2024-09-13T11:07:46.496 INFO:root:teuthology version: 1.2.2.dev2+g3752d38 2024-09-13T11:07:46.519 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-13T11:07:46.606 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/akupczyk-2024-09-13_09:37:56-rados-wip-aclamk-cbt-combined-distro-default-smithi/7903409 branch: wip-aclamk-cbt-combined description: rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{max-simultaneous-scrubs-5} backoff/peering ceph clusters/{fixed-4 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/classic msgr-failures/osd-delay msgr/async-v1only objectstore/{bluestore-options/write$/{write_v2} bluestore/bluestore-stupid} rados supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/admin_socket_objecter_requests} email: akupczyk@redhat.com first_in_suite: false job_id: '7903409' kernel: kdb: 1 sha1: distro ktype: distro last_in_suite: false machine_type: smithi name: akupczyk-2024-09-13_09:37:56-rados-wip-aclamk-cbt-combined-distro-default-smithi no_nested_subset: false openstack: - volumes: count: 4 size: 10 - volumes: count: 4 size: 30 os_type: ubuntu os_version: '22.04' overrides: admin_socket: branch: wip-aclamk-cbt-combined ceph: conf: client.0: admin socket: /var/run/ceph/ceph-$name.asok global: mon client directed command retry: 5 mon election default strategy: 1 ms bind msgr2: false ms inject delay max: 1 ms inject delay probability: 0.005 ms inject delay type: osd ms inject internal delays: 0.002 ms inject socket failures: 2500 ms type: async osd_pool_default_min_size: 2 osd_pool_default_size: 2 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 50 mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 mon scrub interval: 300 paxos service trim min: 10 osd: bdev async discard: true bdev enable discard: true bluestore allocator: stupid bluestore block size: 96636764160 bluestore fsck on mount: true bluestore write v2: true bluestore zero block detection: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd backoff on peering: true osd blocked scrub grace period: 3600 osd debug reject backfill probability: 0.3 osd debug verify cached snaps: true osd debug verify missing on start: true osd delete sleep: 1 osd failsafe full ratio: 0.95 osd inject bad map crc probability: 0.1 osd max backfills: 3 osd max markdown count: 1000 osd max scrubs: 5 osd mclock override recovery settings: true osd mclock profile: high_recovery_ops osd objectstore: bluestore osd op queue: debug_random osd op queue cut off: debug_random osd scrub max interval: 120 osd scrub min interval: 60 osd shutdown pgref assert: true osd snap trim sleep: 2 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(REQUEST_STUCK\) - failed to encode map - \(OSD_SLOW_PING_TIME - but it is still running - objects unfound and apparently lost - overall HEALTH_ - \(OSDMAP_FLAGS\) - \(OSD_ - \(PG_ - \(POOL_ - \(CACHE_POOL_ - \(SMALLER_PGP_NUM\) - \(OBJECT_ - SLOW_OPS - \(REQUEST_SLOW\) - \(TOO_FEW_PGS\) - slow request - timeout on replica - late reservation from - MON_DOWN - OSDMAP_FLAGS - OSD_DOWN - PG_DEGRADED - PG_AVAILABILITY - POOL_APP_NOT_ENABLED - mons down - mon down - out of quorum - noscrub - nodeep-scrub - \(POOL_APP_NOT_ENABLED\) mon_bind_msgr2: false sha1: d58f73069a8157a9ace12a0247ee3e8497707ea9 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: d58f73069a8157a9ace12a0247ee3e8497707ea9 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-aclamk-cbt-combined sha1: d58f73069a8157a9ace12a0247ee3e8497707ea9 owner: scheduled_akupczyk@teuthology priority: 160 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.y - osd.0 - osd.4 - osd.8 - osd.12 - - mon.b - osd.1 - osd.5 - osd.9 - osd.13 - - mon.c - osd.2 - osd.6 - osd.10 - osd.14 - - mgr.x - osd.3 - osd.7 - osd.11 - osd.15 - client.0 seed: 5394 sha1: d58f73069a8157a9ace12a0247ee3e8497707ea9 sleep_before_teardown: 0 subset: 582/12160 suite: rados suite_branch: wip-aclamk-cbt-combined suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_d58f73069a8157a9ace12a0247ee3e8497707ea9/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: d58f73069a8157a9ace12a0247ee3e8497707ea9 targets: smithi001.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPCBua+ZZ7P4Yb2djx+SyGWVot0cO8flB+45cb7GH8y+XVPL1TY5L26qE5XTOM3zgCg9jSHnzqBkdOaG8zj9moU= smithi092.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJwDX5UG6gqL3h+DAMYGoeEhyCSETEk2A9MEUtQzdwwW3GsHQpCTRT2MiYXykpVyRW/Vvi5QDcXBlMSWaETvr80= smithi162.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBE0U3sEdW+13n8O/UQ0/z+qiUKZb6Vil28xhdH0ddggibg460VEQU3kG3IRWKKMoWae7NNNvbEbHdHMmKvoGkGs= smithi177.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKWJ/8t/0RjVNQ3s3lAGz2/jCs/zNbwmvTyMOmsqfB3b/Q47FdXawCs2p1CV8GE6N+jbrr7OSXixRZKM6GdS+kU= tasks: - install: null - ceph: conf: osd: debug monc: 20 - thrashosds: bluestore_new_sharding: random chance_bluestore_reshard: 1 chance_pgnum_grow: 1 chance_pgnum_shrink: 1 chance_pgpnum_fix: 1 min_in: 2 timeout: 1200 - radosbench: clients: - client.0 time: 150 - admin_socket: client.0: objecter_requests: test: http://git.ceph.com/?p={repo};a=blob_plain;f=src/test/admin_socket/objecter_requests;hb={branch} teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a timestamp: 2024-09-13_09:37:56 tube: smithi user: akupczyk verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856 2024-09-13T11:07:46.606 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_d58f73069a8157a9ace12a0247ee3e8497707ea9/qa; will attempt to use it 2024-09-13T11:07:46.607 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_d58f73069a8157a9ace12a0247ee3e8497707ea9/qa/tasks 2024-09-13T11:07:46.607 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-09-13T11:07:46.609 INFO:teuthology.task.internal:Checking packages... 2024-09-13T11:07:46.630 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'default' and ceph hash 'd58f73069a8157a9ace12a0247ee3e8497707ea9' 2024-09-13T11:07:46.630 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-13T11:07:46.630 INFO:teuthology.packaging:ref: None 2024-09-13T11:07:46.630 INFO:teuthology.packaging:tag: None 2024-09-13T11:07:46.630 INFO:teuthology.packaging:branch: wip-aclamk-cbt-combined 2024-09-13T11:07:46.631 INFO:teuthology.packaging:sha1: d58f73069a8157a9ace12a0247ee3e8497707ea9 2024-09-13T11:07:46.631 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&ref=wip-aclamk-cbt-combined 2024-09-13T11:07:46.775 INFO:teuthology.task.internal:Found packages for ceph version 19.3.0-4697-gd58f7306-1jammy 2024-09-13T11:07:46.777 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-09-13T11:07:46.783 INFO:teuthology.task.internal:no buildpackages task found 2024-09-13T11:07:46.783 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-09-13T11:07:46.799 INFO:teuthology.task.internal:Saving configuration 2024-09-13T11:07:46.817 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-09-13T11:07:46.824 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-09-13T11:07:46.846 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi001.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/akupczyk-2024-09-13_09:37:56-rados-wip-aclamk-cbt-combined-distro-default-smithi/7903409', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-13 11:02:00.765262', 'locked_by': 'scheduled_akupczyk@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAICbLhc52UznfyqTyvnHOE/NSfpPakaz6pHYrP57nWBpK'} 2024-09-13T11:07:46.863 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi092.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/akupczyk-2024-09-13_09:37:56-rados-wip-aclamk-cbt-combined-distro-default-smithi/7903409', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-13 11:02:00.766906', 'locked_by': 'scheduled_akupczyk@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIBw2zydBfAw0qcMNjquyK7NyGVG1ZTUGLLgmd9+1l1eY'} 2024-09-13T11:07:46.878 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi162.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/akupczyk-2024-09-13_09:37:56-rados-wip-aclamk-cbt-combined-distro-default-smithi/7903409', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-13 11:02:00.769707', 'locked_by': 'scheduled_akupczyk@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIPLRuWuPEy6Bya2LDRN6qm3saBOTV+FF2737Ex98vEp6'} 2024-09-13T11:07:46.896 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi177.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/akupczyk-2024-09-13_09:37:56-rados-wip-aclamk-cbt-combined-distro-default-smithi/7903409', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-13 11:02:00.768290', 'locked_by': 'scheduled_akupczyk@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIOwhmzIkgd/utDzxtgm4WjTQ3EeCui2DDi853ErGhrVI'} 2024-09-13T11:07:46.896 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-09-13T11:07:46.902 INFO:teuthology.task.internal:roles: ubuntu@smithi001.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'osd.0', 'osd.4', 'osd.8', 'osd.12'] 2024-09-13T11:07:46.903 INFO:teuthology.task.internal:roles: ubuntu@smithi092.front.sepia.ceph.com - ['mon.b', 'osd.1', 'osd.5', 'osd.9', 'osd.13'] 2024-09-13T11:07:46.903 INFO:teuthology.task.internal:roles: ubuntu@smithi162.front.sepia.ceph.com - ['mon.c', 'osd.2', 'osd.6', 'osd.10', 'osd.14'] 2024-09-13T11:07:46.903 INFO:teuthology.task.internal:roles: ubuntu@smithi177.front.sepia.ceph.com - ['mgr.x', 'osd.3', 'osd.7', 'osd.11', 'osd.15', 'client.0'] 2024-09-13T11:07:46.903 INFO:teuthology.run_tasks:Running task console_log... 2024-09-13T11:07:46.997 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f6694bd3be0>, signals=[15]) 2024-09-13T11:07:46.997 INFO:teuthology.run_tasks:Running task internal.connect... 2024-09-13T11:07:47.006 INFO:teuthology.task.internal:Opening connections... 2024-09-13T11:07:47.006 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi001.front.sepia.ceph.com 2024-09-13T11:07:47.007 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi001.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-13T11:07:47.094 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi092.front.sepia.ceph.com 2024-09-13T11:07:47.094 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi092.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-13T11:07:47.173 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi162.front.sepia.ceph.com 2024-09-13T11:07:47.174 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi162.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-13T11:07:47.258 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi177.front.sepia.ceph.com 2024-09-13T11:07:47.259 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi177.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-13T11:07:47.335 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-09-13T11:07:47.342 DEBUG:teuthology.orchestra.run.smithi001:> uname -m 2024-09-13T11:07:47.348 INFO:teuthology.orchestra.run.smithi001.stdout:x86_64 2024-09-13T11:07:47.348 DEBUG:teuthology.orchestra.run.smithi001:> cat /etc/os-release 2024-09-13T11:07:47.396 INFO:teuthology.orchestra.run.smithi001.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS" 2024-09-13T11:07:47.396 INFO:teuthology.orchestra.run.smithi001.stdout:NAME="Ubuntu" 2024-09-13T11:07:47.396 INFO:teuthology.orchestra.run.smithi001.stdout:VERSION_ID="22.04" 2024-09-13T11:07:47.396 INFO:teuthology.orchestra.run.smithi001.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)" 2024-09-13T11:07:47.396 INFO:teuthology.orchestra.run.smithi001.stdout:VERSION_CODENAME=jammy 2024-09-13T11:07:47.396 INFO:teuthology.orchestra.run.smithi001.stdout:ID=ubuntu 2024-09-13T11:07:47.396 INFO:teuthology.orchestra.run.smithi001.stdout:ID_LIKE=debian 2024-09-13T11:07:47.396 INFO:teuthology.orchestra.run.smithi001.stdout:HOME_URL="https://www.ubuntu.com/" 2024-09-13T11:07:47.397 INFO:teuthology.orchestra.run.smithi001.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2024-09-13T11:07:47.397 INFO:teuthology.orchestra.run.smithi001.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2024-09-13T11:07:47.397 INFO:teuthology.orchestra.run.smithi001.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2024-09-13T11:07:47.397 INFO:teuthology.orchestra.run.smithi001.stdout:UBUNTU_CODENAME=jammy 2024-09-13T11:07:47.398 INFO:teuthology.lock.ops:Updating smithi001.front.sepia.ceph.com on lock server 2024-09-13T11:07:47.420 DEBUG:teuthology.orchestra.run.smithi092:> uname -m 2024-09-13T11:07:47.425 INFO:teuthology.orchestra.run.smithi092.stdout:x86_64 2024-09-13T11:07:47.426 DEBUG:teuthology.orchestra.run.smithi092:> cat /etc/os-release 2024-09-13T11:07:47.471 INFO:teuthology.orchestra.run.smithi092.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS" 2024-09-13T11:07:47.471 INFO:teuthology.orchestra.run.smithi092.stdout:NAME="Ubuntu" 2024-09-13T11:07:47.472 INFO:teuthology.orchestra.run.smithi092.stdout:VERSION_ID="22.04" 2024-09-13T11:07:47.472 INFO:teuthology.orchestra.run.smithi092.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)" 2024-09-13T11:07:47.472 INFO:teuthology.orchestra.run.smithi092.stdout:VERSION_CODENAME=jammy 2024-09-13T11:07:47.472 INFO:teuthology.orchestra.run.smithi092.stdout:ID=ubuntu 2024-09-13T11:07:47.472 INFO:teuthology.orchestra.run.smithi092.stdout:ID_LIKE=debian 2024-09-13T11:07:47.472 INFO:teuthology.orchestra.run.smithi092.stdout:HOME_URL="https://www.ubuntu.com/" 2024-09-13T11:07:47.472 INFO:teuthology.orchestra.run.smithi092.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2024-09-13T11:07:47.472 INFO:teuthology.orchestra.run.smithi092.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2024-09-13T11:07:47.472 INFO:teuthology.orchestra.run.smithi092.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2024-09-13T11:07:47.472 INFO:teuthology.orchestra.run.smithi092.stdout:UBUNTU_CODENAME=jammy 2024-09-13T11:07:47.472 INFO:teuthology.lock.ops:Updating smithi092.front.sepia.ceph.com on lock server 2024-09-13T11:07:47.495 DEBUG:teuthology.orchestra.run.smithi162:> uname -m 2024-09-13T11:07:47.501 INFO:teuthology.orchestra.run.smithi162.stdout:x86_64 2024-09-13T11:07:47.501 DEBUG:teuthology.orchestra.run.smithi162:> cat /etc/os-release 2024-09-13T11:07:47.546 INFO:teuthology.orchestra.run.smithi162.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS" 2024-09-13T11:07:47.546 INFO:teuthology.orchestra.run.smithi162.stdout:NAME="Ubuntu" 2024-09-13T11:07:47.546 INFO:teuthology.orchestra.run.smithi162.stdout:VERSION_ID="22.04" 2024-09-13T11:07:47.546 INFO:teuthology.orchestra.run.smithi162.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)" 2024-09-13T11:07:47.546 INFO:teuthology.orchestra.run.smithi162.stdout:VERSION_CODENAME=jammy 2024-09-13T11:07:47.546 INFO:teuthology.orchestra.run.smithi162.stdout:ID=ubuntu 2024-09-13T11:07:47.547 INFO:teuthology.orchestra.run.smithi162.stdout:ID_LIKE=debian 2024-09-13T11:07:47.547 INFO:teuthology.orchestra.run.smithi162.stdout:HOME_URL="https://www.ubuntu.com/" 2024-09-13T11:07:47.547 INFO:teuthology.orchestra.run.smithi162.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2024-09-13T11:07:47.547 INFO:teuthology.orchestra.run.smithi162.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2024-09-13T11:07:47.547 INFO:teuthology.orchestra.run.smithi162.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2024-09-13T11:07:47.547 INFO:teuthology.orchestra.run.smithi162.stdout:UBUNTU_CODENAME=jammy 2024-09-13T11:07:47.547 INFO:teuthology.lock.ops:Updating smithi162.front.sepia.ceph.com on lock server 2024-09-13T11:07:47.570 DEBUG:teuthology.orchestra.run.smithi177:> uname -m 2024-09-13T11:07:47.576 INFO:teuthology.orchestra.run.smithi177.stdout:x86_64 2024-09-13T11:07:47.576 DEBUG:teuthology.orchestra.run.smithi177:> cat /etc/os-release 2024-09-13T11:07:47.624 INFO:teuthology.orchestra.run.smithi177.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS" 2024-09-13T11:07:47.624 INFO:teuthology.orchestra.run.smithi177.stdout:NAME="Ubuntu" 2024-09-13T11:07:47.625 INFO:teuthology.orchestra.run.smithi177.stdout:VERSION_ID="22.04" 2024-09-13T11:07:47.625 INFO:teuthology.orchestra.run.smithi177.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)" 2024-09-13T11:07:47.625 INFO:teuthology.orchestra.run.smithi177.stdout:VERSION_CODENAME=jammy 2024-09-13T11:07:47.625 INFO:teuthology.orchestra.run.smithi177.stdout:ID=ubuntu 2024-09-13T11:07:47.625 INFO:teuthology.orchestra.run.smithi177.stdout:ID_LIKE=debian 2024-09-13T11:07:47.625 INFO:teuthology.orchestra.run.smithi177.stdout:HOME_URL="https://www.ubuntu.com/" 2024-09-13T11:07:47.625 INFO:teuthology.orchestra.run.smithi177.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2024-09-13T11:07:47.625 INFO:teuthology.orchestra.run.smithi177.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2024-09-13T11:07:47.625 INFO:teuthology.orchestra.run.smithi177.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2024-09-13T11:07:47.625 INFO:teuthology.orchestra.run.smithi177.stdout:UBUNTU_CODENAME=jammy 2024-09-13T11:07:47.625 INFO:teuthology.lock.ops:Updating smithi177.front.sepia.ceph.com on lock server 2024-09-13T11:07:47.645 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-09-13T11:07:47.656 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-09-13T11:07:47.820 INFO:teuthology.task.internal:Checking for old test directory... 2024-09-13T11:07:47.821 DEBUG:teuthology.orchestra.run.smithi001:> test '!' -e /home/ubuntu/cephtest 2024-09-13T11:07:47.823 DEBUG:teuthology.orchestra.run.smithi092:> test '!' -e /home/ubuntu/cephtest 2024-09-13T11:07:47.827 DEBUG:teuthology.orchestra.run.smithi162:> test '!' -e /home/ubuntu/cephtest 2024-09-13T11:07:47.830 DEBUG:teuthology.orchestra.run.smithi177:> test '!' -e /home/ubuntu/cephtest 2024-09-13T11:07:47.835 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-09-13T11:07:47.841 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-09-13T11:07:47.842 DEBUG:teuthology.orchestra.run.smithi001:> test -z $(ls -A /var/lib/ceph) 2024-09-13T11:07:47.872 DEBUG:teuthology.orchestra.run.smithi092:> test -z $(ls -A /var/lib/ceph) 2024-09-13T11:07:47.875 DEBUG:teuthology.orchestra.run.smithi162:> test -z $(ls -A /var/lib/ceph) 2024-09-13T11:07:47.877 DEBUG:teuthology.orchestra.run.smithi177:> test -z $(ls -A /var/lib/ceph) 2024-09-13T11:07:48.039 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-09-13T11:07:48.132 INFO:teuthology.run_tasks:Running task kernel... 2024-09-13T11:07:48.146 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-09-13T11:07:48.146 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': 1, 'sha1': 'distro'}, 'mon.b': {'kdb': 1, 'sha1': 'distro'}, 'mon.c': {'kdb': 1, 'sha1': 'distro'}, 'mgr.x': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-09-13T11:07:48.146 DEBUG:teuthology.orchestra.run.smithi001:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-13T11:07:48.147 DEBUG:teuthology.orchestra.run.smithi092:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-13T11:07:48.147 DEBUG:teuthology.orchestra.run.smithi162:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-13T11:07:48.148 DEBUG:teuthology.orchestra.run.smithi177:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-13T11:07:48.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T11:07:48.154 DEBUG:teuthology.orchestra.run.smithi162:> uname -r 2024-09-13T11:07:48.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T11:07:48.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T11:07:48.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T11:07:48.156 DEBUG:teuthology.orchestra.run.smithi177:> uname -r 2024-09-13T11:07:48.157 DEBUG:teuthology.orchestra.run.smithi001:> uname -r 2024-09-13T11:07:48.157 DEBUG:teuthology.orchestra.run.smithi092:> uname -r 2024-09-13T11:07:48.202 INFO:teuthology.orchestra.run.smithi162.stdout:5.15.0-56-generic 2024-09-13T11:07:48.203 INFO:teuthology.task.kernel:Running kernel on smithi162: 5.15.0-56-generic 2024-09-13T11:07:48.203 DEBUG:teuthology.orchestra.run.smithi162:> sudo apt-get clean 2024-09-13T11:07:48.204 INFO:teuthology.orchestra.run.smithi092.stdout:5.15.0-56-generic 2024-09-13T11:07:48.204 INFO:teuthology.orchestra.run.smithi001.stdout:5.15.0-56-generic 2024-09-13T11:07:48.205 INFO:teuthology.task.kernel:Running kernel on smithi092: 5.15.0-56-generic 2024-09-13T11:07:48.205 DEBUG:teuthology.orchestra.run.smithi092:> sudo apt-get clean 2024-09-13T11:07:48.205 INFO:teuthology.task.kernel:Running kernel on smithi001: 5.15.0-56-generic 2024-09-13T11:07:48.205 DEBUG:teuthology.orchestra.run.smithi001:> sudo apt-get clean 2024-09-13T11:07:48.206 INFO:teuthology.orchestra.run.smithi177.stdout:5.15.0-56-generic 2024-09-13T11:07:48.206 INFO:teuthology.task.kernel:Running kernel on smithi177: 5.15.0-56-generic 2024-09-13T11:07:48.206 DEBUG:teuthology.orchestra.run.smithi177:> sudo apt-get clean 2024-09-13T11:07:48.311 DEBUG:teuthology.orchestra.run.smithi001:> sudo apt-get update 2024-09-13T11:07:48.332 DEBUG:teuthology.orchestra.run.smithi162:> sudo apt-get update 2024-09-13T11:07:48.353 DEBUG:teuthology.orchestra.run.smithi177:> sudo apt-get update 2024-09-13T11:07:48.387 DEBUG:teuthology.orchestra.run.smithi092:> sudo apt-get update 2024-09-13T11:07:48.450 INFO:teuthology.orchestra.run.smithi001.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-09-13T11:07:48.451 INFO:teuthology.orchestra.run.smithi001.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [128 kB] 2024-09-13T11:07:48.451 INFO:teuthology.orchestra.run.smithi001.stdout:Get:3 http://security.ubuntu.com/ubuntu jammy-security InRelease [129 kB] 2024-09-13T11:07:48.483 INFO:teuthology.orchestra.run.smithi177.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-09-13T11:07:48.506 INFO:teuthology.orchestra.run.smithi177.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [128 kB] 2024-09-13T11:07:48.523 INFO:teuthology.orchestra.run.smithi001.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease [127 kB] 2024-09-13T11:07:48.579 INFO:teuthology.orchestra.run.smithi177.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-backports InRelease [127 kB] 2024-09-13T11:07:48.599 INFO:teuthology.orchestra.run.smithi177.stdout:Get:4 http://security.ubuntu.com/ubuntu jammy-security InRelease [129 kB] 2024-09-13T11:07:48.607 INFO:teuthology.orchestra.run.smithi162.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-09-13T11:07:48.607 INFO:teuthology.orchestra.run.smithi162.stdout:Get:2 http://security.ubuntu.com/ubuntu jammy-security InRelease [129 kB] 2024-09-13T11:07:48.669 INFO:teuthology.orchestra.run.smithi162.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [128 kB] 2024-09-13T11:07:48.673 INFO:teuthology.orchestra.run.smithi092.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-09-13T11:07:48.675 INFO:teuthology.orchestra.run.smithi092.stdout:Get:2 http://security.ubuntu.com/ubuntu jammy-security InRelease [129 kB] 2024-09-13T11:07:48.698 INFO:teuthology.orchestra.run.smithi001.stdout:Get:5 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages [1,806 kB] 2024-09-13T11:07:48.725 INFO:teuthology.orchestra.run.smithi177.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [2,023 kB] 2024-09-13T11:07:48.729 INFO:teuthology.orchestra.run.smithi092.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [128 kB] 2024-09-13T11:07:48.781 INFO:teuthology.orchestra.run.smithi001.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main i386 Packages [696 kB] 2024-09-13T11:07:48.799 INFO:teuthology.orchestra.run.smithi001.stdout:Get:7 http://security.ubuntu.com/ubuntu jammy-security/main i386 Packages [539 kB] 2024-09-13T11:07:48.805 INFO:teuthology.orchestra.run.smithi001.stdout:Get:8 http://security.ubuntu.com/ubuntu jammy-security/main Translation-en [295 kB] 2024-09-13T11:07:48.808 INFO:teuthology.orchestra.run.smithi001.stdout:Get:9 http://security.ubuntu.com/ubuntu jammy-security/main amd64 c-n-f Metadata [13.3 kB] 2024-09-13T11:07:48.808 INFO:teuthology.orchestra.run.smithi001.stdout:Get:10 http://security.ubuntu.com/ubuntu jammy-security/restricted i386 Packages [37.7 kB] 2024-09-13T11:07:48.809 INFO:teuthology.orchestra.run.smithi001.stdout:Get:11 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [2,377 kB] 2024-09-13T11:07:48.823 INFO:teuthology.orchestra.run.smithi177.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main i386 Packages [696 kB] 2024-09-13T11:07:48.828 INFO:teuthology.orchestra.run.smithi177.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main Translation-en [352 kB] 2024-09-13T11:07:48.831 INFO:teuthology.orchestra.run.smithi177.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 c-n-f Metadata [17.8 kB] 2024-09-13T11:07:48.831 INFO:teuthology.orchestra.run.smithi177.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [2,437 kB] 2024-09-13T11:07:48.833 INFO:teuthology.orchestra.run.smithi001.stdout:Get:12 http://security.ubuntu.com/ubuntu jammy-security/restricted Translation-en [409 kB] 2024-09-13T11:07:48.837 INFO:teuthology.orchestra.run.smithi001.stdout:Get:13 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 c-n-f Metadata [584 B] 2024-09-13T11:07:48.837 INFO:teuthology.orchestra.run.smithi001.stdout:Get:14 http://security.ubuntu.com/ubuntu jammy-security/universe i386 Packages [627 kB] 2024-09-13T11:07:48.839 INFO:teuthology.orchestra.run.smithi001.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [2,023 kB] 2024-09-13T11:07:48.844 INFO:teuthology.orchestra.run.smithi001.stdout:Get:16 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [902 kB] 2024-09-13T11:07:48.850 INFO:teuthology.orchestra.run.smithi177.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy-updates/restricted i386 Packages [39.3 kB] 2024-09-13T11:07:48.850 INFO:teuthology.orchestra.run.smithi177.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy-updates/restricted Translation-en [419 kB] 2024-09-13T11:07:48.853 INFO:teuthology.orchestra.run.smithi177.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 c-n-f Metadata [616 B] 2024-09-13T11:07:48.854 INFO:teuthology.orchestra.run.smithi177.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [1,124 kB] 2024-09-13T11:07:48.854 INFO:teuthology.orchestra.run.smithi001.stdout:Get:17 http://security.ubuntu.com/ubuntu jammy-security/universe Translation-en [176 kB] 2024-09-13T11:07:48.856 INFO:teuthology.orchestra.run.smithi001.stdout:Get:18 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 c-n-f Metadata [19.2 kB] 2024-09-13T11:07:48.856 INFO:teuthology.orchestra.run.smithi001.stdout:Get:19 http://security.ubuntu.com/ubuntu jammy-security/multiverse i386 Packages [1,356 B] 2024-09-13T11:07:48.856 INFO:teuthology.orchestra.run.smithi001.stdout:Get:20 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 Packages [37.2 kB] 2024-09-13T11:07:48.863 INFO:teuthology.orchestra.run.smithi177.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy-updates/universe i386 Packages [732 kB] 2024-09-13T11:07:48.867 INFO:teuthology.orchestra.run.smithi177.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy-updates/universe Translation-en [261 kB] 2024-09-13T11:07:48.870 INFO:teuthology.orchestra.run.smithi177.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 c-n-f Metadata [26.1 kB] 2024-09-13T11:07:48.870 INFO:teuthology.orchestra.run.smithi177.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 Packages [43.3 kB] 2024-09-13T11:07:48.870 INFO:teuthology.orchestra.run.smithi177.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse i386 Packages [4,752 B] 2024-09-13T11:07:48.893 INFO:teuthology.orchestra.run.smithi001.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-updates/main Translation-en [352 kB] 2024-09-13T11:07:48.893 INFO:teuthology.orchestra.run.smithi001.stdout:Get:22 http://security.ubuntu.com/ubuntu jammy-security/multiverse Translation-en [7,588 B] 2024-09-13T11:07:48.894 INFO:teuthology.orchestra.run.smithi001.stdout:Get:23 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 c-n-f Metadata [228 B] 2024-09-13T11:07:48.896 INFO:teuthology.orchestra.run.smithi001.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 c-n-f Metadata [17.8 kB] 2024-09-13T11:07:48.897 INFO:teuthology.orchestra.run.smithi001.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy-updates/restricted i386 Packages [39.3 kB] 2024-09-13T11:07:48.897 INFO:teuthology.orchestra.run.smithi001.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [2,437 kB] 2024-09-13T11:07:48.906 INFO:teuthology.orchestra.run.smithi177.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse Translation-en [10.8 kB] 2024-09-13T11:07:48.907 INFO:teuthology.orchestra.run.smithi177.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 c-n-f Metadata [444 B] 2024-09-13T11:07:48.907 INFO:teuthology.orchestra.run.smithi177.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-backports/main i386 Packages [59.9 kB] 2024-09-13T11:07:48.907 INFO:teuthology.orchestra.run.smithi177.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 Packages [67.8 kB] 2024-09-13T11:07:48.908 INFO:teuthology.orchestra.run.smithi177.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy-backports/main Translation-en [11.1 kB] 2024-09-13T11:07:48.909 INFO:teuthology.orchestra.run.smithi177.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 c-n-f Metadata [388 B] 2024-09-13T11:07:48.909 INFO:teuthology.orchestra.run.smithi177.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy-backports/universe i386 Packages [17.3 kB] 2024-09-13T11:07:48.909 INFO:teuthology.orchestra.run.smithi177.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 Packages [28.8 kB] 2024-09-13T11:07:48.916 INFO:teuthology.orchestra.run.smithi001.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy-updates/restricted Translation-en [419 kB] 2024-09-13T11:07:48.916 INFO:teuthology.orchestra.run.smithi177.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy-backports/universe Translation-en [16.5 kB] 2024-09-13T11:07:48.919 INFO:teuthology.orchestra.run.smithi001.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 c-n-f Metadata [616 B] 2024-09-13T11:07:48.919 INFO:teuthology.orchestra.run.smithi001.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [1,124 kB] 2024-09-13T11:07:48.929 INFO:teuthology.orchestra.run.smithi001.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy-updates/universe i386 Packages [732 kB] 2024-09-13T11:07:48.932 INFO:teuthology.orchestra.run.smithi177.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 c-n-f Metadata [672 B] 2024-09-13T11:07:48.934 INFO:teuthology.orchestra.run.smithi001.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy-updates/universe Translation-en [261 kB] 2024-09-13T11:07:48.936 INFO:teuthology.orchestra.run.smithi001.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 c-n-f Metadata [26.1 kB] 2024-09-13T11:07:48.937 INFO:teuthology.orchestra.run.smithi001.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse i386 Packages [4,752 B] 2024-09-13T11:07:48.937 INFO:teuthology.orchestra.run.smithi001.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 Packages [43.3 kB] 2024-09-13T11:07:48.937 INFO:teuthology.orchestra.run.smithi001.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse Translation-en [10.8 kB] 2024-09-13T11:07:48.947 INFO:teuthology.orchestra.run.smithi001.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 c-n-f Metadata [444 B] 2024-09-13T11:07:48.947 INFO:teuthology.orchestra.run.smithi001.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy-backports/main i386 Packages [59.9 kB] 2024-09-13T11:07:48.948 INFO:teuthology.orchestra.run.smithi001.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 Packages [67.8 kB] 2024-09-13T11:07:48.955 INFO:teuthology.orchestra.run.smithi001.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy-backports/main Translation-en [11.1 kB] 2024-09-13T11:07:48.956 INFO:teuthology.orchestra.run.smithi001.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 c-n-f Metadata [388 B] 2024-09-13T11:07:48.956 INFO:teuthology.orchestra.run.smithi001.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy-backports/universe i386 Packages [17.3 kB] 2024-09-13T11:07:48.956 INFO:teuthology.orchestra.run.smithi001.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 Packages [28.8 kB] 2024-09-13T11:07:48.956 INFO:teuthology.orchestra.run.smithi001.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy-backports/universe Translation-en [16.5 kB] 2024-09-13T11:07:48.964 INFO:teuthology.orchestra.run.smithi001.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 c-n-f Metadata [672 B] 2024-09-13T11:07:49.024 INFO:teuthology.orchestra.run.smithi162.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease [127 kB] 2024-09-13T11:07:49.049 INFO:teuthology.orchestra.run.smithi162.stdout:Get:5 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages [1,806 kB] 2024-09-13T11:07:49.053 INFO:teuthology.orchestra.run.smithi177.stdout:Get:29 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages [1,806 kB] 2024-09-13T11:07:49.084 INFO:teuthology.orchestra.run.smithi092.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease [127 kB] 2024-09-13T11:07:49.102 INFO:teuthology.orchestra.run.smithi092.stdout:Get:5 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages [1,806 kB] 2024-09-13T11:07:49.162 INFO:teuthology.orchestra.run.smithi162.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main i386 Packages [696 kB] 2024-09-13T11:07:49.223 INFO:teuthology.orchestra.run.smithi092.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main i386 Packages [696 kB] 2024-09-13T11:07:49.365 INFO:teuthology.orchestra.run.smithi162.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [2,023 kB] 2024-09-13T11:07:49.429 INFO:teuthology.orchestra.run.smithi092.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [2,023 kB] 2024-09-13T11:07:49.472 INFO:teuthology.orchestra.run.smithi162.stdout:Get:8 http://security.ubuntu.com/ubuntu jammy-security/main i386 Packages [539 kB] 2024-09-13T11:07:49.475 INFO:teuthology.orchestra.run.smithi177.stdout:Get:30 http://security.ubuntu.com/ubuntu jammy-security/main i386 Packages [539 kB] 2024-09-13T11:07:49.488 INFO:teuthology.orchestra.run.smithi162.stdout:Get:9 http://security.ubuntu.com/ubuntu jammy-security/main Translation-en [295 kB] 2024-09-13T11:07:49.492 INFO:teuthology.orchestra.run.smithi177.stdout:Get:31 http://security.ubuntu.com/ubuntu jammy-security/main Translation-en [295 kB] 2024-09-13T11:07:49.496 INFO:teuthology.orchestra.run.smithi162.stdout:Get:10 http://security.ubuntu.com/ubuntu jammy-security/main amd64 c-n-f Metadata [13.3 kB] 2024-09-13T11:07:49.497 INFO:teuthology.orchestra.run.smithi162.stdout:Get:11 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [2,377 kB] 2024-09-13T11:07:49.500 INFO:teuthology.orchestra.run.smithi177.stdout:Get:32 http://security.ubuntu.com/ubuntu jammy-security/main amd64 c-n-f Metadata [13.3 kB] 2024-09-13T11:07:49.501 INFO:teuthology.orchestra.run.smithi177.stdout:Get:33 http://security.ubuntu.com/ubuntu jammy-security/restricted i386 Packages [37.7 kB] 2024-09-13T11:07:49.502 INFO:teuthology.orchestra.run.smithi177.stdout:Get:34 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [2,377 kB] 2024-09-13T11:07:49.516 INFO:teuthology.orchestra.run.smithi162.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy-updates/main Translation-en [352 kB] 2024-09-13T11:07:49.524 INFO:teuthology.orchestra.run.smithi092.stdout:Get:8 http://security.ubuntu.com/ubuntu jammy-security/main i386 Packages [539 kB] 2024-09-13T11:07:49.530 INFO:teuthology.orchestra.run.smithi162.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 c-n-f Metadata [17.8 kB] 2024-09-13T11:07:49.531 INFO:teuthology.orchestra.run.smithi162.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy-updates/restricted i386 Packages [39.3 kB] 2024-09-13T11:07:49.532 INFO:teuthology.orchestra.run.smithi162.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [2,437 kB] 2024-09-13T11:07:49.540 INFO:teuthology.orchestra.run.smithi092.stdout:Get:9 http://security.ubuntu.com/ubuntu jammy-security/main Translation-en [295 kB] 2024-09-13T11:07:49.548 INFO:teuthology.orchestra.run.smithi092.stdout:Get:10 http://security.ubuntu.com/ubuntu jammy-security/main amd64 c-n-f Metadata [13.3 kB] 2024-09-13T11:07:49.549 INFO:teuthology.orchestra.run.smithi092.stdout:Get:11 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [2,377 kB] 2024-09-13T11:07:49.576 INFO:teuthology.orchestra.run.smithi092.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy-updates/main Translation-en [352 kB] 2024-09-13T11:07:49.591 INFO:teuthology.orchestra.run.smithi092.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 c-n-f Metadata [17.8 kB] 2024-09-13T11:07:49.591 INFO:teuthology.orchestra.run.smithi092.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy-updates/restricted i386 Packages [39.3 kB] 2024-09-13T11:07:49.592 INFO:teuthology.orchestra.run.smithi092.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [2,437 kB] 2024-09-13T11:07:49.594 INFO:teuthology.orchestra.run.smithi162.stdout:Get:16 http://security.ubuntu.com/ubuntu jammy-security/restricted i386 Packages [37.7 kB] 2024-09-13T11:07:49.595 INFO:teuthology.orchestra.run.smithi162.stdout:Get:17 http://security.ubuntu.com/ubuntu jammy-security/restricted Translation-en [409 kB] 2024-09-13T11:07:49.599 INFO:teuthology.orchestra.run.smithi177.stdout:Get:35 http://security.ubuntu.com/ubuntu jammy-security/restricted Translation-en [409 kB] 2024-09-13T11:07:49.610 INFO:teuthology.orchestra.run.smithi177.stdout:Get:36 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 c-n-f Metadata [584 B] 2024-09-13T11:07:49.610 INFO:teuthology.orchestra.run.smithi177.stdout:Get:37 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [902 kB] 2024-09-13T11:07:49.645 INFO:teuthology.orchestra.run.smithi092.stdout:Get:16 http://security.ubuntu.com/ubuntu jammy-security/restricted i386 Packages [37.7 kB] 2024-09-13T11:07:49.647 INFO:teuthology.orchestra.run.smithi092.stdout:Get:17 http://security.ubuntu.com/ubuntu jammy-security/restricted Translation-en [409 kB] 2024-09-13T11:07:49.662 INFO:teuthology.orchestra.run.smithi177.stdout:Get:38 http://security.ubuntu.com/ubuntu jammy-security/universe i386 Packages [627 kB] 2024-09-13T11:07:49.666 INFO:teuthology.orchestra.run.smithi162.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy-updates/restricted Translation-en [419 kB] 2024-09-13T11:07:49.666 INFO:teuthology.orchestra.run.smithi162.stdout:Get:19 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 c-n-f Metadata [584 B] 2024-09-13T11:07:49.667 INFO:teuthology.orchestra.run.smithi162.stdout:Get:20 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [902 kB] 2024-09-13T11:07:49.676 INFO:teuthology.orchestra.run.smithi162.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 c-n-f Metadata [616 B] 2024-09-13T11:07:49.677 INFO:teuthology.orchestra.run.smithi162.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy-updates/universe i386 Packages [732 kB] 2024-09-13T11:07:49.681 INFO:teuthology.orchestra.run.smithi177.stdout:Get:39 http://security.ubuntu.com/ubuntu jammy-security/universe Translation-en [176 kB] 2024-09-13T11:07:49.685 INFO:teuthology.orchestra.run.smithi177.stdout:Get:40 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 c-n-f Metadata [19.2 kB] 2024-09-13T11:07:49.686 INFO:teuthology.orchestra.run.smithi162.stdout:Get:23 http://security.ubuntu.com/ubuntu jammy-security/universe i386 Packages [627 kB] 2024-09-13T11:07:49.686 INFO:teuthology.orchestra.run.smithi177.stdout:Get:41 http://security.ubuntu.com/ubuntu jammy-security/multiverse i386 Packages [1,356 B] 2024-09-13T11:07:49.686 INFO:teuthology.orchestra.run.smithi177.stdout:Get:42 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 Packages [37.2 kB] 2024-09-13T11:07:49.687 INFO:teuthology.orchestra.run.smithi177.stdout:Get:43 http://security.ubuntu.com/ubuntu jammy-security/multiverse Translation-en [7,588 B] 2024-09-13T11:07:49.693 INFO:teuthology.orchestra.run.smithi162.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [1,124 kB] 2024-09-13T11:07:49.695 INFO:teuthology.orchestra.run.smithi092.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy-updates/restricted Translation-en [419 kB] 2024-09-13T11:07:49.695 INFO:teuthology.orchestra.run.smithi092.stdout:Get:19 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 c-n-f Metadata [584 B] 2024-09-13T11:07:49.695 INFO:teuthology.orchestra.run.smithi092.stdout:Get:20 http://security.ubuntu.com/ubuntu jammy-security/universe i386 Packages [627 kB] 2024-09-13T11:07:49.699 INFO:teuthology.orchestra.run.smithi162.stdout:Get:25 http://security.ubuntu.com/ubuntu jammy-security/universe Translation-en [176 kB] 2024-09-13T11:07:49.700 INFO:teuthology.orchestra.run.smithi092.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 c-n-f Metadata [616 B] 2024-09-13T11:07:49.700 INFO:teuthology.orchestra.run.smithi092.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [1,124 kB] 2024-09-13T11:07:49.703 INFO:teuthology.orchestra.run.smithi162.stdout:Get:26 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 c-n-f Metadata [19.2 kB] 2024-09-13T11:07:49.704 INFO:teuthology.orchestra.run.smithi162.stdout:Get:27 http://security.ubuntu.com/ubuntu jammy-security/multiverse i386 Packages [1,356 B] 2024-09-13T11:07:49.704 INFO:teuthology.orchestra.run.smithi162.stdout:Get:28 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 Packages [37.2 kB] 2024-09-13T11:07:49.706 INFO:teuthology.orchestra.run.smithi092.stdout:Get:23 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [902 kB] 2024-09-13T11:07:49.712 INFO:teuthology.orchestra.run.smithi162.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy-updates/universe Translation-en [261 kB] 2024-09-13T11:07:49.712 INFO:teuthology.orchestra.run.smithi092.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy-updates/universe i386 Packages [732 kB] 2024-09-13T11:07:49.715 INFO:teuthology.orchestra.run.smithi162.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 c-n-f Metadata [26.1 kB] 2024-09-13T11:07:49.716 INFO:teuthology.orchestra.run.smithi162.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse i386 Packages [4,752 B] 2024-09-13T11:07:49.716 INFO:teuthology.orchestra.run.smithi162.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 Packages [43.3 kB] 2024-09-13T11:07:49.717 INFO:teuthology.orchestra.run.smithi162.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse Translation-en [10.8 kB] 2024-09-13T11:07:49.722 INFO:teuthology.orchestra.run.smithi092.stdout:Get:25 http://security.ubuntu.com/ubuntu jammy-security/universe Translation-en [176 kB] 2024-09-13T11:07:49.725 INFO:teuthology.orchestra.run.smithi092.stdout:Get:26 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 c-n-f Metadata [19.2 kB] 2024-09-13T11:07:49.726 INFO:teuthology.orchestra.run.smithi092.stdout:Get:27 http://security.ubuntu.com/ubuntu jammy-security/multiverse i386 Packages [1,356 B] 2024-09-13T11:07:49.726 INFO:teuthology.orchestra.run.smithi092.stdout:Get:28 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 Packages [37.2 kB] 2024-09-13T11:07:49.728 INFO:teuthology.orchestra.run.smithi092.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy-updates/universe Translation-en [261 kB] 2024-09-13T11:07:49.734 INFO:teuthology.orchestra.run.smithi092.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 c-n-f Metadata [26.1 kB] 2024-09-13T11:07:49.735 INFO:teuthology.orchestra.run.smithi162.stdout:Get:34 http://security.ubuntu.com/ubuntu jammy-security/multiverse Translation-en [7,588 B] 2024-09-13T11:07:49.736 INFO:teuthology.orchestra.run.smithi162.stdout:Get:35 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 c-n-f Metadata [228 B] 2024-09-13T11:07:49.739 INFO:teuthology.orchestra.run.smithi177.stdout:Get:44 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 c-n-f Metadata [228 B] 2024-09-13T11:07:49.739 INFO:teuthology.orchestra.run.smithi092.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse i386 Packages [4,752 B] 2024-09-13T11:07:49.740 INFO:teuthology.orchestra.run.smithi092.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 Packages [43.3 kB] 2024-09-13T11:07:49.741 INFO:teuthology.orchestra.run.smithi092.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse Translation-en [10.8 kB] 2024-09-13T11:07:49.765 INFO:teuthology.orchestra.run.smithi162.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 c-n-f Metadata [444 B] 2024-09-13T11:07:49.765 INFO:teuthology.orchestra.run.smithi162.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy-backports/main i386 Packages [59.9 kB] 2024-09-13T11:07:49.766 INFO:teuthology.orchestra.run.smithi162.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 Packages [67.8 kB] 2024-09-13T11:07:49.786 INFO:teuthology.orchestra.run.smithi092.stdout:Get:34 http://security.ubuntu.com/ubuntu jammy-security/multiverse Translation-en [7,588 B] 2024-09-13T11:07:49.786 INFO:teuthology.orchestra.run.smithi092.stdout:Get:35 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 c-n-f Metadata [228 B] 2024-09-13T11:07:49.799 INFO:teuthology.orchestra.run.smithi162.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy-backports/main Translation-en [11.1 kB] 2024-09-13T11:07:49.799 INFO:teuthology.orchestra.run.smithi162.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 c-n-f Metadata [388 B] 2024-09-13T11:07:49.799 INFO:teuthology.orchestra.run.smithi162.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy-backports/universe i386 Packages [17.3 kB] 2024-09-13T11:07:49.800 INFO:teuthology.orchestra.run.smithi162.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 Packages [28.8 kB] 2024-09-13T11:07:49.800 INFO:teuthology.orchestra.run.smithi162.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy-backports/universe Translation-en [16.5 kB] 2024-09-13T11:07:49.816 INFO:teuthology.orchestra.run.smithi092.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 c-n-f Metadata [444 B] 2024-09-13T11:07:49.816 INFO:teuthology.orchestra.run.smithi092.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 Packages [67.8 kB] 2024-09-13T11:07:49.817 INFO:teuthology.orchestra.run.smithi092.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy-backports/main i386 Packages [59.9 kB] 2024-09-13T11:07:49.818 INFO:teuthology.orchestra.run.smithi092.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy-backports/main Translation-en [11.1 kB] 2024-09-13T11:07:49.818 INFO:teuthology.orchestra.run.smithi092.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 c-n-f Metadata [388 B] 2024-09-13T11:07:49.820 INFO:teuthology.orchestra.run.smithi092.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy-backports/universe i386 Packages [17.3 kB] 2024-09-13T11:07:49.821 INFO:teuthology.orchestra.run.smithi092.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 Packages [28.8 kB] 2024-09-13T11:07:49.846 INFO:teuthology.orchestra.run.smithi162.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 c-n-f Metadata [672 B] 2024-09-13T11:07:49.902 INFO:teuthology.orchestra.run.smithi092.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy-backports/universe Translation-en [16.5 kB] 2024-09-13T11:07:49.902 INFO:teuthology.orchestra.run.smithi092.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 c-n-f Metadata [672 B] 2024-09-13T11:07:55.082 INFO:teuthology.orchestra.run.smithi177.stdout:Fetched 16.0 MB in 2s (7,482 kB/s) 2024-09-13T11:07:55.610 INFO:teuthology.orchestra.run.smithi001.stdout:Fetched 16.0 MB in 2s (7,103 kB/s) 2024-09-13T11:07:55.852 INFO:teuthology.orchestra.run.smithi162.stdout:Fetched 16.0 MB in 3s (5,838 kB/s) 2024-09-13T11:07:55.903 INFO:teuthology.orchestra.run.smithi092.stdout:Fetched 16.0 MB in 3s (5,797 kB/s) 2024-09-13T11:07:56.741 INFO:teuthology.orchestra.run.smithi177.stdout:Reading package lists... 2024-09-13T11:07:56.759 DEBUG:teuthology.orchestra.run.smithi177:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-09-13T11:07:56.826 INFO:teuthology.orchestra.run.smithi177.stdout:Reading package lists... 2024-09-13T11:07:57.047 INFO:teuthology.orchestra.run.smithi177.stdout:Building dependency tree... 2024-09-13T11:07:57.047 INFO:teuthology.orchestra.run.smithi177.stdout:Reading state information... 2024-09-13T11:07:57.100 INFO:teuthology.orchestra.run.smithi001.stdout:Reading package lists... 2024-09-13T11:07:57.119 DEBUG:teuthology.orchestra.run.smithi001:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-09-13T11:07:57.184 INFO:teuthology.orchestra.run.smithi001.stdout:Reading package lists... 2024-09-13T11:07:57.229 INFO:teuthology.orchestra.run.smithi177.stdout:The following packages were automatically installed and are no longer required: 2024-09-13T11:07:57.229 INFO:teuthology.orchestra.run.smithi177.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-09-13T11:07:57.229 INFO:teuthology.orchestra.run.smithi177.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-09-13T11:07:57.229 INFO:teuthology.orchestra.run.smithi177.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-09-13T11:07:57.229 INFO:teuthology.orchestra.run.smithi177.stdout: python2 python2-dev python2-minimal python2.7 python2.7-dev 2024-09-13T11:07:57.229 INFO:teuthology.orchestra.run.smithi177.stdout: python2.7-minimal 2024-09-13T11:07:57.229 INFO:teuthology.orchestra.run.smithi177.stdout:Use 'sudo apt autoremove' to remove them. 2024-09-13T11:07:57.231 INFO:teuthology.orchestra.run.smithi177.stdout:The following additional packages will be installed: 2024-09-13T11:07:57.231 INFO:teuthology.orchestra.run.smithi177.stdout: linux-generic linux-headers-5.15.0-121 linux-headers-5.15.0-121-generic 2024-09-13T11:07:57.231 INFO:teuthology.orchestra.run.smithi177.stdout: linux-headers-generic linux-image-5.15.0-121-generic 2024-09-13T11:07:57.231 INFO:teuthology.orchestra.run.smithi177.stdout: linux-modules-5.15.0-121-generic linux-modules-extra-5.15.0-121-generic 2024-09-13T11:07:57.233 INFO:teuthology.orchestra.run.smithi177.stdout:Suggested packages: 2024-09-13T11:07:57.233 INFO:teuthology.orchestra.run.smithi177.stdout: fdutils linux-doc | linux-source-5.15.0 linux-tools 2024-09-13T11:07:57.233 INFO:teuthology.orchestra.run.smithi177.stdout:Recommended packages: 2024-09-13T11:07:57.233 INFO:teuthology.orchestra.run.smithi177.stdout: thermald 2024-09-13T11:07:57.278 INFO:teuthology.orchestra.run.smithi177.stdout:The following NEW packages will be installed: 2024-09-13T11:07:57.278 INFO:teuthology.orchestra.run.smithi177.stdout: linux-headers-5.15.0-121 linux-headers-5.15.0-121-generic 2024-09-13T11:07:57.278 INFO:teuthology.orchestra.run.smithi177.stdout: linux-image-5.15.0-121-generic linux-modules-5.15.0-121-generic 2024-09-13T11:07:57.278 INFO:teuthology.orchestra.run.smithi177.stdout: linux-modules-extra-5.15.0-121-generic 2024-09-13T11:07:57.279 INFO:teuthology.orchestra.run.smithi177.stdout:The following packages will be upgraded: 2024-09-13T11:07:57.280 INFO:teuthology.orchestra.run.smithi177.stdout: linux-generic linux-headers-generic linux-image-generic 2024-09-13T11:07:57.309 INFO:teuthology.orchestra.run.smithi162.stdout:Reading package lists... 2024-09-13T11:07:57.327 DEBUG:teuthology.orchestra.run.smithi162:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-09-13T11:07:57.330 INFO:teuthology.orchestra.run.smithi177.stdout:3 upgraded, 5 newly installed, 0 to remove and 344 not upgraded. 2024-09-13T11:07:57.330 INFO:teuthology.orchestra.run.smithi177.stdout:Need to get 113 MB of archives. 2024-09-13T11:07:57.330 INFO:teuthology.orchestra.run.smithi177.stdout:After this operation, 584 MB of additional disk space will be used. 2024-09-13T11:07:57.330 INFO:teuthology.orchestra.run.smithi177.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-5.15.0-121-generic amd64 5.15.0-121.131 [22.7 MB] 2024-09-13T11:07:57.389 INFO:teuthology.orchestra.run.smithi162.stdout:Reading package lists... 2024-09-13T11:07:57.397 INFO:teuthology.orchestra.run.smithi001.stdout:Building dependency tree... 2024-09-13T11:07:57.398 INFO:teuthology.orchestra.run.smithi001.stdout:Reading state information... 2024-09-13T11:07:57.574 INFO:teuthology.orchestra.run.smithi001.stdout:The following packages were automatically installed and are no longer required: 2024-09-13T11:07:57.574 INFO:teuthology.orchestra.run.smithi001.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-09-13T11:07:57.575 INFO:teuthology.orchestra.run.smithi001.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-09-13T11:07:57.575 INFO:teuthology.orchestra.run.smithi001.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-09-13T11:07:57.575 INFO:teuthology.orchestra.run.smithi001.stdout: python2 python2-dev python2-minimal python2.7 python2.7-dev 2024-09-13T11:07:57.575 INFO:teuthology.orchestra.run.smithi001.stdout: python2.7-minimal 2024-09-13T11:07:57.575 INFO:teuthology.orchestra.run.smithi001.stdout:Use 'sudo apt autoremove' to remove them. 2024-09-13T11:07:57.576 INFO:teuthology.orchestra.run.smithi001.stdout:The following additional packages will be installed: 2024-09-13T11:07:57.576 INFO:teuthology.orchestra.run.smithi001.stdout: linux-generic linux-headers-5.15.0-121 linux-headers-5.15.0-121-generic 2024-09-13T11:07:57.576 INFO:teuthology.orchestra.run.smithi001.stdout: linux-headers-generic linux-image-5.15.0-121-generic 2024-09-13T11:07:57.577 INFO:teuthology.orchestra.run.smithi001.stdout: linux-modules-5.15.0-121-generic linux-modules-extra-5.15.0-121-generic 2024-09-13T11:07:57.577 INFO:teuthology.orchestra.run.smithi001.stdout:Suggested packages: 2024-09-13T11:07:57.578 INFO:teuthology.orchestra.run.smithi001.stdout: fdutils linux-doc | linux-source-5.15.0 linux-tools 2024-09-13T11:07:57.578 INFO:teuthology.orchestra.run.smithi001.stdout:Recommended packages: 2024-09-13T11:07:57.578 INFO:teuthology.orchestra.run.smithi001.stdout: thermald 2024-09-13T11:07:57.600 INFO:teuthology.orchestra.run.smithi162.stdout:Building dependency tree... 2024-09-13T11:07:57.601 INFO:teuthology.orchestra.run.smithi162.stdout:Reading state information... 2024-09-13T11:07:57.626 INFO:teuthology.orchestra.run.smithi092.stdout:Reading package lists... 2024-09-13T11:07:57.628 INFO:teuthology.orchestra.run.smithi001.stdout:The following NEW packages will be installed: 2024-09-13T11:07:57.628 INFO:teuthology.orchestra.run.smithi001.stdout: linux-headers-5.15.0-121 linux-headers-5.15.0-121-generic 2024-09-13T11:07:57.628 INFO:teuthology.orchestra.run.smithi001.stdout: linux-image-5.15.0-121-generic linux-modules-5.15.0-121-generic 2024-09-13T11:07:57.628 INFO:teuthology.orchestra.run.smithi001.stdout: linux-modules-extra-5.15.0-121-generic 2024-09-13T11:07:57.630 INFO:teuthology.orchestra.run.smithi001.stdout:The following packages will be upgraded: 2024-09-13T11:07:57.630 INFO:teuthology.orchestra.run.smithi001.stdout: linux-generic linux-headers-generic linux-image-generic 2024-09-13T11:07:57.645 DEBUG:teuthology.orchestra.run.smithi092:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-09-13T11:07:57.682 INFO:teuthology.orchestra.run.smithi001.stdout:3 upgraded, 5 newly installed, 0 to remove and 344 not upgraded. 2024-09-13T11:07:57.682 INFO:teuthology.orchestra.run.smithi001.stdout:Need to get 113 MB of archives. 2024-09-13T11:07:57.682 INFO:teuthology.orchestra.run.smithi001.stdout:After this operation, 584 MB of additional disk space will be used. 2024-09-13T11:07:57.682 INFO:teuthology.orchestra.run.smithi001.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-5.15.0-121-generic amd64 5.15.0-121.131 [22.7 MB] 2024-09-13T11:07:57.741 INFO:teuthology.orchestra.run.smithi092.stdout:Reading package lists... 2024-09-13T11:07:57.770 INFO:teuthology.orchestra.run.smithi177.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-5.15.0-121-generic amd64 5.15.0-121.131 [11.5 MB] 2024-09-13T11:07:57.779 INFO:teuthology.orchestra.run.smithi162.stdout:The following packages were automatically installed and are no longer required: 2024-09-13T11:07:57.780 INFO:teuthology.orchestra.run.smithi162.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-09-13T11:07:57.780 INFO:teuthology.orchestra.run.smithi162.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-09-13T11:07:57.780 INFO:teuthology.orchestra.run.smithi162.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-09-13T11:07:57.781 INFO:teuthology.orchestra.run.smithi162.stdout: python2 python2-dev python2-minimal python2.7 python2.7-dev 2024-09-13T11:07:57.781 INFO:teuthology.orchestra.run.smithi162.stdout: python2.7-minimal 2024-09-13T11:07:57.781 INFO:teuthology.orchestra.run.smithi162.stdout:Use 'sudo apt autoremove' to remove them. 2024-09-13T11:07:57.782 INFO:teuthology.orchestra.run.smithi162.stdout:The following additional packages will be installed: 2024-09-13T11:07:57.782 INFO:teuthology.orchestra.run.smithi162.stdout: linux-generic linux-headers-5.15.0-121 linux-headers-5.15.0-121-generic 2024-09-13T11:07:57.782 INFO:teuthology.orchestra.run.smithi162.stdout: linux-headers-generic linux-image-5.15.0-121-generic 2024-09-13T11:07:57.782 INFO:teuthology.orchestra.run.smithi162.stdout: linux-modules-5.15.0-121-generic linux-modules-extra-5.15.0-121-generic 2024-09-13T11:07:57.783 INFO:teuthology.orchestra.run.smithi162.stdout:Suggested packages: 2024-09-13T11:07:57.783 INFO:teuthology.orchestra.run.smithi162.stdout: fdutils linux-doc | linux-source-5.15.0 linux-tools 2024-09-13T11:07:57.783 INFO:teuthology.orchestra.run.smithi162.stdout:Recommended packages: 2024-09-13T11:07:57.784 INFO:teuthology.orchestra.run.smithi162.stdout: thermald 2024-09-13T11:07:57.828 INFO:teuthology.orchestra.run.smithi162.stdout:The following NEW packages will be installed: 2024-09-13T11:07:57.828 INFO:teuthology.orchestra.run.smithi162.stdout: linux-headers-5.15.0-121 linux-headers-5.15.0-121-generic 2024-09-13T11:07:57.828 INFO:teuthology.orchestra.run.smithi162.stdout: linux-image-5.15.0-121-generic linux-modules-5.15.0-121-generic 2024-09-13T11:07:57.828 INFO:teuthology.orchestra.run.smithi162.stdout: linux-modules-extra-5.15.0-121-generic 2024-09-13T11:07:57.829 INFO:teuthology.orchestra.run.smithi162.stdout:The following packages will be upgraded: 2024-09-13T11:07:57.830 INFO:teuthology.orchestra.run.smithi162.stdout: linux-generic linux-headers-generic linux-image-generic 2024-09-13T11:07:57.877 INFO:teuthology.orchestra.run.smithi177.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-extra-5.15.0-121-generic amd64 5.15.0-121.131 [63.9 MB] 2024-09-13T11:07:57.881 INFO:teuthology.orchestra.run.smithi162.stdout:3 upgraded, 5 newly installed, 0 to remove and 344 not upgraded. 2024-09-13T11:07:57.881 INFO:teuthology.orchestra.run.smithi162.stdout:Need to get 113 MB of archives. 2024-09-13T11:07:57.881 INFO:teuthology.orchestra.run.smithi162.stdout:After this operation, 584 MB of additional disk space will be used. 2024-09-13T11:07:57.881 INFO:teuthology.orchestra.run.smithi162.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-5.15.0-121-generic amd64 5.15.0-121.131 [22.7 MB] 2024-09-13T11:07:57.927 INFO:teuthology.orchestra.run.smithi092.stdout:Building dependency tree... 2024-09-13T11:07:57.928 INFO:teuthology.orchestra.run.smithi092.stdout:Reading state information... 2024-09-13T11:07:58.095 INFO:teuthology.orchestra.run.smithi092.stdout:The following packages were automatically installed and are no longer required: 2024-09-13T11:07:58.095 INFO:teuthology.orchestra.run.smithi092.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-09-13T11:07:58.095 INFO:teuthology.orchestra.run.smithi092.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-09-13T11:07:58.096 INFO:teuthology.orchestra.run.smithi092.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-09-13T11:07:58.096 INFO:teuthology.orchestra.run.smithi092.stdout: python2 python2-dev python2-minimal python2.7 python2.7-dev 2024-09-13T11:07:58.096 INFO:teuthology.orchestra.run.smithi092.stdout: python2.7-minimal 2024-09-13T11:07:58.096 INFO:teuthology.orchestra.run.smithi092.stdout:Use 'sudo apt autoremove' to remove them. 2024-09-13T11:07:58.097 INFO:teuthology.orchestra.run.smithi092.stdout:The following additional packages will be installed: 2024-09-13T11:07:58.097 INFO:teuthology.orchestra.run.smithi092.stdout: linux-generic linux-headers-5.15.0-121 linux-headers-5.15.0-121-generic 2024-09-13T11:07:58.097 INFO:teuthology.orchestra.run.smithi092.stdout: linux-headers-generic linux-image-5.15.0-121-generic 2024-09-13T11:07:58.098 INFO:teuthology.orchestra.run.smithi092.stdout: linux-modules-5.15.0-121-generic linux-modules-extra-5.15.0-121-generic 2024-09-13T11:07:58.099 INFO:teuthology.orchestra.run.smithi092.stdout:Suggested packages: 2024-09-13T11:07:58.099 INFO:teuthology.orchestra.run.smithi092.stdout: fdutils linux-doc | linux-source-5.15.0 linux-tools 2024-09-13T11:07:58.099 INFO:teuthology.orchestra.run.smithi092.stdout:Recommended packages: 2024-09-13T11:07:58.099 INFO:teuthology.orchestra.run.smithi092.stdout: thermald 2024-09-13T11:07:58.136 INFO:teuthology.orchestra.run.smithi001.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-5.15.0-121-generic amd64 5.15.0-121.131 [11.5 MB] 2024-09-13T11:07:58.145 INFO:teuthology.orchestra.run.smithi092.stdout:The following NEW packages will be installed: 2024-09-13T11:07:58.145 INFO:teuthology.orchestra.run.smithi092.stdout: linux-headers-5.15.0-121 linux-headers-5.15.0-121-generic 2024-09-13T11:07:58.145 INFO:teuthology.orchestra.run.smithi092.stdout: linux-image-5.15.0-121-generic linux-modules-5.15.0-121-generic 2024-09-13T11:07:58.146 INFO:teuthology.orchestra.run.smithi092.stdout: linux-modules-extra-5.15.0-121-generic 2024-09-13T11:07:58.147 INFO:teuthology.orchestra.run.smithi092.stdout:The following packages will be upgraded: 2024-09-13T11:07:58.147 INFO:teuthology.orchestra.run.smithi092.stdout: linux-generic linux-headers-generic linux-image-generic 2024-09-13T11:07:58.251 INFO:teuthology.orchestra.run.smithi001.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-extra-5.15.0-121-generic amd64 5.15.0-121.131 [63.9 MB] 2024-09-13T11:07:58.308 INFO:teuthology.orchestra.run.smithi162.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-5.15.0-121-generic amd64 5.15.0-121.131 [11.5 MB] 2024-09-13T11:07:58.329 INFO:teuthology.orchestra.run.smithi092.stdout:3 upgraded, 5 newly installed, 0 to remove and 344 not upgraded. 2024-09-13T11:07:58.329 INFO:teuthology.orchestra.run.smithi092.stdout:Need to get 113 MB of archives. 2024-09-13T11:07:58.329 INFO:teuthology.orchestra.run.smithi092.stdout:After this operation, 584 MB of additional disk space will be used. 2024-09-13T11:07:58.329 INFO:teuthology.orchestra.run.smithi092.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-5.15.0-121-generic amd64 5.15.0-121.131 [22.7 MB] 2024-09-13T11:07:58.403 INFO:teuthology.orchestra.run.smithi162.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-extra-5.15.0-121-generic amd64 5.15.0-121.131 [63.9 MB] 2024-09-13T11:07:58.676 INFO:teuthology.orchestra.run.smithi177.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-generic amd64 5.15.0.121.121 [1,694 B] 2024-09-13T11:07:58.676 INFO:teuthology.orchestra.run.smithi177.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-generic amd64 5.15.0.121.121 [2,464 B] 2024-09-13T11:07:58.676 INFO:teuthology.orchestra.run.smithi177.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-121 all 5.15.0-121.131 [12.3 MB] 2024-09-13T11:07:58.795 INFO:teuthology.orchestra.run.smithi177.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-121-generic amd64 5.15.0-121.131 [2,827 kB] 2024-09-13T11:07:58.817 INFO:teuthology.orchestra.run.smithi177.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-generic amd64 5.15.0.121.121 [2,316 B] 2024-09-13T11:07:59.105 INFO:teuthology.orchestra.run.smithi001.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-generic amd64 5.15.0.121.121 [1,694 B] 2024-09-13T11:07:59.105 INFO:teuthology.orchestra.run.smithi001.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-generic amd64 5.15.0.121.121 [2,464 B] 2024-09-13T11:07:59.105 INFO:teuthology.orchestra.run.smithi001.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-121 all 5.15.0-121.131 [12.3 MB] 2024-09-13T11:07:59.205 INFO:teuthology.orchestra.run.smithi162.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-generic amd64 5.15.0.121.121 [1,694 B] 2024-09-13T11:07:59.205 INFO:teuthology.orchestra.run.smithi162.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-generic amd64 5.15.0.121.121 [2,464 B] 2024-09-13T11:07:59.205 INFO:teuthology.orchestra.run.smithi162.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-121 all 5.15.0-121.131 [12.3 MB] 2024-09-13T11:07:59.214 INFO:teuthology.orchestra.run.smithi177.stdout:Fetched 113 MB in 2s (74.0 MB/s) 2024-09-13T11:07:59.238 INFO:teuthology.orchestra.run.smithi001.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-121-generic amd64 5.15.0-121.131 [2,827 kB] 2024-09-13T11:07:59.260 INFO:teuthology.orchestra.run.smithi001.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-generic amd64 5.15.0.121.121 [2,316 B] 2024-09-13T11:07:59.324 INFO:teuthology.orchestra.run.smithi162.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-121-generic amd64 5.15.0-121.131 [2,827 kB] 2024-09-13T11:07:59.398 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package linux-modules-5.15.0-121-generic. 2024-09-13T11:07:59.425 INFO:teuthology.orchestra.run.smithi162.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-generic amd64 5.15.0.121.121 [2,316 B] 2024-09-13T11:07:59.662 INFO:teuthology.orchestra.run.smithi001.stdout:Fetched 113 MB in 2s (69.8 MB/s) 2024-09-13T11:07:59.759 INFO:teuthology.orchestra.run.smithi092.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-5.15.0-121-generic amd64 5.15.0-121.131 [11.5 MB] 2024-09-13T11:07:59.803 INFO:teuthology.orchestra.run.smithi162.stdout:Fetched 113 MB in 2s (71.3 MB/s) 2024-09-13T11:07:59.834 INFO:teuthology.orchestra.run.smithi001.stdout:Selecting previously unselected package linux-modules-5.15.0-121-generic. 2024-09-13T11:07:59.981 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package linux-modules-5.15.0-121-generic. 2024-09-13T11:08:00.109 INFO:teuthology.orchestra.run.smithi092.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-extra-5.15.0-121-generic amd64 5.15.0-121.131 [63.9 MB] 2024-09-13T11:08:01.387 INFO:teuthology.orchestra.run.smithi177.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 129542 files and directories currently installed.) 2024-09-13T11:08:01.391 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../0-linux-modules-5.15.0-121-generic_5.15.0-121.131_amd64.deb ... 2024-09-13T11:08:01.519 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking linux-modules-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-13T11:08:01.804 INFO:teuthology.orchestra.run.smithi001.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 129542 files and directories currently installed.) 2024-09-13T11:08:01.808 INFO:teuthology.orchestra.run.smithi001.stdout:Preparing to unpack .../0-linux-modules-5.15.0-121-generic_5.15.0-121.131_amd64.deb ... 2024-09-13T11:08:01.865 INFO:teuthology.orchestra.run.smithi001.stdout:Unpacking linux-modules-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-13T11:08:01.961 INFO:teuthology.orchestra.run.smithi162.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 129542 files and directories currently installed.) 2024-09-13T11:08:01.966 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../0-linux-modules-5.15.0-121-generic_5.15.0-121.131_amd64.deb ... 2024-09-13T11:08:02.045 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking linux-modules-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-13T11:08:02.607 INFO:teuthology.orchestra.run.smithi092.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-generic amd64 5.15.0.121.121 [1,694 B] 2024-09-13T11:08:02.608 INFO:teuthology.orchestra.run.smithi092.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-generic amd64 5.15.0.121.121 [2,464 B] 2024-09-13T11:08:02.608 INFO:teuthology.orchestra.run.smithi092.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-121 all 5.15.0-121.131 [12.3 MB] 2024-09-13T11:08:02.951 INFO:teuthology.orchestra.run.smithi092.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-121-generic amd64 5.15.0-121.131 [2,827 kB] 2024-09-13T11:08:03.031 INFO:teuthology.orchestra.run.smithi092.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-generic amd64 5.15.0.121.121 [2,316 B] 2024-09-13T11:08:03.477 INFO:teuthology.orchestra.run.smithi092.stdout:Fetched 113 MB in 5s (23.2 MB/s) 2024-09-13T11:08:03.553 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package linux-image-5.15.0-121-generic. 2024-09-13T11:08:03.574 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../1-linux-image-5.15.0-121-generic_5.15.0-121.131_amd64.deb ... 2024-09-13T11:08:03.656 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking linux-image-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-13T11:08:03.664 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package linux-modules-5.15.0-121-generic. 2024-09-13T11:08:03.924 INFO:teuthology.orchestra.run.smithi001.stdout:Selecting previously unselected package linux-image-5.15.0-121-generic. 2024-09-13T11:08:03.946 INFO:teuthology.orchestra.run.smithi001.stdout:Preparing to unpack .../1-linux-image-5.15.0-121-generic_5.15.0-121.131_amd64.deb ... 2024-09-13T11:08:04.035 INFO:teuthology.orchestra.run.smithi001.stdout:Unpacking linux-image-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-13T11:08:04.089 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package linux-modules-extra-5.15.0-121-generic. 2024-09-13T11:08:04.110 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../2-linux-modules-extra-5.15.0-121-generic_5.15.0-121.131_amd64.deb ... 2024-09-13T11:08:04.148 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking linux-modules-extra-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-13T11:08:04.262 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package linux-image-5.15.0-121-generic. 2024-09-13T11:08:04.284 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../1-linux-image-5.15.0-121-generic_5.15.0-121.131_amd64.deb ... 2024-09-13T11:08:04.384 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking linux-image-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-13T11:08:04.711 INFO:teuthology.orchestra.run.smithi001.stdout:Selecting previously unselected package linux-modules-extra-5.15.0-121-generic. 2024-09-13T11:08:04.731 INFO:teuthology.orchestra.run.smithi001.stdout:Preparing to unpack .../2-linux-modules-extra-5.15.0-121-generic_5.15.0-121.131_amd64.deb ... 2024-09-13T11:08:04.769 INFO:teuthology.orchestra.run.smithi001.stdout:Unpacking linux-modules-extra-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-13T11:08:04.849 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package linux-modules-extra-5.15.0-121-generic. 2024-09-13T11:08:04.870 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../2-linux-modules-extra-5.15.0-121-generic_5.15.0-121.131_amd64.deb ... 2024-09-13T11:08:04.899 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking linux-modules-extra-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-13T11:08:05.625 INFO:teuthology.orchestra.run.smithi092.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 129542 files and directories currently installed.) 2024-09-13T11:08:05.629 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../0-linux-modules-5.15.0-121-generic_5.15.0-121.131_amd64.deb ... 2024-09-13T11:08:05.711 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking linux-modules-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-13T11:08:07.787 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package linux-image-5.15.0-121-generic. 2024-09-13T11:08:07.807 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../1-linux-image-5.15.0-121-generic_5.15.0-121.131_amd64.deb ... 2024-09-13T11:08:07.887 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking linux-image-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-13T11:08:08.315 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package linux-modules-extra-5.15.0-121-generic. 2024-09-13T11:08:08.336 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../2-linux-modules-extra-5.15.0-121-generic_5.15.0-121.131_amd64.deb ... 2024-09-13T11:08:08.365 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking linux-modules-extra-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-13T11:08:10.545 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../3-linux-generic_5.15.0.121.121_amd64.deb ... 2024-09-13T11:08:10.672 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking linux-generic (5.15.0.121.121) over (5.15.0.56.54) ... 2024-09-13T11:08:10.996 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../4-linux-image-generic_5.15.0.121.121_amd64.deb ... 2024-09-13T11:08:11.109 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking linux-image-generic (5.15.0.121.121) over (5.15.0.56.54) ... 2024-09-13T11:08:11.284 INFO:teuthology.orchestra.run.smithi001.stdout:Preparing to unpack .../3-linux-generic_5.15.0.121.121_amd64.deb ... 2024-09-13T11:08:11.386 INFO:teuthology.orchestra.run.smithi001.stdout:Unpacking linux-generic (5.15.0.121.121) over (5.15.0.56.54) ... 2024-09-13T11:08:11.395 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package linux-headers-5.15.0-121. 2024-09-13T11:08:11.417 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../5-linux-headers-5.15.0-121_5.15.0-121.131_all.deb ... 2024-09-13T11:08:11.461 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking linux-headers-5.15.0-121 (5.15.0-121.131) ... 2024-09-13T11:08:11.703 INFO:teuthology.orchestra.run.smithi001.stdout:Preparing to unpack .../4-linux-image-generic_5.15.0.121.121_amd64.deb ... 2024-09-13T11:08:11.832 INFO:teuthology.orchestra.run.smithi001.stdout:Unpacking linux-image-generic (5.15.0.121.121) over (5.15.0.56.54) ... 2024-09-13T11:08:11.833 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../3-linux-generic_5.15.0.121.121_amd64.deb ... 2024-09-13T11:08:11.958 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking linux-generic (5.15.0.121.121) over (5.15.0.56.54) ... 2024-09-13T11:08:12.134 INFO:teuthology.orchestra.run.smithi001.stdout:Selecting previously unselected package linux-headers-5.15.0-121. 2024-09-13T11:08:12.157 INFO:teuthology.orchestra.run.smithi001.stdout:Preparing to unpack .../5-linux-headers-5.15.0-121_5.15.0-121.131_all.deb ... 2024-09-13T11:08:12.201 INFO:teuthology.orchestra.run.smithi001.stdout:Unpacking linux-headers-5.15.0-121 (5.15.0-121.131) ... 2024-09-13T11:08:12.273 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../4-linux-image-generic_5.15.0.121.121_amd64.deb ... 2024-09-13T11:08:12.378 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking linux-image-generic (5.15.0.121.121) over (5.15.0.56.54) ... 2024-09-13T11:08:12.689 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package linux-headers-5.15.0-121. 2024-09-13T11:08:12.712 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../5-linux-headers-5.15.0-121_5.15.0-121.131_all.deb ... 2024-09-13T11:08:12.755 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking linux-headers-5.15.0-121 (5.15.0-121.131) ... 2024-09-13T11:08:15.141 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../3-linux-generic_5.15.0.121.121_amd64.deb ... 2024-09-13T11:08:15.285 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking linux-generic (5.15.0.121.121) over (5.15.0.56.54) ... 2024-09-13T11:08:15.627 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../4-linux-image-generic_5.15.0.121.121_amd64.deb ... 2024-09-13T11:08:15.756 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking linux-image-generic (5.15.0.121.121) over (5.15.0.56.54) ... 2024-09-13T11:08:16.033 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package linux-headers-5.15.0-121. 2024-09-13T11:08:16.056 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../5-linux-headers-5.15.0-121_5.15.0-121.131_all.deb ... 2024-09-13T11:08:16.100 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking linux-headers-5.15.0-121 (5.15.0-121.131) ... 2024-09-13T11:08:16.740 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package linux-headers-5.15.0-121-generic. 2024-09-13T11:08:16.777 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../6-linux-headers-5.15.0-121-generic_5.15.0-121.131_amd64.deb ... 2024-09-13T11:08:16.806 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking linux-headers-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-13T11:08:18.116 INFO:teuthology.orchestra.run.smithi001.stdout:Selecting previously unselected package linux-headers-5.15.0-121-generic. 2024-09-13T11:08:18.153 INFO:teuthology.orchestra.run.smithi001.stdout:Preparing to unpack .../6-linux-headers-5.15.0-121-generic_5.15.0-121.131_amd64.deb ... 2024-09-13T11:08:18.182 INFO:teuthology.orchestra.run.smithi001.stdout:Unpacking linux-headers-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-13T11:08:18.839 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package linux-headers-5.15.0-121-generic. 2024-09-13T11:08:18.877 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../6-linux-headers-5.15.0-121-generic_5.15.0-121.131_amd64.deb ... 2024-09-13T11:08:18.906 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking linux-headers-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-13T11:08:19.404 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../7-linux-headers-generic_5.15.0.121.121_amd64.deb ... 2024-09-13T11:08:19.535 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking linux-headers-generic (5.15.0.121.121) over (5.15.0.56.54) ... 2024-09-13T11:08:19.885 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up linux-headers-5.15.0-121 (5.15.0-121.131) ... 2024-09-13T11:08:20.007 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up linux-headers-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-13T11:08:20.133 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up linux-headers-generic (5.15.0.121.121) ... 2024-09-13T11:08:20.260 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up linux-image-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-13T11:08:20.917 INFO:teuthology.orchestra.run.smithi001.stdout:Preparing to unpack .../7-linux-headers-generic_5.15.0.121.121_amd64.deb ... 2024-09-13T11:08:21.035 INFO:teuthology.orchestra.run.smithi001.stdout:Unpacking linux-headers-generic (5.15.0.121.121) over (5.15.0.56.54) ... 2024-09-13T11:08:21.352 INFO:teuthology.orchestra.run.smithi001.stdout:Setting up linux-headers-5.15.0-121 (5.15.0-121.131) ... 2024-09-13T11:08:21.456 INFO:teuthology.orchestra.run.smithi001.stdout:Setting up linux-headers-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-13T11:08:21.538 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package linux-headers-5.15.0-121-generic. 2024-09-13T11:08:21.573 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../7-linux-headers-generic_5.15.0.121.121_amd64.deb ... 2024-09-13T11:08:21.575 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../6-linux-headers-5.15.0-121-generic_5.15.0-121.131_amd64.deb ... 2024-09-13T11:08:21.577 INFO:teuthology.orchestra.run.smithi177.stdout:I: /vmlinuz.old is now a symlink to boot/vmlinuz-5.15.0-56-generic 2024-09-13T11:08:21.577 INFO:teuthology.orchestra.run.smithi177.stdout:I: /initrd.img.old is now a symlink to boot/initrd.img-5.15.0-56-generic 2024-09-13T11:08:21.577 INFO:teuthology.orchestra.run.smithi177.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.15.0-121-generic 2024-09-13T11:08:21.577 INFO:teuthology.orchestra.run.smithi177.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.15.0-121-generic 2024-09-13T11:08:21.582 INFO:teuthology.orchestra.run.smithi001.stdout:Setting up linux-headers-generic (5.15.0.121.121) ... 2024-09-13T11:08:21.605 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking linux-headers-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-13T11:08:21.710 INFO:teuthology.orchestra.run.smithi001.stdout:Setting up linux-image-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-13T11:08:21.718 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking linux-headers-generic (5.15.0.121.121) over (5.15.0.56.54) ... 2024-09-13T11:08:21.730 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up linux-modules-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-13T11:08:22.042 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up linux-headers-5.15.0-121 (5.15.0-121.131) ... 2024-09-13T11:08:22.164 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up linux-headers-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-13T11:08:22.290 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up linux-headers-generic (5.15.0.121.121) ... 2024-09-13T11:08:22.418 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up linux-image-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-13T11:08:23.059 INFO:teuthology.orchestra.run.smithi001.stdout:I: /vmlinuz.old is now a symlink to boot/vmlinuz-5.15.0-56-generic 2024-09-13T11:08:23.060 INFO:teuthology.orchestra.run.smithi001.stdout:I: /initrd.img.old is now a symlink to boot/initrd.img-5.15.0-56-generic 2024-09-13T11:08:23.060 INFO:teuthology.orchestra.run.smithi001.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.15.0-121-generic 2024-09-13T11:08:23.060 INFO:teuthology.orchestra.run.smithi001.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.15.0-121-generic 2024-09-13T11:08:23.131 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up linux-modules-extra-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-13T11:08:23.221 INFO:teuthology.orchestra.run.smithi001.stdout:Setting up linux-modules-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-13T11:08:23.741 INFO:teuthology.orchestra.run.smithi162.stdout:I: /vmlinuz.old is now a symlink to boot/vmlinuz-5.15.0-56-generic 2024-09-13T11:08:23.742 INFO:teuthology.orchestra.run.smithi162.stdout:I: /initrd.img.old is now a symlink to boot/initrd.img-5.15.0-56-generic 2024-09-13T11:08:23.742 INFO:teuthology.orchestra.run.smithi162.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.15.0-121-generic 2024-09-13T11:08:23.742 INFO:teuthology.orchestra.run.smithi162.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.15.0-121-generic 2024-09-13T11:08:24.547 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up linux-modules-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-13T11:08:24.849 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up linux-image-generic (5.15.0.121.121) ... 2024-09-13T11:08:24.975 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up linux-generic (5.15.0.121.121) ... 2024-09-13T11:08:25.047 INFO:teuthology.orchestra.run.smithi001.stdout:Setting up linux-modules-extra-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-13T11:08:25.102 INFO:teuthology.orchestra.run.smithi177.stdout:Processing triggers for linux-image-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-13T11:08:25.155 INFO:teuthology.orchestra.run.smithi177.stdout:/etc/kernel/postinst.d/initramfs-tools: 2024-09-13T11:08:25.155 INFO:teuthology.orchestra.run.smithi177.stdout:update-initramfs: Generating /boot/initrd.img-5.15.0-121-generic 2024-09-13T11:08:25.550 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../7-linux-headers-generic_5.15.0.121.121_amd64.deb ... 2024-09-13T11:08:25.675 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking linux-headers-generic (5.15.0.121.121) over (5.15.0.56.54) ... 2024-09-13T11:08:26.009 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up linux-headers-5.15.0-121 (5.15.0-121.131) ... 2024-09-13T11:08:26.130 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up linux-headers-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-13T11:08:26.256 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up linux-modules-extra-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-13T11:08:26.256 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up linux-headers-generic (5.15.0.121.121) ... 2024-09-13T11:08:26.384 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up linux-image-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-13T11:08:26.466 INFO:teuthology.orchestra.run.smithi001.stdout:Setting up linux-image-generic (5.15.0.121.121) ... 2024-09-13T11:08:26.591 INFO:teuthology.orchestra.run.smithi001.stdout:Setting up linux-generic (5.15.0.121.121) ... 2024-09-13T11:08:26.718 INFO:teuthology.orchestra.run.smithi001.stdout:Processing triggers for linux-image-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-13T11:08:26.785 INFO:teuthology.orchestra.run.smithi001.stdout:/etc/kernel/postinst.d/initramfs-tools: 2024-09-13T11:08:26.785 INFO:teuthology.orchestra.run.smithi001.stdout:update-initramfs: Generating /boot/initrd.img-5.15.0-121-generic 2024-09-13T11:08:27.640 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up linux-image-generic (5.15.0.121.121) ... 2024-09-13T11:08:27.717 INFO:teuthology.orchestra.run.smithi092.stdout:I: /vmlinuz.old is now a symlink to boot/vmlinuz-5.15.0-56-generic 2024-09-13T11:08:27.717 INFO:teuthology.orchestra.run.smithi092.stdout:I: /initrd.img.old is now a symlink to boot/initrd.img-5.15.0-56-generic 2024-09-13T11:08:27.717 INFO:teuthology.orchestra.run.smithi092.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.15.0-121-generic 2024-09-13T11:08:27.717 INFO:teuthology.orchestra.run.smithi092.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.15.0-121-generic 2024-09-13T11:08:27.741 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up linux-generic (5.15.0.121.121) ... 2024-09-13T11:08:27.876 INFO:teuthology.orchestra.run.smithi162.stdout:Processing triggers for linux-image-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-13T11:08:27.889 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up linux-modules-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-13T11:08:27.942 INFO:teuthology.orchestra.run.smithi162.stdout:/etc/kernel/postinst.d/initramfs-tools: 2024-09-13T11:08:27.943 INFO:teuthology.orchestra.run.smithi162.stdout:update-initramfs: Generating /boot/initrd.img-5.15.0-121-generic 2024-09-13T11:08:29.282 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up linux-modules-extra-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-13T11:08:31.108 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up linux-image-generic (5.15.0.121.121) ... 2024-09-13T11:08:31.234 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up linux-generic (5.15.0.121.121) ... 2024-09-13T11:08:31.360 INFO:teuthology.orchestra.run.smithi092.stdout:Processing triggers for linux-image-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-13T11:08:31.426 INFO:teuthology.orchestra.run.smithi092.stdout:/etc/kernel/postinst.d/initramfs-tools: 2024-09-13T11:08:31.426 INFO:teuthology.orchestra.run.smithi092.stdout:update-initramfs: Generating /boot/initrd.img-5.15.0-121-generic 2024-09-13T11:08:39.720 INFO:teuthology.orchestra.run.smithi177.stdout:/etc/kernel/postinst.d/zz-update-grub: 2024-09-13T11:08:39.720 INFO:teuthology.orchestra.run.smithi177.stdout:Sourcing file `/etc/default/grub' 2024-09-13T11:08:39.737 INFO:teuthology.orchestra.run.smithi177.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-09-13T11:08:39.755 INFO:teuthology.orchestra.run.smithi177.stdout:Generating grub configuration file ... 2024-09-13T11:08:40.057 INFO:teuthology.orchestra.run.smithi177.stdout:Found linux image: /boot/vmlinuz-5.15.0-121-generic 2024-09-13T11:08:40.077 INFO:teuthology.orchestra.run.smithi177.stdout:Found initrd image: /boot/initrd.img-5.15.0-121-generic 2024-09-13T11:08:40.342 INFO:teuthology.orchestra.run.smithi177.stdout:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-09-13T11:08:40.348 INFO:teuthology.orchestra.run.smithi177.stdout:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-09-13T11:08:40.427 INFO:teuthology.orchestra.run.smithi177.stdout:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-09-13T11:08:40.433 INFO:teuthology.orchestra.run.smithi177.stdout:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-09-13T11:08:40.596 INFO:teuthology.orchestra.run.smithi177.stdout:Warning: os-prober will not be executed to detect other bootable partitions. 2024-09-13T11:08:40.596 INFO:teuthology.orchestra.run.smithi177.stdout:Systems on them will not be added to the GRUB boot configuration. 2024-09-13T11:08:40.597 INFO:teuthology.orchestra.run.smithi177.stdout:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-09-13T11:08:40.640 INFO:teuthology.orchestra.run.smithi177.stdout:done 2024-09-13T11:08:41.140 DEBUG:teuthology.orchestra.run.smithi177:> dpkg -s linux-image-generic 2024-09-13T11:08:41.170 INFO:teuthology.orchestra.run.smithi177.stdout:Package: linux-image-generic 2024-09-13T11:08:41.170 INFO:teuthology.orchestra.run.smithi177.stdout:Status: install ok installed 2024-09-13T11:08:41.170 INFO:teuthology.orchestra.run.smithi177.stdout:Priority: optional 2024-09-13T11:08:41.170 INFO:teuthology.orchestra.run.smithi177.stdout:Section: kernel 2024-09-13T11:08:41.170 INFO:teuthology.orchestra.run.smithi177.stdout:Installed-Size: 21 2024-09-13T11:08:41.170 INFO:teuthology.orchestra.run.smithi177.stdout:Maintainer: Ubuntu Kernel Team 2024-09-13T11:08:41.171 INFO:teuthology.orchestra.run.smithi177.stdout:Architecture: amd64 2024-09-13T11:08:41.171 INFO:teuthology.orchestra.run.smithi177.stdout:Source: linux-meta 2024-09-13T11:08:41.171 INFO:teuthology.orchestra.run.smithi177.stdout:Version: 5.15.0.121.121 2024-09-13T11:08:41.171 INFO:teuthology.orchestra.run.smithi177.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.4), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-121), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.4) 2024-09-13T11:08:41.171 INFO:teuthology.orchestra.run.smithi177.stdout:Depends: linux-image-5.15.0-121-generic, linux-modules-extra-5.15.0-121-generic, linux-firmware, intel-microcode, amd64-microcode 2024-09-13T11:08:41.171 INFO:teuthology.orchestra.run.smithi177.stdout:Recommends: thermald 2024-09-13T11:08:41.171 INFO:teuthology.orchestra.run.smithi177.stdout:Description: Generic Linux kernel image 2024-09-13T11:08:41.171 INFO:teuthology.orchestra.run.smithi177.stdout: This package will always depend on the latest generic kernel image 2024-09-13T11:08:41.171 INFO:teuthology.orchestra.run.smithi177.stdout: available. 2024-09-13T11:08:41.171 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.15.0-56-generic Expected: 5.15.0-121-generic 2024-09-13T11:08:41.172 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-13T11:08:41.172 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-13T11:08:41.172 INFO:teuthology.task.kernel:Installing distro kernel on mgr.x... 2024-09-13T11:08:41.172 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi177.front.sepia.ceph.com, path=None, version=distro) 2024-09-13T11:08:41.172 DEBUG:teuthology.orchestra.run.smithi177:> sudo apt-get clean 2024-09-13T11:08:41.371 DEBUG:teuthology.orchestra.run.smithi177:> sudo apt-get update 2024-09-13T11:08:41.390 INFO:teuthology.orchestra.run.smithi001.stdout:/etc/kernel/postinst.d/zz-update-grub: 2024-09-13T11:08:41.391 INFO:teuthology.orchestra.run.smithi001.stdout:Sourcing file `/etc/default/grub' 2024-09-13T11:08:41.413 INFO:teuthology.orchestra.run.smithi001.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-09-13T11:08:41.436 INFO:teuthology.orchestra.run.smithi001.stdout:Generating grub configuration file ... 2024-09-13T11:08:41.660 INFO:teuthology.orchestra.run.smithi001.stdout:Found linux image: /boot/vmlinuz-5.15.0-121-generic 2024-09-13T11:08:41.675 INFO:teuthology.orchestra.run.smithi001.stdout:Found initrd image: /boot/initrd.img-5.15.0-121-generic 2024-09-13T11:08:41.961 INFO:teuthology.orchestra.run.smithi001.stdout:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-09-13T11:08:41.969 INFO:teuthology.orchestra.run.smithi001.stdout:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-09-13T11:08:42.040 INFO:teuthology.orchestra.run.smithi162.stdout:/etc/kernel/postinst.d/zz-update-grub: 2024-09-13T11:08:42.041 INFO:teuthology.orchestra.run.smithi162.stdout:Sourcing file `/etc/default/grub' 2024-09-13T11:08:42.046 INFO:teuthology.orchestra.run.smithi001.stdout:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-09-13T11:08:42.052 INFO:teuthology.orchestra.run.smithi001.stdout:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-09-13T11:08:42.061 INFO:teuthology.orchestra.run.smithi162.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-09-13T11:08:42.077 INFO:teuthology.orchestra.run.smithi162.stdout:Generating grub configuration file ... 2024-09-13T11:08:42.209 INFO:teuthology.orchestra.run.smithi001.stdout:Warning: os-prober will not be executed to detect other bootable partitions. 2024-09-13T11:08:42.209 INFO:teuthology.orchestra.run.smithi001.stdout:Systems on them will not be added to the GRUB boot configuration. 2024-09-13T11:08:42.210 INFO:teuthology.orchestra.run.smithi001.stdout:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-09-13T11:08:42.259 INFO:teuthology.orchestra.run.smithi001.stdout:done 2024-09-13T11:08:42.340 INFO:teuthology.orchestra.run.smithi162.stdout:Found linux image: /boot/vmlinuz-5.15.0-121-generic 2024-09-13T11:08:42.370 INFO:teuthology.orchestra.run.smithi162.stdout:Found initrd image: /boot/initrd.img-5.15.0-121-generic 2024-09-13T11:08:42.450 INFO:teuthology.orchestra.run.smithi177.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-09-13T11:08:42.450 INFO:teuthology.orchestra.run.smithi177.stdout:Hit:2 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-09-13T11:08:42.456 INFO:teuthology.orchestra.run.smithi177.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-09-13T11:08:42.473 INFO:teuthology.orchestra.run.smithi177.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-09-13T11:08:42.690 INFO:teuthology.orchestra.run.smithi162.stdout:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-09-13T11:08:42.696 INFO:teuthology.orchestra.run.smithi162.stdout:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-09-13T11:08:42.740 DEBUG:teuthology.orchestra.run.smithi001:> dpkg -s linux-image-generic 2024-09-13T11:08:42.758 INFO:teuthology.orchestra.run.smithi001.stdout:Package: linux-image-generic 2024-09-13T11:08:42.758 INFO:teuthology.orchestra.run.smithi001.stdout:Status: install ok installed 2024-09-13T11:08:42.758 INFO:teuthology.orchestra.run.smithi001.stdout:Priority: optional 2024-09-13T11:08:42.758 INFO:teuthology.orchestra.run.smithi001.stdout:Section: kernel 2024-09-13T11:08:42.758 INFO:teuthology.orchestra.run.smithi001.stdout:Installed-Size: 21 2024-09-13T11:08:42.759 INFO:teuthology.orchestra.run.smithi001.stdout:Maintainer: Ubuntu Kernel Team 2024-09-13T11:08:42.759 INFO:teuthology.orchestra.run.smithi001.stdout:Architecture: amd64 2024-09-13T11:08:42.759 INFO:teuthology.orchestra.run.smithi001.stdout:Source: linux-meta 2024-09-13T11:08:42.759 INFO:teuthology.orchestra.run.smithi001.stdout:Version: 5.15.0.121.121 2024-09-13T11:08:42.759 INFO:teuthology.orchestra.run.smithi001.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.4), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-121), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.4) 2024-09-13T11:08:42.759 INFO:teuthology.orchestra.run.smithi001.stdout:Depends: linux-image-5.15.0-121-generic, linux-modules-extra-5.15.0-121-generic, linux-firmware, intel-microcode, amd64-microcode 2024-09-13T11:08:42.759 INFO:teuthology.orchestra.run.smithi001.stdout:Recommends: thermald 2024-09-13T11:08:42.759 INFO:teuthology.orchestra.run.smithi001.stdout:Description: Generic Linux kernel image 2024-09-13T11:08:42.759 INFO:teuthology.orchestra.run.smithi001.stdout: This package will always depend on the latest generic kernel image 2024-09-13T11:08:42.759 INFO:teuthology.orchestra.run.smithi001.stdout: available. 2024-09-13T11:08:42.760 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.15.0-56-generic Expected: 5.15.0-121-generic 2024-09-13T11:08:42.760 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-13T11:08:42.761 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-13T11:08:42.761 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-09-13T11:08:42.761 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi001.front.sepia.ceph.com, path=None, version=distro) 2024-09-13T11:08:42.761 DEBUG:teuthology.orchestra.run.smithi001:> sudo apt-get clean 2024-09-13T11:08:42.769 INFO:teuthology.orchestra.run.smithi162.stdout:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-09-13T11:08:42.775 INFO:teuthology.orchestra.run.smithi162.stdout:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-09-13T11:08:42.937 DEBUG:teuthology.orchestra.run.smithi001:> sudo apt-get update 2024-09-13T11:08:42.945 INFO:teuthology.orchestra.run.smithi162.stdout:Warning: os-prober will not be executed to detect other bootable partitions. 2024-09-13T11:08:42.945 INFO:teuthology.orchestra.run.smithi162.stdout:Systems on them will not be added to the GRUB boot configuration. 2024-09-13T11:08:42.945 INFO:teuthology.orchestra.run.smithi162.stdout:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-09-13T11:08:42.981 INFO:teuthology.orchestra.run.smithi162.stdout:done 2024-09-13T11:08:43.055 INFO:teuthology.orchestra.run.smithi001.stdout:Hit:1 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-09-13T11:08:43.055 INFO:teuthology.orchestra.run.smithi001.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-09-13T11:08:43.065 INFO:teuthology.orchestra.run.smithi001.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-09-13T11:08:43.083 INFO:teuthology.orchestra.run.smithi001.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-09-13T11:08:43.478 DEBUG:teuthology.orchestra.run.smithi162:> dpkg -s linux-image-generic 2024-09-13T11:08:43.548 INFO:teuthology.orchestra.run.smithi162.stdout:Package: linux-image-generic 2024-09-13T11:08:43.548 INFO:teuthology.orchestra.run.smithi162.stdout:Status: install ok installed 2024-09-13T11:08:43.548 INFO:teuthology.orchestra.run.smithi162.stdout:Priority: optional 2024-09-13T11:08:43.548 INFO:teuthology.orchestra.run.smithi162.stdout:Section: kernel 2024-09-13T11:08:43.548 INFO:teuthology.orchestra.run.smithi162.stdout:Installed-Size: 21 2024-09-13T11:08:43.548 INFO:teuthology.orchestra.run.smithi162.stdout:Maintainer: Ubuntu Kernel Team 2024-09-13T11:08:43.548 INFO:teuthology.orchestra.run.smithi162.stdout:Architecture: amd64 2024-09-13T11:08:43.549 INFO:teuthology.orchestra.run.smithi162.stdout:Source: linux-meta 2024-09-13T11:08:43.549 INFO:teuthology.orchestra.run.smithi162.stdout:Version: 5.15.0.121.121 2024-09-13T11:08:43.549 INFO:teuthology.orchestra.run.smithi162.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.4), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-121), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.4) 2024-09-13T11:08:43.549 INFO:teuthology.orchestra.run.smithi162.stdout:Depends: linux-image-5.15.0-121-generic, linux-modules-extra-5.15.0-121-generic, linux-firmware, intel-microcode, amd64-microcode 2024-09-13T11:08:43.549 INFO:teuthology.orchestra.run.smithi162.stdout:Recommends: thermald 2024-09-13T11:08:43.549 INFO:teuthology.orchestra.run.smithi162.stdout:Description: Generic Linux kernel image 2024-09-13T11:08:43.549 INFO:teuthology.orchestra.run.smithi162.stdout: This package will always depend on the latest generic kernel image 2024-09-13T11:08:43.549 INFO:teuthology.orchestra.run.smithi162.stdout: available. 2024-09-13T11:08:43.549 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.15.0-56-generic Expected: 5.15.0-121-generic 2024-09-13T11:08:43.549 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-13T11:08:43.549 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-13T11:08:43.550 INFO:teuthology.task.kernel:Installing distro kernel on mon.c... 2024-09-13T11:08:43.550 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi162.front.sepia.ceph.com, path=None, version=distro) 2024-09-13T11:08:43.550 DEBUG:teuthology.orchestra.run.smithi162:> sudo apt-get clean 2024-09-13T11:08:43.766 DEBUG:teuthology.orchestra.run.smithi162:> sudo apt-get update 2024-09-13T11:08:43.874 INFO:teuthology.orchestra.run.smithi177.stdout:Reading package lists... 2024-09-13T11:08:43.889 INFO:teuthology.orchestra.run.smithi162.stdout:Hit:1 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-09-13T11:08:43.891 DEBUG:teuthology.orchestra.run.smithi177:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-09-13T11:08:43.956 INFO:teuthology.orchestra.run.smithi177.stdout:Reading package lists... 2024-09-13T11:08:44.010 INFO:teuthology.orchestra.run.smithi162.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-09-13T11:08:44.093 INFO:teuthology.orchestra.run.smithi162.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-09-13T11:08:44.166 INFO:teuthology.orchestra.run.smithi177.stdout:Building dependency tree... 2024-09-13T11:08:44.167 INFO:teuthology.orchestra.run.smithi177.stdout:Reading state information... 2024-09-13T11:08:44.172 INFO:teuthology.orchestra.run.smithi162.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-09-13T11:08:44.355 INFO:teuthology.orchestra.run.smithi177.stdout:linux-image-generic is already the newest version (5.15.0.121.121). 2024-09-13T11:08:44.355 INFO:teuthology.orchestra.run.smithi177.stdout:The following packages were automatically installed and are no longer required: 2024-09-13T11:08:44.355 INFO:teuthology.orchestra.run.smithi177.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-09-13T11:08:44.355 INFO:teuthology.orchestra.run.smithi177.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-09-13T11:08:44.355 INFO:teuthology.orchestra.run.smithi177.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-09-13T11:08:44.356 INFO:teuthology.orchestra.run.smithi177.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-09-13T11:08:44.356 INFO:teuthology.orchestra.run.smithi177.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-09-13T11:08:44.356 INFO:teuthology.orchestra.run.smithi177.stdout:Use 'sudo apt autoremove' to remove them. 2024-09-13T11:08:44.399 INFO:teuthology.orchestra.run.smithi177.stdout:0 upgraded, 0 newly installed, 0 to remove and 344 not upgraded. 2024-09-13T11:08:44.401 DEBUG:teuthology.orchestra.run.smithi177:> dpkg -s linux-image-generic 2024-09-13T11:08:44.417 INFO:teuthology.orchestra.run.smithi177.stdout:Package: linux-image-generic 2024-09-13T11:08:44.417 INFO:teuthology.orchestra.run.smithi177.stdout:Status: install ok installed 2024-09-13T11:08:44.417 INFO:teuthology.orchestra.run.smithi177.stdout:Priority: optional 2024-09-13T11:08:44.417 INFO:teuthology.orchestra.run.smithi177.stdout:Section: kernel 2024-09-13T11:08:44.417 INFO:teuthology.orchestra.run.smithi177.stdout:Installed-Size: 21 2024-09-13T11:08:44.417 INFO:teuthology.orchestra.run.smithi177.stdout:Maintainer: Ubuntu Kernel Team 2024-09-13T11:08:44.417 INFO:teuthology.orchestra.run.smithi177.stdout:Architecture: amd64 2024-09-13T11:08:44.417 INFO:teuthology.orchestra.run.smithi177.stdout:Source: linux-meta 2024-09-13T11:08:44.417 INFO:teuthology.orchestra.run.smithi177.stdout:Version: 5.15.0.121.121 2024-09-13T11:08:44.418 INFO:teuthology.orchestra.run.smithi177.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.4), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-121), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.4) 2024-09-13T11:08:44.418 INFO:teuthology.orchestra.run.smithi177.stdout:Depends: linux-image-5.15.0-121-generic, linux-modules-extra-5.15.0-121-generic, linux-firmware, intel-microcode, amd64-microcode 2024-09-13T11:08:44.418 INFO:teuthology.orchestra.run.smithi177.stdout:Recommends: thermald 2024-09-13T11:08:44.418 INFO:teuthology.orchestra.run.smithi177.stdout:Description: Generic Linux kernel image 2024-09-13T11:08:44.418 INFO:teuthology.orchestra.run.smithi177.stdout: This package will always depend on the latest generic kernel image 2024-09-13T11:08:44.418 INFO:teuthology.orchestra.run.smithi177.stdout: available. 2024-09-13T11:08:44.419 DEBUG:teuthology.orchestra.run.smithi177:> mktemp 2024-09-13T11:08:44.463 INFO:teuthology.orchestra.run.smithi177.stdout:/tmp/tmp.sjMQ8p3W3u 2024-09-13T11:08:44.464 DEBUG:teuthology.orchestra.run.smithi177:> sudo cp /boot/grub/grub.cfg /tmp/tmp.sjMQ8p3W3u 2024-09-13T11:08:44.493 INFO:teuthology.orchestra.run.smithi001.stdout:Reading package lists... 2024-09-13T11:08:44.512 DEBUG:teuthology.orchestra.run.smithi001:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-09-13T11:08:44.523 DEBUG:teuthology.orchestra.run.smithi177:> sudo chmod 0666 /tmp/tmp.sjMQ8p3W3u 2024-09-13T11:08:44.575 INFO:teuthology.orchestra.run.smithi001.stdout:Reading package lists... 2024-09-13T11:08:44.640 DEBUG:teuthology.orchestra.remote:smithi177:/tmp/tmp.sjMQ8p3W3u is 10KB 2024-09-13T11:08:44.653 DEBUG:teuthology.orchestra.run.smithi177:> rm -fr /tmp/tmp.sjMQ8p3W3u 2024-09-13T11:08:44.658 DEBUG:teuthology.orchestra.run.smithi177:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2024-09-13T11:08:44.711 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-09-13T11:08:44.712 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd of=/etc/grub.d/01_ceph_kernel 2024-09-13T11:08:44.712 DEBUG:teuthology.orchestra.run.smithi177:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2024-09-13T11:08:44.781 INFO:teuthology.task.kernel:Distro Kernel Version: 5.15.0-121-generic 2024-09-13T11:08:44.781 DEBUG:teuthology.orchestra.run.smithi177:> sudo update-grub 2024-09-13T11:08:44.788 INFO:teuthology.orchestra.run.smithi001.stdout:Building dependency tree... 2024-09-13T11:08:44.789 INFO:teuthology.orchestra.run.smithi001.stdout:Reading state information... 2024-09-13T11:08:44.988 INFO:teuthology.orchestra.run.smithi001.stdout:linux-image-generic is already the newest version (5.15.0.121.121). 2024-09-13T11:08:44.989 INFO:teuthology.orchestra.run.smithi001.stdout:The following packages were automatically installed and are no longer required: 2024-09-13T11:08:44.989 INFO:teuthology.orchestra.run.smithi001.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-09-13T11:08:44.989 INFO:teuthology.orchestra.run.smithi001.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-09-13T11:08:44.989 INFO:teuthology.orchestra.run.smithi001.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-09-13T11:08:44.989 INFO:teuthology.orchestra.run.smithi001.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-09-13T11:08:44.990 INFO:teuthology.orchestra.run.smithi001.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-09-13T11:08:44.990 INFO:teuthology.orchestra.run.smithi001.stdout:Use 'sudo apt autoremove' to remove them. 2024-09-13T11:08:45.035 INFO:teuthology.orchestra.run.smithi001.stdout:0 upgraded, 0 newly installed, 0 to remove and 344 not upgraded. 2024-09-13T11:08:45.041 DEBUG:teuthology.orchestra.run.smithi001:> dpkg -s linux-image-generic 2024-09-13T11:08:45.057 INFO:teuthology.orchestra.run.smithi001.stdout:Package: linux-image-generic 2024-09-13T11:08:45.058 INFO:teuthology.orchestra.run.smithi001.stdout:Status: install ok installed 2024-09-13T11:08:45.058 INFO:teuthology.orchestra.run.smithi001.stdout:Priority: optional 2024-09-13T11:08:45.058 INFO:teuthology.orchestra.run.smithi001.stdout:Section: kernel 2024-09-13T11:08:45.058 INFO:teuthology.orchestra.run.smithi001.stdout:Installed-Size: 21 2024-09-13T11:08:45.058 INFO:teuthology.orchestra.run.smithi001.stdout:Maintainer: Ubuntu Kernel Team 2024-09-13T11:08:45.058 INFO:teuthology.orchestra.run.smithi001.stdout:Architecture: amd64 2024-09-13T11:08:45.058 INFO:teuthology.orchestra.run.smithi001.stdout:Source: linux-meta 2024-09-13T11:08:45.058 INFO:teuthology.orchestra.run.smithi001.stdout:Version: 5.15.0.121.121 2024-09-13T11:08:45.058 INFO:teuthology.orchestra.run.smithi001.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.4), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-121), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.4) 2024-09-13T11:08:45.058 INFO:teuthology.orchestra.run.smithi001.stdout:Depends: linux-image-5.15.0-121-generic, linux-modules-extra-5.15.0-121-generic, linux-firmware, intel-microcode, amd64-microcode 2024-09-13T11:08:45.059 INFO:teuthology.orchestra.run.smithi001.stdout:Recommends: thermald 2024-09-13T11:08:45.059 INFO:teuthology.orchestra.run.smithi001.stdout:Description: Generic Linux kernel image 2024-09-13T11:08:45.059 INFO:teuthology.orchestra.run.smithi001.stdout: This package will always depend on the latest generic kernel image 2024-09-13T11:08:45.059 INFO:teuthology.orchestra.run.smithi001.stdout: available. 2024-09-13T11:08:45.059 DEBUG:teuthology.orchestra.run.smithi001:> mktemp 2024-09-13T11:08:45.103 INFO:teuthology.orchestra.run.smithi001.stdout:/tmp/tmp.A2iJy6ZmNE 2024-09-13T11:08:45.103 DEBUG:teuthology.orchestra.run.smithi001:> sudo cp /boot/grub/grub.cfg /tmp/tmp.A2iJy6ZmNE 2024-09-13T11:08:45.160 DEBUG:teuthology.orchestra.run.smithi001:> sudo chmod 0666 /tmp/tmp.A2iJy6ZmNE 2024-09-13T11:08:45.211 INFO:teuthology.orchestra.run.smithi092.stdout:/etc/kernel/postinst.d/zz-update-grub: 2024-09-13T11:08:45.211 INFO:teuthology.orchestra.run.smithi092.stdout:Sourcing file `/etc/default/grub' 2024-09-13T11:08:45.240 INFO:teuthology.orchestra.run.smithi092.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-09-13T11:08:45.266 INFO:teuthology.orchestra.run.smithi092.stdout:Generating grub configuration file ... 2024-09-13T11:08:45.284 DEBUG:teuthology.orchestra.remote:smithi001:/tmp/tmp.A2iJy6ZmNE is 10KB 2024-09-13T11:08:45.299 DEBUG:teuthology.orchestra.run.smithi001:> rm -fr /tmp/tmp.A2iJy6ZmNE 2024-09-13T11:08:45.304 DEBUG:teuthology.orchestra.run.smithi001:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2024-09-13T11:08:45.340 INFO:teuthology.orchestra.run.smithi162.stdout:Reading package lists... 2024-09-13T11:08:45.360 DEBUG:teuthology.orchestra.run.smithi162:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-09-13T11:08:45.363 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-09-13T11:08:45.364 DEBUG:teuthology.orchestra.run.smithi001:> sudo dd of=/etc/grub.d/01_ceph_kernel 2024-09-13T11:08:45.364 DEBUG:teuthology.orchestra.run.smithi001:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2024-09-13T11:08:45.423 INFO:teuthology.orchestra.run.smithi162.stdout:Reading package lists... 2024-09-13T11:08:45.429 INFO:teuthology.task.kernel:Distro Kernel Version: 5.15.0-121-generic 2024-09-13T11:08:45.429 DEBUG:teuthology.orchestra.run.smithi001:> sudo update-grub 2024-09-13T11:08:45.513 INFO:teuthology.orchestra.run.smithi092.stdout:Found linux image: /boot/vmlinuz-5.15.0-121-generic 2024-09-13T11:08:45.530 INFO:teuthology.orchestra.run.smithi092.stdout:Found initrd image: /boot/initrd.img-5.15.0-121-generic 2024-09-13T11:08:45.643 INFO:teuthology.orchestra.run.smithi162.stdout:Building dependency tree... 2024-09-13T11:08:45.644 INFO:teuthology.orchestra.run.smithi162.stdout:Reading state information... 2024-09-13T11:08:45.824 INFO:teuthology.orchestra.run.smithi092.stdout:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-09-13T11:08:45.827 INFO:teuthology.orchestra.run.smithi162.stdout:linux-image-generic is already the newest version (5.15.0.121.121). 2024-09-13T11:08:45.828 INFO:teuthology.orchestra.run.smithi162.stdout:The following packages were automatically installed and are no longer required: 2024-09-13T11:08:45.828 INFO:teuthology.orchestra.run.smithi162.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-09-13T11:08:45.828 INFO:teuthology.orchestra.run.smithi162.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-09-13T11:08:45.828 INFO:teuthology.orchestra.run.smithi162.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-09-13T11:08:45.828 INFO:teuthology.orchestra.run.smithi162.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-09-13T11:08:45.828 INFO:teuthology.orchestra.run.smithi162.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-09-13T11:08:45.828 INFO:teuthology.orchestra.run.smithi162.stdout:Use 'sudo apt autoremove' to remove them. 2024-09-13T11:08:45.830 INFO:teuthology.orchestra.run.smithi092.stdout:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-09-13T11:08:45.870 INFO:teuthology.orchestra.run.smithi162.stdout:0 upgraded, 0 newly installed, 0 to remove and 344 not upgraded. 2024-09-13T11:08:45.872 DEBUG:teuthology.orchestra.run.smithi162:> dpkg -s linux-image-generic 2024-09-13T11:08:45.887 INFO:teuthology.orchestra.run.smithi162.stdout:Package: linux-image-generic 2024-09-13T11:08:45.887 INFO:teuthology.orchestra.run.smithi162.stdout:Status: install ok installed 2024-09-13T11:08:45.887 INFO:teuthology.orchestra.run.smithi162.stdout:Priority: optional 2024-09-13T11:08:45.887 INFO:teuthology.orchestra.run.smithi162.stdout:Section: kernel 2024-09-13T11:08:45.887 INFO:teuthology.orchestra.run.smithi162.stdout:Installed-Size: 21 2024-09-13T11:08:45.887 INFO:teuthology.orchestra.run.smithi162.stdout:Maintainer: Ubuntu Kernel Team 2024-09-13T11:08:45.888 INFO:teuthology.orchestra.run.smithi162.stdout:Architecture: amd64 2024-09-13T11:08:45.888 INFO:teuthology.orchestra.run.smithi162.stdout:Source: linux-meta 2024-09-13T11:08:45.888 INFO:teuthology.orchestra.run.smithi162.stdout:Version: 5.15.0.121.121 2024-09-13T11:08:45.888 INFO:teuthology.orchestra.run.smithi162.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.4), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-121), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.4) 2024-09-13T11:08:45.888 INFO:teuthology.orchestra.run.smithi162.stdout:Depends: linux-image-5.15.0-121-generic, linux-modules-extra-5.15.0-121-generic, linux-firmware, intel-microcode, amd64-microcode 2024-09-13T11:08:45.888 INFO:teuthology.orchestra.run.smithi162.stdout:Recommends: thermald 2024-09-13T11:08:45.888 INFO:teuthology.orchestra.run.smithi162.stdout:Description: Generic Linux kernel image 2024-09-13T11:08:45.888 INFO:teuthology.orchestra.run.smithi162.stdout: This package will always depend on the latest generic kernel image 2024-09-13T11:08:45.888 INFO:teuthology.orchestra.run.smithi162.stdout: available. 2024-09-13T11:08:45.888 DEBUG:teuthology.orchestra.run.smithi162:> mktemp 2024-09-13T11:08:45.901 INFO:teuthology.orchestra.run.smithi092.stdout:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-09-13T11:08:45.909 INFO:teuthology.orchestra.run.smithi092.stdout:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-09-13T11:08:45.937 INFO:teuthology.orchestra.run.smithi162.stdout:/tmp/tmp.kO5rSDShab 2024-09-13T11:08:45.937 DEBUG:teuthology.orchestra.run.smithi162:> sudo cp /boot/grub/grub.cfg /tmp/tmp.kO5rSDShab 2024-09-13T11:08:46.002 INFO:teuthology.orchestra.run.smithi177.stderr:Sourcing file `/etc/default/grub' 2024-09-13T11:08:46.004 DEBUG:teuthology.orchestra.run.smithi162:> sudo chmod 0666 /tmp/tmp.kO5rSDShab 2024-09-13T11:08:46.004 INFO:teuthology.orchestra.run.smithi177.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-09-13T11:08:46.005 INFO:teuthology.orchestra.run.smithi177.stderr:Generating grub configuration file ... 2024-09-13T11:08:46.080 INFO:teuthology.orchestra.run.smithi092.stdout:Warning: os-prober will not be executed to detect other bootable partitions. 2024-09-13T11:08:46.081 INFO:teuthology.orchestra.run.smithi092.stdout:Systems on them will not be added to the GRUB boot configuration. 2024-09-13T11:08:46.081 INFO:teuthology.orchestra.run.smithi092.stdout:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-09-13T11:08:46.114 DEBUG:teuthology.orchestra.remote:smithi162:/tmp/tmp.kO5rSDShab is 10KB 2024-09-13T11:08:46.123 INFO:teuthology.orchestra.run.smithi092.stdout:done 2024-09-13T11:08:46.127 DEBUG:teuthology.orchestra.run.smithi162:> rm -fr /tmp/tmp.kO5rSDShab 2024-09-13T11:08:46.131 DEBUG:teuthology.orchestra.run.smithi162:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2024-09-13T11:08:46.181 INFO:teuthology.orchestra.run.smithi177.stderr:Found linux image: /boot/vmlinuz-5.15.0-121-generic 2024-09-13T11:08:46.185 DEBUG:teuthology.orchestra.run.smithi162:> set -ex 2024-09-13T11:08:46.185 DEBUG:teuthology.orchestra.run.smithi162:> sudo dd of=/etc/grub.d/01_ceph_kernel 2024-09-13T11:08:46.185 DEBUG:teuthology.orchestra.run.smithi162:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2024-09-13T11:08:46.187 INFO:teuthology.orchestra.run.smithi177.stderr:Found initrd image: /boot/initrd.img-5.15.0-121-generic 2024-09-13T11:08:46.253 INFO:teuthology.task.kernel:Distro Kernel Version: 5.15.0-121-generic 2024-09-13T11:08:46.253 DEBUG:teuthology.orchestra.run.smithi162:> sudo update-grub 2024-09-13T11:08:46.406 INFO:teuthology.orchestra.run.smithi177.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-09-13T11:08:46.411 INFO:teuthology.orchestra.run.smithi177.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-09-13T11:08:46.450 INFO:teuthology.orchestra.run.smithi177.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-09-13T11:08:46.454 INFO:teuthology.orchestra.run.smithi177.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-09-13T11:08:46.574 DEBUG:teuthology.orchestra.run.smithi092:> dpkg -s linux-image-generic 2024-09-13T11:08:46.583 INFO:teuthology.orchestra.run.smithi177.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2024-09-13T11:08:46.583 INFO:teuthology.orchestra.run.smithi177.stderr:Systems on them will not be added to the GRUB boot configuration. 2024-09-13T11:08:46.583 INFO:teuthology.orchestra.run.smithi177.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-09-13T11:08:46.601 INFO:teuthology.orchestra.run.smithi092.stdout:Package: linux-image-generic 2024-09-13T11:08:46.601 INFO:teuthology.orchestra.run.smithi092.stdout:Status: install ok installed 2024-09-13T11:08:46.601 INFO:teuthology.orchestra.run.smithi092.stdout:Priority: optional 2024-09-13T11:08:46.601 INFO:teuthology.orchestra.run.smithi092.stdout:Section: kernel 2024-09-13T11:08:46.601 INFO:teuthology.orchestra.run.smithi092.stdout:Installed-Size: 21 2024-09-13T11:08:46.601 INFO:teuthology.orchestra.run.smithi092.stdout:Maintainer: Ubuntu Kernel Team 2024-09-13T11:08:46.602 INFO:teuthology.orchestra.run.smithi092.stdout:Architecture: amd64 2024-09-13T11:08:46.602 INFO:teuthology.orchestra.run.smithi092.stdout:Source: linux-meta 2024-09-13T11:08:46.602 INFO:teuthology.orchestra.run.smithi092.stdout:Version: 5.15.0.121.121 2024-09-13T11:08:46.602 INFO:teuthology.orchestra.run.smithi092.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.4), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-121), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.4) 2024-09-13T11:08:46.602 INFO:teuthology.orchestra.run.smithi092.stdout:Depends: linux-image-5.15.0-121-generic, linux-modules-extra-5.15.0-121-generic, linux-firmware, intel-microcode, amd64-microcode 2024-09-13T11:08:46.602 INFO:teuthology.orchestra.run.smithi092.stdout:Recommends: thermald 2024-09-13T11:08:46.602 INFO:teuthology.orchestra.run.smithi092.stdout:Description: Generic Linux kernel image 2024-09-13T11:08:46.602 INFO:teuthology.orchestra.run.smithi092.stdout: This package will always depend on the latest generic kernel image 2024-09-13T11:08:46.602 INFO:teuthology.orchestra.run.smithi092.stdout: available. 2024-09-13T11:08:46.603 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.15.0-56-generic Expected: 5.15.0-121-generic 2024-09-13T11:08:46.603 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-13T11:08:46.603 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-13T11:08:46.603 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-09-13T11:08:46.603 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi092.front.sepia.ceph.com, path=None, version=distro) 2024-09-13T11:08:46.603 DEBUG:teuthology.orchestra.run.smithi092:> sudo apt-get clean 2024-09-13T11:08:46.605 INFO:teuthology.orchestra.run.smithi177.stderr:done 2024-09-13T11:08:46.606 DEBUG:teuthology.orchestra.run.smithi177:> sudo shutdown -r now 2024-09-13T11:08:46.620 INFO:teuthology.orchestra.run.smithi001.stderr:Sourcing file `/etc/default/grub' 2024-09-13T11:08:46.621 INFO:teuthology.orchestra.run.smithi001.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-09-13T11:08:46.623 INFO:teuthology.orchestra.run.smithi001.stderr:Generating grub configuration file ... 2024-09-13T11:08:46.755 DEBUG:teuthology.orchestra.run.smithi092:> sudo apt-get update 2024-09-13T11:08:46.790 INFO:teuthology.orchestra.run.smithi001.stderr:Found linux image: /boot/vmlinuz-5.15.0-121-generic 2024-09-13T11:08:46.797 INFO:teuthology.orchestra.run.smithi001.stderr:Found initrd image: /boot/initrd.img-5.15.0-121-generic 2024-09-13T11:08:46.873 INFO:teuthology.orchestra.run.smithi092.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-09-13T11:08:46.885 INFO:teuthology.orchestra.run.smithi092.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-09-13T11:08:46.901 INFO:teuthology.orchestra.run.smithi092.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-09-13T11:08:47.015 INFO:teuthology.orchestra.run.smithi092.stdout:Hit:4 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-09-13T11:08:47.026 INFO:teuthology.orchestra.run.smithi001.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-09-13T11:08:47.032 INFO:teuthology.orchestra.run.smithi001.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-09-13T11:08:47.077 INFO:teuthology.orchestra.run.smithi001.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-09-13T11:08:47.083 INFO:teuthology.orchestra.run.smithi001.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-09-13T11:08:47.208 INFO:teuthology.orchestra.run.smithi001.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2024-09-13T11:08:47.208 INFO:teuthology.orchestra.run.smithi001.stderr:Systems on them will not be added to the GRUB boot configuration. 2024-09-13T11:08:47.208 INFO:teuthology.orchestra.run.smithi001.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-09-13T11:08:47.226 INFO:teuthology.orchestra.run.smithi001.stderr:done 2024-09-13T11:08:47.239 DEBUG:teuthology.orchestra.run.smithi001:> sudo shutdown -r now 2024-09-13T11:08:47.577 INFO:teuthology.orchestra.run.smithi162.stderr:Sourcing file `/etc/default/grub' 2024-09-13T11:08:47.577 INFO:teuthology.orchestra.run.smithi162.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-09-13T11:08:47.580 INFO:teuthology.orchestra.run.smithi162.stderr:Generating grub configuration file ... 2024-09-13T11:08:47.738 INFO:teuthology.orchestra.run.smithi162.stderr:Found linux image: /boot/vmlinuz-5.15.0-121-generic 2024-09-13T11:08:47.744 INFO:teuthology.orchestra.run.smithi162.stderr:Found initrd image: /boot/initrd.img-5.15.0-121-generic 2024-09-13T11:08:47.962 INFO:teuthology.orchestra.run.smithi162.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-09-13T11:08:47.968 INFO:teuthology.orchestra.run.smithi162.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-09-13T11:08:48.011 INFO:teuthology.orchestra.run.smithi162.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-09-13T11:08:48.016 INFO:teuthology.orchestra.run.smithi162.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-09-13T11:08:48.176 INFO:teuthology.orchestra.run.smithi162.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2024-09-13T11:08:48.176 INFO:teuthology.orchestra.run.smithi162.stderr:Systems on them will not be added to the GRUB boot configuration. 2024-09-13T11:08:48.176 INFO:teuthology.orchestra.run.smithi162.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-09-13T11:08:48.196 INFO:teuthology.orchestra.run.smithi162.stderr:done 2024-09-13T11:08:48.215 DEBUG:teuthology.orchestra.run.smithi162:> sudo shutdown -r now 2024-09-13T11:08:48.308 INFO:teuthology.orchestra.run.smithi092.stdout:Reading package lists... 2024-09-13T11:08:48.325 DEBUG:teuthology.orchestra.run.smithi092:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-09-13T11:08:48.379 INFO:teuthology.orchestra.run.smithi092.stdout:Reading package lists... 2024-09-13T11:08:48.565 INFO:teuthology.orchestra.run.smithi092.stdout:Building dependency tree... 2024-09-13T11:08:48.566 INFO:teuthology.orchestra.run.smithi092.stdout:Reading state information... 2024-09-13T11:08:48.737 INFO:teuthology.orchestra.run.smithi092.stdout:linux-image-generic is already the newest version (5.15.0.121.121). 2024-09-13T11:08:48.737 INFO:teuthology.orchestra.run.smithi092.stdout:The following packages were automatically installed and are no longer required: 2024-09-13T11:08:48.737 INFO:teuthology.orchestra.run.smithi092.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-09-13T11:08:48.737 INFO:teuthology.orchestra.run.smithi092.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-09-13T11:08:48.737 INFO:teuthology.orchestra.run.smithi092.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-09-13T11:08:48.737 INFO:teuthology.orchestra.run.smithi092.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-09-13T11:08:48.737 INFO:teuthology.orchestra.run.smithi092.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-09-13T11:08:48.737 INFO:teuthology.orchestra.run.smithi092.stdout:Use 'sudo apt autoremove' to remove them. 2024-09-13T11:08:48.772 INFO:teuthology.orchestra.run.smithi092.stdout:0 upgraded, 0 newly installed, 0 to remove and 344 not upgraded. 2024-09-13T11:08:48.774 DEBUG:teuthology.orchestra.run.smithi092:> dpkg -s linux-image-generic 2024-09-13T11:08:48.836 INFO:teuthology.orchestra.run.smithi092.stdout:Package: linux-image-generic 2024-09-13T11:08:48.836 INFO:teuthology.orchestra.run.smithi092.stdout:Status: install ok installed 2024-09-13T11:08:48.836 INFO:teuthology.orchestra.run.smithi092.stdout:Priority: optional 2024-09-13T11:08:48.837 INFO:teuthology.orchestra.run.smithi092.stdout:Section: kernel 2024-09-13T11:08:48.837 INFO:teuthology.orchestra.run.smithi092.stdout:Installed-Size: 21 2024-09-13T11:08:48.837 INFO:teuthology.orchestra.run.smithi092.stdout:Maintainer: Ubuntu Kernel Team 2024-09-13T11:08:48.837 INFO:teuthology.orchestra.run.smithi092.stdout:Architecture: amd64 2024-09-13T11:08:48.837 INFO:teuthology.orchestra.run.smithi092.stdout:Source: linux-meta 2024-09-13T11:08:48.837 INFO:teuthology.orchestra.run.smithi092.stdout:Version: 5.15.0.121.121 2024-09-13T11:08:48.837 INFO:teuthology.orchestra.run.smithi092.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.4), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-121), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.4) 2024-09-13T11:08:48.837 INFO:teuthology.orchestra.run.smithi092.stdout:Depends: linux-image-5.15.0-121-generic, linux-modules-extra-5.15.0-121-generic, linux-firmware, intel-microcode, amd64-microcode 2024-09-13T11:08:48.837 INFO:teuthology.orchestra.run.smithi092.stdout:Recommends: thermald 2024-09-13T11:08:48.837 INFO:teuthology.orchestra.run.smithi092.stdout:Description: Generic Linux kernel image 2024-09-13T11:08:48.837 INFO:teuthology.orchestra.run.smithi092.stdout: This package will always depend on the latest generic kernel image 2024-09-13T11:08:48.837 INFO:teuthology.orchestra.run.smithi092.stdout: available. 2024-09-13T11:08:48.838 DEBUG:teuthology.orchestra.run.smithi092:> mktemp 2024-09-13T11:08:48.882 INFO:teuthology.orchestra.run.smithi092.stdout:/tmp/tmp.d9qOysvqD5 2024-09-13T11:08:48.882 DEBUG:teuthology.orchestra.run.smithi092:> sudo cp /boot/grub/grub.cfg /tmp/tmp.d9qOysvqD5 2024-09-13T11:08:48.938 DEBUG:teuthology.orchestra.run.smithi092:> sudo chmod 0666 /tmp/tmp.d9qOysvqD5 2024-09-13T11:08:49.065 DEBUG:teuthology.orchestra.remote:smithi092:/tmp/tmp.d9qOysvqD5 is 10KB 2024-09-13T11:08:49.080 DEBUG:teuthology.orchestra.run.smithi092:> rm -fr /tmp/tmp.d9qOysvqD5 2024-09-13T11:08:49.085 DEBUG:teuthology.orchestra.run.smithi092:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2024-09-13T11:08:49.141 DEBUG:teuthology.orchestra.run.smithi092:> set -ex 2024-09-13T11:08:49.141 DEBUG:teuthology.orchestra.run.smithi092:> sudo dd of=/etc/grub.d/01_ceph_kernel 2024-09-13T11:08:49.142 DEBUG:teuthology.orchestra.run.smithi092:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2024-09-13T11:08:49.209 INFO:teuthology.task.kernel:Distro Kernel Version: 5.15.0-121-generic 2024-09-13T11:08:49.209 DEBUG:teuthology.orchestra.run.smithi092:> sudo update-grub 2024-09-13T11:08:50.420 INFO:teuthology.orchestra.run.smithi092.stderr:Sourcing file `/etc/default/grub' 2024-09-13T11:08:50.421 INFO:teuthology.orchestra.run.smithi092.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-09-13T11:08:50.423 INFO:teuthology.orchestra.run.smithi092.stderr:Generating grub configuration file ... 2024-09-13T11:08:50.581 INFO:teuthology.orchestra.run.smithi092.stderr:Found linux image: /boot/vmlinuz-5.15.0-121-generic 2024-09-13T11:08:50.584 INFO:teuthology.orchestra.run.smithi092.stderr:Found initrd image: /boot/initrd.img-5.15.0-121-generic 2024-09-13T11:08:50.826 INFO:teuthology.orchestra.run.smithi092.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-09-13T11:08:50.832 INFO:teuthology.orchestra.run.smithi092.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-09-13T11:08:50.874 INFO:teuthology.orchestra.run.smithi092.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-09-13T11:08:50.879 INFO:teuthology.orchestra.run.smithi092.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-09-13T11:08:51.011 INFO:teuthology.orchestra.run.smithi092.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2024-09-13T11:08:51.011 INFO:teuthology.orchestra.run.smithi092.stderr:Systems on them will not be added to the GRUB boot configuration. 2024-09-13T11:08:51.011 INFO:teuthology.orchestra.run.smithi092.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-09-13T11:08:51.022 INFO:teuthology.orchestra.run.smithi092.stderr:done 2024-09-13T11:08:51.035 DEBUG:teuthology.orchestra.run.smithi092:> sudo shutdown -r now 2024-09-13T11:09:16.633 INFO:teuthology.task.kernel:Checking client mgr.x for new kernel version... 2024-09-13T11:09:16.634 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi177.front.sepia.ceph.com' 2024-09-13T11:09:16.636 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi177.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-13T11:09:17.245 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-09-13T11:09:17.245 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi001.front.sepia.ceph.com' 2024-09-13T11:09:17.245 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi001.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-13T11:09:18.218 INFO:teuthology.task.kernel:Checking client mon.c for new kernel version... 2024-09-13T11:09:18.219 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi162.front.sepia.ceph.com' 2024-09-13T11:09:18.219 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi162.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-13T11:09:21.040 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-09-13T11:09:21.041 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi092.front.sepia.ceph.com' 2024-09-13T11:09:21.041 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi092.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-13T11:09:35.166 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.177 2024-09-13T11:09:36.701 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.162 2024-09-13T11:09:39.520 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.92 2024-09-13T11:09:44.169 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi177.front.sepia.ceph.com' 2024-09-13T11:09:44.170 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi177.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-13T11:09:45.704 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi162.front.sepia.ceph.com' 2024-09-13T11:09:45.704 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi162.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-13T11:09:47.229 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.177 2024-09-13T11:09:48.523 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi092.front.sepia.ceph.com' 2024-09-13T11:09:48.524 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi092.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-13T11:09:48.765 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.162 2024-09-13T11:09:51.585 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.92 2024-09-13T11:09:51.805 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.1 2024-09-13T11:09:59.237 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi177.front.sepia.ceph.com' 2024-09-13T11:09:59.237 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi177.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-13T11:10:00.767 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi162.front.sepia.ceph.com' 2024-09-13T11:10:00.768 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi162.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-13T11:10:00.806 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi001.front.sepia.ceph.com' 2024-09-13T11:10:00.806 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi001.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-13T11:10:03.589 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi092.front.sepia.ceph.com' 2024-09-13T11:10:03.589 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi092.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-13T11:10:17.661 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.177 2024-09-13T11:10:19.197 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.1 2024-09-13T11:10:19.197 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.162 2024-09-13T11:10:22.013 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.92 2024-09-13T11:10:31.206 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi001.front.sepia.ceph.com' 2024-09-13T11:10:31.208 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi001.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-13T11:10:32.221 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.1 2024-09-13T11:10:32.663 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi177.front.sepia.ceph.com' 2024-09-13T11:10:32.664 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi177.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-13T11:10:32.992 DEBUG:teuthology.orchestra.run.smithi177:> true 2024-09-13T11:10:33.631 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi177.front.sepia.ceph.com' 2024-09-13T11:10:33.631 INFO:teuthology.task.kernel:Checking kernel version of mgr.x, want "5.15.0-121-generic"... 2024-09-13T11:10:33.632 DEBUG:teuthology.orchestra.run.smithi177:> uname -r 2024-09-13T11:10:33.677 INFO:teuthology.orchestra.run.smithi177.stdout:5.15.0-121-generic 2024-09-13T11:10:33.677 DEBUG:teuthology.task.kernel:current kernel version is 5.15.0-121-generic vs 5.15.0-121-generic 2024-09-13T11:10:33.677 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-13T11:10:33.677 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2024-09-13T11:10:34.199 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi162.front.sepia.ceph.com' 2024-09-13T11:10:34.199 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi162.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-13T11:10:34.678 INFO:teuthology.task.kernel:Enabling kdb on mgr.x... 2024-09-13T11:10:34.679 DEBUG:teuthology.orchestra.run.smithi177:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-13T11:10:34.741 DEBUG:teuthology.orchestra.run.smithi162:> true 2024-09-13T11:10:34.766 INFO:teuthology.orchestra.run.smithi177.stdout:ttyS1 2024-09-13T11:10:34.791 DEBUG:teuthology.parallel:result is None 2024-09-13T11:10:35.773 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi162.front.sepia.ceph.com' 2024-09-13T11:10:35.773 INFO:teuthology.task.kernel:Checking kernel version of mon.c, want "5.15.0-121-generic"... 2024-09-13T11:10:35.773 DEBUG:teuthology.orchestra.run.smithi162:> uname -r 2024-09-13T11:10:35.820 INFO:teuthology.orchestra.run.smithi162.stdout:5.15.0-121-generic 2024-09-13T11:10:35.821 DEBUG:teuthology.task.kernel:current kernel version is 5.15.0-121-generic vs 5.15.0-121-generic 2024-09-13T11:10:35.821 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-13T11:10:35.821 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2024-09-13T11:10:36.821 INFO:teuthology.task.kernel:Enabling kdb on mon.c... 2024-09-13T11:10:36.821 DEBUG:teuthology.orchestra.run.smithi162:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-13T11:10:37.013 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi092.front.sepia.ceph.com' 2024-09-13T11:10:37.014 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi092.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-13T11:10:37.085 INFO:teuthology.orchestra.run.smithi162.stdout:ttyS1 2024-09-13T11:10:37.086 DEBUG:teuthology.parallel:result is None 2024-09-13T11:10:37.335 DEBUG:teuthology.orchestra.run.smithi092:> true 2024-09-13T11:10:37.875 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi092.front.sepia.ceph.com' 2024-09-13T11:10:37.875 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.15.0-121-generic"... 2024-09-13T11:10:37.875 DEBUG:teuthology.orchestra.run.smithi092:> uname -r 2024-09-13T11:10:37.921 INFO:teuthology.orchestra.run.smithi092.stdout:5.15.0-121-generic 2024-09-13T11:10:37.922 DEBUG:teuthology.task.kernel:current kernel version is 5.15.0-121-generic vs 5.15.0-121-generic 2024-09-13T11:10:37.922 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-13T11:10:37.922 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2024-09-13T11:10:38.922 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-09-13T11:10:38.923 DEBUG:teuthology.orchestra.run.smithi092:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-13T11:10:38.990 INFO:teuthology.orchestra.run.smithi092.stdout:ttyS1 2024-09-13T11:10:39.035 DEBUG:teuthology.parallel:result is None 2024-09-13T11:10:47.222 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi001.front.sepia.ceph.com' 2024-09-13T11:10:47.223 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi001.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-13T11:10:47.450 DEBUG:teuthology.orchestra.run.smithi001:> true 2024-09-13T11:10:48.041 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi001.front.sepia.ceph.com' 2024-09-13T11:10:48.041 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.15.0-121-generic"... 2024-09-13T11:10:48.041 DEBUG:teuthology.orchestra.run.smithi001:> uname -r 2024-09-13T11:10:48.089 INFO:teuthology.orchestra.run.smithi001.stdout:5.15.0-121-generic 2024-09-13T11:10:48.089 DEBUG:teuthology.task.kernel:current kernel version is 5.15.0-121-generic vs 5.15.0-121-generic 2024-09-13T11:10:48.089 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-13T11:10:48.089 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2024-09-13T11:10:49.091 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-09-13T11:10:49.091 DEBUG:teuthology.orchestra.run.smithi001:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-13T11:10:49.175 INFO:teuthology.orchestra.run.smithi001.stdout:ttyS1 2024-09-13T11:10:49.211 DEBUG:teuthology.parallel:result is None 2024-09-13T11:10:49.211 INFO:teuthology.run_tasks:Running task internal.base... 2024-09-13T11:10:49.220 INFO:teuthology.task.internal:Creating test directory... 2024-09-13T11:10:49.220 DEBUG:teuthology.orchestra.run.smithi001:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-13T11:10:49.222 DEBUG:teuthology.orchestra.run.smithi092:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-13T11:10:49.225 DEBUG:teuthology.orchestra.run.smithi162:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-13T11:10:49.227 DEBUG:teuthology.orchestra.run.smithi177:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-13T11:10:49.234 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-09-13T11:10:49.248 INFO:teuthology.run_tasks:Running task internal.archive... 2024-09-13T11:10:49.263 INFO:teuthology.task.internal:Creating archive directory... 2024-09-13T11:10:49.263 DEBUG:teuthology.orchestra.run.smithi001:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-13T11:10:49.271 DEBUG:teuthology.orchestra.run.smithi092:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-13T11:10:49.274 DEBUG:teuthology.orchestra.run.smithi162:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-13T11:10:49.278 DEBUG:teuthology.orchestra.run.smithi177:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-13T11:10:49.310 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-09-13T11:10:49.321 INFO:teuthology.task.internal:Enabling coredump saving... 2024-09-13T11:10:49.321 DEBUG:teuthology.orchestra.run.smithi001:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-13T11:10:49.330 DEBUG:teuthology.orchestra.run.smithi092:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-13T11:10:49.342 DEBUG:teuthology.orchestra.run.smithi162:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-13T11:10:49.345 DEBUG:teuthology.orchestra.run.smithi177:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-13T11:10:49.352 INFO:teuthology.orchestra.run.smithi001.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T11:10:49.360 INFO:teuthology.orchestra.run.smithi001.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T11:10:49.362 INFO:teuthology.orchestra.run.smithi177.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T11:10:49.367 INFO:teuthology.orchestra.run.smithi162.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T11:10:49.370 INFO:teuthology.orchestra.run.smithi177.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T11:10:49.377 INFO:teuthology.orchestra.run.smithi162.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T11:10:49.403 INFO:teuthology.orchestra.run.smithi092.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T11:10:49.412 INFO:teuthology.orchestra.run.smithi092.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T11:10:49.413 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-09-13T11:10:49.424 INFO:teuthology.task.internal:Configuring sudo... 2024-09-13T11:10:49.424 DEBUG:teuthology.orchestra.run.smithi001:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-13T11:10:49.427 DEBUG:teuthology.orchestra.run.smithi092:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-13T11:10:49.429 DEBUG:teuthology.orchestra.run.smithi162:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-13T11:10:49.431 DEBUG:teuthology.orchestra.run.smithi177:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-13T11:10:49.454 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-09-13T11:10:49.465 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-09-13T11:10:49.465 DEBUG:teuthology.orchestra.run.smithi001:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-13T11:10:49.482 DEBUG:teuthology.orchestra.run.smithi092:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-13T11:10:49.494 DEBUG:teuthology.orchestra.run.smithi162:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-13T11:10:49.496 DEBUG:teuthology.orchestra.run.smithi177:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-13T11:10:49.502 DEBUG:teuthology.orchestra.run.smithi001:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-13T11:10:49.533 DEBUG:teuthology.orchestra.run.smithi001:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-13T11:10:49.581 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-09-13T11:10:49.581 DEBUG:teuthology.orchestra.run.smithi001:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-13T11:10:49.643 DEBUG:teuthology.orchestra.run.smithi092:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-13T11:10:49.649 DEBUG:teuthology.orchestra.run.smithi092:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-13T11:10:49.698 DEBUG:teuthology.orchestra.run.smithi092:> set -ex 2024-09-13T11:10:49.723 DEBUG:teuthology.orchestra.run.smithi092:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-13T11:10:49.762 DEBUG:teuthology.orchestra.run.smithi162:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-13T11:10:49.768 DEBUG:teuthology.orchestra.run.smithi162:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-13T11:10:49.817 DEBUG:teuthology.orchestra.run.smithi162:> set -ex 2024-09-13T11:10:49.817 DEBUG:teuthology.orchestra.run.smithi162:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-13T11:10:49.889 DEBUG:teuthology.orchestra.run.smithi177:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-13T11:10:49.895 DEBUG:teuthology.orchestra.run.smithi177:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-13T11:10:49.941 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-09-13T11:10:49.941 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-13T11:10:50.009 DEBUG:teuthology.orchestra.run.smithi001:> sudo service rsyslog restart 2024-09-13T11:10:50.011 DEBUG:teuthology.orchestra.run.smithi092:> sudo service rsyslog restart 2024-09-13T11:10:50.014 DEBUG:teuthology.orchestra.run.smithi162:> sudo service rsyslog restart 2024-09-13T11:10:50.016 DEBUG:teuthology.orchestra.run.smithi177:> sudo service rsyslog restart 2024-09-13T11:10:50.088 INFO:teuthology.run_tasks:Running task internal.timer... 2024-09-13T11:10:50.099 INFO:teuthology.task.internal:Starting timer... 2024-09-13T11:10:50.099 INFO:teuthology.run_tasks:Running task pcp... 2024-09-13T11:10:50.116 INFO:teuthology.run_tasks:Running task selinux... 2024-09-13T11:10:50.132 DEBUG:teuthology.task.selinux:Excluding smithi001: OS 'ubuntu' does not support SELinux 2024-09-13T11:10:50.132 DEBUG:teuthology.task.selinux:Excluding smithi092: OS 'ubuntu' does not support SELinux 2024-09-13T11:10:50.132 DEBUG:teuthology.task.selinux:Excluding smithi162: OS 'ubuntu' does not support SELinux 2024-09-13T11:10:50.132 DEBUG:teuthology.task.selinux:Excluding smithi177: OS 'ubuntu' does not support SELinux 2024-09-13T11:10:50.132 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-09-13T11:10:50.133 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2024-09-13T11:10:50.133 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-09-13T11:10:50.133 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-09-13T11:10:50.143 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-09-13T11:10:50.155 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-09-13T11:10:50.157 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi001.front.sepia.ceph.com,smithi092.front.sepia.ceph.com,smithi162.front.sepia.ceph.com,smithi177.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-09-13T11:18:34.988 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi001.front.sepia.ceph.com'), Remote(name='ubuntu@smithi092.front.sepia.ceph.com'), Remote(name='ubuntu@smithi162.front.sepia.ceph.com'), Remote(name='ubuntu@smithi177.front.sepia.ceph.com')] 2024-09-13T11:18:34.990 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi001.front.sepia.ceph.com' 2024-09-13T11:18:34.991 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi001.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-13T11:18:35.078 DEBUG:teuthology.orchestra.run.smithi001:> true 2024-09-13T11:18:35.157 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi001.front.sepia.ceph.com' 2024-09-13T11:18:35.157 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi092.front.sepia.ceph.com' 2024-09-13T11:18:35.158 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi092.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-13T11:18:35.238 DEBUG:teuthology.orchestra.run.smithi092:> true 2024-09-13T11:18:35.313 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi092.front.sepia.ceph.com' 2024-09-13T11:18:35.313 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi162.front.sepia.ceph.com' 2024-09-13T11:18:35.313 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi162.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-13T11:18:35.393 DEBUG:teuthology.orchestra.run.smithi162:> true 2024-09-13T11:18:35.469 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi162.front.sepia.ceph.com' 2024-09-13T11:18:35.469 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi177.front.sepia.ceph.com' 2024-09-13T11:18:35.470 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi177.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-13T11:18:35.546 DEBUG:teuthology.orchestra.run.smithi177:> true 2024-09-13T11:18:35.621 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi177.front.sepia.ceph.com' 2024-09-13T11:18:35.622 INFO:teuthology.run_tasks:Running task clock... 2024-09-13T11:18:35.632 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-09-13T11:18:35.633 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-13T11:18:35.633 DEBUG:teuthology.orchestra.run.smithi001:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-13T11:18:35.636 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-13T11:18:35.636 DEBUG:teuthology.orchestra.run.smithi092:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-13T11:18:35.638 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-13T11:18:35.638 DEBUG:teuthology.orchestra.run.smithi162:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-13T11:18:35.640 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-13T11:18:35.641 DEBUG:teuthology.orchestra.run.smithi177:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-13T11:18:35.665 INFO:teuthology.orchestra.run.smithi001.stdout:13 Sep 11:18:35 ntpd[18372]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2024-09-13T11:18:35.666 INFO:teuthology.orchestra.run.smithi001.stdout:13 Sep 11:18:35 ntpd[18372]: Command line: ntpd -gq 2024-09-13T11:18:35.666 INFO:teuthology.orchestra.run.smithi001.stdout:13 Sep 11:18:35 ntpd[18372]: ---------------------------------------------------- 2024-09-13T11:18:35.666 INFO:teuthology.orchestra.run.smithi001.stdout:13 Sep 11:18:35 ntpd[18372]: ntp-4 is maintained by Network Time Foundation, 2024-09-13T11:18:35.666 INFO:teuthology.orchestra.run.smithi001.stdout:13 Sep 11:18:35 ntpd[18372]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2024-09-13T11:18:35.666 INFO:teuthology.orchestra.run.smithi001.stdout:13 Sep 11:18:35 ntpd[18372]: corporation. Support and training for ntp-4 are 2024-09-13T11:18:35.666 INFO:teuthology.orchestra.run.smithi001.stdout:13 Sep 11:18:35 ntpd[18372]: available at https://www.nwtime.org/support 2024-09-13T11:18:35.666 INFO:teuthology.orchestra.run.smithi001.stdout:13 Sep 11:18:35 ntpd[18372]: ---------------------------------------------------- 2024-09-13T11:18:35.666 INFO:teuthology.orchestra.run.smithi001.stdout:13 Sep 11:18:35 ntpd[18372]: proto: precision = 0.047 usec (-24) 2024-09-13T11:18:35.666 INFO:teuthology.orchestra.run.smithi001.stdout:13 Sep 11:18:35 ntpd[18372]: basedate set to 2022-02-04 2024-09-13T11:18:35.666 INFO:teuthology.orchestra.run.smithi001.stdout:13 Sep 11:18:35 ntpd[18372]: gps base set to 2022-02-06 (week 2196) 2024-09-13T11:18:35.666 INFO:teuthology.orchestra.run.smithi001.stdout:13 Sep 11:18:35 ntpd[18372]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-09-13T11:18:35.667 INFO:teuthology.orchestra.run.smithi001.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-09-13T11:18:35.667 INFO:teuthology.orchestra.run.smithi001.stderr:restrict ::: KOD does nothing without LIMITED. 2024-09-13T11:18:35.668 INFO:teuthology.orchestra.run.smithi001.stdout:13 Sep 11:18:35 ntpd[18372]: restrict ::: KOD does nothing without LIMITED. 2024-09-13T11:18:35.668 INFO:teuthology.orchestra.run.smithi001.stdout:13 Sep 11:18:35 ntpd[18372]: Listen and drop on 0 v6wildcard [::]:123 2024-09-13T11:18:35.668 INFO:teuthology.orchestra.run.smithi001.stdout:13 Sep 11:18:35 ntpd[18372]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2024-09-13T11:18:35.668 INFO:teuthology.orchestra.run.smithi001.stdout:13 Sep 11:18:35 ntpd[18372]: Listen normally on 2 lo 127.0.0.1:123 2024-09-13T11:18:35.669 INFO:teuthology.orchestra.run.smithi001.stdout:13 Sep 11:18:35 ntpd[18372]: Listen normally on 3 ens1f0 172.21.15.1:123 2024-09-13T11:18:35.669 INFO:teuthology.orchestra.run.smithi001.stdout:13 Sep 11:18:35 ntpd[18372]: Listen normally on 4 lo [::1]:123 2024-09-13T11:18:35.669 INFO:teuthology.orchestra.run.smithi001.stdout:13 Sep 11:18:35 ntpd[18372]: Listen normally on 5 ens1f0 [fe80::ec4:7aff:febd:15e8%4]:123 2024-09-13T11:18:35.669 INFO:teuthology.orchestra.run.smithi001.stdout:13 Sep 11:18:35 ntpd[18372]: Listening on routing socket on fd #22 for interface updates 2024-09-13T11:18:35.670 INFO:teuthology.orchestra.run.smithi162.stdout:13 Sep 11:18:35 ntpd[18472]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2024-09-13T11:18:35.670 INFO:teuthology.orchestra.run.smithi162.stdout:13 Sep 11:18:35 ntpd[18472]: Command line: ntpd -gq 2024-09-13T11:18:35.670 INFO:teuthology.orchestra.run.smithi162.stdout:13 Sep 11:18:35 ntpd[18472]: ---------------------------------------------------- 2024-09-13T11:18:35.670 INFO:teuthology.orchestra.run.smithi162.stdout:13 Sep 11:18:35 ntpd[18472]: ntp-4 is maintained by Network Time Foundation, 2024-09-13T11:18:35.671 INFO:teuthology.orchestra.run.smithi162.stdout:13 Sep 11:18:35 ntpd[18472]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2024-09-13T11:18:35.671 INFO:teuthology.orchestra.run.smithi162.stdout:13 Sep 11:18:35 ntpd[18472]: corporation. Support and training for ntp-4 are 2024-09-13T11:18:35.671 INFO:teuthology.orchestra.run.smithi162.stdout:13 Sep 11:18:35 ntpd[18472]: available at https://www.nwtime.org/support 2024-09-13T11:18:35.671 INFO:teuthology.orchestra.run.smithi162.stdout:13 Sep 11:18:35 ntpd[18472]: ---------------------------------------------------- 2024-09-13T11:18:35.671 INFO:teuthology.orchestra.run.smithi162.stdout:13 Sep 11:18:35 ntpd[18472]: proto: precision = 0.043 usec (-24) 2024-09-13T11:18:35.671 INFO:teuthology.orchestra.run.smithi162.stdout:13 Sep 11:18:35 ntpd[18472]: basedate set to 2022-02-04 2024-09-13T11:18:35.671 INFO:teuthology.orchestra.run.smithi162.stdout:13 Sep 11:18:35 ntpd[18472]: gps base set to 2022-02-06 (week 2196) 2024-09-13T11:18:35.671 INFO:teuthology.orchestra.run.smithi092.stdout:13 Sep 11:18:35 ntpd[18508]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2024-09-13T11:18:35.671 INFO:teuthology.orchestra.run.smithi092.stdout:13 Sep 11:18:35 ntpd[18508]: Command line: ntpd -gq 2024-09-13T11:18:35.671 INFO:teuthology.orchestra.run.smithi092.stdout:13 Sep 11:18:35 ntpd[18508]: ---------------------------------------------------- 2024-09-13T11:18:35.671 INFO:teuthology.orchestra.run.smithi092.stdout:13 Sep 11:18:35 ntpd[18508]: ntp-4 is maintained by Network Time Foundation, 2024-09-13T11:18:35.671 INFO:teuthology.orchestra.run.smithi092.stdout:13 Sep 11:18:35 ntpd[18508]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2024-09-13T11:18:35.672 INFO:teuthology.orchestra.run.smithi092.stdout:13 Sep 11:18:35 ntpd[18508]: corporation. Support and training for ntp-4 are 2024-09-13T11:18:35.672 INFO:teuthology.orchestra.run.smithi092.stdout:13 Sep 11:18:35 ntpd[18508]: available at https://www.nwtime.org/support 2024-09-13T11:18:35.672 INFO:teuthology.orchestra.run.smithi092.stdout:13 Sep 11:18:35 ntpd[18508]: ---------------------------------------------------- 2024-09-13T11:18:35.672 INFO:teuthology.orchestra.run.smithi092.stdout:13 Sep 11:18:35 ntpd[18508]: proto: precision = 0.045 usec (-24) 2024-09-13T11:18:35.672 INFO:teuthology.orchestra.run.smithi092.stdout:13 Sep 11:18:35 ntpd[18508]: basedate set to 2022-02-04 2024-09-13T11:18:35.672 INFO:teuthology.orchestra.run.smithi092.stdout:13 Sep 11:18:35 ntpd[18508]: gps base set to 2022-02-06 (week 2196) 2024-09-13T11:18:35.672 INFO:teuthology.orchestra.run.smithi092.stdout:13 Sep 11:18:35 ntpd[18508]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-09-13T11:18:35.672 INFO:teuthology.orchestra.run.smithi092.stdout:13 Sep 11:18:35 ntpd[18508]: restrict ::: KOD does nothing without LIMITED. 2024-09-13T11:18:35.672 INFO:teuthology.orchestra.run.smithi162.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-09-13T11:18:35.673 INFO:teuthology.orchestra.run.smithi092.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-09-13T11:18:35.673 INFO:teuthology.orchestra.run.smithi092.stderr:restrict ::: KOD does nothing without LIMITED. 2024-09-13T11:18:35.673 INFO:teuthology.orchestra.run.smithi162.stdout:13 Sep 11:18:35 ntpd[18472]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-09-13T11:18:35.673 INFO:teuthology.orchestra.run.smithi162.stdout:13 Sep 11:18:35 ntpd[18472]: restrict ::: KOD does nothing without LIMITED. 2024-09-13T11:18:35.673 INFO:teuthology.orchestra.run.smithi162.stderr:restrict ::: KOD does nothing without LIMITED. 2024-09-13T11:18:35.675 INFO:teuthology.orchestra.run.smithi092.stdout:13 Sep 11:18:35 ntpd[18508]: Listen and drop on 0 v6wildcard [::]:123 2024-09-13T11:18:35.675 INFO:teuthology.orchestra.run.smithi092.stdout:13 Sep 11:18:35 ntpd[18508]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2024-09-13T11:18:35.675 INFO:teuthology.orchestra.run.smithi092.stdout:13 Sep 11:18:35 ntpd[18508]: Listen normally on 2 lo 127.0.0.1:123 2024-09-13T11:18:35.675 INFO:teuthology.orchestra.run.smithi092.stdout:13 Sep 11:18:35 ntpd[18508]: Listen normally on 3 enp3s0f1 172.21.15.92:123 2024-09-13T11:18:35.675 INFO:teuthology.orchestra.run.smithi092.stdout:13 Sep 11:18:35 ntpd[18508]: Listen normally on 4 lo [::1]:123 2024-09-13T11:18:35.675 INFO:teuthology.orchestra.run.smithi092.stdout:13 Sep 11:18:35 ntpd[18508]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe8f:cd45%5]:123 2024-09-13T11:18:35.675 INFO:teuthology.orchestra.run.smithi092.stdout:13 Sep 11:18:35 ntpd[18508]: Listening on routing socket on fd #22 for interface updates 2024-09-13T11:18:35.675 INFO:teuthology.orchestra.run.smithi162.stdout:13 Sep 11:18:35 ntpd[18472]: Listen and drop on 0 v6wildcard [::]:123 2024-09-13T11:18:35.676 INFO:teuthology.orchestra.run.smithi162.stdout:13 Sep 11:18:35 ntpd[18472]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2024-09-13T11:18:35.676 INFO:teuthology.orchestra.run.smithi162.stdout:13 Sep 11:18:35 ntpd[18472]: Listen normally on 2 lo 127.0.0.1:123 2024-09-13T11:18:35.676 INFO:teuthology.orchestra.run.smithi162.stdout:13 Sep 11:18:35 ntpd[18472]: Listen normally on 3 enp3s0f1 172.21.15.162:123 2024-09-13T11:18:35.676 INFO:teuthology.orchestra.run.smithi162.stdout:13 Sep 11:18:35 ntpd[18472]: Listen normally on 4 lo [::1]:123 2024-09-13T11:18:35.676 INFO:teuthology.orchestra.run.smithi162.stdout:13 Sep 11:18:35 ntpd[18472]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe88:7f41%5]:123 2024-09-13T11:18:35.676 INFO:teuthology.orchestra.run.smithi162.stdout:13 Sep 11:18:35 ntpd[18472]: Listening on routing socket on fd #22 for interface updates 2024-09-13T11:18:35.689 INFO:teuthology.orchestra.run.smithi177.stdout:13 Sep 11:18:35 ntpd[18489]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2024-09-13T11:18:35.689 INFO:teuthology.orchestra.run.smithi177.stdout:13 Sep 11:18:35 ntpd[18489]: Command line: ntpd -gq 2024-09-13T11:18:35.689 INFO:teuthology.orchestra.run.smithi177.stdout:13 Sep 11:18:35 ntpd[18489]: ---------------------------------------------------- 2024-09-13T11:18:35.689 INFO:teuthology.orchestra.run.smithi177.stdout:13 Sep 11:18:35 ntpd[18489]: ntp-4 is maintained by Network Time Foundation, 2024-09-13T11:18:35.690 INFO:teuthology.orchestra.run.smithi177.stdout:13 Sep 11:18:35 ntpd[18489]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2024-09-13T11:18:35.690 INFO:teuthology.orchestra.run.smithi177.stdout:13 Sep 11:18:35 ntpd[18489]: corporation. Support and training for ntp-4 are 2024-09-13T11:18:35.690 INFO:teuthology.orchestra.run.smithi177.stdout:13 Sep 11:18:35 ntpd[18489]: available at https://www.nwtime.org/support 2024-09-13T11:18:35.690 INFO:teuthology.orchestra.run.smithi177.stdout:13 Sep 11:18:35 ntpd[18489]: ---------------------------------------------------- 2024-09-13T11:18:35.690 INFO:teuthology.orchestra.run.smithi177.stdout:13 Sep 11:18:35 ntpd[18489]: proto: precision = 0.066 usec (-24) 2024-09-13T11:18:35.690 INFO:teuthology.orchestra.run.smithi177.stdout:13 Sep 11:18:35 ntpd[18489]: basedate set to 2022-02-04 2024-09-13T11:18:35.691 INFO:teuthology.orchestra.run.smithi177.stdout:13 Sep 11:18:35 ntpd[18489]: gps base set to 2022-02-06 (week 2196) 2024-09-13T11:18:35.691 INFO:teuthology.orchestra.run.smithi177.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-09-13T11:18:35.691 INFO:teuthology.orchestra.run.smithi177.stdout:13 Sep 11:18:35 ntpd[18489]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-09-13T11:18:35.691 INFO:teuthology.orchestra.run.smithi177.stdout:13 Sep 11:18:35 ntpd[18489]: restrict ::: KOD does nothing without LIMITED. 2024-09-13T11:18:35.692 INFO:teuthology.orchestra.run.smithi177.stderr:restrict ::: KOD does nothing without LIMITED. 2024-09-13T11:18:35.692 INFO:teuthology.orchestra.run.smithi177.stdout:13 Sep 11:18:35 ntpd[18489]: Listen and drop on 0 v6wildcard [::]:123 2024-09-13T11:18:35.692 INFO:teuthology.orchestra.run.smithi177.stdout:13 Sep 11:18:35 ntpd[18489]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2024-09-13T11:18:35.693 INFO:teuthology.orchestra.run.smithi177.stdout:13 Sep 11:18:35 ntpd[18489]: Listen normally on 2 lo 127.0.0.1:123 2024-09-13T11:18:35.693 INFO:teuthology.orchestra.run.smithi177.stdout:13 Sep 11:18:35 ntpd[18489]: Listen normally on 3 enp3s0f1 172.21.15.177:123 2024-09-13T11:18:35.693 INFO:teuthology.orchestra.run.smithi177.stdout:13 Sep 11:18:35 ntpd[18489]: Listen normally on 4 lo [::1]:123 2024-09-13T11:18:35.693 INFO:teuthology.orchestra.run.smithi177.stdout:13 Sep 11:18:35 ntpd[18489]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:feea:b04b%5]:123 2024-09-13T11:18:35.693 INFO:teuthology.orchestra.run.smithi177.stdout:13 Sep 11:18:35 ntpd[18489]: Listening on routing socket on fd #22 for interface updates 2024-09-13T11:18:36.665 INFO:teuthology.orchestra.run.smithi001.stderr:13 Sep 11:18:36 ntpd[18372]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-13T11:18:36.665 INFO:teuthology.orchestra.run.smithi001.stderr:13 Sep 11:18:36 ntpd[18372]: can't open /var/log/ntpstats/rawstats.20240913: Permission denied 2024-09-13T11:18:36.666 INFO:teuthology.orchestra.run.smithi001.stderr:13 Sep 11:18:36 ntpd[18372]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-13T11:18:36.666 INFO:teuthology.orchestra.run.smithi001.stderr:13 Sep 11:18:36 ntpd[18372]: can't open /var/log/ntpstats/rawstats.20240913: Permission denied 2024-09-13T11:18:36.666 INFO:teuthology.orchestra.run.smithi001.stderr:13 Sep 11:18:36 ntpd[18372]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-13T11:18:36.666 INFO:teuthology.orchestra.run.smithi001.stderr:13 Sep 11:18:36 ntpd[18372]: can't open /var/log/ntpstats/peerstats.20240913: Permission denied 2024-09-13T11:18:36.667 INFO:teuthology.orchestra.run.smithi162.stderr:13 Sep 11:18:36 ntpd[18472]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-13T11:18:36.667 INFO:teuthology.orchestra.run.smithi162.stderr:13 Sep 11:18:36 ntpd[18472]: can't open /var/log/ntpstats/rawstats.20240913: Permission denied 2024-09-13T11:18:36.667 INFO:teuthology.orchestra.run.smithi162.stderr:13 Sep 11:18:36 ntpd[18472]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-13T11:18:36.667 INFO:teuthology.orchestra.run.smithi162.stderr:13 Sep 11:18:36 ntpd[18472]: can't open /var/log/ntpstats/rawstats.20240913: Permission denied 2024-09-13T11:18:36.667 INFO:teuthology.orchestra.run.smithi162.stderr:13 Sep 11:18:36 ntpd[18472]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-13T11:18:36.667 INFO:teuthology.orchestra.run.smithi162.stderr:13 Sep 11:18:36 ntpd[18472]: can't open /var/log/ntpstats/peerstats.20240913: Permission denied 2024-09-13T11:18:36.668 INFO:teuthology.orchestra.run.smithi092.stderr:13 Sep 11:18:36 ntpd[18508]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-13T11:18:36.668 INFO:teuthology.orchestra.run.smithi092.stderr:13 Sep 11:18:36 ntpd[18508]: can't open /var/log/ntpstats/rawstats.20240913: Permission denied 2024-09-13T11:18:36.668 INFO:teuthology.orchestra.run.smithi092.stderr:13 Sep 11:18:36 ntpd[18508]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-13T11:18:36.668 INFO:teuthology.orchestra.run.smithi092.stderr:13 Sep 11:18:36 ntpd[18508]: can't open /var/log/ntpstats/rawstats.20240913: Permission denied 2024-09-13T11:18:36.668 INFO:teuthology.orchestra.run.smithi092.stderr:13 Sep 11:18:36 ntpd[18508]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-13T11:18:36.668 INFO:teuthology.orchestra.run.smithi092.stderr:13 Sep 11:18:36 ntpd[18508]: can't open /var/log/ntpstats/rawstats.20240913: Permission denied 2024-09-13T11:18:36.668 INFO:teuthology.orchestra.run.smithi092.stderr:13 Sep 11:18:36 ntpd[18508]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-13T11:18:36.668 INFO:teuthology.orchestra.run.smithi092.stderr:13 Sep 11:18:36 ntpd[18508]: can't open /var/log/ntpstats/peerstats.20240913: Permission denied 2024-09-13T11:18:36.669 INFO:teuthology.orchestra.run.smithi092.stderr:13 Sep 11:18:36 ntpd[18508]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-13T11:18:36.669 INFO:teuthology.orchestra.run.smithi092.stderr:13 Sep 11:18:36 ntpd[18508]: can't open /var/log/ntpstats/rawstats.20240913: Permission denied 2024-09-13T11:18:36.669 INFO:teuthology.orchestra.run.smithi092.stderr:13 Sep 11:18:36 ntpd[18508]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-13T11:18:36.669 INFO:teuthology.orchestra.run.smithi092.stderr:13 Sep 11:18:36 ntpd[18508]: can't open /var/log/ntpstats/peerstats.20240913: Permission denied 2024-09-13T11:18:36.690 INFO:teuthology.orchestra.run.smithi177.stderr:13 Sep 11:18:36 ntpd[18489]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-13T11:18:36.690 INFO:teuthology.orchestra.run.smithi177.stderr:13 Sep 11:18:36 ntpd[18489]: can't open /var/log/ntpstats/rawstats.20240913: Permission denied 2024-09-13T11:18:36.690 INFO:teuthology.orchestra.run.smithi177.stderr:13 Sep 11:18:36 ntpd[18489]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-13T11:18:36.690 INFO:teuthology.orchestra.run.smithi177.stderr:13 Sep 11:18:36 ntpd[18489]: can't open /var/log/ntpstats/rawstats.20240913: Permission denied 2024-09-13T11:18:36.690 INFO:teuthology.orchestra.run.smithi177.stderr:13 Sep 11:18:36 ntpd[18489]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-13T11:18:36.691 INFO:teuthology.orchestra.run.smithi177.stderr:13 Sep 11:18:36 ntpd[18489]: can't open /var/log/ntpstats/rawstats.20240913: Permission denied 2024-09-13T11:18:36.691 INFO:teuthology.orchestra.run.smithi177.stderr:13 Sep 11:18:36 ntpd[18489]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-13T11:18:36.691 INFO:teuthology.orchestra.run.smithi177.stderr:13 Sep 11:18:36 ntpd[18489]: can't open /var/log/ntpstats/peerstats.20240913: Permission denied 2024-09-13T11:18:36.691 INFO:teuthology.orchestra.run.smithi177.stderr:13 Sep 11:18:36 ntpd[18489]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-13T11:18:36.691 INFO:teuthology.orchestra.run.smithi177.stderr:13 Sep 11:18:36 ntpd[18489]: can't open /var/log/ntpstats/rawstats.20240913: Permission denied 2024-09-13T11:18:36.691 INFO:teuthology.orchestra.run.smithi177.stderr:13 Sep 11:18:36 ntpd[18489]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-13T11:18:36.691 INFO:teuthology.orchestra.run.smithi177.stderr:13 Sep 11:18:36 ntpd[18489]: can't open /var/log/ntpstats/peerstats.20240913: Permission denied 2024-09-13T11:18:37.690 INFO:teuthology.orchestra.run.smithi177.stderr:13 Sep 11:18:37 ntpd[18489]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-13T11:18:37.690 INFO:teuthology.orchestra.run.smithi177.stderr:13 Sep 11:18:37 ntpd[18489]: can't open /var/log/ntpstats/rawstats.20240913: Permission denied 2024-09-13T11:18:37.690 INFO:teuthology.orchestra.run.smithi177.stderr:13 Sep 11:18:37 ntpd[18489]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-13T11:18:37.690 INFO:teuthology.orchestra.run.smithi177.stderr:13 Sep 11:18:37 ntpd[18489]: can't open /var/log/ntpstats/rawstats.20240913: Permission denied 2024-09-13T11:18:37.690 INFO:teuthology.orchestra.run.smithi177.stderr:13 Sep 11:18:37 ntpd[18489]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-13T11:18:37.690 INFO:teuthology.orchestra.run.smithi177.stderr:13 Sep 11:18:37 ntpd[18489]: can't open /var/log/ntpstats/peerstats.20240913: Permission denied 2024-09-13T11:18:38.666 INFO:teuthology.orchestra.run.smithi001.stderr:13 Sep 11:18:38 ntpd[18372]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-13T11:18:38.666 INFO:teuthology.orchestra.run.smithi001.stderr:13 Sep 11:18:38 ntpd[18372]: can't open /var/log/ntpstats/rawstats.20240913: Permission denied 2024-09-13T11:18:38.666 INFO:teuthology.orchestra.run.smithi001.stderr:13 Sep 11:18:38 ntpd[18372]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-13T11:18:38.666 INFO:teuthology.orchestra.run.smithi001.stderr:13 Sep 11:18:38 ntpd[18372]: can't open /var/log/ntpstats/rawstats.20240913: Permission denied 2024-09-13T11:18:38.666 INFO:teuthology.orchestra.run.smithi001.stderr:13 Sep 11:18:38 ntpd[18372]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-13T11:18:38.667 INFO:teuthology.orchestra.run.smithi001.stderr:13 Sep 11:18:38 ntpd[18372]: can't open /var/log/ntpstats/rawstats.20240913: Permission denied 2024-09-13T11:18:38.667 INFO:teuthology.orchestra.run.smithi001.stderr:13 Sep 11:18:38 ntpd[18372]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-13T11:18:38.667 INFO:teuthology.orchestra.run.smithi001.stderr:13 Sep 11:18:38 ntpd[18372]: can't open /var/log/ntpstats/peerstats.20240913: Permission denied 2024-09-13T11:18:38.668 INFO:teuthology.orchestra.run.smithi162.stderr:13 Sep 11:18:38 ntpd[18472]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-13T11:18:38.669 INFO:teuthology.orchestra.run.smithi162.stderr:13 Sep 11:18:38 ntpd[18472]: can't open /var/log/ntpstats/rawstats.20240913: Permission denied 2024-09-13T11:18:38.669 INFO:teuthology.orchestra.run.smithi162.stderr:13 Sep 11:18:38 ntpd[18472]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-13T11:18:38.669 INFO:teuthology.orchestra.run.smithi162.stderr:13 Sep 11:18:38 ntpd[18472]: can't open /var/log/ntpstats/rawstats.20240913: Permission denied 2024-09-13T11:18:38.669 INFO:teuthology.orchestra.run.smithi162.stderr:13 Sep 11:18:38 ntpd[18472]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-13T11:18:38.669 INFO:teuthology.orchestra.run.smithi162.stderr:13 Sep 11:18:38 ntpd[18472]: can't open /var/log/ntpstats/peerstats.20240913: Permission denied 2024-09-13T11:18:38.670 INFO:teuthology.orchestra.run.smithi092.stderr:13 Sep 11:18:38 ntpd[18508]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-13T11:18:38.670 INFO:teuthology.orchestra.run.smithi092.stderr:13 Sep 11:18:38 ntpd[18508]: can't open /var/log/ntpstats/rawstats.20240913: Permission denied 2024-09-13T11:18:38.670 INFO:teuthology.orchestra.run.smithi092.stderr:13 Sep 11:18:38 ntpd[18508]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-13T11:18:38.670 INFO:teuthology.orchestra.run.smithi092.stderr:13 Sep 11:18:38 ntpd[18508]: can't open /var/log/ntpstats/rawstats.20240913: Permission denied 2024-09-13T11:18:38.670 INFO:teuthology.orchestra.run.smithi092.stderr:13 Sep 11:18:38 ntpd[18508]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-13T11:18:38.670 INFO:teuthology.orchestra.run.smithi092.stderr:13 Sep 11:18:38 ntpd[18508]: can't open /var/log/ntpstats/rawstats.20240913: Permission denied 2024-09-13T11:18:38.670 INFO:teuthology.orchestra.run.smithi092.stderr:13 Sep 11:18:38 ntpd[18508]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-13T11:18:38.670 INFO:teuthology.orchestra.run.smithi092.stderr:13 Sep 11:18:38 ntpd[18508]: can't open /var/log/ntpstats/rawstats.20240913: Permission denied 2024-09-13T11:18:38.670 INFO:teuthology.orchestra.run.smithi092.stderr:13 Sep 11:18:38 ntpd[18508]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-13T11:18:38.670 INFO:teuthology.orchestra.run.smithi092.stderr:13 Sep 11:18:38 ntpd[18508]: can't open /var/log/ntpstats/peerstats.20240913: Permission denied 2024-09-13T11:18:38.670 INFO:teuthology.orchestra.run.smithi092.stderr:13 Sep 11:18:38 ntpd[18508]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-13T11:18:38.671 INFO:teuthology.orchestra.run.smithi092.stderr:13 Sep 11:18:38 ntpd[18508]: can't open /var/log/ntpstats/rawstats.20240913: Permission denied 2024-09-13T11:18:38.671 INFO:teuthology.orchestra.run.smithi092.stderr:13 Sep 11:18:38 ntpd[18508]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-13T11:18:38.671 INFO:teuthology.orchestra.run.smithi092.stderr:13 Sep 11:18:38 ntpd[18508]: can't open /var/log/ntpstats/peerstats.20240913: Permission denied 2024-09-13T11:18:38.691 INFO:teuthology.orchestra.run.smithi177.stderr:13 Sep 11:18:38 ntpd[18489]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-13T11:18:38.692 INFO:teuthology.orchestra.run.smithi177.stderr:13 Sep 11:18:38 ntpd[18489]: can't open /var/log/ntpstats/rawstats.20240913: Permission denied 2024-09-13T11:18:38.692 INFO:teuthology.orchestra.run.smithi177.stderr:13 Sep 11:18:38 ntpd[18489]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-13T11:18:38.692 INFO:teuthology.orchestra.run.smithi177.stderr:13 Sep 11:18:38 ntpd[18489]: can't open /var/log/ntpstats/rawstats.20240913: Permission denied 2024-09-13T11:18:38.692 INFO:teuthology.orchestra.run.smithi177.stderr:13 Sep 11:18:38 ntpd[18489]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-13T11:18:38.692 INFO:teuthology.orchestra.run.smithi177.stderr:13 Sep 11:18:38 ntpd[18489]: can't open /var/log/ntpstats/rawstats.20240913: Permission denied 2024-09-13T11:18:38.692 INFO:teuthology.orchestra.run.smithi177.stderr:13 Sep 11:18:38 ntpd[18489]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-13T11:18:38.692 INFO:teuthology.orchestra.run.smithi177.stderr:13 Sep 11:18:38 ntpd[18489]: can't open /var/log/ntpstats/peerstats.20240913: Permission denied 2024-09-13T11:18:38.692 INFO:teuthology.orchestra.run.smithi177.stderr:13 Sep 11:18:38 ntpd[18489]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-13T11:18:38.692 INFO:teuthology.orchestra.run.smithi177.stderr:13 Sep 11:18:38 ntpd[18489]: can't open /var/log/ntpstats/rawstats.20240913: Permission denied 2024-09-13T11:18:38.692 INFO:teuthology.orchestra.run.smithi177.stderr:13 Sep 11:18:38 ntpd[18489]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-13T11:18:38.692 INFO:teuthology.orchestra.run.smithi177.stderr:13 Sep 11:18:38 ntpd[18489]: can't open /var/log/ntpstats/peerstats.20240913: Permission denied 2024-09-13T11:18:39.665 INFO:teuthology.orchestra.run.smithi001.stderr:13 Sep 11:18:39 ntpd[18372]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-13T11:18:39.666 INFO:teuthology.orchestra.run.smithi001.stderr:13 Sep 11:18:39 ntpd[18372]: can't open /var/log/ntpstats/rawstats.20240913: Permission denied 2024-09-13T11:18:39.666 INFO:teuthology.orchestra.run.smithi001.stderr:13 Sep 11:18:39 ntpd[18372]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-13T11:18:39.666 INFO:teuthology.orchestra.run.smithi001.stderr:13 Sep 11:18:39 ntpd[18372]: can't open /var/log/ntpstats/rawstats.20240913: Permission denied 2024-09-13T11:18:39.666 INFO:teuthology.orchestra.run.smithi001.stderr:13 Sep 11:18:39 ntpd[18372]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-13T11:18:39.666 INFO:teuthology.orchestra.run.smithi001.stderr:13 Sep 11:18:39 ntpd[18372]: can't open /var/log/ntpstats/peerstats.20240913: Permission denied 2024-09-13T11:18:39.667 INFO:teuthology.orchestra.run.smithi162.stderr:13 Sep 11:18:39 ntpd[18472]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-13T11:18:39.667 INFO:teuthology.orchestra.run.smithi162.stderr:13 Sep 11:18:39 ntpd[18472]: can't open /var/log/ntpstats/rawstats.20240913: Permission denied 2024-09-13T11:18:39.667 INFO:teuthology.orchestra.run.smithi162.stderr:13 Sep 11:18:39 ntpd[18472]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-13T11:18:39.668 INFO:teuthology.orchestra.run.smithi162.stderr:13 Sep 11:18:39 ntpd[18472]: can't open /var/log/ntpstats/rawstats.20240913: Permission denied 2024-09-13T11:18:39.668 INFO:teuthology.orchestra.run.smithi162.stderr:13 Sep 11:18:39 ntpd[18472]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-13T11:18:39.668 INFO:teuthology.orchestra.run.smithi162.stderr:13 Sep 11:18:39 ntpd[18472]: can't open /var/log/ntpstats/peerstats.20240913: Permission denied 2024-09-13T11:18:39.690 INFO:teuthology.orchestra.run.smithi177.stderr:13 Sep 11:18:39 ntpd[18489]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-13T11:18:39.690 INFO:teuthology.orchestra.run.smithi177.stderr:13 Sep 11:18:39 ntpd[18489]: can't open /var/log/ntpstats/rawstats.20240913: Permission denied 2024-09-13T11:18:39.690 INFO:teuthology.orchestra.run.smithi177.stderr:13 Sep 11:18:39 ntpd[18489]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-13T11:18:39.690 INFO:teuthology.orchestra.run.smithi177.stderr:13 Sep 11:18:39 ntpd[18489]: can't open /var/log/ntpstats/rawstats.20240913: Permission denied 2024-09-13T11:18:39.690 INFO:teuthology.orchestra.run.smithi177.stderr:13 Sep 11:18:39 ntpd[18489]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-13T11:18:39.690 INFO:teuthology.orchestra.run.smithi177.stderr:13 Sep 11:18:39 ntpd[18489]: can't open /var/log/ntpstats/peerstats.20240913: Permission denied 2024-09-13T11:18:40.666 INFO:teuthology.orchestra.run.smithi001.stderr:13 Sep 11:18:40 ntpd[18372]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-13T11:18:40.666 INFO:teuthology.orchestra.run.smithi001.stderr:13 Sep 11:18:40 ntpd[18372]: can't open /var/log/ntpstats/rawstats.20240913: Permission denied 2024-09-13T11:18:40.666 INFO:teuthology.orchestra.run.smithi001.stderr:13 Sep 11:18:40 ntpd[18372]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-13T11:18:40.666 INFO:teuthology.orchestra.run.smithi001.stderr:13 Sep 11:18:40 ntpd[18372]: can't open /var/log/ntpstats/rawstats.20240913: Permission denied 2024-09-13T11:18:40.666 INFO:teuthology.orchestra.run.smithi001.stderr:13 Sep 11:18:40 ntpd[18372]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-13T11:18:40.666 INFO:teuthology.orchestra.run.smithi001.stderr:13 Sep 11:18:40 ntpd[18372]: can't open /var/log/ntpstats/rawstats.20240913: Permission denied 2024-09-13T11:18:40.666 INFO:teuthology.orchestra.run.smithi001.stderr:13 Sep 11:18:40 ntpd[18372]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-13T11:18:40.666 INFO:teuthology.orchestra.run.smithi001.stderr:13 Sep 11:18:40 ntpd[18372]: can't open /var/log/ntpstats/peerstats.20240913: Permission denied 2024-09-13T11:18:40.667 INFO:teuthology.orchestra.run.smithi001.stderr:13 Sep 11:18:40 ntpd[18372]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-13T11:18:40.667 INFO:teuthology.orchestra.run.smithi001.stderr:13 Sep 11:18:40 ntpd[18372]: can't open /var/log/ntpstats/rawstats.20240913: Permission denied 2024-09-13T11:18:40.667 INFO:teuthology.orchestra.run.smithi001.stderr:13 Sep 11:18:40 ntpd[18372]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-13T11:18:40.667 INFO:teuthology.orchestra.run.smithi001.stderr:13 Sep 11:18:40 ntpd[18372]: can't open /var/log/ntpstats/peerstats.20240913: Permission denied 2024-09-13T11:18:40.669 INFO:teuthology.orchestra.run.smithi162.stderr:13 Sep 11:18:40 ntpd[18472]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-13T11:18:40.669 INFO:teuthology.orchestra.run.smithi162.stderr:13 Sep 11:18:40 ntpd[18472]: can't open /var/log/ntpstats/rawstats.20240913: Permission denied 2024-09-13T11:18:40.669 INFO:teuthology.orchestra.run.smithi162.stderr:13 Sep 11:18:40 ntpd[18472]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-13T11:18:40.669 INFO:teuthology.orchestra.run.smithi162.stderr:13 Sep 11:18:40 ntpd[18472]: can't open /var/log/ntpstats/rawstats.20240913: Permission denied 2024-09-13T11:18:40.669 INFO:teuthology.orchestra.run.smithi162.stderr:13 Sep 11:18:40 ntpd[18472]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-13T11:18:40.669 INFO:teuthology.orchestra.run.smithi162.stderr:13 Sep 11:18:40 ntpd[18472]: can't open /var/log/ntpstats/rawstats.20240913: Permission denied 2024-09-13T11:18:40.670 INFO:teuthology.orchestra.run.smithi162.stderr:13 Sep 11:18:40 ntpd[18472]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-13T11:18:40.670 INFO:teuthology.orchestra.run.smithi162.stderr:13 Sep 11:18:40 ntpd[18472]: can't open /var/log/ntpstats/rawstats.20240913: Permission denied 2024-09-13T11:18:40.670 INFO:teuthology.orchestra.run.smithi162.stderr:13 Sep 11:18:40 ntpd[18472]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-13T11:18:40.670 INFO:teuthology.orchestra.run.smithi162.stderr:13 Sep 11:18:40 ntpd[18472]: can't open /var/log/ntpstats/peerstats.20240913: Permission denied 2024-09-13T11:18:40.670 INFO:teuthology.orchestra.run.smithi162.stderr:13 Sep 11:18:40 ntpd[18472]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-13T11:18:40.670 INFO:teuthology.orchestra.run.smithi162.stderr:13 Sep 11:18:40 ntpd[18472]: can't open /var/log/ntpstats/rawstats.20240913: Permission denied 2024-09-13T11:18:40.670 INFO:teuthology.orchestra.run.smithi162.stderr:13 Sep 11:18:40 ntpd[18472]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-13T11:18:40.670 INFO:teuthology.orchestra.run.smithi162.stderr:13 Sep 11:18:40 ntpd[18472]: can't open /var/log/ntpstats/peerstats.20240913: Permission denied 2024-09-13T11:18:40.671 INFO:teuthology.orchestra.run.smithi092.stderr:13 Sep 11:18:40 ntpd[18508]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-13T11:18:40.671 INFO:teuthology.orchestra.run.smithi092.stderr:13 Sep 11:18:40 ntpd[18508]: can't open /var/log/ntpstats/rawstats.20240913: Permission denied 2024-09-13T11:18:40.671 INFO:teuthology.orchestra.run.smithi092.stderr:13 Sep 11:18:40 ntpd[18508]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-13T11:18:40.671 INFO:teuthology.orchestra.run.smithi092.stderr:13 Sep 11:18:40 ntpd[18508]: can't open /var/log/ntpstats/rawstats.20240913: Permission denied 2024-09-13T11:18:40.671 INFO:teuthology.orchestra.run.smithi092.stderr:13 Sep 11:18:40 ntpd[18508]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-13T11:18:40.671 INFO:teuthology.orchestra.run.smithi092.stderr:13 Sep 11:18:40 ntpd[18508]: can't open /var/log/ntpstats/rawstats.20240913: Permission denied 2024-09-13T11:18:40.671 INFO:teuthology.orchestra.run.smithi092.stderr:13 Sep 11:18:40 ntpd[18508]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-13T11:18:40.671 INFO:teuthology.orchestra.run.smithi092.stderr:13 Sep 11:18:40 ntpd[18508]: can't open /var/log/ntpstats/rawstats.20240913: Permission denied 2024-09-13T11:18:40.671 INFO:teuthology.orchestra.run.smithi092.stderr:13 Sep 11:18:40 ntpd[18508]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-13T11:18:40.671 INFO:teuthology.orchestra.run.smithi092.stderr:13 Sep 11:18:40 ntpd[18508]: can't open /var/log/ntpstats/peerstats.20240913: Permission denied 2024-09-13T11:18:40.671 INFO:teuthology.orchestra.run.smithi092.stderr:13 Sep 11:18:40 ntpd[18508]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-13T11:18:40.671 INFO:teuthology.orchestra.run.smithi092.stderr:13 Sep 11:18:40 ntpd[18508]: can't open /var/log/ntpstats/rawstats.20240913: Permission denied 2024-09-13T11:18:40.671 INFO:teuthology.orchestra.run.smithi092.stderr:13 Sep 11:18:40 ntpd[18508]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-13T11:18:40.672 INFO:teuthology.orchestra.run.smithi092.stderr:13 Sep 11:18:40 ntpd[18508]: can't open /var/log/ntpstats/peerstats.20240913: Permission denied 2024-09-13T11:18:40.672 INFO:teuthology.orchestra.run.smithi092.stderr:13 Sep 11:18:40 ntpd[18508]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-13T11:18:40.672 INFO:teuthology.orchestra.run.smithi092.stderr:13 Sep 11:18:40 ntpd[18508]: can't open /var/log/ntpstats/rawstats.20240913: Permission denied 2024-09-13T11:18:40.672 INFO:teuthology.orchestra.run.smithi092.stderr:13 Sep 11:18:40 ntpd[18508]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-13T11:18:40.672 INFO:teuthology.orchestra.run.smithi092.stderr:13 Sep 11:18:40 ntpd[18508]: can't open /var/log/ntpstats/peerstats.20240913: Permission denied 2024-09-13T11:18:40.690 INFO:teuthology.orchestra.run.smithi177.stderr:13 Sep 11:18:40 ntpd[18489]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-13T11:18:40.690 INFO:teuthology.orchestra.run.smithi177.stderr:13 Sep 11:18:40 ntpd[18489]: can't open /var/log/ntpstats/rawstats.20240913: Permission denied 2024-09-13T11:18:40.690 INFO:teuthology.orchestra.run.smithi177.stderr:13 Sep 11:18:40 ntpd[18489]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-13T11:18:40.691 INFO:teuthology.orchestra.run.smithi177.stderr:13 Sep 11:18:40 ntpd[18489]: can't open /var/log/ntpstats/rawstats.20240913: Permission denied 2024-09-13T11:18:40.691 INFO:teuthology.orchestra.run.smithi177.stderr:13 Sep 11:18:40 ntpd[18489]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-13T11:18:40.691 INFO:teuthology.orchestra.run.smithi177.stderr:13 Sep 11:18:40 ntpd[18489]: can't open /var/log/ntpstats/rawstats.20240913: Permission denied 2024-09-13T11:18:40.691 INFO:teuthology.orchestra.run.smithi177.stderr:13 Sep 11:18:40 ntpd[18489]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-13T11:18:40.691 INFO:teuthology.orchestra.run.smithi177.stderr:13 Sep 11:18:40 ntpd[18489]: can't open /var/log/ntpstats/rawstats.20240913: Permission denied 2024-09-13T11:18:40.691 INFO:teuthology.orchestra.run.smithi177.stderr:13 Sep 11:18:40 ntpd[18489]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-13T11:18:40.691 INFO:teuthology.orchestra.run.smithi177.stderr:13 Sep 11:18:40 ntpd[18489]: can't open /var/log/ntpstats/peerstats.20240913: Permission denied 2024-09-13T11:18:40.691 INFO:teuthology.orchestra.run.smithi177.stderr:13 Sep 11:18:40 ntpd[18489]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-13T11:18:40.691 INFO:teuthology.orchestra.run.smithi177.stderr:13 Sep 11:18:40 ntpd[18489]: can't open /var/log/ntpstats/rawstats.20240913: Permission denied 2024-09-13T11:18:40.691 INFO:teuthology.orchestra.run.smithi177.stderr:13 Sep 11:18:40 ntpd[18489]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-13T11:18:40.691 INFO:teuthology.orchestra.run.smithi177.stderr:13 Sep 11:18:40 ntpd[18489]: can't open /var/log/ntpstats/peerstats.20240913: Permission denied 2024-09-13T11:18:41.665 INFO:teuthology.orchestra.run.smithi001.stderr:13 Sep 11:18:41 ntpd[18372]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-13T11:18:41.666 INFO:teuthology.orchestra.run.smithi001.stderr:13 Sep 11:18:41 ntpd[18372]: can't open /var/log/ntpstats/rawstats.20240913: Permission denied 2024-09-13T11:18:41.666 INFO:teuthology.orchestra.run.smithi001.stderr:13 Sep 11:18:41 ntpd[18372]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-13T11:18:41.666 INFO:teuthology.orchestra.run.smithi001.stderr:13 Sep 11:18:41 ntpd[18372]: can't open /var/log/ntpstats/rawstats.20240913: Permission denied 2024-09-13T11:18:41.666 INFO:teuthology.orchestra.run.smithi001.stderr:13 Sep 11:18:41 ntpd[18372]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-13T11:18:41.666 INFO:teuthology.orchestra.run.smithi001.stderr:13 Sep 11:18:41 ntpd[18372]: can't open /var/log/ntpstats/peerstats.20240913: Permission denied 2024-09-13T11:18:41.667 INFO:teuthology.orchestra.run.smithi162.stderr:13 Sep 11:18:41 ntpd[18472]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-13T11:18:41.667 INFO:teuthology.orchestra.run.smithi162.stderr:13 Sep 11:18:41 ntpd[18472]: can't open /var/log/ntpstats/rawstats.20240913: Permission denied 2024-09-13T11:18:41.667 INFO:teuthology.orchestra.run.smithi162.stderr:13 Sep 11:18:41 ntpd[18472]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-13T11:18:41.667 INFO:teuthology.orchestra.run.smithi162.stderr:13 Sep 11:18:41 ntpd[18472]: can't open /var/log/ntpstats/rawstats.20240913: Permission denied 2024-09-13T11:18:41.667 INFO:teuthology.orchestra.run.smithi162.stderr:13 Sep 11:18:41 ntpd[18472]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-13T11:18:41.667 INFO:teuthology.orchestra.run.smithi162.stderr:13 Sep 11:18:41 ntpd[18472]: can't open /var/log/ntpstats/peerstats.20240913: Permission denied 2024-09-13T11:18:41.690 INFO:teuthology.orchestra.run.smithi177.stderr:13 Sep 11:18:41 ntpd[18489]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-13T11:18:41.690 INFO:teuthology.orchestra.run.smithi177.stderr:13 Sep 11:18:41 ntpd[18489]: can't open /var/log/ntpstats/rawstats.20240913: Permission denied 2024-09-13T11:18:41.690 INFO:teuthology.orchestra.run.smithi177.stderr:13 Sep 11:18:41 ntpd[18489]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-13T11:18:41.690 INFO:teuthology.orchestra.run.smithi177.stderr:13 Sep 11:18:41 ntpd[18489]: can't open /var/log/ntpstats/rawstats.20240913: Permission denied 2024-09-13T11:18:41.690 INFO:teuthology.orchestra.run.smithi177.stderr:13 Sep 11:18:41 ntpd[18489]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-13T11:18:41.690 INFO:teuthology.orchestra.run.smithi177.stderr:13 Sep 11:18:41 ntpd[18489]: can't open /var/log/ntpstats/peerstats.20240913: Permission denied 2024-09-13T11:18:42.665 INFO:teuthology.orchestra.run.smithi001.stderr:13 Sep 11:18:42 ntpd[18372]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-13T11:18:42.666 INFO:teuthology.orchestra.run.smithi001.stderr:13 Sep 11:18:42 ntpd[18372]: can't open /var/log/ntpstats/rawstats.20240913: Permission denied 2024-09-13T11:18:42.666 INFO:teuthology.orchestra.run.smithi001.stderr:13 Sep 11:18:42 ntpd[18372]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-13T11:18:42.666 INFO:teuthology.orchestra.run.smithi001.stderr:13 Sep 11:18:42 ntpd[18372]: can't open /var/log/ntpstats/rawstats.20240913: Permission denied 2024-09-13T11:18:42.666 INFO:teuthology.orchestra.run.smithi001.stderr:13 Sep 11:18:42 ntpd[18372]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-13T11:18:42.666 INFO:teuthology.orchestra.run.smithi001.stderr:13 Sep 11:18:42 ntpd[18372]: can't open /var/log/ntpstats/rawstats.20240913: Permission denied 2024-09-13T11:18:42.666 INFO:teuthology.orchestra.run.smithi001.stderr:13 Sep 11:18:42 ntpd[18372]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-13T11:18:42.666 INFO:teuthology.orchestra.run.smithi001.stderr:13 Sep 11:18:42 ntpd[18372]: can't open /var/log/ntpstats/peerstats.20240913: Permission denied 2024-09-13T11:18:42.667 INFO:teuthology.orchestra.run.smithi001.stdout:13 Sep 11:18:42 ntpd[18372]: ntpd: time slew -0.000746 s 2024-09-13T11:18:42.667 INFO:teuthology.orchestra.run.smithi001.stdout:ntpd: time slew -0.000746s 2024-09-13T11:18:42.667 INFO:teuthology.orchestra.run.smithi001.stderr:13 Sep 11:18:42 ntpd[18372]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2024-09-13T11:18:42.667 INFO:teuthology.orchestra.run.smithi001.stderr:13 Sep 11:18:42 ntpd[18372]: can't open /var/log/ntpstats/loopstats.20240913: Permission denied 2024-09-13T11:18:42.669 INFO:teuthology.orchestra.run.smithi162.stderr:13 Sep 11:18:42 ntpd[18472]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-13T11:18:42.669 INFO:teuthology.orchestra.run.smithi162.stderr:13 Sep 11:18:42 ntpd[18472]: can't open /var/log/ntpstats/rawstats.20240913: Permission denied 2024-09-13T11:18:42.669 INFO:teuthology.orchestra.run.smithi162.stderr:13 Sep 11:18:42 ntpd[18472]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-13T11:18:42.669 INFO:teuthology.orchestra.run.smithi162.stderr:13 Sep 11:18:42 ntpd[18472]: can't open /var/log/ntpstats/rawstats.20240913: Permission denied 2024-09-13T11:18:42.669 INFO:teuthology.orchestra.run.smithi162.stderr:13 Sep 11:18:42 ntpd[18472]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-13T11:18:42.669 INFO:teuthology.orchestra.run.smithi162.stderr:13 Sep 11:18:42 ntpd[18472]: can't open /var/log/ntpstats/rawstats.20240913: Permission denied 2024-09-13T11:18:42.669 INFO:teuthology.orchestra.run.smithi162.stderr:13 Sep 11:18:42 ntpd[18472]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-13T11:18:42.669 INFO:teuthology.orchestra.run.smithi162.stderr:13 Sep 11:18:42 ntpd[18472]: can't open /var/log/ntpstats/peerstats.20240913: Permission denied 2024-09-13T11:18:42.670 INFO:teuthology.orchestra.run.smithi162.stdout:13 Sep 11:18:42 ntpd[18472]: ntpd: time slew -0.000040 s 2024-09-13T11:18:42.670 INFO:teuthology.orchestra.run.smithi162.stdout:ntpd: time slew -0.000040s 2024-09-13T11:18:42.670 INFO:teuthology.orchestra.run.smithi092.stderr:13 Sep 11:18:42 ntpd[18508]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-13T11:18:42.670 INFO:teuthology.orchestra.run.smithi092.stderr:13 Sep 11:18:42 ntpd[18508]: can't open /var/log/ntpstats/rawstats.20240913: Permission denied 2024-09-13T11:18:42.670 INFO:teuthology.orchestra.run.smithi092.stderr:13 Sep 11:18:42 ntpd[18508]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-13T11:18:42.670 INFO:teuthology.orchestra.run.smithi092.stderr:13 Sep 11:18:42 ntpd[18508]: can't open /var/log/ntpstats/rawstats.20240913: Permission denied 2024-09-13T11:18:42.670 INFO:teuthology.orchestra.run.smithi092.stderr:13 Sep 11:18:42 ntpd[18508]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-13T11:18:42.671 INFO:teuthology.orchestra.run.smithi092.stderr:13 Sep 11:18:42 ntpd[18508]: can't open /var/log/ntpstats/rawstats.20240913: Permission denied 2024-09-13T11:18:42.671 INFO:teuthology.orchestra.run.smithi092.stderr:13 Sep 11:18:42 ntpd[18508]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-13T11:18:42.671 INFO:teuthology.orchestra.run.smithi092.stderr:13 Sep 11:18:42 ntpd[18508]: can't open /var/log/ntpstats/rawstats.20240913: Permission denied 2024-09-13T11:18:42.671 INFO:teuthology.orchestra.run.smithi092.stderr:13 Sep 11:18:42 ntpd[18508]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-13T11:18:42.671 INFO:teuthology.orchestra.run.smithi092.stderr:13 Sep 11:18:42 ntpd[18508]: can't open /var/log/ntpstats/peerstats.20240913: Permission denied 2024-09-13T11:18:42.671 INFO:teuthology.orchestra.run.smithi092.stdout:13 Sep 11:18:42 ntpd[18508]: ntpd: time slew -0.000716 s 2024-09-13T11:18:42.671 INFO:teuthology.orchestra.run.smithi092.stdout:ntpd: time slew -0.000716s 2024-09-13T11:18:42.672 INFO:teuthology.orchestra.run.smithi162.stderr:13 Sep 11:18:42 ntpd[18472]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2024-09-13T11:18:42.672 INFO:teuthology.orchestra.run.smithi162.stderr:13 Sep 11:18:42 ntpd[18472]: can't open /var/log/ntpstats/loopstats.20240913: Permission denied 2024-09-13T11:18:42.672 INFO:teuthology.orchestra.run.smithi092.stderr:13 Sep 11:18:42 ntpd[18508]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2024-09-13T11:18:42.672 INFO:teuthology.orchestra.run.smithi092.stderr:13 Sep 11:18:42 ntpd[18508]: can't open /var/log/ntpstats/loopstats.20240913: Permission denied 2024-09-13T11:18:42.690 INFO:teuthology.orchestra.run.smithi177.stderr:13 Sep 11:18:42 ntpd[18489]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-13T11:18:42.690 INFO:teuthology.orchestra.run.smithi177.stderr:13 Sep 11:18:42 ntpd[18489]: can't open /var/log/ntpstats/rawstats.20240913: Permission denied 2024-09-13T11:18:42.690 INFO:teuthology.orchestra.run.smithi177.stderr:13 Sep 11:18:42 ntpd[18489]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-13T11:18:42.690 INFO:teuthology.orchestra.run.smithi177.stderr:13 Sep 11:18:42 ntpd[18489]: can't open /var/log/ntpstats/rawstats.20240913: Permission denied 2024-09-13T11:18:42.690 INFO:teuthology.orchestra.run.smithi177.stderr:13 Sep 11:18:42 ntpd[18489]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-13T11:18:42.690 INFO:teuthology.orchestra.run.smithi177.stderr:13 Sep 11:18:42 ntpd[18489]: can't open /var/log/ntpstats/rawstats.20240913: Permission denied 2024-09-13T11:18:42.690 INFO:teuthology.orchestra.run.smithi177.stderr:13 Sep 11:18:42 ntpd[18489]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-13T11:18:42.690 INFO:teuthology.orchestra.run.smithi177.stderr:13 Sep 11:18:42 ntpd[18489]: can't open /var/log/ntpstats/peerstats.20240913: Permission denied 2024-09-13T11:18:42.691 INFO:teuthology.orchestra.run.smithi177.stdout:13 Sep 11:18:42 ntpd[18489]: ntpd: time slew -0.001004 s 2024-09-13T11:18:42.691 INFO:teuthology.orchestra.run.smithi177.stdout:ntpd: time slew -0.001004s 2024-09-13T11:18:42.691 INFO:teuthology.orchestra.run.smithi177.stderr:13 Sep 11:18:42 ntpd[18489]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2024-09-13T11:18:42.691 INFO:teuthology.orchestra.run.smithi177.stderr:13 Sep 11:18:42 ntpd[18489]: can't open /var/log/ntpstats/loopstats.20240913: Permission denied 2024-09-13T11:18:42.716 INFO:teuthology.orchestra.run.smithi162.stdout: remote refid st t when poll reach delay offset jitter 2024-09-13T11:18:42.716 INFO:teuthology.orchestra.run.smithi162.stdout:============================================================================== 2024-09-13T11:18:42.716 INFO:teuthology.orchestra.run.smithi162.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-13T11:18:42.716 INFO:teuthology.orchestra.run.smithi162.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-13T11:18:42.716 INFO:teuthology.orchestra.run.smithi162.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-13T11:18:42.717 INFO:teuthology.orchestra.run.smithi162.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-13T11:18:42.719 INFO:teuthology.orchestra.run.smithi092.stdout: remote refid st t when poll reach delay offset jitter 2024-09-13T11:18:42.719 INFO:teuthology.orchestra.run.smithi092.stdout:============================================================================== 2024-09-13T11:18:42.719 INFO:teuthology.orchestra.run.smithi092.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-13T11:18:42.719 INFO:teuthology.orchestra.run.smithi092.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-13T11:18:42.719 INFO:teuthology.orchestra.run.smithi092.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-13T11:18:42.719 INFO:teuthology.orchestra.run.smithi092.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-13T11:18:42.720 INFO:teuthology.orchestra.run.smithi001.stdout: remote refid st t when poll reach delay offset jitter 2024-09-13T11:18:42.720 INFO:teuthology.orchestra.run.smithi001.stdout:============================================================================== 2024-09-13T11:18:42.721 INFO:teuthology.orchestra.run.smithi001.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-13T11:18:42.721 INFO:teuthology.orchestra.run.smithi001.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-13T11:18:42.721 INFO:teuthology.orchestra.run.smithi001.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-13T11:18:42.721 INFO:teuthology.orchestra.run.smithi001.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-13T11:18:42.734 INFO:teuthology.orchestra.run.smithi177.stdout: remote refid st t when poll reach delay offset jitter 2024-09-13T11:18:42.734 INFO:teuthology.orchestra.run.smithi177.stdout:============================================================================== 2024-09-13T11:18:42.734 INFO:teuthology.orchestra.run.smithi177.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-13T11:18:42.734 INFO:teuthology.orchestra.run.smithi177.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-13T11:18:42.734 INFO:teuthology.orchestra.run.smithi177.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-13T11:18:42.734 INFO:teuthology.orchestra.run.smithi177.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-13T11:18:42.735 INFO:teuthology.run_tasks:Running task install... 2024-09-13T11:18:42.744 DEBUG:teuthology.task.install:project ceph 2024-09-13T11:18:42.744 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'd58f73069a8157a9ace12a0247ee3e8497707ea9'}} 2024-09-13T11:18:42.744 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': 'd58f73069a8157a9ace12a0247ee3e8497707ea9'} 2024-09-13T11:18:42.744 INFO:teuthology.task.install:Using flavor: default 2024-09-13T11:18:42.750 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-09-13T11:18:42.750 INFO:teuthology.task.install:extra packages: [] 2024-09-13T11:18:42.750 DEBUG:teuthology.orchestra.run.smithi001:> sudo apt-key list | grep Ceph 2024-09-13T11:18:42.751 DEBUG:teuthology.orchestra.run.smithi092:> sudo apt-key list | grep Ceph 2024-09-13T11:18:42.751 DEBUG:teuthology.orchestra.run.smithi162:> sudo apt-key list | grep Ceph 2024-09-13T11:18:42.751 DEBUG:teuthology.orchestra.run.smithi177:> sudo apt-key list | grep Ceph 2024-09-13T11:18:42.814 INFO:teuthology.orchestra.run.smithi092.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2024-09-13T11:18:42.819 INFO:teuthology.orchestra.run.smithi162.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2024-09-13T11:18:42.823 INFO:teuthology.orchestra.run.smithi001.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2024-09-13T11:18:42.836 INFO:teuthology.orchestra.run.smithi177.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2024-09-13T11:18:42.840 INFO:teuthology.orchestra.run.smithi092.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2024-09-13T11:18:42.841 INFO:teuthology.orchestra.run.smithi092.stdout:uid [ unknown] Ceph.com (release key) 2024-09-13T11:18:42.842 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2024-09-13T11:18:42.842 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=d58f73069a8157a9ace12a0247ee3e8497707ea9 2024-09-13T11:18:42.847 INFO:teuthology.orchestra.run.smithi162.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2024-09-13T11:18:42.847 INFO:teuthology.orchestra.run.smithi162.stdout:uid [ unknown] Ceph.com (release key) 2024-09-13T11:18:42.848 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2024-09-13T11:18:42.849 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=d58f73069a8157a9ace12a0247ee3e8497707ea9 2024-09-13T11:18:42.853 INFO:teuthology.orchestra.run.smithi001.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2024-09-13T11:18:42.853 INFO:teuthology.orchestra.run.smithi001.stdout:uid [ unknown] Ceph.com (release key) 2024-09-13T11:18:42.854 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2024-09-13T11:18:42.854 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=d58f73069a8157a9ace12a0247ee3e8497707ea9 2024-09-13T11:18:42.864 INFO:teuthology.orchestra.run.smithi177.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2024-09-13T11:18:42.864 INFO:teuthology.orchestra.run.smithi177.stdout:uid [ unknown] Ceph.com (release key) 2024-09-13T11:18:42.865 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2024-09-13T11:18:42.865 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=d58f73069a8157a9ace12a0247ee3e8497707ea9 2024-09-13T11:18:42.981 INFO:teuthology.task.install.deb:Pulling from https://2.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/ubuntu/jammy/flavors/default/ 2024-09-13T11:18:42.982 INFO:teuthology.task.install.deb:Package version is 19.3.0-4697-gd58f7306-1jammy 2024-09-13T11:18:42.994 INFO:teuthology.task.install.deb:Pulling from https://2.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/ubuntu/jammy/flavors/default/ 2024-09-13T11:18:42.994 INFO:teuthology.task.install.deb:Package version is 19.3.0-4697-gd58f7306-1jammy 2024-09-13T11:18:42.997 INFO:teuthology.task.install.deb:Pulling from https://2.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/ubuntu/jammy/flavors/default/ 2024-09-13T11:18:42.997 INFO:teuthology.task.install.deb:Package version is 19.3.0-4697-gd58f7306-1jammy 2024-09-13T11:18:43.005 INFO:teuthology.task.install.deb:Pulling from https://2.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/ubuntu/jammy/flavors/default/ 2024-09-13T11:18:43.005 INFO:teuthology.task.install.deb:Package version is 19.3.0-4697-gd58f7306-1jammy 2024-09-13T11:18:43.097 DEBUG:teuthology.orchestra.run.smithi092:> set -ex 2024-09-13T11:18:43.097 DEBUG:teuthology.orchestra.run.smithi092:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2024-09-13T11:18:43.103 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-09-13T11:18:43.103 DEBUG:teuthology.orchestra.run.smithi001:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2024-09-13T11:18:43.109 DEBUG:teuthology.orchestra.run.smithi162:> set -ex 2024-09-13T11:18:43.109 DEBUG:teuthology.orchestra.run.smithi162:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2024-09-13T11:18:43.110 DEBUG:teuthology.orchestra.run.smithi092:> sudo apt-get update 2024-09-13T11:18:43.118 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-09-13T11:18:43.118 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2024-09-13T11:18:43.119 DEBUG:teuthology.orchestra.run.smithi001:> sudo apt-get update 2024-09-13T11:18:43.123 DEBUG:teuthology.orchestra.run.smithi162:> sudo apt-get update 2024-09-13T11:18:43.132 DEBUG:teuthology.orchestra.run.smithi177:> sudo apt-get update 2024-09-13T11:18:43.274 INFO:teuthology.orchestra.run.smithi092.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-09-13T11:18:43.285 INFO:teuthology.orchestra.run.smithi092.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-09-13T11:18:43.287 INFO:teuthology.orchestra.run.smithi162.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-09-13T11:18:43.288 INFO:teuthology.orchestra.run.smithi162.stdout:Hit:2 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-09-13T11:18:43.288 INFO:teuthology.orchestra.run.smithi001.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-09-13T11:18:43.297 INFO:teuthology.orchestra.run.smithi162.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-09-13T11:18:43.299 INFO:teuthology.orchestra.run.smithi001.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-09-13T11:18:43.300 INFO:teuthology.orchestra.run.smithi177.stdout:Hit:1 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-09-13T11:18:43.305 INFO:teuthology.orchestra.run.smithi092.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-09-13T11:18:43.315 INFO:teuthology.orchestra.run.smithi162.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-09-13T11:18:43.316 INFO:teuthology.orchestra.run.smithi001.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-09-13T11:18:43.328 INFO:teuthology.orchestra.run.smithi092.stdout:Ign:4 https://2.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/ubuntu/jammy/flavors/default jammy InRelease 2024-09-13T11:18:43.340 INFO:teuthology.orchestra.run.smithi162.stdout:Ign:5 https://2.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/ubuntu/jammy/flavors/default jammy InRelease 2024-09-13T11:18:43.344 INFO:teuthology.orchestra.run.smithi001.stdout:Ign:4 https://2.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/ubuntu/jammy/flavors/default jammy InRelease 2024-09-13T11:18:43.353 INFO:teuthology.orchestra.run.smithi177.stdout:Ign:2 https://2.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/ubuntu/jammy/flavors/default jammy InRelease 2024-09-13T11:18:43.353 INFO:teuthology.orchestra.run.smithi092.stdout:Get:5 https://2.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/ubuntu/jammy/flavors/default jammy Release [7,689 B] 2024-09-13T11:18:43.361 INFO:teuthology.orchestra.run.smithi162.stdout:Get:6 https://2.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/ubuntu/jammy/flavors/default jammy Release [7,689 B] 2024-09-13T11:18:43.365 INFO:teuthology.orchestra.run.smithi001.stdout:Get:5 https://2.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/ubuntu/jammy/flavors/default jammy Release [7,689 B] 2024-09-13T11:18:43.374 INFO:teuthology.orchestra.run.smithi177.stdout:Get:3 https://2.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/ubuntu/jammy/flavors/default jammy Release [7,689 B] 2024-09-13T11:18:43.374 INFO:teuthology.orchestra.run.smithi092.stdout:Ign:6 https://2.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/ubuntu/jammy/flavors/default jammy Release.gpg 2024-09-13T11:18:43.383 INFO:teuthology.orchestra.run.smithi162.stdout:Ign:7 https://2.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/ubuntu/jammy/flavors/default jammy Release.gpg 2024-09-13T11:18:43.387 INFO:teuthology.orchestra.run.smithi001.stdout:Ign:6 https://2.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/ubuntu/jammy/flavors/default jammy Release.gpg 2024-09-13T11:18:43.396 INFO:teuthology.orchestra.run.smithi177.stdout:Ign:4 https://2.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/ubuntu/jammy/flavors/default jammy Release.gpg 2024-09-13T11:18:43.396 INFO:teuthology.orchestra.run.smithi092.stdout:Get:7 https://2.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/ubuntu/jammy/flavors/default jammy/main i386 Packages [3,426 B] 2024-09-13T11:18:43.402 INFO:teuthology.orchestra.run.smithi092.stdout:Hit:8 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-09-13T11:18:43.404 INFO:teuthology.orchestra.run.smithi162.stdout:Get:8 https://2.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/ubuntu/jammy/flavors/default jammy/main i386 Packages [3,426 B] 2024-09-13T11:18:43.408 INFO:teuthology.orchestra.run.smithi001.stdout:Get:7 https://2.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/ubuntu/jammy/flavors/default jammy/main amd64 Packages [18.1 kB] 2024-09-13T11:18:43.415 INFO:teuthology.orchestra.run.smithi001.stdout:Hit:8 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-09-13T11:18:43.417 INFO:teuthology.orchestra.run.smithi177.stdout:Get:5 https://2.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/ubuntu/jammy/flavors/default jammy/main amd64 Packages [18.1 kB] 2024-09-13T11:18:43.423 INFO:teuthology.orchestra.run.smithi092.stdout:Get:9 https://2.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/ubuntu/jammy/flavors/default jammy/main amd64 Packages [18.1 kB] 2024-09-13T11:18:43.424 INFO:teuthology.orchestra.run.smithi177.stdout:Hit:6 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-09-13T11:18:43.431 INFO:teuthology.orchestra.run.smithi162.stdout:Get:9 https://2.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/ubuntu/jammy/flavors/default jammy/main amd64 Packages [18.1 kB] 2024-09-13T11:18:43.436 INFO:teuthology.orchestra.run.smithi001.stdout:Get:9 https://2.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/ubuntu/jammy/flavors/default jammy/main i386 Packages [3,426 B] 2024-09-13T11:18:43.445 INFO:teuthology.orchestra.run.smithi177.stdout:Get:7 https://2.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/ubuntu/jammy/flavors/default jammy/main i386 Packages [3,426 B] 2024-09-13T11:18:43.502 INFO:teuthology.orchestra.run.smithi177.stdout:Hit:8 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-09-13T11:18:43.584 INFO:teuthology.orchestra.run.smithi177.stdout:Hit:9 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-09-13T11:18:43.617 INFO:teuthology.orchestra.run.smithi092.stdout:Fetched 29.3 kB in 0s (88.7 kB/s) 2024-09-13T11:18:43.643 INFO:teuthology.orchestra.run.smithi162.stdout:Fetched 29.3 kB in 0s (85.7 kB/s) 2024-09-13T11:18:43.652 INFO:teuthology.orchestra.run.smithi001.stdout:Fetched 29.3 kB in 0s (83.9 kB/s) 2024-09-13T11:18:43.730 INFO:teuthology.orchestra.run.smithi177.stdout:Fetched 29.3 kB in 0s (71.2 kB/s) 2024-09-13T11:18:44.628 INFO:teuthology.orchestra.run.smithi092.stdout:Reading package lists... 2024-09-13T11:18:44.644 DEBUG:teuthology.orchestra.run.smithi092:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=19.3.0-4697-gd58f7306-1jammy cephadm=19.3.0-4697-gd58f7306-1jammy ceph-mds=19.3.0-4697-gd58f7306-1jammy ceph-mgr=19.3.0-4697-gd58f7306-1jammy ceph-common=19.3.0-4697-gd58f7306-1jammy ceph-fuse=19.3.0-4697-gd58f7306-1jammy ceph-test=19.3.0-4697-gd58f7306-1jammy ceph-volume=19.3.0-4697-gd58f7306-1jammy radosgw=19.3.0-4697-gd58f7306-1jammy python3-rados=19.3.0-4697-gd58f7306-1jammy python3-rgw=19.3.0-4697-gd58f7306-1jammy python3-cephfs=19.3.0-4697-gd58f7306-1jammy python3-rbd=19.3.0-4697-gd58f7306-1jammy libcephfs2=19.3.0-4697-gd58f7306-1jammy libcephfs-dev=19.3.0-4697-gd58f7306-1jammy librados2=19.3.0-4697-gd58f7306-1jammy librbd1=19.3.0-4697-gd58f7306-1jammy rbd-fuse=19.3.0-4697-gd58f7306-1jammy 2024-09-13T11:18:44.678 INFO:teuthology.orchestra.run.smithi162.stdout:Reading package lists... 2024-09-13T11:18:44.688 INFO:teuthology.orchestra.run.smithi001.stdout:Reading package lists... 2024-09-13T11:18:44.694 DEBUG:teuthology.orchestra.run.smithi162:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=19.3.0-4697-gd58f7306-1jammy cephadm=19.3.0-4697-gd58f7306-1jammy ceph-mds=19.3.0-4697-gd58f7306-1jammy ceph-mgr=19.3.0-4697-gd58f7306-1jammy ceph-common=19.3.0-4697-gd58f7306-1jammy ceph-fuse=19.3.0-4697-gd58f7306-1jammy ceph-test=19.3.0-4697-gd58f7306-1jammy ceph-volume=19.3.0-4697-gd58f7306-1jammy radosgw=19.3.0-4697-gd58f7306-1jammy python3-rados=19.3.0-4697-gd58f7306-1jammy python3-rgw=19.3.0-4697-gd58f7306-1jammy python3-cephfs=19.3.0-4697-gd58f7306-1jammy python3-rbd=19.3.0-4697-gd58f7306-1jammy libcephfs2=19.3.0-4697-gd58f7306-1jammy libcephfs-dev=19.3.0-4697-gd58f7306-1jammy librados2=19.3.0-4697-gd58f7306-1jammy librbd1=19.3.0-4697-gd58f7306-1jammy rbd-fuse=19.3.0-4697-gd58f7306-1jammy 2024-09-13T11:18:44.705 DEBUG:teuthology.orchestra.run.smithi001:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=19.3.0-4697-gd58f7306-1jammy cephadm=19.3.0-4697-gd58f7306-1jammy ceph-mds=19.3.0-4697-gd58f7306-1jammy ceph-mgr=19.3.0-4697-gd58f7306-1jammy ceph-common=19.3.0-4697-gd58f7306-1jammy ceph-fuse=19.3.0-4697-gd58f7306-1jammy ceph-test=19.3.0-4697-gd58f7306-1jammy ceph-volume=19.3.0-4697-gd58f7306-1jammy radosgw=19.3.0-4697-gd58f7306-1jammy python3-rados=19.3.0-4697-gd58f7306-1jammy python3-rgw=19.3.0-4697-gd58f7306-1jammy python3-cephfs=19.3.0-4697-gd58f7306-1jammy python3-rbd=19.3.0-4697-gd58f7306-1jammy libcephfs2=19.3.0-4697-gd58f7306-1jammy libcephfs-dev=19.3.0-4697-gd58f7306-1jammy librados2=19.3.0-4697-gd58f7306-1jammy librbd1=19.3.0-4697-gd58f7306-1jammy rbd-fuse=19.3.0-4697-gd58f7306-1jammy 2024-09-13T11:18:44.706 INFO:teuthology.orchestra.run.smithi092.stdout:Reading package lists... 2024-09-13T11:18:44.746 INFO:teuthology.orchestra.run.smithi177.stdout:Reading package lists... 2024-09-13T11:18:44.756 INFO:teuthology.orchestra.run.smithi162.stdout:Reading package lists... 2024-09-13T11:18:44.759 INFO:teuthology.orchestra.run.smithi001.stdout:Reading package lists... 2024-09-13T11:18:44.763 DEBUG:teuthology.orchestra.run.smithi177:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=19.3.0-4697-gd58f7306-1jammy cephadm=19.3.0-4697-gd58f7306-1jammy ceph-mds=19.3.0-4697-gd58f7306-1jammy ceph-mgr=19.3.0-4697-gd58f7306-1jammy ceph-common=19.3.0-4697-gd58f7306-1jammy ceph-fuse=19.3.0-4697-gd58f7306-1jammy ceph-test=19.3.0-4697-gd58f7306-1jammy ceph-volume=19.3.0-4697-gd58f7306-1jammy radosgw=19.3.0-4697-gd58f7306-1jammy python3-rados=19.3.0-4697-gd58f7306-1jammy python3-rgw=19.3.0-4697-gd58f7306-1jammy python3-cephfs=19.3.0-4697-gd58f7306-1jammy python3-rbd=19.3.0-4697-gd58f7306-1jammy libcephfs2=19.3.0-4697-gd58f7306-1jammy libcephfs-dev=19.3.0-4697-gd58f7306-1jammy librados2=19.3.0-4697-gd58f7306-1jammy librbd1=19.3.0-4697-gd58f7306-1jammy rbd-fuse=19.3.0-4697-gd58f7306-1jammy 2024-09-13T11:18:44.825 INFO:teuthology.orchestra.run.smithi177.stdout:Reading package lists... 2024-09-13T11:18:44.914 INFO:teuthology.orchestra.run.smithi092.stdout:Building dependency tree... 2024-09-13T11:18:44.914 INFO:teuthology.orchestra.run.smithi092.stdout:Reading state information... 2024-09-13T11:18:44.957 INFO:teuthology.orchestra.run.smithi001.stdout:Building dependency tree... 2024-09-13T11:18:44.958 INFO:teuthology.orchestra.run.smithi001.stdout:Reading state information... 2024-09-13T11:18:44.959 INFO:teuthology.orchestra.run.smithi162.stdout:Building dependency tree... 2024-09-13T11:18:44.959 INFO:teuthology.orchestra.run.smithi162.stdout:Reading state information... 2024-09-13T11:18:45.045 INFO:teuthology.orchestra.run.smithi177.stdout:Building dependency tree... 2024-09-13T11:18:45.046 INFO:teuthology.orchestra.run.smithi177.stdout:Reading state information... 2024-09-13T11:18:45.108 INFO:teuthology.orchestra.run.smithi092.stdout:The following packages were automatically installed and are no longer required: 2024-09-13T11:18:45.109 INFO:teuthology.orchestra.run.smithi092.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-09-13T11:18:45.109 INFO:teuthology.orchestra.run.smithi092.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-09-13T11:18:45.109 INFO:teuthology.orchestra.run.smithi092.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-09-13T11:18:45.110 INFO:teuthology.orchestra.run.smithi092.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-09-13T11:18:45.110 INFO:teuthology.orchestra.run.smithi092.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-09-13T11:18:45.110 INFO:teuthology.orchestra.run.smithi092.stdout:Use 'sudo apt autoremove' to remove them. 2024-09-13T11:18:45.110 INFO:teuthology.orchestra.run.smithi092.stdout:The following additional packages will be installed: 2024-09-13T11:18:45.110 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2024-09-13T11:18:45.110 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2024-09-13T11:18:45.111 INFO:teuthology.orchestra.run.smithi092.stdout: libdouble-conversion3 libjq1 liblttng-ust1 liblua5.3-dev liboath0 libonig5 2024-09-13T11:18:45.111 INFO:teuthology.orchestra.run.smithi092.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 libradosstriper1 2024-09-13T11:18:45.112 INFO:teuthology.orchestra.run.smithi092.stdout: librdkafka1 libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 2024-09-13T11:18:45.112 INFO:teuthology.orchestra.run.smithi092.stdout: lua-any lua-sec lua-socket lua5.1 luarocks nvme-cli pkg-config 2024-09-13T11:18:45.112 INFO:teuthology.orchestra.run.smithi092.stdout: python-asyncssh-doc python-babel-localedata python-pastedeploy-tpl 2024-09-13T11:18:45.112 INFO:teuthology.orchestra.run.smithi092.stdout: python3-asyncssh python3-babel python3-bcrypt python3-cachetools 2024-09-13T11:18:45.112 INFO:teuthology.orchestra.run.smithi092.stdout: python3-ceph-argparse python3-ceph-common python3-certifi python3-cheroot 2024-09-13T11:18:45.112 INFO:teuthology.orchestra.run.smithi092.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-09-13T11:18:45.112 INFO:teuthology.orchestra.run.smithi092.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-09-13T11:18:45.112 INFO:teuthology.orchestra.run.smithi092.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-09-13T11:18:45.112 INFO:teuthology.orchestra.run.smithi092.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-09-13T11:18:45.112 INFO:teuthology.orchestra.run.smithi092.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-09-13T11:18:45.112 INFO:teuthology.orchestra.run.smithi092.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-09-13T11:18:45.112 INFO:teuthology.orchestra.run.smithi092.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-09-13T11:18:45.112 INFO:teuthology.orchestra.run.smithi092.stdout: python3-pygments python3-pyinotify python3-pytest python3-repoze.lru 2024-09-13T11:18:45.112 INFO:teuthology.orchestra.run.smithi092.stdout: python3-requests python3-requests-oauthlib python3-routes python3-rsa 2024-09-13T11:18:45.113 INFO:teuthology.orchestra.run.smithi092.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2024-09-13T11:18:45.113 INFO:teuthology.orchestra.run.smithi092.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2024-09-13T11:18:45.113 INFO:teuthology.orchestra.run.smithi092.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2024-09-13T11:18:45.113 INFO:teuthology.orchestra.run.smithi092.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2024-09-13T11:18:45.113 INFO:teuthology.orchestra.run.smithi092.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n socat unzip 2024-09-13T11:18:45.113 INFO:teuthology.orchestra.run.smithi092.stdout: xmlstarlet zip 2024-09-13T11:18:45.113 INFO:teuthology.orchestra.run.smithi092.stdout:Suggested packages: 2024-09-13T11:18:45.113 INFO:teuthology.orchestra.run.smithi092.stdout: python3-influxdb readline-doc python-jinja2-doc python3-beaker 2024-09-13T11:18:45.113 INFO:teuthology.orchestra.run.smithi092.stdout: python-mako-doc python-natsort-doc python-openssl-doc python3-openssl-dbg 2024-09-13T11:18:45.114 INFO:teuthology.orchestra.run.smithi092.stdout: httpd-wsgi libapache2-mod-python libapache2-mod-scgi libjs-mochikit 2024-09-13T11:18:45.114 INFO:teuthology.orchestra.run.smithi092.stdout: python-pecan-doc python-psutil-doc subversion python-pygments-doc 2024-09-13T11:18:45.114 INFO:teuthology.orchestra.run.smithi092.stdout: python-pyinotify-doc python3-socks python-requests-doc python3-dap 2024-09-13T11:18:45.114 INFO:teuthology.orchestra.run.smithi092.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2024-09-13T11:18:45.114 INFO:teuthology.orchestra.run.smithi092.stdout: python-webtest-doc python-werkzeug-doc python3-watchdog 2024-09-13T11:18:45.114 INFO:teuthology.orchestra.run.smithi092.stdout:Recommended packages: 2024-09-13T11:18:45.114 INFO:teuthology.orchestra.run.smithi092.stdout: btrfs-tools 2024-09-13T11:18:45.158 INFO:teuthology.orchestra.run.smithi162.stdout:The following packages were automatically installed and are no longer required: 2024-09-13T11:18:45.158 INFO:teuthology.orchestra.run.smithi162.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-09-13T11:18:45.158 INFO:teuthology.orchestra.run.smithi162.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-09-13T11:18:45.158 INFO:teuthology.orchestra.run.smithi162.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-09-13T11:18:45.159 INFO:teuthology.orchestra.run.smithi162.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-09-13T11:18:45.159 INFO:teuthology.orchestra.run.smithi162.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-09-13T11:18:45.159 INFO:teuthology.orchestra.run.smithi162.stdout:Use 'sudo apt autoremove' to remove them. 2024-09-13T11:18:45.159 INFO:teuthology.orchestra.run.smithi162.stdout:The following additional packages will be installed: 2024-09-13T11:18:45.160 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2024-09-13T11:18:45.160 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2024-09-13T11:18:45.160 INFO:teuthology.orchestra.run.smithi162.stdout: libdouble-conversion3 libjq1 liblttng-ust1 liblua5.3-dev liboath0 libonig5 2024-09-13T11:18:45.160 INFO:teuthology.orchestra.run.smithi162.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 libradosstriper1 2024-09-13T11:18:45.161 INFO:teuthology.orchestra.run.smithi001.stdout:The following packages were automatically installed and are no longer required: 2024-09-13T11:18:45.161 INFO:teuthology.orchestra.run.smithi001.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-09-13T11:18:45.162 INFO:teuthology.orchestra.run.smithi001.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-09-13T11:18:45.162 INFO:teuthology.orchestra.run.smithi001.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-09-13T11:18:45.162 INFO:teuthology.orchestra.run.smithi001.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-09-13T11:18:45.162 INFO:teuthology.orchestra.run.smithi001.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-09-13T11:18:45.162 INFO:teuthology.orchestra.run.smithi001.stdout:Use 'sudo apt autoremove' to remove them. 2024-09-13T11:18:45.162 INFO:teuthology.orchestra.run.smithi162.stdout: librdkafka1 libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 2024-09-13T11:18:45.162 INFO:teuthology.orchestra.run.smithi162.stdout: lua-any lua-sec lua-socket lua5.1 luarocks nvme-cli pkg-config 2024-09-13T11:18:45.163 INFO:teuthology.orchestra.run.smithi162.stdout: python-asyncssh-doc python-babel-localedata python-pastedeploy-tpl 2024-09-13T11:18:45.163 INFO:teuthology.orchestra.run.smithi162.stdout: python3-asyncssh python3-babel python3-bcrypt python3-cachetools 2024-09-13T11:18:45.163 INFO:teuthology.orchestra.run.smithi162.stdout: python3-ceph-argparse python3-ceph-common python3-certifi python3-cheroot 2024-09-13T11:18:45.163 INFO:teuthology.orchestra.run.smithi162.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-09-13T11:18:45.163 INFO:teuthology.orchestra.run.smithi162.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-09-13T11:18:45.163 INFO:teuthology.orchestra.run.smithi162.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-09-13T11:18:45.163 INFO:teuthology.orchestra.run.smithi162.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-09-13T11:18:45.163 INFO:teuthology.orchestra.run.smithi162.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-09-13T11:18:45.163 INFO:teuthology.orchestra.run.smithi162.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-09-13T11:18:45.163 INFO:teuthology.orchestra.run.smithi162.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-09-13T11:18:45.163 INFO:teuthology.orchestra.run.smithi162.stdout: python3-pygments python3-pyinotify python3-pytest python3-repoze.lru 2024-09-13T11:18:45.163 INFO:teuthology.orchestra.run.smithi162.stdout: python3-requests python3-requests-oauthlib python3-routes python3-rsa 2024-09-13T11:18:45.163 INFO:teuthology.orchestra.run.smithi162.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2024-09-13T11:18:45.164 INFO:teuthology.orchestra.run.smithi162.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2024-09-13T11:18:45.164 INFO:teuthology.orchestra.run.smithi162.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2024-09-13T11:18:45.164 INFO:teuthology.orchestra.run.smithi162.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2024-09-13T11:18:45.164 INFO:teuthology.orchestra.run.smithi162.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n socat unzip 2024-09-13T11:18:45.164 INFO:teuthology.orchestra.run.smithi162.stdout: xmlstarlet zip 2024-09-13T11:18:45.164 INFO:teuthology.orchestra.run.smithi162.stdout:Suggested packages: 2024-09-13T11:18:45.164 INFO:teuthology.orchestra.run.smithi162.stdout: python3-influxdb readline-doc python-jinja2-doc python3-beaker 2024-09-13T11:18:45.164 INFO:teuthology.orchestra.run.smithi162.stdout: python-mako-doc python-natsort-doc python-openssl-doc python3-openssl-dbg 2024-09-13T11:18:45.164 INFO:teuthology.orchestra.run.smithi162.stdout: httpd-wsgi libapache2-mod-python libapache2-mod-scgi libjs-mochikit 2024-09-13T11:18:45.164 INFO:teuthology.orchestra.run.smithi162.stdout: python-pecan-doc python-psutil-doc subversion python-pygments-doc 2024-09-13T11:18:45.164 INFO:teuthology.orchestra.run.smithi162.stdout: python-pyinotify-doc python3-socks python-requests-doc python3-dap 2024-09-13T11:18:45.164 INFO:teuthology.orchestra.run.smithi162.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2024-09-13T11:18:45.164 INFO:teuthology.orchestra.run.smithi162.stdout: python-webtest-doc python-werkzeug-doc python3-watchdog 2024-09-13T11:18:45.164 INFO:teuthology.orchestra.run.smithi162.stdout:Recommended packages: 2024-09-13T11:18:45.164 INFO:teuthology.orchestra.run.smithi162.stdout: btrfs-tools 2024-09-13T11:18:45.166 INFO:teuthology.orchestra.run.smithi001.stdout:The following additional packages will be installed: 2024-09-13T11:18:45.166 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2024-09-13T11:18:45.166 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2024-09-13T11:18:45.167 INFO:teuthology.orchestra.run.smithi001.stdout: libdouble-conversion3 libjq1 liblttng-ust1 liblua5.3-dev liboath0 libonig5 2024-09-13T11:18:45.167 INFO:teuthology.orchestra.run.smithi001.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 libradosstriper1 2024-09-13T11:18:45.167 INFO:teuthology.orchestra.run.smithi001.stdout: librdkafka1 libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 2024-09-13T11:18:45.167 INFO:teuthology.orchestra.run.smithi001.stdout: lua-any lua-sec lua-socket lua5.1 luarocks nvme-cli pkg-config 2024-09-13T11:18:45.167 INFO:teuthology.orchestra.run.smithi001.stdout: python-asyncssh-doc python-babel-localedata python-pastedeploy-tpl 2024-09-13T11:18:45.167 INFO:teuthology.orchestra.run.smithi001.stdout: python3-asyncssh python3-babel python3-bcrypt python3-cachetools 2024-09-13T11:18:45.167 INFO:teuthology.orchestra.run.smithi001.stdout: python3-ceph-argparse python3-ceph-common python3-certifi python3-cheroot 2024-09-13T11:18:45.167 INFO:teuthology.orchestra.run.smithi001.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-09-13T11:18:45.167 INFO:teuthology.orchestra.run.smithi001.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-09-13T11:18:45.167 INFO:teuthology.orchestra.run.smithi001.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-09-13T11:18:45.167 INFO:teuthology.orchestra.run.smithi001.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-09-13T11:18:45.167 INFO:teuthology.orchestra.run.smithi001.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-09-13T11:18:45.167 INFO:teuthology.orchestra.run.smithi001.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-09-13T11:18:45.167 INFO:teuthology.orchestra.run.smithi001.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-09-13T11:18:45.168 INFO:teuthology.orchestra.run.smithi001.stdout: python3-pygments python3-pyinotify python3-pytest python3-repoze.lru 2024-09-13T11:18:45.168 INFO:teuthology.orchestra.run.smithi001.stdout: python3-requests python3-requests-oauthlib python3-routes python3-rsa 2024-09-13T11:18:45.168 INFO:teuthology.orchestra.run.smithi001.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2024-09-13T11:18:45.168 INFO:teuthology.orchestra.run.smithi001.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2024-09-13T11:18:45.168 INFO:teuthology.orchestra.run.smithi001.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2024-09-13T11:18:45.168 INFO:teuthology.orchestra.run.smithi001.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2024-09-13T11:18:45.168 INFO:teuthology.orchestra.run.smithi001.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n socat unzip 2024-09-13T11:18:45.168 INFO:teuthology.orchestra.run.smithi001.stdout: xmlstarlet zip 2024-09-13T11:18:45.168 INFO:teuthology.orchestra.run.smithi001.stdout:Suggested packages: 2024-09-13T11:18:45.168 INFO:teuthology.orchestra.run.smithi001.stdout: python3-influxdb readline-doc python-jinja2-doc python3-beaker 2024-09-13T11:18:45.168 INFO:teuthology.orchestra.run.smithi001.stdout: python-mako-doc python-natsort-doc python-openssl-doc python3-openssl-dbg 2024-09-13T11:18:45.168 INFO:teuthology.orchestra.run.smithi001.stdout: httpd-wsgi libapache2-mod-python libapache2-mod-scgi libjs-mochikit 2024-09-13T11:18:45.168 INFO:teuthology.orchestra.run.smithi001.stdout: python-pecan-doc python-psutil-doc subversion python-pygments-doc 2024-09-13T11:18:45.168 INFO:teuthology.orchestra.run.smithi001.stdout: python-pyinotify-doc python3-socks python-requests-doc python3-dap 2024-09-13T11:18:45.168 INFO:teuthology.orchestra.run.smithi001.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2024-09-13T11:18:45.169 INFO:teuthology.orchestra.run.smithi001.stdout: python-webtest-doc python-werkzeug-doc python3-watchdog 2024-09-13T11:18:45.169 INFO:teuthology.orchestra.run.smithi001.stdout:Recommended packages: 2024-09-13T11:18:45.169 INFO:teuthology.orchestra.run.smithi001.stdout: btrfs-tools 2024-09-13T11:18:45.216 INFO:teuthology.orchestra.run.smithi092.stdout:The following NEW packages will be installed: 2024-09-13T11:18:45.216 INFO:teuthology.orchestra.run.smithi092.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2024-09-13T11:18:45.216 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2024-09-13T11:18:45.216 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm jq 2024-09-13T11:18:45.216 INFO:teuthology.orchestra.run.smithi092.stdout: libcephfs-dev libcephfs2 libdouble-conversion3 libjq1 liblttng-ust1 2024-09-13T11:18:45.216 INFO:teuthology.orchestra.run.smithi092.stdout: liblua5.3-dev liboath0 libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 2024-09-13T11:18:45.217 INFO:teuthology.orchestra.run.smithi092.stdout: libqt5network5 librados2 libradosstriper1 librbd1 librdkafka1 2024-09-13T11:18:45.218 INFO:teuthology.orchestra.run.smithi092.stdout: libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 lua-any lua-sec 2024-09-13T11:18:45.218 INFO:teuthology.orchestra.run.smithi092.stdout: lua-socket lua5.1 luarocks nvme-cli pkg-config python-asyncssh-doc 2024-09-13T11:18:45.218 INFO:teuthology.orchestra.run.smithi092.stdout: python-babel-localedata python-pastedeploy-tpl python3-asyncssh 2024-09-13T11:18:45.218 INFO:teuthology.orchestra.run.smithi092.stdout: python3-babel python3-bcrypt python3-cachetools python3-ceph-argparse 2024-09-13T11:18:45.218 INFO:teuthology.orchestra.run.smithi092.stdout: python3-ceph-common python3-cephfs python3-certifi python3-cheroot 2024-09-13T11:18:45.218 INFO:teuthology.orchestra.run.smithi092.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-09-13T11:18:45.218 INFO:teuthology.orchestra.run.smithi092.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-09-13T11:18:45.218 INFO:teuthology.orchestra.run.smithi092.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-09-13T11:18:45.218 INFO:teuthology.orchestra.run.smithi092.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-09-13T11:18:45.218 INFO:teuthology.orchestra.run.smithi092.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-09-13T11:18:45.218 INFO:teuthology.orchestra.run.smithi092.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-09-13T11:18:45.219 INFO:teuthology.orchestra.run.smithi092.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-09-13T11:18:45.219 INFO:teuthology.orchestra.run.smithi092.stdout: python3-pygments python3-pyinotify python3-pytest python3-rados python3-rbd 2024-09-13T11:18:45.219 INFO:teuthology.orchestra.run.smithi092.stdout: python3-repoze.lru python3-requests python3-requests-oauthlib python3-rgw 2024-09-13T11:18:45.219 INFO:teuthology.orchestra.run.smithi092.stdout: python3-routes python3-rsa python3-simplegeneric python3-simplejson 2024-09-13T11:18:45.219 INFO:teuthology.orchestra.run.smithi092.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib python3-tempita 2024-09-13T11:18:45.219 INFO:teuthology.orchestra.run.smithi092.stdout: python3-tempora python3-threadpoolctl python3-toml python3-urllib3 2024-09-13T11:18:45.219 INFO:teuthology.orchestra.run.smithi092.stdout: python3-waitress python3-wcwidth python3-webob python3-websocket 2024-09-13T11:18:45.219 INFO:teuthology.orchestra.run.smithi092.stdout: python3-webtest python3-werkzeug python3-zc.lockfile qttranslations5-l10n 2024-09-13T11:18:45.219 INFO:teuthology.orchestra.run.smithi092.stdout: radosgw rbd-fuse socat unzip xmlstarlet zip 2024-09-13T11:18:45.251 INFO:teuthology.orchestra.run.smithi177.stdout:The following packages were automatically installed and are no longer required: 2024-09-13T11:18:45.251 INFO:teuthology.orchestra.run.smithi177.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-09-13T11:18:45.251 INFO:teuthology.orchestra.run.smithi177.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-09-13T11:18:45.251 INFO:teuthology.orchestra.run.smithi177.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-09-13T11:18:45.252 INFO:teuthology.orchestra.run.smithi177.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-09-13T11:18:45.252 INFO:teuthology.orchestra.run.smithi177.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-09-13T11:18:45.252 INFO:teuthology.orchestra.run.smithi177.stdout:Use 'sudo apt autoremove' to remove them. 2024-09-13T11:18:45.253 INFO:teuthology.orchestra.run.smithi177.stdout:The following additional packages will be installed: 2024-09-13T11:18:45.253 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2024-09-13T11:18:45.253 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2024-09-13T11:18:45.253 INFO:teuthology.orchestra.run.smithi177.stdout: libdouble-conversion3 libjq1 liblttng-ust1 liblua5.3-dev liboath0 libonig5 2024-09-13T11:18:45.253 INFO:teuthology.orchestra.run.smithi177.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 libradosstriper1 2024-09-13T11:18:45.254 INFO:teuthology.orchestra.run.smithi177.stdout: librdkafka1 libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 2024-09-13T11:18:45.255 INFO:teuthology.orchestra.run.smithi177.stdout: lua-any lua-sec lua-socket lua5.1 luarocks nvme-cli pkg-config 2024-09-13T11:18:45.255 INFO:teuthology.orchestra.run.smithi177.stdout: python-asyncssh-doc python-babel-localedata python-pastedeploy-tpl 2024-09-13T11:18:45.255 INFO:teuthology.orchestra.run.smithi177.stdout: python3-asyncssh python3-babel python3-bcrypt python3-cachetools 2024-09-13T11:18:45.255 INFO:teuthology.orchestra.run.smithi177.stdout: python3-ceph-argparse python3-ceph-common python3-certifi python3-cheroot 2024-09-13T11:18:45.255 INFO:teuthology.orchestra.run.smithi177.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-09-13T11:18:45.255 INFO:teuthology.orchestra.run.smithi177.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-09-13T11:18:45.255 INFO:teuthology.orchestra.run.smithi177.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-09-13T11:18:45.255 INFO:teuthology.orchestra.run.smithi177.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-09-13T11:18:45.255 INFO:teuthology.orchestra.run.smithi177.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-09-13T11:18:45.255 INFO:teuthology.orchestra.run.smithi177.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-09-13T11:18:45.255 INFO:teuthology.orchestra.run.smithi177.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-09-13T11:18:45.255 INFO:teuthology.orchestra.run.smithi177.stdout: python3-pygments python3-pyinotify python3-pytest python3-repoze.lru 2024-09-13T11:18:45.255 INFO:teuthology.orchestra.run.smithi177.stdout: python3-requests python3-requests-oauthlib python3-routes python3-rsa 2024-09-13T11:18:45.255 INFO:teuthology.orchestra.run.smithi177.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2024-09-13T11:18:45.256 INFO:teuthology.orchestra.run.smithi177.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2024-09-13T11:18:45.256 INFO:teuthology.orchestra.run.smithi177.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2024-09-13T11:18:45.256 INFO:teuthology.orchestra.run.smithi177.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2024-09-13T11:18:45.256 INFO:teuthology.orchestra.run.smithi177.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n socat unzip 2024-09-13T11:18:45.256 INFO:teuthology.orchestra.run.smithi177.stdout: xmlstarlet zip 2024-09-13T11:18:45.256 INFO:teuthology.orchestra.run.smithi177.stdout:Suggested packages: 2024-09-13T11:18:45.256 INFO:teuthology.orchestra.run.smithi177.stdout: python3-influxdb readline-doc python-jinja2-doc python3-beaker 2024-09-13T11:18:45.256 INFO:teuthology.orchestra.run.smithi177.stdout: python-mako-doc python-natsort-doc python-openssl-doc python3-openssl-dbg 2024-09-13T11:18:45.256 INFO:teuthology.orchestra.run.smithi177.stdout: httpd-wsgi libapache2-mod-python libapache2-mod-scgi libjs-mochikit 2024-09-13T11:18:45.256 INFO:teuthology.orchestra.run.smithi177.stdout: python-pecan-doc python-psutil-doc subversion python-pygments-doc 2024-09-13T11:18:45.257 INFO:teuthology.orchestra.run.smithi177.stdout: python-pyinotify-doc python3-socks python-requests-doc python3-dap 2024-09-13T11:18:45.257 INFO:teuthology.orchestra.run.smithi177.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2024-09-13T11:18:45.257 INFO:teuthology.orchestra.run.smithi177.stdout: python-webtest-doc python-werkzeug-doc python3-watchdog 2024-09-13T11:18:45.257 INFO:teuthology.orchestra.run.smithi177.stdout:Recommended packages: 2024-09-13T11:18:45.257 INFO:teuthology.orchestra.run.smithi177.stdout: btrfs-tools 2024-09-13T11:18:45.268 INFO:teuthology.orchestra.run.smithi162.stdout:The following NEW packages will be installed: 2024-09-13T11:18:45.268 INFO:teuthology.orchestra.run.smithi162.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2024-09-13T11:18:45.268 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2024-09-13T11:18:45.268 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm jq 2024-09-13T11:18:45.269 INFO:teuthology.orchestra.run.smithi162.stdout: libcephfs-dev libcephfs2 libdouble-conversion3 libjq1 liblttng-ust1 2024-09-13T11:18:45.269 INFO:teuthology.orchestra.run.smithi162.stdout: liblua5.3-dev liboath0 libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 2024-09-13T11:18:45.269 INFO:teuthology.orchestra.run.smithi162.stdout: libqt5network5 librados2 libradosstriper1 librbd1 librdkafka1 2024-09-13T11:18:45.269 INFO:teuthology.orchestra.run.smithi162.stdout: libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 lua-any lua-sec 2024-09-13T11:18:45.269 INFO:teuthology.orchestra.run.smithi162.stdout: lua-socket lua5.1 luarocks nvme-cli pkg-config python-asyncssh-doc 2024-09-13T11:18:45.269 INFO:teuthology.orchestra.run.smithi162.stdout: python-babel-localedata python-pastedeploy-tpl python3-asyncssh 2024-09-13T11:18:45.269 INFO:teuthology.orchestra.run.smithi162.stdout: python3-babel python3-bcrypt python3-cachetools python3-ceph-argparse 2024-09-13T11:18:45.269 INFO:teuthology.orchestra.run.smithi162.stdout: python3-ceph-common python3-cephfs python3-certifi python3-cheroot 2024-09-13T11:18:45.269 INFO:teuthology.orchestra.run.smithi162.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-09-13T11:18:45.269 INFO:teuthology.orchestra.run.smithi162.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-09-13T11:18:45.269 INFO:teuthology.orchestra.run.smithi162.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-09-13T11:18:45.269 INFO:teuthology.orchestra.run.smithi162.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-09-13T11:18:45.269 INFO:teuthology.orchestra.run.smithi162.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-09-13T11:18:45.270 INFO:teuthology.orchestra.run.smithi162.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-09-13T11:18:45.270 INFO:teuthology.orchestra.run.smithi162.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-09-13T11:18:45.270 INFO:teuthology.orchestra.run.smithi162.stdout: python3-pygments python3-pyinotify python3-pytest python3-rados python3-rbd 2024-09-13T11:18:45.270 INFO:teuthology.orchestra.run.smithi162.stdout: python3-repoze.lru python3-requests python3-requests-oauthlib python3-rgw 2024-09-13T11:18:45.270 INFO:teuthology.orchestra.run.smithi162.stdout: python3-routes python3-rsa python3-simplegeneric python3-simplejson 2024-09-13T11:18:45.270 INFO:teuthology.orchestra.run.smithi162.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib python3-tempita 2024-09-13T11:18:45.270 INFO:teuthology.orchestra.run.smithi162.stdout: python3-tempora python3-threadpoolctl python3-toml python3-urllib3 2024-09-13T11:18:45.270 INFO:teuthology.orchestra.run.smithi162.stdout: python3-waitress python3-wcwidth python3-webob python3-websocket 2024-09-13T11:18:45.270 INFO:teuthology.orchestra.run.smithi162.stdout: python3-webtest python3-werkzeug python3-zc.lockfile qttranslations5-l10n 2024-09-13T11:18:45.270 INFO:teuthology.orchestra.run.smithi162.stdout: radosgw rbd-fuse socat unzip xmlstarlet zip 2024-09-13T11:18:45.272 INFO:teuthology.orchestra.run.smithi001.stdout:The following NEW packages will be installed: 2024-09-13T11:18:45.272 INFO:teuthology.orchestra.run.smithi001.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2024-09-13T11:18:45.272 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2024-09-13T11:18:45.272 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm jq 2024-09-13T11:18:45.274 INFO:teuthology.orchestra.run.smithi001.stdout: libcephfs-dev libcephfs2 libdouble-conversion3 libjq1 liblttng-ust1 2024-09-13T11:18:45.274 INFO:teuthology.orchestra.run.smithi001.stdout: liblua5.3-dev liboath0 libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 2024-09-13T11:18:45.274 INFO:teuthology.orchestra.run.smithi001.stdout: libqt5network5 librados2 libradosstriper1 librbd1 librdkafka1 2024-09-13T11:18:45.274 INFO:teuthology.orchestra.run.smithi001.stdout: libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 lua-any lua-sec 2024-09-13T11:18:45.274 INFO:teuthology.orchestra.run.smithi001.stdout: lua-socket lua5.1 luarocks nvme-cli pkg-config python-asyncssh-doc 2024-09-13T11:18:45.275 INFO:teuthology.orchestra.run.smithi001.stdout: python-babel-localedata python-pastedeploy-tpl python3-asyncssh 2024-09-13T11:18:45.275 INFO:teuthology.orchestra.run.smithi001.stdout: python3-babel python3-bcrypt python3-cachetools python3-ceph-argparse 2024-09-13T11:18:45.275 INFO:teuthology.orchestra.run.smithi001.stdout: python3-ceph-common python3-cephfs python3-certifi python3-cheroot 2024-09-13T11:18:45.275 INFO:teuthology.orchestra.run.smithi001.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-09-13T11:18:45.275 INFO:teuthology.orchestra.run.smithi001.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-09-13T11:18:45.275 INFO:teuthology.orchestra.run.smithi001.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-09-13T11:18:45.275 INFO:teuthology.orchestra.run.smithi001.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-09-13T11:18:45.275 INFO:teuthology.orchestra.run.smithi001.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-09-13T11:18:45.275 INFO:teuthology.orchestra.run.smithi001.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-09-13T11:18:45.275 INFO:teuthology.orchestra.run.smithi001.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-09-13T11:18:45.275 INFO:teuthology.orchestra.run.smithi001.stdout: python3-pygments python3-pyinotify python3-pytest python3-rados python3-rbd 2024-09-13T11:18:45.275 INFO:teuthology.orchestra.run.smithi001.stdout: python3-repoze.lru python3-requests python3-requests-oauthlib python3-rgw 2024-09-13T11:18:45.275 INFO:teuthology.orchestra.run.smithi001.stdout: python3-routes python3-rsa python3-simplegeneric python3-simplejson 2024-09-13T11:18:45.275 INFO:teuthology.orchestra.run.smithi001.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib python3-tempita 2024-09-13T11:18:45.276 INFO:teuthology.orchestra.run.smithi001.stdout: python3-tempora python3-threadpoolctl python3-toml python3-urllib3 2024-09-13T11:18:45.276 INFO:teuthology.orchestra.run.smithi001.stdout: python3-waitress python3-wcwidth python3-webob python3-websocket 2024-09-13T11:18:45.276 INFO:teuthology.orchestra.run.smithi001.stdout: python3-webtest python3-werkzeug python3-zc.lockfile qttranslations5-l10n 2024-09-13T11:18:45.276 INFO:teuthology.orchestra.run.smithi001.stdout: radosgw rbd-fuse socat unzip xmlstarlet zip 2024-09-13T11:18:45.278 INFO:teuthology.orchestra.run.smithi092.stdout:0 upgraded, 118 newly installed, 0 to remove and 331 not upgraded. 2024-09-13T11:18:45.278 INFO:teuthology.orchestra.run.smithi092.stdout:Need to get 251 MB of archives. 2024-09-13T11:18:45.278 INFO:teuthology.orchestra.run.smithi092.stdout:After this operation, 933 MB of additional disk space will be used. 2024-09-13T11:18:45.278 INFO:teuthology.orchestra.run.smithi092.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2024-09-13T11:18:45.330 INFO:teuthology.orchestra.run.smithi162.stdout:0 upgraded, 118 newly installed, 0 to remove and 331 not upgraded. 2024-09-13T11:18:45.330 INFO:teuthology.orchestra.run.smithi162.stdout:Need to get 251 MB of archives. 2024-09-13T11:18:45.330 INFO:teuthology.orchestra.run.smithi162.stdout:After this operation, 933 MB of additional disk space will be used. 2024-09-13T11:18:45.330 INFO:teuthology.orchestra.run.smithi162.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2024-09-13T11:18:45.360 INFO:teuthology.orchestra.run.smithi177.stdout:The following NEW packages will be installed: 2024-09-13T11:18:45.360 INFO:teuthology.orchestra.run.smithi177.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2024-09-13T11:18:45.360 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2024-09-13T11:18:45.360 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm jq 2024-09-13T11:18:45.361 INFO:teuthology.orchestra.run.smithi092.stdout:Get:2 https://2.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 19.3.0-4697-gd58f7306-1jammy [3,631 kB] 2024-09-13T11:18:45.362 INFO:teuthology.orchestra.run.smithi177.stdout: libcephfs-dev libcephfs2 libdouble-conversion3 libjq1 liblttng-ust1 2024-09-13T11:18:45.362 INFO:teuthology.orchestra.run.smithi177.stdout: liblua5.3-dev liboath0 libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 2024-09-13T11:18:45.362 INFO:teuthology.orchestra.run.smithi177.stdout: libqt5network5 librados2 libradosstriper1 librbd1 librdkafka1 2024-09-13T11:18:45.362 INFO:teuthology.orchestra.run.smithi177.stdout: libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 lua-any lua-sec 2024-09-13T11:18:45.362 INFO:teuthology.orchestra.run.smithi177.stdout: lua-socket lua5.1 luarocks nvme-cli pkg-config python-asyncssh-doc 2024-09-13T11:18:45.362 INFO:teuthology.orchestra.run.smithi177.stdout: python-babel-localedata python-pastedeploy-tpl python3-asyncssh 2024-09-13T11:18:45.362 INFO:teuthology.orchestra.run.smithi177.stdout: python3-babel python3-bcrypt python3-cachetools python3-ceph-argparse 2024-09-13T11:18:45.362 INFO:teuthology.orchestra.run.smithi177.stdout: python3-ceph-common python3-cephfs python3-certifi python3-cheroot 2024-09-13T11:18:45.362 INFO:teuthology.orchestra.run.smithi177.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-09-13T11:18:45.362 INFO:teuthology.orchestra.run.smithi177.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-09-13T11:18:45.362 INFO:teuthology.orchestra.run.smithi177.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-09-13T11:18:45.362 INFO:teuthology.orchestra.run.smithi177.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-09-13T11:18:45.363 INFO:teuthology.orchestra.run.smithi177.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-09-13T11:18:45.363 INFO:teuthology.orchestra.run.smithi177.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-09-13T11:18:45.363 INFO:teuthology.orchestra.run.smithi177.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-09-13T11:18:45.363 INFO:teuthology.orchestra.run.smithi177.stdout: python3-pygments python3-pyinotify python3-pytest python3-rados python3-rbd 2024-09-13T11:18:45.363 INFO:teuthology.orchestra.run.smithi177.stdout: python3-repoze.lru python3-requests python3-requests-oauthlib python3-rgw 2024-09-13T11:18:45.363 INFO:teuthology.orchestra.run.smithi177.stdout: python3-routes python3-rsa python3-simplegeneric python3-simplejson 2024-09-13T11:18:45.363 INFO:teuthology.orchestra.run.smithi177.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib python3-tempita 2024-09-13T11:18:45.363 INFO:teuthology.orchestra.run.smithi177.stdout: python3-tempora python3-threadpoolctl python3-toml python3-urllib3 2024-09-13T11:18:45.363 INFO:teuthology.orchestra.run.smithi177.stdout: python3-waitress python3-wcwidth python3-webob python3-websocket 2024-09-13T11:18:45.363 INFO:teuthology.orchestra.run.smithi177.stdout: python3-webtest python3-werkzeug python3-zc.lockfile qttranslations5-l10n 2024-09-13T11:18:45.363 INFO:teuthology.orchestra.run.smithi177.stdout: radosgw rbd-fuse socat unzip xmlstarlet zip 2024-09-13T11:18:45.367 INFO:teuthology.orchestra.run.smithi092.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2024-09-13T11:18:45.370 INFO:teuthology.orchestra.run.smithi092.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2024-09-13T11:18:45.387 INFO:teuthology.orchestra.run.smithi092.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2,006 kB] 2024-09-13T11:18:45.414 INFO:teuthology.orchestra.run.smithi162.stdout:Get:2 https://2.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 19.3.0-4697-gd58f7306-1jammy [3,631 kB] 2024-09-13T11:18:45.417 INFO:teuthology.orchestra.run.smithi162.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2024-09-13T11:18:45.420 INFO:teuthology.orchestra.run.smithi001.stdout:0 upgraded, 118 newly installed, 0 to remove and 331 not upgraded. 2024-09-13T11:18:45.420 INFO:teuthology.orchestra.run.smithi001.stdout:Need to get 251 MB of archives. 2024-09-13T11:18:45.420 INFO:teuthology.orchestra.run.smithi001.stdout:After this operation, 933 MB of additional disk space will be used. 2024-09-13T11:18:45.420 INFO:teuthology.orchestra.run.smithi001.stdout:Get:1 https://2.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 19.3.0-4697-gd58f7306-1jammy [3,631 kB] 2024-09-13T11:18:45.420 INFO:teuthology.orchestra.run.smithi162.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2024-09-13T11:18:45.437 INFO:teuthology.orchestra.run.smithi162.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2,006 kB] 2024-09-13T11:18:45.446 INFO:teuthology.orchestra.run.smithi092.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2024-09-13T11:18:45.451 INFO:teuthology.orchestra.run.smithi092.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2024-09-13T11:18:45.463 INFO:teuthology.orchestra.run.smithi092.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2024-09-13T11:18:45.464 INFO:teuthology.orchestra.run.smithi001.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2024-09-13T11:18:45.467 INFO:teuthology.orchestra.run.smithi092.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2024-09-13T11:18:45.468 INFO:teuthology.orchestra.run.smithi092.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2024-09-13T11:18:45.468 INFO:teuthology.orchestra.run.smithi092.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2024-09-13T11:18:45.477 INFO:teuthology.orchestra.run.smithi092.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy/main amd64 libreadline-dev amd64 8.1.2-1 [166 kB] 2024-09-13T11:18:45.479 INFO:teuthology.orchestra.run.smithi092.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblua5.3-dev amd64 5.3.6-1build1 [167 kB] 2024-09-13T11:18:45.481 INFO:teuthology.orchestra.run.smithi092.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua5.1 amd64 5.1.5-8.1build4 [94.6 kB] 2024-09-13T11:18:45.483 INFO:teuthology.orchestra.run.smithi092.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-any all 27ubuntu1 [5,034 B] 2024-09-13T11:18:45.486 INFO:teuthology.orchestra.run.smithi092.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy/main amd64 zip amd64 3.0-12build2 [176 kB] 2024-09-13T11:18:45.488 INFO:teuthology.orchestra.run.smithi092.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 unzip amd64 6.0-26ubuntu3.2 [175 kB] 2024-09-13T11:18:45.497 INFO:teuthology.orchestra.run.smithi162.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2024-09-13T11:18:45.501 INFO:teuthology.orchestra.run.smithi092.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy/universe amd64 luarocks all 3.8.0+dfsg1-1 [140 kB] 2024-09-13T11:18:45.502 INFO:teuthology.orchestra.run.smithi162.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2024-09-13T11:18:45.503 INFO:teuthology.orchestra.run.smithi092.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy/main amd64 liboath0 amd64 2.6.7-3build1 [41.1 kB] 2024-09-13T11:18:45.504 INFO:teuthology.orchestra.run.smithi092.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9,030 B] 2024-09-13T11:18:45.504 INFO:teuthology.orchestra.run.smithi092.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2024-09-13T11:18:45.505 INFO:teuthology.orchestra.run.smithi092.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6,452 B] 2024-09-13T11:18:45.506 INFO:teuthology.orchestra.run.smithi092.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8,716 B] 2024-09-13T11:18:45.506 INFO:teuthology.orchestra.run.smithi092.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2024-09-13T11:18:45.506 INFO:teuthology.orchestra.run.smithi092.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2024-09-13T11:18:45.514 INFO:teuthology.orchestra.run.smithi162.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2024-09-13T11:18:45.518 INFO:teuthology.orchestra.run.smithi162.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2024-09-13T11:18:45.519 INFO:teuthology.orchestra.run.smithi162.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2024-09-13T11:18:45.519 INFO:teuthology.orchestra.run.smithi162.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2024-09-13T11:18:45.521 INFO:teuthology.orchestra.run.smithi092.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7,240 B] 2024-09-13T11:18:45.522 INFO:teuthology.orchestra.run.smithi177.stdout:0 upgraded, 118 newly installed, 0 to remove and 331 not upgraded. 2024-09-13T11:18:45.522 INFO:teuthology.orchestra.run.smithi177.stdout:Need to get 251 MB of archives. 2024-09-13T11:18:45.522 INFO:teuthology.orchestra.run.smithi177.stdout:After this operation, 933 MB of additional disk space will be used. 2024-09-13T11:18:45.522 INFO:teuthology.orchestra.run.smithi177.stdout:Get:1 https://2.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 19.3.0-4697-gd58f7306-1jammy [3,631 kB] 2024-09-13T11:18:45.523 INFO:teuthology.orchestra.run.smithi092.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8,980 B] 2024-09-13T11:18:45.523 INFO:teuthology.orchestra.run.smithi092.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2024-09-13T11:18:45.528 INFO:teuthology.orchestra.run.smithi162.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy/main amd64 libreadline-dev amd64 8.1.2-1 [166 kB] 2024-09-13T11:18:45.530 INFO:teuthology.orchestra.run.smithi162.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblua5.3-dev amd64 5.3.6-1build1 [167 kB] 2024-09-13T11:18:45.532 INFO:teuthology.orchestra.run.smithi162.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua5.1 amd64 5.1.5-8.1build4 [94.6 kB] 2024-09-13T11:18:45.534 INFO:teuthology.orchestra.run.smithi162.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-any all 27ubuntu1 [5,034 B] 2024-09-13T11:18:45.535 INFO:teuthology.orchestra.run.smithi162.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy/main amd64 zip amd64 3.0-12build2 [176 kB] 2024-09-13T11:18:45.537 INFO:teuthology.orchestra.run.smithi162.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 unzip amd64 6.0-26ubuntu3.2 [175 kB] 2024-09-13T11:18:45.539 INFO:teuthology.orchestra.run.smithi092.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2024-09-13T11:18:45.540 INFO:teuthology.orchestra.run.smithi092.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-logutils all 0.3.3-8 [17.6 kB] 2024-09-13T11:18:45.540 INFO:teuthology.orchestra.run.smithi092.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-markupsafe amd64 2.0.1-2build1 [12.7 kB] 2024-09-13T11:18:45.540 INFO:teuthology.orchestra.run.smithi092.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-mako all 1.1.3+ds1-2ubuntu0.1 [60.5 kB] 2024-09-13T11:18:45.541 INFO:teuthology.orchestra.run.smithi092.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2024-09-13T11:18:45.542 INFO:teuthology.orchestra.run.smithi092.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-singledispatch all 3.4.0.3-3 [7,320 B] 2024-09-13T11:18:45.542 INFO:teuthology.orchestra.run.smithi092.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-webob all 1:1.8.6-1.1ubuntu0.1 [86.7 kB] 2024-09-13T11:18:45.549 INFO:teuthology.orchestra.run.smithi092.stdout:Get:36 https://2.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 19.3.0-4697-gd58f7306-1jammy [3,320 kB] 2024-09-13T11:18:45.552 INFO:teuthology.orchestra.run.smithi162.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy/universe amd64 luarocks all 3.8.0+dfsg1-1 [140 kB] 2024-09-13T11:18:45.552 INFO:teuthology.orchestra.run.smithi177.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2024-09-13T11:18:45.554 INFO:teuthology.orchestra.run.smithi162.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy/main amd64 liboath0 amd64 2.6.7-3build1 [41.1 kB] 2024-09-13T11:18:45.554 INFO:teuthology.orchestra.run.smithi162.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9,030 B] 2024-09-13T11:18:45.554 INFO:teuthology.orchestra.run.smithi162.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2024-09-13T11:18:45.555 INFO:teuthology.orchestra.run.smithi162.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6,452 B] 2024-09-13T11:18:45.556 INFO:teuthology.orchestra.run.smithi092.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-waitress all 1.4.4-1.1ubuntu1 [47.3 kB] 2024-09-13T11:18:45.556 INFO:teuthology.orchestra.run.smithi162.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8,716 B] 2024-09-13T11:18:45.556 INFO:teuthology.orchestra.run.smithi162.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2024-09-13T11:18:45.557 INFO:teuthology.orchestra.run.smithi162.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2024-09-13T11:18:45.557 INFO:teuthology.orchestra.run.smithi092.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempita all 0.5.2-6ubuntu1 [15.1 kB] 2024-09-13T11:18:45.557 INFO:teuthology.orchestra.run.smithi092.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-paste all 3.5.0+dfsg1-1 [456 kB] 2024-09-13T11:18:45.569 INFO:teuthology.orchestra.run.smithi162.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7,240 B] 2024-09-13T11:18:45.570 INFO:teuthology.orchestra.run.smithi162.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8,980 B] 2024-09-13T11:18:45.573 INFO:teuthology.orchestra.run.smithi092.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-pastedeploy-tpl all 2.1.1-1 [4,892 B] 2024-09-13T11:18:45.573 INFO:teuthology.orchestra.run.smithi092.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastedeploy all 2.1.1-1 [26.6 kB] 2024-09-13T11:18:45.574 INFO:teuthology.orchestra.run.smithi092.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webtest all 2.0.35-1 [28.5 kB] 2024-09-13T11:18:45.574 INFO:teuthology.orchestra.run.smithi092.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pecan all 1.3.3-4ubuntu2 [87.3 kB] 2024-09-13T11:18:45.576 INFO:teuthology.orchestra.run.smithi092.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-certifi all 2020.6.20-1 [150 kB] 2024-09-13T11:18:45.577 INFO:teuthology.orchestra.run.smithi092.stdout:Get:45 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-idna all 3.3-1ubuntu0.1 [52.1 kB] 2024-09-13T11:18:45.578 INFO:teuthology.orchestra.run.smithi092.stdout:Get:46 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-urllib3 all 1.26.5-1~exp1ubuntu0.1 [98.2 kB] 2024-09-13T11:18:45.586 INFO:teuthology.orchestra.run.smithi162.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2024-09-13T11:18:45.589 INFO:teuthology.orchestra.run.smithi162.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2024-09-13T11:18:45.589 INFO:teuthology.orchestra.run.smithi162.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-logutils all 0.3.3-8 [17.6 kB] 2024-09-13T11:18:45.590 INFO:teuthology.orchestra.run.smithi092.stdout:Get:47 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-requests all 2.25.1+dfsg-2ubuntu0.1 [48.8 kB] 2024-09-13T11:18:45.590 INFO:teuthology.orchestra.run.smithi162.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-markupsafe amd64 2.0.1-2build1 [12.7 kB] 2024-09-13T11:18:45.591 INFO:teuthology.orchestra.run.smithi162.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-mako all 1.1.3+ds1-2ubuntu0.1 [60.5 kB] 2024-09-13T11:18:45.591 INFO:teuthology.orchestra.run.smithi092.stdout:Get:48 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-werkzeug all 2.0.2+dfsg1-1ubuntu0.22.04.2 [181 kB] 2024-09-13T11:18:45.591 INFO:teuthology.orchestra.run.smithi162.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2024-09-13T11:18:45.592 INFO:teuthology.orchestra.run.smithi162.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-singledispatch all 3.4.0.3-3 [7,320 B] 2024-09-13T11:18:45.592 INFO:teuthology.orchestra.run.smithi162.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-webob all 1:1.8.6-1.1ubuntu0.1 [86.7 kB] 2024-09-13T11:18:45.593 INFO:teuthology.orchestra.run.smithi092.stdout:Get:49 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-bcrypt amd64 3.2.0-1build1 [32.7 kB] 2024-09-13T11:18:45.603 INFO:teuthology.orchestra.run.smithi162.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-waitress all 1.4.4-1.1ubuntu1 [47.3 kB] 2024-09-13T11:18:45.604 INFO:teuthology.orchestra.run.smithi162.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempita all 0.5.2-6ubuntu1 [15.1 kB] 2024-09-13T11:18:45.607 INFO:teuthology.orchestra.run.smithi092.stdout:Get:50 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-openssl all 21.0.0-1 [45.2 kB] 2024-09-13T11:18:45.608 INFO:teuthology.orchestra.run.smithi092.stdout:Get:51 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-asyncssh all 2.5.0-1 [185 kB] 2024-09-13T11:18:45.610 INFO:teuthology.orchestra.run.smithi092.stdout:Get:52 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-babel-localedata all 2.8.0+dfsg.1-7 [4,982 kB] 2024-09-13T11:18:45.621 INFO:teuthology.orchestra.run.smithi162.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-paste all 3.5.0+dfsg1-1 [456 kB] 2024-09-13T11:18:45.626 INFO:teuthology.orchestra.run.smithi162.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-pastedeploy-tpl all 2.1.1-1 [4,892 B] 2024-09-13T11:18:45.626 INFO:teuthology.orchestra.run.smithi162.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastedeploy all 2.1.1-1 [26.6 kB] 2024-09-13T11:18:45.627 INFO:teuthology.orchestra.run.smithi162.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webtest all 2.0.35-1 [28.5 kB] 2024-09-13T11:18:45.627 INFO:teuthology.orchestra.run.smithi162.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pecan all 1.3.3-4ubuntu2 [87.3 kB] 2024-09-13T11:18:45.629 INFO:teuthology.orchestra.run.smithi162.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-certifi all 2020.6.20-1 [150 kB] 2024-09-13T11:18:45.631 INFO:teuthology.orchestra.run.smithi162.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-idna all 3.3-1ubuntu0.1 [52.1 kB] 2024-09-13T11:18:45.631 INFO:teuthology.orchestra.run.smithi162.stdout:Get:45 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-urllib3 all 1.26.5-1~exp1ubuntu0.1 [98.2 kB] 2024-09-13T11:18:45.634 INFO:teuthology.orchestra.run.smithi092.stdout:Get:53 https://2.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 19.3.0-4697-gd58f7306-1jammy [722 kB] 2024-09-13T11:18:45.637 INFO:teuthology.orchestra.run.smithi162.stdout:Get:46 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-requests all 2.25.1+dfsg-2ubuntu0.1 [48.8 kB] 2024-09-13T11:18:45.638 INFO:teuthology.orchestra.run.smithi162.stdout:Get:47 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-werkzeug all 2.0.2+dfsg1-1ubuntu0.22.04.2 [181 kB] 2024-09-13T11:18:45.646 INFO:teuthology.orchestra.run.smithi092.stdout:Get:54 https://2.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 19.3.0-4697-gd58f7306-1jammy [356 kB] 2024-09-13T11:18:45.652 INFO:teuthology.orchestra.run.smithi092.stdout:Get:55 https://2.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 19.3.0-4697-gd58f7306-1jammy [32.8 kB] 2024-09-13T11:18:45.653 INFO:teuthology.orchestra.run.smithi092.stdout:Get:56 https://2.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 19.3.0-4697-gd58f7306-1jammy [180 kB] 2024-09-13T11:18:45.655 INFO:teuthology.orchestra.run.smithi162.stdout:Get:48 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-bcrypt amd64 3.2.0-1build1 [32.7 kB] 2024-09-13T11:18:45.655 INFO:teuthology.orchestra.run.smithi092.stdout:Get:57 https://2.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 19.3.0-4697-gd58f7306-1jammy [71.2 kB] 2024-09-13T11:18:45.656 INFO:teuthology.orchestra.run.smithi162.stdout:Get:49 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-openssl all 21.0.0-1 [45.2 kB] 2024-09-13T11:18:45.656 INFO:teuthology.orchestra.run.smithi162.stdout:Get:50 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-asyncssh all 2.5.0-1 [185 kB] 2024-09-13T11:18:45.656 INFO:teuthology.orchestra.run.smithi092.stdout:Get:58 https://2.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 19.3.0-4697-gd58f7306-1jammy [338 kB] 2024-09-13T11:18:45.659 INFO:teuthology.orchestra.run.smithi162.stdout:Get:51 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-babel-localedata all 2.8.0+dfsg.1-7 [4,982 kB] 2024-09-13T11:18:45.663 INFO:teuthology.orchestra.run.smithi092.stdout:Get:59 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-babel all 2.8.0+dfsg.1-7 [85.1 kB] 2024-09-13T11:18:45.664 INFO:teuthology.orchestra.run.smithi092.stdout:Get:60 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-jinja2 all 3.0.3-1ubuntu0.2 [108 kB] 2024-09-13T11:18:45.666 INFO:teuthology.orchestra.run.smithi092.stdout:Get:61 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2024-09-13T11:18:45.666 INFO:teuthology.orchestra.run.smithi092.stdout:Get:62 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2024-09-13T11:18:45.667 INFO:teuthology.orchestra.run.smithi092.stdout:Get:63 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2,058 kB] 2024-09-13T11:18:45.684 INFO:teuthology.orchestra.run.smithi092.stdout:Get:64 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2024-09-13T11:18:45.685 INFO:teuthology.orchestra.run.smithi092.stdout:Get:65 https://2.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 19.3.0-4697-gd58f7306-1jammy [7,979 kB] 2024-09-13T11:18:45.685 INFO:teuthology.orchestra.run.smithi092.stdout:Get:66 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2024-09-13T11:18:45.686 INFO:teuthology.orchestra.run.smithi092.stdout:Get:67 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1,829 kB] 2024-09-13T11:18:45.700 INFO:teuthology.orchestra.run.smithi092.stdout:Get:68 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9,722 B] 2024-09-13T11:18:45.700 INFO:teuthology.orchestra.run.smithi092.stdout:Get:69 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1 all 0.4.8-1 [50.9 kB] 2024-09-13T11:18:45.705 INFO:teuthology.orchestra.run.smithi092.stdout:Get:70 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1-modules all 0.2.1-1 [33.0 kB] 2024-09-13T11:18:45.705 INFO:teuthology.orchestra.run.smithi092.stdout:Get:71 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2024-09-13T11:18:45.705 INFO:teuthology.orchestra.run.smithi092.stdout:Get:72 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2024-09-13T11:18:45.706 INFO:teuthology.orchestra.run.smithi092.stdout:Get:73 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2024-09-13T11:18:45.706 INFO:teuthology.orchestra.run.smithi092.stdout:Get:74 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2024-09-13T11:18:45.707 INFO:teuthology.orchestra.run.smithi092.stdout:Get:75 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2024-09-13T11:18:45.711 INFO:teuthology.orchestra.run.smithi162.stdout:Get:52 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-babel all 2.8.0+dfsg.1-7 [85.1 kB] 2024-09-13T11:18:45.712 INFO:teuthology.orchestra.run.smithi162.stdout:Get:53 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-jinja2 all 3.0.3-1ubuntu0.2 [108 kB] 2024-09-13T11:18:45.713 INFO:teuthology.orchestra.run.smithi162.stdout:Get:54 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2024-09-13T11:18:45.713 INFO:teuthology.orchestra.run.smithi162.stdout:Get:55 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2024-09-13T11:18:45.714 INFO:teuthology.orchestra.run.smithi162.stdout:Get:56 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2,058 kB] 2024-09-13T11:18:45.721 INFO:teuthology.orchestra.run.smithi092.stdout:Get:76 http://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2024-09-13T11:18:45.723 INFO:teuthology.orchestra.run.smithi092.stdout:Get:77 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjq1 amd64 1.6-2.1ubuntu3 [133 kB] 2024-09-13T11:18:45.724 INFO:teuthology.orchestra.run.smithi092.stdout:Get:78 http://archive.ubuntu.com/ubuntu jammy/main amd64 jq amd64 1.6-2.1ubuntu3 [52.5 kB] 2024-09-13T11:18:45.725 INFO:teuthology.orchestra.run.smithi092.stdout:Get:79 http://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2024-09-13T11:18:45.731 INFO:teuthology.orchestra.run.smithi162.stdout:Get:57 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2024-09-13T11:18:45.733 INFO:teuthology.orchestra.run.smithi162.stdout:Get:58 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2024-09-13T11:18:45.733 INFO:teuthology.orchestra.run.smithi162.stdout:Get:59 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1,829 kB] 2024-09-13T11:18:45.739 INFO:teuthology.orchestra.run.smithi092.stdout:Get:80 http://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2024-09-13T11:18:45.741 INFO:teuthology.orchestra.run.smithi092.stdout:Get:81 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-socket amd64 3.0~rc1+git+ac3201d-6 [78.9 kB] 2024-09-13T11:18:45.742 INFO:teuthology.orchestra.run.smithi092.stdout:Get:82 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-sec amd64 1.0.2-1 [37.6 kB] 2024-09-13T11:18:45.742 INFO:teuthology.orchestra.run.smithi092.stdout:Get:83 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.3 [474 kB] 2024-09-13T11:18:45.746 INFO:teuthology.orchestra.run.smithi092.stdout:Get:84 http://archive.ubuntu.com/ubuntu jammy/main amd64 pkg-config amd64 0.29.2-1ubuntu3 [48.2 kB] 2024-09-13T11:18:45.747 INFO:teuthology.orchestra.run.smithi092.stdout:Get:85 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python-asyncssh-doc all 2.5.0-1 [318 kB] 2024-09-13T11:18:45.748 INFO:teuthology.orchestra.run.smithi162.stdout:Get:60 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9,722 B] 2024-09-13T11:18:45.748 INFO:teuthology.orchestra.run.smithi162.stdout:Get:61 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1 all 0.4.8-1 [50.9 kB] 2024-09-13T11:18:45.748 INFO:teuthology.orchestra.run.smithi162.stdout:Get:62 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1-modules all 0.2.1-1 [33.0 kB] 2024-09-13T11:18:45.748 INFO:teuthology.orchestra.run.smithi162.stdout:Get:63 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2024-09-13T11:18:45.749 INFO:teuthology.orchestra.run.smithi162.stdout:Get:64 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2024-09-13T11:18:45.749 INFO:teuthology.orchestra.run.smithi162.stdout:Get:65 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2024-09-13T11:18:45.756 INFO:teuthology.orchestra.run.smithi092.stdout:Get:86 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6,024 B] 2024-09-13T11:18:45.756 INFO:teuthology.orchestra.run.smithi092.stdout:Get:87 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2024-09-13T11:18:45.756 INFO:teuthology.orchestra.run.smithi092.stdout:Get:88 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2024-09-13T11:18:45.757 INFO:teuthology.orchestra.run.smithi092.stdout:Get:89 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2024-09-13T11:18:45.764 INFO:teuthology.orchestra.run.smithi162.stdout:Get:66 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2024-09-13T11:18:45.765 INFO:teuthology.orchestra.run.smithi162.stdout:Get:67 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2024-09-13T11:18:45.767 INFO:teuthology.orchestra.run.smithi162.stdout:Get:68 http://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2024-09-13T11:18:45.773 INFO:teuthology.orchestra.run.smithi092.stdout:Get:90 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2024-09-13T11:18:45.774 INFO:teuthology.orchestra.run.smithi092.stdout:Get:91 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pygments all 2.11.2+dfsg-2 [750 kB] 2024-09-13T11:18:45.780 INFO:teuthology.orchestra.run.smithi092.stdout:Get:92 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyinotify all 0.9.6-1.3 [24.8 kB] 2024-09-13T11:18:45.780 INFO:teuthology.orchestra.run.smithi092.stdout:Get:93 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2024-09-13T11:18:45.780 INFO:teuthology.orchestra.run.smithi092.stdout:Get:94 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2024-09-13T11:18:45.782 INFO:teuthology.orchestra.run.smithi162.stdout:Get:69 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjq1 amd64 1.6-2.1ubuntu3 [133 kB] 2024-09-13T11:18:45.782 INFO:teuthology.orchestra.run.smithi092.stdout:Get:95 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2024-09-13T11:18:45.783 INFO:teuthology.orchestra.run.smithi162.stdout:Get:70 http://archive.ubuntu.com/ubuntu jammy/main amd64 jq amd64 1.6-2.1ubuntu3 [52.5 kB] 2024-09-13T11:18:45.783 INFO:teuthology.orchestra.run.smithi162.stdout:Get:71 http://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2024-09-13T11:18:45.785 INFO:teuthology.orchestra.run.smithi162.stdout:Get:72 https://2.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 19.3.0-4697-gd58f7306-1jammy [3,320 kB] 2024-09-13T11:18:45.786 INFO:teuthology.orchestra.run.smithi162.stdout:Get:73 http://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2024-09-13T11:18:45.788 INFO:teuthology.orchestra.run.smithi162.stdout:Get:74 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-socket amd64 3.0~rc1+git+ac3201d-6 [78.9 kB] 2024-09-13T11:18:45.789 INFO:teuthology.orchestra.run.smithi162.stdout:Get:75 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-sec amd64 1.0.2-1 [37.6 kB] 2024-09-13T11:18:45.790 INFO:teuthology.orchestra.run.smithi162.stdout:Get:76 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.3 [474 kB] 2024-09-13T11:18:45.790 INFO:teuthology.orchestra.run.smithi092.stdout:Get:96 http://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1,983 kB] 2024-09-13T11:18:45.799 INFO:teuthology.orchestra.run.smithi162.stdout:Get:77 http://archive.ubuntu.com/ubuntu jammy/main amd64 pkg-config amd64 0.29.2-1ubuntu3 [48.2 kB] 2024-09-13T11:18:45.799 INFO:teuthology.orchestra.run.smithi162.stdout:Get:78 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python-asyncssh-doc all 2.5.0-1 [318 kB] 2024-09-13T11:18:45.802 INFO:teuthology.orchestra.run.smithi162.stdout:Get:79 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6,024 B] 2024-09-13T11:18:45.816 INFO:teuthology.orchestra.run.smithi162.stdout:Get:80 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2024-09-13T11:18:45.817 INFO:teuthology.orchestra.run.smithi162.stdout:Get:81 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2024-09-13T11:18:45.817 INFO:teuthology.orchestra.run.smithi162.stdout:Get:82 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2024-09-13T11:18:45.818 INFO:teuthology.orchestra.run.smithi162.stdout:Get:83 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2024-09-13T11:18:45.819 INFO:teuthology.orchestra.run.smithi162.stdout:Get:84 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pygments all 2.11.2+dfsg-2 [750 kB] 2024-09-13T11:18:45.825 INFO:teuthology.orchestra.run.smithi162.stdout:Get:85 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyinotify all 0.9.6-1.3 [24.8 kB] 2024-09-13T11:18:45.825 INFO:teuthology.orchestra.run.smithi162.stdout:Get:86 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2024-09-13T11:18:45.833 INFO:teuthology.orchestra.run.smithi162.stdout:Get:87 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2024-09-13T11:18:45.835 INFO:teuthology.orchestra.run.smithi162.stdout:Get:88 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2024-09-13T11:18:45.836 INFO:teuthology.orchestra.run.smithi162.stdout:Get:89 http://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1,983 kB] 2024-09-13T11:18:45.882 INFO:teuthology.orchestra.run.smithi092.stdout:Get:97 https://2.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 19.3.0-4697-gd58f7306-1jammy [112 kB] 2024-09-13T11:18:45.884 INFO:teuthology.orchestra.run.smithi001.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2024-09-13T11:18:45.885 INFO:teuthology.orchestra.run.smithi092.stdout:Get:98 https://2.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 19.3.0-4697-gd58f7306-1jammy [473 kB] 2024-09-13T11:18:45.894 INFO:teuthology.orchestra.run.smithi092.stdout:Get:99 https://2.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 19.3.0-4697-gd58f7306-1jammy [28.2 MB] 2024-09-13T11:18:45.896 INFO:teuthology.orchestra.run.smithi001.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2024-09-13T11:18:45.954 INFO:teuthology.orchestra.run.smithi001.stdout:Get:5 https://2.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 19.3.0-4697-gd58f7306-1jammy [3,320 kB] 2024-09-13T11:18:45.974 INFO:teuthology.orchestra.run.smithi177.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2024-09-13T11:18:45.977 INFO:teuthology.orchestra.run.smithi001.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2,006 kB] 2024-09-13T11:18:45.987 INFO:teuthology.orchestra.run.smithi177.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2024-09-13T11:18:46.023 INFO:teuthology.orchestra.run.smithi162.stdout:Get:90 https://2.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 19.3.0-4697-gd58f7306-1jammy [722 kB] 2024-09-13T11:18:46.055 INFO:teuthology.orchestra.run.smithi162.stdout:Get:91 https://2.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 19.3.0-4697-gd58f7306-1jammy [356 kB] 2024-09-13T11:18:46.063 INFO:teuthology.orchestra.run.smithi162.stdout:Get:92 https://2.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 19.3.0-4697-gd58f7306-1jammy [32.8 kB] 2024-09-13T11:18:46.064 INFO:teuthology.orchestra.run.smithi162.stdout:Get:93 https://2.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 19.3.0-4697-gd58f7306-1jammy [180 kB] 2024-09-13T11:18:46.068 INFO:teuthology.orchestra.run.smithi162.stdout:Get:94 https://2.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 19.3.0-4697-gd58f7306-1jammy [71.2 kB] 2024-09-13T11:18:46.070 INFO:teuthology.orchestra.run.smithi162.stdout:Get:95 https://2.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 19.3.0-4697-gd58f7306-1jammy [338 kB] 2024-09-13T11:18:46.095 INFO:teuthology.orchestra.run.smithi177.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2,006 kB] 2024-09-13T11:18:46.096 INFO:teuthology.orchestra.run.smithi162.stdout:Get:96 https://2.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 19.3.0-4697-gd58f7306-1jammy [7,979 kB] 2024-09-13T11:18:46.206 INFO:teuthology.orchestra.run.smithi001.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2024-09-13T11:18:46.217 INFO:teuthology.orchestra.run.smithi001.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2024-09-13T11:18:46.247 INFO:teuthology.orchestra.run.smithi001.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2024-09-13T11:18:46.256 INFO:teuthology.orchestra.run.smithi001.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2024-09-13T11:18:46.257 INFO:teuthology.orchestra.run.smithi001.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2024-09-13T11:18:46.258 INFO:teuthology.orchestra.run.smithi001.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2024-09-13T11:18:46.276 INFO:teuthology.orchestra.run.smithi001.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy/main amd64 libreadline-dev amd64 8.1.2-1 [166 kB] 2024-09-13T11:18:46.279 INFO:teuthology.orchestra.run.smithi001.stdout:Get:14 https://2.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 19.3.0-4697-gd58f7306-1jammy [722 kB] 2024-09-13T11:18:46.280 INFO:teuthology.orchestra.run.smithi001.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblua5.3-dev amd64 5.3.6-1build1 [167 kB] 2024-09-13T11:18:46.284 INFO:teuthology.orchestra.run.smithi001.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua5.1 amd64 5.1.5-8.1build4 [94.6 kB] 2024-09-13T11:18:46.288 INFO:teuthology.orchestra.run.smithi001.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-any all 27ubuntu1 [5,034 B] 2024-09-13T11:18:46.355 INFO:teuthology.orchestra.run.smithi177.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2024-09-13T11:18:46.367 INFO:teuthology.orchestra.run.smithi177.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2024-09-13T11:18:46.369 INFO:teuthology.orchestra.run.smithi001.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy/main amd64 zip amd64 3.0-12build2 [176 kB] 2024-09-13T11:18:46.373 INFO:teuthology.orchestra.run.smithi001.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 unzip amd64 6.0-26ubuntu3.2 [175 kB] 2024-09-13T11:18:46.378 INFO:teuthology.orchestra.run.smithi001.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy/universe amd64 luarocks all 3.8.0+dfsg1-1 [140 kB] 2024-09-13T11:18:46.381 INFO:teuthology.orchestra.run.smithi001.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy/main amd64 liboath0 amd64 2.6.7-3build1 [41.1 kB] 2024-09-13T11:18:46.383 INFO:teuthology.orchestra.run.smithi001.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9,030 B] 2024-09-13T11:18:46.383 INFO:teuthology.orchestra.run.smithi001.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2024-09-13T11:18:46.385 INFO:teuthology.orchestra.run.smithi001.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6,452 B] 2024-09-13T11:18:46.386 INFO:teuthology.orchestra.run.smithi001.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8,716 B] 2024-09-13T11:18:46.387 INFO:teuthology.orchestra.run.smithi001.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2024-09-13T11:18:46.396 INFO:teuthology.orchestra.run.smithi001.stdout:Get:27 https://2.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 19.3.0-4697-gd58f7306-1jammy [356 kB] 2024-09-13T11:18:46.398 INFO:teuthology.orchestra.run.smithi177.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2024-09-13T11:18:46.407 INFO:teuthology.orchestra.run.smithi177.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2024-09-13T11:18:46.407 INFO:teuthology.orchestra.run.smithi177.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2024-09-13T11:18:46.409 INFO:teuthology.orchestra.run.smithi177.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2024-09-13T11:18:46.423 INFO:teuthology.orchestra.run.smithi001.stdout:Get:28 https://2.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 19.3.0-4697-gd58f7306-1jammy [32.8 kB] 2024-09-13T11:18:46.424 INFO:teuthology.orchestra.run.smithi001.stdout:Get:29 https://2.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 19.3.0-4697-gd58f7306-1jammy [180 kB] 2024-09-13T11:18:46.428 INFO:teuthology.orchestra.run.smithi177.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy/main amd64 libreadline-dev amd64 8.1.2-1 [166 kB] 2024-09-13T11:18:46.433 INFO:teuthology.orchestra.run.smithi177.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblua5.3-dev amd64 5.3.6-1build1 [167 kB] 2024-09-13T11:18:46.438 INFO:teuthology.orchestra.run.smithi177.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua5.1 amd64 5.1.5-8.1build4 [94.6 kB] 2024-09-13T11:18:46.441 INFO:teuthology.orchestra.run.smithi177.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-any all 27ubuntu1 [5,034 B] 2024-09-13T11:18:46.452 INFO:teuthology.orchestra.run.smithi001.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2024-09-13T11:18:46.470 INFO:teuthology.orchestra.run.smithi001.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7,240 B] 2024-09-13T11:18:46.470 INFO:teuthology.orchestra.run.smithi001.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8,980 B] 2024-09-13T11:18:46.471 INFO:teuthology.orchestra.run.smithi001.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2024-09-13T11:18:46.471 INFO:teuthology.orchestra.run.smithi001.stdout:Get:34 https://2.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 19.3.0-4697-gd58f7306-1jammy [71.2 kB] 2024-09-13T11:18:46.476 INFO:teuthology.orchestra.run.smithi001.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2024-09-13T11:18:46.477 INFO:teuthology.orchestra.run.smithi001.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-logutils all 0.3.3-8 [17.6 kB] 2024-09-13T11:18:46.493 INFO:teuthology.orchestra.run.smithi001.stdout:Get:37 https://2.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 19.3.0-4697-gd58f7306-1jammy [338 kB] 2024-09-13T11:18:46.500 INFO:teuthology.orchestra.run.smithi001.stdout:Get:38 https://2.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 19.3.0-4697-gd58f7306-1jammy [7,979 kB] 2024-09-13T11:18:46.519 INFO:teuthology.orchestra.run.smithi177.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy/main amd64 zip amd64 3.0-12build2 [176 kB] 2024-09-13T11:18:46.523 INFO:teuthology.orchestra.run.smithi177.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 unzip amd64 6.0-26ubuntu3.2 [175 kB] 2024-09-13T11:18:46.524 INFO:teuthology.orchestra.run.smithi162.stdout:Get:97 https://2.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 19.3.0-4697-gd58f7306-1jammy [112 kB] 2024-09-13T11:18:46.527 INFO:teuthology.orchestra.run.smithi162.stdout:Get:98 https://2.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 19.3.0-4697-gd58f7306-1jammy [473 kB] 2024-09-13T11:18:46.527 INFO:teuthology.orchestra.run.smithi177.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy/universe amd64 luarocks all 3.8.0+dfsg1-1 [140 kB] 2024-09-13T11:18:46.531 INFO:teuthology.orchestra.run.smithi177.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy/main amd64 liboath0 amd64 2.6.7-3build1 [41.1 kB] 2024-09-13T11:18:46.532 INFO:teuthology.orchestra.run.smithi177.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9,030 B] 2024-09-13T11:18:46.533 INFO:teuthology.orchestra.run.smithi177.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2024-09-13T11:18:46.534 INFO:teuthology.orchestra.run.smithi001.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-markupsafe amd64 2.0.1-2build1 [12.7 kB] 2024-09-13T11:18:46.534 INFO:teuthology.orchestra.run.smithi001.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-mako all 1.1.3+ds1-2ubuntu0.1 [60.5 kB] 2024-09-13T11:18:46.535 INFO:teuthology.orchestra.run.smithi177.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6,452 B] 2024-09-13T11:18:46.536 INFO:teuthology.orchestra.run.smithi001.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2024-09-13T11:18:46.537 INFO:teuthology.orchestra.run.smithi001.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-singledispatch all 3.4.0.3-3 [7,320 B] 2024-09-13T11:18:46.538 INFO:teuthology.orchestra.run.smithi162.stdout:Get:99 https://2.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 19.3.0-4697-gd58f7306-1jammy [28.2 MB] 2024-09-13T11:18:46.600 INFO:teuthology.orchestra.run.smithi177.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8,716 B] 2024-09-13T11:18:46.601 INFO:teuthology.orchestra.run.smithi177.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2024-09-13T11:18:46.602 INFO:teuthology.orchestra.run.smithi177.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2024-09-13T11:18:46.615 INFO:teuthology.orchestra.run.smithi001.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-webob all 1:1.8.6-1.1ubuntu0.1 [86.7 kB] 2024-09-13T11:18:46.617 INFO:teuthology.orchestra.run.smithi001.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-waitress all 1.4.4-1.1ubuntu1 [47.3 kB] 2024-09-13T11:18:46.618 INFO:teuthology.orchestra.run.smithi001.stdout:Get:45 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempita all 0.5.2-6ubuntu1 [15.1 kB] 2024-09-13T11:18:46.619 INFO:teuthology.orchestra.run.smithi001.stdout:Get:46 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-paste all 3.5.0+dfsg1-1 [456 kB] 2024-09-13T11:18:46.630 INFO:teuthology.orchestra.run.smithi001.stdout:Get:47 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-pastedeploy-tpl all 2.1.1-1 [4,892 B] 2024-09-13T11:18:46.682 INFO:teuthology.orchestra.run.smithi177.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7,240 B] 2024-09-13T11:18:46.683 INFO:teuthology.orchestra.run.smithi177.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8,980 B] 2024-09-13T11:18:46.683 INFO:teuthology.orchestra.run.smithi177.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2024-09-13T11:18:46.688 INFO:teuthology.orchestra.run.smithi177.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2024-09-13T11:18:46.689 INFO:teuthology.orchestra.run.smithi177.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-logutils all 0.3.3-8 [17.6 kB] 2024-09-13T11:18:46.690 INFO:teuthology.orchestra.run.smithi177.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-markupsafe amd64 2.0.1-2build1 [12.7 kB] 2024-09-13T11:18:46.691 INFO:teuthology.orchestra.run.smithi177.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-mako all 1.1.3+ds1-2ubuntu0.1 [60.5 kB] 2024-09-13T11:18:46.696 INFO:teuthology.orchestra.run.smithi001.stdout:Get:48 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastedeploy all 2.1.1-1 [26.6 kB] 2024-09-13T11:18:46.697 INFO:teuthology.orchestra.run.smithi001.stdout:Get:49 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webtest all 2.0.35-1 [28.5 kB] 2024-09-13T11:18:46.698 INFO:teuthology.orchestra.run.smithi001.stdout:Get:50 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pecan all 1.3.3-4ubuntu2 [87.3 kB] 2024-09-13T11:18:46.701 INFO:teuthology.orchestra.run.smithi001.stdout:Get:51 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-certifi all 2020.6.20-1 [150 kB] 2024-09-13T11:18:46.704 INFO:teuthology.orchestra.run.smithi001.stdout:Get:52 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-idna all 3.3-1ubuntu0.1 [52.1 kB] 2024-09-13T11:18:46.763 INFO:teuthology.orchestra.run.smithi177.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2024-09-13T11:18:46.764 INFO:teuthology.orchestra.run.smithi177.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-singledispatch all 3.4.0.3-3 [7,320 B] 2024-09-13T11:18:46.764 INFO:teuthology.orchestra.run.smithi177.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-webob all 1:1.8.6-1.1ubuntu0.1 [86.7 kB] 2024-09-13T11:18:46.777 INFO:teuthology.orchestra.run.smithi001.stdout:Get:53 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-urllib3 all 1.26.5-1~exp1ubuntu0.1 [98.2 kB] 2024-09-13T11:18:46.780 INFO:teuthology.orchestra.run.smithi001.stdout:Get:54 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-requests all 2.25.1+dfsg-2ubuntu0.1 [48.8 kB] 2024-09-13T11:18:46.781 INFO:teuthology.orchestra.run.smithi001.stdout:Get:55 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-werkzeug all 2.0.2+dfsg1-1ubuntu0.22.04.2 [181 kB] 2024-09-13T11:18:46.786 INFO:teuthology.orchestra.run.smithi001.stdout:Get:56 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-bcrypt amd64 3.2.0-1build1 [32.7 kB] 2024-09-13T11:18:46.787 INFO:teuthology.orchestra.run.smithi001.stdout:Get:57 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-openssl all 21.0.0-1 [45.2 kB] 2024-09-13T11:18:46.845 INFO:teuthology.orchestra.run.smithi177.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-waitress all 1.4.4-1.1ubuntu1 [47.3 kB] 2024-09-13T11:18:46.846 INFO:teuthology.orchestra.run.smithi177.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempita all 0.5.2-6ubuntu1 [15.1 kB] 2024-09-13T11:18:46.847 INFO:teuthology.orchestra.run.smithi177.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-paste all 3.5.0+dfsg1-1 [456 kB] 2024-09-13T11:18:46.858 INFO:teuthology.orchestra.run.smithi177.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-pastedeploy-tpl all 2.1.1-1 [4,892 B] 2024-09-13T11:18:46.858 INFO:teuthology.orchestra.run.smithi177.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastedeploy all 2.1.1-1 [26.6 kB] 2024-09-13T11:18:46.858 INFO:teuthology.orchestra.run.smithi001.stdout:Get:58 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-asyncssh all 2.5.0-1 [185 kB] 2024-09-13T11:18:46.859 INFO:teuthology.orchestra.run.smithi177.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webtest all 2.0.35-1 [28.5 kB] 2024-09-13T11:18:46.860 INFO:teuthology.orchestra.run.smithi177.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pecan all 1.3.3-4ubuntu2 [87.3 kB] 2024-09-13T11:18:46.863 INFO:teuthology.orchestra.run.smithi001.stdout:Get:59 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-babel-localedata all 2.8.0+dfsg.1-7 [4,982 kB] 2024-09-13T11:18:46.896 INFO:teuthology.orchestra.run.smithi177.stdout:Get:43 https://2.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 19.3.0-4697-gd58f7306-1jammy [3,320 kB] 2024-09-13T11:18:46.926 INFO:teuthology.orchestra.run.smithi177.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-certifi all 2020.6.20-1 [150 kB] 2024-09-13T11:18:46.930 INFO:teuthology.orchestra.run.smithi177.stdout:Get:45 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-idna all 3.3-1ubuntu0.1 [52.1 kB] 2024-09-13T11:18:46.931 INFO:teuthology.orchestra.run.smithi177.stdout:Get:46 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-urllib3 all 1.26.5-1~exp1ubuntu0.1 [98.2 kB] 2024-09-13T11:18:46.950 INFO:teuthology.orchestra.run.smithi092.stdout:Get:100 https://2.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 19.3.0-4697-gd58f7306-1jammy [5,206 kB] 2024-09-13T11:18:47.008 INFO:teuthology.orchestra.run.smithi177.stdout:Get:47 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-requests all 2.25.1+dfsg-2ubuntu0.1 [48.8 kB] 2024-09-13T11:18:47.009 INFO:teuthology.orchestra.run.smithi177.stdout:Get:48 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-werkzeug all 2.0.2+dfsg1-1ubuntu0.22.04.2 [181 kB] 2024-09-13T11:18:47.014 INFO:teuthology.orchestra.run.smithi177.stdout:Get:49 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-bcrypt amd64 3.2.0-1build1 [32.7 kB] 2024-09-13T11:18:47.015 INFO:teuthology.orchestra.run.smithi177.stdout:Get:50 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-openssl all 21.0.0-1 [45.2 kB] 2024-09-13T11:18:47.016 INFO:teuthology.orchestra.run.smithi177.stdout:Get:51 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-asyncssh all 2.5.0-1 [185 kB] 2024-09-13T11:18:47.021 INFO:teuthology.orchestra.run.smithi177.stdout:Get:52 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-babel-localedata all 2.8.0+dfsg.1-7 [4,982 kB] 2024-09-13T11:18:47.026 INFO:teuthology.orchestra.run.smithi001.stdout:Get:60 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-babel all 2.8.0+dfsg.1-7 [85.1 kB] 2024-09-13T11:18:47.027 INFO:teuthology.orchestra.run.smithi001.stdout:Get:61 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-jinja2 all 3.0.3-1ubuntu0.2 [108 kB] 2024-09-13T11:18:47.030 INFO:teuthology.orchestra.run.smithi001.stdout:Get:62 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2024-09-13T11:18:47.030 INFO:teuthology.orchestra.run.smithi001.stdout:Get:63 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2024-09-13T11:18:47.032 INFO:teuthology.orchestra.run.smithi001.stdout:Get:64 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2,058 kB] 2024-09-13T11:18:47.041 INFO:teuthology.orchestra.run.smithi092.stdout:Get:101 https://2.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 19.3.0-4697-gd58f7306-1jammy [251 kB] 2024-09-13T11:18:47.046 INFO:teuthology.orchestra.run.smithi092.stdout:Get:102 https://2.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 19.3.0-4697-gd58f7306-1jammy [124 kB] 2024-09-13T11:18:47.049 INFO:teuthology.orchestra.run.smithi092.stdout:Get:103 https://2.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 19.3.0-4697-gd58f7306-1jammy [1,561 kB] 2024-09-13T11:18:47.068 INFO:teuthology.orchestra.run.smithi001.stdout:Get:65 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2024-09-13T11:18:47.071 INFO:teuthology.orchestra.run.smithi001.stdout:Get:66 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2024-09-13T11:18:47.072 INFO:teuthology.orchestra.run.smithi001.stdout:Get:67 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1,829 kB] 2024-09-13T11:18:47.105 INFO:teuthology.orchestra.run.smithi092.stdout:Get:104 https://2.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 19.3.0-4697-gd58f7306-1jammy [6,309 kB] 2024-09-13T11:18:47.129 INFO:teuthology.orchestra.run.smithi001.stdout:Get:68 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9,722 B] 2024-09-13T11:18:47.130 INFO:teuthology.orchestra.run.smithi001.stdout:Get:69 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1 all 0.4.8-1 [50.9 kB] 2024-09-13T11:18:47.182 INFO:teuthology.orchestra.run.smithi177.stdout:Get:53 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-babel all 2.8.0+dfsg.1-7 [85.1 kB] 2024-09-13T11:18:47.183 INFO:teuthology.orchestra.run.smithi177.stdout:Get:54 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-jinja2 all 3.0.3-1ubuntu0.2 [108 kB] 2024-09-13T11:18:47.188 INFO:teuthology.orchestra.run.smithi177.stdout:Get:55 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2024-09-13T11:18:47.189 INFO:teuthology.orchestra.run.smithi177.stdout:Get:56 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2024-09-13T11:18:47.189 INFO:teuthology.orchestra.run.smithi177.stdout:Get:57 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2,058 kB] 2024-09-13T11:18:47.192 INFO:teuthology.orchestra.run.smithi001.stdout:Get:70 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1-modules all 0.2.1-1 [33.0 kB] 2024-09-13T11:18:47.193 INFO:teuthology.orchestra.run.smithi001.stdout:Get:71 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2024-09-13T11:18:47.194 INFO:teuthology.orchestra.run.smithi001.stdout:Get:72 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2024-09-13T11:18:47.195 INFO:teuthology.orchestra.run.smithi001.stdout:Get:73 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2024-09-13T11:18:47.196 INFO:teuthology.orchestra.run.smithi001.stdout:Get:74 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2024-09-13T11:18:47.197 INFO:teuthology.orchestra.run.smithi001.stdout:Get:75 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2024-09-13T11:18:47.203 INFO:teuthology.orchestra.run.smithi001.stdout:Get:76 http://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2024-09-13T11:18:47.222 INFO:teuthology.orchestra.run.smithi177.stdout:Get:58 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2024-09-13T11:18:47.256 INFO:teuthology.orchestra.run.smithi177.stdout:Get:59 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2024-09-13T11:18:47.257 INFO:teuthology.orchestra.run.smithi177.stdout:Get:60 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1,829 kB] 2024-09-13T11:18:47.273 INFO:teuthology.orchestra.run.smithi001.stdout:Get:77 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjq1 amd64 1.6-2.1ubuntu3 [133 kB] 2024-09-13T11:18:47.277 INFO:teuthology.orchestra.run.smithi001.stdout:Get:78 http://archive.ubuntu.com/ubuntu jammy/main amd64 jq amd64 1.6-2.1ubuntu3 [52.5 kB] 2024-09-13T11:18:47.278 INFO:teuthology.orchestra.run.smithi001.stdout:Get:79 http://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2024-09-13T11:18:47.286 INFO:teuthology.orchestra.run.smithi177.stdout:Get:61 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9,722 B] 2024-09-13T11:18:47.287 INFO:teuthology.orchestra.run.smithi177.stdout:Get:62 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1 all 0.4.8-1 [50.9 kB] 2024-09-13T11:18:47.338 INFO:teuthology.orchestra.run.smithi092.stdout:Get:105 https://2.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 19.3.0-4697-gd58f7306-1jammy [23.1 MB] 2024-09-13T11:18:47.349 INFO:teuthology.orchestra.run.smithi177.stdout:Get:63 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1-modules all 0.2.1-1 [33.0 kB] 2024-09-13T11:18:47.350 INFO:teuthology.orchestra.run.smithi177.stdout:Get:64 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2024-09-13T11:18:47.351 INFO:teuthology.orchestra.run.smithi177.stdout:Get:65 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2024-09-13T11:18:47.352 INFO:teuthology.orchestra.run.smithi177.stdout:Get:66 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2024-09-13T11:18:47.353 INFO:teuthology.orchestra.run.smithi177.stdout:Get:67 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2024-09-13T11:18:47.354 INFO:teuthology.orchestra.run.smithi177.stdout:Get:68 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2024-09-13T11:18:47.355 INFO:teuthology.orchestra.run.smithi001.stdout:Get:80 http://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2024-09-13T11:18:47.360 INFO:teuthology.orchestra.run.smithi177.stdout:Get:69 http://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2024-09-13T11:18:47.361 INFO:teuthology.orchestra.run.smithi001.stdout:Get:81 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-socket amd64 3.0~rc1+git+ac3201d-6 [78.9 kB] 2024-09-13T11:18:47.363 INFO:teuthology.orchestra.run.smithi001.stdout:Get:82 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-sec amd64 1.0.2-1 [37.6 kB] 2024-09-13T11:18:47.365 INFO:teuthology.orchestra.run.smithi001.stdout:Get:83 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.3 [474 kB] 2024-09-13T11:18:47.376 INFO:teuthology.orchestra.run.smithi001.stdout:Get:84 http://archive.ubuntu.com/ubuntu jammy/main amd64 pkg-config amd64 0.29.2-1ubuntu3 [48.2 kB] 2024-09-13T11:18:47.377 INFO:teuthology.orchestra.run.smithi001.stdout:Get:85 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python-asyncssh-doc all 2.5.0-1 [318 kB] 2024-09-13T11:18:47.385 INFO:teuthology.orchestra.run.smithi001.stdout:Get:86 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6,024 B] 2024-09-13T11:18:47.431 INFO:teuthology.orchestra.run.smithi177.stdout:Get:70 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjq1 amd64 1.6-2.1ubuntu3 [133 kB] 2024-09-13T11:18:47.434 INFO:teuthology.orchestra.run.smithi177.stdout:Get:71 http://archive.ubuntu.com/ubuntu jammy/main amd64 jq amd64 1.6-2.1ubuntu3 [52.5 kB] 2024-09-13T11:18:47.435 INFO:teuthology.orchestra.run.smithi177.stdout:Get:72 http://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2024-09-13T11:18:47.436 INFO:teuthology.orchestra.run.smithi001.stdout:Get:87 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2024-09-13T11:18:47.438 INFO:teuthology.orchestra.run.smithi001.stdout:Get:88 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2024-09-13T11:18:47.439 INFO:teuthology.orchestra.run.smithi001.stdout:Get:89 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2024-09-13T11:18:47.512 INFO:teuthology.orchestra.run.smithi177.stdout:Get:73 http://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2024-09-13T11:18:47.518 INFO:teuthology.orchestra.run.smithi001.stdout:Get:90 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2024-09-13T11:18:47.518 INFO:teuthology.orchestra.run.smithi177.stdout:Get:74 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-socket amd64 3.0~rc1+git+ac3201d-6 [78.9 kB] 2024-09-13T11:18:47.520 INFO:teuthology.orchestra.run.smithi001.stdout:Get:91 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pygments all 2.11.2+dfsg-2 [750 kB] 2024-09-13T11:18:47.520 INFO:teuthology.orchestra.run.smithi177.stdout:Get:75 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-sec amd64 1.0.2-1 [37.6 kB] 2024-09-13T11:18:47.522 INFO:teuthology.orchestra.run.smithi177.stdout:Get:76 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.3 [474 kB] 2024-09-13T11:18:47.533 INFO:teuthology.orchestra.run.smithi177.stdout:Get:77 http://archive.ubuntu.com/ubuntu jammy/main amd64 pkg-config amd64 0.29.2-1ubuntu3 [48.2 kB] 2024-09-13T11:18:47.534 INFO:teuthology.orchestra.run.smithi177.stdout:Get:78 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python-asyncssh-doc all 2.5.0-1 [318 kB] 2024-09-13T11:18:47.536 INFO:teuthology.orchestra.run.smithi001.stdout:Get:92 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyinotify all 0.9.6-1.3 [24.8 kB] 2024-09-13T11:18:47.537 INFO:teuthology.orchestra.run.smithi001.stdout:Get:93 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2024-09-13T11:18:47.538 INFO:teuthology.orchestra.run.smithi001.stdout:Get:94 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2024-09-13T11:18:47.542 INFO:teuthology.orchestra.run.smithi177.stdout:Get:79 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6,024 B] 2024-09-13T11:18:47.543 INFO:teuthology.orchestra.run.smithi001.stdout:Get:95 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2024-09-13T11:18:47.545 INFO:teuthology.orchestra.run.smithi001.stdout:Get:96 http://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1,983 kB] 2024-09-13T11:18:47.593 INFO:teuthology.orchestra.run.smithi177.stdout:Get:80 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2024-09-13T11:18:47.595 INFO:teuthology.orchestra.run.smithi177.stdout:Get:81 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2024-09-13T11:18:47.596 INFO:teuthology.orchestra.run.smithi177.stdout:Get:82 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2024-09-13T11:18:47.603 INFO:teuthology.orchestra.run.smithi001.stdout:Get:97 https://2.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 19.3.0-4697-gd58f7306-1jammy [112 kB] 2024-09-13T11:18:47.612 INFO:teuthology.orchestra.run.smithi001.stdout:Get:98 https://2.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 19.3.0-4697-gd58f7306-1jammy [473 kB] 2024-09-13T11:18:47.675 INFO:teuthology.orchestra.run.smithi177.stdout:Get:83 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2024-09-13T11:18:47.675 INFO:teuthology.orchestra.run.smithi001.stdout:Get:99 https://2.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 19.3.0-4697-gd58f7306-1jammy [28.2 MB] 2024-09-13T11:18:47.677 INFO:teuthology.orchestra.run.smithi177.stdout:Get:84 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pygments all 2.11.2+dfsg-2 [750 kB] 2024-09-13T11:18:47.694 INFO:teuthology.orchestra.run.smithi177.stdout:Get:85 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyinotify all 0.9.6-1.3 [24.8 kB] 2024-09-13T11:18:47.695 INFO:teuthology.orchestra.run.smithi177.stdout:Get:86 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2024-09-13T11:18:47.695 INFO:teuthology.orchestra.run.smithi177.stdout:Get:87 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2024-09-13T11:18:47.700 INFO:teuthology.orchestra.run.smithi177.stdout:Get:88 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2024-09-13T11:18:47.702 INFO:teuthology.orchestra.run.smithi177.stdout:Get:89 http://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1,983 kB] 2024-09-13T11:18:47.793 INFO:teuthology.orchestra.run.smithi177.stdout:Get:90 https://2.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 19.3.0-4697-gd58f7306-1jammy [722 kB] 2024-09-13T11:18:47.866 INFO:teuthology.orchestra.run.smithi162.stdout:Get:100 https://2.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 19.3.0-4697-gd58f7306-1jammy [5,206 kB] 2024-09-13T11:18:47.964 INFO:teuthology.orchestra.run.smithi177.stdout:Get:91 https://2.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 19.3.0-4697-gd58f7306-1jammy [356 kB] 2024-09-13T11:18:48.001 INFO:teuthology.orchestra.run.smithi162.stdout:Get:101 https://2.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 19.3.0-4697-gd58f7306-1jammy [251 kB] 2024-09-13T11:18:48.036 INFO:teuthology.orchestra.run.smithi162.stdout:Get:102 https://2.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 19.3.0-4697-gd58f7306-1jammy [124 kB] 2024-09-13T11:18:48.039 INFO:teuthology.orchestra.run.smithi162.stdout:Get:103 https://2.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 19.3.0-4697-gd58f7306-1jammy [1,561 kB] 2024-09-13T11:18:48.043 INFO:teuthology.orchestra.run.smithi177.stdout:Get:92 https://2.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 19.3.0-4697-gd58f7306-1jammy [32.8 kB] 2024-09-13T11:18:48.044 INFO:teuthology.orchestra.run.smithi177.stdout:Get:93 https://2.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 19.3.0-4697-gd58f7306-1jammy [180 kB] 2024-09-13T11:18:48.105 INFO:teuthology.orchestra.run.smithi162.stdout:Get:104 https://2.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 19.3.0-4697-gd58f7306-1jammy [6,309 kB] 2024-09-13T11:18:48.106 INFO:teuthology.orchestra.run.smithi177.stdout:Get:94 https://2.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 19.3.0-4697-gd58f7306-1jammy [71.2 kB] 2024-09-13T11:18:48.156 INFO:teuthology.orchestra.run.smithi177.stdout:Get:95 https://2.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 19.3.0-4697-gd58f7306-1jammy [338 kB] 2024-09-13T11:18:48.220 INFO:teuthology.orchestra.run.smithi177.stdout:Get:96 https://2.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 19.3.0-4697-gd58f7306-1jammy [7,979 kB] 2024-09-13T11:18:48.228 INFO:teuthology.orchestra.run.smithi092.stdout:Get:106 https://2.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 19.3.0-4697-gd58f7306-1jammy [14.1 kB] 2024-09-13T11:18:48.228 INFO:teuthology.orchestra.run.smithi092.stdout:Get:107 https://2.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 19.3.0-4697-gd58f7306-1jammy [880 kB] 2024-09-13T11:18:48.245 INFO:teuthology.orchestra.run.smithi092.stdout:Get:108 https://2.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 19.3.0-4697-gd58f7306-1jammy [2,433 kB] 2024-09-13T11:18:48.288 INFO:teuthology.orchestra.run.smithi092.stdout:Get:109 https://2.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 19.3.0-4697-gd58f7306-1jammy [812 kB] 2024-09-13T11:18:48.309 INFO:teuthology.orchestra.run.smithi092.stdout:Get:110 https://2.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 19.3.0-4697-gd58f7306-1jammy [160 kB] 2024-09-13T11:18:48.312 INFO:teuthology.orchestra.run.smithi092.stdout:Get:111 https://2.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 19.3.0-4697-gd58f7306-1jammy [61.4 MB] 2024-09-13T11:18:48.366 INFO:teuthology.orchestra.run.smithi162.stdout:Get:105 https://2.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 19.3.0-4697-gd58f7306-1jammy [23.1 MB] 2024-09-13T11:18:49.464 INFO:teuthology.orchestra.run.smithi162.stdout:Get:106 https://2.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 19.3.0-4697-gd58f7306-1jammy [14.1 kB] 2024-09-13T11:18:49.465 INFO:teuthology.orchestra.run.smithi162.stdout:Get:107 https://2.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 19.3.0-4697-gd58f7306-1jammy [880 kB] 2024-09-13T11:18:49.483 INFO:teuthology.orchestra.run.smithi162.stdout:Get:108 https://2.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 19.3.0-4697-gd58f7306-1jammy [2,433 kB] 2024-09-13T11:18:49.544 INFO:teuthology.orchestra.run.smithi162.stdout:Get:109 https://2.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 19.3.0-4697-gd58f7306-1jammy [812 kB] 2024-09-13T11:18:49.582 INFO:teuthology.orchestra.run.smithi162.stdout:Get:110 https://2.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 19.3.0-4697-gd58f7306-1jammy [160 kB] 2024-09-13T11:18:49.585 INFO:teuthology.orchestra.run.smithi162.stdout:Get:111 https://2.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 19.3.0-4697-gd58f7306-1jammy [61.4 MB] 2024-09-13T11:18:50.432 INFO:teuthology.orchestra.run.smithi177.stdout:Get:97 https://2.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 19.3.0-4697-gd58f7306-1jammy [112 kB] 2024-09-13T11:18:50.434 INFO:teuthology.orchestra.run.smithi177.stdout:Get:98 https://2.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 19.3.0-4697-gd58f7306-1jammy [473 kB] 2024-09-13T11:18:50.575 INFO:teuthology.orchestra.run.smithi177.stdout:Get:99 https://2.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 19.3.0-4697-gd58f7306-1jammy [28.2 MB] 2024-09-13T11:18:50.826 INFO:teuthology.orchestra.run.smithi092.stdout:Get:112 https://2.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 19.3.0-4697-gd58f7306-1jammy [8,625 kB] 2024-09-13T11:18:51.043 INFO:teuthology.orchestra.run.smithi092.stdout:Get:113 https://2.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 19.3.0-4697-gd58f7306-1jammy [14.2 kB] 2024-09-13T11:18:51.044 INFO:teuthology.orchestra.run.smithi092.stdout:Get:114 https://2.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 19.3.0-4697-gd58f7306-1jammy [55.7 MB] 2024-09-13T11:18:51.637 INFO:teuthology.orchestra.run.smithi001.stdout:Get:100 https://2.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 19.3.0-4697-gd58f7306-1jammy [5,206 kB] 2024-09-13T11:18:52.293 INFO:teuthology.orchestra.run.smithi001.stdout:Get:101 https://2.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 19.3.0-4697-gd58f7306-1jammy [251 kB] 2024-09-13T11:18:52.322 INFO:teuthology.orchestra.run.smithi001.stdout:Get:102 https://2.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 19.3.0-4697-gd58f7306-1jammy [124 kB] 2024-09-13T11:18:52.325 INFO:teuthology.orchestra.run.smithi001.stdout:Get:103 https://2.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 19.3.0-4697-gd58f7306-1jammy [1,561 kB] 2024-09-13T11:18:52.390 INFO:teuthology.orchestra.run.smithi162.stdout:Get:112 https://2.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 19.3.0-4697-gd58f7306-1jammy [8,625 kB] 2024-09-13T11:18:52.504 INFO:teuthology.orchestra.run.smithi001.stdout:Get:104 https://2.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 19.3.0-4697-gd58f7306-1jammy [6,309 kB] 2024-09-13T11:18:52.644 INFO:teuthology.orchestra.run.smithi162.stdout:Get:113 https://2.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 19.3.0-4697-gd58f7306-1jammy [14.2 kB] 2024-09-13T11:18:52.644 INFO:teuthology.orchestra.run.smithi162.stdout:Get:114 https://2.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 19.3.0-4697-gd58f7306-1jammy [55.7 MB] 2024-09-13T11:18:53.226 INFO:teuthology.orchestra.run.smithi092.stdout:Get:115 https://2.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 19.3.0-4697-gd58f7306-1jammy [130 kB] 2024-09-13T11:18:53.229 INFO:teuthology.orchestra.run.smithi092.stdout:Get:116 https://2.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 19.3.0-4697-gd58f7306-1jammy [38.9 kB] 2024-09-13T11:18:53.230 INFO:teuthology.orchestra.run.smithi092.stdout:Get:117 https://2.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 19.3.0-4697-gd58f7306-1jammy [15.9 MB] 2024-09-13T11:18:53.320 INFO:teuthology.orchestra.run.smithi001.stdout:Get:105 https://2.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 19.3.0-4697-gd58f7306-1jammy [23.1 MB] 2024-09-13T11:18:53.798 INFO:teuthology.orchestra.run.smithi092.stdout:Get:118 https://2.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 19.3.0-4697-gd58f7306-1jammy [91.3 kB] 2024-09-13T11:18:54.386 INFO:teuthology.orchestra.run.smithi092.stdout:Fetched 251 MB in 9s (29.3 MB/s) 2024-09-13T11:18:54.515 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2024-09-13T11:18:54.571 INFO:teuthology.orchestra.run.smithi092.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 165990 files and directories currently installed.) 2024-09-13T11:18:54.576 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../000-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2024-09-13T11:18:54.657 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-09-13T11:18:54.688 INFO:teuthology.orchestra.run.smithi162.stdout:Get:115 https://2.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 19.3.0-4697-gd58f7306-1jammy [130 kB] 2024-09-13T11:18:54.691 INFO:teuthology.orchestra.run.smithi162.stdout:Get:116 https://2.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 19.3.0-4697-gd58f7306-1jammy [38.9 kB] 2024-09-13T11:18:54.692 INFO:teuthology.orchestra.run.smithi162.stdout:Get:117 https://2.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 19.3.0-4697-gd58f7306-1jammy [15.9 MB] 2024-09-13T11:18:54.961 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2024-09-13T11:18:54.987 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../001-libdouble-conversion3_3.1.7-4_amd64.deb ... 2024-09-13T11:18:55.027 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2024-09-13T11:18:55.072 INFO:teuthology.orchestra.run.smithi162.stdout:Get:118 https://2.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 19.3.0-4697-gd58f7306-1jammy [91.3 kB] 2024-09-13T11:18:55.142 INFO:teuthology.orchestra.run.smithi001.stdout:Get:106 https://2.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 19.3.0-4697-gd58f7306-1jammy [14.1 kB] 2024-09-13T11:18:55.143 INFO:teuthology.orchestra.run.smithi001.stdout:Get:107 https://2.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 19.3.0-4697-gd58f7306-1jammy [880 kB] 2024-09-13T11:18:55.160 INFO:teuthology.orchestra.run.smithi001.stdout:Get:108 https://2.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 19.3.0-4697-gd58f7306-1jammy [2,433 kB] 2024-09-13T11:18:55.250 INFO:teuthology.orchestra.run.smithi001.stdout:Get:109 https://2.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 19.3.0-4697-gd58f7306-1jammy [812 kB] 2024-09-13T11:18:55.276 INFO:teuthology.orchestra.run.smithi001.stdout:Get:110 https://2.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 19.3.0-4697-gd58f7306-1jammy [160 kB] 2024-09-13T11:18:55.289 INFO:teuthology.orchestra.run.smithi001.stdout:Get:111 https://2.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 19.3.0-4697-gd58f7306-1jammy [61.4 MB] 2024-09-13T11:18:55.327 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2024-09-13T11:18:55.354 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../002-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2024-09-13T11:18:55.405 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-09-13T11:18:55.496 INFO:teuthology.orchestra.run.smithi177.stdout:Get:100 https://2.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 19.3.0-4697-gd58f7306-1jammy [5,206 kB] 2024-09-13T11:18:55.661 INFO:teuthology.orchestra.run.smithi162.stdout:Fetched 251 MB in 10s (25.6 MB/s) 2024-09-13T11:18:55.828 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2024-09-13T11:18:55.842 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package libqt5core5a:amd64. 2024-09-13T11:18:55.869 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../003-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-09-13T11:18:55.917 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-13T11:18:55.928 INFO:teuthology.orchestra.run.smithi162.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 165990 files and directories currently installed.) 2024-09-13T11:18:55.934 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../000-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2024-09-13T11:18:56.029 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-09-13T11:18:56.225 INFO:teuthology.orchestra.run.smithi177.stdout:Get:101 https://2.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 19.3.0-4697-gd58f7306-1jammy [251 kB] 2024-09-13T11:18:56.274 INFO:teuthology.orchestra.run.smithi177.stdout:Get:102 https://2.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 19.3.0-4697-gd58f7306-1jammy [124 kB] 2024-09-13T11:18:56.276 INFO:teuthology.orchestra.run.smithi177.stdout:Get:103 https://2.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 19.3.0-4697-gd58f7306-1jammy [1,561 kB] 2024-09-13T11:18:56.312 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2024-09-13T11:18:56.339 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../004-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-09-13T11:18:56.379 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-13T11:18:56.527 INFO:teuthology.orchestra.run.smithi177.stdout:Get:104 https://2.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 19.3.0-4697-gd58f7306-1jammy [6,309 kB] 2024-09-13T11:18:56.665 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package libqt5network5:amd64. 2024-09-13T11:18:56.682 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2024-09-13T11:18:56.692 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../005-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-09-13T11:18:56.709 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../001-libdouble-conversion3_3.1.7-4_amd64.deb ... 2024-09-13T11:18:56.723 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-13T11:18:56.749 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2024-09-13T11:18:57.068 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2024-09-13T11:18:57.095 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../006-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2024-09-13T11:18:57.119 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2024-09-13T11:18:57.134 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-09-13T11:18:57.145 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../002-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2024-09-13T11:18:57.277 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-09-13T11:18:57.421 INFO:teuthology.orchestra.run.smithi177.stdout:Get:105 https://2.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 19.3.0-4697-gd58f7306-1jammy [23.1 MB] 2024-09-13T11:18:57.471 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package librados2. 2024-09-13T11:18:57.497 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../007-librados2_19.3.0-4697-gd58f7306-1jammy_amd64.deb ... 2024-09-13T11:18:57.537 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking librados2 (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:18:57.832 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package libqt5core5a:amd64. 2024-09-13T11:18:57.857 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../003-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-09-13T11:18:57.914 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-13T11:18:57.940 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package librbd1. 2024-09-13T11:18:57.967 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../008-librbd1_19.3.0-4697-gd58f7306-1jammy_amd64.deb ... 2024-09-13T11:18:57.998 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking librbd1 (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:18:58.343 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2024-09-13T11:18:58.369 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../004-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-09-13T11:18:58.409 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-13T11:18:58.560 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package libcephfs2. 2024-09-13T11:18:58.586 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../009-libcephfs2_19.3.0-4697-gd58f7306-1jammy_amd64.deb ... 2024-09-13T11:18:58.626 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking libcephfs2 (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:18:58.821 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package libqt5network5:amd64. 2024-09-13T11:18:58.847 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../005-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-09-13T11:18:58.896 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-13T11:18:58.904 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package python3-rados. 2024-09-13T11:18:58.919 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../010-python3-rados_19.3.0-4697-gd58f7306-1jammy_amd64.deb ... 2024-09-13T11:18:58.962 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking python3-rados (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:18:59.282 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package python3-ceph-argparse. 2024-09-13T11:18:59.308 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../011-python3-ceph-argparse_19.3.0-4697-gd58f7306-1jammy_all.deb ... 2024-09-13T11:18:59.316 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2024-09-13T11:18:59.342 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../006-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2024-09-13T11:18:59.348 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking python3-ceph-argparse (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:18:59.390 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-09-13T11:18:59.576 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package python3-cephfs. 2024-09-13T11:18:59.591 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../012-python3-cephfs_19.3.0-4697-gd58f7306-1jammy_amd64.deb ... 2024-09-13T11:18:59.626 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking python3-cephfs (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:18:59.735 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package librados2. 2024-09-13T11:18:59.761 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../007-librados2_19.3.0-4697-gd58f7306-1jammy_amd64.deb ... 2024-09-13T11:18:59.802 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking librados2 (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:18:59.920 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package python3-ceph-common. 2024-09-13T11:18:59.944 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../013-python3-ceph-common_19.3.0-4697-gd58f7306-1jammy_all.deb ... 2024-09-13T11:18:59.987 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking python3-ceph-common (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:19:00.042 INFO:teuthology.orchestra.run.smithi177.stdout:Get:106 https://2.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 19.3.0-4697-gd58f7306-1jammy [14.1 kB] 2024-09-13T11:19:00.043 INFO:teuthology.orchestra.run.smithi177.stdout:Get:107 https://2.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 19.3.0-4697-gd58f7306-1jammy [880 kB] 2024-09-13T11:19:00.060 INFO:teuthology.orchestra.run.smithi177.stdout:Get:108 https://2.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 19.3.0-4697-gd58f7306-1jammy [2,433 kB] 2024-09-13T11:19:00.102 INFO:teuthology.orchestra.run.smithi177.stdout:Get:109 https://2.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 19.3.0-4697-gd58f7306-1jammy [812 kB] 2024-09-13T11:19:00.128 INFO:teuthology.orchestra.run.smithi177.stdout:Get:110 https://2.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 19.3.0-4697-gd58f7306-1jammy [160 kB] 2024-09-13T11:19:00.131 INFO:teuthology.orchestra.run.smithi177.stdout:Get:111 https://2.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 19.3.0-4697-gd58f7306-1jammy [61.4 MB] 2024-09-13T11:19:00.278 INFO:teuthology.orchestra.run.smithi001.stdout:Get:112 https://2.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 19.3.0-4697-gd58f7306-1jammy [8,625 kB] 2024-09-13T11:19:00.315 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package python3-wcwidth. 2024-09-13T11:19:00.330 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package librbd1. 2024-09-13T11:19:00.341 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../014-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2024-09-13T11:19:00.356 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../008-librbd1_19.3.0-4697-gd58f7306-1jammy_amd64.deb ... 2024-09-13T11:19:00.381 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2024-09-13T11:19:00.405 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking librbd1 (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:19:00.432 INFO:teuthology.orchestra.run.smithi001.stdout:Get:113 https://2.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 19.3.0-4697-gd58f7306-1jammy [14.2 kB] 2024-09-13T11:19:00.432 INFO:teuthology.orchestra.run.smithi001.stdout:Get:114 https://2.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 19.3.0-4697-gd58f7306-1jammy [55.7 MB] 2024-09-13T11:19:00.726 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package python3-prettytable. 2024-09-13T11:19:00.752 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../015-python3-prettytable_2.5.0-2_all.deb ... 2024-09-13T11:19:00.793 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2024-09-13T11:19:00.925 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package libcephfs2. 2024-09-13T11:19:00.951 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../009-libcephfs2_19.3.0-4697-gd58f7306-1jammy_amd64.deb ... 2024-09-13T11:19:00.999 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking libcephfs2 (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:19:01.053 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package python3-rbd. 2024-09-13T11:19:01.069 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../016-python3-rbd_19.3.0-4697-gd58f7306-1jammy_amd64.deb ... 2024-09-13T11:19:01.103 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking python3-rbd (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:19:01.335 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package python3-rados. 2024-09-13T11:19:01.362 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../010-python3-rados_19.3.0-4697-gd58f7306-1jammy_amd64.deb ... 2024-09-13T11:19:01.402 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking python3-rados (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:19:01.431 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package librdkafka1:amd64. 2024-09-13T11:19:01.457 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../017-librdkafka1_1.8.0-1build1_amd64.deb ... 2024-09-13T11:19:01.498 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2024-09-13T11:19:01.776 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package libreadline-dev:amd64. 2024-09-13T11:19:01.802 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../018-libreadline-dev_8.1.2-1_amd64.deb ... 2024-09-13T11:19:01.842 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking libreadline-dev:amd64 (8.1.2-1) ... 2024-09-13T11:19:01.920 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package python3-ceph-argparse. 2024-09-13T11:19:01.945 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../011-python3-ceph-argparse_19.3.0-4697-gd58f7306-1jammy_all.deb ... 2024-09-13T11:19:01.997 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking python3-ceph-argparse (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:19:02.137 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package liblua5.3-dev:amd64. 2024-09-13T11:19:02.152 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../019-liblua5.3-dev_5.3.6-1build1_amd64.deb ... 2024-09-13T11:19:02.195 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-09-13T11:19:02.300 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package python3-cephfs. 2024-09-13T11:19:02.326 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../012-python3-cephfs_19.3.0-4697-gd58f7306-1jammy_amd64.deb ... 2024-09-13T11:19:02.374 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking python3-cephfs (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:19:02.429 INFO:teuthology.orchestra.run.smithi001.stdout:Get:115 https://2.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 19.3.0-4697-gd58f7306-1jammy [130 kB] 2024-09-13T11:19:02.432 INFO:teuthology.orchestra.run.smithi001.stdout:Get:116 https://2.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 19.3.0-4697-gd58f7306-1jammy [38.9 kB] 2024-09-13T11:19:02.433 INFO:teuthology.orchestra.run.smithi001.stdout:Get:117 https://2.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 19.3.0-4697-gd58f7306-1jammy [15.9 MB] 2024-09-13T11:19:02.474 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package lua5.1. 2024-09-13T11:19:02.499 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../020-lua5.1_5.1.5-8.1build4_amd64.deb ... 2024-09-13T11:19:02.541 INFO:teuthology.orchestra.run.smithi177.stdout:Get:112 https://2.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 19.3.0-4697-gd58f7306-1jammy [8,625 kB] 2024-09-13T11:19:02.716 INFO:teuthology.orchestra.run.smithi177.stdout:Get:113 https://2.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 19.3.0-4697-gd58f7306-1jammy [14.2 kB] 2024-09-13T11:19:02.717 INFO:teuthology.orchestra.run.smithi177.stdout:Get:114 https://2.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 19.3.0-4697-gd58f7306-1jammy [55.7 MB] 2024-09-13T11:19:02.752 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package python3-ceph-common. 2024-09-13T11:19:02.752 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking lua5.1 (5.1.5-8.1build4) ... 2024-09-13T11:19:02.778 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../013-python3-ceph-common_19.3.0-4697-gd58f7306-1jammy_all.deb ... 2024-09-13T11:19:02.827 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking python3-ceph-common (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:19:02.849 INFO:teuthology.orchestra.run.smithi001.stdout:Get:118 https://2.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 19.3.0-4697-gd58f7306-1jammy [91.3 kB] 2024-09-13T11:19:03.068 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package lua-any. 2024-09-13T11:19:03.084 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../021-lua-any_27ubuntu1_all.deb ... 2024-09-13T11:19:03.127 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking lua-any (27ubuntu1) ... 2024-09-13T11:19:03.189 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package python3-wcwidth. 2024-09-13T11:19:03.215 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../014-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2024-09-13T11:19:03.255 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2024-09-13T11:19:03.388 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package zip. 2024-09-13T11:19:03.414 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../022-zip_3.0-12build2_amd64.deb ... 2024-09-13T11:19:03.445 INFO:teuthology.orchestra.run.smithi001.stdout:Fetched 251 MB in 18s (14.3 MB/s) 2024-09-13T11:19:03.454 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking zip (3.0-12build2) ... 2024-09-13T11:19:03.600 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package python3-prettytable. 2024-09-13T11:19:03.626 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../015-python3-prettytable_2.5.0-2_all.deb ... 2024-09-13T11:19:03.635 INFO:teuthology.orchestra.run.smithi001.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2024-09-13T11:19:03.666 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2024-09-13T11:19:03.738 INFO:teuthology.orchestra.run.smithi001.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 165990 files and directories currently installed.) 2024-09-13T11:19:03.744 INFO:teuthology.orchestra.run.smithi001.stdout:Preparing to unpack .../000-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2024-09-13T11:19:03.850 INFO:teuthology.orchestra.run.smithi001.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-09-13T11:19:03.866 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package unzip. 2024-09-13T11:19:03.881 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../023-unzip_6.0-26ubuntu3.2_amd64.deb ... 2024-09-13T11:19:03.916 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking unzip (6.0-26ubuntu3.2) ... 2024-09-13T11:19:03.919 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package python3-rbd. 2024-09-13T11:19:03.935 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../016-python3-rbd_19.3.0-4697-gd58f7306-1jammy_amd64.deb ... 2024-09-13T11:19:03.969 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking python3-rbd (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:19:04.045 INFO:teuthology.orchestra.run.smithi177.stdout:Get:115 https://2.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 19.3.0-4697-gd58f7306-1jammy [130 kB] 2024-09-13T11:19:04.048 INFO:teuthology.orchestra.run.smithi177.stdout:Get:116 https://2.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 19.3.0-4697-gd58f7306-1jammy [38.9 kB] 2024-09-13T11:19:04.107 INFO:teuthology.orchestra.run.smithi177.stdout:Get:117 https://2.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 19.3.0-4697-gd58f7306-1jammy [15.9 MB] 2024-09-13T11:19:04.228 INFO:teuthology.orchestra.run.smithi001.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2024-09-13T11:19:04.256 INFO:teuthology.orchestra.run.smithi001.stdout:Preparing to unpack .../001-libdouble-conversion3_3.1.7-4_amd64.deb ... 2024-09-13T11:19:04.295 INFO:teuthology.orchestra.run.smithi001.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2024-09-13T11:19:04.314 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package librdkafka1:amd64. 2024-09-13T11:19:04.320 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package luarocks. 2024-09-13T11:19:04.341 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../017-librdkafka1_1.8.0-1build1_amd64.deb ... 2024-09-13T11:19:04.346 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../024-luarocks_3.8.0+dfsg1-1_all.deb ... 2024-09-13T11:19:04.381 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2024-09-13T11:19:04.386 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking luarocks (3.8.0+dfsg1-1) ... 2024-09-13T11:19:04.422 INFO:teuthology.orchestra.run.smithi177.stdout:Get:118 https://2.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 19.3.0-4697-gd58f7306-1jammy [91.3 kB] 2024-09-13T11:19:04.658 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package libreadline-dev:amd64. 2024-09-13T11:19:04.675 INFO:teuthology.orchestra.run.smithi001.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2024-09-13T11:19:04.676 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../018-libreadline-dev_8.1.2-1_amd64.deb ... 2024-09-13T11:19:04.700 INFO:teuthology.orchestra.run.smithi001.stdout:Preparing to unpack .../002-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2024-09-13T11:19:04.716 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking libreadline-dev:amd64 (8.1.2-1) ... 2024-09-13T11:19:04.748 INFO:teuthology.orchestra.run.smithi001.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-09-13T11:19:04.761 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package librgw2. 2024-09-13T11:19:04.787 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../025-librgw2_19.3.0-4697-gd58f7306-1jammy_amd64.deb ... 2024-09-13T11:19:04.827 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking librgw2 (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:19:04.994 INFO:teuthology.orchestra.run.smithi177.stdout:Fetched 251 MB in 19s (13.2 MB/s) 2024-09-13T11:19:05.019 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package liblua5.3-dev:amd64. 2024-09-13T11:19:05.035 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../019-liblua5.3-dev_5.3.6-1build1_amd64.deb ... 2024-09-13T11:19:05.078 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-09-13T11:19:05.128 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2024-09-13T11:19:05.226 INFO:teuthology.orchestra.run.smithi177.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 165990 files and directories currently installed.) 2024-09-13T11:19:05.232 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../000-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2024-09-13T11:19:05.251 INFO:teuthology.orchestra.run.smithi001.stdout:Selecting previously unselected package libqt5core5a:amd64. 2024-09-13T11:19:05.279 INFO:teuthology.orchestra.run.smithi001.stdout:Preparing to unpack .../003-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-09-13T11:19:05.301 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-09-13T11:19:05.326 INFO:teuthology.orchestra.run.smithi001.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-13T11:19:05.373 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package lua5.1. 2024-09-13T11:19:05.400 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../020-lua5.1_5.1.5-8.1build4_amd64.deb ... 2024-09-13T11:19:05.422 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package python3-rgw. 2024-09-13T11:19:05.437 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../026-python3-rgw_19.3.0-4697-gd58f7306-1jammy_amd64.deb ... 2024-09-13T11:19:05.439 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking lua5.1 (5.1.5-8.1build4) ... 2024-09-13T11:19:05.479 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking python3-rgw (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:19:05.588 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2024-09-13T11:19:05.614 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../001-libdouble-conversion3_3.1.7-4_amd64.deb ... 2024-09-13T11:19:05.646 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2024-09-13T11:19:05.751 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package lua-any. 2024-09-13T11:19:05.767 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../021-lua-any_27ubuntu1_all.deb ... 2024-09-13T11:19:05.808 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package liboath0:amd64. 2024-09-13T11:19:05.809 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking lua-any (27ubuntu1) ... 2024-09-13T11:19:05.834 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../027-liboath0_2.6.7-3build1_amd64.deb ... 2024-09-13T11:19:05.863 INFO:teuthology.orchestra.run.smithi001.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2024-09-13T11:19:05.874 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking liboath0:amd64 (2.6.7-3build1) ... 2024-09-13T11:19:05.890 INFO:teuthology.orchestra.run.smithi001.stdout:Preparing to unpack .../004-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-09-13T11:19:05.929 INFO:teuthology.orchestra.run.smithi001.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-13T11:19:05.999 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2024-09-13T11:19:06.025 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../002-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2024-09-13T11:19:06.066 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-09-13T11:19:06.070 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package zip. 2024-09-13T11:19:06.086 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../022-zip_3.0-12build2_amd64.deb ... 2024-09-13T11:19:06.129 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking zip (3.0-12build2) ... 2024-09-13T11:19:06.185 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package libradosstriper1. 2024-09-13T11:19:06.212 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../028-libradosstriper1_19.3.0-4697-gd58f7306-1jammy_amd64.deb ... 2024-09-13T11:19:06.252 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking libradosstriper1 (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:19:06.257 INFO:teuthology.orchestra.run.smithi001.stdout:Selecting previously unselected package libqt5network5:amd64. 2024-09-13T11:19:06.284 INFO:teuthology.orchestra.run.smithi001.stdout:Preparing to unpack .../005-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-09-13T11:19:06.324 INFO:teuthology.orchestra.run.smithi001.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-13T11:19:06.432 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package unzip. 2024-09-13T11:19:06.447 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../023-unzip_6.0-26ubuntu3.2_amd64.deb ... 2024-09-13T11:19:06.482 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking unzip (6.0-26ubuntu3.2) ... 2024-09-13T11:19:06.511 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package libqt5core5a:amd64. 2024-09-13T11:19:06.537 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../003-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-09-13T11:19:06.585 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-13T11:19:06.652 INFO:teuthology.orchestra.run.smithi001.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2024-09-13T11:19:06.679 INFO:teuthology.orchestra.run.smithi001.stdout:Preparing to unpack .../006-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2024-09-13T11:19:06.680 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package ceph-common. 2024-09-13T11:19:06.707 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../029-ceph-common_19.3.0-4697-gd58f7306-1jammy_amd64.deb ... 2024-09-13T11:19:06.719 INFO:teuthology.orchestra.run.smithi001.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-09-13T11:19:06.747 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking ceph-common (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:19:06.886 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package luarocks. 2024-09-13T11:19:06.912 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../024-luarocks_3.8.0+dfsg1-1_all.deb ... 2024-09-13T11:19:06.939 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2024-09-13T11:19:07.108 INFO:teuthology.orchestra.run.smithi001.stdout:Selecting previously unselected package librados2. 2024-09-13T11:19:07.109 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking luarocks (3.8.0+dfsg1-1) ... 2024-09-13T11:19:07.109 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../004-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-09-13T11:19:07.109 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-13T11:19:07.124 INFO:teuthology.orchestra.run.smithi001.stdout:Preparing to unpack .../007-librados2_19.3.0-4697-gd58f7306-1jammy_amd64.deb ... 2024-09-13T11:19:07.163 INFO:teuthology.orchestra.run.smithi001.stdout:Unpacking librados2 (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:19:07.342 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package libqt5network5:amd64. 2024-09-13T11:19:07.368 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../005-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-09-13T11:19:07.408 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-13T11:19:07.528 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package librgw2. 2024-09-13T11:19:07.554 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../025-librgw2_19.3.0-4697-gd58f7306-1jammy_amd64.deb ... 2024-09-13T11:19:07.593 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking librgw2 (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:19:07.616 INFO:teuthology.orchestra.run.smithi001.stdout:Selecting previously unselected package librbd1. 2024-09-13T11:19:07.643 INFO:teuthology.orchestra.run.smithi001.stdout:Preparing to unpack .../008-librbd1_19.3.0-4697-gd58f7306-1jammy_amd64.deb ... 2024-09-13T11:19:07.683 INFO:teuthology.orchestra.run.smithi001.stdout:Unpacking librbd1 (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:19:07.753 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2024-09-13T11:19:07.779 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../006-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2024-09-13T11:19:07.819 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-09-13T11:19:08.086 INFO:teuthology.orchestra.run.smithi001.stdout:Selecting previously unselected package libcephfs2. 2024-09-13T11:19:08.089 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package librados2. 2024-09-13T11:19:08.107 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../007-librados2_19.3.0-4697-gd58f7306-1jammy_amd64.deb ... 2024-09-13T11:19:08.112 INFO:teuthology.orchestra.run.smithi001.stdout:Preparing to unpack .../009-libcephfs2_19.3.0-4697-gd58f7306-1jammy_amd64.deb ... 2024-09-13T11:19:08.138 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking librados2 (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:19:08.160 INFO:teuthology.orchestra.run.smithi001.stdout:Unpacking libcephfs2 (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:19:08.205 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package python3-rgw. 2024-09-13T11:19:08.211 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package ceph-base. 2024-09-13T11:19:08.231 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../026-python3-rgw_19.3.0-4697-gd58f7306-1jammy_amd64.deb ... 2024-09-13T11:19:08.237 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../030-ceph-base_19.3.0-4697-gd58f7306-1jammy_amd64.deb ... 2024-09-13T11:19:08.263 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking python3-rgw (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:19:08.286 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking ceph-base (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:19:08.583 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package liboath0:amd64. 2024-09-13T11:19:08.609 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../027-liboath0_2.6.7-3build1_amd64.deb ... 2024-09-13T11:19:08.621 INFO:teuthology.orchestra.run.smithi001.stdout:Selecting previously unselected package python3-rados. 2024-09-13T11:19:08.648 INFO:teuthology.orchestra.run.smithi001.stdout:Preparing to unpack .../010-python3-rados_19.3.0-4697-gd58f7306-1jammy_amd64.deb ... 2024-09-13T11:19:08.649 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking liboath0:amd64 (2.6.7-3build1) ... 2024-09-13T11:19:08.688 INFO:teuthology.orchestra.run.smithi001.stdout:Unpacking python3-rados (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:19:08.800 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package librbd1. 2024-09-13T11:19:08.826 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../008-librbd1_19.3.0-4697-gd58f7306-1jammy_amd64.deb ... 2024-09-13T11:19:08.840 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package python3-jaraco.functools. 2024-09-13T11:19:08.866 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking librbd1 (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:19:08.866 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../031-python3-jaraco.functools_3.4.0-2_all.deb ... 2024-09-13T11:19:08.898 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2024-09-13T11:19:08.961 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package libradosstriper1. 2024-09-13T11:19:08.987 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../028-libradosstriper1_19.3.0-4697-gd58f7306-1jammy_amd64.deb ... 2024-09-13T11:19:08.999 INFO:teuthology.orchestra.run.smithi001.stdout:Selecting previously unselected package python3-ceph-argparse. 2024-09-13T11:19:09.025 INFO:teuthology.orchestra.run.smithi001.stdout:Preparing to unpack .../011-python3-ceph-argparse_19.3.0-4697-gd58f7306-1jammy_all.deb ... 2024-09-13T11:19:09.027 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking libradosstriper1 (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:19:09.065 INFO:teuthology.orchestra.run.smithi001.stdout:Unpacking python3-ceph-argparse (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:19:09.185 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package python3-cheroot. 2024-09-13T11:19:09.211 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../032-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2024-09-13T11:19:09.251 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-09-13T11:19:09.259 INFO:teuthology.orchestra.run.smithi001.stdout:Selecting previously unselected package python3-cephfs. 2024-09-13T11:19:09.276 INFO:teuthology.orchestra.run.smithi001.stdout:Preparing to unpack .../012-python3-cephfs_19.3.0-4697-gd58f7306-1jammy_amd64.deb ... 2024-09-13T11:19:09.302 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package libcephfs2. 2024-09-13T11:19:09.310 INFO:teuthology.orchestra.run.smithi001.stdout:Unpacking python3-cephfs (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:19:09.328 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../009-libcephfs2_19.3.0-4697-gd58f7306-1jammy_amd64.deb ... 2024-09-13T11:19:09.369 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking libcephfs2 (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:19:09.431 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package ceph-common. 2024-09-13T11:19:09.458 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../029-ceph-common_19.3.0-4697-gd58f7306-1jammy_amd64.deb ... 2024-09-13T11:19:09.489 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking ceph-common (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:19:09.546 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package python3-jaraco.classes. 2024-09-13T11:19:09.572 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../033-python3-jaraco.classes_3.2.1-3_all.deb ... 2024-09-13T11:19:09.604 INFO:teuthology.orchestra.run.smithi001.stdout:Selecting previously unselected package python3-ceph-common. 2024-09-13T11:19:09.612 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2024-09-13T11:19:09.612 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package python3-rados. 2024-09-13T11:19:09.628 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../010-python3-rados_19.3.0-4697-gd58f7306-1jammy_amd64.deb ... 2024-09-13T11:19:09.630 INFO:teuthology.orchestra.run.smithi001.stdout:Preparing to unpack .../013-python3-ceph-common_19.3.0-4697-gd58f7306-1jammy_all.deb ... 2024-09-13T11:19:09.663 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking python3-rados (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:19:09.670 INFO:teuthology.orchestra.run.smithi001.stdout:Unpacking python3-ceph-common (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:19:09.908 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package python3-jaraco.text. 2024-09-13T11:19:09.933 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../034-python3-jaraco.text_3.6.0-2_all.deb ... 2024-09-13T11:19:09.957 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package python3-ceph-argparse. 2024-09-13T11:19:09.974 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2024-09-13T11:19:09.982 INFO:teuthology.orchestra.run.smithi001.stdout:Selecting previously unselected package python3-wcwidth. 2024-09-13T11:19:09.983 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../011-python3-ceph-argparse_19.3.0-4697-gd58f7306-1jammy_all.deb ... 2024-09-13T11:19:10.008 INFO:teuthology.orchestra.run.smithi001.stdout:Preparing to unpack .../014-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2024-09-13T11:19:10.024 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking python3-ceph-argparse (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:19:10.048 INFO:teuthology.orchestra.run.smithi001.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2024-09-13T11:19:10.277 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package python3-jaraco.collections. 2024-09-13T11:19:10.293 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package python3-cephfs. 2024-09-13T11:19:10.303 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../035-python3-jaraco.collections_3.4.0-2_all.deb ... 2024-09-13T11:19:10.321 INFO:teuthology.orchestra.run.smithi001.stdout:Selecting previously unselected package python3-prettytable. 2024-09-13T11:19:10.321 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../012-python3-cephfs_19.3.0-4697-gd58f7306-1jammy_amd64.deb ... 2024-09-13T11:19:10.343 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2024-09-13T11:19:10.347 INFO:teuthology.orchestra.run.smithi001.stdout:Preparing to unpack .../015-python3-prettytable_2.5.0-2_all.deb ... 2024-09-13T11:19:10.359 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking python3-cephfs (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:19:10.378 INFO:teuthology.orchestra.run.smithi001.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2024-09-13T11:19:10.639 INFO:teuthology.orchestra.run.smithi001.stdout:Selecting previously unselected package python3-rbd. 2024-09-13T11:19:10.646 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package python3-tempora. 2024-09-13T11:19:10.655 INFO:teuthology.orchestra.run.smithi001.stdout:Preparing to unpack .../016-python3-rbd_19.3.0-4697-gd58f7306-1jammy_amd64.deb ... 2024-09-13T11:19:10.670 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package python3-ceph-common. 2024-09-13T11:19:10.672 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../036-python3-tempora_4.1.2-1_all.deb ... 2024-09-13T11:19:10.697 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../013-python3-ceph-common_19.3.0-4697-gd58f7306-1jammy_all.deb ... 2024-09-13T11:19:10.697 INFO:teuthology.orchestra.run.smithi001.stdout:Unpacking python3-rbd (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:19:10.712 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking python3-tempora (4.1.2-1) ... 2024-09-13T11:19:10.737 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking python3-ceph-common (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:19:10.974 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package python3-portend. 2024-09-13T11:19:11.000 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../037-python3-portend_3.0.0-1_all.deb ... 2024-09-13T11:19:11.036 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package ceph-base. 2024-09-13T11:19:11.049 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking python3-portend (3.0.0-1) ... 2024-09-13T11:19:11.058 INFO:teuthology.orchestra.run.smithi001.stdout:Selecting previously unselected package librdkafka1:amd64. 2024-09-13T11:19:11.063 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../030-ceph-base_19.3.0-4697-gd58f7306-1jammy_amd64.deb ... 2024-09-13T11:19:11.065 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package python3-wcwidth. 2024-09-13T11:19:11.085 INFO:teuthology.orchestra.run.smithi001.stdout:Preparing to unpack .../017-librdkafka1_1.8.0-1build1_amd64.deb ... 2024-09-13T11:19:11.091 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../014-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2024-09-13T11:19:11.111 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking ceph-base (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:19:11.125 INFO:teuthology.orchestra.run.smithi001.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2024-09-13T11:19:11.131 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2024-09-13T11:19:11.352 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package python3-zc.lockfile. 2024-09-13T11:19:11.378 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../038-python3-zc.lockfile_2.0-1_all.deb ... 2024-09-13T11:19:11.403 INFO:teuthology.orchestra.run.smithi001.stdout:Selecting previously unselected package libreadline-dev:amd64. 2024-09-13T11:19:11.417 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package python3-prettytable. 2024-09-13T11:19:11.418 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2024-09-13T11:19:11.419 INFO:teuthology.orchestra.run.smithi001.stdout:Preparing to unpack .../018-libreadline-dev_8.1.2-1_amd64.deb ... 2024-09-13T11:19:11.444 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../015-python3-prettytable_2.5.0-2_all.deb ... 2024-09-13T11:19:11.453 INFO:teuthology.orchestra.run.smithi001.stdout:Unpacking libreadline-dev:amd64 (8.1.2-1) ... 2024-09-13T11:19:11.476 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2024-09-13T11:19:11.666 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package python3-jaraco.functools. 2024-09-13T11:19:11.692 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../031-python3-jaraco.functools_3.4.0-2_all.deb ... 2024-09-13T11:19:11.695 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package python3-rbd. 2024-09-13T11:19:11.710 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../016-python3-rbd_19.3.0-4697-gd58f7306-1jammy_amd64.deb ... 2024-09-13T11:19:11.714 INFO:teuthology.orchestra.run.smithi001.stdout:Selecting previously unselected package liblua5.3-dev:amd64. 2024-09-13T11:19:11.721 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package python3-cherrypy3. 2024-09-13T11:19:11.729 INFO:teuthology.orchestra.run.smithi001.stdout:Preparing to unpack .../019-liblua5.3-dev_5.3.6-1build1_amd64.deb ... 2024-09-13T11:19:11.732 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2024-09-13T11:19:11.748 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../039-python3-cherrypy3_18.6.1-4_all.deb ... 2024-09-13T11:19:11.753 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking python3-rbd (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:19:11.764 INFO:teuthology.orchestra.run.smithi001.stdout:Unpacking liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-09-13T11:19:11.787 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2024-09-13T11:19:12.010 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package python3-cheroot. 2024-09-13T11:19:12.036 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../032-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2024-09-13T11:19:12.076 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-09-13T11:19:12.084 INFO:teuthology.orchestra.run.smithi001.stdout:Selecting previously unselected package lua5.1. 2024-09-13T11:19:12.097 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package librdkafka1:amd64. 2024-09-13T11:19:12.110 INFO:teuthology.orchestra.run.smithi001.stdout:Preparing to unpack .../020-lua5.1_5.1.5-8.1build4_amd64.deb ... 2024-09-13T11:19:12.124 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../017-librdkafka1_1.8.0-1build1_amd64.deb ... 2024-09-13T11:19:12.149 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package python3-natsort. 2024-09-13T11:19:12.150 INFO:teuthology.orchestra.run.smithi001.stdout:Unpacking lua5.1 (5.1.5-8.1build4) ... 2024-09-13T11:19:12.164 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2024-09-13T11:19:12.176 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../040-python3-natsort_8.0.2-1_all.deb ... 2024-09-13T11:19:12.216 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking python3-natsort (8.0.2-1) ... 2024-09-13T11:19:12.372 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package python3-jaraco.classes. 2024-09-13T11:19:12.398 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../033-python3-jaraco.classes_3.2.1-3_all.deb ... 2024-09-13T11:19:12.425 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package libreadline-dev:amd64. 2024-09-13T11:19:12.438 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2024-09-13T11:19:12.451 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../018-libreadline-dev_8.1.2-1_amd64.deb ... 2024-09-13T11:19:12.478 INFO:teuthology.orchestra.run.smithi001.stdout:Selecting previously unselected package lua-any. 2024-09-13T11:19:12.492 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking libreadline-dev:amd64 (8.1.2-1) ... 2024-09-13T11:19:12.505 INFO:teuthology.orchestra.run.smithi001.stdout:Preparing to unpack .../021-lua-any_27ubuntu1_all.deb ... 2024-09-13T11:19:12.519 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package python3-logutils. 2024-09-13T11:19:12.545 INFO:teuthology.orchestra.run.smithi001.stdout:Unpacking lua-any (27ubuntu1) ... 2024-09-13T11:19:12.546 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../041-python3-logutils_0.3.3-8_all.deb ... 2024-09-13T11:19:12.586 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking python3-logutils (0.3.3-8) ... 2024-09-13T11:19:12.758 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package python3-jaraco.text. 2024-09-13T11:19:12.778 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package liblua5.3-dev:amd64. 2024-09-13T11:19:12.780 INFO:teuthology.orchestra.run.smithi001.stdout:Selecting previously unselected package zip. 2024-09-13T11:19:12.784 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../034-python3-jaraco.text_3.6.0-2_all.deb ... 2024-09-13T11:19:12.793 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../019-liblua5.3-dev_5.3.6-1build1_amd64.deb ... 2024-09-13T11:19:12.796 INFO:teuthology.orchestra.run.smithi001.stdout:Preparing to unpack .../022-zip_3.0-12build2_amd64.deb ... 2024-09-13T11:19:12.824 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2024-09-13T11:19:12.828 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-09-13T11:19:12.830 INFO:teuthology.orchestra.run.smithi001.stdout:Unpacking zip (3.0-12build2) ... 2024-09-13T11:19:12.905 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package python3-markupsafe. 2024-09-13T11:19:12.932 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../042-python3-markupsafe_2.0.1-2build1_amd64.deb ... 2024-09-13T11:19:13.012 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking python3-markupsafe (2.0.1-2build1) ... 2024-09-13T11:19:13.127 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package python3-jaraco.collections. 2024-09-13T11:19:13.131 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package lua5.1. 2024-09-13T11:19:13.142 INFO:teuthology.orchestra.run.smithi001.stdout:Selecting previously unselected package unzip. 2024-09-13T11:19:13.154 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../035-python3-jaraco.collections_3.4.0-2_all.deb ... 2024-09-13T11:19:13.157 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../020-lua5.1_5.1.5-8.1build4_amd64.deb ... 2024-09-13T11:19:13.169 INFO:teuthology.orchestra.run.smithi001.stdout:Preparing to unpack .../023-unzip_6.0-26ubuntu3.2_amd64.deb ... 2024-09-13T11:19:13.194 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2024-09-13T11:19:13.197 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking lua5.1 (5.1.5-8.1build4) ... 2024-09-13T11:19:13.209 INFO:teuthology.orchestra.run.smithi001.stdout:Unpacking unzip (6.0-26ubuntu3.2) ... 2024-09-13T11:19:13.442 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package python3-mako. 2024-09-13T11:19:13.447 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package python3-tempora. 2024-09-13T11:19:13.462 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../036-python3-tempora_4.1.2-1_all.deb ... 2024-09-13T11:19:13.469 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../043-python3-mako_1.1.3+ds1-2ubuntu0.1_all.deb ... 2024-09-13T11:19:13.497 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking python3-tempora (4.1.2-1) ... 2024-09-13T11:19:13.510 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package lua-any. 2024-09-13T11:19:13.525 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-09-13T11:19:13.526 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../021-lua-any_27ubuntu1_all.deb ... 2024-09-13T11:19:13.569 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking lua-any (27ubuntu1) ... 2024-09-13T11:19:13.604 INFO:teuthology.orchestra.run.smithi001.stdout:Selecting previously unselected package luarocks. 2024-09-13T11:19:13.631 INFO:teuthology.orchestra.run.smithi001.stdout:Preparing to unpack .../024-luarocks_3.8.0+dfsg1-1_all.deb ... 2024-09-13T11:19:13.671 INFO:teuthology.orchestra.run.smithi001.stdout:Unpacking luarocks (3.8.0+dfsg1-1) ... 2024-09-13T11:19:13.792 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package python3-portend. 2024-09-13T11:19:13.797 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package zip. 2024-09-13T11:19:13.818 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../037-python3-portend_3.0.0-1_all.deb ... 2024-09-13T11:19:13.823 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../022-zip_3.0-12build2_amd64.deb ... 2024-09-13T11:19:13.853 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package python3-simplegeneric. 2024-09-13T11:19:13.855 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking zip (3.0-12build2) ... 2024-09-13T11:19:13.866 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking python3-portend (3.0.0-1) ... 2024-09-13T11:19:13.880 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../044-python3-simplegeneric_0.8.1-3_all.deb ... 2024-09-13T11:19:13.920 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2024-09-13T11:19:14.054 INFO:teuthology.orchestra.run.smithi001.stdout:Selecting previously unselected package librgw2. 2024-09-13T11:19:14.081 INFO:teuthology.orchestra.run.smithi001.stdout:Preparing to unpack .../025-librgw2_19.3.0-4697-gd58f7306-1jammy_amd64.deb ... 2024-09-13T11:19:14.120 INFO:teuthology.orchestra.run.smithi001.stdout:Unpacking librgw2 (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:19:14.166 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package unzip. 2024-09-13T11:19:14.192 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../023-unzip_6.0-26ubuntu3.2_amd64.deb ... 2024-09-13T11:19:14.224 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package python3-singledispatch. 2024-09-13T11:19:14.233 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking unzip (6.0-26ubuntu3.2) ... 2024-09-13T11:19:14.236 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package python3-zc.lockfile. 2024-09-13T11:19:14.249 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../045-python3-singledispatch_3.4.0.3-3_all.deb ... 2024-09-13T11:19:14.262 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../038-python3-zc.lockfile_2.0-1_all.deb ... 2024-09-13T11:19:14.289 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking python3-singledispatch (3.4.0.3-3) ... 2024-09-13T11:19:14.336 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2024-09-13T11:19:14.559 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package python3-webob. 2024-09-13T11:19:14.574 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../046-python3-webob_1%3a1.8.6-1.1ubuntu0.1_all.deb ... 2024-09-13T11:19:14.609 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking python3-webob (1:1.8.6-1.1ubuntu0.1) ... 2024-09-13T11:19:14.612 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package luarocks. 2024-09-13T11:19:14.638 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../024-luarocks_3.8.0+dfsg1-1_all.deb ... 2024-09-13T11:19:14.673 INFO:teuthology.orchestra.run.smithi001.stdout:Selecting previously unselected package python3-rgw. 2024-09-13T11:19:14.678 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking luarocks (3.8.0+dfsg1-1) ... 2024-09-13T11:19:14.689 INFO:teuthology.orchestra.run.smithi001.stdout:Preparing to unpack .../026-python3-rgw_19.3.0-4697-gd58f7306-1jammy_amd64.deb ... 2024-09-13T11:19:14.706 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package python3-cherrypy3. 2024-09-13T11:19:14.722 INFO:teuthology.orchestra.run.smithi001.stdout:Unpacking python3-rgw (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:19:14.732 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../039-python3-cherrypy3_18.6.1-4_all.deb ... 2024-09-13T11:19:14.781 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2024-09-13T11:19:14.895 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package python3-waitress. 2024-09-13T11:19:14.921 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../047-python3-waitress_1.4.4-1.1ubuntu1_all.deb ... 2024-09-13T11:19:14.957 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking python3-waitress (1.4.4-1.1ubuntu1) ... 2024-09-13T11:19:15.027 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package librgw2. 2024-09-13T11:19:15.045 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../025-librgw2_19.3.0-4697-gd58f7306-1jammy_amd64.deb ... 2024-09-13T11:19:15.075 INFO:teuthology.orchestra.run.smithi001.stdout:Selecting previously unselected package liboath0:amd64. 2024-09-13T11:19:15.077 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking librgw2 (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:19:15.102 INFO:teuthology.orchestra.run.smithi001.stdout:Preparing to unpack .../027-liboath0_2.6.7-3build1_amd64.deb ... 2024-09-13T11:19:15.142 INFO:teuthology.orchestra.run.smithi001.stdout:Unpacking liboath0:amd64 (2.6.7-3build1) ... 2024-09-13T11:19:15.143 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package python3-natsort. 2024-09-13T11:19:15.169 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../040-python3-natsort_8.0.2-1_all.deb ... 2024-09-13T11:19:15.209 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking python3-natsort (8.0.2-1) ... 2024-09-13T11:19:15.265 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package python3-tempita. 2024-09-13T11:19:15.291 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../048-python3-tempita_0.5.2-6ubuntu1_all.deb ... 2024-09-13T11:19:15.331 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking python3-tempita (0.5.2-6ubuntu1) ... 2024-09-13T11:19:15.453 INFO:teuthology.orchestra.run.smithi001.stdout:Selecting previously unselected package libradosstriper1. 2024-09-13T11:19:15.480 INFO:teuthology.orchestra.run.smithi001.stdout:Preparing to unpack .../028-libradosstriper1_19.3.0-4697-gd58f7306-1jammy_amd64.deb ... 2024-09-13T11:19:15.496 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package python3-logutils. 2024-09-13T11:19:15.512 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../041-python3-logutils_0.3.3-8_all.deb ... 2024-09-13T11:19:15.520 INFO:teuthology.orchestra.run.smithi001.stdout:Unpacking libradosstriper1 (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:19:15.546 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking python3-logutils (0.3.3-8) ... 2024-09-13T11:19:15.601 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package python3-paste. 2024-09-13T11:19:15.613 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package python3-rgw. 2024-09-13T11:19:15.628 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../049-python3-paste_3.5.0+dfsg1-1_all.deb ... 2024-09-13T11:19:15.628 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../026-python3-rgw_19.3.0-4697-gd58f7306-1jammy_amd64.deb ... 2024-09-13T11:19:15.667 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking python3-paste (3.5.0+dfsg1-1) ... 2024-09-13T11:19:15.671 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking python3-rgw (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:19:15.857 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package python3-markupsafe. 2024-09-13T11:19:15.865 INFO:teuthology.orchestra.run.smithi001.stdout:Selecting previously unselected package ceph-common. 2024-09-13T11:19:15.883 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../042-python3-markupsafe_2.0.1-2build1_amd64.deb ... 2024-09-13T11:19:15.891 INFO:teuthology.orchestra.run.smithi001.stdout:Preparing to unpack .../029-ceph-common_19.3.0-4697-gd58f7306-1jammy_amd64.deb ... 2024-09-13T11:19:15.923 INFO:teuthology.orchestra.run.smithi001.stdout:Unpacking ceph-common (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:19:15.924 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking python3-markupsafe (2.0.1-2build1) ... 2024-09-13T11:19:15.987 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2024-09-13T11:19:16.003 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../050-python-pastedeploy-tpl_2.1.1-1_all.deb ... 2024-09-13T11:19:16.007 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package liboath0:amd64. 2024-09-13T11:19:16.034 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../027-liboath0_2.6.7-3build1_amd64.deb ... 2024-09-13T11:19:16.037 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking python-pastedeploy-tpl (2.1.1-1) ... 2024-09-13T11:19:16.074 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking liboath0:amd64 (2.6.7-3build1) ... 2024-09-13T11:19:16.227 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package python3-mako. 2024-09-13T11:19:16.253 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../043-python3-mako_1.1.3+ds1-2ubuntu0.1_all.deb ... 2024-09-13T11:19:16.293 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-09-13T11:19:16.306 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package python3-pastedeploy. 2024-09-13T11:19:16.333 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../051-python3-pastedeploy_2.1.1-1_all.deb ... 2024-09-13T11:19:16.373 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking python3-pastedeploy (2.1.1-1) ... 2024-09-13T11:19:16.377 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package libradosstriper1. 2024-09-13T11:19:16.403 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../028-libradosstriper1_19.3.0-4697-gd58f7306-1jammy_amd64.deb ... 2024-09-13T11:19:16.435 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking libradosstriper1 (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:19:16.630 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package python3-simplegeneric. 2024-09-13T11:19:16.657 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../044-python3-simplegeneric_0.8.1-3_all.deb ... 2024-09-13T11:19:16.676 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package python3-webtest. 2024-09-13T11:19:16.697 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2024-09-13T11:19:16.703 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../052-python3-webtest_2.0.35-1_all.deb ... 2024-09-13T11:19:16.743 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking python3-webtest (2.0.35-1) ... 2024-09-13T11:19:16.838 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package ceph-common. 2024-09-13T11:19:16.864 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../029-ceph-common_19.3.0-4697-gd58f7306-1jammy_amd64.deb ... 2024-09-13T11:19:16.904 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking ceph-common (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:19:16.958 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package python3-singledispatch. 2024-09-13T11:19:16.973 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../045-python3-singledispatch_3.4.0.3-3_all.deb ... 2024-09-13T11:19:17.016 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking python3-singledispatch (3.4.0.3-3) ... 2024-09-13T11:19:17.054 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package python3-pecan. 2024-09-13T11:19:17.081 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../053-python3-pecan_1.3.3-4ubuntu2_all.deb ... 2024-09-13T11:19:17.121 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking python3-pecan (1.3.3-4ubuntu2) ... 2024-09-13T11:19:17.319 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package python3-webob. 2024-09-13T11:19:17.345 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../046-python3-webob_1%3a1.8.6-1.1ubuntu0.1_all.deb ... 2024-09-13T11:19:17.385 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking python3-webob (1:1.8.6-1.1ubuntu0.1) ... 2024-09-13T11:19:17.492 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package python3-certifi. 2024-09-13T11:19:17.519 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../054-python3-certifi_2020.6.20-1_all.deb ... 2024-09-13T11:19:17.537 INFO:teuthology.orchestra.run.smithi001.stdout:Selecting previously unselected package ceph-base. 2024-09-13T11:19:17.559 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking python3-certifi (2020.6.20-1) ... 2024-09-13T11:19:17.563 INFO:teuthology.orchestra.run.smithi001.stdout:Preparing to unpack .../030-ceph-base_19.3.0-4697-gd58f7306-1jammy_amd64.deb ... 2024-09-13T11:19:17.612 INFO:teuthology.orchestra.run.smithi001.stdout:Unpacking ceph-base (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:19:17.845 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package python3-idna. 2024-09-13T11:19:17.872 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../055-python3-idna_3.3-1ubuntu0.1_all.deb ... 2024-09-13T11:19:17.903 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking python3-idna (3.3-1ubuntu0.1) ... 2024-09-13T11:19:17.956 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package python3-waitress. 2024-09-13T11:19:17.982 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../047-python3-waitress_1.4.4-1.1ubuntu1_all.deb ... 2024-09-13T11:19:18.025 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking python3-waitress (1.4.4-1.1ubuntu1) ... 2024-09-13T11:19:18.216 INFO:teuthology.orchestra.run.smithi001.stdout:Selecting previously unselected package python3-jaraco.functools. 2024-09-13T11:19:18.223 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package python3-urllib3. 2024-09-13T11:19:18.242 INFO:teuthology.orchestra.run.smithi001.stdout:Preparing to unpack .../031-python3-jaraco.functools_3.4.0-2_all.deb ... 2024-09-13T11:19:18.250 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../056-python3-urllib3_1.26.5-1~exp1ubuntu0.1_all.deb ... 2024-09-13T11:19:18.282 INFO:teuthology.orchestra.run.smithi001.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2024-09-13T11:19:18.289 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-09-13T11:19:18.334 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package python3-tempita. 2024-09-13T11:19:18.360 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../048-python3-tempita_0.5.2-6ubuntu1_all.deb ... 2024-09-13T11:19:18.400 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking python3-tempita (0.5.2-6ubuntu1) ... 2024-09-13T11:19:18.418 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package ceph-base. 2024-09-13T11:19:18.444 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../030-ceph-base_19.3.0-4697-gd58f7306-1jammy_amd64.deb ... 2024-09-13T11:19:18.585 INFO:teuthology.orchestra.run.smithi001.stdout:Selecting previously unselected package python3-cheroot. 2024-09-13T11:19:18.614 INFO:teuthology.orchestra.run.smithi001.stdout:Preparing to unpack .../032-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2024-09-13T11:19:18.636 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking ceph-base (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:19:18.801 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package python3-requests. 2024-09-13T11:19:18.829 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../057-python3-requests_2.25.1+dfsg-2ubuntu0.1_all.deb ... 2024-09-13T11:19:19.026 INFO:teuthology.orchestra.run.smithi001.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-09-13T11:19:19.129 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package python3-paste. 2024-09-13T11:19:19.155 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../049-python3-paste_3.5.0+dfsg1-1_all.deb ... 2024-09-13T11:19:19.226 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-09-13T11:19:19.487 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking python3-paste (3.5.0+dfsg1-1) ... 2024-09-13T11:19:19.738 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package python3-werkzeug. 2024-09-13T11:19:19.765 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../058-python3-werkzeug_2.0.2+dfsg1-1ubuntu0.22.04.2_all.deb ... 2024-09-13T11:19:19.799 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2024-09-13T11:19:19.804 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.2) ... 2024-09-13T11:19:19.825 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../050-python-pastedeploy-tpl_2.1.1-1_all.deb ... 2024-09-13T11:19:19.855 INFO:teuthology.orchestra.run.smithi001.stdout:Selecting previously unselected package python3-jaraco.classes. 2024-09-13T11:19:19.865 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking python-pastedeploy-tpl (2.1.1-1) ... 2024-09-13T11:19:19.882 INFO:teuthology.orchestra.run.smithi001.stdout:Preparing to unpack .../033-python3-jaraco.classes_3.2.1-3_all.deb ... 2024-09-13T11:19:19.913 INFO:teuthology.orchestra.run.smithi001.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2024-09-13T11:19:19.914 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package python3-jaraco.functools. 2024-09-13T11:19:19.940 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../031-python3-jaraco.functools_3.4.0-2_all.deb ... 2024-09-13T11:19:19.980 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2024-09-13T11:19:20.099 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2024-09-13T11:19:20.118 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package python3-pastedeploy. 2024-09-13T11:19:20.126 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../059-ceph-mgr-modules-core_19.3.0-4697-gd58f7306-1jammy_all.deb ... 2024-09-13T11:19:20.141 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../051-python3-pastedeploy_2.1.1-1_all.deb ... 2024-09-13T11:19:20.166 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking ceph-mgr-modules-core (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:19:20.176 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking python3-pastedeploy (2.1.1-1) ... 2024-09-13T11:19:20.216 INFO:teuthology.orchestra.run.smithi001.stdout:Selecting previously unselected package python3-jaraco.text. 2024-09-13T11:19:20.243 INFO:teuthology.orchestra.run.smithi001.stdout:Preparing to unpack .../034-python3-jaraco.text_3.6.0-2_all.deb ... 2024-09-13T11:19:20.282 INFO:teuthology.orchestra.run.smithi001.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2024-09-13T11:19:20.283 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package python3-cheroot. 2024-09-13T11:19:20.309 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../032-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2024-09-13T11:19:20.349 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-09-13T11:19:20.480 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package python3-webtest. 2024-09-13T11:19:20.506 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../052-python3-webtest_2.0.35-1_all.deb ... 2024-09-13T11:19:20.546 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking python3-webtest (2.0.35-1) ... 2024-09-13T11:19:20.562 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2024-09-13T11:19:20.585 INFO:teuthology.orchestra.run.smithi001.stdout:Selecting previously unselected package python3-jaraco.collections. 2024-09-13T11:19:20.589 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../060-libsqlite3-mod-ceph_19.3.0-4697-gd58f7306-1jammy_amd64.deb ... 2024-09-13T11:19:20.612 INFO:teuthology.orchestra.run.smithi001.stdout:Preparing to unpack .../035-python3-jaraco.collections_3.4.0-2_all.deb ... 2024-09-13T11:19:20.628 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking libsqlite3-mod-ceph (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:19:20.652 INFO:teuthology.orchestra.run.smithi001.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2024-09-13T11:19:20.669 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package python3-jaraco.classes. 2024-09-13T11:19:20.696 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../033-python3-jaraco.classes_3.2.1-3_all.deb ... 2024-09-13T11:19:20.736 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2024-09-13T11:19:20.858 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package python3-pecan. 2024-09-13T11:19:20.884 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../053-python3-pecan_1.3.3-4ubuntu2_all.deb ... 2024-09-13T11:19:20.906 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package python3-bcrypt. 2024-09-13T11:19:20.924 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking python3-pecan (1.3.3-4ubuntu2) ... 2024-09-13T11:19:20.930 INFO:teuthology.orchestra.run.smithi001.stdout:Selecting previously unselected package python3-tempora. 2024-09-13T11:19:20.933 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../061-python3-bcrypt_3.2.0-1build1_amd64.deb ... 2024-09-13T11:19:20.957 INFO:teuthology.orchestra.run.smithi001.stdout:Preparing to unpack .../036-python3-tempora_4.1.2-1_all.deb ... 2024-09-13T11:19:20.964 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking python3-bcrypt (3.2.0-1build1) ... 2024-09-13T11:19:20.996 INFO:teuthology.orchestra.run.smithi001.stdout:Unpacking python3-tempora (4.1.2-1) ... 2024-09-13T11:19:21.013 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package python3-jaraco.text. 2024-09-13T11:19:21.040 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../034-python3-jaraco.text_3.6.0-2_all.deb ... 2024-09-13T11:19:21.080 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2024-09-13T11:19:21.266 INFO:teuthology.orchestra.run.smithi001.stdout:Selecting previously unselected package python3-portend. 2024-09-13T11:19:21.268 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package python3-openssl. 2024-09-13T11:19:21.279 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package python3-certifi. 2024-09-13T11:19:21.293 INFO:teuthology.orchestra.run.smithi001.stdout:Preparing to unpack .../037-python3-portend_3.0.0-1_all.deb ... 2024-09-13T11:19:21.294 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../062-python3-openssl_21.0.0-1_all.deb ... 2024-09-13T11:19:21.306 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../054-python3-certifi_2020.6.20-1_all.deb ... 2024-09-13T11:19:21.332 INFO:teuthology.orchestra.run.smithi001.stdout:Unpacking python3-portend (3.0.0-1) ... 2024-09-13T11:19:21.334 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking python3-openssl (21.0.0-1) ... 2024-09-13T11:19:21.345 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking python3-certifi (2020.6.20-1) ... 2024-09-13T11:19:21.366 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package python3-jaraco.collections. 2024-09-13T11:19:21.392 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../035-python3-jaraco.collections_3.4.0-2_all.deb ... 2024-09-13T11:19:21.433 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2024-09-13T11:19:21.618 INFO:teuthology.orchestra.run.smithi001.stdout:Selecting previously unselected package python3-zc.lockfile. 2024-09-13T11:19:21.645 INFO:teuthology.orchestra.run.smithi001.stdout:Preparing to unpack .../038-python3-zc.lockfile_2.0-1_all.deb ... 2024-09-13T11:19:21.670 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package ceph-mgr. 2024-09-13T11:19:21.682 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package python3-idna. 2024-09-13T11:19:21.685 INFO:teuthology.orchestra.run.smithi001.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2024-09-13T11:19:21.694 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package python3-tempora. 2024-09-13T11:19:21.697 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../063-ceph-mgr_19.3.0-4697-gd58f7306-1jammy_amd64.deb ... 2024-09-13T11:19:21.708 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../055-python3-idna_3.3-1ubuntu0.1_all.deb ... 2024-09-13T11:19:21.720 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../036-python3-tempora_4.1.2-1_all.deb ... 2024-09-13T11:19:21.737 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking ceph-mgr (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:19:21.748 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking python3-idna (3.3-1ubuntu0.1) ... 2024-09-13T11:19:21.760 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking python3-tempora (4.1.2-1) ... 2024-09-13T11:19:21.988 INFO:teuthology.orchestra.run.smithi001.stdout:Selecting previously unselected package python3-cherrypy3. 2024-09-13T11:19:22.015 INFO:teuthology.orchestra.run.smithi001.stdout:Preparing to unpack .../039-python3-cherrypy3_18.6.1-4_all.deb ... 2024-09-13T11:19:22.035 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package python3-urllib3. 2024-09-13T11:19:22.054 INFO:teuthology.orchestra.run.smithi001.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2024-09-13T11:19:22.061 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../056-python3-urllib3_1.26.5-1~exp1ubuntu0.1_all.deb ... 2024-09-13T11:19:22.063 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package python3-portend. 2024-09-13T11:19:22.089 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../037-python3-portend_3.0.0-1_all.deb ... 2024-09-13T11:19:22.101 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-09-13T11:19:22.123 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package ceph-mon. 2024-09-13T11:19:22.130 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking python3-portend (3.0.0-1) ... 2024-09-13T11:19:22.150 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../064-ceph-mon_19.3.0-4697-gd58f7306-1jammy_amd64.deb ... 2024-09-13T11:19:22.198 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking ceph-mon (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:19:22.400 INFO:teuthology.orchestra.run.smithi001.stdout:Selecting previously unselected package python3-natsort. 2024-09-13T11:19:22.422 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package python3-requests. 2024-09-13T11:19:22.426 INFO:teuthology.orchestra.run.smithi001.stdout:Preparing to unpack .../040-python3-natsort_8.0.2-1_all.deb ... 2024-09-13T11:19:22.448 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../057-python3-requests_2.25.1+dfsg-2ubuntu0.1_all.deb ... 2024-09-13T11:19:22.457 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package python3-zc.lockfile. 2024-09-13T11:19:22.466 INFO:teuthology.orchestra.run.smithi001.stdout:Unpacking python3-natsort (8.0.2-1) ... 2024-09-13T11:19:22.484 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../038-python3-zc.lockfile_2.0-1_all.deb ... 2024-09-13T11:19:22.488 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-09-13T11:19:22.524 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2024-09-13T11:19:22.777 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package ceph-osd. 2024-09-13T11:19:22.799 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package python3-werkzeug. 2024-09-13T11:19:22.803 INFO:teuthology.orchestra.run.smithi001.stdout:Selecting previously unselected package python3-logutils. 2024-09-13T11:19:22.803 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../065-ceph-osd_19.3.0-4697-gd58f7306-1jammy_amd64.deb ... 2024-09-13T11:19:22.810 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package python3-cherrypy3. 2024-09-13T11:19:22.826 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../058-python3-werkzeug_2.0.2+dfsg1-1ubuntu0.22.04.2_all.deb ... 2024-09-13T11:19:22.829 INFO:teuthology.orchestra.run.smithi001.stdout:Preparing to unpack .../041-python3-logutils_0.3.3-8_all.deb ... 2024-09-13T11:19:22.836 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../039-python3-cherrypy3_18.6.1-4_all.deb ... 2024-09-13T11:19:22.843 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking ceph-osd (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:19:22.866 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.2) ... 2024-09-13T11:19:22.868 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2024-09-13T11:19:22.869 INFO:teuthology.orchestra.run.smithi001.stdout:Unpacking python3-logutils (0.3.3-8) ... 2024-09-13T11:19:23.172 INFO:teuthology.orchestra.run.smithi001.stdout:Selecting previously unselected package python3-markupsafe. 2024-09-13T11:19:23.189 INFO:teuthology.orchestra.run.smithi001.stdout:Preparing to unpack .../042-python3-markupsafe_2.0.1-2build1_amd64.deb ... 2024-09-13T11:19:23.197 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package python3-natsort. 2024-09-13T11:19:23.224 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../040-python3-natsort_8.0.2-1_all.deb ... 2024-09-13T11:19:23.230 INFO:teuthology.orchestra.run.smithi001.stdout:Unpacking python3-markupsafe (2.0.1-2build1) ... 2024-09-13T11:19:23.263 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking python3-natsort (8.0.2-1) ... 2024-09-13T11:19:23.311 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2024-09-13T11:19:23.326 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../059-ceph-mgr-modules-core_19.3.0-4697-gd58f7306-1jammy_all.deb ... 2024-09-13T11:19:23.369 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking ceph-mgr-modules-core (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:19:23.508 INFO:teuthology.orchestra.run.smithi001.stdout:Selecting previously unselected package python3-mako. 2024-09-13T11:19:23.516 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package python3-logutils. 2024-09-13T11:19:23.535 INFO:teuthology.orchestra.run.smithi001.stdout:Preparing to unpack .../043-python3-mako_1.1.3+ds1-2ubuntu0.1_all.deb ... 2024-09-13T11:19:23.542 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../041-python3-logutils_0.3.3-8_all.deb ... 2024-09-13T11:19:23.574 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking python3-logutils (0.3.3-8) ... 2024-09-13T11:19:23.575 INFO:teuthology.orchestra.run.smithi001.stdout:Unpacking python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-09-13T11:19:23.750 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2024-09-13T11:19:23.776 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../060-libsqlite3-mod-ceph_19.3.0-4697-gd58f7306-1jammy_amd64.deb ... 2024-09-13T11:19:23.807 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking libsqlite3-mod-ceph (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:19:23.861 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package python3-markupsafe. 2024-09-13T11:19:23.878 INFO:teuthology.orchestra.run.smithi001.stdout:Selecting previously unselected package python3-simplegeneric. 2024-09-13T11:19:23.887 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../042-python3-markupsafe_2.0.1-2build1_amd64.deb ... 2024-09-13T11:19:23.905 INFO:teuthology.orchestra.run.smithi001.stdout:Preparing to unpack .../044-python3-simplegeneric_0.8.1-3_all.deb ... 2024-09-13T11:19:23.927 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking python3-markupsafe (2.0.1-2build1) ... 2024-09-13T11:19:23.930 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package ceph. 2024-09-13T11:19:23.936 INFO:teuthology.orchestra.run.smithi001.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2024-09-13T11:19:23.945 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../066-ceph_19.3.0-4697-gd58f7306-1jammy_amd64.deb ... 2024-09-13T11:19:23.988 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking ceph (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:19:24.119 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package python3-bcrypt. 2024-09-13T11:19:24.134 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../061-python3-bcrypt_3.2.0-1build1_amd64.deb ... 2024-09-13T11:19:24.177 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking python3-bcrypt (3.2.0-1build1) ... 2024-09-13T11:19:24.247 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package python3-mako. 2024-09-13T11:19:24.247 INFO:teuthology.orchestra.run.smithi001.stdout:Selecting previously unselected package python3-singledispatch. 2024-09-13T11:19:24.273 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../043-python3-mako_1.1.3+ds1-2ubuntu0.1_all.deb ... 2024-09-13T11:19:24.274 INFO:teuthology.orchestra.run.smithi001.stdout:Preparing to unpack .../045-python3-singledispatch_3.4.0.3-3_all.deb ... 2024-09-13T11:19:24.283 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package ceph-fuse. 2024-09-13T11:19:24.309 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../067-ceph-fuse_19.3.0-4697-gd58f7306-1jammy_amd64.deb ... 2024-09-13T11:19:24.313 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-09-13T11:19:24.314 INFO:teuthology.orchestra.run.smithi001.stdout:Unpacking python3-singledispatch (3.4.0.3-3) ... 2024-09-13T11:19:24.349 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking ceph-fuse (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:19:24.447 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package python3-openssl. 2024-09-13T11:19:24.473 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../062-python3-openssl_21.0.0-1_all.deb ... 2024-09-13T11:19:24.513 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking python3-openssl (21.0.0-1) ... 2024-09-13T11:19:24.617 INFO:teuthology.orchestra.run.smithi001.stdout:Selecting previously unselected package python3-webob. 2024-09-13T11:19:24.641 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package python3-simplegeneric. 2024-09-13T11:19:24.643 INFO:teuthology.orchestra.run.smithi001.stdout:Preparing to unpack .../046-python3-webob_1%3a1.8.6-1.1ubuntu0.1_all.deb ... 2024-09-13T11:19:24.667 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../044-python3-simplegeneric_0.8.1-3_all.deb ... 2024-09-13T11:19:24.683 INFO:teuthology.orchestra.run.smithi001.stdout:Unpacking python3-webob (1:1.8.6-1.1ubuntu0.1) ... 2024-09-13T11:19:24.694 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package ceph-mds. 2024-09-13T11:19:24.708 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2024-09-13T11:19:24.721 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../068-ceph-mds_19.3.0-4697-gd58f7306-1jammy_amd64.deb ... 2024-09-13T11:19:24.761 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking ceph-mds (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:19:24.825 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package ceph-mgr. 2024-09-13T11:19:24.851 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../063-ceph-mgr_19.3.0-4697-gd58f7306-1jammy_amd64.deb ... 2024-09-13T11:19:24.891 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking ceph-mgr (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:19:24.977 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package python3-singledispatch. 2024-09-13T11:19:25.004 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../045-python3-singledispatch_3.4.0.3-3_all.deb ... 2024-09-13T11:19:25.035 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking python3-singledispatch (3.4.0.3-3) ... 2024-09-13T11:19:25.036 INFO:teuthology.orchestra.run.smithi001.stdout:Selecting previously unselected package python3-waitress. 2024-09-13T11:19:25.063 INFO:teuthology.orchestra.run.smithi001.stdout:Preparing to unpack .../047-python3-waitress_1.4.4-1.1ubuntu1_all.deb ... 2024-09-13T11:19:25.131 INFO:teuthology.orchestra.run.smithi001.stdout:Unpacking python3-waitress (1.4.4-1.1ubuntu1) ... 2024-09-13T11:19:25.181 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package cephadm. 2024-09-13T11:19:25.207 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../069-cephadm_19.3.0-4697-gd58f7306-1jammy_amd64.deb ... 2024-09-13T11:19:25.247 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking cephadm (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:19:25.311 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package ceph-mon. 2024-09-13T11:19:25.330 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package python3-webob. 2024-09-13T11:19:25.338 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../064-ceph-mon_19.3.0-4697-gd58f7306-1jammy_amd64.deb ... 2024-09-13T11:19:25.357 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../046-python3-webob_1%3a1.8.6-1.1ubuntu0.1_all.deb ... 2024-09-13T11:19:25.377 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking ceph-mon (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:19:25.396 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking python3-webob (1:1.8.6-1.1ubuntu0.1) ... 2024-09-13T11:19:25.447 INFO:teuthology.orchestra.run.smithi001.stdout:Selecting previously unselected package python3-tempita. 2024-09-13T11:19:25.475 INFO:teuthology.orchestra.run.smithi001.stdout:Preparing to unpack .../048-python3-tempita_0.5.2-6ubuntu1_all.deb ... 2024-09-13T11:19:25.514 INFO:teuthology.orchestra.run.smithi001.stdout:Unpacking python3-tempita (0.5.2-6ubuntu1) ... 2024-09-13T11:19:25.567 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package python3-asyncssh. 2024-09-13T11:19:25.594 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../070-python3-asyncssh_2.5.0-1_all.deb ... 2024-09-13T11:19:25.633 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking python3-asyncssh (2.5.0-1) ... 2024-09-13T11:19:25.733 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package python3-waitress. 2024-09-13T11:19:25.759 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../047-python3-waitress_1.4.4-1.1ubuntu1_all.deb ... 2024-09-13T11:19:25.792 INFO:teuthology.orchestra.run.smithi001.stdout:Selecting previously unselected package python3-paste. 2024-09-13T11:19:25.802 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking python3-waitress (1.4.4-1.1ubuntu1) ... 2024-09-13T11:19:25.819 INFO:teuthology.orchestra.run.smithi001.stdout:Preparing to unpack .../049-python3-paste_3.5.0+dfsg1-1_all.deb ... 2024-09-13T11:19:25.850 INFO:teuthology.orchestra.run.smithi001.stdout:Unpacking python3-paste (3.5.0+dfsg1-1) ... 2024-09-13T11:19:25.948 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package ceph-osd. 2024-09-13T11:19:25.952 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package python-babel-localedata. 2024-09-13T11:19:25.975 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../065-ceph-osd_19.3.0-4697-gd58f7306-1jammy_amd64.deb ... 2024-09-13T11:19:25.980 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../071-python-babel-localedata_2.8.0+dfsg.1-7_all.deb ... 2024-09-13T11:19:26.014 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking ceph-osd (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:19:26.019 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-09-13T11:19:26.119 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package python3-tempita. 2024-09-13T11:19:26.145 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../048-python3-tempita_0.5.2-6ubuntu1_all.deb ... 2024-09-13T11:19:26.178 INFO:teuthology.orchestra.run.smithi001.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2024-09-13T11:19:26.185 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking python3-tempita (0.5.2-6ubuntu1) ... 2024-09-13T11:19:26.206 INFO:teuthology.orchestra.run.smithi001.stdout:Preparing to unpack .../050-python-pastedeploy-tpl_2.1.1-1_all.deb ... 2024-09-13T11:19:26.245 INFO:teuthology.orchestra.run.smithi001.stdout:Unpacking python-pastedeploy-tpl (2.1.1-1) ... 2024-09-13T11:19:26.506 INFO:teuthology.orchestra.run.smithi001.stdout:Selecting previously unselected package python3-pastedeploy. 2024-09-13T11:19:26.513 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package python3-paste. 2024-09-13T11:19:26.533 INFO:teuthology.orchestra.run.smithi001.stdout:Preparing to unpack .../051-python3-pastedeploy_2.1.1-1_all.deb ... 2024-09-13T11:19:26.540 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../049-python3-paste_3.5.0+dfsg1-1_all.deb ... 2024-09-13T11:19:26.572 INFO:teuthology.orchestra.run.smithi001.stdout:Unpacking python3-pastedeploy (2.1.1-1) ... 2024-09-13T11:19:26.579 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking python3-paste (3.5.0+dfsg1-1) ... 2024-09-13T11:19:26.875 INFO:teuthology.orchestra.run.smithi001.stdout:Selecting previously unselected package python3-webtest. 2024-09-13T11:19:26.891 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2024-09-13T11:19:26.903 INFO:teuthology.orchestra.run.smithi001.stdout:Preparing to unpack .../052-python3-webtest_2.0.35-1_all.deb ... 2024-09-13T11:19:26.906 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../050-python-pastedeploy-tpl_2.1.1-1_all.deb ... 2024-09-13T11:19:26.942 INFO:teuthology.orchestra.run.smithi001.stdout:Unpacking python3-webtest (2.0.35-1) ... 2024-09-13T11:19:26.949 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking python-pastedeploy-tpl (2.1.1-1) ... 2024-09-13T11:19:27.049 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package python3-babel. 2024-09-13T11:19:27.067 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package ceph. 2024-09-13T11:19:27.076 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../072-python3-babel_2.8.0+dfsg.1-7_all.deb ... 2024-09-13T11:19:27.083 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../066-ceph_19.3.0-4697-gd58f7306-1jammy_amd64.deb ... 2024-09-13T11:19:27.107 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking python3-babel (2.8.0+dfsg.1-7) ... 2024-09-13T11:19:27.126 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking ceph (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:19:27.252 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package python3-pastedeploy. 2024-09-13T11:19:27.262 INFO:teuthology.orchestra.run.smithi001.stdout:Selecting previously unselected package python3-pecan. 2024-09-13T11:19:27.279 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../051-python3-pastedeploy_2.1.1-1_all.deb ... 2024-09-13T11:19:27.289 INFO:teuthology.orchestra.run.smithi001.stdout:Preparing to unpack .../053-python3-pecan_1.3.3-4ubuntu2_all.deb ... 2024-09-13T11:19:27.318 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking python3-pastedeploy (2.1.1-1) ... 2024-09-13T11:19:27.328 INFO:teuthology.orchestra.run.smithi001.stdout:Unpacking python3-pecan (1.3.3-4ubuntu2) ... 2024-09-13T11:19:27.393 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package python3-jinja2. 2024-09-13T11:19:27.420 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../073-python3-jinja2_3.0.3-1ubuntu0.2_all.deb ... 2024-09-13T11:19:27.451 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking python3-jinja2 (3.0.3-1ubuntu0.2) ... 2024-09-13T11:19:27.454 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package ceph-fuse. 2024-09-13T11:19:27.480 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../067-ceph-fuse_19.3.0-4697-gd58f7306-1jammy_amd64.deb ... 2024-09-13T11:19:27.520 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking ceph-fuse (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:19:27.622 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package python3-webtest. 2024-09-13T11:19:27.648 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../052-python3-webtest_2.0.35-1_all.deb ... 2024-09-13T11:19:27.688 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking python3-webtest (2.0.35-1) ... 2024-09-13T11:19:27.716 INFO:teuthology.orchestra.run.smithi001.stdout:Selecting previously unselected package python3-certifi. 2024-09-13T11:19:27.743 INFO:teuthology.orchestra.run.smithi001.stdout:Preparing to unpack .../054-python3-certifi_2020.6.20-1_all.deb ... 2024-09-13T11:19:27.763 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2024-09-13T11:19:27.782 INFO:teuthology.orchestra.run.smithi001.stdout:Unpacking python3-certifi (2020.6.20-1) ... 2024-09-13T11:19:27.790 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../074-ceph-mgr-cephadm_19.3.0-4697-gd58f7306-1jammy_all.deb ... 2024-09-13T11:19:27.821 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking ceph-mgr-cephadm (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:19:27.882 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package ceph-mds. 2024-09-13T11:19:27.909 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../068-ceph-mds_19.3.0-4697-gd58f7306-1jammy_amd64.deb ... 2024-09-13T11:19:27.949 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking ceph-mds (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:19:27.949 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package python3-pecan. 2024-09-13T11:19:27.965 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../053-python3-pecan_1.3.3-4ubuntu2_all.deb ... 2024-09-13T11:19:27.999 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking python3-pecan (1.3.3-4ubuntu2) ... 2024-09-13T11:19:28.085 INFO:teuthology.orchestra.run.smithi001.stdout:Selecting previously unselected package python3-idna. 2024-09-13T11:19:28.113 INFO:teuthology.orchestra.run.smithi001.stdout:Preparing to unpack .../055-python3-idna_3.3-1ubuntu0.1_all.deb ... 2024-09-13T11:19:28.152 INFO:teuthology.orchestra.run.smithi001.stdout:Unpacking python3-idna (3.3-1ubuntu0.1) ... 2024-09-13T11:19:28.183 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package python3-repoze.lru. 2024-09-13T11:19:28.210 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../075-python3-repoze.lru_0.7-2_all.deb ... 2024-09-13T11:19:28.241 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2024-09-13T11:19:28.337 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package python3-certifi. 2024-09-13T11:19:28.363 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../054-python3-certifi_2020.6.20-1_all.deb ... 2024-09-13T11:19:28.403 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking python3-certifi (2020.6.20-1) ... 2024-09-13T11:19:28.426 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package cephadm. 2024-09-13T11:19:28.438 INFO:teuthology.orchestra.run.smithi001.stdout:Selecting previously unselected package python3-urllib3. 2024-09-13T11:19:28.453 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../069-cephadm_19.3.0-4697-gd58f7306-1jammy_amd64.deb ... 2024-09-13T11:19:28.466 INFO:teuthology.orchestra.run.smithi001.stdout:Preparing to unpack .../056-python3-urllib3_1.26.5-1~exp1ubuntu0.1_all.deb ... 2024-09-13T11:19:28.493 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking cephadm (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:19:28.505 INFO:teuthology.orchestra.run.smithi001.stdout:Unpacking python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-09-13T11:19:28.544 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package python3-routes. 2024-09-13T11:19:28.571 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../076-python3-routes_2.5.1-1ubuntu1_all.deb ... 2024-09-13T11:19:28.610 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2024-09-13T11:19:28.715 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package python3-idna. 2024-09-13T11:19:28.741 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../055-python3-idna_3.3-1ubuntu0.1_all.deb ... 2024-09-13T11:19:28.781 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking python3-idna (3.3-1ubuntu0.1) ... 2024-09-13T11:19:28.808 INFO:teuthology.orchestra.run.smithi001.stdout:Selecting previously unselected package python3-requests. 2024-09-13T11:19:28.836 INFO:teuthology.orchestra.run.smithi001.stdout:Preparing to unpack .../057-python3-requests_2.25.1+dfsg-2ubuntu0.1_all.deb ... 2024-09-13T11:19:28.846 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package python3-asyncssh. 2024-09-13T11:19:28.873 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../070-python3-asyncssh_2.5.0-1_all.deb ... 2024-09-13T11:19:28.874 INFO:teuthology.orchestra.run.smithi001.stdout:Unpacking python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-09-13T11:19:28.912 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking python3-asyncssh (2.5.0-1) ... 2024-09-13T11:19:28.931 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2024-09-13T11:19:28.958 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../077-ceph-mgr-dashboard_19.3.0-4697-gd58f7306-1jammy_all.deb ... 2024-09-13T11:19:28.997 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking ceph-mgr-dashboard (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:19:29.017 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package python3-urllib3. 2024-09-13T11:19:29.032 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../056-python3-urllib3_1.26.5-1~exp1ubuntu0.1_all.deb ... 2024-09-13T11:19:29.067 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-09-13T11:19:29.207 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package python-babel-localedata. 2024-09-13T11:19:29.234 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../071-python-babel-localedata_2.8.0+dfsg.1-7_all.deb ... 2024-09-13T11:19:29.274 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-09-13T11:19:29.328 INFO:teuthology.orchestra.run.smithi001.stdout:Selecting previously unselected package python3-werkzeug. 2024-09-13T11:19:29.337 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package python3-requests. 2024-09-13T11:19:29.355 INFO:teuthology.orchestra.run.smithi001.stdout:Preparing to unpack .../058-python3-werkzeug_2.0.2+dfsg1-1ubuntu0.22.04.2_all.deb ... 2024-09-13T11:19:29.363 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../057-python3-requests_2.25.1+dfsg-2ubuntu0.1_all.deb ... 2024-09-13T11:19:29.394 INFO:teuthology.orchestra.run.smithi001.stdout:Unpacking python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.2) ... 2024-09-13T11:19:29.395 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-09-13T11:19:29.681 INFO:teuthology.orchestra.run.smithi001.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2024-09-13T11:19:29.689 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package python3-werkzeug. 2024-09-13T11:19:29.708 INFO:teuthology.orchestra.run.smithi001.stdout:Preparing to unpack .../059-ceph-mgr-modules-core_19.3.0-4697-gd58f7306-1jammy_all.deb ... 2024-09-13T11:19:29.716 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../058-python3-werkzeug_2.0.2+dfsg1-1ubuntu0.22.04.2_all.deb ... 2024-09-13T11:19:29.747 INFO:teuthology.orchestra.run.smithi001.stdout:Unpacking ceph-mgr-modules-core (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:19:29.756 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.2) ... 2024-09-13T11:19:30.025 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2024-09-13T11:19:30.041 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../059-ceph-mgr-modules-core_19.3.0-4697-gd58f7306-1jammy_all.deb ... 2024-09-13T11:19:30.075 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking ceph-mgr-modules-core (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:19:30.152 INFO:teuthology.orchestra.run.smithi001.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2024-09-13T11:19:30.179 INFO:teuthology.orchestra.run.smithi001.stdout:Preparing to unpack .../060-libsqlite3-mod-ceph_19.3.0-4697-gd58f7306-1jammy_amd64.deb ... 2024-09-13T11:19:30.218 INFO:teuthology.orchestra.run.smithi001.stdout:Unpacking libsqlite3-mod-ceph (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:19:30.312 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package python3-babel. 2024-09-13T11:19:30.339 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../072-python3-babel_2.8.0+dfsg.1-7_all.deb ... 2024-09-13T11:19:30.379 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking python3-babel (2.8.0+dfsg.1-7) ... 2024-09-13T11:19:30.455 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2024-09-13T11:19:30.482 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../060-libsqlite3-mod-ceph_19.3.0-4697-gd58f7306-1jammy_amd64.deb ... 2024-09-13T11:19:30.546 INFO:teuthology.orchestra.run.smithi001.stdout:Selecting previously unselected package python3-bcrypt. 2024-09-13T11:19:30.547 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking libsqlite3-mod-ceph (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:19:30.573 INFO:teuthology.orchestra.run.smithi001.stdout:Preparing to unpack .../061-python3-bcrypt_3.2.0-1build1_amd64.deb ... 2024-09-13T11:19:30.613 INFO:teuthology.orchestra.run.smithi001.stdout:Unpacking python3-bcrypt (3.2.0-1build1) ... 2024-09-13T11:19:30.699 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package python3-jinja2. 2024-09-13T11:19:30.725 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../073-python3-jinja2_3.0.3-1ubuntu0.2_all.deb ... 2024-09-13T11:19:30.765 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking python3-jinja2 (3.0.3-1ubuntu0.2) ... 2024-09-13T11:19:30.816 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package python3-bcrypt. 2024-09-13T11:19:30.842 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../061-python3-bcrypt_3.2.0-1build1_amd64.deb ... 2024-09-13T11:19:30.882 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking python3-bcrypt (3.2.0-1build1) ... 2024-09-13T11:19:30.915 INFO:teuthology.orchestra.run.smithi001.stdout:Selecting previously unselected package python3-openssl. 2024-09-13T11:19:30.943 INFO:teuthology.orchestra.run.smithi001.stdout:Preparing to unpack .../062-python3-openssl_21.0.0-1_all.deb ... 2024-09-13T11:19:30.982 INFO:teuthology.orchestra.run.smithi001.stdout:Unpacking python3-openssl (21.0.0-1) ... 2024-09-13T11:19:31.110 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2024-09-13T11:19:31.136 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../074-ceph-mgr-cephadm_19.3.0-4697-gd58f7306-1jammy_all.deb ... 2024-09-13T11:19:31.144 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package python3-openssl. 2024-09-13T11:19:31.170 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../062-python3-openssl_21.0.0-1_all.deb ... 2024-09-13T11:19:31.176 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking ceph-mgr-cephadm (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:19:31.210 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking python3-openssl (21.0.0-1) ... 2024-09-13T11:19:31.327 INFO:teuthology.orchestra.run.smithi001.stdout:Selecting previously unselected package ceph-mgr. 2024-09-13T11:19:31.354 INFO:teuthology.orchestra.run.smithi001.stdout:Preparing to unpack .../063-ceph-mgr_19.3.0-4697-gd58f7306-1jammy_amd64.deb ... 2024-09-13T11:19:31.393 INFO:teuthology.orchestra.run.smithi001.stdout:Unpacking ceph-mgr (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:19:31.522 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package python3-repoze.lru. 2024-09-13T11:19:31.546 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package ceph-mgr. 2024-09-13T11:19:31.548 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../075-python3-repoze.lru_0.7-2_all.deb ... 2024-09-13T11:19:31.573 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../063-ceph-mgr_19.3.0-4697-gd58f7306-1jammy_amd64.deb ... 2024-09-13T11:19:31.588 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2024-09-13T11:19:31.613 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking ceph-mgr (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:19:31.660 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2024-09-13T11:19:31.677 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../078-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2024-09-13T11:19:31.718 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-09-13T11:19:31.780 INFO:teuthology.orchestra.run.smithi001.stdout:Selecting previously unselected package ceph-mon. 2024-09-13T11:19:31.807 INFO:teuthology.orchestra.run.smithi001.stdout:Preparing to unpack .../064-ceph-mon_19.3.0-4697-gd58f7306-1jammy_amd64.deb ... 2024-09-13T11:19:31.846 INFO:teuthology.orchestra.run.smithi001.stdout:Unpacking ceph-mon (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:19:31.850 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package python3-routes. 2024-09-13T11:19:31.876 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../076-python3-routes_2.5.1-1ubuntu1_all.deb ... 2024-09-13T11:19:31.916 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2024-09-13T11:19:32.024 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package ceph-mon. 2024-09-13T11:19:32.051 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../064-ceph-mon_19.3.0-4697-gd58f7306-1jammy_amd64.deb ... 2024-09-13T11:19:32.091 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking ceph-mon (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:19:32.214 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package python3-joblib. 2024-09-13T11:19:32.241 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../079-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2024-09-13T11:19:32.253 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2024-09-13T11:19:32.279 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../077-ceph-mgr-dashboard_19.3.0-4697-gd58f7306-1jammy_all.deb ... 2024-09-13T11:19:32.280 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2024-09-13T11:19:32.310 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking ceph-mgr-dashboard (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:19:32.467 INFO:teuthology.orchestra.run.smithi001.stdout:Selecting previously unselected package ceph-osd. 2024-09-13T11:19:32.494 INFO:teuthology.orchestra.run.smithi001.stdout:Preparing to unpack .../065-ceph-osd_19.3.0-4697-gd58f7306-1jammy_amd64.deb ... 2024-09-13T11:19:32.533 INFO:teuthology.orchestra.run.smithi001.stdout:Unpacking ceph-osd (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:19:32.619 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package python3-threadpoolctl. 2024-09-13T11:19:32.620 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package ceph-osd. 2024-09-13T11:19:32.646 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../065-ceph-osd_19.3.0-4697-gd58f7306-1jammy_amd64.deb ... 2024-09-13T11:19:32.646 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../080-python3-threadpoolctl_3.1.0-1_all.deb ... 2024-09-13T11:19:32.677 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking ceph-osd (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:19:32.685 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2024-09-13T11:19:32.997 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package python3-sklearn. 2024-09-13T11:19:33.024 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../081-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2024-09-13T11:19:33.064 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2024-09-13T11:19:33.696 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2024-09-13T11:19:33.703 INFO:teuthology.orchestra.run.smithi001.stdout:Selecting previously unselected package ceph. 2024-09-13T11:19:33.712 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../082-ceph-mgr-diskprediction-local_19.3.0-4697-gd58f7306-1jammy_all.deb ... 2024-09-13T11:19:33.719 INFO:teuthology.orchestra.run.smithi001.stdout:Preparing to unpack .../066-ceph_19.3.0-4697-gd58f7306-1jammy_amd64.deb ... 2024-09-13T11:19:33.739 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package ceph. 2024-09-13T11:19:33.752 INFO:teuthology.orchestra.run.smithi001.stdout:Unpacking ceph (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:19:33.765 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../066-ceph_19.3.0-4697-gd58f7306-1jammy_amd64.deb ... 2024-09-13T11:19:33.805 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking ceph (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:19:33.823 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking ceph-mgr-diskprediction-local (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:19:34.047 INFO:teuthology.orchestra.run.smithi001.stdout:Selecting previously unselected package ceph-fuse. 2024-09-13T11:19:34.075 INFO:teuthology.orchestra.run.smithi001.stdout:Preparing to unpack .../067-ceph-fuse_19.3.0-4697-gd58f7306-1jammy_amd64.deb ... 2024-09-13T11:19:34.114 INFO:teuthology.orchestra.run.smithi001.stdout:Unpacking ceph-fuse (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:19:34.124 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package ceph-fuse. 2024-09-13T11:19:34.151 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../067-ceph-fuse_19.3.0-4697-gd58f7306-1jammy_amd64.deb ... 2024-09-13T11:19:34.191 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking ceph-fuse (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:19:34.609 INFO:teuthology.orchestra.run.smithi001.stdout:Selecting previously unselected package ceph-mds. 2024-09-13T11:19:34.636 INFO:teuthology.orchestra.run.smithi001.stdout:Preparing to unpack .../068-ceph-mds_19.3.0-4697-gd58f7306-1jammy_amd64.deb ... 2024-09-13T11:19:34.675 INFO:teuthology.orchestra.run.smithi001.stdout:Unpacking ceph-mds (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:19:34.778 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package ceph-mds. 2024-09-13T11:19:34.804 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../068-ceph-mds_19.3.0-4697-gd58f7306-1jammy_amd64.deb ... 2024-09-13T11:19:34.844 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking ceph-mds (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:19:34.933 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2024-09-13T11:19:34.950 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../078-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2024-09-13T11:19:34.991 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-09-13T11:19:35.002 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package python3-cachetools. 2024-09-13T11:19:35.030 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../083-python3-cachetools_5.0.0-1_all.deb ... 2024-09-13T11:19:35.069 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2024-09-13T11:19:35.086 INFO:teuthology.orchestra.run.smithi001.stdout:Selecting previously unselected package cephadm. 2024-09-13T11:19:35.114 INFO:teuthology.orchestra.run.smithi001.stdout:Preparing to unpack .../069-cephadm_19.3.0-4697-gd58f7306-1jammy_amd64.deb ... 2024-09-13T11:19:35.170 INFO:teuthology.orchestra.run.smithi001.stdout:Unpacking cephadm (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:19:35.205 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package cephadm. 2024-09-13T11:19:35.232 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../069-cephadm_19.3.0-4697-gd58f7306-1jammy_amd64.deb ... 2024-09-13T11:19:35.272 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking cephadm (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:19:35.338 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package python3-pyasn1. 2024-09-13T11:19:35.366 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../084-python3-pyasn1_0.4.8-1_all.deb ... 2024-09-13T11:19:35.405 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking python3-pyasn1 (0.4.8-1) ... 2024-09-13T11:19:35.447 INFO:teuthology.orchestra.run.smithi001.stdout:Selecting previously unselected package python3-asyncssh. 2024-09-13T11:19:35.454 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package python3-joblib. 2024-09-13T11:19:35.474 INFO:teuthology.orchestra.run.smithi001.stdout:Preparing to unpack .../070-python3-asyncssh_2.5.0-1_all.deb ... 2024-09-13T11:19:35.481 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../079-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2024-09-13T11:19:35.505 INFO:teuthology.orchestra.run.smithi001.stdout:Unpacking python3-asyncssh (2.5.0-1) ... 2024-09-13T11:19:35.512 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2024-09-13T11:19:35.583 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package python3-asyncssh. 2024-09-13T11:19:35.609 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../070-python3-asyncssh_2.5.0-1_all.deb ... 2024-09-13T11:19:35.649 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking python3-asyncssh (2.5.0-1) ... 2024-09-13T11:19:35.725 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package python3-pyasn1-modules. 2024-09-13T11:19:35.752 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../085-python3-pyasn1-modules_0.2.1-1_all.deb ... 2024-09-13T11:19:35.791 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking python3-pyasn1-modules (0.2.1-1) ... 2024-09-13T11:19:35.800 INFO:teuthology.orchestra.run.smithi001.stdout:Selecting previously unselected package python-babel-localedata. 2024-09-13T11:19:35.827 INFO:teuthology.orchestra.run.smithi001.stdout:Preparing to unpack .../071-python-babel-localedata_2.8.0+dfsg.1-7_all.deb ... 2024-09-13T11:19:35.866 INFO:teuthology.orchestra.run.smithi001.stdout:Unpacking python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-09-13T11:19:35.874 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package python3-threadpoolctl. 2024-09-13T11:19:35.901 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../080-python3-threadpoolctl_3.1.0-1_all.deb ... 2024-09-13T11:19:35.940 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2024-09-13T11:19:35.943 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package python-babel-localedata. 2024-09-13T11:19:35.959 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../071-python-babel-localedata_2.8.0+dfsg.1-7_all.deb ... 2024-09-13T11:19:36.002 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-09-13T11:19:36.094 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package python3-rsa. 2024-09-13T11:19:36.121 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../086-python3-rsa_4.8-1_all.deb ... 2024-09-13T11:19:36.160 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking python3-rsa (4.8-1) ... 2024-09-13T11:19:36.235 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package python3-sklearn. 2024-09-13T11:19:36.262 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../081-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2024-09-13T11:19:36.301 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2024-09-13T11:19:36.464 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package python3-google-auth. 2024-09-13T11:19:36.492 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../087-python3-google-auth_1.5.1-3_all.deb ... 2024-09-13T11:19:36.530 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2024-09-13T11:19:36.859 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package python3-requests-oauthlib. 2024-09-13T11:19:36.880 INFO:teuthology.orchestra.run.smithi001.stdout:Selecting previously unselected package python3-babel. 2024-09-13T11:19:36.887 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../088-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2024-09-13T11:19:36.908 INFO:teuthology.orchestra.run.smithi001.stdout:Preparing to unpack .../072-python3-babel_2.8.0+dfsg.1-7_all.deb ... 2024-09-13T11:19:36.925 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-09-13T11:19:36.946 INFO:teuthology.orchestra.run.smithi001.stdout:Unpacking python3-babel (2.8.0+dfsg.1-7) ... 2024-09-13T11:19:37.007 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package python3-babel. 2024-09-13T11:19:37.034 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../072-python3-babel_2.8.0+dfsg.1-7_all.deb ... 2024-09-13T11:19:37.045 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2024-09-13T11:19:37.073 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking python3-babel (2.8.0+dfsg.1-7) ... 2024-09-13T11:19:37.073 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../082-ceph-mgr-diskprediction-local_19.3.0-4697-gd58f7306-1jammy_all.deb ... 2024-09-13T11:19:37.197 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking ceph-mgr-diskprediction-local (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:19:37.203 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package python3-websocket. 2024-09-13T11:19:37.219 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../089-python3-websocket_1.2.3-1_all.deb ... 2024-09-13T11:19:37.253 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking python3-websocket (1.2.3-1) ... 2024-09-13T11:19:37.258 INFO:teuthology.orchestra.run.smithi001.stdout:Selecting previously unselected package python3-jinja2. 2024-09-13T11:19:37.285 INFO:teuthology.orchestra.run.smithi001.stdout:Preparing to unpack .../073-python3-jinja2_3.0.3-1ubuntu0.2_all.deb ... 2024-09-13T11:19:37.333 INFO:teuthology.orchestra.run.smithi001.stdout:Unpacking python3-jinja2 (3.0.3-1ubuntu0.2) ... 2024-09-13T11:19:37.393 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package python3-jinja2. 2024-09-13T11:19:37.419 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../073-python3-jinja2_3.0.3-1ubuntu0.2_all.deb ... 2024-09-13T11:19:37.459 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking python3-jinja2 (3.0.3-1ubuntu0.2) ... 2024-09-13T11:19:37.581 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package python3-kubernetes. 2024-09-13T11:19:37.609 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../090-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2024-09-13T11:19:37.647 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2024-09-13T11:19:37.685 INFO:teuthology.orchestra.run.smithi001.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2024-09-13T11:19:37.713 INFO:teuthology.orchestra.run.smithi001.stdout:Preparing to unpack .../074-ceph-mgr-cephadm_19.3.0-4697-gd58f7306-1jammy_all.deb ... 2024-09-13T11:19:37.752 INFO:teuthology.orchestra.run.smithi001.stdout:Unpacking ceph-mgr-cephadm (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:19:37.795 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2024-09-13T11:19:37.835 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../074-ceph-mgr-cephadm_19.3.0-4697-gd58f7306-1jammy_all.deb ... 2024-09-13T11:19:37.862 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking ceph-mgr-cephadm (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:19:38.047 INFO:teuthology.orchestra.run.smithi001.stdout:Selecting previously unselected package python3-repoze.lru. 2024-09-13T11:19:38.075 INFO:teuthology.orchestra.run.smithi001.stdout:Preparing to unpack .../075-python3-repoze.lru_0.7-2_all.deb ... 2024-09-13T11:19:38.113 INFO:teuthology.orchestra.run.smithi001.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2024-09-13T11:19:38.142 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package python3-cachetools. 2024-09-13T11:19:38.170 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../083-python3-cachetools_5.0.0-1_all.deb ... 2024-09-13T11:19:38.198 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package python3-repoze.lru. 2024-09-13T11:19:38.209 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2024-09-13T11:19:38.225 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../075-python3-repoze.lru_0.7-2_all.deb ... 2024-09-13T11:19:38.265 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2024-09-13T11:19:38.320 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2024-09-13T11:19:38.337 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../091-ceph-mgr-k8sevents_19.3.0-4697-gd58f7306-1jammy_all.deb ... 2024-09-13T11:19:38.378 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking ceph-mgr-k8sevents (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:19:38.400 INFO:teuthology.orchestra.run.smithi001.stdout:Selecting previously unselected package python3-routes. 2024-09-13T11:19:38.427 INFO:teuthology.orchestra.run.smithi001.stdout:Preparing to unpack .../076-python3-routes_2.5.1-1ubuntu1_all.deb ... 2024-09-13T11:19:38.466 INFO:teuthology.orchestra.run.smithi001.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2024-09-13T11:19:38.529 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package python3-pyasn1. 2024-09-13T11:19:38.556 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../084-python3-pyasn1_0.4.8-1_all.deb ... 2024-09-13T11:19:38.567 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package python3-routes. 2024-09-13T11:19:38.594 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../076-python3-routes_2.5.1-1ubuntu1_all.deb ... 2024-09-13T11:19:38.595 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking python3-pyasn1 (0.4.8-1) ... 2024-09-13T11:19:38.634 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2024-09-13T11:19:38.697 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package libonig5:amd64. 2024-09-13T11:19:38.726 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../092-libonig5_6.9.7.1-2build1_amd64.deb ... 2024-09-13T11:19:38.764 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2024-09-13T11:19:38.802 INFO:teuthology.orchestra.run.smithi001.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2024-09-13T11:19:38.830 INFO:teuthology.orchestra.run.smithi001.stdout:Preparing to unpack .../077-ceph-mgr-dashboard_19.3.0-4697-gd58f7306-1jammy_all.deb ... 2024-09-13T11:19:38.869 INFO:teuthology.orchestra.run.smithi001.stdout:Unpacking ceph-mgr-dashboard (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:19:38.915 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package python3-pyasn1-modules. 2024-09-13T11:19:38.943 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../085-python3-pyasn1-modules_0.2.1-1_all.deb ... 2024-09-13T11:19:38.979 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2024-09-13T11:19:38.982 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking python3-pyasn1-modules (0.2.1-1) ... 2024-09-13T11:19:38.991 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../077-ceph-mgr-dashboard_19.3.0-4697-gd58f7306-1jammy_all.deb ... 2024-09-13T11:19:39.029 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking ceph-mgr-dashboard (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:19:39.067 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package libjq1:amd64. 2024-09-13T11:19:39.095 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../093-libjq1_1.6-2.1ubuntu3_amd64.deb ... 2024-09-13T11:19:39.142 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-09-13T11:19:39.293 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package python3-rsa. 2024-09-13T11:19:39.321 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../086-python3-rsa_4.8-1_all.deb ... 2024-09-13T11:19:39.359 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking python3-rsa (4.8-1) ... 2024-09-13T11:19:39.361 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package jq. 2024-09-13T11:19:39.377 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../094-jq_1.6-2.1ubuntu3_amd64.deb ... 2024-09-13T11:19:39.419 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking jq (1.6-2.1ubuntu3) ... 2024-09-13T11:19:39.739 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package python3-google-auth. 2024-09-13T11:19:39.766 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../087-python3-google-auth_1.5.1-3_all.deb ... 2024-09-13T11:19:39.772 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package socat. 2024-09-13T11:19:39.797 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2024-09-13T11:19:39.800 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../095-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2024-09-13T11:19:39.830 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2024-09-13T11:19:40.075 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package python3-requests-oauthlib. 2024-09-13T11:19:40.091 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../088-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2024-09-13T11:19:40.109 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package xmlstarlet. 2024-09-13T11:19:40.133 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-09-13T11:19:40.136 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../096-xmlstarlet_1.6.1-2.1_amd64.deb ... 2024-09-13T11:19:40.175 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2024-09-13T11:19:40.444 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package python3-websocket. 2024-09-13T11:19:40.465 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../089-python3-websocket_1.2.3-1_all.deb ... 2024-09-13T11:19:40.503 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking python3-websocket (1.2.3-1) ... 2024-09-13T11:19:40.546 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package ceph-test. 2024-09-13T11:19:40.562 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../097-ceph-test_19.3.0-4697-gd58f7306-1jammy_amd64.deb ... 2024-09-13T11:19:40.604 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking ceph-test (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:19:40.823 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package python3-kubernetes. 2024-09-13T11:19:40.850 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../090-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2024-09-13T11:19:40.889 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2024-09-13T11:19:41.503 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2024-09-13T11:19:41.519 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../091-ceph-mgr-k8sevents_19.3.0-4697-gd58f7306-1jammy_all.deb ... 2024-09-13T11:19:41.553 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking ceph-mgr-k8sevents (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:19:41.658 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2024-09-13T11:19:41.666 INFO:teuthology.orchestra.run.smithi001.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2024-09-13T11:19:41.672 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../078-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2024-09-13T11:19:41.696 INFO:teuthology.orchestra.run.smithi001.stdout:Preparing to unpack .../078-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2024-09-13T11:19:41.707 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-09-13T11:19:41.740 INFO:teuthology.orchestra.run.smithi001.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-09-13T11:19:41.889 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package libonig5:amd64. 2024-09-13T11:19:41.918 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../092-libonig5_6.9.7.1-2build1_amd64.deb ... 2024-09-13T11:19:41.955 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2024-09-13T11:19:42.195 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package python3-joblib. 2024-09-13T11:19:42.222 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../079-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2024-09-13T11:19:42.252 INFO:teuthology.orchestra.run.smithi001.stdout:Selecting previously unselected package python3-joblib. 2024-09-13T11:19:42.264 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2024-09-13T11:19:42.280 INFO:teuthology.orchestra.run.smithi001.stdout:Preparing to unpack .../079-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2024-09-13T11:19:42.292 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package libjq1:amd64. 2024-09-13T11:19:42.318 INFO:teuthology.orchestra.run.smithi001.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2024-09-13T11:19:42.320 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../093-libjq1_1.6-2.1ubuntu3_amd64.deb ... 2024-09-13T11:19:42.358 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-09-13T11:19:42.598 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package python3-threadpoolctl. 2024-09-13T11:19:42.602 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package jq. 2024-09-13T11:19:42.614 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../080-python3-threadpoolctl_3.1.0-1_all.deb ... 2024-09-13T11:19:42.630 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../094-jq_1.6-2.1ubuntu3_amd64.deb ... 2024-09-13T11:19:42.647 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2024-09-13T11:19:42.669 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking jq (1.6-2.1ubuntu3) ... 2024-09-13T11:19:42.706 INFO:teuthology.orchestra.run.smithi001.stdout:Selecting previously unselected package python3-threadpoolctl. 2024-09-13T11:19:42.734 INFO:teuthology.orchestra.run.smithi001.stdout:Preparing to unpack .../080-python3-threadpoolctl_3.1.0-1_all.deb ... 2024-09-13T11:19:42.772 INFO:teuthology.orchestra.run.smithi001.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2024-09-13T11:19:42.913 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package socat. 2024-09-13T11:19:42.929 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../095-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2024-09-13T11:19:42.933 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package python3-sklearn. 2024-09-13T11:19:42.961 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../081-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2024-09-13T11:19:43.000 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2024-09-13T11:19:43.013 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2024-09-13T11:19:43.041 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package ceph-volume. 2024-09-13T11:19:43.042 INFO:teuthology.orchestra.run.smithi001.stdout:Selecting previously unselected package python3-sklearn. 2024-09-13T11:19:43.069 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../098-ceph-volume_19.3.0-4697-gd58f7306-1jammy_all.deb ... 2024-09-13T11:19:43.070 INFO:teuthology.orchestra.run.smithi001.stdout:Preparing to unpack .../081-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2024-09-13T11:19:43.107 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking ceph-volume (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:19:43.108 INFO:teuthology.orchestra.run.smithi001.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2024-09-13T11:19:43.317 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package xmlstarlet. 2024-09-13T11:19:43.344 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../096-xmlstarlet_1.6.1-2.1_amd64.deb ... 2024-09-13T11:19:43.383 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2024-09-13T11:19:43.412 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package libcephfs-dev. 2024-09-13T11:19:43.440 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../099-libcephfs-dev_19.3.0-4697-gd58f7306-1jammy_amd64.deb ... 2024-09-13T11:19:43.503 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking libcephfs-dev (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:19:43.675 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2024-09-13T11:19:43.703 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../082-ceph-mgr-diskprediction-local_19.3.0-4697-gd58f7306-1jammy_all.deb ... 2024-09-13T11:19:43.704 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package ceph-test. 2024-09-13T11:19:43.720 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../097-ceph-test_19.3.0-4697-gd58f7306-1jammy_amd64.deb ... 2024-09-13T11:19:43.762 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking ceph-test (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:19:43.808 INFO:teuthology.orchestra.run.smithi001.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2024-09-13T11:19:43.814 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package lua-socket:amd64. 2024-09-13T11:19:43.819 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking ceph-mgr-diskprediction-local (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:19:43.837 INFO:teuthology.orchestra.run.smithi001.stdout:Preparing to unpack .../082-ceph-mgr-diskprediction-local_19.3.0-4697-gd58f7306-1jammy_all.deb ... 2024-09-13T11:19:43.843 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../100-lua-socket_3.0~rc1+git+ac3201d-6_amd64.deb ... 2024-09-13T11:19:43.880 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-09-13T11:19:43.952 INFO:teuthology.orchestra.run.smithi001.stdout:Unpacking ceph-mgr-diskprediction-local (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:19:44.177 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package lua-sec:amd64. 2024-09-13T11:19:44.205 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../101-lua-sec_1.0.2-1_amd64.deb ... 2024-09-13T11:19:44.235 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking lua-sec:amd64 (1.0.2-1) ... 2024-09-13T11:19:44.606 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package nvme-cli. 2024-09-13T11:19:44.634 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../102-nvme-cli_1.16-3ubuntu0.3_amd64.deb ... 2024-09-13T11:19:44.672 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking nvme-cli (1.16-3ubuntu0.3) ... 2024-09-13T11:19:44.739 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package python3-cachetools. 2024-09-13T11:19:44.766 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../083-python3-cachetools_5.0.0-1_all.deb ... 2024-09-13T11:19:44.805 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2024-09-13T11:19:45.102 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package pkg-config. 2024-09-13T11:19:45.106 INFO:teuthology.orchestra.run.smithi001.stdout:Selecting previously unselected package python3-cachetools. 2024-09-13T11:19:45.134 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../103-pkg-config_0.29.2-1ubuntu3_amd64.deb ... 2024-09-13T11:19:45.135 INFO:teuthology.orchestra.run.smithi001.stdout:Preparing to unpack .../083-python3-cachetools_5.0.0-1_all.deb ... 2024-09-13T11:19:45.169 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking pkg-config (0.29.2-1ubuntu3) ... 2024-09-13T11:19:45.172 INFO:teuthology.orchestra.run.smithi001.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2024-09-13T11:19:45.367 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package python3-pyasn1. 2024-09-13T11:19:45.394 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../084-python3-pyasn1_0.4.8-1_all.deb ... 2024-09-13T11:19:45.430 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package python-asyncssh-doc. 2024-09-13T11:19:45.433 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking python3-pyasn1 (0.4.8-1) ... 2024-09-13T11:19:45.433 INFO:teuthology.orchestra.run.smithi001.stdout:Selecting previously unselected package python3-pyasn1. 2024-09-13T11:19:45.446 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../104-python-asyncssh-doc_2.5.0-1_all.deb ... 2024-09-13T11:19:45.461 INFO:teuthology.orchestra.run.smithi001.stdout:Preparing to unpack .../084-python3-pyasn1_0.4.8-1_all.deb ... 2024-09-13T11:19:45.488 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking python-asyncssh-doc (2.5.0-1) ... 2024-09-13T11:19:45.491 INFO:teuthology.orchestra.run.smithi001.stdout:Unpacking python3-pyasn1 (0.4.8-1) ... 2024-09-13T11:19:45.745 INFO:teuthology.orchestra.run.smithi001.stdout:Selecting previously unselected package python3-pyasn1-modules. 2024-09-13T11:19:45.753 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package python3-pyasn1-modules. 2024-09-13T11:19:45.761 INFO:teuthology.orchestra.run.smithi001.stdout:Preparing to unpack .../085-python3-pyasn1-modules_0.2.1-1_all.deb ... 2024-09-13T11:19:45.780 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../085-python3-pyasn1-modules_0.2.1-1_all.deb ... 2024-09-13T11:19:45.794 INFO:teuthology.orchestra.run.smithi001.stdout:Unpacking python3-pyasn1-modules (0.2.1-1) ... 2024-09-13T11:19:45.819 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking python3-pyasn1-modules (0.2.1-1) ... 2024-09-13T11:19:45.891 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package python3-iniconfig. 2024-09-13T11:19:45.919 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../105-python3-iniconfig_1.1.1-2_all.deb ... 2024-09-13T11:19:45.957 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2024-09-13T11:19:46.081 INFO:teuthology.orchestra.run.smithi001.stdout:Selecting previously unselected package python3-rsa. 2024-09-13T11:19:46.109 INFO:teuthology.orchestra.run.smithi001.stdout:Preparing to unpack .../086-python3-rsa_4.8-1_all.deb ... 2024-09-13T11:19:46.131 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package python3-rsa. 2024-09-13T11:19:46.147 INFO:teuthology.orchestra.run.smithi001.stdout:Unpacking python3-rsa (4.8-1) ... 2024-09-13T11:19:46.158 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../086-python3-rsa_4.8-1_all.deb ... 2024-09-13T11:19:46.197 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking python3-rsa (4.8-1) ... 2024-09-13T11:19:46.267 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package ceph-volume. 2024-09-13T11:19:46.269 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package python3-pastescript. 2024-09-13T11:19:46.294 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../098-ceph-volume_19.3.0-4697-gd58f7306-1jammy_all.deb ... 2024-09-13T11:19:46.297 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../106-python3-pastescript_2.0.2-4_all.deb ... 2024-09-13T11:19:46.333 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking ceph-volume (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:19:46.335 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2024-09-13T11:19:46.468 INFO:teuthology.orchestra.run.smithi001.stdout:Selecting previously unselected package python3-google-auth. 2024-09-13T11:19:46.484 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package python3-google-auth. 2024-09-13T11:19:46.496 INFO:teuthology.orchestra.run.smithi001.stdout:Preparing to unpack .../087-python3-google-auth_1.5.1-3_all.deb ... 2024-09-13T11:19:46.512 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../087-python3-google-auth_1.5.1-3_all.deb ... 2024-09-13T11:19:46.534 INFO:teuthology.orchestra.run.smithi001.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2024-09-13T11:19:46.550 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2024-09-13T11:19:46.629 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package libcephfs-dev. 2024-09-13T11:19:46.656 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../099-libcephfs-dev_19.3.0-4697-gd58f7306-1jammy_amd64.deb ... 2024-09-13T11:19:46.680 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package python3-pluggy. 2024-09-13T11:19:46.695 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking libcephfs-dev (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:19:46.708 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../107-python3-pluggy_0.13.0-7.1_all.deb ... 2024-09-13T11:19:46.747 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2024-09-13T11:19:46.821 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package python3-requests-oauthlib. 2024-09-13T11:19:46.848 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../088-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2024-09-13T11:19:46.871 INFO:teuthology.orchestra.run.smithi001.stdout:Selecting previously unselected package python3-requests-oauthlib. 2024-09-13T11:19:46.878 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-09-13T11:19:46.899 INFO:teuthology.orchestra.run.smithi001.stdout:Preparing to unpack .../088-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2024-09-13T11:19:46.937 INFO:teuthology.orchestra.run.smithi001.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-09-13T11:19:47.006 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package lua-socket:amd64. 2024-09-13T11:19:47.034 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../100-lua-socket_3.0~rc1+git+ac3201d-6_amd64.deb ... 2024-09-13T11:19:47.058 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package python3-psutil. 2024-09-13T11:19:47.073 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-09-13T11:19:47.086 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../108-python3-psutil_5.9.0-1build1_amd64.deb ... 2024-09-13T11:19:47.124 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2024-09-13T11:19:47.207 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package python3-websocket. 2024-09-13T11:19:47.234 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../089-python3-websocket_1.2.3-1_all.deb ... 2024-09-13T11:19:47.265 INFO:teuthology.orchestra.run.smithi001.stdout:Selecting previously unselected package python3-websocket. 2024-09-13T11:19:47.273 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking python3-websocket (1.2.3-1) ... 2024-09-13T11:19:47.294 INFO:teuthology.orchestra.run.smithi001.stdout:Preparing to unpack .../089-python3-websocket_1.2.3-1_all.deb ... 2024-09-13T11:19:47.332 INFO:teuthology.orchestra.run.smithi001.stdout:Unpacking python3-websocket (1.2.3-1) ... 2024-09-13T11:19:47.402 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package lua-sec:amd64. 2024-09-13T11:19:47.430 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../101-lua-sec_1.0.2-1_amd64.deb ... 2024-09-13T11:19:47.469 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking lua-sec:amd64 (1.0.2-1) ... 2024-09-13T11:19:47.469 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package python3-py. 2024-09-13T11:19:47.497 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../109-python3-py_1.10.0-1_all.deb ... 2024-09-13T11:19:47.544 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking python3-py (1.10.0-1) ... 2024-09-13T11:19:47.584 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package python3-kubernetes. 2024-09-13T11:19:47.612 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../090-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2024-09-13T11:19:47.627 INFO:teuthology.orchestra.run.smithi001.stdout:Selecting previously unselected package python3-kubernetes. 2024-09-13T11:19:47.651 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2024-09-13T11:19:47.655 INFO:teuthology.orchestra.run.smithi001.stdout:Preparing to unpack .../090-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2024-09-13T11:19:47.685 INFO:teuthology.orchestra.run.smithi001.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2024-09-13T11:19:47.781 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package nvme-cli. 2024-09-13T11:19:47.809 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../102-nvme-cli_1.16-3ubuntu0.3_amd64.deb ... 2024-09-13T11:19:47.839 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking nvme-cli (1.16-3ubuntu0.3) ... 2024-09-13T11:19:47.889 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package python3-pygments. 2024-09-13T11:19:47.917 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../110-python3-pygments_2.11.2+dfsg-2_all.deb ... 2024-09-13T11:19:47.955 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking python3-pygments (2.11.2+dfsg-2) ... 2024-09-13T11:19:48.194 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package pkg-config. 2024-09-13T11:19:48.222 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../103-pkg-config_0.29.2-1ubuntu3_amd64.deb ... 2024-09-13T11:19:48.261 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking pkg-config (0.29.2-1ubuntu3) ... 2024-09-13T11:19:48.273 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2024-09-13T11:19:48.289 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../091-ceph-mgr-k8sevents_19.3.0-4697-gd58f7306-1jammy_all.deb ... 2024-09-13T11:19:48.323 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking ceph-mgr-k8sevents (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:19:48.368 INFO:teuthology.orchestra.run.smithi001.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2024-09-13T11:19:48.396 INFO:teuthology.orchestra.run.smithi001.stdout:Preparing to unpack .../091-ceph-mgr-k8sevents_19.3.0-4697-gd58f7306-1jammy_all.deb ... 2024-09-13T11:19:48.401 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package python3-pyinotify. 2024-09-13T11:19:48.429 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../111-python3-pyinotify_0.9.6-1.3_all.deb ... 2024-09-13T11:19:48.434 INFO:teuthology.orchestra.run.smithi001.stdout:Unpacking ceph-mgr-k8sevents (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:19:48.467 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking python3-pyinotify (0.9.6-1.3) ... 2024-09-13T11:19:48.505 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package python-asyncssh-doc. 2024-09-13T11:19:48.521 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../104-python-asyncssh-doc_2.5.0-1_all.deb ... 2024-09-13T11:19:48.563 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking python-asyncssh-doc (2.5.0-1) ... 2024-09-13T11:19:48.617 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package libonig5:amd64. 2024-09-13T11:19:48.645 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../092-libonig5_6.9.7.1-2build1_amd64.deb ... 2024-09-13T11:19:48.684 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2024-09-13T11:19:48.703 INFO:teuthology.orchestra.run.smithi001.stdout:Selecting previously unselected package libonig5:amd64. 2024-09-13T11:19:48.732 INFO:teuthology.orchestra.run.smithi001.stdout:Preparing to unpack .../092-libonig5_6.9.7.1-2build1_amd64.deb ... 2024-09-13T11:19:48.761 INFO:teuthology.orchestra.run.smithi001.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2024-09-13T11:19:48.770 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package python3-toml. 2024-09-13T11:19:48.798 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../112-python3-toml_0.10.2-1_all.deb ... 2024-09-13T11:19:48.836 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking python3-toml (0.10.2-1) ... 2024-09-13T11:19:49.009 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package python3-iniconfig. 2024-09-13T11:19:49.028 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package libjq1:amd64. 2024-09-13T11:19:49.036 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../105-python3-iniconfig_1.1.1-2_all.deb ... 2024-09-13T11:19:49.056 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../093-libjq1_1.6-2.1ubuntu3_amd64.deb ... 2024-09-13T11:19:49.075 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2024-09-13T11:19:49.095 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-09-13T11:19:49.098 INFO:teuthology.orchestra.run.smithi001.stdout:Selecting previously unselected package libjq1:amd64. 2024-09-13T11:19:49.106 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package python3-pytest. 2024-09-13T11:19:49.126 INFO:teuthology.orchestra.run.smithi001.stdout:Preparing to unpack .../093-libjq1_1.6-2.1ubuntu3_amd64.deb ... 2024-09-13T11:19:49.134 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../113-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2024-09-13T11:19:49.164 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2024-09-13T11:19:49.173 INFO:teuthology.orchestra.run.smithi001.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-09-13T11:19:49.348 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package jq. 2024-09-13T11:19:49.375 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../094-jq_1.6-2.1ubuntu3_amd64.deb ... 2024-09-13T11:19:49.378 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package python3-pastescript. 2024-09-13T11:19:49.405 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../106-python3-pastescript_2.0.2-4_all.deb ... 2024-09-13T11:19:49.414 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking jq (1.6-2.1ubuntu3) ... 2024-09-13T11:19:49.442 INFO:teuthology.orchestra.run.smithi001.stdout:Selecting previously unselected package jq. 2024-09-13T11:19:49.444 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2024-09-13T11:19:49.459 INFO:teuthology.orchestra.run.smithi001.stdout:Preparing to unpack .../094-jq_1.6-2.1ubuntu3_amd64.deb ... 2024-09-13T11:19:49.500 INFO:teuthology.orchestra.run.smithi001.stdout:Unpacking jq (1.6-2.1ubuntu3) ... 2024-09-13T11:19:49.510 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package python3-simplejson. 2024-09-13T11:19:49.538 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../114-python3-simplejson_3.17.6-1build1_amd64.deb ... 2024-09-13T11:19:49.576 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2024-09-13T11:19:49.633 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package socat. 2024-09-13T11:19:49.661 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../095-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2024-09-13T11:19:49.691 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2024-09-13T11:19:49.728 INFO:teuthology.orchestra.run.smithi001.stdout:Selecting previously unselected package socat. 2024-09-13T11:19:49.746 INFO:teuthology.orchestra.run.smithi001.stdout:Preparing to unpack .../095-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2024-09-13T11:19:49.748 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package python3-pluggy. 2024-09-13T11:19:49.775 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../107-python3-pluggy_0.13.0-7.1_all.deb ... 2024-09-13T11:19:49.786 INFO:teuthology.orchestra.run.smithi001.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2024-09-13T11:19:49.806 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2024-09-13T11:19:49.870 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package qttranslations5-l10n. 2024-09-13T11:19:49.887 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../115-qttranslations5-l10n_5.15.3-1_all.deb ... 2024-09-13T11:19:49.928 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2024-09-13T11:19:49.978 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package xmlstarlet. 2024-09-13T11:19:49.994 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../096-xmlstarlet_1.6.1-2.1_amd64.deb ... 2024-09-13T11:19:50.036 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2024-09-13T11:19:50.056 INFO:teuthology.orchestra.run.smithi001.stdout:Selecting previously unselected package xmlstarlet. 2024-09-13T11:19:50.073 INFO:teuthology.orchestra.run.smithi001.stdout:Preparing to unpack .../096-xmlstarlet_1.6.1-2.1_amd64.deb ... 2024-09-13T11:19:50.106 INFO:teuthology.orchestra.run.smithi001.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2024-09-13T11:19:50.109 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package python3-psutil. 2024-09-13T11:19:50.137 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../108-python3-psutil_5.9.0-1build1_amd64.deb ... 2024-09-13T11:19:50.192 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2024-09-13T11:19:50.435 INFO:teuthology.orchestra.run.smithi001.stdout:Selecting previously unselected package ceph-test. 2024-09-13T11:19:50.452 INFO:teuthology.orchestra.run.smithi001.stdout:Preparing to unpack .../097-ceph-test_19.3.0-4697-gd58f7306-1jammy_amd64.deb ... 2024-09-13T11:19:50.473 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package ceph-test. 2024-09-13T11:19:50.485 INFO:teuthology.orchestra.run.smithi001.stdout:Unpacking ceph-test (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:19:50.490 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../097-ceph-test_19.3.0-4697-gd58f7306-1jammy_amd64.deb ... 2024-09-13T11:19:50.495 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package python3-py. 2024-09-13T11:19:50.523 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../109-python3-py_1.10.0-1_all.deb ... 2024-09-13T11:19:50.523 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking ceph-test (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:19:50.561 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking python3-py (1.10.0-1) ... 2024-09-13T11:19:50.599 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package radosgw. 2024-09-13T11:19:50.628 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../116-radosgw_19.3.0-4697-gd58f7306-1jammy_amd64.deb ... 2024-09-13T11:19:50.657 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking radosgw (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:19:50.882 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package python3-pygments. 2024-09-13T11:19:50.910 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../110-python3-pygments_2.11.2+dfsg-2_all.deb ... 2024-09-13T11:19:50.948 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking python3-pygments (2.11.2+dfsg-2) ... 2024-09-13T11:19:51.419 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package python3-pyinotify. 2024-09-13T11:19:51.447 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../111-python3-pyinotify_0.9.6-1.3_all.deb ... 2024-09-13T11:19:51.485 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking python3-pyinotify (0.9.6-1.3) ... 2024-09-13T11:19:51.543 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package rbd-fuse. 2024-09-13T11:19:51.560 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../117-rbd-fuse_19.3.0-4697-gd58f7306-1jammy_amd64.deb ... 2024-09-13T11:19:51.601 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking rbd-fuse (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:19:51.805 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package python3-toml. 2024-09-13T11:19:51.833 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../112-python3-toml_0.10.2-1_all.deb ... 2024-09-13T11:19:51.871 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking python3-toml (0.10.2-1) ... 2024-09-13T11:19:51.897 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2024-09-13T11:19:52.148 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2024-09-13T11:19:52.157 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package python3-pytest. 2024-09-13T11:19:52.173 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../113-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2024-09-13T11:19:52.207 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2024-09-13T11:19:52.249 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up nvme-cli (1.16-3ubuntu0.3) ... 2024-09-13T11:19:52.514 INFO:teuthology.orchestra.run.smithi092.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service -> /lib/systemd/system/nvmefc-boot-connections.service. 2024-09-13T11:19:52.544 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package python3-simplejson. 2024-09-13T11:19:52.572 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../114-python3-simplejson_3.17.6-1build1_amd64.deb ... 2024-09-13T11:19:52.610 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2024-09-13T11:19:52.862 INFO:teuthology.orchestra.run.smithi092.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service -> /lib/systemd/system/nvmf-autoconnect.service. 2024-09-13T11:19:52.863 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package qttranslations5-l10n. 2024-09-13T11:19:52.879 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../115-qttranslations5-l10n_5.15.3-1_all.deb ... 2024-09-13T11:19:52.921 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2024-09-13T11:19:53.078 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package ceph-volume. 2024-09-13T11:19:53.108 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../098-ceph-volume_19.3.0-4697-gd58f7306-1jammy_all.deb ... 2024-09-13T11:19:53.144 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking ceph-volume (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:19:53.156 INFO:teuthology.orchestra.run.smithi001.stdout:Selecting previously unselected package ceph-volume. 2024-09-13T11:19:53.184 INFO:teuthology.orchestra.run.smithi001.stdout:Preparing to unpack .../098-ceph-volume_19.3.0-4697-gd58f7306-1jammy_all.deb ... 2024-09-13T11:19:53.222 INFO:teuthology.orchestra.run.smithi001.stdout:Unpacking ceph-volume (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:19:53.290 INFO:teuthology.orchestra.run.smithi092.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2024-09-13T11:19:53.351 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up cephadm (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:19:53.448 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package libcephfs-dev. 2024-09-13T11:19:53.464 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../099-libcephfs-dev_19.3.0-4697-gd58f7306-1jammy_amd64.deb ... 2024-09-13T11:19:53.506 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking libcephfs-dev (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:19:53.526 INFO:teuthology.orchestra.run.smithi001.stdout:Selecting previously unselected package libcephfs-dev. 2024-09-13T11:19:53.555 INFO:teuthology.orchestra.run.smithi001.stdout:Preparing to unpack .../099-libcephfs-dev_19.3.0-4697-gd58f7306-1jammy_amd64.deb ... 2024-09-13T11:19:53.593 INFO:teuthology.orchestra.run.smithi001.stdout:Unpacking libcephfs-dev (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:19:53.634 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package radosgw. 2024-09-13T11:19:53.662 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../116-radosgw_19.3.0-4697-gd58f7306-1jammy_amd64.deb ... 2024-09-13T11:19:53.700 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking radosgw (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:19:53.792 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package lua-socket:amd64. 2024-09-13T11:19:53.820 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../100-lua-socket_3.0~rc1+git+ac3201d-6_amd64.deb ... 2024-09-13T11:19:53.850 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-09-13T11:19:53.876 INFO:teuthology.orchestra.run.smithi092.stdout:Adding system user cephadm....done 2024-09-13T11:19:53.912 INFO:teuthology.orchestra.run.smithi001.stdout:Selecting previously unselected package lua-socket:amd64. 2024-09-13T11:19:53.941 INFO:teuthology.orchestra.run.smithi001.stdout:Preparing to unpack .../100-lua-socket_3.0~rc1+git+ac3201d-6_amd64.deb ... 2024-09-13T11:19:53.979 INFO:teuthology.orchestra.run.smithi001.stdout:Unpacking lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-09-13T11:19:54.039 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up python3-waitress (1.4.4-1.1ubuntu1) ... 2024-09-13T11:19:54.165 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package lua-sec:amd64. 2024-09-13T11:19:54.193 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../101-lua-sec_1.0.2-1_amd64.deb ... 2024-09-13T11:19:54.231 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking lua-sec:amd64 (1.0.2-1) ... 2024-09-13T11:19:54.299 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2024-09-13T11:19:54.317 INFO:teuthology.orchestra.run.smithi001.stdout:Selecting previously unselected package lua-sec:amd64. 2024-09-13T11:19:54.345 INFO:teuthology.orchestra.run.smithi001.stdout:Preparing to unpack .../101-lua-sec_1.0.2-1_amd64.deb ... 2024-09-13T11:19:54.383 INFO:teuthology.orchestra.run.smithi001.stdout:Unpacking lua-sec:amd64 (1.0.2-1) ... 2024-09-13T11:19:54.551 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up python-asyncssh-doc (2.5.0-1) ... 2024-09-13T11:19:54.593 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package nvme-cli. 2024-09-13T11:19:54.619 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package rbd-fuse. 2024-09-13T11:19:54.622 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../102-nvme-cli_1.16-3ubuntu0.3_amd64.deb ... 2024-09-13T11:19:54.635 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../117-rbd-fuse_19.3.0-4697-gd58f7306-1jammy_amd64.deb ... 2024-09-13T11:19:54.659 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking nvme-cli (1.16-3ubuntu0.3) ... 2024-09-13T11:19:54.677 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2024-09-13T11:19:54.678 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking rbd-fuse (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:19:54.745 INFO:teuthology.orchestra.run.smithi001.stdout:Selecting previously unselected package nvme-cli. 2024-09-13T11:19:54.774 INFO:teuthology.orchestra.run.smithi001.stdout:Preparing to unpack .../102-nvme-cli_1.16-3ubuntu0.3_amd64.deb ... 2024-09-13T11:19:54.812 INFO:teuthology.orchestra.run.smithi001.stdout:Unpacking nvme-cli (1.16-3ubuntu0.3) ... 2024-09-13T11:19:54.928 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up python3-repoze.lru (0.7-2) ... 2024-09-13T11:19:54.974 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2024-09-13T11:19:55.057 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package pkg-config. 2024-09-13T11:19:55.085 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../103-pkg-config_0.29.2-1ubuntu3_amd64.deb ... 2024-09-13T11:19:55.139 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking pkg-config (0.29.2-1ubuntu3) ... 2024-09-13T11:19:55.195 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up liboath0:amd64 (2.6.7-3build1) ... 2024-09-13T11:19:55.224 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2024-09-13T11:19:55.225 INFO:teuthology.orchestra.run.smithi001.stdout:Selecting previously unselected package pkg-config. 2024-09-13T11:19:55.254 INFO:teuthology.orchestra.run.smithi001.stdout:Preparing to unpack .../103-pkg-config_0.29.2-1ubuntu3_amd64.deb ... 2024-09-13T11:19:55.292 INFO:teuthology.orchestra.run.smithi001.stdout:Unpacking pkg-config (0.29.2-1ubuntu3) ... 2024-09-13T11:19:55.330 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up python3-py (1.10.0-1) ... 2024-09-13T11:19:55.359 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up nvme-cli (1.16-3ubuntu0.3) ... 2024-09-13T11:19:55.376 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package python-asyncssh-doc. 2024-09-13T11:19:55.404 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../104-python-asyncssh-doc_2.5.0-1_all.deb ... 2024-09-13T11:19:55.442 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking python-asyncssh-doc (2.5.0-1) ... 2024-09-13T11:19:55.536 INFO:teuthology.orchestra.run.smithi001.stdout:Selecting previously unselected package python-asyncssh-doc. 2024-09-13T11:19:55.554 INFO:teuthology.orchestra.run.smithi001.stdout:Preparing to unpack .../104-python-asyncssh-doc_2.5.0-1_all.deb ... 2024-09-13T11:19:55.594 INFO:teuthology.orchestra.run.smithi162.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service -> /lib/systemd/system/nvmefc-boot-connections.service. 2024-09-13T11:19:55.594 INFO:teuthology.orchestra.run.smithi001.stdout:Unpacking python-asyncssh-doc (2.5.0-1) ... 2024-09-13T11:19:55.640 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2024-09-13T11:19:55.821 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package python3-iniconfig. 2024-09-13T11:19:55.849 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../105-python3-iniconfig_1.1.1-2_all.deb ... 2024-09-13T11:19:55.878 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2024-09-13T11:19:55.924 INFO:teuthology.orchestra.run.smithi162.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service -> /lib/systemd/system/nvmf-autoconnect.service. 2024-09-13T11:19:56.000 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up python3-cachetools (5.0.0-1) ... 2024-09-13T11:19:56.048 INFO:teuthology.orchestra.run.smithi001.stdout:Selecting previously unselected package python3-iniconfig. 2024-09-13T11:19:56.077 INFO:teuthology.orchestra.run.smithi001.stdout:Preparing to unpack .../105-python3-iniconfig_1.1.1-2_all.deb ... 2024-09-13T11:19:56.114 INFO:teuthology.orchestra.run.smithi001.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2024-09-13T11:19:56.181 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package python3-pastescript. 2024-09-13T11:19:56.209 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../106-python3-pastescript_2.0.2-4_all.deb ... 2024-09-13T11:19:56.247 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2024-09-13T11:19:56.260 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up python3-openssl (21.0.0-1) ... 2024-09-13T11:19:56.330 INFO:teuthology.orchestra.run.smithi162.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2024-09-13T11:19:56.394 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up cephadm (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:19:56.417 INFO:teuthology.orchestra.run.smithi001.stdout:Selecting previously unselected package python3-pastescript. 2024-09-13T11:19:56.445 INFO:teuthology.orchestra.run.smithi001.stdout:Preparing to unpack .../106-python3-pastescript_2.0.2-4_all.deb ... 2024-09-13T11:19:56.483 INFO:teuthology.orchestra.run.smithi001.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2024-09-13T11:19:56.528 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up unzip (6.0-26ubuntu3.2) ... 2024-09-13T11:19:56.567 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package python3-pluggy. 2024-09-13T11:19:56.596 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../107-python3-pluggy_0.13.0-7.1_all.deb ... 2024-09-13T11:19:56.634 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2024-09-13T11:19:56.845 INFO:teuthology.orchestra.run.smithi001.stdout:Selecting previously unselected package python3-pluggy. 2024-09-13T11:19:56.846 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up python3-bcrypt (3.2.0-1build1) ... 2024-09-13T11:19:56.868 INFO:teuthology.orchestra.run.smithi162.stdout:Adding system user cephadm....done 2024-09-13T11:19:56.874 INFO:teuthology.orchestra.run.smithi001.stdout:Preparing to unpack .../107-python3-pluggy_0.13.0-7.1_all.deb ... 2024-09-13T11:19:56.911 INFO:teuthology.orchestra.run.smithi001.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2024-09-13T11:19:56.953 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package python3-psutil. 2024-09-13T11:19:56.982 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../108-python3-psutil_5.9.0-1build1_amd64.deb ... 2024-09-13T11:19:57.000 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up python3-waitress (1.4.4-1.1ubuntu1) ... 2024-09-13T11:19:57.020 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2024-09-13T11:19:57.080 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up python3-pyinotify (0.9.6-1.3) ... 2024-09-13T11:19:57.223 INFO:teuthology.orchestra.run.smithi001.stdout:Selecting previously unselected package python3-psutil. 2024-09-13T11:19:57.251 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2024-09-13T11:19:57.252 INFO:teuthology.orchestra.run.smithi001.stdout:Preparing to unpack .../108-python3-psutil_5.9.0-1build1_amd64.deb ... 2024-09-13T11:19:57.289 INFO:teuthology.orchestra.run.smithi001.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2024-09-13T11:19:57.339 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2024-09-13T11:19:57.365 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package python3-py. 2024-09-13T11:19:57.393 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../109-python3-py_1.10.0-1_all.deb ... 2024-09-13T11:19:57.431 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking python3-py (1.10.0-1) ... 2024-09-13T11:19:57.519 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up python-asyncssh-doc (2.5.0-1) ... 2024-09-13T11:19:57.590 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up python3-ceph-argparse (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:19:57.592 INFO:teuthology.orchestra.run.smithi001.stdout:Selecting previously unselected package python3-py. 2024-09-13T11:19:57.609 INFO:teuthology.orchestra.run.smithi001.stdout:Preparing to unpack .../109-python3-py_1.10.0-1_all.deb ... 2024-09-13T11:19:57.642 INFO:teuthology.orchestra.run.smithi001.stdout:Unpacking python3-py (1.10.0-1) ... 2024-09-13T11:19:57.645 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2024-09-13T11:19:57.759 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package python3-pygments. 2024-09-13T11:19:57.788 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../110-python3-pygments_2.11.2+dfsg-2_all.deb ... 2024-09-13T11:19:57.825 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking python3-pygments (2.11.2+dfsg-2) ... 2024-09-13T11:19:57.892 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-09-13T11:19:57.896 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up python3-repoze.lru (0.7-2) ... 2024-09-13T11:19:57.920 INFO:teuthology.orchestra.run.smithi001.stdout:Selecting previously unselected package python3-pygments. 2024-09-13T11:19:57.949 INFO:teuthology.orchestra.run.smithi001.stdout:Preparing to unpack .../110-python3-pygments_2.11.2+dfsg-2_all.deb ... 2024-09-13T11:19:57.987 INFO:teuthology.orchestra.run.smithi001.stdout:Unpacking python3-pygments (2.11.2+dfsg-2) ... 2024-09-13T11:19:58.017 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-09-13T11:19:58.135 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up libreadline-dev:amd64 (8.1.2-1) ... 2024-09-13T11:19:58.139 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up liboath0:amd64 (2.6.7-3build1) ... 2024-09-13T11:19:58.244 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up python3-markupsafe (2.0.1-2build1) ... 2024-09-13T11:19:58.257 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up python3-py (1.10.0-1) ... 2024-09-13T11:19:58.288 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package python3-pyinotify. 2024-09-13T11:19:58.316 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../111-python3-pyinotify_0.9.6-1.3_all.deb ... 2024-09-13T11:19:58.354 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking python3-pyinotify (0.9.6-1.3) ... 2024-09-13T11:19:58.441 INFO:teuthology.orchestra.run.smithi001.stdout:Selecting previously unselected package python3-pyinotify. 2024-09-13T11:19:58.470 INFO:teuthology.orchestra.run.smithi001.stdout:Preparing to unpack .../111-python3-pyinotify_0.9.6-1.3_all.deb ... 2024-09-13T11:19:58.487 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up lua5.1 (5.1.5-8.1build4) ... 2024-09-13T11:19:58.507 INFO:teuthology.orchestra.run.smithi001.stdout:Unpacking python3-pyinotify (0.9.6-1.3) ... 2024-09-13T11:19:58.558 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2024-09-13T11:19:58.657 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package python3-toml. 2024-09-13T11:19:58.686 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../112-python3-toml_0.10.2-1_all.deb ... 2024-09-13T11:19:58.723 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking python3-toml (0.10.2-1) ... 2024-09-13T11:19:58.802 INFO:teuthology.orchestra.run.smithi001.stdout:Selecting previously unselected package python3-toml. 2024-09-13T11:19:58.831 INFO:teuthology.orchestra.run.smithi001.stdout:Preparing to unpack .../112-python3-toml_0.10.2-1_all.deb ... 2024-09-13T11:19:58.868 INFO:teuthology.orchestra.run.smithi001.stdout:Unpacking python3-toml (0.10.2-1) ... 2024-09-13T11:19:58.976 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package python3-pytest. 2024-09-13T11:19:59.001 INFO:teuthology.orchestra.run.smithi092.stdout:update-alternatives: using /usr/bin/lua5.1 to provide /usr/bin/lua (lua-interpreter) in auto mode 2024-09-13T11:19:59.005 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../113-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2024-09-13T11:19:59.042 INFO:teuthology.orchestra.run.smithi092.stdout:update-alternatives: using /usr/bin/luac5.1 to provide /usr/bin/luac (lua-compiler) in auto mode 2024-09-13T11:19:59.043 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2024-09-13T11:19:59.107 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-09-13T11:19:59.146 INFO:teuthology.orchestra.run.smithi001.stdout:Selecting previously unselected package python3-pytest. 2024-09-13T11:19:59.175 INFO:teuthology.orchestra.run.smithi001.stdout:Preparing to unpack .../113-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2024-09-13T11:19:59.212 INFO:teuthology.orchestra.run.smithi001.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2024-09-13T11:19:59.219 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up python3-cachetools (5.0.0-1) ... 2024-09-13T11:19:59.233 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2024-09-13T11:19:59.413 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package python3-simplejson. 2024-09-13T11:19:59.442 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../114-python3-simplejson_3.17.6-1build1_amd64.deb ... 2024-09-13T11:19:59.479 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2024-09-13T11:19:59.495 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up python3-openssl (21.0.0-1) ... 2024-09-13T11:19:59.566 INFO:teuthology.orchestra.run.smithi001.stdout:Selecting previously unselected package python3-simplejson. 2024-09-13T11:19:59.592 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-09-13T11:19:59.595 INFO:teuthology.orchestra.run.smithi001.stdout:Preparing to unpack .../114-python3-simplejson_3.17.6-1build1_amd64.deb ... 2024-09-13T11:19:59.632 INFO:teuthology.orchestra.run.smithi001.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2024-09-13T11:19:59.702 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up python3-natsort (8.0.2-1) ... 2024-09-13T11:19:59.757 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package qttranslations5-l10n. 2024-09-13T11:19:59.771 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up unzip (6.0-26ubuntu3.2) ... 2024-09-13T11:19:59.773 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../115-qttranslations5-l10n_5.15.3-1_all.deb ... 2024-09-13T11:19:59.815 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2024-09-13T11:19:59.910 INFO:teuthology.orchestra.run.smithi001.stdout:Selecting previously unselected package qttranslations5-l10n. 2024-09-13T11:19:59.928 INFO:teuthology.orchestra.run.smithi001.stdout:Preparing to unpack .../115-qttranslations5-l10n_5.15.3-1_all.deb ... 2024-09-13T11:19:59.968 INFO:teuthology.orchestra.run.smithi001.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2024-09-13T11:19:59.969 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2024-09-13T11:20:00.023 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up python3-bcrypt (3.2.0-1build1) ... 2024-09-13T11:20:00.237 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2024-09-13T11:20:00.273 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up python3-pyinotify (0.9.6-1.3) ... 2024-09-13T11:20:00.533 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2024-09-13T11:20:00.538 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up zip (3.0-12build2) ... 2024-09-13T11:20:00.561 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package radosgw. 2024-09-13T11:20:00.590 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../116-radosgw_19.3.0-4697-gd58f7306-1jammy_amd64.deb ... 2024-09-13T11:20:00.627 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking radosgw (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:20:00.648 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up python3-pygments (2.11.2+dfsg-2) ... 2024-09-13T11:20:00.775 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up python3-ceph-argparse (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:20:00.906 INFO:teuthology.orchestra.run.smithi001.stdout:Selecting previously unselected package radosgw. 2024-09-13T11:20:00.934 INFO:teuthology.orchestra.run.smithi001.stdout:Preparing to unpack .../116-radosgw_19.3.0-4697-gd58f7306-1jammy_amd64.deb ... 2024-09-13T11:20:00.972 INFO:teuthology.orchestra.run.smithi001.stdout:Unpacking radosgw (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:20:01.018 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-09-13T11:20:01.144 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-09-13T11:20:01.270 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up libreadline-dev:amd64 (8.1.2-1) ... 2024-09-13T11:20:01.366 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up python3-tempita (0.5.2-6ubuntu1) ... 2024-09-13T11:20:01.397 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up python3-markupsafe (2.0.1-2build1) ... 2024-09-13T11:20:01.488 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package rbd-fuse. 2024-09-13T11:20:01.517 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../117-rbd-fuse_19.3.0-4697-gd58f7306-1jammy_amd64.deb ... 2024-09-13T11:20:01.554 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking rbd-fuse (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:20:01.626 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up python-pastedeploy-tpl (2.1.1-1) ... 2024-09-13T11:20:01.639 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up lua5.1 (5.1.5-8.1build4) ... 2024-09-13T11:20:01.743 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2024-09-13T11:20:01.844 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2024-09-13T11:20:01.859 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2024-09-13T11:20:01.975 INFO:teuthology.orchestra.run.smithi001.stdout:Selecting previously unselected package rbd-fuse. 2024-09-13T11:20:01.993 INFO:teuthology.orchestra.run.smithi001.stdout:Preparing to unpack .../117-rbd-fuse_19.3.0-4697-gd58f7306-1jammy_amd64.deb ... 2024-09-13T11:20:02.025 INFO:teuthology.orchestra.run.smithi001.stdout:Unpacking rbd-fuse (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:20:02.059 INFO:teuthology.orchestra.run.smithi162.stdout:update-alternatives: using /usr/bin/lua5.1 to provide /usr/bin/lua (lua-interpreter) in auto mode 2024-09-13T11:20:02.095 INFO:teuthology.orchestra.run.smithi162.stdout:update-alternatives: using /usr/bin/luac5.1 to provide /usr/bin/luac (lua-compiler) in auto mode 2024-09-13T11:20:02.118 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2024-09-13T11:20:02.150 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-09-13T11:20:02.170 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up python3-asyncssh (2.5.0-1) ... 2024-09-13T11:20:02.244 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up nvme-cli (1.16-3ubuntu0.3) ... 2024-09-13T11:20:02.251 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2024-09-13T11:20:02.322 INFO:teuthology.orchestra.run.smithi001.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2024-09-13T11:20:02.517 INFO:teuthology.orchestra.run.smithi177.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service -> /lib/systemd/system/nvmefc-boot-connections.service. 2024-09-13T11:20:02.538 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up python3-certifi (2020.6.20-1) ... 2024-09-13T11:20:02.580 INFO:teuthology.orchestra.run.smithi001.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2024-09-13T11:20:02.611 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-09-13T11:20:02.706 INFO:teuthology.orchestra.run.smithi001.stdout:Setting up nvme-cli (1.16-3ubuntu0.3) ... 2024-09-13T11:20:02.737 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up python3-natsort (8.0.2-1) ... 2024-09-13T11:20:02.781 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up python3-paste (3.5.0+dfsg1-1) ... 2024-09-13T11:20:02.836 INFO:teuthology.orchestra.run.smithi177.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service -> /lib/systemd/system/nvmf-autoconnect.service. 2024-09-13T11:20:02.980 INFO:teuthology.orchestra.run.smithi001.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service -> /lib/systemd/system/nvmefc-boot-connections.service. 2024-09-13T11:20:03.005 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2024-09-13T11:20:03.149 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-09-13T11:20:03.252 INFO:teuthology.orchestra.run.smithi177.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2024-09-13T11:20:03.273 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2024-09-13T11:20:03.337 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up cephadm (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:20:03.359 INFO:teuthology.orchestra.run.smithi001.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service -> /lib/systemd/system/nvmf-autoconnect.service. 2024-09-13T11:20:03.434 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.2) ... 2024-09-13T11:20:03.574 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up zip (3.0-12build2) ... 2024-09-13T11:20:03.700 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up python3-pygments (2.11.2+dfsg-2) ... 2024-09-13T11:20:03.777 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2024-09-13T11:20:03.800 INFO:teuthology.orchestra.run.smithi001.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2024-09-13T11:20:03.861 INFO:teuthology.orchestra.run.smithi177.stdout:Adding system user cephadm....done 2024-09-13T11:20:03.866 INFO:teuthology.orchestra.run.smithi001.stdout:Setting up cephadm (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:20:03.976 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up python3-waitress (1.4.4-1.1ubuntu1) ... 2024-09-13T11:20:04.002 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2024-09-13T11:20:04.120 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up python3-ceph-common (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:20:04.236 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2024-09-13T11:20:04.343 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up python3-tempita (0.5.2-6ubuntu1) ... 2024-09-13T11:20:04.388 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up python3-idna (3.3-1ubuntu0.1) ... 2024-09-13T11:20:04.495 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up python-asyncssh-doc (2.5.0-1) ... 2024-09-13T11:20:04.540 INFO:teuthology.orchestra.run.smithi001.stdout:Adding system user cephadm....done 2024-09-13T11:20:04.594 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up python-pastedeploy-tpl (2.1.1-1) ... 2024-09-13T11:20:04.622 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2024-09-13T11:20:04.689 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2024-09-13T11:20:04.697 INFO:teuthology.orchestra.run.smithi001.stdout:Setting up python3-waitress (1.4.4-1.1ubuntu1) ... 2024-09-13T11:20:04.738 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2024-09-13T11:20:04.863 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2024-09-13T11:20:04.889 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up python3-repoze.lru (0.7-2) ... 2024-09-13T11:20:04.973 INFO:teuthology.orchestra.run.smithi001.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2024-09-13T11:20:05.141 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up liboath0:amd64 (2.6.7-3build1) ... 2024-09-13T11:20:05.164 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up python3-asyncssh (2.5.0-1) ... 2024-09-13T11:20:05.241 INFO:teuthology.orchestra.run.smithi001.stdout:Setting up python-asyncssh-doc (2.5.0-1) ... 2024-09-13T11:20:05.266 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up python3-py (1.10.0-1) ... 2024-09-13T11:20:05.367 INFO:teuthology.orchestra.run.smithi001.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2024-09-13T11:20:05.565 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up python3-certifi (2020.6.20-1) ... 2024-09-13T11:20:05.660 INFO:teuthology.orchestra.run.smithi001.stdout:Setting up python3-repoze.lru (0.7-2) ... 2024-09-13T11:20:05.668 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2024-09-13T11:20:05.800 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up python3-paste (3.5.0+dfsg1-1) ... 2024-09-13T11:20:05.870 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up pkg-config (0.29.2-1ubuntu3) ... 2024-09-13T11:20:05.919 INFO:teuthology.orchestra.run.smithi001.stdout:Setting up liboath0:amd64 (2.6.7-3build1) ... 2024-09-13T11:20:06.045 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up python3-cachetools (5.0.0-1) ... 2024-09-13T11:20:06.046 INFO:teuthology.orchestra.run.smithi001.stdout:Setting up python3-py (1.10.0-1) ... 2024-09-13T11:20:06.113 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-13T11:20:06.184 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-09-13T11:20:06.248 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up python3-toml (0.10.2-1) ... 2024-09-13T11:20:06.305 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up python3-openssl (21.0.0-1) ... 2024-09-13T11:20:06.364 INFO:teuthology.orchestra.run.smithi001.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2024-09-13T11:20:06.477 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.2) ... 2024-09-13T11:20:06.532 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2024-09-13T11:20:06.572 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up unzip (6.0-26ubuntu3.2) ... 2024-09-13T11:20:06.658 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2024-09-13T11:20:06.741 INFO:teuthology.orchestra.run.smithi001.stdout:Setting up python3-cachetools (5.0.0-1) ... 2024-09-13T11:20:06.784 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-09-13T11:20:06.812 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2024-09-13T11:20:06.824 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up python3-bcrypt (3.2.0-1build1) ... 2024-09-13T11:20:06.984 INFO:teuthology.orchestra.run.smithi001.stdout:Setting up python3-openssl (21.0.0-1) ... 2024-09-13T11:20:07.063 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2024-09-13T11:20:07.066 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up python3-pyinotify (0.9.6-1.3) ... 2024-09-13T11:20:07.077 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2024-09-13T11:20:07.181 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up python3-ceph-common (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:20:07.244 INFO:teuthology.orchestra.run.smithi001.stdout:Setting up unzip (6.0-26ubuntu3.2) ... 2024-09-13T11:20:07.309 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2024-09-13T11:20:07.328 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2024-09-13T11:20:07.490 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up python3-idna (3.3-1ubuntu0.1) ... 2024-09-13T11:20:07.520 INFO:teuthology.orchestra.run.smithi001.stdout:Setting up python3-bcrypt (3.2.0-1build1) ... 2024-09-13T11:20:07.560 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up python3-ceph-argparse (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:20:07.570 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-13T11:20:07.671 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up python3-pyasn1 (0.4.8-1) ... 2024-09-13T11:20:07.787 INFO:teuthology.orchestra.run.smithi001.stdout:Setting up python3-pyinotify (0.9.6-1.3) ... 2024-09-13T11:20:07.791 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2024-09-13T11:20:07.811 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-09-13T11:20:07.937 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-09-13T11:20:07.973 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up python3-singledispatch (3.4.0.3-3) ... 2024-09-13T11:20:08.063 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up libreadline-dev:amd64 (8.1.2-1) ... 2024-09-13T11:20:08.105 INFO:teuthology.orchestra.run.smithi001.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2024-09-13T11:20:08.189 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up python3-markupsafe (2.0.1-2build1) ... 2024-09-13T11:20:08.257 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up python3-logutils (0.3.3-8) ... 2024-09-13T11:20:08.381 INFO:teuthology.orchestra.run.smithi001.stdout:Setting up python3-ceph-argparse (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:20:08.440 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up lua5.1 (5.1.5-8.1build4) ... 2024-09-13T11:20:08.525 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up python3-tempora (4.1.2-1) ... 2024-09-13T11:20:08.649 INFO:teuthology.orchestra.run.smithi001.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-09-13T11:20:08.750 INFO:teuthology.orchestra.run.smithi001.stdout:Setting up lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-09-13T11:20:08.767 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2024-09-13T11:20:08.825 INFO:teuthology.orchestra.run.smithi177.stdout:update-alternatives: using /usr/bin/lua5.1 to provide /usr/bin/lua (lua-interpreter) in auto mode 2024-09-13T11:20:08.855 INFO:teuthology.orchestra.run.smithi177.stdout:update-alternatives: using /usr/bin/luac5.1 to provide /usr/bin/luac (lua-compiler) in auto mode 2024-09-13T11:20:08.867 INFO:teuthology.orchestra.run.smithi001.stdout:Setting up libreadline-dev:amd64 (8.1.2-1) ... 2024-09-13T11:20:08.910 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-09-13T11:20:08.956 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up pkg-config (0.29.2-1ubuntu3) ... 2024-09-13T11:20:08.993 INFO:teuthology.orchestra.run.smithi001.stdout:Setting up python3-markupsafe (2.0.1-2build1) ... 2024-09-13T11:20:09.010 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2024-09-13T11:20:09.018 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up python3-prettytable (2.5.0-2) ... 2024-09-13T11:20:09.207 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-13T11:20:09.252 INFO:teuthology.orchestra.run.smithi001.stdout:Setting up lua5.1 (5.1.5-8.1build4) ... 2024-09-13T11:20:09.286 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-09-13T11:20:09.316 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up python3-toml (0.10.2-1) ... 2024-09-13T11:20:09.370 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-09-13T11:20:09.412 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up python3-websocket (1.2.3-1) ... 2024-09-13T11:20:09.496 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up python3-natsort (8.0.2-1) ... 2024-09-13T11:20:09.576 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2024-09-13T11:20:09.651 INFO:teuthology.orchestra.run.smithi001.stdout:update-alternatives: using /usr/bin/lua5.1 to provide /usr/bin/lua (lua-interpreter) in auto mode 2024-09-13T11:20:09.685 INFO:teuthology.orchestra.run.smithi001.stdout:update-alternatives: using /usr/bin/luac5.1 to provide /usr/bin/luac (lua-compiler) in auto mode 2024-09-13T11:20:09.688 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2024-09-13T11:20:09.702 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2024-09-13T11:20:09.747 INFO:teuthology.orchestra.run.smithi001.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-09-13T11:20:09.764 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2024-09-13T11:20:09.828 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-09-13T11:20:09.889 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-09-13T11:20:09.907 INFO:teuthology.orchestra.run.smithi001.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2024-09-13T11:20:10.023 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2024-09-13T11:20:10.138 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2024-09-13T11:20:10.267 INFO:teuthology.orchestra.run.smithi001.stdout:Setting up python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-09-13T11:20:10.307 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up zip (3.0-12build2) ... 2024-09-13T11:20:10.376 INFO:teuthology.orchestra.run.smithi001.stdout:Setting up python3-natsort (8.0.2-1) ... 2024-09-13T11:20:10.380 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2024-09-13T11:20:10.434 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up python3-pygments (2.11.2+dfsg-2) ... 2024-09-13T11:20:10.524 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up python3-webob (1:1.8.6-1.1ubuntu0.1) ... 2024-09-13T11:20:10.640 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-13T11:20:10.643 INFO:teuthology.orchestra.run.smithi001.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2024-09-13T11:20:10.766 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up python3-pyasn1 (0.4.8-1) ... 2024-09-13T11:20:10.825 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2024-09-13T11:20:10.895 INFO:teuthology.orchestra.run.smithi001.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2024-09-13T11:20:11.076 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-09-13T11:20:11.077 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up python3-singledispatch (3.4.0.3-3) ... 2024-09-13T11:20:11.094 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up python3-tempita (0.5.2-6ubuntu1) ... 2024-09-13T11:20:11.212 INFO:teuthology.orchestra.run.smithi001.stdout:Setting up zip (3.0-12build2) ... 2024-09-13T11:20:11.244 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up lua-sec:amd64 (1.0.2-1) ... 2024-09-13T11:20:11.326 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up python3-logutils (0.3.3-8) ... 2024-09-13T11:20:11.345 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up python-pastedeploy-tpl (2.1.1-1) ... 2024-09-13T11:20:11.363 INFO:teuthology.orchestra.run.smithi001.stdout:Setting up python3-pygments (2.11.2+dfsg-2) ... 2024-09-13T11:20:11.369 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up python3-babel (2.8.0+dfsg.1-7) ... 2024-09-13T11:20:11.458 INFO:teuthology.orchestra.run.smithi092.stdout:update-alternatives: using /usr/bin/pybabel-python3 to provide /usr/bin/pybabel (pybabel) in auto mode 2024-09-13T11:20:11.493 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2024-09-13T11:20:11.586 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up python3-tempora (4.1.2-1) ... 2024-09-13T11:20:11.622 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2024-09-13T11:20:11.688 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-09-13T11:20:11.814 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2024-09-13T11:20:11.862 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2024-09-13T11:20:11.923 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up python3-asyncssh (2.5.0-1) ... 2024-09-13T11:20:12.066 INFO:teuthology.orchestra.run.smithi001.stdout:Setting up python3-tempita (0.5.2-6ubuntu1) ... 2024-09-13T11:20:12.113 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up python3-prettytable (2.5.0-2) ... 2024-09-13T11:20:12.207 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up python3-pastedeploy (2.1.1-1) ... 2024-09-13T11:20:12.307 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up python3-certifi (2020.6.20-1) ... 2024-09-13T11:20:12.316 INFO:teuthology.orchestra.run.smithi001.stdout:Setting up python-pastedeploy-tpl (2.1.1-1) ... 2024-09-13T11:20:12.364 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-09-13T11:20:12.434 INFO:teuthology.orchestra.run.smithi001.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2024-09-13T11:20:12.467 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up lua-any (27ubuntu1) ... 2024-09-13T11:20:12.490 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up python3-websocket (1.2.3-1) ... 2024-09-13T11:20:12.550 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up python3-paste (3.5.0+dfsg1-1) ... 2024-09-13T11:20:12.560 INFO:teuthology.orchestra.run.smithi001.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2024-09-13T11:20:12.576 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up python3-portend (3.0.0-1) ... 2024-09-13T11:20:12.749 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2024-09-13T11:20:12.827 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-13T11:20:12.861 INFO:teuthology.orchestra.run.smithi001.stdout:Setting up python3-asyncssh (2.5.0-1) ... 2024-09-13T11:20:12.875 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-09-13T11:20:12.935 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-09-13T11:20:12.953 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up python3-jinja2 (3.0.3-1ubuntu0.2) ... 2024-09-13T11:20:13.168 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up python3-webob (1:1.8.6-1.1ubuntu0.1) ... 2024-09-13T11:20:13.203 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.2) ... 2024-09-13T11:20:13.279 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up python3-pyasn1-modules (0.2.1-1) ... 2024-09-13T11:20:13.287 INFO:teuthology.orchestra.run.smithi001.stdout:Setting up python3-certifi (2020.6.20-1) ... 2024-09-13T11:20:13.469 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2024-09-13T11:20:13.547 INFO:teuthology.orchestra.run.smithi001.stdout:Setting up python3-paste (3.5.0+dfsg1-1) ... 2024-09-13T11:20:13.554 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2024-09-13T11:20:13.579 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-09-13T11:20:13.729 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-09-13T11:20:13.814 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2024-09-13T11:20:13.829 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up lua-sec:amd64 (1.0.2-1) ... 2024-09-13T11:20:13.872 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up jq (1.6-2.1ubuntu3) ... 2024-09-13T11:20:13.931 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up python3-ceph-common (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:20:13.931 INFO:teuthology.orchestra.run.smithi001.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-09-13T11:20:13.947 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up python3-babel (2.8.0+dfsg.1-7) ... 2024-09-13T11:20:14.007 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up python3-rsa (4.8-1) ... 2024-09-13T11:20:14.037 INFO:teuthology.orchestra.run.smithi162.stdout:update-alternatives: using /usr/bin/pybabel-python3 to provide /usr/bin/pybabel (pybabel) in auto mode 2024-09-13T11:20:14.215 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up python3-idna (3.3-1ubuntu0.1) ... 2024-09-13T11:20:14.225 INFO:teuthology.orchestra.run.smithi001.stdout:Setting up python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.2) ... 2024-09-13T11:20:14.282 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-09-13T11:20:14.299 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up python3-webtest (2.0.35-1) ... 2024-09-13T11:20:14.408 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2024-09-13T11:20:14.542 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2024-09-13T11:20:14.550 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-09-13T11:20:14.551 INFO:teuthology.orchestra.run.smithi001.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2024-09-13T11:20:14.785 INFO:teuthology.orchestra.run.smithi001.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2024-09-13T11:20:14.801 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up python3-pastedeploy (2.1.1-1) ... 2024-09-13T11:20:14.879 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2024-09-13T11:20:14.903 INFO:teuthology.orchestra.run.smithi001.stdout:Setting up python3-ceph-common (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:20:15.077 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up lua-any (27ubuntu1) ... 2024-09-13T11:20:15.212 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up python3-portend (3.0.0-1) ... 2024-09-13T11:20:15.229 INFO:teuthology.orchestra.run.smithi001.stdout:Setting up python3-idna (3.3-1ubuntu0.1) ... 2024-09-13T11:20:15.255 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up python3-pastescript (2.0.2-4) ... 2024-09-13T11:20:15.488 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-13T11:20:15.540 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up python3-pecan (1.3.3-4ubuntu2) ... 2024-09-13T11:20:15.564 INFO:teuthology.orchestra.run.smithi001.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2024-09-13T11:20:15.614 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up python3-jinja2 (3.0.3-1ubuntu0.2) ... 2024-09-13T11:20:15.672 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up pkg-config (0.29.2-1ubuntu3) ... 2024-09-13T11:20:15.876 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-09-13T11:20:15.931 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up python3-pyasn1-modules (0.2.1-1) ... 2024-09-13T11:20:15.932 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-13T11:20:15.985 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up librados2 (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:20:16.074 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up python3-toml (0.10.2-1) ... 2024-09-13T11:20:16.111 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up libsqlite3-mod-ceph (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:20:16.224 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-09-13T11:20:16.254 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up luarocks (3.8.0+dfsg1-1) ... 2024-09-13T11:20:16.350 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2024-09-13T11:20:16.451 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2024-09-13T11:20:16.467 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up jq (1.6-2.1ubuntu3) ... 2024-09-13T11:20:16.569 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-09-13T11:20:16.584 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up libcephfs2 (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:20:16.593 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up python3-rsa (4.8-1) ... 2024-09-13T11:20:16.693 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up libradosstriper1 (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:20:16.745 INFO:teuthology.orchestra.run.smithi001.stdout:Setting up pkg-config (0.29.2-1ubuntu3) ... 2024-09-13T11:20:16.811 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up python3-google-auth (1.5.1-3) ... 2024-09-13T11:20:16.862 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2024-09-13T11:20:16.869 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up python3-webtest (2.0.35-1) ... 2024-09-13T11:20:16.996 INFO:teuthology.orchestra.run.smithi001.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-13T11:20:17.088 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up librbd1 (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:20:17.122 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2024-09-13T11:20:17.130 INFO:teuthology.orchestra.run.smithi001.stdout:Setting up python3-toml (0.10.2-1) ... 2024-09-13T11:20:17.137 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-09-13T11:20:17.213 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up ceph-mgr-modules-core (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:20:17.348 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-13T11:20:17.356 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up ceph-fuse (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:20:17.390 INFO:teuthology.orchestra.run.smithi001.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2024-09-13T11:20:17.465 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2024-09-13T11:20:17.473 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up python3-pyasn1 (0.4.8-1) ... 2024-09-13T11:20:17.492 INFO:teuthology.orchestra.run.smithi001.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2024-09-13T11:20:17.570 INFO:teuthology.orchestra.run.smithi092.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-09-13T11:20:17.570 INFO:teuthology.orchestra.run.smithi092.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-09-13T11:20:17.592 INFO:teuthology.orchestra.run.smithi001.stdout:Setting up python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-09-13T11:20:17.766 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up python3-singledispatch (3.4.0.3-3) ... 2024-09-13T11:20:17.842 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up python3-pastescript (2.0.2-4) ... 2024-09-13T11:20:17.910 INFO:teuthology.orchestra.run.smithi001.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2024-09-13T11:20:18.026 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up python3-logutils (0.3.3-8) ... 2024-09-13T11:20:18.074 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up libcephfs-dev (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:20:18.135 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up python3-pecan (1.3.3-4ubuntu2) ... 2024-09-13T11:20:18.178 INFO:teuthology.orchestra.run.smithi001.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2024-09-13T11:20:18.200 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up python3-rados (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:20:18.285 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up python3-tempora (4.1.2-1) ... 2024-09-13T11:20:18.301 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up librgw2 (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:20:18.427 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2024-09-13T11:20:18.470 INFO:teuthology.orchestra.run.smithi001.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-13T11:20:18.471 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-09-13T11:20:18.553 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2024-09-13T11:20:18.596 INFO:teuthology.orchestra.run.smithi001.stdout:Setting up python3-pyasn1 (0.4.8-1) ... 2024-09-13T11:20:18.605 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up librados2 (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:20:18.731 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up libsqlite3-mod-ceph (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:20:18.796 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up python3-prettytable (2.5.0-2) ... 2024-09-13T11:20:18.857 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up luarocks (3.8.0+dfsg1-1) ... 2024-09-13T11:20:18.898 INFO:teuthology.orchestra.run.smithi001.stdout:Setting up python3-singledispatch (3.4.0.3-3) ... 2024-09-13T11:20:19.055 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-09-13T11:20:19.156 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up python3-websocket (1.2.3-1) ... 2024-09-13T11:20:19.160 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up libcephfs2 (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:20:19.165 INFO:teuthology.orchestra.run.smithi001.stdout:Setting up python3-logutils (0.3.3-8) ... 2024-09-13T11:20:19.286 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up libradosstriper1 (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:20:19.412 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up python3-google-auth (1.5.1-3) ... 2024-09-13T11:20:19.415 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2024-09-13T11:20:19.425 INFO:teuthology.orchestra.run.smithi001.stdout:Setting up python3-tempora (4.1.2-1) ... 2024-09-13T11:20:19.541 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-09-13T11:20:19.589 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up python3-rbd (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:20:19.668 INFO:teuthology.orchestra.run.smithi001.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2024-09-13T11:20:19.688 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up librbd1 (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:20:19.715 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up rbd-fuse (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:20:19.815 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up ceph-mgr-modules-core (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:20:19.834 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up python3-webob (1:1.8.6-1.1ubuntu0.1) ... 2024-09-13T11:20:19.841 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up python3-rgw (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:20:19.910 INFO:teuthology.orchestra.run.smithi001.stdout:Setting up python3-prettytable (2.5.0-2) ... 2024-09-13T11:20:19.941 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up ceph-fuse (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:20:19.958 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up python3-cephfs (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:20:20.060 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up ceph-common (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:20:20.127 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2024-09-13T11:20:20.152 INFO:teuthology.orchestra.run.smithi162.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-09-13T11:20:20.152 INFO:teuthology.orchestra.run.smithi162.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-09-13T11:20:20.178 INFO:teuthology.orchestra.run.smithi001.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-09-13T11:20:20.304 INFO:teuthology.orchestra.run.smithi001.stdout:Setting up python3-websocket (1.2.3-1) ... 2024-09-13T11:20:20.386 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-09-13T11:20:20.512 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up lua-sec:amd64 (1.0.2-1) ... 2024-09-13T11:20:20.588 INFO:teuthology.orchestra.run.smithi001.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2024-09-13T11:20:20.594 INFO:teuthology.orchestra.run.smithi092.stdout:Adding group ceph....done 2024-09-13T11:20:20.608 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up libcephfs-dev (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:20:20.630 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up python3-babel (2.8.0+dfsg.1-7) ... 2024-09-13T11:20:20.703 INFO:teuthology.orchestra.run.smithi177.stdout:update-alternatives: using /usr/bin/pybabel-python3 to provide /usr/bin/pybabel (pybabel) in auto mode 2024-09-13T11:20:20.714 INFO:teuthology.orchestra.run.smithi001.stdout:Setting up python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-09-13T11:20:20.734 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up python3-rados (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:20:20.860 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up librgw2 (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:20:20.931 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-09-13T11:20:20.962 INFO:teuthology.orchestra.run.smithi092.stdout:Adding system user ceph....done 2024-09-13T11:20:20.995 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2024-09-13T11:20:21.024 INFO:teuthology.orchestra.run.smithi001.stdout:Setting up python3-webob (1:1.8.6-1.1ubuntu0.1) ... 2024-09-13T11:20:21.041 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2024-09-13T11:20:21.106 INFO:teuthology.orchestra.run.smithi092.stdout:Setting system user ceph properties....done 2024-09-13T11:20:21.121 INFO:teuthology.orchestra.run.smithi092.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2024-09-13T11:20:21.263 INFO:teuthology.orchestra.run.smithi092.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2024-09-13T11:20:21.375 INFO:teuthology.orchestra.run.smithi001.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2024-09-13T11:20:21.434 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up python3-pastedeploy (2.1.1-1) ... 2024-09-13T11:20:21.593 INFO:teuthology.orchestra.run.smithi092.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2024-09-13T11:20:21.635 INFO:teuthology.orchestra.run.smithi001.stdout:Setting up liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-09-13T11:20:21.693 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up lua-any (27ubuntu1) ... 2024-09-13T11:20:21.735 INFO:teuthology.orchestra.run.smithi001.stdout:Setting up lua-sec:amd64 (1.0.2-1) ... 2024-09-13T11:20:21.836 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up python3-portend (3.0.0-1) ... 2024-09-13T11:20:21.845 INFO:teuthology.orchestra.run.smithi001.stdout:Setting up python3-babel (2.8.0+dfsg.1-7) ... 2024-09-13T11:20:21.934 INFO:teuthology.orchestra.run.smithi001.stdout:update-alternatives: using /usr/bin/pybabel-python3 to provide /usr/bin/pybabel (pybabel) in auto mode 2024-09-13T11:20:22.086 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up ceph-test (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:20:22.095 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-13T11:20:22.205 INFO:teuthology.orchestra.run.smithi001.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-09-13T11:20:22.211 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up radosgw (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:20:22.221 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up python3-jinja2 (3.0.3-1ubuntu0.2) ... 2024-09-13T11:20:22.240 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up python3-rbd (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:20:22.331 INFO:teuthology.orchestra.run.smithi001.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2024-09-13T11:20:22.366 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up rbd-fuse (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:20:22.517 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up python3-rgw (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:20:22.539 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up python3-pyasn1-modules (0.2.1-1) ... 2024-09-13T11:20:22.643 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up python3-cephfs (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:20:22.694 INFO:teuthology.orchestra.run.smithi092.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-09-13T11:20:22.694 INFO:teuthology.orchestra.run.smithi092.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-09-13T11:20:22.716 INFO:teuthology.orchestra.run.smithi001.stdout:Setting up python3-pastedeploy (2.1.1-1) ... 2024-09-13T11:20:22.761 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up ceph-common (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:20:22.815 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-09-13T11:20:22.975 INFO:teuthology.orchestra.run.smithi001.stdout:Setting up lua-any (27ubuntu1) ... 2024-09-13T11:20:23.066 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up jq (1.6-2.1ubuntu3) ... 2024-09-13T11:20:23.076 INFO:teuthology.orchestra.run.smithi001.stdout:Setting up python3-portend (3.0.0-1) ... 2024-09-13T11:20:23.171 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up ceph-base (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:20:23.192 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up python3-rsa (4.8-1) ... 2024-09-13T11:20:23.280 INFO:teuthology.orchestra.run.smithi162.stdout:Adding group ceph....done 2024-09-13T11:20:23.327 INFO:teuthology.orchestra.run.smithi001.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-13T11:20:23.453 INFO:teuthology.orchestra.run.smithi001.stdout:Setting up python3-jinja2 (3.0.3-1ubuntu0.2) ... 2024-09-13T11:20:23.468 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up python3-webtest (2.0.35-1) ... 2024-09-13T11:20:23.494 INFO:teuthology.orchestra.run.smithi092.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2024-09-13T11:20:23.649 INFO:teuthology.orchestra.run.smithi162.stdout:Adding system user ceph....done 2024-09-13T11:20:23.752 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-09-13T11:20:23.787 INFO:teuthology.orchestra.run.smithi001.stdout:Setting up python3-pyasn1-modules (0.2.1-1) ... 2024-09-13T11:20:23.809 INFO:teuthology.orchestra.run.smithi162.stdout:Setting system user ceph properties....done 2024-09-13T11:20:23.826 INFO:teuthology.orchestra.run.smithi162.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2024-09-13T11:20:23.961 INFO:teuthology.orchestra.run.smithi162.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2024-09-13T11:20:23.999 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up ceph-mds (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:20:24.097 INFO:teuthology.orchestra.run.smithi001.stdout:Setting up python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-09-13T11:20:24.114 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2024-09-13T11:20:24.198 INFO:teuthology.orchestra.run.smithi092.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-09-13T11:20:24.199 INFO:teuthology.orchestra.run.smithi092.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-09-13T11:20:24.282 INFO:teuthology.orchestra.run.smithi162.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2024-09-13T11:20:24.373 INFO:teuthology.orchestra.run.smithi001.stdout:Setting up jq (1.6-2.1ubuntu3) ... 2024-09-13T11:20:24.474 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up python3-pastescript (2.0.2-4) ... 2024-09-13T11:20:24.474 INFO:teuthology.orchestra.run.smithi001.stdout:Setting up python3-rsa (4.8-1) ... 2024-09-13T11:20:24.676 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up ceph-mgr (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:20:24.734 INFO:teuthology.orchestra.run.smithi001.stdout:Setting up python3-webtest (2.0.35-1) ... 2024-09-13T11:20:24.758 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up python3-pecan (1.3.3-4ubuntu2) ... 2024-09-13T11:20:24.763 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up ceph-test (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:20:24.889 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up radosgw (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:20:24.893 INFO:teuthology.orchestra.run.smithi092.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-09-13T11:20:24.893 INFO:teuthology.orchestra.run.smithi092.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-09-13T11:20:24.984 INFO:teuthology.orchestra.run.smithi001.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-09-13T11:20:25.086 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-09-13T11:20:25.204 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up librados2 (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:20:25.321 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up libsqlite3-mod-ceph (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:20:25.329 INFO:teuthology.orchestra.run.smithi001.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2024-09-13T11:20:25.358 INFO:teuthology.orchestra.run.smithi162.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-09-13T11:20:25.358 INFO:teuthology.orchestra.run.smithi162.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-09-13T11:20:25.368 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up ceph-osd (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:20:25.431 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up luarocks (3.8.0+dfsg1-1) ... 2024-09-13T11:20:25.646 INFO:teuthology.orchestra.run.smithi092.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-09-13T11:20:25.646 INFO:teuthology.orchestra.run.smithi092.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-09-13T11:20:25.731 INFO:teuthology.orchestra.run.smithi001.stdout:Setting up python3-pastescript (2.0.2-4) ... 2024-09-13T11:20:25.767 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up libcephfs2 (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:20:25.824 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up ceph-base (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:20:25.893 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up libradosstriper1 (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:20:26.019 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up python3-google-auth (1.5.1-3) ... 2024-09-13T11:20:26.041 INFO:teuthology.orchestra.run.smithi001.stdout:Setting up python3-pecan (1.3.3-4ubuntu2) ... 2024-09-13T11:20:26.129 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up ceph-mgr-k8sevents (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:20:26.196 INFO:teuthology.orchestra.run.smithi162.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2024-09-13T11:20:26.238 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up ceph-mgr-diskprediction-local (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:20:26.303 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up librbd1 (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:20:26.364 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up ceph-mon (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:20:26.368 INFO:teuthology.orchestra.run.smithi001.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-09-13T11:20:26.430 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up ceph-mgr-modules-core (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:20:26.495 INFO:teuthology.orchestra.run.smithi001.stdout:Setting up librados2 (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:20:26.539 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up ceph-fuse (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:20:26.558 INFO:teuthology.orchestra.run.smithi092.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-09-13T11:20:26.558 INFO:teuthology.orchestra.run.smithi092.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-09-13T11:20:26.637 INFO:teuthology.orchestra.run.smithi001.stdout:Setting up libsqlite3-mod-ceph (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:20:26.659 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up ceph-mds (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:20:26.737 INFO:teuthology.orchestra.run.smithi177.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-09-13T11:20:26.737 INFO:teuthology.orchestra.run.smithi177.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-09-13T11:20:26.763 INFO:teuthology.orchestra.run.smithi001.stdout:Setting up luarocks (3.8.0+dfsg1-1) ... 2024-09-13T11:20:26.905 INFO:teuthology.orchestra.run.smithi162.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-09-13T11:20:26.906 INFO:teuthology.orchestra.run.smithi162.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-09-13T11:20:27.041 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up ceph-mgr-cephadm (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:20:27.058 INFO:teuthology.orchestra.run.smithi001.stdout:Setting up libcephfs2 (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:20:27.167 INFO:teuthology.orchestra.run.smithi001.stdout:Setting up libradosstriper1 (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:20:27.183 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up ceph (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:20:27.207 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up libcephfs-dev (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:20:27.293 INFO:teuthology.orchestra.run.smithi001.stdout:Setting up python3-google-auth (1.5.1-3) ... 2024-09-13T11:20:27.309 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up ceph-mgr-dashboard (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:20:27.324 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up python3-rados (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:20:27.378 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up ceph-mgr (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:20:27.450 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up librgw2 (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:20:27.460 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up ceph-volume (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:20:27.570 INFO:teuthology.orchestra.run.smithi001.stdout:Setting up librbd1 (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:20:27.576 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2024-09-13T11:20:27.587 INFO:teuthology.orchestra.run.smithi162.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-09-13T11:20:27.587 INFO:teuthology.orchestra.run.smithi162.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-09-13T11:20:27.696 INFO:teuthology.orchestra.run.smithi001.stdout:Setting up ceph-mgr-modules-core (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:20:27.822 INFO:teuthology.orchestra.run.smithi001.stdout:Setting up ceph-fuse (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:20:27.880 INFO:teuthology.orchestra.run.smithi092.stdout:Processing triggers for mailcap (3.70+nmu1ubuntu1) ... 2024-09-13T11:20:27.981 INFO:teuthology.orchestra.run.smithi092.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2024-09-13T11:20:28.040 INFO:teuthology.orchestra.run.smithi001.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-09-13T11:20:28.040 INFO:teuthology.orchestra.run.smithi001.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-09-13T11:20:28.062 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up ceph-osd (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:20:28.125 INFO:teuthology.orchestra.run.smithi092.stdout:Processing triggers for man-db (2.10.2-1) ... 2024-09-13T11:20:28.333 INFO:teuthology.orchestra.run.smithi162.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-09-13T11:20:28.334 INFO:teuthology.orchestra.run.smithi162.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-09-13T11:20:28.523 INFO:teuthology.orchestra.run.smithi001.stdout:Setting up libcephfs-dev (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:20:28.625 INFO:teuthology.orchestra.run.smithi001.stdout:Setting up python3-rados (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:20:28.734 INFO:teuthology.orchestra.run.smithi001.stdout:Setting up librgw2 (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:20:28.821 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up python3-rbd (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:20:28.860 INFO:teuthology.orchestra.run.smithi001.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2024-09-13T11:20:28.869 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up ceph-mgr-k8sevents (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:20:28.956 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up rbd-fuse (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:20:29.082 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up python3-rgw (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:20:29.104 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up ceph-mgr-diskprediction-local (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:20:29.208 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up python3-cephfs (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:20:29.275 INFO:teuthology.orchestra.run.smithi092.stdout:Processing triggers for install-info (6.8-4build1) ... 2024-09-13T11:20:29.315 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up ceph-mon (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:20:29.333 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up ceph-common (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:20:29.583 INFO:teuthology.orchestra.run.smithi162.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-09-13T11:20:29.583 INFO:teuthology.orchestra.run.smithi162.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-09-13T11:20:29.876 INFO:teuthology.orchestra.run.smithi177.stdout:Adding group ceph....done 2024-09-13T11:20:29.934 INFO:teuthology.orchestra.run.smithi092.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-09-13T11:20:29.941 DEBUG:teuthology.parallel:result is None 2024-09-13T11:20:30.042 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up ceph-mgr-cephadm (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:20:30.163 INFO:teuthology.orchestra.run.smithi001.stdout:Setting up python3-rbd (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:20:30.185 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up ceph (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:20:30.227 INFO:teuthology.orchestra.run.smithi177.stdout:Adding system user ceph....done 2024-09-13T11:20:30.289 INFO:teuthology.orchestra.run.smithi001.stdout:Setting up rbd-fuse (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:20:30.294 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up ceph-mgr-dashboard (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:20:30.390 INFO:teuthology.orchestra.run.smithi001.stdout:Setting up python3-rgw (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:20:30.396 INFO:teuthology.orchestra.run.smithi177.stdout:Setting system user ceph properties....done 2024-09-13T11:20:30.411 INFO:teuthology.orchestra.run.smithi177.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2024-09-13T11:20:30.500 INFO:teuthology.orchestra.run.smithi001.stdout:Setting up python3-cephfs (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:20:30.544 INFO:teuthology.orchestra.run.smithi177.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2024-09-13T11:20:30.626 INFO:teuthology.orchestra.run.smithi001.stdout:Setting up ceph-common (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:20:30.843 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up ceph-volume (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:20:30.844 INFO:teuthology.orchestra.run.smithi177.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2024-09-13T11:20:31.275 INFO:teuthology.orchestra.run.smithi162.stdout:Processing triggers for mailcap (3.70+nmu1ubuntu1) ... 2024-09-13T11:20:31.335 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up ceph-test (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:20:31.375 INFO:teuthology.orchestra.run.smithi162.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2024-09-13T11:20:31.428 INFO:teuthology.orchestra.run.smithi001.stdout:Adding group ceph....done 2024-09-13T11:20:31.469 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up radosgw (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:20:31.520 INFO:teuthology.orchestra.run.smithi162.stdout:Processing triggers for man-db (2.10.2-1) ... 2024-09-13T11:20:31.796 INFO:teuthology.orchestra.run.smithi001.stdout:Adding system user ceph....done 2024-09-13T11:20:31.956 INFO:teuthology.orchestra.run.smithi001.stdout:Setting system user ceph properties....done 2024-09-13T11:20:31.969 INFO:teuthology.orchestra.run.smithi001.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2024-09-13T11:20:32.125 INFO:teuthology.orchestra.run.smithi001.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2024-09-13T11:20:32.269 INFO:teuthology.orchestra.run.smithi162.stdout:Processing triggers for install-info (6.8-4build1) ... 2024-09-13T11:20:32.281 INFO:teuthology.orchestra.run.smithi177.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-09-13T11:20:32.281 INFO:teuthology.orchestra.run.smithi177.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-09-13T11:20:32.449 INFO:teuthology.orchestra.run.smithi001.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2024-09-13T11:20:32.763 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up ceph-base (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:20:32.912 INFO:teuthology.orchestra.run.smithi162.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-09-13T11:20:32.918 DEBUG:teuthology.parallel:result is None 2024-09-13T11:20:32.960 INFO:teuthology.orchestra.run.smithi001.stdout:Setting up ceph-test (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:20:33.077 INFO:teuthology.orchestra.run.smithi177.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2024-09-13T11:20:33.105 INFO:teuthology.orchestra.run.smithi001.stdout:Setting up radosgw (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:20:33.540 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up ceph-mds (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:20:33.644 INFO:teuthology.orchestra.run.smithi001.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-09-13T11:20:33.644 INFO:teuthology.orchestra.run.smithi001.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-09-13T11:20:33.763 INFO:teuthology.orchestra.run.smithi177.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-09-13T11:20:33.763 INFO:teuthology.orchestra.run.smithi177.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-09-13T11:20:34.080 INFO:teuthology.orchestra.run.smithi001.stdout:Setting up ceph-base (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:20:34.233 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up ceph-mgr (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:20:34.439 INFO:teuthology.orchestra.run.smithi001.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2024-09-13T11:20:34.461 INFO:teuthology.orchestra.run.smithi177.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-09-13T11:20:34.462 INFO:teuthology.orchestra.run.smithi177.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-09-13T11:20:34.923 INFO:teuthology.orchestra.run.smithi001.stdout:Setting up ceph-mds (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:20:35.037 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up ceph-osd (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:20:35.169 INFO:teuthology.orchestra.run.smithi001.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-09-13T11:20:35.169 INFO:teuthology.orchestra.run.smithi001.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-09-13T11:20:35.317 INFO:teuthology.orchestra.run.smithi177.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-09-13T11:20:35.318 INFO:teuthology.orchestra.run.smithi177.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-09-13T11:20:35.683 INFO:teuthology.orchestra.run.smithi001.stdout:Setting up ceph-mgr (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:20:35.786 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up ceph-mgr-k8sevents (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:20:35.895 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up ceph-mgr-diskprediction-local (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:20:35.904 INFO:teuthology.orchestra.run.smithi001.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-09-13T11:20:35.904 INFO:teuthology.orchestra.run.smithi001.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-09-13T11:20:36.022 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up ceph-mon (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:20:36.251 INFO:teuthology.orchestra.run.smithi177.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-09-13T11:20:36.251 INFO:teuthology.orchestra.run.smithi177.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-09-13T11:20:36.377 INFO:teuthology.orchestra.run.smithi001.stdout:Setting up ceph-osd (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:20:36.659 INFO:teuthology.orchestra.run.smithi001.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-09-13T11:20:36.659 INFO:teuthology.orchestra.run.smithi001.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-09-13T11:20:37.121 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up ceph-mgr-cephadm (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:20:37.170 INFO:teuthology.orchestra.run.smithi001.stdout:Setting up ceph-mgr-k8sevents (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:20:37.277 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up ceph (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:20:37.296 INFO:teuthology.orchestra.run.smithi001.stdout:Setting up ceph-mgr-diskprediction-local (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:20:37.403 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up ceph-mgr-dashboard (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:20:37.473 INFO:teuthology.orchestra.run.smithi001.stdout:Setting up ceph-mon (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:20:37.546 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up ceph-volume (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:20:37.688 INFO:teuthology.orchestra.run.smithi001.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-09-13T11:20:37.688 INFO:teuthology.orchestra.run.smithi001.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-09-13T11:20:37.882 INFO:teuthology.orchestra.run.smithi177.stdout:Processing triggers for mailcap (3.70+nmu1ubuntu1) ... 2024-09-13T11:20:37.999 INFO:teuthology.orchestra.run.smithi177.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2024-09-13T11:20:38.168 INFO:teuthology.orchestra.run.smithi177.stdout:Processing triggers for man-db (2.10.2-1) ... 2024-09-13T11:20:38.266 INFO:teuthology.orchestra.run.smithi001.stdout:Setting up ceph-mgr-cephadm (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:20:38.538 INFO:teuthology.orchestra.run.smithi001.stdout:Setting up ceph (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:20:38.766 INFO:teuthology.orchestra.run.smithi001.stdout:Setting up ceph-mgr-dashboard (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:20:39.011 INFO:teuthology.orchestra.run.smithi001.stdout:Setting up ceph-volume (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:20:39.243 INFO:teuthology.orchestra.run.smithi177.stdout:Processing triggers for install-info (6.8-4build1) ... 2024-09-13T11:20:39.475 INFO:teuthology.orchestra.run.smithi001.stdout:Processing triggers for mailcap (3.70+nmu1ubuntu1) ... 2024-09-13T11:20:39.575 INFO:teuthology.orchestra.run.smithi001.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2024-09-13T11:20:39.727 INFO:teuthology.orchestra.run.smithi001.stdout:Processing triggers for man-db (2.10.2-1) ... 2024-09-13T11:20:39.941 INFO:teuthology.orchestra.run.smithi177.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-09-13T11:20:39.946 DEBUG:teuthology.parallel:result is None 2024-09-13T11:20:40.444 INFO:teuthology.orchestra.run.smithi001.stdout:Processing triggers for install-info (6.8-4build1) ... 2024-09-13T11:20:41.113 INFO:teuthology.orchestra.run.smithi001.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-09-13T11:20:41.117 DEBUG:teuthology.parallel:result is None 2024-09-13T11:20:41.118 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=d58f73069a8157a9ace12a0247ee3e8497707ea9 2024-09-13T11:20:41.275 DEBUG:teuthology.orchestra.run.smithi001:> dpkg-query -W -f '${Version}' ceph 2024-09-13T11:20:41.295 INFO:teuthology.orchestra.run.smithi001.stdout:19.3.0-4697-gd58f7306-1jammy 2024-09-13T11:20:41.295 INFO:teuthology.packaging:The installed version of ceph is 19.3.0-4697-gd58f7306-1jammy 2024-09-13T11:20:41.296 INFO:teuthology.task.install:The correct ceph version 19.3.0-4697-gd58f7306-1jammy is installed. 2024-09-13T11:20:41.298 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=d58f73069a8157a9ace12a0247ee3e8497707ea9 2024-09-13T11:20:41.439 DEBUG:teuthology.orchestra.run.smithi092:> dpkg-query -W -f '${Version}' ceph 2024-09-13T11:20:41.458 INFO:teuthology.orchestra.run.smithi092.stdout:19.3.0-4697-gd58f7306-1jammy 2024-09-13T11:20:41.458 INFO:teuthology.packaging:The installed version of ceph is 19.3.0-4697-gd58f7306-1jammy 2024-09-13T11:20:41.458 INFO:teuthology.task.install:The correct ceph version 19.3.0-4697-gd58f7306-1jammy is installed. 2024-09-13T11:20:41.460 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=d58f73069a8157a9ace12a0247ee3e8497707ea9 2024-09-13T11:20:41.600 DEBUG:teuthology.orchestra.run.smithi162:> dpkg-query -W -f '${Version}' ceph 2024-09-13T11:20:41.619 INFO:teuthology.orchestra.run.smithi162.stdout:19.3.0-4697-gd58f7306-1jammy 2024-09-13T11:20:41.619 INFO:teuthology.packaging:The installed version of ceph is 19.3.0-4697-gd58f7306-1jammy 2024-09-13T11:20:41.619 INFO:teuthology.task.install:The correct ceph version 19.3.0-4697-gd58f7306-1jammy is installed. 2024-09-13T11:20:41.621 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=d58f73069a8157a9ace12a0247ee3e8497707ea9 2024-09-13T11:20:41.764 DEBUG:teuthology.orchestra.run.smithi177:> dpkg-query -W -f '${Version}' ceph 2024-09-13T11:20:41.783 INFO:teuthology.orchestra.run.smithi177.stdout:19.3.0-4697-gd58f7306-1jammy 2024-09-13T11:20:41.783 INFO:teuthology.packaging:The installed version of ceph is 19.3.0-4697-gd58f7306-1jammy 2024-09-13T11:20:41.783 INFO:teuthology.task.install:The correct ceph version 19.3.0-4697-gd58f7306-1jammy is installed. 2024-09-13T11:20:41.786 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-09-13T11:20:41.786 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-09-13T11:20:41.786 DEBUG:teuthology.orchestra.run.smithi001:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-13T11:20:41.801 DEBUG:teuthology.orchestra.run.smithi092:> set -ex 2024-09-13T11:20:41.801 DEBUG:teuthology.orchestra.run.smithi092:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-13T11:20:41.815 DEBUG:teuthology.orchestra.run.smithi162:> set -ex 2024-09-13T11:20:41.816 DEBUG:teuthology.orchestra.run.smithi162:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-13T11:20:41.829 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-09-13T11:20:41.829 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-13T11:20:41.843 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-09-13T11:20:41.843 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-09-13T11:20:41.844 DEBUG:teuthology.orchestra.run.smithi001:> sudo dd of=/usr/bin/daemon-helper 2024-09-13T11:20:41.857 DEBUG:teuthology.orchestra.run.smithi001:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-13T11:20:41.912 DEBUG:teuthology.orchestra.run.smithi092:> set -ex 2024-09-13T11:20:41.912 DEBUG:teuthology.orchestra.run.smithi092:> sudo dd of=/usr/bin/daemon-helper 2024-09-13T11:20:41.925 DEBUG:teuthology.orchestra.run.smithi092:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-13T11:20:41.978 DEBUG:teuthology.orchestra.run.smithi162:> set -ex 2024-09-13T11:20:41.979 DEBUG:teuthology.orchestra.run.smithi162:> sudo dd of=/usr/bin/daemon-helper 2024-09-13T11:20:41.992 DEBUG:teuthology.orchestra.run.smithi162:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-13T11:20:42.048 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-09-13T11:20:42.048 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd of=/usr/bin/daemon-helper 2024-09-13T11:20:42.062 DEBUG:teuthology.orchestra.run.smithi177:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-13T11:20:42.121 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-09-13T11:20:42.121 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-09-13T11:20:42.121 DEBUG:teuthology.orchestra.run.smithi001:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-13T11:20:42.136 DEBUG:teuthology.orchestra.run.smithi001:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-13T11:20:42.193 DEBUG:teuthology.orchestra.run.smithi092:> set -ex 2024-09-13T11:20:42.193 DEBUG:teuthology.orchestra.run.smithi092:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-13T11:20:42.206 DEBUG:teuthology.orchestra.run.smithi092:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-13T11:20:42.263 DEBUG:teuthology.orchestra.run.smithi162:> set -ex 2024-09-13T11:20:42.263 DEBUG:teuthology.orchestra.run.smithi162:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-13T11:20:42.276 DEBUG:teuthology.orchestra.run.smithi162:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-13T11:20:42.331 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-09-13T11:20:42.332 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-13T11:20:42.345 DEBUG:teuthology.orchestra.run.smithi177:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-13T11:20:42.401 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-09-13T11:20:42.401 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-09-13T11:20:42.401 DEBUG:teuthology.orchestra.run.smithi001:> sudo dd of=/usr/bin/stdin-killer 2024-09-13T11:20:42.415 DEBUG:teuthology.orchestra.run.smithi001:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-13T11:20:42.469 DEBUG:teuthology.orchestra.run.smithi092:> set -ex 2024-09-13T11:20:42.469 DEBUG:teuthology.orchestra.run.smithi092:> sudo dd of=/usr/bin/stdin-killer 2024-09-13T11:20:42.483 DEBUG:teuthology.orchestra.run.smithi092:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-13T11:20:42.539 DEBUG:teuthology.orchestra.run.smithi162:> set -ex 2024-09-13T11:20:42.539 DEBUG:teuthology.orchestra.run.smithi162:> sudo dd of=/usr/bin/stdin-killer 2024-09-13T11:20:42.553 DEBUG:teuthology.orchestra.run.smithi162:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-13T11:20:42.607 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-09-13T11:20:42.607 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd of=/usr/bin/stdin-killer 2024-09-13T11:20:42.621 DEBUG:teuthology.orchestra.run.smithi177:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-13T11:20:42.677 INFO:teuthology.run_tasks:Running task ceph... 2024-09-13T11:20:42.766 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-09-13T11:20:42.766 DEBUG:teuthology.orchestra.run.smithi001:> sudo chmod 777 /var/log/ceph 2024-09-13T11:20:42.769 DEBUG:teuthology.orchestra.run.smithi092:> sudo chmod 777 /var/log/ceph 2024-09-13T11:20:42.771 DEBUG:teuthology.orchestra.run.smithi162:> sudo chmod 777 /var/log/ceph 2024-09-13T11:20:42.773 DEBUG:teuthology.orchestra.run.smithi177:> sudo chmod 777 /var/log/ceph 2024-09-13T11:20:42.786 INFO:tasks.ceph:Disabling ceph logrotate... 2024-09-13T11:20:42.786 DEBUG:teuthology.orchestra.run.smithi001:> sudo rm -f -- /etc/logrotate.d/ceph 2024-09-13T11:20:42.826 DEBUG:teuthology.orchestra.run.smithi092:> sudo rm -f -- /etc/logrotate.d/ceph 2024-09-13T11:20:42.828 DEBUG:teuthology.orchestra.run.smithi162:> sudo rm -f -- /etc/logrotate.d/ceph 2024-09-13T11:20:42.830 DEBUG:teuthology.orchestra.run.smithi177:> sudo rm -f -- /etc/logrotate.d/ceph 2024-09-13T11:20:42.843 INFO:tasks.ceph:Creating extra log directories... 2024-09-13T11:20:42.843 DEBUG:teuthology.orchestra.run.smithi001:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-09-13T11:20:42.882 DEBUG:teuthology.orchestra.run.smithi092:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-09-13T11:20:42.884 DEBUG:teuthology.orchestra.run.smithi162:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-09-13T11:20:42.887 DEBUG:teuthology.orchestra.run.smithi177:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-09-13T11:20:42.912 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-09-13T11:20:42.912 INFO:tasks.ceph:config {'conf': {'osd': {'debug monc': 20, 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore write v2': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd backoff on peering': True, 'osd blocked scrub grace period': 3600, 'osd debug reject backfill probability': 0.3, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd delete sleep': 1, 'osd failsafe full ratio': 0.95, 'osd inject bad map crc probability': 0.1, 'osd max backfills': 3, 'osd max markdown count': 1000, 'osd max scrubs': 5, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd snap trim sleep': 2}, 'client.0': {'admin socket': '/var/run/ceph/ceph-$name.asok'}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms bind msgr2': False, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'osd', 'ms inject internal delays': 0.002, 'ms inject socket failures': 2500, 'ms type': 'async', 'osd_pool_default_min_size': 2, 'osd_pool_default_size': 2}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'paxos service trim min': 10}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(REQUEST_STUCK\\)', 'failed to encode map', '\\(OSD_SLOW_PING_TIME', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', 'SLOW_OPS', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', 'mons down', 'mon down', 'out of quorum', 'noscrub', 'nodeep-scrub', '\\(POOL_APP_NOT_ENABLED\\)'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': False, 'mon_bind_addrvec': True} 2024-09-13T11:20:42.913 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/akupczyk-2024-09-13_09:37:56-rados-wip-aclamk-cbt-combined-distro-default-smithi/7903409', 'branch': 'wip-aclamk-cbt-combined', 'description': 'rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{max-simultaneous-scrubs-5} backoff/peering ceph clusters/{fixed-4 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/classic msgr-failures/osd-delay msgr/async-v1only objectstore/{bluestore-options/write$/{write_v2} bluestore/bluestore-stupid} rados supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/admin_socket_objecter_requests}', 'email': 'akupczyk@redhat.com', 'first_in_suite': False, 'job_id': '7903409', 'kernel': {'kdb': 1, 'sha1': 'distro'}, 'ktype': 'distro', 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'akupczyk-2024-09-13_09:37:56-rados-wip-aclamk-cbt-combined-distro-default-smithi', 'no_nested_subset': False, 'openstack': [{'volumes': {'count': 4, 'size': 10}}, {'volumes': {'count': 4, 'size': 30}}], 'os_type': 'ubuntu', 'os_version': '22.04', 'overrides': {'admin_socket': {'branch': 'wip-aclamk-cbt-combined'}, 'ceph': {'conf': {'client.0': {'admin socket': '/var/run/ceph/ceph-$name.asok'}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms bind msgr2': False, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'osd', 'ms inject internal delays': 0.002, 'ms inject socket failures': 2500, 'ms type': 'async', 'osd_pool_default_min_size': 2, 'osd_pool_default_size': 2}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'paxos service trim min': 10}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore write v2': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd backoff on peering': True, 'osd blocked scrub grace period': 3600, 'osd debug reject backfill probability': 0.3, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd delete sleep': 1, 'osd failsafe full ratio': 0.95, 'osd inject bad map crc probability': 0.1, 'osd max backfills': 3, 'osd max markdown count': 1000, 'osd max scrubs': 5, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd snap trim sleep': 2}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(REQUEST_STUCK\\)', 'failed to encode map', '\\(OSD_SLOW_PING_TIME', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', 'SLOW_OPS', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', 'mons down', 'mon down', 'out of quorum', 'noscrub', 'nodeep-scrub', '\\(POOL_APP_NOT_ENABLED\\)'], 'mon_bind_msgr2': False, 'sha1': 'd58f73069a8157a9ace12a0247ee3e8497707ea9'}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'flavor': 'default', 'sha1': 'd58f73069a8157a9ace12a0247ee3e8497707ea9'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-aclamk-cbt-combined', 'sha1': 'd58f73069a8157a9ace12a0247ee3e8497707ea9'}}, 'owner': 'scheduled_akupczyk@teuthology', 'priority': 160, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mgr.y', 'osd.0', 'osd.4', 'osd.8', 'osd.12'], ['mon.b', 'osd.1', 'osd.5', 'osd.9', 'osd.13'], ['mon.c', 'osd.2', 'osd.6', 'osd.10', 'osd.14'], ['mgr.x', 'osd.3', 'osd.7', 'osd.11', 'osd.15', 'client.0']], 'seed': 5394, 'sha1': 'd58f73069a8157a9ace12a0247ee3e8497707ea9', 'sleep_before_teardown': 0, 'subset': '582/12160', 'suite': 'rados', 'suite_branch': 'wip-aclamk-cbt-combined', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_d58f73069a8157a9ace12a0247ee3e8497707ea9/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph-ci.git', 'suite_sha1': 'd58f73069a8157a9ace12a0247ee3e8497707ea9', 'targets': {'smithi001.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPCBua+ZZ7P4Yb2djx+SyGWVot0cO8flB+45cb7GH8y+XVPL1TY5L26qE5XTOM3zgCg9jSHnzqBkdOaG8zj9moU=', 'smithi092.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJwDX5UG6gqL3h+DAMYGoeEhyCSETEk2A9MEUtQzdwwW3GsHQpCTRT2MiYXykpVyRW/Vvi5QDcXBlMSWaETvr80=', 'smithi162.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBE0U3sEdW+13n8O/UQ0/z+qiUKZb6Vil28xhdH0ddggibg460VEQU3kG3IRWKKMoWae7NNNvbEbHdHMmKvoGkGs=', 'smithi177.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKWJ/8t/0RjVNQ3s3lAGz2/jCs/zNbwmvTyMOmsqfB3b/Q47FdXawCs2p1CV8GE6N+jbrr7OSXixRZKM6GdS+kU='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': 1, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': {'conf': {'osd': {'debug monc': 20, 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore write v2': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd backoff on peering': True, 'osd blocked scrub grace period': 3600, 'osd debug reject backfill probability': 0.3, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd delete sleep': 1, 'osd failsafe full ratio': 0.95, 'osd inject bad map crc probability': 0.1, 'osd max backfills': 3, 'osd max markdown count': 1000, 'osd max scrubs': 5, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd snap trim sleep': 2}, 'client.0': {'admin socket': '/var/run/ceph/ceph-$name.asok'}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms bind msgr2': False, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'osd', 'ms inject internal delays': 0.002, 'ms inject socket failures': 2500, 'ms type': 'async', 'osd_pool_default_min_size': 2, 'osd_pool_default_size': 2}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'paxos service trim min': 10}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(REQUEST_STUCK\\)', 'failed to encode map', '\\(OSD_SLOW_PING_TIME', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', 'SLOW_OPS', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', 'mons down', 'mon down', 'out of quorum', 'noscrub', 'nodeep-scrub', '\\(POOL_APP_NOT_ENABLED\\)'], 'mon_bind_msgr2': False, 'sha1': 'd58f73069a8157a9ace12a0247ee3e8497707ea9', 'cluster': 'ceph'}}, {'thrashosds': {'bluestore_new_sharding': 'random', 'chance_bluestore_reshard': 1, 'chance_pgnum_grow': 1, 'chance_pgnum_shrink': 1, 'chance_pgpnum_fix': 1, 'min_in': 2, 'timeout': 1200}}, {'radosbench': {'clients': ['client.0'], 'time': 150}}, {'admin_socket': {'client.0': {'objecter_requests': {'test': 'http://git.ceph.com/?p={repo};a=blob_plain;f=src/test/admin_socket/objecter_requests;hb={branch}'}}}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a', 'timestamp': '2024-09-13_09:37:56', 'tube': 'smithi', 'user': 'akupczyk', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856'} 2024-09-13T11:20:42.913 DEBUG:teuthology.orchestra.run.smithi001:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-09-13T11:20:42.938 DEBUG:teuthology.orchestra.run.smithi092:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-09-13T11:20:42.940 DEBUG:teuthology.orchestra.run.smithi162:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-09-13T11:20:42.943 DEBUG:teuthology.orchestra.run.smithi177:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-09-13T11:20:42.949 DEBUG:teuthology.orchestra.run.smithi001:> sudo install -d -m0777 -- /var/run/ceph 2024-09-13T11:20:42.986 DEBUG:teuthology.orchestra.run.smithi092:> sudo install -d -m0777 -- /var/run/ceph 2024-09-13T11:20:42.988 DEBUG:teuthology.orchestra.run.smithi162:> sudo install -d -m0777 -- /var/run/ceph 2024-09-13T11:20:42.991 DEBUG:teuthology.orchestra.run.smithi177:> sudo install -d -m0777 -- /var/run/ceph 2024-09-13T11:20:43.006 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-09-13T11:20:43.006 DEBUG:teuthology.orchestra.run.smithi001:> dd if=/scratch_devs of=/dev/stdout 2024-09-13T11:20:43.045 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-13T11:20:43.045 DEBUG:teuthology.orchestra.run.smithi001:> stat /dev/vg_nvme/lv_1 2024-09-13T11:20:43.094 INFO:teuthology.orchestra.run.smithi001.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-13T11:20:43.094 INFO:teuthology.orchestra.run.smithi001.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-13T11:20:43.094 INFO:teuthology.orchestra.run.smithi001.stdout:Device: 5h/5d Inode: 886 Links: 1 2024-09-13T11:20:43.094 INFO:teuthology.orchestra.run.smithi001.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-13T11:20:43.094 INFO:teuthology.orchestra.run.smithi001.stdout:Access: 2024-09-13 11:14:55.759475144 +0000 2024-09-13T11:20:43.095 INFO:teuthology.orchestra.run.smithi001.stdout:Modify: 2024-09-13 11:14:55.519477288 +0000 2024-09-13T11:20:43.095 INFO:teuthology.orchestra.run.smithi001.stdout:Change: 2024-09-13 11:14:55.519477288 +0000 2024-09-13T11:20:43.095 INFO:teuthology.orchestra.run.smithi001.stdout: Birth: - 2024-09-13T11:20:43.095 DEBUG:teuthology.orchestra.run.smithi001:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-13T11:20:43.147 INFO:teuthology.orchestra.run.smithi001.stderr:1+0 records in 2024-09-13T11:20:43.147 INFO:teuthology.orchestra.run.smithi001.stderr:1+0 records out 2024-09-13T11:20:43.147 INFO:teuthology.orchestra.run.smithi001.stderr:512 bytes copied, 0.000211837 s, 2.4 MB/s 2024-09-13T11:20:43.148 DEBUG:teuthology.orchestra.run.smithi001:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-13T11:20:43.198 DEBUG:teuthology.orchestra.run.smithi001:> stat /dev/vg_nvme/lv_2 2024-09-13T11:20:43.245 INFO:teuthology.orchestra.run.smithi001.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-13T11:20:43.245 INFO:teuthology.orchestra.run.smithi001.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-13T11:20:43.245 INFO:teuthology.orchestra.run.smithi001.stdout:Device: 5h/5d Inode: 907 Links: 1 2024-09-13T11:20:43.246 INFO:teuthology.orchestra.run.smithi001.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-13T11:20:43.246 INFO:teuthology.orchestra.run.smithi001.stdout:Access: 2024-09-13 11:14:56.219471036 +0000 2024-09-13T11:20:43.246 INFO:teuthology.orchestra.run.smithi001.stdout:Modify: 2024-09-13 11:14:55.995473037 +0000 2024-09-13T11:20:43.246 INFO:teuthology.orchestra.run.smithi001.stdout:Change: 2024-09-13 11:14:55.995473037 +0000 2024-09-13T11:20:43.246 INFO:teuthology.orchestra.run.smithi001.stdout: Birth: - 2024-09-13T11:20:43.246 DEBUG:teuthology.orchestra.run.smithi001:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-13T11:20:43.300 INFO:teuthology.orchestra.run.smithi001.stderr:1+0 records in 2024-09-13T11:20:43.300 INFO:teuthology.orchestra.run.smithi001.stderr:1+0 records out 2024-09-13T11:20:43.300 INFO:teuthology.orchestra.run.smithi001.stderr:512 bytes copied, 0.000316555 s, 1.6 MB/s 2024-09-13T11:20:43.301 DEBUG:teuthology.orchestra.run.smithi001:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-13T11:20:43.350 DEBUG:teuthology.orchestra.run.smithi001:> stat /dev/vg_nvme/lv_3 2024-09-13T11:20:43.397 INFO:teuthology.orchestra.run.smithi001.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-13T11:20:43.397 INFO:teuthology.orchestra.run.smithi001.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-13T11:20:43.397 INFO:teuthology.orchestra.run.smithi001.stdout:Device: 5h/5d Inode: 920 Links: 1 2024-09-13T11:20:43.397 INFO:teuthology.orchestra.run.smithi001.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-13T11:20:43.397 INFO:teuthology.orchestra.run.smithi001.stdout:Access: 2024-09-13 11:14:56.451468963 +0000 2024-09-13T11:20:43.397 INFO:teuthology.orchestra.run.smithi001.stdout:Modify: 2024-09-13 11:14:56.447468998 +0000 2024-09-13T11:20:43.397 INFO:teuthology.orchestra.run.smithi001.stdout:Change: 2024-09-13 11:14:56.447468998 +0000 2024-09-13T11:20:43.398 INFO:teuthology.orchestra.run.smithi001.stdout: Birth: - 2024-09-13T11:20:43.398 DEBUG:teuthology.orchestra.run.smithi001:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-13T11:20:43.451 INFO:teuthology.orchestra.run.smithi001.stderr:1+0 records in 2024-09-13T11:20:43.451 INFO:teuthology.orchestra.run.smithi001.stderr:1+0 records out 2024-09-13T11:20:43.451 INFO:teuthology.orchestra.run.smithi001.stderr:512 bytes copied, 0.000247052 s, 2.1 MB/s 2024-09-13T11:20:43.453 DEBUG:teuthology.orchestra.run.smithi001:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-13T11:20:43.503 DEBUG:teuthology.orchestra.run.smithi001:> stat /dev/vg_nvme/lv_4 2024-09-13T11:20:43.549 INFO:teuthology.orchestra.run.smithi001.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-13T11:20:43.549 INFO:teuthology.orchestra.run.smithi001.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-13T11:20:43.549 INFO:teuthology.orchestra.run.smithi001.stdout:Device: 5h/5d Inode: 937 Links: 1 2024-09-13T11:20:43.549 INFO:teuthology.orchestra.run.smithi001.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-13T11:20:43.550 INFO:teuthology.orchestra.run.smithi001.stdout:Access: 2024-09-13 11:14:56.931464675 +0000 2024-09-13T11:20:43.550 INFO:teuthology.orchestra.run.smithi001.stdout:Modify: 2024-09-13 11:14:56.927464711 +0000 2024-09-13T11:20:43.550 INFO:teuthology.orchestra.run.smithi001.stdout:Change: 2024-09-13 11:14:56.927464711 +0000 2024-09-13T11:20:43.550 INFO:teuthology.orchestra.run.smithi001.stdout: Birth: - 2024-09-13T11:20:43.550 DEBUG:teuthology.orchestra.run.smithi001:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-13T11:20:43.604 INFO:teuthology.orchestra.run.smithi001.stderr:1+0 records in 2024-09-13T11:20:43.604 INFO:teuthology.orchestra.run.smithi001.stderr:1+0 records out 2024-09-13T11:20:43.604 INFO:teuthology.orchestra.run.smithi001.stderr:512 bytes copied, 0.000305868 s, 1.7 MB/s 2024-09-13T11:20:43.605 DEBUG:teuthology.orchestra.run.smithi001:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-13T11:20:43.654 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.8': '/dev/vg_nvme/lv_3', 'osd.12': '/dev/vg_nvme/lv_4'} 2024-09-13T11:20:43.654 DEBUG:teuthology.orchestra.run.smithi092:> set -ex 2024-09-13T11:20:43.654 DEBUG:teuthology.orchestra.run.smithi092:> dd if=/scratch_devs of=/dev/stdout 2024-09-13T11:20:43.660 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-13T11:20:43.660 DEBUG:teuthology.orchestra.run.smithi092:> stat /dev/vg_nvme/lv_1 2024-09-13T11:20:43.708 INFO:teuthology.orchestra.run.smithi092.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-13T11:20:43.708 INFO:teuthology.orchestra.run.smithi092.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-13T11:20:43.708 INFO:teuthology.orchestra.run.smithi092.stdout:Device: 5h/5d Inode: 886 Links: 1 2024-09-13T11:20:43.709 INFO:teuthology.orchestra.run.smithi092.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-13T11:20:43.709 INFO:teuthology.orchestra.run.smithi092.stdout:Access: 2024-09-13 11:14:54.088613366 +0000 2024-09-13T11:20:43.709 INFO:teuthology.orchestra.run.smithi092.stdout:Modify: 2024-09-13 11:14:53.820615760 +0000 2024-09-13T11:20:43.709 INFO:teuthology.orchestra.run.smithi092.stdout:Change: 2024-09-13 11:14:53.820615760 +0000 2024-09-13T11:20:43.709 INFO:teuthology.orchestra.run.smithi092.stdout: Birth: - 2024-09-13T11:20:43.709 DEBUG:teuthology.orchestra.run.smithi092:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-13T11:20:43.762 INFO:teuthology.orchestra.run.smithi092.stderr:1+0 records in 2024-09-13T11:20:43.762 INFO:teuthology.orchestra.run.smithi092.stderr:1+0 records out 2024-09-13T11:20:43.762 INFO:teuthology.orchestra.run.smithi092.stderr:512 bytes copied, 0.000382327 s, 1.3 MB/s 2024-09-13T11:20:43.763 DEBUG:teuthology.orchestra.run.smithi092:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-13T11:20:43.813 DEBUG:teuthology.orchestra.run.smithi092:> stat /dev/vg_nvme/lv_2 2024-09-13T11:20:43.859 INFO:teuthology.orchestra.run.smithi092.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-13T11:20:43.859 INFO:teuthology.orchestra.run.smithi092.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-13T11:20:43.859 INFO:teuthology.orchestra.run.smithi092.stdout:Device: 5h/5d Inode: 898 Links: 1 2024-09-13T11:20:43.859 INFO:teuthology.orchestra.run.smithi092.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-13T11:20:43.859 INFO:teuthology.orchestra.run.smithi092.stdout:Access: 2024-09-13 11:14:54.588608899 +0000 2024-09-13T11:20:43.860 INFO:teuthology.orchestra.run.smithi092.stdout:Modify: 2024-09-13 11:14:54.328611223 +0000 2024-09-13T11:20:43.860 INFO:teuthology.orchestra.run.smithi092.stdout:Change: 2024-09-13 11:14:54.328611223 +0000 2024-09-13T11:20:43.860 INFO:teuthology.orchestra.run.smithi092.stdout: Birth: - 2024-09-13T11:20:43.860 DEBUG:teuthology.orchestra.run.smithi092:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-13T11:20:43.913 INFO:teuthology.orchestra.run.smithi092.stderr:1+0 records in 2024-09-13T11:20:43.913 INFO:teuthology.orchestra.run.smithi092.stderr:1+0 records out 2024-09-13T11:20:43.913 INFO:teuthology.orchestra.run.smithi092.stderr:512 bytes copied, 0.00030715 s, 1.7 MB/s 2024-09-13T11:20:43.914 DEBUG:teuthology.orchestra.run.smithi092:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-13T11:20:43.965 DEBUG:teuthology.orchestra.run.smithi092:> stat /dev/vg_nvme/lv_3 2024-09-13T11:20:44.012 INFO:teuthology.orchestra.run.smithi092.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-13T11:20:44.012 INFO:teuthology.orchestra.run.smithi092.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-13T11:20:44.012 INFO:teuthology.orchestra.run.smithi092.stdout:Device: 5h/5d Inode: 915 Links: 1 2024-09-13T11:20:44.012 INFO:teuthology.orchestra.run.smithi092.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-13T11:20:44.012 INFO:teuthology.orchestra.run.smithi092.stdout:Access: 2024-09-13 11:14:55.068604612 +0000 2024-09-13T11:20:44.012 INFO:teuthology.orchestra.run.smithi092.stdout:Modify: 2024-09-13 11:14:54.820606827 +0000 2024-09-13T11:20:44.012 INFO:teuthology.orchestra.run.smithi092.stdout:Change: 2024-09-13 11:14:54.820606827 +0000 2024-09-13T11:20:44.012 INFO:teuthology.orchestra.run.smithi092.stdout: Birth: - 2024-09-13T11:20:44.012 DEBUG:teuthology.orchestra.run.smithi092:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-13T11:20:44.065 INFO:teuthology.orchestra.run.smithi092.stderr:1+0 records in 2024-09-13T11:20:44.065 INFO:teuthology.orchestra.run.smithi092.stderr:1+0 records out 2024-09-13T11:20:44.065 INFO:teuthology.orchestra.run.smithi092.stderr:512 bytes copied, 0.000414163 s, 1.2 MB/s 2024-09-13T11:20:44.067 DEBUG:teuthology.orchestra.run.smithi092:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-13T11:20:44.116 DEBUG:teuthology.orchestra.run.smithi092:> stat /dev/vg_nvme/lv_4 2024-09-13T11:20:44.163 INFO:teuthology.orchestra.run.smithi092.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-13T11:20:44.164 INFO:teuthology.orchestra.run.smithi092.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-13T11:20:44.164 INFO:teuthology.orchestra.run.smithi092.stdout:Device: 5h/5d Inode: 932 Links: 1 2024-09-13T11:20:44.164 INFO:teuthology.orchestra.run.smithi092.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-13T11:20:44.164 INFO:teuthology.orchestra.run.smithi092.stdout:Access: 2024-09-13 11:14:55.556600254 +0000 2024-09-13T11:20:44.164 INFO:teuthology.orchestra.run.smithi092.stdout:Modify: 2024-09-13 11:14:55.296602576 +0000 2024-09-13T11:20:44.164 INFO:teuthology.orchestra.run.smithi092.stdout:Change: 2024-09-13 11:14:55.296602576 +0000 2024-09-13T11:20:44.164 INFO:teuthology.orchestra.run.smithi092.stdout: Birth: - 2024-09-13T11:20:44.164 DEBUG:teuthology.orchestra.run.smithi092:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-13T11:20:44.219 INFO:teuthology.orchestra.run.smithi092.stderr:1+0 records in 2024-09-13T11:20:44.219 INFO:teuthology.orchestra.run.smithi092.stderr:1+0 records out 2024-09-13T11:20:44.220 INFO:teuthology.orchestra.run.smithi092.stderr:512 bytes copied, 0.000353153 s, 1.4 MB/s 2024-09-13T11:20:44.220 DEBUG:teuthology.orchestra.run.smithi092:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-13T11:20:44.269 INFO:tasks.ceph:osd dev map: {'osd.1': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.9': '/dev/vg_nvme/lv_3', 'osd.13': '/dev/vg_nvme/lv_4'} 2024-09-13T11:20:44.270 DEBUG:teuthology.orchestra.run.smithi162:> set -ex 2024-09-13T11:20:44.270 DEBUG:teuthology.orchestra.run.smithi162:> dd if=/scratch_devs of=/dev/stdout 2024-09-13T11:20:44.275 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-13T11:20:44.276 DEBUG:teuthology.orchestra.run.smithi162:> stat /dev/vg_nvme/lv_1 2024-09-13T11:20:44.325 INFO:teuthology.orchestra.run.smithi162.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-13T11:20:44.325 INFO:teuthology.orchestra.run.smithi162.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-13T11:20:44.325 INFO:teuthology.orchestra.run.smithi162.stdout:Device: 5h/5d Inode: 891 Links: 1 2024-09-13T11:20:44.325 INFO:teuthology.orchestra.run.smithi162.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-13T11:20:44.326 INFO:teuthology.orchestra.run.smithi162.stdout:Access: 2024-09-13 11:14:59.697077536 +0000 2024-09-13T11:20:44.326 INFO:teuthology.orchestra.run.smithi162.stdout:Modify: 2024-09-13 11:14:59.433079895 +0000 2024-09-13T11:20:44.326 INFO:teuthology.orchestra.run.smithi162.stdout:Change: 2024-09-13 11:14:59.433079895 +0000 2024-09-13T11:20:44.326 INFO:teuthology.orchestra.run.smithi162.stdout: Birth: - 2024-09-13T11:20:44.326 DEBUG:teuthology.orchestra.run.smithi162:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-13T11:20:44.380 INFO:teuthology.orchestra.run.smithi162.stderr:1+0 records in 2024-09-13T11:20:44.380 INFO:teuthology.orchestra.run.smithi162.stderr:1+0 records out 2024-09-13T11:20:44.380 INFO:teuthology.orchestra.run.smithi162.stderr:512 bytes copied, 0.000447441 s, 1.1 MB/s 2024-09-13T11:20:44.381 DEBUG:teuthology.orchestra.run.smithi162:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-13T11:20:44.434 DEBUG:teuthology.orchestra.run.smithi162:> stat /dev/vg_nvme/lv_2 2024-09-13T11:20:44.485 INFO:teuthology.orchestra.run.smithi162.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-13T11:20:44.485 INFO:teuthology.orchestra.run.smithi162.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-13T11:20:44.485 INFO:teuthology.orchestra.run.smithi162.stdout:Device: 5h/5d Inode: 907 Links: 1 2024-09-13T11:20:44.485 INFO:teuthology.orchestra.run.smithi162.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-13T11:20:44.485 INFO:teuthology.orchestra.run.smithi162.stdout:Access: 2024-09-13 11:15:00.173073285 +0000 2024-09-13T11:20:44.485 INFO:teuthology.orchestra.run.smithi162.stdout:Modify: 2024-09-13 11:14:59.937075393 +0000 2024-09-13T11:20:44.486 INFO:teuthology.orchestra.run.smithi162.stdout:Change: 2024-09-13 11:14:59.937075393 +0000 2024-09-13T11:20:44.486 INFO:teuthology.orchestra.run.smithi162.stdout: Birth: - 2024-09-13T11:20:44.486 DEBUG:teuthology.orchestra.run.smithi162:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-13T11:20:44.539 INFO:teuthology.orchestra.run.smithi162.stderr:1+0 records in 2024-09-13T11:20:44.539 INFO:teuthology.orchestra.run.smithi162.stderr:1+0 records out 2024-09-13T11:20:44.539 INFO:teuthology.orchestra.run.smithi162.stderr:512 bytes copied, 0.000442164 s, 1.2 MB/s 2024-09-13T11:20:44.540 DEBUG:teuthology.orchestra.run.smithi162:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-13T11:20:44.590 DEBUG:teuthology.orchestra.run.smithi162:> stat /dev/vg_nvme/lv_3 2024-09-13T11:20:44.636 INFO:teuthology.orchestra.run.smithi162.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-13T11:20:44.636 INFO:teuthology.orchestra.run.smithi162.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-13T11:20:44.636 INFO:teuthology.orchestra.run.smithi162.stdout:Device: 5h/5d Inode: 921 Links: 1 2024-09-13T11:20:44.637 INFO:teuthology.orchestra.run.smithi162.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-13T11:20:44.637 INFO:teuthology.orchestra.run.smithi162.stdout:Access: 2024-09-13 11:15:00.437070927 +0000 2024-09-13T11:20:44.637 INFO:teuthology.orchestra.run.smithi162.stdout:Modify: 2024-09-13 11:15:00.433070962 +0000 2024-09-13T11:20:44.637 INFO:teuthology.orchestra.run.smithi162.stdout:Change: 2024-09-13 11:15:00.433070962 +0000 2024-09-13T11:20:44.637 INFO:teuthology.orchestra.run.smithi162.stdout: Birth: - 2024-09-13T11:20:44.637 DEBUG:teuthology.orchestra.run.smithi162:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-13T11:20:44.691 INFO:teuthology.orchestra.run.smithi162.stderr:1+0 records in 2024-09-13T11:20:44.691 INFO:teuthology.orchestra.run.smithi162.stderr:1+0 records out 2024-09-13T11:20:44.691 INFO:teuthology.orchestra.run.smithi162.stderr:512 bytes copied, 0.000418424 s, 1.2 MB/s 2024-09-13T11:20:44.693 DEBUG:teuthology.orchestra.run.smithi162:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-13T11:20:44.742 DEBUG:teuthology.orchestra.run.smithi162:> stat /dev/vg_nvme/lv_4 2024-09-13T11:20:44.788 INFO:teuthology.orchestra.run.smithi162.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-13T11:20:44.789 INFO:teuthology.orchestra.run.smithi162.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-13T11:20:44.789 INFO:teuthology.orchestra.run.smithi162.stdout:Device: 5h/5d Inode: 942 Links: 1 2024-09-13T11:20:44.789 INFO:teuthology.orchestra.run.smithi162.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-13T11:20:44.789 INFO:teuthology.orchestra.run.smithi162.stdout:Access: 2024-09-13 11:15:01.165064424 +0000 2024-09-13T11:20:44.789 INFO:teuthology.orchestra.run.smithi162.stdout:Modify: 2024-09-13 11:15:00.921066604 +0000 2024-09-13T11:20:44.789 INFO:teuthology.orchestra.run.smithi162.stdout:Change: 2024-09-13 11:15:00.921066604 +0000 2024-09-13T11:20:44.789 INFO:teuthology.orchestra.run.smithi162.stdout: Birth: - 2024-09-13T11:20:44.789 DEBUG:teuthology.orchestra.run.smithi162:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-13T11:20:44.843 INFO:teuthology.orchestra.run.smithi162.stderr:1+0 records in 2024-09-13T11:20:44.843 INFO:teuthology.orchestra.run.smithi162.stderr:1+0 records out 2024-09-13T11:20:44.843 INFO:teuthology.orchestra.run.smithi162.stderr:512 bytes copied, 0.000385444 s, 1.3 MB/s 2024-09-13T11:20:44.844 DEBUG:teuthology.orchestra.run.smithi162:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-13T11:20:44.894 INFO:tasks.ceph:osd dev map: {'osd.2': '/dev/vg_nvme/lv_1', 'osd.6': '/dev/vg_nvme/lv_2', 'osd.10': '/dev/vg_nvme/lv_3', 'osd.14': '/dev/vg_nvme/lv_4'} 2024-09-13T11:20:44.894 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-09-13T11:20:44.894 DEBUG:teuthology.orchestra.run.smithi177:> dd if=/scratch_devs of=/dev/stdout 2024-09-13T11:20:44.900 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-13T11:20:44.900 DEBUG:teuthology.orchestra.run.smithi177:> stat /dev/vg_nvme/lv_1 2024-09-13T11:20:44.947 INFO:teuthology.orchestra.run.smithi177.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-13T11:20:44.947 INFO:teuthology.orchestra.run.smithi177.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-13T11:20:44.947 INFO:teuthology.orchestra.run.smithi177.stdout:Device: 5h/5d Inode: 885 Links: 1 2024-09-13T11:20:44.947 INFO:teuthology.orchestra.run.smithi177.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-13T11:20:44.947 INFO:teuthology.orchestra.run.smithi177.stdout:Access: 2024-09-13 11:14:53.792038699 +0000 2024-09-13T11:20:44.947 INFO:teuthology.orchestra.run.smithi177.stdout:Modify: 2024-09-13 11:14:53.528041058 +0000 2024-09-13T11:20:44.947 INFO:teuthology.orchestra.run.smithi177.stdout:Change: 2024-09-13 11:14:53.528041058 +0000 2024-09-13T11:20:44.947 INFO:teuthology.orchestra.run.smithi177.stdout: Birth: - 2024-09-13T11:20:44.947 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-13T11:20:45.001 INFO:teuthology.orchestra.run.smithi177.stderr:1+0 records in 2024-09-13T11:20:45.001 INFO:teuthology.orchestra.run.smithi177.stderr:1+0 records out 2024-09-13T11:20:45.001 INFO:teuthology.orchestra.run.smithi177.stderr:512 bytes copied, 0.000417982 s, 1.2 MB/s 2024-09-13T11:20:45.002 DEBUG:teuthology.orchestra.run.smithi177:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-13T11:20:45.051 DEBUG:teuthology.orchestra.run.smithi177:> stat /dev/vg_nvme/lv_2 2024-09-13T11:20:45.098 INFO:teuthology.orchestra.run.smithi177.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-13T11:20:45.099 INFO:teuthology.orchestra.run.smithi177.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-13T11:20:45.099 INFO:teuthology.orchestra.run.smithi177.stdout:Device: 5h/5d Inode: 908 Links: 1 2024-09-13T11:20:45.099 INFO:teuthology.orchestra.run.smithi177.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-13T11:20:45.099 INFO:teuthology.orchestra.run.smithi177.stdout:Access: 2024-09-13 11:14:54.300034162 +0000 2024-09-13T11:20:45.099 INFO:teuthology.orchestra.run.smithi177.stdout:Modify: 2024-09-13 11:14:54.040036484 +0000 2024-09-13T11:20:45.099 INFO:teuthology.orchestra.run.smithi177.stdout:Change: 2024-09-13 11:14:54.040036484 +0000 2024-09-13T11:20:45.099 INFO:teuthology.orchestra.run.smithi177.stdout: Birth: - 2024-09-13T11:20:45.099 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-13T11:20:45.152 INFO:teuthology.orchestra.run.smithi177.stderr:1+0 records in 2024-09-13T11:20:45.153 INFO:teuthology.orchestra.run.smithi177.stderr:1+0 records out 2024-09-13T11:20:45.153 INFO:teuthology.orchestra.run.smithi177.stderr:512 bytes copied, 0.000422757 s, 1.2 MB/s 2024-09-13T11:20:45.154 DEBUG:teuthology.orchestra.run.smithi177:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-13T11:20:45.203 DEBUG:teuthology.orchestra.run.smithi177:> stat /dev/vg_nvme/lv_3 2024-09-13T11:20:45.250 INFO:teuthology.orchestra.run.smithi177.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-13T11:20:45.250 INFO:teuthology.orchestra.run.smithi177.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-13T11:20:45.250 INFO:teuthology.orchestra.run.smithi177.stdout:Device: 5h/5d Inode: 924 Links: 1 2024-09-13T11:20:45.250 INFO:teuthology.orchestra.run.smithi177.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-13T11:20:45.250 INFO:teuthology.orchestra.run.smithi177.stdout:Access: 2024-09-13 11:14:54.804029660 +0000 2024-09-13T11:20:45.250 INFO:teuthology.orchestra.run.smithi177.stdout:Modify: 2024-09-13 11:14:54.548031947 +0000 2024-09-13T11:20:45.250 INFO:teuthology.orchestra.run.smithi177.stdout:Change: 2024-09-13 11:14:54.548031947 +0000 2024-09-13T11:20:45.251 INFO:teuthology.orchestra.run.smithi177.stdout: Birth: - 2024-09-13T11:20:45.251 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-13T11:20:45.304 INFO:teuthology.orchestra.run.smithi177.stderr:1+0 records in 2024-09-13T11:20:45.305 INFO:teuthology.orchestra.run.smithi177.stderr:1+0 records out 2024-09-13T11:20:45.305 INFO:teuthology.orchestra.run.smithi177.stderr:512 bytes copied, 0.000371178 s, 1.4 MB/s 2024-09-13T11:20:45.306 DEBUG:teuthology.orchestra.run.smithi177:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-13T11:20:45.355 DEBUG:teuthology.orchestra.run.smithi177:> stat /dev/vg_nvme/lv_4 2024-09-13T11:20:45.406 INFO:teuthology.orchestra.run.smithi177.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-13T11:20:45.407 INFO:teuthology.orchestra.run.smithi177.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-13T11:20:45.407 INFO:teuthology.orchestra.run.smithi177.stdout:Device: 5h/5d Inode: 941 Links: 1 2024-09-13T11:20:45.407 INFO:teuthology.orchestra.run.smithi177.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-13T11:20:45.407 INFO:teuthology.orchestra.run.smithi177.stdout:Access: 2024-09-13 11:14:55.328024979 +0000 2024-09-13T11:20:45.407 INFO:teuthology.orchestra.run.smithi177.stdout:Modify: 2024-09-13 11:14:55.060027374 +0000 2024-09-13T11:20:45.407 INFO:teuthology.orchestra.run.smithi177.stdout:Change: 2024-09-13 11:14:55.060027374 +0000 2024-09-13T11:20:45.407 INFO:teuthology.orchestra.run.smithi177.stdout: Birth: - 2024-09-13T11:20:45.407 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-13T11:20:45.461 INFO:teuthology.orchestra.run.smithi177.stderr:1+0 records in 2024-09-13T11:20:45.461 INFO:teuthology.orchestra.run.smithi177.stderr:1+0 records out 2024-09-13T11:20:45.461 INFO:teuthology.orchestra.run.smithi177.stderr:512 bytes copied, 0.000384287 s, 1.3 MB/s 2024-09-13T11:20:45.462 DEBUG:teuthology.orchestra.run.smithi177:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-13T11:20:45.511 INFO:tasks.ceph:osd dev map: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.7': '/dev/vg_nvme/lv_2', 'osd.11': '/dev/vg_nvme/lv_3', 'osd.15': '/dev/vg_nvme/lv_4'} 2024-09-13T11:20:45.511 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi001.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.8': '/dev/vg_nvme/lv_3', 'osd.12': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi092.front.sepia.ceph.com'): {'osd.1': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.9': '/dev/vg_nvme/lv_3', 'osd.13': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi162.front.sepia.ceph.com'): {'osd.2': '/dev/vg_nvme/lv_1', 'osd.6': '/dev/vg_nvme/lv_2', 'osd.10': '/dev/vg_nvme/lv_3', 'osd.14': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi177.front.sepia.ceph.com'): {'osd.3': '/dev/vg_nvme/lv_1', 'osd.7': '/dev/vg_nvme/lv_2', 'osd.11': '/dev/vg_nvme/lv_3', 'osd.15': '/dev/vg_nvme/lv_4'}} 2024-09-13T11:20:45.511 INFO:tasks.ceph:Generating config... 2024-09-13T11:20:45.516 INFO:tasks.ceph:[osd] debug monc = 20 2024-09-13T11:20:45.516 INFO:tasks.ceph:[osd] bdev async discard = True 2024-09-13T11:20:45.517 INFO:tasks.ceph:[osd] bdev enable discard = True 2024-09-13T11:20:45.517 INFO:tasks.ceph:[osd] bluestore allocator = stupid 2024-09-13T11:20:45.517 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-09-13T11:20:45.517 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-09-13T11:20:45.517 INFO:tasks.ceph:[osd] bluestore write v2 = True 2024-09-13T11:20:45.517 INFO:tasks.ceph:[osd] bluestore zero block detection = True 2024-09-13T11:20:45.517 INFO:tasks.ceph:[osd] debug bluefs = 20 2024-09-13T11:20:45.517 INFO:tasks.ceph:[osd] debug bluestore = 20 2024-09-13T11:20:45.517 INFO:tasks.ceph:[osd] debug ms = 1 2024-09-13T11:20:45.517 INFO:tasks.ceph:[osd] debug osd = 20 2024-09-13T11:20:45.517 INFO:tasks.ceph:[osd] debug rocksdb = 10 2024-09-13T11:20:45.517 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-09-13T11:20:45.517 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-09-13T11:20:45.517 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-09-13T11:20:45.518 INFO:tasks.ceph:[osd] osd backoff on peering = True 2024-09-13T11:20:45.518 INFO:tasks.ceph:[osd] osd blocked scrub grace period = 3600 2024-09-13T11:20:45.518 INFO:tasks.ceph:[osd] osd debug reject backfill probability = 0.3 2024-09-13T11:20:45.518 INFO:tasks.ceph:[osd] osd debug verify cached snaps = True 2024-09-13T11:20:45.518 INFO:tasks.ceph:[osd] osd debug verify missing on start = True 2024-09-13T11:20:45.518 INFO:tasks.ceph:[osd] osd delete sleep = 1 2024-09-13T11:20:45.518 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-09-13T11:20:45.518 INFO:tasks.ceph:[osd] osd inject bad map crc probability = 0.1 2024-09-13T11:20:45.518 INFO:tasks.ceph:[osd] osd max backfills = 3 2024-09-13T11:20:45.518 INFO:tasks.ceph:[osd] osd max markdown count = 1000 2024-09-13T11:20:45.518 INFO:tasks.ceph:[osd] osd max scrubs = 5 2024-09-13T11:20:45.518 INFO:tasks.ceph:[osd] osd mclock override recovery settings = True 2024-09-13T11:20:45.518 INFO:tasks.ceph:[osd] osd mclock profile = high_recovery_ops 2024-09-13T11:20:45.519 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-09-13T11:20:45.519 INFO:tasks.ceph:[osd] osd op queue = debug_random 2024-09-13T11:20:45.519 INFO:tasks.ceph:[osd] osd op queue cut off = debug_random 2024-09-13T11:20:45.519 INFO:tasks.ceph:[osd] osd scrub max interval = 120 2024-09-13T11:20:45.519 INFO:tasks.ceph:[osd] osd scrub min interval = 60 2024-09-13T11:20:45.519 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2024-09-13T11:20:45.519 INFO:tasks.ceph:[osd] osd snap trim sleep = 2 2024-09-13T11:20:45.519 INFO:tasks.ceph:[client.0] admin socket = /var/run/ceph/ceph-$name.asok 2024-09-13T11:20:45.519 INFO:tasks.ceph:[global] mon client directed command retry = 5 2024-09-13T11:20:45.519 INFO:tasks.ceph:[global] mon election default strategy = 1 2024-09-13T11:20:45.519 INFO:tasks.ceph:[global] ms bind msgr2 = False 2024-09-13T11:20:45.519 INFO:tasks.ceph:[global] ms inject delay max = 1 2024-09-13T11:20:45.519 INFO:tasks.ceph:[global] ms inject delay probability = 0.005 2024-09-13T11:20:45.519 INFO:tasks.ceph:[global] ms inject delay type = osd 2024-09-13T11:20:45.520 INFO:tasks.ceph:[global] ms inject internal delays = 0.002 2024-09-13T11:20:45.520 INFO:tasks.ceph:[global] ms inject socket failures = 2500 2024-09-13T11:20:45.520 INFO:tasks.ceph:[global] ms type = async 2024-09-13T11:20:45.520 INFO:tasks.ceph:[global] osd_pool_default_min_size = 2 2024-09-13T11:20:45.520 INFO:tasks.ceph:[global] osd_pool_default_size = 2 2024-09-13T11:20:45.520 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-09-13T11:20:45.520 INFO:tasks.ceph:[mgr] debug ms = 1 2024-09-13T11:20:45.520 INFO:tasks.ceph:[mon] debug mon = 20 2024-09-13T11:20:45.520 INFO:tasks.ceph:[mon] debug ms = 1 2024-09-13T11:20:45.520 INFO:tasks.ceph:[mon] debug paxos = 20 2024-09-13T11:20:45.520 INFO:tasks.ceph:[mon] mon min osdmap epochs = 50 2024-09-13T11:20:45.520 INFO:tasks.ceph:[mon] mon osdmap full prune interval = 2 2024-09-13T11:20:45.520 INFO:tasks.ceph:[mon] mon osdmap full prune min = 15 2024-09-13T11:20:45.521 INFO:tasks.ceph:[mon] mon osdmap full prune txsize = 2 2024-09-13T11:20:45.521 INFO:tasks.ceph:[mon] mon scrub interval = 300 2024-09-13T11:20:45.521 INFO:tasks.ceph:[mon] paxos service trim min = 10 2024-09-13T11:20:45.521 INFO:tasks.ceph:Setting up mon.a... 2024-09-13T11:20:45.521 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-09-13T11:20:45.556 INFO:teuthology.orchestra.run.smithi001.stdout:creating /etc/ceph/ceph.keyring 2024-09-13T11:20:45.570 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-09-13T11:20:45.647 DEBUG:teuthology.orchestra.run.smithi001:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-13T11:20:45.701 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '[v1:172.21.15.1:6789]'), ('mon.b', '[v1:172.21.15.92:6789]'), ('mon.c', '[v1:172.21.15.162:6789]')] 2024-09-13T11:20:45.701 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '[v1:172.21.15.1:6789],[v1:172.21.15.92:6789],[v1:172.21.15.162:6789]', 'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms bind msgr2': False, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'osd', 'ms inject internal delays': 0.002, 'ms inject socket failures': 2500, 'ms type': 'async', 'osd_pool_default_min_size': 2, 'osd_pool_default_size': 2}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': 120, 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'debug monc': 20, 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore write v2': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd backoff on peering': True, 'osd blocked scrub grace period': 3600, 'osd debug reject backfill probability': 0.3, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd delete sleep': 1, 'osd failsafe full ratio': 0.95, 'osd inject bad map crc probability': 0.1, 'osd max backfills': 3, 'osd max markdown count': 1000, 'osd max scrubs': 5, 'osd mclock override recovery settings': True, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd snap trim sleep': 2}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'paxos service trim min': 10}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok'}, 'mon.a': {}, 'mon.b': {}, 'mon.c': {}, 'client.0': {'admin socket': '/var/run/ceph/ceph-$name.asok'}} 2024-09-13T11:20:45.702 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-09-13T11:20:45.702 DEBUG:teuthology.orchestra.run.smithi001:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-09-13T11:20:45.749 DEBUG:teuthology.orchestra.run.smithi001:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --addv a '[v1:172.21.15.1:6789]' --addv b '[v1:172.21.15.92:6789]' --addv c '[v1:172.21.15.162:6789]' --print /home/ubuntu/cephtest/ceph.monmap 2024-09-13T11:20:45.837 INFO:teuthology.orchestra.run.smithi001.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-09-13T11:20:45.837 INFO:teuthology.orchestra.run.smithi001.stdout:monmaptool: generated fsid 2fd68ac7-5581-4d90-80a4-30bcf5252a7a 2024-09-13T11:20:45.837 INFO:teuthology.orchestra.run.smithi001.stdout:setting min_mon_release = quincy 2024-09-13T11:20:45.837 INFO:teuthology.orchestra.run.smithi001.stdout:epoch 0 2024-09-13T11:20:45.838 INFO:teuthology.orchestra.run.smithi001.stdout:fsid 2fd68ac7-5581-4d90-80a4-30bcf5252a7a 2024-09-13T11:20:45.838 INFO:teuthology.orchestra.run.smithi001.stdout:last_changed 2024-09-13T11:20:45.836975+0000 2024-09-13T11:20:45.838 INFO:teuthology.orchestra.run.smithi001.stdout:created 2024-09-13T11:20:45.836975+0000 2024-09-13T11:20:45.838 INFO:teuthology.orchestra.run.smithi001.stdout:min_mon_release 17 (quincy) 2024-09-13T11:20:45.838 INFO:teuthology.orchestra.run.smithi001.stdout:election_strategy: 1 2024-09-13T11:20:45.838 INFO:teuthology.orchestra.run.smithi001.stdout:0: v1:172.21.15.1:6789/0 mon.a 2024-09-13T11:20:45.838 INFO:teuthology.orchestra.run.smithi001.stdout:1: v1:172.21.15.92:6789/0 mon.b 2024-09-13T11:20:45.838 INFO:teuthology.orchestra.run.smithi001.stdout:2: v1:172.21.15.162:6789/0 mon.c 2024-09-13T11:20:45.838 INFO:teuthology.orchestra.run.smithi001.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-09-13T11:20:45.840 DEBUG:teuthology.orchestra.run.smithi001:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-09-13T11:20:45.845 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 2fd68ac7-5581-4d90-80a4-30bcf5252a7a... 2024-09-13T11:20:45.846 DEBUG:teuthology.orchestra.run.smithi001:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-09-13T11:20:45.890 DEBUG:teuthology.orchestra.run.smithi092:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-09-13T11:20:45.892 DEBUG:teuthology.orchestra.run.smithi162:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-09-13T11:20:45.895 DEBUG:teuthology.orchestra.run.smithi177:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-09-13T11:20:45.918 INFO:teuthology.orchestra.run.smithi001.stdout:[global] 2024-09-13T11:20:45.918 INFO:teuthology.orchestra.run.smithi001.stdout: chdir = "" 2024-09-13T11:20:45.918 INFO:teuthology.orchestra.run.smithi001.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-13T11:20:45.918 INFO:teuthology.orchestra.run.smithi001.stdout: auth supported = cephx 2024-09-13T11:20:45.919 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-13T11:20:45.919 INFO:teuthology.orchestra.run.smithi001.stdout: filestore xattr use omap = true 2024-09-13T11:20:45.919 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-13T11:20:45.919 INFO:teuthology.orchestra.run.smithi001.stdout: mon clock drift allowed = 1.000 2024-09-13T11:20:45.919 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-13T11:20:45.919 INFO:teuthology.orchestra.run.smithi001.stdout: osd crush chooseleaf type = 0 2024-09-13T11:20:45.919 INFO:teuthology.orchestra.run.smithi001.stdout: auth debug = true 2024-09-13T11:20:45.919 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-13T11:20:45.919 INFO:teuthology.orchestra.run.smithi001.stdout: ms die on old message = true 2024-09-13T11:20:45.919 INFO:teuthology.orchestra.run.smithi001.stdout: ms die on bug = true 2024-09-13T11:20:45.919 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-13T11:20:45.920 INFO:teuthology.orchestra.run.smithi001.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-13T11:20:45.920 INFO:teuthology.orchestra.run.smithi001.stdout: mon pg warn max object skew = 0 2024-09-13T11:20:45.920 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-13T11:20:45.920 INFO:teuthology.orchestra.run.smithi001.stdout: # disable pg_autoscaler by default for new pools 2024-09-13T11:20:45.920 INFO:teuthology.orchestra.run.smithi001.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-13T11:20:45.920 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-13T11:20:45.920 INFO:teuthology.orchestra.run.smithi001.stdout: osd pool default size = 2 2024-09-13T11:20:45.920 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-13T11:20:45.920 INFO:teuthology.orchestra.run.smithi001.stdout: mon osd allow primary affinity = true 2024-09-13T11:20:45.920 INFO:teuthology.orchestra.run.smithi001.stdout: mon osd allow pg remap = true 2024-09-13T11:20:45.920 INFO:teuthology.orchestra.run.smithi001.stdout: mon warn on legacy crush tunables = false 2024-09-13T11:20:45.920 INFO:teuthology.orchestra.run.smithi001.stdout: mon warn on crush straw calc version zero = false 2024-09-13T11:20:45.921 INFO:teuthology.orchestra.run.smithi001.stdout: mon warn on no sortbitwise = false 2024-09-13T11:20:45.921 INFO:teuthology.orchestra.run.smithi001.stdout: mon warn on osd down out interval zero = false 2024-09-13T11:20:45.921 INFO:teuthology.orchestra.run.smithi001.stdout: mon warn on too few osds = false 2024-09-13T11:20:45.921 INFO:teuthology.orchestra.run.smithi001.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-13T11:20:45.921 INFO:teuthology.orchestra.run.smithi001.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-13T11:20:45.921 INFO:teuthology.orchestra.run.smithi001.stdout: mon_allow_pool_size_one = true 2024-09-13T11:20:45.921 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-13T11:20:45.921 INFO:teuthology.orchestra.run.smithi001.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-13T11:20:45.921 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-13T11:20:45.921 INFO:teuthology.orchestra.run.smithi001.stdout: osd default data pool replay window = 5 2024-09-13T11:20:45.925 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-13T11:20:45.925 INFO:teuthology.orchestra.run.smithi001.stdout: mon allow pool delete = true 2024-09-13T11:20:45.925 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-13T11:20:45.925 INFO:teuthology.orchestra.run.smithi001.stdout: mon cluster log file level = debug 2024-09-13T11:20:45.925 INFO:teuthology.orchestra.run.smithi001.stdout: debug asserts on shutdown = true 2024-09-13T11:20:45.925 INFO:teuthology.orchestra.run.smithi001.stdout: mon health detail to clog = false 2024-09-13T11:20:45.925 INFO:teuthology.orchestra.run.smithi001.stdout: mon host = "[v1:172.21.15.1:6789],[v1:172.21.15.92:6789],[v1:172.21.15.162:6789]" 2024-09-13T11:20:45.925 INFO:teuthology.orchestra.run.smithi001.stdout: mon client directed command retry = 5 2024-09-13T11:20:45.925 INFO:teuthology.orchestra.run.smithi001.stdout: mon election default strategy = 1 2024-09-13T11:20:45.926 INFO:teuthology.orchestra.run.smithi001.stdout: ms bind msgr2 = False 2024-09-13T11:20:45.926 INFO:teuthology.orchestra.run.smithi001.stdout: ms inject delay max = 1 2024-09-13T11:20:45.926 INFO:teuthology.orchestra.run.smithi001.stdout: ms inject delay probability = 0.005 2024-09-13T11:20:45.926 INFO:teuthology.orchestra.run.smithi001.stdout: ms inject delay type = osd 2024-09-13T11:20:45.926 INFO:teuthology.orchestra.run.smithi001.stdout: ms inject internal delays = 0.002 2024-09-13T11:20:45.926 INFO:teuthology.orchestra.run.smithi001.stdout: ms inject socket failures = 2500 2024-09-13T11:20:45.926 INFO:teuthology.orchestra.run.smithi001.stdout: ms type = async 2024-09-13T11:20:45.926 INFO:teuthology.orchestra.run.smithi001.stdout: osd_pool_default_min_size = 2 2024-09-13T11:20:45.926 INFO:teuthology.orchestra.run.smithi001.stdout: osd_pool_default_size = 2 2024-09-13T11:20:45.926 INFO:teuthology.orchestra.run.smithi001.stdout: fsid = 2fd68ac7-5581-4d90-80a4-30bcf5252a7a 2024-09-13T11:20:45.926 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-13T11:20:45.926 INFO:teuthology.orchestra.run.smithi001.stdout:[osd] 2024-09-13T11:20:45.927 INFO:teuthology.orchestra.run.smithi001.stdout: osd journal size = 100 2024-09-13T11:20:45.927 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-13T11:20:45.927 INFO:teuthology.orchestra.run.smithi001.stdout: osd scrub load threshold = 5.0 2024-09-13T11:20:45.927 INFO:teuthology.orchestra.run.smithi001.stdout: osd scrub max interval = 120 2024-09-13T11:20:45.927 INFO:teuthology.orchestra.run.smithi001.stdout: osd mclock profile = high_recovery_ops 2024-09-13T11:20:45.927 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-13T11:20:45.927 INFO:teuthology.orchestra.run.smithi001.stdout: osd recover clone overlap = true 2024-09-13T11:20:45.927 INFO:teuthology.orchestra.run.smithi001.stdout: osd recovery max chunk = 1048576 2024-09-13T11:20:45.927 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-13T11:20:45.927 INFO:teuthology.orchestra.run.smithi001.stdout: osd debug shutdown = true 2024-09-13T11:20:45.927 INFO:teuthology.orchestra.run.smithi001.stdout: osd debug op order = true 2024-09-13T11:20:45.928 INFO:teuthology.orchestra.run.smithi001.stdout: osd debug verify stray on activate = true 2024-09-13T11:20:45.928 INFO:teuthology.orchestra.run.smithi001.stdout: osd debug trim objects = true 2024-09-13T11:20:45.928 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-13T11:20:45.928 INFO:teuthology.orchestra.run.smithi001.stdout: osd open classes on start = true 2024-09-13T11:20:45.928 INFO:teuthology.orchestra.run.smithi001.stdout: osd debug pg log writeout = true 2024-09-13T11:20:45.928 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-13T11:20:45.928 INFO:teuthology.orchestra.run.smithi001.stdout: osd deep scrub update digest min age = 30 2024-09-13T11:20:45.928 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-13T11:20:45.928 INFO:teuthology.orchestra.run.smithi001.stdout: osd map max advance = 10 2024-09-13T11:20:45.928 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-13T11:20:45.928 INFO:teuthology.orchestra.run.smithi001.stdout: journal zero on create = true 2024-09-13T11:20:45.928 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-13T11:20:45.928 INFO:teuthology.orchestra.run.smithi001.stdout: filestore ondisk finisher threads = 3 2024-09-13T11:20:45.929 INFO:teuthology.orchestra.run.smithi001.stdout: filestore apply finisher threads = 3 2024-09-13T11:20:45.929 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-13T11:20:45.929 INFO:teuthology.orchestra.run.smithi001.stdout: bdev debug aio = true 2024-09-13T11:20:45.929 INFO:teuthology.orchestra.run.smithi001.stdout: osd debug misdirected ops = true 2024-09-13T11:20:45.929 INFO:teuthology.orchestra.run.smithi001.stdout: debug monc = 20 2024-09-13T11:20:45.929 INFO:teuthology.orchestra.run.smithi001.stdout: bdev async discard = True 2024-09-13T11:20:45.929 INFO:teuthology.orchestra.run.smithi001.stdout: bdev enable discard = True 2024-09-13T11:20:45.929 INFO:teuthology.orchestra.run.smithi001.stdout: bluestore allocator = stupid 2024-09-13T11:20:45.929 INFO:teuthology.orchestra.run.smithi001.stdout: bluestore block size = 96636764160 2024-09-13T11:20:45.929 INFO:teuthology.orchestra.run.smithi001.stdout: bluestore fsck on mount = True 2024-09-13T11:20:45.929 INFO:teuthology.orchestra.run.smithi001.stdout: bluestore write v2 = True 2024-09-13T11:20:45.929 INFO:teuthology.orchestra.run.smithi001.stdout: bluestore zero block detection = True 2024-09-13T11:20:45.929 INFO:teuthology.orchestra.run.smithi001.stdout: debug bluefs = 20 2024-09-13T11:20:45.930 INFO:teuthology.orchestra.run.smithi001.stdout: debug bluestore = 20 2024-09-13T11:20:45.930 INFO:teuthology.orchestra.run.smithi001.stdout: debug ms = 1 2024-09-13T11:20:45.930 INFO:teuthology.orchestra.run.smithi001.stdout: debug osd = 20 2024-09-13T11:20:45.930 INFO:teuthology.orchestra.run.smithi001.stdout: debug rocksdb = 10 2024-09-13T11:20:45.930 INFO:teuthology.orchestra.run.smithi001.stdout: mon osd backfillfull_ratio = 0.85 2024-09-13T11:20:45.930 INFO:teuthology.orchestra.run.smithi001.stdout: mon osd full ratio = 0.9 2024-09-13T11:20:45.930 INFO:teuthology.orchestra.run.smithi001.stdout: mon osd nearfull ratio = 0.8 2024-09-13T11:20:45.930 INFO:teuthology.orchestra.run.smithi001.stdout: osd backoff on peering = True 2024-09-13T11:20:45.930 INFO:teuthology.orchestra.run.smithi001.stdout: osd blocked scrub grace period = 3600 2024-09-13T11:20:45.930 INFO:teuthology.orchestra.run.smithi001.stdout: osd debug reject backfill probability = 0.3 2024-09-13T11:20:45.930 INFO:teuthology.orchestra.run.smithi001.stdout: osd debug verify cached snaps = True 2024-09-13T11:20:45.930 INFO:teuthology.orchestra.run.smithi001.stdout: osd debug verify missing on start = True 2024-09-13T11:20:45.930 INFO:teuthology.orchestra.run.smithi001.stdout: osd delete sleep = 1 2024-09-13T11:20:45.931 INFO:teuthology.orchestra.run.smithi001.stdout: osd failsafe full ratio = 0.95 2024-09-13T11:20:45.931 INFO:teuthology.orchestra.run.smithi001.stdout: osd inject bad map crc probability = 0.1 2024-09-13T11:20:45.931 INFO:teuthology.orchestra.run.smithi001.stdout: osd max backfills = 3 2024-09-13T11:20:45.931 INFO:teuthology.orchestra.run.smithi001.stdout: osd max markdown count = 1000 2024-09-13T11:20:45.931 INFO:teuthology.orchestra.run.smithi001.stdout: osd max scrubs = 5 2024-09-13T11:20:45.931 INFO:teuthology.orchestra.run.smithi001.stdout: osd mclock override recovery settings = True 2024-09-13T11:20:45.931 INFO:teuthology.orchestra.run.smithi001.stdout: osd objectstore = bluestore 2024-09-13T11:20:45.931 INFO:teuthology.orchestra.run.smithi001.stdout: osd op queue = debug_random 2024-09-13T11:20:45.931 INFO:teuthology.orchestra.run.smithi001.stdout: osd op queue cut off = debug_random 2024-09-13T11:20:45.931 INFO:teuthology.orchestra.run.smithi001.stdout: osd scrub min interval = 60 2024-09-13T11:20:45.931 INFO:teuthology.orchestra.run.smithi001.stdout: osd shutdown pgref assert = True 2024-09-13T11:20:45.931 INFO:teuthology.orchestra.run.smithi001.stdout: osd snap trim sleep = 2 2024-09-13T11:20:45.931 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-13T11:20:45.931 INFO:teuthology.orchestra.run.smithi001.stdout:[mgr] 2024-09-13T11:20:45.932 INFO:teuthology.orchestra.run.smithi001.stdout: debug ms = 1 2024-09-13T11:20:45.932 INFO:teuthology.orchestra.run.smithi001.stdout: debug mgr = 20 2024-09-13T11:20:45.932 INFO:teuthology.orchestra.run.smithi001.stdout: debug mon = 20 2024-09-13T11:20:45.932 INFO:teuthology.orchestra.run.smithi001.stdout: debug auth = 20 2024-09-13T11:20:45.932 INFO:teuthology.orchestra.run.smithi001.stdout: mon reweight min pgs per osd = 4 2024-09-13T11:20:45.932 INFO:teuthology.orchestra.run.smithi001.stdout: mon reweight min bytes per osd = 10 2024-09-13T11:20:45.932 INFO:teuthology.orchestra.run.smithi001.stdout: mgr/telemetry/nag = false 2024-09-13T11:20:45.932 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-13T11:20:45.932 INFO:teuthology.orchestra.run.smithi001.stdout:[mon] 2024-09-13T11:20:45.932 INFO:teuthology.orchestra.run.smithi001.stdout: debug ms = 1 2024-09-13T11:20:45.932 INFO:teuthology.orchestra.run.smithi001.stdout: debug mon = 20 2024-09-13T11:20:45.932 INFO:teuthology.orchestra.run.smithi001.stdout: debug paxos = 20 2024-09-13T11:20:45.932 INFO:teuthology.orchestra.run.smithi001.stdout: debug auth = 20 2024-09-13T11:20:45.933 INFO:teuthology.orchestra.run.smithi001.stdout: mon data avail warn = 5 2024-09-13T11:20:45.933 INFO:teuthology.orchestra.run.smithi001.stdout: mon mgr mkfs grace = 240 2024-09-13T11:20:45.933 INFO:teuthology.orchestra.run.smithi001.stdout: mon reweight min pgs per osd = 4 2024-09-13T11:20:45.933 INFO:teuthology.orchestra.run.smithi001.stdout: mon osd reporter subtree level = osd 2024-09-13T11:20:45.933 INFO:teuthology.orchestra.run.smithi001.stdout: mon osd prime pg temp = true 2024-09-13T11:20:45.933 INFO:teuthology.orchestra.run.smithi001.stdout: mon reweight min bytes per osd = 10 2024-09-13T11:20:45.933 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-13T11:20:45.933 INFO:teuthology.orchestra.run.smithi001.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-13T11:20:45.933 INFO:teuthology.orchestra.run.smithi001.stdout: auth mon ticket ttl = 660 # 11m 2024-09-13T11:20:45.933 INFO:teuthology.orchestra.run.smithi001.stdout: auth service ticket ttl = 240 # 4m 2024-09-13T11:20:45.933 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-13T11:20:45.933 INFO:teuthology.orchestra.run.smithi001.stdout: # don't complain about insecure global_id in the test suite 2024-09-13T11:20:45.933 INFO:teuthology.orchestra.run.smithi001.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-13T11:20:45.934 INFO:teuthology.orchestra.run.smithi001.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-13T11:20:45.934 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-13T11:20:45.934 INFO:teuthology.orchestra.run.smithi001.stdout: # 1m isn't quite enough 2024-09-13T11:20:45.934 INFO:teuthology.orchestra.run.smithi001.stdout: mon_down_mkfs_grace = 2m 2024-09-13T11:20:45.934 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-13T11:20:45.934 INFO:teuthology.orchestra.run.smithi001.stdout: mon_warn_on_filestore_osds = false 2024-09-13T11:20:45.934 INFO:teuthology.orchestra.run.smithi001.stdout: mon min osdmap epochs = 50 2024-09-13T11:20:45.934 INFO:teuthology.orchestra.run.smithi001.stdout: mon osdmap full prune interval = 2 2024-09-13T11:20:45.934 INFO:teuthology.orchestra.run.smithi001.stdout: mon osdmap full prune min = 15 2024-09-13T11:20:45.934 INFO:teuthology.orchestra.run.smithi001.stdout: mon osdmap full prune txsize = 2 2024-09-13T11:20:45.934 INFO:teuthology.orchestra.run.smithi001.stdout: mon scrub interval = 300 2024-09-13T11:20:45.934 INFO:teuthology.orchestra.run.smithi001.stdout: paxos service trim min = 10 2024-09-13T11:20:45.934 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-13T11:20:45.934 INFO:teuthology.orchestra.run.smithi001.stdout:[client] 2024-09-13T11:20:45.935 INFO:teuthology.orchestra.run.smithi001.stdout: rgw cache enabled = true 2024-09-13T11:20:45.935 INFO:teuthology.orchestra.run.smithi001.stdout: rgw enable ops log = true 2024-09-13T11:20:45.935 INFO:teuthology.orchestra.run.smithi001.stdout: rgw enable usage log = true 2024-09-13T11:20:45.935 INFO:teuthology.orchestra.run.smithi001.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-13T11:20:45.935 INFO:teuthology.orchestra.run.smithi001.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-13T11:20:45.935 INFO:teuthology.orchestra.run.smithi001.stdout:[mon.a] 2024-09-13T11:20:45.935 INFO:teuthology.orchestra.run.smithi001.stdout:[mon.b] 2024-09-13T11:20:45.935 INFO:teuthology.orchestra.run.smithi001.stdout:[mon.c] 2024-09-13T11:20:45.935 INFO:teuthology.orchestra.run.smithi001.stdout:[client.0] 2024-09-13T11:20:45.935 INFO:teuthology.orchestra.run.smithi001.stdout: admin socket = /var/run/ceph/ceph-$name.asok 2024-09-13T11:20:45.938 INFO:teuthology.orchestra.run.smithi177.stdout:[global] 2024-09-13T11:20:45.939 INFO:teuthology.orchestra.run.smithi177.stdout: chdir = "" 2024-09-13T11:20:45.939 INFO:teuthology.orchestra.run.smithi177.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-13T11:20:45.939 INFO:teuthology.orchestra.run.smithi177.stdout: auth supported = cephx 2024-09-13T11:20:45.939 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-13T11:20:45.939 INFO:teuthology.orchestra.run.smithi177.stdout: filestore xattr use omap = true 2024-09-13T11:20:45.939 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-13T11:20:45.939 INFO:teuthology.orchestra.run.smithi177.stdout: mon clock drift allowed = 1.000 2024-09-13T11:20:45.939 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-13T11:20:45.939 INFO:teuthology.orchestra.run.smithi177.stdout: osd crush chooseleaf type = 0 2024-09-13T11:20:45.939 INFO:teuthology.orchestra.run.smithi177.stdout: auth debug = true 2024-09-13T11:20:45.939 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-13T11:20:45.940 INFO:teuthology.orchestra.run.smithi177.stdout: ms die on old message = true 2024-09-13T11:20:45.940 INFO:teuthology.orchestra.run.smithi177.stdout: ms die on bug = true 2024-09-13T11:20:45.940 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-13T11:20:45.940 INFO:teuthology.orchestra.run.smithi177.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-13T11:20:45.940 INFO:teuthology.orchestra.run.smithi177.stdout: mon pg warn max object skew = 0 2024-09-13T11:20:45.940 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-13T11:20:45.940 INFO:teuthology.orchestra.run.smithi177.stdout: # disable pg_autoscaler by default for new pools 2024-09-13T11:20:45.940 INFO:teuthology.orchestra.run.smithi177.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-13T11:20:45.940 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-13T11:20:45.940 INFO:teuthology.orchestra.run.smithi177.stdout: osd pool default size = 2 2024-09-13T11:20:45.940 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-13T11:20:45.940 INFO:teuthology.orchestra.run.smithi177.stdout: mon osd allow primary affinity = true 2024-09-13T11:20:45.941 INFO:teuthology.orchestra.run.smithi177.stdout: mon osd allow pg remap = true 2024-09-13T11:20:45.941 INFO:teuthology.orchestra.run.smithi177.stdout: mon warn on legacy crush tunables = false 2024-09-13T11:20:45.941 INFO:teuthology.orchestra.run.smithi177.stdout: mon warn on crush straw calc version zero = false 2024-09-13T11:20:45.941 INFO:teuthology.orchestra.run.smithi177.stdout: mon warn on no sortbitwise = false 2024-09-13T11:20:45.941 INFO:teuthology.orchestra.run.smithi177.stdout: mon warn on osd down out interval zero = false 2024-09-13T11:20:45.941 INFO:teuthology.orchestra.run.smithi177.stdout: mon warn on too few osds = false 2024-09-13T11:20:45.941 INFO:teuthology.orchestra.run.smithi177.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-13T11:20:45.941 INFO:teuthology.orchestra.run.smithi177.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-13T11:20:45.941 INFO:teuthology.orchestra.run.smithi177.stdout: mon_allow_pool_size_one = true 2024-09-13T11:20:45.941 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-13T11:20:45.941 INFO:teuthology.orchestra.run.smithi177.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-13T11:20:45.941 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-13T11:20:45.942 INFO:teuthology.orchestra.run.smithi177.stdout: osd default data pool replay window = 5 2024-09-13T11:20:45.942 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-13T11:20:45.942 INFO:teuthology.orchestra.run.smithi177.stdout: mon allow pool delete = true 2024-09-13T11:20:45.942 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-13T11:20:45.942 INFO:teuthology.orchestra.run.smithi177.stdout: mon cluster log file level = debug 2024-09-13T11:20:45.942 INFO:teuthology.orchestra.run.smithi177.stdout: debug asserts on shutdown = true 2024-09-13T11:20:45.942 INFO:teuthology.orchestra.run.smithi177.stdout: mon health detail to clog = false 2024-09-13T11:20:45.942 INFO:teuthology.orchestra.run.smithi177.stdout: mon host = "[v1:172.21.15.1:6789],[v1:172.21.15.92:6789],[v1:172.21.15.162:6789]" 2024-09-13T11:20:45.942 INFO:teuthology.orchestra.run.smithi177.stdout: mon client directed command retry = 5 2024-09-13T11:20:45.942 INFO:teuthology.orchestra.run.smithi177.stdout: mon election default strategy = 1 2024-09-13T11:20:45.942 INFO:teuthology.orchestra.run.smithi177.stdout: ms bind msgr2 = False 2024-09-13T11:20:45.942 INFO:teuthology.orchestra.run.smithi177.stdout: ms inject delay max = 1 2024-09-13T11:20:45.942 INFO:teuthology.orchestra.run.smithi177.stdout: ms inject delay probability = 0.005 2024-09-13T11:20:45.943 INFO:teuthology.orchestra.run.smithi177.stdout: ms inject delay type = osd 2024-09-13T11:20:45.943 INFO:teuthology.orchestra.run.smithi177.stdout: ms inject internal delays = 0.002 2024-09-13T11:20:45.943 INFO:teuthology.orchestra.run.smithi177.stdout: ms inject socket failures = 2500 2024-09-13T11:20:45.943 INFO:teuthology.orchestra.run.smithi177.stdout: ms type = async 2024-09-13T11:20:45.943 INFO:teuthology.orchestra.run.smithi177.stdout: osd_pool_default_min_size = 2 2024-09-13T11:20:45.943 INFO:teuthology.orchestra.run.smithi177.stdout: osd_pool_default_size = 2 2024-09-13T11:20:45.943 INFO:teuthology.orchestra.run.smithi177.stdout: fsid = 2fd68ac7-5581-4d90-80a4-30bcf5252a7a 2024-09-13T11:20:45.943 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-13T11:20:45.943 INFO:teuthology.orchestra.run.smithi177.stdout:[osd] 2024-09-13T11:20:45.943 INFO:teuthology.orchestra.run.smithi177.stdout: osd journal size = 100 2024-09-13T11:20:45.943 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-13T11:20:45.943 INFO:teuthology.orchestra.run.smithi177.stdout: osd scrub load threshold = 5.0 2024-09-13T11:20:45.943 INFO:teuthology.orchestra.run.smithi177.stdout: osd scrub max interval = 120 2024-09-13T11:20:45.944 INFO:teuthology.orchestra.run.smithi177.stdout: osd mclock profile = high_recovery_ops 2024-09-13T11:20:45.944 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-13T11:20:45.944 INFO:teuthology.orchestra.run.smithi177.stdout: osd recover clone overlap = true 2024-09-13T11:20:45.944 INFO:teuthology.orchestra.run.smithi177.stdout: osd recovery max chunk = 1048576 2024-09-13T11:20:45.944 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-13T11:20:45.944 INFO:teuthology.orchestra.run.smithi177.stdout: osd debug shutdown = true 2024-09-13T11:20:45.944 INFO:teuthology.orchestra.run.smithi177.stdout: osd debug op order = true 2024-09-13T11:20:45.944 INFO:teuthology.orchestra.run.smithi177.stdout: osd debug verify stray on activate = true 2024-09-13T11:20:45.944 INFO:teuthology.orchestra.run.smithi177.stdout: osd debug trim objects = true 2024-09-13T11:20:45.945 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-13T11:20:45.945 INFO:teuthology.orchestra.run.smithi177.stdout: osd open classes on start = true 2024-09-13T11:20:45.945 INFO:teuthology.orchestra.run.smithi177.stdout: osd debug pg log writeout = true 2024-09-13T11:20:45.945 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-13T11:20:45.945 INFO:teuthology.orchestra.run.smithi177.stdout: osd deep scrub update digest min age = 30 2024-09-13T11:20:45.945 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-13T11:20:45.945 INFO:teuthology.orchestra.run.smithi177.stdout: osd map max advance = 10 2024-09-13T11:20:45.945 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-13T11:20:45.945 INFO:teuthology.orchestra.run.smithi177.stdout: journal zero on create = true 2024-09-13T11:20:45.945 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-13T11:20:45.945 INFO:teuthology.orchestra.run.smithi177.stdout: filestore ondisk finisher threads = 3 2024-09-13T11:20:45.945 INFO:teuthology.orchestra.run.smithi177.stdout: filestore apply finisher threads = 3 2024-09-13T11:20:45.946 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-13T11:20:45.946 INFO:teuthology.orchestra.run.smithi177.stdout: bdev debug aio = true 2024-09-13T11:20:45.946 INFO:teuthology.orchestra.run.smithi177.stdout: osd debug misdirected ops = true 2024-09-13T11:20:45.946 INFO:teuthology.orchestra.run.smithi177.stdout: debug monc = 20 2024-09-13T11:20:45.946 INFO:teuthology.orchestra.run.smithi177.stdout: bdev async discard = True 2024-09-13T11:20:45.946 INFO:teuthology.orchestra.run.smithi177.stdout: bdev enable discard = True 2024-09-13T11:20:45.946 INFO:teuthology.orchestra.run.smithi177.stdout: bluestore allocator = stupid 2024-09-13T11:20:45.946 INFO:teuthology.orchestra.run.smithi177.stdout: bluestore block size = 96636764160 2024-09-13T11:20:45.946 INFO:teuthology.orchestra.run.smithi177.stdout: bluestore fsck on mount = True 2024-09-13T11:20:45.946 INFO:teuthology.orchestra.run.smithi177.stdout: bluestore write v2 = True 2024-09-13T11:20:45.946 INFO:teuthology.orchestra.run.smithi177.stdout: bluestore zero block detection = True 2024-09-13T11:20:45.946 INFO:teuthology.orchestra.run.smithi177.stdout: debug bluefs = 20 2024-09-13T11:20:45.946 INFO:teuthology.orchestra.run.smithi177.stdout: debug bluestore = 20 2024-09-13T11:20:45.947 INFO:teuthology.orchestra.run.smithi177.stdout: debug ms = 1 2024-09-13T11:20:45.947 INFO:teuthology.orchestra.run.smithi177.stdout: debug osd = 20 2024-09-13T11:20:45.947 INFO:teuthology.orchestra.run.smithi177.stdout: debug rocksdb = 10 2024-09-13T11:20:45.947 INFO:teuthology.orchestra.run.smithi177.stdout: mon osd backfillfull_ratio = 0.85 2024-09-13T11:20:45.947 INFO:teuthology.orchestra.run.smithi177.stdout: mon osd full ratio = 0.9 2024-09-13T11:20:45.947 INFO:teuthology.orchestra.run.smithi177.stdout: mon osd nearfull ratio = 0.8 2024-09-13T11:20:45.947 INFO:teuthology.orchestra.run.smithi177.stdout: osd backoff on peering = True 2024-09-13T11:20:45.947 INFO:teuthology.orchestra.run.smithi177.stdout: osd blocked scrub grace period = 3600 2024-09-13T11:20:45.947 INFO:teuthology.orchestra.run.smithi177.stdout: osd debug reject backfill probability = 0.3 2024-09-13T11:20:45.947 INFO:teuthology.orchestra.run.smithi177.stdout: osd debug verify cached snaps = True 2024-09-13T11:20:45.947 INFO:teuthology.orchestra.run.smithi177.stdout: osd debug verify missing on start = True 2024-09-13T11:20:45.947 INFO:teuthology.orchestra.run.smithi177.stdout: osd delete sleep = 1 2024-09-13T11:20:45.947 INFO:teuthology.orchestra.run.smithi177.stdout: osd failsafe full ratio = 0.95 2024-09-13T11:20:45.948 INFO:teuthology.orchestra.run.smithi177.stdout: osd inject bad map crc probability = 0.1 2024-09-13T11:20:45.948 INFO:teuthology.orchestra.run.smithi177.stdout: osd max backfills = 3 2024-09-13T11:20:45.948 INFO:teuthology.orchestra.run.smithi177.stdout: osd max markdown count = 1000 2024-09-13T11:20:45.948 INFO:teuthology.orchestra.run.smithi177.stdout: osd max scrubs = 5 2024-09-13T11:20:45.948 INFO:teuthology.orchestra.run.smithi177.stdout: osd mclock override recovery settings = True 2024-09-13T11:20:45.948 INFO:teuthology.orchestra.run.smithi177.stdout: osd objectstore = bluestore 2024-09-13T11:20:45.948 INFO:teuthology.orchestra.run.smithi177.stdout: osd op queue = debug_random 2024-09-13T11:20:45.948 INFO:teuthology.orchestra.run.smithi177.stdout: osd op queue cut off = debug_random 2024-09-13T11:20:45.948 INFO:teuthology.orchestra.run.smithi177.stdout: osd scrub min interval = 60 2024-09-13T11:20:45.948 INFO:teuthology.orchestra.run.smithi177.stdout: osd shutdown pgref assert = True 2024-09-13T11:20:45.948 INFO:teuthology.orchestra.run.smithi177.stdout: osd snap trim sleep = 2 2024-09-13T11:20:45.948 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-13T11:20:45.949 INFO:teuthology.orchestra.run.smithi177.stdout:[mgr] 2024-09-13T11:20:45.949 INFO:teuthology.orchestra.run.smithi177.stdout: debug ms = 1 2024-09-13T11:20:45.949 INFO:teuthology.orchestra.run.smithi177.stdout: debug mgr = 20 2024-09-13T11:20:45.949 INFO:teuthology.orchestra.run.smithi177.stdout: debug mon = 20 2024-09-13T11:20:45.949 INFO:teuthology.orchestra.run.smithi177.stdout: debug auth = 20 2024-09-13T11:20:45.949 INFO:teuthology.orchestra.run.smithi177.stdout: mon reweight min pgs per osd = 4 2024-09-13T11:20:45.949 INFO:teuthology.orchestra.run.smithi177.stdout: mon reweight min bytes per osd = 10 2024-09-13T11:20:45.949 INFO:teuthology.orchestra.run.smithi177.stdout: mgr/telemetry/nag = false 2024-09-13T11:20:45.949 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-13T11:20:45.949 INFO:teuthology.orchestra.run.smithi177.stdout:[mon] 2024-09-13T11:20:45.949 INFO:teuthology.orchestra.run.smithi177.stdout: debug ms = 1 2024-09-13T11:20:45.949 INFO:teuthology.orchestra.run.smithi177.stdout: debug mon = 20 2024-09-13T11:20:45.949 INFO:teuthology.orchestra.run.smithi177.stdout: debug paxos = 20 2024-09-13T11:20:45.950 INFO:teuthology.orchestra.run.smithi177.stdout: debug auth = 20 2024-09-13T11:20:45.950 INFO:teuthology.orchestra.run.smithi177.stdout: mon data avail warn = 5 2024-09-13T11:20:45.950 INFO:teuthology.orchestra.run.smithi177.stdout: mon mgr mkfs grace = 240 2024-09-13T11:20:45.950 INFO:teuthology.orchestra.run.smithi177.stdout: mon reweight min pgs per osd = 4 2024-09-13T11:20:45.950 INFO:teuthology.orchestra.run.smithi177.stdout: mon osd reporter subtree level = osd 2024-09-13T11:20:45.950 INFO:teuthology.orchestra.run.smithi177.stdout: mon osd prime pg temp = true 2024-09-13T11:20:45.950 INFO:teuthology.orchestra.run.smithi177.stdout: mon reweight min bytes per osd = 10 2024-09-13T11:20:45.950 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-13T11:20:45.950 INFO:teuthology.orchestra.run.smithi177.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-13T11:20:45.950 INFO:teuthology.orchestra.run.smithi177.stdout: auth mon ticket ttl = 660 # 11m 2024-09-13T11:20:45.950 INFO:teuthology.orchestra.run.smithi177.stdout: auth service ticket ttl = 240 # 4m 2024-09-13T11:20:45.950 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-13T11:20:45.951 INFO:teuthology.orchestra.run.smithi177.stdout: # don't complain about insecure global_id in the test suite 2024-09-13T11:20:45.951 INFO:teuthology.orchestra.run.smithi177.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-13T11:20:45.951 INFO:teuthology.orchestra.run.smithi177.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-13T11:20:45.951 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-13T11:20:45.951 INFO:teuthology.orchestra.run.smithi177.stdout: # 1m isn't quite enough 2024-09-13T11:20:45.951 INFO:teuthology.orchestra.run.smithi177.stdout: mon_down_mkfs_grace = 2m 2024-09-13T11:20:45.951 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-13T11:20:45.951 INFO:teuthology.orchestra.run.smithi177.stdout: mon_warn_on_filestore_osds = false 2024-09-13T11:20:45.951 INFO:teuthology.orchestra.run.smithi177.stdout: mon min osdmap epochs = 50 2024-09-13T11:20:45.951 INFO:teuthology.orchestra.run.smithi177.stdout: mon osdmap full prune interval = 2 2024-09-13T11:20:45.951 INFO:teuthology.orchestra.run.smithi177.stdout: mon osdmap full prune min = 15 2024-09-13T11:20:45.951 INFO:teuthology.orchestra.run.smithi177.stdout: mon osdmap full prune txsize = 2 2024-09-13T11:20:45.951 INFO:teuthology.orchestra.run.smithi177.stdout: mon scrub interval = 300 2024-09-13T11:20:45.952 INFO:teuthology.orchestra.run.smithi177.stdout: paxos service trim min = 10 2024-09-13T11:20:45.952 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-13T11:20:45.952 INFO:teuthology.orchestra.run.smithi177.stdout:[client] 2024-09-13T11:20:45.952 INFO:teuthology.orchestra.run.smithi177.stdout: rgw cache enabled = true 2024-09-13T11:20:45.952 INFO:teuthology.orchestra.run.smithi177.stdout: rgw enable ops log = true 2024-09-13T11:20:45.952 INFO:teuthology.orchestra.run.smithi177.stdout: rgw enable usage log = true 2024-09-13T11:20:45.952 INFO:teuthology.orchestra.run.smithi177.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-13T11:20:45.952 INFO:teuthology.orchestra.run.smithi177.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-13T11:20:45.952 INFO:teuthology.orchestra.run.smithi177.stdout:[mon.a] 2024-09-13T11:20:45.952 INFO:teuthology.orchestra.run.smithi177.stdout:[mon.b] 2024-09-13T11:20:45.952 INFO:teuthology.orchestra.run.smithi177.stdout:[mon.c] 2024-09-13T11:20:45.952 INFO:teuthology.orchestra.run.smithi177.stdout:[client.0] 2024-09-13T11:20:45.952 INFO:teuthology.orchestra.run.smithi177.stdout: admin socket = /var/run/ceph/ceph-$name.asok 2024-09-13T11:20:45.953 INFO:teuthology.orchestra.run.smithi162.stdout:[global] 2024-09-13T11:20:45.953 INFO:teuthology.orchestra.run.smithi162.stdout: chdir = "" 2024-09-13T11:20:45.953 INFO:teuthology.orchestra.run.smithi162.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-13T11:20:45.953 INFO:teuthology.orchestra.run.smithi162.stdout: auth supported = cephx 2024-09-13T11:20:45.953 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-09-13T11:20:45.953 INFO:teuthology.orchestra.run.smithi162.stdout: filestore xattr use omap = true 2024-09-13T11:20:45.953 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-09-13T11:20:45.953 INFO:teuthology.orchestra.run.smithi162.stdout: mon clock drift allowed = 1.000 2024-09-13T11:20:45.953 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-09-13T11:20:45.953 INFO:teuthology.orchestra.run.smithi162.stdout: osd crush chooseleaf type = 0 2024-09-13T11:20:45.954 INFO:teuthology.orchestra.run.smithi162.stdout: auth debug = true 2024-09-13T11:20:45.954 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-09-13T11:20:45.954 INFO:teuthology.orchestra.run.smithi162.stdout: ms die on old message = true 2024-09-13T11:20:45.954 INFO:teuthology.orchestra.run.smithi162.stdout: ms die on bug = true 2024-09-13T11:20:45.954 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-09-13T11:20:45.954 INFO:teuthology.orchestra.run.smithi162.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-13T11:20:45.954 INFO:teuthology.orchestra.run.smithi162.stdout: mon pg warn max object skew = 0 2024-09-13T11:20:45.954 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-09-13T11:20:45.954 INFO:teuthology.orchestra.run.smithi162.stdout: # disable pg_autoscaler by default for new pools 2024-09-13T11:20:45.954 INFO:teuthology.orchestra.run.smithi162.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-13T11:20:45.954 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-09-13T11:20:45.954 INFO:teuthology.orchestra.run.smithi162.stdout: osd pool default size = 2 2024-09-13T11:20:45.954 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-09-13T11:20:45.954 INFO:teuthology.orchestra.run.smithi162.stdout: mon osd allow primary affinity = true 2024-09-13T11:20:45.955 INFO:teuthology.orchestra.run.smithi162.stdout: mon osd allow pg remap = true 2024-09-13T11:20:45.955 INFO:teuthology.orchestra.run.smithi162.stdout: mon warn on legacy crush tunables = false 2024-09-13T11:20:45.955 INFO:teuthology.orchestra.run.smithi162.stdout: mon warn on crush straw calc version zero = false 2024-09-13T11:20:45.955 INFO:teuthology.orchestra.run.smithi162.stdout: mon warn on no sortbitwise = false 2024-09-13T11:20:45.955 INFO:teuthology.orchestra.run.smithi162.stdout: mon warn on osd down out interval zero = false 2024-09-13T11:20:45.955 INFO:teuthology.orchestra.run.smithi162.stdout: mon warn on too few osds = false 2024-09-13T11:20:45.955 INFO:teuthology.orchestra.run.smithi162.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-13T11:20:45.955 INFO:teuthology.orchestra.run.smithi162.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-13T11:20:45.955 INFO:teuthology.orchestra.run.smithi162.stdout: mon_allow_pool_size_one = true 2024-09-13T11:20:45.955 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-09-13T11:20:45.955 INFO:teuthology.orchestra.run.smithi162.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-13T11:20:45.955 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-09-13T11:20:45.955 INFO:teuthology.orchestra.run.smithi162.stdout: osd default data pool replay window = 5 2024-09-13T11:20:45.955 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-09-13T11:20:45.956 INFO:teuthology.orchestra.run.smithi162.stdout: mon allow pool delete = true 2024-09-13T11:20:45.956 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-09-13T11:20:45.956 INFO:teuthology.orchestra.run.smithi162.stdout: mon cluster log file level = debug 2024-09-13T11:20:45.956 INFO:teuthology.orchestra.run.smithi162.stdout: debug asserts on shutdown = true 2024-09-13T11:20:45.956 INFO:teuthology.orchestra.run.smithi162.stdout: mon health detail to clog = false 2024-09-13T11:20:45.956 INFO:teuthology.orchestra.run.smithi162.stdout: mon host = "[v1:172.21.15.1:6789],[v1:172.21.15.92:6789],[v1:172.21.15.162:6789]" 2024-09-13T11:20:45.956 INFO:teuthology.orchestra.run.smithi162.stdout: mon client directed command retry = 5 2024-09-13T11:20:45.956 INFO:teuthology.orchestra.run.smithi162.stdout: mon election default strategy = 1 2024-09-13T11:20:45.956 INFO:teuthology.orchestra.run.smithi162.stdout: ms bind msgr2 = False 2024-09-13T11:20:45.956 INFO:teuthology.orchestra.run.smithi162.stdout: ms inject delay max = 1 2024-09-13T11:20:45.956 INFO:teuthology.orchestra.run.smithi162.stdout: ms inject delay probability = 0.005 2024-09-13T11:20:45.957 INFO:teuthology.orchestra.run.smithi162.stdout: ms inject delay type = osd 2024-09-13T11:20:45.957 INFO:teuthology.orchestra.run.smithi162.stdout: ms inject internal delays = 0.002 2024-09-13T11:20:45.957 INFO:teuthology.orchestra.run.smithi162.stdout: ms inject socket failures = 2500 2024-09-13T11:20:45.957 INFO:teuthology.orchestra.run.smithi162.stdout: ms type = async 2024-09-13T11:20:45.957 INFO:teuthology.orchestra.run.smithi162.stdout: osd_pool_default_min_size = 2 2024-09-13T11:20:45.957 INFO:teuthology.orchestra.run.smithi162.stdout: osd_pool_default_size = 2 2024-09-13T11:20:45.957 INFO:teuthology.orchestra.run.smithi162.stdout: fsid = 2fd68ac7-5581-4d90-80a4-30bcf5252a7a 2024-09-13T11:20:45.957 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-09-13T11:20:45.957 INFO:teuthology.orchestra.run.smithi162.stdout:[osd] 2024-09-13T11:20:45.957 INFO:teuthology.orchestra.run.smithi162.stdout: osd journal size = 100 2024-09-13T11:20:45.957 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-09-13T11:20:45.957 INFO:teuthology.orchestra.run.smithi162.stdout: osd scrub load threshold = 5.0 2024-09-13T11:20:45.958 INFO:teuthology.orchestra.run.smithi162.stdout: osd scrub max interval = 120 2024-09-13T11:20:45.958 INFO:teuthology.orchestra.run.smithi162.stdout: osd mclock profile = high_recovery_ops 2024-09-13T11:20:45.958 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-09-13T11:20:45.958 INFO:teuthology.orchestra.run.smithi162.stdout: osd recover clone overlap = true 2024-09-13T11:20:45.958 INFO:teuthology.orchestra.run.smithi162.stdout: osd recovery max chunk = 1048576 2024-09-13T11:20:45.958 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-09-13T11:20:45.958 INFO:teuthology.orchestra.run.smithi162.stdout: osd debug shutdown = true 2024-09-13T11:20:45.958 INFO:teuthology.orchestra.run.smithi162.stdout: osd debug op order = true 2024-09-13T11:20:45.958 INFO:teuthology.orchestra.run.smithi162.stdout: osd debug verify stray on activate = true 2024-09-13T11:20:45.958 INFO:teuthology.orchestra.run.smithi162.stdout: osd debug trim objects = true 2024-09-13T11:20:45.958 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-09-13T11:20:45.959 INFO:teuthology.orchestra.run.smithi162.stdout: osd open classes on start = true 2024-09-13T11:20:45.959 INFO:teuthology.orchestra.run.smithi162.stdout: osd debug pg log writeout = true 2024-09-13T11:20:45.959 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-09-13T11:20:45.959 INFO:teuthology.orchestra.run.smithi162.stdout: osd deep scrub update digest min age = 30 2024-09-13T11:20:45.959 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-09-13T11:20:45.959 INFO:teuthology.orchestra.run.smithi162.stdout: osd map max advance = 10 2024-09-13T11:20:45.959 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-09-13T11:20:45.959 INFO:teuthology.orchestra.run.smithi162.stdout: journal zero on create = true 2024-09-13T11:20:45.959 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-09-13T11:20:45.959 INFO:teuthology.orchestra.run.smithi162.stdout: filestore ondisk finisher threads = 3 2024-09-13T11:20:45.960 INFO:teuthology.orchestra.run.smithi162.stdout: filestore apply finisher threads = 3 2024-09-13T11:20:45.960 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-09-13T11:20:45.960 INFO:teuthology.orchestra.run.smithi162.stdout: bdev debug aio = true 2024-09-13T11:20:45.960 INFO:teuthology.orchestra.run.smithi162.stdout: osd debug misdirected ops = true 2024-09-13T11:20:45.960 INFO:teuthology.orchestra.run.smithi162.stdout: debug monc = 20 2024-09-13T11:20:45.960 INFO:teuthology.orchestra.run.smithi162.stdout: bdev async discard = True 2024-09-13T11:20:45.960 INFO:teuthology.orchestra.run.smithi162.stdout: bdev enable discard = True 2024-09-13T11:20:45.960 INFO:teuthology.orchestra.run.smithi162.stdout: bluestore allocator = stupid 2024-09-13T11:20:45.960 INFO:teuthology.orchestra.run.smithi162.stdout: bluestore block size = 96636764160 2024-09-13T11:20:45.960 INFO:teuthology.orchestra.run.smithi162.stdout: bluestore fsck on mount = True 2024-09-13T11:20:45.961 INFO:teuthology.orchestra.run.smithi162.stdout: bluestore write v2 = True 2024-09-13T11:20:45.961 INFO:teuthology.orchestra.run.smithi162.stdout: bluestore zero block detection = True 2024-09-13T11:20:45.961 INFO:teuthology.orchestra.run.smithi162.stdout: debug bluefs = 20 2024-09-13T11:20:45.961 INFO:teuthology.orchestra.run.smithi162.stdout: debug bluestore = 20 2024-09-13T11:20:45.961 INFO:teuthology.orchestra.run.smithi162.stdout: debug ms = 1 2024-09-13T11:20:45.961 INFO:teuthology.orchestra.run.smithi162.stdout: debug osd = 20 2024-09-13T11:20:45.961 INFO:teuthology.orchestra.run.smithi162.stdout: debug rocksdb = 10 2024-09-13T11:20:45.961 INFO:teuthology.orchestra.run.smithi162.stdout: mon osd backfillfull_ratio = 0.85 2024-09-13T11:20:45.961 INFO:teuthology.orchestra.run.smithi162.stdout: mon osd full ratio = 0.9 2024-09-13T11:20:45.961 INFO:teuthology.orchestra.run.smithi162.stdout: mon osd nearfull ratio = 0.8 2024-09-13T11:20:45.962 INFO:teuthology.orchestra.run.smithi162.stdout: osd backoff on peering = True 2024-09-13T11:20:45.962 INFO:teuthology.orchestra.run.smithi162.stdout: osd blocked scrub grace period = 3600 2024-09-13T11:20:45.962 INFO:teuthology.orchestra.run.smithi162.stdout: osd debug reject backfill probability = 0.3 2024-09-13T11:20:45.962 INFO:teuthology.orchestra.run.smithi162.stdout: osd debug verify cached snaps = True 2024-09-13T11:20:45.962 INFO:teuthology.orchestra.run.smithi162.stdout: osd debug verify missing on start = True 2024-09-13T11:20:45.962 INFO:teuthology.orchestra.run.smithi162.stdout: osd delete sleep = 1 2024-09-13T11:20:45.962 INFO:teuthology.orchestra.run.smithi162.stdout: osd failsafe full ratio = 0.95 2024-09-13T11:20:45.962 INFO:teuthology.orchestra.run.smithi162.stdout: osd inject bad map crc probability = 0.1 2024-09-13T11:20:45.962 INFO:teuthology.orchestra.run.smithi162.stdout: osd max backfills = 3 2024-09-13T11:20:45.962 INFO:teuthology.orchestra.run.smithi162.stdout: osd max markdown count = 1000 2024-09-13T11:20:45.962 INFO:teuthology.orchestra.run.smithi162.stdout: osd max scrubs = 5 2024-09-13T11:20:45.963 INFO:teuthology.orchestra.run.smithi162.stdout: osd mclock override recovery settings = True 2024-09-13T11:20:45.963 INFO:teuthology.orchestra.run.smithi162.stdout: osd objectstore = bluestore 2024-09-13T11:20:45.963 INFO:teuthology.orchestra.run.smithi162.stdout: osd op queue = debug_random 2024-09-13T11:20:45.963 INFO:teuthology.orchestra.run.smithi162.stdout: osd op queue cut off = debug_random 2024-09-13T11:20:45.963 INFO:teuthology.orchestra.run.smithi162.stdout: osd scrub min interval = 60 2024-09-13T11:20:45.963 INFO:teuthology.orchestra.run.smithi162.stdout: osd shutdown pgref assert = True 2024-09-13T11:20:45.963 INFO:teuthology.orchestra.run.smithi162.stdout: osd snap trim sleep = 2 2024-09-13T11:20:45.963 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-09-13T11:20:45.963 INFO:teuthology.orchestra.run.smithi162.stdout:[mgr] 2024-09-13T11:20:45.963 INFO:teuthology.orchestra.run.smithi162.stdout: debug ms = 1 2024-09-13T11:20:45.963 INFO:teuthology.orchestra.run.smithi162.stdout: debug mgr = 20 2024-09-13T11:20:45.964 INFO:teuthology.orchestra.run.smithi162.stdout: debug mon = 20 2024-09-13T11:20:45.964 INFO:teuthology.orchestra.run.smithi162.stdout: debug auth = 20 2024-09-13T11:20:45.964 INFO:teuthology.orchestra.run.smithi162.stdout: mon reweight min pgs per osd = 4 2024-09-13T11:20:45.964 INFO:teuthology.orchestra.run.smithi162.stdout: mon reweight min bytes per osd = 10 2024-09-13T11:20:45.964 INFO:teuthology.orchestra.run.smithi162.stdout: mgr/telemetry/nag = false 2024-09-13T11:20:45.964 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-09-13T11:20:45.964 INFO:teuthology.orchestra.run.smithi162.stdout:[mon] 2024-09-13T11:20:45.964 INFO:teuthology.orchestra.run.smithi162.stdout: debug ms = 1 2024-09-13T11:20:45.964 INFO:teuthology.orchestra.run.smithi162.stdout: debug mon = 20 2024-09-13T11:20:45.964 INFO:teuthology.orchestra.run.smithi162.stdout: debug paxos = 20 2024-09-13T11:20:45.964 INFO:teuthology.orchestra.run.smithi162.stdout: debug auth = 20 2024-09-13T11:20:45.965 INFO:teuthology.orchestra.run.smithi162.stdout: mon data avail warn = 5 2024-09-13T11:20:45.965 INFO:teuthology.orchestra.run.smithi162.stdout: mon mgr mkfs grace = 240 2024-09-13T11:20:45.965 INFO:teuthology.orchestra.run.smithi162.stdout: mon reweight min pgs per osd = 4 2024-09-13T11:20:45.965 INFO:teuthology.orchestra.run.smithi162.stdout: mon osd reporter subtree level = osd 2024-09-13T11:20:45.965 INFO:teuthology.orchestra.run.smithi162.stdout: mon osd prime pg temp = true 2024-09-13T11:20:45.965 INFO:teuthology.orchestra.run.smithi162.stdout: mon reweight min bytes per osd = 10 2024-09-13T11:20:45.965 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-09-13T11:20:45.965 INFO:teuthology.orchestra.run.smithi162.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-13T11:20:45.965 INFO:teuthology.orchestra.run.smithi162.stdout: auth mon ticket ttl = 660 # 11m 2024-09-13T11:20:45.965 INFO:teuthology.orchestra.run.smithi162.stdout: auth service ticket ttl = 240 # 4m 2024-09-13T11:20:45.965 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-09-13T11:20:45.966 INFO:teuthology.orchestra.run.smithi162.stdout: # don't complain about insecure global_id in the test suite 2024-09-13T11:20:45.966 INFO:teuthology.orchestra.run.smithi162.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-13T11:20:45.966 INFO:teuthology.orchestra.run.smithi162.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-13T11:20:45.966 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-09-13T11:20:45.966 INFO:teuthology.orchestra.run.smithi162.stdout: # 1m isn't quite enough 2024-09-13T11:20:45.966 INFO:teuthology.orchestra.run.smithi162.stdout: mon_down_mkfs_grace = 2m 2024-09-13T11:20:45.966 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-09-13T11:20:45.966 INFO:teuthology.orchestra.run.smithi162.stdout: mon_warn_on_filestore_osds = false 2024-09-13T11:20:45.966 INFO:teuthology.orchestra.run.smithi162.stdout: mon min osdmap epochs = 50 2024-09-13T11:20:45.966 INFO:teuthology.orchestra.run.smithi162.stdout: mon osdmap full prune interval = 2 2024-09-13T11:20:45.966 INFO:teuthology.orchestra.run.smithi162.stdout: mon osdmap full prune min = 15 2024-09-13T11:20:45.967 INFO:teuthology.orchestra.run.smithi162.stdout: mon osdmap full prune txsize = 2 2024-09-13T11:20:45.967 INFO:teuthology.orchestra.run.smithi162.stdout: mon scrub interval = 300 2024-09-13T11:20:45.967 INFO:teuthology.orchestra.run.smithi162.stdout: paxos service trim min = 10 2024-09-13T11:20:45.967 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-09-13T11:20:45.967 INFO:teuthology.orchestra.run.smithi162.stdout:[client] 2024-09-13T11:20:45.967 INFO:teuthology.orchestra.run.smithi162.stdout: rgw cache enabled = true 2024-09-13T11:20:45.967 INFO:teuthology.orchestra.run.smithi162.stdout: rgw enable ops log = true 2024-09-13T11:20:45.967 INFO:teuthology.orchestra.run.smithi162.stdout: rgw enable usage log = true 2024-09-13T11:20:45.967 INFO:teuthology.orchestra.run.smithi162.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-13T11:20:45.967 INFO:teuthology.orchestra.run.smithi162.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-13T11:20:45.967 INFO:teuthology.orchestra.run.smithi162.stdout:[mon.a] 2024-09-13T11:20:45.968 INFO:teuthology.orchestra.run.smithi162.stdout:[mon.b] 2024-09-13T11:20:45.968 INFO:teuthology.orchestra.run.smithi162.stdout:[mon.c] 2024-09-13T11:20:45.968 INFO:teuthology.orchestra.run.smithi162.stdout:[client.0] 2024-09-13T11:20:45.968 INFO:teuthology.orchestra.run.smithi162.stdout: admin socket = /var/run/ceph/ceph-$name.asok 2024-09-13T11:20:45.968 INFO:teuthology.orchestra.run.smithi092.stdout:[global] 2024-09-13T11:20:45.968 INFO:teuthology.orchestra.run.smithi092.stdout: chdir = "" 2024-09-13T11:20:45.968 INFO:teuthology.orchestra.run.smithi092.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-13T11:20:45.968 INFO:teuthology.orchestra.run.smithi092.stdout: auth supported = cephx 2024-09-13T11:20:45.968 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-09-13T11:20:45.968 INFO:teuthology.orchestra.run.smithi092.stdout: filestore xattr use omap = true 2024-09-13T11:20:45.969 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-09-13T11:20:45.969 INFO:teuthology.orchestra.run.smithi092.stdout: mon clock drift allowed = 1.000 2024-09-13T11:20:45.969 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-09-13T11:20:45.969 INFO:teuthology.orchestra.run.smithi092.stdout: osd crush chooseleaf type = 0 2024-09-13T11:20:45.969 INFO:teuthology.orchestra.run.smithi092.stdout: auth debug = true 2024-09-13T11:20:45.969 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-09-13T11:20:45.969 INFO:teuthology.orchestra.run.smithi092.stdout: ms die on old message = true 2024-09-13T11:20:45.969 INFO:teuthology.orchestra.run.smithi092.stdout: ms die on bug = true 2024-09-13T11:20:45.969 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-09-13T11:20:45.969 INFO:teuthology.orchestra.run.smithi092.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-13T11:20:45.969 INFO:teuthology.orchestra.run.smithi092.stdout: mon pg warn max object skew = 0 2024-09-13T11:20:45.969 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-09-13T11:20:45.970 INFO:teuthology.orchestra.run.smithi092.stdout: # disable pg_autoscaler by default for new pools 2024-09-13T11:20:45.970 INFO:teuthology.orchestra.run.smithi092.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-13T11:20:45.970 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-09-13T11:20:45.970 INFO:teuthology.orchestra.run.smithi092.stdout: osd pool default size = 2 2024-09-13T11:20:45.970 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-09-13T11:20:45.970 INFO:teuthology.orchestra.run.smithi092.stdout: mon osd allow primary affinity = true 2024-09-13T11:20:45.970 INFO:teuthology.orchestra.run.smithi092.stdout: mon osd allow pg remap = true 2024-09-13T11:20:45.970 INFO:teuthology.orchestra.run.smithi092.stdout: mon warn on legacy crush tunables = false 2024-09-13T11:20:45.970 INFO:teuthology.orchestra.run.smithi092.stdout: mon warn on crush straw calc version zero = false 2024-09-13T11:20:45.970 INFO:teuthology.orchestra.run.smithi092.stdout: mon warn on no sortbitwise = false 2024-09-13T11:20:45.970 INFO:teuthology.orchestra.run.smithi092.stdout: mon warn on osd down out interval zero = false 2024-09-13T11:20:45.970 INFO:teuthology.orchestra.run.smithi092.stdout: mon warn on too few osds = false 2024-09-13T11:20:45.970 INFO:teuthology.orchestra.run.smithi092.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-13T11:20:45.970 INFO:teuthology.orchestra.run.smithi092.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-13T11:20:45.971 INFO:teuthology.orchestra.run.smithi092.stdout: mon_allow_pool_size_one = true 2024-09-13T11:20:45.971 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-09-13T11:20:45.971 INFO:teuthology.orchestra.run.smithi092.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-13T11:20:45.971 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-09-13T11:20:45.971 INFO:teuthology.orchestra.run.smithi092.stdout: osd default data pool replay window = 5 2024-09-13T11:20:45.971 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-09-13T11:20:45.971 INFO:teuthology.orchestra.run.smithi092.stdout: mon allow pool delete = true 2024-09-13T11:20:45.971 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-09-13T11:20:45.971 INFO:teuthology.orchestra.run.smithi092.stdout: mon cluster log file level = debug 2024-09-13T11:20:45.971 INFO:teuthology.orchestra.run.smithi092.stdout: debug asserts on shutdown = true 2024-09-13T11:20:45.971 INFO:teuthology.orchestra.run.smithi092.stdout: mon health detail to clog = false 2024-09-13T11:20:45.971 INFO:teuthology.orchestra.run.smithi092.stdout: mon host = "[v1:172.21.15.1:6789],[v1:172.21.15.92:6789],[v1:172.21.15.162:6789]" 2024-09-13T11:20:45.971 INFO:teuthology.orchestra.run.smithi092.stdout: mon client directed command retry = 5 2024-09-13T11:20:45.971 INFO:teuthology.orchestra.run.smithi092.stdout: mon election default strategy = 1 2024-09-13T11:20:45.972 INFO:teuthology.orchestra.run.smithi092.stdout: ms bind msgr2 = False 2024-09-13T11:20:45.972 INFO:teuthology.orchestra.run.smithi092.stdout: ms inject delay max = 1 2024-09-13T11:20:45.972 INFO:teuthology.orchestra.run.smithi092.stdout: ms inject delay probability = 0.005 2024-09-13T11:20:45.972 INFO:teuthology.orchestra.run.smithi092.stdout: ms inject delay type = osd 2024-09-13T11:20:45.972 INFO:teuthology.orchestra.run.smithi092.stdout: ms inject internal delays = 0.002 2024-09-13T11:20:45.972 INFO:teuthology.orchestra.run.smithi092.stdout: ms inject socket failures = 2500 2024-09-13T11:20:45.972 INFO:teuthology.orchestra.run.smithi092.stdout: ms type = async 2024-09-13T11:20:45.972 INFO:teuthology.orchestra.run.smithi092.stdout: osd_pool_default_min_size = 2 2024-09-13T11:20:45.972 INFO:teuthology.orchestra.run.smithi092.stdout: osd_pool_default_size = 2 2024-09-13T11:20:45.972 INFO:teuthology.orchestra.run.smithi092.stdout: fsid = 2fd68ac7-5581-4d90-80a4-30bcf5252a7a 2024-09-13T11:20:45.972 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-09-13T11:20:45.972 INFO:teuthology.orchestra.run.smithi092.stdout:[osd] 2024-09-13T11:20:45.972 INFO:teuthology.orchestra.run.smithi092.stdout: osd journal size = 100 2024-09-13T11:20:45.973 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-09-13T11:20:45.973 INFO:teuthology.orchestra.run.smithi092.stdout: osd scrub load threshold = 5.0 2024-09-13T11:20:45.973 INFO:teuthology.orchestra.run.smithi092.stdout: osd scrub max interval = 120 2024-09-13T11:20:45.973 INFO:teuthology.orchestra.run.smithi092.stdout: osd mclock profile = high_recovery_ops 2024-09-13T11:20:45.973 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-09-13T11:20:45.973 INFO:teuthology.orchestra.run.smithi092.stdout: osd recover clone overlap = true 2024-09-13T11:20:45.973 INFO:teuthology.orchestra.run.smithi092.stdout: osd recovery max chunk = 1048576 2024-09-13T11:20:45.973 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-09-13T11:20:45.973 INFO:teuthology.orchestra.run.smithi092.stdout: osd debug shutdown = true 2024-09-13T11:20:45.973 INFO:teuthology.orchestra.run.smithi092.stdout: osd debug op order = true 2024-09-13T11:20:45.973 INFO:teuthology.orchestra.run.smithi092.stdout: osd debug verify stray on activate = true 2024-09-13T11:20:45.973 INFO:teuthology.orchestra.run.smithi092.stdout: osd debug trim objects = true 2024-09-13T11:20:45.973 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-09-13T11:20:45.973 INFO:teuthology.orchestra.run.smithi092.stdout: osd open classes on start = true 2024-09-13T11:20:45.974 INFO:teuthology.orchestra.run.smithi092.stdout: osd debug pg log writeout = true 2024-09-13T11:20:45.974 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-09-13T11:20:45.974 INFO:teuthology.orchestra.run.smithi092.stdout: osd deep scrub update digest min age = 30 2024-09-13T11:20:45.975 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-09-13T11:20:45.975 INFO:teuthology.orchestra.run.smithi092.stdout: osd map max advance = 10 2024-09-13T11:20:45.975 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-09-13T11:20:45.975 INFO:teuthology.orchestra.run.smithi092.stdout: journal zero on create = true 2024-09-13T11:20:45.975 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-09-13T11:20:45.975 INFO:teuthology.orchestra.run.smithi092.stdout: filestore ondisk finisher threads = 3 2024-09-13T11:20:45.975 INFO:teuthology.orchestra.run.smithi092.stdout: filestore apply finisher threads = 3 2024-09-13T11:20:45.975 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-09-13T11:20:45.975 INFO:teuthology.orchestra.run.smithi092.stdout: bdev debug aio = true 2024-09-13T11:20:45.975 INFO:teuthology.orchestra.run.smithi092.stdout: osd debug misdirected ops = true 2024-09-13T11:20:45.976 INFO:teuthology.orchestra.run.smithi092.stdout: debug monc = 20 2024-09-13T11:20:45.976 INFO:teuthology.orchestra.run.smithi092.stdout: bdev async discard = True 2024-09-13T11:20:45.976 INFO:teuthology.orchestra.run.smithi092.stdout: bdev enable discard = True 2024-09-13T11:20:45.976 INFO:teuthology.orchestra.run.smithi092.stdout: bluestore allocator = stupid 2024-09-13T11:20:45.976 INFO:teuthology.orchestra.run.smithi092.stdout: bluestore block size = 96636764160 2024-09-13T11:20:45.976 INFO:teuthology.orchestra.run.smithi092.stdout: bluestore fsck on mount = True 2024-09-13T11:20:45.976 INFO:teuthology.orchestra.run.smithi092.stdout: bluestore write v2 = True 2024-09-13T11:20:45.976 INFO:teuthology.orchestra.run.smithi092.stdout: bluestore zero block detection = True 2024-09-13T11:20:45.976 INFO:teuthology.orchestra.run.smithi092.stdout: debug bluefs = 20 2024-09-13T11:20:45.976 INFO:teuthology.orchestra.run.smithi092.stdout: debug bluestore = 20 2024-09-13T11:20:45.976 INFO:teuthology.orchestra.run.smithi092.stdout: debug ms = 1 2024-09-13T11:20:45.976 INFO:teuthology.orchestra.run.smithi092.stdout: debug osd = 20 2024-09-13T11:20:45.976 INFO:teuthology.orchestra.run.smithi092.stdout: debug rocksdb = 10 2024-09-13T11:20:45.977 INFO:teuthology.orchestra.run.smithi092.stdout: mon osd backfillfull_ratio = 0.85 2024-09-13T11:20:45.977 INFO:teuthology.orchestra.run.smithi092.stdout: mon osd full ratio = 0.9 2024-09-13T11:20:45.977 INFO:teuthology.orchestra.run.smithi092.stdout: mon osd nearfull ratio = 0.8 2024-09-13T11:20:45.977 INFO:teuthology.orchestra.run.smithi092.stdout: osd backoff on peering = True 2024-09-13T11:20:45.977 INFO:teuthology.orchestra.run.smithi092.stdout: osd blocked scrub grace period = 3600 2024-09-13T11:20:45.977 INFO:teuthology.orchestra.run.smithi092.stdout: osd debug reject backfill probability = 0.3 2024-09-13T11:20:45.977 INFO:teuthology.orchestra.run.smithi092.stdout: osd debug verify cached snaps = True 2024-09-13T11:20:45.977 INFO:teuthology.orchestra.run.smithi092.stdout: osd debug verify missing on start = True 2024-09-13T11:20:45.977 INFO:teuthology.orchestra.run.smithi092.stdout: osd delete sleep = 1 2024-09-13T11:20:45.977 INFO:teuthology.orchestra.run.smithi092.stdout: osd failsafe full ratio = 0.95 2024-09-13T11:20:45.977 INFO:teuthology.orchestra.run.smithi092.stdout: osd inject bad map crc probability = 0.1 2024-09-13T11:20:45.977 INFO:teuthology.orchestra.run.smithi092.stdout: osd max backfills = 3 2024-09-13T11:20:45.977 INFO:teuthology.orchestra.run.smithi092.stdout: osd max markdown count = 1000 2024-09-13T11:20:45.978 INFO:teuthology.orchestra.run.smithi092.stdout: osd max scrubs = 5 2024-09-13T11:20:45.978 INFO:teuthology.orchestra.run.smithi092.stdout: osd mclock override recovery settings = True 2024-09-13T11:20:45.978 INFO:teuthology.orchestra.run.smithi092.stdout: osd objectstore = bluestore 2024-09-13T11:20:45.978 INFO:teuthology.orchestra.run.smithi092.stdout: osd op queue = debug_random 2024-09-13T11:20:45.978 INFO:teuthology.orchestra.run.smithi092.stdout: osd op queue cut off = debug_random 2024-09-13T11:20:45.978 INFO:teuthology.orchestra.run.smithi092.stdout: osd scrub min interval = 60 2024-09-13T11:20:45.978 INFO:teuthology.orchestra.run.smithi092.stdout: osd shutdown pgref assert = True 2024-09-13T11:20:45.978 INFO:teuthology.orchestra.run.smithi092.stdout: osd snap trim sleep = 2 2024-09-13T11:20:45.978 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-09-13T11:20:45.978 INFO:teuthology.orchestra.run.smithi092.stdout:[mgr] 2024-09-13T11:20:45.978 INFO:teuthology.orchestra.run.smithi092.stdout: debug ms = 1 2024-09-13T11:20:45.978 INFO:teuthology.orchestra.run.smithi092.stdout: debug mgr = 20 2024-09-13T11:20:45.978 INFO:teuthology.orchestra.run.smithi092.stdout: debug mon = 20 2024-09-13T11:20:45.979 INFO:teuthology.orchestra.run.smithi092.stdout: debug auth = 20 2024-09-13T11:20:45.979 INFO:teuthology.orchestra.run.smithi092.stdout: mon reweight min pgs per osd = 4 2024-09-13T11:20:45.979 INFO:teuthology.orchestra.run.smithi092.stdout: mon reweight min bytes per osd = 10 2024-09-13T11:20:45.979 INFO:teuthology.orchestra.run.smithi092.stdout: mgr/telemetry/nag = false 2024-09-13T11:20:45.979 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-09-13T11:20:45.979 INFO:teuthology.orchestra.run.smithi092.stdout:[mon] 2024-09-13T11:20:45.979 INFO:teuthology.orchestra.run.smithi092.stdout: debug ms = 1 2024-09-13T11:20:45.979 INFO:teuthology.orchestra.run.smithi092.stdout: debug mon = 20 2024-09-13T11:20:45.979 INFO:teuthology.orchestra.run.smithi092.stdout: debug paxos = 20 2024-09-13T11:20:45.979 INFO:teuthology.orchestra.run.smithi092.stdout: debug auth = 20 2024-09-13T11:20:45.979 INFO:teuthology.orchestra.run.smithi092.stdout: mon data avail warn = 5 2024-09-13T11:20:45.979 INFO:teuthology.orchestra.run.smithi092.stdout: mon mgr mkfs grace = 240 2024-09-13T11:20:45.979 INFO:teuthology.orchestra.run.smithi092.stdout: mon reweight min pgs per osd = 4 2024-09-13T11:20:45.979 INFO:teuthology.orchestra.run.smithi092.stdout: mon osd reporter subtree level = osd 2024-09-13T11:20:45.980 INFO:teuthology.orchestra.run.smithi092.stdout: mon osd prime pg temp = true 2024-09-13T11:20:45.980 INFO:teuthology.orchestra.run.smithi092.stdout: mon reweight min bytes per osd = 10 2024-09-13T11:20:45.980 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-09-13T11:20:45.980 INFO:teuthology.orchestra.run.smithi092.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-13T11:20:45.980 INFO:teuthology.orchestra.run.smithi092.stdout: auth mon ticket ttl = 660 # 11m 2024-09-13T11:20:45.980 INFO:teuthology.orchestra.run.smithi092.stdout: auth service ticket ttl = 240 # 4m 2024-09-13T11:20:45.980 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-09-13T11:20:45.980 INFO:teuthology.orchestra.run.smithi092.stdout: # don't complain about insecure global_id in the test suite 2024-09-13T11:20:45.980 INFO:teuthology.orchestra.run.smithi092.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-13T11:20:45.980 INFO:teuthology.orchestra.run.smithi092.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-13T11:20:45.980 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-09-13T11:20:45.980 INFO:teuthology.orchestra.run.smithi092.stdout: # 1m isn't quite enough 2024-09-13T11:20:45.980 INFO:teuthology.orchestra.run.smithi092.stdout: mon_down_mkfs_grace = 2m 2024-09-13T11:20:45.981 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-09-13T11:20:45.981 INFO:teuthology.orchestra.run.smithi092.stdout: mon_warn_on_filestore_osds = false 2024-09-13T11:20:45.981 INFO:teuthology.orchestra.run.smithi092.stdout: mon min osdmap epochs = 50 2024-09-13T11:20:45.981 INFO:teuthology.orchestra.run.smithi092.stdout: mon osdmap full prune interval = 2 2024-09-13T11:20:45.981 INFO:teuthology.orchestra.run.smithi092.stdout: mon osdmap full prune min = 15 2024-09-13T11:20:45.981 INFO:teuthology.orchestra.run.smithi092.stdout: mon osdmap full prune txsize = 2 2024-09-13T11:20:45.981 INFO:teuthology.orchestra.run.smithi092.stdout: mon scrub interval = 300 2024-09-13T11:20:45.981 INFO:teuthology.orchestra.run.smithi092.stdout: paxos service trim min = 10 2024-09-13T11:20:45.981 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-09-13T11:20:45.981 INFO:teuthology.orchestra.run.smithi092.stdout:[client] 2024-09-13T11:20:45.981 INFO:teuthology.orchestra.run.smithi092.stdout: rgw cache enabled = true 2024-09-13T11:20:45.981 INFO:teuthology.orchestra.run.smithi092.stdout: rgw enable ops log = true 2024-09-13T11:20:45.981 INFO:teuthology.orchestra.run.smithi092.stdout: rgw enable usage log = true 2024-09-13T11:20:45.982 INFO:teuthology.orchestra.run.smithi092.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-13T11:20:45.982 INFO:teuthology.orchestra.run.smithi092.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-13T11:20:45.982 INFO:teuthology.orchestra.run.smithi092.stdout:[mon.a] 2024-09-13T11:20:45.982 INFO:teuthology.orchestra.run.smithi092.stdout:[mon.b] 2024-09-13T11:20:45.982 INFO:teuthology.orchestra.run.smithi092.stdout:[mon.c] 2024-09-13T11:20:45.982 INFO:teuthology.orchestra.run.smithi092.stdout:[client.0] 2024-09-13T11:20:45.982 INFO:teuthology.orchestra.run.smithi092.stdout: admin socket = /var/run/ceph/ceph-$name.asok 2024-09-13T11:20:45.982 INFO:tasks.ceph:Creating admin key on mon.a... 2024-09-13T11:20:45.983 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-09-13T11:20:46.033 INFO:tasks.ceph:Copying monmap to all nodes... 2024-09-13T11:20:46.034 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-09-13T11:20:46.034 DEBUG:teuthology.orchestra.run.smithi001:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-09-13T11:20:46.040 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-09-13T11:20:46.040 DEBUG:teuthology.orchestra.run.smithi001:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-09-13T11:20:46.089 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi001.front.sepia.ceph.com 2024-09-13T11:20:46.089 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-09-13T11:20:46.089 DEBUG:teuthology.orchestra.run.smithi001:> sudo dd of=/etc/ceph/ceph.keyring 2024-09-13T11:20:46.089 DEBUG:teuthology.orchestra.run.smithi001:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-13T11:20:46.154 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-09-13T11:20:46.154 DEBUG:teuthology.orchestra.run.smithi001:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-09-13T11:20:46.205 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi092.front.sepia.ceph.com 2024-09-13T11:20:46.205 DEBUG:teuthology.orchestra.run.smithi092:> set -ex 2024-09-13T11:20:46.205 DEBUG:teuthology.orchestra.run.smithi092:> sudo dd of=/etc/ceph/ceph.keyring 2024-09-13T11:20:46.205 DEBUG:teuthology.orchestra.run.smithi092:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-13T11:20:46.228 DEBUG:teuthology.orchestra.run.smithi092:> set -ex 2024-09-13T11:20:46.229 DEBUG:teuthology.orchestra.run.smithi092:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-09-13T11:20:46.276 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi162.front.sepia.ceph.com 2024-09-13T11:20:46.276 DEBUG:teuthology.orchestra.run.smithi162:> set -ex 2024-09-13T11:20:46.276 DEBUG:teuthology.orchestra.run.smithi162:> sudo dd of=/etc/ceph/ceph.keyring 2024-09-13T11:20:46.276 DEBUG:teuthology.orchestra.run.smithi162:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-13T11:20:46.300 DEBUG:teuthology.orchestra.run.smithi162:> set -ex 2024-09-13T11:20:46.301 DEBUG:teuthology.orchestra.run.smithi162:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-09-13T11:20:46.349 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi177.front.sepia.ceph.com 2024-09-13T11:20:46.349 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-09-13T11:20:46.349 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd of=/etc/ceph/ceph.keyring 2024-09-13T11:20:46.350 DEBUG:teuthology.orchestra.run.smithi177:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-13T11:20:46.374 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-09-13T11:20:46.374 DEBUG:teuthology.orchestra.run.smithi177:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-09-13T11:20:46.422 INFO:tasks.ceph:Setting up mon nodes... 2024-09-13T11:20:46.423 INFO:tasks.ceph:Setting up mgr nodes... 2024-09-13T11:20:46.423 DEBUG:teuthology.orchestra.run.smithi001:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2024-09-13T11:20:46.467 INFO:teuthology.orchestra.run.smithi001.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2024-09-13T11:20:46.470 DEBUG:teuthology.orchestra.run.smithi177:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-09-13T11:20:46.514 INFO:teuthology.orchestra.run.smithi177.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-09-13T11:20:46.527 INFO:tasks.ceph:Setting up mds nodes... 2024-09-13T11:20:46.527 INFO:tasks.ceph_client:Setting up client nodes... 2024-09-13T11:20:46.528 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-09-13T11:20:46.603 INFO:teuthology.orchestra.run.smithi177.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-09-13T11:20:46.613 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-09-13T11:20:46.613 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi001.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.8': '/dev/vg_nvme/lv_3', 'osd.12': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi092.front.sepia.ceph.com'): {'osd.1': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.9': '/dev/vg_nvme/lv_3', 'osd.13': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi162.front.sepia.ceph.com'): {'osd.2': '/dev/vg_nvme/lv_1', 'osd.6': '/dev/vg_nvme/lv_2', 'osd.10': '/dev/vg_nvme/lv_3', 'osd.14': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi177.front.sepia.ceph.com'): {'osd.3': '/dev/vg_nvme/lv_1', 'osd.7': '/dev/vg_nvme/lv_2', 'osd.11': '/dev/vg_nvme/lv_3', 'osd.15': '/dev/vg_nvme/lv_4'}} 2024-09-13T11:20:46.613 DEBUG:teuthology.orchestra.run.smithi001:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-09-13T11:20:46.628 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.8': '/dev/vg_nvme/lv_3', 'osd.12': '/dev/vg_nvme/lv_4'} 2024-09-13T11:20:46.628 INFO:tasks.ceph:role: osd.0 2024-09-13T11:20:46.628 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi001.front.sepia.ceph.com 2024-09-13T11:20:46.628 DEBUG:teuthology.orchestra.run.smithi001:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-09-13T11:20:46.685 INFO:teuthology.orchestra.run.smithi001.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-09-13T11:20:46.686 INFO:teuthology.orchestra.run.smithi001.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-13T11:20:46.686 INFO:teuthology.orchestra.run.smithi001.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-13T11:20:46.686 INFO:teuthology.orchestra.run.smithi001.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-09-13T11:20:46.686 INFO:teuthology.orchestra.run.smithi001.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-13T11:20:46.686 INFO:teuthology.orchestra.run.smithi001.stdout: = sunit=0 swidth=0 blks 2024-09-13T11:20:46.686 INFO:teuthology.orchestra.run.smithi001.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-13T11:20:46.686 INFO:teuthology.orchestra.run.smithi001.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-13T11:20:46.686 INFO:teuthology.orchestra.run.smithi001.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-13T11:20:46.686 INFO:teuthology.orchestra.run.smithi001.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-13T11:20:46.709 INFO:teuthology.orchestra.run.smithi001.stdout:Discarding blocks...Done. 2024-09-13T11:20:46.711 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi001.front.sepia.ceph.com -o noatime 2024-09-13T11:20:46.712 DEBUG:teuthology.orchestra.run.smithi001:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-09-13T11:20:46.769 DEBUG:teuthology.orchestra.run.smithi001:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-09-13T11:20:46.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T11:20:46.822 INFO:teuthology.orchestra.run.smithi001.stderr:sudo: /sbin/restorecon: command not found 2024-09-13T11:20:46.822 DEBUG:teuthology.orchestra.run.smithi001:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-09-13T11:20:46.875 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.8': '/dev/vg_nvme/lv_3', 'osd.12': '/dev/vg_nvme/lv_4'} 2024-09-13T11:20:46.876 INFO:tasks.ceph:role: osd.4 2024-09-13T11:20:46.876 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi001.front.sepia.ceph.com 2024-09-13T11:20:46.876 DEBUG:teuthology.orchestra.run.smithi001:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-09-13T11:20:46.930 INFO:teuthology.orchestra.run.smithi001.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-09-13T11:20:46.930 INFO:teuthology.orchestra.run.smithi001.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-13T11:20:46.930 INFO:teuthology.orchestra.run.smithi001.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-13T11:20:46.930 INFO:teuthology.orchestra.run.smithi001.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-09-13T11:20:46.930 INFO:teuthology.orchestra.run.smithi001.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-13T11:20:46.930 INFO:teuthology.orchestra.run.smithi001.stdout: = sunit=0 swidth=0 blks 2024-09-13T11:20:46.930 INFO:teuthology.orchestra.run.smithi001.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-13T11:20:46.931 INFO:teuthology.orchestra.run.smithi001.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-13T11:20:46.931 INFO:teuthology.orchestra.run.smithi001.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-13T11:20:46.931 INFO:teuthology.orchestra.run.smithi001.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-13T11:20:46.954 INFO:teuthology.orchestra.run.smithi001.stdout:Discarding blocks...Done. 2024-09-13T11:20:46.956 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi001.front.sepia.ceph.com -o noatime 2024-09-13T11:20:46.956 DEBUG:teuthology.orchestra.run.smithi001:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-4 2024-09-13T11:20:47.015 DEBUG:teuthology.orchestra.run.smithi001:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-09-13T11:20:47.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T11:20:47.069 INFO:teuthology.orchestra.run.smithi001.stderr:sudo: /sbin/restorecon: command not found 2024-09-13T11:20:47.070 DEBUG:teuthology.orchestra.run.smithi001:> sudo mkdir -p /var/lib/ceph/osd/ceph-8 2024-09-13T11:20:47.127 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.8': '/dev/vg_nvme/lv_3', 'osd.12': '/dev/vg_nvme/lv_4'} 2024-09-13T11:20:47.127 INFO:tasks.ceph:role: osd.8 2024-09-13T11:20:47.127 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi001.front.sepia.ceph.com 2024-09-13T11:20:47.127 DEBUG:teuthology.orchestra.run.smithi001:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-09-13T11:20:47.185 INFO:teuthology.orchestra.run.smithi001.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-09-13T11:20:47.185 INFO:teuthology.orchestra.run.smithi001.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-13T11:20:47.185 INFO:teuthology.orchestra.run.smithi001.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-13T11:20:47.185 INFO:teuthology.orchestra.run.smithi001.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-09-13T11:20:47.185 INFO:teuthology.orchestra.run.smithi001.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-13T11:20:47.185 INFO:teuthology.orchestra.run.smithi001.stdout: = sunit=0 swidth=0 blks 2024-09-13T11:20:47.185 INFO:teuthology.orchestra.run.smithi001.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-13T11:20:47.185 INFO:teuthology.orchestra.run.smithi001.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-13T11:20:47.185 INFO:teuthology.orchestra.run.smithi001.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-13T11:20:47.185 INFO:teuthology.orchestra.run.smithi001.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-13T11:20:47.209 INFO:teuthology.orchestra.run.smithi001.stdout:Discarding blocks...Done. 2024-09-13T11:20:47.211 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi001.front.sepia.ceph.com -o noatime 2024-09-13T11:20:47.211 DEBUG:teuthology.orchestra.run.smithi001:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-8 2024-09-13T11:20:47.270 DEBUG:teuthology.orchestra.run.smithi001:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-8 2024-09-13T11:20:47.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T11:20:47.325 INFO:teuthology.orchestra.run.smithi001.stderr:sudo: /sbin/restorecon: command not found 2024-09-13T11:20:47.326 DEBUG:teuthology.orchestra.run.smithi001:> sudo mkdir -p /var/lib/ceph/osd/ceph-12 2024-09-13T11:20:47.379 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.8': '/dev/vg_nvme/lv_3', 'osd.12': '/dev/vg_nvme/lv_4'} 2024-09-13T11:20:47.380 INFO:tasks.ceph:role: osd.12 2024-09-13T11:20:47.380 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi001.front.sepia.ceph.com 2024-09-13T11:20:47.380 DEBUG:teuthology.orchestra.run.smithi001:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-09-13T11:20:47.436 INFO:teuthology.orchestra.run.smithi001.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-09-13T11:20:47.436 INFO:teuthology.orchestra.run.smithi001.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-13T11:20:47.436 INFO:teuthology.orchestra.run.smithi001.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-13T11:20:47.436 INFO:teuthology.orchestra.run.smithi001.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-09-13T11:20:47.436 INFO:teuthology.orchestra.run.smithi001.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-13T11:20:47.437 INFO:teuthology.orchestra.run.smithi001.stdout: = sunit=0 swidth=0 blks 2024-09-13T11:20:47.437 INFO:teuthology.orchestra.run.smithi001.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-13T11:20:47.437 INFO:teuthology.orchestra.run.smithi001.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-13T11:20:47.437 INFO:teuthology.orchestra.run.smithi001.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-13T11:20:47.437 INFO:teuthology.orchestra.run.smithi001.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-13T11:20:47.460 INFO:teuthology.orchestra.run.smithi001.stdout:Discarding blocks...Done. 2024-09-13T11:20:47.462 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi001.front.sepia.ceph.com -o noatime 2024-09-13T11:20:47.462 DEBUG:teuthology.orchestra.run.smithi001:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-12 2024-09-13T11:20:47.521 DEBUG:teuthology.orchestra.run.smithi001:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-12 2024-09-13T11:20:47.573 INFO:teuthology.orchestra.run.smithi001.stderr:sudo: /sbin/restorecon: command not found 2024-09-13T11:20:47.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T11:20:47.574 DEBUG:teuthology.orchestra.run.smithi001:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-13T11:20:47.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-09-13T11:20:47.682+0000 7fdd9f74f7c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-09-13T11:20:47.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-09-13T11:20:47.682+0000 7fdd9f74f7c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-09-13T11:20:47.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-09-13T11:20:47.682+0000 7fdd9f74f7c0 -1 bdev(0x559bdc1cb800 /var/lib/ceph/osd/ceph-0/block) open stat got: (1) Operation not permitted 2024-09-13T11:20:47.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-09-13T11:20:47.682+0000 7fdd9f74f7c0 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-09-13T11:20:50.869 DEBUG:teuthology.orchestra.run.smithi001:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-09-13T11:20:50.884 DEBUG:teuthology.orchestra.run.smithi001:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-13T11:20:50.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-09-13T11:20:50.970+0000 7f31f53627c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-09-13T11:20:50.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-09-13T11:20:50.970+0000 7f31f53627c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-09-13T11:20:50.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-09-13T11:20:50.970+0000 7f31f53627c0 -1 bdev(0x55565ee51800 /var/lib/ceph/osd/ceph-4/block) open stat got: (1) Operation not permitted 2024-09-13T11:20:50.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-09-13T11:20:50.970+0000 7f31f53627c0 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-09-13T11:20:54.095 DEBUG:teuthology.orchestra.run.smithi001:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-09-13T11:20:54.150 DEBUG:teuthology.orchestra.run.smithi001:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 8 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-13T11:20:54.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-09-13T11:20:54.230+0000 7fa9537657c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-8/keyring: can't open /var/lib/ceph/osd/ceph-8/keyring: (2) No such file or directory 2024-09-13T11:20:54.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-09-13T11:20:54.230+0000 7fa9537657c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-8/keyring 2024-09-13T11:20:54.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-09-13T11:20:54.230+0000 7fa9537657c0 -1 bdev(0x55a91aa4f800 /var/lib/ceph/osd/ceph-8/block) open stat got: (1) Operation not permitted 2024-09-13T11:20:54.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-09-13T11:20:54.230+0000 7fa9537657c0 -1 bluestore(/var/lib/ceph/osd/ceph-8) _read_fsid unparsable uuid 2024-09-13T11:20:57.392 DEBUG:teuthology.orchestra.run.smithi001:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-8 2024-09-13T11:20:57.450 DEBUG:teuthology.orchestra.run.smithi001:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 12 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-13T11:20:57.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-09-13T11:20:57.530+0000 7fde9d1f87c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-12/keyring: can't open /var/lib/ceph/osd/ceph-12/keyring: (2) No such file or directory 2024-09-13T11:20:57.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-09-13T11:20:57.530+0000 7fde9d1f87c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-12/keyring 2024-09-13T11:20:57.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-09-13T11:20:57.530+0000 7fde9d1f87c0 -1 bdev(0x55be4a1ad800 /var/lib/ceph/osd/ceph-12/block) open stat got: (1) Operation not permitted 2024-09-13T11:20:57.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-09-13T11:20:57.530+0000 7fde9d1f87c0 -1 bluestore(/var/lib/ceph/osd/ceph-12) _read_fsid unparsable uuid 2024-09-13T11:21:00.697 DEBUG:teuthology.orchestra.run.smithi001:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-12 2024-09-13T11:21:00.758 DEBUG:teuthology.orchestra.run.smithi092:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-09-13T11:21:00.772 INFO:tasks.ceph:roles_to_devs: {'osd.1': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.9': '/dev/vg_nvme/lv_3', 'osd.13': '/dev/vg_nvme/lv_4'} 2024-09-13T11:21:00.772 INFO:tasks.ceph:role: osd.1 2024-09-13T11:21:00.772 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi092.front.sepia.ceph.com 2024-09-13T11:21:00.772 DEBUG:teuthology.orchestra.run.smithi092:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-09-13T11:21:00.836 INFO:teuthology.orchestra.run.smithi092.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-09-13T11:21:00.836 INFO:teuthology.orchestra.run.smithi092.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-13T11:21:00.836 INFO:teuthology.orchestra.run.smithi092.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-13T11:21:00.836 INFO:teuthology.orchestra.run.smithi092.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-09-13T11:21:00.836 INFO:teuthology.orchestra.run.smithi092.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-13T11:21:00.837 INFO:teuthology.orchestra.run.smithi092.stdout: = sunit=0 swidth=0 blks 2024-09-13T11:21:00.837 INFO:teuthology.orchestra.run.smithi092.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-13T11:21:00.837 INFO:teuthology.orchestra.run.smithi092.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-13T11:21:00.837 INFO:teuthology.orchestra.run.smithi092.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-13T11:21:00.837 INFO:teuthology.orchestra.run.smithi092.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-13T11:21:01.318 INFO:teuthology.orchestra.run.smithi092.stdout:Discarding blocks...Done. 2024-09-13T11:21:01.319 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi092.front.sepia.ceph.com -o noatime 2024-09-13T11:21:01.320 DEBUG:teuthology.orchestra.run.smithi092:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-1 2024-09-13T11:21:01.475 DEBUG:teuthology.orchestra.run.smithi092:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-09-13T11:21:01.486 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: /sbin/restorecon: command not found 2024-09-13T11:21:01.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T11:21:01.487 DEBUG:teuthology.orchestra.run.smithi092:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-09-13T11:21:01.542 INFO:tasks.ceph:roles_to_devs: {'osd.1': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.9': '/dev/vg_nvme/lv_3', 'osd.13': '/dev/vg_nvme/lv_4'} 2024-09-13T11:21:01.543 INFO:tasks.ceph:role: osd.5 2024-09-13T11:21:01.543 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi092.front.sepia.ceph.com 2024-09-13T11:21:01.543 DEBUG:teuthology.orchestra.run.smithi092:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-09-13T11:21:01.607 INFO:teuthology.orchestra.run.smithi092.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-09-13T11:21:01.607 INFO:teuthology.orchestra.run.smithi092.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-13T11:21:01.607 INFO:teuthology.orchestra.run.smithi092.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-13T11:21:01.607 INFO:teuthology.orchestra.run.smithi092.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-09-13T11:21:01.607 INFO:teuthology.orchestra.run.smithi092.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-13T11:21:01.607 INFO:teuthology.orchestra.run.smithi092.stdout: = sunit=0 swidth=0 blks 2024-09-13T11:21:01.607 INFO:teuthology.orchestra.run.smithi092.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-13T11:21:01.607 INFO:teuthology.orchestra.run.smithi092.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-13T11:21:01.608 INFO:teuthology.orchestra.run.smithi092.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-13T11:21:01.608 INFO:teuthology.orchestra.run.smithi092.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-13T11:21:02.053 INFO:teuthology.orchestra.run.smithi092.stdout:Discarding blocks...Done. 2024-09-13T11:21:02.055 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi092.front.sepia.ceph.com -o noatime 2024-09-13T11:21:02.055 DEBUG:teuthology.orchestra.run.smithi092:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2024-09-13T11:21:02.135 DEBUG:teuthology.orchestra.run.smithi092:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-09-13T11:21:02.187 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: /sbin/restorecon: command not found 2024-09-13T11:21:02.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T11:21:02.188 DEBUG:teuthology.orchestra.run.smithi092:> sudo mkdir -p /var/lib/ceph/osd/ceph-9 2024-09-13T11:21:02.240 INFO:tasks.ceph:roles_to_devs: {'osd.1': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.9': '/dev/vg_nvme/lv_3', 'osd.13': '/dev/vg_nvme/lv_4'} 2024-09-13T11:21:02.241 INFO:tasks.ceph:role: osd.9 2024-09-13T11:21:02.241 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi092.front.sepia.ceph.com 2024-09-13T11:21:02.241 DEBUG:teuthology.orchestra.run.smithi092:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-09-13T11:21:02.301 INFO:teuthology.orchestra.run.smithi092.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-09-13T11:21:02.302 INFO:teuthology.orchestra.run.smithi092.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-13T11:21:02.302 INFO:teuthology.orchestra.run.smithi092.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-13T11:21:02.302 INFO:teuthology.orchestra.run.smithi092.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-09-13T11:21:02.302 INFO:teuthology.orchestra.run.smithi092.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-13T11:21:02.302 INFO:teuthology.orchestra.run.smithi092.stdout: = sunit=0 swidth=0 blks 2024-09-13T11:21:02.302 INFO:teuthology.orchestra.run.smithi092.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-13T11:21:02.302 INFO:teuthology.orchestra.run.smithi092.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-13T11:21:02.302 INFO:teuthology.orchestra.run.smithi092.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-13T11:21:02.302 INFO:teuthology.orchestra.run.smithi092.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-13T11:21:02.775 INFO:teuthology.orchestra.run.smithi092.stdout:Discarding blocks...Done. 2024-09-13T11:21:02.777 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi092.front.sepia.ceph.com -o noatime 2024-09-13T11:21:02.777 DEBUG:teuthology.orchestra.run.smithi092:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-9 2024-09-13T11:21:02.921 DEBUG:teuthology.orchestra.run.smithi092:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-9 2024-09-13T11:21:02.932 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: /sbin/restorecon: command not found 2024-09-13T11:21:02.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T11:21:02.934 DEBUG:teuthology.orchestra.run.smithi092:> sudo mkdir -p /var/lib/ceph/osd/ceph-13 2024-09-13T11:21:02.986 INFO:tasks.ceph:roles_to_devs: {'osd.1': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.9': '/dev/vg_nvme/lv_3', 'osd.13': '/dev/vg_nvme/lv_4'} 2024-09-13T11:21:02.986 INFO:tasks.ceph:role: osd.13 2024-09-13T11:21:02.986 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi092.front.sepia.ceph.com 2024-09-13T11:21:02.986 DEBUG:teuthology.orchestra.run.smithi092:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-09-13T11:21:03.050 INFO:teuthology.orchestra.run.smithi092.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-09-13T11:21:03.051 INFO:teuthology.orchestra.run.smithi092.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-13T11:21:03.051 INFO:teuthology.orchestra.run.smithi092.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-13T11:21:03.051 INFO:teuthology.orchestra.run.smithi092.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-09-13T11:21:03.051 INFO:teuthology.orchestra.run.smithi092.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-13T11:21:03.051 INFO:teuthology.orchestra.run.smithi092.stdout: = sunit=0 swidth=0 blks 2024-09-13T11:21:03.051 INFO:teuthology.orchestra.run.smithi092.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-13T11:21:03.051 INFO:teuthology.orchestra.run.smithi092.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-13T11:21:03.051 INFO:teuthology.orchestra.run.smithi092.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-13T11:21:03.051 INFO:teuthology.orchestra.run.smithi092.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-13T11:21:03.536 INFO:teuthology.orchestra.run.smithi092.stdout:Discarding blocks...Done. 2024-09-13T11:21:03.538 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi092.front.sepia.ceph.com -o noatime 2024-09-13T11:21:03.538 DEBUG:teuthology.orchestra.run.smithi092:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-13 2024-09-13T11:21:03.692 DEBUG:teuthology.orchestra.run.smithi092:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-13 2024-09-13T11:21:03.703 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: /sbin/restorecon: command not found 2024-09-13T11:21:03.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T11:21:03.705 DEBUG:teuthology.orchestra.run.smithi092:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-13T11:21:03.810 INFO:teuthology.orchestra.run.smithi092.stderr:2024-09-13T11:21:03.807+0000 7f7242a637c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-09-13T11:21:03.813 INFO:teuthology.orchestra.run.smithi092.stderr:2024-09-13T11:21:03.811+0000 7f7242a637c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-09-13T11:21:03.813 INFO:teuthology.orchestra.run.smithi092.stderr:2024-09-13T11:21:03.811+0000 7f7242a637c0 -1 bdev(0x55bd6b667800 /var/lib/ceph/osd/ceph-1/block) open stat got: (1) Operation not permitted 2024-09-13T11:21:03.813 INFO:teuthology.orchestra.run.smithi092.stderr:2024-09-13T11:21:03.811+0000 7f7242a637c0 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-09-13T11:21:06.985 DEBUG:teuthology.orchestra.run.smithi092:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-09-13T11:21:07.000 DEBUG:teuthology.orchestra.run.smithi092:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-13T11:21:07.081 INFO:teuthology.orchestra.run.smithi092.stderr:2024-09-13T11:21:07.079+0000 7f328679f7c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-09-13T11:21:07.082 INFO:teuthology.orchestra.run.smithi092.stderr:2024-09-13T11:21:07.079+0000 7f328679f7c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-09-13T11:21:07.082 INFO:teuthology.orchestra.run.smithi092.stderr:2024-09-13T11:21:07.079+0000 7f328679f7c0 -1 bdev(0x55d7b286d800 /var/lib/ceph/osd/ceph-5/block) open stat got: (1) Operation not permitted 2024-09-13T11:21:07.082 INFO:teuthology.orchestra.run.smithi092.stderr:2024-09-13T11:21:07.079+0000 7f328679f7c0 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-09-13T11:21:10.246 DEBUG:teuthology.orchestra.run.smithi092:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-09-13T11:21:10.303 DEBUG:teuthology.orchestra.run.smithi092:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 9 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-13T11:21:10.391 INFO:teuthology.orchestra.run.smithi092.stderr:2024-09-13T11:21:10.391+0000 7f6fcb0757c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-9/keyring: can't open /var/lib/ceph/osd/ceph-9/keyring: (2) No such file or directory 2024-09-13T11:21:10.391 INFO:teuthology.orchestra.run.smithi092.stderr:2024-09-13T11:21:10.391+0000 7f6fcb0757c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-9/keyring 2024-09-13T11:21:10.391 INFO:teuthology.orchestra.run.smithi092.stderr:2024-09-13T11:21:10.391+0000 7f6fcb0757c0 -1 bdev(0x55c6e07d3800 /var/lib/ceph/osd/ceph-9/block) open stat got: (1) Operation not permitted 2024-09-13T11:21:10.392 INFO:teuthology.orchestra.run.smithi092.stderr:2024-09-13T11:21:10.391+0000 7f6fcb0757c0 -1 bluestore(/var/lib/ceph/osd/ceph-9) _read_fsid unparsable uuid 2024-09-13T11:21:13.546 DEBUG:teuthology.orchestra.run.smithi092:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-9 2024-09-13T11:21:13.604 DEBUG:teuthology.orchestra.run.smithi092:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 13 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-13T11:21:13.686 INFO:teuthology.orchestra.run.smithi092.stderr:2024-09-13T11:21:13.683+0000 7f217607f7c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-13/keyring: can't open /var/lib/ceph/osd/ceph-13/keyring: (2) No such file or directory 2024-09-13T11:21:13.686 INFO:teuthology.orchestra.run.smithi092.stderr:2024-09-13T11:21:13.687+0000 7f217607f7c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-13/keyring 2024-09-13T11:21:13.686 INFO:teuthology.orchestra.run.smithi092.stderr:2024-09-13T11:21:13.687+0000 7f217607f7c0 -1 bdev(0x55a0cfc6b800 /var/lib/ceph/osd/ceph-13/block) open stat got: (1) Operation not permitted 2024-09-13T11:21:13.687 INFO:teuthology.orchestra.run.smithi092.stderr:2024-09-13T11:21:13.687+0000 7f217607f7c0 -1 bluestore(/var/lib/ceph/osd/ceph-13) _read_fsid unparsable uuid 2024-09-13T11:21:16.822 DEBUG:teuthology.orchestra.run.smithi092:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-13 2024-09-13T11:21:16.880 DEBUG:teuthology.orchestra.run.smithi162:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-09-13T11:21:16.894 INFO:tasks.ceph:roles_to_devs: {'osd.2': '/dev/vg_nvme/lv_1', 'osd.6': '/dev/vg_nvme/lv_2', 'osd.10': '/dev/vg_nvme/lv_3', 'osd.14': '/dev/vg_nvme/lv_4'} 2024-09-13T11:21:16.894 INFO:tasks.ceph:role: osd.2 2024-09-13T11:21:16.894 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi162.front.sepia.ceph.com 2024-09-13T11:21:16.894 DEBUG:teuthology.orchestra.run.smithi162:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-09-13T11:21:16.956 INFO:teuthology.orchestra.run.smithi162.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-09-13T11:21:16.956 INFO:teuthology.orchestra.run.smithi162.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-13T11:21:16.956 INFO:teuthology.orchestra.run.smithi162.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-13T11:21:16.956 INFO:teuthology.orchestra.run.smithi162.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-09-13T11:21:16.957 INFO:teuthology.orchestra.run.smithi162.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-13T11:21:16.957 INFO:teuthology.orchestra.run.smithi162.stdout: = sunit=0 swidth=0 blks 2024-09-13T11:21:16.957 INFO:teuthology.orchestra.run.smithi162.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-13T11:21:16.957 INFO:teuthology.orchestra.run.smithi162.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-13T11:21:16.957 INFO:teuthology.orchestra.run.smithi162.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-13T11:21:16.957 INFO:teuthology.orchestra.run.smithi162.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-13T11:21:17.410 INFO:teuthology.orchestra.run.smithi162.stdout:Discarding blocks...Done. 2024-09-13T11:21:17.413 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi162.front.sepia.ceph.com -o noatime 2024-09-13T11:21:17.413 DEBUG:teuthology.orchestra.run.smithi162:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-2 2024-09-13T11:21:17.500 DEBUG:teuthology.orchestra.run.smithi162:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-09-13T11:21:17.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T11:21:17.510 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: /sbin/restorecon: command not found 2024-09-13T11:21:17.510 DEBUG:teuthology.orchestra.run.smithi162:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2024-09-13T11:21:17.562 INFO:tasks.ceph:roles_to_devs: {'osd.2': '/dev/vg_nvme/lv_1', 'osd.6': '/dev/vg_nvme/lv_2', 'osd.10': '/dev/vg_nvme/lv_3', 'osd.14': '/dev/vg_nvme/lv_4'} 2024-09-13T11:21:17.562 INFO:tasks.ceph:role: osd.6 2024-09-13T11:21:17.562 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi162.front.sepia.ceph.com 2024-09-13T11:21:17.562 DEBUG:teuthology.orchestra.run.smithi162:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-09-13T11:21:17.624 INFO:teuthology.orchestra.run.smithi162.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-09-13T11:21:17.624 INFO:teuthology.orchestra.run.smithi162.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-13T11:21:17.625 INFO:teuthology.orchestra.run.smithi162.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-13T11:21:17.625 INFO:teuthology.orchestra.run.smithi162.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-09-13T11:21:17.625 INFO:teuthology.orchestra.run.smithi162.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-13T11:21:17.625 INFO:teuthology.orchestra.run.smithi162.stdout: = sunit=0 swidth=0 blks 2024-09-13T11:21:17.625 INFO:teuthology.orchestra.run.smithi162.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-13T11:21:17.625 INFO:teuthology.orchestra.run.smithi162.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-13T11:21:17.625 INFO:teuthology.orchestra.run.smithi162.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-13T11:21:17.625 INFO:teuthology.orchestra.run.smithi162.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-13T11:21:18.095 INFO:teuthology.orchestra.run.smithi162.stdout:Discarding blocks...Done. 2024-09-13T11:21:18.097 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi162.front.sepia.ceph.com -o noatime 2024-09-13T11:21:18.097 DEBUG:teuthology.orchestra.run.smithi162:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-6 2024-09-13T11:21:18.245 DEBUG:teuthology.orchestra.run.smithi162:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2024-09-13T11:21:18.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T11:21:18.257 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: /sbin/restorecon: command not found 2024-09-13T11:21:18.257 DEBUG:teuthology.orchestra.run.smithi162:> sudo mkdir -p /var/lib/ceph/osd/ceph-10 2024-09-13T11:21:18.312 INFO:tasks.ceph:roles_to_devs: {'osd.2': '/dev/vg_nvme/lv_1', 'osd.6': '/dev/vg_nvme/lv_2', 'osd.10': '/dev/vg_nvme/lv_3', 'osd.14': '/dev/vg_nvme/lv_4'} 2024-09-13T11:21:18.312 INFO:tasks.ceph:role: osd.10 2024-09-13T11:21:18.312 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi162.front.sepia.ceph.com 2024-09-13T11:21:18.312 DEBUG:teuthology.orchestra.run.smithi162:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-09-13T11:21:18.376 INFO:teuthology.orchestra.run.smithi162.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-09-13T11:21:18.376 INFO:teuthology.orchestra.run.smithi162.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-13T11:21:18.376 INFO:teuthology.orchestra.run.smithi162.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-13T11:21:18.377 INFO:teuthology.orchestra.run.smithi162.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-09-13T11:21:18.377 INFO:teuthology.orchestra.run.smithi162.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-13T11:21:18.377 INFO:teuthology.orchestra.run.smithi162.stdout: = sunit=0 swidth=0 blks 2024-09-13T11:21:18.377 INFO:teuthology.orchestra.run.smithi162.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-13T11:21:18.377 INFO:teuthology.orchestra.run.smithi162.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-13T11:21:18.377 INFO:teuthology.orchestra.run.smithi162.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-13T11:21:18.377 INFO:teuthology.orchestra.run.smithi162.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-13T11:21:18.861 INFO:teuthology.orchestra.run.smithi162.stdout:Discarding blocks...Done. 2024-09-13T11:21:18.863 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi162.front.sepia.ceph.com -o noatime 2024-09-13T11:21:18.863 DEBUG:teuthology.orchestra.run.smithi162:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-10 2024-09-13T11:21:18.996 DEBUG:teuthology.orchestra.run.smithi162:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-10 2024-09-13T11:21:19.007 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: /sbin/restorecon: command not found 2024-09-13T11:21:19.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T11:21:19.008 DEBUG:teuthology.orchestra.run.smithi162:> sudo mkdir -p /var/lib/ceph/osd/ceph-14 2024-09-13T11:21:19.063 INFO:tasks.ceph:roles_to_devs: {'osd.2': '/dev/vg_nvme/lv_1', 'osd.6': '/dev/vg_nvme/lv_2', 'osd.10': '/dev/vg_nvme/lv_3', 'osd.14': '/dev/vg_nvme/lv_4'} 2024-09-13T11:21:19.063 INFO:tasks.ceph:role: osd.14 2024-09-13T11:21:19.063 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi162.front.sepia.ceph.com 2024-09-13T11:21:19.064 DEBUG:teuthology.orchestra.run.smithi162:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-09-13T11:21:19.127 INFO:teuthology.orchestra.run.smithi162.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-09-13T11:21:19.127 INFO:teuthology.orchestra.run.smithi162.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-13T11:21:19.128 INFO:teuthology.orchestra.run.smithi162.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-13T11:21:19.128 INFO:teuthology.orchestra.run.smithi162.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-09-13T11:21:19.128 INFO:teuthology.orchestra.run.smithi162.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-13T11:21:19.128 INFO:teuthology.orchestra.run.smithi162.stdout: = sunit=0 swidth=0 blks 2024-09-13T11:21:19.128 INFO:teuthology.orchestra.run.smithi162.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-13T11:21:19.128 INFO:teuthology.orchestra.run.smithi162.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-13T11:21:19.128 INFO:teuthology.orchestra.run.smithi162.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-13T11:21:19.128 INFO:teuthology.orchestra.run.smithi162.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-13T11:21:19.552 INFO:teuthology.orchestra.run.smithi162.stdout:Discarding blocks...Done. 2024-09-13T11:21:19.554 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi162.front.sepia.ceph.com -o noatime 2024-09-13T11:21:19.555 DEBUG:teuthology.orchestra.run.smithi162:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-14 2024-09-13T11:21:19.631 DEBUG:teuthology.orchestra.run.smithi162:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-14 2024-09-13T11:21:19.684 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: /sbin/restorecon: command not found 2024-09-13T11:21:19.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T11:21:19.685 DEBUG:teuthology.orchestra.run.smithi162:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-13T11:21:19.816 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-13T11:21:19.814+0000 7effad7c87c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-09-13T11:21:19.817 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-13T11:21:19.814+0000 7effad7c87c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-09-13T11:21:19.817 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-13T11:21:19.814+0000 7effad7c87c0 -1 bdev(0x55e0a27bf800 /var/lib/ceph/osd/ceph-2/block) open stat got: (1) Operation not permitted 2024-09-13T11:21:19.817 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-13T11:21:19.814+0000 7effad7c87c0 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-09-13T11:21:22.962 DEBUG:teuthology.orchestra.run.smithi162:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-09-13T11:21:22.976 DEBUG:teuthology.orchestra.run.smithi162:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-13T11:21:23.058 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-13T11:21:23.054+0000 7fe90b2397c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2024-09-13T11:21:23.059 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-13T11:21:23.054+0000 7fe90b2397c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2024-09-13T11:21:23.059 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-13T11:21:23.054+0000 7fe90b2397c0 -1 bdev(0x55ef23d4f800 /var/lib/ceph/osd/ceph-6/block) open stat got: (1) Operation not permitted 2024-09-13T11:21:23.059 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-13T11:21:23.054+0000 7fe90b2397c0 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2024-09-13T11:21:26.196 DEBUG:teuthology.orchestra.run.smithi162:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-09-13T11:21:26.253 DEBUG:teuthology.orchestra.run.smithi162:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 10 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-13T11:21:26.335 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-13T11:21:26.330+0000 7f48d91607c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-10/keyring: can't open /var/lib/ceph/osd/ceph-10/keyring: (2) No such file or directory 2024-09-13T11:21:26.336 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-13T11:21:26.334+0000 7f48d91607c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-10/keyring 2024-09-13T11:21:26.336 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-13T11:21:26.334+0000 7f48d91607c0 -1 bdev(0x5591d5359800 /var/lib/ceph/osd/ceph-10/block) open stat got: (1) Operation not permitted 2024-09-13T11:21:26.336 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-13T11:21:26.334+0000 7f48d91607c0 -1 bluestore(/var/lib/ceph/osd/ceph-10) _read_fsid unparsable uuid 2024-09-13T11:21:29.475 DEBUG:teuthology.orchestra.run.smithi162:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-10 2024-09-13T11:21:29.533 DEBUG:teuthology.orchestra.run.smithi162:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 14 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-13T11:21:29.614 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-13T11:21:29.610+0000 7f92cc14d7c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-14/keyring: can't open /var/lib/ceph/osd/ceph-14/keyring: (2) No such file or directory 2024-09-13T11:21:29.615 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-13T11:21:29.610+0000 7f92cc14d7c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-14/keyring 2024-09-13T11:21:29.615 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-13T11:21:29.610+0000 7f92cc14d7c0 -1 bdev(0x55e08d351800 /var/lib/ceph/osd/ceph-14/block) open stat got: (1) Operation not permitted 2024-09-13T11:21:29.615 INFO:teuthology.orchestra.run.smithi162.stderr:2024-09-13T11:21:29.610+0000 7f92cc14d7c0 -1 bluestore(/var/lib/ceph/osd/ceph-14) _read_fsid unparsable uuid 2024-09-13T11:21:32.754 DEBUG:teuthology.orchestra.run.smithi162:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-14 2024-09-13T11:21:32.808 DEBUG:teuthology.orchestra.run.smithi177:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-09-13T11:21:32.823 INFO:tasks.ceph:roles_to_devs: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.7': '/dev/vg_nvme/lv_2', 'osd.11': '/dev/vg_nvme/lv_3', 'osd.15': '/dev/vg_nvme/lv_4'} 2024-09-13T11:21:32.823 INFO:tasks.ceph:role: osd.3 2024-09-13T11:21:32.823 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi177.front.sepia.ceph.com 2024-09-13T11:21:32.823 DEBUG:teuthology.orchestra.run.smithi177:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-09-13T11:21:32.885 INFO:teuthology.orchestra.run.smithi177.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-09-13T11:21:32.885 INFO:teuthology.orchestra.run.smithi177.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-13T11:21:32.885 INFO:teuthology.orchestra.run.smithi177.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-13T11:21:32.886 INFO:teuthology.orchestra.run.smithi177.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-09-13T11:21:32.886 INFO:teuthology.orchestra.run.smithi177.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-13T11:21:32.886 INFO:teuthology.orchestra.run.smithi177.stdout: = sunit=0 swidth=0 blks 2024-09-13T11:21:32.886 INFO:teuthology.orchestra.run.smithi177.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-13T11:21:32.886 INFO:teuthology.orchestra.run.smithi177.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-13T11:21:32.886 INFO:teuthology.orchestra.run.smithi177.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-13T11:21:32.886 INFO:teuthology.orchestra.run.smithi177.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-13T11:21:33.362 INFO:teuthology.orchestra.run.smithi177.stdout:Discarding blocks...Done. 2024-09-13T11:21:33.364 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi177.front.sepia.ceph.com -o noatime 2024-09-13T11:21:33.364 DEBUG:teuthology.orchestra.run.smithi177:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-3 2024-09-13T11:21:33.510 DEBUG:teuthology.orchestra.run.smithi177:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-09-13T11:21:33.521 INFO:teuthology.orchestra.run.smithi177.stderr:sudo: /sbin/restorecon: command not found 2024-09-13T11:21:33.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T11:21:33.522 DEBUG:teuthology.orchestra.run.smithi177:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2024-09-13T11:21:33.577 INFO:tasks.ceph:roles_to_devs: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.7': '/dev/vg_nvme/lv_2', 'osd.11': '/dev/vg_nvme/lv_3', 'osd.15': '/dev/vg_nvme/lv_4'} 2024-09-13T11:21:33.577 INFO:tasks.ceph:role: osd.7 2024-09-13T11:21:33.577 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi177.front.sepia.ceph.com 2024-09-13T11:21:33.577 DEBUG:teuthology.orchestra.run.smithi177:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-09-13T11:21:33.640 INFO:teuthology.orchestra.run.smithi177.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-09-13T11:21:33.640 INFO:teuthology.orchestra.run.smithi177.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-13T11:21:33.640 INFO:teuthology.orchestra.run.smithi177.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-13T11:21:33.640 INFO:teuthology.orchestra.run.smithi177.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-09-13T11:21:33.640 INFO:teuthology.orchestra.run.smithi177.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-13T11:21:33.640 INFO:teuthology.orchestra.run.smithi177.stdout: = sunit=0 swidth=0 blks 2024-09-13T11:21:33.641 INFO:teuthology.orchestra.run.smithi177.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-13T11:21:33.641 INFO:teuthology.orchestra.run.smithi177.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-13T11:21:33.641 INFO:teuthology.orchestra.run.smithi177.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-13T11:21:33.641 INFO:teuthology.orchestra.run.smithi177.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-13T11:21:34.089 INFO:teuthology.orchestra.run.smithi177.stdout:Discarding blocks...Done. 2024-09-13T11:21:34.091 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi177.front.sepia.ceph.com -o noatime 2024-09-13T11:21:34.091 DEBUG:teuthology.orchestra.run.smithi177:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-7 2024-09-13T11:21:34.184 DEBUG:teuthology.orchestra.run.smithi177:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2024-09-13T11:21:34.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T11:21:34.194 INFO:teuthology.orchestra.run.smithi177.stderr:sudo: /sbin/restorecon: command not found 2024-09-13T11:21:34.194 DEBUG:teuthology.orchestra.run.smithi177:> sudo mkdir -p /var/lib/ceph/osd/ceph-11 2024-09-13T11:21:34.246 INFO:tasks.ceph:roles_to_devs: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.7': '/dev/vg_nvme/lv_2', 'osd.11': '/dev/vg_nvme/lv_3', 'osd.15': '/dev/vg_nvme/lv_4'} 2024-09-13T11:21:34.247 INFO:tasks.ceph:role: osd.11 2024-09-13T11:21:34.247 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi177.front.sepia.ceph.com 2024-09-13T11:21:34.247 DEBUG:teuthology.orchestra.run.smithi177:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-09-13T11:21:34.308 INFO:teuthology.orchestra.run.smithi177.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-09-13T11:21:34.308 INFO:teuthology.orchestra.run.smithi177.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-13T11:21:34.308 INFO:teuthology.orchestra.run.smithi177.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-13T11:21:34.308 INFO:teuthology.orchestra.run.smithi177.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-09-13T11:21:34.308 INFO:teuthology.orchestra.run.smithi177.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-13T11:21:34.308 INFO:teuthology.orchestra.run.smithi177.stdout: = sunit=0 swidth=0 blks 2024-09-13T11:21:34.308 INFO:teuthology.orchestra.run.smithi177.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-13T11:21:34.309 INFO:teuthology.orchestra.run.smithi177.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-13T11:21:34.309 INFO:teuthology.orchestra.run.smithi177.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-13T11:21:34.309 INFO:teuthology.orchestra.run.smithi177.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-13T11:21:34.769 INFO:teuthology.orchestra.run.smithi177.stdout:Discarding blocks...Done. 2024-09-13T11:21:34.770 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi177.front.sepia.ceph.com -o noatime 2024-09-13T11:21:34.770 DEBUG:teuthology.orchestra.run.smithi177:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-11 2024-09-13T11:21:34.904 DEBUG:teuthology.orchestra.run.smithi177:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-11 2024-09-13T11:21:34.915 INFO:teuthology.orchestra.run.smithi177.stderr:sudo: /sbin/restorecon: command not found 2024-09-13T11:21:34.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T11:21:34.916 DEBUG:teuthology.orchestra.run.smithi177:> sudo mkdir -p /var/lib/ceph/osd/ceph-15 2024-09-13T11:21:34.973 INFO:tasks.ceph:roles_to_devs: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.7': '/dev/vg_nvme/lv_2', 'osd.11': '/dev/vg_nvme/lv_3', 'osd.15': '/dev/vg_nvme/lv_4'} 2024-09-13T11:21:34.973 INFO:tasks.ceph:role: osd.15 2024-09-13T11:21:34.973 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi177.front.sepia.ceph.com 2024-09-13T11:21:34.973 DEBUG:teuthology.orchestra.run.smithi177:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-09-13T11:21:35.037 INFO:teuthology.orchestra.run.smithi177.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-09-13T11:21:35.037 INFO:teuthology.orchestra.run.smithi177.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-13T11:21:35.037 INFO:teuthology.orchestra.run.smithi177.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-13T11:21:35.038 INFO:teuthology.orchestra.run.smithi177.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-09-13T11:21:35.038 INFO:teuthology.orchestra.run.smithi177.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-13T11:21:35.038 INFO:teuthology.orchestra.run.smithi177.stdout: = sunit=0 swidth=0 blks 2024-09-13T11:21:35.038 INFO:teuthology.orchestra.run.smithi177.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-13T11:21:35.038 INFO:teuthology.orchestra.run.smithi177.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-13T11:21:35.038 INFO:teuthology.orchestra.run.smithi177.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-13T11:21:35.038 INFO:teuthology.orchestra.run.smithi177.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-13T11:21:35.529 INFO:teuthology.orchestra.run.smithi177.stdout:Discarding blocks...Done. 2024-09-13T11:21:35.531 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi177.front.sepia.ceph.com -o noatime 2024-09-13T11:21:35.531 DEBUG:teuthology.orchestra.run.smithi177:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-15 2024-09-13T11:21:35.665 DEBUG:teuthology.orchestra.run.smithi177:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-15 2024-09-13T11:21:35.683 INFO:teuthology.orchestra.run.smithi177.stderr:sudo: /sbin/restorecon: command not found 2024-09-13T11:21:35.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T11:21:35.685 DEBUG:teuthology.orchestra.run.smithi177:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-13T11:21:35.821 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-13T11:21:35.823+0000 7f8351d4e7c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-09-13T11:21:35.825 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-13T11:21:35.823+0000 7f8351d4e7c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-09-13T11:21:35.825 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-13T11:21:35.823+0000 7f8351d4e7c0 -1 bdev(0x56358cad9800 /var/lib/ceph/osd/ceph-3/block) open stat got: (1) Operation not permitted 2024-09-13T11:21:35.825 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-13T11:21:35.823+0000 7f8351d4e7c0 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-09-13T11:21:38.979 DEBUG:teuthology.orchestra.run.smithi177:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-09-13T11:21:38.994 DEBUG:teuthology.orchestra.run.smithi177:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-13T11:21:39.080 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-13T11:21:39.079+0000 7f9594e107c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2024-09-13T11:21:39.081 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-13T11:21:39.079+0000 7f9594e107c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2024-09-13T11:21:39.081 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-13T11:21:39.083+0000 7f9594e107c0 -1 bdev(0x55a3594ff800 /var/lib/ceph/osd/ceph-7/block) open stat got: (1) Operation not permitted 2024-09-13T11:21:39.081 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-13T11:21:39.083+0000 7f9594e107c0 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2024-09-13T11:21:42.211 DEBUG:teuthology.orchestra.run.smithi177:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-09-13T11:21:42.266 DEBUG:teuthology.orchestra.run.smithi177:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 11 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-13T11:21:42.349 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-13T11:21:42.351+0000 7efec59057c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-11/keyring: can't open /var/lib/ceph/osd/ceph-11/keyring: (2) No such file or directory 2024-09-13T11:21:42.350 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-13T11:21:42.351+0000 7efec59057c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-11/keyring 2024-09-13T11:21:42.350 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-13T11:21:42.351+0000 7efec59057c0 -1 bdev(0x55bf5f903800 /var/lib/ceph/osd/ceph-11/block) open stat got: (1) Operation not permitted 2024-09-13T11:21:42.350 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-13T11:21:42.351+0000 7efec59057c0 -1 bluestore(/var/lib/ceph/osd/ceph-11) _read_fsid unparsable uuid 2024-09-13T11:21:45.495 DEBUG:teuthology.orchestra.run.smithi177:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-11 2024-09-13T11:21:45.550 DEBUG:teuthology.orchestra.run.smithi177:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 15 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-13T11:21:45.631 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-13T11:21:45.631+0000 7fa6990b07c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-15/keyring: can't open /var/lib/ceph/osd/ceph-15/keyring: (2) No such file or directory 2024-09-13T11:21:45.632 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-13T11:21:45.631+0000 7fa6990b07c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-15/keyring 2024-09-13T11:21:45.632 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-13T11:21:45.631+0000 7fa6990b07c0 -1 bdev(0x562561ffb800 /var/lib/ceph/osd/ceph-15/block) open stat got: (1) Operation not permitted 2024-09-13T11:21:45.632 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-13T11:21:45.631+0000 7fa6990b07c0 -1 bluestore(/var/lib/ceph/osd/ceph-15) _read_fsid unparsable uuid 2024-09-13T11:21:48.797 DEBUG:teuthology.orchestra.run.smithi177:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-15 2024-09-13T11:21:48.854 INFO:tasks.ceph:Reading keys from all nodes... 2024-09-13T11:21:48.855 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-09-13T11:21:48.855 DEBUG:teuthology.orchestra.run.smithi001:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2024-09-13T11:21:48.869 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-09-13T11:21:48.870 DEBUG:teuthology.orchestra.run.smithi001:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-09-13T11:21:48.925 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-09-13T11:21:48.925 DEBUG:teuthology.orchestra.run.smithi001:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-09-13T11:21:48.980 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-09-13T11:21:48.981 DEBUG:teuthology.orchestra.run.smithi001:> sudo dd if=/var/lib/ceph/osd/ceph-8/keyring of=/dev/stdout 2024-09-13T11:21:49.037 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-09-13T11:21:49.037 DEBUG:teuthology.orchestra.run.smithi001:> sudo dd if=/var/lib/ceph/osd/ceph-12/keyring of=/dev/stdout 2024-09-13T11:21:49.092 DEBUG:teuthology.orchestra.run.smithi092:> set -ex 2024-09-13T11:21:49.093 DEBUG:teuthology.orchestra.run.smithi092:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-09-13T11:21:49.107 DEBUG:teuthology.orchestra.run.smithi092:> set -ex 2024-09-13T11:21:49.107 DEBUG:teuthology.orchestra.run.smithi092:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-09-13T11:21:49.161 DEBUG:teuthology.orchestra.run.smithi092:> set -ex 2024-09-13T11:21:49.161 DEBUG:teuthology.orchestra.run.smithi092:> sudo dd if=/var/lib/ceph/osd/ceph-9/keyring of=/dev/stdout 2024-09-13T11:21:49.217 DEBUG:teuthology.orchestra.run.smithi092:> set -ex 2024-09-13T11:21:49.217 DEBUG:teuthology.orchestra.run.smithi092:> sudo dd if=/var/lib/ceph/osd/ceph-13/keyring of=/dev/stdout 2024-09-13T11:21:49.274 DEBUG:teuthology.orchestra.run.smithi162:> set -ex 2024-09-13T11:21:49.274 DEBUG:teuthology.orchestra.run.smithi162:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-09-13T11:21:49.289 DEBUG:teuthology.orchestra.run.smithi162:> set -ex 2024-09-13T11:21:49.289 DEBUG:teuthology.orchestra.run.smithi162:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2024-09-13T11:21:49.343 DEBUG:teuthology.orchestra.run.smithi162:> set -ex 2024-09-13T11:21:49.343 DEBUG:teuthology.orchestra.run.smithi162:> sudo dd if=/var/lib/ceph/osd/ceph-10/keyring of=/dev/stdout 2024-09-13T11:21:49.395 DEBUG:teuthology.orchestra.run.smithi162:> set -ex 2024-09-13T11:21:49.395 DEBUG:teuthology.orchestra.run.smithi162:> sudo dd if=/var/lib/ceph/osd/ceph-14/keyring of=/dev/stdout 2024-09-13T11:21:49.450 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-09-13T11:21:49.451 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-09-13T11:21:49.464 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-09-13T11:21:49.464 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-09-13T11:21:49.519 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-09-13T11:21:49.520 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2024-09-13T11:21:49.571 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-09-13T11:21:49.572 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd if=/var/lib/ceph/osd/ceph-11/keyring of=/dev/stdout 2024-09-13T11:21:49.624 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-09-13T11:21:49.624 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd if=/var/lib/ceph/osd/ceph-15/keyring of=/dev/stdout 2024-09-13T11:21:49.680 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-09-13T11:21:49.680 DEBUG:teuthology.orchestra.run.smithi177:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-09-13T11:21:49.729 INFO:tasks.ceph:Adding keys to all mons... 2024-09-13T11:21:49.729 DEBUG:teuthology.orchestra.run.smithi001:> sudo tee -a /etc/ceph/ceph.keyring 2024-09-13T11:21:49.731 DEBUG:teuthology.orchestra.run.smithi092:> sudo tee -a /etc/ceph/ceph.keyring 2024-09-13T11:21:49.734 DEBUG:teuthology.orchestra.run.smithi162:> sudo tee -a /etc/ceph/ceph.keyring 2024-09-13T11:21:49.742 INFO:teuthology.orchestra.run.smithi001.stdout:[mgr.y] 2024-09-13T11:21:49.742 INFO:teuthology.orchestra.run.smithi001.stdout: key = AQAOIORmnc7ZGxAA1P0tqFyLDGZvMFns2QnwCw== 2024-09-13T11:21:49.742 INFO:teuthology.orchestra.run.smithi001.stdout:[osd.0] 2024-09-13T11:21:49.742 INFO:teuthology.orchestra.run.smithi001.stdout: key = AQAPIORmZZPyKBAABXHLkfaYQxgbB4TG5fwIlg== 2024-09-13T11:21:49.742 INFO:teuthology.orchestra.run.smithi001.stdout:[osd.4] 2024-09-13T11:21:49.742 INFO:teuthology.orchestra.run.smithi001.stdout: key = AQASIORm/zTzORAAnl1+SNuvpfm21r0rizlh6g== 2024-09-13T11:21:49.742 INFO:teuthology.orchestra.run.smithi001.stdout:[osd.8] 2024-09-13T11:21:49.742 INFO:teuthology.orchestra.run.smithi001.stdout: key = AQAWIORmcCzbDRAAx+n0T22q0m0x7LYYb5tG6Q== 2024-09-13T11:21:49.742 INFO:teuthology.orchestra.run.smithi001.stdout:[osd.12] 2024-09-13T11:21:49.742 INFO:teuthology.orchestra.run.smithi001.stdout: key = AQAZIORmAfvRHxAAur8S06NIF9JunuSkqKpwRQ== 2024-09-13T11:21:49.742 INFO:teuthology.orchestra.run.smithi001.stdout:[osd.1] 2024-09-13T11:21:49.742 INFO:teuthology.orchestra.run.smithi001.stdout: key = AQAfIORmKr9zMBAAzgT0RTYggRe1LZzX/UStZw== 2024-09-13T11:21:49.743 INFO:teuthology.orchestra.run.smithi001.stdout:[osd.5] 2024-09-13T11:21:49.743 INFO:teuthology.orchestra.run.smithi001.stdout: key = AQAjIORmmZgEBRAAmn+5jYuJt4vWauif/q+SpQ== 2024-09-13T11:21:49.743 INFO:teuthology.orchestra.run.smithi001.stdout:[osd.9] 2024-09-13T11:21:49.743 INFO:teuthology.orchestra.run.smithi001.stdout: key = AQAmIORmyDl5FxAAPo6Q8WHXMHBCzXl19i4FWQ== 2024-09-13T11:21:49.743 INFO:teuthology.orchestra.run.smithi001.stdout:[osd.13] 2024-09-13T11:21:49.743 INFO:teuthology.orchestra.run.smithi001.stdout: key = AQApIORmv0EPKRAAQiQzkSc2lDKGcd+efM0dQA== 2024-09-13T11:21:49.743 INFO:teuthology.orchestra.run.smithi001.stdout:[osd.2] 2024-09-13T11:21:49.743 INFO:teuthology.orchestra.run.smithi001.stdout: key = AQAvIORmOjnEMBAAidJjE+H/y8y9tKgH65PR/Q== 2024-09-13T11:21:49.743 INFO:teuthology.orchestra.run.smithi001.stdout:[osd.6] 2024-09-13T11:21:49.743 INFO:teuthology.orchestra.run.smithi001.stdout: key = AQAzIORmnnSVAxAAIJXcrwcIsyXqowDw5YvT3A== 2024-09-13T11:21:49.743 INFO:teuthology.orchestra.run.smithi001.stdout:[osd.10] 2024-09-13T11:21:49.743 INFO:teuthology.orchestra.run.smithi001.stdout: key = AQA2IORmKCoZFBAA7ZBgAdR+rYekLOIw1m51wQ== 2024-09-13T11:21:49.743 INFO:teuthology.orchestra.run.smithi001.stdout:[osd.14] 2024-09-13T11:21:49.743 INFO:teuthology.orchestra.run.smithi001.stdout: key = AQA5IORmhCS+JBAALAXSWxY3RVGLPQJ9fYAFSQ== 2024-09-13T11:21:49.744 INFO:teuthology.orchestra.run.smithi001.stdout:[mgr.x] 2024-09-13T11:21:49.744 INFO:teuthology.orchestra.run.smithi001.stdout: key = AQAOIORmYMXCHhAAXA/AaPBW+3NXT089AYPr1A== 2024-09-13T11:21:49.744 INFO:teuthology.orchestra.run.smithi001.stdout:[osd.3] 2024-09-13T11:21:49.744 INFO:teuthology.orchestra.run.smithi001.stdout: key = AQA/IORmKxUjMRAAqzEzL1MQKJ74bel9Zsnqgg== 2024-09-13T11:21:49.744 INFO:teuthology.orchestra.run.smithi001.stdout:[osd.7] 2024-09-13T11:21:49.744 INFO:teuthology.orchestra.run.smithi001.stdout: key = AQBDIORmcvL5BBAAZ0NHyU7XwYoKGhCcVGtk0g== 2024-09-13T11:21:49.744 INFO:teuthology.orchestra.run.smithi001.stdout:[osd.11] 2024-09-13T11:21:49.744 INFO:teuthology.orchestra.run.smithi001.stdout: key = AQBGIORme/cEFRAANHdM/H5W/SXUXCTT+0B9yA== 2024-09-13T11:21:49.744 INFO:teuthology.orchestra.run.smithi001.stdout:[osd.15] 2024-09-13T11:21:49.744 INFO:teuthology.orchestra.run.smithi001.stdout: key = AQBJIORmt0nRJRAA+DM592kNDc1m/QAgDDCA4g== 2024-09-13T11:21:49.745 INFO:teuthology.orchestra.run.smithi001.stdout:[client.0] 2024-09-13T11:21:49.745 INFO:teuthology.orchestra.run.smithi001.stdout: key = AQAOIORmmRIcJBAAmZWlfbpSlrtPRpTOF6lULg== 2024-09-13T11:21:49.746 INFO:teuthology.orchestra.run.smithi092.stdout:[mgr.y] 2024-09-13T11:21:49.746 INFO:teuthology.orchestra.run.smithi092.stdout: key = AQAOIORmnc7ZGxAA1P0tqFyLDGZvMFns2QnwCw== 2024-09-13T11:21:49.746 INFO:teuthology.orchestra.run.smithi092.stdout:[osd.0] 2024-09-13T11:21:49.746 INFO:teuthology.orchestra.run.smithi092.stdout: key = AQAPIORmZZPyKBAABXHLkfaYQxgbB4TG5fwIlg== 2024-09-13T11:21:49.746 INFO:teuthology.orchestra.run.smithi092.stdout:[osd.4] 2024-09-13T11:21:49.746 INFO:teuthology.orchestra.run.smithi092.stdout: key = AQASIORm/zTzORAAnl1+SNuvpfm21r0rizlh6g== 2024-09-13T11:21:49.747 INFO:teuthology.orchestra.run.smithi092.stdout:[osd.8] 2024-09-13T11:21:49.747 INFO:teuthology.orchestra.run.smithi092.stdout: key = AQAWIORmcCzbDRAAx+n0T22q0m0x7LYYb5tG6Q== 2024-09-13T11:21:49.747 INFO:teuthology.orchestra.run.smithi092.stdout:[osd.12] 2024-09-13T11:21:49.747 INFO:teuthology.orchestra.run.smithi092.stdout: key = AQAZIORmAfvRHxAAur8S06NIF9JunuSkqKpwRQ== 2024-09-13T11:21:49.747 INFO:teuthology.orchestra.run.smithi092.stdout:[osd.1] 2024-09-13T11:21:49.747 INFO:teuthology.orchestra.run.smithi092.stdout: key = AQAfIORmKr9zMBAAzgT0RTYggRe1LZzX/UStZw== 2024-09-13T11:21:49.747 INFO:teuthology.orchestra.run.smithi092.stdout:[osd.5] 2024-09-13T11:21:49.747 INFO:teuthology.orchestra.run.smithi092.stdout: key = AQAjIORmmZgEBRAAmn+5jYuJt4vWauif/q+SpQ== 2024-09-13T11:21:49.747 INFO:teuthology.orchestra.run.smithi092.stdout:[osd.9] 2024-09-13T11:21:49.747 INFO:teuthology.orchestra.run.smithi092.stdout: key = AQAmIORmyDl5FxAAPo6Q8WHXMHBCzXl19i4FWQ== 2024-09-13T11:21:49.747 INFO:teuthology.orchestra.run.smithi092.stdout:[osd.13] 2024-09-13T11:21:49.747 INFO:teuthology.orchestra.run.smithi092.stdout: key = AQApIORmv0EPKRAAQiQzkSc2lDKGcd+efM0dQA== 2024-09-13T11:21:49.747 INFO:teuthology.orchestra.run.smithi092.stdout:[osd.2] 2024-09-13T11:21:49.747 INFO:teuthology.orchestra.run.smithi092.stdout: key = AQAvIORmOjnEMBAAidJjE+H/y8y9tKgH65PR/Q== 2024-09-13T11:21:49.748 INFO:teuthology.orchestra.run.smithi092.stdout:[osd.6] 2024-09-13T11:21:49.748 INFO:teuthology.orchestra.run.smithi092.stdout: key = AQAzIORmnnSVAxAAIJXcrwcIsyXqowDw5YvT3A== 2024-09-13T11:21:49.748 INFO:teuthology.orchestra.run.smithi092.stdout:[osd.10] 2024-09-13T11:21:49.748 INFO:teuthology.orchestra.run.smithi092.stdout: key = AQA2IORmKCoZFBAA7ZBgAdR+rYekLOIw1m51wQ== 2024-09-13T11:21:49.748 INFO:teuthology.orchestra.run.smithi092.stdout:[osd.14] 2024-09-13T11:21:49.748 INFO:teuthology.orchestra.run.smithi092.stdout: key = AQA5IORmhCS+JBAALAXSWxY3RVGLPQJ9fYAFSQ== 2024-09-13T11:21:49.748 INFO:teuthology.orchestra.run.smithi092.stdout:[mgr.x] 2024-09-13T11:21:49.748 INFO:teuthology.orchestra.run.smithi092.stdout: key = AQAOIORmYMXCHhAAXA/AaPBW+3NXT089AYPr1A== 2024-09-13T11:21:49.748 INFO:teuthology.orchestra.run.smithi092.stdout:[osd.3] 2024-09-13T11:21:49.748 INFO:teuthology.orchestra.run.smithi092.stdout: key = AQA/IORmKxUjMRAAqzEzL1MQKJ74bel9Zsnqgg== 2024-09-13T11:21:49.748 INFO:teuthology.orchestra.run.smithi092.stdout:[osd.7] 2024-09-13T11:21:49.751 INFO:teuthology.orchestra.run.smithi092.stdout: key = AQBDIORmcvL5BBAAZ0NHyU7XwYoKGhCcVGtk0g== 2024-09-13T11:21:49.751 INFO:teuthology.orchestra.run.smithi092.stdout:[osd.11] 2024-09-13T11:21:49.751 INFO:teuthology.orchestra.run.smithi092.stdout: key = AQBGIORme/cEFRAANHdM/H5W/SXUXCTT+0B9yA== 2024-09-13T11:21:49.751 INFO:teuthology.orchestra.run.smithi092.stdout:[osd.15] 2024-09-13T11:21:49.751 INFO:teuthology.orchestra.run.smithi092.stdout: key = AQBJIORmt0nRJRAA+DM592kNDc1m/QAgDDCA4g== 2024-09-13T11:21:49.752 INFO:teuthology.orchestra.run.smithi092.stdout:[client.0] 2024-09-13T11:21:49.752 INFO:teuthology.orchestra.run.smithi092.stdout: key = AQAOIORmmRIcJBAAmZWlfbpSlrtPRpTOF6lULg== 2024-09-13T11:21:49.753 INFO:teuthology.orchestra.run.smithi162.stdout:[mgr.y] 2024-09-13T11:21:49.754 INFO:teuthology.orchestra.run.smithi162.stdout: key = AQAOIORmnc7ZGxAA1P0tqFyLDGZvMFns2QnwCw== 2024-09-13T11:21:49.754 INFO:teuthology.orchestra.run.smithi162.stdout:[osd.0] 2024-09-13T11:21:49.754 INFO:teuthology.orchestra.run.smithi162.stdout: key = AQAPIORmZZPyKBAABXHLkfaYQxgbB4TG5fwIlg== 2024-09-13T11:21:49.754 INFO:teuthology.orchestra.run.smithi162.stdout:[osd.4] 2024-09-13T11:21:49.754 INFO:teuthology.orchestra.run.smithi162.stdout: key = AQASIORm/zTzORAAnl1+SNuvpfm21r0rizlh6g== 2024-09-13T11:21:49.754 INFO:teuthology.orchestra.run.smithi162.stdout:[osd.8] 2024-09-13T11:21:49.755 INFO:teuthology.orchestra.run.smithi162.stdout: key = AQAWIORmcCzbDRAAx+n0T22q0m0x7LYYb5tG6Q== 2024-09-13T11:21:49.755 INFO:teuthology.orchestra.run.smithi162.stdout:[osd.12] 2024-09-13T11:21:49.755 INFO:teuthology.orchestra.run.smithi162.stdout: key = AQAZIORmAfvRHxAAur8S06NIF9JunuSkqKpwRQ== 2024-09-13T11:21:49.755 INFO:teuthology.orchestra.run.smithi162.stdout:[osd.1] 2024-09-13T11:21:49.755 INFO:teuthology.orchestra.run.smithi162.stdout: key = AQAfIORmKr9zMBAAzgT0RTYggRe1LZzX/UStZw== 2024-09-13T11:21:49.756 INFO:teuthology.orchestra.run.smithi162.stdout:[osd.5] 2024-09-13T11:21:49.756 INFO:teuthology.orchestra.run.smithi162.stdout: key = AQAjIORmmZgEBRAAmn+5jYuJt4vWauif/q+SpQ== 2024-09-13T11:21:49.756 INFO:teuthology.orchestra.run.smithi162.stdout:[osd.9] 2024-09-13T11:21:49.756 INFO:teuthology.orchestra.run.smithi162.stdout: key = AQAmIORmyDl5FxAAPo6Q8WHXMHBCzXl19i4FWQ== 2024-09-13T11:21:49.756 INFO:teuthology.orchestra.run.smithi162.stdout:[osd.13] 2024-09-13T11:21:49.756 INFO:teuthology.orchestra.run.smithi162.stdout: key = AQApIORmv0EPKRAAQiQzkSc2lDKGcd+efM0dQA== 2024-09-13T11:21:49.756 INFO:teuthology.orchestra.run.smithi162.stdout:[osd.2] 2024-09-13T11:21:49.757 INFO:teuthology.orchestra.run.smithi162.stdout: key = AQAvIORmOjnEMBAAidJjE+H/y8y9tKgH65PR/Q== 2024-09-13T11:21:49.757 INFO:teuthology.orchestra.run.smithi162.stdout:[osd.6] 2024-09-13T11:21:49.757 INFO:teuthology.orchestra.run.smithi162.stdout: key = AQAzIORmnnSVAxAAIJXcrwcIsyXqowDw5YvT3A== 2024-09-13T11:21:49.757 INFO:teuthology.orchestra.run.smithi162.stdout:[osd.10] 2024-09-13T11:21:49.757 INFO:teuthology.orchestra.run.smithi162.stdout: key = AQA2IORmKCoZFBAA7ZBgAdR+rYekLOIw1m51wQ== 2024-09-13T11:21:49.757 INFO:teuthology.orchestra.run.smithi162.stdout:[osd.14] 2024-09-13T11:21:49.758 INFO:teuthology.orchestra.run.smithi162.stdout: key = AQA5IORmhCS+JBAALAXSWxY3RVGLPQJ9fYAFSQ== 2024-09-13T11:21:49.758 INFO:teuthology.orchestra.run.smithi162.stdout:[mgr.x] 2024-09-13T11:21:49.758 INFO:teuthology.orchestra.run.smithi162.stdout: key = AQAOIORmYMXCHhAAXA/AaPBW+3NXT089AYPr1A== 2024-09-13T11:21:49.758 INFO:teuthology.orchestra.run.smithi162.stdout:[osd.3] 2024-09-13T11:21:49.758 INFO:teuthology.orchestra.run.smithi162.stdout: key = AQA/IORmKxUjMRAAqzEzL1MQKJ74bel9Zsnqgg== 2024-09-13T11:21:49.758 INFO:teuthology.orchestra.run.smithi162.stdout:[osd.7] 2024-09-13T11:21:49.758 INFO:teuthology.orchestra.run.smithi162.stdout: key = AQBDIORmcvL5BBAAZ0NHyU7XwYoKGhCcVGtk0g== 2024-09-13T11:21:49.758 INFO:teuthology.orchestra.run.smithi162.stdout:[osd.11] 2024-09-13T11:21:49.758 INFO:teuthology.orchestra.run.smithi162.stdout: key = AQBGIORme/cEFRAANHdM/H5W/SXUXCTT+0B9yA== 2024-09-13T11:21:49.758 INFO:teuthology.orchestra.run.smithi162.stdout:[osd.15] 2024-09-13T11:21:49.758 INFO:teuthology.orchestra.run.smithi162.stdout: key = AQBJIORmt0nRJRAA+DM592kNDc1m/QAgDDCA4g== 2024-09-13T11:21:49.759 INFO:teuthology.orchestra.run.smithi162.stdout:[client.0] 2024-09-13T11:21:49.759 INFO:teuthology.orchestra.run.smithi162.stdout: key = AQAOIORmmRIcJBAAmZWlfbpSlrtPRpTOF6lULg== 2024-09-13T11:21:49.759 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-13T11:21:49.789 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-13T11:21:49.792 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-13T11:21:49.840 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T11:21:49.878 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T11:21:49.880 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T11:21:49.924 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T11:21:49.926 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T11:21:49.963 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T11:21:50.017 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.8 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T11:21:50.019 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.8 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T11:21:50.055 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.8 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T11:21:50.101 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.12 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T11:21:50.103 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.12 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T11:21:50.143 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.12 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T11:21:50.184 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T11:21:50.186 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T11:21:50.227 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T11:21:50.272 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T11:21:50.274 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T11:21:50.315 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T11:21:50.357 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.9 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T11:21:50.361 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.9 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T11:21:50.403 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.9 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T11:21:50.448 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.13 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T11:21:50.451 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.13 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T11:21:50.487 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.13 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T11:21:50.533 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T11:21:50.541 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T11:21:50.571 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T11:21:50.615 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T11:21:50.629 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T11:21:50.659 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T11:21:50.703 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.10 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T11:21:50.718 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.10 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T11:21:50.720 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.10 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T11:21:50.783 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.14 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T11:21:50.801 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.14 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T11:21:50.804 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.14 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T11:21:50.868 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-13T11:21:50.890 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-13T11:21:50.892 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-13T11:21:50.950 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T11:21:50.953 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T11:21:50.979 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T11:21:51.034 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T11:21:51.041 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T11:21:51.063 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T11:21:51.119 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.11 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T11:21:51.129 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.11 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T11:21:51.151 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.11 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T11:21:51.203 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.15 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T11:21:51.206 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.15 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T11:21:51.208 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.15 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T11:21:51.287 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-13T11:21:51.290 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-13T11:21:51.292 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-13T11:21:51.374 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-09-13T11:21:51.374 DEBUG:teuthology.orchestra.run.smithi001:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-09-13T11:21:51.389 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-13T11:21:51.485 DEBUG:teuthology.orchestra.run.smithi001:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-09-13T11:21:51.498 DEBUG:teuthology.orchestra.run.smithi092:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-09-13T11:21:51.512 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-13T11:21:51.607 DEBUG:teuthology.orchestra.run.smithi092:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-09-13T11:21:51.617 DEBUG:teuthology.orchestra.run.smithi162:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-09-13T11:21:51.632 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-13T11:21:51.732 DEBUG:teuthology.orchestra.run.smithi162:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-09-13T11:21:51.744 DEBUG:teuthology.orchestra.run.smithi001:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-09-13T11:21:51.747 DEBUG:teuthology.orchestra.run.smithi092:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-09-13T11:21:51.749 DEBUG:teuthology.orchestra.run.smithi162:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-09-13T11:21:51.792 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-09-13T11:21:51.792 INFO:tasks.ceph.mon.a:Restarting daemon 2024-09-13T11:21:51.792 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-09-13T11:21:51.797 INFO:tasks.ceph.mon.a:Started 2024-09-13T11:21:51.797 INFO:tasks.ceph.mon.b:Restarting daemon 2024-09-13T11:21:51.798 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-09-13T11:21:51.800 INFO:tasks.ceph.mon.b:Started 2024-09-13T11:21:51.800 INFO:tasks.ceph.mon.c:Restarting daemon 2024-09-13T11:21:51.800 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-09-13T11:21:51.837 INFO:tasks.ceph.mon.c:Started 2024-09-13T11:21:51.837 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-09-13T11:21:51.837 INFO:tasks.ceph.mgr.y:Restarting daemon 2024-09-13T11:21:51.837 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2024-09-13T11:21:51.840 INFO:tasks.ceph.mgr.y:Started 2024-09-13T11:21:51.840 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-09-13T11:21:51.840 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-09-13T11:21:51.842 INFO:tasks.ceph.mgr.x:Started 2024-09-13T11:21:51.843 DEBUG:tasks.ceph:set 0 configs 2024-09-13T11:21:51.843 INFO:tasks.ceph:Setting crush tunables to default 2024-09-13T11:21:51.843 DEBUG:teuthology.orchestra.run.smithi001:> sudo ceph --cluster ceph osd crush tunables default 2024-09-13T11:21:57.384 INFO:teuthology.orchestra.run.smithi001.stderr:adjusted tunables profile to default 2024-09-13T11:21:57.402 INFO:tasks.ceph:check_enable_crimson: False 2024-09-13T11:21:57.403 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-09-13T11:21:57.403 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-09-13T11:21:57.403 DEBUG:teuthology.orchestra.run.smithi001:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-09-13T11:21:57.415 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-09-13T11:21:57.415 DEBUG:teuthology.orchestra.run.smithi001:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-09-13T11:21:57.467 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-09-13T11:21:57.467 DEBUG:teuthology.orchestra.run.smithi001:> sudo dd if=/var/lib/ceph/osd/ceph-8/fsid of=/dev/stdout 2024-09-13T11:21:57.523 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-09-13T11:21:57.523 DEBUG:teuthology.orchestra.run.smithi001:> sudo dd if=/var/lib/ceph/osd/ceph-12/fsid of=/dev/stdout 2024-09-13T11:21:57.577 DEBUG:teuthology.orchestra.run.smithi092:> set -ex 2024-09-13T11:21:57.577 DEBUG:teuthology.orchestra.run.smithi092:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-09-13T11:21:57.591 DEBUG:teuthology.orchestra.run.smithi092:> set -ex 2024-09-13T11:21:57.592 DEBUG:teuthology.orchestra.run.smithi092:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-09-13T11:21:57.645 DEBUG:teuthology.orchestra.run.smithi092:> set -ex 2024-09-13T11:21:57.645 DEBUG:teuthology.orchestra.run.smithi092:> sudo dd if=/var/lib/ceph/osd/ceph-9/fsid of=/dev/stdout 2024-09-13T11:21:57.704 DEBUG:teuthology.orchestra.run.smithi092:> set -ex 2024-09-13T11:21:57.704 DEBUG:teuthology.orchestra.run.smithi092:> sudo dd if=/var/lib/ceph/osd/ceph-13/fsid of=/dev/stdout 2024-09-13T11:21:57.758 DEBUG:teuthology.orchestra.run.smithi162:> set -ex 2024-09-13T11:21:57.758 DEBUG:teuthology.orchestra.run.smithi162:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-09-13T11:21:57.773 DEBUG:teuthology.orchestra.run.smithi162:> set -ex 2024-09-13T11:21:57.773 DEBUG:teuthology.orchestra.run.smithi162:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2024-09-13T11:21:57.828 DEBUG:teuthology.orchestra.run.smithi162:> set -ex 2024-09-13T11:21:57.828 DEBUG:teuthology.orchestra.run.smithi162:> sudo dd if=/var/lib/ceph/osd/ceph-10/fsid of=/dev/stdout 2024-09-13T11:21:57.888 DEBUG:teuthology.orchestra.run.smithi162:> set -ex 2024-09-13T11:21:57.888 DEBUG:teuthology.orchestra.run.smithi162:> sudo dd if=/var/lib/ceph/osd/ceph-14/fsid of=/dev/stdout 2024-09-13T11:21:57.942 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-09-13T11:21:57.942 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-09-13T11:21:57.956 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-09-13T11:21:57.956 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2024-09-13T11:21:58.011 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-09-13T11:21:58.012 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd if=/var/lib/ceph/osd/ceph-11/fsid of=/dev/stdout 2024-09-13T11:21:58.063 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-09-13T11:21:58.063 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd if=/var/lib/ceph/osd/ceph-15/fsid of=/dev/stdout 2024-09-13T11:21:58.116 DEBUG:teuthology.orchestra.run.smithi177:> sudo ceph --cluster ceph osd new 0fc12b3c-e0de-4546-b935-db77efe7c567 0 2024-09-13T11:21:58.374 INFO:teuthology.orchestra.run.smithi177.stdout:0 2024-09-13T11:21:58.389 DEBUG:teuthology.orchestra.run.smithi177:> sudo ceph --cluster ceph osd new b77fe331-cc92-4adb-b146-af5c380f6d4f 1 2024-09-13T11:21:58.598 INFO:teuthology.orchestra.run.smithi177.stdout:1 2024-09-13T11:21:58.614 DEBUG:teuthology.orchestra.run.smithi177:> sudo ceph --cluster ceph osd new c9d031c8-e885-4229-affa-08e1729d00e8 2 2024-09-13T11:21:58.816 INFO:teuthology.orchestra.run.smithi177.stdout:2 2024-09-13T11:21:58.832 DEBUG:teuthology.orchestra.run.smithi177:> sudo ceph --cluster ceph osd new bf50d71a-6c3a-4fbc-a407-cb81bad14a7b 3 2024-09-13T11:21:59.027 INFO:teuthology.orchestra.run.smithi177.stdout:3 2024-09-13T11:21:59.043 DEBUG:teuthology.orchestra.run.smithi177:> sudo ceph --cluster ceph osd new d1fe3f63-df3f-4a73-a1e2-8d7c8a3f1fa0 4 2024-09-13T11:21:59.235 INFO:teuthology.orchestra.run.smithi177.stdout:4 2024-09-13T11:21:59.251 DEBUG:teuthology.orchestra.run.smithi177:> sudo ceph --cluster ceph osd new e2ead447-2f2c-40c9-82e7-53b8dc41e2af 5 2024-09-13T11:21:59.449 INFO:teuthology.orchestra.run.smithi177.stdout:5 2024-09-13T11:21:59.465 DEBUG:teuthology.orchestra.run.smithi177:> sudo ceph --cluster ceph osd new dbc20439-144e-4ec7-89de-674eb783f263 6 2024-09-13T11:21:59.672 INFO:teuthology.orchestra.run.smithi177.stdout:6 2024-09-13T11:21:59.689 DEBUG:teuthology.orchestra.run.smithi177:> sudo ceph --cluster ceph osd new 60aa5206-cace-40f8-8d1a-10ec28d97096 7 2024-09-13T11:21:59.929 INFO:teuthology.orchestra.run.smithi177.stdout:7 2024-09-13T11:21:59.945 DEBUG:teuthology.orchestra.run.smithi177:> sudo ceph --cluster ceph osd new f3d99325-48ff-403c-8662-1d7745f7c97d 8 2024-09-13T11:22:00.153 INFO:tasks.ceph.mgr.x.smithi177.stderr:/usr/lib/python3/dist-packages/scipy/__init__.py:67: UserWarning: NumPy was imported from a Python sub-interpreter but NumPy does not properly support sub-interpreters. This will likely work for most users but might cause hard to track down issues or subtle bugs. A common user of the rare sub-interpreter feature is wsgi which also allows single-interpreter mode. 2024-09-13T11:22:00.153 INFO:tasks.ceph.mgr.x.smithi177.stderr:Improvements in the case of bugs are welcome, but is not on the NumPy roadmap, and full support may require significant effort to achieve. 2024-09-13T11:22:00.153 INFO:tasks.ceph.mgr.x.smithi177.stderr: from numpy import show_config as show_numpy_config 2024-09-13T11:22:00.158 INFO:teuthology.orchestra.run.smithi177.stdout:8 2024-09-13T11:22:00.175 DEBUG:teuthology.orchestra.run.smithi177:> sudo ceph --cluster ceph osd new 88511e24-3675-4551-968c-1a52e7e28f96 9 2024-09-13T11:22:00.337 INFO:tasks.ceph.mgr.y.smithi001.stderr:/usr/lib/python3/dist-packages/scipy/__init__.py:67: UserWarning: NumPy was imported from a Python sub-interpreter but NumPy does not properly support sub-interpreters. This will likely work for most users but might cause hard to track down issues or subtle bugs. A common user of the rare sub-interpreter feature is wsgi which also allows single-interpreter mode. 2024-09-13T11:22:00.337 INFO:tasks.ceph.mgr.y.smithi001.stderr:Improvements in the case of bugs are welcome, but is not on the NumPy roadmap, and full support may require significant effort to achieve. 2024-09-13T11:22:00.338 INFO:tasks.ceph.mgr.y.smithi001.stderr: from numpy import show_config as show_numpy_config 2024-09-13T11:22:00.384 INFO:teuthology.orchestra.run.smithi177.stdout:9 2024-09-13T11:22:00.400 DEBUG:teuthology.orchestra.run.smithi177:> sudo ceph --cluster ceph osd new acf4c64d-99af-42be-8406-413c86f54670 10 2024-09-13T11:22:00.603 INFO:teuthology.orchestra.run.smithi177.stdout:10 2024-09-13T11:22:00.620 DEBUG:teuthology.orchestra.run.smithi177:> sudo ceph --cluster ceph osd new a4bfdfbe-b586-4080-8e09-6f26428e59b5 11 2024-09-13T11:22:00.857 INFO:teuthology.orchestra.run.smithi177.stdout:11 2024-09-13T11:22:00.873 DEBUG:teuthology.orchestra.run.smithi177:> sudo ceph --cluster ceph osd new dc571fb6-d8b5-465b-bb25-afa7d0c47b5a 12 2024-09-13T11:22:01.080 INFO:teuthology.orchestra.run.smithi177.stdout:12 2024-09-13T11:22:01.097 DEBUG:teuthology.orchestra.run.smithi177:> sudo ceph --cluster ceph osd new 5d321d4b-1b26-47fe-bf84-25751b8cde7d 13 2024-09-13T11:22:01.303 INFO:teuthology.orchestra.run.smithi177.stdout:13 2024-09-13T11:22:01.319 DEBUG:teuthology.orchestra.run.smithi177:> sudo ceph --cluster ceph osd new fc3e2499-0c2a-414d-a25e-2628876ae630 14 2024-09-13T11:22:01.525 INFO:teuthology.orchestra.run.smithi177.stdout:14 2024-09-13T11:22:01.541 DEBUG:teuthology.orchestra.run.smithi177:> sudo ceph --cluster ceph osd new d2f69775-edfe-4527-91ec-f7c7e558bec8 15 2024-09-13T11:22:01.748 INFO:teuthology.orchestra.run.smithi177.stdout:15 2024-09-13T11:22:01.766 INFO:tasks.ceph.osd.0:Restarting daemon 2024-09-13T11:22:01.766 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2024-09-13T11:22:01.768 INFO:tasks.ceph.osd.0:Started 2024-09-13T11:22:01.769 INFO:tasks.ceph.osd.4:Restarting daemon 2024-09-13T11:22:01.769 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2024-09-13T11:22:01.771 INFO:tasks.ceph.osd.4:Started 2024-09-13T11:22:01.771 INFO:tasks.ceph.osd.8:Restarting daemon 2024-09-13T11:22:01.771 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 8 2024-09-13T11:22:01.776 INFO:tasks.ceph.osd.8:Started 2024-09-13T11:22:01.777 INFO:tasks.ceph.osd.12:Restarting daemon 2024-09-13T11:22:01.777 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 12 2024-09-13T11:22:01.779 INFO:tasks.ceph.osd.12:Started 2024-09-13T11:22:01.779 INFO:tasks.ceph.osd.1:Restarting daemon 2024-09-13T11:22:01.779 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-09-13T11:22:01.781 INFO:tasks.ceph.osd.1:Started 2024-09-13T11:22:01.782 INFO:tasks.ceph.osd.5:Restarting daemon 2024-09-13T11:22:01.782 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2024-09-13T11:22:01.785 INFO:tasks.ceph.osd.5:Started 2024-09-13T11:22:01.785 INFO:tasks.ceph.osd.9:Restarting daemon 2024-09-13T11:22:01.785 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 9 2024-09-13T11:22:01.787 INFO:tasks.ceph.osd.9:Started 2024-09-13T11:22:01.787 INFO:tasks.ceph.osd.13:Restarting daemon 2024-09-13T11:22:01.787 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 13 2024-09-13T11:22:01.790 INFO:tasks.ceph.osd.13:Started 2024-09-13T11:22:01.790 INFO:tasks.ceph.osd.2:Restarting daemon 2024-09-13T11:22:01.790 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2024-09-13T11:22:01.793 INFO:tasks.ceph.osd.2:Started 2024-09-13T11:22:01.793 INFO:tasks.ceph.osd.6:Restarting daemon 2024-09-13T11:22:01.793 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2024-09-13T11:22:01.795 INFO:tasks.ceph.osd.6:Started 2024-09-13T11:22:01.795 INFO:tasks.ceph.osd.10:Restarting daemon 2024-09-13T11:22:01.795 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 10 2024-09-13T11:22:01.798 INFO:tasks.ceph.osd.10:Started 2024-09-13T11:22:01.798 INFO:tasks.ceph.osd.14:Restarting daemon 2024-09-13T11:22:01.798 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 14 2024-09-13T11:22:01.800 INFO:tasks.ceph.osd.14:Started 2024-09-13T11:22:01.800 INFO:tasks.ceph.osd.3:Restarting daemon 2024-09-13T11:22:01.801 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2024-09-13T11:22:01.803 INFO:tasks.ceph.osd.3:Started 2024-09-13T11:22:01.803 INFO:tasks.ceph.osd.7:Restarting daemon 2024-09-13T11:22:01.803 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2024-09-13T11:22:01.805 INFO:tasks.ceph.osd.7:Started 2024-09-13T11:22:01.806 INFO:tasks.ceph.osd.11:Restarting daemon 2024-09-13T11:22:01.806 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 11 2024-09-13T11:22:01.808 INFO:tasks.ceph.osd.11:Started 2024-09-13T11:22:01.808 INFO:tasks.ceph.osd.15:Restarting daemon 2024-09-13T11:22:01.808 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 15 2024-09-13T11:22:01.810 INFO:tasks.ceph.osd.15:Started 2024-09-13T11:22:01.811 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T11:22:02.009 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-13T11:22:02.009 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":18,"fsid":"2fd68ac7-5581-4d90-80a4-30bcf5252a7a","created":"2024-09-13T11:21:57.223153+0000","modified":"2024-09-13T11:22:01.745814+0000","last_up_change":"0.000000","last_in_change":"2024-09-13T11:22:01.745814+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"0fc12b3c-e0de-4546-b935-db77efe7c567","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b77fe331-cc92-4adb-b146-af5c380f6d4f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c9d031c8-e885-4229-affa-08e1729d00e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"bf50d71a-6c3a-4fbc-a407-cb81bad14a7b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d1fe3f63-df3f-4a73-a1e2-8d7c8a3f1fa0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e2ead447-2f2c-40c9-82e7-53b8dc41e2af","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"dbc20439-144e-4ec7-89de-674eb783f263","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"60aa5206-cace-40f8-8d1a-10ec28d97096","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":8,"uuid":"f3d99325-48ff-403c-8662-1d7745f7c97d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":9,"uuid":"88511e24-3675-4551-968c-1a52e7e28f96","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":10,"uuid":"acf4c64d-99af-42be-8406-413c86f54670","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":11,"uuid":"a4bfdfbe-b586-4080-8e09-6f26428e59b5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":12,"uuid":"dc571fb6-d8b5-465b-bb25-afa7d0c47b5a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":13,"uuid":"5d321d4b-1b26-47fe-bf84-25751b8cde7d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":14,"uuid":"fc3e2499-0c2a-414d-a25e-2628876ae630","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":15,"uuid":"d2f69775-edfe-4527-91ec-f7c7e558bec8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T11:22:02.025 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-09-13T11:22:02.025 INFO:tasks.ceph:Waiting for OSDs to come up 2024-09-13T11:22:02.945 INFO:tasks.ceph.osd.3.smithi177.stderr:2024-09-13T11:22:02.947+0000 7f10b0be87c0 -1 Falling back to public interface 2024-09-13T11:22:03.007 INFO:tasks.ceph.osd.1.smithi092.stderr:2024-09-13T11:22:03.006+0000 7fc2bd0bd7c0 -1 Falling back to public interface 2024-09-13T11:22:03.014 INFO:tasks.ceph.osd.5.smithi092.stderr:2024-09-13T11:22:03.014+0000 7f6f0e7497c0 -1 Falling back to public interface 2024-09-13T11:22:03.050 INFO:tasks.ceph.osd.13.smithi092.stderr:2024-09-13T11:22:03.050+0000 7f3e5e02e7c0 -1 Falling back to public interface 2024-09-13T11:22:03.066 INFO:tasks.ceph.osd.9.smithi092.stderr:2024-09-13T11:22:03.066+0000 7ff5ae4147c0 -1 Falling back to public interface 2024-09-13T11:22:03.125 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-13T11:22:03.121+0000 7f6a6b7b97c0 -1 Falling back to public interface 2024-09-13T11:22:03.131 INFO:tasks.ceph.mgr.x.smithi177.stderr:2024-09-13T11:22:03.131+0000 7fde15788640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-09-13T11:22:03.133 INFO:tasks.ceph.mgr.x.smithi177.stderr:2024-09-13T11:22:03.135+0000 7fde15788640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-09-13T11:22:03.169 INFO:tasks.ceph.osd.15.smithi177.stderr:2024-09-13T11:22:03.171+0000 7f9d6febb7c0 -1 Falling back to public interface 2024-09-13T11:22:03.177 INFO:tasks.ceph.osd.8.smithi001.stderr:2024-09-13T11:22:03.173+0000 7fb9b78f27c0 -1 Falling back to public interface 2024-09-13T11:22:03.193 INFO:tasks.ceph.osd.11.smithi177.stderr:2024-09-13T11:22:03.195+0000 7f247c30b7c0 -1 Falling back to public interface 2024-09-13T11:22:03.205 INFO:tasks.ceph.osd.14.smithi162.stderr:2024-09-13T11:22:03.202+0000 7f235f11f7c0 -1 Falling back to public interface 2024-09-13T11:22:03.205 INFO:tasks.ceph.osd.10.smithi162.stderr:2024-09-13T11:22:03.202+0000 7fb59fa347c0 -1 Falling back to public interface 2024-09-13T11:22:03.405 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-09-13T11:22:03.402+0000 7f97c200d7c0 -1 Falling back to public interface 2024-09-13T11:22:03.410 INFO:tasks.ceph.osd.7.smithi177.stderr:2024-09-13T11:22:03.411+0000 7f897e5617c0 -1 Falling back to public interface 2024-09-13T11:22:03.414 INFO:tasks.ceph.osd.4.smithi001.stderr:2024-09-13T11:22:03.409+0000 7ff8e325c7c0 -1 Falling back to public interface 2024-09-13T11:22:03.425 INFO:tasks.ceph.osd.12.smithi001.stderr:2024-09-13T11:22:03.421+0000 7f892f72a7c0 -1 Falling back to public interface 2024-09-13T11:22:03.465 INFO:tasks.ceph.osd.2.smithi162.stderr:2024-09-13T11:22:03.462+0000 7fe92e39e7c0 -1 Falling back to public interface 2024-09-13T11:22:03.636 DEBUG:teuthology.orchestra.run.smithi001:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-09-13T11:22:03.948 INFO:teuthology.misc.health.smithi001.stdout: 2024-09-13T11:22:03.948 INFO:teuthology.misc.health.smithi001.stdout:{"epoch":18,"fsid":"2fd68ac7-5581-4d90-80a4-30bcf5252a7a","created":"2024-09-13T11:21:57.223153+0000","modified":"2024-09-13T11:22:01.745814+0000","last_up_change":"0.000000","last_in_change":"2024-09-13T11:22:01.745814+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"0fc12b3c-e0de-4546-b935-db77efe7c567","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b77fe331-cc92-4adb-b146-af5c380f6d4f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c9d031c8-e885-4229-affa-08e1729d00e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"bf50d71a-6c3a-4fbc-a407-cb81bad14a7b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d1fe3f63-df3f-4a73-a1e2-8d7c8a3f1fa0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e2ead447-2f2c-40c9-82e7-53b8dc41e2af","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"dbc20439-144e-4ec7-89de-674eb783f263","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"60aa5206-cace-40f8-8d1a-10ec28d97096","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":8,"uuid":"f3d99325-48ff-403c-8662-1d7745f7c97d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":9,"uuid":"88511e24-3675-4551-968c-1a52e7e28f96","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":10,"uuid":"acf4c64d-99af-42be-8406-413c86f54670","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":11,"uuid":"a4bfdfbe-b586-4080-8e09-6f26428e59b5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":12,"uuid":"dc571fb6-d8b5-465b-bb25-afa7d0c47b5a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":13,"uuid":"5d321d4b-1b26-47fe-bf84-25751b8cde7d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":14,"uuid":"fc3e2499-0c2a-414d-a25e-2628876ae630","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":15,"uuid":"d2f69775-edfe-4527-91ec-f7c7e558bec8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T11:22:03.966 DEBUG:teuthology.misc:0 of 16 OSDs are up 2024-09-13T11:22:06.900 INFO:tasks.ceph.osd.3.smithi177.stderr:2024-09-13T11:22:06.898+0000 7f10b0be87c0 -1 osd.3 0 log_to_monitors true 2024-09-13T11:22:06.907 INFO:tasks.ceph.osd.5.smithi092.stderr:2024-09-13T11:22:06.906+0000 7f6f0e7497c0 -1 osd.5 0 log_to_monitors true 2024-09-13T11:22:07.012 INFO:tasks.ceph.osd.1.smithi092.stderr:2024-09-13T11:22:07.010+0000 7fc2bd0bd7c0 -1 osd.1 0 log_to_monitors true 2024-09-13T11:22:07.120 INFO:tasks.ceph.osd.9.smithi092.stderr:2024-09-13T11:22:07.118+0000 7ff5ae4147c0 -1 osd.9 0 log_to_monitors true 2024-09-13T11:22:07.131 INFO:tasks.ceph.osd.15.smithi177.stderr:2024-09-13T11:22:07.130+0000 7f9d6febb7c0 -1 osd.15 0 log_to_monitors true 2024-09-13T11:22:07.188 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-13T11:22:07.185+0000 7f6a6b7b97c0 -1 osd.0 0 log_to_monitors true 2024-09-13T11:22:07.221 INFO:tasks.ceph.osd.11.smithi177.stderr:2024-09-13T11:22:07.222+0000 7f247c30b7c0 -1 osd.11 0 log_to_monitors true 2024-09-13T11:22:07.229 INFO:tasks.ceph.osd.10.smithi162.stderr:2024-09-13T11:22:07.226+0000 7fb59fa347c0 -1 osd.10 0 log_to_monitors true 2024-09-13T11:22:07.238 INFO:tasks.ceph.osd.14.smithi162.stderr:2024-09-13T11:22:07.234+0000 7f235f11f7c0 -1 osd.14 0 log_to_monitors true 2024-09-13T11:22:07.335 INFO:tasks.ceph.osd.13.smithi092.stderr:2024-09-13T11:22:07.334+0000 7f3e5e02e7c0 -1 osd.13 0 log_to_monitors true 2024-09-13T11:22:07.368 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-09-13T11:22:07.366+0000 7f97c200d7c0 -1 osd.6 0 log_to_monitors true 2024-09-13T11:22:07.575 INFO:tasks.ceph.osd.4.smithi001.stderr:2024-09-13T11:22:07.569+0000 7ff8e325c7c0 -1 osd.4 0 log_to_monitors true 2024-09-13T11:22:07.670 INFO:tasks.ceph.osd.12.smithi001.stderr:2024-09-13T11:22:07.665+0000 7f892f72a7c0 -1 osd.12 0 log_to_monitors true 2024-09-13T11:22:07.795 INFO:tasks.ceph.osd.8.smithi001.stderr:2024-09-13T11:22:07.789+0000 7fb9b78f27c0 -1 osd.8 0 log_to_monitors true 2024-09-13T11:22:07.824 INFO:tasks.ceph.osd.7.smithi177.stderr:2024-09-13T11:22:07.822+0000 7f897e5617c0 -1 osd.7 0 log_to_monitors true 2024-09-13T11:22:07.915 INFO:tasks.ceph.osd.2.smithi162.stderr:2024-09-13T11:22:07.910+0000 7fe92e39e7c0 -1 osd.2 0 log_to_monitors true 2024-09-13T11:22:07.979 INFO:tasks.ceph.osd.5.smithi092.stderr:2024-09-13T11:22:07.978+0000 7f6f0a6f2640 -1 osd.5 0 waiting for initial osdmap 2024-09-13T11:22:07.984 INFO:tasks.ceph.osd.3.smithi177.stderr:2024-09-13T11:22:07.982+0000 7f10acb91640 -1 osd.3 0 waiting for initial osdmap 2024-09-13T11:22:07.986 INFO:tasks.ceph.osd.3.smithi177.stderr:2024-09-13T11:22:07.986+0000 7f10a45d5640 -1 osd.3 0 handle_osd_map injecting map crc failure 2024-09-13T11:22:07.993 INFO:tasks.ceph.osd.3.smithi177.stderr:2024-09-13T11:22:07.994+0000 7f10a89cb640 -1 osd.3 20 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-13T11:22:07.993 INFO:tasks.ceph.osd.5.smithi092.stderr:2024-09-13T11:22:07.990+0000 7f6f0652c640 -1 osd.5 20 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-13T11:22:09.006 INFO:tasks.ceph.osd.8.smithi001.stderr:2024-09-13T11:22:09.001+0000 7fb9b389b640 -1 osd.8 0 waiting for initial osdmap 2024-09-13T11:22:09.009 INFO:tasks.ceph.osd.7.smithi177.stderr:2024-09-13T11:22:09.010+0000 7f897a50a640 -1 osd.7 0 waiting for initial osdmap 2024-09-13T11:22:09.010 INFO:tasks.ceph.osd.15.smithi177.stderr:2024-09-13T11:22:09.010+0000 7f9d6be64640 -1 osd.15 0 waiting for initial osdmap 2024-09-13T11:22:09.012 INFO:tasks.ceph.osd.8.smithi001.stderr:2024-09-13T11:22:09.005+0000 7fb9ab2df640 -1 osd.8 0 handle_osd_map injecting map crc failure 2024-09-13T11:22:09.012 INFO:tasks.ceph.osd.15.smithi177.stderr:2024-09-13T11:22:09.010+0000 7f9d638a8640 -1 osd.15 0 handle_osd_map injecting map crc failure 2024-09-13T11:22:09.013 INFO:tasks.ceph.osd.7.smithi177.stderr:2024-09-13T11:22:09.014+0000 7f8971f4e640 -1 osd.7 0 handle_osd_map injecting map crc failure 2024-09-13T11:22:09.017 INFO:tasks.ceph.osd.10.smithi162.stderr:2024-09-13T11:22:09.014+0000 7fb59b9dd640 -1 osd.10 0 waiting for initial osdmap 2024-09-13T11:22:09.017 INFO:tasks.ceph.osd.14.smithi162.stderr:2024-09-13T11:22:09.014+0000 7f235b0c8640 -1 osd.14 0 waiting for initial osdmap 2024-09-13T11:22:09.017 INFO:tasks.ceph.osd.10.smithi162.stderr:2024-09-13T11:22:09.014+0000 7fb593421640 -1 osd.10 0 handle_osd_map injecting map crc failure 2024-09-13T11:22:09.018 INFO:tasks.ceph.osd.14.smithi162.stderr:2024-09-13T11:22:09.014+0000 7f2352b0c640 -1 osd.14 0 handle_osd_map injecting map crc failure 2024-09-13T11:22:09.019 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-09-13T11:22:09.014+0000 7f97bdfb6640 -1 osd.6 0 waiting for initial osdmap 2024-09-13T11:22:09.019 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-13T11:22:09.013+0000 7f6a67762640 -1 osd.0 0 waiting for initial osdmap 2024-09-13T11:22:09.019 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-13T11:22:09.013+0000 7f6a5f1a6640 -1 osd.0 0 handle_osd_map injecting map crc failure 2024-09-13T11:22:09.020 INFO:tasks.ceph.osd.2.smithi162.stderr:2024-09-13T11:22:09.018+0000 7fe92a347640 -1 osd.2 0 waiting for initial osdmap 2024-09-13T11:22:09.020 INFO:tasks.ceph.osd.15.smithi177.stderr:2024-09-13T11:22:09.018+0000 7f9d67c9e640 -1 osd.15 21 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-13T11:22:09.025 INFO:tasks.ceph.osd.8.smithi001.stderr:2024-09-13T11:22:09.021+0000 7fb9af6d5640 -1 osd.8 21 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-13T11:22:09.025 INFO:tasks.ceph.osd.10.smithi162.stderr:2024-09-13T11:22:09.022+0000 7fb597817640 -1 osd.10 21 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-13T11:22:09.025 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-13T11:22:09.021+0000 7f6a6359c640 -1 osd.0 21 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-13T11:22:09.026 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-09-13T11:22:09.018+0000 7f97b59fa640 -1 osd.6 0 handle_osd_map injecting map crc failure 2024-09-13T11:22:09.026 INFO:tasks.ceph.osd.14.smithi162.stderr:2024-09-13T11:22:09.022+0000 7f2356f02640 -1 osd.14 21 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-13T11:22:09.026 INFO:tasks.ceph.osd.7.smithi177.stderr:2024-09-13T11:22:09.026+0000 7f8976344640 -1 osd.7 21 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-13T11:22:09.027 INFO:tasks.ceph.osd.2.smithi162.stderr:2024-09-13T11:22:09.026+0000 7fe926181640 -1 osd.2 21 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-13T11:22:09.031 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-09-13T11:22:09.026+0000 7f97b9df0640 -1 osd.6 21 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-13T11:22:09.856 INFO:tasks.ceph.osd.11.smithi177.stderr:2024-09-13T11:22:09.854+0000 7f24782b4640 -1 osd.11 0 waiting for initial osdmap 2024-09-13T11:22:09.856 INFO:tasks.ceph.osd.11.smithi177.stderr:2024-09-13T11:22:09.854+0000 7f246bcf0640 -1 osd.11 0 handle_osd_map injecting map crc failure 2024-09-13T11:22:09.865 INFO:tasks.ceph.osd.11.smithi177.stderr:2024-09-13T11:22:09.866+0000 7f24700e6640 -1 osd.11 21 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-13T11:22:09.874 INFO:tasks.ceph.osd.12.smithi001.stderr:2024-09-13T11:22:09.869+0000 7f892b6d3640 -1 osd.12 0 waiting for initial osdmap 2024-09-13T11:22:09.875 INFO:tasks.ceph.osd.12.smithi001.stderr:2024-09-13T11:22:09.869+0000 7f891f10f640 -1 osd.12 0 handle_osd_map injecting map crc failure 2024-09-13T11:22:09.882 INFO:tasks.ceph.osd.12.smithi001.stderr:2024-09-13T11:22:09.877+0000 7f8923505640 -1 osd.12 21 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-13T11:22:09.899 INFO:tasks.ceph.osd.4.smithi001.stderr:2024-09-13T11:22:09.893+0000 7ff8df205640 -1 osd.4 0 waiting for initial osdmap 2024-09-13T11:22:09.905 INFO:tasks.ceph.osd.4.smithi001.stderr:2024-09-13T11:22:09.901+0000 7ff8d7037640 -1 osd.4 21 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-13T11:22:10.728 INFO:tasks.ceph.osd.1.smithi092.stderr:2024-09-13T11:22:10.725+0000 7fc2b9878640 -1 osd.1 0 waiting for initial osdmap 2024-09-13T11:22:10.729 INFO:tasks.ceph.osd.1.smithi092.stderr:2024-09-13T11:22:10.725+0000 7fc2acaa2640 -1 osd.1 0 handle_osd_map injecting map crc failure 2024-09-13T11:22:10.736 INFO:tasks.ceph.osd.1.smithi092.stderr:2024-09-13T11:22:10.733+0000 7fc2acaa2640 -1 osd.1 14 handle_osd_map injecting map crc failure 2024-09-13T11:22:10.739 INFO:tasks.ceph.osd.1.smithi092.stderr:2024-09-13T11:22:10.737+0000 7fc2b0e98640 -1 osd.1 22 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-13T11:22:10.755 INFO:tasks.ceph.osd.13.smithi092.stderr:2024-09-13T11:22:10.753+0000 7f3e59fd7640 -1 osd.13 0 waiting for initial osdmap 2024-09-13T11:22:10.755 INFO:tasks.ceph.osd.13.smithi092.stderr:2024-09-13T11:22:10.753+0000 7f3e4da13640 -1 osd.13 0 handle_osd_map injecting map crc failure 2024-09-13T11:22:10.762 INFO:tasks.ceph.osd.13.smithi092.stderr:2024-09-13T11:22:10.757+0000 7f3e51e09640 -1 osd.13 22 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-13T11:22:10.767 INFO:tasks.ceph.osd.9.smithi092.stderr:2024-09-13T11:22:10.765+0000 7ff5aa3bd640 -1 osd.9 0 waiting for initial osdmap 2024-09-13T11:22:10.767 INFO:tasks.ceph.osd.9.smithi092.stderr:2024-09-13T11:22:10.765+0000 7ff59ddf9640 -1 osd.9 0 handle_osd_map injecting map crc failure 2024-09-13T11:22:10.774 INFO:tasks.ceph.osd.9.smithi092.stderr:2024-09-13T11:22:10.773+0000 7ff5a21ef640 -1 osd.9 22 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-13T11:22:11.575 DEBUG:teuthology.orchestra.run.smithi001:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-09-13T11:22:11.895 INFO:teuthology.misc.health.smithi001.stdout: 2024-09-13T11:22:11.895 INFO:teuthology.misc.health.smithi001.stdout:{"epoch":23,"fsid":"2fd68ac7-5581-4d90-80a4-30bcf5252a7a","created":"2024-09-13T11:21:57.223153+0000","modified":"2024-09-13T11:22:11.098388+0000","last_up_change":"2024-09-13T11:22:11.098388+0000","last_in_change":"2024-09-13T11:22:01.745814+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T11:22:10.162266+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{},"read_balance":{"score_type":"Fair distribution","score_acting":15.380000114440918,"score_stable":15.380000114440918,"optimal_score":0.12999999523162842,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"0fc12b3c-e0de-4546-b935-db77efe7c567","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6800","nonce":1544776477}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6801","nonce":1544776477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6803","nonce":1544776477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6802","nonce":1544776477}]},"public_addr":"172.21.15.1:6800/1544776477","cluster_addr":"172.21.15.1:6801/1544776477","heartbeat_back_addr":"172.21.15.1:6803/1544776477","heartbeat_front_addr":"172.21.15.1:6802/1544776477","state":["exists","up"]},{"osd":1,"uuid":"b77fe331-cc92-4adb-b146-af5c380f6d4f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.92:6800","nonce":356319619}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.92:6801","nonce":356319619}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.92:6803","nonce":356319619}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.92:6802","nonce":356319619}]},"public_addr":"172.21.15.92:6800/356319619","cluster_addr":"172.21.15.92:6801/356319619","heartbeat_back_addr":"172.21.15.92:6803/356319619","heartbeat_front_addr":"172.21.15.92:6802/356319619","state":["exists","up"]},{"osd":2,"uuid":"c9d031c8-e885-4229-affa-08e1729d00e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.162:6812","nonce":2712148101}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.162:6813","nonce":2712148101}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.162:6815","nonce":2712148101}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.162:6814","nonce":2712148101}]},"public_addr":"172.21.15.162:6812/2712148101","cluster_addr":"172.21.15.162:6813/2712148101","heartbeat_back_addr":"172.21.15.162:6815/2712148101","heartbeat_front_addr":"172.21.15.162:6814/2712148101","state":["exists","up"]},{"osd":3,"uuid":"bf50d71a-6c3a-4fbc-a407-cb81bad14a7b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.177:6801","nonce":3367751690}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.177:6802","nonce":3367751690}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.177:6804","nonce":3367751690}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.177:6803","nonce":3367751690}]},"public_addr":"172.21.15.177:6801/3367751690","cluster_addr":"172.21.15.177:6802/3367751690","heartbeat_back_addr":"172.21.15.177:6804/3367751690","heartbeat_front_addr":"172.21.15.177:6803/3367751690","state":["exists","up"]},{"osd":4,"uuid":"d1fe3f63-df3f-4a73-a1e2-8d7c8a3f1fa0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6808","nonce":3648540926}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6809","nonce":3648540926}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6811","nonce":3648540926}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6810","nonce":3648540926}]},"public_addr":"172.21.15.1:6808/3648540926","cluster_addr":"172.21.15.1:6809/3648540926","heartbeat_back_addr":"172.21.15.1:6811/3648540926","heartbeat_front_addr":"172.21.15.1:6810/3648540926","state":["exists","up"]},{"osd":5,"uuid":"e2ead447-2f2c-40c9-82e7-53b8dc41e2af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.92:6804","nonce":3399529091}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.92:6805","nonce":3399529091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.92:6807","nonce":3399529091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.92:6806","nonce":3399529091}]},"public_addr":"172.21.15.92:6804/3399529091","cluster_addr":"172.21.15.92:6805/3399529091","heartbeat_back_addr":"172.21.15.92:6807/3399529091","heartbeat_front_addr":"172.21.15.92:6806/3399529091","state":["exists","up"]},{"osd":6,"uuid":"dbc20439-144e-4ec7-89de-674eb783f263","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.162:6808","nonce":2287343806}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.162:6809","nonce":2287343806}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.162:6811","nonce":2287343806}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.162:6810","nonce":2287343806}]},"public_addr":"172.21.15.162:6808/2287343806","cluster_addr":"172.21.15.162:6809/2287343806","heartbeat_back_addr":"172.21.15.162:6811/2287343806","heartbeat_front_addr":"172.21.15.162:6810/2287343806","state":["exists","up"]},{"osd":7,"uuid":"60aa5206-cace-40f8-8d1a-10ec28d97096","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.177:6813","nonce":2110809906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.177:6814","nonce":2110809906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.177:6816","nonce":2110809906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.177:6815","nonce":2110809906}]},"public_addr":"172.21.15.177:6813/2110809906","cluster_addr":"172.21.15.177:6814/2110809906","heartbeat_back_addr":"172.21.15.177:6816/2110809906","heartbeat_front_addr":"172.21.15.177:6815/2110809906","state":["exists","up"]},{"osd":8,"uuid":"f3d99325-48ff-403c-8662-1d7745f7c97d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6804","nonce":2766567910}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6805","nonce":2766567910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6807","nonce":2766567910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6806","nonce":2766567910}]},"public_addr":"172.21.15.1:6804/2766567910","cluster_addr":"172.21.15.1:6805/2766567910","heartbeat_back_addr":"172.21.15.1:6807/2766567910","heartbeat_front_addr":"172.21.15.1:6806/2766567910","state":["exists","up"]},{"osd":9,"uuid":"88511e24-3675-4551-968c-1a52e7e28f96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.92:6812","nonce":2488187511}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.92:6813","nonce":2488187511}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.92:6815","nonce":2488187511}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.92:6814","nonce":2488187511}]},"public_addr":"172.21.15.92:6812/2488187511","cluster_addr":"172.21.15.92:6813/2488187511","heartbeat_back_addr":"172.21.15.92:6815/2488187511","heartbeat_front_addr":"172.21.15.92:6814/2488187511","state":["exists","up"]},{"osd":10,"uuid":"acf4c64d-99af-42be-8406-413c86f54670","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.162:6804","nonce":1450266459}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.162:6805","nonce":1450266459}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.162:6807","nonce":1450266459}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.162:6806","nonce":1450266459}]},"public_addr":"172.21.15.162:6804/1450266459","cluster_addr":"172.21.15.162:6805/1450266459","heartbeat_back_addr":"172.21.15.162:6807/1450266459","heartbeat_front_addr":"172.21.15.162:6806/1450266459","state":["exists","up"]},{"osd":11,"uuid":"a4bfdfbe-b586-4080-8e09-6f26428e59b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.177:6809","nonce":3239521954}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.177:6810","nonce":3239521954}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.177:6812","nonce":3239521954}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.177:6811","nonce":3239521954}]},"public_addr":"172.21.15.177:6809/3239521954","cluster_addr":"172.21.15.177:6810/3239521954","heartbeat_back_addr":"172.21.15.177:6812/3239521954","heartbeat_front_addr":"172.21.15.177:6811/3239521954","state":["exists","up"]},{"osd":12,"uuid":"dc571fb6-d8b5-465b-bb25-afa7d0c47b5a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6812","nonce":4267062742}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6813","nonce":4267062742}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6815","nonce":4267062742}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6814","nonce":4267062742}]},"public_addr":"172.21.15.1:6812/4267062742","cluster_addr":"172.21.15.1:6813/4267062742","heartbeat_back_addr":"172.21.15.1:6815/4267062742","heartbeat_front_addr":"172.21.15.1:6814/4267062742","state":["exists","up"]},{"osd":13,"uuid":"5d321d4b-1b26-47fe-bf84-25751b8cde7d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.92:6808","nonce":1759585046}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.92:6809","nonce":1759585046}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.92:6811","nonce":1759585046}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.92:6810","nonce":1759585046}]},"public_addr":"172.21.15.92:6808/1759585046","cluster_addr":"172.21.15.92:6809/1759585046","heartbeat_back_addr":"172.21.15.92:6811/1759585046","heartbeat_front_addr":"172.21.15.92:6810/1759585046","state":["exists","up"]},{"osd":14,"uuid":"fc3e2499-0c2a-414d-a25e-2628876ae630","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.162:6800","nonce":3905144320}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.162:6801","nonce":3905144320}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.162:6803","nonce":3905144320}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.162:6802","nonce":3905144320}]},"public_addr":"172.21.15.162:6800/3905144320","cluster_addr":"172.21.15.162:6801/3905144320","heartbeat_back_addr":"172.21.15.162:6803/3905144320","heartbeat_front_addr":"172.21.15.162:6802/3905144320","state":["exists","up"]},{"osd":15,"uuid":"d2f69775-edfe-4527-91ec-f7c7e558bec8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.177:6805","nonce":1294337172}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.177:6806","nonce":1294337172}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.177:6808","nonce":1294337172}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.177:6807","nonce":1294337172}]},"public_addr":"172.21.15.177:6805/1294337172","cluster_addr":"172.21.15.177:6806/1294337172","heartbeat_back_addr":"172.21.15.177:6808/1294337172","heartbeat_front_addr":"172.21.15.177:6807/1294337172","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T11:22:08.233274+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T11:22:08.875236+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T11:22:07.936983+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T11:22:08.625189+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T11:22:07.944287+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T11:22:08.351045+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T11:22:08.835090+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T11:22:08.847356+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T11:22:08.218416+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T11:22:08.223142+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T11:22:08.687866+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T11:22:08.223685+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T11:22:08.170891+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T11:22:11.913 DEBUG:teuthology.misc:16 of 16 OSDs are up 2024-09-13T11:22:11.913 INFO:tasks.ceph:Creating RBD pool 2024-09-13T11:22:11.913 DEBUG:teuthology.orchestra.run.smithi001:> sudo ceph --cluster ceph osd pool create rbd 8 2024-09-13T11:22:12.115 INFO:tasks.ceph.osd.4.smithi001.stderr:2024-09-13T11:22:12.109+0000 7ff8d2c41640 -1 osd.4 22 handle_osd_map injecting map crc failure 2024-09-13T11:22:12.826 INFO:tasks.ceph.osd.14.smithi162.stderr:2024-09-13T11:22:12.821+0000 7f2350b08640 -1 osd.14 22 handle_osd_map injecting map crc failure 2024-09-13T11:22:12.878 INFO:tasks.ceph.osd.7.smithi177.stderr:2024-09-13T11:22:12.878+0000 7f896ff4a640 -1 osd.7 22 handle_osd_map injecting map crc failure 2024-09-13T11:22:13.023 INFO:tasks.ceph.osd.3.smithi177.stderr:2024-09-13T11:22:13.022+0000 7f10a25d1640 -1 osd.3 21 handle_osd_map injecting map crc failure 2024-09-13T11:22:13.117 INFO:teuthology.orchestra.run.smithi001.stderr:pool 'rbd' created 2024-09-13T11:22:13.143 DEBUG:teuthology.orchestra.run.smithi001:> rbd --cluster ceph pool init rbd 2024-09-13T11:22:13.392 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-13T11:22:13.389+0000 7f6a5d1a2640 -1 osd.0 24 handle_osd_map injecting map crc failure 2024-09-13T11:22:13.601 INFO:tasks.ceph.osd.5.smithi092.stderr:2024-09-13T11:22:13.597+0000 7f6f00132640 -1 osd.5 21 handle_osd_map injecting map crc failure 2024-09-13T11:22:14.242 INFO:tasks.ceph.osd.7.smithi177.stderr:2024-09-13T11:22:14.242+0000 7f896ff4a640 -1 osd.7 25 handle_osd_map injecting map crc failure 2024-09-13T11:22:15.232 INFO:tasks.ceph.osd.8.smithi001.stderr:2024-09-13T11:22:15.229+0000 7fb9ab2df640 -1 osd.8 26 handle_osd_map injecting map crc failure 2024-09-13T11:22:15.612 INFO:tasks.ceph.osd.5.smithi092.stderr:2024-09-13T11:22:15.613+0000 7f6f00132640 -1 osd.5 26 handle_osd_map injecting map crc failure 2024-09-13T11:22:15.775 INFO:tasks.ceph.osd.12.smithi001.stderr:2024-09-13T11:22:15.773+0000 7f891d10b640 -1 osd.12 26 handle_osd_map injecting map crc failure 2024-09-13T11:22:15.935 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-13T11:22:15.929+0000 7f6a5d1a2640 -1 osd.0 26 handle_osd_map injecting map crc failure 2024-09-13T11:22:16.259 INFO:tasks.ceph.osd.3.smithi177.stderr:2024-09-13T11:22:16.257+0000 7f10a45d5640 -1 osd.3 27 handle_osd_map injecting map crc failure 2024-09-13T11:22:16.279 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-09-13T11:22:16.280 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config log 1 --format=json 2024-09-13T11:22:16.281 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-09-13T11:22:16.607 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-13T11:22:16.626 INFO:teuthology.orchestra.run.smithi001.stdout:[{"version":7,"timestamp":"2024-09-13T11:22:10.767877+0000","name":"","changes":[{"name":"osd.9/osd_mclock_max_capacity_iops_ssd","new_value":"5074.872418"}]}] 2024-09-13T11:22:16.626 INFO:tasks.ceph_manager:config epoch is 7 2024-09-13T11:22:16.626 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-09-13T11:22:16.627 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-09-13T11:22:16.627 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-09-13T11:22:16.881 INFO:tasks.ceph.osd.15.smithi177.stderr:2024-09-13T11:22:16.877+0000 7f9d618a4640 -1 osd.15 27 handle_osd_map injecting map crc failure 2024-09-13T11:22:16.997 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-13T11:22:17.016 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":5,"flags":0,"active_gid":4104,"active_name":"x","active_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.177:6800","nonce":503890672}]},"active_addr":"172.21.15.177:6800/503890672","active_change":"2024-09-13T11:22:02.101341+0000","active_mgr_features":4540719139924082687,"available":true,"standbys":[{"gid":4107,"name":"y","mgr_features":4540719139924082687,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to, use commas to separate multiple","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.27.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:10.4.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_nginx":{"name":"container_image_nginx","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nginx:1.26.1","min":"","max":"","enum_allowed":[],"desc":"Nginx container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.7.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.17","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_oauth2_proxy":{"name":"container_image_oauth2_proxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/oauth2-proxy/oauth2-proxy:v7.6.0","min":"","max":"","enum_allowed":[],"desc":"oauth2-proxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.51.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_requests":{"name":"max_requests","type":"int","level":"advanced","flags":0,"default_value":"500","min":"","max":"","enum_allowed":[],"desc":"Maximum number of requests to keep in memory. When new request comes in, the oldest request will be removed if the number of requests exceeds the max request number.if un-finished request is removed, error message will be logged in the ceph-mgr log.","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"secondary_zone_period_retry_limit":{"name":"secondary_zone_period_retry_limit","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"RGW module period update retry limit for secondary site","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to, use commas to separate multiple","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.27.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:10.4.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_nginx":{"name":"container_image_nginx","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nginx:1.26.1","min":"","max":"","enum_allowed":[],"desc":"Nginx container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.7.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.17","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_oauth2_proxy":{"name":"container_image_oauth2_proxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/oauth2-proxy/oauth2-proxy:v7.6.0","min":"","max":"","enum_allowed":[],"desc":"oauth2-proxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.51.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_requests":{"name":"max_requests","type":"int","level":"advanced","flags":0,"default_value":"500","min":"","max":"","enum_allowed":[],"desc":"Maximum number of requests to keep in memory. When new request comes in, the oldest request will be removed if the number of requests exceeds the max request number.if un-finished request is removed, error message will be logged in the ceph-mgr log.","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"secondary_zone_period_retry_limit":{"name":"secondary_zone_period_retry_limit","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"RGW module period update retry limit for secondary site","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.177:0","nonce":4170554831}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.177:0","nonce":1488131527}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.177:0","nonce":1195511430}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.177:0","nonce":2388565149}]}]} 2024-09-13T11:22:17.021 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-09-13T11:22:17.021 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-09-13T11:22:17.022 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T11:22:17.343 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-13T11:22:17.343 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":28,"fsid":"2fd68ac7-5581-4d90-80a4-30bcf5252a7a","created":"2024-09-13T11:21:57.223153+0000","modified":"2024-09-13T11:22:16.231520+0000","last_up_change":"2024-09-13T11:22:11.098388+0000","last_in_change":"2024-09-13T11:22:01.745814+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T11:22:10.162266+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":15.380000114440918,"score_stable":15.380000114440918,"optimal_score":0.12999999523162842,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-13T11:22:12.235549+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"28","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":28,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4.0199999809265137,"score_stable":4.0199999809265137,"optimal_score":0.56000000238418579,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"0fc12b3c-e0de-4546-b935-db77efe7c567","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6800","nonce":1544776477}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6801","nonce":1544776477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6803","nonce":1544776477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6802","nonce":1544776477}]},"public_addr":"172.21.15.1:6800/1544776477","cluster_addr":"172.21.15.1:6801/1544776477","heartbeat_back_addr":"172.21.15.1:6803/1544776477","heartbeat_front_addr":"172.21.15.1:6802/1544776477","state":["exists","up"]},{"osd":1,"uuid":"b77fe331-cc92-4adb-b146-af5c380f6d4f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.92:6800","nonce":356319619}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.92:6801","nonce":356319619}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.92:6803","nonce":356319619}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.92:6802","nonce":356319619}]},"public_addr":"172.21.15.92:6800/356319619","cluster_addr":"172.21.15.92:6801/356319619","heartbeat_back_addr":"172.21.15.92:6803/356319619","heartbeat_front_addr":"172.21.15.92:6802/356319619","state":["exists","up"]},{"osd":2,"uuid":"c9d031c8-e885-4229-affa-08e1729d00e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.162:6812","nonce":2712148101}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.162:6813","nonce":2712148101}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.162:6815","nonce":2712148101}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.162:6814","nonce":2712148101}]},"public_addr":"172.21.15.162:6812/2712148101","cluster_addr":"172.21.15.162:6813/2712148101","heartbeat_back_addr":"172.21.15.162:6815/2712148101","heartbeat_front_addr":"172.21.15.162:6814/2712148101","state":["exists","up"]},{"osd":3,"uuid":"bf50d71a-6c3a-4fbc-a407-cb81bad14a7b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.177:6801","nonce":3367751690}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.177:6802","nonce":3367751690}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.177:6804","nonce":3367751690}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.177:6803","nonce":3367751690}]},"public_addr":"172.21.15.177:6801/3367751690","cluster_addr":"172.21.15.177:6802/3367751690","heartbeat_back_addr":"172.21.15.177:6804/3367751690","heartbeat_front_addr":"172.21.15.177:6803/3367751690","state":["exists","up"]},{"osd":4,"uuid":"d1fe3f63-df3f-4a73-a1e2-8d7c8a3f1fa0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6808","nonce":3648540926}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6809","nonce":3648540926}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6811","nonce":3648540926}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6810","nonce":3648540926}]},"public_addr":"172.21.15.1:6808/3648540926","cluster_addr":"172.21.15.1:6809/3648540926","heartbeat_back_addr":"172.21.15.1:6811/3648540926","heartbeat_front_addr":"172.21.15.1:6810/3648540926","state":["exists","up"]},{"osd":5,"uuid":"e2ead447-2f2c-40c9-82e7-53b8dc41e2af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.92:6804","nonce":3399529091}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.92:6805","nonce":3399529091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.92:6807","nonce":3399529091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.92:6806","nonce":3399529091}]},"public_addr":"172.21.15.92:6804/3399529091","cluster_addr":"172.21.15.92:6805/3399529091","heartbeat_back_addr":"172.21.15.92:6807/3399529091","heartbeat_front_addr":"172.21.15.92:6806/3399529091","state":["exists","up"]},{"osd":6,"uuid":"dbc20439-144e-4ec7-89de-674eb783f263","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.162:6808","nonce":2287343806}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.162:6809","nonce":2287343806}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.162:6811","nonce":2287343806}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.162:6810","nonce":2287343806}]},"public_addr":"172.21.15.162:6808/2287343806","cluster_addr":"172.21.15.162:6809/2287343806","heartbeat_back_addr":"172.21.15.162:6811/2287343806","heartbeat_front_addr":"172.21.15.162:6810/2287343806","state":["exists","up"]},{"osd":7,"uuid":"60aa5206-cace-40f8-8d1a-10ec28d97096","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.177:6813","nonce":2110809906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.177:6814","nonce":2110809906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.177:6816","nonce":2110809906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.177:6815","nonce":2110809906}]},"public_addr":"172.21.15.177:6813/2110809906","cluster_addr":"172.21.15.177:6814/2110809906","heartbeat_back_addr":"172.21.15.177:6816/2110809906","heartbeat_front_addr":"172.21.15.177:6815/2110809906","state":["exists","up"]},{"osd":8,"uuid":"f3d99325-48ff-403c-8662-1d7745f7c97d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6804","nonce":2766567910}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6805","nonce":2766567910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6807","nonce":2766567910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6806","nonce":2766567910}]},"public_addr":"172.21.15.1:6804/2766567910","cluster_addr":"172.21.15.1:6805/2766567910","heartbeat_back_addr":"172.21.15.1:6807/2766567910","heartbeat_front_addr":"172.21.15.1:6806/2766567910","state":["exists","up"]},{"osd":9,"uuid":"88511e24-3675-4551-968c-1a52e7e28f96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.92:6812","nonce":2488187511}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.92:6813","nonce":2488187511}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.92:6815","nonce":2488187511}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.92:6814","nonce":2488187511}]},"public_addr":"172.21.15.92:6812/2488187511","cluster_addr":"172.21.15.92:6813/2488187511","heartbeat_back_addr":"172.21.15.92:6815/2488187511","heartbeat_front_addr":"172.21.15.92:6814/2488187511","state":["exists","up"]},{"osd":10,"uuid":"acf4c64d-99af-42be-8406-413c86f54670","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.162:6804","nonce":1450266459}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.162:6805","nonce":1450266459}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.162:6807","nonce":1450266459}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.162:6806","nonce":1450266459}]},"public_addr":"172.21.15.162:6804/1450266459","cluster_addr":"172.21.15.162:6805/1450266459","heartbeat_back_addr":"172.21.15.162:6807/1450266459","heartbeat_front_addr":"172.21.15.162:6806/1450266459","state":["exists","up"]},{"osd":11,"uuid":"a4bfdfbe-b586-4080-8e09-6f26428e59b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.177:6809","nonce":3239521954}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.177:6810","nonce":3239521954}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.177:6812","nonce":3239521954}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.177:6811","nonce":3239521954}]},"public_addr":"172.21.15.177:6809/3239521954","cluster_addr":"172.21.15.177:6810/3239521954","heartbeat_back_addr":"172.21.15.177:6812/3239521954","heartbeat_front_addr":"172.21.15.177:6811/3239521954","state":["exists","up"]},{"osd":12,"uuid":"dc571fb6-d8b5-465b-bb25-afa7d0c47b5a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6812","nonce":4267062742}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6813","nonce":4267062742}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6815","nonce":4267062742}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6814","nonce":4267062742}]},"public_addr":"172.21.15.1:6812/4267062742","cluster_addr":"172.21.15.1:6813/4267062742","heartbeat_back_addr":"172.21.15.1:6815/4267062742","heartbeat_front_addr":"172.21.15.1:6814/4267062742","state":["exists","up"]},{"osd":13,"uuid":"5d321d4b-1b26-47fe-bf84-25751b8cde7d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.92:6808","nonce":1759585046}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.92:6809","nonce":1759585046}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.92:6811","nonce":1759585046}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.92:6810","nonce":1759585046}]},"public_addr":"172.21.15.92:6808/1759585046","cluster_addr":"172.21.15.92:6809/1759585046","heartbeat_back_addr":"172.21.15.92:6811/1759585046","heartbeat_front_addr":"172.21.15.92:6810/1759585046","state":["exists","up"]},{"osd":14,"uuid":"fc3e2499-0c2a-414d-a25e-2628876ae630","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.162:6800","nonce":3905144320}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.162:6801","nonce":3905144320}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.162:6803","nonce":3905144320}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.162:6802","nonce":3905144320}]},"public_addr":"172.21.15.162:6800/3905144320","cluster_addr":"172.21.15.162:6801/3905144320","heartbeat_back_addr":"172.21.15.162:6803/3905144320","heartbeat_front_addr":"172.21.15.162:6802/3905144320","state":["exists","up"]},{"osd":15,"uuid":"d2f69775-edfe-4527-91ec-f7c7e558bec8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.177:6805","nonce":1294337172}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.177:6806","nonce":1294337172}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.177:6808","nonce":1294337172}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.177:6807","nonce":1294337172}]},"public_addr":"172.21.15.177:6805/1294337172","cluster_addr":"172.21.15.177:6806/1294337172","heartbeat_back_addr":"172.21.15.177:6808/1294337172","heartbeat_front_addr":"172.21.15.177:6807/1294337172","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T11:22:08.233274+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T11:22:08.032163+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T11:22:08.875236+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T11:22:07.936983+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T11:22:08.625189+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T11:22:07.944287+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T11:22:08.351045+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T11:22:08.835090+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T11:22:08.847356+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T11:22:08.140353+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T11:22:08.218416+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T11:22:08.223142+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T11:22:08.687866+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T11:22:08.347174+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T11:22:08.223685+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T11:22:08.170891+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T11:22:17.360 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-09-13T11:22:17.361 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T11:22:17.662 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-13T11:22:17.662 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":28,"fsid":"2fd68ac7-5581-4d90-80a4-30bcf5252a7a","created":"2024-09-13T11:21:57.223153+0000","modified":"2024-09-13T11:22:16.231520+0000","last_up_change":"2024-09-13T11:22:11.098388+0000","last_in_change":"2024-09-13T11:22:01.745814+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T11:22:10.162266+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":15.380000114440918,"score_stable":15.380000114440918,"optimal_score":0.12999999523162842,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-13T11:22:12.235549+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"28","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":28,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4.0199999809265137,"score_stable":4.0199999809265137,"optimal_score":0.56000000238418579,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"0fc12b3c-e0de-4546-b935-db77efe7c567","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6800","nonce":1544776477}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6801","nonce":1544776477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6803","nonce":1544776477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6802","nonce":1544776477}]},"public_addr":"172.21.15.1:6800/1544776477","cluster_addr":"172.21.15.1:6801/1544776477","heartbeat_back_addr":"172.21.15.1:6803/1544776477","heartbeat_front_addr":"172.21.15.1:6802/1544776477","state":["exists","up"]},{"osd":1,"uuid":"b77fe331-cc92-4adb-b146-af5c380f6d4f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.92:6800","nonce":356319619}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.92:6801","nonce":356319619}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.92:6803","nonce":356319619}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.92:6802","nonce":356319619}]},"public_addr":"172.21.15.92:6800/356319619","cluster_addr":"172.21.15.92:6801/356319619","heartbeat_back_addr":"172.21.15.92:6803/356319619","heartbeat_front_addr":"172.21.15.92:6802/356319619","state":["exists","up"]},{"osd":2,"uuid":"c9d031c8-e885-4229-affa-08e1729d00e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.162:6812","nonce":2712148101}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.162:6813","nonce":2712148101}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.162:6815","nonce":2712148101}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.162:6814","nonce":2712148101}]},"public_addr":"172.21.15.162:6812/2712148101","cluster_addr":"172.21.15.162:6813/2712148101","heartbeat_back_addr":"172.21.15.162:6815/2712148101","heartbeat_front_addr":"172.21.15.162:6814/2712148101","state":["exists","up"]},{"osd":3,"uuid":"bf50d71a-6c3a-4fbc-a407-cb81bad14a7b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.177:6801","nonce":3367751690}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.177:6802","nonce":3367751690}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.177:6804","nonce":3367751690}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.177:6803","nonce":3367751690}]},"public_addr":"172.21.15.177:6801/3367751690","cluster_addr":"172.21.15.177:6802/3367751690","heartbeat_back_addr":"172.21.15.177:6804/3367751690","heartbeat_front_addr":"172.21.15.177:6803/3367751690","state":["exists","up"]},{"osd":4,"uuid":"d1fe3f63-df3f-4a73-a1e2-8d7c8a3f1fa0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6808","nonce":3648540926}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6809","nonce":3648540926}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6811","nonce":3648540926}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6810","nonce":3648540926}]},"public_addr":"172.21.15.1:6808/3648540926","cluster_addr":"172.21.15.1:6809/3648540926","heartbeat_back_addr":"172.21.15.1:6811/3648540926","heartbeat_front_addr":"172.21.15.1:6810/3648540926","state":["exists","up"]},{"osd":5,"uuid":"e2ead447-2f2c-40c9-82e7-53b8dc41e2af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.92:6804","nonce":3399529091}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.92:6805","nonce":3399529091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.92:6807","nonce":3399529091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.92:6806","nonce":3399529091}]},"public_addr":"172.21.15.92:6804/3399529091","cluster_addr":"172.21.15.92:6805/3399529091","heartbeat_back_addr":"172.21.15.92:6807/3399529091","heartbeat_front_addr":"172.21.15.92:6806/3399529091","state":["exists","up"]},{"osd":6,"uuid":"dbc20439-144e-4ec7-89de-674eb783f263","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.162:6808","nonce":2287343806}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.162:6809","nonce":2287343806}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.162:6811","nonce":2287343806}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.162:6810","nonce":2287343806}]},"public_addr":"172.21.15.162:6808/2287343806","cluster_addr":"172.21.15.162:6809/2287343806","heartbeat_back_addr":"172.21.15.162:6811/2287343806","heartbeat_front_addr":"172.21.15.162:6810/2287343806","state":["exists","up"]},{"osd":7,"uuid":"60aa5206-cace-40f8-8d1a-10ec28d97096","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.177:6813","nonce":2110809906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.177:6814","nonce":2110809906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.177:6816","nonce":2110809906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.177:6815","nonce":2110809906}]},"public_addr":"172.21.15.177:6813/2110809906","cluster_addr":"172.21.15.177:6814/2110809906","heartbeat_back_addr":"172.21.15.177:6816/2110809906","heartbeat_front_addr":"172.21.15.177:6815/2110809906","state":["exists","up"]},{"osd":8,"uuid":"f3d99325-48ff-403c-8662-1d7745f7c97d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6804","nonce":2766567910}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6805","nonce":2766567910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6807","nonce":2766567910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6806","nonce":2766567910}]},"public_addr":"172.21.15.1:6804/2766567910","cluster_addr":"172.21.15.1:6805/2766567910","heartbeat_back_addr":"172.21.15.1:6807/2766567910","heartbeat_front_addr":"172.21.15.1:6806/2766567910","state":["exists","up"]},{"osd":9,"uuid":"88511e24-3675-4551-968c-1a52e7e28f96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.92:6812","nonce":2488187511}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.92:6813","nonce":2488187511}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.92:6815","nonce":2488187511}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.92:6814","nonce":2488187511}]},"public_addr":"172.21.15.92:6812/2488187511","cluster_addr":"172.21.15.92:6813/2488187511","heartbeat_back_addr":"172.21.15.92:6815/2488187511","heartbeat_front_addr":"172.21.15.92:6814/2488187511","state":["exists","up"]},{"osd":10,"uuid":"acf4c64d-99af-42be-8406-413c86f54670","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.162:6804","nonce":1450266459}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.162:6805","nonce":1450266459}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.162:6807","nonce":1450266459}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.162:6806","nonce":1450266459}]},"public_addr":"172.21.15.162:6804/1450266459","cluster_addr":"172.21.15.162:6805/1450266459","heartbeat_back_addr":"172.21.15.162:6807/1450266459","heartbeat_front_addr":"172.21.15.162:6806/1450266459","state":["exists","up"]},{"osd":11,"uuid":"a4bfdfbe-b586-4080-8e09-6f26428e59b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.177:6809","nonce":3239521954}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.177:6810","nonce":3239521954}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.177:6812","nonce":3239521954}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.177:6811","nonce":3239521954}]},"public_addr":"172.21.15.177:6809/3239521954","cluster_addr":"172.21.15.177:6810/3239521954","heartbeat_back_addr":"172.21.15.177:6812/3239521954","heartbeat_front_addr":"172.21.15.177:6811/3239521954","state":["exists","up"]},{"osd":12,"uuid":"dc571fb6-d8b5-465b-bb25-afa7d0c47b5a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6812","nonce":4267062742}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6813","nonce":4267062742}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6815","nonce":4267062742}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6814","nonce":4267062742}]},"public_addr":"172.21.15.1:6812/4267062742","cluster_addr":"172.21.15.1:6813/4267062742","heartbeat_back_addr":"172.21.15.1:6815/4267062742","heartbeat_front_addr":"172.21.15.1:6814/4267062742","state":["exists","up"]},{"osd":13,"uuid":"5d321d4b-1b26-47fe-bf84-25751b8cde7d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.92:6808","nonce":1759585046}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.92:6809","nonce":1759585046}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.92:6811","nonce":1759585046}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.92:6810","nonce":1759585046}]},"public_addr":"172.21.15.92:6808/1759585046","cluster_addr":"172.21.15.92:6809/1759585046","heartbeat_back_addr":"172.21.15.92:6811/1759585046","heartbeat_front_addr":"172.21.15.92:6810/1759585046","state":["exists","up"]},{"osd":14,"uuid":"fc3e2499-0c2a-414d-a25e-2628876ae630","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.162:6800","nonce":3905144320}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.162:6801","nonce":3905144320}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.162:6803","nonce":3905144320}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.162:6802","nonce":3905144320}]},"public_addr":"172.21.15.162:6800/3905144320","cluster_addr":"172.21.15.162:6801/3905144320","heartbeat_back_addr":"172.21.15.162:6803/3905144320","heartbeat_front_addr":"172.21.15.162:6802/3905144320","state":["exists","up"]},{"osd":15,"uuid":"d2f69775-edfe-4527-91ec-f7c7e558bec8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.177:6805","nonce":1294337172}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.177:6806","nonce":1294337172}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.177:6808","nonce":1294337172}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.177:6807","nonce":1294337172}]},"public_addr":"172.21.15.177:6805/1294337172","cluster_addr":"172.21.15.177:6806/1294337172","heartbeat_back_addr":"172.21.15.177:6808/1294337172","heartbeat_front_addr":"172.21.15.177:6807/1294337172","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T11:22:08.233274+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T11:22:08.032163+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T11:22:08.875236+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T11:22:07.936983+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T11:22:08.625189+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T11:22:07.944287+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T11:22:08.351045+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T11:22:08.835090+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T11:22:08.847356+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T11:22:08.140353+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T11:22:08.218416+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T11:22:08.223142+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T11:22:08.687866+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T11:22:08.347174+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T11:22:08.223685+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T11:22:08.170891+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T11:22:17.682 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-09-13T11:22:17.682 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-09-13T11:22:17.683 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-09-13T11:22:17.683 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-09-13T11:22:17.683 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2024-09-13T11:22:17.684 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2024-09-13T11:22:17.685 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2024-09-13T11:22:17.685 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2024-09-13T11:22:17.686 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.8 flush_pg_stats 2024-09-13T11:22:17.686 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.9 flush_pg_stats 2024-09-13T11:22:17.687 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.10 flush_pg_stats 2024-09-13T11:22:17.687 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.11 flush_pg_stats 2024-09-13T11:22:17.687 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.12 flush_pg_stats 2024-09-13T11:22:17.688 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.13 flush_pg_stats 2024-09-13T11:22:17.688 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.14 flush_pg_stats 2024-09-13T11:22:17.689 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.15 flush_pg_stats 2024-09-13T11:22:18.113 INFO:teuthology.orchestra.run.smithi001.stdout:94489280516 2024-09-13T11:22:18.113 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-09-13T11:22:18.136 INFO:teuthology.orchestra.run.smithi001.stdout:94489280516 2024-09-13T11:22:18.136 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-13T11:22:18.141 INFO:teuthology.orchestra.run.smithi001.stdout:90194313222 2024-09-13T11:22:18.142 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-09-13T11:22:18.157 INFO:teuthology.orchestra.run.smithi001.stdout:98784247811 2024-09-13T11:22:18.158 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-13T11:22:18.167 INFO:teuthology.orchestra.run.smithi001.stdout:94489280517 2024-09-13T11:22:18.167 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.11 2024-09-13T11:22:18.203 INFO:teuthology.orchestra.run.smithi001.stdout:94489280516 2024-09-13T11:22:18.203 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-09-13T11:22:18.217 INFO:teuthology.orchestra.run.smithi001.stdout:98784247812 2024-09-13T11:22:18.218 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.9 2024-09-13T11:22:18.224 INFO:teuthology.orchestra.run.smithi001.stdout:94489280515 2024-09-13T11:22:18.225 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-13T11:22:18.226 INFO:teuthology.orchestra.run.smithi001.stdout:98784247813 2024-09-13T11:22:18.226 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.13 2024-09-13T11:22:18.228 INFO:teuthology.orchestra.run.smithi001.stdout:90194313220 2024-09-13T11:22:18.228 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-13T11:22:18.240 INFO:teuthology.orchestra.run.smithi001.stdout:94489280516 2024-09-13T11:22:18.240 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.15 2024-09-13T11:22:18.249 INFO:teuthology.orchestra.run.smithi001.stdout:94489280516 2024-09-13T11:22:18.249 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.12 2024-09-13T11:22:18.255 INFO:teuthology.orchestra.run.smithi001.stdout:94489280516 2024-09-13T11:22:18.255 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.8 2024-09-13T11:22:18.261 INFO:teuthology.orchestra.run.smithi001.stdout:94489280515 2024-09-13T11:22:18.262 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.10 2024-09-13T11:22:18.273 INFO:teuthology.orchestra.run.smithi001.stdout:94489280516 2024-09-13T11:22:18.274 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.14 2024-09-13T11:22:18.319 INFO:teuthology.orchestra.run.smithi001.stdout:94489280516 2024-09-13T11:22:18.319 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-09-13T11:22:18.911 INFO:teuthology.orchestra.run.smithi001.stdout:94489280516 2024-09-13T11:22:18.958 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280516 got 94489280516 for osd.4 2024-09-13T11:22:18.959 DEBUG:teuthology.parallel:result is None 2024-09-13T11:22:19.099 INFO:teuthology.orchestra.run.smithi001.stdout:98784247811 2024-09-13T11:22:19.109 INFO:teuthology.orchestra.run.smithi001.stdout:94489280515 2024-09-13T11:22:19.143 INFO:tasks.ceph.ceph_manager.ceph:need seq 98784247811 got 98784247811 for osd.1 2024-09-13T11:22:19.143 DEBUG:teuthology.parallel:result is None 2024-09-13T11:22:19.160 INFO:teuthology.orchestra.run.smithi001.stdout:90194313222 2024-09-13T11:22:19.165 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280516 got 94489280515 for osd.14 2024-09-13T11:22:19.190 INFO:teuthology.orchestra.run.smithi001.stdout:98784247812 2024-09-13T11:22:19.198 INFO:teuthology.orchestra.run.smithi001.stdout:94489280515 2024-09-13T11:22:19.218 INFO:tasks.ceph.ceph_manager.ceph:need seq 90194313222 got 90194313222 for osd.5 2024-09-13T11:22:19.218 DEBUG:teuthology.parallel:result is None 2024-09-13T11:22:19.229 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280516 got 94489280515 for osd.15 2024-09-13T11:22:19.238 INFO:tasks.ceph.ceph_manager.ceph:need seq 98784247813 got 98784247812 for osd.13 2024-09-13T11:22:19.274 INFO:teuthology.orchestra.run.smithi001.stdout:94489280515 2024-09-13T11:22:19.287 INFO:teuthology.orchestra.run.smithi001.stdout:94489280516 2024-09-13T11:22:19.316 INFO:teuthology.orchestra.run.smithi001.stdout:98784247811 2024-09-13T11:22:19.318 INFO:teuthology.orchestra.run.smithi001.stdout:90194313219 2024-09-13T11:22:19.319 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280516 got 94489280515 for osd.6 2024-09-13T11:22:19.328 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280516 got 94489280516 for osd.2 2024-09-13T11:22:19.328 DEBUG:teuthology.parallel:result is None 2024-09-13T11:22:19.329 INFO:teuthology.orchestra.run.smithi001.stdout:94489280515 2024-09-13T11:22:19.350 INFO:teuthology.orchestra.run.smithi001.stdout:94489280515 2024-09-13T11:22:19.351 INFO:tasks.ceph.ceph_manager.ceph:need seq 90194313220 got 90194313219 for osd.3 2024-09-13T11:22:19.355 INFO:tasks.ceph.ceph_manager.ceph:need seq 98784247812 got 98784247811 for osd.9 2024-09-13T11:22:19.356 INFO:teuthology.orchestra.run.smithi001.stdout:94489280514 2024-09-13T11:22:19.357 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280516 got 94489280515 for osd.12 2024-09-13T11:22:19.367 INFO:teuthology.orchestra.run.smithi001.stdout:94489280514 2024-09-13T11:22:19.370 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280516 got 94489280515 for osd.8 2024-09-13T11:22:19.384 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280515 got 94489280514 for osd.0 2024-09-13T11:22:19.385 INFO:teuthology.orchestra.run.smithi001.stdout:94489280517 2024-09-13T11:22:19.392 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280515 got 94489280514 for osd.10 2024-09-13T11:22:19.402 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280517 got 94489280517 for osd.11 2024-09-13T11:22:19.402 DEBUG:teuthology.parallel:result is None 2024-09-13T11:22:19.415 INFO:teuthology.orchestra.run.smithi001.stdout:94489280515 2024-09-13T11:22:19.431 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280516 got 94489280515 for osd.7 2024-09-13T11:22:20.166 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.14 2024-09-13T11:22:20.230 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.15 2024-09-13T11:22:20.239 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.13 2024-09-13T11:22:20.320 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-09-13T11:22:20.351 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-13T11:22:20.356 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.9 2024-09-13T11:22:20.358 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.12 2024-09-13T11:22:20.370 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.8 2024-09-13T11:22:20.386 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-13T11:22:20.393 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.10 2024-09-13T11:22:20.432 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-09-13T11:22:20.746 INFO:teuthology.orchestra.run.smithi001.stdout:94489280516 2024-09-13T11:22:20.791 INFO:teuthology.orchestra.run.smithi001.stdout:94489280516 2024-09-13T11:22:20.814 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280516 got 94489280516 for osd.14 2024-09-13T11:22:20.814 DEBUG:teuthology.parallel:result is None 2024-09-13T11:22:20.845 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280516 got 94489280516 for osd.15 2024-09-13T11:22:20.846 DEBUG:teuthology.parallel:result is None 2024-09-13T11:22:20.900 INFO:teuthology.orchestra.run.smithi001.stdout:98784247813 2024-09-13T11:22:20.949 INFO:tasks.ceph.ceph_manager.ceph:need seq 98784247813 got 98784247813 for osd.13 2024-09-13T11:22:20.950 DEBUG:teuthology.parallel:result is None 2024-09-13T11:22:20.988 INFO:teuthology.orchestra.run.smithi001.stdout:94489280516 2024-09-13T11:22:20.996 INFO:teuthology.orchestra.run.smithi001.stdout:98784247812 2024-09-13T11:22:21.004 INFO:teuthology.orchestra.run.smithi001.stdout:94489280515 2024-09-13T11:22:21.017 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280516 got 94489280516 for osd.6 2024-09-13T11:22:21.017 DEBUG:teuthology.parallel:result is None 2024-09-13T11:22:21.024 INFO:teuthology.orchestra.run.smithi001.stdout:94489280516 2024-09-13T11:22:21.027 INFO:tasks.ceph.ceph_manager.ceph:need seq 98784247812 got 98784247812 for osd.9 2024-09-13T11:22:21.027 DEBUG:teuthology.parallel:result is None 2024-09-13T11:22:21.028 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280515 got 94489280515 for osd.0 2024-09-13T11:22:21.028 DEBUG:teuthology.parallel:result is None 2024-09-13T11:22:21.049 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280516 got 94489280516 for osd.12 2024-09-13T11:22:21.049 DEBUG:teuthology.parallel:result is None 2024-09-13T11:22:21.049 INFO:teuthology.orchestra.run.smithi001.stdout:90194313221 2024-09-13T11:22:21.056 INFO:teuthology.orchestra.run.smithi001.stdout:94489280515 2024-09-13T11:22:21.068 INFO:tasks.ceph.ceph_manager.ceph:need seq 90194313220 got 90194313221 for osd.3 2024-09-13T11:22:21.068 DEBUG:teuthology.parallel:result is None 2024-09-13T11:22:21.081 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280515 got 94489280515 for osd.10 2024-09-13T11:22:21.081 DEBUG:teuthology.parallel:result is None 2024-09-13T11:22:21.103 INFO:teuthology.orchestra.run.smithi001.stdout:94489280516 2024-09-13T11:22:21.110 INFO:teuthology.orchestra.run.smithi001.stdout:94489280516 2024-09-13T11:22:21.121 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280516 got 94489280516 for osd.8 2024-09-13T11:22:21.122 DEBUG:teuthology.parallel:result is None 2024-09-13T11:22:21.128 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280516 got 94489280516 for osd.7 2024-09-13T11:22:21.128 DEBUG:teuthology.parallel:result is None 2024-09-13T11:22:21.129 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-13T11:22:21.129 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-13T11:22:21.460 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-13T11:22:21.461 INFO:teuthology.orchestra.run.smithi001.stderr:dumped all 2024-09-13T11:22:21.477 INFO:teuthology.orchestra.run.smithi001.stdout:{"pg_ready":true,"pg_map":{"version":19,"stamp":"2024-09-13T11:22:20.114840+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":147,"num_write_kb":2674,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":115,"ondisk_log_size":115,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":17,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":10,"kb":1509949440,"kb_used":445380,"kb_used_data":4676,"kb_used_omap":24,"kb_used_meta":440423,"kb_avail":1509504060,"statfs":{"total":1546188226560,"available":1545732157440,"internally_reserved":0,"allocated":4788224,"data_stored":2695514,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":25519,"internal_metadata":450993233},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.001277"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":20,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-13T11:22:16.888721+0000","last_change":"2024-09-13T11:22:16.889130+0000","last_active":"2024-09-13T11:22:16.888721+0000","last_peered":"2024-09-13T11:22:16.888721+0000","last_clean":"2024-09-13T11:22:16.888721+0000","last_became_active":"2024-09-13T11:22:14.192897+0000","last_became_peered":"2024-09-13T11:22:14.192897+0000","last_unstale":"2024-09-13T11:22:16.888721+0000","last_undegraded":"2024-09-13T11:22:16.888721+0000","last_fullsized":"2024-09-13T11:22:16.888721+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:22:13.113563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:23:26.278353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00084655899999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,10],"acting":[15,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":20,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-13T11:22:16.260591+0000","last_change":"2024-09-13T11:22:16.260591+0000","last_active":"2024-09-13T11:22:16.260591+0000","last_peered":"2024-09-13T11:22:16.260591+0000","last_clean":"2024-09-13T11:22:16.260591+0000","last_became_active":"2024-09-13T11:22:14.211923+0000","last_became_peered":"2024-09-13T11:22:14.211923+0000","last_unstale":"2024-09-13T11:22:16.260591+0000","last_undegraded":"2024-09-13T11:22:16.260591+0000","last_fullsized":"2024-09-13T11:22:16.260591+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:22:13.113563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:23:20.320636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011689529999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0],"acting":[9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":20,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-13T11:22:16.503631+0000","last_change":"2024-09-13T11:22:16.504060+0000","last_active":"2024-09-13T11:22:16.503631+0000","last_peered":"2024-09-13T11:22:16.503631+0000","last_clean":"2024-09-13T11:22:16.503631+0000","last_became_active":"2024-09-13T11:22:14.581458+0000","last_became_peered":"2024-09-13T11:22:14.581458+0000","last_unstale":"2024-09-13T11:22:16.503631+0000","last_undegraded":"2024-09-13T11:22:16.503631+0000","last_fullsized":"2024-09-13T11:22:16.503631+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:22:13.113563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:23:37.935010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001026047,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5],"acting":[8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":20,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-13T11:22:16.931914+0000","last_change":"2024-09-13T11:22:16.931914+0000","last_active":"2024-09-13T11:22:16.931914+0000","last_peered":"2024-09-13T11:22:16.931914+0000","last_clean":"2024-09-13T11:22:16.931914+0000","last_became_active":"2024-09-13T11:22:14.146236+0000","last_became_peered":"2024-09-13T11:22:14.146236+0000","last_unstale":"2024-09-13T11:22:16.931914+0000","last_undegraded":"2024-09-13T11:22:16.931914+0000","last_fullsized":"2024-09-13T11:22:16.931914+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:22:13.113563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:23:15.027945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00087476000000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,9],"acting":[1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"28'2","reported_seq":22,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-13T11:22:16.269626+0000","last_change":"2024-09-13T11:22:16.269626+0000","last_active":"2024-09-13T11:22:16.269626+0000","last_peered":"2024-09-13T11:22:16.269626+0000","last_clean":"2024-09-13T11:22:16.269626+0000","last_became_active":"2024-09-13T11:22:14.144472+0000","last_became_peered":"2024-09-13T11:22:14.144472+0000","last_unstale":"2024-09-13T11:22:16.269626+0000","last_undegraded":"2024-09-13T11:22:16.269626+0000","last_fullsized":"2024-09-13T11:22:16.269626+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:22:13.113563+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:23:28.879995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0025826730000000002,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9],"acting":[3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":20,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-13T11:22:16.547196+0000","last_change":"2024-09-13T11:22:16.547619+0000","last_active":"2024-09-13T11:22:16.547196+0000","last_peered":"2024-09-13T11:22:16.547196+0000","last_clean":"2024-09-13T11:22:16.547196+0000","last_became_active":"2024-09-13T11:22:14.142231+0000","last_became_peered":"2024-09-13T11:22:14.142231+0000","last_unstale":"2024-09-13T11:22:16.547196+0000","last_undegraded":"2024-09-13T11:22:16.547196+0000","last_fullsized":"2024-09-13T11:22:16.547196+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:22:13.113563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:23:35.967649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00081225399999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,1],"acting":[14,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":20,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-13T11:22:16.267447+0000","last_change":"2024-09-13T11:22:16.267999+0000","last_active":"2024-09-13T11:22:16.267447+0000","last_peered":"2024-09-13T11:22:16.267447+0000","last_clean":"2024-09-13T11:22:16.267447+0000","last_became_active":"2024-09-13T11:22:14.211435+0000","last_became_peered":"2024-09-13T11:22:14.211435+0000","last_unstale":"2024-09-13T11:22:16.267447+0000","last_undegraded":"2024-09-13T11:22:16.267447+0000","last_fullsized":"2024-09-13T11:22:16.267447+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:22:13.113563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:23:24.076640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010010360000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"26'1","reported_seq":21,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-13T11:22:16.547303+0000","last_change":"2024-09-13T11:22:16.547691+0000","last_active":"2024-09-13T11:22:16.547303+0000","last_peered":"2024-09-13T11:22:16.547303+0000","last_clean":"2024-09-13T11:22:16.547303+0000","last_became_active":"2024-09-13T11:22:14.146507+0000","last_became_peered":"2024-09-13T11:22:14.146507+0000","last_unstale":"2024-09-13T11:22:16.547303+0000","last_undegraded":"2024-09-13T11:22:16.547303+0000","last_fullsized":"2024-09-13T11:22:16.547303+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:22:13.113563+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:23:18.785562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00085610999999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,15],"acting":[14,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"26'112","reported_seq":155,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-13T11:22:17.030100+0000","last_change":"2024-09-13T11:22:12.131711+0000","last_active":"2024-09-13T11:22:17.030100+0000","last_peered":"2024-09-13T11:22:17.030100+0000","last_clean":"2024-09-13T11:22:17.030100+0000","last_became_active":"2024-09-13T11:22:12.129721+0000","last_became_peered":"2024-09-13T11:22:12.129721+0000","last_unstale":"2024-09-13T11:22:17.030100+0000","last_undegraded":"2024-09-13T11:22:17.030100+0000","last_fullsized":"2024-09-13T11:22:17.030100+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:11.098388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:11.098388+0000","last_clean_scrub_stamp":"2024-09-13T11:22:11.098388+0000","objects_scrubbed":0,"log_size":112,"log_dups_size":0,"ondisk_log_size":112,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:23:40.174992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":9},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":112,"ondisk_log_size":112,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":13,"up_from":23,"seq":98784247813,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27244,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344596,"statfs":{"total":96636764160,"available":96608866304,"internally_reserved":0,"allocated":225280,"data_stored":92720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":23,"seq":98784247812,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27252,"kb_used_data":228,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344588,"statfs":{"total":96636764160,"available":96608858112,"internally_reserved":0,"allocated":233472,"data_stored":93019,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,8,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":23,"seq":98784247811,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27260,"kb_used_data":236,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344580,"statfs":{"total":96636764160,"available":96608849920,"internally_reserved":0,"allocated":241664,"data_stored":93840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,4,5,6,7,8,9,11,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":21,"seq":90194313221,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27276,"kb_used_data":252,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":258048,"data_stored":98939,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,2,4,6,7,8,9,10,12,13],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":21,"seq":90194313222,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27268,"kb_used_data":244,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344572,"statfs":{"total":96636764160,"available":96608841728,"internally_reserved":0,"allocated":249856,"data_stored":94997,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":22,"seq":94489280515,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27772,"kb_used_data":748,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344068,"statfs":{"total":96636764160,"available":96608325632,"internally_reserved":0,"allocated":765952,"data_stored":680139,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,3,4,5,6,7,8,10,11,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":22,"seq":94489280516,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27284,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344556,"statfs":{"total":96636764160,"available":96608825344,"internally_reserved":0,"allocated":266240,"data_stored":101514,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,6,7,8,10,11,12,13],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":15,"up_from":22,"seq":94489280516,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27196,"kb_used_data":172,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344644,"statfs":{"total":96636764160,"available":96608915456,"internally_reserved":0,"allocated":176128,"data_stored":90699,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,3,4,5,6,7,10,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":22,"seq":94489280516,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31444,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340396,"statfs":{"total":96636764160,"available":96604565504,"internally_reserved":0,"allocated":266240,"data_stored":101514,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[3,5,7,8,9,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":22,"seq":94489280516,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27796,"kb_used_data":772,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344044,"statfs":{"total":96636764160,"available":96608301056,"internally_reserved":0,"allocated":790528,"data_stored":683090,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,5,7,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":22,"seq":94489280516,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27212,"kb_used_data":188,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344628,"statfs":{"total":96636764160,"available":96608899072,"internally_reserved":0,"allocated":192512,"data_stored":91790,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,5,6,8,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":22,"seq":94489280516,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27268,"kb_used_data":244,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344572,"statfs":{"total":96636764160,"available":96608841728,"internally_reserved":0,"allocated":249856,"data_stored":96496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[3,5,7,9,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":10,"up_from":22,"seq":94489280515,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27212,"kb_used_data":188,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344628,"statfs":{"total":96636764160,"available":96608899072,"internally_reserved":0,"allocated":192512,"data_stored":93118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,4,5,8,9,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":11,"up_from":22,"seq":94489280517,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27240,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":221184,"data_stored":95078,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[2,3,4,5,6,10,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":12,"up_from":22,"seq":94489280516,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31424,"kb_used_data":240,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340416,"statfs":{"total":96636764160,"available":96604585984,"internally_reserved":0,"allocated":245760,"data_stored":96126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[2,3,4,5,6,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":14,"up_from":22,"seq":94489280516,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":208,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":212992,"data_stored":92435,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,5,6,7,8,9,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T11:22:21.478 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-13T11:22:21.780 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-13T11:22:21.781 INFO:teuthology.orchestra.run.smithi001.stderr:dumped all 2024-09-13T11:22:21.797 INFO:teuthology.orchestra.run.smithi001.stdout:{"pg_ready":true,"pg_map":{"version":19,"stamp":"2024-09-13T11:22:20.114840+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":147,"num_write_kb":2674,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":115,"ondisk_log_size":115,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":17,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":10,"kb":1509949440,"kb_used":445380,"kb_used_data":4676,"kb_used_omap":24,"kb_used_meta":440423,"kb_avail":1509504060,"statfs":{"total":1546188226560,"available":1545732157440,"internally_reserved":0,"allocated":4788224,"data_stored":2695514,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":25519,"internal_metadata":450993233},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.001277"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":20,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-13T11:22:16.888721+0000","last_change":"2024-09-13T11:22:16.889130+0000","last_active":"2024-09-13T11:22:16.888721+0000","last_peered":"2024-09-13T11:22:16.888721+0000","last_clean":"2024-09-13T11:22:16.888721+0000","last_became_active":"2024-09-13T11:22:14.192897+0000","last_became_peered":"2024-09-13T11:22:14.192897+0000","last_unstale":"2024-09-13T11:22:16.888721+0000","last_undegraded":"2024-09-13T11:22:16.888721+0000","last_fullsized":"2024-09-13T11:22:16.888721+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:22:13.113563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:23:26.278353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00084655899999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,10],"acting":[15,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":20,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-13T11:22:16.260591+0000","last_change":"2024-09-13T11:22:16.260591+0000","last_active":"2024-09-13T11:22:16.260591+0000","last_peered":"2024-09-13T11:22:16.260591+0000","last_clean":"2024-09-13T11:22:16.260591+0000","last_became_active":"2024-09-13T11:22:14.211923+0000","last_became_peered":"2024-09-13T11:22:14.211923+0000","last_unstale":"2024-09-13T11:22:16.260591+0000","last_undegraded":"2024-09-13T11:22:16.260591+0000","last_fullsized":"2024-09-13T11:22:16.260591+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:22:13.113563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:23:20.320636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011689529999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0],"acting":[9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":20,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-13T11:22:16.503631+0000","last_change":"2024-09-13T11:22:16.504060+0000","last_active":"2024-09-13T11:22:16.503631+0000","last_peered":"2024-09-13T11:22:16.503631+0000","last_clean":"2024-09-13T11:22:16.503631+0000","last_became_active":"2024-09-13T11:22:14.581458+0000","last_became_peered":"2024-09-13T11:22:14.581458+0000","last_unstale":"2024-09-13T11:22:16.503631+0000","last_undegraded":"2024-09-13T11:22:16.503631+0000","last_fullsized":"2024-09-13T11:22:16.503631+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:22:13.113563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:23:37.935010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001026047,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5],"acting":[8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":20,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-13T11:22:16.931914+0000","last_change":"2024-09-13T11:22:16.931914+0000","last_active":"2024-09-13T11:22:16.931914+0000","last_peered":"2024-09-13T11:22:16.931914+0000","last_clean":"2024-09-13T11:22:16.931914+0000","last_became_active":"2024-09-13T11:22:14.146236+0000","last_became_peered":"2024-09-13T11:22:14.146236+0000","last_unstale":"2024-09-13T11:22:16.931914+0000","last_undegraded":"2024-09-13T11:22:16.931914+0000","last_fullsized":"2024-09-13T11:22:16.931914+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:22:13.113563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:23:15.027945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00087476000000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,9],"acting":[1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"28'2","reported_seq":22,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-13T11:22:16.269626+0000","last_change":"2024-09-13T11:22:16.269626+0000","last_active":"2024-09-13T11:22:16.269626+0000","last_peered":"2024-09-13T11:22:16.269626+0000","last_clean":"2024-09-13T11:22:16.269626+0000","last_became_active":"2024-09-13T11:22:14.144472+0000","last_became_peered":"2024-09-13T11:22:14.144472+0000","last_unstale":"2024-09-13T11:22:16.269626+0000","last_undegraded":"2024-09-13T11:22:16.269626+0000","last_fullsized":"2024-09-13T11:22:16.269626+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:22:13.113563+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:23:28.879995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0025826730000000002,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9],"acting":[3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":20,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-13T11:22:16.547196+0000","last_change":"2024-09-13T11:22:16.547619+0000","last_active":"2024-09-13T11:22:16.547196+0000","last_peered":"2024-09-13T11:22:16.547196+0000","last_clean":"2024-09-13T11:22:16.547196+0000","last_became_active":"2024-09-13T11:22:14.142231+0000","last_became_peered":"2024-09-13T11:22:14.142231+0000","last_unstale":"2024-09-13T11:22:16.547196+0000","last_undegraded":"2024-09-13T11:22:16.547196+0000","last_fullsized":"2024-09-13T11:22:16.547196+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:22:13.113563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:23:35.967649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00081225399999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,1],"acting":[14,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":20,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-13T11:22:16.267447+0000","last_change":"2024-09-13T11:22:16.267999+0000","last_active":"2024-09-13T11:22:16.267447+0000","last_peered":"2024-09-13T11:22:16.267447+0000","last_clean":"2024-09-13T11:22:16.267447+0000","last_became_active":"2024-09-13T11:22:14.211435+0000","last_became_peered":"2024-09-13T11:22:14.211435+0000","last_unstale":"2024-09-13T11:22:16.267447+0000","last_undegraded":"2024-09-13T11:22:16.267447+0000","last_fullsized":"2024-09-13T11:22:16.267447+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:22:13.113563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:23:24.076640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010010360000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"26'1","reported_seq":21,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-13T11:22:16.547303+0000","last_change":"2024-09-13T11:22:16.547691+0000","last_active":"2024-09-13T11:22:16.547303+0000","last_peered":"2024-09-13T11:22:16.547303+0000","last_clean":"2024-09-13T11:22:16.547303+0000","last_became_active":"2024-09-13T11:22:14.146507+0000","last_became_peered":"2024-09-13T11:22:14.146507+0000","last_unstale":"2024-09-13T11:22:16.547303+0000","last_undegraded":"2024-09-13T11:22:16.547303+0000","last_fullsized":"2024-09-13T11:22:16.547303+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:22:13.113563+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:23:18.785562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00085610999999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,15],"acting":[14,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"26'112","reported_seq":155,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-13T11:22:17.030100+0000","last_change":"2024-09-13T11:22:12.131711+0000","last_active":"2024-09-13T11:22:17.030100+0000","last_peered":"2024-09-13T11:22:17.030100+0000","last_clean":"2024-09-13T11:22:17.030100+0000","last_became_active":"2024-09-13T11:22:12.129721+0000","last_became_peered":"2024-09-13T11:22:12.129721+0000","last_unstale":"2024-09-13T11:22:17.030100+0000","last_undegraded":"2024-09-13T11:22:17.030100+0000","last_fullsized":"2024-09-13T11:22:17.030100+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:11.098388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:11.098388+0000","last_clean_scrub_stamp":"2024-09-13T11:22:11.098388+0000","objects_scrubbed":0,"log_size":112,"log_dups_size":0,"ondisk_log_size":112,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:23:40.174992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":9},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":112,"ondisk_log_size":112,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":13,"up_from":23,"seq":98784247813,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27244,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344596,"statfs":{"total":96636764160,"available":96608866304,"internally_reserved":0,"allocated":225280,"data_stored":92720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":23,"seq":98784247812,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27252,"kb_used_data":228,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344588,"statfs":{"total":96636764160,"available":96608858112,"internally_reserved":0,"allocated":233472,"data_stored":93019,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,8,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":23,"seq":98784247811,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27260,"kb_used_data":236,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344580,"statfs":{"total":96636764160,"available":96608849920,"internally_reserved":0,"allocated":241664,"data_stored":93840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,4,5,6,7,8,9,11,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":21,"seq":90194313221,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27276,"kb_used_data":252,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":258048,"data_stored":98939,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,2,4,6,7,8,9,10,12,13],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":21,"seq":90194313222,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27268,"kb_used_data":244,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344572,"statfs":{"total":96636764160,"available":96608841728,"internally_reserved":0,"allocated":249856,"data_stored":94997,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":22,"seq":94489280515,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27772,"kb_used_data":748,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344068,"statfs":{"total":96636764160,"available":96608325632,"internally_reserved":0,"allocated":765952,"data_stored":680139,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,3,4,5,6,7,8,10,11,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":22,"seq":94489280516,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27284,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344556,"statfs":{"total":96636764160,"available":96608825344,"internally_reserved":0,"allocated":266240,"data_stored":101514,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,6,7,8,10,11,12,13],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":15,"up_from":22,"seq":94489280516,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27196,"kb_used_data":172,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344644,"statfs":{"total":96636764160,"available":96608915456,"internally_reserved":0,"allocated":176128,"data_stored":90699,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,3,4,5,6,7,10,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":22,"seq":94489280516,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31444,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340396,"statfs":{"total":96636764160,"available":96604565504,"internally_reserved":0,"allocated":266240,"data_stored":101514,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[3,5,7,8,9,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":22,"seq":94489280516,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27796,"kb_used_data":772,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344044,"statfs":{"total":96636764160,"available":96608301056,"internally_reserved":0,"allocated":790528,"data_stored":683090,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,5,7,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":22,"seq":94489280516,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27212,"kb_used_data":188,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344628,"statfs":{"total":96636764160,"available":96608899072,"internally_reserved":0,"allocated":192512,"data_stored":91790,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,5,6,8,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":22,"seq":94489280516,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27268,"kb_used_data":244,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344572,"statfs":{"total":96636764160,"available":96608841728,"internally_reserved":0,"allocated":249856,"data_stored":96496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[3,5,7,9,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":10,"up_from":22,"seq":94489280515,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27212,"kb_used_data":188,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344628,"statfs":{"total":96636764160,"available":96608899072,"internally_reserved":0,"allocated":192512,"data_stored":93118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,4,5,8,9,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":11,"up_from":22,"seq":94489280517,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27240,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":221184,"data_stored":95078,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[2,3,4,5,6,10,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":12,"up_from":22,"seq":94489280516,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31424,"kb_used_data":240,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340416,"statfs":{"total":96636764160,"available":96604585984,"internally_reserved":0,"allocated":245760,"data_stored":96126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[2,3,4,5,6,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":14,"up_from":22,"seq":94489280516,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":208,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":212992,"data_stored":92435,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,5,6,7,8,9,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T11:22:21.798 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-13T11:22:21.798 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-09-13T11:22:21.798 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-09-13T11:22:21.799 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-09-13T11:22:22.137 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-13T11:22:22.137 INFO:teuthology.orchestra.run.smithi001.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-09-13T11:22:22.154 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-09-13T11:22:22.155 INFO:teuthology.run_tasks:Running task thrashosds... 2024-09-13T11:22:22.166 INFO:tasks.thrashosds:config is {'bluestore_new_sharding': 'random', 'chance_bluestore_reshard': 1, 'chance_pgnum_grow': 1, 'chance_pgnum_shrink': 1, 'chance_pgpnum_fix': 1, 'min_in': 2, 'timeout': 1200, 'sighup_delay': 0.1, 'optrack_toggle_delay': 2.0, 'dump_ops_enable': 'true', 'noscrub_toggle_delay': 2.0, 'random_eio': 0.0} 2024-09-13T11:22:22.167 INFO:tasks.thrashosds:overrides is {'admin_socket': {'branch': 'wip-aclamk-cbt-combined'}, 'ceph': {'conf': {'client.0': {'admin socket': '/var/run/ceph/ceph-$name.asok'}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms bind msgr2': False, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'osd', 'ms inject internal delays': 0.002, 'ms inject socket failures': 2500, 'ms type': 'async', 'osd_pool_default_min_size': 2, 'osd_pool_default_size': 2}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'paxos service trim min': 10}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore write v2': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd backoff on peering': True, 'osd blocked scrub grace period': 3600, 'osd debug reject backfill probability': 0.3, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd delete sleep': 1, 'osd failsafe full ratio': 0.95, 'osd inject bad map crc probability': 0.1, 'osd max backfills': 3, 'osd max markdown count': 1000, 'osd max scrubs': 5, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd snap trim sleep': 2}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(REQUEST_STUCK\\)', 'failed to encode map', '\\(OSD_SLOW_PING_TIME', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', 'SLOW_OPS', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', 'mons down', 'mon down', 'out of quorum', 'noscrub', 'nodeep-scrub', '\\(POOL_APP_NOT_ENABLED\\)'], 'mon_bind_msgr2': False, 'sha1': 'd58f73069a8157a9ace12a0247ee3e8497707ea9'}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'flavor': 'default', 'sha1': 'd58f73069a8157a9ace12a0247ee3e8497707ea9'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-aclamk-cbt-combined', 'sha1': 'd58f73069a8157a9ace12a0247ee3e8497707ea9'}} 2024-09-13T11:22:22.167 INFO:tasks.thrashosds:config is {'bluestore_new_sharding': 'random', 'chance_bluestore_reshard': 1, 'chance_pgnum_grow': 1, 'chance_pgnum_shrink': 1, 'chance_pgpnum_fix': 1, 'min_in': 2, 'timeout': 1200, 'sighup_delay': 0.1, 'optrack_toggle_delay': 2.0, 'dump_ops_enable': 'true', 'noscrub_toggle_delay': 2.0, 'random_eio': 0.0, 'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5} 2024-09-13T11:22:22.167 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr_debug_aggressive_pg_num_changes true 2024-09-13T11:22:22.497 INFO:tasks.thrashosds:Beginning thrashosds... 2024-09-13T11:22:22.497 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-13T11:22:22.497 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-13T11:22:22.789 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-13T11:22:22.789 INFO:teuthology.orchestra.run.smithi001.stderr:dumped all 2024-09-13T11:22:22.805 INFO:teuthology.orchestra.run.smithi001.stdout:{"pg_ready":true,"pg_map":{"version":21,"stamp":"2024-09-13T11:22:22.115528+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":147,"num_write_kb":2674,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":115,"ondisk_log_size":115,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":10,"kb":1509949440,"kb_used":445380,"kb_used_data":4676,"kb_used_omap":24,"kb_used_meta":440423,"kb_avail":1509504060,"statfs":{"total":1546188226560,"available":1545732157440,"internally_reserved":0,"allocated":4788224,"data_stored":2695514,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":25519,"internal_metadata":450993233},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001965"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":20,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-13T11:22:16.888721+0000","last_change":"2024-09-13T11:22:16.889130+0000","last_active":"2024-09-13T11:22:16.888721+0000","last_peered":"2024-09-13T11:22:16.888721+0000","last_clean":"2024-09-13T11:22:16.888721+0000","last_became_active":"2024-09-13T11:22:14.192897+0000","last_became_peered":"2024-09-13T11:22:14.192897+0000","last_unstale":"2024-09-13T11:22:16.888721+0000","last_undegraded":"2024-09-13T11:22:16.888721+0000","last_fullsized":"2024-09-13T11:22:16.888721+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:22:13.113563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:23:26.278353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00084655899999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,10],"acting":[15,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":20,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-13T11:22:16.260591+0000","last_change":"2024-09-13T11:22:16.260591+0000","last_active":"2024-09-13T11:22:16.260591+0000","last_peered":"2024-09-13T11:22:16.260591+0000","last_clean":"2024-09-13T11:22:16.260591+0000","last_became_active":"2024-09-13T11:22:14.211923+0000","last_became_peered":"2024-09-13T11:22:14.211923+0000","last_unstale":"2024-09-13T11:22:16.260591+0000","last_undegraded":"2024-09-13T11:22:16.260591+0000","last_fullsized":"2024-09-13T11:22:16.260591+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:22:13.113563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:23:20.320636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011689529999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0],"acting":[9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":20,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-13T11:22:16.503631+0000","last_change":"2024-09-13T11:22:16.504060+0000","last_active":"2024-09-13T11:22:16.503631+0000","last_peered":"2024-09-13T11:22:16.503631+0000","last_clean":"2024-09-13T11:22:16.503631+0000","last_became_active":"2024-09-13T11:22:14.581458+0000","last_became_peered":"2024-09-13T11:22:14.581458+0000","last_unstale":"2024-09-13T11:22:16.503631+0000","last_undegraded":"2024-09-13T11:22:16.503631+0000","last_fullsized":"2024-09-13T11:22:16.503631+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:22:13.113563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:23:37.935010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001026047,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5],"acting":[8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":20,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-13T11:22:16.931914+0000","last_change":"2024-09-13T11:22:16.931914+0000","last_active":"2024-09-13T11:22:16.931914+0000","last_peered":"2024-09-13T11:22:16.931914+0000","last_clean":"2024-09-13T11:22:16.931914+0000","last_became_active":"2024-09-13T11:22:14.146236+0000","last_became_peered":"2024-09-13T11:22:14.146236+0000","last_unstale":"2024-09-13T11:22:16.931914+0000","last_undegraded":"2024-09-13T11:22:16.931914+0000","last_fullsized":"2024-09-13T11:22:16.931914+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:22:13.113563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:23:15.027945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00087476000000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,9],"acting":[1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"28'2","reported_seq":22,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-13T11:22:16.269626+0000","last_change":"2024-09-13T11:22:16.269626+0000","last_active":"2024-09-13T11:22:16.269626+0000","last_peered":"2024-09-13T11:22:16.269626+0000","last_clean":"2024-09-13T11:22:16.269626+0000","last_became_active":"2024-09-13T11:22:14.144472+0000","last_became_peered":"2024-09-13T11:22:14.144472+0000","last_unstale":"2024-09-13T11:22:16.269626+0000","last_undegraded":"2024-09-13T11:22:16.269626+0000","last_fullsized":"2024-09-13T11:22:16.269626+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:22:13.113563+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:23:28.879995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0025826730000000002,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9],"acting":[3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":20,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-13T11:22:16.547196+0000","last_change":"2024-09-13T11:22:16.547619+0000","last_active":"2024-09-13T11:22:16.547196+0000","last_peered":"2024-09-13T11:22:16.547196+0000","last_clean":"2024-09-13T11:22:16.547196+0000","last_became_active":"2024-09-13T11:22:14.142231+0000","last_became_peered":"2024-09-13T11:22:14.142231+0000","last_unstale":"2024-09-13T11:22:16.547196+0000","last_undegraded":"2024-09-13T11:22:16.547196+0000","last_fullsized":"2024-09-13T11:22:16.547196+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:22:13.113563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:23:35.967649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00081225399999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,1],"acting":[14,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":20,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-13T11:22:16.267447+0000","last_change":"2024-09-13T11:22:16.267999+0000","last_active":"2024-09-13T11:22:16.267447+0000","last_peered":"2024-09-13T11:22:16.267447+0000","last_clean":"2024-09-13T11:22:16.267447+0000","last_became_active":"2024-09-13T11:22:14.211435+0000","last_became_peered":"2024-09-13T11:22:14.211435+0000","last_unstale":"2024-09-13T11:22:16.267447+0000","last_undegraded":"2024-09-13T11:22:16.267447+0000","last_fullsized":"2024-09-13T11:22:16.267447+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:22:13.113563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:23:24.076640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010010360000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"26'1","reported_seq":21,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-13T11:22:16.547303+0000","last_change":"2024-09-13T11:22:16.547691+0000","last_active":"2024-09-13T11:22:16.547303+0000","last_peered":"2024-09-13T11:22:16.547303+0000","last_clean":"2024-09-13T11:22:16.547303+0000","last_became_active":"2024-09-13T11:22:14.146507+0000","last_became_peered":"2024-09-13T11:22:14.146507+0000","last_unstale":"2024-09-13T11:22:16.547303+0000","last_undegraded":"2024-09-13T11:22:16.547303+0000","last_fullsized":"2024-09-13T11:22:16.547303+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:22:13.113563+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:23:18.785562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00085610999999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,15],"acting":[14,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"26'112","reported_seq":155,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-13T11:22:17.030100+0000","last_change":"2024-09-13T11:22:12.131711+0000","last_active":"2024-09-13T11:22:17.030100+0000","last_peered":"2024-09-13T11:22:17.030100+0000","last_clean":"2024-09-13T11:22:17.030100+0000","last_became_active":"2024-09-13T11:22:12.129721+0000","last_became_peered":"2024-09-13T11:22:12.129721+0000","last_unstale":"2024-09-13T11:22:17.030100+0000","last_undegraded":"2024-09-13T11:22:17.030100+0000","last_fullsized":"2024-09-13T11:22:17.030100+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:11.098388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:11.098388+0000","last_clean_scrub_stamp":"2024-09-13T11:22:11.098388+0000","objects_scrubbed":0,"log_size":112,"log_dups_size":0,"ondisk_log_size":112,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:23:40.174992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":9},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":112,"ondisk_log_size":112,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":13,"up_from":23,"seq":98784247814,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27244,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344596,"statfs":{"total":96636764160,"available":96608866304,"internally_reserved":0,"allocated":225280,"data_stored":92720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":9,"up_from":23,"seq":98784247813,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27252,"kb_used_data":228,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344588,"statfs":{"total":96636764160,"available":96608858112,"internally_reserved":0,"allocated":233472,"data_stored":93019,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,8,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":23,"seq":98784247812,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27260,"kb_used_data":236,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344580,"statfs":{"total":96636764160,"available":96608849920,"internally_reserved":0,"allocated":241664,"data_stored":93840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,4,5,6,7,8,9,11,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":21,"seq":90194313221,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27276,"kb_used_data":252,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":258048,"data_stored":98939,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,2,4,6,7,8,9,10,12,13],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":21,"seq":90194313223,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27268,"kb_used_data":244,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344572,"statfs":{"total":96636764160,"available":96608841728,"internally_reserved":0,"allocated":249856,"data_stored":94997,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":22,"seq":94489280516,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27772,"kb_used_data":748,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344068,"statfs":{"total":96636764160,"available":96608325632,"internally_reserved":0,"allocated":765952,"data_stored":680139,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,3,4,5,6,7,8,10,11,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":22,"seq":94489280517,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27284,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344556,"statfs":{"total":96636764160,"available":96608825344,"internally_reserved":0,"allocated":266240,"data_stored":101514,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,6,7,8,10,11,12,13],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":15,"up_from":22,"seq":94489280517,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27196,"kb_used_data":172,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344644,"statfs":{"total":96636764160,"available":96608915456,"internally_reserved":0,"allocated":176128,"data_stored":90699,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,3,4,5,6,7,10,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":22,"seq":94489280517,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31444,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340396,"statfs":{"total":96636764160,"available":96604565504,"internally_reserved":0,"allocated":266240,"data_stored":101514,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[3,5,7,8,9,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":22,"seq":94489280517,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27796,"kb_used_data":772,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344044,"statfs":{"total":96636764160,"available":96608301056,"internally_reserved":0,"allocated":790528,"data_stored":683090,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,5,7,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":22,"seq":94489280517,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27212,"kb_used_data":188,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344628,"statfs":{"total":96636764160,"available":96608899072,"internally_reserved":0,"allocated":192512,"data_stored":91790,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,5,6,8,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":22,"seq":94489280517,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27268,"kb_used_data":244,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344572,"statfs":{"total":96636764160,"available":96608841728,"internally_reserved":0,"allocated":249856,"data_stored":96496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[3,5,7,9,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":22,"seq":94489280516,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27212,"kb_used_data":188,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344628,"statfs":{"total":96636764160,"available":96608899072,"internally_reserved":0,"allocated":192512,"data_stored":93118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,4,5,8,9,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":11,"up_from":22,"seq":94489280518,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27240,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":221184,"data_stored":95078,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[2,3,4,5,6,10,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":12,"up_from":22,"seq":94489280517,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31424,"kb_used_data":240,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340416,"statfs":{"total":96636764160,"available":96604585984,"internally_reserved":0,"allocated":245760,"data_stored":96126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[2,3,4,5,6,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":14,"up_from":22,"seq":94489280517,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":208,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":212992,"data_stored":92435,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,5,6,7,8,9,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T11:22:22.806 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-13T11:22:22.808 INFO:tasks.ceph.osd.9.smithi092.stderr:2024-09-13T11:22:22.808+0000 7ff59ddf9640 -1 osd.9 28 handle_osd_map injecting map crc failure 2024-09-13T11:22:23.117 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-13T11:22:23.117 INFO:teuthology.orchestra.run.smithi001.stderr:dumped all 2024-09-13T11:22:23.136 INFO:teuthology.orchestra.run.smithi001.stdout:{"pg_ready":true,"pg_map":{"version":21,"stamp":"2024-09-13T11:22:22.115528+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":147,"num_write_kb":2674,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":115,"ondisk_log_size":115,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":10,"kb":1509949440,"kb_used":445380,"kb_used_data":4676,"kb_used_omap":24,"kb_used_meta":440423,"kb_avail":1509504060,"statfs":{"total":1546188226560,"available":1545732157440,"internally_reserved":0,"allocated":4788224,"data_stored":2695514,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":25519,"internal_metadata":450993233},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001965"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":20,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-13T11:22:16.888721+0000","last_change":"2024-09-13T11:22:16.889130+0000","last_active":"2024-09-13T11:22:16.888721+0000","last_peered":"2024-09-13T11:22:16.888721+0000","last_clean":"2024-09-13T11:22:16.888721+0000","last_became_active":"2024-09-13T11:22:14.192897+0000","last_became_peered":"2024-09-13T11:22:14.192897+0000","last_unstale":"2024-09-13T11:22:16.888721+0000","last_undegraded":"2024-09-13T11:22:16.888721+0000","last_fullsized":"2024-09-13T11:22:16.888721+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:22:13.113563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:23:26.278353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00084655899999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,10],"acting":[15,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":20,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-13T11:22:16.260591+0000","last_change":"2024-09-13T11:22:16.260591+0000","last_active":"2024-09-13T11:22:16.260591+0000","last_peered":"2024-09-13T11:22:16.260591+0000","last_clean":"2024-09-13T11:22:16.260591+0000","last_became_active":"2024-09-13T11:22:14.211923+0000","last_became_peered":"2024-09-13T11:22:14.211923+0000","last_unstale":"2024-09-13T11:22:16.260591+0000","last_undegraded":"2024-09-13T11:22:16.260591+0000","last_fullsized":"2024-09-13T11:22:16.260591+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:22:13.113563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:23:20.320636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011689529999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0],"acting":[9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":20,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-13T11:22:16.503631+0000","last_change":"2024-09-13T11:22:16.504060+0000","last_active":"2024-09-13T11:22:16.503631+0000","last_peered":"2024-09-13T11:22:16.503631+0000","last_clean":"2024-09-13T11:22:16.503631+0000","last_became_active":"2024-09-13T11:22:14.581458+0000","last_became_peered":"2024-09-13T11:22:14.581458+0000","last_unstale":"2024-09-13T11:22:16.503631+0000","last_undegraded":"2024-09-13T11:22:16.503631+0000","last_fullsized":"2024-09-13T11:22:16.503631+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:22:13.113563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:23:37.935010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001026047,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5],"acting":[8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":20,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-13T11:22:16.931914+0000","last_change":"2024-09-13T11:22:16.931914+0000","last_active":"2024-09-13T11:22:16.931914+0000","last_peered":"2024-09-13T11:22:16.931914+0000","last_clean":"2024-09-13T11:22:16.931914+0000","last_became_active":"2024-09-13T11:22:14.146236+0000","last_became_peered":"2024-09-13T11:22:14.146236+0000","last_unstale":"2024-09-13T11:22:16.931914+0000","last_undegraded":"2024-09-13T11:22:16.931914+0000","last_fullsized":"2024-09-13T11:22:16.931914+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:22:13.113563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:23:15.027945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00087476000000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,9],"acting":[1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"28'2","reported_seq":22,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-13T11:22:16.269626+0000","last_change":"2024-09-13T11:22:16.269626+0000","last_active":"2024-09-13T11:22:16.269626+0000","last_peered":"2024-09-13T11:22:16.269626+0000","last_clean":"2024-09-13T11:22:16.269626+0000","last_became_active":"2024-09-13T11:22:14.144472+0000","last_became_peered":"2024-09-13T11:22:14.144472+0000","last_unstale":"2024-09-13T11:22:16.269626+0000","last_undegraded":"2024-09-13T11:22:16.269626+0000","last_fullsized":"2024-09-13T11:22:16.269626+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:22:13.113563+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:23:28.879995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0025826730000000002,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9],"acting":[3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":20,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-13T11:22:16.547196+0000","last_change":"2024-09-13T11:22:16.547619+0000","last_active":"2024-09-13T11:22:16.547196+0000","last_peered":"2024-09-13T11:22:16.547196+0000","last_clean":"2024-09-13T11:22:16.547196+0000","last_became_active":"2024-09-13T11:22:14.142231+0000","last_became_peered":"2024-09-13T11:22:14.142231+0000","last_unstale":"2024-09-13T11:22:16.547196+0000","last_undegraded":"2024-09-13T11:22:16.547196+0000","last_fullsized":"2024-09-13T11:22:16.547196+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:22:13.113563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:23:35.967649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00081225399999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,1],"acting":[14,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":20,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-13T11:22:16.267447+0000","last_change":"2024-09-13T11:22:16.267999+0000","last_active":"2024-09-13T11:22:16.267447+0000","last_peered":"2024-09-13T11:22:16.267447+0000","last_clean":"2024-09-13T11:22:16.267447+0000","last_became_active":"2024-09-13T11:22:14.211435+0000","last_became_peered":"2024-09-13T11:22:14.211435+0000","last_unstale":"2024-09-13T11:22:16.267447+0000","last_undegraded":"2024-09-13T11:22:16.267447+0000","last_fullsized":"2024-09-13T11:22:16.267447+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:22:13.113563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:23:24.076640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010010360000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"26'1","reported_seq":21,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-13T11:22:16.547303+0000","last_change":"2024-09-13T11:22:16.547691+0000","last_active":"2024-09-13T11:22:16.547303+0000","last_peered":"2024-09-13T11:22:16.547303+0000","last_clean":"2024-09-13T11:22:16.547303+0000","last_became_active":"2024-09-13T11:22:14.146507+0000","last_became_peered":"2024-09-13T11:22:14.146507+0000","last_unstale":"2024-09-13T11:22:16.547303+0000","last_undegraded":"2024-09-13T11:22:16.547303+0000","last_fullsized":"2024-09-13T11:22:16.547303+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:22:13.113563+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:23:18.785562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00085610999999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,15],"acting":[14,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"26'112","reported_seq":155,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-13T11:22:17.030100+0000","last_change":"2024-09-13T11:22:12.131711+0000","last_active":"2024-09-13T11:22:17.030100+0000","last_peered":"2024-09-13T11:22:17.030100+0000","last_clean":"2024-09-13T11:22:17.030100+0000","last_became_active":"2024-09-13T11:22:12.129721+0000","last_became_peered":"2024-09-13T11:22:12.129721+0000","last_unstale":"2024-09-13T11:22:17.030100+0000","last_undegraded":"2024-09-13T11:22:17.030100+0000","last_fullsized":"2024-09-13T11:22:17.030100+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:11.098388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:11.098388+0000","last_clean_scrub_stamp":"2024-09-13T11:22:11.098388+0000","objects_scrubbed":0,"log_size":112,"log_dups_size":0,"ondisk_log_size":112,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:23:40.174992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":9},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":112,"ondisk_log_size":112,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":13,"up_from":23,"seq":98784247814,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27244,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344596,"statfs":{"total":96636764160,"available":96608866304,"internally_reserved":0,"allocated":225280,"data_stored":92720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":9,"up_from":23,"seq":98784247813,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27252,"kb_used_data":228,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344588,"statfs":{"total":96636764160,"available":96608858112,"internally_reserved":0,"allocated":233472,"data_stored":93019,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,8,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":23,"seq":98784247812,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27260,"kb_used_data":236,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344580,"statfs":{"total":96636764160,"available":96608849920,"internally_reserved":0,"allocated":241664,"data_stored":93840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,4,5,6,7,8,9,11,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":21,"seq":90194313221,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27276,"kb_used_data":252,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":258048,"data_stored":98939,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,2,4,6,7,8,9,10,12,13],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":21,"seq":90194313223,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27268,"kb_used_data":244,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344572,"statfs":{"total":96636764160,"available":96608841728,"internally_reserved":0,"allocated":249856,"data_stored":94997,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":22,"seq":94489280516,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27772,"kb_used_data":748,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344068,"statfs":{"total":96636764160,"available":96608325632,"internally_reserved":0,"allocated":765952,"data_stored":680139,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,3,4,5,6,7,8,10,11,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":22,"seq":94489280517,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27284,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344556,"statfs":{"total":96636764160,"available":96608825344,"internally_reserved":0,"allocated":266240,"data_stored":101514,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,6,7,8,10,11,12,13],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":15,"up_from":22,"seq":94489280517,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27196,"kb_used_data":172,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344644,"statfs":{"total":96636764160,"available":96608915456,"internally_reserved":0,"allocated":176128,"data_stored":90699,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,3,4,5,6,7,10,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":22,"seq":94489280517,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31444,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340396,"statfs":{"total":96636764160,"available":96604565504,"internally_reserved":0,"allocated":266240,"data_stored":101514,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[3,5,7,8,9,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":22,"seq":94489280517,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27796,"kb_used_data":772,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344044,"statfs":{"total":96636764160,"available":96608301056,"internally_reserved":0,"allocated":790528,"data_stored":683090,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,5,7,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":22,"seq":94489280517,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27212,"kb_used_data":188,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344628,"statfs":{"total":96636764160,"available":96608899072,"internally_reserved":0,"allocated":192512,"data_stored":91790,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,5,6,8,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":22,"seq":94489280517,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27268,"kb_used_data":244,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344572,"statfs":{"total":96636764160,"available":96608841728,"internally_reserved":0,"allocated":249856,"data_stored":96496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[3,5,7,9,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":22,"seq":94489280516,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27212,"kb_used_data":188,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344628,"statfs":{"total":96636764160,"available":96608899072,"internally_reserved":0,"allocated":192512,"data_stored":93118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,4,5,8,9,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":11,"up_from":22,"seq":94489280518,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27240,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":221184,"data_stored":95078,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[2,3,4,5,6,10,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":12,"up_from":22,"seq":94489280517,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31424,"kb_used_data":240,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340416,"statfs":{"total":96636764160,"available":96604585984,"internally_reserved":0,"allocated":245760,"data_stored":96126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[2,3,4,5,6,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":14,"up_from":22,"seq":94489280517,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":208,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":212992,"data_stored":92435,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,5,6,7,8,9,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T11:22:23.137 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-13T11:22:23.137 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump 2024-09-13T11:22:23.275 INFO:tasks.ceph.osd.12.smithi001.stderr:2024-09-13T11:22:23.269+0000 7f891d10b640 -1 osd.12 28 handle_osd_map injecting map crc failure 2024-09-13T11:22:23.355 INFO:tasks.ceph.osd.15.smithi177.stderr:2024-09-13T11:22:23.353+0000 7f9d638a8640 -1 osd.15 28 handle_osd_map injecting map crc failure 2024-09-13T11:22:23.442 INFO:teuthology.orchestra.run.smithi001.stdout:epoch 29 2024-09-13T11:22:23.442 INFO:teuthology.orchestra.run.smithi001.stdout:fsid 2fd68ac7-5581-4d90-80a4-30bcf5252a7a 2024-09-13T11:22:23.442 INFO:teuthology.orchestra.run.smithi001.stdout:created 2024-09-13T11:21:57.223153+0000 2024-09-13T11:22:23.442 INFO:teuthology.orchestra.run.smithi001.stdout:modified 2024-09-13T11:22:21.883543+0000 2024-09-13T11:22:23.442 INFO:teuthology.orchestra.run.smithi001.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2024-09-13T11:22:23.442 INFO:teuthology.orchestra.run.smithi001.stdout:crush_version 5 2024-09-13T11:22:23.442 INFO:teuthology.orchestra.run.smithi001.stdout:full_ratio 0.95 2024-09-13T11:22:23.442 INFO:teuthology.orchestra.run.smithi001.stdout:backfillfull_ratio 0.9 2024-09-13T11:22:23.442 INFO:teuthology.orchestra.run.smithi001.stdout:nearfull_ratio 0.85 2024-09-13T11:22:23.442 INFO:teuthology.orchestra.run.smithi001.stdout:require_min_compat_client luminous 2024-09-13T11:22:23.442 INFO:teuthology.orchestra.run.smithi001.stdout:min_compat_client jewel 2024-09-13T11:22:23.443 INFO:teuthology.orchestra.run.smithi001.stdout:require_osd_release squid 2024-09-13T11:22:23.443 INFO:teuthology.orchestra.run.smithi001.stdout:stretch_mode_enabled false 2024-09-13T11:22:23.443 INFO:teuthology.orchestra.run.smithi001.stdout:pool 1 '.mgr' replicated size 2 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 25 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 15.38 2024-09-13T11:22:23.443 INFO:teuthology.orchestra.run.smithi001.stdout:pool 2 'rbd' replicated size 2 min_size 2 crush_rule 0 object_hash rjenkins pg_num 8 pgp_num 8 autoscale_mode off last_change 28 flags hashpspool,selfmanaged_snaps stripe_width 0 application rbd read_balance_score 4.02 2024-09-13T11:22:23.443 INFO:teuthology.orchestra.run.smithi001.stdout:max_osd 16 2024-09-13T11:22:23.443 INFO:teuthology.orchestra.run.smithi001.stdout:osd.0 up in weight 1 up_from 22 up_thru 0 down_at 0 last_clean_interval [0,0) v1:172.21.15.1:6800/1544776477 v1:172.21.15.1:6801/1544776477 exists,up 0fc12b3c-e0de-4546-b935-db77efe7c567 2024-09-13T11:22:23.443 INFO:teuthology.orchestra.run.smithi001.stdout:osd.1 up in weight 1 up_from 23 up_thru 25 down_at 0 last_clean_interval [0,0) v1:172.21.15.92:6800/356319619 v1:172.21.15.92:6801/356319619 exists,up b77fe331-cc92-4adb-b146-af5c380f6d4f 2024-09-13T11:22:23.443 INFO:teuthology.orchestra.run.smithi001.stdout:osd.2 up in weight 1 up_from 22 up_thru 0 down_at 0 last_clean_interval [0,0) v1:172.21.15.162:6812/2712148101 v1:172.21.15.162:6813/2712148101 exists,up c9d031c8-e885-4229-affa-08e1729d00e8 2024-09-13T11:22:23.443 INFO:teuthology.orchestra.run.smithi001.stdout:osd.3 up in weight 1 up_from 21 up_thru 25 down_at 0 last_clean_interval [0,0) v1:172.21.15.177:6801/3367751690 v1:172.21.15.177:6802/3367751690 exists,up bf50d71a-6c3a-4fbc-a407-cb81bad14a7b 2024-09-13T11:22:23.443 INFO:teuthology.orchestra.run.smithi001.stdout:osd.4 up in weight 1 up_from 22 up_thru 0 down_at 0 last_clean_interval [0,0) v1:172.21.15.1:6808/3648540926 v1:172.21.15.1:6809/3648540926 exists,up d1fe3f63-df3f-4a73-a1e2-8d7c8a3f1fa0 2024-09-13T11:22:23.443 INFO:teuthology.orchestra.run.smithi001.stdout:osd.5 up in weight 1 up_from 21 up_thru 0 down_at 0 last_clean_interval [0,0) v1:172.21.15.92:6804/3399529091 v1:172.21.15.92:6805/3399529091 exists,up e2ead447-2f2c-40c9-82e7-53b8dc41e2af 2024-09-13T11:22:23.443 INFO:teuthology.orchestra.run.smithi001.stdout:osd.6 up in weight 1 up_from 22 up_thru 23 down_at 0 last_clean_interval [0,0) v1:172.21.15.162:6808/2287343806 v1:172.21.15.162:6809/2287343806 exists,up dbc20439-144e-4ec7-89de-674eb783f263 2024-09-13T11:22:23.443 INFO:teuthology.orchestra.run.smithi001.stdout:osd.7 up in weight 1 up_from 22 up_thru 0 down_at 0 last_clean_interval [0,0) v1:172.21.15.177:6813/2110809906 v1:172.21.15.177:6814/2110809906 exists,up 60aa5206-cace-40f8-8d1a-10ec28d97096 2024-09-13T11:22:23.444 INFO:teuthology.orchestra.run.smithi001.stdout:osd.8 up in weight 1 up_from 22 up_thru 25 down_at 0 last_clean_interval [0,0) v1:172.21.15.1:6804/2766567910 v1:172.21.15.1:6805/2766567910 exists,up f3d99325-48ff-403c-8662-1d7745f7c97d 2024-09-13T11:22:23.444 INFO:teuthology.orchestra.run.smithi001.stdout:osd.9 up in weight 1 up_from 23 up_thru 25 down_at 0 last_clean_interval [0,0) v1:172.21.15.92:6812/2488187511 v1:172.21.15.92:6813/2488187511 exists,up 88511e24-3675-4551-968c-1a52e7e28f96 2024-09-13T11:22:23.444 INFO:teuthology.orchestra.run.smithi001.stdout:osd.10 up in weight 1 up_from 22 up_thru 0 down_at 0 last_clean_interval [0,0) v1:172.21.15.162:6804/1450266459 v1:172.21.15.162:6805/1450266459 exists,up acf4c64d-99af-42be-8406-413c86f54670 2024-09-13T11:22:23.444 INFO:teuthology.orchestra.run.smithi001.stdout:osd.11 up in weight 1 up_from 22 up_thru 0 down_at 0 last_clean_interval [0,0) v1:172.21.15.177:6809/3239521954 v1:172.21.15.177:6810/3239521954 exists,up a4bfdfbe-b586-4080-8e09-6f26428e59b5 2024-09-13T11:22:23.444 INFO:teuthology.orchestra.run.smithi001.stdout:osd.12 up in weight 1 up_from 22 up_thru 0 down_at 0 last_clean_interval [0,0) v1:172.21.15.1:6812/4267062742 v1:172.21.15.1:6813/4267062742 exists,up dc571fb6-d8b5-465b-bb25-afa7d0c47b5a 2024-09-13T11:22:23.444 INFO:teuthology.orchestra.run.smithi001.stdout:osd.13 up in weight 1 up_from 23 up_thru 0 down_at 0 last_clean_interval [0,0) v1:172.21.15.92:6808/1759585046 v1:172.21.15.92:6809/1759585046 exists,up 5d321d4b-1b26-47fe-bf84-25751b8cde7d 2024-09-13T11:22:23.444 INFO:teuthology.orchestra.run.smithi001.stdout:osd.14 up in weight 1 up_from 22 up_thru 25 down_at 0 last_clean_interval [0,0) v1:172.21.15.162:6800/3905144320 v1:172.21.15.162:6801/3905144320 exists,up fc3e2499-0c2a-414d-a25e-2628876ae630 2024-09-13T11:22:23.444 INFO:teuthology.orchestra.run.smithi001.stdout:osd.15 up in weight 1 up_from 22 up_thru 25 down_at 0 last_clean_interval [0,0) v1:172.21.15.177:6805/1294337172 v1:172.21.15.177:6806/1294337172 exists,up d2f69775-edfe-4527-91ec-f7c7e558bec8 2024-09-13T11:22:23.459 INFO:tasks.ceph.ceph_manager.ceph:['osd.0 up in weight 1 up_from 22 up_thru 0 down_at 0 last_clean_interval [0,0) v1:172.21.15.1:6800/1544776477 v1:172.21.15.1:6801/1544776477 exists,up 0fc12b3c-e0de-4546-b935-db77efe7c567', 'osd.1 up in weight 1 up_from 23 up_thru 25 down_at 0 last_clean_interval [0,0) v1:172.21.15.92:6800/356319619 v1:172.21.15.92:6801/356319619 exists,up b77fe331-cc92-4adb-b146-af5c380f6d4f', 'osd.2 up in weight 1 up_from 22 up_thru 0 down_at 0 last_clean_interval [0,0) v1:172.21.15.162:6812/2712148101 v1:172.21.15.162:6813/2712148101 exists,up c9d031c8-e885-4229-affa-08e1729d00e8', 'osd.3 up in weight 1 up_from 21 up_thru 25 down_at 0 last_clean_interval [0,0) v1:172.21.15.177:6801/3367751690 v1:172.21.15.177:6802/3367751690 exists,up bf50d71a-6c3a-4fbc-a407-cb81bad14a7b', 'osd.4 up in weight 1 up_from 22 up_thru 0 down_at 0 last_clean_interval [0,0) v1:172.21.15.1:6808/3648540926 v1:172.21.15.1:6809/3648540926 exists,up d1fe3f63-df3f-4a73-a1e2-8d7c8a3f1fa0', 'osd.5 up in weight 1 up_from 21 up_thru 0 down_at 0 last_clean_interval [0,0) v1:172.21.15.92:6804/3399529091 v1:172.21.15.92:6805/3399529091 exists,up e2ead447-2f2c-40c9-82e7-53b8dc41e2af', 'osd.6 up in weight 1 up_from 22 up_thru 23 down_at 0 last_clean_interval [0,0) v1:172.21.15.162:6808/2287343806 v1:172.21.15.162:6809/2287343806 exists,up dbc20439-144e-4ec7-89de-674eb783f263', 'osd.7 up in weight 1 up_from 22 up_thru 0 down_at 0 last_clean_interval [0,0) v1:172.21.15.177:6813/2110809906 v1:172.21.15.177:6814/2110809906 exists,up 60aa5206-cace-40f8-8d1a-10ec28d97096', 'osd.8 up in weight 1 up_from 22 up_thru 25 down_at 0 last_clean_interval [0,0) v1:172.21.15.1:6804/2766567910 v1:172.21.15.1:6805/2766567910 exists,up f3d99325-48ff-403c-8662-1d7745f7c97d', 'osd.9 up in weight 1 up_from 23 up_thru 25 down_at 0 last_clean_interval [0,0) v1:172.21.15.92:6812/2488187511 v1:172.21.15.92:6813/2488187511 exists,up 88511e24-3675-4551-968c-1a52e7e28f96', 'osd.10 up in weight 1 up_from 22 up_thru 0 down_at 0 last_clean_interval [0,0) v1:172.21.15.162:6804/1450266459 v1:172.21.15.162:6805/1450266459 exists,up acf4c64d-99af-42be-8406-413c86f54670', 'osd.11 up in weight 1 up_from 22 up_thru 0 down_at 0 last_clean_interval [0,0) v1:172.21.15.177:6809/3239521954 v1:172.21.15.177:6810/3239521954 exists,up a4bfdfbe-b586-4080-8e09-6f26428e59b5', 'osd.12 up in weight 1 up_from 22 up_thru 0 down_at 0 last_clean_interval [0,0) v1:172.21.15.1:6812/4267062742 v1:172.21.15.1:6813/4267062742 exists,up dc571fb6-d8b5-465b-bb25-afa7d0c47b5a', 'osd.13 up in weight 1 up_from 23 up_thru 0 down_at 0 last_clean_interval [0,0) v1:172.21.15.92:6808/1759585046 v1:172.21.15.92:6809/1759585046 exists,up 5d321d4b-1b26-47fe-bf84-25751b8cde7d', 'osd.14 up in weight 1 up_from 22 up_thru 25 down_at 0 last_clean_interval [0,0) v1:172.21.15.162:6800/3905144320 v1:172.21.15.162:6801/3905144320 exists,up fc3e2499-0c2a-414d-a25e-2628876ae630', 'osd.15 up in weight 1 up_from 22 up_thru 25 down_at 0 last_clean_interval [0,0) v1:172.21.15.177:6805/1294337172 v1:172.21.15.177:6806/1294337172 exists,up d2f69775-edfe-4527-91ec-f7c7e558bec8'] 2024-09-13T11:22:23.459 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mon.a.asok config show 2024-09-13T11:22:23.570 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-13T11:22:23.570 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "mon.a", 2024-09-13T11:22:23.571 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster": "ceph", 2024-09-13T11:22:23.571 INFO:teuthology.orchestra.run.smithi001.stdout: "admin_socket": "/var/run/ceph/ceph-mon.a.asok", 2024-09-13T11:22:23.571 INFO:teuthology.orchestra.run.smithi001.stdout: "admin_socket_mode": "", 2024-09-13T11:22:23.571 INFO:teuthology.orchestra.run.smithi001.stdout: "auth_allow_insecure_global_id_reclaim": "true", 2024-09-13T11:22:23.571 INFO:teuthology.orchestra.run.smithi001.stdout: "auth_client_required": "cephx, none", 2024-09-13T11:22:23.571 INFO:teuthology.orchestra.run.smithi001.stdout: "auth_cluster_required": "cephx", 2024-09-13T11:22:23.571 INFO:teuthology.orchestra.run.smithi001.stdout: "auth_debug": "true", 2024-09-13T11:22:23.571 INFO:teuthology.orchestra.run.smithi001.stdout: "auth_expose_insecure_global_id_reclaim": "true", 2024-09-13T11:22:23.571 INFO:teuthology.orchestra.run.smithi001.stdout: "auth_mon_ticket_ttl": "660.000000", 2024-09-13T11:22:23.571 INFO:teuthology.orchestra.run.smithi001.stdout: "auth_service_required": "cephx", 2024-09-13T11:22:23.571 INFO:teuthology.orchestra.run.smithi001.stdout: "auth_service_ticket_ttl": "240.000000", 2024-09-13T11:22:23.571 INFO:teuthology.orchestra.run.smithi001.stdout: "auth_supported": "cephx", 2024-09-13T11:22:23.571 INFO:teuthology.orchestra.run.smithi001.stdout: "bdev_aio": "true", 2024-09-13T11:22:23.572 INFO:teuthology.orchestra.run.smithi001.stdout: "bdev_aio_max_queue_depth": "1024", 2024-09-13T11:22:23.572 INFO:teuthology.orchestra.run.smithi001.stdout: "bdev_aio_poll_ms": "250", 2024-09-13T11:22:23.572 INFO:teuthology.orchestra.run.smithi001.stdout: "bdev_aio_reap_max": "16", 2024-09-13T11:22:23.572 INFO:teuthology.orchestra.run.smithi001.stdout: "bdev_async_discard_threads": "0", 2024-09-13T11:22:23.572 INFO:teuthology.orchestra.run.smithi001.stdout: "bdev_block_size": "4096", 2024-09-13T11:22:23.572 INFO:teuthology.orchestra.run.smithi001.stdout: "bdev_debug_aio": "false", 2024-09-13T11:22:23.572 INFO:teuthology.orchestra.run.smithi001.stdout: "bdev_debug_aio_log_age": "5.000000", 2024-09-13T11:22:23.572 INFO:teuthology.orchestra.run.smithi001.stdout: "bdev_debug_aio_suicide_timeout": "60.000000", 2024-09-13T11:22:23.572 INFO:teuthology.orchestra.run.smithi001.stdout: "bdev_debug_inflight_ios": "false", 2024-09-13T11:22:23.572 INFO:teuthology.orchestra.run.smithi001.stdout: "bdev_enable_discard": "false", 2024-09-13T11:22:23.572 INFO:teuthology.orchestra.run.smithi001.stdout: "bdev_flock_retry": "3", 2024-09-13T11:22:23.572 INFO:teuthology.orchestra.run.smithi001.stdout: "bdev_flock_retry_interval": "0.100000", 2024-09-13T11:22:23.572 INFO:teuthology.orchestra.run.smithi001.stdout: "bdev_inject_crash": "0", 2024-09-13T11:22:23.573 INFO:teuthology.orchestra.run.smithi001.stdout: "bdev_inject_crash_flush_delay": "2", 2024-09-13T11:22:23.573 INFO:teuthology.orchestra.run.smithi001.stdout: "bdev_ioring": "false", 2024-09-13T11:22:23.573 INFO:teuthology.orchestra.run.smithi001.stdout: "bdev_ioring_hipri": "false", 2024-09-13T11:22:23.573 INFO:teuthology.orchestra.run.smithi001.stdout: "bdev_ioring_sqthread_poll": "false", 2024-09-13T11:22:23.573 INFO:teuthology.orchestra.run.smithi001.stdout: "bdev_nvme_unbind_from_kernel": "false", 2024-09-13T11:22:23.573 INFO:teuthology.orchestra.run.smithi001.stdout: "bdev_read_buffer_alignment": "4096", 2024-09-13T11:22:23.573 INFO:teuthology.orchestra.run.smithi001.stdout: "bdev_read_preallocated_huge_buffers": "", 2024-09-13T11:22:23.573 INFO:teuthology.orchestra.run.smithi001.stdout: "bdev_stalled_read_warn_lifetime": "86400", 2024-09-13T11:22:23.573 INFO:teuthology.orchestra.run.smithi001.stdout: "bdev_stalled_read_warn_threshold": "1", 2024-09-13T11:22:23.573 INFO:teuthology.orchestra.run.smithi001.stdout: "bdev_type": "", 2024-09-13T11:22:23.573 INFO:teuthology.orchestra.run.smithi001.stdout: "bluefs_alloc_size": "1048576", 2024-09-13T11:22:23.573 INFO:teuthology.orchestra.run.smithi001.stdout: "bluefs_allocator": "hybrid", 2024-09-13T11:22:23.573 INFO:teuthology.orchestra.run.smithi001.stdout: "bluefs_buffered_io": "true", 2024-09-13T11:22:23.574 INFO:teuthology.orchestra.run.smithi001.stdout: "bluefs_check_for_zeros": "false", 2024-09-13T11:22:23.574 INFO:teuthology.orchestra.run.smithi001.stdout: "bluefs_check_volume_selector_often": "false", 2024-09-13T11:22:23.574 INFO:teuthology.orchestra.run.smithi001.stdout: "bluefs_check_volume_selector_on_umount": "false", 2024-09-13T11:22:23.574 INFO:teuthology.orchestra.run.smithi001.stdout: "bluefs_compact_log_sync": "false", 2024-09-13T11:22:23.574 INFO:teuthology.orchestra.run.smithi001.stdout: "bluefs_failed_shared_alloc_cooldown": "600.000000", 2024-09-13T11:22:23.574 INFO:teuthology.orchestra.run.smithi001.stdout: "bluefs_log_compact_min_ratio": "5.000000", 2024-09-13T11:22:23.574 INFO:teuthology.orchestra.run.smithi001.stdout: "bluefs_log_compact_min_size": "16777216", 2024-09-13T11:22:23.574 INFO:teuthology.orchestra.run.smithi001.stdout: "bluefs_log_replay_check_allocations": "true", 2024-09-13T11:22:23.574 INFO:teuthology.orchestra.run.smithi001.stdout: "bluefs_max_log_runway": "4194304", 2024-09-13T11:22:23.574 INFO:teuthology.orchestra.run.smithi001.stdout: "bluefs_max_prefetch": "1048576", 2024-09-13T11:22:23.574 INFO:teuthology.orchestra.run.smithi001.stdout: "bluefs_min_flush_size": "524288", 2024-09-13T11:22:23.574 INFO:teuthology.orchestra.run.smithi001.stdout: "bluefs_min_log_runway": "1048576", 2024-09-13T11:22:23.574 INFO:teuthology.orchestra.run.smithi001.stdout: "bluefs_replay_recovery": "false", 2024-09-13T11:22:23.575 INFO:teuthology.orchestra.run.smithi001.stdout: "bluefs_replay_recovery_disable_compact": "false", 2024-09-13T11:22:23.575 INFO:teuthology.orchestra.run.smithi001.stdout: "bluefs_shared_alloc_size": "65536", 2024-09-13T11:22:23.575 INFO:teuthology.orchestra.run.smithi001.stdout: "bluefs_sync_write": "false", 2024-09-13T11:22:23.575 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_2q_cache_kin_ratio": "0.500000", 2024-09-13T11:22:23.575 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_2q_cache_kout_ratio": "0.500000", 2024-09-13T11:22:23.575 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_alloc_stats_dump_interval": "86400.000000", 2024-09-13T11:22:23.575 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_allocation_from_file": "true", 2024-09-13T11:22:23.575 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_allocator": "hybrid", 2024-09-13T11:22:23.575 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_async_db_compaction": "true", 2024-09-13T11:22:23.575 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_avl_alloc_bf_free_pct": "4", 2024-09-13T11:22:23.575 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_avl_alloc_bf_threshold": "131072", 2024-09-13T11:22:23.575 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_avl_alloc_ff_max_search_bytes": "16777216", 2024-09-13T11:22:23.575 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_avl_alloc_ff_max_search_count": "100", 2024-09-13T11:22:23.576 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_bdev_label_multi": "true", 2024-09-13T11:22:23.576 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_bdev_label_multi_upgrade": "false", 2024-09-13T11:22:23.576 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_bdev_label_require_all": "true", 2024-09-13T11:22:23.576 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_bitmapallocator_blocks_per_zone": "1024", 2024-09-13T11:22:23.576 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_bitmapallocator_span_size": "1024", 2024-09-13T11:22:23.576 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_blobid_prealloc": "10240", 2024-09-13T11:22:23.576 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_block_create": "true", 2024-09-13T11:22:23.576 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_block_db_create": "false", 2024-09-13T11:22:23.576 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_block_db_path": "", 2024-09-13T11:22:23.576 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_block_db_size": "0", 2024-09-13T11:22:23.576 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_block_path": "", 2024-09-13T11:22:23.576 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_block_preallocate_file": "false", 2024-09-13T11:22:23.576 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_block_size": "107374182400", 2024-09-13T11:22:23.577 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_block_wal_create": "false", 2024-09-13T11:22:23.577 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_block_wal_path": "", 2024-09-13T11:22:23.577 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_block_wal_size": "100663296", 2024-09-13T11:22:23.577 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_bluefs": "true", 2024-09-13T11:22:23.577 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_bluefs_alloc_failure_dump_interval": "0.000000", 2024-09-13T11:22:23.577 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_bluefs_env_mirror": "false", 2024-09-13T11:22:23.577 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_bluefs_max_free": "10737418240", 2024-09-13T11:22:23.577 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_btree2_alloc_weight_factor": "2.000000", 2024-09-13T11:22:23.577 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_cache_age_bin_interval": "1.000000", 2024-09-13T11:22:23.577 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_cache_age_bins_data": "1 2 6 24 120 720 0 0 0 0", 2024-09-13T11:22:23.577 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_cache_age_bins_kv": "1 2 6 24 120 720 0 0 0 0", 2024-09-13T11:22:23.577 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_cache_age_bins_kv_onode": "0 0 0 0 0 0 0 0 0 720", 2024-09-13T11:22:23.577 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_cache_age_bins_meta": "1 2 6 24 120 720 0 0 0 0", 2024-09-13T11:22:23.577 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_cache_autotune": "true", 2024-09-13T11:22:23.578 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_cache_autotune_interval": "5.000000", 2024-09-13T11:22:23.578 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_cache_kv_onode_ratio": "0.040000", 2024-09-13T11:22:23.578 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_cache_kv_ratio": "0.450000", 2024-09-13T11:22:23.578 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_cache_meta_ratio": "0.450000", 2024-09-13T11:22:23.578 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_cache_size": "0", 2024-09-13T11:22:23.578 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_cache_size_hdd": "1073741824", 2024-09-13T11:22:23.578 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_cache_size_ssd": "3221225472", 2024-09-13T11:22:23.578 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_cache_trim_interval": "0.050000", 2024-09-13T11:22:23.578 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_cache_trim_max_skip_pinned": "1000", 2024-09-13T11:22:23.578 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_cache_type": "2q", 2024-09-13T11:22:23.578 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_cleaner_sleep_interval": "5.000000", 2024-09-13T11:22:23.578 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_clone_cow": "true", 2024-09-13T11:22:23.578 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_compression_algorithm": "snappy", 2024-09-13T11:22:23.578 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_compression_max_blob_size": "0", 2024-09-13T11:22:23.579 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_compression_max_blob_size_hdd": "65536", 2024-09-13T11:22:23.579 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_compression_max_blob_size_ssd": "65536", 2024-09-13T11:22:23.579 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_compression_min_blob_size": "0", 2024-09-13T11:22:23.579 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_compression_min_blob_size_hdd": "8192", 2024-09-13T11:22:23.579 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_compression_min_blob_size_ssd": "65536", 2024-09-13T11:22:23.579 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_compression_mode": "none", 2024-09-13T11:22:23.579 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_compression_required_ratio": "0.875000", 2024-09-13T11:22:23.579 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_csum_type": "crc32c", 2024-09-13T11:22:23.579 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_debug_enforce_min_alloc_size": "0", 2024-09-13T11:22:23.579 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_debug_enforce_settings": "default", 2024-09-13T11:22:23.579 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_debug_freelist": "false", 2024-09-13T11:22:23.579 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_debug_fsck_abort": "false", 2024-09-13T11:22:23.579 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_debug_inject_allocation_from_file_failure": "0.000000", 2024-09-13T11:22:23.580 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_debug_inject_csum_err_probability": "0.000000", 2024-09-13T11:22:23.580 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_debug_inject_read_err": "false", 2024-09-13T11:22:23.580 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_debug_legacy_omap": "false", 2024-09-13T11:22:23.580 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_debug_no_reuse_blocks": "false", 2024-09-13T11:22:23.580 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_debug_omit_block_device_write": "false", 2024-09-13T11:22:23.580 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_debug_omit_kv_commit": "false", 2024-09-13T11:22:23.580 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_debug_permit_any_bdev_label": "false", 2024-09-13T11:22:23.580 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_debug_prefragment_max": "1048576", 2024-09-13T11:22:23.580 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_debug_random_read_err": "0.000000", 2024-09-13T11:22:23.580 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_debug_randomize_serial_transaction": "0", 2024-09-13T11:22:23.580 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_debug_small_allocations": "0", 2024-09-13T11:22:23.580 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_debug_too_many_blobs_threshold": "24576", 2024-09-13T11:22:23.580 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_default_buffered_read": "true", 2024-09-13T11:22:23.580 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_default_buffered_write": "false", 2024-09-13T11:22:23.581 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_deferred_batch_ops": "0", 2024-09-13T11:22:23.581 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_deferred_batch_ops_hdd": "64", 2024-09-13T11:22:23.581 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_deferred_batch_ops_ssd": "16", 2024-09-13T11:22:23.581 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_elastic_shared_blobs": "true", 2024-09-13T11:22:23.581 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_extent_map_inline_shard_prealloc_size": "256", 2024-09-13T11:22:23.581 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_extent_map_shard_max_size": "1200", 2024-09-13T11:22:23.581 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_extent_map_shard_min_size": "150", 2024-09-13T11:22:23.581 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_extent_map_shard_target_size": "500", 2024-09-13T11:22:23.581 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_extent_map_shard_target_size_slop": "0.200000", 2024-09-13T11:22:23.581 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_fail_eio": "false", 2024-09-13T11:22:23.581 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_freelist_blocks_per_key": "128", 2024-09-13T11:22:23.581 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_fsck_error_on_no_per_pg_omap": "false", 2024-09-13T11:22:23.581 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_fsck_error_on_no_per_pool_omap": "false", 2024-09-13T11:22:23.582 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_fsck_error_on_no_per_pool_stats": "false", 2024-09-13T11:22:23.582 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_fsck_on_mkfs": "true", 2024-09-13T11:22:23.582 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_fsck_on_mkfs_deep": "false", 2024-09-13T11:22:23.582 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_fsck_on_mount": "false", 2024-09-13T11:22:23.582 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_fsck_on_mount_deep": "false", 2024-09-13T11:22:23.582 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_fsck_on_umount": "false", 2024-09-13T11:22:23.582 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_fsck_on_umount_deep": "false", 2024-09-13T11:22:23.582 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_fsck_quick_fix_on_mount": "false", 2024-09-13T11:22:23.582 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_fsck_quick_fix_threads": "2", 2024-09-13T11:22:23.582 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_fsck_read_bytes_cap": "67108864", 2024-09-13T11:22:23.582 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_fsck_shared_blob_tracker_size": "0.031250", 2024-09-13T11:22:23.582 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_gc_enable_blob_threshold": "0", 2024-09-13T11:22:23.582 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_gc_enable_total_threshold": "0", 2024-09-13T11:22:23.582 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_hybrid_alloc_mem_cap": "67108864", 2024-09-13T11:22:23.583 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_ignore_data_csum": "false", 2024-09-13T11:22:23.583 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_kv_sync_util_logging_s": "10.000000", 2024-09-13T11:22:23.583 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_kvbackend": "rocksdb", 2024-09-13T11:22:23.583 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_log_collection_list_age": "60.000000", 2024-09-13T11:22:23.583 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_log_omap_iterator_age": "5.000000", 2024-09-13T11:22:23.583 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_log_op_age": "5.000000", 2024-09-13T11:22:23.583 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_max_alloc_size": "0", 2024-09-13T11:22:23.583 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_max_blob_size": "0", 2024-09-13T11:22:23.583 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_max_blob_size_hdd": "65536", 2024-09-13T11:22:23.583 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_max_blob_size_ssd": "65536", 2024-09-13T11:22:23.583 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_max_defer_interval": "3.000000", 2024-09-13T11:22:23.583 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_max_deferred_txc": "32", 2024-09-13T11:22:23.583 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_min_alloc_size": "0", 2024-09-13T11:22:23.583 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_min_alloc_size_hdd": "4096", 2024-09-13T11:22:23.584 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_min_alloc_size_ssd": "4096", 2024-09-13T11:22:23.584 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_nid_prealloc": "1024", 2024-09-13T11:22:23.584 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_prefer_deferred_size": "0", 2024-09-13T11:22:23.584 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_prefer_deferred_size_hdd": "65536", 2024-09-13T11:22:23.584 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_prefer_deferred_size_ssd": "0", 2024-09-13T11:22:23.584 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_qfsck_on_mount": "true", 2024-09-13T11:22:23.584 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_retry_disk_reads": "3", 2024-09-13T11:22:23.584 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_rocksdb_cf": "true", 2024-09-13T11:22:23.584 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_rocksdb_cfs": "m(3) p(3,0-12) O(3,0-13)=block_cache={type=binned_lru} L=min_write_buffer_number_to_merge=32 P=min_write_buffer_number_to_merge=32", 2024-09-13T11:22:23.584 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_rocksdb_options": "compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0", 2024-09-13T11:22:23.584 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_rocksdb_options_annex": "", 2024-09-13T11:22:23.584 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_slow_ops_warn_lifetime": "86400", 2024-09-13T11:22:23.584 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_slow_ops_warn_threshold": "1", 2024-09-13T11:22:23.585 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_spdk_coremask": "0x1", 2024-09-13T11:22:23.585 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_spdk_io_sleep": "5", 2024-09-13T11:22:23.585 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_spdk_max_io_completion": "0", 2024-09-13T11:22:23.585 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_spdk_mem": "512", 2024-09-13T11:22:23.585 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_sync_submit_transaction": "false", 2024-09-13T11:22:23.585 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_throttle_bytes": "67108864", 2024-09-13T11:22:23.585 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_throttle_cost_per_io": "0", 2024-09-13T11:22:23.585 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_throttle_cost_per_io_hdd": "670000", 2024-09-13T11:22:23.585 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_throttle_cost_per_io_ssd": "4000", 2024-09-13T11:22:23.585 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_throttle_deferred_bytes": "134217728", 2024-09-13T11:22:23.585 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_throttle_trace_rate": "0.000000", 2024-09-13T11:22:23.585 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_tracing": "false", 2024-09-13T11:22:23.585 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_use_optimal_io_size_for_min_alloc_size": "false", 2024-09-13T11:22:23.585 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_volume_selection_policy": "use_some_extra", 2024-09-13T11:22:23.586 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_volume_selection_reserved": "0", 2024-09-13T11:22:23.586 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_volume_selection_reserved_factor": "2.000000", 2024-09-13T11:22:23.586 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_warn_on_bluefs_spillover": "true", 2024-09-13T11:22:23.586 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_warn_on_legacy_statfs": "true", 2024-09-13T11:22:23.586 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_warn_on_no_per_pg_omap": "false", 2024-09-13T11:22:23.586 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_warn_on_no_per_pool_omap": "true", 2024-09-13T11:22:23.586 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_warn_on_spurious_read_errors": "true", 2024-09-13T11:22:23.586 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_write_v2": "false", 2024-09-13T11:22:23.586 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_write_v2_random": "false", 2024-09-13T11:22:23.586 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_zero_block_detection": "false", 2024-09-13T11:22:23.586 INFO:teuthology.orchestra.run.smithi001.stdout: "cephadm_path": "/usr/sbin/cephadm", 2024-09-13T11:22:23.586 INFO:teuthology.orchestra.run.smithi001.stdout: "cephfs_mirror_action_update_interval": "2", 2024-09-13T11:22:23.586 INFO:teuthology.orchestra.run.smithi001.stdout: "cephfs_mirror_directory_scan_interval": "10", 2024-09-13T11:22:23.586 INFO:teuthology.orchestra.run.smithi001.stdout: "cephfs_mirror_max_concurrent_directory_syncs": "3", 2024-09-13T11:22:23.587 INFO:teuthology.orchestra.run.smithi001.stdout: "cephfs_mirror_max_consecutive_failures_per_directory": "10", 2024-09-13T11:22:23.587 INFO:teuthology.orchestra.run.smithi001.stdout: "cephfs_mirror_max_snapshot_sync_per_cycle": "3", 2024-09-13T11:22:23.587 INFO:teuthology.orchestra.run.smithi001.stdout: "cephfs_mirror_mount_timeout": "10", 2024-09-13T11:22:23.587 INFO:teuthology.orchestra.run.smithi001.stdout: "cephfs_mirror_perf_stats_prio": "5", 2024-09-13T11:22:23.587 INFO:teuthology.orchestra.run.smithi001.stdout: "cephfs_mirror_restart_mirror_on_blocklist_interval": "30", 2024-09-13T11:22:23.587 INFO:teuthology.orchestra.run.smithi001.stdout: "cephfs_mirror_restart_mirror_on_failure_interval": "20", 2024-09-13T11:22:23.587 INFO:teuthology.orchestra.run.smithi001.stdout: "cephfs_mirror_retry_failed_directories_interval": "60", 2024-09-13T11:22:23.587 INFO:teuthology.orchestra.run.smithi001.stdout: "cephsqlite_blocklist_dead_locker": "true", 2024-09-13T11:22:23.587 INFO:teuthology.orchestra.run.smithi001.stdout: "cephsqlite_lock_renewal_interval": "2000", 2024-09-13T11:22:23.587 INFO:teuthology.orchestra.run.smithi001.stdout: "cephsqlite_lock_renewal_timeout": "30000", 2024-09-13T11:22:23.587 INFO:teuthology.orchestra.run.smithi001.stdout: "cephx_cluster_require_signatures": "false", 2024-09-13T11:22:23.587 INFO:teuthology.orchestra.run.smithi001.stdout: "cephx_cluster_require_version": "2", 2024-09-13T11:22:23.587 INFO:teuthology.orchestra.run.smithi001.stdout: "cephx_require_signatures": "false", 2024-09-13T11:22:23.587 INFO:teuthology.orchestra.run.smithi001.stdout: "cephx_require_version": "2", 2024-09-13T11:22:23.588 INFO:teuthology.orchestra.run.smithi001.stdout: "cephx_service_require_signatures": "false", 2024-09-13T11:22:23.588 INFO:teuthology.orchestra.run.smithi001.stdout: "cephx_service_require_version": "2", 2024-09-13T11:22:23.588 INFO:teuthology.orchestra.run.smithi001.stdout: "cephx_sign_messages": "true", 2024-09-13T11:22:23.588 INFO:teuthology.orchestra.run.smithi001.stdout: "chdir": "", 2024-09-13T11:22:23.588 INFO:teuthology.orchestra.run.smithi001.stdout: "client_acl_type": "", 2024-09-13T11:22:23.588 INFO:teuthology.orchestra.run.smithi001.stdout: "client_asio_thread_count": "2", 2024-09-13T11:22:23.588 INFO:teuthology.orchestra.run.smithi001.stdout: "client_cache_mid": "0.750000", 2024-09-13T11:22:23.588 INFO:teuthology.orchestra.run.smithi001.stdout: "client_cache_size": "16384", 2024-09-13T11:22:23.588 INFO:teuthology.orchestra.run.smithi001.stdout: "client_caps_release_delay": "5", 2024-09-13T11:22:23.588 INFO:teuthology.orchestra.run.smithi001.stdout: "client_check_pool_perm": "true", 2024-09-13T11:22:23.588 INFO:teuthology.orchestra.run.smithi001.stdout: "client_collect_and_send_global_metrics": "false", 2024-09-13T11:22:23.588 INFO:teuthology.orchestra.run.smithi001.stdout: "client_debug_force_sync_read": "false", 2024-09-13T11:22:23.588 INFO:teuthology.orchestra.run.smithi001.stdout: "client_debug_getattr_caps": "false", 2024-09-13T11:22:23.588 INFO:teuthology.orchestra.run.smithi001.stdout: "client_debug_inject_features": "", 2024-09-13T11:22:23.589 INFO:teuthology.orchestra.run.smithi001.stdout: "client_debug_inject_tick_delay": "0", 2024-09-13T11:22:23.589 INFO:teuthology.orchestra.run.smithi001.stdout: "client_die_on_failed_dentry_invalidate": "true", 2024-09-13T11:22:23.589 INFO:teuthology.orchestra.run.smithi001.stdout: "client_die_on_failed_remount": "false", 2024-09-13T11:22:23.589 INFO:teuthology.orchestra.run.smithi001.stdout: "client_dirsize_rbytes": "true", 2024-09-13T11:22:23.589 INFO:teuthology.orchestra.run.smithi001.stdout: "client_force_lazyio": "false", 2024-09-13T11:22:23.589 INFO:teuthology.orchestra.run.smithi001.stdout: "client_fs": "", 2024-09-13T11:22:23.589 INFO:teuthology.orchestra.run.smithi001.stdout: "client_inject_fixed_oldest_tid": "false", 2024-09-13T11:22:23.589 INFO:teuthology.orchestra.run.smithi001.stdout: "client_inject_release_failure": "false", 2024-09-13T11:22:23.589 INFO:teuthology.orchestra.run.smithi001.stdout: "client_max_inline_size": "4096", 2024-09-13T11:22:23.589 INFO:teuthology.orchestra.run.smithi001.stdout: "client_max_retries_on_remount_failure": "5", 2024-09-13T11:22:23.589 INFO:teuthology.orchestra.run.smithi001.stdout: "client_mds_namespace": "", 2024-09-13T11:22:23.589 INFO:teuthology.orchestra.run.smithi001.stdout: "client_metadata": "", 2024-09-13T11:22:23.590 INFO:teuthology.orchestra.run.smithi001.stdout: "client_mount_gid": "-1", 2024-09-13T11:22:23.590 INFO:teuthology.orchestra.run.smithi001.stdout: "client_mount_timeout": "300", 2024-09-13T11:22:23.590 INFO:teuthology.orchestra.run.smithi001.stdout: "client_mount_uid": "-1", 2024-09-13T11:22:23.590 INFO:teuthology.orchestra.run.smithi001.stdout: "client_mountpoint": "/", 2024-09-13T11:22:23.590 INFO:teuthology.orchestra.run.smithi001.stdout: "client_notify_timeout": "10", 2024-09-13T11:22:23.590 INFO:teuthology.orchestra.run.smithi001.stdout: "client_oc": "true", 2024-09-13T11:22:23.590 INFO:teuthology.orchestra.run.smithi001.stdout: "client_oc_max_dirty": "104857600", 2024-09-13T11:22:23.590 INFO:teuthology.orchestra.run.smithi001.stdout: "client_oc_max_dirty_age": "5.000000", 2024-09-13T11:22:23.590 INFO:teuthology.orchestra.run.smithi001.stdout: "client_oc_max_objects": "1000", 2024-09-13T11:22:23.590 INFO:teuthology.orchestra.run.smithi001.stdout: "client_oc_size": "209715200", 2024-09-13T11:22:23.590 INFO:teuthology.orchestra.run.smithi001.stdout: "client_oc_target_dirty": "8388608", 2024-09-13T11:22:23.590 INFO:teuthology.orchestra.run.smithi001.stdout: "client_permissions": "true", 2024-09-13T11:22:23.590 INFO:teuthology.orchestra.run.smithi001.stdout: "client_quota": "true", 2024-09-13T11:22:23.591 INFO:teuthology.orchestra.run.smithi001.stdout: "client_quota_df": "true", 2024-09-13T11:22:23.591 INFO:teuthology.orchestra.run.smithi001.stdout: "client_readahead_max_bytes": "0", 2024-09-13T11:22:23.591 INFO:teuthology.orchestra.run.smithi001.stdout: "client_readahead_max_periods": "4", 2024-09-13T11:22:23.591 INFO:teuthology.orchestra.run.smithi001.stdout: "client_readahead_min": "131072", 2024-09-13T11:22:23.591 INFO:teuthology.orchestra.run.smithi001.stdout: "client_reconnect_stale": "false", 2024-09-13T11:22:23.591 INFO:teuthology.orchestra.run.smithi001.stdout: "client_shutdown_timeout": "30", 2024-09-13T11:22:23.591 INFO:teuthology.orchestra.run.smithi001.stdout: "client_snapdir": ".snap", 2024-09-13T11:22:23.591 INFO:teuthology.orchestra.run.smithi001.stdout: "client_tick_interval": "1", 2024-09-13T11:22:23.591 INFO:teuthology.orchestra.run.smithi001.stdout: "client_trace": "", 2024-09-13T11:22:23.591 INFO:teuthology.orchestra.run.smithi001.stdout: "client_try_dentry_invalidate": "false", 2024-09-13T11:22:23.591 INFO:teuthology.orchestra.run.smithi001.stdout: "client_use_faked_inos": "false", 2024-09-13T11:22:23.591 INFO:teuthology.orchestra.run.smithi001.stdout: "client_use_random_mds": "false", 2024-09-13T11:22:23.591 INFO:teuthology.orchestra.run.smithi001.stdout: "clog_to_graylog": "false", 2024-09-13T11:22:23.591 INFO:teuthology.orchestra.run.smithi001.stdout: "clog_to_graylog_host": "127.0.0.1", 2024-09-13T11:22:23.592 INFO:teuthology.orchestra.run.smithi001.stdout: "clog_to_graylog_port": "12201", 2024-09-13T11:22:23.592 INFO:teuthology.orchestra.run.smithi001.stdout: "clog_to_monitors": "default=true", 2024-09-13T11:22:23.592 INFO:teuthology.orchestra.run.smithi001.stdout: "clog_to_syslog": "false", 2024-09-13T11:22:23.592 INFO:teuthology.orchestra.run.smithi001.stdout: "clog_to_syslog_facility": "default=daemon audit=local0", 2024-09-13T11:22:23.592 INFO:teuthology.orchestra.run.smithi001.stdout: "clog_to_syslog_level": "info", 2024-09-13T11:22:23.592 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_addr": "-", 2024-09-13T11:22:23.592 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_network": "", 2024-09-13T11:22:23.592 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_network_interface": "", 2024-09-13T11:22:23.592 INFO:teuthology.orchestra.run.smithi001.stdout: "compressor_zlib_isal": "false", 2024-09-13T11:22:23.592 INFO:teuthology.orchestra.run.smithi001.stdout: "compressor_zlib_level": "5", 2024-09-13T11:22:23.592 INFO:teuthology.orchestra.run.smithi001.stdout: "compressor_zlib_winsize": "-15", 2024-09-13T11:22:23.592 INFO:teuthology.orchestra.run.smithi001.stdout: "compressor_zstd_level": "1", 2024-09-13T11:22:23.592 INFO:teuthology.orchestra.run.smithi001.stdout: "container_image": "docker.io/ceph/daemon-base:latest-master-devel", 2024-09-13T11:22:23.593 INFO:teuthology.orchestra.run.smithi001.stdout: "crash_dir": "/var/lib/ceph/crash", 2024-09-13T11:22:23.593 INFO:teuthology.orchestra.run.smithi001.stdout: "crimson_alien_op_num_threads": "6", 2024-09-13T11:22:23.593 INFO:teuthology.orchestra.run.smithi001.stdout: "crimson_alien_thread_cpu_cores": "", 2024-09-13T11:22:23.593 INFO:teuthology.orchestra.run.smithi001.stdout: "crimson_osd_obc_lru_size": "512", 2024-09-13T11:22:23.593 INFO:teuthology.orchestra.run.smithi001.stdout: "crimson_osd_scheduler_concurrency": "0", 2024-09-13T11:22:23.593 INFO:teuthology.orchestra.run.smithi001.stdout: "crimson_osd_stat_interval": "0", 2024-09-13T11:22:23.593 INFO:teuthology.orchestra.run.smithi001.stdout: "crimson_seastar_cpu_cores": "", 2024-09-13T11:22:23.593 INFO:teuthology.orchestra.run.smithi001.stdout: "crimson_seastar_num_threads": "0", 2024-09-13T11:22:23.593 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "", 2024-09-13T11:22:23.593 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location_hook": "", 2024-09-13T11:22:23.593 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location_hook_timeout": "10", 2024-09-13T11:22:23.593 INFO:teuthology.orchestra.run.smithi001.stdout: "daemonize": "false", 2024-09-13T11:22:23.594 INFO:teuthology.orchestra.run.smithi001.stdout: "daos_pool": "tank", 2024-09-13T11:22:23.594 INFO:teuthology.orchestra.run.smithi001.stdout: "dbstore_config_uri": "file:/var/lib/ceph/radosgw/dbstore-config.db", 2024-09-13T11:22:23.594 INFO:teuthology.orchestra.run.smithi001.stdout: "dbstore_db_dir": "/var/lib/ceph/radosgw", 2024-09-13T11:22:23.594 INFO:teuthology.orchestra.run.smithi001.stdout: "dbstore_db_name_prefix": "dbstore", 2024-09-13T11:22:23.594 INFO:teuthology.orchestra.run.smithi001.stdout: "debug_alienstore": "0/5", 2024-09-13T11:22:23.594 INFO:teuthology.orchestra.run.smithi001.stdout: "debug_allow_any_pool_priority": "false", 2024-09-13T11:22:23.594 INFO:teuthology.orchestra.run.smithi001.stdout: "debug_asok": "1/5", 2024-09-13T11:22:23.594 INFO:teuthology.orchestra.run.smithi001.stdout: "debug_asok_assert_abort": "false", 2024-09-13T11:22:23.594 INFO:teuthology.orchestra.run.smithi001.stdout: "debug_asserts_on_shutdown": "true", 2024-09-13T11:22:23.594 INFO:teuthology.orchestra.run.smithi001.stdout: "debug_auth": "20/20", 2024-09-13T11:22:23.594 INFO:teuthology.orchestra.run.smithi001.stdout: "debug_bdev": "1/3", 2024-09-13T11:22:23.594 INFO:teuthology.orchestra.run.smithi001.stdout: "debug_bluefs": "1/5", 2024-09-13T11:22:23.594 INFO:teuthology.orchestra.run.smithi001.stdout: "debug_bluestore": "1/5", 2024-09-13T11:22:23.595 INFO:teuthology.orchestra.run.smithi001.stdout: "debug_buffer": "0/1", 2024-09-13T11:22:23.595 INFO:teuthology.orchestra.run.smithi001.stdout: "debug_ceph_dedup": "0/5", 2024-09-13T11:22:23.595 INFO:teuthology.orchestra.run.smithi001.stdout: "debug_ceph_exporter": "1/5", 2024-09-13T11:22:23.595 INFO:teuthology.orchestra.run.smithi001.stdout: "debug_cephfs_mirror": "0/5", 2024-09-13T11:22:23.595 INFO:teuthology.orchestra.run.smithi001.stdout: "debug_cephsqlite": "0/5", 2024-09-13T11:22:23.595 INFO:teuthology.orchestra.run.smithi001.stdout: "debug_client": "0/5", 2024-09-13T11:22:23.595 INFO:teuthology.orchestra.run.smithi001.stdout: "debug_compressor": "1/5", 2024-09-13T11:22:23.595 INFO:teuthology.orchestra.run.smithi001.stdout: "debug_context": "0/1", 2024-09-13T11:22:23.595 INFO:teuthology.orchestra.run.smithi001.stdout: "debug_crimson_interrupt": "0/5", 2024-09-13T11:22:23.595 INFO:teuthology.orchestra.run.smithi001.stdout: "debug_crush": "1/1", 2024-09-13T11:22:23.595 INFO:teuthology.orchestra.run.smithi001.stdout: "debug_crypto": "1/5", 2024-09-13T11:22:23.595 INFO:teuthology.orchestra.run.smithi001.stdout: "debug_cyanstore": "0/5", 2024-09-13T11:22:23.595 INFO:teuthology.orchestra.run.smithi001.stdout: "debug_deliberately_leak_memory": "false", 2024-09-13T11:22:23.596 INFO:teuthology.orchestra.run.smithi001.stdout: "debug_disable_randomized_ping": "false", 2024-09-13T11:22:23.596 INFO:teuthology.orchestra.run.smithi001.stdout: "debug_dpdk": "1/5", 2024-09-13T11:22:23.596 INFO:teuthology.orchestra.run.smithi001.stdout: "debug_eventtrace": "1/5", 2024-09-13T11:22:23.596 INFO:teuthology.orchestra.run.smithi001.stdout: "debug_filer": "0/1", 2024-09-13T11:22:23.596 INFO:teuthology.orchestra.run.smithi001.stdout: "debug_filestore": "1/3", 2024-09-13T11:22:23.596 INFO:teuthology.orchestra.run.smithi001.stdout: "debug_finisher": "1/1", 2024-09-13T11:22:23.596 INFO:teuthology.orchestra.run.smithi001.stdout: "debug_fuse": "1/5", 2024-09-13T11:22:23.596 INFO:teuthology.orchestra.run.smithi001.stdout: "debug_heartbeat_testing_span": "0", 2024-09-13T11:22:23.596 INFO:teuthology.orchestra.run.smithi001.stdout: "debug_heartbeatmap": "1/5", 2024-09-13T11:22:23.596 INFO:teuthology.orchestra.run.smithi001.stdout: "debug_immutable_obj_cache": "0/5", 2024-09-13T11:22:23.596 INFO:teuthology.orchestra.run.smithi001.stdout: "debug_javaclient": "1/5", 2024-09-13T11:22:23.596 INFO:teuthology.orchestra.run.smithi001.stdout: "debug_journal": "1/3", 2024-09-13T11:22:23.596 INFO:teuthology.orchestra.run.smithi001.stdout: "debug_journaler": "0/5", 2024-09-13T11:22:23.596 INFO:teuthology.orchestra.run.smithi001.stdout: "debug_kstore": "1/5", 2024-09-13T11:22:23.597 INFO:teuthology.orchestra.run.smithi001.stdout: "debug_lockdep": "0/1", 2024-09-13T11:22:23.597 INFO:teuthology.orchestra.run.smithi001.stdout: "debug_mclock": "1/5", 2024-09-13T11:22:23.597 INFO:teuthology.orchestra.run.smithi001.stdout: "debug_mds": "1/5", 2024-09-13T11:22:23.597 INFO:teuthology.orchestra.run.smithi001.stdout: "debug_mds_balancer": "1/5", 2024-09-13T11:22:23.597 INFO:teuthology.orchestra.run.smithi001.stdout: "debug_mds_locker": "1/5", 2024-09-13T11:22:23.597 INFO:teuthology.orchestra.run.smithi001.stdout: "debug_mds_log": "1/5", 2024-09-13T11:22:23.597 INFO:teuthology.orchestra.run.smithi001.stdout: "debug_mds_log_expire": "1/5", 2024-09-13T11:22:23.597 INFO:teuthology.orchestra.run.smithi001.stdout: "debug_mds_migrator": "1/5", 2024-09-13T11:22:23.597 INFO:teuthology.orchestra.run.smithi001.stdout: "debug_mds_quiesce": "3/5", 2024-09-13T11:22:23.597 INFO:teuthology.orchestra.run.smithi001.stdout: "debug_memstore": "1/5", 2024-09-13T11:22:23.597 INFO:teuthology.orchestra.run.smithi001.stdout: "debug_mgr": "2/5", 2024-09-13T11:22:23.597 INFO:teuthology.orchestra.run.smithi001.stdout: "debug_mgrc": "1/5", 2024-09-13T11:22:23.598 INFO:teuthology.orchestra.run.smithi001.stdout: "debug_mon": "20/20", 2024-09-13T11:22:23.598 INFO:teuthology.orchestra.run.smithi001.stdout: "debug_monc": "0/10", 2024-09-13T11:22:23.598 INFO:teuthology.orchestra.run.smithi001.stdout: "debug_ms": "1/1", 2024-09-13T11:22:23.598 INFO:teuthology.orchestra.run.smithi001.stdout: "debug_none": "0/5", 2024-09-13T11:22:23.598 INFO:teuthology.orchestra.run.smithi001.stdout: "debug_objclass": "0/5", 2024-09-13T11:22:23.598 INFO:teuthology.orchestra.run.smithi001.stdout: "debug_objectcacher": "0/5", 2024-09-13T11:22:23.598 INFO:teuthology.orchestra.run.smithi001.stdout: "debug_objecter": "0/1", 2024-09-13T11:22:23.598 INFO:teuthology.orchestra.run.smithi001.stdout: "debug_optracker": "0/5", 2024-09-13T11:22:23.598 INFO:teuthology.orchestra.run.smithi001.stdout: "debug_osd": "1/5", 2024-09-13T11:22:23.598 INFO:teuthology.orchestra.run.smithi001.stdout: "debug_paxos": "20/20", 2024-09-13T11:22:23.598 INFO:teuthology.orchestra.run.smithi001.stdout: "debug_perfcounter": "1/5", 2024-09-13T11:22:23.598 INFO:teuthology.orchestra.run.smithi001.stdout: "debug_prioritycache": "1/5", 2024-09-13T11:22:23.598 INFO:teuthology.orchestra.run.smithi001.stdout: "debug_rados": "0/5", 2024-09-13T11:22:23.598 INFO:teuthology.orchestra.run.smithi001.stdout: "debug_rbd": "0/5", 2024-09-13T11:22:23.599 INFO:teuthology.orchestra.run.smithi001.stdout: "debug_rbd_mirror": "0/5", 2024-09-13T11:22:23.599 INFO:teuthology.orchestra.run.smithi001.stdout: "debug_rbd_pwl": "0/5", 2024-09-13T11:22:23.599 INFO:teuthology.orchestra.run.smithi001.stdout: "debug_rbd_replay": "0/5", 2024-09-13T11:22:23.599 INFO:teuthology.orchestra.run.smithi001.stdout: "debug_refs": "0/0", 2024-09-13T11:22:23.599 INFO:teuthology.orchestra.run.smithi001.stdout: "debug_reserver": "1/1", 2024-09-13T11:22:23.599 INFO:teuthology.orchestra.run.smithi001.stdout: "debug_rgw": "1/5", 2024-09-13T11:22:23.599 INFO:teuthology.orchestra.run.smithi001.stdout: "debug_rgw_access": "1/5", 2024-09-13T11:22:23.599 INFO:teuthology.orchestra.run.smithi001.stdout: "debug_rgw_datacache": "1/5", 2024-09-13T11:22:23.599 INFO:teuthology.orchestra.run.smithi001.stdout: "debug_rgw_dbstore": "1/5", 2024-09-13T11:22:23.599 INFO:teuthology.orchestra.run.smithi001.stdout: "debug_rgw_flight": "1/5", 2024-09-13T11:22:23.599 INFO:teuthology.orchestra.run.smithi001.stdout: "debug_rgw_lifecycle": "1/5", 2024-09-13T11:22:23.599 INFO:teuthology.orchestra.run.smithi001.stdout: "debug_rgw_notification": "1/5", 2024-09-13T11:22:23.599 INFO:teuthology.orchestra.run.smithi001.stdout: "debug_rgw_sync": "1/5", 2024-09-13T11:22:23.600 INFO:teuthology.orchestra.run.smithi001.stdout: "debug_rocksdb": "4/5", 2024-09-13T11:22:23.600 INFO:teuthology.orchestra.run.smithi001.stdout: "debug_seastore": "0/5", 2024-09-13T11:22:23.600 INFO:teuthology.orchestra.run.smithi001.stdout: "debug_seastore_backref": "0/5", 2024-09-13T11:22:23.600 INFO:teuthology.orchestra.run.smithi001.stdout: "debug_seastore_cache": "0/5", 2024-09-13T11:22:23.600 INFO:teuthology.orchestra.run.smithi001.stdout: "debug_seastore_cleaner": "0/5", 2024-09-13T11:22:23.600 INFO:teuthology.orchestra.run.smithi001.stdout: "debug_seastore_device": "0/5", 2024-09-13T11:22:23.600 INFO:teuthology.orchestra.run.smithi001.stdout: "debug_seastore_epm": "0/5", 2024-09-13T11:22:23.600 INFO:teuthology.orchestra.run.smithi001.stdout: "debug_seastore_fixedkv_tree": "0/5", 2024-09-13T11:22:23.600 INFO:teuthology.orchestra.run.smithi001.stdout: "debug_seastore_journal": "0/5", 2024-09-13T11:22:23.600 INFO:teuthology.orchestra.run.smithi001.stdout: "debug_seastore_lba": "0/5", 2024-09-13T11:22:23.600 INFO:teuthology.orchestra.run.smithi001.stdout: "debug_seastore_odata": "0/5", 2024-09-13T11:22:23.600 INFO:teuthology.orchestra.run.smithi001.stdout: "debug_seastore_omap": "0/5", 2024-09-13T11:22:23.600 INFO:teuthology.orchestra.run.smithi001.stdout: "debug_seastore_onode": "0/5", 2024-09-13T11:22:23.601 INFO:teuthology.orchestra.run.smithi001.stdout: "debug_seastore_t": "0/5", 2024-09-13T11:22:23.601 INFO:teuthology.orchestra.run.smithi001.stdout: "debug_seastore_tm": "0/5", 2024-09-13T11:22:23.601 INFO:teuthology.orchestra.run.smithi001.stdout: "debug_striper": "0/1", 2024-09-13T11:22:23.601 INFO:teuthology.orchestra.run.smithi001.stdout: "debug_test": "0/5", 2024-09-13T11:22:23.601 INFO:teuthology.orchestra.run.smithi001.stdout: "debug_throttle": "1/1", 2024-09-13T11:22:23.601 INFO:teuthology.orchestra.run.smithi001.stdout: "debug_timer": "0/1", 2024-09-13T11:22:23.601 INFO:teuthology.orchestra.run.smithi001.stdout: "debug_tp": "0/5", 2024-09-13T11:22:23.601 INFO:teuthology.orchestra.run.smithi001.stdout: "debug_trace": "1/5", 2024-09-13T11:22:23.601 INFO:teuthology.orchestra.run.smithi001.stdout: "defer_client_eviction_on_laggy_osds": "false", 2024-09-13T11:22:23.601 INFO:teuthology.orchestra.run.smithi001.stdout: "device_failure_prediction_mode": "none", 2024-09-13T11:22:23.601 INFO:teuthology.orchestra.run.smithi001.stdout: "enable_experimental_unrecoverable_data_corrupting_features": "", 2024-09-13T11:22:23.601 INFO:teuthology.orchestra.run.smithi001.stdout: "erasure_code_dir": "/usr/lib/ceph/erasure-code", 2024-09-13T11:22:23.601 INFO:teuthology.orchestra.run.smithi001.stdout: "err_to_graylog": "false", 2024-09-13T11:22:23.602 INFO:teuthology.orchestra.run.smithi001.stdout: "err_to_journald": "false", 2024-09-13T11:22:23.602 INFO:teuthology.orchestra.run.smithi001.stdout: "err_to_stderr": "true", 2024-09-13T11:22:23.602 INFO:teuthology.orchestra.run.smithi001.stdout: "err_to_syslog": "false", 2024-09-13T11:22:23.602 INFO:teuthology.orchestra.run.smithi001.stdout: "event_tracing": "false", 2024-09-13T11:22:23.602 INFO:teuthology.orchestra.run.smithi001.stdout: "exporter_addr": "0.0.0.0", 2024-09-13T11:22:23.602 INFO:teuthology.orchestra.run.smithi001.stdout: "exporter_cert_file": "", 2024-09-13T11:22:23.602 INFO:teuthology.orchestra.run.smithi001.stdout: "exporter_http_port": "9926", 2024-09-13T11:22:23.602 INFO:teuthology.orchestra.run.smithi001.stdout: "exporter_key_file": "", 2024-09-13T11:22:23.602 INFO:teuthology.orchestra.run.smithi001.stdout: "exporter_prio_limit": "5", 2024-09-13T11:22:23.602 INFO:teuthology.orchestra.run.smithi001.stdout: "exporter_sock_dir": "/var/run/ceph/", 2024-09-13T11:22:23.602 INFO:teuthology.orchestra.run.smithi001.stdout: "exporter_sort_metrics": "true", 2024-09-13T11:22:23.602 INFO:teuthology.orchestra.run.smithi001.stdout: "exporter_stats_period": "5", 2024-09-13T11:22:23.602 INFO:teuthology.orchestra.run.smithi001.stdout: "fake_statfs_for_testing": "0", 2024-09-13T11:22:23.603 INFO:teuthology.orchestra.run.smithi001.stdout: "fatal_signal_handlers": "true", 2024-09-13T11:22:23.603 INFO:teuthology.orchestra.run.smithi001.stdout: "filer_max_purge_ops": "10", 2024-09-13T11:22:23.603 INFO:teuthology.orchestra.run.smithi001.stdout: "filer_max_truncate_ops": "128", 2024-09-13T11:22:23.603 INFO:teuthology.orchestra.run.smithi001.stdout: "filestore_apply_finisher_threads": "1", 2024-09-13T11:22:23.603 INFO:teuthology.orchestra.run.smithi001.stdout: "filestore_blackhole": "false", 2024-09-13T11:22:23.603 INFO:teuthology.orchestra.run.smithi001.stdout: "filestore_btrfs_clone_range": "true", 2024-09-13T11:22:23.603 INFO:teuthology.orchestra.run.smithi001.stdout: "filestore_btrfs_snap": "true", 2024-09-13T11:22:23.603 INFO:teuthology.orchestra.run.smithi001.stdout: "filestore_caller_concurrency": "10", 2024-09-13T11:22:23.603 INFO:teuthology.orchestra.run.smithi001.stdout: "filestore_collect_device_partition_information": "true", 2024-09-13T11:22:23.603 INFO:teuthology.orchestra.run.smithi001.stdout: "filestore_commit_timeout": "600.000000", 2024-09-13T11:22:23.603 INFO:teuthology.orchestra.run.smithi001.stdout: "filestore_debug_inject_read_err": "false", 2024-09-13T11:22:23.603 INFO:teuthology.orchestra.run.smithi001.stdout: "filestore_debug_omap_check": "false", 2024-09-13T11:22:23.603 INFO:teuthology.orchestra.run.smithi001.stdout: "filestore_debug_verify_split": "false", 2024-09-13T11:22:23.603 INFO:teuthology.orchestra.run.smithi001.stdout: "filestore_dump_file": "", 2024-09-13T11:22:23.604 INFO:teuthology.orchestra.run.smithi001.stdout: "filestore_expected_throughput_bytes": "209715200.000000", 2024-09-13T11:22:23.604 INFO:teuthology.orchestra.run.smithi001.stdout: "filestore_expected_throughput_ops": "200.000000", 2024-09-13T11:22:23.604 INFO:teuthology.orchestra.run.smithi001.stdout: "filestore_fadvise": "true", 2024-09-13T11:22:23.604 INFO:teuthology.orchestra.run.smithi001.stdout: "filestore_fail_eio": "true", 2024-09-13T11:22:23.604 INFO:teuthology.orchestra.run.smithi001.stdout: "filestore_fd_cache_shards": "16", 2024-09-13T11:22:23.604 INFO:teuthology.orchestra.run.smithi001.stdout: "filestore_fd_cache_size": "128", 2024-09-13T11:22:23.604 INFO:teuthology.orchestra.run.smithi001.stdout: "filestore_fiemap": "false", 2024-09-13T11:22:23.604 INFO:teuthology.orchestra.run.smithi001.stdout: "filestore_fiemap_threshold": "4096", 2024-09-13T11:22:23.604 INFO:teuthology.orchestra.run.smithi001.stdout: "filestore_fsync_flushes_journal_data": "false", 2024-09-13T11:22:23.604 INFO:teuthology.orchestra.run.smithi001.stdout: "filestore_index_retry_probability": "0.000000", 2024-09-13T11:22:23.604 INFO:teuthology.orchestra.run.smithi001.stdout: "filestore_inject_stall": "0", 2024-09-13T11:22:23.604 INFO:teuthology.orchestra.run.smithi001.stdout: "filestore_journal_parallel": "false", 2024-09-13T11:22:23.604 INFO:teuthology.orchestra.run.smithi001.stdout: "filestore_journal_trailing": "false", 2024-09-13T11:22:23.605 INFO:teuthology.orchestra.run.smithi001.stdout: "filestore_journal_writeahead": "false", 2024-09-13T11:22:23.605 INFO:teuthology.orchestra.run.smithi001.stdout: "filestore_kill_at": "0", 2024-09-13T11:22:23.605 INFO:teuthology.orchestra.run.smithi001.stdout: "filestore_max_alloc_hint_size": "1048576", 2024-09-13T11:22:23.605 INFO:teuthology.orchestra.run.smithi001.stdout: "filestore_max_inline_xattr_size": "0", 2024-09-13T11:22:23.605 INFO:teuthology.orchestra.run.smithi001.stdout: "filestore_max_inline_xattr_size_btrfs": "2048", 2024-09-13T11:22:23.605 INFO:teuthology.orchestra.run.smithi001.stdout: "filestore_max_inline_xattr_size_other": "512", 2024-09-13T11:22:23.605 INFO:teuthology.orchestra.run.smithi001.stdout: "filestore_max_inline_xattr_size_xfs": "65536", 2024-09-13T11:22:23.605 INFO:teuthology.orchestra.run.smithi001.stdout: "filestore_max_inline_xattrs": "0", 2024-09-13T11:22:23.605 INFO:teuthology.orchestra.run.smithi001.stdout: "filestore_max_inline_xattrs_btrfs": "10", 2024-09-13T11:22:23.605 INFO:teuthology.orchestra.run.smithi001.stdout: "filestore_max_inline_xattrs_other": "2", 2024-09-13T11:22:23.605 INFO:teuthology.orchestra.run.smithi001.stdout: "filestore_max_inline_xattrs_xfs": "10", 2024-09-13T11:22:23.605 INFO:teuthology.orchestra.run.smithi001.stdout: "filestore_max_sync_interval": "5.000000", 2024-09-13T11:22:23.605 INFO:teuthology.orchestra.run.smithi001.stdout: "filestore_max_xattr_value_size": "0", 2024-09-13T11:22:23.606 INFO:teuthology.orchestra.run.smithi001.stdout: "filestore_max_xattr_value_size_btrfs": "65536", 2024-09-13T11:22:23.606 INFO:teuthology.orchestra.run.smithi001.stdout: "filestore_max_xattr_value_size_other": "1024", 2024-09-13T11:22:23.606 INFO:teuthology.orchestra.run.smithi001.stdout: "filestore_max_xattr_value_size_xfs": "65536", 2024-09-13T11:22:23.606 INFO:teuthology.orchestra.run.smithi001.stdout: "filestore_merge_threshold": "-10", 2024-09-13T11:22:23.606 INFO:teuthology.orchestra.run.smithi001.stdout: "filestore_min_sync_interval": "0.010000", 2024-09-13T11:22:23.606 INFO:teuthology.orchestra.run.smithi001.stdout: "filestore_odsync_write": "false", 2024-09-13T11:22:23.606 INFO:teuthology.orchestra.run.smithi001.stdout: "filestore_omap_backend": "rocksdb", 2024-09-13T11:22:23.606 INFO:teuthology.orchestra.run.smithi001.stdout: "filestore_omap_backend_path": "", 2024-09-13T11:22:23.606 INFO:teuthology.orchestra.run.smithi001.stdout: "filestore_omap_header_cache_size": "1024", 2024-09-13T11:22:23.606 INFO:teuthology.orchestra.run.smithi001.stdout: "filestore_ondisk_finisher_threads": "1", 2024-09-13T11:22:23.607 INFO:teuthology.orchestra.run.smithi001.stdout: "filestore_op_thread_suicide_timeout": "180", 2024-09-13T11:22:23.607 INFO:teuthology.orchestra.run.smithi001.stdout: "filestore_op_thread_timeout": "60", 2024-09-13T11:22:23.607 INFO:teuthology.orchestra.run.smithi001.stdout: "filestore_op_threads": "2", 2024-09-13T11:22:23.607 INFO:teuthology.orchestra.run.smithi001.stdout: "filestore_punch_hole": "false", 2024-09-13T11:22:23.607 INFO:teuthology.orchestra.run.smithi001.stdout: "filestore_queue_high_delay_multiple": "0.000000", 2024-09-13T11:22:23.607 INFO:teuthology.orchestra.run.smithi001.stdout: "filestore_queue_high_delay_multiple_bytes": "0.000000", 2024-09-13T11:22:23.607 INFO:teuthology.orchestra.run.smithi001.stdout: "filestore_queue_high_delay_multiple_ops": "0.000000", 2024-09-13T11:22:23.607 INFO:teuthology.orchestra.run.smithi001.stdout: "filestore_queue_high_threshhold": "0.900000", 2024-09-13T11:22:23.607 INFO:teuthology.orchestra.run.smithi001.stdout: "filestore_queue_low_threshhold": "0.300000", 2024-09-13T11:22:23.607 INFO:teuthology.orchestra.run.smithi001.stdout: "filestore_queue_max_bytes": "104857600", 2024-09-13T11:22:23.607 INFO:teuthology.orchestra.run.smithi001.stdout: "filestore_queue_max_delay_multiple": "0.000000", 2024-09-13T11:22:23.607 INFO:teuthology.orchestra.run.smithi001.stdout: "filestore_queue_max_delay_multiple_bytes": "0.000000", 2024-09-13T11:22:23.607 INFO:teuthology.orchestra.run.smithi001.stdout: "filestore_queue_max_delay_multiple_ops": "0.000000", 2024-09-13T11:22:23.608 INFO:teuthology.orchestra.run.smithi001.stdout: "filestore_queue_max_ops": "50", 2024-09-13T11:22:23.608 INFO:teuthology.orchestra.run.smithi001.stdout: "filestore_rocksdb_options": "max_background_jobs=10,compaction_readahead_size=2097152,compression=kNoCompression", 2024-09-13T11:22:23.608 INFO:teuthology.orchestra.run.smithi001.stdout: "filestore_seek_data_hole": "false", 2024-09-13T11:22:23.608 INFO:teuthology.orchestra.run.smithi001.stdout: "filestore_sloppy_crc": "false", 2024-09-13T11:22:23.608 INFO:teuthology.orchestra.run.smithi001.stdout: "filestore_sloppy_crc_block_size": "65536", 2024-09-13T11:22:23.608 INFO:teuthology.orchestra.run.smithi001.stdout: "filestore_splice": "false", 2024-09-13T11:22:23.608 INFO:teuthology.orchestra.run.smithi001.stdout: "filestore_split_multiple": "2", 2024-09-13T11:22:23.608 INFO:teuthology.orchestra.run.smithi001.stdout: "filestore_split_rand_factor": "20", 2024-09-13T11:22:23.608 INFO:teuthology.orchestra.run.smithi001.stdout: "filestore_update_to": "1000", 2024-09-13T11:22:23.608 INFO:teuthology.orchestra.run.smithi001.stdout: "filestore_wbthrottle_btrfs_bytes_hard_limit": "419430400", 2024-09-13T11:22:23.608 INFO:teuthology.orchestra.run.smithi001.stdout: "filestore_wbthrottle_btrfs_bytes_start_flusher": "41943040", 2024-09-13T11:22:23.608 INFO:teuthology.orchestra.run.smithi001.stdout: "filestore_wbthrottle_btrfs_inodes_hard_limit": "5000", 2024-09-13T11:22:23.608 INFO:teuthology.orchestra.run.smithi001.stdout: "filestore_wbthrottle_btrfs_inodes_start_flusher": "500", 2024-09-13T11:22:23.609 INFO:teuthology.orchestra.run.smithi001.stdout: "filestore_wbthrottle_btrfs_ios_hard_limit": "5000", 2024-09-13T11:22:23.609 INFO:teuthology.orchestra.run.smithi001.stdout: "filestore_wbthrottle_btrfs_ios_start_flusher": "500", 2024-09-13T11:22:23.609 INFO:teuthology.orchestra.run.smithi001.stdout: "filestore_wbthrottle_enable": "true", 2024-09-13T11:22:23.609 INFO:teuthology.orchestra.run.smithi001.stdout: "filestore_wbthrottle_xfs_bytes_hard_limit": "419430400", 2024-09-13T11:22:23.609 INFO:teuthology.orchestra.run.smithi001.stdout: "filestore_wbthrottle_xfs_bytes_start_flusher": "41943040", 2024-09-13T11:22:23.609 INFO:teuthology.orchestra.run.smithi001.stdout: "filestore_wbthrottle_xfs_inodes_hard_limit": "5000", 2024-09-13T11:22:23.609 INFO:teuthology.orchestra.run.smithi001.stdout: "filestore_wbthrottle_xfs_inodes_start_flusher": "500", 2024-09-13T11:22:23.609 INFO:teuthology.orchestra.run.smithi001.stdout: "filestore_wbthrottle_xfs_ios_hard_limit": "5000", 2024-09-13T11:22:23.609 INFO:teuthology.orchestra.run.smithi001.stdout: "filestore_wbthrottle_xfs_ios_start_flusher": "500", 2024-09-13T11:22:23.609 INFO:teuthology.orchestra.run.smithi001.stdout: "filestore_xfs_extsize": "false", 2024-09-13T11:22:23.609 INFO:teuthology.orchestra.run.smithi001.stdout: "filestore_zfs_snap": "false", 2024-09-13T11:22:23.609 INFO:teuthology.orchestra.run.smithi001.stdout: "fio_dir": "/tmp/fio", 2024-09-13T11:22:23.609 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "2fd68ac7-5581-4d90-80a4-30bcf5252a7a", 2024-09-13T11:22:23.609 INFO:teuthology.orchestra.run.smithi001.stdout: "fuse_allow_other": "true", 2024-09-13T11:22:23.610 INFO:teuthology.orchestra.run.smithi001.stdout: "fuse_atomic_o_trunc": "true", 2024-09-13T11:22:23.610 INFO:teuthology.orchestra.run.smithi001.stdout: "fuse_big_writes": "true", 2024-09-13T11:22:23.610 INFO:teuthology.orchestra.run.smithi001.stdout: "fuse_debug": "false", 2024-09-13T11:22:23.610 INFO:teuthology.orchestra.run.smithi001.stdout: "fuse_default_permissions": "false", 2024-09-13T11:22:23.610 INFO:teuthology.orchestra.run.smithi001.stdout: "fuse_disable_pagecache": "false", 2024-09-13T11:22:23.610 INFO:teuthology.orchestra.run.smithi001.stdout: "fuse_max_write": "0", 2024-09-13T11:22:23.610 INFO:teuthology.orchestra.run.smithi001.stdout: "fuse_multithreaded": "true", 2024-09-13T11:22:23.610 INFO:teuthology.orchestra.run.smithi001.stdout: "fuse_require_active_mds": "true", 2024-09-13T11:22:23.610 INFO:teuthology.orchestra.run.smithi001.stdout: "fuse_set_user_groups": "true", 2024-09-13T11:22:23.610 INFO:teuthology.orchestra.run.smithi001.stdout: "fuse_splice_move": "true", 2024-09-13T11:22:23.610 INFO:teuthology.orchestra.run.smithi001.stdout: "fuse_splice_read": "true", 2024-09-13T11:22:23.610 INFO:teuthology.orchestra.run.smithi001.stdout: "fuse_splice_write": "true", 2024-09-13T11:22:23.610 INFO:teuthology.orchestra.run.smithi001.stdout: "fuse_syncfs_on_mksnap": "true", 2024-09-13T11:22:23.610 INFO:teuthology.orchestra.run.smithi001.stdout: "fuse_use_invalidate_cb": "true", 2024-09-13T11:22:23.611 INFO:teuthology.orchestra.run.smithi001.stdout: "gss_ktab_client_file": "/var/lib/ceph/mon.a/gss_client_mon.a.ktab", 2024-09-13T11:22:23.611 INFO:teuthology.orchestra.run.smithi001.stdout: "gss_target_name": "ceph", 2024-09-13T11:22:23.611 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_file": "", 2024-09-13T11:22:23.611 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_inject_failure": "0", 2024-09-13T11:22:23.611 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_interval": "5", 2024-09-13T11:22:23.611 INFO:teuthology.orchestra.run.smithi001.stdout: "host": "smithi001", 2024-09-13T11:22:23.611 INFO:teuthology.orchestra.run.smithi001.stdout: "immutable_object_cache_client_dedicated_thread_num": "2", 2024-09-13T11:22:23.611 INFO:teuthology.orchestra.run.smithi001.stdout: "immutable_object_cache_max_inflight_ops": "128", 2024-09-13T11:22:23.612 INFO:teuthology.orchestra.run.smithi001.stdout: "immutable_object_cache_max_size": "1073741824", 2024-09-13T11:22:23.612 INFO:teuthology.orchestra.run.smithi001.stdout: "immutable_object_cache_path": "/tmp/ceph_immutable_object_cache", 2024-09-13T11:22:23.612 INFO:teuthology.orchestra.run.smithi001.stdout: "immutable_object_cache_qos_bps_burst": "0", 2024-09-13T11:22:23.612 INFO:teuthology.orchestra.run.smithi001.stdout: "immutable_object_cache_qos_bps_burst_seconds": "1", 2024-09-13T11:22:23.612 INFO:teuthology.orchestra.run.smithi001.stdout: "immutable_object_cache_qos_bps_limit": "0", 2024-09-13T11:22:23.612 INFO:teuthology.orchestra.run.smithi001.stdout: "immutable_object_cache_qos_iops_burst": "0", 2024-09-13T11:22:23.612 INFO:teuthology.orchestra.run.smithi001.stdout: "immutable_object_cache_qos_iops_burst_seconds": "1", 2024-09-13T11:22:23.612 INFO:teuthology.orchestra.run.smithi001.stdout: "immutable_object_cache_qos_iops_limit": "0", 2024-09-13T11:22:23.612 INFO:teuthology.orchestra.run.smithi001.stdout: "immutable_object_cache_qos_schedule_tick_min": "50", 2024-09-13T11:22:23.612 INFO:teuthology.orchestra.run.smithi001.stdout: "immutable_object_cache_sock": "/var/run/ceph/immutable_object_cache_sock", 2024-09-13T11:22:23.612 INFO:teuthology.orchestra.run.smithi001.stdout: "immutable_object_cache_watermark": "0.900000", 2024-09-13T11:22:23.612 INFO:teuthology.orchestra.run.smithi001.stdout: "inject_early_sigterm": "false", 2024-09-13T11:22:23.612 INFO:teuthology.orchestra.run.smithi001.stdout: "jaeger_agent_port": "6799", 2024-09-13T11:22:23.612 INFO:teuthology.orchestra.run.smithi001.stdout: "jaeger_tracing_enable": "false", 2024-09-13T11:22:23.613 INFO:teuthology.orchestra.run.smithi001.stdout: "journal_aio": "true", 2024-09-13T11:22:23.613 INFO:teuthology.orchestra.run.smithi001.stdout: "journal_align_min_size": "65536", 2024-09-13T11:22:23.613 INFO:teuthology.orchestra.run.smithi001.stdout: "journal_block_align": "true", 2024-09-13T11:22:23.613 INFO:teuthology.orchestra.run.smithi001.stdout: "journal_block_size": "4096", 2024-09-13T11:22:23.613 INFO:teuthology.orchestra.run.smithi001.stdout: "journal_dio": "true", 2024-09-13T11:22:23.613 INFO:teuthology.orchestra.run.smithi001.stdout: "journal_discard": "false", 2024-09-13T11:22:23.613 INFO:teuthology.orchestra.run.smithi001.stdout: "journal_force_aio": "false", 2024-09-13T11:22:23.613 INFO:teuthology.orchestra.run.smithi001.stdout: "journal_ignore_corruption": "false", 2024-09-13T11:22:23.613 INFO:teuthology.orchestra.run.smithi001.stdout: "journal_max_write_bytes": "10485760", 2024-09-13T11:22:23.613 INFO:teuthology.orchestra.run.smithi001.stdout: "journal_max_write_entries": "100", 2024-09-13T11:22:23.613 INFO:teuthology.orchestra.run.smithi001.stdout: "journal_replay_from": "0", 2024-09-13T11:22:23.613 INFO:teuthology.orchestra.run.smithi001.stdout: "journal_throttle_high_multiple": "0.000000", 2024-09-13T11:22:23.613 INFO:teuthology.orchestra.run.smithi001.stdout: "journal_throttle_high_threshhold": "0.900000", 2024-09-13T11:22:23.613 INFO:teuthology.orchestra.run.smithi001.stdout: "journal_throttle_low_threshhold": "0.600000", 2024-09-13T11:22:23.614 INFO:teuthology.orchestra.run.smithi001.stdout: "journal_throttle_max_multiple": "0.000000", 2024-09-13T11:22:23.614 INFO:teuthology.orchestra.run.smithi001.stdout: "journal_write_header_frequency": "0", 2024-09-13T11:22:23.614 INFO:teuthology.orchestra.run.smithi001.stdout: "journal_zero_on_create": "false", 2024-09-13T11:22:23.614 INFO:teuthology.orchestra.run.smithi001.stdout: "journaler_prefetch_periods": "10", 2024-09-13T11:22:23.614 INFO:teuthology.orchestra.run.smithi001.stdout: "journaler_prezero_periods": "5", 2024-09-13T11:22:23.614 INFO:teuthology.orchestra.run.smithi001.stdout: "journaler_write_head_interval": "15", 2024-09-13T11:22:23.614 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "", 2024-09-13T11:22:23.614 INFO:teuthology.orchestra.run.smithi001.stdout: "keyfile": "", 2024-09-13T11:22:23.614 INFO:teuthology.orchestra.run.smithi001.stdout: "keyring": "/var/lib/ceph/mon/ceph-a/keyring", 2024-09-13T11:22:23.614 INFO:teuthology.orchestra.run.smithi001.stdout: "kstore_backend": "rocksdb", 2024-09-13T11:22:23.614 INFO:teuthology.orchestra.run.smithi001.stdout: "kstore_default_stripe_size": "65536", 2024-09-13T11:22:23.614 INFO:teuthology.orchestra.run.smithi001.stdout: "kstore_fsck_on_mount": "false", 2024-09-13T11:22:23.614 INFO:teuthology.orchestra.run.smithi001.stdout: "kstore_fsck_on_mount_deep": "true", 2024-09-13T11:22:23.615 INFO:teuthology.orchestra.run.smithi001.stdout: "kstore_max_bytes": "67108864", 2024-09-13T11:22:23.615 INFO:teuthology.orchestra.run.smithi001.stdout: "kstore_max_ops": "512", 2024-09-13T11:22:23.615 INFO:teuthology.orchestra.run.smithi001.stdout: "kstore_nid_prealloc": "1024", 2024-09-13T11:22:23.615 INFO:teuthology.orchestra.run.smithi001.stdout: "kstore_onode_map_size": "1024", 2024-09-13T11:22:23.615 INFO:teuthology.orchestra.run.smithi001.stdout: "kstore_rocksdb_options": "compression=kNoCompression", 2024-09-13T11:22:23.615 INFO:teuthology.orchestra.run.smithi001.stdout: "kstore_sync_submit_transaction": "false", 2024-09-13T11:22:23.615 INFO:teuthology.orchestra.run.smithi001.stdout: "kstore_sync_transaction": "false", 2024-09-13T11:22:23.615 INFO:teuthology.orchestra.run.smithi001.stdout: "librados_thread_count": "2", 2024-09-13T11:22:23.615 INFO:teuthology.orchestra.run.smithi001.stdout: "lockdep": "false", 2024-09-13T11:22:23.615 INFO:teuthology.orchestra.run.smithi001.stdout: "lockdep_force_backtrace": "false", 2024-09-13T11:22:23.615 INFO:teuthology.orchestra.run.smithi001.stdout: "log_coarse_timestamps": "true", 2024-09-13T11:22:23.615 INFO:teuthology.orchestra.run.smithi001.stdout: "log_file": "/var/log/ceph/ceph-mon.a.log", 2024-09-13T11:22:23.615 INFO:teuthology.orchestra.run.smithi001.stdout: "log_flush_on_exit": "false", 2024-09-13T11:22:23.615 INFO:teuthology.orchestra.run.smithi001.stdout: "log_graylog_host": "127.0.0.1", 2024-09-13T11:22:23.616 INFO:teuthology.orchestra.run.smithi001.stdout: "log_graylog_port": "12201", 2024-09-13T11:22:23.616 INFO:teuthology.orchestra.run.smithi001.stdout: "log_max_new": "1000", 2024-09-13T11:22:23.616 INFO:teuthology.orchestra.run.smithi001.stdout: "log_max_recent": "10000", 2024-09-13T11:22:23.616 INFO:teuthology.orchestra.run.smithi001.stdout: "log_stderr_prefix": "", 2024-09-13T11:22:23.616 INFO:teuthology.orchestra.run.smithi001.stdout: "log_stop_at_utilization": "0.970000", 2024-09-13T11:22:23.616 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_file": "true", 2024-09-13T11:22:23.616 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_graylog": "false", 2024-09-13T11:22:23.616 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_journald": "false", 2024-09-13T11:22:23.616 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_stderr": "false", 2024-09-13T11:22:23.616 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_syslog": "false", 2024-09-13T11:22:23.616 INFO:teuthology.orchestra.run.smithi001.stdout: "max_rotating_auth_attempts": "10", 2024-09-13T11:22:23.616 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_abort_on_newly_corrupt_dentry": "true", 2024-09-13T11:22:23.616 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_action_on_write_error": "1", 2024-09-13T11:22:23.617 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_alternate_name_max": "8192", 2024-09-13T11:22:23.617 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_asio_thread_count": "2", 2024-09-13T11:22:23.617 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_bal_export_pin": "true", 2024-09-13T11:22:23.617 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_bal_fragment_dirs": "true", 2024-09-13T11:22:23.617 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_bal_fragment_fast_factor": "1.500000", 2024-09-13T11:22:23.617 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_bal_fragment_interval": "5", 2024-09-13T11:22:23.617 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_bal_fragment_size_max": "100000", 2024-09-13T11:22:23.617 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_bal_idle_threshold": "0.000000", 2024-09-13T11:22:23.617 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_bal_interval": "10", 2024-09-13T11:22:23.617 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_bal_max": "-1", 2024-09-13T11:22:23.617 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_bal_max_until": "-1", 2024-09-13T11:22:23.617 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_bal_merge_size": "50", 2024-09-13T11:22:23.617 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_bal_midchunk": "0.300000", 2024-09-13T11:22:23.617 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_bal_min_rebalance": "0.100000", 2024-09-13T11:22:23.618 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_bal_min_start": "0.200000", 2024-09-13T11:22:23.618 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_bal_minchunk": "0.001000", 2024-09-13T11:22:23.618 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_bal_mode": "0", 2024-09-13T11:22:23.618 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_bal_need_max": "1.200000", 2024-09-13T11:22:23.618 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_bal_need_min": "0.800000", 2024-09-13T11:22:23.618 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_bal_overload_epochs": "2", 2024-09-13T11:22:23.618 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_bal_replicate_threshold": "8000.000000", 2024-09-13T11:22:23.618 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_bal_sample_interval": "3.000000", 2024-09-13T11:22:23.618 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_bal_split_bits": "3", 2024-09-13T11:22:23.618 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_bal_split_rd": "25000.000000", 2024-09-13T11:22:23.618 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_bal_split_size": "10000", 2024-09-13T11:22:23.618 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_bal_split_wr": "10000.000000", 2024-09-13T11:22:23.618 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_bal_target_decay": "10.000000", 2024-09-13T11:22:23.618 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_bal_unreplicate_threshold": "0.000000", 2024-09-13T11:22:23.619 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_beacon_grace": "15.000000", 2024-09-13T11:22:23.619 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_beacon_interval": "4.000000", 2024-09-13T11:22:23.619 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_beacon_mon_down_grace": "60", 2024-09-13T11:22:23.619 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_cache_memory_limit": "4294967296", 2024-09-13T11:22:23.619 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_cache_mid": "0.700000", 2024-09-13T11:22:23.619 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_cache_quiesce_decay_rate": "1.000000", 2024-09-13T11:22:23.619 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_cache_quiesce_delay": "0", 2024-09-13T11:22:23.619 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_cache_quiesce_sleep": "200", 2024-09-13T11:22:23.619 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_cache_quiesce_splitauth": "true", 2024-09-13T11:22:23.619 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_cache_quiesce_threshold": "524288", 2024-09-13T11:22:23.619 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_cache_release_free_interval": "10", 2024-09-13T11:22:23.619 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_cache_reservation": "0.050000", 2024-09-13T11:22:23.619 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_cache_trim_decay_rate": "1.000000", 2024-09-13T11:22:23.619 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_cache_trim_interval": "1", 2024-09-13T11:22:23.620 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_cache_trim_threshold": "262144", 2024-09-13T11:22:23.620 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_cap_acquisition_throttle_retry_request_timeout": "0.500000", 2024-09-13T11:22:23.620 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_cap_revoke_eviction_timeout": "0.000000", 2024-09-13T11:22:23.620 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_client_delegate_inos_pct": "50", 2024-09-13T11:22:23.620 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_client_prealloc_inos": "1000", 2024-09-13T11:22:23.620 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_client_writeable_range_max_inc_objs": "1024", 2024-09-13T11:22:23.620 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_connect_bootstrapping": "false", 2024-09-13T11:22:23.620 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_damage_table_max_entries": "10000", 2024-09-13T11:22:23.620 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_data": "/var/lib/ceph/mds/ceph-a", 2024-09-13T11:22:23.620 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_debug_auth_pins": "false", 2024-09-13T11:22:23.620 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_debug_frag": "false", 2024-09-13T11:22:23.620 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_debug_scatterstat": "false", 2024-09-13T11:22:23.620 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_debug_subtrees": "false", 2024-09-13T11:22:23.620 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_decay_halflife": "5.000000", 2024-09-13T11:22:23.621 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_default_dir_hash": "2", 2024-09-13T11:22:23.621 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_defer_session_stale": "true", 2024-09-13T11:22:23.621 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_deny_all_reconnect": "false", 2024-09-13T11:22:23.621 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_dir_keys_per_op": "16384", 2024-09-13T11:22:23.621 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_dir_max_commit_size": "10", 2024-09-13T11:22:23.621 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_dir_max_entries": "0", 2024-09-13T11:22:23.621 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_dir_prefetch": "true", 2024-09-13T11:22:23.621 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_dirstat_min_interval": "1.000000", 2024-09-13T11:22:23.621 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_dump_cache_after_rejoin": "false", 2024-09-13T11:22:23.621 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_dump_cache_on_map": "false", 2024-09-13T11:22:23.621 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_dump_cache_threshold_file": "0", 2024-09-13T11:22:23.621 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_dump_cache_threshold_formatter": "1073741824", 2024-09-13T11:22:23.621 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_early_reply": "true", 2024-09-13T11:22:23.622 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_enable_op_tracker": "true", 2024-09-13T11:22:23.622 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_enforce_unique_name": "true", 2024-09-13T11:22:23.622 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_export_ephemeral_distributed": "true", 2024-09-13T11:22:23.622 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_export_ephemeral_distributed_factor": "2.000000", 2024-09-13T11:22:23.622 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_export_ephemeral_random": "true", 2024-09-13T11:22:23.622 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_export_ephemeral_random_max": "0.010000", 2024-09-13T11:22:23.622 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_extraordinary_events_dump_interval": "0", 2024-09-13T11:22:23.622 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_forward_all_requests_to_auth": "false", 2024-09-13T11:22:23.622 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_freeze_tree_timeout": "30.000000", 2024-09-13T11:22:23.622 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_fscrypt_last_block_max_size": "4096", 2024-09-13T11:22:23.622 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_go_bad_corrupt_dentry": "true", 2024-09-13T11:22:23.622 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_hack_allow_loading_invalid_metadata": "false", 2024-09-13T11:22:23.622 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_health_cache_threshold": "1.500000", 2024-09-13T11:22:23.622 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_health_summarize_threshold": "10", 2024-09-13T11:22:23.623 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_heartbeat_grace": "15.000000", 2024-09-13T11:22:23.623 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_heartbeat_reset_grace": "1000", 2024-09-13T11:22:23.623 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_inject_health_dummy": "false", 2024-09-13T11:22:23.623 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_inject_journal_corrupt_dentry_first": "0.000000", 2024-09-13T11:22:23.623 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_inject_migrator_session_race": "false", 2024-09-13T11:22:23.623 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_inject_rename_corrupt_dentry_first": "0.000000", 2024-09-13T11:22:23.623 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_inject_skip_replaying_inotable": "false", 2024-09-13T11:22:23.623 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_inject_traceless_reply_probability": "0.000000", 2024-09-13T11:22:23.623 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_join_fs": "", 2024-09-13T11:22:23.623 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_journal_format": "1", 2024-09-13T11:22:23.623 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_kill_after_journal_logs_flushed": "false", 2024-09-13T11:22:23.623 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_kill_create_at": "0", 2024-09-13T11:22:23.623 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_kill_dirfrag_at": "0", 2024-09-13T11:22:23.623 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_kill_export_at": "0", 2024-09-13T11:22:23.624 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_kill_import_at": "0", 2024-09-13T11:22:23.624 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_kill_journal_at": "0", 2024-09-13T11:22:23.624 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_kill_journal_expire_at": "0", 2024-09-13T11:22:23.624 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_kill_journal_replay_at": "0", 2024-09-13T11:22:23.624 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_kill_link_at": "0", 2024-09-13T11:22:23.624 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_kill_mdstable_at": "0", 2024-09-13T11:22:23.624 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_kill_openc_at": "0", 2024-09-13T11:22:23.624 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_kill_rename_at": "0", 2024-09-13T11:22:23.624 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_kill_shutdown_at": "0", 2024-09-13T11:22:23.624 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_log_event_large_threshold": "524288", 2024-09-13T11:22:23.624 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_log_events_per_segment": "1024", 2024-09-13T11:22:23.624 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_log_major_segment_event_ratio": "12", 2024-09-13T11:22:23.624 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_log_max_events": "-1", 2024-09-13T11:22:23.625 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_log_max_segments": "128", 2024-09-13T11:22:23.625 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_log_pause": "false", 2024-09-13T11:22:23.625 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_log_segment_size": "0", 2024-09-13T11:22:23.625 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_log_skip_corrupt_events": "false", 2024-09-13T11:22:23.625 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_log_skip_unbounded_events": "false", 2024-09-13T11:22:23.625 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_log_trim_decay_rate": "1.000000", 2024-09-13T11:22:23.625 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_log_trim_threshold": "128", 2024-09-13T11:22:23.625 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_log_trim_upkeep_interval": "1000", 2024-09-13T11:22:23.625 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_log_warn_factor": "2.000000", 2024-09-13T11:22:23.625 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_max_caps_per_client": "1048576", 2024-09-13T11:22:23.625 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_max_completed_flushes": "100000", 2024-09-13T11:22:23.625 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_max_completed_requests": "100000", 2024-09-13T11:22:23.625 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_max_export_size": "20971520", 2024-09-13T11:22:23.625 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_max_file_recover": "32", 2024-09-13T11:22:23.625 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_max_purge_files": "64", 2024-09-13T11:22:23.626 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_max_purge_ops": "8192", 2024-09-13T11:22:23.626 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_max_purge_ops_per_pg": "0.500000", 2024-09-13T11:22:23.626 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_max_scrub_ops_in_progress": "5", 2024-09-13T11:22:23.626 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_max_snaps_per_dir": "100", 2024-09-13T11:22:23.626 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_metrics_update_interval": "2", 2024-09-13T11:22:23.626 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_min_caps_per_client": "100", 2024-09-13T11:22:23.626 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_min_caps_working_set": "10000", 2024-09-13T11:22:23.626 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_mon_shutdown_timeout": "5.000000", 2024-09-13T11:22:23.626 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_numa_node": "-1", 2024-09-13T11:22:23.626 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_oft_prefetch_dirfrags": "false", 2024-09-13T11:22:23.626 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_op_complaint_time": "30.000000", 2024-09-13T11:22:23.626 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_op_history_duration": "600", 2024-09-13T11:22:23.626 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_op_history_size": "20", 2024-09-13T11:22:23.627 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_op_history_slow_op_size": "20", 2024-09-13T11:22:23.627 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_op_history_slow_op_threshold": "10", 2024-09-13T11:22:23.627 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_op_log_threshold": "5", 2024-09-13T11:22:23.627 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_ping_grace": "15", 2024-09-13T11:22:23.627 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_ping_interval": "5", 2024-09-13T11:22:23.627 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_purge_queue_busy_flush_period": "1.000000", 2024-09-13T11:22:23.627 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_recall_global_max_decay_threshold": "131072", 2024-09-13T11:22:23.627 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_recall_max_caps": "30000", 2024-09-13T11:22:23.627 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_recall_max_decay_rate": "1.500000", 2024-09-13T11:22:23.627 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_recall_max_decay_threshold": "131072", 2024-09-13T11:22:23.627 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_recall_warning_decay_rate": "60.000000", 2024-09-13T11:22:23.627 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_recall_warning_threshold": "262144", 2024-09-13T11:22:23.627 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_reconnect_timeout": "45.000000", 2024-09-13T11:22:23.627 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_replay_interval": "1.000000", 2024-09-13T11:22:23.628 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_replay_unsafe_with_closed_session": "false", 2024-09-13T11:22:23.628 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_request_load_average_decay_rate": "60.000000", 2024-09-13T11:22:23.628 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_root_ino_gid": "0", 2024-09-13T11:22:23.628 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_root_ino_uid": "0", 2024-09-13T11:22:23.628 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_scatter_nudge_interval": "5.000000", 2024-09-13T11:22:23.628 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_server_dispatch_client_request_delay": "0", 2024-09-13T11:22:23.628 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_server_dispatch_killpoint_random": "0.000000", 2024-09-13T11:22:23.628 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_session_blocklist_on_evict": "true", 2024-09-13T11:22:23.628 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_session_blocklist_on_timeout": "true", 2024-09-13T11:22:23.628 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_session_cache_liveness_decay_rate": "300.000000", 2024-09-13T11:22:23.628 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_session_cache_liveness_magnitude": "10", 2024-09-13T11:22:23.628 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_session_cap_acquisition_decay_rate": "30.000000", 2024-09-13T11:22:23.628 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_session_cap_acquisition_throttle": "100000", 2024-09-13T11:22:23.629 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_session_max_caps_throttle_ratio": "1.100000", 2024-09-13T11:22:23.629 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_session_metadata_threshold": "16777216", 2024-09-13T11:22:23.629 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_sessionmap_keys_per_op": "1024", 2024-09-13T11:22:23.629 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_shutdown_check": "0", 2024-09-13T11:22:23.629 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_skip_ino": "0", 2024-09-13T11:22:23.629 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_sleep_rank_change": "0.000000", 2024-09-13T11:22:23.629 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_snap_max_uid": "4294967294", 2024-09-13T11:22:23.629 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_snap_min_uid": "0", 2024-09-13T11:22:23.629 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_snap_rstat": "false", 2024-09-13T11:22:23.629 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_standby_replay_damaged": "false", 2024-09-13T11:22:23.629 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_symlink_recovery": "true", 2024-09-13T11:22:23.629 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_task_status_update_interval": "2.000000", 2024-09-13T11:22:23.629 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_thrash_exports": "0", 2024-09-13T11:22:23.630 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_thrash_fragments": "0", 2024-09-13T11:22:23.630 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_tick_interval": "5.000000", 2024-09-13T11:22:23.630 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_valgrind_exit": "false", 2024-09-13T11:22:23.630 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_verify_backtrace": "1", 2024-09-13T11:22:23.630 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_verify_scatter": "false", 2024-09-13T11:22:23.630 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_wipe_ino_prealloc": "false", 2024-09-13T11:22:23.630 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_wipe_sessions": "false", 2024-09-13T11:22:23.630 INFO:teuthology.orchestra.run.smithi001.stdout: "mempool_debug": "false", 2024-09-13T11:22:23.630 INFO:teuthology.orchestra.run.smithi001.stdout: "memstore_debug_omit_block_device_write": "false", 2024-09-13T11:22:23.630 INFO:teuthology.orchestra.run.smithi001.stdout: "memstore_device_bytes": "1073741824", 2024-09-13T11:22:23.630 INFO:teuthology.orchestra.run.smithi001.stdout: "memstore_page_set": "false", 2024-09-13T11:22:23.630 INFO:teuthology.orchestra.run.smithi001.stdout: "memstore_page_size": "65536", 2024-09-13T11:22:23.630 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr_client_bytes": "134217728", 2024-09-13T11:22:23.630 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr_client_messages": "512", 2024-09-13T11:22:23.631 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr_client_service_daemon_unregister_timeout": "1.000000", 2024-09-13T11:22:23.631 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr_connect_retry_interval": "1.000000", 2024-09-13T11:22:23.631 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr_data": "/var/lib/ceph/mgr/ceph-a", 2024-09-13T11:22:23.631 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr_debug_aggressive_pg_num_changes": "false", 2024-09-13T11:22:23.631 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr_disabled_modules": "", 2024-09-13T11:22:23.631 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr_enable_op_tracker": "true", 2024-09-13T11:22:23.631 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr_initial_modules": "restful iostat nfs", 2024-09-13T11:22:23.631 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr_max_pg_creating": "1024", 2024-09-13T11:22:23.631 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr_max_pg_num_change": "128", 2024-09-13T11:22:23.631 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr_mds_bytes": "134217728", 2024-09-13T11:22:23.631 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr_mds_messages": "128", 2024-09-13T11:22:23.631 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr_module_path": "/usr/share/ceph/mgr", 2024-09-13T11:22:23.631 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr_mon_bytes": "134217728", 2024-09-13T11:22:23.631 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr_mon_messages": "128", 2024-09-13T11:22:23.632 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr_num_op_tracker_shard": "32", 2024-09-13T11:22:23.632 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr_op_complaint_time": "30.000000", 2024-09-13T11:22:23.632 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr_op_history_duration": "600", 2024-09-13T11:22:23.632 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr_op_history_size": "20", 2024-09-13T11:22:23.632 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr_op_history_slow_op_size": "20", 2024-09-13T11:22:23.632 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr_op_history_slow_op_threshold": "10.000000", 2024-09-13T11:22:23.632 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr_op_log_threshold": "5", 2024-09-13T11:22:23.632 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr_osd_bytes": "536870912", 2024-09-13T11:22:23.632 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr_osd_messages": "8192", 2024-09-13T11:22:23.632 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr_pool": "true", 2024-09-13T11:22:23.632 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr_service_beacon_grace": "60.000000", 2024-09-13T11:22:23.633 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr_standby_modules": "true", 2024-09-13T11:22:23.633 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr_stats_period": "5", 2024-09-13T11:22:23.633 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr_stats_threshold": "5", 2024-09-13T11:22:23.633 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr_tick_period": "2", 2024-09-13T11:22:23.633 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr_ttl_cache_expire_seconds": "0", 2024-09-13T11:22:23.633 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_accept_timeout_factor": "2.000000", 2024-09-13T11:22:23.633 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_allow_pool_delete": "true", 2024-09-13T11:22:23.633 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_allow_pool_size_one": "true", 2024-09-13T11:22:23.633 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_auth_validate_all_caps": "true", 2024-09-13T11:22:23.633 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_cache_target_full_warn_ratio": "0.660000", 2024-09-13T11:22:23.633 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_clean_pg_upmaps_per_chunk": "256", 2024-09-13T11:22:23.633 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_client_bytes": "104857600", 2024-09-13T11:22:23.633 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_client_directed_command_retry": "5", 2024-09-13T11:22:23.634 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_client_hunt_interval": "3.000000", 2024-09-13T11:22:23.634 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_client_hunt_interval_backoff": "1.500000", 2024-09-13T11:22:23.634 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_client_hunt_interval_max_multiple": "10.000000", 2024-09-13T11:22:23.634 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_client_hunt_interval_min_multiple": "1.000000", 2024-09-13T11:22:23.634 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_client_hunt_parallel": "3", 2024-09-13T11:22:23.634 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_client_log_interval": "1.000000", 2024-09-13T11:22:23.634 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_client_max_log_entries_per_message": "1000", 2024-09-13T11:22:23.634 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_client_ping_interval": "10.000000", 2024-09-13T11:22:23.634 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_client_ping_timeout": "30.000000", 2024-09-13T11:22:23.634 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_client_target_rank": "-1", 2024-09-13T11:22:23.634 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_clock_drift_allowed": "1.000000", 2024-09-13T11:22:23.634 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_clock_drift_warn_backoff": "5.000000", 2024-09-13T11:22:23.634 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_cluster_log_file": "default=/var/log/ceph/ceph.$channel.log cluster=/var/log/ceph/ceph.log", 2024-09-13T11:22:23.635 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_cluster_log_level": "debug", 2024-09-13T11:22:23.635 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_cluster_log_to_file": "true", 2024-09-13T11:22:23.635 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_cluster_log_to_graylog": "false", 2024-09-13T11:22:23.635 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_cluster_log_to_graylog_host": "127.0.0.1", 2024-09-13T11:22:23.635 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_cluster_log_to_graylog_port": "12201", 2024-09-13T11:22:23.635 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_cluster_log_to_journald": "false", 2024-09-13T11:22:23.635 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_cluster_log_to_stderr": "false", 2024-09-13T11:22:23.635 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_cluster_log_to_syslog": "default=false", 2024-09-13T11:22:23.635 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_cluster_log_to_syslog_facility": "daemon", 2024-09-13T11:22:23.635 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_compact_on_bootstrap": "false", 2024-09-13T11:22:23.635 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_compact_on_start": "false", 2024-09-13T11:22:23.635 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_compact_on_trim": "true", 2024-09-13T11:22:23.635 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_con_tracker_persist_interval": "10", 2024-09-13T11:22:23.635 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_con_tracker_score_halflife": "43200", 2024-09-13T11:22:23.636 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_config_key_max_entry_size": "65536", 2024-09-13T11:22:23.636 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_cpu_threads": "4", 2024-09-13T11:22:23.636 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_crush_min_required_version": "hammer", 2024-09-13T11:22:23.636 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_daemon_bytes": "419430400", 2024-09-13T11:22:23.636 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_data": "/var/lib/ceph/mon/ceph-a", 2024-09-13T11:22:23.636 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_data_avail_crit": "5", 2024-09-13T11:22:23.636 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_data_avail_warn": "5", 2024-09-13T11:22:23.636 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_data_size_warn": "16106127360", 2024-09-13T11:22:23.636 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_debug_block_osdmap_trim": "false", 2024-09-13T11:22:23.636 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_debug_deprecated_as_obsolete": "false", 2024-09-13T11:22:23.636 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_debug_dump_json": "false", 2024-09-13T11:22:23.636 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_debug_dump_location": "/var/log/ceph/ceph-mon.a.tdump", 2024-09-13T11:22:23.636 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_debug_dump_transactions": "false", 2024-09-13T11:22:23.637 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_debug_extra_checks": "false", 2024-09-13T11:22:23.637 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_debug_no_initial_persistent_features": "false", 2024-09-13T11:22:23.637 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_debug_no_require_bluestore_for_ec_overwrites": "false", 2024-09-13T11:22:23.637 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_debug_no_require_reef": "false", 2024-09-13T11:22:23.637 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_debug_no_require_squid": "false", 2024-09-13T11:22:23.637 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_debug_unsafe_allow_tier_with_nonempty_snaps": "false", 2024-09-13T11:22:23.637 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_delta_reset_interval": "10.000000", 2024-09-13T11:22:23.637 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_dns_srv_name": "ceph-mon", 2024-09-13T11:22:23.637 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_down_mkfs_grace": "120", 2024-09-13T11:22:23.637 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_down_uptime_grace": "60", 2024-09-13T11:22:23.637 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_election_default_strategy": "1", 2024-09-13T11:22:23.637 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_election_timeout": "5.000000", 2024-09-13T11:22:23.637 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_elector_ignore_propose_margin": "0.000500", 2024-09-13T11:22:23.637 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_elector_ping_divisor": "2", 2024-09-13T11:22:23.638 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_elector_ping_timeout": "2.000000", 2024-09-13T11:22:23.638 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_enable_op_tracker": "true", 2024-09-13T11:22:23.638 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_fake_pool_delete": "false", 2024-09-13T11:22:23.638 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_force_quorum_join": "false", 2024-09-13T11:22:23.638 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_fsmap_prune_threshold": "300", 2024-09-13T11:22:23.638 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_globalid_prealloc": "10000", 2024-09-13T11:22:23.638 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_health_detail_to_clog": "false", 2024-09-13T11:22:23.638 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_health_log_update_period": "5", 2024-09-13T11:22:23.638 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_health_max_detail": "50", 2024-09-13T11:22:23.638 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_health_to_clog": "true", 2024-09-13T11:22:23.638 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_health_to_clog_interval": "600", 2024-09-13T11:22:23.638 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_health_to_clog_tick_interval": "60.000000", 2024-09-13T11:22:23.639 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_host": "[v1:172.21.15.1:6789],[v1:172.21.15.92:6789],[v1:172.21.15.162:6789]", 2024-09-13T11:22:23.639 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_host_override": "", 2024-09-13T11:22:23.639 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_initial_members": "", 2024-09-13T11:22:23.639 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_inject_pg_merge_bounce_probability": "0.000000", 2024-09-13T11:22:23.639 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_inject_sync_get_chunk_delay": "0.000000", 2024-09-13T11:22:23.639 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_inject_transaction_delay_max": "10.000000", 2024-09-13T11:22:23.639 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_inject_transaction_delay_probability": "0.000000", 2024-09-13T11:22:23.639 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_keyvaluedb": "rocksdb", 2024-09-13T11:22:23.639 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_lease": "5.000000", 2024-09-13T11:22:23.639 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_lease_ack_timeout_factor": "2.000000", 2024-09-13T11:22:23.639 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_lease_renew_interval_factor": "0.600000", 2024-09-13T11:22:23.639 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_log_full_interval": "50", 2024-09-13T11:22:23.639 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_log_max": "10000", 2024-09-13T11:22:23.640 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_log_max_summary": "50", 2024-09-13T11:22:23.640 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_max_log_entries_per_event": "4096", 2024-09-13T11:22:23.640 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_max_log_epochs": "500", 2024-09-13T11:22:23.640 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_max_mdsmap_epochs": "500", 2024-09-13T11:22:23.640 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_max_mgrmap_epochs": "500", 2024-09-13T11:22:23.640 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_max_nvmeof_epochs": "500", 2024-09-13T11:22:23.640 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_max_osd": "10000", 2024-09-13T11:22:23.640 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_max_pg_per_osd": "10000", 2024-09-13T11:22:23.640 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_max_pool_pg_num": "65536", 2024-09-13T11:22:23.640 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_max_snap_prune_per_epoch": "100", 2024-09-13T11:22:23.640 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_mds_blocklist_interval": "86400.000000", 2024-09-13T11:22:23.640 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_mds_force_trim_to": "0", 2024-09-13T11:22:23.640 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_mds_skip_sanity": "false", 2024-09-13T11:22:23.641 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_memory_autotune": "true", 2024-09-13T11:22:23.641 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_memory_target": "2147483648", 2024-09-13T11:22:23.641 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_mgr_beacon_grace": "30", 2024-09-13T11:22:23.641 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_mgr_blocklist_interval": "86400.000000", 2024-09-13T11:22:23.641 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_mgr_digest_period": "5", 2024-09-13T11:22:23.641 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_mgr_inactive_grace": "60", 2024-09-13T11:22:23.641 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_mgr_mkfs_grace": "240", 2024-09-13T11:22:23.641 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_mgr_proxy_client_bytes_ratio": "0.300000", 2024-09-13T11:22:23.641 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_min_osdmap_epochs": "50", 2024-09-13T11:22:23.641 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_nvmeofgw_beacon_grace": "10", 2024-09-13T11:22:23.641 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_nvmeofgw_set_group_id_retry": "1000", 2024-09-13T11:22:23.641 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_op_complaint_time": "30", 2024-09-13T11:22:23.641 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_op_history_duration": "600", 2024-09-13T11:22:23.642 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_op_history_size": "20", 2024-09-13T11:22:23.642 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_op_history_slow_op_size": "20", 2024-09-13T11:22:23.642 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_op_history_slow_op_threshold": "10", 2024-09-13T11:22:23.642 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_op_log_threshold": "5", 2024-09-13T11:22:23.642 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_osd_adjust_down_out_interval": "true", 2024-09-13T11:22:23.642 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_osd_adjust_heartbeat_grace": "true", 2024-09-13T11:22:23.642 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_osd_auto_mark_auto_out_in": "true", 2024-09-13T11:22:23.642 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_osd_auto_mark_in": "false", 2024-09-13T11:22:23.642 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_osd_auto_mark_new_in": "true", 2024-09-13T11:22:23.642 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_osd_backfillfull_ratio": "0.900000", 2024-09-13T11:22:23.642 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_osd_blocklist_default_expire": "3600.000000", 2024-09-13T11:22:23.642 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_osd_cache_size": "500", 2024-09-13T11:22:23.642 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_osd_cache_size_min": "134217728", 2024-09-13T11:22:23.642 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_osd_crush_smoke_test": "true", 2024-09-13T11:22:23.643 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_osd_destroyed_out_interval": "600", 2024-09-13T11:22:23.643 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_osd_down_out_interval": "600", 2024-09-13T11:22:23.643 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_osd_down_out_subtree_limit": "rack", 2024-09-13T11:22:23.643 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_osd_err_op_age_ratio": "128.000000", 2024-09-13T11:22:23.643 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_osd_force_trim_to": "0", 2024-09-13T11:22:23.643 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_osd_full_ratio": "0.950000", 2024-09-13T11:22:23.643 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_osd_initial_require_min_compat_client": "luminous", 2024-09-13T11:22:23.643 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_osd_laggy_halflife": "3600", 2024-09-13T11:22:23.643 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_osd_laggy_max_interval": "300", 2024-09-13T11:22:23.643 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_osd_laggy_weight": "0.300000", 2024-09-13T11:22:23.643 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_osd_mapping_pgs_per_chunk": "4096", 2024-09-13T11:22:23.643 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_osd_max_initial_pgs": "1024", 2024-09-13T11:22:23.643 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_osd_min_down_reporters": "2", 2024-09-13T11:22:23.643 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_osd_min_in_ratio": "0.750000", 2024-09-13T11:22:23.644 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_osd_min_up_ratio": "0.300000", 2024-09-13T11:22:23.644 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_osd_nearfull_ratio": "0.850000", 2024-09-13T11:22:23.644 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_osd_prime_pg_temp": "true", 2024-09-13T11:22:23.644 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_osd_prime_pg_temp_max_estimate": "0.250000", 2024-09-13T11:22:23.644 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_osd_prime_pg_temp_max_time": "0.500000", 2024-09-13T11:22:23.644 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_osd_report_timeout": "900", 2024-09-13T11:22:23.644 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_osd_reporter_subtree_level": "osd", 2024-09-13T11:22:23.644 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_osd_snap_trim_queue_warn_on": "32768", 2024-09-13T11:22:23.644 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_osd_warn_num_repaired": "10", 2024-09-13T11:22:23.644 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_osd_warn_op_age": "32.000000", 2024-09-13T11:22:23.644 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_osdmap_full_prune_enabled": "true", 2024-09-13T11:22:23.644 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_osdmap_full_prune_interval": "2", 2024-09-13T11:22:23.644 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_osdmap_full_prune_min": "15", 2024-09-13T11:22:23.645 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_osdmap_full_prune_txsize": "2", 2024-09-13T11:22:23.645 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_pg_check_down_all_threshold": "0.500000", 2024-09-13T11:22:23.645 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_pg_stuck_threshold": "60", 2024-09-13T11:22:23.645 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_pg_warn_max_object_skew": "0.000000", 2024-09-13T11:22:23.645 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_pg_warn_min_objects": "10000", 2024-09-13T11:22:23.645 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_pg_warn_min_per_osd": "0", 2024-09-13T11:22:23.645 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_pg_warn_min_pool_objects": "1000", 2024-09-13T11:22:23.645 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_pool_quota_crit_threshold": "0", 2024-09-13T11:22:23.645 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_pool_quota_warn_threshold": "0", 2024-09-13T11:22:23.645 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_probe_timeout": "2.000000", 2024-09-13T11:22:23.645 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_reweight_max_change": "0.050000", 2024-09-13T11:22:23.645 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_reweight_max_osds": "4", 2024-09-13T11:22:23.645 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_reweight_min_bytes_per_osd": "10", 2024-09-13T11:22:23.646 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_reweight_min_pgs_per_osd": "4", 2024-09-13T11:22:23.646 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_rocksdb_options": "write_buffer_size=33554432,compression=kNoCompression,level_compaction_dynamic_level_bytes=true", 2024-09-13T11:22:23.646 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_scrub_inject_crc_mismatch": "0.000000", 2024-09-13T11:22:23.646 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_scrub_inject_missing_keys": "0.000000", 2024-09-13T11:22:23.646 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_scrub_interval": "300", 2024-09-13T11:22:23.646 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_scrub_max_keys": "100", 2024-09-13T11:22:23.646 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_scrub_timeout": "300", 2024-09-13T11:22:23.646 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_session_timeout": "300", 2024-09-13T11:22:23.646 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_smart_report_timeout": "5", 2024-09-13T11:22:23.646 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_stat_smooth_intervals": "6", 2024-09-13T11:22:23.646 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_stretch_cluster_recovery_ratio": "0.600000", 2024-09-13T11:22:23.646 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_stretch_pool_min_size": "2", 2024-09-13T11:22:23.646 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_stretch_pool_size": "4", 2024-09-13T11:22:23.647 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_stretch_recovery_min_wait": "15.000000", 2024-09-13T11:22:23.647 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_subscribe_interval": "86400.000000", 2024-09-13T11:22:23.647 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_sync_debug": "false", 2024-09-13T11:22:23.647 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_sync_max_payload_keys": "2000", 2024-09-13T11:22:23.647 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_sync_max_payload_size": "1048576", 2024-09-13T11:22:23.647 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_sync_provider_kill_at": "0", 2024-09-13T11:22:23.647 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_sync_requester_kill_at": "0", 2024-09-13T11:22:23.647 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_sync_timeout": "60.000000", 2024-09-13T11:22:23.647 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_target_pg_per_osd": "100", 2024-09-13T11:22:23.647 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_tick_interval": "5", 2024-09-13T11:22:23.647 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_timecheck_interval": "300.000000", 2024-09-13T11:22:23.647 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_timecheck_skew_interval": "30.000000", 2024-09-13T11:22:23.647 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_use_min_delay_socket": "false", 2024-09-13T11:22:23.647 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_warn_older_version_delay": "604800", 2024-09-13T11:22:23.648 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_warn_on_cache_pools_without_hit_sets": "true", 2024-09-13T11:22:23.648 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_warn_on_crush_straw_calc_version_zero": "false", 2024-09-13T11:22:23.648 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_warn_on_degraded_stretch_mode": "true", 2024-09-13T11:22:23.648 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_warn_on_filestore_osds": "false", 2024-09-13T11:22:23.648 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_warn_on_insecure_global_id_reclaim": "false", 2024-09-13T11:22:23.648 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_warn_on_insecure_global_id_reclaim_allowed": "false", 2024-09-13T11:22:23.648 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_warn_on_legacy_crush_tunables": "false", 2024-09-13T11:22:23.648 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_warn_on_misplaced": "false", 2024-09-13T11:22:23.648 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_warn_on_msgr2_not_enabled": "true", 2024-09-13T11:22:23.648 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_warn_on_older_version": "true", 2024-09-13T11:22:23.648 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_warn_on_osd_down_out_interval_zero": "false", 2024-09-13T11:22:23.648 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_warn_on_pool_no_app": "true", 2024-09-13T11:22:23.648 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_warn_on_pool_no_redundancy": "false", 2024-09-13T11:22:23.649 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_warn_on_pool_pg_num_not_power_of_two": "false", 2024-09-13T11:22:23.649 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_warn_on_slow_ping_ratio": "0.050000", 2024-09-13T11:22:23.649 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_warn_on_slow_ping_time": "0.000000", 2024-09-13T11:22:23.649 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_warn_on_too_few_osds": "false", 2024-09-13T11:22:23.649 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_warn_pg_not_deep_scrubbed_ratio": "0.750000", 2024-09-13T11:22:23.649 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_warn_pg_not_scrubbed_ratio": "0.500000", 2024-09-13T11:22:23.649 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": "", 2024-09-13T11:22:23.649 INFO:teuthology.orchestra.run.smithi001.stdout: "motr_admin_endpoint": "192.168.180.182@tcp:12345:4:1", 2024-09-13T11:22:23.649 INFO:teuthology.orchestra.run.smithi001.stdout: "motr_admin_fid": "0x7200000000000001:0x0", 2024-09-13T11:22:23.649 INFO:teuthology.orchestra.run.smithi001.stdout: "motr_ha_endpoint": "192.168.180.182@tcp:12345:1:1", 2024-09-13T11:22:23.649 INFO:teuthology.orchestra.run.smithi001.stdout: "motr_my_endpoint": "192.168.180.182@tcp:12345:4:1", 2024-09-13T11:22:23.649 INFO:teuthology.orchestra.run.smithi001.stdout: "motr_my_fid": "0x7200000000000001:0x0", 2024-09-13T11:22:23.649 INFO:teuthology.orchestra.run.smithi001.stdout: "motr_profile_fid": "0x7000000000000001:0x0", 2024-09-13T11:22:23.649 INFO:teuthology.orchestra.run.smithi001.stdout: "motr_tracing_enabled": "false", 2024-09-13T11:22:23.650 INFO:teuthology.orchestra.run.smithi001.stdout: "ms_async_op_threads": "3", 2024-09-13T11:22:23.650 INFO:teuthology.orchestra.run.smithi001.stdout: "ms_async_rdma_buffer_size": "131072", 2024-09-13T11:22:23.650 INFO:teuthology.orchestra.run.smithi001.stdout: "ms_async_rdma_cm": "false", 2024-09-13T11:22:23.650 INFO:teuthology.orchestra.run.smithi001.stdout: "ms_async_rdma_device_name": "", 2024-09-13T11:22:23.650 INFO:teuthology.orchestra.run.smithi001.stdout: "ms_async_rdma_dscp": "96", 2024-09-13T11:22:23.650 INFO:teuthology.orchestra.run.smithi001.stdout: "ms_async_rdma_enable_hugepage": "false", 2024-09-13T11:22:23.650 INFO:teuthology.orchestra.run.smithi001.stdout: "ms_async_rdma_gid_idx": "0", 2024-09-13T11:22:23.650 INFO:teuthology.orchestra.run.smithi001.stdout: "ms_async_rdma_local_gid": "", 2024-09-13T11:22:23.650 INFO:teuthology.orchestra.run.smithi001.stdout: "ms_async_rdma_polling_us": "1000", 2024-09-13T11:22:23.650 INFO:teuthology.orchestra.run.smithi001.stdout: "ms_async_rdma_port_num": "1", 2024-09-13T11:22:23.650 INFO:teuthology.orchestra.run.smithi001.stdout: "ms_async_rdma_receive_buffers": "32768", 2024-09-13T11:22:23.650 INFO:teuthology.orchestra.run.smithi001.stdout: "ms_async_rdma_receive_queue_len": "4096", 2024-09-13T11:22:23.650 INFO:teuthology.orchestra.run.smithi001.stdout: "ms_async_rdma_roce_ver": "1", 2024-09-13T11:22:23.650 INFO:teuthology.orchestra.run.smithi001.stdout: "ms_async_rdma_send_buffers": "1024", 2024-09-13T11:22:23.651 INFO:teuthology.orchestra.run.smithi001.stdout: "ms_async_rdma_sl": "3", 2024-09-13T11:22:23.651 INFO:teuthology.orchestra.run.smithi001.stdout: "ms_async_rdma_support_srq": "true", 2024-09-13T11:22:23.651 INFO:teuthology.orchestra.run.smithi001.stdout: "ms_async_rdma_type": "ib", 2024-09-13T11:22:23.651 INFO:teuthology.orchestra.run.smithi001.stdout: "ms_async_reap_threshold": "5", 2024-09-13T11:22:23.651 INFO:teuthology.orchestra.run.smithi001.stdout: "ms_bind_before_connect": "false", 2024-09-13T11:22:23.651 INFO:teuthology.orchestra.run.smithi001.stdout: "ms_bind_ipv4": "true", 2024-09-13T11:22:23.651 INFO:teuthology.orchestra.run.smithi001.stdout: "ms_bind_ipv6": "false", 2024-09-13T11:22:23.651 INFO:teuthology.orchestra.run.smithi001.stdout: "ms_bind_msgr1": "true", 2024-09-13T11:22:23.651 INFO:teuthology.orchestra.run.smithi001.stdout: "ms_bind_msgr2": "false", 2024-09-13T11:22:23.651 INFO:teuthology.orchestra.run.smithi001.stdout: "ms_bind_port_max": "7568", 2024-09-13T11:22:23.651 INFO:teuthology.orchestra.run.smithi001.stdout: "ms_bind_port_min": "6800", 2024-09-13T11:22:23.651 INFO:teuthology.orchestra.run.smithi001.stdout: "ms_bind_prefer_ipv4": "false", 2024-09-13T11:22:23.651 INFO:teuthology.orchestra.run.smithi001.stdout: "ms_bind_retry_count": "3", 2024-09-13T11:22:23.652 INFO:teuthology.orchestra.run.smithi001.stdout: "ms_bind_retry_delay": "5", 2024-09-13T11:22:23.652 INFO:teuthology.orchestra.run.smithi001.stdout: "ms_blackhole_client": "false", 2024-09-13T11:22:23.652 INFO:teuthology.orchestra.run.smithi001.stdout: "ms_blackhole_mds": "false", 2024-09-13T11:22:23.652 INFO:teuthology.orchestra.run.smithi001.stdout: "ms_blackhole_mgr": "false", 2024-09-13T11:22:23.652 INFO:teuthology.orchestra.run.smithi001.stdout: "ms_blackhole_mon": "false", 2024-09-13T11:22:23.652 INFO:teuthology.orchestra.run.smithi001.stdout: "ms_blackhole_osd": "false", 2024-09-13T11:22:23.652 INFO:teuthology.orchestra.run.smithi001.stdout: "ms_client_mode": "crc secure", 2024-09-13T11:22:23.652 INFO:teuthology.orchestra.run.smithi001.stdout: "ms_client_throttle_retry_time_interval": "5000", 2024-09-13T11:22:23.652 INFO:teuthology.orchestra.run.smithi001.stdout: "ms_cluster_mode": "crc secure", 2024-09-13T11:22:23.652 INFO:teuthology.orchestra.run.smithi001.stdout: "ms_cluster_type": "", 2024-09-13T11:22:23.652 INFO:teuthology.orchestra.run.smithi001.stdout: "ms_compress_secure": "false", 2024-09-13T11:22:23.652 INFO:teuthology.orchestra.run.smithi001.stdout: "ms_connection_idle_timeout": "900", 2024-09-13T11:22:23.653 INFO:teuthology.orchestra.run.smithi001.stdout: "ms_connection_ready_timeout": "10", 2024-09-13T11:22:23.653 INFO:teuthology.orchestra.run.smithi001.stdout: "ms_crc_data": "true", 2024-09-13T11:22:23.653 INFO:teuthology.orchestra.run.smithi001.stdout: "ms_crc_header": "true", 2024-09-13T11:22:23.653 INFO:teuthology.orchestra.run.smithi001.stdout: "ms_die_on_bad_msg": "false", 2024-09-13T11:22:23.653 INFO:teuthology.orchestra.run.smithi001.stdout: "ms_die_on_bug": "true", 2024-09-13T11:22:23.653 INFO:teuthology.orchestra.run.smithi001.stdout: "ms_die_on_old_message": "true", 2024-09-13T11:22:23.653 INFO:teuthology.orchestra.run.smithi001.stdout: "ms_die_on_skipped_message": "false", 2024-09-13T11:22:23.653 INFO:teuthology.orchestra.run.smithi001.stdout: "ms_die_on_unhandled_msg": "false", 2024-09-13T11:22:23.653 INFO:teuthology.orchestra.run.smithi001.stdout: "ms_dispatch_throttle_bytes": "104857600", 2024-09-13T11:22:23.653 INFO:teuthology.orchestra.run.smithi001.stdout: "ms_dpdk_coremask": "0xF", 2024-09-13T11:22:23.654 INFO:teuthology.orchestra.run.smithi001.stdout: "ms_dpdk_debug_allow_loopback": "false", 2024-09-13T11:22:23.654 INFO:teuthology.orchestra.run.smithi001.stdout: "ms_dpdk_devs_allowlist": "", 2024-09-13T11:22:23.654 INFO:teuthology.orchestra.run.smithi001.stdout: "ms_dpdk_enable_tso": "true", 2024-09-13T11:22:23.654 INFO:teuthology.orchestra.run.smithi001.stdout: "ms_dpdk_gateway_ipv4_addr": "", 2024-09-13T11:22:23.654 INFO:teuthology.orchestra.run.smithi001.stdout: "ms_dpdk_host_ipv4_addr": "", 2024-09-13T11:22:23.654 INFO:teuthology.orchestra.run.smithi001.stdout: "ms_dpdk_hugepages": "", 2024-09-13T11:22:23.654 INFO:teuthology.orchestra.run.smithi001.stdout: "ms_dpdk_hw_flow_control": "true", 2024-09-13T11:22:23.654 INFO:teuthology.orchestra.run.smithi001.stdout: "ms_dpdk_hw_queue_weight": "1.000000", 2024-09-13T11:22:23.654 INFO:teuthology.orchestra.run.smithi001.stdout: "ms_dpdk_lro": "true", 2024-09-13T11:22:23.654 INFO:teuthology.orchestra.run.smithi001.stdout: "ms_dpdk_memory_channel": "4", 2024-09-13T11:22:23.654 INFO:teuthology.orchestra.run.smithi001.stdout: "ms_dpdk_netmask_ipv4_addr": "", 2024-09-13T11:22:23.654 INFO:teuthology.orchestra.run.smithi001.stdout: "ms_dpdk_pmd": "", 2024-09-13T11:22:23.654 INFO:teuthology.orchestra.run.smithi001.stdout: "ms_dpdk_port_id": "0", 2024-09-13T11:22:23.655 INFO:teuthology.orchestra.run.smithi001.stdout: "ms_dpdk_rx_buffer_count_per_core": "8192", 2024-09-13T11:22:23.655 INFO:teuthology.orchestra.run.smithi001.stdout: "ms_dump_corrupt_message_level": "1", 2024-09-13T11:22:23.655 INFO:teuthology.orchestra.run.smithi001.stdout: "ms_dump_on_send": "false", 2024-09-13T11:22:23.655 INFO:teuthology.orchestra.run.smithi001.stdout: "ms_initial_backoff": "0.200000", 2024-09-13T11:22:23.655 INFO:teuthology.orchestra.run.smithi001.stdout: "ms_inject_delay_max": "1.000000", 2024-09-13T11:22:23.655 INFO:teuthology.orchestra.run.smithi001.stdout: "ms_inject_delay_probability": "0.005000", 2024-09-13T11:22:23.655 INFO:teuthology.orchestra.run.smithi001.stdout: "ms_inject_delay_type": "osd", 2024-09-13T11:22:23.655 INFO:teuthology.orchestra.run.smithi001.stdout: "ms_inject_internal_delays": "0.002000", 2024-09-13T11:22:23.655 INFO:teuthology.orchestra.run.smithi001.stdout: "ms_inject_network_congestion": "0", 2024-09-13T11:22:23.655 INFO:teuthology.orchestra.run.smithi001.stdout: "ms_inject_socket_failures": "2500", 2024-09-13T11:22:23.655 INFO:teuthology.orchestra.run.smithi001.stdout: "ms_learn_addr_from_peer": "true", 2024-09-13T11:22:23.655 INFO:teuthology.orchestra.run.smithi001.stdout: "ms_max_accept_failures": "4", 2024-09-13T11:22:23.655 INFO:teuthology.orchestra.run.smithi001.stdout: "ms_max_backoff": "15.000000", 2024-09-13T11:22:23.655 INFO:teuthology.orchestra.run.smithi001.stdout: "ms_mon_client_mode": "secure crc", 2024-09-13T11:22:23.656 INFO:teuthology.orchestra.run.smithi001.stdout: "ms_mon_cluster_mode": "secure crc", 2024-09-13T11:22:23.656 INFO:teuthology.orchestra.run.smithi001.stdout: "ms_mon_service_mode": "secure crc", 2024-09-13T11:22:23.656 INFO:teuthology.orchestra.run.smithi001.stdout: "ms_osd_compress_min_size": "1024", 2024-09-13T11:22:23.656 INFO:teuthology.orchestra.run.smithi001.stdout: "ms_osd_compress_mode": "none", 2024-09-13T11:22:23.656 INFO:teuthology.orchestra.run.smithi001.stdout: "ms_osd_compression_algorithm": "snappy", 2024-09-13T11:22:23.656 INFO:teuthology.orchestra.run.smithi001.stdout: "ms_pq_max_tokens_per_priority": "16777216", 2024-09-13T11:22:23.656 INFO:teuthology.orchestra.run.smithi001.stdout: "ms_pq_min_cost": "65536", 2024-09-13T11:22:23.656 INFO:teuthology.orchestra.run.smithi001.stdout: "ms_public_type": "", 2024-09-13T11:22:23.656 INFO:teuthology.orchestra.run.smithi001.stdout: "ms_service_mode": "crc secure", 2024-09-13T11:22:23.656 INFO:teuthology.orchestra.run.smithi001.stdout: "ms_tcp_listen_backlog": "512", 2024-09-13T11:22:23.656 INFO:teuthology.orchestra.run.smithi001.stdout: "ms_tcp_nodelay": "true", 2024-09-13T11:22:23.656 INFO:teuthology.orchestra.run.smithi001.stdout: "ms_tcp_prefetch_max_size": "4096", 2024-09-13T11:22:23.656 INFO:teuthology.orchestra.run.smithi001.stdout: "ms_tcp_rcvbuf": "0", 2024-09-13T11:22:23.656 INFO:teuthology.orchestra.run.smithi001.stdout: "ms_time_events_min_wait_interval": "1000", 2024-09-13T11:22:23.657 INFO:teuthology.orchestra.run.smithi001.stdout: "ms_type": "async", 2024-09-13T11:22:23.657 INFO:teuthology.orchestra.run.smithi001.stdout: "no_config_file": "false", 2024-09-13T11:22:23.657 INFO:teuthology.orchestra.run.smithi001.stdout: "nvmeof_mon_client_disconnect_panic": "100", 2024-09-13T11:22:23.657 INFO:teuthology.orchestra.run.smithi001.stdout: "nvmeof_mon_client_tick_period": "2", 2024-09-13T11:22:23.657 INFO:teuthology.orchestra.run.smithi001.stdout: "objecter_completion_locks_per_session": "32", 2024-09-13T11:22:23.657 INFO:teuthology.orchestra.run.smithi001.stdout: "objecter_debug_inject_relock_delay": "false", 2024-09-13T11:22:23.657 INFO:teuthology.orchestra.run.smithi001.stdout: "objecter_inflight_op_bytes": "104857600", 2024-09-13T11:22:23.657 INFO:teuthology.orchestra.run.smithi001.stdout: "objecter_inflight_ops": "1024", 2024-09-13T11:22:23.657 INFO:teuthology.orchestra.run.smithi001.stdout: "objecter_inject_no_watch_ping": "false", 2024-09-13T11:22:23.657 INFO:teuthology.orchestra.run.smithi001.stdout: "objecter_retry_writes_after_first_reply": "false", 2024-09-13T11:22:23.657 INFO:teuthology.orchestra.run.smithi001.stdout: "objecter_tick_interval": "5.000000", 2024-09-13T11:22:23.657 INFO:teuthology.orchestra.run.smithi001.stdout: "objecter_timeout": "10.000000", 2024-09-13T11:22:23.657 INFO:teuthology.orchestra.run.smithi001.stdout: "objectstore_blackhole": "false", 2024-09-13T11:22:23.657 INFO:teuthology.orchestra.run.smithi001.stdout: "objectstore_debug_throw_on_failed_txc": "false", 2024-09-13T11:22:23.658 INFO:teuthology.orchestra.run.smithi001.stdout: "openssl_engine_opts": "", 2024-09-13T11:22:23.658 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_agent_delay_time": "5.000000", 2024-09-13T11:22:23.658 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_agent_hist_halflife": "1000", 2024-09-13T11:22:23.658 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_agent_max_low_ops": "2", 2024-09-13T11:22:23.658 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_agent_max_ops": "4", 2024-09-13T11:22:23.658 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_agent_min_evict_effort": "0.100000", 2024-09-13T11:22:23.658 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_agent_quantize_effort": "0.100000", 2024-09-13T11:22:23.658 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_agent_slop": "0.020000", 2024-09-13T11:22:23.658 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_aggregated_slow_ops_logging": "true", 2024-09-13T11:22:23.658 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_allow_recovery_below_min_size": "true", 2024-09-13T11:22:23.658 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_asio_thread_count": "2", 2024-09-13T11:22:23.658 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_async_recovery_min_cost": "100", 2024-09-13T11:22:23.658 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_auto_mark_unfound_lost": "false", 2024-09-13T11:22:23.658 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_backfill_retry_interval": "30.000000", 2024-09-13T11:22:23.659 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_backfill_scan_max": "512", 2024-09-13T11:22:23.659 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_backfill_scan_min": "64", 2024-09-13T11:22:23.659 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_backoff_on_degraded": "false", 2024-09-13T11:22:23.659 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_backoff_on_peering": "false", 2024-09-13T11:22:23.659 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_backoff_on_unfound": "true", 2024-09-13T11:22:23.659 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_beacon_report_interval": "300", 2024-09-13T11:22:23.659 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_bench_duration": "30", 2024-09-13T11:22:23.659 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_bench_large_size_max_throughput": "104857600", 2024-09-13T11:22:23.659 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_bench_max_block_size": "67108864", 2024-09-13T11:22:23.659 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_bench_small_size_max_iops": "100", 2024-09-13T11:22:23.659 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_blkin_trace_all": "false", 2024-09-13T11:22:23.659 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_blocked_scrub_grace_period": "120", 2024-09-13T11:22:23.659 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_calc_pg_upmaps_aggressively": "true", 2024-09-13T11:22:23.660 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_calc_pg_upmaps_aggressively_fast": "true", 2024-09-13T11:22:23.660 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_calc_pg_upmaps_local_fallback_retries": "100", 2024-09-13T11:22:23.660 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_check_for_log_corruption": "false", 2024-09-13T11:22:23.660 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_check_max_object_name_len_on_startup": "true", 2024-09-13T11:22:23.660 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_class_default_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2024-09-13T11:22:23.660 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_class_dir": "/usr/lib/rados-classes", 2024-09-13T11:22:23.660 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_class_load_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2024-09-13T11:22:23.660 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_class_update_on_start": "true", 2024-09-13T11:22:23.660 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_client_message_cap": "256", 2024-09-13T11:22:23.660 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_client_message_size_cap": "524288000", 2024-09-13T11:22:23.660 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_client_op_priority": "63", 2024-09-13T11:22:23.661 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_client_watch_timeout": "30", 2024-09-13T11:22:23.661 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_command_max_records": "256", 2024-09-13T11:22:23.661 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_compact_on_start": "false", 2024-09-13T11:22:23.661 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_copyfrom_max_chunk": "8388608", 2024-09-13T11:22:23.661 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_crush_chooseleaf_type": "0", 2024-09-13T11:22:23.661 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_crush_initial_weight": "-1.000000", 2024-09-13T11:22:23.661 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_crush_update_on_start": "true", 2024-09-13T11:22:23.661 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_crush_update_weight_set": "true", 2024-09-13T11:22:23.661 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_data": "/var/lib/ceph/osd/ceph-a", 2024-09-13T11:22:23.661 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_debug_crash_on_ignored_backoff": "false", 2024-09-13T11:22:23.662 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_debug_deep_scrub_sleep": "0.000000", 2024-09-13T11:22:23.662 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_debug_drop_ping_duration": "0", 2024-09-13T11:22:23.662 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_debug_drop_ping_probability": "0.000000", 2024-09-13T11:22:23.662 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_debug_feed_pullee": "-1", 2024-09-13T11:22:23.662 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_debug_inject_copyfrom_error": "false", 2024-09-13T11:22:23.662 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_debug_inject_dispatch_delay_duration": "0.100000", 2024-09-13T11:22:23.662 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_debug_inject_dispatch_delay_probability": "0.000000", 2024-09-13T11:22:23.662 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_debug_misdirected_ops": "false", 2024-09-13T11:22:23.662 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_debug_no_acting_change": "false", 2024-09-13T11:22:23.662 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_debug_no_purge_strays": "false", 2024-09-13T11:22:23.662 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_debug_op_order": "false", 2024-09-13T11:22:23.662 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_debug_pg_log_writeout": "false", 2024-09-13T11:22:23.662 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_debug_pretend_recovery_active": "false", 2024-09-13T11:22:23.662 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_debug_random_push_read_error": "0.000000", 2024-09-13T11:22:23.663 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_debug_reject_backfill_probability": "0.000000", 2024-09-13T11:22:23.663 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_debug_shutdown": "false", 2024-09-13T11:22:23.663 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_debug_skip_full_check_in_backfill_reservation": "false", 2024-09-13T11:22:23.663 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_debug_skip_full_check_in_recovery": "false", 2024-09-13T11:22:23.663 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_debug_trim_objects": "false", 2024-09-13T11:22:23.663 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_debug_verify_cached_snaps": "false", 2024-09-13T11:22:23.663 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_debug_verify_missing_on_start": "false", 2024-09-13T11:22:23.663 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_debug_verify_snaps": "false", 2024-09-13T11:22:23.663 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_debug_verify_stray_on_activate": "false", 2024-09-13T11:22:23.663 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_deep_scrub_interval": "604800.000000", 2024-09-13T11:22:23.663 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_deep_scrub_keys": "1024", 2024-09-13T11:22:23.663 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_deep_scrub_large_omap_object_key_threshold": "200000", 2024-09-13T11:22:23.663 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_deep_scrub_large_omap_object_value_sum_threshold": "1073741824", 2024-09-13T11:22:23.663 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_deep_scrub_randomize_ratio": "0.150000", 2024-09-13T11:22:23.664 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_deep_scrub_stride": "524288", 2024-09-13T11:22:23.664 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_deep_scrub_update_digest_min_age": "7200", 2024-09-13T11:22:23.664 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_default_data_pool_replay_window": "5", 2024-09-13T11:22:23.664 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_default_notify_timeout": "30", 2024-09-13T11:22:23.664 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_delete_sleep": "0.000000", 2024-09-13T11:22:23.664 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_delete_sleep_hdd": "5.000000", 2024-09-13T11:22:23.664 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_delete_sleep_hybrid": "1.000000", 2024-09-13T11:22:23.664 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_delete_sleep_ssd": "1.000000", 2024-09-13T11:22:23.664 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_discard_disconnected_ops": "true", 2024-09-13T11:22:23.664 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_ec_partial_reads": "true", 2024-09-13T11:22:23.664 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_enable_op_tracker": "true", 2024-09-13T11:22:23.664 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_erasure_code_plugins": "jerasure lrc", 2024-09-13T11:22:23.664 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_extblkdev_plugins": "vdo", 2024-09-13T11:22:23.665 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_failsafe_full_ratio": "0.970000", 2024-09-13T11:22:23.665 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_fast_fail_on_connection_refused": "true", 2024-09-13T11:22:23.665 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_fast_info": "true", 2024-09-13T11:22:23.665 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_fast_shutdown": "true", 2024-09-13T11:22:23.665 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_fast_shutdown_notify_mon": "true", 2024-09-13T11:22:23.665 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_fast_shutdown_timeout": "15", 2024-09-13T11:22:23.665 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_find_best_info_ignore_history_les": "false", 2024-09-13T11:22:23.665 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_force_auth_primary_missing_objects": "100", 2024-09-13T11:22:23.665 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_force_recovery_pg_log_entries_factor": "1.300000", 2024-09-13T11:22:23.665 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_function_tracing": "false", 2024-09-13T11:22:23.665 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_heartbeat_grace": "20", 2024-09-13T11:22:23.665 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_heartbeat_interval": "6", 2024-09-13T11:22:23.665 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_heartbeat_min_healthy_ratio": "0.330000", 2024-09-13T11:22:23.665 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_heartbeat_min_peers": "10", 2024-09-13T11:22:23.666 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_heartbeat_min_size": "2000", 2024-09-13T11:22:23.666 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_heartbeat_stale": "600", 2024-09-13T11:22:23.666 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_heartbeat_use_min_delay_socket": "false", 2024-09-13T11:22:23.666 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_hit_set_max_size": "100000", 2024-09-13T11:22:23.666 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_hit_set_min_size": "1000", 2024-09-13T11:22:23.666 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_hit_set_namespace": ".ceph-internal", 2024-09-13T11:22:23.666 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_ignore_stale_divergent_priors": "false", 2024-09-13T11:22:23.666 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_inject_bad_map_crc_probability": "0.000000", 2024-09-13T11:22:23.666 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_inject_failure_on_pg_removal": "false", 2024-09-13T11:22:23.666 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_journal": "/var/lib/ceph/osd/ceph-a/journal", 2024-09-13T11:22:23.666 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_journal_flush_on_shutdown": "true", 2024-09-13T11:22:23.666 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_journal_size": "5120", 2024-09-13T11:22:23.666 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_kill_backfill_at": "0", 2024-09-13T11:22:23.666 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_loop_before_reset_tphandle": "64", 2024-09-13T11:22:23.667 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_map_cache_size": "50", 2024-09-13T11:22:23.667 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_map_dedup": "true", 2024-09-13T11:22:23.667 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_map_message_max": "40", 2024-09-13T11:22:23.667 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_map_message_max_bytes": "10485760", 2024-09-13T11:22:23.667 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_map_share_max_epochs": "40", 2024-09-13T11:22:23.667 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_max_attr_name_len": "100", 2024-09-13T11:22:23.667 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_max_attr_size": "0", 2024-09-13T11:22:23.667 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_max_backfills": "1", 2024-09-13T11:22:23.667 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_max_markdown_count": "5", 2024-09-13T11:22:23.667 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_max_markdown_period": "600", 2024-09-13T11:22:23.667 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_max_object_name_len": "2048", 2024-09-13T11:22:23.667 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_max_object_namespace_len": "256", 2024-09-13T11:22:23.667 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_max_object_size": "134217728", 2024-09-13T11:22:23.667 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_max_omap_bytes_per_request": "1073741824", 2024-09-13T11:22:23.668 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_max_omap_entries_per_request": "1024", 2024-09-13T11:22:23.668 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_max_pg_blocked_by": "16", 2024-09-13T11:22:23.668 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_max_pg_log_entries": "10000", 2024-09-13T11:22:23.668 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_max_pg_per_osd_hard_ratio": "3.000000", 2024-09-13T11:22:23.668 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_max_pgls": "1024", 2024-09-13T11:22:23.668 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_max_push_cost": "8388608", 2024-09-13T11:22:23.668 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_max_push_objects": "10", 2024-09-13T11:22:23.668 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_max_scrubs": "3", 2024-09-13T11:22:23.668 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_max_snap_prune_intervals_per_epoch": "512", 2024-09-13T11:22:23.668 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_max_trimming_pgs": "2", 2024-09-13T11:22:23.668 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_max_write_op_reply_len": "64", 2024-09-13T11:22:23.669 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_max_write_size": "90", 2024-09-13T11:22:23.669 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_mclock_force_run_benchmark_on_init": "false", 2024-09-13T11:22:23.669 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_mclock_iops_capacity_threshold_hdd": "500.000000", 2024-09-13T11:22:23.669 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_mclock_iops_capacity_threshold_ssd": "80000.000000", 2024-09-13T11:22:23.669 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_mclock_max_capacity_iops_hdd": "315.000000", 2024-09-13T11:22:23.669 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_mclock_max_capacity_iops_ssd": "21500.000000", 2024-09-13T11:22:23.669 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_mclock_max_sequential_bandwidth_hdd": "157286400", 2024-09-13T11:22:23.669 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_mclock_max_sequential_bandwidth_ssd": "1258291200", 2024-09-13T11:22:23.669 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_mclock_override_recovery_settings": "false", 2024-09-13T11:22:23.669 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_mclock_profile": "balanced", 2024-09-13T11:22:23.669 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_mclock_scheduler_anticipation_timeout": "0.000000", 2024-09-13T11:22:23.669 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_mclock_scheduler_background_best_effort_lim": "0.000000", 2024-09-13T11:22:23.669 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_mclock_scheduler_background_best_effort_res": "0.000000", 2024-09-13T11:22:23.669 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_mclock_scheduler_background_best_effort_wgt": "1", 2024-09-13T11:22:23.670 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_mclock_scheduler_background_recovery_lim": "0.000000", 2024-09-13T11:22:23.670 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_mclock_scheduler_background_recovery_res": "0.000000", 2024-09-13T11:22:23.670 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_mclock_scheduler_background_recovery_wgt": "1", 2024-09-13T11:22:23.670 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_mclock_scheduler_client_lim": "0.000000", 2024-09-13T11:22:23.670 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_mclock_scheduler_client_res": "0.000000", 2024-09-13T11:22:23.670 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_mclock_scheduler_client_wgt": "1", 2024-09-13T11:22:23.670 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_mclock_skip_benchmark": "false", 2024-09-13T11:22:23.670 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_memory_base": "805306368", 2024-09-13T11:22:23.670 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_memory_cache_min": "134217728", 2024-09-13T11:22:23.670 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_memory_cache_resize_interval": "1.000000", 2024-09-13T11:22:23.670 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_memory_expected_fragmentation": "0.150000", 2024-09-13T11:22:23.670 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_memory_target": "4294967296", 2024-09-13T11:22:23.670 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_memory_target_autotune": "false", 2024-09-13T11:22:23.670 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_memory_target_cgroup_limit_ratio": "0.800000", 2024-09-13T11:22:23.671 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_min_pg_log_entries": "250", 2024-09-13T11:22:23.671 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_min_recovery_priority": "0", 2024-09-13T11:22:23.671 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_mon_heartbeat_interval": "30", 2024-09-13T11:22:23.671 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_mon_heartbeat_stat_stale": "3600", 2024-09-13T11:22:23.671 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_mon_report_interval": "5", 2024-09-13T11:22:23.671 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_mon_report_max_in_flight": "2", 2024-09-13T11:22:23.671 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_mon_shutdown_timeout": "5.000000", 2024-09-13T11:22:23.671 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_num_cache_shards": "32", 2024-09-13T11:22:23.671 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_num_op_tracker_shard": "32", 2024-09-13T11:22:23.671 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_numa_auto_affinity": "true", 2024-09-13T11:22:23.671 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_numa_node": "-1", 2024-09-13T11:22:23.671 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_numa_prefer_iface": "true", 2024-09-13T11:22:23.671 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_object_clean_region_max_num_intervals": "10", 2024-09-13T11:22:23.671 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_objecter_finishers": "1", 2024-09-13T11:22:23.671 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_objectstore": "bluestore", 2024-09-13T11:22:23.672 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_objectstore_fuse": "false", 2024-09-13T11:22:23.672 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_objectstore_ideal_list_max": "64", 2024-09-13T11:22:23.672 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_objectstore_tracing": "false", 2024-09-13T11:22:23.672 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_op_complaint_time": "30.000000", 2024-09-13T11:22:23.672 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_op_history_duration": "600", 2024-09-13T11:22:23.672 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_op_history_size": "20", 2024-09-13T11:22:23.672 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_op_history_slow_op_size": "20", 2024-09-13T11:22:23.672 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_op_history_slow_op_threshold": "10.000000", 2024-09-13T11:22:23.672 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_op_log_threshold": "5", 2024-09-13T11:22:23.672 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_op_num_shards": "0", 2024-09-13T11:22:23.672 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_op_num_shards_hdd": "5", 2024-09-13T11:22:23.672 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_op_num_shards_ssd": "8", 2024-09-13T11:22:23.672 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_op_num_threads_per_shard": "0", 2024-09-13T11:22:23.672 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_op_num_threads_per_shard_hdd": "1", 2024-09-13T11:22:23.673 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_op_num_threads_per_shard_ssd": "2", 2024-09-13T11:22:23.673 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_op_pq_max_tokens_per_priority": "4194304", 2024-09-13T11:22:23.673 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_op_pq_min_cost": "65536", 2024-09-13T11:22:23.673 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_op_queue": "mclock_scheduler", 2024-09-13T11:22:23.673 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_op_queue_cut_off": "high", 2024-09-13T11:22:23.673 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_op_thread_suicide_timeout": "150", 2024-09-13T11:22:23.673 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_op_thread_timeout": "15", 2024-09-13T11:22:23.673 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_open_classes_on_start": "true", 2024-09-13T11:22:23.673 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_os_flags": "0", 2024-09-13T11:22:23.673 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_peering_op_priority": "255", 2024-09-13T11:22:23.673 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_pg_delete_cost": "1048576", 2024-09-13T11:22:23.673 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_pg_delete_priority": "5", 2024-09-13T11:22:23.673 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_pg_epoch_max_lag_factor": "2.000000", 2024-09-13T11:22:23.674 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_pg_epoch_persisted_max_stale": "40", 2024-09-13T11:22:23.674 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_pg_log_dups_tracked": "3000", 2024-09-13T11:22:23.674 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_pg_log_trim_max": "10000", 2024-09-13T11:22:23.674 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_pg_log_trim_min": "100", 2024-09-13T11:22:23.674 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_pg_max_concurrent_snap_trims": "2", 2024-09-13T11:22:23.674 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_pg_object_context_cache_count": "64", 2024-09-13T11:22:23.674 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_pg_stat_report_interval_max_epochs": "500", 2024-09-13T11:22:23.674 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_pg_stat_report_interval_max_seconds": "5", 2024-09-13T11:22:23.674 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_pool_default_cache_max_evict_check_size": "10", 2024-09-13T11:22:23.674 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_pool_default_cache_min_evict_age": "0", 2024-09-13T11:22:23.674 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_pool_default_cache_min_flush_age": "0", 2024-09-13T11:22:23.674 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_pool_default_cache_target_dirty_high_ratio": "0.600000", 2024-09-13T11:22:23.674 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_pool_default_cache_target_dirty_ratio": "0.400000", 2024-09-13T11:22:23.674 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_pool_default_cache_target_full_ratio": "0.800000", 2024-09-13T11:22:23.675 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_pool_default_crimson": "false", 2024-09-13T11:22:23.675 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_pool_default_crush_rule": "-1", 2024-09-13T11:22:23.675 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_pool_default_ec_fast_read": "false", 2024-09-13T11:22:23.675 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_pool_default_erasure_code_profile": "plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd", 2024-09-13T11:22:23.675 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_pool_default_flag_bulk": "false", 2024-09-13T11:22:23.675 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_pool_default_flag_hashpspool": "true", 2024-09-13T11:22:23.675 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_pool_default_flag_nodelete": "false", 2024-09-13T11:22:23.675 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_pool_default_flag_nopgchange": "false", 2024-09-13T11:22:23.675 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_pool_default_flag_nosizechange": "false", 2024-09-13T11:22:23.675 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_pool_default_flags": "0", 2024-09-13T11:22:23.675 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_pool_default_hit_set_bloom_fpp": "0.050000", 2024-09-13T11:22:23.675 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_pool_default_min_size": "2", 2024-09-13T11:22:23.675 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_pool_default_pg_autoscale_mode": "off", 2024-09-13T11:22:23.675 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_pool_default_pg_num": "32", 2024-09-13T11:22:23.676 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_pool_default_pgp_num": "0", 2024-09-13T11:22:23.676 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_pool_default_read_lease_ratio": "0.800000", 2024-09-13T11:22:23.676 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_pool_default_read_ratio": "70", 2024-09-13T11:22:23.676 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_pool_default_size": "2", 2024-09-13T11:22:23.676 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_pool_default_type": "replicated", 2024-09-13T11:22:23.676 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_pool_erasure_code_stripe_unit": "4096", 2024-09-13T11:22:23.676 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_pool_use_gmt_hitset": "true", 2024-09-13T11:22:23.676 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_push_per_object_cost": "1000", 2024-09-13T11:22:23.676 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_read_ec_check_for_errors": "false", 2024-09-13T11:22:23.676 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_recover_clone_overlap": "true", 2024-09-13T11:22:23.676 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_recover_clone_overlap_limit": "10", 2024-09-13T11:22:23.676 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_recovery_cost": "20971520", 2024-09-13T11:22:23.676 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_recovery_delay_start": "0.000000", 2024-09-13T11:22:23.677 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_recovery_max_active": "0", 2024-09-13T11:22:23.677 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_recovery_max_active_hdd": "3", 2024-09-13T11:22:23.677 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_recovery_max_active_ssd": "10", 2024-09-13T11:22:23.677 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_recovery_max_chunk": "8388608", 2024-09-13T11:22:23.677 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_recovery_max_omap_entries_per_chunk": "8096", 2024-09-13T11:22:23.677 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_recovery_max_single_start": "1", 2024-09-13T11:22:23.677 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_recovery_op_priority": "3", 2024-09-13T11:22:23.677 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_recovery_op_warn_multiple": "16", 2024-09-13T11:22:23.677 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_recovery_priority": "5", 2024-09-13T11:22:23.677 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_recovery_retry_interval": "30.000000", 2024-09-13T11:22:23.677 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_recovery_sleep": "0.000000", 2024-09-13T11:22:23.677 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_recovery_sleep_hdd": "0.100000", 2024-09-13T11:22:23.677 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_recovery_sleep_hybrid": "0.025000", 2024-09-13T11:22:23.677 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_recovery_sleep_ssd": "0.000000", 2024-09-13T11:22:23.678 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_repair_during_recovery": "false", 2024-09-13T11:22:23.678 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_requested_scrub_priority": "120", 2024-09-13T11:22:23.678 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_rocksdb_iterator_bounds_enabled": "true", 2024-09-13T11:22:23.678 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_rollback_to_cluster_snap": "", 2024-09-13T11:22:23.678 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_scrub_auto_repair": "false", 2024-09-13T11:22:23.678 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_scrub_auto_repair_num_errors": "5", 2024-09-13T11:22:23.678 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_scrub_backoff_ratio": "0.660000", 2024-09-13T11:22:23.678 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_scrub_begin_hour": "0", 2024-09-13T11:22:23.678 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_scrub_begin_week_day": "0", 2024-09-13T11:22:23.678 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_scrub_chunk_max": "25", 2024-09-13T11:22:23.678 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_scrub_chunk_min": "5", 2024-09-13T11:22:23.678 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_scrub_cost": "52428800", 2024-09-13T11:22:23.678 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_scrub_disable_reservation_queuing": "false", 2024-09-13T11:22:23.678 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_scrub_during_recovery": "false", 2024-09-13T11:22:23.679 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_scrub_end_hour": "0", 2024-09-13T11:22:23.679 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_scrub_end_week_day": "0", 2024-09-13T11:22:23.679 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_scrub_event_cost": "4096", 2024-09-13T11:22:23.679 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_scrub_extended_sleep": "0.000000", 2024-09-13T11:22:23.679 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_scrub_interval_randomize_ratio": "0.500000", 2024-09-13T11:22:23.679 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_scrub_invalid_stats": "true", 2024-09-13T11:22:23.679 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_scrub_load_threshold": "0.500000", 2024-09-13T11:22:23.679 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_scrub_max_interval": "604800.000000", 2024-09-13T11:22:23.679 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_scrub_max_preemptions": "5", 2024-09-13T11:22:23.679 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_scrub_min_interval": "86400.000000", 2024-09-13T11:22:23.679 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_scrub_priority": "5", 2024-09-13T11:22:23.679 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_scrub_sleep": "0.000000", 2024-09-13T11:22:23.679 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_shallow_scrub_chunk_max": "100", 2024-09-13T11:22:23.679 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_shallow_scrub_chunk_min": "50", 2024-09-13T11:22:23.680 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_shutdown_pgref_assert": "false", 2024-09-13T11:22:23.680 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_skip_check_past_interval_bounds": "false", 2024-09-13T11:22:23.680 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_skip_data_digest": "false", 2024-09-13T11:22:23.680 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_smart_report_timeout": "5", 2024-09-13T11:22:23.680 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_snap_trim_cost": "1048576", 2024-09-13T11:22:23.680 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_snap_trim_priority": "5", 2024-09-13T11:22:23.680 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_snap_trim_sleep": "0.000000", 2024-09-13T11:22:23.680 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_snap_trim_sleep_hdd": "5.000000", 2024-09-13T11:22:23.680 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_snap_trim_sleep_hybrid": "2.000000", 2024-09-13T11:22:23.680 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_snap_trim_sleep_ssd": "0.000000", 2024-09-13T11:22:23.680 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_stats_update_period_not_scrubbing": "120", 2024-09-13T11:22:23.680 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_stats_update_period_scrubbing": "15", 2024-09-13T11:22:23.680 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_target_pg_log_entries_per_osd": "300000", 2024-09-13T11:22:23.681 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_target_transaction_size": "30", 2024-09-13T11:22:23.681 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_tier_default_cache_hit_set_count": "4", 2024-09-13T11:22:23.681 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_tier_default_cache_hit_set_grade_decay_rate": "20", 2024-09-13T11:22:23.681 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_tier_default_cache_hit_set_period": "1200", 2024-09-13T11:22:23.681 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_tier_default_cache_hit_set_search_last_n": "1", 2024-09-13T11:22:23.681 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_tier_default_cache_hit_set_type": "bloom", 2024-09-13T11:22:23.681 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_tier_default_cache_min_read_recency_for_promote": "1", 2024-09-13T11:22:23.681 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_tier_default_cache_min_write_recency_for_promote": "1", 2024-09-13T11:22:23.681 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_tier_default_cache_mode": "writeback", 2024-09-13T11:22:23.681 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_tier_promote_max_bytes_sec": "5242880", 2024-09-13T11:22:23.681 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_tier_promote_max_objects_sec": "25", 2024-09-13T11:22:23.681 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_tracing": "false", 2024-09-13T11:22:23.681 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_use_stale_snap": "false", 2024-09-13T11:22:23.681 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_uuid": "00000000-0000-0000-0000-000000000000", 2024-09-13T11:22:23.682 INFO:teuthology.orchestra.run.smithi001.stdout: "osdc_blkin_trace_all": "false", 2024-09-13T11:22:23.682 INFO:teuthology.orchestra.run.smithi001.stdout: "paxos_kill_at": "0", 2024-09-13T11:22:23.682 INFO:teuthology.orchestra.run.smithi001.stdout: "paxos_max_join_drift": "10", 2024-09-13T11:22:23.682 INFO:teuthology.orchestra.run.smithi001.stdout: "paxos_min": "500", 2024-09-13T11:22:23.682 INFO:teuthology.orchestra.run.smithi001.stdout: "paxos_min_wait": "0.050000", 2024-09-13T11:22:23.682 INFO:teuthology.orchestra.run.smithi001.stdout: "paxos_propose_interval": "1.000000", 2024-09-13T11:22:23.682 INFO:teuthology.orchestra.run.smithi001.stdout: "paxos_service_trim_max": "500", 2024-09-13T11:22:23.682 INFO:teuthology.orchestra.run.smithi001.stdout: "paxos_service_trim_max_multiplier": "20", 2024-09-13T11:22:23.682 INFO:teuthology.orchestra.run.smithi001.stdout: "paxos_service_trim_min": "10", 2024-09-13T11:22:23.682 INFO:teuthology.orchestra.run.smithi001.stdout: "paxos_stash_full_interval": "25", 2024-09-13T11:22:23.682 INFO:teuthology.orchestra.run.smithi001.stdout: "paxos_trim_max": "500", 2024-09-13T11:22:23.682 INFO:teuthology.orchestra.run.smithi001.stdout: "paxos_trim_min": "250", 2024-09-13T11:22:23.682 INFO:teuthology.orchestra.run.smithi001.stdout: "perf": "true", 2024-09-13T11:22:23.682 INFO:teuthology.orchestra.run.smithi001.stdout: "pid_file": "/var/run/ceph/ceph-mon.a.pid", 2024-09-13T11:22:23.683 INFO:teuthology.orchestra.run.smithi001.stdout: "plugin_crypto_accelerator": "crypto_isal", 2024-09-13T11:22:23.683 INFO:teuthology.orchestra.run.smithi001.stdout: "plugin_dir": "/usr/lib/ceph", 2024-09-13T11:22:23.683 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "-", 2024-09-13T11:22:23.683 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrv": "", 2024-09-13T11:22:23.683 INFO:teuthology.orchestra.run.smithi001.stdout: "public_bind_addr": "-", 2024-09-13T11:22:23.683 INFO:teuthology.orchestra.run.smithi001.stdout: "public_network": "", 2024-09-13T11:22:23.683 INFO:teuthology.orchestra.run.smithi001.stdout: "public_network_interface": "", 2024-09-13T11:22:23.683 INFO:teuthology.orchestra.run.smithi001.stdout: "qat_compressor_busy_polling": "false", 2024-09-13T11:22:23.683 INFO:teuthology.orchestra.run.smithi001.stdout: "qat_compressor_enabled": "false", 2024-09-13T11:22:23.683 INFO:teuthology.orchestra.run.smithi001.stdout: "qat_compressor_session_max_number": "256", 2024-09-13T11:22:23.683 INFO:teuthology.orchestra.run.smithi001.stdout: "rados_mon_op_timeout": "0", 2024-09-13T11:22:23.683 INFO:teuthology.orchestra.run.smithi001.stdout: "rados_osd_op_timeout": "0", 2024-09-13T11:22:23.683 INFO:teuthology.orchestra.run.smithi001.stdout: "rados_tracing": "false", 2024-09-13T11:22:23.683 INFO:teuthology.orchestra.run.smithi001.stdout: "rbd_atime_update_interval": "60", 2024-09-13T11:22:23.684 INFO:teuthology.orchestra.run.smithi001.stdout: "rbd_auto_exclusive_lock_until_manual_request": "true", 2024-09-13T11:22:23.684 INFO:teuthology.orchestra.run.smithi001.stdout: "rbd_balance_parent_reads": "false", 2024-09-13T11:22:23.684 INFO:teuthology.orchestra.run.smithi001.stdout: "rbd_balance_snap_reads": "false", 2024-09-13T11:22:23.684 INFO:teuthology.orchestra.run.smithi001.stdout: "rbd_blkin_trace_all": "false", 2024-09-13T11:22:23.684 INFO:teuthology.orchestra.run.smithi001.stdout: "rbd_blocklist_expire_seconds": "0", 2024-09-13T11:22:23.684 INFO:teuthology.orchestra.run.smithi001.stdout: "rbd_blocklist_on_break_lock": "true", 2024-09-13T11:22:23.684 INFO:teuthology.orchestra.run.smithi001.stdout: "rbd_cache": "true", 2024-09-13T11:22:23.684 INFO:teuthology.orchestra.run.smithi001.stdout: "rbd_cache_block_writes_upfront": "false", 2024-09-13T11:22:23.684 INFO:teuthology.orchestra.run.smithi001.stdout: "rbd_cache_max_dirty": "25165824", 2024-09-13T11:22:23.684 INFO:teuthology.orchestra.run.smithi001.stdout: "rbd_cache_max_dirty_age": "1.000000", 2024-09-13T11:22:23.684 INFO:teuthology.orchestra.run.smithi001.stdout: "rbd_cache_max_dirty_object": "0", 2024-09-13T11:22:23.684 INFO:teuthology.orchestra.run.smithi001.stdout: "rbd_cache_policy": "writearound", 2024-09-13T11:22:23.684 INFO:teuthology.orchestra.run.smithi001.stdout: "rbd_cache_size": "33554432", 2024-09-13T11:22:23.684 INFO:teuthology.orchestra.run.smithi001.stdout: "rbd_cache_target_dirty": "16777216", 2024-09-13T11:22:23.685 INFO:teuthology.orchestra.run.smithi001.stdout: "rbd_cache_writethrough_until_flush": "true", 2024-09-13T11:22:23.685 INFO:teuthology.orchestra.run.smithi001.stdout: "rbd_clone_copy_on_read": "false", 2024-09-13T11:22:23.685 INFO:teuthology.orchestra.run.smithi001.stdout: "rbd_compression_hint": "none", 2024-09-13T11:22:23.685 INFO:teuthology.orchestra.run.smithi001.stdout: "rbd_concurrent_management_ops": "10", 2024-09-13T11:22:23.685 INFO:teuthology.orchestra.run.smithi001.stdout: "rbd_config_pool_override_update_timestamp": "0", 2024-09-13T11:22:23.685 INFO:teuthology.orchestra.run.smithi001.stdout: "rbd_default_clone_format": "auto", 2024-09-13T11:22:23.685 INFO:teuthology.orchestra.run.smithi001.stdout: "rbd_default_data_pool": "", 2024-09-13T11:22:23.685 INFO:teuthology.orchestra.run.smithi001.stdout: "rbd_default_features": "61", 2024-09-13T11:22:23.685 INFO:teuthology.orchestra.run.smithi001.stdout: "rbd_default_format": "2", 2024-09-13T11:22:23.685 INFO:teuthology.orchestra.run.smithi001.stdout: "rbd_default_map_options": "", 2024-09-13T11:22:23.685 INFO:teuthology.orchestra.run.smithi001.stdout: "rbd_default_order": "22", 2024-09-13T11:22:23.685 INFO:teuthology.orchestra.run.smithi001.stdout: "rbd_default_pool": "rbd", 2024-09-13T11:22:23.685 INFO:teuthology.orchestra.run.smithi001.stdout: "rbd_default_snapshot_quiesce_mode": "required", 2024-09-13T11:22:23.685 INFO:teuthology.orchestra.run.smithi001.stdout: "rbd_default_stripe_count": "0", 2024-09-13T11:22:23.686 INFO:teuthology.orchestra.run.smithi001.stdout: "rbd_default_stripe_unit": "0", 2024-09-13T11:22:23.686 INFO:teuthology.orchestra.run.smithi001.stdout: "rbd_disable_zero_copy_writes": "true", 2024-09-13T11:22:23.686 INFO:teuthology.orchestra.run.smithi001.stdout: "rbd_discard_granularity_bytes": "65536", 2024-09-13T11:22:23.686 INFO:teuthology.orchestra.run.smithi001.stdout: "rbd_discard_on_zeroed_write_same": "true", 2024-09-13T11:22:23.686 INFO:teuthology.orchestra.run.smithi001.stdout: "rbd_enable_alloc_hint": "true", 2024-09-13T11:22:23.686 INFO:teuthology.orchestra.run.smithi001.stdout: "rbd_invalidate_object_map_on_timeout": "true", 2024-09-13T11:22:23.686 INFO:teuthology.orchestra.run.smithi001.stdout: "rbd_io_scheduler": "simple", 2024-09-13T11:22:23.686 INFO:teuthology.orchestra.run.smithi001.stdout: "rbd_io_scheduler_simple_max_delay": "0", 2024-09-13T11:22:23.686 INFO:teuthology.orchestra.run.smithi001.stdout: "rbd_journal_commit_age": "5.000000", 2024-09-13T11:22:23.686 INFO:teuthology.orchestra.run.smithi001.stdout: "rbd_journal_max_concurrent_object_sets": "0", 2024-09-13T11:22:23.686 INFO:teuthology.orchestra.run.smithi001.stdout: "rbd_journal_max_payload_bytes": "16384", 2024-09-13T11:22:23.686 INFO:teuthology.orchestra.run.smithi001.stdout: "rbd_journal_object_flush_age": "0.000000", 2024-09-13T11:22:23.686 INFO:teuthology.orchestra.run.smithi001.stdout: "rbd_journal_object_flush_bytes": "1048576", 2024-09-13T11:22:23.686 INFO:teuthology.orchestra.run.smithi001.stdout: "rbd_journal_object_flush_interval": "0", 2024-09-13T11:22:23.687 INFO:teuthology.orchestra.run.smithi001.stdout: "rbd_journal_object_max_in_flight_appends": "0", 2024-09-13T11:22:23.687 INFO:teuthology.orchestra.run.smithi001.stdout: "rbd_journal_object_writethrough_until_flush": "true", 2024-09-13T11:22:23.687 INFO:teuthology.orchestra.run.smithi001.stdout: "rbd_journal_order": "24", 2024-09-13T11:22:23.687 INFO:teuthology.orchestra.run.smithi001.stdout: "rbd_journal_pool": "", 2024-09-13T11:22:23.687 INFO:teuthology.orchestra.run.smithi001.stdout: "rbd_journal_splay_width": "4", 2024-09-13T11:22:23.687 INFO:teuthology.orchestra.run.smithi001.stdout: "rbd_localize_parent_reads": "false", 2024-09-13T11:22:23.687 INFO:teuthology.orchestra.run.smithi001.stdout: "rbd_localize_snap_reads": "false", 2024-09-13T11:22:23.687 INFO:teuthology.orchestra.run.smithi001.stdout: "rbd_mirror_concurrent_image_deletions": "1", 2024-09-13T11:22:23.687 INFO:teuthology.orchestra.run.smithi001.stdout: "rbd_mirror_concurrent_image_syncs": "5", 2024-09-13T11:22:23.687 INFO:teuthology.orchestra.run.smithi001.stdout: "rbd_mirror_delete_retry_interval": "30.000000", 2024-09-13T11:22:23.687 INFO:teuthology.orchestra.run.smithi001.stdout: "rbd_mirror_image_perf_stats_prio": "5", 2024-09-13T11:22:23.687 INFO:teuthology.orchestra.run.smithi001.stdout: "rbd_mirror_image_policy_migration_throttle": "300", 2024-09-13T11:22:23.687 INFO:teuthology.orchestra.run.smithi001.stdout: "rbd_mirror_image_policy_rebalance_timeout": "0.000000", 2024-09-13T11:22:23.687 INFO:teuthology.orchestra.run.smithi001.stdout: "rbd_mirror_image_policy_type": "simple", 2024-09-13T11:22:23.688 INFO:teuthology.orchestra.run.smithi001.stdout: "rbd_mirror_image_policy_update_throttle_interval": "1.000000", 2024-09-13T11:22:23.688 INFO:teuthology.orchestra.run.smithi001.stdout: "rbd_mirror_image_state_check_interval": "30", 2024-09-13T11:22:23.688 INFO:teuthology.orchestra.run.smithi001.stdout: "rbd_mirror_journal_commit_age": "5.000000", 2024-09-13T11:22:23.688 INFO:teuthology.orchestra.run.smithi001.stdout: "rbd_mirror_journal_poll_age": "5.000000", 2024-09-13T11:22:23.688 INFO:teuthology.orchestra.run.smithi001.stdout: "rbd_mirror_leader_heartbeat_interval": "5", 2024-09-13T11:22:23.688 INFO:teuthology.orchestra.run.smithi001.stdout: "rbd_mirror_leader_max_acquire_attempts_before_break": "3", 2024-09-13T11:22:23.688 INFO:teuthology.orchestra.run.smithi001.stdout: "rbd_mirror_leader_max_missed_heartbeats": "2", 2024-09-13T11:22:23.688 INFO:teuthology.orchestra.run.smithi001.stdout: "rbd_mirror_memory_autotune": "true", 2024-09-13T11:22:23.688 INFO:teuthology.orchestra.run.smithi001.stdout: "rbd_mirror_memory_base": "805306368", 2024-09-13T11:22:23.688 INFO:teuthology.orchestra.run.smithi001.stdout: "rbd_mirror_memory_cache_autotune_interval": "30.000000", 2024-09-13T11:22:23.688 INFO:teuthology.orchestra.run.smithi001.stdout: "rbd_mirror_memory_cache_min": "134217728", 2024-09-13T11:22:23.688 INFO:teuthology.orchestra.run.smithi001.stdout: "rbd_mirror_memory_cache_resize_interval": "5.000000", 2024-09-13T11:22:23.688 INFO:teuthology.orchestra.run.smithi001.stdout: "rbd_mirror_memory_expected_fragmentation": "0.150000", 2024-09-13T11:22:23.688 INFO:teuthology.orchestra.run.smithi001.stdout: "rbd_mirror_memory_target": "4294967296", 2024-09-13T11:22:23.689 INFO:teuthology.orchestra.run.smithi001.stdout: "rbd_mirror_perf_stats_prio": "5", 2024-09-13T11:22:23.689 INFO:teuthology.orchestra.run.smithi001.stdout: "rbd_mirror_pool_replayers_refresh_interval": "30", 2024-09-13T11:22:23.689 INFO:teuthology.orchestra.run.smithi001.stdout: "rbd_mirror_sync_point_update_age": "30.000000", 2024-09-13T11:22:23.689 INFO:teuthology.orchestra.run.smithi001.stdout: "rbd_mirroring_delete_delay": "0", 2024-09-13T11:22:23.689 INFO:teuthology.orchestra.run.smithi001.stdout: "rbd_mirroring_max_mirroring_snapshots": "5", 2024-09-13T11:22:23.689 INFO:teuthology.orchestra.run.smithi001.stdout: "rbd_mirroring_replay_delay": "0", 2024-09-13T11:22:23.689 INFO:teuthology.orchestra.run.smithi001.stdout: "rbd_mirroring_resync_after_disconnect": "false", 2024-09-13T11:22:23.689 INFO:teuthology.orchestra.run.smithi001.stdout: "rbd_move_parent_to_trash_on_remove": "false", 2024-09-13T11:22:23.689 INFO:teuthology.orchestra.run.smithi001.stdout: "rbd_move_to_trash_on_remove": "false", 2024-09-13T11:22:23.689 INFO:teuthology.orchestra.run.smithi001.stdout: "rbd_move_to_trash_on_remove_expire_seconds": "0", 2024-09-13T11:22:23.689 INFO:teuthology.orchestra.run.smithi001.stdout: "rbd_mtime_update_interval": "60", 2024-09-13T11:22:23.689 INFO:teuthology.orchestra.run.smithi001.stdout: "rbd_non_blocking_aio": "true", 2024-09-13T11:22:23.689 INFO:teuthology.orchestra.run.smithi001.stdout: "rbd_op_thread_timeout": "60", 2024-09-13T11:22:23.690 INFO:teuthology.orchestra.run.smithi001.stdout: "rbd_op_threads": "1", 2024-09-13T11:22:23.690 INFO:teuthology.orchestra.run.smithi001.stdout: "rbd_parent_cache_enabled": "false", 2024-09-13T11:22:23.690 INFO:teuthology.orchestra.run.smithi001.stdout: "rbd_persistent_cache_mode": "disabled", 2024-09-13T11:22:23.690 INFO:teuthology.orchestra.run.smithi001.stdout: "rbd_persistent_cache_path": "/tmp", 2024-09-13T11:22:23.690 INFO:teuthology.orchestra.run.smithi001.stdout: "rbd_persistent_cache_size": "1073741824", 2024-09-13T11:22:23.690 INFO:teuthology.orchestra.run.smithi001.stdout: "rbd_plugins": "", 2024-09-13T11:22:23.690 INFO:teuthology.orchestra.run.smithi001.stdout: "rbd_qos_bps_burst": "0", 2024-09-13T11:22:23.690 INFO:teuthology.orchestra.run.smithi001.stdout: "rbd_qos_bps_burst_seconds": "1", 2024-09-13T11:22:23.690 INFO:teuthology.orchestra.run.smithi001.stdout: "rbd_qos_bps_limit": "0", 2024-09-13T11:22:23.690 INFO:teuthology.orchestra.run.smithi001.stdout: "rbd_qos_exclude_ops": "0", 2024-09-13T11:22:23.690 INFO:teuthology.orchestra.run.smithi001.stdout: "rbd_qos_iops_burst": "0", 2024-09-13T11:22:23.690 INFO:teuthology.orchestra.run.smithi001.stdout: "rbd_qos_iops_burst_seconds": "1", 2024-09-13T11:22:23.690 INFO:teuthology.orchestra.run.smithi001.stdout: "rbd_qos_iops_limit": "0", 2024-09-13T11:22:23.690 INFO:teuthology.orchestra.run.smithi001.stdout: "rbd_qos_read_bps_burst": "0", 2024-09-13T11:22:23.691 INFO:teuthology.orchestra.run.smithi001.stdout: "rbd_qos_read_bps_burst_seconds": "1", 2024-09-13T11:22:23.691 INFO:teuthology.orchestra.run.smithi001.stdout: "rbd_qos_read_bps_limit": "0", 2024-09-13T11:22:23.691 INFO:teuthology.orchestra.run.smithi001.stdout: "rbd_qos_read_iops_burst": "0", 2024-09-13T11:22:23.691 INFO:teuthology.orchestra.run.smithi001.stdout: "rbd_qos_read_iops_burst_seconds": "1", 2024-09-13T11:22:23.691 INFO:teuthology.orchestra.run.smithi001.stdout: "rbd_qos_read_iops_limit": "0", 2024-09-13T11:22:23.691 INFO:teuthology.orchestra.run.smithi001.stdout: "rbd_qos_schedule_tick_min": "50", 2024-09-13T11:22:23.691 INFO:teuthology.orchestra.run.smithi001.stdout: "rbd_qos_write_bps_burst": "0", 2024-09-13T11:22:23.691 INFO:teuthology.orchestra.run.smithi001.stdout: "rbd_qos_write_bps_burst_seconds": "1", 2024-09-13T11:22:23.691 INFO:teuthology.orchestra.run.smithi001.stdout: "rbd_qos_write_bps_limit": "0", 2024-09-13T11:22:23.691 INFO:teuthology.orchestra.run.smithi001.stdout: "rbd_qos_write_iops_burst": "0", 2024-09-13T11:22:23.691 INFO:teuthology.orchestra.run.smithi001.stdout: "rbd_qos_write_iops_burst_seconds": "1", 2024-09-13T11:22:23.691 INFO:teuthology.orchestra.run.smithi001.stdout: "rbd_qos_write_iops_limit": "0", 2024-09-13T11:22:23.691 INFO:teuthology.orchestra.run.smithi001.stdout: "rbd_quiesce_notification_attempts": "10", 2024-09-13T11:22:23.691 INFO:teuthology.orchestra.run.smithi001.stdout: "rbd_read_from_replica_policy": "default", 2024-09-13T11:22:23.692 INFO:teuthology.orchestra.run.smithi001.stdout: "rbd_readahead_disable_after_bytes": "52428800", 2024-09-13T11:22:23.692 INFO:teuthology.orchestra.run.smithi001.stdout: "rbd_readahead_max_bytes": "524288", 2024-09-13T11:22:23.692 INFO:teuthology.orchestra.run.smithi001.stdout: "rbd_readahead_trigger_requests": "10", 2024-09-13T11:22:23.692 INFO:teuthology.orchestra.run.smithi001.stdout: "rbd_request_timed_out_seconds": "30", 2024-09-13T11:22:23.692 INFO:teuthology.orchestra.run.smithi001.stdout: "rbd_skip_partial_discard": "true", 2024-09-13T11:22:23.692 INFO:teuthology.orchestra.run.smithi001.stdout: "rbd_sparse_read_threshold_bytes": "65536", 2024-09-13T11:22:23.692 INFO:teuthology.orchestra.run.smithi001.stdout: "rbd_tracing": "false", 2024-09-13T11:22:23.692 INFO:teuthology.orchestra.run.smithi001.stdout: "rbd_validate_names": "true", 2024-09-13T11:22:23.692 INFO:teuthology.orchestra.run.smithi001.stdout: "rbd_validate_pool": "true", 2024-09-13T11:22:23.692 INFO:teuthology.orchestra.run.smithi001.stdout: "restapi_base_url": "", 2024-09-13T11:22:23.692 INFO:teuthology.orchestra.run.smithi001.stdout: "restapi_log_level": "", 2024-09-13T11:22:23.692 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_account_default_quota_max_objects": "-1", 2024-09-13T11:22:23.692 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_account_default_quota_max_size": "-1", 2024-09-13T11:22:23.693 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_acl_grants_max_num": "100", 2024-09-13T11:22:23.693 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_admin_entry": "admin", 2024-09-13T11:22:23.693 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_allow_notification_secrets_in_cleartext": "false", 2024-09-13T11:22:23.693 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_asio_assert_yielding": "false", 2024-09-13T11:22:23.693 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_backend_store": "rados", 2024-09-13T11:22:23.693 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_barbican_url": "", 2024-09-13T11:22:23.693 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_beast_enable_async": "true", 2024-09-13T11:22:23.693 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_bucket_counters_cache": "false", 2024-09-13T11:22:23.693 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_bucket_counters_cache_size": "10000", 2024-09-13T11:22:23.693 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_bucket_default_quota_max_objects": "-1", 2024-09-13T11:22:23.693 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_bucket_default_quota_max_size": "-1", 2024-09-13T11:22:23.693 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_bucket_index_max_aio": "128", 2024-09-13T11:22:23.693 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_bucket_index_transaction_instrumentation": "false", 2024-09-13T11:22:23.694 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_bucket_quota_cache_size": "10000", 2024-09-13T11:22:23.694 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_bucket_quota_ttl": "600", 2024-09-13T11:22:23.694 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_bucket_sync_spawn_window": "20", 2024-09-13T11:22:23.694 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_cache_enabled": "true", 2024-09-13T11:22:23.694 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_cache_expiry_interval": "900", 2024-09-13T11:22:23.694 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_cache_lru_size": "25000", 2024-09-13T11:22:23.694 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_config_store": "rados", 2024-09-13T11:22:23.694 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_content_length_compat": "false", 2024-09-13T11:22:23.694 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_copy_obj_progress": "true", 2024-09-13T11:22:23.694 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_copy_obj_progress_every_bytes": "1048576", 2024-09-13T11:22:23.694 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_cors_rules_max_num": "100", 2024-09-13T11:22:23.694 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_cross_domain_policy": "", 2024-09-13T11:22:23.694 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_crypt_default_encryption_key": "", 2024-09-13T11:22:23.694 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_crypt_kmip_addr": "", 2024-09-13T11:22:23.695 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_crypt_kmip_ca_path": "", 2024-09-13T11:22:23.695 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_crypt_kmip_client_cert": "", 2024-09-13T11:22:23.695 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_crypt_kmip_client_key": "", 2024-09-13T11:22:23.695 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_crypt_kmip_kms_key_template": "", 2024-09-13T11:22:23.695 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_crypt_kmip_password": "", 2024-09-13T11:22:23.695 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_crypt_kmip_s3_key_template": "$keyid", 2024-09-13T11:22:23.695 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_crypt_kmip_username": "", 2024-09-13T11:22:23.695 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_crypt_require_ssl": "true", 2024-09-13T11:22:23.695 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_crypt_s3_kms_backend": "barbican", 2024-09-13T11:22:23.695 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_crypt_s3_kms_encryption_keys": "", 2024-09-13T11:22:23.695 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_crypt_sse_s3_backend": "vault", 2024-09-13T11:22:23.695 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_crypt_sse_s3_key_template": "%bucket_id", 2024-09-13T11:22:23.695 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_crypt_sse_s3_vault_addr": "", 2024-09-13T11:22:23.695 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_crypt_sse_s3_vault_auth": "token", 2024-09-13T11:22:23.696 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_crypt_sse_s3_vault_namespace": "", 2024-09-13T11:22:23.696 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_crypt_sse_s3_vault_prefix": "", 2024-09-13T11:22:23.696 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_crypt_sse_s3_vault_secret_engine": "transit", 2024-09-13T11:22:23.696 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_crypt_sse_s3_vault_ssl_cacert": "", 2024-09-13T11:22:23.696 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_crypt_sse_s3_vault_ssl_clientcert": "", 2024-09-13T11:22:23.696 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_crypt_sse_s3_vault_ssl_clientkey": "", 2024-09-13T11:22:23.696 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_crypt_sse_s3_vault_token_file": "", 2024-09-13T11:22:23.696 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_crypt_sse_s3_vault_verify_ssl": "true", 2024-09-13T11:22:23.696 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_crypt_suppress_logs": "true", 2024-09-13T11:22:23.696 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_crypt_vault_addr": "", 2024-09-13T11:22:23.696 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_crypt_vault_auth": "token", 2024-09-13T11:22:23.696 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_crypt_vault_namespace": "", 2024-09-13T11:22:23.696 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_crypt_vault_prefix": "", 2024-09-13T11:22:23.696 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_crypt_vault_secret_engine": "transit", 2024-09-13T11:22:23.697 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_crypt_vault_ssl_cacert": "", 2024-09-13T11:22:23.697 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_crypt_vault_ssl_clientcert": "", 2024-09-13T11:22:23.697 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_crypt_vault_ssl_clientkey": "", 2024-09-13T11:22:23.697 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_crypt_vault_token_file": "", 2024-09-13T11:22:23.697 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_crypt_vault_verify_ssl": "true", 2024-09-13T11:22:23.697 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_curl_buffersize": "524288", 2024-09-13T11:22:23.697 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_curl_low_speed_limit": "1024", 2024-09-13T11:22:23.697 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_curl_low_speed_time": "300", 2024-09-13T11:22:23.697 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_curl_tcp_keepalive": "0", 2024-09-13T11:22:23.697 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_curl_wait_timeout_ms": "1000", 2024-09-13T11:22:23.697 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_d3n_l1_datacache_persistent_path": "/tmp/rgw_datacache/", 2024-09-13T11:22:23.697 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_d3n_l1_datacache_size": "1073741824", 2024-09-13T11:22:23.697 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_d3n_l1_evict_cache_on_start": "true", 2024-09-13T11:22:23.698 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_d3n_l1_eviction_policy": "lru", 2024-09-13T11:22:23.698 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_d3n_l1_fadvise": "4", 2024-09-13T11:22:23.698 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_d3n_l1_local_datacache_enabled": "false", 2024-09-13T11:22:23.698 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_d3n_libaio_aio_num": "64", 2024-09-13T11:22:23.698 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_d3n_libaio_aio_threads": "20", 2024-09-13T11:22:23.698 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_d4n_address": "127.0.0.1:6379", 2024-09-13T11:22:23.698 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_d4n_l1_datacache_address": "127.0.0.1:6379", 2024-09-13T11:22:23.698 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_d4n_l1_datacache_persistent_path": "/tmp/rgw_d4n_datacache/", 2024-09-13T11:22:23.698 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_d4n_l1_datacache_size": "1073741824", 2024-09-13T11:22:23.698 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_d4n_l1_evict_cache_on_start": "true", 2024-09-13T11:22:23.698 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_d4n_l1_fadvise": "4", 2024-09-13T11:22:23.698 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_d4n_libaio_aio_num": "64", 2024-09-13T11:22:23.698 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_d4n_libaio_aio_threads": "20", 2024-09-13T11:22:23.698 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_data": "/var/lib/ceph/radosgw/ceph-a", 2024-09-13T11:22:23.698 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_data_log_changes_size": "1000", 2024-09-13T11:22:23.699 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_data_log_num_shards": "128", 2024-09-13T11:22:23.699 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_data_log_obj_prefix": "data_log", 2024-09-13T11:22:23.699 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_data_log_window": "30", 2024-09-13T11:22:23.699 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_data_notify_interval_msec": "0", 2024-09-13T11:22:23.699 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_data_sync_poll_interval": "20", 2024-09-13T11:22:23.699 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_data_sync_spawn_window": "20", 2024-09-13T11:22:23.699 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_debug_inject_latency_bi_unlink": "0", 2024-09-13T11:22:23.699 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_debug_inject_olh_cancel_modification_err": "false", 2024-09-13T11:22:23.699 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_debug_inject_set_olh_err": "0", 2024-09-13T11:22:23.699 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_default_data_log_backing": "fifo", 2024-09-13T11:22:23.699 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_default_realm_info_oid": "default.realm", 2024-09-13T11:22:23.699 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_default_region_info_oid": "default.region", 2024-09-13T11:22:23.699 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_default_zone_info_oid": "default.zone", 2024-09-13T11:22:23.699 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_default_zonegroup_info_oid": "default.zonegroup", 2024-09-13T11:22:23.700 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_defer_to_bucket_acls": "", 2024-09-13T11:22:23.700 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_delete_multi_obj_max_num": "1000", 2024-09-13T11:22:23.700 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_disable_s3select": "false", 2024-09-13T11:22:23.700 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_dmclock_admin_lim": "0.000000", 2024-09-13T11:22:23.700 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_dmclock_admin_res": "100.000000", 2024-09-13T11:22:23.700 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_dmclock_admin_wgt": "100.000000", 2024-09-13T11:22:23.700 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_dmclock_auth_lim": "0.000000", 2024-09-13T11:22:23.700 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_dmclock_auth_res": "200.000000", 2024-09-13T11:22:23.700 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_dmclock_auth_wgt": "100.000000", 2024-09-13T11:22:23.700 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_dmclock_data_lim": "0.000000", 2024-09-13T11:22:23.700 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_dmclock_data_res": "500.000000", 2024-09-13T11:22:23.700 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_dmclock_data_wgt": "500.000000", 2024-09-13T11:22:23.700 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_dmclock_metadata_lim": "0.000000", 2024-09-13T11:22:23.701 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_dmclock_metadata_res": "500.000000", 2024-09-13T11:22:23.701 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_dmclock_metadata_wgt": "500.000000", 2024-09-13T11:22:23.701 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_dns_name": "", 2024-09-13T11:22:23.701 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_dns_s3website_name": "", 2024-09-13T11:22:23.701 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_dynamic_resharding": "true", 2024-09-13T11:22:23.701 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_dynamic_resharding_may_reduce": "true", 2024-09-13T11:22:23.701 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_dynamic_resharding_reduction_wait": "120", 2024-09-13T11:22:23.701 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_enable_apis": "s3, s3website, swift, swift_auth, admin, sts, iam, notifications", 2024-09-13T11:22:23.701 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_enable_gc_threads": "true", 2024-09-13T11:22:23.701 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_enable_lc_threads": "true", 2024-09-13T11:22:23.701 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_enable_mdsearch": "true", 2024-09-13T11:22:23.701 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_enable_ops_log": "false", 2024-09-13T11:22:23.701 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_enable_quota_threads": "true", 2024-09-13T11:22:23.701 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_enable_static_website": "false", 2024-09-13T11:22:23.702 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_enable_usage_log": "false", 2024-09-13T11:22:23.702 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_enforce_swift_acls": "true", 2024-09-13T11:22:23.702 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_exit_timeout_secs": "120", 2024-09-13T11:22:23.702 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_expose_bucket": "false", 2024-09-13T11:22:23.702 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_extended_http_attrs": "", 2024-09-13T11:22:23.702 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_filter": "none", 2024-09-13T11:22:23.702 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_frontend_defaults": "beast ssl_certificate=config://rgw/cert/$realm/$zone.crt ssl_private_key=config://rgw/cert/$realm/$zone.key", 2024-09-13T11:22:23.702 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_frontends": "beast port=7480", 2024-09-13T11:22:23.702 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_gc_max_concurrent_io": "10", 2024-09-13T11:22:23.702 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_gc_max_deferred": "50", 2024-09-13T11:22:23.702 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_gc_max_deferred_entries_size": "3072", 2024-09-13T11:22:23.702 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_gc_max_objs": "32", 2024-09-13T11:22:23.702 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_gc_max_queue_size": "134213632", 2024-09-13T11:22:23.702 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_gc_max_trim_chunk": "16", 2024-09-13T11:22:23.703 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_gc_obj_min_wait": "7200", 2024-09-13T11:22:23.703 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_gc_processor_max_time": "3600", 2024-09-13T11:22:23.703 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_gc_processor_period": "3600", 2024-09-13T11:22:23.703 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_get_obj_max_req_size": "4194304", 2024-09-13T11:22:23.703 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_get_obj_window_size": "16777216", 2024-09-13T11:22:23.703 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_healthcheck_disabling_path": "", 2024-09-13T11:22:23.703 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_ignore_get_invalid_range": "false", 2024-09-13T11:22:23.703 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_init_timeout": "300", 2024-09-13T11:22:23.703 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_inject_notify_timeout_probability": "0.000000", 2024-09-13T11:22:23.703 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_json_config": "/var/lib/ceph/radosgw/config.json", 2024-09-13T11:22:23.703 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_kafka_connection_idle": "300", 2024-09-13T11:22:23.703 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_kafka_message_timeout": "5000", 2024-09-13T11:22:23.703 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_kafka_sleep_timeout": "10", 2024-09-13T11:22:23.703 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_keystone_accepted_admin_roles": "", 2024-09-13T11:22:23.704 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_keystone_accepted_reader_roles": "", 2024-09-13T11:22:23.704 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_keystone_accepted_roles": "Member, admin", 2024-09-13T11:22:23.704 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_keystone_admin_domain": "", 2024-09-13T11:22:23.704 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_keystone_admin_password": "", 2024-09-13T11:22:23.704 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_keystone_admin_password_path": "", 2024-09-13T11:22:23.704 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_keystone_admin_project": "", 2024-09-13T11:22:23.704 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_keystone_admin_tenant": "", 2024-09-13T11:22:23.704 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_keystone_admin_token": "", 2024-09-13T11:22:23.704 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_keystone_admin_token_path": "", 2024-09-13T11:22:23.704 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_keystone_admin_user": "", 2024-09-13T11:22:23.704 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_keystone_barbican_domain": "", 2024-09-13T11:22:23.704 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_keystone_barbican_password": "", 2024-09-13T11:22:23.704 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_keystone_barbican_project": "", 2024-09-13T11:22:23.704 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_keystone_barbican_tenant": "", 2024-09-13T11:22:23.705 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_keystone_barbican_user": "", 2024-09-13T11:22:23.705 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_keystone_expired_token_cache_expiration": "3600", 2024-09-13T11:22:23.705 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_keystone_implicit_tenants": "false", 2024-09-13T11:22:23.705 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_keystone_service_token_accepted_roles": "admin", 2024-09-13T11:22:23.705 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_keystone_service_token_enabled": "false", 2024-09-13T11:22:23.705 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_keystone_token_cache_size": "10000", 2024-09-13T11:22:23.705 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_keystone_url": "", 2024-09-13T11:22:23.705 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_keystone_verify_ssl": "true", 2024-09-13T11:22:23.705 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_lc_debug_interval": "-1", 2024-09-13T11:22:23.705 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_lc_lock_max_time": "90", 2024-09-13T11:22:23.705 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_lc_max_objs": "32", 2024-09-13T11:22:23.705 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_lc_max_rules": "1000", 2024-09-13T11:22:23.705 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_lc_max_worker": "3", 2024-09-13T11:22:23.705 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_lc_max_wp_worker": "3", 2024-09-13T11:22:23.706 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_lc_thread_delay": "0", 2024-09-13T11:22:23.706 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_ldap_binddn": "uid=admin,cn=users,dc=example,dc=com", 2024-09-13T11:22:23.706 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_ldap_dnattr": "uid", 2024-09-13T11:22:23.706 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_ldap_searchdn": "cn=users,cn=accounts,dc=example,dc=com", 2024-09-13T11:22:23.706 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_ldap_searchfilter": "", 2024-09-13T11:22:23.706 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_ldap_secret": "/etc/openldap/secret", 2024-09-13T11:22:23.706 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_ldap_uri": "", 2024-09-13T11:22:23.706 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_lfuda_sync_frequency": "60", 2024-09-13T11:22:23.706 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_lifecycle_work_time": "00:00-06:00", 2024-09-13T11:22:23.706 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_list_bucket_min_readahead": "1000", 2024-09-13T11:22:23.706 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_list_buckets_max_chunk": "1000", 2024-09-13T11:22:23.706 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_log_http_headers": "", 2024-09-13T11:22:23.706 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_log_nonexistent_bucket": "false", 2024-09-13T11:22:23.706 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_log_object_name": "%Y-%m-%d-%H-%i-%n", 2024-09-13T11:22:23.707 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_log_object_name_utc": "false", 2024-09-13T11:22:23.707 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_lua_max_memory_per_state": "128000", 2024-09-13T11:22:23.707 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_luarocks_location": "/tmp/rgw_luarocks/mon.a", 2024-09-13T11:22:23.707 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_max_attr_name_len": "0", 2024-09-13T11:22:23.707 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_max_attr_size": "0", 2024-09-13T11:22:23.707 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_max_attrs_num_in_req": "0", 2024-09-13T11:22:23.707 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_max_chunk_size": "4194304", 2024-09-13T11:22:23.707 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_max_concurrent_requests": "1024", 2024-09-13T11:22:23.707 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_max_copy_obj_concurrent_io": "10", 2024-09-13T11:22:23.707 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_max_dynamic_shards": "1999", 2024-09-13T11:22:23.707 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_max_listing_results": "1000", 2024-09-13T11:22:23.707 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_max_notify_retries": "10", 2024-09-13T11:22:23.707 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_max_objs_per_shard": "100000", 2024-09-13T11:22:23.708 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_max_put_param_size": "1048576", 2024-09-13T11:22:23.708 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_max_put_size": "5368709120", 2024-09-13T11:22:23.708 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_max_slo_entries": "1000", 2024-09-13T11:22:23.708 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_md_log_max_shards": "64", 2024-09-13T11:22:23.708 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_md_notify_interval_msec": "200", 2024-09-13T11:22:23.708 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_meta_sync_poll_interval": "20", 2024-09-13T11:22:23.708 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_meta_sync_spawn_window": "20", 2024-09-13T11:22:23.708 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_mime_types_file": "/etc/mime.types", 2024-09-13T11:22:23.708 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_mp_lock_max_time": "600", 2024-09-13T11:22:23.708 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_multi_obj_del_max_aio": "16", 2024-09-13T11:22:23.708 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_multipart_min_part_size": "5242880", 2024-09-13T11:22:23.708 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_multipart_part_upload_limit": "10000", 2024-09-13T11:22:23.708 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_nfs_fhcache_partitions": "3", 2024-09-13T11:22:23.708 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_nfs_fhcache_size": "2017", 2024-09-13T11:22:23.709 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_nfs_frontends": "rgw-nfs", 2024-09-13T11:22:23.709 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_nfs_lru_lane_hiwat": "911", 2024-09-13T11:22:23.709 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_nfs_lru_lanes": "5", 2024-09-13T11:22:23.709 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_nfs_max_gc": "300", 2024-09-13T11:22:23.709 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_nfs_namespace_expire_secs": "300", 2024-09-13T11:22:23.709 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_nfs_run_gc_threads": "false", 2024-09-13T11:22:23.709 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_nfs_run_lc_threads": "false", 2024-09-13T11:22:23.709 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_nfs_run_quota_threads": "false", 2024-09-13T11:22:23.709 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_nfs_run_sync_thread": "false", 2024-09-13T11:22:23.709 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_nfs_s3_fast_attrs": "false", 2024-09-13T11:22:23.709 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_nfs_write_completion_interval_s": "10", 2024-09-13T11:22:23.709 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_num_async_rados_threads": "32", 2024-09-13T11:22:23.709 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_num_control_oids": "8", 2024-09-13T11:22:23.709 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_numa_node": "-1", 2024-09-13T11:22:23.710 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_obj_stripe_size": "4194304", 2024-09-13T11:22:23.710 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_obj_tombstone_cache_size": "1000", 2024-09-13T11:22:23.710 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_objexp_chunk_size": "100", 2024-09-13T11:22:23.710 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_objexp_gc_interval": "600", 2024-09-13T11:22:23.710 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_objexp_hints_num_shards": "127", 2024-09-13T11:22:23.710 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_olh_pending_timeout_sec": "3600", 2024-09-13T11:22:23.710 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_op_thread_suicide_timeout": "0", 2024-09-13T11:22:23.710 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_op_thread_timeout": "600", 2024-09-13T11:22:23.710 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_op_tracing": "false", 2024-09-13T11:22:23.710 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_opa_token": "", 2024-09-13T11:22:23.710 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_opa_url": "", 2024-09-13T11:22:23.710 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_opa_verify_ssl": "true", 2024-09-13T11:22:23.710 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_ops_log_data_backlog": "5242880", 2024-09-13T11:22:23.710 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_ops_log_file_path": "/var/log/ceph/ops-log-ceph-mon.a.log", 2024-09-13T11:22:23.711 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_ops_log_rados": "false", 2024-09-13T11:22:23.711 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_ops_log_socket_path": "", 2024-09-13T11:22:23.711 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_override_bucket_index_max_shards": "0", 2024-09-13T11:22:23.711 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_pending_bucket_index_op_expiration": "120", 2024-09-13T11:22:23.711 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_period_latest_epoch_info_oid": ".latest_epoch", 2024-09-13T11:22:23.711 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_period_push_interval": "2.000000", 2024-09-13T11:22:23.711 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_period_push_interval_max": "30.000000", 2024-09-13T11:22:23.711 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_period_root_pool": ".rgw.root", 2024-09-13T11:22:23.711 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_policy_reject_invalid_principals": "true", 2024-09-13T11:22:23.711 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_posix_base_path": "/tmp/rgw_posix_driver", 2024-09-13T11:22:23.711 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_posix_cache_lanes": "3", 2024-09-13T11:22:23.711 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_posix_cache_lmdb_count": "3", 2024-09-13T11:22:23.711 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_posix_cache_max_buckets": "100", 2024-09-13T11:22:23.711 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_posix_cache_partitions": "3", 2024-09-13T11:22:23.712 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_posix_database_root": "/var/lib/ceph/radosgw", 2024-09-13T11:22:23.712 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_print_continue": "true", 2024-09-13T11:22:23.712 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_print_prohibited_content_length": "false", 2024-09-13T11:22:23.712 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_put_obj_max_window_size": "67108864", 2024-09-13T11:22:23.712 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_put_obj_min_window_size": "16777216", 2024-09-13T11:22:23.712 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_rados_pool_autoscale_bias": "4.000000", 2024-09-13T11:22:23.712 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_rados_pool_recovery_priority": "5", 2024-09-13T11:22:23.712 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_rados_tracing": "false", 2024-09-13T11:22:23.712 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_realm": "", 2024-09-13T11:22:23.712 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_realm_id": "", 2024-09-13T11:22:23.712 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_realm_root_pool": ".rgw.root", 2024-09-13T11:22:23.712 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_region": "", 2024-09-13T11:22:23.712 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_region_root_pool": ".rgw.root", 2024-09-13T11:22:23.713 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_relaxed_region_enforcement": "false", 2024-09-13T11:22:23.713 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_relaxed_s3_bucket_names": "false", 2024-09-13T11:22:23.713 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_remote_addr_param": "REMOTE_ADDR", 2024-09-13T11:22:23.713 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_request_uri": "", 2024-09-13T11:22:23.713 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_reshard_batch_size": "64", 2024-09-13T11:22:23.713 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_reshard_bucket_lock_duration": "360", 2024-09-13T11:22:23.713 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_reshard_debug_interval": "-1", 2024-09-13T11:22:23.713 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_reshard_max_aio": "128", 2024-09-13T11:22:23.713 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_reshard_num_logs": "16", 2024-09-13T11:22:23.713 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_reshard_thread_interval": "600", 2024-09-13T11:22:23.713 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_resolve_cname": "false", 2024-09-13T11:22:23.713 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_rest_getusage_op_compat": "false", 2024-09-13T11:22:23.713 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_run_sync_thread": "true", 2024-09-13T11:22:23.713 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_s3_auth_disable_signature_url": "false", 2024-09-13T11:22:23.713 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_s3_auth_order": "sts, external, local", 2024-09-13T11:22:23.714 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_s3_auth_use_keystone": "false", 2024-09-13T11:22:23.714 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_s3_auth_use_ldap": "false", 2024-09-13T11:22:23.714 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_s3_auth_use_rados": "true", 2024-09-13T11:22:23.714 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_s3_auth_use_sts": "false", 2024-09-13T11:22:23.714 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_s3_client_max_sig_ver": "-1", 2024-09-13T11:22:23.714 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_s3_success_create_obj_status": "0", 2024-09-13T11:22:23.714 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_safe_max_objects_per_shard": "102400", 2024-09-13T11:22:23.714 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_scheduler_type": "throttler", 2024-09-13T11:22:23.714 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_script_uri": "", 2024-09-13T11:22:23.714 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_service_provider_name": "", 2024-09-13T11:22:23.714 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_shard_warning_threshold": "90.000000", 2024-09-13T11:22:23.714 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_sts_client_id": "", 2024-09-13T11:22:23.714 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_sts_client_secret": "", 2024-09-13T11:22:23.714 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_sts_entry": "sts", 2024-09-13T11:22:23.715 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_sts_key": "sts", 2024-09-13T11:22:23.715 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_sts_max_session_duration": "43200", 2024-09-13T11:22:23.715 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_sts_min_session_duration": "900", 2024-09-13T11:22:23.715 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_sts_token_introspection_url": "", 2024-09-13T11:22:23.715 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_swift_account_in_url": "false", 2024-09-13T11:22:23.715 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_swift_auth_entry": "auth", 2024-09-13T11:22:23.715 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_swift_auth_url": "", 2024-09-13T11:22:23.715 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_swift_custom_header": "", 2024-09-13T11:22:23.715 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_swift_enforce_content_length": "false", 2024-09-13T11:22:23.715 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_swift_need_stats": "true", 2024-09-13T11:22:23.715 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_swift_tenant_name": "", 2024-09-13T11:22:23.715 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_swift_token_expiration": "86400", 2024-09-13T11:22:23.715 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_swift_url": "", 2024-09-13T11:22:23.716 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_swift_url_prefix": "swift", 2024-09-13T11:22:23.716 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_swift_versioning_enabled": "false", 2024-09-13T11:22:23.716 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_sync_data_full_inject_err_probability": "0.000000", 2024-09-13T11:22:23.716 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_sync_data_inject_err_probability": "0.000000", 2024-09-13T11:22:23.716 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_sync_lease_period": "120", 2024-09-13T11:22:23.716 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_sync_log_trim_concurrent_buckets": "4", 2024-09-13T11:22:23.716 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_sync_log_trim_interval": "1200", 2024-09-13T11:22:23.716 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_sync_log_trim_max_buckets": "16", 2024-09-13T11:22:23.716 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_sync_log_trim_min_cold_buckets": "4", 2024-09-13T11:22:23.716 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_sync_meta_inject_err_probability": "0.000000", 2024-09-13T11:22:23.716 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_sync_obj_etag_verify": "false", 2024-09-13T11:22:23.716 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_sync_trace_history_size": "4096", 2024-09-13T11:22:23.716 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_sync_trace_per_node_log_size": "32", 2024-09-13T11:22:23.716 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_sync_trace_servicemap_update_interval": "10", 2024-09-13T11:22:23.717 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_thread_pool_size": "512", 2024-09-13T11:22:23.717 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_topic_persistency_max_retries": "0", 2024-09-13T11:22:23.717 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_topic_persistency_sleep_duration": "0", 2024-09-13T11:22:23.717 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_topic_persistency_time_to_live": "0", 2024-09-13T11:22:23.717 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_topic_require_publish_policy": "false", 2024-09-13T11:22:23.717 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_torrent_comment": "", 2024-09-13T11:22:23.717 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_torrent_createby": "", 2024-09-13T11:22:23.717 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_torrent_encoding": "", 2024-09-13T11:22:23.717 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_torrent_flag": "false", 2024-09-13T11:22:23.717 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_torrent_max_size": "5368709120", 2024-09-13T11:22:23.717 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_torrent_origin": "", 2024-09-13T11:22:23.717 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_torrent_sha_unit": "524288", 2024-09-13T11:22:23.717 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_torrent_tracker": "", 2024-09-13T11:22:23.717 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_trust_forwarded_https": "false", 2024-09-13T11:22:23.718 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_usage_log_flush_threshold": "1024", 2024-09-13T11:22:23.718 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_usage_log_tick_interval": "30", 2024-09-13T11:22:23.718 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_usage_max_shards": "32", 2024-09-13T11:22:23.718 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_usage_max_user_shards": "1", 2024-09-13T11:22:23.718 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_use_opa_authz": "false", 2024-09-13T11:22:23.718 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_user_counters_cache": "false", 2024-09-13T11:22:23.718 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_user_counters_cache_size": "10000", 2024-09-13T11:22:23.718 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_user_default_quota_max_objects": "-1", 2024-09-13T11:22:23.718 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_user_default_quota_max_size": "-1", 2024-09-13T11:22:23.718 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_user_max_buckets": "1000", 2024-09-13T11:22:23.718 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_user_policies_max_num": "100", 2024-09-13T11:22:23.718 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_user_quota_bucket_sync_interval": "180", 2024-09-13T11:22:23.718 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_user_quota_sync_idle_users": "false", 2024-09-13T11:22:23.719 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_user_quota_sync_interval": "86400", 2024-09-13T11:22:23.719 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_user_quota_sync_wait_time": "86400", 2024-09-13T11:22:23.719 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_user_unique_email": "true", 2024-09-13T11:22:23.719 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_verify_ssl": "true", 2024-09-13T11:22:23.719 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_website_routing_rules_max_num": "50", 2024-09-13T11:22:23.719 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_zone": "", 2024-09-13T11:22:23.719 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_zone_id": "", 2024-09-13T11:22:23.719 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_zone_root_pool": ".rgw.root", 2024-09-13T11:22:23.719 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_zonegroup": "", 2024-09-13T11:22:23.719 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_zonegroup_id": "", 2024-09-13T11:22:23.719 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_zonegroup_root_pool": ".rgw.root", 2024-09-13T11:22:23.719 INFO:teuthology.orchestra.run.smithi001.stdout: "rgwlc_auto_session_clear": "true", 2024-09-13T11:22:23.719 INFO:teuthology.orchestra.run.smithi001.stdout: "rgwlc_skip_bucket_step": "false", 2024-09-13T11:22:23.719 INFO:teuthology.orchestra.run.smithi001.stdout: "rocksdb_block_size": "4096", 2024-09-13T11:22:23.720 INFO:teuthology.orchestra.run.smithi001.stdout: "rocksdb_bloom_bits_per_key": "20", 2024-09-13T11:22:23.720 INFO:teuthology.orchestra.run.smithi001.stdout: "rocksdb_cache_index_and_filter_blocks": "true", 2024-09-13T11:22:23.720 INFO:teuthology.orchestra.run.smithi001.stdout: "rocksdb_cache_index_and_filter_blocks_with_high_priority": "false", 2024-09-13T11:22:23.720 INFO:teuthology.orchestra.run.smithi001.stdout: "rocksdb_cache_row_ratio": "0.000000", 2024-09-13T11:22:23.720 INFO:teuthology.orchestra.run.smithi001.stdout: "rocksdb_cache_shard_bits": "4", 2024-09-13T11:22:23.720 INFO:teuthology.orchestra.run.smithi001.stdout: "rocksdb_cache_size": "536870912", 2024-09-13T11:22:23.720 INFO:teuthology.orchestra.run.smithi001.stdout: "rocksdb_cache_type": "binned_lru", 2024-09-13T11:22:23.720 INFO:teuthology.orchestra.run.smithi001.stdout: "rocksdb_cf_compact_on_deletion": "true", 2024-09-13T11:22:23.720 INFO:teuthology.orchestra.run.smithi001.stdout: "rocksdb_cf_compact_on_deletion_sliding_window": "32768", 2024-09-13T11:22:23.720 INFO:teuthology.orchestra.run.smithi001.stdout: "rocksdb_cf_compact_on_deletion_trigger": "16384", 2024-09-13T11:22:23.720 INFO:teuthology.orchestra.run.smithi001.stdout: "rocksdb_collect_compaction_stats": "false", 2024-09-13T11:22:23.720 INFO:teuthology.orchestra.run.smithi001.stdout: "rocksdb_collect_extended_stats": "false", 2024-09-13T11:22:23.720 INFO:teuthology.orchestra.run.smithi001.stdout: "rocksdb_collect_memory_stats": "false", 2024-09-13T11:22:23.720 INFO:teuthology.orchestra.run.smithi001.stdout: "rocksdb_delete_range_threshold": "1048576", 2024-09-13T11:22:23.721 INFO:teuthology.orchestra.run.smithi001.stdout: "rocksdb_index_type": "binary_search", 2024-09-13T11:22:23.721 INFO:teuthology.orchestra.run.smithi001.stdout: "rocksdb_log_to_ceph_log": "true", 2024-09-13T11:22:23.721 INFO:teuthology.orchestra.run.smithi001.stdout: "rocksdb_metadata_block_size": "4096", 2024-09-13T11:22:23.721 INFO:teuthology.orchestra.run.smithi001.stdout: "rocksdb_partition_filters": "false", 2024-09-13T11:22:23.721 INFO:teuthology.orchestra.run.smithi001.stdout: "rocksdb_perf": "false", 2024-09-13T11:22:23.721 INFO:teuthology.orchestra.run.smithi001.stdout: "rocksdb_pin_l0_filter_and_index_blocks_in_cache": "false", 2024-09-13T11:22:23.721 INFO:teuthology.orchestra.run.smithi001.stdout: "rotating_keys_bootstrap_timeout": "30", 2024-09-13T11:22:23.721 INFO:teuthology.orchestra.run.smithi001.stdout: "rotating_keys_renewal_timeout": "10", 2024-09-13T11:22:23.721 INFO:teuthology.orchestra.run.smithi001.stdout: "run_dir": "/var/run/ceph", 2024-09-13T11:22:23.721 INFO:teuthology.orchestra.run.smithi001.stdout: "seastore_block_create": "true", 2024-09-13T11:22:23.721 INFO:teuthology.orchestra.run.smithi001.stdout: "seastore_cache_lru_size": "67108864", 2024-09-13T11:22:23.721 INFO:teuthology.orchestra.run.smithi001.stdout: "seastore_cbjournal_size": "5368709120", 2024-09-13T11:22:23.721 INFO:teuthology.orchestra.run.smithi001.stdout: "seastore_data_delta_based_overwrite": "0", 2024-09-13T11:22:23.721 INFO:teuthology.orchestra.run.smithi001.stdout: "seastore_default_max_object_size": "16777216", 2024-09-13T11:22:23.722 INFO:teuthology.orchestra.run.smithi001.stdout: "seastore_default_object_metadata_reservation": "16777216", 2024-09-13T11:22:23.722 INFO:teuthology.orchestra.run.smithi001.stdout: "seastore_device_size": "53687091200", 2024-09-13T11:22:23.722 INFO:teuthology.orchestra.run.smithi001.stdout: "seastore_disable_end_to_end_data_protection": "true", 2024-09-13T11:22:23.722 INFO:teuthology.orchestra.run.smithi001.stdout: "seastore_full_integrity_check": "false", 2024-09-13T11:22:23.722 INFO:teuthology.orchestra.run.smithi001.stdout: "seastore_journal_batch_capacity": "16", 2024-09-13T11:22:23.722 INFO:teuthology.orchestra.run.smithi001.stdout: "seastore_journal_batch_flush_size": "16777216", 2024-09-13T11:22:23.722 INFO:teuthology.orchestra.run.smithi001.stdout: "seastore_journal_batch_preferred_fullness": "0.950000", 2024-09-13T11:22:23.722 INFO:teuthology.orchestra.run.smithi001.stdout: "seastore_journal_iodepth_limit": "5", 2024-09-13T11:22:23.722 INFO:teuthology.orchestra.run.smithi001.stdout: "seastore_main_device_type": "SSD", 2024-09-13T11:22:23.722 INFO:teuthology.orchestra.run.smithi001.stdout: "seastore_max_concurrent_transactions": "8", 2024-09-13T11:22:23.722 INFO:teuthology.orchestra.run.smithi001.stdout: "seastore_max_data_allocation_size": "32768", 2024-09-13T11:22:23.722 INFO:teuthology.orchestra.run.smithi001.stdout: "seastore_multiple_tiers_default_evict_ratio": "0.600000", 2024-09-13T11:22:23.722 INFO:teuthology.orchestra.run.smithi001.stdout: "seastore_multiple_tiers_fast_evict_ratio": "0.700000", 2024-09-13T11:22:23.722 INFO:teuthology.orchestra.run.smithi001.stdout: "seastore_multiple_tiers_stop_evict_ratio": "0.500000", 2024-09-13T11:22:23.723 INFO:teuthology.orchestra.run.smithi001.stdout: "seastore_obj_data_write_amplification": "1.250000", 2024-09-13T11:22:23.723 INFO:teuthology.orchestra.run.smithi001.stdout: "seastore_segment_size": "67108864", 2024-09-13T11:22:23.723 INFO:teuthology.orchestra.run.smithi001.stdout: "set_keepcaps": "false", 2024-09-13T11:22:23.723 INFO:teuthology.orchestra.run.smithi001.stdout: "setgroup": "", 2024-09-13T11:22:23.723 INFO:teuthology.orchestra.run.smithi001.stdout: "setuser": "", 2024-09-13T11:22:23.723 INFO:teuthology.orchestra.run.smithi001.stdout: "setuser_match_path": "", 2024-09-13T11:22:23.723 INFO:teuthology.orchestra.run.smithi001.stdout: "target_max_misplaced_ratio": "0.050000", 2024-09-13T11:22:23.723 INFO:teuthology.orchestra.run.smithi001.stdout: "thp": "false", 2024-09-13T11:22:23.723 INFO:teuthology.orchestra.run.smithi001.stdout: "threadpool_default_timeout": "60", 2024-09-13T11:22:23.723 INFO:teuthology.orchestra.run.smithi001.stdout: "threadpool_empty_queue_max_wait": "2", 2024-09-13T11:22:23.723 INFO:teuthology.orchestra.run.smithi001.stdout: "throttler_perf_counter": "true", 2024-09-13T11:22:23.723 INFO:teuthology.orchestra.run.smithi001.stdout: "tmp_dir": "/tmp", 2024-09-13T11:22:23.723 INFO:teuthology.orchestra.run.smithi001.stdout: "tmp_file_template": "/tmp/ceph-mon.a.XXXXXX", 2024-09-13T11:22:23.723 INFO:teuthology.orchestra.run.smithi001.stdout: "uadk_compressor_enabled": "false", 2024-09-13T11:22:23.724 INFO:teuthology.orchestra.run.smithi001.stdout: "uadk_wd_sync_ctx_num": "2" 2024-09-13T11:22:23.724 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-13T11:22:23.726 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'mon.*' injectargs --mon_osd_down_out_interval=0 2024-09-13T11:22:23.879 INFO:teuthology.orchestra.run.smithi001.stdout:mon.a: {} 2024-09-13T11:22:23.879 INFO:teuthology.orchestra.run.smithi001.stderr:mon.a: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-13T11:22:23.898 INFO:teuthology.orchestra.run.smithi001.stdout:mon.b: {} 2024-09-13T11:22:23.898 INFO:teuthology.orchestra.run.smithi001.stderr:mon.b: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-13T11:22:23.917 INFO:teuthology.orchestra.run.smithi001.stdout:mon.c: {} 2024-09-13T11:22:23.917 INFO:teuthology.orchestra.run.smithi001.stderr:mon.c: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-13T11:22:23.933 DEBUG:teuthology.orchestra.run.smithi177:> type ceph-objectstore-tool 2024-09-13T11:22:23.939 INFO:teuthology.orchestra.run.smithi177.stdout:ceph-objectstore-tool is /usr/bin/ceph-objectstore-tool 2024-09-13T11:22:23.939 DEBUG:teuthology.orchestra.run.smithi092:> type ceph-objectstore-tool 2024-09-13T11:22:23.945 INFO:teuthology.orchestra.run.smithi092.stdout:ceph-objectstore-tool is /usr/bin/ceph-objectstore-tool 2024-09-13T11:22:23.945 DEBUG:teuthology.orchestra.run.smithi162:> type ceph-objectstore-tool 2024-09-13T11:22:23.950 INFO:teuthology.orchestra.run.smithi162.stdout:ceph-objectstore-tool is /usr/bin/ceph-objectstore-tool 2024-09-13T11:22:23.951 DEBUG:teuthology.orchestra.run.smithi001:> type ceph-objectstore-tool 2024-09-13T11:22:23.955 INFO:teuthology.orchestra.run.smithi001.stdout:ceph-objectstore-tool is /usr/bin/ceph-objectstore-tool 2024-09-13T11:22:23.956 INFO:teuthology.run_tasks:Running task radosbench... 2024-09-13T11:22:23.967 INFO:tasks.radosbench:Beginning radosbench... 2024-09-13T11:22:23.968 INFO:tasks.ceph.ceph_manager.ceph:creating pool_name unique_pool_0 2024-09-13T11:22:23.968 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create unique_pool_0 16 2024-09-13T11:22:23.968 INFO:tasks.thrashosds.thrasher:starting do_thrash 2024-09-13T11:22:23.968 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15] out_osds: [] dead_osds: [] live_osds: [0, 4, 8, 12, 1, 5, 9, 13, 2, 6, 10, 14, 3, 7, 11, 15] 2024-09-13T11:22:23.968 INFO:tasks.thrashosds.thrasher:choose_action: min_in 2 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-13T11:22:23.969 INFO:tasks.thrashosds.thrasher:Killing osd 1, live_osds are [0, 4, 8, 12, 1, 5, 9, 13, 2, 6, 10, 14, 3, 7, 11, 15] 2024-09-13T11:22:23.969 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-09-13T11:22:23.969 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T11:22:23.969 INFO:tasks.thrashosds.thrasher:starting do_sighup with a delay of 0.1 2024-09-13T11:22:23.969 INFO:tasks.thrashosds.thrasher:starting do_optrack_toggle with a delay of 2.0 2024-09-13T11:22:23.970 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T11:22:23.970 INFO:tasks.thrashosds.thrasher:starting do_dump_ops 2024-09-13T11:22:23.970 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-13T11:22:23.970 INFO:tasks.thrashosds.thrasher:starting do_noscrub_toggle with a delay of 2.0 2024-09-13T11:22:23.971 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-13T11:22:23.992 INFO:tasks.ceph.osd.5.smithi092.stderr:2024-09-13T11:22:23.992+0000 7f6f0b705640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26346) UID: 0 2024-09-13T11:22:24.056 INFO:tasks.ceph.osd.1:Stopped 2024-09-13T11:22:24.056 INFO:tasks.thrashosds.thrasher:Testing ceph-objectstore-tool on down osd.1 2024-09-13T11:22:24.057 INFO:tasks.thrashosds.thrasher:remote for osd 1 is ubuntu@smithi092.front.sepia.ceph.com 2024-09-13T11:22:24.057 DEBUG:teuthology.orchestra.run.smithi092:> type ceph-objectstore-tool 2024-09-13T11:22:24.071 INFO:tasks.ceph.osd.15.smithi177.stderr:2024-09-13T11:22:24.069+0000 7f9d6ce77640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26578) UID: 0 2024-09-13T11:22:24.105 INFO:teuthology.orchestra.run.smithi092.stdout:ceph-objectstore-tool is /usr/bin/ceph-objectstore-tool 2024-09-13T11:22:24.106 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-1 --journal-path /var/lib/ceph/osd/ceph-1/journal --op list-pgs 2024-09-13T11:22:24.116 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-13T11:22:24.116 INFO:teuthology.orchestra.run.smithi001.stdout: "ops": [], 2024-09-13T11:22:24.116 INFO:teuthology.orchestra.run.smithi001.stdout: "num_ops": 0 2024-09-13T11:22:24.116 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-13T11:22:24.132 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-13T11:22:24.140 INFO:teuthology.orchestra.run.smithi001.stdout:osd.0: {} 2024-09-13T11:22:24.140 INFO:teuthology.orchestra.run.smithi001.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T11:22:24.191 INFO:tasks.ceph.osd.5.smithi092.stderr:2024-09-13T11:22:24.192+0000 7f6f0b705640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26346) UID: 0 2024-09-13T11:22:24.244 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-13T11:22:24.245 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:22:24.258 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-13T11:22:24.300 INFO:tasks.ceph.osd.12.smithi001.stderr:2024-09-13T11:22:24.297+0000 7f892c6e6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26314) UID: 0 2024-09-13T11:22:24.374 INFO:tasks.ceph.osd.12.smithi001.stderr:2024-09-13T11:22:24.369+0000 7f892c6e6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26314) UID: 0 2024-09-13T11:22:24.420 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-13T11:22:24.420 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:22:24.434 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-13T11:22:24.475 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-09-13T11:22:24.472+0000 7f97befc9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26315) UID: 0 2024-09-13T11:22:24.510 INFO:teuthology.orchestra.run.smithi092.stdout:2.4 2024-09-13T11:22:24.511 INFO:teuthology.orchestra.run.smithi092.stdout:2.1 2024-09-13T11:22:24.575 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-13T11:22:24.569+0000 7f6a68775640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26307) UID: 0 2024-09-13T11:22:24.602 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-13T11:22:24.603 INFO:teuthology.orchestra.run.smithi001.stdout: "ops": [], 2024-09-13T11:22:24.603 INFO:teuthology.orchestra.run.smithi001.stdout: "num_ops": 0 2024-09-13T11:22:24.603 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-13T11:22:24.617 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-13T11:22:24.676 INFO:tasks.ceph.osd.9.smithi092.stderr:2024-09-13T11:22:24.676+0000 7ff5ab3d0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26348) UID: 0 2024-09-13T11:22:24.777 INFO:tasks.ceph.osd.9.smithi092.stderr:2024-09-13T11:22:24.776+0000 7ff5ab3d0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26348) UID: 0 2024-09-13T11:22:24.786 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-13T11:22:24.786 INFO:teuthology.orchestra.run.smithi001.stdout: "ops": [], 2024-09-13T11:22:24.786 INFO:teuthology.orchestra.run.smithi001.stdout: "complaint_time": 30, 2024-09-13T11:22:24.786 INFO:teuthology.orchestra.run.smithi001.stdout: "num_blocked_ops": 0 2024-09-13T11:22:24.786 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-13T11:22:24.801 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-13T11:22:24.828 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-1 --journal-path /var/lib/ceph/osd/ceph-1/journal --op export --pgid 2.4 --file /var/log/ceph/exp.2.4.1 2024-09-13T11:22:24.878 INFO:tasks.ceph.osd.15.smithi177.stderr:2024-09-13T11:22:24.877+0000 7f9d6ce77640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26578) UID: 0 2024-09-13T11:22:24.908 INFO:teuthology.orchestra.run.smithi001.stderr:pool 'unique_pool_0' created 2024-09-13T11:22:24.912 INFO:tasks.ceph.osd.7.smithi177.stderr:2024-09-13T11:22:24.909+0000 7f8971f4e640 -1 osd.7 29 handle_osd_map injecting map crc failure 2024-09-13T11:22:24.913 INFO:teuthology.orchestra.run.smithi001.stderr:noscrub is set 2024-09-13T11:22:24.926 INFO:tasks.ceph.osd.13.smithi092.stderr:2024-09-13T11:22:24.924+0000 7f3e4ba0f640 -1 osd.13 29 handle_osd_map injecting map crc failure 2024-09-13T11:22:24.933 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool application enable unique_pool_0 rados --yes-i-really-mean-it || true 2024-09-13T11:22:24.934 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-13T11:22:24.934 INFO:teuthology.orchestra.run.smithi001.stdout: "size": 20, 2024-09-13T11:22:24.934 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 600, 2024-09-13T11:22:24.934 INFO:teuthology.orchestra.run.smithi001.stdout: "ops": [] 2024-09-13T11:22:24.935 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-13T11:22:24.951 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-09-13T11:22:24.980 INFO:tasks.ceph.osd.3.smithi177.stderr:2024-09-13T11:22:24.977+0000 7f10adba4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26574) UID: 0 2024-09-13T11:22:25.081 INFO:tasks.ceph.osd.3.smithi177.stderr:2024-09-13T11:22:25.077+0000 7f10adba4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26574) UID: 0 2024-09-13T11:22:25.114 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-13T11:22:25.114 INFO:teuthology.orchestra.run.smithi001.stdout: "ops": [], 2024-09-13T11:22:25.114 INFO:teuthology.orchestra.run.smithi001.stdout: "num_ops": 0 2024-09-13T11:22:25.114 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-13T11:22:25.129 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-09-13T11:22:25.182 INFO:tasks.ceph.osd.3.smithi177.stderr:2024-09-13T11:22:25.181+0000 7f10adba4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26574) UID: 0 2024-09-13T11:22:25.246 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-13T11:22:25.246 INFO:teuthology.orchestra.run.smithi001.stdout: "ops": [], 2024-09-13T11:22:25.246 INFO:teuthology.orchestra.run.smithi001.stdout: "complaint_time": 30, 2024-09-13T11:22:25.246 INFO:teuthology.orchestra.run.smithi001.stdout: "num_blocked_ops": 0 2024-09-13T11:22:25.246 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-13T11:22:25.260 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-09-13T11:22:25.263 INFO:teuthology.orchestra.run.smithi092.stderr:Exporting 2.4 info 2.4( empty local-lis/les=25/26 n=0 ec=25/25 lis/c=25/25 les/c/f=26/26/0 sis=25) 2024-09-13T11:22:25.263 INFO:teuthology.orchestra.run.smithi092.stderr:Export successful 2024-09-13T11:22:25.283 INFO:tasks.ceph.osd.3.smithi177.stderr:2024-09-13T11:22:25.281+0000 7f10adba4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26574) UID: 0 2024-09-13T11:22:25.379 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-13T11:22:25.379 INFO:teuthology.orchestra.run.smithi001.stdout: "size": 20, 2024-09-13T11:22:25.379 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 600, 2024-09-13T11:22:25.380 INFO:teuthology.orchestra.run.smithi001.stdout: "ops": [] 2024-09-13T11:22:25.380 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-13T11:22:25.384 INFO:tasks.ceph.osd.15.smithi177.stderr:2024-09-13T11:22:25.381+0000 7f9d6ce77640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26578) UID: 0 2024-09-13T11:22:25.394 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_ops_in_flight 2024-09-13T11:22:25.485 INFO:tasks.ceph.osd.8.smithi001.stderr:2024-09-13T11:22:25.481+0000 7fb9b48ae640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26312) UID: 0 2024-09-13T11:22:25.524 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-13T11:22:25.524 INFO:teuthology.orchestra.run.smithi001.stdout: "ops": [], 2024-09-13T11:22:25.524 INFO:teuthology.orchestra.run.smithi001.stdout: "num_ops": 0 2024-09-13T11:22:25.524 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-13T11:22:25.538 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_blocked_ops 2024-09-13T11:22:25.569 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-1 --journal-path /var/lib/ceph/osd/ceph-1/journal --force --op remove --pgid 2.4 2024-09-13T11:22:25.586 INFO:tasks.ceph.osd.14.smithi162.stderr:2024-09-13T11:22:25.584+0000 7f235c0db640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26319) UID: 0 2024-09-13T11:22:25.687 INFO:tasks.ceph.osd.5.smithi092.stderr:2024-09-13T11:22:25.688+0000 7f6f0b705640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26346) UID: 0 2024-09-13T11:22:25.709 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-13T11:22:25.710 INFO:teuthology.orchestra.run.smithi001.stdout: "ops": [], 2024-09-13T11:22:25.710 INFO:teuthology.orchestra.run.smithi001.stdout: "complaint_time": 30, 2024-09-13T11:22:25.710 INFO:teuthology.orchestra.run.smithi001.stdout: "num_blocked_ops": 0 2024-09-13T11:22:25.710 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-13T11:22:25.725 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_historic_ops 2024-09-13T11:22:25.789 INFO:tasks.ceph.osd.3.smithi177.stderr:2024-09-13T11:22:25.785+0000 7f10adba4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26574) UID: 0 2024-09-13T11:22:25.845 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-13T11:22:25.845 INFO:teuthology.orchestra.run.smithi001.stdout: "size": 20, 2024-09-13T11:22:25.845 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 600, 2024-09-13T11:22:25.845 INFO:teuthology.orchestra.run.smithi001.stdout: "ops": [] 2024-09-13T11:22:25.845 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-13T11:22:25.860 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-13T11:22:25.890 INFO:tasks.ceph.osd.3.smithi177.stderr:2024-09-13T11:22:25.889+0000 7f10adba4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26574) UID: 0 2024-09-13T11:22:25.909 INFO:teuthology.orchestra.run.smithi001.stderr:enabled application 'rados' on pool 'unique_pool_0' 2024-09-13T11:22:25.911 INFO:tasks.ceph.osd.2.smithi162.stderr:2024-09-13T11:22:25.908+0000 7fe921d8b640 -1 osd.2 30 handle_osd_map injecting map crc failure 2024-09-13T11:22:25.914 INFO:tasks.ceph.osd.13.smithi092.stderr:2024-09-13T11:22:25.912+0000 7f3e4da13640 -1 osd.13 30 handle_osd_map injecting map crc failure 2024-09-13T11:22:25.915 INFO:tasks.ceph.osd.9.smithi092.stderr:2024-09-13T11:22:25.912+0000 7ff59ddf9640 -1 osd.9 30 handle_osd_map injecting map crc failure 2024-09-13T11:22:25.989 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-09-13T11:22:25.989 INFO:teuthology.orchestra.run.smithi092.stdout: "ops": [], 2024-09-13T11:22:25.990 INFO:teuthology.orchestra.run.smithi092.stdout: "num_ops": 0 2024-09-13T11:22:25.990 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-09-13T11:22:25.990 INFO:tasks.ceph.osd.4.smithi001.stderr:2024-09-13T11:22:25.985+0000 7ff8e0218640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26310) UID: 0 2024-09-13T11:22:26.003 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-13T11:22:26.091 INFO:tasks.ceph.osd.10.smithi162.stderr:2024-09-13T11:22:26.088+0000 7fb59c9f0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26317) UID: 0 2024-09-13T11:22:26.119 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-09-13T11:22:26.119 INFO:teuthology.orchestra.run.smithi092.stdout: "ops": [], 2024-09-13T11:22:26.119 INFO:teuthology.orchestra.run.smithi092.stdout: "complaint_time": 30, 2024-09-13T11:22:26.119 INFO:teuthology.orchestra.run.smithi092.stdout: "num_blocked_ops": 0 2024-09-13T11:22:26.119 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-09-13T11:22:26.133 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-13T11:22:26.191 INFO:tasks.ceph.osd.5.smithi092.stderr:2024-09-13T11:22:26.188+0000 7f6f0b705640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26346) UID: 0 2024-09-13T11:22:26.255 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-09-13T11:22:26.255 INFO:teuthology.orchestra.run.smithi092.stdout: "size": 20, 2024-09-13T11:22:26.255 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 600, 2024-09-13T11:22:26.255 INFO:teuthology.orchestra.run.smithi092.stdout: "ops": [] 2024-09-13T11:22:26.255 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-09-13T11:22:26.270 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-09-13T11:22:26.316 INFO:tasks.ceph.osd.13.smithi092.stderr:2024-09-13T11:22:26.316+0000 7f3e5afea640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26350) UID: 0 2024-09-13T11:22:26.395 INFO:tasks.ceph.osd.12.smithi001.stderr:2024-09-13T11:22:26.389+0000 7f892c6e6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26314) UID: 0 2024-09-13T11:22:26.443 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-09-13T11:22:26.443 INFO:teuthology.orchestra.run.smithi092.stdout: "ops": [], 2024-09-13T11:22:26.443 INFO:teuthology.orchestra.run.smithi092.stdout: "num_ops": 0 2024-09-13T11:22:26.443 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-09-13T11:22:26.462 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-09-13T11:22:26.496 INFO:tasks.ceph.osd.3.smithi177.stderr:2024-09-13T11:22:26.493+0000 7f10adba4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26574) UID: 0 2024-09-13T11:22:26.580 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-09-13T11:22:26.580 INFO:teuthology.orchestra.run.smithi092.stdout: "ops": [], 2024-09-13T11:22:26.580 INFO:teuthology.orchestra.run.smithi092.stdout: "complaint_time": 30, 2024-09-13T11:22:26.580 INFO:teuthology.orchestra.run.smithi092.stdout: "num_blocked_ops": 0 2024-09-13T11:22:26.580 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-09-13T11:22:26.594 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-09-13T11:22:26.597 INFO:tasks.ceph.osd.5.smithi092.stderr:2024-09-13T11:22:26.596+0000 7f6f0b705640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26346) UID: 0 2024-09-13T11:22:26.698 INFO:tasks.ceph.osd.3.smithi177.stderr:2024-09-13T11:22:26.697+0000 7f10adba4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26574) UID: 0 2024-09-13T11:22:26.710 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-09-13T11:22:26.710 INFO:teuthology.orchestra.run.smithi092.stdout: "size": 20, 2024-09-13T11:22:26.710 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 600, 2024-09-13T11:22:26.710 INFO:teuthology.orchestra.run.smithi092.stdout: "ops": [ 2024-09-13T11:22:26.710 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:26.710 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop(client.4344.0:5 2.2 e27/25 2:5cea7035:::rbd_info:head v 27'1, mlcod=0'0)", 2024-09-13T11:22:26.711 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:15.250889+0000", 2024-09-13T11:22:26.711 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 11.460823320999999, 2024-09-13T11:22:26.711 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.001990627, 2024-09-13T11:22:26.711 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:22:26.711 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:22:26.711 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:26.711 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:22:26.711 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:26.711 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:22:26.711 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:15.250889+0000", 2024-09-13T11:22:26.711 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:22:26.711 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:26.711 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:26.712 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:22:26.712 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:15.250889+0000", 2024-09-13T11:22:26.712 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:22:26.712 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:26.712 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:26.712 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:22:26.712 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:15.250890+0000", 2024-09-13T11:22:26.712 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.153e-06 2024-09-13T11:22:26.712 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:26.712 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:26.712 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:22:26.712 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:15.250907+0000", 2024-09-13T11:22:26.712 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.6821e-05 2024-09-13T11:22:26.713 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:26.713 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:26.713 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:22:26.713 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:15.250991+0000", 2024-09-13T11:22:26.713 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.4241999999999997e-05 2024-09-13T11:22:26.713 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:26.713 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:26.713 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:22:26.713 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:15.251023+0000", 2024-09-13T11:22:26.713 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.1780999999999999e-05 2024-09-13T11:22:26.713 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:26.713 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:26.714 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:22:26.714 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:15.251178+0000", 2024-09-13T11:22:26.714 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000155064 2024-09-13T11:22:26.714 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:26.714 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:26.714 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:22:26.714 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:15.251215+0000", 2024-09-13T11:22:26.714 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.7160000000000003e-05 2024-09-13T11:22:26.714 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:26.715 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:26.715 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:22:26.715 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:15.252759+0000", 2024-09-13T11:22:26.715 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0015440110000000001 2024-09-13T11:22:26.715 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:26.715 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:26.715 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:22:26.715 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:15.252880+0000", 2024-09-13T11:22:26.715 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000120395 2024-09-13T11:22:26.715 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:22:26.715 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:22:26.715 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:22:26.715 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:26.715 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:26.716 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop(client.4344.0:7 2.2 e28/25 2:5cea7035:::rbd_info:head v 28'2, mlcod=0'0)", 2024-09-13T11:22:26.716 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:16.268518+0000", 2024-09-13T11:22:26.716 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 10.443193721, 2024-09-13T11:22:26.716 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0017167040000000001, 2024-09-13T11:22:26.716 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:22:26.716 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:22:26.716 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:26.716 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:22:26.716 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:26.716 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:22:26.716 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:16.268518+0000", 2024-09-13T11:22:26.716 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:22:26.717 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:26.717 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:26.717 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:22:26.717 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:16.268518+0000", 2024-09-13T11:22:26.717 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:22:26.717 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:26.717 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:26.717 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:22:26.717 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:16.268519+0000", 2024-09-13T11:22:26.717 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.1299999999999999e-07 2024-09-13T11:22:26.717 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:26.717 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:26.717 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:22:26.718 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:16.268539+0000", 2024-09-13T11:22:26.718 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.0214999999999999e-05 2024-09-13T11:22:26.718 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:26.718 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:26.718 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:22:26.718 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:16.268622+0000", 2024-09-13T11:22:26.718 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.2395e-05 2024-09-13T11:22:26.718 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:26.718 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:26.718 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:22:26.718 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:16.268638+0000", 2024-09-13T11:22:26.718 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.6189e-05 2024-09-13T11:22:26.718 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:26.719 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:26.719 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:22:26.719 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:16.268740+0000", 2024-09-13T11:22:26.719 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00010164 2024-09-13T11:22:26.719 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:26.719 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:26.719 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:22:26.719 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:16.268783+0000", 2024-09-13T11:22:26.719 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.2988999999999998e-05 2024-09-13T11:22:26.719 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:26.719 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:26.719 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:22:26.720 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:16.270121+0000", 2024-09-13T11:22:26.720 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00133829 2024-09-13T11:22:26.720 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:26.720 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:26.720 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:22:26.720 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:16.270235+0000", 2024-09-13T11:22:26.720 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000114273 2024-09-13T11:22:26.720 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:22:26.720 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:22:26.720 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:22:26.720 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:22:26.720 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:22:26.720 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-09-13T11:22:26.724 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_ops_in_flight 2024-09-13T11:22:26.799 INFO:tasks.ceph.osd.10.smithi162.stderr:2024-09-13T11:22:26.796+0000 7fb59c9f0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26317) UID: 0 2024-09-13T11:22:26.846 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-09-13T11:22:26.846 INFO:teuthology.orchestra.run.smithi092.stdout: "ops": [], 2024-09-13T11:22:26.846 INFO:teuthology.orchestra.run.smithi092.stdout: "num_ops": 0 2024-09-13T11:22:26.846 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-09-13T11:22:26.860 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_blocked_ops 2024-09-13T11:22:26.903 INFO:tasks.ceph.osd.9.smithi092.stderr:2024-09-13T11:22:26.904+0000 7ff5ab3d0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26348) UID: 0 2024-09-13T11:22:26.913 INFO:tasks.ceph.osd.13.smithi092.stderr:2024-09-13T11:22:26.912+0000 7f3e4da13640 -1 osd.13 31 handle_osd_map injecting map crc failure 2024-09-13T11:22:26.913 INFO:tasks.ceph.osd.10.smithi162.stderr:2024-09-13T11:22:26.912+0000 7fb593421640 -1 osd.10 31 handle_osd_map injecting map crc failure 2024-09-13T11:22:26.936 DEBUG:teuthology.orchestra.run.smithi177:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.0 -b 65536 --object-size 65536 -p unique_pool_0 bench 150 write' 2024-09-13T11:22:26.938 INFO:teuthology.run_tasks:Running task admin_socket... 2024-09-13T11:22:26.951 DEBUG:tasks.admin_socket:Running admin socket tests on client.0 2024-09-13T11:22:26.951 DEBUG:teuthology.orchestra.run.smithi177:> mkdir -- /home/ubuntu/cephtest/admin_socket_client.0 && while test '!' -e /var/run/ceph/ceph-client.0.asok ; do sleep 1 ; done 2024-09-13T11:22:26.951 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-13T11:22:26.986 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-09-13T11:22:26.986 INFO:teuthology.orchestra.run.smithi092.stdout: "ops": [], 2024-09-13T11:22:26.986 INFO:teuthology.orchestra.run.smithi092.stdout: "complaint_time": 30, 2024-09-13T11:22:26.986 INFO:teuthology.orchestra.run.smithi092.stdout: "num_blocked_ops": 0 2024-09-13T11:22:26.986 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-09-13T11:22:26.994 INFO:tasks.radosbench.radosbench.0.smithi177.stdout:hints = 1 2024-09-13T11:22:26.994 INFO:tasks.radosbench.radosbench.0.smithi177.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 150 seconds or 0 objects 2024-09-13T11:22:26.994 INFO:tasks.radosbench.radosbench.0.smithi177.stdout:Object prefix: benchmark_data_smithi177_31116 2024-09-13T11:22:26.995 INFO:tasks.radosbench.radosbench.0.smithi177.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-13T11:22:26.995 INFO:tasks.radosbench.radosbench.0.smithi177.stdout: 0 0 0 0 0 0 - 0 2024-09-13T11:22:27.000 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_historic_ops 2024-09-13T11:22:27.001 INFO:tasks.ceph.osd.2.smithi162.stderr:2024-09-13T11:22:27.000+0000 7fe92b35a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26313) UID: 0 2024-09-13T11:22:27.102 INFO:tasks.ceph.osd.9.smithi092.stderr:2024-09-13T11:22:27.100+0000 7ff5ab3d0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26348) UID: 0 2024-09-13T11:22:27.170 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-09-13T11:22:27.170 INFO:teuthology.orchestra.run.smithi092.stdout: "size": 20, 2024-09-13T11:22:27.170 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 600, 2024-09-13T11:22:27.170 INFO:teuthology.orchestra.run.smithi092.stdout: "ops": [] 2024-09-13T11:22:27.170 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-09-13T11:22:27.186 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-13T11:22:27.202 INFO:tasks.ceph.osd.4.smithi001.stderr:2024-09-13T11:22:27.197+0000 7ff8e0218640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26310) UID: 0 2024-09-13T11:22:27.232 INFO:teuthology.orchestra.run.smithi092.stdout: marking collection for removal 2024-09-13T11:22:27.232 INFO:teuthology.orchestra.run.smithi092.stdout:setting '_remove' omap key 2024-09-13T11:22:27.237 INFO:teuthology.orchestra.run.smithi092.stdout:finish_remove_pgs 2.4_head removing 2.4 2024-09-13T11:22:27.238 INFO:teuthology.orchestra.run.smithi092.stdout:Remove successful 2024-09-13T11:22:27.303 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-09-13T11:22:27.300+0000 7f97befc9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26315) UID: 0 2024-09-13T11:22:27.319 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-13T11:22:27.319 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [], 2024-09-13T11:22:27.319 INFO:teuthology.orchestra.run.smithi162.stdout: "num_ops": 0 2024-09-13T11:22:27.319 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-13T11:22:27.333 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-13T11:22:27.405 INFO:tasks.ceph.osd.8.smithi001.stderr:2024-09-13T11:22:27.401+0000 7fb9b48ae640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26312) UID: 0 2024-09-13T11:22:27.499 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-13T11:22:27.500 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [], 2024-09-13T11:22:27.500 INFO:teuthology.orchestra.run.smithi162.stdout: "complaint_time": 30, 2024-09-13T11:22:27.500 INFO:teuthology.orchestra.run.smithi162.stdout: "num_blocked_ops": 0 2024-09-13T11:22:27.500 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-13T11:22:27.506 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-09-13T11:22:27.504+0000 7f97befc9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26315) UID: 0 2024-09-13T11:22:27.513 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-13T11:22:27.606 INFO:tasks.ceph.osd.2.smithi162.stderr:2024-09-13T11:22:27.604+0000 7fe92b35a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26313) UID: 0 2024-09-13T11:22:27.682 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-13T11:22:27.682 INFO:teuthology.orchestra.run.smithi162.stdout: "size": 20, 2024-09-13T11:22:27.682 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 600, 2024-09-13T11:22:27.682 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [ 2024-09-13T11:22:27.683 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:27.683 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.4665.0:12 3.9 e32/30 3:9dad78b7:::benchmark_data_smithi177_31116_object11:head v 32'1, mlcod=0'0)", 2024-09-13T11:22:27.683 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:22:27.004720+0000", 2024-09-13T11:22:27.683 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 0.67748781700000005, 2024-09-13T11:22:27.683 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.004117791, 2024-09-13T11:22:27.684 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:22:27.684 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:22:27.684 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:27.684 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:22:27.684 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:27.684 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:22:27.685 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.004720+0000", 2024-09-13T11:22:27.685 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:22:27.685 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:27.685 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:27.685 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:22:27.686 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.004720+0000", 2024-09-13T11:22:27.686 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:22:27.686 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:27.686 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:27.686 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:22:27.686 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.004722+0000", 2024-09-13T11:22:27.687 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.483e-06 2024-09-13T11:22:27.687 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:27.687 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:27.687 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:22:27.687 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.004999+0000", 2024-09-13T11:22:27.687 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00027699700000000002 2024-09-13T11:22:27.688 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:27.688 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:27.688 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:22:27.689 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.005067+0000", 2024-09-13T11:22:27.689 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.8162000000000002e-05 2024-09-13T11:22:27.689 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:27.689 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:27.690 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:22:27.690 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.005104+0000", 2024-09-13T11:22:27.690 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.6698e-05 2024-09-13T11:22:27.690 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:27.690 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:27.690 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:22:27.690 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.005253+0000", 2024-09-13T11:22:27.691 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000149111 2024-09-13T11:22:27.691 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:27.691 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:27.692 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:22:27.692 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.005326+0000", 2024-09-13T11:22:27.692 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.2948999999999996e-05 2024-09-13T11:22:27.692 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:27.692 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:27.692 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:22:27.692 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.008734+0000", 2024-09-13T11:22:27.692 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0034086300000000002 2024-09-13T11:22:27.692 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:27.692 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:27.692 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:22:27.692 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.008838+0000", 2024-09-13T11:22:27.693 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000103761 2024-09-13T11:22:27.693 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:22:27.693 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:22:27.693 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:22:27.693 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:27.693 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:27.693 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.4665.0:13 3.9 e32/30 3:967a7538:::benchmark_data_smithi177_31116_object12:head v 32'2, mlcod=0'0)", 2024-09-13T11:22:27.693 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:22:27.006514+0000", 2024-09-13T11:22:27.693 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 0.67569433800000001, 2024-09-13T11:22:27.693 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0070363759999999996, 2024-09-13T11:22:27.694 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:22:27.694 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:22:27.694 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:27.694 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:22:27.694 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:27.694 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:22:27.694 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.006514+0000", 2024-09-13T11:22:27.694 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:22:27.694 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:27.694 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:27.694 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:22:27.694 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.006514+0000", 2024-09-13T11:22:27.694 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:22:27.695 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:27.695 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:27.695 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:22:27.695 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.006517+0000", 2024-09-13T11:22:27.695 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.3900000000000002e-06 2024-09-13T11:22:27.695 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:27.695 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:27.695 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:22:27.695 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.006716+0000", 2024-09-13T11:22:27.695 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000198612 2024-09-13T11:22:27.695 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:27.695 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:27.695 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:22:27.695 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.006803+0000", 2024-09-13T11:22:27.696 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.7544000000000001e-05 2024-09-13T11:22:27.696 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:27.696 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:27.696 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:22:27.696 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.006817+0000", 2024-09-13T11:22:27.696 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4175999999999999e-05 2024-09-13T11:22:27.696 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:27.696 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:27.696 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:22:27.696 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.006947+0000", 2024-09-13T11:22:27.696 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000129209 2024-09-13T11:22:27.696 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:27.696 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:27.696 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:22:27.697 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.007002+0000", 2024-09-13T11:22:27.697 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.5476999999999998e-05 2024-09-13T11:22:27.697 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:27.697 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:27.697 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:22:27.697 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.013448+0000", 2024-09-13T11:22:27.697 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0064460109999999998 2024-09-13T11:22:27.697 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:27.697 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:27.697 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:22:27.697 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.013550+0000", 2024-09-13T11:22:27.697 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000101957 2024-09-13T11:22:27.697 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:22:27.697 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:22:27.698 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:22:27.698 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:27.698 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:27.698 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.4665.0:37 3.9 e32/30 3:962d44ea:::benchmark_data_smithi177_31116_object36:head v 32'3, mlcod=32'1)", 2024-09-13T11:22:27.698 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:22:27.016683+0000", 2024-09-13T11:22:27.698 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 0.66552520500000001, 2024-09-13T11:22:27.698 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0028898940000000001, 2024-09-13T11:22:27.698 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:22:27.698 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:22:27.698 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:27.698 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:22:27.698 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:27.698 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:22:27.698 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.016683+0000", 2024-09-13T11:22:27.699 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:22:27.699 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:27.699 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:27.699 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:22:27.699 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.016683+0000", 2024-09-13T11:22:27.699 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:22:27.699 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:27.699 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:27.699 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:22:27.699 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.016686+0000", 2024-09-13T11:22:27.699 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.2660000000000002e-06 2024-09-13T11:22:27.699 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:27.699 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:27.699 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:22:27.699 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.016767+0000", 2024-09-13T11:22:27.700 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.0328999999999996e-05 2024-09-13T11:22:27.700 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:27.700 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:27.700 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:22:27.700 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.016844+0000", 2024-09-13T11:22:27.700 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.7222e-05 2024-09-13T11:22:27.700 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:27.700 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:27.700 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:22:27.700 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.016858+0000", 2024-09-13T11:22:27.700 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3876000000000001e-05 2024-09-13T11:22:27.700 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:27.700 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:27.701 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:22:27.701 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.016962+0000", 2024-09-13T11:22:27.701 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00010483400000000001 2024-09-13T11:22:27.701 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:27.701 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:27.701 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:22:27.701 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.017002+0000", 2024-09-13T11:22:27.701 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.9467e-05 2024-09-13T11:22:27.701 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:27.701 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:27.701 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:22:27.701 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.019496+0000", 2024-09-13T11:22:27.701 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0024940510000000002 2024-09-13T11:22:27.701 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:27.701 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:27.702 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:22:27.702 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.019573+0000", 2024-09-13T11:22:27.702 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.6848999999999996e-05 2024-09-13T11:22:27.702 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:22:27.702 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:22:27.702 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:22:27.702 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:27.702 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:27.702 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.4665.0:59 3.9 e32/30 3:9499e3d6:::benchmark_data_smithi177_31116_object58:head v 32'4, mlcod=32'2)", 2024-09-13T11:22:27.702 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:22:27.028489+0000", 2024-09-13T11:22:27.702 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 0.65371954700000001, 2024-09-13T11:22:27.702 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0015734430000000001, 2024-09-13T11:22:27.702 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:22:27.702 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:22:27.703 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:27.703 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:22:27.703 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:27.703 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:22:27.703 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.028489+0000", 2024-09-13T11:22:27.703 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:22:27.703 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:27.703 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:27.703 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:22:27.703 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.028489+0000", 2024-09-13T11:22:27.703 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:22:27.703 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:27.703 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:27.704 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:22:27.704 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.028491+0000", 2024-09-13T11:22:27.704 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.9170000000000002e-06 2024-09-13T11:22:27.704 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:27.704 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:27.704 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:22:27.704 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.028547+0000", 2024-09-13T11:22:27.704 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.5832999999999999e-05 2024-09-13T11:22:27.704 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:27.704 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:27.704 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:22:27.704 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.028618+0000", 2024-09-13T11:22:27.704 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.0741999999999995e-05 2024-09-13T11:22:27.704 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:27.705 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:27.705 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:22:27.705 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.028635+0000", 2024-09-13T11:22:27.705 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.6773e-05 2024-09-13T11:22:27.705 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:27.705 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:27.705 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:22:27.705 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.028758+0000", 2024-09-13T11:22:27.705 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00012363200000000001 2024-09-13T11:22:27.705 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:27.706 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:27.706 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:22:27.706 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.028795+0000", 2024-09-13T11:22:27.706 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.6865e-05 2024-09-13T11:22:27.706 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:27.706 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:27.706 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:22:27.706 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.029988+0000", 2024-09-13T11:22:27.706 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001192505 2024-09-13T11:22:27.706 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:27.706 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:27.706 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:22:27.706 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.030062+0000", 2024-09-13T11:22:27.706 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.4176000000000003e-05 2024-09-13T11:22:27.707 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:22:27.707 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:22:27.707 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:22:27.707 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:27.707 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:27.707 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.4665.0:76 3.9 e32/30 3:9e00a8c6:::benchmark_data_smithi177_31116_object75:head v 32'5, mlcod=32'3)", 2024-09-13T11:22:27.707 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:22:27.044735+0000", 2024-09-13T11:22:27.707 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 0.6374727, 2024-09-13T11:22:27.707 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0018264010000000001, 2024-09-13T11:22:27.707 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:22:27.707 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:22:27.707 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:27.707 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:22:27.707 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:27.708 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:22:27.708 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.044735+0000", 2024-09-13T11:22:27.708 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:22:27.708 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:27.708 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:27.708 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:22:27.708 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.044735+0000", 2024-09-13T11:22:27.708 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:22:27.708 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:27.708 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:27.708 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:22:27.708 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.044739+0000", 2024-09-13T11:22:27.708 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.9389999999999999e-06 2024-09-13T11:22:27.708 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:27.709 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:27.709 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:22:27.709 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.044860+0000", 2024-09-13T11:22:27.709 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00012027300000000001 2024-09-13T11:22:27.709 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:27.709 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:27.709 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:22:27.709 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.044931+0000", 2024-09-13T11:22:27.709 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.1575999999999994e-05 2024-09-13T11:22:27.709 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:27.709 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:27.709 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:22:27.709 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.044946+0000", 2024-09-13T11:22:27.709 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4851999999999999e-05 2024-09-13T11:22:27.710 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:27.710 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:27.710 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:22:27.710 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.045050+0000", 2024-09-13T11:22:27.710 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000104264 2024-09-13T11:22:27.710 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:27.710 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:27.710 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:22:27.710 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.045088+0000", 2024-09-13T11:22:27.710 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.7904999999999997e-05 2024-09-13T11:22:27.710 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:27.710 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:27.710 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:22:27.710 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.046490+0000", 2024-09-13T11:22:27.710 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001401619 2024-09-13T11:22:27.711 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:27.711 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:27.711 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:22:27.711 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.046562+0000", 2024-09-13T11:22:27.711 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.1972999999999997e-05 2024-09-13T11:22:27.711 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:22:27.711 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:22:27.712 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:22:27.712 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:22:27.712 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:22:27.712 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-13T11:22:27.714 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-13T11:22:27.715 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-09-13T11:22:27.712+0000 7f97befc9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26315) UID: 0 2024-09-13T11:22:27.772 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-objectstore-tool --err-to-stderr --data-path /var/lib/ceph/osd/ceph-1 --journal-path /var/lib/ceph/osd/ceph-1/journal '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --op import --file /var/log/ceph/exp.2.4.1 2024-09-13T11:22:27.815 INFO:tasks.ceph.osd.7.smithi177.stderr:2024-09-13T11:22:27.817+0000 7f897b51d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26575) UID: 0 2024-09-13T11:22:27.882 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-13T11:22:27.882 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [], 2024-09-13T11:22:27.882 INFO:teuthology.orchestra.run.smithi162.stdout: "num_ops": 0 2024-09-13T11:22:27.883 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-13T11:22:27.896 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-13T11:22:27.929 INFO:teuthology.orchestra.run.smithi001.stderr:nodeep-scrub is set 2024-09-13T11:22:27.941 INFO:tasks.ceph.osd.2.smithi162.stderr:2024-09-13T11:22:27.940+0000 7fe92b35a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26313) UID: 0 2024-09-13T11:22:27.960 DEBUG:tasks.admin_socket:Testing objecter_requests with config {'test': 'http://git.ceph.com/?p={repo};a=blob_plain;f=src/test/admin_socket/objecter_requests;hb={branch}', 'branch': 'wip-aclamk-cbt-combined'} 2024-09-13T11:22:27.960 DEBUG:teuthology.orchestra.run.smithi177:> wget -q -O /home/ubuntu/cephtest/admin_socket_client.0/objecter_requests -- 'http://git.ceph.com/?p=ceph-ci.git;a=blob_plain;f=src/test/admin_socket/objecter_requests;hb=wip-aclamk-cbt-combined' && chmod u=rx -- /home/ubuntu/cephtest/admin_socket_client.0/objecter_requests 2024-09-13T11:22:27.995 INFO:tasks.radosbench.radosbench.0.smithi177.stdout: 1 16 85 69 4.31207 4.3125 0.00997965 0.0188784 2024-09-13T11:22:28.016 INFO:tasks.ceph.osd.14.smithi162.stderr:2024-09-13T11:22:28.016+0000 7f235c0db640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26319) UID: 0 2024-09-13T11:22:28.021 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-13T11:22:28.021 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [], 2024-09-13T11:22:28.021 INFO:teuthology.orchestra.run.smithi162.stdout: "complaint_time": 30, 2024-09-13T11:22:28.021 INFO:teuthology.orchestra.run.smithi162.stdout: "num_blocked_ops": 0 2024-09-13T11:22:28.021 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-13T11:22:28.035 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-13T11:22:28.121 INFO:tasks.ceph.osd.10.smithi162.stderr:2024-09-13T11:22:28.120+0000 7fb59c9f0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26317) UID: 0 2024-09-13T11:22:28.130 INFO:tasks.ceph.osd.10.smithi162.stderr:2024-09-13T11:22:28.128+0000 7fb59141d640 -1 osd.10 32 handle_osd_map injecting map crc failure 2024-09-13T11:22:28.206 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-13T11:22:28.206 INFO:teuthology.orchestra.run.smithi162.stdout: "size": 20, 2024-09-13T11:22:28.206 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 600, 2024-09-13T11:22:28.206 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [ 2024-09-13T11:22:28.207 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.207 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4257.0:45 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-13T11:22:28.207 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:22:12.319311+0000", 2024-09-13T11:22:28.207 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 15.883284262, 2024-09-13T11:22:28.207 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.029681038, 2024-09-13T11:22:28.207 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:22:28.207 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:22:28.207 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:28.207 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:22:28.207 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4257", 2024-09-13T11:22:28.207 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/1488131527", 2024-09-13T11:22:28.207 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 45 2024-09-13T11:22:28.208 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.208 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:22:28.208 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.208 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:22:28.208 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.319311+0000", 2024-09-13T11:22:28.208 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:22:28.208 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.208 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.208 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:22:28.208 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.319311+0000", 2024-09-13T11:22:28.208 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:22:28.208 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.209 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.209 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:22:28.209 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.319313+0000", 2024-09-13T11:22:28.209 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.2859999999999998e-06 2024-09-13T11:22:28.209 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.209 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.209 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:22:28.209 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.319322+0000", 2024-09-13T11:22:28.209 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.4360000000000002e-06 2024-09-13T11:22:28.209 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.209 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.209 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:22:28.209 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.319338+0000", 2024-09-13T11:22:28.210 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.6158999999999999e-05 2024-09-13T11:22:28.210 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.210 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.210 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:22:28.210 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.319352+0000", 2024-09-13T11:22:28.210 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3774999999999999e-05 2024-09-13T11:22:28.210 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.210 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.210 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:22:28.210 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.340424+0000", 2024-09-13T11:22:28.210 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.021072312999999999 2024-09-13T11:22:28.211 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.211 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.211 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:22:28.211 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.340636+0000", 2024-09-13T11:22:28.211 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00021209399999999999 2024-09-13T11:22:28.211 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.211 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.211 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 0", 2024-09-13T11:22:28.211 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.340826+0000", 2024-09-13T11:22:28.211 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00018942199999999999 2024-09-13T11:22:28.211 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.211 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.212 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:22:28.212 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.348717+0000", 2024-09-13T11:22:28.212 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0078914250000000005 2024-09-13T11:22:28.212 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.212 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.212 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:22:28.212 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.348755+0000", 2024-09-13T11:22:28.212 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.8290000000000001e-05 2024-09-13T11:22:28.212 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.212 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.212 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:22:28.212 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.348877+0000", 2024-09-13T11:22:28.212 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00012210500000000001 2024-09-13T11:22:28.213 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.213 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.213 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:22:28.213 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.348992+0000", 2024-09-13T11:22:28.213 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000114733 2024-09-13T11:22:28.213 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:22:28.213 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:22:28.213 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:22:28.213 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.213 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.213 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4257.0:46 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-13T11:22:28.213 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:22:12.319370+0000", 2024-09-13T11:22:28.213 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 15.883225634, 2024-09-13T11:22:28.214 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.029891220999999999, 2024-09-13T11:22:28.214 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:22:28.214 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:22:28.214 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:28.214 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:22:28.214 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4257", 2024-09-13T11:22:28.214 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/1488131527", 2024-09-13T11:22:28.214 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 46 2024-09-13T11:22:28.214 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.214 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:22:28.214 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.214 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:22:28.215 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.319370+0000", 2024-09-13T11:22:28.215 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:22:28.215 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.215 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.215 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:22:28.215 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.319370+0000", 2024-09-13T11:22:28.217 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:22:28.217 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.217 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.217 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:22:28.217 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.319372+0000", 2024-09-13T11:22:28.217 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.81e-06 2024-09-13T11:22:28.217 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.217 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.217 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:22:28.218 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.319379+0000", 2024-09-13T11:22:28.218 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.5619999999999998e-06 2024-09-13T11:22:28.218 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.218 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.218 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:22:28.218 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.319395+0000", 2024-09-13T11:22:28.218 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.6262000000000001e-05 2024-09-13T11:22:28.218 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.218 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.218 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:22:28.218 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.319408+0000", 2024-09-13T11:22:28.218 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3069e-05 2024-09-13T11:22:28.218 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.219 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.219 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:22:28.219 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.341763+0000", 2024-09-13T11:22:28.219 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.022354707000000001 2024-09-13T11:22:28.219 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.219 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.219 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:22:28.219 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.341899+0000", 2024-09-13T11:22:28.219 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000135504 2024-09-13T11:22:28.219 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.219 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.219 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 0", 2024-09-13T11:22:28.219 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.342147+0000", 2024-09-13T11:22:28.220 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00024821700000000001 2024-09-13T11:22:28.220 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.220 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.220 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:22:28.220 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.349019+0000", 2024-09-13T11:22:28.220 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0068718529999999998 2024-09-13T11:22:28.220 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.220 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.220 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:22:28.220 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.349086+0000", 2024-09-13T11:22:28.220 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.7026999999999994e-05 2024-09-13T11:22:28.220 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.220 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.220 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:22:28.221 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.349164+0000", 2024-09-13T11:22:28.221 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.8479000000000005e-05 2024-09-13T11:22:28.221 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.221 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.221 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:22:28.221 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.349261+0000", 2024-09-13T11:22:28.221 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.6731000000000006e-05 2024-09-13T11:22:28.221 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:22:28.221 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:22:28.221 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:22:28.221 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.221 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.221 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4257.0:47 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-13T11:22:28.222 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:22:12.319435+0000", 2024-09-13T11:22:28.222 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 15.883160707, 2024-09-13T11:22:28.222 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.037475999000000003, 2024-09-13T11:22:28.222 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:22:28.222 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:22:28.222 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:28.222 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:22:28.222 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4257", 2024-09-13T11:22:28.222 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/1488131527", 2024-09-13T11:22:28.222 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 47 2024-09-13T11:22:28.222 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.223 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:22:28.223 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.223 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:22:28.223 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.319435+0000", 2024-09-13T11:22:28.223 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:22:28.223 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.223 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.223 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:22:28.223 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.319435+0000", 2024-09-13T11:22:28.223 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:22:28.223 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.223 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.223 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:22:28.224 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.319437+0000", 2024-09-13T11:22:28.224 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.9400000000000001e-06 2024-09-13T11:22:28.224 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.224 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.224 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:22:28.224 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.319521+0000", 2024-09-13T11:22:28.224 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.4178999999999995e-05 2024-09-13T11:22:28.224 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.224 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.224 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:22:28.224 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.319543+0000", 2024-09-13T11:22:28.224 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.1750999999999999e-05 2024-09-13T11:22:28.224 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.225 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.225 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:22:28.225 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.319561+0000", 2024-09-13T11:22:28.225 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.8518000000000001e-05 2024-09-13T11:22:28.225 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.225 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.225 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:22:28.225 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.345664+0000", 2024-09-13T11:22:28.225 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.026102914000000001 2024-09-13T11:22:28.225 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.225 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.225 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:22:28.225 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.345799+0000", 2024-09-13T11:22:28.225 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00013452000000000001 2024-09-13T11:22:28.226 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.226 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.226 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 0", 2024-09-13T11:22:28.226 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.346009+0000", 2024-09-13T11:22:28.226 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00021038799999999999 2024-09-13T11:22:28.226 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.226 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.226 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:22:28.226 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.349395+0000", 2024-09-13T11:22:28.226 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0033864759999999998 2024-09-13T11:22:28.226 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.226 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.226 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:22:28.226 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.356667+0000", 2024-09-13T11:22:28.227 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0072719159999999998 2024-09-13T11:22:28.227 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.227 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.227 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:22:28.227 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.356801+0000", 2024-09-13T11:22:28.227 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00013402999999999999 2024-09-13T11:22:28.227 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.227 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.227 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:22:28.227 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.356911+0000", 2024-09-13T11:22:28.227 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000109367 2024-09-13T11:22:28.228 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:22:28.228 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:22:28.228 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:22:28.228 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.228 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.228 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4257.0:48 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-13T11:22:28.228 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:22:12.319584+0000", 2024-09-13T11:22:28.228 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 15.883011206999999, 2024-09-13T11:22:28.228 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.037877980999999998, 2024-09-13T11:22:28.228 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:22:28.228 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:22:28.228 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:28.228 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:22:28.229 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4257", 2024-09-13T11:22:28.229 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/1488131527", 2024-09-13T11:22:28.229 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 48 2024-09-13T11:22:28.229 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.229 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:22:28.229 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.229 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:22:28.229 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.319584+0000", 2024-09-13T11:22:28.229 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:22:28.229 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.229 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.229 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:22:28.229 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.319584+0000", 2024-09-13T11:22:28.229 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:22:28.230 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.230 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.230 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:22:28.230 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.319586+0000", 2024-09-13T11:22:28.230 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.013e-06 2024-09-13T11:22:28.230 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.230 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.230 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:22:28.230 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.319595+0000", 2024-09-13T11:22:28.230 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.9060000000000008e-06 2024-09-13T11:22:28.230 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.230 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.230 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:22:28.231 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.319615+0000", 2024-09-13T11:22:28.231 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.0378e-05 2024-09-13T11:22:28.231 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.231 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.231 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:22:28.231 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.319632+0000", 2024-09-13T11:22:28.231 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.6858e-05 2024-09-13T11:22:28.231 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.231 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.231 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:22:28.231 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.349488+0000", 2024-09-13T11:22:28.231 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.029856038000000001 2024-09-13T11:22:28.231 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.232 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.232 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:22:28.232 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.349638+0000", 2024-09-13T11:22:28.232 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00014941600000000001 2024-09-13T11:22:28.232 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.232 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.232 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 0", 2024-09-13T11:22:28.232 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.349838+0000", 2024-09-13T11:22:28.232 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000199929 2024-09-13T11:22:28.232 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.232 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.232 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:22:28.232 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.357028+0000", 2024-09-13T11:22:28.233 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0071900799999999997 2024-09-13T11:22:28.233 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.233 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.233 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:22:28.233 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.357243+0000", 2024-09-13T11:22:28.233 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00021529000000000001 2024-09-13T11:22:28.233 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.233 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.233 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:22:28.233 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.357365+0000", 2024-09-13T11:22:28.233 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00012191199999999999 2024-09-13T11:22:28.233 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.233 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.234 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:22:28.234 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.357462+0000", 2024-09-13T11:22:28.234 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.7161e-05 2024-09-13T11:22:28.234 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:22:28.234 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:22:28.234 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:22:28.234 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.234 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.234 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4257.0:49 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-13T11:22:28.234 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:22:12.319663+0000", 2024-09-13T11:22:28.234 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 15.882932168, 2024-09-13T11:22:28.234 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.046281448000000003, 2024-09-13T11:22:28.234 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:22:28.234 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:22:28.235 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:28.235 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:22:28.235 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4257", 2024-09-13T11:22:28.235 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/1488131527", 2024-09-13T11:22:28.235 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 49 2024-09-13T11:22:28.235 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.235 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:22:28.235 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.235 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:22:28.235 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.319663+0000", 2024-09-13T11:22:28.235 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:22:28.235 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.235 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.235 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:22:28.236 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.319663+0000", 2024-09-13T11:22:28.236 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:22:28.236 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.236 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.236 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:22:28.236 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.319665+0000", 2024-09-13T11:22:28.236 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.0030000000000001e-06 2024-09-13T11:22:28.236 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.236 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.236 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:22:28.236 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.319675+0000", 2024-09-13T11:22:28.237 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0019e-05 2024-09-13T11:22:28.237 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.237 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.237 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:22:28.237 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.319697+0000", 2024-09-13T11:22:28.237 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.1387999999999999e-05 2024-09-13T11:22:28.237 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.237 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.237 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:22:28.237 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.319712+0000", 2024-09-13T11:22:28.237 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5068999999999999e-05 2024-09-13T11:22:28.237 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.238 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.238 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:22:28.238 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.352248+0000", 2024-09-13T11:22:28.238 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.032536460000000003 2024-09-13T11:22:28.238 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.238 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.238 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:22:28.238 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.352479+0000", 2024-09-13T11:22:28.238 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00023091499999999999 2024-09-13T11:22:28.238 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.238 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.238 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 0", 2024-09-13T11:22:28.239 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.352882+0000", 2024-09-13T11:22:28.239 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00040249300000000002 2024-09-13T11:22:28.239 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.239 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.239 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:22:28.239 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.357206+0000", 2024-09-13T11:22:28.239 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.004324802 2024-09-13T11:22:28.239 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.239 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.239 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:22:28.239 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.365708+0000", 2024-09-13T11:22:28.239 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0085012509999999996 2024-09-13T11:22:28.240 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.240 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.240 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:22:28.240 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.365839+0000", 2024-09-13T11:22:28.240 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000131202 2024-09-13T11:22:28.240 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.240 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.240 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:22:28.240 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.365945+0000", 2024-09-13T11:22:28.240 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000105846 2024-09-13T11:22:28.240 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:22:28.240 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:22:28.240 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:22:28.240 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.241 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.241 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4257.0:50 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-13T11:22:28.241 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:22:12.319731+0000", 2024-09-13T11:22:28.241 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 15.882864681000001, 2024-09-13T11:22:28.241 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.046650321000000002, 2024-09-13T11:22:28.241 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:22:28.241 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:22:28.241 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:28.241 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:22:28.241 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4257", 2024-09-13T11:22:28.241 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/1488131527", 2024-09-13T11:22:28.242 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 50 2024-09-13T11:22:28.242 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.242 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:22:28.242 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.242 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:22:28.242 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.319731+0000", 2024-09-13T11:22:28.242 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:22:28.242 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.242 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.242 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:22:28.242 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.319731+0000", 2024-09-13T11:22:28.242 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:22:28.242 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.242 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.243 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:22:28.243 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.319732+0000", 2024-09-13T11:22:28.243 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.75e-06 2024-09-13T11:22:28.243 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.243 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.243 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:22:28.243 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.319802+0000", 2024-09-13T11:22:28.243 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.9166999999999997e-05 2024-09-13T11:22:28.243 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.243 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.243 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:22:28.243 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.319826+0000", 2024-09-13T11:22:28.244 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.4464000000000002e-05 2024-09-13T11:22:28.244 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.244 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.244 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:22:28.244 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.319856+0000", 2024-09-13T11:22:28.244 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.9530999999999999e-05 2024-09-13T11:22:28.244 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.244 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.244 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:22:28.244 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.357551+0000", 2024-09-13T11:22:28.244 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.037695687999999998 2024-09-13T11:22:28.244 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.244 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.244 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:22:28.245 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.357707+0000", 2024-09-13T11:22:28.245 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00015562200000000001 2024-09-13T11:22:28.245 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.245 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.245 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 0", 2024-09-13T11:22:28.245 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.357906+0000", 2024-09-13T11:22:28.245 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00019874500000000001 2024-09-13T11:22:28.245 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.245 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.245 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:22:28.245 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.366111+0000", 2024-09-13T11:22:28.245 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.008205815 2024-09-13T11:22:28.245 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.246 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.246 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:22:28.246 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.366149+0000", 2024-09-13T11:22:28.246 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.7113999999999997e-05 2024-09-13T11:22:28.246 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.246 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.246 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:22:28.246 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.366283+0000", 2024-09-13T11:22:28.246 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00013399399999999999 2024-09-13T11:22:28.247 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.247 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.247 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:22:28.247 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.366381+0000", 2024-09-13T11:22:28.247 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.8431000000000007e-05 2024-09-13T11:22:28.247 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:22:28.247 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:22:28.247 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:22:28.247 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.247 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.247 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4257.0:51 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-13T11:22:28.247 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:22:12.319877+0000", 2024-09-13T11:22:28.247 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 15.882718050999999, 2024-09-13T11:22:28.247 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.048107695999999998, 2024-09-13T11:22:28.248 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:22:28.248 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:22:28.248 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:28.248 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:22:28.248 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4257", 2024-09-13T11:22:28.248 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/1488131527", 2024-09-13T11:22:28.248 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 51 2024-09-13T11:22:28.248 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.248 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:22:28.248 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.248 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:22:28.248 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.319877+0000", 2024-09-13T11:22:28.248 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:22:28.249 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.249 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.249 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:22:28.249 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.319877+0000", 2024-09-13T11:22:28.249 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:22:28.249 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.249 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.249 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:22:28.249 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.319879+0000", 2024-09-13T11:22:28.249 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.44e-06 2024-09-13T11:22:28.249 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.249 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.249 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:22:28.250 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.319885+0000", 2024-09-13T11:22:28.250 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.8690000000000001e-06 2024-09-13T11:22:28.250 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.250 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.250 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:22:28.250 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.319894+0000", 2024-09-13T11:22:28.250 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.7389999999999994e-06 2024-09-13T11:22:28.250 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.250 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.250 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:22:28.250 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.319903+0000", 2024-09-13T11:22:28.250 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.2830000000000001e-06 2024-09-13T11:22:28.250 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.251 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.251 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:22:28.251 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.361846+0000", 2024-09-13T11:22:28.251 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.041943041 2024-09-13T11:22:28.251 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.251 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.251 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:22:28.251 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.362090+0000", 2024-09-13T11:22:28.251 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00024382399999999999 2024-09-13T11:22:28.251 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.251 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.251 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 0", 2024-09-13T11:22:28.251 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.362493+0000", 2024-09-13T11:22:28.252 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00040374600000000002 2024-09-13T11:22:28.252 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.252 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.252 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:22:28.252 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.366507+0000", 2024-09-13T11:22:28.252 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0040133369999999996 2024-09-13T11:22:28.252 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.252 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.252 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:22:28.252 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.367751+0000", 2024-09-13T11:22:28.252 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001244118 2024-09-13T11:22:28.252 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.252 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.252 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:22:28.253 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.367873+0000", 2024-09-13T11:22:28.253 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000121972 2024-09-13T11:22:28.253 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.253 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.253 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:22:28.253 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.367985+0000", 2024-09-13T11:22:28.253 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000112327 2024-09-13T11:22:28.253 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:22:28.253 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:22:28.253 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:22:28.253 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.253 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.253 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4257.0:52 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-13T11:22:28.254 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:22:12.319912+0000", 2024-09-13T11:22:28.254 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 15.882683394000001, 2024-09-13T11:22:28.254 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.049068963, 2024-09-13T11:22:28.254 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:22:28.254 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:22:28.254 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:28.254 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:22:28.254 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4257", 2024-09-13T11:22:28.254 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/1488131527", 2024-09-13T11:22:28.254 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 52 2024-09-13T11:22:28.254 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.254 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:22:28.254 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.255 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:22:28.255 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.319912+0000", 2024-09-13T11:22:28.255 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:22:28.255 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.255 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.255 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:22:28.255 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.319912+0000", 2024-09-13T11:22:28.255 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:22:28.255 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.255 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.255 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:22:28.255 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.319913+0000", 2024-09-13T11:22:28.255 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.133e-06 2024-09-13T11:22:28.255 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.256 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.256 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:22:28.256 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.319918+0000", 2024-09-13T11:22:28.256 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.7729999999999999e-06 2024-09-13T11:22:28.256 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.256 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.256 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:22:28.256 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.319927+0000", 2024-09-13T11:22:28.256 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.6750000000000008e-06 2024-09-13T11:22:28.256 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.256 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.256 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:22:28.256 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.319935+0000", 2024-09-13T11:22:28.257 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.9500000000000001e-06 2024-09-13T11:22:28.257 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.257 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.257 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:22:28.257 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.366585+0000", 2024-09-13T11:22:28.257 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.046649991000000002 2024-09-13T11:22:28.257 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.257 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.257 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:22:28.257 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.366832+0000", 2024-09-13T11:22:28.257 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00024709999999999999 2024-09-13T11:22:28.257 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.257 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.257 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 0", 2024-09-13T11:22:28.258 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.367210+0000", 2024-09-13T11:22:28.258 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00037837199999999997 2024-09-13T11:22:28.258 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.258 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.258 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:22:28.258 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.368430+0000", 2024-09-13T11:22:28.258 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00122013 2024-09-13T11:22:28.258 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.258 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.258 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:22:28.258 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.368787+0000", 2024-09-13T11:22:28.258 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00035728300000000001 2024-09-13T11:22:28.258 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.259 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.259 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:22:28.259 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.368874+0000", 2024-09-13T11:22:28.259 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.6876e-05 2024-09-13T11:22:28.259 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.259 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.259 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:22:28.259 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.368981+0000", 2024-09-13T11:22:28.259 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00010668 2024-09-13T11:22:28.259 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:22:28.259 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:22:28.259 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:22:28.259 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.260 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.260 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4257.0:70 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-13T11:22:28.260 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:22:12.458708+0000", 2024-09-13T11:22:28.260 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 15.743887437, 2024-09-13T11:22:28.260 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.036822555, 2024-09-13T11:22:28.260 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:22:28.260 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:22:28.260 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:28.260 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:22:28.260 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4257", 2024-09-13T11:22:28.260 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/1488131527", 2024-09-13T11:22:28.260 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 70 2024-09-13T11:22:28.261 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.261 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:22:28.261 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.261 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:22:28.261 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.458708+0000", 2024-09-13T11:22:28.261 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:22:28.261 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.261 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.261 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:22:28.261 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.458708+0000", 2024-09-13T11:22:28.261 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:22:28.261 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.261 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.262 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:22:28.262 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.458709+0000", 2024-09-13T11:22:28.262 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.316e-06 2024-09-13T11:22:28.262 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.262 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.262 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:22:28.262 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.458717+0000", 2024-09-13T11:22:28.262 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.3390000000000004e-06 2024-09-13T11:22:28.262 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.262 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.262 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:22:28.262 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.458726+0000", 2024-09-13T11:22:28.262 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.2329999999999995e-06 2024-09-13T11:22:28.263 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.263 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.263 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:22:28.263 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.458733+0000", 2024-09-13T11:22:28.263 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.5959999999999998e-06 2024-09-13T11:22:28.263 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.263 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.263 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:22:28.263 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.481591+0000", 2024-09-13T11:22:28.263 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.022857796999999999 2024-09-13T11:22:28.263 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.263 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.264 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:22:28.264 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.481749+0000", 2024-09-13T11:22:28.264 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00015727800000000001 2024-09-13T11:22:28.264 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.264 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.264 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 0", 2024-09-13T11:22:28.264 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.481952+0000", 2024-09-13T11:22:28.264 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00020356500000000001 2024-09-13T11:22:28.264 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.264 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.264 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:22:28.264 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.486070+0000", 2024-09-13T11:22:28.265 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0041175689999999997 2024-09-13T11:22:28.265 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.265 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.265 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:22:28.265 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.495291+0000", 2024-09-13T11:22:28.265 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0092216499999999996 2024-09-13T11:22:28.265 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.265 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.265 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:22:28.265 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.495423+0000", 2024-09-13T11:22:28.265 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000131908 2024-09-13T11:22:28.265 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.265 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.266 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:22:28.266 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.495531+0000", 2024-09-13T11:22:28.266 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000107304 2024-09-13T11:22:28.266 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:22:28.266 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:22:28.266 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:22:28.266 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.266 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.266 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4257.0:71 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-13T11:22:28.266 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:22:12.458743+0000", 2024-09-13T11:22:28.266 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 15.743852604000001, 2024-09-13T11:22:28.266 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.037378219999999997, 2024-09-13T11:22:28.266 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:22:28.266 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:22:28.267 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:28.267 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:22:28.267 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4257", 2024-09-13T11:22:28.267 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/1488131527", 2024-09-13T11:22:28.267 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 71 2024-09-13T11:22:28.267 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.267 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:22:28.267 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.267 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:22:28.267 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.458743+0000", 2024-09-13T11:22:28.267 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:22:28.267 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.267 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.268 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:22:28.268 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.458743+0000", 2024-09-13T11:22:28.268 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:22:28.268 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.268 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.268 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:22:28.268 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.458744+0000", 2024-09-13T11:22:28.268 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.7000000000000003e-07 2024-09-13T11:22:28.268 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.268 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.268 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:22:28.269 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.458748+0000", 2024-09-13T11:22:28.269 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.1060000000000002e-06 2024-09-13T11:22:28.269 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.269 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.269 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:22:28.269 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.458760+0000", 2024-09-13T11:22:28.269 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2316e-05 2024-09-13T11:22:28.269 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.269 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.269 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:22:28.269 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.458768+0000", 2024-09-13T11:22:28.269 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.3690000000000001e-06 2024-09-13T11:22:28.269 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.270 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.270 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:22:28.270 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.486413+0000", 2024-09-13T11:22:28.270 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.027645187000000002 2024-09-13T11:22:28.270 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.270 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.270 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:22:28.270 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.486549+0000", 2024-09-13T11:22:28.270 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000135907 2024-09-13T11:22:28.270 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.270 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.270 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 0", 2024-09-13T11:22:28.270 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.486753+0000", 2024-09-13T11:22:28.270 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00020425700000000001 2024-09-13T11:22:28.271 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.271 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.271 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:22:28.271 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.495649+0000", 2024-09-13T11:22:28.271 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0088957589999999996 2024-09-13T11:22:28.271 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.271 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.271 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:22:28.271 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.495901+0000", 2024-09-13T11:22:28.271 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00025266299999999998 2024-09-13T11:22:28.271 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.271 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.271 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:22:28.272 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.496023+0000", 2024-09-13T11:22:28.272 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000121748 2024-09-13T11:22:28.272 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.272 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.272 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:22:28.272 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.496121+0000", 2024-09-13T11:22:28.272 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.7937999999999996e-05 2024-09-13T11:22:28.272 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:22:28.272 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:22:28.272 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:22:28.272 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.272 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.273 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4257.0:72 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-13T11:22:28.273 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:22:12.458777+0000", 2024-09-13T11:22:28.273 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 15.743818547, 2024-09-13T11:22:28.273 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.041935317999999999, 2024-09-13T11:22:28.273 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:22:28.273 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:22:28.273 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:28.273 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:22:28.273 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4257", 2024-09-13T11:22:28.273 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/1488131527", 2024-09-13T11:22:28.273 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 72 2024-09-13T11:22:28.273 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.273 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:22:28.274 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.274 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:22:28.274 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.458777+0000", 2024-09-13T11:22:28.274 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:22:28.274 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.274 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.274 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:22:28.274 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.458777+0000", 2024-09-13T11:22:28.274 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:22:28.274 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.274 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.274 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:22:28.274 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.458808+0000", 2024-09-13T11:22:28.275 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.0904000000000001e-05 2024-09-13T11:22:28.275 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.275 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.275 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:22:28.275 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.458904+0000", 2024-09-13T11:22:28.275 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.6547999999999998e-05 2024-09-13T11:22:28.275 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.275 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.275 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:22:28.275 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.458918+0000", 2024-09-13T11:22:28.275 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3237999999999999e-05 2024-09-13T11:22:28.275 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.275 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.276 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:22:28.276 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.458926+0000", 2024-09-13T11:22:28.276 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.6929999999999996e-06 2024-09-13T11:22:28.276 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.276 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.276 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:22:28.276 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.490489+0000", 2024-09-13T11:22:28.276 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.031563015 2024-09-13T11:22:28.276 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.276 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.276 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:22:28.276 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.490709+0000", 2024-09-13T11:22:28.277 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00022010700000000001 2024-09-13T11:22:28.277 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.277 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.277 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 0", 2024-09-13T11:22:28.277 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.491113+0000", 2024-09-13T11:22:28.277 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00040412899999999998 2024-09-13T11:22:28.277 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.277 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.277 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:22:28.277 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.495864+0000", 2024-09-13T11:22:28.277 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0047506459999999999 2024-09-13T11:22:28.277 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.277 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.278 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:22:28.278 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.500468+0000", 2024-09-13T11:22:28.278 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0046035670000000002 2024-09-13T11:22:28.278 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.278 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.278 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:22:28.278 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.500598+0000", 2024-09-13T11:22:28.278 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000130541 2024-09-13T11:22:28.278 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.278 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.278 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:22:28.278 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.500712+0000", 2024-09-13T11:22:28.279 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00011393 2024-09-13T11:22:28.279 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:22:28.279 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:22:28.279 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:22:28.279 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.279 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.279 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4257.0:73 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-13T11:22:28.279 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:22:12.458939+0000", 2024-09-13T11:22:28.279 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 15.743656228999999, 2024-09-13T11:22:28.279 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.050922262000000003, 2024-09-13T11:22:28.279 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:22:28.279 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:22:28.279 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:28.280 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:22:28.280 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4257", 2024-09-13T11:22:28.280 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/1488131527", 2024-09-13T11:22:28.280 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 73 2024-09-13T11:22:28.280 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.280 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:22:28.280 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.280 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:22:28.280 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.458939+0000", 2024-09-13T11:22:28.280 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:22:28.280 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.280 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.281 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:22:28.281 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.458939+0000", 2024-09-13T11:22:28.281 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:22:28.281 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.281 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.281 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:22:28.281 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.458940+0000", 2024-09-13T11:22:28.281 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.22e-06 2024-09-13T11:22:28.281 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.281 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.281 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:22:28.281 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.458945+0000", 2024-09-13T11:22:28.281 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.6800000000000001e-06 2024-09-13T11:22:28.282 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.282 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.282 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:22:28.282 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.458954+0000", 2024-09-13T11:22:28.282 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.4589999999999998e-06 2024-09-13T11:22:28.282 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.282 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.282 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:22:28.282 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.458961+0000", 2024-09-13T11:22:28.282 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.2420000000000003e-06 2024-09-13T11:22:28.282 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.282 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.282 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:22:28.282 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.496214+0000", 2024-09-13T11:22:28.283 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.037252805999999999 2024-09-13T11:22:28.283 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.283 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.283 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:22:28.283 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.496446+0000", 2024-09-13T11:22:28.283 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00023255800000000001 2024-09-13T11:22:28.283 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.283 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.283 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 0", 2024-09-13T11:22:28.283 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.496800+0000", 2024-09-13T11:22:28.283 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00035397400000000001 2024-09-13T11:22:28.283 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.284 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.284 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:22:28.284 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.504887+0000", 2024-09-13T11:22:28.284 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0080867769999999999 2024-09-13T11:22:28.284 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.284 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.284 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:22:28.284 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.509579+0000", 2024-09-13T11:22:28.284 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0046924849999999997 2024-09-13T11:22:28.284 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.284 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.284 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:22:28.285 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.509748+0000", 2024-09-13T11:22:28.285 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00016907399999999999 2024-09-13T11:22:28.285 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.285 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.285 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:22:28.285 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.509861+0000", 2024-09-13T11:22:28.285 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00011298699999999999 2024-09-13T11:22:28.285 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:22:28.285 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:22:28.285 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:22:28.285 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.285 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.285 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4257.0:74 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-13T11:22:28.286 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:22:12.458973+0000", 2024-09-13T11:22:28.286 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 15.743622809, 2024-09-13T11:22:28.286 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.055492705000000003, 2024-09-13T11:22:28.286 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:22:28.286 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:22:28.286 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:28.286 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:22:28.286 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4257", 2024-09-13T11:22:28.286 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/1488131527", 2024-09-13T11:22:28.286 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 74 2024-09-13T11:22:28.286 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.286 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:22:28.287 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.287 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:22:28.287 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.458973+0000", 2024-09-13T11:22:28.287 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:22:28.287 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.287 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.287 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:22:28.287 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.458973+0000", 2024-09-13T11:22:28.287 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:22:28.287 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.287 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.287 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:22:28.287 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.458974+0000", 2024-09-13T11:22:28.288 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.3699999999999999e-07 2024-09-13T11:22:28.288 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.288 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.288 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:22:28.288 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.458978+0000", 2024-09-13T11:22:28.288 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.4259999999999996e-06 2024-09-13T11:22:28.288 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.288 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.288 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:22:28.288 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.458994+0000", 2024-09-13T11:22:28.288 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5852e-05 2024-09-13T11:22:28.288 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.288 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.289 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:22:28.289 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.459002+0000", 2024-09-13T11:22:28.289 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.5830000000000007e-06 2024-09-13T11:22:28.289 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.289 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.289 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:22:28.289 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.500820+0000", 2024-09-13T11:22:28.289 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.041817432000000002 2024-09-13T11:22:28.289 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.289 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.289 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:22:28.289 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.500973+0000", 2024-09-13T11:22:28.290 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00015326200000000001 2024-09-13T11:22:28.290 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.290 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.290 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 0", 2024-09-13T11:22:28.290 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.501212+0000", 2024-09-13T11:22:28.290 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00023885099999999999 2024-09-13T11:22:28.291 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.291 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.291 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:22:28.291 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.505008+0000", 2024-09-13T11:22:28.291 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0037961850000000001 2024-09-13T11:22:28.291 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.292 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.292 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:22:28.292 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.514227+0000", 2024-09-13T11:22:28.292 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0092187889999999998 2024-09-13T11:22:28.292 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.292 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.292 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:22:28.292 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.514362+0000", 2024-09-13T11:22:28.292 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000134967 2024-09-13T11:22:28.292 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.292 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.292 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:22:28.292 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.514465+0000", 2024-09-13T11:22:28.293 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000103421 2024-09-13T11:22:28.293 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:22:28.293 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:22:28.293 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:22:28.293 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.293 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.293 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4257.0:75 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-13T11:22:28.293 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:22:12.459011+0000", 2024-09-13T11:22:28.293 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 15.743583912, 2024-09-13T11:22:28.293 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.056002977000000002, 2024-09-13T11:22:28.293 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:22:28.293 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:22:28.293 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:28.294 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:22:28.294 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4257", 2024-09-13T11:22:28.294 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/1488131527", 2024-09-13T11:22:28.294 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 75 2024-09-13T11:22:28.294 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.294 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:22:28.294 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.294 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:22:28.294 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.459011+0000", 2024-09-13T11:22:28.294 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:22:28.294 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.294 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.294 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:22:28.294 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.459011+0000", 2024-09-13T11:22:28.295 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:22:28.295 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.295 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.295 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:22:28.295 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.459012+0000", 2024-09-13T11:22:28.295 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.9999999999999995e-07 2024-09-13T11:22:28.295 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.295 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.295 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:22:28.295 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.459060+0000", 2024-09-13T11:22:28.295 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.7587999999999999e-05 2024-09-13T11:22:28.295 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.295 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.295 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:22:28.296 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.459070+0000", 2024-09-13T11:22:28.296 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.9690000000000006e-06 2024-09-13T11:22:28.296 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.296 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.296 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:22:28.296 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.459078+0000", 2024-09-13T11:22:28.296 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.5329999999999997e-06 2024-09-13T11:22:28.296 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.296 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.296 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:22:28.296 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.505105+0000", 2024-09-13T11:22:28.296 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.046027908999999999 2024-09-13T11:22:28.296 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.297 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.297 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:22:28.297 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.505322+0000", 2024-09-13T11:22:28.297 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00021677600000000001 2024-09-13T11:22:28.297 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.297 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.297 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 0", 2024-09-13T11:22:28.297 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.505675+0000", 2024-09-13T11:22:28.297 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00035293800000000001 2024-09-13T11:22:28.297 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.297 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.297 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:22:28.298 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.514620+0000", 2024-09-13T11:22:28.298 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0089446730000000002 2024-09-13T11:22:28.298 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.298 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.298 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:22:28.298 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.514777+0000", 2024-09-13T11:22:28.298 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00015762999999999999 2024-09-13T11:22:28.298 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.298 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.298 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:22:28.298 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.514903+0000", 2024-09-13T11:22:28.298 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00012559800000000001 2024-09-13T11:22:28.298 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.299 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.299 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:22:28.299 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.515014+0000", 2024-09-13T11:22:28.299 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000111363 2024-09-13T11:22:28.299 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:22:28.299 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:22:28.299 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:22:28.299 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.299 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.299 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4257.0:76 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-13T11:22:28.299 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:22:12.459088+0000", 2024-09-13T11:22:28.299 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 15.743507203, 2024-09-13T11:22:28.299 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.056165815000000001, 2024-09-13T11:22:28.299 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:22:28.300 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:22:28.300 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:28.300 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:22:28.300 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4257", 2024-09-13T11:22:28.300 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/1488131527", 2024-09-13T11:22:28.300 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 76 2024-09-13T11:22:28.300 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.300 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:22:28.300 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.300 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:22:28.300 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.459088+0000", 2024-09-13T11:22:28.300 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:22:28.300 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.301 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.301 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:22:28.301 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.459088+0000", 2024-09-13T11:22:28.301 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:22:28.301 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.301 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.301 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:22:28.301 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.459089+0000", 2024-09-13T11:22:28.301 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.114e-06 2024-09-13T11:22:28.301 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.301 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.301 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:22:28.301 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.459094+0000", 2024-09-13T11:22:28.301 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.0320000000000003e-06 2024-09-13T11:22:28.302 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.302 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.302 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:22:28.302 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.459103+0000", 2024-09-13T11:22:28.302 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.2989999999999997e-06 2024-09-13T11:22:28.302 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.302 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.303 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:22:28.303 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.459110+0000", 2024-09-13T11:22:28.303 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.9970000000000001e-06 2024-09-13T11:22:28.303 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.303 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.303 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:22:28.303 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.509960+0000", 2024-09-13T11:22:28.303 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.050850601000000002 2024-09-13T11:22:28.303 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.303 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.303 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:22:28.303 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.510216+0000", 2024-09-13T11:22:28.303 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00025548299999999998 2024-09-13T11:22:28.303 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.304 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.304 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 0", 2024-09-13T11:22:28.304 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.510559+0000", 2024-09-13T11:22:28.304 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00034306199999999999 2024-09-13T11:22:28.304 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.304 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.304 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:22:28.304 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.514745+0000", 2024-09-13T11:22:28.304 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0041858920000000001 2024-09-13T11:22:28.304 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.304 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.304 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:22:28.304 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.515035+0000", 2024-09-13T11:22:28.305 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00029025600000000002 2024-09-13T11:22:28.305 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.305 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.305 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:22:28.305 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.515152+0000", 2024-09-13T11:22:28.305 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000116872 2024-09-13T11:22:28.305 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.305 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.305 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:22:28.305 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.515254+0000", 2024-09-13T11:22:28.305 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00010220700000000001 2024-09-13T11:22:28.305 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:22:28.305 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:22:28.305 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:22:28.306 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.306 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.306 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4257.0:80 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 262144~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-13T11:22:28.306 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:22:12.516065+0000", 2024-09-13T11:22:28.306 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 15.68653016, 2024-09-13T11:22:28.306 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.46461997999999999, 2024-09-13T11:22:28.306 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:22:28.306 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:22:28.306 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:28.306 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:22:28.306 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4257", 2024-09-13T11:22:28.306 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/1488131527", 2024-09-13T11:22:28.306 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 80 2024-09-13T11:22:28.307 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.307 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:22:28.307 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.307 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:22:28.307 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.516065+0000", 2024-09-13T11:22:28.307 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:22:28.307 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.307 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.307 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:22:28.307 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.516065+0000", 2024-09-13T11:22:28.307 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:22:28.307 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.307 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.307 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:22:28.308 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.516066+0000", 2024-09-13T11:22:28.308 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0869999999999999e-06 2024-09-13T11:22:28.308 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.308 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.308 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:22:28.308 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.516112+0000", 2024-09-13T11:22:28.308 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.5503000000000001e-05 2024-09-13T11:22:28.308 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.308 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.308 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:22:28.308 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.516128+0000", 2024-09-13T11:22:28.308 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5891000000000001e-05 2024-09-13T11:22:28.308 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.309 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.309 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:22:28.309 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.516138+0000", 2024-09-13T11:22:28.309 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.8120000000000002e-06 2024-09-13T11:22:28.309 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.309 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.309 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:22:28.309 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.520398+0000", 2024-09-13T11:22:28.309 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0042605450000000001 2024-09-13T11:22:28.309 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.309 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.309 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:22:28.309 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.520526+0000", 2024-09-13T11:22:28.309 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000128355 2024-09-13T11:22:28.310 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.310 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.310 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 0", 2024-09-13T11:22:28.310 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.520730+0000", 2024-09-13T11:22:28.310 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00020337799999999999 2024-09-13T11:22:28.310 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.310 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.310 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:22:28.310 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.522417+0000", 2024-09-13T11:22:28.310 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001687184 2024-09-13T11:22:28.310 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.310 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.310 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:22:28.310 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.980408+0000", 2024-09-13T11:22:28.311 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.45799068399999998 2024-09-13T11:22:28.311 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.311 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.311 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:22:28.311 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.980565+0000", 2024-09-13T11:22:28.311 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00015760899999999999 2024-09-13T11:22:28.311 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.311 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.311 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:22:28.311 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.980685+0000", 2024-09-13T11:22:28.311 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000119932 2024-09-13T11:22:28.311 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:22:28.311 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:22:28.311 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:22:28.311 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.312 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.312 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop_reply(client.4257.0:80 1.0 e24/23 ondisk, result = 0)", 2024-09-13T11:22:28.312 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:22:12.522938+0000", 2024-09-13T11:22:28.312 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 15.679657127, 2024-09-13T11:22:28.312 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.45776420600000001, 2024-09-13T11:22:28.312 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:22:28.312 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:22:28.312 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:22:28.312 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:22:28.312 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.312 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:22:28.312 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.522938+0000", 2024-09-13T11:22:28.312 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:22:28.313 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.313 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.313 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:22:28.313 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.522938+0000", 2024-09-13T11:22:28.313 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:22:28.313 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.313 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.313 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:22:28.313 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.522941+0000", 2024-09-13T11:22:28.313 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.0290000000000001e-06 2024-09-13T11:22:28.313 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.313 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.313 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:22:28.313 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.522950+0000", 2024-09-13T11:22:28.314 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.7499999999999992e-06 2024-09-13T11:22:28.314 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.314 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.314 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:22:28.314 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.980215+0000", 2024-09-13T11:22:28.314 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.45726523800000002 2024-09-13T11:22:28.314 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.314 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.314 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:22:28.314 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.980229+0000", 2024-09-13T11:22:28.314 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3549e-05 2024-09-13T11:22:28.314 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.314 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.314 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:22:28.315 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.980365+0000", 2024-09-13T11:22:28.315 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00013608700000000001 2024-09-13T11:22:28.315 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.315 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.315 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:22:28.315 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.980388+0000", 2024-09-13T11:22:28.315 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.3112000000000002e-05 2024-09-13T11:22:28.315 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.315 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.315 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:22:28.315 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.980702+0000", 2024-09-13T11:22:28.316 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00031444099999999999 2024-09-13T11:22:28.316 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:22:28.316 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:22:28.316 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:22:28.316 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.316 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.316 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4257.0:96 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-13T11:22:28.316 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:22:13.063144+0000", 2024-09-13T11:22:28.316 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 15.139451669, 2024-09-13T11:22:28.316 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.027851055999999999, 2024-09-13T11:22:28.316 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:22:28.316 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:22:28.316 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:28.317 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:22:28.317 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4257", 2024-09-13T11:22:28.317 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/1488131527", 2024-09-13T11:22:28.317 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 96 2024-09-13T11:22:28.317 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.317 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:22:28.317 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.317 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:22:28.317 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:13.063144+0000", 2024-09-13T11:22:28.317 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:22:28.317 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.317 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.317 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:22:28.317 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:13.063144+0000", 2024-09-13T11:22:28.318 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:22:28.318 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.318 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.318 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:22:28.318 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:13.063145+0000", 2024-09-13T11:22:28.318 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5400000000000001e-06 2024-09-13T11:22:28.318 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.318 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.318 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:22:28.318 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:13.063223+0000", 2024-09-13T11:22:28.318 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.7719000000000006e-05 2024-09-13T11:22:28.318 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.318 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.318 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:22:28.319 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:13.063234+0000", 2024-09-13T11:22:28.319 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0929e-05 2024-09-13T11:22:28.319 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.319 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.319 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:22:28.319 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:13.063240+0000", 2024-09-13T11:22:28.319 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.648e-06 2024-09-13T11:22:28.319 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.319 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.319 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:22:28.320 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:13.081483+0000", 2024-09-13T11:22:28.320 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.018243083 2024-09-13T11:22:28.320 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.320 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.320 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:22:28.320 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:13.081565+0000", 2024-09-13T11:22:28.320 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.2163999999999995e-05 2024-09-13T11:22:28.321 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.321 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.321 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 0", 2024-09-13T11:22:28.321 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:13.081689+0000", 2024-09-13T11:22:28.321 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00012389500000000001 2024-09-13T11:22:28.321 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.321 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.321 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:22:28.322 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:13.085208+0000", 2024-09-13T11:22:28.322 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0035195640000000002 2024-09-13T11:22:28.322 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.322 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.322 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:22:28.322 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:13.090892+0000", 2024-09-13T11:22:28.322 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0056841779999999998 2024-09-13T11:22:28.322 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.323 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.323 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:22:28.323 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:13.090955+0000", 2024-09-13T11:22:28.323 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.2087000000000004e-05 2024-09-13T11:22:28.323 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.323 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.323 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:22:28.323 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:13.090995+0000", 2024-09-13T11:22:28.324 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.0249e-05 2024-09-13T11:22:28.324 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:22:28.324 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:22:28.324 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:22:28.324 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.324 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.324 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4257.0:99 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-13T11:22:28.324 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:22:13.063500+0000", 2024-09-13T11:22:28.324 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 15.139095360000001, 2024-09-13T11:22:28.325 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.029416551999999999, 2024-09-13T11:22:28.325 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:22:28.325 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:22:28.325 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:28.325 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:22:28.325 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4257", 2024-09-13T11:22:28.325 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/1488131527", 2024-09-13T11:22:28.325 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 99 2024-09-13T11:22:28.326 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.326 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:22:28.326 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.326 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:22:28.326 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:13.063500+0000", 2024-09-13T11:22:28.326 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:22:28.326 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.326 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.327 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:22:28.327 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:13.063500+0000", 2024-09-13T11:22:28.327 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:22:28.327 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.327 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.327 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:22:28.327 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:13.063513+0000", 2024-09-13T11:22:28.327 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3004e-05 2024-09-13T11:22:28.328 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.328 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.328 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:22:28.328 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:13.063589+0000", 2024-09-13T11:22:28.328 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.5486000000000002e-05 2024-09-13T11:22:28.328 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.328 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.328 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:22:28.329 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:13.063599+0000", 2024-09-13T11:22:28.329 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0584000000000001e-05 2024-09-13T11:22:28.329 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.329 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.329 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:22:28.329 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:13.063605+0000", 2024-09-13T11:22:28.329 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.8220000000000003e-06 2024-09-13T11:22:28.329 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.329 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.329 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:22:28.329 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:13.088873+0000", 2024-09-13T11:22:28.330 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.025267989000000001 2024-09-13T11:22:28.330 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.330 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.330 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:22:28.330 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:13.088984+0000", 2024-09-13T11:22:28.330 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00011157100000000001 2024-09-13T11:22:28.330 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.330 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.330 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 0", 2024-09-13T11:22:28.330 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:13.089155+0000", 2024-09-13T11:22:28.330 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000170237 2024-09-13T11:22:28.331 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.331 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.331 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:22:28.331 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:13.091189+0000", 2024-09-13T11:22:28.331 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0020344149999999999 2024-09-13T11:22:28.331 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.331 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.331 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:22:28.331 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:13.092847+0000", 2024-09-13T11:22:28.331 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0016581079999999999 2024-09-13T11:22:28.331 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.331 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.332 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:22:28.332 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:13.092887+0000", 2024-09-13T11:22:28.332 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.9508000000000002e-05 2024-09-13T11:22:28.332 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.332 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.332 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:22:28.332 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:13.092917+0000", 2024-09-13T11:22:28.332 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.9828e-05 2024-09-13T11:22:28.332 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:22:28.333 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:22:28.333 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:22:28.333 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.333 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.333 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4257.0:100 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-13T11:22:28.333 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:22:13.063617+0000", 2024-09-13T11:22:28.333 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 15.138978444999999, 2024-09-13T11:22:28.333 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.029894398999999999, 2024-09-13T11:22:28.333 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:22:28.333 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:22:28.334 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:28.334 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:22:28.334 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4257", 2024-09-13T11:22:28.334 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/1488131527", 2024-09-13T11:22:28.334 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 100 2024-09-13T11:22:28.334 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.334 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:22:28.334 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.334 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:22:28.334 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:13.063617+0000", 2024-09-13T11:22:28.334 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:22:28.334 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.334 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.335 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:22:28.335 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:13.063617+0000", 2024-09-13T11:22:28.335 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:22:28.335 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.335 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.335 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:22:28.335 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:13.063618+0000", 2024-09-13T11:22:28.335 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.3099999999999997e-07 2024-09-13T11:22:28.335 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.335 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.335 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:22:28.335 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:13.063621+0000", 2024-09-13T11:22:28.335 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.7170000000000002e-06 2024-09-13T11:22:28.336 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.336 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.336 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:22:28.336 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:13.063628+0000", 2024-09-13T11:22:28.336 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.2419999999999997e-06 2024-09-13T11:22:28.336 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.336 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.336 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:22:28.336 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:13.063631+0000", 2024-09-13T11:22:28.336 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.743e-06 2024-09-13T11:22:28.336 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.336 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.337 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:22:28.337 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:13.091220+0000", 2024-09-13T11:22:28.337 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.027588867999999999 2024-09-13T11:22:28.337 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.337 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.337 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:22:28.337 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:13.091324+0000", 2024-09-13T11:22:28.337 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000103712 2024-09-13T11:22:28.337 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.337 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.337 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 0", 2024-09-13T11:22:28.337 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:13.091485+0000", 2024-09-13T11:22:28.337 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00016127599999999999 2024-09-13T11:22:28.338 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.338 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.338 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:22:28.338 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:13.093127+0000", 2024-09-13T11:22:28.338 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0016413980000000001 2024-09-13T11:22:28.338 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.338 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.338 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:22:28.338 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:13.093401+0000", 2024-09-13T11:22:28.338 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00027388299999999999 2024-09-13T11:22:28.338 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.338 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.338 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:22:28.338 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:13.093462+0000", 2024-09-13T11:22:28.339 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.1816999999999994e-05 2024-09-13T11:22:28.339 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.339 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.339 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:22:28.339 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:13.093511+0000", 2024-09-13T11:22:28.339 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.9011999999999998e-05 2024-09-13T11:22:28.339 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:22:28.339 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:22:28.339 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:22:28.339 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:22:28.339 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:22:28.339 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-13T11:22:28.342 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --admin-daemon /var/run/ceph/ceph-client.0.asok objecter_requests 2024-09-13T11:22:28.342 INFO:tasks.ceph.osd.7.smithi177.stderr:2024-09-13T11:22:28.245+0000 7f896ff4a640 -1 osd.7 32 handle_osd_map injecting map crc failure 2024-09-13T11:22:28.342 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-09-13T11:22:28.343 INFO:tasks.ceph.osd.9.smithi092.stderr:2024-09-13T11:22:28.340+0000 7ff5ab3d0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26348) UID: 0 2024-09-13T11:22:28.441 INFO:tasks.ceph.osd.3.smithi177.stderr:2024-09-13T11:22:28.437+0000 7f10adba4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26574) UID: 0 2024-09-13T11:22:28.463 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-09-13T11:22:28.464 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [ 2024-09-13T11:22:28.464 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:28.464 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 4, 2024-09-13T11:22:28.464 INFO:teuthology.orchestra.run.smithi177.stdout: "pg": "3.399e64e8", 2024-09-13T11:22:28.464 INFO:teuthology.orchestra.run.smithi177.stdout: "osd": 6, 2024-09-13T11:22:28.464 INFO:teuthology.orchestra.run.smithi177.stdout: "object_id": "benchmark_data_smithi177_31116_object3", 2024-09-13T11:22:28.464 INFO:teuthology.orchestra.run.smithi177.stdout: "object_locator": "@3", 2024-09-13T11:22:28.464 INFO:teuthology.orchestra.run.smithi177.stdout: "target_object_id": "benchmark_data_smithi177_31116_object3", 2024-09-13T11:22:28.464 INFO:teuthology.orchestra.run.smithi177.stdout: "target_object_locator": "@3", 2024-09-13T11:22:28.464 INFO:teuthology.orchestra.run.smithi177.stdout: "paused": 0, 2024-09-13T11:22:28.464 INFO:teuthology.orchestra.run.smithi177.stdout: "used_replica": 0, 2024-09-13T11:22:28.465 INFO:teuthology.orchestra.run.smithi177.stdout: "precalc_pgid": 0, 2024-09-13T11:22:28.465 INFO:teuthology.orchestra.run.smithi177.stdout: "last_sent": "738.061016s", 2024-09-13T11:22:28.465 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 1.4679085199999999, 2024-09-13T11:22:28.465 INFO:teuthology.orchestra.run.smithi177.stdout: "attempts": 1, 2024-09-13T11:22:28.465 INFO:teuthology.orchestra.run.smithi177.stdout: "snapid": "head", 2024-09-13T11:22:28.465 INFO:teuthology.orchestra.run.smithi177.stdout: "snap_context": "0=[]", 2024-09-13T11:22:28.465 INFO:teuthology.orchestra.run.smithi177.stdout: "mtime": "2024-09-13T11:22:26.997667+0000", 2024-09-13T11:22:28.465 INFO:teuthology.orchestra.run.smithi177.stdout: "osd_ops": [ 2024-09-13T11:22:28.465 INFO:teuthology.orchestra.run.smithi177.stdout: "set-alloc-hint object_size 65536 write_size 65536", 2024-09-13T11:22:28.465 INFO:teuthology.orchestra.run.smithi177.stdout: "write 0~65536 in=65536b" 2024-09-13T11:22:28.465 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:22:28.465 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:28.466 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:28.466 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 64, 2024-09-13T11:22:28.466 INFO:teuthology.orchestra.run.smithi177.stdout: "pg": "3.3a543d8", 2024-09-13T11:22:28.466 INFO:teuthology.orchestra.run.smithi177.stdout: "osd": 6, 2024-09-13T11:22:28.466 INFO:teuthology.orchestra.run.smithi177.stdout: "object_id": "benchmark_data_smithi177_31116_object63", 2024-09-13T11:22:28.466 INFO:teuthology.orchestra.run.smithi177.stdout: "object_locator": "@3", 2024-09-13T11:22:28.466 INFO:teuthology.orchestra.run.smithi177.stdout: "target_object_id": "benchmark_data_smithi177_31116_object63", 2024-09-13T11:22:28.466 INFO:teuthology.orchestra.run.smithi177.stdout: "target_object_locator": "@3", 2024-09-13T11:22:28.466 INFO:teuthology.orchestra.run.smithi177.stdout: "paused": 0, 2024-09-13T11:22:28.467 INFO:teuthology.orchestra.run.smithi177.stdout: "used_replica": 0, 2024-09-13T11:22:28.467 INFO:teuthology.orchestra.run.smithi177.stdout: "precalc_pgid": 0, 2024-09-13T11:22:28.467 INFO:teuthology.orchestra.run.smithi177.stdout: "last_sent": "0.000000s", 2024-09-13T11:22:28.467 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 739.52892403800001, 2024-09-13T11:22:28.467 INFO:teuthology.orchestra.run.smithi177.stdout: "attempts": 0, 2024-09-13T11:22:28.467 INFO:teuthology.orchestra.run.smithi177.stdout: "snapid": "head", 2024-09-13T11:22:28.467 INFO:teuthology.orchestra.run.smithi177.stdout: "snap_context": "0=[]", 2024-09-13T11:22:28.467 INFO:teuthology.orchestra.run.smithi177.stdout: "mtime": "2024-09-13T11:22:27.031884+0000", 2024-09-13T11:22:28.467 INFO:teuthology.orchestra.run.smithi177.stdout: "osd_ops": [ 2024-09-13T11:22:28.468 INFO:teuthology.orchestra.run.smithi177.stdout: "set-alloc-hint object_size 65536 write_size 65536", 2024-09-13T11:22:28.468 INFO:teuthology.orchestra.run.smithi177.stdout: "write 0~65536 in=65536b" 2024-09-13T11:22:28.468 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:22:28.468 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:28.468 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:28.468 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 3, 2024-09-13T11:22:28.468 INFO:teuthology.orchestra.run.smithi177.stdout: "pg": "3.125e09e4", 2024-09-13T11:22:28.468 INFO:teuthology.orchestra.run.smithi177.stdout: "osd": 12, 2024-09-13T11:22:28.468 INFO:teuthology.orchestra.run.smithi177.stdout: "object_id": "benchmark_data_smithi177_31116_object2", 2024-09-13T11:22:28.468 INFO:teuthology.orchestra.run.smithi177.stdout: "object_locator": "@3", 2024-09-13T11:22:28.468 INFO:teuthology.orchestra.run.smithi177.stdout: "target_object_id": "benchmark_data_smithi177_31116_object2", 2024-09-13T11:22:28.468 INFO:teuthology.orchestra.run.smithi177.stdout: "target_object_locator": "@3", 2024-09-13T11:22:28.468 INFO:teuthology.orchestra.run.smithi177.stdout: "paused": 0, 2024-09-13T11:22:28.469 INFO:teuthology.orchestra.run.smithi177.stdout: "used_replica": 0, 2024-09-13T11:22:28.469 INFO:teuthology.orchestra.run.smithi177.stdout: "precalc_pgid": 0, 2024-09-13T11:22:28.469 INFO:teuthology.orchestra.run.smithi177.stdout: "last_sent": "738.061016s", 2024-09-13T11:22:28.469 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 1.4679085199999999, 2024-09-13T11:22:28.469 INFO:teuthology.orchestra.run.smithi177.stdout: "attempts": 1, 2024-09-13T11:22:28.469 INFO:teuthology.orchestra.run.smithi177.stdout: "snapid": "head", 2024-09-13T11:22:28.469 INFO:teuthology.orchestra.run.smithi177.stdout: "snap_context": "0=[]", 2024-09-13T11:22:28.469 INFO:teuthology.orchestra.run.smithi177.stdout: "mtime": "2024-09-13T11:22:26.997662+0000", 2024-09-13T11:22:28.470 INFO:teuthology.orchestra.run.smithi177.stdout: "osd_ops": [ 2024-09-13T11:22:28.470 INFO:teuthology.orchestra.run.smithi177.stdout: "set-alloc-hint object_size 65536 write_size 65536", 2024-09-13T11:22:28.470 INFO:teuthology.orchestra.run.smithi177.stdout: "write 0~65536 in=65536b" 2024-09-13T11:22:28.470 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:22:28.470 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:28.470 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:28.470 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 29, 2024-09-13T11:22:28.470 INFO:teuthology.orchestra.run.smithi177.stdout: "pg": "3.b7bbfbe4", 2024-09-13T11:22:28.470 INFO:teuthology.orchestra.run.smithi177.stdout: "osd": 12, 2024-09-13T11:22:28.470 INFO:teuthology.orchestra.run.smithi177.stdout: "object_id": "benchmark_data_smithi177_31116_object28", 2024-09-13T11:22:28.470 INFO:teuthology.orchestra.run.smithi177.stdout: "object_locator": "@3", 2024-09-13T11:22:28.470 INFO:teuthology.orchestra.run.smithi177.stdout: "target_object_id": "benchmark_data_smithi177_31116_object28", 2024-09-13T11:22:28.470 INFO:teuthology.orchestra.run.smithi177.stdout: "target_object_locator": "@3", 2024-09-13T11:22:28.470 INFO:teuthology.orchestra.run.smithi177.stdout: "paused": 0, 2024-09-13T11:22:28.471 INFO:teuthology.orchestra.run.smithi177.stdout: "used_replica": 0, 2024-09-13T11:22:28.471 INFO:teuthology.orchestra.run.smithi177.stdout: "precalc_pgid": 0, 2024-09-13T11:22:28.471 INFO:teuthology.orchestra.run.smithi177.stdout: "last_sent": "0.000000s", 2024-09-13T11:22:28.471 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 739.52892403800001, 2024-09-13T11:22:28.471 INFO:teuthology.orchestra.run.smithi177.stdout: "attempts": 0, 2024-09-13T11:22:28.471 INFO:teuthology.orchestra.run.smithi177.stdout: "snapid": "head", 2024-09-13T11:22:28.471 INFO:teuthology.orchestra.run.smithi177.stdout: "snap_context": "0=[]", 2024-09-13T11:22:28.471 INFO:teuthology.orchestra.run.smithi177.stdout: "mtime": "2024-09-13T11:22:27.015131+0000", 2024-09-13T11:22:28.471 INFO:teuthology.orchestra.run.smithi177.stdout: "osd_ops": [ 2024-09-13T11:22:28.471 INFO:teuthology.orchestra.run.smithi177.stdout: "set-alloc-hint object_size 65536 write_size 65536", 2024-09-13T11:22:28.471 INFO:teuthology.orchestra.run.smithi177.stdout: "write 0~65536 in=65536b" 2024-09-13T11:22:28.471 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:22:28.472 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:28.472 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:28.472 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 30, 2024-09-13T11:22:28.472 INFO:teuthology.orchestra.run.smithi177.stdout: "pg": "3.547ec0f4", 2024-09-13T11:22:28.472 INFO:teuthology.orchestra.run.smithi177.stdout: "osd": 12, 2024-09-13T11:22:28.472 INFO:teuthology.orchestra.run.smithi177.stdout: "object_id": "benchmark_data_smithi177_31116_object29", 2024-09-13T11:22:28.472 INFO:teuthology.orchestra.run.smithi177.stdout: "object_locator": "@3", 2024-09-13T11:22:28.472 INFO:teuthology.orchestra.run.smithi177.stdout: "target_object_id": "benchmark_data_smithi177_31116_object29", 2024-09-13T11:22:28.472 INFO:teuthology.orchestra.run.smithi177.stdout: "target_object_locator": "@3", 2024-09-13T11:22:28.472 INFO:teuthology.orchestra.run.smithi177.stdout: "paused": 0, 2024-09-13T11:22:28.472 INFO:teuthology.orchestra.run.smithi177.stdout: "used_replica": 0, 2024-09-13T11:22:28.472 INFO:teuthology.orchestra.run.smithi177.stdout: "precalc_pgid": 0, 2024-09-13T11:22:28.472 INFO:teuthology.orchestra.run.smithi177.stdout: "last_sent": "0.000000s", 2024-09-13T11:22:28.473 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 739.52892403800001, 2024-09-13T11:22:28.473 INFO:teuthology.orchestra.run.smithi177.stdout: "attempts": 0, 2024-09-13T11:22:28.473 INFO:teuthology.orchestra.run.smithi177.stdout: "snapid": "head", 2024-09-13T11:22:28.473 INFO:teuthology.orchestra.run.smithi177.stdout: "snap_context": "0=[]", 2024-09-13T11:22:28.473 INFO:teuthology.orchestra.run.smithi177.stdout: "mtime": "2024-09-13T11:22:27.015307+0000", 2024-09-13T11:22:28.473 INFO:teuthology.orchestra.run.smithi177.stdout: "osd_ops": [ 2024-09-13T11:22:28.473 INFO:teuthology.orchestra.run.smithi177.stdout: "set-alloc-hint object_size 65536 write_size 65536", 2024-09-13T11:22:28.474 INFO:teuthology.orchestra.run.smithi177.stdout: "write 0~65536 in=65536b" 2024-09-13T11:22:28.474 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:22:28.474 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:28.474 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:28.474 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 70, 2024-09-13T11:22:28.474 INFO:teuthology.orchestra.run.smithi177.stdout: "pg": "3.1404a604", 2024-09-13T11:22:28.475 INFO:teuthology.orchestra.run.smithi177.stdout: "osd": 12, 2024-09-13T11:22:28.475 INFO:teuthology.orchestra.run.smithi177.stdout: "object_id": "benchmark_data_smithi177_31116_object69", 2024-09-13T11:22:28.475 INFO:teuthology.orchestra.run.smithi177.stdout: "object_locator": "@3", 2024-09-13T11:22:28.475 INFO:teuthology.orchestra.run.smithi177.stdout: "target_object_id": "benchmark_data_smithi177_31116_object69", 2024-09-13T11:22:28.475 INFO:teuthology.orchestra.run.smithi177.stdout: "target_object_locator": "@3", 2024-09-13T11:22:28.475 INFO:teuthology.orchestra.run.smithi177.stdout: "paused": 0, 2024-09-13T11:22:28.476 INFO:teuthology.orchestra.run.smithi177.stdout: "used_replica": 0, 2024-09-13T11:22:28.476 INFO:teuthology.orchestra.run.smithi177.stdout: "precalc_pgid": 0, 2024-09-13T11:22:28.476 INFO:teuthology.orchestra.run.smithi177.stdout: "last_sent": "0.000000s", 2024-09-13T11:22:28.476 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 739.52892403800001, 2024-09-13T11:22:28.476 INFO:teuthology.orchestra.run.smithi177.stdout: "attempts": 0, 2024-09-13T11:22:28.476 INFO:teuthology.orchestra.run.smithi177.stdout: "snapid": "head", 2024-09-13T11:22:28.476 INFO:teuthology.orchestra.run.smithi177.stdout: "snap_context": "0=[]", 2024-09-13T11:22:28.476 INFO:teuthology.orchestra.run.smithi177.stdout: "mtime": "2024-09-13T11:22:27.035123+0000", 2024-09-13T11:22:28.476 INFO:teuthology.orchestra.run.smithi177.stdout: "osd_ops": [ 2024-09-13T11:22:28.476 INFO:teuthology.orchestra.run.smithi177.stdout: "set-alloc-hint object_size 65536 write_size 65536", 2024-09-13T11:22:28.477 INFO:teuthology.orchestra.run.smithi177.stdout: "write 0~65536 in=65536b" 2024-09-13T11:22:28.477 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:22:28.477 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:28.477 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:28.477 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 7, 2024-09-13T11:22:28.477 INFO:teuthology.orchestra.run.smithi177.stdout: "pg": "3.dc9c1e7b", 2024-09-13T11:22:28.477 INFO:teuthology.orchestra.run.smithi177.stdout: "osd": 13, 2024-09-13T11:22:28.477 INFO:teuthology.orchestra.run.smithi177.stdout: "object_id": "benchmark_data_smithi177_31116_object6", 2024-09-13T11:22:28.477 INFO:teuthology.orchestra.run.smithi177.stdout: "object_locator": "@3", 2024-09-13T11:22:28.477 INFO:teuthology.orchestra.run.smithi177.stdout: "target_object_id": "benchmark_data_smithi177_31116_object6", 2024-09-13T11:22:28.477 INFO:teuthology.orchestra.run.smithi177.stdout: "target_object_locator": "@3", 2024-09-13T11:22:28.477 INFO:teuthology.orchestra.run.smithi177.stdout: "paused": 0, 2024-09-13T11:22:28.477 INFO:teuthology.orchestra.run.smithi177.stdout: "used_replica": 0, 2024-09-13T11:22:28.478 INFO:teuthology.orchestra.run.smithi177.stdout: "precalc_pgid": 0, 2024-09-13T11:22:28.478 INFO:teuthology.orchestra.run.smithi177.stdout: "last_sent": "738.061016s", 2024-09-13T11:22:28.478 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 1.4679085199999999, 2024-09-13T11:22:28.478 INFO:teuthology.orchestra.run.smithi177.stdout: "attempts": 1, 2024-09-13T11:22:28.478 INFO:teuthology.orchestra.run.smithi177.stdout: "snapid": "head", 2024-09-13T11:22:28.478 INFO:teuthology.orchestra.run.smithi177.stdout: "snap_context": "0=[]", 2024-09-13T11:22:28.478 INFO:teuthology.orchestra.run.smithi177.stdout: "mtime": "2024-09-13T11:22:26.997742+0000", 2024-09-13T11:22:28.478 INFO:teuthology.orchestra.run.smithi177.stdout: "osd_ops": [ 2024-09-13T11:22:28.478 INFO:teuthology.orchestra.run.smithi177.stdout: "set-alloc-hint object_size 65536 write_size 65536", 2024-09-13T11:22:28.478 INFO:teuthology.orchestra.run.smithi177.stdout: "write 0~65536 in=65536b" 2024-09-13T11:22:28.478 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:22:28.478 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:28.478 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:28.479 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 9, 2024-09-13T11:22:28.479 INFO:teuthology.orchestra.run.smithi177.stdout: "pg": "3.6ffb5f5b", 2024-09-13T11:22:28.479 INFO:teuthology.orchestra.run.smithi177.stdout: "osd": 13, 2024-09-13T11:22:28.479 INFO:teuthology.orchestra.run.smithi177.stdout: "object_id": "benchmark_data_smithi177_31116_object8", 2024-09-13T11:22:28.479 INFO:teuthology.orchestra.run.smithi177.stdout: "object_locator": "@3", 2024-09-13T11:22:28.479 INFO:teuthology.orchestra.run.smithi177.stdout: "target_object_id": "benchmark_data_smithi177_31116_object8", 2024-09-13T11:22:28.479 INFO:teuthology.orchestra.run.smithi177.stdout: "target_object_locator": "@3", 2024-09-13T11:22:28.479 INFO:teuthology.orchestra.run.smithi177.stdout: "paused": 0, 2024-09-13T11:22:28.479 INFO:teuthology.orchestra.run.smithi177.stdout: "used_replica": 0, 2024-09-13T11:22:28.479 INFO:teuthology.orchestra.run.smithi177.stdout: "precalc_pgid": 0, 2024-09-13T11:22:28.480 INFO:teuthology.orchestra.run.smithi177.stdout: "last_sent": "738.061016s", 2024-09-13T11:22:28.480 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 1.4679085199999999, 2024-09-13T11:22:28.480 INFO:teuthology.orchestra.run.smithi177.stdout: "attempts": 1, 2024-09-13T11:22:28.480 INFO:teuthology.orchestra.run.smithi177.stdout: "snapid": "head", 2024-09-13T11:22:28.480 INFO:teuthology.orchestra.run.smithi177.stdout: "snap_context": "0=[]", 2024-09-13T11:22:28.481 INFO:teuthology.orchestra.run.smithi177.stdout: "mtime": "2024-09-13T11:22:26.997776+0000", 2024-09-13T11:22:28.481 INFO:teuthology.orchestra.run.smithi177.stdout: "osd_ops": [ 2024-09-13T11:22:28.481 INFO:teuthology.orchestra.run.smithi177.stdout: "set-alloc-hint object_size 65536 write_size 65536", 2024-09-13T11:22:28.481 INFO:teuthology.orchestra.run.smithi177.stdout: "write 0~65536 in=65536b" 2024-09-13T11:22:28.481 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:22:28.482 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:28.482 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:28.482 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 10, 2024-09-13T11:22:28.482 INFO:teuthology.orchestra.run.smithi177.stdout: "pg": "3.151407db", 2024-09-13T11:22:28.482 INFO:teuthology.orchestra.run.smithi177.stdout: "osd": 13, 2024-09-13T11:22:28.482 INFO:teuthology.orchestra.run.smithi177.stdout: "object_id": "benchmark_data_smithi177_31116_object9", 2024-09-13T11:22:28.482 INFO:teuthology.orchestra.run.smithi177.stdout: "object_locator": "@3", 2024-09-13T11:22:28.482 INFO:teuthology.orchestra.run.smithi177.stdout: "target_object_id": "benchmark_data_smithi177_31116_object9", 2024-09-13T11:22:28.482 INFO:teuthology.orchestra.run.smithi177.stdout: "target_object_locator": "@3", 2024-09-13T11:22:28.483 INFO:teuthology.orchestra.run.smithi177.stdout: "paused": 0, 2024-09-13T11:22:28.483 INFO:teuthology.orchestra.run.smithi177.stdout: "used_replica": 0, 2024-09-13T11:22:28.483 INFO:teuthology.orchestra.run.smithi177.stdout: "precalc_pgid": 0, 2024-09-13T11:22:28.483 INFO:teuthology.orchestra.run.smithi177.stdout: "last_sent": "738.061016s", 2024-09-13T11:22:28.483 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 1.4679085199999999, 2024-09-13T11:22:28.483 INFO:teuthology.orchestra.run.smithi177.stdout: "attempts": 1, 2024-09-13T11:22:28.483 INFO:teuthology.orchestra.run.smithi177.stdout: "snapid": "head", 2024-09-13T11:22:28.483 INFO:teuthology.orchestra.run.smithi177.stdout: "snap_context": "0=[]", 2024-09-13T11:22:28.483 INFO:teuthology.orchestra.run.smithi177.stdout: "mtime": "2024-09-13T11:22:26.997783+0000", 2024-09-13T11:22:28.483 INFO:teuthology.orchestra.run.smithi177.stdout: "osd_ops": [ 2024-09-13T11:22:28.483 INFO:teuthology.orchestra.run.smithi177.stdout: "set-alloc-hint object_size 65536 write_size 65536", 2024-09-13T11:22:28.483 INFO:teuthology.orchestra.run.smithi177.stdout: "write 0~65536 in=65536b" 2024-09-13T11:22:28.483 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:22:28.483 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:28.484 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:28.484 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 39, 2024-09-13T11:22:28.484 INFO:teuthology.orchestra.run.smithi177.stdout: "pg": "3.cd2a189b", 2024-09-13T11:22:28.484 INFO:teuthology.orchestra.run.smithi177.stdout: "osd": 13, 2024-09-13T11:22:28.484 INFO:teuthology.orchestra.run.smithi177.stdout: "object_id": "benchmark_data_smithi177_31116_object38", 2024-09-13T11:22:28.484 INFO:teuthology.orchestra.run.smithi177.stdout: "object_locator": "@3", 2024-09-13T11:22:28.484 INFO:teuthology.orchestra.run.smithi177.stdout: "target_object_id": "benchmark_data_smithi177_31116_object38", 2024-09-13T11:22:28.484 INFO:teuthology.orchestra.run.smithi177.stdout: "target_object_locator": "@3", 2024-09-13T11:22:28.485 INFO:teuthology.orchestra.run.smithi177.stdout: "paused": 0, 2024-09-13T11:22:28.485 INFO:teuthology.orchestra.run.smithi177.stdout: "used_replica": 0, 2024-09-13T11:22:28.485 INFO:teuthology.orchestra.run.smithi177.stdout: "precalc_pgid": 0, 2024-09-13T11:22:28.485 INFO:teuthology.orchestra.run.smithi177.stdout: "last_sent": "0.000000s", 2024-09-13T11:22:28.485 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 739.52892403800001, 2024-09-13T11:22:28.485 INFO:teuthology.orchestra.run.smithi177.stdout: "attempts": 0, 2024-09-13T11:22:28.485 INFO:teuthology.orchestra.run.smithi177.stdout: "snapid": "head", 2024-09-13T11:22:28.485 INFO:teuthology.orchestra.run.smithi177.stdout: "snap_context": "0=[]", 2024-09-13T11:22:28.485 INFO:teuthology.orchestra.run.smithi177.stdout: "mtime": "2024-09-13T11:22:27.018472+0000", 2024-09-13T11:22:28.485 INFO:teuthology.orchestra.run.smithi177.stdout: "osd_ops": [ 2024-09-13T11:22:28.485 INFO:teuthology.orchestra.run.smithi177.stdout: "set-alloc-hint object_size 65536 write_size 65536", 2024-09-13T11:22:28.485 INFO:teuthology.orchestra.run.smithi177.stdout: "write 0~65536 in=65536b" 2024-09-13T11:22:28.486 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:22:28.486 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:28.486 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:28.486 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 50, 2024-09-13T11:22:28.486 INFO:teuthology.orchestra.run.smithi177.stdout: "pg": "3.1d22838b", 2024-09-13T11:22:28.486 INFO:teuthology.orchestra.run.smithi177.stdout: "osd": 13, 2024-09-13T11:22:28.486 INFO:teuthology.orchestra.run.smithi177.stdout: "object_id": "benchmark_data_smithi177_31116_object49", 2024-09-13T11:22:28.486 INFO:teuthology.orchestra.run.smithi177.stdout: "object_locator": "@3", 2024-09-13T11:22:28.486 INFO:teuthology.orchestra.run.smithi177.stdout: "target_object_id": "benchmark_data_smithi177_31116_object49", 2024-09-13T11:22:28.486 INFO:teuthology.orchestra.run.smithi177.stdout: "target_object_locator": "@3", 2024-09-13T11:22:28.486 INFO:teuthology.orchestra.run.smithi177.stdout: "paused": 0, 2024-09-13T11:22:28.487 INFO:teuthology.orchestra.run.smithi177.stdout: "used_replica": 0, 2024-09-13T11:22:28.487 INFO:teuthology.orchestra.run.smithi177.stdout: "precalc_pgid": 0, 2024-09-13T11:22:28.487 INFO:teuthology.orchestra.run.smithi177.stdout: "last_sent": "0.000000s", 2024-09-13T11:22:28.487 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 739.52892403800001, 2024-09-13T11:22:28.487 INFO:teuthology.orchestra.run.smithi177.stdout: "attempts": 0, 2024-09-13T11:22:28.487 INFO:teuthology.orchestra.run.smithi177.stdout: "snapid": "head", 2024-09-13T11:22:28.487 INFO:teuthology.orchestra.run.smithi177.stdout: "snap_context": "0=[]", 2024-09-13T11:22:28.487 INFO:teuthology.orchestra.run.smithi177.stdout: "mtime": "2024-09-13T11:22:27.024431+0000", 2024-09-13T11:22:28.487 INFO:teuthology.orchestra.run.smithi177.stdout: "osd_ops": [ 2024-09-13T11:22:28.487 INFO:teuthology.orchestra.run.smithi177.stdout: "set-alloc-hint object_size 65536 write_size 65536", 2024-09-13T11:22:28.487 INFO:teuthology.orchestra.run.smithi177.stdout: "write 0~65536 in=65536b" 2024-09-13T11:22:28.487 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:22:28.487 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:28.488 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:28.488 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 60, 2024-09-13T11:22:28.488 INFO:teuthology.orchestra.run.smithi177.stdout: "pg": "3.d184e6b", 2024-09-13T11:22:28.488 INFO:teuthology.orchestra.run.smithi177.stdout: "osd": 13, 2024-09-13T11:22:28.488 INFO:teuthology.orchestra.run.smithi177.stdout: "object_id": "benchmark_data_smithi177_31116_object59", 2024-09-13T11:22:28.488 INFO:teuthology.orchestra.run.smithi177.stdout: "object_locator": "@3", 2024-09-13T11:22:28.489 INFO:teuthology.orchestra.run.smithi177.stdout: "target_object_id": "benchmark_data_smithi177_31116_object59", 2024-09-13T11:22:28.489 INFO:teuthology.orchestra.run.smithi177.stdout: "target_object_locator": "@3", 2024-09-13T11:22:28.489 INFO:teuthology.orchestra.run.smithi177.stdout: "paused": 0, 2024-09-13T11:22:28.489 INFO:teuthology.orchestra.run.smithi177.stdout: "used_replica": 0, 2024-09-13T11:22:28.489 INFO:teuthology.orchestra.run.smithi177.stdout: "precalc_pgid": 0, 2024-09-13T11:22:28.490 INFO:teuthology.orchestra.run.smithi177.stdout: "last_sent": "0.000000s", 2024-09-13T11:22:28.490 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 739.52892403800001, 2024-09-13T11:22:28.490 INFO:teuthology.orchestra.run.smithi177.stdout: "attempts": 0, 2024-09-13T11:22:28.490 INFO:teuthology.orchestra.run.smithi177.stdout: "snapid": "head", 2024-09-13T11:22:28.490 INFO:teuthology.orchestra.run.smithi177.stdout: "snap_context": "0=[]", 2024-09-13T11:22:28.490 INFO:teuthology.orchestra.run.smithi177.stdout: "mtime": "2024-09-13T11:22:27.030904+0000", 2024-09-13T11:22:28.490 INFO:teuthology.orchestra.run.smithi177.stdout: "osd_ops": [ 2024-09-13T11:22:28.490 INFO:teuthology.orchestra.run.smithi177.stdout: "set-alloc-hint object_size 65536 write_size 65536", 2024-09-13T11:22:28.491 INFO:teuthology.orchestra.run.smithi177.stdout: "write 0~65536 in=65536b" 2024-09-13T11:22:28.491 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:22:28.491 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:28.491 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:28.491 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 62, 2024-09-13T11:22:28.491 INFO:teuthology.orchestra.run.smithi177.stdout: "pg": "3.15a7504b", 2024-09-13T11:22:28.491 INFO:teuthology.orchestra.run.smithi177.stdout: "osd": 13, 2024-09-13T11:22:28.491 INFO:teuthology.orchestra.run.smithi177.stdout: "object_id": "benchmark_data_smithi177_31116_object61", 2024-09-13T11:22:28.491 INFO:teuthology.orchestra.run.smithi177.stdout: "object_locator": "@3", 2024-09-13T11:22:28.491 INFO:teuthology.orchestra.run.smithi177.stdout: "target_object_id": "benchmark_data_smithi177_31116_object61", 2024-09-13T11:22:28.491 INFO:teuthology.orchestra.run.smithi177.stdout: "target_object_locator": "@3", 2024-09-13T11:22:28.491 INFO:teuthology.orchestra.run.smithi177.stdout: "paused": 0, 2024-09-13T11:22:28.491 INFO:teuthology.orchestra.run.smithi177.stdout: "used_replica": 0, 2024-09-13T11:22:28.492 INFO:teuthology.orchestra.run.smithi177.stdout: "precalc_pgid": 0, 2024-09-13T11:22:28.492 INFO:teuthology.orchestra.run.smithi177.stdout: "last_sent": "0.000000s", 2024-09-13T11:22:28.492 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 739.52892403800001, 2024-09-13T11:22:28.492 INFO:teuthology.orchestra.run.smithi177.stdout: "attempts": 0, 2024-09-13T11:22:28.492 INFO:teuthology.orchestra.run.smithi177.stdout: "snapid": "head", 2024-09-13T11:22:28.492 INFO:teuthology.orchestra.run.smithi177.stdout: "snap_context": "0=[]", 2024-09-13T11:22:28.492 INFO:teuthology.orchestra.run.smithi177.stdout: "mtime": "2024-09-13T11:22:27.031309+0000", 2024-09-13T11:22:28.492 INFO:teuthology.orchestra.run.smithi177.stdout: "osd_ops": [ 2024-09-13T11:22:28.492 INFO:teuthology.orchestra.run.smithi177.stdout: "set-alloc-hint object_size 65536 write_size 65536", 2024-09-13T11:22:28.492 INFO:teuthology.orchestra.run.smithi177.stdout: "write 0~65536 in=65536b" 2024-09-13T11:22:28.492 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:22:28.492 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:28.492 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:28.493 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 73, 2024-09-13T11:22:28.493 INFO:teuthology.orchestra.run.smithi177.stdout: "pg": "3.fab0600b", 2024-09-13T11:22:28.493 INFO:teuthology.orchestra.run.smithi177.stdout: "osd": 13, 2024-09-13T11:22:28.493 INFO:teuthology.orchestra.run.smithi177.stdout: "object_id": "benchmark_data_smithi177_31116_object72", 2024-09-13T11:22:28.493 INFO:teuthology.orchestra.run.smithi177.stdout: "object_locator": "@3", 2024-09-13T11:22:28.493 INFO:teuthology.orchestra.run.smithi177.stdout: "target_object_id": "benchmark_data_smithi177_31116_object72", 2024-09-13T11:22:28.493 INFO:teuthology.orchestra.run.smithi177.stdout: "target_object_locator": "@3", 2024-09-13T11:22:28.493 INFO:teuthology.orchestra.run.smithi177.stdout: "paused": 0, 2024-09-13T11:22:28.493 INFO:teuthology.orchestra.run.smithi177.stdout: "used_replica": 0, 2024-09-13T11:22:28.493 INFO:teuthology.orchestra.run.smithi177.stdout: "precalc_pgid": 0, 2024-09-13T11:22:28.493 INFO:teuthology.orchestra.run.smithi177.stdout: "last_sent": "0.000000s", 2024-09-13T11:22:28.493 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 739.52892403800001, 2024-09-13T11:22:28.493 INFO:teuthology.orchestra.run.smithi177.stdout: "attempts": 0, 2024-09-13T11:22:28.493 INFO:teuthology.orchestra.run.smithi177.stdout: "snapid": "head", 2024-09-13T11:22:28.494 INFO:teuthology.orchestra.run.smithi177.stdout: "snap_context": "0=[]", 2024-09-13T11:22:28.494 INFO:teuthology.orchestra.run.smithi177.stdout: "mtime": "2024-09-13T11:22:27.038855+0000", 2024-09-13T11:22:28.494 INFO:teuthology.orchestra.run.smithi177.stdout: "osd_ops": [ 2024-09-13T11:22:28.494 INFO:teuthology.orchestra.run.smithi177.stdout: "set-alloc-hint object_size 65536 write_size 65536", 2024-09-13T11:22:28.494 INFO:teuthology.orchestra.run.smithi177.stdout: "write 0~65536 in=65536b" 2024-09-13T11:22:28.494 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:22:28.494 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:28.494 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:28.494 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 81, 2024-09-13T11:22:28.494 INFO:teuthology.orchestra.run.smithi177.stdout: "pg": "3.ebfa0eb", 2024-09-13T11:22:28.494 INFO:teuthology.orchestra.run.smithi177.stdout: "osd": 13, 2024-09-13T11:22:28.494 INFO:teuthology.orchestra.run.smithi177.stdout: "object_id": "benchmark_data_smithi177_31116_object80", 2024-09-13T11:22:28.494 INFO:teuthology.orchestra.run.smithi177.stdout: "object_locator": "@3", 2024-09-13T11:22:28.494 INFO:teuthology.orchestra.run.smithi177.stdout: "target_object_id": "benchmark_data_smithi177_31116_object80", 2024-09-13T11:22:28.495 INFO:teuthology.orchestra.run.smithi177.stdout: "target_object_locator": "@3", 2024-09-13T11:22:28.495 INFO:teuthology.orchestra.run.smithi177.stdout: "paused": 0, 2024-09-13T11:22:28.495 INFO:teuthology.orchestra.run.smithi177.stdout: "used_replica": 0, 2024-09-13T11:22:28.495 INFO:teuthology.orchestra.run.smithi177.stdout: "precalc_pgid": 0, 2024-09-13T11:22:28.495 INFO:teuthology.orchestra.run.smithi177.stdout: "last_sent": "0.000000s", 2024-09-13T11:22:28.495 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 739.52892403800001, 2024-09-13T11:22:28.495 INFO:teuthology.orchestra.run.smithi177.stdout: "attempts": 0, 2024-09-13T11:22:28.495 INFO:teuthology.orchestra.run.smithi177.stdout: "snapid": "head", 2024-09-13T11:22:28.495 INFO:teuthology.orchestra.run.smithi177.stdout: "snap_context": "0=[]", 2024-09-13T11:22:28.495 INFO:teuthology.orchestra.run.smithi177.stdout: "mtime": "2024-09-13T11:22:27.060226+0000", 2024-09-13T11:22:28.495 INFO:teuthology.orchestra.run.smithi177.stdout: "osd_ops": [ 2024-09-13T11:22:28.495 INFO:teuthology.orchestra.run.smithi177.stdout: "set-alloc-hint object_size 65536 write_size 65536", 2024-09-13T11:22:28.495 INFO:teuthology.orchestra.run.smithi177.stdout: "write 0~65536 in=65536b" 2024-09-13T11:22:28.495 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:22:28.496 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:28.496 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:28.496 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 87, 2024-09-13T11:22:28.496 INFO:teuthology.orchestra.run.smithi177.stdout: "pg": "3.b507bebb", 2024-09-13T11:22:28.496 INFO:teuthology.orchestra.run.smithi177.stdout: "osd": 13, 2024-09-13T11:22:28.496 INFO:teuthology.orchestra.run.smithi177.stdout: "object_id": "benchmark_data_smithi177_31116_object86", 2024-09-13T11:22:28.496 INFO:teuthology.orchestra.run.smithi177.stdout: "object_locator": "@3", 2024-09-13T11:22:28.496 INFO:teuthology.orchestra.run.smithi177.stdout: "target_object_id": "benchmark_data_smithi177_31116_object86", 2024-09-13T11:22:28.496 INFO:teuthology.orchestra.run.smithi177.stdout: "target_object_locator": "@3", 2024-09-13T11:22:28.496 INFO:teuthology.orchestra.run.smithi177.stdout: "paused": 0, 2024-09-13T11:22:28.496 INFO:teuthology.orchestra.run.smithi177.stdout: "used_replica": 0, 2024-09-13T11:22:28.496 INFO:teuthology.orchestra.run.smithi177.stdout: "precalc_pgid": 0, 2024-09-13T11:22:28.496 INFO:teuthology.orchestra.run.smithi177.stdout: "last_sent": "0.000000s", 2024-09-13T11:22:28.497 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 739.52892403800001, 2024-09-13T11:22:28.497 INFO:teuthology.orchestra.run.smithi177.stdout: "attempts": 0, 2024-09-13T11:22:28.497 INFO:teuthology.orchestra.run.smithi177.stdout: "snapid": "head", 2024-09-13T11:22:28.497 INFO:teuthology.orchestra.run.smithi177.stdout: "snap_context": "0=[]", 2024-09-13T11:22:28.497 INFO:teuthology.orchestra.run.smithi177.stdout: "mtime": "2024-09-13T11:22:28.341062+0000", 2024-09-13T11:22:28.497 INFO:teuthology.orchestra.run.smithi177.stdout: "osd_ops": [ 2024-09-13T11:22:28.497 INFO:teuthology.orchestra.run.smithi177.stdout: "set-alloc-hint object_size 65536 write_size 65536", 2024-09-13T11:22:28.497 INFO:teuthology.orchestra.run.smithi177.stdout: "write 0~65536 in=65536b" 2024-09-13T11:22:28.497 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:22:28.497 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:28.497 INFO:teuthology.orchestra.run.smithi177.stdout: ], 2024-09-13T11:22:28.497 INFO:teuthology.orchestra.run.smithi177.stdout: "linger_ops": [], 2024-09-13T11:22:28.497 INFO:teuthology.orchestra.run.smithi177.stdout: "pool_ops": [], 2024-09-13T11:22:28.498 INFO:teuthology.orchestra.run.smithi177.stdout: "pool_stat_ops": [], 2024-09-13T11:22:28.498 INFO:teuthology.orchestra.run.smithi177.stdout: "statfs_ops": [], 2024-09-13T11:22:28.498 INFO:teuthology.orchestra.run.smithi177.stdout: "command_ops": [] 2024-09-13T11:22:28.498 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-09-13T11:22:28.499 DEBUG:tasks.admin_socket:admin socket command objecter_requests returned { "ops": [ { "tid": 4, "pg": "3.399e64e8", "osd": 6, "object_id": "benchmark_data_smithi177_31116_object3", "object_locator": "@3", "target_object_id": "benchmark_data_smithi177_31116_object3", "target_object_locator": "@3", "paused": 0, "used_replica": 0, "precalc_pgid": 0, "last_sent": "738.061016s", "age": 1.4679085199999999, "attempts": 1, "snapid": "head", "snap_context": "0=[]", "mtime": "2024-09-13T11:22:26.997667+0000", "osd_ops": [ "set-alloc-hint object_size 65536 write_size 65536", "write 0~65536 in=65536b" ] }, { "tid": 64, "pg": "3.3a543d8", "osd": 6, "object_id": "benchmark_data_smithi177_31116_object63", "object_locator": "@3", "target_object_id": "benchmark_data_smithi177_31116_object63", "target_object_locator": "@3", "paused": 0, "used_replica": 0, "precalc_pgid": 0, "last_sent": "0.000000s", "age": 739.52892403800001, "attempts": 0, "snapid": "head", "snap_context": "0=[]", "mtime": "2024-09-13T11:22:27.031884+0000", "osd_ops": [ "set-alloc-hint object_size 65536 write_size 65536", "write 0~65536 in=65536b" ] }, { "tid": 3, "pg": "3.125e09e4", "osd": 12, "object_id": "benchmark_data_smithi177_31116_object2", "object_locator": "@3", "target_object_id": "benchmark_data_smithi177_31116_object2", "target_object_locator": "@3", "paused": 0, "used_replica": 0, "precalc_pgid": 0, "last_sent": "738.061016s", "age": 1.4679085199999999, "attempts": 1, "snapid": "head", "snap_context": "0=[]", "mtime": "2024-09-13T11:22:26.997662+0000", "osd_ops": [ "set-alloc-hint object_size 65536 write_size 65536", "write 0~65536 in=65536b" ] }, { "tid": 29, "pg": "3.b7bbfbe4", "osd": 12, "object_id": "benchmark_data_smithi177_31116_object28", "object_locator": "@3", "target_object_id": "benchmark_data_smithi177_31116_object28", "target_object_locator": "@3", "paused": 0, "used_replica": 0, "precalc_pgid": 0, "last_sent": "0.000000s", "age": 739.52892403800001, "attempts": 0, "snapid": "head", "snap_context": "0=[]", "mtime": "2024-09-13T11:22:27.015131+0000", "osd_ops": [ "set-alloc-hint object_size 65536 write_size 65536", "write 0~65536 in=65536b" ] }, { "tid": 30, "pg": "3.547ec0f4", "osd": 12, "object_id": "benchmark_data_smithi177_31116_object29", "object_locator": "@3", "target_object_id": "benchmark_data_smithi177_31116_object29", "target_object_locator": "@3", "paused": 0, "used_replica": 0, "precalc_pgid": 0, "last_sent": "0.000000s", "age": 739.52892403800001, "attempts": 0, "snapid": "head", "snap_context": "0=[]", "mtime": "2024-09-13T11:22:27.015307+0000", "osd_ops": [ "set-alloc-hint object_size 65536 write_size 65536", "write 0~65536 in=65536b" ] }, { "tid": 70, "pg": "3.1404a604", "osd": 12, "object_id": "benchmark_data_smithi177_31116_object69", "object_locator": "@3", "target_object_id": "benchmark_data_smithi177_31116_object69", "target_object_locator": "@3", "paused": 0, "used_replica": 0, "precalc_pgid": 0, "last_sent": "0.000000s", "age": 739.52892403800001, "attempts": 0, "snapid": "head", "snap_context": "0=[]", "mtime": "2024-09-13T11:22:27.035123+0000", "osd_ops": [ "set-alloc-hint object_size 65536 write_size 65536", "write 0~65536 in=65536b" ] }, { "tid": 7, "pg": "3.dc9c1e7b", "osd": 13, "object_id": "benchmark_data_smithi177_31116_object6", "object_locator": "@3", "target_object_id": "benchmark_data_smithi177_31116_object6", "target_object_locator": "@3", "paused": 0, "used_replica": 0, "precalc_pgid": 0, "last_sent": "738.061016s", "age": 1.4679085199999999, "attempts": 1, "snapid": "head", "snap_context": "0=[]", "mtime": "2024-09-13T11:22:26.997742+0000", "osd_ops": [ "set-alloc-hint object_size 65536 write_size 65536", "write 0~65536 in=65536b" ] }, { "tid": 9, "pg": "3.6ffb5f5b", "osd": 13, "object_id": "benchmark_data_smithi177_31116_object8", "object_locator": "@3", "target_object_id": "benchmark_data_smithi177_31116_object8", "target_object_locator": "@3", "paused": 0, "used_replica": 0, "precalc_pgid": 0, "last_sent": "738.061016s", "age": 1.4679085199999999, "attempts": 1, "snapid": "head", "snap_context": "0=[]", "mtime": "2024-09-13T11:22:26.997776+0000", "osd_ops": [ "set-alloc-hint object_size 65536 write_size 65536", "write 0~65536 in=65536b" ] }, { "tid": 10, "pg": "3.151407db", "osd": 13, "object_id": "benchmark_data_smithi177_31116_object9", "object_locator": "@3", "target_object_id": "benchmark_data_smithi177_31116_object9", "target_object_locator": "@3", "paused": 0, "used_replica": 0, "precalc_pgid": 0, "last_sent": "738.061016s", "age": 1.4679085199999999, "attempts": 1, "snapid": "head", "snap_context": "0=[]", "mtime": "2024-09-13T11:22:26.997783+0000", "osd_ops": [ "set-alloc-hint object_size 65536 write_size 65536", "write 0~65536 in=65536b" ] }, { "tid": 39, "pg": "3.cd2a189b", "osd": 13, "object_id": "benchmark_data_smithi177_31116_object38", "object_locator": "@3", "target_object_id": "benchmark_data_smithi177_31116_object38", "target_object_locator": "@3", "paused": 0, "used_replica": 0, "precalc_pgid": 0, "last_sent": "0.000000s", "age": 739.52892403800001, "attempts": 0, "snapid": "head", "snap_context": "0=[]", "mtime": "2024-09-13T11:22:27.018472+0000", "osd_ops": [ "set-alloc-hint object_size 65536 write_size 65536", "write 0~65536 in=65536b" ] }, { "tid": 50, "pg": "3.1d22838b", "osd": 13, "object_id": "benchmark_data_smithi177_31116_object49", "object_locator": "@3", "target_object_id": "benchmark_data_smithi177_31116_object49", "target_object_locator": "@3", "paused": 0, "used_replica": 0, "precalc_pgid": 0, "last_sent": "0.000000s", "age": 739.52892403800001, "attempts": 0, "snapid": "head", "snap_context": "0=[]", "mtime": "2024-09-13T11:22:27.024431+0000", "osd_ops": [ "set-alloc-hint object_size 65536 write_size 65536", "write 0~65536 in=65536b" ] }, { "tid": 60, "pg": "3.d184e6b", "osd": 13, "object_id": "benchmark_data_smithi177_31116_object59", "object_locator": "@3", "target_object_id": "benchmark_data_smithi177_31116_object59", "target_object_locator": "@3", "paused": 0, "used_replica": 0, "precalc_pgid": 0, "last_sent": "0.000000s", "age": 739.52892403800001, "attempts": 0, "snapid": "head", "snap_context": "0=[]", "mtime": "2024-09-13T11:22:27.030904+0000", "osd_ops": [ "set-alloc-hint object_size 65536 write_size 65536", "write 0~65536 in=65536b" ] }, { "tid": 62, "pg": "3.15a7504b", "osd": 13, "object_id": "benchmark_data_smithi177_31116_object61", "object_locator": "@3", "target_object_id": "benchmark_data_smithi177_31116_object61", "target_object_locator": "@3", "paused": 0, "used_replica": 0, "precalc_pgid": 0, "last_sent": "0.000000s", "age": 739.52892403800001, "attempts": 0, "snapid": "head", "snap_context": "0=[]", "mtime": "2024-09-13T11:22:27.031309+0000", "osd_ops": [ "set-alloc-hint object_size 65536 write_size 65536", "write 0~65536 in=65536b" ] }, { "tid": 73, "pg": "3.fab0600b", "osd": 13, "object_id": "benchmark_data_smithi177_31116_object72", "object_locator": "@3", "target_object_id": "benchmark_data_smithi177_31116_object72", "target_object_locator": "@3", "paused": 0, "used_replica": 0, "precalc_pgid": 0, "last_sent": "0.000000s", "age": 739.52892403800001, "attempts": 0, "snapid": "head", "snap_context": "0=[]", "mtime": "2024-09-13T11:22:27.038855+0000", "osd_ops": [ "set-alloc-hint object_size 65536 write_size 65536", "write 0~65536 in=65536b" ] }, { "tid": 81, "pg": "3.ebfa0eb", "osd": 13, "object_id": "benchmark_data_smithi177_31116_object80", "object_locator": "@3", "target_object_id": "benchmark_data_smithi177_31116_object80", "target_object_locator": "@3", "paused": 0, "used_replica": 0, "precalc_pgid": 0, "last_sent": "0.000000s", "age": 739.52892403800001, "attempts": 0, "snapid": "head", "snap_context": "0=[]", "mtime": "2024-09-13T11:22:27.060226+0000", "osd_ops": [ "set-alloc-hint object_size 65536 write_size 65536", "write 0~65536 in=65536b" ] }, { "tid": 87, "pg": "3.b507bebb", "osd": 13, "object_id": "benchmark_data_smithi177_31116_object86", "object_locator": "@3", "target_object_id": "benchmark_data_smithi177_31116_object86", "target_object_locator": "@3", "paused": 0, "used_replica": 0, "precalc_pgid": 0, "last_sent": "0.000000s", "age": 739.52892403800001, "attempts": 0, "snapid": "head", "snap_context": "0=[]", "mtime": "2024-09-13T11:22:28.341062+0000", "osd_ops": [ "set-alloc-hint object_size 65536 write_size 65536", "write 0~65536 in=65536b" ] } ], "linger_ops": [], "pool_ops": [], "pool_stat_ops": [], "statfs_ops": [], "command_ops": [] } 2024-09-13T11:22:28.499 DEBUG:teuthology.orchestra.run.smithi177:> /home/ubuntu/cephtest/admin_socket_client.0/objecter_requests 2024-09-13T11:22:28.513 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-13T11:22:28.513 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [], 2024-09-13T11:22:28.513 INFO:teuthology.orchestra.run.smithi162.stdout: "num_ops": 0 2024-09-13T11:22:28.513 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-13T11:22:28.527 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-09-13T11:22:28.542 INFO:tasks.ceph.osd.11.smithi177.stderr:2024-09-13T11:22:28.541+0000 7f24792c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26580) UID: 0 2024-09-13T11:22:28.569 DEBUG:teuthology.orchestra.run.smithi177:> rm -rf -- /home/ubuntu/cephtest/admin_socket_client.0 2024-09-13T11:22:28.617 DEBUG:teuthology.parallel:result is None 2024-09-13T11:22:28.617 DEBUG:teuthology.run_tasks:Unwinding manager radosbench 2024-09-13T11:22:28.627 INFO:tasks.radosbench:joining radosbench (timing out after 4800s) 2024-09-13T11:22:28.627 INFO:teuthology.orchestra.run:waiting for 4800 2024-09-13T11:22:28.643 INFO:tasks.ceph.osd.14.smithi162.stderr:2024-09-13T11:22:28.640+0000 7f235c0db640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26319) UID: 0 2024-09-13T11:22:28.643 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-13T11:22:28.643 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [], 2024-09-13T11:22:28.643 INFO:teuthology.orchestra.run.smithi162.stdout: "complaint_time": 30, 2024-09-13T11:22:28.644 INFO:teuthology.orchestra.run.smithi162.stdout: "num_blocked_ops": 0 2024-09-13T11:22:28.644 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-13T11:22:28.658 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-09-13T11:22:28.743 INFO:tasks.ceph.osd.15.smithi177.stderr:2024-09-13T11:22:28.741+0000 7f9d6ce77640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26578) UID: 0 2024-09-13T11:22:28.827 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-13T11:22:28.828 INFO:teuthology.orchestra.run.smithi162.stdout: "size": 20, 2024-09-13T11:22:28.828 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 600, 2024-09-13T11:22:28.828 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [ 2024-09-13T11:22:28.828 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.828 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:5 3.3 3:c820cfec:::benchmark_data_smithi177_31116_object4:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-09-13T11:22:28.828 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:22:27.001508+0000", 2024-09-13T11:22:28.828 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 1.824239304, 2024-09-13T11:22:28.828 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0089235370000000005, 2024-09-13T11:22:28.828 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:22:28.828 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:22:28.828 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:28.829 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:22:28.829 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:22:28.829 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:22:28.829 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 5 2024-09-13T11:22:28.829 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.829 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:22:28.829 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.829 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:22:28.829 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.001508+0000", 2024-09-13T11:22:28.829 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:22:28.829 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.829 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.829 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:22:28.830 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.001508+0000", 2024-09-13T11:22:28.830 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:22:28.830 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.830 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.830 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:22:28.830 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.001512+0000", 2024-09-13T11:22:28.830 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.7730000000000001e-06 2024-09-13T11:22:28.830 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.830 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.830 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:22:28.830 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.001936+0000", 2024-09-13T11:22:28.830 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00042364399999999998 2024-09-13T11:22:28.830 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.831 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.831 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:22:28.831 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.001967+0000", 2024-09-13T11:22:28.831 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.0747999999999998e-05 2024-09-13T11:22:28.831 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.831 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.831 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:22:28.831 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.001984+0000", 2024-09-13T11:22:28.831 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.7028999999999999e-05 2024-09-13T11:22:28.831 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.831 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.831 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:22:28.831 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.002116+0000", 2024-09-13T11:22:28.832 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000132192 2024-09-13T11:22:28.832 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.832 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.832 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:22:28.832 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.002482+0000", 2024-09-13T11:22:28.832 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00036674899999999998 2024-09-13T11:22:28.832 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.832 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.832 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 7", 2024-09-13T11:22:28.832 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.002714+0000", 2024-09-13T11:22:28.832 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000231759 2024-09-13T11:22:28.832 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.832 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.832 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:22:28.833 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.006493+0000", 2024-09-13T11:22:28.833 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0037787250000000001 2024-09-13T11:22:28.833 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.833 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.833 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:22:28.833 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.010167+0000", 2024-09-13T11:22:28.833 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.003674053 2024-09-13T11:22:28.833 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.833 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.833 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:22:28.833 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.010299+0000", 2024-09-13T11:22:28.833 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00013196599999999999 2024-09-13T11:22:28.834 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.834 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.834 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:22:28.834 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.010432+0000", 2024-09-13T11:22:28.834 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00013289899999999999 2024-09-13T11:22:28.834 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:22:28.834 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:22:28.834 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:22:28.835 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.835 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.835 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop_reply(client.4665.0:5 3.3 e31/30 ondisk, result = 0)", 2024-09-13T11:22:28.835 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:22:27.006210+0000", 2024-09-13T11:22:28.835 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 1.8195375249999999, 2024-09-13T11:22:28.835 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00030806099999999999, 2024-09-13T11:22:28.835 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:22:28.835 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:22:28.835 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:22:28.835 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:22:28.835 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.835 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:22:28.835 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.006210+0000", 2024-09-13T11:22:28.835 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:22:28.836 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.836 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.836 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:22:28.836 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.006210+0000", 2024-09-13T11:22:28.836 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:22:28.836 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.836 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.836 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:22:28.836 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.006214+0000", 2024-09-13T11:22:28.836 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.4230000000000001e-06 2024-09-13T11:22:28.836 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.836 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.836 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:22:28.836 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.006228+0000", 2024-09-13T11:22:28.837 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4195999999999999e-05 2024-09-13T11:22:28.837 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.837 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.837 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:22:28.837 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.006304+0000", 2024-09-13T11:22:28.837 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.6202e-05 2024-09-13T11:22:28.837 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.837 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.837 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:22:28.837 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.006318+0000", 2024-09-13T11:22:28.837 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4005000000000001e-05 2024-09-13T11:22:28.837 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.837 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.837 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:22:28.838 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.006449+0000", 2024-09-13T11:22:28.838 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00013114600000000001 2024-09-13T11:22:28.838 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.838 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.838 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:22:28.838 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.006474+0000", 2024-09-13T11:22:28.838 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.5324999999999999e-05 2024-09-13T11:22:28.838 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.838 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.838 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:22:28.838 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.006518+0000", 2024-09-13T11:22:28.838 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.3764000000000003e-05 2024-09-13T11:22:28.838 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:22:28.839 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:22:28.839 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:22:28.839 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.839 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.839 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:18 3.3 3:c843b42e:::benchmark_data_smithi177_31116_object17:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-09-13T11:22:28.839 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:22:27.008051+0000", 2024-09-13T11:22:28.839 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 1.817696712, 2024-09-13T11:22:28.839 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0056804710000000003, 2024-09-13T11:22:28.839 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:22:28.839 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:22:28.839 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:28.839 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:22:28.839 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:22:28.839 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:22:28.840 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 18 2024-09-13T11:22:28.840 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.840 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:22:28.840 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.840 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:22:28.840 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.008051+0000", 2024-09-13T11:22:28.840 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:22:28.840 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.840 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.840 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:22:28.840 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.008051+0000", 2024-09-13T11:22:28.840 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:22:28.840 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.841 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.841 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:22:28.841 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.008055+0000", 2024-09-13T11:22:28.841 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.1300000000000003e-06 2024-09-13T11:22:28.841 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.841 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.841 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:22:28.841 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.008164+0000", 2024-09-13T11:22:28.841 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000108663 2024-09-13T11:22:28.841 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.841 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.841 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:22:28.841 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.008201+0000", 2024-09-13T11:22:28.841 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.7701999999999999e-05 2024-09-13T11:22:28.842 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.842 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.842 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:22:28.842 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.008223+0000", 2024-09-13T11:22:28.842 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.1974999999999999e-05 2024-09-13T11:22:28.842 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.842 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.842 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:22:28.842 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.008415+0000", 2024-09-13T11:22:28.842 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000191285 2024-09-13T11:22:28.842 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.842 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.842 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:22:28.842 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.008749+0000", 2024-09-13T11:22:28.843 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00033396000000000001 2024-09-13T11:22:28.843 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.843 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.843 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 7", 2024-09-13T11:22:28.843 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.009117+0000", 2024-09-13T11:22:28.843 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00036870799999999998 2024-09-13T11:22:28.843 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.843 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.843 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:22:28.843 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.011237+0000", 2024-09-13T11:22:28.843 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.002119957 2024-09-13T11:22:28.843 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.843 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.843 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:22:28.844 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.013431+0000", 2024-09-13T11:22:28.844 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0021933289999999999 2024-09-13T11:22:28.844 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.844 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.844 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:22:28.844 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.013602+0000", 2024-09-13T11:22:28.844 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000171616 2024-09-13T11:22:28.844 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.844 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.844 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:22:28.844 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.013731+0000", 2024-09-13T11:22:28.844 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00012914600000000001 2024-09-13T11:22:28.844 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:22:28.845 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:22:28.845 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:22:28.845 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.845 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.845 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop_reply(client.4665.0:18 3.3 e32/30 ondisk, result = 0)", 2024-09-13T11:22:28.845 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:22:27.010996+0000", 2024-09-13T11:22:28.845 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 1.8147521310000001, 2024-09-13T11:22:28.845 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00025344499999999998, 2024-09-13T11:22:28.845 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:22:28.845 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:22:28.845 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:22:28.845 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:22:28.845 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.846 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:22:28.846 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.010996+0000", 2024-09-13T11:22:28.846 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:22:28.846 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.846 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.846 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:22:28.846 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.010996+0000", 2024-09-13T11:22:28.846 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:22:28.846 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.846 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.846 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:22:28.846 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.011006+0000", 2024-09-13T11:22:28.846 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0662000000000001e-05 2024-09-13T11:22:28.846 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.847 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.847 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:22:28.847 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.011016+0000", 2024-09-13T11:22:28.847 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0044000000000001e-05 2024-09-13T11:22:28.847 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.847 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.847 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:22:28.847 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.011089+0000", 2024-09-13T11:22:28.847 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.3004999999999996e-05 2024-09-13T11:22:28.847 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.847 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.847 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:22:28.847 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.011102+0000", 2024-09-13T11:22:28.848 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3213e-05 2024-09-13T11:22:28.848 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.848 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.848 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:22:28.848 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.011213+0000", 2024-09-13T11:22:28.848 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00011056300000000001 2024-09-13T11:22:28.848 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.848 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.848 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:22:28.848 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.011227+0000", 2024-09-13T11:22:28.848 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3679000000000001e-05 2024-09-13T11:22:28.848 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.848 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.849 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:22:28.849 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.011249+0000", 2024-09-13T11:22:28.849 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.2279000000000001e-05 2024-09-13T11:22:28.849 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:22:28.849 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:22:28.849 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:22:28.849 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.849 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.849 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:36 3.3 3:cda5fb1c:::benchmark_data_smithi177_31116_object35:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-09-13T11:22:28.849 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:22:27.016028+0000", 2024-09-13T11:22:28.849 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 1.809719922, 2024-09-13T11:22:28.849 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0041210709999999996, 2024-09-13T11:22:28.849 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:22:28.850 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:22:28.850 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:28.850 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:22:28.850 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:22:28.850 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:22:28.850 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 36 2024-09-13T11:22:28.850 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.850 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:22:28.850 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.850 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:22:28.850 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.016028+0000", 2024-09-13T11:22:28.850 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:22:28.851 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.851 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.851 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:22:28.851 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.016028+0000", 2024-09-13T11:22:28.851 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:22:28.851 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.851 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.851 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:22:28.851 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.016032+0000", 2024-09-13T11:22:28.851 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.0770000000000001e-06 2024-09-13T11:22:28.851 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.851 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.851 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:22:28.851 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.016122+0000", 2024-09-13T11:22:28.852 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.0297999999999995e-05 2024-09-13T11:22:28.852 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.852 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.852 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:22:28.852 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.016163+0000", 2024-09-13T11:22:28.852 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.0630000000000002e-05 2024-09-13T11:22:28.852 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.852 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.852 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:22:28.852 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.016179+0000", 2024-09-13T11:22:28.852 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5798999999999999e-05 2024-09-13T11:22:28.852 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.853 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.853 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:22:28.853 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.016327+0000", 2024-09-13T11:22:28.853 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000148009 2024-09-13T11:22:28.853 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.853 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.853 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:22:28.853 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.016703+0000", 2024-09-13T11:22:28.853 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000376824 2024-09-13T11:22:28.853 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.853 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.853 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 7", 2024-09-13T11:22:28.853 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.016951+0000", 2024-09-13T11:22:28.853 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000247641 2024-09-13T11:22:28.854 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.854 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.854 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:22:28.854 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.018415+0000", 2024-09-13T11:22:28.854 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001464443 2024-09-13T11:22:28.854 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.854 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.854 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:22:28.854 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.019894+0000", 2024-09-13T11:22:28.854 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0014790020000000001 2024-09-13T11:22:28.854 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.854 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.854 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:22:28.854 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.020034+0000", 2024-09-13T11:22:28.854 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00013971099999999999 2024-09-13T11:22:28.855 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.855 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.855 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:22:28.855 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.020149+0000", 2024-09-13T11:22:28.855 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00011463699999999999 2024-09-13T11:22:28.855 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:22:28.855 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:22:28.855 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:22:28.855 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.855 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.855 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop_reply(client.4665.0:36 3.3 e32/30 ondisk, result = 0)", 2024-09-13T11:22:28.855 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:22:27.018210+0000", 2024-09-13T11:22:28.856 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 1.807537245, 2024-09-13T11:22:28.856 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00021688, 2024-09-13T11:22:28.856 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:22:28.856 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:22:28.856 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:22:28.856 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:22:28.856 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.856 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:22:28.856 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.018210+0000", 2024-09-13T11:22:28.856 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:22:28.856 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.856 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.856 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:22:28.857 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.018210+0000", 2024-09-13T11:22:28.857 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:22:28.857 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.857 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.857 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:22:28.857 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.018213+0000", 2024-09-13T11:22:28.857 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.0199999999999999e-06 2024-09-13T11:22:28.857 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.857 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.857 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:22:28.857 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.018223+0000", 2024-09-13T11:22:28.857 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.482e-06 2024-09-13T11:22:28.857 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.858 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.858 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:22:28.858 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.018280+0000", 2024-09-13T11:22:28.858 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.7210000000000003e-05 2024-09-13T11:22:28.858 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.858 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.858 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:22:28.858 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.018293+0000", 2024-09-13T11:22:28.858 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3043e-05 2024-09-13T11:22:28.858 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.858 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.858 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:22:28.858 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.018390+0000", 2024-09-13T11:22:28.858 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.7046999999999994e-05 2024-09-13T11:22:28.858 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.859 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.859 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:22:28.859 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.018403+0000", 2024-09-13T11:22:28.859 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2523e-05 2024-09-13T11:22:28.859 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.859 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.859 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:22:28.859 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.018427+0000", 2024-09-13T11:22:28.859 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.4555000000000001e-05 2024-09-13T11:22:28.859 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:22:28.859 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:22:28.859 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:22:28.859 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.859 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.859 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:63 3.3 3:c10de739:::benchmark_data_smithi177_31116_object62:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-09-13T11:22:28.860 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:22:27.030813+0000", 2024-09-13T11:22:28.860 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 1.7949348949999999, 2024-09-13T11:22:28.860 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0025455809999999999, 2024-09-13T11:22:28.860 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:22:28.860 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:22:28.860 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:28.860 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:22:28.860 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:22:28.860 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:22:28.860 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 63 2024-09-13T11:22:28.860 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.860 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:22:28.860 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.860 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:22:28.861 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.030813+0000", 2024-09-13T11:22:28.861 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:22:28.861 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.861 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.861 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:22:28.861 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.030813+0000", 2024-09-13T11:22:28.861 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:22:28.861 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.861 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.861 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:22:28.861 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.030816+0000", 2024-09-13T11:22:28.861 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.563e-06 2024-09-13T11:22:28.861 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.862 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.862 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:22:28.862 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.030871+0000", 2024-09-13T11:22:28.862 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.4243999999999998e-05 2024-09-13T11:22:28.862 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.862 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.862 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:22:28.862 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.030900+0000", 2024-09-13T11:22:28.862 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.9198000000000001e-05 2024-09-13T11:22:28.862 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.862 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.862 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:22:28.862 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.030914+0000", 2024-09-13T11:22:28.863 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4329e-05 2024-09-13T11:22:28.863 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.863 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.863 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:22:28.863 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.031013+0000", 2024-09-13T11:22:28.863 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.8926999999999997e-05 2024-09-13T11:22:28.863 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.863 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.863 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:22:28.863 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.031213+0000", 2024-09-13T11:22:28.863 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00020036500000000001 2024-09-13T11:22:28.863 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.863 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.864 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 7", 2024-09-13T11:22:28.864 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.031497+0000", 2024-09-13T11:22:28.864 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000283923 2024-09-13T11:22:28.864 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.864 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.864 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:22:28.864 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.033152+0000", 2024-09-13T11:22:28.864 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0016544179999999999 2024-09-13T11:22:28.864 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.864 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.864 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:22:28.864 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.033204+0000", 2024-09-13T11:22:28.864 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.2639999999999997e-05 2024-09-13T11:22:28.864 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.865 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.865 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:22:28.865 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.033293+0000", 2024-09-13T11:22:28.865 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.8242e-05 2024-09-13T11:22:28.865 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.865 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.865 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:22:28.865 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.033358+0000", 2024-09-13T11:22:28.865 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.5732e-05 2024-09-13T11:22:28.865 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:22:28.865 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:22:28.865 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:22:28.865 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.866 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.866 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop_reply(client.4665.0:63 3.3 e32/30 ondisk, result = 0)", 2024-09-13T11:22:28.866 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:22:27.032941+0000", 2024-09-13T11:22:28.866 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 1.792806602, 2024-09-13T11:22:28.866 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00023613900000000001, 2024-09-13T11:22:28.866 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:22:28.866 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:22:28.866 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:22:28.866 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:22:28.866 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.866 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:22:28.866 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.032941+0000", 2024-09-13T11:22:28.866 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:22:28.866 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.867 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.867 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:22:28.867 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.032941+0000", 2024-09-13T11:22:28.867 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:22:28.867 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.867 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.867 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:22:28.867 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.032944+0000", 2024-09-13T11:22:28.867 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.1669999999999999e-06 2024-09-13T11:22:28.867 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.867 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.867 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:22:28.867 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.032954+0000", 2024-09-13T11:22:28.867 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.2790000000000004e-06 2024-09-13T11:22:28.868 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.868 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.868 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:22:28.868 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.033011+0000", 2024-09-13T11:22:28.868 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.7646999999999999e-05 2024-09-13T11:22:28.868 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.868 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.868 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:22:28.868 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.033025+0000", 2024-09-13T11:22:28.868 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4171999999999999e-05 2024-09-13T11:22:28.868 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.868 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.868 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:22:28.868 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.033132+0000", 2024-09-13T11:22:28.869 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000106721 2024-09-13T11:22:28.869 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.869 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.869 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:22:28.869 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.033143+0000", 2024-09-13T11:22:28.869 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1449e-05 2024-09-13T11:22:28.869 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.869 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.869 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:22:28.869 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.033177+0000", 2024-09-13T11:22:28.869 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.3704000000000001e-05 2024-09-13T11:22:28.869 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:22:28.869 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:22:28.869 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:22:28.870 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.870 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.870 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:77 3.3 3:c9b6c4f6:::benchmark_data_smithi177_31116_object76:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-09-13T11:22:28.870 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:22:27.047173+0000", 2024-09-13T11:22:28.870 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 1.7785743680000001, 2024-09-13T11:22:28.870 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0023565729999999998, 2024-09-13T11:22:28.870 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:22:28.870 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:22:28.870 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:28.870 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:22:28.870 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:22:28.870 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:22:28.870 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 77 2024-09-13T11:22:28.870 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.871 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:22:28.871 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.871 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:22:28.871 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.047173+0000", 2024-09-13T11:22:28.871 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:22:28.871 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.871 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.871 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:22:28.871 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.047173+0000", 2024-09-13T11:22:28.871 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:22:28.871 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.871 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.871 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:22:28.872 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.047177+0000", 2024-09-13T11:22:28.872 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.393e-06 2024-09-13T11:22:28.872 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.872 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.872 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:22:28.872 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.047282+0000", 2024-09-13T11:22:28.872 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000104867 2024-09-13T11:22:28.872 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.872 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.872 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:22:28.872 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.047310+0000", 2024-09-13T11:22:28.872 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.8034999999999998e-05 2024-09-13T11:22:28.872 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.873 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.873 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:22:28.873 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.047325+0000", 2024-09-13T11:22:28.873 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5122e-05 2024-09-13T11:22:28.873 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.873 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.873 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:22:28.873 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.047465+0000", 2024-09-13T11:22:28.873 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000140149 2024-09-13T11:22:28.873 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.873 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.873 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:22:28.873 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.047673+0000", 2024-09-13T11:22:28.873 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000208464 2024-09-13T11:22:28.874 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.874 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.874 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 7", 2024-09-13T11:22:28.874 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.047911+0000", 2024-09-13T11:22:28.874 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00023797900000000001 2024-09-13T11:22:28.874 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.874 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.874 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:22:28.874 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.049362+0000", 2024-09-13T11:22:28.874 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0014505900000000001 2024-09-13T11:22:28.874 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.874 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.874 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:22:28.875 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.049384+0000", 2024-09-13T11:22:28.875 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.2478999999999999e-05 2024-09-13T11:22:28.875 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.875 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.875 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:22:28.875 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.049464+0000", 2024-09-13T11:22:28.875 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.0129000000000005e-05 2024-09-13T11:22:28.875 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.875 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.875 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:22:28.875 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.049530+0000", 2024-09-13T11:22:28.875 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.5365999999999997e-05 2024-09-13T11:22:28.875 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:22:28.875 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:22:28.876 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:22:28.876 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.876 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.876 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop_reply(client.4665.0:77 3.3 e32/30 ondisk, result = 0)", 2024-09-13T11:22:28.876 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:22:27.049147+0000", 2024-09-13T11:22:28.876 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 1.7766010319999999, 2024-09-13T11:22:28.876 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000226579, 2024-09-13T11:22:28.876 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:22:28.876 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:22:28.876 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:22:28.876 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:22:28.876 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.877 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:22:28.877 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.049147+0000", 2024-09-13T11:22:28.877 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:22:28.877 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.877 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.877 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:22:28.877 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.049147+0000", 2024-09-13T11:22:28.877 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:22:28.877 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.877 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.877 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:22:28.877 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.049150+0000", 2024-09-13T11:22:28.877 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.9900000000000002e-06 2024-09-13T11:22:28.878 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.878 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.878 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:22:28.878 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.049159+0000", 2024-09-13T11:22:28.878 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.3149999999999998e-06 2024-09-13T11:22:28.878 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.878 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.878 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:22:28.878 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.049223+0000", 2024-09-13T11:22:28.878 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.3767000000000001e-05 2024-09-13T11:22:28.878 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.878 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.878 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:22:28.879 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.049239+0000", 2024-09-13T11:22:28.879 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.6745e-05 2024-09-13T11:22:28.879 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.879 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.879 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:22:28.879 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.049340+0000", 2024-09-13T11:22:28.879 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00010066800000000001 2024-09-13T11:22:28.879 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.879 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.879 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:22:28.879 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.049353+0000", 2024-09-13T11:22:28.879 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2913e-05 2024-09-13T11:22:28.879 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.880 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.880 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:22:28.880 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.049373+0000", 2024-09-13T11:22:28.880 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.0180999999999999e-05 2024-09-13T11:22:28.880 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:22:28.880 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:22:28.880 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:22:28.880 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.880 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.880 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:78 3.3 3:c95c4ad0:::benchmark_data_smithi177_31116_object77:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-09-13T11:22:28.880 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:22:27.049761+0000", 2024-09-13T11:22:28.880 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 1.775986289, 2024-09-13T11:22:28.880 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0050178640000000004, 2024-09-13T11:22:28.881 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:22:28.881 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:22:28.881 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:28.881 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:22:28.881 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:22:28.881 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:22:28.881 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 78 2024-09-13T11:22:28.881 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.881 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:22:28.881 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.881 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:22:28.881 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.049761+0000", 2024-09-13T11:22:28.881 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:22:28.882 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.882 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.882 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:22:28.882 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.049761+0000", 2024-09-13T11:22:28.882 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:22:28.882 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.882 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.882 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:22:28.882 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.049764+0000", 2024-09-13T11:22:28.882 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.526e-06 2024-09-13T11:22:28.882 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.882 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.882 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:22:28.882 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.049838+0000", 2024-09-13T11:22:28.882 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.4270000000000006e-05 2024-09-13T11:22:28.883 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.883 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.883 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:22:28.883 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.049862+0000", 2024-09-13T11:22:28.883 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.3428e-05 2024-09-13T11:22:28.883 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.883 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.883 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:22:28.883 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.049875+0000", 2024-09-13T11:22:28.883 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3073e-05 2024-09-13T11:22:28.883 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.883 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.883 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:22:28.884 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.050039+0000", 2024-09-13T11:22:28.884 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000163957 2024-09-13T11:22:28.884 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.884 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.884 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:22:28.884 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.050232+0000", 2024-09-13T11:22:28.884 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00019368799999999999 2024-09-13T11:22:28.884 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.884 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.884 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 7", 2024-09-13T11:22:28.884 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.050472+0000", 2024-09-13T11:22:28.884 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000239986 2024-09-13T11:22:28.885 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.885 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.885 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:22:28.885 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.051919+0000", 2024-09-13T11:22:28.885 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001446403 2024-09-13T11:22:28.885 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.885 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.885 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:22:28.885 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.054619+0000", 2024-09-13T11:22:28.885 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0027000520000000001 2024-09-13T11:22:28.885 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.885 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.885 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:22:28.885 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.054710+0000", 2024-09-13T11:22:28.885 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.1712000000000006e-05 2024-09-13T11:22:28.886 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.886 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.886 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:22:28.886 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.054779+0000", 2024-09-13T11:22:28.886 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.8769000000000005e-05 2024-09-13T11:22:28.886 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:22:28.886 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:22:28.886 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:22:28.886 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.886 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.886 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop_reply(client.4665.0:78 3.3 e32/30 ondisk, result = 0)", 2024-09-13T11:22:28.886 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:22:27.051708+0000", 2024-09-13T11:22:28.886 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 1.774040142, 2024-09-13T11:22:28.887 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000224644, 2024-09-13T11:22:28.887 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:22:28.887 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:22:28.887 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:22:28.887 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:22:28.887 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.887 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:22:28.887 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.051708+0000", 2024-09-13T11:22:28.887 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:22:28.887 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.887 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.887 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:22:28.887 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.051708+0000", 2024-09-13T11:22:28.887 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:22:28.888 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.888 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.888 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:22:28.888 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.051710+0000", 2024-09-13T11:22:28.888 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.977e-06 2024-09-13T11:22:28.888 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.888 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.888 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:22:28.888 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.051719+0000", 2024-09-13T11:22:28.888 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.9460000000000004e-06 2024-09-13T11:22:28.888 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.888 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.888 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:22:28.888 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.051775+0000", 2024-09-13T11:22:28.889 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.5086000000000001e-05 2024-09-13T11:22:28.889 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.889 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.889 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:22:28.889 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.051787+0000", 2024-09-13T11:22:28.889 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2713e-05 2024-09-13T11:22:28.889 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.889 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.889 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:22:28.889 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.051896+0000", 2024-09-13T11:22:28.889 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000108874 2024-09-13T11:22:28.889 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.889 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.890 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:22:28.890 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.051910+0000", 2024-09-13T11:22:28.890 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3631999999999999e-05 2024-09-13T11:22:28.890 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.890 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.890 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:22:28.890 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.051932+0000", 2024-09-13T11:22:28.890 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.2416e-05 2024-09-13T11:22:28.890 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:22:28.890 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:22:28.890 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:22:28.890 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.890 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.890 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:80 3.3 3:c3d99973:::benchmark_data_smithi177_31116_object79:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-09-13T11:22:28.891 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:22:27.056483+0000", 2024-09-13T11:22:28.891 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 1.7692649469999999, 2024-09-13T11:22:28.891 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0023848369999999999, 2024-09-13T11:22:28.891 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:22:28.891 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:22:28.891 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:28.891 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:22:28.891 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:22:28.891 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:22:28.891 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 80 2024-09-13T11:22:28.891 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.891 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:22:28.891 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.891 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:22:28.891 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.056483+0000", 2024-09-13T11:22:28.892 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:22:28.892 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.892 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.892 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:22:28.892 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.056483+0000", 2024-09-13T11:22:28.892 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:22:28.892 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.892 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.892 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:22:28.892 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.056486+0000", 2024-09-13T11:22:28.892 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.36e-06 2024-09-13T11:22:28.892 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.892 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.892 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:22:28.892 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.056535+0000", 2024-09-13T11:22:28.893 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.8964000000000001e-05 2024-09-13T11:22:28.893 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.893 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.893 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:22:28.893 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.056565+0000", 2024-09-13T11:22:28.893 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.0207999999999999e-05 2024-09-13T11:22:28.893 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.893 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.893 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:22:28.893 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.056580+0000", 2024-09-13T11:22:28.893 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4429e-05 2024-09-13T11:22:28.893 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.893 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.893 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:22:28.893 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.056752+0000", 2024-09-13T11:22:28.894 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00017254299999999999 2024-09-13T11:22:28.894 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.894 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.894 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:22:28.894 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.056961+0000", 2024-09-13T11:22:28.894 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00020884000000000001 2024-09-13T11:22:28.894 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.894 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.894 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 7", 2024-09-13T11:22:28.894 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.057172+0000", 2024-09-13T11:22:28.894 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000211358 2024-09-13T11:22:28.894 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.894 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.894 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:22:28.894 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.058646+0000", 2024-09-13T11:22:28.894 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0014738520000000001 2024-09-13T11:22:28.895 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.895 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.895 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:22:28.895 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.058690+0000", 2024-09-13T11:22:28.895 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.4196999999999997e-05 2024-09-13T11:22:28.895 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.895 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.895 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:22:28.895 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.058803+0000", 2024-09-13T11:22:28.895 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00011254700000000001 2024-09-13T11:22:28.895 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.895 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.895 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:22:28.895 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.058868+0000", 2024-09-13T11:22:28.895 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.4539e-05 2024-09-13T11:22:28.896 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:22:28.896 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:22:28.896 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:22:28.896 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.896 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.896 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop_reply(client.4665.0:80 3.3 e32/30 ondisk, result = 0)", 2024-09-13T11:22:28.896 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:22:27.058405+0000", 2024-09-13T11:22:28.896 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 1.7673421789999999, 2024-09-13T11:22:28.896 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00025281899999999999, 2024-09-13T11:22:28.896 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:22:28.896 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:22:28.896 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:22:28.896 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:22:28.896 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.897 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:22:28.897 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.058405+0000", 2024-09-13T11:22:28.897 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:22:28.897 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.897 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.897 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:22:28.897 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.058405+0000", 2024-09-13T11:22:28.897 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:22:28.897 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.897 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.897 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:22:28.897 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.058408+0000", 2024-09-13T11:22:28.897 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.9869999999999999e-06 2024-09-13T11:22:28.897 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.898 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.898 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:22:28.898 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.058424+0000", 2024-09-13T11:22:28.898 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5889e-05 2024-09-13T11:22:28.898 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.898 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.898 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:22:28.898 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.058505+0000", 2024-09-13T11:22:28.898 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.0704999999999994e-05 2024-09-13T11:22:28.898 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.898 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.898 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:22:28.898 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.058524+0000", 2024-09-13T11:22:28.898 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.9292000000000001e-05 2024-09-13T11:22:28.899 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.899 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.899 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:22:28.899 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.058626+0000", 2024-09-13T11:22:28.899 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000101604 2024-09-13T11:22:28.899 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.899 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.899 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:22:28.899 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.058638+0000", 2024-09-13T11:22:28.899 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1843e-05 2024-09-13T11:22:28.899 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:28.899 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:28.899 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:22:28.899 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.058658+0000", 2024-09-13T11:22:28.899 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.0499000000000001e-05 2024-09-13T11:22:28.900 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:22:28.900 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:22:28.900 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:22:28.900 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:22:28.900 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:22:28.900 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-13T11:22:28.901 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_ops_in_flight 2024-09-13T11:22:28.902 INFO:tasks.ceph.osd.12.smithi001.stderr:2024-09-13T11:22:28.897+0000 7f892c6e6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26314) UID: 0 2024-09-13T11:22:28.995 INFO:tasks.radosbench.radosbench.0.smithi177.stdout: 2 16 87 71 2.21848 0.125 0.00468035 0.0234801 2024-09-13T11:22:29.002 INFO:tasks.ceph.osd.11.smithi177.stderr:2024-09-13T11:22:29.000+0000 7f24792c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26580) UID: 0 2024-09-13T11:22:29.030 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-13T11:22:29.030 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [], 2024-09-13T11:22:29.030 INFO:teuthology.orchestra.run.smithi162.stdout: "num_ops": 0 2024-09-13T11:22:29.030 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-13T11:22:29.044 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_blocked_ops 2024-09-13T11:22:29.103 INFO:tasks.ceph.osd.10.smithi162.stderr:2024-09-13T11:22:29.100+0000 7fb59c9f0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26317) UID: 0 2024-09-13T11:22:29.117 INFO:teuthology.orchestra.run.smithi001.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-09-13T11:22:29.150 INFO:teuthology.orchestra.run.smithi001.stdout:osd.2: {} 2024-09-13T11:22:29.150 INFO:teuthology.orchestra.run.smithi001.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T11:22:29.167 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-13T11:22:29.167 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [], 2024-09-13T11:22:29.168 INFO:teuthology.orchestra.run.smithi162.stdout: "complaint_time": 30, 2024-09-13T11:22:29.168 INFO:teuthology.orchestra.run.smithi162.stdout: "num_blocked_ops": 0 2024-09-13T11:22:29.168 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-13T11:22:29.182 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_historic_ops 2024-09-13T11:22:29.192 INFO:teuthology.orchestra.run.smithi001.stdout:osd.3: {} 2024-09-13T11:22:29.192 INFO:teuthology.orchestra.run.smithi001.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T11:22:29.203 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-13T11:22:29.197+0000 7f6a68775640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26307) UID: 0 2024-09-13T11:22:29.233 INFO:teuthology.orchestra.run.smithi001.stdout:osd.4: {} 2024-09-13T11:22:29.233 INFO:teuthology.orchestra.run.smithi001.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T11:22:29.260 INFO:teuthology.orchestra.run.smithi001.stdout:osd.5: {} 2024-09-13T11:22:29.261 INFO:teuthology.orchestra.run.smithi001.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T11:22:29.295 INFO:teuthology.orchestra.run.smithi001.stdout:osd.6: {} 2024-09-13T11:22:29.296 INFO:teuthology.orchestra.run.smithi001.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T11:22:29.303 INFO:teuthology.orchestra.run.smithi092.stdout:get_pg_num_history pg_num_history pg_num_history(e29 pg_nums {1={23=1},2={25=8}} deleted_pools ) 2024-09-13T11:22:29.304 INFO:teuthology.orchestra.run.smithi092.stdout:Importing pgid 2.4 2024-09-13T11:22:29.304 INFO:tasks.ceph.osd.14.smithi162.stderr:2024-09-13T11:22:29.300+0000 7f235c0db640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26319) UID: 0 2024-09-13T11:22:29.304 INFO:teuthology.orchestra.run.smithi092.stdout:write_pg epoch 29 info 2.4( empty local-lis/les=25/26 n=0 ec=25/25 lis/c=25/25 les/c/f=26/26/0 sis=25) 2024-09-13T11:22:29.308 INFO:teuthology.orchestra.run.smithi092.stdout:Import successful 2024-09-13T11:22:29.330 INFO:teuthology.orchestra.run.smithi001.stdout:osd.7: {} 2024-09-13T11:22:29.330 INFO:teuthology.orchestra.run.smithi001.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T11:22:29.355 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-13T11:22:29.355 INFO:teuthology.orchestra.run.smithi162.stdout: "size": 20, 2024-09-13T11:22:29.355 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 600, 2024-09-13T11:22:29.355 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [ 2024-09-13T11:22:29.356 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:29.356 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4344.0:1 2.3 2:c4c92e5a:::rbd_trash:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e26)", 2024-09-13T11:22:29.356 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:22:14.143485+0000", 2024-09-13T11:22:29.356 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 15.209117818999999, 2024-09-13T11:22:29.356 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0061818159999999997, 2024-09-13T11:22:29.356 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:22:29.356 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:22:29.356 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:29.356 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:22:29.356 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4344", 2024-09-13T11:22:29.356 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.1:0/211923147", 2024-09-13T11:22:29.356 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 1 2024-09-13T11:22:29.356 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:29.357 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:22:29.357 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:29.357 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:22:29.357 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:14.143485+0000", 2024-09-13T11:22:29.357 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:22:29.357 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:29.357 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:29.357 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:22:29.357 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:14.143485+0000", 2024-09-13T11:22:29.357 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:22:29.357 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:29.357 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:29.358 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:22:29.358 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:14.143487+0000", 2024-09-13T11:22:29.358 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.933e-06 2024-09-13T11:22:29.358 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:29.358 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:29.358 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:22:29.358 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:14.143496+0000", 2024-09-13T11:22:29.358 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.1929999999999999e-06 2024-09-13T11:22:29.358 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:29.358 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:29.358 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:22:29.358 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:14.143511+0000", 2024-09-13T11:22:29.358 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4995e-05 2024-09-13T11:22:29.359 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:29.359 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:29.359 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:22:29.359 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:14.143523+0000", 2024-09-13T11:22:29.359 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1416999999999999e-05 2024-09-13T11:22:29.359 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:29.359 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:29.359 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:22:29.359 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:14.143579+0000", 2024-09-13T11:22:29.359 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.6598999999999998e-05 2024-09-13T11:22:29.359 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:29.359 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:29.359 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-09-13T11:22:29.359 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:14.143586+0000", 2024-09-13T11:22:29.360 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.8909999999999998e-06 2024-09-13T11:22:29.360 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:29.360 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:29.360 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:22:29.360 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:14.146789+0000", 2024-09-13T11:22:29.360 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0032031350000000002 2024-09-13T11:22:29.360 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:29.360 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:29.360 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:22:29.360 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:14.147054+0000", 2024-09-13T11:22:29.360 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00026497000000000001 2024-09-13T11:22:29.360 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:29.360 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:29.361 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 15", 2024-09-13T11:22:29.361 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:14.147236+0000", 2024-09-13T11:22:29.361 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000181499 2024-09-13T11:22:29.361 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:29.361 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:29.361 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:22:29.361 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:14.148147+0000", 2024-09-13T11:22:29.361 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00091084800000000002 2024-09-13T11:22:29.361 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:29.361 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:29.361 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:22:29.361 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:14.149588+0000", 2024-09-13T11:22:29.361 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001441026 2024-09-13T11:22:29.361 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:29.362 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:29.362 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:22:29.362 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:14.149638+0000", 2024-09-13T11:22:29.362 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.0006000000000002e-05 2024-09-13T11:22:29.362 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:29.362 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:29.362 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:22:29.362 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:14.149667+0000", 2024-09-13T11:22:29.362 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.9303999999999999e-05 2024-09-13T11:22:29.362 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:22:29.362 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:22:29.362 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:22:29.362 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:29.363 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:29.363 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:2 3.2 3:438ada23:::benchmark_data_smithi177_31116_object1:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-09-13T11:22:29.363 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:22:27.001682+0000", 2024-09-13T11:22:29.363 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 2.3509209329999998, 2024-09-13T11:22:29.363 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.012911389000000001, 2024-09-13T11:22:29.363 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:22:29.363 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:22:29.363 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:29.363 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:22:29.363 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:22:29.363 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:22:29.363 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 2 2024-09-13T11:22:29.363 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:29.364 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:22:29.364 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:29.364 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:22:29.364 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.001682+0000", 2024-09-13T11:22:29.364 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:22:29.364 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:29.364 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:29.364 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:22:29.364 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.001682+0000", 2024-09-13T11:22:29.364 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:22:29.364 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:29.364 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:29.365 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:22:29.365 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.001688+0000", 2024-09-13T11:22:29.365 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.7830000000000004e-06 2024-09-13T11:22:29.365 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:29.365 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:29.365 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:22:29.365 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.002086+0000", 2024-09-13T11:22:29.365 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00039873000000000003 2024-09-13T11:22:29.365 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:29.365 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:29.365 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:22:29.365 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.002126+0000", 2024-09-13T11:22:29.365 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.9211000000000001e-05 2024-09-13T11:22:29.365 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:29.366 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:29.366 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:22:29.366 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.002149+0000", 2024-09-13T11:22:29.366 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.3431e-05 2024-09-13T11:22:29.366 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:29.366 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:29.366 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:22:29.366 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.002334+0000", 2024-09-13T11:22:29.366 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000184599 2024-09-13T11:22:29.366 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:29.366 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:29.366 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op must wait for map", 2024-09-13T11:22:29.366 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.002350+0000", 2024-09-13T11:22:29.367 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5979000000000001e-05 2024-09-13T11:22:29.367 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:29.367 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:29.367 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:22:29.367 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.008094+0000", 2024-09-13T11:22:29.367 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0057442869999999998 2024-09-13T11:22:29.367 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:29.367 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:29.367 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:22:29.367 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.008331+0000", 2024-09-13T11:22:29.367 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00023699300000000001 2024-09-13T11:22:29.367 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:29.367 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:29.368 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 12", 2024-09-13T11:22:29.368 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.008608+0000", 2024-09-13T11:22:29.368 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00027721400000000002 2024-09-13T11:22:29.368 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:29.368 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:29.368 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:22:29.368 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.011753+0000", 2024-09-13T11:22:29.368 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0031443090000000001 2024-09-13T11:22:29.368 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:29.368 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:29.368 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:22:29.368 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.014437+0000", 2024-09-13T11:22:29.368 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0026848990000000001 2024-09-13T11:22:29.369 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:29.369 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:29.369 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:22:29.369 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.014524+0000", 2024-09-13T11:22:29.369 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.6821000000000001e-05 2024-09-13T11:22:29.369 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:29.369 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:29.369 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:22:29.369 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.014593+0000", 2024-09-13T11:22:29.369 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.9133000000000004e-05 2024-09-13T11:22:29.369 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:22:29.369 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:22:29.369 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:22:29.370 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:29.370 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:29.370 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.4665.0:8 3.6 e32/30 3:64ae72f1:::benchmark_data_smithi177_31116_object7:head v 32'1, mlcod=0'0)", 2024-09-13T11:22:29.370 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:22:27.003524+0000", 2024-09-13T11:22:29.370 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 2.349078703, 2024-09-13T11:22:29.370 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0051300069999999998, 2024-09-13T11:22:29.370 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:22:29.370 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:22:29.370 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:29.370 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:22:29.370 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:29.370 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:22:29.370 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.003524+0000", 2024-09-13T11:22:29.370 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:22:29.371 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:29.371 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:29.371 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:22:29.371 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.003524+0000", 2024-09-13T11:22:29.371 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:22:29.371 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:29.371 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:29.371 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:22:29.371 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.003526+0000", 2024-09-13T11:22:29.371 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4929999999999999e-06 2024-09-13T11:22:29.371 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:29.371 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:29.372 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:22:29.372 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.003745+0000", 2024-09-13T11:22:29.372 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00021966100000000001 2024-09-13T11:22:29.372 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:29.372 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:29.372 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:22:29.372 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.003839+0000", 2024-09-13T11:22:29.372 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.4036999999999994e-05 2024-09-13T11:22:29.372 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:29.372 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:29.372 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:22:29.372 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.003862+0000", 2024-09-13T11:22:29.372 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.3082e-05 2024-09-13T11:22:29.372 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:29.372 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:29.373 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:22:29.373 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.004005+0000", 2024-09-13T11:22:29.373 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000142812 2024-09-13T11:22:29.373 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:29.373 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:29.373 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:22:29.373 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.004049+0000", 2024-09-13T11:22:29.373 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.3571e-05 2024-09-13T11:22:29.373 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:29.373 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:29.373 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:22:29.373 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.008542+0000", 2024-09-13T11:22:29.373 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0044930910000000003 2024-09-13T11:22:29.373 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:29.373 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:29.374 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:22:29.374 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.008654+0000", 2024-09-13T11:22:29.374 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00011226 2024-09-13T11:22:29.374 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:22:29.374 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:22:29.374 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:22:29.374 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:29.374 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:29.374 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.4665.0:15 3.6 e32/30 3:6288aaff:::benchmark_data_smithi177_31116_object14:head v 32'2, mlcod=0'0)", 2024-09-13T11:22:29.374 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:22:27.004496+0000", 2024-09-13T11:22:29.374 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 2.3481065980000002, 2024-09-13T11:22:29.374 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0042476270000000003, 2024-09-13T11:22:29.374 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:22:29.374 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:22:29.375 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:29.375 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:22:29.375 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:29.375 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:22:29.375 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.004496+0000", 2024-09-13T11:22:29.375 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:22:29.375 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:29.375 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:29.375 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:22:29.375 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.004496+0000", 2024-09-13T11:22:29.375 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:22:29.375 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:29.375 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:29.375 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:22:29.376 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.004507+0000", 2024-09-13T11:22:29.376 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.047e-05 2024-09-13T11:22:29.376 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:29.376 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:29.376 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:22:29.376 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.004664+0000", 2024-09-13T11:22:29.376 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00015729000000000001 2024-09-13T11:22:29.376 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:29.376 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:29.376 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:22:29.376 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.004749+0000", 2024-09-13T11:22:29.376 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.5272000000000006e-05 2024-09-13T11:22:29.376 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:29.376 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:29.377 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:22:29.377 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.004767+0000", 2024-09-13T11:22:29.377 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.7832000000000001e-05 2024-09-13T11:22:29.377 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:29.377 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:29.377 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:22:29.377 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.004948+0000", 2024-09-13T11:22:29.377 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00018096900000000001 2024-09-13T11:22:29.377 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:29.377 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:29.377 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:22:29.377 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.005004+0000", 2024-09-13T11:22:29.377 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.6137000000000001e-05 2024-09-13T11:22:29.378 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:29.378 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:29.378 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:22:29.378 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.008663+0000", 2024-09-13T11:22:29.378 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0036589750000000001 2024-09-13T11:22:29.378 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:29.378 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:29.378 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:22:29.378 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.008744+0000", 2024-09-13T11:22:29.378 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.0681999999999998e-05 2024-09-13T11:22:29.378 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:22:29.378 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:22:29.378 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:22:29.378 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:29.379 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:29.379 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:17 3.2 3:4bd8024b:::benchmark_data_smithi177_31116_object16:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-09-13T11:22:29.379 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:22:27.006937+0000", 2024-09-13T11:22:29.379 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 2.345666144, 2024-09-13T11:22:29.379 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0085759679999999998, 2024-09-13T11:22:29.379 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:22:29.379 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:22:29.379 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:29.379 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:22:29.379 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:22:29.379 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:22:29.379 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 17 2024-09-13T11:22:29.379 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:29.380 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:22:29.380 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:29.380 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:22:29.380 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.006937+0000", 2024-09-13T11:22:29.380 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:22:29.380 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:29.380 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:29.380 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:22:29.380 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.006937+0000", 2024-09-13T11:22:29.380 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:22:29.380 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:29.380 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:29.380 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:22:29.381 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.006941+0000", 2024-09-13T11:22:29.381 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.25e-06 2024-09-13T11:22:29.381 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:29.381 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:29.381 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:22:29.381 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.007103+0000", 2024-09-13T11:22:29.381 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000162251 2024-09-13T11:22:29.381 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:29.381 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:29.381 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:22:29.381 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.007136+0000", 2024-09-13T11:22:29.381 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.2694000000000002e-05 2024-09-13T11:22:29.381 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:29.382 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:29.382 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:22:29.382 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.007156+0000", 2024-09-13T11:22:29.382 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.9585999999999998e-05 2024-09-13T11:22:29.382 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:29.382 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:29.382 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:22:29.382 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.007277+0000", 2024-09-13T11:22:29.382 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000121273 2024-09-13T11:22:29.382 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:29.382 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:29.382 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting_for_map not empty", 2024-09-13T11:22:29.382 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.007285+0000", 2024-09-13T11:22:29.383 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.0860000000000002e-06 2024-09-13T11:22:29.383 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:29.383 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:29.383 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:22:29.383 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.010111+0000", 2024-09-13T11:22:29.383 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0028265310000000002 2024-09-13T11:22:29.383 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:29.383 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:29.383 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:22:29.383 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.010348+0000", 2024-09-13T11:22:29.383 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00023645299999999999 2024-09-13T11:22:29.383 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:29.383 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:29.384 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 12", 2024-09-13T11:22:29.384 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.010754+0000", 2024-09-13T11:22:29.384 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000406458 2024-09-13T11:22:29.384 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:29.384 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:29.384 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:22:29.384 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.012928+0000", 2024-09-13T11:22:29.384 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0021735439999999999 2024-09-13T11:22:29.384 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:29.384 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:29.384 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:22:29.384 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.015262+0000", 2024-09-13T11:22:29.385 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0023336469999999999 2024-09-13T11:22:29.385 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:29.385 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:29.385 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:22:29.385 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.015391+0000", 2024-09-13T11:22:29.385 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00012929900000000001 2024-09-13T11:22:29.385 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:29.385 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:29.385 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:22:29.385 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.015513+0000", 2024-09-13T11:22:29.385 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000121896 2024-09-13T11:22:29.385 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:22:29.385 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:22:29.385 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:22:29.386 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:29.386 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:29.386 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.4665.0:21 3.6 e32/30 3:69e4f35f:::benchmark_data_smithi177_31116_object20:head v 32'3, mlcod=32'1)", 2024-09-13T11:22:29.386 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:22:27.009568+0000", 2024-09-13T11:22:29.386 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 2.343035124, 2024-09-13T11:22:29.386 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0026007949999999999, 2024-09-13T11:22:29.386 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:22:29.386 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:22:29.386 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:29.386 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:22:29.386 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:29.386 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:22:29.386 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.009568+0000", 2024-09-13T11:22:29.387 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:22:29.387 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:29.387 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:29.387 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:22:29.387 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.009568+0000", 2024-09-13T11:22:29.387 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:22:29.387 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:29.387 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:29.387 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:22:29.387 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.009571+0000", 2024-09-13T11:22:29.387 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.8270000000000002e-06 2024-09-13T11:22:29.387 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:29.388 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:29.388 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:22:29.388 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.009691+0000", 2024-09-13T11:22:29.388 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00012037899999999999 2024-09-13T11:22:29.388 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:29.388 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:29.388 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:22:29.388 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.009740+0000", 2024-09-13T11:22:29.388 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.9344000000000001e-05 2024-09-13T11:22:29.388 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:29.388 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:29.388 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:22:29.388 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.009754+0000", 2024-09-13T11:22:29.389 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3366e-05 2024-09-13T11:22:29.389 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:29.389 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:29.389 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:22:29.389 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.009884+0000", 2024-09-13T11:22:29.389 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00012992200000000001 2024-09-13T11:22:29.389 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:29.389 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:29.389 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:22:29.389 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.009929+0000", 2024-09-13T11:22:29.389 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.5223999999999999e-05 2024-09-13T11:22:29.389 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:29.389 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:29.390 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:22:29.390 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.012102+0000", 2024-09-13T11:22:29.390 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0021735270000000002 2024-09-13T11:22:29.390 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:29.390 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:29.390 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:22:29.390 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.012169+0000", 2024-09-13T11:22:29.390 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.6205999999999996e-05 2024-09-13T11:22:29.390 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:22:29.390 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:22:29.390 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:22:29.390 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:29.391 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:29.391 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.4665.0:22 3.6 e32/30 3:6f6b370c:::benchmark_data_smithi177_31116_object21:head v 32'4, mlcod=32'1)", 2024-09-13T11:22:29.391 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:22:27.010057+0000", 2024-09-13T11:22:29.391 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 2.3425454700000001, 2024-09-13T11:22:29.391 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0052449109999999997, 2024-09-13T11:22:29.391 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:22:29.391 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:22:29.391 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:29.391 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:22:29.391 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:29.391 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:22:29.391 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.010057+0000", 2024-09-13T11:22:29.391 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:22:29.392 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:29.392 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:29.392 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:22:29.392 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.010057+0000", 2024-09-13T11:22:29.392 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:22:29.392 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:29.392 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:29.392 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:22:29.392 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.010059+0000", 2024-09-13T11:22:29.392 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.0269999999999998e-06 2024-09-13T11:22:29.392 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:29.392 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:29.392 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:22:29.393 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.010152+0000", 2024-09-13T11:22:29.393 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.2103999999999998e-05 2024-09-13T11:22:29.393 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:29.393 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:29.393 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:22:29.393 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.010203+0000", 2024-09-13T11:22:29.393 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.1384000000000001e-05 2024-09-13T11:22:29.393 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:29.393 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:29.393 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:22:29.393 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.010219+0000", 2024-09-13T11:22:29.393 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.6095000000000001e-05 2024-09-13T11:22:29.393 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:29.393 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:29.394 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:22:29.394 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.010668+0000", 2024-09-13T11:22:29.394 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00044941700000000002 2024-09-13T11:22:29.394 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:29.394 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:29.394 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:22:29.394 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.010723+0000", 2024-09-13T11:22:29.394 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.4447000000000002e-05 2024-09-13T11:22:29.394 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:29.394 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:29.394 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:22:29.394 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.015214+0000", 2024-09-13T11:22:29.394 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0044907280000000003 2024-09-13T11:22:29.394 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:29.395 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:29.395 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:22:29.395 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.015302+0000", 2024-09-13T11:22:29.395 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.8708999999999994e-05 2024-09-13T11:22:29.395 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:22:29.395 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:22:29.395 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:22:29.395 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:29.395 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:29.395 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop_reply(client.4665.0:2 3.2 e32/30 ondisk, result = 0)", 2024-09-13T11:22:29.395 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:22:27.010498+0000", 2024-09-13T11:22:29.395 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 2.342104516, 2024-09-13T11:22:29.396 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001272081, 2024-09-13T11:22:29.396 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:22:29.396 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:22:29.396 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:22:29.396 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:22:29.396 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:29.396 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:22:29.396 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.010498+0000", 2024-09-13T11:22:29.396 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:22:29.396 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:29.396 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:29.396 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:22:29.397 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.010498+0000", 2024-09-13T11:22:29.397 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:22:29.397 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:29.397 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:29.397 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:22:29.397 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.010502+0000", 2024-09-13T11:22:29.397 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.2200000000000001e-06 2024-09-13T11:22:29.397 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:29.397 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:29.397 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:22:29.397 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.010512+0000", 2024-09-13T11:22:29.397 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0006e-05 2024-09-13T11:22:29.397 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:29.398 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:29.398 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:22:29.398 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.010572+0000", 2024-09-13T11:22:29.398 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.0733000000000003e-05 2024-09-13T11:22:29.398 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:29.398 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:29.398 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:22:29.398 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.010589+0000", 2024-09-13T11:22:29.398 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.6901999999999998e-05 2024-09-13T11:22:29.398 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:29.398 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:29.398 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:22:29.398 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.011695+0000", 2024-09-13T11:22:29.399 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001105541 2024-09-13T11:22:29.399 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:29.399 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:29.399 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:22:29.399 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.011737+0000", 2024-09-13T11:22:29.399 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.1724000000000002e-05 2024-09-13T11:22:29.399 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:29.399 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:29.399 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:22:29.399 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.011770+0000", 2024-09-13T11:22:29.399 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.3955000000000003e-05 2024-09-13T11:22:29.399 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:22:29.400 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:22:29.400 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:22:29.400 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:29.400 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:29.400 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.4665.0:24 3.6 e32/30 3:696318df:::benchmark_data_smithi177_31116_object23:head v 32'5, mlcod=32'1)", 2024-09-13T11:22:29.400 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:22:27.010692+0000", 2024-09-13T11:22:29.400 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 2.3419104580000001, 2024-09-13T11:22:29.400 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0046893000000000004, 2024-09-13T11:22:29.400 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:22:29.400 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:22:29.400 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:29.400 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:22:29.400 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:29.401 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:22:29.401 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.010692+0000", 2024-09-13T11:22:29.401 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:22:29.401 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:29.401 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:29.401 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:22:29.401 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.010692+0000", 2024-09-13T11:22:29.401 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:22:29.401 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:29.401 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:29.401 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:22:29.401 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.010695+0000", 2024-09-13T11:22:29.401 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.5000000000000002e-06 2024-09-13T11:22:29.402 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:29.402 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:29.402 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:22:29.402 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.010847+0000", 2024-09-13T11:22:29.402 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00015198800000000001 2024-09-13T11:22:29.402 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:29.402 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:29.402 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:22:29.402 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.010914+0000", 2024-09-13T11:22:29.402 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.7268999999999996e-05 2024-09-13T11:22:29.402 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:29.402 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:29.402 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:22:29.403 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.010931+0000", 2024-09-13T11:22:29.403 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.6492999999999999e-05 2024-09-13T11:22:29.403 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:29.403 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:29.403 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:22:29.403 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.011549+0000", 2024-09-13T11:22:29.403 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00061793899999999997 2024-09-13T11:22:29.403 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:29.403 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:29.403 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:22:29.403 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.011599+0000", 2024-09-13T11:22:29.403 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.0231000000000001e-05 2024-09-13T11:22:29.403 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:29.403 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:29.404 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:22:29.404 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.015315+0000", 2024-09-13T11:22:29.404 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0037164210000000001 2024-09-13T11:22:29.404 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:29.404 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:29.404 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:22:29.404 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.015382+0000", 2024-09-13T11:22:29.404 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.6458999999999995e-05 2024-09-13T11:22:29.404 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:22:29.404 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:22:29.404 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:22:29.404 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:29.404 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:29.404 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.4665.0:20 3.e e32/30 3:794d9a5b:::benchmark_data_smithi177_31116_object19:head v 32'1, mlcod=0'0)", 2024-09-13T11:22:29.405 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:22:27.014769+0000", 2024-09-13T11:22:29.405 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 2.337833984, 2024-09-13T11:22:29.405 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0051222200000000002, 2024-09-13T11:22:29.405 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:22:29.405 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:22:29.405 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:29.405 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:22:29.405 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:29.405 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:22:29.405 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.014769+0000", 2024-09-13T11:22:29.405 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:22:29.405 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:29.406 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:29.406 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:22:29.406 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.014769+0000", 2024-09-13T11:22:29.406 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:22:29.406 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:29.406 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:29.406 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:22:29.406 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.014770+0000", 2024-09-13T11:22:29.406 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.373e-06 2024-09-13T11:22:29.406 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:29.406 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:29.406 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:22:29.406 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.015250+0000", 2024-09-13T11:22:29.407 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00047970800000000002 2024-09-13T11:22:29.407 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:29.407 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:29.407 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:22:29.407 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.015346+0000", 2024-09-13T11:22:29.407 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.5784000000000003e-05 2024-09-13T11:22:29.407 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:29.407 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:29.407 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:22:29.407 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.015359+0000", 2024-09-13T11:22:29.407 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3259e-05 2024-09-13T11:22:29.407 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:29.408 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:29.408 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:22:29.408 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.015463+0000", 2024-09-13T11:22:29.408 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00010442699999999999 2024-09-13T11:22:29.408 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:29.408 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:29.408 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:22:29.408 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.015568+0000", 2024-09-13T11:22:29.408 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000104994 2024-09-13T11:22:29.408 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:29.408 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:29.408 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:22:29.408 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.019795+0000", 2024-09-13T11:22:29.409 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0042267040000000004 2024-09-13T11:22:29.409 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:29.409 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:29.409 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:22:29.409 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.019891+0000", 2024-09-13T11:22:29.409 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.5971000000000007e-05 2024-09-13T11:22:29.409 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:22:29.409 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:22:29.409 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:22:29.409 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:29.409 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:29.409 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.4665.0:31 3.6 e32/30 3:6ecddb78:::benchmark_data_smithi177_31116_object30:head v 32'6, mlcod=32'2)", 2024-09-13T11:22:29.409 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:22:27.014966+0000", 2024-09-13T11:22:29.410 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 2.3376367789999999, 2024-09-13T11:22:29.410 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0040580900000000003, 2024-09-13T11:22:29.410 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:22:29.410 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:22:29.410 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:29.410 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:22:29.410 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:29.410 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:22:29.410 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.014966+0000", 2024-09-13T11:22:29.410 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:22:29.410 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:29.410 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:29.411 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:22:29.411 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.014966+0000", 2024-09-13T11:22:29.411 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:22:29.411 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:29.411 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:29.411 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:22:29.411 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.014968+0000", 2024-09-13T11:22:29.411 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.2560000000000001e-06 2024-09-13T11:22:29.411 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:29.411 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:29.411 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:22:29.411 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.015151+0000", 2024-09-13T11:22:29.411 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00018277899999999999 2024-09-13T11:22:29.411 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:29.412 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:29.412 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:22:29.412 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.015286+0000", 2024-09-13T11:22:29.412 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00013459600000000001 2024-09-13T11:22:29.412 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:29.412 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:29.412 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:22:29.412 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.015303+0000", 2024-09-13T11:22:29.412 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.7221999999999999e-05 2024-09-13T11:22:29.412 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:29.412 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:29.412 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:22:29.412 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.015495+0000", 2024-09-13T11:22:29.413 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00019227099999999999 2024-09-13T11:22:29.413 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:29.413 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:29.413 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:22:29.413 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.015540+0000", 2024-09-13T11:22:29.413 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.4325e-05 2024-09-13T11:22:29.413 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:29.413 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:29.413 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:22:29.413 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.018915+0000", 2024-09-13T11:22:29.413 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0033753049999999999 2024-09-13T11:22:29.413 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:29.413 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:29.414 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:22:29.414 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.019024+0000", 2024-09-13T11:22:29.414 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000109336 2024-09-13T11:22:29.414 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:22:29.414 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:22:29.414 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:22:29.414 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:29.414 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:29.414 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.4665.0:34 3.6 e32/30 3:6628c15c:::benchmark_data_smithi177_31116_object33:head v 32'7, mlcod=32'4)", 2024-09-13T11:22:29.414 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:22:27.016396+0000", 2024-09-13T11:22:29.414 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 2.336206582, 2024-09-13T11:22:29.414 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0038869410000000001, 2024-09-13T11:22:29.414 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:22:29.415 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:22:29.415 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:29.415 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:22:29.415 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:29.415 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:22:29.415 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.016396+0000", 2024-09-13T11:22:29.415 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:22:29.415 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:29.415 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:29.415 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:22:29.415 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.016396+0000", 2024-09-13T11:22:29.415 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:22:29.415 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:29.415 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:29.416 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:22:29.416 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.016399+0000", 2024-09-13T11:22:29.416 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.0800000000000002e-06 2024-09-13T11:22:29.416 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:29.416 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:29.416 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:22:29.416 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.016455+0000", 2024-09-13T11:22:29.416 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.5316999999999999e-05 2024-09-13T11:22:29.416 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:29.416 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:29.416 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:22:29.416 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.016508+0000", 2024-09-13T11:22:29.416 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.3674000000000001e-05 2024-09-13T11:22:29.416 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:29.417 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:29.417 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:22:29.417 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.016542+0000", 2024-09-13T11:22:29.417 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.3877000000000001e-05 2024-09-13T11:22:29.417 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:29.417 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:29.417 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:22:29.417 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.016670+0000", 2024-09-13T11:22:29.417 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000127479 2024-09-13T11:22:29.417 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:29.417 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:29.417 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:22:29.417 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.016721+0000", 2024-09-13T11:22:29.417 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.1374e-05 2024-09-13T11:22:29.418 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:29.418 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:29.418 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:22:29.418 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.020228+0000", 2024-09-13T11:22:29.418 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0035071870000000002 2024-09-13T11:22:29.418 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:29.418 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:29.418 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:22:29.418 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.020283+0000", 2024-09-13T11:22:29.418 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.4953000000000001e-05 2024-09-13T11:22:29.418 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:22:29.418 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:22:29.418 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:22:29.419 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:29.419 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:29.419 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.4665.0:46 3.e e32/30 3:7d541445:::benchmark_data_smithi177_31116_object45:head v 32'2, mlcod=0'0)", 2024-09-13T11:22:29.419 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:22:27.021132+0000", 2024-09-13T11:22:29.419 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 2.3314705780000002, 2024-09-13T11:22:29.419 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001381431, 2024-09-13T11:22:29.419 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:22:29.419 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:22:29.419 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:29.419 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:22:29.419 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:29.419 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:22:29.419 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.021132+0000", 2024-09-13T11:22:29.419 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:22:29.419 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:29.420 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:29.420 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:22:29.420 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.021132+0000", 2024-09-13T11:22:29.420 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:22:29.420 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:29.420 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:29.420 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:22:29.420 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.021135+0000", 2024-09-13T11:22:29.420 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.5730000000000002e-06 2024-09-13T11:22:29.420 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:29.420 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:29.420 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:22:29.420 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.021212+0000", 2024-09-13T11:22:29.420 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.7528999999999996e-05 2024-09-13T11:22:29.421 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:29.421 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:29.421 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:22:29.421 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.021264+0000", 2024-09-13T11:22:29.421 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.1177000000000001e-05 2024-09-13T11:22:29.421 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:29.421 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:29.421 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:22:29.421 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.021278+0000", 2024-09-13T11:22:29.421 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4129e-05 2024-09-13T11:22:29.421 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:29.421 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:29.421 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:22:29.422 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.021360+0000", 2024-09-13T11:22:29.422 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.2729e-05 2024-09-13T11:22:29.422 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:29.422 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:29.422 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:22:29.422 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.021393+0000", 2024-09-13T11:22:29.422 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.2288e-05 2024-09-13T11:22:29.422 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:29.422 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:29.422 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:22:29.422 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.022452+0000", 2024-09-13T11:22:29.422 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00105948 2024-09-13T11:22:29.422 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:29.422 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:29.423 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:22:29.423 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.022514+0000", 2024-09-13T11:22:29.423 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.1525999999999993e-05 2024-09-13T11:22:29.423 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:22:29.423 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:22:29.423 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:22:29.423 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:29.423 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:29.423 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:66 3.2 3:4ab2a1f9:::benchmark_data_smithi177_31116_object65:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-09-13T11:22:29.423 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:22:27.031250+0000", 2024-09-13T11:22:29.423 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 2.3213526189999998, 2024-09-13T11:22:29.423 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0044864989999999997, 2024-09-13T11:22:29.423 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:22:29.423 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:22:29.424 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:29.424 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:22:29.424 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:22:29.424 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:22:29.424 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 66 2024-09-13T11:22:29.424 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:29.424 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:22:29.424 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:29.424 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:22:29.424 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.031250+0000", 2024-09-13T11:22:29.424 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:22:29.424 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:29.424 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:29.424 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:22:29.424 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.031250+0000", 2024-09-13T11:22:29.425 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:22:29.425 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:29.425 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:29.425 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:22:29.425 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.031255+0000", 2024-09-13T11:22:29.425 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.6500000000000004e-06 2024-09-13T11:22:29.425 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:29.425 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:29.425 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:22:29.425 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.031310+0000", 2024-09-13T11:22:29.425 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.5182999999999997e-05 2024-09-13T11:22:29.425 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:29.425 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:29.426 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:22:29.426 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.031340+0000", 2024-09-13T11:22:29.426 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.9519e-05 2024-09-13T11:22:29.426 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:29.426 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:29.426 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:22:29.426 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.031355+0000", 2024-09-13T11:22:29.426 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5009e-05 2024-09-13T11:22:29.426 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:29.426 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:29.426 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:22:29.426 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.031465+0000", 2024-09-13T11:22:29.426 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00010990299999999999 2024-09-13T11:22:29.426 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:29.427 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:29.427 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:22:29.427 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.031721+0000", 2024-09-13T11:22:29.427 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00025666099999999999 2024-09-13T11:22:29.427 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:29.427 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:29.427 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 12", 2024-09-13T11:22:29.427 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.032053+0000", 2024-09-13T11:22:29.427 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00033221999999999999 2024-09-13T11:22:29.427 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:29.427 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:29.427 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:22:29.427 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.033712+0000", 2024-09-13T11:22:29.427 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001658358 2024-09-13T11:22:29.428 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:29.428 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:29.428 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:22:29.428 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.035568+0000", 2024-09-13T11:22:29.428 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0018563360000000001 2024-09-13T11:22:29.428 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:29.428 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:29.428 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:22:29.428 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.035665+0000", 2024-09-13T11:22:29.428 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.6754000000000002e-05 2024-09-13T11:22:29.428 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:29.428 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:29.428 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:22:29.429 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.035737+0000", 2024-09-13T11:22:29.429 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.1905999999999999e-05 2024-09-13T11:22:29.429 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:22:29.429 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:22:29.429 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:22:29.429 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:29.429 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:29.429 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:71 3.2 3:44c580a0:::benchmark_data_smithi177_31116_object70:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-09-13T11:22:29.429 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:22:27.035943+0000", 2024-09-13T11:22:29.430 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 2.3166595920000002, 2024-09-13T11:22:29.430 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.002318821, 2024-09-13T11:22:29.430 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:22:29.430 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:22:29.430 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:29.430 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:22:29.430 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:22:29.430 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:22:29.430 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 71 2024-09-13T11:22:29.430 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:29.430 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:22:29.430 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:29.430 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:22:29.431 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.035943+0000", 2024-09-13T11:22:29.431 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:22:29.431 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:29.431 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:29.431 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:22:29.431 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.035943+0000", 2024-09-13T11:22:29.431 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:22:29.431 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:29.431 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:29.431 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:22:29.431 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.035946+0000", 2024-09-13T11:22:29.431 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.9699999999999999e-06 2024-09-13T11:22:29.431 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:29.432 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:29.432 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:22:29.432 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.036014+0000", 2024-09-13T11:22:29.432 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.7982000000000001e-05 2024-09-13T11:22:29.432 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:29.432 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:29.432 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:22:29.432 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.036044+0000", 2024-09-13T11:22:29.432 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.9442e-05 2024-09-13T11:22:29.432 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:29.432 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:29.432 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:22:29.432 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.036060+0000", 2024-09-13T11:22:29.433 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.6609e-05 2024-09-13T11:22:29.433 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:29.433 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:29.433 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:22:29.433 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.036151+0000", 2024-09-13T11:22:29.433 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.0637000000000007e-05 2024-09-13T11:22:29.433 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:29.433 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:29.433 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:22:29.433 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.036333+0000", 2024-09-13T11:22:29.433 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000182296 2024-09-13T11:22:29.434 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:29.434 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:29.434 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 12", 2024-09-13T11:22:29.434 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.036536+0000", 2024-09-13T11:22:29.434 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00020244800000000001 2024-09-13T11:22:29.434 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:29.434 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:29.434 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:22:29.434 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.037874+0000", 2024-09-13T11:22:29.434 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0013378140000000001 2024-09-13T11:22:29.434 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:29.434 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:29.434 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:22:29.434 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.038118+0000", 2024-09-13T11:22:29.435 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00024411800000000001 2024-09-13T11:22:29.435 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:29.435 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:29.435 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:22:29.435 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.038200+0000", 2024-09-13T11:22:29.435 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.2822000000000001e-05 2024-09-13T11:22:29.435 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:29.435 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:29.435 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:22:29.435 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.038262+0000", 2024-09-13T11:22:29.435 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.1682999999999999e-05 2024-09-13T11:22:29.435 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:22:29.435 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:22:29.435 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:22:29.436 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:29.436 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:29.436 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.4665.0:74 3.6 e32/30 3:65c9aede:::benchmark_data_smithi177_31116_object73:head v 32'8, mlcod=32'6)", 2024-09-13T11:22:29.436 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:22:27.039081+0000", 2024-09-13T11:22:29.436 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 2.3135219669999998, 2024-09-13T11:22:29.436 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001532179, 2024-09-13T11:22:29.436 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:22:29.436 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:22:29.436 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:29.436 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:22:29.436 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:29.436 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:22:29.436 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.039081+0000", 2024-09-13T11:22:29.436 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:22:29.437 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:29.437 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:29.437 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:22:29.437 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.039081+0000", 2024-09-13T11:22:29.437 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:22:29.437 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:29.437 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:29.437 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:22:29.437 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.039084+0000", 2024-09-13T11:22:29.437 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.5499999999999999e-06 2024-09-13T11:22:29.437 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:29.437 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:29.437 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:22:29.437 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.039147+0000", 2024-09-13T11:22:29.438 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.2580000000000001e-05 2024-09-13T11:22:29.438 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:29.438 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:29.438 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:22:29.438 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.039210+0000", 2024-09-13T11:22:29.438 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.2782999999999998e-05 2024-09-13T11:22:29.438 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:29.438 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:29.438 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:22:29.438 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.039223+0000", 2024-09-13T11:22:29.438 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3416e-05 2024-09-13T11:22:29.438 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:29.438 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:29.438 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:22:29.438 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.039337+0000", 2024-09-13T11:22:29.439 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000113393 2024-09-13T11:22:29.439 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:29.439 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:29.439 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:22:29.439 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.039372+0000", 2024-09-13T11:22:29.439 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.5451000000000003e-05 2024-09-13T11:22:29.439 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:29.439 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:29.439 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:22:29.439 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.040528+0000", 2024-09-13T11:22:29.439 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0011557679999999999 2024-09-13T11:22:29.439 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:29.439 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:29.439 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:22:29.440 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.040613+0000", 2024-09-13T11:22:29.440 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.5237999999999999e-05 2024-09-13T11:22:29.440 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:22:29.440 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:22:29.440 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:22:29.440 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:29.440 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:29.440 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:75 3.2 3:415872a1:::benchmark_data_smithi177_31116_object74:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-09-13T11:22:29.440 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:22:27.041206+0000", 2024-09-13T11:22:29.440 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 2.3113970369999999, 2024-09-13T11:22:29.440 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0025752370000000002, 2024-09-13T11:22:29.440 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:22:29.441 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:22:29.441 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:29.441 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:22:29.441 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:22:29.441 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:22:29.441 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 75 2024-09-13T11:22:29.441 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:29.441 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:22:29.441 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:29.441 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:22:29.441 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.041206+0000", 2024-09-13T11:22:29.441 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:22:29.441 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:29.441 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:29.442 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:22:29.442 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.041206+0000", 2024-09-13T11:22:29.442 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:22:29.442 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:29.442 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:29.442 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:22:29.442 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.041208+0000", 2024-09-13T11:22:29.442 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.6199999999999999e-06 2024-09-13T11:22:29.442 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:29.442 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:29.442 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:22:29.442 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.041281+0000", 2024-09-13T11:22:29.443 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.2471999999999993e-05 2024-09-13T11:22:29.443 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:29.443 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:29.443 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:22:29.443 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.041306+0000", 2024-09-13T11:22:29.443 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.5228999999999998e-05 2024-09-13T11:22:29.443 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:29.443 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:29.443 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:22:29.443 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.041321+0000", 2024-09-13T11:22:29.443 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4372999999999999e-05 2024-09-13T11:22:29.443 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:29.443 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:29.444 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:22:29.444 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.041422+0000", 2024-09-13T11:22:29.444 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00010147 2024-09-13T11:22:29.444 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:29.444 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:29.444 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:22:29.444 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.041643+0000", 2024-09-13T11:22:29.444 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000221363 2024-09-13T11:22:29.444 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:29.444 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:29.444 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 12", 2024-09-13T11:22:29.444 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.041861+0000", 2024-09-13T11:22:29.444 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000217314 2024-09-13T11:22:29.444 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:29.445 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:29.445 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:22:29.445 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.043413+0000", 2024-09-13T11:22:29.445 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0015527609999999999 2024-09-13T11:22:29.445 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:29.445 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:29.445 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:22:29.445 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.043634+0000", 2024-09-13T11:22:29.445 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00022012299999999999 2024-09-13T11:22:29.445 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:29.445 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:29.445 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:22:29.445 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.043721+0000", 2024-09-13T11:22:29.445 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.7740999999999999e-05 2024-09-13T11:22:29.446 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:29.446 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:29.446 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:22:29.446 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.043781+0000", 2024-09-13T11:22:29.446 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.9771000000000001e-05 2024-09-13T11:22:29.446 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:22:29.446 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:22:29.446 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:22:29.446 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:29.446 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:29.446 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:82 3.2 3:4daba74e:::benchmark_data_smithi177_31116_object81:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e33)", 2024-09-13T11:22:29.446 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:22:27.950329+0000", 2024-09-13T11:22:29.446 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 1.402274118, 2024-09-13T11:22:29.447 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0041556340000000001, 2024-09-13T11:22:29.447 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:22:29.447 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:22:29.447 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:29.447 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:22:29.447 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:22:29.447 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:22:29.447 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 82 2024-09-13T11:22:29.447 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:29.447 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:22:29.447 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:29.447 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:22:29.447 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.950329+0000", 2024-09-13T11:22:29.448 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:22:29.448 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:29.448 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:29.448 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:22:29.448 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.950329+0000", 2024-09-13T11:22:29.448 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:22:29.448 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:29.448 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:29.448 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:22:29.448 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.950332+0000", 2024-09-13T11:22:29.448 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.5269999999999999e-06 2024-09-13T11:22:29.448 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:29.448 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:29.449 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:22:29.449 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.950503+0000", 2024-09-13T11:22:29.449 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00017101 2024-09-13T11:22:29.449 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:29.449 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:29.449 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:22:29.449 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.950520+0000", 2024-09-13T11:22:29.449 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.6852000000000001e-05 2024-09-13T11:22:29.449 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:29.449 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:29.450 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:22:29.450 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.950529+0000", 2024-09-13T11:22:29.450 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.8980000000000002e-06 2024-09-13T11:22:29.450 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:29.450 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:29.450 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:22:29.450 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.950679+0000", 2024-09-13T11:22:29.450 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000150073 2024-09-13T11:22:29.450 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:29.450 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:29.450 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:22:29.450 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.951050+0000", 2024-09-13T11:22:29.450 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00037121799999999998 2024-09-13T11:22:29.451 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:29.451 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:29.451 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 12", 2024-09-13T11:22:29.451 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.951454+0000", 2024-09-13T11:22:29.451 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00040372200000000002 2024-09-13T11:22:29.451 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:29.451 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:29.451 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:22:29.451 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.953357+0000", 2024-09-13T11:22:29.451 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001902613 2024-09-13T11:22:29.451 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:29.451 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:29.451 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:22:29.451 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.954420+0000", 2024-09-13T11:22:29.452 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0010635550000000001 2024-09-13T11:22:29.452 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:29.452 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:29.452 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:22:29.452 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.954455+0000", 2024-09-13T11:22:29.452 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.5203999999999997e-05 2024-09-13T11:22:29.452 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:29.452 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:29.452 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:22:29.452 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.954484+0000", 2024-09-13T11:22:29.452 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.8962000000000002e-05 2024-09-13T11:22:29.452 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:22:29.452 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:22:29.453 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:22:29.453 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:29.453 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:29.453 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.4665.0:83 3.e e33/30 3:79afbc3f:::benchmark_data_smithi177_31116_object82:head v 33'3, mlcod=32'1)", 2024-09-13T11:22:29.453 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:22:27.963374+0000", 2024-09-13T11:22:29.453 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 1.3892293520000001, 2024-09-13T11:22:29.453 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001227709, 2024-09-13T11:22:29.453 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:22:29.453 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:22:29.453 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:29.453 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:22:29.453 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:29.453 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:22:29.454 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.963374+0000", 2024-09-13T11:22:29.454 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:22:29.454 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:29.454 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:29.454 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:22:29.454 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.963374+0000", 2024-09-13T11:22:29.454 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:22:29.454 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:29.454 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:29.454 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:22:29.454 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.963374+0000", 2024-09-13T11:22:29.454 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.79e-07 2024-09-13T11:22:29.454 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:29.455 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:29.455 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:22:29.455 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.963518+0000", 2024-09-13T11:22:29.455 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00014423100000000001 2024-09-13T11:22:29.455 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:29.455 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:29.455 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:22:29.455 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.963544+0000", 2024-09-13T11:22:29.455 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.6393e-05 2024-09-13T11:22:29.455 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:29.455 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:29.455 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:22:29.455 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.963553+0000", 2024-09-13T11:22:29.455 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.0110000000000006e-06 2024-09-13T11:22:29.456 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:29.456 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:29.456 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:22:29.456 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.963610+0000", 2024-09-13T11:22:29.456 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.6656000000000001e-05 2024-09-13T11:22:29.456 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:29.456 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:29.456 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:22:29.456 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.963629+0000", 2024-09-13T11:22:29.456 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.8678e-05 2024-09-13T11:22:29.456 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:29.456 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:29.456 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:22:29.456 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.964526+0000", 2024-09-13T11:22:29.457 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00089681899999999998 2024-09-13T11:22:29.457 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:29.457 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:29.457 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:22:29.457 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.964601+0000", 2024-09-13T11:22:29.457 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.5642000000000005e-05 2024-09-13T11:22:29.457 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:22:29.457 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:22:29.457 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:22:29.457 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:29.457 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:29.457 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.4665.0:86 3.e e33/30 3:763be569:::benchmark_data_smithi177_31116_object85:head v 33'4, mlcod=32'2)", 2024-09-13T11:22:29.457 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:22:28.336613+0000", 2024-09-13T11:22:29.457 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 1.01599003, 2024-09-13T11:22:29.458 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0023452870000000002, 2024-09-13T11:22:29.458 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:22:29.458 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:22:29.458 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:29.458 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:22:29.458 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:29.458 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:22:29.458 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:28.336613+0000", 2024-09-13T11:22:29.458 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:22:29.458 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:29.458 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:29.458 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:22:29.459 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:28.336613+0000", 2024-09-13T11:22:29.459 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:22:29.459 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:29.459 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:29.459 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:22:29.459 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:28.336617+0000", 2024-09-13T11:22:29.459 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.7790000000000002e-06 2024-09-13T11:22:29.459 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:29.459 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:29.459 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:22:29.459 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:28.336753+0000", 2024-09-13T11:22:29.459 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00013588200000000001 2024-09-13T11:22:29.460 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:29.460 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:29.460 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:22:29.460 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:28.336822+0000", 2024-09-13T11:22:29.460 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.9549999999999996e-05 2024-09-13T11:22:29.460 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:29.460 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:29.460 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:22:29.460 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:28.336837+0000", 2024-09-13T11:22:29.460 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4878999999999999e-05 2024-09-13T11:22:29.460 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:29.460 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:29.460 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:22:29.460 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:28.336989+0000", 2024-09-13T11:22:29.461 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000152134 2024-09-13T11:22:29.461 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:29.461 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:29.461 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:22:29.461 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:28.337051+0000", 2024-09-13T11:22:29.461 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.1443000000000001e-05 2024-09-13T11:22:29.461 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:29.461 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:29.461 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:22:29.461 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:28.338857+0000", 2024-09-13T11:22:29.461 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0018069029999999999 2024-09-13T11:22:29.461 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:29.461 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:29.462 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:22:29.462 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:28.338958+0000", 2024-09-13T11:22:29.462 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000100717 2024-09-13T11:22:29.462 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:22:29.462 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:22:29.462 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:22:29.462 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:22:29.462 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:22:29.462 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-13T11:22:29.463 INFO:teuthology.orchestra.run.smithi001.stdout:osd.8: {} 2024-09-13T11:22:29.464 INFO:teuthology.orchestra.run.smithi001.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-13T11:22:29.464 INFO:teuthology.orchestra.run.smithi001.stdout:osd.9: {} 2024-09-13T11:22:29.464 INFO:teuthology.orchestra.run.smithi001.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-13T11:22:29.465 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-13T11:22:29.465 INFO:teuthology.orchestra.run.smithi001.stdout:osd.10: {} 2024-09-13T11:22:29.466 INFO:teuthology.orchestra.run.smithi001.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-13T11:22:29.466 INFO:tasks.ceph.osd.15.smithi177.stderr:2024-09-13T11:22:29.460+0000 7f9d6ce77640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26578) UID: 0 2024-09-13T11:22:29.474 INFO:teuthology.orchestra.run.smithi001.stdout:osd.11: {} 2024-09-13T11:22:29.474 INFO:teuthology.orchestra.run.smithi001.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-13T11:22:29.510 INFO:teuthology.orchestra.run.smithi001.stdout:osd.12: {} 2024-09-13T11:22:29.511 INFO:teuthology.orchestra.run.smithi001.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-13T11:22:29.536 INFO:teuthology.orchestra.run.smithi001.stdout:osd.13: {} 2024-09-13T11:22:29.537 INFO:teuthology.orchestra.run.smithi001.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-13T11:22:29.564 INFO:tasks.ceph.osd.2.smithi162.stderr:2024-09-13T11:22:29.560+0000 7fe92b35a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26313) UID: 0 2024-09-13T11:22:29.572 INFO:teuthology.orchestra.run.smithi001.stdout:osd.14: {} 2024-09-13T11:22:29.573 INFO:teuthology.orchestra.run.smithi001.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-13T11:22:29.596 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-09-13T11:22:29.597 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:22:29.599 INFO:teuthology.orchestra.run.smithi001.stdout:osd.15: {} 2024-09-13T11:22:29.599 INFO:teuthology.orchestra.run.smithi001.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-13T11:22:29.603 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-13T11:22:29.603 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-13T11:22:29.611 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-13T11:22:29.665 INFO:tasks.ceph.osd.9.smithi092.stderr:2024-09-13T11:22:29.664+0000 7ff5ab3d0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26348) UID: 0 2024-09-13T11:22:29.734 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-09-13T11:22:29.735 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:22:29.749 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-13T11:22:29.766 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-09-13T11:22:29.764+0000 7f97befc9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26315) UID: 0 2024-09-13T11:22:29.852 DEBUG:teuthology.orchestra.run.smithi092:> sudo rm -f /var/log/ceph/exp.2.4.1 2024-09-13T11:22:29.867 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-13T11:22:29.865+0000 7f6a68775640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26307) UID: 0 2024-09-13T11:22:29.871 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-09-13T11:22:29.871 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:22:29.885 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-13T11:22:29.954 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-13T11:22:29.967 INFO:tasks.ceph.osd.10.smithi162.stderr:2024-09-13T11:22:29.964+0000 7fb59c9f0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26317) UID: 0 2024-09-13T11:22:29.995 INFO:tasks.radosbench.radosbench.0.smithi177.stdout: 3 16 87 71 1.47898 0 - 0.0234801 2024-09-13T11:22:30.005 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-09-13T11:22:30.005 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:22:30.019 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-13T11:22:30.068 INFO:tasks.ceph.osd.8.smithi001.stderr:2024-09-13T11:22:30.061+0000 7fb9b48ae640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26312) UID: 0 2024-09-13T11:22:30.138 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-09-13T11:22:30.138 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:22:30.153 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-13T11:22:30.169 INFO:tasks.ceph.osd.10.smithi162.stderr:2024-09-13T11:22:30.168+0000 7fb59c9f0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26317) UID: 0 2024-09-13T11:22:30.269 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-09-13T11:22:30.270 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:22:30.270 INFO:tasks.ceph.osd.7.smithi177.stderr:2024-09-13T11:22:30.268+0000 7f897b51d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26575) UID: 0 2024-09-13T11:22:30.283 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-09-13T11:22:30.371 INFO:tasks.ceph.osd.13.smithi092.stderr:2024-09-13T11:22:30.368+0000 7f3e5afea640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26350) UID: 0 2024-09-13T11:22:30.453 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-09-13T11:22:30.453 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:22:30.466 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-09-13T11:22:30.471 INFO:tasks.ceph.osd.4.smithi001.stderr:2024-09-13T11:22:30.465+0000 7ff8e0218640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26310) UID: 0 2024-09-13T11:22:30.572 INFO:tasks.ceph.osd.3.smithi177.stderr:2024-09-13T11:22:30.568+0000 7f10adba4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26574) UID: 0 2024-09-13T11:22:30.584 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-09-13T11:22:30.585 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:22:30.599 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-09-13T11:22:30.675 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-09-13T11:22:30.672+0000 7f97befc9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26315) UID: 0 2024-09-13T11:22:30.766 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-09-13T11:22:30.767 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:22:30.774 INFO:tasks.ceph.osd.2.smithi162.stderr:2024-09-13T11:22:30.772+0000 7fe92b35a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26313) UID: 0 2024-09-13T11:22:30.782 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_ops_in_flight 2024-09-13T11:22:30.875 INFO:tasks.ceph.osd.9.smithi092.stderr:2024-09-13T11:22:30.876+0000 7ff5ab3d0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26348) UID: 0 2024-09-13T11:22:30.906 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-09-13T11:22:30.906 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:22:30.920 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_blocked_ops 2024-09-13T11:22:30.976 INFO:tasks.ceph.osd.12.smithi001.stderr:2024-09-13T11:22:30.973+0000 7f892c6e6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26314) UID: 0 2024-09-13T11:22:30.996 INFO:tasks.radosbench.radosbench.0.smithi177.stdout: 4 16 87 71 1.10923 0 - 0.0234801 2024-09-13T11:22:31.043 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-09-13T11:22:31.043 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:22:31.057 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_historic_ops 2024-09-13T11:22:31.077 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-09-13T11:22:31.076+0000 7f97befc9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26315) UID: 0 2024-09-13T11:22:31.172 INFO:teuthology.orchestra.run.smithi001.stderr:noscrub is unset 2024-09-13T11:22:31.174 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-09-13T11:22:31.175 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:22:31.177 INFO:tasks.ceph.osd.15.smithi177.stderr:2024-09-13T11:22:31.172+0000 7f9d6ce77640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26578) UID: 0 2024-09-13T11:22:31.189 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-13T11:22:31.209 INFO:tasks.ceph.osd.12.smithi001.stderr:2024-09-13T11:22:31.205+0000 7f891d10b640 -1 osd.12 33 handle_osd_map injecting map crc failure 2024-09-13T11:22:31.278 INFO:tasks.ceph.osd.13.smithi092.stderr:2024-09-13T11:22:31.276+0000 7f3e5afea640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26350) UID: 0 2024-09-13T11:22:31.320 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-13T11:22:31.320 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:22:31.335 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-13T11:22:31.379 INFO:tasks.ceph.osd.10.smithi162.stderr:2024-09-13T11:22:31.376+0000 7fb59c9f0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26317) UID: 0 2024-09-13T11:22:31.479 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-13T11:22:31.472+0000 7f6a68775640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26307) UID: 0 2024-09-13T11:22:31.509 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-13T11:22:31.509 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:22:31.524 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-13T11:22:31.581 INFO:tasks.ceph.osd.15.smithi177.stderr:2024-09-13T11:22:31.576+0000 7f9d6ce77640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26578) UID: 0 2024-09-13T11:22:31.605 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T11:22:31.682 INFO:tasks.ceph.osd.14.smithi162.stderr:2024-09-13T11:22:31.680+0000 7f235c0db640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26319) UID: 0 2024-09-13T11:22:31.694 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-13T11:22:31.694 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:22:31.709 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-13T11:22:31.757 INFO:teuthology.orchestra.run.smithi001.stdout:osd.0: {} 2024-09-13T11:22:31.757 INFO:teuthology.orchestra.run.smithi001.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T11:22:31.757 INFO:teuthology.orchestra.run.smithi001.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-09-13T11:22:31.782 INFO:teuthology.orchestra.run.smithi001.stdout:osd.2: {} 2024-09-13T11:22:31.783 INFO:teuthology.orchestra.run.smithi001.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T11:22:31.783 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-13T11:22:31.776+0000 7f6a68775640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26307) UID: 0 2024-09-13T11:22:31.807 INFO:teuthology.orchestra.run.smithi001.stdout:osd.3: {} 2024-09-13T11:22:31.808 INFO:teuthology.orchestra.run.smithi001.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T11:22:31.829 INFO:teuthology.orchestra.run.smithi001.stdout:osd.4: {} 2024-09-13T11:22:31.829 INFO:teuthology.orchestra.run.smithi001.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T11:22:31.830 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-13T11:22:31.830 INFO:teuthology.orchestra.run.smithi001.stdout: "ops": [], 2024-09-13T11:22:31.830 INFO:teuthology.orchestra.run.smithi001.stdout: "num_ops": 0 2024-09-13T11:22:31.830 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-13T11:22:31.844 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-13T11:22:31.857 INFO:teuthology.orchestra.run.smithi001.stdout:osd.5: {} 2024-09-13T11:22:31.858 INFO:teuthology.orchestra.run.smithi001.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T11:22:31.883 INFO:tasks.ceph.osd.2.smithi162.stderr:2024-09-13T11:22:31.880+0000 7fe92b35a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26313) UID: 0 2024-09-13T11:22:31.886 INFO:teuthology.orchestra.run.smithi001.stdout:osd.6: {} 2024-09-13T11:22:31.887 INFO:teuthology.orchestra.run.smithi001.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T11:22:31.912 INFO:teuthology.orchestra.run.smithi001.stdout:osd.7: {} 2024-09-13T11:22:31.912 INFO:teuthology.orchestra.run.smithi001.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T11:22:31.933 INFO:teuthology.orchestra.run.smithi001.stdout:osd.8: {} 2024-09-13T11:22:31.933 INFO:teuthology.orchestra.run.smithi001.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-13T11:22:31.958 INFO:teuthology.orchestra.run.smithi001.stdout:osd.9: {} 2024-09-13T11:22:31.959 INFO:teuthology.orchestra.run.smithi001.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-13T11:22:31.984 INFO:teuthology.orchestra.run.smithi001.stdout:osd.10: {} 2024-09-13T11:22:31.984 INFO:teuthology.orchestra.run.smithi001.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-13T11:22:31.985 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-13T11:22:31.985 INFO:teuthology.orchestra.run.smithi001.stdout: "ops": [], 2024-09-13T11:22:31.985 INFO:teuthology.orchestra.run.smithi001.stdout: "complaint_time": 30, 2024-09-13T11:22:31.985 INFO:teuthology.orchestra.run.smithi001.stdout: "num_blocked_ops": 0 2024-09-13T11:22:31.985 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-13T11:22:31.985 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-13T11:22:31.980+0000 7f6a68775640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26307) UID: 0 2024-09-13T11:22:31.996 INFO:tasks.radosbench.radosbench.0.smithi177.stdout: 5 16 87 71 0.887382 0 - 0.0234801 2024-09-13T11:22:31.999 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-13T11:22:32.013 INFO:teuthology.orchestra.run.smithi001.stdout:osd.11: {} 2024-09-13T11:22:32.014 INFO:teuthology.orchestra.run.smithi001.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-13T11:22:32.041 INFO:teuthology.orchestra.run.smithi001.stdout:osd.12: {} 2024-09-13T11:22:32.041 INFO:teuthology.orchestra.run.smithi001.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-13T11:22:32.070 INFO:teuthology.orchestra.run.smithi001.stdout:osd.13: {} 2024-09-13T11:22:32.070 INFO:teuthology.orchestra.run.smithi001.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-13T11:22:32.085 INFO:tasks.ceph.osd.9.smithi092.stderr:2024-09-13T11:22:32.083+0000 7ff5ab3d0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26348) UID: 0 2024-09-13T11:22:32.099 INFO:teuthology.orchestra.run.smithi001.stdout:osd.14: {} 2024-09-13T11:22:32.099 INFO:teuthology.orchestra.run.smithi001.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-13T11:22:32.131 INFO:teuthology.orchestra.run.smithi001.stdout:osd.15: {} 2024-09-13T11:22:32.131 INFO:teuthology.orchestra.run.smithi001.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-13T11:22:32.134 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-13T11:22:32.135 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-13T11:22:32.150 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-13T11:22:32.150 INFO:teuthology.orchestra.run.smithi001.stdout: "size": 20, 2024-09-13T11:22:32.150 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 600, 2024-09-13T11:22:32.150 INFO:teuthology.orchestra.run.smithi001.stdout: "ops": [] 2024-09-13T11:22:32.150 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-13T11:22:32.165 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-09-13T11:22:32.212 INFO:tasks.ceph.osd.12.smithi001.stderr:2024-09-13T11:22:32.208+0000 7f892c6e6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26314) UID: 0 2024-09-13T11:22:32.286 INFO:tasks.ceph.osd.14.smithi162.stderr:2024-09-13T11:22:32.284+0000 7f235c0db640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26319) UID: 0 2024-09-13T11:22:32.339 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-13T11:22:32.339 INFO:teuthology.orchestra.run.smithi001.stdout: "ops": [], 2024-09-13T11:22:32.339 INFO:teuthology.orchestra.run.smithi001.stdout: "num_ops": 0 2024-09-13T11:22:32.339 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-13T11:22:32.355 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-09-13T11:22:32.387 INFO:tasks.ceph.osd.7.smithi177.stderr:2024-09-13T11:22:32.384+0000 7f897b51d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26575) UID: 0 2024-09-13T11:22:32.475 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-13T11:22:32.475 INFO:teuthology.orchestra.run.smithi001.stdout: "ops": [], 2024-09-13T11:22:32.475 INFO:teuthology.orchestra.run.smithi001.stdout: "complaint_time": 30, 2024-09-13T11:22:32.475 INFO:teuthology.orchestra.run.smithi001.stdout: "num_blocked_ops": 0 2024-09-13T11:22:32.476 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-13T11:22:32.488 INFO:tasks.ceph.osd.15.smithi177.stderr:2024-09-13T11:22:32.488+0000 7f9d6ce77640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26578) UID: 0 2024-09-13T11:22:32.493 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-09-13T11:22:32.589 INFO:tasks.ceph.osd.14.smithi162.stderr:2024-09-13T11:22:32.588+0000 7f235c0db640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26319) UID: 0 2024-09-13T11:22:32.621 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-13T11:22:32.621 INFO:teuthology.orchestra.run.smithi001.stdout: "size": 20, 2024-09-13T11:22:32.621 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 600, 2024-09-13T11:22:32.622 INFO:teuthology.orchestra.run.smithi001.stdout: "ops": [ 2024-09-13T11:22:32.622 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:32.622 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4665.0:16 3.7 e32/30 3:e0940737:::benchmark_data_smithi177_31116_object15:head v 32'1, mlcod=0'0)", 2024-09-13T11:22:32.622 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:22:27.002190+0000", 2024-09-13T11:22:32.622 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 5.6161627210000002, 2024-09-13T11:22:32.622 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0014874980000000001, 2024-09-13T11:22:32.622 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:22:32.622 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:22:32.622 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:32.622 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:22:32.622 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:32.623 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:22:32.623 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.002190+0000", 2024-09-13T11:22:32.623 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:22:32.623 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:32.623 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:32.623 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:22:32.623 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.002190+0000", 2024-09-13T11:22:32.623 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:22:32.623 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:32.623 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:32.623 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:22:32.624 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.002191+0000", 2024-09-13T11:22:32.624 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.2900000000000002e-07 2024-09-13T11:22:32.624 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:32.624 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:32.624 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:22:32.624 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.002373+0000", 2024-09-13T11:22:32.624 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00018228400000000001 2024-09-13T11:22:32.624 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:32.624 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:32.624 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:22:32.624 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.002414+0000", 2024-09-13T11:22:32.624 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.1162999999999999e-05 2024-09-13T11:22:32.625 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:32.625 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:32.625 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:22:32.625 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.002429+0000", 2024-09-13T11:22:32.625 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.5053e-05 2024-09-13T11:22:32.625 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:32.625 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:32.625 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:22:32.625 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.002490+0000", 2024-09-13T11:22:32.626 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.1316999999999996e-05 2024-09-13T11:22:32.626 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:32.626 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:32.626 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:22:32.626 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.002526+0000", 2024-09-13T11:22:32.626 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.5788999999999999e-05 2024-09-13T11:22:32.626 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:32.626 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:32.626 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:22:32.626 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.003571+0000", 2024-09-13T11:22:32.626 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.001044445 2024-09-13T11:22:32.626 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:32.627 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:32.627 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:22:32.627 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.003677+0000", 2024-09-13T11:22:32.627 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000106518 2024-09-13T11:22:32.627 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:22:32.627 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:22:32.627 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:22:32.627 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:32.627 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:32.627 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4665.0:25 3.7 e32/30 3:e6850330:::benchmark_data_smithi177_31116_object24:head v 32'2, mlcod=0'0)", 2024-09-13T11:22:32.627 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:22:27.012207+0000", 2024-09-13T11:22:32.627 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 5.6061455379999998, 2024-09-13T11:22:32.628 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0015694610000000001, 2024-09-13T11:22:32.628 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:22:32.628 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:22:32.628 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:32.628 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:22:32.628 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:32.628 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:22:32.628 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.012207+0000", 2024-09-13T11:22:32.629 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:22:32.629 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:32.629 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:32.629 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:22:32.629 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.012207+0000", 2024-09-13T11:22:32.629 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:22:32.629 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:32.629 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:32.629 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:22:32.629 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.012209+0000", 2024-09-13T11:22:32.630 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.3070000000000001e-06 2024-09-13T11:22:32.630 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:32.630 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:32.630 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:22:32.630 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.012329+0000", 2024-09-13T11:22:32.630 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000120061 2024-09-13T11:22:32.630 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:32.630 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:32.630 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:22:32.630 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.012445+0000", 2024-09-13T11:22:32.630 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000116104 2024-09-13T11:22:32.631 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:32.631 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:32.631 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:22:32.631 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.012458+0000", 2024-09-13T11:22:32.631 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2723e-05 2024-09-13T11:22:32.631 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:32.631 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:32.632 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:22:32.632 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.012598+0000", 2024-09-13T11:22:32.632 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00014023200000000001 2024-09-13T11:22:32.632 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:32.632 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:32.632 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:22:32.632 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.012633+0000", 2024-09-13T11:22:32.632 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.4643e-05 2024-09-13T11:22:32.632 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:32.633 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:32.633 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:22:32.633 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.013733+0000", 2024-09-13T11:22:32.633 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.001100214 2024-09-13T11:22:32.633 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:32.634 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:32.634 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:22:32.634 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.013776+0000", 2024-09-13T11:22:32.634 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.3176999999999997e-05 2024-09-13T11:22:32.634 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:22:32.634 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:22:32.635 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:22:32.635 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:32.635 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:32.635 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:20 3.e 3:794d9a5b:::benchmark_data_smithi177_31116_object19:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-09-13T11:22:32.635 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:22:27.012384+0000", 2024-09-13T11:22:32.635 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 5.6059685439999996, 2024-09-13T11:22:32.635 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0065062369999999998, 2024-09-13T11:22:32.635 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:22:32.635 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:22:32.635 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:32.635 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:22:32.635 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:22:32.635 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:22:32.636 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 20 2024-09-13T11:22:32.636 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:32.636 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:22:32.636 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:32.636 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:22:32.636 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.012384+0000", 2024-09-13T11:22:32.636 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:22:32.636 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:32.636 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:32.636 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:22:32.636 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.012384+0000", 2024-09-13T11:22:32.636 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:22:32.636 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:32.636 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:32.637 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:22:32.637 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.012386+0000", 2024-09-13T11:22:32.637 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.6800000000000002e-06 2024-09-13T11:22:32.637 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:32.637 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:32.637 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:22:32.637 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.012697+0000", 2024-09-13T11:22:32.637 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00031023200000000002 2024-09-13T11:22:32.637 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:32.637 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:32.637 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:22:32.637 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.012711+0000", 2024-09-13T11:22:32.637 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.4184e-05 2024-09-13T11:22:32.637 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:32.638 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:32.638 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:22:32.638 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.012720+0000", 2024-09-13T11:22:32.638 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.1069999999999994e-06 2024-09-13T11:22:32.638 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:32.638 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:32.638 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:22:32.638 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.012787+0000", 2024-09-13T11:22:32.638 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.6661000000000004e-05 2024-09-13T11:22:32.638 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:32.638 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:32.638 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:22:32.638 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.012898+0000", 2024-09-13T11:22:32.638 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000111309 2024-09-13T11:22:32.639 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:32.639 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:32.639 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 14", 2024-09-13T11:22:32.639 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.013074+0000", 2024-09-13T11:22:32.639 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000176119 2024-09-13T11:22:32.639 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:32.639 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:32.639 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:22:32.639 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.014110+0000", 2024-09-13T11:22:32.639 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0010358100000000001 2024-09-13T11:22:32.639 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:32.639 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:32.639 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:22:32.639 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.018780+0000", 2024-09-13T11:22:32.640 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.004669884 2024-09-13T11:22:32.640 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:32.640 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:32.640 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:22:32.640 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.018841+0000", 2024-09-13T11:22:32.640 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.1309000000000005e-05 2024-09-13T11:22:32.640 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:32.640 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:32.640 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:22:32.640 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.018890+0000", 2024-09-13T11:22:32.640 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.8942e-05 2024-09-13T11:22:32.640 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:22:32.640 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:22:32.641 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:22:32.641 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:32.641 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:32.641 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4665.0:33 3.7 e32/30 3:ebfb0388:::benchmark_data_smithi177_31116_object32:head v 32'3, mlcod=0'0)", 2024-09-13T11:22:32.641 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:22:27.014563+0000", 2024-09-13T11:22:32.641 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 5.6037890319999999, 2024-09-13T11:22:32.641 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.001063968, 2024-09-13T11:22:32.641 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:22:32.641 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:22:32.641 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:32.641 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:22:32.641 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:32.641 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:22:32.641 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.014563+0000", 2024-09-13T11:22:32.642 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:22:32.642 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:32.642 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:32.642 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:22:32.642 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.014563+0000", 2024-09-13T11:22:32.642 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:22:32.642 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:32.642 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:32.642 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:22:32.642 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.014565+0000", 2024-09-13T11:22:32.642 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.263e-06 2024-09-13T11:22:32.642 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:32.642 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:32.643 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:22:32.643 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.014630+0000", 2024-09-13T11:22:32.643 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.5777000000000004e-05 2024-09-13T11:22:32.643 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:32.643 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:32.643 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:22:32.643 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.014660+0000", 2024-09-13T11:22:32.643 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.9660999999999999e-05 2024-09-13T11:22:32.643 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:32.643 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:32.643 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:22:32.643 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.014667+0000", 2024-09-13T11:22:32.643 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.7399999999999998e-06 2024-09-13T11:22:32.643 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:32.643 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:32.644 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:22:32.644 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.014716+0000", 2024-09-13T11:22:32.644 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.9094999999999997e-05 2024-09-13T11:22:32.644 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:32.644 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:32.644 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:22:32.644 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.014733+0000", 2024-09-13T11:22:32.644 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.7027000000000001e-05 2024-09-13T11:22:32.644 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:32.644 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:32.644 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:22:32.644 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.015579+0000", 2024-09-13T11:22:32.644 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000845665 2024-09-13T11:22:32.644 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:32.645 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:32.645 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:22:32.645 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.015627+0000", 2024-09-13T11:22:32.645 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.8739999999999998e-05 2024-09-13T11:22:32.645 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:22:32.645 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:22:32.645 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:22:32.645 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:32.645 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:32.645 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop_reply(client.4665.0:20 3.e e32/30 ondisk, result = 0)", 2024-09-13T11:22:32.645 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:22:27.018628+0000", 2024-09-13T11:22:32.645 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 5.5997246460000003, 2024-09-13T11:22:32.645 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00027831200000000001, 2024-09-13T11:22:32.646 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:22:32.646 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:22:32.646 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:22:32.646 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:22:32.646 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:32.646 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:22:32.646 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.018628+0000", 2024-09-13T11:22:32.646 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:22:32.646 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:32.646 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:32.646 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:22:32.646 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.018628+0000", 2024-09-13T11:22:32.646 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:22:32.646 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:32.647 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:32.647 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:22:32.647 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.018630+0000", 2024-09-13T11:22:32.647 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.2010000000000002e-06 2024-09-13T11:22:32.647 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:32.647 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:32.647 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:22:32.647 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.018638+0000", 2024-09-13T11:22:32.647 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.9990000000000008e-06 2024-09-13T11:22:32.647 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:32.647 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:32.647 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:22:32.647 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.018686+0000", 2024-09-13T11:22:32.648 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.833e-05 2024-09-13T11:22:32.648 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:32.648 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:32.648 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:22:32.648 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.018697+0000", 2024-09-13T11:22:32.648 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.111e-05 2024-09-13T11:22:32.648 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:32.648 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:32.648 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:22:32.648 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.018765+0000", 2024-09-13T11:22:32.648 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.7714999999999998e-05 2024-09-13T11:22:32.648 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:32.648 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:32.648 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:22:32.649 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.018774+0000", 2024-09-13T11:22:32.649 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.6619999999999994e-06 2024-09-13T11:22:32.649 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:32.649 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:32.649 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:22:32.649 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.018906+0000", 2024-09-13T11:22:32.649 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00013229499999999999 2024-09-13T11:22:32.649 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:22:32.649 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:22:32.649 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:22:32.649 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:32.649 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:32.649 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:46 3.e 3:7d541445:::benchmark_data_smithi177_31116_object45:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-09-13T11:22:32.649 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:22:27.019140+0000", 2024-09-13T11:22:32.650 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 5.5992121460000002, 2024-09-13T11:22:32.650 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0023100759999999999, 2024-09-13T11:22:32.650 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:22:32.650 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:22:32.650 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:32.650 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:22:32.650 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:22:32.650 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:22:32.650 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 46 2024-09-13T11:22:32.650 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:32.650 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:22:32.650 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:32.650 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:22:32.651 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.019140+0000", 2024-09-13T11:22:32.651 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:22:32.651 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:32.651 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:32.651 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:22:32.651 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.019140+0000", 2024-09-13T11:22:32.651 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:22:32.651 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:32.651 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:32.651 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:22:32.651 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.019142+0000", 2024-09-13T11:22:32.651 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.9429999999999999e-06 2024-09-13T11:22:32.651 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:32.651 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:32.652 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:22:32.652 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.019211+0000", 2024-09-13T11:22:32.652 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.8855000000000004e-05 2024-09-13T11:22:32.652 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:32.652 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:32.652 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:22:32.652 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.019227+0000", 2024-09-13T11:22:32.652 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.5973000000000001e-05 2024-09-13T11:22:32.652 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:32.652 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:32.652 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:22:32.652 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.019237+0000", 2024-09-13T11:22:32.652 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0438999999999999e-05 2024-09-13T11:22:32.652 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:32.653 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:32.653 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:22:32.653 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.019307+0000", 2024-09-13T11:22:32.653 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.9756000000000001e-05 2024-09-13T11:22:32.653 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:32.653 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:32.653 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:22:32.653 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.019456+0000", 2024-09-13T11:22:32.653 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000149326 2024-09-13T11:22:32.653 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:32.653 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:32.653 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 14", 2024-09-13T11:22:32.653 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.019615+0000", 2024-09-13T11:22:32.653 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00015860900000000001 2024-09-13T11:22:32.654 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:32.654 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:32.654 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:22:32.654 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.020630+0000", 2024-09-13T11:22:32.654 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.001014439 2024-09-13T11:22:32.654 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:32.654 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:32.654 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:22:32.654 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.021369+0000", 2024-09-13T11:22:32.654 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00073971400000000004 2024-09-13T11:22:32.654 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:32.654 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:32.654 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:22:32.654 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.021413+0000", 2024-09-13T11:22:32.655 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.3353999999999999e-05 2024-09-13T11:22:32.655 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:32.655 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:32.655 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:22:32.655 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.021450+0000", 2024-09-13T11:22:32.655 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.7667999999999999e-05 2024-09-13T11:22:32.655 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:22:32.655 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:22:32.655 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:22:32.655 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:32.655 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:32.655 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4665.0:47 3.7 e32/30 3:e22c870b:::benchmark_data_smithi177_31116_object46:head v 32'4, mlcod=32'2)", 2024-09-13T11:22:32.655 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:22:27.020290+0000", 2024-09-13T11:22:32.655 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 5.598062756, 2024-09-13T11:22:32.656 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00097084700000000005, 2024-09-13T11:22:32.656 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:22:32.656 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:22:32.656 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:32.656 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:22:32.656 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:32.656 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:22:32.656 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.020290+0000", 2024-09-13T11:22:32.656 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:22:32.656 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:32.656 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:32.656 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:22:32.656 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.020290+0000", 2024-09-13T11:22:32.656 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:22:32.657 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:32.657 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:32.657 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:22:32.657 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.020291+0000", 2024-09-13T11:22:32.657 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.728e-06 2024-09-13T11:22:32.657 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:32.657 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:32.657 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:22:32.657 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.020356+0000", 2024-09-13T11:22:32.657 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.5105999999999996e-05 2024-09-13T11:22:32.657 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:32.657 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:32.657 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:22:32.657 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.020385+0000", 2024-09-13T11:22:32.658 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.8393000000000001e-05 2024-09-13T11:22:32.658 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:32.658 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:32.658 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:22:32.658 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.020391+0000", 2024-09-13T11:22:32.658 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.4309999999999999e-06 2024-09-13T11:22:32.658 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:32.658 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:32.658 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:22:32.658 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.020480+0000", 2024-09-13T11:22:32.658 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.9163e-05 2024-09-13T11:22:32.658 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:32.658 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:32.658 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:22:32.659 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.020508+0000", 2024-09-13T11:22:32.659 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.7194000000000001e-05 2024-09-13T11:22:32.659 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:32.659 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:32.659 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:22:32.659 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.021217+0000", 2024-09-13T11:22:32.659 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00070979099999999998 2024-09-13T11:22:32.659 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:32.659 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:32.659 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:22:32.659 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.021260+0000", 2024-09-13T11:22:32.659 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.3040999999999997e-05 2024-09-13T11:22:32.659 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:22:32.659 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:22:32.659 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:22:32.660 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:32.660 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:32.660 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop_reply(client.4665.0:46 3.e e32/30 ondisk, result = 0)", 2024-09-13T11:22:32.660 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:22:27.021265+0000", 2024-09-13T11:22:32.660 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 5.5970872319999998, 2024-09-13T11:22:32.660 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00019617699999999999, 2024-09-13T11:22:32.660 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:22:32.660 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:22:32.660 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:22:32.660 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:22:32.660 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:32.660 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:22:32.660 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.021265+0000", 2024-09-13T11:22:32.660 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:22:32.661 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:32.661 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:32.661 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:22:32.661 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.021265+0000", 2024-09-13T11:22:32.661 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:22:32.661 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:32.661 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:32.661 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:22:32.661 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.021266+0000", 2024-09-13T11:22:32.661 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.232e-06 2024-09-13T11:22:32.661 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:32.661 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:32.661 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:22:32.661 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.021271+0000", 2024-09-13T11:22:32.662 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.6040000000000003e-06 2024-09-13T11:22:32.662 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:32.662 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:32.662 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:22:32.662 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.021308+0000", 2024-09-13T11:22:32.662 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.7206000000000003e-05 2024-09-13T11:22:32.662 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:32.662 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:32.662 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:22:32.662 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.021314+0000", 2024-09-13T11:22:32.662 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.8860000000000001e-06 2024-09-13T11:22:32.662 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:32.662 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:32.662 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:22:32.663 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.021358+0000", 2024-09-13T11:22:32.663 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.3708000000000002e-05 2024-09-13T11:22:32.663 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:32.663 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:32.663 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:22:32.663 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.021364+0000", 2024-09-13T11:22:32.663 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.1149999999999999e-06 2024-09-13T11:22:32.663 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:32.663 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:32.663 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:22:32.663 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.021461+0000", 2024-09-13T11:22:32.663 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.7425999999999999e-05 2024-09-13T11:22:32.663 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:22:32.663 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:22:32.664 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:22:32.664 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:32.664 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:32.664 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4665.0:51 3.7 e32/30 3:e2e297cd:::benchmark_data_smithi177_31116_object50:head v 32'5, mlcod=32'3)", 2024-09-13T11:22:32.664 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:22:27.023001+0000", 2024-09-13T11:22:32.664 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 5.5953512490000001, 2024-09-13T11:22:32.664 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00094912600000000001, 2024-09-13T11:22:32.664 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:22:32.664 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:22:32.664 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:32.664 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:22:32.664 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:32.664 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:22:32.664 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.023001+0000", 2024-09-13T11:22:32.665 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:22:32.665 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:32.665 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:32.665 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:22:32.665 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.023001+0000", 2024-09-13T11:22:32.665 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:22:32.665 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:32.665 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:32.665 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:22:32.665 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.023003+0000", 2024-09-13T11:22:32.665 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.685e-06 2024-09-13T11:22:32.665 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:32.665 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:32.665 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:22:32.666 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.023056+0000", 2024-09-13T11:22:32.666 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.3319999999999998e-05 2024-09-13T11:22:32.666 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:32.666 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:32.666 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:22:32.666 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.023092+0000", 2024-09-13T11:22:32.666 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.5716000000000002e-05 2024-09-13T11:22:32.666 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:32.666 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:32.666 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:22:32.666 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.023103+0000", 2024-09-13T11:22:32.666 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.1389000000000001e-05 2024-09-13T11:22:32.666 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:32.666 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:32.667 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:22:32.667 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.023176+0000", 2024-09-13T11:22:32.667 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.2488000000000001e-05 2024-09-13T11:22:32.667 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:32.667 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:32.667 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:22:32.667 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.023194+0000", 2024-09-13T11:22:32.667 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.8800999999999998e-05 2024-09-13T11:22:32.667 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:32.667 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:32.667 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:22:32.667 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.023908+0000", 2024-09-13T11:22:32.667 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00071373900000000002 2024-09-13T11:22:32.667 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:32.668 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:32.668 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:22:32.668 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.023950+0000", 2024-09-13T11:22:32.668 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.1987999999999998e-05 2024-09-13T11:22:32.668 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:22:32.668 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:22:32.668 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:22:32.668 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:32.668 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:32.668 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4665.0:53 3.7 e32/30 3:e998c81b:::benchmark_data_smithi177_31116_object52:head v 32'6, mlcod=32'3)", 2024-09-13T11:22:32.668 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:22:27.024164+0000", 2024-09-13T11:22:32.668 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 5.5941883140000002, 2024-09-13T11:22:32.668 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0010018270000000001, 2024-09-13T11:22:32.669 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:22:32.669 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:22:32.669 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:32.669 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:22:32.669 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:32.669 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:22:32.669 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.024164+0000", 2024-09-13T11:22:32.669 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:22:32.669 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:32.669 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:32.669 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:22:32.669 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.024164+0000", 2024-09-13T11:22:32.669 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:22:32.669 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:32.670 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:32.670 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:22:32.670 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.024165+0000", 2024-09-13T11:22:32.670 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.449e-06 2024-09-13T11:22:32.670 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:32.670 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:32.670 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:22:32.670 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.024234+0000", 2024-09-13T11:22:32.670 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.8758999999999997e-05 2024-09-13T11:22:32.670 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:32.670 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:32.670 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:22:32.670 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.024265+0000", 2024-09-13T11:22:32.670 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.0744999999999998e-05 2024-09-13T11:22:32.671 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:32.671 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:32.671 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:22:32.671 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.024274+0000", 2024-09-13T11:22:32.671 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.4120000000000007e-06 2024-09-13T11:22:32.671 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:32.671 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:32.671 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:22:32.671 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.024377+0000", 2024-09-13T11:22:32.671 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00010253 2024-09-13T11:22:32.671 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:32.671 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:32.671 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:22:32.671 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.024395+0000", 2024-09-13T11:22:32.672 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.7788999999999999e-05 2024-09-13T11:22:32.672 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:32.672 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:32.672 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:22:32.672 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.025123+0000", 2024-09-13T11:22:32.672 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00072863400000000003 2024-09-13T11:22:32.672 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:32.672 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:32.672 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:22:32.672 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.025166+0000", 2024-09-13T11:22:32.672 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.2509000000000002e-05 2024-09-13T11:22:32.672 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:22:32.672 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:22:32.672 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:22:32.673 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:32.673 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:32.673 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4665.0:57 3.7 e32/30 3:e5e6d1ae:::benchmark_data_smithi177_31116_object56:head v 32'7, mlcod=32'5)", 2024-09-13T11:22:32.673 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:22:27.027031+0000", 2024-09-13T11:22:32.673 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 5.5913211069999997, 2024-09-13T11:22:32.673 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0010161809999999999, 2024-09-13T11:22:32.673 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:22:32.673 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:22:32.673 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:32.673 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:22:32.673 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:32.673 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:22:32.673 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.027031+0000", 2024-09-13T11:22:32.674 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:22:32.674 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:32.674 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:32.674 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:22:32.674 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.027031+0000", 2024-09-13T11:22:32.674 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:22:32.674 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:32.674 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:32.674 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:22:32.674 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.027033+0000", 2024-09-13T11:22:32.674 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.576e-06 2024-09-13T11:22:32.674 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:32.674 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:32.675 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:22:32.675 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.027069+0000", 2024-09-13T11:22:32.675 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.6185e-05 2024-09-13T11:22:32.675 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:32.675 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:32.675 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:22:32.675 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.027104+0000", 2024-09-13T11:22:32.675 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.4708e-05 2024-09-13T11:22:32.675 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:32.675 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:32.675 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:22:32.675 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.027110+0000", 2024-09-13T11:22:32.675 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.596e-06 2024-09-13T11:22:32.675 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:32.676 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:32.676 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:22:32.676 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.027171+0000", 2024-09-13T11:22:32.676 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.0851999999999999e-05 2024-09-13T11:22:32.676 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:32.676 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:32.676 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:22:32.676 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.027195+0000", 2024-09-13T11:22:32.676 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.4341e-05 2024-09-13T11:22:32.676 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:32.676 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:32.676 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:22:32.676 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.028005+0000", 2024-09-13T11:22:32.676 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00080958699999999996 2024-09-13T11:22:32.677 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:32.677 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:32.677 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:22:32.677 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.028047+0000", 2024-09-13T11:22:32.677 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.2336000000000003e-05 2024-09-13T11:22:32.677 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:22:32.677 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:22:32.677 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:22:32.677 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:32.677 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:32.677 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:83 3.e 3:79afbc3f:::benchmark_data_smithi177_31116_object82:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e33)", 2024-09-13T11:22:32.677 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:22:27.958896+0000", 2024-09-13T11:22:32.677 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 4.6594564480000003, 2024-09-13T11:22:32.678 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0048065210000000002, 2024-09-13T11:22:32.678 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:22:32.678 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:22:32.678 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:32.678 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:22:32.678 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:22:32.678 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:22:32.678 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 83 2024-09-13T11:22:32.678 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:32.678 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:22:32.678 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:32.678 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:22:32.678 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.958896+0000", 2024-09-13T11:22:32.678 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:22:32.679 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:32.679 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:32.679 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:22:32.679 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.958896+0000", 2024-09-13T11:22:32.679 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:22:32.679 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:32.679 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:32.679 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:22:32.679 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.958900+0000", 2024-09-13T11:22:32.679 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.437e-06 2024-09-13T11:22:32.679 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:32.679 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:32.679 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:22:32.679 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.959128+0000", 2024-09-13T11:22:32.680 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00022734900000000001 2024-09-13T11:22:32.680 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:32.680 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:32.680 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:22:32.680 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.959146+0000", 2024-09-13T11:22:32.680 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.7846000000000002e-05 2024-09-13T11:22:32.680 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:32.680 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:32.680 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:22:32.680 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.959159+0000", 2024-09-13T11:22:32.680 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.326e-05 2024-09-13T11:22:32.680 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:32.680 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:32.681 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:22:32.681 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.959274+0000", 2024-09-13T11:22:32.681 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000114935 2024-09-13T11:22:32.681 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:32.681 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:32.681 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op must wait for map", 2024-09-13T11:22:32.681 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.959280+0000", 2024-09-13T11:22:32.681 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.4919999999999998e-06 2024-09-13T11:22:32.681 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:32.681 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:32.681 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:22:32.681 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.961142+0000", 2024-09-13T11:22:32.681 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0018618689999999999 2024-09-13T11:22:32.682 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:32.682 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:32.682 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:22:32.682 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.961378+0000", 2024-09-13T11:22:32.682 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000236393 2024-09-13T11:22:32.682 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:32.682 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:32.682 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 14", 2024-09-13T11:22:32.682 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.961628+0000", 2024-09-13T11:22:32.682 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00024923300000000002 2024-09-13T11:22:32.682 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:32.682 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:32.682 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:22:32.683 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.963088+0000", 2024-09-13T11:22:32.683 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.001460647 2024-09-13T11:22:32.683 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:32.683 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:32.683 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:22:32.683 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.963546+0000", 2024-09-13T11:22:32.683 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00045760000000000001 2024-09-13T11:22:32.683 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:32.683 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:32.683 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:22:32.683 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.963638+0000", 2024-09-13T11:22:32.683 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.2263999999999996e-05 2024-09-13T11:22:32.683 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:32.683 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:32.684 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:22:32.684 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.963702+0000", 2024-09-13T11:22:32.684 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.4196000000000007e-05 2024-09-13T11:22:32.684 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:22:32.684 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:22:32.684 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:22:32.684 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:32.684 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:32.684 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop_reply(client.4665.0:83 3.e e33/30 ondisk, result = 0)", 2024-09-13T11:22:32.684 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:22:27.963387+0000", 2024-09-13T11:22:32.684 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 4.6549654260000004, 2024-09-13T11:22:32.684 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00032713600000000001, 2024-09-13T11:22:32.684 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:22:32.685 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:22:32.685 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:22:32.685 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:22:32.685 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:32.685 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:22:32.685 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.963387+0000", 2024-09-13T11:22:32.685 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:22:32.685 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:32.685 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:32.685 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:22:32.685 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.963387+0000", 2024-09-13T11:22:32.685 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:22:32.685 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:32.685 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:32.686 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:22:32.686 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.963389+0000", 2024-09-13T11:22:32.686 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.8339999999999999e-06 2024-09-13T11:22:32.686 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:32.686 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:32.686 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:22:32.686 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.963399+0000", 2024-09-13T11:22:32.686 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.8830000000000001e-06 2024-09-13T11:22:32.686 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:32.686 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:32.686 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:22:32.686 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.963436+0000", 2024-09-13T11:22:32.686 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.7536000000000001e-05 2024-09-13T11:22:32.686 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:32.687 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:32.687 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:22:32.687 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.963447+0000", 2024-09-13T11:22:32.687 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.1093e-05 2024-09-13T11:22:32.687 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:32.687 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:32.687 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:22:32.687 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.963524+0000", 2024-09-13T11:22:32.687 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.6433000000000005e-05 2024-09-13T11:22:32.687 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:32.687 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:32.687 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:22:32.687 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.963535+0000", 2024-09-13T11:22:32.687 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.1506e-05 2024-09-13T11:22:32.687 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:32.688 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:32.688 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:22:32.688 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.963714+0000", 2024-09-13T11:22:32.688 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000178851 2024-09-13T11:22:32.688 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:22:32.688 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:22:32.688 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:22:32.688 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:32.688 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:32.688 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:86 3.e 3:763be569:::benchmark_data_smithi177_31116_object85:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e33)", 2024-09-13T11:22:32.688 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:22:28.334113+0000", 2024-09-13T11:22:32.688 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 4.2842393459999997, 2024-09-13T11:22:32.688 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0040445070000000001, 2024-09-13T11:22:32.688 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:22:32.689 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:22:32.689 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:32.689 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:22:32.689 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:22:32.689 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:22:32.689 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 86 2024-09-13T11:22:32.689 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:32.689 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:22:32.689 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:32.689 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:22:32.689 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:28.334113+0000", 2024-09-13T11:22:32.689 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:22:32.689 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:32.689 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:32.690 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:22:32.690 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:28.334113+0000", 2024-09-13T11:22:32.690 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:22:32.690 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:32.690 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:32.690 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:22:32.690 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:28.334117+0000", 2024-09-13T11:22:32.690 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.2359999999999998e-06 2024-09-13T11:22:32.690 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:32.690 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:32.690 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:22:32.690 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:28.334253+0000", 2024-09-13T11:22:32.690 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00013581500000000001 2024-09-13T11:22:32.690 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:32.691 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:32.691 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:22:32.691 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:28.334284+0000", 2024-09-13T11:22:32.691 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.1253e-05 2024-09-13T11:22:32.691 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:32.691 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:32.691 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:22:32.691 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:28.334306+0000", 2024-09-13T11:22:32.691 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.1976000000000001e-05 2024-09-13T11:22:32.691 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:32.691 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:32.691 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:22:32.691 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:28.334489+0000", 2024-09-13T11:22:32.691 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00018322400000000001 2024-09-13T11:22:32.692 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:32.692 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:32.692 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:22:32.692 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:28.334702+0000", 2024-09-13T11:22:32.692 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00021245699999999999 2024-09-13T11:22:32.692 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:32.692 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:32.692 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 14", 2024-09-13T11:22:32.692 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:28.334926+0000", 2024-09-13T11:22:32.692 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00022375699999999999 2024-09-13T11:22:32.692 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:32.692 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:32.692 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:22:32.692 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:28.336643+0000", 2024-09-13T11:22:32.693 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.001717784 2024-09-13T11:22:32.693 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:32.693 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:32.693 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:22:32.693 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:28.337999+0000", 2024-09-13T11:22:32.693 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0013551870000000001 2024-09-13T11:22:32.693 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:32.693 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:32.693 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:22:32.693 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:28.338089+0000", 2024-09-13T11:22:32.696 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.0062000000000006e-05 2024-09-13T11:22:32.696 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:32.696 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:32.696 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:22:32.696 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:28.338157+0000", 2024-09-13T11:22:32.697 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.8756000000000004e-05 2024-09-13T11:22:32.697 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:22:32.697 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:22:32.697 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:22:32.697 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:32.697 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:32.697 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop_reply(client.4665.0:86 3.e e33/30 ondisk, result = 0)", 2024-09-13T11:22:32.697 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:22:28.337787+0000", 2024-09-13T11:22:32.697 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 4.2805651129999998, 2024-09-13T11:22:32.697 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00038257100000000001, 2024-09-13T11:22:32.697 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:22:32.697 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:22:32.697 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:22:32.697 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:22:32.698 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:32.698 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:22:32.698 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:28.337787+0000", 2024-09-13T11:22:32.698 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:22:32.698 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:32.698 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:32.698 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:22:32.698 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:28.337787+0000", 2024-09-13T11:22:32.698 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:22:32.698 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:32.698 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:32.698 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:22:32.698 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:28.337796+0000", 2024-09-13T11:22:32.698 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.8189999999999998e-06 2024-09-13T11:22:32.699 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:32.699 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:32.699 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:22:32.699 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:28.337804+0000", 2024-09-13T11:22:32.699 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.0330000000000008e-06 2024-09-13T11:22:32.699 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:32.699 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:32.699 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:22:32.699 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:28.337864+0000", 2024-09-13T11:22:32.699 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.9500000000000003e-05 2024-09-13T11:22:32.699 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:32.699 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:32.699 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:22:32.700 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:28.337884+0000", 2024-09-13T11:22:32.700 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.0026000000000001e-05 2024-09-13T11:22:32.700 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:32.700 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:32.700 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:22:32.700 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:28.337975+0000", 2024-09-13T11:22:32.700 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.1051999999999995e-05 2024-09-13T11:22:32.700 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:32.700 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:32.700 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:22:32.700 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:28.337990+0000", 2024-09-13T11:22:32.700 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.4936999999999999e-05 2024-09-13T11:22:32.700 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:32.700 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:32.701 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:22:32.701 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:28.338170+0000", 2024-09-13T11:22:32.701 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000180204 2024-09-13T11:22:32.701 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:22:32.701 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:22:32.701 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:22:32.701 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:22:32.701 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:22:32.701 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-13T11:22:32.703 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_ops_in_flight 2024-09-13T11:22:32.703 INFO:tasks.ceph.osd.10.smithi162.stderr:2024-09-13T11:22:32.700+0000 7fb59c9f0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26317) UID: 0 2024-09-13T11:22:32.804 INFO:tasks.ceph.osd.7.smithi177.stderr:2024-09-13T11:22:32.800+0000 7f897b51d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26575) UID: 0 2024-09-13T11:22:32.834 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-13T11:22:32.834 INFO:teuthology.orchestra.run.smithi001.stdout: "ops": [], 2024-09-13T11:22:32.834 INFO:teuthology.orchestra.run.smithi001.stdout: "num_ops": 0 2024-09-13T11:22:32.834 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-13T11:22:32.849 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_blocked_ops 2024-09-13T11:22:32.905 INFO:tasks.ceph.osd.8.smithi001.stderr:2024-09-13T11:22:32.900+0000 7fb9b48ae640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26312) UID: 0 2024-09-13T11:22:32.973 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-13T11:22:32.973 INFO:teuthology.orchestra.run.smithi001.stdout: "ops": [], 2024-09-13T11:22:32.973 INFO:teuthology.orchestra.run.smithi001.stdout: "complaint_time": 30, 2024-09-13T11:22:32.973 INFO:teuthology.orchestra.run.smithi001.stdout: "num_blocked_ops": 0 2024-09-13T11:22:32.974 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-13T11:22:32.988 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_historic_ops 2024-09-13T11:22:32.996 INFO:tasks.radosbench.radosbench.0.smithi177.stdout: 6 16 87 71 0.739484 0 - 0.0234801 2024-09-13T11:22:33.005 INFO:tasks.ceph.osd.13.smithi092.stderr:2024-09-13T11:22:33.003+0000 7f3e5afea640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26350) UID: 0 2024-09-13T11:22:33.107 INFO:tasks.ceph.osd.15.smithi177.stderr:2024-09-13T11:22:33.104+0000 7f9d6ce77640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26578) UID: 0 2024-09-13T11:22:33.161 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-13T11:22:33.161 INFO:teuthology.orchestra.run.smithi001.stdout: "size": 20, 2024-09-13T11:22:33.161 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 600, 2024-09-13T11:22:33.161 INFO:teuthology.orchestra.run.smithi001.stdout: "ops": [ 2024-09-13T11:22:33.161 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:33.161 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:1 3.0 3:0f48e989:::benchmark_data_smithi177_31116_object0:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-09-13T11:22:33.161 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:22:27.000006+0000", 2024-09-13T11:22:33.162 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 6.1583775080000001, 2024-09-13T11:22:33.162 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0051292489999999998, 2024-09-13T11:22:33.162 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:22:33.162 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:22:33.162 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:33.162 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:22:33.162 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:22:33.162 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:22:33.162 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 1 2024-09-13T11:22:33.163 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:33.163 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:22:33.163 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:33.163 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:22:33.163 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.000006+0000", 2024-09-13T11:22:33.163 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:22:33.163 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:33.163 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:33.164 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:22:33.164 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.000006+0000", 2024-09-13T11:22:33.164 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:22:33.164 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:33.164 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:33.164 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:22:33.164 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.000017+0000", 2024-09-13T11:22:33.164 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0613e-05 2024-09-13T11:22:33.164 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:33.164 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:33.164 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:22:33.164 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.000246+0000", 2024-09-13T11:22:33.164 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000229313 2024-09-13T11:22:33.165 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:33.165 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:33.165 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:22:33.165 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.000278+0000", 2024-09-13T11:22:33.165 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.1933000000000001e-05 2024-09-13T11:22:33.165 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:33.165 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:33.165 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:22:33.165 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.000313+0000", 2024-09-13T11:22:33.165 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.4548000000000002e-05 2024-09-13T11:22:33.165 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:33.165 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:33.166 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:22:33.166 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.000401+0000", 2024-09-13T11:22:33.166 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.7800000000000006e-05 2024-09-13T11:22:33.166 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:33.166 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:33.166 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op must wait for map", 2024-09-13T11:22:33.166 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.000407+0000", 2024-09-13T11:22:33.166 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.0519999999999998e-06 2024-09-13T11:22:33.167 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:33.167 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:33.167 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:22:33.167 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.002663+0000", 2024-09-13T11:22:33.167 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0022562720000000001 2024-09-13T11:22:33.167 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:33.167 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:33.167 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:22:33.168 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.002862+0000", 2024-09-13T11:22:33.168 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00019919100000000001 2024-09-13T11:22:33.168 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:33.168 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:33.168 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 15", 2024-09-13T11:22:33.168 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.003014+0000", 2024-09-13T11:22:33.168 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00015144299999999999 2024-09-13T11:22:33.168 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:33.168 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:33.168 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:22:33.168 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.004754+0000", 2024-09-13T11:22:33.168 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.001740103 2024-09-13T11:22:33.168 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:33.169 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:33.169 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:22:33.169 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.005060+0000", 2024-09-13T11:22:33.169 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00030597799999999999 2024-09-13T11:22:33.169 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:33.169 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:33.169 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:22:33.169 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.005101+0000", 2024-09-13T11:22:33.169 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.1618e-05 2024-09-13T11:22:33.169 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:33.169 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:33.169 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:22:33.169 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.005136+0000", 2024-09-13T11:22:33.169 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.4384999999999997e-05 2024-09-13T11:22:33.170 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:22:33.170 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:22:33.170 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:22:33.170 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:33.170 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:33.170 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:3 3.4 3.125e09e4 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-09-13T11:22:33.170 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:22:27.000344+0000", 2024-09-13T11:22:33.170 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 6.1580401949999999, 2024-09-13T11:22:33.170 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0022646469999999998, 2024-09-13T11:22:33.170 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:22:33.170 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:22:33.170 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-13T11:22:33.170 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:22:33.171 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:22:33.171 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:22:33.171 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 3 2024-09-13T11:22:33.171 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:33.171 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:22:33.171 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:33.171 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:22:33.171 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.000344+0000", 2024-09-13T11:22:33.171 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:22:33.171 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:33.171 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:33.171 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:22:33.171 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.000344+0000", 2024-09-13T11:22:33.171 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:22:33.172 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:33.172 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:33.172 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:22:33.172 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.000345+0000", 2024-09-13T11:22:33.172 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.043e-06 2024-09-13T11:22:33.172 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:33.172 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:33.172 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:22:33.172 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.000436+0000", 2024-09-13T11:22:33.172 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.0839000000000002e-05 2024-09-13T11:22:33.172 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:33.172 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:33.172 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:22:33.172 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.000455+0000", 2024-09-13T11:22:33.173 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.9670999999999999e-05 2024-09-13T11:22:33.173 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:33.173 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:33.173 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:22:33.173 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.000463+0000", 2024-09-13T11:22:33.173 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.9319999999999996e-06 2024-09-13T11:22:33.173 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:33.173 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:33.173 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:22:33.173 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.000528+0000", 2024-09-13T11:22:33.173 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.4974999999999994e-05 2024-09-13T11:22:33.173 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:33.173 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:33.173 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op must wait for map", 2024-09-13T11:22:33.174 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.000532+0000", 2024-09-13T11:22:33.174 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.1080000000000003e-06 2024-09-13T11:22:33.174 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:33.174 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:33.174 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:22:33.174 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.002553+0000", 2024-09-13T11:22:33.174 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0020209640000000001 2024-09-13T11:22:33.174 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:33.174 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:33.174 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:22:33.174 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.002608+0000", 2024-09-13T11:22:33.174 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.5114999999999997e-05 2024-09-13T11:22:33.174 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:22:33.174 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:22:33.175 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:22:33.175 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:33.175 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:33.175 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:12 3.9 3:9dad78b7:::benchmark_data_smithi177_31116_object11:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-09-13T11:22:33.175 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:22:27.000484+0000", 2024-09-13T11:22:33.175 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 6.1578998900000004, 2024-09-13T11:22:33.175 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.007402502, 2024-09-13T11:22:33.175 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:22:33.175 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:22:33.175 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:33.175 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:22:33.175 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:22:33.175 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:22:33.175 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 12 2024-09-13T11:22:33.176 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:33.176 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:22:33.176 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:33.176 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:22:33.176 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.000484+0000", 2024-09-13T11:22:33.176 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:22:33.176 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:33.176 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:33.176 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:22:33.176 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.000484+0000", 2024-09-13T11:22:33.176 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:22:33.176 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:33.176 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:33.176 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:22:33.177 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.000485+0000", 2024-09-13T11:22:33.177 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.274e-06 2024-09-13T11:22:33.177 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:33.177 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:33.177 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:22:33.177 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.000632+0000", 2024-09-13T11:22:33.177 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000146756 2024-09-13T11:22:33.177 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:33.177 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:33.177 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:22:33.177 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.000701+0000", 2024-09-13T11:22:33.177 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.9111999999999998e-05 2024-09-13T11:22:33.177 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:33.178 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:33.178 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:22:33.178 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.000713+0000", 2024-09-13T11:22:33.178 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.1552000000000001e-05 2024-09-13T11:22:33.178 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:33.178 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:33.178 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:22:33.178 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.000784+0000", 2024-09-13T11:22:33.178 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.1317999999999997e-05 2024-09-13T11:22:33.178 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:33.178 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:33.178 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op must wait for map", 2024-09-13T11:22:33.178 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.000789+0000", 2024-09-13T11:22:33.178 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.147e-06 2024-09-13T11:22:33.179 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:33.179 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:33.179 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:22:33.179 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.002573+0000", 2024-09-13T11:22:33.179 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.001783532 2024-09-13T11:22:33.179 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:33.179 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:33.179 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:22:33.179 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.002755+0000", 2024-09-13T11:22:33.179 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00018205200000000001 2024-09-13T11:22:33.179 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:33.179 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:33.179 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 2", 2024-09-13T11:22:33.179 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.002959+0000", 2024-09-13T11:22:33.180 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000204358 2024-09-13T11:22:33.180 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:33.180 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:33.180 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:22:33.180 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.004983+0000", 2024-09-13T11:22:33.180 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0020237570000000002 2024-09-13T11:22:33.180 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:33.180 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:33.180 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:22:33.180 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.007718+0000", 2024-09-13T11:22:33.180 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.002734963 2024-09-13T11:22:33.180 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:33.180 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:33.181 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:22:33.181 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.007783+0000", 2024-09-13T11:22:33.181 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.4851000000000006e-05 2024-09-13T11:22:33.181 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:33.181 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:33.181 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:22:33.181 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.007887+0000", 2024-09-13T11:22:33.181 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00010383 2024-09-13T11:22:33.181 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:22:33.181 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:22:33.181 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:22:33.181 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:33.181 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:33.181 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:13 3.9 3:967a7538:::benchmark_data_smithi177_31116_object12:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-09-13T11:22:33.182 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:22:27.000738+0000", 2024-09-13T11:22:33.182 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 6.1576461279999997, 2024-09-13T11:22:33.182 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.011803163, 2024-09-13T11:22:33.182 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:22:33.182 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:22:33.182 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:33.182 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:22:33.182 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:22:33.182 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:22:33.182 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 13 2024-09-13T11:22:33.182 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:33.182 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:22:33.182 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:33.182 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:22:33.183 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.000738+0000", 2024-09-13T11:22:33.183 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:22:33.183 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:33.183 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:33.183 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:22:33.183 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.000738+0000", 2024-09-13T11:22:33.183 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:22:33.183 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:33.183 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:33.183 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:22:33.183 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.000739+0000", 2024-09-13T11:22:33.183 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.652e-06 2024-09-13T11:22:33.183 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:33.183 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:33.184 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:22:33.184 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.000823+0000", 2024-09-13T11:22:33.184 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.3197999999999999e-05 2024-09-13T11:22:33.184 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:33.184 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:33.184 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:22:33.184 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.000836+0000", 2024-09-13T11:22:33.184 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.327e-05 2024-09-13T11:22:33.184 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:33.184 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:33.184 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:22:33.184 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.000842+0000", 2024-09-13T11:22:33.184 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.3160000000000002e-06 2024-09-13T11:22:33.185 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:33.185 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:33.185 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:22:33.185 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.000881+0000", 2024-09-13T11:22:33.185 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.8726999999999997e-05 2024-09-13T11:22:33.185 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:33.185 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:33.185 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting_for_map not empty", 2024-09-13T11:22:33.185 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.000884+0000", 2024-09-13T11:22:33.185 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.2770000000000001e-06 2024-09-13T11:22:33.185 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:33.185 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:33.185 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:22:33.185 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.003919+0000", 2024-09-13T11:22:33.186 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0030346750000000001 2024-09-13T11:22:33.186 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:33.186 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:33.186 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:22:33.186 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.004109+0000", 2024-09-13T11:22:33.186 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00019011500000000001 2024-09-13T11:22:33.186 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:33.186 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:33.186 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 2", 2024-09-13T11:22:33.186 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.004351+0000", 2024-09-13T11:22:33.186 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00024179099999999999 2024-09-13T11:22:33.186 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:33.186 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:33.186 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:22:33.187 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.005708+0000", 2024-09-13T11:22:33.187 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0013570539999999999 2024-09-13T11:22:33.187 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:33.187 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:33.187 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:22:33.187 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.012438+0000", 2024-09-13T11:22:33.187 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0067301210000000004 2024-09-13T11:22:33.187 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:33.187 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:33.187 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:22:33.187 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.012495+0000", 2024-09-13T11:22:33.187 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.7410000000000001e-05 2024-09-13T11:22:33.187 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:33.187 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:33.188 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:22:33.188 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.012541+0000", 2024-09-13T11:22:33.188 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.5556999999999998e-05 2024-09-13T11:22:33.188 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:22:33.188 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:22:33.188 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:22:33.188 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:33.188 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:33.188 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:14 3.0 3:047c9174:::benchmark_data_smithi177_31116_object13:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-09-13T11:22:33.188 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:22:27.000856+0000", 2024-09-13T11:22:33.188 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 6.1575280939999999, 2024-09-13T11:22:33.188 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0064605549999999998, 2024-09-13T11:22:33.188 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:22:33.189 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:22:33.189 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:33.189 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:22:33.189 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:22:33.189 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:22:33.189 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 14 2024-09-13T11:22:33.189 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:33.189 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:22:33.189 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:33.189 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:22:33.189 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.000856+0000", 2024-09-13T11:22:33.189 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:22:33.189 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:33.189 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:33.190 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:22:33.190 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.000856+0000", 2024-09-13T11:22:33.190 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:22:33.190 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:33.190 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:33.190 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:22:33.190 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.000857+0000", 2024-09-13T11:22:33.190 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2330000000000001e-06 2024-09-13T11:22:33.190 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:33.190 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:33.190 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:22:33.190 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.000925+0000", 2024-09-13T11:22:33.190 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.8393999999999996e-05 2024-09-13T11:22:33.190 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:33.191 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:33.191 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:22:33.191 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.000935+0000", 2024-09-13T11:22:33.191 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.4069999999999999e-06 2024-09-13T11:22:33.191 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:33.191 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:33.191 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:22:33.191 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.000941+0000", 2024-09-13T11:22:33.191 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.3799999999999999e-06 2024-09-13T11:22:33.191 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:33.191 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:33.191 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:22:33.191 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.000992+0000", 2024-09-13T11:22:33.192 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.0732999999999997e-05 2024-09-13T11:22:33.192 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:33.192 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:33.192 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting_for_map not empty", 2024-09-13T11:22:33.192 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.000995+0000", 2024-09-13T11:22:33.192 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.354e-06 2024-09-13T11:22:33.192 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:33.192 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:33.192 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:22:33.192 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.003988+0000", 2024-09-13T11:22:33.192 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0029929119999999999 2024-09-13T11:22:33.192 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:33.192 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:33.193 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:22:33.193 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.004102+0000", 2024-09-13T11:22:33.193 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000113501 2024-09-13T11:22:33.193 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:33.193 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:33.193 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 15", 2024-09-13T11:22:33.193 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.004227+0000", 2024-09-13T11:22:33.193 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00012559499999999999 2024-09-13T11:22:33.193 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:33.193 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:33.193 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:22:33.193 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.005668+0000", 2024-09-13T11:22:33.193 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.001440216 2024-09-13T11:22:33.194 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:33.194 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:33.194 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:22:33.194 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.007230+0000", 2024-09-13T11:22:33.194 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0015623200000000001 2024-09-13T11:22:33.194 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:33.194 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:33.194 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:22:33.194 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.007278+0000", 2024-09-13T11:22:33.194 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.7930999999999999e-05 2024-09-13T11:22:33.194 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:33.194 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:33.194 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:22:33.194 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.007316+0000", 2024-09-13T11:22:33.195 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.8578999999999998e-05 2024-09-13T11:22:33.195 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:22:33.195 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:22:33.195 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:22:33.195 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:33.195 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:33.195 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4665.0:11 3.f e32/30 3:f7f390ac:::benchmark_data_smithi177_31116_object10:head v 32'1, mlcod=0'0)", 2024-09-13T11:22:33.195 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:22:27.005110+0000", 2024-09-13T11:22:33.195 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 6.1532738580000004, 2024-09-13T11:22:33.195 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.001113588, 2024-09-13T11:22:33.195 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:22:33.195 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:22:33.195 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:33.195 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:22:33.196 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:33.196 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:22:33.196 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.005110+0000", 2024-09-13T11:22:33.196 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:22:33.196 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:33.196 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:33.196 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:22:33.196 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.005110+0000", 2024-09-13T11:22:33.196 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:22:33.196 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:33.196 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:33.196 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:22:33.196 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.005111+0000", 2024-09-13T11:22:33.196 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.0500000000000003e-07 2024-09-13T11:22:33.197 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:33.197 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:33.197 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:22:33.197 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.005306+0000", 2024-09-13T11:22:33.197 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00019570500000000001 2024-09-13T11:22:33.197 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:33.197 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:33.197 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:22:33.197 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.005345+0000", 2024-09-13T11:22:33.197 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.8618000000000002e-05 2024-09-13T11:22:33.197 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:33.197 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:33.197 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:22:33.197 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.005354+0000", 2024-09-13T11:22:33.198 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.9129999999999992e-06 2024-09-13T11:22:33.198 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:33.198 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:33.198 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:22:33.198 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.005424+0000", 2024-09-13T11:22:33.198 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.9579000000000006e-05 2024-09-13T11:22:33.198 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:33.198 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:33.198 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:22:33.198 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.005439+0000", 2024-09-13T11:22:33.198 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.5653000000000001e-05 2024-09-13T11:22:33.198 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:33.198 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:33.198 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:22:33.199 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.006183+0000", 2024-09-13T11:22:33.199 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00074377099999999995 2024-09-13T11:22:33.199 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:33.199 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:33.199 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:22:33.199 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.006224+0000", 2024-09-13T11:22:33.199 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.0744000000000002e-05 2024-09-13T11:22:33.199 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:22:33.199 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:22:33.199 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:22:33.199 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:33.199 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:33.199 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4665.0:2 3.2 e32/30 3:438ada23:::benchmark_data_smithi177_31116_object1:head v 32'1, mlcod=0'0)", 2024-09-13T11:22:33.200 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:22:27.007858+0000", 2024-09-13T11:22:33.200 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 6.1505264090000002, 2024-09-13T11:22:33.200 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0011826059999999999, 2024-09-13T11:22:33.200 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:22:33.200 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:22:33.200 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:33.200 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:22:33.200 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:33.200 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:22:33.200 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.007858+0000", 2024-09-13T11:22:33.200 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:22:33.200 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:33.200 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:33.200 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:22:33.201 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.007858+0000", 2024-09-13T11:22:33.201 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:22:33.201 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:33.201 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:33.201 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:22:33.201 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.007859+0000", 2024-09-13T11:22:33.201 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.3999999999999999e-06 2024-09-13T11:22:33.201 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:33.201 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:33.201 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:22:33.201 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.008086+0000", 2024-09-13T11:22:33.201 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000226579 2024-09-13T11:22:33.201 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:33.202 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:33.202 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:22:33.202 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.008117+0000", 2024-09-13T11:22:33.202 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.1807000000000002e-05 2024-09-13T11:22:33.202 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:33.202 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:33.202 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:22:33.202 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.008129+0000", 2024-09-13T11:22:33.202 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.136e-05 2024-09-13T11:22:33.202 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:33.202 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:33.202 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:22:33.202 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.008200+0000", 2024-09-13T11:22:33.202 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.1369000000000001e-05 2024-09-13T11:22:33.203 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:33.203 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:33.203 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:22:33.203 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.008217+0000", 2024-09-13T11:22:33.203 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.7396000000000001e-05 2024-09-13T11:22:33.203 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:33.203 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:33.203 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:22:33.203 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.009000+0000", 2024-09-13T11:22:33.203 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00078246800000000003 2024-09-13T11:22:33.203 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:33.203 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:33.203 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:22:33.203 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.009040+0000", 2024-09-13T11:22:33.204 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.0227e-05 2024-09-13T11:22:33.204 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:22:33.204 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:22:33.204 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:22:33.204 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:33.204 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:33.204 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4665.0:27 3.f e32/30 3:fc76794e:::benchmark_data_smithi177_31116_object26:head v 32'2, mlcod=0'0)", 2024-09-13T11:22:33.204 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:22:27.011594+0000", 2024-09-13T11:22:33.204 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 6.1467896719999997, 2024-09-13T11:22:33.204 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00099986799999999989, 2024-09-13T11:22:33.204 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:22:33.204 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:22:33.204 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:33.204 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:22:33.205 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:33.205 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:22:33.205 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.011594+0000", 2024-09-13T11:22:33.205 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:22:33.205 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:33.205 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:33.205 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:22:33.205 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.011594+0000", 2024-09-13T11:22:33.205 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:22:33.205 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:33.205 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:33.205 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:22:33.205 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.011596+0000", 2024-09-13T11:22:33.206 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.753e-06 2024-09-13T11:22:33.206 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:33.206 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:33.206 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:22:33.206 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.011657+0000", 2024-09-13T11:22:33.206 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.0587000000000001e-05 2024-09-13T11:22:33.206 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:33.206 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:33.206 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:22:33.206 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.011696+0000", 2024-09-13T11:22:33.206 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.8874000000000001e-05 2024-09-13T11:22:33.206 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:33.206 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:33.206 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:22:33.207 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.011704+0000", 2024-09-13T11:22:33.207 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.5010000000000004e-06 2024-09-13T11:22:33.207 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:33.207 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:33.207 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:22:33.207 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.011759+0000", 2024-09-13T11:22:33.207 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.4580999999999998e-05 2024-09-13T11:22:33.207 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:33.207 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:33.207 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:22:33.207 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.011775+0000", 2024-09-13T11:22:33.207 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.6348e-05 2024-09-13T11:22:33.207 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:33.208 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:33.208 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:22:33.208 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.012539+0000", 2024-09-13T11:22:33.208 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00076412599999999995 2024-09-13T11:22:33.208 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:33.208 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:33.208 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:22:33.208 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.012594+0000", 2024-09-13T11:22:33.208 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.5098e-05 2024-09-13T11:22:33.208 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:22:33.208 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:22:33.208 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:22:33.208 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:33.208 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:33.209 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:28 3.0 3:0c1f4d63:::benchmark_data_smithi177_31116_object27:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-09-13T11:22:33.209 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:22:27.011908+0000", 2024-09-13T11:22:33.209 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 6.146476281, 2024-09-13T11:22:33.209 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0018107970000000001, 2024-09-13T11:22:33.209 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:22:33.209 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:22:33.209 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:33.209 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:22:33.209 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:22:33.209 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:22:33.209 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 28 2024-09-13T11:22:33.209 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:33.209 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:22:33.209 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:33.210 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:22:33.210 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.011908+0000", 2024-09-13T11:22:33.210 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:22:33.210 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:33.210 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:33.210 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:22:33.210 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.011908+0000", 2024-09-13T11:22:33.210 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:22:33.210 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:33.210 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:33.210 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:22:33.210 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.011910+0000", 2024-09-13T11:22:33.210 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.413e-06 2024-09-13T11:22:33.210 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:33.211 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:33.211 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:22:33.211 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.012021+0000", 2024-09-13T11:22:33.211 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000111202 2024-09-13T11:22:33.211 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:33.211 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:33.211 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:22:33.211 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.012038+0000", 2024-09-13T11:22:33.211 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.6864e-05 2024-09-13T11:22:33.211 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:33.211 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:33.211 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:22:33.211 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.012050+0000", 2024-09-13T11:22:33.211 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.1391e-05 2024-09-13T11:22:33.212 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:33.212 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:33.212 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:22:33.212 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.012111+0000", 2024-09-13T11:22:33.212 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.1055000000000004e-05 2024-09-13T11:22:33.212 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:33.212 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:33.212 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:22:33.212 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.012226+0000", 2024-09-13T11:22:33.212 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000115658 2024-09-13T11:22:33.212 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:33.212 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:33.212 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 15", 2024-09-13T11:22:33.212 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.012376+0000", 2024-09-13T11:22:33.213 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00014960799999999999 2024-09-13T11:22:33.213 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:33.213 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:33.213 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:22:33.213 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.013446+0000", 2024-09-13T11:22:33.213 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.001069656 2024-09-13T11:22:33.213 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:33.213 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:33.213 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:22:33.213 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.013594+0000", 2024-09-13T11:22:33.213 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00014862799999999999 2024-09-13T11:22:33.213 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:33.213 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:33.213 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:22:33.214 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.013671+0000", 2024-09-13T11:22:33.214 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.6577999999999998e-05 2024-09-13T11:22:33.214 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:33.214 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:33.214 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:22:33.214 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.013718+0000", 2024-09-13T11:22:33.214 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.7744000000000002e-05 2024-09-13T11:22:33.214 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:22:33.214 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:22:33.214 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:22:33.214 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:33.214 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:33.214 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:37 3.9 3:962d44ea:::benchmark_data_smithi177_31116_object36:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-09-13T11:22:33.214 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:22:27.014759+0000", 2024-09-13T11:22:33.215 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 6.1436244779999996, 2024-09-13T11:22:33.215 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0038735110000000001, 2024-09-13T11:22:33.215 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:22:33.215 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:22:33.215 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:33.215 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:22:33.215 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:22:33.215 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:22:33.215 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 37 2024-09-13T11:22:33.215 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:33.215 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:22:33.215 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:33.215 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:22:33.216 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.014759+0000", 2024-09-13T11:22:33.216 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:22:33.216 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:33.216 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:33.216 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:22:33.216 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.014759+0000", 2024-09-13T11:22:33.216 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:22:33.216 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:33.216 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:33.216 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:22:33.216 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.014762+0000", 2024-09-13T11:22:33.216 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.441e-06 2024-09-13T11:22:33.216 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:33.216 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:33.217 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:22:33.217 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.014809+0000", 2024-09-13T11:22:33.217 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.7237999999999997e-05 2024-09-13T11:22:33.217 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:33.217 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:33.217 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:22:33.217 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.014828+0000", 2024-09-13T11:22:33.217 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.9182e-05 2024-09-13T11:22:33.217 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:33.217 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:33.217 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:22:33.217 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.014848+0000", 2024-09-13T11:22:33.217 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.9567000000000001e-05 2024-09-13T11:22:33.217 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:33.218 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:33.218 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:22:33.218 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.014917+0000", 2024-09-13T11:22:33.218 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.9430000000000004e-05 2024-09-13T11:22:33.218 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:33.218 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:33.218 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:22:33.218 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.015019+0000", 2024-09-13T11:22:33.218 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000102007 2024-09-13T11:22:33.218 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:33.218 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:33.219 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 2", 2024-09-13T11:22:33.219 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.015140+0000", 2024-09-13T11:22:33.219 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000120732 2024-09-13T11:22:33.219 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:33.219 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:33.219 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:22:33.219 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.016113+0000", 2024-09-13T11:22:33.219 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00097306499999999995 2024-09-13T11:22:33.219 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:33.219 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:33.219 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:22:33.220 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.018476+0000", 2024-09-13T11:22:33.220 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0023628659999999999 2024-09-13T11:22:33.220 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:33.220 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:33.220 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:22:33.220 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.018574+0000", 2024-09-13T11:22:33.220 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.7930000000000006e-05 2024-09-13T11:22:33.220 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:33.220 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:33.220 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:22:33.220 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.018633+0000", 2024-09-13T11:22:33.220 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.9052999999999999e-05 2024-09-13T11:22:33.220 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:22:33.220 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:22:33.221 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:22:33.221 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:33.221 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:33.221 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:42 3.0 3:09c3a999:::benchmark_data_smithi177_31116_object41:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-09-13T11:22:33.221 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:22:27.018149+0000", 2024-09-13T11:22:33.221 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 6.1402348270000005, 2024-09-13T11:22:33.221 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0019727070000000002, 2024-09-13T11:22:33.221 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:22:33.221 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:22:33.221 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:33.221 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:22:33.221 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:22:33.221 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:22:33.221 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 42 2024-09-13T11:22:33.222 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:33.222 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:22:33.222 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:33.222 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:22:33.222 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.018149+0000", 2024-09-13T11:22:33.222 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:22:33.222 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:33.222 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:33.222 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:22:33.222 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.018149+0000", 2024-09-13T11:22:33.222 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:22:33.222 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:33.222 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:33.223 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:22:33.223 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.018151+0000", 2024-09-13T11:22:33.223 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.508e-06 2024-09-13T11:22:33.223 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:33.223 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:33.223 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:22:33.223 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.018213+0000", 2024-09-13T11:22:33.223 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.2137999999999994e-05 2024-09-13T11:22:33.223 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:33.223 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:33.223 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:22:33.223 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.018228+0000", 2024-09-13T11:22:33.223 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.5097e-05 2024-09-13T11:22:33.223 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:33.223 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:33.224 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:22:33.224 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.018236+0000", 2024-09-13T11:22:33.224 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.7840000000000004e-06 2024-09-13T11:22:33.224 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:33.224 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:33.224 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:22:33.224 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.018314+0000", 2024-09-13T11:22:33.224 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.7927000000000001e-05 2024-09-13T11:22:33.224 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:33.224 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:33.224 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:22:33.224 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.018519+0000", 2024-09-13T11:22:33.224 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00020553499999999999 2024-09-13T11:22:33.224 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:33.225 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:33.225 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 15", 2024-09-13T11:22:33.225 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.018698+0000", 2024-09-13T11:22:33.225 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00017922000000000001 2024-09-13T11:22:33.225 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:33.225 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:33.225 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:22:33.225 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.019580+0000", 2024-09-13T11:22:33.225 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00088205599999999999 2024-09-13T11:22:33.225 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:33.225 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:33.225 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:22:33.225 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.020016+0000", 2024-09-13T11:22:33.225 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00043519499999999998 2024-09-13T11:22:33.226 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:33.226 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:33.226 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:22:33.226 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.020082+0000", 2024-09-13T11:22:33.226 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.6783000000000001e-05 2024-09-13T11:22:33.226 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:33.226 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:33.226 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:22:33.226 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.020122+0000", 2024-09-13T11:22:33.226 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.9464e-05 2024-09-13T11:22:33.226 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:22:33.226 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:22:33.226 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:22:33.226 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:33.227 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:33.227 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4665.0:55 3.f e32/30 3:f1c4925a:::benchmark_data_smithi177_31116_object54:head v 32'4, mlcod=32'2)", 2024-09-13T11:22:33.227 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:22:27.025662+0000", 2024-09-13T11:22:33.227 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 6.1327220069999999, 2024-09-13T11:22:33.227 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0014268659999999999, 2024-09-13T11:22:33.227 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:22:33.227 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:22:33.227 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:33.227 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:22:33.227 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:33.227 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:22:33.227 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.025662+0000", 2024-09-13T11:22:33.227 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:22:33.228 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:33.228 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:33.228 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:22:33.228 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.025662+0000", 2024-09-13T11:22:33.228 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:22:33.228 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:33.228 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:33.228 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:22:33.228 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.025664+0000", 2024-09-13T11:22:33.228 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.823e-06 2024-09-13T11:22:33.228 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:33.228 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:33.228 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:22:33.228 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.025736+0000", 2024-09-13T11:22:33.229 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.2162000000000005e-05 2024-09-13T11:22:33.229 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:33.229 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:33.229 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:22:33.229 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.025780+0000", 2024-09-13T11:22:33.229 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.4161e-05 2024-09-13T11:22:33.229 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:33.229 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:33.229 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:22:33.229 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.025788+0000", 2024-09-13T11:22:33.229 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.2239999999999995e-06 2024-09-13T11:22:33.229 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:33.229 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:33.229 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:22:33.230 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.025850+0000", 2024-09-13T11:22:33.230 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.1428999999999997e-05 2024-09-13T11:22:33.230 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:33.230 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:33.230 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:22:33.230 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.025868+0000", 2024-09-13T11:22:33.230 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.7870999999999999e-05 2024-09-13T11:22:33.230 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:33.230 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:33.230 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:22:33.230 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.027017+0000", 2024-09-13T11:22:33.230 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.001149274 2024-09-13T11:22:33.230 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:33.230 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:33.231 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:22:33.231 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.027089+0000", 2024-09-13T11:22:33.231 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.1921999999999993e-05 2024-09-13T11:22:33.231 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:22:33.231 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:22:33.231 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:22:33.231 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:33.231 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:33.231 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:59 3.9 3:9499e3d6:::benchmark_data_smithi177_31116_object58:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-09-13T11:22:33.231 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:22:27.026411+0000", 2024-09-13T11:22:33.231 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 6.1319731370000001, 2024-09-13T11:22:33.231 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0025727300000000001, 2024-09-13T11:22:33.232 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:22:33.232 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:22:33.232 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:33.232 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:22:33.232 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:22:33.232 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:22:33.232 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 59 2024-09-13T11:22:33.232 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:33.232 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:22:33.232 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:33.232 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:22:33.232 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.026411+0000", 2024-09-13T11:22:33.232 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:22:33.233 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:33.233 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:33.233 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:22:33.233 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.026411+0000", 2024-09-13T11:22:33.233 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:22:33.233 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:33.233 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:33.233 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:22:33.233 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.026413+0000", 2024-09-13T11:22:33.233 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.193e-06 2024-09-13T11:22:33.233 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:33.233 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:33.233 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:22:33.234 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.026474+0000", 2024-09-13T11:22:33.234 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.1271000000000003e-05 2024-09-13T11:22:33.234 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:33.234 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:33.234 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:22:33.234 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.026488+0000", 2024-09-13T11:22:33.234 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.3606e-05 2024-09-13T11:22:33.234 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:33.234 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:33.234 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:22:33.234 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.026496+0000", 2024-09-13T11:22:33.234 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.7309999999999993e-06 2024-09-13T11:22:33.234 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:33.234 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:33.235 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:22:33.235 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.026568+0000", 2024-09-13T11:22:33.235 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.2094000000000005e-05 2024-09-13T11:22:33.235 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:33.235 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:33.235 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:22:33.235 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.026703+0000", 2024-09-13T11:22:33.235 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000134919 2024-09-13T11:22:33.235 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:33.235 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:33.235 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 2", 2024-09-13T11:22:33.235 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.026846+0000", 2024-09-13T11:22:33.235 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000143134 2024-09-13T11:22:33.235 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:33.236 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:33.236 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:22:33.236 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.027924+0000", 2024-09-13T11:22:33.236 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0010780290000000001 2024-09-13T11:22:33.236 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:33.236 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:33.236 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:22:33.236 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.028904+0000", 2024-09-13T11:22:33.236 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00098057599999999993 2024-09-13T11:22:33.236 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:33.236 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:33.236 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:22:33.236 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.028948+0000", 2024-09-13T11:22:33.236 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.3541000000000002e-05 2024-09-13T11:22:33.237 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:33.237 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:33.237 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:22:33.237 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.028984+0000", 2024-09-13T11:22:33.237 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.5636000000000002e-05 2024-09-13T11:22:33.237 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:22:33.237 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:22:33.237 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:22:33.237 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:33.237 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:33.237 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:65 3.0 3:01909ae6:::benchmark_data_smithi177_31116_object64:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-09-13T11:22:33.237 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:22:27.029786+0000", 2024-09-13T11:22:33.237 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 6.1285979360000002, 2024-09-13T11:22:33.237 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0020071550000000001, 2024-09-13T11:22:33.238 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:22:33.238 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:22:33.238 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:33.238 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:22:33.238 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:22:33.238 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:22:33.238 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 65 2024-09-13T11:22:33.238 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:33.238 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:22:33.238 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:33.238 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:22:33.238 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.029786+0000", 2024-09-13T11:22:33.239 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:22:33.239 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:33.239 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:33.239 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:22:33.239 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.029786+0000", 2024-09-13T11:22:33.239 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:22:33.239 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:33.239 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:33.239 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:22:33.239 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.029788+0000", 2024-09-13T11:22:33.239 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.8309999999999999e-06 2024-09-13T11:22:33.239 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:33.239 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:33.239 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:22:33.240 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.029880+0000", 2024-09-13T11:22:33.240 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.2053999999999997e-05 2024-09-13T11:22:33.240 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:33.240 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:33.240 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:22:33.240 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.029896+0000", 2024-09-13T11:22:33.240 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.6082e-05 2024-09-13T11:22:33.240 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:33.240 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:33.241 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:22:33.241 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.029904+0000", 2024-09-13T11:22:33.241 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.0839999999999992e-06 2024-09-13T11:22:33.241 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:33.241 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:33.241 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:22:33.241 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.029974+0000", 2024-09-13T11:22:33.241 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.0026999999999999e-05 2024-09-13T11:22:33.241 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:33.241 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:33.241 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:22:33.241 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.030075+0000", 2024-09-13T11:22:33.241 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000100821 2024-09-13T11:22:33.241 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:33.242 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:33.242 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 15", 2024-09-13T11:22:33.242 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.030177+0000", 2024-09-13T11:22:33.242 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000102347 2024-09-13T11:22:33.242 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:33.242 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:33.242 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:22:33.242 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.030984+0000", 2024-09-13T11:22:33.242 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00080681600000000002 2024-09-13T11:22:33.242 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:33.242 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:33.242 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:22:33.242 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.031647+0000", 2024-09-13T11:22:33.243 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00066247599999999995 2024-09-13T11:22:33.243 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:33.243 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:33.243 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:22:33.243 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.031729+0000", 2024-09-13T11:22:33.243 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.2888999999999999e-05 2024-09-13T11:22:33.243 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:33.243 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:33.243 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:22:33.243 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.031793+0000", 2024-09-13T11:22:33.244 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.3727999999999997e-05 2024-09-13T11:22:33.244 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:22:33.244 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:22:33.244 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:22:33.244 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:33.244 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:33.244 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4665.0:66 3.2 e32/30 3:4ab2a1f9:::benchmark_data_smithi177_31116_object65:head v 32'3, mlcod=32'1)", 2024-09-13T11:22:33.244 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:22:27.030909+0000", 2024-09-13T11:22:33.244 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 6.1274753899999999, 2024-09-13T11:22:33.244 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0010672559999999999, 2024-09-13T11:22:33.244 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:22:33.244 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:22:33.244 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:33.245 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:22:33.245 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:33.245 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:22:33.245 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.030909+0000", 2024-09-13T11:22:33.245 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:22:33.245 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:33.245 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:33.245 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:22:33.245 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.030909+0000", 2024-09-13T11:22:33.245 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:22:33.245 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:33.245 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:33.245 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:22:33.246 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.030911+0000", 2024-09-13T11:22:33.246 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.7870000000000002e-06 2024-09-13T11:22:33.246 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:33.246 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:33.246 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:22:33.246 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.031038+0000", 2024-09-13T11:22:33.246 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000126711 2024-09-13T11:22:33.246 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:33.246 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:33.246 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:22:33.246 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.031126+0000", 2024-09-13T11:22:33.246 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.8082000000000001e-05 2024-09-13T11:22:33.246 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:33.246 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:33.247 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:22:33.247 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.031145+0000", 2024-09-13T11:22:33.247 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.8583000000000001e-05 2024-09-13T11:22:33.247 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:33.247 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:33.247 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:22:33.247 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.031228+0000", 2024-09-13T11:22:33.247 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.3511999999999996e-05 2024-09-13T11:22:33.247 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:33.247 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:33.247 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:22:33.247 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.031245+0000", 2024-09-13T11:22:33.248 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.6552999999999999e-05 2024-09-13T11:22:33.248 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:33.248 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:33.248 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:22:33.248 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.031927+0000", 2024-09-13T11:22:33.248 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00068196200000000004 2024-09-13T11:22:33.248 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:33.248 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:33.249 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:22:33.249 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.031976+0000", 2024-09-13T11:22:33.249 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.9066000000000001e-05 2024-09-13T11:22:33.249 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:22:33.249 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:22:33.249 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:22:33.249 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:33.249 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:33.249 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4665.0:71 3.2 e32/30 3:44c580a0:::benchmark_data_smithi177_31116_object70:head v 32'4, mlcod=32'2)", 2024-09-13T11:22:33.250 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:22:27.035414+0000", 2024-09-13T11:22:33.250 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 6.1229699210000001, 2024-09-13T11:22:33.250 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.001027977, 2024-09-13T11:22:33.250 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:22:33.250 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:22:33.250 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:33.250 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:22:33.250 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:33.250 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:22:33.250 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.035414+0000", 2024-09-13T11:22:33.250 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:22:33.250 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:33.250 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:33.251 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:22:33.251 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.035414+0000", 2024-09-13T11:22:33.251 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:22:33.251 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:33.251 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:33.251 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:22:33.251 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.035417+0000", 2024-09-13T11:22:33.251 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.8100000000000002e-06 2024-09-13T11:22:33.251 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:33.251 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:33.251 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:22:33.251 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.035507+0000", 2024-09-13T11:22:33.251 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.9808999999999993e-05 2024-09-13T11:22:33.252 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:33.252 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:33.252 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:22:33.252 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.035563+0000", 2024-09-13T11:22:33.252 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.6215999999999998e-05 2024-09-13T11:22:33.252 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:33.252 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:33.252 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:22:33.252 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.035583+0000", 2024-09-13T11:22:33.252 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.0503e-05 2024-09-13T11:22:33.252 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:33.252 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:33.253 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:22:33.253 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.035673+0000", 2024-09-13T11:22:33.253 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.0019999999999995e-05 2024-09-13T11:22:33.253 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:33.253 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:33.253 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:22:33.253 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.035693+0000", 2024-09-13T11:22:33.253 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.9253999999999999e-05 2024-09-13T11:22:33.253 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:33.253 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:33.253 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:22:33.253 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.036400+0000", 2024-09-13T11:22:33.253 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00070699499999999998 2024-09-13T11:22:33.254 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:33.254 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:33.254 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:22:33.254 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.036442+0000", 2024-09-13T11:22:33.254 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.2370000000000003e-05 2024-09-13T11:22:33.254 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:22:33.254 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:22:33.254 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:22:33.254 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:33.254 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:33.254 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4665.0:75 3.2 e32/30 3:415872a1:::benchmark_data_smithi177_31116_object74:head v 32'5, mlcod=32'3)", 2024-09-13T11:22:33.254 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:22:27.040760+0000", 2024-09-13T11:22:33.254 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 6.117624342, 2024-09-13T11:22:33.255 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0011927999999999999, 2024-09-13T11:22:33.255 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:22:33.255 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:22:33.255 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:33.255 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:22:33.255 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:33.255 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:22:33.255 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.040760+0000", 2024-09-13T11:22:33.255 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:22:33.255 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:33.255 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:33.255 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:22:33.255 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.040760+0000", 2024-09-13T11:22:33.255 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:22:33.256 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:33.256 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:33.256 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:22:33.256 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.040762+0000", 2024-09-13T11:22:33.256 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.9300000000000002e-06 2024-09-13T11:22:33.256 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:33.256 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:33.256 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:22:33.256 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.040854+0000", 2024-09-13T11:22:33.256 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.2145999999999995e-05 2024-09-13T11:22:33.256 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:33.256 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:33.256 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:22:33.257 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.040908+0000", 2024-09-13T11:22:33.257 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.3922000000000003e-05 2024-09-13T11:22:33.257 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:33.257 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:33.257 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:22:33.257 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.040924+0000", 2024-09-13T11:22:33.257 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.5947000000000002e-05 2024-09-13T11:22:33.257 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:33.257 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:33.257 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:22:33.257 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.041062+0000", 2024-09-13T11:22:33.257 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00013851400000000001 2024-09-13T11:22:33.257 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:33.258 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:33.258 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:22:33.258 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.041100+0000", 2024-09-13T11:22:33.258 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.8349999999999997e-05 2024-09-13T11:22:33.258 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:33.258 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:33.258 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:22:33.258 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.041917+0000", 2024-09-13T11:22:33.258 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000816689 2024-09-13T11:22:33.258 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:33.258 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:33.258 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:22:33.258 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.041952+0000", 2024-09-13T11:22:33.259 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.5302000000000002e-05 2024-09-13T11:22:33.259 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:22:33.259 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:22:33.259 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:22:33.259 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:33.259 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:33.259 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:76 3.9 3:9e00a8c6:::benchmark_data_smithi177_31116_object75:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-09-13T11:22:33.259 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:22:27.042773+0000", 2024-09-13T11:22:33.259 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 6.1156111490000002, 2024-09-13T11:22:33.259 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0027323410000000001, 2024-09-13T11:22:33.259 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:22:33.259 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:22:33.259 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:33.259 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:22:33.260 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:22:33.260 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:22:33.260 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 76 2024-09-13T11:22:33.260 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:33.260 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:22:33.260 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:33.260 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:22:33.260 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.042773+0000", 2024-09-13T11:22:33.260 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:22:33.260 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:33.260 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:33.260 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:22:33.260 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.042773+0000", 2024-09-13T11:22:33.261 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:22:33.261 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:33.261 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:33.261 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:22:33.261 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.042776+0000", 2024-09-13T11:22:33.261 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.8619999999999999e-06 2024-09-13T11:22:33.261 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:33.261 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:33.261 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:22:33.261 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.042812+0000", 2024-09-13T11:22:33.261 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.591e-05 2024-09-13T11:22:33.261 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:33.261 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:33.262 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:22:33.262 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.042830+0000", 2024-09-13T11:22:33.262 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.7927999999999998e-05 2024-09-13T11:22:33.262 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:33.262 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:33.262 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:22:33.262 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.042838+0000", 2024-09-13T11:22:33.262 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.6009999999999995e-06 2024-09-13T11:22:33.262 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:33.262 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:33.262 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:22:33.262 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.042898+0000", 2024-09-13T11:22:33.262 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.9851e-05 2024-09-13T11:22:33.262 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:33.263 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:33.263 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:22:33.263 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.043013+0000", 2024-09-13T11:22:33.263 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000115497 2024-09-13T11:22:33.263 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:33.263 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:33.263 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 2", 2024-09-13T11:22:33.263 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.043131+0000", 2024-09-13T11:22:33.263 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000117657 2024-09-13T11:22:33.263 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:33.263 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:33.263 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:22:33.263 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.044018+0000", 2024-09-13T11:22:33.264 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00088696100000000002 2024-09-13T11:22:33.264 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:33.264 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:33.264 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:22:33.264 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.045425+0000", 2024-09-13T11:22:33.264 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0014070350000000001 2024-09-13T11:22:33.264 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:33.264 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:33.264 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:22:33.264 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.045466+0000", 2024-09-13T11:22:33.264 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.0636999999999997e-05 2024-09-13T11:22:33.264 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:33.264 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:33.265 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:22:33.265 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.045505+0000", 2024-09-13T11:22:33.265 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.9402e-05 2024-09-13T11:22:33.265 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:22:33.265 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:22:33.265 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:22:33.265 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:33.265 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:33.265 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4665.0:82 3.2 e33/30 3:4daba74e:::benchmark_data_smithi177_31116_object81:head v 33'6, mlcod=32'4)", 2024-09-13T11:22:33.265 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:22:27.950739+0000", 2024-09-13T11:22:33.265 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 5.2076447090000002, 2024-09-13T11:22:33.265 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.002148507, 2024-09-13T11:22:33.266 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:22:33.266 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:22:33.266 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:33.266 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:22:33.266 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:33.266 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:22:33.266 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.950739+0000", 2024-09-13T11:22:33.266 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:22:33.266 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:33.266 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:33.266 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:22:33.266 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.950739+0000", 2024-09-13T11:22:33.266 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:22:33.267 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:33.267 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:33.267 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:22:33.267 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.950740+0000", 2024-09-13T11:22:33.267 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.0499999999999999e-07 2024-09-13T11:22:33.267 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:33.267 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:33.267 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:22:33.267 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.951138+0000", 2024-09-13T11:22:33.267 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00039828800000000001 2024-09-13T11:22:33.267 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:33.267 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:33.267 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:22:33.268 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.951189+0000", 2024-09-13T11:22:33.268 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.0831000000000002e-05 2024-09-13T11:22:33.268 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:33.268 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:33.268 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:22:33.268 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.951210+0000", 2024-09-13T11:22:33.268 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.1308e-05 2024-09-13T11:22:33.268 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:33.268 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:33.268 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:22:33.268 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.951339+0000", 2024-09-13T11:22:33.268 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000129066 2024-09-13T11:22:33.268 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:33.269 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:33.269 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:22:33.269 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.951376+0000", 2024-09-13T11:22:33.269 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.6505999999999999e-05 2024-09-13T11:22:33.269 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:33.269 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:33.269 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:22:33.269 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.952788+0000", 2024-09-13T11:22:33.269 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.001412368 2024-09-13T11:22:33.269 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:33.269 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:33.269 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:22:33.269 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.952888+0000", 2024-09-13T11:22:33.270 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.9834999999999995e-05 2024-09-13T11:22:33.270 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:22:33.270 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:22:33.270 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:22:33.270 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:33.270 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:33.270 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:85 3.9 3:9b4652c0:::benchmark_data_smithi177_31116_object84:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e33)", 2024-09-13T11:22:33.270 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:22:27.974212+0000", 2024-09-13T11:22:33.270 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 5.1841715339999999, 2024-09-13T11:22:33.270 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0038541299999999999, 2024-09-13T11:22:33.270 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:22:33.270 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:22:33.270 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:33.271 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:22:33.271 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:22:33.271 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:22:33.271 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 85 2024-09-13T11:22:33.271 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:33.271 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:22:33.271 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:33.271 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:22:33.271 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.974212+0000", 2024-09-13T11:22:33.271 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:22:33.271 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:33.271 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:33.271 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:22:33.272 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.974212+0000", 2024-09-13T11:22:33.272 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:22:33.272 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:33.272 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:33.272 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:22:33.272 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.974217+0000", 2024-09-13T11:22:33.272 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.8169999999999998e-06 2024-09-13T11:22:33.272 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:33.272 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:33.272 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:22:33.272 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.974417+0000", 2024-09-13T11:22:33.272 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00019947699999999999 2024-09-13T11:22:33.272 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:33.273 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:33.273 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:22:33.273 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.974447+0000", 2024-09-13T11:22:33.273 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.0349999999999999e-05 2024-09-13T11:22:33.273 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:33.273 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:33.273 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:22:33.273 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.974461+0000", 2024-09-13T11:22:33.273 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.4182000000000001e-05 2024-09-13T11:22:33.273 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:33.273 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:33.273 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:22:33.273 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.974636+0000", 2024-09-13T11:22:33.274 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000174981 2024-09-13T11:22:33.274 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:33.274 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:33.274 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:22:33.274 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.975036+0000", 2024-09-13T11:22:33.274 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000399517 2024-09-13T11:22:33.274 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:33.274 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:33.274 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 2", 2024-09-13T11:22:33.274 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.975431+0000", 2024-09-13T11:22:33.274 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00039529700000000002 2024-09-13T11:22:33.274 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:33.274 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:33.274 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:22:33.275 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.977154+0000", 2024-09-13T11:22:33.275 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0017228160000000001 2024-09-13T11:22:33.275 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:33.275 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:33.275 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:22:33.275 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.977791+0000", 2024-09-13T11:22:33.275 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00063686400000000001 2024-09-13T11:22:33.275 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:33.275 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:33.275 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:22:33.275 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.977949+0000", 2024-09-13T11:22:33.275 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000158758 2024-09-13T11:22:33.276 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:33.276 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:33.276 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:22:33.276 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.978067+0000", 2024-09-13T11:22:33.276 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000117071 2024-09-13T11:22:33.276 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:22:33.276 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:22:33.276 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:22:33.277 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:22:33.277 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:22:33.277 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-13T11:22:33.277 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-13T11:22:33.279 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-13T11:22:33.280 INFO:tasks.ceph.osd.2.smithi162.stderr:2024-09-13T11:22:33.276+0000 7fe92b35a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26313) UID: 0 2024-09-13T11:22:33.380 INFO:tasks.ceph.osd.4.smithi001.stderr:2024-09-13T11:22:33.372+0000 7ff8e0218640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26310) UID: 0 2024-09-13T11:22:33.412 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-09-13T11:22:33.412 INFO:teuthology.orchestra.run.smithi092.stdout: "ops": [], 2024-09-13T11:22:33.413 INFO:teuthology.orchestra.run.smithi092.stdout: "num_ops": 0 2024-09-13T11:22:33.413 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-09-13T11:22:33.426 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-13T11:22:33.481 INFO:tasks.ceph.osd.12.smithi001.stderr:2024-09-13T11:22:33.476+0000 7f892c6e6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26314) UID: 0 2024-09-13T11:22:33.549 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-09-13T11:22:33.549 INFO:teuthology.orchestra.run.smithi092.stdout: "ops": [], 2024-09-13T11:22:33.549 INFO:teuthology.orchestra.run.smithi092.stdout: "complaint_time": 30, 2024-09-13T11:22:33.549 INFO:teuthology.orchestra.run.smithi092.stdout: "num_blocked_ops": 0 2024-09-13T11:22:33.549 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-09-13T11:22:33.563 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-13T11:22:33.608 INFO:tasks.ceph.osd.13.smithi092.stderr:2024-09-13T11:22:33.607+0000 7f3e5afea640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26350) UID: 0 2024-09-13T11:22:33.683 INFO:tasks.ceph.osd.4.smithi001.stderr:2024-09-13T11:22:33.676+0000 7ff8e0218640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26310) UID: 0 2024-09-13T11:22:33.683 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-09-13T11:22:33.684 INFO:teuthology.orchestra.run.smithi092.stdout: "size": 20, 2024-09-13T11:22:33.684 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 600, 2024-09-13T11:22:33.684 INFO:teuthology.orchestra.run.smithi092.stdout: "ops": [] 2024-09-13T11:22:33.684 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-09-13T11:22:33.697 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-09-13T11:22:33.783 INFO:tasks.ceph.osd.8.smithi001.stderr:2024-09-13T11:22:33.780+0000 7fb9b48ae640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26312) UID: 0 2024-09-13T11:22:33.864 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-09-13T11:22:33.865 INFO:teuthology.orchestra.run.smithi092.stdout: "ops": [], 2024-09-13T11:22:33.865 INFO:teuthology.orchestra.run.smithi092.stdout: "num_ops": 0 2024-09-13T11:22:33.865 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-09-13T11:22:33.878 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-09-13T11:22:33.884 INFO:tasks.ceph.osd.4.smithi001.stderr:2024-09-13T11:22:33.880+0000 7ff8e0218640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26310) UID: 0 2024-09-13T11:22:33.985 INFO:tasks.ceph.osd.4.smithi001.stderr:2024-09-13T11:22:33.980+0000 7ff8e0218640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26310) UID: 0 2024-09-13T11:22:33.996 INFO:tasks.radosbench.radosbench.0.smithi177.stdout: 7 16 87 71 0.633841 0 - 0.0234801 2024-09-13T11:22:34.000 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-09-13T11:22:34.000 INFO:teuthology.orchestra.run.smithi092.stdout: "ops": [], 2024-09-13T11:22:34.000 INFO:teuthology.orchestra.run.smithi092.stdout: "complaint_time": 30, 2024-09-13T11:22:34.000 INFO:teuthology.orchestra.run.smithi092.stdout: "num_blocked_ops": 0 2024-09-13T11:22:34.000 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-09-13T11:22:34.014 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-09-13T11:22:34.086 INFO:tasks.ceph.osd.4.smithi001.stderr:2024-09-13T11:22:34.084+0000 7ff8e0218640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26310) UID: 0 2024-09-13T11:22:34.131 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-09-13T11:22:34.132 INFO:teuthology.orchestra.run.smithi092.stdout: "size": 20, 2024-09-13T11:22:34.132 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 600, 2024-09-13T11:22:34.132 INFO:teuthology.orchestra.run.smithi092.stdout: "ops": [ 2024-09-13T11:22:34.132 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:34.132 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop(client.4344.0:5 2.2 e27/25 2:5cea7035:::rbd_info:head v 27'1, mlcod=0'0)", 2024-09-13T11:22:34.132 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:15.250889+0000", 2024-09-13T11:22:34.132 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 18.88198152, 2024-09-13T11:22:34.132 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.001990627, 2024-09-13T11:22:34.132 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:22:34.132 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:22:34.133 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:34.133 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:22:34.133 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:34.133 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:22:34.133 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:15.250889+0000", 2024-09-13T11:22:34.133 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:22:34.133 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:34.133 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:34.133 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:22:34.133 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:15.250889+0000", 2024-09-13T11:22:34.133 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:22:34.133 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:34.133 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:34.134 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:22:34.134 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:15.250890+0000", 2024-09-13T11:22:34.134 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.153e-06 2024-09-13T11:22:34.134 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:34.134 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:34.134 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:22:34.134 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:15.250907+0000", 2024-09-13T11:22:34.134 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.6821e-05 2024-09-13T11:22:34.134 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:34.134 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:34.134 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:22:34.134 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:15.250991+0000", 2024-09-13T11:22:34.134 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.4241999999999997e-05 2024-09-13T11:22:34.134 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:34.135 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:34.135 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:22:34.135 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:15.251023+0000", 2024-09-13T11:22:34.135 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.1780999999999999e-05 2024-09-13T11:22:34.135 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:34.135 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:34.135 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:22:34.135 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:15.251178+0000", 2024-09-13T11:22:34.135 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000155064 2024-09-13T11:22:34.135 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:34.135 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:34.135 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:22:34.135 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:15.251215+0000", 2024-09-13T11:22:34.135 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.7160000000000003e-05 2024-09-13T11:22:34.136 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:34.136 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:34.136 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:22:34.136 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:15.252759+0000", 2024-09-13T11:22:34.136 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0015440110000000001 2024-09-13T11:22:34.136 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:34.136 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:34.136 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:22:34.136 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:15.252880+0000", 2024-09-13T11:22:34.136 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000120395 2024-09-13T11:22:34.136 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:22:34.136 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:22:34.136 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:22:34.137 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:34.137 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:34.137 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop(client.4344.0:7 2.2 e28/25 2:5cea7035:::rbd_info:head v 28'2, mlcod=0'0)", 2024-09-13T11:22:34.137 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:16.268518+0000", 2024-09-13T11:22:34.137 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 17.864351920000001, 2024-09-13T11:22:34.137 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0017167040000000001, 2024-09-13T11:22:34.137 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:22:34.137 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:22:34.137 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:34.137 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:22:34.137 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:34.137 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:22:34.137 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:16.268518+0000", 2024-09-13T11:22:34.138 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:22:34.138 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:34.138 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:34.138 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:22:34.138 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:16.268518+0000", 2024-09-13T11:22:34.138 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:22:34.138 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:34.138 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:34.138 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:22:34.138 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:16.268519+0000", 2024-09-13T11:22:34.138 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.1299999999999999e-07 2024-09-13T11:22:34.138 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:34.139 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:34.139 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:22:34.139 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:16.268539+0000", 2024-09-13T11:22:34.139 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.0214999999999999e-05 2024-09-13T11:22:34.139 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:34.139 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:34.139 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:22:34.139 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:16.268622+0000", 2024-09-13T11:22:34.139 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.2395e-05 2024-09-13T11:22:34.139 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:34.139 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:34.139 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:22:34.139 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:16.268638+0000", 2024-09-13T11:22:34.140 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.6189e-05 2024-09-13T11:22:34.140 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:34.140 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:34.140 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:22:34.140 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:16.268740+0000", 2024-09-13T11:22:34.140 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00010164 2024-09-13T11:22:34.140 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:34.140 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:34.140 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:22:34.140 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:16.268783+0000", 2024-09-13T11:22:34.140 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.2988999999999998e-05 2024-09-13T11:22:34.140 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:34.141 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:34.141 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:22:34.141 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:16.270121+0000", 2024-09-13T11:22:34.141 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00133829 2024-09-13T11:22:34.141 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:34.141 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:34.141 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:22:34.141 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:16.270235+0000", 2024-09-13T11:22:34.141 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000114273 2024-09-13T11:22:34.141 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:22:34.141 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:22:34.141 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:22:34.141 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:22:34.141 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:22:34.142 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-09-13T11:22:34.142 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T11:22:34.146 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_ops_in_flight 2024-09-13T11:22:34.188 INFO:tasks.ceph.osd.8.smithi001.stderr:2024-09-13T11:22:34.184+0000 7fb9b48ae640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26312) UID: 0 2024-09-13T11:22:34.209 INFO:teuthology.orchestra.run.smithi001.stderr:nodeep-scrub is unset 2024-09-13T11:22:34.272 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-09-13T11:22:34.272 INFO:teuthology.orchestra.run.smithi092.stdout: "ops": [], 2024-09-13T11:22:34.273 INFO:teuthology.orchestra.run.smithi092.stdout: "num_ops": 0 2024-09-13T11:22:34.273 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-09-13T11:22:34.286 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_blocked_ops 2024-09-13T11:22:34.288 INFO:tasks.ceph.osd.2.smithi162.stderr:2024-09-13T11:22:34.288+0000 7fe92b35a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26313) UID: 0 2024-09-13T11:22:34.323 INFO:teuthology.orchestra.run.smithi001.stdout:osd.0: {} 2024-09-13T11:22:34.323 INFO:teuthology.orchestra.run.smithi001.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T11:22:34.323 INFO:teuthology.orchestra.run.smithi001.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-09-13T11:22:34.350 INFO:teuthology.orchestra.run.smithi001.stdout:osd.2: {} 2024-09-13T11:22:34.350 INFO:teuthology.orchestra.run.smithi001.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T11:22:34.378 INFO:teuthology.orchestra.run.smithi001.stdout:osd.3: {} 2024-09-13T11:22:34.378 INFO:teuthology.orchestra.run.smithi001.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T11:22:34.389 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-13T11:22:34.384+0000 7f6a68775640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26307) UID: 0 2024-09-13T11:22:34.406 INFO:teuthology.orchestra.run.smithi001.stdout:osd.4: {} 2024-09-13T11:22:34.406 INFO:teuthology.orchestra.run.smithi001.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T11:22:34.408 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-09-13T11:22:34.408 INFO:teuthology.orchestra.run.smithi092.stdout: "ops": [], 2024-09-13T11:22:34.408 INFO:teuthology.orchestra.run.smithi092.stdout: "complaint_time": 30, 2024-09-13T11:22:34.408 INFO:teuthology.orchestra.run.smithi092.stdout: "num_blocked_ops": 0 2024-09-13T11:22:34.408 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-09-13T11:22:34.423 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_historic_ops 2024-09-13T11:22:34.433 INFO:teuthology.orchestra.run.smithi001.stdout:osd.5: {} 2024-09-13T11:22:34.434 INFO:teuthology.orchestra.run.smithi001.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T11:22:34.466 INFO:teuthology.orchestra.run.smithi001.stdout:osd.6: {} 2024-09-13T11:22:34.467 INFO:teuthology.orchestra.run.smithi001.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T11:22:34.490 INFO:tasks.ceph.osd.3.smithi177.stderr:2024-09-13T11:22:34.488+0000 7f10adba4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26574) UID: 0 2024-09-13T11:22:34.499 INFO:teuthology.orchestra.run.smithi001.stdout:osd.7: {} 2024-09-13T11:22:34.499 INFO:teuthology.orchestra.run.smithi001.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T11:22:34.533 INFO:teuthology.orchestra.run.smithi001.stdout:osd.8: {} 2024-09-13T11:22:34.533 INFO:teuthology.orchestra.run.smithi001.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-13T11:22:34.544 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-09-13T11:22:34.544 INFO:teuthology.orchestra.run.smithi092.stdout: "size": 20, 2024-09-13T11:22:34.544 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 600, 2024-09-13T11:22:34.544 INFO:teuthology.orchestra.run.smithi092.stdout: "ops": [ 2024-09-13T11:22:34.544 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:34.544 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:8 3.6 3:64ae72f1:::benchmark_data_smithi177_31116_object7:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-09-13T11:22:34.544 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:27.002767+0000", 2024-09-13T11:22:34.545 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 7.5388396640000002, 2024-09-13T11:22:34.545 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0070672010000000004, 2024-09-13T11:22:34.545 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:22:34.545 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:22:34.545 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:34.545 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:22:34.545 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:22:34.545 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:22:34.545 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 8 2024-09-13T11:22:34.545 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:34.545 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:22:34.545 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:34.545 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:22:34.545 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.002767+0000", 2024-09-13T11:22:34.546 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:22:34.546 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:34.546 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:34.546 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:22:34.546 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.002767+0000", 2024-09-13T11:22:34.546 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:22:34.546 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:34.546 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:34.546 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:22:34.546 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.002768+0000", 2024-09-13T11:22:34.546 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.48e-06 2024-09-13T11:22:34.546 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:34.546 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:34.547 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:22:34.547 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.002919+0000", 2024-09-13T11:22:34.547 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00015046600000000001 2024-09-13T11:22:34.547 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:34.547 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:34.547 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:22:34.547 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.002930+0000", 2024-09-13T11:22:34.547 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.1527e-05 2024-09-13T11:22:34.547 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:34.547 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:34.548 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:22:34.548 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.002939+0000", 2024-09-13T11:22:34.548 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.3929999999999997e-06 2024-09-13T11:22:34.548 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:34.548 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:34.548 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:22:34.548 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.003079+0000", 2024-09-13T11:22:34.548 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00013977 2024-09-13T11:22:34.548 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:34.548 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:34.548 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:22:34.549 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.003401+0000", 2024-09-13T11:22:34.549 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00032173400000000001 2024-09-13T11:22:34.549 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:34.549 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:34.549 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 14", 2024-09-13T11:22:34.549 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.003790+0000", 2024-09-13T11:22:34.549 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00038853599999999998 2024-09-13T11:22:34.549 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:34.549 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:34.549 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:22:34.549 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.005933+0000", 2024-09-13T11:22:34.550 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.002143449 2024-09-13T11:22:34.550 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:34.550 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:34.550 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:22:34.550 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.009779+0000", 2024-09-13T11:22:34.550 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0038457439999999999 2024-09-13T11:22:34.550 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:34.550 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:34.550 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:22:34.550 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.009809+0000", 2024-09-13T11:22:34.550 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.0752e-05 2024-09-13T11:22:34.550 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:34.550 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:34.551 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:22:34.551 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.009834+0000", 2024-09-13T11:22:34.551 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.4349999999999999e-05 2024-09-13T11:22:34.551 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:22:34.551 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:22:34.551 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:22:34.551 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:34.551 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:34.551 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:15 3.6 3:6288aaff:::benchmark_data_smithi177_31116_object14:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-09-13T11:22:34.551 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:27.003252+0000", 2024-09-13T11:22:34.551 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 7.5383540050000004, 2024-09-13T11:22:34.551 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0066852810000000004, 2024-09-13T11:22:34.552 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:22:34.552 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:22:34.552 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:34.552 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:22:34.552 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:22:34.552 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:22:34.552 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 15 2024-09-13T11:22:34.552 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:34.552 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:22:34.552 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:34.552 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:22:34.552 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.003252+0000", 2024-09-13T11:22:34.553 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:22:34.553 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:34.553 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:34.553 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:22:34.553 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.003252+0000", 2024-09-13T11:22:34.553 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:22:34.553 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:34.553 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:34.553 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:22:34.553 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.003253+0000", 2024-09-13T11:22:34.553 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0529999999999999e-06 2024-09-13T11:22:34.554 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:34.554 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:34.554 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:22:34.554 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.003304+0000", 2024-09-13T11:22:34.554 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.0408999999999998e-05 2024-09-13T11:22:34.554 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:34.554 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:34.554 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:22:34.554 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.003313+0000", 2024-09-13T11:22:34.554 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.0750000000000004e-06 2024-09-13T11:22:34.554 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:34.554 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:34.554 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:22:34.555 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.003332+0000", 2024-09-13T11:22:34.555 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.8692e-05 2024-09-13T11:22:34.555 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:34.555 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:34.555 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:22:34.555 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.004981+0000", 2024-09-13T11:22:34.555 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0016496060000000001 2024-09-13T11:22:34.555 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:34.555 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:34.555 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:22:34.555 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.005073+0000", 2024-09-13T11:22:34.555 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.1774999999999995e-05 2024-09-13T11:22:34.555 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:34.556 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:34.556 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 14", 2024-09-13T11:22:34.556 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.005155+0000", 2024-09-13T11:22:34.556 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.2399999999999997e-05 2024-09-13T11:22:34.556 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:34.556 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:34.556 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:22:34.556 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.008629+0000", 2024-09-13T11:22:34.556 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.003473828 2024-09-13T11:22:34.556 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:34.556 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:34.556 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:22:34.556 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.009895+0000", 2024-09-13T11:22:34.557 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.001265396 2024-09-13T11:22:34.557 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:34.557 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:34.557 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:22:34.557 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.009919+0000", 2024-09-13T11:22:34.557 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.4825999999999999e-05 2024-09-13T11:22:34.557 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:34.557 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:34.557 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:22:34.557 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.009938+0000", 2024-09-13T11:22:34.557 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.8221000000000001e-05 2024-09-13T11:22:34.557 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:22:34.557 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:22:34.558 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:22:34.558 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:34.558 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:34.558 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:16 3.7 3:e0940737:::benchmark_data_smithi177_31116_object15:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-09-13T11:22:34.558 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:27.003345+0000", 2024-09-13T11:22:34.558 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 7.5382609699999996, 2024-09-13T11:22:34.558 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0054656790000000002, 2024-09-13T11:22:34.558 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:22:34.558 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:22:34.558 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:34.558 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:22:34.558 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:22:34.558 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:22:34.559 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 16 2024-09-13T11:22:34.559 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:34.559 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:22:34.559 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:34.559 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:22:34.559 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.003345+0000", 2024-09-13T11:22:34.559 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:22:34.559 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:34.559 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:34.559 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:22:34.560 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.003345+0000", 2024-09-13T11:22:34.560 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:22:34.560 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:34.560 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:34.560 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:22:34.560 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.003347+0000", 2024-09-13T11:22:34.560 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.3039999999999999e-06 2024-09-13T11:22:34.560 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:34.560 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:34.560 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:22:34.560 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.003376+0000", 2024-09-13T11:22:34.561 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.9475000000000002e-05 2024-09-13T11:22:34.561 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:34.561 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:34.561 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:22:34.561 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.003385+0000", 2024-09-13T11:22:34.561 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.8310000000000007e-06 2024-09-13T11:22:34.561 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:34.561 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:34.561 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:22:34.561 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.003399+0000", 2024-09-13T11:22:34.561 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.4103e-05 2024-09-13T11:22:34.561 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:34.561 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:34.562 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:22:34.562 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.003505+0000", 2024-09-13T11:22:34.562 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000105739 2024-09-13T11:22:34.562 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:34.562 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:34.562 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:22:34.562 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.003793+0000", 2024-09-13T11:22:34.562 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00028870300000000002 2024-09-13T11:22:34.562 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:34.562 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:34.562 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 8", 2024-09-13T11:22:34.562 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.004014+0000", 2024-09-13T11:22:34.563 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00022092800000000001 2024-09-13T11:22:34.563 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:34.563 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:34.563 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:22:34.563 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.006245+0000", 2024-09-13T11:22:34.563 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.002230788 2024-09-13T11:22:34.563 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:34.563 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:34.563 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:22:34.563 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.008644+0000", 2024-09-13T11:22:34.563 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0023984689999999999 2024-09-13T11:22:34.563 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:34.563 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:34.564 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:22:34.564 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.008724+0000", 2024-09-13T11:22:34.564 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.0005000000000004e-05 2024-09-13T11:22:34.564 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:34.564 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:34.564 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:22:34.564 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.008811+0000", 2024-09-13T11:22:34.564 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.7334000000000001e-05 2024-09-13T11:22:34.564 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:22:34.564 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:22:34.564 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:22:34.564 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:34.564 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:34.565 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:19 3.1 3:8c29d864:::benchmark_data_smithi177_31116_object18:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-09-13T11:22:34.565 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:27.009947+0000", 2024-09-13T11:22:34.565 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 7.5316590869999995, 2024-09-13T11:22:34.565 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.002910733, 2024-09-13T11:22:34.565 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:22:34.565 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:22:34.565 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:34.565 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:22:34.565 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:22:34.565 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:22:34.565 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 19 2024-09-13T11:22:34.565 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:34.565 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:22:34.566 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:34.566 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:22:34.566 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.009947+0000", 2024-09-13T11:22:34.566 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:22:34.566 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:34.566 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:34.566 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:22:34.566 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.009947+0000", 2024-09-13T11:22:34.566 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:22:34.566 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:34.566 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:34.566 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:22:34.566 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.009951+0000", 2024-09-13T11:22:34.567 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.6030000000000001e-06 2024-09-13T11:22:34.567 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:34.567 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:34.567 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:22:34.567 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.010000+0000", 2024-09-13T11:22:34.567 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.9620000000000003e-05 2024-09-13T11:22:34.567 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:34.567 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:34.567 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:22:34.567 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.010035+0000", 2024-09-13T11:22:34.567 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.4626999999999999e-05 2024-09-13T11:22:34.567 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:34.567 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:34.568 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:22:34.568 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.010053+0000", 2024-09-13T11:22:34.568 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.7686e-05 2024-09-13T11:22:34.568 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:34.568 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:34.568 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:22:34.568 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.010134+0000", 2024-09-13T11:22:34.568 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.1074000000000003e-05 2024-09-13T11:22:34.568 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:34.568 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:34.568 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:22:34.568 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.010275+0000", 2024-09-13T11:22:34.569 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00014124100000000001 2024-09-13T11:22:34.569 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:34.569 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:34.569 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 11", 2024-09-13T11:22:34.569 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.010442+0000", 2024-09-13T11:22:34.569 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00016715400000000001 2024-09-13T11:22:34.569 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:34.569 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:34.569 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:22:34.569 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.011925+0000", 2024-09-13T11:22:34.569 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0014823010000000001 2024-09-13T11:22:34.569 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:34.569 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:34.570 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:22:34.570 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.012708+0000", 2024-09-13T11:22:34.570 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00078309199999999999 2024-09-13T11:22:34.570 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:34.570 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:34.570 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:22:34.570 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.012811+0000", 2024-09-13T11:22:34.570 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000103214 2024-09-13T11:22:34.570 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:34.570 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:34.570 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:22:34.570 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.012858+0000", 2024-09-13T11:22:34.571 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.7120999999999998e-05 2024-09-13T11:22:34.571 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:22:34.571 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:22:34.571 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:22:34.571 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:34.571 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:34.571 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:21 3.6 3:69e4f35f:::benchmark_data_smithi177_31116_object20:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-09-13T11:22:34.571 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:27.010086+0000", 2024-09-13T11:22:34.571 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 7.5315199899999996, 2024-09-13T11:22:34.571 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0032553600000000001, 2024-09-13T11:22:34.571 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:22:34.571 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:22:34.571 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:34.571 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:22:34.572 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:22:34.572 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:22:34.572 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 21 2024-09-13T11:22:34.572 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:34.572 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:22:34.572 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:34.572 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:22:34.572 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.010086+0000", 2024-09-13T11:22:34.572 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:22:34.572 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:34.572 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:34.572 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:22:34.573 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.010086+0000", 2024-09-13T11:22:34.573 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:22:34.573 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:34.573 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:34.573 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:22:34.573 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.010088+0000", 2024-09-13T11:22:34.573 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.553e-06 2024-09-13T11:22:34.573 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:34.573 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:34.573 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:22:34.573 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.010130+0000", 2024-09-13T11:22:34.573 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.2049999999999999e-05 2024-09-13T11:22:34.573 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:34.573 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:34.574 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:22:34.574 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.010142+0000", 2024-09-13T11:22:34.574 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.1646e-05 2024-09-13T11:22:34.574 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:34.574 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:34.574 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:22:34.574 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.010162+0000", 2024-09-13T11:22:34.574 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.0642e-05 2024-09-13T11:22:34.574 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:34.574 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:34.574 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:22:34.574 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.010199+0000", 2024-09-13T11:22:34.575 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.7039000000000002e-05 2024-09-13T11:22:34.575 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:34.575 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:34.575 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:22:34.575 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.010267+0000", 2024-09-13T11:22:34.575 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.7488000000000002e-05 2024-09-13T11:22:34.575 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:34.575 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:34.575 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 14", 2024-09-13T11:22:34.575 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.010342+0000", 2024-09-13T11:22:34.575 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.4857000000000005e-05 2024-09-13T11:22:34.575 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:34.575 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:34.576 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:22:34.576 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.011774+0000", 2024-09-13T11:22:34.576 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0014321340000000001 2024-09-13T11:22:34.576 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:34.576 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:34.576 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:22:34.576 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.013275+0000", 2024-09-13T11:22:34.576 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0015013699999999999 2024-09-13T11:22:34.576 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:34.576 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:34.576 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:22:34.576 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.013305+0000", 2024-09-13T11:22:34.576 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.9561e-05 2024-09-13T11:22:34.577 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:34.577 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:34.577 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:22:34.577 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.013342+0000", 2024-09-13T11:22:34.577 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.7020000000000001e-05 2024-09-13T11:22:34.577 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:22:34.577 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:22:34.577 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:22:34.577 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:34.577 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:34.577 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:22 3.6 3:6f6b370c:::benchmark_data_smithi177_31116_object21:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-09-13T11:22:34.577 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:27.010363+0000", 2024-09-13T11:22:34.577 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 7.5312435860000004, 2024-09-13T11:22:34.578 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0063408980000000002, 2024-09-13T11:22:34.578 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:22:34.578 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:22:34.578 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:34.578 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:22:34.578 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:22:34.578 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:22:34.578 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 22 2024-09-13T11:22:34.578 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:34.578 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:22:34.578 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:34.578 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:22:34.578 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.010363+0000", 2024-09-13T11:22:34.579 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:22:34.579 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:34.579 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:34.579 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:22:34.579 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.010363+0000", 2024-09-13T11:22:34.579 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:22:34.579 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:34.579 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:34.579 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:22:34.579 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.010365+0000", 2024-09-13T11:22:34.579 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.2000000000000001e-06 2024-09-13T11:22:34.579 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:34.579 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:34.579 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:22:34.580 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.010563+0000", 2024-09-13T11:22:34.580 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00019851600000000001 2024-09-13T11:22:34.580 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:34.580 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:34.580 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:22:34.580 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.010579+0000", 2024-09-13T11:22:34.580 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.5554999999999999e-05 2024-09-13T11:22:34.580 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:34.580 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:34.580 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:22:34.580 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.010589+0000", 2024-09-13T11:22:34.581 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.9599999999999995e-06 2024-09-13T11:22:34.581 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:34.581 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:34.581 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:22:34.581 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.010657+0000", 2024-09-13T11:22:34.581 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.7615999999999998e-05 2024-09-13T11:22:34.581 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:34.581 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:34.581 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:22:34.581 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.010746+0000", 2024-09-13T11:22:34.581 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.9116000000000005e-05 2024-09-13T11:22:34.581 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:34.581 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:34.582 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 14", 2024-09-13T11:22:34.582 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.010830+0000", 2024-09-13T11:22:34.582 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.4007999999999999e-05 2024-09-13T11:22:34.582 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:34.582 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:34.582 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:22:34.582 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.012764+0000", 2024-09-13T11:22:34.582 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.001934468 2024-09-13T11:22:34.582 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:34.582 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:34.582 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:22:34.582 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.016548+0000", 2024-09-13T11:22:34.583 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.003784103 2024-09-13T11:22:34.583 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:34.583 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:34.583 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:22:34.583 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.016610+0000", 2024-09-13T11:22:34.583 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.2247000000000002e-05 2024-09-13T11:22:34.583 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:34.583 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:34.583 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:22:34.583 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.016704+0000", 2024-09-13T11:22:34.583 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.3109000000000006e-05 2024-09-13T11:22:34.583 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:22:34.583 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:22:34.583 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:22:34.584 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:34.584 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:34.584 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:24 3.6 3:696318df:::benchmark_data_smithi177_31116_object23:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-09-13T11:22:34.584 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:27.010611+0000", 2024-09-13T11:22:34.584 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 7.5309954699999997, 2024-09-13T11:22:34.584 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0061998790000000002, 2024-09-13T11:22:34.584 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:22:34.584 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:22:34.584 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:34.584 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:22:34.585 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:22:34.585 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:22:34.585 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 24 2024-09-13T11:22:34.585 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:34.585 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:22:34.585 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:34.585 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:22:34.585 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.010611+0000", 2024-09-13T11:22:34.585 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:22:34.585 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:34.585 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:34.585 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:22:34.585 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.010611+0000", 2024-09-13T11:22:34.586 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:22:34.586 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:34.586 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:34.586 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:22:34.586 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.010612+0000", 2024-09-13T11:22:34.586 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.57e-06 2024-09-13T11:22:34.586 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:34.586 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:34.586 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:22:34.586 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.010725+0000", 2024-09-13T11:22:34.586 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000112545 2024-09-13T11:22:34.586 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:34.586 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:34.587 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:22:34.587 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.010738+0000", 2024-09-13T11:22:34.587 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.3196e-05 2024-09-13T11:22:34.587 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:34.587 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:34.587 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:22:34.587 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.010747+0000", 2024-09-13T11:22:34.587 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.5730000000000008e-06 2024-09-13T11:22:34.587 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:34.587 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:34.587 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:22:34.587 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.011149+0000", 2024-09-13T11:22:34.587 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00040200899999999999 2024-09-13T11:22:34.588 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:34.588 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:34.588 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:22:34.588 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.011219+0000", 2024-09-13T11:22:34.588 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.0710999999999994e-05 2024-09-13T11:22:34.588 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:34.588 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:34.588 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 14", 2024-09-13T11:22:34.588 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.011301+0000", 2024-09-13T11:22:34.588 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.1278000000000003e-05 2024-09-13T11:22:34.588 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:34.588 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:34.588 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:22:34.589 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.012775+0000", 2024-09-13T11:22:34.589 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.001474174 2024-09-13T11:22:34.589 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:34.589 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:34.589 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:22:34.589 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.016752+0000", 2024-09-13T11:22:34.589 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0039771880000000004 2024-09-13T11:22:34.589 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:34.589 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:34.589 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:22:34.589 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.016787+0000", 2024-09-13T11:22:34.589 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.4937000000000001e-05 2024-09-13T11:22:34.589 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:34.590 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:34.590 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:22:34.590 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.016811+0000", 2024-09-13T11:22:34.590 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.3697999999999999e-05 2024-09-13T11:22:34.590 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:22:34.590 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:22:34.590 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:22:34.590 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:34.590 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:34.590 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop(client.4665.0:23 3.c e32/30 3:31691fed:::benchmark_data_smithi177_31116_object22:head v 32'1, mlcod=0'0)", 2024-09-13T11:22:34.590 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:27.011081+0000", 2024-09-13T11:22:34.590 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 7.5305253360000002, 2024-09-13T11:22:34.590 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0026072109999999999, 2024-09-13T11:22:34.590 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:22:34.591 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:22:34.591 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:34.591 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:22:34.591 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:34.591 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:22:34.591 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.011081+0000", 2024-09-13T11:22:34.591 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:22:34.591 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:34.591 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:34.591 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:22:34.591 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.011081+0000", 2024-09-13T11:22:34.591 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:22:34.591 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:34.592 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:34.592 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:22:34.592 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.011082+0000", 2024-09-13T11:22:34.592 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.28e-06 2024-09-13T11:22:34.592 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:34.592 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:34.592 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:22:34.592 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.011307+0000", 2024-09-13T11:22:34.592 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00022448999999999999 2024-09-13T11:22:34.593 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:34.593 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:34.593 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:22:34.593 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.011374+0000", 2024-09-13T11:22:34.593 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.7732999999999997e-05 2024-09-13T11:22:34.593 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:34.593 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:34.593 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:22:34.593 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.011392+0000", 2024-09-13T11:22:34.593 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.7847999999999999e-05 2024-09-13T11:22:34.593 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:34.593 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:34.593 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:22:34.594 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.011535+0000", 2024-09-13T11:22:34.594 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00014295700000000001 2024-09-13T11:22:34.594 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:34.594 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:34.594 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:22:34.594 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.011587+0000", 2024-09-13T11:22:34.594 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.1826999999999997e-05 2024-09-13T11:22:34.594 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:34.594 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:34.594 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:22:34.594 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.013573+0000", 2024-09-13T11:22:34.595 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0019862539999999998 2024-09-13T11:22:34.595 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:34.595 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:34.595 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:22:34.595 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.013688+0000", 2024-09-13T11:22:34.595 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00011482200000000001 2024-09-13T11:22:34.595 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:22:34.595 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:22:34.595 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:22:34.595 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:34.595 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:34.595 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:25 3.7 3:e6850330:::benchmark_data_smithi177_31116_object24:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-09-13T11:22:34.595 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:27.011586+0000", 2024-09-13T11:22:34.596 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 7.5300202519999999, 2024-09-13T11:22:34.596 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0059918030000000004, 2024-09-13T11:22:34.596 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:22:34.596 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:22:34.596 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:34.596 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:22:34.596 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:22:34.596 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:22:34.596 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 25 2024-09-13T11:22:34.596 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:34.596 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:22:34.596 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:34.596 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:22:34.597 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.011586+0000", 2024-09-13T11:22:34.597 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:22:34.597 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:34.597 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:34.597 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:22:34.597 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.011586+0000", 2024-09-13T11:22:34.597 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:22:34.597 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:34.597 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:34.597 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:22:34.597 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.011588+0000", 2024-09-13T11:22:34.597 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.12e-06 2024-09-13T11:22:34.598 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:34.598 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:34.598 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:22:34.598 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.011688+0000", 2024-09-13T11:22:34.598 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.9920000000000006e-05 2024-09-13T11:22:34.598 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:34.598 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:34.598 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:22:34.598 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.011701+0000", 2024-09-13T11:22:34.598 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.2629999999999999e-05 2024-09-13T11:22:34.598 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:34.598 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:34.599 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:22:34.599 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.011712+0000", 2024-09-13T11:22:34.599 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.1701e-05 2024-09-13T11:22:34.599 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:34.599 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:34.599 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:22:34.599 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.011834+0000", 2024-09-13T11:22:34.599 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000121709 2024-09-13T11:22:34.599 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:34.599 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:34.599 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:22:34.599 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.012145+0000", 2024-09-13T11:22:34.599 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00031069499999999999 2024-09-13T11:22:34.600 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:34.600 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:34.600 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 8", 2024-09-13T11:22:34.600 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.012539+0000", 2024-09-13T11:22:34.600 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000393726 2024-09-13T11:22:34.600 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:34.600 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:34.600 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:22:34.600 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.014263+0000", 2024-09-13T11:22:34.600 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0017240490000000001 2024-09-13T11:22:34.600 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:34.600 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:34.600 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:22:34.601 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.017429+0000", 2024-09-13T11:22:34.601 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0031662890000000001 2024-09-13T11:22:34.601 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:34.601 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:34.601 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:22:34.601 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.017517+0000", 2024-09-13T11:22:34.601 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.7763999999999995e-05 2024-09-13T11:22:34.601 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:34.601 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:34.601 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:22:34.601 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.017578+0000", 2024-09-13T11:22:34.601 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.1199999999999997e-05 2024-09-13T11:22:34.601 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:22:34.601 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:22:34.602 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:22:34.602 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:34.602 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:34.602 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:31 3.6 3:6ecddb78:::benchmark_data_smithi177_31116_object30:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-09-13T11:22:34.602 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:27.015333+0000", 2024-09-13T11:22:34.602 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 7.5262729000000004, 2024-09-13T11:22:34.602 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0049355609999999998, 2024-09-13T11:22:34.602 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:22:34.602 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:22:34.602 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:34.602 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:22:34.602 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:22:34.602 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:22:34.603 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 31 2024-09-13T11:22:34.603 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:34.603 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:22:34.603 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:34.603 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:22:34.603 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.015333+0000", 2024-09-13T11:22:34.603 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:22:34.603 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:34.603 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:34.603 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:22:34.603 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.015333+0000", 2024-09-13T11:22:34.603 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:22:34.603 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:34.603 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:34.604 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:22:34.604 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.015336+0000", 2024-09-13T11:22:34.604 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.7329999999999999e-06 2024-09-13T11:22:34.604 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:34.604 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:34.604 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:22:34.604 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.015455+0000", 2024-09-13T11:22:34.604 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000118406 2024-09-13T11:22:34.604 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:34.604 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:34.604 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:22:34.604 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.015476+0000", 2024-09-13T11:22:34.604 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.1012000000000001e-05 2024-09-13T11:22:34.605 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:34.605 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:34.605 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:22:34.605 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.015489+0000", 2024-09-13T11:22:34.605 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.3739e-05 2024-09-13T11:22:34.605 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:34.605 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:34.605 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:22:34.605 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.015560+0000", 2024-09-13T11:22:34.605 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.1025999999999994e-05 2024-09-13T11:22:34.605 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:34.605 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:34.605 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:22:34.606 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.015635+0000", 2024-09-13T11:22:34.606 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.5041000000000002e-05 2024-09-13T11:22:34.606 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:34.606 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:34.606 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 14", 2024-09-13T11:22:34.606 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.015717+0000", 2024-09-13T11:22:34.606 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.1240000000000001e-05 2024-09-13T11:22:34.606 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:34.606 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:34.606 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:22:34.606 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.017702+0000", 2024-09-13T11:22:34.606 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.001985428 2024-09-13T11:22:34.606 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:34.607 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:34.607 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:22:34.607 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.020150+0000", 2024-09-13T11:22:34.607 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0024484379999999998 2024-09-13T11:22:34.607 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:34.607 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:34.607 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:22:34.607 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.020215+0000", 2024-09-13T11:22:34.607 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.4293000000000003e-05 2024-09-13T11:22:34.607 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:34.607 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:34.607 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:22:34.607 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.020269+0000", 2024-09-13T11:22:34.608 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.4205e-05 2024-09-13T11:22:34.608 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:22:34.608 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:22:34.608 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:22:34.608 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:34.608 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:34.608 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:33 3.7 3:ebfb0388:::benchmark_data_smithi177_31116_object32:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-09-13T11:22:34.608 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:27.015768+0000", 2024-09-13T11:22:34.608 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 7.5258382529999999, 2024-09-13T11:22:34.608 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0027879620000000002, 2024-09-13T11:22:34.608 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:22:34.608 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:22:34.608 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:34.609 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:22:34.609 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:22:34.609 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:22:34.609 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 33 2024-09-13T11:22:34.609 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:34.609 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:22:34.609 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:34.609 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:22:34.609 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.015768+0000", 2024-09-13T11:22:34.609 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:22:34.609 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:34.609 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:34.609 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:22:34.610 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.015768+0000", 2024-09-13T11:22:34.610 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:22:34.610 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:34.610 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:34.610 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:22:34.610 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.015770+0000", 2024-09-13T11:22:34.610 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.3599999999999999e-06 2024-09-13T11:22:34.610 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:34.610 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:34.610 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:22:34.610 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.015829+0000", 2024-09-13T11:22:34.610 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.8461999999999997e-05 2024-09-13T11:22:34.610 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:34.610 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:34.611 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:22:34.611 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.015849+0000", 2024-09-13T11:22:34.611 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.0162000000000001e-05 2024-09-13T11:22:34.611 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:34.611 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:34.611 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:22:34.611 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.015875+0000", 2024-09-13T11:22:34.611 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.5775e-05 2024-09-13T11:22:34.611 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:34.611 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:34.611 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:22:34.611 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.016001+0000", 2024-09-13T11:22:34.611 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00012632499999999999 2024-09-13T11:22:34.612 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:34.612 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:34.612 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:22:34.612 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.016255+0000", 2024-09-13T11:22:34.612 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00025401800000000001 2024-09-13T11:22:34.612 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:34.612 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:34.612 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 8", 2024-09-13T11:22:34.612 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.016587+0000", 2024-09-13T11:22:34.612 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00033234699999999998 2024-09-13T11:22:34.612 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:34.612 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:34.613 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:22:34.613 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.018272+0000", 2024-09-13T11:22:34.613 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.001684266 2024-09-13T11:22:34.613 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:34.613 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:34.613 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:22:34.613 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.018386+0000", 2024-09-13T11:22:34.613 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000113949 2024-09-13T11:22:34.613 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:34.613 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:34.613 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:22:34.613 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.018485+0000", 2024-09-13T11:22:34.613 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.9530000000000004e-05 2024-09-13T11:22:34.614 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:34.614 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:34.614 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:22:34.614 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.018556+0000", 2024-09-13T11:22:34.614 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.0767999999999997e-05 2024-09-13T11:22:34.614 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:22:34.614 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:22:34.614 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:22:34.614 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:34.614 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:34.614 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:34 3.6 3:6628c15c:::benchmark_data_smithi177_31116_object33:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-09-13T11:22:34.614 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:27.016457+0000", 2024-09-13T11:22:34.614 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 7.5251491010000002, 2024-09-13T11:22:34.615 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0050175760000000002, 2024-09-13T11:22:34.615 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:22:34.615 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:22:34.615 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:34.615 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:22:34.615 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:22:34.615 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:22:34.615 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 34 2024-09-13T11:22:34.615 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:34.615 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:22:34.615 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:34.615 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:22:34.616 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.016457+0000", 2024-09-13T11:22:34.616 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:22:34.616 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:34.616 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:34.616 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:22:34.616 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.016457+0000", 2024-09-13T11:22:34.616 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:22:34.616 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:34.616 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:34.616 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:22:34.616 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.016460+0000", 2024-09-13T11:22:34.616 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.5629999999999999e-06 2024-09-13T11:22:34.616 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:34.617 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:34.617 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:22:34.617 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.016595+0000", 2024-09-13T11:22:34.617 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00013488399999999999 2024-09-13T11:22:34.617 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:34.617 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:34.617 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:22:34.617 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.016617+0000", 2024-09-13T11:22:34.617 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.2662000000000001e-05 2024-09-13T11:22:34.617 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:34.617 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:34.617 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:22:34.617 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.016634+0000", 2024-09-13T11:22:34.617 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.6419e-05 2024-09-13T11:22:34.618 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:34.618 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:34.618 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:22:34.618 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.016846+0000", 2024-09-13T11:22:34.618 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000212604 2024-09-13T11:22:34.618 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:34.618 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:34.618 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:22:34.618 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.016988+0000", 2024-09-13T11:22:34.618 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00014186900000000001 2024-09-13T11:22:34.618 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:34.618 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:34.618 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 14", 2024-09-13T11:22:34.619 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.017152+0000", 2024-09-13T11:22:34.619 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000163513 2024-09-13T11:22:34.619 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:34.619 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:34.619 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:22:34.619 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.018388+0000", 2024-09-13T11:22:34.619 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.001235964 2024-09-13T11:22:34.619 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:34.619 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:34.619 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:22:34.619 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.021396+0000", 2024-09-13T11:22:34.619 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0030087400000000002 2024-09-13T11:22:34.619 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:34.620 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:34.620 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:22:34.620 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.021445+0000", 2024-09-13T11:22:34.620 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.8655000000000001e-05 2024-09-13T11:22:34.620 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:34.620 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:34.620 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:22:34.620 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.021475+0000", 2024-09-13T11:22:34.620 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.9703e-05 2024-09-13T11:22:34.620 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:22:34.620 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:22:34.620 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:22:34.620 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:34.621 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:34.621 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:47 3.7 3:e22c870b:::benchmark_data_smithi177_31116_object46:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-09-13T11:22:34.621 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:27.021668+0000", 2024-09-13T11:22:34.621 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 7.5199378709999998, 2024-09-13T11:22:34.621 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0023262040000000001, 2024-09-13T11:22:34.621 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:22:34.621 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:22:34.621 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:34.621 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:22:34.621 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:22:34.621 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:22:34.621 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 47 2024-09-13T11:22:34.621 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:34.622 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:22:34.622 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:34.622 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:22:34.622 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.021668+0000", 2024-09-13T11:22:34.622 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:22:34.622 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:34.622 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:34.622 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:22:34.622 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.021668+0000", 2024-09-13T11:22:34.622 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:22:34.622 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:34.622 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:34.623 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:22:34.623 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.021671+0000", 2024-09-13T11:22:34.623 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.2560000000000001e-06 2024-09-13T11:22:34.623 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:34.623 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:34.623 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:22:34.623 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.021736+0000", 2024-09-13T11:22:34.623 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.5241999999999996e-05 2024-09-13T11:22:34.623 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:34.623 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:34.623 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:22:34.623 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.021751+0000", 2024-09-13T11:22:34.623 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.4718999999999999e-05 2024-09-13T11:22:34.623 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:34.624 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:34.624 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:22:34.624 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.021764+0000", 2024-09-13T11:22:34.624 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.3049e-05 2024-09-13T11:22:34.624 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:34.624 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:34.624 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:22:34.624 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.021904+0000", 2024-09-13T11:22:34.624 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00014050999999999999 2024-09-13T11:22:34.624 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:34.624 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:34.624 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:22:34.624 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.022142+0000", 2024-09-13T11:22:34.625 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00023751100000000001 2024-09-13T11:22:34.625 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:34.625 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:34.625 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 8", 2024-09-13T11:22:34.625 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.022375+0000", 2024-09-13T11:22:34.625 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00023364999999999999 2024-09-13T11:22:34.625 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:34.625 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:34.625 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:22:34.625 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.023308+0000", 2024-09-13T11:22:34.625 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00093230099999999996 2024-09-13T11:22:34.625 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:34.625 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:34.626 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:22:34.626 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.023835+0000", 2024-09-13T11:22:34.626 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00052771100000000004 2024-09-13T11:22:34.626 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:34.626 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:34.626 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:22:34.626 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.023923+0000", 2024-09-13T11:22:34.626 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.7622999999999998e-05 2024-09-13T11:22:34.626 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:34.626 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:34.626 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:22:34.626 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.023995+0000", 2024-09-13T11:22:34.626 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.1631999999999995e-05 2024-09-13T11:22:34.626 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:22:34.627 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:22:34.627 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:22:34.627 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:34.627 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:34.627 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:51 3.7 3:e2e297cd:::benchmark_data_smithi177_31116_object50:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-09-13T11:22:34.627 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:27.024439+0000", 2024-09-13T11:22:34.627 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 7.5171677179999996, 2024-09-13T11:22:34.627 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0028375190000000002, 2024-09-13T11:22:34.627 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:22:34.627 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:22:34.627 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:34.627 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:22:34.627 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:22:34.627 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:22:34.628 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 51 2024-09-13T11:22:34.628 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:34.628 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:22:34.628 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:34.628 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:22:34.628 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.024439+0000", 2024-09-13T11:22:34.628 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:22:34.628 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:34.628 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:34.628 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:22:34.628 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.024439+0000", 2024-09-13T11:22:34.628 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:22:34.628 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:34.629 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:34.629 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:22:34.629 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.024441+0000", 2024-09-13T11:22:34.629 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.9800000000000001e-06 2024-09-13T11:22:34.629 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:34.629 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:34.629 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:22:34.629 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.024509+0000", 2024-09-13T11:22:34.629 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.8479000000000006e-05 2024-09-13T11:22:34.629 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:34.629 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:34.629 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:22:34.629 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.024529+0000", 2024-09-13T11:22:34.630 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.9731000000000001e-05 2024-09-13T11:22:34.630 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:34.630 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:34.630 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:22:34.630 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.024543+0000", 2024-09-13T11:22:34.630 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.3852999999999999e-05 2024-09-13T11:22:34.630 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:34.630 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:34.630 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:22:34.630 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.024661+0000", 2024-09-13T11:22:34.630 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00011882799999999999 2024-09-13T11:22:34.630 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:34.630 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:34.630 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:22:34.631 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.024906+0000", 2024-09-13T11:22:34.631 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00024457 2024-09-13T11:22:34.631 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:34.631 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:34.631 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 8", 2024-09-13T11:22:34.631 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.025143+0000", 2024-09-13T11:22:34.631 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00023677999999999999 2024-09-13T11:22:34.631 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:34.631 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:34.631 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:22:34.631 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.026960+0000", 2024-09-13T11:22:34.631 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0018172799999999999 2024-09-13T11:22:34.631 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:34.632 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:34.632 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:22:34.632 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.027030+0000", 2024-09-13T11:22:34.632 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.0068999999999996e-05 2024-09-13T11:22:34.632 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:34.632 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:34.632 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:22:34.632 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.027172+0000", 2024-09-13T11:22:34.632 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00014156999999999999 2024-09-13T11:22:34.632 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:34.632 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:34.632 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:22:34.632 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.027276+0000", 2024-09-13T11:22:34.632 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000104379 2024-09-13T11:22:34.633 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:22:34.633 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:22:34.633 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:22:34.633 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:34.633 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:34.633 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:53 3.7 3:e998c81b:::benchmark_data_smithi177_31116_object52:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-09-13T11:22:34.633 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:27.024742+0000", 2024-09-13T11:22:34.633 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 7.5168645889999999, 2024-09-13T11:22:34.634 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0031661129999999999, 2024-09-13T11:22:34.634 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:22:34.634 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:22:34.634 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:34.634 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:22:34.634 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:22:34.634 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:22:34.634 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 53 2024-09-13T11:22:34.634 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:34.634 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:22:34.634 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:34.634 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:22:34.634 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.024742+0000", 2024-09-13T11:22:34.634 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:22:34.635 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:34.635 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:34.635 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:22:34.635 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.024742+0000", 2024-09-13T11:22:34.635 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:22:34.635 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:34.635 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:34.635 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:22:34.635 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.024743+0000", 2024-09-13T11:22:34.635 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.4959999999999999e-06 2024-09-13T11:22:34.635 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:34.635 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:34.635 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:22:34.636 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.024822+0000", 2024-09-13T11:22:34.636 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.8792e-05 2024-09-13T11:22:34.636 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:34.636 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:34.636 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:22:34.636 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.024834+0000", 2024-09-13T11:22:34.636 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.2449000000000001e-05 2024-09-13T11:22:34.636 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:34.636 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:34.636 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:22:34.636 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.024850+0000", 2024-09-13T11:22:34.636 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.5855999999999999e-05 2024-09-13T11:22:34.636 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:34.636 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:34.637 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:22:34.637 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.025825+0000", 2024-09-13T11:22:34.637 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00097482799999999996 2024-09-13T11:22:34.637 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:34.637 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:34.637 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:22:34.637 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.026052+0000", 2024-09-13T11:22:34.637 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000226851 2024-09-13T11:22:34.637 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:34.637 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:34.637 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 8", 2024-09-13T11:22:34.637 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.026296+0000", 2024-09-13T11:22:34.637 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00024420600000000001 2024-09-13T11:22:34.638 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:34.638 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:34.638 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:22:34.638 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.027344+0000", 2024-09-13T11:22:34.638 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.001048064 2024-09-13T11:22:34.638 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:34.638 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:34.638 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:22:34.638 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.027757+0000", 2024-09-13T11:22:34.638 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00041227100000000002 2024-09-13T11:22:34.638 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:34.638 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:34.638 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:22:34.638 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.027845+0000", 2024-09-13T11:22:34.639 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.8861000000000002e-05 2024-09-13T11:22:34.639 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:34.639 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:34.639 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:22:34.639 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.027908+0000", 2024-09-13T11:22:34.639 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.2439000000000003e-05 2024-09-13T11:22:34.639 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:22:34.639 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:22:34.639 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:22:34.639 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:34.639 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:34.639 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:57 3.7 3:e5e6d1ae:::benchmark_data_smithi177_31116_object56:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-09-13T11:22:34.639 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:27.028429+0000", 2024-09-13T11:22:34.640 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 7.5131768000000001, 2024-09-13T11:22:34.640 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.002334198, 2024-09-13T11:22:34.640 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:22:34.640 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:22:34.640 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:34.640 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:22:34.640 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:22:34.640 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:22:34.640 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 57 2024-09-13T11:22:34.640 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:34.640 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:22:34.640 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:34.640 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:22:34.641 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.028429+0000", 2024-09-13T11:22:34.641 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:22:34.641 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:34.641 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:34.641 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:22:34.641 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.028429+0000", 2024-09-13T11:22:34.641 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:22:34.641 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:34.641 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:34.641 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:22:34.641 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.028432+0000", 2024-09-13T11:22:34.641 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.39e-06 2024-09-13T11:22:34.641 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:34.642 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:34.642 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:22:34.642 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.028505+0000", 2024-09-13T11:22:34.642 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.3481999999999999e-05 2024-09-13T11:22:34.642 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:34.642 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:34.642 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:22:34.642 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.028537+0000", 2024-09-13T11:22:34.642 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.1294999999999998e-05 2024-09-13T11:22:34.642 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:34.642 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:34.642 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:22:34.642 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.028550+0000", 2024-09-13T11:22:34.643 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.3585e-05 2024-09-13T11:22:34.643 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:34.643 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:34.643 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:22:34.643 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.028669+0000", 2024-09-13T11:22:34.643 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00011893800000000001 2024-09-13T11:22:34.643 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:34.643 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:34.643 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:22:34.643 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.028883+0000", 2024-09-13T11:22:34.643 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00021342600000000001 2024-09-13T11:22:34.643 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:34.643 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:34.644 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 8", 2024-09-13T11:22:34.644 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.029093+0000", 2024-09-13T11:22:34.644 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000210588 2024-09-13T11:22:34.644 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:34.644 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:34.644 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:22:34.644 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.030241+0000", 2024-09-13T11:22:34.644 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.001148264 2024-09-13T11:22:34.644 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:34.644 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:34.644 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:22:34.644 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.030608+0000", 2024-09-13T11:22:34.644 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00036654400000000001 2024-09-13T11:22:34.645 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:34.645 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:34.645 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:22:34.645 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.030697+0000", 2024-09-13T11:22:34.645 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.8700000000000001e-05 2024-09-13T11:22:34.645 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:34.645 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:34.645 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:22:34.645 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.030764+0000", 2024-09-13T11:22:34.645 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.6985999999999999e-05 2024-09-13T11:22:34.645 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:22:34.645 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:22:34.645 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:22:34.646 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:34.646 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:34.646 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:58 3.1 3:893a52d5:::benchmark_data_smithi177_31116_object57:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-09-13T11:22:34.646 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:27.028577+0000", 2024-09-13T11:22:34.646 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 7.5130288329999999, 2024-09-13T11:22:34.646 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0033237890000000002, 2024-09-13T11:22:34.646 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:22:34.646 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:22:34.646 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:34.646 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:22:34.646 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:22:34.646 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:22:34.646 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 58 2024-09-13T11:22:34.646 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:34.647 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:22:34.647 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:34.647 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:22:34.647 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.028577+0000", 2024-09-13T11:22:34.647 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:22:34.647 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:34.647 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:34.647 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:22:34.647 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.028577+0000", 2024-09-13T11:22:34.647 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:22:34.647 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:34.647 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:34.648 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:22:34.648 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.028579+0000", 2024-09-13T11:22:34.648 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.553e-06 2024-09-13T11:22:34.648 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:34.648 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:34.648 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:22:34.648 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.028629+0000", 2024-09-13T11:22:34.648 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.0365999999999999e-05 2024-09-13T11:22:34.648 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:34.648 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:34.648 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:22:34.648 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.028645+0000", 2024-09-13T11:22:34.649 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.5285999999999999e-05 2024-09-13T11:22:34.649 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:34.649 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:34.649 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:22:34.649 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.028653+0000", 2024-09-13T11:22:34.649 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.2500000000000006e-06 2024-09-13T11:22:34.649 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:34.649 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:34.649 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:22:34.649 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.028741+0000", 2024-09-13T11:22:34.650 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.7769000000000006e-05 2024-09-13T11:22:34.650 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:34.650 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:34.650 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:22:34.650 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.028902+0000", 2024-09-13T11:22:34.650 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000161626 2024-09-13T11:22:34.650 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:34.650 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:34.650 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 11", 2024-09-13T11:22:34.650 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.029093+0000", 2024-09-13T11:22:34.650 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000190736 2024-09-13T11:22:34.650 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:34.651 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:34.651 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:22:34.651 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.029959+0000", 2024-09-13T11:22:34.651 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00086562499999999997 2024-09-13T11:22:34.651 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:34.651 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:34.651 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:22:34.651 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.031758+0000", 2024-09-13T11:22:34.651 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0017993110000000001 2024-09-13T11:22:34.651 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:34.651 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:34.651 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:22:34.651 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.031836+0000", 2024-09-13T11:22:34.652 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.7733999999999998e-05 2024-09-13T11:22:34.652 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:34.652 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:34.652 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:22:34.652 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.031901+0000", 2024-09-13T11:22:34.652 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.5532999999999998e-05 2024-09-13T11:22:34.652 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:22:34.652 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:22:34.652 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:22:34.652 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:34.652 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:34.652 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:74 3.6 3:65c9aede:::benchmark_data_smithi177_31116_object73:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-09-13T11:22:34.652 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:27.039439+0000", 2024-09-13T11:22:34.653 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 7.5021673770000001, 2024-09-13T11:22:34.653 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.002444955, 2024-09-13T11:22:34.653 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:22:34.653 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:22:34.653 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:34.653 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:22:34.653 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:22:34.653 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:22:34.653 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 74 2024-09-13T11:22:34.653 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:34.653 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:22:34.653 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:34.653 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:22:34.654 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.039439+0000", 2024-09-13T11:22:34.654 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:22:34.654 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:34.654 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:34.654 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:22:34.654 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.039439+0000", 2024-09-13T11:22:34.654 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:22:34.654 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:34.654 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:34.654 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:22:34.654 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.039442+0000", 2024-09-13T11:22:34.654 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.9170000000000002e-06 2024-09-13T11:22:34.654 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:34.655 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:34.655 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:22:34.655 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.039506+0000", 2024-09-13T11:22:34.655 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.4022000000000005e-05 2024-09-13T11:22:34.655 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:34.655 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:34.655 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:22:34.655 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.039532+0000", 2024-09-13T11:22:34.655 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.5799000000000002e-05 2024-09-13T11:22:34.655 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:34.655 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:34.655 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:22:34.656 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.039548+0000", 2024-09-13T11:22:34.656 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.5971999999999999e-05 2024-09-13T11:22:34.656 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:34.656 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:34.656 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:22:34.656 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.039615+0000", 2024-09-13T11:22:34.656 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.7372999999999993e-05 2024-09-13T11:22:34.656 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:34.656 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:34.656 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:22:34.656 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.039693+0000", 2024-09-13T11:22:34.656 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.8088000000000002e-05 2024-09-13T11:22:34.656 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:34.657 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:34.657 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 14", 2024-09-13T11:22:34.657 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.039772+0000", 2024-09-13T11:22:34.657 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.8974000000000007e-05 2024-09-13T11:22:34.657 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:34.657 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:34.657 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:22:34.657 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.040636+0000", 2024-09-13T11:22:34.657 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00086353899999999997 2024-09-13T11:22:34.657 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:34.657 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:34.657 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:22:34.657 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.041817+0000", 2024-09-13T11:22:34.658 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0011810550000000001 2024-09-13T11:22:34.658 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:34.658 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:34.658 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:22:34.658 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.041859+0000", 2024-09-13T11:22:34.658 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.2571999999999998e-05 2024-09-13T11:22:34.658 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:34.658 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:34.658 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:22:34.658 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.041884+0000", 2024-09-13T11:22:34.658 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.4644e-05 2024-09-13T11:22:34.658 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:22:34.658 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:22:34.659 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:22:34.659 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:34.659 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:34.659 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:79 3.1 3:841df5b5:::benchmark_data_smithi177_31116_object78:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-09-13T11:22:34.659 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:27.056083+0000", 2024-09-13T11:22:34.659 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 7.485523734, 2024-09-13T11:22:34.659 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.89469730199999997, 2024-09-13T11:22:34.659 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:22:34.659 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:22:34.659 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:34.659 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:22:34.659 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:22:34.659 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:22:34.660 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 79 2024-09-13T11:22:34.660 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:34.660 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:22:34.660 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:34.660 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:22:34.660 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.056083+0000", 2024-09-13T11:22:34.660 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:22:34.660 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:34.660 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:34.660 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:22:34.660 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.056083+0000", 2024-09-13T11:22:34.660 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:22:34.660 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:34.661 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:34.661 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:22:34.661 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.056087+0000", 2024-09-13T11:22:34.661 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.0659999999999997e-06 2024-09-13T11:22:34.661 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:34.661 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:34.661 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:22:34.661 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.056148+0000", 2024-09-13T11:22:34.661 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.1376000000000003e-05 2024-09-13T11:22:34.661 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:34.661 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:34.661 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:22:34.661 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.056180+0000", 2024-09-13T11:22:34.662 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.2370999999999999e-05 2024-09-13T11:22:34.662 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:34.662 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:34.662 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:22:34.662 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.056195+0000", 2024-09-13T11:22:34.662 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.5119e-05 2024-09-13T11:22:34.662 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:34.662 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:34.662 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:22:34.662 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.056304+0000", 2024-09-13T11:22:34.662 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000108365 2024-09-13T11:22:34.662 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:34.662 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:34.663 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:22:34.663 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.056478+0000", 2024-09-13T11:22:34.663 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000174542 2024-09-13T11:22:34.663 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:34.663 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:34.663 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 11", 2024-09-13T11:22:34.663 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.056690+0000", 2024-09-13T11:22:34.663 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00021167500000000001 2024-09-13T11:22:34.663 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:34.663 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:34.663 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:22:34.663 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.058063+0000", 2024-09-13T11:22:34.663 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0013731819999999999 2024-09-13T11:22:34.664 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:34.664 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:34.664 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:22:34.664 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.950477+0000", 2024-09-13T11:22:34.664 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.89241369400000004 2024-09-13T11:22:34.664 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:34.664 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:34.664 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:22:34.664 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.950652+0000", 2024-09-13T11:22:34.664 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00017526400000000001 2024-09-13T11:22:34.664 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:34.664 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:34.664 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:22:34.665 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.950780+0000", 2024-09-13T11:22:34.665 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00012764800000000001 2024-09-13T11:22:34.665 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:22:34.665 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:22:34.665 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:22:34.665 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:34.665 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:34.665 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop(client.4665.0:84 3.c e33/30 3:35015570:::benchmark_data_smithi177_31116_object83:head v 33'5, mlcod=32'3)", 2024-09-13T11:22:34.665 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:27.973328+0000", 2024-09-13T11:22:34.665 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 6.5682783809999998, 2024-09-13T11:22:34.665 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0022492699999999998, 2024-09-13T11:22:34.665 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:22:34.665 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:22:34.666 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:34.666 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:22:34.666 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:34.666 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:22:34.666 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.973328+0000", 2024-09-13T11:22:34.666 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:22:34.666 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:34.666 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:34.666 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:22:34.666 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.973328+0000", 2024-09-13T11:22:34.666 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:22:34.666 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:34.666 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:34.667 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:22:34.667 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.973328+0000", 2024-09-13T11:22:34.667 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.0699999999999997e-07 2024-09-13T11:22:34.667 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:34.667 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:34.667 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:22:34.667 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.973563+0000", 2024-09-13T11:22:34.667 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00023503 2024-09-13T11:22:34.667 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:34.667 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:34.667 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:22:34.667 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.973615+0000", 2024-09-13T11:22:34.667 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.1959999999999997e-05 2024-09-13T11:22:34.668 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:34.668 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:34.668 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:22:34.668 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.973631+0000", 2024-09-13T11:22:34.668 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.5235000000000001e-05 2024-09-13T11:22:34.668 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:34.668 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:34.668 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:22:34.668 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.973776+0000", 2024-09-13T11:22:34.668 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00014576300000000001 2024-09-13T11:22:34.668 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:34.668 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:34.668 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:22:34.669 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.973817+0000", 2024-09-13T11:22:34.669 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.0428e-05 2024-09-13T11:22:34.669 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:34.669 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:34.669 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:22:34.669 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.975476+0000", 2024-09-13T11:22:34.669 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.001659111 2024-09-13T11:22:34.669 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:34.669 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:34.669 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:22:34.669 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.975577+0000", 2024-09-13T11:22:34.669 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000101236 2024-09-13T11:22:34.669 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:22:34.670 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:22:34.670 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:22:34.670 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:22:34.670 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:22:34.670 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-09-13T11:22:34.671 INFO:teuthology.orchestra.run.smithi001.stdout:osd.9: {} 2024-09-13T11:22:34.671 INFO:teuthology.orchestra.run.smithi001.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-13T11:22:34.672 INFO:teuthology.orchestra.run.smithi001.stdout:osd.10: {} 2024-09-13T11:22:34.672 INFO:teuthology.orchestra.run.smithi001.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-13T11:22:34.673 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-13T11:22:34.673 INFO:teuthology.orchestra.run.smithi001.stdout:osd.11: {} 2024-09-13T11:22:34.673 INFO:teuthology.orchestra.run.smithi001.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-13T11:22:34.674 INFO:teuthology.orchestra.run.smithi001.stdout:osd.12: {} 2024-09-13T11:22:34.674 INFO:teuthology.orchestra.run.smithi001.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-13T11:22:34.674 INFO:tasks.ceph.osd.14.smithi162.stderr:2024-09-13T11:22:34.668+0000 7f235c0db640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26319) UID: 0 2024-09-13T11:22:34.680 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-09-13T11:22:34.676+0000 7f97b39f6640 -1 osd.6 34 handle_osd_map injecting map crc failure 2024-09-13T11:22:34.704 INFO:teuthology.orchestra.run.smithi001.stdout:osd.13: {} 2024-09-13T11:22:34.704 INFO:teuthology.orchestra.run.smithi001.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-13T11:22:34.737 INFO:teuthology.orchestra.run.smithi001.stdout:osd.14: {} 2024-09-13T11:22:34.737 INFO:teuthology.orchestra.run.smithi001.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-13T11:22:34.772 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-09-13T11:22:34.768+0000 7f97befc9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26315) UID: 0 2024-09-13T11:22:34.774 INFO:teuthology.orchestra.run.smithi001.stdout:osd.15: {} 2024-09-13T11:22:34.774 INFO:teuthology.orchestra.run.smithi001.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-13T11:22:34.781 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-13T11:22:34.781 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-13T11:22:34.807 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-13T11:22:34.807 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:22:34.821 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-13T11:22:34.909 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15] out_osds: [] dead_osds: [1] live_osds: [0, 4, 8, 12, 5, 9, 13, 2, 6, 10, 14, 3, 7, 11, 15] 2024-09-13T11:22:34.909 INFO:tasks.thrashosds.thrasher:choose_action: min_in 2 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-13T11:22:34.909 INFO:tasks.thrashosds.thrasher:inject_pause on osd.6 2024-09-13T11:22:34.909 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-13T11:22:34.909 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-13T11:22:34.910 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok config set filestore_inject_stall 3 2024-09-13T11:22:34.910 INFO:tasks.ceph.osd.14.smithi162.stderr:2024-09-13T11:22:34.908+0000 7f235c0db640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26319) UID: 0 2024-09-13T11:22:34.944 INFO:tasks.ceph.osd.3.smithi177.stderr:2024-09-13T11:22:34.940+0000 7f10a25d1640 -1 osd.3 34 handle_osd_map injecting map crc failure 2024-09-13T11:22:34.944 INFO:tasks.ceph.osd.12.smithi001.stderr:2024-09-13T11:22:34.940+0000 7f891d10b640 -1 osd.12 34 handle_osd_map injecting map crc failure 2024-09-13T11:22:34.973 INFO:tasks.ceph.osd.14.smithi162.stderr:2024-09-13T11:22:34.972+0000 7f235c0db640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26319) UID: 0 2024-09-13T11:22:34.990 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-13T11:22:34.990 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:22:34.996 INFO:tasks.radosbench.radosbench.0.smithi177.stdout: 8 16 87 71 0.55461 0 - 0.0234801 2024-09-13T11:22:35.005 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-13T11:22:35.042 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-13T11:22:35.042 INFO:teuthology.orchestra.run.smithi162.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-09-13T11:22:35.042 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-13T11:22:35.075 INFO:tasks.ceph.osd.11.smithi177.stderr:2024-09-13T11:22:35.072+0000 7f24792c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26580) UID: 0 2024-09-13T11:22:35.171 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-13T11:22:35.172 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:22:35.175 INFO:tasks.ceph.osd.3.smithi177.stderr:2024-09-13T11:22:35.172+0000 7f10adba4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26574) UID: 0 2024-09-13T11:22:35.186 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-13T11:22:35.275 INFO:tasks.ceph.osd.14.smithi162.stderr:2024-09-13T11:22:35.272+0000 7f235c0db640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26319) UID: 0 2024-09-13T11:22:35.310 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-13T11:22:35.310 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:22:35.324 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-13T11:22:35.417 INFO:tasks.ceph.osd.2.smithi162.stderr:2024-09-13T11:22:35.416+0000 7fe92b35a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26313) UID: 0 2024-09-13T11:22:35.477 INFO:tasks.ceph.osd.5.smithi092.stderr:2024-09-13T11:22:35.475+0000 7f6f0b705640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26346) UID: 0 2024-09-13T11:22:35.493 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-13T11:22:35.493 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:22:35.507 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-13T11:22:35.593 INFO:tasks.ceph.osd.2.smithi162.stderr:2024-09-13T11:22:35.592+0000 7fe92b35a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26313) UID: 0 2024-09-13T11:22:35.670 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-13T11:22:35.670 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:22:35.678 INFO:tasks.ceph.osd.8.smithi001.stderr:2024-09-13T11:22:35.676+0000 7fb9b48ae640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26312) UID: 0 2024-09-13T11:22:35.684 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-09-13T11:22:35.779 INFO:tasks.ceph.osd.8.smithi001.stderr:2024-09-13T11:22:35.776+0000 7fb9b48ae640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26312) UID: 0 2024-09-13T11:22:35.853 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-13T11:22:35.853 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:22:35.868 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-09-13T11:22:35.913 INFO:tasks.ceph.osd.14.smithi162.stderr:2024-09-13T11:22:35.912+0000 7f235c0db640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26319) UID: 0 2024-09-13T11:22:35.981 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-13T11:22:35.976+0000 7f6a68775640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26307) UID: 0 2024-09-13T11:22:35.992 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-13T11:22:35.992 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:22:35.996 INFO:tasks.radosbench.radosbench.0.smithi177.stdout: 9 16 87 71 0.492986 0 - 0.0234801 2024-09-13T11:22:36.007 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-09-13T11:22:36.081 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-13T11:22:36.076+0000 7f6a68775640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26307) UID: 0 2024-09-13T11:22:36.177 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-13T11:22:36.177 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:22:36.182 INFO:tasks.ceph.osd.2.smithi162.stderr:2024-09-13T11:22:36.180+0000 7fe92b35a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26313) UID: 0 2024-09-13T11:22:36.191 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_ops_in_flight 2024-09-13T11:22:36.241 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-13T11:22:36.284 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-13T11:22:36.280+0000 7f6a68775640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26307) UID: 0 2024-09-13T11:22:36.357 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-13T11:22:36.358 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:22:36.371 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_blocked_ops 2024-09-13T11:22:36.383 INFO:tasks.ceph.osd.5.smithi092.stderr:2024-09-13T11:22:36.379+0000 7f6f0b705640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26346) UID: 0 2024-09-13T11:22:36.484 INFO:tasks.ceph.osd.2.smithi162.stderr:2024-09-13T11:22:36.480+0000 7fe92b35a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26313) UID: 0 2024-09-13T11:22:36.487 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-13T11:22:36.488 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:22:36.503 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_historic_ops 2024-09-13T11:22:36.585 INFO:tasks.ceph.osd.4.smithi001.stderr:2024-09-13T11:22:36.580+0000 7ff8e0218640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26310) UID: 0 2024-09-13T11:22:36.665 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-13T11:22:36.665 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:22:36.679 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-13T11:22:36.686 INFO:tasks.ceph.osd.10.smithi162.stderr:2024-09-13T11:22:36.684+0000 7fb59c9f0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26317) UID: 0 2024-09-13T11:22:36.782 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T11:22:36.812 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-09-13T11:22:36.812 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:22:36.826 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-13T11:22:36.828 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-13T11:22:36.824+0000 7f6a68775640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26307) UID: 0 2024-09-13T11:22:36.888 INFO:tasks.ceph.osd.9.smithi092.stderr:2024-09-13T11:22:36.887+0000 7ff5ab3d0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26348) UID: 0 2024-09-13T11:22:36.942 INFO:teuthology.orchestra.run.smithi001.stdout:osd.0: {} 2024-09-13T11:22:36.942 INFO:teuthology.orchestra.run.smithi001.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T11:22:36.942 INFO:teuthology.orchestra.run.smithi001.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-09-13T11:22:36.949 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-09-13T11:22:36.949 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:22:36.963 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-13T11:22:36.967 INFO:teuthology.orchestra.run.smithi001.stdout:osd.2: {} 2024-09-13T11:22:36.967 INFO:teuthology.orchestra.run.smithi001.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T11:22:36.990 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-13T11:22:36.984+0000 7f6a68775640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26307) UID: 0 2024-09-13T11:22:36.993 INFO:teuthology.orchestra.run.smithi001.stdout:osd.3: {} 2024-09-13T11:22:36.993 INFO:teuthology.orchestra.run.smithi001.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T11:22:36.997 INFO:tasks.radosbench.radosbench.0.smithi177.stdout: 10 16 87 71 0.443689 0 - 0.0234801 2024-09-13T11:22:37.019 INFO:teuthology.orchestra.run.smithi001.stdout:osd.4: {} 2024-09-13T11:22:37.019 INFO:teuthology.orchestra.run.smithi001.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T11:22:37.048 INFO:teuthology.orchestra.run.smithi001.stdout:osd.5: {} 2024-09-13T11:22:37.048 INFO:teuthology.orchestra.run.smithi001.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T11:22:37.081 INFO:teuthology.orchestra.run.smithi001.stdout:osd.6: {} 2024-09-13T11:22:37.081 INFO:teuthology.orchestra.run.smithi001.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T11:22:37.090 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-09-13T11:22:37.090 INFO:teuthology.orchestra.run.smithi177.stdout: "size": 20, 2024-09-13T11:22:37.090 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 600, 2024-09-13T11:22:37.090 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [ 2024-09-13T11:22:37.090 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.090 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4344.0:2 2.2 2:5cea7035:::rbd_info:head [call rbd.metadata_list in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e26)", 2024-09-13T11:22:37.090 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:22:14.162172+0000", 2024-09-13T11:22:37.090 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 22.927662578, 2024-09-13T11:22:37.090 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000295464, 2024-09-13T11:22:37.090 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:22:37.091 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:22:37.091 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-13T11:22:37.091 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:22:37.091 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4344", 2024-09-13T11:22:37.091 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.1:0/211923147", 2024-09-13T11:22:37.091 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 2 2024-09-13T11:22:37.091 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.091 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:22:37.091 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.091 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:22:37.091 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:14.162172+0000", 2024-09-13T11:22:37.091 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:37.091 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.091 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.092 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:22:37.092 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:14.162172+0000", 2024-09-13T11:22:37.092 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:37.092 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.092 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.092 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:22:37.092 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:14.162185+0000", 2024-09-13T11:22:37.092 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.2259000000000001e-05 2024-09-13T11:22:37.092 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.092 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.092 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:22:37.092 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:14.162196+0000", 2024-09-13T11:22:37.092 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.1202e-05 2024-09-13T11:22:37.093 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.093 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.093 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:22:37.093 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:14.162214+0000", 2024-09-13T11:22:37.093 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.8601999999999999e-05 2024-09-13T11:22:37.093 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.093 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.093 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:22:37.093 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:14.162227+0000", 2024-09-13T11:22:37.093 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.2877999999999999e-05 2024-09-13T11:22:37.093 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.093 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.093 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:22:37.093 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:14.162311+0000", 2024-09-13T11:22:37.094 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.4031999999999998e-05 2024-09-13T11:22:37.094 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.094 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.094 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:22:37.094 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:14.162468+0000", 2024-09-13T11:22:37.094 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000156491 2024-09-13T11:22:37.094 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:37.094 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:22:37.094 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:37.094 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.094 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.094 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4344.0:3 2.2 2:5cea7035:::rbd_info:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e26)", 2024-09-13T11:22:37.094 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:22:14.163003+0000", 2024-09-13T11:22:37.094 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 22.926831858, 2024-09-13T11:22:37.095 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00013798799999999999, 2024-09-13T11:22:37.095 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:22:37.095 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:22:37.095 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-13T11:22:37.095 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:22:37.095 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4344", 2024-09-13T11:22:37.095 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.1:0/211923147", 2024-09-13T11:22:37.095 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 3 2024-09-13T11:22:37.095 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.095 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:22:37.095 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.095 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:22:37.095 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:14.163003+0000", 2024-09-13T11:22:37.096 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:37.096 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.096 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.096 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:22:37.096 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:14.163003+0000", 2024-09-13T11:22:37.096 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:37.096 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.096 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.096 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:22:37.096 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:14.163004+0000", 2024-09-13T11:22:37.096 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3200000000000001e-06 2024-09-13T11:22:37.096 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.096 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.096 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:22:37.097 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:14.163009+0000", 2024-09-13T11:22:37.097 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.5839999999999996e-06 2024-09-13T11:22:37.097 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.097 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.097 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:22:37.097 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:14.163019+0000", 2024-09-13T11:22:37.097 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0433e-05 2024-09-13T11:22:37.097 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.097 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.097 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:22:37.097 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:14.163029+0000", 2024-09-13T11:22:37.097 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.4809999999999995e-06 2024-09-13T11:22:37.097 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.098 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.098 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:22:37.098 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:14.163070+0000", 2024-09-13T11:22:37.098 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.0799e-05 2024-09-13T11:22:37.098 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.098 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.098 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:22:37.098 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:14.163141+0000", 2024-09-13T11:22:37.098 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.1371000000000005e-05 2024-09-13T11:22:37.098 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:37.098 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:22:37.098 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:37.098 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.098 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.099 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4344.0:5 2.2 2:5cea7035:::rbd_info:head [create,write 0~8] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-09-13T11:22:37.099 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:22:15.243111+0000", 2024-09-13T11:22:37.099 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 21.84672424, 2024-09-13T11:22:37.099 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.010509383000000001, 2024-09-13T11:22:37.099 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:22:37.099 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:22:37.099 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:37.099 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:22:37.099 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4344", 2024-09-13T11:22:37.099 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.1:0/211923147", 2024-09-13T11:22:37.099 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 5 2024-09-13T11:22:37.099 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.099 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:22:37.099 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.100 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:22:37.100 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:15.243111+0000", 2024-09-13T11:22:37.100 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:37.100 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.100 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.100 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:22:37.100 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:15.243111+0000", 2024-09-13T11:22:37.100 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:37.100 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.100 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.100 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:22:37.100 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:15.243115+0000", 2024-09-13T11:22:37.100 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.6199999999999998e-06 2024-09-13T11:22:37.100 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.101 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.101 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:22:37.101 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:15.243130+0000", 2024-09-13T11:22:37.101 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5079000000000001e-05 2024-09-13T11:22:37.101 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.101 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.101 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:22:37.102 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:15.243157+0000", 2024-09-13T11:22:37.102 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.6254e-05 2024-09-13T11:22:37.102 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.102 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.102 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:22:37.102 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:15.243175+0000", 2024-09-13T11:22:37.102 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.8062000000000001e-05 2024-09-13T11:22:37.102 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.102 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.102 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:22:37.102 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:15.243327+0000", 2024-09-13T11:22:37.102 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00015213999999999999 2024-09-13T11:22:37.102 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.102 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.103 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op must wait for map", 2024-09-13T11:22:37.103 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:15.243338+0000", 2024-09-13T11:22:37.103 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0866000000000001e-05 2024-09-13T11:22:37.103 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.103 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.103 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:22:37.103 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:15.246468+0000", 2024-09-13T11:22:37.103 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0031299980000000002 2024-09-13T11:22:37.103 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.103 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.103 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:22:37.103 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:15.246813+0000", 2024-09-13T11:22:37.103 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00034492099999999999 2024-09-13T11:22:37.104 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.104 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.104 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 9", 2024-09-13T11:22:37.104 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:15.247226+0000", 2024-09-13T11:22:37.104 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00041305699999999999 2024-09-13T11:22:37.104 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.104 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.104 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:22:37.104 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:15.252047+0000", 2024-09-13T11:22:37.104 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0048209419999999999 2024-09-13T11:22:37.104 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.104 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.104 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:22:37.105 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:15.253466+0000", 2024-09-13T11:22:37.105 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001419844 2024-09-13T11:22:37.105 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.105 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.105 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:22:37.105 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:15.253551+0000", 2024-09-13T11:22:37.105 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.4646000000000002e-05 2024-09-13T11:22:37.105 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.105 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.105 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:22:37.105 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:15.253620+0000", 2024-09-13T11:22:37.105 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.8954000000000004e-05 2024-09-13T11:22:37.105 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:37.105 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:22:37.106 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:37.106 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.106 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.106 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop_reply(client.4344.0:5 2.2 e26/25 ondisk, result = 0)", 2024-09-13T11:22:37.106 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:22:15.253256+0000", 2024-09-13T11:22:37.106 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 21.836578852999999, 2024-09-13T11:22:37.106 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00038215699999999999, 2024-09-13T11:22:37.106 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:22:37.106 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:22:37.106 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:22:37.106 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:22:37.106 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.106 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:22:37.107 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:15.253256+0000", 2024-09-13T11:22:37.107 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:37.107 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.107 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.107 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:22:37.107 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:15.253256+0000", 2024-09-13T11:22:37.107 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:37.107 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.107 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.107 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:22:37.107 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:15.253260+0000", 2024-09-13T11:22:37.107 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.597e-06 2024-09-13T11:22:37.107 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.107 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.108 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:22:37.108 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:15.253271+0000", 2024-09-13T11:22:37.108 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.1712000000000001e-05 2024-09-13T11:22:37.108 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.108 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.108 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:22:37.108 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:15.253339+0000", 2024-09-13T11:22:37.108 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.8001000000000002e-05 2024-09-13T11:22:37.108 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.108 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.108 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:22:37.108 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:15.253354+0000", 2024-09-13T11:22:37.109 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.4786e-05 2024-09-13T11:22:37.109 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.109 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.109 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:22:37.109 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:15.253445+0000", 2024-09-13T11:22:37.109 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.0802000000000002e-05 2024-09-13T11:22:37.109 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.109 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.109 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:22:37.109 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:15.253458+0000", 2024-09-13T11:22:37.109 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3122000000000001e-05 2024-09-13T11:22:37.109 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.109 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.109 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:22:37.110 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:15.253638+0000", 2024-09-13T11:22:37.110 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000180137 2024-09-13T11:22:37.110 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:37.110 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:22:37.110 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:37.110 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.110 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.110 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4344.0:7 2.2 2:5cea7035:::rbd_info:head [write 0~19] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-13T11:22:37.110 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:22:16.258354+0000", 2024-09-13T11:22:37.110 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 20.83148139, 2024-09-13T11:22:37.110 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.012565668, 2024-09-13T11:22:37.110 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:22:37.110 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:22:37.111 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:37.111 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:22:37.111 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4344", 2024-09-13T11:22:37.111 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.1:0/211923147", 2024-09-13T11:22:37.111 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 7 2024-09-13T11:22:37.111 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.111 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:22:37.111 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.111 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:22:37.111 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:16.258354+0000", 2024-09-13T11:22:37.111 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:37.111 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.111 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.112 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:22:37.112 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:16.258354+0000", 2024-09-13T11:22:37.112 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:37.112 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.112 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.112 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:22:37.112 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:16.258358+0000", 2024-09-13T11:22:37.112 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.673e-06 2024-09-13T11:22:37.112 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.112 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.112 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:22:37.112 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:16.258385+0000", 2024-09-13T11:22:37.113 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.6758e-05 2024-09-13T11:22:37.113 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.113 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.113 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:22:37.113 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:16.258417+0000", 2024-09-13T11:22:37.113 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.2336999999999999e-05 2024-09-13T11:22:37.113 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.113 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.113 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:22:37.113 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:16.258436+0000", 2024-09-13T11:22:37.113 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.8491999999999998e-05 2024-09-13T11:22:37.113 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.113 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.113 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:22:37.114 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:16.258609+0000", 2024-09-13T11:22:37.114 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000173282 2024-09-13T11:22:37.114 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.114 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.114 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op must wait for map", 2024-09-13T11:22:37.114 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:16.258627+0000", 2024-09-13T11:22:37.114 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.8312e-05 2024-09-13T11:22:37.114 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.114 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.114 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:22:37.114 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:16.267370+0000", 2024-09-13T11:22:37.114 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0087421310000000002 2024-09-13T11:22:37.114 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.114 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.115 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:22:37.115 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:16.267563+0000", 2024-09-13T11:22:37.115 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00019360699999999999 2024-09-13T11:22:37.115 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.115 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.115 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 9", 2024-09-13T11:22:37.115 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:16.267929+0000", 2024-09-13T11:22:37.115 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00036540599999999998 2024-09-13T11:22:37.115 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.115 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.115 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:22:37.115 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:16.269855+0000", 2024-09-13T11:22:37.115 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001926019 2024-09-13T11:22:37.116 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.116 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.116 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:22:37.116 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:16.270767+0000", 2024-09-13T11:22:37.116 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00091232399999999995 2024-09-13T11:22:37.116 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.116 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.116 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:22:37.116 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:16.270849+0000", 2024-09-13T11:22:37.116 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.1790000000000001e-05 2024-09-13T11:22:37.116 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.116 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.116 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:22:37.116 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:16.270919+0000", 2024-09-13T11:22:37.117 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.0537000000000006e-05 2024-09-13T11:22:37.118 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:37.118 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:22:37.118 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:37.118 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.118 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.118 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop_reply(client.4344.0:7 2.2 e28/25 ondisk, result = 0)", 2024-09-13T11:22:37.118 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:22:16.270558+0000", 2024-09-13T11:22:37.118 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 20.819276471999999, 2024-09-13T11:22:37.118 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00037120600000000001, 2024-09-13T11:22:37.118 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:22:37.118 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:22:37.119 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:22:37.119 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:22:37.119 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.119 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:22:37.119 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:16.270558+0000", 2024-09-13T11:22:37.119 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:37.119 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.119 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.119 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:22:37.119 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:16.270558+0000", 2024-09-13T11:22:37.119 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:37.119 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.119 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.120 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:22:37.120 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:16.270566+0000", 2024-09-13T11:22:37.120 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.2130000000000002e-06 2024-09-13T11:22:37.120 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.120 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.120 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:22:37.120 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:16.270576+0000", 2024-09-13T11:22:37.120 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.8859999999999999e-06 2024-09-13T11:22:37.120 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.120 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.120 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:22:37.120 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:16.270645+0000", 2024-09-13T11:22:37.121 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.9237000000000001e-05 2024-09-13T11:22:37.121 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.121 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.121 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:22:37.121 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:16.270659+0000", 2024-09-13T11:22:37.121 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3719e-05 2024-09-13T11:22:37.121 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.121 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.121 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:22:37.121 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:16.270746+0000", 2024-09-13T11:22:37.121 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.7639000000000006e-05 2024-09-13T11:22:37.121 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.122 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.122 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:22:37.122 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:16.270759+0000", 2024-09-13T11:22:37.122 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.2523e-05 2024-09-13T11:22:37.122 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.122 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.122 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:22:37.122 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:16.270930+0000", 2024-09-13T11:22:37.122 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000170989 2024-09-13T11:22:37.122 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:37.122 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:22:37.122 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:37.122 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.123 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.123 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:35 3.d e32/30 3:b4cacdc6:::benchmark_data_smithi177_31116_object34:head v 32'1, mlcod=0'0)", 2024-09-13T11:22:37.123 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:22:27.022331+0000", 2024-09-13T11:22:37.123 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 10.067504190999999, 2024-09-13T11:22:37.123 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0012703059999999999, 2024-09-13T11:22:37.123 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:22:37.123 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:22:37.123 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:37.123 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:22:37.123 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.123 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:22:37.124 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.022331+0000", 2024-09-13T11:22:37.124 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:37.124 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.124 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.124 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:22:37.124 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.022331+0000", 2024-09-13T11:22:37.124 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:37.124 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.124 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.124 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:22:37.124 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.022331+0000", 2024-09-13T11:22:37.124 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.7899999999999998e-07 2024-09-13T11:22:37.124 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.125 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.125 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:22:37.125 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.022436+0000", 2024-09-13T11:22:37.125 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000104645 2024-09-13T11:22:37.125 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.125 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.125 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:22:37.125 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.022487+0000", 2024-09-13T11:22:37.125 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.1371e-05 2024-09-13T11:22:37.125 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.125 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.125 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:22:37.126 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.022497+0000", 2024-09-13T11:22:37.126 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.5910000000000002e-06 2024-09-13T11:22:37.126 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.126 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.126 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:22:37.126 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.022571+0000", 2024-09-13T11:22:37.126 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.4309999999999998e-05 2024-09-13T11:22:37.126 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.126 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.126 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:22:37.126 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.022590+0000", 2024-09-13T11:22:37.126 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.8252000000000001e-05 2024-09-13T11:22:37.127 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.127 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.127 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:22:37.127 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.023565+0000", 2024-09-13T11:22:37.127 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00097528399999999998 2024-09-13T11:22:37.127 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.127 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.127 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:22:37.127 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.023601+0000", 2024-09-13T11:22:37.127 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.6174000000000003e-05 2024-09-13T11:22:37.127 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:37.127 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:22:37.128 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:37.128 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.128 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.129 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:49 3.d e32/30 3:bbfcc9c5:::benchmark_data_smithi177_31116_object48:head v 32'2, mlcod=0'0)", 2024-09-13T11:22:37.130 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:22:27.023785+0000", 2024-09-13T11:22:37.130 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 10.066049498, 2024-09-13T11:22:37.130 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0019001840000000001, 2024-09-13T11:22:37.130 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:22:37.130 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:22:37.130 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:37.130 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:22:37.130 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.130 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:22:37.130 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.023785+0000", 2024-09-13T11:22:37.130 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:37.131 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.131 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.131 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:22:37.131 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.023785+0000", 2024-09-13T11:22:37.131 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:37.131 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.131 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.131 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:22:37.131 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.023787+0000", 2024-09-13T11:22:37.131 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.2869999999999999e-06 2024-09-13T11:22:37.131 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.131 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.132 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:22:37.132 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.023852+0000", 2024-09-13T11:22:37.132 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.5022000000000002e-05 2024-09-13T11:22:37.132 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.132 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.132 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:22:37.132 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.023888+0000", 2024-09-13T11:22:37.132 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.6219e-05 2024-09-13T11:22:37.132 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.132 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.132 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:22:37.132 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.023896+0000", 2024-09-13T11:22:37.132 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.2919999999999996e-06 2024-09-13T11:22:37.133 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.133 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.133 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:22:37.133 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.023958+0000", 2024-09-13T11:22:37.133 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.2026999999999994e-05 2024-09-13T11:22:37.133 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.133 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.133 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:22:37.133 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.023995+0000", 2024-09-13T11:22:37.133 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.7094000000000001e-05 2024-09-13T11:22:37.133 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.133 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.133 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:22:37.133 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.025642+0000", 2024-09-13T11:22:37.134 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001646296 2024-09-13T11:22:37.134 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.134 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.134 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:22:37.134 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.025686+0000", 2024-09-13T11:22:37.134 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.3946999999999998e-05 2024-09-13T11:22:37.134 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:37.134 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:22:37.134 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:37.134 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.134 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.134 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:41 3.a 3:528fa30e:::benchmark_data_smithi177_31116_object40:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-09-13T11:22:37.134 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:22:27.024470+0000", 2024-09-13T11:22:37.135 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 10.065364531, 2024-09-13T11:22:37.135 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.003018246, 2024-09-13T11:22:37.135 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:22:37.135 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:22:37.135 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:37.135 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:22:37.135 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:22:37.135 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:22:37.135 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 41 2024-09-13T11:22:37.135 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.135 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:22:37.135 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.135 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:22:37.135 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.024470+0000", 2024-09-13T11:22:37.136 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:37.136 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.136 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.136 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:22:37.136 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.024470+0000", 2024-09-13T11:22:37.136 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:37.136 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.136 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.136 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:22:37.136 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.024473+0000", 2024-09-13T11:22:37.136 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.9519999999999999e-06 2024-09-13T11:22:37.136 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.136 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.136 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:22:37.137 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.024579+0000", 2024-09-13T11:22:37.137 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000105423 2024-09-13T11:22:37.137 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.137 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.137 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:22:37.137 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.024594+0000", 2024-09-13T11:22:37.137 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5517999999999999e-05 2024-09-13T11:22:37.137 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.137 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.137 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:22:37.137 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.024602+0000", 2024-09-13T11:22:37.137 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.6860000000000006e-06 2024-09-13T11:22:37.137 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.138 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.138 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:22:37.138 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.024679+0000", 2024-09-13T11:22:37.138 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.6717000000000004e-05 2024-09-13T11:22:37.138 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.138 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.138 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:22:37.138 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.024844+0000", 2024-09-13T11:22:37.138 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00016530000000000001 2024-09-13T11:22:37.138 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.138 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.138 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 7", 2024-09-13T11:22:37.138 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.024942+0000", 2024-09-13T11:22:37.139 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.8357999999999996e-05 2024-09-13T11:22:37.139 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.139 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.139 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:22:37.139 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.027365+0000", 2024-09-13T11:22:37.139 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0024226209999999998 2024-09-13T11:22:37.139 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.139 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.139 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:22:37.139 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.027399+0000", 2024-09-13T11:22:37.139 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.4400999999999998e-05 2024-09-13T11:22:37.139 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.139 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.140 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:22:37.140 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.027450+0000", 2024-09-13T11:22:37.140 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.0351e-05 2024-09-13T11:22:37.140 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.140 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.140 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:22:37.140 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.027489+0000", 2024-09-13T11:22:37.140 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.8918999999999998e-05 2024-09-13T11:22:37.140 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:37.140 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:22:37.140 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:37.140 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.140 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.140 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:44 3.a 3:59e6ff42:::benchmark_data_smithi177_31116_object43:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-09-13T11:22:37.141 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:22:27.024621+0000", 2024-09-13T11:22:37.141 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 10.065213842, 2024-09-13T11:22:37.141 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.004045122, 2024-09-13T11:22:37.141 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:22:37.141 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:22:37.141 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:37.141 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:22:37.141 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:22:37.141 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:22:37.141 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 44 2024-09-13T11:22:37.141 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.141 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:22:37.141 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.142 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:22:37.142 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.024621+0000", 2024-09-13T11:22:37.142 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:37.142 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.142 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.142 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:22:37.142 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.024621+0000", 2024-09-13T11:22:37.142 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:37.142 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.142 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.142 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:22:37.142 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.024622+0000", 2024-09-13T11:22:37.143 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.9299999999999996e-07 2024-09-13T11:22:37.143 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.143 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.143 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:22:37.143 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.024675+0000", 2024-09-13T11:22:37.143 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.3134999999999999e-05 2024-09-13T11:22:37.143 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.143 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.143 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:22:37.143 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.024684+0000", 2024-09-13T11:22:37.143 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.2210000000000003e-06 2024-09-13T11:22:37.143 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.143 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.144 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:22:37.144 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.024691+0000", 2024-09-13T11:22:37.144 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.6320000000000002e-06 2024-09-13T11:22:37.144 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.144 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.144 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:22:37.144 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.025466+0000", 2024-09-13T11:22:37.144 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000774988 2024-09-13T11:22:37.144 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.144 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.144 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:22:37.144 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.025593+0000", 2024-09-13T11:22:37.144 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00012730099999999999 2024-09-13T11:22:37.145 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.145 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.145 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 7", 2024-09-13T11:22:37.145 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.025692+0000", 2024-09-13T11:22:37.145 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.8979000000000002e-05 2024-09-13T11:22:37.145 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.145 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.145 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:22:37.145 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.027771+0000", 2024-09-13T11:22:37.145 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0020787079999999999 2024-09-13T11:22:37.145 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.145 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.145 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:22:37.146 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.028540+0000", 2024-09-13T11:22:37.146 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00076938899999999999 2024-09-13T11:22:37.146 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.146 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.146 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:22:37.146 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.028625+0000", 2024-09-13T11:22:37.146 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.4339999999999995e-05 2024-09-13T11:22:37.146 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.146 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.146 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:22:37.146 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.028666+0000", 2024-09-13T11:22:37.146 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.1535999999999997e-05 2024-09-13T11:22:37.146 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:37.146 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:22:37.147 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:37.147 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.147 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.147 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:45 3.a 3:5af72f92:::benchmark_data_smithi177_31116_object44:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-09-13T11:22:37.147 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:22:27.024706+0000", 2024-09-13T11:22:37.147 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 10.065129233, 2024-09-13T11:22:37.147 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0041289009999999999, 2024-09-13T11:22:37.147 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:22:37.147 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:22:37.147 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:37.147 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:22:37.147 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:22:37.147 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:22:37.147 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 45 2024-09-13T11:22:37.148 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.148 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:22:37.148 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.148 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:22:37.148 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.024706+0000", 2024-09-13T11:22:37.148 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:37.148 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.148 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.148 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:22:37.148 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.024706+0000", 2024-09-13T11:22:37.148 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:37.148 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.148 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.149 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:22:37.149 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.024707+0000", 2024-09-13T11:22:37.149 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0529999999999999e-06 2024-09-13T11:22:37.149 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.149 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.149 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:22:37.149 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.024770+0000", 2024-09-13T11:22:37.149 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.3516999999999995e-05 2024-09-13T11:22:37.149 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.149 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.149 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:22:37.149 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.024780+0000", 2024-09-13T11:22:37.149 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.3680000000000008e-06 2024-09-13T11:22:37.150 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.150 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.150 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:22:37.150 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.024786+0000", 2024-09-13T11:22:37.150 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.5030000000000002e-06 2024-09-13T11:22:37.150 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.150 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.150 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:22:37.150 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.026005+0000", 2024-09-13T11:22:37.150 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0012192240000000001 2024-09-13T11:22:37.150 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.150 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.150 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:22:37.150 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.026102+0000", 2024-09-13T11:22:37.151 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.6142000000000002e-05 2024-09-13T11:22:37.151 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.151 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.151 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 7", 2024-09-13T11:22:37.151 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.026198+0000", 2024-09-13T11:22:37.151 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.6679000000000001e-05 2024-09-13T11:22:37.151 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.151 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.151 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:22:37.151 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.028686+0000", 2024-09-13T11:22:37.151 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.002487646 2024-09-13T11:22:37.151 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.151 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.151 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:22:37.152 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.028724+0000", 2024-09-13T11:22:37.152 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.7663999999999997e-05 2024-09-13T11:22:37.152 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.152 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.152 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:22:37.152 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.028778+0000", 2024-09-13T11:22:37.152 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.4731000000000002e-05 2024-09-13T11:22:37.152 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.152 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.152 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:22:37.152 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.028835+0000", 2024-09-13T11:22:37.152 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.6373999999999999e-05 2024-09-13T11:22:37.152 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:37.153 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:22:37.153 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:37.153 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.153 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.153 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:52 3.a 3:5f2a9152:::benchmark_data_smithi177_31116_object51:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-09-13T11:22:37.153 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:22:27.024802+0000", 2024-09-13T11:22:37.153 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 10.065033365, 2024-09-13T11:22:37.153 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.006439195, 2024-09-13T11:22:37.153 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:22:37.153 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:22:37.153 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:37.153 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:22:37.153 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:22:37.153 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:22:37.154 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 52 2024-09-13T11:22:37.154 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.154 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:22:37.154 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.154 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:22:37.154 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.024802+0000", 2024-09-13T11:22:37.154 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:37.154 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.154 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.154 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:22:37.154 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.024802+0000", 2024-09-13T11:22:37.154 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:37.154 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.154 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.155 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:22:37.155 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.024803+0000", 2024-09-13T11:22:37.155 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.187e-06 2024-09-13T11:22:37.155 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.155 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.155 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:22:37.155 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.024855+0000", 2024-09-13T11:22:37.155 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.2043000000000003e-05 2024-09-13T11:22:37.155 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.155 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.155 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:22:37.155 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.024863+0000", 2024-09-13T11:22:37.155 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.2269999999999993e-06 2024-09-13T11:22:37.156 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.156 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.156 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:22:37.156 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.024868+0000", 2024-09-13T11:22:37.156 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.1209999999999998e-06 2024-09-13T11:22:37.156 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.156 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.156 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:22:37.156 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.026652+0000", 2024-09-13T11:22:37.156 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0017837110000000001 2024-09-13T11:22:37.156 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.156 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.156 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:22:37.157 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.026792+0000", 2024-09-13T11:22:37.157 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00014028700000000001 2024-09-13T11:22:37.157 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.157 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.157 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 7", 2024-09-13T11:22:37.158 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.026946+0000", 2024-09-13T11:22:37.158 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000154067 2024-09-13T11:22:37.158 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.158 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.158 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:22:37.158 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.031062+0000", 2024-09-13T11:22:37.158 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0041160390000000002 2024-09-13T11:22:37.158 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.158 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.158 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:22:37.158 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.031173+0000", 2024-09-13T11:22:37.158 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000110919 2024-09-13T11:22:37.158 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.159 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.159 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:22:37.159 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.031211+0000", 2024-09-13T11:22:37.159 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.7443e-05 2024-09-13T11:22:37.159 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.159 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.159 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:22:37.159 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.031241+0000", 2024-09-13T11:22:37.159 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.0151e-05 2024-09-13T11:22:37.159 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:37.159 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:22:37.159 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:37.159 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.159 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.160 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop_reply(client.4665.0:41 3.a e32/30 ondisk, result = 0)", 2024-09-13T11:22:37.160 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:22:27.026645+0000", 2024-09-13T11:22:37.160 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 10.063189514999999, 2024-09-13T11:22:37.160 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00085708099999999999, 2024-09-13T11:22:37.160 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:22:37.160 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:22:37.160 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:22:37.160 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:22:37.160 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.160 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:22:37.160 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.026645+0000", 2024-09-13T11:22:37.160 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:37.161 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.161 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.161 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:22:37.161 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.026645+0000", 2024-09-13T11:22:37.161 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:37.161 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.161 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.161 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:22:37.161 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.026647+0000", 2024-09-13T11:22:37.161 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.4559999999999999e-06 2024-09-13T11:22:37.161 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.161 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.162 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:22:37.162 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.026651+0000", 2024-09-13T11:22:37.162 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.5639999999999998e-06 2024-09-13T11:22:37.162 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.162 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.162 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:22:37.162 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.026721+0000", 2024-09-13T11:22:37.162 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.9737e-05 2024-09-13T11:22:37.162 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.162 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.162 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:22:37.162 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.026730+0000", 2024-09-13T11:22:37.162 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.5790000000000004e-06 2024-09-13T11:22:37.163 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.163 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.163 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:22:37.163 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.027391+0000", 2024-09-13T11:22:37.163 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00066114600000000004 2024-09-13T11:22:37.163 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.163 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.163 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:22:37.163 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.027396+0000", 2024-09-13T11:22:37.163 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.062e-06 2024-09-13T11:22:37.163 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.164 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.164 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:22:37.164 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.027503+0000", 2024-09-13T11:22:37.164 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000106537 2024-09-13T11:22:37.164 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:37.164 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:22:37.164 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:37.164 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.164 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.164 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop_reply(client.4665.0:44 3.a e32/30 ondisk, result = 0)", 2024-09-13T11:22:37.164 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:22:27.027650+0000", 2024-09-13T11:22:37.164 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 10.062184503999999, 2024-09-13T11:22:37.164 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00012483500000000001, 2024-09-13T11:22:37.165 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:22:37.165 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:22:37.165 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:22:37.165 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:22:37.165 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.165 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:22:37.165 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.027650+0000", 2024-09-13T11:22:37.165 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:37.165 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.165 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.165 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:22:37.165 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.027650+0000", 2024-09-13T11:22:37.165 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:37.165 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.166 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.166 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:22:37.166 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.027653+0000", 2024-09-13T11:22:37.166 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.1430000000000001e-06 2024-09-13T11:22:37.166 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.166 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.166 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:22:37.166 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.027659+0000", 2024-09-13T11:22:37.166 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.2759999999999997e-06 2024-09-13T11:22:37.166 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.166 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.166 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:22:37.166 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.027699+0000", 2024-09-13T11:22:37.166 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.0586000000000001e-05 2024-09-13T11:22:37.167 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.167 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.167 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:22:37.167 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.027707+0000", 2024-09-13T11:22:37.167 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.1199999999999996e-06 2024-09-13T11:22:37.167 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.167 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.167 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:22:37.167 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.027761+0000", 2024-09-13T11:22:37.167 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.4302000000000003e-05 2024-09-13T11:22:37.167 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.167 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.167 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:22:37.168 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.027766+0000", 2024-09-13T11:22:37.168 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.9840000000000001e-06 2024-09-13T11:22:37.168 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.168 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.168 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:22:37.168 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.027775+0000", 2024-09-13T11:22:37.168 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.4239999999999999e-06 2024-09-13T11:22:37.168 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:37.168 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:22:37.168 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:37.168 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.168 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.169 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:56 3.a 3:5c46d3af:::benchmark_data_smithi177_31116_object55:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-09-13T11:22:37.169 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:22:27.027737+0000", 2024-09-13T11:22:37.169 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 10.062098408000001, 2024-09-13T11:22:37.169 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0043087179999999996, 2024-09-13T11:22:37.169 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:22:37.169 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:22:37.169 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:37.169 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:22:37.169 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:22:37.169 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:22:37.169 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 56 2024-09-13T11:22:37.169 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.170 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:22:37.170 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.170 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:22:37.170 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.027737+0000", 2024-09-13T11:22:37.170 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:37.170 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.170 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.170 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:22:37.170 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.027737+0000", 2024-09-13T11:22:37.170 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:37.170 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.170 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.170 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:22:37.171 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.027739+0000", 2024-09-13T11:22:37.171 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.1179999999999999e-06 2024-09-13T11:22:37.171 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.171 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.171 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:22:37.171 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.027803+0000", 2024-09-13T11:22:37.171 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.4523999999999994e-05 2024-09-13T11:22:37.171 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.171 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.171 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:22:37.171 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.027820+0000", 2024-09-13T11:22:37.171 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.6738000000000001e-05 2024-09-13T11:22:37.172 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.172 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.172 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:22:37.172 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.027833+0000", 2024-09-13T11:22:37.172 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3138e-05 2024-09-13T11:22:37.172 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.172 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.172 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:22:37.172 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.027896+0000", 2024-09-13T11:22:37.172 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.2581000000000003e-05 2024-09-13T11:22:37.172 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.172 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.172 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:22:37.173 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.027993+0000", 2024-09-13T11:22:37.173 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.7511999999999997e-05 2024-09-13T11:22:37.173 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.173 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.173 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 7", 2024-09-13T11:22:37.173 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.028139+0000", 2024-09-13T11:22:37.173 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00014552200000000001 2024-09-13T11:22:37.173 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.173 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.173 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:22:37.173 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.031699+0000", 2024-09-13T11:22:37.173 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0035602149999999998 2024-09-13T11:22:37.173 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.174 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.174 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:22:37.174 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.031946+0000", 2024-09-13T11:22:37.174 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00024740100000000001 2024-09-13T11:22:37.174 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.174 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.174 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:22:37.174 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.031986+0000", 2024-09-13T11:22:37.174 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.0083000000000002e-05 2024-09-13T11:22:37.174 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.174 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.174 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:22:37.174 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.032045+0000", 2024-09-13T11:22:37.174 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.8885999999999998e-05 2024-09-13T11:22:37.175 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:37.175 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:22:37.175 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:37.175 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.175 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.175 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop_reply(client.4665.0:45 3.a e32/30 ondisk, result = 0)", 2024-09-13T11:22:37.175 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:22:27.028481+0000", 2024-09-13T11:22:37.175 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 10.061353719, 2024-09-13T11:22:37.175 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00036331599999999998, 2024-09-13T11:22:37.175 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:22:37.175 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:22:37.175 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:22:37.175 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:22:37.176 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.176 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:22:37.176 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.028481+0000", 2024-09-13T11:22:37.176 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:37.176 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.176 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.176 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:22:37.176 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.028481+0000", 2024-09-13T11:22:37.176 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:37.176 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.177 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.177 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:22:37.177 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.028483+0000", 2024-09-13T11:22:37.177 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.7460000000000001e-06 2024-09-13T11:22:37.177 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.177 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.177 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:22:37.177 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.028489+0000", 2024-09-13T11:22:37.177 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.0480000000000004e-06 2024-09-13T11:22:37.177 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.177 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.177 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:22:37.178 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.028529+0000", 2024-09-13T11:22:37.178 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.0306999999999999e-05 2024-09-13T11:22:37.178 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.178 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.178 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:22:37.178 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.028537+0000", 2024-09-13T11:22:37.178 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.0080000000000002e-06 2024-09-13T11:22:37.178 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.178 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.178 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:22:37.178 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.028713+0000", 2024-09-13T11:22:37.178 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000176083 2024-09-13T11:22:37.178 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.178 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.179 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:22:37.179 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.028720+0000", 2024-09-13T11:22:37.179 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.1290000000000001e-06 2024-09-13T11:22:37.179 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.179 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.179 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:22:37.179 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.028845+0000", 2024-09-13T11:22:37.179 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00012499500000000001 2024-09-13T11:22:37.179 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:37.179 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:22:37.179 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:37.179 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.179 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.179 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop_reply(client.4665.0:52 3.a e32/30 ondisk, result = 0)", 2024-09-13T11:22:37.180 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:22:27.031058+0000", 2024-09-13T11:22:37.180 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 10.058777002999999, 2024-09-13T11:22:37.180 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00019034799999999999, 2024-09-13T11:22:37.180 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:22:37.180 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:22:37.180 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:22:37.180 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:22:37.180 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.180 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:22:37.180 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.031058+0000", 2024-09-13T11:22:37.180 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:37.180 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.180 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.181 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:22:37.181 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.031058+0000", 2024-09-13T11:22:37.181 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:37.181 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.181 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.181 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:22:37.181 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.031059+0000", 2024-09-13T11:22:37.181 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5430000000000001e-06 2024-09-13T11:22:37.181 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.181 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.181 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:22:37.181 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.031065+0000", 2024-09-13T11:22:37.181 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.3340000000000001e-06 2024-09-13T11:22:37.181 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.182 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.182 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:22:37.182 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.031108+0000", 2024-09-13T11:22:37.182 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.3166e-05 2024-09-13T11:22:37.182 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.182 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.182 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:22:37.182 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.031117+0000", 2024-09-13T11:22:37.182 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.2119999999999992e-06 2024-09-13T11:22:37.182 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.182 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.182 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:22:37.182 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.031165+0000", 2024-09-13T11:22:37.182 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.7395999999999998e-05 2024-09-13T11:22:37.183 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.183 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.183 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:22:37.183 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.031170+0000", 2024-09-13T11:22:37.183 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.3170000000000001e-06 2024-09-13T11:22:37.183 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.183 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.183 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:22:37.183 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.031248+0000", 2024-09-13T11:22:37.183 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.8380000000000005e-05 2024-09-13T11:22:37.183 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:37.183 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:22:37.183 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:37.183 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.184 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.184 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop_reply(client.4665.0:56 3.a e32/30 ondisk, result = 0)", 2024-09-13T11:22:37.184 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:22:27.031787+0000", 2024-09-13T11:22:37.184 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 10.058048376, 2024-09-13T11:22:37.184 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00026819300000000001, 2024-09-13T11:22:37.184 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:22:37.184 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:22:37.184 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:22:37.184 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:22:37.184 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.184 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:22:37.184 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.031787+0000", 2024-09-13T11:22:37.184 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:37.184 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.185 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.185 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:22:37.185 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.031787+0000", 2024-09-13T11:22:37.185 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:37.185 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.185 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.185 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:22:37.185 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.031788+0000", 2024-09-13T11:22:37.185 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.635e-06 2024-09-13T11:22:37.185 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.185 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.185 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:22:37.185 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.031794+0000", 2024-09-13T11:22:37.185 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.463e-06 2024-09-13T11:22:37.186 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.186 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.186 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:22:37.186 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.031861+0000", 2024-09-13T11:22:37.186 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.7150999999999995e-05 2024-09-13T11:22:37.186 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.186 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.186 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:22:37.186 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.031870+0000", 2024-09-13T11:22:37.186 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.1989999999999995e-06 2024-09-13T11:22:37.186 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.186 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.186 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:22:37.187 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.031937+0000", 2024-09-13T11:22:37.187 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.7360000000000006e-05 2024-09-13T11:22:37.187 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.187 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.187 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:22:37.187 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.031943+0000", 2024-09-13T11:22:37.187 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.524e-06 2024-09-13T11:22:37.187 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.187 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.187 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:22:37.187 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.032055+0000", 2024-09-13T11:22:37.187 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00011186100000000001 2024-09-13T11:22:37.187 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:37.187 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:22:37.188 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:37.188 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.188 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.188 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:72 3.a 3:532c98c2:::benchmark_data_smithi177_31116_object71:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-09-13T11:22:37.188 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:22:27.037426+0000", 2024-09-13T11:22:37.188 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 10.052408517, 2024-09-13T11:22:37.188 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0013739990000000001, 2024-09-13T11:22:37.188 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:22:37.188 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:22:37.188 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:37.188 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:22:37.188 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:22:37.188 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:22:37.188 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 72 2024-09-13T11:22:37.189 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.189 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:22:37.189 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.189 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:22:37.189 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.037426+0000", 2024-09-13T11:22:37.189 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:37.189 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.189 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.189 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:22:37.189 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.037426+0000", 2024-09-13T11:22:37.189 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:37.189 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.189 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.189 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:22:37.190 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.037429+0000", 2024-09-13T11:22:37.190 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.419e-06 2024-09-13T11:22:37.190 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.190 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.190 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:22:37.190 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.037461+0000", 2024-09-13T11:22:37.190 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.1776999999999997e-05 2024-09-13T11:22:37.190 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.190 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.190 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:22:37.190 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.037474+0000", 2024-09-13T11:22:37.190 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3789e-05 2024-09-13T11:22:37.191 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.191 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.191 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:22:37.191 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.037481+0000", 2024-09-13T11:22:37.191 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.8770000000000004e-06 2024-09-13T11:22:37.191 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.191 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.191 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:22:37.191 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.037533+0000", 2024-09-13T11:22:37.191 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.1807000000000001e-05 2024-09-13T11:22:37.191 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.191 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.191 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:22:37.191 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.037624+0000", 2024-09-13T11:22:37.192 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.0921000000000006e-05 2024-09-13T11:22:37.192 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.192 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.192 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 7", 2024-09-13T11:22:37.192 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.037716+0000", 2024-09-13T11:22:37.192 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.1774999999999995e-05 2024-09-13T11:22:37.192 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.192 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.192 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:22:37.192 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.038329+0000", 2024-09-13T11:22:37.192 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00061300600000000003 2024-09-13T11:22:37.192 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.192 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.192 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:22:37.193 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.038742+0000", 2024-09-13T11:22:37.193 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00041313800000000001 2024-09-13T11:22:37.193 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.193 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.193 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:22:37.193 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.038774+0000", 2024-09-13T11:22:37.193 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.1782999999999997e-05 2024-09-13T11:22:37.193 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.193 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.193 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:22:37.193 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.038800+0000", 2024-09-13T11:22:37.193 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.6707e-05 2024-09-13T11:22:37.193 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:37.193 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:22:37.194 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:37.194 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.194 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.194 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop_reply(client.4665.0:72 3.a e32/30 ondisk, result = 0)", 2024-09-13T11:22:37.194 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:22:27.038622+0000", 2024-09-13T11:22:37.194 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 10.051213173000001, 2024-09-13T11:22:37.194 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00018825600000000001, 2024-09-13T11:22:37.194 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:22:37.194 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:22:37.194 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:22:37.194 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:22:37.194 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.194 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:22:37.194 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.038622+0000", 2024-09-13T11:22:37.195 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:37.195 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.195 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.195 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:22:37.195 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.038622+0000", 2024-09-13T11:22:37.195 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:37.195 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.195 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.195 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:22:37.195 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.038623+0000", 2024-09-13T11:22:37.195 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.342e-06 2024-09-13T11:22:37.196 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.196 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.196 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:22:37.196 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.038627+0000", 2024-09-13T11:22:37.196 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.2390000000000004e-06 2024-09-13T11:22:37.196 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.196 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.196 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:22:37.196 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.038654+0000", 2024-09-13T11:22:37.196 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.6628e-05 2024-09-13T11:22:37.196 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.196 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.197 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:22:37.197 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.038674+0000", 2024-09-13T11:22:37.197 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.0418999999999999e-05 2024-09-13T11:22:37.197 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.197 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.197 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:22:37.197 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.038732+0000", 2024-09-13T11:22:37.197 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.7593000000000002e-05 2024-09-13T11:22:37.197 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.197 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.197 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:22:37.197 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.038739+0000", 2024-09-13T11:22:37.197 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.7800000000000003e-06 2024-09-13T11:22:37.197 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.198 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.198 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:22:37.198 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.038810+0000", 2024-09-13T11:22:37.198 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.1254999999999995e-05 2024-09-13T11:22:37.198 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:37.198 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:22:37.198 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:37.198 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:37.198 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:22:37.198 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-09-13T11:22:37.199 INFO:tasks.ceph.osd.8.smithi001.stderr:2024-09-13T11:22:37.084+0000 7fb9b48ae640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26312) UID: 0 2024-09-13T11:22:37.200 INFO:teuthology.orchestra.run.smithi001.stdout:osd.7: {} 2024-09-13T11:22:37.200 INFO:teuthology.orchestra.run.smithi001.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T11:22:37.201 INFO:teuthology.orchestra.run.smithi001.stdout:osd.8: {} 2024-09-13T11:22:37.201 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-13T11:22:37.201 INFO:teuthology.orchestra.run.smithi001.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-13T11:22:37.202 INFO:teuthology.orchestra.run.smithi001.stdout:osd.9: {} 2024-09-13T11:22:37.202 INFO:teuthology.orchestra.run.smithi001.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-13T11:22:37.202 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-13T11:22:37.192+0000 7f6a68775640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26307) UID: 0 2024-09-13T11:22:37.203 INFO:teuthology.orchestra.run.smithi001.stdout:osd.10: {} 2024-09-13T11:22:37.203 INFO:teuthology.orchestra.run.smithi001.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-13T11:22:37.210 INFO:teuthology.orchestra.run.smithi001.stderr:noscrub is set 2024-09-13T11:22:37.238 INFO:teuthology.orchestra.run.smithi001.stdout:osd.11: {} 2024-09-13T11:22:37.238 INFO:teuthology.orchestra.run.smithi001.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-13T11:22:37.266 INFO:teuthology.orchestra.run.smithi001.stdout:osd.12: {} 2024-09-13T11:22:37.266 INFO:teuthology.orchestra.run.smithi001.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-13T11:22:37.297 INFO:teuthology.orchestra.run.smithi001.stdout:osd.13: {} 2024-09-13T11:22:37.297 INFO:teuthology.orchestra.run.smithi001.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-13T11:22:37.301 INFO:tasks.ceph.osd.13.smithi092.stderr:2024-09-13T11:22:37.299+0000 7f3e5afea640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26350) UID: 0 2024-09-13T11:22:37.328 INFO:teuthology.orchestra.run.smithi001.stdout:osd.14: {} 2024-09-13T11:22:37.329 INFO:teuthology.orchestra.run.smithi001.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-13T11:22:37.360 INFO:teuthology.orchestra.run.smithi001.stdout:osd.15: {} 2024-09-13T11:22:37.360 INFO:teuthology.orchestra.run.smithi001.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-13T11:22:37.366 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-13T11:22:37.366 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-13T11:22:37.372 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-09-13T11:22:37.372 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [], 2024-09-13T11:22:37.372 INFO:teuthology.orchestra.run.smithi177.stdout: "num_ops": 0 2024-09-13T11:22:37.372 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-09-13T11:22:37.386 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-13T11:22:37.402 INFO:tasks.ceph.osd.14.smithi162.stderr:2024-09-13T11:22:37.400+0000 7f235c0db640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26319) UID: 0 2024-09-13T11:22:37.503 INFO:tasks.ceph.osd.11.smithi177.stderr:2024-09-13T11:22:37.500+0000 7f24792c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26580) UID: 0 2024-09-13T11:22:37.503 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-09-13T11:22:37.503 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [], 2024-09-13T11:22:37.503 INFO:teuthology.orchestra.run.smithi177.stdout: "complaint_time": 30, 2024-09-13T11:22:37.504 INFO:teuthology.orchestra.run.smithi177.stdout: "num_blocked_ops": 0 2024-09-13T11:22:37.504 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-09-13T11:22:37.518 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-13T11:22:37.581 INFO:tasks.ceph.osd.3.smithi177.stderr:2024-09-13T11:22:37.576+0000 7f10a25d1640 -1 osd.3 35 handle_osd_map injecting map crc failure 2024-09-13T11:22:37.605 INFO:tasks.ceph.osd.15.smithi177.stderr:2024-09-13T11:22:37.600+0000 7f9d6ce77640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26578) UID: 0 2024-09-13T11:22:37.693 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-09-13T11:22:37.694 INFO:teuthology.orchestra.run.smithi177.stdout: "size": 20, 2024-09-13T11:22:37.694 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 600, 2024-09-13T11:22:37.694 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [ 2024-09-13T11:22:37.694 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.694 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:5 3.3 e32/30 3:c820cfec:::benchmark_data_smithi177_31116_object4:head v 32'1, mlcod=0'0)", 2024-09-13T11:22:37.694 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:22:27.005437+0000", 2024-09-13T11:22:37.694 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 10.687153724, 2024-09-13T11:22:37.694 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00130159, 2024-09-13T11:22:37.694 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:22:37.694 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:22:37.694 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:37.695 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:22:37.695 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.695 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:22:37.695 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.005437+0000", 2024-09-13T11:22:37.695 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:37.695 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.695 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.695 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:22:37.695 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.005437+0000", 2024-09-13T11:22:37.695 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:37.695 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.696 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.696 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:22:37.696 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.005438+0000", 2024-09-13T11:22:37.696 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.7800000000000001e-07 2024-09-13T11:22:37.696 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.696 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.696 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:22:37.696 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.005684+0000", 2024-09-13T11:22:37.696 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00024551800000000002 2024-09-13T11:22:37.696 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.696 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.696 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:22:37.697 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.005709+0000", 2024-09-13T11:22:37.697 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.5588999999999998e-05 2024-09-13T11:22:37.697 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.697 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.697 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:22:37.697 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.005717+0000", 2024-09-13T11:22:37.697 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.7249999999999997e-06 2024-09-13T11:22:37.697 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.697 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.697 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:22:37.697 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.005775+0000", 2024-09-13T11:22:37.698 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.8461999999999997e-05 2024-09-13T11:22:37.698 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.698 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.698 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:22:37.698 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.005797+0000", 2024-09-13T11:22:37.698 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.2116e-05 2024-09-13T11:22:37.698 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.698 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.698 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:22:37.698 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.006699+0000", 2024-09-13T11:22:37.698 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00090146499999999995 2024-09-13T11:22:37.699 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.699 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.699 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:22:37.699 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.006739+0000", 2024-09-13T11:22:37.699 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.0037000000000003e-05 2024-09-13T11:22:37.699 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:37.699 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:22:37.699 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:37.699 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.699 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.699 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:18 3.3 e32/30 3:c843b42e:::benchmark_data_smithi177_31116_object17:head v 32'2, mlcod=0'0)", 2024-09-13T11:22:37.699 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:22:27.010629+0000", 2024-09-13T11:22:37.699 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 10.681962308999999, 2024-09-13T11:22:37.700 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001189292, 2024-09-13T11:22:37.700 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:22:37.700 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:22:37.700 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:37.700 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:22:37.700 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.700 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:22:37.700 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.010629+0000", 2024-09-13T11:22:37.700 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:37.700 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.700 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.701 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:22:37.701 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.010629+0000", 2024-09-13T11:22:37.701 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:37.701 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.701 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.701 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:22:37.701 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.010630+0000", 2024-09-13T11:22:37.701 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.46e-06 2024-09-13T11:22:37.701 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.701 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.701 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:22:37.702 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.010734+0000", 2024-09-13T11:22:37.702 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00010415400000000001 2024-09-13T11:22:37.702 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.702 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.702 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:22:37.702 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.010766+0000", 2024-09-13T11:22:37.702 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.1291000000000002e-05 2024-09-13T11:22:37.702 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.702 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.702 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:22:37.702 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.010772+0000", 2024-09-13T11:22:37.703 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.3640000000000004e-06 2024-09-13T11:22:37.703 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.703 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.703 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:22:37.703 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.010823+0000", 2024-09-13T11:22:37.703 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.0767999999999999e-05 2024-09-13T11:22:37.703 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.703 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.703 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:22:37.703 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.010841+0000", 2024-09-13T11:22:37.703 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.7714999999999999e-05 2024-09-13T11:22:37.704 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.704 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.704 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:22:37.704 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.011777+0000", 2024-09-13T11:22:37.704 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00093674899999999996 2024-09-13T11:22:37.704 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.704 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.704 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:22:37.704 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.011818+0000", 2024-09-13T11:22:37.704 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.0791000000000003e-05 2024-09-13T11:22:37.704 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:37.704 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:22:37.705 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:37.705 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.705 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.705 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:36 3.3 e32/30 3:cda5fb1c:::benchmark_data_smithi177_31116_object35:head v 32'3, mlcod=32'1)", 2024-09-13T11:22:37.705 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:22:27.018449+0000", 2024-09-13T11:22:37.705 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 10.674141622, 2024-09-13T11:22:37.705 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00095828600000000003, 2024-09-13T11:22:37.705 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:22:37.705 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:22:37.705 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:37.705 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:22:37.706 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.706 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:22:37.706 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.018449+0000", 2024-09-13T11:22:37.706 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:37.706 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.706 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.706 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:22:37.706 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.018449+0000", 2024-09-13T11:22:37.706 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:37.706 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.706 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.706 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:22:37.707 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.018451+0000", 2024-09-13T11:22:37.707 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.7409999999999999e-06 2024-09-13T11:22:37.707 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.707 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.707 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:22:37.707 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.018511+0000", 2024-09-13T11:22:37.707 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.9451999999999999e-05 2024-09-13T11:22:37.707 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.707 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.707 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:22:37.707 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.018570+0000", 2024-09-13T11:22:37.707 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.9036000000000002e-05 2024-09-13T11:22:37.707 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.708 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.708 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:22:37.708 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.018579+0000", 2024-09-13T11:22:37.708 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.037e-06 2024-09-13T11:22:37.708 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.708 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.708 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:22:37.708 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.018654+0000", 2024-09-13T11:22:37.708 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.5785999999999996e-05 2024-09-13T11:22:37.708 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.708 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.709 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:22:37.709 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.018671+0000", 2024-09-13T11:22:37.709 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.6099999999999998e-05 2024-09-13T11:22:37.709 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.709 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.709 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:22:37.709 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.019355+0000", 2024-09-13T11:22:37.709 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00068475999999999997 2024-09-13T11:22:37.709 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.709 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.709 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:22:37.709 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.019408+0000", 2024-09-13T11:22:37.710 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.2373999999999997e-05 2024-09-13T11:22:37.710 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:37.710 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:22:37.710 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:37.710 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.710 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.710 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:35 3.d 3:b4cacdc6:::benchmark_data_smithi177_31116_object34:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-09-13T11:22:37.710 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:22:27.021453+0000", 2024-09-13T11:22:37.710 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 10.671138261999999, 2024-09-13T11:22:37.710 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0029259860000000002, 2024-09-13T11:22:37.711 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:22:37.711 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:22:37.711 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:37.711 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:22:37.711 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:22:37.711 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:22:37.711 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 35 2024-09-13T11:22:37.711 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.711 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:22:37.711 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.711 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:22:37.711 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.021453+0000", 2024-09-13T11:22:37.712 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:37.712 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.712 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.712 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:22:37.712 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.021453+0000", 2024-09-13T11:22:37.712 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:37.712 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.712 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.712 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:22:37.712 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.021456+0000", 2024-09-13T11:22:37.712 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.033e-06 2024-09-13T11:22:37.712 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.713 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.713 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:22:37.713 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.021570+0000", 2024-09-13T11:22:37.713 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000114354 2024-09-13T11:22:37.713 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.713 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.713 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:22:37.713 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.021584+0000", 2024-09-13T11:22:37.713 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3332999999999999e-05 2024-09-13T11:22:37.713 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.713 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.714 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:22:37.714 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.021591+0000", 2024-09-13T11:22:37.714 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.4159999999999998e-06 2024-09-13T11:22:37.714 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.714 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.714 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:22:37.714 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.021661+0000", 2024-09-13T11:22:37.714 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.9636999999999998e-05 2024-09-13T11:22:37.714 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.714 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.714 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:22:37.714 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.021803+0000", 2024-09-13T11:22:37.715 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000142359 2024-09-13T11:22:37.715 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.715 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.715 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 3", 2024-09-13T11:22:37.715 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.021957+0000", 2024-09-13T11:22:37.715 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00015437 2024-09-13T11:22:37.715 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.715 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.715 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:22:37.715 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.023146+0000", 2024-09-13T11:22:37.715 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0011884759999999999 2024-09-13T11:22:37.715 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.716 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.716 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:22:37.716 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.024289+0000", 2024-09-13T11:22:37.716 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001143217 2024-09-13T11:22:37.716 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.716 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.716 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:22:37.716 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.024335+0000", 2024-09-13T11:22:37.716 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.5995999999999998e-05 2024-09-13T11:22:37.716 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.716 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.716 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:22:37.717 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.024379+0000", 2024-09-13T11:22:37.717 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.3795000000000003e-05 2024-09-13T11:22:37.717 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:37.717 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:22:37.717 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:37.717 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.717 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.717 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:49 3.d 3:bbfcc9c5:::benchmark_data_smithi177_31116_object48:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-09-13T11:22:37.717 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:22:27.023267+0000", 2024-09-13T11:22:37.717 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 10.669324461, 2024-09-13T11:22:37.717 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0027547439999999999, 2024-09-13T11:22:37.718 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:22:37.718 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:22:37.718 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:37.718 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:22:37.718 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:22:37.718 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:22:37.718 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 49 2024-09-13T11:22:37.718 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.718 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:22:37.718 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.718 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:22:37.718 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.023267+0000", 2024-09-13T11:22:37.719 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:37.719 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.719 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.719 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:22:37.719 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.023267+0000", 2024-09-13T11:22:37.719 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:37.719 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.719 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.719 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:22:37.719 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.023269+0000", 2024-09-13T11:22:37.720 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.108e-06 2024-09-13T11:22:37.720 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.720 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.720 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:22:37.720 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.023295+0000", 2024-09-13T11:22:37.720 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.6542000000000001e-05 2024-09-13T11:22:37.720 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.720 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.720 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:22:37.720 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.023309+0000", 2024-09-13T11:22:37.720 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3674e-05 2024-09-13T11:22:37.720 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.721 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.721 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:22:37.721 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.023316+0000", 2024-09-13T11:22:37.721 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.1330000000000001e-06 2024-09-13T11:22:37.721 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.721 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.721 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:22:37.721 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.023379+0000", 2024-09-13T11:22:37.721 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.3223000000000001e-05 2024-09-13T11:22:37.721 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.721 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.722 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:22:37.722 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.023525+0000", 2024-09-13T11:22:37.722 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000145783 2024-09-13T11:22:37.722 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.722 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.722 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 3", 2024-09-13T11:22:37.722 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.023696+0000", 2024-09-13T11:22:37.722 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000170767 2024-09-13T11:22:37.722 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.722 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.722 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:22:37.722 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.025435+0000", 2024-09-13T11:22:37.723 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0017387170000000001 2024-09-13T11:22:37.723 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.723 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.723 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:22:37.723 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.025918+0000", 2024-09-13T11:22:37.723 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00048347400000000001 2024-09-13T11:22:37.723 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.723 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.723 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:22:37.723 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.025973+0000", 2024-09-13T11:22:37.723 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.5117000000000001e-05 2024-09-13T11:22:37.724 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.724 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.724 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:22:37.724 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.026021+0000", 2024-09-13T11:22:37.724 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.8205999999999999e-05 2024-09-13T11:22:37.724 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:37.724 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:22:37.724 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:37.724 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.724 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.724 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop_reply(client.4665.0:35 3.d e31/30 ondisk, result = 0)", 2024-09-13T11:22:37.725 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:22:27.023652+0000", 2024-09-13T11:22:37.725 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 10.668938929999999, 2024-09-13T11:22:37.725 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000739209, 2024-09-13T11:22:37.725 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:22:37.725 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:22:37.725 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:22:37.725 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:22:37.725 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.725 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:22:37.725 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.023652+0000", 2024-09-13T11:22:37.726 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:37.726 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.726 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.726 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:22:37.726 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.023652+0000", 2024-09-13T11:22:37.726 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:37.726 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.726 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.726 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:22:37.726 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.023653+0000", 2024-09-13T11:22:37.726 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.341e-06 2024-09-13T11:22:37.727 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.727 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.727 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:22:37.727 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.023658+0000", 2024-09-13T11:22:37.727 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.3560000000000003e-06 2024-09-13T11:22:37.727 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.727 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.727 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:22:37.727 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.023690+0000", 2024-09-13T11:22:37.727 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.2416999999999998e-05 2024-09-13T11:22:37.727 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.728 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.728 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:22:37.728 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.023698+0000", 2024-09-13T11:22:37.728 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.1419999999999993e-06 2024-09-13T11:22:37.728 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.728 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.728 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:22:37.728 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.024277+0000", 2024-09-13T11:22:37.728 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00057814499999999996 2024-09-13T11:22:37.728 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.729 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.729 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:22:37.729 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.024284+0000", 2024-09-13T11:22:37.729 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.9200000000000004e-06 2024-09-13T11:22:37.729 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.729 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.729 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:22:37.729 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.024391+0000", 2024-09-13T11:22:37.729 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000106888 2024-09-13T11:22:37.729 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:37.729 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:22:37.730 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:37.730 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.730 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.730 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:41 3.a e32/30 3:528fa30e:::benchmark_data_smithi177_31116_object40:head v 32'1, mlcod=0'0)", 2024-09-13T11:22:37.730 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:22:27.025056+0000", 2024-09-13T11:22:37.730 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 10.667534708, 2024-09-13T11:22:37.730 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0015305119999999999, 2024-09-13T11:22:37.730 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:22:37.730 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:22:37.730 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:37.730 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:22:37.731 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.731 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:22:37.731 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.025056+0000", 2024-09-13T11:22:37.731 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:37.731 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.731 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.731 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:22:37.731 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.025056+0000", 2024-09-13T11:22:37.731 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:37.731 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.732 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.732 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:22:37.732 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.025058+0000", 2024-09-13T11:22:37.732 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5200000000000001e-06 2024-09-13T11:22:37.732 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.732 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.732 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:22:37.732 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.025116+0000", 2024-09-13T11:22:37.732 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.8576000000000003e-05 2024-09-13T11:22:37.732 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.732 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.733 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:22:37.733 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.025149+0000", 2024-09-13T11:22:37.733 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.2509999999999999e-05 2024-09-13T11:22:37.733 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.733 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.733 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:22:37.733 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.025156+0000", 2024-09-13T11:22:37.733 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.1250000000000004e-06 2024-09-13T11:22:37.733 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.733 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.733 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:22:37.734 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.025227+0000", 2024-09-13T11:22:37.734 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.0457000000000007e-05 2024-09-13T11:22:37.734 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.734 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.734 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:22:37.734 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.025246+0000", 2024-09-13T11:22:37.734 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.9134999999999999e-05 2024-09-13T11:22:37.734 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.734 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.734 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:22:37.735 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.026544+0000", 2024-09-13T11:22:37.735 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0012978779999999999 2024-09-13T11:22:37.735 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.735 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.735 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:22:37.735 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.026587+0000", 2024-09-13T11:22:37.735 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.3310999999999999e-05 2024-09-13T11:22:37.735 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:37.735 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:22:37.735 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:37.735 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.736 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.736 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop_reply(client.4665.0:49 3.d e32/30 ondisk, result = 0)", 2024-09-13T11:22:37.736 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:22:27.025736+0000", 2024-09-13T11:22:37.736 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 10.666855074000001, 2024-09-13T11:22:37.736 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00029714800000000001, 2024-09-13T11:22:37.736 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:22:37.736 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:22:37.736 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:22:37.736 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:22:37.736 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.736 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:22:37.736 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.025736+0000", 2024-09-13T11:22:37.737 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:37.737 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.737 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.737 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:22:37.737 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.025736+0000", 2024-09-13T11:22:37.737 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:37.737 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.737 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.737 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:22:37.737 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.025738+0000", 2024-09-13T11:22:37.738 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5829999999999999e-06 2024-09-13T11:22:37.738 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.738 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.738 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:22:37.738 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.025744+0000", 2024-09-13T11:22:37.738 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.4829999999999996e-06 2024-09-13T11:22:37.738 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.738 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.738 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:22:37.738 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.025846+0000", 2024-09-13T11:22:37.738 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000102332 2024-09-13T11:22:37.738 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.739 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.739 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:22:37.739 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.025855+0000", 2024-09-13T11:22:37.739 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.6160000000000002e-06 2024-09-13T11:22:37.739 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.739 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.739 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:22:37.739 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.025905+0000", 2024-09-13T11:22:37.739 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.9747999999999999e-05 2024-09-13T11:22:37.739 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.739 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.740 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:22:37.740 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.025913+0000", 2024-09-13T11:22:37.740 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.763e-06 2024-09-13T11:22:37.740 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.740 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.740 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:22:37.740 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.026033+0000", 2024-09-13T11:22:37.740 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000120623 2024-09-13T11:22:37.740 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:37.740 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:22:37.740 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:37.740 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.741 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.741 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:44 3.a e32/30 3:59e6ff42:::benchmark_data_smithi177_31116_object43:head v 32'2, mlcod=0'0)", 2024-09-13T11:22:37.741 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:22:27.025757+0000", 2024-09-13T11:22:37.741 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 10.666834481, 2024-09-13T11:22:37.741 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0018310869999999999, 2024-09-13T11:22:37.741 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:22:37.741 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:22:37.741 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:37.741 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:22:37.741 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.741 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:22:37.742 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.025757+0000", 2024-09-13T11:22:37.742 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:37.742 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.742 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.742 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:22:37.742 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.025757+0000", 2024-09-13T11:22:37.742 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:37.742 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.742 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.742 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:22:37.742 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.025757+0000", 2024-09-13T11:22:37.742 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.5600000000000005e-07 2024-09-13T11:22:37.743 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.743 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.743 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:22:37.743 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.025811+0000", 2024-09-13T11:22:37.743 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.3687999999999998e-05 2024-09-13T11:22:37.743 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.743 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.743 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:22:37.743 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.025874+0000", 2024-09-13T11:22:37.743 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.2884000000000003e-05 2024-09-13T11:22:37.743 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.744 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.744 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:22:37.744 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.025879+0000", 2024-09-13T11:22:37.744 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.9400000000000001e-06 2024-09-13T11:22:37.744 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.744 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.744 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:22:37.744 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.025929+0000", 2024-09-13T11:22:37.744 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.0383000000000002e-05 2024-09-13T11:22:37.744 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.744 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.744 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:22:37.745 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.025944+0000", 2024-09-13T11:22:37.745 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.4873e-05 2024-09-13T11:22:37.745 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.745 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.745 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:22:37.745 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.027546+0000", 2024-09-13T11:22:37.745 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001601875 2024-09-13T11:22:37.745 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.745 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.745 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:22:37.745 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.027588+0000", 2024-09-13T11:22:37.746 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.1788e-05 2024-09-13T11:22:37.746 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:37.746 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:22:37.746 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:37.746 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.746 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.746 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:45 3.a e32/30 3:5af72f92:::benchmark_data_smithi177_31116_object44:head v 32'3, mlcod=0'0)", 2024-09-13T11:22:37.746 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:22:27.026689+0000", 2024-09-13T11:22:37.746 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 10.665902068999999, 2024-09-13T11:22:37.746 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001713352, 2024-09-13T11:22:37.746 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:22:37.746 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:22:37.747 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:37.747 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:22:37.747 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.747 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:22:37.747 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.026689+0000", 2024-09-13T11:22:37.747 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:37.747 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.747 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.747 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:22:37.747 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.026689+0000", 2024-09-13T11:22:37.747 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:37.747 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.748 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.748 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:22:37.748 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.026690+0000", 2024-09-13T11:22:37.748 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.327e-06 2024-09-13T11:22:37.748 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.748 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.748 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:22:37.748 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.026747+0000", 2024-09-13T11:22:37.748 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.6700000000000003e-05 2024-09-13T11:22:37.748 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.749 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.749 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:22:37.749 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.026783+0000", 2024-09-13T11:22:37.749 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.5883999999999997e-05 2024-09-13T11:22:37.749 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.749 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.749 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:22:37.749 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.026789+0000", 2024-09-13T11:22:37.749 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.6380000000000001e-06 2024-09-13T11:22:37.749 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.749 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.749 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:22:37.750 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.026847+0000", 2024-09-13T11:22:37.750 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.8090000000000001e-05 2024-09-13T11:22:37.750 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.750 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.750 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:22:37.750 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.026868+0000", 2024-09-13T11:22:37.750 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.0973999999999999e-05 2024-09-13T11:22:37.750 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.750 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.750 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:22:37.750 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.028358+0000", 2024-09-13T11:22:37.751 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001490328 2024-09-13T11:22:37.751 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.751 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.751 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:22:37.751 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.028402+0000", 2024-09-13T11:22:37.751 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.4410999999999999e-05 2024-09-13T11:22:37.751 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:37.751 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:22:37.751 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:37.751 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.751 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.752 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:52 3.a e32/30 3:5f2a9152:::benchmark_data_smithi177_31116_object51:head v 32'4, mlcod=0'0)", 2024-09-13T11:22:37.752 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:22:27.027044+0000", 2024-09-13T11:22:37.752 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 10.665546752999999, 2024-09-13T11:22:37.752 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0039532619999999999, 2024-09-13T11:22:37.752 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:22:37.752 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:22:37.752 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:37.752 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:22:37.752 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.752 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:22:37.752 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.027044+0000", 2024-09-13T11:22:37.753 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:37.753 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.753 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.753 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:22:37.753 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.027044+0000", 2024-09-13T11:22:37.753 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:37.753 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.753 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.753 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:22:37.753 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.027046+0000", 2024-09-13T11:22:37.753 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.835e-06 2024-09-13T11:22:37.754 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.754 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.754 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:22:37.754 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.027121+0000", 2024-09-13T11:22:37.754 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.5025999999999996e-05 2024-09-13T11:22:37.754 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.754 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.754 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:22:37.754 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.027172+0000", 2024-09-13T11:22:37.754 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.0673000000000001e-05 2024-09-13T11:22:37.754 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.755 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.755 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:22:37.755 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.027181+0000", 2024-09-13T11:22:37.755 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.9879999999999993e-06 2024-09-13T11:22:37.755 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.755 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.755 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:22:37.755 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.027247+0000", 2024-09-13T11:22:37.755 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.5720000000000001e-05 2024-09-13T11:22:37.755 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.755 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.756 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:22:37.756 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.027274+0000", 2024-09-13T11:22:37.756 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.7022e-05 2024-09-13T11:22:37.756 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.756 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.756 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:22:37.756 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.030956+0000", 2024-09-13T11:22:37.756 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0036824430000000001 2024-09-13T11:22:37.756 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.756 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.756 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:22:37.756 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.030998+0000", 2024-09-13T11:22:37.757 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.1554999999999998e-05 2024-09-13T11:22:37.757 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:37.757 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:22:37.757 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:37.757 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.757 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.757 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:56 3.a e32/30 3:5c46d3af:::benchmark_data_smithi177_31116_object55:head v 32'5, mlcod=0'0)", 2024-09-13T11:22:37.757 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:22:27.028284+0000", 2024-09-13T11:22:37.757 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 10.664307542, 2024-09-13T11:22:37.757 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0034464470000000001, 2024-09-13T11:22:37.757 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:22:37.757 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:22:37.758 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:37.758 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:22:37.758 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.758 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:22:37.758 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.028284+0000", 2024-09-13T11:22:37.758 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:37.758 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.758 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.758 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:22:37.758 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.028284+0000", 2024-09-13T11:22:37.758 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:37.759 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.759 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.759 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:22:37.759 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.028285+0000", 2024-09-13T11:22:37.759 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.7179999999999999e-06 2024-09-13T11:22:37.759 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.759 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.759 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:22:37.759 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.028312+0000", 2024-09-13T11:22:37.759 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.6529999999999998e-05 2024-09-13T11:22:37.759 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.759 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.760 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:22:37.760 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.028367+0000", 2024-09-13T11:22:37.760 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.4926000000000002e-05 2024-09-13T11:22:37.760 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.760 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.760 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:22:37.760 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.028374+0000", 2024-09-13T11:22:37.760 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.7230000000000004e-06 2024-09-13T11:22:37.760 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.761 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.761 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:22:37.761 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.028438+0000", 2024-09-13T11:22:37.761 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.3603999999999996e-05 2024-09-13T11:22:37.761 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.761 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.761 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:22:37.761 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.028458+0000", 2024-09-13T11:22:37.761 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.0336e-05 2024-09-13T11:22:37.761 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.762 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.762 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:22:37.762 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.031684+0000", 2024-09-13T11:22:37.762 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0032259430000000002 2024-09-13T11:22:37.762 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.762 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.762 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:22:37.762 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.031730+0000", 2024-09-13T11:22:37.762 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.5667000000000002e-05 2024-09-13T11:22:37.762 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:37.763 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:22:37.763 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:37.763 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.763 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.763 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:63 3.3 e32/30 3:c10de739:::benchmark_data_smithi177_31116_object62:head v 32'4, mlcod=32'2)", 2024-09-13T11:22:37.763 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:22:27.033059+0000", 2024-09-13T11:22:37.763 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 10.659531899999999, 2024-09-13T11:22:37.763 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0010701790000000001, 2024-09-13T11:22:37.763 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:22:37.763 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:22:37.764 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:37.764 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:22:37.764 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.764 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:22:37.764 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.033059+0000", 2024-09-13T11:22:37.764 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:37.764 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.764 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.764 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:22:37.764 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.033059+0000", 2024-09-13T11:22:37.765 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:37.765 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.765 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.765 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:22:37.765 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.033061+0000", 2024-09-13T11:22:37.765 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.7689999999999999e-06 2024-09-13T11:22:37.765 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.765 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.765 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:22:37.765 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.033137+0000", 2024-09-13T11:22:37.766 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.5828999999999995e-05 2024-09-13T11:22:37.766 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.766 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.766 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:22:37.766 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.033175+0000", 2024-09-13T11:22:37.766 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.8084999999999999e-05 2024-09-13T11:22:37.766 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.766 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.766 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:22:37.767 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.033184+0000", 2024-09-13T11:22:37.767 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.8300000000000002e-06 2024-09-13T11:22:37.767 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.767 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.767 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:22:37.767 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.033296+0000", 2024-09-13T11:22:37.767 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00011250400000000001 2024-09-13T11:22:37.767 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.767 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.767 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:22:37.767 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.033323+0000", 2024-09-13T11:22:37.767 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.6999999999999999e-05 2024-09-13T11:22:37.768 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.768 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.768 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:22:37.768 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.034088+0000", 2024-09-13T11:22:37.768 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00076530399999999996 2024-09-13T11:22:37.768 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.768 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.768 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:22:37.768 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.034129+0000", 2024-09-13T11:22:37.769 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.0858000000000001e-05 2024-09-13T11:22:37.769 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:37.769 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:22:37.769 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:37.769 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.769 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.769 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:72 3.a e32/30 3:532c98c2:::benchmark_data_smithi177_31116_object71:head v 32'6, mlcod=32'4)", 2024-09-13T11:22:37.769 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:22:27.037809+0000", 2024-09-13T11:22:37.769 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 10.654782515000001, 2024-09-13T11:22:37.769 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000770154, 2024-09-13T11:22:37.770 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:22:37.770 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:22:37.770 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:37.770 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:22:37.770 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.770 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:22:37.770 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.037809+0000", 2024-09-13T11:22:37.771 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:37.771 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.771 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.771 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:22:37.771 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.037809+0000", 2024-09-13T11:22:37.771 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:37.771 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.771 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.771 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:22:37.771 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.037810+0000", 2024-09-13T11:22:37.772 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.8980000000000001e-06 2024-09-13T11:22:37.772 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.772 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.772 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:22:37.772 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.037848+0000", 2024-09-13T11:22:37.772 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.7183e-05 2024-09-13T11:22:37.772 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.772 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.772 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:22:37.773 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.037901+0000", 2024-09-13T11:22:37.773 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.3269000000000001e-05 2024-09-13T11:22:37.773 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.773 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.773 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:22:37.773 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.037910+0000", 2024-09-13T11:22:37.773 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.2469999999999997e-06 2024-09-13T11:22:37.773 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.773 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.773 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:22:37.774 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.037966+0000", 2024-09-13T11:22:37.774 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.6180000000000001e-05 2024-09-13T11:22:37.774 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.774 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.774 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:22:37.774 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.037980+0000", 2024-09-13T11:22:37.774 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3915e-05 2024-09-13T11:22:37.774 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.774 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.774 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:22:37.775 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.038531+0000", 2024-09-13T11:22:37.775 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00055067600000000005 2024-09-13T11:22:37.775 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.775 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.775 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:22:37.775 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.038579+0000", 2024-09-13T11:22:37.775 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.7786e-05 2024-09-13T11:22:37.775 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:37.775 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:22:37.776 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:37.776 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.776 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.776 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:77 3.3 e32/30 3:c9b6c4f6:::benchmark_data_smithi177_31116_object76:head v 32'5, mlcod=32'3)", 2024-09-13T11:22:37.776 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:22:27.049554+0000", 2024-09-13T11:22:37.776 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 10.643037537, 2024-09-13T11:22:37.776 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00078703900000000001, 2024-09-13T11:22:37.776 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:22:37.776 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:22:37.776 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:37.777 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:22:37.777 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.777 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:22:37.777 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.049554+0000", 2024-09-13T11:22:37.777 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:37.777 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.777 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.777 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:22:37.777 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.049554+0000", 2024-09-13T11:22:37.777 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:37.778 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.778 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.778 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:22:37.778 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.049555+0000", 2024-09-13T11:22:37.778 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.6530000000000001e-06 2024-09-13T11:22:37.778 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.778 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.778 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:22:37.778 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.049593+0000", 2024-09-13T11:22:37.778 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.7814000000000001e-05 2024-09-13T11:22:37.778 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.779 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.779 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:22:37.779 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.049632+0000", 2024-09-13T11:22:37.779 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.8658000000000002e-05 2024-09-13T11:22:37.779 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.779 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.779 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:22:37.779 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.049638+0000", 2024-09-13T11:22:37.779 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.4829999999999996e-06 2024-09-13T11:22:37.779 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.780 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.780 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:22:37.780 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.049693+0000", 2024-09-13T11:22:37.780 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.4741999999999999e-05 2024-09-13T11:22:37.780 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.780 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.780 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:22:37.780 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.049706+0000", 2024-09-13T11:22:37.780 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3623e-05 2024-09-13T11:22:37.780 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.781 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.781 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:22:37.781 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.050301+0000", 2024-09-13T11:22:37.781 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00059447099999999995 2024-09-13T11:22:37.781 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.781 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.781 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:22:37.781 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.050341+0000", 2024-09-13T11:22:37.781 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.9595000000000003e-05 2024-09-13T11:22:37.782 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:37.782 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:22:37.782 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:37.782 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.782 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.782 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:78 3.3 e32/30 3:c95c4ad0:::benchmark_data_smithi177_31116_object77:head v 32'6, mlcod=32'4)", 2024-09-13T11:22:37.782 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:22:27.051984+0000", 2024-09-13T11:22:37.782 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 10.640606942, 2024-09-13T11:22:37.782 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00086381100000000003, 2024-09-13T11:22:37.783 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:22:37.783 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:22:37.783 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:37.783 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:22:37.783 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.783 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:22:37.783 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.051984+0000", 2024-09-13T11:22:37.783 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:37.783 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.783 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.784 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:22:37.784 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.051984+0000", 2024-09-13T11:22:37.784 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:37.784 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.784 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.784 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:22:37.784 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.051986+0000", 2024-09-13T11:22:37.784 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.463e-06 2024-09-13T11:22:37.784 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.784 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.784 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:22:37.785 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.052032+0000", 2024-09-13T11:22:37.785 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.6675999999999998e-05 2024-09-13T11:22:37.785 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.785 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.785 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:22:37.785 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.052063+0000", 2024-09-13T11:22:37.785 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.0641e-05 2024-09-13T11:22:37.785 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.785 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.786 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:22:37.786 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.052070+0000", 2024-09-13T11:22:37.786 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.0330000000000003e-06 2024-09-13T11:22:37.786 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.786 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.786 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:22:37.786 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.052126+0000", 2024-09-13T11:22:37.786 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.6115000000000001e-05 2024-09-13T11:22:37.786 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.786 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.786 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:22:37.787 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.052141+0000", 2024-09-13T11:22:37.787 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.4567e-05 2024-09-13T11:22:37.787 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.787 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.787 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:22:37.787 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.052807+0000", 2024-09-13T11:22:37.787 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00066589100000000005 2024-09-13T11:22:37.787 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.787 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.787 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:22:37.788 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.052848+0000", 2024-09-13T11:22:37.788 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.1424999999999997e-05 2024-09-13T11:22:37.788 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:37.788 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:22:37.788 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:37.788 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.788 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.788 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:80 3.3 e32/30 3:c3d99973:::benchmark_data_smithi177_31116_object79:head v 32'7, mlcod=32'5)", 2024-09-13T11:22:37.788 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:22:27.058673+0000", 2024-09-13T11:22:37.788 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 10.633918020999999, 2024-09-13T11:22:37.789 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00090822699999999995, 2024-09-13T11:22:37.789 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:22:37.789 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:22:37.789 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:37.789 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:22:37.789 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.789 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:22:37.789 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.058673+0000", 2024-09-13T11:22:37.789 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:37.789 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.790 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.790 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:22:37.790 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.058673+0000", 2024-09-13T11:22:37.790 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:37.790 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.790 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.790 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:22:37.790 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.058675+0000", 2024-09-13T11:22:37.790 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5880000000000001e-06 2024-09-13T11:22:37.790 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.790 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.791 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:22:37.791 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.058730+0000", 2024-09-13T11:22:37.791 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.4963000000000002e-05 2024-09-13T11:22:37.791 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.791 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.791 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:22:37.791 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.058765+0000", 2024-09-13T11:22:37.791 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.5438999999999997e-05 2024-09-13T11:22:37.791 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.791 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.792 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:22:37.792 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.058778+0000", 2024-09-13T11:22:37.792 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.2577e-05 2024-09-13T11:22:37.792 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.792 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.792 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:22:37.792 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.058822+0000", 2024-09-13T11:22:37.792 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.4527999999999998e-05 2024-09-13T11:22:37.792 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.792 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.793 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:22:37.793 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.058837+0000", 2024-09-13T11:22:37.793 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.4476e-05 2024-09-13T11:22:37.793 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.793 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.793 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:22:37.793 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.059533+0000", 2024-09-13T11:22:37.793 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00069611599999999999 2024-09-13T11:22:37.793 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:37.793 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:37.794 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:22:37.794 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.059581+0000", 2024-09-13T11:22:37.794 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.8539999999999999e-05 2024-09-13T11:22:37.794 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:37.794 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:22:37.794 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:37.794 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:37.794 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:22:37.794 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-09-13T11:22:37.796 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-09-13T11:22:37.797 INFO:tasks.ceph.osd.14.smithi162.stderr:2024-09-13T11:22:37.792+0000 7f235c0db640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26319) UID: 0 2024-09-13T11:22:37.896 INFO:tasks.ceph.osd.12.smithi001.stderr:2024-09-13T11:22:37.892+0000 7f892c6e6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26314) UID: 0 2024-09-13T11:22:37.966 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-09-13T11:22:37.966 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [], 2024-09-13T11:22:37.966 INFO:teuthology.orchestra.run.smithi177.stdout: "num_ops": 0 2024-09-13T11:22:37.966 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-09-13T11:22:37.980 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-09-13T11:22:37.997 INFO:tasks.radosbench.radosbench.0.smithi177.stdout: 11 16 87 71 0.403356 0 - 0.0234801 2024-09-13T11:22:37.997 INFO:tasks.ceph.osd.15.smithi177.stderr:2024-09-13T11:22:37.992+0000 7f9d6ce77640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26578) UID: 0 2024-09-13T11:22:38.097 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-09-13T11:22:38.098 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [], 2024-09-13T11:22:38.098 INFO:teuthology.orchestra.run.smithi177.stdout: "complaint_time": 30, 2024-09-13T11:22:38.098 INFO:teuthology.orchestra.run.smithi177.stdout: "num_blocked_ops": 0 2024-09-13T11:22:38.098 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-09-13T11:22:38.098 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-13T11:22:38.092+0000 7f6a68775640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26307) UID: 0 2024-09-13T11:22:38.111 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-09-13T11:22:38.199 INFO:tasks.ceph.osd.13.smithi092.stderr:2024-09-13T11:22:38.199+0000 7f3e5afea640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26350) UID: 0 2024-09-13T11:22:38.280 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-09-13T11:22:38.281 INFO:teuthology.orchestra.run.smithi177.stdout: "size": 20, 2024-09-13T11:22:38.281 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 600, 2024-09-13T11:22:38.281 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [ 2024-09-13T11:22:38.281 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.281 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:11 3.f 3:f7f390ac:::benchmark_data_smithi177_31116_object10:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-09-13T11:22:38.281 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:22:27.002444+0000", 2024-09-13T11:22:38.281 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 11.27618268, 2024-09-13T11:22:38.281 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0074315980000000002, 2024-09-13T11:22:38.281 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:22:38.282 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:22:38.282 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:38.282 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:22:38.282 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:22:38.282 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:22:38.282 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 11 2024-09-13T11:22:38.282 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.282 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:22:38.282 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.282 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:22:38.283 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.002444+0000", 2024-09-13T11:22:38.283 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:38.283 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.283 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.283 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:22:38.283 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.002444+0000", 2024-09-13T11:22:38.283 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:38.283 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.283 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.283 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:22:38.284 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.002447+0000", 2024-09-13T11:22:38.284 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.7539999999999998e-06 2024-09-13T11:22:38.284 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.284 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.284 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:22:38.284 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.002508+0000", 2024-09-13T11:22:38.284 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.1556000000000005e-05 2024-09-13T11:22:38.284 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.285 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.285 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:22:38.285 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.002522+0000", 2024-09-13T11:22:38.285 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.4105e-05 2024-09-13T11:22:38.285 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.285 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.285 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:22:38.285 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.002530+0000", 2024-09-13T11:22:38.285 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.5059999999999998e-06 2024-09-13T11:22:38.286 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.286 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.286 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:22:38.286 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.002603+0000", 2024-09-13T11:22:38.286 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.3176999999999994e-05 2024-09-13T11:22:38.286 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.286 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.286 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op must wait for map", 2024-09-13T11:22:38.286 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.002608+0000", 2024-09-13T11:22:38.286 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.3870000000000003e-06 2024-09-13T11:22:38.287 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.287 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.287 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:22:38.287 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.004658+0000", 2024-09-13T11:22:38.287 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0020497129999999999 2024-09-13T11:22:38.287 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.287 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.287 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:22:38.287 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.004758+0000", 2024-09-13T11:22:38.287 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.9982999999999995e-05 2024-09-13T11:22:38.287 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.288 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.288 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 12", 2024-09-13T11:22:38.288 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.004851+0000", 2024-09-13T11:22:38.288 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.3306000000000004e-05 2024-09-13T11:22:38.288 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.288 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.288 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:22:38.288 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.009040+0000", 2024-09-13T11:22:38.288 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0041889689999999999 2024-09-13T11:22:38.289 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.289 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.289 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:22:38.289 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.009796+0000", 2024-09-13T11:22:38.289 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00075592900000000002 2024-09-13T11:22:38.289 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.289 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.289 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:22:38.290 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.009835+0000", 2024-09-13T11:22:38.290 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.8985e-05 2024-09-13T11:22:38.290 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.290 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.290 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:22:38.290 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.009875+0000", 2024-09-13T11:22:38.290 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.0228000000000002e-05 2024-09-13T11:22:38.290 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:38.290 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:22:38.290 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:38.291 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.291 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.291 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:23 3.c 3:31691fed:::benchmark_data_smithi177_31116_object22:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-09-13T11:22:38.291 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:22:27.010745+0000", 2024-09-13T11:22:38.291 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 11.267881374, 2024-09-13T11:22:38.291 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0036433070000000001, 2024-09-13T11:22:38.291 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:22:38.291 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:22:38.291 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:38.291 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:22:38.292 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:22:38.292 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:22:38.292 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 23 2024-09-13T11:22:38.292 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.292 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:22:38.292 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.292 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:22:38.292 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.010745+0000", 2024-09-13T11:22:38.292 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:38.292 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.293 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.293 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:22:38.293 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.010745+0000", 2024-09-13T11:22:38.293 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:38.293 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.293 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.293 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:22:38.293 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.010747+0000", 2024-09-13T11:22:38.293 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.8989999999999999e-06 2024-09-13T11:22:38.293 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.294 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.294 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:22:38.294 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.010775+0000", 2024-09-13T11:22:38.294 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.8421000000000001e-05 2024-09-13T11:22:38.294 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.294 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.294 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:22:38.294 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.010791+0000", 2024-09-13T11:22:38.294 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5942000000000001e-05 2024-09-13T11:22:38.295 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.295 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.295 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:22:38.295 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.010801+0000", 2024-09-13T11:22:38.295 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.3819999999999993e-06 2024-09-13T11:22:38.295 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.295 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.295 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:22:38.295 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.010872+0000", 2024-09-13T11:22:38.295 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.1138999999999998e-05 2024-09-13T11:22:38.295 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.296 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.296 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:22:38.296 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.011045+0000", 2024-09-13T11:22:38.296 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00017294700000000001 2024-09-13T11:22:38.296 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.296 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.296 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 13", 2024-09-13T11:22:38.296 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.011183+0000", 2024-09-13T11:22:38.296 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00013787200000000001 2024-09-13T11:22:38.296 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.297 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.297 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:22:38.297 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.012300+0000", 2024-09-13T11:22:38.297 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0011170100000000001 2024-09-13T11:22:38.297 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.297 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.297 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:22:38.297 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.014327+0000", 2024-09-13T11:22:38.297 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0020277450000000001 2024-09-13T11:22:38.297 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.298 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.298 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:22:38.298 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.014362+0000", 2024-09-13T11:22:38.298 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.4563000000000001e-05 2024-09-13T11:22:38.298 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.298 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.298 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:22:38.298 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.014388+0000", 2024-09-13T11:22:38.298 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.6387e-05 2024-09-13T11:22:38.298 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:38.298 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:22:38.299 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:38.299 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.299 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.299 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:19 3.1 e32/30 3:8c29d864:::benchmark_data_smithi177_31116_object18:head v 32'1, mlcod=0'0)", 2024-09-13T11:22:38.299 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:22:27.010971+0000", 2024-09-13T11:22:38.299 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 11.267655897999999, 2024-09-13T11:22:38.299 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0011065070000000001, 2024-09-13T11:22:38.299 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:22:38.299 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:22:38.299 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:38.300 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:22:38.300 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.300 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:22:38.300 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.010971+0000", 2024-09-13T11:22:38.300 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:38.300 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.300 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.300 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:22:38.300 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.010971+0000", 2024-09-13T11:22:38.300 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:38.301 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.301 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.301 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:22:38.301 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.010971+0000", 2024-09-13T11:22:38.301 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.0100000000000005e-07 2024-09-13T11:22:38.301 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.301 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.301 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:22:38.301 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.011193+0000", 2024-09-13T11:22:38.301 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000221814 2024-09-13T11:22:38.302 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.302 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.302 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:22:38.302 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.011236+0000", 2024-09-13T11:22:38.302 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.3208999999999999e-05 2024-09-13T11:22:38.302 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.302 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.302 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:22:38.302 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.011254+0000", 2024-09-13T11:22:38.303 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.7365000000000001e-05 2024-09-13T11:22:38.303 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.303 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.303 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:22:38.303 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.011309+0000", 2024-09-13T11:22:38.303 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.5686999999999997e-05 2024-09-13T11:22:38.303 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.303 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.303 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:22:38.303 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.011323+0000", 2024-09-13T11:22:38.304 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.4270000000000001e-05 2024-09-13T11:22:38.304 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.304 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.304 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:22:38.304 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.012030+0000", 2024-09-13T11:22:38.304 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00070670100000000001 2024-09-13T11:22:38.304 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.304 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.304 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:22:38.305 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.012077+0000", 2024-09-13T11:22:38.305 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.6860000000000002e-05 2024-09-13T11:22:38.305 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:38.305 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:22:38.305 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:38.305 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.305 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.305 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:27 3.f 3:fc76794e:::benchmark_data_smithi177_31116_object26:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-09-13T11:22:38.305 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:22:27.013790+0000", 2024-09-13T11:22:38.305 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 11.264836453999999, 2024-09-13T11:22:38.305 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0016845689999999999, 2024-09-13T11:22:38.306 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:22:38.306 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:22:38.306 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:38.306 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:22:38.306 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:22:38.306 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:22:38.306 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 27 2024-09-13T11:22:38.306 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.306 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:22:38.306 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.307 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:22:38.307 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.013790+0000", 2024-09-13T11:22:38.307 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:38.307 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.307 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.307 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:22:38.307 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.013790+0000", 2024-09-13T11:22:38.307 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:38.307 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.307 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.308 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:22:38.308 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.013792+0000", 2024-09-13T11:22:38.308 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5820000000000001e-06 2024-09-13T11:22:38.308 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.308 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.308 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:22:38.308 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.013840+0000", 2024-09-13T11:22:38.308 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.8191e-05 2024-09-13T11:22:38.308 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.308 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.309 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:22:38.309 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.013851+0000", 2024-09-13T11:22:38.309 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.1623999999999999e-05 2024-09-13T11:22:38.309 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.309 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.309 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:22:38.309 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.013858+0000", 2024-09-13T11:22:38.309 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.2689999999999996e-06 2024-09-13T11:22:38.309 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.310 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.310 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:22:38.310 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.013915+0000", 2024-09-13T11:22:38.310 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.7321000000000002e-05 2024-09-13T11:22:38.310 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.310 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.310 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:22:38.310 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.014011+0000", 2024-09-13T11:22:38.310 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.6349999999999997e-05 2024-09-13T11:22:38.310 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.311 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.311 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 12", 2024-09-13T11:22:38.311 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.014106+0000", 2024-09-13T11:22:38.311 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.4288000000000003e-05 2024-09-13T11:22:38.311 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.311 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.311 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:22:38.311 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.014863+0000", 2024-09-13T11:22:38.311 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00075756599999999999 2024-09-13T11:22:38.312 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.312 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.312 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:22:38.312 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.015407+0000", 2024-09-13T11:22:38.312 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00054335099999999997 2024-09-13T11:22:38.312 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.312 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.312 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:22:38.312 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.015450+0000", 2024-09-13T11:22:38.312 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.3112000000000003e-05 2024-09-13T11:22:38.312 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.313 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.313 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:22:38.313 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.015475+0000", 2024-09-13T11:22:38.313 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.4915000000000001e-05 2024-09-13T11:22:38.313 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:38.313 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:22:38.313 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:38.313 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.313 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.313 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop_reply(client.4665.0:23 3.c e32/30 ondisk, result = 0)", 2024-09-13T11:22:38.314 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:22:27.014141+0000", 2024-09-13T11:22:38.314 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 11.264485098, 2024-09-13T11:22:38.314 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00025223699999999999, 2024-09-13T11:22:38.314 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:22:38.314 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:22:38.314 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:22:38.314 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:22:38.314 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.314 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:22:38.314 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.014141+0000", 2024-09-13T11:22:38.314 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:38.315 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.315 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.315 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:22:38.315 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.014141+0000", 2024-09-13T11:22:38.315 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:38.315 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.315 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.315 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:22:38.315 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.014143+0000", 2024-09-13T11:22:38.315 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.297e-06 2024-09-13T11:22:38.316 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.316 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.316 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:22:38.316 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.014148+0000", 2024-09-13T11:22:38.316 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.04e-06 2024-09-13T11:22:38.316 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.316 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.316 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:22:38.316 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.014255+0000", 2024-09-13T11:22:38.316 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00010717099999999999 2024-09-13T11:22:38.316 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.317 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.317 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:22:38.317 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.014261+0000", 2024-09-13T11:22:38.317 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.2920000000000001e-06 2024-09-13T11:22:38.317 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.317 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.317 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:22:38.317 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.014312+0000", 2024-09-13T11:22:38.317 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.1309e-05 2024-09-13T11:22:38.317 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.317 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.318 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:22:38.318 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.014318+0000", 2024-09-13T11:22:38.318 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.9309999999999996e-06 2024-09-13T11:22:38.318 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.318 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.318 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:22:38.318 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.014394+0000", 2024-09-13T11:22:38.318 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.5197000000000005e-05 2024-09-13T11:22:38.318 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:38.318 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:22:38.318 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:38.319 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.319 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.319 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:26 3.1 e32/30 3:8a6e2f8a:::benchmark_data_smithi177_31116_object25:head v 32'2, mlcod=0'0)", 2024-09-13T11:22:38.319 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:22:27.014150+0000", 2024-09-13T11:22:38.319 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 11.264476661, 2024-09-13T11:22:38.319 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0011217600000000001, 2024-09-13T11:22:38.319 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:22:38.319 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:22:38.319 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:38.319 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:22:38.320 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.320 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:22:38.320 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.014150+0000", 2024-09-13T11:22:38.320 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:38.320 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.320 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.320 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:22:38.320 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.014150+0000", 2024-09-13T11:22:38.320 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:38.320 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.320 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.321 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:22:38.321 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.014150+0000", 2024-09-13T11:22:38.321 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.1e-07 2024-09-13T11:22:38.321 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.321 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.321 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:22:38.321 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.014229+0000", 2024-09-13T11:22:38.321 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.9387000000000004e-05 2024-09-13T11:22:38.321 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.321 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.322 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:22:38.322 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.014285+0000", 2024-09-13T11:22:38.322 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.5231e-05 2024-09-13T11:22:38.322 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.322 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.322 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:22:38.322 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.014290+0000", 2024-09-13T11:22:38.322 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.3550000000000004e-06 2024-09-13T11:22:38.322 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.322 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.323 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:22:38.323 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.014340+0000", 2024-09-13T11:22:38.323 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.0362000000000003e-05 2024-09-13T11:22:38.323 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.323 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.323 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:22:38.323 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.014362+0000", 2024-09-13T11:22:38.323 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.1362999999999999e-05 2024-09-13T11:22:38.323 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.323 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.323 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:22:38.323 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.015229+0000", 2024-09-13T11:22:38.324 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00086700500000000001 2024-09-13T11:22:38.324 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.324 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.324 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:22:38.324 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.015272+0000", 2024-09-13T11:22:38.324 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.2846999999999998e-05 2024-09-13T11:22:38.324 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:38.324 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:22:38.324 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:38.324 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.324 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.325 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:38 3.f 3:f31cd2bb:::benchmark_data_smithi177_31116_object37:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-09-13T11:22:38.325 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:22:27.018027+0000", 2024-09-13T11:22:38.325 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 11.260599015, 2024-09-13T11:22:38.325 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0017102179999999999, 2024-09-13T11:22:38.325 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:22:38.325 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:22:38.325 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:38.325 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:22:38.325 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:22:38.325 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:22:38.325 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 38 2024-09-13T11:22:38.326 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.326 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:22:38.326 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.326 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:22:38.326 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.018027+0000", 2024-09-13T11:22:38.326 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:38.326 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.326 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.326 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:22:38.326 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.018027+0000", 2024-09-13T11:22:38.326 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:38.327 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.327 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.327 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:22:38.327 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.018029+0000", 2024-09-13T11:22:38.327 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.0109999999999999e-06 2024-09-13T11:22:38.327 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.327 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.327 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:22:38.327 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.018077+0000", 2024-09-13T11:22:38.327 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.7086000000000003e-05 2024-09-13T11:22:38.328 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.328 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.328 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:22:38.328 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.018090+0000", 2024-09-13T11:22:38.328 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.307e-05 2024-09-13T11:22:38.328 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.328 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.328 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:22:38.328 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.018096+0000", 2024-09-13T11:22:38.328 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.7580000000000003e-06 2024-09-13T11:22:38.329 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.329 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.329 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:22:38.329 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.018149+0000", 2024-09-13T11:22:38.329 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.2262000000000002e-05 2024-09-13T11:22:38.329 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.329 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.329 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:22:38.329 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.018238+0000", 2024-09-13T11:22:38.329 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.9126e-05 2024-09-13T11:22:38.329 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.330 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.330 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 12", 2024-09-13T11:22:38.330 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.018375+0000", 2024-09-13T11:22:38.330 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00013688900000000001 2024-09-13T11:22:38.330 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.330 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.330 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:22:38.330 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.019168+0000", 2024-09-13T11:22:38.330 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000793449 2024-09-13T11:22:38.330 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.331 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.331 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:22:38.331 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.019664+0000", 2024-09-13T11:22:38.331 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00049592500000000003 2024-09-13T11:22:38.331 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.331 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.331 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:22:38.331 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.019704+0000", 2024-09-13T11:22:38.331 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.0176000000000003e-05 2024-09-13T11:22:38.331 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.331 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.332 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:22:38.332 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.019738+0000", 2024-09-13T11:22:38.332 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.3466000000000001e-05 2024-09-13T11:22:38.332 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:38.332 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:22:38.332 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:38.332 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.332 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.332 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:40 3.c 3:30812dfc:::benchmark_data_smithi177_31116_object39:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-09-13T11:22:38.332 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:22:27.019015+0000", 2024-09-13T11:22:38.333 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 11.259611705999999, 2024-09-13T11:22:38.333 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001945771, 2024-09-13T11:22:38.333 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:22:38.333 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:22:38.333 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:38.333 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:22:38.333 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:22:38.333 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:22:38.333 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 40 2024-09-13T11:22:38.333 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.333 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:22:38.334 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.334 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:22:38.334 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.019015+0000", 2024-09-13T11:22:38.334 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:38.334 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.334 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.334 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:22:38.334 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.019015+0000", 2024-09-13T11:22:38.334 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:38.334 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.334 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.335 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:22:38.335 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.019017+0000", 2024-09-13T11:22:38.335 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.0260000000000002e-06 2024-09-13T11:22:38.335 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.335 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.335 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:22:38.335 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.019080+0000", 2024-09-13T11:22:38.335 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.3541999999999996e-05 2024-09-13T11:22:38.335 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.335 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.335 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:22:38.335 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.019093+0000", 2024-09-13T11:22:38.336 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3127e-05 2024-09-13T11:22:38.336 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.336 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.336 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:22:38.336 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.019103+0000", 2024-09-13T11:22:38.336 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.1160000000000005e-06 2024-09-13T11:22:38.336 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.336 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.336 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:22:38.336 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.019157+0000", 2024-09-13T11:22:38.337 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.4279e-05 2024-09-13T11:22:38.337 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.337 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.337 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:22:38.337 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.019247+0000", 2024-09-13T11:22:38.337 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.9979e-05 2024-09-13T11:22:38.337 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.337 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.337 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 13", 2024-09-13T11:22:38.337 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.019337+0000", 2024-09-13T11:22:38.338 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.0179000000000005e-05 2024-09-13T11:22:38.338 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.338 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.338 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:22:38.338 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.020496+0000", 2024-09-13T11:22:38.338 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001159338 2024-09-13T11:22:38.338 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.338 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.338 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:22:38.338 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.020860+0000", 2024-09-13T11:22:38.339 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00036339 2024-09-13T11:22:38.339 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.339 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.339 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:22:38.339 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.020915+0000", 2024-09-13T11:22:38.339 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.5788000000000002e-05 2024-09-13T11:22:38.339 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.339 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.339 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:22:38.339 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.020961+0000", 2024-09-13T11:22:38.339 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.5006999999999998e-05 2024-09-13T11:22:38.340 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:38.340 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:22:38.340 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:38.340 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.340 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.340 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop_reply(client.4665.0:40 3.c e32/30 ondisk, result = 0)", 2024-09-13T11:22:38.340 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:22:27.020757+0000", 2024-09-13T11:22:38.340 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 11.257869465000001, 2024-09-13T11:22:38.340 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00021214899999999999, 2024-09-13T11:22:38.340 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:22:38.341 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:22:38.341 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:22:38.341 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:22:38.341 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.341 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:22:38.341 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.020757+0000", 2024-09-13T11:22:38.341 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:38.341 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.341 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.341 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:22:38.341 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.020757+0000", 2024-09-13T11:22:38.342 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:38.342 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.342 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.342 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:22:38.342 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.020759+0000", 2024-09-13T11:22:38.342 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.573e-06 2024-09-13T11:22:38.342 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.342 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.342 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:22:38.342 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.020763+0000", 2024-09-13T11:22:38.343 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.3599999999999998e-06 2024-09-13T11:22:38.343 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.343 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.343 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:22:38.343 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.020790+0000", 2024-09-13T11:22:38.343 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.6639999999999999e-05 2024-09-13T11:22:38.343 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.343 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.343 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:22:38.343 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.020795+0000", 2024-09-13T11:22:38.343 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.6869999999999999e-06 2024-09-13T11:22:38.344 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.344 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.344 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:22:38.344 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.020845+0000", 2024-09-13T11:22:38.344 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.9923999999999998e-05 2024-09-13T11:22:38.344 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.344 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.344 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:22:38.344 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.020854+0000", 2024-09-13T11:22:38.344 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.8950000000000004e-06 2024-09-13T11:22:38.344 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.345 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.345 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:22:38.345 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.020969+0000", 2024-09-13T11:22:38.345 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00011506999999999999 2024-09-13T11:22:38.345 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:38.345 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:22:38.345 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:38.345 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.345 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.345 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:43 3.1 e32/30 3:85e4c473:::benchmark_data_smithi177_31116_object42:head v 32'3, mlcod=32'1)", 2024-09-13T11:22:38.346 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:22:27.021766+0000", 2024-09-13T11:22:38.346 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 11.256860502, 2024-09-13T11:22:38.346 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00098155099999999999, 2024-09-13T11:22:38.346 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:22:38.346 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:22:38.346 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:38.346 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:22:38.346 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.346 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:22:38.346 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.021766+0000", 2024-09-13T11:22:38.346 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:38.347 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.347 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.347 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:22:38.347 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.021766+0000", 2024-09-13T11:22:38.347 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:38.347 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.347 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.347 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:22:38.347 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.021767+0000", 2024-09-13T11:22:38.347 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.246e-06 2024-09-13T11:22:38.348 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.348 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.348 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:22:38.348 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.021829+0000", 2024-09-13T11:22:38.348 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.1481000000000003e-05 2024-09-13T11:22:38.348 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.348 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.348 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:22:38.348 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.021881+0000", 2024-09-13T11:22:38.348 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.2052000000000002e-05 2024-09-13T11:22:38.348 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.349 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.349 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:22:38.349 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.021888+0000", 2024-09-13T11:22:38.349 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.6539999999999999e-06 2024-09-13T11:22:38.349 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.349 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.349 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:22:38.349 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.021959+0000", 2024-09-13T11:22:38.349 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.0562999999999995e-05 2024-09-13T11:22:38.349 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.350 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.350 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:22:38.350 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.021985+0000", 2024-09-13T11:22:38.350 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.6539000000000001e-05 2024-09-13T11:22:38.350 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.350 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.350 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:22:38.350 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.022688+0000", 2024-09-13T11:22:38.350 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00070257100000000003 2024-09-13T11:22:38.350 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.350 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.351 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:22:38.351 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.022747+0000", 2024-09-13T11:22:38.351 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.9444999999999998e-05 2024-09-13T11:22:38.351 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:38.351 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:22:38.351 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:38.351 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.351 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.351 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:54 3.c 3:39f6733d:::benchmark_data_smithi177_31116_object53:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-09-13T11:22:38.351 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:22:27.026161+0000", 2024-09-13T11:22:38.351 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 11.252465013, 2024-09-13T11:22:38.352 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0046609980000000004, 2024-09-13T11:22:38.352 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:22:38.352 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:22:38.352 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:38.352 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:22:38.352 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:22:38.352 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:22:38.352 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 54 2024-09-13T11:22:38.352 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.352 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:22:38.353 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.353 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:22:38.353 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.026161+0000", 2024-09-13T11:22:38.353 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:38.353 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.353 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.353 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:22:38.353 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.026161+0000", 2024-09-13T11:22:38.353 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:38.353 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.354 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.354 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:22:38.354 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.026165+0000", 2024-09-13T11:22:38.354 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.1719999999999999e-06 2024-09-13T11:22:38.354 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.354 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.354 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:22:38.354 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.026225+0000", 2024-09-13T11:22:38.354 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.0302e-05 2024-09-13T11:22:38.354 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.354 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.355 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:22:38.355 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.026243+0000", 2024-09-13T11:22:38.355 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.8359000000000002e-05 2024-09-13T11:22:38.355 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.355 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.355 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:22:38.355 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.026253+0000", 2024-09-13T11:22:38.355 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0036e-05 2024-09-13T11:22:38.355 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.355 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.355 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:22:38.356 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.026345+0000", 2024-09-13T11:22:38.356 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.1507000000000003e-05 2024-09-13T11:22:38.356 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.356 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.356 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:22:38.356 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.026456+0000", 2024-09-13T11:22:38.356 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000111351 2024-09-13T11:22:38.356 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.356 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.357 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 13", 2024-09-13T11:22:38.357 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.026550+0000", 2024-09-13T11:22:38.357 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.3414999999999999e-05 2024-09-13T11:22:38.357 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.357 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.357 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:22:38.357 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.028192+0000", 2024-09-13T11:22:38.357 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001642739 2024-09-13T11:22:38.357 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.357 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.357 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:22:38.358 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.030697+0000", 2024-09-13T11:22:38.358 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0025045139999999998 2024-09-13T11:22:38.358 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.358 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.358 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:22:38.358 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.030773+0000", 2024-09-13T11:22:38.358 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.6100999999999995e-05 2024-09-13T11:22:38.358 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.358 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.358 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:22:38.359 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.030822+0000", 2024-09-13T11:22:38.359 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.9502000000000001e-05 2024-09-13T11:22:38.359 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:38.359 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:22:38.359 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:38.359 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.359 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.359 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:55 3.f 3:f1c4925a:::benchmark_data_smithi177_31116_object54:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-09-13T11:22:38.359 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:22:27.027636+0000", 2024-09-13T11:22:38.360 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 11.250990180000001, 2024-09-13T11:22:38.360 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0041436440000000001, 2024-09-13T11:22:38.360 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:22:38.360 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:22:38.360 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:38.360 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:22:38.360 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:22:38.360 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:22:38.360 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 55 2024-09-13T11:22:38.360 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.361 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:22:38.361 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.361 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:22:38.361 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.027636+0000", 2024-09-13T11:22:38.361 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:38.361 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.361 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.361 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:22:38.361 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.027636+0000", 2024-09-13T11:22:38.361 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:38.361 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.362 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.362 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:22:38.362 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.027638+0000", 2024-09-13T11:22:38.362 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.968e-06 2024-09-13T11:22:38.362 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.362 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.362 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:22:38.362 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.027672+0000", 2024-09-13T11:22:38.362 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.3330000000000001e-05 2024-09-13T11:22:38.362 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.363 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.363 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:22:38.363 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.027687+0000", 2024-09-13T11:22:38.363 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5892e-05 2024-09-13T11:22:38.363 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.363 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.363 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:22:38.363 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.027697+0000", 2024-09-13T11:22:38.363 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.329e-06 2024-09-13T11:22:38.363 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.364 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.364 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:22:38.364 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.027763+0000", 2024-09-13T11:22:38.364 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.5795000000000003e-05 2024-09-13T11:22:38.364 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.364 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.364 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:22:38.364 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.027878+0000", 2024-09-13T11:22:38.364 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000114966 2024-09-13T11:22:38.364 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.364 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.365 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 12", 2024-09-13T11:22:38.365 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.028009+0000", 2024-09-13T11:22:38.365 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000131354 2024-09-13T11:22:38.365 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.365 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.365 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:22:38.365 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.030893+0000", 2024-09-13T11:22:38.365 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0028840290000000002 2024-09-13T11:22:38.365 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.365 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.366 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:22:38.366 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.031607+0000", 2024-09-13T11:22:38.366 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00071392199999999997 2024-09-13T11:22:38.366 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.366 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.366 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:22:38.366 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.031669+0000", 2024-09-13T11:22:38.366 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.1890999999999994e-05 2024-09-13T11:22:38.366 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.366 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.367 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:22:38.367 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.031780+0000", 2024-09-13T11:22:38.367 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000111168 2024-09-13T11:22:38.367 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:38.367 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:22:38.367 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:38.367 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.367 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.367 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:58 3.1 e32/30 3:893a52d5:::benchmark_data_smithi177_31116_object57:head v 32'4, mlcod=32'2)", 2024-09-13T11:22:38.367 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:22:27.029562+0000", 2024-09-13T11:22:38.368 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 11.249064285999999, 2024-09-13T11:22:38.368 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0023439839999999999, 2024-09-13T11:22:38.368 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:22:38.368 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:22:38.368 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:38.368 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:22:38.368 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.368 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:22:38.368 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.029562+0000", 2024-09-13T11:22:38.368 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:38.369 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.369 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.369 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:22:38.369 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.029562+0000", 2024-09-13T11:22:38.369 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:38.369 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.369 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.369 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:22:38.369 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.029564+0000", 2024-09-13T11:22:38.369 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.539e-06 2024-09-13T11:22:38.369 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.370 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.370 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:22:38.370 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.029612+0000", 2024-09-13T11:22:38.370 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.8136000000000001e-05 2024-09-13T11:22:38.370 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.370 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.370 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:22:38.370 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.029644+0000", 2024-09-13T11:22:38.370 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.1689000000000001e-05 2024-09-13T11:22:38.370 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.371 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.371 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:22:38.371 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.029649+0000", 2024-09-13T11:22:38.371 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.7919999999999998e-06 2024-09-13T11:22:38.371 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.371 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.371 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:22:38.371 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.029740+0000", 2024-09-13T11:22:38.371 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.0601999999999998e-05 2024-09-13T11:22:38.371 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.371 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.372 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:22:38.372 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.029765+0000", 2024-09-13T11:22:38.372 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.5188999999999999e-05 2024-09-13T11:22:38.372 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.372 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.372 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:22:38.372 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.031858+0000", 2024-09-13T11:22:38.372 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0020932619999999998 2024-09-13T11:22:38.372 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.372 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.373 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:22:38.373 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.031906+0000", 2024-09-13T11:22:38.373 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.7775000000000003e-05 2024-09-13T11:22:38.373 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:38.373 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:22:38.373 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:38.373 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.373 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.373 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop_reply(client.4665.0:55 3.f e32/30 ondisk, result = 0)", 2024-09-13T11:22:38.373 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:22:27.029808+0000", 2024-09-13T11:22:38.373 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 11.248818485999999, 2024-09-13T11:22:38.374 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0010955800000000001, 2024-09-13T11:22:38.374 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:22:38.374 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:22:38.374 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:22:38.374 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:22:38.374 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.374 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:22:38.374 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.029808+0000", 2024-09-13T11:22:38.374 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:38.374 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.375 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.375 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:22:38.375 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.029808+0000", 2024-09-13T11:22:38.375 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:38.375 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.375 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.375 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:22:38.375 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.029809+0000", 2024-09-13T11:22:38.375 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.393e-06 2024-09-13T11:22:38.375 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.376 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.376 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:22:38.376 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.029817+0000", 2024-09-13T11:22:38.376 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.6389999999999992e-06 2024-09-13T11:22:38.376 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.376 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.376 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:22:38.376 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.029853+0000", 2024-09-13T11:22:38.376 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.6483000000000003e-05 2024-09-13T11:22:38.376 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.377 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.377 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:22:38.377 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.029869+0000", 2024-09-13T11:22:38.377 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5089000000000001e-05 2024-09-13T11:22:38.377 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.377 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.377 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:22:38.377 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.030851+0000", 2024-09-13T11:22:38.377 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00098282100000000004 2024-09-13T11:22:38.378 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.378 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.378 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:22:38.378 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.030881+0000", 2024-09-13T11:22:38.378 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.9595e-05 2024-09-13T11:22:38.378 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.378 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.378 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:22:38.378 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.030904+0000", 2024-09-13T11:22:38.378 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.2560000000000001e-05 2024-09-13T11:22:38.378 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:38.379 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:22:38.379 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:38.379 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.379 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.379 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:67 3.1 e32/30 3:864186ed:::benchmark_data_smithi177_31116_object66:head v 32'5, mlcod=32'3)", 2024-09-13T11:22:38.379 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:22:27.033897+0000", 2024-09-13T11:22:38.379 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 11.24472986, 2024-09-13T11:22:38.379 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00079932199999999999, 2024-09-13T11:22:38.379 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:22:38.379 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:22:38.379 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:38.380 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:22:38.380 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.380 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:22:38.380 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.033897+0000", 2024-09-13T11:22:38.380 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:38.380 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.380 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.380 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:22:38.380 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.033897+0000", 2024-09-13T11:22:38.380 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:38.381 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.381 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.381 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:22:38.381 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.033900+0000", 2024-09-13T11:22:38.381 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.2830000000000002e-06 2024-09-13T11:22:38.381 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.381 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.381 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:22:38.381 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.033958+0000", 2024-09-13T11:22:38.381 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.8375999999999998e-05 2024-09-13T11:22:38.382 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.382 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.382 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:22:38.382 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.033997+0000", 2024-09-13T11:22:38.382 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.8291999999999999e-05 2024-09-13T11:22:38.382 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.382 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.382 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:22:38.382 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.034004+0000", 2024-09-13T11:22:38.382 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.6090000000000003e-06 2024-09-13T11:22:38.383 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.383 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.383 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:22:38.383 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.034061+0000", 2024-09-13T11:22:38.383 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.7138000000000001e-05 2024-09-13T11:22:38.383 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.383 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.383 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:22:38.383 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.034082+0000", 2024-09-13T11:22:38.383 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.0245999999999999e-05 2024-09-13T11:22:38.383 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.384 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.384 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:22:38.384 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.034648+0000", 2024-09-13T11:22:38.384 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00056620200000000002 2024-09-13T11:22:38.384 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.384 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.384 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:22:38.384 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.034696+0000", 2024-09-13T11:22:38.384 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.8176000000000001e-05 2024-09-13T11:22:38.385 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:38.385 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:22:38.385 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:38.385 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.385 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.385 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:68 3.c 3:302541f2:::benchmark_data_smithi177_31116_object67:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-09-13T11:22:38.385 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:22:27.034579+0000", 2024-09-13T11:22:38.385 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 11.244047621, 2024-09-13T11:22:38.385 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.022921435, 2024-09-13T11:22:38.385 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:22:38.385 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:22:38.386 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:38.386 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:22:38.386 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:22:38.386 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:22:38.386 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 68 2024-09-13T11:22:38.386 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.386 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:22:38.386 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.386 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:22:38.386 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.034579+0000", 2024-09-13T11:22:38.386 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:38.386 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.387 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.387 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:22:38.387 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.034579+0000", 2024-09-13T11:22:38.387 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:38.387 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.387 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.387 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:22:38.387 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.034581+0000", 2024-09-13T11:22:38.387 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.8640000000000001e-06 2024-09-13T11:22:38.387 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.387 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.387 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:22:38.388 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.034609+0000", 2024-09-13T11:22:38.388 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.7919000000000002e-05 2024-09-13T11:22:38.388 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.388 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.388 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:22:38.388 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.034621+0000", 2024-09-13T11:22:38.388 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.2116999999999999e-05 2024-09-13T11:22:38.388 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.388 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.388 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:22:38.388 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.034631+0000", 2024-09-13T11:22:38.388 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0083e-05 2024-09-13T11:22:38.389 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.389 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.389 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:22:38.389 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.034696+0000", 2024-09-13T11:22:38.389 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.5158000000000002e-05 2024-09-13T11:22:38.389 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.389 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.389 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:22:38.389 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.034842+0000", 2024-09-13T11:22:38.389 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00014596699999999999 2024-09-13T11:22:38.389 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.390 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.390 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 13", 2024-09-13T11:22:38.390 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.034950+0000", 2024-09-13T11:22:38.390 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000108064 2024-09-13T11:22:38.390 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.390 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.390 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:22:38.390 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.035686+0000", 2024-09-13T11:22:38.390 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00073563700000000003 2024-09-13T11:22:38.390 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.390 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.390 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:22:38.391 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.057431+0000", 2024-09-13T11:22:38.391 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.021744982999999999 2024-09-13T11:22:38.391 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.391 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.391 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:22:38.391 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.057469+0000", 2024-09-13T11:22:38.391 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.8090000000000003e-05 2024-09-13T11:22:38.391 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.391 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.391 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:22:38.391 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.057500+0000", 2024-09-13T11:22:38.391 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.1553000000000001e-05 2024-09-13T11:22:38.392 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:38.392 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:22:38.392 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:38.392 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.392 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.392 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop_reply(client.4665.0:68 3.c e32/30 ondisk, result = 0)", 2024-09-13T11:22:38.392 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:22:27.036487+0000", 2024-09-13T11:22:38.392 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 11.242139503000001, 2024-09-13T11:22:38.392 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.021027839999999999, 2024-09-13T11:22:38.392 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:22:38.392 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:22:38.393 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:22:38.393 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:22:38.393 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.393 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:22:38.393 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.036487+0000", 2024-09-13T11:22:38.393 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:38.393 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.393 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.393 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:22:38.393 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.036487+0000", 2024-09-13T11:22:38.393 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:38.394 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.394 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.394 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:22:38.394 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.036488+0000", 2024-09-13T11:22:38.394 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.415e-06 2024-09-13T11:22:38.394 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.394 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.394 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:22:38.394 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.036493+0000", 2024-09-13T11:22:38.394 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.4159999999999997e-06 2024-09-13T11:22:38.394 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.394 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.395 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:22:38.395 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.057355+0000", 2024-09-13T11:22:38.395 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.020862393 2024-09-13T11:22:38.395 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.395 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.395 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:22:38.395 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.057362+0000", 2024-09-13T11:22:38.395 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.7830000000000001e-06 2024-09-13T11:22:38.395 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.395 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.395 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:22:38.396 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.057422+0000", 2024-09-13T11:22:38.396 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.9858999999999997e-05 2024-09-13T11:22:38.396 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.396 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.396 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:22:38.396 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.057427+0000", 2024-09-13T11:22:38.396 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.5520000000000004e-06 2024-09-13T11:22:38.396 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.396 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.396 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:22:38.397 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.057515+0000", 2024-09-13T11:22:38.397 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.7422000000000004e-05 2024-09-13T11:22:38.397 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:38.397 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:22:38.397 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:38.397 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.397 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.397 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:79 3.1 e32/30 3:841df5b5:::benchmark_data_smithi177_31116_object78:head v 32'6, mlcod=32'4)", 2024-09-13T11:22:38.397 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:22:27.057292+0000", 2024-09-13T11:22:38.397 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 11.221334560000001, 2024-09-13T11:22:38.397 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.89296786800000005, 2024-09-13T11:22:38.397 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:22:38.398 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:22:38.398 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:38.398 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:22:38.398 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.398 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:22:38.398 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.057292+0000", 2024-09-13T11:22:38.398 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:38.398 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.398 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.398 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:22:38.398 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.057292+0000", 2024-09-13T11:22:38.399 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:38.399 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.399 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.399 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:22:38.399 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.057293+0000", 2024-09-13T11:22:38.399 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.6109999999999999e-06 2024-09-13T11:22:38.399 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.399 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.399 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:22:38.399 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.057330+0000", 2024-09-13T11:22:38.399 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.6721999999999998e-05 2024-09-13T11:22:38.400 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.400 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.400 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:22:38.400 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.948197+0000", 2024-09-13T11:22:38.400 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.890867042 2024-09-13T11:22:38.400 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.400 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.400 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:22:38.400 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.948213+0000", 2024-09-13T11:22:38.400 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5826000000000001e-05 2024-09-13T11:22:38.401 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.401 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.401 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:22:38.401 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.948377+0000", 2024-09-13T11:22:38.401 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00016348600000000001 2024-09-13T11:22:38.401 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.401 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.401 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:22:38.401 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.948447+0000", 2024-09-13T11:22:38.401 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.0891999999999998e-05 2024-09-13T11:22:38.402 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.402 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.402 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:22:38.402 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.950150+0000", 2024-09-13T11:22:38.402 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001702329 2024-09-13T11:22:38.402 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.402 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.402 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:22:38.402 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.950260+0000", 2024-09-13T11:22:38.402 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00010996 2024-09-13T11:22:38.402 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:38.402 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:22:38.403 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:38.403 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.403 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.403 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:84 3.c 3:35015570:::benchmark_data_smithi177_31116_object83:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e33)", 2024-09-13T11:22:38.403 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:22:27.966825+0000", 2024-09-13T11:22:38.403 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 10.311800938999999, 2024-09-13T11:22:38.403 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0096167700000000002, 2024-09-13T11:22:38.403 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:22:38.403 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:22:38.403 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:38.404 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:22:38.404 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:22:38.404 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:22:38.404 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 84 2024-09-13T11:22:38.404 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.404 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:22:38.404 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.404 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:22:38.404 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.966825+0000", 2024-09-13T11:22:38.404 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:38.404 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.405 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.405 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:22:38.405 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.966825+0000", 2024-09-13T11:22:38.405 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:38.405 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.405 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.405 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:22:38.405 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.966829+0000", 2024-09-13T11:22:38.405 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.7730000000000001e-06 2024-09-13T11:22:38.405 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.405 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.406 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:22:38.406 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.966864+0000", 2024-09-13T11:22:38.406 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.4817000000000002e-05 2024-09-13T11:22:38.406 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.406 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.406 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:22:38.406 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.966884+0000", 2024-09-13T11:22:38.406 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.0231999999999999e-05 2024-09-13T11:22:38.406 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.406 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.406 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:22:38.407 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.966896+0000", 2024-09-13T11:22:38.407 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.1931e-05 2024-09-13T11:22:38.407 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.407 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.407 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:22:38.407 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.967016+0000", 2024-09-13T11:22:38.407 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000119388 2024-09-13T11:22:38.407 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.407 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.407 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op must wait for map", 2024-09-13T11:22:38.407 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.967022+0000", 2024-09-13T11:22:38.408 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.81e-06 2024-09-13T11:22:38.408 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.408 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.408 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:22:38.408 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.969760+0000", 2024-09-13T11:22:38.408 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.002737752 2024-09-13T11:22:38.408 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.408 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.408 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:22:38.408 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.970016+0000", 2024-09-13T11:22:38.409 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00025577100000000001 2024-09-13T11:22:38.409 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.409 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.409 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 13", 2024-09-13T11:22:38.409 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.970266+0000", 2024-09-13T11:22:38.409 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00024996799999999999 2024-09-13T11:22:38.409 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.409 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.409 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:22:38.409 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.975082+0000", 2024-09-13T11:22:38.410 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0048164189999999997 2024-09-13T11:22:38.410 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.410 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.410 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:22:38.410 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.976282+0000", 2024-09-13T11:22:38.410 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001199526 2024-09-13T11:22:38.410 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.410 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.410 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:22:38.410 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.976375+0000", 2024-09-13T11:22:38.411 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.3393999999999994e-05 2024-09-13T11:22:38.411 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.411 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.411 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:22:38.411 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.976442+0000", 2024-09-13T11:22:38.411 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.6989000000000005e-05 2024-09-13T11:22:38.411 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:38.411 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:22:38.411 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:38.411 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.411 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.411 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop_reply(client.4665.0:84 3.c e32/30 ondisk, result = 0)", 2024-09-13T11:22:38.412 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:22:27.976113+0000", 2024-09-13T11:22:38.412 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 10.302512975000001, 2024-09-13T11:22:38.412 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00034822500000000002, 2024-09-13T11:22:38.412 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:22:38.412 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:22:38.412 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:22:38.412 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:22:38.412 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.412 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:22:38.412 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.976113+0000", 2024-09-13T11:22:38.412 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:38.413 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.413 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.413 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:22:38.413 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.976113+0000", 2024-09-13T11:22:38.413 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:38.413 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.413 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.413 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:22:38.413 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.976116+0000", 2024-09-13T11:22:38.413 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.3599999999999999e-06 2024-09-13T11:22:38.413 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.414 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.414 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:22:38.414 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.976123+0000", 2024-09-13T11:22:38.414 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.6709999999999999e-06 2024-09-13T11:22:38.414 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.414 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.414 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:22:38.414 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.976163+0000", 2024-09-13T11:22:38.414 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.9357000000000003e-05 2024-09-13T11:22:38.414 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.414 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.414 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:22:38.415 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.976171+0000", 2024-09-13T11:22:38.415 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.8140000000000001e-06 2024-09-13T11:22:38.415 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.415 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.415 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:22:38.415 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.976257+0000", 2024-09-13T11:22:38.415 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.6533000000000007e-05 2024-09-13T11:22:38.415 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.415 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.415 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:22:38.415 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.976271+0000", 2024-09-13T11:22:38.415 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.4059e-05 2024-09-13T11:22:38.416 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.416 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.416 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:22:38.416 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.976462+0000", 2024-09-13T11:22:38.416 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00019043099999999999 2024-09-13T11:22:38.416 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:38.416 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:22:38.416 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:38.416 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:38.416 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:22:38.416 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-09-13T11:22:38.418 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_ops_in_flight 2024-09-13T11:22:38.419 INFO:tasks.ceph.osd.9.smithi092.stderr:2024-09-13T11:22:38.415+0000 7ff5ab3d0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26348) UID: 0 2024-09-13T11:22:38.518 INFO:tasks.ceph.osd.7.smithi177.stderr:2024-09-13T11:22:38.516+0000 7f897b51d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26575) UID: 0 2024-09-13T11:22:38.545 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-09-13T11:22:38.545 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [], 2024-09-13T11:22:38.545 INFO:teuthology.orchestra.run.smithi177.stdout: "num_ops": 0 2024-09-13T11:22:38.545 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-09-13T11:22:38.560 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_blocked_ops 2024-09-13T11:22:38.650 INFO:tasks.ceph.osd.11.smithi177.stderr:2024-09-13T11:22:38.648+0000 7f24792c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26580) UID: 0 2024-09-13T11:22:38.721 INFO:tasks.ceph.osd.5.smithi092.stderr:2024-09-13T11:22:38.719+0000 7f6f0b705640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26346) UID: 0 2024-09-13T11:22:38.726 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-09-13T11:22:38.726 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [], 2024-09-13T11:22:38.726 INFO:teuthology.orchestra.run.smithi177.stdout: "complaint_time": 30, 2024-09-13T11:22:38.726 INFO:teuthology.orchestra.run.smithi177.stdout: "num_blocked_ops": 0 2024-09-13T11:22:38.727 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-09-13T11:22:38.741 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_historic_ops 2024-09-13T11:22:38.822 INFO:tasks.ceph.osd.10.smithi162.stderr:2024-09-13T11:22:38.819+0000 7fb59c9f0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26317) UID: 0 2024-09-13T11:22:38.908 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-09-13T11:22:38.908 INFO:teuthology.orchestra.run.smithi177.stdout: "size": 20, 2024-09-13T11:22:38.908 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 600, 2024-09-13T11:22:38.908 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [ 2024-09-13T11:22:38.908 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.908 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4344.0:1 2.3 e26/25 2:c4c92e5a:::rbd_trash:head v 26'1, mlcod=0'0)", 2024-09-13T11:22:38.908 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:22:14.148791+0000", 2024-09-13T11:22:38.908 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 24.75813213, 2024-09-13T11:22:38.908 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001819464, 2024-09-13T11:22:38.909 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:22:38.909 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:22:38.909 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:38.909 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:22:38.909 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.909 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:22:38.909 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:14.148791+0000", 2024-09-13T11:22:38.909 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:38.909 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.909 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.909 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:22:38.909 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:14.148791+0000", 2024-09-13T11:22:38.909 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:38.909 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.910 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.910 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:22:38.910 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:14.148791+0000", 2024-09-13T11:22:38.910 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.1999999999999999e-07 2024-09-13T11:22:38.910 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.910 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.910 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:22:38.910 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:14.148801+0000", 2024-09-13T11:22:38.910 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.3160000000000003e-06 2024-09-13T11:22:38.910 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.910 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.910 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:22:38.910 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:14.148856+0000", 2024-09-13T11:22:38.910 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.4752e-05 2024-09-13T11:22:38.911 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.911 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.911 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:22:38.911 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:14.148871+0000", 2024-09-13T11:22:38.911 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5387999999999999e-05 2024-09-13T11:22:38.911 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.911 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.911 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:22:38.911 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:14.149008+0000", 2024-09-13T11:22:38.911 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000137565 2024-09-13T11:22:38.911 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.911 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.911 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:22:38.912 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:14.149049+0000", 2024-09-13T11:22:38.912 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.0873e-05 2024-09-13T11:22:38.912 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.912 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.912 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:22:38.912 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:14.150530+0000", 2024-09-13T11:22:38.912 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001480631 2024-09-13T11:22:38.912 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.912 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.912 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:22:38.912 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:14.150610+0000", 2024-09-13T11:22:38.912 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.0319000000000001e-05 2024-09-13T11:22:38.912 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:38.912 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:22:38.913 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:38.913 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.913 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.913 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:6 3.5 3:a0cfbf8a:::benchmark_data_smithi177_31116_object5:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-09-13T11:22:38.913 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:22:27.002268+0000", 2024-09-13T11:22:38.913 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 11.904654468, 2024-09-13T11:22:38.913 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0077901719999999997, 2024-09-13T11:22:38.913 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:22:38.913 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:22:38.913 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:38.913 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:22:38.913 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:22:38.913 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:22:38.913 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 6 2024-09-13T11:22:38.914 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.914 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:22:38.914 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.914 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:22:38.914 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.002268+0000", 2024-09-13T11:22:38.914 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:38.914 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.914 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.914 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:22:38.914 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.002268+0000", 2024-09-13T11:22:38.914 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:38.914 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.914 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.915 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:22:38.915 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.002271+0000", 2024-09-13T11:22:38.915 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.9849999999999998e-06 2024-09-13T11:22:38.915 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.915 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.915 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:22:38.915 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.002331+0000", 2024-09-13T11:22:38.915 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.9499000000000001e-05 2024-09-13T11:22:38.915 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.915 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.915 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:22:38.915 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.002346+0000", 2024-09-13T11:22:38.915 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.4800000000000001e-05 2024-09-13T11:22:38.915 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.916 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.916 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:22:38.916 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.002356+0000", 2024-09-13T11:22:38.916 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.024e-05 2024-09-13T11:22:38.916 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.916 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.916 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:22:38.916 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.002437+0000", 2024-09-13T11:22:38.916 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.1102999999999999e-05 2024-09-13T11:22:38.916 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.916 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.916 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op must wait for map", 2024-09-13T11:22:38.916 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.002443+0000", 2024-09-13T11:22:38.917 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.2090000000000002e-06 2024-09-13T11:22:38.917 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.917 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.917 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:22:38.917 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.004598+0000", 2024-09-13T11:22:38.917 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.002154703 2024-09-13T11:22:38.917 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.917 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.917 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:22:38.917 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.004707+0000", 2024-09-13T11:22:38.917 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000108829 2024-09-13T11:22:38.917 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.917 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.918 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 13", 2024-09-13T11:22:38.918 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.004812+0000", 2024-09-13T11:22:38.918 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000104887 2024-09-13T11:22:38.918 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.918 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.918 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:22:38.918 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.009608+0000", 2024-09-13T11:22:38.918 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.004796257 2024-09-13T11:22:38.918 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.918 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.918 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:22:38.918 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.009966+0000", 2024-09-13T11:22:38.918 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00035775 2024-09-13T11:22:38.918 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.919 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.919 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:22:38.919 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.010015+0000", 2024-09-13T11:22:38.919 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.9629000000000002e-05 2024-09-13T11:22:38.919 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.919 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.919 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:22:38.919 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.010059+0000", 2024-09-13T11:22:38.919 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.3281000000000001e-05 2024-09-13T11:22:38.919 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:38.919 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:22:38.919 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:38.919 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.919 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.920 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:1 3.0 e32/30 3:0f48e989:::benchmark_data_smithi177_31116_object0:head v 32'1, mlcod=0'0)", 2024-09-13T11:22:38.920 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:22:27.005948+0000", 2024-09-13T11:22:38.920 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 11.900974903, 2024-09-13T11:22:38.920 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001014619, 2024-09-13T11:22:38.920 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:22:38.920 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:22:38.920 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:38.920 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:22:38.920 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.920 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:22:38.920 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.005948+0000", 2024-09-13T11:22:38.920 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:38.920 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.920 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.921 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:22:38.921 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.005948+0000", 2024-09-13T11:22:38.921 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:38.921 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.921 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.921 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:22:38.921 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.005949+0000", 2024-09-13T11:22:38.921 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.7400000000000003e-07 2024-09-13T11:22:38.921 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.921 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.921 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:22:38.921 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.006212+0000", 2024-09-13T11:22:38.921 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00026340599999999999 2024-09-13T11:22:38.921 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.922 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.922 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:22:38.922 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.006237+0000", 2024-09-13T11:22:38.922 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.4808e-05 2024-09-13T11:22:38.922 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.922 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.922 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:22:38.922 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.006243+0000", 2024-09-13T11:22:38.922 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.4080000000000003e-06 2024-09-13T11:22:38.922 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.922 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.922 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:22:38.922 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.006292+0000", 2024-09-13T11:22:38.923 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.8612000000000001e-05 2024-09-13T11:22:38.923 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.923 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.923 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:22:38.923 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.006306+0000", 2024-09-13T11:22:38.923 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.375e-05 2024-09-13T11:22:38.923 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.923 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.923 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:22:38.923 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.006931+0000", 2024-09-13T11:22:38.923 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00062542399999999997 2024-09-13T11:22:38.923 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.923 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.923 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:22:38.924 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.006963+0000", 2024-09-13T11:22:38.924 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.1537e-05 2024-09-13T11:22:38.924 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:38.924 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:22:38.924 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:38.924 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.924 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.924 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:14 3.0 e32/30 3:047c9174:::benchmark_data_smithi177_31116_object13:head v 32'2, mlcod=0'0)", 2024-09-13T11:22:38.924 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:22:27.007007+0000", 2024-09-13T11:22:38.924 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 11.899916007, 2024-09-13T11:22:38.924 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0026451309999999998, 2024-09-13T11:22:38.924 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:22:38.925 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:22:38.925 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:38.925 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:22:38.925 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.925 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:22:38.925 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.007007+0000", 2024-09-13T11:22:38.925 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:38.925 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.925 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.925 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:22:38.925 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.007007+0000", 2024-09-13T11:22:38.926 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:38.926 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.926 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.926 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:22:38.926 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.007008+0000", 2024-09-13T11:22:38.926 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0920000000000001e-06 2024-09-13T11:22:38.926 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.926 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.926 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:22:38.926 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.007117+0000", 2024-09-13T11:22:38.926 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000109406 2024-09-13T11:22:38.926 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.926 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.926 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:22:38.927 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.007144+0000", 2024-09-13T11:22:38.927 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.6877e-05 2024-09-13T11:22:38.927 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.927 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.927 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:22:38.927 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.007151+0000", 2024-09-13T11:22:38.927 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.2269999999999998e-06 2024-09-13T11:22:38.927 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.927 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.927 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:22:38.927 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.007210+0000", 2024-09-13T11:22:38.927 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.9596000000000003e-05 2024-09-13T11:22:38.927 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.927 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.928 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:22:38.928 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.007233+0000", 2024-09-13T11:22:38.928 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.2524e-05 2024-09-13T11:22:38.928 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.928 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.928 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:22:38.928 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.009606+0000", 2024-09-13T11:22:38.928 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0023731970000000001 2024-09-13T11:22:38.928 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.928 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.928 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:22:38.928 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.009652+0000", 2024-09-13T11:22:38.928 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.6211999999999997e-05 2024-09-13T11:22:38.929 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:38.929 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:22:38.929 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:38.929 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.929 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.929 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop_reply(client.4665.0:6 3.5 e32/30 ondisk, result = 0)", 2024-09-13T11:22:38.929 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:22:27.009479+0000", 2024-09-13T11:22:38.929 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 11.897444423, 2024-09-13T11:22:38.929 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000139455, 2024-09-13T11:22:38.929 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:22:38.929 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:22:38.929 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:22:38.929 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:22:38.929 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.930 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:22:38.930 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.009479+0000", 2024-09-13T11:22:38.930 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:38.930 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.930 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.930 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:22:38.930 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.009479+0000", 2024-09-13T11:22:38.930 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:38.930 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.930 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.930 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:22:38.930 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.009480+0000", 2024-09-13T11:22:38.930 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.751e-06 2024-09-13T11:22:38.930 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.931 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.931 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:22:38.931 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.009486+0000", 2024-09-13T11:22:38.931 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.3319999999999999e-06 2024-09-13T11:22:38.931 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.931 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.931 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:22:38.931 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.009518+0000", 2024-09-13T11:22:38.931 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.2174000000000001e-05 2024-09-13T11:22:38.931 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.931 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.931 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:22:38.932 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.009526+0000", 2024-09-13T11:22:38.932 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.8930000000000005e-06 2024-09-13T11:22:38.932 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.932 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.932 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:22:38.932 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.009592+0000", 2024-09-13T11:22:38.932 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.6234000000000003e-05 2024-09-13T11:22:38.932 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.932 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.932 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:22:38.932 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.009600+0000", 2024-09-13T11:22:38.932 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.3860000000000007e-06 2024-09-13T11:22:38.932 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.932 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.933 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:22:38.933 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.009618+0000", 2024-09-13T11:22:38.933 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.7685000000000001e-05 2024-09-13T11:22:38.933 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:38.933 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:22:38.933 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:38.933 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.933 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.933 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:28 3.0 e32/30 3:0c1f4d63:::benchmark_data_smithi177_31116_object27:head v 32'3, mlcod=32'1)", 2024-09-13T11:22:38.933 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:22:27.015184+0000", 2024-09-13T11:22:38.933 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 11.891739218, 2024-09-13T11:22:38.933 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00078898100000000001, 2024-09-13T11:22:38.933 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:22:38.933 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:22:38.934 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:38.934 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:22:38.934 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.934 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:22:38.934 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.015184+0000", 2024-09-13T11:22:38.934 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:38.934 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.934 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.934 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:22:38.934 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.015184+0000", 2024-09-13T11:22:38.934 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:38.934 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.934 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.934 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:22:38.935 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.015185+0000", 2024-09-13T11:22:38.935 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3459999999999999e-06 2024-09-13T11:22:38.935 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.935 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.935 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:22:38.935 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.015261+0000", 2024-09-13T11:22:38.935 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.6179000000000004e-05 2024-09-13T11:22:38.935 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.935 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.935 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:22:38.935 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.015307+0000", 2024-09-13T11:22:38.935 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.5946999999999999e-05 2024-09-13T11:22:38.935 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.935 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.936 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:22:38.936 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.015314+0000", 2024-09-13T11:22:38.936 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.9829999999999999e-06 2024-09-13T11:22:38.936 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.936 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.936 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:22:38.936 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.015361+0000", 2024-09-13T11:22:38.936 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.6597000000000001e-05 2024-09-13T11:22:38.936 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.936 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.936 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:22:38.936 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.015374+0000", 2024-09-13T11:22:38.936 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3166999999999999e-05 2024-09-13T11:22:38.937 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.937 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.937 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:22:38.937 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.015936+0000", 2024-09-13T11:22:38.937 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00056165000000000002 2024-09-13T11:22:38.937 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.937 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.937 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:22:38.937 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.015973+0000", 2024-09-13T11:22:38.937 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.7112e-05 2024-09-13T11:22:38.937 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:38.937 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:22:38.938 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:38.938 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.938 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.938 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:32 3.5 3:aeeb65b3:::benchmark_data_smithi177_31116_object31:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-09-13T11:22:38.938 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:22:27.015809+0000", 2024-09-13T11:22:38.938 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 11.891114059, 2024-09-13T11:22:38.938 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0027083530000000001, 2024-09-13T11:22:38.938 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:22:38.938 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:22:38.938 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:38.938 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:22:38.938 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:22:38.938 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:22:38.939 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 32 2024-09-13T11:22:38.939 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.939 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:22:38.939 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.939 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:22:38.939 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.015809+0000", 2024-09-13T11:22:38.939 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:38.939 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.939 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.939 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:22:38.939 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.015809+0000", 2024-09-13T11:22:38.939 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:38.939 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.939 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.940 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:22:38.940 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.015811+0000", 2024-09-13T11:22:38.940 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.249e-06 2024-09-13T11:22:38.940 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.940 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.940 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:22:38.940 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.015840+0000", 2024-09-13T11:22:38.940 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.8676000000000001e-05 2024-09-13T11:22:38.940 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.940 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.940 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:22:38.940 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.015850+0000", 2024-09-13T11:22:38.940 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0421000000000001e-05 2024-09-13T11:22:38.941 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.941 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.941 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:22:38.941 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.015859+0000", 2024-09-13T11:22:38.941 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.3559999999999993e-06 2024-09-13T11:22:38.941 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.941 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.941 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:22:38.941 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.015922+0000", 2024-09-13T11:22:38.941 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.2957e-05 2024-09-13T11:22:38.941 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.941 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.941 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:22:38.941 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.016064+0000", 2024-09-13T11:22:38.942 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000141958 2024-09-13T11:22:38.942 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.942 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.942 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 13", 2024-09-13T11:22:38.942 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.016220+0000", 2024-09-13T11:22:38.942 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000155995 2024-09-13T11:22:38.942 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.942 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.942 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:22:38.942 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.016961+0000", 2024-09-13T11:22:38.942 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000741116 2024-09-13T11:22:38.942 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.942 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.943 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:22:38.943 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.018330+0000", 2024-09-13T11:22:38.943 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001369585 2024-09-13T11:22:38.943 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.943 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.943 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:22:38.943 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.018387+0000", 2024-09-13T11:22:38.943 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.6354999999999998e-05 2024-09-13T11:22:38.943 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.943 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.943 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:22:38.943 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.018517+0000", 2024-09-13T11:22:38.943 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00013068500000000001 2024-09-13T11:22:38.943 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:38.944 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:22:38.944 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:38.944 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.944 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.944 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop_reply(client.4665.0:32 3.5 e32/30 ondisk, result = 0)", 2024-09-13T11:22:38.944 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:22:27.018216+0000", 2024-09-13T11:22:38.944 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 11.888706646999999, 2024-09-13T11:22:38.944 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00031082000000000001, 2024-09-13T11:22:38.944 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:22:38.944 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:22:38.944 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:22:38.944 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:22:38.944 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.945 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:22:38.945 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.018216+0000", 2024-09-13T11:22:38.945 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:38.945 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.945 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.945 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:22:38.945 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.018216+0000", 2024-09-13T11:22:38.945 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:38.945 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.945 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.945 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:22:38.945 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.018218+0000", 2024-09-13T11:22:38.945 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.359e-06 2024-09-13T11:22:38.945 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.946 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.946 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:22:38.946 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.018223+0000", 2024-09-13T11:22:38.946 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.6760000000000004e-06 2024-09-13T11:22:38.946 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.946 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.946 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:22:38.946 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.018253+0000", 2024-09-13T11:22:38.946 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.9524000000000001e-05 2024-09-13T11:22:38.946 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.946 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.946 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:22:38.946 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.018259+0000", 2024-09-13T11:22:38.946 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.9490000000000001e-06 2024-09-13T11:22:38.947 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.947 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.947 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:22:38.947 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.018314+0000", 2024-09-13T11:22:38.947 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.5460000000000001e-05 2024-09-13T11:22:38.947 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.947 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.947 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:22:38.947 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.018322+0000", 2024-09-13T11:22:38.947 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.1149999999999994e-06 2024-09-13T11:22:38.947 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.947 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.947 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:22:38.947 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.018527+0000", 2024-09-13T11:22:38.948 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000204737 2024-09-13T11:22:38.948 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:38.948 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:22:38.948 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:38.948 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.948 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.948 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:42 3.0 e32/30 3:09c3a999:::benchmark_data_smithi177_31116_object41:head v 32'4, mlcod=32'2)", 2024-09-13T11:22:38.948 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:22:27.021491+0000", 2024-09-13T11:22:38.948 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 11.885431922, 2024-09-13T11:22:38.948 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00093548800000000005, 2024-09-13T11:22:38.948 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:22:38.948 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:22:38.948 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:38.948 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:22:38.949 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.949 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:22:38.949 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.021491+0000", 2024-09-13T11:22:38.949 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:38.949 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.949 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.949 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:22:38.949 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.021491+0000", 2024-09-13T11:22:38.949 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:38.949 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.949 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.949 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:22:38.949 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.021492+0000", 2024-09-13T11:22:38.949 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3850000000000001e-06 2024-09-13T11:22:38.950 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.950 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.950 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:22:38.950 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.021563+0000", 2024-09-13T11:22:38.950 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.1009999999999999e-05 2024-09-13T11:22:38.950 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.950 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.950 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:22:38.950 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.021592+0000", 2024-09-13T11:22:38.950 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.8311e-05 2024-09-13T11:22:38.950 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.950 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.950 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:22:38.951 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.021599+0000", 2024-09-13T11:22:38.951 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.9720000000000003e-06 2024-09-13T11:22:38.951 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.951 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.951 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:22:38.951 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.021646+0000", 2024-09-13T11:22:38.951 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.7576e-05 2024-09-13T11:22:38.952 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.953 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.953 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:22:38.953 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.021665+0000", 2024-09-13T11:22:38.953 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.8516999999999999e-05 2024-09-13T11:22:38.953 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.953 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.953 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:22:38.953 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.022384+0000", 2024-09-13T11:22:38.953 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00071960399999999999 2024-09-13T11:22:38.953 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.953 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.953 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:22:38.953 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.022427+0000", 2024-09-13T11:22:38.954 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.2113000000000002e-05 2024-09-13T11:22:38.954 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:38.954 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:22:38.954 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:38.954 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.954 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.954 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:48 3.5 3:afe1b749:::benchmark_data_smithi177_31116_object47:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-09-13T11:22:38.954 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:22:27.023008+0000", 2024-09-13T11:22:38.954 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 11.883915305, 2024-09-13T11:22:38.954 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0017631680000000001, 2024-09-13T11:22:38.954 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:22:38.954 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:22:38.954 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:38.954 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:22:38.955 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:22:38.955 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:22:38.955 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 48 2024-09-13T11:22:38.955 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.955 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:22:38.955 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.955 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:22:38.955 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.023008+0000", 2024-09-13T11:22:38.955 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:38.955 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.955 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.955 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:22:38.955 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.023008+0000", 2024-09-13T11:22:38.955 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:38.956 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.956 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.956 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:22:38.956 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.023010+0000", 2024-09-13T11:22:38.956 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.198e-06 2024-09-13T11:22:38.956 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.956 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.956 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:22:38.956 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.023043+0000", 2024-09-13T11:22:38.956 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.3185000000000002e-05 2024-09-13T11:22:38.956 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.956 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.956 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:22:38.956 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.023055+0000", 2024-09-13T11:22:38.957 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.2340999999999999e-05 2024-09-13T11:22:38.957 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.957 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.957 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:22:38.957 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.023062+0000", 2024-09-13T11:22:38.957 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.3069999999999999e-06 2024-09-13T11:22:38.957 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.957 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.957 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:22:38.957 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.023119+0000", 2024-09-13T11:22:38.957 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.7556999999999998e-05 2024-09-13T11:22:38.957 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.957 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.958 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:22:38.958 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.023253+0000", 2024-09-13T11:22:38.958 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00013351899999999999 2024-09-13T11:22:38.958 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.958 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.958 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 13", 2024-09-13T11:22:38.958 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.023390+0000", 2024-09-13T11:22:38.958 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000137637 2024-09-13T11:22:38.958 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.958 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.958 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:22:38.958 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.024635+0000", 2024-09-13T11:22:38.958 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0012449539999999999 2024-09-13T11:22:38.959 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.959 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.959 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:22:38.959 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.024659+0000", 2024-09-13T11:22:38.959 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.3598e-05 2024-09-13T11:22:38.959 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.959 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.959 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:22:38.959 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.024721+0000", 2024-09-13T11:22:38.959 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.1558999999999998e-05 2024-09-13T11:22:38.959 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.959 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.959 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:22:38.959 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.024771+0000", 2024-09-13T11:22:38.960 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.0312999999999998e-05 2024-09-13T11:22:38.960 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:38.960 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:22:38.960 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:38.960 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.960 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.960 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop_reply(client.4665.0:48 3.5 e32/30 ondisk, result = 0)", 2024-09-13T11:22:38.960 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:22:27.024499+0000", 2024-09-13T11:22:38.960 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 11.882424207, 2024-09-13T11:22:38.960 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000146129, 2024-09-13T11:22:38.960 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:22:38.960 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:22:38.961 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:22:38.961 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:22:38.961 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.961 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:22:38.961 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.024499+0000", 2024-09-13T11:22:38.961 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:38.961 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.961 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.961 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:22:38.961 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.024499+0000", 2024-09-13T11:22:38.961 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:38.961 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.961 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.961 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:22:38.962 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.024501+0000", 2024-09-13T11:22:38.962 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.8950000000000001e-06 2024-09-13T11:22:38.962 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.962 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.962 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:22:38.962 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.024506+0000", 2024-09-13T11:22:38.962 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.6239999999999999e-06 2024-09-13T11:22:38.962 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.962 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.962 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:22:38.962 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.024545+0000", 2024-09-13T11:22:38.962 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.8318999999999997e-05 2024-09-13T11:22:38.962 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.962 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.963 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:22:38.963 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.024554+0000", 2024-09-13T11:22:38.963 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.9940000000000006e-06 2024-09-13T11:22:38.963 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.963 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.963 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:22:38.963 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.024623+0000", 2024-09-13T11:22:38.963 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.9721999999999994e-05 2024-09-13T11:22:38.963 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.963 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.963 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:22:38.963 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.024632+0000", 2024-09-13T11:22:38.963 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.3270000000000001e-06 2024-09-13T11:22:38.963 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.963 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.964 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:22:38.964 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.024645+0000", 2024-09-13T11:22:38.964 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3247999999999999e-05 2024-09-13T11:22:38.964 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:38.964 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:22:38.964 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:38.964 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.964 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.964 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:61 3.5 3:a7daa87f:::benchmark_data_smithi177_31116_object60:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-09-13T11:22:38.964 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:22:27.031272+0000", 2024-09-13T11:22:38.964 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 11.875650953999999, 2024-09-13T11:22:38.964 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0016824229999999999, 2024-09-13T11:22:38.964 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:22:38.965 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:22:38.965 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:38.965 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:22:38.965 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:22:38.965 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:22:38.965 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 61 2024-09-13T11:22:38.965 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.965 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:22:38.965 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.965 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:22:38.965 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.031272+0000", 2024-09-13T11:22:38.965 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:38.965 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.965 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.965 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:22:38.966 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.031272+0000", 2024-09-13T11:22:38.966 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:38.966 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.966 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.966 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:22:38.966 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.031274+0000", 2024-09-13T11:22:38.966 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.4650000000000001e-06 2024-09-13T11:22:38.966 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.966 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.966 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:22:38.966 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.031318+0000", 2024-09-13T11:22:38.966 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.3291999999999998e-05 2024-09-13T11:22:38.966 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.966 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.967 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:22:38.967 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.031337+0000", 2024-09-13T11:22:38.967 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.8807000000000001e-05 2024-09-13T11:22:38.967 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.967 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.967 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:22:38.967 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.031345+0000", 2024-09-13T11:22:38.967 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.6449999999999994e-06 2024-09-13T11:22:38.967 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.967 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.967 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:22:38.967 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.031402+0000", 2024-09-13T11:22:38.967 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.6329000000000002e-05 2024-09-13T11:22:38.968 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.968 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.968 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:22:38.968 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.031534+0000", 2024-09-13T11:22:38.968 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000132581 2024-09-13T11:22:38.968 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.968 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.968 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 13", 2024-09-13T11:22:38.968 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.031679+0000", 2024-09-13T11:22:38.968 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000144874 2024-09-13T11:22:38.968 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.968 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.968 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:22:38.968 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.032571+0000", 2024-09-13T11:22:38.969 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00089154900000000005 2024-09-13T11:22:38.969 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.969 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.969 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:22:38.969 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.032838+0000", 2024-09-13T11:22:38.969 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00026793800000000001 2024-09-13T11:22:38.969 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.969 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.969 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:22:38.969 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.032896+0000", 2024-09-13T11:22:38.969 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.7126000000000002e-05 2024-09-13T11:22:38.969 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.969 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.970 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:22:38.970 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.032954+0000", 2024-09-13T11:22:38.970 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.8817000000000003e-05 2024-09-13T11:22:38.970 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:38.970 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:22:38.970 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:38.970 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.970 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.970 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop_reply(client.4665.0:61 3.5 e32/30 ondisk, result = 0)", 2024-09-13T11:22:38.970 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:22:27.032729+0000", 2024-09-13T11:22:38.970 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 11.874193503000001, 2024-09-13T11:22:38.970 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00023284799999999999, 2024-09-13T11:22:38.970 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:22:38.971 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:22:38.971 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:22:38.971 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:22:38.971 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.971 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:22:38.971 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.032729+0000", 2024-09-13T11:22:38.971 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:38.971 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.971 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.971 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:22:38.971 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.032729+0000", 2024-09-13T11:22:38.971 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:38.971 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.971 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.972 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:22:38.972 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.032731+0000", 2024-09-13T11:22:38.972 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.545e-06 2024-09-13T11:22:38.972 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.972 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.972 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:22:38.972 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.032735+0000", 2024-09-13T11:22:38.972 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.4700000000000004e-06 2024-09-13T11:22:38.972 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.972 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.972 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:22:38.972 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.032761+0000", 2024-09-13T11:22:38.972 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.5360000000000001e-05 2024-09-13T11:22:38.973 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.973 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.973 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:22:38.973 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.032770+0000", 2024-09-13T11:22:38.973 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.2229999999999996e-06 2024-09-13T11:22:38.973 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.973 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.973 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:22:38.973 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.032822+0000", 2024-09-13T11:22:38.973 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.2062999999999999e-05 2024-09-13T11:22:38.973 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.973 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.973 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:22:38.974 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.032830+0000", 2024-09-13T11:22:38.974 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.2830000000000001e-06 2024-09-13T11:22:38.974 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.974 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.974 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:22:38.974 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.032962+0000", 2024-09-13T11:22:38.974 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00013190399999999999 2024-09-13T11:22:38.974 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:38.974 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:22:38.974 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:38.974 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.974 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.975 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:65 3.0 e32/30 3:01909ae6:::benchmark_data_smithi177_31116_object64:head v 32'5, mlcod=32'3)", 2024-09-13T11:22:38.975 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:22:27.032982+0000", 2024-09-13T11:22:38.975 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 11.873941432000001, 2024-09-13T11:22:38.975 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001094446, 2024-09-13T11:22:38.975 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:22:38.975 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:22:38.975 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:38.975 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:22:38.975 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.975 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:22:38.975 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.032982+0000", 2024-09-13T11:22:38.975 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:38.975 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.975 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.976 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:22:38.976 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.032982+0000", 2024-09-13T11:22:38.976 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:38.976 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.976 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.976 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:22:38.976 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.032983+0000", 2024-09-13T11:22:38.976 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.7460000000000001e-06 2024-09-13T11:22:38.976 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.976 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.976 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:22:38.976 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.033136+0000", 2024-09-13T11:22:38.976 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00015317099999999999 2024-09-13T11:22:38.977 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.977 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.977 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:22:38.977 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.033173+0000", 2024-09-13T11:22:38.977 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.6625000000000003e-05 2024-09-13T11:22:38.977 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.977 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.977 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:22:38.977 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.033182+0000", 2024-09-13T11:22:38.977 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.7490000000000004e-06 2024-09-13T11:22:38.977 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.978 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.978 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:22:38.978 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.033252+0000", 2024-09-13T11:22:38.978 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.9765999999999996e-05 2024-09-13T11:22:38.978 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.978 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.978 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:22:38.978 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.033275+0000", 2024-09-13T11:22:38.978 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.3563000000000001e-05 2024-09-13T11:22:38.978 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.978 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.978 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:22:38.978 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.034032+0000", 2024-09-13T11:22:38.978 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00075666100000000005 2024-09-13T11:22:38.979 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.979 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.979 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:22:38.979 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.034076+0000", 2024-09-13T11:22:38.979 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.4165000000000002e-05 2024-09-13T11:22:38.979 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:38.979 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:22:38.979 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:38.979 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.979 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.979 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:69 3.5 3:ad4cd57a:::benchmark_data_smithi177_31116_object68:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-09-13T11:22:38.979 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:22:27.034819+0000", 2024-09-13T11:22:38.980 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 11.872104236, 2024-09-13T11:22:38.980 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0024809630000000001, 2024-09-13T11:22:38.980 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:22:38.980 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:22:38.980 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:38.980 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:22:38.980 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:22:38.980 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:22:38.980 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 69 2024-09-13T11:22:38.980 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.980 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:22:38.980 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.980 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:22:38.980 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.034819+0000", 2024-09-13T11:22:38.981 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:38.981 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.981 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.981 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:22:38.981 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.034819+0000", 2024-09-13T11:22:38.981 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:38.981 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.981 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.981 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:22:38.981 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.034820+0000", 2024-09-13T11:22:38.981 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.6819999999999999e-06 2024-09-13T11:22:38.981 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.981 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.981 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:22:38.982 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.034871+0000", 2024-09-13T11:22:38.982 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.0154000000000001e-05 2024-09-13T11:22:38.982 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.982 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.982 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:22:38.982 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.034883+0000", 2024-09-13T11:22:38.982 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.2923e-05 2024-09-13T11:22:38.982 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.982 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.982 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:22:38.982 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.034890+0000", 2024-09-13T11:22:38.982 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.7109999999999998e-06 2024-09-13T11:22:38.982 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.982 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.983 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:22:38.983 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.034945+0000", 2024-09-13T11:22:38.983 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.4824999999999998e-05 2024-09-13T11:22:38.983 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.983 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.983 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:22:38.983 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.035081+0000", 2024-09-13T11:22:38.983 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00013631299999999999 2024-09-13T11:22:38.983 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.983 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.983 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 13", 2024-09-13T11:22:38.983 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.035246+0000", 2024-09-13T11:22:38.983 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00016472899999999999 2024-09-13T11:22:38.983 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.984 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.984 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:22:38.984 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.035975+0000", 2024-09-13T11:22:38.984 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00072935600000000001 2024-09-13T11:22:38.984 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.984 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.984 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:22:38.984 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.037236+0000", 2024-09-13T11:22:38.984 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0012610849999999999 2024-09-13T11:22:38.984 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.984 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.984 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:22:38.984 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.037270+0000", 2024-09-13T11:22:38.984 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.3386999999999997e-05 2024-09-13T11:22:38.985 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.985 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.985 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:22:38.985 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.037300+0000", 2024-09-13T11:22:38.985 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.9798000000000002e-05 2024-09-13T11:22:38.985 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:38.985 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:22:38.985 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:38.985 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.985 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.985 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop_reply(client.4665.0:69 3.5 e32/30 ondisk, result = 0)", 2024-09-13T11:22:38.985 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:22:27.037136+0000", 2024-09-13T11:22:38.985 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 11.869787056, 2024-09-13T11:22:38.985 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00016980399999999999, 2024-09-13T11:22:38.986 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:22:38.986 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:22:38.986 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:22:38.986 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:22:38.986 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.986 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:22:38.986 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.037136+0000", 2024-09-13T11:22:38.986 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:38.986 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.986 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.986 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:22:38.986 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.037136+0000", 2024-09-13T11:22:38.986 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:38.986 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.987 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.987 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:22:38.987 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.037137+0000", 2024-09-13T11:22:38.987 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5260000000000001e-06 2024-09-13T11:22:38.987 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.987 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.987 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:22:38.987 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.037145+0000", 2024-09-13T11:22:38.987 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.1539999999999996e-06 2024-09-13T11:22:38.987 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.987 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.987 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:22:38.987 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.037172+0000", 2024-09-13T11:22:38.987 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.7628e-05 2024-09-13T11:22:38.987 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.988 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.988 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:22:38.988 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.037189+0000", 2024-09-13T11:22:38.988 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.6364000000000001e-05 2024-09-13T11:22:38.988 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.988 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.988 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:22:38.988 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.037228+0000", 2024-09-13T11:22:38.988 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.9919999999999997e-05 2024-09-13T11:22:38.988 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.988 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.988 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:22:38.988 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.037233+0000", 2024-09-13T11:22:38.988 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.9239999999999998e-06 2024-09-13T11:22:38.989 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:38.989 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:38.989 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:22:38.989 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.037306+0000", 2024-09-13T11:22:38.989 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.2287999999999996e-05 2024-09-13T11:22:38.989 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:38.989 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:22:38.989 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:38.989 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:38.989 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:22:38.989 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-09-13T11:22:38.991 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-13T11:22:38.991 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-13T11:22:38.984+0000 7f6a68775640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26307) UID: 0 2024-09-13T11:22:38.997 INFO:tasks.radosbench.radosbench.0.smithi177.stdout: 12 16 87 71 0.369743 0 - 0.0234801 2024-09-13T11:22:39.091 INFO:tasks.ceph.osd.7.smithi177.stderr:2024-09-13T11:22:39.088+0000 7f897b51d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26575) UID: 0 2024-09-13T11:22:39.125 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-13T11:22:39.125 INFO:teuthology.orchestra.run.smithi001.stdout: "ops": [], 2024-09-13T11:22:39.125 INFO:teuthology.orchestra.run.smithi001.stdout: "num_ops": 0 2024-09-13T11:22:39.125 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-13T11:22:39.140 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-13T11:22:39.193 INFO:tasks.ceph.osd.11.smithi177.stderr:2024-09-13T11:22:39.188+0000 7f24792c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26580) UID: 0 2024-09-13T11:22:39.233 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-13T11:22:39.260 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-13T11:22:39.260 INFO:teuthology.orchestra.run.smithi001.stdout: "ops": [], 2024-09-13T11:22:39.260 INFO:teuthology.orchestra.run.smithi001.stdout: "complaint_time": 30, 2024-09-13T11:22:39.260 INFO:teuthology.orchestra.run.smithi001.stdout: "num_blocked_ops": 0 2024-09-13T11:22:39.260 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-13T11:22:39.275 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-13T11:22:39.294 INFO:tasks.ceph.osd.7.smithi177.stderr:2024-09-13T11:22:39.288+0000 7f897b51d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26575) UID: 0 2024-09-13T11:22:39.368 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T11:22:39.395 INFO:tasks.ceph.osd.3.smithi177.stderr:2024-09-13T11:22:39.396+0000 7f10adba4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26574) UID: 0 2024-09-13T11:22:39.446 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-13T11:22:39.446 INFO:teuthology.orchestra.run.smithi001.stdout: "size": 20, 2024-09-13T11:22:39.447 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 600, 2024-09-13T11:22:39.447 INFO:teuthology.orchestra.run.smithi001.stdout: "ops": [ 2024-09-13T11:22:39.447 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:39.447 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4257.0:20 1.0 e24/23 1:30306672:devicehealth::main.db.0000000000000000:head v 24'10, mlcod=24'8)", 2024-09-13T11:22:39.447 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:22:12.167115+0000", 2024-09-13T11:22:39.447 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 27.277020574000002, 2024-09-13T11:22:39.447 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0022173010000000001, 2024-09-13T11:22:39.447 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:22:39.447 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:22:39.447 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:39.447 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:22:39.447 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:39.447 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:22:39.448 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:12.167115+0000", 2024-09-13T11:22:39.448 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:22:39.448 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:39.448 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:39.448 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:22:39.448 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:12.167115+0000", 2024-09-13T11:22:39.448 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:22:39.448 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:39.448 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:39.448 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:22:39.448 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:12.167117+0000", 2024-09-13T11:22:39.448 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.866e-06 2024-09-13T11:22:39.448 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:39.449 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:39.449 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:22:39.449 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:12.167367+0000", 2024-09-13T11:22:39.449 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00024982599999999998 2024-09-13T11:22:39.449 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:39.449 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:39.449 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:22:39.449 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:12.167415+0000", 2024-09-13T11:22:39.449 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.8473000000000002e-05 2024-09-13T11:22:39.449 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:39.449 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:39.449 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:22:39.450 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:12.167426+0000", 2024-09-13T11:22:39.450 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.1234e-05 2024-09-13T11:22:39.450 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:39.450 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:39.450 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:22:39.450 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:12.167563+0000", 2024-09-13T11:22:39.450 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000136613 2024-09-13T11:22:39.450 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:39.450 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:39.450 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:22:39.450 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:12.167602+0000", 2024-09-13T11:22:39.450 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.9236999999999997e-05 2024-09-13T11:22:39.450 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:39.451 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:39.451 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:22:39.451 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:12.169240+0000", 2024-09-13T11:22:39.451 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.001637854 2024-09-13T11:22:39.451 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:39.451 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:39.451 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:22:39.451 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:12.169332+0000", 2024-09-13T11:22:39.451 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.2198000000000001e-05 2024-09-13T11:22:39.451 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:22:39.451 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:22:39.451 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:22:39.451 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:39.451 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:39.452 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4257.0:21 1.0 e24/23 1:30306672:devicehealth::main.db.0000000000000000:head v 24'11, mlcod=24'8)", 2024-09-13T11:22:39.452 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:22:12.168401+0000", 2024-09-13T11:22:39.452 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 27.275734760999999, 2024-09-13T11:22:39.452 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0022842819999999999, 2024-09-13T11:22:39.452 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:22:39.452 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:22:39.452 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:39.452 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:22:39.452 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:39.452 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:22:39.452 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:12.168401+0000", 2024-09-13T11:22:39.452 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:22:39.453 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:39.453 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:39.453 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:22:39.453 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:12.168401+0000", 2024-09-13T11:22:39.453 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:22:39.453 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:39.453 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:39.453 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:22:39.453 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:12.168404+0000", 2024-09-13T11:22:39.453 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.9859999999999999e-06 2024-09-13T11:22:39.453 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:39.453 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:39.453 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:22:39.454 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:12.168562+0000", 2024-09-13T11:22:39.454 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00015851400000000001 2024-09-13T11:22:39.454 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:39.454 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:39.454 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:22:39.454 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:12.168634+0000", 2024-09-13T11:22:39.454 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.1737999999999997e-05 2024-09-13T11:22:39.454 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:39.454 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:39.454 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:22:39.454 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:12.168654+0000", 2024-09-13T11:22:39.454 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.0350999999999999e-05 2024-09-13T11:22:39.454 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:39.455 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:39.455 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:22:39.455 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:12.168801+0000", 2024-09-13T11:22:39.455 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00014716999999999999 2024-09-13T11:22:39.455 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:39.455 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:39.455 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:22:39.455 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:12.168836+0000", 2024-09-13T11:22:39.455 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.4184999999999999e-05 2024-09-13T11:22:39.455 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:39.455 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:39.455 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:22:39.455 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:12.170571+0000", 2024-09-13T11:22:39.455 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.001735826 2024-09-13T11:22:39.456 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:39.456 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:39.456 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:22:39.456 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:12.170685+0000", 2024-09-13T11:22:39.456 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00011351199999999999 2024-09-13T11:22:39.456 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:22:39.456 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:22:39.456 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:22:39.456 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:39.456 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:39.456 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4257.0:64 1.0 e24/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 24'42, mlcod=24'40)", 2024-09-13T11:22:39.456 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:22:12.455922+0000", 2024-09-13T11:22:39.456 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 26.988213107, 2024-09-13T11:22:39.457 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0025220899999999998, 2024-09-13T11:22:39.457 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:22:39.457 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:22:39.457 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:39.457 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:22:39.457 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:39.457 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:22:39.457 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:12.455922+0000", 2024-09-13T11:22:39.457 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:22:39.457 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:39.457 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:39.457 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:22:39.457 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:12.455922+0000", 2024-09-13T11:22:39.457 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:22:39.458 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:39.458 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:39.458 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:22:39.458 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:12.455925+0000", 2024-09-13T11:22:39.458 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.1130000000000001e-06 2024-09-13T11:22:39.458 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:39.458 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:39.458 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:22:39.458 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:12.455948+0000", 2024-09-13T11:22:39.458 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.2133e-05 2024-09-13T11:22:39.458 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:39.458 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:39.458 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:22:39.458 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:12.456021+0000", 2024-09-13T11:22:39.459 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.3489000000000001e-05 2024-09-13T11:22:39.459 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:39.459 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:39.459 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:22:39.459 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:12.456034+0000", 2024-09-13T11:22:39.459 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.3353000000000001e-05 2024-09-13T11:22:39.459 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:39.459 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:39.459 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:22:39.459 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:12.456154+0000", 2024-09-13T11:22:39.459 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00012002699999999999 2024-09-13T11:22:39.459 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:39.459 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:39.459 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:22:39.460 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:12.456191+0000", 2024-09-13T11:22:39.460 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.6505999999999999e-05 2024-09-13T11:22:39.460 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:39.460 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:39.460 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:22:39.460 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:12.458364+0000", 2024-09-13T11:22:39.460 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0021730740000000001 2024-09-13T11:22:39.460 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:39.460 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:39.460 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:22:39.460 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:12.458444+0000", 2024-09-13T11:22:39.460 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.0395000000000005e-05 2024-09-13T11:22:39.460 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:22:39.461 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:22:39.461 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:22:39.461 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:39.461 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:39.461 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4257.0:75 1.0 e24/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 24'53, mlcod=24'49)", 2024-09-13T11:22:39.461 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:22:12.503829+0000", 2024-09-13T11:22:39.461 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 26.940306305, 2024-09-13T11:22:39.461 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0020510960000000001, 2024-09-13T11:22:39.461 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:22:39.461 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:22:39.461 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:39.461 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:22:39.461 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:39.461 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:22:39.462 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:12.503829+0000", 2024-09-13T11:22:39.462 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:22:39.462 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:39.462 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:39.462 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:22:39.462 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:12.503829+0000", 2024-09-13T11:22:39.462 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:22:39.462 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:39.462 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:39.462 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:22:39.462 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:12.503832+0000", 2024-09-13T11:22:39.462 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.7599999999999998e-06 2024-09-13T11:22:39.462 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:39.463 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:39.463 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:22:39.463 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:12.503922+0000", 2024-09-13T11:22:39.463 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.0035000000000001e-05 2024-09-13T11:22:39.463 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:39.463 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:39.463 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:22:39.463 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:12.503980+0000", 2024-09-13T11:22:39.463 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.8421999999999997e-05 2024-09-13T11:22:39.463 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:39.463 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:39.463 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:22:39.463 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:12.503993+0000", 2024-09-13T11:22:39.463 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.313e-05 2024-09-13T11:22:39.464 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:39.464 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:39.464 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:22:39.464 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:12.504099+0000", 2024-09-13T11:22:39.464 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000105624 2024-09-13T11:22:39.464 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:39.464 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:39.464 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:22:39.464 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:12.504138+0000", 2024-09-13T11:22:39.464 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.8736000000000003e-05 2024-09-13T11:22:39.464 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:39.464 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:39.464 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:22:39.465 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:12.505801+0000", 2024-09-13T11:22:39.465 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.001662981 2024-09-13T11:22:39.465 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:39.465 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:39.465 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:22:39.465 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:12.505880+0000", 2024-09-13T11:22:39.465 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.9407999999999996e-05 2024-09-13T11:22:39.465 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:22:39.465 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:22:39.465 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:22:39.465 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:39.465 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:39.465 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4257.0:77 1.0 e24/23 1:30306672:devicehealth::main.db.0000000000000000:head v 24'55, mlcod=24'53)", 2024-09-13T11:22:39.466 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:22:12.514870+0000", 2024-09-13T11:22:39.466 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 26.92926533, 2024-09-13T11:22:39.466 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0022810920000000002, 2024-09-13T11:22:39.466 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:22:39.466 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:22:39.466 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:39.466 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:22:39.466 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:39.466 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:22:39.466 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:12.514870+0000", 2024-09-13T11:22:39.466 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:22:39.466 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:39.466 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:39.467 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:22:39.467 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:12.514870+0000", 2024-09-13T11:22:39.467 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:22:39.467 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:39.467 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:39.467 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:22:39.467 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:12.514873+0000", 2024-09-13T11:22:39.467 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.2839999999999998e-06 2024-09-13T11:22:39.467 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:39.467 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:39.467 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:22:39.467 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:12.514924+0000", 2024-09-13T11:22:39.467 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.1001999999999997e-05 2024-09-13T11:22:39.468 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:39.468 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:39.468 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:22:39.468 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:12.514988+0000", 2024-09-13T11:22:39.468 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.3218000000000004e-05 2024-09-13T11:22:39.468 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:39.468 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:39.468 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:22:39.468 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:12.515001+0000", 2024-09-13T11:22:39.468 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.3502999999999999e-05 2024-09-13T11:22:39.468 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:39.468 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:39.469 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:22:39.469 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:12.515130+0000", 2024-09-13T11:22:39.469 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00012927199999999999 2024-09-13T11:22:39.469 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:39.469 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:39.469 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:22:39.469 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:12.515167+0000", 2024-09-13T11:22:39.469 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.6259e-05 2024-09-13T11:22:39.469 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:39.469 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:39.469 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:22:39.469 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:12.517017+0000", 2024-09-13T11:22:39.469 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0018506270000000001 2024-09-13T11:22:39.470 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:39.470 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:39.470 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:22:39.470 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:12.517151+0000", 2024-09-13T11:22:39.470 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00013392699999999999 2024-09-13T11:22:39.470 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:22:39.470 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:22:39.470 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:22:39.470 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:39.470 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:39.470 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4257.0:78 1.0 e24/23 1:30306672:devicehealth::main.db.0000000000000000:head v 24'56, mlcod=24'53)", 2024-09-13T11:22:39.470 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:22:12.516252+0000", 2024-09-13T11:22:39.470 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 26.927883449999999, 2024-09-13T11:22:39.470 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0023671, 2024-09-13T11:22:39.471 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:22:39.471 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:22:39.471 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:39.471 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:22:39.471 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:39.471 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:22:39.471 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:12.516252+0000", 2024-09-13T11:22:39.471 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:22:39.471 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:39.471 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:39.471 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:22:39.471 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:12.516252+0000", 2024-09-13T11:22:39.471 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:22:39.472 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:39.472 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:39.472 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:22:39.472 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:12.516255+0000", 2024-09-13T11:22:39.472 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.993e-06 2024-09-13T11:22:39.472 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:39.472 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:39.472 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:22:39.472 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:12.516342+0000", 2024-09-13T11:22:39.472 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.7447999999999994e-05 2024-09-13T11:22:39.472 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:39.472 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:39.472 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:22:39.473 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:12.516418+0000", 2024-09-13T11:22:39.473 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.5492000000000002e-05 2024-09-13T11:22:39.473 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:39.473 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:39.473 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:22:39.473 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:12.516442+0000", 2024-09-13T11:22:39.473 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.4105999999999999e-05 2024-09-13T11:22:39.473 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:39.473 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:39.473 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:22:39.473 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:12.516540+0000", 2024-09-13T11:22:39.473 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.7502000000000003e-05 2024-09-13T11:22:39.473 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:39.474 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:39.474 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:22:39.474 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:12.516571+0000", 2024-09-13T11:22:39.474 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.1285000000000003e-05 2024-09-13T11:22:39.474 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:39.474 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:39.474 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:22:39.474 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:12.518498+0000", 2024-09-13T11:22:39.474 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0019272759999999999 2024-09-13T11:22:39.474 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:39.474 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:39.474 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:22:39.475 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:12.518619+0000", 2024-09-13T11:22:39.475 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000120998 2024-09-13T11:22:39.475 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:22:39.475 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:22:39.475 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:22:39.475 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:39.475 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:39.475 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4257.0:101 1.0 e24/23 1:30306672:devicehealth::main.db.0000000000000000:head v 24'75, mlcod=24'73)", 2024-09-13T11:22:39.475 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:22:13.092579+0000", 2024-09-13T11:22:39.475 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 26.351556510999998, 2024-09-13T11:22:39.475 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0025088380000000002, 2024-09-13T11:22:39.475 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:22:39.475 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:22:39.476 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:39.476 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:22:39.476 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:39.476 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:22:39.476 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:13.092579+0000", 2024-09-13T11:22:39.476 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:22:39.476 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:39.476 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:39.476 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:22:39.476 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:13.092579+0000", 2024-09-13T11:22:39.476 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:22:39.476 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:39.476 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:39.477 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:22:39.477 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:13.092582+0000", 2024-09-13T11:22:39.477 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.9569999999999998e-06 2024-09-13T11:22:39.477 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:39.477 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:39.477 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:22:39.477 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:13.092638+0000", 2024-09-13T11:22:39.477 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.5812e-05 2024-09-13T11:22:39.477 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:39.477 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:39.477 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:22:39.477 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:13.092701+0000", 2024-09-13T11:22:39.477 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.2866000000000004e-05 2024-09-13T11:22:39.477 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:39.478 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:39.478 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:22:39.478 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:13.092714+0000", 2024-09-13T11:22:39.478 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.3533000000000001e-05 2024-09-13T11:22:39.478 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:39.478 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:39.478 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:22:39.478 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:13.092798+0000", 2024-09-13T11:22:39.478 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.4132e-05 2024-09-13T11:22:39.478 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:39.478 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:39.478 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:22:39.478 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:13.092828+0000", 2024-09-13T11:22:39.479 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.9349e-05 2024-09-13T11:22:39.479 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:39.479 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:39.479 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:22:39.479 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:13.095016+0000", 2024-09-13T11:22:39.479 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.002188307 2024-09-13T11:22:39.479 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:39.479 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:39.479 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:22:39.480 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:13.095088+0000", 2024-09-13T11:22:39.480 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.1882000000000001e-05 2024-09-13T11:22:39.480 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:22:39.480 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:22:39.480 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:22:39.480 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:39.480 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:39.480 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4257.0:104 1.0 e24/23 1:30306672:devicehealth::main.db.0000000000000000:head v 24'78, mlcod=24'73)", 2024-09-13T11:22:39.480 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:22:13.094467+0000", 2024-09-13T11:22:39.480 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 26.349668140999999, 2024-09-13T11:22:39.480 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0020875860000000002, 2024-09-13T11:22:39.480 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:22:39.480 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:22:39.481 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:39.481 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:22:39.481 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:39.481 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:22:39.481 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:13.094467+0000", 2024-09-13T11:22:39.481 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:22:39.481 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:39.481 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:39.481 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:22:39.481 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:13.094467+0000", 2024-09-13T11:22:39.481 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:22:39.481 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:39.481 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:39.482 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:22:39.482 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:13.094470+0000", 2024-09-13T11:22:39.482 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.9229999999999998e-06 2024-09-13T11:22:39.482 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:39.482 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:39.482 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:22:39.482 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:13.094657+0000", 2024-09-13T11:22:39.482 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00018684600000000001 2024-09-13T11:22:39.482 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:39.482 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:39.482 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:22:39.482 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:13.094753+0000", 2024-09-13T11:22:39.482 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.6021999999999996e-05 2024-09-13T11:22:39.483 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:39.483 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:39.483 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:22:39.483 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:13.094771+0000", 2024-09-13T11:22:39.483 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.7493e-05 2024-09-13T11:22:39.483 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:39.483 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:39.483 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:22:39.483 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:13.095247+0000", 2024-09-13T11:22:39.483 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00047664400000000003 2024-09-13T11:22:39.483 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:39.483 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:39.483 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:22:39.484 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:13.095292+0000", 2024-09-13T11:22:39.484 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.5272999999999998e-05 2024-09-13T11:22:39.484 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:39.484 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:39.484 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:22:39.484 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:13.096492+0000", 2024-09-13T11:22:39.484 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0011997519999999999 2024-09-13T11:22:39.484 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:39.484 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:39.484 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:22:39.484 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:13.096555+0000", 2024-09-13T11:22:39.484 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.2632999999999995e-05 2024-09-13T11:22:39.484 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:22:39.485 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:22:39.485 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:22:39.485 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:39.485 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:39.485 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4257.0:112 1.0 e25/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 25'82, mlcod=24'80)", 2024-09-13T11:22:39.485 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:22:13.681892+0000", 2024-09-13T11:22:39.485 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 25.762243741999999, 2024-09-13T11:22:39.485 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.002076629, 2024-09-13T11:22:39.485 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:22:39.485 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:22:39.485 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:39.485 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:22:39.485 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:39.486 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:22:39.486 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:13.681892+0000", 2024-09-13T11:22:39.486 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:22:39.486 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:39.486 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:39.486 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:22:39.486 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:13.681892+0000", 2024-09-13T11:22:39.486 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:22:39.486 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:39.486 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:39.486 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:22:39.486 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:13.681895+0000", 2024-09-13T11:22:39.486 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.1700000000000001e-06 2024-09-13T11:22:39.487 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:39.487 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:39.487 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:22:39.487 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:13.681908+0000", 2024-09-13T11:22:39.487 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.3196e-05 2024-09-13T11:22:39.487 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:39.487 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:39.487 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:22:39.487 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:13.681943+0000", 2024-09-13T11:22:39.487 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.4962999999999997e-05 2024-09-13T11:22:39.487 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:39.487 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:39.487 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:22:39.487 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:13.681956+0000", 2024-09-13T11:22:39.488 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2833000000000001e-05 2024-09-13T11:22:39.488 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:39.488 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:39.488 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:22:39.488 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:13.682128+0000", 2024-09-13T11:22:39.488 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00017215700000000001 2024-09-13T11:22:39.488 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:39.488 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:39.488 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:22:39.488 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:13.682167+0000", 2024-09-13T11:22:39.488 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.8899000000000001e-05 2024-09-13T11:22:39.488 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:39.488 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:39.489 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:22:39.489 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:13.683894+0000", 2024-09-13T11:22:39.489 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.001727452 2024-09-13T11:22:39.489 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:39.489 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:39.489 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:22:39.489 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:13.683968+0000", 2024-09-13T11:22:39.489 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.3959000000000001e-05 2024-09-13T11:22:39.489 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:22:39.489 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:22:39.489 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:22:39.489 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:39.489 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:39.489 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4257.0:122 1.0 e25/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 25'92, mlcod=25'88)", 2024-09-13T11:22:39.490 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:22:13.704190+0000", 2024-09-13T11:22:39.490 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 25.739945081999998, 2024-09-13T11:22:39.490 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0027194540000000001, 2024-09-13T11:22:39.490 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:22:39.490 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:22:39.490 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:39.490 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:22:39.490 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:39.490 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:22:39.490 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:13.704190+0000", 2024-09-13T11:22:39.490 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:22:39.490 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:39.490 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:39.491 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:22:39.491 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:13.704190+0000", 2024-09-13T11:22:39.491 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:22:39.491 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:39.491 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:39.491 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:22:39.491 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:13.704192+0000", 2024-09-13T11:22:39.491 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.074e-06 2024-09-13T11:22:39.491 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:39.491 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:39.491 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:22:39.491 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:13.704211+0000", 2024-09-13T11:22:39.491 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.8173e-05 2024-09-13T11:22:39.492 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:39.492 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:39.492 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:22:39.492 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:13.704243+0000", 2024-09-13T11:22:39.492 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.2282e-05 2024-09-13T11:22:39.492 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:39.492 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:39.492 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:22:39.492 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:13.704254+0000", 2024-09-13T11:22:39.492 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.1537e-05 2024-09-13T11:22:39.492 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:39.492 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:39.492 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:22:39.493 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:13.704425+0000", 2024-09-13T11:22:39.493 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00017065 2024-09-13T11:22:39.493 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:39.493 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:39.493 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:22:39.493 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:13.704464+0000", 2024-09-13T11:22:39.493 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.9316000000000001e-05 2024-09-13T11:22:39.493 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:39.493 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:39.493 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:22:39.493 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:13.706837+0000", 2024-09-13T11:22:39.493 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0023730800000000001 2024-09-13T11:22:39.493 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:39.494 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:39.494 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:22:39.494 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:13.706910+0000", 2024-09-13T11:22:39.494 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.2342000000000006e-05 2024-09-13T11:22:39.494 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:22:39.494 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:22:39.494 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:22:39.494 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:39.494 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:39.494 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4257.0:123 1.0 e25/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 25'93, mlcod=25'89)", 2024-09-13T11:22:39.494 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:22:13.705726+0000", 2024-09-13T11:22:39.494 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 25.738409782000002, 2024-09-13T11:22:39.494 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0024901849999999998, 2024-09-13T11:22:39.495 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:22:39.495 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:22:39.495 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:39.495 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:22:39.495 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:39.495 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:22:39.495 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:13.705726+0000", 2024-09-13T11:22:39.495 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:22:39.495 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:39.495 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:39.495 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:22:39.495 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:13.705726+0000", 2024-09-13T11:22:39.495 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:22:39.496 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:39.496 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:39.496 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:22:39.496 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:13.705728+0000", 2024-09-13T11:22:39.496 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.3700000000000002e-06 2024-09-13T11:22:39.496 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:39.496 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:39.496 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:22:39.496 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:13.705799+0000", 2024-09-13T11:22:39.496 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.0776000000000001e-05 2024-09-13T11:22:39.496 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:39.496 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:39.496 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:22:39.497 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:13.705835+0000", 2024-09-13T11:22:39.497 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.5918999999999999e-05 2024-09-13T11:22:39.497 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:39.497 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:39.497 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:22:39.497 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:13.705851+0000", 2024-09-13T11:22:39.497 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.6653000000000001e-05 2024-09-13T11:22:39.497 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:39.497 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:39.497 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:22:39.497 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:13.705990+0000", 2024-09-13T11:22:39.497 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000138997 2024-09-13T11:22:39.497 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:39.498 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:39.498 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:22:39.498 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:13.706026+0000", 2024-09-13T11:22:39.498 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.5738999999999998e-05 2024-09-13T11:22:39.498 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:39.498 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:39.498 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:22:39.498 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:13.708154+0000", 2024-09-13T11:22:39.498 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0021283280000000001 2024-09-13T11:22:39.498 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:39.498 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:39.498 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:22:39.498 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:13.708216+0000", 2024-09-13T11:22:39.498 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.1402999999999995e-05 2024-09-13T11:22:39.499 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:22:39.499 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:22:39.499 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:22:39.499 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:39.499 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:39.499 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4257.0:126 1.0 e25/23 1:30306672:devicehealth::main.db.0000000000000000:head v 25'96, mlcod=25'93)", 2024-09-13T11:22:39.499 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:22:13.711692+0000", 2024-09-13T11:22:39.499 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 25.732443325999999, 2024-09-13T11:22:39.499 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0021595410000000001, 2024-09-13T11:22:39.499 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:22:39.499 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:22:39.499 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:39.499 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:22:39.500 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:39.500 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:22:39.500 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:13.711692+0000", 2024-09-13T11:22:39.500 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:22:39.500 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:39.500 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:39.500 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:22:39.500 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:13.711692+0000", 2024-09-13T11:22:39.500 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:22:39.500 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:39.500 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:39.500 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:22:39.500 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:13.711695+0000", 2024-09-13T11:22:39.501 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.5170000000000002e-06 2024-09-13T11:22:39.501 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:39.501 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:39.501 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:22:39.501 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:13.711780+0000", 2024-09-13T11:22:39.501 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.5427999999999996e-05 2024-09-13T11:22:39.501 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:39.501 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:39.501 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:22:39.501 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:13.711819+0000", 2024-09-13T11:22:39.501 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.9303e-05 2024-09-13T11:22:39.501 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:39.501 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:39.501 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:22:39.502 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:13.711860+0000", 2024-09-13T11:22:39.502 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.0194999999999997e-05 2024-09-13T11:22:39.502 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:39.502 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:39.502 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:22:39.502 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:13.712001+0000", 2024-09-13T11:22:39.502 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000141865 2024-09-13T11:22:39.502 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:39.502 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:39.502 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:22:39.502 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:13.712039+0000", 2024-09-13T11:22:39.502 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.7905999999999999e-05 2024-09-13T11:22:39.502 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:39.502 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:39.503 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:22:39.503 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:13.713734+0000", 2024-09-13T11:22:39.503 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.001694426 2024-09-13T11:22:39.503 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:39.503 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:39.503 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:22:39.503 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:13.713852+0000", 2024-09-13T11:22:39.503 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00011790099999999999 2024-09-13T11:22:39.503 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:22:39.503 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:22:39.503 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:22:39.503 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:39.503 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:39.503 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4257.0:136 1.0 e26/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 26'102, mlcod=25'100)", 2024-09-13T11:22:39.504 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:22:14.606007+0000", 2024-09-13T11:22:39.504 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 24.838128478000002, 2024-09-13T11:22:39.504 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0041530050000000004, 2024-09-13T11:22:39.504 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:22:39.504 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:22:39.504 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:39.504 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:22:39.504 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:39.504 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:22:39.504 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:14.606007+0000", 2024-09-13T11:22:39.504 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:22:39.504 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:39.505 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:39.505 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:22:39.505 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:14.606007+0000", 2024-09-13T11:22:39.505 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:22:39.505 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:39.505 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:39.505 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:22:39.505 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:14.606010+0000", 2024-09-13T11:22:39.505 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.0800000000000002e-06 2024-09-13T11:22:39.505 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:39.505 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:39.505 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:22:39.505 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:14.606023+0000", 2024-09-13T11:22:39.506 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.3139000000000001e-05 2024-09-13T11:22:39.506 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:39.506 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:39.506 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:22:39.506 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:14.606096+0000", 2024-09-13T11:22:39.506 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.2941999999999994e-05 2024-09-13T11:22:39.506 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:39.506 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:39.506 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:22:39.506 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:14.606110+0000", 2024-09-13T11:22:39.506 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.3563999999999999e-05 2024-09-13T11:22:39.506 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:39.506 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:39.507 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:22:39.507 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:14.606249+0000", 2024-09-13T11:22:39.507 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00013969300000000001 2024-09-13T11:22:39.507 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:39.507 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:39.507 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:22:39.507 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:14.606306+0000", 2024-09-13T11:22:39.507 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.6938999999999998e-05 2024-09-13T11:22:39.507 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:39.507 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:39.507 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:22:39.507 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:14.610068+0000", 2024-09-13T11:22:39.507 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0037620900000000001 2024-09-13T11:22:39.508 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:39.508 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:39.508 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:22:39.508 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:14.610160+0000", 2024-09-13T11:22:39.508 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.1558000000000007e-05 2024-09-13T11:22:39.508 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:22:39.508 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:22:39.508 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:22:39.508 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:39.508 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:39.508 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4257.0:137 1.0 e26/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 26'103, mlcod=25'100)", 2024-09-13T11:22:39.508 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:22:14.606498+0000", 2024-09-13T11:22:39.509 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 24.837637447999999, 2024-09-13T11:22:39.509 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0037471599999999998, 2024-09-13T11:22:39.509 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:22:39.509 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:22:39.509 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:39.509 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:22:39.509 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:39.509 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:22:39.509 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:14.606498+0000", 2024-09-13T11:22:39.509 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:22:39.509 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:39.509 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:39.509 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:22:39.510 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:14.606498+0000", 2024-09-13T11:22:39.510 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:22:39.510 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:39.510 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:39.510 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:22:39.510 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:14.606501+0000", 2024-09-13T11:22:39.510 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.0800000000000002e-06 2024-09-13T11:22:39.510 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:39.510 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:39.510 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:22:39.510 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:14.606514+0000", 2024-09-13T11:22:39.510 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2853e-05 2024-09-13T11:22:39.510 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:39.511 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:39.511 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:22:39.511 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:14.606582+0000", 2024-09-13T11:22:39.511 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.8119000000000003e-05 2024-09-13T11:22:39.511 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:39.511 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:39.511 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:22:39.511 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:14.606595+0000", 2024-09-13T11:22:39.511 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.3183000000000001e-05 2024-09-13T11:22:39.511 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:39.511 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:39.511 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:22:39.512 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:14.607328+0000", 2024-09-13T11:22:39.512 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00073325599999999995 2024-09-13T11:22:39.512 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:39.512 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:39.512 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:22:39.512 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:14.607368+0000", 2024-09-13T11:22:39.512 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.9192e-05 2024-09-13T11:22:39.512 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:39.512 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:39.512 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:22:39.512 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:14.610169+0000", 2024-09-13T11:22:39.512 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.002801502 2024-09-13T11:22:39.512 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:39.513 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:39.513 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:22:39.513 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:14.610245+0000", 2024-09-13T11:22:39.513 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.5975000000000004e-05 2024-09-13T11:22:39.513 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:22:39.513 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:22:39.513 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:22:39.513 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:39.513 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:39.513 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4257.0:138 1.0 e26/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 26'104, mlcod=25'100)", 2024-09-13T11:22:39.513 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:22:14.607069+0000", 2024-09-13T11:22:39.513 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 24.837066106000002, 2024-09-13T11:22:39.513 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0051436090000000004, 2024-09-13T11:22:39.514 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:22:39.514 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:22:39.514 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:39.514 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:22:39.514 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:39.514 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:22:39.514 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:14.607069+0000", 2024-09-13T11:22:39.514 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:22:39.514 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:39.514 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:39.514 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:22:39.514 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:14.607069+0000", 2024-09-13T11:22:39.514 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:22:39.514 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:39.515 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:39.515 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:22:39.515 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:14.607073+0000", 2024-09-13T11:22:39.515 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.2100000000000002e-06 2024-09-13T11:22:39.515 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:39.515 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:39.515 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:22:39.515 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:14.607311+0000", 2024-09-13T11:22:39.515 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00023832500000000001 2024-09-13T11:22:39.515 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:39.515 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:39.515 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:22:39.515 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:14.607393+0000", 2024-09-13T11:22:39.516 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.1695999999999998e-05 2024-09-13T11:22:39.516 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:39.516 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:39.516 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:22:39.516 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:14.607415+0000", 2024-09-13T11:22:39.516 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.2215999999999999e-05 2024-09-13T11:22:39.516 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:39.516 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:39.516 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:22:39.516 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:14.608040+0000", 2024-09-13T11:22:39.516 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00062501400000000002 2024-09-13T11:22:39.516 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:39.516 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:39.517 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:22:39.517 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:14.608091+0000", 2024-09-13T11:22:39.517 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.1269e-05 2024-09-13T11:22:39.517 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:39.517 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:39.517 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:22:39.517 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:14.612121+0000", 2024-09-13T11:22:39.517 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0040300470000000001 2024-09-13T11:22:39.517 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:39.517 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:39.517 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:22:39.517 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:14.612213+0000", 2024-09-13T11:22:39.517 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.1831999999999998e-05 2024-09-13T11:22:39.517 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:22:39.518 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:22:39.518 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:22:39.518 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:39.518 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:39.518 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4257.0:139 1.0 e26/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 26'105, mlcod=25'100)", 2024-09-13T11:22:39.518 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:22:14.607956+0000", 2024-09-13T11:22:39.518 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 24.836179102999999, 2024-09-13T11:22:39.518 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0043423810000000002, 2024-09-13T11:22:39.518 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:22:39.518 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:22:39.518 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:39.518 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:22:39.518 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:39.519 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:22:39.519 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:14.607956+0000", 2024-09-13T11:22:39.519 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:22:39.519 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:39.519 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:39.519 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:22:39.519 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:14.607956+0000", 2024-09-13T11:22:39.519 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:22:39.519 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:39.519 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:39.519 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:22:39.519 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:14.607959+0000", 2024-09-13T11:22:39.519 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.7259999999999998e-06 2024-09-13T11:22:39.520 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:39.520 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:39.520 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:22:39.520 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:14.607971+0000", 2024-09-13T11:22:39.520 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.19e-05 2024-09-13T11:22:39.520 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:39.520 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:39.520 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:22:39.520 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:14.608043+0000", 2024-09-13T11:22:39.520 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.2062000000000002e-05 2024-09-13T11:22:39.520 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:39.520 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:39.520 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:22:39.520 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:14.608071+0000", 2024-09-13T11:22:39.521 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.8099e-05 2024-09-13T11:22:39.521 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:39.521 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:39.521 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:22:39.521 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:14.609248+0000", 2024-09-13T11:22:39.521 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0011773510000000001 2024-09-13T11:22:39.521 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:39.521 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:39.521 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:22:39.521 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:14.609325+0000", 2024-09-13T11:22:39.521 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.6432000000000003e-05 2024-09-13T11:22:39.521 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:39.521 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:39.522 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:22:39.522 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:14.612226+0000", 2024-09-13T11:22:39.522 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.002900742 2024-09-13T11:22:39.522 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:39.522 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:39.522 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:22:39.522 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:14.612299+0000", 2024-09-13T11:22:39.522 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.3069000000000001e-05 2024-09-13T11:22:39.522 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:22:39.522 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:22:39.522 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:22:39.522 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:39.523 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:39.523 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4257.0:140 1.0 e26/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 26'106, mlcod=25'100)", 2024-09-13T11:22:39.523 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:22:14.608506+0000", 2024-09-13T11:22:39.523 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 24.835629486999999, 2024-09-13T11:22:39.523 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0038706830000000002, 2024-09-13T11:22:39.523 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:22:39.523 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:22:39.523 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:39.523 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:22:39.523 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:39.523 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:22:39.523 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:14.608506+0000", 2024-09-13T11:22:39.523 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:22:39.523 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:39.524 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:39.524 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:22:39.524 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:14.608506+0000", 2024-09-13T11:22:39.524 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:22:39.524 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:39.524 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:39.524 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:22:39.524 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:14.608509+0000", 2024-09-13T11:22:39.524 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.1829999999999998e-06 2024-09-13T11:22:39.524 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:39.524 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:39.524 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:22:39.524 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:14.608526+0000", 2024-09-13T11:22:39.525 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.7178999999999999e-05 2024-09-13T11:22:39.525 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:39.525 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:39.525 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:22:39.525 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:14.608596+0000", 2024-09-13T11:22:39.525 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.9763000000000003e-05 2024-09-13T11:22:39.525 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:39.525 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:39.525 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:22:39.525 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:14.608618+0000", 2024-09-13T11:22:39.525 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.2062e-05 2024-09-13T11:22:39.525 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:39.525 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:39.525 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:22:39.526 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:14.610337+0000", 2024-09-13T11:22:39.526 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0017188640000000001 2024-09-13T11:22:39.526 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:39.526 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:39.526 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:22:39.526 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:14.610375+0000", 2024-09-13T11:22:39.526 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.8488999999999997e-05 2024-09-13T11:22:39.526 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:39.526 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:39.526 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:22:39.526 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:14.612306+0000", 2024-09-13T11:22:39.526 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.001930532 2024-09-13T11:22:39.526 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:39.526 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:39.527 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:22:39.527 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:14.612377+0000", 2024-09-13T11:22:39.527 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.0611000000000005e-05 2024-09-13T11:22:39.527 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:22:39.527 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:22:39.527 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:22:39.527 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:39.527 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:39.527 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4257.0:141 1.0 e26/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 26'107, mlcod=25'100)", 2024-09-13T11:22:39.527 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:22:14.609021+0000", 2024-09-13T11:22:39.527 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 24.835114647000001, 2024-09-13T11:22:39.527 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0041286350000000003, 2024-09-13T11:22:39.527 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:22:39.528 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:22:39.528 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:39.528 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:22:39.528 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:39.528 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:22:39.528 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:14.609021+0000", 2024-09-13T11:22:39.528 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:22:39.528 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:39.528 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:39.528 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:22:39.528 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:14.609021+0000", 2024-09-13T11:22:39.528 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:22:39.528 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:39.528 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:39.529 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:22:39.529 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:14.609023+0000", 2024-09-13T11:22:39.529 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.5160000000000001e-06 2024-09-13T11:22:39.529 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:39.529 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:39.529 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:22:39.529 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:14.609085+0000", 2024-09-13T11:22:39.529 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.1731999999999998e-05 2024-09-13T11:22:39.529 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:39.529 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:39.529 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:22:39.529 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:14.609137+0000", 2024-09-13T11:22:39.529 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.2142000000000003e-05 2024-09-13T11:22:39.529 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:39.530 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:39.530 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:22:39.530 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:14.609148+0000", 2024-09-13T11:22:39.530 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.115e-05 2024-09-13T11:22:39.530 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:39.530 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:39.530 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:22:39.530 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:14.611058+0000", 2024-09-13T11:22:39.530 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.001909289 2024-09-13T11:22:39.530 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:39.530 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:39.530 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:22:39.530 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:14.611109+0000", 2024-09-13T11:22:39.530 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.1595999999999999e-05 2024-09-13T11:22:39.531 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:39.531 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:39.531 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:22:39.531 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:14.613063+0000", 2024-09-13T11:22:39.531 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0019537719999999999 2024-09-13T11:22:39.531 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:39.531 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:39.531 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:22:39.531 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:14.613149+0000", 2024-09-13T11:22:39.531 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.6438000000000001e-05 2024-09-13T11:22:39.531 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:22:39.531 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:22:39.531 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:22:39.531 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:39.532 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:39.532 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4257.0:142 1.0 e26/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 26'108, mlcod=25'100)", 2024-09-13T11:22:39.532 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:22:14.609825+0000", 2024-09-13T11:22:39.532 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 24.834310565999999, 2024-09-13T11:22:39.532 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0041650200000000002, 2024-09-13T11:22:39.532 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:22:39.532 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:22:39.532 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:39.532 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:22:39.532 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:39.532 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:22:39.532 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:14.609825+0000", 2024-09-13T11:22:39.532 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:22:39.532 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:39.533 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:39.533 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:22:39.533 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:14.609825+0000", 2024-09-13T11:22:39.533 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:22:39.533 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:39.533 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:39.533 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:22:39.533 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:14.609828+0000", 2024-09-13T11:22:39.533 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.0429999999999999e-06 2024-09-13T11:22:39.533 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:39.533 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:39.533 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:22:39.533 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:14.609856+0000", 2024-09-13T11:22:39.533 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.7781999999999999e-05 2024-09-13T11:22:39.534 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:39.534 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:39.534 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:22:39.534 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:14.609908+0000", 2024-09-13T11:22:39.534 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.1975999999999998e-05 2024-09-13T11:22:39.534 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:39.534 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:39.534 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:22:39.534 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:14.609919+0000", 2024-09-13T11:22:39.534 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0987e-05 2024-09-13T11:22:39.534 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:39.534 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:39.534 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:22:39.534 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:14.612447+0000", 2024-09-13T11:22:39.535 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0025280429999999998 2024-09-13T11:22:39.535 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:39.535 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:39.535 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:22:39.535 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:14.612483+0000", 2024-09-13T11:22:39.535 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.6599e-05 2024-09-13T11:22:39.535 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:39.535 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:39.535 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:22:39.535 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:14.613906+0000", 2024-09-13T11:22:39.535 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0014225920000000001 2024-09-13T11:22:39.535 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:39.535 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:39.535 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:22:39.536 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:14.613990+0000", 2024-09-13T11:22:39.536 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.3998000000000005e-05 2024-09-13T11:22:39.536 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:22:39.536 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:22:39.536 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:22:39.536 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:39.536 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:39.536 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4257.0:144 1.0 e26/23 1:30306672:devicehealth::main.db.0000000000000000:head v 26'110, mlcod=26'107)", 2024-09-13T11:22:39.536 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:22:14.615772+0000", 2024-09-13T11:22:39.536 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 24.828362990999999, 2024-09-13T11:22:39.536 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0021387350000000001, 2024-09-13T11:22:39.536 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:22:39.536 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:22:39.537 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:39.537 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:22:39.537 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:39.537 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:22:39.537 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:14.615772+0000", 2024-09-13T11:22:39.537 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:22:39.537 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:39.537 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:39.537 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:22:39.537 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:14.615772+0000", 2024-09-13T11:22:39.537 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:22:39.537 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:39.537 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:39.537 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:22:39.537 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:14.615774+0000", 2024-09-13T11:22:39.538 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.623e-06 2024-09-13T11:22:39.538 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:39.538 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:39.538 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:22:39.538 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:14.615859+0000", 2024-09-13T11:22:39.538 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.5382000000000003e-05 2024-09-13T11:22:39.538 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:39.538 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:39.538 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:22:39.538 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:14.615910+0000", 2024-09-13T11:22:39.538 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.1038999999999997e-05 2024-09-13T11:22:39.538 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:39.538 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:39.538 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:22:39.539 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:14.615920+0000", 2024-09-13T11:22:39.539 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.5429999999999999e-06 2024-09-13T11:22:39.539 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:39.539 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:39.539 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:22:39.539 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:14.616157+0000", 2024-09-13T11:22:39.539 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00023733200000000001 2024-09-13T11:22:39.539 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:39.539 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:39.539 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:22:39.539 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:14.616213+0000", 2024-09-13T11:22:39.539 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.5195000000000003e-05 2024-09-13T11:22:39.539 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:39.539 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:39.540 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:22:39.540 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:14.617831+0000", 2024-09-13T11:22:39.540 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.001618179 2024-09-13T11:22:39.540 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:39.540 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:39.540 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:22:39.540 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:14.617911+0000", 2024-09-13T11:22:39.540 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.0442e-05 2024-09-13T11:22:39.540 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:22:39.540 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:22:39.540 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:22:39.540 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:22:39.540 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:22:39.540 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-13T11:22:39.542 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-13T11:22:39.543 INFO:teuthology.orchestra.run.smithi001.stdout:osd.0: {} 2024-09-13T11:22:39.543 INFO:teuthology.orchestra.run.smithi001.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T11:22:39.543 INFO:teuthology.orchestra.run.smithi001.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-09-13T11:22:39.543 INFO:teuthology.orchestra.run.smithi001.stdout:osd.2: {} 2024-09-13T11:22:39.544 INFO:teuthology.orchestra.run.smithi001.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T11:22:39.544 INFO:tasks.ceph.osd.2.smithi162.stderr:2024-09-13T11:22:39.539+0000 7fe92b35a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26313) UID: 0 2024-09-13T11:22:39.571 INFO:teuthology.orchestra.run.smithi001.stdout:osd.3: {} 2024-09-13T11:22:39.571 INFO:teuthology.orchestra.run.smithi001.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T11:22:39.593 INFO:teuthology.orchestra.run.smithi001.stdout:osd.4: {} 2024-09-13T11:22:39.594 INFO:teuthology.orchestra.run.smithi001.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T11:22:39.623 INFO:teuthology.orchestra.run.smithi001.stdout:osd.5: {} 2024-09-13T11:22:39.623 INFO:teuthology.orchestra.run.smithi001.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T11:22:39.642 INFO:tasks.ceph.osd.13.smithi092.stderr:2024-09-13T11:22:39.643+0000 7f3e5afea640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26350) UID: 0 2024-09-13T11:22:39.653 INFO:teuthology.orchestra.run.smithi001.stdout:osd.6: {} 2024-09-13T11:22:39.653 INFO:teuthology.orchestra.run.smithi001.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T11:22:39.673 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-13T11:22:39.674 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:22:39.684 INFO:teuthology.orchestra.run.smithi001.stdout:osd.7: {} 2024-09-13T11:22:39.685 INFO:teuthology.orchestra.run.smithi001.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T11:22:39.688 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-13T11:22:39.711 INFO:teuthology.orchestra.run.smithi001.stdout:osd.8: {} 2024-09-13T11:22:39.711 INFO:teuthology.orchestra.run.smithi001.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-13T11:22:39.742 INFO:teuthology.orchestra.run.smithi001.stdout:osd.9: {} 2024-09-13T11:22:39.743 INFO:teuthology.orchestra.run.smithi001.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-13T11:22:39.743 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-13T11:22:39.736+0000 7f6a68775640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26307) UID: 0 2024-09-13T11:22:39.772 INFO:teuthology.orchestra.run.smithi001.stdout:osd.10: {} 2024-09-13T11:22:39.772 INFO:teuthology.orchestra.run.smithi001.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-13T11:22:39.805 INFO:teuthology.orchestra.run.smithi001.stdout:osd.11: {} 2024-09-13T11:22:39.805 INFO:teuthology.orchestra.run.smithi001.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-13T11:22:39.830 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-13T11:22:39.830 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:22:39.835 INFO:teuthology.orchestra.run.smithi001.stdout:osd.12: {} 2024-09-13T11:22:39.835 INFO:teuthology.orchestra.run.smithi001.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-13T11:22:39.844 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-13T11:22:39.845 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-09-13T11:22:39.839+0000 7f97befc9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26315) UID: 0 2024-09-13T11:22:39.870 INFO:teuthology.orchestra.run.smithi001.stdout:osd.13: {} 2024-09-13T11:22:39.870 INFO:teuthology.orchestra.run.smithi001.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-13T11:22:39.902 INFO:teuthology.orchestra.run.smithi001.stdout:osd.14: {} 2024-09-13T11:22:39.902 INFO:teuthology.orchestra.run.smithi001.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-13T11:22:39.937 INFO:teuthology.orchestra.run.smithi001.stdout:osd.15: {} 2024-09-13T11:22:39.937 INFO:teuthology.orchestra.run.smithi001.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-13T11:22:39.943 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-13T11:22:39.943 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-13T11:22:39.945 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-09-13T11:22:39.943+0000 7f97befc9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26315) UID: 0 2024-09-13T11:22:39.997 INFO:tasks.radosbench.radosbench.0.smithi177.stdout: 13 16 87 71 0.3413 0 - 0.0234801 2024-09-13T11:22:40.002 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-13T11:22:40.002 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:22:40.017 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-09-13T11:22:40.046 INFO:tasks.ceph.osd.14.smithi162.stderr:2024-09-13T11:22:40.043+0000 7f235c0db640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26319) UID: 0 2024-09-13T11:22:40.058 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15] out_osds: [] dead_osds: [1] live_osds: [0, 4, 8, 12, 5, 9, 13, 2, 6, 10, 14, 3, 7, 11, 15] 2024-09-13T11:22:40.058 INFO:tasks.thrashosds.thrasher:Reviving osd 1 2024-09-13T11:22:40.058 INFO:tasks.ceph.osd.1:Restarting daemon 2024-09-13T11:22:40.059 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-09-13T11:22:40.061 INFO:tasks.ceph.osd.1:Started 2024-09-13T11:22:40.061 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-13T11:22:40.144 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-13T11:22:40.144 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:22:40.147 INFO:tasks.ceph.osd.14.smithi162.stderr:2024-09-13T11:22:40.143+0000 7f235c0db640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26319) UID: 0 2024-09-13T11:22:40.158 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-09-13T11:22:40.176 INFO:teuthology.orchestra.run.smithi092.stderr:no valid command found; 10 closest matches: 2024-09-13T11:22:40.176 INFO:teuthology.orchestra.run.smithi092.stderr:0 2024-09-13T11:22:40.176 INFO:teuthology.orchestra.run.smithi092.stderr:1 2024-09-13T11:22:40.176 INFO:teuthology.orchestra.run.smithi092.stderr:2 2024-09-13T11:22:40.176 INFO:teuthology.orchestra.run.smithi092.stderr:abort 2024-09-13T11:22:40.176 INFO:teuthology.orchestra.run.smithi092.stderr:assert 2024-09-13T11:22:40.176 INFO:teuthology.orchestra.run.smithi092.stderr:config diff 2024-09-13T11:22:40.176 INFO:teuthology.orchestra.run.smithi092.stderr:config diff get 2024-09-13T11:22:40.176 INFO:teuthology.orchestra.run.smithi092.stderr:config get 2024-09-13T11:22:40.177 INFO:teuthology.orchestra.run.smithi092.stderr:config help [] 2024-09-13T11:22:40.177 INFO:teuthology.orchestra.run.smithi092.stderr:config set ... 2024-09-13T11:22:40.177 INFO:teuthology.orchestra.run.smithi092.stderr:admin_socket: invalid command 2024-09-13T11:22:40.178 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-13T11:22:40.179 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-1, ['dump_ops_in_flight'] 2024-09-13T11:22:40.222 INFO:teuthology.orchestra.run.smithi001.stderr:nodeep-scrub is set 2024-09-13T11:22:40.247 INFO:tasks.ceph.osd.3.smithi177.stderr:2024-09-13T11:22:40.244+0000 7f10adba4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26574) UID: 0 2024-09-13T11:22:40.286 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-13T11:22:40.286 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:22:40.301 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-09-13T11:22:40.348 INFO:tasks.ceph.osd.15.smithi177.stderr:2024-09-13T11:22:40.344+0000 7f9d6ce77640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26578) UID: 0 2024-09-13T11:22:40.422 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-13T11:22:40.423 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:22:40.437 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_ops_in_flight 2024-09-13T11:22:40.449 INFO:tasks.ceph.osd.5.smithi092.stderr:2024-09-13T11:22:40.447+0000 7f6f0b705640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26346) UID: 0 2024-09-13T11:22:40.550 INFO:tasks.ceph.osd.10.smithi162.stderr:2024-09-13T11:22:40.547+0000 7fb59c9f0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26317) UID: 0 2024-09-13T11:22:40.564 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-13T11:22:40.564 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:22:40.578 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_blocked_ops 2024-09-13T11:22:40.651 INFO:tasks.ceph.osd.12.smithi001.stderr:2024-09-13T11:22:40.644+0000 7f892c6e6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26314) UID: 0 2024-09-13T11:22:40.704 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-13T11:22:40.704 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:22:40.718 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_historic_ops 2024-09-13T11:22:40.752 INFO:tasks.ceph.osd.2.smithi162.stderr:2024-09-13T11:22:40.747+0000 7fe92b35a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26313) UID: 0 2024-09-13T11:22:40.853 INFO:tasks.ceph.osd.9.smithi092.stderr:2024-09-13T11:22:40.851+0000 7ff5ab3d0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26348) UID: 0 2024-09-13T11:22:40.886 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-13T11:22:40.886 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:22:40.901 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-13T11:22:40.955 INFO:tasks.ceph.osd.11.smithi177.stderr:2024-09-13T11:22:40.956+0000 7f24792c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26580) UID: 0 2024-09-13T11:22:40.997 INFO:tasks.radosbench.radosbench.0.smithi177.stdout: 14 16 87 71 0.316921 0 - 0.0234801 2024-09-13T11:22:41.032 INFO:teuthology.orchestra.run.smithi092.stdout:ERROR: (22) Invalid argument 2024-09-13T11:22:41.032 INFO:teuthology.orchestra.run.smithi092.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:22:41.046 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-13T11:22:41.055 INFO:tasks.ceph.osd.3.smithi177.stderr:2024-09-13T11:22:41.056+0000 7f10adba4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26574) UID: 0 2024-09-13T11:22:41.153 INFO:tasks.ceph.osd.1.smithi092.stderr:2024-09-13T11:22:41.151+0000 7fe77ba097c0 -1 Falling back to public interface 2024-09-13T11:22:41.157 INFO:tasks.ceph.osd.8.smithi001.stderr:2024-09-13T11:22:41.152+0000 7fb9b48ae640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26312) UID: 0 2024-09-13T11:22:41.169 INFO:teuthology.orchestra.run.smithi092.stdout:ERROR: (22) Invalid argument 2024-09-13T11:22:41.169 INFO:teuthology.orchestra.run.smithi092.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:22:41.184 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-13T11:22:41.257 INFO:tasks.ceph.osd.5.smithi092.stderr:2024-09-13T11:22:41.255+0000 7f6f0b705640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26346) UID: 0 2024-09-13T11:22:41.300 INFO:teuthology.orchestra.run.smithi092.stdout:ERROR: (22) Invalid argument 2024-09-13T11:22:41.301 INFO:teuthology.orchestra.run.smithi092.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:22:41.314 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-09-13T11:22:41.359 INFO:tasks.ceph.osd.4.smithi001.stderr:2024-09-13T11:22:41.352+0000 7ff8e0218640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26310) UID: 0 2024-09-13T11:22:41.459 INFO:tasks.ceph.osd.9.smithi092.stderr:2024-09-13T11:22:41.455+0000 7ff5ab3d0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26348) UID: 0 2024-09-13T11:22:41.480 INFO:teuthology.orchestra.run.smithi092.stdout:ERROR: (22) Invalid argument 2024-09-13T11:22:41.480 INFO:teuthology.orchestra.run.smithi092.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:22:41.484 INFO:tasks.ceph.osd.4.smithi001.stderr:2024-09-13T11:22:41.480+0000 7ff8d0c3d640 -1 osd.4 36 handle_osd_map injecting map crc failure 2024-09-13T11:22:41.486 INFO:tasks.ceph.osd.4.smithi001.stderr:2024-09-13T11:22:41.480+0000 7ff8d0c3d640 -1 osd.4 36 handle_osd_map injecting map crc failure 2024-09-13T11:22:41.494 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-09-13T11:22:41.584 INFO:tasks.ceph.osd.5.smithi092.stderr:2024-09-13T11:22:41.583+0000 7f6f0b705640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26346) UID: 0 2024-09-13T11:22:41.656 INFO:teuthology.orchestra.run.smithi092.stdout:ERROR: (22) Invalid argument 2024-09-13T11:22:41.656 INFO:teuthology.orchestra.run.smithi092.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:22:41.660 INFO:tasks.ceph.osd.9.smithi092.stderr:2024-09-13T11:22:41.659+0000 7ff5ab3d0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26348) UID: 0 2024-09-13T11:22:41.670 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-09-13T11:22:41.748 INFO:tasks.ceph.osd.14.smithi162.stderr:2024-09-13T11:22:41.743+0000 7f2350b08640 -1 osd.14 36 handle_osd_map injecting map crc failure 2024-09-13T11:22:41.750 INFO:tasks.ceph.osd.14.smithi162.stderr:2024-09-13T11:22:41.747+0000 7f2350b08640 -1 osd.14 36 handle_osd_map injecting map crc failure 2024-09-13T11:22:41.762 INFO:tasks.ceph.osd.15.smithi177.stderr:2024-09-13T11:22:41.760+0000 7f9d6ce77640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26578) UID: 0 2024-09-13T11:22:41.837 INFO:teuthology.orchestra.run.smithi092.stdout:ERROR: (22) Invalid argument 2024-09-13T11:22:41.838 INFO:teuthology.orchestra.run.smithi092.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:22:41.852 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_ops_in_flight 2024-09-13T11:22:41.900 INFO:tasks.ceph.osd.13.smithi092.stderr:2024-09-13T11:22:41.899+0000 7f3e5afea640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26350) UID: 0 2024-09-13T11:22:41.945 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T11:22:41.964 INFO:tasks.ceph.osd.2.smithi162.stderr:2024-09-13T11:22:41.959+0000 7fe92b35a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26313) UID: 0 2024-09-13T11:22:41.976 INFO:teuthology.orchestra.run.smithi092.stdout:ERROR: (22) Invalid argument 2024-09-13T11:22:41.976 INFO:teuthology.orchestra.run.smithi092.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:22:41.991 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_blocked_ops 2024-09-13T11:22:41.997 INFO:tasks.radosbench.radosbench.0.smithi177.stdout: 15 16 87 71 0.295792 0 - 0.0234801 2024-09-13T11:22:42.065 INFO:tasks.ceph.osd.12.smithi001.stderr:2024-09-13T11:22:42.060+0000 7f892c6e6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26314) UID: 0 2024-09-13T11:22:42.112 INFO:teuthology.orchestra.run.smithi001.stdout:osd.0: {} 2024-09-13T11:22:42.113 INFO:teuthology.orchestra.run.smithi001.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T11:22:42.113 INFO:teuthology.orchestra.run.smithi001.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-09-13T11:22:42.138 INFO:teuthology.orchestra.run.smithi001.stdout:osd.2: {} 2024-09-13T11:22:42.138 INFO:teuthology.orchestra.run.smithi001.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T11:22:42.150 INFO:teuthology.orchestra.run.smithi092.stdout:ERROR: (22) Invalid argument 2024-09-13T11:22:42.150 INFO:teuthology.orchestra.run.smithi092.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:22:42.165 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_historic_ops 2024-09-13T11:22:42.167 INFO:teuthology.orchestra.run.smithi001.stdout:osd.3: {} 2024-09-13T11:22:42.167 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-09-13T11:22:42.163+0000 7f97befc9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26315) UID: 0 2024-09-13T11:22:42.167 INFO:teuthology.orchestra.run.smithi001.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T11:22:42.193 INFO:teuthology.orchestra.run.smithi001.stdout:osd.4: {} 2024-09-13T11:22:42.193 INFO:teuthology.orchestra.run.smithi001.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T11:22:42.219 INFO:teuthology.orchestra.run.smithi001.stdout:osd.5: {} 2024-09-13T11:22:42.220 INFO:teuthology.orchestra.run.smithi001.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T11:22:42.248 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-13T11:22:42.250 INFO:teuthology.orchestra.run.smithi001.stdout:osd.6: {} 2024-09-13T11:22:42.250 INFO:teuthology.orchestra.run.smithi001.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T11:22:42.266 INFO:tasks.ceph.osd.15.smithi177.stderr:2024-09-13T11:22:42.264+0000 7f9d6ce77640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26578) UID: 0 2024-09-13T11:22:42.282 INFO:teuthology.orchestra.run.smithi001.stdout:osd.7: {} 2024-09-13T11:22:42.282 INFO:teuthology.orchestra.run.smithi001.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T11:22:42.289 INFO:teuthology.orchestra.run.smithi092.stdout:ERROR: (22) Invalid argument 2024-09-13T11:22:42.289 INFO:teuthology.orchestra.run.smithi092.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:22:42.303 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-13T11:22:42.309 INFO:teuthology.orchestra.run.smithi001.stdout:osd.8: {} 2024-09-13T11:22:42.309 INFO:teuthology.orchestra.run.smithi001.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-13T11:22:42.337 INFO:teuthology.orchestra.run.smithi001.stdout:osd.9: {} 2024-09-13T11:22:42.338 INFO:teuthology.orchestra.run.smithi001.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-13T11:22:42.367 INFO:teuthology.orchestra.run.smithi001.stdout:osd.10: {} 2024-09-13T11:22:42.367 INFO:teuthology.orchestra.run.smithi001.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-13T11:22:42.368 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-13T11:22:42.360+0000 7f6a68775640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26307) UID: 0 2024-09-13T11:22:42.398 INFO:teuthology.orchestra.run.smithi001.stdout:osd.11: {} 2024-09-13T11:22:42.398 INFO:teuthology.orchestra.run.smithi001.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-13T11:22:42.425 INFO:teuthology.orchestra.run.smithi001.stdout:osd.12: {} 2024-09-13T11:22:42.425 INFO:teuthology.orchestra.run.smithi001.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-13T11:22:42.437 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-13T11:22:42.438 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [], 2024-09-13T11:22:42.438 INFO:teuthology.orchestra.run.smithi162.stdout: "num_ops": 0 2024-09-13T11:22:42.438 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-13T11:22:42.452 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-13T11:22:42.456 INFO:teuthology.orchestra.run.smithi001.stdout:osd.13: {} 2024-09-13T11:22:42.457 INFO:teuthology.orchestra.run.smithi001.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-13T11:22:42.468 INFO:tasks.ceph.osd.8.smithi001.stderr:2024-09-13T11:22:42.464+0000 7fb9b48ae640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26312) UID: 0 2024-09-13T11:22:42.483 INFO:teuthology.orchestra.run.smithi001.stdout:osd.14: {} 2024-09-13T11:22:42.483 INFO:teuthology.orchestra.run.smithi001.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-13T11:22:42.513 INFO:teuthology.orchestra.run.smithi001.stdout:osd.15: {} 2024-09-13T11:22:42.513 INFO:teuthology.orchestra.run.smithi001.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-13T11:22:42.521 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-13T11:22:42.521 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-13T11:22:42.569 INFO:tasks.ceph.osd.13.smithi092.stderr:2024-09-13T11:22:42.567+0000 7f3e5afea640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26350) UID: 0 2024-09-13T11:22:42.574 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-13T11:22:42.574 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [], 2024-09-13T11:22:42.575 INFO:teuthology.orchestra.run.smithi162.stdout: "complaint_time": 30, 2024-09-13T11:22:42.575 INFO:teuthology.orchestra.run.smithi162.stdout: "num_blocked_ops": 0 2024-09-13T11:22:42.575 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-13T11:22:42.589 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-13T11:22:42.670 INFO:tasks.ceph.osd.10.smithi162.stderr:2024-09-13T11:22:42.667+0000 7fb59c9f0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26317) UID: 0 2024-09-13T11:22:42.713 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-13T11:22:42.713 INFO:teuthology.orchestra.run.smithi162.stdout: "size": 20, 2024-09-13T11:22:42.713 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 600, 2024-09-13T11:22:42.713 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [ 2024-09-13T11:22:42.713 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:42.713 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.4665.0:12 3.9 e32/30 3:9dad78b7:::benchmark_data_smithi177_31116_object11:head v 32'1, mlcod=0'0)", 2024-09-13T11:22:42.713 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:22:27.004720+0000", 2024-09-13T11:22:42.713 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 15.708029420000001, 2024-09-13T11:22:42.713 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.004117791, 2024-09-13T11:22:42.713 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:22:42.713 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:22:42.714 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:42.714 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:22:42.714 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:42.714 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:22:42.714 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.004720+0000", 2024-09-13T11:22:42.714 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:22:42.714 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:42.714 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:42.714 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:22:42.714 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.004720+0000", 2024-09-13T11:22:42.714 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:22:42.714 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:42.715 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:42.715 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:22:42.715 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.004722+0000", 2024-09-13T11:22:42.715 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.483e-06 2024-09-13T11:22:42.715 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:42.715 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:42.715 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:22:42.715 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.004999+0000", 2024-09-13T11:22:42.715 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00027699700000000002 2024-09-13T11:22:42.715 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:42.715 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:42.715 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:22:42.715 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.005067+0000", 2024-09-13T11:22:42.715 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.8162000000000002e-05 2024-09-13T11:22:42.716 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:42.716 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:42.716 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:22:42.716 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.005104+0000", 2024-09-13T11:22:42.716 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.6698e-05 2024-09-13T11:22:42.716 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:42.716 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:42.716 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:22:42.716 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.005253+0000", 2024-09-13T11:22:42.716 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000149111 2024-09-13T11:22:42.716 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:42.716 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:42.717 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:22:42.717 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.005326+0000", 2024-09-13T11:22:42.717 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.2948999999999996e-05 2024-09-13T11:22:42.717 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:42.717 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:42.717 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:22:42.717 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.008734+0000", 2024-09-13T11:22:42.717 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0034086300000000002 2024-09-13T11:22:42.717 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:42.717 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:42.717 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:22:42.717 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.008838+0000", 2024-09-13T11:22:42.718 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000103761 2024-09-13T11:22:42.718 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:22:42.718 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:22:42.718 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:22:42.718 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:42.718 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:42.718 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.4665.0:13 3.9 e32/30 3:967a7538:::benchmark_data_smithi177_31116_object12:head v 32'2, mlcod=0'0)", 2024-09-13T11:22:42.718 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:22:27.006514+0000", 2024-09-13T11:22:42.718 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 15.706235940999999, 2024-09-13T11:22:42.718 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0070363759999999996, 2024-09-13T11:22:42.718 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:22:42.718 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:22:42.718 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:42.719 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:22:42.719 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:42.719 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:22:42.719 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.006514+0000", 2024-09-13T11:22:42.719 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:22:42.719 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:42.719 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:42.719 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:22:42.719 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.006514+0000", 2024-09-13T11:22:42.719 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:22:42.719 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:42.719 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:42.719 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:22:42.719 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.006517+0000", 2024-09-13T11:22:42.720 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.3900000000000002e-06 2024-09-13T11:22:42.720 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:42.720 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:42.720 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:22:42.720 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.006716+0000", 2024-09-13T11:22:42.720 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000198612 2024-09-13T11:22:42.720 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:42.720 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:42.721 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:22:42.721 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.006803+0000", 2024-09-13T11:22:42.721 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.7544000000000001e-05 2024-09-13T11:22:42.721 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:42.721 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:42.721 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:22:42.721 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.006817+0000", 2024-09-13T11:22:42.721 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4175999999999999e-05 2024-09-13T11:22:42.721 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:42.721 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:42.721 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:22:42.721 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.006947+0000", 2024-09-13T11:22:42.721 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000129209 2024-09-13T11:22:42.722 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:42.722 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:42.722 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:22:42.722 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.007002+0000", 2024-09-13T11:22:42.722 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.5476999999999998e-05 2024-09-13T11:22:42.722 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:42.722 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:42.722 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:22:42.722 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.013448+0000", 2024-09-13T11:22:42.722 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0064460109999999998 2024-09-13T11:22:42.722 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:42.722 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:42.722 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:22:42.723 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.013550+0000", 2024-09-13T11:22:42.723 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000101957 2024-09-13T11:22:42.723 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:22:42.723 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:22:42.723 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:22:42.723 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:42.723 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:42.723 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.4665.0:37 3.9 e32/30 3:962d44ea:::benchmark_data_smithi177_31116_object36:head v 32'3, mlcod=32'1)", 2024-09-13T11:22:42.723 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:22:27.016683+0000", 2024-09-13T11:22:42.723 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 15.696066807999999, 2024-09-13T11:22:42.723 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0028898940000000001, 2024-09-13T11:22:42.723 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:22:42.723 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:22:42.723 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:42.724 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:22:42.724 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:42.724 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:22:42.724 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.016683+0000", 2024-09-13T11:22:42.724 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:22:42.724 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:42.724 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:42.724 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:22:42.724 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.016683+0000", 2024-09-13T11:22:42.724 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:22:42.724 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:42.725 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:42.725 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:22:42.725 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.016686+0000", 2024-09-13T11:22:42.725 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.2660000000000002e-06 2024-09-13T11:22:42.725 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:42.725 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:42.725 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:22:42.725 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.016767+0000", 2024-09-13T11:22:42.725 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.0328999999999996e-05 2024-09-13T11:22:42.725 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:42.725 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:42.725 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:22:42.725 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.016844+0000", 2024-09-13T11:22:42.726 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.7222e-05 2024-09-13T11:22:42.726 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:42.726 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:42.726 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:22:42.726 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.016858+0000", 2024-09-13T11:22:42.726 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3876000000000001e-05 2024-09-13T11:22:42.726 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:42.728 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:42.728 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:22:42.728 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.016962+0000", 2024-09-13T11:22:42.728 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00010483400000000001 2024-09-13T11:22:42.728 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:42.728 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:42.728 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:22:42.728 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.017002+0000", 2024-09-13T11:22:42.728 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.9467e-05 2024-09-13T11:22:42.728 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:42.729 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:42.729 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:22:42.729 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.019496+0000", 2024-09-13T11:22:42.729 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0024940510000000002 2024-09-13T11:22:42.729 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:42.729 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:42.729 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:22:42.729 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.019573+0000", 2024-09-13T11:22:42.729 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.6848999999999996e-05 2024-09-13T11:22:42.729 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:22:42.729 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:22:42.729 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:22:42.729 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:42.730 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:42.730 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.4665.0:59 3.9 e32/30 3:9499e3d6:::benchmark_data_smithi177_31116_object58:head v 32'4, mlcod=32'2)", 2024-09-13T11:22:42.730 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:22:27.028489+0000", 2024-09-13T11:22:42.730 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 15.684261149999999, 2024-09-13T11:22:42.730 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0015734430000000001, 2024-09-13T11:22:42.730 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:22:42.730 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:22:42.730 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:42.730 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:22:42.730 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:42.731 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:22:42.731 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.028489+0000", 2024-09-13T11:22:42.731 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:22:42.731 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:42.731 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:42.731 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:22:42.731 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.028489+0000", 2024-09-13T11:22:42.731 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:22:42.731 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:42.731 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:42.731 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:22:42.732 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.028491+0000", 2024-09-13T11:22:42.732 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.9170000000000002e-06 2024-09-13T11:22:42.732 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:42.732 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:42.732 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:22:42.732 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.028547+0000", 2024-09-13T11:22:42.732 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.5832999999999999e-05 2024-09-13T11:22:42.732 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:42.732 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:42.732 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:22:42.732 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.028618+0000", 2024-09-13T11:22:42.732 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.0741999999999995e-05 2024-09-13T11:22:42.732 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:42.732 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:42.733 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:22:42.733 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.028635+0000", 2024-09-13T11:22:42.733 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.6773e-05 2024-09-13T11:22:42.733 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:42.733 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:42.733 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:22:42.733 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.028758+0000", 2024-09-13T11:22:42.733 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00012363200000000001 2024-09-13T11:22:42.733 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:42.733 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:42.733 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:22:42.733 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.028795+0000", 2024-09-13T11:22:42.733 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.6865e-05 2024-09-13T11:22:42.733 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:42.734 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:42.734 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:22:42.734 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.029988+0000", 2024-09-13T11:22:42.734 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001192505 2024-09-13T11:22:42.734 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:42.734 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:42.734 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:22:42.734 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.030062+0000", 2024-09-13T11:22:42.734 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.4176000000000003e-05 2024-09-13T11:22:42.734 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:22:42.734 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:22:42.734 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:22:42.734 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:42.734 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:42.735 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.4665.0:76 3.9 e32/30 3:9e00a8c6:::benchmark_data_smithi177_31116_object75:head v 32'5, mlcod=32'3)", 2024-09-13T11:22:42.735 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:22:27.044735+0000", 2024-09-13T11:22:42.735 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 15.668014303, 2024-09-13T11:22:42.735 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0018264010000000001, 2024-09-13T11:22:42.735 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:22:42.735 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:22:42.735 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:42.735 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:22:42.735 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:42.735 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:22:42.735 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.044735+0000", 2024-09-13T11:22:42.735 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:22:42.735 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:42.736 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:42.736 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:22:42.736 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.044735+0000", 2024-09-13T11:22:42.736 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:22:42.736 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:42.736 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:42.736 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:22:42.736 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.044739+0000", 2024-09-13T11:22:42.736 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.9389999999999999e-06 2024-09-13T11:22:42.736 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:42.736 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:42.736 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:22:42.736 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.044860+0000", 2024-09-13T11:22:42.736 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00012027300000000001 2024-09-13T11:22:42.737 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:42.737 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:42.737 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:22:42.737 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.044931+0000", 2024-09-13T11:22:42.737 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.1575999999999994e-05 2024-09-13T11:22:42.737 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:42.737 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:42.737 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:22:42.737 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.044946+0000", 2024-09-13T11:22:42.737 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4851999999999999e-05 2024-09-13T11:22:42.737 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:42.738 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:42.738 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:22:42.738 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.045050+0000", 2024-09-13T11:22:42.738 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000104264 2024-09-13T11:22:42.738 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:42.738 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:42.738 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:22:42.738 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.045088+0000", 2024-09-13T11:22:42.738 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.7904999999999997e-05 2024-09-13T11:22:42.738 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:42.738 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:42.738 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:22:42.739 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.046490+0000", 2024-09-13T11:22:42.739 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001401619 2024-09-13T11:22:42.739 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:42.739 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:42.739 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:22:42.739 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.046562+0000", 2024-09-13T11:22:42.739 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.1972999999999997e-05 2024-09-13T11:22:42.739 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:22:42.739 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:22:42.739 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:22:42.739 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:42.739 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:42.739 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.4665.0:85 3.9 e33/30 3:9b4652c0:::benchmark_data_smithi177_31116_object84:head v 33'6, mlcod=32'4)", 2024-09-13T11:22:42.740 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:22:27.977308+0000", 2024-09-13T11:22:42.740 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 14.735441591000001, 2024-09-13T11:22:42.740 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0014741579999999999, 2024-09-13T11:22:42.740 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:22:42.740 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:22:42.740 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:42.740 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:22:42.740 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:42.740 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:22:42.740 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.977308+0000", 2024-09-13T11:22:42.740 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:22:42.740 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:42.740 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:42.741 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:22:42.741 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.977308+0000", 2024-09-13T11:22:42.741 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:22:42.741 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:42.741 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:42.741 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:22:42.741 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.977310+0000", 2024-09-13T11:22:42.741 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.699e-06 2024-09-13T11:22:42.741 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:42.741 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:42.741 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:22:42.741 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.977556+0000", 2024-09-13T11:22:42.741 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00024646299999999999 2024-09-13T11:22:42.742 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:42.742 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:42.742 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:22:42.742 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.977628+0000", 2024-09-13T11:22:42.742 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.2075000000000004e-05 2024-09-13T11:22:42.742 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:42.742 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:42.742 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:22:42.742 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.977647+0000", 2024-09-13T11:22:42.742 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.8542e-05 2024-09-13T11:22:42.742 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:42.742 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:42.742 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:22:42.742 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.977732+0000", 2024-09-13T11:22:42.743 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.5199999999999997e-05 2024-09-13T11:22:42.743 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:42.743 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:42.743 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:22:42.743 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.977752+0000", 2024-09-13T11:22:42.743 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.9400999999999999e-05 2024-09-13T11:22:42.743 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:42.743 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:42.743 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:22:42.743 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.978717+0000", 2024-09-13T11:22:42.743 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00096598400000000003 2024-09-13T11:22:42.744 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:42.744 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:42.744 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:22:42.744 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.978782+0000", 2024-09-13T11:22:42.744 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.4794000000000003e-05 2024-09-13T11:22:42.744 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:22:42.744 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:22:42.744 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:22:42.744 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:22:42.744 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:22:42.744 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-13T11:22:42.746 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-13T11:22:42.771 INFO:tasks.ceph.osd.4.smithi001.stderr:2024-09-13T11:22:42.764+0000 7ff8e0218640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26310) UID: 0 2024-09-13T11:22:42.872 INFO:tasks.ceph.osd.10.smithi162.stderr:2024-09-13T11:22:42.867+0000 7fb59c9f0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26317) UID: 0 2024-09-13T11:22:42.915 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-13T11:22:42.915 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [], 2024-09-13T11:22:42.916 INFO:teuthology.orchestra.run.smithi162.stdout: "num_ops": 0 2024-09-13T11:22:42.916 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-13T11:22:42.929 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-13T11:22:42.973 INFO:tasks.ceph.osd.13.smithi092.stderr:2024-09-13T11:22:42.971+0000 7f3e5afea640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26350) UID: 0 2024-09-13T11:22:42.998 INFO:tasks.radosbench.radosbench.0.smithi177.stdout: 16 16 87 71 0.277305 0 - 0.0234801 2024-09-13T11:22:43.074 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-13T11:22:43.068+0000 7f6a68775640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26307) UID: 0 2024-09-13T11:22:43.098 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-13T11:22:43.098 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [], 2024-09-13T11:22:43.098 INFO:teuthology.orchestra.run.smithi162.stdout: "complaint_time": 30, 2024-09-13T11:22:43.098 INFO:teuthology.orchestra.run.smithi162.stdout: "num_blocked_ops": 0 2024-09-13T11:22:43.098 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-13T11:22:43.112 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-13T11:22:43.175 INFO:tasks.ceph.osd.4.smithi001.stderr:2024-09-13T11:22:43.168+0000 7ff8e0218640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26310) UID: 0 2024-09-13T11:22:43.235 INFO:teuthology.orchestra.run.smithi001.stderr:noscrub is unset 2024-09-13T11:22:43.242 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-13T11:22:43.242 INFO:teuthology.orchestra.run.smithi162.stdout: "size": 20, 2024-09-13T11:22:43.242 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 600, 2024-09-13T11:22:43.242 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [ 2024-09-13T11:22:43.242 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.243 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4257.0:45 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-13T11:22:43.243 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:22:12.319311+0000", 2024-09-13T11:22:43.243 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 30.918735697999999, 2024-09-13T11:22:43.243 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.029681038, 2024-09-13T11:22:43.243 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:22:43.243 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:22:43.243 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:43.243 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:22:43.243 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4257", 2024-09-13T11:22:43.243 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/1488131527", 2024-09-13T11:22:43.243 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 45 2024-09-13T11:22:43.243 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.243 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:22:43.244 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.244 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:22:43.244 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.319311+0000", 2024-09-13T11:22:43.244 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:22:43.244 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.244 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.244 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:22:43.244 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.319311+0000", 2024-09-13T11:22:43.244 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:22:43.244 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.244 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.244 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:22:43.245 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.319313+0000", 2024-09-13T11:22:43.245 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.2859999999999998e-06 2024-09-13T11:22:43.245 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.245 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.245 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:22:43.245 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.319322+0000", 2024-09-13T11:22:43.245 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.4360000000000002e-06 2024-09-13T11:22:43.245 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.245 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.245 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:22:43.246 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.319338+0000", 2024-09-13T11:22:43.246 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.6158999999999999e-05 2024-09-13T11:22:43.246 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.246 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.246 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:22:43.246 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.319352+0000", 2024-09-13T11:22:43.246 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3774999999999999e-05 2024-09-13T11:22:43.246 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.246 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.246 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:22:43.246 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.340424+0000", 2024-09-13T11:22:43.247 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.021072312999999999 2024-09-13T11:22:43.247 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.247 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.247 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:22:43.247 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.340636+0000", 2024-09-13T11:22:43.247 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00021209399999999999 2024-09-13T11:22:43.247 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.247 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.247 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 0", 2024-09-13T11:22:43.248 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.340826+0000", 2024-09-13T11:22:43.248 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00018942199999999999 2024-09-13T11:22:43.248 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.248 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.248 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:22:43.248 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.348717+0000", 2024-09-13T11:22:43.248 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0078914250000000005 2024-09-13T11:22:43.248 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.248 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.248 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:22:43.248 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.348755+0000", 2024-09-13T11:22:43.248 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.8290000000000001e-05 2024-09-13T11:22:43.248 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.248 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.249 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:22:43.249 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.348877+0000", 2024-09-13T11:22:43.249 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00012210500000000001 2024-09-13T11:22:43.249 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.249 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.249 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:22:43.249 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.348992+0000", 2024-09-13T11:22:43.249 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000114733 2024-09-13T11:22:43.249 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:22:43.249 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:22:43.249 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:22:43.249 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.249 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.249 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4257.0:46 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-13T11:22:43.250 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:22:12.319370+0000", 2024-09-13T11:22:43.250 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 30.918677070000001, 2024-09-13T11:22:43.250 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.029891220999999999, 2024-09-13T11:22:43.250 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:22:43.250 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:22:43.250 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:43.250 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:22:43.250 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4257", 2024-09-13T11:22:43.250 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/1488131527", 2024-09-13T11:22:43.250 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 46 2024-09-13T11:22:43.250 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.250 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:22:43.250 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.250 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:22:43.251 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.319370+0000", 2024-09-13T11:22:43.251 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:22:43.251 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.251 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.251 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:22:43.251 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.319370+0000", 2024-09-13T11:22:43.251 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:22:43.251 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.251 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.251 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:22:43.251 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.319372+0000", 2024-09-13T11:22:43.251 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.81e-06 2024-09-13T11:22:43.251 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.252 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.252 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:22:43.252 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.319379+0000", 2024-09-13T11:22:43.252 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.5619999999999998e-06 2024-09-13T11:22:43.252 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.252 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.252 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:22:43.252 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.319395+0000", 2024-09-13T11:22:43.252 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.6262000000000001e-05 2024-09-13T11:22:43.252 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.252 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.252 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:22:43.252 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.319408+0000", 2024-09-13T11:22:43.253 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3069e-05 2024-09-13T11:22:43.253 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.253 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.253 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:22:43.253 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.341763+0000", 2024-09-13T11:22:43.253 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.022354707000000001 2024-09-13T11:22:43.253 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.253 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.253 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:22:43.253 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.341899+0000", 2024-09-13T11:22:43.253 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000135504 2024-09-13T11:22:43.253 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.253 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.254 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 0", 2024-09-13T11:22:43.254 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.342147+0000", 2024-09-13T11:22:43.254 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00024821700000000001 2024-09-13T11:22:43.254 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.254 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.254 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:22:43.254 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.349019+0000", 2024-09-13T11:22:43.254 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0068718529999999998 2024-09-13T11:22:43.254 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.254 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.254 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:22:43.254 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.349086+0000", 2024-09-13T11:22:43.254 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.7026999999999994e-05 2024-09-13T11:22:43.254 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.255 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.255 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:22:43.255 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.349164+0000", 2024-09-13T11:22:43.255 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.8479000000000005e-05 2024-09-13T11:22:43.255 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.255 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.255 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:22:43.255 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.349261+0000", 2024-09-13T11:22:43.255 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.6731000000000006e-05 2024-09-13T11:22:43.255 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:22:43.255 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:22:43.255 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:22:43.255 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.255 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.256 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4257.0:47 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-13T11:22:43.256 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:22:12.319435+0000", 2024-09-13T11:22:43.256 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 30.918612143000001, 2024-09-13T11:22:43.256 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.037475999000000003, 2024-09-13T11:22:43.256 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:22:43.256 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:22:43.256 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:43.256 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:22:43.256 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4257", 2024-09-13T11:22:43.256 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/1488131527", 2024-09-13T11:22:43.256 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 47 2024-09-13T11:22:43.256 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.256 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:22:43.256 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.257 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:22:43.257 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.319435+0000", 2024-09-13T11:22:43.257 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:22:43.257 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.257 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.257 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:22:43.257 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.319435+0000", 2024-09-13T11:22:43.257 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:22:43.257 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.257 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.257 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:22:43.257 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.319437+0000", 2024-09-13T11:22:43.257 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.9400000000000001e-06 2024-09-13T11:22:43.257 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.258 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.258 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:22:43.258 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.319521+0000", 2024-09-13T11:22:43.258 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.4178999999999995e-05 2024-09-13T11:22:43.259 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.259 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.259 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:22:43.259 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.319543+0000", 2024-09-13T11:22:43.259 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.1750999999999999e-05 2024-09-13T11:22:43.259 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.259 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.259 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:22:43.259 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.319561+0000", 2024-09-13T11:22:43.259 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.8518000000000001e-05 2024-09-13T11:22:43.259 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.259 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.259 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:22:43.260 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.345664+0000", 2024-09-13T11:22:43.260 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.026102914000000001 2024-09-13T11:22:43.260 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.260 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.260 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:22:43.260 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.345799+0000", 2024-09-13T11:22:43.260 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00013452000000000001 2024-09-13T11:22:43.260 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.260 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.260 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 0", 2024-09-13T11:22:43.260 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.346009+0000", 2024-09-13T11:22:43.260 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00021038799999999999 2024-09-13T11:22:43.260 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.261 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.261 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:22:43.261 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.349395+0000", 2024-09-13T11:22:43.261 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0033864759999999998 2024-09-13T11:22:43.261 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.261 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.261 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:22:43.261 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.356667+0000", 2024-09-13T11:22:43.261 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0072719159999999998 2024-09-13T11:22:43.261 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.261 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.261 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:22:43.261 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.356801+0000", 2024-09-13T11:22:43.261 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00013402999999999999 2024-09-13T11:22:43.262 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.262 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.262 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:22:43.262 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.356911+0000", 2024-09-13T11:22:43.262 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000109367 2024-09-13T11:22:43.262 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:22:43.262 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:22:43.262 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:22:43.262 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.262 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.262 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4257.0:48 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-13T11:22:43.262 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:22:12.319584+0000", 2024-09-13T11:22:43.262 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 30.918462643000002, 2024-09-13T11:22:43.263 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.037877980999999998, 2024-09-13T11:22:43.263 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:22:43.263 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:22:43.263 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:43.263 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:22:43.263 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4257", 2024-09-13T11:22:43.263 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/1488131527", 2024-09-13T11:22:43.263 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 48 2024-09-13T11:22:43.263 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.263 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:22:43.263 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.263 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:22:43.263 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.319584+0000", 2024-09-13T11:22:43.264 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:22:43.264 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.264 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.264 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:22:43.264 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.319584+0000", 2024-09-13T11:22:43.264 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:22:43.264 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.264 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.264 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:22:43.264 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.319586+0000", 2024-09-13T11:22:43.264 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.013e-06 2024-09-13T11:22:43.264 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.264 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.264 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:22:43.265 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.319595+0000", 2024-09-13T11:22:43.265 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.9060000000000008e-06 2024-09-13T11:22:43.265 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.265 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.265 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:22:43.265 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.319615+0000", 2024-09-13T11:22:43.265 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.0378e-05 2024-09-13T11:22:43.265 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.265 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.265 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:22:43.265 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.319632+0000", 2024-09-13T11:22:43.265 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.6858e-05 2024-09-13T11:22:43.265 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.266 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.266 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:22:43.266 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.349488+0000", 2024-09-13T11:22:43.266 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.029856038000000001 2024-09-13T11:22:43.266 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.266 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.266 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:22:43.266 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.349638+0000", 2024-09-13T11:22:43.266 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00014941600000000001 2024-09-13T11:22:43.266 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.266 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.266 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 0", 2024-09-13T11:22:43.266 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.349838+0000", 2024-09-13T11:22:43.266 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000199929 2024-09-13T11:22:43.267 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.267 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.267 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:22:43.267 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.357028+0000", 2024-09-13T11:22:43.267 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0071900799999999997 2024-09-13T11:22:43.267 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.267 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.267 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:22:43.267 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.357243+0000", 2024-09-13T11:22:43.267 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00021529000000000001 2024-09-13T11:22:43.267 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.267 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.267 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:22:43.268 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.357365+0000", 2024-09-13T11:22:43.268 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00012191199999999999 2024-09-13T11:22:43.268 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.268 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.268 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:22:43.268 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.357462+0000", 2024-09-13T11:22:43.268 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.7161e-05 2024-09-13T11:22:43.268 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:22:43.268 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:22:43.268 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:22:43.268 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.268 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.268 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4257.0:49 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-13T11:22:43.268 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:22:12.319663+0000", 2024-09-13T11:22:43.269 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 30.918383603999999, 2024-09-13T11:22:43.269 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.046281448000000003, 2024-09-13T11:22:43.269 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:22:43.269 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:22:43.269 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:43.269 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:22:43.269 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4257", 2024-09-13T11:22:43.269 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/1488131527", 2024-09-13T11:22:43.269 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 49 2024-09-13T11:22:43.269 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.269 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:22:43.269 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.269 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:22:43.269 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.319663+0000", 2024-09-13T11:22:43.270 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:22:43.270 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.270 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.270 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:22:43.270 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.319663+0000", 2024-09-13T11:22:43.270 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:22:43.270 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.270 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.270 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:22:43.270 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.319665+0000", 2024-09-13T11:22:43.270 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.0030000000000001e-06 2024-09-13T11:22:43.270 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.270 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.271 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:22:43.271 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.319675+0000", 2024-09-13T11:22:43.271 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0019e-05 2024-09-13T11:22:43.271 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.271 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.271 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:22:43.271 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.319697+0000", 2024-09-13T11:22:43.271 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.1387999999999999e-05 2024-09-13T11:22:43.271 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.271 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.271 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:22:43.271 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.319712+0000", 2024-09-13T11:22:43.271 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5068999999999999e-05 2024-09-13T11:22:43.271 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.272 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.272 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:22:43.272 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.352248+0000", 2024-09-13T11:22:43.272 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.032536460000000003 2024-09-13T11:22:43.272 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.272 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.272 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:22:43.272 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.352479+0000", 2024-09-13T11:22:43.272 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00023091499999999999 2024-09-13T11:22:43.272 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.272 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.272 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 0", 2024-09-13T11:22:43.272 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.352882+0000", 2024-09-13T11:22:43.273 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00040249300000000002 2024-09-13T11:22:43.273 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.273 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.273 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:22:43.273 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.357206+0000", 2024-09-13T11:22:43.273 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.004324802 2024-09-13T11:22:43.273 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.273 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.273 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:22:43.273 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.365708+0000", 2024-09-13T11:22:43.273 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0085012509999999996 2024-09-13T11:22:43.273 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.273 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.274 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:22:43.274 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.365839+0000", 2024-09-13T11:22:43.274 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000131202 2024-09-13T11:22:43.274 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.274 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.274 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:22:43.274 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.365945+0000", 2024-09-13T11:22:43.274 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000105846 2024-09-13T11:22:43.274 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:22:43.274 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:22:43.274 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:22:43.274 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.274 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.274 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4257.0:50 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-13T11:22:43.275 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:22:12.319731+0000", 2024-09-13T11:22:43.275 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 30.918316117, 2024-09-13T11:22:43.275 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.046650321000000002, 2024-09-13T11:22:43.275 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:22:43.275 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:22:43.275 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:43.275 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:22:43.275 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4257", 2024-09-13T11:22:43.275 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/1488131527", 2024-09-13T11:22:43.275 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 50 2024-09-13T11:22:43.275 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.275 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:22:43.275 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.275 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:22:43.276 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.319731+0000", 2024-09-13T11:22:43.276 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:22:43.276 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.276 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.276 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:22:43.276 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.319731+0000", 2024-09-13T11:22:43.276 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:22:43.276 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.276 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.276 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:22:43.276 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.319732+0000", 2024-09-13T11:22:43.276 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.75e-06 2024-09-13T11:22:43.276 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.276 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.277 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:22:43.277 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.319802+0000", 2024-09-13T11:22:43.277 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.9166999999999997e-05 2024-09-13T11:22:43.277 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.277 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.277 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:22:43.277 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.319826+0000", 2024-09-13T11:22:43.277 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.4464000000000002e-05 2024-09-13T11:22:43.277 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.277 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.277 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:22:43.277 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.319856+0000", 2024-09-13T11:22:43.277 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.9530999999999999e-05 2024-09-13T11:22:43.277 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.278 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.278 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:22:43.278 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.357551+0000", 2024-09-13T11:22:43.278 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.037695687999999998 2024-09-13T11:22:43.278 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.278 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.278 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:22:43.278 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.357707+0000", 2024-09-13T11:22:43.278 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00015562200000000001 2024-09-13T11:22:43.278 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.278 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.278 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 0", 2024-09-13T11:22:43.278 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.357906+0000", 2024-09-13T11:22:43.279 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00019874500000000001 2024-09-13T11:22:43.279 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.279 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.279 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:22:43.279 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.366111+0000", 2024-09-13T11:22:43.279 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.008205815 2024-09-13T11:22:43.279 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.279 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.279 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:22:43.279 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.366149+0000", 2024-09-13T11:22:43.279 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.7113999999999997e-05 2024-09-13T11:22:43.279 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.279 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.279 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:22:43.279 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.366283+0000", 2024-09-13T11:22:43.280 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00013399399999999999 2024-09-13T11:22:43.280 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.280 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.280 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:22:43.280 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.366381+0000", 2024-09-13T11:22:43.280 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.8431000000000007e-05 2024-09-13T11:22:43.280 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:22:43.280 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:22:43.280 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:22:43.280 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.280 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.280 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4257.0:51 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-13T11:22:43.280 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:22:12.319877+0000", 2024-09-13T11:22:43.280 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 30.918169487, 2024-09-13T11:22:43.281 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.048107695999999998, 2024-09-13T11:22:43.281 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:22:43.281 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:22:43.281 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:43.281 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:22:43.281 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4257", 2024-09-13T11:22:43.281 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/1488131527", 2024-09-13T11:22:43.281 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 51 2024-09-13T11:22:43.281 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.281 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:22:43.282 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.282 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:22:43.282 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.319877+0000", 2024-09-13T11:22:43.282 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:22:43.282 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.282 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.282 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:22:43.282 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.319877+0000", 2024-09-13T11:22:43.282 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:22:43.282 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.282 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.282 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:22:43.282 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.319879+0000", 2024-09-13T11:22:43.283 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.44e-06 2024-09-13T11:22:43.283 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.283 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.283 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:22:43.283 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.319885+0000", 2024-09-13T11:22:43.283 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.8690000000000001e-06 2024-09-13T11:22:43.283 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.283 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.283 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:22:43.283 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.319894+0000", 2024-09-13T11:22:43.283 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.7389999999999994e-06 2024-09-13T11:22:43.283 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.284 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.284 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:22:43.284 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.319903+0000", 2024-09-13T11:22:43.284 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.2830000000000001e-06 2024-09-13T11:22:43.284 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.284 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.284 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:22:43.284 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.361846+0000", 2024-09-13T11:22:43.284 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.041943041 2024-09-13T11:22:43.284 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.284 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.284 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:22:43.284 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.362090+0000", 2024-09-13T11:22:43.284 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00024382399999999999 2024-09-13T11:22:43.285 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.285 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.285 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 0", 2024-09-13T11:22:43.285 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.362493+0000", 2024-09-13T11:22:43.285 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00040374600000000002 2024-09-13T11:22:43.285 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.285 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.285 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:22:43.285 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.366507+0000", 2024-09-13T11:22:43.285 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0040133369999999996 2024-09-13T11:22:43.285 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.285 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.285 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:22:43.286 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.367751+0000", 2024-09-13T11:22:43.286 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001244118 2024-09-13T11:22:43.286 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.286 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.286 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:22:43.286 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.367873+0000", 2024-09-13T11:22:43.286 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000121972 2024-09-13T11:22:43.286 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.286 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.286 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:22:43.286 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.367985+0000", 2024-09-13T11:22:43.286 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000112327 2024-09-13T11:22:43.286 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:22:43.286 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:22:43.287 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:22:43.287 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.287 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.287 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4257.0:52 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-13T11:22:43.287 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:22:12.319912+0000", 2024-09-13T11:22:43.287 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 30.91813483, 2024-09-13T11:22:43.287 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.049068963, 2024-09-13T11:22:43.287 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:22:43.287 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:22:43.287 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:43.287 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:22:43.287 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4257", 2024-09-13T11:22:43.288 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/1488131527", 2024-09-13T11:22:43.288 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 52 2024-09-13T11:22:43.288 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.288 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:22:43.288 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.288 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:22:43.288 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.319912+0000", 2024-09-13T11:22:43.288 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:22:43.288 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.288 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.288 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:22:43.288 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.319912+0000", 2024-09-13T11:22:43.288 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:22:43.289 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.289 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.289 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:22:43.289 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.319913+0000", 2024-09-13T11:22:43.289 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.133e-06 2024-09-13T11:22:43.289 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.289 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.289 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:22:43.289 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.319918+0000", 2024-09-13T11:22:43.289 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.7729999999999999e-06 2024-09-13T11:22:43.289 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.289 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.289 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:22:43.290 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.319927+0000", 2024-09-13T11:22:43.290 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.6750000000000008e-06 2024-09-13T11:22:43.290 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.290 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.290 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:22:43.290 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.319935+0000", 2024-09-13T11:22:43.290 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.9500000000000001e-06 2024-09-13T11:22:43.290 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.290 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.290 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:22:43.290 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.366585+0000", 2024-09-13T11:22:43.290 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.046649991000000002 2024-09-13T11:22:43.290 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.291 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.291 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:22:43.291 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.366832+0000", 2024-09-13T11:22:43.291 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00024709999999999999 2024-09-13T11:22:43.291 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.291 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.291 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 0", 2024-09-13T11:22:43.291 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.367210+0000", 2024-09-13T11:22:43.291 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00037837199999999997 2024-09-13T11:22:43.291 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.291 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.291 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:22:43.291 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.368430+0000", 2024-09-13T11:22:43.291 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00122013 2024-09-13T11:22:43.292 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.292 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.292 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:22:43.292 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.368787+0000", 2024-09-13T11:22:43.292 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00035728300000000001 2024-09-13T11:22:43.292 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.292 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.292 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:22:43.292 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.368874+0000", 2024-09-13T11:22:43.292 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.6876e-05 2024-09-13T11:22:43.292 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.292 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.292 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:22:43.293 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.368981+0000", 2024-09-13T11:22:43.293 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00010668 2024-09-13T11:22:43.293 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:22:43.293 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:22:43.293 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:22:43.293 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.293 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.293 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4257.0:70 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-13T11:22:43.293 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:22:12.458708+0000", 2024-09-13T11:22:43.293 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 30.779338873, 2024-09-13T11:22:43.293 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.036822555, 2024-09-13T11:22:43.293 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:22:43.293 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:22:43.293 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:43.294 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:22:43.294 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4257", 2024-09-13T11:22:43.294 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/1488131527", 2024-09-13T11:22:43.294 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 70 2024-09-13T11:22:43.294 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.294 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:22:43.294 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.294 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:22:43.294 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.458708+0000", 2024-09-13T11:22:43.294 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:22:43.294 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.294 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.294 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:22:43.294 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.458708+0000", 2024-09-13T11:22:43.295 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:22:43.295 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.295 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.295 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:22:43.295 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.458709+0000", 2024-09-13T11:22:43.295 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.316e-06 2024-09-13T11:22:43.295 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.295 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.295 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:22:43.295 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.458717+0000", 2024-09-13T11:22:43.295 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.3390000000000004e-06 2024-09-13T11:22:43.295 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.295 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.295 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:22:43.296 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.458726+0000", 2024-09-13T11:22:43.296 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.2329999999999995e-06 2024-09-13T11:22:43.296 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.296 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.296 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:22:43.296 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.458733+0000", 2024-09-13T11:22:43.296 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.5959999999999998e-06 2024-09-13T11:22:43.296 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.296 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.296 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:22:43.296 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.481591+0000", 2024-09-13T11:22:43.296 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.022857796999999999 2024-09-13T11:22:43.296 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.296 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.297 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:22:43.297 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.481749+0000", 2024-09-13T11:22:43.297 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00015727800000000001 2024-09-13T11:22:43.297 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.297 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.297 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 0", 2024-09-13T11:22:43.297 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.481952+0000", 2024-09-13T11:22:43.297 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00020356500000000001 2024-09-13T11:22:43.297 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.297 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.297 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:22:43.297 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.486070+0000", 2024-09-13T11:22:43.297 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0041175689999999997 2024-09-13T11:22:43.297 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.298 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.298 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:22:43.298 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.495291+0000", 2024-09-13T11:22:43.298 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0092216499999999996 2024-09-13T11:22:43.298 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.298 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.298 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:22:43.298 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.495423+0000", 2024-09-13T11:22:43.298 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000131908 2024-09-13T11:22:43.298 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.298 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.298 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:22:43.298 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.495531+0000", 2024-09-13T11:22:43.298 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000107304 2024-09-13T11:22:43.298 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:22:43.299 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:22:43.299 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:22:43.299 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.299 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.299 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4257.0:71 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-13T11:22:43.299 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:22:12.458743+0000", 2024-09-13T11:22:43.299 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 30.77930404, 2024-09-13T11:22:43.299 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.037378219999999997, 2024-09-13T11:22:43.299 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:22:43.299 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:22:43.299 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:43.299 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:22:43.299 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4257", 2024-09-13T11:22:43.299 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/1488131527", 2024-09-13T11:22:43.300 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 71 2024-09-13T11:22:43.300 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.300 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:22:43.300 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.300 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:22:43.300 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.458743+0000", 2024-09-13T11:22:43.300 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:22:43.300 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.300 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.300 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:22:43.300 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.458743+0000", 2024-09-13T11:22:43.300 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:22:43.300 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.301 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.301 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:22:43.301 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.458744+0000", 2024-09-13T11:22:43.301 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.7000000000000003e-07 2024-09-13T11:22:43.301 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.301 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.301 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:22:43.301 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.458748+0000", 2024-09-13T11:22:43.301 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.1060000000000002e-06 2024-09-13T11:22:43.301 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.301 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.301 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:22:43.301 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.458760+0000", 2024-09-13T11:22:43.301 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2316e-05 2024-09-13T11:22:43.302 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.302 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.302 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:22:43.302 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.458768+0000", 2024-09-13T11:22:43.302 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.3690000000000001e-06 2024-09-13T11:22:43.302 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.302 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.302 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:22:43.302 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.486413+0000", 2024-09-13T11:22:43.302 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.027645187000000002 2024-09-13T11:22:43.302 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.302 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.302 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:22:43.303 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.486549+0000", 2024-09-13T11:22:43.303 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000135907 2024-09-13T11:22:43.303 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.303 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.303 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 0", 2024-09-13T11:22:43.303 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.486753+0000", 2024-09-13T11:22:43.303 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00020425700000000001 2024-09-13T11:22:43.303 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.303 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.303 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:22:43.304 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.495649+0000", 2024-09-13T11:22:43.304 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0088957589999999996 2024-09-13T11:22:43.304 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.304 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.304 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:22:43.304 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.495901+0000", 2024-09-13T11:22:43.304 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00025266299999999998 2024-09-13T11:22:43.304 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.304 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.304 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:22:43.304 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.496023+0000", 2024-09-13T11:22:43.304 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000121748 2024-09-13T11:22:43.304 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.305 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.305 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:22:43.305 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.496121+0000", 2024-09-13T11:22:43.305 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.7937999999999996e-05 2024-09-13T11:22:43.305 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:22:43.305 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:22:43.305 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:22:43.305 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.305 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.305 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4257.0:72 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-13T11:22:43.305 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:22:12.458777+0000", 2024-09-13T11:22:43.305 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 30.779269982999999, 2024-09-13T11:22:43.305 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.041935317999999999, 2024-09-13T11:22:43.305 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:22:43.306 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:22:43.306 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:43.306 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:22:43.306 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4257", 2024-09-13T11:22:43.306 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/1488131527", 2024-09-13T11:22:43.306 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 72 2024-09-13T11:22:43.306 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.306 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:22:43.306 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.306 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:22:43.306 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.458777+0000", 2024-09-13T11:22:43.306 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:22:43.306 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.306 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.307 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:22:43.307 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.458777+0000", 2024-09-13T11:22:43.307 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:22:43.307 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.307 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.307 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:22:43.307 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.458808+0000", 2024-09-13T11:22:43.307 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.0904000000000001e-05 2024-09-13T11:22:43.307 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.307 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.307 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:22:43.307 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.458904+0000", 2024-09-13T11:22:43.307 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.6547999999999998e-05 2024-09-13T11:22:43.307 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.308 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.308 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:22:43.308 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.458918+0000", 2024-09-13T11:22:43.308 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3237999999999999e-05 2024-09-13T11:22:43.308 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.308 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.308 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:22:43.308 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.458926+0000", 2024-09-13T11:22:43.308 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.6929999999999996e-06 2024-09-13T11:22:43.308 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.308 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.308 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:22:43.308 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.490489+0000", 2024-09-13T11:22:43.309 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.031563015 2024-09-13T11:22:43.309 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.309 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.309 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:22:43.309 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.490709+0000", 2024-09-13T11:22:43.309 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00022010700000000001 2024-09-13T11:22:43.309 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.309 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.309 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 0", 2024-09-13T11:22:43.309 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.491113+0000", 2024-09-13T11:22:43.309 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00040412899999999998 2024-09-13T11:22:43.309 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.309 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.309 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:22:43.310 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.495864+0000", 2024-09-13T11:22:43.310 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0047506459999999999 2024-09-13T11:22:43.310 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.310 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.310 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:22:43.310 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.500468+0000", 2024-09-13T11:22:43.310 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0046035670000000002 2024-09-13T11:22:43.310 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.310 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.310 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:22:43.310 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.500598+0000", 2024-09-13T11:22:43.310 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000130541 2024-09-13T11:22:43.310 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.310 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.310 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:22:43.311 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.500712+0000", 2024-09-13T11:22:43.311 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00011393 2024-09-13T11:22:43.311 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:22:43.311 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:22:43.311 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:22:43.311 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.311 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.311 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4257.0:73 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-13T11:22:43.311 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:22:12.458939+0000", 2024-09-13T11:22:43.311 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 30.779107665000002, 2024-09-13T11:22:43.311 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.050922262000000003, 2024-09-13T11:22:43.311 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:22:43.311 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:22:43.311 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:43.312 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:22:43.312 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4257", 2024-09-13T11:22:43.312 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/1488131527", 2024-09-13T11:22:43.312 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 73 2024-09-13T11:22:43.312 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.312 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:22:43.312 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.312 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:22:43.312 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.458939+0000", 2024-09-13T11:22:43.312 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:22:43.312 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.312 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.312 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:22:43.312 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.458939+0000", 2024-09-13T11:22:43.313 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:22:43.313 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.313 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.313 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:22:43.313 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.458940+0000", 2024-09-13T11:22:43.313 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.22e-06 2024-09-13T11:22:43.313 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.313 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.313 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:22:43.313 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.458945+0000", 2024-09-13T11:22:43.313 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.6800000000000001e-06 2024-09-13T11:22:43.313 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.313 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.313 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:22:43.314 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.458954+0000", 2024-09-13T11:22:43.314 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.4589999999999998e-06 2024-09-13T11:22:43.314 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.314 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.314 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:22:43.314 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.458961+0000", 2024-09-13T11:22:43.314 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.2420000000000003e-06 2024-09-13T11:22:43.314 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.314 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.314 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:22:43.314 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.496214+0000", 2024-09-13T11:22:43.314 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.037252805999999999 2024-09-13T11:22:43.314 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.314 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.314 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:22:43.315 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.496446+0000", 2024-09-13T11:22:43.315 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00023255800000000001 2024-09-13T11:22:43.315 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.315 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.315 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 0", 2024-09-13T11:22:43.315 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.496800+0000", 2024-09-13T11:22:43.315 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00035397400000000001 2024-09-13T11:22:43.315 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.315 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.315 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:22:43.315 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.504887+0000", 2024-09-13T11:22:43.315 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0080867769999999999 2024-09-13T11:22:43.315 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.315 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.316 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:22:43.316 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.509579+0000", 2024-09-13T11:22:43.316 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0046924849999999997 2024-09-13T11:22:43.316 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.316 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.316 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:22:43.316 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.509748+0000", 2024-09-13T11:22:43.316 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00016907399999999999 2024-09-13T11:22:43.316 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.317 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.317 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:22:43.317 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.509861+0000", 2024-09-13T11:22:43.317 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00011298699999999999 2024-09-13T11:22:43.317 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:22:43.317 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:22:43.317 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:22:43.317 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.317 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.317 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4257.0:74 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-13T11:22:43.317 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:22:12.458973+0000", 2024-09-13T11:22:43.317 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 30.779074245, 2024-09-13T11:22:43.317 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.055492705000000003, 2024-09-13T11:22:43.317 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:22:43.318 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:22:43.318 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:43.318 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:22:43.318 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4257", 2024-09-13T11:22:43.318 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/1488131527", 2024-09-13T11:22:43.318 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 74 2024-09-13T11:22:43.318 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.318 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:22:43.318 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.318 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:22:43.318 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.458973+0000", 2024-09-13T11:22:43.318 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:22:43.318 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.318 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.319 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:22:43.319 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.458973+0000", 2024-09-13T11:22:43.319 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:22:43.319 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.319 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.319 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:22:43.319 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.458974+0000", 2024-09-13T11:22:43.319 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.3699999999999999e-07 2024-09-13T11:22:43.319 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.319 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.319 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:22:43.319 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.458978+0000", 2024-09-13T11:22:43.319 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.4259999999999996e-06 2024-09-13T11:22:43.319 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.319 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.320 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:22:43.320 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.458994+0000", 2024-09-13T11:22:43.320 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5852e-05 2024-09-13T11:22:43.320 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.320 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.320 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:22:43.320 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.459002+0000", 2024-09-13T11:22:43.320 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.5830000000000007e-06 2024-09-13T11:22:43.320 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.320 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.320 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:22:43.320 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.500820+0000", 2024-09-13T11:22:43.320 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.041817432000000002 2024-09-13T11:22:43.321 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.321 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.321 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:22:43.321 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.500973+0000", 2024-09-13T11:22:43.321 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00015326200000000001 2024-09-13T11:22:43.321 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.321 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.321 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 0", 2024-09-13T11:22:43.321 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.501212+0000", 2024-09-13T11:22:43.321 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00023885099999999999 2024-09-13T11:22:43.321 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.321 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.321 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:22:43.322 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.505008+0000", 2024-09-13T11:22:43.322 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0037961850000000001 2024-09-13T11:22:43.322 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.322 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.322 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:22:43.322 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.514227+0000", 2024-09-13T11:22:43.322 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0092187889999999998 2024-09-13T11:22:43.322 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.322 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.322 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:22:43.322 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.514362+0000", 2024-09-13T11:22:43.322 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000134967 2024-09-13T11:22:43.322 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.322 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.323 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:22:43.323 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.514465+0000", 2024-09-13T11:22:43.323 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000103421 2024-09-13T11:22:43.323 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:22:43.323 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:22:43.323 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:22:43.323 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.323 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.323 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4257.0:75 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-13T11:22:43.323 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:22:12.459011+0000", 2024-09-13T11:22:43.323 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 30.779035348000001, 2024-09-13T11:22:43.323 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.056002977000000002, 2024-09-13T11:22:43.323 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:22:43.323 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:22:43.324 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:43.324 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:22:43.324 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4257", 2024-09-13T11:22:43.324 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/1488131527", 2024-09-13T11:22:43.324 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 75 2024-09-13T11:22:43.324 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.324 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:22:43.324 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.324 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:22:43.324 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.459011+0000", 2024-09-13T11:22:43.324 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:22:43.324 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.324 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.325 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:22:43.325 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.459011+0000", 2024-09-13T11:22:43.325 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:22:43.325 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.325 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.325 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:22:43.325 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.459012+0000", 2024-09-13T11:22:43.325 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.9999999999999995e-07 2024-09-13T11:22:43.325 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.325 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.325 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:22:43.325 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.459060+0000", 2024-09-13T11:22:43.325 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.7587999999999999e-05 2024-09-13T11:22:43.325 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.326 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.326 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:22:43.326 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.459070+0000", 2024-09-13T11:22:43.326 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.9690000000000006e-06 2024-09-13T11:22:43.326 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.326 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.326 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:22:43.326 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.459078+0000", 2024-09-13T11:22:43.326 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.5329999999999997e-06 2024-09-13T11:22:43.326 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.326 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.326 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:22:43.326 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.505105+0000", 2024-09-13T11:22:43.326 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.046027908999999999 2024-09-13T11:22:43.327 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.327 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.327 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:22:43.327 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.505322+0000", 2024-09-13T11:22:43.327 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00021677600000000001 2024-09-13T11:22:43.327 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.327 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.327 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 0", 2024-09-13T11:22:43.327 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.505675+0000", 2024-09-13T11:22:43.327 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00035293800000000001 2024-09-13T11:22:43.327 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.327 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.327 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:22:43.327 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.514620+0000", 2024-09-13T11:22:43.328 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0089446730000000002 2024-09-13T11:22:43.328 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.328 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.328 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:22:43.328 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.514777+0000", 2024-09-13T11:22:43.328 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00015762999999999999 2024-09-13T11:22:43.328 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.328 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.328 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:22:43.328 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.514903+0000", 2024-09-13T11:22:43.328 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00012559800000000001 2024-09-13T11:22:43.328 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.328 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.329 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:22:43.329 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.515014+0000", 2024-09-13T11:22:43.329 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000111363 2024-09-13T11:22:43.329 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:22:43.329 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:22:43.329 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:22:43.329 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.329 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.329 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4257.0:76 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-13T11:22:43.329 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:22:12.459088+0000", 2024-09-13T11:22:43.329 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 30.778958638999999, 2024-09-13T11:22:43.329 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.056165815000000001, 2024-09-13T11:22:43.329 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:22:43.329 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:22:43.330 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:43.330 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:22:43.330 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4257", 2024-09-13T11:22:43.330 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/1488131527", 2024-09-13T11:22:43.330 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 76 2024-09-13T11:22:43.330 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.330 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:22:43.330 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.330 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:22:43.330 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.459088+0000", 2024-09-13T11:22:43.330 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:22:43.330 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.330 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.330 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:22:43.331 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.459088+0000", 2024-09-13T11:22:43.331 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:22:43.331 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.331 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.331 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:22:43.331 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.459089+0000", 2024-09-13T11:22:43.331 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.114e-06 2024-09-13T11:22:43.331 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.331 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.331 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:22:43.331 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.459094+0000", 2024-09-13T11:22:43.331 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.0320000000000003e-06 2024-09-13T11:22:43.331 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.331 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.332 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:22:43.332 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.459103+0000", 2024-09-13T11:22:43.332 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.2989999999999997e-06 2024-09-13T11:22:43.332 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.332 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.332 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:22:43.332 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.459110+0000", 2024-09-13T11:22:43.332 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.9970000000000001e-06 2024-09-13T11:22:43.332 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.332 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.332 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:22:43.332 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.509960+0000", 2024-09-13T11:22:43.332 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.050850601000000002 2024-09-13T11:22:43.332 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.333 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.333 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:22:43.333 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.510216+0000", 2024-09-13T11:22:43.333 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00025548299999999998 2024-09-13T11:22:43.333 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.333 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.333 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 0", 2024-09-13T11:22:43.333 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.510559+0000", 2024-09-13T11:22:43.333 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00034306199999999999 2024-09-13T11:22:43.333 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.333 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.333 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:22:43.333 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.514745+0000", 2024-09-13T11:22:43.334 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0041858920000000001 2024-09-13T11:22:43.334 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.334 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.334 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:22:43.334 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.515035+0000", 2024-09-13T11:22:43.334 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00029025600000000002 2024-09-13T11:22:43.334 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.334 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.334 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:22:43.334 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.515152+0000", 2024-09-13T11:22:43.334 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000116872 2024-09-13T11:22:43.334 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.334 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.334 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:22:43.335 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.515254+0000", 2024-09-13T11:22:43.335 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00010220700000000001 2024-09-13T11:22:43.335 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:22:43.335 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:22:43.335 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:22:43.335 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.335 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.335 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4257.0:80 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 262144~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-13T11:22:43.335 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:22:12.516065+0000", 2024-09-13T11:22:43.335 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 30.721981595999999, 2024-09-13T11:22:43.335 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.46461997999999999, 2024-09-13T11:22:43.335 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:22:43.335 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:22:43.335 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:43.336 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:22:43.336 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4257", 2024-09-13T11:22:43.336 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/1488131527", 2024-09-13T11:22:43.336 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 80 2024-09-13T11:22:43.336 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.336 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:22:43.336 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.336 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:22:43.336 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.516065+0000", 2024-09-13T11:22:43.336 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:22:43.336 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.336 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.336 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:22:43.336 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.516065+0000", 2024-09-13T11:22:43.337 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:22:43.337 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.337 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.337 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:22:43.337 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.516066+0000", 2024-09-13T11:22:43.337 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0869999999999999e-06 2024-09-13T11:22:43.337 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.337 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.337 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:22:43.337 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.516112+0000", 2024-09-13T11:22:43.337 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.5503000000000001e-05 2024-09-13T11:22:43.337 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.337 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.338 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:22:43.338 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.516128+0000", 2024-09-13T11:22:43.338 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5891000000000001e-05 2024-09-13T11:22:43.338 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.338 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.338 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:22:43.338 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.516138+0000", 2024-09-13T11:22:43.338 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.8120000000000002e-06 2024-09-13T11:22:43.338 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.338 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.338 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:22:43.338 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.520398+0000", 2024-09-13T11:22:43.338 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0042605450000000001 2024-09-13T11:22:43.338 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.338 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.339 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:22:43.339 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.520526+0000", 2024-09-13T11:22:43.339 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000128355 2024-09-13T11:22:43.339 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.339 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.339 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 0", 2024-09-13T11:22:43.339 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.520730+0000", 2024-09-13T11:22:43.339 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00020337799999999999 2024-09-13T11:22:43.339 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.339 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.339 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:22:43.339 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.522417+0000", 2024-09-13T11:22:43.339 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001687184 2024-09-13T11:22:43.340 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.340 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.340 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:22:43.340 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.980408+0000", 2024-09-13T11:22:43.340 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.45799068399999998 2024-09-13T11:22:43.340 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.340 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.340 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:22:43.340 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.980565+0000", 2024-09-13T11:22:43.340 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00015760899999999999 2024-09-13T11:22:43.340 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.340 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.340 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:22:43.340 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.980685+0000", 2024-09-13T11:22:43.341 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000119932 2024-09-13T11:22:43.341 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:22:43.341 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:22:43.341 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:22:43.341 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.341 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.341 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop_reply(client.4257.0:80 1.0 e24/23 ondisk, result = 0)", 2024-09-13T11:22:43.341 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:22:12.522938+0000", 2024-09-13T11:22:43.341 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 30.715108563000001, 2024-09-13T11:22:43.341 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.45776420600000001, 2024-09-13T11:22:43.341 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:22:43.341 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:22:43.341 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:22:43.341 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:22:43.342 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.342 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:22:43.342 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.522938+0000", 2024-09-13T11:22:43.342 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:22:43.342 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.342 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.342 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:22:43.342 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.522938+0000", 2024-09-13T11:22:43.342 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:22:43.342 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.342 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.342 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:22:43.342 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.522941+0000", 2024-09-13T11:22:43.343 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.0290000000000001e-06 2024-09-13T11:22:43.343 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.343 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.343 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:22:43.343 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.522950+0000", 2024-09-13T11:22:43.343 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.7499999999999992e-06 2024-09-13T11:22:43.343 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.343 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.343 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:22:43.343 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.980215+0000", 2024-09-13T11:22:43.343 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.45726523800000002 2024-09-13T11:22:43.343 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.343 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.343 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:22:43.344 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.980229+0000", 2024-09-13T11:22:43.344 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3549e-05 2024-09-13T11:22:43.344 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.344 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.344 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:22:43.344 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.980365+0000", 2024-09-13T11:22:43.344 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00013608700000000001 2024-09-13T11:22:43.344 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.344 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.344 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:22:43.344 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.980388+0000", 2024-09-13T11:22:43.344 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.3112000000000002e-05 2024-09-13T11:22:43.344 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.344 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.345 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:22:43.345 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.980702+0000", 2024-09-13T11:22:43.345 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00031444099999999999 2024-09-13T11:22:43.345 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:22:43.345 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:22:43.345 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:22:43.345 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.345 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.345 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4257.0:96 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-13T11:22:43.345 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:22:13.063144+0000", 2024-09-13T11:22:43.345 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 30.174903104999999, 2024-09-13T11:22:43.345 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.027851055999999999, 2024-09-13T11:22:43.345 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:22:43.346 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:22:43.346 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:43.346 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:22:43.348 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4257", 2024-09-13T11:22:43.348 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/1488131527", 2024-09-13T11:22:43.348 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 96 2024-09-13T11:22:43.348 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.348 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:22:43.348 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.348 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:22:43.348 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:13.063144+0000", 2024-09-13T11:22:43.348 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:22:43.348 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.348 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.349 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:22:43.349 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:13.063144+0000", 2024-09-13T11:22:43.349 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:22:43.349 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.349 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.349 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:22:43.349 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:13.063145+0000", 2024-09-13T11:22:43.349 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5400000000000001e-06 2024-09-13T11:22:43.349 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.349 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.349 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:22:43.349 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:13.063223+0000", 2024-09-13T11:22:43.349 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.7719000000000006e-05 2024-09-13T11:22:43.349 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.350 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.350 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:22:43.350 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:13.063234+0000", 2024-09-13T11:22:43.350 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0929e-05 2024-09-13T11:22:43.350 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.350 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.350 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:22:43.350 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:13.063240+0000", 2024-09-13T11:22:43.350 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.648e-06 2024-09-13T11:22:43.350 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.350 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.350 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:22:43.350 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:13.081483+0000", 2024-09-13T11:22:43.350 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.018243083 2024-09-13T11:22:43.350 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.351 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.351 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:22:43.351 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:13.081565+0000", 2024-09-13T11:22:43.351 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.2163999999999995e-05 2024-09-13T11:22:43.351 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.351 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.351 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 0", 2024-09-13T11:22:43.351 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:13.081689+0000", 2024-09-13T11:22:43.351 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00012389500000000001 2024-09-13T11:22:43.351 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.351 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.351 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:22:43.351 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:13.085208+0000", 2024-09-13T11:22:43.351 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0035195640000000002 2024-09-13T11:22:43.352 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.352 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.352 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:22:43.352 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:13.090892+0000", 2024-09-13T11:22:43.352 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0056841779999999998 2024-09-13T11:22:43.352 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.352 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.352 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:22:43.352 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:13.090955+0000", 2024-09-13T11:22:43.352 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.2087000000000004e-05 2024-09-13T11:22:43.352 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.352 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.352 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:22:43.352 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:13.090995+0000", 2024-09-13T11:22:43.353 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.0249e-05 2024-09-13T11:22:43.353 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:22:43.353 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:22:43.353 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:22:43.353 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.353 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.353 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4257.0:99 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-13T11:22:43.353 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:22:13.063500+0000", 2024-09-13T11:22:43.353 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 30.174546796000001, 2024-09-13T11:22:43.353 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.029416551999999999, 2024-09-13T11:22:43.353 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:22:43.353 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:22:43.353 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:43.353 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:22:43.354 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4257", 2024-09-13T11:22:43.354 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/1488131527", 2024-09-13T11:22:43.354 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 99 2024-09-13T11:22:43.354 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.354 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:22:43.354 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.354 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:22:43.354 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:13.063500+0000", 2024-09-13T11:22:43.354 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:22:43.354 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.354 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.354 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:22:43.354 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:13.063500+0000", 2024-09-13T11:22:43.354 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:22:43.355 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.355 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.355 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:22:43.355 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:13.063513+0000", 2024-09-13T11:22:43.355 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3004e-05 2024-09-13T11:22:43.355 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.355 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.355 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:22:43.355 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:13.063589+0000", 2024-09-13T11:22:43.355 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.5486000000000002e-05 2024-09-13T11:22:43.355 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.355 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.355 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:22:43.355 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:13.063599+0000", 2024-09-13T11:22:43.356 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0584000000000001e-05 2024-09-13T11:22:43.356 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.356 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.356 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:22:43.356 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:13.063605+0000", 2024-09-13T11:22:43.356 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.8220000000000003e-06 2024-09-13T11:22:43.356 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.356 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.356 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:22:43.356 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:13.088873+0000", 2024-09-13T11:22:43.356 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.025267989000000001 2024-09-13T11:22:43.356 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.356 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.356 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:22:43.357 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:13.088984+0000", 2024-09-13T11:22:43.357 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00011157100000000001 2024-09-13T11:22:43.357 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.357 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.357 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 0", 2024-09-13T11:22:43.357 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:13.089155+0000", 2024-09-13T11:22:43.357 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000170237 2024-09-13T11:22:43.357 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.357 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.357 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:22:43.357 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:13.091189+0000", 2024-09-13T11:22:43.357 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0020344149999999999 2024-09-13T11:22:43.357 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.357 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.358 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:22:43.358 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:13.092847+0000", 2024-09-13T11:22:43.358 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0016581079999999999 2024-09-13T11:22:43.358 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.358 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.358 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:22:43.358 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:13.092887+0000", 2024-09-13T11:22:43.358 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.9508000000000002e-05 2024-09-13T11:22:43.358 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.358 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.358 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:22:43.358 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:13.092917+0000", 2024-09-13T11:22:43.358 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.9828e-05 2024-09-13T11:22:43.359 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:22:43.359 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:22:43.359 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:22:43.359 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.359 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.359 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4257.0:100 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-13T11:22:43.359 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:22:13.063617+0000", 2024-09-13T11:22:43.359 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 30.174429881000002, 2024-09-13T11:22:43.359 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.029894398999999999, 2024-09-13T11:22:43.359 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:22:43.359 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:22:43.359 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:43.359 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:22:43.359 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4257", 2024-09-13T11:22:43.360 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/1488131527", 2024-09-13T11:22:43.360 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 100 2024-09-13T11:22:43.360 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.360 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:22:43.360 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.360 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:22:43.360 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:13.063617+0000", 2024-09-13T11:22:43.360 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:22:43.360 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.360 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.360 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:22:43.360 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:13.063617+0000", 2024-09-13T11:22:43.360 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:22:43.360 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.361 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.361 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:22:43.361 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:13.063618+0000", 2024-09-13T11:22:43.361 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.3099999999999997e-07 2024-09-13T11:22:43.361 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.361 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.361 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:22:43.361 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:13.063621+0000", 2024-09-13T11:22:43.361 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.7170000000000002e-06 2024-09-13T11:22:43.361 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.361 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.361 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:22:43.361 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:13.063628+0000", 2024-09-13T11:22:43.361 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.2419999999999997e-06 2024-09-13T11:22:43.362 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.362 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.362 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:22:43.362 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:13.063631+0000", 2024-09-13T11:22:43.362 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.743e-06 2024-09-13T11:22:43.362 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.362 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.362 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:22:43.362 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:13.091220+0000", 2024-09-13T11:22:43.362 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.027588867999999999 2024-09-13T11:22:43.362 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.362 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.362 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:22:43.362 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:13.091324+0000", 2024-09-13T11:22:43.363 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000103712 2024-09-13T11:22:43.363 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.363 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.363 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 0", 2024-09-13T11:22:43.363 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:13.091485+0000", 2024-09-13T11:22:43.363 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00016127599999999999 2024-09-13T11:22:43.363 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.363 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.363 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:22:43.363 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:13.093127+0000", 2024-09-13T11:22:43.363 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0016413980000000001 2024-09-13T11:22:43.363 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.363 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.363 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:22:43.364 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:13.093401+0000", 2024-09-13T11:22:43.364 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00027388299999999999 2024-09-13T11:22:43.364 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.364 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.364 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:22:43.364 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:13.093462+0000", 2024-09-13T11:22:43.364 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.1816999999999994e-05 2024-09-13T11:22:43.364 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.364 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.364 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:22:43.364 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:13.093511+0000", 2024-09-13T11:22:43.364 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.9011999999999998e-05 2024-09-13T11:22:43.364 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:22:43.364 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:22:43.365 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:22:43.365 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:22:43.365 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:22:43.365 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-13T11:22:43.367 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-09-13T11:22:43.367 INFO:tasks.ceph.osd.11.smithi177.stderr:2024-09-13T11:22:43.364+0000 7f24792c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26580) UID: 0 2024-09-13T11:22:43.467 INFO:tasks.ceph.osd.2.smithi162.stderr:2024-09-13T11:22:43.463+0000 7fe92b35a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26313) UID: 0 2024-09-13T11:22:43.488 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-13T11:22:43.488 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [], 2024-09-13T11:22:43.488 INFO:teuthology.orchestra.run.smithi162.stdout: "num_ops": 0 2024-09-13T11:22:43.488 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-13T11:22:43.502 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-09-13T11:22:43.589 INFO:tasks.ceph.osd.2.smithi162.stderr:2024-09-13T11:22:43.587+0000 7fe92b35a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26313) UID: 0 2024-09-13T11:22:43.643 INFO:tasks.ceph.osd.15.smithi177.stderr:2024-09-13T11:22:43.640+0000 7f9d618a4640 -1 osd.15 37 handle_osd_map injecting map crc failure 2024-09-13T11:22:43.644 INFO:tasks.ceph.osd.11.smithi177.stderr:2024-09-13T11:22:43.640+0000 7f2469cec640 -1 osd.11 37 handle_osd_map injecting map crc failure 2024-09-13T11:22:43.669 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-13T11:22:43.669 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [], 2024-09-13T11:22:43.669 INFO:teuthology.orchestra.run.smithi162.stdout: "complaint_time": 30, 2024-09-13T11:22:43.669 INFO:teuthology.orchestra.run.smithi162.stdout: "num_blocked_ops": 0 2024-09-13T11:22:43.669 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-13T11:22:43.670 INFO:tasks.ceph.osd.11.smithi177.stderr:2024-09-13T11:22:43.668+0000 7f24792c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26580) UID: 0 2024-09-13T11:22:43.683 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-09-13T11:22:43.773 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-09-13T11:22:43.771+0000 7f97befc9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26315) UID: 0 2024-09-13T11:22:43.851 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-13T11:22:43.852 INFO:teuthology.orchestra.run.smithi162.stdout: "size": 20, 2024-09-13T11:22:43.852 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 600, 2024-09-13T11:22:43.852 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [ 2024-09-13T11:22:43.852 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.852 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:5 3.3 3:c820cfec:::benchmark_data_smithi177_31116_object4:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-09-13T11:22:43.852 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:22:27.001508+0000", 2024-09-13T11:22:43.852 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 16.848261884999999, 2024-09-13T11:22:43.852 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0089235370000000005, 2024-09-13T11:22:43.852 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:22:43.852 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:22:43.853 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:43.853 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:22:43.853 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:22:43.853 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:22:43.853 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 5 2024-09-13T11:22:43.853 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.853 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:22:43.853 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.853 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:22:43.853 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.001508+0000", 2024-09-13T11:22:43.853 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:22:43.853 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.853 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.853 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:22:43.854 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.001508+0000", 2024-09-13T11:22:43.854 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:22:43.854 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.854 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.854 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:22:43.854 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.001512+0000", 2024-09-13T11:22:43.854 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.7730000000000001e-06 2024-09-13T11:22:43.854 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.854 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.854 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:22:43.854 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.001936+0000", 2024-09-13T11:22:43.854 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00042364399999999998 2024-09-13T11:22:43.854 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.854 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.854 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:22:43.855 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.001967+0000", 2024-09-13T11:22:43.855 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.0747999999999998e-05 2024-09-13T11:22:43.855 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.855 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.855 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:22:43.855 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.001984+0000", 2024-09-13T11:22:43.855 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.7028999999999999e-05 2024-09-13T11:22:43.855 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.855 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.855 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:22:43.855 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.002116+0000", 2024-09-13T11:22:43.855 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000132192 2024-09-13T11:22:43.855 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.855 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.856 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:22:43.856 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.002482+0000", 2024-09-13T11:22:43.856 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00036674899999999998 2024-09-13T11:22:43.856 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.856 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.856 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 7", 2024-09-13T11:22:43.856 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.002714+0000", 2024-09-13T11:22:43.856 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000231759 2024-09-13T11:22:43.856 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.856 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.856 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:22:43.856 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.006493+0000", 2024-09-13T11:22:43.856 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0037787250000000001 2024-09-13T11:22:43.857 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.857 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.857 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:22:43.857 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.010167+0000", 2024-09-13T11:22:43.857 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.003674053 2024-09-13T11:22:43.857 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.857 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.857 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:22:43.857 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.010299+0000", 2024-09-13T11:22:43.857 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00013196599999999999 2024-09-13T11:22:43.857 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.857 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.857 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:22:43.858 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.010432+0000", 2024-09-13T11:22:43.858 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00013289899999999999 2024-09-13T11:22:43.858 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:22:43.858 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:22:43.858 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:22:43.858 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.858 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.858 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop_reply(client.4665.0:5 3.3 e31/30 ondisk, result = 0)", 2024-09-13T11:22:43.858 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:22:27.006210+0000", 2024-09-13T11:22:43.858 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 16.843560105999998, 2024-09-13T11:22:43.858 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00030806099999999999, 2024-09-13T11:22:43.858 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:22:43.858 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:22:43.859 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:22:43.859 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:22:43.859 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.859 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:22:43.859 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.006210+0000", 2024-09-13T11:22:43.859 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:22:43.859 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.859 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.859 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:22:43.859 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.006210+0000", 2024-09-13T11:22:43.859 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:22:43.859 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.859 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.860 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:22:43.860 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.006214+0000", 2024-09-13T11:22:43.860 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.4230000000000001e-06 2024-09-13T11:22:43.860 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.860 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.860 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:22:43.860 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.006228+0000", 2024-09-13T11:22:43.860 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4195999999999999e-05 2024-09-13T11:22:43.860 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.860 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.860 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:22:43.860 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.006304+0000", 2024-09-13T11:22:43.860 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.6202e-05 2024-09-13T11:22:43.861 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.861 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.861 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:22:43.861 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.006318+0000", 2024-09-13T11:22:43.861 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4005000000000001e-05 2024-09-13T11:22:43.861 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.861 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.861 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:22:43.861 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.006449+0000", 2024-09-13T11:22:43.861 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00013114600000000001 2024-09-13T11:22:43.861 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.862 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.862 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:22:43.862 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.006474+0000", 2024-09-13T11:22:43.862 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.5324999999999999e-05 2024-09-13T11:22:43.862 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.862 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.862 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:22:43.862 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.006518+0000", 2024-09-13T11:22:43.862 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.3764000000000003e-05 2024-09-13T11:22:43.862 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:22:43.862 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:22:43.862 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:22:43.862 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.863 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.863 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:18 3.3 3:c843b42e:::benchmark_data_smithi177_31116_object17:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-09-13T11:22:43.863 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:22:27.008051+0000", 2024-09-13T11:22:43.863 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 16.841719293000001, 2024-09-13T11:22:43.863 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0056804710000000003, 2024-09-13T11:22:43.863 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:22:43.863 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:22:43.863 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:43.863 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:22:43.863 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:22:43.863 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:22:43.863 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 18 2024-09-13T11:22:43.864 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.864 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:22:43.864 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.864 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:22:43.864 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.008051+0000", 2024-09-13T11:22:43.864 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:22:43.864 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.864 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.864 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:22:43.864 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.008051+0000", 2024-09-13T11:22:43.864 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:22:43.864 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.864 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.864 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:22:43.865 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.008055+0000", 2024-09-13T11:22:43.865 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.1300000000000003e-06 2024-09-13T11:22:43.865 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.865 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.865 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:22:43.865 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.008164+0000", 2024-09-13T11:22:43.865 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000108663 2024-09-13T11:22:43.865 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.865 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.865 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:22:43.865 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.008201+0000", 2024-09-13T11:22:43.865 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.7701999999999999e-05 2024-09-13T11:22:43.865 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.865 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.866 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:22:43.866 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.008223+0000", 2024-09-13T11:22:43.866 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.1974999999999999e-05 2024-09-13T11:22:43.866 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.866 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.866 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:22:43.866 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.008415+0000", 2024-09-13T11:22:43.866 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000191285 2024-09-13T11:22:43.866 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.866 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.866 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:22:43.866 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.008749+0000", 2024-09-13T11:22:43.866 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00033396000000000001 2024-09-13T11:22:43.866 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.867 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.867 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 7", 2024-09-13T11:22:43.867 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.009117+0000", 2024-09-13T11:22:43.867 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00036870799999999998 2024-09-13T11:22:43.867 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.867 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.867 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:22:43.867 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.011237+0000", 2024-09-13T11:22:43.867 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.002119957 2024-09-13T11:22:43.867 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.867 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.867 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:22:43.867 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.013431+0000", 2024-09-13T11:22:43.867 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0021933289999999999 2024-09-13T11:22:43.867 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.868 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.868 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:22:43.868 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.013602+0000", 2024-09-13T11:22:43.868 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000171616 2024-09-13T11:22:43.868 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.868 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.868 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:22:43.868 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.013731+0000", 2024-09-13T11:22:43.868 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00012914600000000001 2024-09-13T11:22:43.868 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:22:43.868 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:22:43.868 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:22:43.868 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.869 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.869 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop_reply(client.4665.0:18 3.3 e32/30 ondisk, result = 0)", 2024-09-13T11:22:43.869 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:22:27.010996+0000", 2024-09-13T11:22:43.869 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 16.838774711999999, 2024-09-13T11:22:43.869 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00025344499999999998, 2024-09-13T11:22:43.869 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:22:43.869 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:22:43.869 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:22:43.869 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:22:43.869 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.869 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:22:43.869 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.010996+0000", 2024-09-13T11:22:43.869 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:22:43.869 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.870 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.870 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:22:43.870 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.010996+0000", 2024-09-13T11:22:43.870 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:22:43.870 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.870 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.870 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:22:43.870 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.011006+0000", 2024-09-13T11:22:43.870 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0662000000000001e-05 2024-09-13T11:22:43.870 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.870 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.870 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:22:43.870 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.011016+0000", 2024-09-13T11:22:43.870 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0044000000000001e-05 2024-09-13T11:22:43.871 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.871 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.871 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:22:43.871 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.011089+0000", 2024-09-13T11:22:43.871 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.3004999999999996e-05 2024-09-13T11:22:43.871 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.871 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.871 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:22:43.871 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.011102+0000", 2024-09-13T11:22:43.871 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3213e-05 2024-09-13T11:22:43.871 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.871 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.871 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:22:43.872 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.011213+0000", 2024-09-13T11:22:43.872 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00011056300000000001 2024-09-13T11:22:43.872 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.872 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.872 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:22:43.872 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.011227+0000", 2024-09-13T11:22:43.872 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3679000000000001e-05 2024-09-13T11:22:43.872 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.872 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.872 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:22:43.872 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.011249+0000", 2024-09-13T11:22:43.872 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.2279000000000001e-05 2024-09-13T11:22:43.872 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:22:43.872 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:22:43.873 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:22:43.873 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.873 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.873 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:36 3.3 3:cda5fb1c:::benchmark_data_smithi177_31116_object35:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-09-13T11:22:43.873 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:22:27.016028+0000", 2024-09-13T11:22:43.873 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 16.833742503, 2024-09-13T11:22:43.873 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0041210709999999996, 2024-09-13T11:22:43.873 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:22:43.873 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:22:43.873 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:43.873 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:22:43.873 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:22:43.873 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:22:43.873 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 36 2024-09-13T11:22:43.874 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.874 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:22:43.874 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.874 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:22:43.874 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.016028+0000", 2024-09-13T11:22:43.874 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:22:43.874 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.874 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.874 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:22:43.874 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.016028+0000", 2024-09-13T11:22:43.874 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:22:43.874 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.874 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.875 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:22:43.875 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.016032+0000", 2024-09-13T11:22:43.875 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.0770000000000001e-06 2024-09-13T11:22:43.875 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.875 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.875 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:22:43.875 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.016122+0000", 2024-09-13T11:22:43.875 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.0297999999999995e-05 2024-09-13T11:22:43.875 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.875 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.875 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:22:43.875 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.016163+0000", 2024-09-13T11:22:43.875 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.0630000000000002e-05 2024-09-13T11:22:43.876 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.876 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.876 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:22:43.876 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.016179+0000", 2024-09-13T11:22:43.876 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5798999999999999e-05 2024-09-13T11:22:43.876 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.876 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.876 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:22:43.876 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.016327+0000", 2024-09-13T11:22:43.876 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000148009 2024-09-13T11:22:43.876 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.876 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.877 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:22:43.877 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.016703+0000", 2024-09-13T11:22:43.877 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000376824 2024-09-13T11:22:43.877 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.877 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.877 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 7", 2024-09-13T11:22:43.877 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.016951+0000", 2024-09-13T11:22:43.877 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000247641 2024-09-13T11:22:43.877 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.877 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.877 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:22:43.877 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.018415+0000", 2024-09-13T11:22:43.877 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001464443 2024-09-13T11:22:43.877 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.878 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.878 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:22:43.878 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.019894+0000", 2024-09-13T11:22:43.878 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0014790020000000001 2024-09-13T11:22:43.878 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.878 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.878 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:22:43.878 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.020034+0000", 2024-09-13T11:22:43.878 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00013971099999999999 2024-09-13T11:22:43.878 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.878 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.878 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:22:43.879 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.020149+0000", 2024-09-13T11:22:43.879 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00011463699999999999 2024-09-13T11:22:43.879 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:22:43.879 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:22:43.879 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:22:43.879 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.879 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.879 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop_reply(client.4665.0:36 3.3 e32/30 ondisk, result = 0)", 2024-09-13T11:22:43.879 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:22:27.018210+0000", 2024-09-13T11:22:43.879 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 16.831559825999999, 2024-09-13T11:22:43.879 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00021688, 2024-09-13T11:22:43.879 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:22:43.879 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:22:43.880 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:22:43.880 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:22:43.880 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.880 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:22:43.880 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.018210+0000", 2024-09-13T11:22:43.880 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:22:43.880 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.880 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.880 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:22:43.880 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.018210+0000", 2024-09-13T11:22:43.880 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:22:43.880 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.880 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.880 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:22:43.881 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.018213+0000", 2024-09-13T11:22:43.881 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.0199999999999999e-06 2024-09-13T11:22:43.881 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.881 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.881 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:22:43.881 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.018223+0000", 2024-09-13T11:22:43.881 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.482e-06 2024-09-13T11:22:43.881 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.881 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.881 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:22:43.882 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.018280+0000", 2024-09-13T11:22:43.882 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.7210000000000003e-05 2024-09-13T11:22:43.882 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.882 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.882 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:22:43.882 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.018293+0000", 2024-09-13T11:22:43.882 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3043e-05 2024-09-13T11:22:43.882 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.882 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.882 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:22:43.882 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.018390+0000", 2024-09-13T11:22:43.882 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.7046999999999994e-05 2024-09-13T11:22:43.882 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.882 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.883 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:22:43.883 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.018403+0000", 2024-09-13T11:22:43.883 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2523e-05 2024-09-13T11:22:43.883 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.883 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.883 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:22:43.883 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.018427+0000", 2024-09-13T11:22:43.883 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.4555000000000001e-05 2024-09-13T11:22:43.883 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:22:43.883 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:22:43.883 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:22:43.884 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.884 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.884 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:63 3.3 3:c10de739:::benchmark_data_smithi177_31116_object62:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-09-13T11:22:43.884 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:22:27.030813+0000", 2024-09-13T11:22:43.884 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 16.818957476000001, 2024-09-13T11:22:43.884 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0025455809999999999, 2024-09-13T11:22:43.884 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:22:43.884 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:22:43.884 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:43.884 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:22:43.884 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:22:43.884 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:22:43.885 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 63 2024-09-13T11:22:43.885 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.885 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:22:43.885 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.885 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:22:43.885 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.030813+0000", 2024-09-13T11:22:43.885 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:22:43.885 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.885 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.885 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:22:43.885 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.030813+0000", 2024-09-13T11:22:43.885 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:22:43.885 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.885 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.886 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:22:43.886 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.030816+0000", 2024-09-13T11:22:43.886 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.563e-06 2024-09-13T11:22:43.886 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.886 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.886 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:22:43.886 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.030871+0000", 2024-09-13T11:22:43.886 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.4243999999999998e-05 2024-09-13T11:22:43.886 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.886 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.886 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:22:43.886 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.030900+0000", 2024-09-13T11:22:43.886 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.9198000000000001e-05 2024-09-13T11:22:43.886 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.887 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.887 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:22:43.887 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.030914+0000", 2024-09-13T11:22:43.887 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4329e-05 2024-09-13T11:22:43.887 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.887 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.887 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:22:43.887 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.031013+0000", 2024-09-13T11:22:43.887 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.8926999999999997e-05 2024-09-13T11:22:43.887 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.887 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.887 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:22:43.887 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.031213+0000", 2024-09-13T11:22:43.887 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00020036500000000001 2024-09-13T11:22:43.888 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.888 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.888 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 7", 2024-09-13T11:22:43.888 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.031497+0000", 2024-09-13T11:22:43.888 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000283923 2024-09-13T11:22:43.888 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.888 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.888 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:22:43.888 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.033152+0000", 2024-09-13T11:22:43.888 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0016544179999999999 2024-09-13T11:22:43.888 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.888 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.888 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:22:43.888 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.033204+0000", 2024-09-13T11:22:43.889 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.2639999999999997e-05 2024-09-13T11:22:43.889 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.889 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.889 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:22:43.889 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.033293+0000", 2024-09-13T11:22:43.889 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.8242e-05 2024-09-13T11:22:43.889 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.889 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.889 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:22:43.889 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.033358+0000", 2024-09-13T11:22:43.889 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.5732e-05 2024-09-13T11:22:43.889 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:22:43.889 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:22:43.889 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:22:43.889 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.890 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.890 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop_reply(client.4665.0:63 3.3 e32/30 ondisk, result = 0)", 2024-09-13T11:22:43.890 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:22:27.032941+0000", 2024-09-13T11:22:43.890 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 16.816829182999999, 2024-09-13T11:22:43.890 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00023613900000000001, 2024-09-13T11:22:43.890 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:22:43.890 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:22:43.890 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:22:43.890 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:22:43.890 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.890 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:22:43.890 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.032941+0000", 2024-09-13T11:22:43.890 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:22:43.890 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.891 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.891 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:22:43.891 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.032941+0000", 2024-09-13T11:22:43.891 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:22:43.891 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.891 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.891 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:22:43.891 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.032944+0000", 2024-09-13T11:22:43.891 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.1669999999999999e-06 2024-09-13T11:22:43.891 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.891 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.891 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:22:43.891 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.032954+0000", 2024-09-13T11:22:43.892 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.2790000000000004e-06 2024-09-13T11:22:43.892 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.892 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.892 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:22:43.892 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.033011+0000", 2024-09-13T11:22:43.892 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.7646999999999999e-05 2024-09-13T11:22:43.892 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.892 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.892 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:22:43.892 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.033025+0000", 2024-09-13T11:22:43.892 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4171999999999999e-05 2024-09-13T11:22:43.892 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.892 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.893 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:22:43.893 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.033132+0000", 2024-09-13T11:22:43.893 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000106721 2024-09-13T11:22:43.893 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.893 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.893 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:22:43.893 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.033143+0000", 2024-09-13T11:22:43.893 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1449e-05 2024-09-13T11:22:43.893 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.893 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.893 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:22:43.893 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.033177+0000", 2024-09-13T11:22:43.893 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.3704000000000001e-05 2024-09-13T11:22:43.893 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:22:43.894 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:22:43.894 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:22:43.894 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.894 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.894 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:77 3.3 3:c9b6c4f6:::benchmark_data_smithi177_31116_object76:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-09-13T11:22:43.894 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:22:27.047173+0000", 2024-09-13T11:22:43.894 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 16.802596949000002, 2024-09-13T11:22:43.894 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0023565729999999998, 2024-09-13T11:22:43.894 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:22:43.894 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:22:43.894 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:43.894 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:22:43.894 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:22:43.894 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:22:43.895 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 77 2024-09-13T11:22:43.895 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.895 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:22:43.895 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.895 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:22:43.895 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.047173+0000", 2024-09-13T11:22:43.895 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:22:43.895 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.895 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.895 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:22:43.895 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.047173+0000", 2024-09-13T11:22:43.895 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:22:43.895 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.896 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.896 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:22:43.896 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.047177+0000", 2024-09-13T11:22:43.896 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.393e-06 2024-09-13T11:22:43.896 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.896 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.896 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:22:43.896 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.047282+0000", 2024-09-13T11:22:43.896 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000104867 2024-09-13T11:22:43.896 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.896 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.896 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:22:43.896 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.047310+0000", 2024-09-13T11:22:43.897 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.8034999999999998e-05 2024-09-13T11:22:43.897 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.897 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.897 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:22:43.897 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.047325+0000", 2024-09-13T11:22:43.897 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5122e-05 2024-09-13T11:22:43.897 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.897 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.897 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:22:43.897 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.047465+0000", 2024-09-13T11:22:43.897 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000140149 2024-09-13T11:22:43.897 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.897 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.897 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:22:43.898 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.047673+0000", 2024-09-13T11:22:43.898 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000208464 2024-09-13T11:22:43.898 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.898 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.898 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 7", 2024-09-13T11:22:43.898 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.047911+0000", 2024-09-13T11:22:43.898 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00023797900000000001 2024-09-13T11:22:43.898 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.898 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.898 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:22:43.898 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.049362+0000", 2024-09-13T11:22:43.898 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0014505900000000001 2024-09-13T11:22:43.898 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.898 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.899 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:22:43.899 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.049384+0000", 2024-09-13T11:22:43.899 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.2478999999999999e-05 2024-09-13T11:22:43.899 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.899 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.899 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:22:43.899 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.049464+0000", 2024-09-13T11:22:43.899 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.0129000000000005e-05 2024-09-13T11:22:43.899 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.899 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.899 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:22:43.899 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.049530+0000", 2024-09-13T11:22:43.899 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.5365999999999997e-05 2024-09-13T11:22:43.900 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:22:43.900 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:22:43.900 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:22:43.900 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.900 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.900 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop_reply(client.4665.0:77 3.3 e32/30 ondisk, result = 0)", 2024-09-13T11:22:43.900 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:22:27.049147+0000", 2024-09-13T11:22:43.900 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 16.800623612999999, 2024-09-13T11:22:43.900 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000226579, 2024-09-13T11:22:43.900 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:22:43.900 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:22:43.900 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:22:43.901 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:22:43.901 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.901 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:22:43.901 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.049147+0000", 2024-09-13T11:22:43.901 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:22:43.901 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.901 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.901 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:22:43.901 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.049147+0000", 2024-09-13T11:22:43.901 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:22:43.901 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.901 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.902 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:22:43.902 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.049150+0000", 2024-09-13T11:22:43.902 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.9900000000000002e-06 2024-09-13T11:22:43.902 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.902 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.902 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:22:43.902 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.049159+0000", 2024-09-13T11:22:43.902 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.3149999999999998e-06 2024-09-13T11:22:43.902 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.902 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.902 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:22:43.902 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.049223+0000", 2024-09-13T11:22:43.903 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.3767000000000001e-05 2024-09-13T11:22:43.903 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.903 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.903 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:22:43.903 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.049239+0000", 2024-09-13T11:22:43.903 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.6745e-05 2024-09-13T11:22:43.903 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.903 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.903 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:22:43.903 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.049340+0000", 2024-09-13T11:22:43.903 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00010066800000000001 2024-09-13T11:22:43.903 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.903 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.904 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:22:43.904 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.049353+0000", 2024-09-13T11:22:43.904 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2913e-05 2024-09-13T11:22:43.904 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.904 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.904 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:22:43.904 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.049373+0000", 2024-09-13T11:22:43.904 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.0180999999999999e-05 2024-09-13T11:22:43.904 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:22:43.904 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:22:43.904 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:22:43.904 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.905 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.905 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:78 3.3 3:c95c4ad0:::benchmark_data_smithi177_31116_object77:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-09-13T11:22:43.905 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:22:27.049761+0000", 2024-09-13T11:22:43.905 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 16.800008869999999, 2024-09-13T11:22:43.905 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0050178640000000004, 2024-09-13T11:22:43.905 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:22:43.905 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:22:43.905 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:43.905 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:22:43.905 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:22:43.905 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:22:43.905 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 78 2024-09-13T11:22:43.905 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.905 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:22:43.906 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.906 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:22:43.906 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.049761+0000", 2024-09-13T11:22:43.906 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:22:43.906 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.906 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.906 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:22:43.906 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.049761+0000", 2024-09-13T11:22:43.906 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:22:43.906 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.906 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.906 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:22:43.907 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.049764+0000", 2024-09-13T11:22:43.907 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.526e-06 2024-09-13T11:22:43.907 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.907 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.907 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:22:43.907 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.049838+0000", 2024-09-13T11:22:43.907 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.4270000000000006e-05 2024-09-13T11:22:43.907 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.907 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.907 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:22:43.907 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.049862+0000", 2024-09-13T11:22:43.907 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.3428e-05 2024-09-13T11:22:43.907 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.907 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.908 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:22:43.908 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.049875+0000", 2024-09-13T11:22:43.908 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3073e-05 2024-09-13T11:22:43.908 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.908 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.908 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:22:43.908 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.050039+0000", 2024-09-13T11:22:43.908 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000163957 2024-09-13T11:22:43.908 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.908 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.908 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:22:43.908 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.050232+0000", 2024-09-13T11:22:43.909 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00019368799999999999 2024-09-13T11:22:43.909 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.909 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.909 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 7", 2024-09-13T11:22:43.909 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.050472+0000", 2024-09-13T11:22:43.909 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000239986 2024-09-13T11:22:43.909 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.909 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.909 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:22:43.909 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.051919+0000", 2024-09-13T11:22:43.909 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001446403 2024-09-13T11:22:43.909 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.909 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.909 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:22:43.910 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.054619+0000", 2024-09-13T11:22:43.910 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0027000520000000001 2024-09-13T11:22:43.910 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.910 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.910 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:22:43.910 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.054710+0000", 2024-09-13T11:22:43.910 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.1712000000000006e-05 2024-09-13T11:22:43.910 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.910 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.910 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:22:43.910 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.054779+0000", 2024-09-13T11:22:43.910 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.8769000000000005e-05 2024-09-13T11:22:43.910 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:22:43.911 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:22:43.911 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:22:43.911 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.911 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.911 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop_reply(client.4665.0:78 3.3 e32/30 ondisk, result = 0)", 2024-09-13T11:22:43.911 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:22:27.051708+0000", 2024-09-13T11:22:43.911 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 16.798062723000001, 2024-09-13T11:22:43.911 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000224644, 2024-09-13T11:22:43.911 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:22:43.911 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:22:43.911 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:22:43.911 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:22:43.911 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.911 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:22:43.911 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.051708+0000", 2024-09-13T11:22:43.912 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:22:43.912 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.912 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.912 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:22:43.912 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.051708+0000", 2024-09-13T11:22:43.912 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:22:43.912 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.912 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.912 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:22:43.912 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.051710+0000", 2024-09-13T11:22:43.912 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.977e-06 2024-09-13T11:22:43.912 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.912 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.913 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:22:43.913 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.051719+0000", 2024-09-13T11:22:43.913 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.9460000000000004e-06 2024-09-13T11:22:43.913 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.913 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.913 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:22:43.913 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.051775+0000", 2024-09-13T11:22:43.913 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.5086000000000001e-05 2024-09-13T11:22:43.913 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.913 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.913 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:22:43.913 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.051787+0000", 2024-09-13T11:22:43.913 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2713e-05 2024-09-13T11:22:43.913 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.914 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.914 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:22:43.914 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.051896+0000", 2024-09-13T11:22:43.914 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000108874 2024-09-13T11:22:43.914 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.914 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.914 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:22:43.914 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.051910+0000", 2024-09-13T11:22:43.914 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3631999999999999e-05 2024-09-13T11:22:43.914 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.914 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.914 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:22:43.914 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.051932+0000", 2024-09-13T11:22:43.914 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.2416e-05 2024-09-13T11:22:43.915 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:22:43.915 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:22:43.915 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:22:43.915 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.915 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.915 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:80 3.3 3:c3d99973:::benchmark_data_smithi177_31116_object79:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-09-13T11:22:43.915 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:22:27.056483+0000", 2024-09-13T11:22:43.915 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 16.793287528, 2024-09-13T11:22:43.915 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0023848369999999999, 2024-09-13T11:22:43.915 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:22:43.915 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:22:43.915 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:43.915 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:22:43.916 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:22:43.916 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:22:43.916 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 80 2024-09-13T11:22:43.916 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.916 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:22:43.916 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.916 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:22:43.916 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.056483+0000", 2024-09-13T11:22:43.916 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:22:43.916 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.916 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.916 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:22:43.916 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.056483+0000", 2024-09-13T11:22:43.916 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:22:43.917 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.917 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.917 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:22:43.917 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.056486+0000", 2024-09-13T11:22:43.917 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.36e-06 2024-09-13T11:22:43.917 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.917 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.917 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:22:43.917 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.056535+0000", 2024-09-13T11:22:43.917 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.8964000000000001e-05 2024-09-13T11:22:43.917 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.917 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.917 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:22:43.917 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.056565+0000", 2024-09-13T11:22:43.918 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.0207999999999999e-05 2024-09-13T11:22:43.918 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.918 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.918 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:22:43.918 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.056580+0000", 2024-09-13T11:22:43.918 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4429e-05 2024-09-13T11:22:43.918 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.918 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.919 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:22:43.919 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.056752+0000", 2024-09-13T11:22:43.919 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00017254299999999999 2024-09-13T11:22:43.919 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.919 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.919 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:22:43.919 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.056961+0000", 2024-09-13T11:22:43.919 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00020884000000000001 2024-09-13T11:22:43.919 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.919 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.919 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 7", 2024-09-13T11:22:43.919 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.057172+0000", 2024-09-13T11:22:43.919 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000211358 2024-09-13T11:22:43.919 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.920 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.920 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:22:43.920 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.058646+0000", 2024-09-13T11:22:43.920 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0014738520000000001 2024-09-13T11:22:43.920 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.920 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.920 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:22:43.920 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.058690+0000", 2024-09-13T11:22:43.920 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.4196999999999997e-05 2024-09-13T11:22:43.920 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.920 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.920 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:22:43.920 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.058803+0000", 2024-09-13T11:22:43.921 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00011254700000000001 2024-09-13T11:22:43.921 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.921 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.921 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:22:43.921 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.058868+0000", 2024-09-13T11:22:43.921 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.4539e-05 2024-09-13T11:22:43.921 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:22:43.921 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:22:43.921 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:22:43.921 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.921 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.921 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop_reply(client.4665.0:80 3.3 e32/30 ondisk, result = 0)", 2024-09-13T11:22:43.921 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:22:27.058405+0000", 2024-09-13T11:22:43.921 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 16.79136476, 2024-09-13T11:22:43.922 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00025281899999999999, 2024-09-13T11:22:43.922 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:22:43.922 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:22:43.922 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:22:43.922 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:22:43.922 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.922 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:22:43.922 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.058405+0000", 2024-09-13T11:22:43.922 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:22:43.922 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.922 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.922 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:22:43.922 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.058405+0000", 2024-09-13T11:22:43.922 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:22:43.923 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.923 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.923 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:22:43.923 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.058408+0000", 2024-09-13T11:22:43.923 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.9869999999999999e-06 2024-09-13T11:22:43.923 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.923 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.923 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:22:43.923 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.058424+0000", 2024-09-13T11:22:43.923 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5889e-05 2024-09-13T11:22:43.923 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.923 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.923 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:22:43.923 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.058505+0000", 2024-09-13T11:22:43.924 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.0704999999999994e-05 2024-09-13T11:22:43.924 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.924 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.924 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:22:43.924 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.058524+0000", 2024-09-13T11:22:43.924 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.9292000000000001e-05 2024-09-13T11:22:43.924 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.924 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.924 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:22:43.924 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.058626+0000", 2024-09-13T11:22:43.924 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000101604 2024-09-13T11:22:43.924 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.924 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.924 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:22:43.925 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.058638+0000", 2024-09-13T11:22:43.925 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1843e-05 2024-09-13T11:22:43.925 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:43.925 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:43.925 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:22:43.925 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.058658+0000", 2024-09-13T11:22:43.925 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.0499000000000001e-05 2024-09-13T11:22:43.925 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:22:43.925 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:22:43.925 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:22:43.925 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:22:43.925 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:22:43.925 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-13T11:22:43.927 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_ops_in_flight 2024-09-13T11:22:43.927 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-13T11:22:43.924+0000 7f6a68775640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26307) UID: 0 2024-09-13T11:22:43.998 INFO:tasks.radosbench.radosbench.0.smithi177.stdout: 17 16 87 71 0.260993 0 - 0.0234801 2024-09-13T11:22:44.027 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-13T11:22:44.020+0000 7f6a68775640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26307) UID: 0 2024-09-13T11:22:44.098 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-13T11:22:44.098 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [], 2024-09-13T11:22:44.098 INFO:teuthology.orchestra.run.smithi162.stdout: "num_ops": 0 2024-09-13T11:22:44.098 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-13T11:22:44.112 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_blocked_ops 2024-09-13T11:22:44.129 INFO:tasks.ceph.osd.8.smithi001.stderr:2024-09-13T11:22:44.124+0000 7fb9b48ae640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26312) UID: 0 2024-09-13T11:22:44.230 INFO:tasks.ceph.osd.7.smithi177.stderr:2024-09-13T11:22:44.224+0000 7f897b51d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26575) UID: 0 2024-09-13T11:22:44.232 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-13T11:22:44.232 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [], 2024-09-13T11:22:44.232 INFO:teuthology.orchestra.run.smithi162.stdout: "complaint_time": 30, 2024-09-13T11:22:44.232 INFO:teuthology.orchestra.run.smithi162.stdout: "num_blocked_ops": 0 2024-09-13T11:22:44.232 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-13T11:22:44.247 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_historic_ops 2024-09-13T11:22:44.331 INFO:tasks.ceph.osd.8.smithi001.stderr:2024-09-13T11:22:44.324+0000 7fb9b48ae640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26312) UID: 0 2024-09-13T11:22:44.394 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-13T11:22:44.394 INFO:teuthology.orchestra.run.smithi162.stdout: "size": 20, 2024-09-13T11:22:44.394 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 600, 2024-09-13T11:22:44.394 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [ 2024-09-13T11:22:44.394 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:44.395 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4344.0:1 2.3 2:c4c92e5a:::rbd_trash:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e26)", 2024-09-13T11:22:44.395 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:22:14.143485+0000", 2024-09-13T11:22:44.395 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 30.247842479999999, 2024-09-13T11:22:44.395 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0061818159999999997, 2024-09-13T11:22:44.395 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:22:44.395 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:22:44.395 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:44.395 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:22:44.395 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4344", 2024-09-13T11:22:44.395 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.1:0/211923147", 2024-09-13T11:22:44.395 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 1 2024-09-13T11:22:44.395 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:44.395 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:22:44.395 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:44.396 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:22:44.396 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:14.143485+0000", 2024-09-13T11:22:44.396 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:22:44.396 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:44.396 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:44.396 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:22:44.396 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:14.143485+0000", 2024-09-13T11:22:44.396 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:22:44.396 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:44.396 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:44.396 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:22:44.396 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:14.143487+0000", 2024-09-13T11:22:44.396 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.933e-06 2024-09-13T11:22:44.397 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:44.397 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:44.397 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:22:44.397 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:14.143496+0000", 2024-09-13T11:22:44.397 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.1929999999999999e-06 2024-09-13T11:22:44.397 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:44.397 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:44.397 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:22:44.397 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:14.143511+0000", 2024-09-13T11:22:44.397 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4995e-05 2024-09-13T11:22:44.397 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:44.397 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:44.397 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:22:44.398 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:14.143523+0000", 2024-09-13T11:22:44.398 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1416999999999999e-05 2024-09-13T11:22:44.398 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:44.398 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:44.398 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:22:44.398 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:14.143579+0000", 2024-09-13T11:22:44.398 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.6598999999999998e-05 2024-09-13T11:22:44.398 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:44.398 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:44.398 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-09-13T11:22:44.398 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:14.143586+0000", 2024-09-13T11:22:44.398 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.8909999999999998e-06 2024-09-13T11:22:44.398 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:44.399 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:44.399 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:22:44.399 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:14.146789+0000", 2024-09-13T11:22:44.399 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0032031350000000002 2024-09-13T11:22:44.399 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:44.399 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:44.399 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:22:44.399 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:14.147054+0000", 2024-09-13T11:22:44.399 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00026497000000000001 2024-09-13T11:22:44.399 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:44.399 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:44.399 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 15", 2024-09-13T11:22:44.399 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:14.147236+0000", 2024-09-13T11:22:44.400 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000181499 2024-09-13T11:22:44.400 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:44.400 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:44.400 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:22:44.400 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:14.148147+0000", 2024-09-13T11:22:44.400 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00091084800000000002 2024-09-13T11:22:44.400 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:44.400 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:44.400 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:22:44.400 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:14.149588+0000", 2024-09-13T11:22:44.400 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001441026 2024-09-13T11:22:44.400 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:44.400 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:44.401 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:22:44.401 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:14.149638+0000", 2024-09-13T11:22:44.401 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.0006000000000002e-05 2024-09-13T11:22:44.401 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:44.401 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:44.401 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:22:44.401 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:14.149667+0000", 2024-09-13T11:22:44.401 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.9303999999999999e-05 2024-09-13T11:22:44.401 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:22:44.401 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:22:44.401 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:22:44.401 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:44.401 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:44.402 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:2 3.2 3:438ada23:::benchmark_data_smithi177_31116_object1:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-09-13T11:22:44.402 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:22:27.001682+0000", 2024-09-13T11:22:44.402 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 17.389645594000001, 2024-09-13T11:22:44.402 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.012911389000000001, 2024-09-13T11:22:44.402 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:22:44.402 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:22:44.402 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:44.402 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:22:44.402 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:22:44.402 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:22:44.402 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 2 2024-09-13T11:22:44.402 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:44.403 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:22:44.403 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:44.403 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:22:44.403 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.001682+0000", 2024-09-13T11:22:44.403 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:22:44.403 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:44.403 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:44.403 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:22:44.403 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.001682+0000", 2024-09-13T11:22:44.403 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:22:44.403 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:44.403 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:44.403 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:22:44.403 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.001688+0000", 2024-09-13T11:22:44.404 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.7830000000000004e-06 2024-09-13T11:22:44.404 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:44.404 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:44.404 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:22:44.404 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.002086+0000", 2024-09-13T11:22:44.404 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00039873000000000003 2024-09-13T11:22:44.404 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:44.404 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:44.404 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:22:44.404 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.002126+0000", 2024-09-13T11:22:44.404 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.9211000000000001e-05 2024-09-13T11:22:44.404 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:44.404 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:44.405 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:22:44.405 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.002149+0000", 2024-09-13T11:22:44.405 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.3431e-05 2024-09-13T11:22:44.405 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:44.405 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:44.405 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:22:44.405 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.002334+0000", 2024-09-13T11:22:44.405 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000184599 2024-09-13T11:22:44.405 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:44.405 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:44.405 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op must wait for map", 2024-09-13T11:22:44.405 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.002350+0000", 2024-09-13T11:22:44.405 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5979000000000001e-05 2024-09-13T11:22:44.406 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:44.406 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:44.406 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:22:44.406 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.008094+0000", 2024-09-13T11:22:44.406 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0057442869999999998 2024-09-13T11:22:44.406 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:44.406 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:44.406 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:22:44.406 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.008331+0000", 2024-09-13T11:22:44.407 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00023699300000000001 2024-09-13T11:22:44.407 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:44.407 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:44.407 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 12", 2024-09-13T11:22:44.407 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.008608+0000", 2024-09-13T11:22:44.407 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00027721400000000002 2024-09-13T11:22:44.407 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:44.407 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:44.407 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:22:44.407 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.011753+0000", 2024-09-13T11:22:44.407 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0031443090000000001 2024-09-13T11:22:44.407 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:44.407 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:44.408 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:22:44.408 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.014437+0000", 2024-09-13T11:22:44.408 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0026848990000000001 2024-09-13T11:22:44.408 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:44.408 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:44.408 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:22:44.408 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.014524+0000", 2024-09-13T11:22:44.408 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.6821000000000001e-05 2024-09-13T11:22:44.408 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:44.408 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:44.408 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:22:44.408 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.014593+0000", 2024-09-13T11:22:44.408 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.9133000000000004e-05 2024-09-13T11:22:44.408 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:22:44.409 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:22:44.409 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:22:44.409 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:44.409 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:44.409 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.4665.0:8 3.6 e32/30 3:64ae72f1:::benchmark_data_smithi177_31116_object7:head v 32'1, mlcod=0'0)", 2024-09-13T11:22:44.409 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:22:27.003524+0000", 2024-09-13T11:22:44.409 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 17.387803364, 2024-09-13T11:22:44.409 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0051300069999999998, 2024-09-13T11:22:44.409 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:22:44.409 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:22:44.409 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:44.409 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:22:44.409 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:44.410 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:22:44.410 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.003524+0000", 2024-09-13T11:22:44.410 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:22:44.410 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:44.410 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:44.410 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:22:44.410 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.003524+0000", 2024-09-13T11:22:44.410 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:22:44.410 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:44.411 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:44.411 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:22:44.411 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.003526+0000", 2024-09-13T11:22:44.411 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4929999999999999e-06 2024-09-13T11:22:44.411 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:44.411 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:44.411 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:22:44.411 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.003745+0000", 2024-09-13T11:22:44.411 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00021966100000000001 2024-09-13T11:22:44.411 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:44.411 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:44.412 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:22:44.412 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.003839+0000", 2024-09-13T11:22:44.412 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.4036999999999994e-05 2024-09-13T11:22:44.412 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:44.412 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:44.412 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:22:44.412 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.003862+0000", 2024-09-13T11:22:44.412 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.3082e-05 2024-09-13T11:22:44.412 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:44.412 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:44.412 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:22:44.412 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.004005+0000", 2024-09-13T11:22:44.413 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000142812 2024-09-13T11:22:44.413 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:44.413 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:44.413 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:22:44.413 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.004049+0000", 2024-09-13T11:22:44.413 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.3571e-05 2024-09-13T11:22:44.413 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:44.413 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:44.413 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:22:44.413 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.008542+0000", 2024-09-13T11:22:44.413 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0044930910000000003 2024-09-13T11:22:44.413 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:44.414 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:44.414 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:22:44.414 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.008654+0000", 2024-09-13T11:22:44.414 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00011226 2024-09-13T11:22:44.414 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:22:44.414 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:22:44.414 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:22:44.414 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:44.414 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:44.414 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.4665.0:15 3.6 e32/30 3:6288aaff:::benchmark_data_smithi177_31116_object14:head v 32'2, mlcod=0'0)", 2024-09-13T11:22:44.414 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:22:27.004496+0000", 2024-09-13T11:22:44.414 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 17.386831259000001, 2024-09-13T11:22:44.415 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0042476270000000003, 2024-09-13T11:22:44.415 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:22:44.415 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:22:44.415 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:44.415 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:22:44.415 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:44.415 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:22:44.415 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.004496+0000", 2024-09-13T11:22:44.415 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:22:44.415 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:44.415 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:44.415 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:22:44.415 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.004496+0000", 2024-09-13T11:22:44.416 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:22:44.416 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:44.416 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:44.416 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:22:44.416 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.004507+0000", 2024-09-13T11:22:44.416 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.047e-05 2024-09-13T11:22:44.416 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:44.416 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:44.416 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:22:44.416 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.004664+0000", 2024-09-13T11:22:44.416 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00015729000000000001 2024-09-13T11:22:44.416 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:44.416 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:44.417 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:22:44.417 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.004749+0000", 2024-09-13T11:22:44.417 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.5272000000000006e-05 2024-09-13T11:22:44.417 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:44.417 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:44.417 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:22:44.417 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.004767+0000", 2024-09-13T11:22:44.417 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.7832000000000001e-05 2024-09-13T11:22:44.417 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:44.417 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:44.417 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:22:44.417 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.004948+0000", 2024-09-13T11:22:44.417 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00018096900000000001 2024-09-13T11:22:44.418 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:44.418 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:44.418 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:22:44.418 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.005004+0000", 2024-09-13T11:22:44.418 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.6137000000000001e-05 2024-09-13T11:22:44.418 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:44.418 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:44.418 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:22:44.418 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.008663+0000", 2024-09-13T11:22:44.418 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0036589750000000001 2024-09-13T11:22:44.418 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:44.418 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:44.418 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:22:44.419 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.008744+0000", 2024-09-13T11:22:44.419 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.0681999999999998e-05 2024-09-13T11:22:44.419 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:22:44.419 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:22:44.419 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:22:44.419 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:44.419 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:44.419 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:17 3.2 3:4bd8024b:::benchmark_data_smithi177_31116_object16:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-09-13T11:22:44.419 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:22:27.006937+0000", 2024-09-13T11:22:44.419 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 17.384390804999999, 2024-09-13T11:22:44.419 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0085759679999999998, 2024-09-13T11:22:44.419 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:22:44.420 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:22:44.420 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:44.420 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:22:44.420 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:22:44.420 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:22:44.420 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 17 2024-09-13T11:22:44.420 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:44.420 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:22:44.420 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:44.420 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:22:44.420 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.006937+0000", 2024-09-13T11:22:44.420 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:22:44.420 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:44.420 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:44.421 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:22:44.421 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.006937+0000", 2024-09-13T11:22:44.421 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:22:44.421 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:44.421 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:44.421 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:22:44.421 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.006941+0000", 2024-09-13T11:22:44.421 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.25e-06 2024-09-13T11:22:44.421 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:44.421 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:44.421 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:22:44.421 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.007103+0000", 2024-09-13T11:22:44.421 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000162251 2024-09-13T11:22:44.421 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:44.422 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:44.422 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:22:44.422 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.007136+0000", 2024-09-13T11:22:44.422 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.2694000000000002e-05 2024-09-13T11:22:44.422 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:44.422 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:44.422 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:22:44.422 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.007156+0000", 2024-09-13T11:22:44.422 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.9585999999999998e-05 2024-09-13T11:22:44.422 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:44.422 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:44.422 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:22:44.422 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.007277+0000", 2024-09-13T11:22:44.423 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000121273 2024-09-13T11:22:44.423 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:44.423 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:44.423 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting_for_map not empty", 2024-09-13T11:22:44.423 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.007285+0000", 2024-09-13T11:22:44.423 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.0860000000000002e-06 2024-09-13T11:22:44.423 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:44.423 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:44.423 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:22:44.423 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.010111+0000", 2024-09-13T11:22:44.423 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0028265310000000002 2024-09-13T11:22:44.423 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:44.423 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:44.423 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:22:44.423 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.010348+0000", 2024-09-13T11:22:44.424 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00023645299999999999 2024-09-13T11:22:44.424 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:44.424 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:44.424 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 12", 2024-09-13T11:22:44.424 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.010754+0000", 2024-09-13T11:22:44.424 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000406458 2024-09-13T11:22:44.424 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:44.424 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:44.424 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:22:44.424 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.012928+0000", 2024-09-13T11:22:44.424 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0021735439999999999 2024-09-13T11:22:44.424 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:44.424 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:44.425 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:22:44.425 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.015262+0000", 2024-09-13T11:22:44.425 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0023336469999999999 2024-09-13T11:22:44.425 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:44.425 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:44.425 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:22:44.425 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.015391+0000", 2024-09-13T11:22:44.425 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00012929900000000001 2024-09-13T11:22:44.425 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:44.425 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:44.425 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:22:44.425 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.015513+0000", 2024-09-13T11:22:44.425 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000121896 2024-09-13T11:22:44.425 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:22:44.426 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:22:44.426 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:22:44.426 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:44.426 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:44.426 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.4665.0:21 3.6 e32/30 3:69e4f35f:::benchmark_data_smithi177_31116_object20:head v 32'3, mlcod=32'1)", 2024-09-13T11:22:44.426 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:22:27.009568+0000", 2024-09-13T11:22:44.426 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 17.381759785, 2024-09-13T11:22:44.426 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0026007949999999999, 2024-09-13T11:22:44.426 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:22:44.426 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:22:44.426 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:44.426 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:22:44.426 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:44.426 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:22:44.426 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.009568+0000", 2024-09-13T11:22:44.427 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:22:44.427 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:44.427 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:44.427 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:22:44.427 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.009568+0000", 2024-09-13T11:22:44.427 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:22:44.427 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:44.427 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:44.427 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:22:44.427 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.009571+0000", 2024-09-13T11:22:44.427 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.8270000000000002e-06 2024-09-13T11:22:44.427 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:44.427 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:44.428 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:22:44.428 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.009691+0000", 2024-09-13T11:22:44.428 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00012037899999999999 2024-09-13T11:22:44.428 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:44.428 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:44.428 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:22:44.428 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.009740+0000", 2024-09-13T11:22:44.428 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.9344000000000001e-05 2024-09-13T11:22:44.428 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:44.428 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:44.428 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:22:44.428 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.009754+0000", 2024-09-13T11:22:44.428 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3366e-05 2024-09-13T11:22:44.429 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:44.429 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:44.429 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:22:44.429 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.009884+0000", 2024-09-13T11:22:44.429 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00012992200000000001 2024-09-13T11:22:44.429 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:44.429 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:44.429 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:22:44.429 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.009929+0000", 2024-09-13T11:22:44.429 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.5223999999999999e-05 2024-09-13T11:22:44.429 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:44.429 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:44.429 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:22:44.430 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.012102+0000", 2024-09-13T11:22:44.430 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0021735270000000002 2024-09-13T11:22:44.430 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:44.430 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:44.430 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:22:44.430 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.012169+0000", 2024-09-13T11:22:44.430 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.6205999999999996e-05 2024-09-13T11:22:44.430 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:22:44.430 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:22:44.430 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:22:44.430 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:44.430 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:44.430 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.4665.0:22 3.6 e32/30 3:6f6b370c:::benchmark_data_smithi177_31116_object21:head v 32'4, mlcod=32'1)", 2024-09-13T11:22:44.431 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:22:27.010057+0000", 2024-09-13T11:22:44.431 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 17.381270131000001, 2024-09-13T11:22:44.431 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0052449109999999997, 2024-09-13T11:22:44.431 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:22:44.431 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:22:44.431 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:44.431 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:22:44.431 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:44.431 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:22:44.431 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.010057+0000", 2024-09-13T11:22:44.431 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:22:44.431 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:44.431 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:44.432 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:22:44.432 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.010057+0000", 2024-09-13T11:22:44.432 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:22:44.432 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:44.432 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:44.432 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:22:44.432 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.010059+0000", 2024-09-13T11:22:44.432 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.0269999999999998e-06 2024-09-13T11:22:44.432 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:44.432 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:44.432 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:22:44.432 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.010152+0000", 2024-09-13T11:22:44.432 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.2103999999999998e-05 2024-09-13T11:22:44.432 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:44.433 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:44.433 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:22:44.433 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.010203+0000", 2024-09-13T11:22:44.433 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.1384000000000001e-05 2024-09-13T11:22:44.433 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:44.433 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:44.433 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:22:44.433 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.010219+0000", 2024-09-13T11:22:44.433 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.6095000000000001e-05 2024-09-13T11:22:44.433 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:44.433 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:44.433 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:22:44.433 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.010668+0000", 2024-09-13T11:22:44.434 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00044941700000000002 2024-09-13T11:22:44.434 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:44.434 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:44.434 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:22:44.434 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.010723+0000", 2024-09-13T11:22:44.434 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.4447000000000002e-05 2024-09-13T11:22:44.434 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:44.434 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:44.434 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:22:44.434 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.015214+0000", 2024-09-13T11:22:44.434 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0044907280000000003 2024-09-13T11:22:44.434 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:44.434 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:44.435 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:22:44.435 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.015302+0000", 2024-09-13T11:22:44.435 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.8708999999999994e-05 2024-09-13T11:22:44.435 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:22:44.435 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:22:44.435 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:22:44.435 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:44.435 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:44.435 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop_reply(client.4665.0:2 3.2 e32/30 ondisk, result = 0)", 2024-09-13T11:22:44.435 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:22:27.010498+0000", 2024-09-13T11:22:44.435 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 17.380829176999999, 2024-09-13T11:22:44.435 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001272081, 2024-09-13T11:22:44.435 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:22:44.436 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:22:44.436 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:22:44.436 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:22:44.436 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:44.436 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:22:44.436 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.010498+0000", 2024-09-13T11:22:44.436 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:22:44.436 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:44.436 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:44.436 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:22:44.436 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.010498+0000", 2024-09-13T11:22:44.436 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:22:44.436 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:44.437 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:44.437 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:22:44.437 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.010502+0000", 2024-09-13T11:22:44.437 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.2200000000000001e-06 2024-09-13T11:22:44.437 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:44.437 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:44.437 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:22:44.437 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.010512+0000", 2024-09-13T11:22:44.437 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0006e-05 2024-09-13T11:22:44.437 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:44.437 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:44.437 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:22:44.437 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.010572+0000", 2024-09-13T11:22:44.437 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.0733000000000003e-05 2024-09-13T11:22:44.438 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:44.438 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:44.438 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:22:44.438 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.010589+0000", 2024-09-13T11:22:44.438 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.6901999999999998e-05 2024-09-13T11:22:44.438 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:44.438 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:44.438 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:22:44.438 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.011695+0000", 2024-09-13T11:22:44.438 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001105541 2024-09-13T11:22:44.438 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:44.438 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:44.438 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:22:44.439 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.011737+0000", 2024-09-13T11:22:44.439 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.1724000000000002e-05 2024-09-13T11:22:44.439 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:44.439 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:44.439 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:22:44.439 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.011770+0000", 2024-09-13T11:22:44.439 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.3955000000000003e-05 2024-09-13T11:22:44.439 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:22:44.439 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:22:44.439 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:22:44.439 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:44.439 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:44.439 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.4665.0:24 3.6 e32/30 3:696318df:::benchmark_data_smithi177_31116_object23:head v 32'5, mlcod=32'1)", 2024-09-13T11:22:44.440 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:22:27.010692+0000", 2024-09-13T11:22:44.440 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 17.380635119000001, 2024-09-13T11:22:44.440 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0046893000000000004, 2024-09-13T11:22:44.440 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:22:44.440 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:22:44.440 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:44.440 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:22:44.440 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:44.440 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:22:44.440 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.010692+0000", 2024-09-13T11:22:44.440 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:22:44.440 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:44.440 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:44.441 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:22:44.441 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.010692+0000", 2024-09-13T11:22:44.441 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:22:44.441 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:44.441 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:44.441 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:22:44.441 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.010695+0000", 2024-09-13T11:22:44.441 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.5000000000000002e-06 2024-09-13T11:22:44.441 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:44.441 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:44.441 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:22:44.441 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.010847+0000", 2024-09-13T11:22:44.441 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00015198800000000001 2024-09-13T11:22:44.441 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:44.442 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:44.442 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:22:44.442 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.010914+0000", 2024-09-13T11:22:44.442 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.7268999999999996e-05 2024-09-13T11:22:44.442 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:44.442 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:44.442 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:22:44.442 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.010931+0000", 2024-09-13T11:22:44.442 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.6492999999999999e-05 2024-09-13T11:22:44.442 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:44.442 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:44.442 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:22:44.442 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.011549+0000", 2024-09-13T11:22:44.442 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00061793899999999997 2024-09-13T11:22:44.443 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:44.443 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:44.443 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:22:44.443 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.011599+0000", 2024-09-13T11:22:44.443 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.0231000000000001e-05 2024-09-13T11:22:44.443 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:44.443 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:44.443 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:22:44.443 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.015315+0000", 2024-09-13T11:22:44.443 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0037164210000000001 2024-09-13T11:22:44.443 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:44.443 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:44.443 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:22:44.444 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.015382+0000", 2024-09-13T11:22:44.444 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.6458999999999995e-05 2024-09-13T11:22:44.444 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:22:44.444 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:22:44.444 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:22:44.444 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:44.444 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:44.444 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.4665.0:20 3.e e32/30 3:794d9a5b:::benchmark_data_smithi177_31116_object19:head v 32'1, mlcod=0'0)", 2024-09-13T11:22:44.444 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:22:27.014769+0000", 2024-09-13T11:22:44.444 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 17.376558644999999, 2024-09-13T11:22:44.444 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0051222200000000002, 2024-09-13T11:22:44.444 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:22:44.444 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:22:44.445 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:44.445 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:22:44.445 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:44.445 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:22:44.445 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.014769+0000", 2024-09-13T11:22:44.445 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:22:44.445 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:44.445 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:44.445 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:22:44.445 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.014769+0000", 2024-09-13T11:22:44.445 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:22:44.445 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:44.446 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:44.446 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:22:44.446 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.014770+0000", 2024-09-13T11:22:44.446 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.373e-06 2024-09-13T11:22:44.446 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:44.446 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:44.446 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:22:44.446 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.015250+0000", 2024-09-13T11:22:44.446 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00047970800000000002 2024-09-13T11:22:44.446 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:44.446 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:44.446 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:22:44.446 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.015346+0000", 2024-09-13T11:22:44.447 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.5784000000000003e-05 2024-09-13T11:22:44.447 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:44.447 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:44.447 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:22:44.447 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.015359+0000", 2024-09-13T11:22:44.447 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3259e-05 2024-09-13T11:22:44.447 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:44.447 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:44.447 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:22:44.447 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.015463+0000", 2024-09-13T11:22:44.447 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00010442699999999999 2024-09-13T11:22:44.447 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:44.447 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:44.447 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:22:44.448 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.015568+0000", 2024-09-13T11:22:44.448 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000104994 2024-09-13T11:22:44.448 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:44.448 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:44.448 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:22:44.448 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.019795+0000", 2024-09-13T11:22:44.448 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0042267040000000004 2024-09-13T11:22:44.448 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:44.448 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:44.448 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:22:44.448 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.019891+0000", 2024-09-13T11:22:44.448 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.5971000000000007e-05 2024-09-13T11:22:44.448 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:22:44.448 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:22:44.449 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:22:44.449 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:44.449 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:44.449 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.4665.0:31 3.6 e32/30 3:6ecddb78:::benchmark_data_smithi177_31116_object30:head v 32'6, mlcod=32'2)", 2024-09-13T11:22:44.449 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:22:27.014966+0000", 2024-09-13T11:22:44.449 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 17.37636144, 2024-09-13T11:22:44.449 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0040580900000000003, 2024-09-13T11:22:44.449 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:22:44.449 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:22:44.449 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:44.449 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:22:44.449 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:44.449 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:22:44.449 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.014966+0000", 2024-09-13T11:22:44.450 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:22:44.450 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:44.450 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:44.450 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:22:44.450 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.014966+0000", 2024-09-13T11:22:44.450 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:22:44.450 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:44.450 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:44.450 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:22:44.450 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.014968+0000", 2024-09-13T11:22:44.450 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.2560000000000001e-06 2024-09-13T11:22:44.450 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:44.450 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:44.451 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:22:44.451 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.015151+0000", 2024-09-13T11:22:44.451 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00018277899999999999 2024-09-13T11:22:44.451 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:44.451 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:44.451 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:22:44.451 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.015286+0000", 2024-09-13T11:22:44.451 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00013459600000000001 2024-09-13T11:22:44.451 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:44.451 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:44.451 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:22:44.451 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.015303+0000", 2024-09-13T11:22:44.451 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.7221999999999999e-05 2024-09-13T11:22:44.452 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:44.452 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:44.452 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:22:44.452 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.015495+0000", 2024-09-13T11:22:44.452 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00019227099999999999 2024-09-13T11:22:44.452 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:44.452 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:44.452 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:22:44.452 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.015540+0000", 2024-09-13T11:22:44.452 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.4325e-05 2024-09-13T11:22:44.452 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:44.452 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:44.452 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:22:44.453 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.018915+0000", 2024-09-13T11:22:44.453 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0033753049999999999 2024-09-13T11:22:44.453 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:44.453 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:44.453 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:22:44.453 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.019024+0000", 2024-09-13T11:22:44.453 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000109336 2024-09-13T11:22:44.453 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:22:44.453 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:22:44.453 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:22:44.453 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:44.453 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:44.453 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.4665.0:34 3.6 e32/30 3:6628c15c:::benchmark_data_smithi177_31116_object33:head v 32'7, mlcod=32'4)", 2024-09-13T11:22:44.454 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:22:27.016396+0000", 2024-09-13T11:22:44.454 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 17.374931242999999, 2024-09-13T11:22:44.454 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0038869410000000001, 2024-09-13T11:22:44.454 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:22:44.454 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:22:44.454 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:44.454 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:22:44.454 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:44.454 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:22:44.454 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.016396+0000", 2024-09-13T11:22:44.455 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:22:44.455 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:44.455 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:44.455 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:22:44.455 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.016396+0000", 2024-09-13T11:22:44.455 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:22:44.455 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:44.455 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:44.455 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:22:44.455 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.016399+0000", 2024-09-13T11:22:44.455 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.0800000000000002e-06 2024-09-13T11:22:44.455 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:44.455 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:44.455 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:22:44.456 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.016455+0000", 2024-09-13T11:22:44.456 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.5316999999999999e-05 2024-09-13T11:22:44.456 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:44.456 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:44.456 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:22:44.456 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.016508+0000", 2024-09-13T11:22:44.456 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.3674000000000001e-05 2024-09-13T11:22:44.456 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:44.456 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:44.456 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:22:44.456 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.016542+0000", 2024-09-13T11:22:44.456 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.3877000000000001e-05 2024-09-13T11:22:44.456 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:44.457 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:44.457 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:22:44.457 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.016670+0000", 2024-09-13T11:22:44.457 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000127479 2024-09-13T11:22:44.457 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:44.457 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:44.457 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:22:44.457 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.016721+0000", 2024-09-13T11:22:44.457 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.1374e-05 2024-09-13T11:22:44.457 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:44.457 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:44.457 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:22:44.457 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.020228+0000", 2024-09-13T11:22:44.458 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0035071870000000002 2024-09-13T11:22:44.458 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:44.458 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:44.458 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:22:44.458 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.020283+0000", 2024-09-13T11:22:44.458 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.4953000000000001e-05 2024-09-13T11:22:44.458 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:22:44.458 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:22:44.458 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:22:44.458 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:44.458 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:44.458 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.4665.0:46 3.e e32/30 3:7d541445:::benchmark_data_smithi177_31116_object45:head v 32'2, mlcod=0'0)", 2024-09-13T11:22:44.458 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:22:27.021132+0000", 2024-09-13T11:22:44.459 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 17.370195239000001, 2024-09-13T11:22:44.459 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001381431, 2024-09-13T11:22:44.459 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:22:44.459 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:22:44.459 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:44.459 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:22:44.459 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:44.459 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:22:44.459 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.021132+0000", 2024-09-13T11:22:44.459 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:22:44.459 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:44.459 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:44.459 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:22:44.460 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.021132+0000", 2024-09-13T11:22:44.460 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:22:44.460 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:44.460 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:44.460 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:22:44.460 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.021135+0000", 2024-09-13T11:22:44.460 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.5730000000000002e-06 2024-09-13T11:22:44.460 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:44.460 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:44.460 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:22:44.460 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.021212+0000", 2024-09-13T11:22:44.460 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.7528999999999996e-05 2024-09-13T11:22:44.460 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:44.460 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:44.461 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:22:44.461 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.021264+0000", 2024-09-13T11:22:44.461 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.1177000000000001e-05 2024-09-13T11:22:44.461 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:44.461 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:44.461 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:22:44.461 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.021278+0000", 2024-09-13T11:22:44.461 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4129e-05 2024-09-13T11:22:44.461 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:44.461 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:44.461 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:22:44.461 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.021360+0000", 2024-09-13T11:22:44.461 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.2729e-05 2024-09-13T11:22:44.462 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:44.462 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:44.462 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:22:44.462 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.021393+0000", 2024-09-13T11:22:44.462 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.2288e-05 2024-09-13T11:22:44.462 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:44.462 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:44.462 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:22:44.462 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.022452+0000", 2024-09-13T11:22:44.462 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00105948 2024-09-13T11:22:44.462 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:44.462 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:44.463 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:22:44.463 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.022514+0000", 2024-09-13T11:22:44.463 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.1525999999999993e-05 2024-09-13T11:22:44.463 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:22:44.463 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:22:44.463 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:22:44.463 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:44.463 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:44.463 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:66 3.2 3:4ab2a1f9:::benchmark_data_smithi177_31116_object65:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-09-13T11:22:44.463 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:22:27.031250+0000", 2024-09-13T11:22:44.463 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 17.360077279999999, 2024-09-13T11:22:44.463 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0044864989999999997, 2024-09-13T11:22:44.463 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:22:44.464 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:22:44.464 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:44.464 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:22:44.464 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:22:44.464 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:22:44.464 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 66 2024-09-13T11:22:44.464 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:44.464 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:22:44.464 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:44.464 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:22:44.464 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.031250+0000", 2024-09-13T11:22:44.464 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:22:44.464 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:44.464 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:44.465 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:22:44.465 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.031250+0000", 2024-09-13T11:22:44.465 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:22:44.465 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:44.465 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:44.465 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:22:44.465 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.031255+0000", 2024-09-13T11:22:44.465 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.6500000000000004e-06 2024-09-13T11:22:44.465 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:44.465 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:44.465 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:22:44.465 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.031310+0000", 2024-09-13T11:22:44.465 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.5182999999999997e-05 2024-09-13T11:22:44.465 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:44.466 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:44.466 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:22:44.466 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.031340+0000", 2024-09-13T11:22:44.466 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.9519e-05 2024-09-13T11:22:44.466 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:44.466 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:44.466 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:22:44.466 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.031355+0000", 2024-09-13T11:22:44.466 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5009e-05 2024-09-13T11:22:44.466 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:44.466 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:44.466 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:22:44.466 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.031465+0000", 2024-09-13T11:22:44.466 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00010990299999999999 2024-09-13T11:22:44.466 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:44.467 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:44.467 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:22:44.467 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.031721+0000", 2024-09-13T11:22:44.467 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00025666099999999999 2024-09-13T11:22:44.467 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:44.467 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:44.467 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 12", 2024-09-13T11:22:44.467 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.032053+0000", 2024-09-13T11:22:44.467 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00033221999999999999 2024-09-13T11:22:44.467 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:44.467 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:44.467 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:22:44.467 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.033712+0000", 2024-09-13T11:22:44.468 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001658358 2024-09-13T11:22:44.468 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:44.468 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:44.468 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:22:44.468 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.035568+0000", 2024-09-13T11:22:44.468 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0018563360000000001 2024-09-13T11:22:44.468 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:44.468 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:44.468 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:22:44.468 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.035665+0000", 2024-09-13T11:22:44.468 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.6754000000000002e-05 2024-09-13T11:22:44.468 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:44.469 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:44.469 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:22:44.469 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.035737+0000", 2024-09-13T11:22:44.469 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.1905999999999999e-05 2024-09-13T11:22:44.469 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:22:44.469 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:22:44.469 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:22:44.469 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:44.469 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:44.469 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:71 3.2 3:44c580a0:::benchmark_data_smithi177_31116_object70:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-09-13T11:22:44.469 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:22:27.035943+0000", 2024-09-13T11:22:44.469 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 17.355384253, 2024-09-13T11:22:44.469 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.002318821, 2024-09-13T11:22:44.469 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:22:44.470 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:22:44.470 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:44.470 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:22:44.470 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:22:44.470 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:22:44.470 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 71 2024-09-13T11:22:44.470 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:44.470 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:22:44.470 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:44.470 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:22:44.470 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.035943+0000", 2024-09-13T11:22:44.470 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:22:44.470 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:44.471 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:44.471 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:22:44.471 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.035943+0000", 2024-09-13T11:22:44.471 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:22:44.471 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:44.471 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:44.471 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:22:44.471 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.035946+0000", 2024-09-13T11:22:44.471 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.9699999999999999e-06 2024-09-13T11:22:44.471 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:44.471 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:44.471 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:22:44.471 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.036014+0000", 2024-09-13T11:22:44.471 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.7982000000000001e-05 2024-09-13T11:22:44.472 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:44.472 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:44.472 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:22:44.472 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.036044+0000", 2024-09-13T11:22:44.472 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.9442e-05 2024-09-13T11:22:44.472 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:44.472 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:44.472 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:22:44.472 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.036060+0000", 2024-09-13T11:22:44.472 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.6609e-05 2024-09-13T11:22:44.472 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:44.472 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:44.472 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:22:44.472 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.036151+0000", 2024-09-13T11:22:44.473 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.0637000000000007e-05 2024-09-13T11:22:44.473 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:44.473 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:44.473 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:22:44.473 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.036333+0000", 2024-09-13T11:22:44.473 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000182296 2024-09-13T11:22:44.473 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:44.473 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:44.473 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 12", 2024-09-13T11:22:44.473 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.036536+0000", 2024-09-13T11:22:44.473 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00020244800000000001 2024-09-13T11:22:44.473 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:44.473 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:44.474 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:22:44.474 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.037874+0000", 2024-09-13T11:22:44.474 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0013378140000000001 2024-09-13T11:22:44.474 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:44.474 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:44.474 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:22:44.474 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.038118+0000", 2024-09-13T11:22:44.474 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00024411800000000001 2024-09-13T11:22:44.474 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:44.474 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:44.474 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:22:44.474 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.038200+0000", 2024-09-13T11:22:44.474 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.2822000000000001e-05 2024-09-13T11:22:44.474 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:44.475 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:44.475 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:22:44.475 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.038262+0000", 2024-09-13T11:22:44.475 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.1682999999999999e-05 2024-09-13T11:22:44.475 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:22:44.475 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:22:44.475 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:22:44.475 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:44.475 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:44.475 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.4665.0:74 3.6 e32/30 3:65c9aede:::benchmark_data_smithi177_31116_object73:head v 32'8, mlcod=32'6)", 2024-09-13T11:22:44.475 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:22:27.039081+0000", 2024-09-13T11:22:44.475 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 17.352246628, 2024-09-13T11:22:44.475 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001532179, 2024-09-13T11:22:44.476 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:22:44.476 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:22:44.476 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:44.476 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:22:44.476 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:44.476 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:22:44.476 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.039081+0000", 2024-09-13T11:22:44.476 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:22:44.476 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:44.476 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:44.476 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:22:44.477 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.039081+0000", 2024-09-13T11:22:44.477 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:22:44.477 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:44.477 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:44.477 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:22:44.477 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.039084+0000", 2024-09-13T11:22:44.477 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.5499999999999999e-06 2024-09-13T11:22:44.477 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:44.477 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:44.477 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:22:44.477 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.039147+0000", 2024-09-13T11:22:44.477 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.2580000000000001e-05 2024-09-13T11:22:44.477 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:44.477 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:44.478 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:22:44.478 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.039210+0000", 2024-09-13T11:22:44.478 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.2782999999999998e-05 2024-09-13T11:22:44.478 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:44.478 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:44.478 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:22:44.478 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.039223+0000", 2024-09-13T11:22:44.478 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3416e-05 2024-09-13T11:22:44.478 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:44.478 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:44.478 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:22:44.478 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.039337+0000", 2024-09-13T11:22:44.478 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000113393 2024-09-13T11:22:44.478 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:44.479 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:44.479 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:22:44.479 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.039372+0000", 2024-09-13T11:22:44.479 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.5451000000000003e-05 2024-09-13T11:22:44.479 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:44.479 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:44.479 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:22:44.479 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.040528+0000", 2024-09-13T11:22:44.479 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0011557679999999999 2024-09-13T11:22:44.479 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:44.479 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:44.479 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:22:44.479 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.040613+0000", 2024-09-13T11:22:44.479 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.5237999999999999e-05 2024-09-13T11:22:44.480 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:22:44.480 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:22:44.480 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:22:44.480 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:44.480 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:44.480 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:75 3.2 3:415872a1:::benchmark_data_smithi177_31116_object74:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-09-13T11:22:44.480 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:22:27.041206+0000", 2024-09-13T11:22:44.480 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 17.350121697999999, 2024-09-13T11:22:44.480 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0025752370000000002, 2024-09-13T11:22:44.480 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:22:44.480 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:22:44.480 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:44.480 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:22:44.481 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:22:44.481 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:22:44.481 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 75 2024-09-13T11:22:44.481 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:44.481 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:22:44.481 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:44.481 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:22:44.481 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.041206+0000", 2024-09-13T11:22:44.481 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:22:44.481 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:44.481 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:44.481 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:22:44.481 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.041206+0000", 2024-09-13T11:22:44.481 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:22:44.482 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:44.482 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:44.482 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:22:44.482 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.041208+0000", 2024-09-13T11:22:44.482 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.6199999999999999e-06 2024-09-13T11:22:44.482 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:44.482 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:44.482 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:22:44.482 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.041281+0000", 2024-09-13T11:22:44.482 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.2471999999999993e-05 2024-09-13T11:22:44.482 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:44.482 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:44.482 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:22:44.482 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.041306+0000", 2024-09-13T11:22:44.483 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.5228999999999998e-05 2024-09-13T11:22:44.483 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:44.483 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:44.483 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:22:44.483 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.041321+0000", 2024-09-13T11:22:44.483 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4372999999999999e-05 2024-09-13T11:22:44.483 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:44.483 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:44.483 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:22:44.484 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.041422+0000", 2024-09-13T11:22:44.484 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00010147 2024-09-13T11:22:44.484 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:44.484 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:44.484 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:22:44.484 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.041643+0000", 2024-09-13T11:22:44.484 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000221363 2024-09-13T11:22:44.484 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:44.484 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:44.484 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 12", 2024-09-13T11:22:44.484 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.041861+0000", 2024-09-13T11:22:44.484 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000217314 2024-09-13T11:22:44.484 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:44.485 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:44.485 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:22:44.485 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.043413+0000", 2024-09-13T11:22:44.485 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0015527609999999999 2024-09-13T11:22:44.485 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:44.485 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:44.485 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:22:44.485 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.043634+0000", 2024-09-13T11:22:44.485 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00022012299999999999 2024-09-13T11:22:44.486 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:44.486 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:44.486 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:22:44.486 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.043721+0000", 2024-09-13T11:22:44.486 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.7740999999999999e-05 2024-09-13T11:22:44.486 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:44.486 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:44.486 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:22:44.486 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.043781+0000", 2024-09-13T11:22:44.486 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.9771000000000001e-05 2024-09-13T11:22:44.486 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:22:44.486 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:22:44.486 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:22:44.487 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:44.487 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:44.487 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:82 3.2 3:4daba74e:::benchmark_data_smithi177_31116_object81:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e33)", 2024-09-13T11:22:44.487 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:22:27.950329+0000", 2024-09-13T11:22:44.487 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 16.440998779000001, 2024-09-13T11:22:44.487 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0041556340000000001, 2024-09-13T11:22:44.487 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:22:44.487 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:22:44.487 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:44.487 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:22:44.487 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:22:44.487 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:22:44.487 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 82 2024-09-13T11:22:44.487 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:44.488 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:22:44.488 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:44.488 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:22:44.488 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.950329+0000", 2024-09-13T11:22:44.488 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:22:44.488 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:44.488 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:44.488 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:22:44.488 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.950329+0000", 2024-09-13T11:22:44.488 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:22:44.488 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:44.488 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:44.488 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:22:44.488 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.950332+0000", 2024-09-13T11:22:44.489 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.5269999999999999e-06 2024-09-13T11:22:44.489 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:44.489 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:44.489 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:22:44.489 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.950503+0000", 2024-09-13T11:22:44.489 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00017101 2024-09-13T11:22:44.489 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:44.489 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:44.489 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:22:44.489 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.950520+0000", 2024-09-13T11:22:44.489 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.6852000000000001e-05 2024-09-13T11:22:44.489 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:44.489 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:44.490 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:22:44.490 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.950529+0000", 2024-09-13T11:22:44.490 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.8980000000000002e-06 2024-09-13T11:22:44.490 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:44.490 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:44.490 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:22:44.490 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.950679+0000", 2024-09-13T11:22:44.490 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000150073 2024-09-13T11:22:44.490 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:44.490 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:44.490 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:22:44.490 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.951050+0000", 2024-09-13T11:22:44.490 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00037121799999999998 2024-09-13T11:22:44.490 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:44.491 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:44.491 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 12", 2024-09-13T11:22:44.491 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.951454+0000", 2024-09-13T11:22:44.491 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00040372200000000002 2024-09-13T11:22:44.491 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:44.491 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:44.491 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:22:44.491 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.953357+0000", 2024-09-13T11:22:44.491 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001902613 2024-09-13T11:22:44.491 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:44.491 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:44.491 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:22:44.491 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.954420+0000", 2024-09-13T11:22:44.491 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0010635550000000001 2024-09-13T11:22:44.492 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:44.492 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:44.492 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:22:44.492 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.954455+0000", 2024-09-13T11:22:44.492 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.5203999999999997e-05 2024-09-13T11:22:44.492 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:44.492 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:44.492 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:22:44.492 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.954484+0000", 2024-09-13T11:22:44.492 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.8962000000000002e-05 2024-09-13T11:22:44.492 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:22:44.492 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:22:44.492 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:22:44.492 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:44.493 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:44.493 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.4665.0:83 3.e e33/30 3:79afbc3f:::benchmark_data_smithi177_31116_object82:head v 33'3, mlcod=32'1)", 2024-09-13T11:22:44.493 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:22:27.963374+0000", 2024-09-13T11:22:44.493 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 16.427954013000001, 2024-09-13T11:22:44.493 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001227709, 2024-09-13T11:22:44.493 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:22:44.493 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:22:44.493 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:44.493 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:22:44.493 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:44.493 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:22:44.493 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.963374+0000", 2024-09-13T11:22:44.493 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:22:44.493 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:44.494 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:44.494 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:22:44.494 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.963374+0000", 2024-09-13T11:22:44.494 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:22:44.494 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:44.494 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:44.494 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:22:44.494 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.963374+0000", 2024-09-13T11:22:44.494 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.79e-07 2024-09-13T11:22:44.494 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:44.494 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:44.494 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:22:44.494 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.963518+0000", 2024-09-13T11:22:44.494 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00014423100000000001 2024-09-13T11:22:44.494 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:44.495 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:44.495 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:22:44.495 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.963544+0000", 2024-09-13T11:22:44.495 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.6393e-05 2024-09-13T11:22:44.495 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:44.495 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:44.495 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:22:44.495 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.963553+0000", 2024-09-13T11:22:44.495 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.0110000000000006e-06 2024-09-13T11:22:44.495 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:44.495 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:44.495 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:22:44.495 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.963610+0000", 2024-09-13T11:22:44.495 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.6656000000000001e-05 2024-09-13T11:22:44.496 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:44.496 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:44.496 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:22:44.496 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.963629+0000", 2024-09-13T11:22:44.496 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.8678e-05 2024-09-13T11:22:44.496 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:44.496 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:44.496 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:22:44.496 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.964526+0000", 2024-09-13T11:22:44.496 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00089681899999999998 2024-09-13T11:22:44.496 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:44.496 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:44.496 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:22:44.496 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:27.964601+0000", 2024-09-13T11:22:44.497 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.5642000000000005e-05 2024-09-13T11:22:44.497 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:22:44.497 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:22:44.497 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:22:44.497 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:44.497 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:44.497 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.4665.0:86 3.e e33/30 3:763be569:::benchmark_data_smithi177_31116_object85:head v 33'4, mlcod=32'2)", 2024-09-13T11:22:44.497 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:22:28.336613+0000", 2024-09-13T11:22:44.497 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 16.054714691000001, 2024-09-13T11:22:44.497 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0023452870000000002, 2024-09-13T11:22:44.497 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:22:44.497 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:22:44.497 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:44.497 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:22:44.498 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:44.498 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:22:44.498 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:28.336613+0000", 2024-09-13T11:22:44.498 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:22:44.498 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:44.498 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:44.498 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:22:44.498 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:28.336613+0000", 2024-09-13T11:22:44.498 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:22:44.498 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:44.498 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:44.498 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:22:44.498 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:28.336617+0000", 2024-09-13T11:22:44.498 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.7790000000000002e-06 2024-09-13T11:22:44.499 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:44.499 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:44.499 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:22:44.499 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:28.336753+0000", 2024-09-13T11:22:44.499 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00013588200000000001 2024-09-13T11:22:44.499 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:44.499 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:44.499 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:22:44.499 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:28.336822+0000", 2024-09-13T11:22:44.499 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.9549999999999996e-05 2024-09-13T11:22:44.499 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:44.499 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:44.499 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:22:44.499 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:28.336837+0000", 2024-09-13T11:22:44.500 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4878999999999999e-05 2024-09-13T11:22:44.500 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:44.500 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:44.500 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:22:44.500 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:28.336989+0000", 2024-09-13T11:22:44.500 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000152134 2024-09-13T11:22:44.500 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:44.500 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:44.500 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:22:44.500 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:28.337051+0000", 2024-09-13T11:22:44.500 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.1443000000000001e-05 2024-09-13T11:22:44.500 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:44.501 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:44.501 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:22:44.501 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:28.338857+0000", 2024-09-13T11:22:44.501 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0018069029999999999 2024-09-13T11:22:44.501 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:44.501 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:44.501 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:22:44.501 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:28.338958+0000", 2024-09-13T11:22:44.501 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000100717 2024-09-13T11:22:44.501 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:22:44.501 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:22:44.501 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:22:44.501 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:22:44.501 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:22:44.501 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-13T11:22:44.503 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-13T11:22:44.504 INFO:tasks.ceph.osd.3.smithi177.stderr:2024-09-13T11:22:44.500+0000 7f10adba4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26574) UID: 0 2024-09-13T11:22:44.522 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T11:22:44.604 INFO:tasks.ceph.osd.11.smithi177.stderr:2024-09-13T11:22:44.600+0000 7f24792c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26580) UID: 0 2024-09-13T11:22:44.639 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-09-13T11:22:44.639 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [], 2024-09-13T11:22:44.639 INFO:teuthology.orchestra.run.smithi177.stdout: "num_ops": 0 2024-09-13T11:22:44.639 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-09-13T11:22:44.653 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-13T11:22:44.689 INFO:teuthology.orchestra.run.smithi001.stdout:osd.0: {} 2024-09-13T11:22:44.689 INFO:teuthology.orchestra.run.smithi001.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T11:22:44.690 INFO:teuthology.orchestra.run.smithi001.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-09-13T11:22:44.717 INFO:teuthology.orchestra.run.smithi001.stdout:osd.2: {} 2024-09-13T11:22:44.717 INFO:teuthology.orchestra.run.smithi001.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T11:22:44.738 INFO:tasks.ceph.osd.3.smithi177.stderr:2024-09-13T11:22:44.736+0000 7f10adba4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26574) UID: 0 2024-09-13T11:22:44.743 INFO:teuthology.orchestra.run.smithi001.stdout:osd.3: {} 2024-09-13T11:22:44.743 INFO:teuthology.orchestra.run.smithi001.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T11:22:44.773 INFO:teuthology.orchestra.run.smithi001.stdout:osd.4: {} 2024-09-13T11:22:44.773 INFO:teuthology.orchestra.run.smithi001.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T11:22:44.804 INFO:teuthology.orchestra.run.smithi001.stdout:osd.5: {} 2024-09-13T11:22:44.804 INFO:teuthology.orchestra.run.smithi001.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T11:22:44.805 INFO:tasks.ceph.osd.4.smithi001.stderr:2024-09-13T11:22:44.800+0000 7ff8e0218640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26310) UID: 0 2024-09-13T11:22:44.817 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-09-13T11:22:44.817 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:22:44.832 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-13T11:22:44.836 INFO:teuthology.orchestra.run.smithi001.stdout:osd.6: {} 2024-09-13T11:22:44.836 INFO:teuthology.orchestra.run.smithi001.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T11:22:44.864 INFO:teuthology.orchestra.run.smithi001.stdout:osd.7: {} 2024-09-13T11:22:44.864 INFO:teuthology.orchestra.run.smithi001.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T11:22:44.894 INFO:teuthology.orchestra.run.smithi001.stdout:osd.8: {} 2024-09-13T11:22:44.895 INFO:teuthology.orchestra.run.smithi001.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-13T11:22:44.906 INFO:tasks.ceph.osd.14.smithi162.stderr:2024-09-13T11:22:44.903+0000 7f235c0db640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26319) UID: 0 2024-09-13T11:22:44.926 INFO:teuthology.orchestra.run.smithi001.stdout:osd.9: {} 2024-09-13T11:22:44.926 INFO:teuthology.orchestra.run.smithi001.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-13T11:22:44.958 INFO:teuthology.orchestra.run.smithi001.stdout:osd.10: {} 2024-09-13T11:22:44.959 INFO:teuthology.orchestra.run.smithi001.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-13T11:22:44.993 INFO:teuthology.orchestra.run.smithi001.stdout:osd.11: {} 2024-09-13T11:22:44.994 INFO:teuthology.orchestra.run.smithi001.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-13T11:22:44.998 INFO:tasks.radosbench.radosbench.0.smithi177.stdout: 18 16 87 71 0.246493 0 - 0.0234801 2024-09-13T11:22:44.999 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-09-13T11:22:44.999 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:22:45.007 INFO:tasks.ceph.osd.5.smithi092.stderr:2024-09-13T11:22:45.007+0000 7f6f0b705640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26346) UID: 0 2024-09-13T11:22:45.014 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-13T11:22:45.028 INFO:teuthology.orchestra.run.smithi001.stdout:osd.12: {} 2024-09-13T11:22:45.028 INFO:teuthology.orchestra.run.smithi001.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-13T11:22:45.063 INFO:teuthology.orchestra.run.smithi001.stdout:osd.13: {} 2024-09-13T11:22:45.064 INFO:teuthology.orchestra.run.smithi001.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-13T11:22:45.098 INFO:teuthology.orchestra.run.smithi001.stdout:osd.14: {} 2024-09-13T11:22:45.098 INFO:teuthology.orchestra.run.smithi001.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-13T11:22:45.108 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-09-13T11:22:45.103+0000 7f97befc9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26315) UID: 0 2024-09-13T11:22:45.132 INFO:teuthology.orchestra.run.smithi001.stdout:osd.15: {} 2024-09-13T11:22:45.132 INFO:teuthology.orchestra.run.smithi001.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-13T11:22:45.136 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-09-13T11:22:45.136 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:22:45.138 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-13T11:22:45.138 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-13T11:22:45.151 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-13T11:22:45.166 INFO:tasks.ceph.osd.1.smithi092.stderr:2024-09-13T11:22:45.163+0000 7fe77ba097c0 -1 osd.1 29 log_to_monitors true 2024-09-13T11:22:45.180 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-13T11:22:45.209 INFO:tasks.ceph.osd.14.smithi162.stderr:2024-09-13T11:22:45.207+0000 7f235c0db640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26319) UID: 0 2024-09-13T11:22:45.269 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-09-13T11:22:45.269 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:22:45.283 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-13T11:22:45.287 INFO:tasks.ceph.osd.1.smithi092.stderr:2024-09-13T11:22:45.283+0000 7fe7737ec640 -1 osd.1 29 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-13T11:22:45.306 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-09-13T11:22:45.306 INFO:teuthology.orchestra.run.smithi092.stdout: "ops": [], 2024-09-13T11:22:45.307 INFO:teuthology.orchestra.run.smithi092.stdout: "num_ops": 0 2024-09-13T11:22:45.307 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-09-13T11:22:45.310 INFO:tasks.ceph.osd.4.smithi001.stderr:2024-09-13T11:22:45.304+0000 7ff8e0218640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26310) UID: 0 2024-09-13T11:22:45.321 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 0 1 2024-09-13T11:22:45.367 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-13T11:22:45.407 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-09-13T11:22:45.407 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:22:45.411 INFO:tasks.ceph.osd.3.smithi177.stderr:2024-09-13T11:22:45.408+0000 7f10adba4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26574) UID: 0 2024-09-13T11:22:45.421 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-09-13T11:22:45.512 INFO:tasks.ceph.osd.8.smithi001.stderr:2024-09-13T11:22:45.508+0000 7fb9b48ae640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26312) UID: 0 2024-09-13T11:22:45.584 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-09-13T11:22:45.584 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:22:45.598 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-09-13T11:22:45.613 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-09-13T11:22:45.607+0000 7f97befc9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26315) UID: 0 2024-09-13T11:22:45.713 INFO:tasks.ceph.osd.15.smithi177.stderr:2024-09-13T11:22:45.708+0000 7f9d6ce77640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26578) UID: 0 2024-09-13T11:22:45.726 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-09-13T11:22:45.726 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:22:45.740 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-09-13T11:22:45.814 INFO:tasks.ceph.osd.14.smithi162.stderr:2024-09-13T11:22:45.811+0000 7f235c0db640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26319) UID: 0 2024-09-13T11:22:45.906 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-09-13T11:22:45.906 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:22:45.915 INFO:tasks.ceph.osd.11.smithi177.stderr:2024-09-13T11:22:45.912+0000 7f24792c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26580) UID: 0 2024-09-13T11:22:45.920 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_ops_in_flight 2024-09-13T11:22:45.998 INFO:tasks.radosbench.radosbench.0.smithi177.stdout: 19 16 87 71 0.23352 0 - 0.0234801 2024-09-13T11:22:46.016 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-09-13T11:22:46.011+0000 7f97befc9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26315) UID: 0 2024-09-13T11:22:46.082 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-09-13T11:22:46.082 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:22:46.096 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_blocked_ops 2024-09-13T11:22:46.138 INFO:tasks.ceph.osd.15.smithi177.stderr:2024-09-13T11:22:46.136+0000 7f9d6ce77640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26578) UID: 0 2024-09-13T11:22:46.161 INFO:tasks.ceph.osd.1.smithi092.stderr:2024-09-13T11:22:46.159+0000 7fe76f3f6640 -1 osd.1 29 handle_osd_map injecting map crc failure 2024-09-13T11:22:46.217 INFO:tasks.ceph.osd.8.smithi001.stderr:2024-09-13T11:22:46.216+0000 7fb9b48ae640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26312) UID: 0 2024-09-13T11:22:46.259 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-09-13T11:22:46.259 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:22:46.273 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_historic_ops 2024-09-13T11:22:46.275 INFO:teuthology.orchestra.run.smithi001.stderr:reweighted osd.0 to 1 (10000) 2024-09-13T11:22:46.276 INFO:teuthology.orchestra.run.smithi001.stderr:nodeep-scrub is unset 2024-09-13T11:22:46.303 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 1 1 2024-09-13T11:22:46.319 INFO:tasks.ceph.osd.14.smithi162.stderr:2024-09-13T11:22:46.315+0000 7f235c0db640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26319) UID: 0 2024-09-13T11:22:46.391 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-09-13T11:22:46.392 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:22:46.405 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-13T11:22:46.419 INFO:tasks.ceph.osd.15.smithi177.stderr:2024-09-13T11:22:46.420+0000 7f9d6ce77640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26578) UID: 0 2024-09-13T11:22:46.521 INFO:tasks.ceph.osd.14.smithi162.stderr:2024-09-13T11:22:46.519+0000 7f235c0db640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26319) UID: 0 2024-09-13T11:22:46.538 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-09-13T11:22:46.538 INFO:teuthology.orchestra.run.smithi092.stdout: "ops": [ 2024-09-13T11:22:46.538 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:46.538 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:3 3.4 3.125e09e4 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e40)", 2024-09-13T11:22:46.538 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:46.304641+0000", 2024-09-13T11:22:46.538 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 0.234428785, 2024-09-13T11:22:46.538 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.23444984999999999, 2024-09-13T11:22:46.539 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:22:46.539 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:22:46.539 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "queued for pg", 2024-09-13T11:22:46.539 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:22:46.539 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:22:46.539 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:22:46.539 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 3 2024-09-13T11:22:46.539 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:46.539 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:22:46.539 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:46.539 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:22:46.539 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.304641+0000", 2024-09-13T11:22:46.539 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:22:46.540 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:46.540 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:46.540 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:22:46.540 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.304641+0000", 2024-09-13T11:22:46.540 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:22:46.540 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:46.540 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:46.540 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:22:46.540 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.304644+0000", 2024-09-13T11:22:46.540 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.5130000000000001e-06 2024-09-13T11:22:46.540 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:46.540 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:46.541 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:22:46.541 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305058+0000", 2024-09-13T11:22:46.541 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000414277 2024-09-13T11:22:46.541 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:46.541 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:46.541 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:22:46.541 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305124+0000", 2024-09-13T11:22:46.541 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.5813000000000002e-05 2024-09-13T11:22:46.541 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:46.541 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:46.541 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:22:46.541 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305156+0000", 2024-09-13T11:22:46.542 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.1489000000000003e-05 2024-09-13T11:22:46.542 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:22:46.542 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:22:46.542 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:22:46.542 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:46.542 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:46.542 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:29 3.4 3.b7bbfbe4 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-09-13T11:22:46.542 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:46.305215+0000", 2024-09-13T11:22:46.542 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 0.233854482, 2024-09-13T11:22:46.542 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.23391128999999999, 2024-09-13T11:22:46.542 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:22:46.542 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:22:46.542 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "queued for pg", 2024-09-13T11:22:46.543 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:22:46.543 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:22:46.543 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:22:46.543 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 29 2024-09-13T11:22:46.543 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:46.543 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:22:46.543 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:46.543 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:22:46.543 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305215+0000", 2024-09-13T11:22:46.543 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:22:46.543 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:46.543 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:46.543 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:22:46.544 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305215+0000", 2024-09-13T11:22:46.544 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:22:46.544 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:46.544 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:46.544 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:22:46.544 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305218+0000", 2024-09-13T11:22:46.544 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.7030000000000002e-06 2024-09-13T11:22:46.544 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:46.544 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:46.544 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:22:46.544 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305366+0000", 2024-09-13T11:22:46.544 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00014825400000000001 2024-09-13T11:22:46.544 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:46.545 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:46.545 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:22:46.545 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305386+0000", 2024-09-13T11:22:46.545 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.9689000000000001e-05 2024-09-13T11:22:46.545 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:46.545 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:46.545 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:22:46.545 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305403+0000", 2024-09-13T11:22:46.545 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.7600000000000001e-05 2024-09-13T11:22:46.545 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:22:46.545 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:22:46.545 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:22:46.546 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:46.546 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:46.546 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:30 3.4 3.547ec0f4 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-09-13T11:22:46.546 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:46.305453+0000", 2024-09-13T11:22:46.546 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 0.233616358, 2024-09-13T11:22:46.546 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.23369837199999999, 2024-09-13T11:22:46.546 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:22:46.546 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:22:46.546 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "queued for pg", 2024-09-13T11:22:46.546 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:22:46.546 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:22:46.546 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:22:46.546 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 30 2024-09-13T11:22:46.547 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:46.547 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:22:46.547 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:46.547 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:22:46.547 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305453+0000", 2024-09-13T11:22:46.547 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:22:46.547 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:46.547 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:46.547 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:22:46.547 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305453+0000", 2024-09-13T11:22:46.547 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:22:46.547 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:46.548 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:46.548 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:22:46.548 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305456+0000", 2024-09-13T11:22:46.548 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.9560000000000002e-06 2024-09-13T11:22:46.548 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:46.548 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:46.548 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:22:46.548 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305597+0000", 2024-09-13T11:22:46.548 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00014122700000000001 2024-09-13T11:22:46.548 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:46.548 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:46.548 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:22:46.549 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305615+0000", 2024-09-13T11:22:46.549 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.8139999999999999e-05 2024-09-13T11:22:46.549 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:46.549 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:46.549 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:22:46.549 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305632+0000", 2024-09-13T11:22:46.549 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.6101999999999999e-05 2024-09-13T11:22:46.549 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:22:46.549 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:22:46.549 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:22:46.549 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:46.549 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:46.549 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:70 3.4 3.1404a604 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-09-13T11:22:46.550 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:46.305665+0000", 2024-09-13T11:22:46.550 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 0.23340421, 2024-09-13T11:22:46.550 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.23351088, 2024-09-13T11:22:46.550 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:22:46.550 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:22:46.550 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "queued for pg", 2024-09-13T11:22:46.550 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:22:46.550 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:22:46.550 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:22:46.550 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 70 2024-09-13T11:22:46.550 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:46.551 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:22:46.551 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:46.551 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:22:46.551 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305665+0000", 2024-09-13T11:22:46.551 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:22:46.551 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:46.551 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:46.551 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:22:46.551 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305665+0000", 2024-09-13T11:22:46.551 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:22:46.551 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:46.551 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:46.551 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:22:46.552 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305668+0000", 2024-09-13T11:22:46.552 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.5890000000000001e-06 2024-09-13T11:22:46.552 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:46.552 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:46.552 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:22:46.552 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305791+0000", 2024-09-13T11:22:46.552 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000123545 2024-09-13T11:22:46.552 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:46.552 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:46.552 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:22:46.552 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305808+0000", 2024-09-13T11:22:46.552 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.668e-05 2024-09-13T11:22:46.553 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:46.553 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:46.553 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:22:46.553 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305824+0000", 2024-09-13T11:22:46.553 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.6385e-05 2024-09-13T11:22:46.553 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:22:46.553 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:22:46.553 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:22:46.553 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:22:46.553 INFO:teuthology.orchestra.run.smithi092.stdout: ], 2024-09-13T11:22:46.553 INFO:teuthology.orchestra.run.smithi092.stdout: "num_ops": 4 2024-09-13T11:22:46.553 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-09-13T11:22:46.554 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-13T11:22:46.622 INFO:tasks.ceph.osd.15.smithi177.stderr:2024-09-13T11:22:46.620+0000 7f9d6ce77640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26578) UID: 0 2024-09-13T11:22:46.678 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-09-13T11:22:46.678 INFO:teuthology.orchestra.run.smithi092.stdout: "ops": [], 2024-09-13T11:22:46.678 INFO:teuthology.orchestra.run.smithi092.stdout: "complaint_time": 30, 2024-09-13T11:22:46.678 INFO:teuthology.orchestra.run.smithi092.stdout: "num_blocked_ops": 0 2024-09-13T11:22:46.678 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-09-13T11:22:46.692 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-13T11:22:46.723 INFO:tasks.ceph.osd.8.smithi001.stderr:2024-09-13T11:22:46.716+0000 7fb9b48ae640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26312) UID: 0 2024-09-13T11:22:46.816 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-09-13T11:22:46.816 INFO:teuthology.orchestra.run.smithi092.stdout: "size": 20, 2024-09-13T11:22:46.816 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 600, 2024-09-13T11:22:46.816 INFO:teuthology.orchestra.run.smithi092.stdout: "ops": [] 2024-09-13T11:22:46.816 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-09-13T11:22:46.824 INFO:tasks.ceph.osd.5.smithi092.stderr:2024-09-13T11:22:46.823+0000 7f6f0b705640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26346) UID: 0 2024-09-13T11:22:46.830 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-13T11:22:46.853 INFO:tasks.ceph.osd.3.smithi177.stderr:2024-09-13T11:22:46.852+0000 7f10a45d5640 -1 osd.3 39 handle_osd_map injecting map crc failure 2024-09-13T11:22:46.925 INFO:tasks.ceph.osd.7.smithi177.stderr:2024-09-13T11:22:46.924+0000 7f897b51d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26575) UID: 0 2024-09-13T11:22:46.965 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-13T11:22:46.965 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:22:46.980 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-13T11:22:46.998 INFO:tasks.radosbench.radosbench.0.smithi177.stdout:2024-09-13T11:22:47.000334+0000 min lat: 0.00151137 max lat: 0.895203 avg lat: 0.0234801 2024-09-13T11:22:46.998 INFO:tasks.radosbench.radosbench.0.smithi177.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-13T11:22:46.998 INFO:tasks.radosbench.radosbench.0.smithi177.stdout: 20 16 87 71 0.221844 0 - 0.0234801 2024-09-13T11:22:47.026 INFO:tasks.ceph.osd.14.smithi162.stderr:2024-09-13T11:22:47.023+0000 7f235c0db640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26319) UID: 0 2024-09-13T11:22:47.106 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-13T11:22:47.106 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:22:47.121 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-13T11:22:47.126 INFO:tasks.ceph.osd.7.smithi177.stderr:2024-09-13T11:22:47.128+0000 7f897b51d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26575) UID: 0 2024-09-13T11:22:47.139 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T11:22:47.227 INFO:tasks.ceph.osd.11.smithi177.stderr:2024-09-13T11:22:47.228+0000 7f24792c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26580) UID: 0 2024-09-13T11:22:47.240 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-13T11:22:47.240 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:22:47.256 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-13T11:22:47.285 INFO:teuthology.orchestra.run.smithi001.stderr:reweighted osd.1 to 1 (10000) 2024-09-13T11:22:47.312 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 2 1 2024-09-13T11:22:47.314 INFO:tasks.ceph.osd.15.smithi177.stderr:2024-09-13T11:22:47.312+0000 7f9d638a8640 -1 osd.15 38 handle_osd_map injecting map crc failure 2024-09-13T11:22:47.330 INFO:teuthology.orchestra.run.smithi001.stdout:osd.0: {} 2024-09-13T11:22:47.330 INFO:teuthology.orchestra.run.smithi001.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T11:22:47.331 INFO:tasks.ceph.osd.4.smithi001.stderr:2024-09-13T11:22:47.324+0000 7ff8e0218640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26310) UID: 0 2024-09-13T11:22:47.357 INFO:teuthology.orchestra.run.smithi001.stdout:osd.1: {} 2024-09-13T11:22:47.357 INFO:teuthology.orchestra.run.smithi001.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T11:22:47.384 INFO:teuthology.orchestra.run.smithi001.stdout:osd.2: {} 2024-09-13T11:22:47.385 INFO:teuthology.orchestra.run.smithi001.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T11:22:47.397 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-13T11:22:47.398 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:22:47.415 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-13T11:22:47.427 INFO:teuthology.orchestra.run.smithi001.stdout:osd.3: {} 2024-09-13T11:22:47.428 INFO:teuthology.orchestra.run.smithi001.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T11:22:47.430 INFO:tasks.ceph.osd.8.smithi001.stderr:2024-09-13T11:22:47.424+0000 7fb9b48ae640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26312) UID: 0 2024-09-13T11:22:47.462 INFO:teuthology.orchestra.run.smithi001.stdout:osd.4: {} 2024-09-13T11:22:47.462 INFO:teuthology.orchestra.run.smithi001.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T11:22:47.488 INFO:teuthology.orchestra.run.smithi001.stdout:osd.5: {} 2024-09-13T11:22:47.489 INFO:teuthology.orchestra.run.smithi001.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T11:22:47.529 INFO:teuthology.orchestra.run.smithi001.stdout:osd.6: {} 2024-09-13T11:22:47.580 INFO:teuthology.orchestra.run.smithi001.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T11:22:47.580 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-13T11:22:47.580 INFO:teuthology.orchestra.run.smithi001.stdout: "ops": [], 2024-09-13T11:22:47.580 INFO:teuthology.orchestra.run.smithi001.stdout: "complaint_time": 30, 2024-09-13T11:22:47.580 INFO:teuthology.orchestra.run.smithi001.stdout: "num_blocked_ops": 0 2024-09-13T11:22:47.580 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-13T11:22:47.581 INFO:tasks.ceph.osd.15.smithi177.stderr:2024-09-13T11:22:47.528+0000 7f9d6ce77640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26578) UID: 0 2024-09-13T11:22:47.582 INFO:tasks.ceph.osd.4.smithi001.stderr:2024-09-13T11:22:47.576+0000 7ff8d0c3d640 -1 osd.4 39 handle_osd_map injecting map crc failure 2024-09-13T11:22:47.585 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-13T11:22:47.630 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-09-13T11:22:47.627+0000 7f97befc9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26315) UID: 0 2024-09-13T11:22:47.669 INFO:teuthology.orchestra.run.smithi001.stdout:osd.7: {} 2024-09-13T11:22:47.669 INFO:teuthology.orchestra.run.smithi001.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T11:22:47.691 INFO:teuthology.orchestra.run.smithi001.stdout:osd.8: {} 2024-09-13T11:22:47.691 INFO:teuthology.orchestra.run.smithi001.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-13T11:22:47.717 INFO:teuthology.orchestra.run.smithi001.stdout:osd.9: {} 2024-09-13T11:22:47.717 INFO:teuthology.orchestra.run.smithi001.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-13T11:22:47.731 INFO:tasks.ceph.osd.3.smithi177.stderr:2024-09-13T11:22:47.732+0000 7f10adba4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26574) UID: 0 2024-09-13T11:22:47.741 INFO:teuthology.orchestra.run.smithi001.stdout:osd.10: {} 2024-09-13T11:22:47.742 INFO:teuthology.orchestra.run.smithi001.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-13T11:22:47.758 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-13T11:22:47.758 INFO:teuthology.orchestra.run.smithi001.stdout: "size": 20, 2024-09-13T11:22:47.758 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 600, 2024-09-13T11:22:47.758 INFO:teuthology.orchestra.run.smithi001.stdout: "ops": [] 2024-09-13T11:22:47.758 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-13T11:22:47.766 INFO:teuthology.orchestra.run.smithi001.stdout:osd.11: {} 2024-09-13T11:22:47.766 INFO:teuthology.orchestra.run.smithi001.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-13T11:22:47.772 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-09-13T11:22:47.789 INFO:teuthology.orchestra.run.smithi001.stdout:osd.12: {} 2024-09-13T11:22:47.789 INFO:teuthology.orchestra.run.smithi001.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-13T11:22:47.818 INFO:teuthology.orchestra.run.smithi001.stdout:osd.13: {} 2024-09-13T11:22:47.819 INFO:teuthology.orchestra.run.smithi001.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-13T11:22:47.832 INFO:tasks.ceph.osd.14.smithi162.stderr:2024-09-13T11:22:47.827+0000 7f235c0db640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26319) UID: 0 2024-09-13T11:22:47.847 INFO:teuthology.orchestra.run.smithi001.stdout:osd.14: {} 2024-09-13T11:22:47.847 INFO:teuthology.orchestra.run.smithi001.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-13T11:22:47.878 INFO:teuthology.orchestra.run.smithi001.stdout:osd.15: {} 2024-09-13T11:22:47.878 INFO:teuthology.orchestra.run.smithi001.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-13T11:22:47.904 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-13T11:22:47.904 INFO:teuthology.orchestra.run.smithi001.stdout: "ops": [], 2024-09-13T11:22:47.904 INFO:teuthology.orchestra.run.smithi001.stdout: "num_ops": 0 2024-09-13T11:22:47.904 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-13T11:22:47.918 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-09-13T11:22:47.933 INFO:tasks.ceph.osd.10.smithi162.stderr:2024-09-13T11:22:47.927+0000 7fb59c9f0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26317) UID: 0 2024-09-13T11:22:47.998 INFO:tasks.radosbench.radosbench.0.smithi177.stdout: 21 16 177 161 0.479099 0.296053 0.0535309 0.769903 2024-09-13T11:22:48.034 INFO:tasks.ceph.osd.2.smithi162.stderr:2024-09-13T11:22:48.031+0000 7fe92b35a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26313) UID: 0 2024-09-13T11:22:48.090 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-13T11:22:48.091 INFO:teuthology.orchestra.run.smithi001.stdout: "ops": [], 2024-09-13T11:22:48.091 INFO:teuthology.orchestra.run.smithi001.stdout: "complaint_time": 30, 2024-09-13T11:22:48.091 INFO:teuthology.orchestra.run.smithi001.stdout: "num_blocked_ops": 0 2024-09-13T11:22:48.091 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-13T11:22:48.104 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-09-13T11:22:48.135 INFO:tasks.ceph.osd.9.smithi092.stderr:2024-09-13T11:22:48.131+0000 7ff5ab3d0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26348) UID: 0 2024-09-13T11:22:48.236 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-13T11:22:48.236 INFO:teuthology.orchestra.run.smithi001.stdout: "size": 20, 2024-09-13T11:22:48.236 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 600, 2024-09-13T11:22:48.236 INFO:teuthology.orchestra.run.smithi001.stdout: "ops": [ 2024-09-13T11:22:48.236 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.236 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4665.0:16 3.7 e32/30 3:e0940737:::benchmark_data_smithi177_31116_object15:head v 32'1, mlcod=0'0)", 2024-09-13T11:22:48.236 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:22:27.002190+0000", 2024-09-13T11:22:48.236 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 21.229822471999999, 2024-09-13T11:22:48.237 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0014874980000000001, 2024-09-13T11:22:48.237 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:22:48.237 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:22:48.237 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:48.237 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:22:48.237 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.237 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:22:48.237 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.002190+0000", 2024-09-13T11:22:48.237 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:22:48.237 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.237 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.237 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:22:48.237 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.002190+0000", 2024-09-13T11:22:48.238 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:22:48.238 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.238 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.238 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:22:48.238 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.002191+0000", 2024-09-13T11:22:48.238 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.2900000000000002e-07 2024-09-13T11:22:48.238 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.238 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.238 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:22:48.238 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.002373+0000", 2024-09-13T11:22:48.238 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00018228400000000001 2024-09-13T11:22:48.238 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.238 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.238 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:22:48.239 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.002414+0000", 2024-09-13T11:22:48.239 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.1162999999999999e-05 2024-09-13T11:22:48.239 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.239 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.239 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:22:48.239 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.002429+0000", 2024-09-13T11:22:48.239 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.5053e-05 2024-09-13T11:22:48.239 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.239 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.239 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:22:48.239 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.002490+0000", 2024-09-13T11:22:48.239 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.1316999999999996e-05 2024-09-13T11:22:48.239 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.240 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.240 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:22:48.240 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.002526+0000", 2024-09-13T11:22:48.240 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.5788999999999999e-05 2024-09-13T11:22:48.240 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.240 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.240 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:22:48.240 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.003571+0000", 2024-09-13T11:22:48.240 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.001044445 2024-09-13T11:22:48.240 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.240 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.241 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:22:48.241 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.003677+0000", 2024-09-13T11:22:48.241 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000106518 2024-09-13T11:22:48.241 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:22:48.241 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:22:48.241 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:22:48.241 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.241 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.241 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4665.0:25 3.7 e32/30 3:e6850330:::benchmark_data_smithi177_31116_object24:head v 32'2, mlcod=0'0)", 2024-09-13T11:22:48.241 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:22:27.012207+0000", 2024-09-13T11:22:48.241 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 21.219805289, 2024-09-13T11:22:48.241 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0015694610000000001, 2024-09-13T11:22:48.241 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:22:48.242 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:22:48.242 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:48.242 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:22:48.242 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.242 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:22:48.242 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.012207+0000", 2024-09-13T11:22:48.242 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:22:48.242 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.242 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.242 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:22:48.242 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.012207+0000", 2024-09-13T11:22:48.242 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:22:48.242 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.243 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.243 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:22:48.243 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.012209+0000", 2024-09-13T11:22:48.243 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.3070000000000001e-06 2024-09-13T11:22:48.243 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.243 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.243 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:22:48.243 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.012329+0000", 2024-09-13T11:22:48.243 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000120061 2024-09-13T11:22:48.243 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.243 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.243 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:22:48.243 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.012445+0000", 2024-09-13T11:22:48.243 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000116104 2024-09-13T11:22:48.244 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.244 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.244 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:22:48.244 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.012458+0000", 2024-09-13T11:22:48.244 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2723e-05 2024-09-13T11:22:48.244 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.244 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.244 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:22:48.244 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.012598+0000", 2024-09-13T11:22:48.244 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00014023200000000001 2024-09-13T11:22:48.244 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.244 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.244 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:22:48.244 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.012633+0000", 2024-09-13T11:22:48.245 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.4643e-05 2024-09-13T11:22:48.245 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.245 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.245 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:22:48.245 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.013733+0000", 2024-09-13T11:22:48.245 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.001100214 2024-09-13T11:22:48.245 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.245 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.245 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:22:48.245 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.013776+0000", 2024-09-13T11:22:48.245 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.3176999999999997e-05 2024-09-13T11:22:48.245 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:22:48.245 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:22:48.246 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:22:48.246 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.246 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.246 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:20 3.e 3:794d9a5b:::benchmark_data_smithi177_31116_object19:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-09-13T11:22:48.246 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:22:27.012384+0000", 2024-09-13T11:22:48.246 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 21.219628295, 2024-09-13T11:22:48.246 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0065062369999999998, 2024-09-13T11:22:48.246 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:22:48.246 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:22:48.246 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:48.246 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:22:48.246 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:22:48.246 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:22:48.247 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 20 2024-09-13T11:22:48.247 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.247 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:22:48.247 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.247 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:22:48.247 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.012384+0000", 2024-09-13T11:22:48.247 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:22:48.247 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.247 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.247 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:22:48.247 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.012384+0000", 2024-09-13T11:22:48.247 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:22:48.247 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.248 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.248 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:22:48.248 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.012386+0000", 2024-09-13T11:22:48.248 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.6800000000000002e-06 2024-09-13T11:22:48.248 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.248 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.248 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:22:48.248 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.012697+0000", 2024-09-13T11:22:48.248 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00031023200000000002 2024-09-13T11:22:48.248 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.249 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.249 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:22:48.249 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.012711+0000", 2024-09-13T11:22:48.249 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.4184e-05 2024-09-13T11:22:48.249 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.249 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.249 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:22:48.249 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.012720+0000", 2024-09-13T11:22:48.249 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.1069999999999994e-06 2024-09-13T11:22:48.249 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.249 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.249 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:22:48.249 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.012787+0000", 2024-09-13T11:22:48.249 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.6661000000000004e-05 2024-09-13T11:22:48.250 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.250 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.250 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:22:48.250 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.012898+0000", 2024-09-13T11:22:48.250 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000111309 2024-09-13T11:22:48.250 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.250 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.250 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 14", 2024-09-13T11:22:48.250 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.013074+0000", 2024-09-13T11:22:48.250 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000176119 2024-09-13T11:22:48.250 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.250 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.250 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:22:48.250 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.014110+0000", 2024-09-13T11:22:48.251 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0010358100000000001 2024-09-13T11:22:48.251 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.251 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.251 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:22:48.251 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.018780+0000", 2024-09-13T11:22:48.251 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.004669884 2024-09-13T11:22:48.251 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.251 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.251 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:22:48.251 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.018841+0000", 2024-09-13T11:22:48.251 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.1309000000000005e-05 2024-09-13T11:22:48.251 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.251 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.252 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:22:48.252 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.018890+0000", 2024-09-13T11:22:48.252 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.8942e-05 2024-09-13T11:22:48.252 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:22:48.252 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:22:48.252 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:22:48.252 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.252 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.252 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4665.0:33 3.7 e32/30 3:ebfb0388:::benchmark_data_smithi177_31116_object32:head v 32'3, mlcod=0'0)", 2024-09-13T11:22:48.252 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:22:27.014563+0000", 2024-09-13T11:22:48.252 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 21.217448782999998, 2024-09-13T11:22:48.252 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.001063968, 2024-09-13T11:22:48.252 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:22:48.253 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:22:48.253 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:48.253 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:22:48.253 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.253 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:22:48.253 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.014563+0000", 2024-09-13T11:22:48.253 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:22:48.253 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.253 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.253 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:22:48.253 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.014563+0000", 2024-09-13T11:22:48.253 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:22:48.253 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.253 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.254 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:22:48.254 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.014565+0000", 2024-09-13T11:22:48.254 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.263e-06 2024-09-13T11:22:48.254 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.254 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.254 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:22:48.254 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.014630+0000", 2024-09-13T11:22:48.254 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.5777000000000004e-05 2024-09-13T11:22:48.254 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.254 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.254 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:22:48.254 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.014660+0000", 2024-09-13T11:22:48.254 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.9660999999999999e-05 2024-09-13T11:22:48.254 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.255 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.255 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:22:48.255 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.014667+0000", 2024-09-13T11:22:48.255 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.7399999999999998e-06 2024-09-13T11:22:48.255 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.255 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.255 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:22:48.255 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.014716+0000", 2024-09-13T11:22:48.255 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.9094999999999997e-05 2024-09-13T11:22:48.255 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.255 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.255 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:22:48.255 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.014733+0000", 2024-09-13T11:22:48.255 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.7027000000000001e-05 2024-09-13T11:22:48.256 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.256 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.256 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:22:48.256 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.015579+0000", 2024-09-13T11:22:48.256 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000845665 2024-09-13T11:22:48.256 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.256 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.256 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:22:48.256 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.015627+0000", 2024-09-13T11:22:48.256 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.8739999999999998e-05 2024-09-13T11:22:48.256 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:22:48.256 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:22:48.256 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:22:48.256 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.257 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.257 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop_reply(client.4665.0:20 3.e e32/30 ondisk, result = 0)", 2024-09-13T11:22:48.257 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:22:27.018628+0000", 2024-09-13T11:22:48.257 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 21.213384396999999, 2024-09-13T11:22:48.257 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00027831200000000001, 2024-09-13T11:22:48.257 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:22:48.257 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:22:48.257 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:22:48.257 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:22:48.257 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.257 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:22:48.257 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.018628+0000", 2024-09-13T11:22:48.258 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:22:48.258 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.258 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.258 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:22:48.258 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.018628+0000", 2024-09-13T11:22:48.258 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:22:48.258 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.258 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.258 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:22:48.258 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.018630+0000", 2024-09-13T11:22:48.258 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.2010000000000002e-06 2024-09-13T11:22:48.258 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.258 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.258 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:22:48.259 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.018638+0000", 2024-09-13T11:22:48.259 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.9990000000000008e-06 2024-09-13T11:22:48.259 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.259 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.259 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:22:48.259 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.018686+0000", 2024-09-13T11:22:48.259 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.833e-05 2024-09-13T11:22:48.259 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.259 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.259 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:22:48.259 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.018697+0000", 2024-09-13T11:22:48.259 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.111e-05 2024-09-13T11:22:48.259 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.260 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.260 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:22:48.260 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.018765+0000", 2024-09-13T11:22:48.260 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.7714999999999998e-05 2024-09-13T11:22:48.260 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.260 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.260 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:22:48.260 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.018774+0000", 2024-09-13T11:22:48.260 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.6619999999999994e-06 2024-09-13T11:22:48.260 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.260 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.260 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:22:48.260 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.018906+0000", 2024-09-13T11:22:48.260 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00013229499999999999 2024-09-13T11:22:48.261 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:22:48.261 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:22:48.261 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:22:48.261 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.261 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.261 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:46 3.e 3:7d541445:::benchmark_data_smithi177_31116_object45:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-09-13T11:22:48.261 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:22:27.019140+0000", 2024-09-13T11:22:48.261 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 21.212871896999999, 2024-09-13T11:22:48.261 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0023100759999999999, 2024-09-13T11:22:48.261 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:22:48.261 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:22:48.261 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:48.261 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:22:48.262 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:22:48.262 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:22:48.262 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 46 2024-09-13T11:22:48.262 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.262 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:22:48.262 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.262 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:22:48.262 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.019140+0000", 2024-09-13T11:22:48.262 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:22:48.262 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.262 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.262 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:22:48.262 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.019140+0000", 2024-09-13T11:22:48.262 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:22:48.262 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.263 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.263 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:22:48.263 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.019142+0000", 2024-09-13T11:22:48.263 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.9429999999999999e-06 2024-09-13T11:22:48.263 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.263 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.263 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:22:48.263 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.019211+0000", 2024-09-13T11:22:48.263 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.8855000000000004e-05 2024-09-13T11:22:48.263 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.263 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.263 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:22:48.263 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.019227+0000", 2024-09-13T11:22:48.263 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.5973000000000001e-05 2024-09-13T11:22:48.264 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.264 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.264 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:22:48.264 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.019237+0000", 2024-09-13T11:22:48.264 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0438999999999999e-05 2024-09-13T11:22:48.264 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.264 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.264 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:22:48.264 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.019307+0000", 2024-09-13T11:22:48.264 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.9756000000000001e-05 2024-09-13T11:22:48.264 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.264 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.264 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:22:48.264 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.019456+0000", 2024-09-13T11:22:48.265 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000149326 2024-09-13T11:22:48.265 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.265 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.265 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 14", 2024-09-13T11:22:48.265 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.019615+0000", 2024-09-13T11:22:48.265 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00015860900000000001 2024-09-13T11:22:48.265 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.265 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.265 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:22:48.265 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.020630+0000", 2024-09-13T11:22:48.265 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.001014439 2024-09-13T11:22:48.265 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.265 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.266 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:22:48.266 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.021369+0000", 2024-09-13T11:22:48.266 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00073971400000000004 2024-09-13T11:22:48.266 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.266 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.266 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:22:48.266 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.021413+0000", 2024-09-13T11:22:48.266 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.3353999999999999e-05 2024-09-13T11:22:48.266 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.266 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.266 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:22:48.266 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.021450+0000", 2024-09-13T11:22:48.266 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.7667999999999999e-05 2024-09-13T11:22:48.266 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:22:48.267 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:22:48.267 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:22:48.267 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.267 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.267 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4665.0:47 3.7 e32/30 3:e22c870b:::benchmark_data_smithi177_31116_object46:head v 32'4, mlcod=32'2)", 2024-09-13T11:22:48.267 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:22:27.020290+0000", 2024-09-13T11:22:48.267 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 21.211722507000001, 2024-09-13T11:22:48.267 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00097084700000000005, 2024-09-13T11:22:48.267 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:22:48.267 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:22:48.267 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:48.267 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:22:48.267 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.267 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:22:48.268 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.020290+0000", 2024-09-13T11:22:48.268 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:22:48.268 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.268 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.268 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:22:48.268 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.020290+0000", 2024-09-13T11:22:48.268 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:22:48.268 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.268 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.268 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:22:48.268 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.020291+0000", 2024-09-13T11:22:48.268 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.728e-06 2024-09-13T11:22:48.268 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.269 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.269 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:22:48.269 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.020356+0000", 2024-09-13T11:22:48.269 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.5105999999999996e-05 2024-09-13T11:22:48.269 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.269 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.269 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:22:48.269 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.020385+0000", 2024-09-13T11:22:48.269 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.8393000000000001e-05 2024-09-13T11:22:48.272 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.272 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.272 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:22:48.272 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.020391+0000", 2024-09-13T11:22:48.272 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.4309999999999999e-06 2024-09-13T11:22:48.272 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.272 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.272 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:22:48.272 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.020480+0000", 2024-09-13T11:22:48.273 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.9163e-05 2024-09-13T11:22:48.273 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.273 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.273 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:22:48.273 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.020508+0000", 2024-09-13T11:22:48.273 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.7194000000000001e-05 2024-09-13T11:22:48.273 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.273 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.273 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:22:48.273 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.021217+0000", 2024-09-13T11:22:48.273 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00070979099999999998 2024-09-13T11:22:48.273 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.273 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.273 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:22:48.274 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.021260+0000", 2024-09-13T11:22:48.274 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.3040999999999997e-05 2024-09-13T11:22:48.274 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:22:48.274 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:22:48.274 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:22:48.274 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.274 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.274 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop_reply(client.4665.0:46 3.e e32/30 ondisk, result = 0)", 2024-09-13T11:22:48.274 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:22:27.021265+0000", 2024-09-13T11:22:48.274 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 21.210746983, 2024-09-13T11:22:48.274 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00019617699999999999, 2024-09-13T11:22:48.274 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:22:48.275 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:22:48.275 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:22:48.275 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:22:48.275 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.275 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:22:48.275 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.021265+0000", 2024-09-13T11:22:48.275 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:22:48.275 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.275 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.275 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:22:48.275 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.021265+0000", 2024-09-13T11:22:48.275 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:22:48.275 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.276 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.276 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:22:48.276 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.021266+0000", 2024-09-13T11:22:48.276 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.232e-06 2024-09-13T11:22:48.276 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.276 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.276 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:22:48.276 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.021271+0000", 2024-09-13T11:22:48.276 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.6040000000000003e-06 2024-09-13T11:22:48.276 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.276 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.276 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:22:48.276 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.021308+0000", 2024-09-13T11:22:48.277 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.7206000000000003e-05 2024-09-13T11:22:48.277 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.277 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.277 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:22:48.277 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.021314+0000", 2024-09-13T11:22:48.277 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.8860000000000001e-06 2024-09-13T11:22:48.277 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.277 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.277 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:22:48.277 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.021358+0000", 2024-09-13T11:22:48.277 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.3708000000000002e-05 2024-09-13T11:22:48.277 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.277 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.278 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:22:48.278 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.021364+0000", 2024-09-13T11:22:48.278 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.1149999999999999e-06 2024-09-13T11:22:48.278 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.278 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.278 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:22:48.278 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.021461+0000", 2024-09-13T11:22:48.278 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.7425999999999999e-05 2024-09-13T11:22:48.278 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:22:48.278 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:22:48.278 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:22:48.278 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.278 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.279 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4665.0:51 3.7 e32/30 3:e2e297cd:::benchmark_data_smithi177_31116_object50:head v 32'5, mlcod=32'3)", 2024-09-13T11:22:48.279 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:22:27.023001+0000", 2024-09-13T11:22:48.279 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 21.209011, 2024-09-13T11:22:48.279 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00094912600000000001, 2024-09-13T11:22:48.279 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:22:48.279 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:22:48.279 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:48.279 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:22:48.279 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.279 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:22:48.279 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.023001+0000", 2024-09-13T11:22:48.279 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:22:48.279 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.279 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.280 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:22:48.280 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.023001+0000", 2024-09-13T11:22:48.280 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:22:48.280 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.280 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.280 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:22:48.280 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.023003+0000", 2024-09-13T11:22:48.280 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.685e-06 2024-09-13T11:22:48.280 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.280 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.280 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:22:48.280 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.023056+0000", 2024-09-13T11:22:48.280 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.3319999999999998e-05 2024-09-13T11:22:48.281 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.281 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.281 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:22:48.281 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.023092+0000", 2024-09-13T11:22:48.281 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.5716000000000002e-05 2024-09-13T11:22:48.281 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.281 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.281 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:22:48.281 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.023103+0000", 2024-09-13T11:22:48.281 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.1389000000000001e-05 2024-09-13T11:22:48.281 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.281 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.281 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:22:48.281 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.023176+0000", 2024-09-13T11:22:48.282 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.2488000000000001e-05 2024-09-13T11:22:48.282 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.282 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.282 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:22:48.282 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.023194+0000", 2024-09-13T11:22:48.282 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.8800999999999998e-05 2024-09-13T11:22:48.282 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.282 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.282 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:22:48.282 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.023908+0000", 2024-09-13T11:22:48.282 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00071373900000000002 2024-09-13T11:22:48.282 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.282 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.282 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:22:48.283 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.023950+0000", 2024-09-13T11:22:48.283 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.1987999999999998e-05 2024-09-13T11:22:48.283 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:22:48.283 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:22:48.283 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:22:48.283 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.283 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.283 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4665.0:53 3.7 e32/30 3:e998c81b:::benchmark_data_smithi177_31116_object52:head v 32'6, mlcod=32'3)", 2024-09-13T11:22:48.283 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:22:27.024164+0000", 2024-09-13T11:22:48.283 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 21.207848065, 2024-09-13T11:22:48.283 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0010018270000000001, 2024-09-13T11:22:48.283 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:22:48.283 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:22:48.284 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:48.284 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:22:48.284 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.284 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:22:48.284 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.024164+0000", 2024-09-13T11:22:48.284 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:22:48.284 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.284 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.284 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:22:48.284 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.024164+0000", 2024-09-13T11:22:48.284 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:22:48.284 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.284 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.284 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:22:48.285 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.024165+0000", 2024-09-13T11:22:48.285 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.449e-06 2024-09-13T11:22:48.285 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.285 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.285 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:22:48.285 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.024234+0000", 2024-09-13T11:22:48.285 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.8758999999999997e-05 2024-09-13T11:22:48.285 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.285 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.285 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:22:48.285 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.024265+0000", 2024-09-13T11:22:48.285 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.0744999999999998e-05 2024-09-13T11:22:48.285 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.285 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.286 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:22:48.286 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.024274+0000", 2024-09-13T11:22:48.286 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.4120000000000007e-06 2024-09-13T11:22:48.286 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.286 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.286 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:22:48.286 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.024377+0000", 2024-09-13T11:22:48.286 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00010253 2024-09-13T11:22:48.286 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.286 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.286 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:22:48.286 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.024395+0000", 2024-09-13T11:22:48.286 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.7788999999999999e-05 2024-09-13T11:22:48.286 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.287 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.287 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:22:48.287 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.025123+0000", 2024-09-13T11:22:48.287 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00072863400000000003 2024-09-13T11:22:48.287 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.287 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.287 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:22:48.287 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.025166+0000", 2024-09-13T11:22:48.287 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.2509000000000002e-05 2024-09-13T11:22:48.287 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:22:48.287 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:22:48.287 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:22:48.287 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.287 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.288 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4665.0:57 3.7 e32/30 3:e5e6d1ae:::benchmark_data_smithi177_31116_object56:head v 32'7, mlcod=32'5)", 2024-09-13T11:22:48.288 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:22:27.027031+0000", 2024-09-13T11:22:48.288 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 21.204980857999999, 2024-09-13T11:22:48.288 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0010161809999999999, 2024-09-13T11:22:48.288 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:22:48.288 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:22:48.288 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:48.288 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:22:48.288 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.288 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:22:48.288 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.027031+0000", 2024-09-13T11:22:48.288 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:22:48.288 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.288 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.289 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:22:48.289 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.027031+0000", 2024-09-13T11:22:48.289 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:22:48.289 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.289 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.289 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:22:48.289 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.027033+0000", 2024-09-13T11:22:48.289 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.576e-06 2024-09-13T11:22:48.289 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.289 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.289 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:22:48.289 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.027069+0000", 2024-09-13T11:22:48.289 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.6185e-05 2024-09-13T11:22:48.289 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.290 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.290 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:22:48.290 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.027104+0000", 2024-09-13T11:22:48.290 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.4708e-05 2024-09-13T11:22:48.290 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.290 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.290 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:22:48.290 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.027110+0000", 2024-09-13T11:22:48.290 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.596e-06 2024-09-13T11:22:48.290 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.290 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.290 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:22:48.290 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.027171+0000", 2024-09-13T11:22:48.290 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.0851999999999999e-05 2024-09-13T11:22:48.290 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.291 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.291 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:22:48.291 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.027195+0000", 2024-09-13T11:22:48.291 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.4341e-05 2024-09-13T11:22:48.291 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.291 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.291 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:22:48.291 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.028005+0000", 2024-09-13T11:22:48.291 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00080958699999999996 2024-09-13T11:22:48.291 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.291 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.291 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:22:48.291 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.028047+0000", 2024-09-13T11:22:48.291 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.2336000000000003e-05 2024-09-13T11:22:48.292 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:22:48.292 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:22:48.292 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:22:48.292 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.292 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.292 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:83 3.e 3:79afbc3f:::benchmark_data_smithi177_31116_object82:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e33)", 2024-09-13T11:22:48.292 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:22:27.958896+0000", 2024-09-13T11:22:48.292 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 20.273116199, 2024-09-13T11:22:48.292 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0048065210000000002, 2024-09-13T11:22:48.292 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:22:48.292 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:22:48.292 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:48.292 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:22:48.292 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:22:48.293 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:22:48.293 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 83 2024-09-13T11:22:48.293 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.293 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:22:48.293 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.293 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:22:48.293 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.958896+0000", 2024-09-13T11:22:48.293 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:22:48.293 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.293 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.293 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:22:48.293 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.958896+0000", 2024-09-13T11:22:48.293 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:22:48.293 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.294 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.294 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:22:48.294 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.958900+0000", 2024-09-13T11:22:48.294 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.437e-06 2024-09-13T11:22:48.294 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.294 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.294 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:22:48.294 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.959128+0000", 2024-09-13T11:22:48.294 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00022734900000000001 2024-09-13T11:22:48.294 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.294 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.294 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:22:48.294 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.959146+0000", 2024-09-13T11:22:48.294 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.7846000000000002e-05 2024-09-13T11:22:48.294 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.295 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.295 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:22:48.295 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.959159+0000", 2024-09-13T11:22:48.295 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.326e-05 2024-09-13T11:22:48.295 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.295 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.295 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:22:48.295 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.959274+0000", 2024-09-13T11:22:48.295 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000114935 2024-09-13T11:22:48.295 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.295 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.295 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op must wait for map", 2024-09-13T11:22:48.295 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.959280+0000", 2024-09-13T11:22:48.295 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.4919999999999998e-06 2024-09-13T11:22:48.296 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.296 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.296 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:22:48.296 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.961142+0000", 2024-09-13T11:22:48.296 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0018618689999999999 2024-09-13T11:22:48.296 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.296 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.296 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:22:48.296 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.961378+0000", 2024-09-13T11:22:48.296 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000236393 2024-09-13T11:22:48.296 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.296 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.296 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 14", 2024-09-13T11:22:48.297 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.961628+0000", 2024-09-13T11:22:48.297 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00024923300000000002 2024-09-13T11:22:48.297 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.297 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.297 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:22:48.297 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.963088+0000", 2024-09-13T11:22:48.297 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.001460647 2024-09-13T11:22:48.297 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.297 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.297 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:22:48.297 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.963546+0000", 2024-09-13T11:22:48.297 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00045760000000000001 2024-09-13T11:22:48.297 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.297 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.298 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:22:48.298 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.963638+0000", 2024-09-13T11:22:48.298 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.2263999999999996e-05 2024-09-13T11:22:48.298 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.298 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.298 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:22:48.298 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.963702+0000", 2024-09-13T11:22:48.298 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.4196000000000007e-05 2024-09-13T11:22:48.298 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:22:48.298 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:22:48.298 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:22:48.298 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.298 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.298 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop_reply(client.4665.0:83 3.e e33/30 ondisk, result = 0)", 2024-09-13T11:22:48.299 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:22:27.963387+0000", 2024-09-13T11:22:48.299 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 20.268625177000001, 2024-09-13T11:22:48.299 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00032713600000000001, 2024-09-13T11:22:48.299 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:22:48.299 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:22:48.299 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:22:48.299 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:22:48.299 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.299 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:22:48.299 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.963387+0000", 2024-09-13T11:22:48.299 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:22:48.299 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.299 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.299 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:22:48.300 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.963387+0000", 2024-09-13T11:22:48.300 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:22:48.300 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.300 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.300 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:22:48.300 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.963389+0000", 2024-09-13T11:22:48.300 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.8339999999999999e-06 2024-09-13T11:22:48.300 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.300 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.300 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:22:48.300 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.963399+0000", 2024-09-13T11:22:48.300 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.8830000000000001e-06 2024-09-13T11:22:48.300 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.300 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.301 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:22:48.301 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.963436+0000", 2024-09-13T11:22:48.301 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.7536000000000001e-05 2024-09-13T11:22:48.301 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.301 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.301 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:22:48.301 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.963447+0000", 2024-09-13T11:22:48.301 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.1093e-05 2024-09-13T11:22:48.301 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.301 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.301 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:22:48.301 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.963524+0000", 2024-09-13T11:22:48.301 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.6433000000000005e-05 2024-09-13T11:22:48.301 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.302 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.302 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:22:48.302 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.963535+0000", 2024-09-13T11:22:48.302 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.1506e-05 2024-09-13T11:22:48.302 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.302 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.302 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:22:48.302 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.963714+0000", 2024-09-13T11:22:48.302 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000178851 2024-09-13T11:22:48.302 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:22:48.302 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:22:48.302 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:22:48.302 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.302 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.303 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:86 3.e 3:763be569:::benchmark_data_smithi177_31116_object85:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e33)", 2024-09-13T11:22:48.303 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:22:28.334113+0000", 2024-09-13T11:22:48.303 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 19.897899097, 2024-09-13T11:22:48.303 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0040445070000000001, 2024-09-13T11:22:48.303 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:22:48.303 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:22:48.303 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:48.303 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:22:48.303 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:22:48.303 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:22:48.303 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 86 2024-09-13T11:22:48.303 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.303 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:22:48.303 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.304 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:22:48.304 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:28.334113+0000", 2024-09-13T11:22:48.304 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:22:48.304 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.304 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.304 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:22:48.304 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:28.334113+0000", 2024-09-13T11:22:48.304 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:22:48.304 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.304 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.304 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:22:48.304 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:28.334117+0000", 2024-09-13T11:22:48.304 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.2359999999999998e-06 2024-09-13T11:22:48.304 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.305 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.305 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:22:48.305 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:28.334253+0000", 2024-09-13T11:22:48.305 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00013581500000000001 2024-09-13T11:22:48.305 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.305 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.305 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:22:48.305 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:28.334284+0000", 2024-09-13T11:22:48.305 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.1253e-05 2024-09-13T11:22:48.305 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.305 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.305 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:22:48.305 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:28.334306+0000", 2024-09-13T11:22:48.305 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.1976000000000001e-05 2024-09-13T11:22:48.306 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.306 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.306 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:22:48.306 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:28.334489+0000", 2024-09-13T11:22:48.306 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00018322400000000001 2024-09-13T11:22:48.306 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.306 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.306 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:22:48.306 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:28.334702+0000", 2024-09-13T11:22:48.306 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00021245699999999999 2024-09-13T11:22:48.306 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.306 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.306 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 14", 2024-09-13T11:22:48.306 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:28.334926+0000", 2024-09-13T11:22:48.307 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00022375699999999999 2024-09-13T11:22:48.307 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.307 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.307 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:22:48.307 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:28.336643+0000", 2024-09-13T11:22:48.307 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.001717784 2024-09-13T11:22:48.307 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.307 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.307 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:22:48.307 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:28.337999+0000", 2024-09-13T11:22:48.307 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0013551870000000001 2024-09-13T11:22:48.307 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.307 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.307 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:22:48.308 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:28.338089+0000", 2024-09-13T11:22:48.308 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.0062000000000006e-05 2024-09-13T11:22:48.308 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.308 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.308 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:22:48.308 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:28.338157+0000", 2024-09-13T11:22:48.308 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.8756000000000004e-05 2024-09-13T11:22:48.308 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:22:48.308 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:22:48.308 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:22:48.308 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.308 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.308 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop_reply(client.4665.0:86 3.e e33/30 ondisk, result = 0)", 2024-09-13T11:22:48.308 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:22:28.337787+0000", 2024-09-13T11:22:48.309 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 19.894224864000002, 2024-09-13T11:22:48.309 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00038257100000000001, 2024-09-13T11:22:48.309 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:22:48.309 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:22:48.309 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:22:48.309 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:22:48.309 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.309 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:22:48.309 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:28.337787+0000", 2024-09-13T11:22:48.309 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:22:48.309 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.309 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.309 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:22:48.309 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:28.337787+0000", 2024-09-13T11:22:48.310 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:22:48.310 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.310 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.310 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:22:48.310 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:28.337796+0000", 2024-09-13T11:22:48.310 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.8189999999999998e-06 2024-09-13T11:22:48.310 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.310 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.310 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:22:48.310 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:28.337804+0000", 2024-09-13T11:22:48.310 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.0330000000000008e-06 2024-09-13T11:22:48.310 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.310 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.310 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:22:48.311 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:28.337864+0000", 2024-09-13T11:22:48.311 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.9500000000000003e-05 2024-09-13T11:22:48.311 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.311 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.311 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:22:48.311 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:28.337884+0000", 2024-09-13T11:22:48.311 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.0026000000000001e-05 2024-09-13T11:22:48.311 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.311 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.311 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:22:48.311 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:28.337975+0000", 2024-09-13T11:22:48.311 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.1051999999999995e-05 2024-09-13T11:22:48.311 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.311 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.311 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:22:48.312 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:28.337990+0000", 2024-09-13T11:22:48.312 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.4936999999999999e-05 2024-09-13T11:22:48.312 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.312 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.312 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:22:48.312 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:28.338170+0000", 2024-09-13T11:22:48.312 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000180204 2024-09-13T11:22:48.312 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:22:48.312 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:22:48.312 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:22:48.312 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.312 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.312 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4665.0:180 3.7 e41/30 3:e7c4f4d4:::benchmark_data_smithi177_31116_object179:head v 41'13, mlcod=41'11)", 2024-09-13T11:22:48.313 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:22:48.075961+0000", 2024-09-13T11:22:48.313 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 0.15605066000000001, 2024-09-13T11:22:48.313 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.001921617, 2024-09-13T11:22:48.313 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:22:48.313 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:22:48.313 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:48.313 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:22:48.313 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.313 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:22:48.313 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:48.075961+0000", 2024-09-13T11:22:48.313 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:22:48.313 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.313 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.313 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:22:48.314 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:48.075961+0000", 2024-09-13T11:22:48.314 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:22:48.314 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.314 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.314 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:22:48.314 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:48.075964+0000", 2024-09-13T11:22:48.314 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.9409999999999999e-06 2024-09-13T11:22:48.314 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.314 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.314 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:22:48.314 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:48.076147+0000", 2024-09-13T11:22:48.314 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00018252700000000001 2024-09-13T11:22:48.314 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.314 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.315 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:22:48.315 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:48.076181+0000", 2024-09-13T11:22:48.315 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.3781000000000001e-05 2024-09-13T11:22:48.315 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.315 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.315 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:22:48.315 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:48.076191+0000", 2024-09-13T11:22:48.315 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.8409999999999995e-06 2024-09-13T11:22:48.315 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.315 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.315 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:22:48.315 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:48.076332+0000", 2024-09-13T11:22:48.315 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00014135800000000001 2024-09-13T11:22:48.315 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.316 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.316 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:22:48.316 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:48.076372+0000", 2024-09-13T11:22:48.316 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.0377000000000003e-05 2024-09-13T11:22:48.316 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.316 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.316 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:22:48.316 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:48.077800+0000", 2024-09-13T11:22:48.316 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0014278629999999999 2024-09-13T11:22:48.316 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.316 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.316 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:22:48.316 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:48.077883+0000", 2024-09-13T11:22:48.316 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.2929000000000005e-05 2024-09-13T11:22:48.317 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:22:48.317 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:22:48.317 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:22:48.317 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.317 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.317 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:194 3.e 3:795d5d56:::benchmark_data_smithi177_31116_object193:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T11:22:48.317 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:22:48.153969+0000", 2024-09-13T11:22:48.317 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 0.078043363000000004, 2024-09-13T11:22:48.317 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.002881279, 2024-09-13T11:22:48.317 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:22:48.317 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:22:48.317 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:48.317 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:22:48.318 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:22:48.318 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:22:48.318 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 194 2024-09-13T11:22:48.318 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.318 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:22:48.318 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.318 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:22:48.318 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:48.153969+0000", 2024-09-13T11:22:48.318 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:22:48.318 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.318 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.318 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:22:48.318 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:48.153969+0000", 2024-09-13T11:22:48.318 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:22:48.318 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.319 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.319 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:22:48.319 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:48.153975+0000", 2024-09-13T11:22:48.319 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.9699999999999996e-06 2024-09-13T11:22:48.319 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.319 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.319 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:22:48.319 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:48.154149+0000", 2024-09-13T11:22:48.319 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000174098 2024-09-13T11:22:48.319 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.319 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.319 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:22:48.319 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:48.154165+0000", 2024-09-13T11:22:48.319 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.6076999999999999e-05 2024-09-13T11:22:48.320 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.320 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.320 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:22:48.320 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:48.154173+0000", 2024-09-13T11:22:48.320 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.4309999999999994e-06 2024-09-13T11:22:48.320 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.320 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.320 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:22:48.320 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:48.154240+0000", 2024-09-13T11:22:48.320 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.6694999999999998e-05 2024-09-13T11:22:48.320 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.320 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.320 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:22:48.321 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:48.154369+0000", 2024-09-13T11:22:48.321 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00012887099999999999 2024-09-13T11:22:48.321 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.321 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.321 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 14", 2024-09-13T11:22:48.321 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:48.154510+0000", 2024-09-13T11:22:48.321 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000141619 2024-09-13T11:22:48.321 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.321 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.321 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:22:48.321 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:48.155413+0000", 2024-09-13T11:22:48.321 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00090269600000000003 2024-09-13T11:22:48.321 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.321 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.322 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:22:48.322 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:48.156690+0000", 2024-09-13T11:22:48.322 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0012763900000000001 2024-09-13T11:22:48.322 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.322 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.322 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:22:48.322 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:48.156782+0000", 2024-09-13T11:22:48.322 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.2274000000000005e-05 2024-09-13T11:22:48.322 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.322 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.322 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:22:48.322 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:48.156850+0000", 2024-09-13T11:22:48.322 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.8157999999999994e-05 2024-09-13T11:22:48.322 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:22:48.323 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:22:48.323 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:22:48.323 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.323 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.323 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop_reply(client.4665.0:194 3.e e41/30 ondisk, result = 0)", 2024-09-13T11:22:48.323 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:22:48.156556+0000", 2024-09-13T11:22:48.323 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 0.075456141000000004, 2024-09-13T11:22:48.323 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00030466100000000002, 2024-09-13T11:22:48.323 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:22:48.323 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:22:48.323 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:22:48.323 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:22:48.323 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.323 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:22:48.324 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:48.156556+0000", 2024-09-13T11:22:48.324 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:22:48.324 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.324 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.324 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:22:48.324 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:48.156556+0000", 2024-09-13T11:22:48.324 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:22:48.324 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.324 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.324 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:22:48.324 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:48.156558+0000", 2024-09-13T11:22:48.324 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.7099999999999999e-06 2024-09-13T11:22:48.324 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.324 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.325 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:22:48.325 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:48.156563+0000", 2024-09-13T11:22:48.325 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.9010000000000002e-06 2024-09-13T11:22:48.325 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.325 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.325 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:22:48.325 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:48.156595+0000", 2024-09-13T11:22:48.325 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.2335000000000001e-05 2024-09-13T11:22:48.325 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.325 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.325 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:22:48.325 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:48.156602+0000", 2024-09-13T11:22:48.325 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.2649999999999999e-06 2024-09-13T11:22:48.325 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.326 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.326 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:22:48.326 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:48.156668+0000", 2024-09-13T11:22:48.326 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.5474000000000004e-05 2024-09-13T11:22:48.326 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.326 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.326 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:22:48.326 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:48.156681+0000", 2024-09-13T11:22:48.326 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.3332999999999999e-05 2024-09-13T11:22:48.326 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.326 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.326 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:22:48.326 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:48.156861+0000", 2024-09-13T11:22:48.326 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000179643 2024-09-13T11:22:48.327 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:22:48.327 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:22:48.327 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:22:48.327 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.327 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.327 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4665.0:199 3.7 e41/30 3:e08cbd3a:::benchmark_data_smithi177_31116_object198:head v 41'14, mlcod=41'12)", 2024-09-13T11:22:48.327 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:22:48.171153+0000", 2024-09-13T11:22:48.327 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 0.060859516000000002, 2024-09-13T11:22:48.327 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0011978030000000001, 2024-09-13T11:22:48.327 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:22:48.327 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:22:48.327 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:48.327 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:22:48.327 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.328 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:22:48.328 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:48.171153+0000", 2024-09-13T11:22:48.328 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:22:48.328 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.328 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.328 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:22:48.328 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:48.171153+0000", 2024-09-13T11:22:48.328 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:22:48.328 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.328 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.328 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:22:48.328 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:48.171156+0000", 2024-09-13T11:22:48.328 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.1470000000000001e-06 2024-09-13T11:22:48.328 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.329 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.329 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:22:48.329 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:48.171206+0000", 2024-09-13T11:22:48.329 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.0300999999999998e-05 2024-09-13T11:22:48.329 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.329 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.329 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:22:48.329 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:48.171250+0000", 2024-09-13T11:22:48.329 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.3778e-05 2024-09-13T11:22:48.329 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.329 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.329 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:22:48.329 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:48.171259+0000", 2024-09-13T11:22:48.329 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.4339999999999998e-06 2024-09-13T11:22:48.330 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.330 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.330 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:22:48.330 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:48.171333+0000", 2024-09-13T11:22:48.330 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.3765999999999998e-05 2024-09-13T11:22:48.330 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.330 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.330 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:22:48.330 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:48.171355+0000", 2024-09-13T11:22:48.330 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.2413e-05 2024-09-13T11:22:48.330 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.330 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.330 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:22:48.330 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:48.172297+0000", 2024-09-13T11:22:48.331 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00094127000000000002 2024-09-13T11:22:48.331 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.331 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.331 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:22:48.331 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:48.172350+0000", 2024-09-13T11:22:48.331 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.3693999999999998e-05 2024-09-13T11:22:48.331 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:22:48.331 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:22:48.331 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:22:48.331 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:22:48.331 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:22:48.331 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-13T11:22:48.332 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-13T11:22:48.333 INFO:tasks.ceph.osd.5.smithi092.stderr:2024-09-13T11:22:48.235+0000 7f6f0b705640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26346) UID: 0 2024-09-13T11:22:48.333 INFO:tasks.ceph.osd.13.smithi092.stderr:2024-09-13T11:22:48.303+0000 7f3e4da13640 -1 osd.13 41 handle_osd_map injecting map crc failure 2024-09-13T11:22:48.334 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_ops_in_flight 2024-09-13T11:22:48.335 INFO:teuthology.orchestra.run.smithi001.stderr:reweighted osd.2 to 1 (10000) 2024-09-13T11:22:48.335 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 3 1 2024-09-13T11:22:48.336 INFO:tasks.ceph.osd.5.smithi092.stderr:2024-09-13T11:22:48.335+0000 7f6f0b705640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26346) UID: 0 2024-09-13T11:22:48.437 INFO:tasks.ceph.osd.7.smithi177.stderr:2024-09-13T11:22:48.436+0000 7f897b51d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26575) UID: 0 2024-09-13T11:22:48.501 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-13T11:22:48.501 INFO:teuthology.orchestra.run.smithi001.stdout: "ops": [], 2024-09-13T11:22:48.501 INFO:teuthology.orchestra.run.smithi001.stdout: "num_ops": 0 2024-09-13T11:22:48.501 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-13T11:22:48.519 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_blocked_ops 2024-09-13T11:22:48.568 INFO:tasks.ceph.osd.4.smithi001.stderr:2024-09-13T11:22:48.564+0000 7ff8e0218640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26310) UID: 0 2024-09-13T11:22:48.632 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-13T11:22:48.632 INFO:teuthology.orchestra.run.smithi001.stdout: "ops": [], 2024-09-13T11:22:48.632 INFO:teuthology.orchestra.run.smithi001.stdout: "complaint_time": 30, 2024-09-13T11:22:48.632 INFO:teuthology.orchestra.run.smithi001.stdout: "num_blocked_ops": 0 2024-09-13T11:22:48.632 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-13T11:22:48.639 INFO:tasks.ceph.osd.2.smithi162.stderr:2024-09-13T11:22:48.635+0000 7fe92b35a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26313) UID: 0 2024-09-13T11:22:48.645 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_historic_ops 2024-09-13T11:22:48.740 INFO:tasks.ceph.osd.3.smithi177.stderr:2024-09-13T11:22:48.740+0000 7f10adba4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26574) UID: 0 2024-09-13T11:22:48.800 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-13T11:22:48.801 INFO:teuthology.orchestra.run.smithi001.stdout: "size": 20, 2024-09-13T11:22:48.801 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 600, 2024-09-13T11:22:48.801 INFO:teuthology.orchestra.run.smithi001.stdout: "ops": [ 2024-09-13T11:22:48.801 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.801 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:1 3.0 3:0f48e989:::benchmark_data_smithi177_31116_object0:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-09-13T11:22:48.801 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:22:27.000006+0000", 2024-09-13T11:22:48.801 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 21.795490814000001, 2024-09-13T11:22:48.801 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0051292489999999998, 2024-09-13T11:22:48.801 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:22:48.801 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:22:48.801 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:48.802 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:22:48.802 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:22:48.802 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:22:48.802 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 1 2024-09-13T11:22:48.802 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.802 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:22:48.802 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.802 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:22:48.802 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.000006+0000", 2024-09-13T11:22:48.802 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:22:48.802 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.802 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.802 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:22:48.803 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.000006+0000", 2024-09-13T11:22:48.803 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:22:48.803 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.803 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.803 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:22:48.803 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.000017+0000", 2024-09-13T11:22:48.803 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0613e-05 2024-09-13T11:22:48.803 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.803 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.803 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:22:48.803 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.000246+0000", 2024-09-13T11:22:48.803 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000229313 2024-09-13T11:22:48.803 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.803 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.804 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:22:48.804 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.000278+0000", 2024-09-13T11:22:48.804 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.1933000000000001e-05 2024-09-13T11:22:48.804 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.804 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.804 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:22:48.804 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.000313+0000", 2024-09-13T11:22:48.804 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.4548000000000002e-05 2024-09-13T11:22:48.804 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.804 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.804 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:22:48.804 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.000401+0000", 2024-09-13T11:22:48.804 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.7800000000000006e-05 2024-09-13T11:22:48.805 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.805 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.805 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op must wait for map", 2024-09-13T11:22:48.805 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.000407+0000", 2024-09-13T11:22:48.805 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.0519999999999998e-06 2024-09-13T11:22:48.805 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.805 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.805 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:22:48.805 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.002663+0000", 2024-09-13T11:22:48.805 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0022562720000000001 2024-09-13T11:22:48.805 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.805 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.805 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:22:48.805 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.002862+0000", 2024-09-13T11:22:48.806 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00019919100000000001 2024-09-13T11:22:48.806 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.806 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.806 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 15", 2024-09-13T11:22:48.806 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.003014+0000", 2024-09-13T11:22:48.806 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00015144299999999999 2024-09-13T11:22:48.806 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.806 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.806 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:22:48.806 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.004754+0000", 2024-09-13T11:22:48.806 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.001740103 2024-09-13T11:22:48.806 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.806 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.806 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:22:48.807 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.005060+0000", 2024-09-13T11:22:48.807 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00030597799999999999 2024-09-13T11:22:48.807 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.807 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.807 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:22:48.807 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.005101+0000", 2024-09-13T11:22:48.807 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.1618e-05 2024-09-13T11:22:48.807 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.807 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.807 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:22:48.807 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.005136+0000", 2024-09-13T11:22:48.808 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.4384999999999997e-05 2024-09-13T11:22:48.808 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:22:48.808 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:22:48.808 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:22:48.808 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.808 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.808 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:3 3.4 3.125e09e4 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-09-13T11:22:48.808 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:22:27.000344+0000", 2024-09-13T11:22:48.808 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 21.795153501000001, 2024-09-13T11:22:48.808 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0022646469999999998, 2024-09-13T11:22:48.808 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:22:48.809 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:22:48.809 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-13T11:22:48.809 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:22:48.809 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:22:48.809 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:22:48.809 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 3 2024-09-13T11:22:48.809 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.809 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:22:48.809 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.809 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:22:48.809 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.000344+0000", 2024-09-13T11:22:48.809 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:22:48.809 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.809 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.810 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:22:48.810 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.000344+0000", 2024-09-13T11:22:48.810 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:22:48.810 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.810 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.810 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:22:48.810 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.000345+0000", 2024-09-13T11:22:48.810 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.043e-06 2024-09-13T11:22:48.810 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.810 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.811 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:22:48.811 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.000436+0000", 2024-09-13T11:22:48.811 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.0839000000000002e-05 2024-09-13T11:22:48.811 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.811 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.811 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:22:48.811 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.000455+0000", 2024-09-13T11:22:48.811 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.9670999999999999e-05 2024-09-13T11:22:48.811 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.811 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.811 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:22:48.811 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.000463+0000", 2024-09-13T11:22:48.811 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.9319999999999996e-06 2024-09-13T11:22:48.811 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.812 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.812 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:22:48.812 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.000528+0000", 2024-09-13T11:22:48.812 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.4974999999999994e-05 2024-09-13T11:22:48.812 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.812 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.812 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op must wait for map", 2024-09-13T11:22:48.812 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.000532+0000", 2024-09-13T11:22:48.812 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.1080000000000003e-06 2024-09-13T11:22:48.812 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.812 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.812 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:22:48.812 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.002553+0000", 2024-09-13T11:22:48.812 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0020209640000000001 2024-09-13T11:22:48.813 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.813 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.813 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:22:48.813 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.002608+0000", 2024-09-13T11:22:48.813 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.5114999999999997e-05 2024-09-13T11:22:48.813 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:22:48.813 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:22:48.813 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:22:48.813 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.813 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.813 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:12 3.9 3:9dad78b7:::benchmark_data_smithi177_31116_object11:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-09-13T11:22:48.813 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:22:27.000484+0000", 2024-09-13T11:22:48.813 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 21.795013195999999, 2024-09-13T11:22:48.814 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.007402502, 2024-09-13T11:22:48.814 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:22:48.814 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:22:48.814 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:48.814 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:22:48.814 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:22:48.814 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:22:48.814 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 12 2024-09-13T11:22:48.814 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.814 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:22:48.814 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.814 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:22:48.814 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.000484+0000", 2024-09-13T11:22:48.814 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:22:48.815 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.815 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.815 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:22:48.815 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.000484+0000", 2024-09-13T11:22:48.815 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:22:48.815 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.815 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.815 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:22:48.815 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.000485+0000", 2024-09-13T11:22:48.815 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.274e-06 2024-09-13T11:22:48.815 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.815 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.815 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:22:48.816 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.000632+0000", 2024-09-13T11:22:48.816 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000146756 2024-09-13T11:22:48.816 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.816 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.816 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:22:48.816 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.000701+0000", 2024-09-13T11:22:48.816 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.9111999999999998e-05 2024-09-13T11:22:48.816 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.816 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.816 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:22:48.816 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.000713+0000", 2024-09-13T11:22:48.816 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.1552000000000001e-05 2024-09-13T11:22:48.816 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.816 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.817 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:22:48.817 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.000784+0000", 2024-09-13T11:22:48.817 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.1317999999999997e-05 2024-09-13T11:22:48.817 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.817 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.817 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op must wait for map", 2024-09-13T11:22:48.817 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.000789+0000", 2024-09-13T11:22:48.817 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.147e-06 2024-09-13T11:22:48.817 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.817 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.817 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:22:48.818 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.002573+0000", 2024-09-13T11:22:48.818 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.001783532 2024-09-13T11:22:48.818 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.818 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.818 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:22:48.818 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.002755+0000", 2024-09-13T11:22:48.818 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00018205200000000001 2024-09-13T11:22:48.818 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.818 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.818 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 2", 2024-09-13T11:22:48.818 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.002959+0000", 2024-09-13T11:22:48.818 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000204358 2024-09-13T11:22:48.818 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.818 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.819 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:22:48.819 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.004983+0000", 2024-09-13T11:22:48.819 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0020237570000000002 2024-09-13T11:22:48.819 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.819 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.819 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:22:48.819 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.007718+0000", 2024-09-13T11:22:48.819 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.002734963 2024-09-13T11:22:48.819 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.819 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.819 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:22:48.819 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.007783+0000", 2024-09-13T11:22:48.819 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.4851000000000006e-05 2024-09-13T11:22:48.820 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.820 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.820 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:22:48.820 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.007887+0000", 2024-09-13T11:22:48.820 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00010383 2024-09-13T11:22:48.820 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:22:48.820 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:22:48.820 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:22:48.820 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.820 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.820 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:13 3.9 3:967a7538:::benchmark_data_smithi177_31116_object12:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-09-13T11:22:48.820 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:22:27.000738+0000", 2024-09-13T11:22:48.820 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 21.794759433999999, 2024-09-13T11:22:48.821 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.011803163, 2024-09-13T11:22:48.821 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:22:48.821 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:22:48.821 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:48.821 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:22:48.821 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:22:48.821 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:22:48.821 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 13 2024-09-13T11:22:48.821 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.821 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:22:48.821 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.821 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:22:48.821 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.000738+0000", 2024-09-13T11:22:48.822 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:22:48.822 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.822 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.822 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:22:48.822 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.000738+0000", 2024-09-13T11:22:48.822 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:22:48.822 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.822 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.822 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:22:48.822 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.000739+0000", 2024-09-13T11:22:48.822 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.652e-06 2024-09-13T11:22:48.823 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.823 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.823 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:22:48.823 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.000823+0000", 2024-09-13T11:22:48.823 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.3197999999999999e-05 2024-09-13T11:22:48.823 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.823 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.823 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:22:48.823 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.000836+0000", 2024-09-13T11:22:48.824 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.327e-05 2024-09-13T11:22:48.824 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.824 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.824 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:22:48.824 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.000842+0000", 2024-09-13T11:22:48.824 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.3160000000000002e-06 2024-09-13T11:22:48.824 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.824 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.824 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:22:48.824 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.000881+0000", 2024-09-13T11:22:48.824 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.8726999999999997e-05 2024-09-13T11:22:48.824 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.824 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.825 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting_for_map not empty", 2024-09-13T11:22:48.825 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.000884+0000", 2024-09-13T11:22:48.825 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.2770000000000001e-06 2024-09-13T11:22:48.825 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.825 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.825 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:22:48.825 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.003919+0000", 2024-09-13T11:22:48.825 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0030346750000000001 2024-09-13T11:22:48.825 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.825 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.825 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:22:48.825 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.004109+0000", 2024-09-13T11:22:48.825 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00019011500000000001 2024-09-13T11:22:48.826 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.826 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.826 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 2", 2024-09-13T11:22:48.826 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.004351+0000", 2024-09-13T11:22:48.826 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00024179099999999999 2024-09-13T11:22:48.826 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.826 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.826 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:22:48.826 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.005708+0000", 2024-09-13T11:22:48.826 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0013570539999999999 2024-09-13T11:22:48.826 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.826 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.826 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:22:48.826 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.012438+0000", 2024-09-13T11:22:48.827 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0067301210000000004 2024-09-13T11:22:48.827 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.827 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.827 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:22:48.827 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.012495+0000", 2024-09-13T11:22:48.827 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.7410000000000001e-05 2024-09-13T11:22:48.827 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.827 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.827 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:22:48.827 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.012541+0000", 2024-09-13T11:22:48.827 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.5556999999999998e-05 2024-09-13T11:22:48.827 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:22:48.827 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:22:48.828 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:22:48.828 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.828 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.828 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:14 3.0 3:047c9174:::benchmark_data_smithi177_31116_object13:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-09-13T11:22:48.828 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:22:27.000856+0000", 2024-09-13T11:22:48.828 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 21.7946414, 2024-09-13T11:22:48.828 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0064605549999999998, 2024-09-13T11:22:48.828 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:22:48.828 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:22:48.828 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:48.828 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:22:48.828 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:22:48.828 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:22:48.829 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 14 2024-09-13T11:22:48.829 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.829 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:22:48.829 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.829 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:22:48.829 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.000856+0000", 2024-09-13T11:22:48.829 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:22:48.829 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.829 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.829 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:22:48.829 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.000856+0000", 2024-09-13T11:22:48.829 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:22:48.829 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.829 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.830 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:22:48.830 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.000857+0000", 2024-09-13T11:22:48.830 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2330000000000001e-06 2024-09-13T11:22:48.830 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.830 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.830 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:22:48.830 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.000925+0000", 2024-09-13T11:22:48.830 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.8393999999999996e-05 2024-09-13T11:22:48.830 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.830 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.830 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:22:48.830 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.000935+0000", 2024-09-13T11:22:48.830 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.4069999999999999e-06 2024-09-13T11:22:48.831 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.831 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.831 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:22:48.831 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.000941+0000", 2024-09-13T11:22:48.831 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.3799999999999999e-06 2024-09-13T11:22:48.831 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.831 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.831 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:22:48.831 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.000992+0000", 2024-09-13T11:22:48.831 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.0732999999999997e-05 2024-09-13T11:22:48.831 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.831 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.831 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting_for_map not empty", 2024-09-13T11:22:48.831 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.000995+0000", 2024-09-13T11:22:48.832 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.354e-06 2024-09-13T11:22:48.832 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.832 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.832 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:22:48.832 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.003988+0000", 2024-09-13T11:22:48.832 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0029929119999999999 2024-09-13T11:22:48.832 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.832 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.832 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:22:48.832 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.004102+0000", 2024-09-13T11:22:48.832 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000113501 2024-09-13T11:22:48.832 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.832 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.833 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 15", 2024-09-13T11:22:48.833 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.004227+0000", 2024-09-13T11:22:48.833 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00012559499999999999 2024-09-13T11:22:48.833 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.833 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.833 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:22:48.833 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.005668+0000", 2024-09-13T11:22:48.833 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.001440216 2024-09-13T11:22:48.833 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.833 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.833 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:22:48.833 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.007230+0000", 2024-09-13T11:22:48.833 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0015623200000000001 2024-09-13T11:22:48.833 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.834 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.834 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:22:48.834 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.007278+0000", 2024-09-13T11:22:48.834 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.7930999999999999e-05 2024-09-13T11:22:48.834 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.834 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.834 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:22:48.834 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.007316+0000", 2024-09-13T11:22:48.834 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.8578999999999998e-05 2024-09-13T11:22:48.834 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:22:48.834 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:22:48.834 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:22:48.834 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.834 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.835 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:28 3.0 3:0c1f4d63:::benchmark_data_smithi177_31116_object27:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-09-13T11:22:48.835 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:22:27.011908+0000", 2024-09-13T11:22:48.835 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 21.783589587000002, 2024-09-13T11:22:48.835 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0018107970000000001, 2024-09-13T11:22:48.835 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:22:48.835 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:22:48.835 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:48.835 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:22:48.835 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:22:48.835 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:22:48.835 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 28 2024-09-13T11:22:48.835 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.835 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:22:48.836 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.836 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:22:48.836 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.011908+0000", 2024-09-13T11:22:48.836 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:22:48.836 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.836 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.836 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:22:48.836 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.011908+0000", 2024-09-13T11:22:48.836 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:22:48.836 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.836 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.836 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:22:48.836 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.011910+0000", 2024-09-13T11:22:48.836 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.413e-06 2024-09-13T11:22:48.837 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.837 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.837 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:22:48.837 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.012021+0000", 2024-09-13T11:22:48.837 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000111202 2024-09-13T11:22:48.837 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.837 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.837 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:22:48.837 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.012038+0000", 2024-09-13T11:22:48.837 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.6864e-05 2024-09-13T11:22:48.837 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.837 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.837 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:22:48.838 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.012050+0000", 2024-09-13T11:22:48.838 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.1391e-05 2024-09-13T11:22:48.838 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.838 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.838 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:22:48.838 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.012111+0000", 2024-09-13T11:22:48.838 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.1055000000000004e-05 2024-09-13T11:22:48.838 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.838 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.838 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:22:48.838 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.012226+0000", 2024-09-13T11:22:48.838 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000115658 2024-09-13T11:22:48.838 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.839 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.839 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 15", 2024-09-13T11:22:48.839 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.012376+0000", 2024-09-13T11:22:48.839 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00014960799999999999 2024-09-13T11:22:48.839 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.839 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.839 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:22:48.839 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.013446+0000", 2024-09-13T11:22:48.839 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.001069656 2024-09-13T11:22:48.839 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.839 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.839 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:22:48.839 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.013594+0000", 2024-09-13T11:22:48.840 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00014862799999999999 2024-09-13T11:22:48.840 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.840 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.840 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:22:48.840 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.013671+0000", 2024-09-13T11:22:48.840 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.6577999999999998e-05 2024-09-13T11:22:48.840 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.840 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.840 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:22:48.840 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.013718+0000", 2024-09-13T11:22:48.840 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.7744000000000002e-05 2024-09-13T11:22:48.840 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:22:48.840 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:22:48.841 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:22:48.841 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.841 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.841 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:37 3.9 3:962d44ea:::benchmark_data_smithi177_31116_object36:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-09-13T11:22:48.841 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:22:27.014759+0000", 2024-09-13T11:22:48.841 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 21.780737783999999, 2024-09-13T11:22:48.841 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0038735110000000001, 2024-09-13T11:22:48.841 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:22:48.841 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:22:48.841 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:48.841 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:22:48.841 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:22:48.841 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:22:48.841 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 37 2024-09-13T11:22:48.842 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.842 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:22:48.842 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.842 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:22:48.842 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.014759+0000", 2024-09-13T11:22:48.842 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:22:48.842 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.842 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.842 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:22:48.842 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.014759+0000", 2024-09-13T11:22:48.842 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:22:48.842 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.842 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.842 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:22:48.843 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.014762+0000", 2024-09-13T11:22:48.843 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.441e-06 2024-09-13T11:22:48.843 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.843 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.843 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:22:48.843 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.014809+0000", 2024-09-13T11:22:48.843 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.7237999999999997e-05 2024-09-13T11:22:48.843 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.843 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.843 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:22:48.843 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.014828+0000", 2024-09-13T11:22:48.843 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.9182e-05 2024-09-13T11:22:48.843 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.843 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.844 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:22:48.844 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.014848+0000", 2024-09-13T11:22:48.844 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.9567000000000001e-05 2024-09-13T11:22:48.844 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.844 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.844 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:22:48.844 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.014917+0000", 2024-09-13T11:22:48.844 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.9430000000000004e-05 2024-09-13T11:22:48.844 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.844 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.844 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:22:48.844 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.015019+0000", 2024-09-13T11:22:48.844 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000102007 2024-09-13T11:22:48.845 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.845 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.845 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 2", 2024-09-13T11:22:48.845 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.015140+0000", 2024-09-13T11:22:48.845 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000120732 2024-09-13T11:22:48.845 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.845 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.845 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:22:48.845 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.016113+0000", 2024-09-13T11:22:48.845 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00097306499999999995 2024-09-13T11:22:48.845 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.845 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.845 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:22:48.846 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.018476+0000", 2024-09-13T11:22:48.846 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0023628659999999999 2024-09-13T11:22:48.846 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.846 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.846 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:22:48.846 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.018574+0000", 2024-09-13T11:22:48.846 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.7930000000000006e-05 2024-09-13T11:22:48.846 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.846 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.846 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:22:48.846 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.018633+0000", 2024-09-13T11:22:48.846 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.9052999999999999e-05 2024-09-13T11:22:48.846 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:22:48.846 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:22:48.847 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:22:48.847 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.847 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.847 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:42 3.0 3:09c3a999:::benchmark_data_smithi177_31116_object41:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-09-13T11:22:48.847 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:22:27.018149+0000", 2024-09-13T11:22:48.847 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 21.777348133, 2024-09-13T11:22:48.847 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0019727070000000002, 2024-09-13T11:22:48.847 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:22:48.847 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:22:48.847 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:48.847 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:22:48.847 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:22:48.847 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:22:48.847 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 42 2024-09-13T11:22:48.848 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.848 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:22:48.848 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.848 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:22:48.848 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.018149+0000", 2024-09-13T11:22:48.848 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:22:48.848 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.848 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.848 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:22:48.848 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.018149+0000", 2024-09-13T11:22:48.848 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:22:48.848 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.848 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.849 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:22:48.849 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.018151+0000", 2024-09-13T11:22:48.849 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.508e-06 2024-09-13T11:22:48.849 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.849 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.849 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:22:48.849 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.018213+0000", 2024-09-13T11:22:48.849 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.2137999999999994e-05 2024-09-13T11:22:48.849 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.849 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.849 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:22:48.849 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.018228+0000", 2024-09-13T11:22:48.849 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.5097e-05 2024-09-13T11:22:48.850 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.850 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.850 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:22:48.850 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.018236+0000", 2024-09-13T11:22:48.850 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.7840000000000004e-06 2024-09-13T11:22:48.850 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.850 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.850 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:22:48.850 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.018314+0000", 2024-09-13T11:22:48.850 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.7927000000000001e-05 2024-09-13T11:22:48.850 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.850 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.850 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:22:48.851 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.018519+0000", 2024-09-13T11:22:48.851 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00020553499999999999 2024-09-13T11:22:48.851 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.851 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.851 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 15", 2024-09-13T11:22:48.851 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.018698+0000", 2024-09-13T11:22:48.851 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00017922000000000001 2024-09-13T11:22:48.851 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.851 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.851 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:22:48.851 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.019580+0000", 2024-09-13T11:22:48.851 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00088205599999999999 2024-09-13T11:22:48.851 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.852 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.852 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:22:48.852 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.020016+0000", 2024-09-13T11:22:48.852 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00043519499999999998 2024-09-13T11:22:48.852 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.852 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.852 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:22:48.852 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.020082+0000", 2024-09-13T11:22:48.852 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.6783000000000001e-05 2024-09-13T11:22:48.852 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.852 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.852 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:22:48.852 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.020122+0000", 2024-09-13T11:22:48.852 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.9464e-05 2024-09-13T11:22:48.853 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:22:48.853 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:22:48.853 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:22:48.853 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.853 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.853 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4665.0:55 3.f e32/30 3:f1c4925a:::benchmark_data_smithi177_31116_object54:head v 32'4, mlcod=32'2)", 2024-09-13T11:22:48.853 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:22:27.025662+0000", 2024-09-13T11:22:48.853 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 21.769835313000002, 2024-09-13T11:22:48.853 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0014268659999999999, 2024-09-13T11:22:48.853 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:22:48.853 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:22:48.853 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:48.853 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:22:48.854 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.854 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:22:48.854 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.025662+0000", 2024-09-13T11:22:48.854 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:22:48.854 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.854 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.854 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:22:48.854 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.025662+0000", 2024-09-13T11:22:48.854 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:22:48.854 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.854 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.854 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:22:48.854 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.025664+0000", 2024-09-13T11:22:48.855 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.823e-06 2024-09-13T11:22:48.855 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.855 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.855 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:22:48.855 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.025736+0000", 2024-09-13T11:22:48.855 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.2162000000000005e-05 2024-09-13T11:22:48.855 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.855 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.855 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:22:48.855 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.025780+0000", 2024-09-13T11:22:48.855 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.4161e-05 2024-09-13T11:22:48.855 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.855 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.856 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:22:48.856 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.025788+0000", 2024-09-13T11:22:48.856 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.2239999999999995e-06 2024-09-13T11:22:48.856 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.856 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.856 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:22:48.856 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.025850+0000", 2024-09-13T11:22:48.856 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.1428999999999997e-05 2024-09-13T11:22:48.856 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.856 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.856 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:22:48.856 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.025868+0000", 2024-09-13T11:22:48.856 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.7870999999999999e-05 2024-09-13T11:22:48.856 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.857 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.857 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:22:48.857 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.027017+0000", 2024-09-13T11:22:48.857 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.001149274 2024-09-13T11:22:48.857 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.857 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.857 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:22:48.857 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.027089+0000", 2024-09-13T11:22:48.857 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.1921999999999993e-05 2024-09-13T11:22:48.857 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:22:48.857 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:22:48.857 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:22:48.857 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.857 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.858 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:59 3.9 3:9499e3d6:::benchmark_data_smithi177_31116_object58:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-09-13T11:22:48.858 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:22:27.026411+0000", 2024-09-13T11:22:48.858 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 21.769086442999999, 2024-09-13T11:22:48.858 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0025727300000000001, 2024-09-13T11:22:48.858 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:22:48.858 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:22:48.858 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:48.858 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:22:48.858 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:22:48.858 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:22:48.858 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 59 2024-09-13T11:22:48.859 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.859 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:22:48.859 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.859 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:22:48.859 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.026411+0000", 2024-09-13T11:22:48.859 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:22:48.859 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.859 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.859 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:22:48.860 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.026411+0000", 2024-09-13T11:22:48.860 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:22:48.860 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.860 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.860 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:22:48.860 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.026413+0000", 2024-09-13T11:22:48.860 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.193e-06 2024-09-13T11:22:48.861 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.861 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.861 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:22:48.861 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.026474+0000", 2024-09-13T11:22:48.861 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.1271000000000003e-05 2024-09-13T11:22:48.861 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.861 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.861 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:22:48.861 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.026488+0000", 2024-09-13T11:22:48.861 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.3606e-05 2024-09-13T11:22:48.861 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.862 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.862 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:22:48.862 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.026496+0000", 2024-09-13T11:22:48.862 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.7309999999999993e-06 2024-09-13T11:22:48.862 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.862 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.862 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:22:48.862 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.026568+0000", 2024-09-13T11:22:48.862 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.2094000000000005e-05 2024-09-13T11:22:48.862 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.862 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.863 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:22:48.863 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.026703+0000", 2024-09-13T11:22:48.863 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000134919 2024-09-13T11:22:48.863 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.863 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.863 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 2", 2024-09-13T11:22:48.863 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.026846+0000", 2024-09-13T11:22:48.864 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000143134 2024-09-13T11:22:48.864 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.864 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.864 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:22:48.864 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.027924+0000", 2024-09-13T11:22:48.864 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0010780290000000001 2024-09-13T11:22:48.864 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.864 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.865 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:22:48.865 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.028904+0000", 2024-09-13T11:22:48.865 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00098057599999999993 2024-09-13T11:22:48.865 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.865 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.865 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:22:48.865 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.028948+0000", 2024-09-13T11:22:48.866 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.3541000000000002e-05 2024-09-13T11:22:48.866 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.866 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.866 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:22:48.866 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.028984+0000", 2024-09-13T11:22:48.866 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.5636000000000002e-05 2024-09-13T11:22:48.866 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:22:48.867 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:22:48.867 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:22:48.867 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.867 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.867 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:65 3.0 3:01909ae6:::benchmark_data_smithi177_31116_object64:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-09-13T11:22:48.867 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:22:27.029786+0000", 2024-09-13T11:22:48.867 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 21.765711241999998, 2024-09-13T11:22:48.868 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0020071550000000001, 2024-09-13T11:22:48.868 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:22:48.868 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:22:48.868 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:48.868 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:22:48.868 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:22:48.868 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:22:48.869 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 65 2024-09-13T11:22:48.869 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.869 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:22:48.869 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.869 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:22:48.869 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.029786+0000", 2024-09-13T11:22:48.869 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:22:48.870 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.870 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.870 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:22:48.870 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.029786+0000", 2024-09-13T11:22:48.870 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:22:48.870 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.870 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.871 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:22:48.871 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.029788+0000", 2024-09-13T11:22:48.871 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.8309999999999999e-06 2024-09-13T11:22:48.871 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.871 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.871 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:22:48.871 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.029880+0000", 2024-09-13T11:22:48.872 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.2053999999999997e-05 2024-09-13T11:22:48.872 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.872 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.872 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:22:48.872 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.029896+0000", 2024-09-13T11:22:48.872 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.6082e-05 2024-09-13T11:22:48.872 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.872 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.872 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:22:48.872 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.029904+0000", 2024-09-13T11:22:48.872 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.0839999999999992e-06 2024-09-13T11:22:48.872 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.872 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.873 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:22:48.873 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.029974+0000", 2024-09-13T11:22:48.873 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.0026999999999999e-05 2024-09-13T11:22:48.873 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.873 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.873 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:22:48.873 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.030075+0000", 2024-09-13T11:22:48.873 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000100821 2024-09-13T11:22:48.873 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.873 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.873 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 15", 2024-09-13T11:22:48.873 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.030177+0000", 2024-09-13T11:22:48.873 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000102347 2024-09-13T11:22:48.873 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.874 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.874 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:22:48.874 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.030984+0000", 2024-09-13T11:22:48.874 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00080681600000000002 2024-09-13T11:22:48.874 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.874 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.874 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:22:48.874 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.031647+0000", 2024-09-13T11:22:48.874 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00066247599999999995 2024-09-13T11:22:48.874 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.874 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.874 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:22:48.874 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.031729+0000", 2024-09-13T11:22:48.874 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.2888999999999999e-05 2024-09-13T11:22:48.875 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.875 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.875 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:22:48.875 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.031793+0000", 2024-09-13T11:22:48.875 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.3727999999999997e-05 2024-09-13T11:22:48.875 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:22:48.875 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:22:48.875 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:22:48.875 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.875 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.875 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:76 3.9 3:9e00a8c6:::benchmark_data_smithi177_31116_object75:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-09-13T11:22:48.875 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:22:27.042773+0000", 2024-09-13T11:22:48.875 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 21.752724454999999, 2024-09-13T11:22:48.876 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0027323410000000001, 2024-09-13T11:22:48.876 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:22:48.876 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:22:48.876 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:48.876 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:22:48.876 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:22:48.876 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:22:48.876 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 76 2024-09-13T11:22:48.876 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.876 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:22:48.876 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.876 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:22:48.876 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.042773+0000", 2024-09-13T11:22:48.877 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:22:48.877 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.877 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.877 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:22:48.877 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.042773+0000", 2024-09-13T11:22:48.877 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:22:48.877 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.877 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.877 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:22:48.877 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.042776+0000", 2024-09-13T11:22:48.878 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.8619999999999999e-06 2024-09-13T11:22:48.878 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.878 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.878 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:22:48.878 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.042812+0000", 2024-09-13T11:22:48.878 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.591e-05 2024-09-13T11:22:48.878 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.878 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.878 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:22:48.878 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.042830+0000", 2024-09-13T11:22:48.878 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.7927999999999998e-05 2024-09-13T11:22:48.879 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.879 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.879 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:22:48.879 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.042838+0000", 2024-09-13T11:22:48.879 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.6009999999999995e-06 2024-09-13T11:22:48.879 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.879 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.879 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:22:48.879 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.042898+0000", 2024-09-13T11:22:48.879 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.9851e-05 2024-09-13T11:22:48.879 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.879 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.880 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:22:48.880 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.043013+0000", 2024-09-13T11:22:48.880 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000115497 2024-09-13T11:22:48.880 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.880 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.880 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 2", 2024-09-13T11:22:48.880 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.043131+0000", 2024-09-13T11:22:48.880 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000117657 2024-09-13T11:22:48.880 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.880 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.880 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:22:48.880 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.044018+0000", 2024-09-13T11:22:48.880 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00088696100000000002 2024-09-13T11:22:48.880 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.881 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.881 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:22:48.881 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.045425+0000", 2024-09-13T11:22:48.881 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0014070350000000001 2024-09-13T11:22:48.881 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.881 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.881 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:22:48.881 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.045466+0000", 2024-09-13T11:22:48.881 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.0636999999999997e-05 2024-09-13T11:22:48.881 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.881 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.881 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:22:48.881 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.045505+0000", 2024-09-13T11:22:48.882 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.9402e-05 2024-09-13T11:22:48.882 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:22:48.882 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:22:48.882 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:22:48.882 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.882 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.882 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4665.0:82 3.2 e33/30 3:4daba74e:::benchmark_data_smithi177_31116_object81:head v 33'6, mlcod=32'4)", 2024-09-13T11:22:48.882 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:22:27.950739+0000", 2024-09-13T11:22:48.882 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 20.844758015, 2024-09-13T11:22:48.882 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.002148507, 2024-09-13T11:22:48.882 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:22:48.882 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:22:48.882 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:48.883 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:22:48.883 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.883 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:22:48.883 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.950739+0000", 2024-09-13T11:22:48.883 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:22:48.883 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.883 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.883 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:22:48.883 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.950739+0000", 2024-09-13T11:22:48.883 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:22:48.883 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.883 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.883 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:22:48.883 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.950740+0000", 2024-09-13T11:22:48.884 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.0499999999999999e-07 2024-09-13T11:22:48.884 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.884 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.884 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:22:48.884 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.951138+0000", 2024-09-13T11:22:48.884 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00039828800000000001 2024-09-13T11:22:48.884 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.884 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.884 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:22:48.884 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.951189+0000", 2024-09-13T11:22:48.884 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.0831000000000002e-05 2024-09-13T11:22:48.884 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.884 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.885 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:22:48.885 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.951210+0000", 2024-09-13T11:22:48.885 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.1308e-05 2024-09-13T11:22:48.885 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.885 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.885 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:22:48.885 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.951339+0000", 2024-09-13T11:22:48.885 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000129066 2024-09-13T11:22:48.885 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.885 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.885 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:22:48.885 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.951376+0000", 2024-09-13T11:22:48.885 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.6505999999999999e-05 2024-09-13T11:22:48.886 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.886 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.886 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:22:48.886 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.952788+0000", 2024-09-13T11:22:48.886 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.001412368 2024-09-13T11:22:48.886 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.886 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.886 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:22:48.886 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.952888+0000", 2024-09-13T11:22:48.886 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.9834999999999995e-05 2024-09-13T11:22:48.886 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:22:48.886 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:22:48.887 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:22:48.887 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.887 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.887 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:85 3.9 3:9b4652c0:::benchmark_data_smithi177_31116_object84:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e33)", 2024-09-13T11:22:48.887 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:22:27.974212+0000", 2024-09-13T11:22:48.887 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 20.821284840000001, 2024-09-13T11:22:48.887 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0038541299999999999, 2024-09-13T11:22:48.887 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:22:48.887 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:22:48.887 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:48.887 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:22:48.887 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:22:48.887 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:22:48.888 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 85 2024-09-13T11:22:48.888 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.888 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:22:48.888 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.888 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:22:48.888 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.974212+0000", 2024-09-13T11:22:48.888 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:22:48.888 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.888 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.888 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:22:48.888 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.974212+0000", 2024-09-13T11:22:48.888 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:22:48.888 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.889 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.889 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:22:48.889 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.974217+0000", 2024-09-13T11:22:48.889 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.8169999999999998e-06 2024-09-13T11:22:48.889 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.889 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.889 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:22:48.889 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.974417+0000", 2024-09-13T11:22:48.889 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00019947699999999999 2024-09-13T11:22:48.889 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.889 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.889 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:22:48.889 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.974447+0000", 2024-09-13T11:22:48.889 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.0349999999999999e-05 2024-09-13T11:22:48.891 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.892 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.892 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:22:48.892 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.974461+0000", 2024-09-13T11:22:48.892 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.4182000000000001e-05 2024-09-13T11:22:48.892 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.892 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.892 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:22:48.892 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.974636+0000", 2024-09-13T11:22:48.892 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000174981 2024-09-13T11:22:48.892 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.892 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.892 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:22:48.893 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.975036+0000", 2024-09-13T11:22:48.893 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000399517 2024-09-13T11:22:48.893 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.893 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.893 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 2", 2024-09-13T11:22:48.893 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.975431+0000", 2024-09-13T11:22:48.893 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00039529700000000002 2024-09-13T11:22:48.893 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.893 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.893 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:22:48.893 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.977154+0000", 2024-09-13T11:22:48.893 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0017228160000000001 2024-09-13T11:22:48.893 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.894 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.894 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:22:48.894 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.977791+0000", 2024-09-13T11:22:48.894 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00063686400000000001 2024-09-13T11:22:48.894 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.894 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.894 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:22:48.894 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.977949+0000", 2024-09-13T11:22:48.894 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000158758 2024-09-13T11:22:48.894 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.894 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.894 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:22:48.894 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:27.978067+0000", 2024-09-13T11:22:48.894 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000117071 2024-09-13T11:22:48.895 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:22:48.895 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:22:48.895 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:22:48.895 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.895 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.895 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop_reply(client.4665.0:167 3.9 e41/30 ondisk, result = 0)", 2024-09-13T11:22:48.895 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:22:47.419696+0000", 2024-09-13T11:22:48.895 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 1.375801719, 2024-09-13T11:22:48.895 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.64615778199999996, 2024-09-13T11:22:48.895 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:22:48.895 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:22:48.895 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:22:48.895 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:22:48.896 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.896 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:22:48.896 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:47.419696+0000", 2024-09-13T11:22:48.896 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:22:48.896 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.896 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.896 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:22:48.896 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:47.419696+0000", 2024-09-13T11:22:48.896 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:22:48.896 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.896 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.896 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:22:48.896 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:47.419697+0000", 2024-09-13T11:22:48.897 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.15e-06 2024-09-13T11:22:48.897 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.897 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.897 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:22:48.897 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:47.419700+0000", 2024-09-13T11:22:48.897 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.2449999999999998e-06 2024-09-13T11:22:48.897 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.897 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.897 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:22:48.897 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:48.065637+0000", 2024-09-13T11:22:48.897 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.64593674300000004 2024-09-13T11:22:48.897 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.897 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.898 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:22:48.898 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:48.065648+0000", 2024-09-13T11:22:48.898 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0865e-05 2024-09-13T11:22:48.898 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.898 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.898 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:22:48.898 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:48.065720+0000", 2024-09-13T11:22:48.898 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.2020999999999994e-05 2024-09-13T11:22:48.898 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.898 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.898 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:22:48.898 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:48.065728+0000", 2024-09-13T11:22:48.898 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.7660000000000004e-06 2024-09-13T11:22:48.899 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.899 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.899 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:22:48.899 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:48.065853+0000", 2024-09-13T11:22:48.899 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00012499200000000001 2024-09-13T11:22:48.899 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:22:48.899 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:22:48.899 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:22:48.899 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.899 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.899 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4665.0:181 3.2 e41/30 3:4b2310c9:::benchmark_data_smithi177_31116_object180:head v 41'11, mlcod=41'9)", 2024-09-13T11:22:48.899 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:22:48.080135+0000", 2024-09-13T11:22:48.899 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 0.71536248599999996, 2024-09-13T11:22:48.900 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0020144130000000001, 2024-09-13T11:22:48.900 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:22:48.900 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:22:48.900 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:48.900 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:22:48.900 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.900 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:22:48.900 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:48.080135+0000", 2024-09-13T11:22:48.900 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:22:48.900 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.900 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.900 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:22:48.900 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:48.080135+0000", 2024-09-13T11:22:48.901 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:22:48.901 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.901 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.901 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:22:48.901 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:48.080138+0000", 2024-09-13T11:22:48.901 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.0060000000000001e-06 2024-09-13T11:22:48.901 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.901 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.901 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:22:48.902 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:48.080328+0000", 2024-09-13T11:22:48.902 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00019049099999999999 2024-09-13T11:22:48.902 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.902 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.902 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:22:48.902 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:48.080394+0000", 2024-09-13T11:22:48.902 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.5356000000000002e-05 2024-09-13T11:22:48.902 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.902 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.902 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:22:48.902 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:48.080418+0000", 2024-09-13T11:22:48.902 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.3980000000000001e-05 2024-09-13T11:22:48.902 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.903 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.903 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:22:48.903 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:48.080540+0000", 2024-09-13T11:22:48.903 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000122225 2024-09-13T11:22:48.903 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.903 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.903 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:22:48.903 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:48.080596+0000", 2024-09-13T11:22:48.903 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.6428999999999998e-05 2024-09-13T11:22:48.903 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.903 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.903 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:22:48.903 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:48.082056+0000", 2024-09-13T11:22:48.904 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0014597270000000001 2024-09-13T11:22:48.904 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.904 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.904 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:22:48.904 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:48.082149+0000", 2024-09-13T11:22:48.904 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.3199e-05 2024-09-13T11:22:48.904 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:22:48.904 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:22:48.904 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:22:48.904 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.904 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.904 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:183 3.9 3:94db3450:::benchmark_data_smithi177_31116_object182:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T11:22:48.904 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:22:48.087335+0000", 2024-09-13T11:22:48.904 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 0.70816245099999997, 2024-09-13T11:22:48.905 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.044469518, 2024-09-13T11:22:48.905 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:22:48.905 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:22:48.905 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:48.905 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:22:48.905 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:22:48.905 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:22:48.905 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 183 2024-09-13T11:22:48.905 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.905 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:22:48.905 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.905 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:22:48.905 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:48.087335+0000", 2024-09-13T11:22:48.905 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:22:48.906 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.906 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.906 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:22:48.906 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:48.087335+0000", 2024-09-13T11:22:48.906 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:22:48.906 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.906 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.906 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:22:48.906 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:48.087339+0000", 2024-09-13T11:22:48.906 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.0430000000000001e-06 2024-09-13T11:22:48.906 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.906 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.907 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:22:48.907 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:48.087491+0000", 2024-09-13T11:22:48.907 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00015205500000000001 2024-09-13T11:22:48.907 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.907 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.907 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:22:48.907 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:48.087521+0000", 2024-09-13T11:22:48.907 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.0187000000000001e-05 2024-09-13T11:22:48.907 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.907 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.907 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:22:48.907 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:48.087536+0000", 2024-09-13T11:22:48.907 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.4572999999999999e-05 2024-09-13T11:22:48.907 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.908 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.908 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:22:48.908 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:48.087649+0000", 2024-09-13T11:22:48.908 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000112852 2024-09-13T11:22:48.908 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.908 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.908 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:22:48.908 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:48.087875+0000", 2024-09-13T11:22:48.908 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00022599100000000001 2024-09-13T11:22:48.908 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.908 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.908 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 2", 2024-09-13T11:22:48.909 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:48.088121+0000", 2024-09-13T11:22:48.909 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00024680699999999998 2024-09-13T11:22:48.909 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.909 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.909 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:22:48.909 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:48.089613+0000", 2024-09-13T11:22:48.909 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0014916319999999999 2024-09-13T11:22:48.909 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.909 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.909 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:22:48.909 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:48.131684+0000", 2024-09-13T11:22:48.909 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.042070667999999999 2024-09-13T11:22:48.909 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.909 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.910 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:22:48.910 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:48.131735+0000", 2024-09-13T11:22:48.910 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.1693000000000001e-05 2024-09-13T11:22:48.910 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.910 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.910 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:22:48.910 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:48.131804+0000", 2024-09-13T11:22:48.910 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.9016999999999993e-05 2024-09-13T11:22:48.910 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:22:48.910 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:22:48.910 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:22:48.910 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.910 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.910 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4665.0:184 3.4 e41/40 3:27ff66b2:::benchmark_data_smithi177_31116_object183:head v 41'8, mlcod=41'6)", 2024-09-13T11:22:48.911 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:22:48.092278+0000", 2024-09-13T11:22:48.911 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 0.70321887599999999, 2024-09-13T11:22:48.911 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0019235879999999999, 2024-09-13T11:22:48.911 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:22:48.911 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:22:48.911 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:48.911 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:22:48.911 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.911 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:22:48.911 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:48.092278+0000", 2024-09-13T11:22:48.911 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:22:48.911 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.911 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.912 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:22:48.912 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:48.092278+0000", 2024-09-13T11:22:48.912 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:22:48.912 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.912 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.912 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:22:48.912 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:48.092281+0000", 2024-09-13T11:22:48.912 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.9500000000000001e-06 2024-09-13T11:22:48.912 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.912 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.912 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:22:48.912 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:48.092491+0000", 2024-09-13T11:22:48.912 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00020925800000000001 2024-09-13T11:22:48.912 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.913 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.913 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:22:48.913 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:48.092552+0000", 2024-09-13T11:22:48.913 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.1796000000000003e-05 2024-09-13T11:22:48.913 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.913 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.913 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:22:48.913 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:48.092565+0000", 2024-09-13T11:22:48.913 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2697000000000001e-05 2024-09-13T11:22:48.913 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.913 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.913 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:22:48.913 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:48.092716+0000", 2024-09-13T11:22:48.913 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000150631 2024-09-13T11:22:48.914 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.914 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.914 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:22:48.914 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:48.092756+0000", 2024-09-13T11:22:48.914 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.0163000000000002e-05 2024-09-13T11:22:48.914 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.914 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.914 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:22:48.914 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:48.094108+0000", 2024-09-13T11:22:48.914 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.001352041 2024-09-13T11:22:48.914 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.914 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.914 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:22:48.914 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:48.094202+0000", 2024-09-13T11:22:48.915 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.4052e-05 2024-09-13T11:22:48.915 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:22:48.915 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:22:48.915 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:22:48.915 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.915 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.915 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:193 3.0 3:00c2fe85:::benchmark_data_smithi177_31116_object192:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T11:22:48.915 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:22:48.150849+0000", 2024-09-13T11:22:48.915 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 0.64464786900000004, 2024-09-13T11:22:48.915 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0027438229999999998, 2024-09-13T11:22:48.915 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:22:48.915 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:22:48.915 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:48.915 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:22:48.916 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:22:48.916 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:22:48.916 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 193 2024-09-13T11:22:48.916 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.916 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:22:48.916 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.916 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:22:48.916 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:48.150849+0000", 2024-09-13T11:22:48.916 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:22:48.916 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.916 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.916 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:22:48.916 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:48.150849+0000", 2024-09-13T11:22:48.917 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:22:48.917 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.917 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.917 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:22:48.917 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:48.150852+0000", 2024-09-13T11:22:48.917 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.7130000000000001e-06 2024-09-13T11:22:48.917 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.917 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.917 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:22:48.917 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:48.150886+0000", 2024-09-13T11:22:48.917 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.43e-05 2024-09-13T11:22:48.917 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.917 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.918 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:22:48.918 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:48.150904+0000", 2024-09-13T11:22:48.918 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.7428999999999999e-05 2024-09-13T11:22:48.918 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.918 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.918 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:22:48.918 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:48.150916+0000", 2024-09-13T11:22:48.918 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2179e-05 2024-09-13T11:22:48.918 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.918 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.918 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:22:48.918 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:48.151001+0000", 2024-09-13T11:22:48.918 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.5198999999999995e-05 2024-09-13T11:22:48.918 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.919 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.919 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:22:48.919 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:48.151213+0000", 2024-09-13T11:22:48.919 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00021130699999999999 2024-09-13T11:22:48.919 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.919 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.919 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 15", 2024-09-13T11:22:48.919 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:48.151429+0000", 2024-09-13T11:22:48.919 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00021653300000000001 2024-09-13T11:22:48.919 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.919 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.919 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:22:48.919 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:48.152459+0000", 2024-09-13T11:22:48.920 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.001029615 2024-09-13T11:22:48.920 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.920 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.920 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:22:48.920 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:48.153514+0000", 2024-09-13T11:22:48.920 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.001055418 2024-09-13T11:22:48.920 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.920 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.920 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:22:48.920 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:48.153556+0000", 2024-09-13T11:22:48.920 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.2250999999999999e-05 2024-09-13T11:22:48.920 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.920 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.920 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:22:48.921 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:48.153593+0000", 2024-09-13T11:22:48.921 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.6878999999999997e-05 2024-09-13T11:22:48.921 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:22:48.921 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:22:48.921 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:22:48.921 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.921 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.921 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:195 3.0 3:0c8cd81f:::benchmark_data_smithi177_31116_object194:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T11:22:48.921 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:22:48.157243+0000", 2024-09-13T11:22:48.921 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 0.63825456700000005, 2024-09-13T11:22:48.921 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0026926879999999999, 2024-09-13T11:22:48.921 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:22:48.921 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:22:48.922 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:48.922 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:22:48.922 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:22:48.922 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:22:48.922 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 195 2024-09-13T11:22:48.922 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.922 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:22:48.922 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.922 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:22:48.922 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:48.157243+0000", 2024-09-13T11:22:48.922 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:22:48.922 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.922 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.922 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:22:48.923 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:48.157243+0000", 2024-09-13T11:22:48.923 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:22:48.923 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.923 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.923 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:22:48.923 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:48.157245+0000", 2024-09-13T11:22:48.923 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.9180000000000001e-06 2024-09-13T11:22:48.923 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.923 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.923 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:22:48.923 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:48.157285+0000", 2024-09-13T11:22:48.923 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.0117999999999998e-05 2024-09-13T11:22:48.923 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.924 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.924 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:22:48.924 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:48.157299+0000", 2024-09-13T11:22:48.924 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.4506e-05 2024-09-13T11:22:48.924 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.924 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.924 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:22:48.924 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:48.157307+0000", 2024-09-13T11:22:48.924 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.7340000000000008e-06 2024-09-13T11:22:48.925 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.925 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.925 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:22:48.925 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:48.157373+0000", 2024-09-13T11:22:48.925 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.6266999999999994e-05 2024-09-13T11:22:48.925 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.925 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.925 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:22:48.925 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:48.157486+0000", 2024-09-13T11:22:48.925 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00011305400000000001 2024-09-13T11:22:48.925 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.925 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.925 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 15", 2024-09-13T11:22:48.926 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:48.157604+0000", 2024-09-13T11:22:48.926 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000118119 2024-09-13T11:22:48.926 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.926 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.926 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:22:48.926 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:48.158420+0000", 2024-09-13T11:22:48.926 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00081547900000000003 2024-09-13T11:22:48.926 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.926 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.926 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:22:48.926 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:48.159827+0000", 2024-09-13T11:22:48.926 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.001406944 2024-09-13T11:22:48.927 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.927 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.927 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:22:48.927 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:48.159881+0000", 2024-09-13T11:22:48.927 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.4011000000000002e-05 2024-09-13T11:22:48.927 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:22:48.927 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:22:48.927 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:22:48.927 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:48.159935+0000", 2024-09-13T11:22:48.927 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.4537999999999999e-05 2024-09-13T11:22:48.927 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:22:48.927 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:22:48.927 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:22:48.928 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:22:48.928 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:22:48.928 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-13T11:22:48.929 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-13T11:22:48.930 INFO:tasks.ceph.osd.11.smithi177.stderr:2024-09-13T11:22:48.928+0000 7f24792c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26580) UID: 0 2024-09-13T11:22:48.998 INFO:tasks.radosbench.radosbench.0.smithi177.stdout: 22 16 200 184 0.522653 1.4375 0.00296188 0.68154 2024-09-13T11:22:49.030 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-13T11:22:49.024+0000 7f6a68775640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26307) UID: 0 2024-09-13T11:22:49.063 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-09-13T11:22:49.063 INFO:teuthology.orchestra.run.smithi092.stdout: "ops": [], 2024-09-13T11:22:49.063 INFO:teuthology.orchestra.run.smithi092.stdout: "num_ops": 0 2024-09-13T11:22:49.063 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-09-13T11:22:49.077 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-13T11:22:49.131 INFO:tasks.ceph.osd.12.smithi001.stderr:2024-09-13T11:22:49.124+0000 7f892c6e6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26314) UID: 0 2024-09-13T11:22:49.186 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-09-13T11:22:49.183+0000 7f97b59fa640 -1 osd.6 41 handle_osd_map injecting map crc failure 2024-09-13T11:22:49.209 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-09-13T11:22:49.209 INFO:teuthology.orchestra.run.smithi092.stdout: "ops": [], 2024-09-13T11:22:49.209 INFO:teuthology.orchestra.run.smithi092.stdout: "complaint_time": 30, 2024-09-13T11:22:49.209 INFO:teuthology.orchestra.run.smithi092.stdout: "num_blocked_ops": 0 2024-09-13T11:22:49.209 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-09-13T11:22:49.229 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-13T11:22:49.232 INFO:tasks.ceph.osd.2.smithi162.stderr:2024-09-13T11:22:49.227+0000 7fe92b35a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26313) UID: 0 2024-09-13T11:22:49.304 INFO:teuthology.orchestra.run.smithi001.stderr:noscrub is set 2024-09-13T11:22:49.311 INFO:tasks.ceph.osd.3.smithi177.stderr:2024-09-13T11:22:49.312+0000 7f10a45d5640 -1 osd.3 42 handle_osd_map injecting map crc failure 2024-09-13T11:22:49.311 INFO:teuthology.orchestra.run.smithi001.stderr:reweighted osd.3 to 1 (10000) 2024-09-13T11:22:49.313 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-13T11:22:49.308+0000 7f6a5d1a2640 -1 osd.0 41 handle_osd_map injecting map crc failure 2024-09-13T11:22:49.332 INFO:tasks.ceph.osd.1.smithi092.stderr:2024-09-13T11:22:49.331+0000 7fe7789c5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33718) UID: 0 2024-09-13T11:22:49.342 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 4 1 2024-09-13T11:22:49.381 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-09-13T11:22:49.381 INFO:teuthology.orchestra.run.smithi092.stdout: "size": 20, 2024-09-13T11:22:49.381 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 600, 2024-09-13T11:22:49.381 INFO:teuthology.orchestra.run.smithi092.stdout: "ops": [] 2024-09-13T11:22:49.381 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-09-13T11:22:49.399 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-09-13T11:22:49.434 INFO:tasks.ceph.osd.15.smithi177.stderr:2024-09-13T11:22:49.432+0000 7f9d6ce77640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26578) UID: 0 2024-09-13T11:22:49.535 INFO:tasks.ceph.osd.10.smithi162.stderr:2024-09-13T11:22:49.531+0000 7fb59c9f0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26317) UID: 0 2024-09-13T11:22:49.571 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-09-13T11:22:49.571 INFO:teuthology.orchestra.run.smithi092.stdout: "ops": [], 2024-09-13T11:22:49.571 INFO:teuthology.orchestra.run.smithi092.stdout: "num_ops": 0 2024-09-13T11:22:49.571 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-09-13T11:22:49.587 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-09-13T11:22:49.636 INFO:tasks.ceph.osd.14.smithi162.stderr:2024-09-13T11:22:49.631+0000 7f235c0db640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26319) UID: 0 2024-09-13T11:22:49.719 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-09-13T11:22:49.720 INFO:teuthology.orchestra.run.smithi092.stdout: "ops": [], 2024-09-13T11:22:49.720 INFO:teuthology.orchestra.run.smithi092.stdout: "complaint_time": 30, 2024-09-13T11:22:49.720 INFO:teuthology.orchestra.run.smithi092.stdout: "num_blocked_ops": 0 2024-09-13T11:22:49.720 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-09-13T11:22:49.737 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-09-13T11:22:49.737 INFO:tasks.ceph.osd.3.smithi177.stderr:2024-09-13T11:22:49.736+0000 7f10adba4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26574) UID: 0 2024-09-13T11:22:49.838 INFO:tasks.ceph.osd.11.smithi177.stderr:2024-09-13T11:22:49.840+0000 7f24792c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26580) UID: 0 2024-09-13T11:22:49.870 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-09-13T11:22:49.870 INFO:teuthology.orchestra.run.smithi092.stdout: "size": 20, 2024-09-13T11:22:49.870 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 600, 2024-09-13T11:22:49.870 INFO:teuthology.orchestra.run.smithi092.stdout: "ops": [ 2024-09-13T11:22:49.870 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:49.870 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop(client.4344.0:5 2.2 e27/25 2:5cea7035:::rbd_info:head v 27'1, mlcod=0'0)", 2024-09-13T11:22:49.870 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:15.250889+0000", 2024-09-13T11:22:49.870 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 34.619885500000002, 2024-09-13T11:22:49.870 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.001990627, 2024-09-13T11:22:49.871 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:22:49.871 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:22:49.871 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:49.871 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:22:49.871 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:49.871 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:22:49.871 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:15.250889+0000", 2024-09-13T11:22:49.871 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:22:49.871 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:49.871 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:49.871 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:22:49.871 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:15.250889+0000", 2024-09-13T11:22:49.872 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:22:49.872 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:49.872 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:49.872 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:22:49.872 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:15.250890+0000", 2024-09-13T11:22:49.872 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.153e-06 2024-09-13T11:22:49.872 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:49.872 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:49.872 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:22:49.872 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:15.250907+0000", 2024-09-13T11:22:49.872 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.6821e-05 2024-09-13T11:22:49.872 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:49.872 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:49.873 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:22:49.873 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:15.250991+0000", 2024-09-13T11:22:49.873 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.4241999999999997e-05 2024-09-13T11:22:49.873 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:49.873 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:49.873 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:22:49.873 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:15.251023+0000", 2024-09-13T11:22:49.873 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.1780999999999999e-05 2024-09-13T11:22:49.873 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:49.873 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:49.873 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:22:49.873 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:15.251178+0000", 2024-09-13T11:22:49.873 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000155064 2024-09-13T11:22:49.874 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:49.874 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:49.874 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:22:49.874 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:15.251215+0000", 2024-09-13T11:22:49.874 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.7160000000000003e-05 2024-09-13T11:22:49.874 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:49.874 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:49.874 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:22:49.874 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:15.252759+0000", 2024-09-13T11:22:49.874 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0015440110000000001 2024-09-13T11:22:49.874 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:49.874 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:49.874 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:22:49.874 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:15.252880+0000", 2024-09-13T11:22:49.875 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000120395 2024-09-13T11:22:49.875 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:22:49.875 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:22:49.875 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:22:49.875 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:49.875 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:49.875 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop(client.4344.0:7 2.2 e28/25 2:5cea7035:::rbd_info:head v 28'2, mlcod=0'0)", 2024-09-13T11:22:49.875 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:16.268518+0000", 2024-09-13T11:22:49.875 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 33.602255900000003, 2024-09-13T11:22:49.875 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0017167040000000001, 2024-09-13T11:22:49.875 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:22:49.875 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:22:49.875 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:49.875 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:22:49.876 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:49.876 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:22:49.876 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:16.268518+0000", 2024-09-13T11:22:49.876 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:22:49.876 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:49.876 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:49.876 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:22:49.876 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:16.268518+0000", 2024-09-13T11:22:49.876 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:22:49.876 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:49.876 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:49.876 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:22:49.876 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:16.268519+0000", 2024-09-13T11:22:49.877 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.1299999999999999e-07 2024-09-13T11:22:49.877 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:49.877 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:49.877 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:22:49.877 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:16.268539+0000", 2024-09-13T11:22:49.877 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.0214999999999999e-05 2024-09-13T11:22:49.877 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:49.877 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:49.877 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:22:49.877 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:16.268622+0000", 2024-09-13T11:22:49.877 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.2395e-05 2024-09-13T11:22:49.877 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:49.877 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:49.877 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:22:49.878 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:16.268638+0000", 2024-09-13T11:22:49.878 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.6189e-05 2024-09-13T11:22:49.878 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:49.878 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:49.878 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:22:49.878 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:16.268740+0000", 2024-09-13T11:22:49.878 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00010164 2024-09-13T11:22:49.878 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:49.878 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:49.878 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:22:49.878 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:16.268783+0000", 2024-09-13T11:22:49.878 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.2988999999999998e-05 2024-09-13T11:22:49.878 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:49.878 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:49.879 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:22:49.879 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:16.270121+0000", 2024-09-13T11:22:49.879 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00133829 2024-09-13T11:22:49.879 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:49.879 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:49.879 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:22:49.879 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:16.270235+0000", 2024-09-13T11:22:49.879 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000114273 2024-09-13T11:22:49.879 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:22:49.879 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:22:49.879 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:22:49.879 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:22:49.879 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:22:49.880 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-09-13T11:22:49.888 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_ops_in_flight 2024-09-13T11:22:49.899 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T11:22:49.939 INFO:tasks.ceph.osd.11.smithi177.stderr:2024-09-13T11:22:49.940+0000 7f24792c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26580) UID: 0 2024-09-13T11:22:49.999 INFO:tasks.radosbench.radosbench.0.smithi177.stdout: 23 16 1586 1570 4.26571 86.625 0.0145673 0.234353 2024-09-13T11:22:50.039 INFO:tasks.ceph.osd.4.smithi001.stderr:2024-09-13T11:22:50.032+0000 7ff8e0218640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26310) UID: 0 2024-09-13T11:22:50.047 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-09-13T11:22:50.048 INFO:teuthology.orchestra.run.smithi092.stdout: "ops": [ 2024-09-13T11:22:50.048 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.048 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:1699 3.6 3:6e6525c3:::benchmark_data_smithi177_31116_object1698:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-09-13T11:22:50.048 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:50.042251+0000", 2024-09-13T11:22:50.048 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 0.0065366390000000003, 2024-09-13T11:22:50.048 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0065629649999999996, 2024-09-13T11:22:50.048 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:22:50.048 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:22:50.048 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "waiting for sub ops", 2024-09-13T11:22:50.048 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:22:50.048 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:22:50.048 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:22:50.049 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 1699 2024-09-13T11:22:50.049 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.049 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:22:50.049 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.049 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:22:50.049 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:50.042251+0000", 2024-09-13T11:22:50.049 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:22:50.049 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.049 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.049 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:22:50.049 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:50.042251+0000", 2024-09-13T11:22:50.049 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:22:50.049 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.050 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.050 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:22:50.050 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:50.042252+0000", 2024-09-13T11:22:50.050 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.268e-06 2024-09-13T11:22:50.050 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.050 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.050 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:22:50.050 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:50.042333+0000", 2024-09-13T11:22:50.050 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.0881e-05 2024-09-13T11:22:50.050 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.050 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.050 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:22:50.050 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:50.042343+0000", 2024-09-13T11:22:50.051 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0181e-05 2024-09-13T11:22:50.051 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.051 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.051 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:22:50.051 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:50.042348+0000", 2024-09-13T11:22:50.051 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.4399999999999998e-06 2024-09-13T11:22:50.051 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.051 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.051 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:22:50.051 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:50.042401+0000", 2024-09-13T11:22:50.051 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.3845999999999999e-05 2024-09-13T11:22:50.051 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.051 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.051 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:22:50.052 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:50.042531+0000", 2024-09-13T11:22:50.052 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00012920100000000001 2024-09-13T11:22:50.052 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.052 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.052 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 14", 2024-09-13T11:22:50.052 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:50.042665+0000", 2024-09-13T11:22:50.052 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00013416599999999999 2024-09-13T11:22:50.052 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.052 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.052 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:22:50.052 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:50.047220+0000", 2024-09-13T11:22:50.053 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0045553039999999996 2024-09-13T11:22:50.053 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:22:50.053 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:22:50.053 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:22:50.053 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.053 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.053 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:1707 3.b 3:d26cba43:::benchmark_data_smithi177_31116_object1706:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-09-13T11:22:50.053 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:50.044833+0000", 2024-09-13T11:22:50.053 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 0.0039544910000000001, 2024-09-13T11:22:50.053 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0040290990000000004, 2024-09-13T11:22:50.053 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:22:50.053 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:22:50.054 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "waiting for sub ops", 2024-09-13T11:22:50.054 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:22:50.054 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:22:50.054 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:22:50.054 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 1707 2024-09-13T11:22:50.054 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.054 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:22:50.054 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.054 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:22:50.054 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:50.044833+0000", 2024-09-13T11:22:50.054 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:22:50.054 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.055 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.055 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:22:50.055 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:50.044833+0000", 2024-09-13T11:22:50.055 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:22:50.055 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.055 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.055 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:22:50.055 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:50.044834+0000", 2024-09-13T11:22:50.055 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.333e-06 2024-09-13T11:22:50.055 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.055 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.055 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:22:50.055 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:50.044898+0000", 2024-09-13T11:22:50.056 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.3781999999999993e-05 2024-09-13T11:22:50.056 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.056 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.056 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:22:50.056 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:50.044908+0000", 2024-09-13T11:22:50.056 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0309000000000001e-05 2024-09-13T11:22:50.056 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.056 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.056 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:22:50.056 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:50.044914+0000", 2024-09-13T11:22:50.056 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.575e-06 2024-09-13T11:22:50.056 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.056 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.057 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:22:50.057 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:50.044955+0000", 2024-09-13T11:22:50.057 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.0790000000000001e-05 2024-09-13T11:22:50.057 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.057 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.057 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:22:50.057 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:50.045046+0000", 2024-09-13T11:22:50.057 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.0906e-05 2024-09-13T11:22:50.057 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.057 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.057 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1", 2024-09-13T11:22:50.057 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:50.045133+0000", 2024-09-13T11:22:50.057 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.7645999999999994e-05 2024-09-13T11:22:50.058 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.058 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.058 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:22:50.058 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:50.047573+0000", 2024-09-13T11:22:50.058 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.002439821 2024-09-13T11:22:50.058 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:22:50.058 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:22:50.058 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:22:50.058 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:22:50.058 INFO:teuthology.orchestra.run.smithi092.stdout: ], 2024-09-13T11:22:50.058 INFO:teuthology.orchestra.run.smithi092.stdout: "num_ops": 2 2024-09-13T11:22:50.058 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-09-13T11:22:50.075 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_blocked_ops 2024-09-13T11:22:50.089 INFO:teuthology.orchestra.run.smithi001.stdout:osd.0: {} 2024-09-13T11:22:50.089 INFO:teuthology.orchestra.run.smithi001.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T11:22:50.115 INFO:teuthology.orchestra.run.smithi001.stdout:osd.1: {} 2024-09-13T11:22:50.116 INFO:teuthology.orchestra.run.smithi001.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T11:22:50.140 INFO:tasks.ceph.osd.11.smithi177.stderr:2024-09-13T11:22:50.140+0000 7f24792c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26580) UID: 0 2024-09-13T11:22:50.143 INFO:teuthology.orchestra.run.smithi001.stdout:osd.2: {} 2024-09-13T11:22:50.143 INFO:teuthology.orchestra.run.smithi001.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T11:22:50.169 INFO:teuthology.orchestra.run.smithi001.stdout:osd.3: {} 2024-09-13T11:22:50.169 INFO:teuthology.orchestra.run.smithi001.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T11:22:50.196 INFO:teuthology.orchestra.run.smithi001.stdout:osd.4: {} 2024-09-13T11:22:50.197 INFO:teuthology.orchestra.run.smithi001.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T11:22:50.223 INFO:teuthology.orchestra.run.smithi001.stdout:osd.5: {} 2024-09-13T11:22:50.223 INFO:teuthology.orchestra.run.smithi001.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T11:22:50.226 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-09-13T11:22:50.226 INFO:teuthology.orchestra.run.smithi092.stdout: "ops": [], 2024-09-13T11:22:50.226 INFO:teuthology.orchestra.run.smithi092.stdout: "complaint_time": 30, 2024-09-13T11:22:50.226 INFO:teuthology.orchestra.run.smithi092.stdout: "num_blocked_ops": 0 2024-09-13T11:22:50.226 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-09-13T11:22:50.240 INFO:tasks.ceph.osd.13.smithi092.stderr:2024-09-13T11:22:50.239+0000 7f3e5afea640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26350) UID: 0 2024-09-13T11:22:50.247 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_historic_ops 2024-09-13T11:22:50.249 INFO:teuthology.orchestra.run.smithi001.stdout:osd.6: {} 2024-09-13T11:22:50.249 INFO:teuthology.orchestra.run.smithi001.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T11:22:50.275 INFO:teuthology.orchestra.run.smithi001.stdout:osd.7: {} 2024-09-13T11:22:50.275 INFO:teuthology.orchestra.run.smithi001.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T11:22:50.308 INFO:teuthology.orchestra.run.smithi001.stdout:osd.8: {} 2024-09-13T11:22:50.308 INFO:teuthology.orchestra.run.smithi001.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-13T11:22:50.324 INFO:teuthology.orchestra.run.smithi001.stderr:reweighted osd.4 to 1 (10000) 2024-09-13T11:22:50.327 INFO:tasks.ceph.osd.11.smithi177.stderr:2024-09-13T11:22:50.328+0000 7f246bcf0640 -1 osd.11 43 handle_osd_map injecting map crc failure 2024-09-13T11:22:50.327 INFO:tasks.ceph.osd.10.smithi162.stderr:2024-09-13T11:22:50.323+0000 7fb593421640 -1 osd.10 43 handle_osd_map injecting map crc failure 2024-09-13T11:22:50.341 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-13T11:22:50.336+0000 7f6a68775640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26307) UID: 0 2024-09-13T11:22:50.347 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 5 1 2024-09-13T11:22:50.347 INFO:teuthology.orchestra.run.smithi001.stdout:osd.9: {} 2024-09-13T11:22:50.348 INFO:teuthology.orchestra.run.smithi001.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-13T11:22:50.377 INFO:teuthology.orchestra.run.smithi001.stdout:osd.10: {} 2024-09-13T11:22:50.377 INFO:teuthology.orchestra.run.smithi001.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-13T11:22:50.401 INFO:teuthology.orchestra.run.smithi001.stdout:osd.11: {} 2024-09-13T11:22:50.402 INFO:teuthology.orchestra.run.smithi001.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-13T11:22:50.427 INFO:teuthology.orchestra.run.smithi001.stdout:osd.12: {} 2024-09-13T11:22:50.427 INFO:teuthology.orchestra.run.smithi001.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-13T11:22:50.440 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-09-13T11:22:50.440 INFO:teuthology.orchestra.run.smithi092.stdout: "size": 20, 2024-09-13T11:22:50.440 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 600, 2024-09-13T11:22:50.440 INFO:teuthology.orchestra.run.smithi092.stdout: "ops": [ 2024-09-13T11:22:50.440 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.440 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:79 3.1 3:841df5b5:::benchmark_data_smithi177_31116_object78:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-09-13T11:22:50.440 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:27.056083+0000", 2024-09-13T11:22:50.440 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 23.382602591000001, 2024-09-13T11:22:50.440 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.89469730199999997, 2024-09-13T11:22:50.440 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:22:50.440 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:22:50.441 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:50.441 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:22:50.441 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:22:50.441 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:22:50.441 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 79 2024-09-13T11:22:50.441 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.441 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:22:50.441 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.441 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:22:50.441 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.056083+0000", 2024-09-13T11:22:50.441 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:22:50.441 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.441 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.441 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:22:50.442 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.056083+0000", 2024-09-13T11:22:50.442 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:22:50.442 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.442 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.442 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:22:50.442 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.056087+0000", 2024-09-13T11:22:50.442 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.0659999999999997e-06 2024-09-13T11:22:50.442 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.442 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.442 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:22:50.443 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.056148+0000", 2024-09-13T11:22:50.443 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.1376000000000003e-05 2024-09-13T11:22:50.443 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.443 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.443 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:22:50.443 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.056180+0000", 2024-09-13T11:22:50.443 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.2370999999999999e-05 2024-09-13T11:22:50.443 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.443 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.443 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:22:50.443 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.056195+0000", 2024-09-13T11:22:50.443 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.5119e-05 2024-09-13T11:22:50.443 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.443 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.444 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:22:50.444 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.056304+0000", 2024-09-13T11:22:50.444 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000108365 2024-09-13T11:22:50.444 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.444 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.444 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:22:50.444 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.056478+0000", 2024-09-13T11:22:50.444 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000174542 2024-09-13T11:22:50.444 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.444 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.444 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 11", 2024-09-13T11:22:50.444 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.056690+0000", 2024-09-13T11:22:50.444 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00021167500000000001 2024-09-13T11:22:50.445 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.445 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.445 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:22:50.445 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.058063+0000", 2024-09-13T11:22:50.445 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0013731819999999999 2024-09-13T11:22:50.445 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.445 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.445 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:22:50.445 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.950477+0000", 2024-09-13T11:22:50.445 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.89241369400000004 2024-09-13T11:22:50.445 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.445 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.445 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:22:50.446 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.950652+0000", 2024-09-13T11:22:50.446 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00017526400000000001 2024-09-13T11:22:50.446 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.446 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.446 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:22:50.446 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.950780+0000", 2024-09-13T11:22:50.446 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00012764800000000001 2024-09-13T11:22:50.446 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:22:50.446 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:22:50.446 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:22:50.446 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.446 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.446 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:7 3.b 3:de78393b:::benchmark_data_smithi177_31116_object6:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T11:22:50.446 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:48.313952+0000", 2024-09-13T11:22:50.447 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 2.1247328419999998, 2024-09-13T11:22:50.447 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.86421288600000001, 2024-09-13T11:22:50.447 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:22:50.447 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:22:50.447 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:50.447 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:22:50.447 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:22:50.447 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:22:50.447 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 7 2024-09-13T11:22:50.447 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.447 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:22:50.447 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.447 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:22:50.447 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.313952+0000", 2024-09-13T11:22:50.447 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:22:50.448 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.448 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.448 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:22:50.448 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.313952+0000", 2024-09-13T11:22:50.448 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:22:50.448 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.448 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.448 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:22:50.448 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.313957+0000", 2024-09-13T11:22:50.448 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.3599999999999998e-06 2024-09-13T11:22:50.448 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.448 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.448 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:22:50.448 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314043+0000", 2024-09-13T11:22:50.449 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.6766000000000002e-05 2024-09-13T11:22:50.449 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.449 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.449 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:22:50.449 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314065+0000", 2024-09-13T11:22:50.449 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.1416e-05 2024-09-13T11:22:50.449 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.449 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.449 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:22:50.449 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314080+0000", 2024-09-13T11:22:50.449 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.5519000000000002e-05 2024-09-13T11:22:50.449 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.449 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.450 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:22:50.450 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315478+0000", 2024-09-13T11:22:50.450 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.001397765 2024-09-13T11:22:50.450 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.450 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.450 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-09-13T11:22:50.450 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315507+0000", 2024-09-13T11:22:50.450 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.9128999999999998e-05 2024-09-13T11:22:50.450 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.450 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.450 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:22:50.450 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.167978+0000", 2024-09-13T11:22:50.450 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.85247068400000003 2024-09-13T11:22:50.450 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.451 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.451 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:22:50.451 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.168307+0000", 2024-09-13T11:22:50.451 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00032886699999999999 2024-09-13T11:22:50.451 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.451 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.451 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1", 2024-09-13T11:22:50.451 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.168647+0000", 2024-09-13T11:22:50.451 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00033998399999999999 2024-09-13T11:22:50.451 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.451 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.451 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:22:50.451 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.178011+0000", 2024-09-13T11:22:50.451 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0093639169999999994 2024-09-13T11:22:50.452 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.452 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.452 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:22:50.452 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.178026+0000", 2024-09-13T11:22:50.452 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.5274999999999999e-05 2024-09-13T11:22:50.452 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.452 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.452 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:22:50.452 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.178080+0000", 2024-09-13T11:22:50.452 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.3601999999999999e-05 2024-09-13T11:22:50.453 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.453 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.453 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:22:50.453 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.178165+0000", 2024-09-13T11:22:50.453 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.5601999999999998e-05 2024-09-13T11:22:50.453 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:22:50.453 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:22:50.453 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:22:50.453 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.453 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.453 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:9 3.b 3:dafadff6:::benchmark_data_smithi177_31116_object8:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T11:22:50.453 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:48.314114+0000", 2024-09-13T11:22:50.453 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 2.1245710280000001, 2024-09-13T11:22:50.453 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.86613615899999996, 2024-09-13T11:22:50.453 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:22:50.454 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:22:50.454 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:50.454 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:22:50.454 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:22:50.454 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:22:50.454 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 9 2024-09-13T11:22:50.454 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.454 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:22:50.454 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.454 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:22:50.454 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314114+0000", 2024-09-13T11:22:50.454 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:22:50.454 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.454 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.455 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:22:50.455 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314114+0000", 2024-09-13T11:22:50.455 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:22:50.455 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.455 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.455 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:22:50.455 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314117+0000", 2024-09-13T11:22:50.455 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.3860000000000001e-06 2024-09-13T11:22:50.455 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.455 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.455 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:22:50.455 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314215+0000", 2024-09-13T11:22:50.455 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.8900000000000005e-05 2024-09-13T11:22:50.456 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.456 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.456 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:22:50.456 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314229+0000", 2024-09-13T11:22:50.456 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.3444999999999999e-05 2024-09-13T11:22:50.456 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.456 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.456 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:22:50.456 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314239+0000", 2024-09-13T11:22:50.456 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.8470000000000008e-06 2024-09-13T11:22:50.456 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.456 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.457 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:22:50.457 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315648+0000", 2024-09-13T11:22:50.457 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.001409291 2024-09-13T11:22:50.457 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.457 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.457 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-09-13T11:22:50.457 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315671+0000", 2024-09-13T11:22:50.457 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.3169000000000001e-05 2024-09-13T11:22:50.457 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.457 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.457 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:22:50.457 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.176036+0000", 2024-09-13T11:22:50.457 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.86036497499999998 2024-09-13T11:22:50.458 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.458 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.458 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:22:50.458 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.176189+0000", 2024-09-13T11:22:50.458 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00015271100000000001 2024-09-13T11:22:50.458 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.458 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.458 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1", 2024-09-13T11:22:50.458 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.176339+0000", 2024-09-13T11:22:50.458 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00015057499999999999 2024-09-13T11:22:50.458 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.458 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.458 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:22:50.458 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.180148+0000", 2024-09-13T11:22:50.458 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.003808891 2024-09-13T11:22:50.459 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.459 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.459 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:22:50.459 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.180168+0000", 2024-09-13T11:22:50.459 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.9507000000000001e-05 2024-09-13T11:22:50.459 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.459 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.459 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:22:50.459 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.180213+0000", 2024-09-13T11:22:50.459 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.5504999999999999e-05 2024-09-13T11:22:50.459 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.459 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.459 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:22:50.460 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.180250+0000", 2024-09-13T11:22:50.460 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.6956999999999999e-05 2024-09-13T11:22:50.460 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:22:50.460 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:22:50.460 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:22:50.460 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.460 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.460 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:10 3.b 3:dbe028a8:::benchmark_data_smithi177_31116_object9:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T11:22:50.460 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:48.314259+0000", 2024-09-13T11:22:50.460 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 2.1244258980000001, 2024-09-13T11:22:50.460 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.86637313599999999, 2024-09-13T11:22:50.460 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:22:50.460 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:22:50.461 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:50.461 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:22:50.461 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:22:50.461 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:22:50.461 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 10 2024-09-13T11:22:50.461 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.461 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:22:50.461 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.461 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:22:50.461 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314259+0000", 2024-09-13T11:22:50.461 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:22:50.461 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.461 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.461 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:22:50.462 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314259+0000", 2024-09-13T11:22:50.462 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:22:50.462 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.462 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.462 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:22:50.462 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314261+0000", 2024-09-13T11:22:50.462 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.1500000000000002e-06 2024-09-13T11:22:50.462 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.462 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.462 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:22:50.463 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314349+0000", 2024-09-13T11:22:50.463 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.8070000000000002e-05 2024-09-13T11:22:50.463 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.463 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.463 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:22:50.463 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314365+0000", 2024-09-13T11:22:50.463 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.5835999999999999e-05 2024-09-13T11:22:50.463 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.463 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.463 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:22:50.463 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314377+0000", 2024-09-13T11:22:50.463 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.1589e-05 2024-09-13T11:22:50.463 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.463 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.464 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:22:50.464 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315788+0000", 2024-09-13T11:22:50.464 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0014109579999999999 2024-09-13T11:22:50.464 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.464 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.464 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-09-13T11:22:50.464 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315809+0000", 2024-09-13T11:22:50.464 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.0985999999999999e-05 2024-09-13T11:22:50.464 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.464 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.464 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:22:50.464 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.176822+0000", 2024-09-13T11:22:50.464 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.86101310799999997 2024-09-13T11:22:50.464 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.465 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.465 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:22:50.465 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.177021+0000", 2024-09-13T11:22:50.465 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00019911199999999999 2024-09-13T11:22:50.465 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.465 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.465 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1", 2024-09-13T11:22:50.465 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.177226+0000", 2024-09-13T11:22:50.465 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00020516600000000001 2024-09-13T11:22:50.465 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.465 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.465 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:22:50.465 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.180264+0000", 2024-09-13T11:22:50.465 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0030381739999999998 2024-09-13T11:22:50.465 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.466 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.466 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:22:50.466 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.180387+0000", 2024-09-13T11:22:50.466 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00012257699999999999 2024-09-13T11:22:50.466 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.466 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.466 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:22:50.466 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.180518+0000", 2024-09-13T11:22:50.466 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00013135800000000001 2024-09-13T11:22:50.466 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.466 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.466 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:22:50.466 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.180632+0000", 2024-09-13T11:22:50.466 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000114052 2024-09-13T11:22:50.467 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:22:50.467 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:22:50.467 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:22:50.467 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.467 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.467 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:39 3.b 3:d91854b3:::benchmark_data_smithi177_31116_object38:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T11:22:50.467 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:48.314393+0000", 2024-09-13T11:22:50.467 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 2.1242923560000002, 2024-09-13T11:22:50.467 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.86850121700000005, 2024-09-13T11:22:50.467 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:22:50.467 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:22:50.467 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:50.467 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:22:50.467 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:22:50.467 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:22:50.468 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 39 2024-09-13T11:22:50.468 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.468 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:22:50.468 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.468 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:22:50.468 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314393+0000", 2024-09-13T11:22:50.468 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:22:50.468 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.468 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.468 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:22:50.468 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314393+0000", 2024-09-13T11:22:50.468 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:22:50.468 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.469 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.469 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:22:50.469 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314394+0000", 2024-09-13T11:22:50.469 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.513e-06 2024-09-13T11:22:50.469 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.469 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.469 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:22:50.469 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314463+0000", 2024-09-13T11:22:50.469 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.8261000000000003e-05 2024-09-13T11:22:50.469 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.469 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.469 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:22:50.469 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314477+0000", 2024-09-13T11:22:50.469 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.4619e-05 2024-09-13T11:22:50.469 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.470 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.470 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:22:50.470 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314485+0000", 2024-09-13T11:22:50.470 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.1699999999999997e-06 2024-09-13T11:22:50.470 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.470 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.470 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:22:50.470 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315917+0000", 2024-09-13T11:22:50.470 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0014316089999999999 2024-09-13T11:22:50.470 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.470 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.470 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-09-13T11:22:50.470 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315933+0000", 2024-09-13T11:22:50.470 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.5999999999999999e-05 2024-09-13T11:22:50.471 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.471 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.471 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:22:50.471 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.178205+0000", 2024-09-13T11:22:50.471 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.86227247100000004 2024-09-13T11:22:50.471 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.471 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.471 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:22:50.471 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.178316+0000", 2024-09-13T11:22:50.471 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000110788 2024-09-13T11:22:50.472 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.472 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.472 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1", 2024-09-13T11:22:50.472 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.178448+0000", 2024-09-13T11:22:50.472 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000132056 2024-09-13T11:22:50.472 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.472 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.472 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:22:50.472 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.182721+0000", 2024-09-13T11:22:50.472 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0042729780000000002 2024-09-13T11:22:50.472 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.472 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.472 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:22:50.473 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.182801+0000", 2024-09-13T11:22:50.473 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.0209000000000004e-05 2024-09-13T11:22:50.473 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.473 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.473 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:22:50.473 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.182855+0000", 2024-09-13T11:22:50.473 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.3749000000000003e-05 2024-09-13T11:22:50.473 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.473 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.473 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:22:50.473 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.182894+0000", 2024-09-13T11:22:50.473 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.8794000000000002e-05 2024-09-13T11:22:50.473 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:22:50.474 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:22:50.474 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:22:50.474 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.474 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.474 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:50 3.b 3:d1c144b8:::benchmark_data_smithi177_31116_object49:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T11:22:50.474 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:48.314498+0000", 2024-09-13T11:22:50.474 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 2.1241869410000001, 2024-09-13T11:22:50.474 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.86878673299999998, 2024-09-13T11:22:50.474 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:22:50.474 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:22:50.474 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:50.474 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:22:50.474 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:22:50.474 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:22:50.475 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 50 2024-09-13T11:22:50.475 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.475 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:22:50.475 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.475 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:22:50.475 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314498+0000", 2024-09-13T11:22:50.475 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:22:50.475 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.475 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.475 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:22:50.475 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314498+0000", 2024-09-13T11:22:50.475 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:22:50.475 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.475 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.476 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:22:50.476 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314499+0000", 2024-09-13T11:22:50.476 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.1969999999999999e-06 2024-09-13T11:22:50.476 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.476 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.476 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:22:50.476 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314565+0000", 2024-09-13T11:22:50.476 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.5628000000000003e-05 2024-09-13T11:22:50.476 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.476 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.476 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:22:50.476 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314579+0000", 2024-09-13T11:22:50.476 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.4222e-05 2024-09-13T11:22:50.476 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.477 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.477 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:22:50.477 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314590+0000", 2024-09-13T11:22:50.477 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.1240000000000001e-05 2024-09-13T11:22:50.477 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.477 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.477 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:22:50.477 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316060+0000", 2024-09-13T11:22:50.477 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.001469515 2024-09-13T11:22:50.477 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.477 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.477 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-09-13T11:22:50.477 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316076+0000", 2024-09-13T11:22:50.477 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.5608999999999999e-05 2024-09-13T11:22:50.478 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.478 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.478 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:22:50.478 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.178857+0000", 2024-09-13T11:22:50.478 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.86278099900000005 2024-09-13T11:22:50.478 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.478 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.478 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:22:50.478 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.178961+0000", 2024-09-13T11:22:50.478 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000104664 2024-09-13T11:22:50.478 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.478 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.478 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1", 2024-09-13T11:22:50.478 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.179123+0000", 2024-09-13T11:22:50.479 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000161738 2024-09-13T11:22:50.479 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.479 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.479 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:22:50.479 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.182755+0000", 2024-09-13T11:22:50.479 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0036323200000000001 2024-09-13T11:22:50.479 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.479 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.479 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:22:50.479 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.183023+0000", 2024-09-13T11:22:50.479 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00026749100000000002 2024-09-13T11:22:50.479 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.479 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.479 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:22:50.479 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.183168+0000", 2024-09-13T11:22:50.480 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00014499099999999999 2024-09-13T11:22:50.480 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.480 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.480 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:22:50.480 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.183285+0000", 2024-09-13T11:22:50.480 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000117119 2024-09-13T11:22:50.480 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:22:50.480 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:22:50.480 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:22:50.480 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.480 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.480 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:60 3.b 3:d67218b0:::benchmark_data_smithi177_31116_object59:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T11:22:50.480 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:48.314603+0000", 2024-09-13T11:22:50.480 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 2.1240823779999998, 2024-09-13T11:22:50.481 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.87106201100000002, 2024-09-13T11:22:50.481 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:22:50.481 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:22:50.481 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:50.481 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:22:50.481 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:22:50.481 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:22:50.481 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 60 2024-09-13T11:22:50.481 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.481 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:22:50.481 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.481 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:22:50.481 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314603+0000", 2024-09-13T11:22:50.482 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:22:50.482 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.482 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.482 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:22:50.482 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314603+0000", 2024-09-13T11:22:50.482 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:22:50.482 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.482 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.482 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:22:50.482 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314604+0000", 2024-09-13T11:22:50.482 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.2359999999999999e-06 2024-09-13T11:22:50.482 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.482 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.482 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:22:50.483 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314667+0000", 2024-09-13T11:22:50.483 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.3077999999999995e-05 2024-09-13T11:22:50.483 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.483 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.483 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:22:50.483 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314699+0000", 2024-09-13T11:22:50.483 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.1965999999999999e-05 2024-09-13T11:22:50.483 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.483 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.483 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:22:50.483 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314710+0000", 2024-09-13T11:22:50.483 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.1148999999999999e-05 2024-09-13T11:22:50.483 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.483 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.484 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:22:50.484 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316291+0000", 2024-09-13T11:22:50.484 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0015804440000000001 2024-09-13T11:22:50.484 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.484 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.484 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-09-13T11:22:50.484 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316310+0000", 2024-09-13T11:22:50.484 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.9799000000000001e-05 2024-09-13T11:22:50.484 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.484 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.484 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:22:50.484 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.179555+0000", 2024-09-13T11:22:50.484 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.86324469500000001 2024-09-13T11:22:50.485 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.485 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.485 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:22:50.485 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.179666+0000", 2024-09-13T11:22:50.485 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000110443 2024-09-13T11:22:50.485 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.485 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.485 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1", 2024-09-13T11:22:50.485 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.179791+0000", 2024-09-13T11:22:50.485 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00012560499999999999 2024-09-13T11:22:50.485 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.485 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.485 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:22:50.485 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.182761+0000", 2024-09-13T11:22:50.486 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0029697019999999998 2024-09-13T11:22:50.486 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.486 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.486 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:22:50.486 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.185572+0000", 2024-09-13T11:22:50.486 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.002811512 2024-09-13T11:22:50.486 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.486 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.486 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:22:50.486 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.185625+0000", 2024-09-13T11:22:50.486 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.2716000000000001e-05 2024-09-13T11:22:50.486 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.486 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.486 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:22:50.487 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.185665+0000", 2024-09-13T11:22:50.487 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.9666000000000003e-05 2024-09-13T11:22:50.487 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:22:50.487 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:22:50.487 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:22:50.487 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.487 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.487 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:62 3.b 3:d20ae5a8:::benchmark_data_smithi177_31116_object61:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T11:22:50.487 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:48.314723+0000", 2024-09-13T11:22:50.487 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 2.1239618359999999, 2024-09-13T11:22:50.487 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.87338985099999999, 2024-09-13T11:22:50.487 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:22:50.487 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:22:50.487 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:50.488 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:22:50.488 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:22:50.488 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:22:50.488 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 62 2024-09-13T11:22:50.488 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.488 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:22:50.488 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.488 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:22:50.488 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314723+0000", 2024-09-13T11:22:50.488 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:22:50.488 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.488 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.488 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:22:50.489 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314723+0000", 2024-09-13T11:22:50.489 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:22:50.489 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.489 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.489 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:22:50.489 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314725+0000", 2024-09-13T11:22:50.489 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.3069999999999999e-06 2024-09-13T11:22:50.489 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.489 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.489 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:22:50.489 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314797+0000", 2024-09-13T11:22:50.489 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.2652999999999997e-05 2024-09-13T11:22:50.489 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.489 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.490 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:22:50.490 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314809+0000", 2024-09-13T11:22:50.490 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.1496e-05 2024-09-13T11:22:50.490 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.490 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.490 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:22:50.490 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314816+0000", 2024-09-13T11:22:50.490 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.6639999999999998e-06 2024-09-13T11:22:50.490 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.490 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.490 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:22:50.490 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316424+0000", 2024-09-13T11:22:50.490 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0016077559999999999 2024-09-13T11:22:50.490 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.491 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.491 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-09-13T11:22:50.491 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316441+0000", 2024-09-13T11:22:50.491 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.7305000000000001e-05 2024-09-13T11:22:50.491 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.491 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.491 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:22:50.491 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.180688+0000", 2024-09-13T11:22:50.491 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.86424692800000003 2024-09-13T11:22:50.491 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.491 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.491 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:22:50.491 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.180807+0000", 2024-09-13T11:22:50.492 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000118977 2024-09-13T11:22:50.492 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.492 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.492 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1", 2024-09-13T11:22:50.492 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.180938+0000", 2024-09-13T11:22:50.492 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00013049700000000001 2024-09-13T11:22:50.492 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.492 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.492 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:22:50.492 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.185680+0000", 2024-09-13T11:22:50.492 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0047417029999999999 2024-09-13T11:22:50.492 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.492 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.493 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:22:50.493 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.187856+0000", 2024-09-13T11:22:50.493 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0021760550000000001 2024-09-13T11:22:50.493 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.493 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.493 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:22:50.493 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.188007+0000", 2024-09-13T11:22:50.493 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00015185099999999999 2024-09-13T11:22:50.493 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.493 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.494 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:22:50.494 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.188113+0000", 2024-09-13T11:22:50.494 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000105659 2024-09-13T11:22:50.494 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:22:50.494 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:22:50.494 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:22:50.494 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.494 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.494 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:73 3.b 3:d0060d5f:::benchmark_data_smithi177_31116_object72:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T11:22:50.494 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:48.314829+0000", 2024-09-13T11:22:50.494 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 2.1238564869999998, 2024-09-13T11:22:50.494 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.87364757199999998, 2024-09-13T11:22:50.494 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:22:50.494 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:22:50.495 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:50.495 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:22:50.495 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:22:50.495 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:22:50.495 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 73 2024-09-13T11:22:50.495 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.495 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:22:50.495 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.495 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:22:50.495 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314829+0000", 2024-09-13T11:22:50.495 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:22:50.495 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.495 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.496 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:22:50.496 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314829+0000", 2024-09-13T11:22:50.496 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:22:50.496 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.496 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.496 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:22:50.496 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314830+0000", 2024-09-13T11:22:50.496 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.243e-06 2024-09-13T11:22:50.496 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.496 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.496 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:22:50.496 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314965+0000", 2024-09-13T11:22:50.496 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00013470800000000001 2024-09-13T11:22:50.497 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.497 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.497 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:22:50.497 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314979+0000", 2024-09-13T11:22:50.497 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.4375999999999999e-05 2024-09-13T11:22:50.497 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.497 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.497 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:22:50.497 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314988+0000", 2024-09-13T11:22:50.497 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.8570000000000001e-06 2024-09-13T11:22:50.497 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.497 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.497 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:22:50.498 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316545+0000", 2024-09-13T11:22:50.498 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.001556878 2024-09-13T11:22:50.498 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.498 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.498 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-09-13T11:22:50.498 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316556+0000", 2024-09-13T11:22:50.498 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.1453e-05 2024-09-13T11:22:50.498 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.498 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.498 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:22:50.498 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.182124+0000", 2024-09-13T11:22:50.498 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.86556744200000002 2024-09-13T11:22:50.498 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.498 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.499 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:22:50.499 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.182246+0000", 2024-09-13T11:22:50.499 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00012210500000000001 2024-09-13T11:22:50.499 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.499 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.499 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1", 2024-09-13T11:22:50.499 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.182375+0000", 2024-09-13T11:22:50.499 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00012920999999999999 2024-09-13T11:22:50.499 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.499 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.499 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:22:50.499 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.185684+0000", 2024-09-13T11:22:50.499 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0033093969999999999 2024-09-13T11:22:50.500 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.500 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.500 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:22:50.500 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.188244+0000", 2024-09-13T11:22:50.500 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0025601550000000002 2024-09-13T11:22:50.500 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.500 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.500 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:22:50.500 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.188379+0000", 2024-09-13T11:22:50.500 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00013463400000000001 2024-09-13T11:22:50.500 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.500 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.500 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:22:50.500 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.188476+0000", 2024-09-13T11:22:50.501 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.7114000000000006e-05 2024-09-13T11:22:50.501 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:22:50.501 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:22:50.501 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:22:50.501 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.501 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.501 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:81 3.b 3:d705fd70:::benchmark_data_smithi177_31116_object80:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T11:22:50.501 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:48.315000+0000", 2024-09-13T11:22:50.501 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 2.12368465, 2024-09-13T11:22:50.501 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.87561418300000005, 2024-09-13T11:22:50.501 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:22:50.501 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:22:50.501 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:50.501 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:22:50.502 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:22:50.502 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:22:50.502 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 81 2024-09-13T11:22:50.502 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.502 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:22:50.502 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.502 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:22:50.502 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315000+0000", 2024-09-13T11:22:50.502 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:22:50.502 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.502 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.502 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:22:50.502 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315000+0000", 2024-09-13T11:22:50.502 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:22:50.503 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.503 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.503 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:22:50.503 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315002+0000", 2024-09-13T11:22:50.503 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.3799999999999999e-06 2024-09-13T11:22:50.503 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.503 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.503 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:22:50.503 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315119+0000", 2024-09-13T11:22:50.503 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000116925 2024-09-13T11:22:50.503 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.503 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.503 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:22:50.504 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315133+0000", 2024-09-13T11:22:50.504 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.3971999999999999e-05 2024-09-13T11:22:50.504 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.504 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.504 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:22:50.504 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315142+0000", 2024-09-13T11:22:50.504 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.4399999999999994e-06 2024-09-13T11:22:50.504 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.504 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.504 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:22:50.504 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316652+0000", 2024-09-13T11:22:50.504 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.001509907 2024-09-13T11:22:50.504 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.505 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.505 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-09-13T11:22:50.505 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316668+0000", 2024-09-13T11:22:50.505 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.5945999999999999e-05 2024-09-13T11:22:50.505 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.505 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.505 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:22:50.505 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.183405+0000", 2024-09-13T11:22:50.505 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.866737384 2024-09-13T11:22:50.505 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.505 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.505 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:22:50.505 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.183701+0000", 2024-09-13T11:22:50.506 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00029511099999999997 2024-09-13T11:22:50.506 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.506 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.506 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1", 2024-09-13T11:22:50.506 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.184078+0000", 2024-09-13T11:22:50.506 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00037741000000000001 2024-09-13T11:22:50.506 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.506 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.506 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:22:50.506 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.188504+0000", 2024-09-13T11:22:50.506 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0044260280000000003 2024-09-13T11:22:50.506 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.506 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.506 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:22:50.507 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.190562+0000", 2024-09-13T11:22:50.507 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0020580099999999999 2024-09-13T11:22:50.507 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.507 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.507 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:22:50.507 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.190592+0000", 2024-09-13T11:22:50.507 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.0057999999999999e-05 2024-09-13T11:22:50.507 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.507 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.507 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:22:50.507 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.190615+0000", 2024-09-13T11:22:50.507 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.2612e-05 2024-09-13T11:22:50.507 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:22:50.507 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:22:50.508 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:22:50.508 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.508 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.508 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:87 3.b 3:dd7de0ad:::benchmark_data_smithi177_31116_object86:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T11:22:50.508 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:48.315155+0000", 2024-09-13T11:22:50.508 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 2.123529853, 2024-09-13T11:22:50.508 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.87555916099999997, 2024-09-13T11:22:50.508 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:22:50.508 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:22:50.508 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:50.508 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:22:50.508 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:22:50.508 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:22:50.509 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 87 2024-09-13T11:22:50.509 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.509 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:22:50.509 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.509 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:22:50.509 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315155+0000", 2024-09-13T11:22:50.509 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:22:50.509 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.509 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.509 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:22:50.509 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315155+0000", 2024-09-13T11:22:50.509 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:22:50.509 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.509 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.510 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:22:50.510 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315157+0000", 2024-09-13T11:22:50.510 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.3829999999999999e-06 2024-09-13T11:22:50.510 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.510 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.510 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:22:50.510 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315271+0000", 2024-09-13T11:22:50.510 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000114539 2024-09-13T11:22:50.510 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.510 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.510 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:22:50.510 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315285+0000", 2024-09-13T11:22:50.510 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.4042e-05 2024-09-13T11:22:50.510 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.510 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.511 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:22:50.511 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315295+0000", 2024-09-13T11:22:50.511 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.2739999999999996e-06 2024-09-13T11:22:50.511 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.511 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.511 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:22:50.511 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316739+0000", 2024-09-13T11:22:50.511 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0014446960000000001 2024-09-13T11:22:50.511 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.511 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.511 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-09-13T11:22:50.511 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316747+0000", 2024-09-13T11:22:50.511 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.0790000000000001e-06 2024-09-13T11:22:50.511 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.511 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.512 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:22:50.512 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.184922+0000", 2024-09-13T11:22:50.512 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.86817446399999998 2024-09-13T11:22:50.512 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.512 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.512 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:22:50.512 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.185037+0000", 2024-09-13T11:22:50.512 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00011565099999999999 2024-09-13T11:22:50.512 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.512 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.512 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1", 2024-09-13T11:22:50.512 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.185171+0000", 2024-09-13T11:22:50.512 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00013316900000000001 2024-09-13T11:22:50.513 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.513 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.513 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:22:50.513 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.188520+0000", 2024-09-13T11:22:50.513 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0033489150000000001 2024-09-13T11:22:50.513 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.513 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.513 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:22:50.513 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.190652+0000", 2024-09-13T11:22:50.513 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0021325649999999999 2024-09-13T11:22:50.513 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.513 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.513 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:22:50.513 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.190686+0000", 2024-09-13T11:22:50.514 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.3827e-05 2024-09-13T11:22:50.514 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.514 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.514 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:22:50.514 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.190714+0000", 2024-09-13T11:22:50.514 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.8557000000000001e-05 2024-09-13T11:22:50.514 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:22:50.514 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:22:50.514 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:22:50.514 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.514 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.514 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:99 3.b 3:d011b600:::benchmark_data_smithi177_31116_object98:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T11:22:50.514 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:48.315307+0000", 2024-09-13T11:22:50.514 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 2.1233780260000001, 2024-09-13T11:22:50.514 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.87551770799999995, 2024-09-13T11:22:50.515 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:22:50.515 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:22:50.515 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:50.515 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:22:50.515 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:22:50.515 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:22:50.515 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 99 2024-09-13T11:22:50.515 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.515 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:22:50.515 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.515 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:22:50.515 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315307+0000", 2024-09-13T11:22:50.515 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:22:50.515 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.516 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.516 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:22:50.516 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315307+0000", 2024-09-13T11:22:50.516 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:22:50.516 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.516 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.516 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:22:50.516 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315308+0000", 2024-09-13T11:22:50.516 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.316e-06 2024-09-13T11:22:50.516 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.516 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.516 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:22:50.516 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315453+0000", 2024-09-13T11:22:50.516 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00014473200000000001 2024-09-13T11:22:50.516 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.517 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.517 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:22:50.517 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315471+0000", 2024-09-13T11:22:50.517 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.8019000000000001e-05 2024-09-13T11:22:50.517 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.517 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.517 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:22:50.517 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315489+0000", 2024-09-13T11:22:50.517 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.7861999999999999e-05 2024-09-13T11:22:50.517 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.517 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.517 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:22:50.517 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316837+0000", 2024-09-13T11:22:50.517 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.001347787 2024-09-13T11:22:50.518 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.518 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.518 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-09-13T11:22:50.518 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316858+0000", 2024-09-13T11:22:50.518 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.1103000000000001e-05 2024-09-13T11:22:50.518 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.518 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.518 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:22:50.518 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.185713+0000", 2024-09-13T11:22:50.518 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.86885540800000005 2024-09-13T11:22:50.518 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.518 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.518 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:22:50.518 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.185820+0000", 2024-09-13T11:22:50.518 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000107081 2024-09-13T11:22:50.519 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.519 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.519 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1", 2024-09-13T11:22:50.519 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.185948+0000", 2024-09-13T11:22:50.519 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00012775800000000001 2024-09-13T11:22:50.519 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.519 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.519 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:22:50.519 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.190527+0000", 2024-09-13T11:22:50.519 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0045783910000000002 2024-09-13T11:22:50.519 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.519 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.519 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:22:50.519 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.190759+0000", 2024-09-13T11:22:50.520 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00023266199999999999 2024-09-13T11:22:50.520 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.520 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.520 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:22:50.520 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.190789+0000", 2024-09-13T11:22:50.520 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.9666999999999999e-05 2024-09-13T11:22:50.520 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.520 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.520 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:22:50.520 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.190825+0000", 2024-09-13T11:22:50.520 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.5921999999999999e-05 2024-09-13T11:22:50.520 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:22:50.520 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:22:50.520 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:22:50.521 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.521 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.521 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:108 3.b 3:d5c90217:::benchmark_data_smithi177_31116_object107:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T11:22:50.521 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:48.315513+0000", 2024-09-13T11:22:50.521 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 2.1231723210000002, 2024-09-13T11:22:50.521 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.87544772500000001, 2024-09-13T11:22:50.521 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:22:50.521 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:22:50.521 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:50.521 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:22:50.521 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:22:50.521 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:22:50.521 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 108 2024-09-13T11:22:50.522 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.522 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:22:50.522 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.522 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:22:50.522 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315513+0000", 2024-09-13T11:22:50.522 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:22:50.522 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.522 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.522 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:22:50.522 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315513+0000", 2024-09-13T11:22:50.522 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:22:50.522 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.522 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.522 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:22:50.523 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315515+0000", 2024-09-13T11:22:50.523 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.9369999999999998e-06 2024-09-13T11:22:50.523 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.523 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.523 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:22:50.523 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315644+0000", 2024-09-13T11:22:50.523 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000129011 2024-09-13T11:22:50.523 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.523 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.523 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:22:50.523 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315657+0000", 2024-09-13T11:22:50.523 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.3689999999999999e-05 2024-09-13T11:22:50.523 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.523 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.524 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:22:50.524 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315668+0000", 2024-09-13T11:22:50.524 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0589e-05 2024-09-13T11:22:50.524 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.524 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.524 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:22:50.524 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316925+0000", 2024-09-13T11:22:50.524 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0012568340000000001 2024-09-13T11:22:50.524 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.524 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.524 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-09-13T11:22:50.524 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316932+0000", 2024-09-13T11:22:50.524 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.2230000000000001e-06 2024-09-13T11:22:50.524 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.525 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.525 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:22:50.525 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.186619+0000", 2024-09-13T11:22:50.525 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.86968679599999998 2024-09-13T11:22:50.525 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.525 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.525 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:22:50.525 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.186914+0000", 2024-09-13T11:22:50.525 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00029528900000000001 2024-09-13T11:22:50.525 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.525 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.525 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1", 2024-09-13T11:22:50.525 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.187217+0000", 2024-09-13T11:22:50.526 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00030232099999999998 2024-09-13T11:22:50.526 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.526 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.526 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:22:50.526 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.190532+0000", 2024-09-13T11:22:50.526 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.003315391 2024-09-13T11:22:50.526 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.526 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.526 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:22:50.526 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.190875+0000", 2024-09-13T11:22:50.526 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00034326300000000001 2024-09-13T11:22:50.526 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.526 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.526 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:22:50.527 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.190925+0000", 2024-09-13T11:22:50.527 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.9429999999999999e-05 2024-09-13T11:22:50.527 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.527 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.527 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:22:50.527 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.190961+0000", 2024-09-13T11:22:50.527 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.5951000000000002e-05 2024-09-13T11:22:50.527 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:22:50.527 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:22:50.527 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:22:50.527 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.527 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.527 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:172 3.b 3:dbec1eeb:::benchmark_data_smithi177_31116_object171:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T11:22:50.527 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:48.315684+0000", 2024-09-13T11:22:50.527 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 2.1230011640000002, 2024-09-13T11:22:50.528 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.87615312999999995, 2024-09-13T11:22:50.528 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:22:50.528 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:22:50.528 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:50.528 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:22:50.528 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:22:50.528 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:22:50.528 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 172 2024-09-13T11:22:50.528 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.528 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:22:50.528 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.528 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:22:50.528 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315684+0000", 2024-09-13T11:22:50.529 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:22:50.529 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.529 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.529 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:22:50.529 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315684+0000", 2024-09-13T11:22:50.529 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:22:50.529 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.529 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.529 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:22:50.529 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315686+0000", 2024-09-13T11:22:50.529 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.886e-06 2024-09-13T11:22:50.529 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.529 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.529 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:22:50.530 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315820+0000", 2024-09-13T11:22:50.530 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00013388500000000001 2024-09-13T11:22:50.530 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.530 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.530 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:22:50.530 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315837+0000", 2024-09-13T11:22:50.530 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.6756e-05 2024-09-13T11:22:50.530 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.531 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.532 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:22:50.532 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315851+0000", 2024-09-13T11:22:50.532 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.4566000000000001e-05 2024-09-13T11:22:50.532 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.532 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.532 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:22:50.532 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.317021+0000", 2024-09-13T11:22:50.532 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0011701769999999999 2024-09-13T11:22:50.532 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.532 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.532 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-09-13T11:22:50.532 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.317037+0000", 2024-09-13T11:22:50.532 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.609e-05 2024-09-13T11:22:50.533 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.533 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.533 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:22:50.533 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.188634+0000", 2024-09-13T11:22:50.533 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.87159711900000003 2024-09-13T11:22:50.533 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.533 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.533 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:22:50.533 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.188974+0000", 2024-09-13T11:22:50.533 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00033922299999999999 2024-09-13T11:22:50.533 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.533 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.533 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1", 2024-09-13T11:22:50.534 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.189510+0000", 2024-09-13T11:22:50.534 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00053618999999999997 2024-09-13T11:22:50.534 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.534 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.534 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:22:50.534 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.191027+0000", 2024-09-13T11:22:50.534 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0015168390000000001 2024-09-13T11:22:50.534 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.534 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.534 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:22:50.534 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.191766+0000", 2024-09-13T11:22:50.534 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000739064 2024-09-13T11:22:50.534 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.534 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.535 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:22:50.535 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.191810+0000", 2024-09-13T11:22:50.535 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.3755999999999999e-05 2024-09-13T11:22:50.535 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.535 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.535 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:22:50.535 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.191837+0000", 2024-09-13T11:22:50.535 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.7579000000000001e-05 2024-09-13T11:22:50.535 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:22:50.535 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:22:50.535 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:22:50.535 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.535 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.535 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:177 3.b 3:de89c779:::benchmark_data_smithi177_31116_object176:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T11:22:50.536 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:48.315898+0000", 2024-09-13T11:22:50.536 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 2.1227868490000001, 2024-09-13T11:22:50.536 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.87600712599999997, 2024-09-13T11:22:50.536 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:22:50.536 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:22:50.536 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:50.536 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:22:50.536 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:22:50.536 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:22:50.536 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 177 2024-09-13T11:22:50.536 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.536 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:22:50.536 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.536 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:22:50.536 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315898+0000", 2024-09-13T11:22:50.537 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:22:50.537 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.537 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.537 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:22:50.537 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315898+0000", 2024-09-13T11:22:50.537 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:22:50.537 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.537 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.537 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:22:50.537 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315900+0000", 2024-09-13T11:22:50.537 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.4929999999999999e-06 2024-09-13T11:22:50.537 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.537 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.537 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:22:50.537 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316027+0000", 2024-09-13T11:22:50.538 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000127568 2024-09-13T11:22:50.538 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.538 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.538 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:22:50.538 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316048+0000", 2024-09-13T11:22:50.538 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.0795999999999999e-05 2024-09-13T11:22:50.538 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.538 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.538 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:22:50.538 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316061+0000", 2024-09-13T11:22:50.538 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.3283e-05 2024-09-13T11:22:50.538 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.538 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.538 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:22:50.539 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.317107+0000", 2024-09-13T11:22:50.539 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0010452949999999999 2024-09-13T11:22:50.539 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.539 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.539 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-09-13T11:22:50.539 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.317114+0000", 2024-09-13T11:22:50.539 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.4569999999999999e-06 2024-09-13T11:22:50.539 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.539 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.539 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:22:50.539 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.190017+0000", 2024-09-13T11:22:50.539 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.87290258499999995 2024-09-13T11:22:50.539 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.539 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.540 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:22:50.540 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.190138+0000", 2024-09-13T11:22:50.540 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000121418 2024-09-13T11:22:50.540 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.540 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.540 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1", 2024-09-13T11:22:50.540 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.190216+0000", 2024-09-13T11:22:50.540 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.7410999999999995e-05 2024-09-13T11:22:50.540 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.540 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.540 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:22:50.540 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.191755+0000", 2024-09-13T11:22:50.540 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0015395420000000001 2024-09-13T11:22:50.541 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.541 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.541 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:22:50.541 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.191843+0000", 2024-09-13T11:22:50.541 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.8022999999999994e-05 2024-09-13T11:22:50.541 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.541 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.541 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:22:50.541 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.191876+0000", 2024-09-13T11:22:50.541 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.2619000000000001e-05 2024-09-13T11:22:50.541 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.541 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.541 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:22:50.541 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.191905+0000", 2024-09-13T11:22:50.542 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.9635999999999999e-05 2024-09-13T11:22:50.542 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:22:50.542 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:22:50.542 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:22:50.542 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.542 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.542 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:187 3.b 3:da03c8dd:::benchmark_data_smithi177_31116_object186:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T11:22:50.542 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:48.316083+0000", 2024-09-13T11:22:50.542 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 2.1226018930000001, 2024-09-13T11:22:50.542 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.87835499500000003, 2024-09-13T11:22:50.542 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:22:50.542 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:22:50.542 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:50.542 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:22:50.543 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:22:50.543 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:22:50.543 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 187 2024-09-13T11:22:50.543 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.543 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:22:50.543 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.543 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:22:50.543 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316083+0000", 2024-09-13T11:22:50.543 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:22:50.543 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.543 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.543 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:22:50.543 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316083+0000", 2024-09-13T11:22:50.543 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:22:50.544 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.544 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.544 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:22:50.544 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316086+0000", 2024-09-13T11:22:50.544 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.5560000000000002e-06 2024-09-13T11:22:50.544 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.544 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.544 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:22:50.544 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316199+0000", 2024-09-13T11:22:50.544 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000113522 2024-09-13T11:22:50.544 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.544 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.544 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:22:50.544 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316215+0000", 2024-09-13T11:22:50.545 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.5265999999999999e-05 2024-09-13T11:22:50.545 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.545 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.545 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:22:50.545 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316226+0000", 2024-09-13T11:22:50.545 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0967e-05 2024-09-13T11:22:50.545 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.545 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.545 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:22:50.545 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.317198+0000", 2024-09-13T11:22:50.545 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00097194100000000004 2024-09-13T11:22:50.545 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.545 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.545 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-09-13T11:22:50.546 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.317213+0000", 2024-09-13T11:22:50.546 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.5472999999999999e-05 2024-09-13T11:22:50.546 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.546 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.546 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:22:50.546 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.191077+0000", 2024-09-13T11:22:50.546 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.87386435200000001 2024-09-13T11:22:50.546 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.546 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.546 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:22:50.546 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.191181+0000", 2024-09-13T11:22:50.546 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000103618 2024-09-13T11:22:50.546 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.546 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.547 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1", 2024-09-13T11:22:50.547 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.191305+0000", 2024-09-13T11:22:50.547 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00012358599999999999 2024-09-13T11:22:50.547 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.547 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.547 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:22:50.547 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.194279+0000", 2024-09-13T11:22:50.547 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0029746970000000001 2024-09-13T11:22:50.547 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.547 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.547 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:22:50.547 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.194336+0000", 2024-09-13T11:22:50.547 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.7244000000000003e-05 2024-09-13T11:22:50.548 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.548 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.548 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:22:50.548 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.194397+0000", 2024-09-13T11:22:50.548 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.0278000000000001e-05 2024-09-13T11:22:50.548 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.548 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.548 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:22:50.548 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.194438+0000", 2024-09-13T11:22:50.548 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.1495000000000002e-05 2024-09-13T11:22:50.548 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:22:50.548 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:22:50.548 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:22:50.548 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.549 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.549 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:200 3.b 3:d90039f1:::benchmark_data_smithi177_31116_object199:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T11:22:50.549 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:48.316280+0000", 2024-09-13T11:22:50.549 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 2.122405428, 2024-09-13T11:22:50.549 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.87879273000000002, 2024-09-13T11:22:50.549 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:22:50.549 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:22:50.549 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:50.549 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:22:50.549 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:22:50.549 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:22:50.549 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 200 2024-09-13T11:22:50.549 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.549 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:22:50.550 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.550 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:22:50.550 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316280+0000", 2024-09-13T11:22:50.550 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:22:50.550 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.550 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.550 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:22:50.550 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316280+0000", 2024-09-13T11:22:50.550 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:22:50.550 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.550 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.550 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:22:50.550 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316282+0000", 2024-09-13T11:22:50.550 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.497e-06 2024-09-13T11:22:50.550 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.551 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.551 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:22:50.551 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316411+0000", 2024-09-13T11:22:50.551 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00012924799999999999 2024-09-13T11:22:50.551 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.551 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.551 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:22:50.551 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316426+0000", 2024-09-13T11:22:50.551 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.4183e-05 2024-09-13T11:22:50.551 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.551 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.551 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:22:50.551 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316435+0000", 2024-09-13T11:22:50.552 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.0990000000000005e-06 2024-09-13T11:22:50.552 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.552 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.552 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:22:50.552 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.317250+0000", 2024-09-13T11:22:50.552 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00081501100000000003 2024-09-13T11:22:50.552 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.552 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.552 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-09-13T11:22:50.552 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.317257+0000", 2024-09-13T11:22:50.552 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.2169999999999997e-06 2024-09-13T11:22:50.552 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.552 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.552 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:22:50.553 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.191936+0000", 2024-09-13T11:22:50.553 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.87467922099999995 2024-09-13T11:22:50.553 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.553 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.553 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:22:50.553 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.192056+0000", 2024-09-13T11:22:50.553 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000119638 2024-09-13T11:22:50.553 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.553 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.553 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1", 2024-09-13T11:22:50.553 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.192184+0000", 2024-09-13T11:22:50.553 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00012813399999999999 2024-09-13T11:22:50.553 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.553 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.554 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:22:50.554 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.194324+0000", 2024-09-13T11:22:50.554 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0021399420000000001 2024-09-13T11:22:50.554 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.554 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.554 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:22:50.554 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.194802+0000", 2024-09-13T11:22:50.554 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00047798799999999999 2024-09-13T11:22:50.554 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.554 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.554 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:22:50.554 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.194951+0000", 2024-09-13T11:22:50.554 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0001494 2024-09-13T11:22:50.555 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.555 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.555 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:22:50.555 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.195072+0000", 2024-09-13T11:22:50.555 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00012115199999999999 2024-09-13T11:22:50.555 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:22:50.555 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:22:50.555 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:22:50.555 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.555 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.555 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:955 3.b 3:d0a0b8ee:::benchmark_data_smithi177_31116_object954:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-09-13T11:22:50.555 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:49.569434+0000", 2024-09-13T11:22:50.555 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 0.86925128200000001, 2024-09-13T11:22:50.555 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.044368339, 2024-09-13T11:22:50.556 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:22:50.556 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:22:50.556 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:50.556 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:22:50.556 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:22:50.556 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:22:50.556 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 955 2024-09-13T11:22:50.556 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.556 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:22:50.556 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.556 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:22:50.556 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.569434+0000", 2024-09-13T11:22:50.556 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:22:50.557 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.557 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.557 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:22:50.557 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.569434+0000", 2024-09-13T11:22:50.557 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:22:50.557 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.557 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.557 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:22:50.557 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.569436+0000", 2024-09-13T11:22:50.557 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.3719999999999999e-06 2024-09-13T11:22:50.557 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.557 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.557 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:22:50.557 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.569510+0000", 2024-09-13T11:22:50.558 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.4095000000000001e-05 2024-09-13T11:22:50.558 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.558 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.558 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:22:50.558 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.569524+0000", 2024-09-13T11:22:50.558 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.3312999999999999e-05 2024-09-13T11:22:50.558 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.558 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.558 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:22:50.558 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.569530+0000", 2024-09-13T11:22:50.558 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.6950000000000003e-06 2024-09-13T11:22:50.558 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.558 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.559 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:22:50.559 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.569596+0000", 2024-09-13T11:22:50.559 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.5294000000000002e-05 2024-09-13T11:22:50.559 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.559 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.559 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:22:50.559 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.569732+0000", 2024-09-13T11:22:50.559 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00013603199999999999 2024-09-13T11:22:50.559 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.559 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.559 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1", 2024-09-13T11:22:50.559 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.569883+0000", 2024-09-13T11:22:50.559 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00015128599999999999 2024-09-13T11:22:50.560 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.560 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.560 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:22:50.560 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.570566+0000", 2024-09-13T11:22:50.560 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00068274099999999995 2024-09-13T11:22:50.560 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.560 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.560 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:22:50.560 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.613737+0000", 2024-09-13T11:22:50.560 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.043171676999999999 2024-09-13T11:22:50.560 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.560 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.560 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:22:50.560 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.613774+0000", 2024-09-13T11:22:50.561 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.6461000000000002e-05 2024-09-13T11:22:50.561 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.561 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.561 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:22:50.561 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.613802+0000", 2024-09-13T11:22:50.561 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.8373000000000001e-05 2024-09-13T11:22:50.561 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:22:50.561 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:22:50.561 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:22:50.561 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.561 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.561 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:1203 3.6 3:6ecdd812:::benchmark_data_smithi177_31116_object1202:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-09-13T11:22:50.561 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:49.776223+0000", 2024-09-13T11:22:50.561 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 0.66246175600000001, 2024-09-13T11:22:50.562 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.048950868000000002, 2024-09-13T11:22:50.562 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:22:50.562 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:22:50.562 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:50.562 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:22:50.562 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:22:50.562 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:22:50.562 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 1203 2024-09-13T11:22:50.562 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.562 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:22:50.562 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.562 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:22:50.562 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.776223+0000", 2024-09-13T11:22:50.562 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:22:50.562 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.563 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.563 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:22:50.563 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.776223+0000", 2024-09-13T11:22:50.563 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:22:50.563 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.563 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.563 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:22:50.563 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.776225+0000", 2024-09-13T11:22:50.563 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.3829999999999999e-06 2024-09-13T11:22:50.563 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.563 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.563 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:22:50.563 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.776348+0000", 2024-09-13T11:22:50.564 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00012361499999999999 2024-09-13T11:22:50.564 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.564 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.564 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:22:50.564 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.776359+0000", 2024-09-13T11:22:50.564 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.1071e-05 2024-09-13T11:22:50.564 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.564 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.564 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:22:50.564 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.776367+0000", 2024-09-13T11:22:50.564 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.8639999999999996e-06 2024-09-13T11:22:50.564 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.564 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.564 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:22:50.565 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.776544+0000", 2024-09-13T11:22:50.565 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000176956 2024-09-13T11:22:50.565 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.565 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.565 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:22:50.565 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.776653+0000", 2024-09-13T11:22:50.565 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00010842 2024-09-13T11:22:50.565 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.565 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.565 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 14", 2024-09-13T11:22:50.565 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.776732+0000", 2024-09-13T11:22:50.565 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.9556999999999997e-05 2024-09-13T11:22:50.565 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.565 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.566 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:22:50.566 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.783781+0000", 2024-09-13T11:22:50.566 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0070488690000000001 2024-09-13T11:22:50.566 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.566 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.566 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:22:50.566 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.825064+0000", 2024-09-13T11:22:50.566 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.041283019999999997 2024-09-13T11:22:50.566 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.566 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.566 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:22:50.566 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.825128+0000", 2024-09-13T11:22:50.566 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.3410000000000004e-05 2024-09-13T11:22:50.567 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.567 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.567 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:22:50.567 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.825174+0000", 2024-09-13T11:22:50.567 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.6703000000000003e-05 2024-09-13T11:22:50.567 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:22:50.567 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:22:50.567 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:22:50.567 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.567 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.567 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:2101 3.6 3:6afe6785:::benchmark_data_smithi177_31116_object2100:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-09-13T11:22:50.567 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:50.188786+0000", 2024-09-13T11:22:50.567 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 0.24989942000000001, 2024-09-13T11:22:50.567 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.04420835, 2024-09-13T11:22:50.568 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:22:50.568 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:22:50.568 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:50.568 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:22:50.568 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:22:50.568 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:22:50.568 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 2101 2024-09-13T11:22:50.568 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.568 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:22:50.568 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.568 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:22:50.568 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:50.188786+0000", 2024-09-13T11:22:50.568 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:22:50.568 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.569 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.569 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:22:50.569 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:50.188786+0000", 2024-09-13T11:22:50.569 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:22:50.569 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.569 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.569 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:22:50.569 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:50.188787+0000", 2024-09-13T11:22:50.569 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.7749999999999999e-06 2024-09-13T11:22:50.569 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.569 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.569 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:22:50.569 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:50.188874+0000", 2024-09-13T11:22:50.570 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.6951000000000001e-05 2024-09-13T11:22:50.570 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.570 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.570 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:22:50.570 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:50.188885+0000", 2024-09-13T11:22:50.570 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0631e-05 2024-09-13T11:22:50.570 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.570 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.570 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:22:50.570 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:50.188893+0000", 2024-09-13T11:22:50.570 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.7950000000000008e-06 2024-09-13T11:22:50.570 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.570 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.570 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:22:50.570 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:50.188934+0000", 2024-09-13T11:22:50.571 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.1189999999999997e-05 2024-09-13T11:22:50.571 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.571 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.571 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:22:50.571 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:50.189016+0000", 2024-09-13T11:22:50.571 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.2134999999999999e-05 2024-09-13T11:22:50.571 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.571 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.571 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 14", 2024-09-13T11:22:50.571 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:50.189098+0000", 2024-09-13T11:22:50.572 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.2220999999999998e-05 2024-09-13T11:22:50.572 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.572 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.572 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:22:50.572 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:50.190608+0000", 2024-09-13T11:22:50.572 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.001509579 2024-09-13T11:22:50.572 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.572 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.572 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:22:50.572 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:50.232847+0000", 2024-09-13T11:22:50.572 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.042238734999999999 2024-09-13T11:22:50.572 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.572 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.572 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:22:50.573 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:50.232948+0000", 2024-09-13T11:22:50.573 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000101018 2024-09-13T11:22:50.573 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:50.573 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:50.573 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:22:50.573 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:50.232994+0000", 2024-09-13T11:22:50.573 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.6319999999999997e-05 2024-09-13T11:22:50.573 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:22:50.573 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:22:50.573 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:22:50.573 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:22:50.573 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:22:50.573 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-09-13T11:22:50.574 INFO:teuthology.orchestra.run.smithi001.stdout:osd.13: {} 2024-09-13T11:22:50.575 INFO:teuthology.orchestra.run.smithi001.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-13T11:22:50.575 INFO:teuthology.orchestra.run.smithi001.stdout:osd.14: {} 2024-09-13T11:22:50.576 INFO:teuthology.orchestra.run.smithi001.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-13T11:22:50.576 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-13T11:22:50.577 INFO:teuthology.orchestra.run.smithi001.stdout:osd.15: {} 2024-09-13T11:22:50.577 INFO:teuthology.orchestra.run.smithi001.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-13T11:22:50.578 INFO:tasks.ceph.osd.7.smithi177.stderr:2024-09-13T11:22:50.575+0000 7f897b51d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26575) UID: 0 2024-09-13T11:22:50.676 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-13T11:22:50.668+0000 7f6a68775640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26307) UID: 0 2024-09-13T11:22:50.730 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-13T11:22:50.730 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:22:50.747 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-13T11:22:50.793 INFO:tasks.ceph.osd.10.smithi162.stderr:2024-09-13T11:22:50.791+0000 7fb59c9f0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26317) UID: 0 2024-09-13T11:22:50.877 INFO:tasks.ceph.osd.7.smithi177.stderr:2024-09-13T11:22:50.875+0000 7f897b51d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26575) UID: 0 2024-09-13T11:22:50.890 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-13T11:22:50.890 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:22:50.910 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-13T11:22:50.977 INFO:tasks.ceph.osd.13.smithi092.stderr:2024-09-13T11:22:50.975+0000 7f3e5afea640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26350) UID: 0 2024-09-13T11:22:50.999 INFO:tasks.radosbench.radosbench.0.smithi177.stdout: 24 15 4179 4164 10.8423 162.125 0.00297317 0.0922083 2024-09-13T11:22:51.079 INFO:tasks.ceph.osd.4.smithi001.stderr:2024-09-13T11:22:51.072+0000 7ff8e0218640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26310) UID: 0 2024-09-13T11:22:51.095 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-13T11:22:51.096 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:22:51.113 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-13T11:22:51.180 INFO:tasks.ceph.osd.11.smithi177.stderr:2024-09-13T11:22:51.179+0000 7f24792c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26580) UID: 0 2024-09-13T11:22:51.246 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-13T11:22:51.246 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:22:51.263 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-13T11:22:51.280 INFO:tasks.ceph.osd.1.smithi092.stderr:2024-09-13T11:22:51.279+0000 7fe7789c5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33718) UID: 0 2024-09-13T11:22:51.338 INFO:teuthology.orchestra.run.smithi001.stderr:reweighted osd.5 to 1 (10000) 2024-09-13T11:22:51.338 INFO:tasks.ceph.osd.12.smithi001.stderr:2024-09-13T11:22:51.332+0000 7f891f10f640 -1 osd.12 44 handle_osd_map injecting map crc failure 2024-09-13T11:22:51.339 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-13T11:22:51.369 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 6 1 2024-09-13T11:22:51.381 INFO:tasks.ceph.osd.5.smithi092.stderr:2024-09-13T11:22:51.379+0000 7f6f0b705640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26346) UID: 0 2024-09-13T11:22:51.395 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-13T11:22:51.395 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:22:51.411 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-13T11:22:51.481 INFO:tasks.ceph.osd.10.smithi162.stderr:2024-09-13T11:22:51.479+0000 7fb59c9f0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26317) UID: 0 2024-09-13T11:22:51.541 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-13T11:22:51.542 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:22:51.557 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-09-13T11:22:51.601 INFO:tasks.ceph.osd.2.smithi162.stderr:2024-09-13T11:22:51.599+0000 7fe92b35a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26313) UID: 0 2024-09-13T11:22:51.684 INFO:tasks.ceph.osd.7.smithi177.stderr:2024-09-13T11:22:51.683+0000 7f897b51d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26575) UID: 0 2024-09-13T11:22:51.727 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-13T11:22:51.727 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:22:51.744 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-09-13T11:22:51.789 INFO:tasks.ceph.osd.10.smithi162.stderr:2024-09-13T11:22:51.787+0000 7fb59c9f0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26317) UID: 0 2024-09-13T11:22:51.873 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-13T11:22:51.874 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:22:51.886 INFO:tasks.ceph.osd.5.smithi092.stderr:2024-09-13T11:22:51.883+0000 7f6f0b705640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26346) UID: 0 2024-09-13T11:22:51.890 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-09-13T11:22:51.987 INFO:tasks.ceph.osd.10.smithi162.stderr:2024-09-13T11:22:51.983+0000 7fb59c9f0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26317) UID: 0 2024-09-13T11:22:51.999 INFO:tasks.radosbench.radosbench.0.smithi177.stdout: 25 16 5747 5731 14.3256 97.9375 0.00274823 0.0697683 2024-09-13T11:22:52.072 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-13T11:22:52.072 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:22:52.089 INFO:tasks.ceph.osd.8.smithi001.stderr:2024-09-13T11:22:52.084+0000 7fb9b48ae640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26312) UID: 0 2024-09-13T11:22:52.089 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_ops_in_flight 2024-09-13T11:22:52.189 INFO:tasks.ceph.osd.7.smithi177.stderr:2024-09-13T11:22:52.187+0000 7f897b51d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26575) UID: 0 2024-09-13T11:22:52.276 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-13T11:22:52.276 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:22:52.290 INFO:tasks.ceph.osd.4.smithi001.stderr:2024-09-13T11:22:52.284+0000 7ff8e0218640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26310) UID: 0 2024-09-13T11:22:52.293 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_blocked_ops 2024-09-13T11:22:52.339 INFO:teuthology.orchestra.run.smithi001.stderr:reweighted osd.6 to 1 (10000) 2024-09-13T11:22:52.340 INFO:teuthology.orchestra.run.smithi001.stderr:nodeep-scrub is set 2024-09-13T11:22:52.341 INFO:tasks.ceph.osd.13.smithi092.stderr:2024-09-13T11:22:52.339+0000 7f3e4da13640 -1 osd.13 45 handle_osd_map injecting map crc failure 2024-09-13T11:22:52.370 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 7 1 2024-09-13T11:22:52.390 INFO:tasks.ceph.osd.11.smithi177.stderr:2024-09-13T11:22:52.391+0000 7f24792c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26580) UID: 0 2024-09-13T11:22:52.427 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-13T11:22:52.427 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:22:52.444 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_historic_ops 2024-09-13T11:22:52.491 INFO:tasks.ceph.osd.12.smithi001.stderr:2024-09-13T11:22:52.484+0000 7f892c6e6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26314) UID: 0 2024-09-13T11:22:52.579 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T11:22:52.587 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-13T11:22:52.587 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:22:52.592 INFO:tasks.ceph.osd.10.smithi162.stderr:2024-09-13T11:22:52.587+0000 7fb59c9f0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26317) UID: 0 2024-09-13T11:22:52.605 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-13T11:22:52.694 INFO:tasks.ceph.osd.5.smithi092.stderr:2024-09-13T11:22:52.691+0000 7f6f0b705640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26346) UID: 0 2024-09-13T11:22:52.751 INFO:tasks.ceph.osd.9.smithi092.stderr:2024-09-13T11:22:52.747+0000 7ff59bdf5640 -1 osd.9 45 handle_osd_map injecting map crc failure 2024-09-13T11:22:52.752 INFO:teuthology.orchestra.run.smithi001.stdout:osd.0: {} 2024-09-13T11:22:52.752 INFO:teuthology.orchestra.run.smithi001.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T11:22:52.757 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-09-13T11:22:52.757 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:22:52.775 INFO:teuthology.orchestra.run.smithi001.stdout:osd.1: {} 2024-09-13T11:22:52.775 INFO:teuthology.orchestra.run.smithi001.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T11:22:52.776 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-13T11:22:52.795 INFO:tasks.ceph.osd.12.smithi001.stderr:2024-09-13T11:22:52.788+0000 7f892c6e6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26314) UID: 0 2024-09-13T11:22:52.803 INFO:teuthology.orchestra.run.smithi001.stdout:osd.2: {} 2024-09-13T11:22:52.803 INFO:teuthology.orchestra.run.smithi001.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T11:22:52.829 INFO:teuthology.orchestra.run.smithi001.stdout:osd.3: {} 2024-09-13T11:22:52.829 INFO:teuthology.orchestra.run.smithi001.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T11:22:52.855 INFO:teuthology.orchestra.run.smithi001.stdout:osd.4: {} 2024-09-13T11:22:52.855 INFO:teuthology.orchestra.run.smithi001.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T11:22:52.882 INFO:teuthology.orchestra.run.smithi001.stdout:osd.5: {} 2024-09-13T11:22:52.882 INFO:teuthology.orchestra.run.smithi001.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T11:22:52.895 INFO:tasks.ceph.osd.3.smithi177.stderr:2024-09-13T11:22:52.895+0000 7f10adba4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26574) UID: 0 2024-09-13T11:22:52.912 INFO:teuthology.orchestra.run.smithi001.stdout:osd.6: {} 2024-09-13T11:22:52.913 INFO:teuthology.orchestra.run.smithi001.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T11:22:52.925 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-09-13T11:22:52.925 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [], 2024-09-13T11:22:52.925 INFO:teuthology.orchestra.run.smithi177.stdout: "complaint_time": 30, 2024-09-13T11:22:52.925 INFO:teuthology.orchestra.run.smithi177.stdout: "num_blocked_ops": 0 2024-09-13T11:22:52.925 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-09-13T11:22:52.935 INFO:teuthology.orchestra.run.smithi001.stdout:osd.7: {} 2024-09-13T11:22:52.935 INFO:teuthology.orchestra.run.smithi001.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T11:22:52.945 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-13T11:22:52.957 INFO:teuthology.orchestra.run.smithi001.stdout:osd.8: {} 2024-09-13T11:22:52.958 INFO:teuthology.orchestra.run.smithi001.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-13T11:22:52.983 INFO:teuthology.orchestra.run.smithi001.stdout:osd.9: {} 2024-09-13T11:22:52.983 INFO:teuthology.orchestra.run.smithi001.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-13T11:22:52.996 INFO:tasks.ceph.osd.5.smithi092.stderr:2024-09-13T11:22:52.994+0000 7f6f0b705640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26346) UID: 0 2024-09-13T11:22:52.999 INFO:tasks.radosbench.radosbench.0.smithi177.stdout: 26 16 7636 7620 18.3148 118.062 0.00349971 0.0545775 2024-09-13T11:22:53.012 INFO:teuthology.orchestra.run.smithi001.stdout:osd.10: {} 2024-09-13T11:22:53.012 INFO:teuthology.orchestra.run.smithi001.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-13T11:22:53.037 INFO:teuthology.orchestra.run.smithi001.stdout:osd.11: {} 2024-09-13T11:22:53.038 INFO:teuthology.orchestra.run.smithi001.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-13T11:22:53.063 INFO:teuthology.orchestra.run.smithi001.stdout:osd.12: {} 2024-09-13T11:22:53.063 INFO:teuthology.orchestra.run.smithi001.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-13T11:22:53.092 INFO:teuthology.orchestra.run.smithi001.stdout:osd.13: {} 2024-09-13T11:22:53.092 INFO:teuthology.orchestra.run.smithi001.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-13T11:22:53.097 INFO:tasks.ceph.osd.4.smithi001.stderr:2024-09-13T11:22:53.092+0000 7ff8e0218640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26310) UID: 0 2024-09-13T11:22:53.121 INFO:teuthology.orchestra.run.smithi001.stdout:osd.14: {} 2024-09-13T11:22:53.121 INFO:teuthology.orchestra.run.smithi001.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-13T11:22:53.144 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-09-13T11:22:53.144 INFO:teuthology.orchestra.run.smithi177.stdout: "size": 20, 2024-09-13T11:22:53.144 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 600, 2024-09-13T11:22:53.144 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [ 2024-09-13T11:22:53.144 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.144 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:831 3.a 3:5e26034b:::benchmark_data_smithi177_31116_object830:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-09-13T11:22:53.144 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:22:49.458901+0000", 2024-09-13T11:22:53.144 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 3.684292551, 2024-09-13T11:22:53.144 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.036200677000000001, 2024-09-13T11:22:53.144 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:22:53.144 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:22:53.144 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:53.145 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:22:53.145 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:22:53.145 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:22:53.145 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 831 2024-09-13T11:22:53.145 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.145 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:22:53.145 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.145 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:22:53.145 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.458901+0000", 2024-09-13T11:22:53.145 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:53.145 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.145 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.145 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:22:53.146 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.458901+0000", 2024-09-13T11:22:53.146 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:53.146 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.146 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.146 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:22:53.146 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.458903+0000", 2024-09-13T11:22:53.146 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.6270000000000001e-06 2024-09-13T11:22:53.146 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.146 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.146 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:22:53.146 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.458929+0000", 2024-09-13T11:22:53.146 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.6342999999999998e-05 2024-09-13T11:22:53.146 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.147 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.147 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:22:53.147 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.458942+0000", 2024-09-13T11:22:53.147 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.2349e-05 2024-09-13T11:22:53.147 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.147 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.147 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:22:53.147 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.458948+0000", 2024-09-13T11:22:53.147 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.635e-06 2024-09-13T11:22:53.147 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.147 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.147 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:22:53.147 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.459000+0000", 2024-09-13T11:22:53.147 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.1474000000000002e-05 2024-09-13T11:22:53.148 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.148 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.148 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:22:53.148 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.459098+0000", 2024-09-13T11:22:53.148 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.8412000000000006e-05 2024-09-13T11:22:53.148 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.148 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.148 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 7", 2024-09-13T11:22:53.148 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.459191+0000", 2024-09-13T11:22:53.148 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.2720999999999995e-05 2024-09-13T11:22:53.148 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.148 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.148 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:22:53.148 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.459920+0000", 2024-09-13T11:22:53.148 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00072961 2024-09-13T11:22:53.149 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.149 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.149 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:22:53.149 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.495017+0000", 2024-09-13T11:22:53.149 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.035096384000000001 2024-09-13T11:22:53.149 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.149 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.149 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:22:53.149 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.495065+0000", 2024-09-13T11:22:53.149 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.7956e-05 2024-09-13T11:22:53.149 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.149 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.149 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:22:53.149 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.495102+0000", 2024-09-13T11:22:53.150 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.7166000000000003e-05 2024-09-13T11:22:53.150 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:53.150 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:22:53.150 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:53.150 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.150 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.150 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:837 3.a 3:5ab13ab0:::benchmark_data_smithi177_31116_object836:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-09-13T11:22:53.150 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:22:49.461597+0000", 2024-09-13T11:22:53.150 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 3.6815970500000001, 2024-09-13T11:22:53.150 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.044395994000000001, 2024-09-13T11:22:53.150 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:22:53.150 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:22:53.150 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:53.150 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:22:53.151 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:22:53.151 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:22:53.151 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 837 2024-09-13T11:22:53.151 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.151 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:22:53.151 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.151 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:22:53.151 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.461597+0000", 2024-09-13T11:22:53.151 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:53.151 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.151 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.151 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:22:53.151 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.461597+0000", 2024-09-13T11:22:53.152 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:53.152 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.152 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.152 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:22:53.152 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.461598+0000", 2024-09-13T11:22:53.152 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.44e-06 2024-09-13T11:22:53.152 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.152 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.152 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:22:53.152 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.461634+0000", 2024-09-13T11:22:53.152 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.5673999999999998e-05 2024-09-13T11:22:53.152 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.152 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.153 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:22:53.153 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.461648+0000", 2024-09-13T11:22:53.153 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.4047e-05 2024-09-13T11:22:53.153 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.153 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.153 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:22:53.153 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.461655+0000", 2024-09-13T11:22:53.153 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.9720000000000003e-06 2024-09-13T11:22:53.153 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.153 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.153 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:22:53.153 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.461715+0000", 2024-09-13T11:22:53.153 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.0195999999999998e-05 2024-09-13T11:22:53.154 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.154 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.154 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:22:53.154 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.461819+0000", 2024-09-13T11:22:53.154 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000103447 2024-09-13T11:22:53.154 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.154 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.154 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 7", 2024-09-13T11:22:53.154 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.461939+0000", 2024-09-13T11:22:53.154 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000120913 2024-09-13T11:22:53.154 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.154 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.154 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:22:53.154 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.462669+0000", 2024-09-13T11:22:53.155 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00072908200000000004 2024-09-13T11:22:53.155 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.155 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.155 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:22:53.155 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.505895+0000", 2024-09-13T11:22:53.155 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.043226597999999998 2024-09-13T11:22:53.155 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.155 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.155 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:22:53.155 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.505941+0000", 2024-09-13T11:22:53.155 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.5503999999999997e-05 2024-09-13T11:22:53.155 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.155 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.156 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:22:53.156 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.505993+0000", 2024-09-13T11:22:53.156 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.2120999999999998e-05 2024-09-13T11:22:53.156 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:53.156 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:22:53.156 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:53.156 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.156 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.156 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop_reply(client.4665.0:837 3.a e43/30 ondisk, result = 0)", 2024-09-13T11:22:53.156 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:22:49.463026+0000", 2024-09-13T11:22:53.156 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 3.6801676510000001, 2024-09-13T11:22:53.156 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.042975286000000001, 2024-09-13T11:22:53.156 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:22:53.156 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:22:53.157 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:22:53.157 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:22:53.157 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.157 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:22:53.157 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.463026+0000", 2024-09-13T11:22:53.157 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:53.157 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.157 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.157 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:22:53.157 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.463026+0000", 2024-09-13T11:22:53.157 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:53.157 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.157 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.158 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:22:53.158 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.463028+0000", 2024-09-13T11:22:53.158 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.443e-06 2024-09-13T11:22:53.158 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.158 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.158 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:22:53.158 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.463033+0000", 2024-09-13T11:22:53.158 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.7830000000000004e-06 2024-09-13T11:22:53.158 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.158 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.158 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:22:53.158 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.505813+0000", 2024-09-13T11:22:53.158 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.042779620999999997 2024-09-13T11:22:53.159 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.159 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.159 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:22:53.159 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.505820+0000", 2024-09-13T11:22:53.159 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.233e-06 2024-09-13T11:22:53.159 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.159 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.159 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:22:53.159 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.505881+0000", 2024-09-13T11:22:53.159 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.0810000000000002e-05 2024-09-13T11:22:53.159 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.159 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.159 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:22:53.159 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.505891+0000", 2024-09-13T11:22:53.160 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.6059999999999992e-06 2024-09-13T11:22:53.160 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.160 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.160 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:22:53.160 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.506001+0000", 2024-09-13T11:22:53.160 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00011079 2024-09-13T11:22:53.160 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:53.160 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:22:53.160 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:53.160 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.160 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.161 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:855 3.a 3:5ce42657:::benchmark_data_smithi177_31116_object854:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-09-13T11:22:53.161 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:22:49.480636+0000", 2024-09-13T11:22:53.161 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 3.662557456, 2024-09-13T11:22:53.161 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.040802389000000001, 2024-09-13T11:22:53.161 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:22:53.161 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:22:53.161 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:53.161 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:22:53.161 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:22:53.161 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:22:53.161 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 855 2024-09-13T11:22:53.161 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.161 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:22:53.162 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.162 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:22:53.162 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.480636+0000", 2024-09-13T11:22:53.162 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:53.162 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.162 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.162 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:22:53.162 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.480636+0000", 2024-09-13T11:22:53.162 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:53.162 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.162 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.163 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:22:53.163 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.480640+0000", 2024-09-13T11:22:53.163 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.0929999999999996e-06 2024-09-13T11:22:53.163 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.163 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.163 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:22:53.163 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.480686+0000", 2024-09-13T11:22:53.163 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.5629e-05 2024-09-13T11:22:53.163 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.163 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.163 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:22:53.163 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.480713+0000", 2024-09-13T11:22:53.163 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.7021000000000001e-05 2024-09-13T11:22:53.164 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.164 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.164 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:22:53.164 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.480730+0000", 2024-09-13T11:22:53.164 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.6835000000000001e-05 2024-09-13T11:22:53.164 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.164 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.164 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:22:53.164 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.480815+0000", 2024-09-13T11:22:53.164 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.4888000000000004e-05 2024-09-13T11:22:53.164 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.164 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.164 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:22:53.165 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.480940+0000", 2024-09-13T11:22:53.165 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00012519000000000001 2024-09-13T11:22:53.165 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.165 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.165 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 7", 2024-09-13T11:22:53.165 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.481056+0000", 2024-09-13T11:22:53.165 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000115467 2024-09-13T11:22:53.165 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.165 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.165 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:22:53.165 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.482076+0000", 2024-09-13T11:22:53.166 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001020038 2024-09-13T11:22:53.166 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.166 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.166 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:22:53.166 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.521353+0000", 2024-09-13T11:22:53.166 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.039277764 2024-09-13T11:22:53.166 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.166 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.166 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:22:53.166 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.521402+0000", 2024-09-13T11:22:53.166 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.8408000000000001e-05 2024-09-13T11:22:53.166 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.166 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.167 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:22:53.167 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.521439+0000", 2024-09-13T11:22:53.167 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.7055999999999999e-05 2024-09-13T11:22:53.167 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:53.167 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:22:53.167 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:53.167 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.167 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.167 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop_reply(client.4665.0:855 3.a e43/30 ondisk, result = 0)", 2024-09-13T11:22:53.167 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:22:49.482718+0000", 2024-09-13T11:22:53.167 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 3.6604760880000002, 2024-09-13T11:22:53.167 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.038728772000000002, 2024-09-13T11:22:53.167 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:22:53.168 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:22:53.168 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:22:53.168 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:22:53.168 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.168 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:22:53.168 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.482718+0000", 2024-09-13T11:22:53.168 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:53.168 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.168 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.168 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:22:53.168 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.482718+0000", 2024-09-13T11:22:53.168 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:53.168 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.169 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.169 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:22:53.169 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.482719+0000", 2024-09-13T11:22:53.169 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.7269999999999999e-06 2024-09-13T11:22:53.169 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.169 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.169 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:22:53.169 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.482725+0000", 2024-09-13T11:22:53.169 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.3650000000000003e-06 2024-09-13T11:22:53.169 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.169 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.169 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:22:53.169 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.521273+0000", 2024-09-13T11:22:53.169 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.038548495000000002 2024-09-13T11:22:53.169 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.170 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.170 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:22:53.170 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.521280+0000", 2024-09-13T11:22:53.170 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.7120000000000003e-06 2024-09-13T11:22:53.170 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.170 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.170 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:22:53.170 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.521341+0000", 2024-09-13T11:22:53.170 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.0693000000000003e-05 2024-09-13T11:22:53.170 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.170 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.170 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:22:53.170 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.521349+0000", 2024-09-13T11:22:53.170 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.174e-06 2024-09-13T11:22:53.170 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.171 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.171 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:22:53.171 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.521447+0000", 2024-09-13T11:22:53.171 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.7606e-05 2024-09-13T11:22:53.171 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:53.171 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:22:53.171 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:53.171 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.171 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.171 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:861 3.a 3:5772350b:::benchmark_data_smithi177_31116_object860:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-09-13T11:22:53.171 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:22:49.487183+0000", 2024-09-13T11:22:53.171 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 3.6560103809999998, 2024-09-13T11:22:53.171 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.042583651, 2024-09-13T11:22:53.171 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:22:53.171 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:22:53.172 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:53.172 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:22:53.172 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:22:53.172 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:22:53.172 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 861 2024-09-13T11:22:53.172 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.172 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:22:53.172 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.172 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:22:53.172 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.487183+0000", 2024-09-13T11:22:53.172 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:53.172 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.172 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.172 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:22:53.173 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.487183+0000", 2024-09-13T11:22:53.173 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:53.173 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.173 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.173 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:22:53.173 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.487186+0000", 2024-09-13T11:22:53.173 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.5409999999999999e-06 2024-09-13T11:22:53.173 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.173 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.173 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:22:53.173 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.487222+0000", 2024-09-13T11:22:53.173 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.5621000000000003e-05 2024-09-13T11:22:53.173 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.174 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.174 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:22:53.174 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.487243+0000", 2024-09-13T11:22:53.174 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.0902000000000001e-05 2024-09-13T11:22:53.174 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.174 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.174 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:22:53.174 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.487253+0000", 2024-09-13T11:22:53.174 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0667e-05 2024-09-13T11:22:53.174 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.174 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.174 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:22:53.175 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.487304+0000", 2024-09-13T11:22:53.175 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.0658000000000002e-05 2024-09-13T11:22:53.175 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.175 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.175 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:22:53.175 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.487392+0000", 2024-09-13T11:22:53.175 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.7824999999999993e-05 2024-09-13T11:22:53.175 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.175 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.175 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 7", 2024-09-13T11:22:53.175 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.487483+0000", 2024-09-13T11:22:53.175 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.0915999999999995e-05 2024-09-13T11:22:53.175 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.175 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.176 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:22:53.176 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.488424+0000", 2024-09-13T11:22:53.176 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00094114899999999996 2024-09-13T11:22:53.176 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.176 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.176 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:22:53.176 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.529663+0000", 2024-09-13T11:22:53.176 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.041239215000000003 2024-09-13T11:22:53.176 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.176 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.176 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:22:53.176 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.529719+0000", 2024-09-13T11:22:53.176 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.6480999999999997e-05 2024-09-13T11:22:53.177 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.177 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.177 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:22:53.177 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.529767+0000", 2024-09-13T11:22:53.177 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.7676000000000002e-05 2024-09-13T11:22:53.177 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:53.177 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:22:53.177 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:53.177 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.177 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.177 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop_reply(client.4665.0:861 3.a e43/30 ondisk, result = 0)", 2024-09-13T11:22:53.177 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:22:49.488684+0000", 2024-09-13T11:22:53.177 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 3.654509929, 2024-09-13T11:22:53.178 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.041092947999999997, 2024-09-13T11:22:53.178 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:22:53.178 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:22:53.178 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:22:53.178 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:22:53.178 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.178 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:22:53.178 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.488684+0000", 2024-09-13T11:22:53.178 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:53.178 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.178 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.178 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:22:53.178 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.488684+0000", 2024-09-13T11:22:53.179 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:53.179 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.179 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.179 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:22:53.179 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.488685+0000", 2024-09-13T11:22:53.179 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.4470000000000001e-06 2024-09-13T11:22:53.179 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.179 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.179 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:22:53.179 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.488690+0000", 2024-09-13T11:22:53.179 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.8520000000000003e-06 2024-09-13T11:22:53.179 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.179 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.179 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:22:53.180 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.529557+0000", 2024-09-13T11:22:53.180 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.040866524000000001 2024-09-13T11:22:53.180 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.180 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.180 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:22:53.180 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.529567+0000", 2024-09-13T11:22:53.180 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.011e-05 2024-09-13T11:22:53.180 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.180 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.180 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:22:53.180 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.529649+0000", 2024-09-13T11:22:53.180 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.2280000000000005e-05 2024-09-13T11:22:53.180 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.181 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.181 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:22:53.181 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.529658+0000", 2024-09-13T11:22:53.181 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.3549999999999994e-06 2024-09-13T11:22:53.181 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.181 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.181 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:22:53.181 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.529777+0000", 2024-09-13T11:22:53.181 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00011838 2024-09-13T11:22:53.181 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:53.184 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:22:53.184 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:53.184 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.184 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.184 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:867 3.a 3:5ca59689:::benchmark_data_smithi177_31116_object866:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-09-13T11:22:53.184 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:22:49.493530+0000", 2024-09-13T11:22:53.184 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 3.6496639370000001, 2024-09-13T11:22:53.184 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.038165363000000001, 2024-09-13T11:22:53.184 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:22:53.184 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:22:53.184 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:53.184 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:22:53.184 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:22:53.185 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:22:53.185 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 867 2024-09-13T11:22:53.185 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.185 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:22:53.185 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.185 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:22:53.185 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.493530+0000", 2024-09-13T11:22:53.185 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:53.185 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.185 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.185 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:22:53.185 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.493530+0000", 2024-09-13T11:22:53.185 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:53.186 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.186 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.186 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:22:53.186 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.493532+0000", 2024-09-13T11:22:53.186 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.5459999999999998e-06 2024-09-13T11:22:53.186 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.186 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.186 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:22:53.186 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.493568+0000", 2024-09-13T11:22:53.186 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.5354000000000001e-05 2024-09-13T11:22:53.186 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.186 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.186 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:22:53.186 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.493595+0000", 2024-09-13T11:22:53.187 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.6823999999999999e-05 2024-09-13T11:22:53.187 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.187 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.187 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:22:53.187 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.493606+0000", 2024-09-13T11:22:53.187 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.1045e-05 2024-09-13T11:22:53.187 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.187 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.187 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:22:53.187 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.493676+0000", 2024-09-13T11:22:53.187 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.0007999999999998e-05 2024-09-13T11:22:53.187 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.187 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.188 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:22:53.188 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.493792+0000", 2024-09-13T11:22:53.188 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00011584800000000001 2024-09-13T11:22:53.188 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.188 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.188 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 7", 2024-09-13T11:22:53.188 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.493926+0000", 2024-09-13T11:22:53.188 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00013482199999999999 2024-09-13T11:22:53.188 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.188 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.188 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:22:53.188 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.494881+0000", 2024-09-13T11:22:53.188 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00095461899999999998 2024-09-13T11:22:53.189 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.189 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.189 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:22:53.189 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.531625+0000", 2024-09-13T11:22:53.189 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.036744106999999998 2024-09-13T11:22:53.189 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.189 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.189 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:22:53.189 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.531663+0000", 2024-09-13T11:22:53.189 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.7962e-05 2024-09-13T11:22:53.189 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.189 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.189 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:22:53.190 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.531695+0000", 2024-09-13T11:22:53.190 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.2227999999999997e-05 2024-09-13T11:22:53.190 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:53.190 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:22:53.190 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:53.190 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.190 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.190 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop_reply(client.4665.0:867 3.a e43/30 ondisk, result = 0)", 2024-09-13T11:22:53.190 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:22:49.494924+0000", 2024-09-13T11:22:53.190 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 3.6482703340000002, 2024-09-13T11:22:53.190 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.036777604999999998, 2024-09-13T11:22:53.190 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:22:53.191 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:22:53.191 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:22:53.191 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:22:53.191 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.191 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:22:53.191 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.494924+0000", 2024-09-13T11:22:53.191 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:53.191 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.191 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.191 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:22:53.191 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.494924+0000", 2024-09-13T11:22:53.191 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:53.191 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.192 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.192 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:22:53.192 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.494925+0000", 2024-09-13T11:22:53.192 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.7129999999999999e-06 2024-09-13T11:22:53.192 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.192 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.192 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:22:53.192 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.494930+0000", 2024-09-13T11:22:53.192 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.7149999999999997e-06 2024-09-13T11:22:53.192 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.192 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.192 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:22:53.192 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.531545+0000", 2024-09-13T11:22:53.193 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.036615279000000001 2024-09-13T11:22:53.193 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.193 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.193 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:22:53.193 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.531552+0000", 2024-09-13T11:22:53.193 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.4150000000000004e-06 2024-09-13T11:22:53.193 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.193 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.193 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:22:53.193 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.531608+0000", 2024-09-13T11:22:53.193 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.6570000000000002e-05 2024-09-13T11:22:53.193 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.193 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.194 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:22:53.194 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.531619+0000", 2024-09-13T11:22:53.194 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.047e-05 2024-09-13T11:22:53.194 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.194 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.194 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:22:53.194 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.531701+0000", 2024-09-13T11:22:53.194 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.2442999999999997e-05 2024-09-13T11:22:53.194 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:53.194 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:22:53.194 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:53.194 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.194 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.194 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:876 3.d e43/30 3:bbe42dec:::benchmark_data_smithi177_31116_object875:head v 43'52, mlcod=43'50)", 2024-09-13T11:22:53.194 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:22:49.505759+0000", 2024-09-13T11:22:53.195 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 3.6374349079999999, 2024-09-13T11:22:53.195 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.037170129000000003, 2024-09-13T11:22:53.195 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:22:53.195 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:22:53.195 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:53.195 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:22:53.195 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.195 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:22:53.195 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.505759+0000", 2024-09-13T11:22:53.195 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:53.195 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.195 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.195 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:22:53.196 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.505759+0000", 2024-09-13T11:22:53.196 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:53.196 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.196 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.196 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:22:53.196 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.505761+0000", 2024-09-13T11:22:53.196 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.0389999999999999e-06 2024-09-13T11:22:53.196 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.196 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.196 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:22:53.196 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.505793+0000", 2024-09-13T11:22:53.196 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.1625999999999998e-05 2024-09-13T11:22:53.196 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.197 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.197 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:22:53.197 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.541869+0000", 2024-09-13T11:22:53.197 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.036076229000000001 2024-09-13T11:22:53.197 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.197 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.197 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:22:53.197 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.541875+0000", 2024-09-13T11:22:53.197 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.2689999999999996e-06 2024-09-13T11:22:53.197 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.197 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.197 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:22:53.197 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.541947+0000", 2024-09-13T11:22:53.197 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.1531000000000003e-05 2024-09-13T11:22:53.198 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.198 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.198 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:22:53.198 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.541967+0000", 2024-09-13T11:22:53.198 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.0835e-05 2024-09-13T11:22:53.198 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.198 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.198 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:22:53.198 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.542886+0000", 2024-09-13T11:22:53.198 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00091859500000000002 2024-09-13T11:22:53.198 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.198 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.199 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:22:53.199 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.542929+0000", 2024-09-13T11:22:53.199 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.3004999999999999e-05 2024-09-13T11:22:53.199 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:53.199 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:22:53.199 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:53.199 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.199 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.199 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:990 3.a 3:508c4f42:::benchmark_data_smithi177_31116_object989:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-09-13T11:22:53.199 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:22:49.601450+0000", 2024-09-13T11:22:53.199 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 3.541743512, 2024-09-13T11:22:53.199 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.036326521000000001, 2024-09-13T11:22:53.199 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:22:53.199 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:22:53.200 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:53.200 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:22:53.200 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:22:53.200 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:22:53.200 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 990 2024-09-13T11:22:53.200 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.200 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:22:53.200 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.200 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:22:53.200 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.601450+0000", 2024-09-13T11:22:53.200 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:53.200 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.200 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.201 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:22:53.201 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.601450+0000", 2024-09-13T11:22:53.201 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:53.201 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.201 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.201 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:22:53.201 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.601452+0000", 2024-09-13T11:22:53.201 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.0700000000000001e-06 2024-09-13T11:22:53.201 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.201 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.201 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:22:53.201 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.601499+0000", 2024-09-13T11:22:53.201 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.6282999999999997e-05 2024-09-13T11:22:53.201 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.202 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.202 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:22:53.202 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.601513+0000", 2024-09-13T11:22:53.202 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.4209000000000001e-05 2024-09-13T11:22:53.202 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.202 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.202 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:22:53.202 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.601521+0000", 2024-09-13T11:22:53.202 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.9370000000000004e-06 2024-09-13T11:22:53.202 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.202 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.202 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:22:53.202 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.601582+0000", 2024-09-13T11:22:53.202 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.0823999999999999e-05 2024-09-13T11:22:53.203 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.203 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.203 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:22:53.203 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.601720+0000", 2024-09-13T11:22:53.203 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00013851099999999999 2024-09-13T11:22:53.203 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.203 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.203 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 7", 2024-09-13T11:22:53.203 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.601843+0000", 2024-09-13T11:22:53.203 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00012291 2024-09-13T11:22:53.203 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.203 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.203 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:22:53.203 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.602668+0000", 2024-09-13T11:22:53.204 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00082519500000000003 2024-09-13T11:22:53.204 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.204 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.204 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:22:53.204 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.637659+0000", 2024-09-13T11:22:53.204 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.034991224000000001 2024-09-13T11:22:53.204 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.204 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.204 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:22:53.204 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.637716+0000", 2024-09-13T11:22:53.204 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.6425999999999998e-05 2024-09-13T11:22:53.204 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.204 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.205 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:22:53.205 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.637777+0000", 2024-09-13T11:22:53.205 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.0931999999999999e-05 2024-09-13T11:22:53.205 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:53.205 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:22:53.205 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:53.205 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.205 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.205 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:1003 3.a 3:5aec7cc6:::benchmark_data_smithi177_31116_object1002:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-09-13T11:22:53.205 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:22:49.614497+0000", 2024-09-13T11:22:53.205 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 3.5286965000000001, 2024-09-13T11:22:53.205 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.040510329999999997, 2024-09-13T11:22:53.205 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:22:53.206 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:22:53.206 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:53.206 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:22:53.206 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:22:53.206 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:22:53.206 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 1003 2024-09-13T11:22:53.206 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.206 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:22:53.206 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.206 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:22:53.206 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.614497+0000", 2024-09-13T11:22:53.206 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:53.206 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.206 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.207 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:22:53.207 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.614497+0000", 2024-09-13T11:22:53.207 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:53.207 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.207 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.207 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:22:53.207 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.614500+0000", 2024-09-13T11:22:53.207 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.3369999999999998e-06 2024-09-13T11:22:53.207 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.207 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.207 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:22:53.207 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.614531+0000", 2024-09-13T11:22:53.207 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.0828999999999999e-05 2024-09-13T11:22:53.208 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.208 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.208 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:22:53.208 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.614546+0000", 2024-09-13T11:22:53.208 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5945000000000001e-05 2024-09-13T11:22:53.208 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.208 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.208 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:22:53.208 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.614557+0000", 2024-09-13T11:22:53.208 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0317e-05 2024-09-13T11:22:53.208 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.208 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.208 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:22:53.208 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.614642+0000", 2024-09-13T11:22:53.209 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.4734999999999994e-05 2024-09-13T11:22:53.209 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.209 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.209 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:22:53.209 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.614794+0000", 2024-09-13T11:22:53.209 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000152027 2024-09-13T11:22:53.209 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.209 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.209 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 7", 2024-09-13T11:22:53.209 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.614980+0000", 2024-09-13T11:22:53.209 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00018625799999999999 2024-09-13T11:22:53.209 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.209 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.210 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:22:53.210 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.615821+0000", 2024-09-13T11:22:53.210 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00084122999999999999 2024-09-13T11:22:53.210 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.210 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.210 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:22:53.210 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.654761+0000", 2024-09-13T11:22:53.210 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.038939559999999998 2024-09-13T11:22:53.210 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.210 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.210 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:22:53.210 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.654897+0000", 2024-09-13T11:22:53.210 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00013600999999999999 2024-09-13T11:22:53.210 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.211 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.211 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:22:53.211 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.655008+0000", 2024-09-13T11:22:53.211 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000111082 2024-09-13T11:22:53.211 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:53.211 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:22:53.211 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:53.211 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.211 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.211 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop_reply(client.4665.0:1003 3.a e43/30 ondisk, result = 0)", 2024-09-13T11:22:53.211 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:22:49.616305+0000", 2024-09-13T11:22:53.211 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 3.5268890600000002, 2024-09-13T11:22:53.211 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.038717949000000002, 2024-09-13T11:22:53.211 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:22:53.212 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:22:53.212 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:22:53.212 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:22:53.212 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.212 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:22:53.212 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.616305+0000", 2024-09-13T11:22:53.212 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:53.212 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.212 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.212 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:22:53.212 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.616305+0000", 2024-09-13T11:22:53.212 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:53.212 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.213 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.213 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:22:53.213 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.616306+0000", 2024-09-13T11:22:53.213 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5200000000000001e-06 2024-09-13T11:22:53.213 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.213 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.213 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:22:53.213 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.616311+0000", 2024-09-13T11:22:53.213 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.5759999999999999e-06 2024-09-13T11:22:53.213 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.213 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.213 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:22:53.213 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.654620+0000", 2024-09-13T11:22:53.213 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.038309187000000001 2024-09-13T11:22:53.214 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.214 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.214 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:22:53.214 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.654628+0000", 2024-09-13T11:22:53.214 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.1669999999999999e-06 2024-09-13T11:22:53.214 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.214 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.214 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:22:53.214 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.654723+0000", 2024-09-13T11:22:53.214 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.4877999999999995e-05 2024-09-13T11:22:53.214 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.214 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.214 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:22:53.214 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.654741+0000", 2024-09-13T11:22:53.215 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.8073000000000001e-05 2024-09-13T11:22:53.215 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.215 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.215 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:22:53.215 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.655023+0000", 2024-09-13T11:22:53.215 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00028154800000000001 2024-09-13T11:22:53.215 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:53.215 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:22:53.215 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:53.215 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.215 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.215 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:1032 3.a 3:52874d26:::benchmark_data_smithi177_31116_object1031:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-09-13T11:22:53.215 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:22:49.639939+0000", 2024-09-13T11:22:53.215 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 3.5032547350000001, 2024-09-13T11:22:53.216 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.036486671999999998, 2024-09-13T11:22:53.216 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:22:53.216 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:22:53.216 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:53.216 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:22:53.216 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:22:53.216 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:22:53.216 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 1032 2024-09-13T11:22:53.216 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.216 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:22:53.216 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.216 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:22:53.216 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.639939+0000", 2024-09-13T11:22:53.216 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:53.217 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.217 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.217 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:22:53.217 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.639939+0000", 2024-09-13T11:22:53.217 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:53.217 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.217 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.217 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:22:53.217 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.639942+0000", 2024-09-13T11:22:53.218 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.4700000000000001e-06 2024-09-13T11:22:53.218 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.218 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.218 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:22:53.218 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.639996+0000", 2024-09-13T11:22:53.218 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.4381e-05 2024-09-13T11:22:53.218 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.218 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.218 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:22:53.218 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.640014+0000", 2024-09-13T11:22:53.218 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.7600000000000001e-05 2024-09-13T11:22:53.218 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.218 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.219 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:22:53.219 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.640023+0000", 2024-09-13T11:22:53.219 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.3160000000000003e-06 2024-09-13T11:22:53.219 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.219 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.219 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:22:53.219 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.640075+0000", 2024-09-13T11:22:53.219 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.2141000000000001e-05 2024-09-13T11:22:53.219 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.219 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.219 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:22:53.219 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.640202+0000", 2024-09-13T11:22:53.219 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00012673400000000001 2024-09-13T11:22:53.219 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.220 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.220 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 7", 2024-09-13T11:22:53.220 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.640310+0000", 2024-09-13T11:22:53.220 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000108063 2024-09-13T11:22:53.220 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.220 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.220 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:22:53.220 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.641366+0000", 2024-09-13T11:22:53.220 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001056544 2024-09-13T11:22:53.220 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.220 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.220 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:22:53.221 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.676337+0000", 2024-09-13T11:22:53.221 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.034970921000000002 2024-09-13T11:22:53.221 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.221 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.221 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:22:53.221 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.676384+0000", 2024-09-13T11:22:53.221 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.6471000000000003e-05 2024-09-13T11:22:53.221 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.221 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.221 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:22:53.221 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.676426+0000", 2024-09-13T11:22:53.221 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.2030999999999998e-05 2024-09-13T11:22:53.221 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:53.221 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:22:53.222 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:53.222 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.222 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.222 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:1122 3.d e43/30 3:b40a2a39:::benchmark_data_smithi177_31116_object1121:head v 43'66, mlcod=43'64)", 2024-09-13T11:22:53.222 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:22:49.712014+0000", 2024-09-13T11:22:53.222 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 3.4311794249999998, 2024-09-13T11:22:53.222 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.040250046999999997, 2024-09-13T11:22:53.222 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:22:53.222 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:22:53.222 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:53.222 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:22:53.222 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.222 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:22:53.223 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.712014+0000", 2024-09-13T11:22:53.223 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:53.223 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.223 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.223 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:22:53.223 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.712014+0000", 2024-09-13T11:22:53.223 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:53.223 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.223 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.223 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:22:53.223 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.712023+0000", 2024-09-13T11:22:53.223 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.4160000000000004e-06 2024-09-13T11:22:53.223 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.224 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.224 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:22:53.224 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.712084+0000", 2024-09-13T11:22:53.224 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.0720999999999997e-05 2024-09-13T11:22:53.224 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.224 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.224 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:22:53.224 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.748087+0000", 2024-09-13T11:22:53.224 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.036003546999999997 2024-09-13T11:22:53.224 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.224 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.224 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:22:53.224 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.748126+0000", 2024-09-13T11:22:53.225 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.9208999999999997e-05 2024-09-13T11:22:53.225 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.225 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.225 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:22:53.225 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.748264+0000", 2024-09-13T11:22:53.225 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00013805600000000001 2024-09-13T11:22:53.225 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.225 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.225 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:22:53.225 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.748313+0000", 2024-09-13T11:22:53.225 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.8575999999999997e-05 2024-09-13T11:22:53.225 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.225 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.226 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:22:53.226 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.752180+0000", 2024-09-13T11:22:53.226 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.003867498 2024-09-13T11:22:53.226 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.226 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.226 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:22:53.226 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.752264+0000", 2024-09-13T11:22:53.226 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.4023999999999994e-05 2024-09-13T11:22:53.226 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:53.226 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:22:53.226 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:53.226 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.226 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.226 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:1177 3.d e43/30 3:bc1c085d:::benchmark_data_smithi177_31116_object1176:head v 43'72, mlcod=43'67)", 2024-09-13T11:22:53.226 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:22:49.759453+0000", 2024-09-13T11:22:53.227 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 3.3837404150000001, 2024-09-13T11:22:53.227 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.036130789000000003, 2024-09-13T11:22:53.227 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:22:53.227 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:22:53.227 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:53.227 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:22:53.227 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.227 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:22:53.227 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.759453+0000", 2024-09-13T11:22:53.227 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:53.227 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.227 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.227 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:22:53.228 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.759453+0000", 2024-09-13T11:22:53.228 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:53.228 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.228 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.228 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:22:53.228 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.759456+0000", 2024-09-13T11:22:53.228 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.7219999999999999e-06 2024-09-13T11:22:53.228 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.228 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.228 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:22:53.228 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.759505+0000", 2024-09-13T11:22:53.228 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.8650999999999999e-05 2024-09-13T11:22:53.228 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.229 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.229 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:22:53.229 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.793964+0000", 2024-09-13T11:22:53.229 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.034459231999999999 2024-09-13T11:22:53.229 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.229 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.229 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:22:53.229 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.793976+0000", 2024-09-13T11:22:53.229 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.2021999999999999e-05 2024-09-13T11:22:53.229 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.229 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.229 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:22:53.229 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.794105+0000", 2024-09-13T11:22:53.229 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000128446 2024-09-13T11:22:53.230 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.230 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.230 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:22:53.230 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.794152+0000", 2024-09-13T11:22:53.230 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.7234999999999998e-05 2024-09-13T11:22:53.230 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.230 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.230 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:22:53.230 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.795480+0000", 2024-09-13T11:22:53.230 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0013285809999999999 2024-09-13T11:22:53.230 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.230 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.230 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:22:53.230 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.795584+0000", 2024-09-13T11:22:53.231 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0001039 2024-09-13T11:22:53.231 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:53.231 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:22:53.231 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:53.231 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.231 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.231 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:1275 3.a 3:5a15da6d:::benchmark_data_smithi177_31116_object1274:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-09-13T11:22:53.231 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:22:49.829287+0000", 2024-09-13T11:22:53.231 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 3.313906539, 2024-09-13T11:22:53.231 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.043769420000000003, 2024-09-13T11:22:53.231 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:22:53.231 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:22:53.231 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:53.231 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:22:53.232 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:22:53.232 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:22:53.232 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 1275 2024-09-13T11:22:53.232 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.232 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:22:53.232 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.232 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:22:53.232 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.829287+0000", 2024-09-13T11:22:53.232 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:53.232 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.232 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.232 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:22:53.232 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.829287+0000", 2024-09-13T11:22:53.233 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:53.233 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.233 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.233 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:22:53.233 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.829291+0000", 2024-09-13T11:22:53.233 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.4529999999999998e-06 2024-09-13T11:22:53.233 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.233 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.233 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:22:53.233 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.829362+0000", 2024-09-13T11:22:53.233 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.1064999999999998e-05 2024-09-13T11:22:53.233 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.234 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.234 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:22:53.234 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.829389+0000", 2024-09-13T11:22:53.234 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.6991999999999998e-05 2024-09-13T11:22:53.234 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.234 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.234 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:22:53.234 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.829408+0000", 2024-09-13T11:22:53.234 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.9477e-05 2024-09-13T11:22:53.234 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.234 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.234 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:22:53.234 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.829514+0000", 2024-09-13T11:22:53.234 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000105832 2024-09-13T11:22:53.234 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.235 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.235 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:22:53.235 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.829663+0000", 2024-09-13T11:22:53.235 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00014837300000000001 2024-09-13T11:22:53.235 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.235 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.235 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 7", 2024-09-13T11:22:53.235 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.829815+0000", 2024-09-13T11:22:53.235 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000152235 2024-09-13T11:22:53.235 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.235 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.235 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:22:53.235 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.832310+0000", 2024-09-13T11:22:53.235 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.002494844 2024-09-13T11:22:53.235 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.236 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.236 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:22:53.236 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.872852+0000", 2024-09-13T11:22:53.236 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.040542424000000001 2024-09-13T11:22:53.236 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.236 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.236 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:22:53.236 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.872981+0000", 2024-09-13T11:22:53.236 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00012902399999999999 2024-09-13T11:22:53.236 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.236 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.236 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:22:53.236 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.873057+0000", 2024-09-13T11:22:53.237 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.5700999999999999e-05 2024-09-13T11:22:53.237 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:53.237 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:22:53.237 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:53.237 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.237 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.237 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop_reply(client.4665.0:1275 3.a e43/30 ondisk, result = 0)", 2024-09-13T11:22:53.237 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:22:49.833244+0000", 2024-09-13T11:22:53.237 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 3.309949633, 2024-09-13T11:22:53.237 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.039823584000000002, 2024-09-13T11:22:53.237 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:22:53.237 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:22:53.237 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:22:53.238 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:22:53.238 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.238 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:22:53.238 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.833244+0000", 2024-09-13T11:22:53.238 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:53.238 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.238 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.238 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:22:53.238 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.833244+0000", 2024-09-13T11:22:53.238 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:53.238 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.238 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.238 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:22:53.238 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.833247+0000", 2024-09-13T11:22:53.239 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.802e-06 2024-09-13T11:22:53.239 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.239 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.239 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:22:53.239 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.833257+0000", 2024-09-13T11:22:53.239 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0068000000000001e-05 2024-09-13T11:22:53.239 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.239 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.239 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:22:53.239 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.872674+0000", 2024-09-13T11:22:53.239 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.039417059999999997 2024-09-13T11:22:53.240 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.240 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.240 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:22:53.240 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.872693+0000", 2024-09-13T11:22:53.240 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.889e-05 2024-09-13T11:22:53.240 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.240 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.240 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:22:53.240 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.872820+0000", 2024-09-13T11:22:53.240 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00012652099999999999 2024-09-13T11:22:53.240 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.241 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.241 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:22:53.241 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.872839+0000", 2024-09-13T11:22:53.241 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.9344999999999999e-05 2024-09-13T11:22:53.241 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.241 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.241 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:22:53.241 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.873068+0000", 2024-09-13T11:22:53.241 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000228898 2024-09-13T11:22:53.241 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:53.241 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:22:53.241 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:53.241 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.242 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.242 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:1282 3.d e43/30 3:b230ba6b:::benchmark_data_smithi177_31116_object1281:head v 43'78, mlcod=43'73)", 2024-09-13T11:22:53.242 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:22:49.834603+0000", 2024-09-13T11:22:53.242 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 3.3085910489999999, 2024-09-13T11:22:53.242 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.043089937000000002, 2024-09-13T11:22:53.242 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:22:53.242 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:22:53.242 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:53.242 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:22:53.242 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.242 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:22:53.242 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.834603+0000", 2024-09-13T11:22:53.242 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:53.243 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.243 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.243 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:22:53.243 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.834603+0000", 2024-09-13T11:22:53.243 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:53.243 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.243 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.243 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:22:53.243 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.834606+0000", 2024-09-13T11:22:53.243 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.7599999999999998e-06 2024-09-13T11:22:53.243 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.243 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.243 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:22:53.243 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.834684+0000", 2024-09-13T11:22:53.243 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.8490000000000002e-05 2024-09-13T11:22:53.244 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.244 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.244 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:22:53.244 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.875920+0000", 2024-09-13T11:22:53.244 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.041235956999999997 2024-09-13T11:22:53.244 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.244 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.244 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:22:53.244 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.875931+0000", 2024-09-13T11:22:53.244 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0834e-05 2024-09-13T11:22:53.244 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.244 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.245 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:22:53.245 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.876037+0000", 2024-09-13T11:22:53.245 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000106582 2024-09-13T11:22:53.245 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.245 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.245 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:22:53.245 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.876077+0000", 2024-09-13T11:22:53.245 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.9993000000000001e-05 2024-09-13T11:22:53.245 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.245 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.245 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:22:53.245 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.877634+0000", 2024-09-13T11:22:53.245 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001556254 2024-09-13T11:22:53.245 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.246 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.246 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:22:53.246 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.877693+0000", 2024-09-13T11:22:53.246 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.9067000000000002e-05 2024-09-13T11:22:53.246 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:53.246 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:22:53.246 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:53.246 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.246 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.246 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:1365 3.d e43/30 3:bf637086:::benchmark_data_smithi177_31116_object1364:head v 43'84, mlcod=43'80)", 2024-09-13T11:22:53.246 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:22:49.890601+0000", 2024-09-13T11:22:53.246 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 3.2525929499999999, 2024-09-13T11:22:53.246 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.040486463, 2024-09-13T11:22:53.247 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:22:53.247 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:22:53.247 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:53.247 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:22:53.247 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.247 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:22:53.247 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.890601+0000", 2024-09-13T11:22:53.247 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:53.247 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.247 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.247 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:22:53.247 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.890601+0000", 2024-09-13T11:22:53.247 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:53.247 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.248 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.248 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:22:53.248 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.890603+0000", 2024-09-13T11:22:53.248 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.2879999999999999e-06 2024-09-13T11:22:53.248 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.248 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.248 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:22:53.248 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.890638+0000", 2024-09-13T11:22:53.248 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.5216999999999998e-05 2024-09-13T11:22:53.248 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.248 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.248 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:22:53.248 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.925869+0000", 2024-09-13T11:22:53.249 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.035230298 2024-09-13T11:22:53.249 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.249 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.249 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:22:53.249 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.925883+0000", 2024-09-13T11:22:53.249 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3973e-05 2024-09-13T11:22:53.249 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.249 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.249 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:22:53.249 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.925987+0000", 2024-09-13T11:22:53.249 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00010407399999999999 2024-09-13T11:22:53.249 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.249 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.250 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:22:53.250 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.926024+0000", 2024-09-13T11:22:53.250 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.7379000000000002e-05 2024-09-13T11:22:53.250 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.250 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.250 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:22:53.250 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.931009+0000", 2024-09-13T11:22:53.250 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0049846719999999999 2024-09-13T11:22:53.250 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.250 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.250 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:22:53.250 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.931087+0000", 2024-09-13T11:22:53.250 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.8561999999999998e-05 2024-09-13T11:22:53.250 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:53.251 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:22:53.251 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:53.251 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:53.251 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:22:53.251 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-09-13T11:22:53.252 INFO:teuthology.orchestra.run.smithi001.stdout:osd.15: {} 2024-09-13T11:22:53.253 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-13T11:22:53.254 INFO:teuthology.orchestra.run.smithi001.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-13T11:22:53.254 INFO:tasks.ceph.osd.4.smithi001.stderr:2024-09-13T11:22:53.248+0000 7ff8e0218640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26310) UID: 0 2024-09-13T11:22:53.345 INFO:teuthology.orchestra.run.smithi001.stderr:reweighted osd.7 to 1 (10000) 2024-09-13T11:22:53.352 INFO:tasks.ceph.osd.7.smithi177.stderr:2024-09-13T11:22:53.351+0000 7f897b51d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26575) UID: 0 2024-09-13T11:22:53.374 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 8 1 2024-09-13T11:22:53.392 INFO:tasks.ceph.osd.2.smithi162.stderr:2024-09-13T11:22:53.387+0000 7fe91fd87640 -1 osd.2 46 handle_osd_map injecting map crc failure 2024-09-13T11:22:53.435 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-09-13T11:22:53.435 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [ 2024-09-13T11:22:53.436 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.436 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:8572 3.d 3:b2201e34:::benchmark_data_smithi177_31116_object8571:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-09-13T11:22:53.436 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:22:53.424573+0000", 2024-09-13T11:22:53.436 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 0.011959562999999999, 2024-09-13T11:22:53.436 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.01198044, 2024-09-13T11:22:53.436 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:22:53.436 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:22:53.436 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "waiting for sub ops", 2024-09-13T11:22:53.436 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:22:53.437 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:22:53.437 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:22:53.437 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 8572 2024-09-13T11:22:53.437 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.437 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:22:53.437 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.437 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:22:53.437 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:53.424573+0000", 2024-09-13T11:22:53.437 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:53.437 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.437 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.437 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:22:53.438 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:53.424573+0000", 2024-09-13T11:22:53.438 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:53.438 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.438 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.438 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:22:53.438 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:53.424576+0000", 2024-09-13T11:22:53.438 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.5239999999999999e-06 2024-09-13T11:22:53.438 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.438 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.438 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:22:53.438 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:53.424605+0000", 2024-09-13T11:22:53.439 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.8921e-05 2024-09-13T11:22:53.439 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.439 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.439 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:22:53.439 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:53.424621+0000", 2024-09-13T11:22:53.439 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5852e-05 2024-09-13T11:22:53.439 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.439 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.439 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:22:53.439 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:53.424629+0000", 2024-09-13T11:22:53.439 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.9120000000000004e-06 2024-09-13T11:22:53.440 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.440 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.440 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:22:53.440 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:53.424697+0000", 2024-09-13T11:22:53.440 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.8029999999999997e-05 2024-09-13T11:22:53.440 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.440 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.440 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:22:53.441 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:53.424846+0000", 2024-09-13T11:22:53.441 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00014871400000000001 2024-09-13T11:22:53.441 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.441 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.441 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 3", 2024-09-13T11:22:53.441 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:53.424970+0000", 2024-09-13T11:22:53.441 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00012426900000000001 2024-09-13T11:22:53.441 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.441 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.441 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:22:53.441 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:53.427613+0000", 2024-09-13T11:22:53.442 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.002642667 2024-09-13T11:22:53.442 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:53.442 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:22:53.442 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:53.442 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.442 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.442 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:8589 3.d 3:b0aaac46:::benchmark_data_smithi177_31116_object8588:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-09-13T11:22:53.442 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:22:53.432442+0000", 2024-09-13T11:22:53.442 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 0.0040908200000000002, 2024-09-13T11:22:53.442 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0041754189999999997, 2024-09-13T11:22:53.442 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:22:53.442 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:22:53.443 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "waiting for sub ops", 2024-09-13T11:22:53.443 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:22:53.443 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:22:53.443 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:22:53.443 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 8589 2024-09-13T11:22:53.443 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.443 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:22:53.443 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.444 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:22:53.444 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:53.432442+0000", 2024-09-13T11:22:53.444 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:53.444 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.444 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.444 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:22:53.444 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:53.432442+0000", 2024-09-13T11:22:53.444 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:53.444 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.444 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.444 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:22:53.445 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:53.432445+0000", 2024-09-13T11:22:53.445 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.6400000000000001e-06 2024-09-13T11:22:53.445 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.445 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.445 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:22:53.445 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:53.432473+0000", 2024-09-13T11:22:53.445 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.8646e-05 2024-09-13T11:22:53.445 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.445 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.445 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:22:53.445 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:53.432488+0000", 2024-09-13T11:22:53.446 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5223e-05 2024-09-13T11:22:53.446 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.446 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.446 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:22:53.446 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:53.432497+0000", 2024-09-13T11:22:53.446 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.0800000000000006e-06 2024-09-13T11:22:53.446 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.446 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.446 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:22:53.446 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:53.433054+0000", 2024-09-13T11:22:53.446 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00055714000000000002 2024-09-13T11:22:53.446 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.447 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.447 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:22:53.447 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:53.433176+0000", 2024-09-13T11:22:53.447 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000122712 2024-09-13T11:22:53.447 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.447 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.447 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 3", 2024-09-13T11:22:53.447 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:53.433285+0000", 2024-09-13T11:22:53.448 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00010893200000000001 2024-09-13T11:22:53.448 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:53.448 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:22:53.448 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:53.448 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:53.448 INFO:teuthology.orchestra.run.smithi177.stdout: ], 2024-09-13T11:22:53.448 INFO:teuthology.orchestra.run.smithi177.stdout: "num_ops": 2 2024-09-13T11:22:53.448 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-09-13T11:22:53.454 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-13T11:22:53.454 INFO:tasks.ceph.osd.3.smithi177.stderr:2024-09-13T11:22:53.451+0000 7f10adba4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26574) UID: 0 2024-09-13T11:22:53.555 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-09-13T11:22:53.551+0000 7f97befc9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26315) UID: 0 2024-09-13T11:22:53.642 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-09-13T11:22:53.642 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [], 2024-09-13T11:22:53.642 INFO:teuthology.orchestra.run.smithi177.stdout: "complaint_time": 30, 2024-09-13T11:22:53.642 INFO:teuthology.orchestra.run.smithi177.stdout: "num_blocked_ops": 0 2024-09-13T11:22:53.642 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-09-13T11:22:53.646 INFO:tasks.ceph.osd.4.smithi001.stderr:2024-09-13T11:22:53.640+0000 7ff8d0c3d640 -1 osd.4 46 handle_osd_map injecting map crc failure 2024-09-13T11:22:53.656 INFO:tasks.ceph.osd.14.smithi162.stderr:2024-09-13T11:22:53.651+0000 7f235c0db640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26319) UID: 0 2024-09-13T11:22:53.660 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-13T11:22:53.757 INFO:tasks.ceph.osd.7.smithi177.stderr:2024-09-13T11:22:53.755+0000 7f897b51d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26575) UID: 0 2024-09-13T11:22:53.800 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-09-13T11:22:53.800 INFO:teuthology.orchestra.run.smithi177.stdout: "size": 20, 2024-09-13T11:22:53.800 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 600, 2024-09-13T11:22:53.800 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [ 2024-09-13T11:22:53.800 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.800 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:830 3.d 3:baefa115:::benchmark_data_smithi177_31116_object829:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-09-13T11:22:53.800 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:22:49.458082+0000", 2024-09-13T11:22:53.800 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 4.3413978140000005, 2024-09-13T11:22:53.801 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.031714818999999998, 2024-09-13T11:22:53.801 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:22:53.801 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:22:53.801 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:53.801 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:22:53.801 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:22:53.801 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:22:53.801 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 830 2024-09-13T11:22:53.801 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.801 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:22:53.801 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.801 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:22:53.802 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.458082+0000", 2024-09-13T11:22:53.802 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:53.802 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.802 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.802 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:22:53.802 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.458082+0000", 2024-09-13T11:22:53.802 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:53.802 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.802 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.802 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:22:53.802 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.458085+0000", 2024-09-13T11:22:53.802 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.0860000000000002e-06 2024-09-13T11:22:53.803 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.803 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.803 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:22:53.803 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.458112+0000", 2024-09-13T11:22:53.803 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.7282999999999999e-05 2024-09-13T11:22:53.803 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.803 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.803 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:22:53.803 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.458127+0000", 2024-09-13T11:22:53.803 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.4241e-05 2024-09-13T11:22:53.804 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.804 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.804 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:22:53.804 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.458133+0000", 2024-09-13T11:22:53.804 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.6440000000000003e-06 2024-09-13T11:22:53.804 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.804 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.804 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:22:53.804 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.458204+0000", 2024-09-13T11:22:53.804 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.0338000000000003e-05 2024-09-13T11:22:53.804 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.804 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.804 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:22:53.805 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.458314+0000", 2024-09-13T11:22:53.805 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000109985 2024-09-13T11:22:53.805 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.805 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.805 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 3", 2024-09-13T11:22:53.805 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.458425+0000", 2024-09-13T11:22:53.805 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000111573 2024-09-13T11:22:53.805 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.805 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.805 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:22:53.805 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.459486+0000", 2024-09-13T11:22:53.805 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00106134 2024-09-13T11:22:53.805 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.805 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.806 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:22:53.806 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.489730+0000", 2024-09-13T11:22:53.806 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.030243545 2024-09-13T11:22:53.806 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.806 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.806 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:22:53.806 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.489766+0000", 2024-09-13T11:22:53.806 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.5975e-05 2024-09-13T11:22:53.806 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.806 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.806 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:22:53.807 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.489797+0000", 2024-09-13T11:22:53.807 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.0809000000000003e-05 2024-09-13T11:22:53.807 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:53.807 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:22:53.807 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:53.807 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.807 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.807 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:876 3.d 3:bbe42dec:::benchmark_data_smithi177_31116_object875:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-09-13T11:22:53.807 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:22:49.505331+0000", 2024-09-13T11:22:53.807 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 4.2941488000000003, 2024-09-13T11:22:53.807 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.03787285, 2024-09-13T11:22:53.807 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:22:53.807 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:22:53.807 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:53.808 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:22:53.808 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:22:53.808 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:22:53.808 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 876 2024-09-13T11:22:53.808 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.808 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:22:53.808 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.809 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:22:53.809 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.505331+0000", 2024-09-13T11:22:53.809 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:53.809 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.809 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.809 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:22:53.809 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.505331+0000", 2024-09-13T11:22:53.809 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:53.809 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.809 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.809 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:22:53.809 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.505334+0000", 2024-09-13T11:22:53.809 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.7039999999999999e-06 2024-09-13T11:22:53.810 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.810 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.810 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:22:53.810 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.505366+0000", 2024-09-13T11:22:53.810 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.2364999999999999e-05 2024-09-13T11:22:53.810 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.810 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.810 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:22:53.810 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.505382+0000", 2024-09-13T11:22:53.810 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5475e-05 2024-09-13T11:22:53.810 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.810 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.811 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:22:53.811 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.505389+0000", 2024-09-13T11:22:53.811 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.4270000000000002e-06 2024-09-13T11:22:53.811 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.811 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.811 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:22:53.811 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.505442+0000", 2024-09-13T11:22:53.811 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.3091999999999999e-05 2024-09-13T11:22:53.811 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.811 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.811 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:22:53.811 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.505526+0000", 2024-09-13T11:22:53.811 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.3466000000000004e-05 2024-09-13T11:22:53.811 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.811 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.812 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 3", 2024-09-13T11:22:53.812 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.505612+0000", 2024-09-13T11:22:53.812 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.6745999999999999e-05 2024-09-13T11:22:53.812 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.812 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.812 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:22:53.812 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.506434+0000", 2024-09-13T11:22:53.812 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00082220800000000005 2024-09-13T11:22:53.812 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.812 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.812 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:22:53.812 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.543113+0000", 2024-09-13T11:22:53.813 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.036678062999999997 2024-09-13T11:22:53.813 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.813 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.813 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:22:53.813 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.543162+0000", 2024-09-13T11:22:53.813 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.9917000000000003e-05 2024-09-13T11:22:53.813 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.813 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.813 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:22:53.813 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.543204+0000", 2024-09-13T11:22:53.813 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.1387000000000002e-05 2024-09-13T11:22:53.813 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:53.813 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:22:53.813 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:53.814 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.814 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.814 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:899 3.d 3:ba88666e:::benchmark_data_smithi177_31116_object898:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-09-13T11:22:53.814 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:22:49.529005+0000", 2024-09-13T11:22:53.814 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 4.2704750020000004, 2024-09-13T11:22:53.814 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.031950366000000001, 2024-09-13T11:22:53.814 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:22:53.814 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:22:53.814 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:53.814 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:22:53.814 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:22:53.814 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:22:53.814 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 899 2024-09-13T11:22:53.815 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.815 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:22:53.815 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.815 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:22:53.815 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.529005+0000", 2024-09-13T11:22:53.815 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:53.815 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.815 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.815 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:22:53.815 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.529005+0000", 2024-09-13T11:22:53.815 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:53.815 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.815 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.816 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:22:53.816 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.529008+0000", 2024-09-13T11:22:53.816 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.3210000000000001e-06 2024-09-13T11:22:53.816 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.816 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.816 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:22:53.816 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.529040+0000", 2024-09-13T11:22:53.816 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.1748000000000002e-05 2024-09-13T11:22:53.816 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.816 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.816 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:22:53.816 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.529057+0000", 2024-09-13T11:22:53.816 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.7598999999999999e-05 2024-09-13T11:22:53.817 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.817 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.817 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:22:53.817 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.529065+0000", 2024-09-13T11:22:53.817 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.8949999999999998e-06 2024-09-13T11:22:53.817 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.817 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.817 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:22:53.817 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.529123+0000", 2024-09-13T11:22:53.817 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.8050999999999997e-05 2024-09-13T11:22:53.817 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.817 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.817 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:22:53.818 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.529264+0000", 2024-09-13T11:22:53.818 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000140183 2024-09-13T11:22:53.818 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.818 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.818 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 3", 2024-09-13T11:22:53.818 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.529381+0000", 2024-09-13T11:22:53.818 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000117058 2024-09-13T11:22:53.818 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.818 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.819 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:22:53.819 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.530760+0000", 2024-09-13T11:22:53.819 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001379183 2024-09-13T11:22:53.819 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.819 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.819 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:22:53.819 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.560883+0000", 2024-09-13T11:22:53.819 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.030123008 2024-09-13T11:22:53.819 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.819 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.819 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:22:53.819 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.560920+0000", 2024-09-13T11:22:53.819 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.7549000000000003e-05 2024-09-13T11:22:53.820 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.820 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.820 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:22:53.820 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.560955+0000", 2024-09-13T11:22:53.820 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.4771000000000003e-05 2024-09-13T11:22:53.820 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:53.820 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:22:53.820 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:53.820 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.820 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.820 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:924 3.d 3:b0ff2987:::benchmark_data_smithi177_31116_object923:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-09-13T11:22:53.820 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:22:49.546192+0000", 2024-09-13T11:22:53.821 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 4.2532878590000003, 2024-09-13T11:22:53.821 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.028314373, 2024-09-13T11:22:53.821 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:22:53.821 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:22:53.821 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:53.821 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:22:53.821 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:22:53.821 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:22:53.821 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 924 2024-09-13T11:22:53.821 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.821 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:22:53.821 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.821 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:22:53.822 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.546192+0000", 2024-09-13T11:22:53.822 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:53.822 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.822 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.822 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:22:53.822 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.546192+0000", 2024-09-13T11:22:53.822 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:53.822 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.822 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.822 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:22:53.822 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.546195+0000", 2024-09-13T11:22:53.822 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.9500000000000001e-06 2024-09-13T11:22:53.822 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.823 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.823 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:22:53.823 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.546234+0000", 2024-09-13T11:22:53.823 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.9106000000000001e-05 2024-09-13T11:22:53.823 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.823 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.823 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:22:53.823 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.546253+0000", 2024-09-13T11:22:53.823 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.8612000000000001e-05 2024-09-13T11:22:53.823 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.823 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.823 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:22:53.823 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.546263+0000", 2024-09-13T11:22:53.824 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0051000000000001e-05 2024-09-13T11:22:53.824 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.824 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.824 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:22:53.824 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.546344+0000", 2024-09-13T11:22:53.824 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.1359000000000005e-05 2024-09-13T11:22:53.824 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.824 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.824 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:22:53.824 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.546460+0000", 2024-09-13T11:22:53.824 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00011600899999999999 2024-09-13T11:22:53.824 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.824 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.824 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 3", 2024-09-13T11:22:53.825 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.546583+0000", 2024-09-13T11:22:53.825 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00012323799999999999 2024-09-13T11:22:53.825 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.825 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.825 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:22:53.825 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.547686+0000", 2024-09-13T11:22:53.825 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001103176 2024-09-13T11:22:53.825 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.825 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.825 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:22:53.825 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.574347+0000", 2024-09-13T11:22:53.825 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.02666083 2024-09-13T11:22:53.825 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.826 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.826 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:22:53.826 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.574421+0000", 2024-09-13T11:22:53.826 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.3787000000000004e-05 2024-09-13T11:22:53.826 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.826 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.826 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:22:53.826 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.574506+0000", 2024-09-13T11:22:53.826 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.5254999999999996e-05 2024-09-13T11:22:53.826 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:53.826 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:22:53.826 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:53.826 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.826 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.827 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:942 3.d 3:bd356909:::benchmark_data_smithi177_31116_object941:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-09-13T11:22:53.827 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:22:49.558916+0000", 2024-09-13T11:22:53.827 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 4.2405637040000004, 2024-09-13T11:22:53.827 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.033402718999999997, 2024-09-13T11:22:53.827 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:22:53.827 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:22:53.827 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:53.827 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:22:53.827 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:22:53.827 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:22:53.827 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 942 2024-09-13T11:22:53.827 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.827 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:22:53.827 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.828 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:22:53.828 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.558916+0000", 2024-09-13T11:22:53.828 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:53.828 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.828 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.828 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:22:53.828 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.558916+0000", 2024-09-13T11:22:53.828 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:53.828 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.828 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.828 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:22:53.828 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.558919+0000", 2024-09-13T11:22:53.828 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.1860000000000001e-06 2024-09-13T11:22:53.829 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.829 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.829 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:22:53.829 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.558952+0000", 2024-09-13T11:22:53.829 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.2452e-05 2024-09-13T11:22:53.829 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.829 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.829 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:22:53.829 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.558970+0000", 2024-09-13T11:22:53.829 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.8712999999999999e-05 2024-09-13T11:22:53.829 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.829 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.829 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:22:53.829 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.558984+0000", 2024-09-13T11:22:53.830 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3551e-05 2024-09-13T11:22:53.830 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.830 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.830 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:22:53.830 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.559057+0000", 2024-09-13T11:22:53.830 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.2791000000000002e-05 2024-09-13T11:22:53.830 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.830 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.830 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:22:53.830 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.559144+0000", 2024-09-13T11:22:53.830 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.6835999999999993e-05 2024-09-13T11:22:53.830 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.830 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.831 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 3", 2024-09-13T11:22:53.831 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.559244+0000", 2024-09-13T11:22:53.831 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000100166 2024-09-13T11:22:53.831 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.831 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.831 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:22:53.831 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.560167+0000", 2024-09-13T11:22:53.831 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00092354500000000001 2024-09-13T11:22:53.831 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.831 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.831 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:22:53.831 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.592148+0000", 2024-09-13T11:22:53.831 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.031980956999999997 2024-09-13T11:22:53.832 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.832 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.832 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:22:53.832 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.592230+0000", 2024-09-13T11:22:53.832 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.1249999999999996e-05 2024-09-13T11:22:53.832 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.832 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.832 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:22:53.832 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.592319+0000", 2024-09-13T11:22:53.832 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.9271999999999995e-05 2024-09-13T11:22:53.832 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:53.832 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:22:53.832 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:53.832 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.833 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.833 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:956 3.d 3:b090e4ba:::benchmark_data_smithi177_31116_object955:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-09-13T11:22:53.833 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:22:49.571417+0000", 2024-09-13T11:22:53.833 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 4.2280632540000003, 2024-09-13T11:22:53.833 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.03028585, 2024-09-13T11:22:53.833 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:22:53.833 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:22:53.833 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:53.833 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:22:53.833 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:22:53.833 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:22:53.833 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 956 2024-09-13T11:22:53.833 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.834 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:22:53.834 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.834 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:22:53.834 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.571417+0000", 2024-09-13T11:22:53.834 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:53.834 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.834 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.834 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:22:53.834 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.571417+0000", 2024-09-13T11:22:53.834 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:53.834 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.834 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.834 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:22:53.835 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.571419+0000", 2024-09-13T11:22:53.835 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.413e-06 2024-09-13T11:22:53.835 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.835 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.835 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:22:53.835 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.571467+0000", 2024-09-13T11:22:53.835 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.7763999999999999e-05 2024-09-13T11:22:53.835 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.835 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.835 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:22:53.835 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.571481+0000", 2024-09-13T11:22:53.835 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.4695000000000001e-05 2024-09-13T11:22:53.835 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.835 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.836 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:22:53.836 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.571491+0000", 2024-09-13T11:22:53.836 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.2560000000000008e-06 2024-09-13T11:22:53.836 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.836 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.836 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:22:53.836 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.571554+0000", 2024-09-13T11:22:53.836 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.3081000000000001e-05 2024-09-13T11:22:53.836 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.836 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.836 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:22:53.836 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.571695+0000", 2024-09-13T11:22:53.836 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00014142500000000001 2024-09-13T11:22:53.837 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.837 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.837 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 3", 2024-09-13T11:22:53.837 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.571783+0000", 2024-09-13T11:22:53.837 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.7707000000000006e-05 2024-09-13T11:22:53.837 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.837 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.837 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:22:53.837 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.573609+0000", 2024-09-13T11:22:53.837 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0018264819999999999 2024-09-13T11:22:53.837 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.837 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.837 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:22:53.838 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.601594+0000", 2024-09-13T11:22:53.838 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.027984594000000002 2024-09-13T11:22:53.838 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.838 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.838 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:22:53.838 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.601656+0000", 2024-09-13T11:22:53.838 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.1927000000000005e-05 2024-09-13T11:22:53.838 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.838 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.838 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:22:53.838 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.601702+0000", 2024-09-13T11:22:53.838 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.6505999999999998e-05 2024-09-13T11:22:53.838 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:53.838 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:22:53.839 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:53.839 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.839 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.839 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:959 3.d 3:b761d58b:::benchmark_data_smithi177_31116_object958:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-09-13T11:22:53.839 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:22:49.572481+0000", 2024-09-13T11:22:53.839 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 4.2269987240000004, 2024-09-13T11:22:53.839 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.032032565999999998, 2024-09-13T11:22:53.839 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:22:53.839 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:22:53.839 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:53.839 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:22:53.839 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:22:53.839 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:22:53.839 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 959 2024-09-13T11:22:53.840 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.840 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:22:53.840 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.840 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:22:53.840 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.572481+0000", 2024-09-13T11:22:53.840 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:53.840 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.840 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.840 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:22:53.840 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.572481+0000", 2024-09-13T11:22:53.840 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:53.840 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.840 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.840 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:22:53.841 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.572483+0000", 2024-09-13T11:22:53.841 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.0640000000000001e-06 2024-09-13T11:22:53.841 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.841 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.841 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:22:53.841 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.572541+0000", 2024-09-13T11:22:53.841 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.7450000000000001e-05 2024-09-13T11:22:53.841 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.841 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.841 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:22:53.841 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.572557+0000", 2024-09-13T11:22:53.841 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.6180000000000001e-05 2024-09-13T11:22:53.841 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.842 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.842 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:22:53.842 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.572565+0000", 2024-09-13T11:22:53.842 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.0600000000000008e-06 2024-09-13T11:22:53.842 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.842 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.842 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:22:53.842 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.572662+0000", 2024-09-13T11:22:53.842 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.7181000000000003e-05 2024-09-13T11:22:53.842 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.843 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.843 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:22:53.843 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.572846+0000", 2024-09-13T11:22:53.843 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00018409599999999999 2024-09-13T11:22:53.843 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.843 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.843 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 3", 2024-09-13T11:22:53.843 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.573027+0000", 2024-09-13T11:22:53.843 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00018115399999999999 2024-09-13T11:22:53.843 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.843 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.843 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:22:53.843 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.573947+0000", 2024-09-13T11:22:53.843 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00091944599999999998 2024-09-13T11:22:53.844 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.844 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.844 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:22:53.844 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.604454+0000", 2024-09-13T11:22:53.844 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.030507593 2024-09-13T11:22:53.844 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.844 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.844 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:22:53.844 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.604488+0000", 2024-09-13T11:22:53.844 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.3379e-05 2024-09-13T11:22:53.844 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.844 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.844 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:22:53.845 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.604514+0000", 2024-09-13T11:22:53.845 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.5962999999999999e-05 2024-09-13T11:22:53.845 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:53.845 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:22:53.845 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:53.845 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.845 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.845 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:987 3.d 3:bda2e645:::benchmark_data_smithi177_31116_object986:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-09-13T11:22:53.845 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:22:49.599396+0000", 2024-09-13T11:22:53.845 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 4.2000842089999999, 2024-09-13T11:22:53.845 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.037252823999999997, 2024-09-13T11:22:53.845 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:22:53.845 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:22:53.846 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:53.846 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:22:53.846 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:22:53.846 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:22:53.846 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 987 2024-09-13T11:22:53.846 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.846 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:22:53.846 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.846 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:22:53.846 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.599396+0000", 2024-09-13T11:22:53.846 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:53.846 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.846 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.846 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:22:53.847 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.599396+0000", 2024-09-13T11:22:53.847 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:53.847 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.847 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.847 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:22:53.847 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.599398+0000", 2024-09-13T11:22:53.847 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.26e-06 2024-09-13T11:22:53.847 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.847 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.847 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:22:53.847 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.599432+0000", 2024-09-13T11:22:53.847 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.4041999999999997e-05 2024-09-13T11:22:53.847 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.847 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.848 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:22:53.848 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.599462+0000", 2024-09-13T11:22:53.848 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.9703e-05 2024-09-13T11:22:53.848 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.848 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.848 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:22:53.848 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.599472+0000", 2024-09-13T11:22:53.848 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0548e-05 2024-09-13T11:22:53.848 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.848 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.848 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:22:53.848 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.599580+0000", 2024-09-13T11:22:53.848 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00010819600000000001 2024-09-13T11:22:53.849 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.849 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.849 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:22:53.849 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.599844+0000", 2024-09-13T11:22:53.849 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000264116 2024-09-13T11:22:53.849 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.849 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.849 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 3", 2024-09-13T11:22:53.849 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.600188+0000", 2024-09-13T11:22:53.849 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00034397099999999999 2024-09-13T11:22:53.849 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.849 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.849 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:22:53.849 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.601291+0000", 2024-09-13T11:22:53.850 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001102172 2024-09-13T11:22:53.850 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.850 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.850 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:22:53.850 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.636578+0000", 2024-09-13T11:22:53.850 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.035287888000000003 2024-09-13T11:22:53.850 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.850 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.850 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:22:53.850 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.636616+0000", 2024-09-13T11:22:53.850 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.7465000000000001e-05 2024-09-13T11:22:53.850 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.850 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.851 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:22:53.851 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.636648+0000", 2024-09-13T11:22:53.851 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.2462999999999997e-05 2024-09-13T11:22:53.851 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:53.851 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:22:53.851 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:53.851 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.851 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.851 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:1055 3.d 3:bde78a31:::benchmark_data_smithi177_31116_object1054:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-09-13T11:22:53.851 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:22:49.656562+0000", 2024-09-13T11:22:53.851 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 4.142917368, 2024-09-13T11:22:53.851 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.034196364, 2024-09-13T11:22:53.851 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:22:53.851 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:22:53.852 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:53.852 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:22:53.852 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:22:53.852 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:22:53.852 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 1055 2024-09-13T11:22:53.852 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.852 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:22:53.852 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.852 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:22:53.852 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.656562+0000", 2024-09-13T11:22:53.852 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:53.852 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.852 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.852 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:22:53.853 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.656562+0000", 2024-09-13T11:22:53.853 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:53.853 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.853 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.853 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:22:53.853 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.656564+0000", 2024-09-13T11:22:53.853 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.8950000000000001e-06 2024-09-13T11:22:53.853 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.853 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.853 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:22:53.853 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.656613+0000", 2024-09-13T11:22:53.853 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.8905e-05 2024-09-13T11:22:53.853 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.854 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.854 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:22:53.854 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.656625+0000", 2024-09-13T11:22:53.854 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.2054e-05 2024-09-13T11:22:53.854 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.854 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.854 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:22:53.854 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.656632+0000", 2024-09-13T11:22:53.854 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.905e-06 2024-09-13T11:22:53.854 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.854 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.854 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:22:53.854 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.656706+0000", 2024-09-13T11:22:53.854 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.3968999999999996e-05 2024-09-13T11:22:53.854 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.855 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.855 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:22:53.855 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.656885+0000", 2024-09-13T11:22:53.855 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000178543 2024-09-13T11:22:53.855 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.855 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.855 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 3", 2024-09-13T11:22:53.855 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.657144+0000", 2024-09-13T11:22:53.855 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00025925699999999999 2024-09-13T11:22:53.855 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.855 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.855 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:22:53.855 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.659172+0000", 2024-09-13T11:22:53.856 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.002028443 2024-09-13T11:22:53.856 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.856 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.856 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:22:53.856 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.690679+0000", 2024-09-13T11:22:53.856 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.031506882 2024-09-13T11:22:53.856 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.856 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.856 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:22:53.856 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.690724+0000", 2024-09-13T11:22:53.856 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.4410999999999999e-05 2024-09-13T11:22:53.856 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.857 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.857 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:22:53.857 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.690759+0000", 2024-09-13T11:22:53.857 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.5099999999999999e-05 2024-09-13T11:22:53.857 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:53.857 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:22:53.857 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:53.857 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.857 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.857 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:1122 3.d 3:b40a2a39:::benchmark_data_smithi177_31116_object1121:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-09-13T11:22:53.857 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:22:49.711239+0000", 2024-09-13T11:22:53.857 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 4.0882403539999999, 2024-09-13T11:22:53.857 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.041526306999999998, 2024-09-13T11:22:53.857 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:22:53.858 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:22:53.858 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:53.858 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:22:53.858 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:22:53.858 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:22:53.858 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 1122 2024-09-13T11:22:53.858 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.858 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:22:53.858 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.858 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:22:53.858 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.711239+0000", 2024-09-13T11:22:53.858 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:53.859 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.859 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.859 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:22:53.859 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.711239+0000", 2024-09-13T11:22:53.859 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:53.859 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.859 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.859 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:22:53.859 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.711244+0000", 2024-09-13T11:22:53.859 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.065e-06 2024-09-13T11:22:53.859 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.859 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.859 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:22:53.859 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.711313+0000", 2024-09-13T11:22:53.860 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.9405000000000004e-05 2024-09-13T11:22:53.860 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.860 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.860 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:22:53.860 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.711334+0000", 2024-09-13T11:22:53.860 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.1186999999999999e-05 2024-09-13T11:22:53.860 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.860 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.860 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:22:53.860 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.711345+0000", 2024-09-13T11:22:53.860 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.1352000000000001e-05 2024-09-13T11:22:53.860 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.860 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.861 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:22:53.861 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.711443+0000", 2024-09-13T11:22:53.861 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.7101000000000004e-05 2024-09-13T11:22:53.861 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.861 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.861 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:22:53.861 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.711626+0000", 2024-09-13T11:22:53.861 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00018319699999999999 2024-09-13T11:22:53.861 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.861 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.861 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 3", 2024-09-13T11:22:53.861 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.711848+0000", 2024-09-13T11:22:53.861 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000221907 2024-09-13T11:22:53.862 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.862 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.862 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:22:53.862 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.713111+0000", 2024-09-13T11:22:53.862 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0012635999999999999 2024-09-13T11:22:53.862 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.862 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.862 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:22:53.862 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.752614+0000", 2024-09-13T11:22:53.862 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.039503038999999997 2024-09-13T11:22:53.862 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.862 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.862 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:22:53.862 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.752701+0000", 2024-09-13T11:22:53.863 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.6206999999999996e-05 2024-09-13T11:22:53.863 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.863 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.863 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:22:53.863 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.752766+0000", 2024-09-13T11:22:53.863 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.5246999999999994e-05 2024-09-13T11:22:53.863 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:53.863 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:22:53.863 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:53.863 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.863 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.863 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:1167 3.d 3:bc23bcc0:::benchmark_data_smithi177_31116_object1166:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-09-13T11:22:53.863 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:22:49.747596+0000", 2024-09-13T11:22:53.863 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 4.0518836570000003, 2024-09-13T11:22:53.864 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.03149366, 2024-09-13T11:22:53.864 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:22:53.864 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:22:53.864 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:53.864 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:22:53.864 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:22:53.864 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:22:53.864 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 1167 2024-09-13T11:22:53.864 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.864 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:22:53.864 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.864 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:22:53.864 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.747596+0000", 2024-09-13T11:22:53.864 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:53.865 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.865 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.865 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:22:53.865 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.747596+0000", 2024-09-13T11:22:53.865 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:53.865 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.865 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.865 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:22:53.865 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.747600+0000", 2024-09-13T11:22:53.865 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.5319999999999998e-06 2024-09-13T11:22:53.865 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.865 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.865 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:22:53.866 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.747673+0000", 2024-09-13T11:22:53.866 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.3547999999999994e-05 2024-09-13T11:22:53.866 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.866 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.866 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:22:53.866 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.747700+0000", 2024-09-13T11:22:53.866 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.6573000000000001e-05 2024-09-13T11:22:53.866 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.866 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.866 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:22:53.866 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.747717+0000", 2024-09-13T11:22:53.866 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.7482999999999998e-05 2024-09-13T11:22:53.866 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.867 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.867 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:22:53.867 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.747843+0000", 2024-09-13T11:22:53.867 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000125757 2024-09-13T11:22:53.867 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.867 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.867 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:22:53.867 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.748060+0000", 2024-09-13T11:22:53.867 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00021707000000000001 2024-09-13T11:22:53.867 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.867 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.867 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 3", 2024-09-13T11:22:53.867 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.748338+0000", 2024-09-13T11:22:53.867 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00027807899999999998 2024-09-13T11:22:53.867 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.868 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.868 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:22:53.868 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.752367+0000", 2024-09-13T11:22:53.868 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0040286920000000004 2024-09-13T11:22:53.868 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.868 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.868 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:22:53.868 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.778966+0000", 2024-09-13T11:22:53.868 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.026598743000000001 2024-09-13T11:22:53.868 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.868 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.868 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:22:53.869 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.779034+0000", 2024-09-13T11:22:53.869 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.8620000000000004e-05 2024-09-13T11:22:53.869 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.869 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.869 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:22:53.869 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.779090+0000", 2024-09-13T11:22:53.869 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.5563000000000003e-05 2024-09-13T11:22:53.869 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:53.869 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:22:53.869 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:53.869 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.869 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.869 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:1173 3.d 3:bc83268f:::benchmark_data_smithi177_31116_object1172:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-09-13T11:22:53.870 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:22:49.755017+0000", 2024-09-13T11:22:53.870 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 4.0444631800000002, 2024-09-13T11:22:53.870 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.029186020999999999, 2024-09-13T11:22:53.870 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:22:53.870 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:22:53.870 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:53.870 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:22:53.870 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:22:53.870 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:22:53.870 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 1173 2024-09-13T11:22:53.870 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.870 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:22:53.870 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.870 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:22:53.871 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.755017+0000", 2024-09-13T11:22:53.871 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:53.871 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.871 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.871 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:22:53.871 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.755017+0000", 2024-09-13T11:22:53.871 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:53.871 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.871 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.871 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:22:53.871 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.755020+0000", 2024-09-13T11:22:53.871 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.743e-06 2024-09-13T11:22:53.871 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.872 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.872 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:22:53.872 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.755075+0000", 2024-09-13T11:22:53.872 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.4138000000000002e-05 2024-09-13T11:22:53.872 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.872 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.872 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:22:53.872 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.755100+0000", 2024-09-13T11:22:53.872 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.5816000000000002e-05 2024-09-13T11:22:53.872 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.872 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.872 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:22:53.872 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.755119+0000", 2024-09-13T11:22:53.873 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.8255999999999999e-05 2024-09-13T11:22:53.873 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.873 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.873 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:22:53.873 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.755233+0000", 2024-09-13T11:22:53.873 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000114399 2024-09-13T11:22:53.873 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.873 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.873 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:22:53.873 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.755404+0000", 2024-09-13T11:22:53.873 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000171218 2024-09-13T11:22:53.873 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.873 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.874 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 3", 2024-09-13T11:22:53.874 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.755594+0000", 2024-09-13T11:22:53.874 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00019030499999999999 2024-09-13T11:22:53.874 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.874 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.874 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:22:53.874 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.756917+0000", 2024-09-13T11:22:53.874 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0013224949999999999 2024-09-13T11:22:53.874 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.874 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.874 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:22:53.874 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.784065+0000", 2024-09-13T11:22:53.874 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.027147999999999999 2024-09-13T11:22:53.874 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.875 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.875 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:22:53.875 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.784153+0000", 2024-09-13T11:22:53.875 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.8004999999999995e-05 2024-09-13T11:22:53.875 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.875 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.875 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:22:53.875 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.784203+0000", 2024-09-13T11:22:53.875 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.9645999999999999e-05 2024-09-13T11:22:53.875 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:53.875 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:22:53.875 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:53.875 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.876 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.876 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:1177 3.d 3:bc1c085d:::benchmark_data_smithi177_31116_object1176:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-09-13T11:22:53.876 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:22:49.758391+0000", 2024-09-13T11:22:53.876 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 4.041088641, 2024-09-13T11:22:53.876 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.037854334000000003, 2024-09-13T11:22:53.876 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:22:53.876 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:22:53.876 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:53.876 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:22:53.876 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:22:53.876 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:22:53.876 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 1177 2024-09-13T11:22:53.876 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.877 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:22:53.877 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.877 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:22:53.877 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.758391+0000", 2024-09-13T11:22:53.877 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:53.877 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.877 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.877 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:22:53.877 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.758391+0000", 2024-09-13T11:22:53.877 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:53.877 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.877 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.877 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:22:53.878 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.758394+0000", 2024-09-13T11:22:53.878 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.9399999999999998e-06 2024-09-13T11:22:53.878 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.878 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.878 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:22:53.878 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.758460+0000", 2024-09-13T11:22:53.878 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.5543999999999995e-05 2024-09-13T11:22:53.878 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.878 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.878 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:22:53.878 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.758487+0000", 2024-09-13T11:22:53.878 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.7742999999999998e-05 2024-09-13T11:22:53.879 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.879 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.879 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:22:53.879 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.758505+0000", 2024-09-13T11:22:53.879 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.7575999999999999e-05 2024-09-13T11:22:53.879 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.879 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.879 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:22:53.879 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.758611+0000", 2024-09-13T11:22:53.879 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00010620000000000001 2024-09-13T11:22:53.879 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.879 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.880 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:22:53.880 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.758919+0000", 2024-09-13T11:22:53.880 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00030832400000000001 2024-09-13T11:22:53.880 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.880 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.880 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 3", 2024-09-13T11:22:53.880 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.759237+0000", 2024-09-13T11:22:53.880 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00031760700000000002 2024-09-13T11:22:53.880 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.880 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.880 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:22:53.880 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.760699+0000", 2024-09-13T11:22:53.881 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0014619419999999999 2024-09-13T11:22:53.881 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.881 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.881 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:22:53.881 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.796021+0000", 2024-09-13T11:22:53.881 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.035321495000000001 2024-09-13T11:22:53.881 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.881 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.881 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:22:53.881 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.796154+0000", 2024-09-13T11:22:53.881 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000133733 2024-09-13T11:22:53.881 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.881 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.882 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:22:53.882 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.796245+0000", 2024-09-13T11:22:53.882 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.1230000000000006e-05 2024-09-13T11:22:53.882 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:53.882 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:22:53.882 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:53.882 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.882 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.882 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:1230 3.d 3:bac30f6b:::benchmark_data_smithi177_31116_object1229:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-09-13T11:22:53.882 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:22:49.797960+0000", 2024-09-13T11:22:53.882 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 4.0015193709999997, 2024-09-13T11:22:53.882 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.033196154999999998, 2024-09-13T11:22:53.882 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:22:53.882 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:22:53.883 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:53.883 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:22:53.883 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:22:53.883 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:22:53.883 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 1230 2024-09-13T11:22:53.883 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.883 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:22:53.883 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.883 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:22:53.883 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.797960+0000", 2024-09-13T11:22:53.883 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:53.883 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.883 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.883 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:22:53.884 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.797960+0000", 2024-09-13T11:22:53.884 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:53.884 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.884 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.884 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:22:53.884 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.797965+0000", 2024-09-13T11:22:53.884 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.0770000000000001e-06 2024-09-13T11:22:53.884 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.884 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.884 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:22:53.884 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.798042+0000", 2024-09-13T11:22:53.884 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.7779000000000002e-05 2024-09-13T11:22:53.884 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.884 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.885 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:22:53.885 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.798073+0000", 2024-09-13T11:22:53.885 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.0465000000000001e-05 2024-09-13T11:22:53.885 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.885 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.885 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:22:53.885 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.798092+0000", 2024-09-13T11:22:53.885 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.8751999999999999e-05 2024-09-13T11:22:53.885 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.886 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.886 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:22:53.886 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.798291+0000", 2024-09-13T11:22:53.886 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000199235 2024-09-13T11:22:53.886 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.886 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.886 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:22:53.886 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.798626+0000", 2024-09-13T11:22:53.886 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00033512599999999999 2024-09-13T11:22:53.886 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.886 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.886 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 3", 2024-09-13T11:22:53.886 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.798872+0000", 2024-09-13T11:22:53.887 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00024637500000000001 2024-09-13T11:22:53.887 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.887 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.887 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:22:53.887 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.800896+0000", 2024-09-13T11:22:53.887 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0020238769999999999 2024-09-13T11:22:53.887 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.887 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.887 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:22:53.887 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.830944+0000", 2024-09-13T11:22:53.887 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.030047447000000001 2024-09-13T11:22:53.887 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.887 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.888 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:22:53.888 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.831066+0000", 2024-09-13T11:22:53.888 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000121963 2024-09-13T11:22:53.888 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.888 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.888 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:22:53.888 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.831157+0000", 2024-09-13T11:22:53.888 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.1058999999999997e-05 2024-09-13T11:22:53.888 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:53.888 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:22:53.888 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:53.888 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.888 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.889 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:1233 3.d 3:b2a2db48:::benchmark_data_smithi177_31116_object1232:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-09-13T11:22:53.889 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:22:49.799393+0000", 2024-09-13T11:22:53.889 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 4.0000867189999996, 2024-09-13T11:22:53.889 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.036109434000000003, 2024-09-13T11:22:53.889 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:22:53.889 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:22:53.889 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:53.889 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:22:53.889 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:22:53.889 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:22:53.889 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 1233 2024-09-13T11:22:53.889 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.890 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:22:53.890 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.890 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:22:53.890 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.799393+0000", 2024-09-13T11:22:53.890 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:53.890 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.890 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.890 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:22:53.890 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.799393+0000", 2024-09-13T11:22:53.890 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:53.890 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.890 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.890 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:22:53.891 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.799397+0000", 2024-09-13T11:22:53.891 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.822e-06 2024-09-13T11:22:53.891 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.891 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.891 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:22:53.891 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.799479+0000", 2024-09-13T11:22:53.891 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.2577000000000006e-05 2024-09-13T11:22:53.891 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.891 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.891 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:22:53.891 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.799506+0000", 2024-09-13T11:22:53.891 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.6835e-05 2024-09-13T11:22:53.891 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.891 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.892 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:22:53.892 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.799524+0000", 2024-09-13T11:22:53.892 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.7334e-05 2024-09-13T11:22:53.892 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.892 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.892 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:22:53.892 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.799625+0000", 2024-09-13T11:22:53.892 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000100895 2024-09-13T11:22:53.892 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.892 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.892 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:22:53.892 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.799913+0000", 2024-09-13T11:22:53.892 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00028851899999999999 2024-09-13T11:22:53.893 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.893 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.893 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 3", 2024-09-13T11:22:53.893 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.800268+0000", 2024-09-13T11:22:53.893 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00035519600000000001 2024-09-13T11:22:53.893 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.893 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.893 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:22:53.893 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.801882+0000", 2024-09-13T11:22:53.893 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001614012 2024-09-13T11:22:53.893 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.893 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.893 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:22:53.894 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.835347+0000", 2024-09-13T11:22:53.894 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.033464498000000002 2024-09-13T11:22:53.894 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.894 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.894 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:22:53.894 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.835438+0000", 2024-09-13T11:22:53.894 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.1442999999999998e-05 2024-09-13T11:22:53.894 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.894 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.894 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:22:53.894 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.835503+0000", 2024-09-13T11:22:53.894 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.4302999999999997e-05 2024-09-13T11:22:53.894 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:53.895 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:22:53.895 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:53.895 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.895 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.895 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:1282 3.d 3:b230ba6b:::benchmark_data_smithi177_31116_object1281:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-09-13T11:22:53.895 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:22:49.833858+0000", 2024-09-13T11:22:53.895 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 3.9656214589999998, 2024-09-13T11:22:53.895 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.045058543, 2024-09-13T11:22:53.895 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:22:53.895 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:22:53.895 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:53.895 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:22:53.895 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:22:53.895 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:22:53.896 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 1282 2024-09-13T11:22:53.896 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.896 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:22:53.896 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.896 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:22:53.896 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.833858+0000", 2024-09-13T11:22:53.896 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:53.896 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.896 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.896 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:22:53.896 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.833858+0000", 2024-09-13T11:22:53.896 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:53.897 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.897 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.897 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:22:53.897 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.833862+0000", 2024-09-13T11:22:53.897 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.6629999999999999e-06 2024-09-13T11:22:53.897 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.897 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.897 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:22:53.897 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.833909+0000", 2024-09-13T11:22:53.897 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.6826999999999997e-05 2024-09-13T11:22:53.897 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.897 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.897 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:22:53.897 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.833933+0000", 2024-09-13T11:22:53.898 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.3952e-05 2024-09-13T11:22:53.898 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.898 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.898 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:22:53.898 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.833949+0000", 2024-09-13T11:22:53.898 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5770999999999999e-05 2024-09-13T11:22:53.898 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.898 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.898 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:22:53.898 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.834033+0000", 2024-09-13T11:22:53.898 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.4028000000000003e-05 2024-09-13T11:22:53.898 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.898 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.899 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:22:53.899 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.834205+0000", 2024-09-13T11:22:53.899 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00017227200000000001 2024-09-13T11:22:53.899 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.899 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.899 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 3", 2024-09-13T11:22:53.899 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.834426+0000", 2024-09-13T11:22:53.899 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00022096 2024-09-13T11:22:53.899 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.899 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.899 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:22:53.899 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.835975+0000", 2024-09-13T11:22:53.899 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001549096 2024-09-13T11:22:53.899 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.900 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.900 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:22:53.900 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.878751+0000", 2024-09-13T11:22:53.900 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.042776531999999999 2024-09-13T11:22:53.900 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.900 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.900 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:22:53.900 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.878853+0000", 2024-09-13T11:22:53.900 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000101796 2024-09-13T11:22:53.900 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.900 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.900 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:22:53.900 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.878917+0000", 2024-09-13T11:22:53.901 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.3645999999999993e-05 2024-09-13T11:22:53.901 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:53.901 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:22:53.901 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:53.901 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.901 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.901 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:1353 3.d 3:bc3a2611:::benchmark_data_smithi177_31116_object1352:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-09-13T11:22:53.901 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:22:49.880965+0000", 2024-09-13T11:22:53.901 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 3.9185145210000001, 2024-09-13T11:22:53.901 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.030798537000000001, 2024-09-13T11:22:53.901 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:22:53.901 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:22:53.901 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:53.902 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:22:53.902 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:22:53.902 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:22:53.902 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 1353 2024-09-13T11:22:53.902 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.902 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:22:53.902 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.902 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:22:53.902 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.880965+0000", 2024-09-13T11:22:53.902 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:53.902 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.902 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.902 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:22:53.902 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.880965+0000", 2024-09-13T11:22:53.903 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:53.903 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.903 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.903 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:22:53.903 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.880969+0000", 2024-09-13T11:22:53.903 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.6500000000000002e-06 2024-09-13T11:22:53.903 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.903 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.903 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:22:53.903 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.881040+0000", 2024-09-13T11:22:53.903 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.1200999999999998e-05 2024-09-13T11:22:53.903 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.903 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.903 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:22:53.904 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.881061+0000", 2024-09-13T11:22:53.904 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.0550000000000001e-05 2024-09-13T11:22:53.904 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.904 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.904 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:22:53.904 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.881077+0000", 2024-09-13T11:22:53.904 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.6273000000000002e-05 2024-09-13T11:22:53.904 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.904 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.904 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:22:53.904 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.881191+0000", 2024-09-13T11:22:53.904 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000114276 2024-09-13T11:22:53.904 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.904 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.905 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:22:53.905 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.881445+0000", 2024-09-13T11:22:53.905 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00025386899999999999 2024-09-13T11:22:53.905 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.905 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.905 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 3", 2024-09-13T11:22:53.905 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.881739+0000", 2024-09-13T11:22:53.905 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00029348900000000002 2024-09-13T11:22:53.905 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.905 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.905 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:22:53.905 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.886023+0000", 2024-09-13T11:22:53.905 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0042843050000000004 2024-09-13T11:22:53.905 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.906 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.906 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:22:53.906 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.911560+0000", 2024-09-13T11:22:53.906 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.025536850999999999 2024-09-13T11:22:53.906 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.906 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.906 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:22:53.906 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.911674+0000", 2024-09-13T11:22:53.906 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00011415400000000001 2024-09-13T11:22:53.906 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.906 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.906 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:22:53.906 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.911764+0000", 2024-09-13T11:22:53.906 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.9919000000000004e-05 2024-09-13T11:22:53.907 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:53.907 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:22:53.907 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:53.907 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.907 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.907 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:1360 3.d 3:b288a8c1:::benchmark_data_smithi177_31116_object1359:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-09-13T11:22:53.907 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:22:49.886795+0000", 2024-09-13T11:22:53.907 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 3.9126845559999999, 2024-09-13T11:22:53.907 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.036075557000000001, 2024-09-13T11:22:53.907 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:22:53.907 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:22:53.907 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:53.907 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:22:53.908 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:22:53.908 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:22:53.908 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 1360 2024-09-13T11:22:53.908 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.908 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:22:53.908 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.908 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:22:53.908 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.886795+0000", 2024-09-13T11:22:53.908 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:53.908 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.908 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.908 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:22:53.908 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.886795+0000", 2024-09-13T11:22:53.909 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:53.909 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.909 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.909 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:22:53.909 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.886798+0000", 2024-09-13T11:22:53.909 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.5620000000000002e-06 2024-09-13T11:22:53.909 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.909 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.909 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:22:53.909 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.886835+0000", 2024-09-13T11:22:53.909 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.7496999999999997e-05 2024-09-13T11:22:53.909 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.909 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.910 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:22:53.910 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.886854+0000", 2024-09-13T11:22:53.910 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.8295e-05 2024-09-13T11:22:53.910 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.910 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.910 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:22:53.910 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.886863+0000", 2024-09-13T11:22:53.910 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.6919999999999997e-06 2024-09-13T11:22:53.910 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.910 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.910 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:22:53.910 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.886971+0000", 2024-09-13T11:22:53.910 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00010777000000000001 2024-09-13T11:22:53.911 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.911 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.911 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:22:53.911 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.887254+0000", 2024-09-13T11:22:53.911 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00028262799999999999 2024-09-13T11:22:53.911 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.911 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.911 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 3", 2024-09-13T11:22:53.911 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.887636+0000", 2024-09-13T11:22:53.911 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00038259699999999998 2024-09-13T11:22:53.911 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.911 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.911 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:22:53.912 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.889772+0000", 2024-09-13T11:22:53.912 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0021356220000000002 2024-09-13T11:22:53.912 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.912 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.912 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:22:53.912 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.922657+0000", 2024-09-13T11:22:53.912 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.032884919999999998 2024-09-13T11:22:53.912 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.912 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.912 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:22:53.912 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.922776+0000", 2024-09-13T11:22:53.912 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000119669 2024-09-13T11:22:53.912 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.913 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.913 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:22:53.913 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.922871+0000", 2024-09-13T11:22:53.913 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.4304999999999999e-05 2024-09-13T11:22:53.913 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:53.913 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:22:53.913 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:53.913 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.913 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.913 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:1365 3.d 3:bf637086:::benchmark_data_smithi177_31116_object1364:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-09-13T11:22:53.913 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:22:49.889970+0000", 2024-09-13T11:22:53.913 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 3.9095099329999998, 2024-09-13T11:22:53.913 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.041599599000000001, 2024-09-13T11:22:53.913 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:22:53.914 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:22:53.914 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:53.914 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:22:53.914 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:22:53.914 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:22:53.914 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 1365 2024-09-13T11:22:53.914 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.914 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:22:53.914 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.914 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:22:53.914 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.889970+0000", 2024-09-13T11:22:53.914 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:53.914 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.914 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.915 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:22:53.915 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.889970+0000", 2024-09-13T11:22:53.915 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:53.915 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.915 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.915 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:22:53.915 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.889973+0000", 2024-09-13T11:22:53.915 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.6840000000000001e-06 2024-09-13T11:22:53.915 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.915 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.915 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:22:53.915 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.890010+0000", 2024-09-13T11:22:53.915 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.7647000000000001e-05 2024-09-13T11:22:53.916 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.916 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.916 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:22:53.916 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.890029+0000", 2024-09-13T11:22:53.916 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.9137e-05 2024-09-13T11:22:53.916 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.916 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.916 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:22:53.916 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.890039+0000", 2024-09-13T11:22:53.916 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0076e-05 2024-09-13T11:22:53.916 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.916 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.916 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:22:53.916 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.890113+0000", 2024-09-13T11:22:53.917 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.3196999999999997e-05 2024-09-13T11:22:53.917 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.917 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.917 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:22:53.917 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.890287+0000", 2024-09-13T11:22:53.917 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000174141 2024-09-13T11:22:53.917 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.917 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.917 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 3", 2024-09-13T11:22:53.917 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.890458+0000", 2024-09-13T11:22:53.917 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000171519 2024-09-13T11:22:53.917 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.917 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.918 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:22:53.918 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.891513+0000", 2024-09-13T11:22:53.918 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0010551200000000001 2024-09-13T11:22:53.918 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.918 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.918 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:22:53.918 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.931392+0000", 2024-09-13T11:22:53.918 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.039878690000000001 2024-09-13T11:22:53.918 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.918 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.918 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:22:53.918 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.931494+0000", 2024-09-13T11:22:53.919 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000101625 2024-09-13T11:22:53.919 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.919 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.919 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:22:53.919 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.931569+0000", 2024-09-13T11:22:53.919 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.5763e-05 2024-09-13T11:22:53.919 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:53.919 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:22:53.919 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:53.919 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.919 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.919 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:8965 3.d 3:b7544a64:::benchmark_data_smithi177_31116_object8964:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-09-13T11:22:53.920 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:22:53.663164+0000", 2024-09-13T11:22:53.920 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 0.136316241, 2024-09-13T11:22:53.920 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.028649096999999998, 2024-09-13T11:22:53.920 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:22:53.920 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:22:53.920 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:53.920 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:22:53.920 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:22:53.920 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:22:53.920 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 8965 2024-09-13T11:22:53.920 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.920 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:22:53.921 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.921 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:22:53.921 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:53.663164+0000", 2024-09-13T11:22:53.921 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:53.921 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.921 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.921 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:22:53.921 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:53.663164+0000", 2024-09-13T11:22:53.921 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:53.921 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.921 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.921 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:22:53.921 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:53.663166+0000", 2024-09-13T11:22:53.922 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.2000000000000001e-06 2024-09-13T11:22:53.922 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.922 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.922 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:22:53.922 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:53.663189+0000", 2024-09-13T11:22:53.922 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.3597000000000001e-05 2024-09-13T11:22:53.922 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.922 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.922 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:22:53.922 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:53.663202+0000", 2024-09-13T11:22:53.922 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.2536e-05 2024-09-13T11:22:53.922 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.923 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.923 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:22:53.923 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:53.663208+0000", 2024-09-13T11:22:53.923 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.0270000000000001e-06 2024-09-13T11:22:53.923 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.923 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.923 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:22:53.923 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:53.663261+0000", 2024-09-13T11:22:53.923 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.2924999999999999e-05 2024-09-13T11:22:53.923 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.923 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.923 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:22:53.923 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:53.663378+0000", 2024-09-13T11:22:53.924 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000117058 2024-09-13T11:22:53.924 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.924 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.924 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 3", 2024-09-13T11:22:53.924 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:53.663482+0000", 2024-09-13T11:22:53.924 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000104109 2024-09-13T11:22:53.924 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.924 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.924 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:22:53.924 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:53.664347+0000", 2024-09-13T11:22:53.924 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00086497099999999997 2024-09-13T11:22:53.924 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.924 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.924 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:22:53.925 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:53.691704+0000", 2024-09-13T11:22:53.925 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.027357178999999999 2024-09-13T11:22:53.925 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.925 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.925 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:22:53.925 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:53.691762+0000", 2024-09-13T11:22:53.925 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.8099000000000001e-05 2024-09-13T11:22:53.925 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:53.925 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:53.925 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:22:53.925 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:53.691813+0000", 2024-09-13T11:22:53.925 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.0396000000000003e-05 2024-09-13T11:22:53.925 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:53.926 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:22:53.926 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:53.926 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:53.926 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:22:53.926 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-09-13T11:22:53.928 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-09-13T11:22:53.928 INFO:tasks.ceph.osd.9.smithi092.stderr:2024-09-13T11:22:53.922+0000 7ff5ab3d0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26348) UID: 0 2024-09-13T11:22:53.999 INFO:tasks.radosbench.radosbench.0.smithi177.stdout: 27 16 9446 9430 21.8258 113.125 0.00289641 0.0457977 2024-09-13T11:22:54.029 INFO:tasks.ceph.osd.14.smithi162.stderr:2024-09-13T11:22:54.023+0000 7f235c0db640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26319) UID: 0 2024-09-13T11:22:54.111 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-09-13T11:22:54.111 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [ 2024-09-13T11:22:54.111 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:54.112 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:9601 3.f 3:fc233cd2:::benchmark_data_smithi177_31116_object9600:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-09-13T11:22:54.112 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:22:54.104008+0000", 2024-09-13T11:22:54.112 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 0.0086619190000000006, 2024-09-13T11:22:54.112 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0086883700000000008, 2024-09-13T11:22:54.112 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:22:54.112 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:22:54.112 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "waiting for sub ops", 2024-09-13T11:22:54.112 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:22:54.112 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:22:54.112 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:22:54.112 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 9601 2024-09-13T11:22:54.112 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:54.112 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:22:54.113 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:54.113 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:22:54.113 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:54.104008+0000", 2024-09-13T11:22:54.113 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:54.113 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:54.113 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:54.113 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:22:54.113 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:54.104008+0000", 2024-09-13T11:22:54.113 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:54.113 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:54.113 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:54.113 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:22:54.113 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:54.104010+0000", 2024-09-13T11:22:54.114 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.621e-06 2024-09-13T11:22:54.114 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:54.114 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:54.114 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:22:54.114 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:54.104045+0000", 2024-09-13T11:22:54.114 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.5135999999999997e-05 2024-09-13T11:22:54.114 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:54.114 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:54.114 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:22:54.114 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:54.104059+0000", 2024-09-13T11:22:54.114 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.4171999999999999e-05 2024-09-13T11:22:54.114 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:54.114 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:54.115 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:22:54.115 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:54.104068+0000", 2024-09-13T11:22:54.115 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.1919999999999994e-06 2024-09-13T11:22:54.115 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:54.115 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:54.115 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:22:54.115 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:54.104153+0000", 2024-09-13T11:22:54.115 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.4703000000000005e-05 2024-09-13T11:22:54.115 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:54.115 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:54.115 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:22:54.115 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:54.104295+0000", 2024-09-13T11:22:54.115 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0001415 2024-09-13T11:22:54.116 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:54.116 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:54.116 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 12", 2024-09-13T11:22:54.116 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:54.104473+0000", 2024-09-13T11:22:54.116 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00017866599999999999 2024-09-13T11:22:54.116 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:54.116 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:54.116 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:22:54.116 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:54.105932+0000", 2024-09-13T11:22:54.116 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0014585449999999999 2024-09-13T11:22:54.116 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:54.116 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:22:54.116 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:54.117 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:54.117 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:54.117 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:9615 3.c 3:3318388e:::benchmark_data_smithi177_31116_object9614:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-09-13T11:22:54.117 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:22:54.111809+0000", 2024-09-13T11:22:54.117 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 0.00086071699999999999, 2024-09-13T11:22:54.117 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00095382300000000002, 2024-09-13T11:22:54.117 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:22:54.117 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:22:54.117 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "waiting for sub ops", 2024-09-13T11:22:54.117 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:22:54.117 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:22:54.117 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:22:54.117 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 9615 2024-09-13T11:22:54.117 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:54.118 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:22:54.118 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:54.118 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:22:54.118 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:54.111809+0000", 2024-09-13T11:22:54.118 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:54.118 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:54.118 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:54.118 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:22:54.118 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:54.111809+0000", 2024-09-13T11:22:54.118 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:54.118 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:54.118 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:54.118 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:22:54.118 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:54.111812+0000", 2024-09-13T11:22:54.119 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.9620000000000002e-06 2024-09-13T11:22:54.119 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:54.119 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:54.119 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:22:54.119 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:54.111846+0000", 2024-09-13T11:22:54.119 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.3589e-05 2024-09-13T11:22:54.119 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:54.119 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:54.119 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:22:54.119 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:54.111870+0000", 2024-09-13T11:22:54.119 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.3960000000000001e-05 2024-09-13T11:22:54.119 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:54.119 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:54.120 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:22:54.120 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:54.111884+0000", 2024-09-13T11:22:54.120 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3633e-05 2024-09-13T11:22:54.120 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:54.120 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:54.120 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:22:54.120 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:54.111948+0000", 2024-09-13T11:22:54.120 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.4772999999999998e-05 2024-09-13T11:22:54.120 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:54.120 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:54.120 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:22:54.120 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:54.112084+0000", 2024-09-13T11:22:54.120 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000135205 2024-09-13T11:22:54.121 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:54.121 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:54.121 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 13", 2024-09-13T11:22:54.121 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:54.112205+0000", 2024-09-13T11:22:54.121 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00012097800000000001 2024-09-13T11:22:54.121 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:54.121 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:22:54.121 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:54.121 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:54.121 INFO:teuthology.orchestra.run.smithi177.stdout: ], 2024-09-13T11:22:54.121 INFO:teuthology.orchestra.run.smithi177.stdout: "num_ops": 2 2024-09-13T11:22:54.121 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-09-13T11:22:54.130 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-09-13T11:22:54.130 INFO:tasks.ceph.osd.12.smithi001.stderr:2024-09-13T11:22:54.124+0000 7f892c6e6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26314) UID: 0 2024-09-13T11:22:54.230 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-09-13T11:22:54.227+0000 7f97befc9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26315) UID: 0 2024-09-13T11:22:54.296 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-09-13T11:22:54.297 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [], 2024-09-13T11:22:54.297 INFO:teuthology.orchestra.run.smithi177.stdout: "complaint_time": 30, 2024-09-13T11:22:54.297 INFO:teuthology.orchestra.run.smithi177.stdout: "num_blocked_ops": 0 2024-09-13T11:22:54.297 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-09-13T11:22:54.310 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-09-13T11:22:54.330 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-09-13T11:22:54.327+0000 7f97befc9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26315) UID: 0 2024-09-13T11:22:54.356 INFO:teuthology.orchestra.run.smithi001.stderr:reweighted osd.8 to 1 (10000) 2024-09-13T11:22:54.370 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-13T11:22:54.387 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 9 1 2024-09-13T11:22:54.431 INFO:tasks.ceph.osd.3.smithi177.stderr:2024-09-13T11:22:54.431+0000 7f10adba4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26574) UID: 0 2024-09-13T11:22:54.442 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-09-13T11:22:54.442 INFO:teuthology.orchestra.run.smithi177.stdout: "size": 20, 2024-09-13T11:22:54.442 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 600, 2024-09-13T11:22:54.442 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [ 2024-09-13T11:22:54.443 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:54.443 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:79 3.1 e32/30 3:841df5b5:::benchmark_data_smithi177_31116_object78:head v 32'6, mlcod=32'4)", 2024-09-13T11:22:54.443 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:22:27.057292+0000", 2024-09-13T11:22:54.443 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 27.384891037999999, 2024-09-13T11:22:54.443 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.89296786800000005, 2024-09-13T11:22:54.443 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:22:54.443 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:22:54.443 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:54.443 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:22:54.443 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:54.443 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:22:54.443 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.057292+0000", 2024-09-13T11:22:54.443 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:54.443 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:54.444 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:54.444 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:22:54.444 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.057292+0000", 2024-09-13T11:22:54.444 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:54.444 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:54.444 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:54.444 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:22:54.444 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.057293+0000", 2024-09-13T11:22:54.444 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.6109999999999999e-06 2024-09-13T11:22:54.444 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:54.444 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:54.444 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:22:54.444 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.057330+0000", 2024-09-13T11:22:54.445 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.6721999999999998e-05 2024-09-13T11:22:54.445 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:54.445 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:54.445 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:22:54.445 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.948197+0000", 2024-09-13T11:22:54.445 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.890867042 2024-09-13T11:22:54.445 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:54.445 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:54.445 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:22:54.445 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.948213+0000", 2024-09-13T11:22:54.445 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5826000000000001e-05 2024-09-13T11:22:54.445 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:54.445 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:54.445 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:22:54.446 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.948377+0000", 2024-09-13T11:22:54.446 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00016348600000000001 2024-09-13T11:22:54.446 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:54.446 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:54.446 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:22:54.446 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.948447+0000", 2024-09-13T11:22:54.446 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.0891999999999998e-05 2024-09-13T11:22:54.446 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:54.446 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:54.446 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:22:54.446 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.950150+0000", 2024-09-13T11:22:54.446 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001702329 2024-09-13T11:22:54.446 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:54.447 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:54.447 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:22:54.447 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.950260+0000", 2024-09-13T11:22:54.447 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00010996 2024-09-13T11:22:54.447 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:54.447 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:22:54.447 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:54.447 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:54.447 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:54.447 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:525 3.f 3:feaa2861:::benchmark_data_smithi177_31116_object524:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-09-13T11:22:54.447 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:22:49.321084+0000", 2024-09-13T11:22:54.447 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 5.1210989370000002, 2024-09-13T11:22:54.447 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.038088766000000003, 2024-09-13T11:22:54.447 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:22:54.448 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:22:54.448 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:54.448 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:22:54.448 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:22:54.448 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:22:54.448 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 525 2024-09-13T11:22:54.448 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:54.448 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:22:54.448 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:54.448 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:22:54.448 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.321084+0000", 2024-09-13T11:22:54.448 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:54.448 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:54.449 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:54.449 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:22:54.449 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.321084+0000", 2024-09-13T11:22:54.449 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:54.449 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:54.449 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:54.449 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:22:54.449 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.321086+0000", 2024-09-13T11:22:54.449 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.951e-06 2024-09-13T11:22:54.449 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:54.449 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:54.449 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:22:54.450 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.321117+0000", 2024-09-13T11:22:54.450 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.1146000000000003e-05 2024-09-13T11:22:54.450 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:54.450 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:54.450 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:22:54.450 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.321133+0000", 2024-09-13T11:22:54.450 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5868000000000001e-05 2024-09-13T11:22:54.450 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:54.450 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:54.450 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:22:54.450 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.321142+0000", 2024-09-13T11:22:54.450 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.9570000000000008e-06 2024-09-13T11:22:54.450 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:54.450 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:54.451 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:22:54.451 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.321206+0000", 2024-09-13T11:22:54.451 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.4184999999999996e-05 2024-09-13T11:22:54.451 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:54.451 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:54.451 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:22:54.451 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.321400+0000", 2024-09-13T11:22:54.451 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000193499 2024-09-13T11:22:54.451 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:54.451 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:54.451 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 12", 2024-09-13T11:22:54.451 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.321571+0000", 2024-09-13T11:22:54.451 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00017127200000000001 2024-09-13T11:22:54.451 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:54.452 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:54.452 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:22:54.452 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.324857+0000", 2024-09-13T11:22:54.452 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.003286518 2024-09-13T11:22:54.452 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:54.452 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:54.452 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:22:54.452 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.359078+0000", 2024-09-13T11:22:54.452 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.034220965999999998 2024-09-13T11:22:54.452 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:54.452 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:54.452 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:22:54.452 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.359124+0000", 2024-09-13T11:22:54.453 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.587e-05 2024-09-13T11:22:54.453 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:54.453 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:54.453 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:22:54.453 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.359173+0000", 2024-09-13T11:22:54.453 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.8534e-05 2024-09-13T11:22:54.453 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:54.453 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:22:54.453 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:54.453 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:54.453 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:54.453 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:812 3.f 3:fbaccdd3:::benchmark_data_smithi177_31116_object811:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-09-13T11:22:54.453 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:22:49.446605+0000", 2024-09-13T11:22:54.453 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 4.9955776140000001, 2024-09-13T11:22:54.454 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.085769887000000003, 2024-09-13T11:22:54.454 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:22:54.454 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:22:54.454 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:54.454 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:22:54.454 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:22:54.454 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:22:54.454 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 812 2024-09-13T11:22:54.454 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:54.454 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:22:54.454 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:54.454 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:22:54.454 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.446605+0000", 2024-09-13T11:22:54.455 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:54.455 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:54.455 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:54.455 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:22:54.455 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.446605+0000", 2024-09-13T11:22:54.455 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:54.455 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:54.455 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:54.455 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:22:54.455 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.446607+0000", 2024-09-13T11:22:54.455 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.934e-06 2024-09-13T11:22:54.455 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:54.455 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:54.455 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:22:54.456 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.446656+0000", 2024-09-13T11:22:54.456 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.8368999999999997e-05 2024-09-13T11:22:54.456 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:54.456 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:54.456 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:22:54.456 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.446677+0000", 2024-09-13T11:22:54.456 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.1367000000000001e-05 2024-09-13T11:22:54.456 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:54.456 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:54.456 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:22:54.456 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.446689+0000", 2024-09-13T11:22:54.456 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.1809e-05 2024-09-13T11:22:54.456 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:54.457 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:54.457 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:22:54.457 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.446766+0000", 2024-09-13T11:22:54.457 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.7092e-05 2024-09-13T11:22:54.457 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:54.457 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:54.457 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:22:54.457 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.446917+0000", 2024-09-13T11:22:54.457 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00015103800000000001 2024-09-13T11:22:54.457 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:54.457 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:54.457 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 12", 2024-09-13T11:22:54.457 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.447076+0000", 2024-09-13T11:22:54.457 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00015952699999999999 2024-09-13T11:22:54.458 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:54.458 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:54.458 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:22:54.458 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.447922+0000", 2024-09-13T11:22:54.458 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00084538099999999998 2024-09-13T11:22:54.458 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:54.458 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:54.458 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:22:54.458 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.532308+0000", 2024-09-13T11:22:54.458 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.084386037999999997 2024-09-13T11:22:54.458 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:54.458 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:54.458 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:22:54.458 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.532345+0000", 2024-09-13T11:22:54.459 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.676e-05 2024-09-13T11:22:54.459 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:54.459 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:54.459 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:22:54.459 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.532375+0000", 2024-09-13T11:22:54.459 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.0571999999999998e-05 2024-09-13T11:22:54.459 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:54.459 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:22:54.459 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:54.459 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:54.459 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:54.459 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop_reply(client.4665.0:812 3.f e43/30 ondisk, result = 0)", 2024-09-13T11:22:54.459 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:22:49.448230+0000", 2024-09-13T11:22:54.459 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 4.9939531200000005, 2024-09-13T11:22:54.460 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.084151144999999997, 2024-09-13T11:22:54.460 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:22:54.460 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:22:54.460 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:22:54.460 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:22:54.460 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:54.460 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:22:54.460 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.448230+0000", 2024-09-13T11:22:54.460 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:54.460 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:54.460 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:54.460 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:22:54.460 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.448230+0000", 2024-09-13T11:22:54.460 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:54.461 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:54.461 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:54.461 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:22:54.461 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.448231+0000", 2024-09-13T11:22:54.461 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.6020000000000001e-06 2024-09-13T11:22:54.461 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:54.461 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:54.461 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:22:54.461 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.448236+0000", 2024-09-13T11:22:54.461 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.4669999999999998e-06 2024-09-13T11:22:54.461 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:54.461 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:54.461 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:22:54.462 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.532240+0000", 2024-09-13T11:22:54.462 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.084004520999999999 2024-09-13T11:22:54.462 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:54.462 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:54.462 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:22:54.462 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.532247+0000", 2024-09-13T11:22:54.462 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.8979999999999999e-06 2024-09-13T11:22:54.462 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:54.462 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:54.462 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:22:54.462 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.532299+0000", 2024-09-13T11:22:54.462 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.1372999999999998e-05 2024-09-13T11:22:54.462 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:54.462 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:54.463 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:22:54.463 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.532304+0000", 2024-09-13T11:22:54.463 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.6219999999999997e-06 2024-09-13T11:22:54.463 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:54.463 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:54.463 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:22:54.463 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.532381+0000", 2024-09-13T11:22:54.463 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.6662000000000006e-05 2024-09-13T11:22:54.463 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:54.463 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:22:54.463 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:54.463 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:54.463 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:54.463 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:996 3.f 3:fe4d616b:::benchmark_data_smithi177_31116_object995:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-09-13T11:22:54.464 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:22:49.607205+0000", 2024-09-13T11:22:54.464 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 4.8349777229999997, 2024-09-13T11:22:54.464 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.040821096000000001, 2024-09-13T11:22:54.464 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:22:54.464 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:22:54.464 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:54.464 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:22:54.464 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:22:54.464 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:22:54.464 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 996 2024-09-13T11:22:54.464 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:54.464 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:22:54.464 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:54.464 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:22:54.465 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.607205+0000", 2024-09-13T11:22:54.465 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:54.465 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:54.465 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:54.465 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:22:54.465 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.607205+0000", 2024-09-13T11:22:54.465 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:54.465 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:54.465 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:54.465 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:22:54.465 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.607208+0000", 2024-09-13T11:22:54.465 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.3539999999999998e-06 2024-09-13T11:22:54.465 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:54.465 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:54.466 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:22:54.466 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.607239+0000", 2024-09-13T11:22:54.466 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.0987e-05 2024-09-13T11:22:54.466 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:54.466 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:54.466 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:22:54.466 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.607256+0000", 2024-09-13T11:22:54.466 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.7853e-05 2024-09-13T11:22:54.466 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:54.466 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:54.466 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:22:54.466 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.607268+0000", 2024-09-13T11:22:54.466 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.2101e-05 2024-09-13T11:22:54.467 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:54.467 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:54.467 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:22:54.467 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.607362+0000", 2024-09-13T11:22:54.467 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.4004000000000003e-05 2024-09-13T11:22:54.467 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:54.467 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:54.467 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:22:54.467 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.607496+0000", 2024-09-13T11:22:54.467 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000133094 2024-09-13T11:22:54.467 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:54.467 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:54.467 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 12", 2024-09-13T11:22:54.467 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.607619+0000", 2024-09-13T11:22:54.467 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00012389000000000001 2024-09-13T11:22:54.468 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:54.468 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:54.468 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:22:54.468 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.608849+0000", 2024-09-13T11:22:54.468 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001229607 2024-09-13T11:22:54.468 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:54.468 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:54.468 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:22:54.468 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.647944+0000", 2024-09-13T11:22:54.468 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.039095169999999999 2024-09-13T11:22:54.468 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:54.468 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:54.468 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:22:54.468 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.647990+0000", 2024-09-13T11:22:54.469 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.5964000000000002e-05 2024-09-13T11:22:54.469 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:54.469 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:54.469 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:22:54.469 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.648026+0000", 2024-09-13T11:22:54.469 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.6072000000000003e-05 2024-09-13T11:22:54.469 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:54.469 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:22:54.469 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:54.469 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:54.469 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:54.469 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop_reply(client.4665.0:996 3.f e43/30 ondisk, result = 0)", 2024-09-13T11:22:54.469 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:22:49.608987+0000", 2024-09-13T11:22:54.470 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 4.8331959639999997, 2024-09-13T11:22:54.470 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.039046384000000003, 2024-09-13T11:22:54.470 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:22:54.470 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:22:54.470 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:22:54.470 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:22:54.470 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:54.470 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:22:54.470 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.608987+0000", 2024-09-13T11:22:54.470 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:54.470 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:54.470 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:54.470 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:22:54.470 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.608987+0000", 2024-09-13T11:22:54.470 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:54.471 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:54.471 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:54.471 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:22:54.471 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.608989+0000", 2024-09-13T11:22:54.471 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.1679999999999998e-06 2024-09-13T11:22:54.471 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:54.471 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:54.471 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:22:54.471 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.609001+0000", 2024-09-13T11:22:54.471 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.1613999999999999e-05 2024-09-13T11:22:54.471 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:54.471 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:54.471 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:22:54.472 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.647842+0000", 2024-09-13T11:22:54.472 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.038841381000000001 2024-09-13T11:22:54.472 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:54.472 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:54.472 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:22:54.472 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.647853+0000", 2024-09-13T11:22:54.472 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0553999999999999e-05 2024-09-13T11:22:54.472 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:54.472 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:54.472 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:22:54.472 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.647927+0000", 2024-09-13T11:22:54.472 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.3980000000000007e-05 2024-09-13T11:22:54.472 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:54.472 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:54.473 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:22:54.473 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.647937+0000", 2024-09-13T11:22:54.473 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0823e-05 2024-09-13T11:22:54.473 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:54.473 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:54.473 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:22:54.473 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.648033+0000", 2024-09-13T11:22:54.473 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.5864000000000002e-05 2024-09-13T11:22:54.473 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:54.473 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:22:54.473 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:54.473 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:54.473 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:54.473 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:1473 3.f 3:f5761887:::benchmark_data_smithi177_31116_object1472:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-09-13T11:22:54.474 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:22:49.949267+0000", 2024-09-13T11:22:54.474 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 4.492915719, 2024-09-13T11:22:54.474 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.043987083000000003, 2024-09-13T11:22:54.474 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:22:54.474 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:22:54.474 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:54.474 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:22:54.474 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:22:54.474 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:22:54.474 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 1473 2024-09-13T11:22:54.474 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:54.474 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:22:54.474 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:54.474 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:22:54.475 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.949267+0000", 2024-09-13T11:22:54.475 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:54.475 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:54.475 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:54.475 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:22:54.475 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.949267+0000", 2024-09-13T11:22:54.475 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:54.475 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:54.475 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:54.475 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:22:54.475 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.949269+0000", 2024-09-13T11:22:54.475 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3999999999999999e-06 2024-09-13T11:22:54.475 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:54.475 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:54.476 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:22:54.476 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.949300+0000", 2024-09-13T11:22:54.476 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.1299e-05 2024-09-13T11:22:54.476 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:54.476 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:54.476 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:22:54.476 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.949310+0000", 2024-09-13T11:22:54.476 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0491e-05 2024-09-13T11:22:54.476 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:54.476 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:54.476 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:22:54.476 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.949317+0000", 2024-09-13T11:22:54.476 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.0849999999999999e-06 2024-09-13T11:22:54.477 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:54.477 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:54.477 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:22:54.477 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.949431+0000", 2024-09-13T11:22:54.477 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00011378399999999999 2024-09-13T11:22:54.477 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:54.477 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:54.477 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:22:54.477 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.949688+0000", 2024-09-13T11:22:54.477 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00025656499999999999 2024-09-13T11:22:54.477 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:54.477 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:54.477 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 12", 2024-09-13T11:22:54.477 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.949976+0000", 2024-09-13T11:22:54.478 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00028794999999999998 2024-09-13T11:22:54.478 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:54.478 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:54.478 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:22:54.478 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.951036+0000", 2024-09-13T11:22:54.478 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001059901 2024-09-13T11:22:54.478 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:54.478 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:54.478 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:22:54.478 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.993077+0000", 2024-09-13T11:22:54.478 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.042041567000000002 2024-09-13T11:22:54.478 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:54.478 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:54.479 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:22:54.479 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.993173+0000", 2024-09-13T11:22:54.479 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.5316999999999996e-05 2024-09-13T11:22:54.479 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:54.479 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:54.479 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:22:54.479 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.993254+0000", 2024-09-13T11:22:54.479 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.1724000000000006e-05 2024-09-13T11:22:54.479 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:54.479 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:22:54.479 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:54.479 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:54.479 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:54.479 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop_reply(client.4665.0:1473 3.f e43/30 ondisk, result = 0)", 2024-09-13T11:22:54.480 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:22:49.951245+0000", 2024-09-13T11:22:54.480 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 4.4909377109999999, 2024-09-13T11:22:54.480 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.042020788000000003, 2024-09-13T11:22:54.480 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:22:54.480 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:22:54.480 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:22:54.480 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:22:54.480 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:54.480 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:22:54.480 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.951245+0000", 2024-09-13T11:22:54.480 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:54.480 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:54.480 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:54.480 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:22:54.481 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.951245+0000", 2024-09-13T11:22:54.481 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:54.481 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:54.481 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:54.481 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:22:54.481 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.951247+0000", 2024-09-13T11:22:54.481 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.1129999999999999e-06 2024-09-13T11:22:54.481 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:54.481 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:54.481 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:22:54.481 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.951257+0000", 2024-09-13T11:22:54.481 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.2799999999999992e-06 2024-09-13T11:22:54.481 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:54.482 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:54.482 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:22:54.482 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.992962+0000", 2024-09-13T11:22:54.482 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.041705684999999999 2024-09-13T11:22:54.482 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:54.482 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:54.482 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:22:54.482 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.992975+0000", 2024-09-13T11:22:54.482 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.2962999999999999e-05 2024-09-13T11:22:54.482 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:54.482 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:54.482 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:22:54.482 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.993056+0000", 2024-09-13T11:22:54.482 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.0294000000000001e-05 2024-09-13T11:22:54.483 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:54.483 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:54.483 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:22:54.483 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.993069+0000", 2024-09-13T11:22:54.483 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3013000000000001e-05 2024-09-13T11:22:54.483 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:54.483 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:54.483 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:22:54.483 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.993266+0000", 2024-09-13T11:22:54.483 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00019744000000000001 2024-09-13T11:22:54.483 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:54.483 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:22:54.483 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:54.483 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:54.484 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:54.484 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:1482 3.f 3:f701ddec:::benchmark_data_smithi177_31116_object1481:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-09-13T11:22:54.484 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:22:49.955615+0000", 2024-09-13T11:22:54.484 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 4.4865674819999999, 2024-09-13T11:22:54.484 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.043939263999999999, 2024-09-13T11:22:54.484 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:22:54.484 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:22:54.484 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:54.484 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:22:54.484 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:22:54.484 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:22:54.484 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 1482 2024-09-13T11:22:54.484 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:54.485 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:22:54.485 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:54.485 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:22:54.485 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.955615+0000", 2024-09-13T11:22:54.485 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:54.485 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:54.485 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:54.485 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:22:54.485 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.955615+0000", 2024-09-13T11:22:54.485 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:54.485 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:54.485 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:54.485 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:22:54.485 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.955617+0000", 2024-09-13T11:22:54.486 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.8470000000000001e-06 2024-09-13T11:22:54.486 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:54.486 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:54.486 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:22:54.486 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.955667+0000", 2024-09-13T11:22:54.486 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.0195000000000003e-05 2024-09-13T11:22:54.486 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:54.486 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:54.486 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:22:54.486 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.955681+0000", 2024-09-13T11:22:54.486 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3750999999999999e-05 2024-09-13T11:22:54.486 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:54.486 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:54.486 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:22:54.487 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.955690+0000", 2024-09-13T11:22:54.487 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.5529999999999993e-06 2024-09-13T11:22:54.487 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:54.487 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:54.487 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:22:54.487 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.955784+0000", 2024-09-13T11:22:54.487 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.3819000000000004e-05 2024-09-13T11:22:54.487 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:54.487 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:54.487 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:22:54.487 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.956000+0000", 2024-09-13T11:22:54.487 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000216739 2024-09-13T11:22:54.487 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:54.487 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:54.488 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 12", 2024-09-13T11:22:54.488 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.956264+0000", 2024-09-13T11:22:54.488 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00026323299999999998 2024-09-13T11:22:54.488 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:54.488 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:54.488 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:22:54.488 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.957764+0000", 2024-09-13T11:22:54.488 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001500349 2024-09-13T11:22:54.488 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:54.488 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:54.488 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:22:54.488 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.999411+0000", 2024-09-13T11:22:54.488 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.041646942999999999 2024-09-13T11:22:54.488 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:54.489 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:54.489 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:22:54.489 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.999491+0000", 2024-09-13T11:22:54.489 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.9654999999999995e-05 2024-09-13T11:22:54.489 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:54.489 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:54.489 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:22:54.489 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.999555+0000", 2024-09-13T11:22:54.489 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.4179999999999999e-05 2024-09-13T11:22:54.489 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:54.489 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:22:54.489 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:54.489 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:54.489 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:54.490 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop_reply(client.4665.0:1482 3.f e43/30 ondisk, result = 0)", 2024-09-13T11:22:54.490 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:22:49.957277+0000", 2024-09-13T11:22:54.490 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 4.4849060109999996, 2024-09-13T11:22:54.490 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.042289684000000001, 2024-09-13T11:22:54.490 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:22:54.490 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:22:54.490 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:22:54.490 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:22:54.490 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:54.490 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:22:54.490 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.957277+0000", 2024-09-13T11:22:54.490 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:54.490 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:54.490 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:54.491 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:22:54.491 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.957277+0000", 2024-09-13T11:22:54.491 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:54.491 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:54.491 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:54.491 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:22:54.491 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.957279+0000", 2024-09-13T11:22:54.491 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.995e-06 2024-09-13T11:22:54.491 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:54.491 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:54.491 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:22:54.491 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.957287+0000", 2024-09-13T11:22:54.491 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.7260000000000002e-06 2024-09-13T11:22:54.492 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:54.492 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:54.492 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:22:54.492 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.999326+0000", 2024-09-13T11:22:54.492 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.042039111999999997 2024-09-13T11:22:54.492 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:54.492 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:54.492 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:22:54.492 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.999339+0000", 2024-09-13T11:22:54.492 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.2945000000000001e-05 2024-09-13T11:22:54.492 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:54.492 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:54.492 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:22:54.493 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.999391+0000", 2024-09-13T11:22:54.493 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.2487000000000001e-05 2024-09-13T11:22:54.493 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:54.493 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:54.493 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:22:54.493 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.999400+0000", 2024-09-13T11:22:54.493 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.3279999999999995e-06 2024-09-13T11:22:54.493 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:54.493 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:54.493 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:22:54.493 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.999567+0000", 2024-09-13T11:22:54.493 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00016609099999999999 2024-09-13T11:22:54.493 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:54.493 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:22:54.494 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:54.494 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:54.494 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:54.494 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:8863 3.c 3:3d097150:::benchmark_data_smithi177_31116_object8862:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-09-13T11:22:54.494 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:22:53.589623+0000", 2024-09-13T11:22:54.494 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 0.85255973200000001, 2024-09-13T11:22:54.494 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.040008196000000003, 2024-09-13T11:22:54.494 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:22:54.494 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:22:54.494 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:54.494 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:22:54.494 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:22:54.494 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:22:54.495 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 8863 2024-09-13T11:22:54.495 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:54.495 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:22:54.495 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:54.495 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:22:54.495 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:53.589623+0000", 2024-09-13T11:22:54.495 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:54.495 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:54.495 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:54.495 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:22:54.495 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:53.589623+0000", 2024-09-13T11:22:54.495 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:54.495 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:54.495 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:54.496 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:22:54.496 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:53.589624+0000", 2024-09-13T11:22:54.496 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.4799999999999997e-07 2024-09-13T11:22:54.496 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:54.496 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:54.496 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:22:54.496 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:53.589642+0000", 2024-09-13T11:22:54.496 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.7679000000000001e-05 2024-09-13T11:22:54.496 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:54.496 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:54.496 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:22:54.496 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:53.589650+0000", 2024-09-13T11:22:54.496 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.1119999999999996e-06 2024-09-13T11:22:54.497 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:54.497 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:54.497 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:22:54.497 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:53.589656+0000", 2024-09-13T11:22:54.497 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.1940000000000003e-06 2024-09-13T11:22:54.497 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:54.497 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:54.497 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:22:54.497 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:53.589695+0000", 2024-09-13T11:22:54.497 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.9096e-05 2024-09-13T11:22:54.497 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:54.497 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:54.497 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:22:54.497 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:53.589785+0000", 2024-09-13T11:22:54.498 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.9808999999999993e-05 2024-09-13T11:22:54.498 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:54.498 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:54.498 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 13", 2024-09-13T11:22:54.498 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:53.589914+0000", 2024-09-13T11:22:54.498 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000129169 2024-09-13T11:22:54.498 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:54.498 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:54.498 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:22:54.498 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:53.592435+0000", 2024-09-13T11:22:54.498 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.002521166 2024-09-13T11:22:54.498 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:54.498 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:54.498 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:22:54.498 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:53.629570+0000", 2024-09-13T11:22:54.499 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.037134524000000002 2024-09-13T11:22:54.499 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:54.499 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:54.499 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:22:54.499 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:53.629604+0000", 2024-09-13T11:22:54.499 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.4116000000000003e-05 2024-09-13T11:22:54.499 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:54.499 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:54.499 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:22:54.499 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:53.629631+0000", 2024-09-13T11:22:54.499 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.7382999999999999e-05 2024-09-13T11:22:54.499 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:54.499 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:22:54.499 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:54.500 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:54.500 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:54.500 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:8884 3.c 3:350c618f:::benchmark_data_smithi177_31116_object8883:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-09-13T11:22:54.500 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:22:53.598209+0000", 2024-09-13T11:22:54.500 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 0.84397366200000001, 2024-09-13T11:22:54.500 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.042377410999999997, 2024-09-13T11:22:54.500 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:22:54.500 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:22:54.500 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:54.500 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:22:54.500 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:22:54.500 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:22:54.500 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 8884 2024-09-13T11:22:54.500 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:54.501 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:22:54.501 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:54.501 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:22:54.501 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:53.598209+0000", 2024-09-13T11:22:54.501 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:54.501 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:54.501 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:54.501 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:22:54.501 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:53.598209+0000", 2024-09-13T11:22:54.501 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:54.501 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:54.501 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:54.501 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:22:54.502 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:53.598212+0000", 2024-09-13T11:22:54.502 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.5040000000000001e-06 2024-09-13T11:22:54.502 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:54.502 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:54.502 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:22:54.502 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:53.598238+0000", 2024-09-13T11:22:54.502 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.6682999999999998e-05 2024-09-13T11:22:54.502 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:54.502 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:54.502 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:22:54.502 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:53.598252+0000", 2024-09-13T11:22:54.502 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3662999999999999e-05 2024-09-13T11:22:54.502 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:54.502 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:54.502 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:22:54.503 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:53.598260+0000", 2024-09-13T11:22:54.503 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.3750000000000003e-06 2024-09-13T11:22:54.503 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:54.503 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:54.503 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:22:54.503 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:53.598332+0000", 2024-09-13T11:22:54.503 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.1261999999999996e-05 2024-09-13T11:22:54.503 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:54.503 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:54.503 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:22:54.503 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:53.598494+0000", 2024-09-13T11:22:54.503 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00016212900000000001 2024-09-13T11:22:54.503 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:54.503 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:54.503 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 13", 2024-09-13T11:22:54.504 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:53.598609+0000", 2024-09-13T11:22:54.504 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000114808 2024-09-13T11:22:54.504 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:54.504 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:54.504 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:22:54.504 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:53.601175+0000", 2024-09-13T11:22:54.504 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.002566249 2024-09-13T11:22:54.504 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:54.504 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:54.504 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:22:54.504 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:53.640497+0000", 2024-09-13T11:22:54.504 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.039321617000000003 2024-09-13T11:22:54.504 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:54.505 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:54.505 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:22:54.505 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:53.640539+0000", 2024-09-13T11:22:54.505 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.2882e-05 2024-09-13T11:22:54.505 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:54.505 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:54.505 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:22:54.505 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:53.640587+0000", 2024-09-13T11:22:54.505 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.7239e-05 2024-09-13T11:22:54.505 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:54.505 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:22:54.505 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:54.505 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:54.506 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:54.506 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:8892 3.c 3:39b8e9bb:::benchmark_data_smithi177_31116_object8891:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-09-13T11:22:54.506 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:22:53.604025+0000", 2024-09-13T11:22:54.506 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 0.83815826900000001, 2024-09-13T11:22:54.506 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.048771272999999997, 2024-09-13T11:22:54.506 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:22:54.506 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:22:54.506 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:54.506 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:22:54.506 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:22:54.506 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:22:54.506 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 8892 2024-09-13T11:22:54.506 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:54.507 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:22:54.507 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:54.507 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:22:54.507 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:53.604025+0000", 2024-09-13T11:22:54.507 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:54.507 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:54.507 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:54.507 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:22:54.507 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:53.604025+0000", 2024-09-13T11:22:54.507 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:54.507 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:54.507 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:54.507 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:22:54.507 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:53.604027+0000", 2024-09-13T11:22:54.508 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.108e-06 2024-09-13T11:22:54.508 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:54.508 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:54.508 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:22:54.508 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:53.604054+0000", 2024-09-13T11:22:54.508 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.7061000000000001e-05 2024-09-13T11:22:54.508 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:54.508 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:54.508 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:22:54.508 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:53.604067+0000", 2024-09-13T11:22:54.508 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3434e-05 2024-09-13T11:22:54.508 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:54.508 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:54.509 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:22:54.509 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:53.604074+0000", 2024-09-13T11:22:54.509 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.6989999999999997e-06 2024-09-13T11:22:54.509 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:54.509 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:54.509 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:22:54.509 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:53.604138+0000", 2024-09-13T11:22:54.509 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.3575e-05 2024-09-13T11:22:54.509 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:54.509 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:54.509 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:22:54.509 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:53.604231+0000", 2024-09-13T11:22:54.509 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.3661999999999999e-05 2024-09-13T11:22:54.510 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:54.510 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:54.510 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 13", 2024-09-13T11:22:54.510 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:53.604318+0000", 2024-09-13T11:22:54.510 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.6835999999999993e-05 2024-09-13T11:22:54.510 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:54.510 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:54.510 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:22:54.510 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:53.605225+0000", 2024-09-13T11:22:54.510 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000906612 2024-09-13T11:22:54.510 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:54.510 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:54.510 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:22:54.510 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:53.652702+0000", 2024-09-13T11:22:54.511 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.047477254000000003 2024-09-13T11:22:54.511 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:54.511 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:54.511 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:22:54.511 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:53.652747+0000", 2024-09-13T11:22:54.511 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.4842999999999997e-05 2024-09-13T11:22:54.511 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:54.511 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:54.511 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:22:54.511 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:53.652796+0000", 2024-09-13T11:22:54.511 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.9188999999999999e-05 2024-09-13T11:22:54.511 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:54.511 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:22:54.512 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:54.512 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:54.512 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:54.512 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:9012 3.c 3:3bd14e8d:::benchmark_data_smithi177_31116_object9011:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-09-13T11:22:54.512 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:22:53.715076+0000", 2024-09-13T11:22:54.512 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 0.72710648899999997, 2024-09-13T11:22:54.512 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.040799415999999998, 2024-09-13T11:22:54.512 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:22:54.512 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:22:54.512 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:54.512 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:22:54.512 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:22:54.513 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:22:54.513 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 9012 2024-09-13T11:22:54.513 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:54.513 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:22:54.513 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:54.513 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:22:54.513 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:53.715076+0000", 2024-09-13T11:22:54.513 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:54.513 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:54.513 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:54.513 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:22:54.513 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:53.715076+0000", 2024-09-13T11:22:54.513 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:54.514 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:54.514 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:54.514 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:22:54.514 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:53.715078+0000", 2024-09-13T11:22:54.514 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.393e-06 2024-09-13T11:22:54.514 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:54.514 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:54.514 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:22:54.514 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:53.715108+0000", 2024-09-13T11:22:54.514 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.9703e-05 2024-09-13T11:22:54.514 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:54.514 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:54.514 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:22:54.514 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:53.715118+0000", 2024-09-13T11:22:54.515 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0180000000000001e-05 2024-09-13T11:22:54.515 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:54.515 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:54.515 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:22:54.515 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:53.715124+0000", 2024-09-13T11:22:54.515 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.9190000000000004e-06 2024-09-13T11:22:54.515 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:54.515 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:54.515 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:22:54.515 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:53.715172+0000", 2024-09-13T11:22:54.515 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.8275000000000001e-05 2024-09-13T11:22:54.515 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:54.515 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:54.515 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:22:54.516 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:53.715265+0000", 2024-09-13T11:22:54.516 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.2839999999999999e-05 2024-09-13T11:22:54.516 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:54.516 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:54.516 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 13", 2024-09-13T11:22:54.516 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:53.715361+0000", 2024-09-13T11:22:54.516 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.5774999999999997e-05 2024-09-13T11:22:54.516 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:54.516 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:54.516 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:22:54.516 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:53.716263+0000", 2024-09-13T11:22:54.517 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00090201500000000004 2024-09-13T11:22:54.517 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:54.517 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:54.517 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:22:54.517 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:53.755766+0000", 2024-09-13T11:22:54.517 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.039503299999999998 2024-09-13T11:22:54.517 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:54.517 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:54.517 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:22:54.517 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:53.755825+0000", 2024-09-13T11:22:54.517 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.9194000000000003e-05 2024-09-13T11:22:54.517 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:54.517 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:54.518 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:22:54.518 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:53.755876+0000", 2024-09-13T11:22:54.518 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.0822000000000002e-05 2024-09-13T11:22:54.518 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:54.518 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:22:54.518 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:54.518 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:54.518 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:54.518 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:9111 3.c 3:3866299c:::benchmark_data_smithi177_31116_object9110:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-09-13T11:22:54.518 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:22:53.781282+0000", 2024-09-13T11:22:54.518 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 0.66090138700000001, 2024-09-13T11:22:54.518 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.044886658000000003, 2024-09-13T11:22:54.518 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:22:54.519 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:22:54.519 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:54.519 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:22:54.519 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:22:54.519 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:22:54.519 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 9111 2024-09-13T11:22:54.519 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:54.519 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:22:54.519 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:54.519 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:22:54.519 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:53.781282+0000", 2024-09-13T11:22:54.519 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:54.519 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:54.519 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:54.520 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:22:54.520 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:53.781282+0000", 2024-09-13T11:22:54.520 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:54.520 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:54.520 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:54.520 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:22:54.520 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:53.781284+0000", 2024-09-13T11:22:54.520 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.548e-06 2024-09-13T11:22:54.520 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:54.520 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:54.520 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:22:54.520 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:53.781308+0000", 2024-09-13T11:22:54.520 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.4060999999999999e-05 2024-09-13T11:22:54.520 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:54.521 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:54.521 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:22:54.521 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:53.781325+0000", 2024-09-13T11:22:54.521 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.6775000000000001e-05 2024-09-13T11:22:54.521 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:54.521 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:54.521 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:22:54.521 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:53.781334+0000", 2024-09-13T11:22:54.521 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.0089999999999996e-06 2024-09-13T11:22:54.521 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:54.521 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:54.521 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:22:54.521 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:53.781417+0000", 2024-09-13T11:22:54.521 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.2817000000000003e-05 2024-09-13T11:22:54.522 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:54.522 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:54.522 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:22:54.522 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:53.781597+0000", 2024-09-13T11:22:54.522 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00018027000000000001 2024-09-13T11:22:54.522 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:54.522 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:54.522 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 13", 2024-09-13T11:22:54.522 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:53.781785+0000", 2024-09-13T11:22:54.522 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000188417 2024-09-13T11:22:54.522 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:54.522 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:54.522 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:22:54.523 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:53.782761+0000", 2024-09-13T11:22:54.523 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00097524299999999999 2024-09-13T11:22:54.523 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:54.523 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:54.523 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:22:54.523 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:53.826105+0000", 2024-09-13T11:22:54.523 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.043343918000000002 2024-09-13T11:22:54.523 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:54.523 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:54.523 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:22:54.523 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:53.826137+0000", 2024-09-13T11:22:54.524 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.2879999999999997e-05 2024-09-13T11:22:54.524 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:54.524 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:54.524 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:22:54.524 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:53.826168+0000", 2024-09-13T11:22:54.524 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.0719999999999997e-05 2024-09-13T11:22:54.524 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:54.524 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:22:54.524 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:54.524 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:54.524 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:54.524 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop_reply(client.4665.0:9111 3.c e47/30 ondisk, result = 0)", 2024-09-13T11:22:54.524 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:22:53.788062+0000", 2024-09-13T11:22:54.524 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 0.65412066000000002, 2024-09-13T11:22:54.525 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.038111977999999998, 2024-09-13T11:22:54.525 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:22:54.525 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:22:54.525 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:22:54.525 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:22:54.525 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:54.525 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:22:54.525 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:53.788062+0000", 2024-09-13T11:22:54.525 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:54.525 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:54.525 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:54.525 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:22:54.525 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:53.788062+0000", 2024-09-13T11:22:54.526 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:54.526 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:54.526 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:54.526 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:22:54.526 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:53.788063+0000", 2024-09-13T11:22:54.526 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.1799999999999999e-06 2024-09-13T11:22:54.526 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:54.526 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:54.526 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:22:54.526 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:53.788067+0000", 2024-09-13T11:22:54.526 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.7519999999999998e-06 2024-09-13T11:22:54.526 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:54.526 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:54.526 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:22:54.527 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:53.826033+0000", 2024-09-13T11:22:54.527 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.037966128000000002 2024-09-13T11:22:54.527 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:54.527 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:54.527 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:22:54.527 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:53.826044+0000", 2024-09-13T11:22:54.527 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.024e-05 2024-09-13T11:22:54.527 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:54.527 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:54.527 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:22:54.527 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:53.826097+0000", 2024-09-13T11:22:54.527 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.3154e-05 2024-09-13T11:22:54.527 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:54.527 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:54.528 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:22:54.528 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:53.826101+0000", 2024-09-13T11:22:54.528 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.476e-06 2024-09-13T11:22:54.528 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:54.528 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:54.528 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:22:54.528 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:53.826174+0000", 2024-09-13T11:22:54.528 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.3047999999999996e-05 2024-09-13T11:22:54.528 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:54.528 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:22:54.528 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:54.528 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:54.528 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:54.529 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:9601 3.f 3:fc233cd2:::benchmark_data_smithi177_31116_object9600:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-09-13T11:22:54.529 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:22:54.104008+0000", 2024-09-13T11:22:54.529 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 0.338174684, 2024-09-13T11:22:54.529 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.046355895000000001, 2024-09-13T11:22:54.529 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:22:54.529 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:22:54.529 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:54.529 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:22:54.529 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:22:54.529 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:22:54.529 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 9601 2024-09-13T11:22:54.529 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:54.529 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:22:54.530 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:54.530 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:22:54.530 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:54.104008+0000", 2024-09-13T11:22:54.530 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:54.530 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:54.530 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:54.530 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:22:54.530 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:54.104008+0000", 2024-09-13T11:22:54.530 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:54.530 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:54.530 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:54.530 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:22:54.530 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:54.104010+0000", 2024-09-13T11:22:54.530 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.621e-06 2024-09-13T11:22:54.530 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:54.531 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:54.531 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:22:54.531 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:54.104045+0000", 2024-09-13T11:22:54.531 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.5135999999999997e-05 2024-09-13T11:22:54.531 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:54.531 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:54.531 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:22:54.531 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:54.104059+0000", 2024-09-13T11:22:54.531 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.4171999999999999e-05 2024-09-13T11:22:54.531 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:54.531 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:54.531 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:22:54.531 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:54.104068+0000", 2024-09-13T11:22:54.532 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.1919999999999994e-06 2024-09-13T11:22:54.532 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:54.532 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:54.532 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:22:54.532 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:54.104153+0000", 2024-09-13T11:22:54.532 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.4703000000000005e-05 2024-09-13T11:22:54.532 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:54.532 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:54.532 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:22:54.532 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:54.104295+0000", 2024-09-13T11:22:54.532 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0001415 2024-09-13T11:22:54.532 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:54.532 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:54.533 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 12", 2024-09-13T11:22:54.533 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:54.104473+0000", 2024-09-13T11:22:54.533 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00017866599999999999 2024-09-13T11:22:54.533 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:54.533 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:54.533 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:22:54.533 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:54.105932+0000", 2024-09-13T11:22:54.533 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0014585449999999999 2024-09-13T11:22:54.533 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:54.533 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:54.533 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:22:54.533 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:54.150271+0000", 2024-09-13T11:22:54.533 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.044339159000000003 2024-09-13T11:22:54.533 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:54.533 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:54.534 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:22:54.534 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:54.150325+0000", 2024-09-13T11:22:54.534 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.4211e-05 2024-09-13T11:22:54.534 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:54.534 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:54.534 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:22:54.534 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:54.150364+0000", 2024-09-13T11:22:54.534 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.8989999999999998e-05 2024-09-13T11:22:54.534 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:54.534 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:22:54.534 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:54.534 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:54.534 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:54.535 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:9903 3.f 3:feb38778:::benchmark_data_smithi177_31116_object9902:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-09-13T11:22:54.535 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:22:54.299893+0000", 2024-09-13T11:22:54.535 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 0.14229025000000001, 2024-09-13T11:22:54.535 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.052386797999999998, 2024-09-13T11:22:54.535 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:22:54.535 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:22:54.535 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:54.535 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:22:54.535 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:22:54.535 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:22:54.535 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 9903 2024-09-13T11:22:54.535 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:54.535 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:22:54.536 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:54.536 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:22:54.536 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:54.299893+0000", 2024-09-13T11:22:54.536 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:54.536 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:54.536 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:54.536 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:22:54.536 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:54.299893+0000", 2024-09-13T11:22:54.536 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:54.536 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:54.536 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:54.536 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:22:54.536 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:54.299895+0000", 2024-09-13T11:22:54.537 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.3039999999999999e-06 2024-09-13T11:22:54.537 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:54.537 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:54.537 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:22:54.537 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:54.299917+0000", 2024-09-13T11:22:54.537 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.2106000000000001e-05 2024-09-13T11:22:54.537 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:54.537 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:54.537 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:22:54.537 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:54.299932+0000", 2024-09-13T11:22:54.537 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5014999999999999e-05 2024-09-13T11:22:54.537 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:54.537 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:54.537 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:22:54.538 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:54.299940+0000", 2024-09-13T11:22:54.538 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.4549999999999998e-06 2024-09-13T11:22:54.538 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:54.538 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:54.538 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:22:54.538 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:54.300014+0000", 2024-09-13T11:22:54.538 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.4845999999999995e-05 2024-09-13T11:22:54.538 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:54.538 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:54.538 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:22:54.538 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:54.300170+0000", 2024-09-13T11:22:54.538 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00015520200000000001 2024-09-13T11:22:54.538 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:54.538 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:54.539 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 12", 2024-09-13T11:22:54.539 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:54.300305+0000", 2024-09-13T11:22:54.539 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00013571899999999999 2024-09-13T11:22:54.539 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:54.539 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:54.539 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:22:54.539 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:54.301265+0000", 2024-09-13T11:22:54.539 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00096003699999999996 2024-09-13T11:22:54.539 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:54.539 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:54.539 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:22:54.539 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:54.352206+0000", 2024-09-13T11:22:54.539 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.050941113000000003 2024-09-13T11:22:54.539 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:54.540 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:54.540 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:22:54.540 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:54.352242+0000", 2024-09-13T11:22:54.540 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.5340999999999999e-05 2024-09-13T11:22:54.540 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:54.540 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:54.540 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:22:54.540 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:54.352279+0000", 2024-09-13T11:22:54.540 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.7660000000000002e-05 2024-09-13T11:22:54.540 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:54.540 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:22:54.540 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:54.540 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:54.541 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:54.541 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:9906 3.f 3:fea9a3d3:::benchmark_data_smithi177_31116_object9905:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-09-13T11:22:54.541 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:22:54.302935+0000", 2024-09-13T11:22:54.541 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 0.13924769400000001, 2024-09-13T11:22:54.541 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.076725138999999998, 2024-09-13T11:22:54.541 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:22:54.541 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:22:54.541 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:54.541 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:22:54.541 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:22:54.541 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:22:54.541 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 9906 2024-09-13T11:22:54.541 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:54.541 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:22:54.542 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:54.542 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:22:54.542 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:54.302935+0000", 2024-09-13T11:22:54.542 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:54.542 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:54.542 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:54.542 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:22:54.542 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:54.302935+0000", 2024-09-13T11:22:54.542 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:54.542 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:54.542 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:54.542 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:22:54.542 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:54.302937+0000", 2024-09-13T11:22:54.542 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.739e-06 2024-09-13T11:22:54.543 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:54.543 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:54.543 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:22:54.543 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:54.302962+0000", 2024-09-13T11:22:54.543 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.4578000000000001e-05 2024-09-13T11:22:54.543 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:54.543 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:54.543 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:22:54.543 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:54.302973+0000", 2024-09-13T11:22:54.543 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.1949e-05 2024-09-13T11:22:54.543 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:54.543 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:54.543 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:22:54.543 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:54.302983+0000", 2024-09-13T11:22:54.544 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.2110000000000004e-06 2024-09-13T11:22:54.544 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:54.544 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:54.544 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:22:54.544 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:54.303039+0000", 2024-09-13T11:22:54.544 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.6409999999999997e-05 2024-09-13T11:22:54.544 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:54.544 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:54.544 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:22:54.544 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:54.303185+0000", 2024-09-13T11:22:54.544 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00014623500000000001 2024-09-13T11:22:54.544 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:54.544 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:54.545 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 12", 2024-09-13T11:22:54.545 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:54.303343+0000", 2024-09-13T11:22:54.545 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00015808799999999999 2024-09-13T11:22:54.545 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:54.545 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:54.545 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:22:54.545 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:54.304222+0000", 2024-09-13T11:22:54.545 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00087887900000000005 2024-09-13T11:22:54.545 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:54.545 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:54.545 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:22:54.545 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:54.379556+0000", 2024-09-13T11:22:54.545 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.075334071000000002 2024-09-13T11:22:54.545 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:54.545 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:54.546 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:22:54.546 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:54.379613+0000", 2024-09-13T11:22:54.546 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.6459000000000003e-05 2024-09-13T11:22:54.546 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:54.546 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:54.546 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:22:54.546 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:54.379660+0000", 2024-09-13T11:22:54.546 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.7519999999999999e-05 2024-09-13T11:22:54.546 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:54.546 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:22:54.546 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:54.546 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:54.546 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:54.546 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop_reply(client.4665.0:9903 3.f e47/30 ondisk, result = 0)", 2024-09-13T11:22:54.547 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:22:54.305432+0000", 2024-09-13T11:22:54.547 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 0.136751346, 2024-09-13T11:22:54.547 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.046852920999999999, 2024-09-13T11:22:54.547 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:22:54.547 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:22:54.547 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:22:54.547 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:22:54.547 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:54.547 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:22:54.547 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:54.305432+0000", 2024-09-13T11:22:54.547 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:54.547 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:54.547 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:54.547 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:22:54.548 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:54.305432+0000", 2024-09-13T11:22:54.548 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:54.548 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:54.548 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:54.548 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:22:54.548 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:54.305433+0000", 2024-09-13T11:22:54.548 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.4899999999999999e-06 2024-09-13T11:22:54.548 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:54.548 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:54.548 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:22:54.548 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:54.305439+0000", 2024-09-13T11:22:54.548 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.5779999999999998e-06 2024-09-13T11:22:54.548 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:54.549 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:54.549 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:22:54.549 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:54.352138+0000", 2024-09-13T11:22:54.549 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.046699362000000001 2024-09-13T11:22:54.549 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:54.549 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:54.549 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:22:54.549 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:54.352144+0000", 2024-09-13T11:22:54.549 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.37e-06 2024-09-13T11:22:54.549 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:54.549 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:54.549 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:22:54.549 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:54.352197+0000", 2024-09-13T11:22:54.549 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.2290999999999998e-05 2024-09-13T11:22:54.550 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:54.550 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:54.550 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:22:54.550 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:54.352203+0000", 2024-09-13T11:22:54.550 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.3820000000000001e-06 2024-09-13T11:22:54.550 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:54.550 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:54.550 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:22:54.550 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:54.352284+0000", 2024-09-13T11:22:54.550 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.1447999999999997e-05 2024-09-13T11:22:54.550 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:54.550 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:22:54.550 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:54.550 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:54.551 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:22:54.551 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-09-13T11:22:54.552 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_ops_in_flight 2024-09-13T11:22:54.553 INFO:tasks.ceph.osd.5.smithi092.stderr:2024-09-13T11:22:54.550+0000 7f6f0b705640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26346) UID: 0 2024-09-13T11:22:54.653 INFO:tasks.ceph.osd.1.smithi092.stderr:2024-09-13T11:22:54.650+0000 7fe7789c5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33718) UID: 0 2024-09-13T11:22:54.727 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-09-13T11:22:54.728 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [ 2024-09-13T11:22:54.728 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:54.728 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:10597 3.5 3:ab149e7d:::benchmark_data_smithi177_31116_object10596:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e48)", 2024-09-13T11:22:54.728 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:22:54.726381+0000", 2024-09-13T11:22:54.728 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 0.0024855609999999998, 2024-09-13T11:22:54.728 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0025047060000000002, 2024-09-13T11:22:54.728 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:22:54.728 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:22:54.728 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "waiting for sub ops", 2024-09-13T11:22:54.728 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:22:54.729 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:22:54.729 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:22:54.729 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 10597 2024-09-13T11:22:54.729 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:54.729 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:22:54.729 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:54.729 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:22:54.729 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:54.726381+0000", 2024-09-13T11:22:54.729 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:54.729 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:54.729 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:54.729 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:22:54.730 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:54.726381+0000", 2024-09-13T11:22:54.730 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:54.730 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:54.730 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:54.730 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:22:54.730 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:54.726383+0000", 2024-09-13T11:22:54.730 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.2479999999999999e-06 2024-09-13T11:22:54.730 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:54.730 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:54.730 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:22:54.730 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:54.726409+0000", 2024-09-13T11:22:54.730 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.5897999999999998e-05 2024-09-13T11:22:54.730 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:54.730 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:54.730 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:22:54.731 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:54.726422+0000", 2024-09-13T11:22:54.731 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.2969999999999999e-05 2024-09-13T11:22:54.731 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:54.731 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:54.731 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:22:54.731 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:54.726430+0000", 2024-09-13T11:22:54.731 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.6230000000000003e-06 2024-09-13T11:22:54.731 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:54.731 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:54.731 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:22:54.731 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:54.726506+0000", 2024-09-13T11:22:54.731 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.5271999999999994e-05 2024-09-13T11:22:54.731 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:54.732 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:54.732 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:22:54.732 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:54.726689+0000", 2024-09-13T11:22:54.732 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000182948 2024-09-13T11:22:54.732 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:54.732 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:54.732 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 13", 2024-09-13T11:22:54.732 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:54.726891+0000", 2024-09-13T11:22:54.732 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00020272399999999999 2024-09-13T11:22:54.732 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:54.732 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:54.732 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:22:54.732 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:54.728341+0000", 2024-09-13T11:22:54.732 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0014499140000000001 2024-09-13T11:22:54.733 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:54.733 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:22:54.733 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:54.733 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:54.733 INFO:teuthology.orchestra.run.smithi177.stdout: ], 2024-09-13T11:22:54.733 INFO:teuthology.orchestra.run.smithi177.stdout: "num_ops": 1 2024-09-13T11:22:54.733 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-09-13T11:22:54.742 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_blocked_ops 2024-09-13T11:22:54.754 INFO:tasks.ceph.osd.8.smithi001.stderr:2024-09-13T11:22:54.748+0000 7fb9b48ae640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26312) UID: 0 2024-09-13T11:22:54.854 INFO:tasks.ceph.osd.4.smithi001.stderr:2024-09-13T11:22:54.848+0000 7ff8e0218640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26310) UID: 0 2024-09-13T11:22:54.871 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-09-13T11:22:54.872 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [], 2024-09-13T11:22:54.872 INFO:teuthology.orchestra.run.smithi177.stdout: "complaint_time": 30, 2024-09-13T11:22:54.872 INFO:teuthology.orchestra.run.smithi177.stdout: "num_blocked_ops": 0 2024-09-13T11:22:54.872 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-09-13T11:22:54.886 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_historic_ops 2024-09-13T11:22:54.887 INFO:tasks.ceph.osd.5.smithi092.stderr:2024-09-13T11:22:54.882+0000 7f6f00132640 -1 osd.5 47 handle_osd_map injecting map crc failure 2024-09-13T11:22:54.955 INFO:tasks.ceph.osd.14.smithi162.stderr:2024-09-13T11:22:54.951+0000 7f235c0db640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26319) UID: 0 2024-09-13T11:22:54.999 INFO:tasks.radosbench.radosbench.0.smithi177.stdout: 28 16 10819 10803 24.1107 85.8125 0.0155494 0.0414443 2024-09-13T11:22:55.011 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-09-13T11:22:55.011 INFO:teuthology.orchestra.run.smithi177.stdout: "size": 20, 2024-09-13T11:22:55.011 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 600, 2024-09-13T11:22:55.011 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [ 2024-09-13T11:22:55.011 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:55.011 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:820 3.5 3:ac53318b:::benchmark_data_smithi177_31116_object819:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-09-13T11:22:55.011 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:22:49.450879+0000", 2024-09-13T11:22:55.011 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 5.5581718259999997, 2024-09-13T11:22:55.011 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.042467982000000001, 2024-09-13T11:22:55.011 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:22:55.011 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:22:55.011 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:55.012 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:22:55.012 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:22:55.012 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:22:55.012 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 820 2024-09-13T11:22:55.012 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:55.012 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:22:55.012 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:55.012 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:22:55.012 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.450879+0000", 2024-09-13T11:22:55.012 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:55.012 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:55.012 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:55.012 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:22:55.013 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.450879+0000", 2024-09-13T11:22:55.013 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:55.013 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:55.013 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:55.013 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:22:55.013 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.450881+0000", 2024-09-13T11:22:55.013 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.069e-06 2024-09-13T11:22:55.013 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:55.013 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:55.013 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:22:55.013 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.450912+0000", 2024-09-13T11:22:55.013 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.0465000000000001e-05 2024-09-13T11:22:55.013 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:55.014 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:55.014 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:22:55.014 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.450923+0000", 2024-09-13T11:22:55.014 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.1905e-05 2024-09-13T11:22:55.014 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:55.014 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:55.014 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:22:55.014 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.450931+0000", 2024-09-13T11:22:55.014 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.0679999999999999e-06 2024-09-13T11:22:55.014 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:55.014 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:55.014 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:22:55.014 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.450984+0000", 2024-09-13T11:22:55.014 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.3356000000000002e-05 2024-09-13T11:22:55.014 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:55.015 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:55.015 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:22:55.015 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.451057+0000", 2024-09-13T11:22:55.015 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.3153999999999998e-05 2024-09-13T11:22:55.015 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:55.015 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:55.015 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 13", 2024-09-13T11:22:55.015 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.451168+0000", 2024-09-13T11:22:55.015 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000110598 2024-09-13T11:22:55.015 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:55.015 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:55.016 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:22:55.016 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.451942+0000", 2024-09-13T11:22:55.016 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00077421800000000002 2024-09-13T11:22:55.016 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:55.016 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:55.016 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:22:55.016 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.493261+0000", 2024-09-13T11:22:55.016 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.041318724000000001 2024-09-13T11:22:55.016 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:55.016 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:55.016 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:22:55.016 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.493306+0000", 2024-09-13T11:22:55.016 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.5575999999999999e-05 2024-09-13T11:22:55.016 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:55.017 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:55.017 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:22:55.017 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.493347+0000", 2024-09-13T11:22:55.017 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.0849000000000001e-05 2024-09-13T11:22:55.017 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:55.017 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:22:55.017 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:55.017 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:55.017 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:55.018 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop_reply(client.4665.0:820 3.5 e43/30 ondisk, result = 0)", 2024-09-13T11:22:55.018 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:22:49.453727+0000", 2024-09-13T11:22:55.018 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 5.5553240739999996, 2024-09-13T11:22:55.018 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.039631573000000003, 2024-09-13T11:22:55.018 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:22:55.018 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:22:55.018 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:22:55.018 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:22:55.018 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:55.018 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:22:55.018 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.453727+0000", 2024-09-13T11:22:55.018 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:55.018 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:55.018 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:55.019 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:22:55.019 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.453727+0000", 2024-09-13T11:22:55.019 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:55.019 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:55.019 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:55.019 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:22:55.019 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.453727+0000", 2024-09-13T11:22:55.019 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.1699999999999998e-07 2024-09-13T11:22:55.019 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:55.020 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:55.020 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:22:55.020 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.453730+0000", 2024-09-13T11:22:55.020 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.3989999999999998e-06 2024-09-13T11:22:55.020 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:55.020 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:55.020 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:22:55.020 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.493183+0000", 2024-09-13T11:22:55.020 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.039453467999999998 2024-09-13T11:22:55.020 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:55.020 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:55.020 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:22:55.020 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.493194+0000", 2024-09-13T11:22:55.021 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.1065e-05 2024-09-13T11:22:55.021 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:55.021 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:55.021 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:22:55.021 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.493249+0000", 2024-09-13T11:22:55.021 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.5010999999999999e-05 2024-09-13T11:22:55.021 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:55.021 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:55.021 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:22:55.021 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.493256+0000", 2024-09-13T11:22:55.021 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.7839999999999998e-06 2024-09-13T11:22:55.021 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:55.021 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:55.022 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:22:55.022 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.493358+0000", 2024-09-13T11:22:55.022 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000102229 2024-09-13T11:22:55.022 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:55.022 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:22:55.022 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:55.022 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:55.022 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:55.022 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:1045 3.5 3:aca982b9:::benchmark_data_smithi177_31116_object1044:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-09-13T11:22:55.022 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:22:49.650700+0000", 2024-09-13T11:22:55.022 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 5.3583504089999998, 2024-09-13T11:22:55.022 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.055297286000000001, 2024-09-13T11:22:55.022 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:22:55.023 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:22:55.023 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:55.023 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:22:55.023 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:22:55.023 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:22:55.023 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 1045 2024-09-13T11:22:55.023 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:55.023 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:22:55.023 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:55.023 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:22:55.024 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.650700+0000", 2024-09-13T11:22:55.024 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:55.024 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:55.024 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:55.024 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:22:55.024 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.650700+0000", 2024-09-13T11:22:55.024 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:55.024 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:55.024 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:55.024 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:22:55.024 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.650702+0000", 2024-09-13T11:22:55.024 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.127e-06 2024-09-13T11:22:55.025 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:55.025 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:55.025 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:22:55.025 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.650746+0000", 2024-09-13T11:22:55.025 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.4097000000000001e-05 2024-09-13T11:22:55.025 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:55.025 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:55.025 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:22:55.025 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.650755+0000", 2024-09-13T11:22:55.025 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.0310000000000004e-06 2024-09-13T11:22:55.025 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:55.025 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:55.025 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:22:55.025 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.650760+0000", 2024-09-13T11:22:55.026 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.3929999999999999e-06 2024-09-13T11:22:55.026 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:55.026 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:55.026 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:22:55.026 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.651076+0000", 2024-09-13T11:22:55.026 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00031611900000000001 2024-09-13T11:22:55.026 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:55.026 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:55.026 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:22:55.026 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.651182+0000", 2024-09-13T11:22:55.026 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00010571199999999999 2024-09-13T11:22:55.026 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:55.026 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:55.027 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 13", 2024-09-13T11:22:55.027 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.651294+0000", 2024-09-13T11:22:55.027 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000112112 2024-09-13T11:22:55.027 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:55.027 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:55.027 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:22:55.027 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.654078+0000", 2024-09-13T11:22:55.027 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.002783832 2024-09-13T11:22:55.027 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:55.027 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:55.027 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:22:55.027 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.705758+0000", 2024-09-13T11:22:55.027 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.051680333000000002 2024-09-13T11:22:55.027 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:55.028 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:55.028 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:22:55.028 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.705896+0000", 2024-09-13T11:22:55.028 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00013765000000000001 2024-09-13T11:22:55.028 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:55.028 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:55.028 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:22:55.028 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.705998+0000", 2024-09-13T11:22:55.028 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00010187999999999999 2024-09-13T11:22:55.028 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:55.028 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:22:55.028 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:55.028 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:55.028 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:55.029 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop_reply(client.4665.0:1045 3.5 e43/30 ondisk, result = 0)", 2024-09-13T11:22:55.029 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:22:49.652166+0000", 2024-09-13T11:22:55.029 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 5.3568845009999997, 2024-09-13T11:22:55.029 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.053848380000000001, 2024-09-13T11:22:55.029 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:22:55.029 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:22:55.029 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:22:55.029 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:22:55.029 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:55.029 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:22:55.029 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.652166+0000", 2024-09-13T11:22:55.029 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:55.029 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:55.029 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:55.030 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:22:55.030 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.652166+0000", 2024-09-13T11:22:55.030 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:55.030 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:55.030 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:55.030 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:22:55.030 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.652168+0000", 2024-09-13T11:22:55.030 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.393e-06 2024-09-13T11:22:55.030 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:55.030 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:55.030 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:22:55.030 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.652172+0000", 2024-09-13T11:22:55.030 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.2989999999999998e-06 2024-09-13T11:22:55.030 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:55.030 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:55.031 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:22:55.031 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.705604+0000", 2024-09-13T11:22:55.031 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.053431575000000002 2024-09-13T11:22:55.031 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:55.031 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:55.031 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:22:55.031 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.705623+0000", 2024-09-13T11:22:55.031 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.9479000000000001e-05 2024-09-13T11:22:55.031 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:55.031 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:55.031 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:22:55.031 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.705728+0000", 2024-09-13T11:22:55.031 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000105338 2024-09-13T11:22:55.031 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:55.032 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:55.032 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:22:55.032 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.705746+0000", 2024-09-13T11:22:55.032 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.7645000000000001e-05 2024-09-13T11:22:55.032 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:55.032 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:55.032 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:22:55.032 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.706015+0000", 2024-09-13T11:22:55.032 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00026865100000000001 2024-09-13T11:22:55.032 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:55.032 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:22:55.032 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:55.032 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:55.033 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:55.033 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:1331 3.5 3:a75b3c55:::benchmark_data_smithi177_31116_object1330:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-09-13T11:22:55.033 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:22:49.866827+0000", 2024-09-13T11:22:55.033 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 5.1422235890000003, 2024-09-13T11:22:55.033 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.033373582999999998, 2024-09-13T11:22:55.033 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:22:55.033 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:22:55.033 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:55.033 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:22:55.033 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:22:55.033 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:22:55.033 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 1331 2024-09-13T11:22:55.033 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:55.034 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:22:55.034 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:55.034 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:22:55.034 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.866827+0000", 2024-09-13T11:22:55.034 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:55.034 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:55.034 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:55.034 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:22:55.034 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.866827+0000", 2024-09-13T11:22:55.034 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:55.034 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:55.034 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:55.034 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:22:55.034 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.866831+0000", 2024-09-13T11:22:55.035 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.5760000000000002e-06 2024-09-13T11:22:55.035 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:55.035 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:55.035 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:22:55.035 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.866881+0000", 2024-09-13T11:22:55.035 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.0454999999999997e-05 2024-09-13T11:22:55.035 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:55.035 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:55.035 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:22:55.035 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.866909+0000", 2024-09-13T11:22:55.035 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.7263999999999998e-05 2024-09-13T11:22:55.035 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:55.035 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:55.035 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:22:55.036 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.866926+0000", 2024-09-13T11:22:55.036 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.7711999999999999e-05 2024-09-13T11:22:55.036 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:55.036 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:55.036 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:22:55.036 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.867035+0000", 2024-09-13T11:22:55.036 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000108392 2024-09-13T11:22:55.036 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:55.036 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:55.036 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:22:55.036 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.867175+0000", 2024-09-13T11:22:55.036 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00014055 2024-09-13T11:22:55.036 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:55.036 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:55.037 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 13", 2024-09-13T11:22:55.037 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.867365+0000", 2024-09-13T11:22:55.037 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00018971299999999999 2024-09-13T11:22:55.037 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:55.037 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:55.037 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:22:55.037 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.868553+0000", 2024-09-13T11:22:55.037 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0011885680000000001 2024-09-13T11:22:55.037 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:55.037 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:55.037 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:22:55.037 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.900054+0000", 2024-09-13T11:22:55.037 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.031500901999999997 2024-09-13T11:22:55.037 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:55.038 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:55.038 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:22:55.038 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.900144+0000", 2024-09-13T11:22:55.040 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.9449000000000004e-05 2024-09-13T11:22:55.040 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:55.040 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:55.040 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:22:55.040 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.900201+0000", 2024-09-13T11:22:55.040 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.7002000000000001e-05 2024-09-13T11:22:55.040 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:55.040 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:22:55.040 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:55.040 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:55.040 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:55.040 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop_reply(client.4665.0:1331 3.5 e43/30 ondisk, result = 0)", 2024-09-13T11:22:55.041 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:22:49.871436+0000", 2024-09-13T11:22:55.041 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 5.1376145339999999, 2024-09-13T11:22:55.041 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.028774013000000001, 2024-09-13T11:22:55.041 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:22:55.041 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:22:55.041 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:22:55.041 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:22:55.041 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:55.041 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:22:55.041 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.871436+0000", 2024-09-13T11:22:55.041 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:55.041 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:55.041 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:55.041 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:22:55.042 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.871436+0000", 2024-09-13T11:22:55.042 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:55.042 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:55.042 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:55.042 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:22:55.042 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.871438+0000", 2024-09-13T11:22:55.042 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.9879999999999999e-06 2024-09-13T11:22:55.042 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:55.042 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:55.042 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:22:55.042 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.871445+0000", 2024-09-13T11:22:55.042 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.246e-06 2024-09-13T11:22:55.042 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:55.043 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:55.043 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:22:55.043 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.899956+0000", 2024-09-13T11:22:55.043 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.028511410000000001 2024-09-13T11:22:55.043 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:55.043 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:55.043 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:22:55.043 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.899968+0000", 2024-09-13T11:22:55.043 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.2534000000000001e-05 2024-09-13T11:22:55.044 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:55.044 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:55.044 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:22:55.044 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.900039+0000", 2024-09-13T11:22:55.044 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.0640000000000001e-05 2024-09-13T11:22:55.044 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:55.044 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:55.044 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:22:55.044 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.900048+0000", 2024-09-13T11:22:55.044 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.1400000000000006e-06 2024-09-13T11:22:55.044 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:55.044 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:55.045 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:22:55.045 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.900210+0000", 2024-09-13T11:22:55.045 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00016205500000000001 2024-09-13T11:22:55.045 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:55.045 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:22:55.045 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:55.045 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:55.045 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:55.045 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:8976 3.5 3:a4524ba3:::benchmark_data_smithi177_31116_object8975:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-09-13T11:22:55.045 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:22:53.681954+0000", 2024-09-13T11:22:55.045 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 1.327096534, 2024-09-13T11:22:55.045 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.033650986000000001, 2024-09-13T11:22:55.045 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:22:55.045 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:22:55.046 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:55.046 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:22:55.046 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:22:55.046 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:22:55.046 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 8976 2024-09-13T11:22:55.046 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:55.046 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:22:55.046 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:55.046 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:22:55.046 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:53.681954+0000", 2024-09-13T11:22:55.047 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:55.047 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:55.047 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:55.047 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:22:55.047 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:53.681954+0000", 2024-09-13T11:22:55.047 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:55.047 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:55.047 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:55.047 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:22:55.048 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:53.681957+0000", 2024-09-13T11:22:55.048 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.1350000000000001e-06 2024-09-13T11:22:55.048 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:55.048 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:55.048 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:22:55.048 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:53.681988+0000", 2024-09-13T11:22:55.048 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.0587999999999999e-05 2024-09-13T11:22:55.048 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:55.048 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:55.048 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:22:55.048 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:53.682005+0000", 2024-09-13T11:22:55.048 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.7431e-05 2024-09-13T11:22:55.048 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:55.049 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:55.049 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:22:55.049 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:53.682015+0000", 2024-09-13T11:22:55.049 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.3379999999999994e-06 2024-09-13T11:22:55.049 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:55.049 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:55.049 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:22:55.049 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:53.682071+0000", 2024-09-13T11:22:55.049 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.6585000000000001e-05 2024-09-13T11:22:55.049 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:55.049 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:55.049 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:22:55.049 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:53.682184+0000", 2024-09-13T11:22:55.050 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000112614 2024-09-13T11:22:55.050 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:55.050 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:55.050 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 13", 2024-09-13T11:22:55.050 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:53.682284+0000", 2024-09-13T11:22:55.050 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000100386 2024-09-13T11:22:55.050 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:55.050 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:55.050 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:22:55.050 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:53.683106+0000", 2024-09-13T11:22:55.050 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00082139099999999998 2024-09-13T11:22:55.050 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:55.050 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:55.050 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:22:55.051 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:53.715530+0000", 2024-09-13T11:22:55.051 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.032424462000000001 2024-09-13T11:22:55.051 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:55.051 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:55.051 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:22:55.051 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:53.715575+0000", 2024-09-13T11:22:55.051 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.4632000000000002e-05 2024-09-13T11:22:55.051 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:55.051 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:55.051 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:22:55.051 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:53.715605+0000", 2024-09-13T11:22:55.051 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.0423999999999999e-05 2024-09-13T11:22:55.051 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:55.051 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:22:55.052 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:55.052 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:55.052 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:55.052 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop_reply(client.4665.0:8976 3.5 e47/30 ondisk, result = 0)", 2024-09-13T11:22:55.052 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:22:53.683954+0000", 2024-09-13T11:22:55.052 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 1.3250971950000001, 2024-09-13T11:22:55.052 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.031666866000000002, 2024-09-13T11:22:55.052 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:22:55.052 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:22:55.052 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:22:55.052 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:22:55.052 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:55.052 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:22:55.053 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:53.683954+0000", 2024-09-13T11:22:55.053 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:55.053 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:55.053 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:55.053 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:22:55.053 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:53.683954+0000", 2024-09-13T11:22:55.053 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:55.053 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:55.053 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:55.053 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:22:55.053 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:53.683955+0000", 2024-09-13T11:22:55.053 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.683e-06 2024-09-13T11:22:55.054 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:55.054 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:55.054 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:22:55.054 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:53.683960+0000", 2024-09-13T11:22:55.054 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.5879999999999999e-06 2024-09-13T11:22:55.054 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:55.054 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:55.054 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:22:55.054 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:53.715462+0000", 2024-09-13T11:22:55.054 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.03150232 2024-09-13T11:22:55.054 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:55.054 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:55.054 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:22:55.054 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:53.715469+0000", 2024-09-13T11:22:55.054 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.075e-06 2024-09-13T11:22:55.055 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:55.055 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:55.055 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:22:55.055 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:53.715520+0000", 2024-09-13T11:22:55.055 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.1122000000000003e-05 2024-09-13T11:22:55.055 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:55.055 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:55.055 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:22:55.055 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:53.715525+0000", 2024-09-13T11:22:55.055 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.6580000000000001e-06 2024-09-13T11:22:55.055 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:55.055 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:55.055 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:22:55.056 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:53.715621+0000", 2024-09-13T11:22:55.056 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.5420000000000005e-05 2024-09-13T11:22:55.056 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:55.056 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:22:55.056 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:55.056 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:55.056 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:55.056 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:9014 3.5 3:aec307b2:::benchmark_data_smithi177_31116_object9013:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-09-13T11:22:55.056 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:22:53.716117+0000", 2024-09-13T11:22:55.056 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 1.292933508, 2024-09-13T11:22:55.056 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.029863041, 2024-09-13T11:22:55.056 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:22:55.056 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:22:55.056 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:55.057 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:22:55.057 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:22:55.057 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:22:55.057 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 9014 2024-09-13T11:22:55.057 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:55.057 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:22:55.057 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:55.057 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:22:55.057 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:53.716117+0000", 2024-09-13T11:22:55.057 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:55.057 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:55.057 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:55.057 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:22:55.058 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:53.716117+0000", 2024-09-13T11:22:55.058 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:55.058 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:55.058 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:55.058 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:22:55.058 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:53.716118+0000", 2024-09-13T11:22:55.058 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0640000000000001e-06 2024-09-13T11:22:55.058 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:55.058 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:55.058 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:22:55.058 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:53.716152+0000", 2024-09-13T11:22:55.058 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.3416e-05 2024-09-13T11:22:55.058 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:55.058 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:55.059 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:22:55.059 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:53.716160+0000", 2024-09-13T11:22:55.059 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.5080000000000005e-06 2024-09-13T11:22:55.059 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:55.059 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:55.059 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:22:55.059 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:53.716166+0000", 2024-09-13T11:22:55.059 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.3550000000000004e-06 2024-09-13T11:22:55.059 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:55.059 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:55.059 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:22:55.059 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:53.716199+0000", 2024-09-13T11:22:55.059 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.3427999999999999e-05 2024-09-13T11:22:55.060 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:55.060 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:55.060 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:22:55.060 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:53.716314+0000", 2024-09-13T11:22:55.060 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000115374 2024-09-13T11:22:55.060 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:55.060 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:55.060 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 13", 2024-09-13T11:22:55.060 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:53.716407+0000", 2024-09-13T11:22:55.060 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.2431999999999999e-05 2024-09-13T11:22:55.060 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:55.060 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:55.060 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:22:55.060 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:53.717190+0000", 2024-09-13T11:22:55.061 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00078350200000000005 2024-09-13T11:22:55.061 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:55.061 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:55.061 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:22:55.061 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:53.745893+0000", 2024-09-13T11:22:55.061 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.028702264000000002 2024-09-13T11:22:55.061 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:55.061 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:55.061 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:22:55.061 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:53.745943+0000", 2024-09-13T11:22:55.061 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.0382e-05 2024-09-13T11:22:55.061 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:55.061 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:55.061 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:22:55.062 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:53.745980+0000", 2024-09-13T11:22:55.062 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.7316e-05 2024-09-13T11:22:55.062 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:55.062 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:22:55.062 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:55.062 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:55.062 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:55.062 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:9099 3.5 3:acd61533:::benchmark_data_smithi177_31116_object9098:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-09-13T11:22:55.062 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:22:53.770313+0000", 2024-09-13T11:22:55.062 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 1.2387381230000001, 2024-09-13T11:22:55.062 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.045420343000000002, 2024-09-13T11:22:55.062 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:22:55.062 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:22:55.062 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:55.063 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:22:55.063 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:22:55.063 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:22:55.063 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 9099 2024-09-13T11:22:55.063 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:55.063 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:22:55.063 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:55.063 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:22:55.063 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:53.770313+0000", 2024-09-13T11:22:55.063 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:55.063 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:55.063 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:55.063 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:22:55.063 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:53.770313+0000", 2024-09-13T11:22:55.064 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:55.064 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:55.064 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:55.064 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:22:55.064 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:53.770314+0000", 2024-09-13T11:22:55.064 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0189999999999999e-06 2024-09-13T11:22:55.064 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:55.064 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:55.064 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:22:55.064 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:53.770334+0000", 2024-09-13T11:22:55.064 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.9899999999999999e-05 2024-09-13T11:22:55.064 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:55.065 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:55.065 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:22:55.065 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:53.770342+0000", 2024-09-13T11:22:55.065 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.8419999999999994e-06 2024-09-13T11:22:55.065 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:55.065 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:55.065 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:22:55.065 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:53.770347+0000", 2024-09-13T11:22:55.065 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.9660000000000004e-06 2024-09-13T11:22:55.065 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:55.065 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:55.065 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:22:55.066 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:53.770403+0000", 2024-09-13T11:22:55.066 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.5185000000000001e-05 2024-09-13T11:22:55.066 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:55.066 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:55.066 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:22:55.066 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:53.770538+0000", 2024-09-13T11:22:55.066 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000134974 2024-09-13T11:22:55.066 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:55.066 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:55.066 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 13", 2024-09-13T11:22:55.066 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:53.770632+0000", 2024-09-13T11:22:55.066 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.4346999999999996e-05 2024-09-13T11:22:55.066 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:55.067 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:55.067 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:22:55.067 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:53.771563+0000", 2024-09-13T11:22:55.067 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00093067300000000003 2024-09-13T11:22:55.067 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:55.067 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:55.067 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:22:55.067 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:53.815623+0000", 2024-09-13T11:22:55.067 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.044060296999999998 2024-09-13T11:22:55.067 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:55.067 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:55.067 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:22:55.067 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:53.815683+0000", 2024-09-13T11:22:55.068 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.9682000000000002e-05 2024-09-13T11:22:55.068 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:55.068 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:55.068 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:22:55.068 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:53.815733+0000", 2024-09-13T11:22:55.068 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.0457999999999997e-05 2024-09-13T11:22:55.068 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:55.068 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:22:55.068 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:55.068 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:55.068 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:55.068 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop_reply(client.4665.0:9099 3.5 e47/30 ondisk, result = 0)", 2024-09-13T11:22:55.068 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:22:53.774767+0000", 2024-09-13T11:22:55.069 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 1.2342839130000001, 2024-09-13T11:22:55.069 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.040976123000000003, 2024-09-13T11:22:55.069 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:22:55.069 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:22:55.069 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:22:55.069 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:22:55.069 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:55.069 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:22:55.069 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:53.774767+0000", 2024-09-13T11:22:55.069 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:55.069 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:55.069 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:55.069 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:22:55.070 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:53.774767+0000", 2024-09-13T11:22:55.070 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:55.070 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:55.070 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:55.070 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:22:55.070 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:53.774769+0000", 2024-09-13T11:22:55.070 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.579e-06 2024-09-13T11:22:55.070 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:55.070 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:55.070 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:22:55.070 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:53.774773+0000", 2024-09-13T11:22:55.070 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.7090000000000001e-06 2024-09-13T11:22:55.070 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:55.070 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:55.071 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:22:55.071 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:53.815552+0000", 2024-09-13T11:22:55.071 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.040778877999999998 2024-09-13T11:22:55.071 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:55.071 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:55.071 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:22:55.071 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:53.815557+0000", 2024-09-13T11:22:55.071 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.0050000000000004e-06 2024-09-13T11:22:55.071 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:55.071 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:55.071 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:22:55.071 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:53.815608+0000", 2024-09-13T11:22:55.071 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.0565000000000001e-05 2024-09-13T11:22:55.071 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:55.072 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:55.072 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:22:55.072 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:53.815616+0000", 2024-09-13T11:22:55.072 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.7900000000000005e-06 2024-09-13T11:22:55.072 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:55.072 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:55.072 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:22:55.072 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:53.815743+0000", 2024-09-13T11:22:55.072 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00012659699999999999 2024-09-13T11:22:55.072 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:55.072 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:22:55.072 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:55.072 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:55.072 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:55.073 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:9187 3.5 3:a591d48f:::benchmark_data_smithi177_31116_object9186:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-09-13T11:22:55.073 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:22:53.834981+0000", 2024-09-13T11:22:55.073 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 1.174070079, 2024-09-13T11:22:55.073 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.030203219999999999, 2024-09-13T11:22:55.073 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:22:55.073 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:22:55.073 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:55.073 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:22:55.073 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:22:55.073 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:22:55.073 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 9187 2024-09-13T11:22:55.073 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:55.073 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:22:55.074 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:55.074 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:22:55.074 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:53.834981+0000", 2024-09-13T11:22:55.074 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:55.074 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:55.074 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:55.074 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:22:55.074 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:53.834981+0000", 2024-09-13T11:22:55.074 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:55.074 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:55.074 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:55.074 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:22:55.074 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:53.834982+0000", 2024-09-13T11:22:55.074 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.601e-06 2024-09-13T11:22:55.075 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:55.075 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:55.075 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:22:55.075 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:53.835004+0000", 2024-09-13T11:22:55.075 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.1827e-05 2024-09-13T11:22:55.075 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:55.075 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:55.075 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:22:55.075 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:53.835016+0000", 2024-09-13T11:22:55.075 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.1677e-05 2024-09-13T11:22:55.075 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:55.075 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:55.075 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:22:55.075 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:53.835024+0000", 2024-09-13T11:22:55.075 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.439e-06 2024-09-13T11:22:55.076 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:55.076 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:55.076 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:22:55.076 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:53.835078+0000", 2024-09-13T11:22:55.076 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.3829999999999998e-05 2024-09-13T11:22:55.076 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:55.076 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:55.076 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:22:55.076 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:53.835170+0000", 2024-09-13T11:22:55.076 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.1763999999999998e-05 2024-09-13T11:22:55.076 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:55.076 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:55.077 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 13", 2024-09-13T11:22:55.077 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:53.835337+0000", 2024-09-13T11:22:55.077 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00016669300000000001 2024-09-13T11:22:55.077 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:55.077 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:55.077 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:22:55.077 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:53.836154+0000", 2024-09-13T11:22:55.077 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00081711100000000003 2024-09-13T11:22:55.077 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:55.077 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:55.077 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:22:55.077 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:53.865112+0000", 2024-09-13T11:22:55.077 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.028958417 2024-09-13T11:22:55.077 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:55.078 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:55.078 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:22:55.078 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:53.865154+0000", 2024-09-13T11:22:55.078 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.1946000000000001e-05 2024-09-13T11:22:55.078 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:55.078 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:55.078 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:22:55.078 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:53.865184+0000", 2024-09-13T11:22:55.078 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.9915000000000001e-05 2024-09-13T11:22:55.078 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:55.078 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:22:55.078 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:55.078 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:55.079 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:55.079 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop_reply(client.4665.0:9187 3.5 e47/30 ondisk, result = 0)", 2024-09-13T11:22:55.079 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:22:53.837090+0000", 2024-09-13T11:22:55.079 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 1.1719604480000001, 2024-09-13T11:22:55.079 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.028101883000000001, 2024-09-13T11:22:55.079 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:22:55.079 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:22:55.079 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:22:55.079 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:22:55.079 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:55.079 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:22:55.079 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:53.837090+0000", 2024-09-13T11:22:55.079 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:55.080 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:55.080 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:55.080 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:22:55.080 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:53.837090+0000", 2024-09-13T11:22:55.080 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:55.080 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:55.080 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:55.080 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:22:55.080 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:53.837092+0000", 2024-09-13T11:22:55.080 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.1999999999999999e-06 2024-09-13T11:22:55.080 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:55.080 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:55.080 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:22:55.080 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:53.837096+0000", 2024-09-13T11:22:55.081 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.9589999999999997e-06 2024-09-13T11:22:55.081 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:55.081 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:55.081 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:22:55.081 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:53.865042+0000", 2024-09-13T11:22:55.081 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.027946861999999999 2024-09-13T11:22:55.081 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:55.081 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:55.081 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:22:55.081 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:53.865050+0000", 2024-09-13T11:22:55.081 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.6229999999999997e-06 2024-09-13T11:22:55.081 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:55.081 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:55.081 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:22:55.082 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:53.865103+0000", 2024-09-13T11:22:55.082 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.2942999999999998e-05 2024-09-13T11:22:55.082 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:55.082 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:55.082 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:22:55.082 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:53.865108+0000", 2024-09-13T11:22:55.082 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.5219999999999998e-06 2024-09-13T11:22:55.082 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:55.082 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:55.082 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:22:55.082 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:53.865192+0000", 2024-09-13T11:22:55.082 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.3774000000000001e-05 2024-09-13T11:22:55.082 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:55.082 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:22:55.083 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:55.083 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:55.083 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:55.083 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:9241 3.5 3:ada5a5f9:::benchmark_data_smithi177_31116_object9240:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-09-13T11:22:55.083 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:22:53.872588+0000", 2024-09-13T11:22:55.083 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 1.1364625749999999, 2024-09-13T11:22:55.083 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.032165796000000003, 2024-09-13T11:22:55.083 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:22:55.083 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:22:55.083 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:55.083 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:22:55.083 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:22:55.083 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:22:55.083 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 9241 2024-09-13T11:22:55.084 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:55.084 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:22:55.084 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:55.084 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:22:55.084 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:53.872588+0000", 2024-09-13T11:22:55.084 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:55.084 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:55.084 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:55.084 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:22:55.084 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:53.872588+0000", 2024-09-13T11:22:55.084 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:55.084 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:55.084 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:55.085 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:22:55.085 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:53.872590+0000", 2024-09-13T11:22:55.085 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.9479999999999998e-06 2024-09-13T11:22:55.085 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:55.085 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:55.085 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:22:55.085 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:53.872624+0000", 2024-09-13T11:22:55.085 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.3921999999999998e-05 2024-09-13T11:22:55.085 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:55.085 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:55.085 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:22:55.085 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:53.872637+0000", 2024-09-13T11:22:55.085 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.253e-05 2024-09-13T11:22:55.086 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:55.086 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:55.086 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:22:55.086 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:53.872645+0000", 2024-09-13T11:22:55.086 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.6109999999999994e-06 2024-09-13T11:22:55.086 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:55.086 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:55.086 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:22:55.086 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:53.872710+0000", 2024-09-13T11:22:55.086 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.4980000000000005e-05 2024-09-13T11:22:55.086 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:55.086 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:55.086 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:22:55.086 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:53.872853+0000", 2024-09-13T11:22:55.087 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00014313600000000001 2024-09-13T11:22:55.087 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:55.087 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:55.087 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 13", 2024-09-13T11:22:55.087 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:53.872994+0000", 2024-09-13T11:22:55.087 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00014030800000000001 2024-09-13T11:22:55.087 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:55.087 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:55.087 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:22:55.087 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:53.873888+0000", 2024-09-13T11:22:55.087 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00089424000000000003 2024-09-13T11:22:55.087 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:55.087 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:55.088 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:22:55.088 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:53.904660+0000", 2024-09-13T11:22:55.088 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.030772536 2024-09-13T11:22:55.088 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:55.088 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:55.088 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:22:55.088 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:53.904706+0000", 2024-09-13T11:22:55.088 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.5989999999999998e-05 2024-09-13T11:22:55.088 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:55.088 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:55.088 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:22:55.088 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:53.904754+0000", 2024-09-13T11:22:55.088 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.7595000000000001e-05 2024-09-13T11:22:55.089 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:55.089 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:22:55.089 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:55.089 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:55.089 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:55.089 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop_reply(client.4665.0:9241 3.5 e47/30 ondisk, result = 0)", 2024-09-13T11:22:55.089 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:22:53.875114+0000", 2024-09-13T11:22:55.089 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 1.1339365400000001, 2024-09-13T11:22:55.089 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.029648994000000001, 2024-09-13T11:22:55.089 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:22:55.089 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:22:55.089 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:22:55.089 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:22:55.089 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:55.090 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:22:55.090 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:53.875114+0000", 2024-09-13T11:22:55.090 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:55.090 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:55.090 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:55.090 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:22:55.090 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:53.875114+0000", 2024-09-13T11:22:55.090 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:55.090 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:55.090 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:55.090 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:22:55.090 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:53.875116+0000", 2024-09-13T11:22:55.090 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.457e-06 2024-09-13T11:22:55.090 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:55.091 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:55.091 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:22:55.091 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:53.875121+0000", 2024-09-13T11:22:55.091 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.4349999999999996e-06 2024-09-13T11:22:55.091 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:55.091 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:55.091 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:22:55.091 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:53.904586+0000", 2024-09-13T11:22:55.091 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.02946491 2024-09-13T11:22:55.091 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:55.091 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:55.091 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:22:55.091 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:53.904592+0000", 2024-09-13T11:22:55.092 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.066e-06 2024-09-13T11:22:55.092 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:55.092 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:55.092 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:22:55.092 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:53.904650+0000", 2024-09-13T11:22:55.092 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.8279000000000002e-05 2024-09-13T11:22:55.092 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:55.092 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:55.092 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:22:55.092 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:53.904657+0000", 2024-09-13T11:22:55.092 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.5980000000000002e-06 2024-09-13T11:22:55.092 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:55.092 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:55.093 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:22:55.093 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:53.904763+0000", 2024-09-13T11:22:55.093 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00010624900000000001 2024-09-13T11:22:55.093 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:55.093 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:22:55.093 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:55.093 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:55.093 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:55.093 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:9510 3.5 3:a7e302ef:::benchmark_data_smithi177_31116_object9509:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-09-13T11:22:55.093 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:22:54.044535+0000", 2024-09-13T11:22:55.093 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 0.96451613300000005, 2024-09-13T11:22:55.093 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.049023708999999999, 2024-09-13T11:22:55.093 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:22:55.093 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:22:55.094 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:55.094 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:22:55.094 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:22:55.094 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:22:55.094 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 9510 2024-09-13T11:22:55.094 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:55.094 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:22:55.094 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:55.094 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:22:55.094 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:54.044535+0000", 2024-09-13T11:22:55.094 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:55.094 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:55.094 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:55.095 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:22:55.095 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:54.044535+0000", 2024-09-13T11:22:55.095 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:55.095 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:55.095 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:55.095 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:22:55.095 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:54.044539+0000", 2024-09-13T11:22:55.095 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.0219999999999998e-06 2024-09-13T11:22:55.095 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:55.095 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:55.095 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:22:55.095 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:54.044570+0000", 2024-09-13T11:22:55.095 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.0914999999999998e-05 2024-09-13T11:22:55.095 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:55.096 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:55.096 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:22:55.096 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:54.044593+0000", 2024-09-13T11:22:55.096 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.3584999999999999e-05 2024-09-13T11:22:55.096 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:55.096 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:55.096 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:22:55.096 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:54.044605+0000", 2024-09-13T11:22:55.096 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.1501000000000001e-05 2024-09-13T11:22:55.096 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:55.096 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:55.096 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:22:55.096 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:54.044681+0000", 2024-09-13T11:22:55.096 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.5894000000000002e-05 2024-09-13T11:22:55.097 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:55.097 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:55.097 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:22:55.097 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:54.044820+0000", 2024-09-13T11:22:55.097 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00013934700000000001 2024-09-13T11:22:55.097 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:55.097 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:55.097 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 13", 2024-09-13T11:22:55.097 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:54.044938+0000", 2024-09-13T11:22:55.097 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000118377 2024-09-13T11:22:55.097 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:55.097 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:55.097 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:22:55.098 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:54.046044+0000", 2024-09-13T11:22:55.098 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001105195 2024-09-13T11:22:55.098 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:55.098 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:55.098 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:22:55.098 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:54.093490+0000", 2024-09-13T11:22:55.098 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.047446691999999999 2024-09-13T11:22:55.098 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:55.098 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:55.098 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:22:55.098 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:54.093525+0000", 2024-09-13T11:22:55.098 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.4422999999999999e-05 2024-09-13T11:22:55.098 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:55.099 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:55.099 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:22:55.099 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:54.093558+0000", 2024-09-13T11:22:55.099 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.3757999999999998e-05 2024-09-13T11:22:55.099 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:55.099 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:22:55.099 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:55.099 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:55.099 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:55.099 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop_reply(client.4665.0:9510 3.5 e47/30 ondisk, result = 0)", 2024-09-13T11:22:55.099 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:22:54.047246+0000", 2024-09-13T11:22:55.099 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 0.96180515200000005, 2024-09-13T11:22:55.099 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.046320735000000002, 2024-09-13T11:22:55.099 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:22:55.100 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:22:55.100 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:22:55.100 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:22:55.100 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:55.100 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:22:55.100 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:54.047246+0000", 2024-09-13T11:22:55.100 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:55.100 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:55.100 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:55.100 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:22:55.100 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:54.047246+0000", 2024-09-13T11:22:55.100 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:55.100 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:55.101 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:55.101 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:22:55.101 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:54.047247+0000", 2024-09-13T11:22:55.101 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5829999999999999e-06 2024-09-13T11:22:55.101 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:55.101 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:55.101 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:22:55.101 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:54.047252+0000", 2024-09-13T11:22:55.101 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.7319999999999998e-06 2024-09-13T11:22:55.101 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:55.101 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:55.101 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:22:55.101 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:54.093433+0000", 2024-09-13T11:22:55.102 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.046181415000000003 2024-09-13T11:22:55.102 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:55.102 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:55.102 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:22:55.102 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:54.093440+0000", 2024-09-13T11:22:55.102 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.0029999999999997e-06 2024-09-13T11:22:55.102 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:55.102 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:55.102 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:22:55.102 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:54.093481+0000", 2024-09-13T11:22:55.102 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.0531999999999998e-05 2024-09-13T11:22:55.102 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:55.102 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:55.103 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:22:55.103 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:54.093486+0000", 2024-09-13T11:22:55.103 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.2700000000000004e-06 2024-09-13T11:22:55.103 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:55.103 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:55.103 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:22:55.103 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:54.093566+0000", 2024-09-13T11:22:55.103 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.0199999999999998e-05 2024-09-13T11:22:55.103 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:55.103 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:22:55.103 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:55.103 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:55.103 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:55.104 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:9988 3.5 3:a30262d1:::benchmark_data_smithi177_31116_object9987:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e48)", 2024-09-13T11:22:55.104 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:22:54.393399+0000", 2024-09-13T11:22:55.104 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 0.61565197800000004, 2024-09-13T11:22:55.104 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.031042665000000001, 2024-09-13T11:22:55.104 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:22:55.104 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:22:55.104 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:55.104 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:22:55.104 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:22:55.104 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:22:55.104 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 9988 2024-09-13T11:22:55.104 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:55.104 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:22:55.105 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:55.105 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:22:55.105 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:54.393399+0000", 2024-09-13T11:22:55.105 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:55.105 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:55.105 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:55.105 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:22:55.105 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:54.393399+0000", 2024-09-13T11:22:55.105 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:55.105 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:55.105 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:55.105 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:22:55.105 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:54.393401+0000", 2024-09-13T11:22:55.105 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.6509999999999999e-06 2024-09-13T11:22:55.106 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:55.106 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:55.106 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:22:55.106 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:54.393427+0000", 2024-09-13T11:22:55.106 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.6432e-05 2024-09-13T11:22:55.106 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:55.106 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:55.106 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:22:55.106 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:54.393442+0000", 2024-09-13T11:22:55.106 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5401e-05 2024-09-13T11:22:55.106 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:55.106 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:55.106 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:22:55.107 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:54.393451+0000", 2024-09-13T11:22:55.107 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.0729999999999994e-06 2024-09-13T11:22:55.107 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:55.107 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:55.107 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:22:55.107 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:54.393500+0000", 2024-09-13T11:22:55.107 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.8841000000000002e-05 2024-09-13T11:22:55.107 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:55.107 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:55.107 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:22:55.107 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:54.393576+0000", 2024-09-13T11:22:55.107 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.5935999999999999e-05 2024-09-13T11:22:55.107 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:55.108 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:55.108 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 13", 2024-09-13T11:22:55.108 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:54.393663+0000", 2024-09-13T11:22:55.108 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.6879999999999995e-05 2024-09-13T11:22:55.108 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:55.108 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:55.108 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:22:55.108 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:54.394494+0000", 2024-09-13T11:22:55.108 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00083127500000000003 2024-09-13T11:22:55.108 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:55.108 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:55.108 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:22:55.108 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:54.424328+0000", 2024-09-13T11:22:55.109 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.029833342999999998 2024-09-13T11:22:55.109 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:55.109 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:55.109 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:22:55.109 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:54.424391+0000", 2024-09-13T11:22:55.109 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.2867000000000006e-05 2024-09-13T11:22:55.109 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:55.109 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:55.109 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:22:55.109 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:54.424442+0000", 2024-09-13T11:22:55.109 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.0966e-05 2024-09-13T11:22:55.109 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:55.109 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:22:55.109 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:55.110 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:55.110 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:55.110 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop_reply(client.4665.0:9988 3.5 e48/30 ondisk, result = 0)", 2024-09-13T11:22:55.110 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:22:54.395251+0000", 2024-09-13T11:22:55.110 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 0.61379994100000002, 2024-09-13T11:22:55.110 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.029199811999999999, 2024-09-13T11:22:55.110 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:22:55.110 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:22:55.110 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:22:55.110 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:22:55.110 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:55.110 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:22:55.110 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:54.395251+0000", 2024-09-13T11:22:55.110 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:55.111 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:55.111 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:55.111 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:22:55.111 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:54.395251+0000", 2024-09-13T11:22:55.111 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:55.111 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:55.111 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:55.111 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:22:55.111 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:54.395252+0000", 2024-09-13T11:22:55.111 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5889999999999999e-06 2024-09-13T11:22:55.111 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:55.111 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:55.111 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:22:55.111 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:54.395257+0000", 2024-09-13T11:22:55.112 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.9899999999999997e-06 2024-09-13T11:22:55.112 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:55.112 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:55.112 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:22:55.112 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:54.424255+0000", 2024-09-13T11:22:55.112 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.028997572999999999 2024-09-13T11:22:55.112 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:55.112 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:55.112 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:22:55.112 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:54.424262+0000", 2024-09-13T11:22:55.112 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.584e-06 2024-09-13T11:22:55.112 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:55.112 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:55.113 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:22:55.113 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:54.424311+0000", 2024-09-13T11:22:55.113 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.9851000000000001e-05 2024-09-13T11:22:55.113 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:55.113 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:55.113 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:22:55.113 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:54.424321+0000", 2024-09-13T11:22:55.113 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.6779999999999995e-06 2024-09-13T11:22:55.113 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:55.113 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:55.113 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:22:55.113 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:54.424451+0000", 2024-09-13T11:22:55.113 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00012954700000000001 2024-09-13T11:22:55.113 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:55.113 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:22:55.114 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:55.114 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:55.114 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:55.114 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:10080 3.5 3:af43f3a5:::benchmark_data_smithi177_31116_object10079:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e48)", 2024-09-13T11:22:55.114 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:22:54.458003+0000", 2024-09-13T11:22:55.114 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 0.55104774999999995, 2024-09-13T11:22:55.114 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.028489071000000001, 2024-09-13T11:22:55.114 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:22:55.114 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:22:55.114 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:55.114 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:22:55.114 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:22:55.114 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:22:55.114 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 10080 2024-09-13T11:22:55.114 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:55.115 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:22:55.115 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:55.115 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:22:55.115 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:54.458003+0000", 2024-09-13T11:22:55.115 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:55.115 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:55.115 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:55.115 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:22:55.115 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:54.458003+0000", 2024-09-13T11:22:55.115 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:22:55.115 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:55.115 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:55.115 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:22:55.115 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:54.458005+0000", 2024-09-13T11:22:55.116 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.9589999999999998e-06 2024-09-13T11:22:55.116 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:55.116 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:55.116 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:22:55.116 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:54.458027+0000", 2024-09-13T11:22:55.116 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.2447e-05 2024-09-13T11:22:55.116 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:55.116 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:55.116 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:22:55.116 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:54.458040+0000", 2024-09-13T11:22:55.116 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.2085e-05 2024-09-13T11:22:55.116 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:55.116 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:55.116 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:22:55.116 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:54.458047+0000", 2024-09-13T11:22:55.117 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.7670000000000003e-06 2024-09-13T11:22:55.117 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:55.117 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:55.117 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:22:55.117 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:54.458102+0000", 2024-09-13T11:22:55.117 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.4685999999999998e-05 2024-09-13T11:22:55.117 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:55.117 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:55.117 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:22:55.117 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:54.458211+0000", 2024-09-13T11:22:55.117 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00010864799999999999 2024-09-13T11:22:55.117 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:55.118 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:55.118 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 13", 2024-09-13T11:22:55.118 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:54.458319+0000", 2024-09-13T11:22:55.118 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00010789300000000001 2024-09-13T11:22:55.118 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:55.118 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:55.118 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:22:55.118 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:54.459215+0000", 2024-09-13T11:22:55.118 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00089625299999999996 2024-09-13T11:22:55.118 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:55.118 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:55.118 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:22:55.118 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:54.486391+0000", 2024-09-13T11:22:55.119 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.02717582 2024-09-13T11:22:55.119 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:55.119 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:55.119 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:22:55.119 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:54.486447+0000", 2024-09-13T11:22:55.119 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.6174000000000001e-05 2024-09-13T11:22:55.119 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:22:55.119 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:22:55.119 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:22:55.119 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:54.486492+0000", 2024-09-13T11:22:55.119 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.5339000000000001e-05 2024-09-13T11:22:55.119 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:55.119 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:22:55.120 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:55.120 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:22:55.120 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:22:55.120 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-09-13T11:22:55.121 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-13T11:22:55.122 INFO:tasks.ceph.osd.9.smithi092.stderr:2024-09-13T11:22:55.118+0000 7ff5ab3d0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26348) UID: 0 2024-09-13T11:22:55.222 INFO:tasks.ceph.osd.8.smithi001.stderr:2024-09-13T11:22:55.216+0000 7fb9b48ae640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26312) UID: 0 2024-09-13T11:22:55.255 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T11:22:55.273 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-09-13T11:22:55.273 INFO:teuthology.orchestra.run.smithi092.stdout: "ops": [], 2024-09-13T11:22:55.273 INFO:teuthology.orchestra.run.smithi092.stdout: "num_ops": 0 2024-09-13T11:22:55.273 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-09-13T11:22:55.290 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-13T11:22:55.336 INFO:tasks.ceph.osd.1.smithi092.stderr:2024-09-13T11:22:55.334+0000 7fe7789c5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33718) UID: 0 2024-09-13T11:22:55.383 INFO:teuthology.orchestra.run.smithi001.stderr:reweighted osd.9 to 1 (10000) 2024-09-13T11:22:55.383 INFO:teuthology.orchestra.run.smithi001.stderr:noscrub is unset 2024-09-13T11:22:55.385 INFO:tasks.ceph.osd.2.smithi162.stderr:2024-09-13T11:22:55.379+0000 7fe921d8b640 -1 osd.2 48 handle_osd_map injecting map crc failure 2024-09-13T11:22:55.386 INFO:tasks.ceph.osd.12.smithi001.stderr:2024-09-13T11:22:55.380+0000 7f891f10f640 -1 osd.12 48 handle_osd_map injecting map crc failure 2024-09-13T11:22:55.394 INFO:tasks.ceph.osd.3.smithi177.stderr:2024-09-13T11:22:55.395+0000 7f10a25d1640 -1 osd.3 48 handle_osd_map injecting map crc failure 2024-09-13T11:22:55.411 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 10 1 2024-09-13T11:22:55.429 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-09-13T11:22:55.429 INFO:teuthology.orchestra.run.smithi092.stdout: "ops": [], 2024-09-13T11:22:55.429 INFO:teuthology.orchestra.run.smithi092.stdout: "complaint_time": 30, 2024-09-13T11:22:55.429 INFO:teuthology.orchestra.run.smithi092.stdout: "num_blocked_ops": 0 2024-09-13T11:22:55.429 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-09-13T11:22:55.440 INFO:teuthology.orchestra.run.smithi001.stdout:osd.0: {} 2024-09-13T11:22:55.440 INFO:teuthology.orchestra.run.smithi001.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T11:22:55.441 INFO:tasks.ceph.osd.8.smithi001.stderr:2024-09-13T11:22:55.436+0000 7fb9b48ae640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26312) UID: 0 2024-09-13T11:22:55.446 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-13T11:22:55.470 INFO:teuthology.orchestra.run.smithi001.stdout:osd.1: {} 2024-09-13T11:22:55.470 INFO:teuthology.orchestra.run.smithi001.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T11:22:55.499 INFO:teuthology.orchestra.run.smithi001.stdout:osd.2: {} 2024-09-13T11:22:55.499 INFO:teuthology.orchestra.run.smithi001.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T11:22:55.524 INFO:tasks.ceph.osd.8.smithi001.stderr:2024-09-13T11:22:55.516+0000 7fb9b48ae640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26312) UID: 0 2024-09-13T11:22:55.527 INFO:teuthology.orchestra.run.smithi001.stdout:osd.3: {} 2024-09-13T11:22:55.527 INFO:teuthology.orchestra.run.smithi001.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T11:22:55.551 INFO:teuthology.orchestra.run.smithi001.stdout:osd.4: {} 2024-09-13T11:22:55.551 INFO:teuthology.orchestra.run.smithi001.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T11:22:55.580 INFO:teuthology.orchestra.run.smithi001.stdout:osd.5: {} 2024-09-13T11:22:55.580 INFO:teuthology.orchestra.run.smithi001.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T11:22:55.619 INFO:teuthology.orchestra.run.smithi001.stdout:osd.6: {} 2024-09-13T11:22:55.619 INFO:teuthology.orchestra.run.smithi001.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T11:22:55.625 INFO:tasks.ceph.osd.14.smithi162.stderr:2024-09-13T11:22:55.623+0000 7f235c0db640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26319) UID: 0 2024-09-13T11:22:55.639 INFO:teuthology.orchestra.run.smithi092.stdout:ERROR: (22) Invalid argument 2024-09-13T11:22:55.639 INFO:teuthology.orchestra.run.smithi092.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:22:55.659 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-13T11:22:55.726 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-09-13T11:22:55.723+0000 7f97befc9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26315) UID: 0 2024-09-13T11:22:55.745 INFO:teuthology.orchestra.run.smithi001.stdout:osd.7: {} 2024-09-13T11:22:55.745 INFO:teuthology.orchestra.run.smithi001.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T11:22:55.772 INFO:teuthology.orchestra.run.smithi001.stdout:osd.8: {} 2024-09-13T11:22:55.772 INFO:teuthology.orchestra.run.smithi001.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-13T11:22:55.798 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-13T11:22:55.799 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:22:55.810 INFO:teuthology.orchestra.run.smithi001.stdout:osd.9: {} 2024-09-13T11:22:55.810 INFO:teuthology.orchestra.run.smithi001.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-13T11:22:55.815 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-13T11:22:55.827 INFO:tasks.ceph.osd.15.smithi177.stderr:2024-09-13T11:22:55.827+0000 7f9d6ce77640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26578) UID: 0 2024-09-13T11:22:55.851 INFO:teuthology.orchestra.run.smithi001.stdout:osd.10: {} 2024-09-13T11:22:55.852 INFO:teuthology.orchestra.run.smithi001.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-13T11:22:55.877 INFO:teuthology.orchestra.run.smithi001.stdout:osd.11: {} 2024-09-13T11:22:55.878 INFO:teuthology.orchestra.run.smithi001.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-13T11:22:55.901 INFO:teuthology.orchestra.run.smithi001.stdout:osd.12: {} 2024-09-13T11:22:55.902 INFO:teuthology.orchestra.run.smithi001.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-13T11:22:55.928 INFO:tasks.ceph.osd.3.smithi177.stderr:2024-09-13T11:22:55.927+0000 7f10adba4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26574) UID: 0 2024-09-13T11:22:55.929 INFO:teuthology.orchestra.run.smithi001.stdout:osd.13: {} 2024-09-13T11:22:55.929 INFO:teuthology.orchestra.run.smithi001.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-13T11:22:55.948 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-13T11:22:55.948 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:22:55.955 INFO:teuthology.orchestra.run.smithi001.stdout:osd.14: {} 2024-09-13T11:22:55.955 INFO:teuthology.orchestra.run.smithi001.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-13T11:22:55.963 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-13T11:22:55.978 INFO:teuthology.orchestra.run.smithi001.stdout:osd.15: {} 2024-09-13T11:22:55.979 INFO:teuthology.orchestra.run.smithi001.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-13T11:22:55.999 INFO:tasks.radosbench.radosbench.0.smithi177.stdout: 29 16 12327 12311 26.5289 94.25 0.00382639 0.037651 2024-09-13T11:22:56.030 INFO:tasks.ceph.osd.5.smithi092.stderr:2024-09-13T11:22:56.026+0000 7f6f0b705640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26346) UID: 0 2024-09-13T11:22:56.110 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-13T11:22:56.110 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:22:56.125 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-13T11:22:56.130 INFO:tasks.ceph.osd.11.smithi177.stderr:2024-09-13T11:22:56.127+0000 7f24792c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26580) UID: 0 2024-09-13T11:22:56.231 INFO:tasks.ceph.osd.10.smithi162.stderr:2024-09-13T11:22:56.227+0000 7fb59c9f0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26317) UID: 0 2024-09-13T11:22:56.260 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-13T11:22:56.260 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:22:56.275 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-13T11:22:56.332 INFO:tasks.ceph.osd.9.smithi092.stderr:2024-09-13T11:22:56.326+0000 7ff5ab3d0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26348) UID: 0 2024-09-13T11:22:56.384 INFO:teuthology.orchestra.run.smithi001.stderr:reweighted osd.10 to 1 (10000) 2024-09-13T11:22:56.397 INFO:tasks.ceph.osd.11.smithi177.stderr:2024-09-13T11:22:56.395+0000 7f2469cec640 -1 osd.11 49 handle_osd_map injecting map crc failure 2024-09-13T11:22:56.407 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 11 1 2024-09-13T11:22:56.416 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-13T11:22:56.416 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:22:56.433 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-13T11:22:56.433 INFO:tasks.ceph.osd.11.smithi177.stderr:2024-09-13T11:22:56.431+0000 7f24792c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26580) UID: 0 2024-09-13T11:22:56.534 INFO:tasks.ceph.osd.10.smithi162.stderr:2024-09-13T11:22:56.531+0000 7fb59c9f0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26317) UID: 0 2024-09-13T11:22:56.614 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-13T11:22:56.614 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:22:56.634 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-09-13T11:22:56.635 INFO:tasks.ceph.osd.8.smithi001.stderr:2024-09-13T11:22:56.628+0000 7fb9b48ae640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26312) UID: 0 2024-09-13T11:22:56.736 INFO:tasks.ceph.osd.5.smithi092.stderr:2024-09-13T11:22:56.730+0000 7f6f0b705640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26346) UID: 0 2024-09-13T11:22:56.763 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-13T11:22:56.763 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:22:56.780 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-09-13T11:22:56.837 INFO:tasks.ceph.osd.3.smithi177.stderr:2024-09-13T11:22:56.835+0000 7f10adba4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26574) UID: 0 2024-09-13T11:22:56.917 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-13T11:22:56.917 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:22:56.935 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-09-13T11:22:56.938 INFO:tasks.ceph.osd.9.smithi092.stderr:2024-09-13T11:22:56.934+0000 7ff5ab3d0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26348) UID: 0 2024-09-13T11:22:56.999 INFO:tasks.radosbench.radosbench.0.smithi177.stdout: 30 16 13961 13945 29.0484 102.125 0.0192515 0.034412 2024-09-13T11:22:57.039 INFO:tasks.ceph.osd.11.smithi177.stderr:2024-09-13T11:22:57.039+0000 7f24792c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26580) UID: 0 2024-09-13T11:22:57.067 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-13T11:22:57.067 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:22:57.083 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_ops_in_flight 2024-09-13T11:22:57.140 INFO:tasks.ceph.osd.15.smithi177.stderr:2024-09-13T11:22:57.139+0000 7f9d6ce77640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26578) UID: 0 2024-09-13T11:22:57.227 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-13T11:22:57.227 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:22:57.241 INFO:tasks.ceph.osd.15.smithi177.stderr:2024-09-13T11:22:57.239+0000 7f9d6ce77640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26578) UID: 0 2024-09-13T11:22:57.243 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_blocked_ops 2024-09-13T11:22:57.342 INFO:tasks.ceph.osd.14.smithi162.stderr:2024-09-13T11:22:57.339+0000 7f235c0db640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26319) UID: 0 2024-09-13T11:22:57.380 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-13T11:22:57.380 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:22:57.394 INFO:teuthology.orchestra.run.smithi001.stderr:reweighted osd.11 to 1 (10000) 2024-09-13T11:22:57.400 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_historic_ops 2024-09-13T11:22:57.403 INFO:tasks.ceph.osd.12.smithi001.stderr:2024-09-13T11:22:57.396+0000 7f891f10f640 -1 osd.12 50 handle_osd_map injecting map crc failure 2024-09-13T11:22:57.412 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-13T11:22:57.419 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 12 1 2024-09-13T11:22:57.442 INFO:tasks.ceph.osd.11.smithi177.stderr:2024-09-13T11:22:57.443+0000 7f24792c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26580) UID: 0 2024-09-13T11:22:57.543 INFO:tasks.ceph.osd.2.smithi162.stderr:2024-09-13T11:22:57.539+0000 7fe92b35a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26313) UID: 0 2024-09-13T11:22:57.544 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-13T11:22:57.545 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:22:57.565 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-13T11:22:57.644 INFO:tasks.ceph.osd.11.smithi177.stderr:2024-09-13T11:22:57.643+0000 7f24792c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26580) UID: 0 2024-09-13T11:22:57.713 INFO:teuthology.orchestra.run.smithi092.stdout:ERROR: (22) Invalid argument 2024-09-13T11:22:57.714 INFO:teuthology.orchestra.run.smithi092.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:22:57.732 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-13T11:22:57.745 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-13T11:22:57.740+0000 7f6a68775640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26307) UID: 0 2024-09-13T11:22:57.846 INFO:tasks.ceph.osd.10.smithi162.stderr:2024-09-13T11:22:57.843+0000 7fb59c9f0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26317) UID: 0 2024-09-13T11:22:57.860 INFO:teuthology.orchestra.run.smithi092.stdout:ERROR: (22) Invalid argument 2024-09-13T11:22:57.861 INFO:teuthology.orchestra.run.smithi092.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:22:57.877 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-13T11:22:57.947 INFO:tasks.ceph.osd.10.smithi162.stderr:2024-09-13T11:22:57.943+0000 7fb59c9f0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26317) UID: 0 2024-09-13T11:22:57.999 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T11:22:58.000 INFO:tasks.radosbench.radosbench.0.smithi177.stdout: 31 16 15254 15238 30.7179 80.8125 0.00224019 0.0325409 2024-09-13T11:22:58.026 INFO:teuthology.orchestra.run.smithi092.stdout:ERROR: (22) Invalid argument 2024-09-13T11:22:58.026 INFO:teuthology.orchestra.run.smithi092.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:22:58.045 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-09-13T11:22:58.047 INFO:tasks.ceph.osd.9.smithi092.stderr:2024-09-13T11:22:58.042+0000 7ff5ab3d0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26348) UID: 0 2024-09-13T11:22:58.149 INFO:tasks.ceph.osd.10.smithi162.stderr:2024-09-13T11:22:58.143+0000 7fb59c9f0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26317) UID: 0 2024-09-13T11:22:58.179 INFO:teuthology.orchestra.run.smithi001.stdout:osd.0: {} 2024-09-13T11:22:58.179 INFO:teuthology.orchestra.run.smithi001.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T11:22:58.194 INFO:teuthology.orchestra.run.smithi092.stdout:ERROR: (22) Invalid argument 2024-09-13T11:22:58.194 INFO:teuthology.orchestra.run.smithi092.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:22:58.204 INFO:teuthology.orchestra.run.smithi001.stdout:osd.1: {} 2024-09-13T11:22:58.204 INFO:teuthology.orchestra.run.smithi001.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T11:22:58.213 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-09-13T11:22:58.234 INFO:teuthology.orchestra.run.smithi001.stdout:osd.2: {} 2024-09-13T11:22:58.234 INFO:teuthology.orchestra.run.smithi001.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T11:22:58.256 INFO:tasks.ceph.osd.1.smithi092.stderr:2024-09-13T11:22:58.254+0000 7fe7789c5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33718) UID: 0 2024-09-13T11:22:58.257 INFO:teuthology.orchestra.run.smithi001.stdout:osd.3: {} 2024-09-13T11:22:58.257 INFO:teuthology.orchestra.run.smithi001.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T11:22:58.283 INFO:teuthology.orchestra.run.smithi001.stdout:osd.4: {} 2024-09-13T11:22:58.283 INFO:teuthology.orchestra.run.smithi001.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T11:22:58.309 INFO:teuthology.orchestra.run.smithi001.stdout:osd.5: {} 2024-09-13T11:22:58.310 INFO:teuthology.orchestra.run.smithi001.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T11:22:58.336 INFO:teuthology.orchestra.run.smithi001.stdout:osd.6: {} 2024-09-13T11:22:58.336 INFO:teuthology.orchestra.run.smithi001.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T11:22:58.344 INFO:teuthology.orchestra.run.smithi092.stdout:ERROR: (22) Invalid argument 2024-09-13T11:22:58.345 INFO:teuthology.orchestra.run.smithi092.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:22:58.350 INFO:tasks.ceph.osd.14.smithi162.stderr:2024-09-13T11:22:58.347+0000 7f235c0db640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26319) UID: 0 2024-09-13T11:22:58.361 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-09-13T11:22:58.363 INFO:teuthology.orchestra.run.smithi001.stdout:osd.7: {} 2024-09-13T11:22:58.363 INFO:teuthology.orchestra.run.smithi001.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T11:22:58.386 INFO:teuthology.orchestra.run.smithi001.stdout:osd.8: {} 2024-09-13T11:22:58.387 INFO:teuthology.orchestra.run.smithi001.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-13T11:22:58.396 INFO:teuthology.orchestra.run.smithi001.stderr:nodeep-scrub is unset 2024-09-13T11:22:58.396 INFO:teuthology.orchestra.run.smithi001.stderr:reweighted osd.12 to 1 (10000) 2024-09-13T11:22:58.397 INFO:tasks.ceph.osd.2.smithi162.stderr:2024-09-13T11:22:58.395+0000 7fe921d8b640 -1 osd.2 51 handle_osd_map injecting map crc failure 2024-09-13T11:22:58.399 INFO:tasks.ceph.osd.12.smithi001.stderr:2024-09-13T11:22:58.392+0000 7f891f10f640 -1 osd.12 51 handle_osd_map injecting map crc failure 2024-09-13T11:22:58.416 INFO:teuthology.orchestra.run.smithi001.stdout:osd.9: {} 2024-09-13T11:22:58.416 INFO:teuthology.orchestra.run.smithi001.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-13T11:22:58.425 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 13 1 2024-09-13T11:22:58.443 INFO:teuthology.orchestra.run.smithi001.stdout:osd.10: {} 2024-09-13T11:22:58.444 INFO:teuthology.orchestra.run.smithi001.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-13T11:22:58.450 INFO:tasks.ceph.osd.2.smithi162.stderr:2024-09-13T11:22:58.447+0000 7fe92b35a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26313) UID: 0 2024-09-13T11:22:58.470 INFO:teuthology.orchestra.run.smithi001.stdout:osd.11: {} 2024-09-13T11:22:58.471 INFO:teuthology.orchestra.run.smithi001.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-13T11:22:58.494 INFO:teuthology.orchestra.run.smithi001.stdout:osd.12: {} 2024-09-13T11:22:58.494 INFO:teuthology.orchestra.run.smithi001.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-13T11:22:58.523 INFO:teuthology.orchestra.run.smithi001.stdout:osd.13: {} 2024-09-13T11:22:58.523 INFO:teuthology.orchestra.run.smithi001.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-13T11:22:58.543 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-09-13T11:22:58.544 INFO:teuthology.orchestra.run.smithi092.stdout: "size": 20, 2024-09-13T11:22:58.544 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 600, 2024-09-13T11:22:58.544 INFO:teuthology.orchestra.run.smithi092.stdout: "ops": [ 2024-09-13T11:22:58.544 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:58.544 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop(client.4344.0:5 2.2 e27/25 2:5cea7035:::rbd_info:head v 27'1, mlcod=0'0)", 2024-09-13T11:22:58.544 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:15.250889+0000", 2024-09-13T11:22:58.544 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 43.293865504000003, 2024-09-13T11:22:58.544 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.001990627, 2024-09-13T11:22:58.544 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:22:58.544 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:22:58.545 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:58.545 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:22:58.545 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:58.545 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:22:58.545 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:15.250889+0000", 2024-09-13T11:22:58.545 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:22:58.545 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:58.545 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:58.545 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:22:58.545 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:15.250889+0000", 2024-09-13T11:22:58.545 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:22:58.545 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:58.546 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:58.546 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:22:58.546 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:15.250890+0000", 2024-09-13T11:22:58.546 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.153e-06 2024-09-13T11:22:58.546 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:58.546 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:58.546 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:22:58.546 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:15.250907+0000", 2024-09-13T11:22:58.546 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.6821e-05 2024-09-13T11:22:58.546 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:58.546 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:58.546 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:22:58.546 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:15.250991+0000", 2024-09-13T11:22:58.547 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.4241999999999997e-05 2024-09-13T11:22:58.547 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:58.547 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:58.547 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:22:58.547 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:15.251023+0000", 2024-09-13T11:22:58.547 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.1780999999999999e-05 2024-09-13T11:22:58.547 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:58.547 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:58.547 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:22:58.547 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:15.251178+0000", 2024-09-13T11:22:58.548 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000155064 2024-09-13T11:22:58.548 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:58.548 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:58.548 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:22:58.548 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:15.251215+0000", 2024-09-13T11:22:58.548 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.7160000000000003e-05 2024-09-13T11:22:58.548 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:58.548 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:58.548 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:22:58.548 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:15.252759+0000", 2024-09-13T11:22:58.548 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0015440110000000001 2024-09-13T11:22:58.548 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:58.548 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:58.549 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:22:58.549 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:15.252880+0000", 2024-09-13T11:22:58.549 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000120395 2024-09-13T11:22:58.549 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:22:58.549 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:22:58.549 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:22:58.549 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:58.549 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:58.549 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop(client.4344.0:7 2.2 e28/25 2:5cea7035:::rbd_info:head v 28'2, mlcod=0'0)", 2024-09-13T11:22:58.549 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:16.268518+0000", 2024-09-13T11:22:58.549 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 42.276235904000004, 2024-09-13T11:22:58.549 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0017167040000000001, 2024-09-13T11:22:58.549 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:22:58.549 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:22:58.550 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:58.550 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:22:58.550 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:58.550 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:22:58.550 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:16.268518+0000", 2024-09-13T11:22:58.550 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:22:58.550 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:58.550 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:58.550 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:22:58.550 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:16.268518+0000", 2024-09-13T11:22:58.550 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:22:58.550 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:58.550 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:58.551 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:22:58.551 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:16.268519+0000", 2024-09-13T11:22:58.551 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.1299999999999999e-07 2024-09-13T11:22:58.551 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:58.551 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:58.551 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:22:58.551 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:16.268539+0000", 2024-09-13T11:22:58.551 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.0214999999999999e-05 2024-09-13T11:22:58.551 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:58.551 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:58.551 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:22:58.551 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:16.268622+0000", 2024-09-13T11:22:58.551 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.2395e-05 2024-09-13T11:22:58.552 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:58.552 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:58.552 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:22:58.552 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:16.268638+0000", 2024-09-13T11:22:58.552 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.6189e-05 2024-09-13T11:22:58.552 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:58.552 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:58.552 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:22:58.552 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:16.268740+0000", 2024-09-13T11:22:58.552 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00010164 2024-09-13T11:22:58.552 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:58.552 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:58.552 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:22:58.552 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:16.268783+0000", 2024-09-13T11:22:58.553 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.2988999999999998e-05 2024-09-13T11:22:58.553 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:58.553 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:58.553 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:22:58.553 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:16.270121+0000", 2024-09-13T11:22:58.553 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00133829 2024-09-13T11:22:58.553 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:58.553 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:58.553 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:22:58.553 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:16.270235+0000", 2024-09-13T11:22:58.553 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000114273 2024-09-13T11:22:58.553 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:22:58.553 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:22:58.554 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:22:58.554 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:22:58.554 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:22:58.554 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-09-13T11:22:58.555 INFO:tasks.ceph.osd.13.smithi092.stderr:2024-09-13T11:22:58.550+0000 7f3e5afea640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26350) UID: 0 2024-09-13T11:22:58.560 INFO:teuthology.orchestra.run.smithi001.stdout:osd.14: {} 2024-09-13T11:22:58.560 INFO:teuthology.orchestra.run.smithi001.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-13T11:22:58.562 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_ops_in_flight 2024-09-13T11:22:58.587 INFO:teuthology.orchestra.run.smithi001.stdout:osd.15: {} 2024-09-13T11:22:58.587 INFO:teuthology.orchestra.run.smithi001.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-13T11:22:58.656 INFO:tasks.ceph.osd.15.smithi177.stderr:2024-09-13T11:22:58.655+0000 7f9d6ce77640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26578) UID: 0 2024-09-13T11:22:58.749 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-09-13T11:22:58.750 INFO:teuthology.orchestra.run.smithi092.stdout: "ops": [ 2024-09-13T11:22:58.750 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:58.750 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:16499 3.7 3:e1aa08a6:::benchmark_data_smithi177_31116_object16498:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-13T11:22:58.750 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:58.742943+0000", 2024-09-13T11:22:58.750 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 0.0073427010000000001, 2024-09-13T11:22:58.750 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0073669110000000003, 2024-09-13T11:22:58.750 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:22:58.750 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:22:58.750 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "waiting for sub ops", 2024-09-13T11:22:58.750 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:22:58.750 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:22:58.751 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:22:58.751 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 16499 2024-09-13T11:22:58.751 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:58.751 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:22:58.751 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:58.751 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:22:58.751 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:58.742943+0000", 2024-09-13T11:22:58.751 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:22:58.751 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:58.751 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:58.751 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:22:58.751 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:58.742943+0000", 2024-09-13T11:22:58.751 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:22:58.752 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:58.752 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:58.752 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:22:58.752 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:58.742944+0000", 2024-09-13T11:22:58.752 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.573e-06 2024-09-13T11:22:58.752 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:58.752 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:58.752 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:22:58.752 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:58.743007+0000", 2024-09-13T11:22:58.752 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.2941999999999995e-05 2024-09-13T11:22:58.752 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:58.752 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:58.752 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:22:58.752 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:58.743019+0000", 2024-09-13T11:22:58.753 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.1415e-05 2024-09-13T11:22:58.753 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:58.753 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:58.753 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:22:58.753 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:58.743025+0000", 2024-09-13T11:22:58.753 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.3029999999999996e-06 2024-09-13T11:22:58.753 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:58.753 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:58.753 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:22:58.753 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:58.743091+0000", 2024-09-13T11:22:58.753 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.5754999999999996e-05 2024-09-13T11:22:58.753 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:58.753 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:58.754 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:22:58.754 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:58.743229+0000", 2024-09-13T11:22:58.754 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00013830900000000001 2024-09-13T11:22:58.754 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:58.754 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:58.754 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 8", 2024-09-13T11:22:58.754 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:58.743368+0000", 2024-09-13T11:22:58.754 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000138622 2024-09-13T11:22:58.754 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:58.754 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:58.754 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:22:58.754 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:58.746078+0000", 2024-09-13T11:22:58.754 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0027097739999999999 2024-09-13T11:22:58.755 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:22:58.755 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:22:58.755 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:22:58.755 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:58.755 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:58.755 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:16502 3.b 3:d6ebe32a:::benchmark_data_smithi177_31116_object16501:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-13T11:22:58.755 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:58.743971+0000", 2024-09-13T11:22:58.755 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 0.0063149670000000003, 2024-09-13T11:22:58.755 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0063965139999999998, 2024-09-13T11:22:58.755 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:22:58.755 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:22:58.755 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "waiting for sub ops", 2024-09-13T11:22:58.755 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:22:58.755 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:22:58.756 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:22:58.756 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 16502 2024-09-13T11:22:58.756 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:58.756 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:22:58.756 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:58.756 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:22:58.756 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:58.743971+0000", 2024-09-13T11:22:58.756 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:22:58.756 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:58.756 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:58.756 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:22:58.756 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:58.743971+0000", 2024-09-13T11:22:58.757 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:22:58.757 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:58.757 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:58.757 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:22:58.757 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:58.743972+0000", 2024-09-13T11:22:58.757 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.1489999999999999e-06 2024-09-13T11:22:58.757 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:58.757 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:58.757 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:22:58.757 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:58.744036+0000", 2024-09-13T11:22:58.757 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.4052000000000003e-05 2024-09-13T11:22:58.757 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:58.757 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:58.758 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:22:58.758 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:58.744045+0000", 2024-09-13T11:22:58.758 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.1239999999999994e-06 2024-09-13T11:22:58.758 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:58.758 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:58.758 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:22:58.758 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:58.744051+0000", 2024-09-13T11:22:58.758 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.0379999999999996e-06 2024-09-13T11:22:58.758 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:58.758 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:58.758 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:22:58.758 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:58.744136+0000", 2024-09-13T11:22:58.758 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.5415999999999997e-05 2024-09-13T11:22:58.758 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:58.759 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:58.759 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:22:58.759 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:58.744296+0000", 2024-09-13T11:22:58.759 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000160123 2024-09-13T11:22:58.759 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:58.759 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:58.759 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1", 2024-09-13T11:22:58.759 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:58.744452+0000", 2024-09-13T11:22:58.759 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00015578799999999999 2024-09-13T11:22:58.759 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:58.759 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:58.759 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:22:58.759 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:58.746467+0000", 2024-09-13T11:22:58.760 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0020143169999999998 2024-09-13T11:22:58.760 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:22:58.760 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:22:58.760 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:22:58.760 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:58.760 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:58.760 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop(client.4665.0:16505 3.c e52/30 3:34308a1c:::benchmark_data_smithi177_31116_object16504:head v 52'1043, mlcod=52'1040)", 2024-09-13T11:22:58.760 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:58.747024+0000", 2024-09-13T11:22:58.760 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 0.0032610460000000001, 2024-09-13T11:22:58.760 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0033970879999999999, 2024-09-13T11:22:58.760 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:22:58.760 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:22:58.760 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-09-13T11:22:58.760 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:22:58.761 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:58.761 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:22:58.761 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:58.747024+0000", 2024-09-13T11:22:58.761 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:22:58.761 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:58.761 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:58.761 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:22:58.761 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:58.747024+0000", 2024-09-13T11:22:58.761 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:22:58.761 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:58.761 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:58.761 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:22:58.761 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:58.747026+0000", 2024-09-13T11:22:58.761 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.753e-06 2024-09-13T11:22:58.762 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:58.762 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:58.762 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:22:58.762 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:58.747066+0000", 2024-09-13T11:22:58.762 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.9854999999999997e-05 2024-09-13T11:22:58.762 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:58.762 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:58.762 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:22:58.762 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:58.747102+0000", 2024-09-13T11:22:58.762 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.5448999999999999e-05 2024-09-13T11:22:58.762 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:58.762 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:58.762 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:22:58.763 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:58.747112+0000", 2024-09-13T11:22:58.763 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0098000000000001e-05 2024-09-13T11:22:58.763 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:58.763 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:58.763 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:22:58.763 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:58.747179+0000", 2024-09-13T11:22:58.763 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.6988000000000003e-05 2024-09-13T11:22:58.763 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:58.763 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:58.763 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:22:58.763 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:58.747196+0000", 2024-09-13T11:22:58.763 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.7609999999999999e-05 2024-09-13T11:22:58.763 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:22:58.763 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:22:58.764 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:22:58.764 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:58.764 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:58.764 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:16508 3.7 3:e2fcd0a8:::benchmark_data_smithi177_31116_object16507:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-13T11:22:58.764 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:58.747833+0000", 2024-09-13T11:22:58.764 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 0.002452676, 2024-09-13T11:22:58.764 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0026288169999999999, 2024-09-13T11:22:58.764 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:22:58.764 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:22:58.764 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "waiting for sub ops", 2024-09-13T11:22:58.764 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:22:58.764 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:22:58.764 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:22:58.765 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 16508 2024-09-13T11:22:58.765 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:58.765 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:22:58.765 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:58.765 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:22:58.765 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:58.747833+0000", 2024-09-13T11:22:58.765 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:22:58.765 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:58.765 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:58.765 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:22:58.765 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:58.747833+0000", 2024-09-13T11:22:58.765 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:22:58.765 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:58.766 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:58.766 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:22:58.766 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:58.747835+0000", 2024-09-13T11:22:58.766 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.7129999999999999e-06 2024-09-13T11:22:58.766 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:58.766 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:58.766 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:22:58.766 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:58.747899+0000", 2024-09-13T11:22:58.766 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.4307999999999995e-05 2024-09-13T11:22:58.766 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:58.766 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:58.766 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:22:58.766 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:58.747910+0000", 2024-09-13T11:22:58.766 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.1533e-05 2024-09-13T11:22:58.767 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:58.767 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:58.767 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:22:58.767 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:58.747917+0000", 2024-09-13T11:22:58.767 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.2910000000000004e-06 2024-09-13T11:22:58.767 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:58.767 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:58.767 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:22:58.767 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:58.747982+0000", 2024-09-13T11:22:58.767 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.5695e-05 2024-09-13T11:22:58.767 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:58.767 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:58.767 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:22:58.768 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:58.748101+0000", 2024-09-13T11:22:58.768 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000118363 2024-09-13T11:22:58.768 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:58.768 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:58.768 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 8", 2024-09-13T11:22:58.768 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:58.748220+0000", 2024-09-13T11:22:58.768 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000119021 2024-09-13T11:22:58.768 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:22:58.768 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:22:58.768 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:22:58.768 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:58.768 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:58.768 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:16509 3.b 3:d67b09f2:::benchmark_data_smithi177_31116_object16508:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-13T11:22:58.768 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:58.748394+0000", 2024-09-13T11:22:58.769 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 0.001891446, 2024-09-13T11:22:58.769 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0021117269999999999, 2024-09-13T11:22:58.769 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:22:58.769 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:22:58.769 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "waiting for sub ops", 2024-09-13T11:22:58.769 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:22:58.769 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:22:58.769 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:22:58.769 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 16509 2024-09-13T11:22:58.769 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:58.769 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:22:58.769 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:58.769 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:22:58.770 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:58.748394+0000", 2024-09-13T11:22:58.770 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:22:58.770 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:58.770 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:58.770 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:22:58.770 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:58.748394+0000", 2024-09-13T11:22:58.770 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:22:58.770 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:58.770 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:58.770 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:22:58.770 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:58.748395+0000", 2024-09-13T11:22:58.770 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.1820000000000001e-06 2024-09-13T11:22:58.770 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:58.771 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:58.771 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:22:58.771 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:58.748463+0000", 2024-09-13T11:22:58.771 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.8071999999999995e-05 2024-09-13T11:22:58.771 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:58.771 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:58.771 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:22:58.771 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:58.748475+0000", 2024-09-13T11:22:58.771 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.2165e-05 2024-09-13T11:22:58.771 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:58.771 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:58.771 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:22:58.771 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:58.748481+0000", 2024-09-13T11:22:58.772 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.2460000000000003e-06 2024-09-13T11:22:58.772 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:58.772 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:58.772 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:22:58.772 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:58.748561+0000", 2024-09-13T11:22:58.772 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.9951999999999996e-05 2024-09-13T11:22:58.772 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:58.772 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:58.772 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:22:58.772 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:58.748721+0000", 2024-09-13T11:22:58.772 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00015980399999999999 2024-09-13T11:22:58.772 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:58.772 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:58.773 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1", 2024-09-13T11:22:58.773 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:58.748871+0000", 2024-09-13T11:22:58.773 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00015018400000000001 2024-09-13T11:22:58.773 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:22:58.773 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:22:58.773 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:22:58.773 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:58.773 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:58.773 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop_reply(client.4665.0:16508 3.7 e52/30)", 2024-09-13T11:22:58.773 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:58.749286+0000", 2024-09-13T11:22:58.773 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 0.00099973100000000001, 2024-09-13T11:22:58.773 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0012585840000000001, 2024-09-13T11:22:58.773 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:22:58.774 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:22:58.774 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "queued for pg", 2024-09-13T11:22:58.774 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:22:58.774 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:58.774 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:22:58.774 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:58.749286+0000", 2024-09-13T11:22:58.774 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:22:58.774 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:58.774 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:58.774 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:22:58.774 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:58.749286+0000", 2024-09-13T11:22:58.774 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:22:58.774 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:58.774 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:58.775 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:22:58.775 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:58.749287+0000", 2024-09-13T11:22:58.775 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.6050000000000001e-06 2024-09-13T11:22:58.775 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:58.775 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:58.775 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:22:58.775 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:58.749292+0000", 2024-09-13T11:22:58.775 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.9740000000000001e-06 2024-09-13T11:22:58.775 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:58.775 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:58.775 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:22:58.775 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:58.749322+0000", 2024-09-13T11:22:58.775 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.0031000000000001e-05 2024-09-13T11:22:58.776 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:58.776 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:58.776 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:22:58.776 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:58.749332+0000", 2024-09-13T11:22:58.776 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.1079999999999999e-06 2024-09-13T11:22:58.776 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:22:58.776 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:22:58.776 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:22:58.776 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:58.776 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:58.776 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:16510 3.1 3:8467a2dc:::benchmark_data_smithi177_31116_object16509:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-13T11:22:58.776 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:58.749650+0000", 2024-09-13T11:22:58.776 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 0.00063554499999999995, 2024-09-13T11:22:58.776 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00092411800000000003, 2024-09-13T11:22:58.777 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:22:58.777 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:22:58.777 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "waiting for sub ops", 2024-09-13T11:22:58.777 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:22:58.777 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:22:58.777 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:22:58.777 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 16510 2024-09-13T11:22:58.777 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:58.777 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:22:58.777 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:58.777 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:22:58.777 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:58.749650+0000", 2024-09-13T11:22:58.777 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:22:58.778 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:58.778 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:58.778 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:22:58.778 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:58.749650+0000", 2024-09-13T11:22:58.778 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:22:58.778 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:58.778 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:58.778 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:22:58.778 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:58.749652+0000", 2024-09-13T11:22:58.778 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.759e-06 2024-09-13T11:22:58.778 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:58.778 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:58.778 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:22:58.778 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:58.749711+0000", 2024-09-13T11:22:58.779 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.8928000000000002e-05 2024-09-13T11:22:58.779 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:58.779 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:58.779 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:22:58.779 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:58.749725+0000", 2024-09-13T11:22:58.779 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.4355999999999999e-05 2024-09-13T11:22:58.779 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:58.779 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:58.779 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:22:58.779 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:58.749734+0000", 2024-09-13T11:22:58.779 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.4829999999999999e-06 2024-09-13T11:22:58.779 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:58.779 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:58.779 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:22:58.780 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:58.749821+0000", 2024-09-13T11:22:58.780 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.7499999999999999e-05 2024-09-13T11:22:58.780 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:58.780 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:58.780 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:22:58.780 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:58.749973+0000", 2024-09-13T11:22:58.780 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00015184700000000001 2024-09-13T11:22:58.780 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:58.780 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:58.780 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 11", 2024-09-13T11:22:58.780 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:58.750131+0000", 2024-09-13T11:22:58.780 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000158338 2024-09-13T11:22:58.780 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:22:58.781 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:22:58.781 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:22:58.781 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:22:58.781 INFO:teuthology.orchestra.run.smithi092.stdout: ], 2024-09-13T11:22:58.781 INFO:teuthology.orchestra.run.smithi092.stdout: "num_ops": 7 2024-09-13T11:22:58.781 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-09-13T11:22:58.782 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_blocked_ops 2024-09-13T11:22:58.783 INFO:tasks.ceph.osd.11.smithi177.stderr:2024-09-13T11:22:58.783+0000 7f24792c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26580) UID: 0 2024-09-13T11:22:58.883 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-09-13T11:22:58.879+0000 7f97befc9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26315) UID: 0 2024-09-13T11:22:58.918 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-09-13T11:22:58.918 INFO:teuthology.orchestra.run.smithi092.stdout: "ops": [], 2024-09-13T11:22:58.919 INFO:teuthology.orchestra.run.smithi092.stdout: "complaint_time": 30, 2024-09-13T11:22:58.919 INFO:teuthology.orchestra.run.smithi092.stdout: "num_blocked_ops": 0 2024-09-13T11:22:58.919 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-09-13T11:22:58.933 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_historic_ops 2024-09-13T11:22:58.984 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-09-13T11:22:58.979+0000 7f97befc9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26315) UID: 0 2024-09-13T11:22:59.000 INFO:tasks.radosbench.radosbench.0.smithi177.stdout: 32 16 16753 16737 32.6853 93.6875 0.00822389 0.0305852 2024-09-13T11:22:59.074 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-09-13T11:22:59.074 INFO:teuthology.orchestra.run.smithi092.stdout: "size": 20, 2024-09-13T11:22:59.074 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 600, 2024-09-13T11:22:59.074 INFO:teuthology.orchestra.run.smithi092.stdout: "ops": [ 2024-09-13T11:22:59.074 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.074 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:79 3.1 3:841df5b5:::benchmark_data_smithi177_31116_object78:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-09-13T11:22:59.074 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:27.056083+0000", 2024-09-13T11:22:59.075 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 32.017074522000001, 2024-09-13T11:22:59.075 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.89469730199999997, 2024-09-13T11:22:59.075 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:22:59.075 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:22:59.075 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:59.075 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:22:59.075 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:22:59.075 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:22:59.075 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 79 2024-09-13T11:22:59.075 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.075 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:22:59.075 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.076 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:22:59.076 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.056083+0000", 2024-09-13T11:22:59.076 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:22:59.076 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.076 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.076 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:22:59.076 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.056083+0000", 2024-09-13T11:22:59.076 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:22:59.076 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.076 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.076 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:22:59.076 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.056087+0000", 2024-09-13T11:22:59.076 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.0659999999999997e-06 2024-09-13T11:22:59.077 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.077 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.077 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:22:59.077 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.056148+0000", 2024-09-13T11:22:59.077 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.1376000000000003e-05 2024-09-13T11:22:59.077 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.077 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.077 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:22:59.077 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.056180+0000", 2024-09-13T11:22:59.077 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.2370999999999999e-05 2024-09-13T11:22:59.079 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.079 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.079 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:22:59.079 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.056195+0000", 2024-09-13T11:22:59.079 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.5119e-05 2024-09-13T11:22:59.079 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.079 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.079 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:22:59.079 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.056304+0000", 2024-09-13T11:22:59.079 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000108365 2024-09-13T11:22:59.079 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.080 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.080 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:22:59.080 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.056478+0000", 2024-09-13T11:22:59.080 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000174542 2024-09-13T11:22:59.080 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.080 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.080 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 11", 2024-09-13T11:22:59.080 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.056690+0000", 2024-09-13T11:22:59.080 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00021167500000000001 2024-09-13T11:22:59.080 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.080 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.080 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:22:59.080 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.058063+0000", 2024-09-13T11:22:59.081 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0013731819999999999 2024-09-13T11:22:59.081 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.081 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.081 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:22:59.081 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.950477+0000", 2024-09-13T11:22:59.081 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.89241369400000004 2024-09-13T11:22:59.081 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.081 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.081 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:22:59.081 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.950652+0000", 2024-09-13T11:22:59.081 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00017526400000000001 2024-09-13T11:22:59.081 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.082 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.082 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:22:59.082 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.950780+0000", 2024-09-13T11:22:59.082 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00012764800000000001 2024-09-13T11:22:59.082 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:22:59.082 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:22:59.082 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:22:59.082 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.082 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.082 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:7 3.b 3:de78393b:::benchmark_data_smithi177_31116_object6:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T11:22:59.082 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:48.313952+0000", 2024-09-13T11:22:59.082 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 10.759204773, 2024-09-13T11:22:59.082 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.86421288600000001, 2024-09-13T11:22:59.083 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:22:59.083 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:22:59.083 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:59.083 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:22:59.083 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:22:59.083 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:22:59.083 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 7 2024-09-13T11:22:59.083 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.083 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:22:59.083 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.083 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:22:59.083 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.313952+0000", 2024-09-13T11:22:59.084 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:22:59.084 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.084 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.084 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:22:59.084 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.313952+0000", 2024-09-13T11:22:59.084 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:22:59.084 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.084 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.084 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:22:59.084 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.313957+0000", 2024-09-13T11:22:59.084 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.3599999999999998e-06 2024-09-13T11:22:59.084 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.084 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.085 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:22:59.085 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314043+0000", 2024-09-13T11:22:59.085 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.6766000000000002e-05 2024-09-13T11:22:59.085 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.085 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.085 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:22:59.085 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314065+0000", 2024-09-13T11:22:59.085 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.1416e-05 2024-09-13T11:22:59.085 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.085 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.085 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:22:59.085 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314080+0000", 2024-09-13T11:22:59.085 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.5519000000000002e-05 2024-09-13T11:22:59.086 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.086 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.086 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:22:59.086 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315478+0000", 2024-09-13T11:22:59.086 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.001397765 2024-09-13T11:22:59.086 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.086 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.086 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-09-13T11:22:59.086 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315507+0000", 2024-09-13T11:22:59.086 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.9128999999999998e-05 2024-09-13T11:22:59.086 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.086 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.086 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:22:59.086 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.167978+0000", 2024-09-13T11:22:59.087 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.85247068400000003 2024-09-13T11:22:59.087 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.087 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.087 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:22:59.087 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.168307+0000", 2024-09-13T11:22:59.087 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00032886699999999999 2024-09-13T11:22:59.087 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.087 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.087 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1", 2024-09-13T11:22:59.087 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.168647+0000", 2024-09-13T11:22:59.087 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00033998399999999999 2024-09-13T11:22:59.087 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.087 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.088 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:22:59.088 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.178011+0000", 2024-09-13T11:22:59.088 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0093639169999999994 2024-09-13T11:22:59.088 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.088 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.088 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:22:59.088 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.178026+0000", 2024-09-13T11:22:59.088 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.5274999999999999e-05 2024-09-13T11:22:59.088 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.088 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.088 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:22:59.088 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.178080+0000", 2024-09-13T11:22:59.088 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.3601999999999999e-05 2024-09-13T11:22:59.088 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.089 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.089 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:22:59.089 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.178165+0000", 2024-09-13T11:22:59.089 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.5601999999999998e-05 2024-09-13T11:22:59.089 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:22:59.089 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:22:59.089 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:22:59.089 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.089 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.089 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:9 3.b 3:dafadff6:::benchmark_data_smithi177_31116_object8:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T11:22:59.089 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:48.314114+0000", 2024-09-13T11:22:59.089 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 10.759042959, 2024-09-13T11:22:59.089 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.86613615899999996, 2024-09-13T11:22:59.089 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:22:59.090 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:22:59.090 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:59.090 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:22:59.090 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:22:59.090 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:22:59.090 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 9 2024-09-13T11:22:59.090 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.090 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:22:59.090 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.090 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:22:59.090 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314114+0000", 2024-09-13T11:22:59.090 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:22:59.090 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.091 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.091 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:22:59.091 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314114+0000", 2024-09-13T11:22:59.091 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:22:59.091 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.091 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.091 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:22:59.091 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314117+0000", 2024-09-13T11:22:59.091 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.3860000000000001e-06 2024-09-13T11:22:59.091 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.091 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.091 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:22:59.092 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314215+0000", 2024-09-13T11:22:59.092 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.8900000000000005e-05 2024-09-13T11:22:59.092 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.092 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.092 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:22:59.092 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314229+0000", 2024-09-13T11:22:59.092 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.3444999999999999e-05 2024-09-13T11:22:59.092 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.092 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.092 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:22:59.092 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314239+0000", 2024-09-13T11:22:59.092 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.8470000000000008e-06 2024-09-13T11:22:59.093 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.093 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.093 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:22:59.093 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315648+0000", 2024-09-13T11:22:59.093 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.001409291 2024-09-13T11:22:59.093 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.093 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.093 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-09-13T11:22:59.093 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315671+0000", 2024-09-13T11:22:59.093 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.3169000000000001e-05 2024-09-13T11:22:59.093 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.093 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.093 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:22:59.094 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.176036+0000", 2024-09-13T11:22:59.094 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.86036497499999998 2024-09-13T11:22:59.094 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.094 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.094 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:22:59.094 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.176189+0000", 2024-09-13T11:22:59.094 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00015271100000000001 2024-09-13T11:22:59.094 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.094 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.094 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1", 2024-09-13T11:22:59.094 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.176339+0000", 2024-09-13T11:22:59.094 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00015057499999999999 2024-09-13T11:22:59.094 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.094 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.095 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:22:59.095 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.180148+0000", 2024-09-13T11:22:59.095 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.003808891 2024-09-13T11:22:59.095 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.095 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.095 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:22:59.095 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.180168+0000", 2024-09-13T11:22:59.095 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.9507000000000001e-05 2024-09-13T11:22:59.095 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.095 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.095 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:22:59.095 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.180213+0000", 2024-09-13T11:22:59.095 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.5504999999999999e-05 2024-09-13T11:22:59.096 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.096 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.096 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:22:59.096 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.180250+0000", 2024-09-13T11:22:59.096 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.6956999999999999e-05 2024-09-13T11:22:59.096 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:22:59.096 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:22:59.096 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:22:59.096 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.096 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.096 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:10 3.b 3:dbe028a8:::benchmark_data_smithi177_31116_object9:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T11:22:59.096 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:48.314259+0000", 2024-09-13T11:22:59.096 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 10.758897829, 2024-09-13T11:22:59.097 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.86637313599999999, 2024-09-13T11:22:59.097 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:22:59.097 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:22:59.097 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:59.097 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:22:59.097 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:22:59.097 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:22:59.097 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 10 2024-09-13T11:22:59.097 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.097 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:22:59.097 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.097 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:22:59.097 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314259+0000", 2024-09-13T11:22:59.097 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:22:59.098 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.098 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.098 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:22:59.098 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314259+0000", 2024-09-13T11:22:59.098 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:22:59.098 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.098 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.098 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:22:59.098 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314261+0000", 2024-09-13T11:22:59.098 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.1500000000000002e-06 2024-09-13T11:22:59.098 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.098 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.098 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:22:59.099 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314349+0000", 2024-09-13T11:22:59.099 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.8070000000000002e-05 2024-09-13T11:22:59.099 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.099 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.099 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:22:59.099 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314365+0000", 2024-09-13T11:22:59.099 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.5835999999999999e-05 2024-09-13T11:22:59.099 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.099 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.099 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:22:59.099 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314377+0000", 2024-09-13T11:22:59.099 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.1589e-05 2024-09-13T11:22:59.099 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.099 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.100 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:22:59.100 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315788+0000", 2024-09-13T11:22:59.100 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0014109579999999999 2024-09-13T11:22:59.100 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.100 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.100 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-09-13T11:22:59.100 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315809+0000", 2024-09-13T11:22:59.100 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.0985999999999999e-05 2024-09-13T11:22:59.100 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.100 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.100 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:22:59.100 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.176822+0000", 2024-09-13T11:22:59.100 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.86101310799999997 2024-09-13T11:22:59.100 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.101 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.101 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:22:59.101 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.177021+0000", 2024-09-13T11:22:59.101 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00019911199999999999 2024-09-13T11:22:59.101 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.101 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.101 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1", 2024-09-13T11:22:59.101 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.177226+0000", 2024-09-13T11:22:59.101 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00020516600000000001 2024-09-13T11:22:59.101 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.101 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.101 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:22:59.102 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.180264+0000", 2024-09-13T11:22:59.102 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0030381739999999998 2024-09-13T11:22:59.102 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.102 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.102 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:22:59.102 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.180387+0000", 2024-09-13T11:22:59.102 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00012257699999999999 2024-09-13T11:22:59.102 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.102 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.102 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:22:59.102 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.180518+0000", 2024-09-13T11:22:59.102 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00013135800000000001 2024-09-13T11:22:59.102 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.102 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.103 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:22:59.103 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.180632+0000", 2024-09-13T11:22:59.103 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000114052 2024-09-13T11:22:59.103 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:22:59.103 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:22:59.103 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:22:59.103 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.103 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.103 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:39 3.b 3:d91854b3:::benchmark_data_smithi177_31116_object38:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T11:22:59.103 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:48.314393+0000", 2024-09-13T11:22:59.103 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 10.758764287, 2024-09-13T11:22:59.103 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.86850121700000005, 2024-09-13T11:22:59.104 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:22:59.104 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:22:59.104 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:59.104 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:22:59.104 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:22:59.104 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:22:59.104 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 39 2024-09-13T11:22:59.104 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.104 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:22:59.104 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.104 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:22:59.104 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314393+0000", 2024-09-13T11:22:59.104 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:22:59.105 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.105 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.105 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:22:59.105 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314393+0000", 2024-09-13T11:22:59.105 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:22:59.105 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.105 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.105 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:22:59.105 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314394+0000", 2024-09-13T11:22:59.105 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.513e-06 2024-09-13T11:22:59.105 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.105 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.105 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:22:59.105 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314463+0000", 2024-09-13T11:22:59.106 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.8261000000000003e-05 2024-09-13T11:22:59.106 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.106 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.106 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:22:59.106 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314477+0000", 2024-09-13T11:22:59.106 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.4619e-05 2024-09-13T11:22:59.106 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.106 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.106 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:22:59.106 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314485+0000", 2024-09-13T11:22:59.106 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.1699999999999997e-06 2024-09-13T11:22:59.106 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.106 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.107 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:22:59.107 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315917+0000", 2024-09-13T11:22:59.107 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0014316089999999999 2024-09-13T11:22:59.107 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.107 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.107 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-09-13T11:22:59.107 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315933+0000", 2024-09-13T11:22:59.107 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.5999999999999999e-05 2024-09-13T11:22:59.107 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.107 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.107 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:22:59.107 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.178205+0000", 2024-09-13T11:22:59.107 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.86227247100000004 2024-09-13T11:22:59.108 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.108 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.108 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:22:59.108 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.178316+0000", 2024-09-13T11:22:59.108 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000110788 2024-09-13T11:22:59.108 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.108 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.108 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1", 2024-09-13T11:22:59.108 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.178448+0000", 2024-09-13T11:22:59.108 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000132056 2024-09-13T11:22:59.108 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.108 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.108 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:22:59.109 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.182721+0000", 2024-09-13T11:22:59.109 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0042729780000000002 2024-09-13T11:22:59.109 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.109 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.109 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:22:59.109 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.182801+0000", 2024-09-13T11:22:59.109 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.0209000000000004e-05 2024-09-13T11:22:59.109 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.109 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.109 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:22:59.109 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.182855+0000", 2024-09-13T11:22:59.109 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.3749000000000003e-05 2024-09-13T11:22:59.109 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.109 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.110 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:22:59.110 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.182894+0000", 2024-09-13T11:22:59.110 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.8794000000000002e-05 2024-09-13T11:22:59.110 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:22:59.110 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:22:59.110 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:22:59.110 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.110 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.110 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:50 3.b 3:d1c144b8:::benchmark_data_smithi177_31116_object49:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T11:22:59.110 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:48.314498+0000", 2024-09-13T11:22:59.110 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 10.758658872, 2024-09-13T11:22:59.110 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.86878673299999998, 2024-09-13T11:22:59.110 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:22:59.111 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:22:59.111 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:59.111 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:22:59.111 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:22:59.111 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:22:59.111 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 50 2024-09-13T11:22:59.111 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.111 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:22:59.111 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.111 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:22:59.111 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314498+0000", 2024-09-13T11:22:59.111 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:22:59.111 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.112 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.112 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:22:59.112 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314498+0000", 2024-09-13T11:22:59.112 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:22:59.112 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.112 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.112 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:22:59.112 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314499+0000", 2024-09-13T11:22:59.112 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.1969999999999999e-06 2024-09-13T11:22:59.112 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.112 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.112 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:22:59.112 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314565+0000", 2024-09-13T11:22:59.112 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.5628000000000003e-05 2024-09-13T11:22:59.113 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.113 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.113 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:22:59.113 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314579+0000", 2024-09-13T11:22:59.113 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.4222e-05 2024-09-13T11:22:59.113 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.113 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.113 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:22:59.113 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314590+0000", 2024-09-13T11:22:59.113 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.1240000000000001e-05 2024-09-13T11:22:59.113 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.113 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.113 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:22:59.114 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316060+0000", 2024-09-13T11:22:59.114 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.001469515 2024-09-13T11:22:59.114 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.114 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.114 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-09-13T11:22:59.114 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316076+0000", 2024-09-13T11:22:59.114 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.5608999999999999e-05 2024-09-13T11:22:59.114 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.114 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.114 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:22:59.114 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.178857+0000", 2024-09-13T11:22:59.114 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.86278099900000005 2024-09-13T11:22:59.114 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.114 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.115 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:22:59.115 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.178961+0000", 2024-09-13T11:22:59.115 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000104664 2024-09-13T11:22:59.115 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.115 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.115 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1", 2024-09-13T11:22:59.115 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.179123+0000", 2024-09-13T11:22:59.115 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000161738 2024-09-13T11:22:59.115 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.115 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.115 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:22:59.115 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.182755+0000", 2024-09-13T11:22:59.115 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0036323200000000001 2024-09-13T11:22:59.115 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.116 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.116 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:22:59.116 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.183023+0000", 2024-09-13T11:22:59.116 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00026749100000000002 2024-09-13T11:22:59.116 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.116 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.116 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:22:59.116 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.183168+0000", 2024-09-13T11:22:59.116 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00014499099999999999 2024-09-13T11:22:59.116 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.116 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.116 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:22:59.116 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.183285+0000", 2024-09-13T11:22:59.116 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000117119 2024-09-13T11:22:59.117 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:22:59.117 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:22:59.117 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:22:59.117 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.117 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.117 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:60 3.b 3:d67218b0:::benchmark_data_smithi177_31116_object59:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T11:22:59.117 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:48.314603+0000", 2024-09-13T11:22:59.117 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 10.758554309000001, 2024-09-13T11:22:59.117 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.87106201100000002, 2024-09-13T11:22:59.117 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:22:59.117 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:22:59.117 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:59.117 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:22:59.117 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:22:59.118 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:22:59.118 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 60 2024-09-13T11:22:59.118 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.118 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:22:59.118 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.118 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:22:59.118 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314603+0000", 2024-09-13T11:22:59.118 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:22:59.118 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.118 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.118 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:22:59.118 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314603+0000", 2024-09-13T11:22:59.118 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:22:59.118 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.119 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.119 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:22:59.119 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314604+0000", 2024-09-13T11:22:59.119 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.2359999999999999e-06 2024-09-13T11:22:59.119 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.119 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.119 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:22:59.119 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314667+0000", 2024-09-13T11:22:59.119 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.3077999999999995e-05 2024-09-13T11:22:59.119 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.119 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.119 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:22:59.119 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314699+0000", 2024-09-13T11:22:59.120 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.1965999999999999e-05 2024-09-13T11:22:59.120 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.120 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.120 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:22:59.120 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314710+0000", 2024-09-13T11:22:59.120 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.1148999999999999e-05 2024-09-13T11:22:59.120 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.120 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.120 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:22:59.120 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316291+0000", 2024-09-13T11:22:59.120 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0015804440000000001 2024-09-13T11:22:59.120 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.121 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.121 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-09-13T11:22:59.121 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316310+0000", 2024-09-13T11:22:59.121 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.9799000000000001e-05 2024-09-13T11:22:59.121 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.121 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.121 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:22:59.121 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.179555+0000", 2024-09-13T11:22:59.121 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.86324469500000001 2024-09-13T11:22:59.121 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.121 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.121 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:22:59.121 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.179666+0000", 2024-09-13T11:22:59.121 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000110443 2024-09-13T11:22:59.122 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.122 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.122 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1", 2024-09-13T11:22:59.122 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.179791+0000", 2024-09-13T11:22:59.122 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00012560499999999999 2024-09-13T11:22:59.122 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.122 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.122 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:22:59.122 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.182761+0000", 2024-09-13T11:22:59.122 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0029697019999999998 2024-09-13T11:22:59.122 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.122 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.122 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:22:59.122 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.185572+0000", 2024-09-13T11:22:59.122 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.002811512 2024-09-13T11:22:59.123 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.123 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.123 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:22:59.123 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.185625+0000", 2024-09-13T11:22:59.123 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.2716000000000001e-05 2024-09-13T11:22:59.123 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.123 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.123 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:22:59.123 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.185665+0000", 2024-09-13T11:22:59.123 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.9666000000000003e-05 2024-09-13T11:22:59.123 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:22:59.123 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:22:59.124 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:22:59.124 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.124 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.124 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:62 3.b 3:d20ae5a8:::benchmark_data_smithi177_31116_object61:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T11:22:59.124 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:48.314723+0000", 2024-09-13T11:22:59.124 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 10.758433767, 2024-09-13T11:22:59.124 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.87338985099999999, 2024-09-13T11:22:59.124 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:22:59.124 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:22:59.124 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:59.124 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:22:59.124 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:22:59.124 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:22:59.124 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 62 2024-09-13T11:22:59.125 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.125 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:22:59.125 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.125 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:22:59.125 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314723+0000", 2024-09-13T11:22:59.125 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:22:59.125 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.125 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.125 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:22:59.125 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314723+0000", 2024-09-13T11:22:59.125 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:22:59.125 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.125 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.125 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:22:59.126 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314725+0000", 2024-09-13T11:22:59.126 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.3069999999999999e-06 2024-09-13T11:22:59.126 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.126 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.126 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:22:59.126 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314797+0000", 2024-09-13T11:22:59.126 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.2652999999999997e-05 2024-09-13T11:22:59.126 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.126 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.126 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:22:59.126 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314809+0000", 2024-09-13T11:22:59.126 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.1496e-05 2024-09-13T11:22:59.126 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.126 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.127 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:22:59.127 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314816+0000", 2024-09-13T11:22:59.127 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.6639999999999998e-06 2024-09-13T11:22:59.127 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.127 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.127 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:22:59.127 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316424+0000", 2024-09-13T11:22:59.127 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0016077559999999999 2024-09-13T11:22:59.127 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.127 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.127 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-09-13T11:22:59.127 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316441+0000", 2024-09-13T11:22:59.127 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.7305000000000001e-05 2024-09-13T11:22:59.127 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.128 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.128 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:22:59.128 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.180688+0000", 2024-09-13T11:22:59.128 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.86424692800000003 2024-09-13T11:22:59.128 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.128 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.128 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:22:59.128 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.180807+0000", 2024-09-13T11:22:59.128 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000118977 2024-09-13T11:22:59.128 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.128 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.128 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1", 2024-09-13T11:22:59.129 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.180938+0000", 2024-09-13T11:22:59.129 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00013049700000000001 2024-09-13T11:22:59.129 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.129 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.129 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:22:59.129 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.185680+0000", 2024-09-13T11:22:59.129 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0047417029999999999 2024-09-13T11:22:59.129 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.129 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.129 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:22:59.129 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.187856+0000", 2024-09-13T11:22:59.129 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0021760550000000001 2024-09-13T11:22:59.129 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.129 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.130 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:22:59.130 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.188007+0000", 2024-09-13T11:22:59.130 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00015185099999999999 2024-09-13T11:22:59.130 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.130 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.130 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:22:59.130 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.188113+0000", 2024-09-13T11:22:59.130 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000105659 2024-09-13T11:22:59.130 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:22:59.130 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:22:59.130 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:22:59.130 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.130 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.130 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:73 3.b 3:d0060d5f:::benchmark_data_smithi177_31116_object72:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T11:22:59.130 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:48.314829+0000", 2024-09-13T11:22:59.131 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 10.758328418, 2024-09-13T11:22:59.131 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.87364757199999998, 2024-09-13T11:22:59.131 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:22:59.131 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:22:59.131 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:59.131 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:22:59.131 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:22:59.131 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:22:59.131 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 73 2024-09-13T11:22:59.131 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.131 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:22:59.131 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.131 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:22:59.131 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314829+0000", 2024-09-13T11:22:59.132 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:22:59.132 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.132 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.132 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:22:59.132 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314829+0000", 2024-09-13T11:22:59.132 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:22:59.132 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.132 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.132 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:22:59.132 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314830+0000", 2024-09-13T11:22:59.132 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.243e-06 2024-09-13T11:22:59.132 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.132 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.132 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:22:59.133 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314965+0000", 2024-09-13T11:22:59.133 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00013470800000000001 2024-09-13T11:22:59.133 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.133 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.133 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:22:59.133 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314979+0000", 2024-09-13T11:22:59.133 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.4375999999999999e-05 2024-09-13T11:22:59.133 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.133 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.133 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:22:59.133 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314988+0000", 2024-09-13T11:22:59.133 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.8570000000000001e-06 2024-09-13T11:22:59.133 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.134 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.134 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:22:59.134 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316545+0000", 2024-09-13T11:22:59.134 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.001556878 2024-09-13T11:22:59.134 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.134 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.134 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-09-13T11:22:59.134 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316556+0000", 2024-09-13T11:22:59.134 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.1453e-05 2024-09-13T11:22:59.134 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.134 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.134 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:22:59.134 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.182124+0000", 2024-09-13T11:22:59.134 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.86556744200000002 2024-09-13T11:22:59.134 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.135 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.135 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:22:59.135 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.182246+0000", 2024-09-13T11:22:59.135 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00012210500000000001 2024-09-13T11:22:59.135 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.135 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.135 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1", 2024-09-13T11:22:59.135 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.182375+0000", 2024-09-13T11:22:59.135 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00012920999999999999 2024-09-13T11:22:59.135 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.135 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.135 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:22:59.136 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.185684+0000", 2024-09-13T11:22:59.136 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0033093969999999999 2024-09-13T11:22:59.136 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.136 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.136 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:22:59.136 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.188244+0000", 2024-09-13T11:22:59.136 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0025601550000000002 2024-09-13T11:22:59.136 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.136 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.136 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:22:59.136 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.188379+0000", 2024-09-13T11:22:59.136 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00013463400000000001 2024-09-13T11:22:59.137 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.137 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.137 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:22:59.137 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.188476+0000", 2024-09-13T11:22:59.137 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.7114000000000006e-05 2024-09-13T11:22:59.137 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:22:59.137 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:22:59.137 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:22:59.137 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.137 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.137 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:81 3.b 3:d705fd70:::benchmark_data_smithi177_31116_object80:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T11:22:59.137 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:48.315000+0000", 2024-09-13T11:22:59.137 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 10.758156581, 2024-09-13T11:22:59.137 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.87561418300000005, 2024-09-13T11:22:59.138 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:22:59.138 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:22:59.138 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:59.138 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:22:59.138 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:22:59.138 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:22:59.138 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 81 2024-09-13T11:22:59.138 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.138 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:22:59.138 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.138 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:22:59.138 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315000+0000", 2024-09-13T11:22:59.138 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:22:59.139 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.139 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.139 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:22:59.139 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315000+0000", 2024-09-13T11:22:59.139 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:22:59.139 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.139 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.139 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:22:59.139 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315002+0000", 2024-09-13T11:22:59.139 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.3799999999999999e-06 2024-09-13T11:22:59.139 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.140 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.140 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:22:59.140 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315119+0000", 2024-09-13T11:22:59.140 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000116925 2024-09-13T11:22:59.140 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.140 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.140 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:22:59.140 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315133+0000", 2024-09-13T11:22:59.140 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.3971999999999999e-05 2024-09-13T11:22:59.140 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.140 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.140 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:22:59.141 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315142+0000", 2024-09-13T11:22:59.141 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.4399999999999994e-06 2024-09-13T11:22:59.141 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.141 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.141 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:22:59.141 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316652+0000", 2024-09-13T11:22:59.141 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.001509907 2024-09-13T11:22:59.141 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.141 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.141 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-09-13T11:22:59.141 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316668+0000", 2024-09-13T11:22:59.141 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.5945999999999999e-05 2024-09-13T11:22:59.141 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.142 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.142 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:22:59.142 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.183405+0000", 2024-09-13T11:22:59.142 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.866737384 2024-09-13T11:22:59.142 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.142 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.142 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:22:59.142 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.183701+0000", 2024-09-13T11:22:59.142 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00029511099999999997 2024-09-13T11:22:59.142 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.142 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.142 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1", 2024-09-13T11:22:59.142 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.184078+0000", 2024-09-13T11:22:59.142 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00037741000000000001 2024-09-13T11:22:59.143 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.143 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.143 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:22:59.143 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.188504+0000", 2024-09-13T11:22:59.143 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0044260280000000003 2024-09-13T11:22:59.143 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.143 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.143 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:22:59.143 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.190562+0000", 2024-09-13T11:22:59.143 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0020580099999999999 2024-09-13T11:22:59.143 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.143 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.143 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:22:59.144 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.190592+0000", 2024-09-13T11:22:59.144 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.0057999999999999e-05 2024-09-13T11:22:59.144 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.144 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.144 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:22:59.144 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.190615+0000", 2024-09-13T11:22:59.144 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.2612e-05 2024-09-13T11:22:59.144 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:22:59.144 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:22:59.144 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:22:59.144 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.144 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.144 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:87 3.b 3:dd7de0ad:::benchmark_data_smithi177_31116_object86:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T11:22:59.144 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:48.315155+0000", 2024-09-13T11:22:59.145 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 10.758001783999999, 2024-09-13T11:22:59.145 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.87555916099999997, 2024-09-13T11:22:59.145 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:22:59.145 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:22:59.145 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:59.145 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:22:59.145 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:22:59.145 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:22:59.145 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 87 2024-09-13T11:22:59.145 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.145 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:22:59.145 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.145 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:22:59.146 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315155+0000", 2024-09-13T11:22:59.146 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:22:59.146 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.146 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.146 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:22:59.146 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315155+0000", 2024-09-13T11:22:59.146 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:22:59.146 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.146 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.146 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:22:59.146 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315157+0000", 2024-09-13T11:22:59.146 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.3829999999999999e-06 2024-09-13T11:22:59.146 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.147 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.147 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:22:59.147 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315271+0000", 2024-09-13T11:22:59.147 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000114539 2024-09-13T11:22:59.147 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.147 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.147 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:22:59.147 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315285+0000", 2024-09-13T11:22:59.147 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.4042e-05 2024-09-13T11:22:59.147 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.147 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.147 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:22:59.147 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315295+0000", 2024-09-13T11:22:59.147 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.2739999999999996e-06 2024-09-13T11:22:59.148 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.148 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.148 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:22:59.148 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316739+0000", 2024-09-13T11:22:59.148 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0014446960000000001 2024-09-13T11:22:59.148 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.148 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.148 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-09-13T11:22:59.148 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316747+0000", 2024-09-13T11:22:59.148 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.0790000000000001e-06 2024-09-13T11:22:59.148 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.148 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.148 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:22:59.149 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.184922+0000", 2024-09-13T11:22:59.149 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.86817446399999998 2024-09-13T11:22:59.149 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.149 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.149 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:22:59.149 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.185037+0000", 2024-09-13T11:22:59.149 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00011565099999999999 2024-09-13T11:22:59.149 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.149 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.149 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1", 2024-09-13T11:22:59.149 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.185171+0000", 2024-09-13T11:22:59.149 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00013316900000000001 2024-09-13T11:22:59.149 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.149 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.150 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:22:59.150 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.188520+0000", 2024-09-13T11:22:59.150 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0033489150000000001 2024-09-13T11:22:59.150 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.150 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.150 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:22:59.150 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.190652+0000", 2024-09-13T11:22:59.150 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0021325649999999999 2024-09-13T11:22:59.150 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.150 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.150 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:22:59.150 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.190686+0000", 2024-09-13T11:22:59.150 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.3827e-05 2024-09-13T11:22:59.150 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.151 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.151 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:22:59.151 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.190714+0000", 2024-09-13T11:22:59.151 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.8557000000000001e-05 2024-09-13T11:22:59.151 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:22:59.151 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:22:59.151 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:22:59.151 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.151 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.151 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:99 3.b 3:d011b600:::benchmark_data_smithi177_31116_object98:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T11:22:59.151 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:48.315307+0000", 2024-09-13T11:22:59.151 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 10.757849956999999, 2024-09-13T11:22:59.151 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.87551770799999995, 2024-09-13T11:22:59.152 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:22:59.152 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:22:59.152 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:59.152 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:22:59.152 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:22:59.152 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:22:59.152 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 99 2024-09-13T11:22:59.152 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.152 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:22:59.152 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.152 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:22:59.152 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315307+0000", 2024-09-13T11:22:59.153 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:22:59.153 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.153 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.153 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:22:59.153 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315307+0000", 2024-09-13T11:22:59.153 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:22:59.153 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.153 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.153 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:22:59.153 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315308+0000", 2024-09-13T11:22:59.153 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.316e-06 2024-09-13T11:22:59.153 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.154 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.154 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:22:59.154 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315453+0000", 2024-09-13T11:22:59.154 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00014473200000000001 2024-09-13T11:22:59.154 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.154 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.154 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:22:59.154 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315471+0000", 2024-09-13T11:22:59.154 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.8019000000000001e-05 2024-09-13T11:22:59.154 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.154 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.154 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:22:59.154 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315489+0000", 2024-09-13T11:22:59.154 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.7861999999999999e-05 2024-09-13T11:22:59.155 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.155 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.155 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:22:59.155 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316837+0000", 2024-09-13T11:22:59.155 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.001347787 2024-09-13T11:22:59.155 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.155 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.155 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-09-13T11:22:59.155 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316858+0000", 2024-09-13T11:22:59.155 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.1103000000000001e-05 2024-09-13T11:22:59.155 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.155 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.156 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:22:59.156 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.185713+0000", 2024-09-13T11:22:59.156 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.86885540800000005 2024-09-13T11:22:59.156 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.156 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.156 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:22:59.156 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.185820+0000", 2024-09-13T11:22:59.156 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000107081 2024-09-13T11:22:59.156 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.156 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.156 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1", 2024-09-13T11:22:59.156 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.185948+0000", 2024-09-13T11:22:59.156 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00012775800000000001 2024-09-13T11:22:59.157 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.157 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.157 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:22:59.157 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.190527+0000", 2024-09-13T11:22:59.157 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0045783910000000002 2024-09-13T11:22:59.157 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.157 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.157 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:22:59.157 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.190759+0000", 2024-09-13T11:22:59.157 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00023266199999999999 2024-09-13T11:22:59.157 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.158 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.158 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:22:59.158 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.190789+0000", 2024-09-13T11:22:59.158 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.9666999999999999e-05 2024-09-13T11:22:59.158 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.158 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.158 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:22:59.158 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.190825+0000", 2024-09-13T11:22:59.158 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.5921999999999999e-05 2024-09-13T11:22:59.158 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:22:59.158 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:22:59.158 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:22:59.158 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.159 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.159 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:108 3.b 3:d5c90217:::benchmark_data_smithi177_31116_object107:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T11:22:59.159 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:48.315513+0000", 2024-09-13T11:22:59.159 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 10.757644252, 2024-09-13T11:22:59.159 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.87544772500000001, 2024-09-13T11:22:59.159 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:22:59.159 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:22:59.159 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:59.159 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:22:59.159 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:22:59.159 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:22:59.159 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 108 2024-09-13T11:22:59.159 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.159 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:22:59.160 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.160 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:22:59.160 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315513+0000", 2024-09-13T11:22:59.160 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:22:59.160 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.160 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.160 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:22:59.160 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315513+0000", 2024-09-13T11:22:59.160 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:22:59.160 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.160 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.160 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:22:59.160 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315515+0000", 2024-09-13T11:22:59.161 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.9369999999999998e-06 2024-09-13T11:22:59.161 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.161 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.161 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:22:59.161 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315644+0000", 2024-09-13T11:22:59.161 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000129011 2024-09-13T11:22:59.161 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.161 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.161 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:22:59.161 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315657+0000", 2024-09-13T11:22:59.161 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.3689999999999999e-05 2024-09-13T11:22:59.161 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.162 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.162 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:22:59.162 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315668+0000", 2024-09-13T11:22:59.162 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0589e-05 2024-09-13T11:22:59.162 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.162 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.162 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:22:59.162 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316925+0000", 2024-09-13T11:22:59.162 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0012568340000000001 2024-09-13T11:22:59.162 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.162 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.162 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-09-13T11:22:59.162 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316932+0000", 2024-09-13T11:22:59.162 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.2230000000000001e-06 2024-09-13T11:22:59.163 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.163 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.163 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:22:59.163 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.186619+0000", 2024-09-13T11:22:59.163 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.86968679599999998 2024-09-13T11:22:59.163 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.163 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.163 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:22:59.163 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.186914+0000", 2024-09-13T11:22:59.163 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00029528900000000001 2024-09-13T11:22:59.163 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.163 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.163 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1", 2024-09-13T11:22:59.163 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.187217+0000", 2024-09-13T11:22:59.164 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00030232099999999998 2024-09-13T11:22:59.164 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.164 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.164 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:22:59.164 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.190532+0000", 2024-09-13T11:22:59.164 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.003315391 2024-09-13T11:22:59.164 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.164 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.164 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:22:59.164 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.190875+0000", 2024-09-13T11:22:59.164 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00034326300000000001 2024-09-13T11:22:59.164 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.165 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.165 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:22:59.165 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.190925+0000", 2024-09-13T11:22:59.165 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.9429999999999999e-05 2024-09-13T11:22:59.165 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.165 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.165 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:22:59.165 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.190961+0000", 2024-09-13T11:22:59.165 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.5951000000000002e-05 2024-09-13T11:22:59.165 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:22:59.165 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:22:59.165 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:22:59.165 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.165 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.166 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:172 3.b 3:dbec1eeb:::benchmark_data_smithi177_31116_object171:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T11:22:59.166 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:48.315684+0000", 2024-09-13T11:22:59.166 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 10.757473095, 2024-09-13T11:22:59.166 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.87615312999999995, 2024-09-13T11:22:59.166 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:22:59.166 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:22:59.166 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:59.166 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:22:59.166 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:22:59.166 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:22:59.166 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 172 2024-09-13T11:22:59.166 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.166 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:22:59.167 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.167 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:22:59.167 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315684+0000", 2024-09-13T11:22:59.167 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:22:59.167 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.167 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.167 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:22:59.167 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315684+0000", 2024-09-13T11:22:59.167 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:22:59.167 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.167 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.167 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:22:59.167 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315686+0000", 2024-09-13T11:22:59.167 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.886e-06 2024-09-13T11:22:59.168 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.168 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.168 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:22:59.168 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315820+0000", 2024-09-13T11:22:59.168 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00013388500000000001 2024-09-13T11:22:59.168 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.168 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.168 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:22:59.168 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315837+0000", 2024-09-13T11:22:59.168 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.6756e-05 2024-09-13T11:22:59.168 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.168 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.168 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:22:59.168 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315851+0000", 2024-09-13T11:22:59.169 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.4566000000000001e-05 2024-09-13T11:22:59.169 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.169 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.169 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:22:59.169 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.317021+0000", 2024-09-13T11:22:59.169 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0011701769999999999 2024-09-13T11:22:59.169 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.169 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.169 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-09-13T11:22:59.169 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.317037+0000", 2024-09-13T11:22:59.169 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.609e-05 2024-09-13T11:22:59.169 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.169 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.169 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:22:59.170 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.188634+0000", 2024-09-13T11:22:59.170 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.87159711900000003 2024-09-13T11:22:59.170 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.170 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.170 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:22:59.170 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.188974+0000", 2024-09-13T11:22:59.170 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00033922299999999999 2024-09-13T11:22:59.170 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.170 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.170 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1", 2024-09-13T11:22:59.170 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.189510+0000", 2024-09-13T11:22:59.170 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00053618999999999997 2024-09-13T11:22:59.170 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.170 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.171 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:22:59.171 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.191027+0000", 2024-09-13T11:22:59.171 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0015168390000000001 2024-09-13T11:22:59.171 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.171 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.171 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:22:59.171 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.191766+0000", 2024-09-13T11:22:59.171 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000739064 2024-09-13T11:22:59.171 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.171 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.171 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:22:59.171 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.191810+0000", 2024-09-13T11:22:59.171 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.3755999999999999e-05 2024-09-13T11:22:59.171 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.171 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.172 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:22:59.172 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.191837+0000", 2024-09-13T11:22:59.172 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.7579000000000001e-05 2024-09-13T11:22:59.172 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:22:59.172 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:22:59.172 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:22:59.172 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.172 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.172 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:177 3.b 3:de89c779:::benchmark_data_smithi177_31116_object176:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T11:22:59.172 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:48.315898+0000", 2024-09-13T11:22:59.172 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 10.757258780000001, 2024-09-13T11:22:59.172 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.87600712599999997, 2024-09-13T11:22:59.172 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:22:59.173 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:22:59.173 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:59.173 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:22:59.173 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:22:59.173 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:22:59.173 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 177 2024-09-13T11:22:59.173 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.173 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:22:59.173 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.173 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:22:59.173 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315898+0000", 2024-09-13T11:22:59.173 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:22:59.173 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.173 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.174 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:22:59.174 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315898+0000", 2024-09-13T11:22:59.174 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:22:59.174 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.174 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.174 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:22:59.174 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315900+0000", 2024-09-13T11:22:59.174 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.4929999999999999e-06 2024-09-13T11:22:59.174 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.174 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.174 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:22:59.174 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316027+0000", 2024-09-13T11:22:59.174 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000127568 2024-09-13T11:22:59.174 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.175 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.175 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:22:59.175 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316048+0000", 2024-09-13T11:22:59.175 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.0795999999999999e-05 2024-09-13T11:22:59.175 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.175 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.175 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:22:59.175 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316061+0000", 2024-09-13T11:22:59.175 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.3283e-05 2024-09-13T11:22:59.175 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.175 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.175 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:22:59.175 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.317107+0000", 2024-09-13T11:22:59.175 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0010452949999999999 2024-09-13T11:22:59.176 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.176 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.176 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-09-13T11:22:59.176 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.317114+0000", 2024-09-13T11:22:59.176 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.4569999999999999e-06 2024-09-13T11:22:59.176 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.176 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.176 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:22:59.176 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.190017+0000", 2024-09-13T11:22:59.176 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.87290258499999995 2024-09-13T11:22:59.176 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.176 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.176 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:22:59.176 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.190138+0000", 2024-09-13T11:22:59.177 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000121418 2024-09-13T11:22:59.177 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.177 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.177 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1", 2024-09-13T11:22:59.177 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.190216+0000", 2024-09-13T11:22:59.177 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.7410999999999995e-05 2024-09-13T11:22:59.177 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.177 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.177 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:22:59.177 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.191755+0000", 2024-09-13T11:22:59.177 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0015395420000000001 2024-09-13T11:22:59.177 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.177 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.178 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:22:59.178 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.191843+0000", 2024-09-13T11:22:59.178 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.8022999999999994e-05 2024-09-13T11:22:59.178 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.178 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.178 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:22:59.178 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.191876+0000", 2024-09-13T11:22:59.178 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.2619000000000001e-05 2024-09-13T11:22:59.178 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.178 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.178 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:22:59.178 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.191905+0000", 2024-09-13T11:22:59.179 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.9635999999999999e-05 2024-09-13T11:22:59.179 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:22:59.179 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:22:59.179 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:22:59.179 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.179 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.179 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:187 3.b 3:da03c8dd:::benchmark_data_smithi177_31116_object186:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T11:22:59.179 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:48.316083+0000", 2024-09-13T11:22:59.179 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 10.757073824000001, 2024-09-13T11:22:59.179 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.87835499500000003, 2024-09-13T11:22:59.179 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:22:59.179 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:22:59.179 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:59.180 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:22:59.180 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:22:59.180 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:22:59.180 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 187 2024-09-13T11:22:59.180 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.180 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:22:59.180 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.180 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:22:59.180 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316083+0000", 2024-09-13T11:22:59.180 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:22:59.180 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.180 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.180 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:22:59.180 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316083+0000", 2024-09-13T11:22:59.181 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:22:59.181 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.181 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.181 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:22:59.181 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316086+0000", 2024-09-13T11:22:59.181 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.5560000000000002e-06 2024-09-13T11:22:59.181 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.181 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.181 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:22:59.181 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316199+0000", 2024-09-13T11:22:59.181 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000113522 2024-09-13T11:22:59.181 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.181 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.181 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:22:59.182 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316215+0000", 2024-09-13T11:22:59.182 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.5265999999999999e-05 2024-09-13T11:22:59.182 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.182 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.182 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:22:59.182 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316226+0000", 2024-09-13T11:22:59.182 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0967e-05 2024-09-13T11:22:59.182 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.182 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.182 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:22:59.182 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.317198+0000", 2024-09-13T11:22:59.182 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00097194100000000004 2024-09-13T11:22:59.182 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.182 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.183 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-09-13T11:22:59.183 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.317213+0000", 2024-09-13T11:22:59.183 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.5472999999999999e-05 2024-09-13T11:22:59.183 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.183 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.183 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:22:59.183 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.191077+0000", 2024-09-13T11:22:59.183 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.87386435200000001 2024-09-13T11:22:59.183 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.183 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.183 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:22:59.183 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.191181+0000", 2024-09-13T11:22:59.183 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000103618 2024-09-13T11:22:59.183 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.183 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.184 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1", 2024-09-13T11:22:59.184 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.191305+0000", 2024-09-13T11:22:59.184 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00012358599999999999 2024-09-13T11:22:59.184 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.184 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.184 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:22:59.184 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.194279+0000", 2024-09-13T11:22:59.184 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0029746970000000001 2024-09-13T11:22:59.184 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.184 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.184 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:22:59.184 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.194336+0000", 2024-09-13T11:22:59.184 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.7244000000000003e-05 2024-09-13T11:22:59.185 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.185 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.185 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:22:59.185 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.194397+0000", 2024-09-13T11:22:59.185 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.0278000000000001e-05 2024-09-13T11:22:59.185 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.185 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.185 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:22:59.185 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.194438+0000", 2024-09-13T11:22:59.185 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.1495000000000002e-05 2024-09-13T11:22:59.185 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:22:59.185 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:22:59.185 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:22:59.185 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.186 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.186 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:200 3.b 3:d90039f1:::benchmark_data_smithi177_31116_object199:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T11:22:59.186 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:48.316280+0000", 2024-09-13T11:22:59.186 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 10.756877359000001, 2024-09-13T11:22:59.186 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.87879273000000002, 2024-09-13T11:22:59.186 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:22:59.186 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:22:59.186 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:59.186 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:22:59.186 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:22:59.186 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:22:59.186 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 200 2024-09-13T11:22:59.186 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.186 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:22:59.187 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.187 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:22:59.187 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316280+0000", 2024-09-13T11:22:59.187 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:22:59.187 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.187 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.187 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:22:59.187 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316280+0000", 2024-09-13T11:22:59.187 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:22:59.187 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.187 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.187 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:22:59.187 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316282+0000", 2024-09-13T11:22:59.187 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.497e-06 2024-09-13T11:22:59.188 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.188 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.188 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:22:59.188 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316411+0000", 2024-09-13T11:22:59.188 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00012924799999999999 2024-09-13T11:22:59.188 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.188 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.188 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:22:59.188 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316426+0000", 2024-09-13T11:22:59.188 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.4183e-05 2024-09-13T11:22:59.188 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.188 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.188 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:22:59.189 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316435+0000", 2024-09-13T11:22:59.189 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.0990000000000005e-06 2024-09-13T11:22:59.189 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.189 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.189 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:22:59.189 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.317250+0000", 2024-09-13T11:22:59.189 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00081501100000000003 2024-09-13T11:22:59.189 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.189 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.189 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-09-13T11:22:59.189 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.317257+0000", 2024-09-13T11:22:59.189 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.2169999999999997e-06 2024-09-13T11:22:59.189 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.190 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.190 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:22:59.190 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.191936+0000", 2024-09-13T11:22:59.190 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.87467922099999995 2024-09-13T11:22:59.190 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.190 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.190 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:22:59.190 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.192056+0000", 2024-09-13T11:22:59.190 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000119638 2024-09-13T11:22:59.190 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.190 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.190 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1", 2024-09-13T11:22:59.190 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.192184+0000", 2024-09-13T11:22:59.190 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00012813399999999999 2024-09-13T11:22:59.191 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.191 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.191 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:22:59.191 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.194324+0000", 2024-09-13T11:22:59.191 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0021399420000000001 2024-09-13T11:22:59.191 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.191 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.191 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:22:59.191 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.194802+0000", 2024-09-13T11:22:59.191 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00047798799999999999 2024-09-13T11:22:59.191 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.191 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.191 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:22:59.191 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.194951+0000", 2024-09-13T11:22:59.192 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0001494 2024-09-13T11:22:59.192 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.192 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.192 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:22:59.192 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.195072+0000", 2024-09-13T11:22:59.192 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00012115199999999999 2024-09-13T11:22:59.192 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:22:59.192 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:22:59.192 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:22:59.192 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.192 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.192 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:16548 3.b 3:d719617f:::benchmark_data_smithi177_31116_object16547:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-13T11:22:59.192 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:58.771284+0000", 2024-09-13T11:22:59.193 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 0.30187282700000001, 2024-09-13T11:22:59.193 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.19129597600000001, 2024-09-13T11:22:59.193 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:22:59.193 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:22:59.193 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:59.193 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:22:59.193 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:22:59.193 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:22:59.193 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 16548 2024-09-13T11:22:59.193 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.193 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:22:59.193 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.193 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:22:59.193 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:58.771284+0000", 2024-09-13T11:22:59.194 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:22:59.194 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.194 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.194 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:22:59.194 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:58.771284+0000", 2024-09-13T11:22:59.194 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:22:59.194 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.194 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.194 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:22:59.194 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:58.771289+0000", 2024-09-13T11:22:59.194 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.5499999999999996e-06 2024-09-13T11:22:59.194 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.194 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.195 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:22:59.195 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:58.771345+0000", 2024-09-13T11:22:59.195 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.5890999999999997e-05 2024-09-13T11:22:59.195 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.195 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.195 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:22:59.195 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:58.771378+0000", 2024-09-13T11:22:59.195 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.3145999999999997e-05 2024-09-13T11:22:59.195 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.195 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.195 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:22:59.195 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:58.771402+0000", 2024-09-13T11:22:59.196 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.3839e-05 2024-09-13T11:22:59.196 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.196 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.196 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:22:59.196 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:58.771527+0000", 2024-09-13T11:22:59.196 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000124876 2024-09-13T11:22:59.196 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.196 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.196 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:22:59.196 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:58.771665+0000", 2024-09-13T11:22:59.196 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00013799400000000001 2024-09-13T11:22:59.196 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.196 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.197 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1", 2024-09-13T11:22:59.197 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:58.771775+0000", 2024-09-13T11:22:59.197 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000110474 2024-09-13T11:22:59.197 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.197 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.197 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:22:59.197 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:58.773238+0000", 2024-09-13T11:22:59.197 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.001463186 2024-09-13T11:22:59.197 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.197 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.197 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:22:59.197 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:58.962439+0000", 2024-09-13T11:22:59.197 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.18920100400000001 2024-09-13T11:22:59.197 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.198 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.198 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:22:59.198 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:58.962517+0000", 2024-09-13T11:22:59.198 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.8207000000000005e-05 2024-09-13T11:22:59.198 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.198 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.198 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:22:59.198 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:58.962580+0000", 2024-09-13T11:22:59.198 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.2809000000000001e-05 2024-09-13T11:22:59.198 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:22:59.198 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:22:59.198 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:22:59.198 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.199 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.199 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:16556 3.b 3:d4e63cbf:::benchmark_data_smithi177_31116_object16555:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-13T11:22:59.199 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:58.774574+0000", 2024-09-13T11:22:59.199 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 0.29858285099999998, 2024-09-13T11:22:59.199 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.19064982999999999, 2024-09-13T11:22:59.199 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:22:59.199 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:22:59.199 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:59.199 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:22:59.199 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:22:59.199 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:22:59.199 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 16556 2024-09-13T11:22:59.199 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.199 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:22:59.200 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.200 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:22:59.200 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:58.774574+0000", 2024-09-13T11:22:59.200 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:22:59.200 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.200 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.200 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:22:59.200 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:58.774574+0000", 2024-09-13T11:22:59.200 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:22:59.200 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.200 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.200 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:22:59.200 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:58.774576+0000", 2024-09-13T11:22:59.200 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.9429999999999999e-06 2024-09-13T11:22:59.201 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.201 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.201 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:22:59.201 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:58.774613+0000", 2024-09-13T11:22:59.201 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.6409000000000003e-05 2024-09-13T11:22:59.201 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.201 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.201 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:22:59.201 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:58.774628+0000", 2024-09-13T11:22:59.201 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.5098e-05 2024-09-13T11:22:59.201 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.201 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.201 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:22:59.201 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:58.774638+0000", 2024-09-13T11:22:59.202 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0040999999999999e-05 2024-09-13T11:22:59.202 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.202 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.202 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:22:59.202 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:58.774710+0000", 2024-09-13T11:22:59.202 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.2323999999999994e-05 2024-09-13T11:22:59.202 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.202 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.202 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:22:59.202 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:58.774856+0000", 2024-09-13T11:22:59.202 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00014571399999999999 2024-09-13T11:22:59.202 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.202 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.202 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1", 2024-09-13T11:22:59.203 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:58.775018+0000", 2024-09-13T11:22:59.203 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000161774 2024-09-13T11:22:59.203 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.203 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.203 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:22:59.203 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:58.776116+0000", 2024-09-13T11:22:59.203 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0010987130000000001 2024-09-13T11:22:59.203 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.203 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.203 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:22:59.203 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:58.965117+0000", 2024-09-13T11:22:59.203 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.189001217 2024-09-13T11:22:59.203 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.204 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.204 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:22:59.204 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:58.965174+0000", 2024-09-13T11:22:59.204 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.6891999999999997e-05 2024-09-13T11:22:59.204 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.204 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.204 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:22:59.204 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:58.965224+0000", 2024-09-13T11:22:59.204 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.9704999999999999e-05 2024-09-13T11:22:59.204 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:22:59.204 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:22:59.204 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:22:59.204 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.205 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.205 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:16586 3.b 3:d47b54db:::benchmark_data_smithi177_31116_object16585:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-13T11:22:59.205 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:58.803730+0000", 2024-09-13T11:22:59.205 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 0.26942683299999998, 2024-09-13T11:22:59.205 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.16205857500000001, 2024-09-13T11:22:59.205 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:22:59.205 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:22:59.205 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:59.205 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:22:59.205 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:22:59.205 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:22:59.205 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 16586 2024-09-13T11:22:59.205 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.206 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:22:59.206 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.206 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:22:59.206 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:58.803730+0000", 2024-09-13T11:22:59.206 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:22:59.206 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.206 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.206 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:22:59.206 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:58.803730+0000", 2024-09-13T11:22:59.206 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:22:59.206 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.206 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.206 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:22:59.206 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:58.803731+0000", 2024-09-13T11:22:59.207 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.2079999999999999e-06 2024-09-13T11:22:59.207 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.207 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.207 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:22:59.207 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:58.803797+0000", 2024-09-13T11:22:59.207 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.5742999999999997e-05 2024-09-13T11:22:59.207 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.207 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.207 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:22:59.207 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:58.803806+0000", 2024-09-13T11:22:59.207 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.8769999999999999e-06 2024-09-13T11:22:59.207 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.207 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.207 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:22:59.208 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:58.803811+0000", 2024-09-13T11:22:59.208 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.101e-06 2024-09-13T11:22:59.208 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.208 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.208 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:22:59.208 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:58.803852+0000", 2024-09-13T11:22:59.208 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.0357e-05 2024-09-13T11:22:59.208 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.208 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.208 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:22:59.208 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:58.803930+0000", 2024-09-13T11:22:59.208 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.8763999999999993e-05 2024-09-13T11:22:59.208 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.208 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.209 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1", 2024-09-13T11:22:59.209 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:58.804010+0000", 2024-09-13T11:22:59.209 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.0068999999999995e-05 2024-09-13T11:22:59.209 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.209 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.209 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:22:59.209 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:58.805528+0000", 2024-09-13T11:22:59.209 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00151779 2024-09-13T11:22:59.209 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.209 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.209 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:22:59.209 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:58.965720+0000", 2024-09-13T11:22:59.209 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.16019212499999999 2024-09-13T11:22:59.209 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.210 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.210 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:22:59.210 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:58.965752+0000", 2024-09-13T11:22:59.210 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.1801999999999998e-05 2024-09-13T11:22:59.210 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:22:59.210 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:22:59.210 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:22:59.210 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:58.965789+0000", 2024-09-13T11:22:59.210 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.6739000000000002e-05 2024-09-13T11:22:59.210 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:22:59.210 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:22:59.210 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:22:59.210 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:22:59.210 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:22:59.211 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-09-13T11:22:59.212 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-13T11:22:59.213 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-09-13T11:22:59.207+0000 7f97befc9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26315) UID: 0 2024-09-13T11:22:59.315 INFO:tasks.ceph.osd.5.smithi092.stderr:2024-09-13T11:22:59.310+0000 7f6f0b705640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26346) UID: 0 2024-09-13T11:22:59.368 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-13T11:22:59.369 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [], 2024-09-13T11:22:59.369 INFO:teuthology.orchestra.run.smithi162.stdout: "num_ops": 0 2024-09-13T11:22:59.369 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-13T11:22:59.386 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-13T11:22:59.409 INFO:teuthology.orchestra.run.smithi001.stderr:reweighted osd.13 to 1 (10000) 2024-09-13T11:22:59.409 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-13T11:22:59.404+0000 7f6a5f1a6640 -1 osd.0 52 handle_osd_map injecting map crc failure 2024-09-13T11:22:59.411 INFO:tasks.ceph.osd.10.smithi162.stderr:2024-09-13T11:22:59.407+0000 7fb593421640 -1 osd.10 52 handle_osd_map injecting map crc failure 2024-09-13T11:22:59.413 INFO:tasks.ceph.osd.3.smithi177.stderr:2024-09-13T11:22:59.411+0000 7f10a45d5640 -1 osd.3 52 handle_osd_map injecting map crc failure 2024-09-13T11:22:59.414 INFO:tasks.ceph.osd.10.smithi162.stderr:2024-09-13T11:22:59.411+0000 7fb59c9f0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26317) UID: 0 2024-09-13T11:22:59.437 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 14 1 2024-09-13T11:22:59.514 INFO:tasks.ceph.osd.2.smithi162.stderr:2024-09-13T11:22:59.511+0000 7fe92b35a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26313) UID: 0 2024-09-13T11:22:59.527 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-13T11:22:59.527 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [], 2024-09-13T11:22:59.527 INFO:teuthology.orchestra.run.smithi162.stdout: "complaint_time": 30, 2024-09-13T11:22:59.527 INFO:teuthology.orchestra.run.smithi162.stdout: "num_blocked_ops": 0 2024-09-13T11:22:59.527 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-13T11:22:59.546 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-13T11:22:59.625 INFO:tasks.ceph.osd.9.smithi092.stderr:2024-09-13T11:22:59.622+0000 7ff59bdf5640 -1 osd.9 52 handle_osd_map injecting map crc failure 2024-09-13T11:22:59.633 INFO:tasks.ceph.osd.2.smithi162.stderr:2024-09-13T11:22:59.631+0000 7fe92b35a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26313) UID: 0 2024-09-13T11:22:59.716 INFO:tasks.ceph.osd.3.smithi177.stderr:2024-09-13T11:22:59.715+0000 7f10adba4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26574) UID: 0 2024-09-13T11:22:59.729 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-13T11:22:59.729 INFO:teuthology.orchestra.run.smithi162.stdout: "size": 20, 2024-09-13T11:22:59.729 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 600, 2024-09-13T11:22:59.730 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [ 2024-09-13T11:22:59.730 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:59.730 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.4665.0:119 3.9 e41/30 3:98957a18:::benchmark_data_smithi177_31116_object118:head v 41'9, mlcod=33'6)", 2024-09-13T11:22:59.730 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:22:47.342628+0000", 2024-09-13T11:22:59.730 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 12.385341201999999, 2024-09-13T11:22:59.730 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.057932441000000001, 2024-09-13T11:22:59.730 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:22:59.730 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:22:59.730 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:59.730 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:22:59.730 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:59.730 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:22:59.730 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:47.342628+0000", 2024-09-13T11:22:59.731 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:22:59.731 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:59.731 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:59.731 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:22:59.731 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:47.342628+0000", 2024-09-13T11:22:59.731 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:22:59.731 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:59.731 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:59.731 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:22:59.731 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:47.342632+0000", 2024-09-13T11:22:59.731 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.5700000000000001e-06 2024-09-13T11:22:59.731 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:59.731 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:59.731 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:22:59.732 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:47.342705+0000", 2024-09-13T11:22:59.732 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.3220999999999996e-05 2024-09-13T11:22:59.732 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:59.732 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:59.732 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:22:59.732 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:47.398889+0000", 2024-09-13T11:22:59.732 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.056183956 2024-09-13T11:22:59.732 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:59.732 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:59.732 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:22:59.732 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:47.398904+0000", 2024-09-13T11:22:59.732 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4902999999999999e-05 2024-09-13T11:22:59.732 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:59.733 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:59.733 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:22:59.733 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:47.399009+0000", 2024-09-13T11:22:59.733 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000105037 2024-09-13T11:22:59.733 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:59.733 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:59.733 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:22:59.733 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:47.399047+0000", 2024-09-13T11:22:59.733 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.7474000000000001e-05 2024-09-13T11:22:59.733 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:59.733 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:59.733 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:22:59.733 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:47.400461+0000", 2024-09-13T11:22:59.733 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0014144400000000001 2024-09-13T11:22:59.734 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:59.734 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:59.734 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:22:59.734 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:47.400561+0000", 2024-09-13T11:22:59.734 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.9840000000000006e-05 2024-09-13T11:22:59.734 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:22:59.734 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:22:59.734 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:22:59.734 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:59.734 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:59.734 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.4665.0:171 3.9 e41/30 3:9e1c7eeb:::benchmark_data_smithi177_31116_object170:head v 41'13, mlcod=41'9)", 2024-09-13T11:22:59.734 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:22:47.418602+0000", 2024-09-13T11:22:59.734 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 12.309367436, 2024-09-13T11:22:59.734 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.67235367300000004, 2024-09-13T11:22:59.735 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:22:59.735 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:22:59.735 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:59.735 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:22:59.735 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:59.735 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:22:59.735 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:47.418602+0000", 2024-09-13T11:22:59.735 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:22:59.735 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:59.735 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:59.735 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:22:59.735 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:47.418602+0000", 2024-09-13T11:22:59.735 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:22:59.735 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:59.736 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:59.736 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:22:59.736 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:47.418605+0000", 2024-09-13T11:22:59.736 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.2200000000000001e-06 2024-09-13T11:22:59.736 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:59.736 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:59.736 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:22:59.736 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:47.418668+0000", 2024-09-13T11:22:59.736 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.2921000000000003e-05 2024-09-13T11:22:59.736 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:59.736 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:59.736 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:22:59.736 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:48.089380+0000", 2024-09-13T11:22:59.737 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.67071191100000005 2024-09-13T11:22:59.737 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:59.737 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:59.737 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:22:59.737 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:48.089423+0000", 2024-09-13T11:22:59.737 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.2916e-05 2024-09-13T11:22:59.737 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:59.738 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:59.738 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:22:59.738 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:48.089533+0000", 2024-09-13T11:22:59.738 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000109996 2024-09-13T11:22:59.738 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:59.738 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:59.738 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:22:59.738 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:48.089570+0000", 2024-09-13T11:22:59.738 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.7314999999999998e-05 2024-09-13T11:22:59.738 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:59.739 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:59.739 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:22:59.739 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:48.090852+0000", 2024-09-13T11:22:59.739 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001281641 2024-09-13T11:22:59.739 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:59.739 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:59.739 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:22:59.739 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:48.090956+0000", 2024-09-13T11:22:59.739 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00010375299999999999 2024-09-13T11:22:59.739 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:22:59.739 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:22:59.740 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:22:59.740 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:59.740 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:59.740 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.4665.0:183 3.9 e41/30 3:94db3450:::benchmark_data_smithi177_31116_object182:head v 41'14, mlcod=41'11)", 2024-09-13T11:22:59.740 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:22:48.089206+0000", 2024-09-13T11:22:59.740 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 11.638763981, 2024-09-13T11:22:59.740 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.042973540999999997, 2024-09-13T11:22:59.740 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:22:59.740 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:22:59.740 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:59.741 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:22:59.741 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:59.741 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:22:59.741 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:48.089206+0000", 2024-09-13T11:22:59.741 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:22:59.741 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:59.742 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:59.742 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:22:59.742 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:48.089206+0000", 2024-09-13T11:22:59.742 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:22:59.742 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:59.742 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:59.742 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:22:59.742 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:48.089209+0000", 2024-09-13T11:22:59.742 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.726e-06 2024-09-13T11:22:59.742 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:59.742 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:59.743 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:22:59.743 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:48.089341+0000", 2024-09-13T11:22:59.743 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00013116000000000001 2024-09-13T11:22:59.743 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:59.743 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:59.743 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:22:59.743 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:48.130485+0000", 2024-09-13T11:22:59.743 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.041144095999999998 2024-09-13T11:22:59.743 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:59.743 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:59.743 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:22:59.743 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:48.130498+0000", 2024-09-13T11:22:59.743 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3436e-05 2024-09-13T11:22:59.743 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:59.744 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:59.744 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:22:59.744 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:48.130647+0000", 2024-09-13T11:22:59.744 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00014904100000000001 2024-09-13T11:22:59.744 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:59.744 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:59.744 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:22:59.744 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:48.130681+0000", 2024-09-13T11:22:59.744 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.3763000000000002e-05 2024-09-13T11:22:59.744 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:59.744 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:59.745 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:22:59.745 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:48.132077+0000", 2024-09-13T11:22:59.745 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001396266 2024-09-13T11:22:59.745 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:59.745 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:59.745 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:22:59.745 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:48.132179+0000", 2024-09-13T11:22:59.745 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00010205299999999999 2024-09-13T11:22:59.745 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:22:59.745 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:22:59.745 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:22:59.746 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:59.746 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:59.746 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.4665.0:432 3.9 e42/30 3:95b50337:::benchmark_data_smithi177_31116_object431:head v 42'38, mlcod=42'35)", 2024-09-13T11:22:59.746 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:22:49.280914+0000", 2024-09-13T11:22:59.746 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 10.447055592, 2024-09-13T11:22:59.746 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.035195901000000002, 2024-09-13T11:22:59.746 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:22:59.746 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:22:59.746 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:59.746 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:22:59.746 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:59.746 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:22:59.746 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:49.280914+0000", 2024-09-13T11:22:59.746 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:22:59.747 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:59.747 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:59.747 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:22:59.747 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:49.280914+0000", 2024-09-13T11:22:59.747 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:22:59.747 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:59.747 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:59.747 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:22:59.747 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:49.280917+0000", 2024-09-13T11:22:59.747 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.4300000000000002e-06 2024-09-13T11:22:59.747 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:59.747 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:59.748 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:22:59.748 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:49.280993+0000", 2024-09-13T11:22:59.748 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.5154000000000006e-05 2024-09-13T11:22:59.748 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:59.748 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:59.748 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:22:59.748 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:49.314658+0000", 2024-09-13T11:22:59.748 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.033665050000000002 2024-09-13T11:22:59.748 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:59.748 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:59.748 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:22:59.748 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:49.314671+0000", 2024-09-13T11:22:59.748 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3506000000000001e-05 2024-09-13T11:22:59.749 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:59.749 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:59.749 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:22:59.749 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:49.314786+0000", 2024-09-13T11:22:59.749 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000115053 2024-09-13T11:22:59.749 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:59.749 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:59.749 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:22:59.749 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:49.314821+0000", 2024-09-13T11:22:59.749 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.4978999999999998e-05 2024-09-13T11:22:59.749 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:59.749 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:59.749 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:22:59.749 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:49.316033+0000", 2024-09-13T11:22:59.750 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001211571 2024-09-13T11:22:59.750 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:59.750 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:59.750 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:22:59.750 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:49.316110+0000", 2024-09-13T11:22:59.750 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.7157999999999995e-05 2024-09-13T11:22:59.750 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:22:59.750 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:22:59.750 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:22:59.750 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:59.750 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:59.750 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.4665.0:765 3.9 e43/30 3:9e2afeb7:::benchmark_data_smithi177_31116_object764:head v 43'63, mlcod=43'59)", 2024-09-13T11:22:59.750 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:22:49.424535+0000", 2024-09-13T11:22:59.750 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 10.303434963999999, 2024-09-13T11:22:59.751 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.024580944, 2024-09-13T11:22:59.751 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:22:59.751 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:22:59.751 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:59.751 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:22:59.751 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:59.751 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:22:59.751 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:49.424535+0000", 2024-09-13T11:22:59.751 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:22:59.751 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:59.752 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:59.752 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:22:59.752 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:49.424535+0000", 2024-09-13T11:22:59.752 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:22:59.752 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:59.752 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:59.752 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:22:59.752 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:49.424536+0000", 2024-09-13T11:22:59.752 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.8050000000000001e-06 2024-09-13T11:22:59.752 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:59.752 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:59.752 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:22:59.752 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:49.424610+0000", 2024-09-13T11:22:59.752 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.3557000000000001e-05 2024-09-13T11:22:59.753 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:59.753 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:59.753 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:22:59.753 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:49.447721+0000", 2024-09-13T11:22:59.753 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.023110815 2024-09-13T11:22:59.753 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:59.753 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:59.753 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:22:59.753 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:49.447730+0000", 2024-09-13T11:22:59.753 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.2690000000000005e-06 2024-09-13T11:22:59.756 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:59.756 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:59.756 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:22:59.756 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:49.447834+0000", 2024-09-13T11:22:59.756 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000104193 2024-09-13T11:22:59.756 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:59.756 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:59.756 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:22:59.756 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:49.447868+0000", 2024-09-13T11:22:59.756 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.3426999999999997e-05 2024-09-13T11:22:59.756 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:59.756 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:59.757 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:22:59.757 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:49.449047+0000", 2024-09-13T11:22:59.757 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0011788840000000001 2024-09-13T11:22:59.757 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:59.757 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:59.757 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:22:59.757 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:49.449116+0000", 2024-09-13T11:22:59.757 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.8993999999999997e-05 2024-09-13T11:22:59.757 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:22:59.757 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:22:59.757 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:22:59.757 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:59.757 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:59.757 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.4665.0:824 3.9 e43/30 3:97455708:::benchmark_data_smithi177_31116_object823:head v 43'65, mlcod=43'62)", 2024-09-13T11:22:59.758 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:22:49.453080+0000", 2024-09-13T11:22:59.758 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 10.27488921, 2024-09-13T11:22:59.758 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.032738985999999998, 2024-09-13T11:22:59.758 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:22:59.758 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:22:59.758 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:59.758 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:22:59.758 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:59.758 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:22:59.758 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:49.453080+0000", 2024-09-13T11:22:59.758 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:22:59.758 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:59.758 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:59.759 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:22:59.759 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:49.453080+0000", 2024-09-13T11:22:59.759 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:22:59.759 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:59.759 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:59.759 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:22:59.759 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:49.453082+0000", 2024-09-13T11:22:59.759 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.0420000000000001e-06 2024-09-13T11:22:59.759 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:59.759 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:59.759 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:22:59.759 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:49.453129+0000", 2024-09-13T11:22:59.759 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.6329000000000003e-05 2024-09-13T11:22:59.759 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:59.760 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:59.760 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:22:59.760 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:49.484468+0000", 2024-09-13T11:22:59.760 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.031339321000000003 2024-09-13T11:22:59.760 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:59.760 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:59.760 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:22:59.760 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:49.484480+0000", 2024-09-13T11:22:59.760 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2069000000000001e-05 2024-09-13T11:22:59.760 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:59.760 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:59.760 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:22:59.760 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:49.484579+0000", 2024-09-13T11:22:59.760 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.8533e-05 2024-09-13T11:22:59.761 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:59.761 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:59.761 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:22:59.761 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:49.484615+0000", 2024-09-13T11:22:59.761 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.5932000000000001e-05 2024-09-13T11:22:59.761 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:59.761 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:59.761 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:22:59.761 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:49.485741+0000", 2024-09-13T11:22:59.761 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0011268490000000001 2024-09-13T11:22:59.761 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:59.761 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:59.761 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:22:59.761 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:49.485819+0000", 2024-09-13T11:22:59.761 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.7911000000000007e-05 2024-09-13T11:22:59.762 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:22:59.762 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:22:59.762 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:22:59.762 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:59.762 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:59.762 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.4665.0:871 3.9 e43/30 3:96b617d2:::benchmark_data_smithi177_31116_object870:head v 43'68, mlcod=43'65)", 2024-09-13T11:22:59.762 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:22:49.499029+0000", 2024-09-13T11:22:59.762 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 10.228940691, 2024-09-13T11:22:59.762 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.022790472999999999, 2024-09-13T11:22:59.762 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:22:59.762 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:22:59.762 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:59.762 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:22:59.763 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:59.763 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:22:59.763 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:49.499029+0000", 2024-09-13T11:22:59.763 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:22:59.763 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:59.763 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:59.763 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:22:59.763 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:49.499029+0000", 2024-09-13T11:22:59.763 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:22:59.763 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:59.763 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:59.763 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:22:59.763 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:49.499031+0000", 2024-09-13T11:22:59.764 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.874e-06 2024-09-13T11:22:59.764 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:59.764 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:59.764 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:22:59.764 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:49.499080+0000", 2024-09-13T11:22:59.764 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.9376000000000003e-05 2024-09-13T11:22:59.764 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:59.764 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:59.764 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:22:59.764 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:49.520094+0000", 2024-09-13T11:22:59.764 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.021013344 2024-09-13T11:22:59.765 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:59.765 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:59.765 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:22:59.765 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:49.520108+0000", 2024-09-13T11:22:59.765 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4212999999999999e-05 2024-09-13T11:22:59.765 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:59.765 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:59.765 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:22:59.766 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:49.520206+0000", 2024-09-13T11:22:59.766 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.8462999999999996e-05 2024-09-13T11:22:59.766 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:59.766 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:59.766 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:22:59.766 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:49.520250+0000", 2024-09-13T11:22:59.766 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.4095999999999999e-05 2024-09-13T11:22:59.766 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:59.766 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:59.766 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:22:59.766 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:49.521730+0000", 2024-09-13T11:22:59.766 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00148018 2024-09-13T11:22:59.766 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:59.766 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:59.767 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:22:59.767 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:49.521819+0000", 2024-09-13T11:22:59.767 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.8926999999999997e-05 2024-09-13T11:22:59.767 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:22:59.767 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:22:59.767 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:22:59.767 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:59.767 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:59.767 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.4665.0:915 3.9 e43/30 3:9b5a11fd:::benchmark_data_smithi177_31116_object914:head v 43'73, mlcod=43'70)", 2024-09-13T11:22:59.767 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:22:49.539211+0000", 2024-09-13T11:22:59.767 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 10.188758116000001, 2024-09-13T11:22:59.767 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.063010919999999998, 2024-09-13T11:22:59.767 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:22:59.767 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:22:59.768 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:59.768 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:22:59.768 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:59.768 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:22:59.768 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:49.539211+0000", 2024-09-13T11:22:59.768 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:22:59.768 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:59.768 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:59.768 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:22:59.768 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:49.539211+0000", 2024-09-13T11:22:59.768 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:22:59.768 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:59.768 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:59.768 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:22:59.769 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:49.539214+0000", 2024-09-13T11:22:59.769 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.4279999999999999e-06 2024-09-13T11:22:59.769 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:59.769 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:59.769 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:22:59.769 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:49.539257+0000", 2024-09-13T11:22:59.769 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.2648000000000002e-05 2024-09-13T11:22:59.769 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:59.769 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:59.769 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:22:59.769 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:49.600637+0000", 2024-09-13T11:22:59.769 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.061380067000000003 2024-09-13T11:22:59.769 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:59.770 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:59.770 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:22:59.770 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:49.600652+0000", 2024-09-13T11:22:59.770 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5727e-05 2024-09-13T11:22:59.770 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:59.770 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:59.770 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:22:59.770 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:49.600775+0000", 2024-09-13T11:22:59.770 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00012288199999999999 2024-09-13T11:22:59.770 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:59.770 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:59.770 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:22:59.770 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:49.600813+0000", 2024-09-13T11:22:59.770 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.8078999999999999e-05 2024-09-13T11:22:59.771 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:59.771 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:59.771 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:22:59.771 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:49.602102+0000", 2024-09-13T11:22:59.771 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001288443 2024-09-13T11:22:59.771 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:59.771 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:59.771 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:22:59.771 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:49.602222+0000", 2024-09-13T11:22:59.771 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000120646 2024-09-13T11:22:59.771 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:22:59.771 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:22:59.771 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:22:59.771 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:59.772 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:59.772 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.4665.0:1019 3.9 e43/30 3:9e876c33:::benchmark_data_smithi177_31116_object1018:head v 43'79, mlcod=43'75)", 2024-09-13T11:22:59.772 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:22:49.631331+0000", 2024-09-13T11:22:59.772 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 10.096638971000001, 2024-09-13T11:22:59.772 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.024334492999999999, 2024-09-13T11:22:59.772 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:22:59.772 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:22:59.772 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:59.772 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:22:59.772 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:59.772 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:22:59.772 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:49.631331+0000", 2024-09-13T11:22:59.772 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:22:59.772 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:59.773 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:59.773 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:22:59.773 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:49.631331+0000", 2024-09-13T11:22:59.773 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:22:59.773 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:59.773 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:59.773 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:22:59.773 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:49.631334+0000", 2024-09-13T11:22:59.773 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.7799999999999998e-06 2024-09-13T11:22:59.773 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:59.773 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:59.773 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:22:59.773 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:49.631396+0000", 2024-09-13T11:22:59.774 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.2040999999999998e-05 2024-09-13T11:22:59.774 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:59.774 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:59.774 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:22:59.774 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:49.653893+0000", 2024-09-13T11:22:59.774 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.022496972 2024-09-13T11:22:59.774 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:59.774 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:59.774 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:22:59.774 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:49.653907+0000", 2024-09-13T11:22:59.774 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3703000000000001e-05 2024-09-13T11:22:59.774 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:59.774 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:59.774 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:22:59.775 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:49.654013+0000", 2024-09-13T11:22:59.775 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00010565000000000001 2024-09-13T11:22:59.775 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:59.775 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:59.775 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:22:59.775 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:49.654047+0000", 2024-09-13T11:22:59.775 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.4348999999999999e-05 2024-09-13T11:22:59.775 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:59.775 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:59.775 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:22:59.775 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:49.655554+0000", 2024-09-13T11:22:59.775 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0015069020000000001 2024-09-13T11:22:59.775 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:59.775 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:59.775 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:22:59.776 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:49.655665+0000", 2024-09-13T11:22:59.776 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00011109599999999999 2024-09-13T11:22:59.776 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:22:59.776 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:22:59.776 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:22:59.776 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:59.776 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:59.776 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.4665.0:1130 3.9 e43/30 3:9b77df09:::benchmark_data_smithi177_31116_object1129:head v 43'85, mlcod=43'82)", 2024-09-13T11:22:59.776 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:22:49.718258+0000", 2024-09-13T11:22:59.776 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 10.009711589, 2024-09-13T11:22:59.776 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.024223326999999999, 2024-09-13T11:22:59.776 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:22:59.776 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:22:59.777 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:59.777 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:22:59.777 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:59.777 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:22:59.777 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:49.718258+0000", 2024-09-13T11:22:59.777 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:22:59.777 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:59.777 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:59.777 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:22:59.777 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:49.718258+0000", 2024-09-13T11:22:59.777 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:22:59.777 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:59.777 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:59.777 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:22:59.777 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:49.718262+0000", 2024-09-13T11:22:59.778 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.053e-06 2024-09-13T11:22:59.778 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:59.778 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:59.778 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:22:59.778 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:49.718341+0000", 2024-09-13T11:22:59.778 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.9274e-05 2024-09-13T11:22:59.778 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:59.778 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:59.778 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:22:59.778 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:49.740805+0000", 2024-09-13T11:22:59.778 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.022463796000000001 2024-09-13T11:22:59.778 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:59.778 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:59.779 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:22:59.779 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:49.740818+0000", 2024-09-13T11:22:59.779 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3234000000000001e-05 2024-09-13T11:22:59.779 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:59.779 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:59.779 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:22:59.779 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:49.740924+0000", 2024-09-13T11:22:59.779 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000105696 2024-09-13T11:22:59.779 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:59.779 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:59.779 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:22:59.779 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:49.740964+0000", 2024-09-13T11:22:59.779 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.9678999999999997e-05 2024-09-13T11:22:59.779 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:59.780 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:59.780 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:22:59.780 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:49.742409+0000", 2024-09-13T11:22:59.780 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001445034 2024-09-13T11:22:59.780 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:59.780 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:59.780 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:22:59.780 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:49.742481+0000", 2024-09-13T11:22:59.780 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.2560999999999999e-05 2024-09-13T11:22:59.780 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:22:59.780 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:22:59.780 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:22:59.780 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:59.781 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:59.781 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.4665.0:16333 3.9 e52/30 3:980c1bbf:::benchmark_data_smithi177_31116_object16332:head v 52'1001, mlcod=52'998)", 2024-09-13T11:22:59.781 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:22:58.654899+0000", 2024-09-13T11:22:59.781 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 1.0730708280000001, 2024-09-13T11:22:59.781 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.026363133, 2024-09-13T11:22:59.781 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:22:59.781 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:22:59.781 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:59.781 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:22:59.781 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:59.781 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:22:59.781 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:58.654899+0000", 2024-09-13T11:22:59.781 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:22:59.781 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:59.782 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:59.782 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:22:59.782 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:58.654899+0000", 2024-09-13T11:22:59.782 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:22:59.782 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:59.782 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:59.782 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:22:59.782 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:58.654900+0000", 2024-09-13T11:22:59.782 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.6029999999999999e-06 2024-09-13T11:22:59.782 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:59.782 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:59.782 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:22:59.782 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:58.654970+0000", 2024-09-13T11:22:59.782 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.9536999999999995e-05 2024-09-13T11:22:59.783 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:59.783 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:59.783 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:22:59.783 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:58.678510+0000", 2024-09-13T11:22:59.783 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.023540271000000002 2024-09-13T11:22:59.783 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:59.783 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:59.783 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:22:59.783 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:58.678532+0000", 2024-09-13T11:22:59.783 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.1965999999999999e-05 2024-09-13T11:22:59.783 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:59.783 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:59.783 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:22:59.784 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:58.678669+0000", 2024-09-13T11:22:59.784 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000136499 2024-09-13T11:22:59.784 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:59.784 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:59.784 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:22:59.784 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:58.678704+0000", 2024-09-13T11:22:59.784 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.5746e-05 2024-09-13T11:22:59.784 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:59.784 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:59.784 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:22:59.784 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:58.681140+0000", 2024-09-13T11:22:59.784 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00243574 2024-09-13T11:22:59.784 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:59.784 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:59.785 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:22:59.785 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:58.681262+0000", 2024-09-13T11:22:59.785 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000121771 2024-09-13T11:22:59.785 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:22:59.785 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:22:59.785 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:22:59.785 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:59.785 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:59.785 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.4665.0:16404 3.9 e52/30 3:943ffae1:::benchmark_data_smithi177_31116_object16403:head v 52'1003, mlcod=52'1000)", 2024-09-13T11:22:59.785 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:22:58.690281+0000", 2024-09-13T11:22:59.785 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 1.037688519, 2024-09-13T11:22:59.785 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.024581548000000002, 2024-09-13T11:22:59.785 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:22:59.785 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:22:59.786 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:59.786 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:22:59.786 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:59.786 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:22:59.786 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:58.690281+0000", 2024-09-13T11:22:59.786 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:22:59.786 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:59.786 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:59.786 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:22:59.786 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:58.690281+0000", 2024-09-13T11:22:59.786 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:22:59.786 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:59.786 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:59.786 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:22:59.787 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:58.690284+0000", 2024-09-13T11:22:59.787 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.3909999999999998e-06 2024-09-13T11:22:59.787 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:59.787 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:59.787 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:22:59.787 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:58.690353+0000", 2024-09-13T11:22:59.787 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.8638000000000003e-05 2024-09-13T11:22:59.787 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:59.787 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:59.787 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:22:59.787 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:58.710785+0000", 2024-09-13T11:22:59.787 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.020431988000000002 2024-09-13T11:22:59.787 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:59.787 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:59.788 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:22:59.788 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:58.710803+0000", 2024-09-13T11:22:59.788 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.7611000000000001e-05 2024-09-13T11:22:59.788 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:59.788 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:59.788 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:22:59.788 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:58.710926+0000", 2024-09-13T11:22:59.788 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000123034 2024-09-13T11:22:59.788 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:59.788 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:59.788 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:22:59.788 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:58.710978+0000", 2024-09-13T11:22:59.788 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.2303999999999999e-05 2024-09-13T11:22:59.788 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:59.788 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:59.789 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:22:59.789 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:58.714770+0000", 2024-09-13T11:22:59.789 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0037920079999999999 2024-09-13T11:22:59.789 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:59.789 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:59.789 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:22:59.789 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:58.714863+0000", 2024-09-13T11:22:59.789 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.2573999999999998e-05 2024-09-13T11:22:59.789 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:22:59.789 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:22:59.789 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:22:59.789 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:59.789 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:59.789 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.4665.0:16574 3.9 e52/30 3:929985e1:::benchmark_data_smithi177_31116_object16573:head v 52'1018, mlcod=52'1014)", 2024-09-13T11:22:59.790 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:22:58.787960+0000", 2024-09-13T11:22:59.790 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 0.94001002700000003, 2024-09-13T11:22:59.790 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.021548823000000002, 2024-09-13T11:22:59.790 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:22:59.790 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:22:59.790 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:59.790 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:22:59.790 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:59.790 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:22:59.790 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:58.787960+0000", 2024-09-13T11:22:59.790 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:22:59.790 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:59.790 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:59.790 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:22:59.791 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:58.787960+0000", 2024-09-13T11:22:59.791 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:22:59.791 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:59.791 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:59.791 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:22:59.791 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:58.787963+0000", 2024-09-13T11:22:59.791 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.1300000000000001e-06 2024-09-13T11:22:59.791 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:59.791 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:59.791 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:22:59.791 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:58.788035+0000", 2024-09-13T11:22:59.791 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.2720999999999997e-05 2024-09-13T11:22:59.791 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:59.791 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:59.792 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:22:59.792 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:58.807330+0000", 2024-09-13T11:22:59.792 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.019294494999999998 2024-09-13T11:22:59.792 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:59.792 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:59.792 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:22:59.792 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:58.807343+0000", 2024-09-13T11:22:59.792 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3492999999999999e-05 2024-09-13T11:22:59.792 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:59.792 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:59.792 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:22:59.792 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:58.807487+0000", 2024-09-13T11:22:59.792 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00014401299999999999 2024-09-13T11:22:59.793 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:59.793 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:59.793 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:22:59.793 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:58.807549+0000", 2024-09-13T11:22:59.793 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.1490999999999998e-05 2024-09-13T11:22:59.793 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:59.793 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:59.793 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:22:59.793 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:58.809435+0000", 2024-09-13T11:22:59.793 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001886485 2024-09-13T11:22:59.793 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:59.793 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:59.793 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:22:59.793 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:58.809508+0000", 2024-09-13T11:22:59.794 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.2995000000000002e-05 2024-09-13T11:22:59.794 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:22:59.794 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:22:59.794 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:22:59.794 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:59.794 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:59.794 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.4665.0:16592 3.9 e52/30 3:9b63e372:::benchmark_data_smithi177_31116_object16591:head v 52'1020, mlcod=52'1017)", 2024-09-13T11:22:59.794 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:22:58.811049+0000", 2024-09-13T11:22:59.794 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 0.91692026100000001, 2024-09-13T11:22:59.794 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.024313276000000002, 2024-09-13T11:22:59.794 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:22:59.794 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:22:59.794 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:59.794 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:22:59.795 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:59.795 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:22:59.795 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:58.811049+0000", 2024-09-13T11:22:59.795 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:22:59.795 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:59.795 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:59.795 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:22:59.795 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:58.811049+0000", 2024-09-13T11:22:59.795 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:22:59.795 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:59.795 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:59.795 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:22:59.795 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:58.811053+0000", 2024-09-13T11:22:59.796 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.3469999999999999e-06 2024-09-13T11:22:59.796 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:59.796 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:59.796 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:22:59.796 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:58.811111+0000", 2024-09-13T11:22:59.796 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.8168000000000003e-05 2024-09-13T11:22:59.796 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:59.796 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:59.796 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:22:59.796 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:58.833141+0000", 2024-09-13T11:22:59.796 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.022030319 2024-09-13T11:22:59.796 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:59.796 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:59.796 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:22:59.797 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:58.833161+0000", 2024-09-13T11:22:59.797 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.0100000000000001e-05 2024-09-13T11:22:59.797 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:59.797 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:59.797 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:22:59.797 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:58.833315+0000", 2024-09-13T11:22:59.797 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000153552 2024-09-13T11:22:59.797 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:59.797 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:59.797 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:22:59.797 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:58.833374+0000", 2024-09-13T11:22:59.797 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.9125000000000001e-05 2024-09-13T11:22:59.797 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:59.797 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:59.798 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:22:59.798 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:58.835288+0000", 2024-09-13T11:22:59.798 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001914231 2024-09-13T11:22:59.798 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:59.798 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:59.798 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:22:59.798 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:58.835363+0000", 2024-09-13T11:22:59.798 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.4433999999999999e-05 2024-09-13T11:22:59.798 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:22:59.798 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:22:59.798 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:22:59.798 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:59.798 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:59.798 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.4665.0:16625 3.9 e52/30 3:9a13a72a:::benchmark_data_smithi177_31116_object16624:head v 52'1022, mlcod=52'1019)", 2024-09-13T11:22:59.799 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:22:58.851689+0000", 2024-09-13T11:22:59.799 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 0.876280964, 2024-09-13T11:22:59.799 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.038130886000000003, 2024-09-13T11:22:59.799 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:22:59.799 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:22:59.799 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:59.799 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:22:59.799 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:59.799 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:22:59.799 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:58.851689+0000", 2024-09-13T11:22:59.799 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:22:59.799 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:59.799 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:59.799 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:22:59.800 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:58.851689+0000", 2024-09-13T11:22:59.800 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:22:59.800 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:59.800 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:59.800 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:22:59.800 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:58.851700+0000", 2024-09-13T11:22:59.800 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1206000000000001e-05 2024-09-13T11:22:59.800 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:59.800 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:59.801 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:22:59.801 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:58.851762+0000", 2024-09-13T11:22:59.801 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.2142000000000002e-05 2024-09-13T11:22:59.801 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:59.801 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:59.801 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:22:59.801 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:58.887294+0000", 2024-09-13T11:22:59.801 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.035531528999999999 2024-09-13T11:22:59.801 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:59.801 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:59.801 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:22:59.801 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:58.887308+0000", 2024-09-13T11:22:59.801 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4932e-05 2024-09-13T11:22:59.802 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:59.802 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:59.802 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:22:59.802 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:58.887465+0000", 2024-09-13T11:22:59.802 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000156506 2024-09-13T11:22:59.802 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:59.802 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:59.802 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:22:59.802 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:58.887524+0000", 2024-09-13T11:22:59.802 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.8878000000000001e-05 2024-09-13T11:22:59.802 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:59.802 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:59.802 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:22:59.802 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:58.889752+0000", 2024-09-13T11:22:59.803 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0022285949999999999 2024-09-13T11:22:59.803 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:59.803 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:59.803 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:22:59.803 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:58.889820+0000", 2024-09-13T11:22:59.803 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.7098e-05 2024-09-13T11:22:59.803 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:22:59.803 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:22:59.803 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:22:59.803 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:59.803 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:59.803 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.4665.0:16651 3.9 e52/30 3:93c589f9:::benchmark_data_smithi177_31116_object16650:head v 52'1023, mlcod=52'1020)", 2024-09-13T11:22:59.803 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:22:58.887189+0000", 2024-09-13T11:22:59.803 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 0.84078097100000004, 2024-09-13T11:22:59.804 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.032800454, 2024-09-13T11:22:59.804 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:22:59.804 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:22:59.804 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:59.804 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:22:59.804 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:59.804 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:22:59.804 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:58.887189+0000", 2024-09-13T11:22:59.804 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:22:59.804 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:59.804 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:59.804 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:22:59.804 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:58.887189+0000", 2024-09-13T11:22:59.804 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:22:59.805 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:59.805 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:59.805 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:22:59.805 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:58.887192+0000", 2024-09-13T11:22:59.805 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.6789999999999998e-06 2024-09-13T11:22:59.805 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:59.805 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:59.805 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:22:59.805 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:58.887249+0000", 2024-09-13T11:22:59.805 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.6999000000000001e-05 2024-09-13T11:22:59.805 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:59.805 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:59.805 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:22:59.805 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:58.917802+0000", 2024-09-13T11:22:59.806 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.030552617000000001 2024-09-13T11:22:59.806 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:59.806 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:59.806 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:22:59.806 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:58.917816+0000", 2024-09-13T11:22:59.806 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4195e-05 2024-09-13T11:22:59.806 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:59.806 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:59.806 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:22:59.806 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:58.917954+0000", 2024-09-13T11:22:59.806 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00013805999999999999 2024-09-13T11:22:59.806 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:59.806 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:59.806 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:22:59.807 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:58.917988+0000", 2024-09-13T11:22:59.807 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.3875999999999999e-05 2024-09-13T11:22:59.807 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:59.807 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:59.807 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:22:59.807 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:58.919919+0000", 2024-09-13T11:22:59.807 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0019310659999999999 2024-09-13T11:22:59.807 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:59.807 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:59.807 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:22:59.807 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:58.919989+0000", 2024-09-13T11:22:59.807 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.9962000000000005e-05 2024-09-13T11:22:59.807 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:22:59.807 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:22:59.808 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:22:59.808 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:59.808 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:59.808 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.4665.0:16680 3.9 e52/30 3:9dfae4aa:::benchmark_data_smithi177_31116_object16679:head v 52'1026, mlcod=52'1023)", 2024-09-13T11:22:59.808 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:22:58.943541+0000", 2024-09-13T11:22:59.808 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 0.78442882999999997, 2024-09-13T11:22:59.808 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.023183431000000001, 2024-09-13T11:22:59.808 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:22:59.808 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:22:59.808 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:59.808 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:22:59.808 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:59.808 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:22:59.808 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:58.943541+0000", 2024-09-13T11:22:59.809 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:22:59.809 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:59.809 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:59.809 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:22:59.809 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:58.943541+0000", 2024-09-13T11:22:59.809 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:22:59.809 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:59.809 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:59.809 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:22:59.809 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:58.943544+0000", 2024-09-13T11:22:59.809 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.1429999999999998e-06 2024-09-13T11:22:59.809 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:59.809 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:59.809 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:22:59.810 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:58.943598+0000", 2024-09-13T11:22:59.810 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.4175999999999998e-05 2024-09-13T11:22:59.810 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:59.810 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:59.810 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:22:59.810 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:58.964801+0000", 2024-09-13T11:22:59.810 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.021202858000000002 2024-09-13T11:22:59.810 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:59.810 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:59.810 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:22:59.810 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:58.964815+0000", 2024-09-13T11:22:59.810 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3803e-05 2024-09-13T11:22:59.810 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:59.810 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:59.811 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:22:59.811 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:58.964941+0000", 2024-09-13T11:22:59.811 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00012598700000000001 2024-09-13T11:22:59.811 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:59.811 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:59.811 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:22:59.811 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:58.964981+0000", 2024-09-13T11:22:59.811 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.9839000000000003e-05 2024-09-13T11:22:59.811 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:59.811 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:59.811 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:22:59.811 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:58.966653+0000", 2024-09-13T11:22:59.811 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001672914 2024-09-13T11:22:59.811 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:59.812 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:59.812 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:22:59.812 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:58.966724+0000", 2024-09-13T11:22:59.812 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.0710999999999994e-05 2024-09-13T11:22:59.812 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:22:59.812 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:22:59.812 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:22:59.812 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:59.812 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:59.812 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.4665.0:16733 3.9 e52/30 3:9a5f4302:::benchmark_data_smithi177_31116_object16732:head v 52'1032, mlcod=52'1028)", 2024-09-13T11:22:59.812 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:22:58.990074+0000", 2024-09-13T11:22:59.812 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 0.73789598899999997, 2024-09-13T11:22:59.812 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.022030092000000001, 2024-09-13T11:22:59.812 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:22:59.813 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:22:59.813 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:59.813 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:22:59.813 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:59.813 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:22:59.813 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:58.990074+0000", 2024-09-13T11:22:59.813 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:22:59.813 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:59.813 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:59.813 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:22:59.813 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:58.990074+0000", 2024-09-13T11:22:59.813 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:22:59.813 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:59.813 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:59.814 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:22:59.814 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:58.990077+0000", 2024-09-13T11:22:59.814 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.0230000000000001e-06 2024-09-13T11:22:59.814 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:59.814 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:59.814 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:22:59.814 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:58.990143+0000", 2024-09-13T11:22:59.814 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.6107999999999998e-05 2024-09-13T11:22:59.814 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:59.815 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:59.815 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:22:59.815 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:59.010035+0000", 2024-09-13T11:22:59.815 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.019892401000000001 2024-09-13T11:22:59.815 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:59.815 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:59.815 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:22:59.815 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:59.010056+0000", 2024-09-13T11:22:59.815 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.0982999999999999e-05 2024-09-13T11:22:59.815 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:59.815 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:59.815 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:22:59.815 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:59.010203+0000", 2024-09-13T11:22:59.816 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000146689 2024-09-13T11:22:59.816 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:59.816 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:59.816 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:22:59.816 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:59.010241+0000", 2024-09-13T11:22:59.816 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.7978999999999997e-05 2024-09-13T11:22:59.816 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:59.816 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:59.816 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:22:59.816 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:59.012023+0000", 2024-09-13T11:22:59.816 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001782014 2024-09-13T11:22:59.816 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:59.816 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:59.816 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:22:59.817 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:59.012104+0000", 2024-09-13T11:22:59.817 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.0895000000000004e-05 2024-09-13T11:22:59.817 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:22:59.817 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:22:59.817 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:22:59.817 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:59.817 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:59.817 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.4665.0:16769 3.9 e52/30 3:919a172b:::benchmark_data_smithi177_31116_object16768:head v 52'1033, mlcod=52'1030)", 2024-09-13T11:22:59.817 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:22:59.009884+0000", 2024-09-13T11:22:59.817 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 0.71808590699999997, 2024-09-13T11:22:59.817 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.041620747, 2024-09-13T11:22:59.817 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:22:59.817 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:22:59.817 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:59.818 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:22:59.818 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:59.818 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:22:59.818 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:59.009884+0000", 2024-09-13T11:22:59.818 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:22:59.818 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:59.818 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:59.818 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:22:59.818 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:59.009884+0000", 2024-09-13T11:22:59.818 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:22:59.818 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:59.818 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:59.818 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:22:59.818 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:59.009888+0000", 2024-09-13T11:22:59.819 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.9199999999999997e-06 2024-09-13T11:22:59.819 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:59.819 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:59.819 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:22:59.819 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:59.009984+0000", 2024-09-13T11:22:59.819 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.6207999999999998e-05 2024-09-13T11:22:59.819 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:59.819 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:59.819 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:22:59.819 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:59.048236+0000", 2024-09-13T11:22:59.819 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.038252183000000002 2024-09-13T11:22:59.819 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:59.819 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:59.819 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:22:59.820 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:59.048251+0000", 2024-09-13T11:22:59.820 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.503e-05 2024-09-13T11:22:59.820 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:59.820 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:59.820 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:22:59.820 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:59.048384+0000", 2024-09-13T11:22:59.820 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000133289 2024-09-13T11:22:59.820 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:59.820 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:59.820 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:22:59.820 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:59.048424+0000", 2024-09-13T11:22:59.820 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.9332000000000002e-05 2024-09-13T11:22:59.820 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:59.820 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:59.821 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:22:59.821 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:59.051412+0000", 2024-09-13T11:22:59.821 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.002988044 2024-09-13T11:22:59.821 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:59.821 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:59.821 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:22:59.821 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:59.051504+0000", 2024-09-13T11:22:59.821 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.2740999999999999e-05 2024-09-13T11:22:59.821 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:22:59.821 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:22:59.821 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:22:59.821 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:59.821 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:59.822 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.4665.0:16884 3.9 e52/30 3:9d264015:::benchmark_data_smithi177_31116_object16883:head v 52'1040, mlcod=52'1037)", 2024-09-13T11:22:59.822 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:22:59.077723+0000", 2024-09-13T11:22:59.822 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 0.65024616400000002, 2024-09-13T11:22:59.822 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.02966214, 2024-09-13T11:22:59.822 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:22:59.822 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:22:59.822 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:22:59.822 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:22:59.822 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:59.822 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:22:59.822 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:59.077723+0000", 2024-09-13T11:22:59.822 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:22:59.822 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:59.822 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:59.823 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:22:59.823 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:59.077723+0000", 2024-09-13T11:22:59.823 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:22:59.823 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:59.823 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:59.823 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:22:59.823 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:59.077727+0000", 2024-09-13T11:22:59.823 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.8890000000000004e-06 2024-09-13T11:22:59.823 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:59.823 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:59.823 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:22:59.823 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:59.077800+0000", 2024-09-13T11:22:59.823 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.2554999999999999e-05 2024-09-13T11:22:59.823 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:59.823 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:59.824 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:22:59.824 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:59.105288+0000", 2024-09-13T11:22:59.824 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.027487810000000001 2024-09-13T11:22:59.824 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:59.824 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:59.824 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:22:59.824 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:59.105301+0000", 2024-09-13T11:22:59.824 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3797e-05 2024-09-13T11:22:59.824 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:59.824 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:59.824 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:22:59.824 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:59.105412+0000", 2024-09-13T11:22:59.824 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00011044000000000001 2024-09-13T11:22:59.824 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:59.825 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:59.825 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:22:59.825 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:59.105450+0000", 2024-09-13T11:22:59.825 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.7762000000000002e-05 2024-09-13T11:22:59.825 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:59.825 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:59.825 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:22:59.825 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:59.107276+0000", 2024-09-13T11:22:59.825 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0018265499999999999 2024-09-13T11:22:59.825 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:22:59.825 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:22:59.825 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:22:59.825 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:59.107386+0000", 2024-09-13T11:22:59.825 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000109337 2024-09-13T11:22:59.826 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:22:59.826 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:22:59.826 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:22:59.826 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:22:59.826 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:22:59.826 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-13T11:22:59.827 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-13T11:22:59.828 INFO:tasks.ceph.osd.11.smithi177.stderr:2024-09-13T11:22:59.827+0000 7f24792c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26580) UID: 0 2024-09-13T11:22:59.928 INFO:tasks.ceph.osd.12.smithi001.stderr:2024-09-13T11:22:59.924+0000 7f892c6e6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26314) UID: 0 2024-09-13T11:23:00.000 INFO:tasks.radosbench.radosbench.0.smithi177.stdout: 33 16 18468 18452 34.9426 107.188 0.00533225 0.0286087 2024-09-13T11:23:00.023 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-13T11:23:00.023 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [], 2024-09-13T11:23:00.023 INFO:teuthology.orchestra.run.smithi162.stdout: "num_ops": 0 2024-09-13T11:23:00.023 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-13T11:23:00.029 INFO:tasks.ceph.osd.2.smithi162.stderr:2024-09-13T11:23:00.027+0000 7fe92b35a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26313) UID: 0 2024-09-13T11:23:00.039 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-13T11:23:00.130 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-09-13T11:23:00.127+0000 7f97befc9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26315) UID: 0 2024-09-13T11:23:00.224 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-13T11:23:00.224 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [], 2024-09-13T11:23:00.225 INFO:teuthology.orchestra.run.smithi162.stdout: "complaint_time": 30, 2024-09-13T11:23:00.225 INFO:teuthology.orchestra.run.smithi162.stdout: "num_blocked_ops": 0 2024-09-13T11:23:00.225 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-13T11:23:00.231 INFO:tasks.ceph.osd.14.smithi162.stderr:2024-09-13T11:23:00.227+0000 7f235c0db640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26319) UID: 0 2024-09-13T11:23:00.243 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-13T11:23:00.332 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-09-13T11:23:00.327+0000 7f97befc9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26315) UID: 0 2024-09-13T11:23:00.417 INFO:teuthology.orchestra.run.smithi001.stderr:reweighted osd.14 to 1 (10000) 2024-09-13T11:23:00.420 INFO:tasks.ceph.osd.13.smithi092.stderr:2024-09-13T11:23:00.414+0000 7f3e4da13640 -1 osd.13 53 handle_osd_map injecting map crc failure 2024-09-13T11:23:00.426 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-13T11:23:00.426 INFO:teuthology.orchestra.run.smithi162.stdout: "size": 20, 2024-09-13T11:23:00.426 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 600, 2024-09-13T11:23:00.426 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [ 2024-09-13T11:23:00.426 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:00.427 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4257.0:47 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-13T11:23:00.427 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:22:12.319435+0000", 2024-09-13T11:23:00.427 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 48.104621977999997, 2024-09-13T11:23:00.427 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.037475999000000003, 2024-09-13T11:23:00.427 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:23:00.427 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:23:00.427 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:00.427 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:23:00.427 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4257", 2024-09-13T11:23:00.427 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/1488131527", 2024-09-13T11:23:00.427 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 47 2024-09-13T11:23:00.428 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:00.428 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:23:00.428 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:00.428 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:23:00.428 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.319435+0000", 2024-09-13T11:23:00.428 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:00.428 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:00.428 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:00.428 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:23:00.428 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.319435+0000", 2024-09-13T11:23:00.428 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:00.429 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:00.429 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:00.429 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:23:00.429 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.319437+0000", 2024-09-13T11:23:00.430 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.9400000000000001e-06 2024-09-13T11:23:00.430 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:00.430 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:00.430 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:23:00.430 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.319521+0000", 2024-09-13T11:23:00.430 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.4178999999999995e-05 2024-09-13T11:23:00.430 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:00.430 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:00.431 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:23:00.431 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.319543+0000", 2024-09-13T11:23:00.431 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.1750999999999999e-05 2024-09-13T11:23:00.431 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:00.431 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:00.431 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:23:00.431 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.319561+0000", 2024-09-13T11:23:00.431 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.8518000000000001e-05 2024-09-13T11:23:00.431 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:00.431 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:00.431 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:23:00.431 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.345664+0000", 2024-09-13T11:23:00.431 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.026102914000000001 2024-09-13T11:23:00.432 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:00.432 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:00.432 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:23:00.433 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.345799+0000", 2024-09-13T11:23:00.433 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00013452000000000001 2024-09-13T11:23:00.433 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:00.433 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:00.433 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 0", 2024-09-13T11:23:00.433 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.346009+0000", 2024-09-13T11:23:00.434 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00021038799999999999 2024-09-13T11:23:00.434 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:00.434 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:00.434 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:00.434 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.349395+0000", 2024-09-13T11:23:00.434 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0033864759999999998 2024-09-13T11:23:00.434 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:00.434 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:00.434 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:23:00.434 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.356667+0000", 2024-09-13T11:23:00.434 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0072719159999999998 2024-09-13T11:23:00.434 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:00.435 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:00.435 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:23:00.435 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.356801+0000", 2024-09-13T11:23:00.435 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00013402999999999999 2024-09-13T11:23:00.435 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:00.435 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:00.435 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:23:00.435 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.356911+0000", 2024-09-13T11:23:00.435 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000109367 2024-09-13T11:23:00.435 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:00.435 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:23:00.435 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:00.436 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:00.436 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:00.436 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4257.0:48 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-13T11:23:00.436 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:22:12.319584+0000", 2024-09-13T11:23:00.436 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 48.104472477999998, 2024-09-13T11:23:00.436 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.037877980999999998, 2024-09-13T11:23:00.436 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:23:00.436 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:23:00.436 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:00.436 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:23:00.436 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4257", 2024-09-13T11:23:00.436 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/1488131527", 2024-09-13T11:23:00.437 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 48 2024-09-13T11:23:00.437 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:00.437 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:23:00.437 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:00.437 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:23:00.437 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.319584+0000", 2024-09-13T11:23:00.437 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:00.437 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:00.437 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:00.437 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:23:00.437 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.319584+0000", 2024-09-13T11:23:00.437 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:00.437 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:00.438 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:00.438 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:23:00.438 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.319586+0000", 2024-09-13T11:23:00.438 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.013e-06 2024-09-13T11:23:00.438 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:00.438 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:00.438 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:23:00.438 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.319595+0000", 2024-09-13T11:23:00.438 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.9060000000000008e-06 2024-09-13T11:23:00.438 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:00.438 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:00.439 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:23:00.439 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.319615+0000", 2024-09-13T11:23:00.439 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.0378e-05 2024-09-13T11:23:00.439 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:00.439 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:00.439 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:23:00.439 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.319632+0000", 2024-09-13T11:23:00.439 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.6858e-05 2024-09-13T11:23:00.439 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:00.440 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:00.440 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:23:00.440 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.349488+0000", 2024-09-13T11:23:00.440 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.029856038000000001 2024-09-13T11:23:00.440 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:00.441 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:00.441 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:23:00.441 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.349638+0000", 2024-09-13T11:23:00.441 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00014941600000000001 2024-09-13T11:23:00.441 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:00.441 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:00.441 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 0", 2024-09-13T11:23:00.441 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.349838+0000", 2024-09-13T11:23:00.441 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000199929 2024-09-13T11:23:00.442 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:00.442 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:00.442 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:00.442 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.357028+0000", 2024-09-13T11:23:00.442 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0071900799999999997 2024-09-13T11:23:00.442 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:00.442 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:00.442 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:23:00.442 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.357243+0000", 2024-09-13T11:23:00.442 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00021529000000000001 2024-09-13T11:23:00.442 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:00.442 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:00.442 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:23:00.443 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.357365+0000", 2024-09-13T11:23:00.443 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00012191199999999999 2024-09-13T11:23:00.443 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:00.443 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:00.443 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:23:00.443 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.357462+0000", 2024-09-13T11:23:00.443 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.7161e-05 2024-09-13T11:23:00.444 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:00.444 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:23:00.444 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:00.444 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:00.444 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:00.444 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4257.0:49 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-13T11:23:00.444 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:22:12.319663+0000", 2024-09-13T11:23:00.444 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 48.104393438999999, 2024-09-13T11:23:00.445 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.046281448000000003, 2024-09-13T11:23:00.445 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:23:00.445 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:23:00.445 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:00.445 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:23:00.445 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4257", 2024-09-13T11:23:00.445 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/1488131527", 2024-09-13T11:23:00.446 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 49 2024-09-13T11:23:00.446 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:00.446 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:23:00.446 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:00.446 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:23:00.446 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.319663+0000", 2024-09-13T11:23:00.446 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:00.447 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:00.447 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:00.447 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:23:00.447 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.319663+0000", 2024-09-13T11:23:00.447 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:00.447 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:00.447 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:00.447 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:23:00.447 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.319665+0000", 2024-09-13T11:23:00.447 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.0030000000000001e-06 2024-09-13T11:23:00.448 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:00.448 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:00.448 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:23:00.448 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.319675+0000", 2024-09-13T11:23:00.448 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0019e-05 2024-09-13T11:23:00.448 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:00.448 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:00.448 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:23:00.448 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.319697+0000", 2024-09-13T11:23:00.448 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.1387999999999999e-05 2024-09-13T11:23:00.448 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:00.448 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:00.448 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:23:00.449 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.319712+0000", 2024-09-13T11:23:00.449 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5068999999999999e-05 2024-09-13T11:23:00.449 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:00.449 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:00.449 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:23:00.449 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.352248+0000", 2024-09-13T11:23:00.449 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.032536460000000003 2024-09-13T11:23:00.449 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:00.449 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:00.449 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:23:00.449 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.352479+0000", 2024-09-13T11:23:00.449 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00023091499999999999 2024-09-13T11:23:00.450 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:00.450 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:00.450 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 0", 2024-09-13T11:23:00.450 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.352882+0000", 2024-09-13T11:23:00.450 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00040249300000000002 2024-09-13T11:23:00.450 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:00.450 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:00.450 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:00.450 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.357206+0000", 2024-09-13T11:23:00.450 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.004324802 2024-09-13T11:23:00.450 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:00.450 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:00.450 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:23:00.450 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.365708+0000", 2024-09-13T11:23:00.451 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0085012509999999996 2024-09-13T11:23:00.451 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:00.451 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:00.451 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:23:00.451 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.365839+0000", 2024-09-13T11:23:00.451 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000131202 2024-09-13T11:23:00.451 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:00.451 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:00.451 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:23:00.451 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.365945+0000", 2024-09-13T11:23:00.451 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000105846 2024-09-13T11:23:00.451 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:00.451 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:23:00.452 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:00.452 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:00.452 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:00.452 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4257.0:50 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-13T11:23:00.452 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:22:12.319731+0000", 2024-09-13T11:23:00.452 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 48.104325952000003, 2024-09-13T11:23:00.452 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.046650321000000002, 2024-09-13T11:23:00.452 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:23:00.452 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:23:00.452 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:00.452 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:23:00.453 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4257", 2024-09-13T11:23:00.453 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/1488131527", 2024-09-13T11:23:00.453 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 50 2024-09-13T11:23:00.453 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:00.453 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:23:00.453 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:00.453 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:23:00.453 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.319731+0000", 2024-09-13T11:23:00.453 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:00.453 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:00.453 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:00.453 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:23:00.454 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.319731+0000", 2024-09-13T11:23:00.454 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:00.454 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:00.454 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:00.454 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:23:00.454 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.319732+0000", 2024-09-13T11:23:00.454 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.75e-06 2024-09-13T11:23:00.454 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:00.454 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:00.454 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:23:00.454 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.319802+0000", 2024-09-13T11:23:00.454 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.9166999999999997e-05 2024-09-13T11:23:00.455 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:00.455 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:00.455 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:23:00.455 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.319826+0000", 2024-09-13T11:23:00.455 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.4464000000000002e-05 2024-09-13T11:23:00.455 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:00.455 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:00.455 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:23:00.455 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.319856+0000", 2024-09-13T11:23:00.455 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.9530999999999999e-05 2024-09-13T11:23:00.455 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:00.456 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:00.456 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:23:00.456 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.357551+0000", 2024-09-13T11:23:00.456 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.037695687999999998 2024-09-13T11:23:00.456 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:00.456 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:00.456 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:23:00.456 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.357707+0000", 2024-09-13T11:23:00.456 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00015562200000000001 2024-09-13T11:23:00.456 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:00.456 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:00.456 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 0", 2024-09-13T11:23:00.456 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.357906+0000", 2024-09-13T11:23:00.456 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00019874500000000001 2024-09-13T11:23:00.457 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:00.457 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:00.457 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:00.457 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.366111+0000", 2024-09-13T11:23:00.457 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.008205815 2024-09-13T11:23:00.457 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:00.457 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:00.457 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:23:00.457 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.366149+0000", 2024-09-13T11:23:00.457 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.7113999999999997e-05 2024-09-13T11:23:00.458 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:00.458 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:00.458 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:23:00.458 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.366283+0000", 2024-09-13T11:23:00.458 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00013399399999999999 2024-09-13T11:23:00.458 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:00.458 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:00.458 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:23:00.458 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.366381+0000", 2024-09-13T11:23:00.458 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.8431000000000007e-05 2024-09-13T11:23:00.458 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:00.458 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:23:00.458 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:00.459 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:00.459 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:00.459 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4257.0:51 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-13T11:23:00.459 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:22:12.319877+0000", 2024-09-13T11:23:00.459 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 48.104179322, 2024-09-13T11:23:00.459 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.048107695999999998, 2024-09-13T11:23:00.459 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:23:00.459 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:23:00.459 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:00.459 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:23:00.459 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4257", 2024-09-13T11:23:00.459 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/1488131527", 2024-09-13T11:23:00.459 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 51 2024-09-13T11:23:00.459 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:00.460 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:23:00.460 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:00.460 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:23:00.460 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.319877+0000", 2024-09-13T11:23:00.460 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:00.460 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:00.460 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:00.460 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:23:00.460 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.319877+0000", 2024-09-13T11:23:00.460 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:00.460 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:00.460 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:00.460 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:23:00.461 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.319879+0000", 2024-09-13T11:23:00.461 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.44e-06 2024-09-13T11:23:00.461 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:00.461 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:00.461 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:23:00.461 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.319885+0000", 2024-09-13T11:23:00.461 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.8690000000000001e-06 2024-09-13T11:23:00.461 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:00.461 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:00.461 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:23:00.461 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.319894+0000", 2024-09-13T11:23:00.461 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.7389999999999994e-06 2024-09-13T11:23:00.461 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:00.462 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:00.462 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:23:00.462 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.319903+0000", 2024-09-13T11:23:00.462 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.2830000000000001e-06 2024-09-13T11:23:00.462 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:00.462 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:00.462 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:23:00.462 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.361846+0000", 2024-09-13T11:23:00.462 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.041943041 2024-09-13T11:23:00.462 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:00.462 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:00.462 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:23:00.463 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.362090+0000", 2024-09-13T11:23:00.463 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00024382399999999999 2024-09-13T11:23:00.463 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:00.463 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:00.463 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 0", 2024-09-13T11:23:00.463 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.362493+0000", 2024-09-13T11:23:00.463 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00040374600000000002 2024-09-13T11:23:00.463 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:00.463 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:00.463 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:00.463 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.366507+0000", 2024-09-13T11:23:00.463 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0040133369999999996 2024-09-13T11:23:00.463 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:00.463 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:00.464 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:23:00.464 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.367751+0000", 2024-09-13T11:23:00.464 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001244118 2024-09-13T11:23:00.464 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:00.464 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:00.464 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:23:00.464 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.367873+0000", 2024-09-13T11:23:00.464 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000121972 2024-09-13T11:23:00.464 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:00.464 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:00.464 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:23:00.464 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.367985+0000", 2024-09-13T11:23:00.465 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000112327 2024-09-13T11:23:00.465 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:00.465 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:23:00.465 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:00.465 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:00.465 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:00.465 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4257.0:52 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-13T11:23:00.465 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:22:12.319912+0000", 2024-09-13T11:23:00.465 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 48.104144665, 2024-09-13T11:23:00.466 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.049068963, 2024-09-13T11:23:00.466 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:23:00.466 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:23:00.466 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:00.466 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:23:00.466 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4257", 2024-09-13T11:23:00.466 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/1488131527", 2024-09-13T11:23:00.466 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 52 2024-09-13T11:23:00.466 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:00.466 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:23:00.466 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:00.466 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:23:00.466 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.319912+0000", 2024-09-13T11:23:00.466 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:00.467 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:00.467 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:00.467 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:23:00.467 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.319912+0000", 2024-09-13T11:23:00.467 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:00.467 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:00.467 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:00.467 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:23:00.467 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.319913+0000", 2024-09-13T11:23:00.468 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.133e-06 2024-09-13T11:23:00.468 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:00.468 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:00.468 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:23:00.468 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.319918+0000", 2024-09-13T11:23:00.468 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.7729999999999999e-06 2024-09-13T11:23:00.468 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:00.468 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:00.468 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:23:00.468 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.319927+0000", 2024-09-13T11:23:00.468 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.6750000000000008e-06 2024-09-13T11:23:00.469 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:00.469 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:00.469 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:23:00.469 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.319935+0000", 2024-09-13T11:23:00.469 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.9500000000000001e-06 2024-09-13T11:23:00.469 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:00.469 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:00.469 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:23:00.469 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.366585+0000", 2024-09-13T11:23:00.469 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.046649991000000002 2024-09-13T11:23:00.470 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:00.470 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:00.470 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:23:00.470 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.366832+0000", 2024-09-13T11:23:00.470 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00024709999999999999 2024-09-13T11:23:00.470 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:00.470 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:00.470 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 0", 2024-09-13T11:23:00.470 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.367210+0000", 2024-09-13T11:23:00.470 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00037837199999999997 2024-09-13T11:23:00.470 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:00.470 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:00.471 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:23:00.471 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.368430+0000", 2024-09-13T11:23:00.471 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00122013 2024-09-13T11:23:00.471 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:00.471 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:00.471 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:00.471 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.368787+0000", 2024-09-13T11:23:00.471 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00035728300000000001 2024-09-13T11:23:00.471 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:00.471 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:00.471 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:23:00.471 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.368874+0000", 2024-09-13T11:23:00.471 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.6876e-05 2024-09-13T11:23:00.472 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:00.472 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:00.472 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:23:00.472 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.368981+0000", 2024-09-13T11:23:00.472 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00010668 2024-09-13T11:23:00.472 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:00.472 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:23:00.472 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:00.472 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:00.472 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:00.472 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4257.0:70 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-13T11:23:00.472 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:22:12.458708+0000", 2024-09-13T11:23:00.472 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 47.965348708, 2024-09-13T11:23:00.472 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.036822555, 2024-09-13T11:23:00.473 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:23:00.473 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:23:00.473 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:00.473 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:23:00.473 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4257", 2024-09-13T11:23:00.473 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/1488131527", 2024-09-13T11:23:00.473 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 70 2024-09-13T11:23:00.473 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:00.473 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:23:00.473 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:00.473 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:23:00.473 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.458708+0000", 2024-09-13T11:23:00.474 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:00.474 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:00.474 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:00.474 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:23:00.474 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.458708+0000", 2024-09-13T11:23:00.474 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:00.474 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:00.474 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:00.474 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:23:00.474 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.458709+0000", 2024-09-13T11:23:00.474 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.316e-06 2024-09-13T11:23:00.474 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:00.474 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:00.474 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:23:00.475 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.458717+0000", 2024-09-13T11:23:00.475 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.3390000000000004e-06 2024-09-13T11:23:00.475 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:00.475 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:00.475 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:23:00.475 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.458726+0000", 2024-09-13T11:23:00.475 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.2329999999999995e-06 2024-09-13T11:23:00.475 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:00.475 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:00.475 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:23:00.475 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.458733+0000", 2024-09-13T11:23:00.475 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.5959999999999998e-06 2024-09-13T11:23:00.475 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:00.476 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:00.476 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:23:00.476 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.481591+0000", 2024-09-13T11:23:00.476 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.022857796999999999 2024-09-13T11:23:00.476 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:00.476 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:00.476 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:23:00.476 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.481749+0000", 2024-09-13T11:23:00.476 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00015727800000000001 2024-09-13T11:23:00.476 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:00.476 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:00.476 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 0", 2024-09-13T11:23:00.476 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.481952+0000", 2024-09-13T11:23:00.477 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00020356500000000001 2024-09-13T11:23:00.477 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:00.477 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:00.477 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:00.477 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.486070+0000", 2024-09-13T11:23:00.477 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0041175689999999997 2024-09-13T11:23:00.477 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:00.477 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:00.477 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:23:00.477 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.495291+0000", 2024-09-13T11:23:00.477 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0092216499999999996 2024-09-13T11:23:00.477 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:00.477 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:00.478 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:23:00.478 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.495423+0000", 2024-09-13T11:23:00.478 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000131908 2024-09-13T11:23:00.478 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:00.478 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:00.478 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:23:00.478 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.495531+0000", 2024-09-13T11:23:00.478 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000107304 2024-09-13T11:23:00.478 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:00.478 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:23:00.478 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:00.478 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:00.478 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:00.478 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4257.0:71 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-13T11:23:00.479 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:22:12.458743+0000", 2024-09-13T11:23:00.479 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 47.965313875, 2024-09-13T11:23:00.479 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.037378219999999997, 2024-09-13T11:23:00.479 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:23:00.479 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:23:00.479 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:00.479 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:23:00.479 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4257", 2024-09-13T11:23:00.479 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/1488131527", 2024-09-13T11:23:00.479 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 71 2024-09-13T11:23:00.479 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:00.479 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:23:00.480 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:00.480 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:23:00.480 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.458743+0000", 2024-09-13T11:23:00.480 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:00.480 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:00.480 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:00.480 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:23:00.480 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.458743+0000", 2024-09-13T11:23:00.480 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:00.480 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:00.480 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:00.480 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:23:00.481 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.458744+0000", 2024-09-13T11:23:00.481 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.7000000000000003e-07 2024-09-13T11:23:00.481 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:00.481 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:00.481 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:23:00.481 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.458748+0000", 2024-09-13T11:23:00.481 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.1060000000000002e-06 2024-09-13T11:23:00.481 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:00.481 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:00.481 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:23:00.481 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.458760+0000", 2024-09-13T11:23:00.481 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2316e-05 2024-09-13T11:23:00.481 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:00.482 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:00.482 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:23:00.482 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.458768+0000", 2024-09-13T11:23:00.482 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.3690000000000001e-06 2024-09-13T11:23:00.482 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:00.482 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:00.482 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:23:00.482 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.486413+0000", 2024-09-13T11:23:00.482 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.027645187000000002 2024-09-13T11:23:00.482 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:00.482 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:00.483 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:23:00.483 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.486549+0000", 2024-09-13T11:23:00.483 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000135907 2024-09-13T11:23:00.483 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:00.483 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:00.483 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 0", 2024-09-13T11:23:00.483 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.486753+0000", 2024-09-13T11:23:00.483 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00020425700000000001 2024-09-13T11:23:00.484 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:00.484 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:00.484 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:00.484 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.495649+0000", 2024-09-13T11:23:00.484 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0088957589999999996 2024-09-13T11:23:00.484 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:00.484 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:00.484 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:23:00.484 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.495901+0000", 2024-09-13T11:23:00.484 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00025266299999999998 2024-09-13T11:23:00.484 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:00.484 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:00.485 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:23:00.485 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.496023+0000", 2024-09-13T11:23:00.485 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000121748 2024-09-13T11:23:00.485 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:00.485 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:00.485 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:23:00.485 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.496121+0000", 2024-09-13T11:23:00.485 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.7937999999999996e-05 2024-09-13T11:23:00.485 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:00.485 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:23:00.485 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:00.486 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:00.486 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:00.486 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4257.0:72 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-13T11:23:00.486 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:22:12.458777+0000", 2024-09-13T11:23:00.486 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 47.965279817999999, 2024-09-13T11:23:00.486 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.041935317999999999, 2024-09-13T11:23:00.486 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:23:00.486 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:23:00.486 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:00.486 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:23:00.486 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4257", 2024-09-13T11:23:00.486 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/1488131527", 2024-09-13T11:23:00.486 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 72 2024-09-13T11:23:00.487 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:00.487 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:23:00.487 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:00.487 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:23:00.487 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.458777+0000", 2024-09-13T11:23:00.487 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:00.487 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:00.487 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:00.487 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:23:00.487 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.458777+0000", 2024-09-13T11:23:00.487 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:00.487 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:00.487 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:00.488 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:23:00.488 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.458808+0000", 2024-09-13T11:23:00.488 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.0904000000000001e-05 2024-09-13T11:23:00.488 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:00.488 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:00.488 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:23:00.488 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.458904+0000", 2024-09-13T11:23:00.488 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.6547999999999998e-05 2024-09-13T11:23:00.488 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:00.488 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:00.488 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:23:00.488 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.458918+0000", 2024-09-13T11:23:00.489 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3237999999999999e-05 2024-09-13T11:23:00.489 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:00.489 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:00.489 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:23:00.489 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.458926+0000", 2024-09-13T11:23:00.489 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.6929999999999996e-06 2024-09-13T11:23:00.489 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:00.489 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:00.489 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:23:00.489 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.490489+0000", 2024-09-13T11:23:00.489 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.031563015 2024-09-13T11:23:00.489 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:00.489 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:00.490 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:23:00.490 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.490709+0000", 2024-09-13T11:23:00.490 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00022010700000000001 2024-09-13T11:23:00.490 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:00.490 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:00.490 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 0", 2024-09-13T11:23:00.490 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.491113+0000", 2024-09-13T11:23:00.490 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00040412899999999998 2024-09-13T11:23:00.490 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:00.490 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:00.490 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:00.490 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.495864+0000", 2024-09-13T11:23:00.490 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0047506459999999999 2024-09-13T11:23:00.491 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:00.491 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:00.491 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:23:00.491 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.500468+0000", 2024-09-13T11:23:00.491 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0046035670000000002 2024-09-13T11:23:00.491 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:00.491 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:00.491 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:23:00.491 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.500598+0000", 2024-09-13T11:23:00.491 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000130541 2024-09-13T11:23:00.491 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:00.492 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:00.492 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:23:00.492 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.500712+0000", 2024-09-13T11:23:00.492 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00011393 2024-09-13T11:23:00.492 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:00.492 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:23:00.492 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:00.492 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:00.492 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:00.492 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4257.0:73 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-13T11:23:00.492 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:22:12.458939+0000", 2024-09-13T11:23:00.492 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 47.965117499999998, 2024-09-13T11:23:00.493 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.050922262000000003, 2024-09-13T11:23:00.493 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:23:00.493 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:23:00.493 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:00.493 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:23:00.493 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4257", 2024-09-13T11:23:00.493 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/1488131527", 2024-09-13T11:23:00.493 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 73 2024-09-13T11:23:00.493 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:00.493 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:23:00.493 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:00.493 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:23:00.493 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.458939+0000", 2024-09-13T11:23:00.494 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:00.494 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:00.494 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:00.494 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:23:00.494 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.458939+0000", 2024-09-13T11:23:00.494 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:00.494 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:00.494 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:00.494 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:23:00.494 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.458940+0000", 2024-09-13T11:23:00.494 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.22e-06 2024-09-13T11:23:00.494 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:00.495 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:00.495 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:23:00.495 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.458945+0000", 2024-09-13T11:23:00.495 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.6800000000000001e-06 2024-09-13T11:23:00.495 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:00.495 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:00.495 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:23:00.495 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.458954+0000", 2024-09-13T11:23:00.495 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.4589999999999998e-06 2024-09-13T11:23:00.495 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:00.495 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:00.496 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:23:00.496 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.458961+0000", 2024-09-13T11:23:00.496 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.2420000000000003e-06 2024-09-13T11:23:00.496 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:00.496 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:00.496 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:23:00.496 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.496214+0000", 2024-09-13T11:23:00.496 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.037252805999999999 2024-09-13T11:23:00.496 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:00.496 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:00.496 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:23:00.497 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.496446+0000", 2024-09-13T11:23:00.497 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00023255800000000001 2024-09-13T11:23:00.497 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:00.497 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:00.497 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 0", 2024-09-13T11:23:00.497 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.496800+0000", 2024-09-13T11:23:00.497 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00035397400000000001 2024-09-13T11:23:00.497 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:00.497 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:00.497 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:00.498 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.504887+0000", 2024-09-13T11:23:00.498 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0080867769999999999 2024-09-13T11:23:00.498 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:00.498 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:00.498 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:23:00.498 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.509579+0000", 2024-09-13T11:23:00.498 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0046924849999999997 2024-09-13T11:23:00.498 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:00.498 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:00.498 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:23:00.498 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.509748+0000", 2024-09-13T11:23:00.498 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00016907399999999999 2024-09-13T11:23:00.498 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:00.498 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:00.499 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:23:00.499 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.509861+0000", 2024-09-13T11:23:00.499 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00011298699999999999 2024-09-13T11:23:00.499 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:00.499 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:23:00.499 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:00.499 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:00.499 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:00.499 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4257.0:74 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-13T11:23:00.499 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:22:12.458973+0000", 2024-09-13T11:23:00.499 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 47.965084079999997, 2024-09-13T11:23:00.499 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.055492705000000003, 2024-09-13T11:23:00.499 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:23:00.500 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:23:00.500 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:00.500 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:23:00.500 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4257", 2024-09-13T11:23:00.500 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/1488131527", 2024-09-13T11:23:00.500 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 74 2024-09-13T11:23:00.500 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:00.500 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:23:00.500 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:00.500 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:23:00.500 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.458973+0000", 2024-09-13T11:23:00.501 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:00.501 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:00.501 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:00.501 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:23:00.501 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.458973+0000", 2024-09-13T11:23:00.501 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:00.501 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:00.501 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:00.501 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:23:00.501 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.458974+0000", 2024-09-13T11:23:00.501 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.3699999999999999e-07 2024-09-13T11:23:00.501 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:00.501 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:00.502 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:23:00.502 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.458978+0000", 2024-09-13T11:23:00.502 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.4259999999999996e-06 2024-09-13T11:23:00.502 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:00.502 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:00.502 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:23:00.503 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.458994+0000", 2024-09-13T11:23:00.503 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5852e-05 2024-09-13T11:23:00.503 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:00.503 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:00.503 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:23:00.503 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.459002+0000", 2024-09-13T11:23:00.503 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.5830000000000007e-06 2024-09-13T11:23:00.503 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:00.503 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:00.503 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:23:00.504 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.500820+0000", 2024-09-13T11:23:00.504 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.041817432000000002 2024-09-13T11:23:00.504 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:00.504 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:00.504 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:23:00.504 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.500973+0000", 2024-09-13T11:23:00.504 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00015326200000000001 2024-09-13T11:23:00.504 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:00.504 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:00.504 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 0", 2024-09-13T11:23:00.504 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.501212+0000", 2024-09-13T11:23:00.504 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00023885099999999999 2024-09-13T11:23:00.505 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:00.505 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:00.505 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:00.505 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.505008+0000", 2024-09-13T11:23:00.505 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0037961850000000001 2024-09-13T11:23:00.505 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:00.505 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:00.505 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:23:00.505 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.514227+0000", 2024-09-13T11:23:00.505 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0092187889999999998 2024-09-13T11:23:00.505 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:00.506 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:00.506 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:23:00.506 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.514362+0000", 2024-09-13T11:23:00.506 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000134967 2024-09-13T11:23:00.506 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:00.506 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:00.506 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:23:00.506 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.514465+0000", 2024-09-13T11:23:00.506 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000103421 2024-09-13T11:23:00.506 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:00.507 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:23:00.507 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:00.507 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:00.507 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:00.507 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4257.0:75 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-13T11:23:00.507 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:22:12.459011+0000", 2024-09-13T11:23:00.507 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 47.965045183000001, 2024-09-13T11:23:00.508 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.056002977000000002, 2024-09-13T11:23:00.508 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:23:00.508 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:23:00.508 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:00.508 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:23:00.508 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4257", 2024-09-13T11:23:00.508 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/1488131527", 2024-09-13T11:23:00.508 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 75 2024-09-13T11:23:00.508 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:00.508 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:23:00.508 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:00.509 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:23:00.509 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.459011+0000", 2024-09-13T11:23:00.509 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:00.509 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:00.509 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:00.509 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:23:00.509 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.459011+0000", 2024-09-13T11:23:00.509 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:00.509 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:00.509 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:00.509 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:23:00.509 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.459012+0000", 2024-09-13T11:23:00.509 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.9999999999999995e-07 2024-09-13T11:23:00.509 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:00.510 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:00.510 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:23:00.510 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.459060+0000", 2024-09-13T11:23:00.510 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.7587999999999999e-05 2024-09-13T11:23:00.510 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:00.510 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:00.510 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:23:00.511 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.459070+0000", 2024-09-13T11:23:00.511 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.9690000000000006e-06 2024-09-13T11:23:00.511 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:00.511 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:00.511 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:23:00.512 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.459078+0000", 2024-09-13T11:23:00.512 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.5329999999999997e-06 2024-09-13T11:23:00.512 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:00.512 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:00.512 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:23:00.512 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.505105+0000", 2024-09-13T11:23:00.512 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.046027908999999999 2024-09-13T11:23:00.512 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:00.512 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:00.513 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:23:00.513 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.505322+0000", 2024-09-13T11:23:00.513 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00021677600000000001 2024-09-13T11:23:00.513 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:00.513 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:00.513 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 0", 2024-09-13T11:23:00.513 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.505675+0000", 2024-09-13T11:23:00.513 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00035293800000000001 2024-09-13T11:23:00.513 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:00.513 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:00.513 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:00.514 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.514620+0000", 2024-09-13T11:23:00.514 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0089446730000000002 2024-09-13T11:23:00.514 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:00.514 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:00.514 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:23:00.515 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.514777+0000", 2024-09-13T11:23:00.515 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00015762999999999999 2024-09-13T11:23:00.515 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:00.515 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:00.515 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:23:00.515 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.514903+0000", 2024-09-13T11:23:00.515 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00012559800000000001 2024-09-13T11:23:00.516 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:00.516 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:00.516 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:23:00.516 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.515014+0000", 2024-09-13T11:23:00.516 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000111363 2024-09-13T11:23:00.516 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:00.517 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:23:00.517 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:00.517 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:00.517 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:00.517 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4257.0:76 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-13T11:23:00.517 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:22:12.459088+0000", 2024-09-13T11:23:00.517 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 47.964968474000003, 2024-09-13T11:23:00.517 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.056165815000000001, 2024-09-13T11:23:00.518 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:23:00.518 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:23:00.518 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:00.518 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:23:00.518 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4257", 2024-09-13T11:23:00.518 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/1488131527", 2024-09-13T11:23:00.518 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 76 2024-09-13T11:23:00.519 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:00.519 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:23:00.519 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:00.519 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:23:00.519 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.459088+0000", 2024-09-13T11:23:00.519 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:00.520 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:00.520 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:00.520 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:23:00.520 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.459088+0000", 2024-09-13T11:23:00.520 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:00.520 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:00.521 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:00.521 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:23:00.521 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.459089+0000", 2024-09-13T11:23:00.521 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.114e-06 2024-09-13T11:23:00.521 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:00.521 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:00.521 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:23:00.522 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.459094+0000", 2024-09-13T11:23:00.522 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.0320000000000003e-06 2024-09-13T11:23:00.522 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:00.522 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:00.522 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:23:00.522 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.459103+0000", 2024-09-13T11:23:00.522 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.2989999999999997e-06 2024-09-13T11:23:00.522 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:00.522 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:00.522 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:23:00.522 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.459110+0000", 2024-09-13T11:23:00.522 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.9970000000000001e-06 2024-09-13T11:23:00.523 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:00.523 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:00.523 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:23:00.523 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.509960+0000", 2024-09-13T11:23:00.523 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.050850601000000002 2024-09-13T11:23:00.523 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:00.523 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:00.523 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:23:00.523 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.510216+0000", 2024-09-13T11:23:00.523 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00025548299999999998 2024-09-13T11:23:00.524 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:00.524 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:00.524 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 0", 2024-09-13T11:23:00.524 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.510559+0000", 2024-09-13T11:23:00.524 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00034306199999999999 2024-09-13T11:23:00.524 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:00.524 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:00.524 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:00.524 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.514745+0000", 2024-09-13T11:23:00.524 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0041858920000000001 2024-09-13T11:23:00.524 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:00.524 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:00.524 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:23:00.525 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.515035+0000", 2024-09-13T11:23:00.525 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00029025600000000002 2024-09-13T11:23:00.525 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:00.525 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:00.525 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:23:00.525 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.515152+0000", 2024-09-13T11:23:00.525 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000116872 2024-09-13T11:23:00.525 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:00.525 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:00.525 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:23:00.525 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.515254+0000", 2024-09-13T11:23:00.525 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00010220700000000001 2024-09-13T11:23:00.525 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:00.526 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:23:00.526 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:00.526 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:00.526 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:00.526 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4257.0:80 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 262144~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-13T11:23:00.526 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:22:12.516065+0000", 2024-09-13T11:23:00.526 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 47.907991430999999, 2024-09-13T11:23:00.526 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.46461997999999999, 2024-09-13T11:23:00.526 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:23:00.526 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:23:00.526 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:00.526 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:23:00.526 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4257", 2024-09-13T11:23:00.526 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/1488131527", 2024-09-13T11:23:00.527 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 80 2024-09-13T11:23:00.527 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:00.527 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:23:00.527 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:00.527 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:23:00.527 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.516065+0000", 2024-09-13T11:23:00.527 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:00.527 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:00.527 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:00.527 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:23:00.527 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.516065+0000", 2024-09-13T11:23:00.527 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:00.527 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:00.528 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:00.528 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:23:00.528 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.516066+0000", 2024-09-13T11:23:00.528 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0869999999999999e-06 2024-09-13T11:23:00.528 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:00.528 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:00.528 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:23:00.528 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.516112+0000", 2024-09-13T11:23:00.528 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.5503000000000001e-05 2024-09-13T11:23:00.528 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:00.529 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:00.529 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:23:00.529 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.516128+0000", 2024-09-13T11:23:00.529 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5891000000000001e-05 2024-09-13T11:23:00.529 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:00.529 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:00.529 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:23:00.529 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.516138+0000", 2024-09-13T11:23:00.529 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.8120000000000002e-06 2024-09-13T11:23:00.529 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:00.529 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:00.530 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:23:00.530 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.520398+0000", 2024-09-13T11:23:00.530 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0042605450000000001 2024-09-13T11:23:00.530 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:00.530 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:00.530 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:23:00.530 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.520526+0000", 2024-09-13T11:23:00.530 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000128355 2024-09-13T11:23:00.530 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:00.530 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:00.530 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 0", 2024-09-13T11:23:00.530 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.520730+0000", 2024-09-13T11:23:00.530 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00020337799999999999 2024-09-13T11:23:00.531 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:00.531 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:00.531 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:23:00.531 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.522417+0000", 2024-09-13T11:23:00.531 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001687184 2024-09-13T11:23:00.531 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:00.531 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:00.531 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:00.531 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.980408+0000", 2024-09-13T11:23:00.531 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.45799068399999998 2024-09-13T11:23:00.531 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:00.531 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:00.531 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:23:00.532 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.980565+0000", 2024-09-13T11:23:00.532 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00015760899999999999 2024-09-13T11:23:00.532 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:00.532 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:00.532 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:23:00.532 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.980685+0000", 2024-09-13T11:23:00.532 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000119932 2024-09-13T11:23:00.532 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:00.532 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:23:00.532 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:00.532 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:00.532 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:00.533 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop_reply(client.4257.0:80 1.0 e24/23 ondisk, result = 0)", 2024-09-13T11:23:00.533 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:22:12.522938+0000", 2024-09-13T11:23:00.533 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 47.901118398000001, 2024-09-13T11:23:00.533 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.45776420600000001, 2024-09-13T11:23:00.533 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:23:00.533 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:23:00.533 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:23:00.533 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:23:00.533 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:00.533 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:23:00.534 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.522938+0000", 2024-09-13T11:23:00.534 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:00.534 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:00.534 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:00.534 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:23:00.534 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.522938+0000", 2024-09-13T11:23:00.534 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:00.534 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:00.534 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:00.534 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:23:00.534 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.522941+0000", 2024-09-13T11:23:00.535 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.0290000000000001e-06 2024-09-13T11:23:00.535 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:00.535 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:00.535 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:23:00.535 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.522950+0000", 2024-09-13T11:23:00.535 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.7499999999999992e-06 2024-09-13T11:23:00.535 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:00.535 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:00.535 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:23:00.535 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.980215+0000", 2024-09-13T11:23:00.535 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.45726523800000002 2024-09-13T11:23:00.535 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:00.536 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:00.536 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:23:00.536 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.980229+0000", 2024-09-13T11:23:00.536 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3549e-05 2024-09-13T11:23:00.536 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:00.536 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:00.536 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:23:00.536 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.980365+0000", 2024-09-13T11:23:00.536 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00013608700000000001 2024-09-13T11:23:00.536 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:00.536 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:00.536 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:23:00.536 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.980388+0000", 2024-09-13T11:23:00.537 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.3112000000000002e-05 2024-09-13T11:23:00.537 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:00.537 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:00.537 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:23:00.537 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.980702+0000", 2024-09-13T11:23:00.537 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00031444099999999999 2024-09-13T11:23:00.537 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:00.537 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:23:00.537 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:00.537 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:00.537 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:00.537 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:9761 3.8 3:1c01147e:::benchmark_data_smithi177_31116_object9760:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-09-13T11:23:00.537 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:22:54.198844+0000", 2024-09-13T11:23:00.538 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 6.2252130760000002, 2024-09-13T11:23:00.538 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.051690649999999998, 2024-09-13T11:23:00.538 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:23:00.538 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:23:00.538 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:00.538 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:23:00.538 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:23:00.538 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:00.538 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 9761 2024-09-13T11:23:00.538 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:00.538 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:23:00.538 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:00.539 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:23:00.539 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:54.198844+0000", 2024-09-13T11:23:00.539 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:00.539 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:00.539 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:00.539 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:23:00.539 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:54.198844+0000", 2024-09-13T11:23:00.539 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:00.539 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:00.539 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:00.539 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:23:00.539 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:54.198847+0000", 2024-09-13T11:23:00.540 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.1520000000000001e-06 2024-09-13T11:23:00.540 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:00.540 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:00.540 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:23:00.540 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:54.198906+0000", 2024-09-13T11:23:00.540 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.9315999999999999e-05 2024-09-13T11:23:00.540 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:00.540 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:00.540 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:23:00.540 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:54.198928+0000", 2024-09-13T11:23:00.540 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.2053999999999999e-05 2024-09-13T11:23:00.540 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:00.540 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:00.541 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:23:00.541 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:54.198947+0000", 2024-09-13T11:23:00.541 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.8482e-05 2024-09-13T11:23:00.541 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:00.541 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:00.541 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:23:00.541 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:54.199074+0000", 2024-09-13T11:23:00.541 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00012774600000000001 2024-09-13T11:23:00.541 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:00.541 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:00.541 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:23:00.541 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:54.199351+0000", 2024-09-13T11:23:00.541 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00027705900000000002 2024-09-13T11:23:00.542 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:00.542 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:00.542 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 1", 2024-09-13T11:23:00.542 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:54.199707+0000", 2024-09-13T11:23:00.542 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00035590600000000002 2024-09-13T11:23:00.542 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:00.542 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:00.542 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:23:00.542 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:54.201465+0000", 2024-09-13T11:23:00.542 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0017581179999999999 2024-09-13T11:23:00.542 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:00.542 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:00.542 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:00.543 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:54.250374+0000", 2024-09-13T11:23:00.543 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.048908419000000002 2024-09-13T11:23:00.543 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:00.543 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:00.543 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:23:00.543 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:54.250467+0000", 2024-09-13T11:23:00.543 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.3367000000000002e-05 2024-09-13T11:23:00.543 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:00.543 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:00.543 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:23:00.543 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:54.250534+0000", 2024-09-13T11:23:00.544 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.7031000000000002e-05 2024-09-13T11:23:00.544 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:00.544 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:23:00.544 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:00.544 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:00.544 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:00.544 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:9859 3.8 3:13ff5619:::benchmark_data_smithi177_31116_object9858:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-09-13T11:23:00.544 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:22:54.263848+0000", 2024-09-13T11:23:00.544 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 6.1602088349999997, 2024-09-13T11:23:00.545 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.043382326999999998, 2024-09-13T11:23:00.545 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:23:00.545 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:23:00.545 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:00.545 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:23:00.545 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:23:00.545 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:00.545 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 9859 2024-09-13T11:23:00.545 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:00.546 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:23:00.546 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:00.546 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:23:00.546 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:54.263848+0000", 2024-09-13T11:23:00.546 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:00.546 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:00.546 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:00.546 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:23:00.546 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:54.263848+0000", 2024-09-13T11:23:00.546 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:00.546 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:00.546 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:00.547 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:23:00.547 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:54.263852+0000", 2024-09-13T11:23:00.547 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.4469999999999999e-06 2024-09-13T11:23:00.547 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:00.547 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:00.547 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:23:00.547 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:54.263938+0000", 2024-09-13T11:23:00.547 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.5494000000000005e-05 2024-09-13T11:23:00.547 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:00.547 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:00.548 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:23:00.548 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:54.263975+0000", 2024-09-13T11:23:00.548 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.7109e-05 2024-09-13T11:23:00.548 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:00.548 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:00.548 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:23:00.548 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:54.263996+0000", 2024-09-13T11:23:00.548 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.0656e-05 2024-09-13T11:23:00.548 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:00.548 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:00.548 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:23:00.548 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:54.264144+0000", 2024-09-13T11:23:00.549 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00014840900000000001 2024-09-13T11:23:00.549 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:00.549 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:00.549 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:23:00.549 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:54.264369+0000", 2024-09-13T11:23:00.549 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00022494600000000001 2024-09-13T11:23:00.549 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:00.549 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:00.549 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 1", 2024-09-13T11:23:00.550 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:54.264580+0000", 2024-09-13T11:23:00.550 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00021120399999999999 2024-09-13T11:23:00.550 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:00.550 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:00.550 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:23:00.550 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:54.266244+0000", 2024-09-13T11:23:00.550 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001663662 2024-09-13T11:23:00.550 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:00.550 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:00.550 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:00.550 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:54.306964+0000", 2024-09-13T11:23:00.551 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.040719995000000002 2024-09-13T11:23:00.551 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:00.551 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:00.551 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:23:00.551 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:54.307111+0000", 2024-09-13T11:23:00.551 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00014726600000000001 2024-09-13T11:23:00.551 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:00.551 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:00.551 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:23:00.551 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:54.307230+0000", 2024-09-13T11:23:00.551 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000119139 2024-09-13T11:23:00.551 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:00.551 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:23:00.552 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:00.552 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:00.552 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:00.552 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:9925 3.8 3:1a3e39f4:::benchmark_data_smithi177_31116_object9924:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-09-13T11:23:00.552 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:22:54.321245+0000", 2024-09-13T11:23:00.552 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 6.1028115359999999, 2024-09-13T11:23:00.553 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.035621280999999998, 2024-09-13T11:23:00.553 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:23:00.553 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:23:00.553 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:00.553 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:23:00.553 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:23:00.553 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:00.554 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 9925 2024-09-13T11:23:00.554 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:00.555 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:23:00.555 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:00.555 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:23:00.555 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:54.321245+0000", 2024-09-13T11:23:00.555 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:00.555 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:00.555 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:00.555 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:23:00.555 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:54.321245+0000", 2024-09-13T11:23:00.555 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:00.556 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:00.556 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:00.556 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:23:00.556 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:54.321255+0000", 2024-09-13T11:23:00.556 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.6819999999999998e-06 2024-09-13T11:23:00.556 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:00.556 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:00.556 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:23:00.556 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:54.321309+0000", 2024-09-13T11:23:00.556 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.4605000000000003e-05 2024-09-13T11:23:00.556 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:00.556 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:00.557 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:23:00.557 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:54.321337+0000", 2024-09-13T11:23:00.557 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.7555999999999998e-05 2024-09-13T11:23:00.557 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:00.557 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:00.557 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:23:00.557 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:54.321352+0000", 2024-09-13T11:23:00.557 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4934e-05 2024-09-13T11:23:00.558 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:00.558 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:00.558 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:23:00.558 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:54.321491+0000", 2024-09-13T11:23:00.558 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000139398 2024-09-13T11:23:00.558 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:00.558 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:00.558 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:23:00.558 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:54.321716+0000", 2024-09-13T11:23:00.558 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00022440900000000001 2024-09-13T11:23:00.558 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:00.559 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:00.559 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 1", 2024-09-13T11:23:00.559 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:54.321931+0000", 2024-09-13T11:23:00.559 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000215353 2024-09-13T11:23:00.559 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:00.559 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:00.559 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:23:00.559 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:54.323707+0000", 2024-09-13T11:23:00.559 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0017756029999999999 2024-09-13T11:23:00.559 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:00.559 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:00.559 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:00.559 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:54.356721+0000", 2024-09-13T11:23:00.560 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.033014565000000003 2024-09-13T11:23:00.560 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:00.560 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:00.560 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:23:00.560 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:54.356803+0000", 2024-09-13T11:23:00.560 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.1557999999999994e-05 2024-09-13T11:23:00.560 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:00.560 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:00.560 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:23:00.560 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:54.356866+0000", 2024-09-13T11:23:00.560 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.3618e-05 2024-09-13T11:23:00.560 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:00.560 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:23:00.561 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:00.561 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:00.561 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:00.561 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:9940 3.8 3:1dba252f:::benchmark_data_smithi177_31116_object9939:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-09-13T11:23:00.561 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:22:54.337660+0000", 2024-09-13T11:23:00.561 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 6.0863963759999997, 2024-09-13T11:23:00.561 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.039237778000000001, 2024-09-13T11:23:00.561 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:23:00.561 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:23:00.561 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:00.561 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:23:00.561 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:23:00.561 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:00.561 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 9940 2024-09-13T11:23:00.562 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:00.562 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:23:00.562 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:00.562 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:23:00.562 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:54.337660+0000", 2024-09-13T11:23:00.562 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:00.562 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:00.562 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:00.562 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:23:00.562 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:54.337660+0000", 2024-09-13T11:23:00.562 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:00.562 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:00.562 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:00.563 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:23:00.563 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:54.337669+0000", 2024-09-13T11:23:00.563 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.0799999999999995e-06 2024-09-13T11:23:00.563 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:00.563 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:00.563 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:23:00.563 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:54.337749+0000", 2024-09-13T11:23:00.563 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.9599999999999997e-05 2024-09-13T11:23:00.563 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:00.563 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:00.563 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:23:00.563 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:54.337776+0000", 2024-09-13T11:23:00.563 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.7336000000000001e-05 2024-09-13T11:23:00.563 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:00.564 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:00.564 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:23:00.564 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:54.337790+0000", 2024-09-13T11:23:00.564 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4088e-05 2024-09-13T11:23:00.564 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:00.564 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:00.564 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:23:00.564 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:54.337938+0000", 2024-09-13T11:23:00.564 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00014714799999999999 2024-09-13T11:23:00.564 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:00.564 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:00.564 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:23:00.564 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:54.338223+0000", 2024-09-13T11:23:00.565 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00028498500000000002 2024-09-13T11:23:00.565 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:00.565 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:00.565 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 1", 2024-09-13T11:23:00.565 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:54.338512+0000", 2024-09-13T11:23:00.565 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00028928500000000002 2024-09-13T11:23:00.565 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:00.565 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:00.565 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:23:00.565 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:54.340477+0000", 2024-09-13T11:23:00.565 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001965212 2024-09-13T11:23:00.565 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:00.566 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:00.566 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:00.566 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:54.376726+0000", 2024-09-13T11:23:00.566 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.036249129999999997 2024-09-13T11:23:00.566 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:00.566 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:00.566 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:23:00.566 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:54.376821+0000", 2024-09-13T11:23:00.566 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.5004e-05 2024-09-13T11:23:00.566 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:00.566 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:00.566 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:23:00.567 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:54.376898+0000", 2024-09-13T11:23:00.567 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.6909999999999994e-05 2024-09-13T11:23:00.567 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:00.567 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:23:00.567 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:00.567 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:00.567 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:00.567 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:9957 3.8 3:194c212b:::benchmark_data_smithi177_31116_object9956:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-09-13T11:23:00.567 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:22:54.353667+0000", 2024-09-13T11:23:00.567 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 6.0703894079999996, 2024-09-13T11:23:00.567 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.034347623000000001, 2024-09-13T11:23:00.567 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:23:00.567 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:23:00.568 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:00.568 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:23:00.568 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:23:00.568 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:00.568 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 9957 2024-09-13T11:23:00.568 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:00.568 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:23:00.568 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:00.568 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:23:00.568 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:54.353667+0000", 2024-09-13T11:23:00.568 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:00.568 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:00.569 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:00.569 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:23:00.569 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:54.353667+0000", 2024-09-13T11:23:00.569 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:00.569 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:00.569 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:00.569 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:23:00.569 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:54.353670+0000", 2024-09-13T11:23:00.569 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.1200000000000002e-06 2024-09-13T11:23:00.570 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:00.570 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:00.570 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:23:00.570 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:54.353744+0000", 2024-09-13T11:23:00.570 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.3478000000000004e-05 2024-09-13T11:23:00.570 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:00.573 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:00.573 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:23:00.573 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:54.353772+0000", 2024-09-13T11:23:00.573 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.7653000000000001e-05 2024-09-13T11:23:00.573 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:00.573 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:00.573 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:23:00.574 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:54.353785+0000", 2024-09-13T11:23:00.574 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.38e-05 2024-09-13T11:23:00.574 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:00.574 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:00.574 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:23:00.574 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:54.353931+0000", 2024-09-13T11:23:00.574 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00014533499999999999 2024-09-13T11:23:00.574 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:00.574 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:00.574 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:23:00.574 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:54.354156+0000", 2024-09-13T11:23:00.574 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00022579300000000001 2024-09-13T11:23:00.574 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:00.575 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:00.575 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 1", 2024-09-13T11:23:00.575 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:54.354378+0000", 2024-09-13T11:23:00.575 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000221883 2024-09-13T11:23:00.575 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:00.575 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:00.575 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:23:00.575 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:54.355982+0000", 2024-09-13T11:23:00.575 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001604065 2024-09-13T11:23:00.575 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:00.575 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:00.575 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:00.575 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:54.387759+0000", 2024-09-13T11:23:00.576 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.031776186999999997 2024-09-13T11:23:00.576 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:00.576 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:00.576 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:23:00.576 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:54.387904+0000", 2024-09-13T11:23:00.576 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00014515200000000001 2024-09-13T11:23:00.576 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:00.576 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:00.576 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:23:00.576 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:54.388015+0000", 2024-09-13T11:23:00.576 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00011115700000000001 2024-09-13T11:23:00.576 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:00.576 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:23:00.577 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:00.577 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:00.577 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:23:00.577 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-13T11:23:00.577 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-13T11:23:00.579 INFO:tasks.ceph.osd.5.smithi092.stderr:2024-09-13T11:23:00.574+0000 7f6f0b705640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26346) UID: 0 2024-09-13T11:23:00.580 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 15 1 2024-09-13T11:23:00.580 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-09-13T11:23:00.610 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T11:23:00.679 INFO:tasks.ceph.osd.7.smithi177.stderr:2024-09-13T11:23:00.679+0000 7f897b51d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26575) UID: 0 2024-09-13T11:23:00.770 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-13T11:23:00.770 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [ 2024-09-13T11:23:00.771 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:00.771 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:20106 3.3 3:c9ac27ed:::benchmark_data_smithi177_31116_object20105:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e54)", 2024-09-13T11:23:00.771 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:00.761781+0000", 2024-09-13T11:23:00.771 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 0.0084911280000000006, 2024-09-13T11:23:00.771 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0085205669999999997, 2024-09-13T11:23:00.771 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:23:00.771 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:23:00.771 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "waiting for sub ops", 2024-09-13T11:23:00.771 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:23:00.771 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:23:00.771 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:00.771 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 20106 2024-09-13T11:23:00.771 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:00.771 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:23:00.772 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:00.772 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:23:00.772 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:00.761781+0000", 2024-09-13T11:23:00.772 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:00.772 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:00.772 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:00.772 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:23:00.772 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:00.761781+0000", 2024-09-13T11:23:00.772 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:00.772 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:00.772 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:00.772 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:23:00.772 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:00.761784+0000", 2024-09-13T11:23:00.773 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.2119999999999999e-06 2024-09-13T11:23:00.773 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:00.773 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:00.773 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:23:00.773 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:00.761845+0000", 2024-09-13T11:23:00.773 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.0198000000000002e-05 2024-09-13T11:23:00.773 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:00.773 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:00.773 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:23:00.774 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:00.761874+0000", 2024-09-13T11:23:00.774 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.9502999999999999e-05 2024-09-13T11:23:00.774 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:00.774 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:00.774 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:23:00.774 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:00.761890+0000", 2024-09-13T11:23:00.774 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5957999999999999e-05 2024-09-13T11:23:00.774 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:00.775 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:00.775 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:23:00.775 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:00.761985+0000", 2024-09-13T11:23:00.775 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.4611999999999995e-05 2024-09-13T11:23:00.775 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:00.775 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:00.775 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:23:00.775 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:00.762196+0000", 2024-09-13T11:23:00.776 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00021149799999999999 2024-09-13T11:23:00.776 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:00.776 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:00.776 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 7", 2024-09-13T11:23:00.776 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:00.762338+0000", 2024-09-13T11:23:00.776 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00014227399999999999 2024-09-13T11:23:00.776 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:00.776 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:00.776 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:23:00.777 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:00.764994+0000", 2024-09-13T11:23:00.777 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0026555229999999999 2024-09-13T11:23:00.777 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:00.777 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:23:00.777 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:00.777 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:00.780 INFO:teuthology.orchestra.run.smithi162.stdout: ], 2024-09-13T11:23:00.780 INFO:teuthology.orchestra.run.smithi162.stdout: "num_ops": 1 2024-09-13T11:23:00.780 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-13T11:23:00.781 INFO:tasks.ceph.osd.3.smithi177.stderr:2024-09-13T11:23:00.779+0000 7f10adba4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26574) UID: 0 2024-09-13T11:23:00.787 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-09-13T11:23:00.827 INFO:teuthology.orchestra.run.smithi001.stdout:osd.0: {} 2024-09-13T11:23:00.827 INFO:teuthology.orchestra.run.smithi001.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T11:23:00.867 INFO:teuthology.orchestra.run.smithi001.stdout:osd.1: {} 2024-09-13T11:23:00.867 INFO:teuthology.orchestra.run.smithi001.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T11:23:00.882 INFO:tasks.ceph.osd.3.smithi177.stderr:2024-09-13T11:23:00.883+0000 7f10adba4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26574) UID: 0 2024-09-13T11:23:00.896 INFO:teuthology.orchestra.run.smithi001.stdout:osd.2: {} 2024-09-13T11:23:00.896 INFO:teuthology.orchestra.run.smithi001.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T11:23:00.924 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-13T11:23:00.924 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [], 2024-09-13T11:23:00.924 INFO:teuthology.orchestra.run.smithi162.stdout: "complaint_time": 30, 2024-09-13T11:23:00.924 INFO:teuthology.orchestra.run.smithi162.stdout: "num_blocked_ops": 0 2024-09-13T11:23:00.924 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-13T11:23:00.936 INFO:teuthology.orchestra.run.smithi001.stdout:osd.3: {} 2024-09-13T11:23:00.936 INFO:teuthology.orchestra.run.smithi001.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T11:23:00.940 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-09-13T11:23:00.962 INFO:teuthology.orchestra.run.smithi001.stdout:osd.4: {} 2024-09-13T11:23:00.962 INFO:teuthology.orchestra.run.smithi001.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T11:23:00.983 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-13T11:23:00.976+0000 7f6a68775640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26307) UID: 0 2024-09-13T11:23:00.989 INFO:teuthology.orchestra.run.smithi001.stdout:osd.5: {} 2024-09-13T11:23:00.989 INFO:teuthology.orchestra.run.smithi001.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T11:23:01.000 INFO:tasks.radosbench.radosbench.0.smithi177.stdout: 34 16 20649 20633 37.9236 136.312 0.00589462 0.0263616 2024-09-13T11:23:01.019 INFO:teuthology.orchestra.run.smithi001.stdout:osd.6: {} 2024-09-13T11:23:01.019 INFO:teuthology.orchestra.run.smithi001.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T11:23:01.058 INFO:teuthology.orchestra.run.smithi001.stdout:osd.7: {} 2024-09-13T11:23:01.058 INFO:teuthology.orchestra.run.smithi001.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T11:23:01.075 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-13T11:23:01.075 INFO:teuthology.orchestra.run.smithi162.stdout: "size": 20, 2024-09-13T11:23:01.075 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 600, 2024-09-13T11:23:01.075 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [ 2024-09-13T11:23:01.075 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:01.075 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:1843 3.3 3:cb86274f:::benchmark_data_smithi177_31116_object1842:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-09-13T11:23:01.075 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:22:50.096099+0000", 2024-09-13T11:23:01.075 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 10.976812528, 2024-09-13T11:23:01.075 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.031518746, 2024-09-13T11:23:01.076 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:23:01.076 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:23:01.076 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:01.076 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:23:01.076 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:23:01.076 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:01.076 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 1843 2024-09-13T11:23:01.076 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:01.076 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:23:01.076 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:01.077 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:23:01.077 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:50.096099+0000", 2024-09-13T11:23:01.077 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:01.077 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:01.077 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:01.077 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:23:01.077 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:50.096099+0000", 2024-09-13T11:23:01.077 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:01.077 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:01.077 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:01.077 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:23:01.077 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:50.096101+0000", 2024-09-13T11:23:01.077 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.013e-06 2024-09-13T11:23:01.078 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:01.078 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:01.078 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:23:01.078 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:50.096162+0000", 2024-09-13T11:23:01.078 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.0579999999999999e-05 2024-09-13T11:23:01.078 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:01.078 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:01.078 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:23:01.078 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:50.096178+0000", 2024-09-13T11:23:01.078 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.6209e-05 2024-09-13T11:23:01.078 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:01.078 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:01.078 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:23:01.078 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:50.096187+0000", 2024-09-13T11:23:01.079 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.2539999999999998e-06 2024-09-13T11:23:01.079 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:01.079 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:01.079 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:23:01.079 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:50.096269+0000", 2024-09-13T11:23:01.079 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.1777e-05 2024-09-13T11:23:01.079 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:01.079 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:01.079 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:23:01.079 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:50.096480+0000", 2024-09-13T11:23:01.079 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00021105599999999999 2024-09-13T11:23:01.079 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:01.079 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:01.079 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 7", 2024-09-13T11:23:01.080 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:50.096733+0000", 2024-09-13T11:23:01.080 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00025259700000000002 2024-09-13T11:23:01.080 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:01.080 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:01.080 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:23:01.080 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:50.098381+0000", 2024-09-13T11:23:01.080 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0016479960000000001 2024-09-13T11:23:01.080 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:01.080 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:01.080 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:01.080 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:50.127463+0000", 2024-09-13T11:23:01.080 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.029082202000000001 2024-09-13T11:23:01.080 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:01.081 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:01.081 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:23:01.081 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:50.127552+0000", 2024-09-13T11:23:01.081 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.9265999999999995e-05 2024-09-13T11:23:01.081 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:01.081 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:01.081 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:23:01.081 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:50.127618+0000", 2024-09-13T11:23:01.082 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.5796000000000005e-05 2024-09-13T11:23:01.082 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:01.082 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:23:01.082 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:01.082 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:01.082 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:01.082 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop_reply(client.4665.0:1843 3.3 e43/30 ondisk, result = 0)", 2024-09-13T11:23:01.082 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:22:50.098092+0000", 2024-09-13T11:23:01.082 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 10.974819290999999, 2024-09-13T11:23:01.082 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.029535087000000002, 2024-09-13T11:23:01.082 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:23:01.082 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:23:01.082 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:23:01.082 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:23:01.083 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:01.083 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:23:01.083 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:50.098092+0000", 2024-09-13T11:23:01.083 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:01.083 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:01.083 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:01.083 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:23:01.083 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:50.098092+0000", 2024-09-13T11:23:01.083 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:01.083 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:01.083 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:01.083 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:23:01.084 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:50.098095+0000", 2024-09-13T11:23:01.084 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.08e-06 2024-09-13T11:23:01.084 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:01.084 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:01.084 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:23:01.084 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:50.098102+0000", 2024-09-13T11:23:01.084 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.1300000000000003e-06 2024-09-13T11:23:01.084 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:01.084 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:01.084 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:23:01.084 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:50.127085+0000", 2024-09-13T11:23:01.084 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.028983045999999998 2024-09-13T11:23:01.085 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:01.085 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:01.085 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:23:01.085 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:50.127108+0000", 2024-09-13T11:23:01.085 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.3643e-05 2024-09-13T11:23:01.085 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:01.085 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:01.085 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:23:01.085 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:50.127444+0000", 2024-09-13T11:23:01.085 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00033556800000000001 2024-09-13T11:23:01.086 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:01.086 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:01.086 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:23:01.086 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:50.127455+0000", 2024-09-13T11:23:01.086 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1316e-05 2024-09-13T11:23:01.086 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:01.086 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:01.086 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:23:01.086 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:50.127628+0000", 2024-09-13T11:23:01.086 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000172304 2024-09-13T11:23:01.086 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:01.086 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:23:01.087 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:01.087 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:01.087 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:01.087 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:2015 3.3 3:c1c42616:::benchmark_data_smithi177_31116_object2014:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-09-13T11:23:01.087 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:22:50.158555+0000", 2024-09-13T11:23:01.087 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 10.914357116, 2024-09-13T11:23:01.087 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.025814133999999999, 2024-09-13T11:23:01.087 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:23:01.087 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:23:01.087 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:01.087 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:23:01.087 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:23:01.088 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:01.088 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 2015 2024-09-13T11:23:01.088 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:01.088 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:23:01.088 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:01.088 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:23:01.088 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:50.158555+0000", 2024-09-13T11:23:01.088 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:01.088 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:01.088 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:01.088 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:23:01.088 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:50.158555+0000", 2024-09-13T11:23:01.088 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:01.089 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:01.089 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:01.089 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:23:01.089 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:50.158556+0000", 2024-09-13T11:23:01.089 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.796e-06 2024-09-13T11:23:01.089 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:01.089 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:01.089 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:23:01.089 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:50.158605+0000", 2024-09-13T11:23:01.089 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.8619999999999999e-05 2024-09-13T11:23:01.089 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:01.089 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:01.090 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:23:01.090 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:50.158617+0000", 2024-09-13T11:23:01.090 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1970999999999999e-05 2024-09-13T11:23:01.090 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:01.090 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:01.090 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:23:01.090 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:50.158624+0000", 2024-09-13T11:23:01.090 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.2169999999999997e-06 2024-09-13T11:23:01.090 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:01.091 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:01.091 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:23:01.091 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:50.158697+0000", 2024-09-13T11:23:01.091 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.2896000000000001e-05 2024-09-13T11:23:01.091 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:01.091 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:01.092 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:23:01.092 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:50.158828+0000", 2024-09-13T11:23:01.092 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000131058 2024-09-13T11:23:01.092 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:01.092 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:01.093 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 7", 2024-09-13T11:23:01.093 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:50.158941+0000", 2024-09-13T11:23:01.093 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00011312 2024-09-13T11:23:01.093 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:01.093 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:01.093 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:23:01.093 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:50.161526+0000", 2024-09-13T11:23:01.094 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.002584631 2024-09-13T11:23:01.094 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:01.094 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:01.094 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:01.094 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:50.184269+0000", 2024-09-13T11:23:01.094 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.022742853 2024-09-13T11:23:01.094 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:01.094 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:01.095 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:23:01.095 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:50.184330+0000", 2024-09-13T11:23:01.095 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.109e-05 2024-09-13T11:23:01.095 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:01.095 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:01.095 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:23:01.096 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:50.184369+0000", 2024-09-13T11:23:01.096 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.8881999999999998e-05 2024-09-13T11:23:01.096 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:01.096 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:23:01.096 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:01.096 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:01.097 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:01.097 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:2082 3.3 3:c7cabb72:::benchmark_data_smithi177_31116_object2081:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-09-13T11:23:01.097 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:22:50.180715+0000", 2024-09-13T11:23:01.097 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 10.892196343, 2024-09-13T11:23:01.097 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.027514294000000002, 2024-09-13T11:23:01.098 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:23:01.098 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:23:01.098 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:01.098 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:23:01.098 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:23:01.099 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:01.099 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 2082 2024-09-13T11:23:01.099 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:01.099 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:23:01.099 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:01.099 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:23:01.099 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:50.180715+0000", 2024-09-13T11:23:01.099 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:01.099 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:01.099 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:01.099 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:23:01.099 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:50.180715+0000", 2024-09-13T11:23:01.100 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:01.100 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:01.100 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:01.100 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:23:01.100 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:50.180721+0000", 2024-09-13T11:23:01.100 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.0419999999999999e-06 2024-09-13T11:23:01.100 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:01.100 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:01.100 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:23:01.100 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:50.180774+0000", 2024-09-13T11:23:01.100 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.2332e-05 2024-09-13T11:23:01.100 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:01.100 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:01.101 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:23:01.101 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:50.180790+0000", 2024-09-13T11:23:01.101 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.6025e-05 2024-09-13T11:23:01.101 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:01.101 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:01.101 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:23:01.101 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:50.180800+0000", 2024-09-13T11:23:01.101 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.9879999999999999e-06 2024-09-13T11:23:01.101 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:01.101 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:01.101 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:23:01.101 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:50.180890+0000", 2024-09-13T11:23:01.101 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.0680000000000006e-05 2024-09-13T11:23:01.102 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:01.102 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:01.102 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:23:01.102 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:50.181093+0000", 2024-09-13T11:23:01.102 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00020274400000000001 2024-09-13T11:23:01.102 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:01.102 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:01.102 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 7", 2024-09-13T11:23:01.102 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:50.181316+0000", 2024-09-13T11:23:01.102 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00022227400000000001 2024-09-13T11:23:01.102 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:01.102 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:01.102 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:23:01.103 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:50.182603+0000", 2024-09-13T11:23:01.103 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0012872339999999999 2024-09-13T11:23:01.103 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:01.103 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:01.103 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:01.103 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:50.208113+0000", 2024-09-13T11:23:01.103 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.025510623 2024-09-13T11:23:01.103 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:01.103 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:01.103 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:23:01.103 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:50.208191+0000", 2024-09-13T11:23:01.103 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.7843999999999995e-05 2024-09-13T11:23:01.104 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:01.104 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:01.104 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:23:01.104 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:50.208230+0000", 2024-09-13T11:23:01.104 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.8507999999999998e-05 2024-09-13T11:23:01.104 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:01.104 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:23:01.104 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:01.104 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:01.104 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:01.104 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop_reply(client.4665.0:2082 3.3 e43/30 ondisk, result = 0)", 2024-09-13T11:23:01.104 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:22:50.184167+0000", 2024-09-13T11:23:01.104 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 10.888744864, 2024-09-13T11:23:01.105 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.024069184, 2024-09-13T11:23:01.105 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:23:01.105 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:23:01.105 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:23:01.105 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:23:01.105 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:01.105 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:23:01.105 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:50.184167+0000", 2024-09-13T11:23:01.105 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:01.105 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:01.106 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:01.106 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:23:01.106 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:50.184167+0000", 2024-09-13T11:23:01.106 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:01.106 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:01.106 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:01.106 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:23:01.106 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:50.184169+0000", 2024-09-13T11:23:01.106 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.6500000000000001e-06 2024-09-13T11:23:01.106 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:01.106 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:01.106 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:23:01.106 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:50.184173+0000", 2024-09-13T11:23:01.106 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.0239999999999999e-06 2024-09-13T11:23:01.107 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:01.107 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:01.107 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:23:01.107 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:50.208028+0000", 2024-09-13T11:23:01.107 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.023855343000000001 2024-09-13T11:23:01.107 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:01.107 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:01.107 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:23:01.107 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:50.208034+0000", 2024-09-13T11:23:01.107 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.6790000000000002e-06 2024-09-13T11:23:01.107 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:01.107 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:01.107 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:23:01.107 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:50.208082+0000", 2024-09-13T11:23:01.108 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.8804000000000002e-05 2024-09-13T11:23:01.108 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:01.108 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:01.108 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:23:01.108 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:50.208105+0000", 2024-09-13T11:23:01.108 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.2665000000000001e-05 2024-09-13T11:23:01.108 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:01.108 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:01.108 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:23:01.108 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:50.208236+0000", 2024-09-13T11:23:01.108 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00013101899999999999 2024-09-13T11:23:01.108 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:01.109 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:23:01.109 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:01.109 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:01.109 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:01.109 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:2251 3.3 3:cdfe3214:::benchmark_data_smithi177_31116_object2250:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-09-13T11:23:01.109 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:22:50.241594+0000", 2024-09-13T11:23:01.109 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 10.831317981, 2024-09-13T11:23:01.109 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.033010455000000001, 2024-09-13T11:23:01.109 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:23:01.109 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:23:01.109 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:01.109 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:23:01.109 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:23:01.109 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:01.110 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 2251 2024-09-13T11:23:01.110 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:01.110 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:23:01.110 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:01.110 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:23:01.110 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:50.241594+0000", 2024-09-13T11:23:01.110 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:01.110 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:01.110 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:01.110 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:23:01.110 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:50.241594+0000", 2024-09-13T11:23:01.110 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:01.110 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:01.111 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:01.111 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:23:01.111 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:50.241596+0000", 2024-09-13T11:23:01.111 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.1179999999999999e-06 2024-09-13T11:23:01.111 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:01.111 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:01.111 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:23:01.111 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:50.241668+0000", 2024-09-13T11:23:01.111 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.2241000000000002e-05 2024-09-13T11:23:01.111 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:01.111 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:01.111 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:23:01.111 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:50.241685+0000", 2024-09-13T11:23:01.112 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.6829000000000001e-05 2024-09-13T11:23:01.112 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:01.112 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:01.112 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:23:01.112 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:50.241694+0000", 2024-09-13T11:23:01.112 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.8140000000000007e-06 2024-09-13T11:23:01.112 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:01.112 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:01.112 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:23:01.112 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:50.241804+0000", 2024-09-13T11:23:01.112 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000110481 2024-09-13T11:23:01.112 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:01.112 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:01.113 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:23:01.113 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:50.242059+0000", 2024-09-13T11:23:01.113 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00025521499999999999 2024-09-13T11:23:01.113 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:01.113 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:01.113 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 7", 2024-09-13T11:23:01.113 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:50.242367+0000", 2024-09-13T11:23:01.113 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00030775000000000003 2024-09-13T11:23:01.113 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:01.113 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:01.113 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:23:01.113 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:50.243771+0000", 2024-09-13T11:23:01.113 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001403877 2024-09-13T11:23:01.114 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:01.114 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:01.114 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:01.114 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:50.274487+0000", 2024-09-13T11:23:01.114 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.030716291999999999 2024-09-13T11:23:01.114 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:01.114 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:01.114 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:23:01.114 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:50.274552+0000", 2024-09-13T11:23:01.114 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.4818000000000002e-05 2024-09-13T11:23:01.114 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:01.114 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:01.114 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:23:01.115 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:50.274604+0000", 2024-09-13T11:23:01.115 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.202e-05 2024-09-13T11:23:01.115 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:01.115 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:23:01.115 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:01.115 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:01.115 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:01.115 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop_reply(client.4665.0:2251 3.3 e43/30 ondisk, result = 0)", 2024-09-13T11:23:01.115 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:22:50.245239+0000", 2024-09-13T11:23:01.115 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 10.82767247, 2024-09-13T11:23:01.115 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.029408732999999999, 2024-09-13T11:23:01.115 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:23:01.115 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:23:01.116 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:23:01.116 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:23:01.116 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:01.116 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:23:01.116 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:50.245239+0000", 2024-09-13T11:23:01.116 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:01.116 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:01.116 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:01.116 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:23:01.116 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:50.245239+0000", 2024-09-13T11:23:01.116 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:01.116 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:01.117 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:01.117 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:23:01.117 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:50.245241+0000", 2024-09-13T11:23:01.117 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.7990000000000001e-06 2024-09-13T11:23:01.117 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:01.117 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:01.117 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:23:01.117 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:50.245250+0000", 2024-09-13T11:23:01.117 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.1460000000000002e-06 2024-09-13T11:23:01.117 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:01.117 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:01.117 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:23:01.117 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:50.270892+0000", 2024-09-13T11:23:01.118 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.025641740999999999 2024-09-13T11:23:01.118 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:01.118 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:01.118 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:23:01.118 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:50.270953+0000", 2024-09-13T11:23:01.118 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.1307000000000001e-05 2024-09-13T11:23:01.118 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:01.118 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:01.118 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:23:01.118 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:50.274471+0000", 2024-09-13T11:23:01.118 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0035181129999999998 2024-09-13T11:23:01.118 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:01.118 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:01.119 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:23:01.119 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:50.274481+0000", 2024-09-13T11:23:01.119 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.5799999999999998e-06 2024-09-13T11:23:01.119 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:01.119 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:01.119 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:23:01.119 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:50.274648+0000", 2024-09-13T11:23:01.119 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000167047 2024-09-13T11:23:01.119 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:01.120 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:23:01.120 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:01.120 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:01.120 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:01.120 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:2435 3.3 3:ca9ec8bd:::benchmark_data_smithi177_31116_object2434:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-09-13T11:23:01.120 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:22:50.323974+0000", 2024-09-13T11:23:01.120 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 10.748937648, 2024-09-13T11:23:01.120 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.024100244999999999, 2024-09-13T11:23:01.120 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:23:01.120 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:23:01.121 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:01.121 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:23:01.121 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:23:01.121 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:01.121 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 2435 2024-09-13T11:23:01.121 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:01.122 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:23:01.122 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:01.122 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:23:01.122 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:50.323974+0000", 2024-09-13T11:23:01.122 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:01.122 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:01.122 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:01.123 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:23:01.123 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:50.323974+0000", 2024-09-13T11:23:01.123 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:01.123 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:01.123 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:01.123 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:23:01.124 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:50.323981+0000", 2024-09-13T11:23:01.124 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.1600000000000001e-06 2024-09-13T11:23:01.124 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:01.124 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:01.124 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:23:01.124 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:50.324033+0000", 2024-09-13T11:23:01.124 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.1464000000000001e-05 2024-09-13T11:23:01.124 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:01.124 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:01.124 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:23:01.125 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:50.324053+0000", 2024-09-13T11:23:01.125 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.9973e-05 2024-09-13T11:23:01.125 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:01.125 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:01.125 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:23:01.125 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:50.324062+0000", 2024-09-13T11:23:01.125 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.6199999999999994e-06 2024-09-13T11:23:01.125 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:01.125 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:01.126 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:23:01.126 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:50.324152+0000", 2024-09-13T11:23:01.126 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.9169e-05 2024-09-13T11:23:01.126 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:01.126 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:01.126 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op must wait for map", 2024-09-13T11:23:01.126 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:50.324161+0000", 2024-09-13T11:23:01.126 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.6010000000000001e-06 2024-09-13T11:23:01.127 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:01.127 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:01.127 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:23:01.127 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:50.330438+0000", 2024-09-13T11:23:01.127 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0062771900000000002 2024-09-13T11:23:01.127 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:01.127 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:01.127 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:23:01.127 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:50.330653+0000", 2024-09-13T11:23:01.128 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000214255 2024-09-13T11:23:01.128 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:01.128 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:01.128 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 7", 2024-09-13T11:23:01.128 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:50.330877+0000", 2024-09-13T11:23:01.128 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00022451099999999999 2024-09-13T11:23:01.128 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:01.129 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:01.129 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:23:01.129 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:50.332419+0000", 2024-09-13T11:23:01.129 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0015419279999999999 2024-09-13T11:23:01.129 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:01.129 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:01.129 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:01.129 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:50.347910+0000", 2024-09-13T11:23:01.130 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.015490947999999999 2024-09-13T11:23:01.130 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:01.130 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:01.130 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:23:01.130 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:50.348016+0000", 2024-09-13T11:23:01.130 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00010572400000000001 2024-09-13T11:23:01.130 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:01.130 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:01.131 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:23:01.131 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:50.348074+0000", 2024-09-13T11:23:01.131 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.8702000000000001e-05 2024-09-13T11:23:01.131 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:01.131 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:23:01.131 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:01.131 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:01.132 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:01.132 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop_reply(client.4665.0:7655 3.3 e46/30 ondisk, result = 0)", 2024-09-13T11:23:01.132 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:22:53.009318+0000", 2024-09-13T11:23:01.132 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 8.0635939140000001, 2024-09-13T11:23:01.132 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.037971667000000001, 2024-09-13T11:23:01.132 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:23:01.132 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:23:01.132 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:23:01.133 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:23:01.133 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:01.133 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:23:01.133 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:53.009318+0000", 2024-09-13T11:23:01.133 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:01.133 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:01.133 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:01.133 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:23:01.133 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:53.009318+0000", 2024-09-13T11:23:01.134 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:01.134 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:01.134 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:01.134 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:23:01.134 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:53.009321+0000", 2024-09-13T11:23:01.134 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.4850000000000001e-06 2024-09-13T11:23:01.134 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:01.134 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:01.135 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:23:01.135 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:53.009331+0000", 2024-09-13T11:23:01.135 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.4939999999999992e-06 2024-09-13T11:23:01.135 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:01.135 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:01.135 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:23:01.135 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:53.046792+0000", 2024-09-13T11:23:01.135 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.037460836999999997 2024-09-13T11:23:01.136 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:01.136 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:01.136 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:23:01.136 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:53.046809+0000", 2024-09-13T11:23:01.136 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.6965000000000001e-05 2024-09-13T11:23:01.136 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:01.136 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:01.136 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:23:01.137 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:53.046973+0000", 2024-09-13T11:23:01.137 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000164585 2024-09-13T11:23:01.137 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:01.137 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:01.137 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:23:01.137 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:53.046995+0000", 2024-09-13T11:23:01.137 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.1529e-05 2024-09-13T11:23:01.137 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:01.138 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:01.138 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:23:01.138 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:53.047290+0000", 2024-09-13T11:23:01.138 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000294772 2024-09-13T11:23:01.138 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:01.138 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:23:01.138 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:01.138 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:01.138 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:01.139 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:11801 3.3 3:cae0ce13:::benchmark_data_smithi177_31116_object11800:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-13T11:23:01.139 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:22:55.643405+0000", 2024-09-13T11:23:01.139 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 5.4295062969999996, 2024-09-13T11:23:01.139 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.035702266000000003, 2024-09-13T11:23:01.139 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:23:01.139 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:23:01.139 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:01.139 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:23:01.139 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:23:01.139 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:01.139 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 11801 2024-09-13T11:23:01.140 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:01.140 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:23:01.140 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:01.140 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:23:01.140 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:55.643405+0000", 2024-09-13T11:23:01.140 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:01.140 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:01.140 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:01.140 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:23:01.140 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:55.643405+0000", 2024-09-13T11:23:01.141 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:01.141 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:01.141 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:01.141 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:23:01.141 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:55.643416+0000", 2024-09-13T11:23:01.141 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0054999999999999e-05 2024-09-13T11:23:01.141 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:01.141 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:01.141 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:23:01.141 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:55.643466+0000", 2024-09-13T11:23:01.141 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.0408999999999998e-05 2024-09-13T11:23:01.142 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:01.142 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:01.142 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:23:01.142 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:55.643490+0000", 2024-09-13T11:23:01.142 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.4128999999999999e-05 2024-09-13T11:23:01.142 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:01.142 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:01.143 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:23:01.143 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:55.643502+0000", 2024-09-13T11:23:01.143 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.149e-05 2024-09-13T11:23:01.143 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:01.143 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:01.143 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:23:01.143 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:55.643624+0000", 2024-09-13T11:23:01.143 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00012277300000000001 2024-09-13T11:23:01.143 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:01.143 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:01.143 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:23:01.143 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:55.643915+0000", 2024-09-13T11:23:01.143 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000290754 2024-09-13T11:23:01.144 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:01.144 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:01.144 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 7", 2024-09-13T11:23:01.144 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:55.644248+0000", 2024-09-13T11:23:01.144 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00033311099999999998 2024-09-13T11:23:01.144 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:01.144 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:01.144 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:23:01.144 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:55.645823+0000", 2024-09-13T11:23:01.144 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0015748369999999999 2024-09-13T11:23:01.144 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:01.144 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:01.144 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:01.145 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:55.678842+0000", 2024-09-13T11:23:01.145 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.033018919000000001 2024-09-13T11:23:01.145 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:01.145 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:01.145 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:23:01.145 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:55.679013+0000", 2024-09-13T11:23:01.145 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00017098600000000001 2024-09-13T11:23:01.145 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:01.145 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:01.145 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:23:01.145 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:55.679108+0000", 2024-09-13T11:23:01.145 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.4802999999999993e-05 2024-09-13T11:23:01.146 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:01.146 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:23:01.146 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:01.147 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:01.147 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:01.147 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop_reply(client.4665.0:15910 3.3 e52/30 ondisk, result = 0)", 2024-09-13T11:23:01.147 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:22:58.421847+0000", 2024-09-13T11:23:01.147 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 2.6510644839999999, 2024-09-13T11:23:01.147 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.040755588000000002, 2024-09-13T11:23:01.147 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:23:01.147 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:23:01.147 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:23:01.147 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:23:01.147 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:01.147 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:23:01.148 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:58.421847+0000", 2024-09-13T11:23:01.148 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:01.148 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:01.148 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:01.148 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:23:01.148 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:58.421847+0000", 2024-09-13T11:23:01.148 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:01.148 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:01.148 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:01.148 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:23:01.148 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:58.421850+0000", 2024-09-13T11:23:01.148 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.8289999999999999e-06 2024-09-13T11:23:01.148 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:01.149 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:01.149 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:23:01.149 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:58.421859+0000", 2024-09-13T11:23:01.149 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.2650000000000002e-06 2024-09-13T11:23:01.149 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:01.149 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:01.149 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:23:01.149 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:58.460051+0000", 2024-09-13T11:23:01.149 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.038192049999999998 2024-09-13T11:23:01.149 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:01.149 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:01.149 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:23:01.149 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:58.460085+0000", 2024-09-13T11:23:01.149 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.3318999999999997e-05 2024-09-13T11:23:01.150 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:01.150 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:01.150 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:23:01.150 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:58.462416+0000", 2024-09-13T11:23:01.150 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0023308729999999998 2024-09-13T11:23:01.150 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:01.150 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:01.150 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:23:01.150 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:58.462425+0000", 2024-09-13T11:23:01.150 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.9250000000000001e-06 2024-09-13T11:23:01.150 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:01.150 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:01.150 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:23:01.151 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:58.462603+0000", 2024-09-13T11:23:01.151 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00017832699999999999 2024-09-13T11:23:01.151 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:01.151 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:23:01.151 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:01.151 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:01.151 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:01.151 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:15989 3.3 3:c09b4b9d:::benchmark_data_smithi177_31116_object15988:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-13T11:23:01.151 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:22:58.466255+0000", 2024-09-13T11:23:01.151 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 2.6066570110000002, 2024-09-13T11:23:01.151 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.024894293000000001, 2024-09-13T11:23:01.151 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:23:01.151 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:23:01.152 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:01.152 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:23:01.152 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:23:01.152 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:01.152 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 15989 2024-09-13T11:23:01.152 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:01.152 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:23:01.152 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:01.152 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:23:01.152 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:58.466255+0000", 2024-09-13T11:23:01.152 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:01.152 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:01.152 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:01.153 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:23:01.153 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:58.466255+0000", 2024-09-13T11:23:01.153 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:01.153 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:01.153 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:01.153 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:23:01.153 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:58.466258+0000", 2024-09-13T11:23:01.153 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.92e-06 2024-09-13T11:23:01.153 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:01.153 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:01.154 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:23:01.154 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:58.466321+0000", 2024-09-13T11:23:01.154 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.2883000000000001e-05 2024-09-13T11:23:01.154 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:01.154 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:01.154 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:23:01.154 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:58.466346+0000", 2024-09-13T11:23:01.155 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.5772999999999999e-05 2024-09-13T11:23:01.155 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:01.155 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:01.155 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:23:01.155 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:58.466361+0000", 2024-09-13T11:23:01.155 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4944e-05 2024-09-13T11:23:01.155 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:01.155 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:01.155 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:23:01.155 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:58.466521+0000", 2024-09-13T11:23:01.155 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00015927899999999999 2024-09-13T11:23:01.155 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:01.156 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:01.156 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:23:01.156 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:58.466790+0000", 2024-09-13T11:23:01.156 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00026906800000000002 2024-09-13T11:23:01.156 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:01.156 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:01.156 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 7", 2024-09-13T11:23:01.156 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:58.467019+0000", 2024-09-13T11:23:01.156 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00022986299999999999 2024-09-13T11:23:01.156 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:01.156 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:01.156 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:23:01.156 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:58.468845+0000", 2024-09-13T11:23:01.157 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0018255000000000001 2024-09-13T11:23:01.157 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:01.157 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:01.157 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:01.157 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:58.491012+0000", 2024-09-13T11:23:01.157 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.022166844000000002 2024-09-13T11:23:01.157 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:01.157 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:01.157 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:23:01.157 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:58.491089+0000", 2024-09-13T11:23:01.157 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.6790000000000002e-05 2024-09-13T11:23:01.157 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:01.157 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:01.158 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:23:01.158 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:58.491149+0000", 2024-09-13T11:23:01.158 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.0429e-05 2024-09-13T11:23:01.158 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:01.158 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:23:01.158 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:01.158 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:01.158 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:01.158 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:16026 3.3 3:c6ae1378:::benchmark_data_smithi177_31116_object16025:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-13T11:23:01.158 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:22:58.488003+0000", 2024-09-13T11:23:01.158 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 2.5849091880000001, 2024-09-13T11:23:01.158 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.025525003000000001, 2024-09-13T11:23:01.158 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:23:01.158 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:23:01.159 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:01.159 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:23:01.159 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:23:01.159 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:01.159 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 16026 2024-09-13T11:23:01.159 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:01.159 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:23:01.159 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:01.159 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:23:01.159 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:58.488003+0000", 2024-09-13T11:23:01.159 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:01.159 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:01.159 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:01.160 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:23:01.160 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:58.488003+0000", 2024-09-13T11:23:01.160 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:01.160 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:01.160 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:01.160 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:23:01.160 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:58.488006+0000", 2024-09-13T11:23:01.160 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.36e-06 2024-09-13T11:23:01.160 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:01.160 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:01.160 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:23:01.160 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:58.488079+0000", 2024-09-13T11:23:01.160 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.2704000000000001e-05 2024-09-13T11:23:01.161 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:01.161 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:01.161 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:23:01.161 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:58.488109+0000", 2024-09-13T11:23:01.161 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.0485e-05 2024-09-13T11:23:01.161 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:01.161 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:01.161 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:23:01.161 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:58.488131+0000", 2024-09-13T11:23:01.161 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.1673999999999999e-05 2024-09-13T11:23:01.161 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:01.161 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:01.161 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:23:01.161 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:58.488229+0000", 2024-09-13T11:23:01.162 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.7904000000000003e-05 2024-09-13T11:23:01.162 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:01.162 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:01.162 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:23:01.162 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:58.488425+0000", 2024-09-13T11:23:01.162 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00019670500000000001 2024-09-13T11:23:01.163 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:01.163 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:01.163 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 7", 2024-09-13T11:23:01.163 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:58.488626+0000", 2024-09-13T11:23:01.163 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00020016899999999999 2024-09-13T11:23:01.164 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:01.164 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:01.164 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:23:01.164 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:58.490163+0000", 2024-09-13T11:23:01.164 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001537891 2024-09-13T11:23:01.165 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:01.165 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:01.165 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:01.165 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:58.513323+0000", 2024-09-13T11:23:01.165 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.023159664 2024-09-13T11:23:01.166 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:01.166 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:01.166 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:23:01.166 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:58.513440+0000", 2024-09-13T11:23:01.166 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000116514 2024-09-13T11:23:01.166 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:01.166 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:01.166 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:23:01.167 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:58.513528+0000", 2024-09-13T11:23:01.167 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.7933e-05 2024-09-13T11:23:01.167 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:01.167 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:23:01.167 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:01.167 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:01.167 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:01.167 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:16182 3.3 3:cb9b6b32:::benchmark_data_smithi177_31116_object16181:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-13T11:23:01.167 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:22:58.570069+0000", 2024-09-13T11:23:01.167 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 2.5028430830000001, 2024-09-13T11:23:01.167 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.026861664, 2024-09-13T11:23:01.167 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:23:01.167 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:23:01.168 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:01.168 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:23:01.168 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:23:01.168 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:01.168 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 16182 2024-09-13T11:23:01.168 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:01.168 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:23:01.168 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:01.169 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:23:01.169 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:58.570069+0000", 2024-09-13T11:23:01.169 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:01.169 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:01.169 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:01.169 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:23:01.169 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:58.570069+0000", 2024-09-13T11:23:01.169 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:01.169 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:01.169 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:01.169 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:23:01.169 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:58.570072+0000", 2024-09-13T11:23:01.169 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.0479999999999999e-06 2024-09-13T11:23:01.170 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:01.170 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:01.170 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:23:01.170 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:58.570123+0000", 2024-09-13T11:23:01.170 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.0770999999999999e-05 2024-09-13T11:23:01.170 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:01.170 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:01.170 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:23:01.170 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:58.570147+0000", 2024-09-13T11:23:01.170 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.4539e-05 2024-09-13T11:23:01.170 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:01.170 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:01.170 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:23:01.171 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:58.570164+0000", 2024-09-13T11:23:01.171 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.679e-05 2024-09-13T11:23:01.171 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:01.171 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:01.171 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:23:01.171 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:58.570305+0000", 2024-09-13T11:23:01.171 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000140853 2024-09-13T11:23:01.171 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:01.171 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:01.172 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:23:01.172 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:58.570583+0000", 2024-09-13T11:23:01.172 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00027865700000000003 2024-09-13T11:23:01.172 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:01.172 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:01.172 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 7", 2024-09-13T11:23:01.172 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:58.570971+0000", 2024-09-13T11:23:01.172 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00038734199999999999 2024-09-13T11:23:01.172 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:01.172 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:01.172 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:23:01.172 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:58.573679+0000", 2024-09-13T11:23:01.173 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0027080709999999998 2024-09-13T11:23:01.173 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:01.173 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:01.173 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:01.173 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:58.596816+0000", 2024-09-13T11:23:01.173 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.023137548000000001 2024-09-13T11:23:01.173 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:01.173 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:01.173 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:23:01.173 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:58.596879+0000", 2024-09-13T11:23:01.173 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.2403999999999994e-05 2024-09-13T11:23:01.173 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:01.173 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:01.174 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:23:01.174 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:58.596930+0000", 2024-09-13T11:23:01.174 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.1641000000000002e-05 2024-09-13T11:23:01.174 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:01.174 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:23:01.174 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:01.174 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:01.174 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:01.174 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:16223 3.3 3:c30b4ac3:::benchmark_data_smithi177_31116_object16222:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-13T11:23:01.174 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:22:58.594235+0000", 2024-09-13T11:23:01.174 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 2.478676358, 2024-09-13T11:23:01.174 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.024696195000000001, 2024-09-13T11:23:01.174 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:23:01.174 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:23:01.175 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:01.175 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:23:01.175 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:23:01.175 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:01.175 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 16223 2024-09-13T11:23:01.175 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:01.175 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:23:01.175 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:01.175 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:23:01.175 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:58.594235+0000", 2024-09-13T11:23:01.175 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:01.176 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:01.176 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:01.176 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:23:01.176 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:58.594235+0000", 2024-09-13T11:23:01.176 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:01.176 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:01.176 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:01.176 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:23:01.176 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:58.594238+0000", 2024-09-13T11:23:01.176 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.83e-06 2024-09-13T11:23:01.176 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:01.177 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:01.177 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:23:01.177 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:58.594287+0000", 2024-09-13T11:23:01.177 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.8381999999999999e-05 2024-09-13T11:23:01.177 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:01.177 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:01.177 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:23:01.177 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:58.594316+0000", 2024-09-13T11:23:01.177 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.8928000000000002e-05 2024-09-13T11:23:01.177 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:01.177 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:01.177 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:23:01.177 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:58.594327+0000", 2024-09-13T11:23:01.178 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1236999999999999e-05 2024-09-13T11:23:01.178 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:01.178 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:01.178 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:23:01.178 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:58.595275+0000", 2024-09-13T11:23:01.178 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00094867499999999995 2024-09-13T11:23:01.178 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:01.178 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:01.178 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:23:01.178 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:58.595442+0000", 2024-09-13T11:23:01.178 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000166556 2024-09-13T11:23:01.178 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:01.178 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:01.179 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 7", 2024-09-13T11:23:01.179 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:58.595599+0000", 2024-09-13T11:23:01.179 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00015734399999999999 2024-09-13T11:23:01.179 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:01.179 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:01.179 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:23:01.179 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:58.599105+0000", 2024-09-13T11:23:01.179 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0035061139999999998 2024-09-13T11:23:01.179 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:01.179 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:01.179 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:01.179 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:58.618736+0000", 2024-09-13T11:23:01.179 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.019630904000000001 2024-09-13T11:23:01.179 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:01.180 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:01.180 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:23:01.180 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:58.618832+0000", 2024-09-13T11:23:01.180 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.5492e-05 2024-09-13T11:23:01.180 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:01.180 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:01.180 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:23:01.180 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:58.618932+0000", 2024-09-13T11:23:01.180 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.9733000000000002e-05 2024-09-13T11:23:01.180 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:01.180 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:23:01.180 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:01.180 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:01.181 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:01.181 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:16292 3.3 3:cb66b543:::benchmark_data_smithi177_31116_object16291:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-13T11:23:01.181 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:22:58.632124+0000", 2024-09-13T11:23:01.181 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 2.4407875749999999, 2024-09-13T11:23:01.181 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.025034259999999999, 2024-09-13T11:23:01.181 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:23:01.181 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:23:01.181 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:01.181 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:23:01.181 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:23:01.181 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:01.181 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 16292 2024-09-13T11:23:01.181 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:01.182 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:23:01.182 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:01.182 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:23:01.182 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:58.632124+0000", 2024-09-13T11:23:01.182 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:01.182 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:01.182 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:01.182 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:23:01.182 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:58.632124+0000", 2024-09-13T11:23:01.182 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:01.182 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:01.182 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:01.182 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:23:01.182 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:58.632128+0000", 2024-09-13T11:23:01.183 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.7720000000000001e-06 2024-09-13T11:23:01.183 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:01.183 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:01.183 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:23:01.183 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:58.632193+0000", 2024-09-13T11:23:01.183 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.5203999999999994e-05 2024-09-13T11:23:01.183 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:01.183 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:01.183 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:23:01.183 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:58.632223+0000", 2024-09-13T11:23:01.183 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.9751e-05 2024-09-13T11:23:01.184 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:01.184 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:01.184 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:23:01.184 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:58.632242+0000", 2024-09-13T11:23:01.184 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.9045000000000001e-05 2024-09-13T11:23:01.184 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:01.184 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:01.184 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:23:01.184 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:58.632488+0000", 2024-09-13T11:23:01.184 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00024573599999999998 2024-09-13T11:23:01.184 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:01.184 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:01.184 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:23:01.185 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:58.632702+0000", 2024-09-13T11:23:01.185 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00021425799999999999 2024-09-13T11:23:01.185 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:01.185 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:01.185 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 7", 2024-09-13T11:23:01.185 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:58.632880+0000", 2024-09-13T11:23:01.185 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000177799 2024-09-13T11:23:01.185 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:01.185 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:01.185 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:23:01.185 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:58.634455+0000", 2024-09-13T11:23:01.185 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0015756699999999999 2024-09-13T11:23:01.185 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:01.185 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:01.185 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:01.186 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:58.656959+0000", 2024-09-13T11:23:01.186 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.022503690999999999 2024-09-13T11:23:01.186 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:01.188 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:01.188 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:23:01.188 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:58.657080+0000", 2024-09-13T11:23:01.188 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000120594 2024-09-13T11:23:01.188 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:01.188 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:01.188 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:23:01.188 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:58.657158+0000", 2024-09-13T11:23:01.188 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.8739999999999995e-05 2024-09-13T11:23:01.188 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:01.189 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:23:01.189 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:01.189 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:01.189 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:01.189 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:20167 3.3 3:cca39706:::benchmark_data_smithi177_31116_object20166:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e54)", 2024-09-13T11:23:01.189 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:00.791331+0000", 2024-09-13T11:23:01.189 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 0.28158109399999998, 2024-09-13T11:23:01.189 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.024118292999999999, 2024-09-13T11:23:01.189 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:23:01.189 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:23:01.189 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:01.189 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:23:01.189 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:23:01.189 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:01.190 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 20167 2024-09-13T11:23:01.190 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:01.190 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:23:01.190 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:01.190 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:23:01.190 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:00.791331+0000", 2024-09-13T11:23:01.190 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:01.190 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:01.190 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:01.190 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:23:01.190 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:00.791331+0000", 2024-09-13T11:23:01.190 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:01.190 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:01.190 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:01.191 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:23:01.191 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:00.791334+0000", 2024-09-13T11:23:01.191 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.9960000000000002e-06 2024-09-13T11:23:01.191 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:01.191 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:01.191 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:23:01.191 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:00.791388+0000", 2024-09-13T11:23:01.191 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.4255999999999997e-05 2024-09-13T11:23:01.191 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:01.191 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:01.191 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:23:01.191 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:00.791411+0000", 2024-09-13T11:23:01.191 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.3373000000000001e-05 2024-09-13T11:23:01.191 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:01.192 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:01.192 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:23:01.192 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:00.791423+0000", 2024-09-13T11:23:01.192 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1839e-05 2024-09-13T11:23:01.192 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:01.192 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:01.192 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:23:01.192 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:00.791530+0000", 2024-09-13T11:23:01.192 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000107042 2024-09-13T11:23:01.192 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:01.192 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:01.192 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:23:01.192 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:00.791676+0000", 2024-09-13T11:23:01.192 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00014558600000000001 2024-09-13T11:23:01.193 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:01.193 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:01.193 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 7", 2024-09-13T11:23:01.193 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:00.791795+0000", 2024-09-13T11:23:01.193 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000118739 2024-09-13T11:23:01.193 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:01.193 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:01.193 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:23:01.193 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:00.792855+0000", 2024-09-13T11:23:01.193 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001060526 2024-09-13T11:23:01.193 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:01.193 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:01.193 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:01.193 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:00.815369+0000", 2024-09-13T11:23:01.193 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.022513532999999999 2024-09-13T11:23:01.194 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:01.194 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:01.194 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:23:01.194 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:00.815411+0000", 2024-09-13T11:23:01.194 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.2376000000000002e-05 2024-09-13T11:23:01.194 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:01.194 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:01.194 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:23:01.194 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:00.815449+0000", 2024-09-13T11:23:01.194 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.8027e-05 2024-09-13T11:23:01.194 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:01.194 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:23:01.194 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:01.195 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:01.195 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:01.195 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:20309 3.3 3:caddc263:::benchmark_data_smithi177_31116_object20308:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e54)", 2024-09-13T11:23:01.195 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:00.847841+0000", 2024-09-13T11:23:01.195 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 0.225070457, 2024-09-13T11:23:01.195 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.024822764000000001, 2024-09-13T11:23:01.195 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:23:01.195 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:23:01.195 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:01.195 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:23:01.195 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:23:01.195 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:01.195 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 20309 2024-09-13T11:23:01.196 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:01.196 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:23:01.196 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:01.196 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:23:01.196 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:00.847841+0000", 2024-09-13T11:23:01.196 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:01.196 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:01.196 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:01.196 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:23:01.196 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:00.847841+0000", 2024-09-13T11:23:01.196 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:01.196 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:01.196 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:01.196 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:23:01.197 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:00.847843+0000", 2024-09-13T11:23:01.197 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.474e-06 2024-09-13T11:23:01.197 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:01.197 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:01.197 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:23:01.197 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:00.847952+0000", 2024-09-13T11:23:01.197 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00010891 2024-09-13T11:23:01.197 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:01.197 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:01.197 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:23:01.197 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:00.847963+0000", 2024-09-13T11:23:01.197 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1626e-05 2024-09-13T11:23:01.197 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:01.197 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:01.197 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:23:01.198 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:00.847969+0000", 2024-09-13T11:23:01.198 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.9460000000000003e-06 2024-09-13T11:23:01.198 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:01.198 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:01.198 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:23:01.198 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:00.848072+0000", 2024-09-13T11:23:01.198 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00010264000000000001 2024-09-13T11:23:01.198 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:01.198 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:01.198 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:23:01.198 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:00.848153+0000", 2024-09-13T11:23:01.198 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.0939999999999994e-05 2024-09-13T11:23:01.198 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:01.198 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:01.199 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 7", 2024-09-13T11:23:01.199 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:00.848239+0000", 2024-09-13T11:23:01.199 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.6059999999999999e-05 2024-09-13T11:23:01.199 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:01.199 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:01.199 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:23:01.199 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:00.851261+0000", 2024-09-13T11:23:01.199 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0030222000000000001 2024-09-13T11:23:01.199 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:01.199 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:01.199 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:01.199 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:00.872559+0000", 2024-09-13T11:23:01.199 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.021298379999999999 2024-09-13T11:23:01.199 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:01.200 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:01.200 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:23:01.200 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:00.872619+0000", 2024-09-13T11:23:01.200 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.9299000000000003e-05 2024-09-13T11:23:01.200 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:01.200 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:01.200 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:23:01.200 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:00.872664+0000", 2024-09-13T11:23:01.200 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.5289e-05 2024-09-13T11:23:01.200 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:01.200 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:23:01.200 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:01.200 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:01.201 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:01.201 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:20503 3.3 3:c2cf9bec:::benchmark_data_smithi177_31116_object20502:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e54)", 2024-09-13T11:23:01.201 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:00.934025+0000", 2024-09-13T11:23:01.201 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 0.138886969, 2024-09-13T11:23:01.201 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.040488244999999999, 2024-09-13T11:23:01.201 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:23:01.201 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:23:01.201 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:01.201 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:23:01.201 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:23:01.201 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:01.201 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 20503 2024-09-13T11:23:01.201 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:01.201 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:23:01.202 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:01.202 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:23:01.202 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:00.934025+0000", 2024-09-13T11:23:01.202 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:01.202 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:01.202 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:01.202 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:23:01.202 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:00.934025+0000", 2024-09-13T11:23:01.202 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:01.202 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:01.202 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:01.202 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:23:01.202 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:00.934028+0000", 2024-09-13T11:23:01.202 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.7199999999999998e-06 2024-09-13T11:23:01.203 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:01.203 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:01.203 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:23:01.203 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:00.934065+0000", 2024-09-13T11:23:01.203 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.7199000000000001e-05 2024-09-13T11:23:01.203 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:01.203 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:01.203 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:23:01.203 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:00.934082+0000", 2024-09-13T11:23:01.203 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.7286e-05 2024-09-13T11:23:01.203 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:01.203 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:01.203 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:23:01.203 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:00.934096+0000", 2024-09-13T11:23:01.204 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4429e-05 2024-09-13T11:23:01.204 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:01.204 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:01.204 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:23:01.204 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:00.934178+0000", 2024-09-13T11:23:01.204 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.1284000000000003e-05 2024-09-13T11:23:01.204 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:01.204 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:01.204 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:23:01.204 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:00.934369+0000", 2024-09-13T11:23:01.204 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00019146699999999999 2024-09-13T11:23:01.204 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:01.204 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:01.204 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 7", 2024-09-13T11:23:01.205 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:00.934566+0000", 2024-09-13T11:23:01.205 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000196646 2024-09-13T11:23:01.205 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:01.205 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:01.205 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:23:01.205 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:00.935850+0000", 2024-09-13T11:23:01.205 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0012837930000000001 2024-09-13T11:23:01.205 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:01.205 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:01.205 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:01.205 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:00.974449+0000", 2024-09-13T11:23:01.205 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.038599708000000003 2024-09-13T11:23:01.205 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:01.205 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:01.206 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:23:01.206 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:00.974484+0000", 2024-09-13T11:23:01.206 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.4311999999999999e-05 2024-09-13T11:23:01.206 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:01.206 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:01.206 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:23:01.206 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:00.974513+0000", 2024-09-13T11:23:01.206 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.9400999999999999e-05 2024-09-13T11:23:01.206 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:01.206 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:23:01.206 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:01.206 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:01.206 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:01.206 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop_reply(client.4665.0:20503 3.3 e54/30 ondisk, result = 0)", 2024-09-13T11:23:01.206 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:00.936548+0000", 2024-09-13T11:23:01.207 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 0.13636356999999999, 2024-09-13T11:23:01.207 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.037970861000000002, 2024-09-13T11:23:01.207 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:23:01.207 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:23:01.207 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:23:01.207 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:23:01.207 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:01.207 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:23:01.207 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:00.936548+0000", 2024-09-13T11:23:01.207 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:01.207 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:01.207 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:01.207 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:23:01.207 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:00.936548+0000", 2024-09-13T11:23:01.208 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:01.208 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:01.208 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:01.208 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:23:01.208 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:00.936550+0000", 2024-09-13T11:23:01.208 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3090000000000001e-06 2024-09-13T11:23:01.208 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:01.208 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:01.208 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:23:01.208 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:00.936554+0000", 2024-09-13T11:23:01.208 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.2540000000000003e-06 2024-09-13T11:23:01.208 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:01.208 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:01.208 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:23:01.209 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:00.974382+0000", 2024-09-13T11:23:01.209 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.037828529 2024-09-13T11:23:01.209 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:01.209 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:01.209 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:23:01.209 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:00.974389+0000", 2024-09-13T11:23:01.209 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.5189999999999998e-06 2024-09-13T11:23:01.209 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:01.209 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:01.209 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:23:01.209 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:00.974440+0000", 2024-09-13T11:23:01.209 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.1201e-05 2024-09-13T11:23:01.209 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:01.209 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:01.210 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:23:01.210 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:00.974445+0000", 2024-09-13T11:23:01.210 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.6750000000000001e-06 2024-09-13T11:23:01.210 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:01.210 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:01.210 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:23:01.210 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:00.974519+0000", 2024-09-13T11:23:01.210 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.4374000000000003e-05 2024-09-13T11:23:01.210 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:01.210 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:23:01.210 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:01.210 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:01.210 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:23:01.210 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-13T11:23:01.211 INFO:teuthology.orchestra.run.smithi001.stdout:osd.8: {} 2024-09-13T11:23:01.212 INFO:teuthology.orchestra.run.smithi001.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-13T11:23:01.212 INFO:teuthology.orchestra.run.smithi001.stdout:osd.9: {} 2024-09-13T11:23:01.213 INFO:teuthology.orchestra.run.smithi001.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-13T11:23:01.213 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_ops_in_flight 2024-09-13T11:23:01.214 INFO:teuthology.orchestra.run.smithi001.stdout:osd.10: {} 2024-09-13T11:23:01.214 INFO:teuthology.orchestra.run.smithi001.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-13T11:23:01.214 INFO:teuthology.orchestra.run.smithi001.stdout:osd.11: {} 2024-09-13T11:23:01.214 INFO:teuthology.orchestra.run.smithi001.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-13T11:23:01.215 INFO:teuthology.orchestra.run.smithi001.stdout:osd.12: {} 2024-09-13T11:23:01.215 INFO:teuthology.orchestra.run.smithi001.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-13T11:23:01.215 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-13T11:23:01.204+0000 7f6a68775640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26307) UID: 0 2024-09-13T11:23:01.231 INFO:teuthology.orchestra.run.smithi001.stdout:osd.13: {} 2024-09-13T11:23:01.231 INFO:teuthology.orchestra.run.smithi001.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-13T11:23:01.259 INFO:teuthology.orchestra.run.smithi001.stdout:osd.14: {} 2024-09-13T11:23:01.260 INFO:teuthology.orchestra.run.smithi001.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-13T11:23:01.284 INFO:teuthology.orchestra.run.smithi001.stdout:osd.15: {} 2024-09-13T11:23:01.284 INFO:teuthology.orchestra.run.smithi001.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-13T11:23:01.312 INFO:tasks.ceph.osd.11.smithi177.stderr:2024-09-13T11:23:01.311+0000 7f24792c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26580) UID: 0 2024-09-13T11:23:01.352 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-13T11:23:01.353 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:23:01.366 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_blocked_ops 2024-09-13T11:23:01.413 INFO:tasks.ceph.osd.10.smithi162.stderr:2024-09-13T11:23:01.411+0000 7fb59c9f0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26317) UID: 0 2024-09-13T11:23:01.425 INFO:teuthology.orchestra.run.smithi001.stderr:noscrub is set 2024-09-13T11:23:01.432 INFO:teuthology.orchestra.run.smithi001.stderr:reweighted osd.15 to 1 (10000) 2024-09-13T11:23:01.456 INFO:tasks.ceph.ceph_manager.ceph:waiting for recovery to complete 2024-09-13T11:23:01.457 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-13T11:23:01.459 INFO:tasks.ceph.osd.11.smithi177.stderr:2024-09-13T11:23:01.455+0000 7f2469cec640 -1 osd.11 54 handle_osd_map injecting map crc failure 2024-09-13T11:23:01.489 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-13T11:23:01.489 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:23:01.505 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_historic_ops 2024-09-13T11:23:01.514 INFO:tasks.ceph.osd.7.smithi177.stderr:2024-09-13T11:23:01.511+0000 7f897b51d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26575) UID: 0 2024-09-13T11:23:01.614 INFO:tasks.ceph.osd.2.smithi162.stderr:2024-09-13T11:23:01.611+0000 7fe92b35a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26313) UID: 0 2024-09-13T11:23:01.686 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-13T11:23:01.686 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:23:01.703 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-13T11:23:01.748 INFO:tasks.ceph.osd.5.smithi092.stderr:2024-09-13T11:23:01.746+0000 7f6f00132640 -1 osd.5 54 handle_osd_map injecting map crc failure 2024-09-13T11:23:01.750 INFO:tasks.ceph.osd.11.smithi177.stderr:2024-09-13T11:23:01.751+0000 7f24792c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26580) UID: 0 2024-09-13T11:23:01.817 INFO:tasks.ceph.osd.10.smithi162.stderr:2024-09-13T11:23:01.811+0000 7fb59c9f0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26317) UID: 0 2024-09-13T11:23:01.857 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-09-13T11:23:01.857 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:23:01.867 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-13T11:23:01.868 INFO:teuthology.orchestra.run.smithi001.stderr:dumped all 2024-09-13T11:23:01.871 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-13T11:23:01.886 INFO:teuthology.orchestra.run.smithi001.stdout:{"pg_ready":true,"pg_map":{"version":66,"stamp":"2024-09-13T11:23:01.428193+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1179451955,"num_objects":17992,"num_object_clones":0,"num_object_copies":35984,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17992,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":18135,"num_write_kb":1153906,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":18103,"ondisk_log_size":18103,"up":50,"acting":50,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":50,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":15,"kb":1509949440,"kb_used":2798340,"kb_used_data":2265284,"kb_used_omap":27,"kb_used_meta":532772,"kb_avail":1507151100,"statfs":{"total":1546188226560,"available":1543322726400,"internally_reserved":0,"allocated":2319650816,"data_stored":2315855351,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28358,"internal_metadata":545558842},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,0,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":340525056,"num_objects":5196,"num_object_clones":0,"num_object_copies":10392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5196,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5196,"num_write_kb":332544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.032204"},"pg_stats":[{"pgid":"3.f","version":"50'917","reported_seq":966,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-13T11:22:57.013275+0000","last_change":"2024-09-13T11:22:25.925951+0000","last_active":"2024-09-13T11:22:57.013275+0000","last_peered":"2024-09-13T11:22:57.013275+0000","last_clean":"2024-09-13T11:22:57.013275+0000","last_became_active":"2024-09-13T11:22:25.925169+0000","last_became_peered":"2024-09-13T11:22:25.925169+0000","last_unstale":"2024-09-13T11:22:57.013275+0000","last_undegraded":"2024-09-13T11:22:57.013275+0000","last_fullsized":"2024-09-13T11:22:57.013275+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:22:24.896055+0000","objects_scrubbed":0,"log_size":917,"log_dups_size":0,"ondisk_log_size":917,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:23:38.143926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":60096512,"num_objects":917,"num_object_clones":0,"num_object_copies":1834,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":917,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":917,"num_write_kb":58688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,12],"acting":[11,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.e","version":"54'1377","reported_seq":1432,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-13T11:23:01.012203+0000","last_change":"2024-09-13T11:22:25.923829+0000","last_active":"2024-09-13T11:23:01.012203+0000","last_peered":"2024-09-13T11:23:01.012203+0000","last_clean":"2024-09-13T11:23:01.012203+0000","last_became_active":"2024-09-13T11:22:25.923498+0000","last_became_peered":"2024-09-13T11:22:25.923498+0000","last_unstale":"2024-09-13T11:23:01.012203+0000","last_undegraded":"2024-09-13T11:23:01.012203+0000","last_fullsized":"2024-09-13T11:23:01.012203+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:22:24.896055+0000","objects_scrubbed":0,"log_size":1377,"log_dups_size":0,"ondisk_log_size":1377,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:23:50.687438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90243072,"num_objects":1377,"num_object_clones":0,"num_object_copies":2754,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1377,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1377,"num_write_kb":88128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,14],"acting":[8,14],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.d","version":"54'1291","reported_seq":1346,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-13T11:23:01.014481+0000","last_change":"2024-09-13T11:22:25.927325+0000","last_active":"2024-09-13T11:23:01.014481+0000","last_peered":"2024-09-13T11:23:01.014481+0000","last_clean":"2024-09-13T11:23:01.014481+0000","last_became_active":"2024-09-13T11:22:25.926433+0000","last_became_peered":"2024-09-13T11:22:25.926433+0000","last_unstale":"2024-09-13T11:23:01.014481+0000","last_undegraded":"2024-09-13T11:23:01.014481+0000","last_fullsized":"2024-09-13T11:23:01.014481+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:22:24.896055+0000","objects_scrubbed":0,"log_size":1291,"log_dups_size":0,"ondisk_log_size":1291,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:23:39.281487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":84606976,"num_objects":1291,"num_object_clones":0,"num_object_copies":2582,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1291,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1291,"num_write_kb":82624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.c","version":"50'882","reported_seq":931,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-13T11:22:56.989062+0000","last_change":"2024-09-13T11:22:25.934619+0000","last_active":"2024-09-13T11:22:56.989062+0000","last_peered":"2024-09-13T11:22:56.989062+0000","last_clean":"2024-09-13T11:22:56.989062+0000","last_became_active":"2024-09-13T11:22:25.933854+0000","last_became_peered":"2024-09-13T11:22:25.933854+0000","last_unstale":"2024-09-13T11:22:56.989062+0000","last_undegraded":"2024-09-13T11:22:56.989062+0000","last_fullsized":"2024-09-13T11:22:56.989062+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:22:24.896055+0000","objects_scrubbed":0,"log_size":882,"log_dups_size":0,"ondisk_log_size":882,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:23:42.213952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":57802752,"num_objects":882,"num_object_clones":0,"num_object_copies":1764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":882,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":882,"num_write_kb":56448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,13],"acting":[11,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.b","version":"50'886","reported_seq":935,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-13T11:22:56.994381+0000","last_change":"2024-09-13T11:22:49.175600+0000","last_active":"2024-09-13T11:22:56.994381+0000","last_peered":"2024-09-13T11:22:56.994381+0000","last_clean":"2024-09-13T11:22:56.994381+0000","last_became_active":"2024-09-13T11:22:49.167083+0000","last_became_peered":"2024-09-13T11:22:49.167083+0000","last_unstale":"2024-09-13T11:22:56.994381+0000","last_undegraded":"2024-09-13T11:22:56.994381+0000","last_fullsized":"2024-09-13T11:22:56.994381+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:22:24.896055+0000","objects_scrubbed":0,"log_size":886,"log_dups_size":0,"ondisk_log_size":886,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:23:45.371983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":58064896,"num_objects":886,"num_object_clones":0,"num_object_copies":1772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":886,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":886,"num_write_kb":56704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,1],"acting":[13,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"3.a","version":"53'1150","reported_seq":1205,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-13T11:22:59.968907+0000","last_change":"2024-09-13T11:22:25.932217+0000","last_active":"2024-09-13T11:22:59.968907+0000","last_peered":"2024-09-13T11:22:59.968907+0000","last_clean":"2024-09-13T11:22:59.968907+0000","last_became_active":"2024-09-13T11:22:25.931493+0000","last_became_peered":"2024-09-13T11:22:25.931493+0000","last_unstale":"2024-09-13T11:22:59.968907+0000","last_undegraded":"2024-09-13T11:22:59.968907+0000","last_fullsized":"2024-09-13T11:22:59.968907+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:22:24.896055+0000","objects_scrubbed":0,"log_size":1150,"log_dups_size":0,"ondisk_log_size":1150,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:23:33.302955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":75366400,"num_objects":1150,"num_object_clones":0,"num_object_copies":2300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1150,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1150,"num_write_kb":73600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.9","version":"54'1283","reported_seq":1339,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-13T11:23:00.989373+0000","last_change":"2024-09-13T11:22:25.922876+0000","last_active":"2024-09-13T11:23:00.989373+0000","last_peered":"2024-09-13T11:23:00.989373+0000","last_clean":"2024-09-13T11:23:00.989373+0000","last_became_active":"2024-09-13T11:22:25.922498+0000","last_became_peered":"2024-09-13T11:22:25.922498+0000","last_unstale":"2024-09-13T11:23:00.989373+0000","last_undegraded":"2024-09-13T11:23:00.989373+0000","last_fullsized":"2024-09-13T11:23:00.989373+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:22:24.896055+0000","objects_scrubbed":0,"log_size":1283,"log_dups_size":0,"ondisk_log_size":1283,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:23:29.835976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":84082688,"num_objects":1283,"num_object_clones":0,"num_object_copies":2566,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1283,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1283,"num_write_kb":82112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,2],"acting":[12,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[]},{"pgid":"3.8","version":"54'1273","reported_seq":1331,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-13T11:23:01.017914+0000","last_change":"2024-09-13T11:22:47.301262+0000","last_active":"2024-09-13T11:23:01.017914+0000","last_peered":"2024-09-13T11:23:01.017914+0000","last_clean":"2024-09-13T11:23:01.017914+0000","last_became_active":"2024-09-13T11:22:47.296719+0000","last_became_peered":"2024-09-13T11:22:47.296719+0000","last_unstale":"2024-09-13T11:23:01.017914+0000","last_undegraded":"2024-09-13T11:23:01.017914+0000","last_fullsized":"2024-09-13T11:23:01.017914+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:22:24.896055+0000","objects_scrubbed":0,"log_size":1273,"log_dups_size":0,"ondisk_log_size":1273,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:23:54.227881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":83427328,"num_objects":1273,"num_object_clones":0,"num_object_copies":2546,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1273,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1273,"num_write_kb":81472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"26'112","reported_seq":206,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-13T11:23:00.440566+0000","last_change":"2024-09-13T11:22:12.131711+0000","last_active":"2024-09-13T11:23:00.440566+0000","last_peered":"2024-09-13T11:23:00.440566+0000","last_clean":"2024-09-13T11:23:00.440566+0000","last_became_active":"2024-09-13T11:22:12.129721+0000","last_became_peered":"2024-09-13T11:22:12.129721+0000","last_unstale":"2024-09-13T11:23:00.440566+0000","last_undegraded":"2024-09-13T11:23:00.440566+0000","last_fullsized":"2024-09-13T11:23:00.440566+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:11.098388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:11.098388+0000","last_clean_scrub_stamp":"2024-09-13T11:22:11.098388+0000","objects_scrubbed":0,"log_size":112,"log_dups_size":0,"ondisk_log_size":112,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:23:40.174992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.3","version":"26'1","reported_seq":73,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-13T11:23:00.425006+0000","last_change":"2024-09-13T11:22:16.547691+0000","last_active":"2024-09-13T11:23:00.425006+0000","last_peered":"2024-09-13T11:23:00.425006+0000","last_clean":"2024-09-13T11:23:00.425006+0000","last_became_active":"2024-09-13T11:22:14.146507+0000","last_became_peered":"2024-09-13T11:22:14.146507+0000","last_unstale":"2024-09-13T11:23:00.425006+0000","last_undegraded":"2024-09-13T11:23:00.425006+0000","last_fullsized":"2024-09-13T11:23:00.425006+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:22:13.113563+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:23:18.785562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00085610999999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,15],"acting":[14,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"3.2","version":"54'1283","reported_seq":1340,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-13T11:23:01.028296+0000","last_change":"2024-09-13T11:22:25.923575+0000","last_active":"2024-09-13T11:23:01.028296+0000","last_peered":"2024-09-13T11:23:01.028296+0000","last_clean":"2024-09-13T11:23:01.028296+0000","last_became_active":"2024-09-13T11:22:25.922665+0000","last_became_peered":"2024-09-13T11:22:25.922665+0000","last_unstale":"2024-09-13T11:23:01.028296+0000","last_undegraded":"2024-09-13T11:23:01.028296+0000","last_fullsized":"2024-09-13T11:23:01.028296+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:22:24.896055+0000","objects_scrubbed":0,"log_size":1283,"log_dups_size":0,"ondisk_log_size":1283,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:23:37.611518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":84082688,"num_objects":1283,"num_object_clones":0,"num_object_copies":2566,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1283,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1283,"num_write_kb":82112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,12],"acting":[14,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":70,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-13T11:22:59.422067+0000","last_change":"2024-09-13T11:22:16.267999+0000","last_active":"2024-09-13T11:22:59.422067+0000","last_peered":"2024-09-13T11:22:59.422067+0000","last_clean":"2024-09-13T11:22:59.422067+0000","last_became_active":"2024-09-13T11:22:14.211435+0000","last_became_peered":"2024-09-13T11:22:14.211435+0000","last_unstale":"2024-09-13T11:22:59.422067+0000","last_undegraded":"2024-09-13T11:22:59.422067+0000","last_fullsized":"2024-09-13T11:22:59.422067+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:22:13.113563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:23:24.076640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010010360000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"50'858","reported_seq":906,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-13T11:22:57.002260+0000","last_change":"2024-09-13T11:22:25.933242+0000","last_active":"2024-09-13T11:22:57.002260+0000","last_peered":"2024-09-13T11:22:57.002260+0000","last_clean":"2024-09-13T11:22:57.002260+0000","last_became_active":"2024-09-13T11:22:25.932868+0000","last_became_peered":"2024-09-13T11:22:25.932868+0000","last_unstale":"2024-09-13T11:22:57.002260+0000","last_undegraded":"2024-09-13T11:22:57.002260+0000","last_fullsized":"2024-09-13T11:22:57.002260+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:22:24.896055+0000","objects_scrubbed":0,"log_size":858,"log_dups_size":0,"ondisk_log_size":858,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:23:38.369067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56229888,"num_objects":858,"num_object_clones":0,"num_object_copies":1716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":858,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":858,"num_write_kb":54912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,11],"acting":[13,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":81,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-13T11:23:00.424989+0000","last_change":"2024-09-13T11:22:47.298206+0000","last_active":"2024-09-13T11:23:00.424989+0000","last_peered":"2024-09-13T11:23:00.424989+0000","last_clean":"2024-09-13T11:23:00.424989+0000","last_became_active":"2024-09-13T11:22:47.297319+0000","last_became_peered":"2024-09-13T11:22:47.297319+0000","last_unstale":"2024-09-13T11:23:00.424989+0000","last_undegraded":"2024-09-13T11:23:00.424989+0000","last_fullsized":"2024-09-13T11:23:00.424989+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:22:13.113563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:23:23.483800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00081225399999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,1],"acting":[14,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"3.0","version":"54'1291","reported_seq":1347,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-13T11:23:01.006156+0000","last_change":"2024-09-13T11:22:25.926230+0000","last_active":"2024-09-13T11:23:01.006156+0000","last_peered":"2024-09-13T11:23:01.006156+0000","last_clean":"2024-09-13T11:23:01.006156+0000","last_became_active":"2024-09-13T11:22:25.925791+0000","last_became_peered":"2024-09-13T11:22:25.925791+0000","last_unstale":"2024-09-13T11:23:01.006156+0000","last_undegraded":"2024-09-13T11:23:01.006156+0000","last_fullsized":"2024-09-13T11:23:01.006156+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:22:24.896055+0000","objects_scrubbed":0,"log_size":1291,"log_dups_size":0,"ondisk_log_size":1291,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:23:38.551603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":84606976,"num_objects":1291,"num_object_clones":0,"num_object_copies":2582,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1291,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1291,"num_write_kb":82624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,15],"acting":[12,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[]},{"pgid":"2.2","version":"28'2","reported_seq":72,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-13T11:22:59.422104+0000","last_change":"2024-09-13T11:22:16.269626+0000","last_active":"2024-09-13T11:22:59.422104+0000","last_peered":"2024-09-13T11:22:59.422104+0000","last_clean":"2024-09-13T11:22:59.422104+0000","last_became_active":"2024-09-13T11:22:14.144472+0000","last_became_peered":"2024-09-13T11:22:14.144472+0000","last_unstale":"2024-09-13T11:22:59.422104+0000","last_undegraded":"2024-09-13T11:22:59.422104+0000","last_fullsized":"2024-09-13T11:22:59.422104+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:22:13.113563+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:23:28.879995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0025826730000000002,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9],"acting":[3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.3","version":"54'1290","reported_seq":1345,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-13T11:23:01.026174+0000","last_change":"2024-09-13T11:22:25.930558+0000","last_active":"2024-09-13T11:23:01.026174+0000","last_peered":"2024-09-13T11:23:01.026174+0000","last_clean":"2024-09-13T11:23:01.026174+0000","last_became_active":"2024-09-13T11:22:25.930023+0000","last_became_peered":"2024-09-13T11:22:25.930023+0000","last_unstale":"2024-09-13T11:23:01.026174+0000","last_undegraded":"2024-09-13T11:23:01.026174+0000","last_fullsized":"2024-09-13T11:23:01.026174+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:22:24.896055+0000","objects_scrubbed":0,"log_size":1290,"log_dups_size":0,"ondisk_log_size":1290,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:23:33.097704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":84541440,"num_objects":1290,"num_object_clones":0,"num_object_copies":2580,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1290,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1290,"num_write_kb":82560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,7],"acting":[10,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.5","version":"54'1259","reported_seq":1315,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-13T11:23:01.021747+0000","last_change":"2024-09-13T11:22:25.930005+0000","last_active":"2024-09-13T11:23:01.021747+0000","last_peered":"2024-09-13T11:23:01.021747+0000","last_clean":"2024-09-13T11:23:01.021747+0000","last_became_active":"2024-09-13T11:22:25.929242+0000","last_became_peered":"2024-09-13T11:22:25.929242+0000","last_unstale":"2024-09-13T11:23:01.021747+0000","last_undegraded":"2024-09-13T11:23:01.021747+0000","last_fullsized":"2024-09-13T11:23:01.021747+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:22:24.896055+0000","objects_scrubbed":0,"log_size":1259,"log_dups_size":0,"ondisk_log_size":1259,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:23:41.561894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":82509824,"num_objects":1259,"num_object_clones":0,"num_object_copies":2518,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1259,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1259,"num_write_kb":80576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,13],"acting":[15,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":57,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-13T11:22:59.424762+0000","last_change":"2024-09-13T11:22:47.720313+0000","last_active":"2024-09-13T11:22:59.424762+0000","last_peered":"2024-09-13T11:22:59.424762+0000","last_clean":"2024-09-13T11:22:59.424762+0000","last_became_active":"2024-09-13T11:22:47.719636+0000","last_became_peered":"2024-09-13T11:22:47.719636+0000","last_unstale":"2024-09-13T11:22:59.424762+0000","last_undegraded":"2024-09-13T11:22:59.424762+0000","last_fullsized":"2024-09-13T11:22:59.424762+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:22:13.113563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:23:13.812273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00087476000000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,9],"acting":[1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"53'1217","reported_seq":1250,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-13T11:23:00.275446+0000","last_change":"2024-09-13T11:22:47.293720+0000","last_active":"2024-09-13T11:23:00.275446+0000","last_peered":"2024-09-13T11:23:00.275446+0000","last_clean":"2024-09-13T11:23:00.275446+0000","last_became_active":"2024-09-13T11:22:47.292998+0000","last_became_peered":"2024-09-13T11:22:47.292998+0000","last_unstale":"2024-09-13T11:23:00.275446+0000","last_undegraded":"2024-09-13T11:23:00.275446+0000","last_fullsized":"2024-09-13T11:23:00.275446+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:22:24.896055+0000","objects_scrubbed":0,"log_size":1217,"log_dups_size":0,"ondisk_log_size":1217,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:23:42.001016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":79757312,"num_objects":1217,"num_object_clones":0,"num_object_copies":2434,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1217,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1217,"num_write_kb":77888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,12],"acting":[1,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":71,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-13T11:23:00.419532+0000","last_change":"2024-09-13T11:22:16.504060+0000","last_active":"2024-09-13T11:23:00.419532+0000","last_peered":"2024-09-13T11:23:00.419532+0000","last_clean":"2024-09-13T11:23:00.419532+0000","last_became_active":"2024-09-13T11:22:14.581458+0000","last_became_peered":"2024-09-13T11:22:14.581458+0000","last_unstale":"2024-09-13T11:23:00.419532+0000","last_undegraded":"2024-09-13T11:23:00.419532+0000","last_fullsized":"2024-09-13T11:23:00.419532+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:22:13.113563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:23:37.935010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001026047,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5],"acting":[8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.7","version":"50'842","reported_seq":890,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-13T11:22:56.985659+0000","last_change":"2024-09-13T11:22:25.924462+0000","last_active":"2024-09-13T11:22:56.985659+0000","last_peered":"2024-09-13T11:22:56.985659+0000","last_clean":"2024-09-13T11:22:56.985659+0000","last_became_active":"2024-09-13T11:22:25.924170+0000","last_became_peered":"2024-09-13T11:22:25.924170+0000","last_unstale":"2024-09-13T11:22:56.985659+0000","last_undegraded":"2024-09-13T11:22:56.985659+0000","last_fullsized":"2024-09-13T11:22:56.985659+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:22:24.896055+0000","objects_scrubbed":0,"log_size":842,"log_dups_size":0,"ondisk_log_size":842,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:23:51.945140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":55181312,"num_objects":842,"num_object_clones":0,"num_object_copies":1684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":842,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":842,"num_write_kb":53888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,8],"acting":[13,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":63,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-13T11:22:56.651159+0000","last_change":"2024-09-13T11:22:16.260591+0000","last_active":"2024-09-13T11:22:56.651159+0000","last_peered":"2024-09-13T11:22:56.651159+0000","last_clean":"2024-09-13T11:22:56.651159+0000","last_became_active":"2024-09-13T11:22:14.211923+0000","last_became_peered":"2024-09-13T11:22:14.211923+0000","last_unstale":"2024-09-13T11:22:56.651159+0000","last_undegraded":"2024-09-13T11:22:56.651159+0000","last_fullsized":"2024-09-13T11:22:56.651159+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:22:13.113563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:23:20.320636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011689529999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0],"acting":[9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.6","version":"50'889","reported_seq":936,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-13T11:22:57.025650+0000","last_change":"2024-09-13T11:22:25.928269+0000","last_active":"2024-09-13T11:22:57.025650+0000","last_peered":"2024-09-13T11:22:57.025650+0000","last_clean":"2024-09-13T11:22:57.025650+0000","last_became_active":"2024-09-13T11:22:25.927961+0000","last_became_peered":"2024-09-13T11:22:25.927961+0000","last_unstale":"2024-09-13T11:22:57.025650+0000","last_undegraded":"2024-09-13T11:22:57.025650+0000","last_fullsized":"2024-09-13T11:22:57.025650+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:22:24.896055+0000","objects_scrubbed":0,"log_size":889,"log_dups_size":0,"ondisk_log_size":889,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:23:25.574436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":58261504,"num_objects":889,"num_object_clones":0,"num_object_copies":1778,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":889,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":889,"num_write_kb":56896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,14],"acting":[13,14],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":71,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-13T11:23:00.426547+0000","last_change":"2024-09-13T11:22:16.889130+0000","last_active":"2024-09-13T11:23:00.426547+0000","last_peered":"2024-09-13T11:23:00.426547+0000","last_clean":"2024-09-13T11:23:00.426547+0000","last_became_active":"2024-09-13T11:22:14.192897+0000","last_became_peered":"2024-09-13T11:22:14.192897+0000","last_unstale":"2024-09-13T11:23:00.426547+0000","last_undegraded":"2024-09-13T11:23:00.426547+0000","last_fullsized":"2024-09-13T11:23:00.426547+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:22:13.113563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:23:26.278353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00084655899999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,10],"acting":[15,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":1178861568,"num_objects":17988,"num_object_clones":0,"num_object_copies":35976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17988,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17988,"num_write_kb":1151232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2431188992,"data_stored":2431188992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17988,"ondisk_log_size":17988,"up":32,"acting":32,"num_store_stats":12},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":9},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":112,"ondisk_log_size":112,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":13,"up_from":23,"seq":98784247821,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":330712,"kb_used_data":303688,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94041128,"statfs":{"total":96636764160,"available":96298115072,"internally_reserved":0,"allocated":310976512,"data_stored":310755089,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,6,8,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":9,"up_from":23,"seq":98784247820,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27496,"kb_used_data":472,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344344,"statfs":{"total":96636764160,"available":96608608256,"internally_reserved":0,"allocated":483328,"data_stored":253241,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,2,3,8,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":40,"seq":171798691844,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":267600,"kb_used_data":227264,"kb_used_omap":4,"kb_used_meta":40315,"kb_avail":94104240,"statfs":{"total":96636764160,"available":96362741760,"internally_reserved":0,"allocated":232718336,"data_stored":232468949,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4433,"internal_metadata":41283247},"hb_peers":[0,2,3,4,5,6,7,8,9,11,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":21,"seq":90194313229,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":180124,"kb_used_data":144780,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94191716,"statfs":{"total":96636764160,"available":96452317184,"internally_reserved":0,"allocated":148254720,"data_stored":147998346,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,2,4,5,7,8,9,10,12,13],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":21,"seq":90194313231,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27572,"kb_used_data":548,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344268,"statfs":{"total":96636764160,"available":96608530432,"internally_reserved":0,"allocated":561152,"data_stored":285775,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":22,"seq":94489280524,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28052,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343788,"statfs":{"total":96636764160,"available":96608038912,"internally_reserved":0,"allocated":1052672,"data_stored":862564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,3,4,5,6,7,8,10,11,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":22,"seq":94489280525,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":111472,"kb_used_data":80288,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94260368,"statfs":{"total":96636764160,"available":96522616832,"internally_reserved":0,"allocated":82214912,"data_stored":81948484,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,6,7,8,10,11,12,13],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":15,"up_from":22,"seq":94489280525,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":186968,"kb_used_data":151624,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94184872,"statfs":{"total":96636764160,"available":96445308928,"internally_reserved":0,"allocated":155262976,"data_stored":155069744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,3,4,5,6,7,10,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":22,"seq":94489280525,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31724,"kb_used_data":540,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340116,"statfs":{"total":96636764160,"available":96604278784,"internally_reserved":0,"allocated":552960,"data_stored":283963,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[3,5,7,8,9,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":22,"seq":94489280525,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":107256,"kb_used_data":76072,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94264584,"statfs":{"total":96636764160,"available":96526934016,"internally_reserved":0,"allocated":77897728,"data_stored":77674803,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,3,5,7,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,0,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":22,"seq":94489280525,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":279476,"kb_used_data":239972,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94092364,"statfs":{"total":96636764160,"available":96350580736,"internally_reserved":0,"allocated":245731328,"data_stored":245517008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":40434122},"hb_peers":[0,3,6,8,9,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":22,"seq":94489280525,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":194748,"kb_used_data":159404,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94177092,"statfs":{"total":96636764160,"available":96437342208,"internally_reserved":0,"allocated":163229696,"data_stored":162948130,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[3,5,7,9,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":22,"seq":94489280524,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":105960,"kb_used_data":74776,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94265880,"statfs":{"total":96636764160,"available":96528261120,"internally_reserved":0,"allocated":76570624,"data_stored":76363107,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,5,7,8,9,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":11,"up_from":22,"seq":94489280525,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":193108,"kb_used_data":157764,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94178732,"statfs":{"total":96636764160,"available":96439021568,"internally_reserved":0,"allocated":161550336,"data_stored":161336127,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":36174276},"hb_peers":[2,3,4,5,6,10,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":12,"up_from":22,"seq":94489280525,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":441188,"kb_used_data":401684,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":93930652,"statfs":{"total":96636764160,"available":96184987648,"internally_reserved":0,"allocated":411324416,"data_stored":411065812,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[2,3,4,5,6,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":14,"up_from":22,"seq":94489280525,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":284884,"kb_used_data":245380,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94086956,"statfs":{"total":96636764160,"available":96345042944,"internally_reserved":0,"allocated":251269120,"data_stored":251024209,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,5,6,7,8,9,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":236191744,"data_stored":236191744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":84148224,"data_stored":84148224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":151388160,"data_stored":151388160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":83427328,"data_stored":83427328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":253231104,"data_stored":253231104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":172032000,"data_stored":172032000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":84541440,"data_stored":84541440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":174194688,"data_stored":174194688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":425066496,"data_stored":425066496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":340066304,"data_stored":340066304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":259653632,"data_stored":259653632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":167247872,"data_stored":167247872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T11:23:01.888 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-13T11:23:01.917 INFO:tasks.ceph.osd.5.smithi092.stderr:2024-09-13T11:23:01.914+0000 7f6f0b705640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26346) UID: 0 2024-09-13T11:23:02.000 INFO:tasks.radosbench.radosbench.0.smithi177.stdout: 35 16 21817 21801 38.9256 73 0.222176 0.0255892 2024-09-13T11:23:02.018 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-09-13T11:23:02.015+0000 7f97befc9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26315) UID: 0 2024-09-13T11:23:02.050 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-09-13T11:23:02.050 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:23:02.070 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-13T11:23:02.076 INFO:tasks.ceph.osd.9.smithi092.stderr:2024-09-13T11:23:02.070+0000 7ff59bdf5640 -1 osd.9 54 handle_osd_map injecting map crc failure 2024-09-13T11:23:02.119 INFO:tasks.ceph.osd.13.smithi092.stderr:2024-09-13T11:23:02.114+0000 7f3e5afea640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26350) UID: 0 2024-09-13T11:23:02.220 INFO:tasks.ceph.osd.11.smithi177.stderr:2024-09-13T11:23:02.219+0000 7f24792c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26580) UID: 0 2024-09-13T11:23:02.222 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-09-13T11:23:02.223 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:23:02.242 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-13T11:23:02.247 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-13T11:23:02.247 INFO:teuthology.orchestra.run.smithi001.stderr:dumped all 2024-09-13T11:23:02.271 INFO:teuthology.orchestra.run.smithi001.stdout:{"pg_ready":true,"pg_map":{"version":67,"stamp":"2024-09-13T11:23:02.126829+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1368719923,"num_objects":20880,"num_object_clones":0,"num_object_copies":41760,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20880,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":21023,"num_write_kb":1338738,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":20991,"ondisk_log_size":20991,"up":50,"acting":50,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":50,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":15,"kb":1509949440,"kb_used":3098464,"kb_used_data":2561248,"kb_used_omap":27,"kb_used_meta":536932,"kb_avail":1506850976,"statfs":{"total":1546188226560,"available":1543015399424,"internally_reserved":0,"allocated":2622717952,"data_stored":2618863585,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28358,"internal_metadata":549818682},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,0,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":529793024,"num_objects":8084,"num_object_clones":0,"num_object_copies":16168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8084,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8084,"num_write_kb":517376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000749"},"pg_stats":[{"pgid":"3.f","version":"55'1385","reported_seq":1444,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-13T11:23:01.817226+0000","last_change":"2024-09-13T11:22:25.925951+0000","last_active":"2024-09-13T11:23:01.817226+0000","last_peered":"2024-09-13T11:23:01.817226+0000","last_clean":"2024-09-13T11:23:01.817226+0000","last_became_active":"2024-09-13T11:22:25.925169+0000","last_became_peered":"2024-09-13T11:22:25.925169+0000","last_unstale":"2024-09-13T11:23:01.817226+0000","last_undegraded":"2024-09-13T11:23:01.817226+0000","last_fullsized":"2024-09-13T11:23:01.817226+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:22:24.896055+0000","objects_scrubbed":0,"log_size":1385,"log_dups_size":0,"ondisk_log_size":1385,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:23:38.143926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90767360,"num_objects":1385,"num_object_clones":0,"num_object_copies":2770,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1385,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1385,"num_write_kb":88640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,12],"acting":[11,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.e","version":"54'1377","reported_seq":1432,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-13T11:23:01.012203+0000","last_change":"2024-09-13T11:22:25.923829+0000","last_active":"2024-09-13T11:23:01.012203+0000","last_peered":"2024-09-13T11:23:01.012203+0000","last_clean":"2024-09-13T11:23:01.012203+0000","last_became_active":"2024-09-13T11:22:25.923498+0000","last_became_peered":"2024-09-13T11:22:25.923498+0000","last_unstale":"2024-09-13T11:23:01.012203+0000","last_undegraded":"2024-09-13T11:23:01.012203+0000","last_fullsized":"2024-09-13T11:23:01.012203+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:22:24.896055+0000","objects_scrubbed":0,"log_size":1377,"log_dups_size":0,"ondisk_log_size":1377,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:23:50.687438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90243072,"num_objects":1377,"num_object_clones":0,"num_object_copies":2754,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1377,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1377,"num_write_kb":88128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,14],"acting":[8,14],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.d","version":"54'1291","reported_seq":1346,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-13T11:23:01.014481+0000","last_change":"2024-09-13T11:22:25.927325+0000","last_active":"2024-09-13T11:23:01.014481+0000","last_peered":"2024-09-13T11:23:01.014481+0000","last_clean":"2024-09-13T11:23:01.014481+0000","last_became_active":"2024-09-13T11:22:25.926433+0000","last_became_peered":"2024-09-13T11:22:25.926433+0000","last_unstale":"2024-09-13T11:23:01.014481+0000","last_undegraded":"2024-09-13T11:23:01.014481+0000","last_fullsized":"2024-09-13T11:23:01.014481+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:22:24.896055+0000","objects_scrubbed":0,"log_size":1291,"log_dups_size":0,"ondisk_log_size":1291,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:23:39.281487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":84606976,"num_objects":1291,"num_object_clones":0,"num_object_copies":2582,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1291,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1291,"num_write_kb":82624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.c","version":"55'1371","reported_seq":1430,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-13T11:23:02.018208+0000","last_change":"2024-09-13T11:22:25.934619+0000","last_active":"2024-09-13T11:23:02.018208+0000","last_peered":"2024-09-13T11:23:02.018208+0000","last_clean":"2024-09-13T11:23:02.018208+0000","last_became_active":"2024-09-13T11:22:25.933854+0000","last_became_peered":"2024-09-13T11:22:25.933854+0000","last_unstale":"2024-09-13T11:23:02.018208+0000","last_undegraded":"2024-09-13T11:23:02.018208+0000","last_fullsized":"2024-09-13T11:23:02.018208+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:22:24.896055+0000","objects_scrubbed":0,"log_size":1371,"log_dups_size":0,"ondisk_log_size":1371,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:23:42.213952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":89849856,"num_objects":1371,"num_object_clones":0,"num_object_copies":2742,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1371,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1371,"num_write_kb":87744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,13],"acting":[11,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.b","version":"55'1407","reported_seq":1465,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-13T11:23:02.010994+0000","last_change":"2024-09-13T11:22:49.175600+0000","last_active":"2024-09-13T11:23:02.010994+0000","last_peered":"2024-09-13T11:23:02.010994+0000","last_clean":"2024-09-13T11:23:02.010994+0000","last_became_active":"2024-09-13T11:22:49.167083+0000","last_became_peered":"2024-09-13T11:22:49.167083+0000","last_unstale":"2024-09-13T11:23:02.010994+0000","last_undegraded":"2024-09-13T11:23:02.010994+0000","last_fullsized":"2024-09-13T11:23:02.010994+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:22:24.896055+0000","objects_scrubbed":0,"log_size":1407,"log_dups_size":0,"ondisk_log_size":1407,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:23:45.371983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92209152,"num_objects":1407,"num_object_clones":0,"num_object_copies":2814,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1407,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1407,"num_write_kb":90048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,1],"acting":[13,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"3.a","version":"53'1150","reported_seq":1205,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-13T11:22:59.968907+0000","last_change":"2024-09-13T11:22:25.932217+0000","last_active":"2024-09-13T11:22:59.968907+0000","last_peered":"2024-09-13T11:22:59.968907+0000","last_clean":"2024-09-13T11:22:59.968907+0000","last_became_active":"2024-09-13T11:22:25.931493+0000","last_became_peered":"2024-09-13T11:22:25.931493+0000","last_unstale":"2024-09-13T11:22:59.968907+0000","last_undegraded":"2024-09-13T11:22:59.968907+0000","last_fullsized":"2024-09-13T11:22:59.968907+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:22:24.896055+0000","objects_scrubbed":0,"log_size":1150,"log_dups_size":0,"ondisk_log_size":1150,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:23:33.302955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":75366400,"num_objects":1150,"num_object_clones":0,"num_object_copies":2300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1150,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1150,"num_write_kb":73600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.9","version":"54'1283","reported_seq":1339,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-13T11:23:00.989373+0000","last_change":"2024-09-13T11:22:25.922876+0000","last_active":"2024-09-13T11:23:00.989373+0000","last_peered":"2024-09-13T11:23:00.989373+0000","last_clean":"2024-09-13T11:23:00.989373+0000","last_became_active":"2024-09-13T11:22:25.922498+0000","last_became_peered":"2024-09-13T11:22:25.922498+0000","last_unstale":"2024-09-13T11:23:00.989373+0000","last_undegraded":"2024-09-13T11:23:00.989373+0000","last_fullsized":"2024-09-13T11:23:00.989373+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:22:24.896055+0000","objects_scrubbed":0,"log_size":1283,"log_dups_size":0,"ondisk_log_size":1283,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:23:29.835976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":84082688,"num_objects":1283,"num_object_clones":0,"num_object_copies":2566,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1283,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1283,"num_write_kb":82112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,2],"acting":[12,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[]},{"pgid":"3.8","version":"54'1273","reported_seq":1331,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-13T11:23:01.017914+0000","last_change":"2024-09-13T11:22:47.301262+0000","last_active":"2024-09-13T11:23:01.017914+0000","last_peered":"2024-09-13T11:23:01.017914+0000","last_clean":"2024-09-13T11:23:01.017914+0000","last_became_active":"2024-09-13T11:22:47.296719+0000","last_became_peered":"2024-09-13T11:22:47.296719+0000","last_unstale":"2024-09-13T11:23:01.017914+0000","last_undegraded":"2024-09-13T11:23:01.017914+0000","last_fullsized":"2024-09-13T11:23:01.017914+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:22:24.896055+0000","objects_scrubbed":0,"log_size":1273,"log_dups_size":0,"ondisk_log_size":1273,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:23:54.227881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":83427328,"num_objects":1273,"num_object_clones":0,"num_object_copies":2546,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1273,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1273,"num_write_kb":81472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"26'112","reported_seq":206,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-13T11:23:00.440566+0000","last_change":"2024-09-13T11:22:12.131711+0000","last_active":"2024-09-13T11:23:00.440566+0000","last_peered":"2024-09-13T11:23:00.440566+0000","last_clean":"2024-09-13T11:23:00.440566+0000","last_became_active":"2024-09-13T11:22:12.129721+0000","last_became_peered":"2024-09-13T11:22:12.129721+0000","last_unstale":"2024-09-13T11:23:00.440566+0000","last_undegraded":"2024-09-13T11:23:00.440566+0000","last_fullsized":"2024-09-13T11:23:00.440566+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:11.098388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:11.098388+0000","last_clean_scrub_stamp":"2024-09-13T11:22:11.098388+0000","objects_scrubbed":0,"log_size":112,"log_dups_size":0,"ondisk_log_size":112,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:23:40.174992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.3","version":"26'1","reported_seq":73,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-13T11:23:00.425006+0000","last_change":"2024-09-13T11:22:16.547691+0000","last_active":"2024-09-13T11:23:00.425006+0000","last_peered":"2024-09-13T11:23:00.425006+0000","last_clean":"2024-09-13T11:23:00.425006+0000","last_became_active":"2024-09-13T11:22:14.146507+0000","last_became_peered":"2024-09-13T11:22:14.146507+0000","last_unstale":"2024-09-13T11:23:00.425006+0000","last_undegraded":"2024-09-13T11:23:00.425006+0000","last_fullsized":"2024-09-13T11:23:00.425006+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:22:13.113563+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:23:18.785562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00085610999999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,15],"acting":[14,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"3.2","version":"54'1283","reported_seq":1340,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-13T11:23:01.028296+0000","last_change":"2024-09-13T11:22:25.923575+0000","last_active":"2024-09-13T11:23:01.028296+0000","last_peered":"2024-09-13T11:23:01.028296+0000","last_clean":"2024-09-13T11:23:01.028296+0000","last_became_active":"2024-09-13T11:22:25.922665+0000","last_became_peered":"2024-09-13T11:22:25.922665+0000","last_unstale":"2024-09-13T11:23:01.028296+0000","last_undegraded":"2024-09-13T11:23:01.028296+0000","last_fullsized":"2024-09-13T11:23:01.028296+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:22:24.896055+0000","objects_scrubbed":0,"log_size":1283,"log_dups_size":0,"ondisk_log_size":1283,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:23:37.611518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":84082688,"num_objects":1283,"num_object_clones":0,"num_object_copies":2566,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1283,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1283,"num_write_kb":82112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,12],"acting":[14,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":70,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-13T11:22:59.422067+0000","last_change":"2024-09-13T11:22:16.267999+0000","last_active":"2024-09-13T11:22:59.422067+0000","last_peered":"2024-09-13T11:22:59.422067+0000","last_clean":"2024-09-13T11:22:59.422067+0000","last_became_active":"2024-09-13T11:22:14.211435+0000","last_became_peered":"2024-09-13T11:22:14.211435+0000","last_unstale":"2024-09-13T11:22:59.422067+0000","last_undegraded":"2024-09-13T11:22:59.422067+0000","last_fullsized":"2024-09-13T11:22:59.422067+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:22:13.113563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:23:24.076640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010010360000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"55'1330","reported_seq":1388,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-13T11:23:01.795235+0000","last_change":"2024-09-13T11:22:25.933242+0000","last_active":"2024-09-13T11:23:01.795235+0000","last_peered":"2024-09-13T11:23:01.795235+0000","last_clean":"2024-09-13T11:23:01.795235+0000","last_became_active":"2024-09-13T11:22:25.932868+0000","last_became_peered":"2024-09-13T11:22:25.932868+0000","last_unstale":"2024-09-13T11:23:01.795235+0000","last_undegraded":"2024-09-13T11:23:01.795235+0000","last_fullsized":"2024-09-13T11:23:01.795235+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:22:24.896055+0000","objects_scrubbed":0,"log_size":1330,"log_dups_size":0,"ondisk_log_size":1330,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:23:38.369067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":87162880,"num_objects":1330,"num_object_clones":0,"num_object_copies":2660,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1330,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1330,"num_write_kb":85120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,11],"acting":[13,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":81,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-13T11:23:00.424989+0000","last_change":"2024-09-13T11:22:47.298206+0000","last_active":"2024-09-13T11:23:00.424989+0000","last_peered":"2024-09-13T11:23:00.424989+0000","last_clean":"2024-09-13T11:23:00.424989+0000","last_became_active":"2024-09-13T11:22:47.297319+0000","last_became_peered":"2024-09-13T11:22:47.297319+0000","last_unstale":"2024-09-13T11:23:00.424989+0000","last_undegraded":"2024-09-13T11:23:00.424989+0000","last_fullsized":"2024-09-13T11:23:00.424989+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:22:13.113563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:23:23.483800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00081225399999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,1],"acting":[14,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"3.0","version":"54'1291","reported_seq":1347,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-13T11:23:01.006156+0000","last_change":"2024-09-13T11:22:25.926230+0000","last_active":"2024-09-13T11:23:01.006156+0000","last_peered":"2024-09-13T11:23:01.006156+0000","last_clean":"2024-09-13T11:23:01.006156+0000","last_became_active":"2024-09-13T11:22:25.925791+0000","last_became_peered":"2024-09-13T11:22:25.925791+0000","last_unstale":"2024-09-13T11:23:01.006156+0000","last_undegraded":"2024-09-13T11:23:01.006156+0000","last_fullsized":"2024-09-13T11:23:01.006156+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:22:24.896055+0000","objects_scrubbed":0,"log_size":1291,"log_dups_size":0,"ondisk_log_size":1291,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:23:38.551603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":84606976,"num_objects":1291,"num_object_clones":0,"num_object_copies":2582,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1291,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1291,"num_write_kb":82624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,15],"acting":[12,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[]},{"pgid":"2.2","version":"28'2","reported_seq":72,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-13T11:22:59.422104+0000","last_change":"2024-09-13T11:22:16.269626+0000","last_active":"2024-09-13T11:22:59.422104+0000","last_peered":"2024-09-13T11:22:59.422104+0000","last_clean":"2024-09-13T11:22:59.422104+0000","last_became_active":"2024-09-13T11:22:14.144472+0000","last_became_peered":"2024-09-13T11:22:14.144472+0000","last_unstale":"2024-09-13T11:22:59.422104+0000","last_undegraded":"2024-09-13T11:22:59.422104+0000","last_fullsized":"2024-09-13T11:22:59.422104+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:22:13.113563+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:23:28.879995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0025826730000000002,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9],"acting":[3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.3","version":"54'1290","reported_seq":1345,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-13T11:23:01.026174+0000","last_change":"2024-09-13T11:22:25.930558+0000","last_active":"2024-09-13T11:23:01.026174+0000","last_peered":"2024-09-13T11:23:01.026174+0000","last_clean":"2024-09-13T11:23:01.026174+0000","last_became_active":"2024-09-13T11:22:25.930023+0000","last_became_peered":"2024-09-13T11:22:25.930023+0000","last_unstale":"2024-09-13T11:23:01.026174+0000","last_undegraded":"2024-09-13T11:23:01.026174+0000","last_fullsized":"2024-09-13T11:23:01.026174+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:22:24.896055+0000","objects_scrubbed":0,"log_size":1290,"log_dups_size":0,"ondisk_log_size":1290,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:23:33.097704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":84541440,"num_objects":1290,"num_object_clones":0,"num_object_copies":2580,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1290,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1290,"num_write_kb":82560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,7],"acting":[10,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.5","version":"54'1259","reported_seq":1315,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-13T11:23:01.021747+0000","last_change":"2024-09-13T11:22:25.930005+0000","last_active":"2024-09-13T11:23:01.021747+0000","last_peered":"2024-09-13T11:23:01.021747+0000","last_clean":"2024-09-13T11:23:01.021747+0000","last_became_active":"2024-09-13T11:22:25.929242+0000","last_became_peered":"2024-09-13T11:22:25.929242+0000","last_unstale":"2024-09-13T11:23:01.021747+0000","last_undegraded":"2024-09-13T11:23:01.021747+0000","last_fullsized":"2024-09-13T11:23:01.021747+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:22:24.896055+0000","objects_scrubbed":0,"log_size":1259,"log_dups_size":0,"ondisk_log_size":1259,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:23:41.561894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":82509824,"num_objects":1259,"num_object_clones":0,"num_object_copies":2518,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1259,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1259,"num_write_kb":80576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,13],"acting":[15,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":57,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-13T11:22:59.424762+0000","last_change":"2024-09-13T11:22:47.720313+0000","last_active":"2024-09-13T11:22:59.424762+0000","last_peered":"2024-09-13T11:22:59.424762+0000","last_clean":"2024-09-13T11:22:59.424762+0000","last_became_active":"2024-09-13T11:22:47.719636+0000","last_became_peered":"2024-09-13T11:22:47.719636+0000","last_unstale":"2024-09-13T11:22:59.424762+0000","last_undegraded":"2024-09-13T11:22:59.424762+0000","last_fullsized":"2024-09-13T11:22:59.424762+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:22:13.113563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:23:13.812273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00087476000000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,9],"acting":[1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"53'1217","reported_seq":1250,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-13T11:23:00.275446+0000","last_change":"2024-09-13T11:22:47.293720+0000","last_active":"2024-09-13T11:23:00.275446+0000","last_peered":"2024-09-13T11:23:00.275446+0000","last_clean":"2024-09-13T11:23:00.275446+0000","last_became_active":"2024-09-13T11:22:47.292998+0000","last_became_peered":"2024-09-13T11:22:47.292998+0000","last_unstale":"2024-09-13T11:23:00.275446+0000","last_undegraded":"2024-09-13T11:23:00.275446+0000","last_fullsized":"2024-09-13T11:23:00.275446+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:22:24.896055+0000","objects_scrubbed":0,"log_size":1217,"log_dups_size":0,"ondisk_log_size":1217,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:23:42.001016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":79757312,"num_objects":1217,"num_object_clones":0,"num_object_copies":2434,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1217,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1217,"num_write_kb":77888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,12],"acting":[1,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":71,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-13T11:23:00.419532+0000","last_change":"2024-09-13T11:22:16.504060+0000","last_active":"2024-09-13T11:23:00.419532+0000","last_peered":"2024-09-13T11:23:00.419532+0000","last_clean":"2024-09-13T11:23:00.419532+0000","last_became_active":"2024-09-13T11:22:14.581458+0000","last_became_peered":"2024-09-13T11:22:14.581458+0000","last_unstale":"2024-09-13T11:23:00.419532+0000","last_undegraded":"2024-09-13T11:23:00.419532+0000","last_fullsized":"2024-09-13T11:23:00.419532+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:22:13.113563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:23:37.935010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001026047,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5],"acting":[8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.7","version":"55'1304","reported_seq":1362,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-13T11:23:01.800327+0000","last_change":"2024-09-13T11:22:25.924462+0000","last_active":"2024-09-13T11:23:01.800327+0000","last_peered":"2024-09-13T11:23:01.800327+0000","last_clean":"2024-09-13T11:23:01.800327+0000","last_became_active":"2024-09-13T11:22:25.924170+0000","last_became_peered":"2024-09-13T11:22:25.924170+0000","last_unstale":"2024-09-13T11:23:01.800327+0000","last_undegraded":"2024-09-13T11:23:01.800327+0000","last_fullsized":"2024-09-13T11:23:01.800327+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:22:24.896055+0000","objects_scrubbed":0,"log_size":1304,"log_dups_size":0,"ondisk_log_size":1304,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:23:51.945140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":85458944,"num_objects":1304,"num_object_clones":0,"num_object_copies":2608,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1304,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1304,"num_write_kb":83456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,8],"acting":[13,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":71,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-13T11:23:00.974086+0000","last_change":"2024-09-13T11:22:16.260591+0000","last_active":"2024-09-13T11:23:00.974086+0000","last_peered":"2024-09-13T11:23:00.974086+0000","last_clean":"2024-09-13T11:23:00.974086+0000","last_became_active":"2024-09-13T11:22:14.211923+0000","last_became_peered":"2024-09-13T11:22:14.211923+0000","last_unstale":"2024-09-13T11:23:00.974086+0000","last_undegraded":"2024-09-13T11:23:00.974086+0000","last_fullsized":"2024-09-13T11:23:00.974086+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:22:13.113563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:23:20.320636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011689529999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0],"acting":[9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.6","version":"55'1365","reported_seq":1423,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-13T11:23:02.013524+0000","last_change":"2024-09-13T11:22:25.928269+0000","last_active":"2024-09-13T11:23:02.013524+0000","last_peered":"2024-09-13T11:23:02.013524+0000","last_clean":"2024-09-13T11:23:02.013524+0000","last_became_active":"2024-09-13T11:22:25.927961+0000","last_became_peered":"2024-09-13T11:22:25.927961+0000","last_unstale":"2024-09-13T11:23:02.013524+0000","last_undegraded":"2024-09-13T11:23:02.013524+0000","last_fullsized":"2024-09-13T11:23:02.013524+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:22:24.896055+0000","objects_scrubbed":0,"log_size":1365,"log_dups_size":0,"ondisk_log_size":1365,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:23:25.574436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":89456640,"num_objects":1365,"num_object_clones":0,"num_object_copies":2730,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1365,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1365,"num_write_kb":87360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,14],"acting":[13,14],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":71,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-13T11:23:00.426547+0000","last_change":"2024-09-13T11:22:16.889130+0000","last_active":"2024-09-13T11:23:00.426547+0000","last_peered":"2024-09-13T11:23:00.426547+0000","last_clean":"2024-09-13T11:23:00.426547+0000","last_became_active":"2024-09-13T11:22:14.192897+0000","last_became_peered":"2024-09-13T11:22:14.192897+0000","last_unstale":"2024-09-13T11:23:00.426547+0000","last_undegraded":"2024-09-13T11:23:00.426547+0000","last_fullsized":"2024-09-13T11:23:00.426547+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:22:13.113563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:23:26.278353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00084655899999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,10],"acting":[15,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":1368129536,"num_objects":20876,"num_object_clones":0,"num_object_copies":41752,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20876,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20876,"num_write_kb":1336064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2716008448,"data_stored":2716008448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":20876,"ondisk_log_size":20876,"up":32,"acting":32,"num_store_stats":12},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":9},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":112,"ondisk_log_size":112,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":13,"up_from":23,"seq":98784247822,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":540816,"kb_used_data":501312,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":93831024,"statfs":{"total":96636764160,"available":96082968576,"internally_reserved":0,"allocated":513343488,"data_stored":513100754,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,3,4,6,8,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":9,"up_from":23,"seq":98784247821,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":516,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":528384,"data_stored":282017,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,2,3,8,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":40,"seq":171798691844,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":267600,"kb_used_data":227264,"kb_used_omap":4,"kb_used_meta":40315,"kb_avail":94104240,"statfs":{"total":96636764160,"available":96362741760,"internally_reserved":0,"allocated":232718336,"data_stored":232468949,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4433,"internal_metadata":41283247},"hb_peers":[0,2,3,4,5,6,7,8,9,11,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":21,"seq":90194313229,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":180124,"kb_used_data":144780,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94191716,"statfs":{"total":96636764160,"available":96452317184,"internally_reserved":0,"allocated":148254720,"data_stored":147998346,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,2,4,5,7,8,9,10,12,13],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":21,"seq":90194313231,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27572,"kb_used_data":548,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344268,"statfs":{"total":96636764160,"available":96608530432,"internally_reserved":0,"allocated":561152,"data_stored":285775,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":22,"seq":94489280524,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28052,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343788,"statfs":{"total":96636764160,"available":96608038912,"internally_reserved":0,"allocated":1052672,"data_stored":862564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,3,4,5,6,7,8,10,11,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":22,"seq":94489280525,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":111472,"kb_used_data":80288,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94260368,"statfs":{"total":96636764160,"available":96522616832,"internally_reserved":0,"allocated":82214912,"data_stored":81948484,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,6,7,8,10,11,12,13],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":15,"up_from":22,"seq":94489280525,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":186968,"kb_used_data":151624,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94184872,"statfs":{"total":96636764160,"available":96445308928,"internally_reserved":0,"allocated":155262976,"data_stored":155069744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,3,4,5,6,7,10,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":22,"seq":94489280525,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31724,"kb_used_data":540,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340116,"statfs":{"total":96636764160,"available":96604278784,"internally_reserved":0,"allocated":552960,"data_stored":283963,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[3,5,7,8,9,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":22,"seq":94489280525,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":107256,"kb_used_data":76072,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94264584,"statfs":{"total":96636764160,"available":96526934016,"internally_reserved":0,"allocated":77897728,"data_stored":77674803,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,3,5,7,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,0,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":22,"seq":94489280525,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":279476,"kb_used_data":239972,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94092364,"statfs":{"total":96636764160,"available":96350580736,"internally_reserved":0,"allocated":245731328,"data_stored":245517008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":40434122},"hb_peers":[0,3,6,8,9,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":22,"seq":94489280525,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":194748,"kb_used_data":159404,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94177092,"statfs":{"total":96636764160,"available":96437342208,"internally_reserved":0,"allocated":163229696,"data_stored":162948130,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[3,5,7,9,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":22,"seq":94489280524,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":105960,"kb_used_data":74776,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94265880,"statfs":{"total":96636764160,"available":96528261120,"internally_reserved":0,"allocated":76570624,"data_stored":76363107,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,5,7,8,9,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":11,"up_from":22,"seq":94489280526,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":283084,"kb_used_data":256060,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94088756,"statfs":{"total":96636764160,"available":96346886144,"internally_reserved":0,"allocated":262205440,"data_stored":261969920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[2,3,4,5,6,10,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":12,"up_from":22,"seq":94489280525,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":441188,"kb_used_data":401684,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":93930652,"statfs":{"total":96636764160,"available":96184987648,"internally_reserved":0,"allocated":411324416,"data_stored":411065812,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[2,3,4,5,6,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":14,"up_from":22,"seq":94489280525,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":284884,"kb_used_data":245380,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94086956,"statfs":{"total":96636764160,"available":96345042944,"internally_reserved":0,"allocated":251269120,"data_stored":251024209,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,5,6,7,8,9,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":236191744,"data_stored":236191744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":84148224,"data_stored":84148224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":151388160,"data_stored":151388160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":83427328,"data_stored":83427328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":253231104,"data_stored":253231104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":172032000,"data_stored":172032000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":84541440,"data_stored":84541440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":267845632,"data_stored":267845632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":425066496,"data_stored":425066496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":531234816,"data_stored":531234816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":259653632,"data_stored":259653632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":167247872,"data_stored":167247872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T11:23:02.273 INFO:tasks.ceph.ceph_manager.ceph:recovered! 2024-09-13T11:23:02.273 INFO:tasks.thrashosds.thrasher:choose_action: min_in 2 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-13T11:23:02.273 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2024-09-13T11:23:02.273 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump -f json-pretty 2024-09-13T11:23:02.321 INFO:tasks.ceph.osd.9.smithi092.stderr:2024-09-13T11:23:02.318+0000 7ff5ab3d0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26348) UID: 0 2024-09-13T11:23:02.423 INFO:tasks.ceph.osd.10.smithi162.stderr:2024-09-13T11:23:02.419+0000 7fb59c9f0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26317) UID: 0 2024-09-13T11:23:02.439 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-09-13T11:23:02.439 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:23:02.457 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-13T11:23:02.524 INFO:tasks.ceph.osd.5.smithi092.stderr:2024-09-13T11:23:02.522+0000 7f6f0b705640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26346) UID: 0 2024-09-13T11:23:02.591 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-09-13T11:23:02.591 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:23:02.609 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-13T11:23:02.613 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-13T11:23:02.613 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-13T11:23:02.613 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 55, 2024-09-13T11:23:02.613 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "2fd68ac7-5581-4d90-80a4-30bcf5252a7a", 2024-09-13T11:23:02.613 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-09-13T11:21:57.223153+0000", 2024-09-13T11:23:02.614 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-09-13T11:23:01.422282+0000", 2024-09-13T11:23:02.614 INFO:teuthology.orchestra.run.smithi001.stdout: "last_up_change": "2024-09-13T11:22:46.271946+0000", 2024-09-13T11:23:02.614 INFO:teuthology.orchestra.run.smithi001.stdout: "last_in_change": "2024-09-13T11:22:01.745814+0000", 2024-09-13T11:23:02.614 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": "noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-13T11:23:02.614 INFO:teuthology.orchestra.run.smithi001.stdout: "flags_num": 5801984, 2024-09-13T11:23:02.615 INFO:teuthology.orchestra.run.smithi001.stdout: "flags_set": [ 2024-09-13T11:23:02.615 INFO:teuthology.orchestra.run.smithi001.stdout: "noscrub", 2024-09-13T11:23:02.615 INFO:teuthology.orchestra.run.smithi001.stdout: "pglog_hardlimit", 2024-09-13T11:23:02.615 INFO:teuthology.orchestra.run.smithi001.stdout: "purged_snapdirs", 2024-09-13T11:23:02.615 INFO:teuthology.orchestra.run.smithi001.stdout: "recovery_deletes", 2024-09-13T11:23:02.616 INFO:teuthology.orchestra.run.smithi001.stdout: "sortbitwise" 2024-09-13T11:23:02.616 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-09-13T11:23:02.616 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_version": 6, 2024-09-13T11:23:02.616 INFO:teuthology.orchestra.run.smithi001.stdout: "full_ratio": 0.94999998807907104, 2024-09-13T11:23:02.616 INFO:teuthology.orchestra.run.smithi001.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-09-13T11:23:02.616 INFO:teuthology.orchestra.run.smithi001.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-13T11:23:02.617 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_snapshot": "", 2024-09-13T11:23:02.617 INFO:teuthology.orchestra.run.smithi001.stdout: "pool_max": 3, 2024-09-13T11:23:02.617 INFO:teuthology.orchestra.run.smithi001.stdout: "max_osd": 16, 2024-09-13T11:23:02.617 INFO:teuthology.orchestra.run.smithi001.stdout: "require_min_compat_client": "luminous", 2024-09-13T11:23:02.617 INFO:teuthology.orchestra.run.smithi001.stdout: "min_compat_client": "jewel", 2024-09-13T11:23:02.618 INFO:teuthology.orchestra.run.smithi001.stdout: "require_osd_release": "squid", 2024-09-13T11:23:02.618 INFO:teuthology.orchestra.run.smithi001.stdout: "allow_crimson": false, 2024-09-13T11:23:02.618 INFO:teuthology.orchestra.run.smithi001.stdout: "pools": [ 2024-09-13T11:23:02.618 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:02.618 INFO:teuthology.orchestra.run.smithi001.stdout: "pool": 1, 2024-09-13T11:23:02.618 INFO:teuthology.orchestra.run.smithi001.stdout: "pool_name": ".mgr", 2024-09-13T11:23:02.619 INFO:teuthology.orchestra.run.smithi001.stdout: "create_time": "2024-09-13T11:22:10.162266+0000", 2024-09-13T11:23:02.619 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-09-13T11:23:02.619 INFO:teuthology.orchestra.run.smithi001.stdout: "flags_names": "hashpspool", 2024-09-13T11:23:02.619 INFO:teuthology.orchestra.run.smithi001.stdout: "type": 1, 2024-09-13T11:23:02.619 INFO:teuthology.orchestra.run.smithi001.stdout: "size": 2, 2024-09-13T11:23:02.620 INFO:teuthology.orchestra.run.smithi001.stdout: "min_size": 2, 2024-09-13T11:23:02.620 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_rule": 0, 2024-09-13T11:23:02.620 INFO:teuthology.orchestra.run.smithi001.stdout: "peering_crush_bucket_count": 0, 2024-09-13T11:23:02.620 INFO:teuthology.orchestra.run.smithi001.stdout: "peering_crush_bucket_target": 0, 2024-09-13T11:23:02.620 INFO:teuthology.orchestra.run.smithi001.stdout: "peering_crush_bucket_barrier": 0, 2024-09-13T11:23:02.621 INFO:teuthology.orchestra.run.smithi001.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-13T11:23:02.621 INFO:teuthology.orchestra.run.smithi001.stdout: "is_stretch_pool": false, 2024-09-13T11:23:02.621 INFO:teuthology.orchestra.run.smithi001.stdout: "object_hash": 2, 2024-09-13T11:23:02.621 INFO:teuthology.orchestra.run.smithi001.stdout: "pg_autoscale_mode": "off", 2024-09-13T11:23:02.621 INFO:teuthology.orchestra.run.smithi001.stdout: "pg_num": 1, 2024-09-13T11:23:02.621 INFO:teuthology.orchestra.run.smithi001.stdout: "pg_placement_num": 1, 2024-09-13T11:23:02.622 INFO:teuthology.orchestra.run.smithi001.stdout: "pg_placement_num_target": 1, 2024-09-13T11:23:02.622 INFO:teuthology.orchestra.run.smithi001.stdout: "pg_num_target": 1, 2024-09-13T11:23:02.622 INFO:teuthology.orchestra.run.smithi001.stdout: "pg_num_pending": 1, 2024-09-13T11:23:02.622 INFO:teuthology.orchestra.run.smithi001.stdout: "last_pg_merge_meta": { 2024-09-13T11:23:02.622 INFO:teuthology.orchestra.run.smithi001.stdout: "source_pgid": "0.0", 2024-09-13T11:23:02.623 INFO:teuthology.orchestra.run.smithi001.stdout: "ready_epoch": 0, 2024-09-13T11:23:02.623 INFO:teuthology.orchestra.run.smithi001.stdout: "last_epoch_started": 0, 2024-09-13T11:23:02.623 INFO:teuthology.orchestra.run.smithi001.stdout: "last_epoch_clean": 0, 2024-09-13T11:23:02.623 INFO:teuthology.orchestra.run.smithi001.stdout: "source_version": "0'0", 2024-09-13T11:23:02.623 INFO:teuthology.orchestra.run.smithi001.stdout: "target_version": "0'0" 2024-09-13T11:23:02.623 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:02.624 INFO:teuthology.orchestra.run.smithi001.stdout: "last_change": "25", 2024-09-13T11:23:02.624 INFO:teuthology.orchestra.run.smithi001.stdout: "last_force_op_resend": "0", 2024-09-13T11:23:02.624 INFO:teuthology.orchestra.run.smithi001.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-13T11:23:02.624 INFO:teuthology.orchestra.run.smithi001.stdout: "last_force_op_resend_preluminous": "0", 2024-09-13T11:23:02.624 INFO:teuthology.orchestra.run.smithi001.stdout: "auid": 0, 2024-09-13T11:23:02.625 INFO:teuthology.orchestra.run.smithi001.stdout: "snap_mode": "selfmanaged", 2024-09-13T11:23:02.625 INFO:teuthology.orchestra.run.smithi001.stdout: "snap_seq": 0, 2024-09-13T11:23:02.625 INFO:teuthology.orchestra.run.smithi001.stdout: "snap_epoch": 0, 2024-09-13T11:23:02.625 INFO:teuthology.orchestra.run.smithi001.stdout: "pool_snaps": [], 2024-09-13T11:23:02.625 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_snaps": "[]", 2024-09-13T11:23:02.625 INFO:teuthology.orchestra.run.smithi001.stdout: "quota_max_bytes": 0, 2024-09-13T11:23:02.626 INFO:teuthology.orchestra.run.smithi001.stdout: "quota_max_objects": 0, 2024-09-13T11:23:02.626 INFO:teuthology.orchestra.run.smithi001.stdout: "tiers": [], 2024-09-13T11:23:02.626 INFO:teuthology.orchestra.run.smithi001.stdout: "tier_of": -1, 2024-09-13T11:23:02.626 INFO:teuthology.orchestra.run.smithi001.stdout: "read_tier": -1, 2024-09-13T11:23:02.626 INFO:teuthology.orchestra.run.smithi001.stdout: "write_tier": -1, 2024-09-13T11:23:02.627 INFO:teuthology.orchestra.run.smithi001.stdout: "cache_mode": "none", 2024-09-13T11:23:02.627 INFO:teuthology.orchestra.run.smithi001.stdout: "target_max_bytes": 0, 2024-09-13T11:23:02.627 INFO:teuthology.orchestra.run.smithi001.stdout: "target_max_objects": 0, 2024-09-13T11:23:02.627 INFO:teuthology.orchestra.run.smithi001.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-13T11:23:02.627 INFO:teuthology.orchestra.run.smithi001.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-13T11:23:02.628 INFO:teuthology.orchestra.run.smithi001.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-13T11:23:02.628 INFO:teuthology.orchestra.run.smithi001.stdout: "cache_min_flush_age": 0, 2024-09-13T11:23:02.628 INFO:teuthology.orchestra.run.smithi001.stdout: "cache_min_evict_age": 0, 2024-09-13T11:23:02.628 INFO:teuthology.orchestra.run.smithi001.stdout: "erasure_code_profile": "", 2024-09-13T11:23:02.628 INFO:teuthology.orchestra.run.smithi001.stdout: "hit_set_params": { 2024-09-13T11:23:02.628 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "none" 2024-09-13T11:23:02.629 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:02.629 INFO:teuthology.orchestra.run.smithi001.stdout: "hit_set_period": 0, 2024-09-13T11:23:02.629 INFO:teuthology.orchestra.run.smithi001.stdout: "hit_set_count": 0, 2024-09-13T11:23:02.629 INFO:teuthology.orchestra.run.smithi001.stdout: "use_gmt_hitset": true, 2024-09-13T11:23:02.629 INFO:teuthology.orchestra.run.smithi001.stdout: "min_read_recency_for_promote": 0, 2024-09-13T11:23:02.630 INFO:teuthology.orchestra.run.smithi001.stdout: "min_write_recency_for_promote": 0, 2024-09-13T11:23:02.630 INFO:teuthology.orchestra.run.smithi001.stdout: "hit_set_grade_decay_rate": 0, 2024-09-13T11:23:02.630 INFO:teuthology.orchestra.run.smithi001.stdout: "hit_set_search_last_n": 0, 2024-09-13T11:23:02.630 INFO:teuthology.orchestra.run.smithi001.stdout: "grade_table": [], 2024-09-13T11:23:02.630 INFO:teuthology.orchestra.run.smithi001.stdout: "stripe_width": 0, 2024-09-13T11:23:02.631 INFO:teuthology.orchestra.run.smithi001.stdout: "expected_num_objects": 0, 2024-09-13T11:23:02.631 INFO:teuthology.orchestra.run.smithi001.stdout: "fast_read": false, 2024-09-13T11:23:02.631 INFO:teuthology.orchestra.run.smithi001.stdout: "options": { 2024-09-13T11:23:02.631 INFO:teuthology.orchestra.run.smithi001.stdout: "pg_num_max": 32, 2024-09-13T11:23:02.631 INFO:teuthology.orchestra.run.smithi001.stdout: "pg_num_min": 1 2024-09-13T11:23:02.632 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:02.632 INFO:teuthology.orchestra.run.smithi001.stdout: "application_metadata": { 2024-09-13T11:23:02.632 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": {} 2024-09-13T11:23:02.632 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:02.632 INFO:teuthology.orchestra.run.smithi001.stdout: "read_balance": { 2024-09-13T11:23:02.633 INFO:teuthology.orchestra.run.smithi001.stdout: "score_type": "Fair distribution", 2024-09-13T11:23:02.633 INFO:teuthology.orchestra.run.smithi001.stdout: "score_acting": 15.380000114440918, 2024-09-13T11:23:02.633 INFO:teuthology.orchestra.run.smithi001.stdout: "score_stable": 15.380000114440918, 2024-09-13T11:23:02.633 INFO:teuthology.orchestra.run.smithi001.stdout: "optimal_score": 0.12999999523162842, 2024-09-13T11:23:02.633 INFO:teuthology.orchestra.run.smithi001.stdout: "raw_score_acting": 2, 2024-09-13T11:23:02.633 INFO:teuthology.orchestra.run.smithi001.stdout: "raw_score_stable": 2, 2024-09-13T11:23:02.634 INFO:teuthology.orchestra.run.smithi001.stdout: "primary_affinity_weighted": 1, 2024-09-13T11:23:02.634 INFO:teuthology.orchestra.run.smithi001.stdout: "average_primary_affinity": 1, 2024-09-13T11:23:02.634 INFO:teuthology.orchestra.run.smithi001.stdout: "average_primary_affinity_weighted": 1 2024-09-13T11:23:02.634 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:02.634 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:02.635 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:02.635 INFO:teuthology.orchestra.run.smithi001.stdout: "pool": 2, 2024-09-13T11:23:02.635 INFO:teuthology.orchestra.run.smithi001.stdout: "pool_name": "rbd", 2024-09-13T11:23:02.635 INFO:teuthology.orchestra.run.smithi001.stdout: "create_time": "2024-09-13T11:22:12.235549+0000", 2024-09-13T11:23:02.635 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 8193, 2024-09-13T11:23:02.635 INFO:teuthology.orchestra.run.smithi001.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-09-13T11:23:02.636 INFO:teuthology.orchestra.run.smithi001.stdout: "type": 1, 2024-09-13T11:23:02.636 INFO:teuthology.orchestra.run.smithi001.stdout: "size": 2, 2024-09-13T11:23:02.636 INFO:teuthology.orchestra.run.smithi001.stdout: "min_size": 2, 2024-09-13T11:23:02.636 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_rule": 0, 2024-09-13T11:23:02.636 INFO:teuthology.orchestra.run.smithi001.stdout: "peering_crush_bucket_count": 0, 2024-09-13T11:23:02.636 INFO:teuthology.orchestra.run.smithi001.stdout: "peering_crush_bucket_target": 0, 2024-09-13T11:23:02.636 INFO:teuthology.orchestra.run.smithi001.stdout: "peering_crush_bucket_barrier": 0, 2024-09-13T11:23:02.636 INFO:teuthology.orchestra.run.smithi001.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-13T11:23:02.637 INFO:teuthology.orchestra.run.smithi001.stdout: "is_stretch_pool": false, 2024-09-13T11:23:02.637 INFO:teuthology.orchestra.run.smithi001.stdout: "object_hash": 2, 2024-09-13T11:23:02.637 INFO:teuthology.orchestra.run.smithi001.stdout: "pg_autoscale_mode": "off", 2024-09-13T11:23:02.637 INFO:teuthology.orchestra.run.smithi001.stdout: "pg_num": 8, 2024-09-13T11:23:02.637 INFO:teuthology.orchestra.run.smithi001.stdout: "pg_placement_num": 8, 2024-09-13T11:23:02.637 INFO:teuthology.orchestra.run.smithi001.stdout: "pg_placement_num_target": 8, 2024-09-13T11:23:02.638 INFO:teuthology.orchestra.run.smithi001.stdout: "pg_num_target": 8, 2024-09-13T11:23:02.638 INFO:teuthology.orchestra.run.smithi001.stdout: "pg_num_pending": 8, 2024-09-13T11:23:02.638 INFO:teuthology.orchestra.run.smithi001.stdout: "last_pg_merge_meta": { 2024-09-13T11:23:02.638 INFO:teuthology.orchestra.run.smithi001.stdout: "source_pgid": "0.0", 2024-09-13T11:23:02.638 INFO:teuthology.orchestra.run.smithi001.stdout: "ready_epoch": 0, 2024-09-13T11:23:02.638 INFO:teuthology.orchestra.run.smithi001.stdout: "last_epoch_started": 0, 2024-09-13T11:23:02.638 INFO:teuthology.orchestra.run.smithi001.stdout: "last_epoch_clean": 0, 2024-09-13T11:23:02.638 INFO:teuthology.orchestra.run.smithi001.stdout: "source_version": "0'0", 2024-09-13T11:23:02.638 INFO:teuthology.orchestra.run.smithi001.stdout: "target_version": "0'0" 2024-09-13T11:23:02.638 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:02.638 INFO:teuthology.orchestra.run.smithi001.stdout: "last_change": "28", 2024-09-13T11:23:02.638 INFO:teuthology.orchestra.run.smithi001.stdout: "last_force_op_resend": "0", 2024-09-13T11:23:02.638 INFO:teuthology.orchestra.run.smithi001.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-13T11:23:02.639 INFO:teuthology.orchestra.run.smithi001.stdout: "last_force_op_resend_preluminous": "0", 2024-09-13T11:23:02.639 INFO:teuthology.orchestra.run.smithi001.stdout: "auid": 0, 2024-09-13T11:23:02.639 INFO:teuthology.orchestra.run.smithi001.stdout: "snap_mode": "selfmanaged", 2024-09-13T11:23:02.639 INFO:teuthology.orchestra.run.smithi001.stdout: "snap_seq": 2, 2024-09-13T11:23:02.639 INFO:teuthology.orchestra.run.smithi001.stdout: "snap_epoch": 28, 2024-09-13T11:23:02.639 INFO:teuthology.orchestra.run.smithi001.stdout: "pool_snaps": [], 2024-09-13T11:23:02.639 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_snaps": "[]", 2024-09-13T11:23:02.639 INFO:teuthology.orchestra.run.smithi001.stdout: "quota_max_bytes": 0, 2024-09-13T11:23:02.639 INFO:teuthology.orchestra.run.smithi001.stdout: "quota_max_objects": 0, 2024-09-13T11:23:02.639 INFO:teuthology.orchestra.run.smithi001.stdout: "tiers": [], 2024-09-13T11:23:02.639 INFO:teuthology.orchestra.run.smithi001.stdout: "tier_of": -1, 2024-09-13T11:23:02.639 INFO:teuthology.orchestra.run.smithi001.stdout: "read_tier": -1, 2024-09-13T11:23:02.639 INFO:teuthology.orchestra.run.smithi001.stdout: "write_tier": -1, 2024-09-13T11:23:02.639 INFO:teuthology.orchestra.run.smithi001.stdout: "cache_mode": "none", 2024-09-13T11:23:02.640 INFO:teuthology.orchestra.run.smithi001.stdout: "target_max_bytes": 0, 2024-09-13T11:23:02.640 INFO:teuthology.orchestra.run.smithi001.stdout: "target_max_objects": 0, 2024-09-13T11:23:02.640 INFO:teuthology.orchestra.run.smithi001.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-13T11:23:02.640 INFO:teuthology.orchestra.run.smithi001.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-13T11:23:02.640 INFO:teuthology.orchestra.run.smithi001.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-13T11:23:02.640 INFO:teuthology.orchestra.run.smithi001.stdout: "cache_min_flush_age": 0, 2024-09-13T11:23:02.640 INFO:teuthology.orchestra.run.smithi001.stdout: "cache_min_evict_age": 0, 2024-09-13T11:23:02.640 INFO:teuthology.orchestra.run.smithi001.stdout: "erasure_code_profile": "", 2024-09-13T11:23:02.640 INFO:teuthology.orchestra.run.smithi001.stdout: "hit_set_params": { 2024-09-13T11:23:02.640 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "none" 2024-09-13T11:23:02.640 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:02.640 INFO:teuthology.orchestra.run.smithi001.stdout: "hit_set_period": 0, 2024-09-13T11:23:02.640 INFO:teuthology.orchestra.run.smithi001.stdout: "hit_set_count": 0, 2024-09-13T11:23:02.640 INFO:teuthology.orchestra.run.smithi001.stdout: "use_gmt_hitset": true, 2024-09-13T11:23:02.641 INFO:teuthology.orchestra.run.smithi001.stdout: "min_read_recency_for_promote": 0, 2024-09-13T11:23:02.641 INFO:teuthology.orchestra.run.smithi001.stdout: "min_write_recency_for_promote": 0, 2024-09-13T11:23:02.641 INFO:teuthology.orchestra.run.smithi001.stdout: "hit_set_grade_decay_rate": 0, 2024-09-13T11:23:02.641 INFO:teuthology.orchestra.run.smithi001.stdout: "hit_set_search_last_n": 0, 2024-09-13T11:23:02.641 INFO:teuthology.orchestra.run.smithi001.stdout: "grade_table": [], 2024-09-13T11:23:02.641 INFO:teuthology.orchestra.run.smithi001.stdout: "stripe_width": 0, 2024-09-13T11:23:02.641 INFO:teuthology.orchestra.run.smithi001.stdout: "expected_num_objects": 0, 2024-09-13T11:23:02.641 INFO:teuthology.orchestra.run.smithi001.stdout: "fast_read": false, 2024-09-13T11:23:02.641 INFO:teuthology.orchestra.run.smithi001.stdout: "options": {}, 2024-09-13T11:23:02.641 INFO:teuthology.orchestra.run.smithi001.stdout: "application_metadata": { 2024-09-13T11:23:02.641 INFO:teuthology.orchestra.run.smithi001.stdout: "rbd": {} 2024-09-13T11:23:02.641 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:02.641 INFO:teuthology.orchestra.run.smithi001.stdout: "read_balance": { 2024-09-13T11:23:02.641 INFO:teuthology.orchestra.run.smithi001.stdout: "score_type": "Fair distribution", 2024-09-13T11:23:02.641 INFO:teuthology.orchestra.run.smithi001.stdout: "score_acting": 4.0199999809265137, 2024-09-13T11:23:02.642 INFO:teuthology.orchestra.run.smithi001.stdout: "score_stable": 4.0199999809265137, 2024-09-13T11:23:02.642 INFO:teuthology.orchestra.run.smithi001.stdout: "optimal_score": 0.56000000238418579, 2024-09-13T11:23:02.642 INFO:teuthology.orchestra.run.smithi001.stdout: "raw_score_acting": 2.25, 2024-09-13T11:23:02.642 INFO:teuthology.orchestra.run.smithi001.stdout: "raw_score_stable": 2.25, 2024-09-13T11:23:02.642 INFO:teuthology.orchestra.run.smithi001.stdout: "primary_affinity_weighted": 1, 2024-09-13T11:23:02.642 INFO:teuthology.orchestra.run.smithi001.stdout: "average_primary_affinity": 1, 2024-09-13T11:23:02.642 INFO:teuthology.orchestra.run.smithi001.stdout: "average_primary_affinity_weighted": 1 2024-09-13T11:23:02.642 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:02.642 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:02.643 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:02.643 INFO:teuthology.orchestra.run.smithi001.stdout: "pool": 3, 2024-09-13T11:23:02.643 INFO:teuthology.orchestra.run.smithi001.stdout: "pool_name": "unique_pool_0", 2024-09-13T11:23:02.643 INFO:teuthology.orchestra.run.smithi001.stdout: "create_time": "2024-09-13T11:22:24.305521+0000", 2024-09-13T11:23:02.643 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-09-13T11:23:02.643 INFO:teuthology.orchestra.run.smithi001.stdout: "flags_names": "hashpspool", 2024-09-13T11:23:02.643 INFO:teuthology.orchestra.run.smithi001.stdout: "type": 1, 2024-09-13T11:23:02.643 INFO:teuthology.orchestra.run.smithi001.stdout: "size": 2, 2024-09-13T11:23:02.643 INFO:teuthology.orchestra.run.smithi001.stdout: "min_size": 2, 2024-09-13T11:23:02.643 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_rule": 0, 2024-09-13T11:23:02.643 INFO:teuthology.orchestra.run.smithi001.stdout: "peering_crush_bucket_count": 0, 2024-09-13T11:23:02.644 INFO:teuthology.orchestra.run.smithi001.stdout: "peering_crush_bucket_target": 0, 2024-09-13T11:23:02.644 INFO:teuthology.orchestra.run.smithi001.stdout: "peering_crush_bucket_barrier": 0, 2024-09-13T11:23:02.644 INFO:teuthology.orchestra.run.smithi001.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-13T11:23:02.644 INFO:teuthology.orchestra.run.smithi001.stdout: "is_stretch_pool": false, 2024-09-13T11:23:02.644 INFO:teuthology.orchestra.run.smithi001.stdout: "object_hash": 2, 2024-09-13T11:23:02.644 INFO:teuthology.orchestra.run.smithi001.stdout: "pg_autoscale_mode": "off", 2024-09-13T11:23:02.644 INFO:teuthology.orchestra.run.smithi001.stdout: "pg_num": 16, 2024-09-13T11:23:02.644 INFO:teuthology.orchestra.run.smithi001.stdout: "pg_placement_num": 16, 2024-09-13T11:23:02.644 INFO:teuthology.orchestra.run.smithi001.stdout: "pg_placement_num_target": 16, 2024-09-13T11:23:02.644 INFO:teuthology.orchestra.run.smithi001.stdout: "pg_num_target": 16, 2024-09-13T11:23:02.644 INFO:teuthology.orchestra.run.smithi001.stdout: "pg_num_pending": 16, 2024-09-13T11:23:02.644 INFO:teuthology.orchestra.run.smithi001.stdout: "last_pg_merge_meta": { 2024-09-13T11:23:02.644 INFO:teuthology.orchestra.run.smithi001.stdout: "source_pgid": "0.0", 2024-09-13T11:23:02.645 INFO:teuthology.orchestra.run.smithi001.stdout: "ready_epoch": 0, 2024-09-13T11:23:02.645 INFO:teuthology.orchestra.run.smithi001.stdout: "last_epoch_started": 0, 2024-09-13T11:23:02.645 INFO:teuthology.orchestra.run.smithi001.stdout: "last_epoch_clean": 0, 2024-09-13T11:23:02.645 INFO:teuthology.orchestra.run.smithi001.stdout: "source_version": "0'0", 2024-09-13T11:23:02.645 INFO:teuthology.orchestra.run.smithi001.stdout: "target_version": "0'0" 2024-09-13T11:23:02.645 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:02.645 INFO:teuthology.orchestra.run.smithi001.stdout: "last_change": "32", 2024-09-13T11:23:02.645 INFO:teuthology.orchestra.run.smithi001.stdout: "last_force_op_resend": "0", 2024-09-13T11:23:02.645 INFO:teuthology.orchestra.run.smithi001.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-13T11:23:02.645 INFO:teuthology.orchestra.run.smithi001.stdout: "last_force_op_resend_preluminous": "0", 2024-09-13T11:23:02.645 INFO:teuthology.orchestra.run.smithi001.stdout: "auid": 0, 2024-09-13T11:23:02.645 INFO:teuthology.orchestra.run.smithi001.stdout: "snap_mode": "selfmanaged", 2024-09-13T11:23:02.645 INFO:teuthology.orchestra.run.smithi001.stdout: "snap_seq": 0, 2024-09-13T11:23:02.645 INFO:teuthology.orchestra.run.smithi001.stdout: "snap_epoch": 0, 2024-09-13T11:23:02.646 INFO:teuthology.orchestra.run.smithi001.stdout: "pool_snaps": [], 2024-09-13T11:23:02.646 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_snaps": "[]", 2024-09-13T11:23:02.646 INFO:teuthology.orchestra.run.smithi001.stdout: "quota_max_bytes": 0, 2024-09-13T11:23:02.646 INFO:teuthology.orchestra.run.smithi001.stdout: "quota_max_objects": 0, 2024-09-13T11:23:02.646 INFO:teuthology.orchestra.run.smithi001.stdout: "tiers": [], 2024-09-13T11:23:02.646 INFO:teuthology.orchestra.run.smithi001.stdout: "tier_of": -1, 2024-09-13T11:23:02.646 INFO:teuthology.orchestra.run.smithi001.stdout: "read_tier": -1, 2024-09-13T11:23:02.646 INFO:teuthology.orchestra.run.smithi001.stdout: "write_tier": -1, 2024-09-13T11:23:02.647 INFO:teuthology.orchestra.run.smithi001.stdout: "cache_mode": "none", 2024-09-13T11:23:02.647 INFO:teuthology.orchestra.run.smithi001.stdout: "target_max_bytes": 0, 2024-09-13T11:23:02.647 INFO:teuthology.orchestra.run.smithi001.stdout: "target_max_objects": 0, 2024-09-13T11:23:02.647 INFO:teuthology.orchestra.run.smithi001.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-13T11:23:02.647 INFO:teuthology.orchestra.run.smithi001.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-13T11:23:02.647 INFO:teuthology.orchestra.run.smithi001.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-13T11:23:02.647 INFO:teuthology.orchestra.run.smithi001.stdout: "cache_min_flush_age": 0, 2024-09-13T11:23:02.647 INFO:teuthology.orchestra.run.smithi001.stdout: "cache_min_evict_age": 0, 2024-09-13T11:23:02.647 INFO:teuthology.orchestra.run.smithi001.stdout: "erasure_code_profile": "", 2024-09-13T11:23:02.647 INFO:teuthology.orchestra.run.smithi001.stdout: "hit_set_params": { 2024-09-13T11:23:02.647 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "none" 2024-09-13T11:23:02.647 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:02.648 INFO:teuthology.orchestra.run.smithi001.stdout: "hit_set_period": 0, 2024-09-13T11:23:02.648 INFO:teuthology.orchestra.run.smithi001.stdout: "hit_set_count": 0, 2024-09-13T11:23:02.648 INFO:teuthology.orchestra.run.smithi001.stdout: "use_gmt_hitset": true, 2024-09-13T11:23:02.648 INFO:teuthology.orchestra.run.smithi001.stdout: "min_read_recency_for_promote": 0, 2024-09-13T11:23:02.648 INFO:teuthology.orchestra.run.smithi001.stdout: "min_write_recency_for_promote": 0, 2024-09-13T11:23:02.648 INFO:teuthology.orchestra.run.smithi001.stdout: "hit_set_grade_decay_rate": 0, 2024-09-13T11:23:02.648 INFO:teuthology.orchestra.run.smithi001.stdout: "hit_set_search_last_n": 0, 2024-09-13T11:23:02.649 INFO:teuthology.orchestra.run.smithi001.stdout: "grade_table": [], 2024-09-13T11:23:02.649 INFO:teuthology.orchestra.run.smithi001.stdout: "stripe_width": 0, 2024-09-13T11:23:02.649 INFO:teuthology.orchestra.run.smithi001.stdout: "expected_num_objects": 0, 2024-09-13T11:23:02.649 INFO:teuthology.orchestra.run.smithi001.stdout: "fast_read": false, 2024-09-13T11:23:02.649 INFO:teuthology.orchestra.run.smithi001.stdout: "options": {}, 2024-09-13T11:23:02.649 INFO:teuthology.orchestra.run.smithi001.stdout: "application_metadata": { 2024-09-13T11:23:02.649 INFO:teuthology.orchestra.run.smithi001.stdout: "rados": {} 2024-09-13T11:23:02.649 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:02.649 INFO:teuthology.orchestra.run.smithi001.stdout: "read_balance": { 2024-09-13T11:23:02.649 INFO:teuthology.orchestra.run.smithi001.stdout: "score_type": "Fair distribution", 2024-09-13T11:23:02.649 INFO:teuthology.orchestra.run.smithi001.stdout: "score_acting": 4, 2024-09-13T11:23:02.649 INFO:teuthology.orchestra.run.smithi001.stdout: "score_stable": 4, 2024-09-13T11:23:02.649 INFO:teuthology.orchestra.run.smithi001.stdout: "optimal_score": 0.75, 2024-09-13T11:23:02.650 INFO:teuthology.orchestra.run.smithi001.stdout: "raw_score_acting": 3, 2024-09-13T11:23:02.650 INFO:teuthology.orchestra.run.smithi001.stdout: "raw_score_stable": 3, 2024-09-13T11:23:02.650 INFO:teuthology.orchestra.run.smithi001.stdout: "primary_affinity_weighted": 1, 2024-09-13T11:23:02.650 INFO:teuthology.orchestra.run.smithi001.stdout: "average_primary_affinity": 1, 2024-09-13T11:23:02.650 INFO:teuthology.orchestra.run.smithi001.stdout: "average_primary_affinity_weighted": 1 2024-09-13T11:23:02.650 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:02.650 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:02.650 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-09-13T11:23:02.650 INFO:teuthology.orchestra.run.smithi001.stdout: "osds": [ 2024-09-13T11:23:02.650 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:02.650 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": 0, 2024-09-13T11:23:02.650 INFO:teuthology.orchestra.run.smithi001.stdout: "uuid": "0fc12b3c-e0de-4546-b935-db77efe7c567", 2024-09-13T11:23:02.650 INFO:teuthology.orchestra.run.smithi001.stdout: "up": 1, 2024-09-13T11:23:02.650 INFO:teuthology.orchestra.run.smithi001.stdout: "in": 1, 2024-09-13T11:23:02.651 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 1, 2024-09-13T11:23:02.651 INFO:teuthology.orchestra.run.smithi001.stdout: "primary_affinity": 1, 2024-09-13T11:23:02.651 INFO:teuthology.orchestra.run.smithi001.stdout: "last_clean_begin": 0, 2024-09-13T11:23:02.651 INFO:teuthology.orchestra.run.smithi001.stdout: "last_clean_end": 0, 2024-09-13T11:23:02.651 INFO:teuthology.orchestra.run.smithi001.stdout: "up_from": 22, 2024-09-13T11:23:02.651 INFO:teuthology.orchestra.run.smithi001.stdout: "up_thru": 0, 2024-09-13T11:23:02.651 INFO:teuthology.orchestra.run.smithi001.stdout: "down_at": 0, 2024-09-13T11:23:02.651 INFO:teuthology.orchestra.run.smithi001.stdout: "lost_at": 0, 2024-09-13T11:23:02.651 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-09-13T11:23:02.651 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:23:02.652 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:02.652 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:23:02.652 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6800", 2024-09-13T11:23:02.652 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 1544776477 2024-09-13T11:23:02.652 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:02.652 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:02.652 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:02.652 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_addrs": { 2024-09-13T11:23:02.653 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:23:02.653 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:02.653 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:23:02.653 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6801", 2024-09-13T11:23:02.653 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 1544776477 2024-09-13T11:23:02.653 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:02.653 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:02.653 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:02.653 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_back_addrs": { 2024-09-13T11:23:02.653 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:23:02.654 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:02.654 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:23:02.654 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6803", 2024-09-13T11:23:02.654 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 1544776477 2024-09-13T11:23:02.654 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:02.654 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:02.655 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:02.655 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_front_addrs": { 2024-09-13T11:23:02.655 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:23:02.655 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:02.655 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:23:02.655 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6802", 2024-09-13T11:23:02.655 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 1544776477 2024-09-13T11:23:02.655 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:02.655 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:02.655 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:02.655 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6800/1544776477", 2024-09-13T11:23:02.655 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_addr": "172.21.15.1:6801/1544776477", 2024-09-13T11:23:02.655 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_back_addr": "172.21.15.1:6803/1544776477", 2024-09-13T11:23:02.656 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_front_addr": "172.21.15.1:6802/1544776477", 2024-09-13T11:23:02.656 INFO:teuthology.orchestra.run.smithi001.stdout: "state": [ 2024-09-13T11:23:02.656 INFO:teuthology.orchestra.run.smithi001.stdout: "exists", 2024-09-13T11:23:02.656 INFO:teuthology.orchestra.run.smithi001.stdout: "up" 2024-09-13T11:23:02.656 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:02.656 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:02.656 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:02.656 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": 1, 2024-09-13T11:23:02.656 INFO:teuthology.orchestra.run.smithi001.stdout: "uuid": "b77fe331-cc92-4adb-b146-af5c380f6d4f", 2024-09-13T11:23:02.656 INFO:teuthology.orchestra.run.smithi001.stdout: "up": 1, 2024-09-13T11:23:02.656 INFO:teuthology.orchestra.run.smithi001.stdout: "in": 1, 2024-09-13T11:23:02.656 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 1, 2024-09-13T11:23:02.656 INFO:teuthology.orchestra.run.smithi001.stdout: "primary_affinity": 1, 2024-09-13T11:23:02.657 INFO:teuthology.orchestra.run.smithi001.stdout: "last_clean_begin": 23, 2024-09-13T11:23:02.657 INFO:teuthology.orchestra.run.smithi001.stdout: "last_clean_end": 29, 2024-09-13T11:23:02.657 INFO:teuthology.orchestra.run.smithi001.stdout: "up_from": 40, 2024-09-13T11:23:02.657 INFO:teuthology.orchestra.run.smithi001.stdout: "up_thru": 40, 2024-09-13T11:23:02.657 INFO:teuthology.orchestra.run.smithi001.stdout: "down_at": 30, 2024-09-13T11:23:02.657 INFO:teuthology.orchestra.run.smithi001.stdout: "lost_at": 0, 2024-09-13T11:23:02.657 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-09-13T11:23:02.657 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:23:02.657 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:02.657 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:23:02.657 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.92:6800", 2024-09-13T11:23:02.657 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 3591430847 2024-09-13T11:23:02.657 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:02.658 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:02.658 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:02.658 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_addrs": { 2024-09-13T11:23:02.658 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:23:02.658 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:02.658 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:23:02.658 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.92:6801", 2024-09-13T11:23:02.658 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 3591430847 2024-09-13T11:23:02.659 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:02.659 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:02.659 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:02.659 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_back_addrs": { 2024-09-13T11:23:02.659 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:23:02.659 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:02.659 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:23:02.659 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.92:6803", 2024-09-13T11:23:02.659 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 3591430847 2024-09-13T11:23:02.659 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:02.660 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:02.660 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:02.660 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_front_addrs": { 2024-09-13T11:23:02.660 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:23:02.660 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:02.660 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:23:02.660 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.92:6802", 2024-09-13T11:23:02.660 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 3591430847 2024-09-13T11:23:02.660 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:02.660 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:02.660 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:02.660 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.92:6800/3591430847", 2024-09-13T11:23:02.660 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_addr": "172.21.15.92:6801/3591430847", 2024-09-13T11:23:02.661 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_back_addr": "172.21.15.92:6803/3591430847", 2024-09-13T11:23:02.661 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_front_addr": "172.21.15.92:6802/3591430847", 2024-09-13T11:23:02.661 INFO:teuthology.orchestra.run.smithi001.stdout: "state": [ 2024-09-13T11:23:02.661 INFO:teuthology.orchestra.run.smithi001.stdout: "exists", 2024-09-13T11:23:02.661 INFO:teuthology.orchestra.run.smithi001.stdout: "up" 2024-09-13T11:23:02.661 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:02.661 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:02.661 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:02.661 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": 2, 2024-09-13T11:23:02.661 INFO:teuthology.orchestra.run.smithi001.stdout: "uuid": "c9d031c8-e885-4229-affa-08e1729d00e8", 2024-09-13T11:23:02.661 INFO:teuthology.orchestra.run.smithi001.stdout: "up": 1, 2024-09-13T11:23:02.661 INFO:teuthology.orchestra.run.smithi001.stdout: "in": 1, 2024-09-13T11:23:02.661 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 1, 2024-09-13T11:23:02.661 INFO:teuthology.orchestra.run.smithi001.stdout: "primary_affinity": 1, 2024-09-13T11:23:02.662 INFO:teuthology.orchestra.run.smithi001.stdout: "last_clean_begin": 0, 2024-09-13T11:23:02.662 INFO:teuthology.orchestra.run.smithi001.stdout: "last_clean_end": 0, 2024-09-13T11:23:02.662 INFO:teuthology.orchestra.run.smithi001.stdout: "up_from": 22, 2024-09-13T11:23:02.662 INFO:teuthology.orchestra.run.smithi001.stdout: "up_thru": 0, 2024-09-13T11:23:02.662 INFO:teuthology.orchestra.run.smithi001.stdout: "down_at": 0, 2024-09-13T11:23:02.662 INFO:teuthology.orchestra.run.smithi001.stdout: "lost_at": 0, 2024-09-13T11:23:02.662 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-09-13T11:23:02.662 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:23:02.662 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:02.662 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:23:02.662 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.162:6812", 2024-09-13T11:23:02.662 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 2712148101 2024-09-13T11:23:02.663 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:02.663 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:02.663 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:02.663 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_addrs": { 2024-09-13T11:23:02.663 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:23:02.663 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:02.663 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:23:02.663 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.162:6813", 2024-09-13T11:23:02.663 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 2712148101 2024-09-13T11:23:02.663 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:02.663 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:02.663 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:02.663 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_back_addrs": { 2024-09-13T11:23:02.664 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:23:02.664 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:02.664 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:23:02.664 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.162:6815", 2024-09-13T11:23:02.664 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 2712148101 2024-09-13T11:23:02.664 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:02.664 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:02.664 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:02.664 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_front_addrs": { 2024-09-13T11:23:02.664 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:23:02.664 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:02.664 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:23:02.664 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.162:6814", 2024-09-13T11:23:02.665 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 2712148101 2024-09-13T11:23:02.665 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:02.665 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:02.665 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:02.665 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.162:6812/2712148101", 2024-09-13T11:23:02.665 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_addr": "172.21.15.162:6813/2712148101", 2024-09-13T11:23:02.665 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_back_addr": "172.21.15.162:6815/2712148101", 2024-09-13T11:23:02.665 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_front_addr": "172.21.15.162:6814/2712148101", 2024-09-13T11:23:02.665 INFO:teuthology.orchestra.run.smithi001.stdout: "state": [ 2024-09-13T11:23:02.665 INFO:teuthology.orchestra.run.smithi001.stdout: "exists", 2024-09-13T11:23:02.665 INFO:teuthology.orchestra.run.smithi001.stdout: "up" 2024-09-13T11:23:02.665 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:02.665 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:02.665 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:02.666 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": 3, 2024-09-13T11:23:02.666 INFO:teuthology.orchestra.run.smithi001.stdout: "uuid": "bf50d71a-6c3a-4fbc-a407-cb81bad14a7b", 2024-09-13T11:23:02.666 INFO:teuthology.orchestra.run.smithi001.stdout: "up": 1, 2024-09-13T11:23:02.666 INFO:teuthology.orchestra.run.smithi001.stdout: "in": 1, 2024-09-13T11:23:02.666 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 1, 2024-09-13T11:23:02.666 INFO:teuthology.orchestra.run.smithi001.stdout: "primary_affinity": 1, 2024-09-13T11:23:02.666 INFO:teuthology.orchestra.run.smithi001.stdout: "last_clean_begin": 0, 2024-09-13T11:23:02.666 INFO:teuthology.orchestra.run.smithi001.stdout: "last_clean_end": 0, 2024-09-13T11:23:02.666 INFO:teuthology.orchestra.run.smithi001.stdout: "up_from": 21, 2024-09-13T11:23:02.666 INFO:teuthology.orchestra.run.smithi001.stdout: "up_thru": 30, 2024-09-13T11:23:02.666 INFO:teuthology.orchestra.run.smithi001.stdout: "down_at": 0, 2024-09-13T11:23:02.666 INFO:teuthology.orchestra.run.smithi001.stdout: "lost_at": 0, 2024-09-13T11:23:02.667 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-09-13T11:23:02.667 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:23:02.667 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:02.667 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:23:02.667 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.177:6801", 2024-09-13T11:23:02.667 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 3367751690 2024-09-13T11:23:02.667 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:02.667 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:02.667 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:02.667 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_addrs": { 2024-09-13T11:23:02.667 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:23:02.667 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:02.668 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:23:02.668 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.177:6802", 2024-09-13T11:23:02.668 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 3367751690 2024-09-13T11:23:02.668 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:02.668 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:02.668 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:02.668 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_back_addrs": { 2024-09-13T11:23:02.668 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:23:02.668 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:02.668 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:23:02.668 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.177:6804", 2024-09-13T11:23:02.668 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 3367751690 2024-09-13T11:23:02.668 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:02.669 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:02.669 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:02.669 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_front_addrs": { 2024-09-13T11:23:02.669 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:23:02.669 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:02.669 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:23:02.669 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.177:6803", 2024-09-13T11:23:02.669 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 3367751690 2024-09-13T11:23:02.669 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:02.669 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:02.670 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:02.670 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.177:6801/3367751690", 2024-09-13T11:23:02.670 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_addr": "172.21.15.177:6802/3367751690", 2024-09-13T11:23:02.670 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_back_addr": "172.21.15.177:6804/3367751690", 2024-09-13T11:23:02.670 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_front_addr": "172.21.15.177:6803/3367751690", 2024-09-13T11:23:02.670 INFO:teuthology.orchestra.run.smithi001.stdout: "state": [ 2024-09-13T11:23:02.670 INFO:teuthology.orchestra.run.smithi001.stdout: "exists", 2024-09-13T11:23:02.670 INFO:teuthology.orchestra.run.smithi001.stdout: "up" 2024-09-13T11:23:02.670 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:02.670 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:02.670 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:02.670 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": 4, 2024-09-13T11:23:02.671 INFO:teuthology.orchestra.run.smithi001.stdout: "uuid": "d1fe3f63-df3f-4a73-a1e2-8d7c8a3f1fa0", 2024-09-13T11:23:02.671 INFO:teuthology.orchestra.run.smithi001.stdout: "up": 1, 2024-09-13T11:23:02.671 INFO:teuthology.orchestra.run.smithi001.stdout: "in": 1, 2024-09-13T11:23:02.671 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 1, 2024-09-13T11:23:02.671 INFO:teuthology.orchestra.run.smithi001.stdout: "primary_affinity": 1, 2024-09-13T11:23:02.671 INFO:teuthology.orchestra.run.smithi001.stdout: "last_clean_begin": 0, 2024-09-13T11:23:02.671 INFO:teuthology.orchestra.run.smithi001.stdout: "last_clean_end": 0, 2024-09-13T11:23:02.671 INFO:teuthology.orchestra.run.smithi001.stdout: "up_from": 22, 2024-09-13T11:23:02.671 INFO:teuthology.orchestra.run.smithi001.stdout: "up_thru": 0, 2024-09-13T11:23:02.671 INFO:teuthology.orchestra.run.smithi001.stdout: "down_at": 0, 2024-09-13T11:23:02.671 INFO:teuthology.orchestra.run.smithi001.stdout: "lost_at": 0, 2024-09-13T11:23:02.672 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-09-13T11:23:02.672 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:23:02.672 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:02.672 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:23:02.672 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6808", 2024-09-13T11:23:02.672 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 3648540926 2024-09-13T11:23:02.672 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:02.672 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:02.672 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:02.672 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_addrs": { 2024-09-13T11:23:02.672 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:23:02.672 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:02.672 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:23:02.673 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6809", 2024-09-13T11:23:02.673 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 3648540926 2024-09-13T11:23:02.673 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:02.673 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:02.673 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:02.673 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_back_addrs": { 2024-09-13T11:23:02.673 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:23:02.673 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:02.673 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:23:02.673 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6811", 2024-09-13T11:23:02.673 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 3648540926 2024-09-13T11:23:02.673 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:02.673 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:02.673 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:02.674 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_front_addrs": { 2024-09-13T11:23:02.674 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:23:02.674 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:02.674 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:23:02.674 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6810", 2024-09-13T11:23:02.674 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 3648540926 2024-09-13T11:23:02.674 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:02.674 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:02.674 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:02.674 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6808/3648540926", 2024-09-13T11:23:02.674 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_addr": "172.21.15.1:6809/3648540926", 2024-09-13T11:23:02.674 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_back_addr": "172.21.15.1:6811/3648540926", 2024-09-13T11:23:02.674 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_front_addr": "172.21.15.1:6810/3648540926", 2024-09-13T11:23:02.674 INFO:teuthology.orchestra.run.smithi001.stdout: "state": [ 2024-09-13T11:23:02.675 INFO:teuthology.orchestra.run.smithi001.stdout: "exists", 2024-09-13T11:23:02.675 INFO:teuthology.orchestra.run.smithi001.stdout: "up" 2024-09-13T11:23:02.675 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:02.675 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:02.675 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:02.675 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": 5, 2024-09-13T11:23:02.675 INFO:teuthology.orchestra.run.smithi001.stdout: "uuid": "e2ead447-2f2c-40c9-82e7-53b8dc41e2af", 2024-09-13T11:23:02.675 INFO:teuthology.orchestra.run.smithi001.stdout: "up": 1, 2024-09-13T11:23:02.675 INFO:teuthology.orchestra.run.smithi001.stdout: "in": 1, 2024-09-13T11:23:02.675 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 1, 2024-09-13T11:23:02.675 INFO:teuthology.orchestra.run.smithi001.stdout: "primary_affinity": 1, 2024-09-13T11:23:02.675 INFO:teuthology.orchestra.run.smithi001.stdout: "last_clean_begin": 0, 2024-09-13T11:23:02.675 INFO:teuthology.orchestra.run.smithi001.stdout: "last_clean_end": 0, 2024-09-13T11:23:02.676 INFO:teuthology.orchestra.run.smithi001.stdout: "up_from": 21, 2024-09-13T11:23:02.676 INFO:teuthology.orchestra.run.smithi001.stdout: "up_thru": 0, 2024-09-13T11:23:02.676 INFO:teuthology.orchestra.run.smithi001.stdout: "down_at": 0, 2024-09-13T11:23:02.676 INFO:teuthology.orchestra.run.smithi001.stdout: "lost_at": 0, 2024-09-13T11:23:02.676 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-09-13T11:23:02.676 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:23:02.676 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:02.676 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:23:02.676 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.92:6804", 2024-09-13T11:23:02.676 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 3399529091 2024-09-13T11:23:02.676 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:02.676 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:02.676 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:02.677 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_addrs": { 2024-09-13T11:23:02.677 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:23:02.677 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:02.677 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:23:02.677 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.92:6805", 2024-09-13T11:23:02.677 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 3399529091 2024-09-13T11:23:02.677 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:02.678 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:02.678 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:02.678 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_back_addrs": { 2024-09-13T11:23:02.678 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:23:02.678 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:02.678 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:23:02.678 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.92:6807", 2024-09-13T11:23:02.678 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 3399529091 2024-09-13T11:23:02.678 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:02.678 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:02.678 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:02.678 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_front_addrs": { 2024-09-13T11:23:02.678 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:23:02.679 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:02.679 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:23:02.679 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.92:6806", 2024-09-13T11:23:02.679 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 3399529091 2024-09-13T11:23:02.679 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:02.679 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:02.679 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:02.679 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.92:6804/3399529091", 2024-09-13T11:23:02.679 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_addr": "172.21.15.92:6805/3399529091", 2024-09-13T11:23:02.679 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_back_addr": "172.21.15.92:6807/3399529091", 2024-09-13T11:23:02.680 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_front_addr": "172.21.15.92:6806/3399529091", 2024-09-13T11:23:02.680 INFO:teuthology.orchestra.run.smithi001.stdout: "state": [ 2024-09-13T11:23:02.680 INFO:teuthology.orchestra.run.smithi001.stdout: "exists", 2024-09-13T11:23:02.680 INFO:teuthology.orchestra.run.smithi001.stdout: "up" 2024-09-13T11:23:02.680 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:02.680 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:02.680 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:02.680 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": 6, 2024-09-13T11:23:02.680 INFO:teuthology.orchestra.run.smithi001.stdout: "uuid": "dbc20439-144e-4ec7-89de-674eb783f263", 2024-09-13T11:23:02.681 INFO:teuthology.orchestra.run.smithi001.stdout: "up": 1, 2024-09-13T11:23:02.681 INFO:teuthology.orchestra.run.smithi001.stdout: "in": 1, 2024-09-13T11:23:02.681 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 1, 2024-09-13T11:23:02.681 INFO:teuthology.orchestra.run.smithi001.stdout: "primary_affinity": 1, 2024-09-13T11:23:02.681 INFO:teuthology.orchestra.run.smithi001.stdout: "last_clean_begin": 0, 2024-09-13T11:23:02.681 INFO:teuthology.orchestra.run.smithi001.stdout: "last_clean_end": 0, 2024-09-13T11:23:02.682 INFO:teuthology.orchestra.run.smithi001.stdout: "up_from": 22, 2024-09-13T11:23:02.682 INFO:teuthology.orchestra.run.smithi001.stdout: "up_thru": 40, 2024-09-13T11:23:02.682 INFO:teuthology.orchestra.run.smithi001.stdout: "down_at": 0, 2024-09-13T11:23:02.682 INFO:teuthology.orchestra.run.smithi001.stdout: "lost_at": 0, 2024-09-13T11:23:02.682 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-09-13T11:23:02.682 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:23:02.682 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:02.682 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:23:02.682 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.162:6808", 2024-09-13T11:23:02.682 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 2287343806 2024-09-13T11:23:02.682 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:02.682 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:02.682 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:02.683 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_addrs": { 2024-09-13T11:23:02.683 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:23:02.683 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:02.683 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:23:02.683 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.162:6809", 2024-09-13T11:23:02.683 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 2287343806 2024-09-13T11:23:02.683 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:02.683 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:02.683 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:02.683 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_back_addrs": { 2024-09-13T11:23:02.683 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:23:02.683 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:02.683 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:23:02.683 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.162:6811", 2024-09-13T11:23:02.684 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 2287343806 2024-09-13T11:23:02.684 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:02.684 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:02.684 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:02.684 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_front_addrs": { 2024-09-13T11:23:02.684 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:23:02.684 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:02.684 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:23:02.684 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.162:6810", 2024-09-13T11:23:02.684 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 2287343806 2024-09-13T11:23:02.684 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:02.684 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:02.684 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:02.685 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.162:6808/2287343806", 2024-09-13T11:23:02.685 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_addr": "172.21.15.162:6809/2287343806", 2024-09-13T11:23:02.685 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_back_addr": "172.21.15.162:6811/2287343806", 2024-09-13T11:23:02.685 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_front_addr": "172.21.15.162:6810/2287343806", 2024-09-13T11:23:02.685 INFO:teuthology.orchestra.run.smithi001.stdout: "state": [ 2024-09-13T11:23:02.685 INFO:teuthology.orchestra.run.smithi001.stdout: "exists", 2024-09-13T11:23:02.685 INFO:teuthology.orchestra.run.smithi001.stdout: "up" 2024-09-13T11:23:02.685 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:02.685 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:02.685 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:02.685 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": 7, 2024-09-13T11:23:02.685 INFO:teuthology.orchestra.run.smithi001.stdout: "uuid": "60aa5206-cace-40f8-8d1a-10ec28d97096", 2024-09-13T11:23:02.685 INFO:teuthology.orchestra.run.smithi001.stdout: "up": 1, 2024-09-13T11:23:02.685 INFO:teuthology.orchestra.run.smithi001.stdout: "in": 1, 2024-09-13T11:23:02.686 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 1, 2024-09-13T11:23:02.686 INFO:teuthology.orchestra.run.smithi001.stdout: "primary_affinity": 1, 2024-09-13T11:23:02.686 INFO:teuthology.orchestra.run.smithi001.stdout: "last_clean_begin": 0, 2024-09-13T11:23:02.686 INFO:teuthology.orchestra.run.smithi001.stdout: "last_clean_end": 0, 2024-09-13T11:23:02.686 INFO:teuthology.orchestra.run.smithi001.stdout: "up_from": 22, 2024-09-13T11:23:02.686 INFO:teuthology.orchestra.run.smithi001.stdout: "up_thru": 30, 2024-09-13T11:23:02.686 INFO:teuthology.orchestra.run.smithi001.stdout: "down_at": 0, 2024-09-13T11:23:02.686 INFO:teuthology.orchestra.run.smithi001.stdout: "lost_at": 0, 2024-09-13T11:23:02.686 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-09-13T11:23:02.686 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:23:02.686 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:02.687 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:23:02.687 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.177:6813", 2024-09-13T11:23:02.687 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 2110809906 2024-09-13T11:23:02.687 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:02.687 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:02.687 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:02.687 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_addrs": { 2024-09-13T11:23:02.687 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:23:02.687 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:02.687 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:23:02.687 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.177:6814", 2024-09-13T11:23:02.687 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 2110809906 2024-09-13T11:23:02.688 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:02.688 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:02.688 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:02.688 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_back_addrs": { 2024-09-13T11:23:02.688 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:23:02.688 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:02.688 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:23:02.688 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.177:6816", 2024-09-13T11:23:02.688 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 2110809906 2024-09-13T11:23:02.688 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:02.688 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:02.688 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:02.688 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_front_addrs": { 2024-09-13T11:23:02.688 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:23:02.689 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:02.689 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:23:02.689 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.177:6815", 2024-09-13T11:23:02.689 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 2110809906 2024-09-13T11:23:02.689 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:02.689 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:02.689 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:02.689 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.177:6813/2110809906", 2024-09-13T11:23:02.689 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_addr": "172.21.15.177:6814/2110809906", 2024-09-13T11:23:02.689 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_back_addr": "172.21.15.177:6816/2110809906", 2024-09-13T11:23:02.689 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_front_addr": "172.21.15.177:6815/2110809906", 2024-09-13T11:23:02.689 INFO:teuthology.orchestra.run.smithi001.stdout: "state": [ 2024-09-13T11:23:02.689 INFO:teuthology.orchestra.run.smithi001.stdout: "exists", 2024-09-13T11:23:02.689 INFO:teuthology.orchestra.run.smithi001.stdout: "up" 2024-09-13T11:23:02.690 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:02.690 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:02.690 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:02.690 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": 8, 2024-09-13T11:23:02.690 INFO:teuthology.orchestra.run.smithi001.stdout: "uuid": "f3d99325-48ff-403c-8662-1d7745f7c97d", 2024-09-13T11:23:02.690 INFO:teuthology.orchestra.run.smithi001.stdout: "up": 1, 2024-09-13T11:23:02.690 INFO:teuthology.orchestra.run.smithi001.stdout: "in": 1, 2024-09-13T11:23:02.690 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 1, 2024-09-13T11:23:02.690 INFO:teuthology.orchestra.run.smithi001.stdout: "primary_affinity": 1, 2024-09-13T11:23:02.690 INFO:teuthology.orchestra.run.smithi001.stdout: "last_clean_begin": 0, 2024-09-13T11:23:02.690 INFO:teuthology.orchestra.run.smithi001.stdout: "last_clean_end": 0, 2024-09-13T11:23:02.690 INFO:teuthology.orchestra.run.smithi001.stdout: "up_from": 22, 2024-09-13T11:23:02.690 INFO:teuthology.orchestra.run.smithi001.stdout: "up_thru": 30, 2024-09-13T11:23:02.691 INFO:teuthology.orchestra.run.smithi001.stdout: "down_at": 0, 2024-09-13T11:23:02.691 INFO:teuthology.orchestra.run.smithi001.stdout: "lost_at": 0, 2024-09-13T11:23:02.691 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-09-13T11:23:02.691 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:23:02.691 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:02.691 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:23:02.691 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6804", 2024-09-13T11:23:02.691 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 2766567910 2024-09-13T11:23:02.691 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:02.691 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:02.691 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:02.691 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_addrs": { 2024-09-13T11:23:02.691 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:23:02.691 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:02.692 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:23:02.692 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6805", 2024-09-13T11:23:02.692 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 2766567910 2024-09-13T11:23:02.692 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:02.692 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:02.692 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:02.692 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_back_addrs": { 2024-09-13T11:23:02.692 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:23:02.692 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:02.692 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:23:02.692 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6807", 2024-09-13T11:23:02.692 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 2766567910 2024-09-13T11:23:02.692 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:02.692 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:02.693 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:02.693 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_front_addrs": { 2024-09-13T11:23:02.693 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:23:02.693 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:02.693 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:23:02.693 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6806", 2024-09-13T11:23:02.693 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 2766567910 2024-09-13T11:23:02.693 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:02.693 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:02.693 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:02.693 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6804/2766567910", 2024-09-13T11:23:02.693 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_addr": "172.21.15.1:6805/2766567910", 2024-09-13T11:23:02.693 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_back_addr": "172.21.15.1:6807/2766567910", 2024-09-13T11:23:02.693 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_front_addr": "172.21.15.1:6806/2766567910", 2024-09-13T11:23:02.694 INFO:teuthology.orchestra.run.smithi001.stdout: "state": [ 2024-09-13T11:23:02.694 INFO:teuthology.orchestra.run.smithi001.stdout: "exists", 2024-09-13T11:23:02.694 INFO:teuthology.orchestra.run.smithi001.stdout: "up" 2024-09-13T11:23:02.694 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:02.694 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:02.694 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:02.694 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": 9, 2024-09-13T11:23:02.694 INFO:teuthology.orchestra.run.smithi001.stdout: "uuid": "88511e24-3675-4551-968c-1a52e7e28f96", 2024-09-13T11:23:02.694 INFO:teuthology.orchestra.run.smithi001.stdout: "up": 1, 2024-09-13T11:23:02.694 INFO:teuthology.orchestra.run.smithi001.stdout: "in": 1, 2024-09-13T11:23:02.694 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 1, 2024-09-13T11:23:02.694 INFO:teuthology.orchestra.run.smithi001.stdout: "primary_affinity": 1, 2024-09-13T11:23:02.694 INFO:teuthology.orchestra.run.smithi001.stdout: "last_clean_begin": 0, 2024-09-13T11:23:02.694 INFO:teuthology.orchestra.run.smithi001.stdout: "last_clean_end": 0, 2024-09-13T11:23:02.695 INFO:teuthology.orchestra.run.smithi001.stdout: "up_from": 23, 2024-09-13T11:23:02.695 INFO:teuthology.orchestra.run.smithi001.stdout: "up_thru": 30, 2024-09-13T11:23:02.695 INFO:teuthology.orchestra.run.smithi001.stdout: "down_at": 0, 2024-09-13T11:23:02.695 INFO:teuthology.orchestra.run.smithi001.stdout: "lost_at": 0, 2024-09-13T11:23:02.695 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-09-13T11:23:02.695 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:23:02.695 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:02.695 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:23:02.695 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.92:6812", 2024-09-13T11:23:02.695 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 2488187511 2024-09-13T11:23:02.695 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:02.695 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:02.695 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:02.695 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_addrs": { 2024-09-13T11:23:02.696 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:23:02.696 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:02.696 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:23:02.696 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.92:6813", 2024-09-13T11:23:02.696 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 2488187511 2024-09-13T11:23:02.696 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:02.696 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:02.696 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:02.696 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_back_addrs": { 2024-09-13T11:23:02.696 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:23:02.696 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:02.696 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:23:02.696 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.92:6815", 2024-09-13T11:23:02.697 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 2488187511 2024-09-13T11:23:02.697 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:02.697 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:02.697 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:02.697 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_front_addrs": { 2024-09-13T11:23:02.697 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:23:02.697 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:02.697 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:23:02.697 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.92:6814", 2024-09-13T11:23:02.697 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 2488187511 2024-09-13T11:23:02.697 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:02.697 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:02.697 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:02.698 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.92:6812/2488187511", 2024-09-13T11:23:02.698 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_addr": "172.21.15.92:6813/2488187511", 2024-09-13T11:23:02.698 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_back_addr": "172.21.15.92:6815/2488187511", 2024-09-13T11:23:02.698 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_front_addr": "172.21.15.92:6814/2488187511", 2024-09-13T11:23:02.698 INFO:teuthology.orchestra.run.smithi001.stdout: "state": [ 2024-09-13T11:23:02.698 INFO:teuthology.orchestra.run.smithi001.stdout: "exists", 2024-09-13T11:23:02.698 INFO:teuthology.orchestra.run.smithi001.stdout: "up" 2024-09-13T11:23:02.698 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:02.698 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:02.698 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:02.698 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": 10, 2024-09-13T11:23:02.699 INFO:teuthology.orchestra.run.smithi001.stdout: "uuid": "acf4c64d-99af-42be-8406-413c86f54670", 2024-09-13T11:23:02.699 INFO:teuthology.orchestra.run.smithi001.stdout: "up": 1, 2024-09-13T11:23:02.699 INFO:teuthology.orchestra.run.smithi001.stdout: "in": 1, 2024-09-13T11:23:02.699 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 1, 2024-09-13T11:23:02.699 INFO:teuthology.orchestra.run.smithi001.stdout: "primary_affinity": 1, 2024-09-13T11:23:02.699 INFO:teuthology.orchestra.run.smithi001.stdout: "last_clean_begin": 0, 2024-09-13T11:23:02.699 INFO:teuthology.orchestra.run.smithi001.stdout: "last_clean_end": 0, 2024-09-13T11:23:02.699 INFO:teuthology.orchestra.run.smithi001.stdout: "up_from": 22, 2024-09-13T11:23:02.699 INFO:teuthology.orchestra.run.smithi001.stdout: "up_thru": 30, 2024-09-13T11:23:02.699 INFO:teuthology.orchestra.run.smithi001.stdout: "down_at": 0, 2024-09-13T11:23:02.699 INFO:teuthology.orchestra.run.smithi001.stdout: "lost_at": 0, 2024-09-13T11:23:02.699 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-09-13T11:23:02.699 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:23:02.699 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:02.700 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:23:02.700 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.162:6804", 2024-09-13T11:23:02.700 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 1450266459 2024-09-13T11:23:02.700 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:02.700 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:02.700 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:02.700 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_addrs": { 2024-09-13T11:23:02.700 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:23:02.700 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:02.700 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:23:02.700 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.162:6805", 2024-09-13T11:23:02.700 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 1450266459 2024-09-13T11:23:02.700 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:02.701 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:02.701 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:02.701 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_back_addrs": { 2024-09-13T11:23:02.701 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:23:02.701 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:02.701 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:23:02.701 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.162:6807", 2024-09-13T11:23:02.701 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 1450266459 2024-09-13T11:23:02.701 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:02.701 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:02.701 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:02.701 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_front_addrs": { 2024-09-13T11:23:02.701 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:23:02.701 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:02.702 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:23:02.702 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.162:6806", 2024-09-13T11:23:02.702 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 1450266459 2024-09-13T11:23:02.702 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:02.702 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:02.702 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:02.702 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.162:6804/1450266459", 2024-09-13T11:23:02.702 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_addr": "172.21.15.162:6805/1450266459", 2024-09-13T11:23:02.702 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_back_addr": "172.21.15.162:6807/1450266459", 2024-09-13T11:23:02.702 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_front_addr": "172.21.15.162:6806/1450266459", 2024-09-13T11:23:02.702 INFO:teuthology.orchestra.run.smithi001.stdout: "state": [ 2024-09-13T11:23:02.702 INFO:teuthology.orchestra.run.smithi001.stdout: "exists", 2024-09-13T11:23:02.702 INFO:teuthology.orchestra.run.smithi001.stdout: "up" 2024-09-13T11:23:02.703 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:02.703 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:02.703 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:02.703 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": 11, 2024-09-13T11:23:02.703 INFO:teuthology.orchestra.run.smithi001.stdout: "uuid": "a4bfdfbe-b586-4080-8e09-6f26428e59b5", 2024-09-13T11:23:02.703 INFO:teuthology.orchestra.run.smithi001.stdout: "up": 1, 2024-09-13T11:23:02.703 INFO:teuthology.orchestra.run.smithi001.stdout: "in": 1, 2024-09-13T11:23:02.703 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 1, 2024-09-13T11:23:02.703 INFO:teuthology.orchestra.run.smithi001.stdout: "primary_affinity": 1, 2024-09-13T11:23:02.703 INFO:teuthology.orchestra.run.smithi001.stdout: "last_clean_begin": 0, 2024-09-13T11:23:02.703 INFO:teuthology.orchestra.run.smithi001.stdout: "last_clean_end": 0, 2024-09-13T11:23:02.703 INFO:teuthology.orchestra.run.smithi001.stdout: "up_from": 22, 2024-09-13T11:23:02.703 INFO:teuthology.orchestra.run.smithi001.stdout: "up_thru": 30, 2024-09-13T11:23:02.703 INFO:teuthology.orchestra.run.smithi001.stdout: "down_at": 0, 2024-09-13T11:23:02.704 INFO:teuthology.orchestra.run.smithi001.stdout: "lost_at": 0, 2024-09-13T11:23:02.704 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-09-13T11:23:02.704 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:23:02.704 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:02.704 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:23:02.704 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.177:6809", 2024-09-13T11:23:02.704 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 3239521954 2024-09-13T11:23:02.704 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:02.704 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:02.704 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:02.704 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_addrs": { 2024-09-13T11:23:02.704 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:23:02.704 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:02.704 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:23:02.705 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.177:6810", 2024-09-13T11:23:02.705 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 3239521954 2024-09-13T11:23:02.705 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:02.705 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:02.705 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:02.705 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_back_addrs": { 2024-09-13T11:23:02.705 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:23:02.705 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:02.705 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:23:02.705 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.177:6812", 2024-09-13T11:23:02.705 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 3239521954 2024-09-13T11:23:02.705 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:02.705 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:02.705 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:02.706 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_front_addrs": { 2024-09-13T11:23:02.706 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:23:02.706 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:02.706 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:23:02.706 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.177:6811", 2024-09-13T11:23:02.706 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 3239521954 2024-09-13T11:23:02.706 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:02.706 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:02.706 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:02.706 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.177:6809/3239521954", 2024-09-13T11:23:02.706 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_addr": "172.21.15.177:6810/3239521954", 2024-09-13T11:23:02.706 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_back_addr": "172.21.15.177:6812/3239521954", 2024-09-13T11:23:02.706 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_front_addr": "172.21.15.177:6811/3239521954", 2024-09-13T11:23:02.706 INFO:teuthology.orchestra.run.smithi001.stdout: "state": [ 2024-09-13T11:23:02.707 INFO:teuthology.orchestra.run.smithi001.stdout: "exists", 2024-09-13T11:23:02.707 INFO:teuthology.orchestra.run.smithi001.stdout: "up" 2024-09-13T11:23:02.707 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:02.707 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:02.707 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:02.707 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": 12, 2024-09-13T11:23:02.707 INFO:teuthology.orchestra.run.smithi001.stdout: "uuid": "dc571fb6-d8b5-465b-bb25-afa7d0c47b5a", 2024-09-13T11:23:02.707 INFO:teuthology.orchestra.run.smithi001.stdout: "up": 1, 2024-09-13T11:23:02.707 INFO:teuthology.orchestra.run.smithi001.stdout: "in": 1, 2024-09-13T11:23:02.707 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 1, 2024-09-13T11:23:02.707 INFO:teuthology.orchestra.run.smithi001.stdout: "primary_affinity": 1, 2024-09-13T11:23:02.707 INFO:teuthology.orchestra.run.smithi001.stdout: "last_clean_begin": 0, 2024-09-13T11:23:02.707 INFO:teuthology.orchestra.run.smithi001.stdout: "last_clean_end": 0, 2024-09-13T11:23:02.707 INFO:teuthology.orchestra.run.smithi001.stdout: "up_from": 22, 2024-09-13T11:23:02.707 INFO:teuthology.orchestra.run.smithi001.stdout: "up_thru": 30, 2024-09-13T11:23:02.708 INFO:teuthology.orchestra.run.smithi001.stdout: "down_at": 0, 2024-09-13T11:23:02.708 INFO:teuthology.orchestra.run.smithi001.stdout: "lost_at": 0, 2024-09-13T11:23:02.708 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-09-13T11:23:02.708 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:23:02.708 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:02.708 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:23:02.708 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6812", 2024-09-13T11:23:02.708 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 4267062742 2024-09-13T11:23:02.708 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:02.708 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:02.708 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:02.708 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_addrs": { 2024-09-13T11:23:02.708 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:23:02.709 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:02.709 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:23:02.709 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6813", 2024-09-13T11:23:02.709 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 4267062742 2024-09-13T11:23:02.709 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:02.709 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:02.709 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:02.709 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_back_addrs": { 2024-09-13T11:23:02.709 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:23:02.709 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:02.709 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:23:02.709 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6815", 2024-09-13T11:23:02.709 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 4267062742 2024-09-13T11:23:02.710 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:02.710 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:02.710 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:02.710 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_front_addrs": { 2024-09-13T11:23:02.710 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:23:02.710 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:02.710 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:23:02.710 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6814", 2024-09-13T11:23:02.710 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 4267062742 2024-09-13T11:23:02.710 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:02.710 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:02.710 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:02.710 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6812/4267062742", 2024-09-13T11:23:02.710 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_addr": "172.21.15.1:6813/4267062742", 2024-09-13T11:23:02.711 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_back_addr": "172.21.15.1:6815/4267062742", 2024-09-13T11:23:02.711 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_front_addr": "172.21.15.1:6814/4267062742", 2024-09-13T11:23:02.711 INFO:teuthology.orchestra.run.smithi001.stdout: "state": [ 2024-09-13T11:23:02.711 INFO:teuthology.orchestra.run.smithi001.stdout: "exists", 2024-09-13T11:23:02.711 INFO:teuthology.orchestra.run.smithi001.stdout: "up" 2024-09-13T11:23:02.711 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:02.711 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:02.711 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:02.711 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": 13, 2024-09-13T11:23:02.711 INFO:teuthology.orchestra.run.smithi001.stdout: "uuid": "5d321d4b-1b26-47fe-bf84-25751b8cde7d", 2024-09-13T11:23:02.711 INFO:teuthology.orchestra.run.smithi001.stdout: "up": 1, 2024-09-13T11:23:02.711 INFO:teuthology.orchestra.run.smithi001.stdout: "in": 1, 2024-09-13T11:23:02.711 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 1, 2024-09-13T11:23:02.711 INFO:teuthology.orchestra.run.smithi001.stdout: "primary_affinity": 1, 2024-09-13T11:23:02.711 INFO:teuthology.orchestra.run.smithi001.stdout: "last_clean_begin": 0, 2024-09-13T11:23:02.712 INFO:teuthology.orchestra.run.smithi001.stdout: "last_clean_end": 0, 2024-09-13T11:23:02.712 INFO:teuthology.orchestra.run.smithi001.stdout: "up_from": 23, 2024-09-13T11:23:02.712 INFO:teuthology.orchestra.run.smithi001.stdout: "up_thru": 41, 2024-09-13T11:23:02.712 INFO:teuthology.orchestra.run.smithi001.stdout: "down_at": 0, 2024-09-13T11:23:02.712 INFO:teuthology.orchestra.run.smithi001.stdout: "lost_at": 0, 2024-09-13T11:23:02.712 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-09-13T11:23:02.712 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:23:02.712 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:02.712 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:23:02.712 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.92:6808", 2024-09-13T11:23:02.712 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 1759585046 2024-09-13T11:23:02.712 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:02.712 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:02.712 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:02.713 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_addrs": { 2024-09-13T11:23:02.713 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:23:02.713 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:02.713 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:23:02.713 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.92:6809", 2024-09-13T11:23:02.713 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 1759585046 2024-09-13T11:23:02.713 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:02.713 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:02.713 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:02.713 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_back_addrs": { 2024-09-13T11:23:02.713 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:23:02.713 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:02.713 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:23:02.713 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.92:6811", 2024-09-13T11:23:02.714 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 1759585046 2024-09-13T11:23:02.714 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:02.714 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:02.714 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:02.714 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_front_addrs": { 2024-09-13T11:23:02.714 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:23:02.714 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:02.714 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:23:02.714 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.92:6810", 2024-09-13T11:23:02.714 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 1759585046 2024-09-13T11:23:02.714 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:02.714 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:02.714 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:02.715 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.92:6808/1759585046", 2024-09-13T11:23:02.715 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_addr": "172.21.15.92:6809/1759585046", 2024-09-13T11:23:02.715 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_back_addr": "172.21.15.92:6811/1759585046", 2024-09-13T11:23:02.715 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_front_addr": "172.21.15.92:6810/1759585046", 2024-09-13T11:23:02.715 INFO:teuthology.orchestra.run.smithi001.stdout: "state": [ 2024-09-13T11:23:02.715 INFO:teuthology.orchestra.run.smithi001.stdout: "exists", 2024-09-13T11:23:02.715 INFO:teuthology.orchestra.run.smithi001.stdout: "up" 2024-09-13T11:23:02.715 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:02.715 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:02.715 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:02.715 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": 14, 2024-09-13T11:23:02.715 INFO:teuthology.orchestra.run.smithi001.stdout: "uuid": "fc3e2499-0c2a-414d-a25e-2628876ae630", 2024-09-13T11:23:02.715 INFO:teuthology.orchestra.run.smithi001.stdout: "up": 1, 2024-09-13T11:23:02.715 INFO:teuthology.orchestra.run.smithi001.stdout: "in": 1, 2024-09-13T11:23:02.716 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 1, 2024-09-13T11:23:02.716 INFO:teuthology.orchestra.run.smithi001.stdout: "primary_affinity": 1, 2024-09-13T11:23:02.716 INFO:teuthology.orchestra.run.smithi001.stdout: "last_clean_begin": 0, 2024-09-13T11:23:02.716 INFO:teuthology.orchestra.run.smithi001.stdout: "last_clean_end": 0, 2024-09-13T11:23:02.716 INFO:teuthology.orchestra.run.smithi001.stdout: "up_from": 22, 2024-09-13T11:23:02.716 INFO:teuthology.orchestra.run.smithi001.stdout: "up_thru": 40, 2024-09-13T11:23:02.716 INFO:teuthology.orchestra.run.smithi001.stdout: "down_at": 0, 2024-09-13T11:23:02.716 INFO:teuthology.orchestra.run.smithi001.stdout: "lost_at": 0, 2024-09-13T11:23:02.716 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-09-13T11:23:02.716 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:23:02.716 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:02.716 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:23:02.716 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.162:6800", 2024-09-13T11:23:02.716 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 3905144320 2024-09-13T11:23:02.717 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:02.717 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:02.717 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:02.717 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_addrs": { 2024-09-13T11:23:02.717 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:23:02.717 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:02.717 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:23:02.717 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.162:6801", 2024-09-13T11:23:02.717 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 3905144320 2024-09-13T11:23:02.717 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:02.717 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:02.717 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:02.717 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_back_addrs": { 2024-09-13T11:23:02.717 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:23:02.718 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:02.718 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:23:02.718 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.162:6803", 2024-09-13T11:23:02.718 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 3905144320 2024-09-13T11:23:02.718 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:02.718 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:02.718 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:02.718 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_front_addrs": { 2024-09-13T11:23:02.718 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:23:02.718 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:02.718 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:23:02.718 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.162:6802", 2024-09-13T11:23:02.718 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 3905144320 2024-09-13T11:23:02.718 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:02.719 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:02.719 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:02.719 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.162:6800/3905144320", 2024-09-13T11:23:02.719 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_addr": "172.21.15.162:6801/3905144320", 2024-09-13T11:23:02.719 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_back_addr": "172.21.15.162:6803/3905144320", 2024-09-13T11:23:02.719 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_front_addr": "172.21.15.162:6802/3905144320", 2024-09-13T11:23:02.719 INFO:teuthology.orchestra.run.smithi001.stdout: "state": [ 2024-09-13T11:23:02.719 INFO:teuthology.orchestra.run.smithi001.stdout: "exists", 2024-09-13T11:23:02.719 INFO:teuthology.orchestra.run.smithi001.stdout: "up" 2024-09-13T11:23:02.719 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:02.719 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:02.719 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:02.719 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": 15, 2024-09-13T11:23:02.720 INFO:teuthology.orchestra.run.smithi001.stdout: "uuid": "d2f69775-edfe-4527-91ec-f7c7e558bec8", 2024-09-13T11:23:02.720 INFO:teuthology.orchestra.run.smithi001.stdout: "up": 1, 2024-09-13T11:23:02.720 INFO:teuthology.orchestra.run.smithi001.stdout: "in": 1, 2024-09-13T11:23:02.720 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 1, 2024-09-13T11:23:02.720 INFO:teuthology.orchestra.run.smithi001.stdout: "primary_affinity": 1, 2024-09-13T11:23:02.720 INFO:teuthology.orchestra.run.smithi001.stdout: "last_clean_begin": 0, 2024-09-13T11:23:02.720 INFO:teuthology.orchestra.run.smithi001.stdout: "last_clean_end": 0, 2024-09-13T11:23:02.720 INFO:teuthology.orchestra.run.smithi001.stdout: "up_from": 22, 2024-09-13T11:23:02.720 INFO:teuthology.orchestra.run.smithi001.stdout: "up_thru": 30, 2024-09-13T11:23:02.720 INFO:teuthology.orchestra.run.smithi001.stdout: "down_at": 0, 2024-09-13T11:23:02.720 INFO:teuthology.orchestra.run.smithi001.stdout: "lost_at": 0, 2024-09-13T11:23:02.720 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-09-13T11:23:02.720 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:23:02.720 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:02.721 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:23:02.721 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.177:6805", 2024-09-13T11:23:02.721 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 1294337172 2024-09-13T11:23:02.721 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:02.721 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:02.721 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:02.721 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_addrs": { 2024-09-13T11:23:02.721 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:23:02.721 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:02.721 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:23:02.721 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.177:6806", 2024-09-13T11:23:02.721 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 1294337172 2024-09-13T11:23:02.721 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:02.721 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:02.722 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:02.722 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_back_addrs": { 2024-09-13T11:23:02.722 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:23:02.722 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:02.722 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:23:02.722 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.177:6808", 2024-09-13T11:23:02.722 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 1294337172 2024-09-13T11:23:02.722 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:02.722 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:02.722 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:02.722 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_front_addrs": { 2024-09-13T11:23:02.722 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:23:02.722 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:02.722 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:23:02.723 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.177:6807", 2024-09-13T11:23:02.723 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 1294337172 2024-09-13T11:23:02.723 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:02.723 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:02.723 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:02.723 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.177:6805/1294337172", 2024-09-13T11:23:02.723 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_addr": "172.21.15.177:6806/1294337172", 2024-09-13T11:23:02.723 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_back_addr": "172.21.15.177:6808/1294337172", 2024-09-13T11:23:02.723 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_front_addr": "172.21.15.177:6807/1294337172", 2024-09-13T11:23:02.723 INFO:teuthology.orchestra.run.smithi001.stdout: "state": [ 2024-09-13T11:23:02.723 INFO:teuthology.orchestra.run.smithi001.stdout: "exists", 2024-09-13T11:23:02.723 INFO:teuthology.orchestra.run.smithi001.stdout: "up" 2024-09-13T11:23:02.723 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:02.723 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:02.724 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-09-13T11:23:02.724 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_xinfo": [ 2024-09-13T11:23:02.724 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:02.724 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": 0, 2024-09-13T11:23:02.724 INFO:teuthology.orchestra.run.smithi001.stdout: "down_stamp": "0.000000", 2024-09-13T11:23:02.724 INFO:teuthology.orchestra.run.smithi001.stdout: "laggy_probability": 0, 2024-09-13T11:23:02.724 INFO:teuthology.orchestra.run.smithi001.stdout: "laggy_interval": 0, 2024-09-13T11:23:02.724 INFO:teuthology.orchestra.run.smithi001.stdout: "features": 4540719139924082687, 2024-09-13T11:23:02.724 INFO:teuthology.orchestra.run.smithi001.stdout: "old_weight": 0, 2024-09-13T11:23:02.724 INFO:teuthology.orchestra.run.smithi001.stdout: "last_purged_snaps_scrub": "2024-09-13T11:22:08.233274+0000", 2024-09-13T11:23:02.724 INFO:teuthology.orchestra.run.smithi001.stdout: "dead_epoch": 0 2024-09-13T11:23:02.724 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:02.724 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:02.724 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": 1, 2024-09-13T11:23:02.725 INFO:teuthology.orchestra.run.smithi001.stdout: "down_stamp": "0.000000", 2024-09-13T11:23:02.725 INFO:teuthology.orchestra.run.smithi001.stdout: "laggy_probability": 0, 2024-09-13T11:23:02.725 INFO:teuthology.orchestra.run.smithi001.stdout: "laggy_interval": 0, 2024-09-13T11:23:02.725 INFO:teuthology.orchestra.run.smithi001.stdout: "features": 4540719139924082687, 2024-09-13T11:23:02.725 INFO:teuthology.orchestra.run.smithi001.stdout: "old_weight": 0, 2024-09-13T11:23:02.725 INFO:teuthology.orchestra.run.smithi001.stdout: "last_purged_snaps_scrub": "2024-09-13T11:22:08.032163+0000", 2024-09-13T11:23:02.725 INFO:teuthology.orchestra.run.smithi001.stdout: "dead_epoch": 30 2024-09-13T11:23:02.725 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:02.725 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:02.725 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": 2, 2024-09-13T11:23:02.725 INFO:teuthology.orchestra.run.smithi001.stdout: "down_stamp": "0.000000", 2024-09-13T11:23:02.725 INFO:teuthology.orchestra.run.smithi001.stdout: "laggy_probability": 0, 2024-09-13T11:23:02.725 INFO:teuthology.orchestra.run.smithi001.stdout: "laggy_interval": 0, 2024-09-13T11:23:02.725 INFO:teuthology.orchestra.run.smithi001.stdout: "features": 4540719139924082687, 2024-09-13T11:23:02.726 INFO:teuthology.orchestra.run.smithi001.stdout: "old_weight": 0, 2024-09-13T11:23:02.726 INFO:teuthology.orchestra.run.smithi001.stdout: "last_purged_snaps_scrub": "2024-09-13T11:22:08.875236+0000", 2024-09-13T11:23:02.726 INFO:teuthology.orchestra.run.smithi001.stdout: "dead_epoch": 0 2024-09-13T11:23:02.726 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:02.726 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:02.726 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": 3, 2024-09-13T11:23:02.726 INFO:teuthology.orchestra.run.smithi001.stdout: "down_stamp": "0.000000", 2024-09-13T11:23:02.726 INFO:teuthology.orchestra.run.smithi001.stdout: "laggy_probability": 0, 2024-09-13T11:23:02.726 INFO:teuthology.orchestra.run.smithi001.stdout: "laggy_interval": 0, 2024-09-13T11:23:02.726 INFO:teuthology.orchestra.run.smithi001.stdout: "features": 4540719139924082687, 2024-09-13T11:23:02.726 INFO:teuthology.orchestra.run.smithi001.stdout: "old_weight": 0, 2024-09-13T11:23:02.726 INFO:teuthology.orchestra.run.smithi001.stdout: "last_purged_snaps_scrub": "2024-09-13T11:22:07.936983+0000", 2024-09-13T11:23:02.726 INFO:teuthology.orchestra.run.smithi001.stdout: "dead_epoch": 0 2024-09-13T11:23:02.726 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:02.727 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:02.727 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": 4, 2024-09-13T11:23:02.727 INFO:teuthology.orchestra.run.smithi001.stdout: "down_stamp": "0.000000", 2024-09-13T11:23:02.727 INFO:teuthology.orchestra.run.smithi001.stdout: "laggy_probability": 0, 2024-09-13T11:23:02.727 INFO:teuthology.orchestra.run.smithi001.stdout: "laggy_interval": 0, 2024-09-13T11:23:02.727 INFO:teuthology.orchestra.run.smithi001.stdout: "features": 4540719139924082687, 2024-09-13T11:23:02.727 INFO:teuthology.orchestra.run.smithi001.stdout: "old_weight": 0, 2024-09-13T11:23:02.727 INFO:teuthology.orchestra.run.smithi001.stdout: "last_purged_snaps_scrub": "2024-09-13T11:22:08.625189+0000", 2024-09-13T11:23:02.727 INFO:teuthology.orchestra.run.smithi001.stdout: "dead_epoch": 0 2024-09-13T11:23:02.727 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:02.727 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:02.727 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": 5, 2024-09-13T11:23:02.727 INFO:teuthology.orchestra.run.smithi001.stdout: "down_stamp": "0.000000", 2024-09-13T11:23:02.728 INFO:teuthology.orchestra.run.smithi001.stdout: "laggy_probability": 0, 2024-09-13T11:23:02.728 INFO:teuthology.orchestra.run.smithi001.stdout: "laggy_interval": 0, 2024-09-13T11:23:02.728 INFO:teuthology.orchestra.run.smithi001.stdout: "features": 4540719139924082687, 2024-09-13T11:23:02.728 INFO:teuthology.orchestra.run.smithi001.stdout: "old_weight": 0, 2024-09-13T11:23:02.728 INFO:teuthology.orchestra.run.smithi001.stdout: "last_purged_snaps_scrub": "2024-09-13T11:22:07.944287+0000", 2024-09-13T11:23:02.728 INFO:teuthology.orchestra.run.smithi001.stdout: "dead_epoch": 0 2024-09-13T11:23:02.728 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:02.728 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:02.728 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": 6, 2024-09-13T11:23:02.728 INFO:teuthology.orchestra.run.smithi001.stdout: "down_stamp": "0.000000", 2024-09-13T11:23:02.728 INFO:teuthology.orchestra.run.smithi001.stdout: "laggy_probability": 0, 2024-09-13T11:23:02.728 INFO:teuthology.orchestra.run.smithi001.stdout: "laggy_interval": 0, 2024-09-13T11:23:02.728 INFO:teuthology.orchestra.run.smithi001.stdout: "features": 4540719139924082687, 2024-09-13T11:23:02.729 INFO:teuthology.orchestra.run.smithi001.stdout: "old_weight": 0, 2024-09-13T11:23:02.729 INFO:teuthology.orchestra.run.smithi001.stdout: "last_purged_snaps_scrub": "2024-09-13T11:22:08.351045+0000", 2024-09-13T11:23:02.729 INFO:teuthology.orchestra.run.smithi001.stdout: "dead_epoch": 0 2024-09-13T11:23:02.729 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:02.729 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:02.729 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": 7, 2024-09-13T11:23:02.729 INFO:teuthology.orchestra.run.smithi001.stdout: "down_stamp": "0.000000", 2024-09-13T11:23:02.729 INFO:teuthology.orchestra.run.smithi001.stdout: "laggy_probability": 0, 2024-09-13T11:23:02.729 INFO:teuthology.orchestra.run.smithi001.stdout: "laggy_interval": 0, 2024-09-13T11:23:02.729 INFO:teuthology.orchestra.run.smithi001.stdout: "features": 4540719139924082687, 2024-09-13T11:23:02.729 INFO:teuthology.orchestra.run.smithi001.stdout: "old_weight": 0, 2024-09-13T11:23:02.729 INFO:teuthology.orchestra.run.smithi001.stdout: "last_purged_snaps_scrub": "2024-09-13T11:22:08.835090+0000", 2024-09-13T11:23:02.729 INFO:teuthology.orchestra.run.smithi001.stdout: "dead_epoch": 0 2024-09-13T11:23:02.729 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:02.730 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:02.730 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": 8, 2024-09-13T11:23:02.730 INFO:teuthology.orchestra.run.smithi001.stdout: "down_stamp": "0.000000", 2024-09-13T11:23:02.730 INFO:teuthology.orchestra.run.smithi001.stdout: "laggy_probability": 0, 2024-09-13T11:23:02.730 INFO:teuthology.orchestra.run.smithi001.stdout: "laggy_interval": 0, 2024-09-13T11:23:02.730 INFO:teuthology.orchestra.run.smithi001.stdout: "features": 4540719139924082687, 2024-09-13T11:23:02.730 INFO:teuthology.orchestra.run.smithi001.stdout: "old_weight": 0, 2024-09-13T11:23:02.730 INFO:teuthology.orchestra.run.smithi001.stdout: "last_purged_snaps_scrub": "2024-09-13T11:22:08.847356+0000", 2024-09-13T11:23:02.730 INFO:teuthology.orchestra.run.smithi001.stdout: "dead_epoch": 0 2024-09-13T11:23:02.730 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:02.730 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:02.730 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": 9, 2024-09-13T11:23:02.731 INFO:teuthology.orchestra.run.smithi001.stdout: "down_stamp": "0.000000", 2024-09-13T11:23:02.731 INFO:teuthology.orchestra.run.smithi001.stdout: "laggy_probability": 0, 2024-09-13T11:23:02.731 INFO:teuthology.orchestra.run.smithi001.stdout: "laggy_interval": 0, 2024-09-13T11:23:02.731 INFO:teuthology.orchestra.run.smithi001.stdout: "features": 4540719139924082687, 2024-09-13T11:23:02.731 INFO:teuthology.orchestra.run.smithi001.stdout: "old_weight": 0, 2024-09-13T11:23:02.731 INFO:teuthology.orchestra.run.smithi001.stdout: "last_purged_snaps_scrub": "2024-09-13T11:22:08.140353+0000", 2024-09-13T11:23:02.731 INFO:teuthology.orchestra.run.smithi001.stdout: "dead_epoch": 0 2024-09-13T11:23:02.731 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:02.731 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:02.731 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": 10, 2024-09-13T11:23:02.731 INFO:teuthology.orchestra.run.smithi001.stdout: "down_stamp": "0.000000", 2024-09-13T11:23:02.731 INFO:teuthology.orchestra.run.smithi001.stdout: "laggy_probability": 0, 2024-09-13T11:23:02.731 INFO:teuthology.orchestra.run.smithi001.stdout: "laggy_interval": 0, 2024-09-13T11:23:02.732 INFO:teuthology.orchestra.run.smithi001.stdout: "features": 4540719139924082687, 2024-09-13T11:23:02.732 INFO:teuthology.orchestra.run.smithi001.stdout: "old_weight": 0, 2024-09-13T11:23:02.732 INFO:teuthology.orchestra.run.smithi001.stdout: "last_purged_snaps_scrub": "2024-09-13T11:22:08.218416+0000", 2024-09-13T11:23:02.732 INFO:teuthology.orchestra.run.smithi001.stdout: "dead_epoch": 0 2024-09-13T11:23:02.732 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:02.732 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:02.732 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": 11, 2024-09-13T11:23:02.732 INFO:teuthology.orchestra.run.smithi001.stdout: "down_stamp": "0.000000", 2024-09-13T11:23:02.732 INFO:teuthology.orchestra.run.smithi001.stdout: "laggy_probability": 0, 2024-09-13T11:23:02.732 INFO:teuthology.orchestra.run.smithi001.stdout: "laggy_interval": 0, 2024-09-13T11:23:02.732 INFO:teuthology.orchestra.run.smithi001.stdout: "features": 4540719139924082687, 2024-09-13T11:23:02.732 INFO:teuthology.orchestra.run.smithi001.stdout: "old_weight": 0, 2024-09-13T11:23:02.732 INFO:teuthology.orchestra.run.smithi001.stdout: "last_purged_snaps_scrub": "2024-09-13T11:22:08.223142+0000", 2024-09-13T11:23:02.732 INFO:teuthology.orchestra.run.smithi001.stdout: "dead_epoch": 0 2024-09-13T11:23:02.733 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:02.733 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:02.733 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": 12, 2024-09-13T11:23:02.733 INFO:teuthology.orchestra.run.smithi001.stdout: "down_stamp": "0.000000", 2024-09-13T11:23:02.733 INFO:teuthology.orchestra.run.smithi001.stdout: "laggy_probability": 0, 2024-09-13T11:23:02.733 INFO:teuthology.orchestra.run.smithi001.stdout: "laggy_interval": 0, 2024-09-13T11:23:02.733 INFO:teuthology.orchestra.run.smithi001.stdout: "features": 4540719139924082687, 2024-09-13T11:23:02.733 INFO:teuthology.orchestra.run.smithi001.stdout: "old_weight": 0, 2024-09-13T11:23:02.733 INFO:teuthology.orchestra.run.smithi001.stdout: "last_purged_snaps_scrub": "2024-09-13T11:22:08.687866+0000", 2024-09-13T11:23:02.733 INFO:teuthology.orchestra.run.smithi001.stdout: "dead_epoch": 0 2024-09-13T11:23:02.733 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:02.733 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:02.733 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": 13, 2024-09-13T11:23:02.733 INFO:teuthology.orchestra.run.smithi001.stdout: "down_stamp": "0.000000", 2024-09-13T11:23:02.734 INFO:teuthology.orchestra.run.smithi001.stdout: "laggy_probability": 0, 2024-09-13T11:23:02.734 INFO:teuthology.orchestra.run.smithi001.stdout: "laggy_interval": 0, 2024-09-13T11:23:02.734 INFO:teuthology.orchestra.run.smithi001.stdout: "features": 4540719139924082687, 2024-09-13T11:23:02.734 INFO:teuthology.orchestra.run.smithi001.stdout: "old_weight": 0, 2024-09-13T11:23:02.734 INFO:teuthology.orchestra.run.smithi001.stdout: "last_purged_snaps_scrub": "2024-09-13T11:22:08.347174+0000", 2024-09-13T11:23:02.734 INFO:teuthology.orchestra.run.smithi001.stdout: "dead_epoch": 0 2024-09-13T11:23:02.734 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:02.734 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:02.734 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": 14, 2024-09-13T11:23:02.734 INFO:teuthology.orchestra.run.smithi001.stdout: "down_stamp": "0.000000", 2024-09-13T11:23:02.734 INFO:teuthology.orchestra.run.smithi001.stdout: "laggy_probability": 0, 2024-09-13T11:23:02.734 INFO:teuthology.orchestra.run.smithi001.stdout: "laggy_interval": 0, 2024-09-13T11:23:02.734 INFO:teuthology.orchestra.run.smithi001.stdout: "features": 4540719139924082687, 2024-09-13T11:23:02.734 INFO:teuthology.orchestra.run.smithi001.stdout: "old_weight": 0, 2024-09-13T11:23:02.735 INFO:teuthology.orchestra.run.smithi001.stdout: "last_purged_snaps_scrub": "2024-09-13T11:22:08.223685+0000", 2024-09-13T11:23:02.735 INFO:teuthology.orchestra.run.smithi001.stdout: "dead_epoch": 0 2024-09-13T11:23:02.735 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:02.735 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:02.735 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": 15, 2024-09-13T11:23:02.735 INFO:teuthology.orchestra.run.smithi001.stdout: "down_stamp": "0.000000", 2024-09-13T11:23:02.735 INFO:teuthology.orchestra.run.smithi001.stdout: "laggy_probability": 0, 2024-09-13T11:23:02.735 INFO:teuthology.orchestra.run.smithi001.stdout: "laggy_interval": 0, 2024-09-13T11:23:02.735 INFO:teuthology.orchestra.run.smithi001.stdout: "features": 4540719139924082687, 2024-09-13T11:23:02.735 INFO:teuthology.orchestra.run.smithi001.stdout: "old_weight": 0, 2024-09-13T11:23:02.735 INFO:teuthology.orchestra.run.smithi001.stdout: "last_purged_snaps_scrub": "2024-09-13T11:22:08.170891+0000", 2024-09-13T11:23:02.735 INFO:teuthology.orchestra.run.smithi001.stdout: "dead_epoch": 0 2024-09-13T11:23:02.735 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:02.735 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-09-13T11:23:02.736 INFO:teuthology.orchestra.run.smithi001.stdout: "pg_upmap": [], 2024-09-13T11:23:02.736 INFO:teuthology.orchestra.run.smithi001.stdout: "pg_upmap_items": [], 2024-09-13T11:23:02.736 INFO:teuthology.orchestra.run.smithi001.stdout: "pg_upmap_primaries": [], 2024-09-13T11:23:02.736 INFO:teuthology.orchestra.run.smithi001.stdout: "pg_temp": [], 2024-09-13T11:23:02.736 INFO:teuthology.orchestra.run.smithi001.stdout: "primary_temp": [], 2024-09-13T11:23:02.736 INFO:teuthology.orchestra.run.smithi001.stdout: "blocklist": {}, 2024-09-13T11:23:02.736 INFO:teuthology.orchestra.run.smithi001.stdout: "range_blocklist": {}, 2024-09-13T11:23:02.736 INFO:teuthology.orchestra.run.smithi001.stdout: "erasure_code_profiles": { 2024-09-13T11:23:02.736 INFO:teuthology.orchestra.run.smithi001.stdout: "default": { 2024-09-13T11:23:02.736 INFO:teuthology.orchestra.run.smithi001.stdout: "crush-failure-domain": "osd", 2024-09-13T11:23:02.736 INFO:teuthology.orchestra.run.smithi001.stdout: "k": "2", 2024-09-13T11:23:02.736 INFO:teuthology.orchestra.run.smithi001.stdout: "m": "1", 2024-09-13T11:23:02.736 INFO:teuthology.orchestra.run.smithi001.stdout: "plugin": "jerasure", 2024-09-13T11:23:02.737 INFO:teuthology.orchestra.run.smithi001.stdout: "technique": "reed_sol_van" 2024-09-13T11:23:02.737 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:02.737 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:02.737 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_snaps_queue": [], 2024-09-13T11:23:02.737 INFO:teuthology.orchestra.run.smithi001.stdout: "new_removed_snaps": [], 2024-09-13T11:23:02.737 INFO:teuthology.orchestra.run.smithi001.stdout: "new_purged_snaps": [], 2024-09-13T11:23:02.737 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_node_flags": {}, 2024-09-13T11:23:02.737 INFO:teuthology.orchestra.run.smithi001.stdout: "device_class_flags": {}, 2024-09-13T11:23:02.737 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": { 2024-09-13T11:23:02.737 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode_enabled": false, 2024-09-13T11:23:02.737 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_bucket_count": 0, 2024-09-13T11:23:02.737 INFO:teuthology.orchestra.run.smithi001.stdout: "degraded_stretch_mode": 0, 2024-09-13T11:23:02.737 INFO:teuthology.orchestra.run.smithi001.stdout: "recovering_stretch_mode": 0, 2024-09-13T11:23:02.737 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode_bucket": 0 2024-09-13T11:23:02.738 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:02.738 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-13T11:23:02.740 INFO:tasks.thrashosds.thrasher:j is {'epoch': 55, 'fsid': '2fd68ac7-5581-4d90-80a4-30bcf5252a7a', 'created': '2024-09-13T11:21:57.223153+0000', 'modified': '2024-09-13T11:23:01.422282+0000', 'last_up_change': '2024-09-13T11:22:46.271946+0000', 'last_in_change': '2024-09-13T11:22:01.745814+0000', 'flags': 'noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5801984, 'flags_set': ['noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 6, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 3, 'max_osd': 16, 'require_min_compat_client': 'luminous', 'min_compat_client': 'jewel', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-13T11:22:10.162266+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 15.380000114440918, 'score_stable': 15.380000114440918, 'optimal_score': 0.12999999523162842, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2024-09-13T11:22:12.235549+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '28', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 2, 'snap_epoch': 28, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 4.019999980926514, 'score_stable': 4.019999980926514, 'optimal_score': 0.5600000023841858, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'unique_pool_0', 'create_time': '2024-09-13T11:22:24.305521+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '32', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 4, 'score_stable': 4, 'optimal_score': 0.75, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}], 'osds': [{'osd': 0, 'uuid': '0fc12b3c-e0de-4546-b935-db77efe7c567', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 22, 'up_thru': 0, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.1:6800', 'nonce': 1544776477}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.1:6801', 'nonce': 1544776477}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.1:6803', 'nonce': 1544776477}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.1:6802', 'nonce': 1544776477}]}, 'public_addr': '172.21.15.1:6800/1544776477', 'cluster_addr': '172.21.15.1:6801/1544776477', 'heartbeat_back_addr': '172.21.15.1:6803/1544776477', 'heartbeat_front_addr': '172.21.15.1:6802/1544776477', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': 'b77fe331-cc92-4adb-b146-af5c380f6d4f', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 23, 'last_clean_end': 29, 'up_from': 40, 'up_thru': 40, 'down_at': 30, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.92:6800', 'nonce': 3591430847}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.92:6801', 'nonce': 3591430847}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.92:6803', 'nonce': 3591430847}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.92:6802', 'nonce': 3591430847}]}, 'public_addr': '172.21.15.92:6800/3591430847', 'cluster_addr': '172.21.15.92:6801/3591430847', 'heartbeat_back_addr': '172.21.15.92:6803/3591430847', 'heartbeat_front_addr': '172.21.15.92:6802/3591430847', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': 'c9d031c8-e885-4229-affa-08e1729d00e8', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 22, 'up_thru': 0, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.162:6812', 'nonce': 2712148101}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.162:6813', 'nonce': 2712148101}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.162:6815', 'nonce': 2712148101}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.162:6814', 'nonce': 2712148101}]}, 'public_addr': '172.21.15.162:6812/2712148101', 'cluster_addr': '172.21.15.162:6813/2712148101', 'heartbeat_back_addr': '172.21.15.162:6815/2712148101', 'heartbeat_front_addr': '172.21.15.162:6814/2712148101', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': 'bf50d71a-6c3a-4fbc-a407-cb81bad14a7b', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 21, 'up_thru': 30, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.177:6801', 'nonce': 3367751690}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.177:6802', 'nonce': 3367751690}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.177:6804', 'nonce': 3367751690}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.177:6803', 'nonce': 3367751690}]}, 'public_addr': '172.21.15.177:6801/3367751690', 'cluster_addr': '172.21.15.177:6802/3367751690', 'heartbeat_back_addr': '172.21.15.177:6804/3367751690', 'heartbeat_front_addr': '172.21.15.177:6803/3367751690', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': 'd1fe3f63-df3f-4a73-a1e2-8d7c8a3f1fa0', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 22, 'up_thru': 0, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.1:6808', 'nonce': 3648540926}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.1:6809', 'nonce': 3648540926}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.1:6811', 'nonce': 3648540926}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.1:6810', 'nonce': 3648540926}]}, 'public_addr': '172.21.15.1:6808/3648540926', 'cluster_addr': '172.21.15.1:6809/3648540926', 'heartbeat_back_addr': '172.21.15.1:6811/3648540926', 'heartbeat_front_addr': '172.21.15.1:6810/3648540926', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': 'e2ead447-2f2c-40c9-82e7-53b8dc41e2af', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 21, 'up_thru': 0, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.92:6804', 'nonce': 3399529091}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.92:6805', 'nonce': 3399529091}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.92:6807', 'nonce': 3399529091}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.92:6806', 'nonce': 3399529091}]}, 'public_addr': '172.21.15.92:6804/3399529091', 'cluster_addr': '172.21.15.92:6805/3399529091', 'heartbeat_back_addr': '172.21.15.92:6807/3399529091', 'heartbeat_front_addr': '172.21.15.92:6806/3399529091', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': 'dbc20439-144e-4ec7-89de-674eb783f263', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 22, 'up_thru': 40, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.162:6808', 'nonce': 2287343806}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.162:6809', 'nonce': 2287343806}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.162:6811', 'nonce': 2287343806}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.162:6810', 'nonce': 2287343806}]}, 'public_addr': '172.21.15.162:6808/2287343806', 'cluster_addr': '172.21.15.162:6809/2287343806', 'heartbeat_back_addr': '172.21.15.162:6811/2287343806', 'heartbeat_front_addr': '172.21.15.162:6810/2287343806', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '60aa5206-cace-40f8-8d1a-10ec28d97096', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 22, 'up_thru': 30, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.177:6813', 'nonce': 2110809906}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.177:6814', 'nonce': 2110809906}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.177:6816', 'nonce': 2110809906}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.177:6815', 'nonce': 2110809906}]}, 'public_addr': '172.21.15.177:6813/2110809906', 'cluster_addr': '172.21.15.177:6814/2110809906', 'heartbeat_back_addr': '172.21.15.177:6816/2110809906', 'heartbeat_front_addr': '172.21.15.177:6815/2110809906', 'state': ['exists', 'up']}, {'osd': 8, 'uuid': 'f3d99325-48ff-403c-8662-1d7745f7c97d', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 22, 'up_thru': 30, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.1:6804', 'nonce': 2766567910}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.1:6805', 'nonce': 2766567910}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.1:6807', 'nonce': 2766567910}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.1:6806', 'nonce': 2766567910}]}, 'public_addr': '172.21.15.1:6804/2766567910', 'cluster_addr': '172.21.15.1:6805/2766567910', 'heartbeat_back_addr': '172.21.15.1:6807/2766567910', 'heartbeat_front_addr': '172.21.15.1:6806/2766567910', 'state': ['exists', 'up']}, {'osd': 9, 'uuid': '88511e24-3675-4551-968c-1a52e7e28f96', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 23, 'up_thru': 30, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.92:6812', 'nonce': 2488187511}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.92:6813', 'nonce': 2488187511}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.92:6815', 'nonce': 2488187511}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.92:6814', 'nonce': 2488187511}]}, 'public_addr': '172.21.15.92:6812/2488187511', 'cluster_addr': '172.21.15.92:6813/2488187511', 'heartbeat_back_addr': '172.21.15.92:6815/2488187511', 'heartbeat_front_addr': '172.21.15.92:6814/2488187511', 'state': ['exists', 'up']}, {'osd': 10, 'uuid': 'acf4c64d-99af-42be-8406-413c86f54670', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 22, 'up_thru': 30, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.162:6804', 'nonce': 1450266459}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.162:6805', 'nonce': 1450266459}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.162:6807', 'nonce': 1450266459}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.162:6806', 'nonce': 1450266459}]}, 'public_addr': '172.21.15.162:6804/1450266459', 'cluster_addr': '172.21.15.162:6805/1450266459', 'heartbeat_back_addr': '172.21.15.162:6807/1450266459', 'heartbeat_front_addr': '172.21.15.162:6806/1450266459', 'state': ['exists', 'up']}, {'osd': 11, 'uuid': 'a4bfdfbe-b586-4080-8e09-6f26428e59b5', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 22, 'up_thru': 30, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.177:6809', 'nonce': 3239521954}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.177:6810', 'nonce': 3239521954}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.177:6812', 'nonce': 3239521954}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.177:6811', 'nonce': 3239521954}]}, 'public_addr': '172.21.15.177:6809/3239521954', 'cluster_addr': '172.21.15.177:6810/3239521954', 'heartbeat_back_addr': '172.21.15.177:6812/3239521954', 'heartbeat_front_addr': '172.21.15.177:6811/3239521954', 'state': ['exists', 'up']}, {'osd': 12, 'uuid': 'dc571fb6-d8b5-465b-bb25-afa7d0c47b5a', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 22, 'up_thru': 30, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.1:6812', 'nonce': 4267062742}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.1:6813', 'nonce': 4267062742}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.1:6815', 'nonce': 4267062742}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.1:6814', 'nonce': 4267062742}]}, 'public_addr': '172.21.15.1:6812/4267062742', 'cluster_addr': '172.21.15.1:6813/4267062742', 'heartbeat_back_addr': '172.21.15.1:6815/4267062742', 'heartbeat_front_addr': '172.21.15.1:6814/4267062742', 'state': ['exists', 'up']}, {'osd': 13, 'uuid': '5d321d4b-1b26-47fe-bf84-25751b8cde7d', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 23, 'up_thru': 41, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.92:6808', 'nonce': 1759585046}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.92:6809', 'nonce': 1759585046}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.92:6811', 'nonce': 1759585046}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.92:6810', 'nonce': 1759585046}]}, 'public_addr': '172.21.15.92:6808/1759585046', 'cluster_addr': '172.21.15.92:6809/1759585046', 'heartbeat_back_addr': '172.21.15.92:6811/1759585046', 'heartbeat_front_addr': '172.21.15.92:6810/1759585046', 'state': ['exists', 'up']}, {'osd': 14, 'uuid': 'fc3e2499-0c2a-414d-a25e-2628876ae630', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 22, 'up_thru': 40, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.162:6800', 'nonce': 3905144320}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.162:6801', 'nonce': 3905144320}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.162:6803', 'nonce': 3905144320}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.162:6802', 'nonce': 3905144320}]}, 'public_addr': '172.21.15.162:6800/3905144320', 'cluster_addr': '172.21.15.162:6801/3905144320', 'heartbeat_back_addr': '172.21.15.162:6803/3905144320', 'heartbeat_front_addr': '172.21.15.162:6802/3905144320', 'state': ['exists', 'up']}, {'osd': 15, 'uuid': 'd2f69775-edfe-4527-91ec-f7c7e558bec8', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 22, 'up_thru': 30, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.177:6805', 'nonce': 1294337172}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.177:6806', 'nonce': 1294337172}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.177:6808', 'nonce': 1294337172}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.177:6807', 'nonce': 1294337172}]}, 'public_addr': '172.21.15.177:6805/1294337172', 'cluster_addr': '172.21.15.177:6806/1294337172', 'heartbeat_back_addr': '172.21.15.177:6808/1294337172', 'heartbeat_front_addr': '172.21.15.177:6807/1294337172', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-13T11:22:08.233274+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-13T11:22:08.032163+0000', 'dead_epoch': 30}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-13T11:22:08.875236+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-13T11:22:07.936983+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-13T11:22:08.625189+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-13T11:22:07.944287+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-13T11:22:08.351045+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-13T11:22:08.835090+0000', 'dead_epoch': 0}, {'osd': 8, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-13T11:22:08.847356+0000', 'dead_epoch': 0}, {'osd': 9, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-13T11:22:08.140353+0000', 'dead_epoch': 0}, {'osd': 10, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-13T11:22:08.218416+0000', 'dead_epoch': 0}, {'osd': 11, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-13T11:22:08.223142+0000', 'dead_epoch': 0}, {'osd': 12, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-13T11:22:08.687866+0000', 'dead_epoch': 0}, {'osd': 13, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-13T11:22:08.347174+0000', 'dead_epoch': 0}, {'osd': 14, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-13T11:22:08.223685+0000', 'dead_epoch': 0}, {'osd': 15, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-13T11:22:08.170891+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-09-13T11:23:02.740 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-13T11:23:02.741 INFO:tasks.ceph.osd.2.smithi162.stderr:2024-09-13T11:23:02.735+0000 7fe92b35a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26313) UID: 0 2024-09-13T11:23:02.744 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-09-13T11:23:02.745 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:23:02.761 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-09-13T11:23:02.840 INFO:tasks.ceph.osd.4.smithi001.stderr:2024-09-13T11:23:02.836+0000 7ff8e0218640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26310) UID: 0 2024-09-13T11:23:02.890 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-09-13T11:23:02.890 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:23:02.905 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-09-13T11:23:02.954 INFO:tasks.ceph.osd.15.smithi177.stderr:2024-09-13T11:23:02.955+0000 7f9d6ce77640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26578) UID: 0 2024-09-13T11:23:03.000 INFO:tasks.radosbench.radosbench.0.smithi177.stdout: 36 16 23043 23027 39.9725 76.625 0.0632431 0.0250063 2024-09-13T11:23:03.041 INFO:tasks.ceph.osd.13.smithi092.stderr:2024-09-13T11:23:03.038+0000 7f3e5afea640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26350) UID: 0 2024-09-13T11:23:03.049 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-09-13T11:23:03.050 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:23:03.067 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-09-13T11:23:03.075 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-13T11:23:03.076 INFO:teuthology.orchestra.run.smithi001.stderr:dumped all 2024-09-13T11:23:03.096 INFO:teuthology.orchestra.run.smithi001.stdout:{"pg_ready":true,"pg_map":{"version":67,"stamp":"2024-09-13T11:23:02.126829+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1368719923,"num_objects":20880,"num_object_clones":0,"num_object_copies":41760,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20880,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":21023,"num_write_kb":1338738,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":20991,"ondisk_log_size":20991,"up":50,"acting":50,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":50,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":15,"kb":1509949440,"kb_used":3098464,"kb_used_data":2561248,"kb_used_omap":27,"kb_used_meta":536932,"kb_avail":1506850976,"statfs":{"total":1546188226560,"available":1543015399424,"internally_reserved":0,"allocated":2622717952,"data_stored":2618863585,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28358,"internal_metadata":549818682},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,0,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":529793024,"num_objects":8084,"num_object_clones":0,"num_object_copies":16168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8084,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8084,"num_write_kb":517376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000749"},"pg_stats":[{"pgid":"3.f","version":"55'1385","reported_seq":1444,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-13T11:23:01.817226+0000","last_change":"2024-09-13T11:22:25.925951+0000","last_active":"2024-09-13T11:23:01.817226+0000","last_peered":"2024-09-13T11:23:01.817226+0000","last_clean":"2024-09-13T11:23:01.817226+0000","last_became_active":"2024-09-13T11:22:25.925169+0000","last_became_peered":"2024-09-13T11:22:25.925169+0000","last_unstale":"2024-09-13T11:23:01.817226+0000","last_undegraded":"2024-09-13T11:23:01.817226+0000","last_fullsized":"2024-09-13T11:23:01.817226+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:22:24.896055+0000","objects_scrubbed":0,"log_size":1385,"log_dups_size":0,"ondisk_log_size":1385,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:23:38.143926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90767360,"num_objects":1385,"num_object_clones":0,"num_object_copies":2770,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1385,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1385,"num_write_kb":88640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,12],"acting":[11,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.e","version":"54'1377","reported_seq":1432,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-13T11:23:01.012203+0000","last_change":"2024-09-13T11:22:25.923829+0000","last_active":"2024-09-13T11:23:01.012203+0000","last_peered":"2024-09-13T11:23:01.012203+0000","last_clean":"2024-09-13T11:23:01.012203+0000","last_became_active":"2024-09-13T11:22:25.923498+0000","last_became_peered":"2024-09-13T11:22:25.923498+0000","last_unstale":"2024-09-13T11:23:01.012203+0000","last_undegraded":"2024-09-13T11:23:01.012203+0000","last_fullsized":"2024-09-13T11:23:01.012203+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:22:24.896055+0000","objects_scrubbed":0,"log_size":1377,"log_dups_size":0,"ondisk_log_size":1377,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:23:50.687438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90243072,"num_objects":1377,"num_object_clones":0,"num_object_copies":2754,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1377,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1377,"num_write_kb":88128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,14],"acting":[8,14],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.d","version":"54'1291","reported_seq":1346,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-13T11:23:01.014481+0000","last_change":"2024-09-13T11:22:25.927325+0000","last_active":"2024-09-13T11:23:01.014481+0000","last_peered":"2024-09-13T11:23:01.014481+0000","last_clean":"2024-09-13T11:23:01.014481+0000","last_became_active":"2024-09-13T11:22:25.926433+0000","last_became_peered":"2024-09-13T11:22:25.926433+0000","last_unstale":"2024-09-13T11:23:01.014481+0000","last_undegraded":"2024-09-13T11:23:01.014481+0000","last_fullsized":"2024-09-13T11:23:01.014481+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:22:24.896055+0000","objects_scrubbed":0,"log_size":1291,"log_dups_size":0,"ondisk_log_size":1291,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:23:39.281487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":84606976,"num_objects":1291,"num_object_clones":0,"num_object_copies":2582,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1291,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1291,"num_write_kb":82624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.c","version":"55'1371","reported_seq":1430,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-13T11:23:02.018208+0000","last_change":"2024-09-13T11:22:25.934619+0000","last_active":"2024-09-13T11:23:02.018208+0000","last_peered":"2024-09-13T11:23:02.018208+0000","last_clean":"2024-09-13T11:23:02.018208+0000","last_became_active":"2024-09-13T11:22:25.933854+0000","last_became_peered":"2024-09-13T11:22:25.933854+0000","last_unstale":"2024-09-13T11:23:02.018208+0000","last_undegraded":"2024-09-13T11:23:02.018208+0000","last_fullsized":"2024-09-13T11:23:02.018208+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:22:24.896055+0000","objects_scrubbed":0,"log_size":1371,"log_dups_size":0,"ondisk_log_size":1371,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:23:42.213952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":89849856,"num_objects":1371,"num_object_clones":0,"num_object_copies":2742,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1371,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1371,"num_write_kb":87744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,13],"acting":[11,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.b","version":"55'1407","reported_seq":1465,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-13T11:23:02.010994+0000","last_change":"2024-09-13T11:22:49.175600+0000","last_active":"2024-09-13T11:23:02.010994+0000","last_peered":"2024-09-13T11:23:02.010994+0000","last_clean":"2024-09-13T11:23:02.010994+0000","last_became_active":"2024-09-13T11:22:49.167083+0000","last_became_peered":"2024-09-13T11:22:49.167083+0000","last_unstale":"2024-09-13T11:23:02.010994+0000","last_undegraded":"2024-09-13T11:23:02.010994+0000","last_fullsized":"2024-09-13T11:23:02.010994+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:22:24.896055+0000","objects_scrubbed":0,"log_size":1407,"log_dups_size":0,"ondisk_log_size":1407,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:23:45.371983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92209152,"num_objects":1407,"num_object_clones":0,"num_object_copies":2814,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1407,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1407,"num_write_kb":90048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,1],"acting":[13,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"3.a","version":"53'1150","reported_seq":1205,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-13T11:22:59.968907+0000","last_change":"2024-09-13T11:22:25.932217+0000","last_active":"2024-09-13T11:22:59.968907+0000","last_peered":"2024-09-13T11:22:59.968907+0000","last_clean":"2024-09-13T11:22:59.968907+0000","last_became_active":"2024-09-13T11:22:25.931493+0000","last_became_peered":"2024-09-13T11:22:25.931493+0000","last_unstale":"2024-09-13T11:22:59.968907+0000","last_undegraded":"2024-09-13T11:22:59.968907+0000","last_fullsized":"2024-09-13T11:22:59.968907+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:22:24.896055+0000","objects_scrubbed":0,"log_size":1150,"log_dups_size":0,"ondisk_log_size":1150,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:23:33.302955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":75366400,"num_objects":1150,"num_object_clones":0,"num_object_copies":2300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1150,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1150,"num_write_kb":73600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.9","version":"54'1283","reported_seq":1339,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-13T11:23:00.989373+0000","last_change":"2024-09-13T11:22:25.922876+0000","last_active":"2024-09-13T11:23:00.989373+0000","last_peered":"2024-09-13T11:23:00.989373+0000","last_clean":"2024-09-13T11:23:00.989373+0000","last_became_active":"2024-09-13T11:22:25.922498+0000","last_became_peered":"2024-09-13T11:22:25.922498+0000","last_unstale":"2024-09-13T11:23:00.989373+0000","last_undegraded":"2024-09-13T11:23:00.989373+0000","last_fullsized":"2024-09-13T11:23:00.989373+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:22:24.896055+0000","objects_scrubbed":0,"log_size":1283,"log_dups_size":0,"ondisk_log_size":1283,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:23:29.835976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":84082688,"num_objects":1283,"num_object_clones":0,"num_object_copies":2566,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1283,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1283,"num_write_kb":82112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,2],"acting":[12,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[]},{"pgid":"3.8","version":"54'1273","reported_seq":1331,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-13T11:23:01.017914+0000","last_change":"2024-09-13T11:22:47.301262+0000","last_active":"2024-09-13T11:23:01.017914+0000","last_peered":"2024-09-13T11:23:01.017914+0000","last_clean":"2024-09-13T11:23:01.017914+0000","last_became_active":"2024-09-13T11:22:47.296719+0000","last_became_peered":"2024-09-13T11:22:47.296719+0000","last_unstale":"2024-09-13T11:23:01.017914+0000","last_undegraded":"2024-09-13T11:23:01.017914+0000","last_fullsized":"2024-09-13T11:23:01.017914+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:22:24.896055+0000","objects_scrubbed":0,"log_size":1273,"log_dups_size":0,"ondisk_log_size":1273,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:23:54.227881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":83427328,"num_objects":1273,"num_object_clones":0,"num_object_copies":2546,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1273,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1273,"num_write_kb":81472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"26'112","reported_seq":206,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-13T11:23:00.440566+0000","last_change":"2024-09-13T11:22:12.131711+0000","last_active":"2024-09-13T11:23:00.440566+0000","last_peered":"2024-09-13T11:23:00.440566+0000","last_clean":"2024-09-13T11:23:00.440566+0000","last_became_active":"2024-09-13T11:22:12.129721+0000","last_became_peered":"2024-09-13T11:22:12.129721+0000","last_unstale":"2024-09-13T11:23:00.440566+0000","last_undegraded":"2024-09-13T11:23:00.440566+0000","last_fullsized":"2024-09-13T11:23:00.440566+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:11.098388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:11.098388+0000","last_clean_scrub_stamp":"2024-09-13T11:22:11.098388+0000","objects_scrubbed":0,"log_size":112,"log_dups_size":0,"ondisk_log_size":112,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:23:40.174992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.3","version":"26'1","reported_seq":73,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-13T11:23:00.425006+0000","last_change":"2024-09-13T11:22:16.547691+0000","last_active":"2024-09-13T11:23:00.425006+0000","last_peered":"2024-09-13T11:23:00.425006+0000","last_clean":"2024-09-13T11:23:00.425006+0000","last_became_active":"2024-09-13T11:22:14.146507+0000","last_became_peered":"2024-09-13T11:22:14.146507+0000","last_unstale":"2024-09-13T11:23:00.425006+0000","last_undegraded":"2024-09-13T11:23:00.425006+0000","last_fullsized":"2024-09-13T11:23:00.425006+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:22:13.113563+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:23:18.785562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00085610999999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,15],"acting":[14,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"3.2","version":"54'1283","reported_seq":1340,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-13T11:23:01.028296+0000","last_change":"2024-09-13T11:22:25.923575+0000","last_active":"2024-09-13T11:23:01.028296+0000","last_peered":"2024-09-13T11:23:01.028296+0000","last_clean":"2024-09-13T11:23:01.028296+0000","last_became_active":"2024-09-13T11:22:25.922665+0000","last_became_peered":"2024-09-13T11:22:25.922665+0000","last_unstale":"2024-09-13T11:23:01.028296+0000","last_undegraded":"2024-09-13T11:23:01.028296+0000","last_fullsized":"2024-09-13T11:23:01.028296+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:22:24.896055+0000","objects_scrubbed":0,"log_size":1283,"log_dups_size":0,"ondisk_log_size":1283,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:23:37.611518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":84082688,"num_objects":1283,"num_object_clones":0,"num_object_copies":2566,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1283,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1283,"num_write_kb":82112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,12],"acting":[14,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":70,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-13T11:22:59.422067+0000","last_change":"2024-09-13T11:22:16.267999+0000","last_active":"2024-09-13T11:22:59.422067+0000","last_peered":"2024-09-13T11:22:59.422067+0000","last_clean":"2024-09-13T11:22:59.422067+0000","last_became_active":"2024-09-13T11:22:14.211435+0000","last_became_peered":"2024-09-13T11:22:14.211435+0000","last_unstale":"2024-09-13T11:22:59.422067+0000","last_undegraded":"2024-09-13T11:22:59.422067+0000","last_fullsized":"2024-09-13T11:22:59.422067+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:22:13.113563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:23:24.076640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010010360000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"55'1330","reported_seq":1388,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-13T11:23:01.795235+0000","last_change":"2024-09-13T11:22:25.933242+0000","last_active":"2024-09-13T11:23:01.795235+0000","last_peered":"2024-09-13T11:23:01.795235+0000","last_clean":"2024-09-13T11:23:01.795235+0000","last_became_active":"2024-09-13T11:22:25.932868+0000","last_became_peered":"2024-09-13T11:22:25.932868+0000","last_unstale":"2024-09-13T11:23:01.795235+0000","last_undegraded":"2024-09-13T11:23:01.795235+0000","last_fullsized":"2024-09-13T11:23:01.795235+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:22:24.896055+0000","objects_scrubbed":0,"log_size":1330,"log_dups_size":0,"ondisk_log_size":1330,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:23:38.369067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":87162880,"num_objects":1330,"num_object_clones":0,"num_object_copies":2660,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1330,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1330,"num_write_kb":85120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,11],"acting":[13,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":81,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-13T11:23:00.424989+0000","last_change":"2024-09-13T11:22:47.298206+0000","last_active":"2024-09-13T11:23:00.424989+0000","last_peered":"2024-09-13T11:23:00.424989+0000","last_clean":"2024-09-13T11:23:00.424989+0000","last_became_active":"2024-09-13T11:22:47.297319+0000","last_became_peered":"2024-09-13T11:22:47.297319+0000","last_unstale":"2024-09-13T11:23:00.424989+0000","last_undegraded":"2024-09-13T11:23:00.424989+0000","last_fullsized":"2024-09-13T11:23:00.424989+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:22:13.113563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:23:23.483800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00081225399999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,1],"acting":[14,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"3.0","version":"54'1291","reported_seq":1347,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-13T11:23:01.006156+0000","last_change":"2024-09-13T11:22:25.926230+0000","last_active":"2024-09-13T11:23:01.006156+0000","last_peered":"2024-09-13T11:23:01.006156+0000","last_clean":"2024-09-13T11:23:01.006156+0000","last_became_active":"2024-09-13T11:22:25.925791+0000","last_became_peered":"2024-09-13T11:22:25.925791+0000","last_unstale":"2024-09-13T11:23:01.006156+0000","last_undegraded":"2024-09-13T11:23:01.006156+0000","last_fullsized":"2024-09-13T11:23:01.006156+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:22:24.896055+0000","objects_scrubbed":0,"log_size":1291,"log_dups_size":0,"ondisk_log_size":1291,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:23:38.551603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":84606976,"num_objects":1291,"num_object_clones":0,"num_object_copies":2582,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1291,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1291,"num_write_kb":82624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,15],"acting":[12,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[]},{"pgid":"2.2","version":"28'2","reported_seq":72,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-13T11:22:59.422104+0000","last_change":"2024-09-13T11:22:16.269626+0000","last_active":"2024-09-13T11:22:59.422104+0000","last_peered":"2024-09-13T11:22:59.422104+0000","last_clean":"2024-09-13T11:22:59.422104+0000","last_became_active":"2024-09-13T11:22:14.144472+0000","last_became_peered":"2024-09-13T11:22:14.144472+0000","last_unstale":"2024-09-13T11:22:59.422104+0000","last_undegraded":"2024-09-13T11:22:59.422104+0000","last_fullsized":"2024-09-13T11:22:59.422104+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:22:13.113563+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:23:28.879995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0025826730000000002,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9],"acting":[3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.3","version":"54'1290","reported_seq":1345,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-13T11:23:01.026174+0000","last_change":"2024-09-13T11:22:25.930558+0000","last_active":"2024-09-13T11:23:01.026174+0000","last_peered":"2024-09-13T11:23:01.026174+0000","last_clean":"2024-09-13T11:23:01.026174+0000","last_became_active":"2024-09-13T11:22:25.930023+0000","last_became_peered":"2024-09-13T11:22:25.930023+0000","last_unstale":"2024-09-13T11:23:01.026174+0000","last_undegraded":"2024-09-13T11:23:01.026174+0000","last_fullsized":"2024-09-13T11:23:01.026174+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:22:24.896055+0000","objects_scrubbed":0,"log_size":1290,"log_dups_size":0,"ondisk_log_size":1290,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:23:33.097704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":84541440,"num_objects":1290,"num_object_clones":0,"num_object_copies":2580,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1290,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1290,"num_write_kb":82560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,7],"acting":[10,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.5","version":"54'1259","reported_seq":1315,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-13T11:23:01.021747+0000","last_change":"2024-09-13T11:22:25.930005+0000","last_active":"2024-09-13T11:23:01.021747+0000","last_peered":"2024-09-13T11:23:01.021747+0000","last_clean":"2024-09-13T11:23:01.021747+0000","last_became_active":"2024-09-13T11:22:25.929242+0000","last_became_peered":"2024-09-13T11:22:25.929242+0000","last_unstale":"2024-09-13T11:23:01.021747+0000","last_undegraded":"2024-09-13T11:23:01.021747+0000","last_fullsized":"2024-09-13T11:23:01.021747+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:22:24.896055+0000","objects_scrubbed":0,"log_size":1259,"log_dups_size":0,"ondisk_log_size":1259,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:23:41.561894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":82509824,"num_objects":1259,"num_object_clones":0,"num_object_copies":2518,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1259,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1259,"num_write_kb":80576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,13],"acting":[15,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":57,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-13T11:22:59.424762+0000","last_change":"2024-09-13T11:22:47.720313+0000","last_active":"2024-09-13T11:22:59.424762+0000","last_peered":"2024-09-13T11:22:59.424762+0000","last_clean":"2024-09-13T11:22:59.424762+0000","last_became_active":"2024-09-13T11:22:47.719636+0000","last_became_peered":"2024-09-13T11:22:47.719636+0000","last_unstale":"2024-09-13T11:22:59.424762+0000","last_undegraded":"2024-09-13T11:22:59.424762+0000","last_fullsized":"2024-09-13T11:22:59.424762+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:22:13.113563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:23:13.812273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00087476000000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,9],"acting":[1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"53'1217","reported_seq":1250,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-13T11:23:00.275446+0000","last_change":"2024-09-13T11:22:47.293720+0000","last_active":"2024-09-13T11:23:00.275446+0000","last_peered":"2024-09-13T11:23:00.275446+0000","last_clean":"2024-09-13T11:23:00.275446+0000","last_became_active":"2024-09-13T11:22:47.292998+0000","last_became_peered":"2024-09-13T11:22:47.292998+0000","last_unstale":"2024-09-13T11:23:00.275446+0000","last_undegraded":"2024-09-13T11:23:00.275446+0000","last_fullsized":"2024-09-13T11:23:00.275446+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:22:24.896055+0000","objects_scrubbed":0,"log_size":1217,"log_dups_size":0,"ondisk_log_size":1217,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:23:42.001016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":79757312,"num_objects":1217,"num_object_clones":0,"num_object_copies":2434,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1217,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1217,"num_write_kb":77888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,12],"acting":[1,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":71,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-13T11:23:00.419532+0000","last_change":"2024-09-13T11:22:16.504060+0000","last_active":"2024-09-13T11:23:00.419532+0000","last_peered":"2024-09-13T11:23:00.419532+0000","last_clean":"2024-09-13T11:23:00.419532+0000","last_became_active":"2024-09-13T11:22:14.581458+0000","last_became_peered":"2024-09-13T11:22:14.581458+0000","last_unstale":"2024-09-13T11:23:00.419532+0000","last_undegraded":"2024-09-13T11:23:00.419532+0000","last_fullsized":"2024-09-13T11:23:00.419532+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:22:13.113563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:23:37.935010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001026047,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5],"acting":[8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.7","version":"55'1304","reported_seq":1362,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-13T11:23:01.800327+0000","last_change":"2024-09-13T11:22:25.924462+0000","last_active":"2024-09-13T11:23:01.800327+0000","last_peered":"2024-09-13T11:23:01.800327+0000","last_clean":"2024-09-13T11:23:01.800327+0000","last_became_active":"2024-09-13T11:22:25.924170+0000","last_became_peered":"2024-09-13T11:22:25.924170+0000","last_unstale":"2024-09-13T11:23:01.800327+0000","last_undegraded":"2024-09-13T11:23:01.800327+0000","last_fullsized":"2024-09-13T11:23:01.800327+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:22:24.896055+0000","objects_scrubbed":0,"log_size":1304,"log_dups_size":0,"ondisk_log_size":1304,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:23:51.945140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":85458944,"num_objects":1304,"num_object_clones":0,"num_object_copies":2608,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1304,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1304,"num_write_kb":83456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,8],"acting":[13,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":71,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-13T11:23:00.974086+0000","last_change":"2024-09-13T11:22:16.260591+0000","last_active":"2024-09-13T11:23:00.974086+0000","last_peered":"2024-09-13T11:23:00.974086+0000","last_clean":"2024-09-13T11:23:00.974086+0000","last_became_active":"2024-09-13T11:22:14.211923+0000","last_became_peered":"2024-09-13T11:22:14.211923+0000","last_unstale":"2024-09-13T11:23:00.974086+0000","last_undegraded":"2024-09-13T11:23:00.974086+0000","last_fullsized":"2024-09-13T11:23:00.974086+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:22:13.113563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:23:20.320636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011689529999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0],"acting":[9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.6","version":"55'1365","reported_seq":1423,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-13T11:23:02.013524+0000","last_change":"2024-09-13T11:22:25.928269+0000","last_active":"2024-09-13T11:23:02.013524+0000","last_peered":"2024-09-13T11:23:02.013524+0000","last_clean":"2024-09-13T11:23:02.013524+0000","last_became_active":"2024-09-13T11:22:25.927961+0000","last_became_peered":"2024-09-13T11:22:25.927961+0000","last_unstale":"2024-09-13T11:23:02.013524+0000","last_undegraded":"2024-09-13T11:23:02.013524+0000","last_fullsized":"2024-09-13T11:23:02.013524+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:22:24.896055+0000","objects_scrubbed":0,"log_size":1365,"log_dups_size":0,"ondisk_log_size":1365,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:23:25.574436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":89456640,"num_objects":1365,"num_object_clones":0,"num_object_copies":2730,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1365,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1365,"num_write_kb":87360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,14],"acting":[13,14],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":71,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-13T11:23:00.426547+0000","last_change":"2024-09-13T11:22:16.889130+0000","last_active":"2024-09-13T11:23:00.426547+0000","last_peered":"2024-09-13T11:23:00.426547+0000","last_clean":"2024-09-13T11:23:00.426547+0000","last_became_active":"2024-09-13T11:22:14.192897+0000","last_became_peered":"2024-09-13T11:22:14.192897+0000","last_unstale":"2024-09-13T11:23:00.426547+0000","last_undegraded":"2024-09-13T11:23:00.426547+0000","last_fullsized":"2024-09-13T11:23:00.426547+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:22:13.113563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:23:26.278353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00084655899999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,10],"acting":[15,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":1368129536,"num_objects":20876,"num_object_clones":0,"num_object_copies":41752,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20876,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20876,"num_write_kb":1336064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2716008448,"data_stored":2716008448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":20876,"ondisk_log_size":20876,"up":32,"acting":32,"num_store_stats":12},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":9},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":112,"ondisk_log_size":112,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":13,"up_from":23,"seq":98784247822,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":540816,"kb_used_data":501312,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":93831024,"statfs":{"total":96636764160,"available":96082968576,"internally_reserved":0,"allocated":513343488,"data_stored":513100754,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,3,4,6,8,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":9,"up_from":23,"seq":98784247821,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":516,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":528384,"data_stored":282017,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,2,3,8,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":40,"seq":171798691844,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":267600,"kb_used_data":227264,"kb_used_omap":4,"kb_used_meta":40315,"kb_avail":94104240,"statfs":{"total":96636764160,"available":96362741760,"internally_reserved":0,"allocated":232718336,"data_stored":232468949,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4433,"internal_metadata":41283247},"hb_peers":[0,2,3,4,5,6,7,8,9,11,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":21,"seq":90194313229,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":180124,"kb_used_data":144780,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94191716,"statfs":{"total":96636764160,"available":96452317184,"internally_reserved":0,"allocated":148254720,"data_stored":147998346,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,2,4,5,7,8,9,10,12,13],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":21,"seq":90194313231,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27572,"kb_used_data":548,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344268,"statfs":{"total":96636764160,"available":96608530432,"internally_reserved":0,"allocated":561152,"data_stored":285775,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":22,"seq":94489280524,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28052,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343788,"statfs":{"total":96636764160,"available":96608038912,"internally_reserved":0,"allocated":1052672,"data_stored":862564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,3,4,5,6,7,8,10,11,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":22,"seq":94489280525,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":111472,"kb_used_data":80288,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94260368,"statfs":{"total":96636764160,"available":96522616832,"internally_reserved":0,"allocated":82214912,"data_stored":81948484,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,6,7,8,10,11,12,13],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":15,"up_from":22,"seq":94489280525,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":186968,"kb_used_data":151624,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94184872,"statfs":{"total":96636764160,"available":96445308928,"internally_reserved":0,"allocated":155262976,"data_stored":155069744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,3,4,5,6,7,10,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":22,"seq":94489280525,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31724,"kb_used_data":540,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340116,"statfs":{"total":96636764160,"available":96604278784,"internally_reserved":0,"allocated":552960,"data_stored":283963,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[3,5,7,8,9,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":22,"seq":94489280525,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":107256,"kb_used_data":76072,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94264584,"statfs":{"total":96636764160,"available":96526934016,"internally_reserved":0,"allocated":77897728,"data_stored":77674803,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,3,5,7,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,0,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":22,"seq":94489280525,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":279476,"kb_used_data":239972,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94092364,"statfs":{"total":96636764160,"available":96350580736,"internally_reserved":0,"allocated":245731328,"data_stored":245517008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":40434122},"hb_peers":[0,3,6,8,9,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":22,"seq":94489280525,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":194748,"kb_used_data":159404,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94177092,"statfs":{"total":96636764160,"available":96437342208,"internally_reserved":0,"allocated":163229696,"data_stored":162948130,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[3,5,7,9,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":22,"seq":94489280524,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":105960,"kb_used_data":74776,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94265880,"statfs":{"total":96636764160,"available":96528261120,"internally_reserved":0,"allocated":76570624,"data_stored":76363107,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,5,7,8,9,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":11,"up_from":22,"seq":94489280526,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":283084,"kb_used_data":256060,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94088756,"statfs":{"total":96636764160,"available":96346886144,"internally_reserved":0,"allocated":262205440,"data_stored":261969920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[2,3,4,5,6,10,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":12,"up_from":22,"seq":94489280525,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":441188,"kb_used_data":401684,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":93930652,"statfs":{"total":96636764160,"available":96184987648,"internally_reserved":0,"allocated":411324416,"data_stored":411065812,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[2,3,4,5,6,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":14,"up_from":22,"seq":94489280525,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":284884,"kb_used_data":245380,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94086956,"statfs":{"total":96636764160,"available":96345042944,"internally_reserved":0,"allocated":251269120,"data_stored":251024209,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,5,6,7,8,9,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":236191744,"data_stored":236191744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":84148224,"data_stored":84148224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":151388160,"data_stored":151388160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":83427328,"data_stored":83427328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":253231104,"data_stored":253231104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":172032000,"data_stored":172032000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":84541440,"data_stored":84541440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":267845632,"data_stored":267845632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":425066496,"data_stored":425066496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":531234816,"data_stored":531234816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":259653632,"data_stored":259653632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":167247872,"data_stored":167247872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T11:23:03.097 INFO:tasks.thrashosds.thrasher:Setting 3.e to [0, 1] 2024-09-13T11:23:03.097 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap', '3.e', '0', '1'] 2024-09-13T11:23:03.098 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pg-upmap 3.e 0 1 2024-09-13T11:23:03.143 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-09-13T11:23:03.139+0000 7f97befc9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26315) UID: 0 2024-09-13T11:23:03.244 INFO:tasks.ceph.osd.7.smithi177.stderr:2024-09-13T11:23:03.243+0000 7f897b51d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26575) UID: 0 2024-09-13T11:23:03.247 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-09-13T11:23:03.248 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:23:03.268 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_ops_in_flight 2024-09-13T11:23:03.305 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T11:23:03.345 INFO:tasks.ceph.osd.5.smithi092.stderr:2024-09-13T11:23:03.342+0000 7f6f0b705640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26346) UID: 0 2024-09-13T11:23:03.445 INFO:tasks.ceph.osd.13.smithi092.stderr:2024-09-13T11:23:03.442+0000 7f3e5afea640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26350) UID: 0 2024-09-13T11:23:03.459 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-13T11:23:03.469 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-09-13T11:23:03.469 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:23:03.492 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_blocked_ops 2024-09-13T11:23:03.494 INFO:teuthology.orchestra.run.smithi001.stdout:osd.0: {} 2024-09-13T11:23:03.495 INFO:teuthology.orchestra.run.smithi001.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T11:23:03.534 INFO:teuthology.orchestra.run.smithi001.stdout:osd.1: {} 2024-09-13T11:23:03.534 INFO:teuthology.orchestra.run.smithi001.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T11:23:03.546 INFO:tasks.ceph.osd.1.smithi092.stderr:2024-09-13T11:23:03.542+0000 7fe7789c5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33718) UID: 0 2024-09-13T11:23:03.574 INFO:teuthology.orchestra.run.smithi001.stdout:osd.2: {} 2024-09-13T11:23:03.574 INFO:teuthology.orchestra.run.smithi001.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T11:23:03.601 INFO:teuthology.orchestra.run.smithi001.stdout:osd.3: {} 2024-09-13T11:23:03.601 INFO:teuthology.orchestra.run.smithi001.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T11:23:03.631 INFO:teuthology.orchestra.run.smithi001.stdout:osd.4: {} 2024-09-13T11:23:03.631 INFO:teuthology.orchestra.run.smithi001.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T11:23:03.647 INFO:tasks.ceph.osd.9.smithi092.stderr:2024-09-13T11:23:03.642+0000 7ff5ab3d0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26348) UID: 0 2024-09-13T11:23:03.651 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-09-13T11:23:03.651 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:23:03.657 INFO:teuthology.orchestra.run.smithi001.stdout:osd.5: {} 2024-09-13T11:23:03.657 INFO:teuthology.orchestra.run.smithi001.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T11:23:03.674 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_historic_ops 2024-09-13T11:23:03.686 INFO:teuthology.orchestra.run.smithi001.stdout:osd.6: {} 2024-09-13T11:23:03.686 INFO:teuthology.orchestra.run.smithi001.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T11:23:03.710 INFO:teuthology.orchestra.run.smithi001.stdout:osd.7: {} 2024-09-13T11:23:03.710 INFO:teuthology.orchestra.run.smithi001.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T11:23:03.737 INFO:teuthology.orchestra.run.smithi001.stdout:osd.8: {} 2024-09-13T11:23:03.737 INFO:teuthology.orchestra.run.smithi001.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-13T11:23:03.748 INFO:tasks.ceph.osd.9.smithi092.stderr:2024-09-13T11:23:03.746+0000 7ff5ab3d0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26348) UID: 0 2024-09-13T11:23:03.763 INFO:teuthology.orchestra.run.smithi001.stdout:osd.9: {} 2024-09-13T11:23:03.763 INFO:teuthology.orchestra.run.smithi001.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-13T11:23:03.793 INFO:teuthology.orchestra.run.smithi001.stdout:osd.10: {} 2024-09-13T11:23:03.793 INFO:teuthology.orchestra.run.smithi001.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-13T11:23:03.820 INFO:teuthology.orchestra.run.smithi001.stdout:osd.11: {} 2024-09-13T11:23:03.820 INFO:teuthology.orchestra.run.smithi001.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-13T11:23:03.841 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-09-13T11:23:03.841 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:23:03.846 INFO:teuthology.orchestra.run.smithi001.stdout:osd.12: {} 2024-09-13T11:23:03.846 INFO:teuthology.orchestra.run.smithi001.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-13T11:23:03.849 INFO:tasks.ceph.osd.7.smithi177.stderr:2024-09-13T11:23:03.847+0000 7f897b51d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26575) UID: 0 2024-09-13T11:23:03.863 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-13T11:23:03.870 INFO:teuthology.orchestra.run.smithi001.stdout:osd.13: {} 2024-09-13T11:23:03.870 INFO:teuthology.orchestra.run.smithi001.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-13T11:23:03.897 INFO:teuthology.orchestra.run.smithi001.stdout:osd.14: {} 2024-09-13T11:23:03.897 INFO:teuthology.orchestra.run.smithi001.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-13T11:23:03.920 INFO:teuthology.orchestra.run.smithi001.stdout:osd.15: {} 2024-09-13T11:23:03.920 INFO:teuthology.orchestra.run.smithi001.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-13T11:23:03.951 INFO:tasks.ceph.osd.2.smithi162.stderr:2024-09-13T11:23:03.947+0000 7fe92b35a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26313) UID: 0 2024-09-13T11:23:04.000 INFO:tasks.radosbench.radosbench.0.smithi177.stdout: 37 16 24946 24930 42.1064 118.938 0.002013 0.0237394 2024-09-13T11:23:04.003 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-09-13T11:23:04.003 INFO:teuthology.orchestra.run.smithi092.stdout: "ops": [ 2024-09-13T11:23:04.003 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:04.003 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:24889 3.4 3:2f091b55:::benchmark_data_smithi177_31116_object24888:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e55)", 2024-09-13T11:23:04.003 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:03.967225+0000", 2024-09-13T11:23:04.003 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 0.036854046000000001, 2024-09-13T11:23:04.003 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.036891538000000001, 2024-09-13T11:23:04.003 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:23:04.003 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:23:04.003 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "waiting for sub ops", 2024-09-13T11:23:04.004 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:23:04.004 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:23:04.004 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:04.004 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 24889 2024-09-13T11:23:04.004 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:04.004 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:23:04.004 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:04.004 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:23:04.004 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:03.967225+0000", 2024-09-13T11:23:04.004 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:04.004 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:04.004 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:04.004 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:23:04.004 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:03.967225+0000", 2024-09-13T11:23:04.005 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:04.005 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:04.005 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:04.005 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:23:04.005 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:03.967227+0000", 2024-09-13T11:23:04.005 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.917e-06 2024-09-13T11:23:04.005 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:04.005 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:04.005 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:23:04.005 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:03.967292+0000", 2024-09-13T11:23:04.005 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.5242999999999999e-05 2024-09-13T11:23:04.005 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:04.005 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:04.005 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:23:04.005 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:03.967304+0000", 2024-09-13T11:23:04.006 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.182e-05 2024-09-13T11:23:04.006 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:04.006 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:04.006 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:23:04.006 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:03.967311+0000", 2024-09-13T11:23:04.006 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.7429999999999996e-06 2024-09-13T11:23:04.006 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:04.006 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:04.006 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:23:04.006 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:03.967366+0000", 2024-09-13T11:23:04.006 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.5186000000000003e-05 2024-09-13T11:23:04.006 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:04.006 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:04.006 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:23:04.007 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:03.967507+0000", 2024-09-13T11:23:04.007 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00014144299999999999 2024-09-13T11:23:04.007 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:04.007 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:04.007 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 12", 2024-09-13T11:23:04.007 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:03.967656+0000", 2024-09-13T11:23:04.007 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000149152 2024-09-13T11:23:04.007 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:04.007 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:04.007 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:23:04.007 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:03.969107+0000", 2024-09-13T11:23:04.007 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0014509060000000001 2024-09-13T11:23:04.007 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:04.008 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:23:04.008 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:04.008 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:04.008 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:04.008 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop(client.4665.0:24943 3.8 e55/40 3:107ac2f9:::benchmark_data_smithi177_31116_object24942:head v 55'1543, mlcod=55'1540)", 2024-09-13T11:23:04.008 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:04.003365+0000", 2024-09-13T11:23:04.008 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 0.00071433599999999999, 2024-09-13T11:23:04.008 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00086565899999999996, 2024-09-13T11:23:04.008 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:23:04.008 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:23:04.008 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-09-13T11:23:04.008 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:23:04.008 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:04.008 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:23:04.009 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.003365+0000", 2024-09-13T11:23:04.009 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:04.009 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:04.009 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:04.009 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:23:04.009 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.003365+0000", 2024-09-13T11:23:04.009 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:04.009 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:04.009 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:04.009 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:23:04.009 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.003366+0000", 2024-09-13T11:23:04.009 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.088e-06 2024-09-13T11:23:04.009 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:04.009 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:04.010 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:23:04.010 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.003444+0000", 2024-09-13T11:23:04.010 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.8510000000000006e-05 2024-09-13T11:23:04.010 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:04.010 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:04.010 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:23:04.010 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.003470+0000", 2024-09-13T11:23:04.010 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.5765000000000001e-05 2024-09-13T11:23:04.010 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:04.010 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:04.010 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:23:04.010 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.003479+0000", 2024-09-13T11:23:04.010 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.3070000000000008e-06 2024-09-13T11:23:04.010 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:04.011 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:04.011 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:23:04.011 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.003525+0000", 2024-09-13T11:23:04.011 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.5631999999999999e-05 2024-09-13T11:23:04.011 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:04.011 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:04.011 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:23:04.011 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.003545+0000", 2024-09-13T11:23:04.011 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.9936e-05 2024-09-13T11:23:04.011 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:04.011 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:23:04.011 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:04.012 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:04.012 INFO:teuthology.orchestra.run.smithi092.stdout: ], 2024-09-13T11:23:04.012 INFO:teuthology.orchestra.run.smithi092.stdout: "num_ops": 2 2024-09-13T11:23:04.012 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-09-13T11:23:04.018 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-13T11:23:04.053 INFO:tasks.ceph.osd.12.smithi001.stderr:2024-09-13T11:23:04.048+0000 7f892c6e6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26314) UID: 0 2024-09-13T11:23:04.154 INFO:tasks.ceph.osd.15.smithi177.stderr:2024-09-13T11:23:04.151+0000 7f9d6ce77640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26578) UID: 0 2024-09-13T11:23:04.159 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-09-13T11:23:04.159 INFO:teuthology.orchestra.run.smithi092.stdout: "ops": [], 2024-09-13T11:23:04.159 INFO:teuthology.orchestra.run.smithi092.stdout: "complaint_time": 30, 2024-09-13T11:23:04.159 INFO:teuthology.orchestra.run.smithi092.stdout: "num_blocked_ops": 0 2024-09-13T11:23:04.159 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-09-13T11:23:04.176 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-13T11:23:04.254 INFO:tasks.ceph.osd.1.smithi092.stderr:2024-09-13T11:23:04.250+0000 7fe7789c5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33718) UID: 0 2024-09-13T11:23:04.313 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-09-13T11:23:04.313 INFO:teuthology.orchestra.run.smithi092.stdout: "size": 20, 2024-09-13T11:23:04.313 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 600, 2024-09-13T11:23:04.313 INFO:teuthology.orchestra.run.smithi092.stdout: "ops": [ 2024-09-13T11:23:04.313 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:04.313 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:3 3.4 3.125e09e4 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e40)", 2024-09-13T11:23:04.313 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:46.304641+0000", 2024-09-13T11:23:04.313 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 18.007681993999999, 2024-09-13T11:23:04.313 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.374372119, 2024-09-13T11:23:04.313 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:23:04.314 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:23:04.314 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "reached pg", 2024-09-13T11:23:04.314 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:23:04.314 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:23:04.314 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:04.314 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 3 2024-09-13T11:23:04.314 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:04.314 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:23:04.314 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:04.314 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:23:04.314 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.304641+0000", 2024-09-13T11:23:04.314 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:04.314 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:04.315 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:04.315 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:23:04.315 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.304641+0000", 2024-09-13T11:23:04.315 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:04.315 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:04.315 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:04.315 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:23:04.315 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.304644+0000", 2024-09-13T11:23:04.315 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.5130000000000001e-06 2024-09-13T11:23:04.315 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:04.315 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:04.315 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:23:04.315 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305058+0000", 2024-09-13T11:23:04.316 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000414277 2024-09-13T11:23:04.316 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:04.316 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:04.316 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:23:04.316 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305124+0000", 2024-09-13T11:23:04.316 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.5813000000000002e-05 2024-09-13T11:23:04.316 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:04.316 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:04.316 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:23:04.316 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305156+0000", 2024-09-13T11:23:04.316 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.1489000000000003e-05 2024-09-13T11:23:04.316 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:04.316 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:04.316 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:23:04.317 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.678939+0000", 2024-09-13T11:23:04.317 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.37378339999999999 2024-09-13T11:23:04.317 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:04.317 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:04.317 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:23:04.317 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.679013+0000", 2024-09-13T11:23:04.317 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.3627000000000005e-05 2024-09-13T11:23:04.317 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:04.317 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:23:04.317 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:04.317 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:04.317 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:04.317 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:29 3.4 3.b7bbfbe4 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-09-13T11:23:04.318 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:46.305215+0000", 2024-09-13T11:23:04.318 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 18.007107691000002, 2024-09-13T11:23:04.318 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.37387964600000001, 2024-09-13T11:23:04.318 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:23:04.318 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:23:04.318 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "reached pg", 2024-09-13T11:23:04.318 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:23:04.318 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:23:04.318 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:04.318 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 29 2024-09-13T11:23:04.318 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:04.318 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:23:04.318 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:04.318 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:23:04.319 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305215+0000", 2024-09-13T11:23:04.319 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:04.319 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:04.319 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:04.319 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:23:04.319 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305215+0000", 2024-09-13T11:23:04.319 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:04.319 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:04.319 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:04.319 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:23:04.319 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305218+0000", 2024-09-13T11:23:04.319 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.7030000000000002e-06 2024-09-13T11:23:04.319 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:04.320 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:04.320 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:23:04.320 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305366+0000", 2024-09-13T11:23:04.320 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00014825400000000001 2024-09-13T11:23:04.320 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:04.320 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:04.320 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:23:04.320 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305386+0000", 2024-09-13T11:23:04.320 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.9689000000000001e-05 2024-09-13T11:23:04.320 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:04.320 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:04.320 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:23:04.320 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305403+0000", 2024-09-13T11:23:04.320 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.7600000000000001e-05 2024-09-13T11:23:04.321 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:04.321 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:04.321 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:23:04.321 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.679067+0000", 2024-09-13T11:23:04.321 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.37366339100000001 2024-09-13T11:23:04.321 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:04.321 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:04.321 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:23:04.321 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.679095+0000", 2024-09-13T11:23:04.321 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.8008999999999999e-05 2024-09-13T11:23:04.321 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:04.321 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:23:04.321 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:04.322 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:04.322 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:04.322 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:30 3.4 3.547ec0f4 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-09-13T11:23:04.322 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:46.305453+0000", 2024-09-13T11:23:04.322 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 18.006869566999999, 2024-09-13T11:23:04.322 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.373764914, 2024-09-13T11:23:04.322 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:23:04.322 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:23:04.322 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "reached pg", 2024-09-13T11:23:04.322 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:23:04.322 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:23:04.322 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:04.322 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 30 2024-09-13T11:23:04.323 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:04.323 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:23:04.323 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:04.323 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:23:04.323 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305453+0000", 2024-09-13T11:23:04.323 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:04.323 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:04.323 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:04.323 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:23:04.323 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305453+0000", 2024-09-13T11:23:04.323 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:04.323 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:04.323 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:04.324 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:23:04.324 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305456+0000", 2024-09-13T11:23:04.324 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.9560000000000002e-06 2024-09-13T11:23:04.324 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:04.324 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:04.324 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:23:04.324 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305597+0000", 2024-09-13T11:23:04.324 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00014122700000000001 2024-09-13T11:23:04.324 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:04.324 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:04.324 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:23:04.324 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305615+0000", 2024-09-13T11:23:04.324 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.8139999999999999e-05 2024-09-13T11:23:04.325 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:04.325 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:04.325 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:23:04.325 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305632+0000", 2024-09-13T11:23:04.325 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.6101999999999999e-05 2024-09-13T11:23:04.325 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:04.325 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:04.325 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:23:04.325 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.679195+0000", 2024-09-13T11:23:04.325 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.373563486 2024-09-13T11:23:04.325 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:04.325 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:04.325 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:23:04.325 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.679218+0000", 2024-09-13T11:23:04.326 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.3003e-05 2024-09-13T11:23:04.326 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:04.326 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:23:04.326 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:04.326 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:04.326 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:04.326 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:70 3.4 3.1404a604 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-09-13T11:23:04.326 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:46.305665+0000", 2024-09-13T11:23:04.326 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 18.006657419, 2024-09-13T11:23:04.326 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.37365550199999997, 2024-09-13T11:23:04.326 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:23:04.326 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:23:04.326 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "reached pg", 2024-09-13T11:23:04.327 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:23:04.327 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:23:04.327 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:04.327 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 70 2024-09-13T11:23:04.327 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:04.327 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:23:04.327 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:04.327 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:23:04.327 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305665+0000", 2024-09-13T11:23:04.327 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:04.327 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:04.327 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:04.327 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:23:04.328 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305665+0000", 2024-09-13T11:23:04.328 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:04.328 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:04.328 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:04.328 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:23:04.328 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305668+0000", 2024-09-13T11:23:04.328 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.5890000000000001e-06 2024-09-13T11:23:04.328 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:04.328 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:04.328 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:23:04.329 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305791+0000", 2024-09-13T11:23:04.329 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000123545 2024-09-13T11:23:04.329 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:04.329 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:04.329 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:23:04.329 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305808+0000", 2024-09-13T11:23:04.329 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.668e-05 2024-09-13T11:23:04.329 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:04.329 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:04.329 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:23:04.329 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305824+0000", 2024-09-13T11:23:04.329 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.6385e-05 2024-09-13T11:23:04.330 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:04.330 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:04.330 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:23:04.330 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.679305+0000", 2024-09-13T11:23:04.330 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.37348059900000002 2024-09-13T11:23:04.330 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:04.330 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:04.330 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:23:04.330 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.679321+0000", 2024-09-13T11:23:04.330 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.5704000000000001e-05 2024-09-13T11:23:04.330 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:04.330 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:23:04.330 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:04.331 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:04.331 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:04.331 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop(client.4665.0:860 3.b e43/40 3:d95a5791:::benchmark_data_smithi177_31116_object859:head v 43'56, mlcod=43'53)", 2024-09-13T11:23:04.331 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:49.485537+0000", 2024-09-13T11:23:04.331 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 14.826785299999999, 2024-09-13T11:23:04.331 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.039386523, 2024-09-13T11:23:04.331 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:23:04.331 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:23:04.331 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:04.331 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:23:04.331 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:04.331 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:23:04.331 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.485537+0000", 2024-09-13T11:23:04.332 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:04.332 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:04.332 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:04.332 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:23:04.332 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.485537+0000", 2024-09-13T11:23:04.332 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:04.332 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:04.332 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:04.332 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:23:04.332 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.485539+0000", 2024-09-13T11:23:04.332 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.9989999999999998e-06 2024-09-13T11:23:04.332 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:04.332 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:04.333 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:23:04.333 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.485563+0000", 2024-09-13T11:23:04.333 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.3662000000000001e-05 2024-09-13T11:23:04.333 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:04.333 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:04.333 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:23:04.333 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.524281+0000", 2024-09-13T11:23:04.333 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.038717768999999999 2024-09-13T11:23:04.333 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:04.333 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:04.333 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:23:04.333 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.524291+0000", 2024-09-13T11:23:04.333 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.9129999999999998e-06 2024-09-13T11:23:04.333 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:04.334 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:04.334 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:23:04.334 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.524356+0000", 2024-09-13T11:23:04.334 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.5018999999999995e-05 2024-09-13T11:23:04.334 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:04.334 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:04.334 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:23:04.334 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.524371+0000", 2024-09-13T11:23:04.334 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.5328e-05 2024-09-13T11:23:04.335 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:04.335 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:04.335 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:23:04.335 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.524867+0000", 2024-09-13T11:23:04.335 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00049635900000000004 2024-09-13T11:23:04.335 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:04.335 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:04.335 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:23:04.335 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.524924+0000", 2024-09-13T11:23:04.335 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.6474000000000002e-05 2024-09-13T11:23:04.335 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:04.335 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:23:04.336 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:04.336 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:04.336 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:04.336 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop(client.4665.0:955 3.b e43/40 3:d0a0b8ee:::benchmark_data_smithi177_31116_object954:head v 43'60, mlcod=43'57)", 2024-09-13T11:23:04.336 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:49.570002+0000", 2024-09-13T11:23:04.336 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 14.742321135999999, 2024-09-13T11:23:04.336 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.043591372000000003, 2024-09-13T11:23:04.336 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:23:04.336 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:23:04.336 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:04.336 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:23:04.336 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:04.336 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:23:04.337 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.570002+0000", 2024-09-13T11:23:04.337 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:04.337 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:04.337 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:04.337 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:23:04.337 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.570002+0000", 2024-09-13T11:23:04.337 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:04.337 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:04.337 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:04.337 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:23:04.337 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.570003+0000", 2024-09-13T11:23:04.337 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.2839999999999999e-06 2024-09-13T11:23:04.337 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:04.337 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:04.338 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:23:04.338 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.570023+0000", 2024-09-13T11:23:04.338 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.0596000000000001e-05 2024-09-13T11:23:04.338 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:04.338 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:04.338 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:23:04.338 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.612857+0000", 2024-09-13T11:23:04.338 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.042833610000000001 2024-09-13T11:23:04.338 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:04.338 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:04.338 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:23:04.338 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.612869+0000", 2024-09-13T11:23:04.338 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.1608e-05 2024-09-13T11:23:04.338 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:04.339 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:04.339 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:23:04.339 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.612928+0000", 2024-09-13T11:23:04.339 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.8924e-05 2024-09-13T11:23:04.339 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:04.339 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:04.339 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:23:04.339 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.612945+0000", 2024-09-13T11:23:04.339 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.7628000000000001e-05 2024-09-13T11:23:04.339 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:04.339 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:04.340 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:23:04.340 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.613554+0000", 2024-09-13T11:23:04.340 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00060919200000000005 2024-09-13T11:23:04.340 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:04.340 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:04.340 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:23:04.340 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.613593+0000", 2024-09-13T11:23:04.340 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.8529999999999999e-05 2024-09-13T11:23:04.340 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:04.340 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:23:04.340 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:04.340 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:04.341 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:04.341 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:8929 3.4 3:2df06198:::benchmark_data_smithi177_31116_object8928:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-09-13T11:23:04.341 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:53.626810+0000", 2024-09-13T11:23:04.341 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 10.68551297, 2024-09-13T11:23:04.341 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.042891885999999997, 2024-09-13T11:23:04.341 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:23:04.341 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:23:04.341 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:04.341 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:23:04.341 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:23:04.341 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:04.341 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 8929 2024-09-13T11:23:04.341 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:04.342 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:23:04.342 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:04.342 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:23:04.342 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:53.626810+0000", 2024-09-13T11:23:04.342 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:04.342 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:04.342 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:04.342 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:23:04.342 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:53.626810+0000", 2024-09-13T11:23:04.342 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:04.342 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:04.342 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:04.342 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:23:04.343 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:53.626812+0000", 2024-09-13T11:23:04.343 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.5569999999999998e-06 2024-09-13T11:23:04.343 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:04.343 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:04.343 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:23:04.343 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:53.626853+0000", 2024-09-13T11:23:04.343 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.0383000000000003e-05 2024-09-13T11:23:04.343 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:04.343 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:04.343 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:23:04.343 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:53.626871+0000", 2024-09-13T11:23:04.343 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.8590999999999999e-05 2024-09-13T11:23:04.343 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:04.344 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:04.344 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:23:04.344 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:53.626881+0000", 2024-09-13T11:23:04.344 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.5780000000000005e-06 2024-09-13T11:23:04.344 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:04.344 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:04.344 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:23:04.344 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:53.626985+0000", 2024-09-13T11:23:04.344 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000104392 2024-09-13T11:23:04.344 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:04.344 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:04.344 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:23:04.344 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:53.627227+0000", 2024-09-13T11:23:04.345 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000241831 2024-09-13T11:23:04.345 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:04.345 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:04.345 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 12", 2024-09-13T11:23:04.345 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:53.627479+0000", 2024-09-13T11:23:04.345 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000251931 2024-09-13T11:23:04.345 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:04.345 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:04.345 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:23:04.345 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:53.629352+0000", 2024-09-13T11:23:04.345 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0018730050000000001 2024-09-13T11:23:04.345 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:04.345 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:04.345 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:04.346 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:53.669552+0000", 2024-09-13T11:23:04.346 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.040200333999999997 2024-09-13T11:23:04.346 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:04.346 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:04.346 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:23:04.346 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:53.669633+0000", 2024-09-13T11:23:04.346 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.1037000000000004e-05 2024-09-13T11:23:04.346 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:04.346 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:04.346 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:23:04.346 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:53.669702+0000", 2024-09-13T11:23:04.346 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.8246999999999999e-05 2024-09-13T11:23:04.346 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:04.347 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:23:04.347 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:04.347 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:04.347 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:04.347 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:8943 3.4 3:2e96fe7b:::benchmark_data_smithi177_31116_object8942:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-09-13T11:23:04.347 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:53.640283+0000", 2024-09-13T11:23:04.347 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 10.672039267000001, 2024-09-13T11:23:04.347 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.071508547000000006, 2024-09-13T11:23:04.347 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:23:04.347 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:23:04.347 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:04.347 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:23:04.347 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:23:04.348 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:04.348 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 8943 2024-09-13T11:23:04.348 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:04.348 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:23:04.348 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:04.348 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:23:04.348 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:53.640283+0000", 2024-09-13T11:23:04.348 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:04.348 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:04.348 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:04.348 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:23:04.349 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:53.640283+0000", 2024-09-13T11:23:04.349 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:04.349 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:04.349 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:04.349 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:23:04.349 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:53.640286+0000", 2024-09-13T11:23:04.349 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.5519999999999999e-06 2024-09-13T11:23:04.349 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:04.349 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:04.349 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:23:04.350 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:53.640336+0000", 2024-09-13T11:23:04.350 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.9676999999999999e-05 2024-09-13T11:23:04.350 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:04.350 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:04.350 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:23:04.350 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:53.640354+0000", 2024-09-13T11:23:04.350 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.8082999999999999e-05 2024-09-13T11:23:04.350 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:04.350 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:04.350 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:23:04.350 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:53.640362+0000", 2024-09-13T11:23:04.350 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.5739999999999996e-06 2024-09-13T11:23:04.350 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:04.351 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:04.351 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:23:04.351 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:53.640442+0000", 2024-09-13T11:23:04.351 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.9500999999999997e-05 2024-09-13T11:23:04.351 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:04.351 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:04.351 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:23:04.351 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:53.640593+0000", 2024-09-13T11:23:04.351 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000151123 2024-09-13T11:23:04.351 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:04.351 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:04.351 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 12", 2024-09-13T11:23:04.351 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:53.640717+0000", 2024-09-13T11:23:04.351 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00012440199999999999 2024-09-13T11:23:04.352 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:04.352 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:04.352 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:23:04.352 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:53.641838+0000", 2024-09-13T11:23:04.352 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.001120675 2024-09-13T11:23:04.352 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:04.352 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:04.352 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:04.352 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:53.711666+0000", 2024-09-13T11:23:04.352 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.069827881999999994 2024-09-13T11:23:04.352 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:04.352 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:04.352 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:23:04.353 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:53.711737+0000", 2024-09-13T11:23:04.353 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.1274999999999998e-05 2024-09-13T11:23:04.353 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:04.353 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:04.353 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:23:04.353 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:53.711792+0000", 2024-09-13T11:23:04.353 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.4802999999999997e-05 2024-09-13T11:23:04.353 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:04.353 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:23:04.353 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:04.353 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:04.353 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:04.354 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:8968 3.4 3:244f32fb:::benchmark_data_smithi177_31116_object8967:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-09-13T11:23:04.354 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:53.666630+0000", 2024-09-13T11:23:04.354 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 10.645692851, 2024-09-13T11:23:04.354 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.055866648999999997, 2024-09-13T11:23:04.354 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:23:04.354 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:23:04.354 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:04.354 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:23:04.354 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:23:04.354 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:04.354 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 8968 2024-09-13T11:23:04.354 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:04.354 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:23:04.355 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:04.355 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:23:04.355 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:53.666630+0000", 2024-09-13T11:23:04.355 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:04.355 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:04.355 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:04.355 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:23:04.355 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:53.666630+0000", 2024-09-13T11:23:04.355 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:04.355 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:04.355 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:04.355 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:23:04.355 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:53.666632+0000", 2024-09-13T11:23:04.356 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.4490000000000002e-06 2024-09-13T11:23:04.356 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:04.356 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:04.356 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:23:04.356 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:53.666677+0000", 2024-09-13T11:23:04.356 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.4836999999999998e-05 2024-09-13T11:23:04.356 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:04.356 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:04.356 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:23:04.356 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:53.666692+0000", 2024-09-13T11:23:04.356 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.4496e-05 2024-09-13T11:23:04.356 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:04.356 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:04.357 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:23:04.357 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:53.666698+0000", 2024-09-13T11:23:04.357 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.4509999999999997e-06 2024-09-13T11:23:04.357 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:04.357 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:04.357 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:23:04.357 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:53.666797+0000", 2024-09-13T11:23:04.357 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.8743e-05 2024-09-13T11:23:04.357 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:04.357 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:04.357 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:23:04.357 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:53.667028+0000", 2024-09-13T11:23:04.357 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00023124300000000001 2024-09-13T11:23:04.357 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:04.358 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:04.358 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 12", 2024-09-13T11:23:04.358 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:53.667245+0000", 2024-09-13T11:23:04.358 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000216973 2024-09-13T11:23:04.358 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:04.358 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:04.358 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:23:04.358 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:53.671114+0000", 2024-09-13T11:23:04.358 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00386908 2024-09-13T11:23:04.358 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:04.358 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:04.358 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:04.359 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:53.722251+0000", 2024-09-13T11:23:04.359 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.051137254 2024-09-13T11:23:04.359 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:04.359 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:04.359 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:23:04.359 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:53.722389+0000", 2024-09-13T11:23:04.359 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00013790100000000001 2024-09-13T11:23:04.359 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:04.359 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:04.359 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:23:04.359 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:53.722496+0000", 2024-09-13T11:23:04.359 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000107222 2024-09-13T11:23:04.359 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:04.360 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:23:04.360 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:04.360 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:04.360 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:04.360 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop_reply(client.4665.0:8943 3.4 e47/40 ondisk, result = 0)", 2024-09-13T11:23:04.360 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:53.669396+0000", 2024-09-13T11:23:04.360 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 10.642926621999999, 2024-09-13T11:23:04.360 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.042405259000000001, 2024-09-13T11:23:04.360 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:23:04.360 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:23:04.360 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-09-13T11:23:04.360 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:23:04.360 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:04.361 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:23:04.361 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:53.669396+0000", 2024-09-13T11:23:04.361 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:04.361 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:04.361 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:04.361 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:23:04.361 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:53.669396+0000", 2024-09-13T11:23:04.361 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:04.361 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:04.361 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:04.361 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:23:04.361 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:53.669397+0000", 2024-09-13T11:23:04.362 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.263e-06 2024-09-13T11:23:04.362 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:04.362 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:04.362 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:23:04.362 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:53.669402+0000", 2024-09-13T11:23:04.362 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.1969999999999998e-06 2024-09-13T11:23:04.362 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:04.362 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:04.362 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:23:04.362 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:53.711534+0000", 2024-09-13T11:23:04.362 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.042132256 2024-09-13T11:23:04.362 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:04.362 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:04.363 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:23:04.363 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:53.711545+0000", 2024-09-13T11:23:04.363 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0757000000000001e-05 2024-09-13T11:23:04.363 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:04.363 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:04.363 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:23:04.363 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:53.711648+0000", 2024-09-13T11:23:04.363 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000103591 2024-09-13T11:23:04.363 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:04.363 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:04.363 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:23:04.363 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:53.711659+0000", 2024-09-13T11:23:04.363 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0659000000000001e-05 2024-09-13T11:23:04.363 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:04.364 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:04.364 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:23:04.364 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:53.711801+0000", 2024-09-13T11:23:04.364 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00014253599999999999 2024-09-13T11:23:04.364 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:04.364 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:23:04.364 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:04.364 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:04.364 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:04.364 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:9301 3.4 3:2c2635c1:::benchmark_data_smithi177_31116_object9300:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-09-13T11:23:04.364 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:53.910351+0000", 2024-09-13T11:23:04.364 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 10.401971541, 2024-09-13T11:23:04.365 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.061138634999999997, 2024-09-13T11:23:04.365 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:23:04.365 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:23:04.365 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:04.365 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:23:04.365 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:23:04.365 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:04.365 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 9301 2024-09-13T11:23:04.365 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:04.365 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:23:04.365 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:04.365 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:23:04.365 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:53.910351+0000", 2024-09-13T11:23:04.365 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:04.366 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:04.366 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:04.366 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:23:04.366 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:53.910351+0000", 2024-09-13T11:23:04.366 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:04.366 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:04.366 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:04.366 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:23:04.366 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:53.910353+0000", 2024-09-13T11:23:04.366 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.8139999999999999e-06 2024-09-13T11:23:04.366 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:04.366 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:04.366 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:23:04.367 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:53.910417+0000", 2024-09-13T11:23:04.367 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.4354000000000001e-05 2024-09-13T11:23:04.367 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:04.367 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:04.367 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:23:04.367 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:53.910441+0000", 2024-09-13T11:23:04.367 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.3306e-05 2024-09-13T11:23:04.367 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:04.367 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:04.367 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:23:04.367 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:53.910450+0000", 2024-09-13T11:23:04.367 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.6849999999999996e-06 2024-09-13T11:23:04.368 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:04.368 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:04.368 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:23:04.368 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:53.910580+0000", 2024-09-13T11:23:04.368 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00012942699999999999 2024-09-13T11:23:04.368 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:04.368 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:04.368 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:23:04.368 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:53.910808+0000", 2024-09-13T11:23:04.368 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00022840100000000001 2024-09-13T11:23:04.368 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:04.368 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:04.368 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 12", 2024-09-13T11:23:04.368 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:53.911027+0000", 2024-09-13T11:23:04.369 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000218421 2024-09-13T11:23:04.369 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:04.369 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:04.369 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:23:04.369 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:53.913684+0000", 2024-09-13T11:23:04.369 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0026571780000000001 2024-09-13T11:23:04.369 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:04.369 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:04.369 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:04.369 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:53.971316+0000", 2024-09-13T11:23:04.369 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.057631911000000001 2024-09-13T11:23:04.369 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:04.369 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:04.370 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:23:04.370 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:53.971414+0000", 2024-09-13T11:23:04.370 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.8793000000000001e-05 2024-09-13T11:23:04.370 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:04.370 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:04.370 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:23:04.370 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:53.971490+0000", 2024-09-13T11:23:04.370 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.5345000000000005e-05 2024-09-13T11:23:04.370 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:04.370 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:23:04.370 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:04.370 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:04.370 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:04.371 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop_reply(client.4665.0:9301 3.4 e47/40 ondisk, result = 0)", 2024-09-13T11:23:04.371 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:53.913196+0000", 2024-09-13T11:23:04.371 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 10.399126597, 2024-09-13T11:23:04.371 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.058304652999999998, 2024-09-13T11:23:04.371 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:23:04.371 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:23:04.371 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-09-13T11:23:04.371 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:23:04.371 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:04.371 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:23:04.371 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:53.913196+0000", 2024-09-13T11:23:04.371 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:04.371 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:04.372 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:04.372 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:23:04.372 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:53.913196+0000", 2024-09-13T11:23:04.372 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:04.372 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:04.372 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:04.372 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:23:04.372 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:53.913199+0000", 2024-09-13T11:23:04.372 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.5799999999999999e-06 2024-09-13T11:23:04.372 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:04.372 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:04.372 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:23:04.372 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:53.913210+0000", 2024-09-13T11:23:04.373 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.083e-05 2024-09-13T11:23:04.373 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:04.373 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:04.373 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:23:04.373 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:53.971150+0000", 2024-09-13T11:23:04.373 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.057940525 2024-09-13T11:23:04.373 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:04.373 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:04.373 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:23:04.373 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:53.971161+0000", 2024-09-13T11:23:04.373 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0774999999999999e-05 2024-09-13T11:23:04.373 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:04.374 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:04.374 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:23:04.374 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:53.971292+0000", 2024-09-13T11:23:04.374 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00013122000000000001 2024-09-13T11:23:04.374 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:04.374 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:04.374 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:23:04.374 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:53.971305+0000", 2024-09-13T11:23:04.374 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.3115000000000001e-05 2024-09-13T11:23:04.374 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:04.374 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:04.374 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:23:04.374 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:53.971501+0000", 2024-09-13T11:23:04.374 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000195608 2024-09-13T11:23:04.375 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:04.375 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:23:04.375 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:04.375 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:04.375 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:04.375 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop(client.4665.0:9761 3.8 e47/40 3:1c01147e:::benchmark_data_smithi177_31116_object9760:head v 47'635, mlcod=47'632)", 2024-09-13T11:23:04.375 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:54.200926+0000", 2024-09-13T11:23:04.375 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 10.111396464, 2024-09-13T11:23:04.375 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.047231686000000002, 2024-09-13T11:23:04.375 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:23:04.375 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:23:04.375 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:04.375 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:23:04.376 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:04.376 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:23:04.376 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:54.200926+0000", 2024-09-13T11:23:04.376 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:04.376 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:04.376 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:04.376 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:23:04.376 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:54.200926+0000", 2024-09-13T11:23:04.376 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:04.376 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:04.376 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:04.376 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:23:04.376 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:54.200928+0000", 2024-09-13T11:23:04.377 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.483e-06 2024-09-13T11:23:04.377 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:04.377 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:04.377 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:23:04.377 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:54.200995+0000", 2024-09-13T11:23:04.377 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.7651999999999995e-05 2024-09-13T11:23:04.377 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:04.377 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:04.377 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:23:04.377 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:54.247048+0000", 2024-09-13T11:23:04.377 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.046052203999999999 2024-09-13T11:23:04.377 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:04.377 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:04.378 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:23:04.378 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:54.247057+0000", 2024-09-13T11:23:04.378 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.4029999999999996e-06 2024-09-13T11:23:04.378 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:04.378 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:04.378 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:23:04.378 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:54.247129+0000", 2024-09-13T11:23:04.378 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.1903999999999995e-05 2024-09-13T11:23:04.378 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:04.378 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:04.378 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:23:04.378 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:54.247146+0000", 2024-09-13T11:23:04.378 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.7592999999999999e-05 2024-09-13T11:23:04.378 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:04.379 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:04.379 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:23:04.379 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:54.248117+0000", 2024-09-13T11:23:04.379 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00097079399999999998 2024-09-13T11:23:04.379 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:04.379 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:04.379 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:23:04.379 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:54.248158+0000", 2024-09-13T11:23:04.379 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.0652999999999999e-05 2024-09-13T11:23:04.379 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:04.379 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:23:04.379 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:04.379 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:04.380 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:04.380 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop(client.4665.0:9859 3.8 e47/40 3:13ff5619:::benchmark_data_smithi177_31116_object9858:head v 47'640, mlcod=47'637)", 2024-09-13T11:23:04.380 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:54.265763+0000", 2024-09-13T11:23:04.380 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 10.046559434000001, 2024-09-13T11:23:04.380 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.041827754000000002, 2024-09-13T11:23:04.380 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:23:04.380 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:23:04.380 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:04.380 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:23:04.380 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:04.380 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:23:04.380 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:54.265763+0000", 2024-09-13T11:23:04.380 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:04.380 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:04.381 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:04.381 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:23:04.381 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:54.265763+0000", 2024-09-13T11:23:04.381 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:04.381 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:04.381 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:04.381 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:23:04.381 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:54.265765+0000", 2024-09-13T11:23:04.381 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.79e-06 2024-09-13T11:23:04.381 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:04.381 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:04.381 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:23:04.381 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:54.265841+0000", 2024-09-13T11:23:04.382 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.5671000000000001e-05 2024-09-13T11:23:04.382 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:04.382 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:04.382 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:23:04.382 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:54.306452+0000", 2024-09-13T11:23:04.382 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.040611119000000001 2024-09-13T11:23:04.382 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:04.382 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:04.382 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:23:04.382 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:54.306461+0000", 2024-09-13T11:23:04.382 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.2399999999999996e-06 2024-09-13T11:23:04.383 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:04.383 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:04.383 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:23:04.383 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:54.306539+0000", 2024-09-13T11:23:04.383 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.8023999999999997e-05 2024-09-13T11:23:04.383 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:04.383 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:04.383 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:23:04.383 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:54.306565+0000", 2024-09-13T11:23:04.384 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.6122000000000002e-05 2024-09-13T11:23:04.384 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:04.384 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:04.384 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:23:04.384 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:54.307504+0000", 2024-09-13T11:23:04.384 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00093926100000000005 2024-09-13T11:23:04.384 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:04.384 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:04.384 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:23:04.384 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:54.307591+0000", 2024-09-13T11:23:04.384 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.6526999999999993e-05 2024-09-13T11:23:04.384 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:04.384 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:23:04.385 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:04.385 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:04.385 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:04.385 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop(client.4665.0:9940 3.8 e47/40 3:1dba252f:::benchmark_data_smithi177_31116_object9939:head v 47'646, mlcod=47'642)", 2024-09-13T11:23:04.385 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:54.339754+0000", 2024-09-13T11:23:04.385 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 9.9725689440000007, 2024-09-13T11:23:04.385 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.037601664999999999, 2024-09-13T11:23:04.385 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:23:04.385 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:23:04.385 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:04.385 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:23:04.385 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:04.385 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:23:04.386 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:54.339754+0000", 2024-09-13T11:23:04.386 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:04.386 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:04.386 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:04.386 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:23:04.386 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:54.339754+0000", 2024-09-13T11:23:04.386 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:04.386 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:04.386 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:04.386 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:23:04.386 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:54.339756+0000", 2024-09-13T11:23:04.386 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.9980000000000002e-06 2024-09-13T11:23:04.386 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:04.386 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:04.387 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:23:04.387 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:54.339807+0000", 2024-09-13T11:23:04.387 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.1449999999999997e-05 2024-09-13T11:23:04.387 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:04.387 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:04.387 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:23:04.387 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:54.374014+0000", 2024-09-13T11:23:04.387 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.034206703999999998 2024-09-13T11:23:04.387 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:04.387 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:04.387 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:23:04.387 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:54.374027+0000", 2024-09-13T11:23:04.387 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.2608e-05 2024-09-13T11:23:04.388 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:04.388 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:04.388 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:23:04.388 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:54.375907+0000", 2024-09-13T11:23:04.388 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.001880872 2024-09-13T11:23:04.388 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:04.388 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:04.388 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:23:04.388 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:54.375933+0000", 2024-09-13T11:23:04.388 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.5420000000000001e-05 2024-09-13T11:23:04.388 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:04.388 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:04.388 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:23:04.389 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:54.377317+0000", 2024-09-13T11:23:04.389 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.001383752 2024-09-13T11:23:04.389 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:04.389 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:04.389 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:23:04.389 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:54.377355+0000", 2024-09-13T11:23:04.389 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.8860999999999999e-05 2024-09-13T11:23:04.389 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:04.389 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:23:04.389 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:04.389 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:04.389 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:04.389 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:17682 3.4 3:255b08e5:::benchmark_data_smithi177_31116_object17681:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-13T11:23:04.390 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:59.548043+0000", 2024-09-13T11:23:04.390 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 4.7642801759999998, 2024-09-13T11:23:04.390 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.042123406000000002, 2024-09-13T11:23:04.390 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:23:04.390 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:23:04.390 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:04.390 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:23:04.390 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:23:04.390 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:04.390 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 17682 2024-09-13T11:23:04.390 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:04.390 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:23:04.390 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:04.391 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:23:04.391 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:59.548043+0000", 2024-09-13T11:23:04.391 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:04.391 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:04.391 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:04.391 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:23:04.391 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:59.548043+0000", 2024-09-13T11:23:04.391 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:04.391 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:04.391 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:04.391 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:23:04.391 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:59.548044+0000", 2024-09-13T11:23:04.391 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.973e-06 2024-09-13T11:23:04.391 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:04.392 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:04.392 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:23:04.392 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:59.548157+0000", 2024-09-13T11:23:04.392 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000112329 2024-09-13T11:23:04.392 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:04.392 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:04.392 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:23:04.392 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:59.548171+0000", 2024-09-13T11:23:04.392 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.3919e-05 2024-09-13T11:23:04.392 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:04.392 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:04.393 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:23:04.393 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:59.548177+0000", 2024-09-13T11:23:04.393 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.6279999999999999e-06 2024-09-13T11:23:04.393 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:04.393 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:04.393 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:23:04.393 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:59.548238+0000", 2024-09-13T11:23:04.393 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.0754999999999997e-05 2024-09-13T11:23:04.393 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:04.393 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:04.393 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:23:04.393 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:59.548366+0000", 2024-09-13T11:23:04.393 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000128057 2024-09-13T11:23:04.394 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:04.394 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:04.394 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 12", 2024-09-13T11:23:04.394 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:59.548468+0000", 2024-09-13T11:23:04.394 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000102249 2024-09-13T11:23:04.394 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:04.394 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:04.394 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:23:04.394 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:59.549882+0000", 2024-09-13T11:23:04.394 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0014131320000000001 2024-09-13T11:23:04.394 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:04.394 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:04.394 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:04.395 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:59.590061+0000", 2024-09-13T11:23:04.395 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.040179399999999997 2024-09-13T11:23:04.395 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:04.395 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:04.395 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:23:04.395 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:59.590117+0000", 2024-09-13T11:23:04.395 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.5726000000000001e-05 2024-09-13T11:23:04.395 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:04.395 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:04.395 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:23:04.395 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:59.590166+0000", 2024-09-13T11:23:04.395 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.9237999999999999e-05 2024-09-13T11:23:04.395 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:04.395 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:23:04.396 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:04.396 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:04.396 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:04.396 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:24889 3.4 3:2f091b55:::benchmark_data_smithi177_31116_object24888:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e55)", 2024-09-13T11:23:04.396 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:03.967225+0000", 2024-09-13T11:23:04.396 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 0.34509774700000001, 2024-09-13T11:23:04.396 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.046161196000000002, 2024-09-13T11:23:04.396 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:23:04.396 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:23:04.396 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:04.396 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:23:04.396 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:23:04.396 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:04.397 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 24889 2024-09-13T11:23:04.397 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:04.397 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:23:04.397 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:04.397 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:23:04.397 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:03.967225+0000", 2024-09-13T11:23:04.397 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:04.397 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:04.397 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:04.397 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:23:04.397 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:03.967225+0000", 2024-09-13T11:23:04.397 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:04.397 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:04.398 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:04.398 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:23:04.398 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:03.967227+0000", 2024-09-13T11:23:04.398 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.917e-06 2024-09-13T11:23:04.398 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:04.398 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:04.398 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:23:04.398 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:03.967292+0000", 2024-09-13T11:23:04.398 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.5242999999999999e-05 2024-09-13T11:23:04.398 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:04.398 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:04.398 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:23:04.398 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:03.967304+0000", 2024-09-13T11:23:04.399 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.182e-05 2024-09-13T11:23:04.399 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:04.399 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:04.399 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:23:04.399 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:03.967311+0000", 2024-09-13T11:23:04.399 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.7429999999999996e-06 2024-09-13T11:23:04.399 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:04.399 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:04.399 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:23:04.399 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:03.967366+0000", 2024-09-13T11:23:04.399 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.5186000000000003e-05 2024-09-13T11:23:04.399 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:04.399 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:04.400 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:23:04.400 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:03.967507+0000", 2024-09-13T11:23:04.400 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00014144299999999999 2024-09-13T11:23:04.400 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:04.400 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:04.400 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 12", 2024-09-13T11:23:04.400 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:03.967656+0000", 2024-09-13T11:23:04.400 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000149152 2024-09-13T11:23:04.400 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:04.400 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:04.400 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:23:04.400 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:03.969107+0000", 2024-09-13T11:23:04.400 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0014509060000000001 2024-09-13T11:23:04.400 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:04.401 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:04.401 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:04.401 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.013242+0000", 2024-09-13T11:23:04.401 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.044134410999999998 2024-09-13T11:23:04.401 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:04.401 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:04.401 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:23:04.401 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.013320+0000", 2024-09-13T11:23:04.401 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.8307999999999996e-05 2024-09-13T11:23:04.401 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:04.401 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:04.401 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:23:04.401 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.013386+0000", 2024-09-13T11:23:04.402 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.6067000000000003e-05 2024-09-13T11:23:04.402 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:04.402 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:23:04.402 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:04.402 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:04.402 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:04.402 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:25128 3.4 3:2a00e065:::benchmark_data_smithi177_31116_object25127:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e55)", 2024-09-13T11:23:04.402 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:04.107380+0000", 2024-09-13T11:23:04.402 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 0.20494225499999999, 2024-09-13T11:23:04.402 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.044004359, 2024-09-13T11:23:04.402 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:23:04.402 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:23:04.402 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:04.402 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:23:04.403 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:23:04.403 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:04.403 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 25128 2024-09-13T11:23:04.403 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:04.403 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:23:04.403 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:04.403 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:23:04.403 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.107380+0000", 2024-09-13T11:23:04.403 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:04.403 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:04.403 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:04.403 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:23:04.404 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.107380+0000", 2024-09-13T11:23:04.404 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:04.404 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:04.404 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:04.404 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:23:04.404 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.107382+0000", 2024-09-13T11:23:04.404 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.381e-06 2024-09-13T11:23:04.404 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:04.404 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:04.404 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:23:04.404 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.107420+0000", 2024-09-13T11:23:04.404 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.8655999999999997e-05 2024-09-13T11:23:04.404 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:04.405 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:04.405 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:23:04.405 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.107431+0000", 2024-09-13T11:23:04.405 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0626e-05 2024-09-13T11:23:04.405 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:04.405 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:04.405 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:23:04.405 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.107438+0000", 2024-09-13T11:23:04.405 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.4579999999999998e-06 2024-09-13T11:23:04.405 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:04.405 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:04.405 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:23:04.405 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.107490+0000", 2024-09-13T11:23:04.405 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.2225000000000002e-05 2024-09-13T11:23:04.406 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:04.406 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:04.406 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:23:04.406 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.107635+0000", 2024-09-13T11:23:04.406 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00014496100000000001 2024-09-13T11:23:04.406 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:04.406 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:04.406 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 12", 2024-09-13T11:23:04.406 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.107777+0000", 2024-09-13T11:23:04.406 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000142292 2024-09-13T11:23:04.406 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:04.406 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:04.406 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:23:04.406 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.108983+0000", 2024-09-13T11:23:04.407 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0012058539999999999 2024-09-13T11:23:04.407 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:04.407 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:04.407 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:04.407 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.151279+0000", 2024-09-13T11:23:04.407 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.042296400999999997 2024-09-13T11:23:04.407 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:04.407 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:04.407 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:23:04.407 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.151335+0000", 2024-09-13T11:23:04.407 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.5327000000000001e-05 2024-09-13T11:23:04.407 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:04.407 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:04.408 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:23:04.408 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.151385+0000", 2024-09-13T11:23:04.408 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.0178e-05 2024-09-13T11:23:04.408 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:04.408 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:23:04.408 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:04.408 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:04.408 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:04.408 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:25187 3.4 3:2d897667:::benchmark_data_smithi177_31116_object25186:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e55)", 2024-09-13T11:23:04.408 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:04.190304+0000", 2024-09-13T11:23:04.408 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 0.122018267, 2024-09-13T11:23:04.408 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.050102254999999998, 2024-09-13T11:23:04.408 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:23:04.408 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:23:04.409 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:04.409 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:23:04.409 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:23:04.409 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:04.409 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 25187 2024-09-13T11:23:04.409 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:04.409 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:23:04.409 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:04.409 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:23:04.409 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.190304+0000", 2024-09-13T11:23:04.409 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:04.409 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:04.409 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:04.410 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:23:04.410 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.190304+0000", 2024-09-13T11:23:04.410 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:04.410 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:04.410 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:04.410 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:23:04.410 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.190307+0000", 2024-09-13T11:23:04.410 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.5000000000000002e-06 2024-09-13T11:23:04.410 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:04.410 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:04.410 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:23:04.410 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.190406+0000", 2024-09-13T11:23:04.410 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.9469000000000006e-05 2024-09-13T11:23:04.411 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:04.411 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:04.411 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:23:04.411 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.190430+0000", 2024-09-13T11:23:04.411 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.3105999999999999e-05 2024-09-13T11:23:04.411 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:04.411 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:04.411 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:23:04.411 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.190441+0000", 2024-09-13T11:23:04.411 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.1732999999999999e-05 2024-09-13T11:23:04.411 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:04.411 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:04.411 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:23:04.412 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.190523+0000", 2024-09-13T11:23:04.412 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.1909999999999993e-05 2024-09-13T11:23:04.412 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:04.412 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:04.412 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:23:04.412 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.190676+0000", 2024-09-13T11:23:04.412 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00015268000000000001 2024-09-13T11:23:04.412 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:04.412 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:04.412 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 12", 2024-09-13T11:23:04.412 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.190822+0000", 2024-09-13T11:23:04.412 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00014630099999999999 2024-09-13T11:23:04.412 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:04.413 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:04.413 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:23:04.413 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.194639+0000", 2024-09-13T11:23:04.413 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.003816963 2024-09-13T11:23:04.413 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:04.413 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:04.413 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:04.413 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.240341+0000", 2024-09-13T11:23:04.413 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.045701861000000003 2024-09-13T11:23:04.413 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:04.413 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:04.413 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:23:04.413 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.240377+0000", 2024-09-13T11:23:04.414 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.6429e-05 2024-09-13T11:23:04.414 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:04.414 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:04.414 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:23:04.414 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.240407+0000", 2024-09-13T11:23:04.414 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.9303e-05 2024-09-13T11:23:04.414 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:04.414 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:23:04.414 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:04.414 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:04.414 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:04.414 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:25234 3.4 3:2d0fda32:::benchmark_data_smithi177_31116_object25233:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e55)", 2024-09-13T11:23:04.414 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:04.243554+0000", 2024-09-13T11:23:04.415 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 0.068769091000000004, 2024-09-13T11:23:04.415 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.039204449000000002, 2024-09-13T11:23:04.415 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:23:04.415 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:23:04.415 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:04.415 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:23:04.415 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:23:04.415 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:04.415 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 25234 2024-09-13T11:23:04.415 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:04.415 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:23:04.415 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:04.416 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:23:04.416 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.243554+0000", 2024-09-13T11:23:04.416 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:04.416 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:04.416 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:04.416 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:23:04.416 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.243554+0000", 2024-09-13T11:23:04.416 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:04.416 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:04.416 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:04.416 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:23:04.416 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.243556+0000", 2024-09-13T11:23:04.417 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.08e-06 2024-09-13T11:23:04.417 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:04.417 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:04.417 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:23:04.417 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.243625+0000", 2024-09-13T11:23:04.417 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.9679999999999997e-05 2024-09-13T11:23:04.417 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:04.417 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:04.417 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:23:04.417 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.243639+0000", 2024-09-13T11:23:04.417 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.3259e-05 2024-09-13T11:23:04.417 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:04.418 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:04.418 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:23:04.418 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.243645+0000", 2024-09-13T11:23:04.418 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.7000000000000002e-06 2024-09-13T11:23:04.418 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:04.418 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:04.418 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:23:04.418 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.243691+0000", 2024-09-13T11:23:04.418 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.6162999999999998e-05 2024-09-13T11:23:04.418 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:04.418 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:04.418 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:23:04.418 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.243805+0000", 2024-09-13T11:23:04.419 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000113978 2024-09-13T11:23:04.419 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:04.419 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:04.419 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 12", 2024-09-13T11:23:04.419 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.243893+0000", 2024-09-13T11:23:04.419 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.7802999999999999e-05 2024-09-13T11:23:04.419 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:04.419 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:04.419 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:23:04.419 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.245147+0000", 2024-09-13T11:23:04.419 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0012536400000000001 2024-09-13T11:23:04.419 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:04.419 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:04.419 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:04.420 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.282692+0000", 2024-09-13T11:23:04.420 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.037545575999999997 2024-09-13T11:23:04.420 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:04.420 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:04.420 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:23:04.420 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.282730+0000", 2024-09-13T11:23:04.420 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.7110000000000002e-05 2024-09-13T11:23:04.420 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:04.420 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:04.420 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:23:04.420 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.282758+0000", 2024-09-13T11:23:04.420 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.8459999999999999e-05 2024-09-13T11:23:04.420 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:04.421 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:23:04.421 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:04.421 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:04.421 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:23:04.421 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-09-13T11:23:04.423 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-13T11:23:04.423 INFO:tasks.ceph.osd.15.smithi177.stderr:2024-09-13T11:23:04.423+0000 7f9d6ce77640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26578) UID: 0 2024-09-13T11:23:04.464 INFO:teuthology.orchestra.run.smithi001.stderr:set 3.e pg_upmap mapping to [0,1] 2024-09-13T11:23:04.464 INFO:teuthology.orchestra.run.smithi001.stderr:nodeep-scrub is set 2024-09-13T11:23:04.523 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-09-13T11:23:04.519+0000 7f97befc9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26315) UID: 0 2024-09-13T11:23:04.566 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-13T11:23:04.566 INFO:teuthology.orchestra.run.smithi001.stdout: "ops": [], 2024-09-13T11:23:04.566 INFO:teuthology.orchestra.run.smithi001.stdout: "num_ops": 0 2024-09-13T11:23:04.566 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-13T11:23:04.580 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-13T11:23:04.624 INFO:tasks.ceph.osd.2.smithi162.stderr:2024-09-13T11:23:04.619+0000 7fe92b35a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26313) UID: 0 2024-09-13T11:23:04.705 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-13T11:23:04.705 INFO:teuthology.orchestra.run.smithi001.stdout: "ops": [], 2024-09-13T11:23:04.705 INFO:teuthology.orchestra.run.smithi001.stdout: "complaint_time": 30, 2024-09-13T11:23:04.705 INFO:teuthology.orchestra.run.smithi001.stdout: "num_blocked_ops": 0 2024-09-13T11:23:04.705 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-13T11:23:04.721 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-13T11:23:04.724 INFO:tasks.ceph.osd.5.smithi092.stderr:2024-09-13T11:23:04.722+0000 7f6f0b705640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26346) UID: 0 2024-09-13T11:23:04.825 INFO:tasks.ceph.osd.11.smithi177.stderr:2024-09-13T11:23:04.823+0000 7f24792c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26580) UID: 0 2024-09-13T11:23:04.859 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-13T11:23:04.860 INFO:teuthology.orchestra.run.smithi001.stdout: "size": 20, 2024-09-13T11:23:04.860 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 600, 2024-09-13T11:23:04.860 INFO:teuthology.orchestra.run.smithi001.stdout: "ops": [ 2024-09-13T11:23:04.860 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:04.860 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4257.0:20 1.0 e24/23 1:30306672:devicehealth::main.db.0000000000000000:head v 24'10, mlcod=24'8)", 2024-09-13T11:23:04.860 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:22:12.167115+0000", 2024-09-13T11:23:04.860 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 52.688650778000003, 2024-09-13T11:23:04.860 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0022173010000000001, 2024-09-13T11:23:04.860 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:23:04.860 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:23:04.861 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:04.861 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:23:04.861 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:04.861 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:23:04.861 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:12.167115+0000", 2024-09-13T11:23:04.861 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:04.861 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:04.861 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:04.861 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:23:04.861 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:12.167115+0000", 2024-09-13T11:23:04.861 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:04.861 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:04.862 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:04.862 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:23:04.862 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:12.167117+0000", 2024-09-13T11:23:04.862 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.866e-06 2024-09-13T11:23:04.862 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:04.862 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:04.862 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:23:04.862 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:12.167367+0000", 2024-09-13T11:23:04.863 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00024982599999999998 2024-09-13T11:23:04.863 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:04.863 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:04.863 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:23:04.863 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:12.167415+0000", 2024-09-13T11:23:04.863 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.8473000000000002e-05 2024-09-13T11:23:04.863 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:04.863 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:04.863 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:23:04.863 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:12.167426+0000", 2024-09-13T11:23:04.863 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.1234e-05 2024-09-13T11:23:04.864 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:04.864 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:04.864 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:04.864 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:12.167563+0000", 2024-09-13T11:23:04.864 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000136613 2024-09-13T11:23:04.864 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:04.864 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:04.864 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:23:04.864 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:12.167602+0000", 2024-09-13T11:23:04.865 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.9236999999999997e-05 2024-09-13T11:23:04.865 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:04.865 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:04.865 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:23:04.865 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:12.169240+0000", 2024-09-13T11:23:04.865 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.001637854 2024-09-13T11:23:04.865 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:04.865 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:04.865 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:23:04.865 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:12.169332+0000", 2024-09-13T11:23:04.865 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.2198000000000001e-05 2024-09-13T11:23:04.865 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:04.866 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:04.866 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:04.866 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:04.866 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:04.866 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4257.0:21 1.0 e24/23 1:30306672:devicehealth::main.db.0000000000000000:head v 24'11, mlcod=24'8)", 2024-09-13T11:23:04.866 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:22:12.168401+0000", 2024-09-13T11:23:04.866 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 52.687364965, 2024-09-13T11:23:04.866 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0022842819999999999, 2024-09-13T11:23:04.866 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:23:04.867 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:23:04.867 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:04.867 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:23:04.867 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:04.867 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:23:04.867 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:12.168401+0000", 2024-09-13T11:23:04.867 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:04.867 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:04.867 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:04.867 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:23:04.868 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:12.168401+0000", 2024-09-13T11:23:04.868 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:04.868 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:04.868 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:04.868 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:23:04.868 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:12.168404+0000", 2024-09-13T11:23:04.868 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.9859999999999999e-06 2024-09-13T11:23:04.868 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:04.868 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:04.869 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:23:04.869 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:12.168562+0000", 2024-09-13T11:23:04.869 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00015851400000000001 2024-09-13T11:23:04.869 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:04.869 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:04.869 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:23:04.869 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:12.168634+0000", 2024-09-13T11:23:04.869 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.1737999999999997e-05 2024-09-13T11:23:04.869 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:04.869 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:04.869 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:23:04.869 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:12.168654+0000", 2024-09-13T11:23:04.870 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.0350999999999999e-05 2024-09-13T11:23:04.870 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:04.870 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:04.870 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:04.870 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:12.168801+0000", 2024-09-13T11:23:04.870 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00014716999999999999 2024-09-13T11:23:04.870 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:04.870 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:04.870 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:23:04.870 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:12.168836+0000", 2024-09-13T11:23:04.870 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.4184999999999999e-05 2024-09-13T11:23:04.870 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:04.870 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:04.871 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:23:04.871 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:12.170571+0000", 2024-09-13T11:23:04.871 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.001735826 2024-09-13T11:23:04.871 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:04.871 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:04.871 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:23:04.871 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:12.170685+0000", 2024-09-13T11:23:04.871 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00011351199999999999 2024-09-13T11:23:04.871 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:04.871 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:04.871 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:04.871 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:04.871 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:04.872 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4257.0:64 1.0 e24/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 24'42, mlcod=24'40)", 2024-09-13T11:23:04.872 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:22:12.455922+0000", 2024-09-13T11:23:04.872 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 52.399843310999998, 2024-09-13T11:23:04.872 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0025220899999999998, 2024-09-13T11:23:04.872 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:23:04.872 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:23:04.872 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:04.872 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:23:04.872 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:04.872 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:23:04.872 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:12.455922+0000", 2024-09-13T11:23:04.872 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:04.873 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:04.873 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:04.873 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:23:04.873 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:12.455922+0000", 2024-09-13T11:23:04.873 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:04.873 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:04.873 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:04.873 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:23:04.873 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:12.455925+0000", 2024-09-13T11:23:04.873 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.1130000000000001e-06 2024-09-13T11:23:04.873 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:04.873 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:04.873 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:23:04.873 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:12.455948+0000", 2024-09-13T11:23:04.874 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.2133e-05 2024-09-13T11:23:04.874 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:04.874 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:04.874 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:23:04.874 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:12.456021+0000", 2024-09-13T11:23:04.874 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.3489000000000001e-05 2024-09-13T11:23:04.874 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:04.874 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:04.874 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:23:04.874 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:12.456034+0000", 2024-09-13T11:23:04.875 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.3353000000000001e-05 2024-09-13T11:23:04.875 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:04.875 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:04.875 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:04.875 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:12.456154+0000", 2024-09-13T11:23:04.875 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00012002699999999999 2024-09-13T11:23:04.875 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:04.876 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:04.876 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:23:04.876 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:12.456191+0000", 2024-09-13T11:23:04.876 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.6505999999999999e-05 2024-09-13T11:23:04.876 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:04.876 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:04.876 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:23:04.876 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:12.458364+0000", 2024-09-13T11:23:04.876 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0021730740000000001 2024-09-13T11:23:04.877 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:04.877 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:04.877 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:23:04.877 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:12.458444+0000", 2024-09-13T11:23:04.877 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.0395000000000005e-05 2024-09-13T11:23:04.877 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:04.877 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:04.877 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:04.877 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:04.877 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:04.878 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4257.0:75 1.0 e24/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 24'53, mlcod=24'49)", 2024-09-13T11:23:04.878 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:22:12.503829+0000", 2024-09-13T11:23:04.878 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 52.351936508999998, 2024-09-13T11:23:04.878 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0020510960000000001, 2024-09-13T11:23:04.878 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:23:04.878 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:23:04.878 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:04.878 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:23:04.878 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:04.879 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:23:04.879 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:12.503829+0000", 2024-09-13T11:23:04.879 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:04.879 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:04.879 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:04.879 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:23:04.879 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:12.503829+0000", 2024-09-13T11:23:04.879 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:04.879 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:04.879 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:04.880 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:23:04.880 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:12.503832+0000", 2024-09-13T11:23:04.880 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.7599999999999998e-06 2024-09-13T11:23:04.880 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:04.880 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:04.880 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:23:04.880 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:12.503922+0000", 2024-09-13T11:23:04.880 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.0035000000000001e-05 2024-09-13T11:23:04.880 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:04.880 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:04.881 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:23:04.881 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:12.503980+0000", 2024-09-13T11:23:04.881 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.8421999999999997e-05 2024-09-13T11:23:04.881 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:04.881 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:04.881 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:23:04.881 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:12.503993+0000", 2024-09-13T11:23:04.881 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.313e-05 2024-09-13T11:23:04.881 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:04.881 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:04.881 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:04.881 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:12.504099+0000", 2024-09-13T11:23:04.882 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000105624 2024-09-13T11:23:04.882 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:04.882 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:04.882 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:23:04.882 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:12.504138+0000", 2024-09-13T11:23:04.882 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.8736000000000003e-05 2024-09-13T11:23:04.882 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:04.882 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:04.882 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:23:04.882 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:12.505801+0000", 2024-09-13T11:23:04.882 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.001662981 2024-09-13T11:23:04.882 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:04.883 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:04.883 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:23:04.883 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:12.505880+0000", 2024-09-13T11:23:04.883 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.9407999999999996e-05 2024-09-13T11:23:04.883 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:04.883 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:04.883 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:04.883 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:04.883 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:04.883 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4257.0:77 1.0 e24/23 1:30306672:devicehealth::main.db.0000000000000000:head v 24'55, mlcod=24'53)", 2024-09-13T11:23:04.883 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:22:12.514870+0000", 2024-09-13T11:23:04.883 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 52.340895533999998, 2024-09-13T11:23:04.884 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0022810920000000002, 2024-09-13T11:23:04.884 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:23:04.884 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:23:04.884 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:04.884 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:23:04.884 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:04.884 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:23:04.884 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:12.514870+0000", 2024-09-13T11:23:04.885 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:04.885 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:04.885 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:04.885 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:23:04.885 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:12.514870+0000", 2024-09-13T11:23:04.885 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:04.886 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:04.886 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:04.886 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:23:04.886 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:12.514873+0000", 2024-09-13T11:23:04.886 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.2839999999999998e-06 2024-09-13T11:23:04.886 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:04.886 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:04.886 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:23:04.886 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:12.514924+0000", 2024-09-13T11:23:04.886 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.1001999999999997e-05 2024-09-13T11:23:04.886 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:04.886 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:04.887 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:23:04.887 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:12.514988+0000", 2024-09-13T11:23:04.887 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.3218000000000004e-05 2024-09-13T11:23:04.887 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:04.887 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:04.887 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:23:04.887 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:12.515001+0000", 2024-09-13T11:23:04.887 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.3502999999999999e-05 2024-09-13T11:23:04.887 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:04.887 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:04.887 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:04.887 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:12.515130+0000", 2024-09-13T11:23:04.888 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00012927199999999999 2024-09-13T11:23:04.888 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:04.888 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:04.888 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:23:04.888 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:12.515167+0000", 2024-09-13T11:23:04.888 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.6259e-05 2024-09-13T11:23:04.888 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:04.888 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:04.888 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:23:04.888 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:12.517017+0000", 2024-09-13T11:23:04.888 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0018506270000000001 2024-09-13T11:23:04.888 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:04.888 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:04.889 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:23:04.889 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:12.517151+0000", 2024-09-13T11:23:04.889 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00013392699999999999 2024-09-13T11:23:04.889 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:04.889 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:04.889 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:04.889 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:04.889 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:04.889 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4257.0:78 1.0 e24/23 1:30306672:devicehealth::main.db.0000000000000000:head v 24'56, mlcod=24'53)", 2024-09-13T11:23:04.889 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:22:12.516252+0000", 2024-09-13T11:23:04.889 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 52.339513654000001, 2024-09-13T11:23:04.889 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0023671, 2024-09-13T11:23:04.890 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:23:04.890 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:23:04.890 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:04.890 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:23:04.890 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:04.890 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:23:04.890 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:12.516252+0000", 2024-09-13T11:23:04.890 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:04.890 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:04.890 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:04.890 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:23:04.890 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:12.516252+0000", 2024-09-13T11:23:04.890 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:04.891 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:04.891 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:04.891 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:23:04.891 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:12.516255+0000", 2024-09-13T11:23:04.891 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.993e-06 2024-09-13T11:23:04.891 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:04.891 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:04.891 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:23:04.891 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:12.516342+0000", 2024-09-13T11:23:04.891 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.7447999999999994e-05 2024-09-13T11:23:04.891 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:04.891 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:04.892 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:23:04.892 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:12.516418+0000", 2024-09-13T11:23:04.892 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.5492000000000002e-05 2024-09-13T11:23:04.892 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:04.892 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:04.892 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:23:04.892 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:12.516442+0000", 2024-09-13T11:23:04.892 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.4105999999999999e-05 2024-09-13T11:23:04.892 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:04.893 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:04.893 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:04.893 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:12.516540+0000", 2024-09-13T11:23:04.893 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.7502000000000003e-05 2024-09-13T11:23:04.893 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:04.894 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:04.894 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:23:04.894 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:12.516571+0000", 2024-09-13T11:23:04.894 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.1285000000000003e-05 2024-09-13T11:23:04.894 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:04.894 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:04.894 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:23:04.894 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:12.518498+0000", 2024-09-13T11:23:04.894 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0019272759999999999 2024-09-13T11:23:04.895 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:04.895 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:04.895 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:23:04.895 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:12.518619+0000", 2024-09-13T11:23:04.895 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000120998 2024-09-13T11:23:04.895 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:04.895 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:04.895 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:04.895 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:04.895 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:04.895 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4257.0:101 1.0 e24/23 1:30306672:devicehealth::main.db.0000000000000000:head v 24'75, mlcod=24'73)", 2024-09-13T11:23:04.895 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:22:13.092579+0000", 2024-09-13T11:23:04.895 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 51.763186715000003, 2024-09-13T11:23:04.895 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0025088380000000002, 2024-09-13T11:23:04.896 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:23:04.896 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:23:04.896 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:04.896 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:23:04.896 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:04.896 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:23:04.896 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:13.092579+0000", 2024-09-13T11:23:04.896 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:04.896 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:04.896 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:04.896 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:23:04.896 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:13.092579+0000", 2024-09-13T11:23:04.897 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:04.897 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:04.897 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:04.897 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:23:04.897 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:13.092582+0000", 2024-09-13T11:23:04.897 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.9569999999999998e-06 2024-09-13T11:23:04.897 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:04.897 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:04.897 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:23:04.897 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:13.092638+0000", 2024-09-13T11:23:04.897 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.5812e-05 2024-09-13T11:23:04.897 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:04.897 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:04.898 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:23:04.898 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:13.092701+0000", 2024-09-13T11:23:04.898 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.2866000000000004e-05 2024-09-13T11:23:04.898 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:04.898 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:04.898 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:23:04.898 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:13.092714+0000", 2024-09-13T11:23:04.898 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.3533000000000001e-05 2024-09-13T11:23:04.898 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:04.898 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:04.898 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:04.898 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:13.092798+0000", 2024-09-13T11:23:04.898 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.4132e-05 2024-09-13T11:23:04.899 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:04.899 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:04.899 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:23:04.899 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:13.092828+0000", 2024-09-13T11:23:04.899 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.9349e-05 2024-09-13T11:23:04.899 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:04.899 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:04.899 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:23:04.899 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:13.095016+0000", 2024-09-13T11:23:04.899 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.002188307 2024-09-13T11:23:04.899 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:04.899 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:04.899 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:23:04.900 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:13.095088+0000", 2024-09-13T11:23:04.900 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.1882000000000001e-05 2024-09-13T11:23:04.900 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:04.900 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:04.900 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:04.900 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:04.900 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:04.900 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4257.0:104 1.0 e24/23 1:30306672:devicehealth::main.db.0000000000000000:head v 24'78, mlcod=24'73)", 2024-09-13T11:23:04.900 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:22:13.094467+0000", 2024-09-13T11:23:04.900 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 51.761298345, 2024-09-13T11:23:04.900 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0020875860000000002, 2024-09-13T11:23:04.900 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:23:04.900 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:23:04.901 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:04.901 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:23:04.901 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:04.901 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:23:04.901 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:13.094467+0000", 2024-09-13T11:23:04.901 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:04.901 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:04.901 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:04.901 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:23:04.901 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:13.094467+0000", 2024-09-13T11:23:04.901 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:04.901 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:04.901 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:04.901 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:23:04.902 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:13.094470+0000", 2024-09-13T11:23:04.902 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.9229999999999998e-06 2024-09-13T11:23:04.902 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:04.902 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:04.902 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:23:04.902 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:13.094657+0000", 2024-09-13T11:23:04.902 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00018684600000000001 2024-09-13T11:23:04.902 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:04.902 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:04.902 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:23:04.902 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:13.094753+0000", 2024-09-13T11:23:04.902 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.6021999999999996e-05 2024-09-13T11:23:04.902 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:04.902 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:04.903 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:23:04.903 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:13.094771+0000", 2024-09-13T11:23:04.903 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.7493e-05 2024-09-13T11:23:04.903 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:04.903 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:04.903 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:04.903 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:13.095247+0000", 2024-09-13T11:23:04.903 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00047664400000000003 2024-09-13T11:23:04.903 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:04.903 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:04.903 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:23:04.903 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:13.095292+0000", 2024-09-13T11:23:04.903 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.5272999999999998e-05 2024-09-13T11:23:04.904 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:04.904 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:04.904 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:23:04.904 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:13.096492+0000", 2024-09-13T11:23:04.904 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0011997519999999999 2024-09-13T11:23:04.904 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:04.904 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:04.904 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:23:04.904 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:13.096555+0000", 2024-09-13T11:23:04.904 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.2632999999999995e-05 2024-09-13T11:23:04.904 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:04.904 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:04.904 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:04.904 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:04.905 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:04.905 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4257.0:112 1.0 e25/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 25'82, mlcod=24'80)", 2024-09-13T11:23:04.905 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:22:13.681892+0000", 2024-09-13T11:23:04.905 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 51.173873946000001, 2024-09-13T11:23:04.905 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.002076629, 2024-09-13T11:23:04.905 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:23:04.905 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:23:04.905 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:04.905 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:23:04.905 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:04.905 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:23:04.905 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:13.681892+0000", 2024-09-13T11:23:04.905 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:04.906 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:04.906 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:04.906 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:23:04.906 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:13.681892+0000", 2024-09-13T11:23:04.906 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:04.906 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:04.906 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:04.906 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:23:04.906 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:13.681895+0000", 2024-09-13T11:23:04.906 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.1700000000000001e-06 2024-09-13T11:23:04.906 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:04.906 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:04.906 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:23:04.906 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:13.681908+0000", 2024-09-13T11:23:04.907 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.3196e-05 2024-09-13T11:23:04.907 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:04.907 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:04.907 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:23:04.907 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:13.681943+0000", 2024-09-13T11:23:04.907 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.4962999999999997e-05 2024-09-13T11:23:04.907 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:04.907 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:04.907 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:23:04.907 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:13.681956+0000", 2024-09-13T11:23:04.907 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2833000000000001e-05 2024-09-13T11:23:04.907 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:04.907 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:04.907 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:04.908 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:13.682128+0000", 2024-09-13T11:23:04.908 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00017215700000000001 2024-09-13T11:23:04.908 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:04.908 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:04.908 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:23:04.908 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:13.682167+0000", 2024-09-13T11:23:04.908 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.8899000000000001e-05 2024-09-13T11:23:04.908 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:04.908 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:04.908 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:23:04.908 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:13.683894+0000", 2024-09-13T11:23:04.908 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.001727452 2024-09-13T11:23:04.908 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:04.908 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:04.909 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:23:04.909 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:13.683968+0000", 2024-09-13T11:23:04.909 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.3959000000000001e-05 2024-09-13T11:23:04.909 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:04.909 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:04.909 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:04.909 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:04.909 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:04.909 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4257.0:122 1.0 e25/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 25'92, mlcod=25'88)", 2024-09-13T11:23:04.909 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:22:13.704190+0000", 2024-09-13T11:23:04.909 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 51.151575286000003, 2024-09-13T11:23:04.909 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0027194540000000001, 2024-09-13T11:23:04.909 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:23:04.909 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:23:04.910 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:04.910 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:23:04.910 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:04.910 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:23:04.910 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:13.704190+0000", 2024-09-13T11:23:04.910 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:04.910 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:04.910 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:04.910 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:23:04.910 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:13.704190+0000", 2024-09-13T11:23:04.910 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:04.910 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:04.910 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:04.910 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:23:04.911 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:13.704192+0000", 2024-09-13T11:23:04.911 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.074e-06 2024-09-13T11:23:04.911 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:04.911 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:04.911 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:23:04.911 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:13.704211+0000", 2024-09-13T11:23:04.911 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.8173e-05 2024-09-13T11:23:04.911 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:04.911 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:04.912 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:23:04.912 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:13.704243+0000", 2024-09-13T11:23:04.912 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.2282e-05 2024-09-13T11:23:04.912 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:04.912 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:04.912 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:23:04.912 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:13.704254+0000", 2024-09-13T11:23:04.912 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.1537e-05 2024-09-13T11:23:04.912 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:04.912 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:04.912 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:04.912 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:13.704425+0000", 2024-09-13T11:23:04.912 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00017065 2024-09-13T11:23:04.913 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:04.913 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:04.913 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:23:04.913 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:13.704464+0000", 2024-09-13T11:23:04.913 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.9316000000000001e-05 2024-09-13T11:23:04.913 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:04.913 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:04.913 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:23:04.913 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:13.706837+0000", 2024-09-13T11:23:04.913 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0023730800000000001 2024-09-13T11:23:04.913 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:04.913 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:04.913 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:23:04.913 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:13.706910+0000", 2024-09-13T11:23:04.914 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.2342000000000006e-05 2024-09-13T11:23:04.914 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:04.914 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:04.914 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:04.914 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:04.914 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:04.914 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4257.0:123 1.0 e25/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 25'93, mlcod=25'89)", 2024-09-13T11:23:04.914 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:22:13.705726+0000", 2024-09-13T11:23:04.914 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 51.150039986000003, 2024-09-13T11:23:04.914 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0024901849999999998, 2024-09-13T11:23:04.914 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:23:04.914 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:23:04.914 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:04.915 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:23:04.915 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:04.915 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:23:04.915 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:13.705726+0000", 2024-09-13T11:23:04.915 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:04.915 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:04.915 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:04.915 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:23:04.915 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:13.705726+0000", 2024-09-13T11:23:04.915 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:04.915 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:04.915 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:04.915 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:23:04.916 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:13.705728+0000", 2024-09-13T11:23:04.916 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.3700000000000002e-06 2024-09-13T11:23:04.916 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:04.916 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:04.916 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:23:04.916 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:13.705799+0000", 2024-09-13T11:23:04.916 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.0776000000000001e-05 2024-09-13T11:23:04.916 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:04.916 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:04.916 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:23:04.916 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:13.705835+0000", 2024-09-13T11:23:04.916 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.5918999999999999e-05 2024-09-13T11:23:04.916 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:04.916 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:04.917 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:23:04.917 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:13.705851+0000", 2024-09-13T11:23:04.917 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.6653000000000001e-05 2024-09-13T11:23:04.917 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:04.917 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:04.917 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:04.917 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:13.705990+0000", 2024-09-13T11:23:04.917 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000138997 2024-09-13T11:23:04.917 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:04.917 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:04.917 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:23:04.917 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:13.706026+0000", 2024-09-13T11:23:04.917 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.5738999999999998e-05 2024-09-13T11:23:04.918 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:04.918 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:04.918 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:23:04.918 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:13.708154+0000", 2024-09-13T11:23:04.918 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0021283280000000001 2024-09-13T11:23:04.918 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:04.918 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:04.918 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:23:04.918 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:13.708216+0000", 2024-09-13T11:23:04.918 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.1402999999999995e-05 2024-09-13T11:23:04.918 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:04.918 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:04.918 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:04.918 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:04.919 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:04.919 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4257.0:126 1.0 e25/23 1:30306672:devicehealth::main.db.0000000000000000:head v 25'96, mlcod=25'93)", 2024-09-13T11:23:04.919 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:22:13.711692+0000", 2024-09-13T11:23:04.919 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 51.14407353, 2024-09-13T11:23:04.919 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0021595410000000001, 2024-09-13T11:23:04.919 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:23:04.919 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:23:04.919 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:04.919 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:23:04.919 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:04.919 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:23:04.919 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:13.711692+0000", 2024-09-13T11:23:04.920 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:04.920 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:04.920 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:04.920 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:23:04.920 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:13.711692+0000", 2024-09-13T11:23:04.920 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:04.920 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:04.920 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:04.920 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:23:04.920 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:13.711695+0000", 2024-09-13T11:23:04.920 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.5170000000000002e-06 2024-09-13T11:23:04.920 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:04.920 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:04.921 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:23:04.921 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:13.711780+0000", 2024-09-13T11:23:04.921 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.5427999999999996e-05 2024-09-13T11:23:04.921 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:04.921 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:04.921 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:23:04.921 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:13.711819+0000", 2024-09-13T11:23:04.921 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.9303e-05 2024-09-13T11:23:04.921 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:04.921 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:04.921 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:23:04.921 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:13.711860+0000", 2024-09-13T11:23:04.921 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.0194999999999997e-05 2024-09-13T11:23:04.922 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:04.922 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:04.922 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:04.922 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:13.712001+0000", 2024-09-13T11:23:04.922 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000141865 2024-09-13T11:23:04.922 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:04.922 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:04.922 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:23:04.922 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:13.712039+0000", 2024-09-13T11:23:04.922 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.7905999999999999e-05 2024-09-13T11:23:04.922 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:04.922 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:04.922 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:23:04.923 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:13.713734+0000", 2024-09-13T11:23:04.923 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.001694426 2024-09-13T11:23:04.923 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:04.923 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:04.923 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:23:04.923 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:13.713852+0000", 2024-09-13T11:23:04.923 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00011790099999999999 2024-09-13T11:23:04.923 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:04.923 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:04.923 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:04.923 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:04.923 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:04.923 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4257.0:136 1.0 e26/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 26'102, mlcod=25'100)", 2024-09-13T11:23:04.923 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:22:14.606007+0000", 2024-09-13T11:23:04.924 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 50.249758682, 2024-09-13T11:23:04.924 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0041530050000000004, 2024-09-13T11:23:04.924 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:23:04.924 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:23:04.924 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:04.924 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:23:04.924 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:04.924 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:23:04.924 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:14.606007+0000", 2024-09-13T11:23:04.924 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:04.924 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:04.924 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:04.924 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:23:04.925 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:14.606007+0000", 2024-09-13T11:23:04.925 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:04.925 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:04.925 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:04.925 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:23:04.925 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:14.606010+0000", 2024-09-13T11:23:04.925 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.0800000000000002e-06 2024-09-13T11:23:04.925 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:04.925 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:04.925 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:23:04.925 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:14.606023+0000", 2024-09-13T11:23:04.925 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.3139000000000001e-05 2024-09-13T11:23:04.925 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:04.925 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:04.926 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:23:04.926 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:14.606096+0000", 2024-09-13T11:23:04.926 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.2941999999999994e-05 2024-09-13T11:23:04.926 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:04.926 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:04.926 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:23:04.926 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:14.606110+0000", 2024-09-13T11:23:04.926 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.3563999999999999e-05 2024-09-13T11:23:04.926 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:04.926 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:04.926 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:04.926 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:14.606249+0000", 2024-09-13T11:23:04.926 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00013969300000000001 2024-09-13T11:23:04.927 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:04.927 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:04.927 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:23:04.927 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:14.606306+0000", 2024-09-13T11:23:04.927 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.6938999999999998e-05 2024-09-13T11:23:04.927 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:04.927 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:04.927 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:23:04.927 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:14.610068+0000", 2024-09-13T11:23:04.927 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0037620900000000001 2024-09-13T11:23:04.927 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:04.927 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:04.927 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:23:04.928 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:14.610160+0000", 2024-09-13T11:23:04.928 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.1558000000000007e-05 2024-09-13T11:23:04.928 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:04.928 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:04.928 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:04.928 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:04.928 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:04.928 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4257.0:137 1.0 e26/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 26'103, mlcod=25'100)", 2024-09-13T11:23:04.928 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:22:14.606498+0000", 2024-09-13T11:23:04.928 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 50.249267652, 2024-09-13T11:23:04.928 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0037471599999999998, 2024-09-13T11:23:04.928 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:23:04.928 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:23:04.928 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:04.929 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:23:04.929 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:04.929 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:23:04.929 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:14.606498+0000", 2024-09-13T11:23:04.929 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:04.929 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:04.929 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:04.929 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:23:04.929 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:14.606498+0000", 2024-09-13T11:23:04.929 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:04.929 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:04.929 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:04.929 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:23:04.930 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:14.606501+0000", 2024-09-13T11:23:04.930 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.0800000000000002e-06 2024-09-13T11:23:04.930 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:04.930 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:04.930 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:23:04.930 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:14.606514+0000", 2024-09-13T11:23:04.930 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2853e-05 2024-09-13T11:23:04.930 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:04.930 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:04.930 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:23:04.930 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:14.606582+0000", 2024-09-13T11:23:04.930 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.8119000000000003e-05 2024-09-13T11:23:04.930 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:04.931 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:04.931 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:23:04.931 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:14.606595+0000", 2024-09-13T11:23:04.931 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.3183000000000001e-05 2024-09-13T11:23:04.931 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:04.931 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:04.931 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:04.931 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:14.607328+0000", 2024-09-13T11:23:04.931 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00073325599999999995 2024-09-13T11:23:04.931 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:04.931 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:04.931 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:23:04.931 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:14.607368+0000", 2024-09-13T11:23:04.932 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.9192e-05 2024-09-13T11:23:04.932 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:04.932 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:04.932 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:23:04.932 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:14.610169+0000", 2024-09-13T11:23:04.932 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.002801502 2024-09-13T11:23:04.932 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:04.932 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:04.932 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:23:04.932 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:14.610245+0000", 2024-09-13T11:23:04.932 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.5975000000000004e-05 2024-09-13T11:23:04.932 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:04.933 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:04.933 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:04.933 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:04.933 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:04.933 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4257.0:138 1.0 e26/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 26'104, mlcod=25'100)", 2024-09-13T11:23:04.933 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:22:14.607069+0000", 2024-09-13T11:23:04.933 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 50.24869631, 2024-09-13T11:23:04.933 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0051436090000000004, 2024-09-13T11:23:04.933 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:23:04.933 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:23:04.933 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:04.933 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:23:04.933 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:04.933 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:23:04.934 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:14.607069+0000", 2024-09-13T11:23:04.934 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:04.934 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:04.934 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:04.934 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:23:04.934 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:14.607069+0000", 2024-09-13T11:23:04.934 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:04.934 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:04.934 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:04.934 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:23:04.934 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:14.607073+0000", 2024-09-13T11:23:04.934 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.2100000000000002e-06 2024-09-13T11:23:04.934 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:04.935 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:04.935 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:23:04.935 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:14.607311+0000", 2024-09-13T11:23:04.935 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00023832500000000001 2024-09-13T11:23:04.935 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:04.935 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:04.935 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:23:04.935 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:14.607393+0000", 2024-09-13T11:23:04.935 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.1695999999999998e-05 2024-09-13T11:23:04.935 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:04.935 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:04.935 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:23:04.935 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:14.607415+0000", 2024-09-13T11:23:04.936 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.2215999999999999e-05 2024-09-13T11:23:04.936 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:04.936 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:04.936 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:04.936 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:14.608040+0000", 2024-09-13T11:23:04.936 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00062501400000000002 2024-09-13T11:23:04.936 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:04.936 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:04.936 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:23:04.936 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:14.608091+0000", 2024-09-13T11:23:04.936 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.1269e-05 2024-09-13T11:23:04.936 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:04.936 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:04.937 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:23:04.937 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:14.612121+0000", 2024-09-13T11:23:04.937 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0040300470000000001 2024-09-13T11:23:04.937 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:04.937 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:04.937 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:23:04.937 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:14.612213+0000", 2024-09-13T11:23:04.937 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.1831999999999998e-05 2024-09-13T11:23:04.937 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:04.937 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:04.937 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:04.937 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:04.937 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:04.938 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4257.0:139 1.0 e26/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 26'105, mlcod=25'100)", 2024-09-13T11:23:04.938 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:22:14.607956+0000", 2024-09-13T11:23:04.938 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 50.247809306999997, 2024-09-13T11:23:04.938 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0043423810000000002, 2024-09-13T11:23:04.938 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:23:04.938 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:23:04.938 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:04.938 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:23:04.938 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:04.938 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:23:04.938 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:14.607956+0000", 2024-09-13T11:23:04.938 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:04.938 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:04.939 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:04.939 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:23:04.939 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:14.607956+0000", 2024-09-13T11:23:04.939 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:04.939 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:04.939 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:04.939 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:23:04.939 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:14.607959+0000", 2024-09-13T11:23:04.939 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.7259999999999998e-06 2024-09-13T11:23:04.939 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:04.939 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:04.939 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:23:04.939 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:14.607971+0000", 2024-09-13T11:23:04.939 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.19e-05 2024-09-13T11:23:04.940 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:04.940 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:04.940 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:23:04.940 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:14.608043+0000", 2024-09-13T11:23:04.940 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.2062000000000002e-05 2024-09-13T11:23:04.940 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:04.940 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:04.940 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:23:04.940 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:14.608071+0000", 2024-09-13T11:23:04.940 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.8099e-05 2024-09-13T11:23:04.940 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:04.940 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:04.940 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:04.940 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:14.609248+0000", 2024-09-13T11:23:04.941 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0011773510000000001 2024-09-13T11:23:04.941 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:04.941 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:04.941 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:23:04.941 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:14.609325+0000", 2024-09-13T11:23:04.941 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.6432000000000003e-05 2024-09-13T11:23:04.941 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:04.941 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:04.941 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:23:04.941 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:14.612226+0000", 2024-09-13T11:23:04.941 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.002900742 2024-09-13T11:23:04.941 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:04.942 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:04.942 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:23:04.942 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:14.612299+0000", 2024-09-13T11:23:04.942 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.3069000000000001e-05 2024-09-13T11:23:04.942 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:04.942 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:04.942 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:04.942 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:04.942 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:04.942 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4257.0:140 1.0 e26/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 26'106, mlcod=25'100)", 2024-09-13T11:23:04.942 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:22:14.608506+0000", 2024-09-13T11:23:04.942 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 50.247259690999996, 2024-09-13T11:23:04.942 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0038706830000000002, 2024-09-13T11:23:04.942 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:23:04.943 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:23:04.943 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:04.943 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:23:04.943 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:04.943 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:23:04.943 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:14.608506+0000", 2024-09-13T11:23:04.943 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:04.943 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:04.943 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:04.943 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:23:04.943 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:14.608506+0000", 2024-09-13T11:23:04.943 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:04.943 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:04.943 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:04.944 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:23:04.944 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:14.608509+0000", 2024-09-13T11:23:04.944 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.1829999999999998e-06 2024-09-13T11:23:04.944 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:04.944 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:04.944 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:23:04.944 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:14.608526+0000", 2024-09-13T11:23:04.944 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.7178999999999999e-05 2024-09-13T11:23:04.944 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:04.944 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:04.944 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:23:04.945 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:14.608596+0000", 2024-09-13T11:23:04.945 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.9763000000000003e-05 2024-09-13T11:23:04.945 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:04.945 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:04.945 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:23:04.945 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:14.608618+0000", 2024-09-13T11:23:04.945 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.2062e-05 2024-09-13T11:23:04.945 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:04.945 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:04.945 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:04.945 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:14.610337+0000", 2024-09-13T11:23:04.945 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0017188640000000001 2024-09-13T11:23:04.945 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:04.945 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:04.946 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:23:04.946 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:14.610375+0000", 2024-09-13T11:23:04.946 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.8488999999999997e-05 2024-09-13T11:23:04.946 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:04.946 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:04.946 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:23:04.946 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:14.612306+0000", 2024-09-13T11:23:04.946 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.001930532 2024-09-13T11:23:04.946 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:04.946 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:04.946 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:23:04.946 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:14.612377+0000", 2024-09-13T11:23:04.946 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.0611000000000005e-05 2024-09-13T11:23:04.947 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:04.947 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:04.947 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:04.947 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:04.947 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:04.947 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4257.0:141 1.0 e26/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 26'107, mlcod=25'100)", 2024-09-13T11:23:04.947 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:22:14.609021+0000", 2024-09-13T11:23:04.947 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 50.246744851000003, 2024-09-13T11:23:04.947 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0041286350000000003, 2024-09-13T11:23:04.947 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:23:04.947 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:23:04.947 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:04.947 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:23:04.948 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:04.948 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:23:04.948 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:14.609021+0000", 2024-09-13T11:23:04.948 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:04.948 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:04.948 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:04.948 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:23:04.948 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:14.609021+0000", 2024-09-13T11:23:04.948 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:04.948 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:04.948 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:04.948 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:23:04.948 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:14.609023+0000", 2024-09-13T11:23:04.949 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.5160000000000001e-06 2024-09-13T11:23:04.949 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:04.949 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:04.949 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:23:04.949 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:14.609085+0000", 2024-09-13T11:23:04.949 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.1731999999999998e-05 2024-09-13T11:23:04.949 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:04.949 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:04.949 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:23:04.949 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:14.609137+0000", 2024-09-13T11:23:04.949 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.2142000000000003e-05 2024-09-13T11:23:04.949 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:04.949 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:04.949 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:23:04.950 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:14.609148+0000", 2024-09-13T11:23:04.950 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.115e-05 2024-09-13T11:23:04.950 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:04.950 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:04.950 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:04.950 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:14.611058+0000", 2024-09-13T11:23:04.950 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.001909289 2024-09-13T11:23:04.950 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:04.950 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:04.950 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:23:04.950 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:14.611109+0000", 2024-09-13T11:23:04.950 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.1595999999999999e-05 2024-09-13T11:23:04.950 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:04.950 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:04.951 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:23:04.951 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:14.613063+0000", 2024-09-13T11:23:04.951 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0019537719999999999 2024-09-13T11:23:04.951 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:04.951 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:04.951 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:23:04.951 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:14.613149+0000", 2024-09-13T11:23:04.951 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.6438000000000001e-05 2024-09-13T11:23:04.951 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:04.951 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:04.951 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:04.951 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:04.951 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:04.952 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4257.0:142 1.0 e26/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 26'108, mlcod=25'100)", 2024-09-13T11:23:04.952 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:22:14.609825+0000", 2024-09-13T11:23:04.952 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 50.245940769999997, 2024-09-13T11:23:04.952 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0041650200000000002, 2024-09-13T11:23:04.952 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:23:04.952 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:23:04.952 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:04.952 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:23:04.952 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:04.952 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:23:04.952 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:14.609825+0000", 2024-09-13T11:23:04.952 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:04.952 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:04.952 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:04.953 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:23:04.953 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:14.609825+0000", 2024-09-13T11:23:04.953 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:04.953 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:04.953 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:04.953 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:23:04.953 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:14.609828+0000", 2024-09-13T11:23:04.953 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.0429999999999999e-06 2024-09-13T11:23:04.953 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:04.953 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:04.953 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:23:04.953 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:14.609856+0000", 2024-09-13T11:23:04.953 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.7781999999999999e-05 2024-09-13T11:23:04.953 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:04.954 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:04.954 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:23:04.954 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:14.609908+0000", 2024-09-13T11:23:04.954 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.1975999999999998e-05 2024-09-13T11:23:04.954 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:04.954 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:04.954 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:23:04.954 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:14.609919+0000", 2024-09-13T11:23:04.954 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0987e-05 2024-09-13T11:23:04.954 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:04.954 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:04.954 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:04.954 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:14.612447+0000", 2024-09-13T11:23:04.954 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0025280429999999998 2024-09-13T11:23:04.955 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:04.955 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:04.955 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:23:04.955 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:14.612483+0000", 2024-09-13T11:23:04.955 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.6599e-05 2024-09-13T11:23:04.955 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:04.955 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:04.955 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:23:04.955 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:14.613906+0000", 2024-09-13T11:23:04.955 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0014225920000000001 2024-09-13T11:23:04.955 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:04.955 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:04.955 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:23:04.956 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:14.613990+0000", 2024-09-13T11:23:04.956 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.3998000000000005e-05 2024-09-13T11:23:04.956 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:04.956 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:04.956 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:04.956 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:04.956 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:04.956 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4257.0:144 1.0 e26/23 1:30306672:devicehealth::main.db.0000000000000000:head v 26'110, mlcod=26'107)", 2024-09-13T11:23:04.956 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:22:14.615772+0000", 2024-09-13T11:23:04.956 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 50.239993194999997, 2024-09-13T11:23:04.956 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0021387350000000001, 2024-09-13T11:23:04.956 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:23:04.956 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:23:04.957 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:04.957 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:23:04.957 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:04.957 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:23:04.957 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:14.615772+0000", 2024-09-13T11:23:04.957 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:04.957 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:04.957 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:04.957 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:23:04.957 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:14.615772+0000", 2024-09-13T11:23:04.957 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:04.957 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:04.957 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:04.958 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:23:04.958 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:14.615774+0000", 2024-09-13T11:23:04.958 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.623e-06 2024-09-13T11:23:04.958 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:04.958 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:04.958 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:23:04.958 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:14.615859+0000", 2024-09-13T11:23:04.958 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.5382000000000003e-05 2024-09-13T11:23:04.958 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:04.958 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:04.958 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:23:04.958 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:14.615910+0000", 2024-09-13T11:23:04.958 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.1038999999999997e-05 2024-09-13T11:23:04.958 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:04.959 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:04.959 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:23:04.959 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:14.615920+0000", 2024-09-13T11:23:04.959 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.5429999999999999e-06 2024-09-13T11:23:04.959 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:04.959 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:04.959 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:04.959 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:14.616157+0000", 2024-09-13T11:23:04.959 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00023733200000000001 2024-09-13T11:23:04.959 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:04.959 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:04.959 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:23:04.959 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:14.616213+0000", 2024-09-13T11:23:04.960 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.5195000000000003e-05 2024-09-13T11:23:04.960 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:04.960 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:04.960 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:23:04.960 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:14.617831+0000", 2024-09-13T11:23:04.960 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.001618179 2024-09-13T11:23:04.960 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:04.960 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:04.960 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:23:04.960 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:14.617911+0000", 2024-09-13T11:23:04.960 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.0442e-05 2024-09-13T11:23:04.960 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:04.960 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:04.961 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:04.961 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:04.961 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:04.961 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-13T11:23:04.962 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-13T11:23:04.963 INFO:tasks.ceph.osd.7.smithi177.stderr:2024-09-13T11:23:04.959+0000 7f897b51d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26575) UID: 0 2024-09-13T11:23:05.001 INFO:tasks.radosbench.radosbench.0.smithi177.stdout: 38 16 25526 25510 41.9521 36.25 0.0148177 0.023674 2024-09-13T11:23:05.063 INFO:tasks.ceph.osd.4.smithi001.stderr:2024-09-13T11:23:05.056+0000 7ff8e0218640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26310) UID: 0 2024-09-13T11:23:05.097 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-13T11:23:05.098 INFO:teuthology.orchestra.run.smithi001.stdout: "ops": [], 2024-09-13T11:23:05.098 INFO:teuthology.orchestra.run.smithi001.stdout: "num_ops": 0 2024-09-13T11:23:05.098 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-13T11:23:05.112 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-13T11:23:05.164 INFO:tasks.ceph.osd.3.smithi177.stderr:2024-09-13T11:23:05.163+0000 7f10adba4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26574) UID: 0 2024-09-13T11:23:05.265 INFO:tasks.ceph.osd.11.smithi177.stderr:2024-09-13T11:23:05.263+0000 7f24792c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26580) UID: 0 2024-09-13T11:23:05.288 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-13T11:23:05.288 INFO:teuthology.orchestra.run.smithi001.stdout: "ops": [], 2024-09-13T11:23:05.288 INFO:teuthology.orchestra.run.smithi001.stdout: "complaint_time": 30, 2024-09-13T11:23:05.288 INFO:teuthology.orchestra.run.smithi001.stdout: "num_blocked_ops": 0 2024-09-13T11:23:05.288 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-13T11:23:05.302 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-13T11:23:05.366 INFO:tasks.ceph.osd.13.smithi092.stderr:2024-09-13T11:23:05.366+0000 7f3e5afea640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26350) UID: 0 2024-09-13T11:23:05.428 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-13T11:23:05.429 INFO:teuthology.orchestra.run.smithi001.stdout: "size": 20, 2024-09-13T11:23:05.429 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 600, 2024-09-13T11:23:05.429 INFO:teuthology.orchestra.run.smithi001.stdout: "ops": [] 2024-09-13T11:23:05.429 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-13T11:23:05.443 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-09-13T11:23:05.467 INFO:tasks.ceph.osd.10.smithi162.stderr:2024-09-13T11:23:05.463+0000 7fb59c9f0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26317) UID: 0 2024-09-13T11:23:05.474 INFO:tasks.ceph.osd.8.smithi001.stderr:2024-09-13T11:23:05.468+0000 7fb9a92db640 -1 osd.8 56 handle_osd_map injecting map crc failure 2024-09-13T11:23:05.569 INFO:tasks.ceph.osd.5.smithi092.stderr:2024-09-13T11:23:05.566+0000 7f6f0b705640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26346) UID: 0 2024-09-13T11:23:05.575 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-13T11:23:05.575 INFO:teuthology.orchestra.run.smithi001.stdout: "ops": [], 2024-09-13T11:23:05.575 INFO:teuthology.orchestra.run.smithi001.stdout: "num_ops": 0 2024-09-13T11:23:05.575 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-13T11:23:05.590 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-09-13T11:23:05.670 INFO:tasks.ceph.osd.7.smithi177.stderr:2024-09-13T11:23:05.671+0000 7f897b51d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26575) UID: 0 2024-09-13T11:23:05.758 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-13T11:23:05.758 INFO:teuthology.orchestra.run.smithi001.stdout: "ops": [], 2024-09-13T11:23:05.758 INFO:teuthology.orchestra.run.smithi001.stdout: "complaint_time": 30, 2024-09-13T11:23:05.758 INFO:teuthology.orchestra.run.smithi001.stdout: "num_blocked_ops": 0 2024-09-13T11:23:05.758 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-13T11:23:05.771 INFO:tasks.ceph.osd.4.smithi001.stderr:2024-09-13T11:23:05.764+0000 7ff8e0218640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26310) UID: 0 2024-09-13T11:23:05.772 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-09-13T11:23:05.871 INFO:tasks.ceph.osd.11.smithi177.stderr:2024-09-13T11:23:05.871+0000 7f24792c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26580) UID: 0 2024-09-13T11:23:05.939 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T11:23:05.941 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-13T11:23:05.941 INFO:teuthology.orchestra.run.smithi001.stdout: "size": 20, 2024-09-13T11:23:05.942 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 600, 2024-09-13T11:23:05.942 INFO:teuthology.orchestra.run.smithi001.stdout: "ops": [ 2024-09-13T11:23:05.942 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:05.942 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:1015 3.e 3:7d9221f5:::benchmark_data_smithi177_31116_object1014:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-09-13T11:23:05.942 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:22:49.621640+0000", 2024-09-13T11:23:05.943 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 16.313992785, 2024-09-13T11:23:05.943 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.046225468999999998, 2024-09-13T11:23:05.943 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:23:05.943 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:23:05.943 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:05.944 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:23:05.944 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:23:05.944 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:05.944 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 1015 2024-09-13T11:23:05.944 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:05.945 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:23:05.945 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:05.945 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:23:05.945 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:49.621640+0000", 2024-09-13T11:23:05.945 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:05.946 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:05.946 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:05.946 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:23:05.946 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:49.621640+0000", 2024-09-13T11:23:05.946 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:05.946 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:05.947 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:05.947 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:23:05.947 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:49.621642+0000", 2024-09-13T11:23:05.947 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.542e-06 2024-09-13T11:23:05.947 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:05.947 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:05.948 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:23:05.948 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:49.621687+0000", 2024-09-13T11:23:05.948 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.5257999999999999e-05 2024-09-13T11:23:05.948 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:05.948 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:05.949 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:23:05.949 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:49.621708+0000", 2024-09-13T11:23:05.949 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.0771000000000002e-05 2024-09-13T11:23:05.949 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:05.949 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:05.950 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:23:05.950 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:49.621725+0000", 2024-09-13T11:23:05.950 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.6578e-05 2024-09-13T11:23:05.950 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:05.950 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:05.951 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:05.951 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:49.621806+0000", 2024-09-13T11:23:05.951 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.0872999999999996e-05 2024-09-13T11:23:05.951 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:05.951 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:05.952 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:23:05.952 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:49.621948+0000", 2024-09-13T11:23:05.952 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00014266600000000001 2024-09-13T11:23:05.952 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:05.952 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:05.952 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 14", 2024-09-13T11:23:05.953 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:49.622096+0000", 2024-09-13T11:23:05.953 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00014749500000000001 2024-09-13T11:23:05.953 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:05.953 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:05.953 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:23:05.954 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:49.623017+0000", 2024-09-13T11:23:05.954 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000920914 2024-09-13T11:23:05.954 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:05.954 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:05.954 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:05.955 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:49.667733+0000", 2024-09-13T11:23:05.955 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.044715866 2024-09-13T11:23:05.955 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:05.955 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:05.955 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:23:05.956 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:49.667793+0000", 2024-09-13T11:23:05.956 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.0003999999999997e-05 2024-09-13T11:23:05.956 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:05.956 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:05.956 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:23:05.957 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:49.667865+0000", 2024-09-13T11:23:05.957 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.2502000000000005e-05 2024-09-13T11:23:05.957 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:05.957 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:05.957 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:05.958 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:05.958 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:05.958 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:8893 3.e 3:7d8ed229:::benchmark_data_smithi177_31116_object8892:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-09-13T11:23:05.958 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:22:53.602023+0000", 2024-09-13T11:23:05.958 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 12.333608927, 2024-09-13T11:23:05.958 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.049691079999999999, 2024-09-13T11:23:05.959 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:23:05.959 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:23:05.959 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:05.959 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:23:05.959 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:23:05.960 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:05.960 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 8893 2024-09-13T11:23:05.960 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:05.960 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:23:05.960 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:05.961 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:23:05.961 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:53.602023+0000", 2024-09-13T11:23:05.961 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:05.961 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:05.961 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:05.962 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:23:05.962 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:53.602023+0000", 2024-09-13T11:23:05.962 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:05.962 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:05.962 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:05.963 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:23:05.963 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:53.602026+0000", 2024-09-13T11:23:05.963 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.1229999999999998e-06 2024-09-13T11:23:05.963 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:05.963 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:05.964 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:23:05.964 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:53.602069+0000", 2024-09-13T11:23:05.964 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.3115000000000003e-05 2024-09-13T11:23:05.964 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:05.965 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:05.965 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:23:05.965 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:53.602082+0000", 2024-09-13T11:23:05.965 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.3487e-05 2024-09-13T11:23:05.965 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:05.966 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:05.966 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:23:05.966 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:53.602091+0000", 2024-09-13T11:23:05.966 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.5439999999999999e-06 2024-09-13T11:23:05.966 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:05.966 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:05.967 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:05.967 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:53.602146+0000", 2024-09-13T11:23:05.967 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.5213000000000001e-05 2024-09-13T11:23:05.967 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:05.967 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:05.968 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:23:05.968 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:53.602245+0000", 2024-09-13T11:23:05.968 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.9054000000000004e-05 2024-09-13T11:23:05.968 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:05.968 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:05.969 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 14", 2024-09-13T11:23:05.969 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:53.602382+0000", 2024-09-13T11:23:05.969 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00013682599999999999 2024-09-13T11:23:05.969 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:05.969 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:05.970 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:23:05.970 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:53.603247+0000", 2024-09-13T11:23:05.970 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00086561099999999996 2024-09-13T11:23:05.970 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:05.970 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:05.971 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:05.971 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:53.651548+0000", 2024-09-13T11:23:05.971 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.048300269 2024-09-13T11:23:05.971 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:05.971 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:05.972 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:23:05.972 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:53.651627+0000", 2024-09-13T11:23:05.972 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.9751000000000003e-05 2024-09-13T11:23:05.972 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:05.972 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:05.972 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:23:05.973 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:53.651714+0000", 2024-09-13T11:23:05.973 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.7087000000000002e-05 2024-09-13T11:23:05.973 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:05.973 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:05.973 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:05.974 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:05.974 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:05.974 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop_reply(client.4665.0:8893 3.e e47/30 ondisk, result = 0)", 2024-09-13T11:23:05.974 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:22:53.604561+0000", 2024-09-13T11:23:05.974 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 12.331071457, 2024-09-13T11:23:05.975 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.047164077999999998, 2024-09-13T11:23:05.975 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:23:05.975 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:23:05.975 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:23:05.975 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:23:05.976 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:05.976 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:23:05.976 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:53.604561+0000", 2024-09-13T11:23:05.976 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:05.976 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:05.977 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:05.977 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:23:05.977 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:53.604561+0000", 2024-09-13T11:23:05.977 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:05.977 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:05.978 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:05.978 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:23:05.978 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:53.604562+0000", 2024-09-13T11:23:05.978 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.437e-06 2024-09-13T11:23:05.978 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:05.978 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:05.979 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:23:05.979 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:53.604571+0000", 2024-09-13T11:23:05.979 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.3089999999999996e-06 2024-09-13T11:23:05.979 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:05.979 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:05.980 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:23:05.980 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:53.651421+0000", 2024-09-13T11:23:05.980 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.046850783999999999 2024-09-13T11:23:05.980 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:05.980 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:05.981 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:23:05.981 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:53.651428+0000", 2024-09-13T11:23:05.981 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.4649999999999999e-06 2024-09-13T11:23:05.981 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:05.981 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:05.982 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:05.982 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:53.651528+0000", 2024-09-13T11:23:05.982 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.9784999999999994e-05 2024-09-13T11:23:05.982 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:05.982 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:05.982 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:23:05.983 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:53.651539+0000", 2024-09-13T11:23:05.983 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.1717e-05 2024-09-13T11:23:05.983 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:05.983 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:05.983 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:23:05.984 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:53.651725+0000", 2024-09-13T11:23:05.984 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00018558100000000001 2024-09-13T11:23:05.984 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:05.984 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:05.984 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:05.985 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:05.985 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:05.985 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:9335 3.e 3:705e648d:::benchmark_data_smithi177_31116_object9334:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-09-13T11:23:05.985 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:22:53.933071+0000", 2024-09-13T11:23:05.985 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 12.002560931, 2024-09-13T11:23:05.986 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.04835395, 2024-09-13T11:23:05.986 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:23:05.986 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:23:05.986 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:05.986 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:23:05.987 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:23:05.987 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:05.987 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 9335 2024-09-13T11:23:05.987 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:05.987 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:23:05.987 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:05.988 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:23:05.988 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:53.933071+0000", 2024-09-13T11:23:05.988 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:05.988 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:05.988 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:05.989 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:23:05.989 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:53.933071+0000", 2024-09-13T11:23:05.989 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:05.989 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:05.989 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:05.990 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:23:05.990 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:53.933075+0000", 2024-09-13T11:23:05.990 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.9530000000000001e-06 2024-09-13T11:23:05.990 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:05.990 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:05.990 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:23:05.991 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:53.933140+0000", 2024-09-13T11:23:05.991 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.4839999999999996e-05 2024-09-13T11:23:05.991 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:05.991 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:05.991 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:23:05.992 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:53.933177+0000", 2024-09-13T11:23:05.992 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.7026000000000001e-05 2024-09-13T11:23:05.992 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:05.992 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:05.992 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:23:05.993 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:53.933213+0000", 2024-09-13T11:23:05.993 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.6183000000000002e-05 2024-09-13T11:23:05.993 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:05.993 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:05.993 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:05.994 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:53.933458+0000", 2024-09-13T11:23:05.994 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00024508 2024-09-13T11:23:05.994 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:05.994 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:05.994 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:23:05.995 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:53.933774+0000", 2024-09-13T11:23:05.995 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000315254 2024-09-13T11:23:05.995 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:05.995 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:05.995 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 14", 2024-09-13T11:23:05.996 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:53.933994+0000", 2024-09-13T11:23:05.996 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00022038599999999999 2024-09-13T11:23:05.996 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:05.996 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:05.996 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:23:05.996 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:53.935731+0000", 2024-09-13T11:23:05.997 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.001736448 2024-09-13T11:23:05.997 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:05.997 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:05.997 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:05.997 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:53.981301+0000", 2024-09-13T11:23:05.998 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.045570301000000001 2024-09-13T11:23:05.998 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:05.998 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:05.998 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:23:05.998 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:53.981372+0000", 2024-09-13T11:23:05.998 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.1386999999999999e-05 2024-09-13T11:23:05.999 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:05.999 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:05.999 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:23:05.999 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:53.981425+0000", 2024-09-13T11:23:05.999 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.3091999999999999e-05 2024-09-13T11:23:06.000 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:06.000 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:06.000 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:06.000 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.000 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.001 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop_reply(client.4665.0:9335 3.e e47/30 ondisk, result = 0)", 2024-09-13T11:23:06.001 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:22:53.938016+0000", 2024-09-13T11:23:06.001 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 11.997616306999999, 2024-09-13T11:23:06.001 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.043418678000000002, 2024-09-13T11:23:06.002 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:23:06.002 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:23:06.002 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:23:06.002 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:23:06.002 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.002 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:23:06.003 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:53.938016+0000", 2024-09-13T11:23:06.003 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:06.003 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.003 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.003 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:23:06.004 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:53.938016+0000", 2024-09-13T11:23:06.004 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:06.004 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.004 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.004 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:23:06.004 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:53.938019+0000", 2024-09-13T11:23:06.005 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.1760000000000002e-06 2024-09-13T11:23:06.005 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.005 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.005 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:23:06.005 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:53.938029+0000", 2024-09-13T11:23:06.005 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.8369999999999992e-06 2024-09-13T11:23:06.005 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.005 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.005 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:23:06.006 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:53.981150+0000", 2024-09-13T11:23:06.006 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.043120641000000001 2024-09-13T11:23:06.006 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.006 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.006 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:23:06.006 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:53.981165+0000", 2024-09-13T11:23:06.006 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.4932999999999999e-05 2024-09-13T11:23:06.006 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.006 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.006 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:06.006 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:53.981276+0000", 2024-09-13T11:23:06.006 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00011131199999999999 2024-09-13T11:23:06.006 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.006 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.007 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:23:06.007 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:53.981292+0000", 2024-09-13T11:23:06.007 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.6110999999999999e-05 2024-09-13T11:23:06.007 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.007 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.007 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:23:06.007 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:53.981435+0000", 2024-09-13T11:23:06.007 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00014266800000000001 2024-09-13T11:23:06.007 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:06.007 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:06.007 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:06.007 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.007 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.007 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:9889 3.e 3:711155e6:::benchmark_data_smithi177_31116_object9888:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-09-13T11:23:06.008 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:22:54.281236+0000", 2024-09-13T11:23:06.008 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 11.654395942000001, 2024-09-13T11:23:06.008 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.054312457000000001, 2024-09-13T11:23:06.008 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:23:06.008 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:23:06.008 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:06.008 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:23:06.008 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:23:06.008 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:06.008 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 9889 2024-09-13T11:23:06.008 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.008 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:23:06.008 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.009 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:23:06.009 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:54.281236+0000", 2024-09-13T11:23:06.009 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:06.009 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.009 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.009 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:23:06.009 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:54.281236+0000", 2024-09-13T11:23:06.009 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:06.009 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.009 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.009 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:23:06.009 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:54.281239+0000", 2024-09-13T11:23:06.009 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.2139999999999999e-06 2024-09-13T11:23:06.009 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.010 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.010 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:23:06.010 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:54.281288+0000", 2024-09-13T11:23:06.010 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.9280000000000003e-05 2024-09-13T11:23:06.010 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.010 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.010 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:23:06.010 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:54.281309+0000", 2024-09-13T11:23:06.010 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.1461999999999999e-05 2024-09-13T11:23:06.010 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.010 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.010 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:23:06.010 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:54.281322+0000", 2024-09-13T11:23:06.011 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2557e-05 2024-09-13T11:23:06.011 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.011 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.011 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:06.011 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:54.281442+0000", 2024-09-13T11:23:06.011 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00012048300000000001 2024-09-13T11:23:06.011 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.011 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.011 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:23:06.011 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:54.281640+0000", 2024-09-13T11:23:06.011 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00019767 2024-09-13T11:23:06.011 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.011 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.011 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 14", 2024-09-13T11:23:06.012 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:54.281824+0000", 2024-09-13T11:23:06.012 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000183907 2024-09-13T11:23:06.012 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.012 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.012 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:23:06.012 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:54.283416+0000", 2024-09-13T11:23:06.012 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0015919339999999999 2024-09-13T11:23:06.012 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.012 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.012 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:06.012 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:54.335364+0000", 2024-09-13T11:23:06.012 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.051948558999999998 2024-09-13T11:23:06.012 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.013 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.013 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:23:06.013 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:54.335483+0000", 2024-09-13T11:23:06.013 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000118175 2024-09-13T11:23:06.013 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.013 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.013 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:23:06.013 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:54.335549+0000", 2024-09-13T11:23:06.013 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.6216000000000004e-05 2024-09-13T11:23:06.013 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:06.013 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:06.013 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:06.013 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.013 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.014 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop_reply(client.4665.0:9889 3.e e47/30 ondisk, result = 0)", 2024-09-13T11:23:06.014 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:22:54.284595+0000", 2024-09-13T11:23:06.014 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 11.651037842000001, 2024-09-13T11:23:06.014 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.050965249999999997, 2024-09-13T11:23:06.014 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:23:06.014 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:23:06.014 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:23:06.014 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:23:06.014 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.014 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:23:06.014 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:54.284595+0000", 2024-09-13T11:23:06.014 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:06.014 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.014 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.015 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:23:06.015 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:54.284595+0000", 2024-09-13T11:23:06.015 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:06.015 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.015 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.015 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:23:06.015 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:54.284604+0000", 2024-09-13T11:23:06.015 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.0529999999999996e-06 2024-09-13T11:23:06.015 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.015 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.015 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:23:06.015 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:54.284613+0000", 2024-09-13T11:23:06.015 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.3770000000000002e-06 2024-09-13T11:23:06.016 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.016 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.016 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:23:06.016 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:54.335182+0000", 2024-09-13T11:23:06.016 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.050569127999999998 2024-09-13T11:23:06.016 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.016 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.016 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:23:06.016 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:54.335195+0000", 2024-09-13T11:23:06.016 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.3179999999999999e-05 2024-09-13T11:23:06.016 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.016 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.016 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:06.017 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:54.335339+0000", 2024-09-13T11:23:06.017 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00014390099999999999 2024-09-13T11:23:06.017 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.017 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.017 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:23:06.017 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:54.335355+0000", 2024-09-13T11:23:06.017 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.6356999999999999e-05 2024-09-13T11:23:06.017 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.017 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.017 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:23:06.017 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:54.335560+0000", 2024-09-13T11:23:06.017 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00020425400000000001 2024-09-13T11:23:06.017 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:06.018 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:06.018 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:06.018 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.018 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.018 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:9959 3.e 3:77912156:::benchmark_data_smithi177_31116_object9958:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-09-13T11:23:06.018 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:22:54.356380+0000", 2024-09-13T11:23:06.018 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 11.579252502999999, 2024-09-13T11:23:06.018 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.060511429999999998, 2024-09-13T11:23:06.018 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:23:06.018 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:23:06.018 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:06.018 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:23:06.018 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:23:06.018 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:06.019 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 9959 2024-09-13T11:23:06.019 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.019 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:23:06.019 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.019 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:23:06.019 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:54.356380+0000", 2024-09-13T11:23:06.019 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:06.019 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.019 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.019 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:23:06.019 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:54.356380+0000", 2024-09-13T11:23:06.019 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:06.019 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.020 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.020 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:23:06.020 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:54.356384+0000", 2024-09-13T11:23:06.020 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.7230000000000002e-06 2024-09-13T11:23:06.020 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.020 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.020 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:23:06.020 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:54.356440+0000", 2024-09-13T11:23:06.020 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.6703000000000002e-05 2024-09-13T11:23:06.020 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.020 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.020 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:23:06.020 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:54.356472+0000", 2024-09-13T11:23:06.020 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.1888999999999999e-05 2024-09-13T11:23:06.021 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.021 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.021 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:23:06.021 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:54.356495+0000", 2024-09-13T11:23:06.021 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.3306999999999999e-05 2024-09-13T11:23:06.021 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.021 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.021 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:06.021 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:54.356687+0000", 2024-09-13T11:23:06.021 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00019194399999999999 2024-09-13T11:23:06.021 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.021 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.021 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:23:06.021 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:54.357065+0000", 2024-09-13T11:23:06.022 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00037750899999999999 2024-09-13T11:23:06.022 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.022 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.022 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 14", 2024-09-13T11:23:06.022 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:54.357623+0000", 2024-09-13T11:23:06.022 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00055840700000000002 2024-09-13T11:23:06.022 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.022 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.022 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:23:06.022 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:54.360103+0000", 2024-09-13T11:23:06.022 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0024792989999999999 2024-09-13T11:23:06.022 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.022 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.022 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:06.023 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:54.416745+0000", 2024-09-13T11:23:06.023 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.056641944999999999 2024-09-13T11:23:06.023 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.023 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.023 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:23:06.023 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:54.416828+0000", 2024-09-13T11:23:06.023 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.3275000000000005e-05 2024-09-13T11:23:06.023 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.023 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.023 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:23:06.023 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:54.416891+0000", 2024-09-13T11:23:06.023 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.3429000000000005e-05 2024-09-13T11:23:06.023 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:06.024 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:06.024 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:06.024 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.024 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.024 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop_reply(client.4665.0:9959 3.e e47/30 ondisk, result = 0)", 2024-09-13T11:23:06.024 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:22:54.360272+0000", 2024-09-13T11:23:06.024 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 11.575360294999999, 2024-09-13T11:23:06.024 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.056629291999999998, 2024-09-13T11:23:06.024 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:23:06.024 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:23:06.024 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:23:06.024 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:23:06.024 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.025 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:23:06.025 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:54.360272+0000", 2024-09-13T11:23:06.025 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:06.025 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.025 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.025 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:23:06.025 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:54.360272+0000", 2024-09-13T11:23:06.025 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:06.025 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.025 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.025 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:23:06.025 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:54.360276+0000", 2024-09-13T11:23:06.025 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.5829999999999998e-06 2024-09-13T11:23:06.025 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.026 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.026 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:23:06.026 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:54.360286+0000", 2024-09-13T11:23:06.026 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0434e-05 2024-09-13T11:23:06.026 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.026 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.026 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:23:06.026 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:54.416602+0000", 2024-09-13T11:23:06.026 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.056316135000000003 2024-09-13T11:23:06.026 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.026 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.026 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:23:06.026 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:54.416614+0000", 2024-09-13T11:23:06.027 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2289e-05 2024-09-13T11:23:06.027 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.027 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.027 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:06.027 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:54.416721+0000", 2024-09-13T11:23:06.027 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000106193 2024-09-13T11:23:06.027 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.027 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.027 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:23:06.027 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:54.416736+0000", 2024-09-13T11:23:06.027 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.5216e-05 2024-09-13T11:23:06.027 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.027 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.028 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:23:06.028 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:54.416901+0000", 2024-09-13T11:23:06.028 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00016544199999999999 2024-09-13T11:23:06.028 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:06.028 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:06.028 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:06.028 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.028 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.028 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:9962 3.e 3:7450abb7:::benchmark_data_smithi177_31116_object9961:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e48)", 2024-09-13T11:23:06.028 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:22:54.360795+0000", 2024-09-13T11:23:06.028 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 11.574837788, 2024-09-13T11:23:06.028 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.061794618000000003, 2024-09-13T11:23:06.028 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:23:06.029 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:23:06.029 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:06.029 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:23:06.029 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:23:06.029 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:06.029 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 9962 2024-09-13T11:23:06.029 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.029 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:23:06.029 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.029 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:23:06.029 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:54.360795+0000", 2024-09-13T11:23:06.029 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:06.029 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.029 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.030 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:23:06.030 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:54.360795+0000", 2024-09-13T11:23:06.030 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:06.030 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.030 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.030 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:23:06.030 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:54.360799+0000", 2024-09-13T11:23:06.030 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.6539999999999998e-06 2024-09-13T11:23:06.030 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.030 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.030 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:23:06.030 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:54.360894+0000", 2024-09-13T11:23:06.030 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.4602e-05 2024-09-13T11:23:06.030 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.031 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.031 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:23:06.031 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:54.360932+0000", 2024-09-13T11:23:06.031 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.8299e-05 2024-09-13T11:23:06.031 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.031 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.031 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:23:06.031 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:54.360956+0000", 2024-09-13T11:23:06.031 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.4369999999999999e-05 2024-09-13T11:23:06.031 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.031 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.031 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:06.031 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:54.361062+0000", 2024-09-13T11:23:06.031 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00010568700000000001 2024-09-13T11:23:06.031 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.032 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.032 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op must wait for map", 2024-09-13T11:23:06.032 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:54.361068+0000", 2024-09-13T11:23:06.032 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.9689999999999999e-06 2024-09-13T11:23:06.032 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.032 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.032 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:06.032 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:54.363985+0000", 2024-09-13T11:23:06.032 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0029172820000000002 2024-09-13T11:23:06.032 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.032 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.032 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:23:06.032 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:54.364160+0000", 2024-09-13T11:23:06.033 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00017481800000000001 2024-09-13T11:23:06.033 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.033 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.033 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 14", 2024-09-13T11:23:06.033 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:54.364326+0000", 2024-09-13T11:23:06.033 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00016597199999999999 2024-09-13T11:23:06.033 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.033 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.033 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:23:06.033 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:54.365863+0000", 2024-09-13T11:23:06.033 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0015368599999999999 2024-09-13T11:23:06.033 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.033 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.033 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:06.034 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:54.422496+0000", 2024-09-13T11:23:06.034 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.056632770999999998 2024-09-13T11:23:06.034 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.034 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.034 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:23:06.034 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:54.422544+0000", 2024-09-13T11:23:06.034 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.8000000000000001e-05 2024-09-13T11:23:06.034 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.034 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.034 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:23:06.034 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:54.422589+0000", 2024-09-13T11:23:06.034 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.5334000000000003e-05 2024-09-13T11:23:06.034 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:06.034 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:06.034 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:06.035 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.035 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.035 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop_reply(client.4665.0:9962 3.e e48/30 ondisk, result = 0)", 2024-09-13T11:23:06.035 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:22:54.367440+0000", 2024-09-13T11:23:06.035 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 11.568192681999999, 2024-09-13T11:23:06.035 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.055157683999999998, 2024-09-13T11:23:06.035 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:23:06.035 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:23:06.035 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:23:06.035 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:23:06.035 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.035 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:23:06.035 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:54.367440+0000", 2024-09-13T11:23:06.035 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:06.036 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.036 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.036 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:23:06.036 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:54.367440+0000", 2024-09-13T11:23:06.036 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:06.036 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.036 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.036 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:23:06.036 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:54.367443+0000", 2024-09-13T11:23:06.036 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.0520000000000002e-06 2024-09-13T11:23:06.036 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.036 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.036 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:23:06.036 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:54.367452+0000", 2024-09-13T11:23:06.037 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.0329999999999997e-06 2024-09-13T11:23:06.037 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.037 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.037 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:23:06.037 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:54.422418+0000", 2024-09-13T11:23:06.037 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.054965950999999999 2024-09-13T11:23:06.037 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.037 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.037 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:23:06.037 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:54.422428+0000", 2024-09-13T11:23:06.037 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.8439999999999993e-06 2024-09-13T11:23:06.037 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.037 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.037 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:06.038 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:54.422485+0000", 2024-09-13T11:23:06.038 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.7391e-05 2024-09-13T11:23:06.038 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.038 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.038 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:23:06.038 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:54.422492+0000", 2024-09-13T11:23:06.038 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.6069999999999996e-06 2024-09-13T11:23:06.038 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.038 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.038 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:23:06.038 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:54.422597+0000", 2024-09-13T11:23:06.038 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000105806 2024-09-13T11:23:06.038 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:06.038 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:06.039 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:06.039 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.039 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.039 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:16881 3.e 3:7aa4020e:::benchmark_data_smithi177_31116_object16880:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-13T11:23:06.039 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:22:59.074935+0000", 2024-09-13T11:23:06.039 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 6.8606977269999998, 2024-09-13T11:23:06.039 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.045276227000000002, 2024-09-13T11:23:06.039 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:23:06.039 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:23:06.039 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:06.039 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:23:06.039 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:23:06.039 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:06.039 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 16881 2024-09-13T11:23:06.040 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.040 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:23:06.040 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.040 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:23:06.040 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:59.074935+0000", 2024-09-13T11:23:06.040 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:06.040 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.040 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.040 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:23:06.040 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:59.074935+0000", 2024-09-13T11:23:06.040 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:06.040 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.040 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.040 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:23:06.041 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:59.074937+0000", 2024-09-13T11:23:06.041 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.5739999999999998e-06 2024-09-13T11:23:06.041 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.041 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.041 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:23:06.041 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:59.074988+0000", 2024-09-13T11:23:06.041 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.1119000000000003e-05 2024-09-13T11:23:06.041 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.041 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.041 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:23:06.041 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:59.075016+0000", 2024-09-13T11:23:06.041 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.722e-05 2024-09-13T11:23:06.041 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.041 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.042 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:23:06.042 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:59.075033+0000", 2024-09-13T11:23:06.042 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.7506000000000001e-05 2024-09-13T11:23:06.042 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.042 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.042 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:06.042 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:59.075136+0000", 2024-09-13T11:23:06.042 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000102723 2024-09-13T11:23:06.042 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.042 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.042 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:23:06.042 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:59.075353+0000", 2024-09-13T11:23:06.042 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00021747800000000001 2024-09-13T11:23:06.042 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.042 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.043 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 14", 2024-09-13T11:23:06.043 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:59.075574+0000", 2024-09-13T11:23:06.043 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00022087800000000001 2024-09-13T11:23:06.043 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.043 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.043 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:23:06.043 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:59.078089+0000", 2024-09-13T11:23:06.043 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0025151280000000002 2024-09-13T11:23:06.043 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.043 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.043 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:06.043 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:59.119967+0000", 2024-09-13T11:23:06.043 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.041878253999999997 2024-09-13T11:23:06.043 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.044 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.044 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:23:06.044 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:59.120092+0000", 2024-09-13T11:23:06.044 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00012474799999999999 2024-09-13T11:23:06.044 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.044 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.044 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:23:06.044 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:59.120211+0000", 2024-09-13T11:23:06.044 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00011859899999999999 2024-09-13T11:23:06.044 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:06.044 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:06.044 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:06.044 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.044 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.045 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:17072 3.e 3:7a54cc2f:::benchmark_data_smithi177_31116_object17071:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-13T11:23:06.045 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:22:59.205812+0000", 2024-09-13T11:23:06.045 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 6.7298206690000004, 2024-09-13T11:23:06.045 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.042897009, 2024-09-13T11:23:06.045 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:23:06.045 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:23:06.045 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:06.045 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:23:06.045 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:23:06.045 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:06.045 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 17072 2024-09-13T11:23:06.045 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.045 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:23:06.046 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.046 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:23:06.046 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:59.205812+0000", 2024-09-13T11:23:06.046 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:06.046 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.046 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.046 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:23:06.046 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:59.205812+0000", 2024-09-13T11:23:06.046 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:06.046 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.046 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.046 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:23:06.046 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:59.205815+0000", 2024-09-13T11:23:06.046 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.27e-06 2024-09-13T11:23:06.047 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.047 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.047 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:23:06.047 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:59.205881+0000", 2024-09-13T11:23:06.047 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.5908999999999995e-05 2024-09-13T11:23:06.047 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.047 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.047 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:23:06.047 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:59.205915+0000", 2024-09-13T11:23:06.047 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.4344000000000002e-05 2024-09-13T11:23:06.047 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.047 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.047 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:23:06.047 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:59.205930+0000", 2024-09-13T11:23:06.047 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.4949999999999999e-05 2024-09-13T11:23:06.048 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.048 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.048 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:06.048 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:59.206046+0000", 2024-09-13T11:23:06.048 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000115973 2024-09-13T11:23:06.048 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.048 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.048 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:23:06.048 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:59.206343+0000", 2024-09-13T11:23:06.048 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000297306 2024-09-13T11:23:06.048 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.048 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.048 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 14", 2024-09-13T11:23:06.048 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:59.206701+0000", 2024-09-13T11:23:06.049 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00035758200000000001 2024-09-13T11:23:06.049 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.049 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.049 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:23:06.049 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:59.208591+0000", 2024-09-13T11:23:06.049 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.001890203 2024-09-13T11:23:06.049 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.049 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.049 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:06.049 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:59.248476+0000", 2024-09-13T11:23:06.049 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.039884815999999997 2024-09-13T11:23:06.049 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.049 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.049 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:23:06.050 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:59.248603+0000", 2024-09-13T11:23:06.050 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00012715900000000001 2024-09-13T11:23:06.050 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.050 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.050 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:23:06.050 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:59.248709+0000", 2024-09-13T11:23:06.050 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000105497 2024-09-13T11:23:06.050 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:06.050 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:06.050 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:06.050 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.050 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.051 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:17137 3.e 3:73c6ddc2:::benchmark_data_smithi177_31116_object17136:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-13T11:23:06.051 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:22:59.244270+0000", 2024-09-13T11:23:06.051 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 6.6913624399999998, 2024-09-13T11:23:06.051 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.060039742, 2024-09-13T11:23:06.051 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:23:06.051 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:23:06.051 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:06.051 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:23:06.051 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:23:06.051 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:06.051 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 17137 2024-09-13T11:23:06.051 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.051 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:23:06.052 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.052 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:23:06.052 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:59.244270+0000", 2024-09-13T11:23:06.052 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:06.052 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.052 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.052 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:23:06.052 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:59.244270+0000", 2024-09-13T11:23:06.052 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:06.052 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.052 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.052 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:23:06.052 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:59.244282+0000", 2024-09-13T11:23:06.053 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2225000000000001e-05 2024-09-13T11:23:06.053 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.053 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.053 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:23:06.053 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:59.244356+0000", 2024-09-13T11:23:06.053 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.3837000000000005e-05 2024-09-13T11:23:06.053 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.053 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.053 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:23:06.053 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:59.244396+0000", 2024-09-13T11:23:06.053 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.0494000000000002e-05 2024-09-13T11:23:06.053 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.053 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.053 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:23:06.053 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:59.244410+0000", 2024-09-13T11:23:06.054 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.3403e-05 2024-09-13T11:23:06.054 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.054 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.054 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:06.054 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:59.244575+0000", 2024-09-13T11:23:06.054 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00016559499999999999 2024-09-13T11:23:06.054 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.054 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.054 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:23:06.054 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:59.244814+0000", 2024-09-13T11:23:06.054 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00023877499999999999 2024-09-13T11:23:06.054 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.054 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.054 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 14", 2024-09-13T11:23:06.055 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:59.245036+0000", 2024-09-13T11:23:06.055 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00022129499999999999 2024-09-13T11:23:06.055 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.055 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.055 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:23:06.055 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:59.247349+0000", 2024-09-13T11:23:06.055 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0023137029999999999 2024-09-13T11:23:06.055 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.055 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.055 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:06.055 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:59.304151+0000", 2024-09-13T11:23:06.055 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.056801429000000001 2024-09-13T11:23:06.055 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.055 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.056 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:23:06.056 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:59.304233+0000", 2024-09-13T11:23:06.056 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.2066999999999999e-05 2024-09-13T11:23:06.056 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.056 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.056 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:23:06.056 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:59.304310+0000", 2024-09-13T11:23:06.056 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.6919e-05 2024-09-13T11:23:06.056 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:06.056 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:06.056 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:06.056 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.056 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.056 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop_reply(client.4665.0:17137 3.e e52/30 ondisk, result = 0)", 2024-09-13T11:23:06.057 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:22:59.248231+0000", 2024-09-13T11:23:06.057 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 6.6874013310000002, 2024-09-13T11:23:06.057 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.056092270999999999, 2024-09-13T11:23:06.057 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:23:06.057 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:23:06.057 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:23:06.057 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:23:06.057 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.057 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:23:06.057 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:59.248231+0000", 2024-09-13T11:23:06.057 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:06.057 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.057 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.058 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:23:06.058 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:59.248231+0000", 2024-09-13T11:23:06.058 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:06.058 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.058 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.058 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:23:06.058 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:59.248234+0000", 2024-09-13T11:23:06.058 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.1870000000000001e-06 2024-09-13T11:23:06.058 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.058 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.058 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:23:06.058 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:59.248246+0000", 2024-09-13T11:23:06.058 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.1620000000000001e-05 2024-09-13T11:23:06.058 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.058 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.059 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:23:06.059 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:59.304048+0000", 2024-09-13T11:23:06.059 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.055802233999999999 2024-09-13T11:23:06.059 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.059 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.059 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:23:06.059 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:59.304062+0000", 2024-09-13T11:23:06.059 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.3570000000000001e-05 2024-09-13T11:23:06.059 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.059 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.059 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:06.059 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:59.304138+0000", 2024-09-13T11:23:06.059 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.6159e-05 2024-09-13T11:23:06.059 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.060 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.060 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:23:06.060 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:59.304146+0000", 2024-09-13T11:23:06.060 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.7279999999999995e-06 2024-09-13T11:23:06.060 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.060 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.060 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:23:06.060 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:59.304323+0000", 2024-09-13T11:23:06.060 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00017777299999999999 2024-09-13T11:23:06.060 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:06.060 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:06.060 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:06.060 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.060 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.061 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4665.0:17772 3.7 e53/30 3:efa118de:::benchmark_data_smithi177_31116_object17771:head v 53'1070, mlcod=53'1067)", 2024-09-13T11:23:06.061 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:22:59.605894+0000", 2024-09-13T11:23:06.061 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 6.329738442, 2024-09-13T11:23:06.061 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.088358904000000002, 2024-09-13T11:23:06.061 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:23:06.061 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:23:06.061 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:06.061 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:23:06.061 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.061 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:23:06.061 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:59.605894+0000", 2024-09-13T11:23:06.061 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:06.061 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.061 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.062 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:23:06.062 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:59.605894+0000", 2024-09-13T11:23:06.062 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:06.062 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.062 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.062 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:23:06.062 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:59.605895+0000", 2024-09-13T11:23:06.062 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2589999999999999e-06 2024-09-13T11:23:06.062 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.062 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.062 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:23:06.062 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:59.605947+0000", 2024-09-13T11:23:06.062 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.1788e-05 2024-09-13T11:23:06.062 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.063 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.063 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:23:06.063 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:59.692920+0000", 2024-09-13T11:23:06.063 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.086972663000000006 2024-09-13T11:23:06.063 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.063 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.063 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:23:06.063 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:59.692926+0000", 2024-09-13T11:23:06.063 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.0870000000000003e-06 2024-09-13T11:23:06.063 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.063 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.063 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:06.063 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:59.692979+0000", 2024-09-13T11:23:06.064 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.3060999999999999e-05 2024-09-13T11:23:06.064 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.064 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.064 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:23:06.064 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:59.692993+0000", 2024-09-13T11:23:06.064 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.3753e-05 2024-09-13T11:23:06.064 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.064 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.064 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:23:06.064 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:59.694204+0000", 2024-09-13T11:23:06.064 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.001211571 2024-09-13T11:23:06.064 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.064 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.065 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:23:06.065 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:59.694253+0000", 2024-09-13T11:23:06.065 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.8721999999999999e-05 2024-09-13T11:23:06.065 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:06.065 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:06.065 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:06.065 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.065 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.065 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4665.0:25336 3.7 e55/30 3:e5532806:::benchmark_data_smithi177_31116_object25335:head v 55'1529, mlcod=55'1524)", 2024-09-13T11:23:06.065 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:04.385742+0000", 2024-09-13T11:23:06.065 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 1.549890228, 2024-09-13T11:23:06.065 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.051148206000000002, 2024-09-13T11:23:06.065 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:23:06.065 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:23:06.066 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:06.066 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:23:06.066 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.066 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:23:06.066 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.385742+0000", 2024-09-13T11:23:06.066 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:06.066 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.066 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.066 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:23:06.066 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.385742+0000", 2024-09-13T11:23:06.066 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:06.066 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.066 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.066 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:23:06.067 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.385744+0000", 2024-09-13T11:23:06.067 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.857e-06 2024-09-13T11:23:06.067 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.067 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.067 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:23:06.067 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.385776+0000", 2024-09-13T11:23:06.067 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.2459000000000002e-05 2024-09-13T11:23:06.067 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.067 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.067 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:23:06.067 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.434669+0000", 2024-09-13T11:23:06.067 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.048892113000000001 2024-09-13T11:23:06.067 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.067 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.068 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:23:06.068 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.434684+0000", 2024-09-13T11:23:06.068 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.5596000000000001e-05 2024-09-13T11:23:06.068 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.068 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.068 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:06.068 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.434794+0000", 2024-09-13T11:23:06.068 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000109819 2024-09-13T11:23:06.068 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.068 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.068 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:23:06.068 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.434827+0000", 2024-09-13T11:23:06.068 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.3435000000000001e-05 2024-09-13T11:23:06.068 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.069 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.069 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:23:06.069 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.436840+0000", 2024-09-13T11:23:06.069 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0020130220000000002 2024-09-13T11:23:06.069 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.069 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.069 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:23:06.069 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.436890+0000", 2024-09-13T11:23:06.069 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.9904999999999997e-05 2024-09-13T11:23:06.069 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:06.069 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:06.069 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:06.070 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.070 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.070 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4665.0:25388 3.7 e55/30 3:e5933cef:::benchmark_data_smithi177_31116_object25387:head v 55'1533, mlcod=55'1528)", 2024-09-13T11:23:06.070 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:04.456355+0000", 2024-09-13T11:23:06.070 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 1.479276925, 2024-09-13T11:23:06.070 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.22062032300000001, 2024-09-13T11:23:06.070 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:23:06.070 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:23:06.070 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:06.070 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:23:06.070 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.070 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:23:06.070 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.456355+0000", 2024-09-13T11:23:06.071 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:06.071 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.071 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.071 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:23:06.071 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.456355+0000", 2024-09-13T11:23:06.071 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:06.071 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.071 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.071 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:23:06.071 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.456357+0000", 2024-09-13T11:23:06.071 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.945e-06 2024-09-13T11:23:06.071 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.071 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.071 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:23:06.072 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.456398+0000", 2024-09-13T11:23:06.072 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.0423999999999998e-05 2024-09-13T11:23:06.072 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.072 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.072 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:23:06.072 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.675229+0000", 2024-09-13T11:23:06.072 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.21883077000000001 2024-09-13T11:23:06.072 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.072 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.072 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:23:06.072 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.675240+0000", 2024-09-13T11:23:06.072 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.1817999999999999e-05 2024-09-13T11:23:06.072 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.073 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.073 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:06.073 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.675393+0000", 2024-09-13T11:23:06.073 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000152476 2024-09-13T11:23:06.073 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.073 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.073 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:23:06.073 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.675416+0000", 2024-09-13T11:23:06.073 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.2934000000000001e-05 2024-09-13T11:23:06.073 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.073 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.073 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:23:06.073 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.676931+0000", 2024-09-13T11:23:06.074 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.001514835 2024-09-13T11:23:06.074 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.074 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.074 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:23:06.074 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.676976+0000", 2024-09-13T11:23:06.074 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.5120999999999997e-05 2024-09-13T11:23:06.074 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:06.074 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:06.074 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:06.074 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.074 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.074 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4665.0:25431 3.7 e56/30 3:e2b1f5d7:::benchmark_data_smithi177_31116_object25430:head v 56'1536, mlcod=55'1532)", 2024-09-13T11:23:06.074 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:04.715725+0000", 2024-09-13T11:23:06.074 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 1.219907372, 2024-09-13T11:23:06.074 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.15174399699999999, 2024-09-13T11:23:06.075 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:23:06.075 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:23:06.075 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:06.075 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:23:06.075 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.075 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:23:06.075 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.715725+0000", 2024-09-13T11:23:06.075 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:06.075 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.075 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.075 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:23:06.075 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.715725+0000", 2024-09-13T11:23:06.075 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:06.076 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.076 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.076 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:23:06.076 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.715728+0000", 2024-09-13T11:23:06.076 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.92e-06 2024-09-13T11:23:06.076 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.076 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.076 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:23:06.076 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.715772+0000", 2024-09-13T11:23:06.076 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.4141000000000003e-05 2024-09-13T11:23:06.076 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.076 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.076 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:23:06.076 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.865148+0000", 2024-09-13T11:23:06.077 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.149376012 2024-09-13T11:23:06.077 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.077 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.077 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:23:06.077 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.865163+0000", 2024-09-13T11:23:06.077 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.5106000000000001e-05 2024-09-13T11:23:06.077 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.077 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.077 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:06.077 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.865341+0000", 2024-09-13T11:23:06.077 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000178269 2024-09-13T11:23:06.077 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.077 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.077 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:23:06.077 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.865380+0000", 2024-09-13T11:23:06.078 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.8432000000000001e-05 2024-09-13T11:23:06.078 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.078 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.078 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:23:06.078 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.867412+0000", 2024-09-13T11:23:06.078 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0020325830000000001 2024-09-13T11:23:06.078 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.078 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.078 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:23:06.078 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.867469+0000", 2024-09-13T11:23:06.078 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.6533999999999998e-05 2024-09-13T11:23:06.078 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:06.078 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:06.078 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:06.078 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.079 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.079 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4665.0:25486 3.7 e56/30 3:e7acd6c0:::benchmark_data_smithi177_31116_object25485:head v 56'1537, mlcod=55'1533)", 2024-09-13T11:23:06.079 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:04.865042+0000", 2024-09-13T11:23:06.079 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 1.070590631, 2024-09-13T11:23:06.079 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.060232964, 2024-09-13T11:23:06.079 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:23:06.079 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:23:06.079 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:06.079 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:23:06.079 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.079 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:23:06.079 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.865042+0000", 2024-09-13T11:23:06.079 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:06.079 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.080 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.080 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:23:06.080 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.865042+0000", 2024-09-13T11:23:06.080 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:06.080 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.080 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.080 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:23:06.080 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.865046+0000", 2024-09-13T11:23:06.080 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.1230000000000002e-06 2024-09-13T11:23:06.080 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.080 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.080 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:23:06.080 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.865103+0000", 2024-09-13T11:23:06.080 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.7321999999999998e-05 2024-09-13T11:23:06.081 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.081 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.081 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:23:06.081 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.922874+0000", 2024-09-13T11:23:06.081 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.057771050999999997 2024-09-13T11:23:06.081 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.081 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.081 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:23:06.081 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.922884+0000", 2024-09-13T11:23:06.081 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.6169999999999996e-06 2024-09-13T11:23:06.081 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.081 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.081 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:06.081 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.922979+0000", 2024-09-13T11:23:06.082 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.5058999999999999e-05 2024-09-13T11:23:06.082 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.082 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.082 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:23:06.082 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.923040+0000", 2024-09-13T11:23:06.082 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.1396999999999995e-05 2024-09-13T11:23:06.082 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.082 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.082 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:23:06.082 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.925196+0000", 2024-09-13T11:23:06.082 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.002155952 2024-09-13T11:23:06.082 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.082 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.082 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:23:06.082 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.925275+0000", 2024-09-13T11:23:06.083 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.8442999999999994e-05 2024-09-13T11:23:06.083 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:06.083 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:06.083 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:06.083 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:06.083 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:06.083 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-13T11:23:06.084 INFO:tasks.radosbench.radosbench.0.smithi177.stdout: 39 16 25546 25530 40.9085 1.25 0.00502529 0.0236681 2024-09-13T11:23:06.085 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_ops_in_flight 2024-09-13T11:23:06.086 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-09-13T11:23:06.083+0000 7f97befc9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26315) UID: 0 2024-09-13T11:23:06.186 INFO:tasks.ceph.osd.1.smithi092.stderr:2024-09-13T11:23:06.182+0000 7fe7789c5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33718) UID: 0 2024-09-13T11:23:06.246 INFO:teuthology.orchestra.run.smithi001.stdout:osd.0: {} 2024-09-13T11:23:06.247 INFO:teuthology.orchestra.run.smithi001.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T11:23:06.251 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-13T11:23:06.251 INFO:teuthology.orchestra.run.smithi001.stdout: "ops": [], 2024-09-13T11:23:06.251 INFO:teuthology.orchestra.run.smithi001.stdout: "num_ops": 0 2024-09-13T11:23:06.251 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-13T11:23:06.266 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_blocked_ops 2024-09-13T11:23:06.272 INFO:teuthology.orchestra.run.smithi001.stdout:osd.1: {} 2024-09-13T11:23:06.272 INFO:teuthology.orchestra.run.smithi001.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T11:23:06.286 INFO:tasks.ceph.osd.4.smithi001.stderr:2024-09-13T11:23:06.280+0000 7ff8e0218640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26310) UID: 0 2024-09-13T11:23:06.297 INFO:teuthology.orchestra.run.smithi001.stdout:osd.2: {} 2024-09-13T11:23:06.298 INFO:teuthology.orchestra.run.smithi001.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T11:23:06.323 INFO:teuthology.orchestra.run.smithi001.stdout:osd.3: {} 2024-09-13T11:23:06.324 INFO:teuthology.orchestra.run.smithi001.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T11:23:06.347 INFO:teuthology.orchestra.run.smithi001.stdout:osd.4: {} 2024-09-13T11:23:06.347 INFO:teuthology.orchestra.run.smithi001.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T11:23:06.377 INFO:teuthology.orchestra.run.smithi001.stdout:osd.5: {} 2024-09-13T11:23:06.377 INFO:teuthology.orchestra.run.smithi001.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T11:23:06.387 INFO:tasks.ceph.osd.4.smithi001.stderr:2024-09-13T11:23:06.380+0000 7ff8e0218640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26310) UID: 0 2024-09-13T11:23:06.390 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-13T11:23:06.391 INFO:teuthology.orchestra.run.smithi001.stdout: "ops": [], 2024-09-13T11:23:06.391 INFO:teuthology.orchestra.run.smithi001.stdout: "complaint_time": 30, 2024-09-13T11:23:06.391 INFO:teuthology.orchestra.run.smithi001.stdout: "num_blocked_ops": 0 2024-09-13T11:23:06.391 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-13T11:23:06.405 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_historic_ops 2024-09-13T11:23:06.407 INFO:teuthology.orchestra.run.smithi001.stdout:osd.6: {} 2024-09-13T11:23:06.408 INFO:teuthology.orchestra.run.smithi001.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T11:23:06.433 INFO:teuthology.orchestra.run.smithi001.stdout:osd.7: {} 2024-09-13T11:23:06.433 INFO:teuthology.orchestra.run.smithi001.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T11:23:06.455 INFO:teuthology.orchestra.run.smithi001.stdout:osd.8: {} 2024-09-13T11:23:06.455 INFO:teuthology.orchestra.run.smithi001.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-13T11:23:06.481 INFO:teuthology.orchestra.run.smithi001.stdout:osd.9: {} 2024-09-13T11:23:06.482 INFO:teuthology.orchestra.run.smithi001.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-13T11:23:06.488 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-09-13T11:23:06.483+0000 7f97befc9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26315) UID: 0 2024-09-13T11:23:06.493 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-13T11:23:06.511 INFO:tasks.ceph.osd.5.smithi092.stderr:2024-09-13T11:23:06.506+0000 7f6f00132640 -1 osd.5 57 handle_osd_map injecting map crc failure 2024-09-13T11:23:06.511 INFO:teuthology.orchestra.run.smithi001.stdout:osd.10: {} 2024-09-13T11:23:06.512 INFO:teuthology.orchestra.run.smithi001.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-13T11:23:06.545 INFO:teuthology.orchestra.run.smithi001.stdout:osd.11: {} 2024-09-13T11:23:06.545 INFO:teuthology.orchestra.run.smithi001.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-13T11:23:06.557 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-13T11:23:06.558 INFO:teuthology.orchestra.run.smithi001.stdout: "size": 20, 2024-09-13T11:23:06.558 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 600, 2024-09-13T11:23:06.558 INFO:teuthology.orchestra.run.smithi001.stdout: "ops": [ 2024-09-13T11:23:06.558 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.558 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop_reply(client.4665.0:167 3.9 e41/30 ondisk, result = 0)", 2024-09-13T11:23:06.558 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:22:47.419696+0000", 2024-09-13T11:23:06.558 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 19.134614060000001, 2024-09-13T11:23:06.558 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.64615778199999996, 2024-09-13T11:23:06.558 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:23:06.558 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:23:06.558 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:23:06.558 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:23:06.558 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.559 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:23:06.559 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:47.419696+0000", 2024-09-13T11:23:06.559 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:06.559 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.559 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.559 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:23:06.559 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:47.419696+0000", 2024-09-13T11:23:06.559 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:06.559 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.559 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.559 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:23:06.559 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:47.419697+0000", 2024-09-13T11:23:06.559 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.15e-06 2024-09-13T11:23:06.559 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.560 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.560 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:23:06.560 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:47.419700+0000", 2024-09-13T11:23:06.560 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.2449999999999998e-06 2024-09-13T11:23:06.560 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.560 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.560 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:23:06.560 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:48.065637+0000", 2024-09-13T11:23:06.560 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.64593674300000004 2024-09-13T11:23:06.560 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.560 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.560 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:23:06.560 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:48.065648+0000", 2024-09-13T11:23:06.561 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0865e-05 2024-09-13T11:23:06.561 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.561 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.561 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:06.561 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:48.065720+0000", 2024-09-13T11:23:06.561 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.2020999999999994e-05 2024-09-13T11:23:06.561 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.561 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.561 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:23:06.561 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:48.065728+0000", 2024-09-13T11:23:06.561 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.7660000000000004e-06 2024-09-13T11:23:06.561 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.562 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.562 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:23:06.562 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:48.065853+0000", 2024-09-13T11:23:06.562 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00012499200000000001 2024-09-13T11:23:06.562 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:06.562 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:06.562 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:06.562 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.562 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.562 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:915 3.9 3:9b5a11fd:::benchmark_data_smithi177_31116_object914:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-09-13T11:23:06.562 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:22:49.537855+0000", 2024-09-13T11:23:06.562 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 17.01645499, 2024-09-13T11:23:06.562 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.063882247000000003, 2024-09-13T11:23:06.562 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:23:06.563 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:23:06.563 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:06.563 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:23:06.563 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:23:06.563 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:06.563 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 915 2024-09-13T11:23:06.563 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.563 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:23:06.563 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.563 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:23:06.563 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:49.537855+0000", 2024-09-13T11:23:06.563 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:06.563 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.564 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.564 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:23:06.564 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:49.537855+0000", 2024-09-13T11:23:06.564 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:06.564 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.564 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.564 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:23:06.564 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:49.537856+0000", 2024-09-13T11:23:06.564 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.4589999999999999e-06 2024-09-13T11:23:06.564 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.564 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.564 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:23:06.565 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:49.537884+0000", 2024-09-13T11:23:06.565 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.8203000000000001e-05 2024-09-13T11:23:06.565 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.565 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.565 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:23:06.565 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:49.537895+0000", 2024-09-13T11:23:06.565 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.1035e-05 2024-09-13T11:23:06.565 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.565 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.565 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:23:06.565 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:49.537901+0000", 2024-09-13T11:23:06.565 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.5550000000000002e-06 2024-09-13T11:23:06.566 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.566 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.566 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:06.566 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:49.537966+0000", 2024-09-13T11:23:06.566 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.5303999999999997e-05 2024-09-13T11:23:06.566 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.566 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.566 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:23:06.566 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:49.538122+0000", 2024-09-13T11:23:06.566 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00015589099999999999 2024-09-13T11:23:06.566 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.566 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.566 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 2", 2024-09-13T11:23:06.566 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:49.538227+0000", 2024-09-13T11:23:06.566 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000104875 2024-09-13T11:23:06.567 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.567 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.567 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:23:06.567 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:49.538799+0000", 2024-09-13T11:23:06.567 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00057212799999999998 2024-09-13T11:23:06.567 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.567 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.567 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:06.567 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:49.601616+0000", 2024-09-13T11:23:06.567 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.062817387000000002 2024-09-13T11:23:06.567 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.567 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.567 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:23:06.568 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:49.601684+0000", 2024-09-13T11:23:06.568 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.7348999999999995e-05 2024-09-13T11:23:06.568 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.568 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.568 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:23:06.568 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:49.601737+0000", 2024-09-13T11:23:06.568 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.3060999999999999e-05 2024-09-13T11:23:06.568 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:06.568 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:06.568 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:06.568 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.568 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.568 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:10697 3.0 3:08f06ac6:::benchmark_data_smithi177_31116_object10696:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e48)", 2024-09-13T11:23:06.568 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:22:54.806675+0000", 2024-09-13T11:23:06.569 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 11.747635042000001, 2024-09-13T11:23:06.569 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.102587161, 2024-09-13T11:23:06.569 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:23:06.569 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:23:06.569 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:06.569 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:23:06.569 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:23:06.569 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:06.569 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 10697 2024-09-13T11:23:06.569 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.569 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:23:06.569 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.569 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:23:06.569 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:54.806675+0000", 2024-09-13T11:23:06.569 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:06.570 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.570 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.570 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:23:06.570 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:54.806675+0000", 2024-09-13T11:23:06.570 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:06.570 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.570 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.570 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:23:06.570 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:54.806680+0000", 2024-09-13T11:23:06.570 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.2970000000000003e-06 2024-09-13T11:23:06.570 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.570 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.570 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:23:06.571 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:54.806731+0000", 2024-09-13T11:23:06.571 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.1048999999999999e-05 2024-09-13T11:23:06.571 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.571 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.571 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:23:06.571 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:54.806743+0000", 2024-09-13T11:23:06.571 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2264e-05 2024-09-13T11:23:06.571 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.571 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.571 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:23:06.571 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:54.806752+0000", 2024-09-13T11:23:06.571 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.3259999999999996e-06 2024-09-13T11:23:06.571 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.571 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.571 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:06.572 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:54.806812+0000", 2024-09-13T11:23:06.572 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.0433000000000002e-05 2024-09-13T11:23:06.572 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.572 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.572 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:23:06.572 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:54.806930+0000", 2024-09-13T11:23:06.572 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000117805 2024-09-13T11:23:06.572 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.572 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.572 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 15", 2024-09-13T11:23:06.572 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:54.807042+0000", 2024-09-13T11:23:06.572 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000112528 2024-09-13T11:23:06.573 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.573 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.573 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:23:06.573 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:54.807872+0000", 2024-09-13T11:23:06.573 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00082981600000000004 2024-09-13T11:23:06.573 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.573 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.573 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:06.573 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:54.908964+0000", 2024-09-13T11:23:06.573 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.101092056 2024-09-13T11:23:06.573 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.573 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.573 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:23:06.574 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:54.909137+0000", 2024-09-13T11:23:06.574 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00017322799999999999 2024-09-13T11:23:06.574 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.574 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.574 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:23:06.574 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:54.909262+0000", 2024-09-13T11:23:06.574 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00012435899999999999 2024-09-13T11:23:06.574 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:06.574 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:06.574 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:06.574 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.574 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.574 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop_reply(client.4665.0:10697 3.0 e48/30 ondisk, result = 0)", 2024-09-13T11:23:06.574 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:22:54.808193+0000", 2024-09-13T11:23:06.575 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 11.746117006, 2024-09-13T11:23:06.575 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.10108661100000001, 2024-09-13T11:23:06.575 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:23:06.575 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:23:06.575 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:23:06.575 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:23:06.575 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.575 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:23:06.575 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:54.808193+0000", 2024-09-13T11:23:06.575 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:06.575 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.575 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.575 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:23:06.575 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:54.808193+0000", 2024-09-13T11:23:06.576 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:06.576 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.576 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.576 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:23:06.576 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:54.808194+0000", 2024-09-13T11:23:06.576 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.392e-06 2024-09-13T11:23:06.576 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.576 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.576 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:23:06.576 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:54.808198+0000", 2024-09-13T11:23:06.576 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.9099999999999998e-06 2024-09-13T11:23:06.576 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.576 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.576 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:23:06.577 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:54.908728+0000", 2024-09-13T11:23:06.577 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.100530419 2024-09-13T11:23:06.577 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.577 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.577 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:23:06.577 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:54.908775+0000", 2024-09-13T11:23:06.577 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.6479e-05 2024-09-13T11:23:06.577 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.577 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.577 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:06.577 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:54.908920+0000", 2024-09-13T11:23:06.577 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00014498500000000001 2024-09-13T11:23:06.577 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.577 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.578 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:23:06.578 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:54.908944+0000", 2024-09-13T11:23:06.578 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.4493000000000001e-05 2024-09-13T11:23:06.578 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.578 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.578 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:23:06.578 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:54.909279+0000", 2024-09-13T11:23:06.578 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00033493299999999999 2024-09-13T11:23:06.578 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:06.578 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:06.578 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:06.578 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.578 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.579 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:10715 3.9 3:9813c6ac:::benchmark_data_smithi177_31116_object10714:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e48)", 2024-09-13T11:23:06.579 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:22:54.824503+0000", 2024-09-13T11:23:06.579 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 11.729806537, 2024-09-13T11:23:06.579 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.088715206000000005, 2024-09-13T11:23:06.579 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:23:06.579 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:23:06.579 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:06.579 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:23:06.579 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:23:06.579 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:06.579 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 10715 2024-09-13T11:23:06.579 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.579 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:23:06.580 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.580 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:23:06.580 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:54.824503+0000", 2024-09-13T11:23:06.580 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:06.580 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.580 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.580 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:23:06.580 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:54.824503+0000", 2024-09-13T11:23:06.580 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:06.580 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.580 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.580 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:23:06.581 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:54.824505+0000", 2024-09-13T11:23:06.581 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.967e-06 2024-09-13T11:23:06.581 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.581 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.581 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:23:06.581 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:54.824555+0000", 2024-09-13T11:23:06.581 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.0419e-05 2024-09-13T11:23:06.581 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.581 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.581 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:23:06.581 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:54.824573+0000", 2024-09-13T11:23:06.581 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.7499999999999998e-05 2024-09-13T11:23:06.581 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.581 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.582 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:23:06.582 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:54.824597+0000", 2024-09-13T11:23:06.582 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.4389e-05 2024-09-13T11:23:06.582 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.582 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.582 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:06.582 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:54.824674+0000", 2024-09-13T11:23:06.582 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.6808000000000001e-05 2024-09-13T11:23:06.582 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.582 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.582 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:23:06.582 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:54.824781+0000", 2024-09-13T11:23:06.582 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00010696099999999999 2024-09-13T11:23:06.583 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.583 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.583 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 2", 2024-09-13T11:23:06.583 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:54.824894+0000", 2024-09-13T11:23:06.583 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000112785 2024-09-13T11:23:06.583 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.583 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.583 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:23:06.583 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:54.825714+0000", 2024-09-13T11:23:06.583 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00082003999999999996 2024-09-13T11:23:06.583 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.583 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.584 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:06.584 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:54.913065+0000", 2024-09-13T11:23:06.584 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.087351028999999997 2024-09-13T11:23:06.584 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.584 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.584 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:23:06.584 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:54.913152+0000", 2024-09-13T11:23:06.584 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.7162000000000003e-05 2024-09-13T11:23:06.584 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.584 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.584 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:23:06.584 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:54.913218+0000", 2024-09-13T11:23:06.584 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.6146e-05 2024-09-13T11:23:06.585 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:06.585 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:06.585 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:06.585 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.585 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.585 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop_reply(client.4665.0:10715 3.9 e48/30 ondisk, result = 0)", 2024-09-13T11:23:06.585 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:22:54.827201+0000", 2024-09-13T11:23:06.585 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 11.727108833999999, 2024-09-13T11:23:06.585 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.086028436, 2024-09-13T11:23:06.585 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:23:06.585 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:23:06.585 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:23:06.585 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:23:06.585 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.585 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:23:06.586 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:54.827201+0000", 2024-09-13T11:23:06.586 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:06.586 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.586 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.586 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:23:06.586 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:54.827201+0000", 2024-09-13T11:23:06.586 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:06.586 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.586 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.586 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:23:06.586 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:54.827202+0000", 2024-09-13T11:23:06.586 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.6029999999999999e-06 2024-09-13T11:23:06.586 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.586 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.587 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:23:06.587 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:54.827208+0000", 2024-09-13T11:23:06.587 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.2089999999999996e-06 2024-09-13T11:23:06.587 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.587 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.587 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:23:06.587 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:54.912940+0000", 2024-09-13T11:23:06.587 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.085732357999999995 2024-09-13T11:23:06.587 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.587 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.587 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:23:06.587 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:54.912959+0000", 2024-09-13T11:23:06.587 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.8773000000000001e-05 2024-09-13T11:23:06.588 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.588 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.588 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:06.588 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:54.913046+0000", 2024-09-13T11:23:06.588 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.7522999999999995e-05 2024-09-13T11:23:06.588 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.588 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.588 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:23:06.588 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:54.913057+0000", 2024-09-13T11:23:06.588 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0796e-05 2024-09-13T11:23:06.588 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.588 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.588 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:23:06.588 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:54.913229+0000", 2024-09-13T11:23:06.589 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00017217400000000001 2024-09-13T11:23:06.589 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:06.589 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:06.589 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:06.589 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.589 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.589 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4665.0:10755 3.2 e48/30 3:4d45beba:::benchmark_data_smithi177_31116_object10754:head v 48'674, mlcod=48'670)", 2024-09-13T11:23:06.589 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:22:54.883593+0000", 2024-09-13T11:23:06.589 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 11.670716925000001, 2024-09-13T11:23:06.589 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.071458590000000002, 2024-09-13T11:23:06.589 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:23:06.589 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:23:06.589 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:06.589 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:23:06.589 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.590 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:23:06.590 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:54.883593+0000", 2024-09-13T11:23:06.590 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:06.590 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.590 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.590 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:23:06.590 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:54.883593+0000", 2024-09-13T11:23:06.590 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:06.590 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.590 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.590 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:23:06.590 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:54.883594+0000", 2024-09-13T11:23:06.590 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.6139999999999999e-06 2024-09-13T11:23:06.590 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.591 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.591 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:23:06.591 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:54.883622+0000", 2024-09-13T11:23:06.591 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.8045e-05 2024-09-13T11:23:06.591 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.591 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.591 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:23:06.591 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:54.953147+0000", 2024-09-13T11:23:06.591 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.069524444000000005 2024-09-13T11:23:06.591 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.591 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.591 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:23:06.592 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:54.953160+0000", 2024-09-13T11:23:06.592 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.3216e-05 2024-09-13T11:23:06.592 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.592 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.592 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:06.592 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:54.953287+0000", 2024-09-13T11:23:06.592 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00012703899999999999 2024-09-13T11:23:06.592 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.592 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.592 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:23:06.592 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:54.953320+0000", 2024-09-13T11:23:06.592 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.2653000000000001e-05 2024-09-13T11:23:06.592 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.593 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.593 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:23:06.593 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:54.954947+0000", 2024-09-13T11:23:06.593 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0016272909999999999 2024-09-13T11:23:06.593 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.593 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.593 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:23:06.593 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:54.955051+0000", 2024-09-13T11:23:06.594 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000104288 2024-09-13T11:23:06.594 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:06.594 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:06.594 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:06.594 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.594 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.594 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:10763 3.0 3:04821bc6:::benchmark_data_smithi177_31116_object10762:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e48)", 2024-09-13T11:23:06.594 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:22:54.906425+0000", 2024-09-13T11:23:06.594 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 11.647884345, 2024-09-13T11:23:06.594 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.068511506999999999, 2024-09-13T11:23:06.594 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:23:06.594 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:23:06.594 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:06.594 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:23:06.595 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:23:06.595 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:06.595 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 10763 2024-09-13T11:23:06.595 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.595 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:23:06.595 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.595 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:23:06.595 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:54.906425+0000", 2024-09-13T11:23:06.595 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:06.595 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.595 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.595 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:23:06.595 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:54.906425+0000", 2024-09-13T11:23:06.595 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:06.596 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.596 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.596 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:23:06.596 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:54.906430+0000", 2024-09-13T11:23:06.596 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.4700000000000004e-06 2024-09-13T11:23:06.596 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.596 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.596 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:23:06.596 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:54.906486+0000", 2024-09-13T11:23:06.596 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.5968999999999999e-05 2024-09-13T11:23:06.597 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.597 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.597 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:23:06.597 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:54.906517+0000", 2024-09-13T11:23:06.597 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.0876000000000001e-05 2024-09-13T11:23:06.597 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.597 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.597 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:23:06.597 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:54.906532+0000", 2024-09-13T11:23:06.597 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.5702999999999999e-05 2024-09-13T11:23:06.597 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.598 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.598 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:06.598 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:54.906667+0000", 2024-09-13T11:23:06.598 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00013423299999999999 2024-09-13T11:23:06.598 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.598 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.598 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:23:06.598 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:54.906898+0000", 2024-09-13T11:23:06.598 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00023149299999999999 2024-09-13T11:23:06.598 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.598 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.598 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 15", 2024-09-13T11:23:06.598 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:54.907131+0000", 2024-09-13T11:23:06.599 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000233451 2024-09-13T11:23:06.599 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.599 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.599 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:23:06.599 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:54.909555+0000", 2024-09-13T11:23:06.599 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0024232350000000001 2024-09-13T11:23:06.599 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.599 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.599 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:06.599 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:54.974778+0000", 2024-09-13T11:23:06.599 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.065223549000000006 2024-09-13T11:23:06.599 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.599 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.599 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:23:06.599 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:54.974865+0000", 2024-09-13T11:23:06.600 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.6462e-05 2024-09-13T11:23:06.600 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.600 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.600 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:23:06.600 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:54.974937+0000", 2024-09-13T11:23:06.600 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.2065999999999997e-05 2024-09-13T11:23:06.600 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:06.600 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:06.600 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:06.600 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.600 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.600 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop_reply(client.4665.0:10763 3.0 e48/30 ondisk, result = 0)", 2024-09-13T11:23:06.600 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:22:54.908674+0000", 2024-09-13T11:23:06.601 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 11.645635672000001, 2024-09-13T11:23:06.601 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.066274364000000002, 2024-09-13T11:23:06.601 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:23:06.601 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:23:06.601 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:23:06.601 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:23:06.601 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.601 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:23:06.601 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:54.908674+0000", 2024-09-13T11:23:06.601 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:06.601 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.601 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.601 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:23:06.601 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:54.908674+0000", 2024-09-13T11:23:06.602 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:06.602 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.602 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.602 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:23:06.602 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:54.908678+0000", 2024-09-13T11:23:06.602 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.6770000000000001e-06 2024-09-13T11:23:06.602 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.602 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.602 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:23:06.602 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:54.908690+0000", 2024-09-13T11:23:06.602 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2387e-05 2024-09-13T11:23:06.602 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.602 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.603 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:23:06.603 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:54.974636+0000", 2024-09-13T11:23:06.603 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.065946500000000005 2024-09-13T11:23:06.603 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.603 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.603 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:23:06.603 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:54.974655+0000", 2024-09-13T11:23:06.603 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.8167000000000001e-05 2024-09-13T11:23:06.603 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.603 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.603 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:06.603 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:54.974756+0000", 2024-09-13T11:23:06.603 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00010088499999999999 2024-09-13T11:23:06.603 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.604 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.604 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:23:06.604 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:54.974769+0000", 2024-09-13T11:23:06.604 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.3377000000000001e-05 2024-09-13T11:23:06.604 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.604 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.604 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:23:06.604 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:54.974948+0000", 2024-09-13T11:23:06.604 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000179371 2024-09-13T11:23:06.604 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:06.604 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:06.604 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:06.604 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.605 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.605 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:10980 3.9 3:90a31275:::benchmark_data_smithi177_31116_object10979:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e48)", 2024-09-13T11:23:06.605 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:22:55.120713+0000", 2024-09-13T11:23:06.605 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 11.433596186000001, 2024-09-13T11:23:06.605 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.058755816000000002, 2024-09-13T11:23:06.605 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:23:06.605 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:23:06.605 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:06.605 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:23:06.605 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:23:06.605 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:06.605 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 10980 2024-09-13T11:23:06.605 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.606 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:23:06.606 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.606 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:23:06.606 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:55.120713+0000", 2024-09-13T11:23:06.606 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:06.606 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.606 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.606 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:23:06.606 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:55.120713+0000", 2024-09-13T11:23:06.606 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:06.606 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.606 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.606 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:23:06.606 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:55.120717+0000", 2024-09-13T11:23:06.607 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.8859999999999997e-06 2024-09-13T11:23:06.607 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.607 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.607 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:23:06.607 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:55.120790+0000", 2024-09-13T11:23:06.607 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.2689999999999997e-05 2024-09-13T11:23:06.607 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.607 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.607 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:23:06.607 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:55.120819+0000", 2024-09-13T11:23:06.607 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.9149000000000002e-05 2024-09-13T11:23:06.607 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.607 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.607 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:23:06.608 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:55.120839+0000", 2024-09-13T11:23:06.608 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.014e-05 2024-09-13T11:23:06.608 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.608 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.608 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:06.608 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:55.120955+0000", 2024-09-13T11:23:06.608 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000115466 2024-09-13T11:23:06.608 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.608 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.608 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:23:06.608 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:55.121294+0000", 2024-09-13T11:23:06.608 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00033957300000000002 2024-09-13T11:23:06.608 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.608 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.609 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 2", 2024-09-13T11:23:06.609 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:55.121669+0000", 2024-09-13T11:23:06.609 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00037502199999999997 2024-09-13T11:23:06.609 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.609 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.609 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:23:06.609 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:55.123503+0000", 2024-09-13T11:23:06.609 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.001833777 2024-09-13T11:23:06.609 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.609 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.609 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:06.609 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:55.179294+0000", 2024-09-13T11:23:06.609 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.055791111999999997 2024-09-13T11:23:06.609 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.609 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.610 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:23:06.610 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:55.179389+0000", 2024-09-13T11:23:06.610 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.4732e-05 2024-09-13T11:23:06.610 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.610 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.610 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:23:06.610 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:55.179469+0000", 2024-09-13T11:23:06.610 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.0269e-05 2024-09-13T11:23:06.610 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:06.610 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:06.610 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:06.610 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.610 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.610 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4665.0:11121 3.2 e48/30 3:4b00b743:::benchmark_data_smithi177_31116_object11120:head v 48'693, mlcod=48'690)", 2024-09-13T11:23:06.610 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:22:55.229888+0000", 2024-09-13T11:23:06.611 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 11.324421172999999, 2024-09-13T11:23:06.611 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.069340866000000001, 2024-09-13T11:23:06.611 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:23:06.611 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:23:06.611 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:06.611 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:23:06.611 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.611 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:23:06.611 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:55.229888+0000", 2024-09-13T11:23:06.611 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:06.611 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.611 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.611 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:23:06.611 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:55.229888+0000", 2024-09-13T11:23:06.611 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:06.612 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.612 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.612 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:23:06.612 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:55.229891+0000", 2024-09-13T11:23:06.612 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.5600000000000001e-06 2024-09-13T11:23:06.612 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.612 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.612 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:23:06.612 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:55.229972+0000", 2024-09-13T11:23:06.612 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.1138999999999997e-05 2024-09-13T11:23:06.612 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.612 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.612 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:23:06.612 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:55.297933+0000", 2024-09-13T11:23:06.612 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.067960975000000007 2024-09-13T11:23:06.613 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.613 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.613 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:23:06.613 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:55.297962+0000", 2024-09-13T11:23:06.613 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.8736999999999999e-05 2024-09-13T11:23:06.613 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.613 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.613 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:06.613 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:55.298077+0000", 2024-09-13T11:23:06.613 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000115359 2024-09-13T11:23:06.613 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.613 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.613 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:23:06.614 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:55.298095+0000", 2024-09-13T11:23:06.614 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.8148999999999999e-05 2024-09-13T11:23:06.614 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.614 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.614 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:23:06.614 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:55.299190+0000", 2024-09-13T11:23:06.614 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.001094294 2024-09-13T11:23:06.614 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.614 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.614 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:23:06.614 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:55.299229+0000", 2024-09-13T11:23:06.614 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.9653000000000001e-05 2024-09-13T11:23:06.614 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:06.615 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:06.615 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:06.615 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.615 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.615 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:16546 3.0 3:0fd86ef0:::benchmark_data_smithi177_31116_object16545:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-13T11:23:06.615 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:22:58.765077+0000", 2024-09-13T11:23:06.615 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 7.7892329480000004, 2024-09-13T11:23:06.615 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.058105758, 2024-09-13T11:23:06.615 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:23:06.615 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:23:06.615 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:06.615 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:23:06.615 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:23:06.616 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:06.616 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 16546 2024-09-13T11:23:06.616 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.616 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:23:06.616 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.616 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:23:06.616 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:58.765077+0000", 2024-09-13T11:23:06.616 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:06.616 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.616 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.616 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:23:06.616 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:58.765077+0000", 2024-09-13T11:23:06.616 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:06.617 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.617 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.617 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:23:06.617 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:58.765078+0000", 2024-09-13T11:23:06.617 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.192e-06 2024-09-13T11:23:06.617 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.617 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.617 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:23:06.617 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:58.765148+0000", 2024-09-13T11:23:06.617 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.0061999999999994e-05 2024-09-13T11:23:06.617 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.617 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.617 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:23:06.618 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:58.765159+0000", 2024-09-13T11:23:06.618 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0688e-05 2024-09-13T11:23:06.618 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.618 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.618 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:23:06.618 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:58.765166+0000", 2024-09-13T11:23:06.618 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.7149999999999998e-06 2024-09-13T11:23:06.618 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.618 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.618 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:06.618 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:58.765211+0000", 2024-09-13T11:23:06.618 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.4703999999999998e-05 2024-09-13T11:23:06.618 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.619 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.619 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:23:06.619 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:58.765291+0000", 2024-09-13T11:23:06.619 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.9745000000000003e-05 2024-09-13T11:23:06.619 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.619 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.619 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 15", 2024-09-13T11:23:06.619 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:58.765371+0000", 2024-09-13T11:23:06.619 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.0332000000000003e-05 2024-09-13T11:23:06.619 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.619 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.619 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:23:06.619 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:58.766168+0000", 2024-09-13T11:23:06.619 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00079666399999999999 2024-09-13T11:23:06.620 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.620 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.620 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:06.620 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:58.823069+0000", 2024-09-13T11:23:06.620 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.056900918000000002 2024-09-13T11:23:06.620 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.620 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.620 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:23:06.620 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:58.823124+0000", 2024-09-13T11:23:06.620 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.516e-05 2024-09-13T11:23:06.620 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.620 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.620 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:23:06.621 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:58.823182+0000", 2024-09-13T11:23:06.621 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.8578000000000001e-05 2024-09-13T11:23:06.621 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:06.621 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:06.621 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:06.621 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.621 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.621 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:16657 3.0 3:0b771969:::benchmark_data_smithi177_31116_object16656:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-13T11:23:06.621 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:22:58.893528+0000", 2024-09-13T11:23:06.621 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 7.6607814359999997, 2024-09-13T11:23:06.621 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.064654485999999997, 2024-09-13T11:23:06.621 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:23:06.621 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:23:06.622 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:06.622 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:23:06.622 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:23:06.622 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:06.622 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 16657 2024-09-13T11:23:06.622 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.622 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:23:06.622 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.622 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:23:06.622 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:58.893528+0000", 2024-09-13T11:23:06.622 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:06.622 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.623 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.623 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:23:06.623 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:58.893528+0000", 2024-09-13T11:23:06.623 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:06.623 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.623 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.623 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:23:06.623 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:58.893529+0000", 2024-09-13T11:23:06.623 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.3239999999999999e-06 2024-09-13T11:23:06.623 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.623 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.623 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:23:06.623 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:58.893570+0000", 2024-09-13T11:23:06.624 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.0193e-05 2024-09-13T11:23:06.624 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.624 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.624 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:23:06.624 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:58.893580+0000", 2024-09-13T11:23:06.624 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.9590000000000007e-06 2024-09-13T11:23:06.624 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.624 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.624 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:23:06.624 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:58.893585+0000", 2024-09-13T11:23:06.624 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.0289999999999996e-06 2024-09-13T11:23:06.624 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.624 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.625 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:06.625 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:58.893704+0000", 2024-09-13T11:23:06.625 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00011903 2024-09-13T11:23:06.625 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.625 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.625 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:23:06.625 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:58.893872+0000", 2024-09-13T11:23:06.625 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00016827500000000001 2024-09-13T11:23:06.625 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.625 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.625 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 15", 2024-09-13T11:23:06.625 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:58.894032+0000", 2024-09-13T11:23:06.625 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00015952100000000001 2024-09-13T11:23:06.626 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.626 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.626 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:23:06.626 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:58.895575+0000", 2024-09-13T11:23:06.626 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.001543623 2024-09-13T11:23:06.626 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.626 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.626 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:06.626 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:58.957980+0000", 2024-09-13T11:23:06.626 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.062405108000000001 2024-09-13T11:23:06.626 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.626 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.626 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:23:06.627 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:58.958066+0000", 2024-09-13T11:23:06.627 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.5861999999999999e-05 2024-09-13T11:23:06.627 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.627 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.627 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:23:06.627 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:58.958183+0000", 2024-09-13T11:23:06.627 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000116562 2024-09-13T11:23:06.627 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:06.627 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:06.627 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:06.627 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.627 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.627 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop_reply(client.4665.0:16657 3.0 e52/30 ondisk, result = 0)", 2024-09-13T11:23:06.627 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:22:58.896836+0000", 2024-09-13T11:23:06.628 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 7.6574732159999996, 2024-09-13T11:23:06.628 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.061360296000000002, 2024-09-13T11:23:06.628 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:23:06.628 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:23:06.628 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:23:06.628 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:23:06.628 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.628 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:23:06.628 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:58.896836+0000", 2024-09-13T11:23:06.628 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:06.628 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.628 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.628 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:23:06.629 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:58.896836+0000", 2024-09-13T11:23:06.629 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:06.629 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.629 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.629 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:23:06.629 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:58.896839+0000", 2024-09-13T11:23:06.629 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.4030000000000001e-06 2024-09-13T11:23:06.629 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.629 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.629 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:23:06.629 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:58.896847+0000", 2024-09-13T11:23:06.629 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.9070000000000007e-06 2024-09-13T11:23:06.629 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.630 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.630 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:23:06.630 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:58.957814+0000", 2024-09-13T11:23:06.630 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.060967649999999998 2024-09-13T11:23:06.630 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.630 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.630 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:23:06.630 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:58.957827+0000", 2024-09-13T11:23:06.630 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2483999999999999e-05 2024-09-13T11:23:06.630 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.630 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.630 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:06.630 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:58.957956+0000", 2024-09-13T11:23:06.631 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000129531 2024-09-13T11:23:06.631 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.631 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.631 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:23:06.631 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:58.957971+0000", 2024-09-13T11:23:06.631 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.5007e-05 2024-09-13T11:23:06.631 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.631 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.631 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:23:06.631 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:58.958197+0000", 2024-09-13T11:23:06.631 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000225314 2024-09-13T11:23:06.631 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:06.631 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:06.632 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:06.632 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.632 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.632 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:17634 3.9 3:9e121bd5:::benchmark_data_smithi177_31116_object17633:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-13T11:23:06.632 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:22:59.522129+0000", 2024-09-13T11:23:06.632 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 7.0321807060000001, 2024-09-13T11:23:06.632 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.076237751000000006, 2024-09-13T11:23:06.632 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:23:06.632 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:23:06.632 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:06.632 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:23:06.632 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:23:06.632 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:06.632 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 17634 2024-09-13T11:23:06.633 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.633 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:23:06.633 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.633 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:23:06.633 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:59.522129+0000", 2024-09-13T11:23:06.633 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:06.633 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.633 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.633 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:23:06.633 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:59.522129+0000", 2024-09-13T11:23:06.633 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:06.633 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.633 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.634 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:23:06.634 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:59.522130+0000", 2024-09-13T11:23:06.634 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.5379999999999999e-06 2024-09-13T11:23:06.634 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.634 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.634 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:23:06.634 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:59.522213+0000", 2024-09-13T11:23:06.634 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.2256000000000007e-05 2024-09-13T11:23:06.634 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.634 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.634 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:23:06.634 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:59.522223+0000", 2024-09-13T11:23:06.634 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0564000000000001e-05 2024-09-13T11:23:06.634 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.635 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.635 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:23:06.635 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:59.522231+0000", 2024-09-13T11:23:06.635 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.5680000000000002e-06 2024-09-13T11:23:06.635 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.635 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.635 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:06.635 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:59.522279+0000", 2024-09-13T11:23:06.635 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.8127999999999997e-05 2024-09-13T11:23:06.635 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.635 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.635 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:23:06.635 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:59.522376+0000", 2024-09-13T11:23:06.635 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.6873000000000006e-05 2024-09-13T11:23:06.636 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.636 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.636 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 2", 2024-09-13T11:23:06.636 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:59.522467+0000", 2024-09-13T11:23:06.636 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.0816999999999995e-05 2024-09-13T11:23:06.636 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.636 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.636 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:23:06.636 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:59.523626+0000", 2024-09-13T11:23:06.636 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.001159263 2024-09-13T11:23:06.636 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.636 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.636 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:06.636 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:59.598309+0000", 2024-09-13T11:23:06.637 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.074683278000000006 2024-09-13T11:23:06.637 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.637 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.637 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:23:06.637 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:59.598342+0000", 2024-09-13T11:23:06.637 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.2512999999999999e-05 2024-09-13T11:23:06.637 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.637 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.637 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:23:06.637 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:59.598367+0000", 2024-09-13T11:23:06.637 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.4953e-05 2024-09-13T11:23:06.637 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:06.637 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:06.637 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:06.638 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.638 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.638 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop_reply(client.4665.0:17634 3.9 e53/30 ondisk, result = 0)", 2024-09-13T11:23:06.638 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:22:59.524210+0000", 2024-09-13T11:23:06.638 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 7.0300996649999998, 2024-09-13T11:23:06.638 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.074161042999999996, 2024-09-13T11:23:06.638 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:23:06.638 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:23:06.638 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:23:06.638 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:23:06.638 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.638 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:23:06.639 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:59.524210+0000", 2024-09-13T11:23:06.639 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:06.639 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.639 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.639 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:23:06.639 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:59.524210+0000", 2024-09-13T11:23:06.639 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:06.639 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.639 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.639 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:23:06.639 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:59.524211+0000", 2024-09-13T11:23:06.639 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.158e-06 2024-09-13T11:23:06.639 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.639 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.639 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:23:06.640 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:59.524215+0000", 2024-09-13T11:23:06.640 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.5750000000000001e-06 2024-09-13T11:23:06.640 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.640 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.640 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:23:06.640 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:59.598261+0000", 2024-09-13T11:23:06.640 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.074046325999999996 2024-09-13T11:23:06.640 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.640 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.640 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:23:06.640 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:59.598266+0000", 2024-09-13T11:23:06.640 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.1420000000000001e-06 2024-09-13T11:23:06.640 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.641 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.641 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:06.641 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:59.598301+0000", 2024-09-13T11:23:06.641 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.5002000000000001e-05 2024-09-13T11:23:06.641 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.641 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.641 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:23:06.641 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:59.598306+0000", 2024-09-13T11:23:06.641 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.9239999999999998e-06 2024-09-13T11:23:06.641 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.641 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.641 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:23:06.641 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:59.598371+0000", 2024-09-13T11:23:06.641 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.4916e-05 2024-09-13T11:23:06.642 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:06.642 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:06.642 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:06.642 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.642 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.642 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4665.0:25290 3.4 e55/40 3:2c6b3a0e:::benchmark_data_smithi177_31116_object25289:head v 55'1638, mlcod=55'1635)", 2024-09-13T11:23:06.642 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:04.309280+0000", 2024-09-13T11:23:06.642 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 2.245029331, 2024-09-13T11:23:06.642 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.094030580000000002, 2024-09-13T11:23:06.642 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:23:06.642 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:23:06.642 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:06.642 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:23:06.643 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.643 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:23:06.643 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.309280+0000", 2024-09-13T11:23:06.643 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:06.643 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.643 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.643 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:23:06.643 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.309280+0000", 2024-09-13T11:23:06.643 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:06.643 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.643 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.643 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:23:06.643 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.309283+0000", 2024-09-13T11:23:06.643 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.7800000000000001e-06 2024-09-13T11:23:06.644 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.644 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.644 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:23:06.644 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.309334+0000", 2024-09-13T11:23:06.644 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.0510000000000003e-05 2024-09-13T11:23:06.644 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.644 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.644 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:23:06.644 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.400659+0000", 2024-09-13T11:23:06.644 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.091325274999999997 2024-09-13T11:23:06.644 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.644 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.644 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:23:06.645 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.400676+0000", 2024-09-13T11:23:06.645 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.7537000000000002e-05 2024-09-13T11:23:06.645 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.645 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.645 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:06.645 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.400785+0000", 2024-09-13T11:23:06.645 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000108756 2024-09-13T11:23:06.645 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.645 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.645 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:23:06.645 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.400818+0000", 2024-09-13T11:23:06.645 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.2635999999999997e-05 2024-09-13T11:23:06.645 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.646 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.646 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:23:06.646 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.403208+0000", 2024-09-13T11:23:06.646 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0023897900000000001 2024-09-13T11:23:06.646 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.646 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.646 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:23:06.646 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.403311+0000", 2024-09-13T11:23:06.646 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00010329599999999999 2024-09-13T11:23:06.646 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:06.646 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:06.646 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:06.646 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.646 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.647 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:25383 3.0 3:0a6bb869:::benchmark_data_smithi177_31116_object25382:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e55)", 2024-09-13T11:23:06.647 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:04.447376+0000", 2024-09-13T11:23:06.647 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 2.1069333910000001, 2024-09-13T11:23:06.647 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.21239864, 2024-09-13T11:23:06.647 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:23:06.647 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:23:06.647 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:06.647 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:23:06.647 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:23:06.647 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:06.647 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 25383 2024-09-13T11:23:06.647 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.647 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:23:06.647 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.647 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:23:06.648 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.447376+0000", 2024-09-13T11:23:06.648 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:06.648 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.648 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.648 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:23:06.648 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.447376+0000", 2024-09-13T11:23:06.648 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:06.648 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.648 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.648 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:23:06.648 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.447378+0000", 2024-09-13T11:23:06.648 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.6729999999999999e-06 2024-09-13T11:23:06.648 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.649 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.649 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:23:06.649 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.447433+0000", 2024-09-13T11:23:06.649 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.5371000000000002e-05 2024-09-13T11:23:06.649 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.649 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.649 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:23:06.649 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.447446+0000", 2024-09-13T11:23:06.649 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.3179999999999999e-05 2024-09-13T11:23:06.649 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.649 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.649 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:23:06.649 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.447456+0000", 2024-09-13T11:23:06.649 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0008e-05 2024-09-13T11:23:06.650 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.650 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.650 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:06.650 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.447514+0000", 2024-09-13T11:23:06.650 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.8023999999999999e-05 2024-09-13T11:23:06.650 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.650 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.650 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:23:06.650 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.447642+0000", 2024-09-13T11:23:06.650 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000127296 2024-09-13T11:23:06.650 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.650 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.650 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 15", 2024-09-13T11:23:06.650 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.447767+0000", 2024-09-13T11:23:06.650 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000125547 2024-09-13T11:23:06.651 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.651 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.651 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:23:06.651 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.449188+0000", 2024-09-13T11:23:06.651 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.001420604 2024-09-13T11:23:06.651 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.651 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.651 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:06.651 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.659668+0000", 2024-09-13T11:23:06.651 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.210479685 2024-09-13T11:23:06.651 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.651 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.651 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:23:06.652 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.659716+0000", 2024-09-13T11:23:06.652 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.8609999999999997e-05 2024-09-13T11:23:06.652 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.652 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.652 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:23:06.652 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.659775+0000", 2024-09-13T11:23:06.652 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.8641999999999999e-05 2024-09-13T11:23:06.652 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:06.652 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:06.652 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:06.652 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.652 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.652 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4665.0:25408 3.4 e56/40 3:294e23bd:::benchmark_data_smithi177_31116_object25407:head v 56'1645, mlcod=55'1642)", 2024-09-13T11:23:06.652 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:04.494686+0000", 2024-09-13T11:23:06.653 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 2.0596237319999999, 2024-09-13T11:23:06.653 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.29725566399999998, 2024-09-13T11:23:06.653 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:23:06.653 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:23:06.653 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:06.653 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:23:06.653 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.653 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:23:06.653 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.494686+0000", 2024-09-13T11:23:06.653 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:06.653 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.653 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.653 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:23:06.653 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.494686+0000", 2024-09-13T11:23:06.654 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:06.654 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.654 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.654 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:23:06.654 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.494686+0000", 2024-09-13T11:23:06.654 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.2399999999999999e-07 2024-09-13T11:23:06.654 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.654 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.654 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:23:06.654 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.494809+0000", 2024-09-13T11:23:06.654 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00012308600000000001 2024-09-13T11:23:06.654 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.654 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.654 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:23:06.655 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.789410+0000", 2024-09-13T11:23:06.655 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.29460066899999998 2024-09-13T11:23:06.655 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.655 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.655 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:23:06.655 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.789425+0000", 2024-09-13T11:23:06.655 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.4600000000000001e-05 2024-09-13T11:23:06.655 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.655 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.655 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:06.655 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.789507+0000", 2024-09-13T11:23:06.655 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.2785000000000001e-05 2024-09-13T11:23:06.655 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.655 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.656 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:23:06.656 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.789525+0000", 2024-09-13T11:23:06.656 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.8117999999999998e-05 2024-09-13T11:23:06.656 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.656 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.656 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:23:06.656 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.791852+0000", 2024-09-13T11:23:06.656 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0023264459999999998 2024-09-13T11:23:06.656 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.656 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.656 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:23:06.656 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.791942+0000", 2024-09-13T11:23:06.656 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.9635999999999994e-05 2024-09-13T11:23:06.656 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:06.657 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:06.657 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:06.657 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.657 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.657 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:25420 3.0 3:03f6c3d6:::benchmark_data_smithi177_31116_object25419:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e56)", 2024-09-13T11:23:06.657 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:04.655682+0000", 2024-09-13T11:23:06.657 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 1.898627997, 2024-09-13T11:23:06.657 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.074957139000000006, 2024-09-13T11:23:06.657 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:23:06.657 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:23:06.657 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:06.657 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:23:06.657 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:23:06.657 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:06.658 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 25420 2024-09-13T11:23:06.658 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.658 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:23:06.658 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.658 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:23:06.658 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.655682+0000", 2024-09-13T11:23:06.658 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:06.658 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.658 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.658 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:23:06.658 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.655682+0000", 2024-09-13T11:23:06.658 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:06.658 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.658 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.658 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:23:06.659 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.655685+0000", 2024-09-13T11:23:06.659 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.653e-06 2024-09-13T11:23:06.659 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.659 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.659 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:23:06.659 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.655744+0000", 2024-09-13T11:23:06.659 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.8508999999999998e-05 2024-09-13T11:23:06.659 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.659 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.659 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:23:06.659 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.655773+0000", 2024-09-13T11:23:06.659 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.9357e-05 2024-09-13T11:23:06.659 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.659 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.660 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:23:06.660 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.655787+0000", 2024-09-13T11:23:06.660 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.4263e-05 2024-09-13T11:23:06.660 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.660 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.660 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:06.660 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.655923+0000", 2024-09-13T11:23:06.660 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00013515200000000001 2024-09-13T11:23:06.660 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.660 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.660 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:23:06.660 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.656163+0000", 2024-09-13T11:23:06.660 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000240268 2024-09-13T11:23:06.660 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.661 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.661 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 15", 2024-09-13T11:23:06.661 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.656393+0000", 2024-09-13T11:23:06.661 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00022986799999999999 2024-09-13T11:23:06.661 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.661 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.661 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:23:06.661 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.658693+0000", 2024-09-13T11:23:06.661 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.002300074 2024-09-13T11:23:06.661 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.661 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.661 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:06.661 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.730425+0000", 2024-09-13T11:23:06.661 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.071732196999999998 2024-09-13T11:23:06.661 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.662 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.662 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:23:06.662 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.730528+0000", 2024-09-13T11:23:06.662 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000103132 2024-09-13T11:23:06.662 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:06.662 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:06.662 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:23:06.662 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.730639+0000", 2024-09-13T11:23:06.662 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000110666 2024-09-13T11:23:06.662 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:06.662 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:06.662 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:06.662 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:06.662 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:06.662 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-13T11:23:06.663 INFO:teuthology.orchestra.run.smithi001.stdout:osd.12: {} 2024-09-13T11:23:06.664 INFO:teuthology.orchestra.run.smithi001.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-13T11:23:06.665 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-13T11:23:06.665 INFO:teuthology.orchestra.run.smithi001.stdout:osd.13: {} 2024-09-13T11:23:06.665 INFO:teuthology.orchestra.run.smithi001.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-13T11:23:06.666 INFO:teuthology.orchestra.run.smithi001.stdout:osd.14: {} 2024-09-13T11:23:06.666 INFO:teuthology.orchestra.run.smithi001.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-13T11:23:06.666 INFO:teuthology.orchestra.run.smithi001.stdout:osd.15: {} 2024-09-13T11:23:06.667 INFO:teuthology.orchestra.run.smithi001.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-13T11:23:06.667 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-09-13T11:23:06.659+0000 7f97befc9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26315) UID: 0 2024-09-13T11:23:06.764 INFO:tasks.ceph.osd.15.smithi177.stderr:2024-09-13T11:23:06.763+0000 7f9d6ce77640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26578) UID: 0 2024-09-13T11:23:06.829 INFO:teuthology.orchestra.run.smithi092.stdout:ERROR: (22) Invalid argument 2024-09-13T11:23:06.829 INFO:teuthology.orchestra.run.smithi092.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:23:06.851 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-13T11:23:06.866 INFO:tasks.ceph.osd.10.smithi162.stderr:2024-09-13T11:23:06.863+0000 7fb59c9f0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26317) UID: 0 2024-09-13T11:23:06.966 INFO:tasks.ceph.osd.14.smithi162.stderr:2024-09-13T11:23:06.963+0000 7f235c0db640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26319) UID: 0 2024-09-13T11:23:07.001 INFO:tasks.radosbench.radosbench.0.smithi177.stdout:2024-09-13T11:23:07.002397+0000 min lat: 0.00139681 max lat: 22.1835 avg lat: 0.0242834 2024-09-13T11:23:07.001 INFO:tasks.radosbench.radosbench.0.smithi177.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-13T11:23:07.001 INFO:tasks.radosbench.radosbench.0.smithi177.stdout: 40 16 26367 26351 41.1685 51.3125 0.0158797 0.0242834 2024-09-13T11:23:07.002 INFO:teuthology.orchestra.run.smithi092.stdout:ERROR: (22) Invalid argument 2024-09-13T11:23:07.002 INFO:teuthology.orchestra.run.smithi092.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:23:07.024 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-13T11:23:07.067 INFO:tasks.ceph.osd.14.smithi162.stderr:2024-09-13T11:23:07.063+0000 7f235c0db640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26319) UID: 0 2024-09-13T11:23:07.168 INFO:tasks.ceph.osd.1.smithi092.stderr:2024-09-13T11:23:07.166+0000 7fe7789c5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33718) UID: 0 2024-09-13T11:23:07.179 INFO:teuthology.orchestra.run.smithi092.stdout:ERROR: (22) Invalid argument 2024-09-13T11:23:07.179 INFO:teuthology.orchestra.run.smithi092.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:23:07.201 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-09-13T11:23:07.269 INFO:tasks.ceph.osd.15.smithi177.stderr:2024-09-13T11:23:07.266+0000 7f9d6ce77640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26578) UID: 0 2024-09-13T11:23:07.370 INFO:tasks.ceph.osd.1.smithi092.stderr:2024-09-13T11:23:07.366+0000 7fe7789c5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33718) UID: 0 2024-09-13T11:23:07.397 INFO:teuthology.orchestra.run.smithi092.stdout:ERROR: (22) Invalid argument 2024-09-13T11:23:07.397 INFO:teuthology.orchestra.run.smithi092.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:23:07.418 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-09-13T11:23:07.470 INFO:tasks.ceph.osd.11.smithi177.stderr:2024-09-13T11:23:07.466+0000 7f24792c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26580) UID: 0 2024-09-13T11:23:07.503 INFO:tasks.ceph.osd.7.smithi177.stderr:2024-09-13T11:23:07.502+0000 7f8971f4e640 -1 osd.7 58 handle_osd_map injecting map crc failure 2024-09-13T11:23:07.506 INFO:teuthology.orchestra.run.smithi001.stderr:noscrub is unset 2024-09-13T11:23:07.510 INFO:tasks.ceph.osd.12.smithi001.stderr:2024-09-13T11:23:07.504+0000 7f891d10b640 -1 osd.12 58 handle_osd_map injecting map crc failure 2024-09-13T11:23:07.571 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-13T11:23:07.564+0000 7f6a68775640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26307) UID: 0 2024-09-13T11:23:07.617 INFO:teuthology.orchestra.run.smithi092.stdout:ERROR: (22) Invalid argument 2024-09-13T11:23:07.618 INFO:teuthology.orchestra.run.smithi092.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:23:07.639 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-09-13T11:23:07.672 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-13T11:23:07.668+0000 7f6a68775640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26307) UID: 0 2024-09-13T11:23:07.773 INFO:tasks.ceph.osd.13.smithi092.stderr:2024-09-13T11:23:07.770+0000 7f3e5afea640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26350) UID: 0 2024-09-13T11:23:07.782 INFO:teuthology.orchestra.run.smithi092.stdout:ERROR: (22) Invalid argument 2024-09-13T11:23:07.782 INFO:teuthology.orchestra.run.smithi092.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:23:07.800 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_ops_in_flight 2024-09-13T11:23:07.888 INFO:tasks.ceph.osd.13.smithi092.stderr:2024-09-13T11:23:07.886+0000 7f3e5afea640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26350) UID: 0 2024-09-13T11:23:07.976 INFO:tasks.ceph.osd.1.smithi092.stderr:2024-09-13T11:23:07.974+0000 7fe7789c5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33718) UID: 0 2024-09-13T11:23:07.980 INFO:teuthology.orchestra.run.smithi092.stdout:ERROR: (22) Invalid argument 2024-09-13T11:23:07.981 INFO:teuthology.orchestra.run.smithi092.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:23:07.998 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_blocked_ops 2024-09-13T11:23:08.001 INFO:tasks.radosbench.radosbench.0.smithi177.stdout: 41 16 28609 28593 43.5817 140.125 0.0036884 0.0229381 2024-09-13T11:23:08.077 INFO:tasks.ceph.osd.7.smithi177.stderr:2024-09-13T11:23:08.074+0000 7f897b51d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26575) UID: 0 2024-09-13T11:23:08.173 INFO:teuthology.orchestra.run.smithi092.stdout:ERROR: (22) Invalid argument 2024-09-13T11:23:08.173 INFO:teuthology.orchestra.run.smithi092.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:23:08.178 INFO:tasks.ceph.osd.10.smithi162.stderr:2024-09-13T11:23:08.175+0000 7fb59c9f0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26317) UID: 0 2024-09-13T11:23:08.191 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_historic_ops 2024-09-13T11:23:08.279 INFO:tasks.ceph.osd.2.smithi162.stderr:2024-09-13T11:23:08.275+0000 7fe92b35a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26313) UID: 0 2024-09-13T11:23:08.318 INFO:teuthology.orchestra.run.smithi092.stdout:ERROR: (22) Invalid argument 2024-09-13T11:23:08.318 INFO:teuthology.orchestra.run.smithi092.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:23:08.334 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-13T11:23:08.380 INFO:tasks.ceph.osd.13.smithi092.stderr:2024-09-13T11:23:08.378+0000 7f3e5afea640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26350) UID: 0 2024-09-13T11:23:08.481 INFO:tasks.ceph.osd.3.smithi177.stderr:2024-09-13T11:23:08.478+0000 7f10adba4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26574) UID: 0 2024-09-13T11:23:08.486 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-13T11:23:08.486 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:23:08.504 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-13T11:23:08.582 INFO:tasks.ceph.osd.5.smithi092.stderr:2024-09-13T11:23:08.578+0000 7f6f0b705640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26346) UID: 0 2024-09-13T11:23:08.644 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-13T11:23:08.644 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:23:08.662 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-13T11:23:08.683 INFO:tasks.ceph.osd.11.smithi177.stderr:2024-09-13T11:23:08.682+0000 7f24792c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26580) UID: 0 2024-09-13T11:23:08.691 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T11:23:08.785 INFO:tasks.ceph.osd.9.smithi092.stderr:2024-09-13T11:23:08.782+0000 7ff5ab3d0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26348) UID: 0 2024-09-13T11:23:08.794 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-13T11:23:08.794 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:23:08.809 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-13T11:23:08.858 INFO:teuthology.orchestra.run.smithi001.stdout:osd.0: {} 2024-09-13T11:23:08.858 INFO:teuthology.orchestra.run.smithi001.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T11:23:08.885 INFO:teuthology.orchestra.run.smithi001.stdout:osd.1: {} 2024-09-13T11:23:08.885 INFO:teuthology.orchestra.run.smithi001.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T11:23:08.886 INFO:tasks.ceph.osd.7.smithi177.stderr:2024-09-13T11:23:08.882+0000 7f897b51d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26575) UID: 0 2024-09-13T11:23:08.908 INFO:teuthology.orchestra.run.smithi001.stdout:osd.2: {} 2024-09-13T11:23:08.909 INFO:teuthology.orchestra.run.smithi001.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T11:23:08.934 INFO:teuthology.orchestra.run.smithi001.stdout:osd.3: {} 2024-09-13T11:23:08.934 INFO:teuthology.orchestra.run.smithi001.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T11:23:08.941 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-13T11:23:08.941 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:23:08.957 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-13T11:23:08.958 INFO:teuthology.orchestra.run.smithi001.stdout:osd.4: {} 2024-09-13T11:23:08.958 INFO:teuthology.orchestra.run.smithi001.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T11:23:08.983 INFO:teuthology.orchestra.run.smithi001.stdout:osd.5: {} 2024-09-13T11:23:08.983 INFO:teuthology.orchestra.run.smithi001.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T11:23:08.987 INFO:tasks.ceph.osd.13.smithi092.stderr:2024-09-13T11:23:08.982+0000 7f3e5afea640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26350) UID: 0 2024-09-13T11:23:09.001 INFO:tasks.radosbench.radosbench.0.smithi177.stdout: 42 16 30203 30187 44.9158 99.625 0.0131435 0.0222448 2024-09-13T11:23:09.009 INFO:teuthology.orchestra.run.smithi001.stdout:osd.6: {} 2024-09-13T11:23:09.009 INFO:teuthology.orchestra.run.smithi001.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T11:23:09.079 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-13T11:23:09.080 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [], 2024-09-13T11:23:09.080 INFO:teuthology.orchestra.run.smithi162.stdout: "complaint_time": 30, 2024-09-13T11:23:09.080 INFO:teuthology.orchestra.run.smithi162.stdout: "num_blocked_ops": 0 2024-09-13T11:23:09.080 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-13T11:23:09.087 INFO:tasks.ceph.osd.2.smithi162.stderr:2024-09-13T11:23:09.082+0000 7fe92b35a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26313) UID: 0 2024-09-13T11:23:09.095 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-13T11:23:09.139 INFO:teuthology.orchestra.run.smithi001.stdout:osd.7: {} 2024-09-13T11:23:09.139 INFO:teuthology.orchestra.run.smithi001.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T11:23:09.172 INFO:teuthology.orchestra.run.smithi001.stdout:osd.8: {} 2024-09-13T11:23:09.173 INFO:teuthology.orchestra.run.smithi001.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-13T11:23:09.188 INFO:tasks.ceph.osd.8.smithi001.stderr:2024-09-13T11:23:09.184+0000 7fb9b48ae640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26312) UID: 0 2024-09-13T11:23:09.200 INFO:teuthology.orchestra.run.smithi001.stdout:osd.9: {} 2024-09-13T11:23:09.200 INFO:teuthology.orchestra.run.smithi001.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-13T11:23:09.228 INFO:teuthology.orchestra.run.smithi001.stdout:osd.10: {} 2024-09-13T11:23:09.228 INFO:teuthology.orchestra.run.smithi001.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-13T11:23:09.257 INFO:teuthology.orchestra.run.smithi001.stdout:osd.11: {} 2024-09-13T11:23:09.257 INFO:teuthology.orchestra.run.smithi001.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-13T11:23:09.280 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-13T11:23:09.280 INFO:teuthology.orchestra.run.smithi162.stdout: "size": 20, 2024-09-13T11:23:09.280 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 600, 2024-09-13T11:23:09.280 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [ 2024-09-13T11:23:09.280 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.280 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4257.0:49 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-13T11:23:09.280 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:22:12.319663+0000", 2024-09-13T11:23:09.281 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 56.956097595999999, 2024-09-13T11:23:09.281 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.046281448000000003, 2024-09-13T11:23:09.281 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:23:09.281 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:23:09.281 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:09.281 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:23:09.282 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4257", 2024-09-13T11:23:09.282 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/1488131527", 2024-09-13T11:23:09.282 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 49 2024-09-13T11:23:09.282 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.282 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:23:09.282 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.282 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:23:09.282 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.319663+0000", 2024-09-13T11:23:09.283 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:09.283 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.283 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.283 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:23:09.283 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.319663+0000", 2024-09-13T11:23:09.283 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:09.283 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.283 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.283 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:23:09.283 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.319665+0000", 2024-09-13T11:23:09.283 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.0030000000000001e-06 2024-09-13T11:23:09.284 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.284 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.284 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:23:09.284 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.319675+0000", 2024-09-13T11:23:09.284 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0019e-05 2024-09-13T11:23:09.284 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.284 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.284 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:23:09.284 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.319697+0000", 2024-09-13T11:23:09.285 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.1387999999999999e-05 2024-09-13T11:23:09.285 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.285 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.285 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:23:09.285 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.319712+0000", 2024-09-13T11:23:09.285 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5068999999999999e-05 2024-09-13T11:23:09.285 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.285 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.285 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:23:09.285 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.352248+0000", 2024-09-13T11:23:09.285 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.032536460000000003 2024-09-13T11:23:09.286 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.286 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.286 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:23:09.286 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.352479+0000", 2024-09-13T11:23:09.286 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00023091499999999999 2024-09-13T11:23:09.286 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.286 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.286 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 0", 2024-09-13T11:23:09.287 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.352882+0000", 2024-09-13T11:23:09.287 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00040249300000000002 2024-09-13T11:23:09.287 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.287 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.287 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:09.287 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.357206+0000", 2024-09-13T11:23:09.287 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.004324802 2024-09-13T11:23:09.287 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.287 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.287 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:23:09.287 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.365708+0000", 2024-09-13T11:23:09.287 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0085012509999999996 2024-09-13T11:23:09.288 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.288 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.288 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:23:09.288 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.365839+0000", 2024-09-13T11:23:09.288 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000131202 2024-09-13T11:23:09.288 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.288 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.288 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:23:09.288 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.365945+0000", 2024-09-13T11:23:09.288 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000105846 2024-09-13T11:23:09.288 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:09.288 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:23:09.289 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:09.289 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.289 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.289 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4257.0:50 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-13T11:23:09.289 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:22:12.319731+0000", 2024-09-13T11:23:09.289 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 56.956030108999997, 2024-09-13T11:23:09.289 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.046650321000000002, 2024-09-13T11:23:09.289 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:23:09.289 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:23:09.289 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:09.289 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:23:09.289 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4257", 2024-09-13T11:23:09.290 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/1488131527", 2024-09-13T11:23:09.290 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 50 2024-09-13T11:23:09.290 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.290 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:23:09.290 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.290 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:23:09.291 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.319731+0000", 2024-09-13T11:23:09.291 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:09.291 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.291 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.291 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:23:09.291 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.319731+0000", 2024-09-13T11:23:09.291 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:09.291 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.291 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.291 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:23:09.292 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.319732+0000", 2024-09-13T11:23:09.292 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.75e-06 2024-09-13T11:23:09.292 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.292 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.292 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:23:09.292 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.319802+0000", 2024-09-13T11:23:09.292 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.9166999999999997e-05 2024-09-13T11:23:09.292 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.292 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.292 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:23:09.292 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.319826+0000", 2024-09-13T11:23:09.292 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.4464000000000002e-05 2024-09-13T11:23:09.293 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.293 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.293 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:23:09.293 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.319856+0000", 2024-09-13T11:23:09.293 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.9530999999999999e-05 2024-09-13T11:23:09.293 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.293 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.293 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:23:09.293 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.357551+0000", 2024-09-13T11:23:09.293 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.037695687999999998 2024-09-13T11:23:09.293 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.293 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.294 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:23:09.294 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.357707+0000", 2024-09-13T11:23:09.294 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00015562200000000001 2024-09-13T11:23:09.294 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.294 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.294 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 0", 2024-09-13T11:23:09.294 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.357906+0000", 2024-09-13T11:23:09.294 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00019874500000000001 2024-09-13T11:23:09.294 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.294 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.295 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:09.295 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.366111+0000", 2024-09-13T11:23:09.295 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.008205815 2024-09-13T11:23:09.295 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.295 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.295 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:23:09.295 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.366149+0000", 2024-09-13T11:23:09.295 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.7113999999999997e-05 2024-09-13T11:23:09.295 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.295 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.295 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:23:09.295 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.366283+0000", 2024-09-13T11:23:09.295 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00013399399999999999 2024-09-13T11:23:09.295 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.296 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.296 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:23:09.296 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.366381+0000", 2024-09-13T11:23:09.296 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.8431000000000007e-05 2024-09-13T11:23:09.296 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:09.296 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:23:09.296 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:09.296 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.296 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.297 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4257.0:51 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-13T11:23:09.297 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:22:12.319877+0000", 2024-09-13T11:23:09.297 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 56.955883479000001, 2024-09-13T11:23:09.297 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.048107695999999998, 2024-09-13T11:23:09.297 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:23:09.297 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:23:09.298 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:09.298 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:23:09.298 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4257", 2024-09-13T11:23:09.298 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/1488131527", 2024-09-13T11:23:09.298 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 51 2024-09-13T11:23:09.299 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.299 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:23:09.299 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.299 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:23:09.299 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.319877+0000", 2024-09-13T11:23:09.299 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:09.299 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.299 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.299 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:23:09.299 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.319877+0000", 2024-09-13T11:23:09.300 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:09.300 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.300 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.300 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:23:09.300 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.319879+0000", 2024-09-13T11:23:09.300 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.44e-06 2024-09-13T11:23:09.300 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.300 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.300 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:23:09.300 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.319885+0000", 2024-09-13T11:23:09.300 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.8690000000000001e-06 2024-09-13T11:23:09.300 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.300 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.301 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:23:09.301 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.319894+0000", 2024-09-13T11:23:09.301 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.7389999999999994e-06 2024-09-13T11:23:09.301 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.301 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.301 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:23:09.301 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.319903+0000", 2024-09-13T11:23:09.301 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.2830000000000001e-06 2024-09-13T11:23:09.301 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.301 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.302 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:23:09.302 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.361846+0000", 2024-09-13T11:23:09.302 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.041943041 2024-09-13T11:23:09.302 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.302 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.302 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:23:09.302 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.362090+0000", 2024-09-13T11:23:09.302 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00024382399999999999 2024-09-13T11:23:09.302 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.302 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.302 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 0", 2024-09-13T11:23:09.302 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.362493+0000", 2024-09-13T11:23:09.302 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00040374600000000002 2024-09-13T11:23:09.302 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.303 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.303 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:09.303 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.366507+0000", 2024-09-13T11:23:09.303 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0040133369999999996 2024-09-13T11:23:09.303 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.303 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.303 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:23:09.304 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.367751+0000", 2024-09-13T11:23:09.304 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001244118 2024-09-13T11:23:09.304 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.304 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.304 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:23:09.304 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.367873+0000", 2024-09-13T11:23:09.304 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000121972 2024-09-13T11:23:09.304 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.304 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.304 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:23:09.304 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.367985+0000", 2024-09-13T11:23:09.305 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000112327 2024-09-13T11:23:09.305 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:09.305 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:23:09.305 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:09.305 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.305 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.305 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4257.0:52 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-13T11:23:09.305 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:22:12.319912+0000", 2024-09-13T11:23:09.305 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 56.955848822, 2024-09-13T11:23:09.305 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.049068963, 2024-09-13T11:23:09.305 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:23:09.305 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:23:09.306 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:09.306 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:23:09.306 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4257", 2024-09-13T11:23:09.306 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/1488131527", 2024-09-13T11:23:09.306 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 52 2024-09-13T11:23:09.306 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.306 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:23:09.306 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.306 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:23:09.306 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.319912+0000", 2024-09-13T11:23:09.307 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:09.307 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.307 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.307 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:23:09.307 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.319912+0000", 2024-09-13T11:23:09.307 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:09.307 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.308 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.308 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:23:09.308 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.319913+0000", 2024-09-13T11:23:09.308 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.133e-06 2024-09-13T11:23:09.308 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.308 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.308 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:23:09.308 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.319918+0000", 2024-09-13T11:23:09.309 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.7729999999999999e-06 2024-09-13T11:23:09.309 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.309 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.309 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:23:09.309 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.319927+0000", 2024-09-13T11:23:09.309 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.6750000000000008e-06 2024-09-13T11:23:09.309 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.309 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.310 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:23:09.310 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.319935+0000", 2024-09-13T11:23:09.310 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.9500000000000001e-06 2024-09-13T11:23:09.310 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.310 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.310 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:23:09.310 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.366585+0000", 2024-09-13T11:23:09.311 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.046649991000000002 2024-09-13T11:23:09.311 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.311 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.311 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:23:09.311 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.366832+0000", 2024-09-13T11:23:09.311 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00024709999999999999 2024-09-13T11:23:09.311 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.311 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.312 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 0", 2024-09-13T11:23:09.312 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.367210+0000", 2024-09-13T11:23:09.312 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00037837199999999997 2024-09-13T11:23:09.312 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.312 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.312 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:23:09.313 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.368430+0000", 2024-09-13T11:23:09.313 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00122013 2024-09-13T11:23:09.313 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.313 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.313 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:09.313 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.368787+0000", 2024-09-13T11:23:09.313 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00035728300000000001 2024-09-13T11:23:09.313 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.314 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.314 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:23:09.314 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.368874+0000", 2024-09-13T11:23:09.314 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.6876e-05 2024-09-13T11:23:09.314 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.314 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.314 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:23:09.314 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.368981+0000", 2024-09-13T11:23:09.315 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00010668 2024-09-13T11:23:09.315 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:09.315 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:23:09.315 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:09.315 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.315 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.315 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4257.0:73 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-13T11:23:09.315 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:22:12.458939+0000", 2024-09-13T11:23:09.316 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 56.816821656999998, 2024-09-13T11:23:09.316 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.050922262000000003, 2024-09-13T11:23:09.316 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:23:09.316 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:23:09.316 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:09.316 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:23:09.316 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4257", 2024-09-13T11:23:09.316 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/1488131527", 2024-09-13T11:23:09.317 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 73 2024-09-13T11:23:09.317 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.317 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:23:09.317 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.317 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:23:09.317 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.458939+0000", 2024-09-13T11:23:09.317 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:09.317 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.318 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.318 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:23:09.318 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.458939+0000", 2024-09-13T11:23:09.318 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:09.318 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.318 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.318 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:23:09.318 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.458940+0000", 2024-09-13T11:23:09.319 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.22e-06 2024-09-13T11:23:09.319 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.319 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.319 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:23:09.319 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.458945+0000", 2024-09-13T11:23:09.319 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.6800000000000001e-06 2024-09-13T11:23:09.319 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.319 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.320 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:23:09.320 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.458954+0000", 2024-09-13T11:23:09.320 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.4589999999999998e-06 2024-09-13T11:23:09.320 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.320 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.320 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:23:09.320 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.458961+0000", 2024-09-13T11:23:09.320 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.2420000000000003e-06 2024-09-13T11:23:09.321 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.321 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.321 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:23:09.321 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.496214+0000", 2024-09-13T11:23:09.321 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.037252805999999999 2024-09-13T11:23:09.321 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.321 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.321 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:23:09.322 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.496446+0000", 2024-09-13T11:23:09.322 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00023255800000000001 2024-09-13T11:23:09.322 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.322 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.322 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 0", 2024-09-13T11:23:09.322 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.496800+0000", 2024-09-13T11:23:09.322 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00035397400000000001 2024-09-13T11:23:09.322 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.323 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.323 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:09.323 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.504887+0000", 2024-09-13T11:23:09.323 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0080867769999999999 2024-09-13T11:23:09.323 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.323 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.323 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:23:09.324 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.509579+0000", 2024-09-13T11:23:09.324 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0046924849999999997 2024-09-13T11:23:09.324 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.324 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.324 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:23:09.324 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.509748+0000", 2024-09-13T11:23:09.324 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00016907399999999999 2024-09-13T11:23:09.324 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.324 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.324 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:23:09.325 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.509861+0000", 2024-09-13T11:23:09.325 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00011298699999999999 2024-09-13T11:23:09.325 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:09.325 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:23:09.325 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:09.325 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.325 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.325 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4257.0:74 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-13T11:23:09.325 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:22:12.458973+0000", 2024-09-13T11:23:09.325 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 56.816788236999997, 2024-09-13T11:23:09.325 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.055492705000000003, 2024-09-13T11:23:09.325 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:23:09.326 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:23:09.326 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:09.326 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:23:09.326 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4257", 2024-09-13T11:23:09.326 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/1488131527", 2024-09-13T11:23:09.326 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 74 2024-09-13T11:23:09.327 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.327 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:23:09.327 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.327 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:23:09.327 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.458973+0000", 2024-09-13T11:23:09.327 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:09.328 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.328 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.328 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:23:09.328 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.458973+0000", 2024-09-13T11:23:09.328 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:09.328 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.328 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.328 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:23:09.328 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.458974+0000", 2024-09-13T11:23:09.329 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.3699999999999999e-07 2024-09-13T11:23:09.329 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.329 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.329 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:23:09.329 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.458978+0000", 2024-09-13T11:23:09.330 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.4259999999999996e-06 2024-09-13T11:23:09.330 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.330 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.330 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:23:09.330 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.458994+0000", 2024-09-13T11:23:09.330 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5852e-05 2024-09-13T11:23:09.330 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.330 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.330 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:23:09.331 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.459002+0000", 2024-09-13T11:23:09.331 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.5830000000000007e-06 2024-09-13T11:23:09.331 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.331 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.331 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:23:09.331 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.500820+0000", 2024-09-13T11:23:09.331 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.041817432000000002 2024-09-13T11:23:09.331 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.331 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.331 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:23:09.332 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.500973+0000", 2024-09-13T11:23:09.332 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00015326200000000001 2024-09-13T11:23:09.332 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.332 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.332 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 0", 2024-09-13T11:23:09.332 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.501212+0000", 2024-09-13T11:23:09.332 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00023885099999999999 2024-09-13T11:23:09.332 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.332 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.332 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:09.333 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.505008+0000", 2024-09-13T11:23:09.333 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0037961850000000001 2024-09-13T11:23:09.333 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.333 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.333 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:23:09.333 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.514227+0000", 2024-09-13T11:23:09.333 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0092187889999999998 2024-09-13T11:23:09.333 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.333 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.333 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:23:09.334 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.514362+0000", 2024-09-13T11:23:09.334 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000134967 2024-09-13T11:23:09.334 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.334 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.334 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:23:09.334 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.514465+0000", 2024-09-13T11:23:09.334 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000103421 2024-09-13T11:23:09.334 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:09.334 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:23:09.334 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:09.334 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.334 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.335 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4257.0:75 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-13T11:23:09.335 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:22:12.459011+0000", 2024-09-13T11:23:09.335 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 56.816749340000001, 2024-09-13T11:23:09.335 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.056002977000000002, 2024-09-13T11:23:09.335 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:23:09.335 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:23:09.335 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:09.335 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:23:09.335 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4257", 2024-09-13T11:23:09.335 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/1488131527", 2024-09-13T11:23:09.335 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 75 2024-09-13T11:23:09.336 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.336 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:23:09.336 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.336 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:23:09.336 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.459011+0000", 2024-09-13T11:23:09.336 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:09.336 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.336 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.336 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:23:09.336 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.459011+0000", 2024-09-13T11:23:09.336 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:09.336 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.336 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.337 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:23:09.337 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.459012+0000", 2024-09-13T11:23:09.337 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.9999999999999995e-07 2024-09-13T11:23:09.337 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.337 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.337 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:23:09.337 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.459060+0000", 2024-09-13T11:23:09.337 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.7587999999999999e-05 2024-09-13T11:23:09.337 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.337 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.337 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:23:09.338 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.459070+0000", 2024-09-13T11:23:09.338 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.9690000000000006e-06 2024-09-13T11:23:09.338 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.338 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.338 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:23:09.338 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.459078+0000", 2024-09-13T11:23:09.338 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.5329999999999997e-06 2024-09-13T11:23:09.338 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.339 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.339 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:23:09.339 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.505105+0000", 2024-09-13T11:23:09.339 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.046027908999999999 2024-09-13T11:23:09.339 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.339 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.339 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:23:09.339 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.505322+0000", 2024-09-13T11:23:09.339 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00021677600000000001 2024-09-13T11:23:09.339 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.339 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.339 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 0", 2024-09-13T11:23:09.340 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.505675+0000", 2024-09-13T11:23:09.340 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00035293800000000001 2024-09-13T11:23:09.340 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.340 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.340 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:09.340 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.514620+0000", 2024-09-13T11:23:09.340 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0089446730000000002 2024-09-13T11:23:09.340 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.341 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.341 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:23:09.341 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.514777+0000", 2024-09-13T11:23:09.341 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00015762999999999999 2024-09-13T11:23:09.341 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.341 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.341 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:23:09.341 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.514903+0000", 2024-09-13T11:23:09.341 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00012559800000000001 2024-09-13T11:23:09.342 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.342 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.342 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:23:09.342 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.515014+0000", 2024-09-13T11:23:09.342 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000111363 2024-09-13T11:23:09.342 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:09.342 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:23:09.342 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:09.343 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.343 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.343 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4257.0:76 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-13T11:23:09.343 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:22:12.459088+0000", 2024-09-13T11:23:09.343 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 56.816672631000003, 2024-09-13T11:23:09.344 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.056165815000000001, 2024-09-13T11:23:09.344 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:23:09.344 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:23:09.344 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:09.344 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:23:09.344 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4257", 2024-09-13T11:23:09.345 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/1488131527", 2024-09-13T11:23:09.345 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 76 2024-09-13T11:23:09.345 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.345 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:23:09.345 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.346 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:23:09.346 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.459088+0000", 2024-09-13T11:23:09.346 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:09.346 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.346 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.346 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:23:09.346 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.459088+0000", 2024-09-13T11:23:09.346 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:09.346 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.346 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.346 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:23:09.346 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.459089+0000", 2024-09-13T11:23:09.346 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.114e-06 2024-09-13T11:23:09.347 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.347 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.347 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:23:09.347 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.459094+0000", 2024-09-13T11:23:09.347 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.0320000000000003e-06 2024-09-13T11:23:09.348 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.348 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.348 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:23:09.348 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.459103+0000", 2024-09-13T11:23:09.348 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.2989999999999997e-06 2024-09-13T11:23:09.348 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.348 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.348 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:23:09.348 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.459110+0000", 2024-09-13T11:23:09.348 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.9970000000000001e-06 2024-09-13T11:23:09.348 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.349 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.349 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:23:09.349 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.509960+0000", 2024-09-13T11:23:09.349 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.050850601000000002 2024-09-13T11:23:09.349 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.349 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.349 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:23:09.349 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.510216+0000", 2024-09-13T11:23:09.349 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00025548299999999998 2024-09-13T11:23:09.349 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.349 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.349 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 0", 2024-09-13T11:23:09.350 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.510559+0000", 2024-09-13T11:23:09.350 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00034306199999999999 2024-09-13T11:23:09.350 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.350 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.350 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:09.350 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.514745+0000", 2024-09-13T11:23:09.350 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0041858920000000001 2024-09-13T11:23:09.351 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.351 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.351 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:23:09.351 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.515035+0000", 2024-09-13T11:23:09.351 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00029025600000000002 2024-09-13T11:23:09.351 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.351 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.351 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:23:09.351 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.515152+0000", 2024-09-13T11:23:09.352 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000116872 2024-09-13T11:23:09.352 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.352 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.352 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:23:09.352 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.515254+0000", 2024-09-13T11:23:09.352 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00010220700000000001 2024-09-13T11:23:09.352 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:09.352 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:23:09.352 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:09.352 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.352 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.352 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4257.0:80 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 262144~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-13T11:23:09.353 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:22:12.516065+0000", 2024-09-13T11:23:09.353 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 56.759695588, 2024-09-13T11:23:09.353 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.46461997999999999, 2024-09-13T11:23:09.353 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:23:09.353 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:23:09.353 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:09.353 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:23:09.353 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4257", 2024-09-13T11:23:09.353 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/1488131527", 2024-09-13T11:23:09.353 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 80 2024-09-13T11:23:09.354 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.354 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:23:09.354 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.354 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:23:09.354 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.516065+0000", 2024-09-13T11:23:09.354 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:09.354 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.354 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.354 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:23:09.354 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.516065+0000", 2024-09-13T11:23:09.354 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:09.354 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.355 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.355 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:23:09.355 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.516066+0000", 2024-09-13T11:23:09.355 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0869999999999999e-06 2024-09-13T11:23:09.355 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.355 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.355 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:23:09.355 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.516112+0000", 2024-09-13T11:23:09.355 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.5503000000000001e-05 2024-09-13T11:23:09.355 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.356 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.356 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:23:09.356 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.516128+0000", 2024-09-13T11:23:09.356 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5891000000000001e-05 2024-09-13T11:23:09.356 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.356 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.356 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:23:09.356 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.516138+0000", 2024-09-13T11:23:09.356 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.8120000000000002e-06 2024-09-13T11:23:09.356 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.357 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.357 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:23:09.357 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.520398+0000", 2024-09-13T11:23:09.357 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0042605450000000001 2024-09-13T11:23:09.357 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.357 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.357 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:23:09.357 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.520526+0000", 2024-09-13T11:23:09.357 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000128355 2024-09-13T11:23:09.357 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.357 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.358 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 0", 2024-09-13T11:23:09.358 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.520730+0000", 2024-09-13T11:23:09.358 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00020337799999999999 2024-09-13T11:23:09.358 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.358 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.358 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:23:09.358 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.522417+0000", 2024-09-13T11:23:09.358 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001687184 2024-09-13T11:23:09.358 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.358 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.358 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:09.359 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.980408+0000", 2024-09-13T11:23:09.359 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.45799068399999998 2024-09-13T11:23:09.359 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.359 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.359 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:23:09.359 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.980565+0000", 2024-09-13T11:23:09.359 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00015760899999999999 2024-09-13T11:23:09.359 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.359 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.360 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:23:09.360 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.980685+0000", 2024-09-13T11:23:09.360 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000119932 2024-09-13T11:23:09.360 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:09.360 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:23:09.360 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:09.361 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.361 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.361 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop_reply(client.4257.0:80 1.0 e24/23 ondisk, result = 0)", 2024-09-13T11:23:09.361 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:22:12.522938+0000", 2024-09-13T11:23:09.361 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 56.752822555000002, 2024-09-13T11:23:09.361 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.45776420600000001, 2024-09-13T11:23:09.361 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:23:09.361 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:23:09.361 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:23:09.361 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:23:09.362 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.362 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:23:09.362 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.522938+0000", 2024-09-13T11:23:09.362 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:09.362 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.362 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.362 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:23:09.362 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.522938+0000", 2024-09-13T11:23:09.362 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:09.362 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.362 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.362 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:23:09.362 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.522941+0000", 2024-09-13T11:23:09.363 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.0290000000000001e-06 2024-09-13T11:23:09.363 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.363 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.363 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:23:09.363 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.522950+0000", 2024-09-13T11:23:09.363 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.7499999999999992e-06 2024-09-13T11:23:09.363 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.363 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.363 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:23:09.363 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.980215+0000", 2024-09-13T11:23:09.363 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.45726523800000002 2024-09-13T11:23:09.363 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.364 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.364 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:23:09.364 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.980229+0000", 2024-09-13T11:23:09.364 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3549e-05 2024-09-13T11:23:09.364 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.364 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.364 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:23:09.364 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.980365+0000", 2024-09-13T11:23:09.364 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00013608700000000001 2024-09-13T11:23:09.364 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.364 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.364 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:23:09.365 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.980388+0000", 2024-09-13T11:23:09.365 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.3112000000000002e-05 2024-09-13T11:23:09.365 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.365 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.365 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:23:09.365 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.980702+0000", 2024-09-13T11:23:09.365 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00031444099999999999 2024-09-13T11:23:09.365 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:09.365 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:23:09.365 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:09.365 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.365 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.366 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:9761 3.8 3:1c01147e:::benchmark_data_smithi177_31116_object9760:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-09-13T11:23:09.366 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:22:54.198844+0000", 2024-09-13T11:23:09.366 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 15.076917233, 2024-09-13T11:23:09.366 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.051690649999999998, 2024-09-13T11:23:09.366 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:23:09.366 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:23:09.366 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:09.366 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:23:09.366 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:23:09.366 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:09.366 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 9761 2024-09-13T11:23:09.366 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.367 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:23:09.367 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.367 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:23:09.367 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:54.198844+0000", 2024-09-13T11:23:09.367 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:09.367 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.367 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.367 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:23:09.367 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:54.198844+0000", 2024-09-13T11:23:09.367 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:09.367 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.367 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.368 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:23:09.368 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:54.198847+0000", 2024-09-13T11:23:09.368 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.1520000000000001e-06 2024-09-13T11:23:09.368 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.368 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.368 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:23:09.369 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:54.198906+0000", 2024-09-13T11:23:09.369 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.9315999999999999e-05 2024-09-13T11:23:09.369 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.369 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.369 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:23:09.369 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:54.198928+0000", 2024-09-13T11:23:09.369 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.2053999999999999e-05 2024-09-13T11:23:09.369 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.369 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.370 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:23:09.370 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:54.198947+0000", 2024-09-13T11:23:09.370 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.8482e-05 2024-09-13T11:23:09.370 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.370 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.370 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:23:09.370 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:54.199074+0000", 2024-09-13T11:23:09.370 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00012774600000000001 2024-09-13T11:23:09.370 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.370 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.370 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:23:09.370 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:54.199351+0000", 2024-09-13T11:23:09.371 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00027705900000000002 2024-09-13T11:23:09.371 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.371 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.371 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 1", 2024-09-13T11:23:09.371 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:54.199707+0000", 2024-09-13T11:23:09.371 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00035590600000000002 2024-09-13T11:23:09.371 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.371 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.371 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:23:09.371 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:54.201465+0000", 2024-09-13T11:23:09.371 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0017581179999999999 2024-09-13T11:23:09.372 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.372 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.372 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:09.372 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:54.250374+0000", 2024-09-13T11:23:09.372 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.048908419000000002 2024-09-13T11:23:09.372 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.372 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.372 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:23:09.372 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:54.250467+0000", 2024-09-13T11:23:09.372 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.3367000000000002e-05 2024-09-13T11:23:09.373 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.373 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.373 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:23:09.373 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:54.250534+0000", 2024-09-13T11:23:09.373 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.7031000000000002e-05 2024-09-13T11:23:09.373 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:09.373 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:23:09.373 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:09.373 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.373 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.373 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:9859 3.8 3:13ff5619:::benchmark_data_smithi177_31116_object9858:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-09-13T11:23:09.373 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:22:54.263848+0000", 2024-09-13T11:23:09.373 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 15.011912991999999, 2024-09-13T11:23:09.374 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.043382326999999998, 2024-09-13T11:23:09.374 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:23:09.374 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:23:09.374 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:09.374 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:23:09.374 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:23:09.374 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:09.374 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 9859 2024-09-13T11:23:09.374 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.374 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:23:09.374 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.374 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:23:09.374 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:54.263848+0000", 2024-09-13T11:23:09.374 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:09.375 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.375 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.375 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:23:09.375 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:54.263848+0000", 2024-09-13T11:23:09.375 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:09.375 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.375 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.375 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:23:09.375 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:54.263852+0000", 2024-09-13T11:23:09.375 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.4469999999999999e-06 2024-09-13T11:23:09.375 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.375 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.375 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:23:09.376 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:54.263938+0000", 2024-09-13T11:23:09.376 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.5494000000000005e-05 2024-09-13T11:23:09.376 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.376 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.376 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:23:09.376 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:54.263975+0000", 2024-09-13T11:23:09.376 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.7109e-05 2024-09-13T11:23:09.376 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.376 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.376 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:23:09.376 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:54.263996+0000", 2024-09-13T11:23:09.376 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.0656e-05 2024-09-13T11:23:09.376 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.376 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.377 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:23:09.377 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:54.264144+0000", 2024-09-13T11:23:09.377 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00014840900000000001 2024-09-13T11:23:09.377 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.377 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.377 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:23:09.377 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:54.264369+0000", 2024-09-13T11:23:09.377 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00022494600000000001 2024-09-13T11:23:09.377 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.377 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.377 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 1", 2024-09-13T11:23:09.377 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:54.264580+0000", 2024-09-13T11:23:09.377 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00021120399999999999 2024-09-13T11:23:09.378 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.378 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.378 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:23:09.378 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:54.266244+0000", 2024-09-13T11:23:09.378 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001663662 2024-09-13T11:23:09.378 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.378 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.378 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:09.378 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:54.306964+0000", 2024-09-13T11:23:09.378 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.040719995000000002 2024-09-13T11:23:09.378 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.378 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.378 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:23:09.378 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:54.307111+0000", 2024-09-13T11:23:09.379 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00014726600000000001 2024-09-13T11:23:09.379 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.379 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.379 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:23:09.379 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:54.307230+0000", 2024-09-13T11:23:09.379 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000119139 2024-09-13T11:23:09.379 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:09.379 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:23:09.379 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:09.379 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.379 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.379 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:24758 3.8 3:1ae6e252:::benchmark_data_smithi177_31116_object24757:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e55)", 2024-09-13T11:23:09.379 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:03.894679+0000", 2024-09-13T11:23:09.380 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 5.3810816169999995, 2024-09-13T11:23:09.380 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.055283855999999999, 2024-09-13T11:23:09.380 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:23:09.380 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:23:09.380 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:09.380 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:23:09.380 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:23:09.380 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:09.380 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 24758 2024-09-13T11:23:09.380 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.380 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:23:09.380 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.380 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:23:09.381 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:03.894679+0000", 2024-09-13T11:23:09.381 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:09.381 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.381 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.381 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:23:09.381 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:03.894679+0000", 2024-09-13T11:23:09.381 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:09.381 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.381 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.381 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:23:09.381 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:03.894682+0000", 2024-09-13T11:23:09.381 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.6359999999999998e-06 2024-09-13T11:23:09.382 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.382 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.382 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:23:09.382 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:03.894747+0000", 2024-09-13T11:23:09.382 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.5475999999999994e-05 2024-09-13T11:23:09.382 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.382 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.382 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:23:09.382 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:03.894770+0000", 2024-09-13T11:23:09.382 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.2832000000000001e-05 2024-09-13T11:23:09.382 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.382 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.382 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:23:09.382 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:03.894784+0000", 2024-09-13T11:23:09.383 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3893000000000001e-05 2024-09-13T11:23:09.383 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.383 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.383 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:23:09.383 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:03.896605+0000", 2024-09-13T11:23:09.383 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0018213610000000001 2024-09-13T11:23:09.383 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.383 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.383 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:23:09.383 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:03.896754+0000", 2024-09-13T11:23:09.383 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00014863300000000001 2024-09-13T11:23:09.383 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.383 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.383 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 1", 2024-09-13T11:23:09.383 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:03.896957+0000", 2024-09-13T11:23:09.384 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00020267500000000001 2024-09-13T11:23:09.384 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.384 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.384 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:23:09.384 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:03.900344+0000", 2024-09-13T11:23:09.384 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0033871930000000001 2024-09-13T11:23:09.384 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.384 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.384 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:09.384 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:03.949787+0000", 2024-09-13T11:23:09.384 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.049443560999999997 2024-09-13T11:23:09.384 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.384 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.385 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:23:09.385 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:03.949898+0000", 2024-09-13T11:23:09.385 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000110066 2024-09-13T11:23:09.385 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.385 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.385 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:23:09.385 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:03.949963+0000", 2024-09-13T11:23:09.385 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.5530000000000004e-05 2024-09-13T11:23:09.385 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:09.385 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:23:09.385 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:09.385 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.385 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.385 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop_reply(client.4665.0:24758 3.8 e55/40 ondisk, result = 0)", 2024-09-13T11:23:09.386 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:03.898443+0000", 2024-09-13T11:23:09.386 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 5.3773181340000002, 2024-09-13T11:23:09.386 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.051533658000000003, 2024-09-13T11:23:09.386 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:23:09.386 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:23:09.386 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:23:09.386 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:23:09.386 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.386 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:23:09.386 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:03.898443+0000", 2024-09-13T11:23:09.386 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:09.386 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.386 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.387 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:23:09.387 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:03.898443+0000", 2024-09-13T11:23:09.387 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:09.387 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.387 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.387 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:23:09.387 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:03.898445+0000", 2024-09-13T11:23:09.387 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.04e-06 2024-09-13T11:23:09.387 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.387 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.387 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:23:09.387 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:03.898451+0000", 2024-09-13T11:23:09.387 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.5270000000000004e-06 2024-09-13T11:23:09.387 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.388 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.388 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:23:09.388 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:03.949564+0000", 2024-09-13T11:23:09.388 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.051113020000000002 2024-09-13T11:23:09.388 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.388 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.388 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:23:09.388 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:03.949578+0000", 2024-09-13T11:23:09.388 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3643e-05 2024-09-13T11:23:09.388 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.388 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.388 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:23:09.388 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:03.949735+0000", 2024-09-13T11:23:09.388 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00015688900000000001 2024-09-13T11:23:09.389 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.389 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.389 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:23:09.389 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:03.949754+0000", 2024-09-13T11:23:09.389 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.9193000000000001e-05 2024-09-13T11:23:09.389 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.389 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.389 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:23:09.389 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:03.949976+0000", 2024-09-13T11:23:09.389 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000222346 2024-09-13T11:23:09.389 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:09.389 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:23:09.389 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:09.389 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.390 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.390 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:25378 3.8 3:19339ced:::benchmark_data_smithi177_31116_object25377:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e55)", 2024-09-13T11:23:09.390 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:04.442409+0000", 2024-09-13T11:23:09.390 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 4.8333515, 2024-09-13T11:23:09.390 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.064890903999999999, 2024-09-13T11:23:09.390 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:23:09.390 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:23:09.390 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:09.390 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:23:09.390 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:23:09.390 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:09.390 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 25378 2024-09-13T11:23:09.390 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.390 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:23:09.390 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.391 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:23:09.391 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.442409+0000", 2024-09-13T11:23:09.391 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:09.391 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.391 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.391 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:23:09.391 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.442409+0000", 2024-09-13T11:23:09.391 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:09.391 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.391 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.391 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:23:09.391 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.442415+0000", 2024-09-13T11:23:09.391 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.2800000000000003e-06 2024-09-13T11:23:09.391 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.392 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.392 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:23:09.392 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.442484+0000", 2024-09-13T11:23:09.392 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.8918000000000007e-05 2024-09-13T11:23:09.392 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.392 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.392 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:23:09.392 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.442515+0000", 2024-09-13T11:23:09.392 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.1489000000000003e-05 2024-09-13T11:23:09.392 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.392 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.392 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:23:09.392 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.442531+0000", 2024-09-13T11:23:09.392 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.6120000000000002e-05 2024-09-13T11:23:09.393 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.393 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.393 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:23:09.393 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.442664+0000", 2024-09-13T11:23:09.393 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00013289000000000001 2024-09-13T11:23:09.393 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.393 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.393 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:23:09.393 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.442915+0000", 2024-09-13T11:23:09.393 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00025073100000000002 2024-09-13T11:23:09.393 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.393 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.393 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 1", 2024-09-13T11:23:09.394 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.443137+0000", 2024-09-13T11:23:09.394 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00022200799999999999 2024-09-13T11:23:09.394 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.394 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.394 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:23:09.394 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.445390+0000", 2024-09-13T11:23:09.394 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.002252754 2024-09-13T11:23:09.394 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.394 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.394 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:09.394 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.507042+0000", 2024-09-13T11:23:09.394 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.061652271000000002 2024-09-13T11:23:09.394 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.394 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.395 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:23:09.395 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.507224+0000", 2024-09-13T11:23:09.395 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00018251500000000001 2024-09-13T11:23:09.395 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.395 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.395 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:23:09.395 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.507300+0000", 2024-09-13T11:23:09.395 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.5927999999999995e-05 2024-09-13T11:23:09.395 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:09.395 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:23:09.395 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:09.395 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.395 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.395 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop_reply(client.4665.0:25378 3.8 e55/40 ondisk, result = 0)", 2024-09-13T11:23:09.396 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:04.446177+0000", 2024-09-13T11:23:09.396 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 4.8295841719999997, 2024-09-13T11:23:09.396 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.061139145999999998, 2024-09-13T11:23:09.396 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:23:09.396 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:23:09.396 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:23:09.396 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:23:09.396 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.396 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:23:09.396 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.446177+0000", 2024-09-13T11:23:09.396 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:09.396 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.396 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.396 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:23:09.397 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.446177+0000", 2024-09-13T11:23:09.397 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:09.400 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.400 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.400 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:23:09.400 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.446187+0000", 2024-09-13T11:23:09.400 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0236e-05 2024-09-13T11:23:09.400 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.400 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.401 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:23:09.401 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.446199+0000", 2024-09-13T11:23:09.401 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1756000000000001e-05 2024-09-13T11:23:09.401 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.401 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.401 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:23:09.401 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.506855+0000", 2024-09-13T11:23:09.401 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.060655964999999999 2024-09-13T11:23:09.401 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.401 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.401 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:23:09.401 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.506870+0000", 2024-09-13T11:23:09.401 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.552e-05 2024-09-13T11:23:09.401 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.402 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.402 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:23:09.402 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.507000+0000", 2024-09-13T11:23:09.402 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00012941999999999999 2024-09-13T11:23:09.402 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.402 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.402 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:23:09.402 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.507024+0000", 2024-09-13T11:23:09.402 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.4213e-05 2024-09-13T11:23:09.402 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.402 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.402 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:23:09.402 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.507316+0000", 2024-09-13T11:23:09.402 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00029203600000000002 2024-09-13T11:23:09.403 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:09.403 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:23:09.403 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:09.403 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.403 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.403 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:25416 3.8 3:1a1fbdd1:::benchmark_data_smithi177_31116_object25415:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e56)", 2024-09-13T11:23:09.403 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:04.512188+0000", 2024-09-13T11:23:09.403 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 4.7635725879999997, 2024-09-13T11:23:09.403 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.210208697, 2024-09-13T11:23:09.403 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:23:09.403 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:23:09.403 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:09.403 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:23:09.403 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:23:09.404 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:09.404 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 25416 2024-09-13T11:23:09.404 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.404 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:23:09.404 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.404 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:23:09.404 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.512188+0000", 2024-09-13T11:23:09.404 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:09.404 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.404 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.404 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:23:09.404 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.512188+0000", 2024-09-13T11:23:09.404 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:09.405 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.405 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.405 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:23:09.405 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.512192+0000", 2024-09-13T11:23:09.405 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.2660000000000003e-06 2024-09-13T11:23:09.405 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.405 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.405 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:23:09.405 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.512247+0000", 2024-09-13T11:23:09.405 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.4889000000000003e-05 2024-09-13T11:23:09.405 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.405 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.405 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:23:09.405 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.512279+0000", 2024-09-13T11:23:09.406 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.1146000000000003e-05 2024-09-13T11:23:09.406 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.406 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.406 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:23:09.406 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.512297+0000", 2024-09-13T11:23:09.406 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.825e-05 2024-09-13T11:23:09.406 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.406 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.406 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:23:09.406 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.512377+0000", 2024-09-13T11:23:09.406 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.9875000000000004e-05 2024-09-13T11:23:09.406 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.406 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.406 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:23:09.407 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.512570+0000", 2024-09-13T11:23:09.407 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00019312100000000001 2024-09-13T11:23:09.407 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.407 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.407 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 1", 2024-09-13T11:23:09.407 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.512798+0000", 2024-09-13T11:23:09.407 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00022847899999999999 2024-09-13T11:23:09.407 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.407 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.407 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:23:09.407 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.514975+0000", 2024-09-13T11:23:09.407 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0021769979999999999 2024-09-13T11:23:09.407 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.407 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.408 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:09.408 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.722207+0000", 2024-09-13T11:23:09.408 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.20723217099999999 2024-09-13T11:23:09.408 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.408 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.408 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:23:09.408 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.722314+0000", 2024-09-13T11:23:09.408 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00010648400000000001 2024-09-13T11:23:09.408 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.408 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.408 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:23:09.408 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.722397+0000", 2024-09-13T11:23:09.408 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.3017999999999997e-05 2024-09-13T11:23:09.409 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:09.409 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:23:09.409 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:09.409 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.409 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.409 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop_reply(client.4665.0:25416 3.8 e56/40 ondisk, result = 0)", 2024-09-13T11:23:09.409 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:04.515635+0000", 2024-09-13T11:23:09.409 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 4.7601259359999997, 2024-09-13T11:23:09.409 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.20677600500000001, 2024-09-13T11:23:09.409 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:23:09.409 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:23:09.409 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:23:09.409 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:23:09.410 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.410 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:23:09.410 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.515635+0000", 2024-09-13T11:23:09.410 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:09.410 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.410 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.410 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:23:09.410 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.515635+0000", 2024-09-13T11:23:09.410 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:09.410 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.410 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.411 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:23:09.411 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.515639+0000", 2024-09-13T11:23:09.411 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.619e-06 2024-09-13T11:23:09.411 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.411 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.411 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:23:09.411 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.515650+0000", 2024-09-13T11:23:09.411 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.111e-05 2024-09-13T11:23:09.411 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.411 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.411 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:23:09.411 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.722004+0000", 2024-09-13T11:23:09.411 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.206354805 2024-09-13T11:23:09.412 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.412 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.412 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:23:09.412 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.722017+0000", 2024-09-13T11:23:09.412 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2840000000000001e-05 2024-09-13T11:23:09.412 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.412 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.412 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:23:09.412 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.722186+0000", 2024-09-13T11:23:09.412 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00016898299999999999 2024-09-13T11:23:09.412 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.412 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.413 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:23:09.413 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.722199+0000", 2024-09-13T11:23:09.413 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.289e-05 2024-09-13T11:23:09.413 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.413 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.413 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:23:09.413 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.722411+0000", 2024-09-13T11:23:09.413 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00021175799999999999 2024-09-13T11:23:09.413 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:09.413 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:23:09.413 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:09.413 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.413 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.413 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:25432 3.8 3:19671730:::benchmark_data_smithi177_31116_object25431:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e56)", 2024-09-13T11:23:09.414 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:04.717754+0000", 2024-09-13T11:23:09.414 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 4.5580069210000005, 2024-09-13T11:23:09.414 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.102412884, 2024-09-13T11:23:09.414 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:23:09.414 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:23:09.414 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:09.414 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:23:09.414 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:23:09.414 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:09.414 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 25432 2024-09-13T11:23:09.414 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.414 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:23:09.415 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.415 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:23:09.415 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.717754+0000", 2024-09-13T11:23:09.415 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:09.415 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.415 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.415 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:23:09.415 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.717754+0000", 2024-09-13T11:23:09.415 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:09.415 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.415 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.415 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:23:09.415 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.717759+0000", 2024-09-13T11:23:09.416 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.04e-06 2024-09-13T11:23:09.416 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.416 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.416 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:23:09.416 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.717820+0000", 2024-09-13T11:23:09.416 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.0900999999999998e-05 2024-09-13T11:23:09.416 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.416 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.416 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:23:09.416 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.717854+0000", 2024-09-13T11:23:09.416 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.3797000000000002e-05 2024-09-13T11:23:09.416 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.416 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.416 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:23:09.417 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.717871+0000", 2024-09-13T11:23:09.417 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.6898999999999999e-05 2024-09-13T11:23:09.417 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.417 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.417 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:23:09.417 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.717993+0000", 2024-09-13T11:23:09.417 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00012248399999999999 2024-09-13T11:23:09.417 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.417 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.417 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:23:09.417 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.718222+0000", 2024-09-13T11:23:09.417 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00022848500000000001 2024-09-13T11:23:09.417 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.417 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.418 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 1", 2024-09-13T11:23:09.418 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.718447+0000", 2024-09-13T11:23:09.418 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00022588400000000001 2024-09-13T11:23:09.418 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.418 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.418 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:23:09.418 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.720604+0000", 2024-09-13T11:23:09.418 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0021561430000000001 2024-09-13T11:23:09.418 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.418 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.418 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:09.418 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.820012+0000", 2024-09-13T11:23:09.418 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.099408924999999995 2024-09-13T11:23:09.418 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.419 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.419 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:23:09.419 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.820097+0000", 2024-09-13T11:23:09.419 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.4147999999999995e-05 2024-09-13T11:23:09.419 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.419 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.419 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:23:09.419 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.820167+0000", 2024-09-13T11:23:09.419 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.0178000000000005e-05 2024-09-13T11:23:09.419 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:09.419 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:23:09.419 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:09.419 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.419 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.419 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop_reply(client.4665.0:25432 3.8 e56/40 ondisk, result = 0)", 2024-09-13T11:23:09.420 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:04.721952+0000", 2024-09-13T11:23:09.420 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 4.5538085800000001, 2024-09-13T11:23:09.420 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.098227587000000005, 2024-09-13T11:23:09.420 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:23:09.420 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:23:09.420 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:23:09.420 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:23:09.420 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.420 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:23:09.420 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.721952+0000", 2024-09-13T11:23:09.420 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:09.420 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.420 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.421 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:23:09.421 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.721952+0000", 2024-09-13T11:23:09.421 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:09.421 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.421 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.421 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:23:09.421 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.721956+0000", 2024-09-13T11:23:09.421 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.8130000000000002e-06 2024-09-13T11:23:09.421 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.421 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.421 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:23:09.421 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.721967+0000", 2024-09-13T11:23:09.421 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1229e-05 2024-09-13T11:23:09.421 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.422 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.422 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:23:09.422 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.819880+0000", 2024-09-13T11:23:09.422 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.097912274999999993 2024-09-13T11:23:09.422 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.422 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.422 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:23:09.422 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.819894+0000", 2024-09-13T11:23:09.422 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.416e-05 2024-09-13T11:23:09.422 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.422 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.422 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:23:09.422 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.819993+0000", 2024-09-13T11:23:09.422 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.8859999999999999e-05 2024-09-13T11:23:09.423 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.423 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.423 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:23:09.423 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.820005+0000", 2024-09-13T11:23:09.423 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1960000000000001e-05 2024-09-13T11:23:09.423 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.423 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.423 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:23:09.423 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.820180+0000", 2024-09-13T11:23:09.423 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00017529000000000001 2024-09-13T11:23:09.423 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:09.423 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:23:09.423 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:09.423 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:09.424 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:23:09.424 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-13T11:23:09.425 INFO:teuthology.orchestra.run.smithi001.stdout:osd.12: {} 2024-09-13T11:23:09.425 INFO:teuthology.orchestra.run.smithi001.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-13T11:23:09.426 INFO:teuthology.orchestra.run.smithi001.stdout:osd.13: {} 2024-09-13T11:23:09.426 INFO:teuthology.orchestra.run.smithi001.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-13T11:23:09.426 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-09-13T11:23:09.427 INFO:teuthology.orchestra.run.smithi001.stdout:osd.14: {} 2024-09-13T11:23:09.427 INFO:teuthology.orchestra.run.smithi001.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-13T11:23:09.427 INFO:teuthology.orchestra.run.smithi001.stdout:osd.15: {} 2024-09-13T11:23:09.428 INFO:teuthology.orchestra.run.smithi001.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-13T11:23:09.429 INFO:tasks.ceph.osd.5.smithi092.stderr:2024-09-13T11:23:09.422+0000 7f6f0b705640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26346) UID: 0 2024-09-13T11:23:09.491 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15] out_osds: [] dead_osds: [] live_osds: [0, 4, 8, 12, 5, 9, 13, 2, 6, 10, 14, 3, 7, 11, 15, 1] 2024-09-13T11:23:09.491 INFO:tasks.thrashosds.thrasher:choose_action: min_in 2 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-13T11:23:09.492 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2024-09-13T11:23:09.492 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump -f json-pretty 2024-09-13T11:23:09.532 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-13T11:23:09.540 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-13T11:23:09.536+0000 7f6a68775640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26307) UID: 0 2024-09-13T11:23:09.568 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-13T11:23:09.568 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [], 2024-09-13T11:23:09.568 INFO:teuthology.orchestra.run.smithi162.stdout: "num_ops": 0 2024-09-13T11:23:09.568 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-13T11:23:09.582 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-09-13T11:23:09.627 INFO:tasks.ceph.osd.1.smithi092.stderr:2024-09-13T11:23:09.622+0000 7fe7789c5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33718) UID: 0 2024-09-13T11:23:09.703 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-13T11:23:09.704 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [], 2024-09-13T11:23:09.704 INFO:teuthology.orchestra.run.smithi162.stdout: "complaint_time": 30, 2024-09-13T11:23:09.704 INFO:teuthology.orchestra.run.smithi162.stdout: "num_blocked_ops": 0 2024-09-13T11:23:09.704 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-13T11:23:09.719 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-09-13T11:23:09.765 INFO:tasks.ceph.osd.14.smithi162.stderr:2024-09-13T11:23:09.762+0000 7f235c0db640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26319) UID: 0 2024-09-13T11:23:09.829 INFO:tasks.ceph.osd.3.smithi177.stderr:2024-09-13T11:23:09.826+0000 7f10adba4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26574) UID: 0 2024-09-13T11:23:09.849 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-13T11:23:09.849 INFO:teuthology.orchestra.run.smithi162.stdout: "size": 20, 2024-09-13T11:23:09.849 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 600, 2024-09-13T11:23:09.849 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [ 2024-09-13T11:23:09.849 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.850 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:2251 3.3 3:cdfe3214:::benchmark_data_smithi177_31116_object2250:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-09-13T11:23:09.850 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:22:50.241594+0000", 2024-09-13T11:23:09.850 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 19.604195441999998, 2024-09-13T11:23:09.850 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.033010455000000001, 2024-09-13T11:23:09.850 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:23:09.850 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:23:09.850 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:09.850 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:23:09.850 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:23:09.850 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:09.850 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 2251 2024-09-13T11:23:09.850 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.850 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:23:09.851 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.851 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:23:09.851 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:50.241594+0000", 2024-09-13T11:23:09.851 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:09.851 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.851 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.851 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:23:09.851 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:50.241594+0000", 2024-09-13T11:23:09.851 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:09.851 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.851 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.851 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:23:09.852 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:50.241596+0000", 2024-09-13T11:23:09.852 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.1179999999999999e-06 2024-09-13T11:23:09.852 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.852 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.852 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:23:09.852 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:50.241668+0000", 2024-09-13T11:23:09.852 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.2241000000000002e-05 2024-09-13T11:23:09.852 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.852 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.852 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:23:09.852 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:50.241685+0000", 2024-09-13T11:23:09.852 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.6829000000000001e-05 2024-09-13T11:23:09.852 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.853 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.853 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:23:09.853 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:50.241694+0000", 2024-09-13T11:23:09.853 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.8140000000000007e-06 2024-09-13T11:23:09.853 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.853 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.853 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:23:09.853 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:50.241804+0000", 2024-09-13T11:23:09.853 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000110481 2024-09-13T11:23:09.853 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.853 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.853 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:23:09.854 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:50.242059+0000", 2024-09-13T11:23:09.854 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00025521499999999999 2024-09-13T11:23:09.854 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.854 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.854 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 7", 2024-09-13T11:23:09.854 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:50.242367+0000", 2024-09-13T11:23:09.854 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00030775000000000003 2024-09-13T11:23:09.854 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.854 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.854 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:23:09.854 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:50.243771+0000", 2024-09-13T11:23:09.854 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001403877 2024-09-13T11:23:09.854 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.854 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.855 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:09.855 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:50.274487+0000", 2024-09-13T11:23:09.855 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.030716291999999999 2024-09-13T11:23:09.855 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.855 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.855 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:23:09.855 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:50.274552+0000", 2024-09-13T11:23:09.855 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.4818000000000002e-05 2024-09-13T11:23:09.855 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.855 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.855 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:23:09.855 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:50.274604+0000", 2024-09-13T11:23:09.855 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.202e-05 2024-09-13T11:23:09.855 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:09.856 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:23:09.856 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:09.856 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.856 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.856 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop_reply(client.4665.0:7655 3.3 e46/30 ondisk, result = 0)", 2024-09-13T11:23:09.856 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:22:53.009318+0000", 2024-09-13T11:23:09.856 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 16.836471374999999, 2024-09-13T11:23:09.856 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.037971667000000001, 2024-09-13T11:23:09.856 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:23:09.856 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:23:09.856 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:23:09.856 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:23:09.856 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.856 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:23:09.857 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:53.009318+0000", 2024-09-13T11:23:09.857 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:09.857 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.857 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.857 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:23:09.857 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:53.009318+0000", 2024-09-13T11:23:09.857 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:09.857 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.857 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.857 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:23:09.858 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:53.009321+0000", 2024-09-13T11:23:09.858 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.4850000000000001e-06 2024-09-13T11:23:09.858 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.858 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.858 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:23:09.858 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:53.009331+0000", 2024-09-13T11:23:09.858 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.4939999999999992e-06 2024-09-13T11:23:09.858 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.858 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.858 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:23:09.858 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:53.046792+0000", 2024-09-13T11:23:09.858 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.037460836999999997 2024-09-13T11:23:09.858 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.858 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.859 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:23:09.859 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:53.046809+0000", 2024-09-13T11:23:09.859 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.6965000000000001e-05 2024-09-13T11:23:09.859 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.859 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.859 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:23:09.859 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:53.046973+0000", 2024-09-13T11:23:09.859 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000164585 2024-09-13T11:23:09.859 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.859 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.859 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:23:09.859 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:53.046995+0000", 2024-09-13T11:23:09.859 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.1529e-05 2024-09-13T11:23:09.859 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.859 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.860 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:23:09.860 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:53.047290+0000", 2024-09-13T11:23:09.860 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000294772 2024-09-13T11:23:09.860 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:09.860 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:23:09.860 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:09.860 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.860 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.860 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:11801 3.3 3:cae0ce13:::benchmark_data_smithi177_31116_object11800:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-13T11:23:09.860 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:22:55.643405+0000", 2024-09-13T11:23:09.860 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 14.202383758, 2024-09-13T11:23:09.860 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.035702266000000003, 2024-09-13T11:23:09.861 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:23:09.861 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:23:09.861 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:09.861 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:23:09.861 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:23:09.861 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:09.861 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 11801 2024-09-13T11:23:09.861 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.861 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:23:09.861 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.861 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:23:09.861 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:55.643405+0000", 2024-09-13T11:23:09.861 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:09.862 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.862 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.862 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:23:09.862 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:55.643405+0000", 2024-09-13T11:23:09.862 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:09.862 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.862 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.862 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:23:09.862 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:55.643416+0000", 2024-09-13T11:23:09.862 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0054999999999999e-05 2024-09-13T11:23:09.862 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.862 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.862 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:23:09.862 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:55.643466+0000", 2024-09-13T11:23:09.862 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.0408999999999998e-05 2024-09-13T11:23:09.863 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.863 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.863 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:23:09.863 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:55.643490+0000", 2024-09-13T11:23:09.863 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.4128999999999999e-05 2024-09-13T11:23:09.863 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.863 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.863 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:23:09.863 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:55.643502+0000", 2024-09-13T11:23:09.863 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.149e-05 2024-09-13T11:23:09.863 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.863 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.863 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:23:09.863 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:55.643624+0000", 2024-09-13T11:23:09.864 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00012277300000000001 2024-09-13T11:23:09.864 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.864 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.864 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:23:09.864 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:55.643915+0000", 2024-09-13T11:23:09.864 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000290754 2024-09-13T11:23:09.864 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.864 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.864 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 7", 2024-09-13T11:23:09.864 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:55.644248+0000", 2024-09-13T11:23:09.864 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00033311099999999998 2024-09-13T11:23:09.864 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.865 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.865 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:23:09.865 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:55.645823+0000", 2024-09-13T11:23:09.865 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0015748369999999999 2024-09-13T11:23:09.865 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.865 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.865 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:09.865 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:55.678842+0000", 2024-09-13T11:23:09.865 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.033018919000000001 2024-09-13T11:23:09.865 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.866 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.866 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:23:09.866 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:55.679013+0000", 2024-09-13T11:23:09.866 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00017098600000000001 2024-09-13T11:23:09.866 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.866 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.866 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:23:09.866 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:55.679108+0000", 2024-09-13T11:23:09.866 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.4802999999999993e-05 2024-09-13T11:23:09.866 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:09.866 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:23:09.866 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:09.867 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.867 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.867 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop_reply(client.4665.0:15910 3.3 e52/30 ondisk, result = 0)", 2024-09-13T11:23:09.867 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:22:58.421847+0000", 2024-09-13T11:23:09.867 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 11.423941944999999, 2024-09-13T11:23:09.867 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.040755588000000002, 2024-09-13T11:23:09.867 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:23:09.867 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:23:09.867 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:23:09.867 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:23:09.867 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.867 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:23:09.867 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:58.421847+0000", 2024-09-13T11:23:09.867 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:09.868 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.868 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.868 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:23:09.868 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:58.421847+0000", 2024-09-13T11:23:09.868 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:09.868 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.868 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.868 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:23:09.868 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:58.421850+0000", 2024-09-13T11:23:09.868 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.8289999999999999e-06 2024-09-13T11:23:09.868 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.868 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.868 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:23:09.869 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:58.421859+0000", 2024-09-13T11:23:09.869 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.2650000000000002e-06 2024-09-13T11:23:09.869 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.869 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.869 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:23:09.869 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:58.460051+0000", 2024-09-13T11:23:09.869 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.038192049999999998 2024-09-13T11:23:09.869 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.869 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.869 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:23:09.869 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:58.460085+0000", 2024-09-13T11:23:09.869 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.3318999999999997e-05 2024-09-13T11:23:09.869 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.869 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.870 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:23:09.870 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:58.462416+0000", 2024-09-13T11:23:09.870 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0023308729999999998 2024-09-13T11:23:09.870 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.870 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.870 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:23:09.870 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:58.462425+0000", 2024-09-13T11:23:09.870 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.9250000000000001e-06 2024-09-13T11:23:09.870 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.870 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.870 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:23:09.870 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:58.462603+0000", 2024-09-13T11:23:09.870 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00017832699999999999 2024-09-13T11:23:09.870 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:09.871 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:23:09.871 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:09.871 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.871 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.871 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:20503 3.3 3:c2cf9bec:::benchmark_data_smithi177_31116_object20502:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e54)", 2024-09-13T11:23:09.871 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:00.934025+0000", 2024-09-13T11:23:09.871 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 8.9117644299999998, 2024-09-13T11:23:09.871 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.040488244999999999, 2024-09-13T11:23:09.871 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:23:09.871 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:23:09.871 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:09.871 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:23:09.871 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:23:09.871 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:09.871 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 20503 2024-09-13T11:23:09.872 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.872 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:23:09.872 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.872 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:23:09.872 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:00.934025+0000", 2024-09-13T11:23:09.872 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:09.872 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.872 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.872 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:23:09.872 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:00.934025+0000", 2024-09-13T11:23:09.872 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:09.872 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.872 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.873 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:23:09.873 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:00.934028+0000", 2024-09-13T11:23:09.873 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.7199999999999998e-06 2024-09-13T11:23:09.873 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.873 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.873 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:23:09.873 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:00.934065+0000", 2024-09-13T11:23:09.873 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.7199000000000001e-05 2024-09-13T11:23:09.873 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.873 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.873 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:23:09.873 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:00.934082+0000", 2024-09-13T11:23:09.873 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.7286e-05 2024-09-13T11:23:09.874 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.874 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.874 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:23:09.874 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:00.934096+0000", 2024-09-13T11:23:09.874 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4429e-05 2024-09-13T11:23:09.874 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.874 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.874 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:23:09.874 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:00.934178+0000", 2024-09-13T11:23:09.874 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.1284000000000003e-05 2024-09-13T11:23:09.874 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.874 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.874 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:23:09.874 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:00.934369+0000", 2024-09-13T11:23:09.874 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00019146699999999999 2024-09-13T11:23:09.875 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.875 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.875 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 7", 2024-09-13T11:23:09.875 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:00.934566+0000", 2024-09-13T11:23:09.875 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000196646 2024-09-13T11:23:09.875 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.875 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.875 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:23:09.875 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:00.935850+0000", 2024-09-13T11:23:09.875 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0012837930000000001 2024-09-13T11:23:09.875 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.875 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.875 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:09.875 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:00.974449+0000", 2024-09-13T11:23:09.876 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.038599708000000003 2024-09-13T11:23:09.876 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.876 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.876 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:23:09.876 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:00.974484+0000", 2024-09-13T11:23:09.876 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.4311999999999999e-05 2024-09-13T11:23:09.876 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.876 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.876 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:23:09.876 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:00.974513+0000", 2024-09-13T11:23:09.876 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.9400999999999999e-05 2024-09-13T11:23:09.876 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:09.876 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:23:09.877 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:09.877 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.877 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.877 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop_reply(client.4665.0:20503 3.3 e54/30 ondisk, result = 0)", 2024-09-13T11:23:09.877 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:00.936548+0000", 2024-09-13T11:23:09.877 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 8.9092410310000005, 2024-09-13T11:23:09.877 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.037970861000000002, 2024-09-13T11:23:09.877 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:23:09.877 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:23:09.877 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:23:09.877 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:23:09.877 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.877 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:23:09.877 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:00.936548+0000", 2024-09-13T11:23:09.878 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:09.878 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.878 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.878 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:23:09.878 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:00.936548+0000", 2024-09-13T11:23:09.878 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:09.878 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.878 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.878 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:23:09.878 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:00.936550+0000", 2024-09-13T11:23:09.878 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3090000000000001e-06 2024-09-13T11:23:09.878 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.878 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.879 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:23:09.879 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:00.936554+0000", 2024-09-13T11:23:09.879 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.2540000000000003e-06 2024-09-13T11:23:09.879 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.879 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.879 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:23:09.879 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:00.974382+0000", 2024-09-13T11:23:09.879 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.037828529 2024-09-13T11:23:09.879 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.879 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.879 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:23:09.879 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:00.974389+0000", 2024-09-13T11:23:09.879 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.5189999999999998e-06 2024-09-13T11:23:09.880 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.880 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.880 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:23:09.880 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:00.974440+0000", 2024-09-13T11:23:09.880 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.1201e-05 2024-09-13T11:23:09.880 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.880 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.880 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:23:09.880 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:00.974445+0000", 2024-09-13T11:23:09.880 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.6750000000000001e-06 2024-09-13T11:23:09.880 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.880 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.880 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:23:09.880 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:00.974519+0000", 2024-09-13T11:23:09.881 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.4374000000000003e-05 2024-09-13T11:23:09.881 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:09.881 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:23:09.881 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:09.881 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.881 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.881 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:25019 3.3 3:c7aa74ef:::benchmark_data_smithi177_31116_object25018:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e55)", 2024-09-13T11:23:09.881 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:04.041317+0000", 2024-09-13T11:23:09.881 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 5.8044725459999995, 2024-09-13T11:23:09.881 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.046382957000000002, 2024-09-13T11:23:09.881 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:23:09.881 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:23:09.881 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:09.882 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:23:09.882 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:23:09.882 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:09.882 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 25019 2024-09-13T11:23:09.882 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.882 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:23:09.882 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.882 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:23:09.882 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.041317+0000", 2024-09-13T11:23:09.882 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:09.882 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.882 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.882 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:23:09.882 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.041317+0000", 2024-09-13T11:23:09.883 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:09.883 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.883 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.883 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:23:09.883 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.041325+0000", 2024-09-13T11:23:09.883 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.2770000000000005e-06 2024-09-13T11:23:09.883 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.883 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.883 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:23:09.883 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.041384+0000", 2024-09-13T11:23:09.883 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.8984000000000003e-05 2024-09-13T11:23:09.883 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.883 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.883 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:23:09.883 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.041407+0000", 2024-09-13T11:23:09.884 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.3031e-05 2024-09-13T11:23:09.884 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.884 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.884 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:23:09.884 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.041419+0000", 2024-09-13T11:23:09.884 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2014000000000001e-05 2024-09-13T11:23:09.884 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.884 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.884 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:23:09.884 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.041575+0000", 2024-09-13T11:23:09.884 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00015568400000000001 2024-09-13T11:23:09.884 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.884 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.885 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:23:09.885 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.041927+0000", 2024-09-13T11:23:09.885 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00035198199999999998 2024-09-13T11:23:09.885 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.885 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.885 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 7", 2024-09-13T11:23:09.885 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.042213+0000", 2024-09-13T11:23:09.885 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00028670300000000002 2024-09-13T11:23:09.885 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.885 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.885 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:23:09.885 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.044793+0000", 2024-09-13T11:23:09.885 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.002579928 2024-09-13T11:23:09.885 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.886 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.886 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:09.886 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.087540+0000", 2024-09-13T11:23:09.886 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.042746900999999997 2024-09-13T11:23:09.886 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.886 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.886 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:23:09.886 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.087631+0000", 2024-09-13T11:23:09.886 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.1230999999999994e-05 2024-09-13T11:23:09.886 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.886 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.886 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:23:09.886 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.087700+0000", 2024-09-13T11:23:09.886 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.8221999999999998e-05 2024-09-13T11:23:09.887 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:09.887 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:23:09.887 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:09.887 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.887 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.887 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:25110 3.3 3:c25a0b62:::benchmark_data_smithi177_31116_object25109:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e55)", 2024-09-13T11:23:09.887 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:04.094554+0000", 2024-09-13T11:23:09.887 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 5.7512348150000001, 2024-09-13T11:23:09.887 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.052119406, 2024-09-13T11:23:09.887 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:23:09.887 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:23:09.887 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:09.887 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:23:09.887 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:23:09.888 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:09.888 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 25110 2024-09-13T11:23:09.888 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.888 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:23:09.888 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.888 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:23:09.888 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.094554+0000", 2024-09-13T11:23:09.888 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:09.888 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.888 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.888 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:23:09.888 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.094554+0000", 2024-09-13T11:23:09.888 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:09.888 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.889 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.889 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:23:09.889 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.094558+0000", 2024-09-13T11:23:09.889 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.6859999999999999e-06 2024-09-13T11:23:09.889 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.889 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.890 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:23:09.890 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.094612+0000", 2024-09-13T11:23:09.890 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.4185999999999999e-05 2024-09-13T11:23:09.890 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.890 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.890 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:23:09.891 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.094642+0000", 2024-09-13T11:23:09.891 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.0199e-05 2024-09-13T11:23:09.891 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.891 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.891 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:23:09.891 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.094661+0000", 2024-09-13T11:23:09.891 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.8023e-05 2024-09-13T11:23:09.891 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.891 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.891 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:23:09.891 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.094785+0000", 2024-09-13T11:23:09.892 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00012498699999999999 2024-09-13T11:23:09.892 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.892 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.892 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:23:09.892 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.095012+0000", 2024-09-13T11:23:09.892 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000226238 2024-09-13T11:23:09.892 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.892 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.892 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 7", 2024-09-13T11:23:09.892 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.095257+0000", 2024-09-13T11:23:09.892 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00024494000000000002 2024-09-13T11:23:09.892 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.892 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.893 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:23:09.893 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.097213+0000", 2024-09-13T11:23:09.893 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001956312 2024-09-13T11:23:09.893 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.893 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.893 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:09.893 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.146492+0000", 2024-09-13T11:23:09.893 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.049278897000000002 2024-09-13T11:23:09.893 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.893 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.893 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:23:09.893 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.146593+0000", 2024-09-13T11:23:09.893 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00010145999999999999 2024-09-13T11:23:09.893 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.894 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.894 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:23:09.894 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.146674+0000", 2024-09-13T11:23:09.894 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.0477999999999998e-05 2024-09-13T11:23:09.894 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:09.894 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:23:09.894 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:09.894 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.894 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.894 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:25231 3.3 3:c5e4549b:::benchmark_data_smithi177_31116_object25230:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e55)", 2024-09-13T11:23:09.894 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:04.239701+0000", 2024-09-13T11:23:09.894 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 5.6060882979999995, 2024-09-13T11:23:09.894 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.038002049000000003, 2024-09-13T11:23:09.894 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:23:09.895 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:23:09.895 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:09.895 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:23:09.895 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:23:09.895 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:09.895 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 25231 2024-09-13T11:23:09.895 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.895 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:23:09.895 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.895 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:23:09.895 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.239701+0000", 2024-09-13T11:23:09.895 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:09.895 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.896 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.896 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:23:09.896 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.239701+0000", 2024-09-13T11:23:09.896 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:09.896 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.896 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.896 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:23:09.896 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.239708+0000", 2024-09-13T11:23:09.896 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.8499999999999996e-06 2024-09-13T11:23:09.896 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.896 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.896 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:23:09.896 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.239760+0000", 2024-09-13T11:23:09.896 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.1955e-05 2024-09-13T11:23:09.897 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.897 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.897 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:23:09.897 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.239781+0000", 2024-09-13T11:23:09.897 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.1330000000000001e-05 2024-09-13T11:23:09.897 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.897 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.897 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:23:09.897 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.239794+0000", 2024-09-13T11:23:09.897 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3039e-05 2024-09-13T11:23:09.897 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.897 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.897 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:23:09.898 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.239887+0000", 2024-09-13T11:23:09.898 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.2764999999999997e-05 2024-09-13T11:23:09.898 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.898 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.898 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:23:09.898 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.240086+0000", 2024-09-13T11:23:09.898 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00019904499999999999 2024-09-13T11:23:09.898 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.898 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.898 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 7", 2024-09-13T11:23:09.898 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.240428+0000", 2024-09-13T11:23:09.898 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00034170199999999999 2024-09-13T11:23:09.899 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.899 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.899 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:23:09.899 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.242679+0000", 2024-09-13T11:23:09.899 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00225173 2024-09-13T11:23:09.899 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.899 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.899 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:09.899 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.277454+0000", 2024-09-13T11:23:09.899 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.034774741999999997 2024-09-13T11:23:09.899 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.899 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.899 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:23:09.899 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.277542+0000", 2024-09-13T11:23:09.900 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.7521000000000005e-05 2024-09-13T11:23:09.900 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.900 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.900 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:23:09.900 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.277703+0000", 2024-09-13T11:23:09.900 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00016137000000000001 2024-09-13T11:23:09.900 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:09.900 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:23:09.900 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:09.900 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.900 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.900 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop_reply(client.4665.0:25231 3.3 e55/30 ondisk, result = 0)", 2024-09-13T11:23:09.900 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:04.243524+0000", 2024-09-13T11:23:09.901 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 5.6022649449999999, 2024-09-13T11:23:09.901 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.034205897999999998, 2024-09-13T11:23:09.901 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:23:09.901 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:23:09.901 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:23:09.901 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:23:09.901 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.901 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:23:09.901 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.243524+0000", 2024-09-13T11:23:09.901 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:09.901 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.901 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.901 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:23:09.901 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.243524+0000", 2024-09-13T11:23:09.902 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:09.902 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.902 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.902 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:23:09.902 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.243527+0000", 2024-09-13T11:23:09.902 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.0489999999999999e-06 2024-09-13T11:23:09.902 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.902 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.902 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:23:09.902 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.243536+0000", 2024-09-13T11:23:09.902 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.8869999999999998e-06 2024-09-13T11:23:09.902 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.902 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.902 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:23:09.903 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.277288+0000", 2024-09-13T11:23:09.903 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.033752200000000003 2024-09-13T11:23:09.903 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.903 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.903 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:23:09.903 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.277311+0000", 2024-09-13T11:23:09.903 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.3071999999999999e-05 2024-09-13T11:23:09.903 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.903 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.903 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:23:09.903 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.277433+0000", 2024-09-13T11:23:09.903 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00012128700000000001 2024-09-13T11:23:09.903 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.903 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.904 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:23:09.904 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.277445+0000", 2024-09-13T11:23:09.904 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.22e-05 2024-09-13T11:23:09.904 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.904 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.904 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:23:09.904 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.277730+0000", 2024-09-13T11:23:09.904 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00028520299999999999 2024-09-13T11:23:09.904 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:09.904 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:23:09.904 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:09.904 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.904 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.905 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:25257 3.3 3:cb7fec05:::benchmark_data_smithi177_31116_object25256:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e55)", 2024-09-13T11:23:09.905 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:04.273763+0000", 2024-09-13T11:23:09.905 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 5.5720258620000003, 2024-09-13T11:23:09.905 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.035474191000000002, 2024-09-13T11:23:09.905 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:23:09.905 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:23:09.905 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:09.905 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:23:09.905 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:23:09.905 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:09.905 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 25257 2024-09-13T11:23:09.905 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.905 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:23:09.905 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.906 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:23:09.906 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.273763+0000", 2024-09-13T11:23:09.906 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:09.906 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.906 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.906 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:23:09.906 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.273763+0000", 2024-09-13T11:23:09.906 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:09.906 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.906 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.906 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:23:09.906 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.273767+0000", 2024-09-13T11:23:09.906 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.1060000000000002e-06 2024-09-13T11:23:09.906 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.907 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.907 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:23:09.907 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.273841+0000", 2024-09-13T11:23:09.907 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.3648999999999999e-05 2024-09-13T11:23:09.907 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.907 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.907 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:23:09.907 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.273882+0000", 2024-09-13T11:23:09.907 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.0425e-05 2024-09-13T11:23:09.907 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.907 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.907 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:23:09.907 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.273904+0000", 2024-09-13T11:23:09.907 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.2466000000000001e-05 2024-09-13T11:23:09.908 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.908 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.908 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:23:09.908 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.274046+0000", 2024-09-13T11:23:09.908 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00014219999999999999 2024-09-13T11:23:09.908 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.908 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.908 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:23:09.908 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.274280+0000", 2024-09-13T11:23:09.908 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00023350500000000001 2024-09-13T11:23:09.908 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.908 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.909 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 7", 2024-09-13T11:23:09.909 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.274506+0000", 2024-09-13T11:23:09.909 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00022615800000000001 2024-09-13T11:23:09.909 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.909 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.909 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:23:09.909 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.277142+0000", 2024-09-13T11:23:09.909 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0026361700000000002 2024-09-13T11:23:09.909 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.909 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.909 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:09.909 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.309084+0000", 2024-09-13T11:23:09.910 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.031942269000000002 2024-09-13T11:23:09.910 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.910 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.910 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:23:09.910 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.309165+0000", 2024-09-13T11:23:09.910 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.0191999999999994e-05 2024-09-13T11:23:09.910 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.910 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.910 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:23:09.910 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.309238+0000", 2024-09-13T11:23:09.910 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.3051000000000002e-05 2024-09-13T11:23:09.910 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:09.910 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:23:09.910 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:09.911 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.911 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.911 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop_reply(client.4665.0:25257 3.3 e55/30 ondisk, result = 0)", 2024-09-13T11:23:09.911 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:04.277239+0000", 2024-09-13T11:23:09.911 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 5.5685500350000003, 2024-09-13T11:23:09.911 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.032015871000000001, 2024-09-13T11:23:09.911 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:23:09.911 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:23:09.911 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:23:09.911 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:23:09.911 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.911 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:23:09.911 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.277239+0000", 2024-09-13T11:23:09.912 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:09.912 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.912 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.912 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:23:09.912 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.277239+0000", 2024-09-13T11:23:09.912 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:09.912 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.912 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.912 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:23:09.912 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.277243+0000", 2024-09-13T11:23:09.912 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.6500000000000002e-06 2024-09-13T11:23:09.912 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.912 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.913 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:23:09.913 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.277252+0000", 2024-09-13T11:23:09.913 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.5300000000000002e-06 2024-09-13T11:23:09.913 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.913 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.913 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:23:09.913 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.308932+0000", 2024-09-13T11:23:09.913 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.031680080999999999 2024-09-13T11:23:09.913 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.913 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.913 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:23:09.913 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.308945+0000", 2024-09-13T11:23:09.913 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2687000000000001e-05 2024-09-13T11:23:09.913 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.914 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.914 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:23:09.914 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.309061+0000", 2024-09-13T11:23:09.914 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00011592 2024-09-13T11:23:09.914 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.914 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.914 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:23:09.914 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.309076+0000", 2024-09-13T11:23:09.914 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5034e-05 2024-09-13T11:23:09.914 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.914 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.914 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:23:09.914 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.309255+0000", 2024-09-13T11:23:09.915 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00017896899999999999 2024-09-13T11:23:09.915 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:09.915 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:23:09.915 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:09.915 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.915 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.915 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:25286 3.3 3:ccb7e8b2:::benchmark_data_smithi177_31116_object25285:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e55)", 2024-09-13T11:23:09.915 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:04.305812+0000", 2024-09-13T11:23:09.915 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 5.539976996, 2024-09-13T11:23:09.915 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.036073121, 2024-09-13T11:23:09.915 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:23:09.915 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:23:09.915 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:09.915 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:23:09.916 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:23:09.916 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:09.916 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 25286 2024-09-13T11:23:09.916 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.916 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:23:09.916 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.916 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:23:09.916 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.305812+0000", 2024-09-13T11:23:09.916 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:09.916 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.916 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.916 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:23:09.916 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.305812+0000", 2024-09-13T11:23:09.916 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:09.917 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.917 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.917 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:23:09.917 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.305817+0000", 2024-09-13T11:23:09.917 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.9529999999999998e-06 2024-09-13T11:23:09.917 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.917 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.917 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:23:09.917 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.305895+0000", 2024-09-13T11:23:09.917 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.8108000000000005e-05 2024-09-13T11:23:09.917 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.917 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.917 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:23:09.918 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.305934+0000", 2024-09-13T11:23:09.918 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.8229000000000003e-05 2024-09-13T11:23:09.918 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.918 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.918 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:23:09.918 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.305956+0000", 2024-09-13T11:23:09.918 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.2269999999999999e-05 2024-09-13T11:23:09.918 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.918 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.918 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:23:09.918 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.306104+0000", 2024-09-13T11:23:09.918 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00014809600000000001 2024-09-13T11:23:09.918 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.918 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.918 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:23:09.919 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.306341+0000", 2024-09-13T11:23:09.919 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000236894 2024-09-13T11:23:09.919 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.919 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.919 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 7", 2024-09-13T11:23:09.919 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.306570+0000", 2024-09-13T11:23:09.919 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00022894199999999999 2024-09-13T11:23:09.919 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.919 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.919 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:23:09.919 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.309277+0000", 2024-09-13T11:23:09.919 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0027069860000000002 2024-09-13T11:23:09.919 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.919 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.920 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:09.920 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.341727+0000", 2024-09-13T11:23:09.920 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.032450237 2024-09-13T11:23:09.920 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.920 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.920 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:23:09.920 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.341816+0000", 2024-09-13T11:23:09.920 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.9297999999999998e-05 2024-09-13T11:23:09.920 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.920 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.920 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:23:09.920 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.341885+0000", 2024-09-13T11:23:09.920 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.9108000000000003e-05 2024-09-13T11:23:09.921 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:09.921 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:23:09.921 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:09.921 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.921 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.921 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop_reply(client.4665.0:25286 3.3 e55/30 ondisk, result = 0)", 2024-09-13T11:23:09.921 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:04.308887+0000", 2024-09-13T11:23:09.921 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 5.536902156, 2024-09-13T11:23:09.921 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.033008603999999997, 2024-09-13T11:23:09.921 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:23:09.921 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:23:09.921 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:23:09.921 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:23:09.922 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.922 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:23:09.922 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.308887+0000", 2024-09-13T11:23:09.922 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:09.922 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.922 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.922 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:23:09.922 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.308887+0000", 2024-09-13T11:23:09.922 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:09.922 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.922 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.922 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:23:09.922 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.308890+0000", 2024-09-13T11:23:09.923 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.6400000000000001e-06 2024-09-13T11:23:09.923 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.923 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.923 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:23:09.923 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.308898+0000", 2024-09-13T11:23:09.923 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.67e-06 2024-09-13T11:23:09.923 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.923 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.923 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:23:09.923 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.341563+0000", 2024-09-13T11:23:09.923 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.032664385999999997 2024-09-13T11:23:09.923 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.923 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.923 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:23:09.923 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.341576+0000", 2024-09-13T11:23:09.924 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3546e-05 2024-09-13T11:23:09.924 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.924 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.924 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:23:09.924 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.341703+0000", 2024-09-13T11:23:09.924 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00012625700000000001 2024-09-13T11:23:09.924 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.924 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.924 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:23:09.924 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.341716+0000", 2024-09-13T11:23:09.924 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3529999999999999e-05 2024-09-13T11:23:09.924 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.924 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.924 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:23:09.925 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.341896+0000", 2024-09-13T11:23:09.925 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00017957499999999999 2024-09-13T11:23:09.925 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:09.925 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:23:09.925 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:09.925 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.925 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.925 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:25335 3.3 3:c8830d6d:::benchmark_data_smithi177_31116_object25334:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e55)", 2024-09-13T11:23:09.925 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:04.381894+0000", 2024-09-13T11:23:09.925 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 5.4638952100000004, 2024-09-13T11:23:09.925 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.050377281000000003, 2024-09-13T11:23:09.925 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:23:09.925 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:23:09.926 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:09.926 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:23:09.926 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:23:09.926 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:09.926 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 25335 2024-09-13T11:23:09.926 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.926 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:23:09.926 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.926 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:23:09.926 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.381894+0000", 2024-09-13T11:23:09.926 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:09.926 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.926 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.927 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:23:09.927 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.381894+0000", 2024-09-13T11:23:09.927 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:09.927 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.927 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.927 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:23:09.927 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.381898+0000", 2024-09-13T11:23:09.927 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.5329999999999999e-06 2024-09-13T11:23:09.927 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.927 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.927 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:23:09.927 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.381955+0000", 2024-09-13T11:23:09.927 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.7015000000000002e-05 2024-09-13T11:23:09.927 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.928 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.928 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:23:09.928 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.381983+0000", 2024-09-13T11:23:09.928 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.7966e-05 2024-09-13T11:23:09.928 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.928 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.928 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:23:09.928 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.381997+0000", 2024-09-13T11:23:09.928 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4547e-05 2024-09-13T11:23:09.928 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.928 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.928 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:23:09.928 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.382106+0000", 2024-09-13T11:23:09.929 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00010883999999999999 2024-09-13T11:23:09.929 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.929 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.929 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:23:09.929 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.382327+0000", 2024-09-13T11:23:09.929 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00022106200000000001 2024-09-13T11:23:09.929 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.929 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.929 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 7", 2024-09-13T11:23:09.929 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.382549+0000", 2024-09-13T11:23:09.929 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000222034 2024-09-13T11:23:09.929 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.929 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.929 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:23:09.930 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.384911+0000", 2024-09-13T11:23:09.930 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0023616739999999998 2024-09-13T11:23:09.930 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.930 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.930 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:09.930 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.432107+0000", 2024-09-13T11:23:09.930 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.047196693999999997 2024-09-13T11:23:09.930 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.930 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.930 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:23:09.930 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.432198+0000", 2024-09-13T11:23:09.930 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.1033999999999996e-05 2024-09-13T11:23:09.930 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.930 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.931 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:23:09.931 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.432271+0000", 2024-09-13T11:23:09.931 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.2881999999999998e-05 2024-09-13T11:23:09.931 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:09.931 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:23:09.931 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:09.931 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.931 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.931 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop_reply(client.4665.0:25335 3.3 e55/30 ondisk, result = 0)", 2024-09-13T11:23:09.931 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:04.385453+0000", 2024-09-13T11:23:09.931 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 5.4603359649999996, 2024-09-13T11:23:09.931 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.046835388999999998, 2024-09-13T11:23:09.931 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:23:09.931 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:23:09.932 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:23:09.932 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:23:09.932 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.932 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:23:09.932 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.385453+0000", 2024-09-13T11:23:09.932 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:09.932 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.932 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.932 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:23:09.932 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.385453+0000", 2024-09-13T11:23:09.932 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:09.932 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.932 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.932 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:23:09.933 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.385456+0000", 2024-09-13T11:23:09.933 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.9859999999999999e-06 2024-09-13T11:23:09.933 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.933 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.933 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:23:09.933 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.385465+0000", 2024-09-13T11:23:09.933 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.2199999999999998e-06 2024-09-13T11:23:09.933 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.933 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.933 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:23:09.933 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.431957+0000", 2024-09-13T11:23:09.933 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.046491167 2024-09-13T11:23:09.933 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.933 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.934 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:23:09.934 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.431970+0000", 2024-09-13T11:23:09.934 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.362e-05 2024-09-13T11:23:09.934 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.934 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.934 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:23:09.934 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.432084+0000", 2024-09-13T11:23:09.934 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000113514 2024-09-13T11:23:09.934 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.934 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.934 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:23:09.934 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.432096+0000", 2024-09-13T11:23:09.934 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2687000000000001e-05 2024-09-13T11:23:09.934 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.935 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.935 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:23:09.935 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.432289+0000", 2024-09-13T11:23:09.935 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00019219500000000001 2024-09-13T11:23:09.935 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:09.935 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:23:09.935 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:09.935 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.935 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.935 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:25365 3.3 3:c0d2f9d0:::benchmark_data_smithi177_31116_object25364:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e55)", 2024-09-13T11:23:09.935 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:04.427876+0000", 2024-09-13T11:23:09.935 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 5.4179130449999997, 2024-09-13T11:23:09.935 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.042917626, 2024-09-13T11:23:09.935 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:23:09.936 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:23:09.936 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:09.936 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:23:09.936 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:23:09.936 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:09.936 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 25365 2024-09-13T11:23:09.936 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.936 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:23:09.936 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.936 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:23:09.936 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.427876+0000", 2024-09-13T11:23:09.936 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:09.936 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.936 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.937 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:23:09.937 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.427876+0000", 2024-09-13T11:23:09.937 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:09.937 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.937 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.937 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:23:09.937 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.427881+0000", 2024-09-13T11:23:09.937 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.4970000000000003e-06 2024-09-13T11:23:09.937 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.937 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.937 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:23:09.937 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.427941+0000", 2024-09-13T11:23:09.937 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.0645e-05 2024-09-13T11:23:09.938 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.938 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.938 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:23:09.938 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.427973+0000", 2024-09-13T11:23:09.938 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.2146e-05 2024-09-13T11:23:09.938 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.938 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.938 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:23:09.938 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.427992+0000", 2024-09-13T11:23:09.938 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.8983000000000001e-05 2024-09-13T11:23:09.938 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.938 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.938 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:23:09.939 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.428134+0000", 2024-09-13T11:23:09.939 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00014130300000000001 2024-09-13T11:23:09.939 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.939 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.939 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:23:09.939 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.428360+0000", 2024-09-13T11:23:09.939 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000226555 2024-09-13T11:23:09.939 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.939 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.939 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 7", 2024-09-13T11:23:09.939 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.428586+0000", 2024-09-13T11:23:09.939 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00022557100000000001 2024-09-13T11:23:09.939 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.939 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.940 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:23:09.940 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.430881+0000", 2024-09-13T11:23:09.940 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0022950959999999999 2024-09-13T11:23:09.940 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.940 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.940 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:09.940 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.470628+0000", 2024-09-13T11:23:09.940 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.039746803999999997 2024-09-13T11:23:09.940 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.940 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.940 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:23:09.940 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.470717+0000", 2024-09-13T11:23:09.940 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.9520999999999999e-05 2024-09-13T11:23:09.940 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.941 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.941 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:23:09.941 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.470794+0000", 2024-09-13T11:23:09.941 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.6505e-05 2024-09-13T11:23:09.941 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:09.941 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:23:09.941 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:09.941 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.941 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.941 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop_reply(client.4665.0:25365 3.3 e55/30 ondisk, result = 0)", 2024-09-13T11:23:09.941 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:04.431913+0000", 2024-09-13T11:23:09.941 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 5.4138759710000004, 2024-09-13T11:23:09.941 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.038893091999999997, 2024-09-13T11:23:09.941 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:23:09.942 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:23:09.942 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:23:09.942 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:23:09.942 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.942 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:23:09.942 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.431913+0000", 2024-09-13T11:23:09.942 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:09.942 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.942 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.942 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:23:09.942 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.431913+0000", 2024-09-13T11:23:09.942 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:09.942 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.942 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.942 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:23:09.943 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.431916+0000", 2024-09-13T11:23:09.943 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.7439999999999999e-06 2024-09-13T11:23:09.943 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.943 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.943 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:23:09.943 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.431925+0000", 2024-09-13T11:23:09.943 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.1400000000000006e-06 2024-09-13T11:23:09.943 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.943 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.943 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:23:09.943 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.468467+0000", 2024-09-13T11:23:09.943 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.036541975999999997 2024-09-13T11:23:09.943 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.943 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.944 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:23:09.944 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.468487+0000", 2024-09-13T11:23:09.944 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.9579999999999999e-05 2024-09-13T11:23:09.944 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.944 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.944 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:23:09.944 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.470605+0000", 2024-09-13T11:23:09.944 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0021182330000000002 2024-09-13T11:23:09.944 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.944 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.944 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:23:09.944 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.470619+0000", 2024-09-13T11:23:09.944 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3613e-05 2024-09-13T11:23:09.945 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.945 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.945 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:23:09.945 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.470806+0000", 2024-09-13T11:23:09.945 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000187806 2024-09-13T11:23:09.945 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:09.945 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:23:09.945 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:09.945 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.945 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.945 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:25395 3.3 3:c1df8ccf:::benchmark_data_smithi177_31116_object25394:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e55)", 2024-09-13T11:23:09.945 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:04.465425+0000", 2024-09-13T11:23:09.945 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 5.3803638759999997, 2024-09-13T11:23:09.945 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.18641877200000001, 2024-09-13T11:23:09.946 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:23:09.946 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:23:09.946 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:09.946 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:23:09.946 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:23:09.946 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:09.946 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 25395 2024-09-13T11:23:09.946 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.946 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:23:09.946 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.946 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:23:09.946 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.465425+0000", 2024-09-13T11:23:09.946 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:09.946 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.947 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.947 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:23:09.947 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.465425+0000", 2024-09-13T11:23:09.947 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:09.947 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.947 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.947 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:23:09.947 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.465427+0000", 2024-09-13T11:23:09.947 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.9970000000000001e-06 2024-09-13T11:23:09.947 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.947 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.947 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:23:09.947 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.465516+0000", 2024-09-13T11:23:09.948 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.8181000000000002e-05 2024-09-13T11:23:09.948 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.948 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.948 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:23:09.948 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.465534+0000", 2024-09-13T11:23:09.948 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.8204000000000001e-05 2024-09-13T11:23:09.948 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.948 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.948 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:23:09.948 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.465548+0000", 2024-09-13T11:23:09.948 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4195999999999999e-05 2024-09-13T11:23:09.948 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.948 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.948 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:23:09.949 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.468249+0000", 2024-09-13T11:23:09.949 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0027014729999999998 2024-09-13T11:23:09.949 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.949 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.949 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:23:09.949 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.468626+0000", 2024-09-13T11:23:09.949 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000376721 2024-09-13T11:23:09.949 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.949 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.949 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 7", 2024-09-13T11:23:09.949 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.469016+0000", 2024-09-13T11:23:09.949 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00038975000000000001 2024-09-13T11:23:09.949 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.950 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.950 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:23:09.950 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.471140+0000", 2024-09-13T11:23:09.950 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0021243400000000002 2024-09-13T11:23:09.950 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.950 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.950 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:09.950 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.651698+0000", 2024-09-13T11:23:09.950 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.18055781700000001 2024-09-13T11:23:09.950 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.950 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.950 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:23:09.950 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.651781+0000", 2024-09-13T11:23:09.950 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.3350999999999995e-05 2024-09-13T11:23:09.951 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.951 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.951 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:23:09.951 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.651844+0000", 2024-09-13T11:23:09.951 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.2742000000000003e-05 2024-09-13T11:23:09.951 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:09.951 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:23:09.951 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:09.951 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.951 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.951 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop_reply(client.4665.0:25395 3.3 e55/30 ondisk, result = 0)", 2024-09-13T11:23:09.951 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:04.471832+0000", 2024-09-13T11:23:09.951 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 5.3739575740000003, 2024-09-13T11:23:09.951 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.18002319, 2024-09-13T11:23:09.952 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:23:09.952 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:23:09.952 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:23:09.952 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:23:09.952 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.952 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:23:09.952 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.471832+0000", 2024-09-13T11:23:09.952 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:09.952 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.952 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.952 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:23:09.952 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.471832+0000", 2024-09-13T11:23:09.952 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:09.952 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.953 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.953 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:23:09.953 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.471840+0000", 2024-09-13T11:23:09.953 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.7900000000000005e-06 2024-09-13T11:23:09.953 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.953 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.953 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:23:09.953 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.471850+0000", 2024-09-13T11:23:09.953 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.3899999999999999e-06 2024-09-13T11:23:09.953 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.953 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.953 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:23:09.953 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.651528+0000", 2024-09-13T11:23:09.953 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.17967789100000001 2024-09-13T11:23:09.953 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.954 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.954 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:23:09.954 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.651540+0000", 2024-09-13T11:23:09.954 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2e-05 2024-09-13T11:23:09.954 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.954 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.954 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:23:09.954 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.651676+0000", 2024-09-13T11:23:09.954 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000136033 2024-09-13T11:23:09.954 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.954 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.954 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:23:09.954 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.651689+0000", 2024-09-13T11:23:09.955 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.364e-05 2024-09-13T11:23:09.955 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:09.955 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:09.955 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:23:09.955 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.651855+0000", 2024-09-13T11:23:09.955 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000165446 2024-09-13T11:23:09.955 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:09.955 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:23:09.955 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:09.955 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:09.955 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:23:09.955 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-13T11:23:09.958 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_ops_in_flight 2024-09-13T11:23:09.958 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-13T11:23:09.958 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-13T11:23:09.958 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 59, 2024-09-13T11:23:09.958 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "2fd68ac7-5581-4d90-80a4-30bcf5252a7a", 2024-09-13T11:23:09.959 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-09-13T11:21:57.223153+0000", 2024-09-13T11:23:09.959 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-09-13T11:23:07.489625+0000", 2024-09-13T11:23:09.959 INFO:teuthology.orchestra.run.smithi001.stdout: "last_up_change": "2024-09-13T11:22:46.271946+0000", 2024-09-13T11:23:09.959 INFO:teuthology.orchestra.run.smithi001.stdout: "last_in_change": "2024-09-13T11:22:01.745814+0000", 2024-09-13T11:23:09.959 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": "nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-13T11:23:09.959 INFO:teuthology.orchestra.run.smithi001.stdout: "flags_num": 5804032, 2024-09-13T11:23:09.959 INFO:teuthology.orchestra.run.smithi001.stdout: "flags_set": [ 2024-09-13T11:23:09.959 INFO:teuthology.orchestra.run.smithi001.stdout: "nodeep-scrub", 2024-09-13T11:23:09.959 INFO:teuthology.orchestra.run.smithi001.stdout: "pglog_hardlimit", 2024-09-13T11:23:09.959 INFO:teuthology.orchestra.run.smithi001.stdout: "purged_snapdirs", 2024-09-13T11:23:09.959 INFO:teuthology.orchestra.run.smithi001.stdout: "recovery_deletes", 2024-09-13T11:23:09.959 INFO:teuthology.orchestra.run.smithi001.stdout: "sortbitwise" 2024-09-13T11:23:09.959 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-09-13T11:23:09.960 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_version": 6, 2024-09-13T11:23:09.960 INFO:teuthology.orchestra.run.smithi001.stdout: "full_ratio": 0.94999998807907104, 2024-09-13T11:23:09.960 INFO:teuthology.orchestra.run.smithi001.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-09-13T11:23:09.960 INFO:teuthology.orchestra.run.smithi001.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-13T11:23:09.960 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_snapshot": "", 2024-09-13T11:23:09.960 INFO:teuthology.orchestra.run.smithi001.stdout: "pool_max": 3, 2024-09-13T11:23:09.960 INFO:teuthology.orchestra.run.smithi001.stdout: "max_osd": 16, 2024-09-13T11:23:09.960 INFO:teuthology.orchestra.run.smithi001.stdout: "require_min_compat_client": "luminous", 2024-09-13T11:23:09.960 INFO:teuthology.orchestra.run.smithi001.stdout: "min_compat_client": "luminous", 2024-09-13T11:23:09.960 INFO:teuthology.orchestra.run.smithi001.stdout: "require_osd_release": "squid", 2024-09-13T11:23:09.960 INFO:teuthology.orchestra.run.smithi001.stdout: "allow_crimson": false, 2024-09-13T11:23:09.960 INFO:teuthology.orchestra.run.smithi001.stdout: "pools": [ 2024-09-13T11:23:09.960 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:09.960 INFO:teuthology.orchestra.run.smithi001.stdout: "pool": 1, 2024-09-13T11:23:09.961 INFO:teuthology.orchestra.run.smithi001.stdout: "pool_name": ".mgr", 2024-09-13T11:23:09.961 INFO:teuthology.orchestra.run.smithi001.stdout: "create_time": "2024-09-13T11:22:10.162266+0000", 2024-09-13T11:23:09.961 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-09-13T11:23:09.961 INFO:teuthology.orchestra.run.smithi001.stdout: "flags_names": "hashpspool", 2024-09-13T11:23:09.961 INFO:teuthology.orchestra.run.smithi001.stdout: "type": 1, 2024-09-13T11:23:09.961 INFO:teuthology.orchestra.run.smithi001.stdout: "size": 2, 2024-09-13T11:23:09.961 INFO:teuthology.orchestra.run.smithi001.stdout: "min_size": 2, 2024-09-13T11:23:09.961 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_rule": 0, 2024-09-13T11:23:09.961 INFO:teuthology.orchestra.run.smithi001.stdout: "peering_crush_bucket_count": 0, 2024-09-13T11:23:09.961 INFO:teuthology.orchestra.run.smithi001.stdout: "peering_crush_bucket_target": 0, 2024-09-13T11:23:09.961 INFO:teuthology.orchestra.run.smithi001.stdout: "peering_crush_bucket_barrier": 0, 2024-09-13T11:23:09.961 INFO:teuthology.orchestra.run.smithi001.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-13T11:23:09.961 INFO:teuthology.orchestra.run.smithi001.stdout: "is_stretch_pool": false, 2024-09-13T11:23:09.961 INFO:teuthology.orchestra.run.smithi001.stdout: "object_hash": 2, 2024-09-13T11:23:09.962 INFO:teuthology.orchestra.run.smithi001.stdout: "pg_autoscale_mode": "off", 2024-09-13T11:23:09.962 INFO:teuthology.orchestra.run.smithi001.stdout: "pg_num": 1, 2024-09-13T11:23:09.962 INFO:teuthology.orchestra.run.smithi001.stdout: "pg_placement_num": 1, 2024-09-13T11:23:09.962 INFO:teuthology.orchestra.run.smithi001.stdout: "pg_placement_num_target": 1, 2024-09-13T11:23:09.962 INFO:teuthology.orchestra.run.smithi001.stdout: "pg_num_target": 1, 2024-09-13T11:23:09.962 INFO:teuthology.orchestra.run.smithi001.stdout: "pg_num_pending": 1, 2024-09-13T11:23:09.962 INFO:teuthology.orchestra.run.smithi001.stdout: "last_pg_merge_meta": { 2024-09-13T11:23:09.962 INFO:teuthology.orchestra.run.smithi001.stdout: "source_pgid": "0.0", 2024-09-13T11:23:09.962 INFO:teuthology.orchestra.run.smithi001.stdout: "ready_epoch": 0, 2024-09-13T11:23:09.962 INFO:teuthology.orchestra.run.smithi001.stdout: "last_epoch_started": 0, 2024-09-13T11:23:09.962 INFO:teuthology.orchestra.run.smithi001.stdout: "last_epoch_clean": 0, 2024-09-13T11:23:09.962 INFO:teuthology.orchestra.run.smithi001.stdout: "source_version": "0'0", 2024-09-13T11:23:09.962 INFO:teuthology.orchestra.run.smithi001.stdout: "target_version": "0'0" 2024-09-13T11:23:09.963 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:09.963 INFO:teuthology.orchestra.run.smithi001.stdout: "last_change": "25", 2024-09-13T11:23:09.963 INFO:teuthology.orchestra.run.smithi001.stdout: "last_force_op_resend": "0", 2024-09-13T11:23:09.963 INFO:teuthology.orchestra.run.smithi001.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-13T11:23:09.963 INFO:teuthology.orchestra.run.smithi001.stdout: "last_force_op_resend_preluminous": "0", 2024-09-13T11:23:09.963 INFO:teuthology.orchestra.run.smithi001.stdout: "auid": 0, 2024-09-13T11:23:09.963 INFO:teuthology.orchestra.run.smithi001.stdout: "snap_mode": "selfmanaged", 2024-09-13T11:23:09.963 INFO:teuthology.orchestra.run.smithi001.stdout: "snap_seq": 0, 2024-09-13T11:23:09.963 INFO:teuthology.orchestra.run.smithi001.stdout: "snap_epoch": 0, 2024-09-13T11:23:09.963 INFO:teuthology.orchestra.run.smithi001.stdout: "pool_snaps": [], 2024-09-13T11:23:09.963 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_snaps": "[]", 2024-09-13T11:23:09.963 INFO:teuthology.orchestra.run.smithi001.stdout: "quota_max_bytes": 0, 2024-09-13T11:23:09.963 INFO:teuthology.orchestra.run.smithi001.stdout: "quota_max_objects": 0, 2024-09-13T11:23:09.963 INFO:teuthology.orchestra.run.smithi001.stdout: "tiers": [], 2024-09-13T11:23:09.964 INFO:teuthology.orchestra.run.smithi001.stdout: "tier_of": -1, 2024-09-13T11:23:09.964 INFO:teuthology.orchestra.run.smithi001.stdout: "read_tier": -1, 2024-09-13T11:23:09.964 INFO:teuthology.orchestra.run.smithi001.stdout: "write_tier": -1, 2024-09-13T11:23:09.964 INFO:teuthology.orchestra.run.smithi001.stdout: "cache_mode": "none", 2024-09-13T11:23:09.964 INFO:teuthology.orchestra.run.smithi001.stdout: "target_max_bytes": 0, 2024-09-13T11:23:09.964 INFO:teuthology.orchestra.run.smithi001.stdout: "target_max_objects": 0, 2024-09-13T11:23:09.964 INFO:teuthology.orchestra.run.smithi001.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-13T11:23:09.964 INFO:teuthology.orchestra.run.smithi001.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-13T11:23:09.964 INFO:teuthology.orchestra.run.smithi001.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-13T11:23:09.964 INFO:teuthology.orchestra.run.smithi001.stdout: "cache_min_flush_age": 0, 2024-09-13T11:23:09.964 INFO:teuthology.orchestra.run.smithi001.stdout: "cache_min_evict_age": 0, 2024-09-13T11:23:09.964 INFO:teuthology.orchestra.run.smithi001.stdout: "erasure_code_profile": "", 2024-09-13T11:23:09.964 INFO:teuthology.orchestra.run.smithi001.stdout: "hit_set_params": { 2024-09-13T11:23:09.964 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "none" 2024-09-13T11:23:09.965 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:09.965 INFO:teuthology.orchestra.run.smithi001.stdout: "hit_set_period": 0, 2024-09-13T11:23:09.965 INFO:teuthology.orchestra.run.smithi001.stdout: "hit_set_count": 0, 2024-09-13T11:23:09.965 INFO:teuthology.orchestra.run.smithi001.stdout: "use_gmt_hitset": true, 2024-09-13T11:23:09.965 INFO:teuthology.orchestra.run.smithi001.stdout: "min_read_recency_for_promote": 0, 2024-09-13T11:23:09.965 INFO:teuthology.orchestra.run.smithi001.stdout: "min_write_recency_for_promote": 0, 2024-09-13T11:23:09.965 INFO:teuthology.orchestra.run.smithi001.stdout: "hit_set_grade_decay_rate": 0, 2024-09-13T11:23:09.965 INFO:teuthology.orchestra.run.smithi001.stdout: "hit_set_search_last_n": 0, 2024-09-13T11:23:09.965 INFO:teuthology.orchestra.run.smithi001.stdout: "grade_table": [], 2024-09-13T11:23:09.965 INFO:teuthology.orchestra.run.smithi001.stdout: "stripe_width": 0, 2024-09-13T11:23:09.965 INFO:teuthology.orchestra.run.smithi001.stdout: "expected_num_objects": 0, 2024-09-13T11:23:09.965 INFO:teuthology.orchestra.run.smithi001.stdout: "fast_read": false, 2024-09-13T11:23:09.965 INFO:teuthology.orchestra.run.smithi001.stdout: "options": { 2024-09-13T11:23:09.965 INFO:teuthology.orchestra.run.smithi001.stdout: "pg_num_max": 32, 2024-09-13T11:23:09.966 INFO:teuthology.orchestra.run.smithi001.stdout: "pg_num_min": 1 2024-09-13T11:23:09.966 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:09.966 INFO:teuthology.orchestra.run.smithi001.stdout: "application_metadata": { 2024-09-13T11:23:09.966 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": {} 2024-09-13T11:23:09.966 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:09.966 INFO:teuthology.orchestra.run.smithi001.stdout: "read_balance": { 2024-09-13T11:23:09.966 INFO:teuthology.orchestra.run.smithi001.stdout: "score_type": "Fair distribution", 2024-09-13T11:23:09.966 INFO:teuthology.orchestra.run.smithi001.stdout: "score_acting": 15.380000114440918, 2024-09-13T11:23:09.966 INFO:teuthology.orchestra.run.smithi001.stdout: "score_stable": 15.380000114440918, 2024-09-13T11:23:09.966 INFO:teuthology.orchestra.run.smithi001.stdout: "optimal_score": 0.12999999523162842, 2024-09-13T11:23:09.966 INFO:teuthology.orchestra.run.smithi001.stdout: "raw_score_acting": 2, 2024-09-13T11:23:09.966 INFO:teuthology.orchestra.run.smithi001.stdout: "raw_score_stable": 2, 2024-09-13T11:23:09.966 INFO:teuthology.orchestra.run.smithi001.stdout: "primary_affinity_weighted": 1, 2024-09-13T11:23:09.966 INFO:teuthology.orchestra.run.smithi001.stdout: "average_primary_affinity": 1, 2024-09-13T11:23:09.967 INFO:teuthology.orchestra.run.smithi001.stdout: "average_primary_affinity_weighted": 1 2024-09-13T11:23:09.967 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:09.967 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:09.967 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:09.967 INFO:teuthology.orchestra.run.smithi001.stdout: "pool": 2, 2024-09-13T11:23:09.967 INFO:teuthology.orchestra.run.smithi001.stdout: "pool_name": "rbd", 2024-09-13T11:23:09.967 INFO:teuthology.orchestra.run.smithi001.stdout: "create_time": "2024-09-13T11:22:12.235549+0000", 2024-09-13T11:23:09.967 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 8193, 2024-09-13T11:23:09.967 INFO:teuthology.orchestra.run.smithi001.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-09-13T11:23:09.967 INFO:teuthology.orchestra.run.smithi001.stdout: "type": 1, 2024-09-13T11:23:09.967 INFO:teuthology.orchestra.run.smithi001.stdout: "size": 2, 2024-09-13T11:23:09.967 INFO:teuthology.orchestra.run.smithi001.stdout: "min_size": 2, 2024-09-13T11:23:09.967 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_rule": 0, 2024-09-13T11:23:09.967 INFO:teuthology.orchestra.run.smithi001.stdout: "peering_crush_bucket_count": 0, 2024-09-13T11:23:09.968 INFO:teuthology.orchestra.run.smithi001.stdout: "peering_crush_bucket_target": 0, 2024-09-13T11:23:09.968 INFO:teuthology.orchestra.run.smithi001.stdout: "peering_crush_bucket_barrier": 0, 2024-09-13T11:23:09.968 INFO:teuthology.orchestra.run.smithi001.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-13T11:23:09.968 INFO:teuthology.orchestra.run.smithi001.stdout: "is_stretch_pool": false, 2024-09-13T11:23:09.968 INFO:teuthology.orchestra.run.smithi001.stdout: "object_hash": 2, 2024-09-13T11:23:09.968 INFO:teuthology.orchestra.run.smithi001.stdout: "pg_autoscale_mode": "off", 2024-09-13T11:23:09.968 INFO:teuthology.orchestra.run.smithi001.stdout: "pg_num": 8, 2024-09-13T11:23:09.968 INFO:teuthology.orchestra.run.smithi001.stdout: "pg_placement_num": 8, 2024-09-13T11:23:09.968 INFO:teuthology.orchestra.run.smithi001.stdout: "pg_placement_num_target": 8, 2024-09-13T11:23:09.968 INFO:teuthology.orchestra.run.smithi001.stdout: "pg_num_target": 8, 2024-09-13T11:23:09.968 INFO:teuthology.orchestra.run.smithi001.stdout: "pg_num_pending": 8, 2024-09-13T11:23:09.968 INFO:teuthology.orchestra.run.smithi001.stdout: "last_pg_merge_meta": { 2024-09-13T11:23:09.968 INFO:teuthology.orchestra.run.smithi001.stdout: "source_pgid": "0.0", 2024-09-13T11:23:09.969 INFO:teuthology.orchestra.run.smithi001.stdout: "ready_epoch": 0, 2024-09-13T11:23:09.969 INFO:teuthology.orchestra.run.smithi001.stdout: "last_epoch_started": 0, 2024-09-13T11:23:09.969 INFO:teuthology.orchestra.run.smithi001.stdout: "last_epoch_clean": 0, 2024-09-13T11:23:09.969 INFO:teuthology.orchestra.run.smithi001.stdout: "source_version": "0'0", 2024-09-13T11:23:09.969 INFO:teuthology.orchestra.run.smithi001.stdout: "target_version": "0'0" 2024-09-13T11:23:09.969 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:09.969 INFO:teuthology.orchestra.run.smithi001.stdout: "last_change": "28", 2024-09-13T11:23:09.969 INFO:teuthology.orchestra.run.smithi001.stdout: "last_force_op_resend": "0", 2024-09-13T11:23:09.969 INFO:teuthology.orchestra.run.smithi001.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-13T11:23:09.969 INFO:teuthology.orchestra.run.smithi001.stdout: "last_force_op_resend_preluminous": "0", 2024-09-13T11:23:09.969 INFO:teuthology.orchestra.run.smithi001.stdout: "auid": 0, 2024-09-13T11:23:09.969 INFO:teuthology.orchestra.run.smithi001.stdout: "snap_mode": "selfmanaged", 2024-09-13T11:23:09.969 INFO:teuthology.orchestra.run.smithi001.stdout: "snap_seq": 2, 2024-09-13T11:23:09.969 INFO:teuthology.orchestra.run.smithi001.stdout: "snap_epoch": 28, 2024-09-13T11:23:09.970 INFO:teuthology.orchestra.run.smithi001.stdout: "pool_snaps": [], 2024-09-13T11:23:09.970 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_snaps": "[]", 2024-09-13T11:23:09.970 INFO:teuthology.orchestra.run.smithi001.stdout: "quota_max_bytes": 0, 2024-09-13T11:23:09.970 INFO:teuthology.orchestra.run.smithi001.stdout: "quota_max_objects": 0, 2024-09-13T11:23:09.970 INFO:teuthology.orchestra.run.smithi001.stdout: "tiers": [], 2024-09-13T11:23:09.970 INFO:teuthology.orchestra.run.smithi001.stdout: "tier_of": -1, 2024-09-13T11:23:09.970 INFO:teuthology.orchestra.run.smithi001.stdout: "read_tier": -1, 2024-09-13T11:23:09.970 INFO:teuthology.orchestra.run.smithi001.stdout: "write_tier": -1, 2024-09-13T11:23:09.970 INFO:teuthology.orchestra.run.smithi001.stdout: "cache_mode": "none", 2024-09-13T11:23:09.970 INFO:teuthology.orchestra.run.smithi001.stdout: "target_max_bytes": 0, 2024-09-13T11:23:09.970 INFO:teuthology.orchestra.run.smithi001.stdout: "target_max_objects": 0, 2024-09-13T11:23:09.970 INFO:teuthology.orchestra.run.smithi001.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-13T11:23:09.970 INFO:teuthology.orchestra.run.smithi001.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-13T11:23:09.970 INFO:teuthology.orchestra.run.smithi001.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-13T11:23:09.971 INFO:teuthology.orchestra.run.smithi001.stdout: "cache_min_flush_age": 0, 2024-09-13T11:23:09.971 INFO:teuthology.orchestra.run.smithi001.stdout: "cache_min_evict_age": 0, 2024-09-13T11:23:09.971 INFO:teuthology.orchestra.run.smithi001.stdout: "erasure_code_profile": "", 2024-09-13T11:23:09.971 INFO:teuthology.orchestra.run.smithi001.stdout: "hit_set_params": { 2024-09-13T11:23:09.971 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "none" 2024-09-13T11:23:09.971 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:09.971 INFO:teuthology.orchestra.run.smithi001.stdout: "hit_set_period": 0, 2024-09-13T11:23:09.971 INFO:teuthology.orchestra.run.smithi001.stdout: "hit_set_count": 0, 2024-09-13T11:23:09.971 INFO:teuthology.orchestra.run.smithi001.stdout: "use_gmt_hitset": true, 2024-09-13T11:23:09.971 INFO:teuthology.orchestra.run.smithi001.stdout: "min_read_recency_for_promote": 0, 2024-09-13T11:23:09.971 INFO:teuthology.orchestra.run.smithi001.stdout: "min_write_recency_for_promote": 0, 2024-09-13T11:23:09.971 INFO:teuthology.orchestra.run.smithi001.stdout: "hit_set_grade_decay_rate": 0, 2024-09-13T11:23:09.971 INFO:teuthology.orchestra.run.smithi001.stdout: "hit_set_search_last_n": 0, 2024-09-13T11:23:09.971 INFO:teuthology.orchestra.run.smithi001.stdout: "grade_table": [], 2024-09-13T11:23:09.972 INFO:teuthology.orchestra.run.smithi001.stdout: "stripe_width": 0, 2024-09-13T11:23:09.972 INFO:teuthology.orchestra.run.smithi001.stdout: "expected_num_objects": 0, 2024-09-13T11:23:09.972 INFO:teuthology.orchestra.run.smithi001.stdout: "fast_read": false, 2024-09-13T11:23:09.972 INFO:teuthology.orchestra.run.smithi001.stdout: "options": {}, 2024-09-13T11:23:09.972 INFO:teuthology.orchestra.run.smithi001.stdout: "application_metadata": { 2024-09-13T11:23:09.972 INFO:teuthology.orchestra.run.smithi001.stdout: "rbd": {} 2024-09-13T11:23:09.972 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:09.972 INFO:teuthology.orchestra.run.smithi001.stdout: "read_balance": { 2024-09-13T11:23:09.972 INFO:teuthology.orchestra.run.smithi001.stdout: "score_type": "Fair distribution", 2024-09-13T11:23:09.972 INFO:teuthology.orchestra.run.smithi001.stdout: "score_acting": 4.0199999809265137, 2024-09-13T11:23:09.972 INFO:teuthology.orchestra.run.smithi001.stdout: "score_stable": 4.0199999809265137, 2024-09-13T11:23:09.972 INFO:teuthology.orchestra.run.smithi001.stdout: "optimal_score": 0.56000000238418579, 2024-09-13T11:23:09.972 INFO:teuthology.orchestra.run.smithi001.stdout: "raw_score_acting": 2.25, 2024-09-13T11:23:09.973 INFO:teuthology.orchestra.run.smithi001.stdout: "raw_score_stable": 2.25, 2024-09-13T11:23:09.973 INFO:teuthology.orchestra.run.smithi001.stdout: "primary_affinity_weighted": 1, 2024-09-13T11:23:09.973 INFO:teuthology.orchestra.run.smithi001.stdout: "average_primary_affinity": 1, 2024-09-13T11:23:09.973 INFO:teuthology.orchestra.run.smithi001.stdout: "average_primary_affinity_weighted": 1 2024-09-13T11:23:09.973 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:09.973 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:09.973 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:09.973 INFO:teuthology.orchestra.run.smithi001.stdout: "pool": 3, 2024-09-13T11:23:09.973 INFO:teuthology.orchestra.run.smithi001.stdout: "pool_name": "unique_pool_0", 2024-09-13T11:23:09.973 INFO:teuthology.orchestra.run.smithi001.stdout: "create_time": "2024-09-13T11:22:24.305521+0000", 2024-09-13T11:23:09.973 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-09-13T11:23:09.973 INFO:teuthology.orchestra.run.smithi001.stdout: "flags_names": "hashpspool", 2024-09-13T11:23:09.973 INFO:teuthology.orchestra.run.smithi001.stdout: "type": 1, 2024-09-13T11:23:09.973 INFO:teuthology.orchestra.run.smithi001.stdout: "size": 2, 2024-09-13T11:23:09.974 INFO:teuthology.orchestra.run.smithi001.stdout: "min_size": 2, 2024-09-13T11:23:09.974 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_rule": 0, 2024-09-13T11:23:09.974 INFO:teuthology.orchestra.run.smithi001.stdout: "peering_crush_bucket_count": 0, 2024-09-13T11:23:09.974 INFO:teuthology.orchestra.run.smithi001.stdout: "peering_crush_bucket_target": 0, 2024-09-13T11:23:09.974 INFO:teuthology.orchestra.run.smithi001.stdout: "peering_crush_bucket_barrier": 0, 2024-09-13T11:23:09.974 INFO:teuthology.orchestra.run.smithi001.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-13T11:23:09.974 INFO:teuthology.orchestra.run.smithi001.stdout: "is_stretch_pool": false, 2024-09-13T11:23:09.974 INFO:teuthology.orchestra.run.smithi001.stdout: "object_hash": 2, 2024-09-13T11:23:09.974 INFO:teuthology.orchestra.run.smithi001.stdout: "pg_autoscale_mode": "off", 2024-09-13T11:23:09.974 INFO:teuthology.orchestra.run.smithi001.stdout: "pg_num": 16, 2024-09-13T11:23:09.974 INFO:teuthology.orchestra.run.smithi001.stdout: "pg_placement_num": 16, 2024-09-13T11:23:09.974 INFO:teuthology.orchestra.run.smithi001.stdout: "pg_placement_num_target": 16, 2024-09-13T11:23:09.974 INFO:teuthology.orchestra.run.smithi001.stdout: "pg_num_target": 16, 2024-09-13T11:23:09.974 INFO:teuthology.orchestra.run.smithi001.stdout: "pg_num_pending": 16, 2024-09-13T11:23:09.974 INFO:teuthology.orchestra.run.smithi001.stdout: "last_pg_merge_meta": { 2024-09-13T11:23:09.975 INFO:teuthology.orchestra.run.smithi001.stdout: "source_pgid": "0.0", 2024-09-13T11:23:09.975 INFO:teuthology.orchestra.run.smithi001.stdout: "ready_epoch": 0, 2024-09-13T11:23:09.975 INFO:teuthology.orchestra.run.smithi001.stdout: "last_epoch_started": 0, 2024-09-13T11:23:09.975 INFO:teuthology.orchestra.run.smithi001.stdout: "last_epoch_clean": 0, 2024-09-13T11:23:09.975 INFO:teuthology.orchestra.run.smithi001.stdout: "source_version": "0'0", 2024-09-13T11:23:09.975 INFO:teuthology.orchestra.run.smithi001.stdout: "target_version": "0'0" 2024-09-13T11:23:09.975 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:09.975 INFO:teuthology.orchestra.run.smithi001.stdout: "last_change": "32", 2024-09-13T11:23:09.975 INFO:teuthology.orchestra.run.smithi001.stdout: "last_force_op_resend": "0", 2024-09-13T11:23:09.975 INFO:teuthology.orchestra.run.smithi001.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-13T11:23:09.975 INFO:teuthology.orchestra.run.smithi001.stdout: "last_force_op_resend_preluminous": "0", 2024-09-13T11:23:09.975 INFO:teuthology.orchestra.run.smithi001.stdout: "auid": 0, 2024-09-13T11:23:09.975 INFO:teuthology.orchestra.run.smithi001.stdout: "snap_mode": "selfmanaged", 2024-09-13T11:23:09.976 INFO:teuthology.orchestra.run.smithi001.stdout: "snap_seq": 0, 2024-09-13T11:23:09.976 INFO:teuthology.orchestra.run.smithi001.stdout: "snap_epoch": 0, 2024-09-13T11:23:09.976 INFO:teuthology.orchestra.run.smithi001.stdout: "pool_snaps": [], 2024-09-13T11:23:09.976 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_snaps": "[]", 2024-09-13T11:23:09.976 INFO:teuthology.orchestra.run.smithi001.stdout: "quota_max_bytes": 0, 2024-09-13T11:23:09.976 INFO:teuthology.orchestra.run.smithi001.stdout: "quota_max_objects": 0, 2024-09-13T11:23:09.976 INFO:teuthology.orchestra.run.smithi001.stdout: "tiers": [], 2024-09-13T11:23:09.976 INFO:teuthology.orchestra.run.smithi001.stdout: "tier_of": -1, 2024-09-13T11:23:09.976 INFO:teuthology.orchestra.run.smithi001.stdout: "read_tier": -1, 2024-09-13T11:23:09.976 INFO:teuthology.orchestra.run.smithi001.stdout: "write_tier": -1, 2024-09-13T11:23:09.976 INFO:teuthology.orchestra.run.smithi001.stdout: "cache_mode": "none", 2024-09-13T11:23:09.976 INFO:teuthology.orchestra.run.smithi001.stdout: "target_max_bytes": 0, 2024-09-13T11:23:09.976 INFO:teuthology.orchestra.run.smithi001.stdout: "target_max_objects": 0, 2024-09-13T11:23:09.976 INFO:teuthology.orchestra.run.smithi001.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-13T11:23:09.977 INFO:teuthology.orchestra.run.smithi001.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-13T11:23:09.977 INFO:teuthology.orchestra.run.smithi001.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-13T11:23:09.977 INFO:teuthology.orchestra.run.smithi001.stdout: "cache_min_flush_age": 0, 2024-09-13T11:23:09.977 INFO:teuthology.orchestra.run.smithi001.stdout: "cache_min_evict_age": 0, 2024-09-13T11:23:09.977 INFO:teuthology.orchestra.run.smithi001.stdout: "erasure_code_profile": "", 2024-09-13T11:23:09.977 INFO:teuthology.orchestra.run.smithi001.stdout: "hit_set_params": { 2024-09-13T11:23:09.977 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "none" 2024-09-13T11:23:09.977 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:09.977 INFO:teuthology.orchestra.run.smithi001.stdout: "hit_set_period": 0, 2024-09-13T11:23:09.977 INFO:teuthology.orchestra.run.smithi001.stdout: "hit_set_count": 0, 2024-09-13T11:23:09.977 INFO:teuthology.orchestra.run.smithi001.stdout: "use_gmt_hitset": true, 2024-09-13T11:23:09.977 INFO:teuthology.orchestra.run.smithi001.stdout: "min_read_recency_for_promote": 0, 2024-09-13T11:23:09.977 INFO:teuthology.orchestra.run.smithi001.stdout: "min_write_recency_for_promote": 0, 2024-09-13T11:23:09.977 INFO:teuthology.orchestra.run.smithi001.stdout: "hit_set_grade_decay_rate": 0, 2024-09-13T11:23:09.978 INFO:teuthology.orchestra.run.smithi001.stdout: "hit_set_search_last_n": 0, 2024-09-13T11:23:09.978 INFO:teuthology.orchestra.run.smithi001.stdout: "grade_table": [], 2024-09-13T11:23:09.978 INFO:teuthology.orchestra.run.smithi001.stdout: "stripe_width": 0, 2024-09-13T11:23:09.978 INFO:teuthology.orchestra.run.smithi001.stdout: "expected_num_objects": 0, 2024-09-13T11:23:09.978 INFO:teuthology.orchestra.run.smithi001.stdout: "fast_read": false, 2024-09-13T11:23:09.978 INFO:teuthology.orchestra.run.smithi001.stdout: "options": {}, 2024-09-13T11:23:09.978 INFO:teuthology.orchestra.run.smithi001.stdout: "application_metadata": { 2024-09-13T11:23:09.978 INFO:teuthology.orchestra.run.smithi001.stdout: "rados": {} 2024-09-13T11:23:09.978 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:09.978 INFO:teuthology.orchestra.run.smithi001.stdout: "read_balance": { 2024-09-13T11:23:09.978 INFO:teuthology.orchestra.run.smithi001.stdout: "score_type": "Fair distribution", 2024-09-13T11:23:09.978 INFO:teuthology.orchestra.run.smithi001.stdout: "score_acting": 4.0100002288818359, 2024-09-13T11:23:09.978 INFO:teuthology.orchestra.run.smithi001.stdout: "score_stable": 4.0100002288818359, 2024-09-13T11:23:09.978 INFO:teuthology.orchestra.run.smithi001.stdout: "optimal_score": 0.81000000238418579, 2024-09-13T11:23:09.979 INFO:teuthology.orchestra.run.smithi001.stdout: "raw_score_acting": 3.25, 2024-09-13T11:23:09.979 INFO:teuthology.orchestra.run.smithi001.stdout: "raw_score_stable": 3.25, 2024-09-13T11:23:09.979 INFO:teuthology.orchestra.run.smithi001.stdout: "primary_affinity_weighted": 1, 2024-09-13T11:23:09.979 INFO:teuthology.orchestra.run.smithi001.stdout: "average_primary_affinity": 1, 2024-09-13T11:23:09.979 INFO:teuthology.orchestra.run.smithi001.stdout: "average_primary_affinity_weighted": 1 2024-09-13T11:23:09.979 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:09.979 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:09.979 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-09-13T11:23:09.979 INFO:teuthology.orchestra.run.smithi001.stdout: "osds": [ 2024-09-13T11:23:09.979 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:09.979 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": 0, 2024-09-13T11:23:09.979 INFO:teuthology.orchestra.run.smithi001.stdout: "uuid": "0fc12b3c-e0de-4546-b935-db77efe7c567", 2024-09-13T11:23:09.979 INFO:teuthology.orchestra.run.smithi001.stdout: "up": 1, 2024-09-13T11:23:09.979 INFO:teuthology.orchestra.run.smithi001.stdout: "in": 1, 2024-09-13T11:23:09.980 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 1, 2024-09-13T11:23:09.980 INFO:teuthology.orchestra.run.smithi001.stdout: "primary_affinity": 1, 2024-09-13T11:23:09.980 INFO:teuthology.orchestra.run.smithi001.stdout: "last_clean_begin": 0, 2024-09-13T11:23:09.980 INFO:teuthology.orchestra.run.smithi001.stdout: "last_clean_end": 0, 2024-09-13T11:23:09.980 INFO:teuthology.orchestra.run.smithi001.stdout: "up_from": 22, 2024-09-13T11:23:09.980 INFO:teuthology.orchestra.run.smithi001.stdout: "up_thru": 56, 2024-09-13T11:23:09.980 INFO:teuthology.orchestra.run.smithi001.stdout: "down_at": 0, 2024-09-13T11:23:09.980 INFO:teuthology.orchestra.run.smithi001.stdout: "lost_at": 0, 2024-09-13T11:23:09.980 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-09-13T11:23:09.980 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:23:09.980 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:09.980 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:23:09.980 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6800", 2024-09-13T11:23:09.980 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 1544776477 2024-09-13T11:23:09.981 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:09.981 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:09.981 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:09.981 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_addrs": { 2024-09-13T11:23:09.981 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:23:09.981 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:09.981 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:23:09.981 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6801", 2024-09-13T11:23:09.981 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 1544776477 2024-09-13T11:23:09.981 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:09.981 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:09.981 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:09.981 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_back_addrs": { 2024-09-13T11:23:09.982 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:23:09.982 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:09.982 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:23:09.982 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6803", 2024-09-13T11:23:09.982 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 1544776477 2024-09-13T11:23:09.982 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:09.982 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:09.982 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:09.982 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_front_addrs": { 2024-09-13T11:23:09.982 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:23:09.982 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:09.982 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:23:09.982 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6802", 2024-09-13T11:23:09.982 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 1544776477 2024-09-13T11:23:09.983 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:09.983 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:09.983 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:09.983 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6800/1544776477", 2024-09-13T11:23:09.983 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_addr": "172.21.15.1:6801/1544776477", 2024-09-13T11:23:09.983 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_back_addr": "172.21.15.1:6803/1544776477", 2024-09-13T11:23:09.983 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_front_addr": "172.21.15.1:6802/1544776477", 2024-09-13T11:23:09.983 INFO:teuthology.orchestra.run.smithi001.stdout: "state": [ 2024-09-13T11:23:09.983 INFO:teuthology.orchestra.run.smithi001.stdout: "exists", 2024-09-13T11:23:09.983 INFO:teuthology.orchestra.run.smithi001.stdout: "up" 2024-09-13T11:23:09.983 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:09.983 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:09.983 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:09.983 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": 1, 2024-09-13T11:23:09.984 INFO:teuthology.orchestra.run.smithi001.stdout: "uuid": "b77fe331-cc92-4adb-b146-af5c380f6d4f", 2024-09-13T11:23:09.984 INFO:teuthology.orchestra.run.smithi001.stdout: "up": 1, 2024-09-13T11:23:09.984 INFO:teuthology.orchestra.run.smithi001.stdout: "in": 1, 2024-09-13T11:23:09.984 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 1, 2024-09-13T11:23:09.984 INFO:teuthology.orchestra.run.smithi001.stdout: "primary_affinity": 1, 2024-09-13T11:23:09.984 INFO:teuthology.orchestra.run.smithi001.stdout: "last_clean_begin": 23, 2024-09-13T11:23:09.984 INFO:teuthology.orchestra.run.smithi001.stdout: "last_clean_end": 29, 2024-09-13T11:23:09.984 INFO:teuthology.orchestra.run.smithi001.stdout: "up_from": 40, 2024-09-13T11:23:09.984 INFO:teuthology.orchestra.run.smithi001.stdout: "up_thru": 40, 2024-09-13T11:23:09.984 INFO:teuthology.orchestra.run.smithi001.stdout: "down_at": 30, 2024-09-13T11:23:09.984 INFO:teuthology.orchestra.run.smithi001.stdout: "lost_at": 0, 2024-09-13T11:23:09.984 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-09-13T11:23:09.984 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:23:09.984 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:09.985 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:23:09.985 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.92:6800", 2024-09-13T11:23:09.985 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 3591430847 2024-09-13T11:23:09.985 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:09.985 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:09.985 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:09.985 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_addrs": { 2024-09-13T11:23:09.985 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:23:09.985 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:09.985 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:23:09.985 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.92:6801", 2024-09-13T11:23:09.985 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 3591430847 2024-09-13T11:23:09.985 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:09.985 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:09.986 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:09.986 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_back_addrs": { 2024-09-13T11:23:09.986 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:23:09.986 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:09.986 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:23:09.986 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.92:6803", 2024-09-13T11:23:09.986 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 3591430847 2024-09-13T11:23:09.986 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:09.986 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:09.986 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:09.986 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_front_addrs": { 2024-09-13T11:23:09.986 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:23:09.986 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:09.986 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:23:09.987 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.92:6802", 2024-09-13T11:23:09.987 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 3591430847 2024-09-13T11:23:09.987 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:09.987 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:09.987 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:09.987 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.92:6800/3591430847", 2024-09-13T11:23:09.987 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_addr": "172.21.15.92:6801/3591430847", 2024-09-13T11:23:09.987 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_back_addr": "172.21.15.92:6803/3591430847", 2024-09-13T11:23:09.987 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_front_addr": "172.21.15.92:6802/3591430847", 2024-09-13T11:23:09.987 INFO:teuthology.orchestra.run.smithi001.stdout: "state": [ 2024-09-13T11:23:09.987 INFO:teuthology.orchestra.run.smithi001.stdout: "exists", 2024-09-13T11:23:09.987 INFO:teuthology.orchestra.run.smithi001.stdout: "up" 2024-09-13T11:23:09.987 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:09.987 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:09.988 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:09.988 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": 2, 2024-09-13T11:23:09.988 INFO:teuthology.orchestra.run.smithi001.stdout: "uuid": "c9d031c8-e885-4229-affa-08e1729d00e8", 2024-09-13T11:23:09.988 INFO:teuthology.orchestra.run.smithi001.stdout: "up": 1, 2024-09-13T11:23:09.988 INFO:teuthology.orchestra.run.smithi001.stdout: "in": 1, 2024-09-13T11:23:09.988 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 1, 2024-09-13T11:23:09.988 INFO:teuthology.orchestra.run.smithi001.stdout: "primary_affinity": 1, 2024-09-13T11:23:09.988 INFO:teuthology.orchestra.run.smithi001.stdout: "last_clean_begin": 0, 2024-09-13T11:23:09.988 INFO:teuthology.orchestra.run.smithi001.stdout: "last_clean_end": 0, 2024-09-13T11:23:09.988 INFO:teuthology.orchestra.run.smithi001.stdout: "up_from": 22, 2024-09-13T11:23:09.988 INFO:teuthology.orchestra.run.smithi001.stdout: "up_thru": 0, 2024-09-13T11:23:09.988 INFO:teuthology.orchestra.run.smithi001.stdout: "down_at": 0, 2024-09-13T11:23:09.988 INFO:teuthology.orchestra.run.smithi001.stdout: "lost_at": 0, 2024-09-13T11:23:09.989 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-09-13T11:23:09.989 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:23:09.989 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:09.989 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:23:09.989 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.162:6812", 2024-09-13T11:23:09.989 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 2712148101 2024-09-13T11:23:09.989 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:09.989 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:09.989 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:09.989 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_addrs": { 2024-09-13T11:23:09.989 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:23:09.989 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:09.989 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:23:09.989 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.162:6813", 2024-09-13T11:23:09.990 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 2712148101 2024-09-13T11:23:09.990 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:09.990 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:09.990 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:09.990 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_back_addrs": { 2024-09-13T11:23:09.990 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:23:09.990 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:09.990 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:23:09.990 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.162:6815", 2024-09-13T11:23:09.990 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 2712148101 2024-09-13T11:23:09.990 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:09.990 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:09.990 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:09.991 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_front_addrs": { 2024-09-13T11:23:09.991 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:23:09.991 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:09.991 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:23:09.991 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.162:6814", 2024-09-13T11:23:09.991 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 2712148101 2024-09-13T11:23:09.991 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:09.991 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:09.991 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:09.991 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.162:6812/2712148101", 2024-09-13T11:23:09.991 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_addr": "172.21.15.162:6813/2712148101", 2024-09-13T11:23:09.991 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_back_addr": "172.21.15.162:6815/2712148101", 2024-09-13T11:23:09.991 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_front_addr": "172.21.15.162:6814/2712148101", 2024-09-13T11:23:09.991 INFO:teuthology.orchestra.run.smithi001.stdout: "state": [ 2024-09-13T11:23:09.991 INFO:teuthology.orchestra.run.smithi001.stdout: "exists", 2024-09-13T11:23:09.992 INFO:teuthology.orchestra.run.smithi001.stdout: "up" 2024-09-13T11:23:09.992 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:09.992 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:09.992 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:09.992 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": 3, 2024-09-13T11:23:09.992 INFO:teuthology.orchestra.run.smithi001.stdout: "uuid": "bf50d71a-6c3a-4fbc-a407-cb81bad14a7b", 2024-09-13T11:23:09.992 INFO:teuthology.orchestra.run.smithi001.stdout: "up": 1, 2024-09-13T11:23:09.992 INFO:teuthology.orchestra.run.smithi001.stdout: "in": 1, 2024-09-13T11:23:09.992 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 1, 2024-09-13T11:23:09.992 INFO:teuthology.orchestra.run.smithi001.stdout: "primary_affinity": 1, 2024-09-13T11:23:09.992 INFO:teuthology.orchestra.run.smithi001.stdout: "last_clean_begin": 0, 2024-09-13T11:23:09.992 INFO:teuthology.orchestra.run.smithi001.stdout: "last_clean_end": 0, 2024-09-13T11:23:09.992 INFO:teuthology.orchestra.run.smithi001.stdout: "up_from": 21, 2024-09-13T11:23:09.993 INFO:teuthology.orchestra.run.smithi001.stdout: "up_thru": 30, 2024-09-13T11:23:09.993 INFO:teuthology.orchestra.run.smithi001.stdout: "down_at": 0, 2024-09-13T11:23:09.993 INFO:teuthology.orchestra.run.smithi001.stdout: "lost_at": 0, 2024-09-13T11:23:09.993 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-09-13T11:23:09.993 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:23:09.993 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:09.993 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:23:09.993 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.177:6801", 2024-09-13T11:23:09.993 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 3367751690 2024-09-13T11:23:09.993 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:09.993 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:09.993 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:09.993 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_addrs": { 2024-09-13T11:23:09.993 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:23:09.994 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:09.994 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:23:09.994 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.177:6802", 2024-09-13T11:23:09.994 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 3367751690 2024-09-13T11:23:09.994 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:09.994 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:09.994 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:09.994 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_back_addrs": { 2024-09-13T11:23:09.994 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:23:09.994 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:09.994 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:23:09.994 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.177:6804", 2024-09-13T11:23:09.995 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 3367751690 2024-09-13T11:23:09.995 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:09.995 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:09.995 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:09.995 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_front_addrs": { 2024-09-13T11:23:09.995 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:23:09.995 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:09.995 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:23:09.995 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.177:6803", 2024-09-13T11:23:09.995 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 3367751690 2024-09-13T11:23:09.995 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:09.995 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:09.995 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:09.995 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.177:6801/3367751690", 2024-09-13T11:23:09.996 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_addr": "172.21.15.177:6802/3367751690", 2024-09-13T11:23:09.996 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_back_addr": "172.21.15.177:6804/3367751690", 2024-09-13T11:23:09.996 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_front_addr": "172.21.15.177:6803/3367751690", 2024-09-13T11:23:09.996 INFO:teuthology.orchestra.run.smithi001.stdout: "state": [ 2024-09-13T11:23:09.996 INFO:teuthology.orchestra.run.smithi001.stdout: "exists", 2024-09-13T11:23:09.996 INFO:teuthology.orchestra.run.smithi001.stdout: "up" 2024-09-13T11:23:09.996 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:09.996 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:09.996 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:09.996 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": 4, 2024-09-13T11:23:09.996 INFO:teuthology.orchestra.run.smithi001.stdout: "uuid": "d1fe3f63-df3f-4a73-a1e2-8d7c8a3f1fa0", 2024-09-13T11:23:09.996 INFO:teuthology.orchestra.run.smithi001.stdout: "up": 1, 2024-09-13T11:23:09.996 INFO:teuthology.orchestra.run.smithi001.stdout: "in": 1, 2024-09-13T11:23:09.997 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 1, 2024-09-13T11:23:09.997 INFO:teuthology.orchestra.run.smithi001.stdout: "primary_affinity": 1, 2024-09-13T11:23:09.997 INFO:teuthology.orchestra.run.smithi001.stdout: "last_clean_begin": 0, 2024-09-13T11:23:09.997 INFO:teuthology.orchestra.run.smithi001.stdout: "last_clean_end": 0, 2024-09-13T11:23:09.997 INFO:teuthology.orchestra.run.smithi001.stdout: "up_from": 22, 2024-09-13T11:23:09.997 INFO:teuthology.orchestra.run.smithi001.stdout: "up_thru": 0, 2024-09-13T11:23:09.997 INFO:teuthology.orchestra.run.smithi001.stdout: "down_at": 0, 2024-09-13T11:23:09.997 INFO:teuthology.orchestra.run.smithi001.stdout: "lost_at": 0, 2024-09-13T11:23:09.997 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-09-13T11:23:09.997 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:23:09.997 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:09.997 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:23:09.997 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6808", 2024-09-13T11:23:09.997 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 3648540926 2024-09-13T11:23:09.998 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:09.998 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:09.998 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:09.998 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_addrs": { 2024-09-13T11:23:09.998 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:23:09.998 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:09.998 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:23:09.998 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6809", 2024-09-13T11:23:09.998 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 3648540926 2024-09-13T11:23:09.998 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:09.998 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:09.998 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:09.998 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_back_addrs": { 2024-09-13T11:23:09.998 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:23:09.999 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:09.999 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:23:09.999 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6811", 2024-09-13T11:23:09.999 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 3648540926 2024-09-13T11:23:09.999 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:09.999 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:09.999 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:09.999 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_front_addrs": { 2024-09-13T11:23:09.999 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:23:09.999 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:09.999 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:23:09.999 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6810", 2024-09-13T11:23:09.999 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 3648540926 2024-09-13T11:23:10.000 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:10.000 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:10.000 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:10.000 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6808/3648540926", 2024-09-13T11:23:10.000 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_addr": "172.21.15.1:6809/3648540926", 2024-09-13T11:23:10.000 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_back_addr": "172.21.15.1:6811/3648540926", 2024-09-13T11:23:10.000 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_front_addr": "172.21.15.1:6810/3648540926", 2024-09-13T11:23:10.000 INFO:teuthology.orchestra.run.smithi001.stdout: "state": [ 2024-09-13T11:23:10.000 INFO:teuthology.orchestra.run.smithi001.stdout: "exists", 2024-09-13T11:23:10.000 INFO:teuthology.orchestra.run.smithi001.stdout: "up" 2024-09-13T11:23:10.000 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:10.000 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:10.000 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:10.000 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": 5, 2024-09-13T11:23:10.001 INFO:teuthology.orchestra.run.smithi001.stdout: "uuid": "e2ead447-2f2c-40c9-82e7-53b8dc41e2af", 2024-09-13T11:23:10.001 INFO:teuthology.orchestra.run.smithi001.stdout: "up": 1, 2024-09-13T11:23:10.001 INFO:teuthology.orchestra.run.smithi001.stdout: "in": 1, 2024-09-13T11:23:10.001 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 1, 2024-09-13T11:23:10.001 INFO:teuthology.orchestra.run.smithi001.stdout: "primary_affinity": 1, 2024-09-13T11:23:10.001 INFO:teuthology.orchestra.run.smithi001.stdout: "last_clean_begin": 0, 2024-09-13T11:23:10.001 INFO:teuthology.orchestra.run.smithi001.stdout: "last_clean_end": 0, 2024-09-13T11:23:10.001 INFO:teuthology.orchestra.run.smithi001.stdout: "up_from": 21, 2024-09-13T11:23:10.001 INFO:teuthology.orchestra.run.smithi001.stdout: "up_thru": 0, 2024-09-13T11:23:10.001 INFO:teuthology.orchestra.run.smithi001.stdout: "down_at": 0, 2024-09-13T11:23:10.001 INFO:teuthology.orchestra.run.smithi001.stdout: "lost_at": 0, 2024-09-13T11:23:10.001 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-09-13T11:23:10.001 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:23:10.001 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:10.002 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:23:10.002 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.92:6804", 2024-09-13T11:23:10.002 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 3399529091 2024-09-13T11:23:10.002 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:10.002 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:10.002 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:10.002 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_addrs": { 2024-09-13T11:23:10.002 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:23:10.002 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:10.002 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:23:10.002 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.92:6805", 2024-09-13T11:23:10.002 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 3399529091 2024-09-13T11:23:10.002 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:10.002 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:10.003 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:10.003 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_back_addrs": { 2024-09-13T11:23:10.003 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:23:10.003 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:10.003 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:23:10.003 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.92:6807", 2024-09-13T11:23:10.003 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 3399529091 2024-09-13T11:23:10.003 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:10.003 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:10.003 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:10.003 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_front_addrs": { 2024-09-13T11:23:10.003 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:23:10.003 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:10.003 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:23:10.004 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.92:6806", 2024-09-13T11:23:10.004 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 3399529091 2024-09-13T11:23:10.004 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:10.004 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:10.004 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:10.004 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.92:6804/3399529091", 2024-09-13T11:23:10.004 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_addr": "172.21.15.92:6805/3399529091", 2024-09-13T11:23:10.004 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_back_addr": "172.21.15.92:6807/3399529091", 2024-09-13T11:23:10.004 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_front_addr": "172.21.15.92:6806/3399529091", 2024-09-13T11:23:10.004 INFO:teuthology.orchestra.run.smithi001.stdout: "state": [ 2024-09-13T11:23:10.004 INFO:teuthology.orchestra.run.smithi001.stdout: "exists", 2024-09-13T11:23:10.004 INFO:teuthology.orchestra.run.smithi001.stdout: "up" 2024-09-13T11:23:10.004 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:10.005 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:10.005 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:10.005 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": 6, 2024-09-13T11:23:10.005 INFO:teuthology.orchestra.run.smithi001.stdout: "uuid": "dbc20439-144e-4ec7-89de-674eb783f263", 2024-09-13T11:23:10.005 INFO:teuthology.orchestra.run.smithi001.stdout: "up": 1, 2024-09-13T11:23:10.005 INFO:teuthology.orchestra.run.smithi001.stdout: "in": 1, 2024-09-13T11:23:10.005 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 1, 2024-09-13T11:23:10.005 INFO:teuthology.orchestra.run.smithi001.stdout: "primary_affinity": 1, 2024-09-13T11:23:10.005 INFO:teuthology.orchestra.run.smithi001.stdout: "last_clean_begin": 0, 2024-09-13T11:23:10.005 INFO:teuthology.orchestra.run.smithi001.stdout: "last_clean_end": 0, 2024-09-13T11:23:10.005 INFO:teuthology.orchestra.run.smithi001.stdout: "up_from": 22, 2024-09-13T11:23:10.005 INFO:teuthology.orchestra.run.smithi001.stdout: "up_thru": 40, 2024-09-13T11:23:10.005 INFO:teuthology.orchestra.run.smithi001.stdout: "down_at": 0, 2024-09-13T11:23:10.005 INFO:teuthology.orchestra.run.smithi001.stdout: "lost_at": 0, 2024-09-13T11:23:10.006 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-09-13T11:23:10.006 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:23:10.006 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:10.006 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:23:10.006 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.162:6808", 2024-09-13T11:23:10.006 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 2287343806 2024-09-13T11:23:10.006 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:10.006 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:10.006 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:10.006 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_addrs": { 2024-09-13T11:23:10.006 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:23:10.006 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:10.006 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:23:10.006 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.162:6809", 2024-09-13T11:23:10.007 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 2287343806 2024-09-13T11:23:10.007 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:10.007 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:10.007 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:10.007 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_back_addrs": { 2024-09-13T11:23:10.007 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:23:10.007 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:10.007 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:23:10.007 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.162:6811", 2024-09-13T11:23:10.007 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 2287343806 2024-09-13T11:23:10.007 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:10.007 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:10.007 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:10.007 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_front_addrs": { 2024-09-13T11:23:10.007 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:23:10.008 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:10.008 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:23:10.008 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.162:6810", 2024-09-13T11:23:10.008 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 2287343806 2024-09-13T11:23:10.008 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:10.008 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:10.008 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:10.008 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.162:6808/2287343806", 2024-09-13T11:23:10.008 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_addr": "172.21.15.162:6809/2287343806", 2024-09-13T11:23:10.008 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_back_addr": "172.21.15.162:6811/2287343806", 2024-09-13T11:23:10.008 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_front_addr": "172.21.15.162:6810/2287343806", 2024-09-13T11:23:10.008 INFO:teuthology.orchestra.run.smithi001.stdout: "state": [ 2024-09-13T11:23:10.008 INFO:teuthology.orchestra.run.smithi001.stdout: "exists", 2024-09-13T11:23:10.008 INFO:teuthology.orchestra.run.smithi001.stdout: "up" 2024-09-13T11:23:10.009 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:10.009 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:10.009 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:10.009 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": 7, 2024-09-13T11:23:10.009 INFO:teuthology.orchestra.run.smithi001.stdout: "uuid": "60aa5206-cace-40f8-8d1a-10ec28d97096", 2024-09-13T11:23:10.009 INFO:teuthology.orchestra.run.smithi001.stdout: "up": 1, 2024-09-13T11:23:10.012 INFO:teuthology.orchestra.run.smithi001.stdout: "in": 1, 2024-09-13T11:23:10.012 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 1, 2024-09-13T11:23:10.012 INFO:teuthology.orchestra.run.smithi001.stdout: "primary_affinity": 1, 2024-09-13T11:23:10.012 INFO:teuthology.orchestra.run.smithi001.stdout: "last_clean_begin": 0, 2024-09-13T11:23:10.012 INFO:teuthology.orchestra.run.smithi001.stdout: "last_clean_end": 0, 2024-09-13T11:23:10.012 INFO:teuthology.orchestra.run.smithi001.stdout: "up_from": 22, 2024-09-13T11:23:10.012 INFO:teuthology.orchestra.run.smithi001.stdout: "up_thru": 30, 2024-09-13T11:23:10.012 INFO:teuthology.orchestra.run.smithi001.stdout: "down_at": 0, 2024-09-13T11:23:10.012 INFO:teuthology.orchestra.run.smithi001.stdout: "lost_at": 0, 2024-09-13T11:23:10.012 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-09-13T11:23:10.012 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:23:10.013 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:10.013 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:23:10.013 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.177:6813", 2024-09-13T11:23:10.013 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 2110809906 2024-09-13T11:23:10.013 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:10.013 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:10.013 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:10.013 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_addrs": { 2024-09-13T11:23:10.013 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:23:10.013 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:10.013 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:23:10.013 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.177:6814", 2024-09-13T11:23:10.013 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 2110809906 2024-09-13T11:23:10.013 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:10.013 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:10.014 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:10.014 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_back_addrs": { 2024-09-13T11:23:10.014 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:23:10.014 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:10.014 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:23:10.014 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.177:6816", 2024-09-13T11:23:10.014 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 2110809906 2024-09-13T11:23:10.014 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:10.014 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:10.014 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:10.014 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_front_addrs": { 2024-09-13T11:23:10.014 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:23:10.014 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:10.014 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:23:10.015 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.177:6815", 2024-09-13T11:23:10.015 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 2110809906 2024-09-13T11:23:10.015 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:10.015 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:10.015 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:10.015 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.177:6813/2110809906", 2024-09-13T11:23:10.015 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_addr": "172.21.15.177:6814/2110809906", 2024-09-13T11:23:10.015 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_back_addr": "172.21.15.177:6816/2110809906", 2024-09-13T11:23:10.015 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_front_addr": "172.21.15.177:6815/2110809906", 2024-09-13T11:23:10.015 INFO:teuthology.orchestra.run.smithi001.stdout: "state": [ 2024-09-13T11:23:10.015 INFO:teuthology.orchestra.run.smithi001.stdout: "exists", 2024-09-13T11:23:10.015 INFO:teuthology.orchestra.run.smithi001.stdout: "up" 2024-09-13T11:23:10.015 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:10.016 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:10.016 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:10.016 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": 8, 2024-09-13T11:23:10.016 INFO:teuthology.orchestra.run.smithi001.stdout: "uuid": "f3d99325-48ff-403c-8662-1d7745f7c97d", 2024-09-13T11:23:10.016 INFO:teuthology.orchestra.run.smithi001.stdout: "up": 1, 2024-09-13T11:23:10.016 INFO:teuthology.orchestra.run.smithi001.stdout: "in": 1, 2024-09-13T11:23:10.016 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 1, 2024-09-13T11:23:10.016 INFO:teuthology.orchestra.run.smithi001.stdout: "primary_affinity": 1, 2024-09-13T11:23:10.016 INFO:teuthology.orchestra.run.smithi001.stdout: "last_clean_begin": 0, 2024-09-13T11:23:10.016 INFO:teuthology.orchestra.run.smithi001.stdout: "last_clean_end": 0, 2024-09-13T11:23:10.016 INFO:teuthology.orchestra.run.smithi001.stdout: "up_from": 22, 2024-09-13T11:23:10.016 INFO:teuthology.orchestra.run.smithi001.stdout: "up_thru": 57, 2024-09-13T11:23:10.016 INFO:teuthology.orchestra.run.smithi001.stdout: "down_at": 0, 2024-09-13T11:23:10.016 INFO:teuthology.orchestra.run.smithi001.stdout: "lost_at": 0, 2024-09-13T11:23:10.017 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-09-13T11:23:10.017 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:23:10.017 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:10.017 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:23:10.017 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6804", 2024-09-13T11:23:10.017 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 2766567910 2024-09-13T11:23:10.017 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:10.017 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:10.017 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:10.017 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_addrs": { 2024-09-13T11:23:10.017 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:23:10.017 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:10.017 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:23:10.017 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6805", 2024-09-13T11:23:10.018 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 2766567910 2024-09-13T11:23:10.018 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:10.018 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:10.018 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:10.018 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_back_addrs": { 2024-09-13T11:23:10.018 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:23:10.018 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:10.018 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:23:10.018 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6807", 2024-09-13T11:23:10.018 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 2766567910 2024-09-13T11:23:10.018 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:10.018 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:10.018 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:10.018 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_front_addrs": { 2024-09-13T11:23:10.018 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:23:10.019 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:10.019 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:23:10.019 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6806", 2024-09-13T11:23:10.019 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 2766567910 2024-09-13T11:23:10.019 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:10.019 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:10.019 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:10.019 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6804/2766567910", 2024-09-13T11:23:10.019 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_addr": "172.21.15.1:6805/2766567910", 2024-09-13T11:23:10.019 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_back_addr": "172.21.15.1:6807/2766567910", 2024-09-13T11:23:10.019 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_front_addr": "172.21.15.1:6806/2766567910", 2024-09-13T11:23:10.019 INFO:teuthology.orchestra.run.smithi001.stdout: "state": [ 2024-09-13T11:23:10.019 INFO:teuthology.orchestra.run.smithi001.stdout: "exists", 2024-09-13T11:23:10.019 INFO:teuthology.orchestra.run.smithi001.stdout: "up" 2024-09-13T11:23:10.020 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:10.020 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:10.020 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:10.020 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": 9, 2024-09-13T11:23:10.020 INFO:teuthology.orchestra.run.smithi001.stdout: "uuid": "88511e24-3675-4551-968c-1a52e7e28f96", 2024-09-13T11:23:10.020 INFO:teuthology.orchestra.run.smithi001.stdout: "up": 1, 2024-09-13T11:23:10.020 INFO:teuthology.orchestra.run.smithi001.stdout: "in": 1, 2024-09-13T11:23:10.020 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 1, 2024-09-13T11:23:10.020 INFO:teuthology.orchestra.run.smithi001.stdout: "primary_affinity": 1, 2024-09-13T11:23:10.020 INFO:teuthology.orchestra.run.smithi001.stdout: "last_clean_begin": 0, 2024-09-13T11:23:10.020 INFO:teuthology.orchestra.run.smithi001.stdout: "last_clean_end": 0, 2024-09-13T11:23:10.020 INFO:teuthology.orchestra.run.smithi001.stdout: "up_from": 23, 2024-09-13T11:23:10.020 INFO:teuthology.orchestra.run.smithi001.stdout: "up_thru": 30, 2024-09-13T11:23:10.020 INFO:teuthology.orchestra.run.smithi001.stdout: "down_at": 0, 2024-09-13T11:23:10.021 INFO:teuthology.orchestra.run.smithi001.stdout: "lost_at": 0, 2024-09-13T11:23:10.021 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-09-13T11:23:10.021 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:23:10.021 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:10.021 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:23:10.021 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.92:6812", 2024-09-13T11:23:10.021 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 2488187511 2024-09-13T11:23:10.021 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:10.021 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:10.021 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:10.021 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_addrs": { 2024-09-13T11:23:10.021 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:23:10.021 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:10.021 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:23:10.022 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.92:6813", 2024-09-13T11:23:10.022 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 2488187511 2024-09-13T11:23:10.022 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:10.022 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:10.022 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:10.022 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_back_addrs": { 2024-09-13T11:23:10.022 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:23:10.022 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:10.022 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:23:10.022 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.92:6815", 2024-09-13T11:23:10.022 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 2488187511 2024-09-13T11:23:10.022 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:10.022 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:10.023 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:10.023 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_front_addrs": { 2024-09-13T11:23:10.023 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:23:10.023 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:10.023 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:23:10.023 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.92:6814", 2024-09-13T11:23:10.023 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 2488187511 2024-09-13T11:23:10.023 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:10.023 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:10.023 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:10.023 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.92:6812/2488187511", 2024-09-13T11:23:10.023 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_addr": "172.21.15.92:6813/2488187511", 2024-09-13T11:23:10.023 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_back_addr": "172.21.15.92:6815/2488187511", 2024-09-13T11:23:10.023 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_front_addr": "172.21.15.92:6814/2488187511", 2024-09-13T11:23:10.023 INFO:teuthology.orchestra.run.smithi001.stdout: "state": [ 2024-09-13T11:23:10.024 INFO:teuthology.orchestra.run.smithi001.stdout: "exists", 2024-09-13T11:23:10.024 INFO:teuthology.orchestra.run.smithi001.stdout: "up" 2024-09-13T11:23:10.024 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:10.024 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:10.024 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:10.024 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": 10, 2024-09-13T11:23:10.024 INFO:teuthology.orchestra.run.smithi001.stdout: "uuid": "acf4c64d-99af-42be-8406-413c86f54670", 2024-09-13T11:23:10.024 INFO:teuthology.orchestra.run.smithi001.stdout: "up": 1, 2024-09-13T11:23:10.024 INFO:teuthology.orchestra.run.smithi001.stdout: "in": 1, 2024-09-13T11:23:10.024 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 1, 2024-09-13T11:23:10.024 INFO:teuthology.orchestra.run.smithi001.stdout: "primary_affinity": 1, 2024-09-13T11:23:10.024 INFO:teuthology.orchestra.run.smithi001.stdout: "last_clean_begin": 0, 2024-09-13T11:23:10.024 INFO:teuthology.orchestra.run.smithi001.stdout: "last_clean_end": 0, 2024-09-13T11:23:10.025 INFO:teuthology.orchestra.run.smithi001.stdout: "up_from": 22, 2024-09-13T11:23:10.025 INFO:teuthology.orchestra.run.smithi001.stdout: "up_thru": 30, 2024-09-13T11:23:10.025 INFO:teuthology.orchestra.run.smithi001.stdout: "down_at": 0, 2024-09-13T11:23:10.025 INFO:teuthology.orchestra.run.smithi001.stdout: "lost_at": 0, 2024-09-13T11:23:10.025 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-09-13T11:23:10.025 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:23:10.025 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:10.025 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:23:10.025 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.162:6804", 2024-09-13T11:23:10.025 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 1450266459 2024-09-13T11:23:10.025 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:10.025 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:10.025 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:10.025 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_addrs": { 2024-09-13T11:23:10.026 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:23:10.026 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:10.026 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:23:10.026 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.162:6805", 2024-09-13T11:23:10.026 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 1450266459 2024-09-13T11:23:10.026 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:10.026 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:10.026 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:10.026 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_back_addrs": { 2024-09-13T11:23:10.026 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:23:10.026 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:10.026 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:23:10.026 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.162:6807", 2024-09-13T11:23:10.026 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 1450266459 2024-09-13T11:23:10.027 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:10.027 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:10.027 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:10.027 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_front_addrs": { 2024-09-13T11:23:10.027 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:23:10.027 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:10.027 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:23:10.027 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.162:6806", 2024-09-13T11:23:10.027 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 1450266459 2024-09-13T11:23:10.027 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:10.027 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:10.027 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:10.027 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.162:6804/1450266459", 2024-09-13T11:23:10.027 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_addr": "172.21.15.162:6805/1450266459", 2024-09-13T11:23:10.028 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_back_addr": "172.21.15.162:6807/1450266459", 2024-09-13T11:23:10.028 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_front_addr": "172.21.15.162:6806/1450266459", 2024-09-13T11:23:10.028 INFO:teuthology.orchestra.run.smithi001.stdout: "state": [ 2024-09-13T11:23:10.028 INFO:teuthology.orchestra.run.smithi001.stdout: "exists", 2024-09-13T11:23:10.028 INFO:teuthology.orchestra.run.smithi001.stdout: "up" 2024-09-13T11:23:10.028 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:10.028 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:10.028 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:10.028 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": 11, 2024-09-13T11:23:10.028 INFO:teuthology.orchestra.run.smithi001.stdout: "uuid": "a4bfdfbe-b586-4080-8e09-6f26428e59b5", 2024-09-13T11:23:10.028 INFO:teuthology.orchestra.run.smithi001.stdout: "up": 1, 2024-09-13T11:23:10.028 INFO:teuthology.orchestra.run.smithi001.stdout: "in": 1, 2024-09-13T11:23:10.028 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 1, 2024-09-13T11:23:10.029 INFO:teuthology.orchestra.run.smithi001.stdout: "primary_affinity": 1, 2024-09-13T11:23:10.029 INFO:teuthology.orchestra.run.smithi001.stdout: "last_clean_begin": 0, 2024-09-13T11:23:10.029 INFO:teuthology.orchestra.run.smithi001.stdout: "last_clean_end": 0, 2024-09-13T11:23:10.029 INFO:teuthology.orchestra.run.smithi001.stdout: "up_from": 22, 2024-09-13T11:23:10.029 INFO:teuthology.orchestra.run.smithi001.stdout: "up_thru": 30, 2024-09-13T11:23:10.029 INFO:teuthology.orchestra.run.smithi001.stdout: "down_at": 0, 2024-09-13T11:23:10.029 INFO:teuthology.orchestra.run.smithi001.stdout: "lost_at": 0, 2024-09-13T11:23:10.029 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-09-13T11:23:10.029 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:23:10.029 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:10.029 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:23:10.029 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.177:6809", 2024-09-13T11:23:10.029 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 3239521954 2024-09-13T11:23:10.029 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:10.030 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:10.030 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:10.030 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_addrs": { 2024-09-13T11:23:10.030 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:23:10.030 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:10.030 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:23:10.030 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.177:6810", 2024-09-13T11:23:10.030 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 3239521954 2024-09-13T11:23:10.030 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:10.030 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:10.030 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:10.030 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_back_addrs": { 2024-09-13T11:23:10.030 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:23:10.030 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:10.031 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:23:10.031 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.177:6812", 2024-09-13T11:23:10.031 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 3239521954 2024-09-13T11:23:10.031 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:10.031 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:10.031 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:10.031 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_front_addrs": { 2024-09-13T11:23:10.031 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:23:10.031 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:10.031 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:23:10.031 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.177:6811", 2024-09-13T11:23:10.031 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 3239521954 2024-09-13T11:23:10.031 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:10.031 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:10.031 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:10.032 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.177:6809/3239521954", 2024-09-13T11:23:10.032 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_addr": "172.21.15.177:6810/3239521954", 2024-09-13T11:23:10.032 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_back_addr": "172.21.15.177:6812/3239521954", 2024-09-13T11:23:10.032 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_front_addr": "172.21.15.177:6811/3239521954", 2024-09-13T11:23:10.032 INFO:teuthology.orchestra.run.smithi001.stdout: "state": [ 2024-09-13T11:23:10.032 INFO:teuthology.orchestra.run.smithi001.stdout: "exists", 2024-09-13T11:23:10.032 INFO:teuthology.orchestra.run.smithi001.stdout: "up" 2024-09-13T11:23:10.032 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:10.032 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:10.032 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:10.032 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": 12, 2024-09-13T11:23:10.032 INFO:teuthology.orchestra.run.smithi001.stdout: "uuid": "dc571fb6-d8b5-465b-bb25-afa7d0c47b5a", 2024-09-13T11:23:10.032 INFO:teuthology.orchestra.run.smithi001.stdout: "up": 1, 2024-09-13T11:23:10.032 INFO:teuthology.orchestra.run.smithi001.stdout: "in": 1, 2024-09-13T11:23:10.033 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 1, 2024-09-13T11:23:10.033 INFO:teuthology.orchestra.run.smithi001.stdout: "primary_affinity": 1, 2024-09-13T11:23:10.033 INFO:teuthology.orchestra.run.smithi001.stdout: "last_clean_begin": 0, 2024-09-13T11:23:10.033 INFO:teuthology.orchestra.run.smithi001.stdout: "last_clean_end": 0, 2024-09-13T11:23:10.033 INFO:teuthology.orchestra.run.smithi001.stdout: "up_from": 22, 2024-09-13T11:23:10.033 INFO:teuthology.orchestra.run.smithi001.stdout: "up_thru": 30, 2024-09-13T11:23:10.033 INFO:teuthology.orchestra.run.smithi001.stdout: "down_at": 0, 2024-09-13T11:23:10.033 INFO:teuthology.orchestra.run.smithi001.stdout: "lost_at": 0, 2024-09-13T11:23:10.033 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-09-13T11:23:10.033 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:23:10.033 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:10.033 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:23:10.033 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6812", 2024-09-13T11:23:10.033 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 4267062742 2024-09-13T11:23:10.034 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:10.034 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:10.034 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:10.034 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_addrs": { 2024-09-13T11:23:10.034 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:23:10.034 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:10.034 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:23:10.034 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6813", 2024-09-13T11:23:10.034 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 4267062742 2024-09-13T11:23:10.034 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:10.034 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:10.034 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:10.034 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_back_addrs": { 2024-09-13T11:23:10.035 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:23:10.035 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:10.035 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:23:10.035 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6815", 2024-09-13T11:23:10.035 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 4267062742 2024-09-13T11:23:10.035 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:10.035 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:10.035 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:10.035 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_front_addrs": { 2024-09-13T11:23:10.035 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:23:10.035 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:10.035 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:23:10.035 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6814", 2024-09-13T11:23:10.035 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 4267062742 2024-09-13T11:23:10.036 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:10.036 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:10.036 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:10.036 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6812/4267062742", 2024-09-13T11:23:10.036 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_addr": "172.21.15.1:6813/4267062742", 2024-09-13T11:23:10.036 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_back_addr": "172.21.15.1:6815/4267062742", 2024-09-13T11:23:10.036 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_front_addr": "172.21.15.1:6814/4267062742", 2024-09-13T11:23:10.036 INFO:teuthology.orchestra.run.smithi001.stdout: "state": [ 2024-09-13T11:23:10.036 INFO:teuthology.orchestra.run.smithi001.stdout: "exists", 2024-09-13T11:23:10.036 INFO:teuthology.orchestra.run.smithi001.stdout: "up" 2024-09-13T11:23:10.036 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:10.036 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:10.036 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:10.037 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": 13, 2024-09-13T11:23:10.037 INFO:teuthology.orchestra.run.smithi001.stdout: "uuid": "5d321d4b-1b26-47fe-bf84-25751b8cde7d", 2024-09-13T11:23:10.037 INFO:teuthology.orchestra.run.smithi001.stdout: "up": 1, 2024-09-13T11:23:10.037 INFO:teuthology.orchestra.run.smithi001.stdout: "in": 1, 2024-09-13T11:23:10.037 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 1, 2024-09-13T11:23:10.037 INFO:teuthology.orchestra.run.smithi001.stdout: "primary_affinity": 1, 2024-09-13T11:23:10.037 INFO:teuthology.orchestra.run.smithi001.stdout: "last_clean_begin": 0, 2024-09-13T11:23:10.037 INFO:teuthology.orchestra.run.smithi001.stdout: "last_clean_end": 0, 2024-09-13T11:23:10.037 INFO:teuthology.orchestra.run.smithi001.stdout: "up_from": 23, 2024-09-13T11:23:10.037 INFO:teuthology.orchestra.run.smithi001.stdout: "up_thru": 41, 2024-09-13T11:23:10.037 INFO:teuthology.orchestra.run.smithi001.stdout: "down_at": 0, 2024-09-13T11:23:10.037 INFO:teuthology.orchestra.run.smithi001.stdout: "lost_at": 0, 2024-09-13T11:23:10.037 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-09-13T11:23:10.037 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:23:10.038 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:10.038 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:23:10.038 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.92:6808", 2024-09-13T11:23:10.038 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 1759585046 2024-09-13T11:23:10.038 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:10.038 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:10.038 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:10.038 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_addrs": { 2024-09-13T11:23:10.038 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:23:10.038 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:10.038 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:23:10.038 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.92:6809", 2024-09-13T11:23:10.038 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 1759585046 2024-09-13T11:23:10.038 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:10.039 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:10.039 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:10.039 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_back_addrs": { 2024-09-13T11:23:10.039 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:23:10.039 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:10.039 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:23:10.039 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.92:6811", 2024-09-13T11:23:10.039 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 1759585046 2024-09-13T11:23:10.039 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:10.039 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:10.039 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:10.039 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_front_addrs": { 2024-09-13T11:23:10.039 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:23:10.039 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:10.040 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:23:10.040 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.92:6810", 2024-09-13T11:23:10.040 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 1759585046 2024-09-13T11:23:10.040 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:10.040 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:10.040 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:10.040 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.92:6808/1759585046", 2024-09-13T11:23:10.040 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_addr": "172.21.15.92:6809/1759585046", 2024-09-13T11:23:10.040 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_back_addr": "172.21.15.92:6811/1759585046", 2024-09-13T11:23:10.040 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_front_addr": "172.21.15.92:6810/1759585046", 2024-09-13T11:23:10.040 INFO:teuthology.orchestra.run.smithi001.stdout: "state": [ 2024-09-13T11:23:10.040 INFO:teuthology.orchestra.run.smithi001.stdout: "exists", 2024-09-13T11:23:10.040 INFO:teuthology.orchestra.run.smithi001.stdout: "up" 2024-09-13T11:23:10.041 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:10.041 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:10.041 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:10.041 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": 14, 2024-09-13T11:23:10.041 INFO:teuthology.orchestra.run.smithi001.stdout: "uuid": "fc3e2499-0c2a-414d-a25e-2628876ae630", 2024-09-13T11:23:10.041 INFO:teuthology.orchestra.run.smithi001.stdout: "up": 1, 2024-09-13T11:23:10.041 INFO:teuthology.orchestra.run.smithi001.stdout: "in": 1, 2024-09-13T11:23:10.041 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 1, 2024-09-13T11:23:10.041 INFO:teuthology.orchestra.run.smithi001.stdout: "primary_affinity": 1, 2024-09-13T11:23:10.041 INFO:teuthology.orchestra.run.smithi001.stdout: "last_clean_begin": 0, 2024-09-13T11:23:10.041 INFO:teuthology.orchestra.run.smithi001.stdout: "last_clean_end": 0, 2024-09-13T11:23:10.041 INFO:teuthology.orchestra.run.smithi001.stdout: "up_from": 22, 2024-09-13T11:23:10.041 INFO:teuthology.orchestra.run.smithi001.stdout: "up_thru": 40, 2024-09-13T11:23:10.041 INFO:teuthology.orchestra.run.smithi001.stdout: "down_at": 0, 2024-09-13T11:23:10.042 INFO:teuthology.orchestra.run.smithi001.stdout: "lost_at": 0, 2024-09-13T11:23:10.042 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-09-13T11:23:10.042 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:23:10.042 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:10.042 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:23:10.042 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.162:6800", 2024-09-13T11:23:10.042 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 3905144320 2024-09-13T11:23:10.042 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:10.042 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:10.042 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:10.042 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_addrs": { 2024-09-13T11:23:10.042 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:23:10.042 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:10.042 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:23:10.043 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.162:6801", 2024-09-13T11:23:10.043 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 3905144320 2024-09-13T11:23:10.043 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:10.043 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:10.043 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:10.043 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_back_addrs": { 2024-09-13T11:23:10.043 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:23:10.043 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:10.043 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:23:10.043 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.162:6803", 2024-09-13T11:23:10.043 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 3905144320 2024-09-13T11:23:10.043 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:10.043 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:10.043 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:10.044 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_front_addrs": { 2024-09-13T11:23:10.044 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:23:10.044 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:10.044 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:23:10.044 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.162:6802", 2024-09-13T11:23:10.044 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 3905144320 2024-09-13T11:23:10.044 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:10.044 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:10.044 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:10.044 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.162:6800/3905144320", 2024-09-13T11:23:10.044 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_addr": "172.21.15.162:6801/3905144320", 2024-09-13T11:23:10.044 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_back_addr": "172.21.15.162:6803/3905144320", 2024-09-13T11:23:10.044 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_front_addr": "172.21.15.162:6802/3905144320", 2024-09-13T11:23:10.044 INFO:teuthology.orchestra.run.smithi001.stdout: "state": [ 2024-09-13T11:23:10.045 INFO:teuthology.orchestra.run.smithi001.stdout: "exists", 2024-09-13T11:23:10.045 INFO:teuthology.orchestra.run.smithi001.stdout: "up" 2024-09-13T11:23:10.045 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:10.045 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:10.045 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:10.045 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": 15, 2024-09-13T11:23:10.045 INFO:teuthology.orchestra.run.smithi001.stdout: "uuid": "d2f69775-edfe-4527-91ec-f7c7e558bec8", 2024-09-13T11:23:10.045 INFO:teuthology.orchestra.run.smithi001.stdout: "up": 1, 2024-09-13T11:23:10.045 INFO:teuthology.orchestra.run.smithi001.stdout: "in": 1, 2024-09-13T11:23:10.045 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 1, 2024-09-13T11:23:10.045 INFO:teuthology.orchestra.run.smithi001.stdout: "primary_affinity": 1, 2024-09-13T11:23:10.045 INFO:teuthology.orchestra.run.smithi001.stdout: "last_clean_begin": 0, 2024-09-13T11:23:10.045 INFO:teuthology.orchestra.run.smithi001.stdout: "last_clean_end": 0, 2024-09-13T11:23:10.045 INFO:teuthology.orchestra.run.smithi001.stdout: "up_from": 22, 2024-09-13T11:23:10.046 INFO:teuthology.orchestra.run.smithi001.stdout: "up_thru": 30, 2024-09-13T11:23:10.046 INFO:teuthology.orchestra.run.smithi001.stdout: "down_at": 0, 2024-09-13T11:23:10.046 INFO:teuthology.orchestra.run.smithi001.stdout: "lost_at": 0, 2024-09-13T11:23:10.046 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-09-13T11:23:10.046 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:23:10.046 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:10.046 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:23:10.046 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.177:6805", 2024-09-13T11:23:10.046 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 1294337172 2024-09-13T11:23:10.046 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:10.046 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:10.046 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:10.046 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_addrs": { 2024-09-13T11:23:10.046 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:23:10.046 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:10.047 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:23:10.047 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.177:6806", 2024-09-13T11:23:10.047 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 1294337172 2024-09-13T11:23:10.047 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:10.047 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:10.047 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:10.047 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_back_addrs": { 2024-09-13T11:23:10.047 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:23:10.047 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:10.047 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:23:10.047 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.177:6808", 2024-09-13T11:23:10.047 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 1294337172 2024-09-13T11:23:10.047 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:10.047 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:10.048 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:10.048 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_front_addrs": { 2024-09-13T11:23:10.048 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:23:10.048 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:10.048 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:23:10.048 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.177:6807", 2024-09-13T11:23:10.048 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 1294337172 2024-09-13T11:23:10.048 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:10.048 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:10.048 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:10.048 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.177:6805/1294337172", 2024-09-13T11:23:10.048 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_addr": "172.21.15.177:6806/1294337172", 2024-09-13T11:23:10.048 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_back_addr": "172.21.15.177:6808/1294337172", 2024-09-13T11:23:10.048 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_front_addr": "172.21.15.177:6807/1294337172", 2024-09-13T11:23:10.049 INFO:teuthology.orchestra.run.smithi001.stdout: "state": [ 2024-09-13T11:23:10.049 INFO:teuthology.orchestra.run.smithi001.stdout: "exists", 2024-09-13T11:23:10.049 INFO:teuthology.orchestra.run.smithi001.stdout: "up" 2024-09-13T11:23:10.049 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:10.049 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:10.049 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-09-13T11:23:10.049 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_xinfo": [ 2024-09-13T11:23:10.049 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:10.049 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": 0, 2024-09-13T11:23:10.049 INFO:teuthology.orchestra.run.smithi001.stdout: "down_stamp": "0.000000", 2024-09-13T11:23:10.049 INFO:teuthology.orchestra.run.smithi001.stdout: "laggy_probability": 0, 2024-09-13T11:23:10.049 INFO:teuthology.orchestra.run.smithi001.stdout: "laggy_interval": 0, 2024-09-13T11:23:10.049 INFO:teuthology.orchestra.run.smithi001.stdout: "features": 4540719139924082687, 2024-09-13T11:23:10.049 INFO:teuthology.orchestra.run.smithi001.stdout: "old_weight": 0, 2024-09-13T11:23:10.050 INFO:teuthology.orchestra.run.smithi001.stdout: "last_purged_snaps_scrub": "2024-09-13T11:22:08.233274+0000", 2024-09-13T11:23:10.050 INFO:teuthology.orchestra.run.smithi001.stdout: "dead_epoch": 0 2024-09-13T11:23:10.050 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:10.050 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:10.050 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": 1, 2024-09-13T11:23:10.050 INFO:teuthology.orchestra.run.smithi001.stdout: "down_stamp": "0.000000", 2024-09-13T11:23:10.050 INFO:teuthology.orchestra.run.smithi001.stdout: "laggy_probability": 0, 2024-09-13T11:23:10.050 INFO:teuthology.orchestra.run.smithi001.stdout: "laggy_interval": 0, 2024-09-13T11:23:10.050 INFO:teuthology.orchestra.run.smithi001.stdout: "features": 4540719139924082687, 2024-09-13T11:23:10.050 INFO:teuthology.orchestra.run.smithi001.stdout: "old_weight": 0, 2024-09-13T11:23:10.050 INFO:teuthology.orchestra.run.smithi001.stdout: "last_purged_snaps_scrub": "2024-09-13T11:22:08.032163+0000", 2024-09-13T11:23:10.050 INFO:teuthology.orchestra.run.smithi001.stdout: "dead_epoch": 30 2024-09-13T11:23:10.050 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:10.050 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:10.051 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": 2, 2024-09-13T11:23:10.051 INFO:teuthology.orchestra.run.smithi001.stdout: "down_stamp": "0.000000", 2024-09-13T11:23:10.051 INFO:teuthology.orchestra.run.smithi001.stdout: "laggy_probability": 0, 2024-09-13T11:23:10.051 INFO:teuthology.orchestra.run.smithi001.stdout: "laggy_interval": 0, 2024-09-13T11:23:10.051 INFO:teuthology.orchestra.run.smithi001.stdout: "features": 4540719139924082687, 2024-09-13T11:23:10.051 INFO:teuthology.orchestra.run.smithi001.stdout: "old_weight": 0, 2024-09-13T11:23:10.051 INFO:teuthology.orchestra.run.smithi001.stdout: "last_purged_snaps_scrub": "2024-09-13T11:22:08.875236+0000", 2024-09-13T11:23:10.051 INFO:teuthology.orchestra.run.smithi001.stdout: "dead_epoch": 0 2024-09-13T11:23:10.051 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:10.051 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:10.051 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": 3, 2024-09-13T11:23:10.051 INFO:teuthology.orchestra.run.smithi001.stdout: "down_stamp": "0.000000", 2024-09-13T11:23:10.051 INFO:teuthology.orchestra.run.smithi001.stdout: "laggy_probability": 0, 2024-09-13T11:23:10.051 INFO:teuthology.orchestra.run.smithi001.stdout: "laggy_interval": 0, 2024-09-13T11:23:10.052 INFO:teuthology.orchestra.run.smithi001.stdout: "features": 4540719139924082687, 2024-09-13T11:23:10.052 INFO:teuthology.orchestra.run.smithi001.stdout: "old_weight": 0, 2024-09-13T11:23:10.052 INFO:teuthology.orchestra.run.smithi001.stdout: "last_purged_snaps_scrub": "2024-09-13T11:22:07.936983+0000", 2024-09-13T11:23:10.052 INFO:teuthology.orchestra.run.smithi001.stdout: "dead_epoch": 0 2024-09-13T11:23:10.052 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:10.052 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:10.052 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": 4, 2024-09-13T11:23:10.052 INFO:teuthology.orchestra.run.smithi001.stdout: "down_stamp": "0.000000", 2024-09-13T11:23:10.052 INFO:teuthology.orchestra.run.smithi001.stdout: "laggy_probability": 0, 2024-09-13T11:23:10.052 INFO:teuthology.orchestra.run.smithi001.stdout: "laggy_interval": 0, 2024-09-13T11:23:10.052 INFO:teuthology.orchestra.run.smithi001.stdout: "features": 4540719139924082687, 2024-09-13T11:23:10.052 INFO:teuthology.orchestra.run.smithi001.stdout: "old_weight": 0, 2024-09-13T11:23:10.052 INFO:teuthology.orchestra.run.smithi001.stdout: "last_purged_snaps_scrub": "2024-09-13T11:22:08.625189+0000", 2024-09-13T11:23:10.052 INFO:teuthology.orchestra.run.smithi001.stdout: "dead_epoch": 0 2024-09-13T11:23:10.053 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:10.053 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:10.053 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": 5, 2024-09-13T11:23:10.053 INFO:teuthology.orchestra.run.smithi001.stdout: "down_stamp": "0.000000", 2024-09-13T11:23:10.053 INFO:teuthology.orchestra.run.smithi001.stdout: "laggy_probability": 0, 2024-09-13T11:23:10.053 INFO:teuthology.orchestra.run.smithi001.stdout: "laggy_interval": 0, 2024-09-13T11:23:10.053 INFO:teuthology.orchestra.run.smithi001.stdout: "features": 4540719139924082687, 2024-09-13T11:23:10.053 INFO:teuthology.orchestra.run.smithi001.stdout: "old_weight": 0, 2024-09-13T11:23:10.053 INFO:teuthology.orchestra.run.smithi001.stdout: "last_purged_snaps_scrub": "2024-09-13T11:22:07.944287+0000", 2024-09-13T11:23:10.053 INFO:teuthology.orchestra.run.smithi001.stdout: "dead_epoch": 0 2024-09-13T11:23:10.053 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:10.053 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:10.053 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": 6, 2024-09-13T11:23:10.053 INFO:teuthology.orchestra.run.smithi001.stdout: "down_stamp": "0.000000", 2024-09-13T11:23:10.054 INFO:teuthology.orchestra.run.smithi001.stdout: "laggy_probability": 0, 2024-09-13T11:23:10.054 INFO:teuthology.orchestra.run.smithi001.stdout: "laggy_interval": 0, 2024-09-13T11:23:10.054 INFO:teuthology.orchestra.run.smithi001.stdout: "features": 4540719139924082687, 2024-09-13T11:23:10.054 INFO:teuthology.orchestra.run.smithi001.stdout: "old_weight": 0, 2024-09-13T11:23:10.054 INFO:teuthology.orchestra.run.smithi001.stdout: "last_purged_snaps_scrub": "2024-09-13T11:22:08.351045+0000", 2024-09-13T11:23:10.054 INFO:teuthology.orchestra.run.smithi001.stdout: "dead_epoch": 0 2024-09-13T11:23:10.054 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:10.054 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:10.054 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": 7, 2024-09-13T11:23:10.054 INFO:teuthology.orchestra.run.smithi001.stdout: "down_stamp": "0.000000", 2024-09-13T11:23:10.054 INFO:teuthology.orchestra.run.smithi001.stdout: "laggy_probability": 0, 2024-09-13T11:23:10.054 INFO:teuthology.orchestra.run.smithi001.stdout: "laggy_interval": 0, 2024-09-13T11:23:10.054 INFO:teuthology.orchestra.run.smithi001.stdout: "features": 4540719139924082687, 2024-09-13T11:23:10.054 INFO:teuthology.orchestra.run.smithi001.stdout: "old_weight": 0, 2024-09-13T11:23:10.055 INFO:teuthology.orchestra.run.smithi001.stdout: "last_purged_snaps_scrub": "2024-09-13T11:22:08.835090+0000", 2024-09-13T11:23:10.055 INFO:teuthology.orchestra.run.smithi001.stdout: "dead_epoch": 0 2024-09-13T11:23:10.055 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:10.055 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:10.055 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": 8, 2024-09-13T11:23:10.055 INFO:teuthology.orchestra.run.smithi001.stdout: "down_stamp": "0.000000", 2024-09-13T11:23:10.055 INFO:teuthology.orchestra.run.smithi001.stdout: "laggy_probability": 0, 2024-09-13T11:23:10.055 INFO:teuthology.orchestra.run.smithi001.stdout: "laggy_interval": 0, 2024-09-13T11:23:10.055 INFO:teuthology.orchestra.run.smithi001.stdout: "features": 4540719139924082687, 2024-09-13T11:23:10.055 INFO:teuthology.orchestra.run.smithi001.stdout: "old_weight": 0, 2024-09-13T11:23:10.055 INFO:teuthology.orchestra.run.smithi001.stdout: "last_purged_snaps_scrub": "2024-09-13T11:22:08.847356+0000", 2024-09-13T11:23:10.055 INFO:teuthology.orchestra.run.smithi001.stdout: "dead_epoch": 0 2024-09-13T11:23:10.055 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:10.055 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:10.055 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": 9, 2024-09-13T11:23:10.056 INFO:teuthology.orchestra.run.smithi001.stdout: "down_stamp": "0.000000", 2024-09-13T11:23:10.056 INFO:teuthology.orchestra.run.smithi001.stdout: "laggy_probability": 0, 2024-09-13T11:23:10.056 INFO:teuthology.orchestra.run.smithi001.stdout: "laggy_interval": 0, 2024-09-13T11:23:10.056 INFO:teuthology.orchestra.run.smithi001.stdout: "features": 4540719139924082687, 2024-09-13T11:23:10.056 INFO:teuthology.orchestra.run.smithi001.stdout: "old_weight": 0, 2024-09-13T11:23:10.056 INFO:teuthology.orchestra.run.smithi001.stdout: "last_purged_snaps_scrub": "2024-09-13T11:22:08.140353+0000", 2024-09-13T11:23:10.056 INFO:teuthology.orchestra.run.smithi001.stdout: "dead_epoch": 0 2024-09-13T11:23:10.056 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:10.056 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:10.056 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": 10, 2024-09-13T11:23:10.056 INFO:teuthology.orchestra.run.smithi001.stdout: "down_stamp": "0.000000", 2024-09-13T11:23:10.056 INFO:teuthology.orchestra.run.smithi001.stdout: "laggy_probability": 0, 2024-09-13T11:23:10.056 INFO:teuthology.orchestra.run.smithi001.stdout: "laggy_interval": 0, 2024-09-13T11:23:10.057 INFO:teuthology.orchestra.run.smithi001.stdout: "features": 4540719139924082687, 2024-09-13T11:23:10.057 INFO:teuthology.orchestra.run.smithi001.stdout: "old_weight": 0, 2024-09-13T11:23:10.057 INFO:teuthology.orchestra.run.smithi001.stdout: "last_purged_snaps_scrub": "2024-09-13T11:22:08.218416+0000", 2024-09-13T11:23:10.057 INFO:teuthology.orchestra.run.smithi001.stdout: "dead_epoch": 0 2024-09-13T11:23:10.057 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:10.057 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:10.057 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": 11, 2024-09-13T11:23:10.057 INFO:teuthology.orchestra.run.smithi001.stdout: "down_stamp": "0.000000", 2024-09-13T11:23:10.057 INFO:teuthology.orchestra.run.smithi001.stdout: "laggy_probability": 0, 2024-09-13T11:23:10.057 INFO:teuthology.orchestra.run.smithi001.stdout: "laggy_interval": 0, 2024-09-13T11:23:10.057 INFO:teuthology.orchestra.run.smithi001.stdout: "features": 4540719139924082687, 2024-09-13T11:23:10.057 INFO:teuthology.orchestra.run.smithi001.stdout: "old_weight": 0, 2024-09-13T11:23:10.057 INFO:teuthology.orchestra.run.smithi001.stdout: "last_purged_snaps_scrub": "2024-09-13T11:22:08.223142+0000", 2024-09-13T11:23:10.057 INFO:teuthology.orchestra.run.smithi001.stdout: "dead_epoch": 0 2024-09-13T11:23:10.057 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:10.058 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:10.058 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": 12, 2024-09-13T11:23:10.058 INFO:teuthology.orchestra.run.smithi001.stdout: "down_stamp": "0.000000", 2024-09-13T11:23:10.058 INFO:teuthology.orchestra.run.smithi001.stdout: "laggy_probability": 0, 2024-09-13T11:23:10.058 INFO:teuthology.orchestra.run.smithi001.stdout: "laggy_interval": 0, 2024-09-13T11:23:10.058 INFO:teuthology.orchestra.run.smithi001.stdout: "features": 4540719139924082687, 2024-09-13T11:23:10.058 INFO:teuthology.orchestra.run.smithi001.stdout: "old_weight": 0, 2024-09-13T11:23:10.058 INFO:teuthology.orchestra.run.smithi001.stdout: "last_purged_snaps_scrub": "2024-09-13T11:22:08.687866+0000", 2024-09-13T11:23:10.058 INFO:teuthology.orchestra.run.smithi001.stdout: "dead_epoch": 0 2024-09-13T11:23:10.058 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:10.058 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:10.058 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": 13, 2024-09-13T11:23:10.058 INFO:teuthology.orchestra.run.smithi001.stdout: "down_stamp": "0.000000", 2024-09-13T11:23:10.058 INFO:teuthology.orchestra.run.smithi001.stdout: "laggy_probability": 0, 2024-09-13T11:23:10.059 INFO:teuthology.orchestra.run.smithi001.stdout: "laggy_interval": 0, 2024-09-13T11:23:10.059 INFO:teuthology.orchestra.run.smithi001.stdout: "features": 4540719139924082687, 2024-09-13T11:23:10.059 INFO:teuthology.orchestra.run.smithi001.stdout: "old_weight": 0, 2024-09-13T11:23:10.059 INFO:teuthology.orchestra.run.smithi001.stdout: "last_purged_snaps_scrub": "2024-09-13T11:22:08.347174+0000", 2024-09-13T11:23:10.059 INFO:teuthology.orchestra.run.smithi001.stdout: "dead_epoch": 0 2024-09-13T11:23:10.059 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:10.059 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:10.059 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": 14, 2024-09-13T11:23:10.059 INFO:teuthology.orchestra.run.smithi001.stdout: "down_stamp": "0.000000", 2024-09-13T11:23:10.059 INFO:teuthology.orchestra.run.smithi001.stdout: "laggy_probability": 0, 2024-09-13T11:23:10.059 INFO:teuthology.orchestra.run.smithi001.stdout: "laggy_interval": 0, 2024-09-13T11:23:10.059 INFO:teuthology.orchestra.run.smithi001.stdout: "features": 4540719139924082687, 2024-09-13T11:23:10.059 INFO:teuthology.orchestra.run.smithi001.stdout: "old_weight": 0, 2024-09-13T11:23:10.059 INFO:teuthology.orchestra.run.smithi001.stdout: "last_purged_snaps_scrub": "2024-09-13T11:22:08.223685+0000", 2024-09-13T11:23:10.060 INFO:teuthology.orchestra.run.smithi001.stdout: "dead_epoch": 0 2024-09-13T11:23:10.060 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:10.060 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:10.060 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": 15, 2024-09-13T11:23:10.060 INFO:teuthology.orchestra.run.smithi001.stdout: "down_stamp": "0.000000", 2024-09-13T11:23:10.060 INFO:teuthology.orchestra.run.smithi001.stdout: "laggy_probability": 0, 2024-09-13T11:23:10.060 INFO:teuthology.orchestra.run.smithi001.stdout: "laggy_interval": 0, 2024-09-13T11:23:10.060 INFO:teuthology.orchestra.run.smithi001.stdout: "features": 4540719139924082687, 2024-09-13T11:23:10.060 INFO:teuthology.orchestra.run.smithi001.stdout: "old_weight": 0, 2024-09-13T11:23:10.060 INFO:teuthology.orchestra.run.smithi001.stdout: "last_purged_snaps_scrub": "2024-09-13T11:22:08.170891+0000", 2024-09-13T11:23:10.060 INFO:teuthology.orchestra.run.smithi001.stdout: "dead_epoch": 0 2024-09-13T11:23:10.060 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:10.060 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-09-13T11:23:10.060 INFO:teuthology.orchestra.run.smithi001.stdout: "pg_upmap": [ 2024-09-13T11:23:10.061 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:10.061 INFO:teuthology.orchestra.run.smithi001.stdout: "pgid": "3.e", 2024-09-13T11:23:10.061 INFO:teuthology.orchestra.run.smithi001.stdout: "osds": [ 2024-09-13T11:23:10.061 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-09-13T11:23:10.061 INFO:teuthology.orchestra.run.smithi001.stdout: 1 2024-09-13T11:23:10.061 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:10.061 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:10.061 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-09-13T11:23:10.061 INFO:teuthology.orchestra.run.smithi001.stdout: "pg_upmap_items": [], 2024-09-13T11:23:10.061 INFO:teuthology.orchestra.run.smithi001.stdout: "pg_upmap_primaries": [], 2024-09-13T11:23:10.061 INFO:teuthology.orchestra.run.smithi001.stdout: "pg_temp": [ 2024-09-13T11:23:10.061 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:10.061 INFO:teuthology.orchestra.run.smithi001.stdout: "pgid": "3.e", 2024-09-13T11:23:10.062 INFO:teuthology.orchestra.run.smithi001.stdout: "osds": [ 2024-09-13T11:23:10.062 INFO:teuthology.orchestra.run.smithi001.stdout: 8, 2024-09-13T11:23:10.062 INFO:teuthology.orchestra.run.smithi001.stdout: 0 2024-09-13T11:23:10.062 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:10.062 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:10.062 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-09-13T11:23:10.062 INFO:teuthology.orchestra.run.smithi001.stdout: "primary_temp": [], 2024-09-13T11:23:10.062 INFO:teuthology.orchestra.run.smithi001.stdout: "blocklist": {}, 2024-09-13T11:23:10.062 INFO:teuthology.orchestra.run.smithi001.stdout: "range_blocklist": {}, 2024-09-13T11:23:10.062 INFO:teuthology.orchestra.run.smithi001.stdout: "erasure_code_profiles": { 2024-09-13T11:23:10.062 INFO:teuthology.orchestra.run.smithi001.stdout: "default": { 2024-09-13T11:23:10.062 INFO:teuthology.orchestra.run.smithi001.stdout: "crush-failure-domain": "osd", 2024-09-13T11:23:10.062 INFO:teuthology.orchestra.run.smithi001.stdout: "k": "2", 2024-09-13T11:23:10.062 INFO:teuthology.orchestra.run.smithi001.stdout: "m": "1", 2024-09-13T11:23:10.063 INFO:teuthology.orchestra.run.smithi001.stdout: "plugin": "jerasure", 2024-09-13T11:23:10.063 INFO:teuthology.orchestra.run.smithi001.stdout: "technique": "reed_sol_van" 2024-09-13T11:23:10.063 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:10.063 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:10.063 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_snaps_queue": [], 2024-09-13T11:23:10.063 INFO:teuthology.orchestra.run.smithi001.stdout: "new_removed_snaps": [], 2024-09-13T11:23:10.063 INFO:teuthology.orchestra.run.smithi001.stdout: "new_purged_snaps": [], 2024-09-13T11:23:10.063 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_node_flags": {}, 2024-09-13T11:23:10.063 INFO:teuthology.orchestra.run.smithi001.stdout: "device_class_flags": {}, 2024-09-13T11:23:10.063 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": { 2024-09-13T11:23:10.063 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode_enabled": false, 2024-09-13T11:23:10.063 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_bucket_count": 0, 2024-09-13T11:23:10.063 INFO:teuthology.orchestra.run.smithi001.stdout: "degraded_stretch_mode": 0, 2024-09-13T11:23:10.063 INFO:teuthology.orchestra.run.smithi001.stdout: "recovering_stretch_mode": 0, 2024-09-13T11:23:10.064 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode_bucket": 0 2024-09-13T11:23:10.064 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:10.064 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-13T11:23:10.065 INFO:tasks.thrashosds.thrasher:j is {'epoch': 59, 'fsid': '2fd68ac7-5581-4d90-80a4-30bcf5252a7a', 'created': '2024-09-13T11:21:57.223153+0000', 'modified': '2024-09-13T11:23:07.489625+0000', 'last_up_change': '2024-09-13T11:22:46.271946+0000', 'last_in_change': '2024-09-13T11:22:01.745814+0000', 'flags': 'nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5804032, 'flags_set': ['nodeep-scrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 6, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 3, 'max_osd': 16, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-13T11:22:10.162266+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 15.380000114440918, 'score_stable': 15.380000114440918, 'optimal_score': 0.12999999523162842, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2024-09-13T11:22:12.235549+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '28', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 2, 'snap_epoch': 28, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 4.019999980926514, 'score_stable': 4.019999980926514, 'optimal_score': 0.5600000023841858, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'unique_pool_0', 'create_time': '2024-09-13T11:22:24.305521+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '32', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 4.010000228881836, 'score_stable': 4.010000228881836, 'optimal_score': 0.8100000023841858, 'raw_score_acting': 3.25, 'raw_score_stable': 3.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}], 'osds': [{'osd': 0, 'uuid': '0fc12b3c-e0de-4546-b935-db77efe7c567', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 22, 'up_thru': 56, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.1:6800', 'nonce': 1544776477}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.1:6801', 'nonce': 1544776477}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.1:6803', 'nonce': 1544776477}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.1:6802', 'nonce': 1544776477}]}, 'public_addr': '172.21.15.1:6800/1544776477', 'cluster_addr': '172.21.15.1:6801/1544776477', 'heartbeat_back_addr': '172.21.15.1:6803/1544776477', 'heartbeat_front_addr': '172.21.15.1:6802/1544776477', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': 'b77fe331-cc92-4adb-b146-af5c380f6d4f', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 23, 'last_clean_end': 29, 'up_from': 40, 'up_thru': 40, 'down_at': 30, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.92:6800', 'nonce': 3591430847}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.92:6801', 'nonce': 3591430847}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.92:6803', 'nonce': 3591430847}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.92:6802', 'nonce': 3591430847}]}, 'public_addr': '172.21.15.92:6800/3591430847', 'cluster_addr': '172.21.15.92:6801/3591430847', 'heartbeat_back_addr': '172.21.15.92:6803/3591430847', 'heartbeat_front_addr': '172.21.15.92:6802/3591430847', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': 'c9d031c8-e885-4229-affa-08e1729d00e8', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 22, 'up_thru': 0, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.162:6812', 'nonce': 2712148101}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.162:6813', 'nonce': 2712148101}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.162:6815', 'nonce': 2712148101}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.162:6814', 'nonce': 2712148101}]}, 'public_addr': '172.21.15.162:6812/2712148101', 'cluster_addr': '172.21.15.162:6813/2712148101', 'heartbeat_back_addr': '172.21.15.162:6815/2712148101', 'heartbeat_front_addr': '172.21.15.162:6814/2712148101', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': 'bf50d71a-6c3a-4fbc-a407-cb81bad14a7b', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 21, 'up_thru': 30, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.177:6801', 'nonce': 3367751690}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.177:6802', 'nonce': 3367751690}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.177:6804', 'nonce': 3367751690}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.177:6803', 'nonce': 3367751690}]}, 'public_addr': '172.21.15.177:6801/3367751690', 'cluster_addr': '172.21.15.177:6802/3367751690', 'heartbeat_back_addr': '172.21.15.177:6804/3367751690', 'heartbeat_front_addr': '172.21.15.177:6803/3367751690', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': 'd1fe3f63-df3f-4a73-a1e2-8d7c8a3f1fa0', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 22, 'up_thru': 0, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.1:6808', 'nonce': 3648540926}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.1:6809', 'nonce': 3648540926}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.1:6811', 'nonce': 3648540926}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.1:6810', 'nonce': 3648540926}]}, 'public_addr': '172.21.15.1:6808/3648540926', 'cluster_addr': '172.21.15.1:6809/3648540926', 'heartbeat_back_addr': '172.21.15.1:6811/3648540926', 'heartbeat_front_addr': '172.21.15.1:6810/3648540926', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': 'e2ead447-2f2c-40c9-82e7-53b8dc41e2af', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 21, 'up_thru': 0, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.92:6804', 'nonce': 3399529091}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.92:6805', 'nonce': 3399529091}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.92:6807', 'nonce': 3399529091}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.92:6806', 'nonce': 3399529091}]}, 'public_addr': '172.21.15.92:6804/3399529091', 'cluster_addr': '172.21.15.92:6805/3399529091', 'heartbeat_back_addr': '172.21.15.92:6807/3399529091', 'heartbeat_front_addr': '172.21.15.92:6806/3399529091', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': 'dbc20439-144e-4ec7-89de-674eb783f263', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 22, 'up_thru': 40, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.162:6808', 'nonce': 2287343806}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.162:6809', 'nonce': 2287343806}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.162:6811', 'nonce': 2287343806}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.162:6810', 'nonce': 2287343806}]}, 'public_addr': '172.21.15.162:6808/2287343806', 'cluster_addr': '172.21.15.162:6809/2287343806', 'heartbeat_back_addr': '172.21.15.162:6811/2287343806', 'heartbeat_front_addr': '172.21.15.162:6810/2287343806', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '60aa5206-cace-40f8-8d1a-10ec28d97096', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 22, 'up_thru': 30, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.177:6813', 'nonce': 2110809906}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.177:6814', 'nonce': 2110809906}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.177:6816', 'nonce': 2110809906}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.177:6815', 'nonce': 2110809906}]}, 'public_addr': '172.21.15.177:6813/2110809906', 'cluster_addr': '172.21.15.177:6814/2110809906', 'heartbeat_back_addr': '172.21.15.177:6816/2110809906', 'heartbeat_front_addr': '172.21.15.177:6815/2110809906', 'state': ['exists', 'up']}, {'osd': 8, 'uuid': 'f3d99325-48ff-403c-8662-1d7745f7c97d', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 22, 'up_thru': 57, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.1:6804', 'nonce': 2766567910}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.1:6805', 'nonce': 2766567910}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.1:6807', 'nonce': 2766567910}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.1:6806', 'nonce': 2766567910}]}, 'public_addr': '172.21.15.1:6804/2766567910', 'cluster_addr': '172.21.15.1:6805/2766567910', 'heartbeat_back_addr': '172.21.15.1:6807/2766567910', 'heartbeat_front_addr': '172.21.15.1:6806/2766567910', 'state': ['exists', 'up']}, {'osd': 9, 'uuid': '88511e24-3675-4551-968c-1a52e7e28f96', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 23, 'up_thru': 30, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.92:6812', 'nonce': 2488187511}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.92:6813', 'nonce': 2488187511}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.92:6815', 'nonce': 2488187511}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.92:6814', 'nonce': 2488187511}]}, 'public_addr': '172.21.15.92:6812/2488187511', 'cluster_addr': '172.21.15.92:6813/2488187511', 'heartbeat_back_addr': '172.21.15.92:6815/2488187511', 'heartbeat_front_addr': '172.21.15.92:6814/2488187511', 'state': ['exists', 'up']}, {'osd': 10, 'uuid': 'acf4c64d-99af-42be-8406-413c86f54670', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 22, 'up_thru': 30, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.162:6804', 'nonce': 1450266459}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.162:6805', 'nonce': 1450266459}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.162:6807', 'nonce': 1450266459}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.162:6806', 'nonce': 1450266459}]}, 'public_addr': '172.21.15.162:6804/1450266459', 'cluster_addr': '172.21.15.162:6805/1450266459', 'heartbeat_back_addr': '172.21.15.162:6807/1450266459', 'heartbeat_front_addr': '172.21.15.162:6806/1450266459', 'state': ['exists', 'up']}, {'osd': 11, 'uuid': 'a4bfdfbe-b586-4080-8e09-6f26428e59b5', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 22, 'up_thru': 30, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.177:6809', 'nonce': 3239521954}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.177:6810', 'nonce': 3239521954}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.177:6812', 'nonce': 3239521954}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.177:6811', 'nonce': 3239521954}]}, 'public_addr': '172.21.15.177:6809/3239521954', 'cluster_addr': '172.21.15.177:6810/3239521954', 'heartbeat_back_addr': '172.21.15.177:6812/3239521954', 'heartbeat_front_addr': '172.21.15.177:6811/3239521954', 'state': ['exists', 'up']}, {'osd': 12, 'uuid': 'dc571fb6-d8b5-465b-bb25-afa7d0c47b5a', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 22, 'up_thru': 30, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.1:6812', 'nonce': 4267062742}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.1:6813', 'nonce': 4267062742}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.1:6815', 'nonce': 4267062742}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.1:6814', 'nonce': 4267062742}]}, 'public_addr': '172.21.15.1:6812/4267062742', 'cluster_addr': '172.21.15.1:6813/4267062742', 'heartbeat_back_addr': '172.21.15.1:6815/4267062742', 'heartbeat_front_addr': '172.21.15.1:6814/4267062742', 'state': ['exists', 'up']}, {'osd': 13, 'uuid': '5d321d4b-1b26-47fe-bf84-25751b8cde7d', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 23, 'up_thru': 41, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.92:6808', 'nonce': 1759585046}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.92:6809', 'nonce': 1759585046}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.92:6811', 'nonce': 1759585046}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.92:6810', 'nonce': 1759585046}]}, 'public_addr': '172.21.15.92:6808/1759585046', 'cluster_addr': '172.21.15.92:6809/1759585046', 'heartbeat_back_addr': '172.21.15.92:6811/1759585046', 'heartbeat_front_addr': '172.21.15.92:6810/1759585046', 'state': ['exists', 'up']}, {'osd': 14, 'uuid': 'fc3e2499-0c2a-414d-a25e-2628876ae630', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 22, 'up_thru': 40, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.162:6800', 'nonce': 3905144320}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.162:6801', 'nonce': 3905144320}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.162:6803', 'nonce': 3905144320}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.162:6802', 'nonce': 3905144320}]}, 'public_addr': '172.21.15.162:6800/3905144320', 'cluster_addr': '172.21.15.162:6801/3905144320', 'heartbeat_back_addr': '172.21.15.162:6803/3905144320', 'heartbeat_front_addr': '172.21.15.162:6802/3905144320', 'state': ['exists', 'up']}, {'osd': 15, 'uuid': 'd2f69775-edfe-4527-91ec-f7c7e558bec8', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 22, 'up_thru': 30, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.177:6805', 'nonce': 1294337172}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.177:6806', 'nonce': 1294337172}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.177:6808', 'nonce': 1294337172}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.177:6807', 'nonce': 1294337172}]}, 'public_addr': '172.21.15.177:6805/1294337172', 'cluster_addr': '172.21.15.177:6806/1294337172', 'heartbeat_back_addr': '172.21.15.177:6808/1294337172', 'heartbeat_front_addr': '172.21.15.177:6807/1294337172', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-13T11:22:08.233274+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-13T11:22:08.032163+0000', 'dead_epoch': 30}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-13T11:22:08.875236+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-13T11:22:07.936983+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-13T11:22:08.625189+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-13T11:22:07.944287+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-13T11:22:08.351045+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-13T11:22:08.835090+0000', 'dead_epoch': 0}, {'osd': 8, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-13T11:22:08.847356+0000', 'dead_epoch': 0}, {'osd': 9, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-13T11:22:08.140353+0000', 'dead_epoch': 0}, {'osd': 10, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-13T11:22:08.218416+0000', 'dead_epoch': 0}, {'osd': 11, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-13T11:22:08.223142+0000', 'dead_epoch': 0}, {'osd': 12, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-13T11:22:08.687866+0000', 'dead_epoch': 0}, {'osd': 13, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-13T11:22:08.347174+0000', 'dead_epoch': 0}, {'osd': 14, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-13T11:22:08.223685+0000', 'dead_epoch': 0}, {'osd': 15, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-13T11:22:08.170891+0000', 'dead_epoch': 0}], 'pg_upmap': [{'pgid': '3.e', 'osds': [0, 1]}], 'pg_upmap_items': [], 'pg_upmap_primaries': [], 'pg_temp': [{'pgid': '3.e', 'osds': [8, 0]}], 'primary_temp': [], 'blocklist': {}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-09-13T11:23:10.065 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-13T11:23:10.066 INFO:tasks.ceph.osd.4.smithi001.stderr:2024-09-13T11:23:09.952+0000 7ff8e0218640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26310) UID: 0 2024-09-13T11:23:10.067 INFO:tasks.radosbench.radosbench.0.smithi177.stdout: 43 16 31137 31121 45.2286 58.375 0.0237381 0.0220959 2024-09-13T11:23:10.067 INFO:tasks.ceph.osd.5.smithi092.stderr:2024-09-13T11:23:10.062+0000 7f6f0b705640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26346) UID: 0 2024-09-13T11:23:10.168 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-13T11:23:10.164+0000 7f6a68775640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26307) UID: 0 2024-09-13T11:23:10.221 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-13T11:23:10.221 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [ 2024-09-13T11:23:10.221 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:10.221 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:31529 3.2 3:49df26c7:::benchmark_data_smithi177_31116_object31528:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e59)", 2024-09-13T11:23:10.221 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:10.220101+0000", 2024-09-13T11:23:10.221 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 0.00059481599999999996, 2024-09-13T11:23:10.222 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00062553700000000001, 2024-09-13T11:23:10.222 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:23:10.222 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:23:10.222 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "waiting for sub ops", 2024-09-13T11:23:10.222 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:23:10.222 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:23:10.222 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:10.222 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 31529 2024-09-13T11:23:10.222 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:10.222 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:23:10.222 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:10.222 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:23:10.222 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:10.220101+0000", 2024-09-13T11:23:10.222 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:10.223 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:10.223 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:10.223 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:23:10.223 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:10.220101+0000", 2024-09-13T11:23:10.223 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:10.223 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:10.223 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:10.223 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:23:10.223 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:10.220104+0000", 2024-09-13T11:23:10.223 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.4959999999999999e-06 2024-09-13T11:23:10.223 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:10.223 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:10.223 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:23:10.223 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:10.220152+0000", 2024-09-13T11:23:10.224 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.8303000000000002e-05 2024-09-13T11:23:10.224 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:10.224 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:10.224 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:23:10.224 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:10.220187+0000", 2024-09-13T11:23:10.224 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.5037999999999999e-05 2024-09-13T11:23:10.224 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:10.224 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:10.224 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:23:10.224 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:10.220196+0000", 2024-09-13T11:23:10.224 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.0019999999999995e-06 2024-09-13T11:23:10.224 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:10.224 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:10.225 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:23:10.225 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:10.220272+0000", 2024-09-13T11:23:10.225 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.6056999999999994e-05 2024-09-13T11:23:10.225 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:10.225 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:10.225 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:23:10.225 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:10.220458+0000", 2024-09-13T11:23:10.225 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000186107 2024-09-13T11:23:10.225 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:10.225 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:10.225 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 12", 2024-09-13T11:23:10.225 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:10.220601+0000", 2024-09-13T11:23:10.225 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000142829 2024-09-13T11:23:10.225 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:10.226 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:23:10.226 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:10.226 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:10.226 INFO:teuthology.orchestra.run.smithi162.stdout: ], 2024-09-13T11:23:10.226 INFO:teuthology.orchestra.run.smithi162.stdout: "num_ops": 1 2024-09-13T11:23:10.226 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-13T11:23:10.238 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_blocked_ops 2024-09-13T11:23:10.269 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-13T11:23:10.264+0000 7f6a68775640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26307) UID: 0 2024-09-13T11:23:10.365 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-13T11:23:10.365 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [], 2024-09-13T11:23:10.365 INFO:teuthology.orchestra.run.smithi162.stdout: "complaint_time": 30, 2024-09-13T11:23:10.365 INFO:teuthology.orchestra.run.smithi162.stdout: "num_blocked_ops": 0 2024-09-13T11:23:10.366 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-13T11:23:10.370 INFO:tasks.ceph.osd.11.smithi177.stderr:2024-09-13T11:23:10.366+0000 7f24792c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26580) UID: 0 2024-09-13T11:23:10.384 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_historic_ops 2024-09-13T11:23:10.468 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-13T11:23:10.469 INFO:teuthology.orchestra.run.smithi001.stderr:dumped all 2024-09-13T11:23:10.470 INFO:tasks.ceph.osd.10.smithi162.stderr:2024-09-13T11:23:10.466+0000 7fb59c9f0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26317) UID: 0 2024-09-13T11:23:10.494 INFO:teuthology.orchestra.run.smithi001.stdout:{"pg_ready":true,"pg_map":{"version":75,"stamp":"2024-09-13T11:23:10.128879+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1719337523,"num_objects":26230,"num_object_clones":0,"num_object_copies":52460,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26230,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":26373,"num_write_kb":1681138,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":26341,"ondisk_log_size":26341,"up":50,"acting":50,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":50,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":15,"kb":1509949440,"kb_used":3817160,"kb_used_data":3321544,"kb_used_omap":27,"kb_used_meta":495332,"kb_avail":1506132280,"statfs":{"total":1546188226560,"available":1542279454720,"internally_reserved":0,"allocated":3401261056,"data_stored":3397189695,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28358,"internal_metadata":507220282},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,1,0,1,0,0,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":350617600,"num_objects":5350,"num_object_clones":0,"num_object_copies":10700,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5350,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5350,"num_write_kb":342400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.662360"},"pg_stats":[{"pgid":"3.f","version":"58'1652","reported_seq":1716,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-13T11:23:07.032543+0000","last_change":"2024-09-13T11:22:25.925951+0000","last_active":"2024-09-13T11:23:07.032543+0000","last_peered":"2024-09-13T11:23:07.032543+0000","last_clean":"2024-09-13T11:23:07.032543+0000","last_became_active":"2024-09-13T11:22:25.925169+0000","last_became_peered":"2024-09-13T11:22:25.925169+0000","last_unstale":"2024-09-13T11:23:07.032543+0000","last_undegraded":"2024-09-13T11:23:07.032543+0000","last_fullsized":"2024-09-13T11:23:07.032543+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:22:24.896055+0000","objects_scrubbed":0,"log_size":1652,"log_dups_size":0,"ondisk_log_size":1652,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:23:38.143926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":108265472,"num_objects":1652,"num_object_clones":0,"num_object_copies":3304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1652,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1652,"num_write_kb":105728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,12],"acting":[11,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.e","version":"55'1641","reported_seq":1700,"reported_epoch":55,"state":"remapped+peering","last_fresh":"2024-09-13T11:23:04.462865+0000","last_change":"2024-09-13T11:23:05.477988+0000","last_active":"2024-09-13T11:23:05.477988+0000","last_peered":"2024-09-13T11:23:04.462865+0000","last_clean":"2024-09-13T11:23:04.462865+0000","last_became_active":"2024-09-13T11:22:25.923498+0000","last_became_peered":"2024-09-13T11:22:25.923498+0000","last_unstale":"2024-09-13T11:23:04.462865+0000","last_undegraded":"2024-09-13T11:23:04.462865+0000","last_fullsized":"2024-09-13T11:23:04.462865+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:22:24.896055+0000","objects_scrubbed":0,"log_size":1641,"log_dups_size":0,"ondisk_log_size":1641,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":107544576,"num_objects":1641,"num_object_clones":0,"num_object_copies":3282,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1641,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1641,"num_write_kb":105024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,1,14],"up_primary":0,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.d","version":"56'1560","reported_seq":1621,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-13T11:23:05.471248+0000","last_change":"2024-09-13T11:22:25.927325+0000","last_active":"2024-09-13T11:23:05.471248+0000","last_peered":"2024-09-13T11:23:05.471248+0000","last_clean":"2024-09-13T11:23:05.471248+0000","last_became_active":"2024-09-13T11:22:25.926433+0000","last_became_peered":"2024-09-13T11:22:25.926433+0000","last_unstale":"2024-09-13T11:23:05.471248+0000","last_undegraded":"2024-09-13T11:23:05.471248+0000","last_fullsized":"2024-09-13T11:23:05.471248+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:22:24.896055+0000","objects_scrubbed":0,"log_size":1560,"log_dups_size":0,"ondisk_log_size":1560,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:23:39.281487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":102236160,"num_objects":1560,"num_object_clones":0,"num_object_copies":3120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1560,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1560,"num_write_kb":99840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.c","version":"58'1638","reported_seq":1701,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-13T11:23:07.033361+0000","last_change":"2024-09-13T11:22:25.934619+0000","last_active":"2024-09-13T11:23:07.033361+0000","last_peered":"2024-09-13T11:23:07.033361+0000","last_clean":"2024-09-13T11:23:07.033361+0000","last_became_active":"2024-09-13T11:22:25.933854+0000","last_became_peered":"2024-09-13T11:22:25.933854+0000","last_unstale":"2024-09-13T11:23:07.033361+0000","last_undegraded":"2024-09-13T11:23:07.033361+0000","last_fullsized":"2024-09-13T11:23:07.033361+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:22:24.896055+0000","objects_scrubbed":0,"log_size":1638,"log_dups_size":0,"ondisk_log_size":1638,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:23:42.213952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":107347968,"num_objects":1638,"num_object_clones":0,"num_object_copies":3276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1638,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1638,"num_write_kb":104832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,13],"acting":[11,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.b","version":"58'1699","reported_seq":1762,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-13T11:23:07.028829+0000","last_change":"2024-09-13T11:22:49.175600+0000","last_active":"2024-09-13T11:23:07.028829+0000","last_peered":"2024-09-13T11:23:07.028829+0000","last_clean":"2024-09-13T11:23:07.028829+0000","last_became_active":"2024-09-13T11:22:49.167083+0000","last_became_peered":"2024-09-13T11:22:49.167083+0000","last_unstale":"2024-09-13T11:23:07.028829+0000","last_undegraded":"2024-09-13T11:23:07.028829+0000","last_fullsized":"2024-09-13T11:23:07.028829+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:22:24.896055+0000","objects_scrubbed":0,"log_size":1699,"log_dups_size":0,"ondisk_log_size":1699,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:23:45.371983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":111345664,"num_objects":1699,"num_object_clones":0,"num_object_copies":3398,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1699,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1699,"num_write_kb":108736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,1],"acting":[13,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"3.a","version":"59'1963","reported_seq":2029,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-13T11:23:09.956241+0000","last_change":"2024-09-13T11:22:25.932217+0000","last_active":"2024-09-13T11:23:09.956241+0000","last_peered":"2024-09-13T11:23:09.956241+0000","last_clean":"2024-09-13T11:23:09.956241+0000","last_became_active":"2024-09-13T11:22:25.931493+0000","last_became_peered":"2024-09-13T11:22:25.931493+0000","last_unstale":"2024-09-13T11:23:09.956241+0000","last_undegraded":"2024-09-13T11:23:09.956241+0000","last_fullsized":"2024-09-13T11:23:09.956241+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:22:24.896055+0000","objects_scrubbed":0,"log_size":1963,"log_dups_size":0,"ondisk_log_size":1963,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:23:33.302955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":128647168,"num_objects":1963,"num_object_clones":0,"num_object_copies":3926,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1963,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1963,"num_write_kb":125632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.9","version":"56'1607","reported_seq":1667,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-13T11:23:04.880477+0000","last_change":"2024-09-13T11:22:25.922876+0000","last_active":"2024-09-13T11:23:04.880477+0000","last_peered":"2024-09-13T11:23:04.880477+0000","last_clean":"2024-09-13T11:23:04.880477+0000","last_became_active":"2024-09-13T11:22:25.922498+0000","last_became_peered":"2024-09-13T11:22:25.922498+0000","last_unstale":"2024-09-13T11:23:04.880477+0000","last_undegraded":"2024-09-13T11:23:04.880477+0000","last_fullsized":"2024-09-13T11:23:04.880477+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:22:24.896055+0000","objects_scrubbed":0,"log_size":1607,"log_dups_size":0,"ondisk_log_size":1607,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:23:29.835976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":105316352,"num_objects":1607,"num_object_clones":0,"num_object_copies":3214,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1607,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1607,"num_write_kb":102848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,2],"acting":[12,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[]},{"pgid":"3.8","version":"56'1570","reported_seq":1632,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-13T11:23:05.254522+0000","last_change":"2024-09-13T11:22:47.301262+0000","last_active":"2024-09-13T11:23:05.254522+0000","last_peered":"2024-09-13T11:23:05.254522+0000","last_clean":"2024-09-13T11:23:05.254522+0000","last_became_active":"2024-09-13T11:22:47.296719+0000","last_became_peered":"2024-09-13T11:22:47.296719+0000","last_unstale":"2024-09-13T11:23:05.254522+0000","last_undegraded":"2024-09-13T11:23:05.254522+0000","last_fullsized":"2024-09-13T11:23:05.254522+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:22:24.896055+0000","objects_scrubbed":0,"log_size":1570,"log_dups_size":0,"ondisk_log_size":1570,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:23:54.227881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":102891520,"num_objects":1570,"num_object_clones":0,"num_object_copies":3140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1570,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1570,"num_write_kb":100480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"26'112","reported_seq":210,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-13T11:23:04.499494+0000","last_change":"2024-09-13T11:22:12.131711+0000","last_active":"2024-09-13T11:23:04.499494+0000","last_peered":"2024-09-13T11:23:04.499494+0000","last_clean":"2024-09-13T11:23:04.499494+0000","last_became_active":"2024-09-13T11:22:12.129721+0000","last_became_peered":"2024-09-13T11:22:12.129721+0000","last_unstale":"2024-09-13T11:23:04.499494+0000","last_undegraded":"2024-09-13T11:23:04.499494+0000","last_fullsized":"2024-09-13T11:23:04.499494+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:11.098388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:11.098388+0000","last_clean_scrub_stamp":"2024-09-13T11:22:11.098388+0000","objects_scrubbed":0,"log_size":112,"log_dups_size":0,"ondisk_log_size":112,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:23:40.174992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.3","version":"26'1","reported_seq":79,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-13T11:23:05.481711+0000","last_change":"2024-09-13T11:22:16.547691+0000","last_active":"2024-09-13T11:23:05.481711+0000","last_peered":"2024-09-13T11:23:05.481711+0000","last_clean":"2024-09-13T11:23:05.481711+0000","last_became_active":"2024-09-13T11:22:14.146507+0000","last_became_peered":"2024-09-13T11:22:14.146507+0000","last_unstale":"2024-09-13T11:23:05.481711+0000","last_undegraded":"2024-09-13T11:23:05.481711+0000","last_fullsized":"2024-09-13T11:23:05.481711+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:22:13.113563+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:23:18.785562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00085610999999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,15],"acting":[14,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"3.2","version":"56'1600","reported_seq":1663,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-13T11:23:05.481681+0000","last_change":"2024-09-13T11:22:25.923575+0000","last_active":"2024-09-13T11:23:05.481681+0000","last_peered":"2024-09-13T11:23:05.481681+0000","last_clean":"2024-09-13T11:23:05.481681+0000","last_became_active":"2024-09-13T11:22:25.922665+0000","last_became_peered":"2024-09-13T11:22:25.922665+0000","last_unstale":"2024-09-13T11:23:05.481681+0000","last_undegraded":"2024-09-13T11:23:05.481681+0000","last_fullsized":"2024-09-13T11:23:05.481681+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:22:24.896055+0000","objects_scrubbed":0,"log_size":1600,"log_dups_size":0,"ondisk_log_size":1600,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:23:37.611518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":104857600,"num_objects":1600,"num_object_clones":0,"num_object_copies":3200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1600,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1600,"num_write_kb":102400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,12],"acting":[14,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":81,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-13T11:23:07.506026+0000","last_change":"2024-09-13T11:22:16.267999+0000","last_active":"2024-09-13T11:23:07.506026+0000","last_peered":"2024-09-13T11:23:07.506026+0000","last_clean":"2024-09-13T11:23:07.506026+0000","last_became_active":"2024-09-13T11:22:14.211435+0000","last_became_peered":"2024-09-13T11:22:14.211435+0000","last_unstale":"2024-09-13T11:23:07.506026+0000","last_undegraded":"2024-09-13T11:23:07.506026+0000","last_fullsized":"2024-09-13T11:23:07.506026+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:22:13.113563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:23:24.076640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010010360000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"58'1630","reported_seq":1694,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-13T11:23:07.009154+0000","last_change":"2024-09-13T11:22:25.933242+0000","last_active":"2024-09-13T11:23:07.009154+0000","last_peered":"2024-09-13T11:23:07.009154+0000","last_clean":"2024-09-13T11:23:07.009154+0000","last_became_active":"2024-09-13T11:22:25.932868+0000","last_became_peered":"2024-09-13T11:22:25.932868+0000","last_unstale":"2024-09-13T11:23:07.009154+0000","last_undegraded":"2024-09-13T11:23:07.009154+0000","last_fullsized":"2024-09-13T11:23:07.009154+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:22:24.896055+0000","objects_scrubbed":0,"log_size":1630,"log_dups_size":0,"ondisk_log_size":1630,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:23:38.369067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":106823680,"num_objects":1630,"num_object_clones":0,"num_object_copies":3260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1630,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1630,"num_write_kb":104320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,11],"acting":[13,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":87,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-13T11:23:05.481705+0000","last_change":"2024-09-13T11:22:47.298206+0000","last_active":"2024-09-13T11:23:05.481705+0000","last_peered":"2024-09-13T11:23:05.481705+0000","last_clean":"2024-09-13T11:23:05.481705+0000","last_became_active":"2024-09-13T11:22:47.297319+0000","last_became_peered":"2024-09-13T11:22:47.297319+0000","last_unstale":"2024-09-13T11:23:05.481705+0000","last_undegraded":"2024-09-13T11:23:05.481705+0000","last_fullsized":"2024-09-13T11:23:05.481705+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:22:13.113563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:23:23.483800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00081225399999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,1],"acting":[14,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"3.0","version":"56'1633","reported_seq":1693,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-13T11:23:05.248861+0000","last_change":"2024-09-13T11:22:25.926230+0000","last_active":"2024-09-13T11:23:05.248861+0000","last_peered":"2024-09-13T11:23:05.248861+0000","last_clean":"2024-09-13T11:23:05.248861+0000","last_became_active":"2024-09-13T11:22:25.925791+0000","last_became_peered":"2024-09-13T11:22:25.925791+0000","last_unstale":"2024-09-13T11:23:05.248861+0000","last_undegraded":"2024-09-13T11:23:05.248861+0000","last_fullsized":"2024-09-13T11:23:05.248861+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:22:24.896055+0000","objects_scrubbed":0,"log_size":1633,"log_dups_size":0,"ondisk_log_size":1633,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:23:38.551603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":107020288,"num_objects":1633,"num_object_clones":0,"num_object_copies":3266,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1633,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1633,"num_write_kb":104512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,15],"acting":[12,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[]},{"pgid":"2.2","version":"28'2","reported_seq":83,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-13T11:23:07.506052+0000","last_change":"2024-09-13T11:22:16.269626+0000","last_active":"2024-09-13T11:23:07.506052+0000","last_peered":"2024-09-13T11:23:07.506052+0000","last_clean":"2024-09-13T11:23:07.506052+0000","last_became_active":"2024-09-13T11:22:14.144472+0000","last_became_peered":"2024-09-13T11:22:14.144472+0000","last_unstale":"2024-09-13T11:23:07.506052+0000","last_undegraded":"2024-09-13T11:23:07.506052+0000","last_fullsized":"2024-09-13T11:23:07.506052+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:22:13.113563+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:23:28.879995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0025826730000000002,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9],"acting":[3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.3","version":"56'1596","reported_seq":1658,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-13T11:23:05.991940+0000","last_change":"2024-09-13T11:22:25.930558+0000","last_active":"2024-09-13T11:23:05.991940+0000","last_peered":"2024-09-13T11:23:05.991940+0000","last_clean":"2024-09-13T11:23:05.991940+0000","last_became_active":"2024-09-13T11:22:25.930023+0000","last_became_peered":"2024-09-13T11:22:25.930023+0000","last_unstale":"2024-09-13T11:23:05.991940+0000","last_undegraded":"2024-09-13T11:23:05.991940+0000","last_fullsized":"2024-09-13T11:23:05.991940+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:22:24.896055+0000","objects_scrubbed":0,"log_size":1596,"log_dups_size":0,"ondisk_log_size":1596,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:23:33.097704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":104595456,"num_objects":1596,"num_object_clones":0,"num_object_copies":3192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1596,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1596,"num_write_kb":102144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,7],"acting":[10,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.5","version":"56'1534","reported_seq":1594,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-13T11:23:04.879016+0000","last_change":"2024-09-13T11:22:25.930005+0000","last_active":"2024-09-13T11:23:04.879016+0000","last_peered":"2024-09-13T11:23:04.879016+0000","last_clean":"2024-09-13T11:23:04.879016+0000","last_became_active":"2024-09-13T11:22:25.929242+0000","last_became_peered":"2024-09-13T11:22:25.929242+0000","last_unstale":"2024-09-13T11:23:04.879016+0000","last_undegraded":"2024-09-13T11:23:04.879016+0000","last_fullsized":"2024-09-13T11:23:04.879016+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:22:24.896055+0000","objects_scrubbed":0,"log_size":1534,"log_dups_size":0,"ondisk_log_size":1534,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:23:41.561894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":100532224,"num_objects":1534,"num_object_clones":0,"num_object_copies":3068,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1534,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1534,"num_write_kb":98176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,13],"acting":[15,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":63,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-13T11:23:04.487530+0000","last_change":"2024-09-13T11:22:47.720313+0000","last_active":"2024-09-13T11:23:04.487530+0000","last_peered":"2024-09-13T11:23:04.487530+0000","last_clean":"2024-09-13T11:23:04.487530+0000","last_became_active":"2024-09-13T11:22:47.719636+0000","last_became_peered":"2024-09-13T11:22:47.719636+0000","last_unstale":"2024-09-13T11:23:04.487530+0000","last_undegraded":"2024-09-13T11:23:04.487530+0000","last_fullsized":"2024-09-13T11:23:04.487530+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:22:13.113563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:23:13.812273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00087476000000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,9],"acting":[1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"56'1649","reported_seq":1689,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-13T11:23:04.925306+0000","last_change":"2024-09-13T11:22:47.293720+0000","last_active":"2024-09-13T11:23:04.925306+0000","last_peered":"2024-09-13T11:23:04.925306+0000","last_clean":"2024-09-13T11:23:04.925306+0000","last_became_active":"2024-09-13T11:22:47.292998+0000","last_became_peered":"2024-09-13T11:22:47.292998+0000","last_unstale":"2024-09-13T11:23:04.925306+0000","last_undegraded":"2024-09-13T11:23:04.925306+0000","last_fullsized":"2024-09-13T11:23:04.925306+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:22:24.896055+0000","objects_scrubbed":0,"log_size":1649,"log_dups_size":0,"ondisk_log_size":1649,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:23:42.001016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":108068864,"num_objects":1649,"num_object_clones":0,"num_object_copies":3298,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1649,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1649,"num_write_kb":105536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,12],"acting":[1,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":77,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-13T11:23:05.477734+0000","last_change":"2024-09-13T11:22:16.504060+0000","last_active":"2024-09-13T11:23:05.477734+0000","last_peered":"2024-09-13T11:23:05.477734+0000","last_clean":"2024-09-13T11:23:05.477734+0000","last_became_active":"2024-09-13T11:22:14.581458+0000","last_became_peered":"2024-09-13T11:22:14.581458+0000","last_unstale":"2024-09-13T11:23:05.477734+0000","last_undegraded":"2024-09-13T11:23:05.477734+0000","last_fullsized":"2024-09-13T11:23:05.477734+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:22:13.113563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:23:37.935010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001026047,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5],"acting":[8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.7","version":"58'1594","reported_seq":1658,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-13T11:23:07.031754+0000","last_change":"2024-09-13T11:22:25.924462+0000","last_active":"2024-09-13T11:23:07.031754+0000","last_peered":"2024-09-13T11:23:07.031754+0000","last_clean":"2024-09-13T11:23:07.031754+0000","last_became_active":"2024-09-13T11:22:25.924170+0000","last_became_peered":"2024-09-13T11:22:25.924170+0000","last_unstale":"2024-09-13T11:23:07.031754+0000","last_undegraded":"2024-09-13T11:23:07.031754+0000","last_fullsized":"2024-09-13T11:23:07.031754+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:22:24.896055+0000","objects_scrubbed":0,"log_size":1594,"log_dups_size":0,"ondisk_log_size":1594,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:23:51.945140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":104464384,"num_objects":1594,"num_object_clones":0,"num_object_copies":3188,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1594,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1594,"num_write_kb":102016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,8],"acting":[13,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":78,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-13T11:23:06.495452+0000","last_change":"2024-09-13T11:22:16.260591+0000","last_active":"2024-09-13T11:23:06.495452+0000","last_peered":"2024-09-13T11:23:06.495452+0000","last_clean":"2024-09-13T11:23:06.495452+0000","last_became_active":"2024-09-13T11:22:14.211923+0000","last_became_peered":"2024-09-13T11:22:14.211923+0000","last_unstale":"2024-09-13T11:23:06.495452+0000","last_undegraded":"2024-09-13T11:23:06.495452+0000","last_fullsized":"2024-09-13T11:23:06.495452+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:22:13.113563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:23:20.320636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011689529999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0],"acting":[9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.6","version":"58'1660","reported_seq":1724,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-13T11:23:07.011467+0000","last_change":"2024-09-13T11:22:25.928269+0000","last_active":"2024-09-13T11:23:07.011467+0000","last_peered":"2024-09-13T11:23:07.011467+0000","last_clean":"2024-09-13T11:23:07.011467+0000","last_became_active":"2024-09-13T11:22:25.927961+0000","last_became_peered":"2024-09-13T11:22:25.927961+0000","last_unstale":"2024-09-13T11:23:07.011467+0000","last_undegraded":"2024-09-13T11:23:07.011467+0000","last_fullsized":"2024-09-13T11:23:07.011467+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:22:24.896055+0000","objects_scrubbed":0,"log_size":1660,"log_dups_size":0,"ondisk_log_size":1660,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:23:25.574436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":108789760,"num_objects":1660,"num_object_clones":0,"num_object_copies":3320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1660,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1660,"num_write_kb":106240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,14],"acting":[13,14],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":75,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-13T11:23:04.475101+0000","last_change":"2024-09-13T11:22:16.889130+0000","last_active":"2024-09-13T11:23:04.475101+0000","last_peered":"2024-09-13T11:23:04.475101+0000","last_clean":"2024-09-13T11:23:04.475101+0000","last_became_active":"2024-09-13T11:22:14.192897+0000","last_became_peered":"2024-09-13T11:22:14.192897+0000","last_unstale":"2024-09-13T11:23:04.475101+0000","last_undegraded":"2024-09-13T11:23:04.475101+0000","last_fullsized":"2024-09-13T11:23:04.475101+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:22:13.113563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:23:26.278353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00084655899999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,10],"acting":[15,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":1718747136,"num_objects":26226,"num_object_clones":0,"num_object_copies":52452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26226,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":26226,"num_write_kb":1678464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3426811904,"data_stored":3426811904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":26226,"ondisk_log_size":26226,"up":32,"acting":32,"num_store_stats":13},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":9},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":112,"ondisk_log_size":112,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":13,"up_from":23,"seq":98784247823,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":633192,"kb_used_data":606168,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93738648,"statfs":{"total":96636764160,"available":95988375552,"internally_reserved":0,"allocated":620716032,"data_stored":620463286,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,6,8,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":9,"up_from":23,"seq":98784247822,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27584,"kb_used_data":560,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344256,"statfs":{"total":96636764160,"available":96608518144,"internally_reserved":0,"allocated":573440,"data_stored":310988,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,2,3,8,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":40,"seq":171798691845,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":338868,"kb_used_data":311012,"kb_used_omap":4,"kb_used_meta":27835,"kb_avail":94032972,"statfs":{"total":96636764160,"available":96289763328,"internally_reserved":0,"allocated":318476288,"data_stored":318211858,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4433,"internal_metadata":28503727},"hb_peers":[0,2,3,4,5,6,7,8,9,11,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":21,"seq":90194313231,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":286496,"kb_used_data":246992,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94085344,"statfs":{"total":96636764160,"available":96343392256,"internally_reserved":0,"allocated":252919808,"data_stored":252637337,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40434119},"hb_peers":[0,2,4,5,7,8,9,10,12,13],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":21,"seq":90194313233,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27592,"kb_used_data":568,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344248,"statfs":{"total":96636764160,"available":96608509952,"internally_reserved":0,"allocated":581632,"data_stored":300087,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":22,"seq":94489280525,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28088,"kb_used_data":1064,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343752,"statfs":{"total":96636764160,"available":96608002048,"internally_reserved":0,"allocated":1089536,"data_stored":884385,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,3,4,5,6,7,8,10,11,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":22,"seq":94489280526,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":134600,"kb_used_data":103416,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94237240,"statfs":{"total":96636764160,"available":96498933760,"internally_reserved":0,"allocated":105897984,"data_stored":105621544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,6,7,8,10,11,12,13],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":15,"up_from":22,"seq":94489280526,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":242684,"kb_used_data":203180,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94129156,"statfs":{"total":96636764160,"available":96388255744,"internally_reserved":0,"allocated":208056320,"data_stored":207848045,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40434118},"hb_peers":[0,3,4,5,6,7,10,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":22,"seq":94489280526,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31772,"kb_used_data":588,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340068,"statfs":{"total":96636764160,"available":96604229632,"internally_reserved":0,"allocated":602112,"data_stored":313128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[3,5,7,8,9,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":22,"seq":94489280526,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":132764,"kb_used_data":101580,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94239076,"statfs":{"total":96636764160,"available":96500813824,"internally_reserved":0,"allocated":104017920,"data_stored":103779952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,3,5,7,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":22,"seq":94489280526,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":331992,"kb_used_data":304968,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94039848,"statfs":{"total":96636764160,"available":96296804352,"internally_reserved":0,"allocated":312287232,"data_stored":312057956,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,3,6,8,9,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":22,"seq":94489280526,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":243612,"kb_used_data":204108,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94128228,"statfs":{"total":96636764160,"available":96387305472,"internally_reserved":0,"allocated":209006592,"data_stored":208713885,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[3,5,7,9,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":22,"seq":94489280525,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":133836,"kb_used_data":102652,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94238004,"statfs":{"total":96636764160,"available":96499716096,"internally_reserved":0,"allocated":105115648,"data_stored":104893088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,5,7,8,9,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":11,"up_from":22,"seq":94489280527,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":336288,"kb_used_data":305104,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94035552,"statfs":{"total":96636764160,"available":96292405248,"internally_reserved":0,"allocated":312426496,"data_stored":312184808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[2,3,4,5,6,10,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":12,"up_from":22,"seq":94489280526,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":550076,"kb_used_data":518892,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93821764,"statfs":{"total":96636764160,"available":96073486336,"internally_reserved":0,"allocated":531345408,"data_stored":531076792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[2,3,4,5,6,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":14,"up_from":22,"seq":94489280526,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":337716,"kb_used_data":310692,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94034124,"statfs":{"total":96636764160,"available":96290942976,"internally_reserved":0,"allocated":318148608,"data_stored":317892556,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,5,6,7,8,9,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":318177280,"data_stored":318177280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":105316352,"data_stored":105316352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":253820928,"data_stored":253820928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":102891520,"data_stored":102891520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":311754752,"data_stored":311754752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":208404480,"data_stored":208404480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":104595456,"data_stored":104595456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":322437120,"data_stored":322437120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":530776064,"data_stored":530776064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":643497984,"data_stored":643497984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":317587456,"data_stored":317587456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":207552512,"data_stored":207552512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T11:23:10.496 INFO:tasks.thrashosds.thrasher:Setting 2.2 to [2, 8] 2024-09-13T11:23:10.496 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap', '2.2', '2', '8'] 2024-09-13T11:23:10.496 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pg-upmap 2.2 2 8 2024-09-13T11:23:10.521 INFO:teuthology.orchestra.run.smithi001.stderr:nodeep-scrub is unset 2024-09-13T11:23:10.524 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-13T11:23:10.524 INFO:teuthology.orchestra.run.smithi162.stdout: "size": 20, 2024-09-13T11:23:10.524 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 600, 2024-09-13T11:23:10.524 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [ 2024-09-13T11:23:10.524 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:10.524 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.4665.0:1203 3.6 e43/30 3:6ecdd812:::benchmark_data_smithi177_31116_object1202:head v 43'82, mlcod=43'78)", 2024-09-13T11:23:10.524 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:22:49.775920+0000", 2024-09-13T11:23:10.524 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 20.745791730000001, 2024-09-13T11:23:10.524 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.047933733999999999, 2024-09-13T11:23:10.524 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:23:10.524 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:23:10.524 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:10.525 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:23:10.525 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:10.525 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:23:10.525 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:49.775920+0000", 2024-09-13T11:23:10.525 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:10.525 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:10.525 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:10.525 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:23:10.525 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:49.775920+0000", 2024-09-13T11:23:10.525 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:10.525 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:10.525 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:10.525 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:23:10.525 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:49.775922+0000", 2024-09-13T11:23:10.526 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.1739999999999999e-06 2024-09-13T11:23:10.526 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:10.526 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:10.526 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:23:10.526 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:49.775997+0000", 2024-09-13T11:23:10.526 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.5199999999999998e-05 2024-09-13T11:23:10.526 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:10.526 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:10.526 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:23:10.528 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:49.816064+0000", 2024-09-13T11:23:10.528 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.040066709999999998 2024-09-13T11:23:10.528 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:10.528 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:10.528 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:23:10.528 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:49.816076+0000", 2024-09-13T11:23:10.528 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2449000000000001e-05 2024-09-13T11:23:10.528 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:10.528 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:10.528 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:23:10.528 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:49.816188+0000", 2024-09-13T11:23:10.528 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000111303 2024-09-13T11:23:10.529 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:10.529 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:10.529 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:23:10.529 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:49.816234+0000", 2024-09-13T11:23:10.529 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.6202000000000003e-05 2024-09-13T11:23:10.529 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:10.529 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:10.529 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:23:10.529 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:49.823738+0000", 2024-09-13T11:23:10.529 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0075036369999999996 2024-09-13T11:23:10.529 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:10.529 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:10.529 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:23:10.529 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:49.823854+0000", 2024-09-13T11:23:10.530 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00011605899999999999 2024-09-13T11:23:10.530 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:10.530 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:23:10.530 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:10.530 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:10.530 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:10.530 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:8136 3.2 3:4dc5c6b7:::benchmark_data_smithi177_31116_object8135:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T11:23:10.530 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:22:53.204680+0000", 2024-09-13T11:23:10.530 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 17.317032057999999, 2024-09-13T11:23:10.530 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.047720889000000002, 2024-09-13T11:23:10.530 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:23:10.530 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:23:10.530 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:10.531 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:23:10.531 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:23:10.531 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:10.531 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 8136 2024-09-13T11:23:10.531 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:10.531 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:23:10.531 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:10.531 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:23:10.531 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:53.204680+0000", 2024-09-13T11:23:10.531 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:10.531 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:10.531 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:10.531 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:23:10.532 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:53.204680+0000", 2024-09-13T11:23:10.532 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:10.532 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:10.532 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:10.532 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:23:10.532 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:53.204681+0000", 2024-09-13T11:23:10.532 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.085e-06 2024-09-13T11:23:10.532 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:10.532 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:10.532 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:23:10.532 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:53.204770+0000", 2024-09-13T11:23:10.532 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.9161999999999998e-05 2024-09-13T11:23:10.532 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:10.532 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:10.533 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:23:10.533 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:53.204780+0000", 2024-09-13T11:23:10.533 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0319999999999999e-05 2024-09-13T11:23:10.533 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:10.533 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:10.533 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:23:10.533 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:53.204786+0000", 2024-09-13T11:23:10.533 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.0959999999999997e-06 2024-09-13T11:23:10.533 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:10.534 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:10.534 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:23:10.534 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:53.206012+0000", 2024-09-13T11:23:10.534 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0012255600000000001 2024-09-13T11:23:10.534 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:10.534 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:10.534 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:23:10.534 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:53.206225+0000", 2024-09-13T11:23:10.534 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00021282300000000001 2024-09-13T11:23:10.534 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:10.534 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:10.534 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 12", 2024-09-13T11:23:10.534 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:53.206463+0000", 2024-09-13T11:23:10.534 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000238321 2024-09-13T11:23:10.535 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:10.535 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:10.535 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:23:10.535 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:53.207831+0000", 2024-09-13T11:23:10.535 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0013682709999999999 2024-09-13T11:23:10.535 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:10.535 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:10.535 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:10.535 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:53.252275+0000", 2024-09-13T11:23:10.535 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.044443573 2024-09-13T11:23:10.535 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:10.535 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:10.535 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:23:10.535 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:53.252342+0000", 2024-09-13T11:23:10.536 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.7386999999999997e-05 2024-09-13T11:23:10.536 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:10.536 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:10.536 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:23:10.536 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:53.252400+0000", 2024-09-13T11:23:10.536 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.8291000000000001e-05 2024-09-13T11:23:10.536 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:10.536 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:23:10.536 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:10.536 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:10.536 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:10.536 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.4665.0:9905 3.6 e47/30 3:639d4f69:::benchmark_data_smithi177_31116_object9904:head v 47'621, mlcod=47'618)", 2024-09-13T11:23:10.536 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:22:54.301066+0000", 2024-09-13T11:23:10.537 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 16.220645329, 2024-09-13T11:23:10.537 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.11187942300000001, 2024-09-13T11:23:10.537 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:23:10.537 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:23:10.537 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:10.537 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:23:10.537 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:10.537 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:23:10.537 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:54.301066+0000", 2024-09-13T11:23:10.537 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:10.537 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:10.537 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:10.537 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:23:10.537 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:54.301066+0000", 2024-09-13T11:23:10.537 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:10.538 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:10.538 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:10.538 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:23:10.538 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:54.301077+0000", 2024-09-13T11:23:10.538 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1154e-05 2024-09-13T11:23:10.538 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:10.538 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:10.538 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:23:10.538 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:54.301136+0000", 2024-09-13T11:23:10.538 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.8480999999999998e-05 2024-09-13T11:23:10.538 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:10.538 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:10.538 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:23:10.538 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:54.410822+0000", 2024-09-13T11:23:10.539 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.109685649 2024-09-13T11:23:10.539 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:10.539 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:10.539 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:23:10.539 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:54.410829+0000", 2024-09-13T11:23:10.539 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.7470000000000005e-06 2024-09-13T11:23:10.539 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:10.539 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:10.539 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:23:10.539 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:54.410995+0000", 2024-09-13T11:23:10.539 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00016614200000000001 2024-09-13T11:23:10.539 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:10.539 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:10.540 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:23:10.540 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:54.411056+0000", 2024-09-13T11:23:10.540 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.0174999999999999e-05 2024-09-13T11:23:10.540 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:10.540 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:10.540 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:23:10.540 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:54.412869+0000", 2024-09-13T11:23:10.540 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001813423 2024-09-13T11:23:10.541 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:10.541 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:10.541 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:23:10.541 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:54.412946+0000", 2024-09-13T11:23:10.541 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.6651999999999997e-05 2024-09-13T11:23:10.541 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:10.541 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:23:10.541 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:10.541 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:10.541 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:10.541 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.4665.0:10283 3.6 e48/30 3:6e9a3211:::benchmark_data_smithi177_31116_object10282:head v 48'639, mlcod=48'634)", 2024-09-13T11:23:10.541 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:22:54.551316+0000", 2024-09-13T11:23:10.542 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 15.970395205999999, 2024-09-13T11:23:10.542 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.049481734999999999, 2024-09-13T11:23:10.542 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:23:10.542 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:23:10.542 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:10.542 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:23:10.542 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:10.542 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:23:10.542 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:54.551316+0000", 2024-09-13T11:23:10.542 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:10.542 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:10.542 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:10.542 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:23:10.543 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:54.551316+0000", 2024-09-13T11:23:10.543 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:10.543 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:10.543 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:10.543 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:23:10.543 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:54.551320+0000", 2024-09-13T11:23:10.543 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.4379999999999999e-06 2024-09-13T11:23:10.543 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:10.543 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:10.543 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:23:10.543 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:54.551388+0000", 2024-09-13T11:23:10.543 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.8040000000000006e-05 2024-09-13T11:23:10.543 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:10.543 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:10.544 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:23:10.544 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:54.599005+0000", 2024-09-13T11:23:10.544 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.047616671999999999 2024-09-13T11:23:10.544 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:10.544 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:10.544 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:23:10.544 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:54.599013+0000", 2024-09-13T11:23:10.544 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.2400000000000007e-06 2024-09-13T11:23:10.544 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:10.544 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:10.544 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:23:10.544 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:54.599098+0000", 2024-09-13T11:23:10.545 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.4917e-05 2024-09-13T11:23:10.545 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:10.545 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:10.545 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:23:10.545 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:54.599125+0000", 2024-09-13T11:23:10.545 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.7004000000000001e-05 2024-09-13T11:23:10.545 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:10.545 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:10.545 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:23:10.545 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:54.600741+0000", 2024-09-13T11:23:10.545 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0016163359999999999 2024-09-13T11:23:10.545 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:10.545 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:10.546 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:23:10.546 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:54.600798+0000", 2024-09-13T11:23:10.546 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.7088e-05 2024-09-13T11:23:10.546 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:10.546 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:23:10.546 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:10.546 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:10.546 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:10.546 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:10602 3.2 3:4e5af7b0:::benchmark_data_smithi177_31116_object10601:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e48)", 2024-09-13T11:23:10.546 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:22:54.727904+0000", 2024-09-13T11:23:10.546 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 15.793807599999999, 2024-09-13T11:23:10.546 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.05044945, 2024-09-13T11:23:10.546 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:23:10.546 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:23:10.547 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:10.547 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:23:10.547 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:23:10.547 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:10.547 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 10602 2024-09-13T11:23:10.547 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:10.547 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:23:10.547 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:10.547 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:23:10.547 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:54.727904+0000", 2024-09-13T11:23:10.547 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:10.547 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:10.548 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:10.548 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:23:10.548 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:54.727904+0000", 2024-09-13T11:23:10.548 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:10.548 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:10.548 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:10.548 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:23:10.548 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:54.727907+0000", 2024-09-13T11:23:10.548 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.7240000000000001e-06 2024-09-13T11:23:10.548 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:10.548 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:10.548 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:23:10.548 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:54.727953+0000", 2024-09-13T11:23:10.548 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.6520999999999997e-05 2024-09-13T11:23:10.549 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:10.549 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:10.549 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:23:10.549 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:54.727975+0000", 2024-09-13T11:23:10.549 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.1675000000000002e-05 2024-09-13T11:23:10.549 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:10.549 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:10.549 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:23:10.549 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:54.728008+0000", 2024-09-13T11:23:10.549 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.3188999999999997e-05 2024-09-13T11:23:10.549 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:10.549 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:10.549 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:23:10.549 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:54.728097+0000", 2024-09-13T11:23:10.550 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.8707000000000003e-05 2024-09-13T11:23:10.550 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:10.550 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:10.550 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:23:10.550 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:54.728285+0000", 2024-09-13T11:23:10.550 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000187894 2024-09-13T11:23:10.550 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:10.550 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:10.550 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 12", 2024-09-13T11:23:10.550 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:54.728464+0000", 2024-09-13T11:23:10.550 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00017899999999999999 2024-09-13T11:23:10.550 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:10.550 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:10.550 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:23:10.550 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:54.730056+0000", 2024-09-13T11:23:10.551 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0015921760000000001 2024-09-13T11:23:10.551 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:10.551 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:10.551 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:10.551 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:54.778206+0000", 2024-09-13T11:23:10.551 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.048150432999999999 2024-09-13T11:23:10.551 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:10.551 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:10.551 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:23:10.551 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:54.778282+0000", 2024-09-13T11:23:10.551 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.5644999999999998e-05 2024-09-13T11:23:10.551 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:10.551 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:10.552 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:23:10.552 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:54.778353+0000", 2024-09-13T11:23:10.552 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.1486e-05 2024-09-13T11:23:10.552 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:10.552 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:23:10.552 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:10.552 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:10.552 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:10.552 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:10614 3.2 3:467c4800:::benchmark_data_smithi177_31116_object10613:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e48)", 2024-09-13T11:23:10.552 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:22:54.737035+0000", 2024-09-13T11:23:10.552 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 15.784676884, 2024-09-13T11:23:10.552 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.074763275000000004, 2024-09-13T11:23:10.552 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:23:10.553 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:23:10.553 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:10.553 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:23:10.553 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:23:10.553 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:10.553 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 10614 2024-09-13T11:23:10.553 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:10.553 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:23:10.553 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:10.553 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:23:10.553 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:54.737035+0000", 2024-09-13T11:23:10.553 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:10.553 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:10.553 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:10.554 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:23:10.554 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:54.737035+0000", 2024-09-13T11:23:10.554 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:10.554 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:10.554 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:10.554 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:23:10.554 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:54.737038+0000", 2024-09-13T11:23:10.554 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.4029999999999999e-06 2024-09-13T11:23:10.554 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:10.554 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:10.554 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:23:10.554 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:54.737110+0000", 2024-09-13T11:23:10.555 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.1538000000000005e-05 2024-09-13T11:23:10.555 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:10.555 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:10.555 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:23:10.555 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:54.737137+0000", 2024-09-13T11:23:10.555 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.7172999999999999e-05 2024-09-13T11:23:10.555 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:10.555 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:10.555 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:23:10.555 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:54.737154+0000", 2024-09-13T11:23:10.555 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.6708e-05 2024-09-13T11:23:10.555 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:10.555 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:10.556 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:23:10.556 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:54.737240+0000", 2024-09-13T11:23:10.556 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.6516999999999999e-05 2024-09-13T11:23:10.556 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:10.556 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:10.556 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:23:10.556 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:54.737418+0000", 2024-09-13T11:23:10.556 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00017780199999999999 2024-09-13T11:23:10.556 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:10.556 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:10.556 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 12", 2024-09-13T11:23:10.556 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:54.737598+0000", 2024-09-13T11:23:10.556 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00017990200000000001 2024-09-13T11:23:10.557 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:10.557 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:10.557 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:23:10.557 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:54.739568+0000", 2024-09-13T11:23:10.557 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001970547 2024-09-13T11:23:10.557 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:10.557 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:10.557 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:10.557 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:54.811623+0000", 2024-09-13T11:23:10.557 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.072054887999999997 2024-09-13T11:23:10.557 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:10.557 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:10.557 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:23:10.558 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:54.811727+0000", 2024-09-13T11:23:10.558 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000104063 2024-09-13T11:23:10.558 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:10.558 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:10.558 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:23:10.558 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:54.811798+0000", 2024-09-13T11:23:10.558 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.0734000000000004e-05 2024-09-13T11:23:10.558 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:10.558 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:23:10.558 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:10.558 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:10.558 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:10.558 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:10663 3.2 3:45c0af2a:::benchmark_data_smithi177_31116_object10662:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e48)", 2024-09-13T11:23:10.558 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:22:54.775964+0000", 2024-09-13T11:23:10.559 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 15.745748042000001, 2024-09-13T11:23:10.559 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.063256978000000005, 2024-09-13T11:23:10.559 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:23:10.559 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:23:10.559 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:10.559 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:23:10.559 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:23:10.559 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:10.559 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 10663 2024-09-13T11:23:10.559 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:10.559 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:23:10.559 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:10.559 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:23:10.559 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:54.775964+0000", 2024-09-13T11:23:10.560 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:10.560 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:10.560 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:10.560 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:23:10.560 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:54.775964+0000", 2024-09-13T11:23:10.560 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:10.560 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:10.560 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:10.560 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:23:10.560 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:54.775967+0000", 2024-09-13T11:23:10.560 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.574e-06 2024-09-13T11:23:10.560 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:10.560 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:10.560 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:23:10.561 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:54.776018+0000", 2024-09-13T11:23:10.561 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.0970000000000002e-05 2024-09-13T11:23:10.561 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:10.561 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:10.561 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:23:10.561 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:54.776039+0000", 2024-09-13T11:23:10.561 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.0928999999999999e-05 2024-09-13T11:23:10.561 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:10.561 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:10.561 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:23:10.561 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:54.776049+0000", 2024-09-13T11:23:10.561 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0108e-05 2024-09-13T11:23:10.561 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:10.561 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:10.562 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:23:10.562 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:54.776135+0000", 2024-09-13T11:23:10.562 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.6211000000000005e-05 2024-09-13T11:23:10.562 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:10.562 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:10.562 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:23:10.562 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:54.776306+0000", 2024-09-13T11:23:10.562 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00017098100000000001 2024-09-13T11:23:10.562 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:10.562 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:10.562 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 12", 2024-09-13T11:23:10.562 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:54.776526+0000", 2024-09-13T11:23:10.562 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00021961699999999999 2024-09-13T11:23:10.562 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:10.562 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:10.563 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:23:10.563 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:54.778807+0000", 2024-09-13T11:23:10.563 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.002281464 2024-09-13T11:23:10.563 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:10.563 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:10.563 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:10.563 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:54.838969+0000", 2024-09-13T11:23:10.563 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.060161961999999999 2024-09-13T11:23:10.563 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:10.563 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:10.563 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:23:10.563 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:54.839105+0000", 2024-09-13T11:23:10.563 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00013577899999999999 2024-09-13T11:23:10.564 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:10.564 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:10.564 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:23:10.564 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:54.839221+0000", 2024-09-13T11:23:10.564 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000115383 2024-09-13T11:23:10.564 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:10.564 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:23:10.564 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:10.564 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:10.564 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:10.564 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:10699 3.2 3:474fc186:::benchmark_data_smithi177_31116_object10698:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e48)", 2024-09-13T11:23:10.564 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:22:54.809344+0000", 2024-09-13T11:23:10.564 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 15.712367349999999, 2024-09-13T11:23:10.564 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.077043713, 2024-09-13T11:23:10.564 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:23:10.565 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:23:10.565 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:10.565 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:23:10.565 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:23:10.565 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:10.565 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 10699 2024-09-13T11:23:10.565 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:10.565 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:23:10.565 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:10.565 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:23:10.565 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:54.809344+0000", 2024-09-13T11:23:10.565 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:10.565 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:10.566 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:10.566 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:23:10.566 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:54.809344+0000", 2024-09-13T11:23:10.566 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:10.566 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:10.566 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:10.566 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:23:10.566 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:54.809350+0000", 2024-09-13T11:23:10.566 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.3669999999999996e-06 2024-09-13T11:23:10.566 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:10.566 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:10.566 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:23:10.566 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:54.809412+0000", 2024-09-13T11:23:10.566 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.2157999999999997e-05 2024-09-13T11:23:10.567 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:10.567 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:10.567 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:23:10.567 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:54.809440+0000", 2024-09-13T11:23:10.567 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.7798999999999999e-05 2024-09-13T11:23:10.567 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:10.567 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:10.567 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:23:10.567 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:54.809454+0000", 2024-09-13T11:23:10.567 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4732999999999999e-05 2024-09-13T11:23:10.567 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:10.567 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:10.567 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:23:10.568 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:54.809564+0000", 2024-09-13T11:23:10.568 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000109413 2024-09-13T11:23:10.568 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:10.568 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:10.568 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:23:10.568 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:54.809800+0000", 2024-09-13T11:23:10.568 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000236488 2024-09-13T11:23:10.568 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:10.568 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:10.568 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 12", 2024-09-13T11:23:10.568 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:54.809945+0000", 2024-09-13T11:23:10.568 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00014512300000000001 2024-09-13T11:23:10.568 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:10.569 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:10.569 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:23:10.569 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:54.811417+0000", 2024-09-13T11:23:10.569 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001471305 2024-09-13T11:23:10.569 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:10.569 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:10.569 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:10.569 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:54.886237+0000", 2024-09-13T11:23:10.569 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.074819906000000005 2024-09-13T11:23:10.569 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:10.569 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:10.569 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:23:10.569 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:54.886323+0000", 2024-09-13T11:23:10.570 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.6286999999999996e-05 2024-09-13T11:23:10.570 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:10.570 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:10.570 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:23:10.570 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:54.886388+0000", 2024-09-13T11:23:10.570 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.5134000000000004e-05 2024-09-13T11:23:10.570 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:10.570 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:23:10.570 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:10.570 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:10.570 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:10.570 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:10728 3.2 3:43fb4edc:::benchmark_data_smithi177_31116_object10727:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e48)", 2024-09-13T11:23:10.570 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:22:54.836511+0000", 2024-09-13T11:23:10.571 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 15.68520011, 2024-09-13T11:23:10.571 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.106223183, 2024-09-13T11:23:10.571 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:23:10.571 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:23:10.571 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:10.571 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:23:10.571 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:23:10.571 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:10.571 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 10728 2024-09-13T11:23:10.571 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:10.571 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:23:10.571 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:10.571 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:23:10.571 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:54.836511+0000", 2024-09-13T11:23:10.572 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:10.572 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:10.572 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:10.572 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:23:10.572 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:54.836511+0000", 2024-09-13T11:23:10.572 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:10.572 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:10.572 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:10.572 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:23:10.572 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:54.836514+0000", 2024-09-13T11:23:10.572 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.948e-06 2024-09-13T11:23:10.572 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:10.572 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:10.573 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:23:10.573 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:54.836564+0000", 2024-09-13T11:23:10.573 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.9129999999999999e-05 2024-09-13T11:23:10.573 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:10.573 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:10.573 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:23:10.573 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:54.836584+0000", 2024-09-13T11:23:10.573 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.0684000000000001e-05 2024-09-13T11:23:10.573 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:10.573 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:10.573 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:23:10.573 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:54.836599+0000", 2024-09-13T11:23:10.573 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4409e-05 2024-09-13T11:23:10.574 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:10.574 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:10.574 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:23:10.574 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:54.836738+0000", 2024-09-13T11:23:10.574 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00013955799999999999 2024-09-13T11:23:10.574 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:10.574 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:10.574 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:23:10.574 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:54.836985+0000", 2024-09-13T11:23:10.574 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000247159 2024-09-13T11:23:10.574 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:10.574 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:10.574 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 12", 2024-09-13T11:23:10.575 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:54.837238+0000", 2024-09-13T11:23:10.575 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00025263299999999999 2024-09-13T11:23:10.575 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:10.575 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:10.575 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:23:10.575 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:54.839339+0000", 2024-09-13T11:23:10.575 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0021009380000000001 2024-09-13T11:23:10.575 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:10.575 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:10.575 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:10.575 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:54.942575+0000", 2024-09-13T11:23:10.575 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.103236046 2024-09-13T11:23:10.575 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:10.576 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:10.576 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:23:10.576 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:54.942665+0000", 2024-09-13T11:23:10.576 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.9747000000000007e-05 2024-09-13T11:23:10.576 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:10.576 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:10.576 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:23:10.576 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:54.942735+0000", 2024-09-13T11:23:10.576 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.9931000000000005e-05 2024-09-13T11:23:10.576 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:10.576 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:23:10.576 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:10.576 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:10.577 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:10.577 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop_reply(client.4665.0:10699 3.2 e48/30 ondisk, result = 0)", 2024-09-13T11:23:10.577 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:22:54.838735+0000", 2024-09-13T11:23:10.577 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 15.682976105, 2024-09-13T11:23:10.577 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.047663175000000002, 2024-09-13T11:23:10.577 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:23:10.577 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:23:10.577 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:23:10.577 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:23:10.577 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:10.577 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:23:10.577 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:54.838735+0000", 2024-09-13T11:23:10.577 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:10.578 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:10.578 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:10.578 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:23:10.578 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:54.838735+0000", 2024-09-13T11:23:10.578 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:10.578 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:10.578 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:10.578 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:23:10.578 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:54.838739+0000", 2024-09-13T11:23:10.578 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.366e-06 2024-09-13T11:23:10.578 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:10.578 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:10.578 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:23:10.578 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:54.838749+0000", 2024-09-13T11:23:10.579 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.8099999999999992e-06 2024-09-13T11:23:10.579 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:10.579 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:10.579 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:23:10.579 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:54.886089+0000", 2024-09-13T11:23:10.579 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.047339945000000001 2024-09-13T11:23:10.579 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:10.579 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:10.579 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:23:10.579 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:54.886102+0000", 2024-09-13T11:23:10.579 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3063e-05 2024-09-13T11:23:10.579 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:10.579 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:10.579 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:23:10.580 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:54.886212+0000", 2024-09-13T11:23:10.580 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00011074 2024-09-13T11:23:10.580 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:10.580 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:10.580 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:23:10.580 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:54.886228+0000", 2024-09-13T11:23:10.580 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5252999999999999e-05 2024-09-13T11:23:10.580 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:10.580 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:10.581 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:23:10.581 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:54.886399+0000", 2024-09-13T11:23:10.581 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00017099800000000001 2024-09-13T11:23:10.581 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:10.581 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:23:10.581 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:10.581 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:10.581 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:10.581 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:10755 3.2 3:4d45beba:::benchmark_data_smithi177_31116_object10754:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e48)", 2024-09-13T11:23:10.581 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:22:54.883336+0000", 2024-09-13T11:23:10.581 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 15.638376029, 2024-09-13T11:23:10.581 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.072969299000000001, 2024-09-13T11:23:10.581 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:23:10.582 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:23:10.582 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:10.582 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:23:10.582 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:23:10.582 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:10.582 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 10755 2024-09-13T11:23:10.582 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:10.582 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:23:10.582 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:10.582 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:23:10.582 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:54.883336+0000", 2024-09-13T11:23:10.582 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:10.582 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:10.583 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:10.583 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:23:10.583 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:54.883336+0000", 2024-09-13T11:23:10.583 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:10.583 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:10.583 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:10.583 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:23:10.583 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:54.883340+0000", 2024-09-13T11:23:10.583 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.0659999999999997e-06 2024-09-13T11:23:10.583 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:10.583 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:10.583 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:23:10.583 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:54.883396+0000", 2024-09-13T11:23:10.584 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.6765000000000003e-05 2024-09-13T11:23:10.584 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:10.584 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:10.584 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:23:10.584 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:54.883428+0000", 2024-09-13T11:23:10.584 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.1275999999999997e-05 2024-09-13T11:23:10.584 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:10.584 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:10.584 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:23:10.584 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:54.883449+0000", 2024-09-13T11:23:10.584 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.1362999999999999e-05 2024-09-13T11:23:10.584 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:10.584 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:10.584 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:23:10.585 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:54.883578+0000", 2024-09-13T11:23:10.585 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000128603 2024-09-13T11:23:10.585 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:10.585 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:10.585 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:23:10.585 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:54.883794+0000", 2024-09-13T11:23:10.585 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00021602899999999999 2024-09-13T11:23:10.585 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:10.585 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:10.585 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 12", 2024-09-13T11:23:10.585 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:54.884014+0000", 2024-09-13T11:23:10.585 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00022015700000000001 2024-09-13T11:23:10.585 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:10.586 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:10.586 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:23:10.586 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:54.885788+0000", 2024-09-13T11:23:10.586 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0017737460000000001 2024-09-13T11:23:10.586 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:10.586 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:10.586 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:10.586 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:54.956152+0000", 2024-09-13T11:23:10.586 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.070364950999999995 2024-09-13T11:23:10.586 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:10.586 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:10.586 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:23:10.586 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:54.956237+0000", 2024-09-13T11:23:10.586 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.4275000000000002e-05 2024-09-13T11:23:10.587 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:10.587 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:10.587 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:23:10.587 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:54.956305+0000", 2024-09-13T11:23:10.587 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.8067999999999999e-05 2024-09-13T11:23:10.587 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:10.587 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:23:10.587 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:10.587 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:10.587 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:10.587 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop_reply(client.4665.0:10728 3.2 e48/30 ondisk, result = 0)", 2024-09-13T11:23:10.587 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:22:54.886042+0000", 2024-09-13T11:23:10.588 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 15.635669776, 2024-09-13T11:23:10.588 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.056703652, 2024-09-13T11:23:10.588 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:23:10.588 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:23:10.588 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:23:10.588 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:23:10.588 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:10.588 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:23:10.588 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:54.886042+0000", 2024-09-13T11:23:10.588 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:10.588 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:10.588 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:10.588 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:23:10.588 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:54.886042+0000", 2024-09-13T11:23:10.588 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:10.589 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:10.589 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:10.589 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:23:10.589 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:54.886045+0000", 2024-09-13T11:23:10.589 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.1439999999999999e-06 2024-09-13T11:23:10.589 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:10.589 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:10.589 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:23:10.589 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:54.886056+0000", 2024-09-13T11:23:10.589 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1059e-05 2024-09-13T11:23:10.589 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:10.589 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:10.589 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:23:10.589 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:54.942439+0000", 2024-09-13T11:23:10.590 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.056382699000000001 2024-09-13T11:23:10.590 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:10.590 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:10.590 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:23:10.590 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:54.942452+0000", 2024-09-13T11:23:10.590 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3736e-05 2024-09-13T11:23:10.590 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:10.590 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:10.590 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:23:10.590 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:54.942554+0000", 2024-09-13T11:23:10.590 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000101283 2024-09-13T11:23:10.590 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:10.590 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:10.590 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:23:10.591 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:54.942566+0000", 2024-09-13T11:23:10.591 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2537e-05 2024-09-13T11:23:10.591 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:10.591 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:10.591 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:23:10.591 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:54.942745+0000", 2024-09-13T11:23:10.591 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00017919400000000001 2024-09-13T11:23:10.591 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:10.591 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:23:10.591 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:10.591 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:10.591 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:10.591 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:11121 3.2 3:4b00b743:::benchmark_data_smithi177_31116_object11120:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e48)", 2024-09-13T11:23:10.591 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:22:55.229228+0000", 2024-09-13T11:23:10.591 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 15.292483291, 2024-09-13T11:23:10.591 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.071321041000000002, 2024-09-13T11:23:10.592 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:23:10.592 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:23:10.592 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:10.592 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:23:10.592 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:23:10.592 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:10.592 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 11121 2024-09-13T11:23:10.592 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:10.592 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:23:10.592 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:10.592 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:23:10.592 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:55.229228+0000", 2024-09-13T11:23:10.592 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:10.592 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:10.593 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:10.593 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:23:10.593 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:55.229228+0000", 2024-09-13T11:23:10.593 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:10.593 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:10.593 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:10.593 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:23:10.593 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:55.229232+0000", 2024-09-13T11:23:10.593 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.3299999999999999e-06 2024-09-13T11:23:10.593 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:10.593 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:10.593 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:23:10.593 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:55.229301+0000", 2024-09-13T11:23:10.593 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.9855000000000001e-05 2024-09-13T11:23:10.594 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:10.594 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:10.594 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:23:10.594 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:55.229329+0000", 2024-09-13T11:23:10.594 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.7599000000000001e-05 2024-09-13T11:23:10.594 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:10.594 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:10.594 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:23:10.594 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:55.229343+0000", 2024-09-13T11:23:10.594 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4025999999999999e-05 2024-09-13T11:23:10.594 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:10.594 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:10.594 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:23:10.594 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:55.229480+0000", 2024-09-13T11:23:10.595 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00013678299999999999 2024-09-13T11:23:10.595 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:10.595 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:10.595 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:23:10.595 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:55.229848+0000", 2024-09-13T11:23:10.595 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00036796600000000001 2024-09-13T11:23:10.595 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:10.595 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:10.595 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 12", 2024-09-13T11:23:10.595 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:55.230377+0000", 2024-09-13T11:23:10.595 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00052884799999999995 2024-09-13T11:23:10.595 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:10.595 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:10.595 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:23:10.596 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:55.232523+0000", 2024-09-13T11:23:10.596 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0021465579999999998 2024-09-13T11:23:10.596 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:10.596 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:10.596 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:10.596 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:55.300309+0000", 2024-09-13T11:23:10.596 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.067785887000000003 2024-09-13T11:23:10.596 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:10.596 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:10.596 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:23:10.596 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:55.300477+0000", 2024-09-13T11:23:10.596 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000167808 2024-09-13T11:23:10.596 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:10.597 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:10.597 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:23:10.597 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:55.300549+0000", 2024-09-13T11:23:10.597 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.2380999999999997e-05 2024-09-13T11:23:10.597 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:10.597 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:23:10.597 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:10.597 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:10.597 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:10.597 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.4665.0:11275 3.6 e48/30 3:6a6723af:::benchmark_data_smithi177_31116_object11274:head v 48'710, mlcod=48'706)", 2024-09-13T11:23:10.597 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:22:55.358219+0000", 2024-09-13T11:23:10.597 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 15.163492231999999, 2024-09-13T11:23:10.597 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.050991611999999999, 2024-09-13T11:23:10.598 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:23:10.598 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:23:10.598 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:10.598 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:23:10.598 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:10.598 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:23:10.598 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:55.358219+0000", 2024-09-13T11:23:10.598 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:10.598 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:10.598 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:10.598 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:23:10.598 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:55.358219+0000", 2024-09-13T11:23:10.598 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:10.598 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:10.599 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:10.599 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:23:10.599 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:55.358222+0000", 2024-09-13T11:23:10.599 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.79e-06 2024-09-13T11:23:10.599 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:10.599 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:10.599 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:23:10.599 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:55.358288+0000", 2024-09-13T11:23:10.599 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.5838000000000002e-05 2024-09-13T11:23:10.599 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:10.599 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:10.599 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:23:10.599 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:55.406879+0000", 2024-09-13T11:23:10.599 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.048591267 2024-09-13T11:23:10.600 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:10.600 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:10.600 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:23:10.600 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:55.406899+0000", 2024-09-13T11:23:10.600 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.9279999999999998e-05 2024-09-13T11:23:10.600 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:10.600 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:10.600 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:23:10.600 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:55.407310+0000", 2024-09-13T11:23:10.600 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00041140799999999999 2024-09-13T11:23:10.600 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:10.600 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:10.600 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:23:10.600 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:55.407349+0000", 2024-09-13T11:23:10.601 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.9566e-05 2024-09-13T11:23:10.601 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:10.601 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:10.601 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:23:10.601 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:55.409131+0000", 2024-09-13T11:23:10.601 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0017810250000000001 2024-09-13T11:23:10.601 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:10.601 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:10.601 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:23:10.601 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:55.409211+0000", 2024-09-13T11:23:10.601 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.0438000000000005e-05 2024-09-13T11:23:10.601 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:10.602 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:23:10.602 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:10.602 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:10.602 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:10.602 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.4665.0:12166 3.e e49/30 3:7e73cb7a:::benchmark_data_smithi177_31116_object12165:head v 49'801, mlcod=49'795)", 2024-09-13T11:23:10.602 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:22:55.851739+0000", 2024-09-13T11:23:10.602 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 14.669972605, 2024-09-13T11:23:10.602 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.055096021000000002, 2024-09-13T11:23:10.602 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:23:10.602 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:23:10.602 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:10.602 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:23:10.602 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:10.602 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:23:10.603 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:55.851739+0000", 2024-09-13T11:23:10.603 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:10.603 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:10.603 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:10.603 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:23:10.603 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:55.851739+0000", 2024-09-13T11:23:10.603 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:10.603 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:10.603 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:10.603 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:23:10.603 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:55.851742+0000", 2024-09-13T11:23:10.603 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.1360000000000001e-06 2024-09-13T11:23:10.603 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:10.604 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:10.604 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:23:10.604 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:55.851809+0000", 2024-09-13T11:23:10.604 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.7345e-05 2024-09-13T11:23:10.604 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:10.604 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:10.604 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:23:10.604 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:55.904734+0000", 2024-09-13T11:23:10.604 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.052924811000000002 2024-09-13T11:23:10.604 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:10.604 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:10.604 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:23:10.604 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:55.904744+0000", 2024-09-13T11:23:10.604 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0086e-05 2024-09-13T11:23:10.605 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:10.605 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:10.605 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:23:10.605 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:55.904880+0000", 2024-09-13T11:23:10.605 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00013551899999999999 2024-09-13T11:23:10.605 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:10.605 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:10.605 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:23:10.605 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:55.904945+0000", 2024-09-13T11:23:10.605 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.5331000000000002e-05 2024-09-13T11:23:10.605 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:10.605 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:10.605 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:23:10.605 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:55.906723+0000", 2024-09-13T11:23:10.606 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0017778830000000001 2024-09-13T11:23:10.606 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:10.606 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:10.606 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:23:10.606 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:55.906835+0000", 2024-09-13T11:23:10.606 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00011191 2024-09-13T11:23:10.606 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:10.606 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:23:10.606 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:10.606 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:10.606 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:10.606 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.4665.0:12168 3.e e49/30 3:7d73611a:::benchmark_data_smithi177_31116_object12167:head v 49'802, mlcod=49'796)", 2024-09-13T11:23:10.606 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:22:55.853146+0000", 2024-09-13T11:23:10.607 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 14.668565919000001, 2024-09-13T11:23:10.607 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.056655908999999997, 2024-09-13T11:23:10.607 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:23:10.607 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:23:10.607 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:10.607 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:23:10.607 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:10.607 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:23:10.607 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:55.853146+0000", 2024-09-13T11:23:10.607 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:10.607 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:10.607 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:10.607 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:23:10.608 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:55.853146+0000", 2024-09-13T11:23:10.608 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:10.608 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:10.608 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:10.608 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:23:10.608 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:55.853154+0000", 2024-09-13T11:23:10.608 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.2029999999999992e-06 2024-09-13T11:23:10.608 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:10.608 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:10.608 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:23:10.608 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:55.853209+0000", 2024-09-13T11:23:10.608 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.4679000000000003e-05 2024-09-13T11:23:10.609 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:10.609 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:10.609 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:23:10.609 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:55.907991+0000", 2024-09-13T11:23:10.609 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.054782072000000001 2024-09-13T11:23:10.609 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:10.609 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:10.609 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:23:10.609 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:55.908002+0000", 2024-09-13T11:23:10.609 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1884e-05 2024-09-13T11:23:10.609 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:10.609 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:10.609 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:23:10.610 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:55.908104+0000", 2024-09-13T11:23:10.610 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00010127 2024-09-13T11:23:10.610 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:10.610 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:10.610 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:23:10.610 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:55.908142+0000", 2024-09-13T11:23:10.610 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.8661999999999997e-05 2024-09-13T11:23:10.610 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:10.610 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:10.610 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:23:10.610 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:55.909718+0000", 2024-09-13T11:23:10.610 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001575271 2024-09-13T11:23:10.611 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:10.611 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:10.611 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:23:10.611 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:55.909802+0000", 2024-09-13T11:23:10.611 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.3868000000000004e-05 2024-09-13T11:23:10.611 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:10.611 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:23:10.611 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:10.611 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:10.611 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:10.611 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.4665.0:12202 3.e e49/30 3:79a193d0:::benchmark_data_smithi177_31116_object12201:head v 49'803, mlcod=49'798)", 2024-09-13T11:23:10.611 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:22:55.875406+0000", 2024-09-13T11:23:10.611 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 14.646305455, 2024-09-13T11:23:10.612 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.067950287999999998, 2024-09-13T11:23:10.612 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:23:10.612 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:23:10.612 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:10.612 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:23:10.612 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:10.612 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:23:10.612 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:55.875406+0000", 2024-09-13T11:23:10.612 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:10.612 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:10.612 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:10.612 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:23:10.612 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:55.875406+0000", 2024-09-13T11:23:10.613 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:10.613 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:10.613 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:10.613 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:23:10.613 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:55.875415+0000", 2024-09-13T11:23:10.613 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.8559999999999996e-06 2024-09-13T11:23:10.613 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:10.613 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:10.613 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:23:10.613 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:55.875471+0000", 2024-09-13T11:23:10.613 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.5905000000000001e-05 2024-09-13T11:23:10.613 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:10.613 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:10.614 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:23:10.614 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:55.941380+0000", 2024-09-13T11:23:10.614 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.065908777000000002 2024-09-13T11:23:10.614 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:10.614 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:10.614 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:23:10.614 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:55.941397+0000", 2024-09-13T11:23:10.614 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.6892999999999999e-05 2024-09-13T11:23:10.614 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:10.614 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:10.614 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:23:10.614 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:55.941512+0000", 2024-09-13T11:23:10.614 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00011496 2024-09-13T11:23:10.614 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:10.615 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:10.615 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:23:10.615 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:55.941548+0000", 2024-09-13T11:23:10.615 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.6375000000000003e-05 2024-09-13T11:23:10.615 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:10.615 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:10.615 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:23:10.615 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:55.943273+0000", 2024-09-13T11:23:10.615 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0017248509999999999 2024-09-13T11:23:10.615 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:10.615 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:10.615 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:23:10.615 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:55.943356+0000", 2024-09-13T11:23:10.615 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.3671000000000006e-05 2024-09-13T11:23:10.616 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:10.616 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:23:10.616 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:10.616 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:10.616 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:10.616 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:19098 3.2 3:46319609:::benchmark_data_smithi177_31116_object19097:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-13T11:23:10.616 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:00.271981+0000", 2024-09-13T11:23:10.616 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 10.249730724999999, 2024-09-13T11:23:10.616 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.056514595000000001, 2024-09-13T11:23:10.616 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:23:10.616 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:23:10.616 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:10.617 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:23:10.617 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:23:10.617 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:10.617 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 19098 2024-09-13T11:23:10.617 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:10.617 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:23:10.617 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:10.617 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:23:10.617 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:00.271981+0000", 2024-09-13T11:23:10.617 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:10.617 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:10.617 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:10.617 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:23:10.617 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:00.271981+0000", 2024-09-13T11:23:10.618 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:10.618 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:10.618 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:10.618 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:23:10.618 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:00.271983+0000", 2024-09-13T11:23:10.618 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.736e-06 2024-09-13T11:23:10.618 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:10.618 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:10.618 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:23:10.618 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:00.272057+0000", 2024-09-13T11:23:10.618 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.4350999999999993e-05 2024-09-13T11:23:10.618 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:10.618 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:10.619 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:23:10.619 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:00.272068+0000", 2024-09-13T11:23:10.619 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1284000000000001e-05 2024-09-13T11:23:10.619 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:10.619 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:10.619 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:23:10.619 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:00.272074+0000", 2024-09-13T11:23:10.619 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.9839999999999998e-06 2024-09-13T11:23:10.619 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:10.619 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:10.619 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:23:10.619 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:00.272118+0000", 2024-09-13T11:23:10.619 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.4155e-05 2024-09-13T11:23:10.620 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:10.620 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:10.620 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:23:10.620 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:00.272240+0000", 2024-09-13T11:23:10.620 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000121706 2024-09-13T11:23:10.620 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:10.620 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:10.620 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 12", 2024-09-13T11:23:10.620 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:00.272356+0000", 2024-09-13T11:23:10.620 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00011628700000000001 2024-09-13T11:23:10.620 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:10.620 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:10.620 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:23:10.620 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:00.276008+0000", 2024-09-13T11:23:10.621 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0036515520000000002 2024-09-13T11:23:10.621 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:10.621 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:10.621 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:10.621 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:00.328408+0000", 2024-09-13T11:23:10.621 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.052400265000000001 2024-09-13T11:23:10.621 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:10.621 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:10.621 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:23:10.621 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:00.328462+0000", 2024-09-13T11:23:10.621 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.3409000000000003e-05 2024-09-13T11:23:10.621 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:10.621 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:10.622 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:23:10.622 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:00.328495+0000", 2024-09-13T11:23:10.622 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.3865999999999997e-05 2024-09-13T11:23:10.622 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:10.622 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:23:10.622 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:10.622 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:10.622 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:10.622 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop_reply(client.4665.0:19098 3.2 e53/30 ondisk, result = 0)", 2024-09-13T11:23:10.622 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:00.273923+0000", 2024-09-13T11:23:10.622 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 10.247788244000001, 2024-09-13T11:23:10.622 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.054583803, 2024-09-13T11:23:10.623 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:23:10.623 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:23:10.623 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:23:10.623 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:23:10.623 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:10.623 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:23:10.623 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:00.273923+0000", 2024-09-13T11:23:10.623 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:10.623 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:10.623 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:10.623 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:23:10.623 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:00.273923+0000", 2024-09-13T11:23:10.624 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:10.624 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:10.624 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:10.624 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:23:10.624 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:00.273925+0000", 2024-09-13T11:23:10.624 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2589999999999999e-06 2024-09-13T11:23:10.624 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:10.624 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:10.624 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:23:10.624 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:00.273930+0000", 2024-09-13T11:23:10.624 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.6779999999999997e-06 2024-09-13T11:23:10.624 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:10.625 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:10.625 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:23:10.625 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:00.327623+0000", 2024-09-13T11:23:10.625 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.053692462000000003 2024-09-13T11:23:10.625 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:10.625 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:10.625 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:23:10.625 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:00.327631+0000", 2024-09-13T11:23:10.625 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.9789999999999993e-06 2024-09-13T11:23:10.625 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:10.625 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:10.625 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:23:10.625 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:00.328393+0000", 2024-09-13T11:23:10.626 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00076212299999999997 2024-09-13T11:23:10.626 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:10.626 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:10.626 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:23:10.626 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:00.328401+0000", 2024-09-13T11:23:10.626 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.9529999999999999e-06 2024-09-13T11:23:10.626 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:10.626 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:10.626 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:23:10.626 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:00.328507+0000", 2024-09-13T11:23:10.626 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00010634899999999999 2024-09-13T11:23:10.626 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:10.626 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:23:10.627 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:10.627 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:10.627 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:10.627 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:19844 3.2 3:47619240:::benchmark_data_smithi177_31116_object19843:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e54)", 2024-09-13T11:23:10.627 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:00.641375+0000", 2024-09-13T11:23:10.627 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 9.8803360710000003, 2024-09-13T11:23:10.627 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.049986293000000001, 2024-09-13T11:23:10.627 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:23:10.627 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:23:10.627 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:10.627 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:23:10.627 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:23:10.628 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:10.628 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 19844 2024-09-13T11:23:10.628 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:10.628 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:23:10.628 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:10.628 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:23:10.628 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:00.641375+0000", 2024-09-13T11:23:10.628 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:10.628 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:10.628 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:10.628 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:23:10.628 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:00.641375+0000", 2024-09-13T11:23:10.628 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:10.629 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:10.629 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:10.629 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:23:10.629 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:00.641379+0000", 2024-09-13T11:23:10.629 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.326e-06 2024-09-13T11:23:10.629 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:10.629 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:10.629 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:23:10.629 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:00.641435+0000", 2024-09-13T11:23:10.629 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.6143000000000001e-05 2024-09-13T11:23:10.629 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:10.629 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:10.629 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:23:10.630 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:00.641457+0000", 2024-09-13T11:23:10.630 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.2123000000000001e-05 2024-09-13T11:23:10.630 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:10.630 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:10.630 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:23:10.630 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:00.641471+0000", 2024-09-13T11:23:10.630 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3896e-05 2024-09-13T11:23:10.630 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:10.630 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:10.630 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:23:10.630 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:00.641552+0000", 2024-09-13T11:23:10.630 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.1334000000000004e-05 2024-09-13T11:23:10.630 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:10.631 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:10.631 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:23:10.631 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:00.641748+0000", 2024-09-13T11:23:10.631 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000195568 2024-09-13T11:23:10.631 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:10.631 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:10.631 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 12", 2024-09-13T11:23:10.631 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:00.641878+0000", 2024-09-13T11:23:10.631 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000129973 2024-09-13T11:23:10.631 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:10.631 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:10.631 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:23:10.631 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:00.643220+0000", 2024-09-13T11:23:10.632 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0013425869999999999 2024-09-13T11:23:10.632 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:10.632 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:10.632 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:10.632 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:00.691255+0000", 2024-09-13T11:23:10.632 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.048034133999999999 2024-09-13T11:23:10.632 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:10.632 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:10.632 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:23:10.632 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:00.691315+0000", 2024-09-13T11:23:10.632 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.0383000000000001e-05 2024-09-13T11:23:10.632 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:10.632 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:10.633 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:23:10.633 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:00.691362+0000", 2024-09-13T11:23:10.633 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.6826000000000002e-05 2024-09-13T11:23:10.633 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:10.633 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:23:10.633 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:10.633 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:10.633 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:23:10.633 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-13T11:23:10.634 INFO:tasks.ceph.osd.15.smithi177.stderr:2024-09-13T11:23:10.538+0000 7f9d618a4640 -1 osd.15 59 handle_osd_map injecting map crc failure 2024-09-13T11:23:10.634 INFO:tasks.ceph.osd.12.smithi001.stderr:2024-09-13T11:23:10.528+0000 7f891d10b640 -1 osd.12 59 handle_osd_map injecting map crc failure 2024-09-13T11:23:10.636 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-13T11:23:10.636 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-09-13T11:23:10.630+0000 7f97befc9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26315) UID: 0 2024-09-13T11:23:10.736 INFO:tasks.ceph.osd.7.smithi177.stderr:2024-09-13T11:23:10.734+0000 7f897b51d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26575) UID: 0 2024-09-13T11:23:10.786 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-09-13T11:23:10.786 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [ 2024-09-13T11:23:10.786 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:10.786 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:32355 3.a 3:5c11d29f:::benchmark_data_smithi177_31116_object32354:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e60)", 2024-09-13T11:23:10.786 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:10.783319+0000", 2024-09-13T11:23:10.786 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 0.0034341350000000001, 2024-09-13T11:23:10.787 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0034654659999999999, 2024-09-13T11:23:10.787 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:23:10.787 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:23:10.787 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "waiting for sub ops", 2024-09-13T11:23:10.787 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:23:10.787 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:23:10.787 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:10.787 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 32355 2024-09-13T11:23:10.787 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:10.787 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:23:10.787 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:10.787 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:23:10.788 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:10.783319+0000", 2024-09-13T11:23:10.788 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:10.788 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:10.788 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:10.788 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:23:10.788 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:10.783319+0000", 2024-09-13T11:23:10.788 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:10.788 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:10.788 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:10.788 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:23:10.788 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:10.783322+0000", 2024-09-13T11:23:10.788 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.0600000000000002e-06 2024-09-13T11:23:10.788 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:10.789 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:10.789 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:23:10.789 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:10.783348+0000", 2024-09-13T11:23:10.789 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.6740000000000001e-05 2024-09-13T11:23:10.789 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:10.789 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:10.789 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:23:10.789 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:10.783362+0000", 2024-09-13T11:23:10.789 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3964999999999999e-05 2024-09-13T11:23:10.789 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:10.789 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:10.789 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:23:10.789 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:10.783371+0000", 2024-09-13T11:23:10.789 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.2420000000000006e-06 2024-09-13T11:23:10.790 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:10.790 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:10.790 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:23:10.790 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:10.784028+0000", 2024-09-13T11:23:10.790 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000656888 2024-09-13T11:23:10.790 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:10.790 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:10.790 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:23:10.790 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:10.784179+0000", 2024-09-13T11:23:10.790 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000150569 2024-09-13T11:23:10.790 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:10.790 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:10.790 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 7", 2024-09-13T11:23:10.790 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:10.784334+0000", 2024-09-13T11:23:10.791 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00015547900000000001 2024-09-13T11:23:10.791 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:10.791 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:10.791 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:23:10.791 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:10.785769+0000", 2024-09-13T11:23:10.791 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001434191 2024-09-13T11:23:10.791 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:10.791 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:23:10.791 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:10.791 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:10.791 INFO:teuthology.orchestra.run.smithi177.stdout: ], 2024-09-13T11:23:10.791 INFO:teuthology.orchestra.run.smithi177.stdout: "num_ops": 1 2024-09-13T11:23:10.791 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-09-13T11:23:10.805 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-13T11:23:10.836 INFO:tasks.ceph.osd.5.smithi092.stderr:2024-09-13T11:23:10.834+0000 7f6f0b705640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26346) UID: 0 2024-09-13T11:23:10.937 INFO:tasks.ceph.osd.12.smithi001.stderr:2024-09-13T11:23:10.932+0000 7f892c6e6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26314) UID: 0 2024-09-13T11:23:10.999 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-09-13T11:23:10.999 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [], 2024-09-13T11:23:10.999 INFO:teuthology.orchestra.run.smithi177.stdout: "complaint_time": 30, 2024-09-13T11:23:10.999 INFO:teuthology.orchestra.run.smithi177.stdout: "num_blocked_ops": 0 2024-09-13T11:23:10.999 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-09-13T11:23:11.001 INFO:tasks.radosbench.radosbench.0.smithi177.stdout: 44 15 32722 32707 46.4533 99.125 0.00259813 0.02152 2024-09-13T11:23:11.019 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-13T11:23:11.038 INFO:tasks.ceph.osd.5.smithi092.stderr:2024-09-13T11:23:11.034+0000 7f6f0b705640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26346) UID: 0 2024-09-13T11:23:11.139 INFO:tasks.ceph.osd.13.smithi092.stderr:2024-09-13T11:23:11.134+0000 7f3e5afea640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26350) UID: 0 2024-09-13T11:23:11.159 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-09-13T11:23:11.159 INFO:teuthology.orchestra.run.smithi177.stdout: "size": 20, 2024-09-13T11:23:11.159 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 600, 2024-09-13T11:23:11.160 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [ 2024-09-13T11:23:11.160 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:11.160 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:837 3.a 3:5ab13ab0:::benchmark_data_smithi177_31116_object836:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-09-13T11:23:11.160 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:22:49.461597+0000", 2024-09-13T11:23:11.160 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 21.696532385000001, 2024-09-13T11:23:11.160 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.044395994000000001, 2024-09-13T11:23:11.160 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:23:11.160 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:23:11.160 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:11.160 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:23:11.160 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:23:11.160 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:11.161 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 837 2024-09-13T11:23:11.161 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:11.161 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:23:11.161 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:11.161 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:23:11.161 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.461597+0000", 2024-09-13T11:23:11.161 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:11.161 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:11.161 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:11.161 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:23:11.161 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.461597+0000", 2024-09-13T11:23:11.161 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:11.162 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:11.162 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:11.162 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:23:11.162 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.461598+0000", 2024-09-13T11:23:11.162 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.44e-06 2024-09-13T11:23:11.162 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:11.162 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:11.162 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:23:11.162 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.461634+0000", 2024-09-13T11:23:11.162 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.5673999999999998e-05 2024-09-13T11:23:11.162 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:11.162 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:11.162 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:23:11.163 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.461648+0000", 2024-09-13T11:23:11.163 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.4047e-05 2024-09-13T11:23:11.163 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:11.163 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:11.163 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:23:11.163 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.461655+0000", 2024-09-13T11:23:11.163 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.9720000000000003e-06 2024-09-13T11:23:11.163 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:11.163 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:11.163 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:23:11.163 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.461715+0000", 2024-09-13T11:23:11.163 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.0195999999999998e-05 2024-09-13T11:23:11.163 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:11.164 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:11.164 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:23:11.164 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.461819+0000", 2024-09-13T11:23:11.164 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000103447 2024-09-13T11:23:11.164 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:11.164 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:11.164 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 7", 2024-09-13T11:23:11.164 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.461939+0000", 2024-09-13T11:23:11.164 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000120913 2024-09-13T11:23:11.164 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:11.164 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:11.164 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:23:11.164 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.462669+0000", 2024-09-13T11:23:11.164 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00072908200000000004 2024-09-13T11:23:11.165 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:11.165 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:11.165 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:11.165 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.505895+0000", 2024-09-13T11:23:11.165 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.043226597999999998 2024-09-13T11:23:11.165 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:11.165 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:11.165 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:23:11.165 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.505941+0000", 2024-09-13T11:23:11.165 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.5503999999999997e-05 2024-09-13T11:23:11.165 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:11.165 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:11.165 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:23:11.166 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.505993+0000", 2024-09-13T11:23:11.166 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.2120999999999998e-05 2024-09-13T11:23:11.166 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:11.166 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:23:11.166 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:11.166 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:11.166 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:11.166 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop_reply(client.4665.0:837 3.a e43/30 ondisk, result = 0)", 2024-09-13T11:23:11.166 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:22:49.463026+0000", 2024-09-13T11:23:11.166 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 21.695102985999998, 2024-09-13T11:23:11.166 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.042975286000000001, 2024-09-13T11:23:11.166 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:23:11.166 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:23:11.167 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:23:11.167 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:23:11.167 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:11.167 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:23:11.167 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.463026+0000", 2024-09-13T11:23:11.167 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:11.167 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:11.167 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:11.167 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:23:11.167 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.463026+0000", 2024-09-13T11:23:11.167 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:11.167 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:11.167 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:11.167 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:23:11.168 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.463028+0000", 2024-09-13T11:23:11.168 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.443e-06 2024-09-13T11:23:11.168 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:11.168 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:11.168 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:23:11.168 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.463033+0000", 2024-09-13T11:23:11.168 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.7830000000000004e-06 2024-09-13T11:23:11.168 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:11.168 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:11.168 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:23:11.168 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.505813+0000", 2024-09-13T11:23:11.168 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.042779620999999997 2024-09-13T11:23:11.168 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:11.169 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:11.169 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:23:11.169 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.505820+0000", 2024-09-13T11:23:11.169 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.233e-06 2024-09-13T11:23:11.169 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:11.169 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:11.169 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:23:11.169 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.505881+0000", 2024-09-13T11:23:11.169 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.0810000000000002e-05 2024-09-13T11:23:11.169 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:11.169 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:11.169 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:23:11.169 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.505891+0000", 2024-09-13T11:23:11.169 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.6059999999999992e-06 2024-09-13T11:23:11.170 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:11.170 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:11.170 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:23:11.170 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.506001+0000", 2024-09-13T11:23:11.170 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00011079 2024-09-13T11:23:11.170 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:11.170 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:23:11.170 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:11.170 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:11.170 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:11.170 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:855 3.a 3:5ce42657:::benchmark_data_smithi177_31116_object854:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-09-13T11:23:11.170 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:22:49.480636+0000", 2024-09-13T11:23:11.170 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 21.677492790999999, 2024-09-13T11:23:11.170 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.040802389000000001, 2024-09-13T11:23:11.171 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:23:11.171 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:23:11.171 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:11.171 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:23:11.171 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:23:11.171 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:11.171 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 855 2024-09-13T11:23:11.171 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:11.171 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:23:11.171 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:11.171 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:23:11.171 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.480636+0000", 2024-09-13T11:23:11.171 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:11.172 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:11.172 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:11.172 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:23:11.172 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.480636+0000", 2024-09-13T11:23:11.172 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:11.172 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:11.172 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:11.172 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:23:11.172 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.480640+0000", 2024-09-13T11:23:11.172 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.0929999999999996e-06 2024-09-13T11:23:11.172 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:11.172 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:11.172 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:23:11.172 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.480686+0000", 2024-09-13T11:23:11.173 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.5629e-05 2024-09-13T11:23:11.173 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:11.173 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:11.173 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:23:11.173 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.480713+0000", 2024-09-13T11:23:11.173 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.7021000000000001e-05 2024-09-13T11:23:11.173 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:11.173 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:11.173 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:23:11.173 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.480730+0000", 2024-09-13T11:23:11.173 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.6835000000000001e-05 2024-09-13T11:23:11.173 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:11.174 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:11.174 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:23:11.174 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.480815+0000", 2024-09-13T11:23:11.174 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.4888000000000004e-05 2024-09-13T11:23:11.174 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:11.174 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:11.174 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:23:11.174 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.480940+0000", 2024-09-13T11:23:11.174 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00012519000000000001 2024-09-13T11:23:11.174 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:11.174 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:11.174 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 7", 2024-09-13T11:23:11.175 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.481056+0000", 2024-09-13T11:23:11.175 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000115467 2024-09-13T11:23:11.175 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:11.175 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:11.175 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:23:11.175 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.482076+0000", 2024-09-13T11:23:11.175 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001020038 2024-09-13T11:23:11.175 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:11.175 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:11.175 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:11.175 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.521353+0000", 2024-09-13T11:23:11.175 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.039277764 2024-09-13T11:23:11.175 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:11.175 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:11.176 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:23:11.176 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.521402+0000", 2024-09-13T11:23:11.176 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.8408000000000001e-05 2024-09-13T11:23:11.176 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:11.176 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:11.176 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:23:11.176 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.521439+0000", 2024-09-13T11:23:11.176 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.7055999999999999e-05 2024-09-13T11:23:11.176 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:11.176 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:23:11.176 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:11.176 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:11.176 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:11.176 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop_reply(client.4665.0:855 3.a e43/30 ondisk, result = 0)", 2024-09-13T11:23:11.177 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:22:49.482718+0000", 2024-09-13T11:23:11.177 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 21.675411423, 2024-09-13T11:23:11.177 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.038728772000000002, 2024-09-13T11:23:11.177 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:23:11.177 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:23:11.177 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:23:11.177 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:23:11.177 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:11.177 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:23:11.177 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.482718+0000", 2024-09-13T11:23:11.177 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:11.177 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:11.177 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:11.178 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:23:11.178 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.482718+0000", 2024-09-13T11:23:11.178 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:11.178 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:11.178 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:11.178 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:23:11.178 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.482719+0000", 2024-09-13T11:23:11.178 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.7269999999999999e-06 2024-09-13T11:23:11.178 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:11.178 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:11.178 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:23:11.178 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.482725+0000", 2024-09-13T11:23:11.178 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.3650000000000003e-06 2024-09-13T11:23:11.178 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:11.179 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:11.179 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:23:11.179 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.521273+0000", 2024-09-13T11:23:11.179 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.038548495000000002 2024-09-13T11:23:11.179 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:11.179 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:11.179 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:23:11.179 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.521280+0000", 2024-09-13T11:23:11.179 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.7120000000000003e-06 2024-09-13T11:23:11.179 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:11.179 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:11.179 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:23:11.179 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.521341+0000", 2024-09-13T11:23:11.179 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.0693000000000003e-05 2024-09-13T11:23:11.180 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:11.180 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:11.180 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:23:11.180 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.521349+0000", 2024-09-13T11:23:11.180 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.174e-06 2024-09-13T11:23:11.180 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:11.180 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:11.180 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:23:11.180 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.521447+0000", 2024-09-13T11:23:11.180 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.7606e-05 2024-09-13T11:23:11.180 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:11.180 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:23:11.180 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:11.180 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:11.181 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:11.181 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:861 3.a 3:5772350b:::benchmark_data_smithi177_31116_object860:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-09-13T11:23:11.181 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:22:49.487183+0000", 2024-09-13T11:23:11.181 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 21.670945715999999, 2024-09-13T11:23:11.181 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.042583651, 2024-09-13T11:23:11.181 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:23:11.181 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:23:11.181 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:11.181 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:23:11.181 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:23:11.181 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:11.181 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 861 2024-09-13T11:23:11.181 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:11.181 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:23:11.182 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:11.182 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:23:11.182 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.487183+0000", 2024-09-13T11:23:11.182 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:11.182 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:11.182 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:11.182 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:23:11.182 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.487183+0000", 2024-09-13T11:23:11.182 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:11.182 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:11.182 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:11.182 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:23:11.182 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.487186+0000", 2024-09-13T11:23:11.182 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.5409999999999999e-06 2024-09-13T11:23:11.183 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:11.183 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:11.183 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:23:11.183 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.487222+0000", 2024-09-13T11:23:11.183 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.5621000000000003e-05 2024-09-13T11:23:11.183 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:11.183 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:11.183 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:23:11.183 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.487243+0000", 2024-09-13T11:23:11.183 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.0902000000000001e-05 2024-09-13T11:23:11.183 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:11.183 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:11.184 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:23:11.184 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.487253+0000", 2024-09-13T11:23:11.184 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0667e-05 2024-09-13T11:23:11.184 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:11.184 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:11.184 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:23:11.184 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.487304+0000", 2024-09-13T11:23:11.184 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.0658000000000002e-05 2024-09-13T11:23:11.184 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:11.184 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:11.184 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:23:11.184 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.487392+0000", 2024-09-13T11:23:11.184 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.7824999999999993e-05 2024-09-13T11:23:11.184 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:11.185 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:11.185 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 7", 2024-09-13T11:23:11.185 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.487483+0000", 2024-09-13T11:23:11.185 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.0915999999999995e-05 2024-09-13T11:23:11.185 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:11.185 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:11.185 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:23:11.185 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.488424+0000", 2024-09-13T11:23:11.185 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00094114899999999996 2024-09-13T11:23:11.185 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:11.185 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:11.185 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:11.186 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.529663+0000", 2024-09-13T11:23:11.186 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.041239215000000003 2024-09-13T11:23:11.186 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:11.186 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:11.186 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:23:11.186 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.529719+0000", 2024-09-13T11:23:11.186 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.6480999999999997e-05 2024-09-13T11:23:11.186 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:11.186 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:11.186 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:23:11.186 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.529767+0000", 2024-09-13T11:23:11.186 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.7676000000000002e-05 2024-09-13T11:23:11.186 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:11.186 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:23:11.186 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:11.187 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:11.187 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:11.187 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop_reply(client.4665.0:861 3.a e43/30 ondisk, result = 0)", 2024-09-13T11:23:11.187 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:22:49.488684+0000", 2024-09-13T11:23:11.187 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 21.669445264, 2024-09-13T11:23:11.187 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.041092947999999997, 2024-09-13T11:23:11.187 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:23:11.187 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:23:11.187 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:23:11.187 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:23:11.187 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:11.187 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:23:11.187 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.488684+0000", 2024-09-13T11:23:11.188 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:11.188 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:11.188 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:11.188 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:23:11.188 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.488684+0000", 2024-09-13T11:23:11.188 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:11.188 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:11.188 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:11.188 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:23:11.188 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.488685+0000", 2024-09-13T11:23:11.188 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.4470000000000001e-06 2024-09-13T11:23:11.188 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:11.188 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:11.188 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:23:11.189 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.488690+0000", 2024-09-13T11:23:11.189 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.8520000000000003e-06 2024-09-13T11:23:11.189 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:11.189 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:11.189 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:23:11.189 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.529557+0000", 2024-09-13T11:23:11.189 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.040866524000000001 2024-09-13T11:23:11.189 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:11.189 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:11.189 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:23:11.189 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.529567+0000", 2024-09-13T11:23:11.189 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.011e-05 2024-09-13T11:23:11.189 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:11.190 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:11.190 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:23:11.190 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.529649+0000", 2024-09-13T11:23:11.190 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.2280000000000005e-05 2024-09-13T11:23:11.191 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:11.191 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:11.191 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:23:11.191 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.529658+0000", 2024-09-13T11:23:11.191 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.3549999999999994e-06 2024-09-13T11:23:11.191 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:11.191 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:11.191 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:23:11.191 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.529777+0000", 2024-09-13T11:23:11.191 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00011838 2024-09-13T11:23:11.191 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:11.191 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:23:11.191 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:11.192 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:11.192 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:11.192 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:867 3.a 3:5ca59689:::benchmark_data_smithi177_31116_object866:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-09-13T11:23:11.192 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:22:49.493530+0000", 2024-09-13T11:23:11.192 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 21.664599272, 2024-09-13T11:23:11.192 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.038165363000000001, 2024-09-13T11:23:11.192 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:23:11.192 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:23:11.192 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:11.192 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:23:11.192 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:23:11.192 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:11.192 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 867 2024-09-13T11:23:11.193 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:11.193 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:23:11.193 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:11.193 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:23:11.193 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.493530+0000", 2024-09-13T11:23:11.193 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:11.193 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:11.193 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:11.193 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:23:11.193 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.493530+0000", 2024-09-13T11:23:11.193 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:11.193 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:11.193 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:11.194 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:23:11.194 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.493532+0000", 2024-09-13T11:23:11.194 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.5459999999999998e-06 2024-09-13T11:23:11.194 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:11.194 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:11.194 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:23:11.194 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.493568+0000", 2024-09-13T11:23:11.194 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.5354000000000001e-05 2024-09-13T11:23:11.194 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:11.194 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:11.194 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:23:11.194 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.493595+0000", 2024-09-13T11:23:11.194 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.6823999999999999e-05 2024-09-13T11:23:11.195 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:11.195 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:11.195 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:23:11.195 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.493606+0000", 2024-09-13T11:23:11.195 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.1045e-05 2024-09-13T11:23:11.195 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:11.195 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:11.195 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:23:11.195 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.493676+0000", 2024-09-13T11:23:11.195 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.0007999999999998e-05 2024-09-13T11:23:11.195 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:11.195 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:11.196 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:23:11.196 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.493792+0000", 2024-09-13T11:23:11.196 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00011584800000000001 2024-09-13T11:23:11.196 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:11.196 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:11.196 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 7", 2024-09-13T11:23:11.196 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.493926+0000", 2024-09-13T11:23:11.196 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00013482199999999999 2024-09-13T11:23:11.196 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:11.196 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:11.196 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:23:11.196 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.494881+0000", 2024-09-13T11:23:11.197 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00095461899999999998 2024-09-13T11:23:11.197 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:11.197 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:11.197 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:11.197 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.531625+0000", 2024-09-13T11:23:11.197 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.036744106999999998 2024-09-13T11:23:11.197 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:11.197 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:11.197 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:23:11.197 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.531663+0000", 2024-09-13T11:23:11.197 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.7962e-05 2024-09-13T11:23:11.197 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:11.197 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:11.198 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:23:11.198 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.531695+0000", 2024-09-13T11:23:11.198 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.2227999999999997e-05 2024-09-13T11:23:11.198 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:11.198 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:23:11.198 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:11.198 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:11.198 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:11.198 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:1003 3.a 3:5aec7cc6:::benchmark_data_smithi177_31116_object1002:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-09-13T11:23:11.198 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:22:49.614497+0000", 2024-09-13T11:23:11.198 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 21.543631834999999, 2024-09-13T11:23:11.198 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.040510329999999997, 2024-09-13T11:23:11.198 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:23:11.199 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:23:11.199 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:11.199 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:23:11.199 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:23:11.199 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:11.199 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 1003 2024-09-13T11:23:11.199 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:11.199 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:23:11.199 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:11.199 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:23:11.199 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.614497+0000", 2024-09-13T11:23:11.199 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:11.199 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:11.200 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:11.200 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:23:11.200 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.614497+0000", 2024-09-13T11:23:11.200 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:11.200 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:11.200 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:11.200 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:23:11.200 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.614500+0000", 2024-09-13T11:23:11.200 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.3369999999999998e-06 2024-09-13T11:23:11.200 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:11.200 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:11.200 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:23:11.200 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.614531+0000", 2024-09-13T11:23:11.201 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.0828999999999999e-05 2024-09-13T11:23:11.201 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:11.201 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:11.201 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:23:11.201 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.614546+0000", 2024-09-13T11:23:11.201 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5945000000000001e-05 2024-09-13T11:23:11.201 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:11.201 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:11.201 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:23:11.201 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.614557+0000", 2024-09-13T11:23:11.201 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0317e-05 2024-09-13T11:23:11.201 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:11.201 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:11.202 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:23:11.202 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.614642+0000", 2024-09-13T11:23:11.202 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.4734999999999994e-05 2024-09-13T11:23:11.202 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:11.202 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:11.202 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:23:11.202 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.614794+0000", 2024-09-13T11:23:11.202 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000152027 2024-09-13T11:23:11.202 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:11.202 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:11.202 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 7", 2024-09-13T11:23:11.202 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.614980+0000", 2024-09-13T11:23:11.203 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00018625799999999999 2024-09-13T11:23:11.203 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:11.203 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:11.203 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:23:11.203 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.615821+0000", 2024-09-13T11:23:11.203 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00084122999999999999 2024-09-13T11:23:11.203 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:11.203 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:11.203 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:11.203 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.654761+0000", 2024-09-13T11:23:11.203 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.038939559999999998 2024-09-13T11:23:11.203 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:11.203 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:11.204 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:23:11.204 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.654897+0000", 2024-09-13T11:23:11.204 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00013600999999999999 2024-09-13T11:23:11.204 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:11.204 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:11.204 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:23:11.204 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.655008+0000", 2024-09-13T11:23:11.204 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000111082 2024-09-13T11:23:11.204 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:11.204 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:23:11.204 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:11.204 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:11.204 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:11.205 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop_reply(client.4665.0:1003 3.a e43/30 ondisk, result = 0)", 2024-09-13T11:23:11.205 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:22:49.616305+0000", 2024-09-13T11:23:11.205 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 21.541824394999999, 2024-09-13T11:23:11.205 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.038717949000000002, 2024-09-13T11:23:11.205 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:23:11.205 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:23:11.205 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:23:11.205 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:23:11.205 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:11.205 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:23:11.205 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.616305+0000", 2024-09-13T11:23:11.205 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:11.205 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:11.206 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:11.206 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:23:11.206 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.616305+0000", 2024-09-13T11:23:11.206 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:11.206 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:11.206 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:11.206 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:23:11.206 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.616306+0000", 2024-09-13T11:23:11.206 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5200000000000001e-06 2024-09-13T11:23:11.206 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:11.206 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:11.206 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:23:11.206 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.616311+0000", 2024-09-13T11:23:11.207 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.5759999999999999e-06 2024-09-13T11:23:11.207 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:11.207 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:11.207 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:23:11.207 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.654620+0000", 2024-09-13T11:23:11.207 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.038309187000000001 2024-09-13T11:23:11.207 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:11.207 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:11.207 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:23:11.207 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.654628+0000", 2024-09-13T11:23:11.207 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.1669999999999999e-06 2024-09-13T11:23:11.207 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:11.208 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:11.208 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:23:11.208 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.654723+0000", 2024-09-13T11:23:11.208 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.4877999999999995e-05 2024-09-13T11:23:11.208 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:11.208 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:11.208 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:23:11.208 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.654741+0000", 2024-09-13T11:23:11.208 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.8073000000000001e-05 2024-09-13T11:23:11.208 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:11.208 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:11.208 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:23:11.209 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.655023+0000", 2024-09-13T11:23:11.209 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00028154800000000001 2024-09-13T11:23:11.209 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:11.209 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:23:11.209 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:11.209 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:11.209 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:11.209 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:1122 3.d e43/30 3:b40a2a39:::benchmark_data_smithi177_31116_object1121:head v 43'66, mlcod=43'64)", 2024-09-13T11:23:11.209 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:22:49.712014+0000", 2024-09-13T11:23:11.209 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 21.44611476, 2024-09-13T11:23:11.209 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.040250046999999997, 2024-09-13T11:23:11.209 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:23:11.209 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:23:11.210 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:11.210 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:23:11.210 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:11.210 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:23:11.210 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.712014+0000", 2024-09-13T11:23:11.210 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:11.210 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:11.210 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:11.210 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:23:11.210 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.712014+0000", 2024-09-13T11:23:11.210 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:11.210 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:11.210 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:11.211 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:23:11.211 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.712023+0000", 2024-09-13T11:23:11.211 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.4160000000000004e-06 2024-09-13T11:23:11.211 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:11.211 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:11.211 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:23:11.211 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.712084+0000", 2024-09-13T11:23:11.211 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.0720999999999997e-05 2024-09-13T11:23:11.211 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:11.211 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:11.211 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:23:11.211 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.748087+0000", 2024-09-13T11:23:11.211 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.036003546999999997 2024-09-13T11:23:11.212 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:11.212 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:11.212 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:23:11.212 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.748126+0000", 2024-09-13T11:23:11.212 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.9208999999999997e-05 2024-09-13T11:23:11.212 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:11.212 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:11.212 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:23:11.212 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.748264+0000", 2024-09-13T11:23:11.212 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00013805600000000001 2024-09-13T11:23:11.212 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:11.212 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:11.212 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:23:11.213 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.748313+0000", 2024-09-13T11:23:11.213 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.8575999999999997e-05 2024-09-13T11:23:11.213 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:11.213 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:11.213 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:23:11.213 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.752180+0000", 2024-09-13T11:23:11.213 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.003867498 2024-09-13T11:23:11.213 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:11.213 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:11.213 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:23:11.213 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.752264+0000", 2024-09-13T11:23:11.213 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.4023999999999994e-05 2024-09-13T11:23:11.213 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:11.213 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:23:11.214 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:11.214 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:11.214 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:11.214 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:1275 3.a 3:5a15da6d:::benchmark_data_smithi177_31116_object1274:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-09-13T11:23:11.214 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:22:49.829287+0000", 2024-09-13T11:23:11.214 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 21.328841873999998, 2024-09-13T11:23:11.214 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.043769420000000003, 2024-09-13T11:23:11.214 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:23:11.214 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:23:11.214 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:11.214 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:23:11.214 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:23:11.215 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:11.215 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 1275 2024-09-13T11:23:11.215 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:11.215 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:23:11.215 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:11.215 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:23:11.215 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.829287+0000", 2024-09-13T11:23:11.215 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:11.215 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:11.215 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:11.215 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:23:11.215 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.829287+0000", 2024-09-13T11:23:11.215 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:11.216 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:11.216 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:11.216 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:23:11.216 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.829291+0000", 2024-09-13T11:23:11.216 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.4529999999999998e-06 2024-09-13T11:23:11.216 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:11.216 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:11.216 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:23:11.216 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.829362+0000", 2024-09-13T11:23:11.216 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.1064999999999998e-05 2024-09-13T11:23:11.216 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:11.216 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:11.216 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:23:11.216 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.829389+0000", 2024-09-13T11:23:11.217 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.6991999999999998e-05 2024-09-13T11:23:11.217 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:11.217 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:11.217 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:23:11.217 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.829408+0000", 2024-09-13T11:23:11.217 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.9477e-05 2024-09-13T11:23:11.217 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:11.217 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:11.217 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:23:11.217 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.829514+0000", 2024-09-13T11:23:11.217 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000105832 2024-09-13T11:23:11.217 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:11.217 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:11.217 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:23:11.218 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.829663+0000", 2024-09-13T11:23:11.218 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00014837300000000001 2024-09-13T11:23:11.218 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:11.218 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:11.218 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 7", 2024-09-13T11:23:11.218 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.829815+0000", 2024-09-13T11:23:11.218 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000152235 2024-09-13T11:23:11.218 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:11.218 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:11.218 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:23:11.218 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.832310+0000", 2024-09-13T11:23:11.219 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.002494844 2024-09-13T11:23:11.219 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:11.219 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:11.219 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:11.219 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.872852+0000", 2024-09-13T11:23:11.219 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.040542424000000001 2024-09-13T11:23:11.219 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:11.219 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:11.219 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:23:11.219 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.872981+0000", 2024-09-13T11:23:11.219 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00012902399999999999 2024-09-13T11:23:11.220 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:11.220 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:11.220 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:23:11.220 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.873057+0000", 2024-09-13T11:23:11.220 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.5700999999999999e-05 2024-09-13T11:23:11.220 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:11.220 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:23:11.220 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:11.220 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:11.220 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:11.220 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop_reply(client.4665.0:1275 3.a e43/30 ondisk, result = 0)", 2024-09-13T11:23:11.220 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:22:49.833244+0000", 2024-09-13T11:23:11.220 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 21.324884967999999, 2024-09-13T11:23:11.221 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.039823584000000002, 2024-09-13T11:23:11.221 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:23:11.221 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:23:11.221 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:23:11.221 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:23:11.221 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:11.221 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:23:11.221 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.833244+0000", 2024-09-13T11:23:11.221 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:11.221 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:11.221 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:11.221 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:23:11.221 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.833244+0000", 2024-09-13T11:23:11.221 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:11.222 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:11.222 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:11.222 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:23:11.222 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.833247+0000", 2024-09-13T11:23:11.222 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.802e-06 2024-09-13T11:23:11.222 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:11.222 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:11.222 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:23:11.222 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.833257+0000", 2024-09-13T11:23:11.222 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0068000000000001e-05 2024-09-13T11:23:11.222 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:11.222 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:11.222 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:23:11.223 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.872674+0000", 2024-09-13T11:23:11.223 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.039417059999999997 2024-09-13T11:23:11.223 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:11.223 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:11.223 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:23:11.223 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.872693+0000", 2024-09-13T11:23:11.223 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.889e-05 2024-09-13T11:23:11.223 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:11.223 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:11.223 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:23:11.223 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.872820+0000", 2024-09-13T11:23:11.223 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00012652099999999999 2024-09-13T11:23:11.223 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:11.224 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:11.224 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:23:11.224 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.872839+0000", 2024-09-13T11:23:11.224 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.9344999999999999e-05 2024-09-13T11:23:11.224 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:11.224 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:11.224 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:23:11.224 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.873068+0000", 2024-09-13T11:23:11.224 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000228898 2024-09-13T11:23:11.224 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:11.224 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:23:11.224 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:11.224 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:11.225 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:11.225 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:1282 3.d e43/30 3:b230ba6b:::benchmark_data_smithi177_31116_object1281:head v 43'78, mlcod=43'73)", 2024-09-13T11:23:11.225 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:22:49.834603+0000", 2024-09-13T11:23:11.225 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 21.323526384000001, 2024-09-13T11:23:11.225 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.043089937000000002, 2024-09-13T11:23:11.225 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:23:11.225 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:23:11.225 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:11.225 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:23:11.225 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:11.225 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:23:11.225 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.834603+0000", 2024-09-13T11:23:11.226 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:11.226 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:11.226 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:11.226 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:23:11.226 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.834603+0000", 2024-09-13T11:23:11.226 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:11.226 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:11.226 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:11.226 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:23:11.226 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.834606+0000", 2024-09-13T11:23:11.226 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.7599999999999998e-06 2024-09-13T11:23:11.226 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:11.226 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:11.227 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:23:11.227 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.834684+0000", 2024-09-13T11:23:11.227 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.8490000000000002e-05 2024-09-13T11:23:11.227 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:11.227 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:11.227 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:23:11.227 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.875920+0000", 2024-09-13T11:23:11.227 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.041235956999999997 2024-09-13T11:23:11.227 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:11.227 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:11.227 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:23:11.227 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.875931+0000", 2024-09-13T11:23:11.227 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0834e-05 2024-09-13T11:23:11.227 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:11.228 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:11.228 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:23:11.228 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.876037+0000", 2024-09-13T11:23:11.228 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000106582 2024-09-13T11:23:11.228 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:11.228 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:11.228 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:23:11.228 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.876077+0000", 2024-09-13T11:23:11.228 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.9993000000000001e-05 2024-09-13T11:23:11.228 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:11.228 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:11.228 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:23:11.229 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.877634+0000", 2024-09-13T11:23:11.229 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001556254 2024-09-13T11:23:11.229 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:11.229 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:11.229 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:23:11.229 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.877693+0000", 2024-09-13T11:23:11.229 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.9067000000000002e-05 2024-09-13T11:23:11.229 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:11.229 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:23:11.229 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:11.229 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:11.229 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:11.230 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:1365 3.d e43/30 3:bf637086:::benchmark_data_smithi177_31116_object1364:head v 43'84, mlcod=43'80)", 2024-09-13T11:23:11.230 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:22:49.890601+0000", 2024-09-13T11:23:11.230 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 21.267528285000001, 2024-09-13T11:23:11.230 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.040486463, 2024-09-13T11:23:11.230 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:23:11.230 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:23:11.230 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:11.230 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:23:11.230 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:11.230 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:23:11.230 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.890601+0000", 2024-09-13T11:23:11.230 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:11.231 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:11.231 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:11.231 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:23:11.231 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.890601+0000", 2024-09-13T11:23:11.231 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:11.231 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:11.231 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:11.231 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:23:11.231 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.890603+0000", 2024-09-13T11:23:11.231 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.2879999999999999e-06 2024-09-13T11:23:11.231 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:11.231 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:11.231 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:23:11.232 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.890638+0000", 2024-09-13T11:23:11.232 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.5216999999999998e-05 2024-09-13T11:23:11.232 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:11.232 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:11.232 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:23:11.232 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.925869+0000", 2024-09-13T11:23:11.232 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.035230298 2024-09-13T11:23:11.232 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:11.232 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:11.232 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:23:11.232 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.925883+0000", 2024-09-13T11:23:11.232 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3973e-05 2024-09-13T11:23:11.233 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:11.233 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:11.233 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:23:11.233 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.925987+0000", 2024-09-13T11:23:11.233 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00010407399999999999 2024-09-13T11:23:11.233 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:11.233 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:11.233 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:23:11.233 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.926024+0000", 2024-09-13T11:23:11.233 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.7379000000000002e-05 2024-09-13T11:23:11.233 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:11.233 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:11.233 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:23:11.234 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.931009+0000", 2024-09-13T11:23:11.234 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0049846719999999999 2024-09-13T11:23:11.234 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:11.234 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:11.234 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:23:11.234 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.931087+0000", 2024-09-13T11:23:11.234 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.8561999999999998e-05 2024-09-13T11:23:11.234 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:11.234 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:23:11.234 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:11.234 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:11.234 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:11.235 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:10761 3.d e48/30 3:b65e51ff:::benchmark_data_smithi177_31116_object10760:head v 48'678, mlcod=48'675)", 2024-09-13T11:23:11.235 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:22:54.904092+0000", 2024-09-13T11:23:11.235 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 16.254036718999998, 2024-09-13T11:23:11.235 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.043534784, 2024-09-13T11:23:11.235 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:23:11.235 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:23:11.235 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:11.235 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:23:11.235 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:11.235 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:23:11.235 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:54.904092+0000", 2024-09-13T11:23:11.236 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:11.236 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:11.236 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:11.236 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:23:11.236 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:54.904092+0000", 2024-09-13T11:23:11.236 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:11.236 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:11.236 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:11.236 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:23:11.236 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:54.904106+0000", 2024-09-13T11:23:11.236 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3196999999999999e-05 2024-09-13T11:23:11.236 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:11.237 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:11.237 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:23:11.237 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:54.904141+0000", 2024-09-13T11:23:11.237 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.5607e-05 2024-09-13T11:23:11.237 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:11.237 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:11.237 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:23:11.237 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:54.946464+0000", 2024-09-13T11:23:11.237 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.042322384999999997 2024-09-13T11:23:11.237 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:11.237 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:11.237 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:23:11.237 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:54.946473+0000", 2024-09-13T11:23:11.238 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.842e-06 2024-09-13T11:23:11.238 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:11.238 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:11.238 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:23:11.238 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:54.946542+0000", 2024-09-13T11:23:11.238 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.8382999999999999e-05 2024-09-13T11:23:11.238 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:11.238 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:11.238 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:23:11.238 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:54.946568+0000", 2024-09-13T11:23:11.238 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.6135999999999998e-05 2024-09-13T11:23:11.238 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:11.238 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:11.239 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:23:11.239 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:54.947588+0000", 2024-09-13T11:23:11.239 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0010200400000000001 2024-09-13T11:23:11.239 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:11.239 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:11.239 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:23:11.239 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:54.947627+0000", 2024-09-13T11:23:11.239 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.9193999999999998e-05 2024-09-13T11:23:11.239 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:11.239 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:23:11.239 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:11.239 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:11.239 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:11.239 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:16941 3.a 3:5049f2a8:::benchmark_data_smithi177_31116_object16940:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-13T11:23:11.240 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:22:59.119359+0000", 2024-09-13T11:23:11.240 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 12.038770213999999, 2024-09-13T11:23:11.240 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.045603890000000001, 2024-09-13T11:23:11.240 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:23:11.240 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:23:11.240 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:11.240 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:23:11.240 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:23:11.240 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:11.240 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 16941 2024-09-13T11:23:11.240 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:11.240 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:23:11.240 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:11.241 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:23:11.241 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:59.119359+0000", 2024-09-13T11:23:11.241 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:11.241 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:11.241 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:11.241 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:23:11.241 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:59.119359+0000", 2024-09-13T11:23:11.241 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:11.241 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:11.241 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:11.241 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:23:11.241 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:59.119362+0000", 2024-09-13T11:23:11.242 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.427e-06 2024-09-13T11:23:11.242 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:11.242 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:11.242 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:23:11.242 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:59.119399+0000", 2024-09-13T11:23:11.242 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.6315e-05 2024-09-13T11:23:11.242 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:11.242 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:11.242 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:23:11.242 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:59.119422+0000", 2024-09-13T11:23:11.242 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.2945000000000001e-05 2024-09-13T11:23:11.242 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:11.243 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:11.243 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:23:11.243 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:59.119432+0000", 2024-09-13T11:23:11.243 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0796e-05 2024-09-13T11:23:11.243 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:11.243 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:11.243 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:23:11.243 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:59.119530+0000", 2024-09-13T11:23:11.243 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.7323000000000003e-05 2024-09-13T11:23:11.243 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:11.243 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:11.243 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:23:11.243 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:59.119756+0000", 2024-09-13T11:23:11.244 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00022613700000000001 2024-09-13T11:23:11.244 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:11.244 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:11.244 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 7", 2024-09-13T11:23:11.244 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:59.120020+0000", 2024-09-13T11:23:11.244 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00026381100000000003 2024-09-13T11:23:11.244 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:11.244 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:11.244 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:23:11.244 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:59.121549+0000", 2024-09-13T11:23:11.244 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001529725 2024-09-13T11:23:11.244 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:11.244 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:11.245 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:11.245 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:59.164794+0000", 2024-09-13T11:23:11.245 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.043244432999999999 2024-09-13T11:23:11.245 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:11.245 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:11.245 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:23:11.245 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:59.164883+0000", 2024-09-13T11:23:11.245 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.9058e-05 2024-09-13T11:23:11.245 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:11.245 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:11.245 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:23:11.245 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:59.164963+0000", 2024-09-13T11:23:11.246 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.9919999999999994e-05 2024-09-13T11:23:11.246 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:11.246 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:23:11.246 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:11.246 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:11.246 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:11.246 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:17011 3.a 3:5e3c00d0:::benchmark_data_smithi177_31116_object17010:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-13T11:23:11.246 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:22:59.165305+0000", 2024-09-13T11:23:11.246 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 11.992824541999999, 2024-09-13T11:23:11.246 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.042388409000000002, 2024-09-13T11:23:11.246 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:23:11.246 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:23:11.246 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:11.246 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:23:11.246 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:23:11.247 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:11.247 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 17011 2024-09-13T11:23:11.247 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:11.247 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:23:11.247 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:11.247 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:23:11.247 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:59.165305+0000", 2024-09-13T11:23:11.247 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:11.247 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:11.247 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:11.247 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:23:11.247 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:59.165305+0000", 2024-09-13T11:23:11.247 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:11.248 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:11.248 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:11.248 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:23:11.248 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:59.165306+0000", 2024-09-13T11:23:11.248 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5999999999999999e-06 2024-09-13T11:23:11.248 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:11.248 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:11.248 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:23:11.248 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:59.165352+0000", 2024-09-13T11:23:11.248 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.6180000000000002e-05 2024-09-13T11:23:11.248 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:11.248 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:11.248 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:23:11.248 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:59.165368+0000", 2024-09-13T11:23:11.249 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5854000000000001e-05 2024-09-13T11:23:11.249 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:11.249 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:11.249 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:23:11.249 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:59.165378+0000", 2024-09-13T11:23:11.249 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0188e-05 2024-09-13T11:23:11.249 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:11.249 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:11.249 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:23:11.249 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:59.165458+0000", 2024-09-13T11:23:11.249 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.9163000000000001e-05 2024-09-13T11:23:11.249 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:11.249 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:11.250 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:23:11.250 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:59.165673+0000", 2024-09-13T11:23:11.250 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00021530200000000001 2024-09-13T11:23:11.250 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:11.250 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:11.250 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 7", 2024-09-13T11:23:11.250 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:59.165942+0000", 2024-09-13T11:23:11.250 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00026858200000000001 2024-09-13T11:23:11.250 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:11.250 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:11.250 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:23:11.250 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:59.167472+0000", 2024-09-13T11:23:11.250 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0015307540000000001 2024-09-13T11:23:11.251 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:11.251 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:11.251 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:11.251 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:59.207563+0000", 2024-09-13T11:23:11.251 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.040090354000000002 2024-09-13T11:23:11.251 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:11.251 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:11.251 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:23:11.251 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:59.207634+0000", 2024-09-13T11:23:11.251 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.1601999999999996e-05 2024-09-13T11:23:11.251 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:11.251 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:11.251 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:23:11.252 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:59.207693+0000", 2024-09-13T11:23:11.252 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.8829999999999997e-05 2024-09-13T11:23:11.252 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:11.252 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:23:11.252 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:11.252 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:11.252 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:11.252 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:30852 3.a 3:522fb0d3:::benchmark_data_smithi177_31116_object30851:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e59)", 2024-09-13T11:23:11.252 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:09.538712+0000", 2024-09-13T11:23:11.252 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 1.6194171370000001, 2024-09-13T11:23:11.252 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.19661263300000001, 2024-09-13T11:23:11.252 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:23:11.252 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:23:11.253 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:11.253 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:23:11.253 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:23:11.253 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:11.253 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 30852 2024-09-13T11:23:11.253 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:11.253 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:23:11.253 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:11.253 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:23:11.253 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.538712+0000", 2024-09-13T11:23:11.253 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:11.253 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:11.254 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:11.254 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:23:11.254 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.538712+0000", 2024-09-13T11:23:11.254 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:11.254 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:11.254 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:11.254 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:23:11.254 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.538714+0000", 2024-09-13T11:23:11.254 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.658e-06 2024-09-13T11:23:11.254 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:11.254 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:11.254 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:23:11.254 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.538735+0000", 2024-09-13T11:23:11.254 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.141e-05 2024-09-13T11:23:11.255 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:11.255 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:11.255 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:23:11.255 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.538747+0000", 2024-09-13T11:23:11.255 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.2258e-05 2024-09-13T11:23:11.255 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:11.255 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:11.255 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:23:11.255 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.538754+0000", 2024-09-13T11:23:11.255 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.2859999999999996e-06 2024-09-13T11:23:11.255 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:11.255 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:11.255 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:23:11.255 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.538798+0000", 2024-09-13T11:23:11.256 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.4258000000000002e-05 2024-09-13T11:23:11.256 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:11.256 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:11.256 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:23:11.256 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.538890+0000", 2024-09-13T11:23:11.256 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.1676999999999997e-05 2024-09-13T11:23:11.256 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:11.256 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:11.256 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 7", 2024-09-13T11:23:11.256 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.538991+0000", 2024-09-13T11:23:11.256 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00010173700000000001 2024-09-13T11:23:11.256 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:11.256 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:11.257 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:23:11.257 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.540171+0000", 2024-09-13T11:23:11.257 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0011800479999999999 2024-09-13T11:23:11.257 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:11.257 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:11.257 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:11.257 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.735031+0000", 2024-09-13T11:23:11.257 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.194859798 2024-09-13T11:23:11.257 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:11.257 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:11.257 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:23:11.257 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.735197+0000", 2024-09-13T11:23:11.257 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00016569499999999999 2024-09-13T11:23:11.258 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:11.258 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:11.258 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:23:11.258 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.735325+0000", 2024-09-13T11:23:11.258 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00012780800000000001 2024-09-13T11:23:11.258 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:11.258 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:23:11.258 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:11.258 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:11.258 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:11.258 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop_reply(client.4665.0:30852 3.a e59/30 ondisk, result = 0)", 2024-09-13T11:23:11.258 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:09.540458+0000", 2024-09-13T11:23:11.258 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 1.617671649, 2024-09-13T11:23:11.259 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.194887, 2024-09-13T11:23:11.259 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:23:11.259 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:23:11.259 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:23:11.259 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:23:11.259 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:11.259 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:23:11.259 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.540458+0000", 2024-09-13T11:23:11.259 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:11.259 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:11.259 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:11.259 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:23:11.259 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.540458+0000", 2024-09-13T11:23:11.259 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:11.260 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:11.260 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:11.260 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:23:11.260 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.540459+0000", 2024-09-13T11:23:11.260 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.2699999999999999e-06 2024-09-13T11:23:11.260 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:11.260 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:11.260 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:23:11.260 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.540463+0000", 2024-09-13T11:23:11.260 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.8500000000000004e-06 2024-09-13T11:23:11.260 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:11.260 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:11.260 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:23:11.261 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.734813+0000", 2024-09-13T11:23:11.261 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.194350047 2024-09-13T11:23:11.261 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:11.261 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:11.261 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:23:11.261 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.734832+0000", 2024-09-13T11:23:11.261 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.9610999999999999e-05 2024-09-13T11:23:11.261 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:11.261 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:11.261 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:23:11.261 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.734989+0000", 2024-09-13T11:23:11.261 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00015694799999999999 2024-09-13T11:23:11.261 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:11.261 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:11.262 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:23:11.262 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.735015+0000", 2024-09-13T11:23:11.262 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.6165000000000001e-05 2024-09-13T11:23:11.262 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:11.262 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:11.262 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:23:11.262 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.735345+0000", 2024-09-13T11:23:11.262 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00032910900000000001 2024-09-13T11:23:11.262 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:11.262 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:23:11.262 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:11.262 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:11.262 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:11.263 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:31097 3.d e59/30 3:bc3dc2d6:::benchmark_data_smithi177_31116_object31096:head v 59'1908, mlcod=59'1904)", 2024-09-13T11:23:11.263 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:09.957883+0000", 2024-09-13T11:23:11.263 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 1.200246175, 2024-09-13T11:23:11.263 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.042855150000000002, 2024-09-13T11:23:11.263 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:23:11.263 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:23:11.263 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:11.263 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:23:11.263 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:11.263 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:23:11.263 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.957883+0000", 2024-09-13T11:23:11.263 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:11.263 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:11.264 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:11.264 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:23:11.264 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.957883+0000", 2024-09-13T11:23:11.264 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:11.264 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:11.264 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:11.264 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:23:11.264 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.957886+0000", 2024-09-13T11:23:11.264 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.0910000000000001e-06 2024-09-13T11:23:11.264 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:11.264 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:11.264 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:23:11.264 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.957937+0000", 2024-09-13T11:23:11.265 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.1233000000000002e-05 2024-09-13T11:23:11.265 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:11.265 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:11.265 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:23:11.265 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.997999+0000", 2024-09-13T11:23:11.265 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.040061306999999997 2024-09-13T11:23:11.265 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:11.265 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:11.265 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:23:11.265 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.998017+0000", 2024-09-13T11:23:11.265 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.8587999999999999e-05 2024-09-13T11:23:11.265 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:11.265 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:11.266 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:23:11.266 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.998151+0000", 2024-09-13T11:23:11.266 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000133781 2024-09-13T11:23:11.266 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:11.266 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:11.266 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:23:11.266 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.998188+0000", 2024-09-13T11:23:11.266 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.6915999999999997e-05 2024-09-13T11:23:11.266 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:11.266 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:11.266 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:23:11.266 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:10.000640+0000", 2024-09-13T11:23:11.266 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0024520459999999998 2024-09-13T11:23:11.266 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:11.267 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:11.267 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:23:11.267 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:10.000738+0000", 2024-09-13T11:23:11.267 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.8188000000000002e-05 2024-09-13T11:23:11.267 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:11.267 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:23:11.267 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:11.267 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:11.267 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:23:11.267 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-09-13T11:23:11.269 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-13T11:23:11.269 INFO:tasks.ceph.osd.15.smithi177.stderr:2024-09-13T11:23:11.266+0000 7f9d6ce77640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26578) UID: 0 2024-09-13T11:23:11.370 INFO:tasks.ceph.osd.3.smithi177.stderr:2024-09-13T11:23:11.366+0000 7f10adba4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26574) UID: 0 2024-09-13T11:23:11.429 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T11:23:11.455 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-09-13T11:23:11.455 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [ 2024-09-13T11:23:11.456 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:11.456 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:33464 3.d 3:b1628af6:::benchmark_data_smithi177_31116_object33463:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e60)", 2024-09-13T11:23:11.456 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:11.448688+0000", 2024-09-13T11:23:11.456 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 0.0075544100000000001, 2024-09-13T11:23:11.456 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0075751899999999999, 2024-09-13T11:23:11.456 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:23:11.456 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:23:11.456 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "waiting for sub ops", 2024-09-13T11:23:11.456 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:23:11.456 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:23:11.456 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:11.456 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 33464 2024-09-13T11:23:11.457 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:11.457 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:23:11.457 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:11.457 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:23:11.457 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:11.448688+0000", 2024-09-13T11:23:11.457 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:11.457 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:11.457 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:11.457 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:23:11.457 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:11.448688+0000", 2024-09-13T11:23:11.457 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:11.457 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:11.457 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:11.458 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:23:11.458 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:11.448690+0000", 2024-09-13T11:23:11.458 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.734e-06 2024-09-13T11:23:11.458 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:11.458 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:11.458 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:23:11.458 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:11.448713+0000", 2024-09-13T11:23:11.458 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.3091999999999998e-05 2024-09-13T11:23:11.458 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:11.458 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:11.458 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:23:11.458 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:11.448729+0000", 2024-09-13T11:23:11.458 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.6010000000000001e-05 2024-09-13T11:23:11.459 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:11.459 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:11.459 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:23:11.459 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:11.448737+0000", 2024-09-13T11:23:11.459 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.1969999999999999e-06 2024-09-13T11:23:11.459 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:11.459 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:11.459 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:23:11.459 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:11.448802+0000", 2024-09-13T11:23:11.459 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.4968000000000005e-05 2024-09-13T11:23:11.459 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:11.459 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:11.459 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:23:11.459 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:11.448954+0000", 2024-09-13T11:23:11.460 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00015227200000000001 2024-09-13T11:23:11.460 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:11.460 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:11.460 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 3", 2024-09-13T11:23:11.460 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:11.449108+0000", 2024-09-13T11:23:11.460 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00015447 2024-09-13T11:23:11.460 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:11.460 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:11.460 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:23:11.460 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:11.452081+0000", 2024-09-13T11:23:11.460 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00297263 2024-09-13T11:23:11.460 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:11.460 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:23:11.461 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:11.461 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:11.461 INFO:teuthology.orchestra.run.smithi177.stdout: ], 2024-09-13T11:23:11.461 INFO:teuthology.orchestra.run.smithi177.stdout: "num_ops": 1 2024-09-13T11:23:11.461 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-09-13T11:23:11.470 INFO:tasks.ceph.osd.14.smithi162.stderr:2024-09-13T11:23:11.466+0000 7f235c0db640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26319) UID: 0 2024-09-13T11:23:11.472 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-13T11:23:11.529 INFO:teuthology.orchestra.run.smithi001.stderr:set 2.2 pg_upmap mapping to [2,8] 2024-09-13T11:23:11.536 INFO:tasks.ceph.osd.2.smithi162.stderr:2024-09-13T11:23:11.530+0000 7fe91fd87640 -1 osd.2 60 handle_osd_map injecting map crc failure 2024-09-13T11:23:11.571 INFO:tasks.ceph.osd.12.smithi001.stderr:2024-09-13T11:23:11.564+0000 7f892c6e6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26314) UID: 0 2024-09-13T11:23:11.602 INFO:teuthology.orchestra.run.smithi001.stdout:osd.0: {} 2024-09-13T11:23:11.603 INFO:teuthology.orchestra.run.smithi001.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T11:23:11.632 INFO:teuthology.orchestra.run.smithi001.stdout:osd.1: {} 2024-09-13T11:23:11.633 INFO:teuthology.orchestra.run.smithi001.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T11:23:11.664 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-09-13T11:23:11.664 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [], 2024-09-13T11:23:11.664 INFO:teuthology.orchestra.run.smithi177.stdout: "complaint_time": 30, 2024-09-13T11:23:11.664 INFO:teuthology.orchestra.run.smithi177.stdout: "num_blocked_ops": 0 2024-09-13T11:23:11.664 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-09-13T11:23:11.665 INFO:teuthology.orchestra.run.smithi001.stdout:osd.2: {} 2024-09-13T11:23:11.665 INFO:teuthology.orchestra.run.smithi001.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T11:23:11.672 INFO:tasks.ceph.osd.13.smithi092.stderr:2024-09-13T11:23:11.666+0000 7f3e5afea640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26350) UID: 0 2024-09-13T11:23:11.683 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-13T11:23:11.691 INFO:teuthology.orchestra.run.smithi001.stdout:osd.3: {} 2024-09-13T11:23:11.691 INFO:teuthology.orchestra.run.smithi001.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T11:23:11.713 INFO:teuthology.orchestra.run.smithi001.stdout:osd.4: {} 2024-09-13T11:23:11.714 INFO:teuthology.orchestra.run.smithi001.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T11:23:11.738 INFO:teuthology.orchestra.run.smithi001.stdout:osd.5: {} 2024-09-13T11:23:11.738 INFO:teuthology.orchestra.run.smithi001.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T11:23:11.768 INFO:teuthology.orchestra.run.smithi001.stdout:osd.6: {} 2024-09-13T11:23:11.768 INFO:teuthology.orchestra.run.smithi001.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T11:23:11.773 INFO:tasks.ceph.osd.12.smithi001.stderr:2024-09-13T11:23:11.768+0000 7f892c6e6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26314) UID: 0 2024-09-13T11:23:11.802 INFO:teuthology.orchestra.run.smithi001.stdout:osd.7: {} 2024-09-13T11:23:11.830 INFO:teuthology.orchestra.run.smithi001.stdout:osd.8: {} 2024-09-13T11:23:11.830 INFO:teuthology.orchestra.run.smithi001.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-13T11:23:11.831 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-09-13T11:23:11.831 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:23:11.853 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-09-13T11:23:11.860 INFO:teuthology.orchestra.run.smithi001.stdout:osd.9: {} 2024-09-13T11:23:11.861 INFO:teuthology.orchestra.run.smithi001.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-13T11:23:11.874 INFO:tasks.ceph.osd.8.smithi001.stderr:2024-09-13T11:23:11.868+0000 7fb9b48ae640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26312) UID: 0 2024-09-13T11:23:11.885 INFO:teuthology.orchestra.run.smithi001.stdout:osd.10: {} 2024-09-13T11:23:11.885 INFO:teuthology.orchestra.run.smithi001.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-13T11:23:11.910 INFO:teuthology.orchestra.run.smithi001.stdout:osd.11: {} 2024-09-13T11:23:11.910 INFO:teuthology.orchestra.run.smithi001.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-13T11:23:11.938 INFO:teuthology.orchestra.run.smithi001.stdout:osd.12: {} 2024-09-13T11:23:11.938 INFO:teuthology.orchestra.run.smithi001.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-13T11:23:11.970 INFO:teuthology.orchestra.run.smithi001.stdout:osd.13: {} 2024-09-13T11:23:11.971 INFO:teuthology.orchestra.run.smithi001.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-13T11:23:11.975 INFO:tasks.ceph.osd.1.smithi092.stderr:2024-09-13T11:23:11.970+0000 7fe7789c5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33718) UID: 0 2024-09-13T11:23:11.990 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-09-13T11:23:11.991 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:23:11.998 INFO:teuthology.orchestra.run.smithi001.stdout:osd.14: {} 2024-09-13T11:23:11.998 INFO:teuthology.orchestra.run.smithi001.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-13T11:23:12.002 INFO:tasks.radosbench.radosbench.0.smithi177.stdout: 45 16 34321 34305 47.6402 99.875 0.00832341 0.0209812 2024-09-13T11:23:12.009 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-09-13T11:23:12.024 INFO:teuthology.orchestra.run.smithi001.stdout:osd.15: {} 2024-09-13T11:23:12.024 INFO:teuthology.orchestra.run.smithi001.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-13T11:23:12.076 INFO:tasks.ceph.osd.13.smithi092.stderr:2024-09-13T11:23:12.070+0000 7f3e5afea640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26350) UID: 0 2024-09-13T11:23:12.134 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-09-13T11:23:12.134 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:23:12.150 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-09-13T11:23:12.177 INFO:tasks.ceph.osd.2.smithi162.stderr:2024-09-13T11:23:12.174+0000 7fe92b35a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26313) UID: 0 2024-09-13T11:23:12.279 INFO:tasks.ceph.osd.11.smithi177.stderr:2024-09-13T11:23:12.278+0000 7f24792c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26580) UID: 0 2024-09-13T11:23:12.288 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-09-13T11:23:12.288 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:23:12.307 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_ops_in_flight 2024-09-13T11:23:12.380 INFO:tasks.ceph.osd.1.smithi092.stderr:2024-09-13T11:23:12.373+0000 7fe7789c5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33718) UID: 0 2024-09-13T11:23:12.481 INFO:tasks.ceph.osd.13.smithi092.stderr:2024-09-13T11:23:12.477+0000 7f3e5afea640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26350) UID: 0 2024-09-13T11:23:12.484 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-09-13T11:23:12.484 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:23:12.503 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_blocked_ops 2024-09-13T11:23:12.534 INFO:tasks.ceph.osd.8.smithi001.stderr:2024-09-13T11:23:12.528+0000 7fb9ab2df640 -1 osd.8 61 handle_osd_map injecting map crc failure 2024-09-13T11:23:12.583 INFO:tasks.ceph.osd.14.smithi162.stderr:2024-09-13T11:23:12.578+0000 7f235c0db640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26319) UID: 0 2024-09-13T11:23:12.632 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-09-13T11:23:12.633 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:23:12.637 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-13T11:23:12.647 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_historic_ops 2024-09-13T11:23:12.687 INFO:tasks.ceph.osd.4.smithi001.stderr:2024-09-13T11:23:12.680+0000 7ff8d0c3d640 -1 osd.4 61 handle_osd_map injecting map crc failure 2024-09-13T11:23:12.691 INFO:tasks.ceph.osd.7.smithi177.stderr:2024-09-13T11:23:12.690+0000 7f897b51d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26575) UID: 0 2024-09-13T11:23:12.784 INFO:tasks.ceph.osd.8.smithi001.stderr:2024-09-13T11:23:12.780+0000 7fb9b48ae640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26312) UID: 0 2024-09-13T11:23:12.788 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-09-13T11:23:12.788 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:23:12.806 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-13T11:23:12.885 INFO:tasks.ceph.osd.1.smithi092.stderr:2024-09-13T11:23:12.881+0000 7fe7789c5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33718) UID: 0 2024-09-13T11:23:12.962 INFO:teuthology.orchestra.run.smithi092.stdout:ERROR: (22) Invalid argument 2024-09-13T11:23:12.962 INFO:teuthology.orchestra.run.smithi092.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:23:12.985 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-13T11:23:12.986 INFO:tasks.ceph.osd.3.smithi177.stderr:2024-09-13T11:23:12.982+0000 7f10adba4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26574) UID: 0 2024-09-13T11:23:13.002 INFO:tasks.radosbench.radosbench.0.smithi177.stdout: 46 16 35812 35796 48.6302 93.1875 0.00881708 0.0205556 2024-09-13T11:23:13.090 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-13T11:23:13.084+0000 7f6a68775640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26307) UID: 0 2024-09-13T11:23:13.175 INFO:teuthology.orchestra.run.smithi092.stdout:ERROR: (22) Invalid argument 2024-09-13T11:23:13.175 INFO:teuthology.orchestra.run.smithi092.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:23:13.191 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-13T11:23:13.192 INFO:tasks.ceph.osd.14.smithi162.stderr:2024-09-13T11:23:13.186+0000 7f235c0db640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26319) UID: 0 2024-09-13T11:23:13.292 INFO:tasks.ceph.osd.12.smithi001.stderr:2024-09-13T11:23:13.284+0000 7f892c6e6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26314) UID: 0 2024-09-13T11:23:13.312 INFO:teuthology.orchestra.run.smithi092.stdout:ERROR: (22) Invalid argument 2024-09-13T11:23:13.312 INFO:teuthology.orchestra.run.smithi092.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:23:13.330 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-13T11:23:13.393 INFO:tasks.ceph.osd.15.smithi177.stderr:2024-09-13T11:23:13.390+0000 7f9d6ce77640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26578) UID: 0 2024-09-13T11:23:13.493 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-13T11:23:13.493 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:23:13.494 INFO:tasks.ceph.osd.5.smithi092.stderr:2024-09-13T11:23:13.489+0000 7f6f0b705640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26346) UID: 0 2024-09-13T11:23:13.517 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-13T11:23:13.537 INFO:teuthology.orchestra.run.smithi001.stderr:noscrub is set 2024-09-13T11:23:13.595 INFO:tasks.ceph.osd.13.smithi092.stderr:2024-09-13T11:23:13.589+0000 7f3e5afea640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26350) UID: 0 2024-09-13T11:23:13.682 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-13T11:23:13.682 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:23:13.696 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-09-13T11:23:13.690+0000 7f97befc9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26315) UID: 0 2024-09-13T11:23:13.704 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-13T11:23:13.796 INFO:tasks.ceph.osd.11.smithi177.stderr:2024-09-13T11:23:13.794+0000 7f24792c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26580) UID: 0 2024-09-13T11:23:13.867 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-13T11:23:13.868 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:23:13.890 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-13T11:23:13.897 INFO:tasks.ceph.osd.11.smithi177.stderr:2024-09-13T11:23:13.894+0000 7f24792c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26580) UID: 0 2024-09-13T11:23:13.998 INFO:tasks.ceph.osd.14.smithi162.stderr:2024-09-13T11:23:13.994+0000 7f235c0db640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26319) UID: 0 2024-09-13T11:23:14.002 INFO:tasks.radosbench.radosbench.0.smithi177.stdout: 47 16 37435 37419 49.7535 101.438 0.00807662 0.0200899 2024-09-13T11:23:14.043 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-13T11:23:14.043 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:23:14.045 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T11:23:14.063 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-13T11:23:14.099 INFO:tasks.ceph.osd.2.smithi162.stderr:2024-09-13T11:23:14.094+0000 7fe92b35a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26313) UID: 0 2024-09-13T11:23:14.200 INFO:tasks.ceph.osd.8.smithi001.stderr:2024-09-13T11:23:14.192+0000 7fb9b48ae640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26312) UID: 0 2024-09-13T11:23:14.225 INFO:teuthology.orchestra.run.smithi001.stdout:osd.0: {} 2024-09-13T11:23:14.226 INFO:teuthology.orchestra.run.smithi001.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T11:23:14.248 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-13T11:23:14.248 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:23:14.254 INFO:teuthology.orchestra.run.smithi001.stdout:osd.1: {} 2024-09-13T11:23:14.254 INFO:teuthology.orchestra.run.smithi001.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T11:23:14.267 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-13T11:23:14.286 INFO:teuthology.orchestra.run.smithi001.stdout:osd.2: {} 2024-09-13T11:23:14.286 INFO:teuthology.orchestra.run.smithi001.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T11:23:14.316 INFO:teuthology.orchestra.run.smithi001.stdout:osd.3: {} 2024-09-13T11:23:14.317 INFO:teuthology.orchestra.run.smithi001.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T11:23:14.317 INFO:tasks.ceph.osd.4.smithi001.stderr:2024-09-13T11:23:14.308+0000 7ff8e0218640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26310) UID: 0 2024-09-13T11:23:14.343 INFO:teuthology.orchestra.run.smithi001.stdout:osd.4: {} 2024-09-13T11:23:14.343 INFO:teuthology.orchestra.run.smithi001.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T11:23:14.381 INFO:teuthology.orchestra.run.smithi001.stdout:osd.5: {} 2024-09-13T11:23:14.381 INFO:teuthology.orchestra.run.smithi001.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T11:23:14.401 INFO:tasks.ceph.osd.8.smithi001.stderr:2024-09-13T11:23:14.396+0000 7fb9b48ae640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26312) UID: 0 2024-09-13T11:23:14.420 INFO:teuthology.orchestra.run.smithi001.stdout:osd.6: {} 2024-09-13T11:23:14.421 INFO:teuthology.orchestra.run.smithi001.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T11:23:14.434 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-13T11:23:14.434 INFO:teuthology.orchestra.run.smithi001.stdout: "size": 20, 2024-09-13T11:23:14.434 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 600, 2024-09-13T11:23:14.434 INFO:teuthology.orchestra.run.smithi001.stdout: "ops": [] 2024-09-13T11:23:14.435 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-13T11:23:14.453 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-09-13T11:23:14.545 INFO:teuthology.orchestra.run.smithi001.stdout:osd.7: {} 2024-09-13T11:23:14.545 INFO:teuthology.orchestra.run.smithi001.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T11:23:14.545 INFO:tasks.ceph.osd.12.smithi001.stderr:2024-09-13T11:23:14.540+0000 7f892c6e6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26314) UID: 0 2024-09-13T11:23:14.573 INFO:teuthology.orchestra.run.smithi001.stdout:osd.8: {} 2024-09-13T11:23:14.573 INFO:teuthology.orchestra.run.smithi001.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-13T11:23:14.599 INFO:teuthology.orchestra.run.smithi001.stdout:osd.9: {} 2024-09-13T11:23:14.600 INFO:teuthology.orchestra.run.smithi001.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-13T11:23:14.602 INFO:tasks.ceph.osd.12.smithi001.stderr:2024-09-13T11:23:14.596+0000 7f892c6e6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26314) UID: 0 2024-09-13T11:23:14.631 INFO:teuthology.orchestra.run.smithi001.stdout:osd.10: {} 2024-09-13T11:23:14.631 INFO:teuthology.orchestra.run.smithi001.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-13T11:23:14.639 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-13T11:23:14.639 INFO:teuthology.orchestra.run.smithi001.stdout: "ops": [ 2024-09-13T11:23:14.640 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:14.640 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:38319 3.e 3:7756459a:::benchmark_data_smithi177_31116_object38318:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e63)", 2024-09-13T11:23:14.640 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:14.633684+0000", 2024-09-13T11:23:14.640 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 0.0049795660000000004, 2024-09-13T11:23:14.640 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.005007671, 2024-09-13T11:23:14.640 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:23:14.640 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:23:14.640 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "waiting for sub ops", 2024-09-13T11:23:14.640 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:23:14.640 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:23:14.640 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:14.640 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 38319 2024-09-13T11:23:14.640 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:14.641 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:23:14.641 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:14.641 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:23:14.641 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:14.633684+0000", 2024-09-13T11:23:14.641 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:14.641 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:14.641 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:14.641 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:23:14.641 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:14.633684+0000", 2024-09-13T11:23:14.641 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:14.641 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:14.641 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:14.641 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:23:14.641 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:14.633686+0000", 2024-09-13T11:23:14.642 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.7910000000000001e-06 2024-09-13T11:23:14.642 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:14.642 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:14.642 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:23:14.642 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:14.633764+0000", 2024-09-13T11:23:14.642 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.8837000000000004e-05 2024-09-13T11:23:14.642 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:14.642 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:14.642 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:23:14.642 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:14.633775+0000", 2024-09-13T11:23:14.642 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0156999999999999e-05 2024-09-13T11:23:14.642 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:14.642 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:14.642 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:23:14.643 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:14.633782+0000", 2024-09-13T11:23:14.643 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.9190000000000002e-06 2024-09-13T11:23:14.643 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:14.643 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:14.643 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:14.643 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:14.633886+0000", 2024-09-13T11:23:14.643 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000104505 2024-09-13T11:23:14.643 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:14.643 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:14.643 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:23:14.643 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:14.634008+0000", 2024-09-13T11:23:14.643 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000121648 2024-09-13T11:23:14.644 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:14.644 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:14.644 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 0,1", 2024-09-13T11:23:14.644 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:14.634187+0000", 2024-09-13T11:23:14.644 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000179063 2024-09-13T11:23:14.644 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:14.644 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:14.644 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:23:14.644 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:14.637452+0000", 2024-09-13T11:23:14.644 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.003265511 2024-09-13T11:23:14.644 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:14.644 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:14.644 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:14.644 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:14.645 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:14.645 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:38323 3.e 3:7f6aa5db:::benchmark_data_smithi177_31116_object38322:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e63)", 2024-09-13T11:23:14.645 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:14.635054+0000", 2024-09-13T11:23:14.645 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 0.003609555, 2024-09-13T11:23:14.645 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0037287079999999998, 2024-09-13T11:23:14.645 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:23:14.645 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:23:14.645 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "waiting for sub ops", 2024-09-13T11:23:14.645 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:23:14.645 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:23:14.645 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:14.645 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 38323 2024-09-13T11:23:14.645 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:14.646 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:23:14.646 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:14.646 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:23:14.646 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:14.635054+0000", 2024-09-13T11:23:14.646 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:14.646 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:14.646 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:14.646 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:23:14.646 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:14.635054+0000", 2024-09-13T11:23:14.646 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:14.646 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:14.646 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:14.646 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:23:14.647 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:14.635055+0000", 2024-09-13T11:23:14.647 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2979999999999999e-06 2024-09-13T11:23:14.647 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:14.647 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:14.647 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:23:14.647 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:14.635124+0000", 2024-09-13T11:23:14.647 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.9128000000000006e-05 2024-09-13T11:23:14.647 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:14.647 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:14.647 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:23:14.647 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:14.635134+0000", 2024-09-13T11:23:14.647 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0128e-05 2024-09-13T11:23:14.647 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:14.647 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:14.648 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:23:14.648 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:14.635143+0000", 2024-09-13T11:23:14.648 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.7970000000000006e-06 2024-09-13T11:23:14.648 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:14.648 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:14.648 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:14.648 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:14.635318+0000", 2024-09-13T11:23:14.648 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00017499099999999999 2024-09-13T11:23:14.648 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:14.648 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:14.648 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:23:14.648 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:14.635470+0000", 2024-09-13T11:23:14.648 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000151972 2024-09-13T11:23:14.649 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:14.649 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:14.649 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 0,1", 2024-09-13T11:23:14.649 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:14.635632+0000", 2024-09-13T11:23:14.649 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00016208499999999999 2024-09-13T11:23:14.649 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:14.649 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:14.649 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:23:14.649 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:14.637622+0000", 2024-09-13T11:23:14.649 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0019897859999999999 2024-09-13T11:23:14.649 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:14.649 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:14.649 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:14.649 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:14.649 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:14.650 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:38326 3.e 3:7d6a3ebe:::benchmark_data_smithi177_31116_object38325:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e63)", 2024-09-13T11:23:14.650 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:14.637935+0000", 2024-09-13T11:23:14.650 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 0.00072871899999999996, 2024-09-13T11:23:14.650 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00092670399999999998, 2024-09-13T11:23:14.650 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:23:14.650 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:23:14.650 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:23:14.650 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:23:14.650 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:23:14.650 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:14.650 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 38326 2024-09-13T11:23:14.650 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:14.650 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:23:14.651 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:14.651 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:23:14.651 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:14.637935+0000", 2024-09-13T11:23:14.651 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:14.651 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:14.651 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:14.651 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:23:14.651 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:14.637935+0000", 2024-09-13T11:23:14.651 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:14.651 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:14.651 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:14.651 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:23:14.651 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:14.637937+0000", 2024-09-13T11:23:14.651 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.1260000000000001e-06 2024-09-13T11:23:14.652 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:14.652 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:14.652 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:23:14.652 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:14.637998+0000", 2024-09-13T11:23:14.652 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.1044999999999996e-05 2024-09-13T11:23:14.652 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:14.652 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:14.652 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:23:14.652 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:14.638011+0000", 2024-09-13T11:23:14.652 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.3519999999999999e-05 2024-09-13T11:23:14.652 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:14.652 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:14.652 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:23:14.652 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:14.638018+0000", 2024-09-13T11:23:14.653 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.9539999999999998e-06 2024-09-13T11:23:14.653 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:14.653 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:14.653 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:14.653 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:14.638476+0000", 2024-09-13T11:23:14.653 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00045728199999999999 2024-09-13T11:23:14.653 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:14.653 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:14.653 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:23:14.653 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:14.638674+0000", 2024-09-13T11:23:14.653 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00019850400000000001 2024-09-13T11:23:14.653 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:14.654 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:14.654 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 0,1", 2024-09-13T11:23:14.654 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:14.638865+0000", 2024-09-13T11:23:14.654 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00019075 2024-09-13T11:23:14.654 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:14.654 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:14.654 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:14.654 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:14.654 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-09-13T11:23:14.654 INFO:teuthology.orchestra.run.smithi001.stdout: "num_ops": 3 2024-09-13T11:23:14.654 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-13T11:23:14.658 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-09-13T11:23:14.661 INFO:teuthology.orchestra.run.smithi001.stdout:osd.11: {} 2024-09-13T11:23:14.662 INFO:teuthology.orchestra.run.smithi001.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-13T11:23:14.687 INFO:teuthology.orchestra.run.smithi001.stdout:osd.12: {} 2024-09-13T11:23:14.687 INFO:teuthology.orchestra.run.smithi001.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-13T11:23:14.703 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-13T11:23:14.696+0000 7f6a68775640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26307) UID: 0 2024-09-13T11:23:14.713 INFO:teuthology.orchestra.run.smithi001.stdout:osd.13: {} 2024-09-13T11:23:14.713 INFO:teuthology.orchestra.run.smithi001.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-13T11:23:14.754 INFO:teuthology.orchestra.run.smithi001.stdout:osd.14: {} 2024-09-13T11:23:14.755 INFO:teuthology.orchestra.run.smithi001.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-13T11:23:14.782 INFO:teuthology.orchestra.run.smithi001.stdout:osd.15: {} 2024-09-13T11:23:14.783 INFO:teuthology.orchestra.run.smithi001.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-13T11:23:14.804 INFO:tasks.ceph.osd.10.smithi162.stderr:2024-09-13T11:23:14.798+0000 7fb59c9f0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26317) UID: 0 2024-09-13T11:23:14.810 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-13T11:23:14.811 INFO:teuthology.orchestra.run.smithi001.stdout: "ops": [], 2024-09-13T11:23:14.811 INFO:teuthology.orchestra.run.smithi001.stdout: "complaint_time": 30, 2024-09-13T11:23:14.811 INFO:teuthology.orchestra.run.smithi001.stdout: "num_blocked_ops": 0 2024-09-13T11:23:14.811 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-13T11:23:14.824 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-09-13T11:23:14.912 INFO:tasks.ceph.osd.8.smithi001.stderr:2024-09-13T11:23:14.908+0000 7fb9b48ae640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26312) UID: 0 2024-09-13T11:23:14.991 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-13T11:23:14.992 INFO:teuthology.orchestra.run.smithi001.stdout: "size": 20, 2024-09-13T11:23:14.992 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 600, 2024-09-13T11:23:14.992 INFO:teuthology.orchestra.run.smithi001.stdout: "ops": [ 2024-09-13T11:23:14.992 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:14.992 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:9889 3.e 3:711155e6:::benchmark_data_smithi177_31116_object9888:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-09-13T11:23:14.992 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:22:54.281236+0000", 2024-09-13T11:23:14.992 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 20.707431383999999, 2024-09-13T11:23:14.992 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.054312457000000001, 2024-09-13T11:23:14.992 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:23:14.992 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:23:14.993 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:14.993 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:23:14.993 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:23:14.993 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:14.993 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 9889 2024-09-13T11:23:14.993 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:14.993 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:23:14.993 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:14.993 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:23:14.993 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:54.281236+0000", 2024-09-13T11:23:14.993 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:14.993 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:14.994 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:14.994 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:23:14.994 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:54.281236+0000", 2024-09-13T11:23:14.994 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:14.994 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:14.994 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:14.994 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:23:14.994 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:54.281239+0000", 2024-09-13T11:23:14.994 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.2139999999999999e-06 2024-09-13T11:23:14.994 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:14.994 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:14.994 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:23:14.994 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:54.281288+0000", 2024-09-13T11:23:14.995 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.9280000000000003e-05 2024-09-13T11:23:14.995 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:14.995 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:14.995 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:23:14.995 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:54.281309+0000", 2024-09-13T11:23:14.995 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.1461999999999999e-05 2024-09-13T11:23:14.995 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:14.995 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:14.995 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:23:14.995 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:54.281322+0000", 2024-09-13T11:23:14.995 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2557e-05 2024-09-13T11:23:14.995 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:14.996 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:14.996 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:14.996 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:54.281442+0000", 2024-09-13T11:23:14.996 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00012048300000000001 2024-09-13T11:23:14.996 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:14.996 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:14.996 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:23:14.996 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:54.281640+0000", 2024-09-13T11:23:14.996 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00019767 2024-09-13T11:23:14.996 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:14.996 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:14.996 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 14", 2024-09-13T11:23:14.997 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:54.281824+0000", 2024-09-13T11:23:14.997 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000183907 2024-09-13T11:23:14.997 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:14.997 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:14.997 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:23:14.997 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:54.283416+0000", 2024-09-13T11:23:14.997 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0015919339999999999 2024-09-13T11:23:14.997 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:14.997 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:14.997 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:14.997 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:54.335364+0000", 2024-09-13T11:23:14.997 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.051948558999999998 2024-09-13T11:23:14.998 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:14.998 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:14.998 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:23:14.998 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:54.335483+0000", 2024-09-13T11:23:14.998 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000118175 2024-09-13T11:23:14.998 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:14.998 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:14.998 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:23:14.998 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:54.335549+0000", 2024-09-13T11:23:14.999 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.6216000000000004e-05 2024-09-13T11:23:14.999 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:14.999 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:14.999 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:14.999 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:14.999 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:14.999 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop_reply(client.4665.0:9889 3.e e47/30 ondisk, result = 0)", 2024-09-13T11:23:14.999 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:22:54.284595+0000", 2024-09-13T11:23:14.999 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 20.704073284, 2024-09-13T11:23:15.000 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.050965249999999997, 2024-09-13T11:23:15.000 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:23:15.000 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:23:15.000 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:23:15.000 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:23:15.000 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.000 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:23:15.000 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:54.284595+0000", 2024-09-13T11:23:15.000 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:15.000 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.000 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.000 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:23:15.000 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:54.284595+0000", 2024-09-13T11:23:15.001 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:15.001 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.001 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.001 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:23:15.001 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:54.284604+0000", 2024-09-13T11:23:15.001 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.0529999999999996e-06 2024-09-13T11:23:15.001 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.001 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.001 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:23:15.001 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:54.284613+0000", 2024-09-13T11:23:15.001 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.3770000000000002e-06 2024-09-13T11:23:15.001 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.002 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.002 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:23:15.002 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:54.335182+0000", 2024-09-13T11:23:15.002 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.050569127999999998 2024-09-13T11:23:15.002 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.002 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.002 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:23:15.002 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:54.335195+0000", 2024-09-13T11:23:15.002 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.3179999999999999e-05 2024-09-13T11:23:15.002 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.002 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.002 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:15.002 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:54.335339+0000", 2024-09-13T11:23:15.003 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00014390099999999999 2024-09-13T11:23:15.003 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.003 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.003 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:23:15.003 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:54.335355+0000", 2024-09-13T11:23:15.003 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.6356999999999999e-05 2024-09-13T11:23:15.003 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.003 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.003 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:23:15.003 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:54.335560+0000", 2024-09-13T11:23:15.003 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00020425400000000001 2024-09-13T11:23:15.003 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:15.003 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:15.004 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:15.004 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.004 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.004 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:9959 3.e 3:77912156:::benchmark_data_smithi177_31116_object9958:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-09-13T11:23:15.004 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:22:54.356380+0000", 2024-09-13T11:23:15.004 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 20.632287945000002, 2024-09-13T11:23:15.004 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.060511429999999998, 2024-09-13T11:23:15.004 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:23:15.004 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:23:15.004 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:15.004 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:23:15.004 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:23:15.004 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:15.005 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 9959 2024-09-13T11:23:15.005 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.005 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:23:15.005 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.005 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:23:15.005 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:54.356380+0000", 2024-09-13T11:23:15.005 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:15.005 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.005 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.005 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:23:15.005 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:54.356380+0000", 2024-09-13T11:23:15.005 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:15.005 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.006 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.006 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:23:15.006 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:54.356384+0000", 2024-09-13T11:23:15.006 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.7230000000000002e-06 2024-09-13T11:23:15.006 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.007 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.007 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:23:15.007 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:54.356440+0000", 2024-09-13T11:23:15.007 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.6703000000000002e-05 2024-09-13T11:23:15.007 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.007 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.008 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:23:15.008 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:54.356472+0000", 2024-09-13T11:23:15.008 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.1888999999999999e-05 2024-09-13T11:23:15.008 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.008 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.008 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:23:15.008 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:54.356495+0000", 2024-09-13T11:23:15.008 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.3306999999999999e-05 2024-09-13T11:23:15.008 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.008 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.009 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:15.009 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:54.356687+0000", 2024-09-13T11:23:15.009 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00019194399999999999 2024-09-13T11:23:15.009 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.009 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.009 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:23:15.009 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:54.357065+0000", 2024-09-13T11:23:15.009 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00037750899999999999 2024-09-13T11:23:15.009 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.009 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.009 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 14", 2024-09-13T11:23:15.009 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:54.357623+0000", 2024-09-13T11:23:15.010 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00055840700000000002 2024-09-13T11:23:15.010 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.010 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.010 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:23:15.010 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:54.360103+0000", 2024-09-13T11:23:15.010 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0024792989999999999 2024-09-13T11:23:15.010 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.010 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.010 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:15.010 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:54.416745+0000", 2024-09-13T11:23:15.010 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.056641944999999999 2024-09-13T11:23:15.010 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.011 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.011 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:23:15.011 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:54.416828+0000", 2024-09-13T11:23:15.011 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.3275000000000005e-05 2024-09-13T11:23:15.011 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.011 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.011 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:23:15.011 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:54.416891+0000", 2024-09-13T11:23:15.011 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.3429000000000005e-05 2024-09-13T11:23:15.011 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:15.011 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:15.011 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:15.011 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.012 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.012 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop_reply(client.4665.0:9959 3.e e47/30 ondisk, result = 0)", 2024-09-13T11:23:15.012 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:22:54.360272+0000", 2024-09-13T11:23:15.012 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 20.628395736999998, 2024-09-13T11:23:15.012 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.056629291999999998, 2024-09-13T11:23:15.012 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:23:15.012 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:23:15.012 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:23:15.012 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:23:15.012 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.012 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:23:15.013 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:54.360272+0000", 2024-09-13T11:23:15.013 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:15.013 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.013 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.013 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:23:15.013 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:54.360272+0000", 2024-09-13T11:23:15.013 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:15.013 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.013 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.013 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:23:15.013 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:54.360276+0000", 2024-09-13T11:23:15.013 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.5829999999999998e-06 2024-09-13T11:23:15.013 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.014 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.014 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:23:15.014 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:54.360286+0000", 2024-09-13T11:23:15.014 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0434e-05 2024-09-13T11:23:15.014 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.014 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.014 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:23:15.014 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:54.416602+0000", 2024-09-13T11:23:15.014 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.056316135000000003 2024-09-13T11:23:15.014 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.014 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.014 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:23:15.014 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:54.416614+0000", 2024-09-13T11:23:15.014 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2289e-05 2024-09-13T11:23:15.015 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.015 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.015 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:15.015 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:54.416721+0000", 2024-09-13T11:23:15.015 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000106193 2024-09-13T11:23:15.015 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.015 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.015 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:23:15.015 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:54.416736+0000", 2024-09-13T11:23:15.015 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.5216e-05 2024-09-13T11:23:15.015 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.015 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.015 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:23:15.016 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:54.416901+0000", 2024-09-13T11:23:15.016 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00016544199999999999 2024-09-13T11:23:15.016 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:15.016 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:15.016 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:15.016 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.016 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.016 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:9962 3.e 3:7450abb7:::benchmark_data_smithi177_31116_object9961:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e48)", 2024-09-13T11:23:15.016 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:22:54.360795+0000", 2024-09-13T11:23:15.016 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 20.627873229999999, 2024-09-13T11:23:15.016 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.061794618000000003, 2024-09-13T11:23:15.016 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:23:15.016 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:23:15.017 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:15.017 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:23:15.017 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:23:15.017 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:15.017 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 9962 2024-09-13T11:23:15.017 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.017 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:23:15.017 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.017 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:23:15.017 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:54.360795+0000", 2024-09-13T11:23:15.017 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:15.017 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.017 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.017 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:23:15.018 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:54.360795+0000", 2024-09-13T11:23:15.018 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:15.018 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.018 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.018 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:23:15.018 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:54.360799+0000", 2024-09-13T11:23:15.018 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.6539999999999998e-06 2024-09-13T11:23:15.018 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.018 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.018 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:23:15.018 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:54.360894+0000", 2024-09-13T11:23:15.018 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.4602e-05 2024-09-13T11:23:15.019 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.019 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.019 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:23:15.019 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:54.360932+0000", 2024-09-13T11:23:15.019 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.8299e-05 2024-09-13T11:23:15.019 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.019 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.019 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:23:15.019 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:54.360956+0000", 2024-09-13T11:23:15.019 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.4369999999999999e-05 2024-09-13T11:23:15.019 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.019 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.019 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:15.019 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:54.361062+0000", 2024-09-13T11:23:15.020 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00010568700000000001 2024-09-13T11:23:15.020 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.020 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.020 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op must wait for map", 2024-09-13T11:23:15.020 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:54.361068+0000", 2024-09-13T11:23:15.020 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.9689999999999999e-06 2024-09-13T11:23:15.020 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.020 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.020 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:15.020 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:54.363985+0000", 2024-09-13T11:23:15.020 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0029172820000000002 2024-09-13T11:23:15.020 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.020 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.021 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:23:15.021 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:54.364160+0000", 2024-09-13T11:23:15.021 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00017481800000000001 2024-09-13T11:23:15.021 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.021 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.021 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 14", 2024-09-13T11:23:15.021 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:54.364326+0000", 2024-09-13T11:23:15.021 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00016597199999999999 2024-09-13T11:23:15.021 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.021 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.022 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:23:15.022 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:54.365863+0000", 2024-09-13T11:23:15.022 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0015368599999999999 2024-09-13T11:23:15.022 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.022 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.022 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:15.022 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:54.422496+0000", 2024-09-13T11:23:15.022 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.056632770999999998 2024-09-13T11:23:15.022 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.022 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.022 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:23:15.022 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:54.422544+0000", 2024-09-13T11:23:15.022 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.8000000000000001e-05 2024-09-13T11:23:15.023 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.023 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.023 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:23:15.023 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:54.422589+0000", 2024-09-13T11:23:15.023 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.5334000000000003e-05 2024-09-13T11:23:15.023 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:15.023 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:15.023 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:15.023 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.023 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.023 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop_reply(client.4665.0:9962 3.e e48/30 ondisk, result = 0)", 2024-09-13T11:23:15.023 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:22:54.367440+0000", 2024-09-13T11:23:15.023 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 20.621228124000002, 2024-09-13T11:23:15.024 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.055157683999999998, 2024-09-13T11:23:15.024 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:23:15.024 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:23:15.024 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:23:15.024 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:23:15.024 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.024 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:23:15.024 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:54.367440+0000", 2024-09-13T11:23:15.024 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:15.024 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.024 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.024 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:23:15.024 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:54.367440+0000", 2024-09-13T11:23:15.025 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:15.025 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.025 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.025 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:23:15.025 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:54.367443+0000", 2024-09-13T11:23:15.025 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.0520000000000002e-06 2024-09-13T11:23:15.025 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.025 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.025 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:23:15.025 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:54.367452+0000", 2024-09-13T11:23:15.025 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.0329999999999997e-06 2024-09-13T11:23:15.025 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.025 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.026 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:23:15.026 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:54.422418+0000", 2024-09-13T11:23:15.026 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.054965950999999999 2024-09-13T11:23:15.026 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.026 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.026 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:23:15.026 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:54.422428+0000", 2024-09-13T11:23:15.026 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.8439999999999993e-06 2024-09-13T11:23:15.026 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.026 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.026 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:15.026 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:54.422485+0000", 2024-09-13T11:23:15.026 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.7391e-05 2024-09-13T11:23:15.026 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.027 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.027 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:23:15.027 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:54.422492+0000", 2024-09-13T11:23:15.027 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.6069999999999996e-06 2024-09-13T11:23:15.027 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.027 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.027 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:23:15.027 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:54.422597+0000", 2024-09-13T11:23:15.027 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000105806 2024-09-13T11:23:15.027 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:15.027 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:15.027 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:15.027 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.027 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.028 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:17137 3.e 3:73c6ddc2:::benchmark_data_smithi177_31116_object17136:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-13T11:23:15.028 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:22:59.244270+0000", 2024-09-13T11:23:15.028 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 15.744397881999999, 2024-09-13T11:23:15.028 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.060039742, 2024-09-13T11:23:15.028 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:23:15.028 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:23:15.028 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:15.028 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:23:15.028 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:23:15.028 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:15.028 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 17137 2024-09-13T11:23:15.028 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.029 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:23:15.029 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.029 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:23:15.029 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:59.244270+0000", 2024-09-13T11:23:15.029 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:15.029 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.029 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.029 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:23:15.029 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:59.244270+0000", 2024-09-13T11:23:15.029 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:15.029 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.029 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.029 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:23:15.030 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:59.244282+0000", 2024-09-13T11:23:15.030 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2225000000000001e-05 2024-09-13T11:23:15.030 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.030 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.030 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:23:15.030 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:59.244356+0000", 2024-09-13T11:23:15.030 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.3837000000000005e-05 2024-09-13T11:23:15.030 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.030 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.031 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:23:15.031 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:59.244396+0000", 2024-09-13T11:23:15.031 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.0494000000000002e-05 2024-09-13T11:23:15.031 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.031 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.031 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:23:15.031 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:59.244410+0000", 2024-09-13T11:23:15.031 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.3403e-05 2024-09-13T11:23:15.031 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.031 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.031 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:15.031 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:59.244575+0000", 2024-09-13T11:23:15.031 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00016559499999999999 2024-09-13T11:23:15.032 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.032 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.032 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:23:15.032 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:59.244814+0000", 2024-09-13T11:23:15.032 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00023877499999999999 2024-09-13T11:23:15.032 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.032 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.032 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 14", 2024-09-13T11:23:15.032 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:59.245036+0000", 2024-09-13T11:23:15.032 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00022129499999999999 2024-09-13T11:23:15.032 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.032 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.032 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:23:15.033 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:59.247349+0000", 2024-09-13T11:23:15.033 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0023137029999999999 2024-09-13T11:23:15.033 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.033 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.033 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:15.033 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:59.304151+0000", 2024-09-13T11:23:15.033 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.056801429000000001 2024-09-13T11:23:15.033 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.033 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.033 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:23:15.034 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:59.304233+0000", 2024-09-13T11:23:15.034 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.2066999999999999e-05 2024-09-13T11:23:15.034 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.034 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.034 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:23:15.034 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:59.304310+0000", 2024-09-13T11:23:15.034 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.6919e-05 2024-09-13T11:23:15.034 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:15.034 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:15.034 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:15.034 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.035 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.035 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop_reply(client.4665.0:17137 3.e e52/30 ondisk, result = 0)", 2024-09-13T11:23:15.035 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:22:59.248231+0000", 2024-09-13T11:23:15.035 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 15.740436773000001, 2024-09-13T11:23:15.035 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.056092270999999999, 2024-09-13T11:23:15.035 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:23:15.035 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:23:15.035 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:23:15.035 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:23:15.035 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.035 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:23:15.035 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:59.248231+0000", 2024-09-13T11:23:15.035 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:15.036 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.036 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.036 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:23:15.036 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:59.248231+0000", 2024-09-13T11:23:15.036 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:15.036 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.036 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.036 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:23:15.036 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:59.248234+0000", 2024-09-13T11:23:15.036 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.1870000000000001e-06 2024-09-13T11:23:15.036 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.036 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.036 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:23:15.036 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:59.248246+0000", 2024-09-13T11:23:15.037 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.1620000000000001e-05 2024-09-13T11:23:15.037 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.037 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.037 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:23:15.037 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:59.304048+0000", 2024-09-13T11:23:15.037 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.055802233999999999 2024-09-13T11:23:15.037 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.037 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.037 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:23:15.037 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:59.304062+0000", 2024-09-13T11:23:15.037 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.3570000000000001e-05 2024-09-13T11:23:15.037 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.037 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.037 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:15.038 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:59.304138+0000", 2024-09-13T11:23:15.038 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.6159e-05 2024-09-13T11:23:15.038 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.038 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.038 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:23:15.038 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:59.304146+0000", 2024-09-13T11:23:15.038 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.7279999999999995e-06 2024-09-13T11:23:15.038 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.038 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.038 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:23:15.038 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:59.304323+0000", 2024-09-13T11:23:15.038 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00017777299999999999 2024-09-13T11:23:15.038 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:15.039 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:15.039 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:15.039 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.039 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.039 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4665.0:17772 3.7 e53/30 3:efa118de:::benchmark_data_smithi177_31116_object17771:head v 53'1070, mlcod=53'1067)", 2024-09-13T11:23:15.039 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:22:59.605894+0000", 2024-09-13T11:23:15.039 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 15.382773884000001, 2024-09-13T11:23:15.039 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.088358904000000002, 2024-09-13T11:23:15.039 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:23:15.039 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:23:15.039 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:15.039 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:23:15.039 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.039 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:23:15.040 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:59.605894+0000", 2024-09-13T11:23:15.040 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:15.040 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.040 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.040 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:23:15.041 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:59.605894+0000", 2024-09-13T11:23:15.041 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:15.041 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.041 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.041 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:23:15.041 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:59.605895+0000", 2024-09-13T11:23:15.041 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2589999999999999e-06 2024-09-13T11:23:15.041 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.041 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.041 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:23:15.041 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:59.605947+0000", 2024-09-13T11:23:15.041 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.1788e-05 2024-09-13T11:23:15.041 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.041 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.042 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:23:15.042 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:59.692920+0000", 2024-09-13T11:23:15.042 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.086972663000000006 2024-09-13T11:23:15.042 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.042 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.042 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:23:15.042 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:59.692926+0000", 2024-09-13T11:23:15.042 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.0870000000000003e-06 2024-09-13T11:23:15.042 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.042 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.042 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:15.042 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:59.692979+0000", 2024-09-13T11:23:15.042 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.3060999999999999e-05 2024-09-13T11:23:15.042 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.043 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.043 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:23:15.043 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:59.692993+0000", 2024-09-13T11:23:15.043 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.3753e-05 2024-09-13T11:23:15.043 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.043 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.043 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:23:15.043 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:59.694204+0000", 2024-09-13T11:23:15.043 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.001211571 2024-09-13T11:23:15.043 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.043 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.043 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:23:15.043 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:59.694253+0000", 2024-09-13T11:23:15.043 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.8721999999999999e-05 2024-09-13T11:23:15.044 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:15.044 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:15.044 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:15.044 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.044 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.044 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4665.0:25336 3.7 e55/30 3:e5532806:::benchmark_data_smithi177_31116_object25335:head v 55'1529, mlcod=55'1524)", 2024-09-13T11:23:15.044 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:04.385742+0000", 2024-09-13T11:23:15.044 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 10.602925669999999, 2024-09-13T11:23:15.044 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.051148206000000002, 2024-09-13T11:23:15.044 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:23:15.044 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:23:15.044 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:15.044 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:23:15.044 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.045 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:23:15.045 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.385742+0000", 2024-09-13T11:23:15.045 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:15.045 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.045 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.045 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:23:15.045 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.385742+0000", 2024-09-13T11:23:15.045 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:15.045 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.045 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.045 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:23:15.045 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.385744+0000", 2024-09-13T11:23:15.045 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.857e-06 2024-09-13T11:23:15.045 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.046 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.046 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:23:15.046 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.385776+0000", 2024-09-13T11:23:15.046 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.2459000000000002e-05 2024-09-13T11:23:15.046 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.046 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.046 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:23:15.046 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.434669+0000", 2024-09-13T11:23:15.046 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.048892113000000001 2024-09-13T11:23:15.046 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.046 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.046 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:23:15.046 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.434684+0000", 2024-09-13T11:23:15.047 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.5596000000000001e-05 2024-09-13T11:23:15.047 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.047 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.047 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:15.047 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.434794+0000", 2024-09-13T11:23:15.047 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000109819 2024-09-13T11:23:15.047 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.047 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.047 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:23:15.047 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.434827+0000", 2024-09-13T11:23:15.047 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.3435000000000001e-05 2024-09-13T11:23:15.047 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.047 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.047 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:23:15.048 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.436840+0000", 2024-09-13T11:23:15.048 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0020130220000000002 2024-09-13T11:23:15.048 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.048 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.048 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:23:15.048 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.436890+0000", 2024-09-13T11:23:15.048 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.9904999999999997e-05 2024-09-13T11:23:15.048 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:15.048 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:15.048 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:15.048 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.048 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.048 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4665.0:25388 3.7 e55/30 3:e5933cef:::benchmark_data_smithi177_31116_object25387:head v 55'1533, mlcod=55'1528)", 2024-09-13T11:23:15.048 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:04.456355+0000", 2024-09-13T11:23:15.049 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 10.532312366999999, 2024-09-13T11:23:15.049 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.22062032300000001, 2024-09-13T11:23:15.049 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:23:15.049 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:23:15.049 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:15.049 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:23:15.049 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.049 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:23:15.049 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.456355+0000", 2024-09-13T11:23:15.049 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:15.049 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.049 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.049 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:23:15.049 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.456355+0000", 2024-09-13T11:23:15.050 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:15.050 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.050 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.050 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:23:15.050 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.456357+0000", 2024-09-13T11:23:15.050 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.945e-06 2024-09-13T11:23:15.050 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.050 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.050 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:23:15.050 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.456398+0000", 2024-09-13T11:23:15.050 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.0423999999999998e-05 2024-09-13T11:23:15.050 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.050 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.050 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:23:15.051 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.675229+0000", 2024-09-13T11:23:15.051 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.21883077000000001 2024-09-13T11:23:15.051 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.051 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.051 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:23:15.051 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.675240+0000", 2024-09-13T11:23:15.051 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.1817999999999999e-05 2024-09-13T11:23:15.051 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.051 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.051 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:15.051 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.675393+0000", 2024-09-13T11:23:15.051 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000152476 2024-09-13T11:23:15.051 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.051 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.052 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:23:15.052 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.675416+0000", 2024-09-13T11:23:15.052 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.2934000000000001e-05 2024-09-13T11:23:15.052 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.052 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.052 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:23:15.052 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.676931+0000", 2024-09-13T11:23:15.052 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.001514835 2024-09-13T11:23:15.052 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.052 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.052 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:23:15.052 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.676976+0000", 2024-09-13T11:23:15.052 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.5120999999999997e-05 2024-09-13T11:23:15.052 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:15.053 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:15.053 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:15.053 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.053 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.053 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4665.0:25431 3.7 e56/30 3:e2b1f5d7:::benchmark_data_smithi177_31116_object25430:head v 56'1536, mlcod=55'1532)", 2024-09-13T11:23:15.053 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:04.715725+0000", 2024-09-13T11:23:15.053 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 10.272942814, 2024-09-13T11:23:15.053 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.15174399699999999, 2024-09-13T11:23:15.053 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:23:15.053 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:23:15.053 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:15.053 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:23:15.054 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.054 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:23:15.054 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.715725+0000", 2024-09-13T11:23:15.054 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:15.054 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.054 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.054 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:23:15.054 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.715725+0000", 2024-09-13T11:23:15.054 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:15.054 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.054 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.054 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:23:15.054 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.715728+0000", 2024-09-13T11:23:15.055 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.92e-06 2024-09-13T11:23:15.055 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.055 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.055 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:23:15.055 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.715772+0000", 2024-09-13T11:23:15.055 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.4141000000000003e-05 2024-09-13T11:23:15.055 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.055 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.055 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:23:15.055 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.865148+0000", 2024-09-13T11:23:15.055 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.149376012 2024-09-13T11:23:15.055 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.055 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.055 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:23:15.056 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.865163+0000", 2024-09-13T11:23:15.056 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.5106000000000001e-05 2024-09-13T11:23:15.056 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.056 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.056 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:15.056 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.865341+0000", 2024-09-13T11:23:15.056 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000178269 2024-09-13T11:23:15.056 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.056 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.056 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:23:15.056 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.865380+0000", 2024-09-13T11:23:15.056 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.8432000000000001e-05 2024-09-13T11:23:15.056 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.056 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.057 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:23:15.057 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.867412+0000", 2024-09-13T11:23:15.057 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0020325830000000001 2024-09-13T11:23:15.057 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.057 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.057 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:23:15.057 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.867469+0000", 2024-09-13T11:23:15.057 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.6533999999999998e-05 2024-09-13T11:23:15.057 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:15.057 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:15.057 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:15.057 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.057 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.057 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4665.0:25486 3.7 e56/30 3:e7acd6c0:::benchmark_data_smithi177_31116_object25485:head v 56'1537, mlcod=55'1533)", 2024-09-13T11:23:15.058 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:04.865042+0000", 2024-09-13T11:23:15.058 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 10.123626073000001, 2024-09-13T11:23:15.058 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.060232964, 2024-09-13T11:23:15.058 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:23:15.058 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:23:15.058 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:15.058 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:23:15.058 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.058 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:23:15.058 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.865042+0000", 2024-09-13T11:23:15.058 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:15.058 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.058 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.058 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:23:15.058 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.865042+0000", 2024-09-13T11:23:15.059 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:15.059 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.059 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.059 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:23:15.059 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.865046+0000", 2024-09-13T11:23:15.059 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.1230000000000002e-06 2024-09-13T11:23:15.059 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.059 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.059 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:23:15.059 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.865103+0000", 2024-09-13T11:23:15.059 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.7321999999999998e-05 2024-09-13T11:23:15.059 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.059 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.060 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:23:15.060 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.922874+0000", 2024-09-13T11:23:15.060 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.057771050999999997 2024-09-13T11:23:15.060 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.060 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.060 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:23:15.060 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.922884+0000", 2024-09-13T11:23:15.060 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.6169999999999996e-06 2024-09-13T11:23:15.060 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.060 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.060 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:15.060 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.922979+0000", 2024-09-13T11:23:15.060 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.5058999999999999e-05 2024-09-13T11:23:15.061 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.061 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.061 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:23:15.061 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.923040+0000", 2024-09-13T11:23:15.061 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.1396999999999995e-05 2024-09-13T11:23:15.061 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.061 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.061 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:23:15.061 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.925196+0000", 2024-09-13T11:23:15.061 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.002155952 2024-09-13T11:23:15.061 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.061 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.061 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:23:15.062 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.925275+0000", 2024-09-13T11:23:15.062 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.8442999999999994e-05 2024-09-13T11:23:15.062 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:15.062 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:15.062 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:15.062 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.062 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.062 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDPGPushReply(3.e 59/57 [PushReplyOp(3:7269999f:::benchmark_data_smithi177_31116_object2401:head)])", 2024-09-13T11:23:15.062 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:09.179472+0000", 2024-09-13T11:23:15.062 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 5.8091956549999999, 2024-09-13T11:23:15.062 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.062501552000000002, 2024-09-13T11:23:15.062 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:23:15.062 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:23:15.062 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-13T11:23:15.063 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:23:15.063 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.063 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:23:15.063 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.179472+0000", 2024-09-13T11:23:15.063 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:15.063 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.063 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.063 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:23:15.063 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.179472+0000", 2024-09-13T11:23:15.063 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:15.063 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.063 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.063 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:23:15.063 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.179474+0000", 2024-09-13T11:23:15.064 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.6300000000000001e-06 2024-09-13T11:23:15.064 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.064 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.064 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:23:15.064 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.179480+0000", 2024-09-13T11:23:15.064 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.2990000000000002e-06 2024-09-13T11:23:15.064 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.064 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.064 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:23:15.064 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.240453+0000", 2024-09-13T11:23:15.064 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.060972814 2024-09-13T11:23:15.064 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.064 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.064 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:23:15.065 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.240459+0000", 2024-09-13T11:23:15.065 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.3779999999999998e-06 2024-09-13T11:23:15.065 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.065 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.065 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:15.065 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.240527+0000", 2024-09-13T11:23:15.065 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.7664999999999997e-05 2024-09-13T11:23:15.065 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.065 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.065 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:23:15.065 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.241974+0000", 2024-09-13T11:23:15.065 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0014467659999999999 2024-09-13T11:23:15.065 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:15.065 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:15.066 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:15.066 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.066 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.066 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:30779 3.e 3:77c07d05:::benchmark_data_smithi177_31116_object30778:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e59)", 2024-09-13T11:23:15.066 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:09.473906+0000", 2024-09-13T11:23:15.066 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 5.5147620750000002, 2024-09-13T11:23:15.066 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.224949224, 2024-09-13T11:23:15.066 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:23:15.066 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:23:15.066 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:15.066 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:23:15.066 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:23:15.066 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:15.066 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 30779 2024-09-13T11:23:15.067 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.067 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:23:15.067 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.067 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:23:15.067 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.473906+0000", 2024-09-13T11:23:15.067 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:15.067 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.067 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.067 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:23:15.067 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.473906+0000", 2024-09-13T11:23:15.067 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:15.067 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.067 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.067 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:23:15.068 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.473908+0000", 2024-09-13T11:23:15.068 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.6230000000000001e-06 2024-09-13T11:23:15.068 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.068 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.068 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:23:15.068 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.473961+0000", 2024-09-13T11:23:15.068 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.2374999999999999e-05 2024-09-13T11:23:15.068 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.068 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.068 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:23:15.068 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.473976+0000", 2024-09-13T11:23:15.068 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.56e-05 2024-09-13T11:23:15.069 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.069 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.069 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:23:15.069 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.473987+0000", 2024-09-13T11:23:15.069 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0394000000000001e-05 2024-09-13T11:23:15.069 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.069 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.069 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:15.069 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.474275+0000", 2024-09-13T11:23:15.069 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00028800800000000002 2024-09-13T11:23:15.069 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.070 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.070 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:23:15.070 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.474466+0000", 2024-09-13T11:23:15.070 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000191499 2024-09-13T11:23:15.070 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.070 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.070 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 0,1", 2024-09-13T11:23:15.070 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.474666+0000", 2024-09-13T11:23:15.070 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00019994899999999999 2024-09-13T11:23:15.070 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.070 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.070 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:15.070 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.478354+0000", 2024-09-13T11:23:15.071 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0036876779999999998 2024-09-13T11:23:15.071 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.071 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.071 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:23:15.071 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.483128+0000", 2024-09-13T11:23:15.071 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0047739410000000003 2024-09-13T11:23:15.071 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.071 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.071 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:15.071 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.697695+0000", 2024-09-13T11:23:15.071 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.214566955 2024-09-13T11:23:15.071 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.071 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.071 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:23:15.072 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.697786+0000", 2024-09-13T11:23:15.072 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.1310000000000005e-05 2024-09-13T11:23:15.072 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.072 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.072 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:23:15.072 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.698855+0000", 2024-09-13T11:23:15.072 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0010688920000000001 2024-09-13T11:23:15.072 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:15.072 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:15.072 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:15.072 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.072 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.072 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:30785 3.e 3:7c135e1c:::benchmark_data_smithi177_31116_object30784:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e59)", 2024-09-13T11:23:15.072 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:09.483451+0000", 2024-09-13T11:23:15.073 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 5.5052166199999997, 2024-09-13T11:23:15.073 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.223624553, 2024-09-13T11:23:15.073 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:23:15.073 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:23:15.073 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:15.073 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:23:15.073 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:23:15.073 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:15.073 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 30785 2024-09-13T11:23:15.073 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.073 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:23:15.073 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.073 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:23:15.073 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.483451+0000", 2024-09-13T11:23:15.073 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:15.074 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.074 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.074 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:23:15.074 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.483451+0000", 2024-09-13T11:23:15.074 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:15.074 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.074 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.074 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:23:15.074 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.483453+0000", 2024-09-13T11:23:15.074 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.0530000000000001e-06 2024-09-13T11:23:15.074 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.074 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.074 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:23:15.075 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.483484+0000", 2024-09-13T11:23:15.075 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.1163e-05 2024-09-13T11:23:15.075 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.075 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.075 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:23:15.075 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.483495+0000", 2024-09-13T11:23:15.075 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0202e-05 2024-09-13T11:23:15.075 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.075 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.075 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:23:15.075 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.483501+0000", 2024-09-13T11:23:15.075 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.3169999999999998e-06 2024-09-13T11:23:15.075 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.075 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.076 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:15.076 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.483562+0000", 2024-09-13T11:23:15.076 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.1465999999999997e-05 2024-09-13T11:23:15.076 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.076 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.076 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:23:15.076 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.483699+0000", 2024-09-13T11:23:15.076 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000136293 2024-09-13T11:23:15.076 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.076 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.076 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 0,1", 2024-09-13T11:23:15.076 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.483819+0000", 2024-09-13T11:23:15.076 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000119972 2024-09-13T11:23:15.076 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.077 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.077 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:23:15.077 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.485318+0000", 2024-09-13T11:23:15.077 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0014994660000000001 2024-09-13T11:23:15.077 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.077 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.077 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:15.077 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.488237+0000", 2024-09-13T11:23:15.077 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0029186699999999999 2024-09-13T11:23:15.077 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.077 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.077 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:15.077 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.706356+0000", 2024-09-13T11:23:15.078 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.218119067 2024-09-13T11:23:15.078 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.078 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.078 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:23:15.078 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.706398+0000", 2024-09-13T11:23:15.078 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.2156000000000001e-05 2024-09-13T11:23:15.078 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.078 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.078 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:23:15.078 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.707076+0000", 2024-09-13T11:23:15.078 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00067772799999999995 2024-09-13T11:23:15.078 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:15.078 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:15.078 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:15.079 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.079 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.079 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:30801 3.e 3:73b0f21a:::benchmark_data_smithi177_31116_object30800:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e59)", 2024-09-13T11:23:15.079 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:09.485629+0000", 2024-09-13T11:23:15.079 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 5.5030392729999997, 2024-09-13T11:23:15.079 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.22604919900000001, 2024-09-13T11:23:15.079 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:23:15.079 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:23:15.079 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:15.079 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:23:15.079 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:23:15.079 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:15.079 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 30801 2024-09-13T11:23:15.079 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.080 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:23:15.080 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.080 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:23:15.080 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.485629+0000", 2024-09-13T11:23:15.080 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:15.080 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.080 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.080 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:23:15.080 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.485629+0000", 2024-09-13T11:23:15.080 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:15.080 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.080 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.080 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:23:15.080 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.485631+0000", 2024-09-13T11:23:15.081 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.9989999999999998e-06 2024-09-13T11:23:15.081 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.081 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.081 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:23:15.081 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.485664+0000", 2024-09-13T11:23:15.081 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.3896000000000002e-05 2024-09-13T11:23:15.081 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.081 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.081 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:23:15.081 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.485679+0000", 2024-09-13T11:23:15.081 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.4438e-05 2024-09-13T11:23:15.081 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.081 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.082 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:23:15.082 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.485688+0000", 2024-09-13T11:23:15.082 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.0180000000000007e-06 2024-09-13T11:23:15.082 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.082 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.082 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:15.082 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.485742+0000", 2024-09-13T11:23:15.082 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.4026000000000001e-05 2024-09-13T11:23:15.082 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.082 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.082 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:23:15.082 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.485841+0000", 2024-09-13T11:23:15.082 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.8875000000000005e-05 2024-09-13T11:23:15.082 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.083 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.083 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 0,1", 2024-09-13T11:23:15.083 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.486005+0000", 2024-09-13T11:23:15.083 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000163747 2024-09-13T11:23:15.083 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.083 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.083 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:23:15.083 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.487867+0000", 2024-09-13T11:23:15.083 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.001862945 2024-09-13T11:23:15.083 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.083 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.083 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:15.084 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.488570+0000", 2024-09-13T11:23:15.084 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00070255700000000003 2024-09-13T11:23:15.084 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.084 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.084 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:15.084 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.711140+0000", 2024-09-13T11:23:15.084 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.222570348 2024-09-13T11:23:15.084 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.084 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.084 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:23:15.084 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.711180+0000", 2024-09-13T11:23:15.085 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.9273999999999997e-05 2024-09-13T11:23:15.085 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.085 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.085 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:23:15.085 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.711678+0000", 2024-09-13T11:23:15.085 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00049807599999999997 2024-09-13T11:23:15.085 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:15.085 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:15.085 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:15.085 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.085 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.085 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:30820 3.e 3:7ad3d310:::benchmark_data_smithi177_31116_object30819:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e59)", 2024-09-13T11:23:15.085 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:09.495542+0000", 2024-09-13T11:23:15.086 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 5.4931255129999998, 2024-09-13T11:23:15.086 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.22088603900000001, 2024-09-13T11:23:15.086 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:23:15.086 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:23:15.086 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:15.086 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:23:15.086 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:23:15.086 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:15.086 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 30820 2024-09-13T11:23:15.086 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.086 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:23:15.086 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.086 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:23:15.086 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.495542+0000", 2024-09-13T11:23:15.087 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:15.087 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.087 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.087 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:23:15.087 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.495542+0000", 2024-09-13T11:23:15.087 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:15.087 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.087 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.087 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:23:15.087 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.495545+0000", 2024-09-13T11:23:15.087 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.689e-06 2024-09-13T11:23:15.087 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.087 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.087 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:23:15.088 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.495596+0000", 2024-09-13T11:23:15.088 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.1066999999999997e-05 2024-09-13T11:23:15.088 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.088 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.088 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:23:15.088 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.495611+0000", 2024-09-13T11:23:15.088 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.5298000000000002e-05 2024-09-13T11:23:15.088 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.088 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.088 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:23:15.088 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.495620+0000", 2024-09-13T11:23:15.088 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.9509999999999995e-06 2024-09-13T11:23:15.088 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.089 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.089 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:15.089 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.495689+0000", 2024-09-13T11:23:15.089 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.8901999999999999e-05 2024-09-13T11:23:15.089 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.089 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.089 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:23:15.089 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.495881+0000", 2024-09-13T11:23:15.089 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00019190900000000001 2024-09-13T11:23:15.089 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.089 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.089 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 0,1", 2024-09-13T11:23:15.089 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.496036+0000", 2024-09-13T11:23:15.090 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00015474999999999999 2024-09-13T11:23:15.090 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.090 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.090 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:23:15.090 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.499142+0000", 2024-09-13T11:23:15.090 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00310589 2024-09-13T11:23:15.090 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.090 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.090 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:15.090 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.499593+0000", 2024-09-13T11:23:15.090 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00045114599999999998 2024-09-13T11:23:15.090 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.090 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.090 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:15.091 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.715504+0000", 2024-09-13T11:23:15.091 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.21591141599999999 2024-09-13T11:23:15.091 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.091 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.091 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:23:15.091 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.715573+0000", 2024-09-13T11:23:15.091 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.8779e-05 2024-09-13T11:23:15.091 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.091 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.091 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:23:15.091 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.716428+0000", 2024-09-13T11:23:15.091 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00085524199999999998 2024-09-13T11:23:15.091 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:15.091 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:15.092 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:15.092 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.092 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.092 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:30823 3.e 3:7b66f30d:::benchmark_data_smithi177_31116_object30822:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e59)", 2024-09-13T11:23:15.092 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:09.496838+0000", 2024-09-13T11:23:15.092 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 5.4918294320000003, 2024-09-13T11:23:15.092 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.220684879, 2024-09-13T11:23:15.092 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:23:15.092 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:23:15.092 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:15.092 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:23:15.092 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:23:15.092 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:15.092 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 30823 2024-09-13T11:23:15.092 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.093 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:23:15.093 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.093 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:23:15.093 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.496838+0000", 2024-09-13T11:23:15.093 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:15.093 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.093 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.093 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:23:15.093 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.496838+0000", 2024-09-13T11:23:15.093 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:15.093 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.093 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.093 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:23:15.094 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.496840+0000", 2024-09-13T11:23:15.094 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.314e-06 2024-09-13T11:23:15.094 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.094 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.094 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:23:15.094 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.496914+0000", 2024-09-13T11:23:15.094 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.4103999999999994e-05 2024-09-13T11:23:15.094 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.094 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.094 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:23:15.094 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.496924+0000", 2024-09-13T11:23:15.094 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0324e-05 2024-09-13T11:23:15.094 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.094 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.094 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:23:15.095 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.496930+0000", 2024-09-13T11:23:15.095 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.6710000000000004e-06 2024-09-13T11:23:15.095 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.095 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.095 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:15.095 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.497361+0000", 2024-09-13T11:23:15.095 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00043119499999999999 2024-09-13T11:23:15.095 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.095 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.095 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:23:15.095 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.497576+0000", 2024-09-13T11:23:15.095 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00021456099999999999 2024-09-13T11:23:15.095 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.095 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.096 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 0,1", 2024-09-13T11:23:15.096 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.497771+0000", 2024-09-13T11:23:15.096 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000195832 2024-09-13T11:23:15.096 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.096 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.096 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:23:15.096 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.499618+0000", 2024-09-13T11:23:15.096 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0018469879999999999 2024-09-13T11:23:15.096 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.096 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.096 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:15.096 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.499871+0000", 2024-09-13T11:23:15.096 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00025242899999999998 2024-09-13T11:23:15.096 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.097 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.097 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:15.097 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.716535+0000", 2024-09-13T11:23:15.097 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.21666413500000001 2024-09-13T11:23:15.097 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.097 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.097 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:23:15.097 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.716642+0000", 2024-09-13T11:23:15.097 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000106786 2024-09-13T11:23:15.097 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.097 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.097 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:23:15.097 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.717523+0000", 2024-09-13T11:23:15.097 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00088153999999999999 2024-09-13T11:23:15.097 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:15.098 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:15.098 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:15.098 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.098 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.098 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:30839 3.e 3:7b4f601e:::benchmark_data_smithi177_31116_object30838:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e59)", 2024-09-13T11:23:15.098 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:09.512124+0000", 2024-09-13T11:23:15.098 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 5.4765436980000004, 2024-09-13T11:23:15.098 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.20664021399999999, 2024-09-13T11:23:15.098 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:23:15.098 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:23:15.098 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:15.098 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:23:15.098 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:23:15.098 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:15.098 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 30839 2024-09-13T11:23:15.099 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.099 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:23:15.099 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.099 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:23:15.099 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.512124+0000", 2024-09-13T11:23:15.099 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:15.099 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.099 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.099 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:23:15.099 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.512124+0000", 2024-09-13T11:23:15.099 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:15.099 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.099 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.099 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:23:15.100 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.512126+0000", 2024-09-13T11:23:15.100 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.2369999999999999e-06 2024-09-13T11:23:15.100 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.100 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.100 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:23:15.100 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.512191+0000", 2024-09-13T11:23:15.100 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.4237000000000002e-05 2024-09-13T11:23:15.100 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.100 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.100 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:23:15.100 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.512202+0000", 2024-09-13T11:23:15.100 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.1929e-05 2024-09-13T11:23:15.100 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.100 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.100 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:23:15.101 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.512208+0000", 2024-09-13T11:23:15.101 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.5310000000000001e-06 2024-09-13T11:23:15.101 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.101 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.101 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:15.101 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.512259+0000", 2024-09-13T11:23:15.101 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.0578999999999998e-05 2024-09-13T11:23:15.101 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.101 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.101 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:23:15.101 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.512370+0000", 2024-09-13T11:23:15.101 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000111384 2024-09-13T11:23:15.102 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.102 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.102 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 0,1", 2024-09-13T11:23:15.102 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.512483+0000", 2024-09-13T11:23:15.102 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000112868 2024-09-13T11:23:15.102 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.102 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.102 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:23:15.102 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.514007+0000", 2024-09-13T11:23:15.102 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.001524397 2024-09-13T11:23:15.102 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.102 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.102 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:15.102 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.514718+0000", 2024-09-13T11:23:15.103 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00071077900000000003 2024-09-13T11:23:15.103 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.103 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.103 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:15.103 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.717710+0000", 2024-09-13T11:23:15.103 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.20299189200000001 2024-09-13T11:23:15.103 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.103 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.103 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:23:15.103 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.717810+0000", 2024-09-13T11:23:15.103 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.9949000000000001e-05 2024-09-13T11:23:15.103 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.103 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.103 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:23:15.104 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.718764+0000", 2024-09-13T11:23:15.104 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00095443199999999996 2024-09-13T11:23:15.104 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:15.104 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:15.104 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:15.104 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:15.104 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:15.104 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-13T11:23:15.105 INFO:tasks.radosbench.radosbench.0.smithi177.stdout: 48 16 38767 38751 50.4512 83.25 0.0146582 0.0198134 2024-09-13T11:23:15.106 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_ops_in_flight 2024-09-13T11:23:15.106 INFO:tasks.ceph.osd.12.smithi001.stderr:2024-09-13T11:23:15.100+0000 7f892c6e6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26314) UID: 0 2024-09-13T11:23:15.206 INFO:tasks.ceph.osd.5.smithi092.stderr:2024-09-13T11:23:15.201+0000 7f6f0b705640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26346) UID: 0 2024-09-13T11:23:15.294 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-13T11:23:15.294 INFO:teuthology.orchestra.run.smithi001.stdout: "ops": [ 2024-09-13T11:23:15.294 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.294 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:39132 3.9 3:9c3ff9f3:::benchmark_data_smithi177_31116_object39131:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e63)", 2024-09-13T11:23:15.294 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:15.292178+0000", 2024-09-13T11:23:15.294 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 0.00061319499999999997, 2024-09-13T11:23:15.294 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00065478900000000002, 2024-09-13T11:23:15.294 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:23:15.294 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:23:15.294 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "waiting for sub ops", 2024-09-13T11:23:15.295 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:23:15.295 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:23:15.295 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:15.295 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 39132 2024-09-13T11:23:15.295 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.295 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:23:15.295 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.295 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:23:15.295 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:15.292178+0000", 2024-09-13T11:23:15.295 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:15.295 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.295 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.295 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:23:15.296 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:15.292178+0000", 2024-09-13T11:23:15.296 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:15.296 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.296 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.296 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:23:15.296 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:15.292181+0000", 2024-09-13T11:23:15.296 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.224e-06 2024-09-13T11:23:15.296 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.296 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.296 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:23:15.296 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:15.292231+0000", 2024-09-13T11:23:15.296 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.0059999999999998e-05 2024-09-13T11:23:15.296 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.297 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.297 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:23:15.297 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:15.292253+0000", 2024-09-13T11:23:15.297 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.2027000000000001e-05 2024-09-13T11:23:15.297 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.297 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.297 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:23:15.297 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:15.292264+0000", 2024-09-13T11:23:15.297 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.116e-05 2024-09-13T11:23:15.297 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.297 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.297 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:15.297 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:15.292343+0000", 2024-09-13T11:23:15.298 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.9167999999999998e-05 2024-09-13T11:23:15.298 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.298 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.298 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:23:15.298 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:15.292535+0000", 2024-09-13T11:23:15.298 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00019192699999999999 2024-09-13T11:23:15.298 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.298 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.298 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 2", 2024-09-13T11:23:15.298 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:15.292733+0000", 2024-09-13T11:23:15.298 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00019799100000000001 2024-09-13T11:23:15.298 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:15.298 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:15.298 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:15.299 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.299 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.299 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:39110 3.9 3:9f0d9203:::benchmark_data_smithi177_31116_object39109:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e63)", 2024-09-13T11:23:15.299 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:15.277624+0000", 2024-09-13T11:23:15.299 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 0.015167389999999999, 2024-09-13T11:23:15.299 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.015314925, 2024-09-13T11:23:15.299 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:23:15.299 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:23:15.299 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "waiting for sub ops", 2024-09-13T11:23:15.299 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:23:15.299 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:23:15.299 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:15.299 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 39110 2024-09-13T11:23:15.300 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.300 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:23:15.300 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.300 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:23:15.300 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:15.277624+0000", 2024-09-13T11:23:15.300 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:15.300 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.300 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.300 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:23:15.300 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:15.277624+0000", 2024-09-13T11:23:15.300 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:15.300 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.300 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.300 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:23:15.301 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:15.277625+0000", 2024-09-13T11:23:15.301 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0860000000000001e-06 2024-09-13T11:23:15.301 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.301 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.301 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:23:15.301 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:15.277703+0000", 2024-09-13T11:23:15.301 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.7888999999999999e-05 2024-09-13T11:23:15.301 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.301 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.301 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:23:15.301 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:15.277711+0000", 2024-09-13T11:23:15.301 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.5669999999999995e-06 2024-09-13T11:23:15.301 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.302 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.302 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:23:15.302 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:15.277716+0000", 2024-09-13T11:23:15.302 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.4789999999999998e-06 2024-09-13T11:23:15.302 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.302 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.302 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:15.302 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:15.278334+0000", 2024-09-13T11:23:15.302 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00061806600000000001 2024-09-13T11:23:15.302 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.302 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.302 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:23:15.302 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:15.278453+0000", 2024-09-13T11:23:15.302 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000119175 2024-09-13T11:23:15.303 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.303 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.303 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 2", 2024-09-13T11:23:15.303 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:15.278558+0000", 2024-09-13T11:23:15.303 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00010556300000000001 2024-09-13T11:23:15.303 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.303 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.303 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:23:15.303 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:15.279793+0000", 2024-09-13T11:23:15.303 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0012348630000000001 2024-09-13T11:23:15.303 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:15.303 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:15.303 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:15.304 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:15.304 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-09-13T11:23:15.304 INFO:teuthology.orchestra.run.smithi001.stdout: "num_ops": 2 2024-09-13T11:23:15.304 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-13T11:23:15.307 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_blocked_ops 2024-09-13T11:23:15.308 INFO:tasks.ceph.osd.10.smithi162.stderr:2024-09-13T11:23:15.302+0000 7fb59c9f0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26317) UID: 0 2024-09-13T11:23:15.408 INFO:tasks.ceph.osd.1.smithi092.stderr:2024-09-13T11:23:15.401+0000 7fe7789c5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33718) UID: 0 2024-09-13T11:23:15.447 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-13T11:23:15.447 INFO:teuthology.orchestra.run.smithi001.stdout: "ops": [], 2024-09-13T11:23:15.447 INFO:teuthology.orchestra.run.smithi001.stdout: "complaint_time": 30, 2024-09-13T11:23:15.447 INFO:teuthology.orchestra.run.smithi001.stdout: "num_blocked_ops": 0 2024-09-13T11:23:15.447 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-13T11:23:15.464 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_historic_ops 2024-09-13T11:23:15.509 INFO:tasks.ceph.osd.2.smithi162.stderr:2024-09-13T11:23:15.502+0000 7fe92b35a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26313) UID: 0 2024-09-13T11:23:15.567 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-13T11:23:15.560+0000 7f6a5d1a2640 -1 osd.0 63 handle_osd_map injecting map crc failure 2024-09-13T11:23:15.570 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-13T11:23:15.610 INFO:tasks.ceph.osd.7.smithi177.stderr:2024-09-13T11:23:15.606+0000 7f897b51d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26575) UID: 0 2024-09-13T11:23:15.610 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-13T11:23:15.610 INFO:teuthology.orchestra.run.smithi001.stdout: "size": 20, 2024-09-13T11:23:15.610 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 600, 2024-09-13T11:23:15.610 INFO:teuthology.orchestra.run.smithi001.stdout: "ops": [ 2024-09-13T11:23:15.610 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.610 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop_reply(client.4665.0:167 3.9 e41/30 ondisk, result = 0)", 2024-09-13T11:23:15.611 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:22:47.419696+0000", 2024-09-13T11:23:15.611 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 28.187529617999999, 2024-09-13T11:23:15.611 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.64615778199999996, 2024-09-13T11:23:15.611 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:23:15.611 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:23:15.611 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:23:15.611 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:23:15.611 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.611 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:23:15.611 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:47.419696+0000", 2024-09-13T11:23:15.611 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:15.611 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.611 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.611 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:23:15.612 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:47.419696+0000", 2024-09-13T11:23:15.612 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:15.612 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.612 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.612 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:23:15.612 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:47.419697+0000", 2024-09-13T11:23:15.612 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.15e-06 2024-09-13T11:23:15.612 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.612 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.612 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:23:15.612 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:47.419700+0000", 2024-09-13T11:23:15.612 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.2449999999999998e-06 2024-09-13T11:23:15.612 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.613 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.613 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:23:15.613 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:48.065637+0000", 2024-09-13T11:23:15.613 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.64593674300000004 2024-09-13T11:23:15.613 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.613 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.613 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:23:15.613 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:48.065648+0000", 2024-09-13T11:23:15.613 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0865e-05 2024-09-13T11:23:15.613 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.613 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.614 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:15.614 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:48.065720+0000", 2024-09-13T11:23:15.614 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.2020999999999994e-05 2024-09-13T11:23:15.614 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.614 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.614 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:23:15.614 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:48.065728+0000", 2024-09-13T11:23:15.614 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.7660000000000004e-06 2024-09-13T11:23:15.614 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.614 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.614 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:23:15.614 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:48.065853+0000", 2024-09-13T11:23:15.614 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00012499200000000001 2024-09-13T11:23:15.615 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:15.615 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:15.615 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:15.615 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.615 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.615 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:10697 3.0 3:08f06ac6:::benchmark_data_smithi177_31116_object10696:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e48)", 2024-09-13T11:23:15.615 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:22:54.806675+0000", 2024-09-13T11:23:15.615 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 20.800550600000001, 2024-09-13T11:23:15.615 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.102587161, 2024-09-13T11:23:15.615 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:23:15.615 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:23:15.615 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:15.615 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:23:15.616 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:23:15.616 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:15.616 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 10697 2024-09-13T11:23:15.616 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.616 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:23:15.616 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.616 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:23:15.616 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:54.806675+0000", 2024-09-13T11:23:15.616 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:15.616 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.616 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.616 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:23:15.616 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:54.806675+0000", 2024-09-13T11:23:15.616 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:15.617 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.617 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.617 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:23:15.617 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:54.806680+0000", 2024-09-13T11:23:15.617 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.2970000000000003e-06 2024-09-13T11:23:15.617 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.617 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.617 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:23:15.617 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:54.806731+0000", 2024-09-13T11:23:15.617 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.1048999999999999e-05 2024-09-13T11:23:15.617 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.617 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.617 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:23:15.618 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:54.806743+0000", 2024-09-13T11:23:15.618 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2264e-05 2024-09-13T11:23:15.618 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.618 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.618 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:23:15.618 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:54.806752+0000", 2024-09-13T11:23:15.618 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.3259999999999996e-06 2024-09-13T11:23:15.618 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.618 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.618 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:15.618 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:54.806812+0000", 2024-09-13T11:23:15.618 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.0433000000000002e-05 2024-09-13T11:23:15.618 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.619 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.619 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:23:15.619 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:54.806930+0000", 2024-09-13T11:23:15.619 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000117805 2024-09-13T11:23:15.619 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.619 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.619 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 15", 2024-09-13T11:23:15.619 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:54.807042+0000", 2024-09-13T11:23:15.619 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000112528 2024-09-13T11:23:15.619 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.619 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.619 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:23:15.619 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:54.807872+0000", 2024-09-13T11:23:15.619 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00082981600000000004 2024-09-13T11:23:15.620 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.620 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.620 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:15.620 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:54.908964+0000", 2024-09-13T11:23:15.620 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.101092056 2024-09-13T11:23:15.620 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.620 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.620 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:23:15.620 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:54.909137+0000", 2024-09-13T11:23:15.620 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00017322799999999999 2024-09-13T11:23:15.620 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.620 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.620 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:23:15.621 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:54.909262+0000", 2024-09-13T11:23:15.621 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00012435899999999999 2024-09-13T11:23:15.621 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:15.621 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:15.621 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:15.621 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.621 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.621 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop_reply(client.4665.0:10697 3.0 e48/30 ondisk, result = 0)", 2024-09-13T11:23:15.621 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:22:54.808193+0000", 2024-09-13T11:23:15.621 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 20.799032564000001, 2024-09-13T11:23:15.621 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.10108661100000001, 2024-09-13T11:23:15.621 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:23:15.621 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:23:15.622 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:23:15.622 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:23:15.622 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.622 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:23:15.622 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:54.808193+0000", 2024-09-13T11:23:15.622 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:15.622 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.622 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.622 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:23:15.622 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:54.808193+0000", 2024-09-13T11:23:15.622 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:15.622 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.622 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.623 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:23:15.623 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:54.808194+0000", 2024-09-13T11:23:15.623 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.392e-06 2024-09-13T11:23:15.623 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.623 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.623 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:23:15.623 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:54.808198+0000", 2024-09-13T11:23:15.623 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.9099999999999998e-06 2024-09-13T11:23:15.623 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.623 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.623 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:23:15.623 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:54.908728+0000", 2024-09-13T11:23:15.623 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.100530419 2024-09-13T11:23:15.623 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.624 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.624 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:23:15.624 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:54.908775+0000", 2024-09-13T11:23:15.624 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.6479e-05 2024-09-13T11:23:15.624 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.624 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.624 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:15.624 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:54.908920+0000", 2024-09-13T11:23:15.624 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00014498500000000001 2024-09-13T11:23:15.624 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.624 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.624 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:23:15.624 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:54.908944+0000", 2024-09-13T11:23:15.625 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.4493000000000001e-05 2024-09-13T11:23:15.625 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.625 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.625 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:23:15.625 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:54.909279+0000", 2024-09-13T11:23:15.625 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00033493299999999999 2024-09-13T11:23:15.625 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:15.625 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:15.625 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:15.625 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.625 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.625 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:10715 3.9 3:9813c6ac:::benchmark_data_smithi177_31116_object10714:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e48)", 2024-09-13T11:23:15.625 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:22:54.824503+0000", 2024-09-13T11:23:15.626 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 20.782722095, 2024-09-13T11:23:15.626 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.088715206000000005, 2024-09-13T11:23:15.626 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:23:15.626 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:23:15.626 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:15.626 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:23:15.626 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:23:15.626 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:15.626 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 10715 2024-09-13T11:23:15.626 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.626 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:23:15.626 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.626 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:23:15.626 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:54.824503+0000", 2024-09-13T11:23:15.627 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:15.627 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.627 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.627 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:23:15.627 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:54.824503+0000", 2024-09-13T11:23:15.627 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:15.627 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.627 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.627 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:23:15.627 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:54.824505+0000", 2024-09-13T11:23:15.627 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.967e-06 2024-09-13T11:23:15.627 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.627 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.627 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:23:15.628 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:54.824555+0000", 2024-09-13T11:23:15.628 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.0419e-05 2024-09-13T11:23:15.628 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.628 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.628 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:23:15.628 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:54.824573+0000", 2024-09-13T11:23:15.629 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.7499999999999998e-05 2024-09-13T11:23:15.629 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.629 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.629 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:23:15.629 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:54.824597+0000", 2024-09-13T11:23:15.629 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.4389e-05 2024-09-13T11:23:15.629 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.629 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.629 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:15.629 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:54.824674+0000", 2024-09-13T11:23:15.629 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.6808000000000001e-05 2024-09-13T11:23:15.629 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.629 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.629 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:23:15.630 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:54.824781+0000", 2024-09-13T11:23:15.630 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00010696099999999999 2024-09-13T11:23:15.630 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.630 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.630 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 2", 2024-09-13T11:23:15.630 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:54.824894+0000", 2024-09-13T11:23:15.630 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000112785 2024-09-13T11:23:15.630 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.630 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.630 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:23:15.630 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:54.825714+0000", 2024-09-13T11:23:15.630 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00082003999999999996 2024-09-13T11:23:15.630 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.630 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.631 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:15.631 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:54.913065+0000", 2024-09-13T11:23:15.631 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.087351028999999997 2024-09-13T11:23:15.631 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.631 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.631 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:23:15.631 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:54.913152+0000", 2024-09-13T11:23:15.631 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.7162000000000003e-05 2024-09-13T11:23:15.631 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.631 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.631 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:23:15.631 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:54.913218+0000", 2024-09-13T11:23:15.631 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.6146e-05 2024-09-13T11:23:15.632 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:15.632 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:15.632 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:15.632 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.632 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.632 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop_reply(client.4665.0:10715 3.9 e48/30 ondisk, result = 0)", 2024-09-13T11:23:15.632 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:22:54.827201+0000", 2024-09-13T11:23:15.632 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 20.780024392000001, 2024-09-13T11:23:15.632 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.086028436, 2024-09-13T11:23:15.632 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:23:15.632 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:23:15.632 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:23:15.632 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:23:15.633 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.633 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:23:15.633 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:54.827201+0000", 2024-09-13T11:23:15.633 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:15.633 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.633 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.633 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:23:15.633 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:54.827201+0000", 2024-09-13T11:23:15.633 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:15.633 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.633 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.633 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:23:15.633 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:54.827202+0000", 2024-09-13T11:23:15.633 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.6029999999999999e-06 2024-09-13T11:23:15.634 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.634 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.634 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:23:15.634 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:54.827208+0000", 2024-09-13T11:23:15.634 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.2089999999999996e-06 2024-09-13T11:23:15.634 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.634 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.634 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:23:15.634 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:54.912940+0000", 2024-09-13T11:23:15.634 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.085732357999999995 2024-09-13T11:23:15.634 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.634 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.634 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:23:15.634 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:54.912959+0000", 2024-09-13T11:23:15.635 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.8773000000000001e-05 2024-09-13T11:23:15.635 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.635 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.635 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:15.635 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:54.913046+0000", 2024-09-13T11:23:15.635 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.7522999999999995e-05 2024-09-13T11:23:15.635 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.635 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.635 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:23:15.635 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:54.913057+0000", 2024-09-13T11:23:15.635 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0796e-05 2024-09-13T11:23:15.635 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.635 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.635 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:23:15.636 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:54.913229+0000", 2024-09-13T11:23:15.636 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00017217400000000001 2024-09-13T11:23:15.636 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:15.636 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:15.636 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:15.636 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.636 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.636 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4665.0:10755 3.2 e48/30 3:4d45beba:::benchmark_data_smithi177_31116_object10754:head v 48'674, mlcod=48'670)", 2024-09-13T11:23:15.636 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:22:54.883593+0000", 2024-09-13T11:23:15.636 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 20.723632482999999, 2024-09-13T11:23:15.636 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.071458590000000002, 2024-09-13T11:23:15.636 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:23:15.636 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:23:15.637 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:15.637 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:23:15.637 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.637 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:23:15.637 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:54.883593+0000", 2024-09-13T11:23:15.637 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:15.637 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.637 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.637 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:23:15.637 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:54.883593+0000", 2024-09-13T11:23:15.637 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:15.637 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.637 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.637 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:23:15.638 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:54.883594+0000", 2024-09-13T11:23:15.638 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.6139999999999999e-06 2024-09-13T11:23:15.638 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.638 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.638 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:23:15.638 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:54.883622+0000", 2024-09-13T11:23:15.638 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.8045e-05 2024-09-13T11:23:15.638 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.638 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.638 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:23:15.638 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:54.953147+0000", 2024-09-13T11:23:15.638 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.069524444000000005 2024-09-13T11:23:15.638 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.638 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.639 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:23:15.639 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:54.953160+0000", 2024-09-13T11:23:15.639 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.3216e-05 2024-09-13T11:23:15.639 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.639 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.639 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:15.639 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:54.953287+0000", 2024-09-13T11:23:15.639 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00012703899999999999 2024-09-13T11:23:15.639 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.639 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.639 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:23:15.639 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:54.953320+0000", 2024-09-13T11:23:15.639 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.2653000000000001e-05 2024-09-13T11:23:15.640 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.640 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.640 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:23:15.640 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:54.954947+0000", 2024-09-13T11:23:15.640 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0016272909999999999 2024-09-13T11:23:15.640 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.640 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.640 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:23:15.640 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:54.955051+0000", 2024-09-13T11:23:15.640 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000104288 2024-09-13T11:23:15.640 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:15.640 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:15.641 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:15.641 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.641 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.641 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:17634 3.9 3:9e121bd5:::benchmark_data_smithi177_31116_object17633:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-13T11:23:15.641 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:22:59.522129+0000", 2024-09-13T11:23:15.641 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 16.085096264000001, 2024-09-13T11:23:15.641 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.076237751000000006, 2024-09-13T11:23:15.641 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:23:15.641 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:23:15.641 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:15.642 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:23:15.642 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:23:15.642 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:15.642 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 17634 2024-09-13T11:23:15.643 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.643 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:23:15.643 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.643 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:23:15.643 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:59.522129+0000", 2024-09-13T11:23:15.643 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:15.643 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.644 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.644 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:23:15.645 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:59.522129+0000", 2024-09-13T11:23:15.645 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:15.645 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.645 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.645 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:23:15.645 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:59.522130+0000", 2024-09-13T11:23:15.645 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.5379999999999999e-06 2024-09-13T11:23:15.645 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.645 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.645 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:23:15.645 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:59.522213+0000", 2024-09-13T11:23:15.645 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.2256000000000007e-05 2024-09-13T11:23:15.645 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.645 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.646 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:23:15.646 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:59.522223+0000", 2024-09-13T11:23:15.646 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0564000000000001e-05 2024-09-13T11:23:15.646 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.646 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.646 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:23:15.646 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:59.522231+0000", 2024-09-13T11:23:15.646 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.5680000000000002e-06 2024-09-13T11:23:15.646 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.646 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.646 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:15.646 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:59.522279+0000", 2024-09-13T11:23:15.646 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.8127999999999997e-05 2024-09-13T11:23:15.646 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.647 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.647 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:23:15.647 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:59.522376+0000", 2024-09-13T11:23:15.647 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.6873000000000006e-05 2024-09-13T11:23:15.647 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.647 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.647 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 2", 2024-09-13T11:23:15.647 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:59.522467+0000", 2024-09-13T11:23:15.647 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.0816999999999995e-05 2024-09-13T11:23:15.647 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.647 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.647 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:23:15.647 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:59.523626+0000", 2024-09-13T11:23:15.647 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.001159263 2024-09-13T11:23:15.648 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.648 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.648 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:15.648 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:59.598309+0000", 2024-09-13T11:23:15.648 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.074683278000000006 2024-09-13T11:23:15.648 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.648 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.648 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:23:15.648 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:59.598342+0000", 2024-09-13T11:23:15.648 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.2512999999999999e-05 2024-09-13T11:23:15.649 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.649 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.649 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:23:15.649 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:59.598367+0000", 2024-09-13T11:23:15.649 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.4953e-05 2024-09-13T11:23:15.649 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:15.649 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:15.649 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:15.649 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.649 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.649 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop_reply(client.4665.0:17634 3.9 e53/30 ondisk, result = 0)", 2024-09-13T11:23:15.649 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:22:59.524210+0000", 2024-09-13T11:23:15.649 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 16.083015223, 2024-09-13T11:23:15.649 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.074161042999999996, 2024-09-13T11:23:15.650 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:23:15.650 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:23:15.650 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:23:15.650 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:23:15.650 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.650 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:23:15.650 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:59.524210+0000", 2024-09-13T11:23:15.650 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:15.650 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.650 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.650 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:23:15.650 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:59.524210+0000", 2024-09-13T11:23:15.650 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:15.650 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.651 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.651 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:23:15.651 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:59.524211+0000", 2024-09-13T11:23:15.651 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.158e-06 2024-09-13T11:23:15.651 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.651 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.651 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:23:15.651 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:59.524215+0000", 2024-09-13T11:23:15.651 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.5750000000000001e-06 2024-09-13T11:23:15.651 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.651 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.651 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:23:15.651 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:59.598261+0000", 2024-09-13T11:23:15.651 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.074046325999999996 2024-09-13T11:23:15.652 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.652 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.652 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:23:15.652 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:59.598266+0000", 2024-09-13T11:23:15.652 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.1420000000000001e-06 2024-09-13T11:23:15.652 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.652 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.652 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:15.652 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:59.598301+0000", 2024-09-13T11:23:15.652 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.5002000000000001e-05 2024-09-13T11:23:15.652 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.652 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.652 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:23:15.652 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:59.598306+0000", 2024-09-13T11:23:15.653 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.9239999999999998e-06 2024-09-13T11:23:15.653 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.653 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.653 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:23:15.653 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:59.598371+0000", 2024-09-13T11:23:15.653 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.4916e-05 2024-09-13T11:23:15.653 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:15.653 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:15.653 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:15.653 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.653 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.653 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4665.0:25290 3.4 e55/40 3:2c6b3a0e:::benchmark_data_smithi177_31116_object25289:head v 55'1638, mlcod=55'1635)", 2024-09-13T11:23:15.653 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:04.309280+0000", 2024-09-13T11:23:15.653 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 11.297944889, 2024-09-13T11:23:15.654 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.094030580000000002, 2024-09-13T11:23:15.654 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:23:15.654 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:23:15.654 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:15.654 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:23:15.654 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.654 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:23:15.654 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.309280+0000", 2024-09-13T11:23:15.654 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:15.654 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.654 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.654 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:23:15.654 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.309280+0000", 2024-09-13T11:23:15.654 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:15.654 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.655 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.655 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:23:15.655 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.309283+0000", 2024-09-13T11:23:15.655 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.7800000000000001e-06 2024-09-13T11:23:15.655 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.655 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.655 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:23:15.655 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.309334+0000", 2024-09-13T11:23:15.655 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.0510000000000003e-05 2024-09-13T11:23:15.655 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.655 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.655 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:23:15.655 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.400659+0000", 2024-09-13T11:23:15.655 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.091325274999999997 2024-09-13T11:23:15.656 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.656 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.656 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:23:15.656 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.400676+0000", 2024-09-13T11:23:15.656 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.7537000000000002e-05 2024-09-13T11:23:15.656 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.656 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.656 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:15.656 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.400785+0000", 2024-09-13T11:23:15.656 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000108756 2024-09-13T11:23:15.656 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.656 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.656 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:23:15.656 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.400818+0000", 2024-09-13T11:23:15.657 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.2635999999999997e-05 2024-09-13T11:23:15.657 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.657 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.657 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:23:15.657 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.403208+0000", 2024-09-13T11:23:15.657 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0023897900000000001 2024-09-13T11:23:15.657 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.657 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.657 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:23:15.657 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.403311+0000", 2024-09-13T11:23:15.657 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00010329599999999999 2024-09-13T11:23:15.657 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:15.657 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:15.657 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:15.658 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.658 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.658 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:25383 3.0 3:0a6bb869:::benchmark_data_smithi177_31116_object25382:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e55)", 2024-09-13T11:23:15.658 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:04.447376+0000", 2024-09-13T11:23:15.658 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 11.159848949000001, 2024-09-13T11:23:15.658 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.21239864, 2024-09-13T11:23:15.658 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:23:15.658 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:23:15.658 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:15.658 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:23:15.658 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:23:15.658 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:15.658 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 25383 2024-09-13T11:23:15.658 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.659 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:23:15.659 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.659 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:23:15.659 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.447376+0000", 2024-09-13T11:23:15.659 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:15.659 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.659 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.659 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:23:15.659 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.447376+0000", 2024-09-13T11:23:15.659 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:15.659 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.659 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.659 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:23:15.659 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.447378+0000", 2024-09-13T11:23:15.660 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.6729999999999999e-06 2024-09-13T11:23:15.660 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.660 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.660 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:23:15.660 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.447433+0000", 2024-09-13T11:23:15.660 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.5371000000000002e-05 2024-09-13T11:23:15.660 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.660 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.660 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:23:15.660 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.447446+0000", 2024-09-13T11:23:15.660 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.3179999999999999e-05 2024-09-13T11:23:15.660 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.660 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.660 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:23:15.661 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.447456+0000", 2024-09-13T11:23:15.661 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0008e-05 2024-09-13T11:23:15.661 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.661 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.661 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:15.661 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.447514+0000", 2024-09-13T11:23:15.661 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.8023999999999999e-05 2024-09-13T11:23:15.661 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.661 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.661 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:23:15.661 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.447642+0000", 2024-09-13T11:23:15.661 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000127296 2024-09-13T11:23:15.661 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.661 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.661 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 15", 2024-09-13T11:23:15.662 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.447767+0000", 2024-09-13T11:23:15.662 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000125547 2024-09-13T11:23:15.662 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.662 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.662 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:23:15.662 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.449188+0000", 2024-09-13T11:23:15.662 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.001420604 2024-09-13T11:23:15.662 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.662 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.662 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:15.662 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.659668+0000", 2024-09-13T11:23:15.662 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.210479685 2024-09-13T11:23:15.662 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.663 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.663 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:23:15.663 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.659716+0000", 2024-09-13T11:23:15.663 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.8609999999999997e-05 2024-09-13T11:23:15.663 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.663 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.663 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:23:15.663 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.659775+0000", 2024-09-13T11:23:15.663 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.8641999999999999e-05 2024-09-13T11:23:15.663 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:15.663 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:15.663 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:15.663 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.663 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.663 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4665.0:25408 3.4 e56/40 3:294e23bd:::benchmark_data_smithi177_31116_object25407:head v 56'1645, mlcod=55'1642)", 2024-09-13T11:23:15.664 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:04.494686+0000", 2024-09-13T11:23:15.664 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 11.112539290000001, 2024-09-13T11:23:15.664 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.29725566399999998, 2024-09-13T11:23:15.664 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:23:15.664 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:23:15.664 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:15.664 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:23:15.664 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.664 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:23:15.664 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.494686+0000", 2024-09-13T11:23:15.664 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:15.664 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.664 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.664 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:23:15.665 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.494686+0000", 2024-09-13T11:23:15.665 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:15.665 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.665 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.665 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:23:15.665 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.494686+0000", 2024-09-13T11:23:15.665 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.2399999999999999e-07 2024-09-13T11:23:15.665 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.665 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.665 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:23:15.665 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.494809+0000", 2024-09-13T11:23:15.665 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00012308600000000001 2024-09-13T11:23:15.665 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.665 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.666 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:23:15.666 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.789410+0000", 2024-09-13T11:23:15.666 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.29460066899999998 2024-09-13T11:23:15.666 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.666 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.666 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:23:15.666 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.789425+0000", 2024-09-13T11:23:15.666 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.4600000000000001e-05 2024-09-13T11:23:15.666 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.666 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.666 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:15.666 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.789507+0000", 2024-09-13T11:23:15.666 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.2785000000000001e-05 2024-09-13T11:23:15.666 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.666 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.667 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:23:15.667 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.789525+0000", 2024-09-13T11:23:15.667 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.8117999999999998e-05 2024-09-13T11:23:15.667 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.667 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.667 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:23:15.667 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.791852+0000", 2024-09-13T11:23:15.667 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0023264459999999998 2024-09-13T11:23:15.667 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.667 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.667 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:23:15.667 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.791942+0000", 2024-09-13T11:23:15.667 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.9635999999999994e-05 2024-09-13T11:23:15.667 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:15.668 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:15.668 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:15.668 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.668 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.668 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:25420 3.0 3:03f6c3d6:::benchmark_data_smithi177_31116_object25419:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e56)", 2024-09-13T11:23:15.668 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:04.655682+0000", 2024-09-13T11:23:15.668 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 10.951543555000001, 2024-09-13T11:23:15.668 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.074957139000000006, 2024-09-13T11:23:15.668 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:23:15.668 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:23:15.668 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:15.668 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:23:15.668 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:23:15.668 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:15.668 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 25420 2024-09-13T11:23:15.669 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.669 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:23:15.669 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.669 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:23:15.669 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.655682+0000", 2024-09-13T11:23:15.669 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:15.669 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.669 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.669 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:23:15.669 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.655682+0000", 2024-09-13T11:23:15.670 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:15.670 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.670 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.670 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:23:15.670 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.655685+0000", 2024-09-13T11:23:15.670 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.653e-06 2024-09-13T11:23:15.670 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.670 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.670 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:23:15.670 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.655744+0000", 2024-09-13T11:23:15.670 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.8508999999999998e-05 2024-09-13T11:23:15.670 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.670 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.671 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:23:15.671 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.655773+0000", 2024-09-13T11:23:15.671 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.9357e-05 2024-09-13T11:23:15.671 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.671 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.671 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:23:15.671 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.655787+0000", 2024-09-13T11:23:15.671 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.4263e-05 2024-09-13T11:23:15.671 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.671 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.671 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:15.671 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.655923+0000", 2024-09-13T11:23:15.671 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00013515200000000001 2024-09-13T11:23:15.671 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.671 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.672 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:23:15.672 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.656163+0000", 2024-09-13T11:23:15.672 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000240268 2024-09-13T11:23:15.672 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.672 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.672 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 15", 2024-09-13T11:23:15.672 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.656393+0000", 2024-09-13T11:23:15.672 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00022986799999999999 2024-09-13T11:23:15.672 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.672 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.672 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:23:15.672 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.658693+0000", 2024-09-13T11:23:15.672 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.002300074 2024-09-13T11:23:15.672 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.673 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.673 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:15.673 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.730425+0000", 2024-09-13T11:23:15.673 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.071732196999999998 2024-09-13T11:23:15.673 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.673 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.673 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:23:15.673 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.730528+0000", 2024-09-13T11:23:15.673 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000103132 2024-09-13T11:23:15.673 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.673 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.673 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:23:15.673 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.730639+0000", 2024-09-13T11:23:15.673 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000110666 2024-09-13T11:23:15.674 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:15.674 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:15.674 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:15.674 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.674 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.674 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4665.0:30815 3.f e59/30 3:fde17cb1:::benchmark_data_smithi177_31116_object30814:head v 59'1919, mlcod=59'1914)", 2024-09-13T11:23:15.674 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:09.494342+0000", 2024-09-13T11:23:15.674 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 6.1128833040000004, 2024-09-13T11:23:15.674 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.22787855700000001, 2024-09-13T11:23:15.674 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:23:15.674 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:23:15.674 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:15.674 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:23:15.674 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.675 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:23:15.675 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.494342+0000", 2024-09-13T11:23:15.675 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:15.675 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.675 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.675 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:23:15.675 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.494342+0000", 2024-09-13T11:23:15.675 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:15.675 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.675 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.675 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:23:15.675 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.494343+0000", 2024-09-13T11:23:15.675 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.251e-06 2024-09-13T11:23:15.675 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.675 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.676 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:23:15.676 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.494387+0000", 2024-09-13T11:23:15.676 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.4276000000000001e-05 2024-09-13T11:23:15.676 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.676 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.676 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:23:15.676 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.720454+0000", 2024-09-13T11:23:15.676 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.226066614 2024-09-13T11:23:15.676 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.676 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.676 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:23:15.676 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.720462+0000", 2024-09-13T11:23:15.676 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.9240000000000007e-06 2024-09-13T11:23:15.677 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.677 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.677 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:15.677 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.720537+0000", 2024-09-13T11:23:15.677 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.4967000000000002e-05 2024-09-13T11:23:15.677 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.677 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.677 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:23:15.677 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.720563+0000", 2024-09-13T11:23:15.677 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.5630999999999999e-05 2024-09-13T11:23:15.677 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.677 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.677 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:23:15.677 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.722183+0000", 2024-09-13T11:23:15.677 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0016209670000000001 2024-09-13T11:23:15.678 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.678 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.678 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:23:15.678 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.722220+0000", 2024-09-13T11:23:15.678 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.6927000000000001e-05 2024-09-13T11:23:15.678 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:15.678 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:15.678 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:15.678 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.678 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.678 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4665.0:30828 3.f e59/30 3:f3a9c6a9:::benchmark_data_smithi177_31116_object30827:head v 59'1920, mlcod=59'1916)", 2024-09-13T11:23:15.678 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:09.502607+0000", 2024-09-13T11:23:15.678 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 6.1046182470000003, 2024-09-13T11:23:15.678 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.26374408300000002, 2024-09-13T11:23:15.679 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:23:15.679 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:23:15.679 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:15.679 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:23:15.679 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.679 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:23:15.679 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.502607+0000", 2024-09-13T11:23:15.679 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:15.679 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.679 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.679 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:23:15.679 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.502607+0000", 2024-09-13T11:23:15.679 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:15.679 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.679 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.680 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:23:15.680 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.502609+0000", 2024-09-13T11:23:15.680 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.84e-06 2024-09-13T11:23:15.680 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.680 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.680 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:23:15.680 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.502663+0000", 2024-09-13T11:23:15.680 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.3971000000000002e-05 2024-09-13T11:23:15.680 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.680 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.680 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:23:15.680 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.763912+0000", 2024-09-13T11:23:15.680 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.26124904300000001 2024-09-13T11:23:15.680 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.681 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.681 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:23:15.681 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.763920+0000", 2024-09-13T11:23:15.681 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.1149999999999994e-06 2024-09-13T11:23:15.681 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.681 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.681 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:15.681 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.764024+0000", 2024-09-13T11:23:15.681 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00010380899999999999 2024-09-13T11:23:15.681 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.681 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.681 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:23:15.681 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.764068+0000", 2024-09-13T11:23:15.681 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.4638999999999997e-05 2024-09-13T11:23:15.682 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.682 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.682 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:23:15.682 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.766308+0000", 2024-09-13T11:23:15.682 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0022392039999999998 2024-09-13T11:23:15.682 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.682 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.682 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:23:15.682 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.766351+0000", 2024-09-13T11:23:15.682 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.3461999999999998e-05 2024-09-13T11:23:15.682 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:15.682 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:15.682 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:15.682 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.682 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.683 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4665.0:30976 3.f e59/30 3:f39fdccb:::benchmark_data_smithi177_31116_object30975:head v 59'1926, mlcod=59'1922)", 2024-09-13T11:23:15.683 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:09.826192+0000", 2024-09-13T11:23:15.683 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 5.7810335249999998, 2024-09-13T11:23:15.683 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.071880974, 2024-09-13T11:23:15.683 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:23:15.683 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:23:15.683 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:15.683 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:23:15.683 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.683 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:23:15.683 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.826192+0000", 2024-09-13T11:23:15.683 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:15.683 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.683 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.684 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:23:15.684 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.826192+0000", 2024-09-13T11:23:15.684 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:15.684 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.684 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.684 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:23:15.684 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.826194+0000", 2024-09-13T11:23:15.684 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.886e-06 2024-09-13T11:23:15.684 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.684 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.684 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:23:15.684 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.826277+0000", 2024-09-13T11:23:15.684 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.3409000000000001e-05 2024-09-13T11:23:15.684 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.685 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.685 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:23:15.685 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.896616+0000", 2024-09-13T11:23:15.685 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.070339473 2024-09-13T11:23:15.685 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.685 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.685 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:23:15.685 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.896623+0000", 2024-09-13T11:23:15.685 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.7109999999999998e-06 2024-09-13T11:23:15.685 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.685 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.685 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:15.685 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.896670+0000", 2024-09-13T11:23:15.685 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.6944000000000003e-05 2024-09-13T11:23:15.686 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.686 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.686 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:23:15.686 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.896685+0000", 2024-09-13T11:23:15.686 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.4491000000000001e-05 2024-09-13T11:23:15.686 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.686 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.686 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:23:15.686 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.898022+0000", 2024-09-13T11:23:15.686 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0013378280000000001 2024-09-13T11:23:15.686 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.686 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.686 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:23:15.686 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.898073+0000", 2024-09-13T11:23:15.686 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.0232000000000003e-05 2024-09-13T11:23:15.687 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:15.687 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:15.687 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:15.687 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.687 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.687 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4665.0:30979 3.f e59/30 3:fe31d07f:::benchmark_data_smithi177_31116_object30978:head v 59'1927, mlcod=59'1922)", 2024-09-13T11:23:15.687 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:09.829194+0000", 2024-09-13T11:23:15.687 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 5.7780307010000005, 2024-09-13T11:23:15.687 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.088362964000000002, 2024-09-13T11:23:15.687 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:23:15.687 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:23:15.687 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:15.687 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:23:15.688 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.688 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:23:15.688 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.829194+0000", 2024-09-13T11:23:15.688 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:15.688 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.688 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.688 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:23:15.688 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.829194+0000", 2024-09-13T11:23:15.688 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:15.688 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.688 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.688 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:23:15.688 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.829196+0000", 2024-09-13T11:23:15.688 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.4559999999999999e-06 2024-09-13T11:23:15.689 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.689 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.689 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:23:15.689 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.829320+0000", 2024-09-13T11:23:15.689 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00012374200000000001 2024-09-13T11:23:15.689 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.689 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.689 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:23:15.689 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.916142+0000", 2024-09-13T11:23:15.689 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.086822193000000006 2024-09-13T11:23:15.689 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.689 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.689 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:23:15.689 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.916150+0000", 2024-09-13T11:23:15.689 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.0760000000000003e-06 2024-09-13T11:23:15.690 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.690 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.690 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:15.690 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.916200+0000", 2024-09-13T11:23:15.690 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.0207000000000002e-05 2024-09-13T11:23:15.690 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.690 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.690 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:23:15.690 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.916214+0000", 2024-09-13T11:23:15.690 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.4266e-05 2024-09-13T11:23:15.690 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.690 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.690 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:23:15.690 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.917506+0000", 2024-09-13T11:23:15.691 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.001291553 2024-09-13T11:23:15.691 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.691 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.691 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:23:15.691 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.917557+0000", 2024-09-13T11:23:15.691 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.1471000000000002e-05 2024-09-13T11:23:15.691 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:15.691 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:15.691 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:15.691 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.691 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.691 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4665.0:31065 3.f e59/30 3:f622697a:::benchmark_data_smithi177_31116_object31064:head v 59'1930, mlcod=59'1925)", 2024-09-13T11:23:15.691 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:09.919087+0000", 2024-09-13T11:23:15.691 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 5.6881378570000001, 2024-09-13T11:23:15.692 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.078316636999999995, 2024-09-13T11:23:15.692 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:23:15.692 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:23:15.692 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:15.692 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:23:15.692 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.692 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:23:15.692 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.919087+0000", 2024-09-13T11:23:15.692 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:15.692 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.692 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.692 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:23:15.692 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.919087+0000", 2024-09-13T11:23:15.692 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:15.693 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.693 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.693 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:23:15.693 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.919088+0000", 2024-09-13T11:23:15.693 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0920000000000001e-06 2024-09-13T11:23:15.693 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.693 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.693 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:23:15.693 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.919122+0000", 2024-09-13T11:23:15.693 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.3272000000000003e-05 2024-09-13T11:23:15.693 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.693 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.693 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:23:15.693 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.996003+0000", 2024-09-13T11:23:15.694 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.076881489999999997 2024-09-13T11:23:15.694 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.694 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.694 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:23:15.694 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.996010+0000", 2024-09-13T11:23:15.694 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.883e-06 2024-09-13T11:23:15.694 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.694 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.694 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:15.694 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.996062+0000", 2024-09-13T11:23:15.694 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.1909000000000001e-05 2024-09-13T11:23:15.694 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.694 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.694 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:23:15.695 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.996077+0000", 2024-09-13T11:23:15.695 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.5475999999999999e-05 2024-09-13T11:23:15.695 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.695 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.695 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:23:15.695 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.997359+0000", 2024-09-13T11:23:15.695 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0012811960000000001 2024-09-13T11:23:15.695 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.695 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.695 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:23:15.695 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.997404+0000", 2024-09-13T11:23:15.695 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.5318999999999998e-05 2024-09-13T11:23:15.695 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:15.695 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:15.696 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:15.696 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.696 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.696 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4665.0:31074 3.f e59/30 3:f7a42e3b:::benchmark_data_smithi177_31116_object31073:head v 59'1931, mlcod=59'1927)", 2024-09-13T11:23:15.696 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:09.927808+0000", 2024-09-13T11:23:15.696 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 5.6794167839999998, 2024-09-13T11:23:15.696 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.075142754000000006, 2024-09-13T11:23:15.696 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:23:15.696 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:23:15.696 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:15.696 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:23:15.696 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.696 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:23:15.696 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.927808+0000", 2024-09-13T11:23:15.697 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:15.697 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.697 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.697 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:23:15.697 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.927808+0000", 2024-09-13T11:23:15.697 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:15.697 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.697 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.697 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:23:15.697 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.927810+0000", 2024-09-13T11:23:15.697 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2750000000000001e-06 2024-09-13T11:23:15.697 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.697 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.697 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:23:15.698 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.927868+0000", 2024-09-13T11:23:15.698 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.8724000000000002e-05 2024-09-13T11:23:15.698 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.698 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.698 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:23:15.698 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:10.001398+0000", 2024-09-13T11:23:15.698 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.073529702000000002 2024-09-13T11:23:15.698 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.698 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.698 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:23:15.698 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:10.001406+0000", 2024-09-13T11:23:15.698 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.5100000000000001e-06 2024-09-13T11:23:15.698 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.698 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.699 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:15.699 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:10.001466+0000", 2024-09-13T11:23:15.699 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.0480999999999999e-05 2024-09-13T11:23:15.699 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.699 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.699 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:23:15.699 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:10.001490+0000", 2024-09-13T11:23:15.699 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.4253999999999999e-05 2024-09-13T11:23:15.699 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.699 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.699 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:23:15.699 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:10.002913+0000", 2024-09-13T11:23:15.699 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0014225100000000001 2024-09-13T11:23:15.699 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.700 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.700 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:23:15.700 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:10.002951+0000", 2024-09-13T11:23:15.700 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.8297999999999998e-05 2024-09-13T11:23:15.700 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:15.700 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:15.700 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:15.700 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.700 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.700 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:38564 3.9 3:98f60bf4:::benchmark_data_smithi177_31116_object38563:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e63)", 2024-09-13T11:23:15.700 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:14.823608+0000", 2024-09-13T11:23:15.700 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 0.78361704600000004, 2024-09-13T11:23:15.700 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.080796259999999995, 2024-09-13T11:23:15.700 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:23:15.700 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:23:15.701 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:15.701 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:23:15.701 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:23:15.701 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:15.701 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 38564 2024-09-13T11:23:15.701 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.701 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:23:15.701 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.701 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:23:15.701 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:14.823608+0000", 2024-09-13T11:23:15.701 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:15.701 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.701 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.701 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:23:15.702 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:14.823608+0000", 2024-09-13T11:23:15.702 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:15.702 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.702 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.702 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:23:15.702 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:14.823610+0000", 2024-09-13T11:23:15.702 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.84e-06 2024-09-13T11:23:15.702 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.702 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.702 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:23:15.702 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:14.823678+0000", 2024-09-13T11:23:15.702 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.7668000000000003e-05 2024-09-13T11:23:15.702 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.702 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.703 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:23:15.703 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:14.823687+0000", 2024-09-13T11:23:15.703 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.6220000000000004e-06 2024-09-13T11:23:15.703 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.703 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.703 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:23:15.703 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:14.823692+0000", 2024-09-13T11:23:15.703 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.9690000000000002e-06 2024-09-13T11:23:15.703 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.703 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.703 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:15.703 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:14.823738+0000", 2024-09-13T11:23:15.703 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.5537999999999997e-05 2024-09-13T11:23:15.703 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.704 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.704 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:23:15.704 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:14.823894+0000", 2024-09-13T11:23:15.704 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000156033 2024-09-13T11:23:15.704 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.704 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.704 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 2", 2024-09-13T11:23:15.704 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:14.824051+0000", 2024-09-13T11:23:15.704 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00015744000000000001 2024-09-13T11:23:15.704 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.704 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.704 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:23:15.704 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:14.825470+0000", 2024-09-13T11:23:15.704 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.001419233 2024-09-13T11:23:15.705 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.705 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.705 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:15.705 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:14.904337+0000", 2024-09-13T11:23:15.705 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.078866461999999998 2024-09-13T11:23:15.705 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.705 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.705 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:23:15.705 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:14.904376+0000", 2024-09-13T11:23:15.705 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.8676e-05 2024-09-13T11:23:15.705 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.705 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.706 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:23:15.706 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:14.904404+0000", 2024-09-13T11:23:15.706 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.8779e-05 2024-09-13T11:23:15.706 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:15.706 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:15.706 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:15.706 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.706 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.706 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:38580 3.9 3:9ba6ecd4:::benchmark_data_smithi177_31116_object38579:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e63)", 2024-09-13T11:23:15.706 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:14.841663+0000", 2024-09-13T11:23:15.706 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 0.76556204100000003, 2024-09-13T11:23:15.707 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.092241380999999997, 2024-09-13T11:23:15.707 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:23:15.707 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:23:15.707 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:15.707 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:23:15.707 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:23:15.707 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:15.707 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 38580 2024-09-13T11:23:15.707 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.707 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:23:15.707 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.707 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:23:15.707 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:14.841663+0000", 2024-09-13T11:23:15.707 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:15.707 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.708 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.708 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:23:15.708 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:14.841663+0000", 2024-09-13T11:23:15.708 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:15.708 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.708 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.708 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:23:15.708 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:14.841665+0000", 2024-09-13T11:23:15.708 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.2409999999999998e-06 2024-09-13T11:23:15.708 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.708 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.708 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:23:15.708 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:14.841782+0000", 2024-09-13T11:23:15.709 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00011616999999999999 2024-09-13T11:23:15.709 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.709 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.709 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:23:15.709 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:14.841796+0000", 2024-09-13T11:23:15.709 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.4482e-05 2024-09-13T11:23:15.709 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.709 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.709 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:23:15.709 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:14.841804+0000", 2024-09-13T11:23:15.709 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.5610000000000001e-06 2024-09-13T11:23:15.709 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.709 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.709 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:15.710 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:14.841863+0000", 2024-09-13T11:23:15.710 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.9537999999999998e-05 2024-09-13T11:23:15.710 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.710 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.710 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:23:15.710 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:14.841960+0000", 2024-09-13T11:23:15.710 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.7227999999999998e-05 2024-09-13T11:23:15.710 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.710 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.710 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 2", 2024-09-13T11:23:15.710 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:14.842047+0000", 2024-09-13T11:23:15.710 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.6422999999999996e-05 2024-09-13T11:23:15.710 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.710 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.710 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:23:15.711 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:14.843303+0000", 2024-09-13T11:23:15.711 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0012565390000000001 2024-09-13T11:23:15.711 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.711 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.711 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:15.711 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:14.933814+0000", 2024-09-13T11:23:15.711 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.090510254999999998 2024-09-13T11:23:15.711 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.711 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.711 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:23:15.711 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:14.933851+0000", 2024-09-13T11:23:15.711 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.7837999999999999e-05 2024-09-13T11:23:15.711 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:15.711 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:15.711 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:23:15.712 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:14.933905+0000", 2024-09-13T11:23:15.712 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.3106000000000003e-05 2024-09-13T11:23:15.712 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:15.712 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:15.712 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:15.712 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:15.712 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:15.712 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-13T11:23:15.713 INFO:tasks.ceph.osd.15.smithi177.stderr:2024-09-13T11:23:15.710+0000 7f9d6ce77640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26578) UID: 0 2024-09-13T11:23:15.714 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-13T11:23:15.814 INFO:tasks.ceph.osd.2.smithi162.stderr:2024-09-13T11:23:15.810+0000 7fe92b35a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26313) UID: 0 2024-09-13T11:23:15.847 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-09-13T11:23:15.847 INFO:teuthology.orchestra.run.smithi092.stdout: "ops": [], 2024-09-13T11:23:15.847 INFO:teuthology.orchestra.run.smithi092.stdout: "num_ops": 0 2024-09-13T11:23:15.847 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-09-13T11:23:15.863 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-13T11:23:15.915 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-13T11:23:15.908+0000 7f6a68775640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26307) UID: 0 2024-09-13T11:23:16.000 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-09-13T11:23:16.000 INFO:teuthology.orchestra.run.smithi092.stdout: "ops": [], 2024-09-13T11:23:16.000 INFO:teuthology.orchestra.run.smithi092.stdout: "complaint_time": 30, 2024-09-13T11:23:16.000 INFO:teuthology.orchestra.run.smithi092.stdout: "num_blocked_ops": 0 2024-09-13T11:23:16.000 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-09-13T11:23:16.002 INFO:tasks.radosbench.radosbench.0.smithi177.stdout: 49 16 40189 40173 51.2352 88.875 0.00710615 0.0195088 2024-09-13T11:23:16.015 INFO:tasks.ceph.osd.4.smithi001.stderr:2024-09-13T11:23:16.008+0000 7ff8e0218640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26310) UID: 0 2024-09-13T11:23:16.022 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-13T11:23:16.117 INFO:tasks.ceph.osd.14.smithi162.stderr:2024-09-13T11:23:16.110+0000 7f235c0db640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26319) UID: 0 2024-09-13T11:23:16.164 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-09-13T11:23:16.164 INFO:teuthology.orchestra.run.smithi092.stdout: "size": 20, 2024-09-13T11:23:16.165 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 600, 2024-09-13T11:23:16.165 INFO:teuthology.orchestra.run.smithi092.stdout: "ops": [] 2024-09-13T11:23:16.165 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-09-13T11:23:16.183 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-09-13T11:23:16.218 INFO:tasks.ceph.osd.15.smithi177.stderr:2024-09-13T11:23:16.214+0000 7f9d6ce77640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26578) UID: 0 2024-09-13T11:23:16.318 INFO:tasks.ceph.osd.2.smithi162.stderr:2024-09-13T11:23:16.314+0000 7fe92b35a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26313) UID: 0 2024-09-13T11:23:16.326 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-09-13T11:23:16.326 INFO:teuthology.orchestra.run.smithi092.stdout: "ops": [], 2024-09-13T11:23:16.326 INFO:teuthology.orchestra.run.smithi092.stdout: "num_ops": 0 2024-09-13T11:23:16.326 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-09-13T11:23:16.346 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-09-13T11:23:16.420 INFO:tasks.ceph.osd.15.smithi177.stderr:2024-09-13T11:23:16.418+0000 7f9d6ce77640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26578) UID: 0 2024-09-13T11:23:16.487 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-09-13T11:23:16.487 INFO:teuthology.orchestra.run.smithi092.stdout: "ops": [], 2024-09-13T11:23:16.488 INFO:teuthology.orchestra.run.smithi092.stdout: "complaint_time": 30, 2024-09-13T11:23:16.488 INFO:teuthology.orchestra.run.smithi092.stdout: "num_blocked_ops": 0 2024-09-13T11:23:16.488 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-09-13T11:23:16.507 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-09-13T11:23:16.521 INFO:tasks.ceph.osd.15.smithi177.stderr:2024-09-13T11:23:16.518+0000 7f9d6ce77640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26578) UID: 0 2024-09-13T11:23:16.554 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15] out_osds: [] dead_osds: [] live_osds: [0, 4, 8, 12, 5, 9, 13, 2, 6, 10, 14, 3, 7, 11, 15, 1] 2024-09-13T11:23:16.554 INFO:tasks.thrashosds.thrasher:choose_action: min_in 2 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-13T11:23:16.555 INFO:tasks.thrashosds.thrasher:Growing pool unique_pool_0 2024-09-13T11:23:16.555 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-13T11:23:16.566 INFO:teuthology.orchestra.run.smithi001.stderr:nodeep-scrub is set 2024-09-13T11:23:16.570 INFO:tasks.ceph.osd.2.smithi162.stderr:2024-09-13T11:23:16.566+0000 7fe91fd87640 -1 osd.2 64 handle_osd_map injecting map crc failure 2024-09-13T11:23:16.622 INFO:tasks.ceph.osd.11.smithi177.stderr:2024-09-13T11:23:16.618+0000 7f24792c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26580) UID: 0 2024-09-13T11:23:16.641 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-09-13T11:23:16.641 INFO:teuthology.orchestra.run.smithi092.stdout: "size": 20, 2024-09-13T11:23:16.641 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 600, 2024-09-13T11:23:16.642 INFO:teuthology.orchestra.run.smithi092.stdout: "ops": [ 2024-09-13T11:23:16.642 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:16.642 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop(client.4344.0:5 2.2 e27/25 2:5cea7035:::rbd_info:head v 27'1, mlcod=0'0)", 2024-09-13T11:23:16.642 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:15.250889+0000", 2024-09-13T11:23:16.642 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 61.390996530000002, 2024-09-13T11:23:16.642 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.001990627, 2024-09-13T11:23:16.642 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:23:16.642 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:23:16.642 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:16.642 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:23:16.642 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:16.642 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:23:16.642 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:15.250889+0000", 2024-09-13T11:23:16.643 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:16.643 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:16.643 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:16.643 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:23:16.643 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:15.250889+0000", 2024-09-13T11:23:16.643 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:16.643 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:16.643 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:16.643 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:23:16.643 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:15.250890+0000", 2024-09-13T11:23:16.643 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.153e-06 2024-09-13T11:23:16.643 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:16.643 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:16.643 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:23:16.644 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:15.250907+0000", 2024-09-13T11:23:16.644 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.6821e-05 2024-09-13T11:23:16.644 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:16.644 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:16.644 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:23:16.644 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:15.250991+0000", 2024-09-13T11:23:16.644 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.4241999999999997e-05 2024-09-13T11:23:16.644 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:16.644 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:16.644 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:23:16.644 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:15.251023+0000", 2024-09-13T11:23:16.644 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.1780999999999999e-05 2024-09-13T11:23:16.644 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:16.645 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:16.645 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:23:16.645 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:15.251178+0000", 2024-09-13T11:23:16.645 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000155064 2024-09-13T11:23:16.645 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:16.645 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:16.645 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:23:16.645 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:15.251215+0000", 2024-09-13T11:23:16.645 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.7160000000000003e-05 2024-09-13T11:23:16.645 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:16.645 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:16.645 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:23:16.645 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:15.252759+0000", 2024-09-13T11:23:16.646 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0015440110000000001 2024-09-13T11:23:16.646 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:16.646 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:16.646 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:23:16.646 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:15.252880+0000", 2024-09-13T11:23:16.646 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000120395 2024-09-13T11:23:16.646 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:16.646 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:23:16.646 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:16.646 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:16.646 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:16.646 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop(client.4344.0:7 2.2 e28/25 2:5cea7035:::rbd_info:head v 28'2, mlcod=0'0)", 2024-09-13T11:23:16.646 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:16.268518+0000", 2024-09-13T11:23:16.646 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 60.373366930000003, 2024-09-13T11:23:16.647 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0017167040000000001, 2024-09-13T11:23:16.647 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:23:16.647 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:23:16.647 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:16.647 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:23:16.647 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:16.647 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:23:16.647 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:16.268518+0000", 2024-09-13T11:23:16.647 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:16.647 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:16.647 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:16.647 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:23:16.647 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:16.268518+0000", 2024-09-13T11:23:16.647 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:16.648 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:16.648 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:16.648 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:23:16.648 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:16.268519+0000", 2024-09-13T11:23:16.648 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.1299999999999999e-07 2024-09-13T11:23:16.648 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:16.648 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:16.648 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:23:16.648 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:16.268539+0000", 2024-09-13T11:23:16.649 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.0214999999999999e-05 2024-09-13T11:23:16.649 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:16.649 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:16.649 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:23:16.649 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:16.268622+0000", 2024-09-13T11:23:16.649 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.2395e-05 2024-09-13T11:23:16.649 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:16.649 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:16.649 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:23:16.649 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:16.268638+0000", 2024-09-13T11:23:16.649 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.6189e-05 2024-09-13T11:23:16.649 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:16.650 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:16.650 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:23:16.650 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:16.268740+0000", 2024-09-13T11:23:16.650 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00010164 2024-09-13T11:23:16.650 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:16.650 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:16.650 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:23:16.650 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:16.268783+0000", 2024-09-13T11:23:16.650 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.2988999999999998e-05 2024-09-13T11:23:16.650 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:16.650 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:16.650 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:23:16.650 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:16.270121+0000", 2024-09-13T11:23:16.650 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00133829 2024-09-13T11:23:16.651 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:16.651 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:16.651 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:23:16.651 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:16.270235+0000", 2024-09-13T11:23:16.651 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000114273 2024-09-13T11:23:16.651 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:16.651 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:23:16.651 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:16.651 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:16.651 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:23:16.651 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-09-13T11:23:16.660 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_ops_in_flight 2024-09-13T11:23:16.723 INFO:tasks.ceph.osd.9.smithi092.stderr:2024-09-13T11:23:16.717+0000 7ff5ab3d0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26348) UID: 0 2024-09-13T11:23:16.792 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-09-13T11:23:16.792 INFO:teuthology.orchestra.run.smithi092.stdout: "ops": [ 2024-09-13T11:23:16.792 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:16.793 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:41224 3.b 3:dbdd1f33:::benchmark_data_smithi177_31116_object41223:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e65)", 2024-09-13T11:23:16.793 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:16.769166+0000", 2024-09-13T11:23:16.793 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 0.023413538000000001, 2024-09-13T11:23:16.793 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.023435827999999999, 2024-09-13T11:23:16.793 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:23:16.793 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:23:16.793 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "waiting for sub ops", 2024-09-13T11:23:16.793 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:23:16.793 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:23:16.793 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:16.793 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 41224 2024-09-13T11:23:16.793 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:16.793 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:23:16.794 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:16.794 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:23:16.794 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:16.769166+0000", 2024-09-13T11:23:16.794 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:16.794 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:16.794 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:16.794 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:23:16.794 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:16.769166+0000", 2024-09-13T11:23:16.794 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:16.794 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:16.794 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:16.794 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:23:16.794 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:16.769168+0000", 2024-09-13T11:23:16.795 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.5099999999999999e-06 2024-09-13T11:23:16.795 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:16.795 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:16.795 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:23:16.795 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:16.769269+0000", 2024-09-13T11:23:16.795 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000100832 2024-09-13T11:23:16.795 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:16.795 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:16.795 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:23:16.795 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:16.769277+0000", 2024-09-13T11:23:16.795 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.3499999999999997e-06 2024-09-13T11:23:16.795 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:16.795 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:16.796 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:23:16.796 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:16.769284+0000", 2024-09-13T11:23:16.796 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.9349999999999997e-06 2024-09-13T11:23:16.796 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:16.796 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:16.796 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:23:16.796 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:16.769333+0000", 2024-09-13T11:23:16.796 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.8702999999999998e-05 2024-09-13T11:23:16.796 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:16.796 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:16.796 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:23:16.796 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:16.769435+0000", 2024-09-13T11:23:16.796 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000102042 2024-09-13T11:23:16.797 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:16.797 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:16.797 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1", 2024-09-13T11:23:16.797 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:16.769525+0000", 2024-09-13T11:23:16.797 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.0520999999999996e-05 2024-09-13T11:23:16.797 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:16.797 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:16.797 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:23:16.797 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:16.771581+0000", 2024-09-13T11:23:16.797 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0020556060000000002 2024-09-13T11:23:16.797 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:16.797 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:23:16.797 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:16.798 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:16.798 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:16.798 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:41238 3.1 3:8aa6a4e7:::benchmark_data_smithi177_31116_object41237:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e65)", 2024-09-13T11:23:16.798 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:16.779698+0000", 2024-09-13T11:23:16.798 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 0.012881861, 2024-09-13T11:23:16.798 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.012951038999999999, 2024-09-13T11:23:16.798 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:23:16.798 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:23:16.798 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "waiting for sub ops", 2024-09-13T11:23:16.798 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:23:16.798 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:23:16.798 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:16.798 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 41238 2024-09-13T11:23:16.798 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:16.799 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:23:16.799 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:16.799 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:23:16.799 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:16.779698+0000", 2024-09-13T11:23:16.799 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:16.799 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:16.799 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:16.799 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:23:16.799 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:16.779698+0000", 2024-09-13T11:23:16.799 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:16.799 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:16.799 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:16.800 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:23:16.800 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:16.779700+0000", 2024-09-13T11:23:16.800 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.4789999999999999e-06 2024-09-13T11:23:16.800 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:16.800 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:16.800 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:23:16.800 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:16.779764+0000", 2024-09-13T11:23:16.800 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.4133000000000004e-05 2024-09-13T11:23:16.800 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:16.800 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:16.800 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:23:16.800 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:16.779776+0000", 2024-09-13T11:23:16.800 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.1955e-05 2024-09-13T11:23:16.801 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:16.801 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:16.801 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:23:16.801 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:16.779783+0000", 2024-09-13T11:23:16.801 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.5170000000000002e-06 2024-09-13T11:23:16.801 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:16.801 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:16.801 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:23:16.801 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:16.782503+0000", 2024-09-13T11:23:16.801 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0027199450000000001 2024-09-13T11:23:16.801 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:16.801 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:16.801 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:23:16.801 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:16.782642+0000", 2024-09-13T11:23:16.802 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00013883299999999999 2024-09-13T11:23:16.802 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:16.802 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:16.802 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 11", 2024-09-13T11:23:16.802 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:16.782786+0000", 2024-09-13T11:23:16.802 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00014353900000000001 2024-09-13T11:23:16.802 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:16.802 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:16.802 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:23:16.802 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:16.784449+0000", 2024-09-13T11:23:16.802 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00166333 2024-09-13T11:23:16.802 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:16.802 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:23:16.803 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:16.803 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:16.803 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:16.803 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:41201 3.6 3:69e3c6b8:::benchmark_data_smithi177_31116_object41200:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e65)", 2024-09-13T11:23:16.803 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:16.748075+0000", 2024-09-13T11:23:16.803 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 0.044504601999999997, 2024-09-13T11:23:16.803 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.044613470000000002, 2024-09-13T11:23:16.803 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:23:16.803 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:23:16.803 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "waiting for sub ops", 2024-09-13T11:23:16.803 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:23:16.803 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:23:16.803 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:16.803 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 41201 2024-09-13T11:23:16.804 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:16.804 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:23:16.804 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:16.804 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:23:16.804 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:16.748075+0000", 2024-09-13T11:23:16.805 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:16.805 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:16.805 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:16.805 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:23:16.805 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:16.748075+0000", 2024-09-13T11:23:16.805 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:16.805 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:16.805 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:16.805 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:23:16.805 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:16.748077+0000", 2024-09-13T11:23:16.805 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.105e-06 2024-09-13T11:23:16.805 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:16.806 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:16.806 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:23:16.806 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:16.748139+0000", 2024-09-13T11:23:16.806 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.2500000000000001e-05 2024-09-13T11:23:16.806 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:16.806 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:16.806 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:23:16.806 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:16.748150+0000", 2024-09-13T11:23:16.806 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0746e-05 2024-09-13T11:23:16.806 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:16.806 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:16.806 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:23:16.806 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:16.748157+0000", 2024-09-13T11:23:16.806 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.8499999999999996e-06 2024-09-13T11:23:16.807 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:16.807 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:16.807 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:23:16.807 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:16.748205+0000", 2024-09-13T11:23:16.807 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.8576999999999999e-05 2024-09-13T11:23:16.807 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:16.807 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:16.807 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:23:16.807 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:16.748327+0000", 2024-09-13T11:23:16.807 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000121599 2024-09-13T11:23:16.807 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:16.807 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:16.807 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 14", 2024-09-13T11:23:16.808 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:16.748412+0000", 2024-09-13T11:23:16.808 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.5142999999999994e-05 2024-09-13T11:23:16.808 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:16.808 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:16.808 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:23:16.808 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:16.749948+0000", 2024-09-13T11:23:16.808 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.001535851 2024-09-13T11:23:16.808 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:16.808 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:23:16.808 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:16.808 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:16.808 INFO:teuthology.orchestra.run.smithi092.stdout: ], 2024-09-13T11:23:16.808 INFO:teuthology.orchestra.run.smithi092.stdout: "num_ops": 3 2024-09-13T11:23:16.809 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-09-13T11:23:16.809 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T11:23:16.810 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_blocked_ops 2024-09-13T11:23:16.853 INFO:tasks.ceph.osd.5.smithi092.stderr:2024-09-13T11:23:16.849+0000 7f6f0b705640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26346) UID: 0 2024-09-13T11:23:16.924 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-09-13T11:23:16.918+0000 7f97befc9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26315) UID: 0 2024-09-13T11:23:16.932 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-13T11:23:16.933 INFO:teuthology.orchestra.run.smithi001.stderr:dumped all 2024-09-13T11:23:16.958 INFO:teuthology.orchestra.run.smithi001.stdout:{"pg_ready":true,"pg_map":{"version":84,"stamp":"2024-09-13T11:23:16.565157+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2480669235,"num_objects":37847,"num_object_clones":0,"num_object_copies":75694,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1814,"num_objects_unfound":0,"num_objects_dirty":37847,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":37990,"num_write_kb":2424626,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":735,"num_bytes_recovered":48103443,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":37958,"ondisk_log_size":37958,"up":50,"acting":50,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":54,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":15,"kb":1509949440,"kb_used":5493696,"kb_used_data":4977216,"kb_used_omap":27,"kb_used_meta":516132,"kb_avail":1504455744,"statfs":{"total":1546188226560,"available":1540562681856,"internally_reserved":0,"allocated":5096669184,"data_stored":5092156406,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28358,"internal_metadata":528519482},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,3,3,1,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":303759360,"num_objects":4635,"num_object_clones":0,"num_object_copies":9270,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-910,"num_objects_unfound":0,"num_objects_dirty":4635,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4635,"num_write_kb":296640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":456,"num_bytes_recovered":29818899,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.435770"},"pg_stats":[{"pgid":"3.f","version":"61'2142","reported_seq":2212,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T11:23:12.013557+0000","last_change":"2024-09-13T11:22:25.925951+0000","last_active":"2024-09-13T11:23:12.013557+0000","last_peered":"2024-09-13T11:23:12.013557+0000","last_clean":"2024-09-13T11:23:12.013557+0000","last_became_active":"2024-09-13T11:22:25.925169+0000","last_became_peered":"2024-09-13T11:22:25.925169+0000","last_unstale":"2024-09-13T11:23:12.013557+0000","last_undegraded":"2024-09-13T11:23:12.013557+0000","last_fullsized":"2024-09-13T11:23:12.013557+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:22:24.896055+0000","objects_scrubbed":0,"log_size":2142,"log_dups_size":0,"ondisk_log_size":2142,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:23:38.143926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":140378112,"num_objects":2142,"num_object_clones":0,"num_object_copies":4284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2142,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2142,"num_write_kb":137088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,12],"acting":[11,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.e","version":"64'2574","reported_seq":4855,"reported_epoch":64,"state":"active+recovering+remapped","last_fresh":"2024-09-13T11:23:16.018534+0000","last_change":"2024-09-13T11:23:06.617265+0000","last_active":"2024-09-13T11:23:16.018534+0000","last_peered":"2024-09-13T11:23:16.018534+0000","last_clean":"2024-09-13T11:23:04.462865+0000","last_became_active":"2024-09-13T11:23:06.595158+0000","last_became_peered":"2024-09-13T11:23:06.595158+0000","last_unstale":"2024-09-13T11:23:16.018534+0000","last_undegraded":"2024-09-13T11:23:16.018534+0000","last_fullsized":"2024-09-13T11:23:16.018534+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:22:24.896055+0000","objects_scrubbed":0,"log_size":2574,"log_dups_size":0,"ondisk_log_size":2574,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":168689664,"num_objects":2574,"num_object_clones":0,"num_object_copies":5148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1814,"num_objects_unfound":0,"num_objects_dirty":2574,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2574,"num_write_kb":164736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":734,"num_bytes_recovered":48103424,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[8,0],"avail_no_missing":["8","14"],"object_location_counts":[{"shards":"0,8","objects":1667},{"shards":"8,14","objects":907}],"blocked_by":[],"up_primary":0,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.d","version":"64'2458","reported_seq":2533,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-13T11:23:16.001624+0000","last_change":"2024-09-13T11:22:25.927325+0000","last_active":"2024-09-13T11:23:16.001624+0000","last_peered":"2024-09-13T11:23:16.001624+0000","last_clean":"2024-09-13T11:23:16.001624+0000","last_became_active":"2024-09-13T11:22:25.926433+0000","last_became_peered":"2024-09-13T11:22:25.926433+0000","last_unstale":"2024-09-13T11:23:16.001624+0000","last_undegraded":"2024-09-13T11:23:16.001624+0000","last_fullsized":"2024-09-13T11:23:16.001624+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:22:24.896055+0000","objects_scrubbed":0,"log_size":2458,"log_dups_size":0,"ondisk_log_size":2458,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:23:39.281487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":161087488,"num_objects":2458,"num_object_clones":0,"num_object_copies":4916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2458,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2458,"num_write_kb":157312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.c","version":"61'2134","reported_seq":2203,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T11:23:12.006132+0000","last_change":"2024-09-13T11:22:25.934619+0000","last_active":"2024-09-13T11:23:12.006132+0000","last_peered":"2024-09-13T11:23:12.006132+0000","last_clean":"2024-09-13T11:23:12.006132+0000","last_became_active":"2024-09-13T11:22:25.933854+0000","last_became_peered":"2024-09-13T11:22:25.933854+0000","last_unstale":"2024-09-13T11:23:12.006132+0000","last_undegraded":"2024-09-13T11:23:12.006132+0000","last_fullsized":"2024-09-13T11:23:12.006132+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:22:24.896055+0000","objects_scrubbed":0,"log_size":2134,"log_dups_size":0,"ondisk_log_size":2134,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:23:42.213952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":139853824,"num_objects":2134,"num_object_clones":0,"num_object_copies":4268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2134,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2134,"num_write_kb":136576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,13],"acting":[11,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.b","version":"61'2159","reported_seq":2230,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T11:23:12.025444+0000","last_change":"2024-09-13T11:22:49.175600+0000","last_active":"2024-09-13T11:23:12.025444+0000","last_peered":"2024-09-13T11:23:12.025444+0000","last_clean":"2024-09-13T11:23:12.025444+0000","last_became_active":"2024-09-13T11:22:49.167083+0000","last_became_peered":"2024-09-13T11:22:49.167083+0000","last_unstale":"2024-09-13T11:23:12.025444+0000","last_undegraded":"2024-09-13T11:23:12.025444+0000","last_fullsized":"2024-09-13T11:23:12.025444+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:22:24.896055+0000","objects_scrubbed":0,"log_size":2159,"log_dups_size":0,"ondisk_log_size":2159,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:23:45.371983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":141492224,"num_objects":2159,"num_object_clones":0,"num_object_copies":4318,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2159,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2159,"num_write_kb":138176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,1],"acting":[13,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"3.a","version":"63'2438","reported_seq":2512,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-13T11:23:14.971715+0000","last_change":"2024-09-13T11:22:25.932217+0000","last_active":"2024-09-13T11:23:14.971715+0000","last_peered":"2024-09-13T11:23:14.971715+0000","last_clean":"2024-09-13T11:23:14.971715+0000","last_became_active":"2024-09-13T11:22:25.931493+0000","last_became_peered":"2024-09-13T11:22:25.931493+0000","last_unstale":"2024-09-13T11:23:14.971715+0000","last_undegraded":"2024-09-13T11:23:14.971715+0000","last_fullsized":"2024-09-13T11:23:14.971715+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:22:24.896055+0000","objects_scrubbed":0,"log_size":2438,"log_dups_size":0,"ondisk_log_size":2438,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:23:33.302955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":159776768,"num_objects":2438,"num_object_clones":0,"num_object_copies":4876,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2438,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2438,"num_write_kb":156032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.9","version":"64'2500","reported_seq":2573,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-13T11:23:16.005797+0000","last_change":"2024-09-13T11:22:25.922876+0000","last_active":"2024-09-13T11:23:16.005797+0000","last_peered":"2024-09-13T11:23:16.005797+0000","last_clean":"2024-09-13T11:23:16.005797+0000","last_became_active":"2024-09-13T11:22:25.922498+0000","last_became_peered":"2024-09-13T11:22:25.922498+0000","last_unstale":"2024-09-13T11:23:16.005797+0000","last_undegraded":"2024-09-13T11:23:16.005797+0000","last_fullsized":"2024-09-13T11:23:16.005797+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:22:24.896055+0000","objects_scrubbed":0,"log_size":2500,"log_dups_size":0,"ondisk_log_size":2500,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:23:29.835976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":163840000,"num_objects":2500,"num_object_clones":0,"num_object_copies":5000,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2500,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2500,"num_write_kb":160000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,2],"acting":[12,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[]},{"pgid":"3.8","version":"64'2487","reported_seq":2564,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-13T11:23:16.011289+0000","last_change":"2024-09-13T11:22:47.301262+0000","last_active":"2024-09-13T11:23:16.011289+0000","last_peered":"2024-09-13T11:23:16.011289+0000","last_clean":"2024-09-13T11:23:16.011289+0000","last_became_active":"2024-09-13T11:22:47.296719+0000","last_became_peered":"2024-09-13T11:22:47.296719+0000","last_unstale":"2024-09-13T11:23:16.011289+0000","last_undegraded":"2024-09-13T11:23:16.011289+0000","last_fullsized":"2024-09-13T11:23:16.011289+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:22:24.896055+0000","objects_scrubbed":0,"log_size":2487,"log_dups_size":0,"ondisk_log_size":2487,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:23:54.227881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":162988032,"num_objects":2487,"num_object_clones":0,"num_object_copies":4974,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2487,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2487,"num_write_kb":159168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"26'112","reported_seq":225,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-13T11:23:15.565834+0000","last_change":"2024-09-13T11:22:12.131711+0000","last_active":"2024-09-13T11:23:15.565834+0000","last_peered":"2024-09-13T11:23:15.565834+0000","last_clean":"2024-09-13T11:23:15.565834+0000","last_became_active":"2024-09-13T11:22:12.129721+0000","last_became_peered":"2024-09-13T11:22:12.129721+0000","last_unstale":"2024-09-13T11:23:15.565834+0000","last_undegraded":"2024-09-13T11:23:15.565834+0000","last_fullsized":"2024-09-13T11:23:15.565834+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:11.098388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:11.098388+0000","last_clean_scrub_stamp":"2024-09-13T11:22:11.098388+0000","objects_scrubbed":0,"log_size":112,"log_dups_size":0,"ondisk_log_size":112,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:23:40.174992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.3","version":"26'1","reported_seq":93,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-13T11:23:15.568431+0000","last_change":"2024-09-13T11:22:16.547691+0000","last_active":"2024-09-13T11:23:15.568431+0000","last_peered":"2024-09-13T11:23:15.568431+0000","last_clean":"2024-09-13T11:23:15.568431+0000","last_became_active":"2024-09-13T11:22:14.146507+0000","last_became_peered":"2024-09-13T11:22:14.146507+0000","last_unstale":"2024-09-13T11:23:15.568431+0000","last_undegraded":"2024-09-13T11:23:15.568431+0000","last_fullsized":"2024-09-13T11:23:15.568431+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:22:13.113563+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:23:18.785562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00085610999999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,15],"acting":[14,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"3.2","version":"64'2533","reported_seq":2610,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-13T11:23:16.029730+0000","last_change":"2024-09-13T11:22:25.923575+0000","last_active":"2024-09-13T11:23:16.029730+0000","last_peered":"2024-09-13T11:23:16.029730+0000","last_clean":"2024-09-13T11:23:16.029730+0000","last_became_active":"2024-09-13T11:22:25.922665+0000","last_became_peered":"2024-09-13T11:22:25.922665+0000","last_unstale":"2024-09-13T11:23:16.029730+0000","last_undegraded":"2024-09-13T11:23:16.029730+0000","last_fullsized":"2024-09-13T11:23:16.029730+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:22:24.896055+0000","objects_scrubbed":0,"log_size":2533,"log_dups_size":0,"ondisk_log_size":2533,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:23:37.611518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":166002688,"num_objects":2533,"num_object_clones":0,"num_object_copies":5066,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2533,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2533,"num_write_kb":162112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,12],"acting":[14,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":89,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-13T11:23:13.542861+0000","last_change":"2024-09-13T11:22:16.267999+0000","last_active":"2024-09-13T11:23:13.542861+0000","last_peered":"2024-09-13T11:23:13.542861+0000","last_clean":"2024-09-13T11:23:13.542861+0000","last_became_active":"2024-09-13T11:22:14.211435+0000","last_became_peered":"2024-09-13T11:22:14.211435+0000","last_unstale":"2024-09-13T11:23:13.542861+0000","last_undegraded":"2024-09-13T11:23:13.542861+0000","last_fullsized":"2024-09-13T11:23:13.542861+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:22:13.113563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:23:24.076640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010010360000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"61'2121","reported_seq":2189,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T11:23:12.021385+0000","last_change":"2024-09-13T11:22:25.933242+0000","last_active":"2024-09-13T11:23:12.021385+0000","last_peered":"2024-09-13T11:23:12.021385+0000","last_clean":"2024-09-13T11:23:12.021385+0000","last_became_active":"2024-09-13T11:22:25.932868+0000","last_became_peered":"2024-09-13T11:22:25.932868+0000","last_unstale":"2024-09-13T11:23:12.021385+0000","last_undegraded":"2024-09-13T11:23:12.021385+0000","last_fullsized":"2024-09-13T11:23:12.021385+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:22:24.896055+0000","objects_scrubbed":0,"log_size":2121,"log_dups_size":0,"ondisk_log_size":2121,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:23:38.369067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":139001856,"num_objects":2121,"num_object_clones":0,"num_object_copies":4242,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2121,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2121,"num_write_kb":135744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,11],"acting":[13,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":101,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-13T11:23:15.568363+0000","last_change":"2024-09-13T11:22:47.298206+0000","last_active":"2024-09-13T11:23:15.568363+0000","last_peered":"2024-09-13T11:23:15.568363+0000","last_clean":"2024-09-13T11:23:15.568363+0000","last_became_active":"2024-09-13T11:22:47.297319+0000","last_became_peered":"2024-09-13T11:22:47.297319+0000","last_unstale":"2024-09-13T11:23:15.568363+0000","last_undegraded":"2024-09-13T11:23:15.568363+0000","last_fullsized":"2024-09-13T11:23:15.568363+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:22:13.113563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:23:23.483800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00081225399999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,1],"acting":[14,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"3.0","version":"64'2557","reported_seq":2632,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-13T11:23:15.989828+0000","last_change":"2024-09-13T11:22:25.926230+0000","last_active":"2024-09-13T11:23:15.989828+0000","last_peered":"2024-09-13T11:23:15.989828+0000","last_clean":"2024-09-13T11:23:15.989828+0000","last_became_active":"2024-09-13T11:22:25.925791+0000","last_became_peered":"2024-09-13T11:22:25.925791+0000","last_unstale":"2024-09-13T11:23:15.989828+0000","last_undegraded":"2024-09-13T11:23:15.989828+0000","last_fullsized":"2024-09-13T11:23:15.989828+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:22:24.896055+0000","objects_scrubbed":0,"log_size":2557,"log_dups_size":0,"ondisk_log_size":2557,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:23:38.551603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":167575552,"num_objects":2557,"num_object_clones":0,"num_object_copies":5114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2557,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2557,"num_write_kb":163648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,15],"acting":[12,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[]},{"pgid":"2.2","version":"28'2","reported_seq":21,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-13T11:23:15.583689+0000","last_change":"2024-09-13T11:23:12.560311+0000","last_active":"2024-09-13T11:23:15.583689+0000","last_peered":"2024-09-13T11:23:15.583689+0000","last_clean":"2024-09-13T11:23:15.583689+0000","last_became_active":"2024-09-13T11:23:12.546348+0000","last_became_peered":"2024-09-13T11:23:12.546348+0000","last_unstale":"2024-09-13T11:23:15.583689+0000","last_undegraded":"2024-09-13T11:23:15.583689+0000","last_fullsized":"2024-09-13T11:23:15.583689+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:22:13.113563+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:23:19.329317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0025826730000000002,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8],"acting":[2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"64'2563","reported_seq":2639,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-13T11:23:16.001591+0000","last_change":"2024-09-13T11:22:25.930558+0000","last_active":"2024-09-13T11:23:16.001591+0000","last_peered":"2024-09-13T11:23:16.001591+0000","last_clean":"2024-09-13T11:23:16.001591+0000","last_became_active":"2024-09-13T11:22:25.930023+0000","last_became_peered":"2024-09-13T11:22:25.930023+0000","last_unstale":"2024-09-13T11:23:16.001591+0000","last_undegraded":"2024-09-13T11:23:16.001591+0000","last_fullsized":"2024-09-13T11:23:16.001591+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:22:24.896055+0000","objects_scrubbed":0,"log_size":2563,"log_dups_size":0,"ondisk_log_size":2563,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:23:33.097704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":167968768,"num_objects":2563,"num_object_clones":0,"num_object_copies":5126,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2563,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2563,"num_write_kb":164032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,7],"acting":[10,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.5","version":"64'2464","reported_seq":2540,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-13T11:23:15.984411+0000","last_change":"2024-09-13T11:22:25.930005+0000","last_active":"2024-09-13T11:23:15.984411+0000","last_peered":"2024-09-13T11:23:15.984411+0000","last_clean":"2024-09-13T11:23:15.984411+0000","last_became_active":"2024-09-13T11:22:25.929242+0000","last_became_peered":"2024-09-13T11:22:25.929242+0000","last_unstale":"2024-09-13T11:23:15.984411+0000","last_undegraded":"2024-09-13T11:23:15.984411+0000","last_fullsized":"2024-09-13T11:23:15.984411+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:22:24.896055+0000","objects_scrubbed":0,"log_size":2464,"log_dups_size":0,"ondisk_log_size":2464,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:23:41.561894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":161480704,"num_objects":2464,"num_object_clones":0,"num_object_copies":4928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2464,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2464,"num_write_kb":157696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,13],"acting":[15,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":77,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-13T11:23:13.550344+0000","last_change":"2024-09-13T11:22:47.720313+0000","last_active":"2024-09-13T11:23:13.550344+0000","last_peered":"2024-09-13T11:23:13.550344+0000","last_clean":"2024-09-13T11:23:13.550344+0000","last_became_active":"2024-09-13T11:22:47.719636+0000","last_became_peered":"2024-09-13T11:22:47.719636+0000","last_unstale":"2024-09-13T11:23:13.550344+0000","last_undegraded":"2024-09-13T11:23:13.550344+0000","last_fullsized":"2024-09-13T11:23:13.550344+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:22:13.113563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:23:13.812273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00087476000000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,9],"acting":[1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"63'2432","reported_seq":2485,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-13T11:23:15.290086+0000","last_change":"2024-09-13T11:22:47.293720+0000","last_active":"2024-09-13T11:23:15.290086+0000","last_peered":"2024-09-13T11:23:15.290086+0000","last_clean":"2024-09-13T11:23:15.290086+0000","last_became_active":"2024-09-13T11:22:47.292998+0000","last_became_peered":"2024-09-13T11:22:47.292998+0000","last_unstale":"2024-09-13T11:23:15.290086+0000","last_undegraded":"2024-09-13T11:23:15.290086+0000","last_fullsized":"2024-09-13T11:23:15.290086+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:22:24.896055+0000","objects_scrubbed":0,"log_size":2432,"log_dups_size":0,"ondisk_log_size":2432,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:23:42.001016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":159383552,"num_objects":2432,"num_object_clones":0,"num_object_copies":4864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2432,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2432,"num_write_kb":155648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,12],"acting":[1,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":91,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-13T11:23:15.561289+0000","last_change":"2024-09-13T11:22:16.504060+0000","last_active":"2024-09-13T11:23:15.561289+0000","last_peered":"2024-09-13T11:23:15.561289+0000","last_clean":"2024-09-13T11:23:15.561289+0000","last_became_active":"2024-09-13T11:22:14.581458+0000","last_became_peered":"2024-09-13T11:22:14.581458+0000","last_unstale":"2024-09-13T11:23:15.561289+0000","last_undegraded":"2024-09-13T11:23:15.561289+0000","last_fullsized":"2024-09-13T11:23:15.561289+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:22:13.113563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:23:37.935010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001026047,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5],"acting":[8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.7","version":"61'2097","reported_seq":2167,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T11:23:12.018764+0000","last_change":"2024-09-13T11:22:25.924462+0000","last_active":"2024-09-13T11:23:12.018764+0000","last_peered":"2024-09-13T11:23:12.018764+0000","last_clean":"2024-09-13T11:23:12.018764+0000","last_became_active":"2024-09-13T11:22:25.924170+0000","last_became_peered":"2024-09-13T11:22:25.924170+0000","last_unstale":"2024-09-13T11:23:12.018764+0000","last_undegraded":"2024-09-13T11:23:12.018764+0000","last_fullsized":"2024-09-13T11:23:12.018764+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:22:24.896055+0000","objects_scrubbed":0,"log_size":2097,"log_dups_size":0,"ondisk_log_size":2097,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:23:51.945140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":137428992,"num_objects":2097,"num_object_clones":0,"num_object_copies":4194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2097,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2097,"num_write_kb":134208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,8],"acting":[13,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":84,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T11:23:11.548648+0000","last_change":"2024-09-13T11:22:16.260591+0000","last_active":"2024-09-13T11:23:11.548648+0000","last_peered":"2024-09-13T11:23:11.548648+0000","last_clean":"2024-09-13T11:23:11.548648+0000","last_became_active":"2024-09-13T11:22:14.211923+0000","last_became_peered":"2024-09-13T11:22:14.211923+0000","last_unstale":"2024-09-13T11:23:11.548648+0000","last_undegraded":"2024-09-13T11:23:11.548648+0000","last_fullsized":"2024-09-13T11:23:11.548648+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:22:13.113563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:23:20.320636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011689529999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0],"acting":[9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.6","version":"61'2184","reported_seq":2254,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T11:23:12.018996+0000","last_change":"2024-09-13T11:22:25.928269+0000","last_active":"2024-09-13T11:23:12.018996+0000","last_peered":"2024-09-13T11:23:12.018996+0000","last_clean":"2024-09-13T11:23:12.018996+0000","last_became_active":"2024-09-13T11:22:25.927961+0000","last_became_peered":"2024-09-13T11:22:25.927961+0000","last_unstale":"2024-09-13T11:23:12.018996+0000","last_undegraded":"2024-09-13T11:23:12.018996+0000","last_fullsized":"2024-09-13T11:23:12.018996+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:22:24.896055+0000","objects_scrubbed":0,"log_size":2184,"log_dups_size":0,"ondisk_log_size":2184,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:23:25.574436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":143130624,"num_objects":2184,"num_object_clones":0,"num_object_copies":4368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2184,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2184,"num_write_kb":139776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,14],"acting":[13,14],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":91,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-13T11:23:15.572918+0000","last_change":"2024-09-13T11:22:16.889130+0000","last_active":"2024-09-13T11:23:15.572918+0000","last_peered":"2024-09-13T11:23:15.572918+0000","last_clean":"2024-09-13T11:23:15.572918+0000","last_became_active":"2024-09-13T11:22:14.192897+0000","last_became_peered":"2024-09-13T11:22:14.192897+0000","last_unstale":"2024-09-13T11:23:15.572918+0000","last_undegraded":"2024-09-13T11:23:15.572918+0000","last_fullsized":"2024-09-13T11:23:15.572918+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:22:13.113563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:23:26.278353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00084655899999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,10],"acting":[15,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":2480078848,"num_objects":37843,"num_object_clones":0,"num_object_copies":75686,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1814,"num_objects_unfound":0,"num_objects_dirty":37843,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":37843,"num_write_kb":2421952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":734,"num_bytes_recovered":48103424,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":5182783488,"data_stored":5182783488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":37843,"ondisk_log_size":37843,"up":32,"acting":32,"num_store_stats":13},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":10},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":112,"ondisk_log_size":112,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":13,"up_from":23,"seq":98784247824,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":823768,"kb_used_data":788424,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93548072,"statfs":{"total":96636764160,"available":95793225728,"internally_reserved":0,"allocated":807346176,"data_stored":807073551,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,6,8,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":9,"up_from":23,"seq":98784247823,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27616,"kb_used_data":592,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344224,"statfs":{"total":96636764160,"available":96608485376,"internally_reserved":0,"allocated":606208,"data_stored":332694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,2,3,8,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":40,"seq":171798691847,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":594116,"kb_used_data":566260,"kb_used_omap":4,"kb_used_meta":27835,"kb_avail":93777724,"statfs":{"total":96636764160,"available":96028389376,"internally_reserved":0,"allocated":579850240,"data_stored":579554977,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4433,"internal_metadata":28503727},"hb_peers":[0,2,5,6,7,8,9,11,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":21,"seq":90194313232,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":333772,"kb_used_data":306748,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94038068,"statfs":{"total":96636764160,"available":96294981632,"internally_reserved":0,"allocated":314109952,"data_stored":313811808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,2,4,5,7,8,9,10,12,13],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":21,"seq":90194313235,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27684,"kb_used_data":660,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344156,"statfs":{"total":96636764160,"available":96608415744,"internally_reserved":0,"allocated":675840,"data_stored":358589,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":22,"seq":94489280527,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":133068,"kb_used_data":101884,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94238772,"statfs":{"total":96636764160,"available":96500502528,"internally_reserved":0,"allocated":104329216,"data_stored":104089417,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,3,5,6,7,8,10,11,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":22,"seq":94489280528,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":194780,"kb_used_data":159436,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94177060,"statfs":{"total":96636764160,"available":96437309440,"internally_reserved":0,"allocated":163262464,"data_stored":162957681,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,7,8,9,10,11,12,13],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":15,"up_from":22,"seq":94489280528,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":345548,"kb_used_data":314364,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94026292,"statfs":{"total":96636764160,"available":96282923008,"internally_reserved":0,"allocated":321908736,"data_stored":321669665,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,3,4,5,6,7,10,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":22,"seq":94489280528,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31824,"kb_used_data":640,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340016,"statfs":{"total":96636764160,"available":96604176384,"internally_reserved":0,"allocated":655360,"data_stored":349312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[3,5,7,8,9,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":22,"seq":94489280528,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":191084,"kb_used_data":155740,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94180756,"statfs":{"total":96636764160,"available":96441094144,"internally_reserved":0,"allocated":159477760,"data_stored":159208988,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,3,5,7,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":22,"seq":94489280528,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":517612,"kb_used_data":478108,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":93854228,"statfs":{"total":96636764160,"available":96106729472,"internally_reserved":0,"allocated":489582592,"data_stored":489318710,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":40434122},"hb_peers":[0,3,6,8,9,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":22,"seq":94489280528,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":350512,"kb_used_data":319328,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94021328,"statfs":{"total":96636764160,"available":96277839872,"internally_reserved":0,"allocated":326991872,"data_stored":326664326,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,5,7,9,10,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":22,"seq":94489280527,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":194016,"kb_used_data":158672,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94177824,"statfs":{"total":96636764160,"available":96438091776,"internally_reserved":0,"allocated":162480128,"data_stored":162222920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,5,7,8,9,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":11,"up_from":22,"seq":94489280528,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":430992,"kb_used_data":395584,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93940848,"statfs":{"total":96636764160,"available":96195428352,"internally_reserved":0,"allocated":405078016,"data_stored":404816449,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":36174276},"hb_peers":[2,3,4,5,6,10,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":12,"up_from":22,"seq":94489280528,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":833232,"kb_used_data":802048,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93538608,"statfs":{"total":96636764160,"available":95783534592,"internally_reserved":0,"allocated":821297152,"data_stored":821000505,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[2,3,4,5,6,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,0,1,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":14,"up_from":22,"seq":94489280528,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":464072,"kb_used_data":428728,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93907768,"statfs":{"total":96636764160,"available":96161554432,"internally_reserved":0,"allocated":439017472,"data_stored":438726814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,5,6,7,8,9,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":109379584,"data_stored":109379584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":582352896,"data_stored":582352896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":163840000,"data_stored":163840000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":315359232,"data_stored":315359232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":162988032,"data_stored":162988032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":494403584,"data_stored":494403584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":329842688,"data_stored":329842688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":167968768,"data_stored":167968768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":419168256,"data_stored":419168256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":827064320,"data_stored":827064320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":839057408,"data_stored":839057408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":442236928,"data_stored":442236928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":329121792,"data_stored":329121792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T11:23:16.959 INFO:tasks.ceph.ceph_manager.ceph:increase pool size by 10 2024-09-13T11:23:16.959 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool set unique_pool_0 pg_num 26 2024-09-13T11:23:16.991 INFO:teuthology.orchestra.run.smithi001.stdout:osd.0: {} 2024-09-13T11:23:16.991 INFO:teuthology.orchestra.run.smithi001.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T11:23:16.992 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-09-13T11:23:16.992 INFO:teuthology.orchestra.run.smithi092.stdout: "ops": [], 2024-09-13T11:23:16.992 INFO:teuthology.orchestra.run.smithi092.stdout: "complaint_time": 30, 2024-09-13T11:23:16.992 INFO:teuthology.orchestra.run.smithi092.stdout: "num_blocked_ops": 0 2024-09-13T11:23:16.992 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-09-13T11:23:17.002 INFO:tasks.radosbench.radosbench.0.smithi177.stdout: 50 16 41453 41437 51.7903 79 0.0133127 0.0193006 2024-09-13T11:23:17.010 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_historic_ops 2024-09-13T11:23:17.024 INFO:teuthology.orchestra.run.smithi001.stdout:osd.1: {} 2024-09-13T11:23:17.024 INFO:teuthology.orchestra.run.smithi001.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T11:23:17.054 INFO:teuthology.orchestra.run.smithi001.stdout:osd.2: {} 2024-09-13T11:23:17.054 INFO:teuthology.orchestra.run.smithi001.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T11:23:17.057 INFO:tasks.ceph.osd.9.smithi092.stderr:2024-09-13T11:23:17.053+0000 7ff5ab3d0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26348) UID: 0 2024-09-13T11:23:17.079 INFO:teuthology.orchestra.run.smithi001.stdout:osd.3: {} 2024-09-13T11:23:17.080 INFO:teuthology.orchestra.run.smithi001.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T11:23:17.106 INFO:teuthology.orchestra.run.smithi001.stdout:osd.4: {} 2024-09-13T11:23:17.106 INFO:teuthology.orchestra.run.smithi001.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T11:23:17.125 INFO:tasks.ceph.osd.9.smithi092.stderr:2024-09-13T11:23:17.121+0000 7ff5ab3d0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26348) UID: 0 2024-09-13T11:23:17.135 INFO:teuthology.orchestra.run.smithi001.stdout:osd.5: {} 2024-09-13T11:23:17.135 INFO:teuthology.orchestra.run.smithi001.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T11:23:17.154 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-09-13T11:23:17.154 INFO:teuthology.orchestra.run.smithi092.stdout: "size": 20, 2024-09-13T11:23:17.154 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 600, 2024-09-13T11:23:17.155 INFO:teuthology.orchestra.run.smithi092.stdout: "ops": [ 2024-09-13T11:23:17.155 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.155 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:79 3.1 3:841df5b5:::benchmark_data_smithi177_31116_object78:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-09-13T11:23:17.155 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:27.056083+0000", 2024-09-13T11:23:17.155 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 50.096242195000002, 2024-09-13T11:23:17.155 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.89469730199999997, 2024-09-13T11:23:17.155 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:23:17.155 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:23:17.155 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:17.155 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:23:17.155 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:23:17.155 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:17.155 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 79 2024-09-13T11:23:17.156 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.156 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:23:17.156 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.156 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:23:17.156 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.056083+0000", 2024-09-13T11:23:17.156 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:17.156 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.156 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.156 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:23:17.157 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.056083+0000", 2024-09-13T11:23:17.157 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:17.157 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.157 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.157 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:23:17.157 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.056087+0000", 2024-09-13T11:23:17.157 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.0659999999999997e-06 2024-09-13T11:23:17.157 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.157 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.157 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:23:17.157 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.056148+0000", 2024-09-13T11:23:17.158 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.1376000000000003e-05 2024-09-13T11:23:17.158 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.158 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.158 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:23:17.158 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.056180+0000", 2024-09-13T11:23:17.158 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.2370999999999999e-05 2024-09-13T11:23:17.158 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.158 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.158 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:23:17.158 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.056195+0000", 2024-09-13T11:23:17.158 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.5119e-05 2024-09-13T11:23:17.158 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.158 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.159 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:23:17.159 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.056304+0000", 2024-09-13T11:23:17.159 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000108365 2024-09-13T11:23:17.159 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.159 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.159 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:23:17.159 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.056478+0000", 2024-09-13T11:23:17.159 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000174542 2024-09-13T11:23:17.159 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.159 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.159 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 11", 2024-09-13T11:23:17.159 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.056690+0000", 2024-09-13T11:23:17.160 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00021167500000000001 2024-09-13T11:23:17.160 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.160 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.160 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:23:17.160 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.058063+0000", 2024-09-13T11:23:17.160 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0013731819999999999 2024-09-13T11:23:17.160 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.160 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.160 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:17.160 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.950477+0000", 2024-09-13T11:23:17.160 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.89241369400000004 2024-09-13T11:23:17.160 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.160 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.161 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:23:17.161 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.950652+0000", 2024-09-13T11:23:17.161 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00017526400000000001 2024-09-13T11:23:17.161 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.161 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.161 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:23:17.161 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.950780+0000", 2024-09-13T11:23:17.161 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00012764800000000001 2024-09-13T11:23:17.161 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:17.161 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:23:17.161 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:17.161 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.161 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.162 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:7 3.b 3:de78393b:::benchmark_data_smithi177_31116_object6:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T11:23:17.162 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:48.313952+0000", 2024-09-13T11:23:17.162 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 28.838372446000001, 2024-09-13T11:23:17.162 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.86421288600000001, 2024-09-13T11:23:17.162 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:23:17.162 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:23:17.162 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:17.162 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:23:17.162 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:23:17.162 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:17.162 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 7 2024-09-13T11:23:17.162 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.162 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:23:17.162 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.163 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:23:17.163 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.313952+0000", 2024-09-13T11:23:17.163 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:17.163 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.163 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.163 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:23:17.163 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.313952+0000", 2024-09-13T11:23:17.163 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:17.163 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.163 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.163 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:23:17.163 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.313957+0000", 2024-09-13T11:23:17.164 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.3599999999999998e-06 2024-09-13T11:23:17.164 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.164 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.164 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:23:17.164 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314043+0000", 2024-09-13T11:23:17.164 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.6766000000000002e-05 2024-09-13T11:23:17.164 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.164 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.164 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:23:17.164 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314065+0000", 2024-09-13T11:23:17.164 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.1416e-05 2024-09-13T11:23:17.164 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.164 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.164 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:23:17.165 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314080+0000", 2024-09-13T11:23:17.165 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.5519000000000002e-05 2024-09-13T11:23:17.165 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.165 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.165 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:23:17.165 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315478+0000", 2024-09-13T11:23:17.166 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.001397765 2024-09-13T11:23:17.166 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.166 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.166 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-09-13T11:23:17.166 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315507+0000", 2024-09-13T11:23:17.166 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.9128999999999998e-05 2024-09-13T11:23:17.166 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.166 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.166 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:23:17.166 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.167978+0000", 2024-09-13T11:23:17.166 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.85247068400000003 2024-09-13T11:23:17.166 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.166 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.166 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:23:17.167 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.168307+0000", 2024-09-13T11:23:17.167 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00032886699999999999 2024-09-13T11:23:17.167 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.167 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.167 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1", 2024-09-13T11:23:17.167 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.168647+0000", 2024-09-13T11:23:17.167 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00033998399999999999 2024-09-13T11:23:17.167 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.167 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.167 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:17.167 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.178011+0000", 2024-09-13T11:23:17.167 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0093639169999999994 2024-09-13T11:23:17.167 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.168 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.168 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:23:17.168 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.178026+0000", 2024-09-13T11:23:17.168 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.5274999999999999e-05 2024-09-13T11:23:17.168 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.168 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.168 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:23:17.168 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.178080+0000", 2024-09-13T11:23:17.168 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.3601999999999999e-05 2024-09-13T11:23:17.168 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.168 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.168 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:23:17.168 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.178165+0000", 2024-09-13T11:23:17.169 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.5601999999999998e-05 2024-09-13T11:23:17.169 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:17.169 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:23:17.169 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:17.169 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.169 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.169 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:9 3.b 3:dafadff6:::benchmark_data_smithi177_31116_object8:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T11:23:17.169 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:48.314114+0000", 2024-09-13T11:23:17.169 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 28.838210631999999, 2024-09-13T11:23:17.169 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.86613615899999996, 2024-09-13T11:23:17.169 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:23:17.169 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:23:17.169 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:17.170 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:23:17.170 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:23:17.170 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:17.170 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 9 2024-09-13T11:23:17.170 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.170 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:23:17.170 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.170 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:23:17.171 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314114+0000", 2024-09-13T11:23:17.171 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:17.171 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.171 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.171 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:23:17.171 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314114+0000", 2024-09-13T11:23:17.171 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:17.171 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.171 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.171 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:23:17.171 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314117+0000", 2024-09-13T11:23:17.171 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.3860000000000001e-06 2024-09-13T11:23:17.172 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.172 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.172 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:23:17.172 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314215+0000", 2024-09-13T11:23:17.172 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.8900000000000005e-05 2024-09-13T11:23:17.172 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.172 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.172 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:23:17.172 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314229+0000", 2024-09-13T11:23:17.172 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.3444999999999999e-05 2024-09-13T11:23:17.172 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.172 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.172 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:23:17.172 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314239+0000", 2024-09-13T11:23:17.173 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.8470000000000008e-06 2024-09-13T11:23:17.173 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.173 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.173 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:23:17.173 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315648+0000", 2024-09-13T11:23:17.173 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.001409291 2024-09-13T11:23:17.173 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.173 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.173 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-09-13T11:23:17.173 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315671+0000", 2024-09-13T11:23:17.173 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.3169000000000001e-05 2024-09-13T11:23:17.174 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.174 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.174 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:23:17.174 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.176036+0000", 2024-09-13T11:23:17.174 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.86036497499999998 2024-09-13T11:23:17.174 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.174 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.174 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:23:17.175 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.176189+0000", 2024-09-13T11:23:17.175 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00015271100000000001 2024-09-13T11:23:17.175 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.175 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.175 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1", 2024-09-13T11:23:17.175 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.176339+0000", 2024-09-13T11:23:17.175 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00015057499999999999 2024-09-13T11:23:17.175 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.176 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.176 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:17.176 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.180148+0000", 2024-09-13T11:23:17.176 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.003808891 2024-09-13T11:23:17.176 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.176 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.176 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:23:17.176 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.180168+0000", 2024-09-13T11:23:17.176 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.9507000000000001e-05 2024-09-13T11:23:17.176 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.176 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.176 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:23:17.176 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.180213+0000", 2024-09-13T11:23:17.176 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.5504999999999999e-05 2024-09-13T11:23:17.177 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.177 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.177 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:23:17.177 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.180250+0000", 2024-09-13T11:23:17.177 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.6956999999999999e-05 2024-09-13T11:23:17.177 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:17.177 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:23:17.177 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:17.177 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.177 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.177 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:10 3.b 3:dbe028a8:::benchmark_data_smithi177_31116_object9:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T11:23:17.178 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:48.314259+0000", 2024-09-13T11:23:17.178 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 28.838065501999999, 2024-09-13T11:23:17.178 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.86637313599999999, 2024-09-13T11:23:17.178 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:23:17.178 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:23:17.178 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:17.178 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:23:17.178 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:23:17.178 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:17.178 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 10 2024-09-13T11:23:17.178 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.178 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:23:17.178 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.178 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:23:17.179 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314259+0000", 2024-09-13T11:23:17.179 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:17.179 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.179 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.179 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:23:17.179 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314259+0000", 2024-09-13T11:23:17.179 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:17.179 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.179 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.179 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:23:17.179 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314261+0000", 2024-09-13T11:23:17.179 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.1500000000000002e-06 2024-09-13T11:23:17.179 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.179 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.180 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:23:17.180 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314349+0000", 2024-09-13T11:23:17.180 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.8070000000000002e-05 2024-09-13T11:23:17.180 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.180 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.180 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:23:17.180 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314365+0000", 2024-09-13T11:23:17.180 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.5835999999999999e-05 2024-09-13T11:23:17.180 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.180 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.180 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:23:17.180 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314377+0000", 2024-09-13T11:23:17.180 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.1589e-05 2024-09-13T11:23:17.181 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.181 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.181 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:23:17.181 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315788+0000", 2024-09-13T11:23:17.181 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0014109579999999999 2024-09-13T11:23:17.181 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.181 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.181 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-09-13T11:23:17.181 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315809+0000", 2024-09-13T11:23:17.181 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.0985999999999999e-05 2024-09-13T11:23:17.181 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.181 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.181 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:23:17.182 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.176822+0000", 2024-09-13T11:23:17.182 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.86101310799999997 2024-09-13T11:23:17.182 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.182 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.182 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:23:17.182 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.177021+0000", 2024-09-13T11:23:17.182 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00019911199999999999 2024-09-13T11:23:17.182 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.183 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.183 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1", 2024-09-13T11:23:17.183 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.177226+0000", 2024-09-13T11:23:17.183 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00020516600000000001 2024-09-13T11:23:17.183 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.183 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.183 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:23:17.183 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.180264+0000", 2024-09-13T11:23:17.183 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0030381739999999998 2024-09-13T11:23:17.183 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.183 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.183 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:17.183 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.180387+0000", 2024-09-13T11:23:17.183 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00012257699999999999 2024-09-13T11:23:17.184 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.184 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.184 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:23:17.184 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.180518+0000", 2024-09-13T11:23:17.184 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00013135800000000001 2024-09-13T11:23:17.184 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.184 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.184 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:23:17.184 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.180632+0000", 2024-09-13T11:23:17.184 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000114052 2024-09-13T11:23:17.184 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:17.184 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:23:17.184 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:17.185 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.185 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.185 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:39 3.b 3:d91854b3:::benchmark_data_smithi177_31116_object38:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T11:23:17.185 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:48.314393+0000", 2024-09-13T11:23:17.185 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 28.837931959999999, 2024-09-13T11:23:17.185 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.86850121700000005, 2024-09-13T11:23:17.185 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:23:17.185 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:23:17.185 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:17.185 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:23:17.186 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:23:17.186 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:17.186 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 39 2024-09-13T11:23:17.186 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.186 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:23:17.186 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.186 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:23:17.186 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314393+0000", 2024-09-13T11:23:17.186 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:17.186 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.186 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.186 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:23:17.186 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314393+0000", 2024-09-13T11:23:17.187 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:17.187 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.187 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.187 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:23:17.187 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314394+0000", 2024-09-13T11:23:17.187 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.513e-06 2024-09-13T11:23:17.187 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.187 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.187 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:23:17.187 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314463+0000", 2024-09-13T11:23:17.187 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.8261000000000003e-05 2024-09-13T11:23:17.187 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.187 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.188 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:23:17.188 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314477+0000", 2024-09-13T11:23:17.188 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.4619e-05 2024-09-13T11:23:17.188 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.188 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.188 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:23:17.188 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314485+0000", 2024-09-13T11:23:17.188 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.1699999999999997e-06 2024-09-13T11:23:17.188 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.188 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.188 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:23:17.188 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315917+0000", 2024-09-13T11:23:17.189 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0014316089999999999 2024-09-13T11:23:17.189 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.189 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.192 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-09-13T11:23:17.192 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315933+0000", 2024-09-13T11:23:17.192 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.5999999999999999e-05 2024-09-13T11:23:17.192 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.192 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.193 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:23:17.193 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.178205+0000", 2024-09-13T11:23:17.193 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.86227247100000004 2024-09-13T11:23:17.193 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.193 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.193 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:23:17.194 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.178316+0000", 2024-09-13T11:23:17.194 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000110788 2024-09-13T11:23:17.194 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.194 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.194 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1", 2024-09-13T11:23:17.194 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.178448+0000", 2024-09-13T11:23:17.195 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000132056 2024-09-13T11:23:17.195 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.195 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.195 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:23:17.195 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.182721+0000", 2024-09-13T11:23:17.195 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0042729780000000002 2024-09-13T11:23:17.196 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.196 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.196 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:17.196 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.182801+0000", 2024-09-13T11:23:17.196 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.0209000000000004e-05 2024-09-13T11:23:17.196 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.197 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.197 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:23:17.197 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.182855+0000", 2024-09-13T11:23:17.197 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.3749000000000003e-05 2024-09-13T11:23:17.197 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.197 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.198 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:23:17.198 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.182894+0000", 2024-09-13T11:23:17.198 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.8794000000000002e-05 2024-09-13T11:23:17.198 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:17.198 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:23:17.199 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:17.199 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.199 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.199 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:50 3.b 3:d1c144b8:::benchmark_data_smithi177_31116_object49:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T11:23:17.199 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:48.314498+0000", 2024-09-13T11:23:17.199 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 28.837826544999999, 2024-09-13T11:23:17.200 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.86878673299999998, 2024-09-13T11:23:17.200 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:23:17.200 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:23:17.200 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:17.200 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:23:17.200 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:23:17.201 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:17.201 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 50 2024-09-13T11:23:17.201 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.201 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:23:17.201 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.201 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:23:17.202 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314498+0000", 2024-09-13T11:23:17.202 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:17.202 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.202 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.202 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:23:17.202 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314498+0000", 2024-09-13T11:23:17.202 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:17.202 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.202 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.202 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:23:17.202 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314499+0000", 2024-09-13T11:23:17.202 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.1969999999999999e-06 2024-09-13T11:23:17.202 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.203 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.203 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:23:17.203 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314565+0000", 2024-09-13T11:23:17.203 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.5628000000000003e-05 2024-09-13T11:23:17.203 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.203 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.203 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:23:17.203 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314579+0000", 2024-09-13T11:23:17.203 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.4222e-05 2024-09-13T11:23:17.203 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.203 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.204 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:23:17.204 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314590+0000", 2024-09-13T11:23:17.204 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.1240000000000001e-05 2024-09-13T11:23:17.204 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.204 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.204 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:23:17.204 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316060+0000", 2024-09-13T11:23:17.204 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.001469515 2024-09-13T11:23:17.204 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.204 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.204 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-09-13T11:23:17.205 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316076+0000", 2024-09-13T11:23:17.205 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.5608999999999999e-05 2024-09-13T11:23:17.205 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.205 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.205 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:23:17.205 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.178857+0000", 2024-09-13T11:23:17.205 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.86278099900000005 2024-09-13T11:23:17.205 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.205 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.205 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:23:17.205 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.178961+0000", 2024-09-13T11:23:17.205 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000104664 2024-09-13T11:23:17.205 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.206 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.206 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1", 2024-09-13T11:23:17.206 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.179123+0000", 2024-09-13T11:23:17.206 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000161738 2024-09-13T11:23:17.206 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.206 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.206 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:23:17.206 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.182755+0000", 2024-09-13T11:23:17.206 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0036323200000000001 2024-09-13T11:23:17.206 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.206 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.207 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:17.207 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.183023+0000", 2024-09-13T11:23:17.207 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00026749100000000002 2024-09-13T11:23:17.207 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.207 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.207 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:23:17.207 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.183168+0000", 2024-09-13T11:23:17.207 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00014499099999999999 2024-09-13T11:23:17.207 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.207 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.207 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:23:17.207 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.183285+0000", 2024-09-13T11:23:17.207 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000117119 2024-09-13T11:23:17.208 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:17.208 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:23:17.208 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:17.208 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.208 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.208 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:60 3.b 3:d67218b0:::benchmark_data_smithi177_31116_object59:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T11:23:17.208 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:48.314603+0000", 2024-09-13T11:23:17.208 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 28.837721982000001, 2024-09-13T11:23:17.208 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.87106201100000002, 2024-09-13T11:23:17.209 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:23:17.209 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:23:17.209 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:17.209 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:23:17.209 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:23:17.209 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:17.209 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 60 2024-09-13T11:23:17.209 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.209 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:23:17.209 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.209 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:23:17.209 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314603+0000", 2024-09-13T11:23:17.209 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:17.210 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.210 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.210 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:23:17.210 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314603+0000", 2024-09-13T11:23:17.210 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:17.210 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.210 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.210 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:23:17.210 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314604+0000", 2024-09-13T11:23:17.210 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.2359999999999999e-06 2024-09-13T11:23:17.210 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.210 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.210 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:23:17.211 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314667+0000", 2024-09-13T11:23:17.211 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.3077999999999995e-05 2024-09-13T11:23:17.211 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.211 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.211 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:23:17.211 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314699+0000", 2024-09-13T11:23:17.212 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.1965999999999999e-05 2024-09-13T11:23:17.212 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.212 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.212 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:23:17.212 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314710+0000", 2024-09-13T11:23:17.213 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.1148999999999999e-05 2024-09-13T11:23:17.213 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.213 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.213 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:23:17.213 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316291+0000", 2024-09-13T11:23:17.213 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0015804440000000001 2024-09-13T11:23:17.214 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.214 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.214 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-09-13T11:23:17.214 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316310+0000", 2024-09-13T11:23:17.214 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.9799000000000001e-05 2024-09-13T11:23:17.214 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.214 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.215 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:23:17.215 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.179555+0000", 2024-09-13T11:23:17.215 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.86324469500000001 2024-09-13T11:23:17.215 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.215 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.215 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:23:17.215 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.179666+0000", 2024-09-13T11:23:17.216 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000110443 2024-09-13T11:23:17.216 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.216 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.216 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1", 2024-09-13T11:23:17.216 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.179791+0000", 2024-09-13T11:23:17.216 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00012560499999999999 2024-09-13T11:23:17.216 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.216 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.217 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:23:17.217 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.182761+0000", 2024-09-13T11:23:17.217 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0029697019999999998 2024-09-13T11:23:17.217 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.217 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.218 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:17.218 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.185572+0000", 2024-09-13T11:23:17.218 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.002811512 2024-09-13T11:23:17.218 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.218 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.218 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:23:17.219 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.185625+0000", 2024-09-13T11:23:17.219 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.2716000000000001e-05 2024-09-13T11:23:17.219 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.219 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.219 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:23:17.219 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.185665+0000", 2024-09-13T11:23:17.220 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.9666000000000003e-05 2024-09-13T11:23:17.220 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:17.220 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:23:17.220 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:17.220 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.220 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.220 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:62 3.b 3:d20ae5a8:::benchmark_data_smithi177_31116_object61:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T11:23:17.220 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:48.314723+0000", 2024-09-13T11:23:17.221 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 28.83760144, 2024-09-13T11:23:17.221 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.87338985099999999, 2024-09-13T11:23:17.221 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:23:17.221 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:23:17.221 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:17.221 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:23:17.221 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:23:17.221 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:17.221 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 62 2024-09-13T11:23:17.221 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.221 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:23:17.221 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.221 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:23:17.222 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314723+0000", 2024-09-13T11:23:17.222 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:17.222 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.222 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.222 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:23:17.222 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314723+0000", 2024-09-13T11:23:17.222 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:17.222 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.222 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.222 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:23:17.222 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314725+0000", 2024-09-13T11:23:17.222 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.3069999999999999e-06 2024-09-13T11:23:17.222 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.222 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.223 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:23:17.223 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314797+0000", 2024-09-13T11:23:17.223 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.2652999999999997e-05 2024-09-13T11:23:17.223 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.223 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.223 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:23:17.223 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314809+0000", 2024-09-13T11:23:17.223 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.1496e-05 2024-09-13T11:23:17.223 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.223 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.223 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:23:17.223 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314816+0000", 2024-09-13T11:23:17.223 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.6639999999999998e-06 2024-09-13T11:23:17.224 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.224 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.224 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:23:17.224 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316424+0000", 2024-09-13T11:23:17.224 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0016077559999999999 2024-09-13T11:23:17.224 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.224 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.224 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-09-13T11:23:17.224 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316441+0000", 2024-09-13T11:23:17.224 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.7305000000000001e-05 2024-09-13T11:23:17.224 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.224 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.225 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:23:17.225 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.180688+0000", 2024-09-13T11:23:17.225 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.86424692800000003 2024-09-13T11:23:17.225 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.225 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.225 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:23:17.225 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.180807+0000", 2024-09-13T11:23:17.225 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000118977 2024-09-13T11:23:17.225 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.225 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.226 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1", 2024-09-13T11:23:17.226 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.180938+0000", 2024-09-13T11:23:17.226 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00013049700000000001 2024-09-13T11:23:17.226 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.226 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.226 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:23:17.226 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.185680+0000", 2024-09-13T11:23:17.226 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0047417029999999999 2024-09-13T11:23:17.226 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.226 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.226 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:17.226 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.187856+0000", 2024-09-13T11:23:17.226 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0021760550000000001 2024-09-13T11:23:17.226 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.227 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.227 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:23:17.227 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.188007+0000", 2024-09-13T11:23:17.227 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00015185099999999999 2024-09-13T11:23:17.227 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.227 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.227 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:23:17.227 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.188113+0000", 2024-09-13T11:23:17.227 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000105659 2024-09-13T11:23:17.227 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:17.227 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:23:17.227 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:17.227 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.228 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.228 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:73 3.b 3:d0060d5f:::benchmark_data_smithi177_31116_object72:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T11:23:17.228 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:48.314829+0000", 2024-09-13T11:23:17.228 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 28.837496090999998, 2024-09-13T11:23:17.228 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.87364757199999998, 2024-09-13T11:23:17.228 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:23:17.228 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:23:17.228 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:17.228 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:23:17.228 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:23:17.228 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:17.228 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 73 2024-09-13T11:23:17.228 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.228 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:23:17.229 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.229 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:23:17.229 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314829+0000", 2024-09-13T11:23:17.229 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:17.229 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.229 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.229 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:23:17.229 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314829+0000", 2024-09-13T11:23:17.229 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:17.230 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.230 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.230 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:23:17.230 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314830+0000", 2024-09-13T11:23:17.230 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.243e-06 2024-09-13T11:23:17.230 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.230 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.230 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:23:17.230 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314965+0000", 2024-09-13T11:23:17.230 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00013470800000000001 2024-09-13T11:23:17.230 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.230 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.230 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:23:17.231 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314979+0000", 2024-09-13T11:23:17.231 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.4375999999999999e-05 2024-09-13T11:23:17.231 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.231 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.231 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:23:17.231 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314988+0000", 2024-09-13T11:23:17.231 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.8570000000000001e-06 2024-09-13T11:23:17.231 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.231 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.231 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:23:17.231 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316545+0000", 2024-09-13T11:23:17.231 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.001556878 2024-09-13T11:23:17.232 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.232 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.232 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-09-13T11:23:17.232 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316556+0000", 2024-09-13T11:23:17.232 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.1453e-05 2024-09-13T11:23:17.232 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.232 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.232 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:23:17.232 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.182124+0000", 2024-09-13T11:23:17.232 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.86556744200000002 2024-09-13T11:23:17.232 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.232 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.232 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:23:17.232 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.182246+0000", 2024-09-13T11:23:17.233 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00012210500000000001 2024-09-13T11:23:17.233 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.233 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.233 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1", 2024-09-13T11:23:17.233 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.182375+0000", 2024-09-13T11:23:17.233 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00012920999999999999 2024-09-13T11:23:17.233 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.233 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.233 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:23:17.233 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.185684+0000", 2024-09-13T11:23:17.233 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0033093969999999999 2024-09-13T11:23:17.233 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.234 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.234 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:17.234 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.188244+0000", 2024-09-13T11:23:17.234 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0025601550000000002 2024-09-13T11:23:17.234 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.234 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.234 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:23:17.234 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.188379+0000", 2024-09-13T11:23:17.234 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00013463400000000001 2024-09-13T11:23:17.234 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.234 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.234 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:23:17.235 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.188476+0000", 2024-09-13T11:23:17.235 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.7114000000000006e-05 2024-09-13T11:23:17.235 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:17.235 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:23:17.235 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:17.235 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.235 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.235 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:81 3.b 3:d705fd70:::benchmark_data_smithi177_31116_object80:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T11:23:17.236 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:48.315000+0000", 2024-09-13T11:23:17.236 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 28.837324253999999, 2024-09-13T11:23:17.236 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.87561418300000005, 2024-09-13T11:23:17.236 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:23:17.236 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:23:17.236 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:17.236 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:23:17.237 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:23:17.237 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:17.237 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 81 2024-09-13T11:23:17.237 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.237 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:23:17.237 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.237 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:23:17.237 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315000+0000", 2024-09-13T11:23:17.237 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:17.237 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.237 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.237 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:23:17.238 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315000+0000", 2024-09-13T11:23:17.238 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:17.238 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.238 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.238 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:23:17.238 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315002+0000", 2024-09-13T11:23:17.238 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.3799999999999999e-06 2024-09-13T11:23:17.238 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.238 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.238 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:23:17.238 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315119+0000", 2024-09-13T11:23:17.238 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000116925 2024-09-13T11:23:17.238 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.239 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.239 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:23:17.239 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315133+0000", 2024-09-13T11:23:17.239 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.3971999999999999e-05 2024-09-13T11:23:17.239 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.239 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.239 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:23:17.239 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315142+0000", 2024-09-13T11:23:17.239 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.4399999999999994e-06 2024-09-13T11:23:17.239 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.239 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.239 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:23:17.239 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316652+0000", 2024-09-13T11:23:17.239 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.001509907 2024-09-13T11:23:17.240 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.240 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.240 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-09-13T11:23:17.240 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316668+0000", 2024-09-13T11:23:17.240 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.5945999999999999e-05 2024-09-13T11:23:17.240 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.240 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.240 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:23:17.240 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.183405+0000", 2024-09-13T11:23:17.240 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.866737384 2024-09-13T11:23:17.240 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.240 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.240 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:23:17.241 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.183701+0000", 2024-09-13T11:23:17.241 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00029511099999999997 2024-09-13T11:23:17.241 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.241 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.241 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1", 2024-09-13T11:23:17.241 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.184078+0000", 2024-09-13T11:23:17.241 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00037741000000000001 2024-09-13T11:23:17.241 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.241 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.241 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:23:17.241 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.188504+0000", 2024-09-13T11:23:17.241 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0044260280000000003 2024-09-13T11:23:17.241 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.242 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.242 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:17.242 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.190562+0000", 2024-09-13T11:23:17.242 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0020580099999999999 2024-09-13T11:23:17.242 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.242 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.242 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:23:17.242 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.190592+0000", 2024-09-13T11:23:17.242 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.0057999999999999e-05 2024-09-13T11:23:17.242 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.242 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.242 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:23:17.243 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.190615+0000", 2024-09-13T11:23:17.243 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.2612e-05 2024-09-13T11:23:17.243 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:17.243 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:23:17.243 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:17.243 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.243 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.243 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:87 3.b 3:dd7de0ad:::benchmark_data_smithi177_31116_object86:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T11:23:17.243 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:48.315155+0000", 2024-09-13T11:23:17.243 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 28.837169457000002, 2024-09-13T11:23:17.243 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.87555916099999997, 2024-09-13T11:23:17.243 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:23:17.243 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:23:17.243 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:17.244 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:23:17.244 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:23:17.244 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:17.244 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 87 2024-09-13T11:23:17.244 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.244 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:23:17.244 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.244 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:23:17.244 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315155+0000", 2024-09-13T11:23:17.244 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:17.244 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.244 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.244 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:23:17.245 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315155+0000", 2024-09-13T11:23:17.245 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:17.245 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.245 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.245 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:23:17.245 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315157+0000", 2024-09-13T11:23:17.245 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.3829999999999999e-06 2024-09-13T11:23:17.245 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.245 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.245 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:23:17.245 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315271+0000", 2024-09-13T11:23:17.246 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000114539 2024-09-13T11:23:17.246 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.246 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.246 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:23:17.246 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315285+0000", 2024-09-13T11:23:17.246 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.4042e-05 2024-09-13T11:23:17.246 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.246 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.246 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:23:17.246 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315295+0000", 2024-09-13T11:23:17.246 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.2739999999999996e-06 2024-09-13T11:23:17.246 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.246 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.247 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:23:17.247 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316739+0000", 2024-09-13T11:23:17.247 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0014446960000000001 2024-09-13T11:23:17.247 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.247 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.247 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-09-13T11:23:17.247 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316747+0000", 2024-09-13T11:23:17.247 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.0790000000000001e-06 2024-09-13T11:23:17.247 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.247 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.247 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:23:17.247 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.184922+0000", 2024-09-13T11:23:17.247 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.86817446399999998 2024-09-13T11:23:17.247 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.248 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.248 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:23:17.248 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.185037+0000", 2024-09-13T11:23:17.248 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00011565099999999999 2024-09-13T11:23:17.248 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.248 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.248 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1", 2024-09-13T11:23:17.248 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.185171+0000", 2024-09-13T11:23:17.248 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00013316900000000001 2024-09-13T11:23:17.249 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.249 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.249 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:23:17.249 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.188520+0000", 2024-09-13T11:23:17.249 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0033489150000000001 2024-09-13T11:23:17.249 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.250 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.250 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:17.250 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.190652+0000", 2024-09-13T11:23:17.250 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0021325649999999999 2024-09-13T11:23:17.250 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.250 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.250 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:23:17.250 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.190686+0000", 2024-09-13T11:23:17.251 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.3827e-05 2024-09-13T11:23:17.251 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.251 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.251 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:23:17.251 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.190714+0000", 2024-09-13T11:23:17.251 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.8557000000000001e-05 2024-09-13T11:23:17.251 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:17.251 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:23:17.251 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:17.251 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.251 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.251 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:99 3.b 3:d011b600:::benchmark_data_smithi177_31116_object98:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T11:23:17.251 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:48.315307+0000", 2024-09-13T11:23:17.251 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 28.837017629999998, 2024-09-13T11:23:17.252 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.87551770799999995, 2024-09-13T11:23:17.252 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:23:17.252 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:23:17.252 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:17.252 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:23:17.252 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:23:17.252 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:17.252 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 99 2024-09-13T11:23:17.252 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.252 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:23:17.252 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.253 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:23:17.253 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315307+0000", 2024-09-13T11:23:17.253 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:17.253 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.253 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.254 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:23:17.254 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315307+0000", 2024-09-13T11:23:17.254 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:17.254 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.254 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.254 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:23:17.254 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315308+0000", 2024-09-13T11:23:17.254 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.316e-06 2024-09-13T11:23:17.254 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.254 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.254 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:23:17.254 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315453+0000", 2024-09-13T11:23:17.255 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00014473200000000001 2024-09-13T11:23:17.255 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.255 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.255 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:23:17.255 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315471+0000", 2024-09-13T11:23:17.255 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.8019000000000001e-05 2024-09-13T11:23:17.255 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.255 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.255 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:23:17.255 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315489+0000", 2024-09-13T11:23:17.255 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.7861999999999999e-05 2024-09-13T11:23:17.255 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.255 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.256 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:23:17.256 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316837+0000", 2024-09-13T11:23:17.256 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.001347787 2024-09-13T11:23:17.256 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.256 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.256 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-09-13T11:23:17.256 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316858+0000", 2024-09-13T11:23:17.256 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.1103000000000001e-05 2024-09-13T11:23:17.256 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.256 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.256 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:23:17.256 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.185713+0000", 2024-09-13T11:23:17.256 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.86885540800000005 2024-09-13T11:23:17.257 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.257 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.257 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:23:17.257 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.185820+0000", 2024-09-13T11:23:17.257 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000107081 2024-09-13T11:23:17.257 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.257 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.257 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1", 2024-09-13T11:23:17.257 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.185948+0000", 2024-09-13T11:23:17.257 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00012775800000000001 2024-09-13T11:23:17.257 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.257 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.257 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:23:17.257 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.190527+0000", 2024-09-13T11:23:17.257 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0045783910000000002 2024-09-13T11:23:17.258 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.258 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.258 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:17.258 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.190759+0000", 2024-09-13T11:23:17.258 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00023266199999999999 2024-09-13T11:23:17.258 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.258 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.258 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:23:17.258 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.190789+0000", 2024-09-13T11:23:17.258 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.9666999999999999e-05 2024-09-13T11:23:17.258 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.258 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.258 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:23:17.259 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.190825+0000", 2024-09-13T11:23:17.259 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.5921999999999999e-05 2024-09-13T11:23:17.259 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:17.259 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:23:17.259 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:17.259 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.259 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.259 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:108 3.b 3:d5c90217:::benchmark_data_smithi177_31116_object107:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T11:23:17.259 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:48.315513+0000", 2024-09-13T11:23:17.259 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 28.836811924999999, 2024-09-13T11:23:17.259 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.87544772500000001, 2024-09-13T11:23:17.259 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:23:17.259 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:23:17.259 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:17.260 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:23:17.260 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:23:17.260 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:17.260 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 108 2024-09-13T11:23:17.260 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.260 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:23:17.260 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.260 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:23:17.260 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315513+0000", 2024-09-13T11:23:17.260 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:17.260 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.260 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.261 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:23:17.261 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315513+0000", 2024-09-13T11:23:17.261 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:17.261 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.261 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.261 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:23:17.261 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315515+0000", 2024-09-13T11:23:17.261 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.9369999999999998e-06 2024-09-13T11:23:17.261 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.261 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.261 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:23:17.261 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315644+0000", 2024-09-13T11:23:17.261 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000129011 2024-09-13T11:23:17.262 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.262 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.262 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:23:17.262 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315657+0000", 2024-09-13T11:23:17.262 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.3689999999999999e-05 2024-09-13T11:23:17.262 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.262 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.262 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:23:17.262 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315668+0000", 2024-09-13T11:23:17.262 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0589e-05 2024-09-13T11:23:17.262 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.262 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.263 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:23:17.263 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316925+0000", 2024-09-13T11:23:17.263 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0012568340000000001 2024-09-13T11:23:17.263 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.263 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.263 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-09-13T11:23:17.263 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316932+0000", 2024-09-13T11:23:17.263 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.2230000000000001e-06 2024-09-13T11:23:17.263 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.263 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.263 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:23:17.263 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.186619+0000", 2024-09-13T11:23:17.263 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.86968679599999998 2024-09-13T11:23:17.264 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.264 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.264 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:23:17.264 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.186914+0000", 2024-09-13T11:23:17.264 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00029528900000000001 2024-09-13T11:23:17.264 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.264 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.264 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1", 2024-09-13T11:23:17.264 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.187217+0000", 2024-09-13T11:23:17.264 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00030232099999999998 2024-09-13T11:23:17.264 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.264 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.264 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:23:17.265 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.190532+0000", 2024-09-13T11:23:17.265 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.003315391 2024-09-13T11:23:17.265 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.265 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.265 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:17.265 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.190875+0000", 2024-09-13T11:23:17.265 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00034326300000000001 2024-09-13T11:23:17.265 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.265 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.265 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:23:17.266 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.190925+0000", 2024-09-13T11:23:17.266 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.9429999999999999e-05 2024-09-13T11:23:17.266 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.266 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.266 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:23:17.266 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.190961+0000", 2024-09-13T11:23:17.266 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.5951000000000002e-05 2024-09-13T11:23:17.266 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:17.266 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:23:17.266 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:17.266 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.266 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.266 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:172 3.b 3:dbec1eeb:::benchmark_data_smithi177_31116_object171:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T11:23:17.267 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:48.315684+0000", 2024-09-13T11:23:17.267 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 28.836640767999999, 2024-09-13T11:23:17.267 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.87615312999999995, 2024-09-13T11:23:17.267 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:23:17.267 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:23:17.267 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:17.267 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:23:17.267 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:23:17.267 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:17.267 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 172 2024-09-13T11:23:17.267 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.267 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:23:17.267 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.268 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:23:17.268 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315684+0000", 2024-09-13T11:23:17.268 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:17.268 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.268 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.268 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:23:17.268 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315684+0000", 2024-09-13T11:23:17.268 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:17.268 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.268 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.268 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:23:17.268 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315686+0000", 2024-09-13T11:23:17.269 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.886e-06 2024-09-13T11:23:17.269 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.269 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.269 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:23:17.269 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315820+0000", 2024-09-13T11:23:17.269 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00013388500000000001 2024-09-13T11:23:17.269 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.269 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.269 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:23:17.269 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315837+0000", 2024-09-13T11:23:17.269 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.6756e-05 2024-09-13T11:23:17.269 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.269 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.269 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:23:17.270 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315851+0000", 2024-09-13T11:23:17.270 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.4566000000000001e-05 2024-09-13T11:23:17.270 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.270 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.270 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:23:17.270 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.317021+0000", 2024-09-13T11:23:17.270 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0011701769999999999 2024-09-13T11:23:17.270 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.270 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.270 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-09-13T11:23:17.270 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.317037+0000", 2024-09-13T11:23:17.270 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.609e-05 2024-09-13T11:23:17.270 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.271 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.271 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:23:17.271 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.188634+0000", 2024-09-13T11:23:17.271 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.87159711900000003 2024-09-13T11:23:17.271 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.271 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.271 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:23:17.271 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.188974+0000", 2024-09-13T11:23:17.271 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00033922299999999999 2024-09-13T11:23:17.271 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.271 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.272 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1", 2024-09-13T11:23:17.272 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.189510+0000", 2024-09-13T11:23:17.272 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00053618999999999997 2024-09-13T11:23:17.272 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.272 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.272 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:17.272 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.191027+0000", 2024-09-13T11:23:17.272 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0015168390000000001 2024-09-13T11:23:17.272 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.272 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.272 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:23:17.272 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.191766+0000", 2024-09-13T11:23:17.273 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000739064 2024-09-13T11:23:17.273 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.273 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.273 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:23:17.273 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.191810+0000", 2024-09-13T11:23:17.273 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.3755999999999999e-05 2024-09-13T11:23:17.273 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.273 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.273 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:23:17.273 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.191837+0000", 2024-09-13T11:23:17.273 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.7579000000000001e-05 2024-09-13T11:23:17.274 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:17.274 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:23:17.274 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:17.274 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.274 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.274 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:177 3.b 3:de89c779:::benchmark_data_smithi177_31116_object176:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T11:23:17.274 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:48.315898+0000", 2024-09-13T11:23:17.274 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 28.836426453000001, 2024-09-13T11:23:17.274 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.87600712599999997, 2024-09-13T11:23:17.274 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:23:17.274 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:23:17.274 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:17.274 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:23:17.274 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:23:17.275 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:17.275 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 177 2024-09-13T11:23:17.275 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.275 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:23:17.275 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.275 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:23:17.275 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315898+0000", 2024-09-13T11:23:17.275 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:17.275 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.276 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.276 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:23:17.276 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315898+0000", 2024-09-13T11:23:17.276 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:17.276 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.276 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.277 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:23:17.277 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315900+0000", 2024-09-13T11:23:17.277 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.4929999999999999e-06 2024-09-13T11:23:17.277 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.277 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.277 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:23:17.277 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316027+0000", 2024-09-13T11:23:17.278 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000127568 2024-09-13T11:23:17.278 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.278 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.278 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:23:17.278 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316048+0000", 2024-09-13T11:23:17.278 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.0795999999999999e-05 2024-09-13T11:23:17.279 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.279 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.279 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:23:17.279 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316061+0000", 2024-09-13T11:23:17.279 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.3283e-05 2024-09-13T11:23:17.279 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.279 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.280 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:23:17.280 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.317107+0000", 2024-09-13T11:23:17.280 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0010452949999999999 2024-09-13T11:23:17.280 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.280 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.280 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-09-13T11:23:17.280 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.317114+0000", 2024-09-13T11:23:17.280 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.4569999999999999e-06 2024-09-13T11:23:17.281 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.281 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.281 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:23:17.281 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.190017+0000", 2024-09-13T11:23:17.281 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.87290258499999995 2024-09-13T11:23:17.281 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.282 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.282 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:23:17.282 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.190138+0000", 2024-09-13T11:23:17.282 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000121418 2024-09-13T11:23:17.282 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.282 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.283 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1", 2024-09-13T11:23:17.283 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.190216+0000", 2024-09-13T11:23:17.283 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.7410999999999995e-05 2024-09-13T11:23:17.283 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.283 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.283 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:17.283 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.191755+0000", 2024-09-13T11:23:17.284 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0015395420000000001 2024-09-13T11:23:17.284 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.284 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.284 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:23:17.284 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.191843+0000", 2024-09-13T11:23:17.284 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.8022999999999994e-05 2024-09-13T11:23:17.284 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.284 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.284 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:23:17.284 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.191876+0000", 2024-09-13T11:23:17.284 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.2619000000000001e-05 2024-09-13T11:23:17.285 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.285 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.285 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:23:17.285 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.191905+0000", 2024-09-13T11:23:17.285 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.9635999999999999e-05 2024-09-13T11:23:17.285 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:17.285 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:23:17.285 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:17.285 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.285 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.285 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:187 3.b 3:da03c8dd:::benchmark_data_smithi177_31116_object186:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T11:23:17.286 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:48.316083+0000", 2024-09-13T11:23:17.286 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 28.836241497, 2024-09-13T11:23:17.286 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.87835499500000003, 2024-09-13T11:23:17.286 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:23:17.286 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:23:17.286 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:17.286 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:23:17.286 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:23:17.286 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:17.286 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 187 2024-09-13T11:23:17.286 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.286 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:23:17.286 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.287 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:23:17.287 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316083+0000", 2024-09-13T11:23:17.287 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:17.287 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.287 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.287 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:23:17.287 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316083+0000", 2024-09-13T11:23:17.287 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:17.287 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.287 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.287 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:23:17.287 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316086+0000", 2024-09-13T11:23:17.287 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.5560000000000002e-06 2024-09-13T11:23:17.288 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.288 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.288 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:23:17.288 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316199+0000", 2024-09-13T11:23:17.288 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000113522 2024-09-13T11:23:17.288 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.288 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.288 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:23:17.288 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316215+0000", 2024-09-13T11:23:17.288 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.5265999999999999e-05 2024-09-13T11:23:17.288 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.288 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.288 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:23:17.289 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316226+0000", 2024-09-13T11:23:17.289 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0967e-05 2024-09-13T11:23:17.289 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.289 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.289 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:23:17.289 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.317198+0000", 2024-09-13T11:23:17.289 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00097194100000000004 2024-09-13T11:23:17.289 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.289 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.289 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-09-13T11:23:17.289 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.317213+0000", 2024-09-13T11:23:17.289 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.5472999999999999e-05 2024-09-13T11:23:17.289 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.289 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.290 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:23:17.290 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.191077+0000", 2024-09-13T11:23:17.290 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.87386435200000001 2024-09-13T11:23:17.290 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.290 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.290 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:23:17.290 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.191181+0000", 2024-09-13T11:23:17.290 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000103618 2024-09-13T11:23:17.290 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.290 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.290 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1", 2024-09-13T11:23:17.290 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.191305+0000", 2024-09-13T11:23:17.290 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00012358599999999999 2024-09-13T11:23:17.290 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.291 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.291 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:17.291 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.194279+0000", 2024-09-13T11:23:17.291 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0029746970000000001 2024-09-13T11:23:17.291 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.291 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.291 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:23:17.291 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.194336+0000", 2024-09-13T11:23:17.291 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.7244000000000003e-05 2024-09-13T11:23:17.291 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.291 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.291 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:23:17.291 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.194397+0000", 2024-09-13T11:23:17.291 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.0278000000000001e-05 2024-09-13T11:23:17.292 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.292 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.292 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:23:17.292 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.194438+0000", 2024-09-13T11:23:17.292 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.1495000000000002e-05 2024-09-13T11:23:17.292 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:17.292 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:23:17.292 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:17.292 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.292 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.293 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:200 3.b 3:d90039f1:::benchmark_data_smithi177_31116_object199:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T11:23:17.293 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:48.316280+0000", 2024-09-13T11:23:17.293 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 28.836045032000001, 2024-09-13T11:23:17.293 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.87879273000000002, 2024-09-13T11:23:17.293 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:23:17.293 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:23:17.293 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:17.293 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:23:17.293 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:23:17.293 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:17.293 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 200 2024-09-13T11:23:17.293 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.293 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:23:17.293 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.294 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:23:17.294 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316280+0000", 2024-09-13T11:23:17.294 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:17.294 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.294 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.294 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:23:17.294 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316280+0000", 2024-09-13T11:23:17.294 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:17.294 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.294 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.294 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:23:17.294 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316282+0000", 2024-09-13T11:23:17.294 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.497e-06 2024-09-13T11:23:17.294 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.295 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.295 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:23:17.295 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316411+0000", 2024-09-13T11:23:17.295 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00012924799999999999 2024-09-13T11:23:17.295 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.295 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.295 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:23:17.295 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316426+0000", 2024-09-13T11:23:17.295 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.4183e-05 2024-09-13T11:23:17.295 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.295 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.296 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:23:17.296 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316435+0000", 2024-09-13T11:23:17.296 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.0990000000000005e-06 2024-09-13T11:23:17.296 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.296 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.296 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:23:17.296 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.317250+0000", 2024-09-13T11:23:17.296 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00081501100000000003 2024-09-13T11:23:17.296 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.296 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.296 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-09-13T11:23:17.296 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.317257+0000", 2024-09-13T11:23:17.296 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.2169999999999997e-06 2024-09-13T11:23:17.297 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.297 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.297 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:23:17.297 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.191936+0000", 2024-09-13T11:23:17.297 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.87467922099999995 2024-09-13T11:23:17.297 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.297 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.297 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:23:17.297 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.192056+0000", 2024-09-13T11:23:17.297 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000119638 2024-09-13T11:23:17.297 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.297 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.297 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1", 2024-09-13T11:23:17.298 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.192184+0000", 2024-09-13T11:23:17.298 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00012813399999999999 2024-09-13T11:23:17.298 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.298 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.298 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:17.298 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.194324+0000", 2024-09-13T11:23:17.298 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0021399420000000001 2024-09-13T11:23:17.298 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.298 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.298 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:23:17.298 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.194802+0000", 2024-09-13T11:23:17.298 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00047798799999999999 2024-09-13T11:23:17.299 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.299 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.299 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:23:17.299 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.194951+0000", 2024-09-13T11:23:17.299 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0001494 2024-09-13T11:23:17.299 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.299 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.299 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:23:17.299 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.195072+0000", 2024-09-13T11:23:17.299 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00012115199999999999 2024-09-13T11:23:17.299 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:17.299 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:23:17.299 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:17.299 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.299 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.300 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:25385 3.7 3:e9a54943:::benchmark_data_smithi177_31116_object25384:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e55)", 2024-09-13T11:23:17.300 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:04.452404+0000", 2024-09-13T11:23:17.300 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 12.699920634, 2024-09-13T11:23:17.300 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.22598749700000001, 2024-09-13T11:23:17.300 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:23:17.300 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:23:17.300 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:17.300 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:23:17.300 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:23:17.300 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:17.300 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 25385 2024-09-13T11:23:17.300 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.300 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:23:17.301 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.301 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:23:17.301 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.452404+0000", 2024-09-13T11:23:17.301 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:17.301 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.301 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.301 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:23:17.301 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.452404+0000", 2024-09-13T11:23:17.301 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:17.301 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.301 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.301 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:23:17.301 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.452406+0000", 2024-09-13T11:23:17.302 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.57e-06 2024-09-13T11:23:17.302 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.302 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.302 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:23:17.302 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.452461+0000", 2024-09-13T11:23:17.302 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.4892000000000002e-05 2024-09-13T11:23:17.302 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.302 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.302 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:23:17.302 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.452473+0000", 2024-09-13T11:23:17.302 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.2703e-05 2024-09-13T11:23:17.302 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.302 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.303 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:23:17.303 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.452482+0000", 2024-09-13T11:23:17.303 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.6999999999999997e-06 2024-09-13T11:23:17.303 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.303 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.303 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:23:17.303 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.452605+0000", 2024-09-13T11:23:17.303 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00012328099999999999 2024-09-13T11:23:17.303 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.303 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.303 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:23:17.303 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.452930+0000", 2024-09-13T11:23:17.303 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00032516000000000001 2024-09-13T11:23:17.304 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.304 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.304 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 8", 2024-09-13T11:23:17.304 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.453272+0000", 2024-09-13T11:23:17.304 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00034195100000000001 2024-09-13T11:23:17.304 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.304 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.304 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:23:17.304 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.456573+0000", 2024-09-13T11:23:17.304 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0033003619999999998 2024-09-13T11:23:17.304 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.304 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.304 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:17.305 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.678212+0000", 2024-09-13T11:23:17.305 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.22163946600000001 2024-09-13T11:23:17.305 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.305 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.305 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:23:17.305 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.678325+0000", 2024-09-13T11:23:17.305 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00011263400000000001 2024-09-13T11:23:17.305 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.305 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.305 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:23:17.305 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.678392+0000", 2024-09-13T11:23:17.305 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.6778000000000003e-05 2024-09-13T11:23:17.306 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:17.306 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:23:17.306 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:17.306 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.306 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.306 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:25525 3.1 3:8e5634af:::benchmark_data_smithi177_31116_object25524:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e56)", 2024-09-13T11:23:17.306 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:04.981590+0000", 2024-09-13T11:23:17.306 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 12.170734755, 2024-09-13T11:23:17.306 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.23642401599999999, 2024-09-13T11:23:17.307 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:23:17.307 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:23:17.307 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:17.307 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:23:17.307 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:23:17.307 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:17.307 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 25525 2024-09-13T11:23:17.307 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.307 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:23:17.307 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.307 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:23:17.307 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.981590+0000", 2024-09-13T11:23:17.307 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:17.307 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.308 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.308 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:23:17.308 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.981590+0000", 2024-09-13T11:23:17.308 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:17.308 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.308 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.308 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:23:17.308 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.981598+0000", 2024-09-13T11:23:17.308 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.4230000000000005e-06 2024-09-13T11:23:17.309 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.309 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.309 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:23:17.309 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.981652+0000", 2024-09-13T11:23:17.309 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.3226000000000002e-05 2024-09-13T11:23:17.309 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.309 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.309 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:23:17.309 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.981680+0000", 2024-09-13T11:23:17.309 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.8348999999999999e-05 2024-09-13T11:23:17.309 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.309 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.309 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:23:17.309 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.981695+0000", 2024-09-13T11:23:17.310 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.5085999999999999e-05 2024-09-13T11:23:17.310 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.310 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.310 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:23:17.310 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.981809+0000", 2024-09-13T11:23:17.310 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00011382700000000001 2024-09-13T11:23:17.310 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.310 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.310 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:23:17.310 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.982042+0000", 2024-09-13T11:23:17.310 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000232697 2024-09-13T11:23:17.310 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.310 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.311 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 11", 2024-09-13T11:23:17.311 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.982261+0000", 2024-09-13T11:23:17.311 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00021918400000000001 2024-09-13T11:23:17.311 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.311 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.311 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:23:17.311 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.984553+0000", 2024-09-13T11:23:17.311 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0022918460000000002 2024-09-13T11:23:17.311 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.311 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.311 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:17.311 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:05.217842+0000", 2024-09-13T11:23:17.311 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.233289106 2024-09-13T11:23:17.311 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.312 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.312 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:23:17.312 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:05.217940+0000", 2024-09-13T11:23:17.312 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.7936999999999994e-05 2024-09-13T11:23:17.312 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.312 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.312 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:23:17.312 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:05.218014+0000", 2024-09-13T11:23:17.312 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.4334999999999999e-05 2024-09-13T11:23:17.312 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:17.313 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:23:17.313 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:17.313 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.313 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.313 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop_reply(client.4665.0:25525 3.1 e56/30 ondisk, result = 0)", 2024-09-13T11:23:17.313 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:04.985699+0000", 2024-09-13T11:23:17.313 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 12.166626023999999, 2024-09-13T11:23:17.313 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.23233051800000001, 2024-09-13T11:23:17.313 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:23:17.313 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:23:17.314 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-09-13T11:23:17.314 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:23:17.314 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.314 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:23:17.314 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.985699+0000", 2024-09-13T11:23:17.314 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:17.314 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.314 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.314 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:23:17.314 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.985699+0000", 2024-09-13T11:23:17.314 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:17.314 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.314 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.314 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:23:17.315 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.985702+0000", 2024-09-13T11:23:17.315 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.0570000000000001e-06 2024-09-13T11:23:17.315 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.315 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.315 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:23:17.315 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.985711+0000", 2024-09-13T11:23:17.315 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.1060000000000006e-06 2024-09-13T11:23:17.315 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.315 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.315 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:23:17.315 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:05.217667+0000", 2024-09-13T11:23:17.315 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.231955774 2024-09-13T11:23:17.315 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.316 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.316 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:23:17.316 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:05.217685+0000", 2024-09-13T11:23:17.316 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.791e-05 2024-09-13T11:23:17.316 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.316 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.316 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:23:17.316 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:05.217818+0000", 2024-09-13T11:23:17.316 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000133036 2024-09-13T11:23:17.316 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.316 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.316 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:23:17.316 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:05.217833+0000", 2024-09-13T11:23:17.316 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.519e-05 2024-09-13T11:23:17.317 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:17.317 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:17.317 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:23:17.317 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:05.218029+0000", 2024-09-13T11:23:17.317 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00019644500000000001 2024-09-13T11:23:17.317 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:17.317 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:23:17.317 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:17.317 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:17.317 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:23:17.318 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-09-13T11:23:17.319 INFO:teuthology.orchestra.run.smithi001.stdout:osd.6: {} 2024-09-13T11:23:17.320 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-09-13T11:23:17.314+0000 7f97befc9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26315) UID: 0 2024-09-13T11:23:17.321 INFO:teuthology.orchestra.run.smithi001.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T11:23:17.322 INFO:teuthology.orchestra.run.smithi001.stdout:osd.7: {} 2024-09-13T11:23:17.322 INFO:teuthology.orchestra.run.smithi001.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T11:23:17.322 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-13T11:23:17.323 INFO:teuthology.orchestra.run.smithi001.stdout:osd.8: {} 2024-09-13T11:23:17.323 INFO:teuthology.orchestra.run.smithi001.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-13T11:23:17.323 INFO:teuthology.orchestra.run.smithi001.stdout:osd.9: {} 2024-09-13T11:23:17.324 INFO:teuthology.orchestra.run.smithi001.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-13T11:23:17.324 INFO:teuthology.orchestra.run.smithi001.stdout:osd.10: {} 2024-09-13T11:23:17.325 INFO:teuthology.orchestra.run.smithi001.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-13T11:23:17.326 INFO:teuthology.orchestra.run.smithi001.stdout:osd.11: {} 2024-09-13T11:23:17.326 INFO:teuthology.orchestra.run.smithi001.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-13T11:23:17.326 INFO:teuthology.orchestra.run.smithi001.stdout:osd.12: {} 2024-09-13T11:23:17.327 INFO:teuthology.orchestra.run.smithi001.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-13T11:23:17.343 INFO:teuthology.orchestra.run.smithi001.stdout:osd.13: {} 2024-09-13T11:23:17.344 INFO:teuthology.orchestra.run.smithi001.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-13T11:23:17.372 INFO:teuthology.orchestra.run.smithi001.stdout:osd.14: {} 2024-09-13T11:23:17.372 INFO:teuthology.orchestra.run.smithi001.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-13T11:23:17.399 INFO:teuthology.orchestra.run.smithi001.stdout:osd.15: {} 2024-09-13T11:23:17.399 INFO:teuthology.orchestra.run.smithi001.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-13T11:23:17.420 INFO:tasks.ceph.osd.12.smithi001.stderr:2024-09-13T11:23:17.412+0000 7f892c6e6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26314) UID: 0 2024-09-13T11:23:17.470 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-13T11:23:17.470 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:23:17.486 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-13T11:23:17.521 INFO:tasks.ceph.osd.8.smithi001.stderr:2024-09-13T11:23:17.516+0000 7fb9b48ae640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26312) UID: 0 2024-09-13T11:23:17.617 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-13T11:23:17.618 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:23:17.622 INFO:tasks.ceph.osd.15.smithi177.stderr:2024-09-13T11:23:17.618+0000 7f9d6ce77640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26578) UID: 0 2024-09-13T11:23:17.635 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-13T11:23:17.722 INFO:tasks.ceph.osd.11.smithi177.stderr:2024-09-13T11:23:17.718+0000 7f24792c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26580) UID: 0 2024-09-13T11:23:17.764 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-13T11:23:17.765 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:23:17.781 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-13T11:23:17.823 INFO:tasks.ceph.osd.3.smithi177.stderr:2024-09-13T11:23:17.822+0000 7f10adba4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26574) UID: 0 2024-09-13T11:23:17.901 INFO:teuthology.orchestra.run.smithi001.stderr:set pool 3 pg_num to 26 2024-09-13T11:23:17.904 INFO:tasks.ceph.osd.14.smithi162.stderr:2024-09-13T11:23:17.898+0000 7f2352b0c640 -1 osd.14 65 handle_osd_map injecting map crc failure 2024-09-13T11:23:17.921 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-13T11:23:17.921 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:23:17.924 INFO:tasks.ceph.osd.13.smithi092.stderr:2024-09-13T11:23:17.917+0000 7f3e5afea640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26350) UID: 0 2024-09-13T11:23:17.942 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-13T11:23:18.003 INFO:tasks.radosbench.radosbench.0.smithi177.stdout: 51 16 42893 42877 52.5394 90 0.0078282 0.0190257 2024-09-13T11:23:18.025 INFO:tasks.ceph.osd.11.smithi177.stderr:2024-09-13T11:23:18.022+0000 7f24792c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26580) UID: 0 2024-09-13T11:23:18.070 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-13T11:23:18.070 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:23:18.085 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-13T11:23:18.127 INFO:tasks.ceph.osd.13.smithi092.stderr:2024-09-13T11:23:18.121+0000 7f3e5afea640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26350) UID: 0 2024-09-13T11:23:18.220 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-13T11:23:18.272 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:23:18.273 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-09-13T11:23:18.274 INFO:tasks.ceph.osd.10.smithi162.stderr:2024-09-13T11:23:18.266+0000 7fb59c9f0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26317) UID: 0 2024-09-13T11:23:18.374 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-13T11:23:18.368+0000 7f6a68775640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26307) UID: 0 2024-09-13T11:23:18.456 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-13T11:23:18.456 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:23:18.472 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-09-13T11:23:18.474 INFO:tasks.ceph.osd.3.smithi177.stderr:2024-09-13T11:23:18.469+0000 7f10adba4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26574) UID: 0 2024-09-13T11:23:18.575 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-09-13T11:23:18.570+0000 7f97befc9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26315) UID: 0 2024-09-13T11:23:18.597 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-13T11:23:18.600 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-13T11:23:18.600 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:23:18.617 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-09-13T11:23:18.706 INFO:tasks.ceph.osd.10.smithi162.stderr:2024-09-13T11:23:18.702+0000 7fb59c9f0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26317) UID: 0 2024-09-13T11:23:18.776 INFO:tasks.ceph.osd.13.smithi092.stderr:2024-09-13T11:23:18.769+0000 7f3e5afea640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26350) UID: 0 2024-09-13T11:23:18.798 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-13T11:23:18.799 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:23:18.814 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_ops_in_flight 2024-09-13T11:23:18.906 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-09-13T11:23:18.902+0000 7f97befc9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26315) UID: 0 2024-09-13T11:23:18.911 INFO:tasks.ceph.osd.10.smithi162.stderr:2024-09-13T11:23:18.906+0000 7fb593421640 -1 osd.10 66 handle_osd_map injecting map crc failure 2024-09-13T11:23:18.912 INFO:tasks.ceph.osd.8.smithi001.stderr:2024-09-13T11:23:18.904+0000 7fb9ab2df640 -1 osd.8 66 handle_osd_map injecting map crc failure 2024-09-13T11:23:18.977 INFO:tasks.ceph.osd.1.smithi092.stderr:2024-09-13T11:23:18.973+0000 7fe7789c5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33718) UID: 0 2024-09-13T11:23:19.003 INFO:tasks.radosbench.radosbench.0.smithi177.stdout: 52 16 44093 44077 52.9712 75 0.00635658 0.0188399 2024-09-13T11:23:19.011 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-13T11:23:19.012 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:23:19.025 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_blocked_ops 2024-09-13T11:23:19.078 INFO:tasks.ceph.osd.15.smithi177.stderr:2024-09-13T11:23:19.073+0000 7f9d6ce77640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26578) UID: 0 2024-09-13T11:23:19.180 INFO:tasks.ceph.osd.15.smithi177.stderr:2024-09-13T11:23:19.177+0000 7f9d6ce77640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26578) UID: 0 2024-09-13T11:23:19.190 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-13T11:23:19.190 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:23:19.204 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_historic_ops 2024-09-13T11:23:19.280 INFO:tasks.ceph.osd.3.smithi177.stderr:2024-09-13T11:23:19.277+0000 7f10adba4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26574) UID: 0 2024-09-13T11:23:19.328 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-13T11:23:19.329 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:23:19.344 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-13T11:23:19.381 INFO:tasks.ceph.osd.8.smithi001.stderr:2024-09-13T11:23:19.376+0000 7fb9b48ae640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26312) UID: 0 2024-09-13T11:23:19.421 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T11:23:19.477 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-09-13T11:23:19.477 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:23:19.481 INFO:tasks.ceph.osd.12.smithi001.stderr:2024-09-13T11:23:19.476+0000 7f892c6e6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26314) UID: 0 2024-09-13T11:23:19.491 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-13T11:23:19.582 INFO:tasks.ceph.osd.10.smithi162.stderr:2024-09-13T11:23:19.578+0000 7fb59c9f0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26317) UID: 0 2024-09-13T11:23:19.590 INFO:teuthology.orchestra.run.smithi001.stdout:osd.0: {} 2024-09-13T11:23:19.590 INFO:teuthology.orchestra.run.smithi001.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T11:23:19.612 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-09-13T11:23:19.613 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:23:19.618 INFO:teuthology.orchestra.run.smithi001.stdout:osd.1: {} 2024-09-13T11:23:19.618 INFO:teuthology.orchestra.run.smithi001.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T11:23:19.627 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-13T11:23:19.645 INFO:teuthology.orchestra.run.smithi001.stdout:osd.2: {} 2024-09-13T11:23:19.646 INFO:teuthology.orchestra.run.smithi001.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T11:23:19.671 INFO:teuthology.orchestra.run.smithi001.stdout:osd.3: {} 2024-09-13T11:23:19.671 INFO:teuthology.orchestra.run.smithi001.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T11:23:19.683 INFO:tasks.ceph.osd.12.smithi001.stderr:2024-09-13T11:23:19.676+0000 7f892c6e6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26314) UID: 0 2024-09-13T11:23:19.695 INFO:teuthology.orchestra.run.smithi001.stdout:osd.4: {} 2024-09-13T11:23:19.695 INFO:teuthology.orchestra.run.smithi001.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T11:23:19.723 INFO:teuthology.orchestra.run.smithi001.stdout:osd.5: {} 2024-09-13T11:23:19.723 INFO:teuthology.orchestra.run.smithi001.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T11:23:19.749 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-09-13T11:23:19.749 INFO:teuthology.orchestra.run.smithi177.stdout: "size": 20, 2024-09-13T11:23:19.750 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 600, 2024-09-13T11:23:19.750 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [ 2024-09-13T11:23:19.750 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:19.750 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:837 3.a 3:5ab13ab0:::benchmark_data_smithi177_31116_object836:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-09-13T11:23:19.750 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:22:49.461597+0000", 2024-09-13T11:23:19.750 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 30.285945038000001, 2024-09-13T11:23:19.751 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.044395994000000001, 2024-09-13T11:23:19.751 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:23:19.751 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:23:19.751 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:19.751 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:23:19.752 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:23:19.752 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:19.752 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 837 2024-09-13T11:23:19.752 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:19.752 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:23:19.752 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:19.753 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:23:19.753 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.461597+0000", 2024-09-13T11:23:19.753 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:19.753 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:19.753 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:19.754 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:23:19.754 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.461597+0000", 2024-09-13T11:23:19.754 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:19.754 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:19.754 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:19.754 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:23:19.755 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.461598+0000", 2024-09-13T11:23:19.755 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.44e-06 2024-09-13T11:23:19.755 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:19.755 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:19.755 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:23:19.756 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.461634+0000", 2024-09-13T11:23:19.756 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.5673999999999998e-05 2024-09-13T11:23:19.756 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:19.756 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:19.756 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:23:19.756 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.461648+0000", 2024-09-13T11:23:19.757 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.4047e-05 2024-09-13T11:23:19.757 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:19.757 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:19.757 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:23:19.757 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.461655+0000", 2024-09-13T11:23:19.757 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.9720000000000003e-06 2024-09-13T11:23:19.758 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:19.758 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:19.758 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:23:19.758 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.461715+0000", 2024-09-13T11:23:19.758 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.0195999999999998e-05 2024-09-13T11:23:19.759 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:19.759 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:19.759 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:23:19.759 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.461819+0000", 2024-09-13T11:23:19.759 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000103447 2024-09-13T11:23:19.759 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:19.759 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:19.759 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 7", 2024-09-13T11:23:19.759 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.461939+0000", 2024-09-13T11:23:19.759 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000120913 2024-09-13T11:23:19.760 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:19.760 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:19.760 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:23:19.760 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.462669+0000", 2024-09-13T11:23:19.760 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00072908200000000004 2024-09-13T11:23:19.760 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:19.760 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:19.760 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:19.760 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.505895+0000", 2024-09-13T11:23:19.760 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.043226597999999998 2024-09-13T11:23:19.760 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:19.760 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:19.761 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:23:19.761 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.505941+0000", 2024-09-13T11:23:19.761 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.5503999999999997e-05 2024-09-13T11:23:19.761 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:19.761 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:19.761 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:23:19.762 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.505993+0000", 2024-09-13T11:23:19.762 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.2120999999999998e-05 2024-09-13T11:23:19.762 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:19.762 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:23:19.762 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:19.762 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:19.762 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:19.762 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop_reply(client.4665.0:837 3.a e43/30 ondisk, result = 0)", 2024-09-13T11:23:19.762 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:22:49.463026+0000", 2024-09-13T11:23:19.762 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 30.284515638999999, 2024-09-13T11:23:19.762 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.042975286000000001, 2024-09-13T11:23:19.763 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:23:19.763 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:23:19.763 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:23:19.763 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:23:19.763 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:19.763 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:23:19.763 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.463026+0000", 2024-09-13T11:23:19.763 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:19.763 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:19.763 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:19.763 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:23:19.763 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.463026+0000", 2024-09-13T11:23:19.763 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:19.763 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:19.764 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:19.764 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:23:19.764 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.463028+0000", 2024-09-13T11:23:19.764 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.443e-06 2024-09-13T11:23:19.764 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:19.764 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:19.764 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:23:19.764 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.463033+0000", 2024-09-13T11:23:19.764 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.7830000000000004e-06 2024-09-13T11:23:19.764 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:19.764 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:19.764 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:23:19.764 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.505813+0000", 2024-09-13T11:23:19.764 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.042779620999999997 2024-09-13T11:23:19.765 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:19.765 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:19.765 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:23:19.765 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.505820+0000", 2024-09-13T11:23:19.765 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.233e-06 2024-09-13T11:23:19.765 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:19.765 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:19.765 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:23:19.765 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.505881+0000", 2024-09-13T11:23:19.765 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.0810000000000002e-05 2024-09-13T11:23:19.765 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:19.765 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:19.765 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:23:19.766 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.505891+0000", 2024-09-13T11:23:19.766 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.6059999999999992e-06 2024-09-13T11:23:19.766 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:19.766 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:19.766 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:23:19.766 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.506001+0000", 2024-09-13T11:23:19.766 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00011079 2024-09-13T11:23:19.766 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:19.766 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:23:19.766 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:19.766 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:19.766 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:19.766 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:855 3.a 3:5ce42657:::benchmark_data_smithi177_31116_object854:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-09-13T11:23:19.767 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:22:49.480636+0000", 2024-09-13T11:23:19.767 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 30.266905443999999, 2024-09-13T11:23:19.767 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.040802389000000001, 2024-09-13T11:23:19.767 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:23:19.767 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:23:19.767 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:19.767 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:23:19.767 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:23:19.767 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:19.767 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 855 2024-09-13T11:23:19.767 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:19.767 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:23:19.767 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:19.768 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:23:19.768 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.480636+0000", 2024-09-13T11:23:19.768 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:19.768 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:19.768 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:19.768 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:23:19.768 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.480636+0000", 2024-09-13T11:23:19.768 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:19.768 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:19.768 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:19.768 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:23:19.768 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.480640+0000", 2024-09-13T11:23:19.768 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.0929999999999996e-06 2024-09-13T11:23:19.769 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:19.769 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:19.769 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:23:19.769 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.480686+0000", 2024-09-13T11:23:19.769 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.5629e-05 2024-09-13T11:23:19.769 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:19.769 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:19.769 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:23:19.769 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.480713+0000", 2024-09-13T11:23:19.770 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.7021000000000001e-05 2024-09-13T11:23:19.770 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:19.770 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:19.770 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:23:19.770 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.480730+0000", 2024-09-13T11:23:19.770 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.6835000000000001e-05 2024-09-13T11:23:19.770 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:19.770 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:19.770 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:23:19.770 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.480815+0000", 2024-09-13T11:23:19.771 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.4888000000000004e-05 2024-09-13T11:23:19.771 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:19.771 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:19.771 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:23:19.771 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.480940+0000", 2024-09-13T11:23:19.771 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00012519000000000001 2024-09-13T11:23:19.771 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:19.771 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:19.771 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 7", 2024-09-13T11:23:19.771 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.481056+0000", 2024-09-13T11:23:19.771 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000115467 2024-09-13T11:23:19.771 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:19.772 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:19.772 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:23:19.772 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.482076+0000", 2024-09-13T11:23:19.772 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001020038 2024-09-13T11:23:19.772 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:19.772 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:19.772 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:19.772 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.521353+0000", 2024-09-13T11:23:19.772 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.039277764 2024-09-13T11:23:19.773 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:19.773 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:19.773 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:23:19.773 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.521402+0000", 2024-09-13T11:23:19.773 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.8408000000000001e-05 2024-09-13T11:23:19.773 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:19.773 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:19.773 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:23:19.773 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.521439+0000", 2024-09-13T11:23:19.773 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.7055999999999999e-05 2024-09-13T11:23:19.773 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:19.773 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:23:19.774 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:19.774 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:19.774 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:19.774 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop_reply(client.4665.0:855 3.a e43/30 ondisk, result = 0)", 2024-09-13T11:23:19.774 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:22:49.482718+0000", 2024-09-13T11:23:19.774 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 30.264824076, 2024-09-13T11:23:19.774 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.038728772000000002, 2024-09-13T11:23:19.774 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:23:19.774 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:23:19.774 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:23:19.774 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:23:19.774 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:19.774 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:23:19.775 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.482718+0000", 2024-09-13T11:23:19.775 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:19.775 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:19.775 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:19.775 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:23:19.775 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.482718+0000", 2024-09-13T11:23:19.775 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:19.775 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:19.775 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:19.775 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:23:19.775 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.482719+0000", 2024-09-13T11:23:19.775 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.7269999999999999e-06 2024-09-13T11:23:19.775 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:19.775 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:19.776 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:23:19.776 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.482725+0000", 2024-09-13T11:23:19.776 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.3650000000000003e-06 2024-09-13T11:23:19.776 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:19.776 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:19.776 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:23:19.776 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.521273+0000", 2024-09-13T11:23:19.776 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.038548495000000002 2024-09-13T11:23:19.776 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:19.776 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:19.776 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:23:19.777 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.521280+0000", 2024-09-13T11:23:19.777 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.7120000000000003e-06 2024-09-13T11:23:19.777 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:19.777 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:19.777 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:23:19.777 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.521341+0000", 2024-09-13T11:23:19.777 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.0693000000000003e-05 2024-09-13T11:23:19.777 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:19.777 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:19.777 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:23:19.777 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.521349+0000", 2024-09-13T11:23:19.777 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.174e-06 2024-09-13T11:23:19.777 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:19.778 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:19.778 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:23:19.778 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.521447+0000", 2024-09-13T11:23:19.778 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.7606e-05 2024-09-13T11:23:19.778 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:19.778 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:23:19.778 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:19.778 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:19.778 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:19.778 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:861 3.a 3:5772350b:::benchmark_data_smithi177_31116_object860:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-09-13T11:23:19.778 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:22:49.487183+0000", 2024-09-13T11:23:19.778 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 30.260358368999999, 2024-09-13T11:23:19.779 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.042583651, 2024-09-13T11:23:19.779 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:23:19.779 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:23:19.779 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:19.779 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:23:19.779 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:23:19.779 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:19.779 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 861 2024-09-13T11:23:19.779 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:19.779 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:23:19.779 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:19.780 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:23:19.780 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.487183+0000", 2024-09-13T11:23:19.780 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:19.780 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:19.780 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:19.780 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:23:19.780 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.487183+0000", 2024-09-13T11:23:19.781 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:19.781 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:19.781 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:19.781 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:23:19.781 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.487186+0000", 2024-09-13T11:23:19.781 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.5409999999999999e-06 2024-09-13T11:23:19.781 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:19.781 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:19.781 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:23:19.781 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.487222+0000", 2024-09-13T11:23:19.781 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.5621000000000003e-05 2024-09-13T11:23:19.781 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:19.782 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:19.782 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:23:19.782 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.487243+0000", 2024-09-13T11:23:19.782 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.0902000000000001e-05 2024-09-13T11:23:19.782 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:19.782 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:19.782 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:23:19.782 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.487253+0000", 2024-09-13T11:23:19.782 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0667e-05 2024-09-13T11:23:19.783 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:19.783 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:19.783 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:23:19.783 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.487304+0000", 2024-09-13T11:23:19.783 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.0658000000000002e-05 2024-09-13T11:23:19.783 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:19.783 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:19.783 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:23:19.783 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.487392+0000", 2024-09-13T11:23:19.783 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.7824999999999993e-05 2024-09-13T11:23:19.783 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:19.783 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:19.784 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 7", 2024-09-13T11:23:19.784 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.487483+0000", 2024-09-13T11:23:19.784 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.0915999999999995e-05 2024-09-13T11:23:19.784 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:19.784 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:19.784 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:23:19.784 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.488424+0000", 2024-09-13T11:23:19.784 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00094114899999999996 2024-09-13T11:23:19.784 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:19.784 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:19.784 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:19.784 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.529663+0000", 2024-09-13T11:23:19.784 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.041239215000000003 2024-09-13T11:23:19.784 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:19.785 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:19.785 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:23:19.785 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.529719+0000", 2024-09-13T11:23:19.785 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.6480999999999997e-05 2024-09-13T11:23:19.785 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:19.785 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:19.785 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:23:19.785 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.529767+0000", 2024-09-13T11:23:19.785 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.7676000000000002e-05 2024-09-13T11:23:19.785 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:19.785 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:23:19.785 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:19.786 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:19.786 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:19.786 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop_reply(client.4665.0:861 3.a e43/30 ondisk, result = 0)", 2024-09-13T11:23:19.786 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:22:49.488684+0000", 2024-09-13T11:23:19.786 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 30.258857917, 2024-09-13T11:23:19.786 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.041092947999999997, 2024-09-13T11:23:19.786 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:23:19.786 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:23:19.786 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:23:19.786 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:23:19.786 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:19.786 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:23:19.786 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.488684+0000", 2024-09-13T11:23:19.786 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:19.787 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:19.787 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:19.787 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:23:19.787 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.488684+0000", 2024-09-13T11:23:19.787 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:19.787 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:19.787 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:19.787 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:23:19.787 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.488685+0000", 2024-09-13T11:23:19.787 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.4470000000000001e-06 2024-09-13T11:23:19.787 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:19.787 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:19.787 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:23:19.787 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.488690+0000", 2024-09-13T11:23:19.788 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.8520000000000003e-06 2024-09-13T11:23:19.788 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:19.788 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:19.788 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:23:19.788 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.529557+0000", 2024-09-13T11:23:19.788 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.040866524000000001 2024-09-13T11:23:19.788 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:19.788 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:19.788 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:23:19.788 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.529567+0000", 2024-09-13T11:23:19.788 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.011e-05 2024-09-13T11:23:19.788 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:19.789 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:19.789 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:23:19.789 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.529649+0000", 2024-09-13T11:23:19.789 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.2280000000000005e-05 2024-09-13T11:23:19.789 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:19.789 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:19.789 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:23:19.789 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.529658+0000", 2024-09-13T11:23:19.789 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.3549999999999994e-06 2024-09-13T11:23:19.789 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:19.789 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:19.790 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:23:19.790 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.529777+0000", 2024-09-13T11:23:19.790 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00011838 2024-09-13T11:23:19.790 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:19.790 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:23:19.790 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:19.790 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:19.790 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:19.790 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:867 3.a 3:5ca59689:::benchmark_data_smithi177_31116_object866:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-09-13T11:23:19.790 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:22:49.493530+0000", 2024-09-13T11:23:19.790 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 30.254011925, 2024-09-13T11:23:19.791 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.038165363000000001, 2024-09-13T11:23:19.791 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:23:19.791 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:23:19.791 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:19.791 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:23:19.791 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:23:19.792 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:19.792 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 867 2024-09-13T11:23:19.792 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:19.792 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:23:19.792 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:19.793 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:23:19.793 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.493530+0000", 2024-09-13T11:23:19.793 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:19.793 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:19.793 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:19.794 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:23:19.794 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.493530+0000", 2024-09-13T11:23:19.794 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:19.794 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:19.794 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:19.794 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:23:19.794 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.493532+0000", 2024-09-13T11:23:19.794 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.5459999999999998e-06 2024-09-13T11:23:19.794 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:19.795 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:19.795 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:23:19.795 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.493568+0000", 2024-09-13T11:23:19.795 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.5354000000000001e-05 2024-09-13T11:23:19.795 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:19.795 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:19.795 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:23:19.795 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.493595+0000", 2024-09-13T11:23:19.795 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.6823999999999999e-05 2024-09-13T11:23:19.795 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:19.795 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:19.795 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:23:19.796 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.493606+0000", 2024-09-13T11:23:19.796 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.1045e-05 2024-09-13T11:23:19.796 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:19.796 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:19.796 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:23:19.796 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.493676+0000", 2024-09-13T11:23:19.796 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.0007999999999998e-05 2024-09-13T11:23:19.796 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:19.796 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:19.796 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:23:19.796 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.493792+0000", 2024-09-13T11:23:19.796 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00011584800000000001 2024-09-13T11:23:19.797 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:19.797 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:19.797 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 7", 2024-09-13T11:23:19.797 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.493926+0000", 2024-09-13T11:23:19.797 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00013482199999999999 2024-09-13T11:23:19.797 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:19.797 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:19.797 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:23:19.797 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.494881+0000", 2024-09-13T11:23:19.797 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00095461899999999998 2024-09-13T11:23:19.797 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:19.797 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:19.798 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:19.798 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.531625+0000", 2024-09-13T11:23:19.798 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.036744106999999998 2024-09-13T11:23:19.798 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:19.798 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:19.798 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:23:19.798 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.531663+0000", 2024-09-13T11:23:19.798 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.7962e-05 2024-09-13T11:23:19.798 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:19.798 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:19.798 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:23:19.798 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.531695+0000", 2024-09-13T11:23:19.798 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.2227999999999997e-05 2024-09-13T11:23:19.799 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:19.799 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:23:19.799 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:19.799 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:19.799 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:19.799 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:1003 3.a 3:5aec7cc6:::benchmark_data_smithi177_31116_object1002:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-09-13T11:23:19.799 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:22:49.614497+0000", 2024-09-13T11:23:19.799 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 30.133044487999999, 2024-09-13T11:23:19.799 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.040510329999999997, 2024-09-13T11:23:19.799 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:23:19.799 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:23:19.799 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:19.799 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:23:19.800 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:23:19.800 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:19.800 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 1003 2024-09-13T11:23:19.800 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:19.800 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:23:19.800 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:19.800 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:23:19.800 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.614497+0000", 2024-09-13T11:23:19.800 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:19.800 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:19.800 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:19.800 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:23:19.801 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.614497+0000", 2024-09-13T11:23:19.801 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:19.801 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:19.801 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:19.801 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:23:19.801 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.614500+0000", 2024-09-13T11:23:19.801 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.3369999999999998e-06 2024-09-13T11:23:19.801 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:19.801 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:19.802 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:23:19.802 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.614531+0000", 2024-09-13T11:23:19.802 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.0828999999999999e-05 2024-09-13T11:23:19.802 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:19.802 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:19.802 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:23:19.803 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.614546+0000", 2024-09-13T11:23:19.803 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5945000000000001e-05 2024-09-13T11:23:19.803 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:19.803 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:19.803 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:23:19.803 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.614557+0000", 2024-09-13T11:23:19.803 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0317e-05 2024-09-13T11:23:19.803 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:19.803 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:19.804 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:23:19.804 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.614642+0000", 2024-09-13T11:23:19.804 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.4734999999999994e-05 2024-09-13T11:23:19.804 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:19.804 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:19.804 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:23:19.804 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.614794+0000", 2024-09-13T11:23:19.804 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000152027 2024-09-13T11:23:19.805 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:19.805 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:19.805 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 7", 2024-09-13T11:23:19.805 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.614980+0000", 2024-09-13T11:23:19.805 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00018625799999999999 2024-09-13T11:23:19.806 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:19.806 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:19.806 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:23:19.806 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.615821+0000", 2024-09-13T11:23:19.806 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00084122999999999999 2024-09-13T11:23:19.806 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:19.806 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:19.806 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:19.806 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.654761+0000", 2024-09-13T11:23:19.806 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.038939559999999998 2024-09-13T11:23:19.806 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:19.806 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:19.806 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:23:19.806 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.654897+0000", 2024-09-13T11:23:19.807 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00013600999999999999 2024-09-13T11:23:19.807 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:19.807 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:19.807 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:23:19.807 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.655008+0000", 2024-09-13T11:23:19.807 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000111082 2024-09-13T11:23:19.807 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:19.807 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:23:19.807 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:19.807 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:19.807 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:19.807 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop_reply(client.4665.0:1003 3.a e43/30 ondisk, result = 0)", 2024-09-13T11:23:19.807 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:22:49.616305+0000", 2024-09-13T11:23:19.807 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 30.131237047999999, 2024-09-13T11:23:19.808 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.038717949000000002, 2024-09-13T11:23:19.808 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:23:19.808 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:23:19.808 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:23:19.808 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:23:19.808 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:19.808 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:23:19.808 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.616305+0000", 2024-09-13T11:23:19.808 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:19.808 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:19.808 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:19.808 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:23:19.808 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.616305+0000", 2024-09-13T11:23:19.809 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:19.809 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:19.809 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:19.809 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:23:19.809 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.616306+0000", 2024-09-13T11:23:19.809 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5200000000000001e-06 2024-09-13T11:23:19.809 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:19.809 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:19.809 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:23:19.809 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.616311+0000", 2024-09-13T11:23:19.809 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.5759999999999999e-06 2024-09-13T11:23:19.809 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:19.809 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:19.810 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:23:19.810 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.654620+0000", 2024-09-13T11:23:19.810 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.038309187000000001 2024-09-13T11:23:19.810 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:19.810 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:19.810 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:23:19.810 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.654628+0000", 2024-09-13T11:23:19.810 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.1669999999999999e-06 2024-09-13T11:23:19.811 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:19.811 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:19.811 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:23:19.811 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.654723+0000", 2024-09-13T11:23:19.811 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.4877999999999995e-05 2024-09-13T11:23:19.811 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:19.811 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:19.811 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:23:19.811 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.654741+0000", 2024-09-13T11:23:19.811 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.8073000000000001e-05 2024-09-13T11:23:19.812 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:19.812 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:19.812 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:23:19.812 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.655023+0000", 2024-09-13T11:23:19.812 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00028154800000000001 2024-09-13T11:23:19.812 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:19.812 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:23:19.812 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:19.813 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:19.813 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:19.813 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:1122 3.d e43/30 3:b40a2a39:::benchmark_data_smithi177_31116_object1121:head v 43'66, mlcod=43'64)", 2024-09-13T11:23:19.813 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:22:49.712014+0000", 2024-09-13T11:23:19.813 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 30.035527413000001, 2024-09-13T11:23:19.813 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.040250046999999997, 2024-09-13T11:23:19.813 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:23:19.813 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:23:19.813 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:19.813 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:23:19.813 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:19.813 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:23:19.813 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.712014+0000", 2024-09-13T11:23:19.814 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:19.814 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:19.814 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:19.814 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:23:19.814 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.712014+0000", 2024-09-13T11:23:19.814 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:19.814 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:19.814 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:19.814 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:23:19.814 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.712023+0000", 2024-09-13T11:23:19.814 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.4160000000000004e-06 2024-09-13T11:23:19.814 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:19.814 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:19.814 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:23:19.815 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.712084+0000", 2024-09-13T11:23:19.815 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.0720999999999997e-05 2024-09-13T11:23:19.815 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:19.815 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:19.815 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:23:19.815 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.748087+0000", 2024-09-13T11:23:19.815 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.036003546999999997 2024-09-13T11:23:19.815 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:19.815 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:19.816 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:23:19.816 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.748126+0000", 2024-09-13T11:23:19.816 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.9208999999999997e-05 2024-09-13T11:23:19.816 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:19.816 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:19.816 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:23:19.816 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.748264+0000", 2024-09-13T11:23:19.816 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00013805600000000001 2024-09-13T11:23:19.817 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:19.817 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:19.817 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:23:19.817 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.748313+0000", 2024-09-13T11:23:19.817 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.8575999999999997e-05 2024-09-13T11:23:19.817 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:19.817 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:19.817 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:23:19.817 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.752180+0000", 2024-09-13T11:23:19.817 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.003867498 2024-09-13T11:23:19.817 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:19.818 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:19.818 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:23:19.818 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.752264+0000", 2024-09-13T11:23:19.818 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.4023999999999994e-05 2024-09-13T11:23:19.818 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:19.819 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:23:19.819 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:19.819 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:19.819 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:19.819 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:1275 3.a 3:5a15da6d:::benchmark_data_smithi177_31116_object1274:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-09-13T11:23:19.819 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:22:49.829287+0000", 2024-09-13T11:23:19.819 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 29.918254526999998, 2024-09-13T11:23:19.820 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.043769420000000003, 2024-09-13T11:23:19.820 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:23:19.820 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:23:19.820 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:19.820 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:23:19.820 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:23:19.820 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:19.820 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 1275 2024-09-13T11:23:19.820 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:19.820 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:23:19.820 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:19.821 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:23:19.821 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.829287+0000", 2024-09-13T11:23:19.821 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:19.821 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:19.821 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:19.821 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:23:19.821 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.829287+0000", 2024-09-13T11:23:19.821 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:19.821 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:19.821 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:19.821 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:23:19.821 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.829291+0000", 2024-09-13T11:23:19.821 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.4529999999999998e-06 2024-09-13T11:23:19.822 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:19.822 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:19.822 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:23:19.822 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.829362+0000", 2024-09-13T11:23:19.822 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.1064999999999998e-05 2024-09-13T11:23:19.822 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:19.822 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:19.822 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:23:19.823 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.829389+0000", 2024-09-13T11:23:19.823 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.6991999999999998e-05 2024-09-13T11:23:19.823 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:19.823 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:19.823 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:23:19.823 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.829408+0000", 2024-09-13T11:23:19.823 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.9477e-05 2024-09-13T11:23:19.823 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:19.823 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:19.823 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:23:19.823 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.829514+0000", 2024-09-13T11:23:19.824 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000105832 2024-09-13T11:23:19.824 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:19.824 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:19.824 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:23:19.824 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.829663+0000", 2024-09-13T11:23:19.824 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00014837300000000001 2024-09-13T11:23:19.824 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:19.824 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:19.824 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 7", 2024-09-13T11:23:19.824 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.829815+0000", 2024-09-13T11:23:19.824 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000152235 2024-09-13T11:23:19.824 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:19.824 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:19.824 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:23:19.825 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.832310+0000", 2024-09-13T11:23:19.825 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.002494844 2024-09-13T11:23:19.825 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:19.825 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:19.825 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:19.825 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.872852+0000", 2024-09-13T11:23:19.825 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.040542424000000001 2024-09-13T11:23:19.825 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:19.825 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:19.825 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:23:19.825 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.872981+0000", 2024-09-13T11:23:19.825 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00012902399999999999 2024-09-13T11:23:19.826 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:19.826 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:19.826 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:23:19.826 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.873057+0000", 2024-09-13T11:23:19.826 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.5700999999999999e-05 2024-09-13T11:23:19.826 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:19.826 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:23:19.826 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:19.826 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:19.826 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:19.826 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop_reply(client.4665.0:1275 3.a e43/30 ondisk, result = 0)", 2024-09-13T11:23:19.827 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:22:49.833244+0000", 2024-09-13T11:23:19.827 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 29.914297620999999, 2024-09-13T11:23:19.827 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.039823584000000002, 2024-09-13T11:23:19.827 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:23:19.827 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:23:19.827 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:23:19.827 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:23:19.827 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:19.827 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:23:19.827 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.833244+0000", 2024-09-13T11:23:19.827 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:19.827 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:19.828 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:19.828 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:23:19.828 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.833244+0000", 2024-09-13T11:23:19.828 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:19.828 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:19.828 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:19.828 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:23:19.828 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.833247+0000", 2024-09-13T11:23:19.828 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.802e-06 2024-09-13T11:23:19.828 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:19.828 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:19.828 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:23:19.829 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.833257+0000", 2024-09-13T11:23:19.829 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0068000000000001e-05 2024-09-13T11:23:19.829 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:19.829 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:19.829 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:23:19.829 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.872674+0000", 2024-09-13T11:23:19.829 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.039417059999999997 2024-09-13T11:23:19.829 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:19.829 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:19.829 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:23:19.829 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.872693+0000", 2024-09-13T11:23:19.829 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.889e-05 2024-09-13T11:23:19.830 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:19.830 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:19.830 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:23:19.830 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.872820+0000", 2024-09-13T11:23:19.830 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00012652099999999999 2024-09-13T11:23:19.830 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:19.830 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:19.830 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:23:19.830 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.872839+0000", 2024-09-13T11:23:19.830 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.9344999999999999e-05 2024-09-13T11:23:19.830 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:19.830 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:19.830 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:23:19.831 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.873068+0000", 2024-09-13T11:23:19.831 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000228898 2024-09-13T11:23:19.831 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:19.831 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:23:19.831 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:19.831 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:19.831 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:19.831 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:1282 3.d e43/30 3:b230ba6b:::benchmark_data_smithi177_31116_object1281:head v 43'78, mlcod=43'73)", 2024-09-13T11:23:19.831 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:22:49.834603+0000", 2024-09-13T11:23:19.831 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 29.912939037000001, 2024-09-13T11:23:19.831 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.043089937000000002, 2024-09-13T11:23:19.831 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:23:19.831 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:23:19.832 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:19.832 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:23:19.832 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:19.832 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:23:19.832 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.834603+0000", 2024-09-13T11:23:19.832 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:19.832 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:19.832 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:19.832 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:23:19.832 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.834603+0000", 2024-09-13T11:23:19.832 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:19.832 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:19.833 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:19.833 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:23:19.833 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.834606+0000", 2024-09-13T11:23:19.833 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.7599999999999998e-06 2024-09-13T11:23:19.833 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:19.833 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:19.833 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:23:19.833 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.834684+0000", 2024-09-13T11:23:19.833 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.8490000000000002e-05 2024-09-13T11:23:19.834 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:19.834 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:19.834 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:23:19.834 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.875920+0000", 2024-09-13T11:23:19.834 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.041235956999999997 2024-09-13T11:23:19.834 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:19.834 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:19.834 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:23:19.834 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.875931+0000", 2024-09-13T11:23:19.834 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0834e-05 2024-09-13T11:23:19.834 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:19.834 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:19.834 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:23:19.835 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.876037+0000", 2024-09-13T11:23:19.835 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000106582 2024-09-13T11:23:19.835 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:19.835 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:19.835 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:23:19.835 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.876077+0000", 2024-09-13T11:23:19.835 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.9993000000000001e-05 2024-09-13T11:23:19.835 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:19.835 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:19.835 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:23:19.835 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.877634+0000", 2024-09-13T11:23:19.835 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001556254 2024-09-13T11:23:19.835 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:19.835 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:19.836 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:23:19.836 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.877693+0000", 2024-09-13T11:23:19.836 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.9067000000000002e-05 2024-09-13T11:23:19.836 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:19.836 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:23:19.836 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:19.836 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:19.836 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:19.836 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:1365 3.d e43/30 3:bf637086:::benchmark_data_smithi177_31116_object1364:head v 43'84, mlcod=43'80)", 2024-09-13T11:23:19.836 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:22:49.890601+0000", 2024-09-13T11:23:19.836 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 29.856940938000001, 2024-09-13T11:23:19.836 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.040486463, 2024-09-13T11:23:19.836 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:23:19.837 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:23:19.837 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:19.837 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:23:19.837 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:19.837 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:23:19.837 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.890601+0000", 2024-09-13T11:23:19.837 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:19.837 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:19.837 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:19.837 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:23:19.837 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.890601+0000", 2024-09-13T11:23:19.837 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:19.837 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:19.838 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:19.838 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:23:19.838 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.890603+0000", 2024-09-13T11:23:19.838 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.2879999999999999e-06 2024-09-13T11:23:19.838 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:19.838 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:19.838 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:23:19.838 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.890638+0000", 2024-09-13T11:23:19.838 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.5216999999999998e-05 2024-09-13T11:23:19.838 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:19.838 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:19.838 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:23:19.838 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.925869+0000", 2024-09-13T11:23:19.839 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.035230298 2024-09-13T11:23:19.839 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:19.839 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:19.839 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:23:19.839 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.925883+0000", 2024-09-13T11:23:19.839 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3973e-05 2024-09-13T11:23:19.839 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:19.839 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:19.839 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:23:19.839 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.925987+0000", 2024-09-13T11:23:19.839 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00010407399999999999 2024-09-13T11:23:19.839 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:19.839 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:19.840 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:23:19.840 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.926024+0000", 2024-09-13T11:23:19.840 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.7379000000000002e-05 2024-09-13T11:23:19.840 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:19.840 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:19.840 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:23:19.840 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.931009+0000", 2024-09-13T11:23:19.840 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0049846719999999999 2024-09-13T11:23:19.840 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:19.840 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:19.840 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:23:19.840 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.931087+0000", 2024-09-13T11:23:19.840 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.8561999999999998e-05 2024-09-13T11:23:19.840 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:19.841 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:23:19.841 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:19.841 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:19.841 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:19.841 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:10761 3.d e48/30 3:b65e51ff:::benchmark_data_smithi177_31116_object10760:head v 48'678, mlcod=48'675)", 2024-09-13T11:23:19.841 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:22:54.904092+0000", 2024-09-13T11:23:19.841 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 24.843449371999998, 2024-09-13T11:23:19.841 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.043534784, 2024-09-13T11:23:19.841 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:23:19.841 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:23:19.841 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:19.841 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:23:19.841 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:19.842 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:23:19.842 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:54.904092+0000", 2024-09-13T11:23:19.842 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:19.842 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:19.842 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:19.842 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:23:19.842 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:54.904092+0000", 2024-09-13T11:23:19.842 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:19.842 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:19.842 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:19.842 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:23:19.842 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:54.904106+0000", 2024-09-13T11:23:19.842 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3196999999999999e-05 2024-09-13T11:23:19.842 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:19.843 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:19.843 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:23:19.843 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:54.904141+0000", 2024-09-13T11:23:19.843 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.5607e-05 2024-09-13T11:23:19.843 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:19.843 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:19.843 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:23:19.844 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:54.946464+0000", 2024-09-13T11:23:19.844 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.042322384999999997 2024-09-13T11:23:19.844 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:19.844 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:19.845 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:23:19.845 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:54.946473+0000", 2024-09-13T11:23:19.845 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.842e-06 2024-09-13T11:23:19.845 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:19.845 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:19.845 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:23:19.845 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:54.946542+0000", 2024-09-13T11:23:19.845 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.8382999999999999e-05 2024-09-13T11:23:19.845 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:19.846 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:19.846 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:23:19.846 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:54.946568+0000", 2024-09-13T11:23:19.846 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.6135999999999998e-05 2024-09-13T11:23:19.846 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:19.846 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:19.847 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:23:19.847 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:54.947588+0000", 2024-09-13T11:23:19.847 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0010200400000000001 2024-09-13T11:23:19.847 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:19.847 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:19.847 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:23:19.847 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:54.947627+0000", 2024-09-13T11:23:19.847 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.9193999999999998e-05 2024-09-13T11:23:19.847 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:19.848 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:23:19.848 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:19.848 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:19.848 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:19.848 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:16941 3.a 3:5049f2a8:::benchmark_data_smithi177_31116_object16940:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-13T11:23:19.849 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:22:59.119359+0000", 2024-09-13T11:23:19.849 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 20.628182867, 2024-09-13T11:23:19.849 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.045603890000000001, 2024-09-13T11:23:19.849 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:23:19.849 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:23:19.849 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:19.849 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:23:19.849 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:23:19.849 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:19.849 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 16941 2024-09-13T11:23:19.850 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:19.850 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:23:19.850 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:19.850 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:23:19.850 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:59.119359+0000", 2024-09-13T11:23:19.850 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:19.850 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:19.850 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:19.850 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:23:19.850 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:59.119359+0000", 2024-09-13T11:23:19.850 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:19.850 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:19.851 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:19.851 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:23:19.851 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:59.119362+0000", 2024-09-13T11:23:19.851 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.427e-06 2024-09-13T11:23:19.851 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:19.851 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:19.851 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:23:19.851 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:59.119399+0000", 2024-09-13T11:23:19.851 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.6315e-05 2024-09-13T11:23:19.851 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:19.851 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:19.851 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:23:19.851 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:59.119422+0000", 2024-09-13T11:23:19.852 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.2945000000000001e-05 2024-09-13T11:23:19.852 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:19.852 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:19.852 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:23:19.852 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:59.119432+0000", 2024-09-13T11:23:19.852 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0796e-05 2024-09-13T11:23:19.852 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:19.852 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:19.852 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:23:19.852 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:59.119530+0000", 2024-09-13T11:23:19.852 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.7323000000000003e-05 2024-09-13T11:23:19.852 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:19.852 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:19.853 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:23:19.853 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:59.119756+0000", 2024-09-13T11:23:19.853 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00022613700000000001 2024-09-13T11:23:19.856 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:19.856 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:19.856 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 7", 2024-09-13T11:23:19.856 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:59.120020+0000", 2024-09-13T11:23:19.856 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00026381100000000003 2024-09-13T11:23:19.857 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:19.857 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:19.857 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:23:19.857 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:59.121549+0000", 2024-09-13T11:23:19.857 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001529725 2024-09-13T11:23:19.857 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:19.857 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:19.857 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:19.857 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:59.164794+0000", 2024-09-13T11:23:19.857 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.043244432999999999 2024-09-13T11:23:19.857 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:19.857 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:19.857 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:23:19.857 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:59.164883+0000", 2024-09-13T11:23:19.858 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.9058e-05 2024-09-13T11:23:19.858 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:19.858 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:19.858 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:23:19.858 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:59.164963+0000", 2024-09-13T11:23:19.858 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.9919999999999994e-05 2024-09-13T11:23:19.858 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:19.858 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:23:19.858 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:19.859 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:19.859 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:19.859 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:17011 3.a 3:5e3c00d0:::benchmark_data_smithi177_31116_object17010:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-13T11:23:19.859 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:22:59.165305+0000", 2024-09-13T11:23:19.859 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 20.582237195000001, 2024-09-13T11:23:19.859 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.042388409000000002, 2024-09-13T11:23:19.859 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:23:19.859 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:23:19.859 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:19.859 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:23:19.859 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:23:19.859 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:19.859 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 17011 2024-09-13T11:23:19.860 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:19.860 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:23:19.860 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:19.860 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:23:19.860 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:59.165305+0000", 2024-09-13T11:23:19.860 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:19.860 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:19.860 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:19.860 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:23:19.860 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:59.165305+0000", 2024-09-13T11:23:19.860 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:19.860 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:19.860 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:19.860 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:23:19.861 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:59.165306+0000", 2024-09-13T11:23:19.861 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5999999999999999e-06 2024-09-13T11:23:19.861 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:19.861 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:19.861 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:23:19.861 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:59.165352+0000", 2024-09-13T11:23:19.861 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.6180000000000002e-05 2024-09-13T11:23:19.861 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:19.861 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:19.861 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:23:19.861 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:59.165368+0000", 2024-09-13T11:23:19.861 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5854000000000001e-05 2024-09-13T11:23:19.861 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:19.861 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:19.862 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:23:19.862 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:59.165378+0000", 2024-09-13T11:23:19.862 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0188e-05 2024-09-13T11:23:19.862 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:19.862 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:19.862 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:23:19.862 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:59.165458+0000", 2024-09-13T11:23:19.862 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.9163000000000001e-05 2024-09-13T11:23:19.862 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:19.862 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:19.862 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:23:19.862 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:59.165673+0000", 2024-09-13T11:23:19.862 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00021530200000000001 2024-09-13T11:23:19.862 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:19.863 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:19.863 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 7", 2024-09-13T11:23:19.863 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:59.165942+0000", 2024-09-13T11:23:19.863 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00026858200000000001 2024-09-13T11:23:19.863 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:19.863 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:19.863 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:23:19.863 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:59.167472+0000", 2024-09-13T11:23:19.863 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0015307540000000001 2024-09-13T11:23:19.863 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:19.863 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:19.863 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:19.863 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:59.207563+0000", 2024-09-13T11:23:19.863 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.040090354000000002 2024-09-13T11:23:19.864 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:19.864 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:19.864 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:23:19.864 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:59.207634+0000", 2024-09-13T11:23:19.864 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.1601999999999996e-05 2024-09-13T11:23:19.864 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:19.864 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:19.864 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:23:19.864 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:59.207693+0000", 2024-09-13T11:23:19.864 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.8829999999999997e-05 2024-09-13T11:23:19.864 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:19.864 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:23:19.864 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:19.864 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:19.864 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:19.864 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:30852 3.a 3:522fb0d3:::benchmark_data_smithi177_31116_object30851:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e59)", 2024-09-13T11:23:19.865 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:09.538712+0000", 2024-09-13T11:23:19.865 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 10.208829789999999, 2024-09-13T11:23:19.865 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.19661263300000001, 2024-09-13T11:23:19.865 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:23:19.865 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:23:19.865 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:19.865 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:23:19.865 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:23:19.865 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:19.865 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 30852 2024-09-13T11:23:19.865 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:19.865 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:23:19.865 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:19.865 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:23:19.865 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.538712+0000", 2024-09-13T11:23:19.866 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:19.866 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:19.866 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:19.866 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:23:19.866 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.538712+0000", 2024-09-13T11:23:19.866 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:19.866 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:19.866 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:19.866 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:23:19.866 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.538714+0000", 2024-09-13T11:23:19.866 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.658e-06 2024-09-13T11:23:19.866 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:19.866 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:19.866 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:23:19.867 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.538735+0000", 2024-09-13T11:23:19.867 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.141e-05 2024-09-13T11:23:19.867 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:19.867 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:19.867 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:23:19.867 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.538747+0000", 2024-09-13T11:23:19.867 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.2258e-05 2024-09-13T11:23:19.867 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:19.867 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:19.867 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:23:19.867 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.538754+0000", 2024-09-13T11:23:19.867 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.2859999999999996e-06 2024-09-13T11:23:19.867 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:19.867 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:19.868 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:23:19.868 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.538798+0000", 2024-09-13T11:23:19.868 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.4258000000000002e-05 2024-09-13T11:23:19.868 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:19.868 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:19.868 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:23:19.868 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.538890+0000", 2024-09-13T11:23:19.868 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.1676999999999997e-05 2024-09-13T11:23:19.868 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:19.868 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:19.869 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 7", 2024-09-13T11:23:19.869 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.538991+0000", 2024-09-13T11:23:19.869 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00010173700000000001 2024-09-13T11:23:19.869 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:19.869 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:19.869 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:23:19.869 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.540171+0000", 2024-09-13T11:23:19.869 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0011800479999999999 2024-09-13T11:23:19.869 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:19.869 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:19.869 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:19.869 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.735031+0000", 2024-09-13T11:23:19.869 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.194859798 2024-09-13T11:23:19.869 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:19.870 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:19.870 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:23:19.870 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.735197+0000", 2024-09-13T11:23:19.870 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00016569499999999999 2024-09-13T11:23:19.870 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:19.870 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:19.870 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:23:19.870 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.735325+0000", 2024-09-13T11:23:19.870 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00012780800000000001 2024-09-13T11:23:19.870 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:19.870 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:23:19.870 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:19.870 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:19.871 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:19.871 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop_reply(client.4665.0:30852 3.a e59/30 ondisk, result = 0)", 2024-09-13T11:23:19.871 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:09.540458+0000", 2024-09-13T11:23:19.871 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 10.207084302, 2024-09-13T11:23:19.871 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.194887, 2024-09-13T11:23:19.871 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:23:19.871 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:23:19.871 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:23:19.871 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:23:19.871 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:19.871 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:23:19.872 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.540458+0000", 2024-09-13T11:23:19.872 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:19.872 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:19.872 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:19.872 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:23:19.872 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.540458+0000", 2024-09-13T11:23:19.872 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:19.872 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:19.872 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:19.872 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:23:19.872 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.540459+0000", 2024-09-13T11:23:19.872 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.2699999999999999e-06 2024-09-13T11:23:19.872 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:19.873 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:19.873 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:23:19.873 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.540463+0000", 2024-09-13T11:23:19.873 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.8500000000000004e-06 2024-09-13T11:23:19.873 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:19.873 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:19.873 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:23:19.873 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.734813+0000", 2024-09-13T11:23:19.873 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.194350047 2024-09-13T11:23:19.873 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:19.873 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:19.874 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:23:19.874 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.734832+0000", 2024-09-13T11:23:19.874 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.9610999999999999e-05 2024-09-13T11:23:19.874 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:19.874 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:19.874 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:23:19.874 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.734989+0000", 2024-09-13T11:23:19.874 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00015694799999999999 2024-09-13T11:23:19.874 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:19.874 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:19.874 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:23:19.874 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.735015+0000", 2024-09-13T11:23:19.874 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.6165000000000001e-05 2024-09-13T11:23:19.874 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:19.874 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:19.875 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:23:19.875 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.735345+0000", 2024-09-13T11:23:19.875 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00032910900000000001 2024-09-13T11:23:19.875 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:19.875 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:23:19.875 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:19.875 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:19.875 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:19.875 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:31097 3.d e59/30 3:bc3dc2d6:::benchmark_data_smithi177_31116_object31096:head v 59'1908, mlcod=59'1904)", 2024-09-13T11:23:19.875 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:09.957883+0000", 2024-09-13T11:23:19.875 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 9.7896588280000003, 2024-09-13T11:23:19.875 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.042855150000000002, 2024-09-13T11:23:19.875 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:23:19.875 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:23:19.876 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:19.876 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:23:19.876 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:19.876 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:23:19.876 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.957883+0000", 2024-09-13T11:23:19.876 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:19.876 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:19.876 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:19.876 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:23:19.876 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.957883+0000", 2024-09-13T11:23:19.876 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:19.876 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:19.876 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:19.877 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:23:19.877 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.957886+0000", 2024-09-13T11:23:19.877 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.0910000000000001e-06 2024-09-13T11:23:19.877 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:19.877 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:19.877 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:23:19.877 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.957937+0000", 2024-09-13T11:23:19.877 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.1233000000000002e-05 2024-09-13T11:23:19.878 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:19.878 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:19.878 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:23:19.878 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.997999+0000", 2024-09-13T11:23:19.878 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.040061306999999997 2024-09-13T11:23:19.878 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:19.878 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:19.878 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:23:19.878 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.998017+0000", 2024-09-13T11:23:19.878 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.8587999999999999e-05 2024-09-13T11:23:19.879 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:19.879 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:19.879 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:23:19.879 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.998151+0000", 2024-09-13T11:23:19.879 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000133781 2024-09-13T11:23:19.879 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:19.879 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:19.879 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:23:19.879 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.998188+0000", 2024-09-13T11:23:19.879 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.6915999999999997e-05 2024-09-13T11:23:19.879 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:19.879 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:19.879 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:23:19.879 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:10.000640+0000", 2024-09-13T11:23:19.879 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0024520459999999998 2024-09-13T11:23:19.880 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:19.880 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:19.880 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:23:19.880 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:10.000738+0000", 2024-09-13T11:23:19.880 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.8188000000000002e-05 2024-09-13T11:23:19.880 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:19.880 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:23:19.880 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:19.880 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:19.880 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:23:19.880 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-09-13T11:23:19.881 INFO:teuthology.orchestra.run.smithi001.stdout:osd.6: {} 2024-09-13T11:23:19.882 INFO:teuthology.orchestra.run.smithi001.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T11:23:19.882 INFO:teuthology.orchestra.run.smithi001.stdout:osd.7: {} 2024-09-13T11:23:19.883 INFO:teuthology.orchestra.run.smithi001.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T11:23:19.883 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-13T11:23:19.884 INFO:teuthology.orchestra.run.smithi001.stdout:osd.8: {} 2024-09-13T11:23:19.884 INFO:teuthology.orchestra.run.smithi001.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-13T11:23:19.884 INFO:teuthology.orchestra.run.smithi001.stdout:osd.9: {} 2024-09-13T11:23:19.884 INFO:teuthology.orchestra.run.smithi001.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-13T11:23:19.885 INFO:teuthology.orchestra.run.smithi001.stdout:osd.10: {} 2024-09-13T11:23:19.885 INFO:teuthology.orchestra.run.smithi001.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-13T11:23:19.885 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-13T11:23:19.876+0000 7f6a68775640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26307) UID: 0 2024-09-13T11:23:19.905 INFO:teuthology.orchestra.run.smithi001.stdout:osd.11: {} 2024-09-13T11:23:19.905 INFO:teuthology.orchestra.run.smithi001.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-13T11:23:19.916 INFO:teuthology.orchestra.run.smithi001.stderr:noscrub is unset 2024-09-13T11:23:19.925 INFO:tasks.ceph.osd.10.smithi162.stderr:2024-09-13T11:23:19.918+0000 7fb593421640 -1 osd.10 67 handle_osd_map injecting map crc failure 2024-09-13T11:23:19.946 INFO:teuthology.orchestra.run.smithi001.stdout:osd.12: {} 2024-09-13T11:23:19.946 INFO:teuthology.orchestra.run.smithi001.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-13T11:23:19.975 INFO:teuthology.orchestra.run.smithi001.stdout:osd.13: {} 2024-09-13T11:23:19.975 INFO:teuthology.orchestra.run.smithi001.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-13T11:23:19.983 INFO:tasks.ceph.osd.15.smithi177.stderr:2024-09-13T11:23:19.977+0000 7f9d6ce77640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26578) UID: 0 2024-09-13T11:23:20.003 INFO:tasks.radosbench.radosbench.0.smithi177.stdout: 53 15 44213 44198 52.1144 7.5625 0.00408401 0.0191831 2024-09-13T11:23:20.009 INFO:teuthology.orchestra.run.smithi001.stdout:osd.14: {} 2024-09-13T11:23:20.009 INFO:teuthology.orchestra.run.smithi001.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-13T11:23:20.040 INFO:teuthology.orchestra.run.smithi001.stdout:osd.15: {} 2024-09-13T11:23:20.040 INFO:teuthology.orchestra.run.smithi001.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-13T11:23:20.048 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-09-13T11:23:20.049 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [ 2024-09-13T11:23:20.049 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:20.049 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:44316 3.d 3:b25d0e74:::benchmark_data_smithi177_31116_object44315:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e68)", 2024-09-13T11:23:20.049 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:20.047298+0000", 2024-09-13T11:23:20.049 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 0.0013110890000000001, 2024-09-13T11:23:20.049 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0013398469999999999, 2024-09-13T11:23:20.050 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:23:20.050 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:23:20.050 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "waiting for sub ops", 2024-09-13T11:23:20.050 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:23:20.050 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:23:20.050 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:20.050 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 44316 2024-09-13T11:23:20.050 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:20.050 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:23:20.051 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:20.051 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:23:20.051 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:20.047298+0000", 2024-09-13T11:23:20.051 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:20.051 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:20.051 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:20.051 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:23:20.051 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:20.047298+0000", 2024-09-13T11:23:20.051 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:20.051 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:20.051 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:20.051 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:23:20.052 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:20.047300+0000", 2024-09-13T11:23:20.052 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.7760000000000002e-06 2024-09-13T11:23:20.052 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:20.052 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:20.052 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:23:20.052 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:20.047335+0000", 2024-09-13T11:23:20.052 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.4337e-05 2024-09-13T11:23:20.052 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:20.052 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:20.052 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:23:20.052 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:20.047353+0000", 2024-09-13T11:23:20.052 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.7816e-05 2024-09-13T11:23:20.053 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:20.053 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:20.053 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:23:20.053 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:20.047362+0000", 2024-09-13T11:23:20.053 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.3880000000000006e-06 2024-09-13T11:23:20.053 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:20.053 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:20.053 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:23:20.053 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:20.047430+0000", 2024-09-13T11:23:20.053 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.7632000000000006e-05 2024-09-13T11:23:20.053 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:20.054 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:20.054 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:23:20.054 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:20.047602+0000", 2024-09-13T11:23:20.054 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00017249 2024-09-13T11:23:20.054 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:20.054 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:20.054 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 3", 2024-09-13T11:23:20.054 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:20.047762+0000", 2024-09-13T11:23:20.054 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00016026300000000001 2024-09-13T11:23:20.054 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:20.054 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:23:20.054 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:20.055 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:20.055 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:20.055 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:44317 3.a e68/30 3:5cfe0f71:::benchmark_data_smithi177_31116_object44316:head v 68'2747, mlcod=68'2745)", 2024-09-13T11:23:20.055 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:20.047910+0000", 2024-09-13T11:23:20.055 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 0.00069864700000000005, 2024-09-13T11:23:20.055 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00079528800000000001, 2024-09-13T11:23:20.055 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:23:20.055 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:23:20.055 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:23:20.055 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:23:20.055 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:20.055 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:23:20.055 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:20.047910+0000", 2024-09-13T11:23:20.056 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:20.056 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:20.056 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:20.056 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:23:20.056 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:20.047910+0000", 2024-09-13T11:23:20.056 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:20.056 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:20.056 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:20.056 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:23:20.056 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:20.047912+0000", 2024-09-13T11:23:20.056 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.466e-06 2024-09-13T11:23:20.056 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:20.056 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:20.057 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:23:20.057 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:20.047943+0000", 2024-09-13T11:23:20.057 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.1848999999999999e-05 2024-09-13T11:23:20.057 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:20.057 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:20.057 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:23:20.057 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:20.047972+0000", 2024-09-13T11:23:20.057 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.8721000000000002e-05 2024-09-13T11:23:20.057 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:20.057 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:20.057 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:23:20.057 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:20.047979+0000", 2024-09-13T11:23:20.057 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.6270000000000002e-06 2024-09-13T11:23:20.057 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:20.058 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:20.058 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:23:20.058 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:20.048042+0000", 2024-09-13T11:23:20.058 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.3723e-05 2024-09-13T11:23:20.058 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:20.058 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:20.058 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:23:20.058 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:20.048061+0000", 2024-09-13T11:23:20.058 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.8243000000000001e-05 2024-09-13T11:23:20.058 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:20.058 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:23:20.059 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:20.059 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:20.059 INFO:teuthology.orchestra.run.smithi177.stdout: ], 2024-09-13T11:23:20.059 INFO:teuthology.orchestra.run.smithi177.stdout: "num_ops": 2 2024-09-13T11:23:20.059 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-09-13T11:23:20.074 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-13T11:23:20.108 INFO:tasks.ceph.osd.8.smithi001.stderr:2024-09-13T11:23:20.104+0000 7fb9b48ae640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26312) UID: 0 2024-09-13T11:23:20.184 INFO:tasks.ceph.osd.7.smithi177.stderr:2024-09-13T11:23:20.181+0000 7f897b51d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26575) UID: 0 2024-09-13T11:23:20.278 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-09-13T11:23:20.278 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [], 2024-09-13T11:23:20.278 INFO:teuthology.orchestra.run.smithi177.stdout: "complaint_time": 30, 2024-09-13T11:23:20.278 INFO:teuthology.orchestra.run.smithi177.stdout: "num_blocked_ops": 0 2024-09-13T11:23:20.278 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-09-13T11:23:20.285 INFO:tasks.ceph.osd.11.smithi177.stderr:2024-09-13T11:23:20.281+0000 7f24792c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26580) UID: 0 2024-09-13T11:23:20.298 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-13T11:23:20.385 INFO:tasks.ceph.osd.1.smithi092.stderr:2024-09-13T11:23:20.385+0000 7fe7789c5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33718) UID: 0 2024-09-13T11:23:20.486 INFO:tasks.ceph.osd.14.smithi162.stderr:2024-09-13T11:23:20.482+0000 7f235c0db640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26319) UID: 0 2024-09-13T11:23:20.496 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-09-13T11:23:20.496 INFO:teuthology.orchestra.run.smithi177.stdout: "size": 20, 2024-09-13T11:23:20.496 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 600, 2024-09-13T11:23:20.496 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [ 2024-09-13T11:23:20.496 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:20.496 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:1122 3.d 3:b40a2a39:::benchmark_data_smithi177_31116_object1121:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-09-13T11:23:20.496 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:22:49.711239+0000", 2024-09-13T11:23:20.496 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 30.783374665, 2024-09-13T11:23:20.496 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.041526306999999998, 2024-09-13T11:23:20.496 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:23:20.496 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:23:20.497 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:20.497 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:23:20.497 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:23:20.497 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:20.497 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 1122 2024-09-13T11:23:20.497 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:20.497 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:23:20.497 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:20.497 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:23:20.497 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.711239+0000", 2024-09-13T11:23:20.497 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:20.497 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:20.498 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:20.498 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:23:20.498 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.711239+0000", 2024-09-13T11:23:20.498 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:20.498 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:20.498 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:20.498 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:23:20.498 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.711244+0000", 2024-09-13T11:23:20.498 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.065e-06 2024-09-13T11:23:20.498 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:20.498 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:20.499 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:23:20.499 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.711313+0000", 2024-09-13T11:23:20.499 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.9405000000000004e-05 2024-09-13T11:23:20.499 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:20.499 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:20.499 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:23:20.499 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.711334+0000", 2024-09-13T11:23:20.499 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.1186999999999999e-05 2024-09-13T11:23:20.499 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:20.499 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:20.499 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:23:20.499 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.711345+0000", 2024-09-13T11:23:20.499 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.1352000000000001e-05 2024-09-13T11:23:20.499 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:20.500 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:20.500 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:23:20.500 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.711443+0000", 2024-09-13T11:23:20.500 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.7101000000000004e-05 2024-09-13T11:23:20.500 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:20.500 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:20.500 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:23:20.500 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.711626+0000", 2024-09-13T11:23:20.500 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00018319699999999999 2024-09-13T11:23:20.500 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:20.500 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:20.500 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 3", 2024-09-13T11:23:20.500 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.711848+0000", 2024-09-13T11:23:20.501 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000221907 2024-09-13T11:23:20.501 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:20.501 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:20.501 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:23:20.501 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.713111+0000", 2024-09-13T11:23:20.501 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0012635999999999999 2024-09-13T11:23:20.501 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:20.501 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:20.501 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:20.501 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.752614+0000", 2024-09-13T11:23:20.501 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.039503038999999997 2024-09-13T11:23:20.501 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:20.501 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:20.501 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:23:20.502 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.752701+0000", 2024-09-13T11:23:20.502 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.6206999999999996e-05 2024-09-13T11:23:20.502 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:20.502 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:20.502 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:23:20.502 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.752766+0000", 2024-09-13T11:23:20.502 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.5246999999999994e-05 2024-09-13T11:23:20.502 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:20.502 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:23:20.502 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:20.502 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:20.502 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:20.502 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:1282 3.d 3:b230ba6b:::benchmark_data_smithi177_31116_object1281:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-09-13T11:23:20.502 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:22:49.833858+0000", 2024-09-13T11:23:20.503 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 30.660755770000002, 2024-09-13T11:23:20.503 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.045058543, 2024-09-13T11:23:20.503 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:23:20.503 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:23:20.503 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:20.503 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:23:20.503 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:23:20.503 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:20.503 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 1282 2024-09-13T11:23:20.503 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:20.503 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:23:20.503 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:20.503 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:23:20.503 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.833858+0000", 2024-09-13T11:23:20.504 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:20.504 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:20.504 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:20.504 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:23:20.504 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.833858+0000", 2024-09-13T11:23:20.504 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:20.504 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:20.504 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:20.504 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:23:20.504 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.833862+0000", 2024-09-13T11:23:20.504 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.6629999999999999e-06 2024-09-13T11:23:20.504 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:20.504 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:20.505 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:23:20.505 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.833909+0000", 2024-09-13T11:23:20.505 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.6826999999999997e-05 2024-09-13T11:23:20.505 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:20.505 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:20.505 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:23:20.505 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.833933+0000", 2024-09-13T11:23:20.505 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.3952e-05 2024-09-13T11:23:20.505 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:20.505 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:20.505 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:23:20.505 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.833949+0000", 2024-09-13T11:23:20.505 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5770999999999999e-05 2024-09-13T11:23:20.505 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:20.506 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:20.506 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:23:20.506 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.834033+0000", 2024-09-13T11:23:20.506 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.4028000000000003e-05 2024-09-13T11:23:20.506 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:20.506 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:20.506 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:23:20.506 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.834205+0000", 2024-09-13T11:23:20.506 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00017227200000000001 2024-09-13T11:23:20.506 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:20.506 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:20.506 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 3", 2024-09-13T11:23:20.506 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.834426+0000", 2024-09-13T11:23:20.506 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00022096 2024-09-13T11:23:20.507 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:20.507 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:20.507 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:23:20.507 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.835975+0000", 2024-09-13T11:23:20.507 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001549096 2024-09-13T11:23:20.507 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:20.507 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:20.507 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:20.507 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.878751+0000", 2024-09-13T11:23:20.508 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.042776531999999999 2024-09-13T11:23:20.508 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:20.508 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:20.508 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:23:20.508 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.878853+0000", 2024-09-13T11:23:20.508 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000101796 2024-09-13T11:23:20.508 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:20.508 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:20.508 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:23:20.508 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.878917+0000", 2024-09-13T11:23:20.508 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.3645999999999993e-05 2024-09-13T11:23:20.508 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:20.508 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:23:20.508 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:20.509 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:20.509 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:20.509 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:1365 3.d 3:bf637086:::benchmark_data_smithi177_31116_object1364:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-09-13T11:23:20.509 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:22:49.889970+0000", 2024-09-13T11:23:20.509 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 30.604644243999999, 2024-09-13T11:23:20.509 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.041599599000000001, 2024-09-13T11:23:20.509 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:23:20.510 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:23:20.510 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:20.510 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:23:20.510 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:23:20.510 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:20.510 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 1365 2024-09-13T11:23:20.510 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:20.510 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:23:20.510 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:20.510 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:23:20.510 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.889970+0000", 2024-09-13T11:23:20.510 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:20.510 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:20.510 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:20.511 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:23:20.511 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.889970+0000", 2024-09-13T11:23:20.511 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:20.511 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:20.511 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:20.511 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:23:20.511 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.889973+0000", 2024-09-13T11:23:20.511 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.6840000000000001e-06 2024-09-13T11:23:20.511 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:20.511 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:20.511 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:23:20.511 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.890010+0000", 2024-09-13T11:23:20.511 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.7647000000000001e-05 2024-09-13T11:23:20.511 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:20.512 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:20.512 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:23:20.512 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.890029+0000", 2024-09-13T11:23:20.512 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.9137e-05 2024-09-13T11:23:20.512 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:20.512 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:20.512 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:23:20.512 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.890039+0000", 2024-09-13T11:23:20.512 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0076e-05 2024-09-13T11:23:20.512 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:20.512 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:20.512 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:23:20.512 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.890113+0000", 2024-09-13T11:23:20.513 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.3196999999999997e-05 2024-09-13T11:23:20.513 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:20.513 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:20.513 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:23:20.513 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.890287+0000", 2024-09-13T11:23:20.513 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000174141 2024-09-13T11:23:20.513 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:20.513 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:20.513 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 3", 2024-09-13T11:23:20.513 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.890458+0000", 2024-09-13T11:23:20.513 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000171519 2024-09-13T11:23:20.513 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:20.513 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:20.513 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:23:20.514 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.891513+0000", 2024-09-13T11:23:20.514 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0010551200000000001 2024-09-13T11:23:20.514 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:20.514 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:20.514 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:20.514 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.931392+0000", 2024-09-13T11:23:20.514 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.039878690000000001 2024-09-13T11:23:20.514 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:20.515 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:20.515 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:23:20.515 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.931494+0000", 2024-09-13T11:23:20.515 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000101625 2024-09-13T11:23:20.515 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:20.515 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:20.515 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:23:20.515 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.931569+0000", 2024-09-13T11:23:20.515 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.5763e-05 2024-09-13T11:23:20.515 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:20.515 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:23:20.515 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:20.515 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:20.515 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:20.516 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:10761 3.d 3:b65e51ff:::benchmark_data_smithi177_31116_object10760:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e48)", 2024-09-13T11:23:20.516 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:22:54.903597+0000", 2024-09-13T11:23:20.516 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 25.591017597, 2024-09-13T11:23:20.516 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.044312925000000003, 2024-09-13T11:23:20.516 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:23:20.516 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:23:20.516 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:20.516 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:23:20.516 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:23:20.516 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:20.516 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 10761 2024-09-13T11:23:20.516 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:20.516 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:23:20.517 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:20.517 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:23:20.517 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:54.903597+0000", 2024-09-13T11:23:20.517 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:20.517 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:20.517 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:20.517 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:23:20.517 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:54.903597+0000", 2024-09-13T11:23:20.517 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:20.517 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:20.517 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:20.517 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:23:20.518 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:54.903598+0000", 2024-09-13T11:23:20.518 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.7680000000000001e-06 2024-09-13T11:23:20.518 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:20.518 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:20.518 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:23:20.518 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:54.903621+0000", 2024-09-13T11:23:20.518 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.2707000000000001e-05 2024-09-13T11:23:20.518 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:20.518 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:20.518 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:23:20.518 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:54.903632+0000", 2024-09-13T11:23:20.518 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0641e-05 2024-09-13T11:23:20.518 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:20.518 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:20.519 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:23:20.519 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:54.903637+0000", 2024-09-13T11:23:20.519 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.733e-06 2024-09-13T11:23:20.519 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:20.519 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:20.519 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:23:20.519 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:54.903699+0000", 2024-09-13T11:23:20.519 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.1723000000000005e-05 2024-09-13T11:23:20.519 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:20.519 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:20.519 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:23:20.519 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:54.903856+0000", 2024-09-13T11:23:20.519 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00015686199999999999 2024-09-13T11:23:20.520 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:20.520 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:20.520 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 3", 2024-09-13T11:23:20.520 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:54.904016+0000", 2024-09-13T11:23:20.520 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000159777 2024-09-13T11:23:20.520 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:20.520 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:20.520 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:23:20.520 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:54.905147+0000", 2024-09-13T11:23:20.520 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001131176 2024-09-13T11:23:20.520 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:20.520 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:20.520 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:20.521 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:54.947809+0000", 2024-09-13T11:23:20.521 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.042661835000000002 2024-09-13T11:23:20.521 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:20.521 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:20.521 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:23:20.521 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:54.947866+0000", 2024-09-13T11:23:20.521 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.7077999999999998e-05 2024-09-13T11:23:20.521 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:20.521 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:20.521 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:23:20.521 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:54.947909+0000", 2024-09-13T11:23:20.521 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.3625000000000003e-05 2024-09-13T11:23:20.521 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:20.522 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:23:20.522 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:20.522 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:20.522 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:20.522 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:15875 3.d 3:b68b87bc:::benchmark_data_smithi177_31116_object15874:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-13T11:23:20.522 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:22:58.390100+0000", 2024-09-13T11:23:20.522 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 22.104514353999999, 2024-09-13T11:23:20.522 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.042592756000000002, 2024-09-13T11:23:20.522 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:23:20.522 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:23:20.522 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:20.522 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:23:20.522 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:23:20.523 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:20.523 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 15875 2024-09-13T11:23:20.523 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:20.523 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:23:20.523 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:20.523 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:23:20.523 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:58.390100+0000", 2024-09-13T11:23:20.523 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:20.523 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:20.523 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:20.523 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:23:20.523 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:58.390100+0000", 2024-09-13T11:23:20.523 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:20.524 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:20.524 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:20.524 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:23:20.524 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:58.390102+0000", 2024-09-13T11:23:20.524 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.492e-06 2024-09-13T11:23:20.524 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:20.524 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:20.524 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:23:20.524 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:58.390175+0000", 2024-09-13T11:23:20.524 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.3127999999999995e-05 2024-09-13T11:23:20.524 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:20.524 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:20.525 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:23:20.525 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:58.390193+0000", 2024-09-13T11:23:20.525 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.7740999999999998e-05 2024-09-13T11:23:20.525 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:20.525 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:20.525 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:23:20.525 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:58.390207+0000", 2024-09-13T11:23:20.525 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3594000000000001e-05 2024-09-13T11:23:20.525 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:20.525 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:20.525 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:23:20.525 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:58.390284+0000", 2024-09-13T11:23:20.525 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.7547999999999997e-05 2024-09-13T11:23:20.526 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:20.526 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:20.526 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:23:20.526 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:58.390445+0000", 2024-09-13T11:23:20.526 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000160457 2024-09-13T11:23:20.526 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:20.526 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:20.526 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 3", 2024-09-13T11:23:20.526 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:58.390616+0000", 2024-09-13T11:23:20.526 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000170932 2024-09-13T11:23:20.526 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:20.526 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:20.527 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:23:20.527 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:58.393640+0000", 2024-09-13T11:23:20.527 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0030246880000000002 2024-09-13T11:23:20.527 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:20.527 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:20.527 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:20.527 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:58.432548+0000", 2024-09-13T11:23:20.527 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.038907616999999999 2024-09-13T11:23:20.527 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:20.527 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:20.527 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:23:20.527 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:58.432632+0000", 2024-09-13T11:23:20.527 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.3974999999999995e-05 2024-09-13T11:23:20.528 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:20.528 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:20.528 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:23:20.528 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:58.432693+0000", 2024-09-13T11:23:20.528 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.0584000000000001e-05 2024-09-13T11:23:20.528 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:20.528 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:23:20.528 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:20.528 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:20.528 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:20.528 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:16619 3.d 3:b2c0dd0e:::benchmark_data_smithi177_31116_object16618:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-13T11:23:20.528 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:22:58.844888+0000", 2024-09-13T11:23:20.528 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 21.649725676999999, 2024-09-13T11:23:20.529 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.055397954999999999, 2024-09-13T11:23:20.529 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:23:20.529 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:23:20.529 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:20.529 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:23:20.529 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:23:20.529 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:20.529 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 16619 2024-09-13T11:23:20.529 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:20.529 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:23:20.529 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:20.529 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:23:20.530 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:58.844888+0000", 2024-09-13T11:23:20.530 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:20.530 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:20.530 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:20.530 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:23:20.530 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:58.844888+0000", 2024-09-13T11:23:20.530 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:20.530 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:20.530 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:20.530 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:23:20.530 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:58.844891+0000", 2024-09-13T11:23:20.530 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.1840000000000002e-06 2024-09-13T11:23:20.530 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:20.531 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:20.531 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:23:20.531 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:58.844918+0000", 2024-09-13T11:23:20.531 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.7596000000000001e-05 2024-09-13T11:23:20.531 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:20.531 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:20.531 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:23:20.531 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:58.844933+0000", 2024-09-13T11:23:20.531 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.4914e-05 2024-09-13T11:23:20.531 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:20.531 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:20.531 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:23:20.531 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:58.844940+0000", 2024-09-13T11:23:20.532 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.8159999999999996e-06 2024-09-13T11:23:20.532 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:20.532 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:20.532 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:23:20.532 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:58.845004+0000", 2024-09-13T11:23:20.532 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.4243000000000001e-05 2024-09-13T11:23:20.532 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:20.532 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:20.532 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:23:20.532 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:58.845111+0000", 2024-09-13T11:23:20.532 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00010719 2024-09-13T11:23:20.532 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:20.532 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:20.533 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 3", 2024-09-13T11:23:20.533 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:58.845212+0000", 2024-09-13T11:23:20.533 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00010024 2024-09-13T11:23:20.533 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:20.533 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:20.533 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:23:20.533 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:58.846588+0000", 2024-09-13T11:23:20.533 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0013761439999999999 2024-09-13T11:23:20.533 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:20.533 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:20.533 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:20.533 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:58.900164+0000", 2024-09-13T11:23:20.533 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.053576231000000002 2024-09-13T11:23:20.534 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:20.534 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:20.534 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:23:20.534 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:58.900230+0000", 2024-09-13T11:23:20.534 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.6500000000000004e-05 2024-09-13T11:23:20.534 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:20.534 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:20.534 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:23:20.534 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:58.900286+0000", 2024-09-13T11:23:20.534 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.5896999999999997e-05 2024-09-13T11:23:20.534 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:20.534 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:23:20.534 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:20.535 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:20.535 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:20.535 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop_reply(client.4665.0:16619 3.d e52/30 ondisk, result = 0)", 2024-09-13T11:23:20.535 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:22:58.846542+0000", 2024-09-13T11:23:20.535 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 21.648071907999999, 2024-09-13T11:23:20.535 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.053753741000000001, 2024-09-13T11:23:20.535 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:23:20.535 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:23:20.535 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:23:20.535 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:23:20.535 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:20.535 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:23:20.535 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:58.846542+0000", 2024-09-13T11:23:20.535 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:20.536 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:20.536 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:20.536 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:23:20.536 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:58.846542+0000", 2024-09-13T11:23:20.536 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:20.536 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:20.536 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:20.536 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:23:20.536 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:58.846544+0000", 2024-09-13T11:23:20.536 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.48e-06 2024-09-13T11:23:20.536 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:20.536 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:20.536 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:23:20.537 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:58.846548+0000", 2024-09-13T11:23:20.537 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.5909999999999998e-06 2024-09-13T11:23:20.537 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:20.537 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:20.537 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:23:20.537 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:58.899994+0000", 2024-09-13T11:23:20.537 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.053445626000000003 2024-09-13T11:23:20.537 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:20.537 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:20.537 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:23:20.537 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:58.900013+0000", 2024-09-13T11:23:20.537 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.9409999999999999e-05 2024-09-13T11:23:20.537 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:20.537 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:20.538 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:23:20.538 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:58.900145+0000", 2024-09-13T11:23:20.538 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000131278 2024-09-13T11:23:20.538 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:20.538 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:20.538 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:23:20.538 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:58.900157+0000", 2024-09-13T11:23:20.538 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.1993e-05 2024-09-13T11:23:20.538 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:20.538 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:20.538 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:23:20.538 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:58.900296+0000", 2024-09-13T11:23:20.538 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00013936299999999999 2024-09-13T11:23:20.539 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:20.539 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:23:20.539 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:20.539 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:20.539 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:20.539 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:16671 3.d 3:bf7f11f9:::benchmark_data_smithi177_31116_object16670:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-13T11:23:20.539 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:22:58.912164+0000", 2024-09-13T11:23:20.539 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 21.582449711999999, 2024-09-13T11:23:20.539 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.040706978999999997, 2024-09-13T11:23:20.539 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:23:20.539 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:23:20.539 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:20.539 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:23:20.539 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:23:20.540 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:20.540 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 16671 2024-09-13T11:23:20.540 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:20.540 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:23:20.540 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:20.540 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:23:20.540 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:58.912164+0000", 2024-09-13T11:23:20.540 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:20.540 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:20.540 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:20.540 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:23:20.540 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:58.912164+0000", 2024-09-13T11:23:20.540 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:20.541 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:20.541 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:20.541 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:23:20.541 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:58.912168+0000", 2024-09-13T11:23:20.541 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.6339999999999999e-06 2024-09-13T11:23:20.541 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:20.541 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:20.541 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:23:20.541 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:58.912226+0000", 2024-09-13T11:23:20.541 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.8251000000000002e-05 2024-09-13T11:23:20.541 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:20.541 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:20.541 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:23:20.542 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:58.912253+0000", 2024-09-13T11:23:20.542 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.6302999999999999e-05 2024-09-13T11:23:20.542 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:20.542 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:20.542 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:23:20.542 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:58.912274+0000", 2024-09-13T11:23:20.542 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.1899000000000002e-05 2024-09-13T11:23:20.542 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:20.542 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:20.542 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:23:20.542 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:58.912369+0000", 2024-09-13T11:23:20.542 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.4118999999999998e-05 2024-09-13T11:23:20.542 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:20.542 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:20.543 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:23:20.543 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:58.912604+0000", 2024-09-13T11:23:20.543 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00023492900000000001 2024-09-13T11:23:20.543 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:20.543 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:20.543 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 3", 2024-09-13T11:23:20.543 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:58.912786+0000", 2024-09-13T11:23:20.543 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00018197499999999999 2024-09-13T11:23:20.543 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:20.543 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:20.543 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:23:20.543 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:58.914454+0000", 2024-09-13T11:23:20.543 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0016681809999999999 2024-09-13T11:23:20.543 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:20.544 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:20.544 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:20.544 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:58.952590+0000", 2024-09-13T11:23:20.544 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.038135972999999997 2024-09-13T11:23:20.544 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:20.544 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:20.544 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:23:20.544 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:58.952750+0000", 2024-09-13T11:23:20.544 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000160032 2024-09-13T11:23:20.544 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:20.544 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:20.544 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:23:20.544 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:58.952871+0000", 2024-09-13T11:23:20.545 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00012168299999999999 2024-09-13T11:23:20.545 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:20.545 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:23:20.545 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:20.545 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:20.545 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:20.545 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:17011 3.a e52/30 3:5e3c00d0:::benchmark_data_smithi177_31116_object17010:head v 52'1050, mlcod=52'1046)", 2024-09-13T11:23:20.545 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:22:59.166097+0000", 2024-09-13T11:23:20.545 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 21.328517202, 2024-09-13T11:23:20.545 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.041158551000000002, 2024-09-13T11:23:20.545 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:23:20.545 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:23:20.545 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:20.546 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:23:20.546 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:20.546 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:23:20.546 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:59.166097+0000", 2024-09-13T11:23:20.546 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:20.546 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:20.546 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:20.546 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:23:20.546 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:59.166097+0000", 2024-09-13T11:23:20.546 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:20.546 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:20.546 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:20.546 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:23:20.547 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:59.166099+0000", 2024-09-13T11:23:20.547 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.2579999999999998e-06 2024-09-13T11:23:20.547 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:20.547 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:20.547 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:23:20.547 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:59.166141+0000", 2024-09-13T11:23:20.547 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.2292000000000001e-05 2024-09-13T11:23:20.547 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:20.547 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:20.547 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:23:20.547 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:59.200687+0000", 2024-09-13T11:23:20.547 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.034545619 2024-09-13T11:23:20.547 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:20.548 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:20.548 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:23:20.548 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:59.200701+0000", 2024-09-13T11:23:20.548 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3694e-05 2024-09-13T11:23:20.548 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:20.548 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:20.548 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:23:20.548 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:59.200839+0000", 2024-09-13T11:23:20.548 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00013825400000000001 2024-09-13T11:23:20.548 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:20.548 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:20.548 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:23:20.548 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:59.200871+0000", 2024-09-13T11:23:20.549 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.1720999999999997e-05 2024-09-13T11:23:20.549 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:20.549 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:20.549 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:23:20.549 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:59.207167+0000", 2024-09-13T11:23:20.549 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0062967500000000003 2024-09-13T11:23:20.549 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:20.549 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:20.549 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:23:20.549 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:59.207255+0000", 2024-09-13T11:23:20.549 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.7962999999999998e-05 2024-09-13T11:23:20.550 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:20.550 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:23:20.550 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:20.550 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:20.550 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:20.550 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:25019 3.3 e55/30 3:c7aa74ef:::benchmark_data_smithi177_31116_object25018:head v 55'1561, mlcod=55'1558)", 2024-09-13T11:23:20.550 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:04.043764+0000", 2024-09-13T11:23:20.550 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 16.450849785999999, 2024-09-13T11:23:20.550 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.044760751000000001, 2024-09-13T11:23:20.550 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:23:20.550 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:23:20.550 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:20.550 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:23:20.550 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:20.551 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:23:20.551 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.043764+0000", 2024-09-13T11:23:20.551 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:20.551 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:20.551 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:20.551 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:23:20.551 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.043764+0000", 2024-09-13T11:23:20.551 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:20.551 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:20.551 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:20.551 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:23:20.551 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.043766+0000", 2024-09-13T11:23:20.551 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5859999999999999e-06 2024-09-13T11:23:20.551 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:20.552 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:20.552 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:23:20.552 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.043822+0000", 2024-09-13T11:23:20.552 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.6181999999999998e-05 2024-09-13T11:23:20.552 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:20.552 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:20.552 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:23:20.552 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.087256+0000", 2024-09-13T11:23:20.552 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.043433893000000001 2024-09-13T11:23:20.552 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:20.552 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:20.552 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:23:20.552 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.087263+0000", 2024-09-13T11:23:20.552 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.6250000000000001e-06 2024-09-13T11:23:20.553 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:20.553 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:20.553 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:23:20.553 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.087309+0000", 2024-09-13T11:23:20.553 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.6354999999999999e-05 2024-09-13T11:23:20.553 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:20.553 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:20.553 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:23:20.553 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.087325+0000", 2024-09-13T11:23:20.553 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.6201e-05 2024-09-13T11:23:20.553 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:20.553 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:20.553 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:23:20.554 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.088492+0000", 2024-09-13T11:23:20.554 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001166858 2024-09-13T11:23:20.554 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:20.554 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:20.554 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:23:20.554 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.088525+0000", 2024-09-13T11:23:20.554 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.3050999999999999e-05 2024-09-13T11:23:20.554 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:20.554 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:23:20.554 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:20.554 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:20.554 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:20.554 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:25110 3.3 e55/30 3:c25a0b62:::benchmark_data_smithi177_31116_object25109:head v 55'1564, mlcod=55'1561)", 2024-09-13T11:23:20.554 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:04.096802+0000", 2024-09-13T11:23:20.555 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 16.397812564999999, 2024-09-13T11:23:20.555 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.050712050000000002, 2024-09-13T11:23:20.555 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:23:20.555 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:23:20.555 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:20.555 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:23:20.555 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:20.555 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:23:20.555 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.096802+0000", 2024-09-13T11:23:20.555 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:20.555 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:20.555 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:20.556 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:23:20.556 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.096802+0000", 2024-09-13T11:23:20.556 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:20.556 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:20.556 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:20.556 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:23:20.556 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.096803+0000", 2024-09-13T11:23:20.556 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.42e-06 2024-09-13T11:23:20.556 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:20.556 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:20.556 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:23:20.556 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.096857+0000", 2024-09-13T11:23:20.556 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.4521000000000002e-05 2024-09-13T11:23:20.556 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:20.557 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:20.557 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:23:20.557 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.146038+0000", 2024-09-13T11:23:20.557 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.049180486000000002 2024-09-13T11:23:20.557 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:20.557 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:20.557 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:23:20.557 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.146044+0000", 2024-09-13T11:23:20.557 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.6230000000000002e-06 2024-09-13T11:23:20.557 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:20.557 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:20.557 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:23:20.557 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.146118+0000", 2024-09-13T11:23:20.558 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.4649999999999998e-05 2024-09-13T11:23:20.558 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:20.558 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:20.558 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:23:20.558 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.146139+0000", 2024-09-13T11:23:20.558 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.0920999999999998e-05 2024-09-13T11:23:20.558 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:20.558 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:20.558 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:23:20.558 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.147469+0000", 2024-09-13T11:23:20.558 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0013296740000000001 2024-09-13T11:23:20.558 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:20.558 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:20.558 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:23:20.559 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.147514+0000", 2024-09-13T11:23:20.559 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.4755000000000001e-05 2024-09-13T11:23:20.559 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:20.559 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:23:20.559 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:20.559 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:20.559 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:20.559 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:25155 3.d 3:b276f5eb:::benchmark_data_smithi177_31116_object25154:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e55)", 2024-09-13T11:23:20.559 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:04.143935+0000", 2024-09-13T11:23:20.559 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 16.350679493000001, 2024-09-13T11:23:20.559 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.046542231000000003, 2024-09-13T11:23:20.559 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:23:20.559 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:23:20.559 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:20.560 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:23:20.560 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:23:20.560 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:20.560 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 25155 2024-09-13T11:23:20.560 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:20.560 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:23:20.560 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:20.560 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:23:20.560 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.143935+0000", 2024-09-13T11:23:20.560 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:20.560 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:20.560 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:20.560 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:23:20.561 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.143935+0000", 2024-09-13T11:23:20.561 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:20.561 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:20.561 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:20.561 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:23:20.561 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.143937+0000", 2024-09-13T11:23:20.561 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.757e-06 2024-09-13T11:23:20.561 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:20.561 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:20.561 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:23:20.561 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.143964+0000", 2024-09-13T11:23:20.561 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.6854000000000001e-05 2024-09-13T11:23:20.561 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:20.562 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:20.562 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:23:20.562 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.143979+0000", 2024-09-13T11:23:20.562 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.4883e-05 2024-09-13T11:23:20.562 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:20.562 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:20.562 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:23:20.562 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.143987+0000", 2024-09-13T11:23:20.562 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.6429999999999995e-06 2024-09-13T11:23:20.562 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:20.562 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:20.562 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:23:20.562 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.144056+0000", 2024-09-13T11:23:20.562 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.9610000000000006e-05 2024-09-13T11:23:20.562 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:20.563 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:20.563 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:23:20.563 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.144218+0000", 2024-09-13T11:23:20.563 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00016203300000000001 2024-09-13T11:23:20.563 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:20.563 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:20.563 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 3", 2024-09-13T11:23:20.563 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.144380+0000", 2024-09-13T11:23:20.563 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000161776 2024-09-13T11:23:20.563 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:20.563 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:20.563 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:23:20.563 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.145780+0000", 2024-09-13T11:23:20.563 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0013996620000000001 2024-09-13T11:23:20.563 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:20.564 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:20.564 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:20.564 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.190380+0000", 2024-09-13T11:23:20.564 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.044600534999999997 2024-09-13T11:23:20.564 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:20.564 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:20.564 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:23:20.564 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.190443+0000", 2024-09-13T11:23:20.564 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.2353000000000004e-05 2024-09-13T11:23:20.564 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:20.564 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:20.564 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:23:20.564 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.190477+0000", 2024-09-13T11:23:20.565 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.4125000000000003e-05 2024-09-13T11:23:20.565 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:20.565 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:23:20.565 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:20.565 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:20.565 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:20.565 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop_reply(client.4665.0:25155 3.d e55/30 ondisk, result = 0)", 2024-09-13T11:23:20.565 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:04.146494+0000", 2024-09-13T11:23:20.565 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 16.348119763, 2024-09-13T11:23:20.565 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.043989937999999999, 2024-09-13T11:23:20.565 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:23:20.565 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:23:20.565 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:23:20.566 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:23:20.566 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:20.566 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:23:20.566 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.146494+0000", 2024-09-13T11:23:20.566 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:20.566 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:20.566 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:20.566 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:23:20.566 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.146494+0000", 2024-09-13T11:23:20.566 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:20.566 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:20.566 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:20.566 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:23:20.567 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.146496+0000", 2024-09-13T11:23:20.567 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.2079999999999999e-06 2024-09-13T11:23:20.567 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:20.567 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:20.567 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:23:20.567 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.146500+0000", 2024-09-13T11:23:20.567 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.1230000000000002e-06 2024-09-13T11:23:20.567 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:20.567 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:20.567 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:23:20.567 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.190294+0000", 2024-09-13T11:23:20.567 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.043793987999999999 2024-09-13T11:23:20.567 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:20.567 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:20.568 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:23:20.568 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.190303+0000", 2024-09-13T11:23:20.568 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.1249999999999999e-06 2024-09-13T11:23:20.568 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:20.568 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:20.568 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:23:20.568 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.190364+0000", 2024-09-13T11:23:20.568 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.1110000000000003e-05 2024-09-13T11:23:20.568 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:20.568 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:20.568 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:23:20.568 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.190373+0000", 2024-09-13T11:23:20.568 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.7630000000000006e-06 2024-09-13T11:23:20.569 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:20.569 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:20.569 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:23:20.569 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.190484+0000", 2024-09-13T11:23:20.569 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00011162099999999999 2024-09-13T11:23:20.569 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:20.569 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:23:20.569 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:20.569 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:20.569 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:20.569 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:30284 3.3 e59/30 3:c4c1f51c:::benchmark_data_smithi177_31116_object30283:head v 59'1908, mlcod=59'1904)", 2024-09-13T11:23:20.569 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:09.095510+0000", 2024-09-13T11:23:20.570 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 11.399103634999999, 2024-09-13T11:23:20.570 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.067957288000000005, 2024-09-13T11:23:20.570 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:23:20.570 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:23:20.570 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:20.570 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:23:20.570 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:20.570 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:23:20.570 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.095510+0000", 2024-09-13T11:23:20.570 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:20.570 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:20.570 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:20.570 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:23:20.570 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.095510+0000", 2024-09-13T11:23:20.571 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:20.571 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:20.571 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:20.571 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:23:20.571 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.095513+0000", 2024-09-13T11:23:20.571 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.1770000000000001e-06 2024-09-13T11:23:20.571 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:20.571 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:20.571 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:23:20.571 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.095583+0000", 2024-09-13T11:23:20.571 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.0383999999999996e-05 2024-09-13T11:23:20.571 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:20.571 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:20.572 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:23:20.572 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.160868+0000", 2024-09-13T11:23:20.572 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.065284810999999998 2024-09-13T11:23:20.572 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:20.572 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:20.572 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:23:20.572 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.160875+0000", 2024-09-13T11:23:20.572 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.2760000000000003e-06 2024-09-13T11:23:20.572 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:20.572 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:20.572 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:23:20.572 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.160966+0000", 2024-09-13T11:23:20.572 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.0421999999999996e-05 2024-09-13T11:23:20.573 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:20.573 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:20.573 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:23:20.573 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.161006+0000", 2024-09-13T11:23:20.573 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.0809999999999997e-05 2024-09-13T11:23:20.573 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:20.573 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:20.573 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:23:20.573 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.163367+0000", 2024-09-13T11:23:20.573 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0023606040000000001 2024-09-13T11:23:20.573 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:20.573 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:20.573 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:23:20.573 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.163468+0000", 2024-09-13T11:23:20.573 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00010080400000000001 2024-09-13T11:23:20.574 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:20.574 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:23:20.574 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:20.574 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:20.574 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:20.574 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:31097 3.d 3:bc3dc2d6:::benchmark_data_smithi177_31116_object31096:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e59)", 2024-09-13T11:23:20.574 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:09.956171+0000", 2024-09-13T11:23:20.574 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 10.538443411999999, 2024-09-13T11:23:20.574 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.045191461000000002, 2024-09-13T11:23:20.574 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:23:20.574 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:23:20.574 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:20.574 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:23:20.574 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:23:20.575 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:20.575 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 31097 2024-09-13T11:23:20.575 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:20.575 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:23:20.575 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:20.575 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:23:20.575 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.956171+0000", 2024-09-13T11:23:20.575 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:20.575 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:20.575 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:20.575 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:23:20.575 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.956171+0000", 2024-09-13T11:23:20.575 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:20.575 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:20.576 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:20.576 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:23:20.576 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.956174+0000", 2024-09-13T11:23:20.576 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.1729999999999999e-06 2024-09-13T11:23:20.576 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:20.576 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:20.576 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:23:20.576 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.956252+0000", 2024-09-13T11:23:20.576 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.8590000000000005e-05 2024-09-13T11:23:20.576 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:20.576 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:20.576 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:23:20.576 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.956280+0000", 2024-09-13T11:23:20.576 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.7735000000000001e-05 2024-09-13T11:23:20.577 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:20.577 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:20.577 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:23:20.577 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.956297+0000", 2024-09-13T11:23:20.577 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.6478999999999999e-05 2024-09-13T11:23:20.577 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:20.577 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:20.577 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:23:20.577 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.957164+0000", 2024-09-13T11:23:20.577 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00086682800000000004 2024-09-13T11:23:20.577 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:20.577 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:20.578 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:23:20.578 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.957408+0000", 2024-09-13T11:23:20.578 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00024422000000000002 2024-09-13T11:23:20.578 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:20.578 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:20.578 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 3", 2024-09-13T11:23:20.578 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.957689+0000", 2024-09-13T11:23:20.578 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00028150199999999999 2024-09-13T11:23:20.578 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:20.578 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:20.578 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:23:20.578 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.960119+0000", 2024-09-13T11:23:20.578 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0024294199999999998 2024-09-13T11:23:20.578 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:20.579 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:20.579 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:20.579 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:10.001161+0000", 2024-09-13T11:23:20.579 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.041042793000000001 2024-09-13T11:23:20.579 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:20.579 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:20.579 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:23:20.579 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:10.001275+0000", 2024-09-13T11:23:20.579 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000113433 2024-09-13T11:23:20.579 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:20.579 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:20.579 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:23:20.579 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:10.001362+0000", 2024-09-13T11:23:20.579 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.7287999999999995e-05 2024-09-13T11:23:20.580 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:20.580 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:23:20.580 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:20.580 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:20.580 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:20.580 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:31207 3.d 3:b93b1bf0:::benchmark_data_smithi177_31116_object31206:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e59)", 2024-09-13T11:23:20.580 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:10.061635+0000", 2024-09-13T11:23:20.580 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 10.432979394, 2024-09-13T11:23:20.580 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.039943030999999997, 2024-09-13T11:23:20.580 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:23:20.580 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:23:20.580 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:20.580 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:23:20.580 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:23:20.581 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:20.581 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 31207 2024-09-13T11:23:20.581 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:20.581 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:23:20.581 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:20.581 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:23:20.581 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:10.061635+0000", 2024-09-13T11:23:20.581 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:20.581 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:20.581 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:20.581 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:23:20.581 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:10.061635+0000", 2024-09-13T11:23:20.581 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:20.582 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:20.582 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:20.582 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:23:20.582 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:10.061639+0000", 2024-09-13T11:23:20.582 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.5539999999999999e-06 2024-09-13T11:23:20.582 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:20.582 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:20.582 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:23:20.582 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:10.061692+0000", 2024-09-13T11:23:20.582 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.2435999999999997e-05 2024-09-13T11:23:20.582 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:20.582 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:20.582 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:23:20.582 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:10.061727+0000", 2024-09-13T11:23:20.582 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.4848999999999998e-05 2024-09-13T11:23:20.583 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:20.583 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:20.583 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:23:20.583 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:10.061746+0000", 2024-09-13T11:23:20.583 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.9381e-05 2024-09-13T11:23:20.583 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:20.583 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:20.583 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:23:20.583 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:10.061863+0000", 2024-09-13T11:23:20.583 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00011680699999999999 2024-09-13T11:23:20.583 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:20.583 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:20.583 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:23:20.584 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:10.062017+0000", 2024-09-13T11:23:20.584 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000154456 2024-09-13T11:23:20.584 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:20.584 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:20.584 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 3", 2024-09-13T11:23:20.584 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:10.062190+0000", 2024-09-13T11:23:20.584 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00017256500000000001 2024-09-13T11:23:20.584 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:20.584 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:20.584 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:23:20.584 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:10.067196+0000", 2024-09-13T11:23:20.584 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.005005888 2024-09-13T11:23:20.584 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:20.585 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:20.585 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:20.585 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:10.101494+0000", 2024-09-13T11:23:20.585 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.034298593000000002 2024-09-13T11:23:20.585 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:20.585 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:20.585 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:23:20.585 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:10.101540+0000", 2024-09-13T11:23:20.585 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.5636999999999997e-05 2024-09-13T11:23:20.585 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:20.585 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:20.585 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:23:20.585 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:10.101578+0000", 2024-09-13T11:23:20.585 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.7864999999999997e-05 2024-09-13T11:23:20.586 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:20.586 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:23:20.586 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:20.586 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:20.586 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:20.586 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:31212 3.d 3:b65a0c64:::benchmark_data_smithi177_31116_object31211:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e59)", 2024-09-13T11:23:20.586 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:10.063100+0000", 2024-09-13T11:23:20.586 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 10.431514487999999, 2024-09-13T11:23:20.586 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.038612104000000001, 2024-09-13T11:23:20.586 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:23:20.586 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:23:20.586 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:20.586 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:23:20.586 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:23:20.587 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:20.587 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 31212 2024-09-13T11:23:20.587 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:20.587 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:23:20.587 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:20.587 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:23:20.587 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:10.063100+0000", 2024-09-13T11:23:20.587 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:20.587 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:20.587 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:20.587 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:23:20.587 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:10.063100+0000", 2024-09-13T11:23:20.587 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:20.587 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:20.588 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:20.588 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:23:20.588 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:10.063103+0000", 2024-09-13T11:23:20.588 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.45e-06 2024-09-13T11:23:20.588 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:20.588 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:20.588 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:23:20.588 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:10.063154+0000", 2024-09-13T11:23:20.588 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.0939999999999997e-05 2024-09-13T11:23:20.588 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:20.588 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:20.588 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:23:20.588 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:10.063187+0000", 2024-09-13T11:23:20.589 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.2951999999999999e-05 2024-09-13T11:23:20.589 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:20.589 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:20.589 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:23:20.589 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:10.063203+0000", 2024-09-13T11:23:20.589 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5835999999999999e-05 2024-09-13T11:23:20.589 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:20.589 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:20.589 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:23:20.589 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:10.064010+0000", 2024-09-13T11:23:20.589 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00080766500000000005 2024-09-13T11:23:20.589 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:20.589 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:20.590 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:23:20.590 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:10.064275+0000", 2024-09-13T11:23:20.590 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00026461099999999999 2024-09-13T11:23:20.590 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:20.590 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:20.590 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 3", 2024-09-13T11:23:20.590 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:10.064509+0000", 2024-09-13T11:23:20.590 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00023429200000000001 2024-09-13T11:23:20.590 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:20.590 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:20.590 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:23:20.590 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:10.067659+0000", 2024-09-13T11:23:20.590 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0031491689999999998 2024-09-13T11:23:20.590 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:20.590 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:20.591 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:20.591 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:10.101620+0000", 2024-09-13T11:23:20.591 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.033961586000000002 2024-09-13T11:23:20.591 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:20.591 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:20.591 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:23:20.591 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:10.101660+0000", 2024-09-13T11:23:20.591 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.0061999999999997e-05 2024-09-13T11:23:20.591 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:20.591 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:20.591 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:23:20.591 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:10.101712+0000", 2024-09-13T11:23:20.591 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.1541e-05 2024-09-13T11:23:20.591 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:20.591 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:23:20.592 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:20.592 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:20.592 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:20.592 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:39878 3.3 e64/30 3:c69ff34d:::benchmark_data_smithi177_31116_object39877:head v 64'2548, mlcod=64'2545)", 2024-09-13T11:23:20.592 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:15.737082+0000", 2024-09-13T11:23:20.592 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 4.7575316929999998, 2024-09-13T11:23:20.592 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.038156546999999999, 2024-09-13T11:23:20.592 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:23:20.592 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:23:20.592 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:20.592 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:23:20.592 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:20.592 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:23:20.592 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:15.737082+0000", 2024-09-13T11:23:20.593 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:20.593 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:20.593 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:20.593 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:23:20.593 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:15.737082+0000", 2024-09-13T11:23:20.593 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:20.593 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:20.593 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:20.593 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:23:20.593 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:15.737085+0000", 2024-09-13T11:23:20.593 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.5600000000000001e-06 2024-09-13T11:23:20.593 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:20.593 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:20.594 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:23:20.594 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:15.737138+0000", 2024-09-13T11:23:20.594 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.2905999999999998e-05 2024-09-13T11:23:20.594 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:20.594 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:20.594 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:23:20.594 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:15.773682+0000", 2024-09-13T11:23:20.594 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.036543921 2024-09-13T11:23:20.594 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:20.594 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:20.594 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:23:20.594 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:15.773690+0000", 2024-09-13T11:23:20.594 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.9489999999999996e-06 2024-09-13T11:23:20.594 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:20.595 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:20.595 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:23:20.595 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:15.773742+0000", 2024-09-13T11:23:20.595 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.2589000000000001e-05 2024-09-13T11:23:20.595 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:20.595 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:20.595 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:23:20.595 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:15.773755+0000", 2024-09-13T11:23:20.595 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3127e-05 2024-09-13T11:23:20.595 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:20.595 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:20.595 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:23:20.595 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:15.775190+0000", 2024-09-13T11:23:20.595 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0014350019999999999 2024-09-13T11:23:20.595 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:20.596 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:20.596 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:23:20.596 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:15.775239+0000", 2024-09-13T11:23:20.596 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.8492999999999998e-05 2024-09-13T11:23:20.596 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:20.596 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:23:20.596 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:20.596 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:20.596 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:20.596 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:39935 3.3 e64/30 3:c6ae38b7:::benchmark_data_smithi177_31116_object39934:head v 64'2551, mlcod=64'2548)", 2024-09-13T11:23:20.596 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:15.786939+0000", 2024-09-13T11:23:20.596 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 4.7076755510000003, 2024-09-13T11:23:20.596 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.058508338999999999, 2024-09-13T11:23:20.596 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:23:20.597 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:23:20.597 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:20.597 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:23:20.597 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:20.597 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:23:20.597 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:15.786939+0000", 2024-09-13T11:23:20.597 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:20.597 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:20.597 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:20.597 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:23:20.597 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:15.786939+0000", 2024-09-13T11:23:20.597 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:20.597 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:20.598 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:20.598 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:23:20.598 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:15.786940+0000", 2024-09-13T11:23:20.598 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.4619999999999999e-06 2024-09-13T11:23:20.598 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:20.598 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:20.598 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:23:20.598 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:15.786989+0000", 2024-09-13T11:23:20.598 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.9082000000000002e-05 2024-09-13T11:23:20.598 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:20.598 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:20.598 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:23:20.598 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:15.843720+0000", 2024-09-13T11:23:20.598 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.056730506999999999 2024-09-13T11:23:20.599 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:20.599 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:20.599 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:23:20.599 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:15.843726+0000", 2024-09-13T11:23:20.599 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.9050000000000002e-06 2024-09-13T11:23:20.599 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:20.599 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:20.599 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:23:20.599 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:15.843777+0000", 2024-09-13T11:23:20.599 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.1891000000000002e-05 2024-09-13T11:23:20.599 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:20.599 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:20.599 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:23:20.599 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:15.843794+0000", 2024-09-13T11:23:20.600 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.6787999999999999e-05 2024-09-13T11:23:20.600 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:20.600 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:20.600 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:23:20.600 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:15.845414+0000", 2024-09-13T11:23:20.600 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001620291 2024-09-13T11:23:20.600 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:20.600 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:20.600 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:23:20.600 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:15.845447+0000", 2024-09-13T11:23:20.600 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.2413000000000003e-05 2024-09-13T11:23:20.600 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:20.600 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:23:20.601 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:20.601 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:20.601 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:20.601 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:39999 3.3 e64/30 3:cf9d09f2:::benchmark_data_smithi177_31116_object39998:head v 64'2552, mlcod=64'2549)", 2024-09-13T11:23:20.601 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:15.843659+0000", 2024-09-13T11:23:20.601 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 4.650954714, 2024-09-13T11:23:20.601 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.038964875000000003, 2024-09-13T11:23:20.601 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:23:20.601 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:23:20.601 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:20.601 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:23:20.601 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:20.601 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:23:20.602 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:15.843659+0000", 2024-09-13T11:23:20.602 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:20.602 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:20.602 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:20.602 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:23:20.602 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:15.843659+0000", 2024-09-13T11:23:20.602 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:20.602 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:20.602 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:20.602 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:23:20.602 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:15.843661+0000", 2024-09-13T11:23:20.602 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.615e-06 2024-09-13T11:23:20.602 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:20.602 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:20.603 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:23:20.603 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:15.843703+0000", 2024-09-13T11:23:20.603 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.1575000000000001e-05 2024-09-13T11:23:20.603 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:20.603 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:20.603 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:23:20.603 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:15.880875+0000", 2024-09-13T11:23:20.603 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.037172822000000001 2024-09-13T11:23:20.603 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:20.603 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:20.603 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:23:20.603 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:15.880881+0000", 2024-09-13T11:23:20.603 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.558e-06 2024-09-13T11:23:20.603 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:20.604 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:20.604 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:23:20.604 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:15.880920+0000", 2024-09-13T11:23:20.604 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.8708999999999998e-05 2024-09-13T11:23:20.604 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:20.604 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:20.604 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:23:20.604 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:15.880932+0000", 2024-09-13T11:23:20.604 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.2662e-05 2024-09-13T11:23:20.604 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:20.604 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:20.604 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:23:20.604 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:15.882579+0000", 2024-09-13T11:23:20.605 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0016464159999999999 2024-09-13T11:23:20.605 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:20.605 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:20.605 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:23:20.605 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:15.882624+0000", 2024-09-13T11:23:20.605 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.5518e-05 2024-09-13T11:23:20.605 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:20.605 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:23:20.605 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:20.605 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:20.605 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:23:20.605 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-09-13T11:23:20.607 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-09-13T11:23:20.608 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-13T11:23:20.600+0000 7f6a68775640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26307) UID: 0 2024-09-13T11:23:20.708 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-13T11:23:20.704+0000 7f6a68775640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26307) UID: 0 2024-09-13T11:23:20.744 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-09-13T11:23:20.745 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [ 2024-09-13T11:23:20.745 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:20.745 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:45514 3.c 3:3713d632:::benchmark_data_smithi177_31116_object45513:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e68)", 2024-09-13T11:23:20.745 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:20.740552+0000", 2024-09-13T11:23:20.745 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 0.0042155090000000001, 2024-09-13T11:23:20.745 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0042408480000000002, 2024-09-13T11:23:20.745 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:23:20.745 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:23:20.745 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "waiting for sub ops", 2024-09-13T11:23:20.745 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:23:20.745 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:23:20.745 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:20.746 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 45514 2024-09-13T11:23:20.746 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:20.746 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:23:20.746 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:20.746 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:23:20.746 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:20.740552+0000", 2024-09-13T11:23:20.746 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:20.746 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:20.746 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:20.746 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:23:20.746 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:20.740552+0000", 2024-09-13T11:23:20.746 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:20.746 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:20.746 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:20.747 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:23:20.747 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:20.740554+0000", 2024-09-13T11:23:20.747 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3009999999999999e-06 2024-09-13T11:23:20.747 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:20.747 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:20.747 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:23:20.747 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:20.740581+0000", 2024-09-13T11:23:20.747 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.7563999999999999e-05 2024-09-13T11:23:20.747 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:20.747 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:20.747 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:23:20.747 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:20.740591+0000", 2024-09-13T11:23:20.747 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.3370000000000006e-06 2024-09-13T11:23:20.748 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:20.748 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:20.748 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:23:20.748 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:20.740596+0000", 2024-09-13T11:23:20.748 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.2129999999999999e-06 2024-09-13T11:23:20.748 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:20.748 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:20.748 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:23:20.748 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:20.740639+0000", 2024-09-13T11:23:20.748 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.3359999999999998e-05 2024-09-13T11:23:20.748 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:20.748 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:20.748 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:23:20.749 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:20.740731+0000", 2024-09-13T11:23:20.749 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.1951000000000001e-05 2024-09-13T11:23:20.749 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:20.749 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:20.749 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 13", 2024-09-13T11:23:20.749 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:20.740810+0000", 2024-09-13T11:23:20.749 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.8386000000000005e-05 2024-09-13T11:23:20.749 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:20.749 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:20.749 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:23:20.749 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:20.743448+0000", 2024-09-13T11:23:20.749 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0026383660000000001 2024-09-13T11:23:20.749 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:20.749 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:23:20.750 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:20.750 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:20.750 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:20.750 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:45515 3.c 3:3109a554:::benchmark_data_smithi177_31116_object45514:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e68)", 2024-09-13T11:23:20.750 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:20.741820+0000", 2024-09-13T11:23:20.750 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 0.0029475230000000001, 2024-09-13T11:23:20.750 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0030564730000000001, 2024-09-13T11:23:20.750 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:23:20.750 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:23:20.750 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "waiting for sub ops", 2024-09-13T11:23:20.750 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:23:20.750 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:23:20.750 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:20.751 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 45515 2024-09-13T11:23:20.751 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:20.751 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:23:20.751 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:20.751 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:23:20.751 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:20.741820+0000", 2024-09-13T11:23:20.751 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:20.751 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:20.751 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:20.751 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:23:20.751 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:20.741820+0000", 2024-09-13T11:23:20.751 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:20.751 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:20.751 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:20.752 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:23:20.752 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:20.741822+0000", 2024-09-13T11:23:20.752 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.508e-06 2024-09-13T11:23:20.752 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:20.752 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:20.752 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:23:20.752 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:20.741856+0000", 2024-09-13T11:23:20.752 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.3905000000000001e-05 2024-09-13T11:23:20.752 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:20.752 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:20.752 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:23:20.752 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:20.741866+0000", 2024-09-13T11:23:20.753 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0239e-05 2024-09-13T11:23:20.753 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:20.753 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:20.753 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:23:20.753 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:20.741875+0000", 2024-09-13T11:23:20.753 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.8710000000000003e-06 2024-09-13T11:23:20.753 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:20.753 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:20.753 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:23:20.753 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:20.742266+0000", 2024-09-13T11:23:20.753 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00039116999999999998 2024-09-13T11:23:20.753 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:20.754 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:20.754 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:23:20.754 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:20.742372+0000", 2024-09-13T11:23:20.754 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000106331 2024-09-13T11:23:20.754 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:20.754 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:20.754 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 13", 2024-09-13T11:23:20.754 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:20.742456+0000", 2024-09-13T11:23:20.754 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.3275000000000005e-05 2024-09-13T11:23:20.754 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:20.754 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:20.754 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:23:20.754 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:20.743770+0000", 2024-09-13T11:23:20.755 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0013139530000000001 2024-09-13T11:23:20.755 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:20.755 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:23:20.755 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:20.755 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:20.755 INFO:teuthology.orchestra.run.smithi177.stdout: ], 2024-09-13T11:23:20.755 INFO:teuthology.orchestra.run.smithi177.stdout: "num_ops": 2 2024-09-13T11:23:20.755 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-09-13T11:23:20.762 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-09-13T11:23:20.809 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-09-13T11:23:20.802+0000 7f97befc9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26315) UID: 0 2024-09-13T11:23:20.901 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-09-13T11:23:20.901 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [], 2024-09-13T11:23:20.901 INFO:teuthology.orchestra.run.smithi177.stdout: "complaint_time": 30, 2024-09-13T11:23:20.901 INFO:teuthology.orchestra.run.smithi177.stdout: "num_blocked_ops": 0 2024-09-13T11:23:20.901 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-09-13T11:23:20.910 INFO:tasks.ceph.osd.14.smithi162.stderr:2024-09-13T11:23:20.906+0000 7f235c0db640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26319) UID: 0 2024-09-13T11:23:20.921 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-09-13T11:23:20.938 INFO:tasks.ceph.osd.14.smithi162.stderr:2024-09-13T11:23:20.934+0000 7f2350b08640 -1 osd.14 68 handle_osd_map injecting map crc failure 2024-09-13T11:23:20.942 INFO:tasks.ceph.osd.7.smithi177.stderr:2024-09-13T11:23:20.937+0000 7f896ff4a640 -1 osd.7 68 handle_osd_map injecting map crc failure 2024-09-13T11:23:21.003 INFO:tasks.radosbench.radosbench.0.smithi177.stdout: 54 16 45789 45773 52.9721 98.4375 0.00445913 0.0188502 2024-09-13T11:23:21.010 INFO:tasks.ceph.osd.11.smithi177.stderr:2024-09-13T11:23:21.005+0000 7f24792c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26580) UID: 0 2024-09-13T11:23:21.036 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-09-13T11:23:21.036 INFO:teuthology.orchestra.run.smithi177.stdout: "size": 20, 2024-09-13T11:23:21.036 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 600, 2024-09-13T11:23:21.036 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [ 2024-09-13T11:23:21.036 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.036 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:79 3.1 e32/30 3:841df5b5:::benchmark_data_smithi177_31116_object78:head v 32'6, mlcod=32'4)", 2024-09-13T11:23:21.037 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:22:27.057292+0000", 2024-09-13T11:23:21.037 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 53.977810040000001, 2024-09-13T11:23:21.037 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.89296786800000005, 2024-09-13T11:23:21.037 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:23:21.037 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:23:21.037 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:21.037 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:23:21.037 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.037 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:23:21.037 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.057292+0000", 2024-09-13T11:23:21.037 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:21.037 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.037 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.038 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:23:21.038 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.057292+0000", 2024-09-13T11:23:21.038 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:21.038 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.038 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.038 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:23:21.038 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.057293+0000", 2024-09-13T11:23:21.038 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.6109999999999999e-06 2024-09-13T11:23:21.038 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.038 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.038 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:23:21.038 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.057330+0000", 2024-09-13T11:23:21.038 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.6721999999999998e-05 2024-09-13T11:23:21.039 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.039 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.039 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:23:21.039 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.948197+0000", 2024-09-13T11:23:21.039 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.890867042 2024-09-13T11:23:21.039 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.039 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.039 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:23:21.039 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.948213+0000", 2024-09-13T11:23:21.039 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5826000000000001e-05 2024-09-13T11:23:21.039 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.039 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.039 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:23:21.039 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.948377+0000", 2024-09-13T11:23:21.040 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00016348600000000001 2024-09-13T11:23:21.040 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.040 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.040 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:23:21.040 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.948447+0000", 2024-09-13T11:23:21.040 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.0891999999999998e-05 2024-09-13T11:23:21.040 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.040 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.040 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:23:21.040 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.950150+0000", 2024-09-13T11:23:21.040 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001702329 2024-09-13T11:23:21.040 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.040 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.041 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:23:21.041 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.950260+0000", 2024-09-13T11:23:21.041 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00010996 2024-09-13T11:23:21.041 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:21.041 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:23:21.041 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:21.041 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.041 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.041 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:812 3.f 3:fbaccdd3:::benchmark_data_smithi177_31116_object811:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-09-13T11:23:21.041 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:22:49.446605+0000", 2024-09-13T11:23:21.041 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 31.588496616, 2024-09-13T11:23:21.041 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.085769887000000003, 2024-09-13T11:23:21.041 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:23:21.042 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:23:21.042 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:21.042 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:23:21.042 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:23:21.042 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:21.042 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 812 2024-09-13T11:23:21.042 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.042 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:23:21.042 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.042 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:23:21.042 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.446605+0000", 2024-09-13T11:23:21.042 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:21.042 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.042 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.043 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:23:21.043 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.446605+0000", 2024-09-13T11:23:21.043 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:21.043 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.043 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.043 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:23:21.043 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.446607+0000", 2024-09-13T11:23:21.043 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.934e-06 2024-09-13T11:23:21.043 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.043 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.043 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:23:21.043 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.446656+0000", 2024-09-13T11:23:21.043 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.8368999999999997e-05 2024-09-13T11:23:21.044 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.044 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.044 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:23:21.044 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.446677+0000", 2024-09-13T11:23:21.044 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.1367000000000001e-05 2024-09-13T11:23:21.044 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.044 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.044 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:23:21.044 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.446689+0000", 2024-09-13T11:23:21.044 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.1809e-05 2024-09-13T11:23:21.044 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.044 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.044 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:23:21.044 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.446766+0000", 2024-09-13T11:23:21.045 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.7092e-05 2024-09-13T11:23:21.045 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.045 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.045 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:23:21.045 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.446917+0000", 2024-09-13T11:23:21.045 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00015103800000000001 2024-09-13T11:23:21.045 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.045 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.045 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 12", 2024-09-13T11:23:21.045 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.447076+0000", 2024-09-13T11:23:21.045 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00015952699999999999 2024-09-13T11:23:21.045 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.046 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.046 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:23:21.046 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.447922+0000", 2024-09-13T11:23:21.046 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00084538099999999998 2024-09-13T11:23:21.046 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.046 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.046 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:21.046 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.532308+0000", 2024-09-13T11:23:21.046 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.084386037999999997 2024-09-13T11:23:21.046 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.046 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.046 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:23:21.046 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.532345+0000", 2024-09-13T11:23:21.046 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.676e-05 2024-09-13T11:23:21.047 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.047 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.047 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:23:21.047 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.532375+0000", 2024-09-13T11:23:21.047 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.0571999999999998e-05 2024-09-13T11:23:21.047 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:21.047 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:23:21.047 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:21.047 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.047 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.047 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop_reply(client.4665.0:812 3.f e43/30 ondisk, result = 0)", 2024-09-13T11:23:21.047 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:22:49.448230+0000", 2024-09-13T11:23:21.048 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 31.586872121999999, 2024-09-13T11:23:21.048 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.084151144999999997, 2024-09-13T11:23:21.048 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:23:21.048 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:23:21.048 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:23:21.048 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:23:21.048 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.048 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:23:21.048 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.448230+0000", 2024-09-13T11:23:21.048 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:21.048 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.048 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.048 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:23:21.049 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.448230+0000", 2024-09-13T11:23:21.049 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:21.049 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.049 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.049 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:23:21.049 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.448231+0000", 2024-09-13T11:23:21.049 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.6020000000000001e-06 2024-09-13T11:23:21.049 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.049 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.049 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:23:21.049 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.448236+0000", 2024-09-13T11:23:21.049 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.4669999999999998e-06 2024-09-13T11:23:21.049 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.050 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.050 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:23:21.050 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.532240+0000", 2024-09-13T11:23:21.050 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.084004520999999999 2024-09-13T11:23:21.050 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.050 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.050 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:23:21.050 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.532247+0000", 2024-09-13T11:23:21.050 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.8979999999999999e-06 2024-09-13T11:23:21.050 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.050 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.050 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:23:21.050 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.532299+0000", 2024-09-13T11:23:21.051 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.1372999999999998e-05 2024-09-13T11:23:21.051 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.051 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.051 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:23:21.051 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.532304+0000", 2024-09-13T11:23:21.051 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.6219999999999997e-06 2024-09-13T11:23:21.051 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.051 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.051 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:23:21.051 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.532381+0000", 2024-09-13T11:23:21.051 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.6662000000000006e-05 2024-09-13T11:23:21.051 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:21.052 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:23:21.052 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:21.052 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.052 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.052 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:9906 3.f 3:fea9a3d3:::benchmark_data_smithi177_31116_object9905:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-09-13T11:23:21.052 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:22:54.302935+0000", 2024-09-13T11:23:21.052 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 26.732166696, 2024-09-13T11:23:21.052 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.076725138999999998, 2024-09-13T11:23:21.052 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:23:21.052 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:23:21.052 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:21.052 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:23:21.052 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:23:21.053 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:21.053 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 9906 2024-09-13T11:23:21.053 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.053 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:23:21.053 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.053 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:23:21.053 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:54.302935+0000", 2024-09-13T11:23:21.053 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:21.053 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.053 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.053 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:23:21.053 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:54.302935+0000", 2024-09-13T11:23:21.053 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:21.054 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.054 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.054 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:23:21.054 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:54.302937+0000", 2024-09-13T11:23:21.054 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.739e-06 2024-09-13T11:23:21.054 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.054 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.054 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:23:21.054 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:54.302962+0000", 2024-09-13T11:23:21.054 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.4578000000000001e-05 2024-09-13T11:23:21.054 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.054 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.054 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:23:21.055 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:54.302973+0000", 2024-09-13T11:23:21.055 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.1949e-05 2024-09-13T11:23:21.055 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.055 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.055 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:23:21.055 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:54.302983+0000", 2024-09-13T11:23:21.055 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.2110000000000004e-06 2024-09-13T11:23:21.055 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.055 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.055 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:23:21.055 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:54.303039+0000", 2024-09-13T11:23:21.055 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.6409999999999997e-05 2024-09-13T11:23:21.055 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.056 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.056 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:23:21.056 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:54.303185+0000", 2024-09-13T11:23:21.056 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00014623500000000001 2024-09-13T11:23:21.056 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.056 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.056 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 12", 2024-09-13T11:23:21.056 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:54.303343+0000", 2024-09-13T11:23:21.056 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00015808799999999999 2024-09-13T11:23:21.056 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.056 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.056 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:23:21.057 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:54.304222+0000", 2024-09-13T11:23:21.057 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00087887900000000005 2024-09-13T11:23:21.057 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.057 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.057 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:21.057 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:54.379556+0000", 2024-09-13T11:23:21.057 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.075334071000000002 2024-09-13T11:23:21.057 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.057 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.057 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:23:21.057 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:54.379613+0000", 2024-09-13T11:23:21.057 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.6459000000000003e-05 2024-09-13T11:23:21.057 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.058 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.058 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:23:21.058 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:54.379660+0000", 2024-09-13T11:23:21.058 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.7519999999999999e-05 2024-09-13T11:23:21.058 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:21.058 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:23:21.058 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:21.058 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.058 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.058 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:10754 3.1 e48/30 3:834ae29b:::benchmark_data_smithi177_31116_object10753:head v 48'647, mlcod=48'644)", 2024-09-13T11:23:21.058 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:22:54.882468+0000", 2024-09-13T11:23:21.058 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 26.152634176999999, 2024-09-13T11:23:21.058 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.070095397000000004, 2024-09-13T11:23:21.058 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:23:21.059 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:23:21.059 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:21.059 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:23:21.059 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.059 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:23:21.059 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:54.882468+0000", 2024-09-13T11:23:21.059 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:21.059 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.059 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.059 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:23:21.059 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:54.882468+0000", 2024-09-13T11:23:21.059 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:21.059 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.060 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.060 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:23:21.060 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:54.882471+0000", 2024-09-13T11:23:21.060 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.2990000000000001e-06 2024-09-13T11:23:21.060 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.060 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.060 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:23:21.060 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:54.882523+0000", 2024-09-13T11:23:21.060 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.1520000000000001e-05 2024-09-13T11:23:21.060 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.060 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.060 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:23:21.060 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:54.951519+0000", 2024-09-13T11:23:21.060 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.068996601000000005 2024-09-13T11:23:21.061 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.061 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.061 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:23:21.061 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:54.951528+0000", 2024-09-13T11:23:21.061 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.9439999999999994e-06 2024-09-13T11:23:21.061 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.061 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.061 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:23:21.061 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:54.951577+0000", 2024-09-13T11:23:21.061 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.9070000000000003e-05 2024-09-13T11:23:21.061 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.061 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.061 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:23:21.062 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:54.951596+0000", 2024-09-13T11:23:21.062 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.863e-05 2024-09-13T11:23:21.062 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.062 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.062 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:23:21.062 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:54.952498+0000", 2024-09-13T11:23:21.062 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000902577 2024-09-13T11:23:21.062 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.062 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.062 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:23:21.062 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:54.952563+0000", 2024-09-13T11:23:21.062 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.4756000000000001e-05 2024-09-13T11:23:21.062 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:21.063 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:23:21.063 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:21.063 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.063 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.063 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:11189 3.f 3:f4bfb447:::benchmark_data_smithi177_31116_object11188:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e48)", 2024-09-13T11:23:21.063 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:22:55.281736+0000", 2024-09-13T11:23:21.063 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 25.753366308, 2024-09-13T11:23:21.063 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.12444208399999999, 2024-09-13T11:23:21.063 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:23:21.063 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:23:21.063 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:21.063 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:23:21.063 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:23:21.063 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:21.064 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 11189 2024-09-13T11:23:21.064 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.064 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:23:21.064 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.064 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:23:21.064 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:55.281736+0000", 2024-09-13T11:23:21.064 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:21.064 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.064 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.064 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:23:21.064 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:55.281736+0000", 2024-09-13T11:23:21.064 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:21.064 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.065 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.065 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:23:21.065 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:55.281738+0000", 2024-09-13T11:23:21.065 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.824e-06 2024-09-13T11:23:21.065 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.065 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.065 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:23:21.065 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:55.281795+0000", 2024-09-13T11:23:21.065 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.7058000000000002e-05 2024-09-13T11:23:21.065 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.065 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.065 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:23:21.065 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:55.281816+0000", 2024-09-13T11:23:21.066 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.0435e-05 2024-09-13T11:23:21.066 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.066 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.066 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:23:21.066 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:55.281830+0000", 2024-09-13T11:23:21.066 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3696000000000001e-05 2024-09-13T11:23:21.066 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.066 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.066 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:23:21.066 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:55.281957+0000", 2024-09-13T11:23:21.066 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00012741400000000001 2024-09-13T11:23:21.066 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.066 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.067 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:23:21.067 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:55.282237+0000", 2024-09-13T11:23:21.067 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00027982400000000002 2024-09-13T11:23:21.067 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.067 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.067 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 12", 2024-09-13T11:23:21.067 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:55.282550+0000", 2024-09-13T11:23:21.067 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00031267300000000002 2024-09-13T11:23:21.067 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.067 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.067 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:23:21.067 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:55.284360+0000", 2024-09-13T11:23:21.067 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0018107189999999999 2024-09-13T11:23:21.067 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.068 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.068 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:21.068 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:55.406042+0000", 2024-09-13T11:23:21.068 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.121681656 2024-09-13T11:23:21.068 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.068 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.068 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:23:21.068 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:55.406122+0000", 2024-09-13T11:23:21.068 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.0222000000000005e-05 2024-09-13T11:23:21.068 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.068 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.068 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:23:21.068 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:55.406178+0000", 2024-09-13T11:23:21.068 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.5563000000000003e-05 2024-09-13T11:23:21.069 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:21.069 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:23:21.069 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:21.069 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.069 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.069 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop_reply(client.4665.0:11189 3.f e48/30 ondisk, result = 0)", 2024-09-13T11:23:21.069 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:22:55.284415+0000", 2024-09-13T11:23:21.069 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 25.750687134, 2024-09-13T11:23:21.069 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.12177515, 2024-09-13T11:23:21.069 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:23:21.069 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:23:21.069 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:23:21.069 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:23:21.069 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.070 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:23:21.070 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:55.284415+0000", 2024-09-13T11:23:21.070 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:21.070 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.070 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.070 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:23:21.070 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:55.284415+0000", 2024-09-13T11:23:21.070 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:21.070 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.070 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.070 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:23:21.070 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:55.284424+0000", 2024-09-13T11:23:21.070 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.1339999999999993e-06 2024-09-13T11:23:21.070 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.071 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.071 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:23:21.071 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:55.284434+0000", 2024-09-13T11:23:21.071 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.5920000000000007e-06 2024-09-13T11:23:21.071 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.071 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.071 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:23:21.071 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:55.405920+0000", 2024-09-13T11:23:21.071 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.121486093 2024-09-13T11:23:21.071 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.071 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.071 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:23:21.072 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:55.405926+0000", 2024-09-13T11:23:21.072 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.3160000000000002e-06 2024-09-13T11:23:21.072 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.072 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.072 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:23:21.072 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:55.406012+0000", 2024-09-13T11:23:21.072 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.6236000000000006e-05 2024-09-13T11:23:21.072 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.072 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.072 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:23:21.072 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:55.406029+0000", 2024-09-13T11:23:21.072 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.7269e-05 2024-09-13T11:23:21.072 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.072 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.073 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:23:21.073 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:55.406190+0000", 2024-09-13T11:23:21.073 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00016050999999999999 2024-09-13T11:23:21.073 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:21.073 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:23:21.073 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:21.073 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.073 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.073 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:25146 3.f 3:fa7cef74:::benchmark_data_smithi177_31116_object25145:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e55)", 2024-09-13T11:23:21.073 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:04.128665+0000", 2024-09-13T11:23:21.073 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 16.906436802000002, 2024-09-13T11:23:21.073 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.060252115000000002, 2024-09-13T11:23:21.073 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:23:21.074 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:23:21.074 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:21.074 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:23:21.074 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:23:21.074 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:21.074 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 25146 2024-09-13T11:23:21.074 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.074 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:23:21.074 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.074 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:23:21.074 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.128665+0000", 2024-09-13T11:23:21.074 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:21.074 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.074 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.075 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:23:21.075 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.128665+0000", 2024-09-13T11:23:21.075 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:21.075 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.075 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.075 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:23:21.075 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.128667+0000", 2024-09-13T11:23:21.075 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.1780000000000002e-06 2024-09-13T11:23:21.075 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.075 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.075 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:23:21.075 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.128693+0000", 2024-09-13T11:23:21.076 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.5607999999999999e-05 2024-09-13T11:23:21.076 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.076 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.076 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:23:21.076 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.128707+0000", 2024-09-13T11:23:21.076 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.4491000000000001e-05 2024-09-13T11:23:21.076 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.076 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.076 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:23:21.076 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.128715+0000", 2024-09-13T11:23:21.076 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.2370000000000003e-06 2024-09-13T11:23:21.076 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.076 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.076 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:23:21.077 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.128770+0000", 2024-09-13T11:23:21.077 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.5745000000000002e-05 2024-09-13T11:23:21.077 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.077 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.077 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:23:21.077 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.128874+0000", 2024-09-13T11:23:21.077 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00010369099999999999 2024-09-13T11:23:21.077 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.077 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.077 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 12", 2024-09-13T11:23:21.077 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.128970+0000", 2024-09-13T11:23:21.077 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.6211000000000004e-05 2024-09-13T11:23:21.077 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.077 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.078 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:23:21.078 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.130169+0000", 2024-09-13T11:23:21.078 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0011986740000000001 2024-09-13T11:23:21.078 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.078 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.078 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:21.078 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.188743+0000", 2024-09-13T11:23:21.078 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.058574307999999999 2024-09-13T11:23:21.078 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.078 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.078 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:23:21.078 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.188850+0000", 2024-09-13T11:23:21.078 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000106599 2024-09-13T11:23:21.078 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.079 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.079 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:23:21.079 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.188917+0000", 2024-09-13T11:23:21.079 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.7372999999999993e-05 2024-09-13T11:23:21.079 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:21.079 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:23:21.079 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:21.079 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.079 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.079 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop_reply(client.4665.0:25146 3.f e55/30 ondisk, result = 0)", 2024-09-13T11:23:21.079 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:04.131215+0000", 2024-09-13T11:23:21.079 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 16.903886784000001, 2024-09-13T11:23:21.079 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.057715051000000003, 2024-09-13T11:23:21.079 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:23:21.080 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:23:21.080 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:23:21.080 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:23:21.080 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.080 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:23:21.080 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.131215+0000", 2024-09-13T11:23:21.080 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:21.080 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.080 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.080 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:23:21.080 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.131215+0000", 2024-09-13T11:23:21.080 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:21.080 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.081 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.081 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:23:21.081 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.131216+0000", 2024-09-13T11:23:21.081 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.19e-06 2024-09-13T11:23:21.081 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.081 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.081 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:23:21.081 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.131220+0000", 2024-09-13T11:23:21.081 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.535e-06 2024-09-13T11:23:21.081 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.081 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.081 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:23:21.081 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.188634+0000", 2024-09-13T11:23:21.082 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.057414272000000002 2024-09-13T11:23:21.082 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.082 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.082 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:23:21.082 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.188644+0000", 2024-09-13T11:23:21.082 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.5319999999999995e-06 2024-09-13T11:23:21.082 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.082 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.082 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:23:21.082 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.188723+0000", 2024-09-13T11:23:21.082 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.9264999999999994e-05 2024-09-13T11:23:21.082 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.082 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.083 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:23:21.083 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.188735+0000", 2024-09-13T11:23:21.083 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.1715999999999999e-05 2024-09-13T11:23:21.083 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.083 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.083 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:23:21.083 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.188930+0000", 2024-09-13T11:23:21.083 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000195541 2024-09-13T11:23:21.083 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:21.083 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:23:21.083 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:21.083 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.083 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.083 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:25219 3.f 3:fc69bbba:::benchmark_data_smithi177_31116_object25218:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e55)", 2024-09-13T11:23:21.084 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:04.227827+0000", 2024-09-13T11:23:21.084 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 16.807274821, 2024-09-13T11:23:21.084 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.096429679000000004, 2024-09-13T11:23:21.084 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:23:21.084 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:23:21.084 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:21.084 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:23:21.084 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:23:21.084 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:21.084 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 25219 2024-09-13T11:23:21.084 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.084 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:23:21.084 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.085 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:23:21.085 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.227827+0000", 2024-09-13T11:23:21.085 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:21.085 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.085 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.085 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:23:21.085 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.227827+0000", 2024-09-13T11:23:21.085 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:21.085 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.085 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.085 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:23:21.085 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.227829+0000", 2024-09-13T11:23:21.085 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.1399999999999998e-06 2024-09-13T11:23:21.086 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.086 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.086 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:23:21.086 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.227881+0000", 2024-09-13T11:23:21.086 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.1575e-05 2024-09-13T11:23:21.086 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.086 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.086 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:23:21.086 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.227892+0000", 2024-09-13T11:23:21.086 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0847e-05 2024-09-13T11:23:21.086 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.086 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.086 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:23:21.087 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.227899+0000", 2024-09-13T11:23:21.087 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.5170000000000002e-06 2024-09-13T11:23:21.087 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.087 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.087 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:23:21.087 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.227974+0000", 2024-09-13T11:23:21.087 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.5105999999999995e-05 2024-09-13T11:23:21.087 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.087 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.087 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:23:21.087 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.228144+0000", 2024-09-13T11:23:21.087 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00016960200000000001 2024-09-13T11:23:21.087 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.088 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.088 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 12", 2024-09-13T11:23:21.088 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.228297+0000", 2024-09-13T11:23:21.088 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000153029 2024-09-13T11:23:21.088 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.088 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.088 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:23:21.088 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.230747+0000", 2024-09-13T11:23:21.088 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0024503350000000001 2024-09-13T11:23:21.088 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.088 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.088 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:21.088 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.324031+0000", 2024-09-13T11:23:21.089 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.093283378 2024-09-13T11:23:21.089 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.089 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.089 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:23:21.089 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.324162+0000", 2024-09-13T11:23:21.089 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00013150299999999999 2024-09-13T11:23:21.089 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.089 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.089 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:23:21.089 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.324257+0000", 2024-09-13T11:23:21.089 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.4647000000000004e-05 2024-09-13T11:23:21.089 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:21.089 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:23:21.089 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:21.090 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.090 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.090 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop_reply(client.4665.0:25219 3.f e55/30 ondisk, result = 0)", 2024-09-13T11:23:21.090 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:04.231094+0000", 2024-09-13T11:23:21.090 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 16.804007601999999, 2024-09-13T11:23:21.090 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.093178698000000004, 2024-09-13T11:23:21.090 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:23:21.090 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:23:21.090 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:23:21.090 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:23:21.090 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.090 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:23:21.090 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.231094+0000", 2024-09-13T11:23:21.090 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:21.091 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.091 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.091 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:23:21.091 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.231094+0000", 2024-09-13T11:23:21.091 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:21.091 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.091 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.091 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:23:21.091 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.231096+0000", 2024-09-13T11:23:21.091 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5659999999999999e-06 2024-09-13T11:23:21.091 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.091 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.091 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:23:21.091 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.231101+0000", 2024-09-13T11:23:21.092 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.536e-06 2024-09-13T11:23:21.092 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.092 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.092 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:23:21.092 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.323914+0000", 2024-09-13T11:23:21.092 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.092812243000000003 2024-09-13T11:23:21.092 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.092 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.092 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:23:21.092 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.323924+0000", 2024-09-13T11:23:21.092 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0665999999999999e-05 2024-09-13T11:23:21.092 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.092 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.093 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:23:21.093 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.324014+0000", 2024-09-13T11:23:21.093 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.9332000000000005e-05 2024-09-13T11:23:21.093 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.093 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.093 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:23:21.093 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.324024+0000", 2024-09-13T11:23:21.093 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0188e-05 2024-09-13T11:23:21.093 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.093 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.093 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:23:21.093 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.324273+0000", 2024-09-13T11:23:21.093 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00024916700000000001 2024-09-13T11:23:21.093 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:21.094 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:23:21.094 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:21.094 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.094 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.094 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:25309 3.f 3:f08e6eac:::benchmark_data_smithi177_31116_object25308:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e55)", 2024-09-13T11:23:21.094 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:04.335206+0000", 2024-09-13T11:23:21.094 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 16.699895515000001, 2024-09-13T11:23:21.094 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.059217060000000002, 2024-09-13T11:23:21.094 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:23:21.094 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:23:21.094 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:21.094 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:23:21.094 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:23:21.095 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:21.095 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 25309 2024-09-13T11:23:21.095 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.095 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:23:21.095 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.095 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:23:21.095 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.335206+0000", 2024-09-13T11:23:21.095 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:21.095 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.095 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.095 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:23:21.095 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.335206+0000", 2024-09-13T11:23:21.095 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:21.096 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.096 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.096 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:23:21.096 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.335209+0000", 2024-09-13T11:23:21.096 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.7949999999999999e-06 2024-09-13T11:23:21.096 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.096 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.096 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:23:21.096 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.335264+0000", 2024-09-13T11:23:21.096 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.4772999999999999e-05 2024-09-13T11:23:21.096 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.096 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.096 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:23:21.096 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.335285+0000", 2024-09-13T11:23:21.097 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.1305999999999999e-05 2024-09-13T11:23:21.097 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.097 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.097 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:23:21.097 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.335301+0000", 2024-09-13T11:23:21.097 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5378000000000001e-05 2024-09-13T11:23:21.097 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.097 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.097 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:23:21.097 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.335372+0000", 2024-09-13T11:23:21.097 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.1660000000000002e-05 2024-09-13T11:23:21.097 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.097 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.098 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:23:21.098 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.335464+0000", 2024-09-13T11:23:21.098 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.2151000000000006e-05 2024-09-13T11:23:21.098 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.098 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.098 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 12", 2024-09-13T11:23:21.098 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.335555+0000", 2024-09-13T11:23:21.098 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.0730999999999996e-05 2024-09-13T11:23:21.098 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.098 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.098 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:23:21.098 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.336987+0000", 2024-09-13T11:23:21.099 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0014320470000000001 2024-09-13T11:23:21.099 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.099 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.099 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:21.099 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.394348+0000", 2024-09-13T11:23:21.099 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.057360577000000003 2024-09-13T11:23:21.099 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.099 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.099 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:23:21.099 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.394389+0000", 2024-09-13T11:23:21.099 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.1402000000000001e-05 2024-09-13T11:23:21.099 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.099 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.100 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:23:21.100 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.394423+0000", 2024-09-13T11:23:21.100 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.4239999999999997e-05 2024-09-13T11:23:21.100 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:21.100 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:23:21.100 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:21.100 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.100 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.100 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:30815 3.f 3:fde17cb1:::benchmark_data_smithi177_31116_object30814:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e59)", 2024-09-13T11:23:21.100 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:09.495035+0000", 2024-09-13T11:23:21.100 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 11.540066875000001, 2024-09-13T11:23:21.101 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.229307711, 2024-09-13T11:23:21.101 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:23:21.101 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:23:21.101 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:21.101 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:23:21.101 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:23:21.101 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:21.101 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 30815 2024-09-13T11:23:21.101 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.101 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:23:21.101 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.101 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:23:21.101 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.495035+0000", 2024-09-13T11:23:21.102 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:21.102 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.102 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.102 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:23:21.102 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.495035+0000", 2024-09-13T11:23:21.102 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:21.102 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.102 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.102 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:23:21.102 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.495037+0000", 2024-09-13T11:23:21.102 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.871e-06 2024-09-13T11:23:21.102 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.102 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.103 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:23:21.103 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.495062+0000", 2024-09-13T11:23:21.103 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.5164000000000002e-05 2024-09-13T11:23:21.103 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.103 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.103 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:23:21.103 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.495075+0000", 2024-09-13T11:23:21.103 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3369e-05 2024-09-13T11:23:21.103 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.103 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.103 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:23:21.103 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.495084+0000", 2024-09-13T11:23:21.104 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.3510000000000002e-06 2024-09-13T11:23:21.104 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.104 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.104 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:23:21.104 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.495346+0000", 2024-09-13T11:23:21.104 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00026264000000000002 2024-09-13T11:23:21.104 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.104 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.104 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:23:21.104 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.495523+0000", 2024-09-13T11:23:21.104 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00017640000000000001 2024-09-13T11:23:21.104 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.105 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.105 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 12", 2024-09-13T11:23:21.105 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.495699+0000", 2024-09-13T11:23:21.105 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00017653600000000001 2024-09-13T11:23:21.105 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.105 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.105 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:23:21.105 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.497040+0000", 2024-09-13T11:23:21.105 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001340473 2024-09-13T11:23:21.105 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.105 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.105 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:21.106 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.724049+0000", 2024-09-13T11:23:21.106 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.22700931799999999 2024-09-13T11:23:21.106 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.106 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.106 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:23:21.106 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.724215+0000", 2024-09-13T11:23:21.106 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000165988 2024-09-13T11:23:21.106 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.106 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.106 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:23:21.106 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.724343+0000", 2024-09-13T11:23:21.106 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000127601 2024-09-13T11:23:21.107 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:21.107 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:23:21.107 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:21.107 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.107 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.107 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:30828 3.f 3:f3a9c6a9:::benchmark_data_smithi177_31116_object30827:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e59)", 2024-09-13T11:23:21.107 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:09.503661+0000", 2024-09-13T11:23:21.107 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 11.531440691, 2024-09-13T11:23:21.107 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.26477667100000002, 2024-09-13T11:23:21.107 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:23:21.107 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:23:21.107 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:21.107 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:23:21.107 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:23:21.108 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:21.108 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 30828 2024-09-13T11:23:21.108 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.108 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:23:21.108 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.108 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:23:21.108 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.503661+0000", 2024-09-13T11:23:21.108 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:21.108 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.108 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.108 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:23:21.108 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.503661+0000", 2024-09-13T11:23:21.108 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:21.109 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.109 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.109 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:23:21.109 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.503663+0000", 2024-09-13T11:23:21.109 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.968e-06 2024-09-13T11:23:21.109 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.109 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.109 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:23:21.109 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.503687+0000", 2024-09-13T11:23:21.109 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.3983999999999999e-05 2024-09-13T11:23:21.109 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.109 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.109 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:23:21.109 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.503701+0000", 2024-09-13T11:23:21.110 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3416e-05 2024-09-13T11:23:21.110 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.110 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.110 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:23:21.110 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.503708+0000", 2024-09-13T11:23:21.110 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.9530000000000002e-06 2024-09-13T11:23:21.110 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.110 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.110 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:23:21.110 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.503759+0000", 2024-09-13T11:23:21.110 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.1075999999999997e-05 2024-09-13T11:23:21.110 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.110 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.111 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:23:21.111 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.503861+0000", 2024-09-13T11:23:21.111 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000102372 2024-09-13T11:23:21.111 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.111 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.111 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 12", 2024-09-13T11:23:21.111 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.503952+0000", 2024-09-13T11:23:21.111 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.1135000000000001e-05 2024-09-13T11:23:21.111 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.111 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.111 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:23:21.111 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.505340+0000", 2024-09-13T11:23:21.111 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0013877169999999999 2024-09-13T11:23:21.112 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.112 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.112 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:21.112 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.768281+0000", 2024-09-13T11:23:21.112 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.26294092400000002 2024-09-13T11:23:21.112 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.112 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.112 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:23:21.112 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.768369+0000", 2024-09-13T11:23:21.112 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.7818999999999994e-05 2024-09-13T11:23:21.112 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.112 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.112 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:23:21.113 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.768438+0000", 2024-09-13T11:23:21.113 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.9307000000000006e-05 2024-09-13T11:23:21.113 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:21.113 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:23:21.113 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:21.113 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.113 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.113 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:30864 3.f 3:f839a50b:::benchmark_data_smithi177_31116_object30863:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e59)", 2024-09-13T11:23:21.113 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:09.720850+0000", 2024-09-13T11:23:21.113 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 11.314252035999999, 2024-09-13T11:23:21.113 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.056434030000000003, 2024-09-13T11:23:21.113 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:23:21.113 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:23:21.113 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:21.114 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:23:21.114 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:23:21.114 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:21.114 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 30864 2024-09-13T11:23:21.114 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.114 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:23:21.114 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.114 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:23:21.114 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.720850+0000", 2024-09-13T11:23:21.114 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:21.114 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.114 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.114 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:23:21.115 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.720850+0000", 2024-09-13T11:23:21.115 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:21.115 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.115 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.115 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:23:21.115 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.720854+0000", 2024-09-13T11:23:21.115 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.5519999999999998e-06 2024-09-13T11:23:21.115 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.115 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.115 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:23:21.115 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.720906+0000", 2024-09-13T11:23:21.115 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.1955e-05 2024-09-13T11:23:21.115 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.116 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.116 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:23:21.116 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.720940+0000", 2024-09-13T11:23:21.116 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.3766999999999997e-05 2024-09-13T11:23:21.116 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.116 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.116 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:23:21.116 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.720957+0000", 2024-09-13T11:23:21.116 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.6498999999999999e-05 2024-09-13T11:23:21.116 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.116 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.116 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:23:21.117 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.721096+0000", 2024-09-13T11:23:21.117 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00013951400000000001 2024-09-13T11:23:21.117 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.117 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.117 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:23:21.117 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.721335+0000", 2024-09-13T11:23:21.117 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00023907 2024-09-13T11:23:21.117 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.117 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.117 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 12", 2024-09-13T11:23:21.117 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.721567+0000", 2024-09-13T11:23:21.117 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00023199900000000001 2024-09-13T11:23:21.117 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.118 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.118 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:23:21.118 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.724625+0000", 2024-09-13T11:23:21.118 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.003057597 2024-09-13T11:23:21.118 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.118 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.118 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:21.118 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.777073+0000", 2024-09-13T11:23:21.118 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.052447763000000001 2024-09-13T11:23:21.118 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.118 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.118 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:23:21.118 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.777158+0000", 2024-09-13T11:23:21.118 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.4932000000000006e-05 2024-09-13T11:23:21.119 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.119 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.119 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:23:21.119 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.777284+0000", 2024-09-13T11:23:21.119 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000126382 2024-09-13T11:23:21.119 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:21.119 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:23:21.119 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:21.119 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.119 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.119 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:30976 3.f 3:f39fdccb:::benchmark_data_smithi177_31116_object30975:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e59)", 2024-09-13T11:23:21.119 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:09.826371+0000", 2024-09-13T11:23:21.119 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 11.208731350000001, 2024-09-13T11:23:21.120 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.073771945000000005, 2024-09-13T11:23:21.120 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:23:21.120 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:23:21.120 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:21.120 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:23:21.120 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:23:21.120 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:21.120 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 30976 2024-09-13T11:23:21.120 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.120 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:23:21.120 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.120 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:23:21.120 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.826371+0000", 2024-09-13T11:23:21.121 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:21.121 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.121 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.121 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:23:21.121 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.826371+0000", 2024-09-13T11:23:21.121 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:21.121 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.121 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.121 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:23:21.121 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.826375+0000", 2024-09-13T11:23:21.121 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.296e-06 2024-09-13T11:23:21.121 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.121 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.121 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:23:21.122 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.826422+0000", 2024-09-13T11:23:21.122 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.6968999999999997e-05 2024-09-13T11:23:21.122 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.122 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.122 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:23:21.122 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.826452+0000", 2024-09-13T11:23:21.122 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.9791e-05 2024-09-13T11:23:21.122 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.122 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.122 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:23:21.122 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.826474+0000", 2024-09-13T11:23:21.122 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.2257999999999999e-05 2024-09-13T11:23:21.122 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.123 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.123 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:23:21.123 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.826627+0000", 2024-09-13T11:23:21.123 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00015291299999999999 2024-09-13T11:23:21.123 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.123 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.123 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:23:21.123 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.826984+0000", 2024-09-13T11:23:21.123 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00035701800000000002 2024-09-13T11:23:21.124 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.124 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.124 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 12", 2024-09-13T11:23:21.124 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.827330+0000", 2024-09-13T11:23:21.124 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00034660700000000002 2024-09-13T11:23:21.124 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.124 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.124 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:23:21.124 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.832085+0000", 2024-09-13T11:23:21.124 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0047545340000000004 2024-09-13T11:23:21.124 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.124 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.125 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:21.125 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.899871+0000", 2024-09-13T11:23:21.125 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.067785709999999999 2024-09-13T11:23:21.125 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.125 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.125 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:23:21.125 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.900018+0000", 2024-09-13T11:23:21.125 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000147403 2024-09-13T11:23:21.125 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.125 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.125 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:23:21.125 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.900143+0000", 2024-09-13T11:23:21.125 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00012444600000000001 2024-09-13T11:23:21.126 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:21.126 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:23:21.126 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:21.126 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.126 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.126 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:30979 3.f 3:fe31d07f:::benchmark_data_smithi177_31116_object30978:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e59)", 2024-09-13T11:23:21.126 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:09.828735+0000", 2024-09-13T11:23:21.126 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 11.206366647999999, 2024-09-13T11:23:21.126 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.093285357999999999, 2024-09-13T11:23:21.126 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:23:21.126 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:23:21.126 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:21.127 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:23:21.127 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:23:21.127 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:21.127 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 30979 2024-09-13T11:23:21.127 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.127 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:23:21.127 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.127 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:23:21.127 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.828735+0000", 2024-09-13T11:23:21.127 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:21.127 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.127 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.127 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:23:21.127 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.828735+0000", 2024-09-13T11:23:21.128 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:21.128 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.128 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.128 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:23:21.128 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.828740+0000", 2024-09-13T11:23:21.128 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.5589999999999999e-06 2024-09-13T11:23:21.128 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.128 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.128 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:23:21.128 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.828802+0000", 2024-09-13T11:23:21.128 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.2626999999999995e-05 2024-09-13T11:23:21.128 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.128 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.129 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:23:21.129 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.828839+0000", 2024-09-13T11:23:21.129 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.6233999999999999e-05 2024-09-13T11:23:21.129 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.129 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.129 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:23:21.129 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.828866+0000", 2024-09-13T11:23:21.129 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.7353999999999999e-05 2024-09-13T11:23:21.129 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.129 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.129 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:23:21.129 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.829778+0000", 2024-09-13T11:23:21.129 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00091205799999999999 2024-09-13T11:23:21.130 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.130 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.130 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:23:21.130 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.830009+0000", 2024-09-13T11:23:21.130 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00023056699999999999 2024-09-13T11:23:21.130 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.130 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.130 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 12", 2024-09-13T11:23:21.130 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.830424+0000", 2024-09-13T11:23:21.130 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00041555500000000002 2024-09-13T11:23:21.130 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.130 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.130 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:23:21.130 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.835151+0000", 2024-09-13T11:23:21.131 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0047271370000000002 2024-09-13T11:23:21.131 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.131 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.131 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:21.131 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.921784+0000", 2024-09-13T11:23:21.131 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.086632434999999994 2024-09-13T11:23:21.131 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.131 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.131 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:23:21.131 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.921941+0000", 2024-09-13T11:23:21.131 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000157479 2024-09-13T11:23:21.131 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.131 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.132 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:23:21.132 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.922021+0000", 2024-09-13T11:23:21.132 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.9352999999999997e-05 2024-09-13T11:23:21.132 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:21.132 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:23:21.132 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:21.132 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.132 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.132 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:31045 3.1 e59/30 3:856c2a29:::benchmark_data_smithi177_31116_object31044:head v 59'1923, mlcod=59'1921)", 2024-09-13T11:23:21.132 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:09.893626+0000", 2024-09-13T11:23:21.132 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 11.141475762000001, 2024-09-13T11:23:21.132 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.068932729999999998, 2024-09-13T11:23:21.132 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:23:21.132 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:23:21.133 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:21.133 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:23:21.133 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.133 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:23:21.133 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.893626+0000", 2024-09-13T11:23:21.133 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:21.133 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.133 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.133 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:23:21.133 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.893626+0000", 2024-09-13T11:23:21.133 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:21.133 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.133 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.133 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:23:21.134 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.893631+0000", 2024-09-13T11:23:21.134 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.3989999999999997e-06 2024-09-13T11:23:21.134 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.134 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.134 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:23:21.134 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.893698+0000", 2024-09-13T11:23:21.134 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.7606000000000003e-05 2024-09-13T11:23:21.134 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.134 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.134 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:23:21.134 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.959845+0000", 2024-09-13T11:23:21.134 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.066147062000000006 2024-09-13T11:23:21.134 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.135 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.135 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:23:21.135 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.959863+0000", 2024-09-13T11:23:21.135 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.7717999999999999e-05 2024-09-13T11:23:21.135 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.135 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.135 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:23:21.135 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.959969+0000", 2024-09-13T11:23:21.135 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000106536 2024-09-13T11:23:21.135 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.135 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.135 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:23:21.135 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.959996+0000", 2024-09-13T11:23:21.135 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.6939000000000001e-05 2024-09-13T11:23:21.136 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.136 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.136 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:23:21.136 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.962507+0000", 2024-09-13T11:23:21.136 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.002510675 2024-09-13T11:23:21.136 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.136 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.136 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:23:21.136 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.962559+0000", 2024-09-13T11:23:21.136 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.1795000000000001e-05 2024-09-13T11:23:21.136 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:21.136 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:23:21.136 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:21.137 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.137 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.137 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:31065 3.f 3:f622697a:::benchmark_data_smithi177_31116_object31064:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e59)", 2024-09-13T11:23:21.137 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:09.918572+0000", 2024-09-13T11:23:21.137 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 11.116530044999999, 2024-09-13T11:23:21.137 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.080739476000000004, 2024-09-13T11:23:21.137 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:23:21.137 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:23:21.137 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:21.137 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:23:21.137 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:23:21.137 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:21.137 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 31065 2024-09-13T11:23:21.137 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.138 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:23:21.138 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.138 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:23:21.138 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.918572+0000", 2024-09-13T11:23:21.138 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:21.138 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.138 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.138 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:23:21.138 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.918572+0000", 2024-09-13T11:23:21.138 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:21.138 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.138 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.138 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:23:21.139 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.918575+0000", 2024-09-13T11:23:21.139 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.1590000000000002e-06 2024-09-13T11:23:21.139 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.139 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.139 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:23:21.139 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.918623+0000", 2024-09-13T11:23:21.139 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.7945999999999998e-05 2024-09-13T11:23:21.139 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.139 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.139 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:23:21.139 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.918652+0000", 2024-09-13T11:23:21.139 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.9493999999999999e-05 2024-09-13T11:23:21.139 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.139 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.139 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:23:21.140 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.918668+0000", 2024-09-13T11:23:21.140 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5424999999999999e-05 2024-09-13T11:23:21.140 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.140 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.140 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:23:21.140 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.919719+0000", 2024-09-13T11:23:21.140 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0010511749999999999 2024-09-13T11:23:21.140 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.140 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.140 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:23:21.140 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.919990+0000", 2024-09-13T11:23:21.140 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00027083400000000002 2024-09-13T11:23:21.140 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.141 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.141 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 12", 2024-09-13T11:23:21.141 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.920300+0000", 2024-09-13T11:23:21.141 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00031055500000000001 2024-09-13T11:23:21.141 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.141 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.141 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:23:21.141 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.922791+0000", 2024-09-13T11:23:21.141 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0024907919999999999 2024-09-13T11:23:21.141 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.141 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.141 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:21.141 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.999173+0000", 2024-09-13T11:23:21.142 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.076381334999999995 2024-09-13T11:23:21.142 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.142 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.142 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:23:21.142 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.999248+0000", 2024-09-13T11:23:21.142 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.5727000000000002e-05 2024-09-13T11:23:21.142 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.142 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.142 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:23:21.142 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.999311+0000", 2024-09-13T11:23:21.142 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.3034000000000007e-05 2024-09-13T11:23:21.142 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:21.142 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:23:21.143 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:21.143 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.143 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.143 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:31074 3.f 3:f7a42e3b:::benchmark_data_smithi177_31116_object31073:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e59)", 2024-09-13T11:23:21.143 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:09.928503+0000", 2024-09-13T11:23:21.143 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 11.106598891999999, 2024-09-13T11:23:21.143 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.080440859000000003, 2024-09-13T11:23:21.143 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:23:21.143 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:23:21.143 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:21.143 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:23:21.143 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:23:21.143 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:21.143 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 31074 2024-09-13T11:23:21.144 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.144 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:23:21.144 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.144 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:23:21.144 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.928503+0000", 2024-09-13T11:23:21.144 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:21.144 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.144 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.144 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:23:21.144 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.928503+0000", 2024-09-13T11:23:21.144 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:21.144 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.144 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.145 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:23:21.145 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.928507+0000", 2024-09-13T11:23:21.145 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.6169999999999998e-06 2024-09-13T11:23:21.145 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.145 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.145 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:23:21.145 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.928574+0000", 2024-09-13T11:23:21.145 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.6884999999999994e-05 2024-09-13T11:23:21.145 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.145 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.145 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:23:21.145 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.928600+0000", 2024-09-13T11:23:21.145 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.6789000000000001e-05 2024-09-13T11:23:21.145 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.146 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.146 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:23:21.146 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.928615+0000", 2024-09-13T11:23:21.146 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.4430000000000001e-05 2024-09-13T11:23:21.146 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.146 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.146 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:23:21.146 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.928726+0000", 2024-09-13T11:23:21.146 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000110794 2024-09-13T11:23:21.146 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.146 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.146 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:23:21.146 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.928893+0000", 2024-09-13T11:23:21.147 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00016779800000000001 2024-09-13T11:23:21.147 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.147 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.147 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 12", 2024-09-13T11:23:21.147 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.929052+0000", 2024-09-13T11:23:21.147 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000159037 2024-09-13T11:23:21.147 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.147 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.147 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:23:21.147 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.931260+0000", 2024-09-13T11:23:21.147 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.002208041 2024-09-13T11:23:21.147 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.147 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.147 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:21.147 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:10.008837+0000", 2024-09-13T11:23:21.148 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.077576459 2024-09-13T11:23:21.148 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.148 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.148 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:23:21.148 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:10.008903+0000", 2024-09-13T11:23:21.148 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.6093999999999994e-05 2024-09-13T11:23:21.148 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.148 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.148 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:23:21.148 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:10.008944+0000", 2024-09-13T11:23:21.148 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.0914999999999997e-05 2024-09-13T11:23:21.149 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:21.149 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:23:21.149 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:21.149 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:21.149 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:23:21.149 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-09-13T11:23:21.151 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_ops_in_flight 2024-09-13T11:23:21.151 INFO:tasks.ceph.osd.3.smithi177.stderr:2024-09-13T11:23:21.145+0000 7f10adba4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26574) UID: 0 2024-09-13T11:23:21.252 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-09-13T11:23:21.246+0000 7f97befc9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26315) UID: 0 2024-09-13T11:23:21.313 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-09-13T11:23:21.314 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [], 2024-09-13T11:23:21.314 INFO:teuthology.orchestra.run.smithi177.stdout: "num_ops": 0 2024-09-13T11:23:21.314 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-09-13T11:23:21.328 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_blocked_ops 2024-09-13T11:23:21.352 INFO:tasks.ceph.osd.12.smithi001.stderr:2024-09-13T11:23:21.348+0000 7f892c6e6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26314) UID: 0 2024-09-13T11:23:21.438 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-09-13T11:23:21.438 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [], 2024-09-13T11:23:21.439 INFO:teuthology.orchestra.run.smithi177.stdout: "complaint_time": 30, 2024-09-13T11:23:21.439 INFO:teuthology.orchestra.run.smithi177.stdout: "num_blocked_ops": 0 2024-09-13T11:23:21.439 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-09-13T11:23:21.451 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_historic_ops 2024-09-13T11:23:21.453 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-09-13T11:23:21.446+0000 7f97befc9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26315) UID: 0 2024-09-13T11:23:21.553 INFO:tasks.ceph.osd.4.smithi001.stderr:2024-09-13T11:23:21.548+0000 7ff8e0218640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26310) UID: 0 2024-09-13T11:23:21.574 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-09-13T11:23:21.574 INFO:teuthology.orchestra.run.smithi177.stdout: "size": 20, 2024-09-13T11:23:21.574 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 600, 2024-09-13T11:23:21.574 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [ 2024-09-13T11:23:21.574 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.574 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:1045 3.5 3:aca982b9:::benchmark_data_smithi177_31116_object1044:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-09-13T11:23:21.574 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:22:49.650700+0000", 2024-09-13T11:23:21.574 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 31.920022528000001, 2024-09-13T11:23:21.574 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.055297286000000001, 2024-09-13T11:23:21.574 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:23:21.574 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:23:21.575 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:21.575 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:23:21.575 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:23:21.575 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:21.575 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 1045 2024-09-13T11:23:21.575 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.575 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:23:21.575 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.575 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:23:21.575 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.650700+0000", 2024-09-13T11:23:21.575 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:21.575 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.576 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.576 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:23:21.576 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.650700+0000", 2024-09-13T11:23:21.576 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:21.576 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.576 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.576 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:23:21.576 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.650702+0000", 2024-09-13T11:23:21.576 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.127e-06 2024-09-13T11:23:21.576 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.576 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.577 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:23:21.577 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.650746+0000", 2024-09-13T11:23:21.577 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.4097000000000001e-05 2024-09-13T11:23:21.577 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.577 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.577 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:23:21.577 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.650755+0000", 2024-09-13T11:23:21.578 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.0310000000000004e-06 2024-09-13T11:23:21.578 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.578 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.578 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:23:21.578 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.650760+0000", 2024-09-13T11:23:21.578 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.3929999999999999e-06 2024-09-13T11:23:21.578 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.579 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.579 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:23:21.579 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.651076+0000", 2024-09-13T11:23:21.579 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00031611900000000001 2024-09-13T11:23:21.579 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.580 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.580 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:23:21.580 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.651182+0000", 2024-09-13T11:23:21.580 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00010571199999999999 2024-09-13T11:23:21.580 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.580 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.580 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 13", 2024-09-13T11:23:21.581 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.651294+0000", 2024-09-13T11:23:21.581 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000112112 2024-09-13T11:23:21.581 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.581 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.581 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:23:21.581 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.654078+0000", 2024-09-13T11:23:21.581 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.002783832 2024-09-13T11:23:21.582 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.582 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.582 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:21.582 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.705758+0000", 2024-09-13T11:23:21.582 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.051680333000000002 2024-09-13T11:23:21.582 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.583 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.583 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:23:21.583 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.705896+0000", 2024-09-13T11:23:21.583 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00013765000000000001 2024-09-13T11:23:21.583 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.583 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.584 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:23:21.584 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.705998+0000", 2024-09-13T11:23:21.584 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00010187999999999999 2024-09-13T11:23:21.584 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:21.584 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:23:21.584 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:21.585 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.585 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.585 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop_reply(client.4665.0:1045 3.5 e43/30 ondisk, result = 0)", 2024-09-13T11:23:21.585 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:22:49.652166+0000", 2024-09-13T11:23:21.585 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 31.91855662, 2024-09-13T11:23:21.585 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.053848380000000001, 2024-09-13T11:23:21.586 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:23:21.586 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:23:21.586 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:23:21.586 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:23:21.586 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.586 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:23:21.586 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.652166+0000", 2024-09-13T11:23:21.587 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:21.587 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.587 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.587 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:23:21.587 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.652166+0000", 2024-09-13T11:23:21.587 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:21.587 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.588 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.588 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:23:21.588 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.652168+0000", 2024-09-13T11:23:21.588 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.393e-06 2024-09-13T11:23:21.588 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.588 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.588 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:23:21.589 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.652172+0000", 2024-09-13T11:23:21.589 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.2989999999999998e-06 2024-09-13T11:23:21.589 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.589 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.589 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:23:21.589 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.705604+0000", 2024-09-13T11:23:21.590 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.053431575000000002 2024-09-13T11:23:21.590 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.590 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.590 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:23:21.590 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.705623+0000", 2024-09-13T11:23:21.590 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.9479000000000001e-05 2024-09-13T11:23:21.591 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.591 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.591 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:23:21.591 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.705728+0000", 2024-09-13T11:23:21.591 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000105338 2024-09-13T11:23:21.591 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.591 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.592 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:23:21.592 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.705746+0000", 2024-09-13T11:23:21.592 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.7645000000000001e-05 2024-09-13T11:23:21.592 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.592 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.592 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:23:21.592 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.706015+0000", 2024-09-13T11:23:21.592 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00026865100000000001 2024-09-13T11:23:21.592 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:21.592 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:23:21.592 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:21.592 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.593 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.593 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:9510 3.5 3:a7e302ef:::benchmark_data_smithi177_31116_object9509:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-09-13T11:23:21.593 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:22:54.044535+0000", 2024-09-13T11:23:21.593 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 27.526188252000001, 2024-09-13T11:23:21.593 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.049023708999999999, 2024-09-13T11:23:21.593 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:23:21.593 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:23:21.593 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:21.593 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:23:21.593 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:23:21.593 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:21.594 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 9510 2024-09-13T11:23:21.594 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.594 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:23:21.594 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.594 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:23:21.594 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:54.044535+0000", 2024-09-13T11:23:21.594 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:21.594 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.594 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.594 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:23:21.594 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:54.044535+0000", 2024-09-13T11:23:21.595 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:21.595 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.595 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.595 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:23:21.595 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:54.044539+0000", 2024-09-13T11:23:21.595 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.0219999999999998e-06 2024-09-13T11:23:21.595 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.595 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.595 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:23:21.595 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:54.044570+0000", 2024-09-13T11:23:21.595 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.0914999999999998e-05 2024-09-13T11:23:21.595 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.595 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.596 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:23:21.596 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:54.044593+0000", 2024-09-13T11:23:21.596 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.3584999999999999e-05 2024-09-13T11:23:21.596 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.596 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.596 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:23:21.596 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:54.044605+0000", 2024-09-13T11:23:21.596 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.1501000000000001e-05 2024-09-13T11:23:21.596 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.596 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.596 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:23:21.597 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:54.044681+0000", 2024-09-13T11:23:21.597 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.5894000000000002e-05 2024-09-13T11:23:21.597 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.597 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.597 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:23:21.597 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:54.044820+0000", 2024-09-13T11:23:21.597 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00013934700000000001 2024-09-13T11:23:21.597 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.597 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.598 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 13", 2024-09-13T11:23:21.598 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:54.044938+0000", 2024-09-13T11:23:21.598 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000118377 2024-09-13T11:23:21.598 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.598 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.598 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:23:21.598 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:54.046044+0000", 2024-09-13T11:23:21.598 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001105195 2024-09-13T11:23:21.598 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.598 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.598 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:21.599 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:54.093490+0000", 2024-09-13T11:23:21.599 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.047446691999999999 2024-09-13T11:23:21.599 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.599 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.599 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:23:21.599 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:54.093525+0000", 2024-09-13T11:23:21.599 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.4422999999999999e-05 2024-09-13T11:23:21.599 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.599 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.599 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:23:21.599 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:54.093558+0000", 2024-09-13T11:23:21.600 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.3757999999999998e-05 2024-09-13T11:23:21.600 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:21.600 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:23:21.600 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:21.600 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.600 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.600 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop_reply(client.4665.0:9510 3.5 e47/30 ondisk, result = 0)", 2024-09-13T11:23:21.600 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:22:54.047246+0000", 2024-09-13T11:23:21.600 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 27.523477271000001, 2024-09-13T11:23:21.600 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.046320735000000002, 2024-09-13T11:23:21.600 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:23:21.600 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:23:21.601 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:23:21.601 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:23:21.601 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.601 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:23:21.601 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:54.047246+0000", 2024-09-13T11:23:21.601 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:21.601 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.601 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.601 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:23:21.601 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:54.047246+0000", 2024-09-13T11:23:21.601 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:21.602 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.602 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.602 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:23:21.602 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:54.047247+0000", 2024-09-13T11:23:21.602 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5829999999999999e-06 2024-09-13T11:23:21.602 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.602 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.602 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:23:21.602 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:54.047252+0000", 2024-09-13T11:23:21.602 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.7319999999999998e-06 2024-09-13T11:23:21.603 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.603 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.603 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:23:21.603 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:54.093433+0000", 2024-09-13T11:23:21.603 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.046181415000000003 2024-09-13T11:23:21.603 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.603 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.603 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:23:21.603 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:54.093440+0000", 2024-09-13T11:23:21.603 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.0029999999999997e-06 2024-09-13T11:23:21.603 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.604 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.604 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:23:21.604 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:54.093481+0000", 2024-09-13T11:23:21.604 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.0531999999999998e-05 2024-09-13T11:23:21.604 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.604 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.604 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:23:21.604 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:54.093486+0000", 2024-09-13T11:23:21.604 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.2700000000000004e-06 2024-09-13T11:23:21.604 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.604 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.605 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:23:21.605 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:54.093566+0000", 2024-09-13T11:23:21.605 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.0199999999999998e-05 2024-09-13T11:23:21.605 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:21.605 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:23:21.605 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:21.605 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.605 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.605 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:25138 3.5 3:ab2ac965:::benchmark_data_smithi177_31116_object25137:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e55)", 2024-09-13T11:23:21.605 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:04.117349+0000", 2024-09-13T11:23:21.605 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 17.453374135000001, 2024-09-13T11:23:21.605 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.050459565999999997, 2024-09-13T11:23:21.606 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:23:21.606 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:23:21.606 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:21.606 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:23:21.606 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:23:21.606 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:21.606 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 25138 2024-09-13T11:23:21.606 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.606 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:23:21.606 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.606 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:23:21.606 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.117349+0000", 2024-09-13T11:23:21.606 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:21.607 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.607 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.607 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:23:21.607 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.117349+0000", 2024-09-13T11:23:21.607 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:21.607 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.607 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.607 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:23:21.608 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.117350+0000", 2024-09-13T11:23:21.608 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5230000000000001e-06 2024-09-13T11:23:21.608 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.608 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.608 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:23:21.608 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.117384+0000", 2024-09-13T11:23:21.608 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.3711000000000003e-05 2024-09-13T11:23:21.608 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.608 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.608 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:23:21.609 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.117397+0000", 2024-09-13T11:23:21.609 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3274e-05 2024-09-13T11:23:21.609 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.609 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.609 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:23:21.609 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.117404+0000", 2024-09-13T11:23:21.609 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.9190000000000002e-06 2024-09-13T11:23:21.609 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.609 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.609 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:23:21.609 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.117443+0000", 2024-09-13T11:23:21.609 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.8733000000000003e-05 2024-09-13T11:23:21.610 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.610 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.610 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:23:21.610 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.117534+0000", 2024-09-13T11:23:21.610 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.0526999999999996e-05 2024-09-13T11:23:21.610 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.610 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.610 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 13", 2024-09-13T11:23:21.610 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.117618+0000", 2024-09-13T11:23:21.610 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.4214000000000004e-05 2024-09-13T11:23:21.610 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.610 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.610 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:23:21.610 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.118475+0000", 2024-09-13T11:23:21.611 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00085723699999999995 2024-09-13T11:23:21.611 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.611 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.611 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:21.611 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.167731+0000", 2024-09-13T11:23:21.611 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.049256105000000001 2024-09-13T11:23:21.611 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.611 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.611 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:23:21.611 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.167775+0000", 2024-09-13T11:23:21.611 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.3906000000000003e-05 2024-09-13T11:23:21.611 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.612 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.612 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:23:21.612 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.167808+0000", 2024-09-13T11:23:21.612 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.3417000000000002e-05 2024-09-13T11:23:21.612 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:21.612 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:23:21.612 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:21.612 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.612 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.612 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:25383 3.0 e55/30 3:0a6bb869:::benchmark_data_smithi177_31116_object25382:head v 55'1620, mlcod=55'1617)", 2024-09-13T11:23:21.612 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:04.449956+0000", 2024-09-13T11:23:21.613 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 17.120766879000001, 2024-09-13T11:23:21.613 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.21143479600000001, 2024-09-13T11:23:21.613 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:23:21.613 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:23:21.613 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:21.613 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:23:21.613 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.613 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:23:21.613 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.449956+0000", 2024-09-13T11:23:21.613 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:21.613 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.614 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.614 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:23:21.614 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.449956+0000", 2024-09-13T11:23:21.614 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:21.614 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.614 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.614 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:23:21.614 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.449959+0000", 2024-09-13T11:23:21.614 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.393e-06 2024-09-13T11:23:21.614 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.614 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.614 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:23:21.614 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.450014+0000", 2024-09-13T11:23:21.615 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.4200000000000003e-05 2024-09-13T11:23:21.615 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.615 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.615 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:23:21.615 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.658637+0000", 2024-09-13T11:23:21.615 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.20862334799999999 2024-09-13T11:23:21.615 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.615 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.615 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:23:21.615 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.658650+0000", 2024-09-13T11:23:21.616 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3148e-05 2024-09-13T11:23:21.616 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.616 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.616 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:23:21.616 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.658867+0000", 2024-09-13T11:23:21.616 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00021673499999999999 2024-09-13T11:23:21.616 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.616 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.616 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:23:21.616 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.658931+0000", 2024-09-13T11:23:21.617 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.4045000000000001e-05 2024-09-13T11:23:21.617 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.617 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.617 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:23:21.617 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.661317+0000", 2024-09-13T11:23:21.617 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.002385823 2024-09-13T11:23:21.617 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.617 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.617 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:23:21.617 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.661391+0000", 2024-09-13T11:23:21.617 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.4103999999999994e-05 2024-09-13T11:23:21.617 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:21.618 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:23:21.618 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:21.618 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.618 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.618 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:25420 3.0 e56/30 3:03f6c3d6:::benchmark_data_smithi177_31116_object25419:head v 56'1621, mlcod=55'1618)", 2024-09-13T11:23:21.618 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:04.658522+0000", 2024-09-13T11:23:21.618 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 16.912200836, 2024-09-13T11:23:21.618 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.073500560000000006, 2024-09-13T11:23:21.619 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:23:21.619 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:23:21.619 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:21.619 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:23:21.619 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.619 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:23:21.619 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.658522+0000", 2024-09-13T11:23:21.619 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:21.619 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.619 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.619 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:23:21.620 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.658522+0000", 2024-09-13T11:23:21.620 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:21.620 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.620 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.620 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:23:21.620 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.658523+0000", 2024-09-13T11:23:21.620 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.9999999999999997e-07 2024-09-13T11:23:21.620 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.620 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.621 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:23:21.621 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.658603+0000", 2024-09-13T11:23:21.621 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.0413000000000004e-05 2024-09-13T11:23:21.621 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.621 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.621 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:23:21.621 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.729224+0000", 2024-09-13T11:23:21.621 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.070621192999999999 2024-09-13T11:23:21.621 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.621 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.622 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:23:21.622 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.729239+0000", 2024-09-13T11:23:21.622 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5143999999999999e-05 2024-09-13T11:23:21.622 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.622 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.622 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:23:21.622 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.729386+0000", 2024-09-13T11:23:21.622 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00014676399999999999 2024-09-13T11:23:21.622 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.622 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.622 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:23:21.623 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.729427+0000", 2024-09-13T11:23:21.623 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.0451000000000003e-05 2024-09-13T11:23:21.624 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.624 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.625 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:23:21.625 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.731905+0000", 2024-09-13T11:23:21.625 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0024786949999999999 2024-09-13T11:23:21.625 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.625 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.625 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:23:21.625 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.732023+0000", 2024-09-13T11:23:21.625 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00011730000000000001 2024-09-13T11:23:21.625 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:21.625 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:23:21.626 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:21.626 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.626 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.626 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:25436 3.0 e56/30 3:07027188:::benchmark_data_smithi177_31116_object25435:head v 56'1622, mlcod=55'1619)", 2024-09-13T11:23:21.626 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:04.729114+0000", 2024-09-13T11:23:21.626 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 16.841608676, 2024-09-13T11:23:21.626 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.046114465, 2024-09-13T11:23:21.626 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:23:21.626 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:23:21.626 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:21.626 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:23:21.626 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.627 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:23:21.627 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.729114+0000", 2024-09-13T11:23:21.627 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:21.627 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.627 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.627 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:23:21.627 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.729114+0000", 2024-09-13T11:23:21.627 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:21.627 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.627 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.627 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:23:21.628 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.729118+0000", 2024-09-13T11:23:21.628 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.9230000000000004e-06 2024-09-13T11:23:21.628 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.628 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.628 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:23:21.628 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.729181+0000", 2024-09-13T11:23:21.628 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.2504999999999999e-05 2024-09-13T11:23:21.628 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.629 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.629 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:23:21.629 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.772755+0000", 2024-09-13T11:23:21.629 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.043574553000000002 2024-09-13T11:23:21.629 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.629 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.629 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:23:21.629 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.772771+0000", 2024-09-13T11:23:21.629 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5277999999999998e-05 2024-09-13T11:23:21.629 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.629 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.630 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:23:21.630 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.772914+0000", 2024-09-13T11:23:21.630 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00014371699999999999 2024-09-13T11:23:21.630 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.630 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.630 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:23:21.630 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.772953+0000", 2024-09-13T11:23:21.630 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.8858e-05 2024-09-13T11:23:21.630 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.630 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.630 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:23:21.631 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.775151+0000", 2024-09-13T11:23:21.631 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0021981050000000001 2024-09-13T11:23:21.631 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.631 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.631 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:23:21.631 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.775229+0000", 2024-09-13T11:23:21.631 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.7526000000000003e-05 2024-09-13T11:23:21.631 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:21.631 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:23:21.632 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:21.632 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.632 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.632 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:32785 3.5 3:ad14c732:::benchmark_data_smithi177_31116_object32784:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e60)", 2024-09-13T11:23:21.632 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:11.046235+0000", 2024-09-13T11:23:21.632 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 10.524487532, 2024-09-13T11:23:21.632 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.054203993999999998, 2024-09-13T11:23:21.632 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:23:21.632 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:23:21.632 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:21.632 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:23:21.632 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:23:21.633 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:21.633 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 32785 2024-09-13T11:23:21.633 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.633 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:23:21.633 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.633 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:23:21.633 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:11.046235+0000", 2024-09-13T11:23:21.633 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:21.633 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.633 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.633 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:23:21.633 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:11.046235+0000", 2024-09-13T11:23:21.633 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:21.634 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.634 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.634 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:23:21.634 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:11.046239+0000", 2024-09-13T11:23:21.634 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.095e-06 2024-09-13T11:23:21.634 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.634 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.634 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:23:21.634 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:11.046270+0000", 2024-09-13T11:23:21.634 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.1761000000000003e-05 2024-09-13T11:23:21.634 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.634 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.634 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:23:21.635 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:11.046292+0000", 2024-09-13T11:23:21.635 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.1736999999999999e-05 2024-09-13T11:23:21.635 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.635 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.635 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:23:21.635 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:11.046302+0000", 2024-09-13T11:23:21.635 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0413e-05 2024-09-13T11:23:21.635 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.635 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.635 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:23:21.635 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:11.046360+0000", 2024-09-13T11:23:21.635 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.7816999999999999e-05 2024-09-13T11:23:21.635 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.635 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.636 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:23:21.636 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:11.046481+0000", 2024-09-13T11:23:21.636 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000120846 2024-09-13T11:23:21.636 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.636 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.636 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 13", 2024-09-13T11:23:21.636 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:11.046576+0000", 2024-09-13T11:23:21.636 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.5189e-05 2024-09-13T11:23:21.636 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.636 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.636 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:23:21.636 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:11.047858+0000", 2024-09-13T11:23:21.636 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001281379 2024-09-13T11:23:21.637 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.637 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.637 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:21.637 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:11.100355+0000", 2024-09-13T11:23:21.637 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.052497061999999997 2024-09-13T11:23:21.637 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.637 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.637 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:23:21.637 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:11.100393+0000", 2024-09-13T11:23:21.637 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.8132e-05 2024-09-13T11:23:21.637 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.638 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.638 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:23:21.638 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:11.100439+0000", 2024-09-13T11:23:21.638 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.6563000000000001e-05 2024-09-13T11:23:21.638 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:21.638 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:23:21.638 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:21.638 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.638 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.638 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:32800 3.5 3:a2df9f69:::benchmark_data_smithi177_31116_object32799:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e60)", 2024-09-13T11:23:21.638 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:11.055291+0000", 2024-09-13T11:23:21.638 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 10.515431982999999, 2024-09-13T11:23:21.638 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.085442275999999998, 2024-09-13T11:23:21.639 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:23:21.639 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:23:21.639 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:21.639 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:23:21.639 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:23:21.639 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:21.639 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 32800 2024-09-13T11:23:21.639 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.639 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:23:21.639 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.639 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:23:21.639 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:11.055291+0000", 2024-09-13T11:23:21.640 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:21.640 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.640 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.640 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:23:21.640 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:11.055291+0000", 2024-09-13T11:23:21.640 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:21.640 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.640 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.640 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:23:21.640 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:11.055294+0000", 2024-09-13T11:23:21.641 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.411e-06 2024-09-13T11:23:21.641 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.641 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.641 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:23:21.641 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:11.055326+0000", 2024-09-13T11:23:21.641 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.1257999999999998e-05 2024-09-13T11:23:21.641 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.641 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.641 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:23:21.641 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:11.055346+0000", 2024-09-13T11:23:21.641 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.0483e-05 2024-09-13T11:23:21.641 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.641 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.641 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:23:21.642 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:11.055356+0000", 2024-09-13T11:23:21.642 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0217999999999999e-05 2024-09-13T11:23:21.642 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.642 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.642 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:23:21.642 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:11.055415+0000", 2024-09-13T11:23:21.642 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.8350999999999997e-05 2024-09-13T11:23:21.642 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.642 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.642 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:23:21.642 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:11.055514+0000", 2024-09-13T11:23:21.642 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.9380000000000001e-05 2024-09-13T11:23:21.642 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.643 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.643 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 13", 2024-09-13T11:23:21.643 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:11.055610+0000", 2024-09-13T11:23:21.643 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.5694999999999998e-05 2024-09-13T11:23:21.643 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.643 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.643 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:23:21.643 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:11.057365+0000", 2024-09-13T11:23:21.643 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0017549309999999999 2024-09-13T11:23:21.643 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.644 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.644 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:21.644 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:11.140661+0000", 2024-09-13T11:23:21.644 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.083296408000000002 2024-09-13T11:23:21.644 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.644 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.644 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:23:21.644 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:11.140703+0000", 2024-09-13T11:23:21.644 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.1823000000000003e-05 2024-09-13T11:23:21.644 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.644 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.644 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:23:21.644 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:11.140733+0000", 2024-09-13T11:23:21.644 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.0318e-05 2024-09-13T11:23:21.645 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:21.645 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:23:21.645 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:21.645 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.645 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.645 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:40949 3.0 e64/30 3:051bf38c:::benchmark_data_smithi177_31116_object40948:head v 64'2598, mlcod=64'2595)", 2024-09-13T11:23:21.646 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:16.543416+0000", 2024-09-13T11:23:21.646 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 5.0273074050000002, 2024-09-13T11:23:21.646 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.058856275999999999, 2024-09-13T11:23:21.646 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:23:21.646 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:23:21.646 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:21.646 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:23:21.646 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.646 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:23:21.647 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:16.543416+0000", 2024-09-13T11:23:21.647 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:21.647 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.647 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.647 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:23:21.647 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:16.543416+0000", 2024-09-13T11:23:21.647 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:21.647 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.647 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.647 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:23:21.647 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:16.543418+0000", 2024-09-13T11:23:21.647 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.9129999999999999e-06 2024-09-13T11:23:21.648 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.648 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.648 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:23:21.648 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:16.543467+0000", 2024-09-13T11:23:21.648 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.8825000000000001e-05 2024-09-13T11:23:21.648 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.648 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.648 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:23:21.648 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:16.600673+0000", 2024-09-13T11:23:21.648 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.057205473 2024-09-13T11:23:21.649 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.649 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.649 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:23:21.649 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:16.600680+0000", 2024-09-13T11:23:21.649 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.2300000000000002e-06 2024-09-13T11:23:21.649 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.649 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.649 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:23:21.649 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:16.600749+0000", 2024-09-13T11:23:21.649 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.9255e-05 2024-09-13T11:23:21.649 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.649 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.649 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:23:21.650 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:16.600768+0000", 2024-09-13T11:23:21.650 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.8348000000000001e-05 2024-09-13T11:23:21.650 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.650 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.650 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:23:21.650 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:16.602234+0000", 2024-09-13T11:23:21.650 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0014659499999999999 2024-09-13T11:23:21.650 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.650 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.650 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:23:21.650 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:16.602272+0000", 2024-09-13T11:23:21.650 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.8281999999999997e-05 2024-09-13T11:23:21.651 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:21.651 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:23:21.651 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:21.651 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.651 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.651 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:41125 3.0 e65/30 3:03dd7acf:::benchmark_data_smithi177_31116_object41124:head v 65'2605, mlcod=65'2602)", 2024-09-13T11:23:21.651 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:16.687513+0000", 2024-09-13T11:23:21.651 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 4.8832102260000001, 2024-09-13T11:23:21.651 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.058003289, 2024-09-13T11:23:21.651 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:23:21.651 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:23:21.651 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:21.651 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:23:21.652 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.652 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:23:21.652 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:16.687513+0000", 2024-09-13T11:23:21.652 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:21.652 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.652 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.652 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:23:21.652 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:16.687513+0000", 2024-09-13T11:23:21.652 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:21.652 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.652 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.652 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:23:21.652 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:16.687515+0000", 2024-09-13T11:23:21.653 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.1749999999999999e-06 2024-09-13T11:23:21.653 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.653 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.653 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:23:21.653 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:16.687560+0000", 2024-09-13T11:23:21.653 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.5080000000000002e-05 2024-09-13T11:23:21.653 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.653 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.653 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:23:21.653 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:16.743482+0000", 2024-09-13T11:23:21.653 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.055922472000000001 2024-09-13T11:23:21.653 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.653 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.653 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:23:21.654 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:16.743493+0000", 2024-09-13T11:23:21.654 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0633e-05 2024-09-13T11:23:21.654 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.654 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.654 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:23:21.654 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:16.743568+0000", 2024-09-13T11:23:21.654 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.5176e-05 2024-09-13T11:23:21.654 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.654 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.654 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:23:21.654 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:16.743593+0000", 2024-09-13T11:23:21.654 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.4924000000000001e-05 2024-09-13T11:23:21.655 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.655 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.655 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:23:21.655 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:16.745468+0000", 2024-09-13T11:23:21.655 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0018745159999999999 2024-09-13T11:23:21.655 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.655 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.655 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:23:21.655 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:16.745516+0000", 2024-09-13T11:23:21.656 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.8313000000000003e-05 2024-09-13T11:23:21.656 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:21.656 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:23:21.656 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:21.656 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.656 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.656 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:41164 3.5 3:a54052b0:::benchmark_data_smithi177_31116_object41163:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e65)", 2024-09-13T11:23:21.656 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:16.718639+0000", 2024-09-13T11:23:21.656 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 4.8520835370000004, 2024-09-13T11:23:21.656 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.053671647000000003, 2024-09-13T11:23:21.656 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:23:21.656 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:23:21.657 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:21.657 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:23:21.657 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:23:21.657 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:21.657 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 41164 2024-09-13T11:23:21.657 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.657 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:23:21.657 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.657 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:23:21.657 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:16.718639+0000", 2024-09-13T11:23:21.657 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:21.657 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.657 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.658 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:23:21.658 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:16.718639+0000", 2024-09-13T11:23:21.658 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:21.658 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.658 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.658 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:23:21.658 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:16.718641+0000", 2024-09-13T11:23:21.658 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.466e-06 2024-09-13T11:23:21.658 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.658 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.658 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:23:21.658 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:16.718674+0000", 2024-09-13T11:23:21.658 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.3456e-05 2024-09-13T11:23:21.658 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.659 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.659 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:23:21.659 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:16.718687+0000", 2024-09-13T11:23:21.659 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.2877999999999999e-05 2024-09-13T11:23:21.659 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.659 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.659 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:23:21.659 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:16.718693+0000", 2024-09-13T11:23:21.659 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.1890000000000004e-06 2024-09-13T11:23:21.659 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.659 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.659 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:23:21.659 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:16.718758+0000", 2024-09-13T11:23:21.660 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.4474999999999995e-05 2024-09-13T11:23:21.660 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.660 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.660 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:23:21.660 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:16.718885+0000", 2024-09-13T11:23:21.660 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00012749300000000001 2024-09-13T11:23:21.660 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.660 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.660 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 13", 2024-09-13T11:23:21.660 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:16.719015+0000", 2024-09-13T11:23:21.660 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00012969699999999999 2024-09-13T11:23:21.660 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.661 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.661 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:23:21.661 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:16.720517+0000", 2024-09-13T11:23:21.661 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0015014799999999999 2024-09-13T11:23:21.661 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.661 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.661 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:21.661 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:16.772214+0000", 2024-09-13T11:23:21.661 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.051697476999999999 2024-09-13T11:23:21.661 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.661 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.661 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:23:21.661 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:16.772266+0000", 2024-09-13T11:23:21.662 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.2012999999999998e-05 2024-09-13T11:23:21.662 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.662 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.662 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:23:21.662 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:16.772311+0000", 2024-09-13T11:23:21.662 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.5022999999999999e-05 2024-09-13T11:23:21.662 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:21.662 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:23:21.662 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:21.662 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.662 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.662 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop_reply(client.4665.0:41164 3.5 e65/30 ondisk, result = 0)", 2024-09-13T11:23:21.662 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:16.721259+0000", 2024-09-13T11:23:21.662 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 4.8494637259999998, 2024-09-13T11:23:21.663 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.051059460000000001, 2024-09-13T11:23:21.663 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:23:21.663 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:23:21.663 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:23:21.663 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:23:21.663 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.663 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:23:21.663 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:16.721259+0000", 2024-09-13T11:23:21.663 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:21.663 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.663 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.663 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:23:21.663 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:16.721259+0000", 2024-09-13T11:23:21.664 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:21.664 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.664 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.664 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:23:21.664 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:16.721261+0000", 2024-09-13T11:23:21.664 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.4839999999999999e-06 2024-09-13T11:23:21.664 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.665 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.665 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:23:21.665 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:16.721266+0000", 2024-09-13T11:23:21.665 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.2859999999999999e-06 2024-09-13T11:23:21.665 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.665 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.665 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:23:21.665 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:16.772136+0000", 2024-09-13T11:23:21.665 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.050870163000000003 2024-09-13T11:23:21.665 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.666 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.666 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:23:21.666 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:16.772145+0000", 2024-09-13T11:23:21.666 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.3349999999999996e-06 2024-09-13T11:23:21.666 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.666 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.666 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:23:21.666 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:16.772204+0000", 2024-09-13T11:23:21.666 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.8455999999999997e-05 2024-09-13T11:23:21.666 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.666 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.667 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:23:21.667 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:16.772210+0000", 2024-09-13T11:23:21.667 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.0129999999999999e-06 2024-09-13T11:23:21.667 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.667 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.667 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:23:21.667 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:16.772319+0000", 2024-09-13T11:23:21.667 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000108723 2024-09-13T11:23:21.667 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:21.667 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:23:21.667 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:21.667 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.667 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.667 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:41229 3.0 e65/30 3:005e9408:::benchmark_data_smithi177_31116_object41228:head v 65'2610, mlcod=65'2607)", 2024-09-13T11:23:21.668 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:16.772798+0000", 2024-09-13T11:23:21.668 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 4.7979252160000003, 2024-09-13T11:23:21.668 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.076228748999999998, 2024-09-13T11:23:21.668 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:23:21.668 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:23:21.668 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:21.668 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:23:21.668 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.669 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:23:21.669 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:16.772798+0000", 2024-09-13T11:23:21.669 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:21.669 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.669 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.669 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:23:21.669 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:16.772798+0000", 2024-09-13T11:23:21.669 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:21.669 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.669 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.670 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:23:21.670 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:16.772799+0000", 2024-09-13T11:23:21.670 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.73e-06 2024-09-13T11:23:21.670 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.670 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.670 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:23:21.670 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:16.772874+0000", 2024-09-13T11:23:21.670 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.4714000000000003e-05 2024-09-13T11:23:21.671 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.671 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.671 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:23:21.671 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:16.847048+0000", 2024-09-13T11:23:21.671 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.074173825999999998 2024-09-13T11:23:21.671 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.671 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.671 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:23:21.671 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:16.847058+0000", 2024-09-13T11:23:21.671 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.6679999999999996e-06 2024-09-13T11:23:21.671 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.672 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.672 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:23:21.672 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:16.847135+0000", 2024-09-13T11:23:21.672 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.7511999999999999e-05 2024-09-13T11:23:21.672 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.672 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.672 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:23:21.672 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:16.847162+0000", 2024-09-13T11:23:21.672 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.7036999999999999e-05 2024-09-13T11:23:21.672 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.672 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.672 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:23:21.673 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:16.848961+0000", 2024-09-13T11:23:21.673 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001799036 2024-09-13T11:23:21.673 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.673 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.673 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:23:21.673 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:16.849026+0000", 2024-09-13T11:23:21.673 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.5226000000000002e-05 2024-09-13T11:23:21.673 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:21.673 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:23:21.673 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:21.673 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.673 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.674 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:41278 3.5 3:a4c6a539:::benchmark_data_smithi177_31116_object41277:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e65)", 2024-09-13T11:23:21.674 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:16.819499+0000", 2024-09-13T11:23:21.674 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 4.7512241809999995, 2024-09-13T11:23:21.674 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.082103348000000007, 2024-09-13T11:23:21.674 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:23:21.674 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:23:21.674 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:21.674 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:23:21.674 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:23:21.674 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:21.674 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 41278 2024-09-13T11:23:21.674 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.674 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:23:21.674 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.675 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:23:21.675 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:16.819499+0000", 2024-09-13T11:23:21.675 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:21.675 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.675 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.675 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:23:21.675 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:16.819499+0000", 2024-09-13T11:23:21.675 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:21.675 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.675 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.675 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:23:21.675 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:16.819502+0000", 2024-09-13T11:23:21.675 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.1599999999999998e-06 2024-09-13T11:23:21.676 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.676 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.676 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:23:21.676 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:16.819530+0000", 2024-09-13T11:23:21.676 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.7613000000000001e-05 2024-09-13T11:23:21.676 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.676 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.676 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:23:21.676 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:16.819548+0000", 2024-09-13T11:23:21.676 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.8091e-05 2024-09-13T11:23:21.676 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.676 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.677 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:23:21.677 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:16.819557+0000", 2024-09-13T11:23:21.677 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.4050000000000006e-06 2024-09-13T11:23:21.677 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.677 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.677 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:23:21.677 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:16.819635+0000", 2024-09-13T11:23:21.677 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.7990000000000004e-05 2024-09-13T11:23:21.677 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.677 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.677 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:23:21.678 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:16.819813+0000", 2024-09-13T11:23:21.678 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00017749899999999999 2024-09-13T11:23:21.678 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.678 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.678 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 13", 2024-09-13T11:23:21.678 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:16.819984+0000", 2024-09-13T11:23:21.678 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00017193899999999999 2024-09-13T11:23:21.678 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.678 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.678 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:23:21.678 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:16.821659+0000", 2024-09-13T11:23:21.678 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0016748329999999999 2024-09-13T11:23:21.679 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.679 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.679 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:21.679 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:16.901504+0000", 2024-09-13T11:23:21.679 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.079845149000000004 2024-09-13T11:23:21.679 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.679 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.679 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:23:21.679 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:16.901547+0000", 2024-09-13T11:23:21.679 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.2553999999999999e-05 2024-09-13T11:23:21.679 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.679 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.679 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:23:21.680 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:16.901602+0000", 2024-09-13T11:23:21.680 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.5114999999999997e-05 2024-09-13T11:23:21.680 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:21.680 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:23:21.680 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:21.680 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.680 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.680 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop_reply(client.4665.0:41278 3.5 e65/30 ondisk, result = 0)", 2024-09-13T11:23:21.680 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:16.821657+0000", 2024-09-13T11:23:21.680 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 4.7490657189999999, 2024-09-13T11:23:21.681 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.079952737999999995, 2024-09-13T11:23:21.681 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:23:21.681 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:23:21.681 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:23:21.681 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:23:21.681 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.681 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:23:21.681 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:16.821657+0000", 2024-09-13T11:23:21.681 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:21.681 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.681 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.682 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:23:21.682 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:16.821657+0000", 2024-09-13T11:23:21.682 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:21.682 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.682 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.682 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:23:21.682 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:16.821659+0000", 2024-09-13T11:23:21.682 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.801e-06 2024-09-13T11:23:21.682 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.682 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.682 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:23:21.682 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:16.821666+0000", 2024-09-13T11:23:21.683 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.6669999999999999e-06 2024-09-13T11:23:21.683 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.683 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.683 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:23:21.683 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:16.901428+0000", 2024-09-13T11:23:21.683 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.079762258000000003 2024-09-13T11:23:21.683 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.683 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.683 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:23:21.683 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:16.901436+0000", 2024-09-13T11:23:21.683 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.0460000000000006e-06 2024-09-13T11:23:21.683 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.684 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.684 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:23:21.684 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:16.901493+0000", 2024-09-13T11:23:21.684 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.6511999999999997e-05 2024-09-13T11:23:21.684 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.684 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.684 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:23:21.684 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:16.901500+0000", 2024-09-13T11:23:21.684 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.8990000000000001e-06 2024-09-13T11:23:21.684 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.684 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.684 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:23:21.685 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:16.901610+0000", 2024-09-13T11:23:21.685 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000109555 2024-09-13T11:23:21.685 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:21.685 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:23:21.685 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:21.685 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.685 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.685 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:41301 3.0 e65/30 3:02cfc2d3:::benchmark_data_smithi177_31116_object41300:head v 65'2613, mlcod=65'2610)", 2024-09-13T11:23:21.685 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:16.852266+0000", 2024-09-13T11:23:21.685 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 4.7184573499999996, 2024-09-13T11:23:21.685 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.046276194, 2024-09-13T11:23:21.685 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:23:21.686 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:23:21.686 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:21.686 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:23:21.686 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.686 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:23:21.686 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:16.852266+0000", 2024-09-13T11:23:21.686 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:21.686 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.686 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.686 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:23:21.686 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:16.852266+0000", 2024-09-13T11:23:21.686 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:21.686 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.687 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.687 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:23:21.687 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:16.852267+0000", 2024-09-13T11:23:21.687 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5460000000000001e-06 2024-09-13T11:23:21.687 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.687 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.687 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:23:21.687 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:16.852316+0000", 2024-09-13T11:23:21.687 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.8584000000000001e-05 2024-09-13T11:23:21.687 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.687 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.688 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:23:21.688 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:16.896704+0000", 2024-09-13T11:23:21.688 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.044388351 2024-09-13T11:23:21.688 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.688 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.688 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:23:21.688 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:16.896714+0000", 2024-09-13T11:23:21.688 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.7289999999999995e-06 2024-09-13T11:23:21.688 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.688 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.688 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:23:21.688 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:16.896790+0000", 2024-09-13T11:23:21.689 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.6594999999999994e-05 2024-09-13T11:23:21.689 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.689 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.689 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:23:21.689 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:16.896810+0000", 2024-09-13T11:23:21.689 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.9423999999999999e-05 2024-09-13T11:23:21.689 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.689 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.689 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:23:21.689 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:16.898504+0000", 2024-09-13T11:23:21.689 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001694617 2024-09-13T11:23:21.689 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.690 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.690 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:23:21.690 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:16.898542+0000", 2024-09-13T11:23:21.690 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.7348000000000002e-05 2024-09-13T11:23:21.690 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:21.690 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:23:21.690 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:21.690 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.690 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.690 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:41541 3.5 3:a5fb3e8a:::benchmark_data_smithi177_31116_object41540:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e65)", 2024-09-13T11:23:21.691 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:17.081524+0000", 2024-09-13T11:23:21.691 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 4.4891991960000004, 2024-09-13T11:23:21.691 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.054097655000000001, 2024-09-13T11:23:21.691 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:23:21.691 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:23:21.691 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:21.691 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:23:21.691 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:23:21.691 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:21.691 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 41541 2024-09-13T11:23:21.691 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.691 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:23:21.691 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.691 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:23:21.692 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:17.081524+0000", 2024-09-13T11:23:21.692 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:21.692 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.692 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.692 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:23:21.692 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:17.081524+0000", 2024-09-13T11:23:21.692 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:21.692 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.692 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.692 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:23:21.692 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:17.081527+0000", 2024-09-13T11:23:21.692 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.0280000000000001e-06 2024-09-13T11:23:21.692 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.693 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.693 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:23:21.693 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:17.081560+0000", 2024-09-13T11:23:21.693 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.3148999999999997e-05 2024-09-13T11:23:21.693 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.693 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.693 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:23:21.693 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:17.081581+0000", 2024-09-13T11:23:21.693 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.0905999999999999e-05 2024-09-13T11:23:21.693 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.693 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.693 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:23:21.696 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:17.081592+0000", 2024-09-13T11:23:21.696 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0879999999999999e-05 2024-09-13T11:23:21.696 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.696 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.696 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:23:21.696 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:17.081661+0000", 2024-09-13T11:23:21.696 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.8992000000000006e-05 2024-09-13T11:23:21.696 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.697 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.697 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:23:21.697 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:17.081782+0000", 2024-09-13T11:23:21.697 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000121159 2024-09-13T11:23:21.697 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.697 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.697 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 13", 2024-09-13T11:23:21.697 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:17.081926+0000", 2024-09-13T11:23:21.697 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000144454 2024-09-13T11:23:21.697 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.697 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.697 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:23:21.697 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:17.083717+0000", 2024-09-13T11:23:21.697 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001790343 2024-09-13T11:23:21.698 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.698 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.698 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:21.698 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:17.135532+0000", 2024-09-13T11:23:21.698 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.051815426999999997 2024-09-13T11:23:21.698 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.698 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.698 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:23:21.698 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:17.135572+0000", 2024-09-13T11:23:21.698 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.0136000000000003e-05 2024-09-13T11:23:21.698 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.698 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.698 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:23:21.699 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:17.135621+0000", 2024-09-13T11:23:21.699 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.9181000000000002e-05 2024-09-13T11:23:21.699 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:21.699 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:23:21.699 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:21.699 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.699 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.699 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:41562 3.5 3:a22c5a6f:::benchmark_data_smithi177_31116_object41561:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e65)", 2024-09-13T11:23:21.699 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:17.098774+0000", 2024-09-13T11:23:21.699 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 4.4719492240000003, 2024-09-13T11:23:21.699 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.052671197000000003, 2024-09-13T11:23:21.699 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:23:21.700 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:23:21.700 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:21.700 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:23:21.700 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:23:21.700 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:21.700 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 41562 2024-09-13T11:23:21.700 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.700 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:23:21.700 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.700 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:23:21.700 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:17.098774+0000", 2024-09-13T11:23:21.700 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:21.700 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.701 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.701 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:23:21.701 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:17.098774+0000", 2024-09-13T11:23:21.701 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:21.701 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.701 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.701 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:23:21.702 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:17.098777+0000", 2024-09-13T11:23:21.702 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.5140000000000002e-06 2024-09-13T11:23:21.702 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.702 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.702 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:23:21.702 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:17.098816+0000", 2024-09-13T11:23:21.702 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.8911000000000001e-05 2024-09-13T11:23:21.703 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.703 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.703 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:23:21.703 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:17.098838+0000", 2024-09-13T11:23:21.703 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.1665e-05 2024-09-13T11:23:21.703 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.703 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.703 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:23:21.703 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:17.098850+0000", 2024-09-13T11:23:21.703 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.2091e-05 2024-09-13T11:23:21.703 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.703 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.703 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:23:21.704 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:17.098923+0000", 2024-09-13T11:23:21.704 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.3165999999999997e-05 2024-09-13T11:23:21.704 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.704 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.704 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:23:21.704 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:17.099032+0000", 2024-09-13T11:23:21.704 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000108964 2024-09-13T11:23:21.704 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.704 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.704 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 13", 2024-09-13T11:23:21.704 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:17.099143+0000", 2024-09-13T11:23:21.704 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00011065000000000001 2024-09-13T11:23:21.704 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.704 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.705 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:23:21.705 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:17.101083+0000", 2024-09-13T11:23:21.705 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0019405449999999999 2024-09-13T11:23:21.705 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.705 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.705 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:21.705 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:17.151206+0000", 2024-09-13T11:23:21.705 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.050122672 2024-09-13T11:23:21.705 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.705 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.705 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:23:21.705 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:17.151357+0000", 2024-09-13T11:23:21.705 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00015066100000000001 2024-09-13T11:23:21.705 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:21.706 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:21.706 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:23:21.706 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:17.151445+0000", 2024-09-13T11:23:21.706 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.8357999999999997e-05 2024-09-13T11:23:21.706 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:21.706 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:23:21.706 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:21.706 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:21.706 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:23:21.706 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-09-13T11:23:21.708 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-13T11:23:21.709 INFO:tasks.ceph.osd.5.smithi092.stderr:2024-09-13T11:23:21.705+0000 7f6f0b705640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26346) UID: 0 2024-09-13T11:23:21.808 INFO:tasks.ceph.osd.5.smithi092.stderr:2024-09-13T11:23:21.805+0000 7f6f0b705640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26346) UID: 0 2024-09-13T11:23:21.842 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-09-13T11:23:21.842 INFO:teuthology.orchestra.run.smithi092.stdout: "ops": [], 2024-09-13T11:23:21.842 INFO:teuthology.orchestra.run.smithi092.stdout: "num_ops": 0 2024-09-13T11:23:21.842 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-09-13T11:23:21.857 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-13T11:23:21.907 INFO:tasks.ceph.osd.10.smithi162.stderr:2024-09-13T11:23:21.902+0000 7fb593421640 -1 osd.10 69 handle_osd_map injecting map crc failure 2024-09-13T11:23:21.945 INFO:tasks.ceph.osd.13.smithi092.stderr:2024-09-13T11:23:21.945+0000 7f3e5afea640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26350) UID: 0 2024-09-13T11:23:21.951 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-13T11:23:22.003 INFO:tasks.radosbench.radosbench.0.smithi177.stdout: 55 16 45810 45794 52.0328 1.3125 0.00480203 0.0188666 2024-09-13T11:23:22.011 INFO:tasks.ceph.osd.5.smithi092.stderr:2024-09-13T11:23:22.009+0000 7f6f0b705640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26346) UID: 0 2024-09-13T11:23:22.028 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-09-13T11:23:22.028 INFO:teuthology.orchestra.run.smithi092.stdout: "ops": [], 2024-09-13T11:23:22.029 INFO:teuthology.orchestra.run.smithi092.stdout: "complaint_time": 30, 2024-09-13T11:23:22.029 INFO:teuthology.orchestra.run.smithi092.stdout: "num_blocked_ops": 0 2024-09-13T11:23:22.029 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-09-13T11:23:22.044 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-13T11:23:22.064 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T11:23:22.112 INFO:tasks.ceph.osd.14.smithi162.stderr:2024-09-13T11:23:22.106+0000 7f235c0db640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26319) UID: 0 2024-09-13T11:23:22.210 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-09-13T11:23:22.211 INFO:teuthology.orchestra.run.smithi092.stdout: "size": 20, 2024-09-13T11:23:22.211 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 600, 2024-09-13T11:23:22.211 INFO:teuthology.orchestra.run.smithi092.stdout: "ops": [ 2024-09-13T11:23:22.211 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:22.211 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:3 3.4 3.125e09e4 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e40)", 2024-09-13T11:23:22.211 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:46.304641+0000", 2024-09-13T11:23:22.211 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 35.903067503000003, 2024-09-13T11:23:22.211 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.374372119, 2024-09-13T11:23:22.211 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:23:22.211 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:23:22.211 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "reached pg", 2024-09-13T11:23:22.212 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:23:22.212 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:23:22.212 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:22.212 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 3 2024-09-13T11:23:22.212 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:22.212 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:23:22.212 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:22.212 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:23:22.212 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.304641+0000", 2024-09-13T11:23:22.212 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:22.212 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:22.213 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:22.213 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:23:22.213 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.304641+0000", 2024-09-13T11:23:22.213 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:22.213 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:22.213 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:22.213 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:23:22.213 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.304644+0000", 2024-09-13T11:23:22.213 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.5130000000000001e-06 2024-09-13T11:23:22.213 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:22.213 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:22.214 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:23:22.214 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305058+0000", 2024-09-13T11:23:22.214 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000414277 2024-09-13T11:23:22.214 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:22.214 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:22.214 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:23:22.214 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305124+0000", 2024-09-13T11:23:22.214 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.5813000000000002e-05 2024-09-13T11:23:22.214 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:22.214 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:22.214 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:23:22.214 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305156+0000", 2024-09-13T11:23:22.215 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.1489000000000003e-05 2024-09-13T11:23:22.215 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:22.215 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:22.215 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:23:22.215 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.678939+0000", 2024-09-13T11:23:22.215 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.37378339999999999 2024-09-13T11:23:22.215 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:22.215 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:22.215 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:23:22.215 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.679013+0000", 2024-09-13T11:23:22.216 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.3627000000000005e-05 2024-09-13T11:23:22.216 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:22.216 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:23:22.216 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:22.216 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:22.216 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:22.216 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:29 3.4 3.b7bbfbe4 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-09-13T11:23:22.216 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:46.305215+0000", 2024-09-13T11:23:22.216 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 35.902493200000002, 2024-09-13T11:23:22.216 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.37387964600000001, 2024-09-13T11:23:22.216 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:23:22.217 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:23:22.217 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "reached pg", 2024-09-13T11:23:22.217 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:23:22.217 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:23:22.217 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:22.217 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 29 2024-09-13T11:23:22.217 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:22.217 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:23:22.217 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:22.217 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:23:22.217 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305215+0000", 2024-09-13T11:23:22.217 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:22.217 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:22.218 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:22.218 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:23:22.218 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305215+0000", 2024-09-13T11:23:22.218 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:22.218 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:22.218 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:22.218 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:23:22.218 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305218+0000", 2024-09-13T11:23:22.218 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.7030000000000002e-06 2024-09-13T11:23:22.218 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:22.218 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:22.218 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:23:22.218 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305366+0000", 2024-09-13T11:23:22.219 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00014825400000000001 2024-09-13T11:23:22.219 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:22.219 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:22.219 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:23:22.219 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305386+0000", 2024-09-13T11:23:22.219 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.9689000000000001e-05 2024-09-13T11:23:22.219 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:22.219 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:22.219 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:23:22.219 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305403+0000", 2024-09-13T11:23:22.219 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.7600000000000001e-05 2024-09-13T11:23:22.219 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:22.220 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:22.220 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:23:22.220 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.679067+0000", 2024-09-13T11:23:22.220 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.37366339100000001 2024-09-13T11:23:22.220 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:22.220 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:22.220 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:23:22.220 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.679095+0000", 2024-09-13T11:23:22.220 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.8008999999999999e-05 2024-09-13T11:23:22.220 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:22.220 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:23:22.220 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:22.221 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:22.221 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:22.221 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:30 3.4 3.547ec0f4 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-09-13T11:23:22.221 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:46.305453+0000", 2024-09-13T11:23:22.221 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 35.902255076000003, 2024-09-13T11:23:22.221 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.373764914, 2024-09-13T11:23:22.221 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:23:22.221 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:23:22.221 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "reached pg", 2024-09-13T11:23:22.221 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:23:22.221 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:23:22.221 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:22.221 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 30 2024-09-13T11:23:22.222 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:22.222 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:23:22.222 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:22.222 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:23:22.222 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305453+0000", 2024-09-13T11:23:22.222 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:22.222 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:22.222 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:22.222 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:23:22.222 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305453+0000", 2024-09-13T11:23:22.222 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:22.222 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:22.222 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:22.223 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:23:22.223 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305456+0000", 2024-09-13T11:23:22.223 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.9560000000000002e-06 2024-09-13T11:23:22.223 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:22.223 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:22.223 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:23:22.223 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305597+0000", 2024-09-13T11:23:22.223 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00014122700000000001 2024-09-13T11:23:22.223 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:22.223 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:22.223 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:23:22.223 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305615+0000", 2024-09-13T11:23:22.223 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.8139999999999999e-05 2024-09-13T11:23:22.223 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:22.223 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:22.224 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:23:22.224 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305632+0000", 2024-09-13T11:23:22.224 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.6101999999999999e-05 2024-09-13T11:23:22.224 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:22.224 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:22.224 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:23:22.224 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.679195+0000", 2024-09-13T11:23:22.224 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.373563486 2024-09-13T11:23:22.224 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:22.224 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:22.224 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:23:22.224 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.679218+0000", 2024-09-13T11:23:22.225 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.3003e-05 2024-09-13T11:23:22.225 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:22.225 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:23:22.225 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:22.225 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:22.225 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:22.225 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:70 3.4 3.1404a604 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-09-13T11:23:22.225 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:46.305665+0000", 2024-09-13T11:23:22.225 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 35.902042928, 2024-09-13T11:23:22.225 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.37365550199999997, 2024-09-13T11:23:22.225 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:23:22.225 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:23:22.226 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "reached pg", 2024-09-13T11:23:22.226 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:23:22.226 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:23:22.226 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:22.226 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 70 2024-09-13T11:23:22.226 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:22.226 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:23:22.226 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:22.226 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:23:22.226 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305665+0000", 2024-09-13T11:23:22.226 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:22.226 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:22.226 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:22.226 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:23:22.227 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305665+0000", 2024-09-13T11:23:22.227 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:22.227 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:22.227 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:22.227 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:23:22.227 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305668+0000", 2024-09-13T11:23:22.227 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.5890000000000001e-06 2024-09-13T11:23:22.227 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:22.227 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:22.227 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:23:22.227 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305791+0000", 2024-09-13T11:23:22.227 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000123545 2024-09-13T11:23:22.227 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:22.227 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:22.228 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:23:22.228 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305808+0000", 2024-09-13T11:23:22.228 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.668e-05 2024-09-13T11:23:22.228 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:22.228 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:22.228 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:23:22.228 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305824+0000", 2024-09-13T11:23:22.228 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.6385e-05 2024-09-13T11:23:22.228 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:22.228 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:22.228 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:23:22.228 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.679305+0000", 2024-09-13T11:23:22.228 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.37348059900000002 2024-09-13T11:23:22.229 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:22.229 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:22.229 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:23:22.229 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.679321+0000", 2024-09-13T11:23:22.229 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.5704000000000001e-05 2024-09-13T11:23:22.229 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:22.229 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:23:22.229 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:22.229 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:22.229 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:22.229 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:8943 3.4 3:2e96fe7b:::benchmark_data_smithi177_31116_object8942:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-09-13T11:23:22.229 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:53.640283+0000", 2024-09-13T11:23:22.229 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 28.567424775999999, 2024-09-13T11:23:22.229 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.071508547000000006, 2024-09-13T11:23:22.230 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:23:22.230 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:23:22.230 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:22.230 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:23:22.230 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:23:22.230 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:22.230 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 8943 2024-09-13T11:23:22.230 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:22.230 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:23:22.230 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:22.230 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:23:22.230 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:53.640283+0000", 2024-09-13T11:23:22.230 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:22.230 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:22.231 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:22.231 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:23:22.231 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:53.640283+0000", 2024-09-13T11:23:22.231 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:22.231 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:22.231 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:22.231 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:23:22.231 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:53.640286+0000", 2024-09-13T11:23:22.231 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.5519999999999999e-06 2024-09-13T11:23:22.231 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:22.231 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:22.231 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:23:22.231 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:53.640336+0000", 2024-09-13T11:23:22.232 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.9676999999999999e-05 2024-09-13T11:23:22.232 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:22.232 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:22.232 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:23:22.232 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:53.640354+0000", 2024-09-13T11:23:22.232 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.8082999999999999e-05 2024-09-13T11:23:22.232 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:22.232 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:22.232 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:23:22.232 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:53.640362+0000", 2024-09-13T11:23:22.232 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.5739999999999996e-06 2024-09-13T11:23:22.232 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:22.232 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:22.232 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:23:22.233 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:53.640442+0000", 2024-09-13T11:23:22.233 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.9500999999999997e-05 2024-09-13T11:23:22.233 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:22.233 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:22.233 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:23:22.233 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:53.640593+0000", 2024-09-13T11:23:22.233 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000151123 2024-09-13T11:23:22.233 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:22.233 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:22.233 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 12", 2024-09-13T11:23:22.233 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:53.640717+0000", 2024-09-13T11:23:22.233 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00012440199999999999 2024-09-13T11:23:22.233 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:22.234 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:22.234 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:23:22.234 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:53.641838+0000", 2024-09-13T11:23:22.234 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.001120675 2024-09-13T11:23:22.234 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:22.234 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:22.234 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:22.234 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:53.711666+0000", 2024-09-13T11:23:22.234 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.069827881999999994 2024-09-13T11:23:22.234 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:22.234 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:22.234 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:23:22.234 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:53.711737+0000", 2024-09-13T11:23:22.234 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.1274999999999998e-05 2024-09-13T11:23:22.235 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:22.235 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:22.235 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:23:22.235 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:53.711792+0000", 2024-09-13T11:23:22.235 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.4802999999999997e-05 2024-09-13T11:23:22.235 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:22.235 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:23:22.235 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:22.235 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:22.235 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:22.235 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:9301 3.4 3:2c2635c1:::benchmark_data_smithi177_31116_object9300:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-09-13T11:23:22.235 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:53.910351+0000", 2024-09-13T11:23:22.235 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 28.297357049999999, 2024-09-13T11:23:22.235 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.061138634999999997, 2024-09-13T11:23:22.235 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:23:22.236 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:23:22.236 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:22.236 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:23:22.236 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:23:22.236 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:22.236 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 9301 2024-09-13T11:23:22.236 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:22.236 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:23:22.236 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:22.236 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:23:22.236 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:53.910351+0000", 2024-09-13T11:23:22.236 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:22.236 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:22.237 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:22.237 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:23:22.237 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:53.910351+0000", 2024-09-13T11:23:22.237 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:22.237 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:22.237 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:22.237 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:23:22.237 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:53.910353+0000", 2024-09-13T11:23:22.237 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.8139999999999999e-06 2024-09-13T11:23:22.237 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:22.237 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:22.237 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:23:22.237 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:53.910417+0000", 2024-09-13T11:23:22.237 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.4354000000000001e-05 2024-09-13T11:23:22.238 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:22.238 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:22.238 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:23:22.238 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:53.910441+0000", 2024-09-13T11:23:22.238 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.3306e-05 2024-09-13T11:23:22.238 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:22.238 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:22.238 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:23:22.238 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:53.910450+0000", 2024-09-13T11:23:22.238 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.6849999999999996e-06 2024-09-13T11:23:22.238 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:22.238 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:22.238 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:23:22.238 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:53.910580+0000", 2024-09-13T11:23:22.238 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00012942699999999999 2024-09-13T11:23:22.239 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:22.239 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:22.239 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:23:22.239 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:53.910808+0000", 2024-09-13T11:23:22.239 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00022840100000000001 2024-09-13T11:23:22.239 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:22.239 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:22.239 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 12", 2024-09-13T11:23:22.239 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:53.911027+0000", 2024-09-13T11:23:22.239 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000218421 2024-09-13T11:23:22.239 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:22.239 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:22.239 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:23:22.239 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:53.913684+0000", 2024-09-13T11:23:22.239 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0026571780000000001 2024-09-13T11:23:22.239 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:22.240 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:22.240 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:22.240 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:53.971316+0000", 2024-09-13T11:23:22.240 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.057631911000000001 2024-09-13T11:23:22.240 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:22.240 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:22.240 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:23:22.240 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:53.971414+0000", 2024-09-13T11:23:22.240 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.8793000000000001e-05 2024-09-13T11:23:22.240 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:22.240 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:22.240 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:23:22.240 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:53.971490+0000", 2024-09-13T11:23:22.241 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.5345000000000005e-05 2024-09-13T11:23:22.241 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:22.241 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:23:22.241 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:22.241 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:22.241 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:22.241 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop_reply(client.4665.0:9301 3.4 e47/40 ondisk, result = 0)", 2024-09-13T11:23:22.241 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:53.913196+0000", 2024-09-13T11:23:22.241 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 28.294512105999999, 2024-09-13T11:23:22.241 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.058304652999999998, 2024-09-13T11:23:22.241 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:23:22.241 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:23:22.241 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-09-13T11:23:22.242 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:23:22.242 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:22.242 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:23:22.242 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:53.913196+0000", 2024-09-13T11:23:22.242 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:22.242 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:22.242 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:22.242 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:23:22.242 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:53.913196+0000", 2024-09-13T11:23:22.242 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:22.242 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:22.242 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:22.242 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:23:22.242 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:53.913199+0000", 2024-09-13T11:23:22.243 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.5799999999999999e-06 2024-09-13T11:23:22.243 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:22.243 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:22.243 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:23:22.243 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:53.913210+0000", 2024-09-13T11:23:22.243 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.083e-05 2024-09-13T11:23:22.243 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:22.243 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:22.243 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:23:22.243 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:53.971150+0000", 2024-09-13T11:23:22.243 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.057940525 2024-09-13T11:23:22.243 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:22.243 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:22.243 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:23:22.243 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:53.971161+0000", 2024-09-13T11:23:22.244 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0774999999999999e-05 2024-09-13T11:23:22.244 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:22.244 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:22.244 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:23:22.244 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:53.971292+0000", 2024-09-13T11:23:22.244 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00013122000000000001 2024-09-13T11:23:22.244 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:22.244 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:22.244 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:23:22.244 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:53.971305+0000", 2024-09-13T11:23:22.244 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.3115000000000001e-05 2024-09-13T11:23:22.244 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:22.244 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:22.244 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:23:22.245 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:53.971501+0000", 2024-09-13T11:23:22.245 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000195608 2024-09-13T11:23:22.245 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:22.245 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:23:22.245 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:22.245 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:22.245 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:22.245 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:25290 3.4 3:2c6b3a0e:::benchmark_data_smithi177_31116_object25289:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e55)", 2024-09-13T11:23:22.245 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:04.310470+0000", 2024-09-13T11:23:22.245 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 17.897238201, 2024-09-13T11:23:22.245 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.095019785999999995, 2024-09-13T11:23:22.245 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:23:22.245 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:23:22.245 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:22.246 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:23:22.246 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:23:22.246 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:22.246 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 25290 2024-09-13T11:23:22.246 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:22.246 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:23:22.246 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:22.246 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:23:22.246 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.310470+0000", 2024-09-13T11:23:22.246 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:22.246 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:22.246 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:22.246 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:23:22.247 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.310470+0000", 2024-09-13T11:23:22.247 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:22.247 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:22.247 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:22.247 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:23:22.247 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.310472+0000", 2024-09-13T11:23:22.247 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.481e-06 2024-09-13T11:23:22.247 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:22.247 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:22.247 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:23:22.247 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.310527+0000", 2024-09-13T11:23:22.248 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.4999e-05 2024-09-13T11:23:22.248 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:22.248 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:22.248 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:23:22.248 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.310540+0000", 2024-09-13T11:23:22.248 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.2206e-05 2024-09-13T11:23:22.248 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:22.248 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:22.248 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:23:22.248 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.310545+0000", 2024-09-13T11:23:22.248 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.7749999999999998e-06 2024-09-13T11:23:22.248 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:22.249 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:22.249 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:23:22.249 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.310598+0000", 2024-09-13T11:23:22.249 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.2095999999999997e-05 2024-09-13T11:23:22.249 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:22.249 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:22.249 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:23:22.249 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.310706+0000", 2024-09-13T11:23:22.249 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000108655 2024-09-13T11:23:22.249 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:22.249 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:22.249 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 12", 2024-09-13T11:23:22.249 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.310793+0000", 2024-09-13T11:23:22.249 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.7023999999999999e-05 2024-09-13T11:23:22.250 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:22.250 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:22.250 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:23:22.250 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.311820+0000", 2024-09-13T11:23:22.250 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.001027062 2024-09-13T11:23:22.250 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:22.250 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:22.250 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:22.250 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.405301+0000", 2024-09-13T11:23:22.250 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.093480806 2024-09-13T11:23:22.251 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:22.251 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:22.251 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:23:22.251 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.405400+0000", 2024-09-13T11:23:22.251 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.9279999999999998e-05 2024-09-13T11:23:22.251 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:22.251 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:22.251 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:23:22.251 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.405490+0000", 2024-09-13T11:23:22.251 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.9401999999999995e-05 2024-09-13T11:23:22.251 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:22.252 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:23:22.252 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:22.252 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:22.252 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:22.252 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:25408 3.4 3:294e23bd:::benchmark_data_smithi177_31116_object25407:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e56)", 2024-09-13T11:23:22.252 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:04.495455+0000", 2024-09-13T11:23:22.252 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 17.712253275999998, 2024-09-13T11:23:22.252 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.29897367600000002, 2024-09-13T11:23:22.252 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:23:22.252 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:23:22.252 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:22.253 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:23:22.253 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:23:22.253 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:22.253 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 25408 2024-09-13T11:23:22.253 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:22.253 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:23:22.253 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:22.253 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:23:22.253 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.495455+0000", 2024-09-13T11:23:22.253 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:22.253 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:22.253 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:22.253 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:23:22.254 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.495455+0000", 2024-09-13T11:23:22.254 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:22.254 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:22.254 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:22.254 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:23:22.254 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.495459+0000", 2024-09-13T11:23:22.254 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.5630000000000002e-06 2024-09-13T11:23:22.254 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:22.254 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:22.254 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:23:22.254 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.495545+0000", 2024-09-13T11:23:22.255 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.5704000000000005e-05 2024-09-13T11:23:22.255 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:22.255 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:22.255 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:23:22.255 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.495581+0000", 2024-09-13T11:23:22.255 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.5689000000000003e-05 2024-09-13T11:23:22.255 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:22.255 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:22.255 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:23:22.255 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.495597+0000", 2024-09-13T11:23:22.255 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.6053e-05 2024-09-13T11:23:22.255 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:22.255 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:22.255 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:23:22.256 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.495721+0000", 2024-09-13T11:23:22.256 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00012379099999999999 2024-09-13T11:23:22.256 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:22.256 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:22.256 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:23:22.256 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.495940+0000", 2024-09-13T11:23:22.256 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00021967699999999999 2024-09-13T11:23:22.256 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:22.256 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:22.256 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 12", 2024-09-13T11:23:22.256 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.496162+0000", 2024-09-13T11:23:22.256 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000221917 2024-09-13T11:23:22.257 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:22.257 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:22.257 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:23:22.257 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.498699+0000", 2024-09-13T11:23:22.257 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0025366690000000001 2024-09-13T11:23:22.257 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:22.257 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:22.257 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:22.257 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.794139+0000", 2024-09-13T11:23:22.257 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.295440025 2024-09-13T11:23:22.257 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:22.257 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:22.257 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:23:22.257 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.794306+0000", 2024-09-13T11:23:22.258 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00016689799999999999 2024-09-13T11:23:22.258 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:22.258 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:22.258 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:23:22.258 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.794429+0000", 2024-09-13T11:23:22.258 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00012269000000000001 2024-09-13T11:23:22.258 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:22.258 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:23:22.258 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:22.258 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:22.259 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:22.259 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop(client.4665.0:30053 3.e e59/57 3:7e3b05ed:::benchmark_data_smithi177_31116_object30052:head v 59'1945, mlcod=43'155)", 2024-09-13T11:23:22.259 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:08.869231+0000", 2024-09-13T11:23:22.259 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 13.338477163, 2024-09-13T11:23:22.259 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.073509376000000001, 2024-09-13T11:23:22.259 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:23:22.259 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:23:22.259 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:22.259 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:23:22.259 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:22.259 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:23:22.259 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:08.869231+0000", 2024-09-13T11:23:22.259 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:22.260 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:22.260 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:22.260 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:23:22.260 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:08.869231+0000", 2024-09-13T11:23:22.260 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:22.260 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:22.261 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:22.261 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:23:22.261 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:08.869233+0000", 2024-09-13T11:23:22.261 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.7260000000000001e-06 2024-09-13T11:23:22.261 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:22.261 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:22.262 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:23:22.262 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:08.869278+0000", 2024-09-13T11:23:22.262 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.5677000000000003e-05 2024-09-13T11:23:22.262 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:22.262 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:22.262 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:23:22.263 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:08.940547+0000", 2024-09-13T11:23:22.263 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.071268189999999995 2024-09-13T11:23:22.263 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:22.263 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:22.263 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:23:22.263 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:08.940556+0000", 2024-09-13T11:23:22.264 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.3030000000000005e-06 2024-09-13T11:23:22.264 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:22.264 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:22.264 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:23:22.264 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:08.940636+0000", 2024-09-13T11:23:22.264 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.9964999999999998e-05 2024-09-13T11:23:22.265 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:22.265 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:22.265 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:23:22.265 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:08.940661+0000", 2024-09-13T11:23:22.265 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.4947e-05 2024-09-13T11:23:22.265 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:22.266 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:22.266 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:23:22.266 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:08.942703+0000", 2024-09-13T11:23:22.266 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0020419430000000001 2024-09-13T11:23:22.266 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:22.266 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:22.267 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:23:22.267 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:08.942740+0000", 2024-09-13T11:23:22.267 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.7625e-05 2024-09-13T11:23:22.267 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:22.267 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:23:22.268 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:22.268 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:22.268 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:22.268 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "MOSDPGPush(3.e 59/57 [PushOp(3:70ec9a0c:::benchmark_data_smithi177_31116_object2293:head, version: 43'157, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:70ec9a0c:::benchmark_data_smithi177_31116_object2293:head@43'157, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:23:22.268 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:08.878216+0000", 2024-09-13T11:23:22.269 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 13.329491789, 2024-09-13T11:23:22.269 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.076943826000000007, 2024-09-13T11:23:22.269 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:23:22.269 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:23:22.269 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-09-13T11:23:22.269 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:23:22.270 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:22.270 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:23:22.270 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:08.878216+0000", 2024-09-13T11:23:22.270 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:22.270 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:22.271 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:22.271 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:23:22.271 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:08.878216+0000", 2024-09-13T11:23:22.271 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:22.271 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:22.271 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:22.272 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:23:22.272 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:08.878218+0000", 2024-09-13T11:23:22.272 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.798e-06 2024-09-13T11:23:22.272 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:22.272 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:22.272 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:23:22.273 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:08.878299+0000", 2024-09-13T11:23:22.273 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.0929999999999999e-05 2024-09-13T11:23:22.273 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:22.273 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:22.273 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:23:22.274 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:08.953946+0000", 2024-09-13T11:23:22.274 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.075646586000000002 2024-09-13T11:23:22.274 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:22.274 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:22.274 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:23:22.274 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:08.953964+0000", 2024-09-13T11:23:22.274 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.8715999999999998e-05 2024-09-13T11:23:22.275 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:22.275 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:22.275 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:23:22.275 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:08.954080+0000", 2024-09-13T11:23:22.275 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000115734 2024-09-13T11:23:22.275 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:22.276 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:22.276 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:23:22.276 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:08.954095+0000", 2024-09-13T11:23:22.276 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.5024999999999999e-05 2024-09-13T11:23:22.276 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:22.276 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:22.277 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:23:22.277 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:08.955160+0000", 2024-09-13T11:23:22.277 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.001065037 2024-09-13T11:23:22.277 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:22.277 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:23:22.277 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:22.278 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:22.278 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:22.278 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop(client.4665.0:30353 3.b e59/40 3:da72bd4f:::benchmark_data_smithi177_31116_object30352:head v 59'1909, mlcod=59'1906)", 2024-09-13T11:23:22.278 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:09.165566+0000", 2024-09-13T11:23:22.278 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 13.042142189, 2024-09-13T11:23:22.279 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.090209561999999993, 2024-09-13T11:23:22.279 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:23:22.279 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:23:22.279 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:22.279 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:23:22.279 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:22.280 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:23:22.280 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:09.165566+0000", 2024-09-13T11:23:22.280 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:22.280 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:22.280 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:22.280 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:23:22.281 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:09.165566+0000", 2024-09-13T11:23:22.281 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:22.281 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:22.281 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:22.281 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:23:22.281 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:09.165568+0000", 2024-09-13T11:23:22.282 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.4449999999999999e-06 2024-09-13T11:23:22.282 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:22.282 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:22.282 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:23:22.282 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:09.165621+0000", 2024-09-13T11:23:22.282 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.2533e-05 2024-09-13T11:23:22.283 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:22.283 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:22.283 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:23:22.283 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:09.249936+0000", 2024-09-13T11:23:22.283 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.084315347999999998 2024-09-13T11:23:22.283 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:22.284 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:22.284 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:23:22.284 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:09.249954+0000", 2024-09-13T11:23:22.284 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.7479e-05 2024-09-13T11:23:22.284 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:22.285 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:22.285 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:23:22.285 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:09.250071+0000", 2024-09-13T11:23:22.285 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000116827 2024-09-13T11:23:22.285 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:22.285 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:22.286 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:23:22.286 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:09.250105+0000", 2024-09-13T11:23:22.286 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.4212999999999999e-05 2024-09-13T11:23:22.286 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:22.286 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:22.286 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:23:22.286 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:09.255685+0000", 2024-09-13T11:23:22.287 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.005580588 2024-09-13T11:23:22.287 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:22.287 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:22.287 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:23:22.287 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:09.255776+0000", 2024-09-13T11:23:22.288 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.0129000000000004e-05 2024-09-13T11:23:22.288 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:22.288 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:23:22.288 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:22.288 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:22.288 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:22.288 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop(client.4665.0:30360 3.e e59/57 3:7373925b:::benchmark_data_smithi177_31116_object30359:head v 59'1966, mlcod=43'163)", 2024-09-13T11:23:22.288 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:09.176488+0000", 2024-09-13T11:23:22.289 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 13.031219772, 2024-09-13T11:23:22.289 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.065334847000000001, 2024-09-13T11:23:22.289 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:23:22.289 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:23:22.289 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:22.289 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:23:22.289 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:22.289 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:23:22.289 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:09.176488+0000", 2024-09-13T11:23:22.289 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:22.289 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:22.289 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:22.290 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:23:22.290 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:09.176488+0000", 2024-09-13T11:23:22.290 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:22.290 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:22.290 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:22.290 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:23:22.290 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:09.176490+0000", 2024-09-13T11:23:22.290 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.325e-06 2024-09-13T11:23:22.290 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:22.290 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:22.290 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:23:22.290 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:09.176548+0000", 2024-09-13T11:23:22.291 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.8167000000000001e-05 2024-09-13T11:23:22.291 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:22.291 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:22.291 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:23:22.291 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:09.236693+0000", 2024-09-13T11:23:22.291 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.060145536999999999 2024-09-13T11:23:22.291 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:22.291 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:22.291 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:23:22.291 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:09.236707+0000", 2024-09-13T11:23:22.291 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.3531e-05 2024-09-13T11:23:22.291 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:22.292 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:22.292 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:23:22.292 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:09.236806+0000", 2024-09-13T11:23:22.292 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.9074999999999996e-05 2024-09-13T11:23:22.292 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:22.292 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:22.292 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:23:22.292 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:09.236840+0000", 2024-09-13T11:23:22.292 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.4415000000000002e-05 2024-09-13T11:23:22.292 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:22.292 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:22.292 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:23:22.292 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:09.241749+0000", 2024-09-13T11:23:22.293 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0049085509999999997 2024-09-13T11:23:22.293 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:22.293 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:22.293 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:23:22.293 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:09.241823+0000", 2024-09-13T11:23:22.293 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.4245999999999994e-05 2024-09-13T11:23:22.293 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:22.293 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:23:22.293 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:22.293 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:22.293 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:22.294 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop(client.4665.0:30855 3.b e59/40 3:dc59714f:::benchmark_data_smithi177_31116_object30854:head v 59'1930, mlcod=59'1927)", 2024-09-13T11:23:22.294 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:09.550713+0000", 2024-09-13T11:23:22.294 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 12.656995483999999, 2024-09-13T11:23:22.294 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.17337753, 2024-09-13T11:23:22.294 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:23:22.294 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:23:22.294 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:22.294 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:23:22.294 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:22.294 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:23:22.294 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:09.550713+0000", 2024-09-13T11:23:22.294 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:22.295 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:22.295 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:22.295 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:23:22.295 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:09.550713+0000", 2024-09-13T11:23:22.295 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:22.295 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:22.295 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:22.295 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:23:22.295 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:09.550716+0000", 2024-09-13T11:23:22.295 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.5099999999999999e-06 2024-09-13T11:23:22.295 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:22.295 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:22.296 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:23:22.296 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:09.550764+0000", 2024-09-13T11:23:22.296 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.7389000000000003e-05 2024-09-13T11:23:22.296 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:22.296 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:22.296 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:23:22.296 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:09.719062+0000", 2024-09-13T11:23:22.296 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.16829854499999999 2024-09-13T11:23:22.296 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:22.296 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:22.296 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:23:22.296 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:09.719075+0000", 2024-09-13T11:23:22.296 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.2969000000000001e-05 2024-09-13T11:23:22.296 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:22.297 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:22.297 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:23:22.297 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:09.719216+0000", 2024-09-13T11:23:22.297 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000141266 2024-09-13T11:23:22.297 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:22.297 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:22.297 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:23:22.297 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:09.719277+0000", 2024-09-13T11:23:22.297 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.0244999999999997e-05 2024-09-13T11:23:22.297 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:22.297 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:22.297 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:23:22.297 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:09.724019+0000", 2024-09-13T11:23:22.298 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0047426020000000003 2024-09-13T11:23:22.298 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:22.298 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:22.298 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:23:22.298 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:09.724090+0000", 2024-09-13T11:23:22.298 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.1004e-05 2024-09-13T11:23:22.298 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:22.298 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:23:22.298 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:22.298 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:22.298 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:22.298 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:38307 3.4 3:2a0b6b65:::benchmark_data_smithi177_31116_object38306:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e63)", 2024-09-13T11:23:22.298 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:14.624057+0000", 2024-09-13T11:23:22.299 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 7.583651326, 2024-09-13T11:23:22.299 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.067672300000000005, 2024-09-13T11:23:22.299 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:23:22.299 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:23:22.299 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:22.299 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:23:22.299 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:23:22.299 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:22.299 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 38307 2024-09-13T11:23:22.299 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:22.299 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:23:22.299 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:22.299 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:23:22.299 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:14.624057+0000", 2024-09-13T11:23:22.299 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:22.300 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:22.300 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:22.300 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:23:22.300 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:14.624057+0000", 2024-09-13T11:23:22.300 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:22.300 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:22.300 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:22.300 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:23:22.300 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:14.624060+0000", 2024-09-13T11:23:22.300 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.6179999999999999e-06 2024-09-13T11:23:22.300 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:22.300 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:22.300 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:23:22.300 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:14.624120+0000", 2024-09-13T11:23:22.301 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.9527000000000001e-05 2024-09-13T11:23:22.301 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:22.301 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:22.301 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:23:22.301 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:14.624145+0000", 2024-09-13T11:23:22.301 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.5208e-05 2024-09-13T11:23:22.301 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:22.301 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:22.301 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:23:22.301 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:14.624158+0000", 2024-09-13T11:23:22.301 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.3239e-05 2024-09-13T11:23:22.301 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:22.301 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:22.302 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:23:22.302 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:14.624236+0000", 2024-09-13T11:23:22.302 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.7806999999999996e-05 2024-09-13T11:23:22.302 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:22.302 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:22.302 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:23:22.302 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:14.624424+0000", 2024-09-13T11:23:22.302 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00018798600000000001 2024-09-13T11:23:22.302 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:22.302 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:22.302 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 12", 2024-09-13T11:23:22.302 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:14.624626+0000", 2024-09-13T11:23:22.302 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00020203699999999999 2024-09-13T11:23:22.302 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:22.302 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:22.302 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:23:22.303 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:14.627248+0000", 2024-09-13T11:23:22.303 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0026215679999999999 2024-09-13T11:23:22.303 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:22.303 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:22.303 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:22.303 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:14.691646+0000", 2024-09-13T11:23:22.303 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.064397839999999998 2024-09-13T11:23:22.303 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:22.303 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:22.303 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:23:22.303 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:14.691696+0000", 2024-09-13T11:23:22.303 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.9938999999999997e-05 2024-09-13T11:23:22.303 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:22.304 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:22.304 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:23:22.304 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:14.691729+0000", 2024-09-13T11:23:22.304 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.3531000000000001e-05 2024-09-13T11:23:22.304 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:22.304 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:23:22.304 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:22.304 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:22.304 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:22.304 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:38991 3.4 3:23410d4c:::benchmark_data_smithi177_31116_object38990:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e63)", 2024-09-13T11:23:22.304 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:15.188590+0000", 2024-09-13T11:23:22.304 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 7.0191182039999997, 2024-09-13T11:23:22.305 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.066209300999999998, 2024-09-13T11:23:22.305 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:23:22.305 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:23:22.305 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:22.305 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:23:22.308 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:23:22.308 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:22.308 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 38991 2024-09-13T11:23:22.308 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:22.308 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:23:22.308 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:22.308 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:23:22.308 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:15.188590+0000", 2024-09-13T11:23:22.309 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:22.309 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:22.309 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:22.309 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:23:22.309 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:15.188590+0000", 2024-09-13T11:23:22.309 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:22.309 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:22.309 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:22.309 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:23:22.309 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:15.188593+0000", 2024-09-13T11:23:22.309 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.965e-06 2024-09-13T11:23:22.309 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:22.309 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:22.310 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:23:22.310 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:15.188635+0000", 2024-09-13T11:23:22.310 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.2519999999999999e-05 2024-09-13T11:23:22.310 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:22.310 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:22.310 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:23:22.310 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:15.188653+0000", 2024-09-13T11:23:22.310 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.7575999999999999e-05 2024-09-13T11:23:22.310 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:22.310 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:22.310 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:23:22.310 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:15.188662+0000", 2024-09-13T11:23:22.310 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.7530000000000007e-06 2024-09-13T11:23:22.310 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:22.311 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:22.311 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:23:22.311 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:15.188770+0000", 2024-09-13T11:23:22.311 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000108417 2024-09-13T11:23:22.311 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:22.311 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:22.311 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:23:22.311 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:15.188942+0000", 2024-09-13T11:23:22.311 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000171485 2024-09-13T11:23:22.311 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:22.311 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:22.311 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 12", 2024-09-13T11:23:22.312 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:15.189070+0000", 2024-09-13T11:23:22.312 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000128477 2024-09-13T11:23:22.312 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:22.312 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:22.312 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:23:22.312 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:15.196172+0000", 2024-09-13T11:23:22.312 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0071015990000000001 2024-09-13T11:23:22.312 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:22.312 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:22.312 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:22.313 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:15.254654+0000", 2024-09-13T11:23:22.313 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.058482264999999999 2024-09-13T11:23:22.313 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:22.313 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:22.313 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:23:22.313 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:15.254737+0000", 2024-09-13T11:23:22.313 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.2515999999999994e-05 2024-09-13T11:23:22.313 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:22.313 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:22.313 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:23:22.313 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:15.254799+0000", 2024-09-13T11:23:22.313 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.2728e-05 2024-09-13T11:23:22.313 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:22.314 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:23:22.314 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:22.314 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:22.314 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:22.314 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:40441 3.4 3:22d6107f:::benchmark_data_smithi177_31116_object40440:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-13T11:23:22.314 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:16.194725+0000", 2024-09-13T11:23:22.314 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 6.0129828060000001, 2024-09-13T11:23:22.314 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.078031000000000003, 2024-09-13T11:23:22.314 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:23:22.314 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:23:22.314 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:22.314 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:23:22.314 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:23:22.314 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:22.315 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 40441 2024-09-13T11:23:22.315 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:22.315 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:23:22.315 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:22.315 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:23:22.315 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:16.194725+0000", 2024-09-13T11:23:22.315 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:22.315 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:22.315 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:22.315 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:23:22.316 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:16.194725+0000", 2024-09-13T11:23:22.316 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:22.316 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:22.316 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:22.316 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:23:22.316 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:16.194727+0000", 2024-09-13T11:23:22.317 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.607e-06 2024-09-13T11:23:22.317 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:22.317 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:22.317 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:23:22.317 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:16.194776+0000", 2024-09-13T11:23:22.317 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.9268000000000003e-05 2024-09-13T11:23:22.317 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:22.317 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:22.317 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:23:22.317 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:16.194789+0000", 2024-09-13T11:23:22.318 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.3142000000000001e-05 2024-09-13T11:23:22.318 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:22.318 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:22.318 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:23:22.318 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:16.194798+0000", 2024-09-13T11:23:22.318 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.6629999999999999e-06 2024-09-13T11:23:22.318 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:22.318 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:22.318 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:23:22.318 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:16.195669+0000", 2024-09-13T11:23:22.318 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00087054600000000004 2024-09-13T11:23:22.318 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:22.318 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:22.319 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:23:22.319 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:16.195807+0000", 2024-09-13T11:23:22.319 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00013808700000000001 2024-09-13T11:23:22.319 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:22.319 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:22.319 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 12", 2024-09-13T11:23:22.319 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:16.195942+0000", 2024-09-13T11:23:22.319 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00013483500000000001 2024-09-13T11:23:22.319 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:22.319 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:22.320 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:23:22.320 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:16.197510+0000", 2024-09-13T11:23:22.320 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0015679979999999999 2024-09-13T11:23:22.320 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:22.320 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:22.320 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:22.320 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:16.272690+0000", 2024-09-13T11:23:22.320 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.075180012000000004 2024-09-13T11:23:22.320 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:22.320 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:22.320 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:23:22.320 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:16.272724+0000", 2024-09-13T11:23:22.320 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.4755000000000002e-05 2024-09-13T11:23:22.321 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:22.321 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:22.321 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:23:22.321 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:16.272756+0000", 2024-09-13T11:23:22.321 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.2087e-05 2024-09-13T11:23:22.321 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:22.321 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:23:22.321 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:22.321 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:22.321 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:22.321 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:40465 3.4 3:25013390:::benchmark_data_smithi177_31116_object40464:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-13T11:23:22.321 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:16.216068+0000", 2024-09-13T11:23:22.322 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 5.991640071, 2024-09-13T11:23:22.322 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.069806731999999996, 2024-09-13T11:23:22.322 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:23:22.322 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:23:22.322 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:22.322 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:23:22.322 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:23:22.322 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:22.322 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 40465 2024-09-13T11:23:22.322 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:22.322 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:23:22.323 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:22.323 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:23:22.323 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:16.216068+0000", 2024-09-13T11:23:22.323 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:22.323 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:22.323 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:22.323 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:23:22.323 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:16.216068+0000", 2024-09-13T11:23:22.323 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:22.323 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:22.323 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:22.323 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:23:22.323 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:16.216070+0000", 2024-09-13T11:23:22.324 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.334e-06 2024-09-13T11:23:22.324 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:22.324 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:22.324 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:23:22.324 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:16.216102+0000", 2024-09-13T11:23:22.324 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.1993999999999999e-05 2024-09-13T11:23:22.324 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:22.324 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:22.324 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:23:22.325 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:16.216115+0000", 2024-09-13T11:23:22.325 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.2948999999999999e-05 2024-09-13T11:23:22.325 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:22.325 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:22.325 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:23:22.325 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:16.216123+0000", 2024-09-13T11:23:22.325 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.8460000000000008e-06 2024-09-13T11:23:22.325 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:22.325 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:22.325 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:23:22.326 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:16.216187+0000", 2024-09-13T11:23:22.326 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.3361999999999994e-05 2024-09-13T11:23:22.326 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:22.326 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:22.326 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:23:22.326 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:16.216355+0000", 2024-09-13T11:23:22.326 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000168197 2024-09-13T11:23:22.326 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:22.326 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:22.326 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 12", 2024-09-13T11:23:22.326 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:16.216538+0000", 2024-09-13T11:23:22.326 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000182733 2024-09-13T11:23:22.326 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:22.326 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:22.327 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:23:22.327 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:16.218859+0000", 2024-09-13T11:23:22.327 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0023214720000000002 2024-09-13T11:23:22.327 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:22.327 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:22.327 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:22.327 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:16.285782+0000", 2024-09-13T11:23:22.327 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.066923273000000005 2024-09-13T11:23:22.327 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:22.327 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:22.327 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:23:22.327 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:16.285841+0000", 2024-09-13T11:23:22.327 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.8557000000000002e-05 2024-09-13T11:23:22.328 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:22.328 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:22.328 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:23:22.328 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:16.285875+0000", 2024-09-13T11:23:22.328 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.4014999999999999e-05 2024-09-13T11:23:22.328 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:22.328 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:23:22.328 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:22.328 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:22.328 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:22.328 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop(client.4665.0:41257 3.b e65/40 3:d342dab2:::benchmark_data_smithi177_31116_object41256:head v 65'2613, mlcod=65'2610)", 2024-09-13T11:23:22.328 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:16.800467+0000", 2024-09-13T11:23:22.328 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 5.4072412380000001, 2024-09-13T11:23:22.329 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.095071575000000005, 2024-09-13T11:23:22.329 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:23:22.329 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:23:22.329 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:22.329 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:23:22.329 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:22.329 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:23:22.329 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:16.800467+0000", 2024-09-13T11:23:22.329 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:22.329 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:22.329 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:22.329 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:23:22.330 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:16.800467+0000", 2024-09-13T11:23:22.330 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:22.330 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:22.330 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:22.330 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:23:22.330 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:16.800469+0000", 2024-09-13T11:23:22.330 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.2340000000000001e-06 2024-09-13T11:23:22.330 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:22.330 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:22.330 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:23:22.330 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:16.800495+0000", 2024-09-13T11:23:22.331 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.5434000000000001e-05 2024-09-13T11:23:22.331 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:22.331 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:22.331 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:23:22.331 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:16.893496+0000", 2024-09-13T11:23:22.331 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.093001167999999995 2024-09-13T11:23:22.331 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:22.331 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:22.331 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:23:22.331 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:16.893503+0000", 2024-09-13T11:23:22.331 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.029e-06 2024-09-13T11:23:22.331 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:22.331 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:22.332 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:23:22.332 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:16.893569+0000", 2024-09-13T11:23:22.332 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.6291999999999995e-05 2024-09-13T11:23:22.332 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:22.332 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:22.332 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:23:22.332 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:16.893595+0000", 2024-09-13T11:23:22.332 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.5377000000000001e-05 2024-09-13T11:23:22.332 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:22.332 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:22.332 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:23:22.332 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:16.895501+0000", 2024-09-13T11:23:22.332 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.001906865 2024-09-13T11:23:22.332 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:22.333 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:22.333 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:23:22.333 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:16.895539+0000", 2024-09-13T11:23:22.333 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.7175999999999998e-05 2024-09-13T11:23:22.333 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:22.333 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:23:22.333 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:22.333 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:22.333 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:22.333 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop(client.4665.0:45770 3.b e68/40 3:d2ece503:::benchmark_data_smithi177_31116_object45769:head v 68'2878, mlcod=68'2875)", 2024-09-13T11:23:22.333 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:20.925073+0000", 2024-09-13T11:23:22.333 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 1.282635258, 2024-09-13T11:23:22.333 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.14884730400000001, 2024-09-13T11:23:22.334 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:23:22.334 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:23:22.334 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:22.334 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:23:22.334 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:22.334 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:23:22.334 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:20.925073+0000", 2024-09-13T11:23:22.334 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:22.334 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:22.334 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:22.334 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:23:22.335 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:20.925073+0000", 2024-09-13T11:23:22.335 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:22.335 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:22.335 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:22.335 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:23:22.335 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:20.925075+0000", 2024-09-13T11:23:22.335 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.9360000000000002e-06 2024-09-13T11:23:22.335 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:22.335 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:22.335 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:23:22.335 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:20.925100+0000", 2024-09-13T11:23:22.335 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.5602e-05 2024-09-13T11:23:22.336 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:22.336 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:22.336 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:23:22.336 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:21.070362+0000", 2024-09-13T11:23:22.336 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.14526132 2024-09-13T11:23:22.336 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:22.336 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:22.336 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:23:22.336 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:21.070375+0000", 2024-09-13T11:23:22.336 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.3636e-05 2024-09-13T11:23:22.336 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:22.337 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:22.337 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:23:22.337 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:21.070523+0000", 2024-09-13T11:23:22.337 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00014711999999999999 2024-09-13T11:23:22.337 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:22.337 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:22.337 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:23:22.337 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:21.070577+0000", 2024-09-13T11:23:22.337 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.3987000000000003e-05 2024-09-13T11:23:22.337 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:22.337 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:22.338 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:23:22.338 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:21.073840+0000", 2024-09-13T11:23:22.338 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0032633050000000002 2024-09-13T11:23:22.338 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:22.338 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:22.338 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:23:22.338 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:21.073920+0000", 2024-09-13T11:23:22.338 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.0397999999999998e-05 2024-09-13T11:23:22.338 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:22.338 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:23:22.338 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:22.338 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:22.338 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:23:22.339 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-09-13T11:23:22.340 INFO:teuthology.orchestra.run.smithi001.stdout:osd.0: {} 2024-09-13T11:23:22.340 INFO:teuthology.orchestra.run.smithi001.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T11:23:22.341 INFO:teuthology.orchestra.run.smithi001.stdout:osd.1: {} 2024-09-13T11:23:22.341 INFO:teuthology.orchestra.run.smithi001.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T11:23:22.342 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-13T11:23:22.342 INFO:teuthology.orchestra.run.smithi001.stdout:osd.2: {} 2024-09-13T11:23:22.342 INFO:teuthology.orchestra.run.smithi001.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T11:23:22.343 INFO:teuthology.orchestra.run.smithi001.stdout:osd.3: {} 2024-09-13T11:23:22.343 INFO:teuthology.orchestra.run.smithi001.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T11:23:22.343 INFO:teuthology.orchestra.run.smithi001.stdout:osd.4: {} 2024-09-13T11:23:22.344 INFO:teuthology.orchestra.run.smithi001.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T11:23:22.344 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-09-13T11:23:22.334+0000 7f97befc9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26315) UID: 0 2024-09-13T11:23:22.353 INFO:teuthology.orchestra.run.smithi001.stdout:osd.5: {} 2024-09-13T11:23:22.353 INFO:teuthology.orchestra.run.smithi001.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T11:23:22.381 INFO:teuthology.orchestra.run.smithi001.stdout:osd.6: {} 2024-09-13T11:23:22.381 INFO:teuthology.orchestra.run.smithi001.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T11:23:22.408 INFO:teuthology.orchestra.run.smithi001.stdout:osd.7: {} 2024-09-13T11:23:22.408 INFO:teuthology.orchestra.run.smithi001.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T11:23:22.429 INFO:teuthology.orchestra.run.smithi001.stdout:osd.8: {} 2024-09-13T11:23:22.429 INFO:teuthology.orchestra.run.smithi001.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-13T11:23:22.440 INFO:tasks.ceph.osd.15.smithi177.stderr:2024-09-13T11:23:22.437+0000 7f9d6ce77640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26578) UID: 0 2024-09-13T11:23:22.458 INFO:teuthology.orchestra.run.smithi001.stdout:osd.9: {} 2024-09-13T11:23:22.458 INFO:teuthology.orchestra.run.smithi001.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-13T11:23:22.471 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-13T11:23:22.471 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:23:22.485 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-13T11:23:22.488 INFO:teuthology.orchestra.run.smithi001.stdout:osd.10: {} 2024-09-13T11:23:22.488 INFO:teuthology.orchestra.run.smithi001.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-13T11:23:22.517 INFO:teuthology.orchestra.run.smithi001.stdout:osd.11: {} 2024-09-13T11:23:22.517 INFO:teuthology.orchestra.run.smithi001.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-13T11:23:22.541 INFO:teuthology.orchestra.run.smithi001.stdout:osd.12: {} 2024-09-13T11:23:22.542 INFO:teuthology.orchestra.run.smithi001.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-13T11:23:22.542 INFO:tasks.ceph.osd.2.smithi162.stderr:2024-09-13T11:23:22.538+0000 7fe92b35a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26313) UID: 0 2024-09-13T11:23:22.570 INFO:teuthology.orchestra.run.smithi001.stdout:osd.13: {} 2024-09-13T11:23:22.570 INFO:teuthology.orchestra.run.smithi001.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-13T11:23:22.606 INFO:teuthology.orchestra.run.smithi001.stdout:osd.14: {} 2024-09-13T11:23:22.606 INFO:teuthology.orchestra.run.smithi001.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-13T11:23:22.609 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-13T11:23:22.609 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:23:22.625 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-13T11:23:22.638 INFO:teuthology.orchestra.run.smithi001.stdout:osd.15: {} 2024-09-13T11:23:22.638 INFO:teuthology.orchestra.run.smithi001.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-13T11:23:22.642 INFO:tasks.ceph.osd.11.smithi177.stderr:2024-09-13T11:23:22.637+0000 7f24792c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26580) UID: 0 2024-09-13T11:23:22.744 INFO:tasks.ceph.osd.14.smithi162.stderr:2024-09-13T11:23:22.738+0000 7f235c0db640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26319) UID: 0 2024-09-13T11:23:22.772 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-13T11:23:22.772 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:23:22.787 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-13T11:23:22.844 INFO:tasks.ceph.osd.2.smithi162.stderr:2024-09-13T11:23:22.838+0000 7fe92b35a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26313) UID: 0 2024-09-13T11:23:22.904 INFO:teuthology.orchestra.run.smithi001.stderr:nodeep-scrub is unset 2024-09-13T11:23:22.908 INFO:tasks.ceph.osd.12.smithi001.stderr:2024-09-13T11:23:22.900+0000 7f891f10f640 -1 osd.12 70 handle_osd_map injecting map crc failure 2024-09-13T11:23:22.920 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-13T11:23:22.920 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:23:22.928 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15] out_osds: [] dead_osds: [] live_osds: [0, 4, 8, 12, 5, 9, 13, 2, 6, 10, 14, 3, 7, 11, 15, 1] 2024-09-13T11:23:22.929 INFO:tasks.thrashosds.thrasher:choose_action: min_in 2 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-13T11:23:22.929 INFO:tasks.thrashosds.thrasher:inject_pause on osd.15 2024-09-13T11:23:22.929 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-13T11:23:22.929 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-13T11:23:22.929 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok config set filestore_inject_stall 3 2024-09-13T11:23:22.946 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-13T11:23:22.947 INFO:tasks.ceph.osd.4.smithi001.stderr:2024-09-13T11:23:22.940+0000 7ff8e0218640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26310) UID: 0 2024-09-13T11:23:23.003 INFO:tasks.radosbench.radosbench.0.smithi177.stdout: 56 16 45818 45802 51.1126 0.5 0.00392015 0.0189333 2024-09-13T11:23:23.017 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-09-13T11:23:23.010+0000 7f97b39f6640 -1 osd.6 70 handle_osd_map injecting map crc failure 2024-09-13T11:23:23.046 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-13T11:23:23.040+0000 7f6a68775640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26307) UID: 0 2024-09-13T11:23:23.092 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-09-13T11:23:23.092 INFO:teuthology.orchestra.run.smithi177.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-09-13T11:23:23.093 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-09-13T11:23:23.138 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-13T11:23:23.138 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:23:23.147 INFO:tasks.ceph.osd.11.smithi177.stderr:2024-09-13T11:23:23.141+0000 7f24792c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26580) UID: 0 2024-09-13T11:23:23.161 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-13T11:23:23.247 INFO:tasks.ceph.osd.3.smithi177.stderr:2024-09-13T11:23:23.245+0000 7f10adba4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26574) UID: 0 2024-09-13T11:23:23.350 INFO:tasks.ceph.osd.11.smithi177.stderr:2024-09-13T11:23:23.345+0000 7f24792c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26580) UID: 0 2024-09-13T11:23:23.369 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-13T11:23:23.369 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:23:23.397 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-09-13T11:23:23.449 INFO:tasks.ceph.osd.8.smithi001.stderr:2024-09-13T11:23:23.444+0000 7fb9b48ae640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26312) UID: 0 2024-09-13T11:23:23.551 INFO:tasks.ceph.osd.8.smithi001.stderr:2024-09-13T11:23:23.544+0000 7fb9b48ae640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26312) UID: 0 2024-09-13T11:23:23.554 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-13T11:23:23.554 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:23:23.578 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-09-13T11:23:23.664 INFO:tasks.ceph.osd.8.smithi001.stderr:2024-09-13T11:23:23.660+0000 7fb9b48ae640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26312) UID: 0 2024-09-13T11:23:23.752 INFO:tasks.ceph.osd.15.smithi177.stderr:2024-09-13T11:23:23.749+0000 7f9d6ce77640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26578) UID: 0 2024-09-13T11:23:23.768 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-13T11:23:23.768 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:23:23.791 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-09-13T11:23:23.853 INFO:tasks.ceph.osd.2.smithi162.stderr:2024-09-13T11:23:23.846+0000 7fe92b35a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26313) UID: 0 2024-09-13T11:23:23.954 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-09-13T11:23:23.950+0000 7f97befc9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26315) UID: 0 2024-09-13T11:23:23.997 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-13T11:23:23.998 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:23:24.003 INFO:tasks.radosbench.radosbench.0.smithi177.stdout: 57 16 47079 47063 51.5984 78.8125 0.00629898 0.0193743 2024-09-13T11:23:24.024 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_ops_in_flight 2024-09-13T11:23:24.055 INFO:tasks.ceph.osd.2.smithi162.stderr:2024-09-13T11:23:24.050+0000 7fe92b35a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26313) UID: 0 2024-09-13T11:23:24.156 INFO:tasks.ceph.osd.14.smithi162.stderr:2024-09-13T11:23:24.150+0000 7f235c0db640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26319) UID: 0 2024-09-13T11:23:24.181 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-13T11:23:24.181 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:23:24.202 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_blocked_ops 2024-09-13T11:23:24.257 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-09-13T11:23:24.250+0000 7f97befc9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26315) UID: 0 2024-09-13T11:23:24.351 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-13T11:23:24.351 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:23:24.358 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-13T11:23:24.352+0000 7f6a68775640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26307) UID: 0 2024-09-13T11:23:24.371 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_historic_ops 2024-09-13T11:23:24.458 INFO:tasks.ceph.osd.14.smithi162.stderr:2024-09-13T11:23:24.454+0000 7f235c0db640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26319) UID: 0 2024-09-13T11:23:24.540 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-13T11:23:24.540 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:23:24.556 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-13T11:23:24.559 INFO:tasks.ceph.osd.2.smithi162.stderr:2024-09-13T11:23:24.554+0000 7fe92b35a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26313) UID: 0 2024-09-13T11:23:24.660 INFO:tasks.ceph.osd.8.smithi001.stderr:2024-09-13T11:23:24.656+0000 7fb9b48ae640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26312) UID: 0 2024-09-13T11:23:24.661 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T11:23:24.700 INFO:teuthology.orchestra.run.smithi092.stdout:ERROR: (22) Invalid argument 2024-09-13T11:23:24.700 INFO:teuthology.orchestra.run.smithi092.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:23:24.718 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-13T11:23:24.761 INFO:tasks.ceph.osd.8.smithi001.stderr:2024-09-13T11:23:24.756+0000 7fb9b48ae640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26312) UID: 0 2024-09-13T11:23:24.831 INFO:teuthology.orchestra.run.smithi001.stdout:osd.0: {} 2024-09-13T11:23:24.831 INFO:teuthology.orchestra.run.smithi001.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T11:23:24.862 INFO:teuthology.orchestra.run.smithi001.stdout:osd.1: {} 2024-09-13T11:23:24.863 INFO:teuthology.orchestra.run.smithi001.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T11:23:24.863 INFO:tasks.ceph.osd.8.smithi001.stderr:2024-09-13T11:23:24.856+0000 7fb9b48ae640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26312) UID: 0 2024-09-13T11:23:24.867 INFO:teuthology.orchestra.run.smithi092.stdout:ERROR: (22) Invalid argument 2024-09-13T11:23:24.867 INFO:teuthology.orchestra.run.smithi092.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:23:24.887 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-13T11:23:24.889 INFO:teuthology.orchestra.run.smithi001.stdout:osd.2: {} 2024-09-13T11:23:24.889 INFO:teuthology.orchestra.run.smithi001.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T11:23:24.913 INFO:teuthology.orchestra.run.smithi001.stdout:osd.3: {} 2024-09-13T11:23:24.913 INFO:teuthology.orchestra.run.smithi001.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T11:23:24.933 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-13T11:23:24.937 INFO:teuthology.orchestra.run.smithi001.stdout:osd.4: {} 2024-09-13T11:23:24.937 INFO:teuthology.orchestra.run.smithi001.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T11:23:24.963 INFO:tasks.ceph.osd.8.smithi001.stderr:2024-09-13T11:23:24.956+0000 7fb9b48ae640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26312) UID: 0 2024-09-13T11:23:24.967 INFO:teuthology.orchestra.run.smithi001.stdout:osd.5: {} 2024-09-13T11:23:24.967 INFO:teuthology.orchestra.run.smithi001.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T11:23:24.995 INFO:teuthology.orchestra.run.smithi001.stdout:osd.6: {} 2024-09-13T11:23:24.995 INFO:teuthology.orchestra.run.smithi001.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T11:23:25.004 INFO:tasks.radosbench.radosbench.0.smithi177.stdout: 58 16 48396 48380 52.1278 82.3125 0.00736094 0.0191683 2024-09-13T11:23:25.021 INFO:teuthology.orchestra.run.smithi001.stdout:osd.7: {} 2024-09-13T11:23:25.021 INFO:teuthology.orchestra.run.smithi001.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T11:23:25.024 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-09-13T11:23:25.024 INFO:teuthology.orchestra.run.smithi092.stdout: "size": 20, 2024-09-13T11:23:25.024 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 600, 2024-09-13T11:23:25.024 INFO:teuthology.orchestra.run.smithi092.stdout: "ops": [] 2024-09-13T11:23:25.024 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-09-13T11:23:25.043 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-09-13T11:23:25.049 INFO:teuthology.orchestra.run.smithi001.stdout:osd.8: {} 2024-09-13T11:23:25.049 INFO:teuthology.orchestra.run.smithi001.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-13T11:23:25.075 INFO:teuthology.orchestra.run.smithi001.stdout:osd.9: {} 2024-09-13T11:23:25.075 INFO:teuthology.orchestra.run.smithi001.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-13T11:23:25.093 INFO:tasks.ceph.osd.5.smithi092.stderr:2024-09-13T11:23:25.093+0000 7f6f0b705640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26346) UID: 0 2024-09-13T11:23:25.102 INFO:teuthology.orchestra.run.smithi001.stdout:osd.10: {} 2024-09-13T11:23:25.102 INFO:teuthology.orchestra.run.smithi001.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-13T11:23:25.128 INFO:teuthology.orchestra.run.smithi001.stdout:osd.11: {} 2024-09-13T11:23:25.129 INFO:teuthology.orchestra.run.smithi001.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-13T11:23:25.156 INFO:teuthology.orchestra.run.smithi001.stdout:osd.12: {} 2024-09-13T11:23:25.157 INFO:teuthology.orchestra.run.smithi001.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-13T11:23:25.165 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-13T11:23:25.160+0000 7f6a68775640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26307) UID: 0 2024-09-13T11:23:25.184 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-09-13T11:23:25.184 INFO:teuthology.orchestra.run.smithi092.stdout: "ops": [], 2024-09-13T11:23:25.185 INFO:teuthology.orchestra.run.smithi092.stdout: "num_ops": 0 2024-09-13T11:23:25.185 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-09-13T11:23:25.194 INFO:teuthology.orchestra.run.smithi001.stdout:osd.13: {} 2024-09-13T11:23:25.194 INFO:teuthology.orchestra.run.smithi001.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-13T11:23:25.204 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-09-13T11:23:25.220 INFO:teuthology.orchestra.run.smithi001.stdout:osd.14: {} 2024-09-13T11:23:25.220 INFO:teuthology.orchestra.run.smithi001.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-13T11:23:25.251 INFO:teuthology.orchestra.run.smithi001.stdout:osd.15: {} 2024-09-13T11:23:25.251 INFO:teuthology.orchestra.run.smithi001.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-13T11:23:25.266 INFO:tasks.ceph.osd.15.smithi177.stderr:2024-09-13T11:23:25.261+0000 7f9d6ce77640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26578) UID: 0 2024-09-13T11:23:25.367 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-13T11:23:25.360+0000 7f6a68775640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26307) UID: 0 2024-09-13T11:23:25.384 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-09-13T11:23:25.384 INFO:teuthology.orchestra.run.smithi092.stdout: "ops": [], 2024-09-13T11:23:25.384 INFO:teuthology.orchestra.run.smithi092.stdout: "complaint_time": 30, 2024-09-13T11:23:25.384 INFO:teuthology.orchestra.run.smithi092.stdout: "num_blocked_ops": 0 2024-09-13T11:23:25.384 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-09-13T11:23:25.403 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-09-13T11:23:25.468 INFO:tasks.ceph.osd.8.smithi001.stderr:2024-09-13T11:23:25.464+0000 7fb9b48ae640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26312) UID: 0 2024-09-13T11:23:25.532 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-09-13T11:23:25.532 INFO:teuthology.orchestra.run.smithi092.stdout: "size": 20, 2024-09-13T11:23:25.532 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 600, 2024-09-13T11:23:25.532 INFO:teuthology.orchestra.run.smithi092.stdout: "ops": [ 2024-09-13T11:23:25.532 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:25.532 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop(client.4344.0:5 2.2 e27/25 2:5cea7035:::rbd_info:head v 27'1, mlcod=0'0)", 2024-09-13T11:23:25.532 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:15.250889+0000", 2024-09-13T11:23:25.532 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 70.279436321000006, 2024-09-13T11:23:25.533 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.001990627, 2024-09-13T11:23:25.533 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:23:25.533 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:23:25.533 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:25.533 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:23:25.533 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:25.533 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:23:25.533 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:15.250889+0000", 2024-09-13T11:23:25.533 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:25.533 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:25.533 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:25.533 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:23:25.534 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:15.250889+0000", 2024-09-13T11:23:25.534 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:25.534 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:25.534 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:25.534 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:23:25.534 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:15.250890+0000", 2024-09-13T11:23:25.534 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.153e-06 2024-09-13T11:23:25.534 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:25.534 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:25.534 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:23:25.534 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:15.250907+0000", 2024-09-13T11:23:25.534 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.6821e-05 2024-09-13T11:23:25.534 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:25.535 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:25.535 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:23:25.535 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:15.250991+0000", 2024-09-13T11:23:25.535 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.4241999999999997e-05 2024-09-13T11:23:25.535 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:25.535 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:25.535 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:23:25.535 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:15.251023+0000", 2024-09-13T11:23:25.535 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.1780999999999999e-05 2024-09-13T11:23:25.535 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:25.535 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:25.535 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:23:25.535 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:15.251178+0000", 2024-09-13T11:23:25.536 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000155064 2024-09-13T11:23:25.536 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:25.536 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:25.536 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:23:25.536 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:15.251215+0000", 2024-09-13T11:23:25.536 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.7160000000000003e-05 2024-09-13T11:23:25.536 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:25.536 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:25.536 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:23:25.536 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:15.252759+0000", 2024-09-13T11:23:25.536 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0015440110000000001 2024-09-13T11:23:25.536 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:25.536 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:25.537 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:23:25.537 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:15.252880+0000", 2024-09-13T11:23:25.537 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000120395 2024-09-13T11:23:25.537 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:25.537 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:23:25.537 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:25.537 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:25.537 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:25.538 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop(client.4344.0:7 2.2 e28/25 2:5cea7035:::rbd_info:head v 28'2, mlcod=0'0)", 2024-09-13T11:23:25.538 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:16.268518+0000", 2024-09-13T11:23:25.538 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 69.261806720999999, 2024-09-13T11:23:25.538 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0017167040000000001, 2024-09-13T11:23:25.538 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:23:25.538 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:23:25.538 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:25.538 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:23:25.539 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:25.539 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:23:25.539 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:16.268518+0000", 2024-09-13T11:23:25.539 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:25.539 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:25.539 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:25.539 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:23:25.539 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:16.268518+0000", 2024-09-13T11:23:25.539 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:25.539 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:25.540 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:25.540 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:23:25.540 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:16.268519+0000", 2024-09-13T11:23:25.540 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.1299999999999999e-07 2024-09-13T11:23:25.540 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:25.540 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:25.540 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:23:25.540 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:16.268539+0000", 2024-09-13T11:23:25.540 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.0214999999999999e-05 2024-09-13T11:23:25.540 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:25.541 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:25.541 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:23:25.541 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:16.268622+0000", 2024-09-13T11:23:25.541 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.2395e-05 2024-09-13T11:23:25.541 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:25.541 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:25.541 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:23:25.541 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:16.268638+0000", 2024-09-13T11:23:25.541 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.6189e-05 2024-09-13T11:23:25.541 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:25.541 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:25.541 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:23:25.541 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:16.268740+0000", 2024-09-13T11:23:25.542 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00010164 2024-09-13T11:23:25.542 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:25.542 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:25.542 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:23:25.542 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:16.268783+0000", 2024-09-13T11:23:25.542 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.2988999999999998e-05 2024-09-13T11:23:25.542 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:25.542 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:25.542 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:23:25.543 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:16.270121+0000", 2024-09-13T11:23:25.543 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00133829 2024-09-13T11:23:25.543 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:25.543 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:25.543 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:23:25.543 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:16.270235+0000", 2024-09-13T11:23:25.543 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000114273 2024-09-13T11:23:25.543 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:25.543 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:23:25.543 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:25.543 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:25.543 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:25.543 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop(client.4665.0:48473 3.19 e71/70 3:9b87d3f3:::benchmark_data_smithi177_31116_object48472:head v 71'2882, mlcod=44'262)", 2024-09-13T11:23:25.544 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:25.081737+0000", 2024-09-13T11:23:25.544 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 0.448588559, 2024-09-13T11:23:25.544 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.002851705, 2024-09-13T11:23:25.544 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:23:25.544 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:23:25.544 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:25.544 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:23:25.544 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:25.544 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:23:25.544 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:25.081737+0000", 2024-09-13T11:23:25.544 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:25.544 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:25.544 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:25.545 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:23:25.545 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:25.081737+0000", 2024-09-13T11:23:25.545 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:25.545 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:25.545 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:25.545 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:23:25.545 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:25.081739+0000", 2024-09-13T11:23:25.545 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.3010000000000001e-06 2024-09-13T11:23:25.545 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:25.545 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:25.545 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:23:25.545 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:25.081769+0000", 2024-09-13T11:23:25.545 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.0385000000000001e-05 2024-09-13T11:23:25.546 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:25.546 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:25.546 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:23:25.546 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:25.081808+0000", 2024-09-13T11:23:25.546 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.8287000000000001e-05 2024-09-13T11:23:25.546 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:25.546 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:25.546 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:23:25.546 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:25.081817+0000", 2024-09-13T11:23:25.546 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.9600000000000006e-06 2024-09-13T11:23:25.546 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:25.546 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:25.546 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:23:25.547 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:25.081876+0000", 2024-09-13T11:23:25.547 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.9094e-05 2024-09-13T11:23:25.547 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:25.547 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:25.547 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:23:25.547 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:25.081900+0000", 2024-09-13T11:23:25.547 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.3978e-05 2024-09-13T11:23:25.547 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:25.547 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:25.547 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:23:25.547 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:25.084539+0000", 2024-09-13T11:23:25.547 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0026395009999999998 2024-09-13T11:23:25.547 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:25.547 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:25.548 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:23:25.548 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:25.084588+0000", 2024-09-13T11:23:25.548 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.9199000000000001e-05 2024-09-13T11:23:25.548 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:25.548 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:23:25.548 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:25.548 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:25.548 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:25.548 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "MOSDPGPush(3.19 71/70 [PushOp(3:9a16236c:::benchmark_data_smithi177_31116_object4235:head, version: 44'265, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:9a16236c:::benchmark_data_smithi177_31116_object4235:head@44'265, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:23:25.549 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:25.092876+0000", 2024-09-13T11:23:25.549 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 0.43744926899999997, 2024-09-13T11:23:25.549 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.001076498, 2024-09-13T11:23:25.549 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:23:25.550 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:23:25.550 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-09-13T11:23:25.550 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:23:25.550 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:25.550 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:23:25.550 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:25.092876+0000", 2024-09-13T11:23:25.550 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:25.550 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:25.550 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:25.550 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:23:25.550 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:25.092876+0000", 2024-09-13T11:23:25.550 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:25.550 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:25.551 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:25.551 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:23:25.551 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:25.092892+0000", 2024-09-13T11:23:25.551 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.5858999999999998e-05 2024-09-13T11:23:25.551 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:25.551 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:25.551 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:23:25.551 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:25.092963+0000", 2024-09-13T11:23:25.551 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.1583999999999998e-05 2024-09-13T11:23:25.551 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:25.551 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:25.551 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:23:25.551 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:25.093010+0000", 2024-09-13T11:23:25.551 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.6193999999999999e-05 2024-09-13T11:23:25.551 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:25.551 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:25.552 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:23:25.552 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:25.093024+0000", 2024-09-13T11:23:25.552 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.4357e-05 2024-09-13T11:23:25.552 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:25.552 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:25.552 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:23:25.552 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:25.093122+0000", 2024-09-13T11:23:25.552 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.7843000000000005e-05 2024-09-13T11:23:25.552 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:25.552 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:25.552 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:23:25.552 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:25.093138+0000", 2024-09-13T11:23:25.552 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.647e-05 2024-09-13T11:23:25.553 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:25.553 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:25.553 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:23:25.553 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:25.093952+0000", 2024-09-13T11:23:25.553 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00081419100000000002 2024-09-13T11:23:25.553 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:25.553 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:23:25.553 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:25.553 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:25.553 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:25.553 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop(client.4665.0:48499 3.19 e71/70 3:9a22555d:::benchmark_data_smithi177_31116_object48498:head v 71'2883, mlcod=44'264)", 2024-09-13T11:23:25.553 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:25.108146+0000", 2024-09-13T11:23:25.553 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 0.42217887300000001, 2024-09-13T11:23:25.554 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0048232129999999998, 2024-09-13T11:23:25.554 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:23:25.554 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:23:25.554 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:25.554 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:23:25.554 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:25.554 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:23:25.554 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:25.108146+0000", 2024-09-13T11:23:25.554 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:25.554 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:25.554 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:25.554 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:23:25.554 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:25.108146+0000", 2024-09-13T11:23:25.554 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:25.555 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:25.555 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:25.555 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:23:25.555 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:25.108149+0000", 2024-09-13T11:23:25.555 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.2180000000000002e-06 2024-09-13T11:23:25.555 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:25.555 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:25.555 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:23:25.555 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:25.108207+0000", 2024-09-13T11:23:25.555 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.8706999999999999e-05 2024-09-13T11:23:25.555 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:25.555 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:25.555 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:23:25.555 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:25.108243+0000", 2024-09-13T11:23:25.556 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.5509000000000002e-05 2024-09-13T11:23:25.556 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:25.556 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:25.556 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:23:25.556 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:25.108253+0000", 2024-09-13T11:23:25.556 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.8660000000000001e-06 2024-09-13T11:23:25.556 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:25.556 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:25.556 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:23:25.556 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:25.108315+0000", 2024-09-13T11:23:25.556 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.2032000000000005e-05 2024-09-13T11:23:25.556 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:25.556 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:25.556 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:23:25.556 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:25.108342+0000", 2024-09-13T11:23:25.556 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.6971999999999999e-05 2024-09-13T11:23:25.557 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:25.557 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:25.557 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:23:25.557 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:25.112927+0000", 2024-09-13T11:23:25.557 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0045848720000000003 2024-09-13T11:23:25.557 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:25.557 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:25.557 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:23:25.557 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:25.112970+0000", 2024-09-13T11:23:25.557 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.3037000000000001e-05 2024-09-13T11:23:25.557 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:25.557 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:23:25.557 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:25.557 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:25.558 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:25.558 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop(client.4665.0:48504 3.19 e71/70 3:9ebd3c6a:::benchmark_data_smithi177_31116_object48503:head v 71'2884, mlcod=44'264)", 2024-09-13T11:23:25.558 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:25.110909+0000", 2024-09-13T11:23:25.558 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 0.41941662600000001, 2024-09-13T11:23:25.558 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0041018560000000001, 2024-09-13T11:23:25.558 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:23:25.558 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:23:25.558 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:25.558 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:23:25.558 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:25.558 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:23:25.558 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:25.110909+0000", 2024-09-13T11:23:25.558 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:25.559 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:25.559 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:25.559 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:23:25.559 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:25.110909+0000", 2024-09-13T11:23:25.559 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:25.559 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:25.559 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:25.559 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:23:25.559 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:25.110911+0000", 2024-09-13T11:23:25.559 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.9300000000000002e-06 2024-09-13T11:23:25.559 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:25.559 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:25.559 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:23:25.559 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:25.111020+0000", 2024-09-13T11:23:25.560 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000109484 2024-09-13T11:23:25.560 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:25.560 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:25.560 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:23:25.560 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:25.111053+0000", 2024-09-13T11:23:25.560 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.3470000000000003e-05 2024-09-13T11:23:25.560 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:25.560 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:25.560 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:23:25.560 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:25.111133+0000", 2024-09-13T11:23:25.560 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.9633000000000001e-05 2024-09-13T11:23:25.560 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:25.560 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:25.560 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:23:25.561 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:25.111357+0000", 2024-09-13T11:23:25.561 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00022399299999999999 2024-09-13T11:23:25.561 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:25.561 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:25.561 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:23:25.561 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:25.111382+0000", 2024-09-13T11:23:25.561 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.5286000000000002e-05 2024-09-13T11:23:25.561 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:25.561 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:25.561 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:23:25.561 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:25.114976+0000", 2024-09-13T11:23:25.561 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0035934029999999998 2024-09-13T11:23:25.561 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:25.561 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:25.562 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:23:25.562 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:25.115010+0000", 2024-09-13T11:23:25.562 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.4656999999999997e-05 2024-09-13T11:23:25.562 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:25.562 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:23:25.562 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:25.562 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:25.562 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:25.562 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "MOSDPGPush(3.19 71/70 [PushOp(3:980058cd:::benchmark_data_smithi177_31116_object4309:head, version: 44'269, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:980058cd:::benchmark_data_smithi177_31116_object4309:head@44'269, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:23:25.562 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:25.121441+0000", 2024-09-13T11:23:25.562 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 0.40888398100000001, 2024-09-13T11:23:25.562 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0010737310000000001, 2024-09-13T11:23:25.562 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:23:25.563 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:23:25.563 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-09-13T11:23:25.563 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:23:25.563 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:25.563 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:23:25.563 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:25.121441+0000", 2024-09-13T11:23:25.563 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:25.563 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:25.563 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:25.563 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:23:25.563 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:25.121441+0000", 2024-09-13T11:23:25.563 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:25.563 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:25.563 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:25.564 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:23:25.564 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:25.121444+0000", 2024-09-13T11:23:25.564 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.6110000000000001e-06 2024-09-13T11:23:25.564 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:25.564 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:25.564 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:23:25.564 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:25.121541+0000", 2024-09-13T11:23:25.564 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.7354000000000003e-05 2024-09-13T11:23:25.564 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:25.564 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:25.564 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:23:25.564 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:25.121587+0000", 2024-09-13T11:23:25.564 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.5806000000000002e-05 2024-09-13T11:23:25.564 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:25.565 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:25.565 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:23:25.565 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:25.121607+0000", 2024-09-13T11:23:25.565 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.9862000000000001e-05 2024-09-13T11:23:25.565 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:25.565 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:25.565 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:23:25.565 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:25.121703+0000", 2024-09-13T11:23:25.565 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.5951000000000003e-05 2024-09-13T11:23:25.565 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:25.565 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:25.565 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:23:25.565 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:25.121715+0000", 2024-09-13T11:23:25.565 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.2551000000000001e-05 2024-09-13T11:23:25.566 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:25.566 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:25.566 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:23:25.566 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:25.122515+0000", 2024-09-13T11:23:25.566 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00079959599999999997 2024-09-13T11:23:25.566 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:25.566 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:23:25.566 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:25.566 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:25.566 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:25.566 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop(client.4665.0:48707 3.19 e71/70 3:98e8cb5d:::benchmark_data_smithi177_31116_object48706:head v 71'2885, mlcod=44'265)", 2024-09-13T11:23:25.566 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:25.298293+0000", 2024-09-13T11:23:25.566 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 0.23203226699999999, 2024-09-13T11:23:25.566 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0018579219999999999, 2024-09-13T11:23:25.566 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:23:25.567 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:23:25.567 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:25.567 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:23:25.567 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:25.567 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:23:25.567 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:25.298293+0000", 2024-09-13T11:23:25.567 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:25.567 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:25.567 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:25.567 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:23:25.567 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:25.298293+0000", 2024-09-13T11:23:25.567 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:25.567 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:25.567 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:25.567 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:23:25.568 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:25.298294+0000", 2024-09-13T11:23:25.568 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.4139999999999999e-06 2024-09-13T11:23:25.568 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:25.568 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:25.568 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:23:25.568 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:25.298374+0000", 2024-09-13T11:23:25.568 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.0127e-05 2024-09-13T11:23:25.568 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:25.568 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:25.568 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:23:25.568 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:25.298405+0000", 2024-09-13T11:23:25.568 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.0511e-05 2024-09-13T11:23:25.568 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:25.568 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:25.569 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:23:25.569 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:25.298413+0000", 2024-09-13T11:23:25.569 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.6459999999999993e-06 2024-09-13T11:23:25.569 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:25.569 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:25.569 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:23:25.569 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:25.298464+0000", 2024-09-13T11:23:25.569 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.1678000000000002e-05 2024-09-13T11:23:25.569 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:25.569 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:25.569 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:23:25.569 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:25.298482+0000", 2024-09-13T11:23:25.569 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.7405999999999999e-05 2024-09-13T11:23:25.569 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:25.569 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:25.570 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:23:25.570 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:25.300106+0000", 2024-09-13T11:23:25.570 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0016242839999999999 2024-09-13T11:23:25.570 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:25.570 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:25.570 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:23:25.570 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:25.300151+0000", 2024-09-13T11:23:25.570 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.4855999999999999e-05 2024-09-13T11:23:25.570 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:25.570 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:23:25.570 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:25.570 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:25.570 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:25.570 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "MOSDPGPush(3.19 71/70 [PushOp(3:9a3e75c7:::benchmark_data_smithi177_31116_object4324:head, version: 44'271, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:9a3e75c7:::benchmark_data_smithi177_31116_object4324:head@44'271, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:23:25.571 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:25.304337+0000", 2024-09-13T11:23:25.571 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 0.22598812900000001, 2024-09-13T11:23:25.571 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0011522990000000001, 2024-09-13T11:23:25.571 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:23:25.571 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:23:25.571 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-09-13T11:23:25.571 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:23:25.572 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:25.572 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:23:25.572 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:25.304337+0000", 2024-09-13T11:23:25.572 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:25.572 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:25.572 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:25.572 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:23:25.572 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:25.304337+0000", 2024-09-13T11:23:25.572 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:25.572 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:25.572 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:25.572 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:23:25.572 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:25.304340+0000", 2024-09-13T11:23:25.572 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.773e-06 2024-09-13T11:23:25.573 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:25.573 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:25.573 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:23:25.573 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:25.304405+0000", 2024-09-13T11:23:25.573 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.4801000000000005e-05 2024-09-13T11:23:25.573 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:25.573 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:25.573 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:23:25.573 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:25.304460+0000", 2024-09-13T11:23:25.573 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.575e-05 2024-09-13T11:23:25.573 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:25.573 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:25.574 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:23:25.574 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:25.304479+0000", 2024-09-13T11:23:25.574 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.8351000000000001e-05 2024-09-13T11:23:25.574 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:25.574 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:25.574 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:23:25.574 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:25.304621+0000", 2024-09-13T11:23:25.574 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00014192499999999999 2024-09-13T11:23:25.574 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:25.574 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:25.574 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:23:25.574 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:25.304637+0000", 2024-09-13T11:23:25.574 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.6005999999999999e-05 2024-09-13T11:23:25.574 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:25.575 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:25.575 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:23:25.575 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:25.305489+0000", 2024-09-13T11:23:25.575 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00085269299999999996 2024-09-13T11:23:25.575 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:25.575 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:23:25.575 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:25.575 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:25.575 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:25.575 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "MOSDPGPush(3.19 71/70 [PushOp(3:9e752df7:::benchmark_data_smithi177_31116_object4411:head, version: 44'274, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:9e752df7:::benchmark_data_smithi177_31116_object4411:head@44'274, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:23:25.575 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:25.314529+0000", 2024-09-13T11:23:25.575 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 0.21579669000000001, 2024-09-13T11:23:25.575 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.001556078, 2024-09-13T11:23:25.576 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:23:25.576 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:23:25.576 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-09-13T11:23:25.576 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:23:25.576 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:25.576 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:23:25.576 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:25.314529+0000", 2024-09-13T11:23:25.576 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:25.576 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:25.576 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:25.576 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:23:25.576 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:25.314529+0000", 2024-09-13T11:23:25.576 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:25.577 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:25.577 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:25.577 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:23:25.577 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:25.314531+0000", 2024-09-13T11:23:25.577 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.1730000000000002e-06 2024-09-13T11:23:25.577 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:25.577 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:25.577 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:23:25.577 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:25.315237+0000", 2024-09-13T11:23:25.577 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00070645799999999998 2024-09-13T11:23:25.577 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:25.577 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:25.577 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:23:25.578 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:25.315282+0000", 2024-09-13T11:23:25.578 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.4648999999999999e-05 2024-09-13T11:23:25.578 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:25.578 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:25.578 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:23:25.578 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:25.315301+0000", 2024-09-13T11:23:25.578 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.9542999999999999e-05 2024-09-13T11:23:25.578 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:25.578 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:25.578 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:23:25.578 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:25.315421+0000", 2024-09-13T11:23:25.578 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0001192 2024-09-13T11:23:25.578 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:25.578 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:25.579 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:23:25.579 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:25.315440+0000", 2024-09-13T11:23:25.579 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.9315999999999999e-05 2024-09-13T11:23:25.579 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:25.579 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:25.579 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:23:25.579 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:25.316085+0000", 2024-09-13T11:23:25.579 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00064473899999999997 2024-09-13T11:23:25.579 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:25.579 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:23:25.579 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:25.579 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:25.579 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:25.579 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "MOSDPGPush(3.19 71/70 [PushOp(3:9d6d6fa0:::benchmark_data_smithi177_31116_object4456:head, version: 44'275, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:9d6d6fa0:::benchmark_data_smithi177_31116_object4456:head@44'275, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:23:25.580 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:25.323915+0000", 2024-09-13T11:23:25.580 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 0.20641067799999999, 2024-09-13T11:23:25.580 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.001035029, 2024-09-13T11:23:25.580 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:23:25.580 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:23:25.580 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-09-13T11:23:25.580 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:23:25.580 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:25.580 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:23:25.580 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:25.323915+0000", 2024-09-13T11:23:25.580 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:25.580 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:25.580 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:25.581 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:23:25.581 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:25.323915+0000", 2024-09-13T11:23:25.581 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:25.581 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:25.581 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:25.581 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:23:25.581 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:25.323917+0000", 2024-09-13T11:23:25.581 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.3989999999999998e-06 2024-09-13T11:23:25.581 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:25.581 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:25.581 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:23:25.581 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:25.323979+0000", 2024-09-13T11:23:25.581 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.1939000000000004e-05 2024-09-13T11:23:25.582 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:25.582 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:25.582 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:23:25.582 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:25.324020+0000", 2024-09-13T11:23:25.582 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.0958000000000003e-05 2024-09-13T11:23:25.582 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:25.582 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:25.582 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:23:25.582 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:25.324038+0000", 2024-09-13T11:23:25.582 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.8508e-05 2024-09-13T11:23:25.582 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:25.582 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:25.582 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:23:25.583 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:25.324147+0000", 2024-09-13T11:23:25.583 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00010891 2024-09-13T11:23:25.583 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:25.583 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:25.583 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:23:25.583 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:25.324161+0000", 2024-09-13T11:23:25.583 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.3412e-05 2024-09-13T11:23:25.583 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:25.583 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:25.583 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:23:25.583 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:25.324950+0000", 2024-09-13T11:23:25.583 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00078890299999999998 2024-09-13T11:23:25.583 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:25.584 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:23:25.584 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:25.584 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:25.584 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:25.584 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "MOSDPGPush(3.19 71/70 [PushOp(3:98aa237c:::benchmark_data_smithi177_31116_object4537:head, version: 44'281, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:98aa237c:::benchmark_data_smithi177_31116_object4537:head@44'281, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:23:25.584 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:25.331753+0000", 2024-09-13T11:23:25.584 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 0.19857238999999999, 2024-09-13T11:23:25.584 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.001022689, 2024-09-13T11:23:25.584 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:23:25.584 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:23:25.584 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-09-13T11:23:25.584 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:23:25.584 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:25.584 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:23:25.585 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:25.331753+0000", 2024-09-13T11:23:25.585 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:25.585 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:25.585 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:25.585 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:23:25.585 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:25.331753+0000", 2024-09-13T11:23:25.585 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:25.585 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:25.585 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:25.585 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:23:25.585 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:25.331755+0000", 2024-09-13T11:23:25.585 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.1880000000000001e-06 2024-09-13T11:23:25.585 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:25.585 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:25.586 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:23:25.586 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:25.331831+0000", 2024-09-13T11:23:25.586 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.6057999999999996e-05 2024-09-13T11:23:25.586 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:25.586 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:25.586 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:23:25.586 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:25.331878+0000", 2024-09-13T11:23:25.586 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.7002000000000002e-05 2024-09-13T11:23:25.586 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:25.586 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:25.586 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:23:25.586 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:25.331897+0000", 2024-09-13T11:23:25.586 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.9012999999999999e-05 2024-09-13T11:23:25.586 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:25.587 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:25.587 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:23:25.587 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:25.332001+0000", 2024-09-13T11:23:25.587 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00010385100000000001 2024-09-13T11:23:25.587 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:25.587 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:25.587 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:23:25.587 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:25.332015+0000", 2024-09-13T11:23:25.587 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.4368e-05 2024-09-13T11:23:25.587 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:25.587 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:25.587 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:23:25.587 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:25.332776+0000", 2024-09-13T11:23:25.587 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00076020899999999997 2024-09-13T11:23:25.588 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:25.588 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:23:25.588 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:25.588 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:25.588 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:25.588 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "MOSDPGPush(3.19 71/70 [PushOp(3:9f9046ad:::benchmark_data_smithi177_31116_object4606:head, version: 44'283, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:9f9046ad:::benchmark_data_smithi177_31116_object4606:head@44'283, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:23:25.588 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:25.342287+0000", 2024-09-13T11:23:25.588 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 0.18803830299999999, 2024-09-13T11:23:25.588 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.001110111, 2024-09-13T11:23:25.588 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:23:25.588 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:23:25.588 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-09-13T11:23:25.588 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:23:25.588 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:25.589 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:23:25.589 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:25.342287+0000", 2024-09-13T11:23:25.589 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:25.589 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:25.589 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:25.589 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:23:25.589 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:25.342287+0000", 2024-09-13T11:23:25.589 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:25.589 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:25.589 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:25.589 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:23:25.589 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:25.342290+0000", 2024-09-13T11:23:25.589 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.8399999999999999e-06 2024-09-13T11:23:25.589 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:25.590 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:25.590 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:23:25.590 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:25.342441+0000", 2024-09-13T11:23:25.590 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000151364 2024-09-13T11:23:25.590 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:25.590 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:25.590 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:23:25.590 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:25.342492+0000", 2024-09-13T11:23:25.590 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.0871999999999997e-05 2024-09-13T11:23:25.590 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:25.590 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:25.590 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:23:25.590 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:25.342515+0000", 2024-09-13T11:23:25.590 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.2580000000000001e-05 2024-09-13T11:23:25.591 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:25.591 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:25.591 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:23:25.591 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:25.342624+0000", 2024-09-13T11:23:25.591 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000109373 2024-09-13T11:23:25.591 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:25.591 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:25.591 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:23:25.591 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:25.342640+0000", 2024-09-13T11:23:25.591 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.6044000000000001e-05 2024-09-13T11:23:25.591 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:25.591 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:25.591 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:23:25.592 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:25.343397+0000", 2024-09-13T11:23:25.592 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00075703800000000003 2024-09-13T11:23:25.592 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:25.592 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:23:25.592 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:25.592 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:25.592 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:25.592 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "MOSDPGPush(3.19 71/70 [PushOp(3:9fb0c3ee:::benchmark_data_smithi177_31116_object4675:head, version: 44'287, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:9fb0c3ee:::benchmark_data_smithi177_31116_object4675:head@44'287, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:23:25.592 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:25.382627+0000", 2024-09-13T11:23:25.592 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 0.14769861100000001, 2024-09-13T11:23:25.592 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00096409100000000004, 2024-09-13T11:23:25.592 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:23:25.592 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:23:25.593 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-09-13T11:23:25.593 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:23:25.593 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:25.593 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:23:25.593 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:25.382627+0000", 2024-09-13T11:23:25.593 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:25.593 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:25.593 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:25.593 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:23:25.596 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:25.382627+0000", 2024-09-13T11:23:25.596 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:25.596 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:25.596 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:25.596 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:23:25.596 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:25.382629+0000", 2024-09-13T11:23:25.597 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.5660000000000001e-06 2024-09-13T11:23:25.597 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:25.597 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:25.597 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:23:25.597 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:25.382716+0000", 2024-09-13T11:23:25.597 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.7038000000000003e-05 2024-09-13T11:23:25.597 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:25.597 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:25.597 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:23:25.597 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:25.382768+0000", 2024-09-13T11:23:25.597 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.1558999999999999e-05 2024-09-13T11:23:25.597 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:25.597 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:25.598 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:23:25.598 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:25.382791+0000", 2024-09-13T11:23:25.598 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.3047000000000001e-05 2024-09-13T11:23:25.598 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:25.598 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:25.598 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:23:25.598 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:25.382897+0000", 2024-09-13T11:23:25.598 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000106487 2024-09-13T11:23:25.598 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:25.598 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:25.598 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:23:25.598 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:25.382912+0000", 2024-09-13T11:23:25.598 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.4222999999999999e-05 2024-09-13T11:23:25.598 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:25.599 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:25.599 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:23:25.599 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:25.383591+0000", 2024-09-13T11:23:25.599 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00067917100000000001 2024-09-13T11:23:25.599 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:25.599 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:23:25.599 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:25.599 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:25.599 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:25.599 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop(client.4665.0:48830 3.19 e71/70 3:9b52cdff:::benchmark_data_smithi177_31116_object48829:head v 71'2886, mlcod=44'270)", 2024-09-13T11:23:25.599 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:25.397130+0000", 2024-09-13T11:23:25.599 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 0.13319555, 2024-09-13T11:23:25.599 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0023311389999999999, 2024-09-13T11:23:25.600 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:23:25.600 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:23:25.600 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:25.600 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:23:25.600 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:25.600 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:23:25.600 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:25.397130+0000", 2024-09-13T11:23:25.600 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:25.600 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:25.600 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:25.600 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:23:25.600 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:25.397130+0000", 2024-09-13T11:23:25.600 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:25.600 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:25.601 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:25.601 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:23:25.601 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:25.397131+0000", 2024-09-13T11:23:25.601 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.2610000000000001e-06 2024-09-13T11:23:25.601 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:25.601 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:25.601 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:23:25.601 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:25.397211+0000", 2024-09-13T11:23:25.601 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.0129999999999993e-05 2024-09-13T11:23:25.601 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:25.601 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:25.601 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:23:25.601 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:25.397238+0000", 2024-09-13T11:23:25.601 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.7344000000000001e-05 2024-09-13T11:23:25.602 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:25.602 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:25.602 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:23:25.602 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:25.397245+0000", 2024-09-13T11:23:25.602 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.3570000000000003e-06 2024-09-13T11:23:25.602 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:25.602 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:25.602 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:23:25.602 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:25.397493+0000", 2024-09-13T11:23:25.602 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00024778099999999998 2024-09-13T11:23:25.602 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:25.602 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:25.602 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:23:25.602 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:25.397534+0000", 2024-09-13T11:23:25.603 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.1616999999999998e-05 2024-09-13T11:23:25.603 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:25.603 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:25.603 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:23:25.603 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:25.399419+0000", 2024-09-13T11:23:25.603 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0018846010000000001 2024-09-13T11:23:25.603 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:25.603 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:25.603 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:23:25.603 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:25.399461+0000", 2024-09-13T11:23:25.603 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.2048000000000001e-05 2024-09-13T11:23:25.603 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:25.603 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:23:25.603 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:25.603 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:25.604 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:25.604 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop(client.4665.0:48836 3.19 e71/70 3:98d2e6eb:::benchmark_data_smithi177_31116_object48835:head v 71'2887, mlcod=44'290)", 2024-09-13T11:23:25.604 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:25.401237+0000", 2024-09-13T11:23:25.604 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 0.12908846800000001, 2024-09-13T11:23:25.604 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0021295440000000001, 2024-09-13T11:23:25.604 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:23:25.604 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:23:25.604 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:25.604 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:23:25.604 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:25.604 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:23:25.604 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:25.401237+0000", 2024-09-13T11:23:25.604 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:25.605 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:25.605 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:25.605 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:23:25.605 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:25.401237+0000", 2024-09-13T11:23:25.605 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:25.605 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:25.605 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:25.605 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:23:25.605 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:25.401239+0000", 2024-09-13T11:23:25.605 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.0049999999999999e-06 2024-09-13T11:23:25.605 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:25.605 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:25.605 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:23:25.605 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:25.401310+0000", 2024-09-13T11:23:25.606 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.1707999999999999e-05 2024-09-13T11:23:25.606 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:25.606 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:25.606 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:23:25.606 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:25.401347+0000", 2024-09-13T11:23:25.606 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.6510000000000001e-05 2024-09-13T11:23:25.606 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:25.606 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:25.606 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:23:25.606 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:25.401355+0000", 2024-09-13T11:23:25.606 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.1699999999999997e-06 2024-09-13T11:23:25.606 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:25.606 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:25.606 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:23:25.607 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:25.401465+0000", 2024-09-13T11:23:25.607 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00010967999999999999 2024-09-13T11:23:25.607 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:25.607 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:25.607 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:23:25.607 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:25.401490+0000", 2024-09-13T11:23:25.607 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.4910999999999999e-05 2024-09-13T11:23:25.607 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:25.607 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:25.607 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:23:25.607 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:25.403315+0000", 2024-09-13T11:23:25.607 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.001825215 2024-09-13T11:23:25.607 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:25.608 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:25.608 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:23:25.608 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:25.403366+0000", 2024-09-13T11:23:25.608 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.1344999999999997e-05 2024-09-13T11:23:25.608 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:25.608 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:23:25.608 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:25.608 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:25.608 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:25.608 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "MOSDPGPush(3.19 71/70 [PushOp(3:9ccfbe4c:::benchmark_data_smithi177_31116_object4690:head, version: 44'291, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:9ccfbe4c:::benchmark_data_smithi177_31116_object4690:head@44'291, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:23:25.608 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:25.407401+0000", 2024-09-13T11:23:25.608 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 0.122924188, 2024-09-13T11:23:25.608 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0010619749999999999, 2024-09-13T11:23:25.608 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:23:25.609 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:23:25.609 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-09-13T11:23:25.609 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:23:25.609 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:25.609 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:23:25.609 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:25.407401+0000", 2024-09-13T11:23:25.609 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:25.609 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:25.609 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:25.609 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:23:25.609 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:25.407401+0000", 2024-09-13T11:23:25.609 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:25.609 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:25.609 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:25.610 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:23:25.610 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:25.407403+0000", 2024-09-13T11:23:25.610 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.2809999999999998e-06 2024-09-13T11:23:25.610 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:25.610 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:25.610 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:23:25.610 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:25.407447+0000", 2024-09-13T11:23:25.610 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.4057000000000002e-05 2024-09-13T11:23:25.610 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:25.610 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:25.610 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:23:25.610 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:25.407489+0000", 2024-09-13T11:23:25.610 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.1587e-05 2024-09-13T11:23:25.611 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:25.611 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:25.611 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:23:25.611 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:25.407504+0000", 2024-09-13T11:23:25.611 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.5048999999999999e-05 2024-09-13T11:23:25.611 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:25.611 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:25.611 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:23:25.611 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:25.407593+0000", 2024-09-13T11:23:25.611 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.9208000000000004e-05 2024-09-13T11:23:25.611 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:25.611 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:25.611 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:23:25.611 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:25.407612+0000", 2024-09-13T11:23:25.612 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.8987e-05 2024-09-13T11:23:25.612 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:25.612 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:25.612 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:23:25.612 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:25.408463+0000", 2024-09-13T11:23:25.612 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00085080599999999996 2024-09-13T11:23:25.612 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:25.612 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:23:25.612 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:25.612 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:25.612 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:25.612 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop(client.4665.0:48851 3.19 e71/70 3:9feb6a49:::benchmark_data_smithi177_31116_object48850:head v 71'2888, mlcod=44'290)", 2024-09-13T11:23:25.612 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:25.410126+0000", 2024-09-13T11:23:25.613 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 0.120199504, 2024-09-13T11:23:25.613 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0027265150000000001, 2024-09-13T11:23:25.613 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:23:25.613 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:23:25.613 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:25.613 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:23:25.613 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:25.613 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:23:25.613 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:25.410126+0000", 2024-09-13T11:23:25.613 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:25.613 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:25.613 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:25.613 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:23:25.613 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:25.410126+0000", 2024-09-13T11:23:25.614 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:25.614 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:25.614 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:25.614 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:23:25.614 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:25.410127+0000", 2024-09-13T11:23:25.614 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.516e-06 2024-09-13T11:23:25.614 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:25.614 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:25.614 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:23:25.614 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:25.410187+0000", 2024-09-13T11:23:25.614 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.9590999999999999e-05 2024-09-13T11:23:25.614 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:25.614 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:25.614 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:23:25.615 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:25.410223+0000", 2024-09-13T11:23:25.615 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.5927999999999999e-05 2024-09-13T11:23:25.615 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:25.615 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:25.615 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:23:25.615 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:25.410230+0000", 2024-09-13T11:23:25.615 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.323e-06 2024-09-13T11:23:25.615 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:25.615 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:25.615 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:23:25.615 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:25.410286+0000", 2024-09-13T11:23:25.615 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.6234999999999999e-05 2024-09-13T11:23:25.615 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:25.615 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:25.616 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:23:25.616 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:25.410322+0000", 2024-09-13T11:23:25.616 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.5626000000000001e-05 2024-09-13T11:23:25.616 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:25.616 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:25.616 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:23:25.616 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:25.412798+0000", 2024-09-13T11:23:25.616 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.002475653 2024-09-13T11:23:25.616 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:25.616 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:25.616 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:23:25.616 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:25.412852+0000", 2024-09-13T11:23:25.616 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.4642999999999998e-05 2024-09-13T11:23:25.617 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:25.617 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:23:25.617 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:25.617 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:25.617 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:25.617 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop(client.4665.0:48909 3.19 e71/70 3:9ef23a04:::benchmark_data_smithi177_31116_object48908:head v 71'2889, mlcod=44'291)", 2024-09-13T11:23:25.617 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:25.455670+0000", 2024-09-13T11:23:25.617 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 0.074655438000000005, 2024-09-13T11:23:25.617 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0035573779999999999, 2024-09-13T11:23:25.617 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:23:25.617 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:23:25.617 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:25.617 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:23:25.617 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:25.618 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:23:25.618 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:25.455670+0000", 2024-09-13T11:23:25.618 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:25.618 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:25.618 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:25.618 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:23:25.618 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:25.455670+0000", 2024-09-13T11:23:25.618 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:25.618 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:25.618 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:25.618 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:23:25.618 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:25.455673+0000", 2024-09-13T11:23:25.618 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.7580000000000001e-06 2024-09-13T11:23:25.618 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:25.619 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:25.619 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:23:25.619 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:25.455756+0000", 2024-09-13T11:23:25.619 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.3848000000000001e-05 2024-09-13T11:23:25.619 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:25.619 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:25.619 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:23:25.619 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:25.455800+0000", 2024-09-13T11:23:25.619 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.3361e-05 2024-09-13T11:23:25.619 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:25.619 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:25.619 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:23:25.619 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:25.455809+0000", 2024-09-13T11:23:25.619 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.7560000000000005e-06 2024-09-13T11:23:25.620 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:25.620 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:25.620 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:23:25.620 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:25.457208+0000", 2024-09-13T11:23:25.620 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.001399721 2024-09-13T11:23:25.620 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:25.620 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:25.620 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:23:25.620 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:25.457234+0000", 2024-09-13T11:23:25.620 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.6174999999999999e-05 2024-09-13T11:23:25.620 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:25.620 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:25.620 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:23:25.621 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:25.459186+0000", 2024-09-13T11:23:25.621 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00195145 2024-09-13T11:23:25.621 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:25.621 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:25.621 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:23:25.621 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:25.459227+0000", 2024-09-13T11:23:25.621 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.1309e-05 2024-09-13T11:23:25.621 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:25.621 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:23:25.621 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:25.621 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:25.621 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:25.621 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "MOSDPGPush(3.19 71/70 [PushOp(3:9b41e30d:::benchmark_data_smithi177_31116_object4694:head, version: 44'293, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:9b41e30d:::benchmark_data_smithi177_31116_object4694:head@44'293, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:23:25.622 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:25.465281+0000", 2024-09-13T11:23:25.622 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 0.065044167, 2024-09-13T11:23:25.622 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00104952, 2024-09-13T11:23:25.622 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:23:25.622 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:23:25.622 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-09-13T11:23:25.622 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:23:25.622 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:25.622 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:23:25.622 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:25.465281+0000", 2024-09-13T11:23:25.622 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:25.622 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:25.622 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:25.623 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:23:25.623 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:25.465281+0000", 2024-09-13T11:23:25.623 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:25.623 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:25.623 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:25.623 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:23:25.623 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:25.465283+0000", 2024-09-13T11:23:25.623 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.3269999999999999e-06 2024-09-13T11:23:25.623 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:25.623 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:25.623 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:23:25.623 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:25.465337+0000", 2024-09-13T11:23:25.623 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.329e-05 2024-09-13T11:23:25.623 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:25.624 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:25.624 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:23:25.624 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:25.465376+0000", 2024-09-13T11:23:25.624 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.8881000000000003e-05 2024-09-13T11:23:25.624 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:25.624 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:25.624 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:23:25.624 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:25.465389+0000", 2024-09-13T11:23:25.624 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.3556e-05 2024-09-13T11:23:25.624 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:25.624 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:25.624 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:23:25.624 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:25.465488+0000", 2024-09-13T11:23:25.624 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.8608000000000002e-05 2024-09-13T11:23:25.625 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:25.625 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:25.625 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:23:25.625 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:25.465502+0000", 2024-09-13T11:23:25.625 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.45e-05 2024-09-13T11:23:25.625 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:25.625 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:25.625 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:23:25.625 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:25.466331+0000", 2024-09-13T11:23:25.625 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00082835799999999996 2024-09-13T11:23:25.625 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:25.625 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:23:25.625 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:25.626 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:25.626 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:23:25.626 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-09-13T11:23:25.627 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_ops_in_flight 2024-09-13T11:23:25.628 INFO:tasks.ceph.osd.9.smithi092.stderr:2024-09-13T11:23:25.625+0000 7ff5ab3d0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26348) UID: 0 2024-09-13T11:23:25.730 INFO:tasks.ceph.osd.10.smithi162.stderr:2024-09-13T11:23:25.726+0000 7fb59c9f0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26317) UID: 0 2024-09-13T11:23:25.820 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-09-13T11:23:25.821 INFO:teuthology.orchestra.run.smithi092.stdout: "ops": [ 2024-09-13T11:23:25.821 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:25.821 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "MOSDPGPushReply(3.16 71/70 [PushReplyOp(3:68bd3bcc:::benchmark_data_smithi177_31116_object6699:head)])", 2024-09-13T11:23:25.821 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:25.819920+0000", 2024-09-13T11:23:25.821 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 0.00042485999999999999, 2024-09-13T11:23:25.821 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00043841099999999999, 2024-09-13T11:23:25.821 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:23:25.821 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:23:25.821 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "queued for pg", 2024-09-13T11:23:25.821 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:23:25.821 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:25.821 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:23:25.822 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:25.819920+0000", 2024-09-13T11:23:25.822 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:25.822 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:25.822 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:25.822 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:23:25.822 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:25.819920+0000", 2024-09-13T11:23:25.822 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:25.822 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:25.822 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:25.822 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:23:25.822 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:25.819920+0000", 2024-09-13T11:23:25.822 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.7800000000000001e-07 2024-09-13T11:23:25.822 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:25.823 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:25.823 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:23:25.823 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:25.819924+0000", 2024-09-13T11:23:25.823 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.1479999999999999e-06 2024-09-13T11:23:25.823 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:25.823 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:25.823 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:23:25.823 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:25.819958+0000", 2024-09-13T11:23:25.823 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.4076999999999999e-05 2024-09-13T11:23:25.823 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:25.823 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:25.823 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:23:25.823 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:25.819965+0000", 2024-09-13T11:23:25.823 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.7939999999999997e-06 2024-09-13T11:23:25.823 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:25.824 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:23:25.824 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:25.824 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:25.824 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:25.824 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:49305 3.16 3:6ef9500b:::benchmark_data_smithi177_31116_object49304:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e71)", 2024-09-13T11:23:25.824 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:25.808849+0000", 2024-09-13T11:23:25.824 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 0.011495079, 2024-09-13T11:23:25.824 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.011541355999999999, 2024-09-13T11:23:25.824 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:23:25.824 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:23:25.824 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:25.824 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:23:25.824 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:23:25.825 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:25.825 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 49305 2024-09-13T11:23:25.825 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:25.825 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:23:25.825 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:25.825 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:23:25.825 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:25.808849+0000", 2024-09-13T11:23:25.825 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:25.825 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:25.825 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:25.825 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:23:25.825 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:25.808849+0000", 2024-09-13T11:23:25.825 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:25.825 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:25.826 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:25.826 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:23:25.826 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:25.808851+0000", 2024-09-13T11:23:25.826 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.398e-06 2024-09-13T11:23:25.826 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:25.826 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:25.826 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:23:25.826 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:25.808927+0000", 2024-09-13T11:23:25.826 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.5784999999999994e-05 2024-09-13T11:23:25.826 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:25.826 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:25.826 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:23:25.826 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:25.808938+0000", 2024-09-13T11:23:25.826 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.1649e-05 2024-09-13T11:23:25.827 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:25.827 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:25.827 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:23:25.827 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:25.808945+0000", 2024-09-13T11:23:25.827 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.6919999999999996e-06 2024-09-13T11:23:25.827 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:25.827 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:25.827 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:23:25.827 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:25.813366+0000", 2024-09-13T11:23:25.827 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0044210370000000001 2024-09-13T11:23:25.827 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:25.827 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:25.827 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:23:25.828 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:25.813478+0000", 2024-09-13T11:23:25.828 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000111864 2024-09-13T11:23:25.828 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:25.828 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:25.828 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 11,15", 2024-09-13T11:23:25.828 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:25.813601+0000", 2024-09-13T11:23:25.828 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00012327599999999999 2024-09-13T11:23:25.828 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:25.828 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:25.828 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:23:25.828 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:25.814928+0000", 2024-09-13T11:23:25.828 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0013268290000000001 2024-09-13T11:23:25.828 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:25.829 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:25.829 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:25.829 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:25.819939+0000", 2024-09-13T11:23:25.829 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0050111319999999997 2024-09-13T11:23:25.829 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:25.829 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:25.829 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:25.829 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:25.819992+0000", 2024-09-13T11:23:25.829 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.2617999999999997e-05 2024-09-13T11:23:25.829 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:25.829 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:25.829 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:23:25.829 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:25.820083+0000", 2024-09-13T11:23:25.830 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.1259000000000002e-05 2024-09-13T11:23:25.830 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:25.830 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:23:25.830 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:25.830 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:25.830 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:25.830 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:49315 3.6 3:67fd4283:::benchmark_data_smithi177_31116_object49314:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e71)", 2024-09-13T11:23:25.830 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:25.813503+0000", 2024-09-13T11:23:25.830 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 0.0068420440000000003, 2024-09-13T11:23:25.830 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0069378570000000004, 2024-09-13T11:23:25.830 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:23:25.830 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:23:25.830 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "waiting for sub ops", 2024-09-13T11:23:25.830 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:23:25.831 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:23:25.831 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:25.831 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 49315 2024-09-13T11:23:25.831 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:25.831 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:23:25.831 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:25.831 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:23:25.831 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:25.813503+0000", 2024-09-13T11:23:25.831 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:25.831 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:25.831 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:25.831 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:23:25.832 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:25.813503+0000", 2024-09-13T11:23:25.832 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:25.832 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:25.832 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:25.832 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:23:25.832 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:25.813504+0000", 2024-09-13T11:23:25.832 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.5379999999999999e-06 2024-09-13T11:23:25.832 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:25.832 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:25.832 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:23:25.832 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:25.813552+0000", 2024-09-13T11:23:25.833 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.7908999999999998e-05 2024-09-13T11:23:25.833 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:25.833 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:25.833 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:23:25.833 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:25.813564+0000", 2024-09-13T11:23:25.833 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.1756999999999999e-05 2024-09-13T11:23:25.833 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:25.833 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:25.833 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:23:25.833 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:25.813570+0000", 2024-09-13T11:23:25.833 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.7440000000000001e-06 2024-09-13T11:23:25.833 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:25.833 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:25.833 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:23:25.834 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:25.813618+0000", 2024-09-13T11:23:25.834 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.7377999999999999e-05 2024-09-13T11:23:25.834 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:25.834 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:25.834 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:23:25.834 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:25.813734+0000", 2024-09-13T11:23:25.834 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000116592 2024-09-13T11:23:25.834 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:25.834 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:25.834 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 14", 2024-09-13T11:23:25.834 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:25.813853+0000", 2024-09-13T11:23:25.834 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00011831 2024-09-13T11:23:25.834 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:25.835 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:25.835 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:23:25.835 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:25.815167+0000", 2024-09-13T11:23:25.835 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.001313909 2024-09-13T11:23:25.835 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:25.835 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:23:25.835 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:25.835 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:25.835 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:25.835 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop(client.4665.0:49318 3.5 e71/67 3:a5056a3a:::benchmark_data_smithi177_31116_object49317:head v 71'2871, mlcod=71'2869)", 2024-09-13T11:23:25.835 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:25.819662+0000", 2024-09-13T11:23:25.835 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 0.00068253400000000005, 2024-09-13T11:23:25.835 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00081600599999999998, 2024-09-13T11:23:25.835 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:23:25.836 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:23:25.836 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-09-13T11:23:25.836 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:23:25.836 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:25.836 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:23:25.836 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:25.819662+0000", 2024-09-13T11:23:25.836 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:25.836 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:25.836 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:25.836 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:23:25.836 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:25.819662+0000", 2024-09-13T11:23:25.836 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:25.837 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:25.837 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:25.837 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:23:25.837 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:25.819664+0000", 2024-09-13T11:23:25.837 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.607e-06 2024-09-13T11:23:25.837 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:25.837 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:25.837 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:23:25.837 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:25.819712+0000", 2024-09-13T11:23:25.837 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.8529999999999998e-05 2024-09-13T11:23:25.837 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:25.837 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:25.838 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:23:25.838 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:25.819737+0000", 2024-09-13T11:23:25.838 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.4862e-05 2024-09-13T11:23:25.838 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:25.838 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:25.838 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:23:25.838 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:25.819742+0000", 2024-09-13T11:23:25.838 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.0000000000000004e-06 2024-09-13T11:23:25.838 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:25.838 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:25.838 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:23:25.838 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:25.819787+0000", 2024-09-13T11:23:25.838 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.5362999999999999e-05 2024-09-13T11:23:25.838 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:25.838 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:25.839 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:23:25.839 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:25.819803+0000", 2024-09-13T11:23:25.839 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.5248e-05 2024-09-13T11:23:25.839 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:25.839 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:23:25.839 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:25.839 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:25.839 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:25.839 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop_reply(client.4665.0:49305 3.16 e71/70 ondisk, result = 0)", 2024-09-13T11:23:25.839 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:25.819901+0000", 2024-09-13T11:23:25.839 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 0.00044316899999999998, 2024-09-13T11:23:25.839 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00060091300000000001, 2024-09-13T11:23:25.839 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:23:25.839 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:23:25.840 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-09-13T11:23:25.840 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:23:25.840 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:25.840 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:23:25.840 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:25.819901+0000", 2024-09-13T11:23:25.840 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:25.840 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:25.840 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:25.840 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:23:25.840 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:25.819901+0000", 2024-09-13T11:23:25.840 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:25.840 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:25.840 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:25.840 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:23:25.840 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:25.819902+0000", 2024-09-13T11:23:25.841 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.1699999999999998e-07 2024-09-13T11:23:25.841 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:25.841 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:25.841 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:23:25.841 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:25.819905+0000", 2024-09-13T11:23:25.841 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.6359999999999998e-06 2024-09-13T11:23:25.841 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:25.841 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:25.841 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:23:25.841 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:25.819934+0000", 2024-09-13T11:23:25.841 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.9555000000000001e-05 2024-09-13T11:23:25.841 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:25.841 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:25.842 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:23:25.842 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:25.819939+0000", 2024-09-13T11:23:25.842 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.967e-06 2024-09-13T11:23:25.842 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:25.842 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:25.842 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:23:25.842 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:25.819974+0000", 2024-09-13T11:23:25.842 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.4638000000000003e-05 2024-09-13T11:23:25.842 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:25.842 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:25.842 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:23:25.842 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:25.819981+0000", 2024-09-13T11:23:25.842 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.481e-06 2024-09-13T11:23:25.842 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:25.843 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:23:25.843 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:25.843 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:25.843 INFO:teuthology.orchestra.run.smithi092.stdout: ], 2024-09-13T11:23:25.843 INFO:teuthology.orchestra.run.smithi092.stdout: "num_ops": 5 2024-09-13T11:23:25.843 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-09-13T11:23:25.844 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_blocked_ops 2024-09-13T11:23:25.845 INFO:tasks.ceph.osd.3.smithi177.stderr:2024-09-13T11:23:25.841+0000 7f10adba4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26574) UID: 0 2024-09-13T11:23:25.946 INFO:tasks.ceph.osd.15.smithi177.stderr:2024-09-13T11:23:25.941+0000 7f9d6ce77640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26578) UID: 0 2024-09-13T11:23:25.965 INFO:teuthology.orchestra.run.smithi001.stderr:noscrub is set 2024-09-13T11:23:25.970 INFO:tasks.ceph.osd.13.smithi092.stderr:2024-09-13T11:23:25.968+0000 7f3e4da13640 -1 osd.13 71 handle_osd_map injecting map crc failure 2024-09-13T11:23:25.971 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-13T11:23:25.964+0000 7f6a5d1a2640 -1 osd.0 71 handle_osd_map injecting map crc failure 2024-09-13T11:23:25.973 INFO:tasks.ceph.osd.8.smithi001.stderr:2024-09-13T11:23:25.968+0000 7fb9a92db640 -1 osd.8 71 handle_osd_map injecting map crc failure 2024-09-13T11:23:26.004 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-09-13T11:23:26.005 INFO:teuthology.orchestra.run.smithi092.stdout: "ops": [], 2024-09-13T11:23:26.005 INFO:teuthology.orchestra.run.smithi092.stdout: "complaint_time": 30, 2024-09-13T11:23:26.005 INFO:teuthology.orchestra.run.smithi092.stdout: "num_blocked_ops": 0 2024-09-13T11:23:26.005 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-09-13T11:23:26.005 INFO:tasks.radosbench.radosbench.0.smithi177.stdout: 59 16 49563 49547 52.4804 72.9375 0.00606964 0.0190472 2024-09-13T11:23:26.022 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_historic_ops 2024-09-13T11:23:26.045 INFO:tasks.ceph.osd.12.smithi001.stderr:2024-09-13T11:23:26.040+0000 7f892c6e6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26314) UID: 0 2024-09-13T11:23:26.146 INFO:tasks.ceph.osd.15.smithi177.stderr:2024-09-13T11:23:26.141+0000 7f9d6ce77640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26578) UID: 0 2024-09-13T11:23:26.168 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-09-13T11:23:26.168 INFO:teuthology.orchestra.run.smithi092.stdout: "size": 20, 2024-09-13T11:23:26.168 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 600, 2024-09-13T11:23:26.168 INFO:teuthology.orchestra.run.smithi092.stdout: "ops": [ 2024-09-13T11:23:26.168 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.169 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:79 3.1 3:841df5b5:::benchmark_data_smithi177_31116_object78:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-09-13T11:23:26.169 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:27.056083+0000", 2024-09-13T11:23:26.169 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 59.109843697000002, 2024-09-13T11:23:26.169 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.89469730199999997, 2024-09-13T11:23:26.169 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:23:26.169 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:23:26.169 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:26.169 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:23:26.169 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:23:26.169 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:26.169 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 79 2024-09-13T11:23:26.169 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.169 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:23:26.170 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.170 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:23:26.170 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.056083+0000", 2024-09-13T11:23:26.170 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:26.170 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.170 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.170 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:23:26.170 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.056083+0000", 2024-09-13T11:23:26.170 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:26.170 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.170 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.170 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:23:26.170 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.056087+0000", 2024-09-13T11:23:26.170 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.0659999999999997e-06 2024-09-13T11:23:26.171 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.171 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.171 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:23:26.171 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.056148+0000", 2024-09-13T11:23:26.171 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.1376000000000003e-05 2024-09-13T11:23:26.171 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.171 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.171 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:23:26.171 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.056180+0000", 2024-09-13T11:23:26.171 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.2370999999999999e-05 2024-09-13T11:23:26.171 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.171 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.171 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:23:26.171 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.056195+0000", 2024-09-13T11:23:26.172 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.5119e-05 2024-09-13T11:23:26.172 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.172 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.172 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:23:26.172 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.056304+0000", 2024-09-13T11:23:26.172 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000108365 2024-09-13T11:23:26.172 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.172 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.172 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:23:26.172 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.056478+0000", 2024-09-13T11:23:26.172 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000174542 2024-09-13T11:23:26.172 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.172 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.173 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 11", 2024-09-13T11:23:26.173 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.056690+0000", 2024-09-13T11:23:26.173 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00021167500000000001 2024-09-13T11:23:26.173 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.173 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.173 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:23:26.173 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.058063+0000", 2024-09-13T11:23:26.174 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0013731819999999999 2024-09-13T11:23:26.174 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.174 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.174 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:26.174 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.950477+0000", 2024-09-13T11:23:26.174 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.89241369400000004 2024-09-13T11:23:26.174 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.174 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.174 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:23:26.175 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.950652+0000", 2024-09-13T11:23:26.175 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00017526400000000001 2024-09-13T11:23:26.175 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.175 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.175 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:23:26.175 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.950780+0000", 2024-09-13T11:23:26.175 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00012764800000000001 2024-09-13T11:23:26.175 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:26.175 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:23:26.175 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:26.176 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.176 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.176 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:7 3.b 3:de78393b:::benchmark_data_smithi177_31116_object6:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T11:23:26.176 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:48.313952+0000", 2024-09-13T11:23:26.176 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 37.851973948000001, 2024-09-13T11:23:26.176 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.86421288600000001, 2024-09-13T11:23:26.176 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:23:26.176 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:23:26.176 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:26.176 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:23:26.176 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:23:26.176 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:26.176 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 7 2024-09-13T11:23:26.176 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.177 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:23:26.177 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.177 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:23:26.177 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.313952+0000", 2024-09-13T11:23:26.177 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:26.177 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.177 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.177 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:23:26.177 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.313952+0000", 2024-09-13T11:23:26.177 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:26.177 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.177 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.177 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:23:26.177 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.313957+0000", 2024-09-13T11:23:26.178 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.3599999999999998e-06 2024-09-13T11:23:26.178 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.178 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.178 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:23:26.178 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314043+0000", 2024-09-13T11:23:26.178 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.6766000000000002e-05 2024-09-13T11:23:26.178 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.178 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.178 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:23:26.178 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314065+0000", 2024-09-13T11:23:26.178 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.1416e-05 2024-09-13T11:23:26.178 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.178 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.178 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:23:26.179 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314080+0000", 2024-09-13T11:23:26.179 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.5519000000000002e-05 2024-09-13T11:23:26.179 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.179 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.179 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:23:26.179 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315478+0000", 2024-09-13T11:23:26.179 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.001397765 2024-09-13T11:23:26.179 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.179 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.179 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-09-13T11:23:26.179 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315507+0000", 2024-09-13T11:23:26.179 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.9128999999999998e-05 2024-09-13T11:23:26.179 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.179 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.180 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:23:26.180 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.167978+0000", 2024-09-13T11:23:26.180 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.85247068400000003 2024-09-13T11:23:26.180 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.180 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.180 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:23:26.180 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.168307+0000", 2024-09-13T11:23:26.180 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00032886699999999999 2024-09-13T11:23:26.180 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.180 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.180 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1", 2024-09-13T11:23:26.180 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.168647+0000", 2024-09-13T11:23:26.180 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00033998399999999999 2024-09-13T11:23:26.181 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.181 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.181 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:26.181 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.178011+0000", 2024-09-13T11:23:26.181 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0093639169999999994 2024-09-13T11:23:26.181 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.181 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.181 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:23:26.181 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.178026+0000", 2024-09-13T11:23:26.181 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.5274999999999999e-05 2024-09-13T11:23:26.181 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.181 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.181 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:23:26.181 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.178080+0000", 2024-09-13T11:23:26.182 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.3601999999999999e-05 2024-09-13T11:23:26.182 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.182 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.182 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:23:26.182 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.178165+0000", 2024-09-13T11:23:26.182 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.5601999999999998e-05 2024-09-13T11:23:26.182 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:26.182 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:23:26.182 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:26.182 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.182 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.182 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:9 3.b 3:dafadff6:::benchmark_data_smithi177_31116_object8:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T11:23:26.182 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:48.314114+0000", 2024-09-13T11:23:26.182 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 37.851812133999999, 2024-09-13T11:23:26.183 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.86613615899999996, 2024-09-13T11:23:26.183 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:23:26.183 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:23:26.183 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:26.183 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:23:26.183 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:23:26.183 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:26.183 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 9 2024-09-13T11:23:26.183 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.183 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:23:26.183 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.183 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:23:26.183 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314114+0000", 2024-09-13T11:23:26.183 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:26.184 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.184 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.184 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:23:26.184 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314114+0000", 2024-09-13T11:23:26.184 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:26.184 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.184 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.184 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:23:26.184 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314117+0000", 2024-09-13T11:23:26.184 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.3860000000000001e-06 2024-09-13T11:23:26.184 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.184 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.184 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:23:26.185 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314215+0000", 2024-09-13T11:23:26.185 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.8900000000000005e-05 2024-09-13T11:23:26.185 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.185 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.185 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:23:26.185 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314229+0000", 2024-09-13T11:23:26.185 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.3444999999999999e-05 2024-09-13T11:23:26.185 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.185 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.185 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:23:26.185 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314239+0000", 2024-09-13T11:23:26.185 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.8470000000000008e-06 2024-09-13T11:23:26.185 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.185 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.186 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:23:26.186 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315648+0000", 2024-09-13T11:23:26.186 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.001409291 2024-09-13T11:23:26.186 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.186 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.186 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-09-13T11:23:26.186 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315671+0000", 2024-09-13T11:23:26.186 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.3169000000000001e-05 2024-09-13T11:23:26.186 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.186 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.186 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:23:26.186 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.176036+0000", 2024-09-13T11:23:26.186 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.86036497499999998 2024-09-13T11:23:26.187 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.187 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.187 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:23:26.187 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.176189+0000", 2024-09-13T11:23:26.187 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00015271100000000001 2024-09-13T11:23:26.187 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.187 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.187 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1", 2024-09-13T11:23:26.187 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.176339+0000", 2024-09-13T11:23:26.187 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00015057499999999999 2024-09-13T11:23:26.187 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.187 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.187 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:26.187 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.180148+0000", 2024-09-13T11:23:26.188 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.003808891 2024-09-13T11:23:26.188 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.188 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.188 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:23:26.188 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.180168+0000", 2024-09-13T11:23:26.188 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.9507000000000001e-05 2024-09-13T11:23:26.188 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.188 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.188 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:23:26.188 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.180213+0000", 2024-09-13T11:23:26.188 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.5504999999999999e-05 2024-09-13T11:23:26.188 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.188 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.188 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:23:26.189 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.180250+0000", 2024-09-13T11:23:26.189 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.6956999999999999e-05 2024-09-13T11:23:26.189 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:26.189 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:23:26.189 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:26.189 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.189 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.189 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:10 3.b 3:dbe028a8:::benchmark_data_smithi177_31116_object9:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T11:23:26.189 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:48.314259+0000", 2024-09-13T11:23:26.189 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 37.851667003999999, 2024-09-13T11:23:26.189 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.86637313599999999, 2024-09-13T11:23:26.189 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:23:26.189 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:23:26.190 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:26.190 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:23:26.190 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:23:26.190 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:26.190 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 10 2024-09-13T11:23:26.190 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.190 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:23:26.190 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.190 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:23:26.190 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314259+0000", 2024-09-13T11:23:26.190 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:26.190 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.190 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.190 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:23:26.191 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314259+0000", 2024-09-13T11:23:26.191 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:26.191 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.191 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.191 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:23:26.191 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314261+0000", 2024-09-13T11:23:26.191 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.1500000000000002e-06 2024-09-13T11:23:26.191 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.191 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.191 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:23:26.191 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314349+0000", 2024-09-13T11:23:26.191 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.8070000000000002e-05 2024-09-13T11:23:26.192 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.192 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.192 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:23:26.192 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314365+0000", 2024-09-13T11:23:26.192 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.5835999999999999e-05 2024-09-13T11:23:26.192 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.192 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.192 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:23:26.192 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314377+0000", 2024-09-13T11:23:26.192 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.1589e-05 2024-09-13T11:23:26.192 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.192 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.192 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:23:26.193 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315788+0000", 2024-09-13T11:23:26.193 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0014109579999999999 2024-09-13T11:23:26.193 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.193 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.193 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-09-13T11:23:26.193 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315809+0000", 2024-09-13T11:23:26.193 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.0985999999999999e-05 2024-09-13T11:23:26.193 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.193 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.193 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:23:26.193 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.176822+0000", 2024-09-13T11:23:26.193 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.86101310799999997 2024-09-13T11:23:26.193 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.194 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.194 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:23:26.194 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.177021+0000", 2024-09-13T11:23:26.194 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00019911199999999999 2024-09-13T11:23:26.194 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.194 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.194 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1", 2024-09-13T11:23:26.194 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.177226+0000", 2024-09-13T11:23:26.194 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00020516600000000001 2024-09-13T11:23:26.194 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.194 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.194 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:23:26.194 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.180264+0000", 2024-09-13T11:23:26.194 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0030381739999999998 2024-09-13T11:23:26.195 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.195 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.195 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:26.195 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.180387+0000", 2024-09-13T11:23:26.195 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00012257699999999999 2024-09-13T11:23:26.195 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.195 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.195 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:23:26.195 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.180518+0000", 2024-09-13T11:23:26.195 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00013135800000000001 2024-09-13T11:23:26.195 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.195 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.196 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:23:26.196 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.180632+0000", 2024-09-13T11:23:26.196 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000114052 2024-09-13T11:23:26.196 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:26.196 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:23:26.196 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:26.196 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.196 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.196 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:39 3.b 3:d91854b3:::benchmark_data_smithi177_31116_object38:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T11:23:26.196 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:48.314393+0000", 2024-09-13T11:23:26.196 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 37.851533461999999, 2024-09-13T11:23:26.196 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.86850121700000005, 2024-09-13T11:23:26.196 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:23:26.196 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:23:26.197 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:26.197 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:23:26.197 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:23:26.197 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:26.197 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 39 2024-09-13T11:23:26.197 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.197 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:23:26.197 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.197 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:23:26.197 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314393+0000", 2024-09-13T11:23:26.197 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:26.197 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.197 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.198 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:23:26.198 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314393+0000", 2024-09-13T11:23:26.198 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:26.198 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.198 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.198 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:23:26.198 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314394+0000", 2024-09-13T11:23:26.198 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.513e-06 2024-09-13T11:23:26.198 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.198 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.198 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:23:26.198 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314463+0000", 2024-09-13T11:23:26.198 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.8261000000000003e-05 2024-09-13T11:23:26.198 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.199 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.199 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:23:26.199 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314477+0000", 2024-09-13T11:23:26.199 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.4619e-05 2024-09-13T11:23:26.199 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.199 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.199 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:23:26.199 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314485+0000", 2024-09-13T11:23:26.199 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.1699999999999997e-06 2024-09-13T11:23:26.199 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.199 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.199 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:23:26.199 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315917+0000", 2024-09-13T11:23:26.199 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0014316089999999999 2024-09-13T11:23:26.199 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.200 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.200 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-09-13T11:23:26.200 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315933+0000", 2024-09-13T11:23:26.200 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.5999999999999999e-05 2024-09-13T11:23:26.200 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.200 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.200 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:23:26.200 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.178205+0000", 2024-09-13T11:23:26.200 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.86227247100000004 2024-09-13T11:23:26.200 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.200 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.200 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:23:26.201 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.178316+0000", 2024-09-13T11:23:26.201 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000110788 2024-09-13T11:23:26.204 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.204 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.204 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1", 2024-09-13T11:23:26.204 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.178448+0000", 2024-09-13T11:23:26.205 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000132056 2024-09-13T11:23:26.205 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.205 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.205 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:23:26.205 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.182721+0000", 2024-09-13T11:23:26.205 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0042729780000000002 2024-09-13T11:23:26.205 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.205 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.205 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:26.205 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.182801+0000", 2024-09-13T11:23:26.205 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.0209000000000004e-05 2024-09-13T11:23:26.205 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.205 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.205 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:23:26.206 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.182855+0000", 2024-09-13T11:23:26.206 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.3749000000000003e-05 2024-09-13T11:23:26.206 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.206 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.206 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:23:26.206 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.182894+0000", 2024-09-13T11:23:26.206 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.8794000000000002e-05 2024-09-13T11:23:26.206 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:26.206 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:23:26.206 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:26.206 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.206 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.206 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:50 3.b 3:d1c144b8:::benchmark_data_smithi177_31116_object49:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T11:23:26.206 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:48.314498+0000", 2024-09-13T11:23:26.207 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 37.851428046999999, 2024-09-13T11:23:26.207 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.86878673299999998, 2024-09-13T11:23:26.207 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:23:26.207 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:23:26.207 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:26.207 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:23:26.207 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:23:26.207 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:26.207 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 50 2024-09-13T11:23:26.207 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.207 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:23:26.207 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.207 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:23:26.208 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314498+0000", 2024-09-13T11:23:26.208 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:26.208 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.208 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.208 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:23:26.208 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314498+0000", 2024-09-13T11:23:26.208 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:26.208 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.208 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.208 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:23:26.208 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314499+0000", 2024-09-13T11:23:26.208 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.1969999999999999e-06 2024-09-13T11:23:26.208 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.209 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.209 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:23:26.209 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314565+0000", 2024-09-13T11:23:26.209 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.5628000000000003e-05 2024-09-13T11:23:26.209 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.209 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.209 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:23:26.209 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314579+0000", 2024-09-13T11:23:26.209 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.4222e-05 2024-09-13T11:23:26.209 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.210 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.210 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:23:26.210 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314590+0000", 2024-09-13T11:23:26.210 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.1240000000000001e-05 2024-09-13T11:23:26.210 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.210 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.210 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:23:26.210 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316060+0000", 2024-09-13T11:23:26.210 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.001469515 2024-09-13T11:23:26.210 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.210 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.210 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-09-13T11:23:26.210 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316076+0000", 2024-09-13T11:23:26.210 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.5608999999999999e-05 2024-09-13T11:23:26.211 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.211 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.211 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:23:26.211 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.178857+0000", 2024-09-13T11:23:26.211 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.86278099900000005 2024-09-13T11:23:26.211 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.211 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.211 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:23:26.211 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.178961+0000", 2024-09-13T11:23:26.211 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000104664 2024-09-13T11:23:26.211 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.211 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.211 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1", 2024-09-13T11:23:26.211 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.179123+0000", 2024-09-13T11:23:26.211 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000161738 2024-09-13T11:23:26.212 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.212 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.212 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:23:26.212 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.182755+0000", 2024-09-13T11:23:26.212 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0036323200000000001 2024-09-13T11:23:26.212 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.212 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.212 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:26.212 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.183023+0000", 2024-09-13T11:23:26.212 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00026749100000000002 2024-09-13T11:23:26.212 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.212 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.212 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:23:26.213 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.183168+0000", 2024-09-13T11:23:26.213 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00014499099999999999 2024-09-13T11:23:26.213 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.213 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.213 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:23:26.213 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.183285+0000", 2024-09-13T11:23:26.213 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000117119 2024-09-13T11:23:26.213 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:26.213 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:23:26.213 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:26.213 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.213 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.213 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:60 3.b 3:d67218b0:::benchmark_data_smithi177_31116_object59:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T11:23:26.214 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:48.314603+0000", 2024-09-13T11:23:26.214 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 37.851323483999998, 2024-09-13T11:23:26.214 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.87106201100000002, 2024-09-13T11:23:26.214 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:23:26.214 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:23:26.214 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:26.214 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:23:26.214 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:23:26.214 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:26.214 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 60 2024-09-13T11:23:26.214 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.214 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:23:26.214 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.214 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:23:26.215 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314603+0000", 2024-09-13T11:23:26.215 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:26.215 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.215 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.215 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:23:26.215 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314603+0000", 2024-09-13T11:23:26.215 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:26.215 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.215 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.215 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:23:26.215 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314604+0000", 2024-09-13T11:23:26.215 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.2359999999999999e-06 2024-09-13T11:23:26.215 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.216 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.216 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:23:26.216 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314667+0000", 2024-09-13T11:23:26.216 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.3077999999999995e-05 2024-09-13T11:23:26.216 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.216 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.216 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:23:26.216 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314699+0000", 2024-09-13T11:23:26.216 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.1965999999999999e-05 2024-09-13T11:23:26.216 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.216 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.216 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:23:26.216 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314710+0000", 2024-09-13T11:23:26.216 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.1148999999999999e-05 2024-09-13T11:23:26.217 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.217 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.217 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:23:26.217 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316291+0000", 2024-09-13T11:23:26.217 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0015804440000000001 2024-09-13T11:23:26.217 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.217 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.217 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-09-13T11:23:26.217 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316310+0000", 2024-09-13T11:23:26.217 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.9799000000000001e-05 2024-09-13T11:23:26.217 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.217 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.217 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:23:26.217 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.179555+0000", 2024-09-13T11:23:26.218 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.86324469500000001 2024-09-13T11:23:26.218 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.218 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.218 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:23:26.218 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.179666+0000", 2024-09-13T11:23:26.218 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000110443 2024-09-13T11:23:26.218 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.218 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.218 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1", 2024-09-13T11:23:26.218 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.179791+0000", 2024-09-13T11:23:26.218 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00012560499999999999 2024-09-13T11:23:26.218 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.218 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.218 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:23:26.219 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.182761+0000", 2024-09-13T11:23:26.219 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0029697019999999998 2024-09-13T11:23:26.219 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.219 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.219 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:26.219 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.185572+0000", 2024-09-13T11:23:26.220 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.002811512 2024-09-13T11:23:26.220 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.220 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.220 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:23:26.220 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.185625+0000", 2024-09-13T11:23:26.220 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.2716000000000001e-05 2024-09-13T11:23:26.220 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.220 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.220 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:23:26.220 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.185665+0000", 2024-09-13T11:23:26.220 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.9666000000000003e-05 2024-09-13T11:23:26.220 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:26.220 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:23:26.221 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:26.221 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.221 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.221 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:62 3.b 3:d20ae5a8:::benchmark_data_smithi177_31116_object61:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T11:23:26.221 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:48.314723+0000", 2024-09-13T11:23:26.221 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 37.851202942, 2024-09-13T11:23:26.221 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.87338985099999999, 2024-09-13T11:23:26.221 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:23:26.221 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:23:26.221 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:26.221 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:23:26.221 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:23:26.221 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:26.221 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 62 2024-09-13T11:23:26.222 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.222 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:23:26.222 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.222 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:23:26.222 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314723+0000", 2024-09-13T11:23:26.222 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:26.222 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.222 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.222 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:23:26.222 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314723+0000", 2024-09-13T11:23:26.222 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:26.222 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.222 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.223 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:23:26.223 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314725+0000", 2024-09-13T11:23:26.223 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.3069999999999999e-06 2024-09-13T11:23:26.223 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.223 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.223 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:23:26.223 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314797+0000", 2024-09-13T11:23:26.223 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.2652999999999997e-05 2024-09-13T11:23:26.223 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.223 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.223 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:23:26.223 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314809+0000", 2024-09-13T11:23:26.223 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.1496e-05 2024-09-13T11:23:26.223 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.224 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.224 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:23:26.224 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314816+0000", 2024-09-13T11:23:26.224 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.6639999999999998e-06 2024-09-13T11:23:26.224 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.224 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.224 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:23:26.224 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316424+0000", 2024-09-13T11:23:26.224 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0016077559999999999 2024-09-13T11:23:26.224 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.224 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.224 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-09-13T11:23:26.224 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316441+0000", 2024-09-13T11:23:26.224 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.7305000000000001e-05 2024-09-13T11:23:26.225 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.225 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.225 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:23:26.225 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.180688+0000", 2024-09-13T11:23:26.225 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.86424692800000003 2024-09-13T11:23:26.225 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.225 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.225 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:23:26.225 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.180807+0000", 2024-09-13T11:23:26.225 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000118977 2024-09-13T11:23:26.225 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.225 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.225 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1", 2024-09-13T11:23:26.225 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.180938+0000", 2024-09-13T11:23:26.226 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00013049700000000001 2024-09-13T11:23:26.226 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.226 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.226 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:23:26.226 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.185680+0000", 2024-09-13T11:23:26.226 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0047417029999999999 2024-09-13T11:23:26.226 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.226 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.226 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:26.226 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.187856+0000", 2024-09-13T11:23:26.226 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0021760550000000001 2024-09-13T11:23:26.226 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.226 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.226 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:23:26.227 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.188007+0000", 2024-09-13T11:23:26.227 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00015185099999999999 2024-09-13T11:23:26.227 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.227 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.227 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:23:26.227 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.188113+0000", 2024-09-13T11:23:26.227 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000105659 2024-09-13T11:23:26.227 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:26.227 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:23:26.227 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:26.227 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.228 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.228 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:73 3.b 3:d0060d5f:::benchmark_data_smithi177_31116_object72:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T11:23:26.228 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:48.314829+0000", 2024-09-13T11:23:26.228 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 37.851097592999999, 2024-09-13T11:23:26.228 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.87364757199999998, 2024-09-13T11:23:26.228 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:23:26.228 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:23:26.228 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:26.228 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:23:26.228 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:23:26.228 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:26.228 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 73 2024-09-13T11:23:26.228 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.229 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:23:26.229 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.229 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:23:26.229 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314829+0000", 2024-09-13T11:23:26.229 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:26.229 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.229 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.229 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:23:26.229 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314829+0000", 2024-09-13T11:23:26.229 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:26.229 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.229 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.229 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:23:26.230 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314830+0000", 2024-09-13T11:23:26.230 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.243e-06 2024-09-13T11:23:26.230 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.230 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.230 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:23:26.230 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314965+0000", 2024-09-13T11:23:26.230 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00013470800000000001 2024-09-13T11:23:26.230 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.230 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.230 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:23:26.230 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314979+0000", 2024-09-13T11:23:26.230 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.4375999999999999e-05 2024-09-13T11:23:26.230 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.230 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.230 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:23:26.231 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314988+0000", 2024-09-13T11:23:26.231 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.8570000000000001e-06 2024-09-13T11:23:26.231 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.231 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.231 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:23:26.231 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316545+0000", 2024-09-13T11:23:26.231 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.001556878 2024-09-13T11:23:26.231 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.231 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.231 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-09-13T11:23:26.231 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316556+0000", 2024-09-13T11:23:26.231 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.1453e-05 2024-09-13T11:23:26.231 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.232 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.232 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:23:26.232 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.182124+0000", 2024-09-13T11:23:26.232 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.86556744200000002 2024-09-13T11:23:26.232 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.232 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.232 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:23:26.232 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.182246+0000", 2024-09-13T11:23:26.232 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00012210500000000001 2024-09-13T11:23:26.232 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.232 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.232 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1", 2024-09-13T11:23:26.232 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.182375+0000", 2024-09-13T11:23:26.233 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00012920999999999999 2024-09-13T11:23:26.233 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.233 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.233 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:23:26.233 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.185684+0000", 2024-09-13T11:23:26.233 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0033093969999999999 2024-09-13T11:23:26.233 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.233 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.233 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:26.233 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.188244+0000", 2024-09-13T11:23:26.233 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0025601550000000002 2024-09-13T11:23:26.233 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.233 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.234 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:23:26.234 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.188379+0000", 2024-09-13T11:23:26.234 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00013463400000000001 2024-09-13T11:23:26.234 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.234 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.234 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:23:26.234 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.188476+0000", 2024-09-13T11:23:26.234 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.7114000000000006e-05 2024-09-13T11:23:26.234 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:26.234 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:23:26.234 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:26.234 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.234 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.235 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:81 3.b 3:d705fd70:::benchmark_data_smithi177_31116_object80:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T11:23:26.235 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:48.315000+0000", 2024-09-13T11:23:26.235 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 37.850925756000002, 2024-09-13T11:23:26.235 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.87561418300000005, 2024-09-13T11:23:26.235 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:23:26.235 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:23:26.235 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:26.235 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:23:26.235 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:23:26.235 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:26.235 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 81 2024-09-13T11:23:26.235 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.235 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:23:26.236 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.236 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:23:26.236 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315000+0000", 2024-09-13T11:23:26.236 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:26.236 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.236 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.236 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:23:26.236 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315000+0000", 2024-09-13T11:23:26.236 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:26.236 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.236 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.236 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:23:26.236 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315002+0000", 2024-09-13T11:23:26.236 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.3799999999999999e-06 2024-09-13T11:23:26.237 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.237 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.237 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:23:26.237 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315119+0000", 2024-09-13T11:23:26.237 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000116925 2024-09-13T11:23:26.237 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.237 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.237 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:23:26.237 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315133+0000", 2024-09-13T11:23:26.237 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.3971999999999999e-05 2024-09-13T11:23:26.237 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.237 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.237 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:23:26.238 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315142+0000", 2024-09-13T11:23:26.238 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.4399999999999994e-06 2024-09-13T11:23:26.238 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.238 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.238 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:23:26.238 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316652+0000", 2024-09-13T11:23:26.238 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.001509907 2024-09-13T11:23:26.238 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.238 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.238 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-09-13T11:23:26.238 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316668+0000", 2024-09-13T11:23:26.238 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.5945999999999999e-05 2024-09-13T11:23:26.238 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.238 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.239 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:23:26.239 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.183405+0000", 2024-09-13T11:23:26.239 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.866737384 2024-09-13T11:23:26.239 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.239 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.239 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:23:26.239 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.183701+0000", 2024-09-13T11:23:26.239 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00029511099999999997 2024-09-13T11:23:26.239 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.239 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.239 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1", 2024-09-13T11:23:26.239 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.184078+0000", 2024-09-13T11:23:26.239 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00037741000000000001 2024-09-13T11:23:26.240 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.240 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.240 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:23:26.240 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.188504+0000", 2024-09-13T11:23:26.240 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0044260280000000003 2024-09-13T11:23:26.240 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.240 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.240 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:26.240 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.190562+0000", 2024-09-13T11:23:26.241 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0020580099999999999 2024-09-13T11:23:26.241 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.241 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.241 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:23:26.241 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.190592+0000", 2024-09-13T11:23:26.241 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.0057999999999999e-05 2024-09-13T11:23:26.241 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.241 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.241 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:23:26.241 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.190615+0000", 2024-09-13T11:23:26.241 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.2612e-05 2024-09-13T11:23:26.241 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:26.241 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:23:26.241 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:26.242 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.242 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.242 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:87 3.b 3:dd7de0ad:::benchmark_data_smithi177_31116_object86:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T11:23:26.242 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:48.315155+0000", 2024-09-13T11:23:26.242 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 37.850770959000002, 2024-09-13T11:23:26.242 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.87555916099999997, 2024-09-13T11:23:26.242 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:23:26.242 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:23:26.242 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:26.242 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:23:26.242 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:23:26.242 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:26.242 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 87 2024-09-13T11:23:26.242 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.243 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:23:26.243 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.243 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:23:26.243 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315155+0000", 2024-09-13T11:23:26.243 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:26.243 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.243 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.243 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:23:26.243 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315155+0000", 2024-09-13T11:23:26.243 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:26.243 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.243 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.243 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:23:26.243 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315157+0000", 2024-09-13T11:23:26.244 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.3829999999999999e-06 2024-09-13T11:23:26.244 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.244 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.244 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:23:26.244 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315271+0000", 2024-09-13T11:23:26.244 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000114539 2024-09-13T11:23:26.244 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.244 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.244 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:23:26.244 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315285+0000", 2024-09-13T11:23:26.244 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.4042e-05 2024-09-13T11:23:26.244 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.244 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.245 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:23:26.245 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315295+0000", 2024-09-13T11:23:26.245 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.2739999999999996e-06 2024-09-13T11:23:26.245 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.245 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.245 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:23:26.245 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316739+0000", 2024-09-13T11:23:26.245 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0014446960000000001 2024-09-13T11:23:26.245 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.245 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.245 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-09-13T11:23:26.245 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316747+0000", 2024-09-13T11:23:26.245 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.0790000000000001e-06 2024-09-13T11:23:26.245 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.246 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.246 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:23:26.246 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.184922+0000", 2024-09-13T11:23:26.246 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.86817446399999998 2024-09-13T11:23:26.246 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.246 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.246 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:23:26.246 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.185037+0000", 2024-09-13T11:23:26.246 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00011565099999999999 2024-09-13T11:23:26.246 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.246 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.246 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1", 2024-09-13T11:23:26.246 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.185171+0000", 2024-09-13T11:23:26.246 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00013316900000000001 2024-09-13T11:23:26.247 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.247 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.247 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:23:26.247 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.188520+0000", 2024-09-13T11:23:26.247 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0033489150000000001 2024-09-13T11:23:26.247 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.247 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.247 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:26.247 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.190652+0000", 2024-09-13T11:23:26.247 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0021325649999999999 2024-09-13T11:23:26.247 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.247 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.247 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:23:26.247 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.190686+0000", 2024-09-13T11:23:26.247 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.3827e-05 2024-09-13T11:23:26.248 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.248 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.248 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:23:26.248 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.190714+0000", 2024-09-13T11:23:26.248 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.8557000000000001e-05 2024-09-13T11:23:26.248 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:26.248 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:23:26.248 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:26.248 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.248 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.248 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:99 3.b 3:d011b600:::benchmark_data_smithi177_31116_object98:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T11:23:26.248 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:48.315307+0000", 2024-09-13T11:23:26.248 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 37.850619131999999, 2024-09-13T11:23:26.249 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.87551770799999995, 2024-09-13T11:23:26.249 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:23:26.249 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:23:26.249 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:26.249 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:23:26.249 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:23:26.249 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:26.249 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 99 2024-09-13T11:23:26.249 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.249 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:23:26.249 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.249 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:23:26.249 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315307+0000", 2024-09-13T11:23:26.249 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:26.250 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.250 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.250 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:23:26.250 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315307+0000", 2024-09-13T11:23:26.250 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:26.250 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.250 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.250 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:23:26.250 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315308+0000", 2024-09-13T11:23:26.250 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.316e-06 2024-09-13T11:23:26.250 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.250 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.250 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:23:26.250 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315453+0000", 2024-09-13T11:23:26.251 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00014473200000000001 2024-09-13T11:23:26.251 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.251 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.251 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:23:26.251 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315471+0000", 2024-09-13T11:23:26.251 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.8019000000000001e-05 2024-09-13T11:23:26.251 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.251 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.251 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:23:26.251 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315489+0000", 2024-09-13T11:23:26.251 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.7861999999999999e-05 2024-09-13T11:23:26.251 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.251 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.251 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:23:26.251 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316837+0000", 2024-09-13T11:23:26.252 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.001347787 2024-09-13T11:23:26.252 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.252 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.252 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-09-13T11:23:26.252 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316858+0000", 2024-09-13T11:23:26.252 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.1103000000000001e-05 2024-09-13T11:23:26.252 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.252 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.252 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:23:26.252 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.185713+0000", 2024-09-13T11:23:26.252 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.86885540800000005 2024-09-13T11:23:26.252 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.252 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.252 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:23:26.253 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.185820+0000", 2024-09-13T11:23:26.253 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000107081 2024-09-13T11:23:26.253 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.253 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.253 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1", 2024-09-13T11:23:26.253 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.185948+0000", 2024-09-13T11:23:26.253 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00012775800000000001 2024-09-13T11:23:26.253 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.253 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.253 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:23:26.253 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.190527+0000", 2024-09-13T11:23:26.253 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0045783910000000002 2024-09-13T11:23:26.253 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.253 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.254 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:26.254 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.190759+0000", 2024-09-13T11:23:26.254 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00023266199999999999 2024-09-13T11:23:26.254 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.254 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.254 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:23:26.254 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.190789+0000", 2024-09-13T11:23:26.254 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.9666999999999999e-05 2024-09-13T11:23:26.254 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.254 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.254 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:23:26.254 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.190825+0000", 2024-09-13T11:23:26.254 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.5921999999999999e-05 2024-09-13T11:23:26.254 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:26.254 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:23:26.255 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:26.255 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.255 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.255 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:108 3.b 3:d5c90217:::benchmark_data_smithi177_31116_object107:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T11:23:26.255 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:48.315513+0000", 2024-09-13T11:23:26.255 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 37.850413426999999, 2024-09-13T11:23:26.255 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.87544772500000001, 2024-09-13T11:23:26.255 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:23:26.255 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:23:26.255 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:26.255 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:23:26.255 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:23:26.255 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:26.255 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 108 2024-09-13T11:23:26.256 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.256 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:23:26.256 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.256 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:23:26.256 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315513+0000", 2024-09-13T11:23:26.256 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:26.256 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.256 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.256 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:23:26.256 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315513+0000", 2024-09-13T11:23:26.256 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:26.256 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.256 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.256 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:23:26.257 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315515+0000", 2024-09-13T11:23:26.257 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.9369999999999998e-06 2024-09-13T11:23:26.257 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.257 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.257 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:23:26.257 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315644+0000", 2024-09-13T11:23:26.257 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000129011 2024-09-13T11:23:26.257 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.257 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.257 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:23:26.257 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315657+0000", 2024-09-13T11:23:26.257 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.3689999999999999e-05 2024-09-13T11:23:26.257 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.257 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.258 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:23:26.258 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315668+0000", 2024-09-13T11:23:26.258 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0589e-05 2024-09-13T11:23:26.258 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.258 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.258 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:23:26.258 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316925+0000", 2024-09-13T11:23:26.258 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0012568340000000001 2024-09-13T11:23:26.258 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.258 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.258 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-09-13T11:23:26.258 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316932+0000", 2024-09-13T11:23:26.258 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.2230000000000001e-06 2024-09-13T11:23:26.259 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.259 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.259 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:23:26.259 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.186619+0000", 2024-09-13T11:23:26.259 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.86968679599999998 2024-09-13T11:23:26.259 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.259 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.259 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:23:26.259 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.186914+0000", 2024-09-13T11:23:26.259 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00029528900000000001 2024-09-13T11:23:26.259 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.259 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.259 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1", 2024-09-13T11:23:26.259 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.187217+0000", 2024-09-13T11:23:26.259 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00030232099999999998 2024-09-13T11:23:26.260 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.260 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.260 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:23:26.260 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.190532+0000", 2024-09-13T11:23:26.260 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.003315391 2024-09-13T11:23:26.260 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.260 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.260 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:26.260 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.190875+0000", 2024-09-13T11:23:26.260 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00034326300000000001 2024-09-13T11:23:26.260 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.260 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.260 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:23:26.261 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.190925+0000", 2024-09-13T11:23:26.261 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.9429999999999999e-05 2024-09-13T11:23:26.261 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.261 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.261 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:23:26.261 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.190961+0000", 2024-09-13T11:23:26.261 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.5951000000000002e-05 2024-09-13T11:23:26.261 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:26.261 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:23:26.261 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:26.261 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.261 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.261 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:172 3.b 3:dbec1eeb:::benchmark_data_smithi177_31116_object171:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T11:23:26.262 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:48.315684+0000", 2024-09-13T11:23:26.262 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 37.850242270000003, 2024-09-13T11:23:26.262 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.87615312999999995, 2024-09-13T11:23:26.262 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:23:26.262 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:23:26.262 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:26.262 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:23:26.262 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:23:26.262 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:26.262 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 172 2024-09-13T11:23:26.262 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.262 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:23:26.262 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.262 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:23:26.263 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315684+0000", 2024-09-13T11:23:26.263 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:26.263 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.263 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.263 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:23:26.263 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315684+0000", 2024-09-13T11:23:26.263 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:26.263 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.263 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.263 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:23:26.263 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315686+0000", 2024-09-13T11:23:26.263 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.886e-06 2024-09-13T11:23:26.263 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.263 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.264 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:23:26.264 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315820+0000", 2024-09-13T11:23:26.264 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00013388500000000001 2024-09-13T11:23:26.264 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.264 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.264 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:23:26.264 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315837+0000", 2024-09-13T11:23:26.264 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.6756e-05 2024-09-13T11:23:26.264 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.264 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.264 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:23:26.264 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315851+0000", 2024-09-13T11:23:26.264 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.4566000000000001e-05 2024-09-13T11:23:26.265 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.265 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.265 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:23:26.265 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.317021+0000", 2024-09-13T11:23:26.265 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0011701769999999999 2024-09-13T11:23:26.265 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.265 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.265 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-09-13T11:23:26.265 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.317037+0000", 2024-09-13T11:23:26.265 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.609e-05 2024-09-13T11:23:26.265 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.265 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.265 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:23:26.265 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.188634+0000", 2024-09-13T11:23:26.266 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.87159711900000003 2024-09-13T11:23:26.266 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.266 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.266 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:23:26.266 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.188974+0000", 2024-09-13T11:23:26.266 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00033922299999999999 2024-09-13T11:23:26.266 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.266 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.266 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1", 2024-09-13T11:23:26.266 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.189510+0000", 2024-09-13T11:23:26.266 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00053618999999999997 2024-09-13T11:23:26.266 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.266 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.266 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:26.267 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.191027+0000", 2024-09-13T11:23:26.267 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0015168390000000001 2024-09-13T11:23:26.267 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.267 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.267 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:23:26.267 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.191766+0000", 2024-09-13T11:23:26.267 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000739064 2024-09-13T11:23:26.267 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.267 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.267 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:23:26.267 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.191810+0000", 2024-09-13T11:23:26.267 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.3755999999999999e-05 2024-09-13T11:23:26.267 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.267 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.267 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:23:26.268 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.191837+0000", 2024-09-13T11:23:26.268 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.7579000000000001e-05 2024-09-13T11:23:26.268 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:26.268 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:23:26.268 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:26.268 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.268 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.268 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:177 3.b 3:de89c779:::benchmark_data_smithi177_31116_object176:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T11:23:26.268 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:48.315898+0000", 2024-09-13T11:23:26.268 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 37.850027955000002, 2024-09-13T11:23:26.268 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.87600712599999997, 2024-09-13T11:23:26.268 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:23:26.269 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:23:26.269 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:26.269 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:23:26.269 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:23:26.269 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:26.269 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 177 2024-09-13T11:23:26.269 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.269 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:23:26.269 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.269 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:23:26.269 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315898+0000", 2024-09-13T11:23:26.269 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:26.269 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.270 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.270 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:23:26.270 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315898+0000", 2024-09-13T11:23:26.270 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:26.270 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.270 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.270 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:23:26.270 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315900+0000", 2024-09-13T11:23:26.270 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.4929999999999999e-06 2024-09-13T11:23:26.270 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.270 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.270 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:23:26.270 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316027+0000", 2024-09-13T11:23:26.271 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000127568 2024-09-13T11:23:26.271 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.271 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.271 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:23:26.271 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316048+0000", 2024-09-13T11:23:26.271 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.0795999999999999e-05 2024-09-13T11:23:26.271 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.271 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.271 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:23:26.271 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316061+0000", 2024-09-13T11:23:26.271 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.3283e-05 2024-09-13T11:23:26.271 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.271 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.271 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:23:26.272 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.317107+0000", 2024-09-13T11:23:26.272 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0010452949999999999 2024-09-13T11:23:26.272 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.272 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.272 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-09-13T11:23:26.272 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.317114+0000", 2024-09-13T11:23:26.272 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.4569999999999999e-06 2024-09-13T11:23:26.272 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.272 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.272 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:23:26.272 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.190017+0000", 2024-09-13T11:23:26.272 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.87290258499999995 2024-09-13T11:23:26.272 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.273 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.273 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:23:26.273 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.190138+0000", 2024-09-13T11:23:26.273 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000121418 2024-09-13T11:23:26.273 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.273 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.273 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1", 2024-09-13T11:23:26.273 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.190216+0000", 2024-09-13T11:23:26.273 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.7410999999999995e-05 2024-09-13T11:23:26.273 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.273 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.273 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:26.273 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.191755+0000", 2024-09-13T11:23:26.274 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0015395420000000001 2024-09-13T11:23:26.274 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.274 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.274 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:23:26.274 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.191843+0000", 2024-09-13T11:23:26.274 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.8022999999999994e-05 2024-09-13T11:23:26.274 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.274 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.274 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:23:26.275 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.191876+0000", 2024-09-13T11:23:26.275 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.2619000000000001e-05 2024-09-13T11:23:26.275 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.275 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.275 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:23:26.275 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.191905+0000", 2024-09-13T11:23:26.275 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.9635999999999999e-05 2024-09-13T11:23:26.275 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:26.275 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:23:26.275 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:26.275 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.275 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.275 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:187 3.b 3:da03c8dd:::benchmark_data_smithi177_31116_object186:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T11:23:26.276 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:48.316083+0000", 2024-09-13T11:23:26.276 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 37.849842999000003, 2024-09-13T11:23:26.276 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.87835499500000003, 2024-09-13T11:23:26.276 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:23:26.276 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:23:26.276 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:26.276 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:23:26.276 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:23:26.276 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:26.276 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 187 2024-09-13T11:23:26.276 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.276 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:23:26.277 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.277 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:23:26.277 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316083+0000", 2024-09-13T11:23:26.277 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:26.277 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.277 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.277 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:23:26.277 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316083+0000", 2024-09-13T11:23:26.277 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:26.277 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.277 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.277 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:23:26.277 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316086+0000", 2024-09-13T11:23:26.278 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.5560000000000002e-06 2024-09-13T11:23:26.278 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.278 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.278 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:23:26.278 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316199+0000", 2024-09-13T11:23:26.278 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000113522 2024-09-13T11:23:26.278 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.278 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.278 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:23:26.278 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316215+0000", 2024-09-13T11:23:26.278 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.5265999999999999e-05 2024-09-13T11:23:26.278 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.278 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.278 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:23:26.279 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316226+0000", 2024-09-13T11:23:26.279 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0967e-05 2024-09-13T11:23:26.279 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.279 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.279 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:23:26.279 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.317198+0000", 2024-09-13T11:23:26.279 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00097194100000000004 2024-09-13T11:23:26.279 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.279 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.279 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-09-13T11:23:26.279 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.317213+0000", 2024-09-13T11:23:26.279 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.5472999999999999e-05 2024-09-13T11:23:26.279 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.280 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.280 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:23:26.280 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.191077+0000", 2024-09-13T11:23:26.280 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.87386435200000001 2024-09-13T11:23:26.280 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.280 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.280 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:23:26.280 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.191181+0000", 2024-09-13T11:23:26.280 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000103618 2024-09-13T11:23:26.281 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.281 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.281 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1", 2024-09-13T11:23:26.281 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.191305+0000", 2024-09-13T11:23:26.281 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00012358599999999999 2024-09-13T11:23:26.281 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.281 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.281 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:26.281 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.194279+0000", 2024-09-13T11:23:26.281 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0029746970000000001 2024-09-13T11:23:26.281 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.281 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.282 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:23:26.282 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.194336+0000", 2024-09-13T11:23:26.282 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.7244000000000003e-05 2024-09-13T11:23:26.282 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.282 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.282 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:23:26.282 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.194397+0000", 2024-09-13T11:23:26.282 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.0278000000000001e-05 2024-09-13T11:23:26.282 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.282 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.282 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:23:26.282 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.194438+0000", 2024-09-13T11:23:26.282 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.1495000000000002e-05 2024-09-13T11:23:26.282 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:26.283 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:23:26.283 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:26.283 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.283 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.283 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:200 3.b 3:d90039f1:::benchmark_data_smithi177_31116_object199:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T11:23:26.283 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:48.316280+0000", 2024-09-13T11:23:26.283 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 37.849646534000001, 2024-09-13T11:23:26.283 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.87879273000000002, 2024-09-13T11:23:26.283 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:23:26.283 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:23:26.283 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:26.283 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:23:26.283 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:23:26.283 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:26.284 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 200 2024-09-13T11:23:26.284 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.284 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:23:26.284 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.284 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:23:26.284 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316280+0000", 2024-09-13T11:23:26.284 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:26.284 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.284 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.284 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:23:26.284 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316280+0000", 2024-09-13T11:23:26.284 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:26.284 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.285 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.285 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:23:26.285 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316282+0000", 2024-09-13T11:23:26.285 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.497e-06 2024-09-13T11:23:26.285 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.285 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.285 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:23:26.285 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316411+0000", 2024-09-13T11:23:26.285 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00012924799999999999 2024-09-13T11:23:26.285 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.285 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.285 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:23:26.285 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316426+0000", 2024-09-13T11:23:26.286 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.4183e-05 2024-09-13T11:23:26.286 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.286 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.286 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:23:26.286 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316435+0000", 2024-09-13T11:23:26.286 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.0990000000000005e-06 2024-09-13T11:23:26.286 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.286 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.286 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:23:26.286 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.317250+0000", 2024-09-13T11:23:26.286 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00081501100000000003 2024-09-13T11:23:26.286 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.286 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.287 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-09-13T11:23:26.287 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.317257+0000", 2024-09-13T11:23:26.287 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.2169999999999997e-06 2024-09-13T11:23:26.287 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.287 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.287 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:23:26.287 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.191936+0000", 2024-09-13T11:23:26.287 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.87467922099999995 2024-09-13T11:23:26.287 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.287 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.287 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:23:26.287 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.192056+0000", 2024-09-13T11:23:26.287 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000119638 2024-09-13T11:23:26.288 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.288 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.288 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1", 2024-09-13T11:23:26.288 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.192184+0000", 2024-09-13T11:23:26.288 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00012813399999999999 2024-09-13T11:23:26.288 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.288 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.288 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:26.288 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.194324+0000", 2024-09-13T11:23:26.288 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0021399420000000001 2024-09-13T11:23:26.288 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.288 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.288 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:23:26.289 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.194802+0000", 2024-09-13T11:23:26.289 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00047798799999999999 2024-09-13T11:23:26.289 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.289 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.289 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:23:26.289 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.194951+0000", 2024-09-13T11:23:26.289 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0001494 2024-09-13T11:23:26.289 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.289 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.289 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:23:26.289 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.195072+0000", 2024-09-13T11:23:26.289 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00012115199999999999 2024-09-13T11:23:26.290 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:26.290 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:23:26.290 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:26.290 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.290 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.290 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:25385 3.7 3:e9a54943:::benchmark_data_smithi177_31116_object25384:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e55)", 2024-09-13T11:23:26.290 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:04.452404+0000", 2024-09-13T11:23:26.290 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 21.713522136000002, 2024-09-13T11:23:26.290 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.22598749700000001, 2024-09-13T11:23:26.290 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:23:26.290 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:23:26.290 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:26.290 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:23:26.291 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:23:26.291 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:26.291 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 25385 2024-09-13T11:23:26.291 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.291 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:23:26.291 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.291 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:23:26.291 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.452404+0000", 2024-09-13T11:23:26.291 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:26.291 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.291 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.291 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:23:26.292 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.452404+0000", 2024-09-13T11:23:26.292 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:26.292 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.292 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.292 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:23:26.292 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.452406+0000", 2024-09-13T11:23:26.292 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.57e-06 2024-09-13T11:23:26.292 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.292 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.292 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:23:26.292 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.452461+0000", 2024-09-13T11:23:26.292 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.4892000000000002e-05 2024-09-13T11:23:26.292 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.293 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.293 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:23:26.293 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.452473+0000", 2024-09-13T11:23:26.293 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.2703e-05 2024-09-13T11:23:26.293 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.293 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.293 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:23:26.293 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.452482+0000", 2024-09-13T11:23:26.293 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.6999999999999997e-06 2024-09-13T11:23:26.293 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.293 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.293 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:23:26.294 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.452605+0000", 2024-09-13T11:23:26.294 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00012328099999999999 2024-09-13T11:23:26.294 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.294 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.294 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:23:26.294 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.452930+0000", 2024-09-13T11:23:26.294 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00032516000000000001 2024-09-13T11:23:26.294 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.294 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.294 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 8", 2024-09-13T11:23:26.294 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.453272+0000", 2024-09-13T11:23:26.294 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00034195100000000001 2024-09-13T11:23:26.294 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.294 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.295 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:23:26.295 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.456573+0000", 2024-09-13T11:23:26.295 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0033003619999999998 2024-09-13T11:23:26.295 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.295 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.295 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:26.295 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.678212+0000", 2024-09-13T11:23:26.295 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.22163946600000001 2024-09-13T11:23:26.295 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.295 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.295 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:23:26.295 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.678325+0000", 2024-09-13T11:23:26.296 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00011263400000000001 2024-09-13T11:23:26.296 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.296 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.296 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:23:26.296 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.678392+0000", 2024-09-13T11:23:26.296 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.6778000000000003e-05 2024-09-13T11:23:26.296 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:26.296 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:23:26.296 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:26.296 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.296 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.296 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:25525 3.1 3:8e5634af:::benchmark_data_smithi177_31116_object25524:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e56)", 2024-09-13T11:23:26.296 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:04.981590+0000", 2024-09-13T11:23:26.297 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 21.184336256999998, 2024-09-13T11:23:26.297 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.23642401599999999, 2024-09-13T11:23:26.297 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:23:26.297 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:23:26.297 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:26.297 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:23:26.297 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:23:26.297 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:26.297 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 25525 2024-09-13T11:23:26.297 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.297 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:23:26.297 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.297 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:23:26.297 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.981590+0000", 2024-09-13T11:23:26.298 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:26.298 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.298 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.298 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:23:26.298 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.981590+0000", 2024-09-13T11:23:26.298 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:26.298 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.298 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.298 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:23:26.298 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.981598+0000", 2024-09-13T11:23:26.298 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.4230000000000005e-06 2024-09-13T11:23:26.298 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.298 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.298 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:23:26.299 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.981652+0000", 2024-09-13T11:23:26.299 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.3226000000000002e-05 2024-09-13T11:23:26.299 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.299 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.299 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:23:26.299 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.981680+0000", 2024-09-13T11:23:26.299 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.8348999999999999e-05 2024-09-13T11:23:26.299 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.299 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.299 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:23:26.299 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.981695+0000", 2024-09-13T11:23:26.299 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.5085999999999999e-05 2024-09-13T11:23:26.299 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.299 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.300 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:23:26.300 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.981809+0000", 2024-09-13T11:23:26.300 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00011382700000000001 2024-09-13T11:23:26.300 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.300 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.300 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:23:26.300 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.982042+0000", 2024-09-13T11:23:26.300 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000232697 2024-09-13T11:23:26.300 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.300 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.300 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 11", 2024-09-13T11:23:26.300 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.982261+0000", 2024-09-13T11:23:26.301 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00021918400000000001 2024-09-13T11:23:26.301 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.301 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.301 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:23:26.301 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.984553+0000", 2024-09-13T11:23:26.301 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0022918460000000002 2024-09-13T11:23:26.301 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.301 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.301 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:26.301 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:05.217842+0000", 2024-09-13T11:23:26.301 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.233289106 2024-09-13T11:23:26.301 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.301 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.302 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:23:26.302 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:05.217940+0000", 2024-09-13T11:23:26.302 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.7936999999999994e-05 2024-09-13T11:23:26.302 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.302 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.302 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:23:26.302 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:05.218014+0000", 2024-09-13T11:23:26.302 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.4334999999999999e-05 2024-09-13T11:23:26.302 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:26.302 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:23:26.302 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:26.302 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.302 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.302 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop_reply(client.4665.0:25525 3.1 e56/30 ondisk, result = 0)", 2024-09-13T11:23:26.303 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:04.985699+0000", 2024-09-13T11:23:26.303 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 21.180227525999999, 2024-09-13T11:23:26.303 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.23233051800000001, 2024-09-13T11:23:26.303 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:23:26.303 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:23:26.303 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-09-13T11:23:26.303 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:23:26.303 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.303 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:23:26.303 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.985699+0000", 2024-09-13T11:23:26.303 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:26.303 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.303 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.303 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:23:26.304 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.985699+0000", 2024-09-13T11:23:26.304 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:26.304 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.304 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.304 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:23:26.304 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.985702+0000", 2024-09-13T11:23:26.304 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.0570000000000001e-06 2024-09-13T11:23:26.304 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.304 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.304 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:23:26.304 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.985711+0000", 2024-09-13T11:23:26.304 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.1060000000000006e-06 2024-09-13T11:23:26.304 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.305 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.305 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:23:26.305 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:05.217667+0000", 2024-09-13T11:23:26.305 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.231955774 2024-09-13T11:23:26.305 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.305 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.305 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:23:26.305 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:05.217685+0000", 2024-09-13T11:23:26.305 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.791e-05 2024-09-13T11:23:26.305 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.305 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.305 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:23:26.305 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:05.217818+0000", 2024-09-13T11:23:26.305 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000133036 2024-09-13T11:23:26.306 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.306 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.306 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:23:26.306 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:05.217833+0000", 2024-09-13T11:23:26.306 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.519e-05 2024-09-13T11:23:26.306 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:26.306 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:26.306 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:23:26.306 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:05.218029+0000", 2024-09-13T11:23:26.306 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00019644500000000001 2024-09-13T11:23:26.306 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:26.306 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:23:26.306 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:26.307 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:26.307 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:23:26.307 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-09-13T11:23:26.308 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-13T11:23:26.309 INFO:tasks.ceph.osd.10.smithi162.stderr:2024-09-13T11:23:26.302+0000 7fb59c9f0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26317) UID: 0 2024-09-13T11:23:26.412 INFO:tasks.ceph.osd.10.smithi162.stderr:2024-09-13T11:23:26.402+0000 7fb59c9f0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26317) UID: 0 2024-09-13T11:23:26.445 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-13T11:23:26.446 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [], 2024-09-13T11:23:26.446 INFO:teuthology.orchestra.run.smithi162.stdout: "num_ops": 0 2024-09-13T11:23:26.446 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-13T11:23:26.460 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-13T11:23:26.510 INFO:tasks.ceph.osd.15.smithi177.stderr:2024-09-13T11:23:26.505+0000 7f9d6ce77640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26578) UID: 0 2024-09-13T11:23:26.614 INFO:tasks.ceph.osd.9.smithi092.stderr:2024-09-13T11:23:26.612+0000 7ff5ab3d0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26348) UID: 0 2024-09-13T11:23:26.647 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-13T11:23:26.647 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [], 2024-09-13T11:23:26.647 INFO:teuthology.orchestra.run.smithi162.stdout: "complaint_time": 30, 2024-09-13T11:23:26.647 INFO:teuthology.orchestra.run.smithi162.stdout: "num_blocked_ops": 0 2024-09-13T11:23:26.647 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-13T11:23:26.668 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-13T11:23:26.714 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-13T11:23:26.708+0000 7f6a68775640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26307) UID: 0 2024-09-13T11:23:26.817 INFO:tasks.ceph.osd.8.smithi001.stderr:2024-09-13T11:23:26.812+0000 7fb9b48ae640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26312) UID: 0 2024-09-13T11:23:26.818 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-13T11:23:26.818 INFO:teuthology.orchestra.run.smithi162.stdout: "size": 20, 2024-09-13T11:23:26.818 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 600, 2024-09-13T11:23:26.818 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [ 2024-09-13T11:23:26.819 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:26.819 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.4665.0:119 3.9 e41/30 3:98957a18:::benchmark_data_smithi177_31116_object118:head v 41'9, mlcod=33'6)", 2024-09-13T11:23:26.819 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:22:47.342628+0000", 2024-09-13T11:23:26.819 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 39.470372759999997, 2024-09-13T11:23:26.819 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.057932441000000001, 2024-09-13T11:23:26.819 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:23:26.819 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:23:26.819 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:26.819 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:23:26.819 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:26.819 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:23:26.819 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:47.342628+0000", 2024-09-13T11:23:26.819 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:26.820 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:26.820 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:26.820 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:23:26.820 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:47.342628+0000", 2024-09-13T11:23:26.820 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:26.820 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:26.820 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:26.820 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:23:26.820 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:47.342632+0000", 2024-09-13T11:23:26.820 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.5700000000000001e-06 2024-09-13T11:23:26.820 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:26.820 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:26.821 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:23:26.821 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:47.342705+0000", 2024-09-13T11:23:26.821 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.3220999999999996e-05 2024-09-13T11:23:26.821 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:26.821 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:26.821 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:23:26.821 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:47.398889+0000", 2024-09-13T11:23:26.821 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.056183956 2024-09-13T11:23:26.821 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:26.821 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:26.821 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:23:26.821 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:47.398904+0000", 2024-09-13T11:23:26.821 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4902999999999999e-05 2024-09-13T11:23:26.822 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:26.822 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:26.822 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:23:26.822 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:47.399009+0000", 2024-09-13T11:23:26.822 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000105037 2024-09-13T11:23:26.822 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:26.822 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:26.822 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:23:26.822 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:47.399047+0000", 2024-09-13T11:23:26.822 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.7474000000000001e-05 2024-09-13T11:23:26.822 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:26.822 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:26.822 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:23:26.823 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:47.400461+0000", 2024-09-13T11:23:26.823 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0014144400000000001 2024-09-13T11:23:26.823 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:26.823 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:26.823 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:23:26.823 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:47.400561+0000", 2024-09-13T11:23:26.823 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.9840000000000006e-05 2024-09-13T11:23:26.823 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:26.823 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:23:26.823 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:26.823 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:26.823 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:26.823 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.4665.0:171 3.9 e41/30 3:9e1c7eeb:::benchmark_data_smithi177_31116_object170:head v 41'13, mlcod=41'9)", 2024-09-13T11:23:26.823 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:22:47.418602+0000", 2024-09-13T11:23:26.824 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 39.394398993999999, 2024-09-13T11:23:26.824 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.67235367300000004, 2024-09-13T11:23:26.824 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:23:26.824 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:23:26.824 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:26.824 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:23:26.824 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:26.824 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:23:26.824 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:47.418602+0000", 2024-09-13T11:23:26.824 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:26.824 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:26.824 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:26.824 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:23:26.825 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:47.418602+0000", 2024-09-13T11:23:26.825 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:26.825 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:26.825 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:26.825 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:23:26.825 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:47.418605+0000", 2024-09-13T11:23:26.825 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.2200000000000001e-06 2024-09-13T11:23:26.825 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:26.825 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:26.825 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:23:26.825 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:47.418668+0000", 2024-09-13T11:23:26.825 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.2921000000000003e-05 2024-09-13T11:23:26.826 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:26.826 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:26.826 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:23:26.826 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:48.089380+0000", 2024-09-13T11:23:26.826 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.67071191100000005 2024-09-13T11:23:26.826 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:26.826 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:26.826 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:23:26.826 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:48.089423+0000", 2024-09-13T11:23:26.826 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.2916e-05 2024-09-13T11:23:26.826 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:26.826 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:26.826 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:23:26.826 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:48.089533+0000", 2024-09-13T11:23:26.827 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000109996 2024-09-13T11:23:26.827 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:26.827 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:26.827 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:23:26.827 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:48.089570+0000", 2024-09-13T11:23:26.827 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.7314999999999998e-05 2024-09-13T11:23:26.827 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:26.827 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:26.827 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:23:26.827 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:48.090852+0000", 2024-09-13T11:23:26.827 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001281641 2024-09-13T11:23:26.827 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:26.827 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:26.828 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:23:26.828 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:48.090956+0000", 2024-09-13T11:23:26.828 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00010375299999999999 2024-09-13T11:23:26.828 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:26.828 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:23:26.828 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:26.828 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:26.828 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:26.828 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.4665.0:183 3.9 e41/30 3:94db3450:::benchmark_data_smithi177_31116_object182:head v 41'14, mlcod=41'11)", 2024-09-13T11:23:26.828 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:22:48.089206+0000", 2024-09-13T11:23:26.828 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 38.723795539000001, 2024-09-13T11:23:26.828 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.042973540999999997, 2024-09-13T11:23:26.828 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:23:26.829 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:23:26.829 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:26.829 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:23:26.829 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:26.829 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:23:26.829 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:48.089206+0000", 2024-09-13T11:23:26.829 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:26.829 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:26.829 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:26.829 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:23:26.829 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:48.089206+0000", 2024-09-13T11:23:26.829 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:26.829 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:26.830 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:26.830 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:23:26.830 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:48.089209+0000", 2024-09-13T11:23:26.830 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.726e-06 2024-09-13T11:23:26.830 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:26.830 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:26.830 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:23:26.830 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:48.089341+0000", 2024-09-13T11:23:26.830 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00013116000000000001 2024-09-13T11:23:26.830 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:26.830 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:26.830 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:23:26.830 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:48.130485+0000", 2024-09-13T11:23:26.830 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.041144095999999998 2024-09-13T11:23:26.831 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:26.831 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:26.831 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:23:26.831 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:48.130498+0000", 2024-09-13T11:23:26.831 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3436e-05 2024-09-13T11:23:26.831 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:26.831 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:26.831 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:23:26.831 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:48.130647+0000", 2024-09-13T11:23:26.831 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00014904100000000001 2024-09-13T11:23:26.831 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:26.831 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:26.831 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:23:26.831 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:48.130681+0000", 2024-09-13T11:23:26.832 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.3763000000000002e-05 2024-09-13T11:23:26.832 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:26.832 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:26.832 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:23:26.832 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:48.132077+0000", 2024-09-13T11:23:26.832 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001396266 2024-09-13T11:23:26.832 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:26.832 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:26.832 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:23:26.832 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:48.132179+0000", 2024-09-13T11:23:26.832 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00010205299999999999 2024-09-13T11:23:26.832 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:26.832 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:23:26.832 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:26.833 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:26.833 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:26.833 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.4665.0:915 3.9 e43/30 3:9b5a11fd:::benchmark_data_smithi177_31116_object914:head v 43'73, mlcod=43'70)", 2024-09-13T11:23:26.833 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:22:49.539211+0000", 2024-09-13T11:23:26.833 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 37.273789674, 2024-09-13T11:23:26.833 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.063010919999999998, 2024-09-13T11:23:26.833 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:23:26.833 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:23:26.833 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:26.833 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:23:26.833 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:26.833 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:23:26.833 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:49.539211+0000", 2024-09-13T11:23:26.833 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:26.834 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:26.834 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:26.834 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:23:26.834 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:49.539211+0000", 2024-09-13T11:23:26.834 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:26.834 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:26.834 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:26.834 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:23:26.834 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:49.539214+0000", 2024-09-13T11:23:26.834 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.4279999999999999e-06 2024-09-13T11:23:26.834 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:26.834 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:26.835 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:23:26.835 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:49.539257+0000", 2024-09-13T11:23:26.835 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.2648000000000002e-05 2024-09-13T11:23:26.835 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:26.835 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:26.835 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:23:26.835 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:49.600637+0000", 2024-09-13T11:23:26.835 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.061380067000000003 2024-09-13T11:23:26.835 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:26.835 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:26.835 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:23:26.835 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:49.600652+0000", 2024-09-13T11:23:26.836 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5727e-05 2024-09-13T11:23:26.836 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:26.836 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:26.836 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:23:26.836 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:49.600775+0000", 2024-09-13T11:23:26.836 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00012288199999999999 2024-09-13T11:23:26.836 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:26.836 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:26.836 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:23:26.836 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:49.600813+0000", 2024-09-13T11:23:26.836 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.8078999999999999e-05 2024-09-13T11:23:26.836 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:26.836 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:26.837 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:23:26.837 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:49.602102+0000", 2024-09-13T11:23:26.837 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001288443 2024-09-13T11:23:26.837 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:26.837 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:26.837 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:23:26.837 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:49.602222+0000", 2024-09-13T11:23:26.837 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000120646 2024-09-13T11:23:26.837 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:26.837 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:23:26.837 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:26.837 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:26.838 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:26.838 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.4665.0:16769 3.9 e52/30 3:919a172b:::benchmark_data_smithi177_31116_object16768:head v 52'1033, mlcod=52'1030)", 2024-09-13T11:23:26.838 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:22:59.009884+0000", 2024-09-13T11:23:26.838 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 27.803117465, 2024-09-13T11:23:26.838 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.041620747, 2024-09-13T11:23:26.838 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:23:26.838 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:23:26.838 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:26.838 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:23:26.838 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:26.838 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:23:26.838 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:59.009884+0000", 2024-09-13T11:23:26.838 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:26.839 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:26.839 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:26.839 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:23:26.839 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:59.009884+0000", 2024-09-13T11:23:26.839 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:26.839 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:26.839 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:26.839 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:23:26.839 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:59.009888+0000", 2024-09-13T11:23:26.839 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.9199999999999997e-06 2024-09-13T11:23:26.839 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:26.839 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:26.839 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:23:26.839 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:59.009984+0000", 2024-09-13T11:23:26.840 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.6207999999999998e-05 2024-09-13T11:23:26.840 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:26.840 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:26.840 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:23:26.840 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:59.048236+0000", 2024-09-13T11:23:26.840 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.038252183000000002 2024-09-13T11:23:26.840 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:26.840 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:26.840 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:23:26.840 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:59.048251+0000", 2024-09-13T11:23:26.840 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.503e-05 2024-09-13T11:23:26.840 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:26.840 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:26.841 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:23:26.841 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:59.048384+0000", 2024-09-13T11:23:26.841 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000133289 2024-09-13T11:23:26.841 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:26.841 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:26.841 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:23:26.841 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:59.048424+0000", 2024-09-13T11:23:26.841 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.9332000000000002e-05 2024-09-13T11:23:26.841 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:26.841 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:26.841 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:23:26.841 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:59.051412+0000", 2024-09-13T11:23:26.841 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.002988044 2024-09-13T11:23:26.841 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:26.842 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:26.842 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:23:26.842 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:59.051504+0000", 2024-09-13T11:23:26.842 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.2740999999999999e-05 2024-09-13T11:23:26.842 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:26.842 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:23:26.842 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:26.842 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:26.842 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:26.842 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.4665.0:30236 3.9 e59/30 3:965dc9f3:::benchmark_data_smithi177_31116_object30235:head v 59'1906, mlcod=59'1903)", 2024-09-13T11:23:26.842 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:09.035179+0000", 2024-09-13T11:23:26.843 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 17.777822225000001, 2024-09-13T11:23:26.843 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.049294138000000001, 2024-09-13T11:23:26.843 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:23:26.843 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:23:26.843 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:26.843 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:23:26.843 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:26.843 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:23:26.843 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.035179+0000", 2024-09-13T11:23:26.844 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:26.844 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:26.844 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:26.844 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:23:26.844 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.035179+0000", 2024-09-13T11:23:26.844 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:26.844 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:26.844 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:26.845 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:23:26.845 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.035181+0000", 2024-09-13T11:23:26.845 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.2280000000000001e-06 2024-09-13T11:23:26.845 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:26.845 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:26.845 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:23:26.845 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.035250+0000", 2024-09-13T11:23:26.845 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.8632000000000003e-05 2024-09-13T11:23:26.845 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:26.845 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:26.845 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:23:26.845 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.082879+0000", 2024-09-13T11:23:26.845 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.047629684999999998 2024-09-13T11:23:26.845 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:26.846 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:26.846 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:23:26.846 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.082888+0000", 2024-09-13T11:23:26.846 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.2630000000000003e-06 2024-09-13T11:23:26.846 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:26.846 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:26.846 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:23:26.846 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.082948+0000", 2024-09-13T11:23:26.846 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.0368999999999997e-05 2024-09-13T11:23:26.846 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:26.846 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:26.846 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:23:26.846 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.082966+0000", 2024-09-13T11:23:26.847 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.7564999999999999e-05 2024-09-13T11:23:26.847 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:26.847 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:26.847 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:23:26.847 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.084427+0000", 2024-09-13T11:23:26.847 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0014614960000000001 2024-09-13T11:23:26.847 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:26.847 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:26.847 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:23:26.847 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.084473+0000", 2024-09-13T11:23:26.847 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.5899999999999998e-05 2024-09-13T11:23:26.847 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:26.847 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:23:26.847 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:26.848 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:26.848 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:26.848 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.4665.0:38407 3.9 e63/30 3:9a75baaa:::benchmark_data_smithi177_31116_object38406:head v 63'2401, mlcod=63'2398)", 2024-09-13T11:23:26.848 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:14.703927+0000", 2024-09-13T11:23:26.848 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 12.109074383999999, 2024-09-13T11:23:26.848 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.049196825999999999, 2024-09-13T11:23:26.848 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:23:26.848 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:23:26.848 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:26.848 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:23:26.848 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:26.848 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:23:26.848 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:14.703927+0000", 2024-09-13T11:23:26.848 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:26.849 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:26.849 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:26.849 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:23:26.849 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:14.703927+0000", 2024-09-13T11:23:26.849 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:26.849 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:26.849 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:26.849 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:23:26.849 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:14.703936+0000", 2024-09-13T11:23:26.849 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.8230000000000001e-06 2024-09-13T11:23:26.849 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:26.849 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:26.849 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:23:26.850 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:14.703994+0000", 2024-09-13T11:23:26.850 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.8783000000000003e-05 2024-09-13T11:23:26.850 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:26.850 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:26.850 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:23:26.850 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:14.748652+0000", 2024-09-13T11:23:26.850 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.044657177999999999 2024-09-13T11:23:26.850 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:26.850 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:26.850 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:23:26.850 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:14.748667+0000", 2024-09-13T11:23:26.850 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5526e-05 2024-09-13T11:23:26.850 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:26.850 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:26.851 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:23:26.851 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:14.748822+0000", 2024-09-13T11:23:26.851 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000154778 2024-09-13T11:23:26.851 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:26.851 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:26.851 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:23:26.851 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:14.748885+0000", 2024-09-13T11:23:26.851 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.3602999999999994e-05 2024-09-13T11:23:26.851 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:26.851 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:26.851 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:23:26.851 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:14.753022+0000", 2024-09-13T11:23:26.851 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0041365550000000001 2024-09-13T11:23:26.851 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:26.852 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:26.852 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:23:26.852 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:14.753124+0000", 2024-09-13T11:23:26.852 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00010158 2024-09-13T11:23:26.852 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:26.852 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:23:26.852 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:26.852 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:26.852 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:26.852 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.4665.0:38580 3.9 e63/30 3:9ba6ecd4:::benchmark_data_smithi177_31116_object38579:head v 63'2413, mlcod=63'2409)", 2024-09-13T11:23:26.852 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:14.842663+0000", 2024-09-13T11:23:26.852 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 11.970337657, 2024-09-13T11:23:26.852 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.061908754000000003, 2024-09-13T11:23:26.852 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:23:26.853 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:23:26.853 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:26.853 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:23:26.853 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:26.853 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:23:26.853 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:14.842663+0000", 2024-09-13T11:23:26.853 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:26.853 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:26.853 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:26.853 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:23:26.853 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:14.842663+0000", 2024-09-13T11:23:26.853 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:26.853 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:26.853 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:26.854 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:23:26.854 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:14.842667+0000", 2024-09-13T11:23:26.854 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.8999999999999999e-06 2024-09-13T11:23:26.854 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:26.854 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:26.854 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:23:26.854 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:14.842762+0000", 2024-09-13T11:23:26.854 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.4758000000000003e-05 2024-09-13T11:23:26.854 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:26.854 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:26.854 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:23:26.854 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:14.901081+0000", 2024-09-13T11:23:26.854 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.058318983999999997 2024-09-13T11:23:26.855 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:26.855 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:26.855 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:23:26.855 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:14.901097+0000", 2024-09-13T11:23:26.855 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5401999999999999e-05 2024-09-13T11:23:26.855 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:26.855 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:26.855 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:23:26.855 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:14.901253+0000", 2024-09-13T11:23:26.855 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00015671500000000001 2024-09-13T11:23:26.855 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:26.855 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:26.855 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:23:26.856 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:14.901318+0000", 2024-09-13T11:23:26.856 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.5129999999999995e-05 2024-09-13T11:23:26.856 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:26.856 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:26.856 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:23:26.856 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:14.904473+0000", 2024-09-13T11:23:26.856 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0031549640000000001 2024-09-13T11:23:26.856 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:26.856 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:26.856 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:23:26.856 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:14.904572+0000", 2024-09-13T11:23:26.856 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.8900999999999994e-05 2024-09-13T11:23:26.856 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:26.856 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:23:26.857 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:26.857 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:26.857 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:26.857 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.4665.0:39821 3.9 e64/30 3:9a53a2fb:::benchmark_data_smithi177_31116_object39820:head v 64'2482, mlcod=64'2477)", 2024-09-13T11:23:26.857 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:15.690133+0000", 2024-09-13T11:23:26.857 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 11.122868132000001, 2024-09-13T11:23:26.857 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.053716400999999997, 2024-09-13T11:23:26.857 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:23:26.857 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:23:26.857 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:26.857 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:23:26.857 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:26.857 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:23:26.858 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:15.690133+0000", 2024-09-13T11:23:26.858 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:26.858 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:26.858 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:26.858 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:23:26.858 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:15.690133+0000", 2024-09-13T11:23:26.858 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:26.858 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:26.858 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:26.858 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:23:26.858 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:15.690136+0000", 2024-09-13T11:23:26.858 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.2219999999999998e-06 2024-09-13T11:23:26.858 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:26.858 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:26.859 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:23:26.859 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:15.690197+0000", 2024-09-13T11:23:26.859 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.122e-05 2024-09-13T11:23:26.859 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:26.859 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:26.859 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:23:26.859 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:15.740573+0000", 2024-09-13T11:23:26.859 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.050375939000000002 2024-09-13T11:23:26.859 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:26.859 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:26.859 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:23:26.859 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:15.740586+0000", 2024-09-13T11:23:26.859 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2401999999999999e-05 2024-09-13T11:23:26.859 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:26.859 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:26.860 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:23:26.860 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:15.740705+0000", 2024-09-13T11:23:26.860 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00011892699999999999 2024-09-13T11:23:26.860 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:26.860 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:26.860 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:23:26.860 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:15.740741+0000", 2024-09-13T11:23:26.860 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.5797999999999998e-05 2024-09-13T11:23:26.860 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:26.860 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:26.860 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:23:26.860 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:15.743746+0000", 2024-09-13T11:23:26.860 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0030052220000000001 2024-09-13T11:23:26.860 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:26.861 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:26.861 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:23:26.861 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:15.743849+0000", 2024-09-13T11:23:26.861 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000103671 2024-09-13T11:23:26.861 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:26.861 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:23:26.861 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:26.861 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:26.861 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:26.861 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.4665.0:39894 3.9 e64/30 3:9190c04c:::benchmark_data_smithi177_31116_object39893:head v 64'2485, mlcod=64'2479)", 2024-09-13T11:23:26.861 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:15.747823+0000", 2024-09-13T11:23:26.861 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 11.065178181, 2024-09-13T11:23:26.861 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.041695300999999997, 2024-09-13T11:23:26.861 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:23:26.861 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:23:26.862 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:26.862 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:23:26.862 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:26.862 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:23:26.862 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:15.747823+0000", 2024-09-13T11:23:26.862 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:26.862 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:26.862 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:26.862 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:23:26.862 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:15.747823+0000", 2024-09-13T11:23:26.862 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:26.862 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:26.862 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:26.863 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:23:26.863 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:15.747824+0000", 2024-09-13T11:23:26.863 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.268e-06 2024-09-13T11:23:26.863 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:26.863 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:26.863 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:23:26.863 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:15.747899+0000", 2024-09-13T11:23:26.863 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.5185999999999995e-05 2024-09-13T11:23:26.863 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:26.863 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:26.863 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:23:26.863 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:15.785774+0000", 2024-09-13T11:23:26.863 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.037874554999999997 2024-09-13T11:23:26.863 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:26.864 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:26.864 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:23:26.864 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:15.785789+0000", 2024-09-13T11:23:26.864 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4883e-05 2024-09-13T11:23:26.864 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:26.864 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:26.864 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:23:26.864 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:15.785937+0000", 2024-09-13T11:23:26.864 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000147699 2024-09-13T11:23:26.864 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:26.864 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:26.864 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:23:26.864 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:15.786000+0000", 2024-09-13T11:23:26.865 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.3280000000000004e-05 2024-09-13T11:23:26.865 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:26.865 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:26.865 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:23:26.865 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:15.789412+0000", 2024-09-13T11:23:26.865 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0034117330000000001 2024-09-13T11:23:26.865 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:26.865 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:26.865 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:23:26.865 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:15.789518+0000", 2024-09-13T11:23:26.865 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000106697 2024-09-13T11:23:26.865 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:26.865 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:23:26.865 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:26.865 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:26.866 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:26.866 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.4665.0:39960 3.9 e64/30 3:9ee3e6ef:::benchmark_data_smithi177_31116_object39959:head v 64'2487, mlcod=64'2482)", 2024-09-13T11:23:26.866 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:15.804811+0000", 2024-09-13T11:23:26.866 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 11.008190151999999, 2024-09-13T11:23:26.866 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.070576958999999995, 2024-09-13T11:23:26.866 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:23:26.866 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:23:26.866 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:26.866 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:23:26.866 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:26.866 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:23:26.866 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:15.804811+0000", 2024-09-13T11:23:26.866 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:26.866 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:26.867 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:26.867 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:23:26.867 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:15.804811+0000", 2024-09-13T11:23:26.867 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:26.867 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:26.867 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:26.867 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:23:26.867 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:15.804815+0000", 2024-09-13T11:23:26.867 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.1330000000000001e-06 2024-09-13T11:23:26.867 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:26.867 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:26.867 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:23:26.867 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:15.804890+0000", 2024-09-13T11:23:26.868 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.5096000000000001e-05 2024-09-13T11:23:26.868 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:26.868 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:26.868 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:23:26.868 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:15.872369+0000", 2024-09-13T11:23:26.868 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.067478350000000006 2024-09-13T11:23:26.868 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:26.868 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:26.868 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:23:26.868 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:15.872387+0000", 2024-09-13T11:23:26.868 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.8315e-05 2024-09-13T11:23:26.868 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:26.868 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:26.869 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:23:26.869 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:15.872516+0000", 2024-09-13T11:23:26.869 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00012954000000000001 2024-09-13T11:23:26.869 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:26.869 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:26.869 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:23:26.869 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:15.872556+0000", 2024-09-13T11:23:26.869 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.0055000000000002e-05 2024-09-13T11:23:26.869 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:26.869 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:26.869 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:23:26.869 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:15.875302+0000", 2024-09-13T11:23:26.869 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0027456149999999999 2024-09-13T11:23:26.869 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:26.870 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:26.870 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:23:26.870 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:15.875388+0000", 2024-09-13T11:23:26.870 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.5854999999999997e-05 2024-09-13T11:23:26.870 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:26.870 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:23:26.870 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:26.870 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:26.870 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:26.870 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.4665.0:40059 3.9 e64/30 3:9817045a:::benchmark_data_smithi177_31116_object40058:head v 64'2496, mlcod=64'2490)", 2024-09-13T11:23:26.870 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:15.898342+0000", 2024-09-13T11:23:26.870 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 10.914658887, 2024-09-13T11:23:26.870 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.061227383000000003, 2024-09-13T11:23:26.871 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:23:26.871 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:23:26.871 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:26.871 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:23:26.871 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:26.871 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:23:26.871 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:15.898342+0000", 2024-09-13T11:23:26.871 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:26.871 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:26.871 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:26.871 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:23:26.871 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:15.898342+0000", 2024-09-13T11:23:26.871 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:26.871 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:26.872 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:26.872 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:23:26.872 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:15.898345+0000", 2024-09-13T11:23:26.872 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.1939999999999998e-06 2024-09-13T11:23:26.872 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:26.872 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:26.872 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:23:26.872 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:15.898419+0000", 2024-09-13T11:23:26.872 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.3739000000000007e-05 2024-09-13T11:23:26.872 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:26.872 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:26.872 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:23:26.872 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:15.955391+0000", 2024-09-13T11:23:26.872 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.056971567000000001 2024-09-13T11:23:26.873 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:26.873 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:26.873 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:23:26.873 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:15.955405+0000", 2024-09-13T11:23:26.873 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4662e-05 2024-09-13T11:23:26.873 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:26.873 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:26.873 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:23:26.873 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:15.955569+0000", 2024-09-13T11:23:26.873 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000163265 2024-09-13T11:23:26.873 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:26.873 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:26.873 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:23:26.873 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:15.955630+0000", 2024-09-13T11:23:26.874 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.1643000000000006e-05 2024-09-13T11:23:26.874 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:26.874 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:26.874 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:23:26.874 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:15.959488+0000", 2024-09-13T11:23:26.874 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0038577009999999998 2024-09-13T11:23:26.874 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:26.874 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:26.874 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:23:26.874 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:15.959570+0000", 2024-09-13T11:23:26.874 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.1612000000000004e-05 2024-09-13T11:23:26.874 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:26.874 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:23:26.874 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:26.874 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:26.875 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:26.875 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.4665.0:48216 3.9 e71/67 3:927b9ff4:::benchmark_data_smithi177_31116_object48215:head v 71'2887, mlcod=71'2883)", 2024-09-13T11:23:26.875 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:24.835666+0000", 2024-09-13T11:23:26.875 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 1.9773353550000001, 2024-09-13T11:23:26.875 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.085651284999999994, 2024-09-13T11:23:26.875 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:23:26.875 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:23:26.875 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:26.875 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:23:26.875 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:26.875 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:23:26.875 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:24.835666+0000", 2024-09-13T11:23:26.875 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:26.876 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:26.876 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:26.876 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:23:26.876 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:24.835666+0000", 2024-09-13T11:23:26.876 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:26.876 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:26.876 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:26.876 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:23:26.876 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:24.835668+0000", 2024-09-13T11:23:26.876 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.778e-06 2024-09-13T11:23:26.876 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:26.876 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:26.876 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:23:26.877 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:24.835723+0000", 2024-09-13T11:23:26.877 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.5433999999999998e-05 2024-09-13T11:23:26.877 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:26.877 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:26.877 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:23:26.877 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:24.919015+0000", 2024-09-13T11:23:26.877 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.083291986999999998 2024-09-13T11:23:26.877 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:26.877 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:26.877 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:23:26.877 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:24.919025+0000", 2024-09-13T11:23:26.878 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0234000000000001e-05 2024-09-13T11:23:26.878 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:26.878 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:26.878 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:23:26.878 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:24.919115+0000", 2024-09-13T11:23:26.878 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.9969000000000005e-05 2024-09-13T11:23:26.878 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:26.878 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:26.878 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:23:26.878 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:24.919151+0000", 2024-09-13T11:23:26.878 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.6170000000000001e-05 2024-09-13T11:23:26.878 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:26.878 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:26.879 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:23:26.879 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:24.921265+0000", 2024-09-13T11:23:26.879 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0021133380000000002 2024-09-13T11:23:26.879 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:26.879 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:26.879 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:23:26.879 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:24.921317+0000", 2024-09-13T11:23:26.879 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.2374999999999999e-05 2024-09-13T11:23:26.879 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:26.879 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:23:26.879 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:26.879 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:26.879 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:26.879 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPushReply(3.19 71/70 [PushReplyOp(3:998ae847:::benchmark_data_smithi177_31116_object4216:head)])", 2024-09-13T11:23:26.880 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:24.931374+0000", 2024-09-13T11:23:26.880 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 1.8816272139999999, 2024-09-13T11:23:26.880 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.073131357999999994, 2024-09-13T11:23:26.880 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:23:26.880 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:23:26.880 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-09-13T11:23:26.880 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:23:26.880 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:26.880 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:23:26.880 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:24.931374+0000", 2024-09-13T11:23:26.880 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:26.880 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:26.880 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:26.881 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:23:26.881 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:24.931374+0000", 2024-09-13T11:23:26.881 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:26.881 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:26.881 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:26.881 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:23:26.881 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:24.931375+0000", 2024-09-13T11:23:26.881 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1820000000000001e-06 2024-09-13T11:23:26.881 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:26.881 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:26.881 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:23:26.881 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:24.931380+0000", 2024-09-13T11:23:26.881 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.0540000000000002e-06 2024-09-13T11:23:26.881 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:26.882 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:26.882 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:23:26.882 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.002979+0000", 2024-09-13T11:23:26.882 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.071598740999999994 2024-09-13T11:23:26.882 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:26.882 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:26.882 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:23:26.882 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.003001+0000", 2024-09-13T11:23:26.882 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.2402e-05 2024-09-13T11:23:26.882 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:26.882 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:26.882 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:23:26.882 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.003125+0000", 2024-09-13T11:23:26.883 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00012376499999999999 2024-09-13T11:23:26.883 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:26.883 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:26.883 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:23:26.883 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.004505+0000", 2024-09-13T11:23:26.883 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0013802140000000001 2024-09-13T11:23:26.883 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:26.883 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:23:26.883 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:26.883 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:26.883 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:26.883 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPushReply(3.19 71/70 [PushReplyOp(3:9f04fe53:::benchmark_data_smithi177_31116_object4225:head)])", 2024-09-13T11:23:26.883 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:25.012367+0000", 2024-09-13T11:23:26.883 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 1.8006338720000001, 2024-09-13T11:23:26.884 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.072652915999999998, 2024-09-13T11:23:26.884 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:23:26.884 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:23:26.884 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-09-13T11:23:26.884 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:23:26.884 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:26.884 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:23:26.884 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.012367+0000", 2024-09-13T11:23:26.884 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:26.884 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:26.884 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:26.884 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:23:26.884 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.012367+0000", 2024-09-13T11:23:26.885 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:26.885 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:26.885 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:26.885 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:23:26.885 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.012370+0000", 2024-09-13T11:23:26.885 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.4760000000000001e-06 2024-09-13T11:23:26.885 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:26.885 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:26.885 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:23:26.885 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.012378+0000", 2024-09-13T11:23:26.885 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.2649999999999996e-06 2024-09-13T11:23:26.885 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:26.885 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:26.885 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:23:26.886 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.083643+0000", 2024-09-13T11:23:26.886 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.071265257999999998 2024-09-13T11:23:26.886 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:26.886 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:26.886 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:23:26.886 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.083655+0000", 2024-09-13T11:23:26.886 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2004999999999999e-05 2024-09-13T11:23:26.886 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:26.886 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:26.886 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:23:26.886 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.083753+0000", 2024-09-13T11:23:26.886 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.7572999999999996e-05 2024-09-13T11:23:26.886 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:26.886 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:26.887 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:23:26.887 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.085020+0000", 2024-09-13T11:23:26.887 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0012673389999999999 2024-09-13T11:23:26.887 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:26.887 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:23:26.887 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:26.887 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:26.887 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:26.887 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPushReply(3.19 71/70 [PushReplyOp(3:980058cd:::benchmark_data_smithi177_31116_object4309:head)])", 2024-09-13T11:23:26.887 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:25.123140+0000", 2024-09-13T11:23:26.887 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 1.6898613149999999, 2024-09-13T11:23:26.887 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.16436381899999999, 2024-09-13T11:23:26.887 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:23:26.887 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:23:26.888 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-09-13T11:23:26.888 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:23:26.888 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:26.888 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:23:26.888 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.123140+0000", 2024-09-13T11:23:26.888 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:26.888 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:26.888 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:26.888 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:23:26.888 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.123140+0000", 2024-09-13T11:23:26.888 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:26.888 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:26.888 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:26.888 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:23:26.889 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.123141+0000", 2024-09-13T11:23:26.889 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.562e-06 2024-09-13T11:23:26.889 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:26.889 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:26.889 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:23:26.889 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.123151+0000", 2024-09-13T11:23:26.889 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.3030000000000005e-06 2024-09-13T11:23:26.889 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:26.889 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:26.889 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:23:26.889 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.285828+0000", 2024-09-13T11:23:26.889 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.16267689399999999 2024-09-13T11:23:26.889 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:26.889 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:26.889 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:23:26.890 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.285838+0000", 2024-09-13T11:23:26.890 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0016e-05 2024-09-13T11:23:26.890 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:26.890 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:26.890 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:23:26.890 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.285966+0000", 2024-09-13T11:23:26.890 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00012870500000000001 2024-09-13T11:23:26.890 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:26.890 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:26.890 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:23:26.890 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.287504+0000", 2024-09-13T11:23:26.890 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001537339 2024-09-13T11:23:26.890 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:26.891 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:23:26.891 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:26.891 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:26.891 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:26.891 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:48909 3.19 3:9ef23a04:::benchmark_data_smithi177_31116_object48908:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e71)", 2024-09-13T11:23:26.891 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:25.454486+0000", 2024-09-13T11:23:26.891 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 1.3585154509999999, 2024-09-13T11:23:26.891 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.072962329000000006, 2024-09-13T11:23:26.891 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:23:26.891 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:23:26.891 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:26.891 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:23:26.891 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:23:26.891 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:26.892 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 48909 2024-09-13T11:23:26.892 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:26.892 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:23:26.892 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:26.892 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:23:26.892 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.454486+0000", 2024-09-13T11:23:26.892 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:26.892 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:26.892 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:26.892 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:23:26.892 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.454486+0000", 2024-09-13T11:23:26.892 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:26.892 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:26.893 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:26.893 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:23:26.893 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.454488+0000", 2024-09-13T11:23:26.893 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.723e-06 2024-09-13T11:23:26.893 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:26.893 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:26.893 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:23:26.893 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.454567+0000", 2024-09-13T11:23:26.893 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.8671000000000006e-05 2024-09-13T11:23:26.893 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:26.893 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:26.893 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:23:26.893 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.454590+0000", 2024-09-13T11:23:26.894 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.2483000000000001e-05 2024-09-13T11:23:26.894 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:26.894 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:26.894 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:23:26.894 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.454602+0000", 2024-09-13T11:23:26.894 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2709e-05 2024-09-13T11:23:26.894 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:26.894 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:26.894 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:23:26.894 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.454672+0000", 2024-09-13T11:23:26.894 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.9487999999999996e-05 2024-09-13T11:23:26.894 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:26.894 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:26.895 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:23:26.895 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.454816+0000", 2024-09-13T11:23:26.895 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00014411 2024-09-13T11:23:26.895 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:26.895 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:26.895 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 1,9", 2024-09-13T11:23:26.895 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.454952+0000", 2024-09-13T11:23:26.895 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00013627999999999999 2024-09-13T11:23:26.895 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:26.895 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:26.895 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:23:26.895 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.456063+0000", 2024-09-13T11:23:26.895 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001110411 2024-09-13T11:23:26.895 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:26.896 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:26.896 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:26.896 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.458692+0000", 2024-09-13T11:23:26.896 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00262962 2024-09-13T11:23:26.896 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:26.896 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:26.896 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:26.896 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.526869+0000", 2024-09-13T11:23:26.896 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.068176666999999996 2024-09-13T11:23:26.896 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:26.896 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:26.896 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:23:26.896 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.526930+0000", 2024-09-13T11:23:26.897 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.0822000000000001e-05 2024-09-13T11:23:26.897 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:26.897 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:26.897 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:23:26.897 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.527448+0000", 2024-09-13T11:23:26.897 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00051834500000000003 2024-09-13T11:23:26.897 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:26.897 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:23:26.897 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:26.897 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:26.897 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:26.897 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop_reply(client.4665.0:48909 3.19 e71/70 ondisk, result = 0)", 2024-09-13T11:23:26.897 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:25.458751+0000", 2024-09-13T11:23:26.897 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 1.354250062, 2024-09-13T11:23:26.898 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.068704475000000001, 2024-09-13T11:23:26.898 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:23:26.898 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:23:26.898 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:23:26.898 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:23:26.898 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:26.898 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:23:26.898 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.458751+0000", 2024-09-13T11:23:26.898 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:26.898 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:26.898 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:26.898 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:23:26.898 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.458751+0000", 2024-09-13T11:23:26.899 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:26.899 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:26.899 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:26.899 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:23:26.899 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.458752+0000", 2024-09-13T11:23:26.899 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.2200000000000003e-07 2024-09-13T11:23:26.899 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:26.899 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:26.899 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:23:26.899 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.458755+0000", 2024-09-13T11:23:26.899 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.377e-06 2024-09-13T11:23:26.899 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:26.899 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:26.900 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:23:26.900 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.526795+0000", 2024-09-13T11:23:26.900 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.068040001000000003 2024-09-13T11:23:26.900 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:26.900 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:26.900 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:23:26.900 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.526802+0000", 2024-09-13T11:23:26.900 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.3620000000000002e-06 2024-09-13T11:23:26.900 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:26.900 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:26.900 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:23:26.900 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.526857+0000", 2024-09-13T11:23:26.900 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.5087999999999998e-05 2024-09-13T11:23:26.901 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:26.901 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:26.901 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:23:26.901 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.526864+0000", 2024-09-13T11:23:26.901 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.6109999999999996e-06 2024-09-13T11:23:26.901 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:26.901 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:26.901 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:23:26.901 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.527456+0000", 2024-09-13T11:23:26.901 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00059121399999999995 2024-09-13T11:23:26.901 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:26.901 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:23:26.901 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:26.901 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:26.902 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:26.902 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPushReply(3.19 71/70 [PushReplyOp(3:9b41e30d:::benchmark_data_smithi177_31116_object4694:head)])", 2024-09-13T11:23:26.902 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:25.466329+0000", 2024-09-13T11:23:26.902 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 1.3466717340000001, 2024-09-13T11:23:26.902 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.084015425000000005, 2024-09-13T11:23:26.902 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:23:26.902 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:23:26.902 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-09-13T11:23:26.902 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:23:26.902 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:26.902 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:23:26.902 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.466329+0000", 2024-09-13T11:23:26.902 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:26.902 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:26.903 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:26.903 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:23:26.903 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.466329+0000", 2024-09-13T11:23:26.903 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:26.903 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:26.903 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:26.903 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:23:26.903 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.466331+0000", 2024-09-13T11:23:26.903 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4500000000000001e-06 2024-09-13T11:23:26.903 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:26.903 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:26.903 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:23:26.903 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.466336+0000", 2024-09-13T11:23:26.903 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.5029999999999997e-06 2024-09-13T11:23:26.904 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:26.904 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:26.904 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:23:26.904 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.548447+0000", 2024-09-13T11:23:26.904 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.082110965999999994 2024-09-13T11:23:26.904 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:26.904 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:26.904 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:23:26.904 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.548469+0000", 2024-09-13T11:23:26.904 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.1628999999999999e-05 2024-09-13T11:23:26.904 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:26.904 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:26.904 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:23:26.904 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.548626+0000", 2024-09-13T11:23:26.905 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00015717699999999999 2024-09-13T11:23:26.905 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:26.905 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:26.905 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:23:26.905 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.550345+0000", 2024-09-13T11:23:26.905 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0017187000000000001 2024-09-13T11:23:26.905 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:26.905 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:23:26.905 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:26.905 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:26.905 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:26.905 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPushReply(3.19 72/70 [PushReplyOp(3:9ae6d658:::benchmark_data_smithi177_31116_object6031:head)])", 2024-09-13T11:23:26.905 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:26.072435+0000", 2024-09-13T11:23:26.905 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 0.74056611800000005, 2024-09-13T11:23:26.906 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.065378444999999993, 2024-09-13T11:23:26.906 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:23:26.906 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:23:26.906 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-09-13T11:23:26.906 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:23:26.906 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:26.906 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:23:26.906 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:26.072435+0000", 2024-09-13T11:23:26.906 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:26.906 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:26.906 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:26.906 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:23:26.907 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:26.072435+0000", 2024-09-13T11:23:26.907 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:26.907 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:26.907 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:26.907 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:23:26.907 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:26.072436+0000", 2024-09-13T11:23:26.907 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3039999999999999e-06 2024-09-13T11:23:26.907 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:26.907 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:26.907 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:23:26.907 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:26.072440+0000", 2024-09-13T11:23:26.907 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.2210000000000002e-06 2024-09-13T11:23:26.907 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:26.907 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:26.908 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:23:26.908 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:26.136932+0000", 2024-09-13T11:23:26.908 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.064492298000000003 2024-09-13T11:23:26.908 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:26.908 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:26.908 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:23:26.908 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:26.136939+0000", 2024-09-13T11:23:26.908 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.1369999999999996e-06 2024-09-13T11:23:26.908 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:26.908 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:26.908 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:23:26.908 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:26.136993+0000", 2024-09-13T11:23:26.908 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.4431000000000001e-05 2024-09-13T11:23:26.909 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:26.909 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:26.909 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:23:26.909 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:26.137813+0000", 2024-09-13T11:23:26.909 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00082005399999999997 2024-09-13T11:23:26.909 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:26.909 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:23:26.909 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:26.909 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:26.909 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:23:26.909 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-13T11:23:26.910 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-13T11:23:26.954 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-09-13T11:23:26.950+0000 7f97befc9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26315) UID: 0 2024-09-13T11:23:27.004 INFO:tasks.radosbench.radosbench.0.smithi177.stdout:2024-09-13T11:23:27.004198+0000 min lat: 0.00139681 max lat: 22.1835 avg lat: 0.0189102 2024-09-13T11:23:27.004 INFO:tasks.radosbench.radosbench.0.smithi177.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-13T11:23:27.005 INFO:tasks.radosbench.radosbench.0.smithi177.stdout: 60 16 50772 50756 52.865 75.5625 0.0165141 0.0189102 2024-09-13T11:23:27.021 INFO:tasks.ceph.osd.9.smithi092.stderr:2024-09-13T11:23:27.020+0000 7ff5ab3d0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26348) UID: 0 2024-09-13T11:23:27.057 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-13T11:23:27.057 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [], 2024-09-13T11:23:27.057 INFO:teuthology.orchestra.run.smithi162.stdout: "num_ops": 0 2024-09-13T11:23:27.057 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-13T11:23:27.077 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-13T11:23:27.122 INFO:tasks.ceph.osd.15.smithi177.stderr:2024-09-13T11:23:27.117+0000 7f9d6ce77640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26578) UID: 0 2024-09-13T11:23:27.225 INFO:tasks.ceph.osd.15.smithi177.stderr:2024-09-13T11:23:27.221+0000 7f9d6ce77640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26578) UID: 0 2024-09-13T11:23:27.265 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-13T11:23:27.265 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [], 2024-09-13T11:23:27.265 INFO:teuthology.orchestra.run.smithi162.stdout: "complaint_time": 30, 2024-09-13T11:23:27.265 INFO:teuthology.orchestra.run.smithi162.stdout: "num_blocked_ops": 0 2024-09-13T11:23:27.265 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-13T11:23:27.279 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T11:23:27.283 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-13T11:23:27.328 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-13T11:23:27.324+0000 7f6a68775640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26307) UID: 0 2024-09-13T11:23:27.428 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-13T11:23:27.428 INFO:teuthology.orchestra.run.smithi162.stdout: "size": 20, 2024-09-13T11:23:27.428 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 600, 2024-09-13T11:23:27.429 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [ 2024-09-13T11:23:27.429 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:27.429 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4257.0:80 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 262144~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-13T11:23:27.429 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:22:12.516065+0000", 2024-09-13T11:23:27.429 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 74.907395917000002, 2024-09-13T11:23:27.429 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.46461997999999999, 2024-09-13T11:23:27.429 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:23:27.429 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:23:27.429 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:27.429 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:23:27.429 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4257", 2024-09-13T11:23:27.429 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/1488131527", 2024-09-13T11:23:27.429 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 80 2024-09-13T11:23:27.430 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:27.430 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:23:27.430 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:27.430 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:23:27.430 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.516065+0000", 2024-09-13T11:23:27.430 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:27.430 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:27.430 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:27.430 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:23:27.430 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.516065+0000", 2024-09-13T11:23:27.430 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:27.430 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:27.430 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:27.431 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:23:27.431 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.516066+0000", 2024-09-13T11:23:27.431 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0869999999999999e-06 2024-09-13T11:23:27.431 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:27.431 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:27.431 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:23:27.431 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.516112+0000", 2024-09-13T11:23:27.431 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.5503000000000001e-05 2024-09-13T11:23:27.431 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:27.431 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:27.431 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:23:27.431 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.516128+0000", 2024-09-13T11:23:27.432 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5891000000000001e-05 2024-09-13T11:23:27.432 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:27.432 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:27.432 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:23:27.432 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.516138+0000", 2024-09-13T11:23:27.432 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.8120000000000002e-06 2024-09-13T11:23:27.432 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:27.432 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:27.432 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:23:27.432 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.520398+0000", 2024-09-13T11:23:27.432 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0042605450000000001 2024-09-13T11:23:27.432 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:27.432 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:27.433 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:23:27.433 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.520526+0000", 2024-09-13T11:23:27.433 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000128355 2024-09-13T11:23:27.433 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:27.433 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:27.433 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 0", 2024-09-13T11:23:27.433 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.520730+0000", 2024-09-13T11:23:27.433 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00020337799999999999 2024-09-13T11:23:27.433 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:27.433 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:27.433 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:23:27.433 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.522417+0000", 2024-09-13T11:23:27.433 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001687184 2024-09-13T11:23:27.434 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:27.434 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:27.434 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:27.434 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.980408+0000", 2024-09-13T11:23:27.434 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.45799068399999998 2024-09-13T11:23:27.434 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:27.434 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:27.434 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:23:27.434 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.980565+0000", 2024-09-13T11:23:27.434 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00015760899999999999 2024-09-13T11:23:27.434 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:27.434 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:27.435 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:23:27.435 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.980685+0000", 2024-09-13T11:23:27.435 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000119932 2024-09-13T11:23:27.435 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:27.435 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:23:27.435 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:27.435 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:27.435 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:27.435 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop_reply(client.4257.0:80 1.0 e24/23 ondisk, result = 0)", 2024-09-13T11:23:27.435 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:22:12.522938+0000", 2024-09-13T11:23:27.435 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 74.900522883999997, 2024-09-13T11:23:27.435 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.45776420600000001, 2024-09-13T11:23:27.435 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:23:27.436 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:23:27.436 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:23:27.436 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:23:27.436 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:27.436 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:23:27.436 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.522938+0000", 2024-09-13T11:23:27.436 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:27.436 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:27.436 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:27.436 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:23:27.436 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.522938+0000", 2024-09-13T11:23:27.436 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:27.436 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:27.436 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:27.437 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:23:27.437 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.522941+0000", 2024-09-13T11:23:27.437 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.0290000000000001e-06 2024-09-13T11:23:27.437 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:27.437 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:27.437 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:23:27.437 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.522950+0000", 2024-09-13T11:23:27.437 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.7499999999999992e-06 2024-09-13T11:23:27.437 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:27.437 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:27.437 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:23:27.437 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.980215+0000", 2024-09-13T11:23:27.437 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.45726523800000002 2024-09-13T11:23:27.438 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:27.438 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:27.438 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:23:27.438 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.980229+0000", 2024-09-13T11:23:27.438 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3549e-05 2024-09-13T11:23:27.438 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:27.438 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:27.438 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:23:27.438 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.980365+0000", 2024-09-13T11:23:27.438 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00013608700000000001 2024-09-13T11:23:27.438 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:27.438 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:27.438 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:23:27.438 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.980388+0000", 2024-09-13T11:23:27.439 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.3112000000000002e-05 2024-09-13T11:23:27.439 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:27.439 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:27.439 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:23:27.439 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.980702+0000", 2024-09-13T11:23:27.439 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00031444099999999999 2024-09-13T11:23:27.439 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:27.439 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:23:27.439 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:27.439 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:27.439 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:27.439 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:25378 3.8 3:19339ced:::benchmark_data_smithi177_31116_object25377:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e55)", 2024-09-13T11:23:27.439 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:04.442409+0000", 2024-09-13T11:23:27.439 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 22.981051828999998, 2024-09-13T11:23:27.440 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.064890903999999999, 2024-09-13T11:23:27.440 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:23:27.440 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:23:27.440 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:27.440 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:23:27.440 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:23:27.440 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:27.440 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 25378 2024-09-13T11:23:27.440 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:27.440 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:23:27.440 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:27.440 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:23:27.440 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.442409+0000", 2024-09-13T11:23:27.441 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:27.441 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:27.441 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:27.441 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:23:27.441 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.442409+0000", 2024-09-13T11:23:27.441 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:27.441 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:27.441 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:27.441 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:23:27.441 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.442415+0000", 2024-09-13T11:23:27.441 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.2800000000000003e-06 2024-09-13T11:23:27.441 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:27.441 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:27.442 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:23:27.442 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.442484+0000", 2024-09-13T11:23:27.442 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.8918000000000007e-05 2024-09-13T11:23:27.442 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:27.442 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:27.442 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:23:27.442 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.442515+0000", 2024-09-13T11:23:27.442 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.1489000000000003e-05 2024-09-13T11:23:27.442 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:27.442 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:27.442 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:23:27.442 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.442531+0000", 2024-09-13T11:23:27.442 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.6120000000000002e-05 2024-09-13T11:23:27.443 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:27.443 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:27.443 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:23:27.443 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.442664+0000", 2024-09-13T11:23:27.443 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00013289000000000001 2024-09-13T11:23:27.443 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:27.443 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:27.443 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:23:27.443 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.442915+0000", 2024-09-13T11:23:27.443 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00025073100000000002 2024-09-13T11:23:27.443 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:27.443 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:27.443 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 1", 2024-09-13T11:23:27.444 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.443137+0000", 2024-09-13T11:23:27.444 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00022200799999999999 2024-09-13T11:23:27.444 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:27.444 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:27.444 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:23:27.444 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.445390+0000", 2024-09-13T11:23:27.444 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.002252754 2024-09-13T11:23:27.444 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:27.444 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:27.444 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:27.444 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.507042+0000", 2024-09-13T11:23:27.444 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.061652271000000002 2024-09-13T11:23:27.444 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:27.445 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:27.445 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:23:27.445 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.507224+0000", 2024-09-13T11:23:27.445 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00018251500000000001 2024-09-13T11:23:27.445 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:27.445 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:27.445 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:23:27.445 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.507300+0000", 2024-09-13T11:23:27.445 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.5927999999999995e-05 2024-09-13T11:23:27.445 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:27.445 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:23:27.445 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:27.445 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:27.445 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:27.446 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:25416 3.8 3:1a1fbdd1:::benchmark_data_smithi177_31116_object25415:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e56)", 2024-09-13T11:23:27.446 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:04.512188+0000", 2024-09-13T11:23:27.446 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 22.911272917000002, 2024-09-13T11:23:27.446 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.210208697, 2024-09-13T11:23:27.446 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:23:27.446 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:23:27.446 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:27.446 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:23:27.446 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:23:27.446 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:27.446 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 25416 2024-09-13T11:23:27.446 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:27.446 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:23:27.446 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:27.447 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:23:27.447 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.512188+0000", 2024-09-13T11:23:27.447 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:27.447 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:27.447 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:27.447 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:23:27.447 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.512188+0000", 2024-09-13T11:23:27.447 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:27.447 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:27.447 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:27.447 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:23:27.447 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.512192+0000", 2024-09-13T11:23:27.448 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.2660000000000003e-06 2024-09-13T11:23:27.448 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:27.448 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:27.448 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:23:27.448 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.512247+0000", 2024-09-13T11:23:27.448 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.4889000000000003e-05 2024-09-13T11:23:27.448 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:27.448 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:27.448 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:23:27.448 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.512279+0000", 2024-09-13T11:23:27.448 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.1146000000000003e-05 2024-09-13T11:23:27.448 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:27.448 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:27.448 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:23:27.449 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.512297+0000", 2024-09-13T11:23:27.449 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.825e-05 2024-09-13T11:23:27.449 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:27.449 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:27.449 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:23:27.449 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.512377+0000", 2024-09-13T11:23:27.449 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.9875000000000004e-05 2024-09-13T11:23:27.449 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:27.449 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:27.449 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:23:27.449 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.512570+0000", 2024-09-13T11:23:27.449 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00019312100000000001 2024-09-13T11:23:27.449 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:27.450 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:27.450 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 1", 2024-09-13T11:23:27.450 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.512798+0000", 2024-09-13T11:23:27.450 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00022847899999999999 2024-09-13T11:23:27.450 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:27.450 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:27.450 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:23:27.450 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.514975+0000", 2024-09-13T11:23:27.450 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0021769979999999999 2024-09-13T11:23:27.450 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:27.450 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:27.450 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:27.450 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.722207+0000", 2024-09-13T11:23:27.450 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.20723217099999999 2024-09-13T11:23:27.451 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:27.451 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:27.451 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:23:27.451 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.722314+0000", 2024-09-13T11:23:27.451 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00010648400000000001 2024-09-13T11:23:27.451 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:27.451 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:27.451 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:23:27.451 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.722397+0000", 2024-09-13T11:23:27.451 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.3017999999999997e-05 2024-09-13T11:23:27.451 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:27.451 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:23:27.451 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:27.451 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:27.452 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:27.452 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop_reply(client.4665.0:25416 3.8 e56/40 ondisk, result = 0)", 2024-09-13T11:23:27.452 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:04.515635+0000", 2024-09-13T11:23:27.452 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 22.907826265000001, 2024-09-13T11:23:27.452 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.20677600500000001, 2024-09-13T11:23:27.452 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:23:27.452 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:23:27.452 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:23:27.452 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:23:27.452 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:27.452 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:23:27.452 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.515635+0000", 2024-09-13T11:23:27.452 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:27.453 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:27.453 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:27.453 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:23:27.453 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.515635+0000", 2024-09-13T11:23:27.453 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:27.453 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:27.453 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:27.453 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:23:27.453 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.515639+0000", 2024-09-13T11:23:27.453 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.619e-06 2024-09-13T11:23:27.453 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:27.453 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:27.453 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:23:27.453 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.515650+0000", 2024-09-13T11:23:27.454 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.111e-05 2024-09-13T11:23:27.454 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:27.454 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:27.454 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:23:27.454 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.722004+0000", 2024-09-13T11:23:27.454 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.206354805 2024-09-13T11:23:27.454 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:27.454 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:27.454 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:23:27.454 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.722017+0000", 2024-09-13T11:23:27.454 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2840000000000001e-05 2024-09-13T11:23:27.454 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:27.454 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:27.455 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:23:27.455 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.722186+0000", 2024-09-13T11:23:27.455 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00016898299999999999 2024-09-13T11:23:27.455 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:27.455 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:27.455 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:23:27.455 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.722199+0000", 2024-09-13T11:23:27.455 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.289e-05 2024-09-13T11:23:27.455 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:27.455 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:27.455 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:23:27.455 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.722411+0000", 2024-09-13T11:23:27.455 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00021175799999999999 2024-09-13T11:23:27.455 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:27.456 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:23:27.456 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:27.456 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:27.456 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:27.456 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:25432 3.8 3:19671730:::benchmark_data_smithi177_31116_object25431:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e56)", 2024-09-13T11:23:27.456 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:04.717754+0000", 2024-09-13T11:23:27.456 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 22.70570725, 2024-09-13T11:23:27.456 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.102412884, 2024-09-13T11:23:27.456 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:23:27.456 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:23:27.456 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:27.456 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:23:27.456 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:23:27.457 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:27.457 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 25432 2024-09-13T11:23:27.457 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:27.457 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:23:27.457 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:27.457 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:23:27.457 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.717754+0000", 2024-09-13T11:23:27.457 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:27.457 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:27.457 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:27.457 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:23:27.457 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.717754+0000", 2024-09-13T11:23:27.457 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:27.458 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:27.458 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:27.458 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:23:27.458 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.717759+0000", 2024-09-13T11:23:27.458 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.04e-06 2024-09-13T11:23:27.458 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:27.458 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:27.458 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:23:27.458 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.717820+0000", 2024-09-13T11:23:27.458 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.0900999999999998e-05 2024-09-13T11:23:27.458 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:27.458 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:27.458 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:23:27.459 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.717854+0000", 2024-09-13T11:23:27.459 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.3797000000000002e-05 2024-09-13T11:23:27.459 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:27.459 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:27.459 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:23:27.459 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.717871+0000", 2024-09-13T11:23:27.459 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.6898999999999999e-05 2024-09-13T11:23:27.459 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:27.459 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:27.459 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:23:27.459 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.717993+0000", 2024-09-13T11:23:27.459 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00012248399999999999 2024-09-13T11:23:27.459 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:27.459 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:27.459 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:23:27.460 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.718222+0000", 2024-09-13T11:23:27.460 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00022848500000000001 2024-09-13T11:23:27.460 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:27.460 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:27.460 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 1", 2024-09-13T11:23:27.460 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.718447+0000", 2024-09-13T11:23:27.460 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00022588400000000001 2024-09-13T11:23:27.460 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:27.460 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:27.460 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:23:27.460 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.720604+0000", 2024-09-13T11:23:27.460 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0021561430000000001 2024-09-13T11:23:27.460 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:27.461 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:27.461 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:27.461 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.820012+0000", 2024-09-13T11:23:27.461 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.099408924999999995 2024-09-13T11:23:27.461 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:27.461 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:27.461 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:23:27.461 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.820097+0000", 2024-09-13T11:23:27.461 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.4147999999999995e-05 2024-09-13T11:23:27.461 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:27.461 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:27.461 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:23:27.461 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.820167+0000", 2024-09-13T11:23:27.461 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.0178000000000005e-05 2024-09-13T11:23:27.462 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:27.462 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:23:27.462 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:27.462 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:27.462 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:27.462 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop_reply(client.4665.0:25432 3.8 e56/40 ondisk, result = 0)", 2024-09-13T11:23:27.462 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:04.721952+0000", 2024-09-13T11:23:27.462 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 22.701508909000001, 2024-09-13T11:23:27.462 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.098227587000000005, 2024-09-13T11:23:27.462 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:23:27.462 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:23:27.462 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:23:27.462 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:23:27.462 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:27.463 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:23:27.463 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.721952+0000", 2024-09-13T11:23:27.463 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:27.463 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:27.463 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:27.463 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:23:27.463 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.721952+0000", 2024-09-13T11:23:27.463 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:27.463 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:27.463 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:27.463 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:23:27.463 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.721956+0000", 2024-09-13T11:23:27.463 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.8130000000000002e-06 2024-09-13T11:23:27.463 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:27.464 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:27.464 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:23:27.464 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.721967+0000", 2024-09-13T11:23:27.464 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1229e-05 2024-09-13T11:23:27.464 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:27.464 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:27.464 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:23:27.464 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.819880+0000", 2024-09-13T11:23:27.464 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.097912274999999993 2024-09-13T11:23:27.464 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:27.464 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:27.464 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:23:27.464 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.819894+0000", 2024-09-13T11:23:27.464 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.416e-05 2024-09-13T11:23:27.465 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:27.465 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:27.465 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:23:27.465 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.819993+0000", 2024-09-13T11:23:27.465 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.8859999999999999e-05 2024-09-13T11:23:27.465 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:27.465 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:27.465 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:23:27.465 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.820005+0000", 2024-09-13T11:23:27.465 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1960000000000001e-05 2024-09-13T11:23:27.465 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:27.465 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:27.465 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:23:27.465 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.820180+0000", 2024-09-13T11:23:27.466 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00017529000000000001 2024-09-13T11:23:27.466 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:27.466 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:23:27.466 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:27.466 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:27.466 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:27.466 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:30833 3.8 3:1c203f8d:::benchmark_data_smithi177_31116_object30832:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e59)", 2024-09-13T11:23:27.466 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:09.506656+0000", 2024-09-13T11:23:27.466 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 17.916805366999998, 2024-09-13T11:23:27.466 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.224862282, 2024-09-13T11:23:27.466 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:23:27.466 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:23:27.466 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:27.467 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:23:27.467 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:23:27.467 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:27.467 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 30833 2024-09-13T11:23:27.467 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:27.467 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:23:27.467 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:27.467 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:23:27.467 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.506656+0000", 2024-09-13T11:23:27.467 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:27.467 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:27.467 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:27.467 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:23:27.467 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.506656+0000", 2024-09-13T11:23:27.468 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:27.468 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:27.468 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:27.468 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:23:27.468 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.506660+0000", 2024-09-13T11:23:27.468 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.6e-06 2024-09-13T11:23:27.468 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:27.468 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:27.468 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:23:27.468 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.506717+0000", 2024-09-13T11:23:27.468 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.6131999999999997e-05 2024-09-13T11:23:27.468 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:27.468 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:27.469 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:23:27.469 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.506751+0000", 2024-09-13T11:23:27.469 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.4746000000000002e-05 2024-09-13T11:23:27.469 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:27.469 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:27.469 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:23:27.469 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.506774+0000", 2024-09-13T11:23:27.469 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.2636000000000001e-05 2024-09-13T11:23:27.469 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:27.469 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:27.469 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:23:27.469 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.506878+0000", 2024-09-13T11:23:27.469 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000104057 2024-09-13T11:23:27.469 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:27.470 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:27.470 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:23:27.470 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.507052+0000", 2024-09-13T11:23:27.470 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00017402800000000001 2024-09-13T11:23:27.470 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:27.470 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:27.470 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 1", 2024-09-13T11:23:27.470 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.507217+0000", 2024-09-13T11:23:27.470 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00016516 2024-09-13T11:23:27.470 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:27.470 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:27.470 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:23:27.470 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.508992+0000", 2024-09-13T11:23:27.470 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0017743990000000001 2024-09-13T11:23:27.471 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:27.471 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:27.471 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:27.471 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.731429+0000", 2024-09-13T11:23:27.471 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.22243718000000001 2024-09-13T11:23:27.471 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:27.471 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:27.471 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:23:27.471 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.731479+0000", 2024-09-13T11:23:27.471 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.0146999999999999e-05 2024-09-13T11:23:27.471 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:27.471 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:27.471 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:23:27.472 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.731518+0000", 2024-09-13T11:23:27.472 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.9196999999999998e-05 2024-09-13T11:23:27.472 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:27.472 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:23:27.472 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:27.472 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:27.472 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:27.472 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop_reply(client.4665.0:30833 3.8 e59/40 ondisk, result = 0)", 2024-09-13T11:23:27.472 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:09.508907+0000", 2024-09-13T11:23:27.472 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 17.914554273, 2024-09-13T11:23:27.472 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.222618862, 2024-09-13T11:23:27.472 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:23:27.472 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:23:27.472 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:23:27.473 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:23:27.473 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:27.473 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:23:27.473 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.508907+0000", 2024-09-13T11:23:27.473 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:27.473 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:27.473 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:27.473 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:23:27.473 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.508907+0000", 2024-09-13T11:23:27.473 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:27.473 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:27.473 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:27.473 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:23:27.473 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.508910+0000", 2024-09-13T11:23:27.474 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.993e-06 2024-09-13T11:23:27.474 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:27.474 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:27.474 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:23:27.474 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.508918+0000", 2024-09-13T11:23:27.474 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.7000000000000008e-06 2024-09-13T11:23:27.474 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:27.474 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:27.474 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:23:27.474 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.731341+0000", 2024-09-13T11:23:27.474 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.22242308699999999 2024-09-13T11:23:27.474 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:27.474 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:27.475 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:23:27.475 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.731350+0000", 2024-09-13T11:23:27.475 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.3179999999999996e-06 2024-09-13T11:23:27.475 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:27.475 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:27.475 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:23:27.475 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.731418+0000", 2024-09-13T11:23:27.475 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.7586e-05 2024-09-13T11:23:27.475 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:27.475 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:27.475 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:23:27.475 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.731424+0000", 2024-09-13T11:23:27.475 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.844e-06 2024-09-13T11:23:27.475 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:27.475 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:27.476 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:23:27.476 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.731526+0000", 2024-09-13T11:23:27.476 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000101334 2024-09-13T11:23:27.476 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:27.476 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:23:27.476 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:27.476 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:27.476 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:27.476 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:30922 3.8 3:126029c8:::benchmark_data_smithi177_31116_object30921:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e59)", 2024-09-13T11:23:27.476 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:09.776775+0000", 2024-09-13T11:23:27.476 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 17.646686555999999, 2024-09-13T11:23:27.476 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.063352835999999996, 2024-09-13T11:23:27.476 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:23:27.477 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:23:27.477 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:27.477 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:23:27.477 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:23:27.477 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:27.477 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 30922 2024-09-13T11:23:27.477 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:27.477 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:23:27.477 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:27.477 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:23:27.477 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.776775+0000", 2024-09-13T11:23:27.477 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:27.477 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:27.478 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:27.478 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:23:27.478 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.776775+0000", 2024-09-13T11:23:27.478 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:27.478 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:27.478 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:27.478 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:23:27.478 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.776782+0000", 2024-09-13T11:23:27.478 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.6329999999999996e-06 2024-09-13T11:23:27.478 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:27.478 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:27.479 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:23:27.479 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.776815+0000", 2024-09-13T11:23:27.479 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.2857000000000001e-05 2024-09-13T11:23:27.479 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:27.479 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:27.479 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:23:27.479 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.776836+0000", 2024-09-13T11:23:27.479 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.1197999999999999e-05 2024-09-13T11:23:27.479 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:27.480 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:27.480 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:23:27.480 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.776848+0000", 2024-09-13T11:23:27.480 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1235e-05 2024-09-13T11:23:27.480 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:27.480 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:27.480 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:23:27.480 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.776931+0000", 2024-09-13T11:23:27.480 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.3172999999999998e-05 2024-09-13T11:23:27.480 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:27.480 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:27.480 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:23:27.480 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.777149+0000", 2024-09-13T11:23:27.481 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000218433 2024-09-13T11:23:27.481 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:27.481 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:27.481 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 1", 2024-09-13T11:23:27.481 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.777367+0000", 2024-09-13T11:23:27.481 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000217493 2024-09-13T11:23:27.481 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:27.481 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:27.481 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:23:27.481 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.779004+0000", 2024-09-13T11:23:27.481 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001637042 2024-09-13T11:23:27.481 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:27.481 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:27.481 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:27.482 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.839949+0000", 2024-09-13T11:23:27.482 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.060945680000000002 2024-09-13T11:23:27.482 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:27.482 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:27.482 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:23:27.482 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.840049+0000", 2024-09-13T11:23:27.482 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.9271000000000006e-05 2024-09-13T11:23:27.482 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:27.482 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:27.482 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:23:27.482 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.840127+0000", 2024-09-13T11:23:27.482 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.8820999999999996e-05 2024-09-13T11:23:27.482 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:27.482 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:23:27.483 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:27.483 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:27.483 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:27.483 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:48977 3.8 3:1030343e:::benchmark_data_smithi177_31116_object48976:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e71)", 2024-09-13T11:23:27.483 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:25.517299+0000", 2024-09-13T11:23:27.483 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 1.906161979, 2024-09-13T11:23:27.483 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.14514738599999999, 2024-09-13T11:23:27.483 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:23:27.483 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:23:27.483 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:27.483 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:23:27.483 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:23:27.483 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:27.483 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 48977 2024-09-13T11:23:27.484 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:27.484 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:23:27.484 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:27.484 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:23:27.484 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.517299+0000", 2024-09-13T11:23:27.484 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:27.484 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:27.484 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:27.484 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:23:27.484 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.517299+0000", 2024-09-13T11:23:27.484 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:27.484 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:27.484 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:27.485 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:23:27.485 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.517301+0000", 2024-09-13T11:23:27.485 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.7060000000000001e-06 2024-09-13T11:23:27.485 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:27.485 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:27.485 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:23:27.485 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.517354+0000", 2024-09-13T11:23:27.485 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.3148e-05 2024-09-13T11:23:27.485 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:27.485 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:27.485 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:23:27.485 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.517365+0000", 2024-09-13T11:23:27.485 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1389000000000001e-05 2024-09-13T11:23:27.486 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:27.486 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:27.486 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:23:27.486 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.517374+0000", 2024-09-13T11:23:27.486 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.6740000000000003e-06 2024-09-13T11:23:27.486 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:27.486 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:27.486 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:23:27.486 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.517454+0000", 2024-09-13T11:23:27.486 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.9968000000000004e-05 2024-09-13T11:23:27.486 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:27.486 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:27.486 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:23:27.487 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.517561+0000", 2024-09-13T11:23:27.487 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00010655 2024-09-13T11:23:27.487 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:27.487 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:27.487 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 1", 2024-09-13T11:23:27.487 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.517664+0000", 2024-09-13T11:23:27.487 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000103409 2024-09-13T11:23:27.487 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:27.487 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:27.487 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:23:27.487 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.521381+0000", 2024-09-13T11:23:27.487 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0037174019999999999 2024-09-13T11:23:27.487 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:27.488 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:27.488 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:27.488 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.662307+0000", 2024-09-13T11:23:27.488 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.14092524000000001 2024-09-13T11:23:27.488 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:27.488 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:27.488 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:23:27.488 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.662382+0000", 2024-09-13T11:23:27.488 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.5331000000000001e-05 2024-09-13T11:23:27.488 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:27.488 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:27.488 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:23:27.489 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.662447+0000", 2024-09-13T11:23:27.489 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.4568999999999998e-05 2024-09-13T11:23:27.489 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:27.489 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:23:27.489 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:27.489 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:27.489 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:27.489 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:49109 3.8 3:1587435f:::benchmark_data_smithi177_31116_object49108:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e71)", 2024-09-13T11:23:27.489 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:25.659607+0000", 2024-09-13T11:23:27.489 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 1.763853895, 2024-09-13T11:23:27.489 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.077972397999999998, 2024-09-13T11:23:27.489 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:23:27.489 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:23:27.489 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:27.490 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:23:27.490 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:23:27.490 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:27.490 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 49109 2024-09-13T11:23:27.490 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:27.490 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:23:27.490 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:27.490 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:23:27.490 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.659607+0000", 2024-09-13T11:23:27.490 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:27.490 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:27.490 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:27.490 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:23:27.491 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.659607+0000", 2024-09-13T11:23:27.491 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:27.491 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:27.491 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:27.491 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:23:27.491 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.659612+0000", 2024-09-13T11:23:27.491 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.5630000000000002e-06 2024-09-13T11:23:27.491 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:27.491 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:27.491 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:23:27.491 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.659667+0000", 2024-09-13T11:23:27.491 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.4898000000000002e-05 2024-09-13T11:23:27.491 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:27.492 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:27.492 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:23:27.492 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.659696+0000", 2024-09-13T11:23:27.492 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.9776000000000001e-05 2024-09-13T11:23:27.492 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:27.492 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:27.492 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:23:27.492 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.659719+0000", 2024-09-13T11:23:27.492 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.2106000000000001e-05 2024-09-13T11:23:27.492 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:27.492 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:27.492 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:23:27.492 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.659883+0000", 2024-09-13T11:23:27.493 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00016433499999999999 2024-09-13T11:23:27.493 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:27.493 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:27.493 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:23:27.493 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.660207+0000", 2024-09-13T11:23:27.493 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00032428099999999999 2024-09-13T11:23:27.493 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:27.493 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:27.493 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 1", 2024-09-13T11:23:27.493 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.660564+0000", 2024-09-13T11:23:27.493 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00035704500000000001 2024-09-13T11:23:27.493 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:27.493 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:27.494 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:23:27.494 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.662883+0000", 2024-09-13T11:23:27.494 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0023183829999999998 2024-09-13T11:23:27.494 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:27.494 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:27.494 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:27.494 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.737361+0000", 2024-09-13T11:23:27.494 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.074478279999999994 2024-09-13T11:23:27.494 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:27.494 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:27.494 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:23:27.494 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.737486+0000", 2024-09-13T11:23:27.494 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00012487500000000001 2024-09-13T11:23:27.495 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:27.495 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:27.495 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:23:27.495 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.737580+0000", 2024-09-13T11:23:27.495 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.3856000000000004e-05 2024-09-13T11:23:27.495 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:27.495 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:23:27.495 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:27.495 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:27.495 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:27.495 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:49273 3.8 3:17bb4794:::benchmark_data_smithi177_31116_object49272:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e71)", 2024-09-13T11:23:27.495 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:25.787492+0000", 2024-09-13T11:23:27.496 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 1.6359694569999998, 2024-09-13T11:23:27.496 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.073929259999999997, 2024-09-13T11:23:27.496 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:23:27.496 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:23:27.496 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:27.496 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:23:27.496 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:23:27.496 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:27.496 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 49273 2024-09-13T11:23:27.496 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:27.496 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:23:27.496 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:27.496 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:23:27.497 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.787492+0000", 2024-09-13T11:23:27.497 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:27.497 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:27.497 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:27.497 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:23:27.497 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.787492+0000", 2024-09-13T11:23:27.497 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:27.497 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:27.497 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:27.497 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:23:27.497 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.787494+0000", 2024-09-13T11:23:27.498 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.554e-06 2024-09-13T11:23:27.498 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:27.498 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:27.498 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:23:27.498 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.787551+0000", 2024-09-13T11:23:27.498 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.6499000000000002e-05 2024-09-13T11:23:27.498 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:27.498 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:27.498 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:23:27.498 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.787569+0000", 2024-09-13T11:23:27.498 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.8490000000000001e-05 2024-09-13T11:23:27.498 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:27.498 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:27.498 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:23:27.499 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.787578+0000", 2024-09-13T11:23:27.499 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.7639999999999994e-06 2024-09-13T11:23:27.499 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:27.499 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:27.499 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:23:27.499 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.787645+0000", 2024-09-13T11:23:27.499 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.7157999999999996e-05 2024-09-13T11:23:27.499 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:27.499 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:27.499 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:23:27.499 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.787818+0000", 2024-09-13T11:23:27.499 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00017271800000000001 2024-09-13T11:23:27.499 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:27.500 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:27.500 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 1", 2024-09-13T11:23:27.500 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.788024+0000", 2024-09-13T11:23:27.500 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00020625299999999999 2024-09-13T11:23:27.500 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:27.500 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:27.500 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:23:27.500 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.789566+0000", 2024-09-13T11:23:27.500 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001542086 2024-09-13T11:23:27.500 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:27.500 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:27.500 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:27.501 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.861283+0000", 2024-09-13T11:23:27.501 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.071717078000000004 2024-09-13T11:23:27.501 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:27.501 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:27.501 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:23:27.501 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.861362+0000", 2024-09-13T11:23:27.501 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.9066999999999994e-05 2024-09-13T11:23:27.501 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:27.501 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:27.501 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:23:27.501 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.861421+0000", 2024-09-13T11:23:27.501 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.8593e-05 2024-09-13T11:23:27.501 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:27.502 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:23:27.502 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:27.502 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:27.502 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:27.502 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:49280 3.8 3:11086105:::benchmark_data_smithi177_31116_object49279:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e71)", 2024-09-13T11:23:27.502 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:25.791647+0000", 2024-09-13T11:23:27.502 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 1.6318139700000001, 2024-09-13T11:23:27.502 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.083081314000000003, 2024-09-13T11:23:27.502 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:23:27.502 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:23:27.502 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:27.502 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:23:27.502 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:23:27.503 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:27.503 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 49280 2024-09-13T11:23:27.503 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:27.503 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:23:27.503 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:27.503 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:23:27.503 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.791647+0000", 2024-09-13T11:23:27.503 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:27.503 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:27.503 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:27.503 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:23:27.503 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.791647+0000", 2024-09-13T11:23:27.503 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:27.504 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:27.504 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:27.504 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:23:27.504 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.791652+0000", 2024-09-13T11:23:27.504 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.2220000000000001e-06 2024-09-13T11:23:27.504 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:27.504 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:27.504 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:23:27.504 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.791707+0000", 2024-09-13T11:23:27.504 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.5053999999999998e-05 2024-09-13T11:23:27.504 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:27.504 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:27.504 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:23:27.505 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.791731+0000", 2024-09-13T11:23:27.505 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.3261e-05 2024-09-13T11:23:27.505 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:27.505 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:27.505 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:23:27.505 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.791743+0000", 2024-09-13T11:23:27.505 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2037e-05 2024-09-13T11:23:27.505 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:27.505 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:27.505 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:23:27.505 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.793450+0000", 2024-09-13T11:23:27.505 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0017074830000000001 2024-09-13T11:23:27.505 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:27.506 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:27.506 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:23:27.506 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.793671+0000", 2024-09-13T11:23:27.506 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000220891 2024-09-13T11:23:27.506 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:27.506 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:27.506 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 1", 2024-09-13T11:23:27.506 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.793883+0000", 2024-09-13T11:23:27.506 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00021190200000000001 2024-09-13T11:23:27.506 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:27.506 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:27.506 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:23:27.506 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.795973+0000", 2024-09-13T11:23:27.507 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0020898380000000001 2024-09-13T11:23:27.507 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:27.507 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:27.507 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:27.507 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.874651+0000", 2024-09-13T11:23:27.507 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.078678389000000001 2024-09-13T11:23:27.507 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:27.507 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:27.507 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:23:27.507 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.874694+0000", 2024-09-13T11:23:27.507 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.3071999999999997e-05 2024-09-13T11:23:27.507 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:27.508 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:27.508 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:23:27.508 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.874728+0000", 2024-09-13T11:23:27.508 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.4165000000000002e-05 2024-09-13T11:23:27.508 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:27.508 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:23:27.508 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:27.508 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:27.508 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:27.508 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:49373 3.8 3:17a23476:::benchmark_data_smithi177_31116_object49372:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e71)", 2024-09-13T11:23:27.508 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:25.858495+0000", 2024-09-13T11:23:27.508 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 1.56496579, 2024-09-13T11:23:27.508 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.088819487000000003, 2024-09-13T11:23:27.509 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:23:27.509 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:23:27.509 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:27.509 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:23:27.509 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:23:27.509 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:27.509 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 49373 2024-09-13T11:23:27.509 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:27.509 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:23:27.509 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:27.509 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:23:27.509 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.858495+0000", 2024-09-13T11:23:27.509 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:27.510 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:27.510 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:27.510 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:23:27.510 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.858495+0000", 2024-09-13T11:23:27.510 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:27.510 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:27.510 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:27.510 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:23:27.510 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.858509+0000", 2024-09-13T11:23:27.510 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3195e-05 2024-09-13T11:23:27.510 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:27.511 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:27.511 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:23:27.511 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.858546+0000", 2024-09-13T11:23:27.511 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.7842999999999997e-05 2024-09-13T11:23:27.511 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:27.511 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:27.511 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:23:27.511 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.858562+0000", 2024-09-13T11:23:27.511 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.6030000000000001e-05 2024-09-13T11:23:27.511 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:27.511 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:27.511 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:23:27.511 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.858571+0000", 2024-09-13T11:23:27.512 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.4970000000000001e-06 2024-09-13T11:23:27.512 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:27.512 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:27.512 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:23:27.512 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.858724+0000", 2024-09-13T11:23:27.512 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00015293500000000001 2024-09-13T11:23:27.512 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:27.512 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:27.512 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:23:27.512 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.858934+0000", 2024-09-13T11:23:27.512 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00020973500000000001 2024-09-13T11:23:27.512 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:27.512 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:27.513 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 1", 2024-09-13T11:23:27.513 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.859197+0000", 2024-09-13T11:23:27.513 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00026385999999999998 2024-09-13T11:23:27.513 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:27.513 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:27.513 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:23:27.513 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.861435+0000", 2024-09-13T11:23:27.513 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.002237555 2024-09-13T11:23:27.513 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:27.513 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:27.513 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:27.513 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.947194+0000", 2024-09-13T11:23:27.513 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.085758501000000001 2024-09-13T11:23:27.513 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:27.514 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:27.514 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:23:27.514 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.947260+0000", 2024-09-13T11:23:27.514 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.6675999999999997e-05 2024-09-13T11:23:27.514 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:27.514 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:27.514 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:23:27.514 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.947315+0000", 2024-09-13T11:23:27.514 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.4660000000000002e-05 2024-09-13T11:23:27.514 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:27.514 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:23:27.514 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:27.514 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:27.514 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:27.515 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:49573 3.8 3:16fffb91:::benchmark_data_smithi177_31116_object49572:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e72)", 2024-09-13T11:23:27.515 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:26.017955+0000", 2024-09-13T11:23:27.515 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 1.405506184, 2024-09-13T11:23:27.515 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.063194602000000002, 2024-09-13T11:23:27.515 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:23:27.515 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:23:27.515 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:27.515 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:23:27.515 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:23:27.515 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:27.515 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 49573 2024-09-13T11:23:27.515 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:27.516 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:23:27.516 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:27.516 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:23:27.516 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:26.017955+0000", 2024-09-13T11:23:27.516 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:27.516 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:27.516 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:27.516 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:23:27.516 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:26.017955+0000", 2024-09-13T11:23:27.516 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:27.516 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:27.516 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:27.516 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:23:27.517 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:26.017958+0000", 2024-09-13T11:23:27.517 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.7269999999999999e-06 2024-09-13T11:23:27.517 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:27.517 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:27.517 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:23:27.517 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:26.018000+0000", 2024-09-13T11:23:27.517 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.2619e-05 2024-09-13T11:23:27.517 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:27.517 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:27.517 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:23:27.517 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:26.018021+0000", 2024-09-13T11:23:27.517 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.1162000000000002e-05 2024-09-13T11:23:27.518 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:27.518 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:27.518 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:23:27.518 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:26.018029+0000", 2024-09-13T11:23:27.518 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.554e-06 2024-09-13T11:23:27.518 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:27.518 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:27.518 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:23:27.518 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:26.018079+0000", 2024-09-13T11:23:27.518 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.9650000000000001e-05 2024-09-13T11:23:27.518 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:27.518 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:27.519 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:23:27.519 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:26.018186+0000", 2024-09-13T11:23:27.519 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00010737499999999999 2024-09-13T11:23:27.519 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:27.519 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:27.519 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 1", 2024-09-13T11:23:27.519 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:26.018284+0000", 2024-09-13T11:23:27.519 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.8411000000000004e-05 2024-09-13T11:23:27.519 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:27.519 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:27.519 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:23:27.519 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:26.019357+0000", 2024-09-13T11:23:27.520 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0010729050000000001 2024-09-13T11:23:27.520 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:27.520 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:27.520 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:27.520 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:26.081082+0000", 2024-09-13T11:23:27.520 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.061724723000000002 2024-09-13T11:23:27.520 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:27.520 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:27.520 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:23:27.520 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:26.081118+0000", 2024-09-13T11:23:27.520 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.6300000000000001e-05 2024-09-13T11:23:27.520 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:27.520 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:27.521 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:23:27.521 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:26.081150+0000", 2024-09-13T11:23:27.521 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.1176000000000001e-05 2024-09-13T11:23:27.521 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:27.521 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:23:27.521 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:27.521 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:27.521 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:27.521 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:49666 3.8 3:121be161:::benchmark_data_smithi177_31116_object49665:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e72)", 2024-09-13T11:23:27.521 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:26.078875+0000", 2024-09-13T11:23:27.521 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 1.3445860970000001, 2024-09-13T11:23:27.521 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.067728236999999997, 2024-09-13T11:23:27.522 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:23:27.522 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:23:27.522 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:27.522 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:23:27.522 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:23:27.522 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:27.522 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 49666 2024-09-13T11:23:27.522 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:27.522 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:23:27.522 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:27.522 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:23:27.522 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:26.078875+0000", 2024-09-13T11:23:27.522 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:27.523 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:27.523 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:27.523 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:23:27.523 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:26.078875+0000", 2024-09-13T11:23:27.523 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:27.523 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:27.523 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:27.523 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:23:27.523 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:26.078878+0000", 2024-09-13T11:23:27.523 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.134e-06 2024-09-13T11:23:27.523 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:27.523 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:27.523 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:23:27.523 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:26.078914+0000", 2024-09-13T11:23:27.524 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.6007999999999998e-05 2024-09-13T11:23:27.524 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:27.524 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:27.524 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:23:27.524 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:26.078928+0000", 2024-09-13T11:23:27.524 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4131000000000001e-05 2024-09-13T11:23:27.524 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:27.524 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:27.524 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:23:27.524 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:26.078935+0000", 2024-09-13T11:23:27.524 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.1729999999999998e-06 2024-09-13T11:23:27.524 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:27.525 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:27.525 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:23:27.525 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:26.078997+0000", 2024-09-13T11:23:27.525 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.1039999999999998e-05 2024-09-13T11:23:27.525 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:27.525 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:27.525 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:23:27.525 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:26.079134+0000", 2024-09-13T11:23:27.525 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00013730599999999999 2024-09-13T11:23:27.525 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:27.525 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:27.525 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 1", 2024-09-13T11:23:27.525 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:26.079227+0000", 2024-09-13T11:23:27.525 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.2813999999999996e-05 2024-09-13T11:23:27.526 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:27.526 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:27.526 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:23:27.526 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:26.080435+0000", 2024-09-13T11:23:27.526 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0012080159999999999 2024-09-13T11:23:27.526 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:27.526 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:27.526 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:27.526 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:26.146520+0000", 2024-09-13T11:23:27.526 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.066085099999999994 2024-09-13T11:23:27.526 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:27.526 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:27.526 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:23:27.527 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:26.146566+0000", 2024-09-13T11:23:27.527 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.5804999999999999e-05 2024-09-13T11:23:27.527 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:27.527 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:27.527 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:23:27.527 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:26.146603+0000", 2024-09-13T11:23:27.527 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.7710000000000003e-05 2024-09-13T11:23:27.527 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:27.527 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:23:27.527 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:27.527 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:27.527 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:27.527 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.4665.0:49741 3.10 e72/70 3:085ab63e:::benchmark_data_smithi177_31116_object49740:head v 72'2952, mlcod=45'353)", 2024-09-13T11:23:27.527 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:26.142310+0000", 2024-09-13T11:23:27.527 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 1.2811510130000001, 2024-09-13T11:23:27.528 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.077780545000000006, 2024-09-13T11:23:27.528 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:23:27.528 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:23:27.528 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:27.528 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:23:27.528 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:27.528 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:23:27.528 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:26.142310+0000", 2024-09-13T11:23:27.528 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:27.528 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:27.528 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:27.528 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:23:27.528 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:26.142310+0000", 2024-09-13T11:23:27.528 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:27.529 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:27.529 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:27.529 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:23:27.529 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:26.142313+0000", 2024-09-13T11:23:27.529 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.6469999999999998e-06 2024-09-13T11:23:27.529 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:27.529 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:27.529 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:23:27.529 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:26.142365+0000", 2024-09-13T11:23:27.529 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.27e-05 2024-09-13T11:23:27.529 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:27.529 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:27.529 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:23:27.530 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:26.216413+0000", 2024-09-13T11:23:27.530 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.074047846 2024-09-13T11:23:27.530 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:27.530 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:27.530 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:23:27.530 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:26.216422+0000", 2024-09-13T11:23:27.530 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.4330000000000004e-06 2024-09-13T11:23:27.530 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:27.530 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:27.530 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:23:27.530 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:26.216480+0000", 2024-09-13T11:23:27.530 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.7871999999999997e-05 2024-09-13T11:23:27.530 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:27.530 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:27.531 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:23:27.531 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:26.216508+0000", 2024-09-13T11:23:27.531 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.8133999999999999e-05 2024-09-13T11:23:27.531 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:27.531 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:27.531 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:23:27.531 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:26.220041+0000", 2024-09-13T11:23:27.531 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.003533062 2024-09-13T11:23:27.531 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:27.531 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:27.531 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:23:27.531 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:26.220091+0000", 2024-09-13T11:23:27.531 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.9851000000000001e-05 2024-09-13T11:23:27.531 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:27.532 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:23:27.532 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:27.532 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:27.532 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:27.532 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPush(3.10 72/70 [PushOp(3:0f544ab4:::benchmark_data_smithi177_31116_object5644:head, version: 45'361, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:0f544ab4:::benchmark_data_smithi177_31116_object5644:head@45'361, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:23:27.532 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:26.315293+0000", 2024-09-13T11:23:27.532 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 1.108167728, 2024-09-13T11:23:27.532 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.071768672000000006, 2024-09-13T11:23:27.532 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:23:27.532 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:23:27.532 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:23:27.532 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:23:27.532 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:27.533 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:23:27.533 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:26.315293+0000", 2024-09-13T11:23:27.533 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:27.533 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:27.533 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:27.533 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:23:27.533 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:26.315293+0000", 2024-09-13T11:23:27.533 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:27.533 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:27.533 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:27.533 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:23:27.533 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:26.315297+0000", 2024-09-13T11:23:27.533 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.3220000000000001e-06 2024-09-13T11:23:27.533 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:27.534 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:27.534 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:23:27.534 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:26.315372+0000", 2024-09-13T11:23:27.534 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.5358999999999995e-05 2024-09-13T11:23:27.534 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:27.534 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:27.534 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:23:27.534 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:26.386241+0000", 2024-09-13T11:23:27.534 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.070869086999999997 2024-09-13T11:23:27.534 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:27.534 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:27.534 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:23:27.534 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:26.386255+0000", 2024-09-13T11:23:27.534 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3542e-05 2024-09-13T11:23:27.535 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:27.535 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:27.535 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:23:27.535 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:26.386334+0000", 2024-09-13T11:23:27.535 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.9050999999999999e-05 2024-09-13T11:23:27.535 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:27.535 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:27.535 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:23:27.535 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:26.386344+0000", 2024-09-13T11:23:27.535 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0086e-05 2024-09-13T11:23:27.535 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:27.535 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:27.535 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:23:27.535 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:26.387062+0000", 2024-09-13T11:23:27.536 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00071822499999999996 2024-09-13T11:23:27.536 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:27.536 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:23:27.536 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:27.536 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:27.536 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:27.536 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPush(3.10 72/70 [PushOp(3:0f8adc10:::benchmark_data_smithi177_31116_object5718:head, version: 45'362, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:0f8adc10:::benchmark_data_smithi177_31116_object5718:head@45'362, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:23:27.536 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:26.392100+0000", 2024-09-13T11:23:27.536 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 1.031361046, 2024-09-13T11:23:27.536 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.079916466000000005, 2024-09-13T11:23:27.536 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:23:27.536 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:23:27.536 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:23:27.536 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:23:27.537 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:27.537 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:23:27.537 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:26.392100+0000", 2024-09-13T11:23:27.537 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:27.537 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:27.537 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:27.537 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:23:27.537 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:26.392100+0000", 2024-09-13T11:23:27.537 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:27.537 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:27.537 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:27.537 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:23:27.537 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:26.392101+0000", 2024-09-13T11:23:27.538 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.226e-06 2024-09-13T11:23:27.538 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:27.538 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:27.538 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:23:27.538 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:26.392183+0000", 2024-09-13T11:23:27.538 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.1460999999999998e-05 2024-09-13T11:23:27.538 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:27.538 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:27.538 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:23:27.538 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:26.470887+0000", 2024-09-13T11:23:27.538 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.078704619000000003 2024-09-13T11:23:27.538 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:27.538 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:27.538 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:23:27.538 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:26.470901+0000", 2024-09-13T11:23:27.539 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3191e-05 2024-09-13T11:23:27.539 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:27.539 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:27.539 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:23:27.539 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:26.471023+0000", 2024-09-13T11:23:27.539 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00012229 2024-09-13T11:23:27.539 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:27.539 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:27.539 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:23:27.539 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:26.471045+0000", 2024-09-13T11:23:27.539 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.1670000000000001e-05 2024-09-13T11:23:27.539 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:27.539 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:27.539 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:23:27.540 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:26.472017+0000", 2024-09-13T11:23:27.540 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00097200900000000002 2024-09-13T11:23:27.540 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:27.540 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:23:27.540 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:27.540 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:27.540 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:23:27.540 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-13T11:23:27.541 INFO:teuthology.orchestra.run.smithi001.stdout:osd.0: {} 2024-09-13T11:23:27.542 INFO:tasks.ceph.osd.11.smithi177.stderr:2024-09-13T11:23:27.537+0000 7f24792c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26580) UID: 0 2024-09-13T11:23:27.542 INFO:teuthology.orchestra.run.smithi001.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T11:23:27.543 INFO:teuthology.orchestra.run.smithi001.stdout:osd.1: {} 2024-09-13T11:23:27.543 INFO:teuthology.orchestra.run.smithi001.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T11:23:27.543 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-09-13T11:23:27.544 INFO:teuthology.orchestra.run.smithi001.stdout:osd.2: {} 2024-09-13T11:23:27.544 INFO:teuthology.orchestra.run.smithi001.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T11:23:27.557 INFO:teuthology.orchestra.run.smithi001.stdout:osd.3: {} 2024-09-13T11:23:27.557 INFO:teuthology.orchestra.run.smithi001.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T11:23:27.589 INFO:teuthology.orchestra.run.smithi001.stdout:osd.4: {} 2024-09-13T11:23:27.589 INFO:teuthology.orchestra.run.smithi001.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T11:23:27.621 INFO:teuthology.orchestra.run.smithi001.stdout:osd.5: {} 2024-09-13T11:23:27.621 INFO:teuthology.orchestra.run.smithi001.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T11:23:27.643 INFO:tasks.ceph.osd.15.smithi177.stderr:2024-09-13T11:23:27.637+0000 7f9d6ce77640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26578) UID: 0 2024-09-13T11:23:27.645 INFO:teuthology.orchestra.run.smithi001.stdout:osd.6: {} 2024-09-13T11:23:27.645 INFO:teuthology.orchestra.run.smithi001.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T11:23:27.675 INFO:teuthology.orchestra.run.smithi001.stdout:osd.7: {} 2024-09-13T11:23:27.676 INFO:teuthology.orchestra.run.smithi001.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T11:23:27.686 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-13T11:23:27.687 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [], 2024-09-13T11:23:27.687 INFO:teuthology.orchestra.run.smithi162.stdout: "num_ops": 0 2024-09-13T11:23:27.687 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-13T11:23:27.702 INFO:teuthology.orchestra.run.smithi001.stdout:osd.8: {} 2024-09-13T11:23:27.703 INFO:teuthology.orchestra.run.smithi001.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-13T11:23:27.704 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-09-13T11:23:27.731 INFO:teuthology.orchestra.run.smithi001.stdout:osd.9: {} 2024-09-13T11:23:27.732 INFO:teuthology.orchestra.run.smithi001.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-13T11:23:27.743 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-13T11:23:27.736+0000 7f6a68775640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26307) UID: 0 2024-09-13T11:23:27.757 INFO:teuthology.orchestra.run.smithi001.stdout:osd.10: {} 2024-09-13T11:23:27.757 INFO:teuthology.orchestra.run.smithi001.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-13T11:23:27.782 INFO:teuthology.orchestra.run.smithi001.stdout:osd.11: {} 2024-09-13T11:23:27.783 INFO:teuthology.orchestra.run.smithi001.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-13T11:23:27.806 INFO:teuthology.orchestra.run.smithi001.stdout:osd.12: {} 2024-09-13T11:23:27.806 INFO:teuthology.orchestra.run.smithi001.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-13T11:23:27.834 INFO:teuthology.orchestra.run.smithi001.stdout:osd.13: {} 2024-09-13T11:23:27.834 INFO:teuthology.orchestra.run.smithi001.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-13T11:23:27.835 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-13T11:23:27.835 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:23:27.843 INFO:tasks.ceph.osd.4.smithi001.stderr:2024-09-13T11:23:27.836+0000 7ff8e0218640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26310) UID: 0 2024-09-13T11:23:27.853 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-09-13T11:23:27.856 INFO:teuthology.orchestra.run.smithi001.stdout:osd.14: {} 2024-09-13T11:23:27.856 INFO:teuthology.orchestra.run.smithi001.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-13T11:23:27.884 INFO:teuthology.orchestra.run.smithi001.stdout:osd.15: {} 2024-09-13T11:23:27.885 INFO:teuthology.orchestra.run.smithi001.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-13T11:23:27.944 INFO:tasks.ceph.osd.9.smithi092.stderr:2024-09-13T11:23:27.940+0000 7ff5ab3d0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26348) UID: 0 2024-09-13T11:23:27.973 INFO:tasks.ceph.osd.14.smithi162.stderr:2024-09-13T11:23:27.966+0000 7f2350b08640 -1 osd.14 72 handle_osd_map injecting map crc failure 2024-09-13T11:23:27.993 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-13T11:23:27.993 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:23:27.994 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-13T11:23:28.004 INFO:tasks.radosbench.radosbench.0.smithi177.stdout: 61 16 52207 52191 53.4685 89.6875 0.00343795 0.0186932 2024-09-13T11:23:28.014 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_ops_in_flight 2024-09-13T11:23:28.045 INFO:tasks.ceph.osd.4.smithi001.stderr:2024-09-13T11:23:28.040+0000 7ff8e0218640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26310) UID: 0 2024-09-13T11:23:28.110 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15] out_osds: [] dead_osds: [] live_osds: [0, 4, 8, 12, 5, 9, 13, 2, 6, 10, 14, 3, 7, 11, 15, 1] 2024-09-13T11:23:28.110 INFO:tasks.thrashosds.thrasher:choose_action: min_in 2 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-13T11:23:28.110 INFO:tasks.thrashosds.thrasher:Growing pool unique_pool_0 2024-09-13T11:23:28.111 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-13T11:23:28.146 INFO:tasks.ceph.osd.13.smithi092.stderr:2024-09-13T11:23:28.144+0000 7f3e5afea640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26350) UID: 0 2024-09-13T11:23:28.157 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-13T11:23:28.157 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:23:28.176 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_blocked_ops 2024-09-13T11:23:28.247 INFO:tasks.ceph.osd.3.smithi177.stderr:2024-09-13T11:23:28.241+0000 7f10adba4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26574) UID: 0 2024-09-13T11:23:28.307 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-13T11:23:28.307 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:23:28.324 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_historic_ops 2024-09-13T11:23:28.348 INFO:tasks.ceph.osd.5.smithi092.stderr:2024-09-13T11:23:28.344+0000 7f6f0b705640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26346) UID: 0 2024-09-13T11:23:28.449 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-13T11:23:28.444+0000 7f6a68775640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26307) UID: 0 2024-09-13T11:23:28.453 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-13T11:23:28.454 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:23:28.470 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-13T11:23:28.496 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-13T11:23:28.497 INFO:teuthology.orchestra.run.smithi001.stderr:dumped all 2024-09-13T11:23:28.516 INFO:teuthology.orchestra.run.smithi001.stdout:{"pg_ready":true,"pg_map":{"version":98,"stamp":"2024-09-13T11:23:28.133510+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":3276538419,"num_objects":49994,"num_object_clones":0,"num_object_copies":99988,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1456,"num_objects_misplaced":19941,"num_objects_unfound":0,"num_objects_dirty":49994,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":35878,"num_write_kb":2289266,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3432,"num_bytes_recovered":224854035,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":77559,"ondisk_log_size":77559,"up":70,"acting":70,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":86,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":16,"kb":1509949440,"kb_used":7372624,"kb_used_data":6766032,"kb_used_omap":27,"kb_used_meta":606308,"kb_avail":1502576816,"statfs":{"total":1546188226560,"available":1538638659584,"internally_reserved":0,"allocated":6928416768,"data_stored":6923329014,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28358,"internal_metadata":620859706},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,3,2,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":749797376,"num_objects":11443,"num_object_clones":0,"num_object_copies":22886,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1456,"num_objects_misplaced":18847,"num_objects_unfound":0,"num_objects_dirty":11443,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7060,"num_write_kb":451712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2336,"num_bytes_recovered":153092096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001573"},"pg_stats":[{"pgid":"3.19","version":"72'2911","reported_seq":3572,"reported_epoch":72,"state":"active+recovering+remapped","last_fresh":"2024-09-13T11:23:26.033523+0000","last_change":"2024-09-13T11:23:23.288284+0000","last_active":"2024-09-13T11:23:26.033523+0000","last_peered":"2024-09-13T11:23:26.033523+0000","last_clean":"2024-09-13T11:23:20.891203+0000","last_became_active":"2024-09-13T11:23:23.272826+0000","last_became_peered":"2024-09-13T11:23:23.272826+0000","last_unstale":"2024-09-13T11:23:26.033523+0000","last_undegraded":"2024-09-13T11:23:26.033523+0000","last_fullsized":"2024-09-13T11:23:26.033523+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:22:24.896055+0000","objects_scrubbed":0,"log_size":2911,"log_dups_size":0,"ondisk_log_size":2911,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":100335616,"num_objects":1531,"num_object_clones":0,"num_object_copies":3062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2404,"num_objects_unfound":0,"num_objects_dirty":1531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1531,"num_write_kb":97984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":187,"num_bytes_recovered":12255232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1],"acting":[2,1],"avail_no_missing":["2","12"],"object_location_counts":[{"shards":"1,2","objects":329},{"shards":"2,12","objects":1202}],"blocked_by":[],"up_primary":9,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.18","version":"71'2880","reported_seq":2986,"reported_epoch":71,"state":"active+recovery_wait+degraded+remapped","last_fresh":"2024-09-13T11:23:25.275782+0000","last_change":"2024-09-13T11:23:23.217053+0000","last_active":"2024-09-13T11:23:25.275782+0000","last_peered":"2024-09-13T11:23:25.275782+0000","last_clean":"2024-09-13T11:23:20.906701+0000","last_became_active":"2024-09-13T11:23:23.213658+0000","last_became_peered":"2024-09-13T11:23:23.213658+0000","last_unstale":"2024-09-13T11:23:25.275782+0000","last_undegraded":"2024-09-13T11:23:23.015081+0000","last_fullsized":"2024-09-13T11:23:25.275782+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:22:24.896055+0000","objects_scrubbed":0,"log_size":2880,"log_dups_size":0,"ondisk_log_size":2880,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":99155968,"num_objects":1513,"num_object_clones":0,"num_object_copies":3026,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1456,"num_objects_misplaced":1456,"num_objects_unfound":0,"num_objects_dirty":1513,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1513,"num_write_kb":96832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":131072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0],"acting":[1,0],"avail_no_missing":["1","6"],"object_location_counts":[{"shards":"0,1","objects":57},{"shards":"1,6","objects":1456}],"blocked_by":[],"up_primary":10,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.17","version":"72'2864","reported_seq":3341,"reported_epoch":72,"state":"active+recovering+remapped","last_fresh":"2024-09-13T11:23:26.016986+0000","last_change":"2024-09-13T11:23:23.032017+0000","last_active":"2024-09-13T11:23:26.016986+0000","last_peered":"2024-09-13T11:23:26.016986+0000","last_clean":"2024-09-13T11:23:20.855652+0000","last_became_active":"2024-09-13T11:23:23.020866+0000","last_became_peered":"2024-09-13T11:23:23.020866+0000","last_unstale":"2024-09-13T11:23:26.016986+0000","last_undegraded":"2024-09-13T11:23:26.016986+0000","last_fullsized":"2024-09-13T11:23:26.016986+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:22:24.896055+0000","objects_scrubbed":0,"log_size":2864,"log_dups_size":0,"ondisk_log_size":2864,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":98861056,"num_objects":1509,"num_object_clones":0,"num_object_copies":3018,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2572,"num_objects_unfound":0,"num_objects_dirty":1509,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1509,"num_write_kb":96544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":125,"num_bytes_recovered":8192000,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3],"acting":[8,3],"avail_no_missing":["8","13"],"object_location_counts":[{"shards":"3,8","objects":223},{"shards":"8,13","objects":1286}],"blocked_by":[],"up_primary":10,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.16","version":"72'3074","reported_seq":4035,"reported_epoch":72,"state":"active+recovering+remapped","last_fresh":"2024-09-13T11:23:27.035007+0000","last_change":"2024-09-13T11:23:23.056115+0000","last_active":"2024-09-13T11:23:27.035007+0000","last_peered":"2024-09-13T11:23:27.035007+0000","last_clean":"2024-09-13T11:23:20.915563+0000","last_became_active":"2024-09-13T11:23:23.027108+0000","last_became_peered":"2024-09-13T11:23:23.027108+0000","last_unstale":"2024-09-13T11:23:27.035007+0000","last_undegraded":"2024-09-13T11:23:27.035007+0000","last_fullsized":"2024-09-13T11:23:27.035007+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:22:24.896055+0000","objects_scrubbed":0,"log_size":3074,"log_dups_size":0,"ondisk_log_size":3074,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":108298240,"num_objects":1653,"num_object_clones":0,"num_object_copies":3306,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2402,"num_objects_unfound":0,"num_objects_dirty":1653,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1653,"num_write_kb":105760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":286,"num_bytes_recovered":18743296,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,15],"acting":[13,11],"avail_no_missing":["13","14"],"object_location_counts":[{"shards":"11,13","objects":452},{"shards":"13,14","objects":1201}],"blocked_by":[],"up_primary":11,"acting_primary":13,"purged_snaps":[]},{"pgid":"3.15","version":"72'2905","reported_seq":3520,"reported_epoch":72,"state":"active+recovering+remapped","last_fresh":"2024-09-13T11:23:27.035267+0000","last_change":"2024-09-13T11:23:23.229204+0000","last_active":"2024-09-13T11:23:27.035267+0000","last_peered":"2024-09-13T11:23:27.035267+0000","last_clean":"2024-09-13T11:23:20.918282+0000","last_became_active":"2024-09-13T11:23:23.204617+0000","last_became_peered":"2024-09-13T11:23:23.204617+0000","last_unstale":"2024-09-13T11:23:27.035267+0000","last_undegraded":"2024-09-13T11:23:27.035267+0000","last_fullsized":"2024-09-13T11:23:27.035267+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:22:24.896055+0000","objects_scrubbed":0,"log_size":2905,"log_dups_size":0,"ondisk_log_size":2905,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":101810176,"num_objects":1554,"num_object_clones":0,"num_object_copies":3108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2470,"num_objects_unfound":0,"num_objects_dirty":1554,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1554,"num_write_kb":99424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":171,"num_bytes_recovered":11206656,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,0],"acting":[13,0],"avail_no_missing":["13","15"],"object_location_counts":[{"shards":"0,13","objects":319},{"shards":"13,15","objects":1235}],"blocked_by":[],"up_primary":14,"acting_primary":13,"purged_snaps":[]},{"pgid":"3.14","version":"71'2864","reported_seq":3332,"reported_epoch":71,"state":"active+recovering+remapped","last_fresh":"2024-09-13T11:23:25.299772+0000","last_change":"2024-09-13T11:23:23.050524+0000","last_active":"2024-09-13T11:23:25.299772+0000","last_peered":"2024-09-13T11:23:25.299772+0000","last_clean":"2024-09-13T11:23:20.921598+0000","last_became_active":"2024-09-13T11:23:23.039246+0000","last_became_peered":"2024-09-13T11:23:23.039246+0000","last_unstale":"2024-09-13T11:23:25.299772+0000","last_undegraded":"2024-09-13T11:23:25.299772+0000","last_fullsized":"2024-09-13T11:23:25.299772+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:22:24.896055+0000","objects_scrubbed":0,"log_size":2864,"log_dups_size":0,"ondisk_log_size":2864,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":97943552,"num_objects":1495,"num_object_clones":0,"num_object_copies":2990,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2448,"num_objects_unfound":0,"num_objects_dirty":1495,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1495,"num_write_kb":95648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":129,"num_bytes_recovered":8454144,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,14],"acting":[1,3],"avail_no_missing":["1","12"],"object_location_counts":[{"shards":"1,3","objects":271},{"shards":"1,12","objects":1224}],"blocked_by":[],"up_primary":3,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":107,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-13T11:23:25.981979+0000","last_change":"2024-09-13T11:22:16.889130+0000","last_active":"2024-09-13T11:23:25.981979+0000","last_peered":"2024-09-13T11:23:25.981979+0000","last_clean":"2024-09-13T11:23:25.981979+0000","last_became_active":"2024-09-13T11:22:14.192897+0000","last_became_peered":"2024-09-13T11:22:14.192897+0000","last_unstale":"2024-09-13T11:23:25.981979+0000","last_undegraded":"2024-09-13T11:23:25.981979+0000","last_fullsized":"2024-09-13T11:23:25.981979+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:22:13.113563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:23:26.278353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00084655899999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,10],"acting":[15,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"3.6","version":"72'3066","reported_seq":3188,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-13T11:23:27.011776+0000","last_change":"2024-09-13T11:23:22.656608+0000","last_active":"2024-09-13T11:23:27.011776+0000","last_peered":"2024-09-13T11:23:27.011776+0000","last_clean":"2024-09-13T11:23:27.011776+0000","last_became_active":"2024-09-13T11:23:19.939432+0000","last_became_peered":"2024-09-13T11:23:19.939432+0000","last_unstale":"2024-09-13T11:23:27.011776+0000","last_undegraded":"2024-09-13T11:23:27.011776+0000","last_fullsized":"2024-09-13T11:23:27.011776+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'2909","last_scrub_stamp":"2024-09-13T11:23:22.656323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:22.656323+0000","objects_scrubbed":1487,"log_size":3066,"log_dups_size":0,"ondisk_log_size":3066,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:24:48.036709+0000","scrub_duration":323,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":107741184,"num_objects":1644,"num_object_clones":0,"num_object_copies":3288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1644,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":157,"num_write_kb":10048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,14],"acting":[13,14],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":114,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-13T11:23:25.984384+0000","last_change":"2024-09-13T11:23:21.875855+0000","last_active":"2024-09-13T11:23:25.984384+0000","last_peered":"2024-09-13T11:23:25.984384+0000","last_clean":"2024-09-13T11:23:25.984384+0000","last_became_active":"2024-09-13T11:22:14.211923+0000","last_became_peered":"2024-09-13T11:22:14.211923+0000","last_unstale":"2024-09-13T11:23:25.984384+0000","last_undegraded":"2024-09-13T11:23:25.984384+0000","last_fullsized":"2024-09-13T11:23:25.984384+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:23:21.875683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:23:21.875683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:24:40.678136+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0011689529999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0],"acting":[9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"72'2921","reported_seq":3044,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-13T11:23:27.033009+0000","last_change":"2024-09-13T11:23:20.443418+0000","last_active":"2024-09-13T11:23:27.033009+0000","last_peered":"2024-09-13T11:23:27.033009+0000","last_clean":"2024-09-13T11:23:27.033009+0000","last_became_active":"2024-09-13T11:23:19.927680+0000","last_became_peered":"2024-09-13T11:23:19.927680+0000","last_unstale":"2024-09-13T11:23:27.033009+0000","last_undegraded":"2024-09-13T11:23:27.033009+0000","last_fullsized":"2024-09-13T11:23:27.033009+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'2736","last_scrub_stamp":"2024-09-13T11:23:20.443329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:20.443329+0000","objects_scrubbed":1369,"log_size":2921,"log_dups_size":0,"ondisk_log_size":2921,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:24:29.180950+0000","scrub_duration":183,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":102039552,"num_objects":1557,"num_object_clones":0,"num_object_copies":3114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1557,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":188,"num_write_kb":12032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,8],"acting":[13,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":107,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-13T11:23:25.974870+0000","last_change":"2024-09-13T11:22:16.504060+0000","last_active":"2024-09-13T11:23:25.974870+0000","last_peered":"2024-09-13T11:23:25.974870+0000","last_clean":"2024-09-13T11:23:25.974870+0000","last_became_active":"2024-09-13T11:22:14.581458+0000","last_became_peered":"2024-09-13T11:22:14.581458+0000","last_unstale":"2024-09-13T11:23:25.974870+0000","last_undegraded":"2024-09-13T11:23:25.974870+0000","last_fullsized":"2024-09-13T11:23:25.974870+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:22:13.113563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:23:37.935010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001026047,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5],"acting":[8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"71'2872","reported_seq":2973,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-13T11:23:25.251960+0000","last_change":"2024-09-13T11:23:20.254213+0000","last_active":"2024-09-13T11:23:25.251960+0000","last_peered":"2024-09-13T11:23:25.251960+0000","last_clean":"2024-09-13T11:23:25.251960+0000","last_became_active":"2024-09-13T11:23:19.925399+0000","last_became_peered":"2024-09-13T11:23:19.925399+0000","last_unstale":"2024-09-13T11:23:25.251960+0000","last_undegraded":"2024-09-13T11:23:25.251960+0000","last_fullsized":"2024-09-13T11:23:25.251960+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'2755","last_scrub_stamp":"2024-09-13T11:23:20.254124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:20.254124+0000","objects_scrubbed":1437,"log_size":2872,"log_dups_size":0,"ondisk_log_size":2872,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:24:36.891863+0000","scrub_duration":187,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":102170624,"num_objects":1559,"num_object_clones":0,"num_object_copies":3118,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1559,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":122,"num_write_kb":7808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,12],"acting":[1,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":93,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-13T11:23:22.913822+0000","last_change":"2024-09-13T11:22:47.720313+0000","last_active":"2024-09-13T11:23:22.913822+0000","last_peered":"2024-09-13T11:23:22.913822+0000","last_clean":"2024-09-13T11:23:22.913822+0000","last_became_active":"2024-09-13T11:22:47.719636+0000","last_became_peered":"2024-09-13T11:22:47.719636+0000","last_unstale":"2024-09-13T11:23:22.913822+0000","last_undegraded":"2024-09-13T11:23:22.913822+0000","last_fullsized":"2024-09-13T11:23:22.913822+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:22:13.113563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00087476000000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,9],"acting":[1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"72'2881","reported_seq":3004,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-13T11:23:25.982035+0000","last_change":"2024-09-13T11:23:20.686446+0000","last_active":"2024-09-13T11:23:25.982035+0000","last_peered":"2024-09-13T11:23:25.982035+0000","last_clean":"2024-09-13T11:23:25.982035+0000","last_became_active":"2024-09-13T11:23:19.939695+0000","last_became_peered":"2024-09-13T11:23:19.939695+0000","last_unstale":"2024-09-13T11:23:25.982035+0000","last_undegraded":"2024-09-13T11:23:25.982035+0000","last_fullsized":"2024-09-13T11:23:25.982035+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'2739","last_scrub_stamp":"2024-09-13T11:23:20.686358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:20.686358+0000","objects_scrubbed":1386,"log_size":2881,"log_dups_size":0,"ondisk_log_size":2881,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:24:23.429344+0000","scrub_duration":271,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":100204544,"num_objects":1529,"num_object_clones":0,"num_object_copies":3058,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1529,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":143,"num_write_kb":9152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,13],"acting":[15,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"3.3","version":"72'2958","reported_seq":3082,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-13T11:23:25.968816+0000","last_change":"2024-09-13T11:23:20.431214+0000","last_active":"2024-09-13T11:23:25.968816+0000","last_peered":"2024-09-13T11:23:25.968816+0000","last_clean":"2024-09-13T11:23:25.968816+0000","last_became_active":"2024-09-13T11:23:19.937240+0000","last_became_peered":"2024-09-13T11:23:19.937240+0000","last_unstale":"2024-09-13T11:23:25.968816+0000","last_undegraded":"2024-09-13T11:23:25.968816+0000","last_fullsized":"2024-09-13T11:23:25.968816+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'2828","last_scrub_stamp":"2024-09-13T11:23:20.431125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:20.431125+0000","objects_scrubbed":1442,"log_size":2958,"log_dups_size":0,"ondisk_log_size":2958,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:24:47.674925+0000","scrub_duration":311,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":103809024,"num_objects":1584,"num_object_clones":0,"num_object_copies":3168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1584,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":142,"num_write_kb":9088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,7],"acting":[10,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"28'2","reported_seq":37,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-13T11:23:25.973455+0000","last_change":"2024-09-13T11:23:12.560311+0000","last_active":"2024-09-13T11:23:25.973455+0000","last_peered":"2024-09-13T11:23:25.973455+0000","last_clean":"2024-09-13T11:23:25.973455+0000","last_became_active":"2024-09-13T11:23:12.546348+0000","last_became_peered":"2024-09-13T11:23:12.546348+0000","last_unstale":"2024-09-13T11:23:25.973455+0000","last_undegraded":"2024-09-13T11:23:25.973455+0000","last_fullsized":"2024-09-13T11:23:25.973455+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:22:13.113563+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0025826730000000002,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8],"acting":[2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"72'2965","reported_seq":3087,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-13T11:23:25.969505+0000","last_change":"2024-09-13T11:23:20.977801+0000","last_active":"2024-09-13T11:23:25.969505+0000","last_peered":"2024-09-13T11:23:25.969505+0000","last_clean":"2024-09-13T11:23:25.969505+0000","last_became_active":"2024-09-13T11:23:19.934960+0000","last_became_peered":"2024-09-13T11:23:19.934960+0000","last_unstale":"2024-09-13T11:23:25.969505+0000","last_undegraded":"2024-09-13T11:23:25.969505+0000","last_fullsized":"2024-09-13T11:23:25.969505+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'2855","last_scrub_stamp":"2024-09-13T11:23:20.977695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:20.977695+0000","objects_scrubbed":1500,"log_size":2965,"log_dups_size":0,"ondisk_log_size":2965,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:24:30.985361+0000","scrub_duration":267,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":105840640,"num_objects":1615,"num_object_clones":0,"num_object_copies":3230,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1615,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":115,"num_write_kb":7360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,15],"acting":[12,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":117,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-13T11:23:25.972180+0000","last_change":"2024-09-13T11:22:47.298206+0000","last_active":"2024-09-13T11:23:25.972180+0000","last_peered":"2024-09-13T11:23:25.972180+0000","last_clean":"2024-09-13T11:23:25.972180+0000","last_became_active":"2024-09-13T11:22:47.297319+0000","last_became_peered":"2024-09-13T11:22:47.297319+0000","last_unstale":"2024-09-13T11:23:25.972180+0000","last_undegraded":"2024-09-13T11:23:25.972180+0000","last_fullsized":"2024-09-13T11:23:25.972180+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:22:13.113563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00081225399999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,1],"acting":[14,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"3.1","version":"72'2949","reported_seq":3070,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-13T11:23:27.033280+0000","last_change":"2024-09-13T11:23:21.595780+0000","last_active":"2024-09-13T11:23:27.033280+0000","last_peered":"2024-09-13T11:23:27.033280+0000","last_clean":"2024-09-13T11:23:27.033280+0000","last_became_active":"2024-09-13T11:23:19.936563+0000","last_became_peered":"2024-09-13T11:23:19.936563+0000","last_unstale":"2024-09-13T11:23:27.033280+0000","last_undegraded":"2024-09-13T11:23:27.033280+0000","last_fullsized":"2024-09-13T11:23:27.033280+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'2784","last_scrub_stamp":"2024-09-13T11:23:21.595419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:21.595419+0000","objects_scrubbed":1386,"log_size":2949,"log_dups_size":0,"ondisk_log_size":2949,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:24:31.525521+0000","scrub_duration":311,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":101646336,"num_objects":1551,"num_object_clones":0,"num_object_copies":3102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1551,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":165,"num_write_kb":10560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,11],"acting":[13,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":105,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-13T11:23:22.976174+0000","last_change":"2024-09-13T11:22:16.267999+0000","last_active":"2024-09-13T11:23:22.976174+0000","last_peered":"2024-09-13T11:23:22.976174+0000","last_clean":"2024-09-13T11:23:22.976174+0000","last_became_active":"2024-09-13T11:22:14.211435+0000","last_became_peered":"2024-09-13T11:22:14.211435+0000","last_unstale":"2024-09-13T11:23:22.976174+0000","last_undegraded":"2024-09-13T11:23:22.976174+0000","last_fullsized":"2024-09-13T11:23:22.976174+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:22:13.113563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:23:24.076640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010010360000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2","version":"72'2899","reported_seq":3024,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-13T11:23:26.045584+0000","last_change":"2024-09-13T11:23:21.197028+0000","last_active":"2024-09-13T11:23:26.045584+0000","last_peered":"2024-09-13T11:23:26.045584+0000","last_clean":"2024-09-13T11:23:26.045584+0000","last_became_active":"2024-09-13T11:23:19.937528+0000","last_became_peered":"2024-09-13T11:23:19.937528+0000","last_unstale":"2024-09-13T11:23:26.045584+0000","last_undegraded":"2024-09-13T11:23:26.045584+0000","last_fullsized":"2024-09-13T11:23:26.045584+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'2800","last_scrub_stamp":"2024-09-13T11:23:21.196792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:21.196792+0000","objects_scrubbed":1425,"log_size":2899,"log_dups_size":0,"ondisk_log_size":2899,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:24:39.687395+0000","scrub_duration":475,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":100007936,"num_objects":1526,"num_object_clones":0,"num_object_copies":3052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1526,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":101,"num_write_kb":6464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,12],"acting":[14,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"2.3","version":"26'1","reported_seq":109,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-13T11:23:25.972274+0000","last_change":"2024-09-13T11:22:16.547691+0000","last_active":"2024-09-13T11:23:25.972274+0000","last_peered":"2024-09-13T11:23:25.972274+0000","last_clean":"2024-09-13T11:23:25.972274+0000","last_became_active":"2024-09-13T11:22:14.146507+0000","last_became_peered":"2024-09-13T11:22:14.146507+0000","last_unstale":"2024-09-13T11:23:25.972274+0000","last_undegraded":"2024-09-13T11:23:25.972274+0000","last_fullsized":"2024-09-13T11:23:25.972274+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:22:13.113563+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00085610999999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,15],"acting":[14,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"1.0","version":"26'112","reported_seq":241,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-13T11:23:25.970639+0000","last_change":"2024-09-13T11:22:12.131711+0000","last_active":"2024-09-13T11:23:25.970639+0000","last_peered":"2024-09-13T11:23:25.970639+0000","last_clean":"2024-09-13T11:23:25.970639+0000","last_became_active":"2024-09-13T11:22:12.129721+0000","last_became_peered":"2024-09-13T11:22:12.129721+0000","last_unstale":"2024-09-13T11:23:25.970639+0000","last_undegraded":"2024-09-13T11:23:25.970639+0000","last_fullsized":"2024-09-13T11:23:25.970639+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:11.098388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:11.098388+0000","last_clean_scrub_stamp":"2024-09-13T11:22:11.098388+0000","objects_scrubbed":0,"log_size":112,"log_dups_size":0,"ondisk_log_size":112,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:23:40.174992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.8","version":"72'2899","reported_seq":3022,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-13T11:23:25.998167+0000","last_change":"2024-09-13T11:23:20.644952+0000","last_active":"2024-09-13T11:23:25.998167+0000","last_peered":"2024-09-13T11:23:25.998167+0000","last_clean":"2024-09-13T11:23:25.998167+0000","last_became_active":"2024-09-13T11:23:19.925802+0000","last_became_peered":"2024-09-13T11:23:19.925802+0000","last_unstale":"2024-09-13T11:23:25.998167+0000","last_undegraded":"2024-09-13T11:23:25.998167+0000","last_fullsized":"2024-09-13T11:23:25.998167+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'2771","last_scrub_stamp":"2024-09-13T11:23:20.644747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:20.644747+0000","objects_scrubbed":1356,"log_size":2899,"log_dups_size":0,"ondisk_log_size":2899,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:24:44.760037+0000","scrub_duration":411,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":98041856,"num_objects":1496,"num_object_clones":0,"num_object_copies":2992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1496,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":140,"num_write_kb":8960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.9","version":"71'2941","reported_seq":3063,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-13T11:23:25.968897+0000","last_change":"2024-09-13T11:23:22.033678+0000","last_active":"2024-09-13T11:23:25.968897+0000","last_peered":"2024-09-13T11:23:25.968897+0000","last_clean":"2024-09-13T11:23:25.968897+0000","last_became_active":"2024-09-13T11:23:19.935114+0000","last_became_peered":"2024-09-13T11:23:19.935114+0000","last_unstale":"2024-09-13T11:23:25.968897+0000","last_undegraded":"2024-09-13T11:23:25.968897+0000","last_fullsized":"2024-09-13T11:23:25.968897+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'2808","last_scrub_stamp":"2024-09-13T11:23:22.033572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:22.033572+0000","objects_scrubbed":1471,"log_size":2941,"log_dups_size":0,"ondisk_log_size":2941,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:24:23.022638+0000","scrub_duration":287,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":105119744,"num_objects":1604,"num_object_clones":0,"num_object_copies":3208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1604,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":133,"num_write_kb":8512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,2],"acting":[12,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[]},{"pgid":"3.a","version":"71'3012","reported_seq":3105,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-13T11:23:24.991065+0000","last_change":"2024-09-13T11:22:25.932217+0000","last_active":"2024-09-13T11:23:24.991065+0000","last_peered":"2024-09-13T11:23:24.991065+0000","last_clean":"2024-09-13T11:23:24.991065+0000","last_became_active":"2024-09-13T11:22:25.931493+0000","last_became_peered":"2024-09-13T11:22:25.931493+0000","last_unstale":"2024-09-13T11:23:24.991065+0000","last_undegraded":"2024-09-13T11:23:24.991065+0000","last_fullsized":"2024-09-13T11:23:24.991065+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:22:24.896055+0000","objects_scrubbed":0,"log_size":3012,"log_dups_size":0,"ondisk_log_size":3012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:23:40.032463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":197394432,"num_objects":3012,"num_object_clones":0,"num_object_copies":6024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3012,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3012,"num_write_kb":192768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.b","version":"72'3219","reported_seq":3315,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-13T11:23:27.031115+0000","last_change":"2024-09-13T11:22:49.175600+0000","last_active":"2024-09-13T11:23:27.031115+0000","last_peered":"2024-09-13T11:23:27.031115+0000","last_clean":"2024-09-13T11:23:27.031115+0000","last_became_active":"2024-09-13T11:22:49.167083+0000","last_became_peered":"2024-09-13T11:22:49.167083+0000","last_unstale":"2024-09-13T11:23:27.031115+0000","last_undegraded":"2024-09-13T11:23:27.031115+0000","last_fullsized":"2024-09-13T11:23:27.031115+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:22:24.896055+0000","objects_scrubbed":0,"log_size":3219,"log_dups_size":0,"ondisk_log_size":3219,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:23:53.109043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":210960384,"num_objects":3219,"num_object_clones":0,"num_object_copies":6438,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3219,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3219,"num_write_kb":206016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,1],"acting":[13,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"3.c","version":"72'3198","reported_seq":3293,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-13T11:23:27.019958+0000","last_change":"2024-09-13T11:22:25.934619+0000","last_active":"2024-09-13T11:23:27.019958+0000","last_peered":"2024-09-13T11:23:27.019958+0000","last_clean":"2024-09-13T11:23:27.019958+0000","last_became_active":"2024-09-13T11:22:25.933854+0000","last_became_peered":"2024-09-13T11:22:25.933854+0000","last_unstale":"2024-09-13T11:23:27.019958+0000","last_undegraded":"2024-09-13T11:23:27.019958+0000","last_fullsized":"2024-09-13T11:23:27.019958+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:22:24.896055+0000","objects_scrubbed":0,"log_size":3198,"log_dups_size":0,"ondisk_log_size":3198,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:23:33.648534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":209584128,"num_objects":3198,"num_object_clones":0,"num_object_copies":6396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3198,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3198,"num_write_kb":204672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,13],"acting":[11,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.d","version":"72'3039","reported_seq":3132,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-13T11:23:26.006642+0000","last_change":"2024-09-13T11:22:25.927325+0000","last_active":"2024-09-13T11:23:26.006642+0000","last_peered":"2024-09-13T11:23:26.006642+0000","last_clean":"2024-09-13T11:23:26.006642+0000","last_became_active":"2024-09-13T11:22:25.926433+0000","last_became_peered":"2024-09-13T11:22:25.926433+0000","last_unstale":"2024-09-13T11:23:26.006642+0000","last_undegraded":"2024-09-13T11:23:26.006642+0000","last_fullsized":"2024-09-13T11:23:26.006642+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:22:24.896055+0000","objects_scrubbed":0,"log_size":3039,"log_dups_size":0,"ondisk_log_size":3039,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:23:50.633436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":199163904,"num_objects":3039,"num_object_clones":0,"num_object_copies":6078,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3039,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3039,"num_write_kb":194496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.e","version":"72'3195","reported_seq":7824,"reported_epoch":72,"state":"active+recovering+remapped","last_fresh":"2024-09-13T11:23:26.021050+0000","last_change":"2024-09-13T11:23:06.617265+0000","last_active":"2024-09-13T11:23:26.021050+0000","last_peered":"2024-09-13T11:23:26.021050+0000","last_clean":"2024-09-13T11:23:04.462865+0000","last_became_active":"2024-09-13T11:23:06.595158+0000","last_became_peered":"2024-09-13T11:23:06.595158+0000","last_unstale":"2024-09-13T11:23:26.021050+0000","last_undegraded":"2024-09-13T11:23:26.021050+0000","last_fullsized":"2024-09-13T11:23:26.021050+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:22:24.896055+0000","objects_scrubbed":0,"log_size":3195,"log_dups_size":0,"ondisk_log_size":3195,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":209387520,"num_objects":3195,"num_object_clones":0,"num_object_copies":6390,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":260,"num_objects_unfound":0,"num_objects_dirty":3195,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3195,"num_write_kb":204480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1511,"num_bytes_recovered":99024896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[8,0],"avail_no_missing":["8","14"],"object_location_counts":[{"shards":"0,8","objects":3065},{"shards":"8,14","objects":130}],"blocked_by":[],"up_primary":0,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.f","version":"72'3167","reported_seq":3262,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-13T11:23:27.007665+0000","last_change":"2024-09-13T11:22:25.925951+0000","last_active":"2024-09-13T11:23:27.007665+0000","last_peered":"2024-09-13T11:23:27.007665+0000","last_clean":"2024-09-13T11:23:27.007665+0000","last_became_active":"2024-09-13T11:22:25.925169+0000","last_became_peered":"2024-09-13T11:22:25.925169+0000","last_unstale":"2024-09-13T11:23:27.007665+0000","last_undegraded":"2024-09-13T11:23:27.007665+0000","last_fullsized":"2024-09-13T11:23:27.007665+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:22:24.896055+0000","objects_scrubbed":0,"log_size":3167,"log_dups_size":0,"ondisk_log_size":3167,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:23:47.710319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":207552512,"num_objects":3167,"num_object_clones":0,"num_object_copies":6334,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3167,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3167,"num_write_kb":202688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,12],"acting":[11,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.10","version":"72'2946","reported_seq":3517,"reported_epoch":72,"state":"active+recovering+remapped","last_fresh":"2024-09-13T11:23:26.022722+0000","last_change":"2024-09-13T11:23:23.229267+0000","last_active":"2024-09-13T11:23:26.022722+0000","last_peered":"2024-09-13T11:23:26.022722+0000","last_clean":"2024-09-13T11:23:20.925888+0000","last_became_active":"2024-09-13T11:23:23.209229+0000","last_became_peered":"2024-09-13T11:23:23.209229+0000","last_unstale":"2024-09-13T11:23:26.022722+0000","last_undegraded":"2024-09-13T11:23:26.022722+0000","last_fullsized":"2024-09-13T11:23:26.022722+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:22:24.896055+0000","objects_scrubbed":0,"log_size":2946,"log_dups_size":0,"ondisk_log_size":2946,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":101810176,"num_objects":1554,"num_object_clones":0,"num_object_copies":3108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2485,"num_objects_unfound":0,"num_objects_dirty":1554,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1554,"num_write_kb":99424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":156,"num_bytes_recovered":10223616,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[12,0],"avail_no_missing":["12","15"],"object_location_counts":[{"shards":"0,12","objects":311},{"shards":"0,12,15","objects":1},{"shards":"12,15","objects":1242}],"blocked_by":[],"up_primary":6,"acting_primary":12,"purged_snaps":[]},{"pgid":"3.11","version":"72'2941","reported_seq":3618,"reported_epoch":72,"state":"active+recovering+remapped","last_fresh":"2024-09-13T11:23:27.033775+0000","last_change":"2024-09-13T11:23:23.159477+0000","last_active":"2024-09-13T11:23:27.033775+0000","last_peered":"2024-09-13T11:23:27.033775+0000","last_clean":"2024-09-13T11:23:20.915926+0000","last_became_active":"2024-09-13T11:23:23.145118+0000","last_became_peered":"2024-09-13T11:23:23.145118+0000","last_unstale":"2024-09-13T11:23:27.033775+0000","last_undegraded":"2024-09-13T11:23:27.033775+0000","last_fullsized":"2024-09-13T11:23:27.033775+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:22:24.896055+0000","objects_scrubbed":0,"log_size":2941,"log_dups_size":0,"ondisk_log_size":2941,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":103350272,"num_objects":1577,"num_object_clones":0,"num_object_copies":3154,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1172,"num_objects_unfound":0,"num_objects_dirty":1577,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1577,"num_write_kb":100928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":288,"num_bytes_recovered":18874368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11],"acting":[11,4],"avail_no_missing":["11","13"],"object_location_counts":[{"shards":"4,11","objects":405},{"shards":"11,13","objects":1172}],"blocked_by":[],"up_primary":4,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.12","version":"72'2926","reported_seq":3713,"reported_epoch":72,"state":"active+recovering","last_fresh":"2024-09-13T11:23:26.060412+0000","last_change":"2024-09-13T11:23:22.028697+0000","last_active":"2024-09-13T11:23:26.060412+0000","last_peered":"2024-09-13T11:23:26.060412+0000","last_clean":"2024-09-13T11:23:20.895038+0000","last_became_active":"2024-09-13T11:23:21.998877+0000","last_became_peered":"2024-09-13T11:23:21.998877+0000","last_unstale":"2024-09-13T11:23:26.060412+0000","last_undegraded":"2024-09-13T11:23:26.060412+0000","last_fullsized":"2024-09-13T11:23:26.060412+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:22:24.896055+0000","objects_scrubbed":0,"log_size":2926,"log_dups_size":0,"ondisk_log_size":2926,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":101646336,"num_objects":1551,"num_object_clones":0,"num_object_copies":3102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1085,"num_objects_unfound":0,"num_objects_dirty":1551,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1551,"num_write_kb":99264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":342,"num_bytes_recovered":22413312,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,7],"acting":[14,7],"avail_no_missing":["14","12"],"object_location_counts":[{"shards":"7,14","objects":466},{"shards":"12,14","objects":1085}],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"3.13","version":"72'2952","reported_seq":3522,"reported_epoch":72,"state":"active+recovering+remapped","last_fresh":"2024-09-13T11:23:26.027401+0000","last_change":"2024-09-13T11:23:23.021842+0000","last_active":"2024-09-13T11:23:26.027401+0000","last_peered":"2024-09-13T11:23:26.027401+0000","last_clean":"2024-09-13T11:23:20.875718+0000","last_became_active":"2024-09-13T11:23:23.005094+0000","last_became_peered":"2024-09-13T11:23:23.005094+0000","last_unstale":"2024-09-13T11:23:26.027401+0000","last_undegraded":"2024-09-13T11:23:26.027401+0000","last_fullsized":"2024-09-13T11:23:26.027401+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:22:24.896055+0000","objects_scrubbed":0,"log_size":2952,"log_dups_size":0,"ondisk_log_size":2952,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":102072320,"num_objects":1558,"num_object_clones":0,"num_object_copies":3116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1187,"num_objects_unfound":0,"num_objects_dirty":1558,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1558,"num_write_kb":99680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":234,"num_bytes_recovered":15335424,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7],"acting":[7,11],"avail_no_missing":["7","10"],"object_location_counts":[{"shards":"7,10","objects":1187},{"shards":"7,11","objects":371}],"blocked_by":[],"up_primary":11,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":26,"stat_sum":{"num_bytes":3275948032,"num_objects":49990,"num_object_clones":0,"num_object_copies":99980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1456,"num_objects_misplaced":19941,"num_objects_unfound":0,"num_objects_dirty":49990,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":35731,"num_write_kb":2286592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3431,"num_bytes_recovered":224854016,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":7041384448,"data_stored":7041384448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":77444,"ondisk_log_size":77444,"up":52,"acting":52,"num_store_stats":15},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":10},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":112,"ondisk_log_size":112,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":13,"up_from":23,"seq":98784247827,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1233636,"kb_used_data":1188564,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":93138204,"statfs":{"total":96636764160,"available":95373520896,"internally_reserved":0,"allocated":1217089536,"data_stored":1216763916,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[0,1,3,4,6,8,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":9,"up_from":23,"seq":98784247826,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":53724,"kb_used_data":22540,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94318116,"statfs":{"total":96636764160,"available":96581750784,"internally_reserved":0,"allocated":23080960,"data_stored":22763401,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,8,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":40,"seq":171798691849,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":818148,"kb_used_data":777812,"kb_used_omap":4,"kb_used_meta":40315,"kb_avail":93553692,"statfs":{"total":96636764160,"available":95798980608,"internally_reserved":0,"allocated":796479488,"data_stored":796146838,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4433,"internal_metadata":41283247},"hb_peers":[0,2,5,6,7,8,9,11,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,1,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":21,"seq":90194313234,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":439084,"kb_used_data":403740,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93932756,"statfs":{"total":96636764160,"available":96187142144,"internally_reserved":0,"allocated":413429760,"data_stored":413094229,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,7,9,10,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":21,"seq":90194313237,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27768,"kb_used_data":744,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344072,"statfs":{"total":96636764160,"available":96608329728,"internally_reserved":0,"allocated":761856,"data_stored":412120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":22,"seq":94489280529,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":254308,"kb_used_data":214804,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94117532,"statfs":{"total":96636764160,"available":96376352768,"internally_reserved":0,"allocated":219959296,"data_stored":219689424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[1,3,5,6,7,8,10,11,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":22,"seq":94489280530,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":237036,"kb_used_data":197532,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94134804,"statfs":{"total":96636764160,"available":96394039296,"internally_reserved":0,"allocated":202272768,"data_stored":201939352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,3,7,8,9,10,11,12,13],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":15,"up_from":22,"seq":94489280530,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":429928,"kb_used_data":394584,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93941912,"statfs":{"total":96636764160,"available":96196517888,"internally_reserved":0,"allocated":404054016,"data_stored":403781304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,3,4,5,6,7,10,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":22,"seq":94489280530,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":44284,"kb_used_data":13100,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94327556,"statfs":{"total":96636764160,"available":96591417344,"internally_reserved":0,"allocated":13414400,"data_stored":13066013,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[3,5,8,9,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":22,"seq":94489280530,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":240520,"kb_used_data":201016,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94131320,"statfs":{"total":96636764160,"available":96390471680,"internally_reserved":0,"allocated":205840384,"data_stored":205537971,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40434118},"hb_peers":[0,1,3,5,7,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":22,"seq":94489280530,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":645048,"kb_used_data":613864,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93726792,"statfs":{"total":96636764160,"available":95976235008,"internally_reserved":0,"allocated":628596736,"data_stored":628307627,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,3,6,8,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":22,"seq":94489280530,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":429120,"kb_used_data":393776,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93942720,"statfs":{"total":96636764160,"available":96197345280,"internally_reserved":0,"allocated":403226624,"data_stored":402870105,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,5,7,9,10,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":22,"seq":94489280529,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":247096,"kb_used_data":207592,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94124744,"statfs":{"total":96636764160,"available":96383737856,"internally_reserved":0,"allocated":212574208,"data_stored":212286921,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":40434112},"hb_peers":[0,1,3,6,7,8,9,11,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":11,"up_from":22,"seq":94489280531,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":681696,"kb_used_data":646352,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93690144,"statfs":{"total":96636764160,"available":95938707456,"internally_reserved":0,"allocated":661864448,"data_stored":661546034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":36174276},"hb_peers":[2,4,5,6,7,10,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":12,"up_from":22,"seq":94489280530,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1020192,"kb_used_data":962640,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":93351648,"statfs":{"total":96636764160,"available":95592087552,"internally_reserved":0,"allocated":985743360,"data_stored":985417974,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[0,2,3,5,6,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":14,"up_from":22,"seq":94489280530,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":571036,"kb_used_data":527372,"kb_used_omap":1,"kb_used_meta":43646,"kb_avail":93800804,"statfs":{"total":96636764160,"available":96052023296,"internally_reserved":0,"allocated":540028928,"data_stored":539705785,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":44693957},"hb_peers":[0,1,5,6,7,8,9,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":242024448,"data_stored":242024448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":800587776,"data_stored":800587776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":202637312,"data_stored":202637312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":412745728,"data_stored":412745728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":21102592,"data_stored":21102592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":211353600,"data_stored":211353600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":636485632,"data_stored":636485632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":408027136,"data_stored":408027136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":24117248,"data_stored":24117248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":219873280,"data_stored":219873280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":681246720,"data_stored":681246720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":989986816,"data_stored":989986816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":1232732160,"data_stored":1232732160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":545849344,"data_stored":545849344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":412614656,"data_stored":412614656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T11:23:28.518 INFO:tasks.ceph.ceph_manager.ceph:increase pool size by 10 2024-09-13T11:23:28.518 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool set unique_pool_0 pg_num 36 2024-09-13T11:23:28.549 INFO:tasks.ceph.osd.9.smithi092.stderr:2024-09-13T11:23:28.548+0000 7ff5ab3d0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26348) UID: 0 2024-09-13T11:23:28.608 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-09-13T11:23:28.608 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:23:28.629 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-13T11:23:28.672 INFO:tasks.ceph.osd.15.smithi177.stderr:2024-09-13T11:23:28.669+0000 7f9d6ce77640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26578) UID: 0 2024-09-13T11:23:28.751 INFO:tasks.ceph.osd.7.smithi177.stderr:2024-09-13T11:23:28.745+0000 7f897b51d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26575) UID: 0 2024-09-13T11:23:28.776 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-09-13T11:23:28.776 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:23:28.798 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-13T11:23:28.852 INFO:tasks.ceph.osd.12.smithi001.stderr:2024-09-13T11:23:28.848+0000 7f892c6e6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26314) UID: 0 2024-09-13T11:23:28.953 INFO:tasks.ceph.osd.11.smithi177.stderr:2024-09-13T11:23:28.949+0000 7f24792c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26580) UID: 0 2024-09-13T11:23:28.975 INFO:teuthology.orchestra.run.smithi001.stderr:nodeep-scrub is set 2024-09-13T11:23:28.975 INFO:teuthology.orchestra.run.smithi001.stderr:set pool 3 pg_num to 36 2024-09-13T11:23:28.978 INFO:tasks.ceph.osd.11.smithi177.stderr:2024-09-13T11:23:28.973+0000 7f246bcf0640 -1 osd.11 73 handle_osd_map injecting map crc failure 2024-09-13T11:23:28.983 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-09-13T11:23:28.983 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:23:29.002 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-13T11:23:29.004 INFO:tasks.radosbench.radosbench.0.smithi177.stdout: 62 16 52380 52364 52.7805 10.8125 0.0280064 0.0187685 2024-09-13T11:23:29.031 INFO:tasks.ceph.osd.2.smithi162.stderr:2024-09-13T11:23:29.026+0000 7fe91fd87640 -1 osd.2 73 handle_osd_map injecting map crc failure 2024-09-13T11:23:29.054 INFO:tasks.ceph.osd.13.smithi092.stderr:2024-09-13T11:23:29.052+0000 7f3e5afea640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26350) UID: 0 2024-09-13T11:23:29.155 INFO:tasks.ceph.osd.8.smithi001.stderr:2024-09-13T11:23:29.148+0000 7fb9b48ae640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26312) UID: 0 2024-09-13T11:23:29.189 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-09-13T11:23:29.189 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:23:29.216 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-13T11:23:29.260 INFO:tasks.ceph.osd.3.smithi177.stderr:2024-09-13T11:23:29.257+0000 7f10adba4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26574) UID: 0 2024-09-13T11:23:29.357 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-09-13T11:23:29.350+0000 7f97befc9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26315) UID: 0 2024-09-13T11:23:29.390 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-09-13T11:23:29.390 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:23:29.416 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-13T11:23:29.460 INFO:tasks.ceph.osd.3.smithi177.stderr:2024-09-13T11:23:29.457+0000 7f10adba4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26574) UID: 0 2024-09-13T11:23:29.559 INFO:tasks.ceph.osd.14.smithi162.stderr:2024-09-13T11:23:29.554+0000 7f235c0db640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26319) UID: 0 2024-09-13T11:23:29.607 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-09-13T11:23:29.607 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:23:29.627 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-09-13T11:23:29.660 INFO:tasks.ceph.osd.8.smithi001.stderr:2024-09-13T11:23:29.656+0000 7fb9b48ae640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26312) UID: 0 2024-09-13T11:23:29.761 INFO:tasks.ceph.osd.2.smithi162.stderr:2024-09-13T11:23:29.754+0000 7fe92b35a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26313) UID: 0 2024-09-13T11:23:29.777 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-09-13T11:23:29.777 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:23:29.799 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-09-13T11:23:29.862 INFO:tasks.ceph.osd.7.smithi177.stderr:2024-09-13T11:23:29.857+0000 7f897b51d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26575) UID: 0 2024-09-13T11:23:29.907 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T11:23:29.949 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-09-13T11:23:29.949 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:23:29.963 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-09-13T11:23:29.957+0000 7f97befc9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26315) UID: 0 2024-09-13T11:23:29.967 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-09-13T11:23:30.004 INFO:tasks.radosbench.radosbench.0.smithi177.stdout: 63 16 53649 53633 53.2015 79.3125 0.00805682 0.0187896 2024-09-13T11:23:30.064 INFO:tasks.ceph.osd.13.smithi092.stderr:2024-09-13T11:23:30.060+0000 7f3e5afea640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26350) UID: 0 2024-09-13T11:23:30.099 INFO:teuthology.orchestra.run.smithi001.stdout:osd.0: {} 2024-09-13T11:23:30.100 INFO:teuthology.orchestra.run.smithi001.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T11:23:30.126 INFO:teuthology.orchestra.run.smithi001.stdout:osd.1: {} 2024-09-13T11:23:30.126 INFO:teuthology.orchestra.run.smithi001.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T11:23:30.152 INFO:teuthology.orchestra.run.smithi001.stdout:osd.2: {} 2024-09-13T11:23:30.152 INFO:teuthology.orchestra.run.smithi001.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T11:23:30.159 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-09-13T11:23:30.159 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:23:30.165 INFO:tasks.ceph.osd.1.smithi092.stderr:2024-09-13T11:23:30.160+0000 7fe7789c5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33718) UID: 0 2024-09-13T11:23:30.179 INFO:teuthology.orchestra.run.smithi001.stdout:osd.3: {} 2024-09-13T11:23:30.179 INFO:teuthology.orchestra.run.smithi001.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T11:23:30.183 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_ops_in_flight 2024-09-13T11:23:30.207 INFO:teuthology.orchestra.run.smithi001.stdout:osd.4: {} 2024-09-13T11:23:30.207 INFO:teuthology.orchestra.run.smithi001.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T11:23:30.234 INFO:teuthology.orchestra.run.smithi001.stdout:osd.5: {} 2024-09-13T11:23:30.235 INFO:teuthology.orchestra.run.smithi001.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T11:23:30.265 INFO:tasks.ceph.osd.9.smithi092.stderr:2024-09-13T11:23:30.264+0000 7ff5ab3d0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26348) UID: 0 2024-09-13T11:23:30.332 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-09-13T11:23:30.332 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:23:30.350 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_blocked_ops 2024-09-13T11:23:30.365 INFO:teuthology.orchestra.run.smithi001.stdout:osd.6: {} 2024-09-13T11:23:30.365 INFO:teuthology.orchestra.run.smithi001.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T11:23:30.366 INFO:tasks.ceph.osd.14.smithi162.stderr:2024-09-13T11:23:30.361+0000 7f235c0db640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26319) UID: 0 2024-09-13T11:23:30.391 INFO:teuthology.orchestra.run.smithi001.stdout:osd.7: {} 2024-09-13T11:23:30.391 INFO:teuthology.orchestra.run.smithi001.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T11:23:30.422 INFO:teuthology.orchestra.run.smithi001.stdout:osd.8: {} 2024-09-13T11:23:30.422 INFO:teuthology.orchestra.run.smithi001.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-13T11:23:30.449 INFO:teuthology.orchestra.run.smithi001.stdout:osd.9: {} 2024-09-13T11:23:30.450 INFO:teuthology.orchestra.run.smithi001.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-13T11:23:30.467 INFO:tasks.ceph.osd.3.smithi177.stderr:2024-09-13T11:23:30.461+0000 7f10adba4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26574) UID: 0 2024-09-13T11:23:30.477 INFO:teuthology.orchestra.run.smithi001.stdout:osd.10: {} 2024-09-13T11:23:30.478 INFO:teuthology.orchestra.run.smithi001.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-13T11:23:30.488 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-09-13T11:23:30.488 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:23:30.508 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_historic_ops 2024-09-13T11:23:30.509 INFO:teuthology.orchestra.run.smithi001.stdout:osd.11: {} 2024-09-13T11:23:30.509 INFO:teuthology.orchestra.run.smithi001.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-13T11:23:30.534 INFO:teuthology.orchestra.run.smithi001.stdout:osd.12: {} 2024-09-13T11:23:30.534 INFO:teuthology.orchestra.run.smithi001.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-13T11:23:30.566 INFO:teuthology.orchestra.run.smithi001.stdout:osd.13: {} 2024-09-13T11:23:30.566 INFO:teuthology.orchestra.run.smithi001.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-13T11:23:30.593 INFO:teuthology.orchestra.run.smithi001.stdout:osd.14: {} 2024-09-13T11:23:30.593 INFO:teuthology.orchestra.run.smithi001.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-13T11:23:30.596 INFO:tasks.ceph.osd.7.smithi177.stderr:2024-09-13T11:23:30.593+0000 7f897b51d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26575) UID: 0 2024-09-13T11:23:30.621 INFO:teuthology.orchestra.run.smithi001.stdout:osd.15: {} 2024-09-13T11:23:30.621 INFO:teuthology.orchestra.run.smithi001.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-13T11:23:30.668 INFO:tasks.ceph.osd.2.smithi162.stderr:2024-09-13T11:23:30.661+0000 7fe92b35a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26313) UID: 0 2024-09-13T11:23:30.710 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-09-13T11:23:30.710 INFO:teuthology.orchestra.run.smithi177.stdout: "size": 20, 2024-09-13T11:23:30.710 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 600, 2024-09-13T11:23:30.710 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [ 2024-09-13T11:23:30.710 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:30.710 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:1045 3.5 3:aca982b9:::benchmark_data_smithi177_31116_object1044:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-09-13T11:23:30.711 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:22:49.650700+0000", 2024-09-13T11:23:30.711 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 41.057586972000003, 2024-09-13T11:23:30.711 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.055297286000000001, 2024-09-13T11:23:30.711 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:23:30.711 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:23:30.711 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:30.711 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:23:30.711 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:23:30.711 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:30.711 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 1045 2024-09-13T11:23:30.711 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:30.711 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:23:30.711 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:30.712 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:23:30.712 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.650700+0000", 2024-09-13T11:23:30.712 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:30.712 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:30.712 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:30.712 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:23:30.712 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.650700+0000", 2024-09-13T11:23:30.712 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:30.712 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:30.712 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:30.712 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:23:30.712 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.650702+0000", 2024-09-13T11:23:30.712 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.127e-06 2024-09-13T11:23:30.713 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:30.713 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:30.713 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:23:30.713 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.650746+0000", 2024-09-13T11:23:30.713 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.4097000000000001e-05 2024-09-13T11:23:30.713 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:30.713 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:30.713 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:23:30.713 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.650755+0000", 2024-09-13T11:23:30.713 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.0310000000000004e-06 2024-09-13T11:23:30.713 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:30.713 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:30.713 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:23:30.714 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.650760+0000", 2024-09-13T11:23:30.714 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.3929999999999999e-06 2024-09-13T11:23:30.714 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:30.714 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:30.714 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:23:30.714 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.651076+0000", 2024-09-13T11:23:30.714 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00031611900000000001 2024-09-13T11:23:30.714 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:30.714 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:30.714 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:23:30.714 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.651182+0000", 2024-09-13T11:23:30.714 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00010571199999999999 2024-09-13T11:23:30.714 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:30.715 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:30.715 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 13", 2024-09-13T11:23:30.715 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.651294+0000", 2024-09-13T11:23:30.715 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000112112 2024-09-13T11:23:30.715 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:30.715 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:30.715 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:23:30.715 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.654078+0000", 2024-09-13T11:23:30.715 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.002783832 2024-09-13T11:23:30.715 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:30.715 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:30.715 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:30.715 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.705758+0000", 2024-09-13T11:23:30.716 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.051680333000000002 2024-09-13T11:23:30.716 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:30.716 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:30.716 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:23:30.716 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.705896+0000", 2024-09-13T11:23:30.716 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00013765000000000001 2024-09-13T11:23:30.716 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:30.716 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:30.716 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:23:30.716 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.705998+0000", 2024-09-13T11:23:30.716 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00010187999999999999 2024-09-13T11:23:30.716 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:30.716 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:23:30.717 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:30.717 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:30.717 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:30.717 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop_reply(client.4665.0:1045 3.5 e43/30 ondisk, result = 0)", 2024-09-13T11:23:30.717 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:22:49.652166+0000", 2024-09-13T11:23:30.717 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 41.056121064000003, 2024-09-13T11:23:30.717 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.053848380000000001, 2024-09-13T11:23:30.717 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:23:30.717 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:23:30.717 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:23:30.717 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:23:30.717 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:30.717 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:23:30.718 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.652166+0000", 2024-09-13T11:23:30.718 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:30.718 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:30.718 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:30.718 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:23:30.718 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.652166+0000", 2024-09-13T11:23:30.718 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:30.718 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:30.718 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:30.718 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:23:30.718 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.652168+0000", 2024-09-13T11:23:30.718 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.393e-06 2024-09-13T11:23:30.718 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:30.719 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:30.719 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:23:30.719 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.652172+0000", 2024-09-13T11:23:30.719 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.2989999999999998e-06 2024-09-13T11:23:30.719 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:30.719 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:30.719 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:23:30.719 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.705604+0000", 2024-09-13T11:23:30.719 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.053431575000000002 2024-09-13T11:23:30.719 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:30.719 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:30.719 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:23:30.720 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.705623+0000", 2024-09-13T11:23:30.720 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.9479000000000001e-05 2024-09-13T11:23:30.720 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:30.720 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:30.720 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:23:30.720 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.705728+0000", 2024-09-13T11:23:30.720 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000105338 2024-09-13T11:23:30.720 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:30.720 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:30.720 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:23:30.720 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.705746+0000", 2024-09-13T11:23:30.720 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.7645000000000001e-05 2024-09-13T11:23:30.720 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:30.721 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:30.721 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:23:30.721 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:49.706015+0000", 2024-09-13T11:23:30.721 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00026865100000000001 2024-09-13T11:23:30.721 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:30.721 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:23:30.721 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:30.721 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:30.721 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:30.722 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:9510 3.5 3:a7e302ef:::benchmark_data_smithi177_31116_object9509:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-09-13T11:23:30.722 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:22:54.044535+0000", 2024-09-13T11:23:30.722 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 36.663752696000003, 2024-09-13T11:23:30.722 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.049023708999999999, 2024-09-13T11:23:30.722 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:23:30.722 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:23:30.722 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:30.722 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:23:30.722 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:23:30.722 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:30.722 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 9510 2024-09-13T11:23:30.723 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:30.723 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:23:30.723 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:30.723 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:23:30.723 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:54.044535+0000", 2024-09-13T11:23:30.723 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:30.723 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:30.723 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:30.723 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:23:30.723 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:54.044535+0000", 2024-09-13T11:23:30.723 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:30.723 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:30.723 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:30.724 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:23:30.724 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:54.044539+0000", 2024-09-13T11:23:30.724 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.0219999999999998e-06 2024-09-13T11:23:30.724 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:30.724 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:30.724 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:23:30.724 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:54.044570+0000", 2024-09-13T11:23:30.724 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.0914999999999998e-05 2024-09-13T11:23:30.724 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:30.724 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:30.724 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:23:30.724 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:54.044593+0000", 2024-09-13T11:23:30.724 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.3584999999999999e-05 2024-09-13T11:23:30.725 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:30.725 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:30.725 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:23:30.725 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:54.044605+0000", 2024-09-13T11:23:30.725 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.1501000000000001e-05 2024-09-13T11:23:30.725 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:30.725 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:30.725 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:23:30.725 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:54.044681+0000", 2024-09-13T11:23:30.725 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.5894000000000002e-05 2024-09-13T11:23:30.725 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:30.725 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:30.725 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:23:30.726 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:54.044820+0000", 2024-09-13T11:23:30.726 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00013934700000000001 2024-09-13T11:23:30.726 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:30.726 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:30.726 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 13", 2024-09-13T11:23:30.726 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:54.044938+0000", 2024-09-13T11:23:30.726 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000118377 2024-09-13T11:23:30.726 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:30.726 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:30.726 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:23:30.726 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:54.046044+0000", 2024-09-13T11:23:30.726 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001105195 2024-09-13T11:23:30.727 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:30.727 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:30.727 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:30.727 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:54.093490+0000", 2024-09-13T11:23:30.727 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.047446691999999999 2024-09-13T11:23:30.727 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:30.727 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:30.727 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:23:30.727 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:54.093525+0000", 2024-09-13T11:23:30.727 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.4422999999999999e-05 2024-09-13T11:23:30.727 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:30.727 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:30.727 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:23:30.727 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:54.093558+0000", 2024-09-13T11:23:30.727 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.3757999999999998e-05 2024-09-13T11:23:30.728 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:30.728 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:23:30.728 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:30.728 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:30.728 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:30.728 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop_reply(client.4665.0:9510 3.5 e47/30 ondisk, result = 0)", 2024-09-13T11:23:30.728 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:22:54.047246+0000", 2024-09-13T11:23:30.728 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 36.661041715000003, 2024-09-13T11:23:30.728 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.046320735000000002, 2024-09-13T11:23:30.728 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:23:30.728 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:23:30.728 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:23:30.728 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:23:30.728 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:30.729 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:23:30.729 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:54.047246+0000", 2024-09-13T11:23:30.729 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:30.729 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:30.729 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:30.729 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:23:30.729 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:54.047246+0000", 2024-09-13T11:23:30.729 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:30.729 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:30.729 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:30.729 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:23:30.729 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:54.047247+0000", 2024-09-13T11:23:30.729 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5829999999999999e-06 2024-09-13T11:23:30.730 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:30.730 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:30.730 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:23:30.730 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:54.047252+0000", 2024-09-13T11:23:30.730 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.7319999999999998e-06 2024-09-13T11:23:30.730 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:30.730 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:30.730 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:23:30.730 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:54.093433+0000", 2024-09-13T11:23:30.730 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.046181415000000003 2024-09-13T11:23:30.730 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:30.730 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:30.730 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:23:30.730 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:54.093440+0000", 2024-09-13T11:23:30.731 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.0029999999999997e-06 2024-09-13T11:23:30.731 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:30.731 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:30.731 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:23:30.731 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:54.093481+0000", 2024-09-13T11:23:30.731 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.0531999999999998e-05 2024-09-13T11:23:30.731 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:30.731 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:30.731 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:23:30.731 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:54.093486+0000", 2024-09-13T11:23:30.731 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.2700000000000004e-06 2024-09-13T11:23:30.731 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:30.731 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:30.732 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:23:30.732 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:54.093566+0000", 2024-09-13T11:23:30.732 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.0199999999999998e-05 2024-09-13T11:23:30.732 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:30.732 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:23:30.732 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:30.732 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:30.732 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:30.732 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:25138 3.5 3:ab2ac965:::benchmark_data_smithi177_31116_object25137:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e55)", 2024-09-13T11:23:30.732 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:04.117349+0000", 2024-09-13T11:23:30.732 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 26.590938578999999, 2024-09-13T11:23:30.732 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.050459565999999997, 2024-09-13T11:23:30.732 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:23:30.733 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:23:30.733 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:30.733 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:23:30.733 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:23:30.733 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:30.733 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 25138 2024-09-13T11:23:30.733 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:30.733 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:23:30.733 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:30.733 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:23:30.733 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.117349+0000", 2024-09-13T11:23:30.733 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:30.733 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:30.734 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:30.734 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:23:30.734 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.117349+0000", 2024-09-13T11:23:30.734 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:30.734 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:30.734 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:30.734 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:23:30.734 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.117350+0000", 2024-09-13T11:23:30.734 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5230000000000001e-06 2024-09-13T11:23:30.735 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:30.735 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:30.735 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:23:30.735 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.117384+0000", 2024-09-13T11:23:30.735 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.3711000000000003e-05 2024-09-13T11:23:30.735 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:30.735 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:30.735 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:23:30.735 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.117397+0000", 2024-09-13T11:23:30.735 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3274e-05 2024-09-13T11:23:30.736 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:30.736 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:30.736 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:23:30.736 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.117404+0000", 2024-09-13T11:23:30.736 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.9190000000000002e-06 2024-09-13T11:23:30.736 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:30.736 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:30.736 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:23:30.736 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.117443+0000", 2024-09-13T11:23:30.736 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.8733000000000003e-05 2024-09-13T11:23:30.736 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:30.736 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:30.736 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:23:30.737 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.117534+0000", 2024-09-13T11:23:30.737 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.0526999999999996e-05 2024-09-13T11:23:30.737 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:30.737 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:30.737 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 13", 2024-09-13T11:23:30.737 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.117618+0000", 2024-09-13T11:23:30.737 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.4214000000000004e-05 2024-09-13T11:23:30.737 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:30.738 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:30.738 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:23:30.739 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.118475+0000", 2024-09-13T11:23:30.739 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00085723699999999995 2024-09-13T11:23:30.739 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:30.739 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:30.739 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:30.739 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.167731+0000", 2024-09-13T11:23:30.739 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.049256105000000001 2024-09-13T11:23:30.740 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:30.740 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:30.740 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:23:30.740 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.167775+0000", 2024-09-13T11:23:30.740 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.3906000000000003e-05 2024-09-13T11:23:30.740 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:30.740 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:30.740 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:23:30.740 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.167808+0000", 2024-09-13T11:23:30.740 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.3417000000000002e-05 2024-09-13T11:23:30.741 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:30.741 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:23:30.741 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:30.741 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:30.741 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:30.741 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:25383 3.0 e55/30 3:0a6bb869:::benchmark_data_smithi177_31116_object25382:head v 55'1620, mlcod=55'1617)", 2024-09-13T11:23:30.741 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:04.449956+0000", 2024-09-13T11:23:30.741 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 26.258331323, 2024-09-13T11:23:30.741 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.21143479600000001, 2024-09-13T11:23:30.741 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:23:30.741 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:23:30.742 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:30.742 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:23:30.742 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:30.742 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:23:30.742 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.449956+0000", 2024-09-13T11:23:30.742 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:30.742 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:30.742 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:30.742 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:23:30.742 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.449956+0000", 2024-09-13T11:23:30.742 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:30.742 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:30.743 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:30.743 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:23:30.743 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.449959+0000", 2024-09-13T11:23:30.743 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.393e-06 2024-09-13T11:23:30.743 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:30.743 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:30.743 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:23:30.743 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.450014+0000", 2024-09-13T11:23:30.743 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.4200000000000003e-05 2024-09-13T11:23:30.743 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:30.743 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:30.743 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:23:30.744 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.658637+0000", 2024-09-13T11:23:30.744 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.20862334799999999 2024-09-13T11:23:30.744 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:30.744 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:30.744 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:23:30.744 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.658650+0000", 2024-09-13T11:23:30.744 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3148e-05 2024-09-13T11:23:30.744 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:30.744 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:30.744 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:23:30.744 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.658867+0000", 2024-09-13T11:23:30.744 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00021673499999999999 2024-09-13T11:23:30.745 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:30.745 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:30.745 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:23:30.745 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.658931+0000", 2024-09-13T11:23:30.745 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.4045000000000001e-05 2024-09-13T11:23:30.745 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:30.745 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:30.745 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:23:30.745 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.661317+0000", 2024-09-13T11:23:30.745 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.002385823 2024-09-13T11:23:30.745 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:30.746 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:30.746 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:23:30.746 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.661391+0000", 2024-09-13T11:23:30.746 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.4103999999999994e-05 2024-09-13T11:23:30.746 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:30.746 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:23:30.746 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:30.746 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:30.746 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:30.746 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:25420 3.0 e56/30 3:03f6c3d6:::benchmark_data_smithi177_31116_object25419:head v 56'1621, mlcod=55'1618)", 2024-09-13T11:23:30.746 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:04.658522+0000", 2024-09-13T11:23:30.746 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 26.049765279999999, 2024-09-13T11:23:30.746 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.073500560000000006, 2024-09-13T11:23:30.747 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:23:30.747 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:23:30.747 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:30.747 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:23:30.747 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:30.747 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:23:30.747 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.658522+0000", 2024-09-13T11:23:30.747 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:30.747 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:30.747 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:30.747 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:23:30.747 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.658522+0000", 2024-09-13T11:23:30.747 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:30.748 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:30.748 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:30.748 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:23:30.748 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.658523+0000", 2024-09-13T11:23:30.748 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.9999999999999997e-07 2024-09-13T11:23:30.748 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:30.748 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:30.748 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:23:30.748 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.658603+0000", 2024-09-13T11:23:30.748 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.0413000000000004e-05 2024-09-13T11:23:30.748 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:30.748 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:30.748 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:23:30.748 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.729224+0000", 2024-09-13T11:23:30.749 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.070621192999999999 2024-09-13T11:23:30.749 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:30.749 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:30.749 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:23:30.749 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.729239+0000", 2024-09-13T11:23:30.749 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5143999999999999e-05 2024-09-13T11:23:30.749 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:30.749 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:30.749 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:23:30.749 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.729386+0000", 2024-09-13T11:23:30.749 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00014676399999999999 2024-09-13T11:23:30.749 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:30.749 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:30.750 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:23:30.750 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.729427+0000", 2024-09-13T11:23:30.750 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.0451000000000003e-05 2024-09-13T11:23:30.750 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:30.750 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:30.750 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:23:30.750 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.731905+0000", 2024-09-13T11:23:30.750 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0024786949999999999 2024-09-13T11:23:30.750 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:30.750 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:30.750 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:23:30.750 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.732023+0000", 2024-09-13T11:23:30.750 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00011730000000000001 2024-09-13T11:23:30.751 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:30.751 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:23:30.751 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:30.751 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:30.751 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:30.751 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:25436 3.0 e56/30 3:07027188:::benchmark_data_smithi177_31116_object25435:head v 56'1622, mlcod=55'1619)", 2024-09-13T11:23:30.751 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:04.729114+0000", 2024-09-13T11:23:30.751 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 25.979173119999999, 2024-09-13T11:23:30.751 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.046114465, 2024-09-13T11:23:30.751 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:23:30.751 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:23:30.751 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:30.751 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:23:30.751 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:30.752 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:23:30.752 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.729114+0000", 2024-09-13T11:23:30.752 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:30.752 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:30.752 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:30.752 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:23:30.752 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.729114+0000", 2024-09-13T11:23:30.752 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:30.752 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:30.752 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:30.752 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:23:30.752 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.729118+0000", 2024-09-13T11:23:30.752 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.9230000000000004e-06 2024-09-13T11:23:30.753 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:30.753 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:30.753 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:23:30.753 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.729181+0000", 2024-09-13T11:23:30.753 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.2504999999999999e-05 2024-09-13T11:23:30.753 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:30.753 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:30.753 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:23:30.753 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.772755+0000", 2024-09-13T11:23:30.753 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.043574553000000002 2024-09-13T11:23:30.753 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:30.753 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:30.753 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:23:30.753 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.772771+0000", 2024-09-13T11:23:30.754 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5277999999999998e-05 2024-09-13T11:23:30.754 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:30.754 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:30.754 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:23:30.754 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.772914+0000", 2024-09-13T11:23:30.754 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00014371699999999999 2024-09-13T11:23:30.754 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:30.754 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:30.754 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:23:30.754 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.772953+0000", 2024-09-13T11:23:30.754 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.8858e-05 2024-09-13T11:23:30.754 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:30.754 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:30.755 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:23:30.755 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.775151+0000", 2024-09-13T11:23:30.755 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0021981050000000001 2024-09-13T11:23:30.755 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:30.755 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:30.755 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:23:30.755 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.775229+0000", 2024-09-13T11:23:30.755 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.7526000000000003e-05 2024-09-13T11:23:30.755 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:30.755 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:23:30.755 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:30.755 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:30.755 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:30.756 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:32785 3.5 3:ad14c732:::benchmark_data_smithi177_31116_object32784:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e60)", 2024-09-13T11:23:30.756 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:11.046235+0000", 2024-09-13T11:23:30.756 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 19.662051976000001, 2024-09-13T11:23:30.756 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.054203993999999998, 2024-09-13T11:23:30.756 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:23:30.756 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:23:30.756 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:30.756 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:23:30.756 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:23:30.756 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:30.756 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 32785 2024-09-13T11:23:30.756 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:30.756 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:23:30.756 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:30.757 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:23:30.757 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:11.046235+0000", 2024-09-13T11:23:30.757 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:30.757 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:30.757 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:30.757 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:23:30.757 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:11.046235+0000", 2024-09-13T11:23:30.757 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:30.757 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:30.757 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:30.757 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:23:30.757 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:11.046239+0000", 2024-09-13T11:23:30.757 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.095e-06 2024-09-13T11:23:30.758 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:30.758 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:30.758 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:23:30.758 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:11.046270+0000", 2024-09-13T11:23:30.758 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.1761000000000003e-05 2024-09-13T11:23:30.758 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:30.758 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:30.758 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:23:30.758 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:11.046292+0000", 2024-09-13T11:23:30.758 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.1736999999999999e-05 2024-09-13T11:23:30.758 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:30.758 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:30.758 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:23:30.759 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:11.046302+0000", 2024-09-13T11:23:30.759 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0413e-05 2024-09-13T11:23:30.759 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:30.759 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:30.759 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:23:30.759 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:11.046360+0000", 2024-09-13T11:23:30.759 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.7816999999999999e-05 2024-09-13T11:23:30.759 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:30.759 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:30.759 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:23:30.759 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:11.046481+0000", 2024-09-13T11:23:30.759 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000120846 2024-09-13T11:23:30.759 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:30.760 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:30.760 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 13", 2024-09-13T11:23:30.760 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:11.046576+0000", 2024-09-13T11:23:30.760 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.5189e-05 2024-09-13T11:23:30.760 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:30.760 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:30.760 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:23:30.760 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:11.047858+0000", 2024-09-13T11:23:30.760 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001281379 2024-09-13T11:23:30.760 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:30.760 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:30.760 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:30.761 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:11.100355+0000", 2024-09-13T11:23:30.761 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.052497061999999997 2024-09-13T11:23:30.761 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:30.761 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:30.761 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:23:30.761 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:11.100393+0000", 2024-09-13T11:23:30.761 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.8132e-05 2024-09-13T11:23:30.761 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:30.761 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:30.761 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:23:30.761 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:11.100439+0000", 2024-09-13T11:23:30.761 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.6563000000000001e-05 2024-09-13T11:23:30.761 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:30.761 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:23:30.762 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:30.762 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:30.762 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:30.762 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:32800 3.5 3:a2df9f69:::benchmark_data_smithi177_31116_object32799:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e60)", 2024-09-13T11:23:30.762 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:11.055291+0000", 2024-09-13T11:23:30.762 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 19.652996427000001, 2024-09-13T11:23:30.762 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.085442275999999998, 2024-09-13T11:23:30.762 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:23:30.762 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:23:30.762 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:30.762 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:23:30.762 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:23:30.762 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:30.763 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 32800 2024-09-13T11:23:30.763 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:30.763 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:23:30.763 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:30.763 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:23:30.763 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:11.055291+0000", 2024-09-13T11:23:30.763 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:30.763 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:30.763 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:30.763 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:23:30.763 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:11.055291+0000", 2024-09-13T11:23:30.763 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:30.764 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:30.764 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:30.764 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:23:30.764 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:11.055294+0000", 2024-09-13T11:23:30.764 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.411e-06 2024-09-13T11:23:30.764 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:30.764 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:30.764 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:23:30.764 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:11.055326+0000", 2024-09-13T11:23:30.764 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.1257999999999998e-05 2024-09-13T11:23:30.764 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:30.764 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:30.764 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:23:30.764 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:11.055346+0000", 2024-09-13T11:23:30.765 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.0483e-05 2024-09-13T11:23:30.765 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:30.765 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:30.765 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:23:30.765 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:11.055356+0000", 2024-09-13T11:23:30.765 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0217999999999999e-05 2024-09-13T11:23:30.765 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:30.765 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:30.765 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:23:30.765 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:11.055415+0000", 2024-09-13T11:23:30.765 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.8350999999999997e-05 2024-09-13T11:23:30.765 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:30.765 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:30.766 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:23:30.766 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:11.055514+0000", 2024-09-13T11:23:30.766 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.9380000000000001e-05 2024-09-13T11:23:30.766 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:30.766 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:30.766 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 13", 2024-09-13T11:23:30.766 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:11.055610+0000", 2024-09-13T11:23:30.766 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.5694999999999998e-05 2024-09-13T11:23:30.766 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:30.766 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:30.766 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:23:30.766 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:11.057365+0000", 2024-09-13T11:23:30.767 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0017549309999999999 2024-09-13T11:23:30.767 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:30.767 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:30.767 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:30.767 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:11.140661+0000", 2024-09-13T11:23:30.767 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.083296408000000002 2024-09-13T11:23:30.767 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:30.767 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:30.767 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:23:30.767 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:11.140703+0000", 2024-09-13T11:23:30.767 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.1823000000000003e-05 2024-09-13T11:23:30.767 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:30.767 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:30.767 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:23:30.767 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:11.140733+0000", 2024-09-13T11:23:30.768 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.0318e-05 2024-09-13T11:23:30.768 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:30.768 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:23:30.768 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:30.768 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:30.768 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:30.768 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:40949 3.0 e64/30 3:051bf38c:::benchmark_data_smithi177_31116_object40948:head v 64'2598, mlcod=64'2595)", 2024-09-13T11:23:30.768 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:16.543416+0000", 2024-09-13T11:23:30.768 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 14.164871849000001, 2024-09-13T11:23:30.768 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.058856275999999999, 2024-09-13T11:23:30.768 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:23:30.768 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:23:30.768 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:30.769 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:23:30.769 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:30.769 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:23:30.769 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:16.543416+0000", 2024-09-13T11:23:30.769 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:30.769 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:30.769 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:30.769 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:23:30.769 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:16.543416+0000", 2024-09-13T11:23:30.769 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:30.769 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:30.769 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:30.769 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:23:30.769 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:16.543418+0000", 2024-09-13T11:23:30.770 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.9129999999999999e-06 2024-09-13T11:23:30.770 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:30.770 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:30.770 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:23:30.770 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:16.543467+0000", 2024-09-13T11:23:30.770 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.8825000000000001e-05 2024-09-13T11:23:30.770 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:30.770 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:30.770 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:23:30.770 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:16.600673+0000", 2024-09-13T11:23:30.770 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.057205473 2024-09-13T11:23:30.770 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:30.770 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:30.770 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:23:30.771 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:16.600680+0000", 2024-09-13T11:23:30.771 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.2300000000000002e-06 2024-09-13T11:23:30.771 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:30.771 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:30.771 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:23:30.771 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:16.600749+0000", 2024-09-13T11:23:30.771 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.9255e-05 2024-09-13T11:23:30.771 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:30.771 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:30.771 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:23:30.772 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:16.600768+0000", 2024-09-13T11:23:30.772 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.8348000000000001e-05 2024-09-13T11:23:30.772 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:30.772 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:30.772 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:23:30.772 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:16.602234+0000", 2024-09-13T11:23:30.772 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0014659499999999999 2024-09-13T11:23:30.772 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:30.772 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:30.772 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:23:30.772 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:16.602272+0000", 2024-09-13T11:23:30.772 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.8281999999999997e-05 2024-09-13T11:23:30.772 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:30.773 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:23:30.773 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:30.773 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:30.773 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:30.773 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:41125 3.0 e65/30 3:03dd7acf:::benchmark_data_smithi177_31116_object41124:head v 65'2605, mlcod=65'2602)", 2024-09-13T11:23:30.773 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:16.687513+0000", 2024-09-13T11:23:30.773 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 14.02077467, 2024-09-13T11:23:30.773 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.058003289, 2024-09-13T11:23:30.773 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:23:30.773 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:23:30.773 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:30.773 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:23:30.773 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:30.774 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:23:30.774 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:16.687513+0000", 2024-09-13T11:23:30.774 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:30.774 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:30.774 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:30.774 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:23:30.774 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:16.687513+0000", 2024-09-13T11:23:30.774 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:30.774 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:30.774 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:30.774 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:23:30.774 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:16.687515+0000", 2024-09-13T11:23:30.774 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.1749999999999999e-06 2024-09-13T11:23:30.774 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:30.775 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:30.775 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:23:30.775 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:16.687560+0000", 2024-09-13T11:23:30.775 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.5080000000000002e-05 2024-09-13T11:23:30.775 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:30.775 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:30.775 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:23:30.775 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:16.743482+0000", 2024-09-13T11:23:30.775 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.055922472000000001 2024-09-13T11:23:30.775 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:30.775 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:30.775 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:23:30.775 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:16.743493+0000", 2024-09-13T11:23:30.776 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0633e-05 2024-09-13T11:23:30.776 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:30.776 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:30.776 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:23:30.776 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:16.743568+0000", 2024-09-13T11:23:30.776 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.5176e-05 2024-09-13T11:23:30.776 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:30.776 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:30.776 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:23:30.776 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:16.743593+0000", 2024-09-13T11:23:30.776 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.4924000000000001e-05 2024-09-13T11:23:30.776 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:30.776 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:30.777 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:23:30.777 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:16.745468+0000", 2024-09-13T11:23:30.777 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0018745159999999999 2024-09-13T11:23:30.777 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:30.777 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:30.777 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:23:30.777 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:16.745516+0000", 2024-09-13T11:23:30.777 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.8313000000000003e-05 2024-09-13T11:23:30.777 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:30.777 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:23:30.777 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:30.777 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:30.777 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:30.777 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:41164 3.5 3:a54052b0:::benchmark_data_smithi177_31116_object41163:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e65)", 2024-09-13T11:23:30.778 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:16.718639+0000", 2024-09-13T11:23:30.778 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 13.989647980999999, 2024-09-13T11:23:30.778 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.053671647000000003, 2024-09-13T11:23:30.778 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:23:30.778 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:23:30.778 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:30.778 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:23:30.778 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:23:30.778 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:30.778 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 41164 2024-09-13T11:23:30.778 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:30.778 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:23:30.778 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:30.779 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:23:30.779 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:16.718639+0000", 2024-09-13T11:23:30.779 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:30.779 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:30.779 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:30.779 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:23:30.779 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:16.718639+0000", 2024-09-13T11:23:30.779 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:30.779 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:30.779 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:30.779 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:23:30.779 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:16.718641+0000", 2024-09-13T11:23:30.779 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.466e-06 2024-09-13T11:23:30.780 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:30.780 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:30.780 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:23:30.780 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:16.718674+0000", 2024-09-13T11:23:30.780 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.3456e-05 2024-09-13T11:23:30.780 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:30.780 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:30.780 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:23:30.780 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:16.718687+0000", 2024-09-13T11:23:30.780 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.2877999999999999e-05 2024-09-13T11:23:30.780 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:30.780 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:30.780 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:23:30.781 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:16.718693+0000", 2024-09-13T11:23:30.781 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.1890000000000004e-06 2024-09-13T11:23:30.781 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:30.781 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:30.781 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:23:30.781 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:16.718758+0000", 2024-09-13T11:23:30.781 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.4474999999999995e-05 2024-09-13T11:23:30.781 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:30.781 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:30.781 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:23:30.781 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:16.718885+0000", 2024-09-13T11:23:30.781 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00012749300000000001 2024-09-13T11:23:30.781 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:30.782 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:30.782 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 13", 2024-09-13T11:23:30.782 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:16.719015+0000", 2024-09-13T11:23:30.782 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00012969699999999999 2024-09-13T11:23:30.782 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:30.782 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:30.782 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:23:30.782 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:16.720517+0000", 2024-09-13T11:23:30.782 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0015014799999999999 2024-09-13T11:23:30.782 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:30.782 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:30.782 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:30.782 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:16.772214+0000", 2024-09-13T11:23:30.782 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.051697476999999999 2024-09-13T11:23:30.783 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:30.783 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:30.783 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:23:30.783 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:16.772266+0000", 2024-09-13T11:23:30.783 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.2012999999999998e-05 2024-09-13T11:23:30.783 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:30.783 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:30.783 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:23:30.783 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:16.772311+0000", 2024-09-13T11:23:30.783 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.5022999999999999e-05 2024-09-13T11:23:30.784 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:30.784 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:23:30.784 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:30.784 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:30.784 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:30.784 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop_reply(client.4665.0:41164 3.5 e65/30 ondisk, result = 0)", 2024-09-13T11:23:30.784 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:16.721259+0000", 2024-09-13T11:23:30.784 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 13.98702817, 2024-09-13T11:23:30.784 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.051059460000000001, 2024-09-13T11:23:30.785 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:23:30.785 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:23:30.785 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:23:30.785 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:23:30.785 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:30.785 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:23:30.785 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:16.721259+0000", 2024-09-13T11:23:30.785 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:30.785 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:30.785 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:30.785 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:23:30.785 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:16.721259+0000", 2024-09-13T11:23:30.785 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:30.785 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:30.786 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:30.786 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:23:30.786 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:16.721261+0000", 2024-09-13T11:23:30.786 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.4839999999999999e-06 2024-09-13T11:23:30.786 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:30.786 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:30.786 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:23:30.786 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:16.721266+0000", 2024-09-13T11:23:30.786 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.2859999999999999e-06 2024-09-13T11:23:30.786 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:30.786 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:30.786 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:23:30.786 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:16.772136+0000", 2024-09-13T11:23:30.786 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.050870163000000003 2024-09-13T11:23:30.787 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:30.787 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:30.787 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:23:30.787 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:16.772145+0000", 2024-09-13T11:23:30.787 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.3349999999999996e-06 2024-09-13T11:23:30.787 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:30.787 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:30.787 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:23:30.787 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:16.772204+0000", 2024-09-13T11:23:30.787 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.8455999999999997e-05 2024-09-13T11:23:30.787 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:30.787 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:30.787 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:23:30.788 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:16.772210+0000", 2024-09-13T11:23:30.788 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.0129999999999999e-06 2024-09-13T11:23:30.788 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:30.788 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:30.788 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:23:30.788 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:16.772319+0000", 2024-09-13T11:23:30.788 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000108723 2024-09-13T11:23:30.788 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:30.788 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:23:30.788 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:30.788 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:30.788 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:30.789 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:41229 3.0 e65/30 3:005e9408:::benchmark_data_smithi177_31116_object41228:head v 65'2610, mlcod=65'2607)", 2024-09-13T11:23:30.789 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:16.772798+0000", 2024-09-13T11:23:30.789 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 13.93548966, 2024-09-13T11:23:30.789 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.076228748999999998, 2024-09-13T11:23:30.789 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:23:30.789 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:23:30.789 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:30.789 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:23:30.789 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:30.789 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:23:30.789 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:16.772798+0000", 2024-09-13T11:23:30.789 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:30.789 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:30.789 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:30.790 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:23:30.790 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:16.772798+0000", 2024-09-13T11:23:30.790 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:30.790 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:30.790 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:30.790 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:23:30.790 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:16.772799+0000", 2024-09-13T11:23:30.790 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.73e-06 2024-09-13T11:23:30.790 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:30.790 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:30.790 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:23:30.790 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:16.772874+0000", 2024-09-13T11:23:30.790 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.4714000000000003e-05 2024-09-13T11:23:30.790 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:30.791 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:30.791 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:23:30.791 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:16.847048+0000", 2024-09-13T11:23:30.791 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.074173825999999998 2024-09-13T11:23:30.791 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:30.791 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:30.791 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:23:30.791 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:16.847058+0000", 2024-09-13T11:23:30.791 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.6679999999999996e-06 2024-09-13T11:23:30.791 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:30.792 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:30.792 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:23:30.792 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:16.847135+0000", 2024-09-13T11:23:30.792 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.7511999999999999e-05 2024-09-13T11:23:30.792 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:30.792 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:30.792 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:23:30.792 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:16.847162+0000", 2024-09-13T11:23:30.792 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.7036999999999999e-05 2024-09-13T11:23:30.792 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:30.792 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:30.792 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:23:30.792 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:16.848961+0000", 2024-09-13T11:23:30.793 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001799036 2024-09-13T11:23:30.793 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:30.793 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:30.793 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:23:30.793 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:16.849026+0000", 2024-09-13T11:23:30.793 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.5226000000000002e-05 2024-09-13T11:23:30.793 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:30.793 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:23:30.793 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:30.793 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:30.793 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:30.793 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:41278 3.5 3:a4c6a539:::benchmark_data_smithi177_31116_object41277:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e65)", 2024-09-13T11:23:30.793 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:16.819499+0000", 2024-09-13T11:23:30.794 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 13.888788625, 2024-09-13T11:23:30.794 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.082103348000000007, 2024-09-13T11:23:30.794 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:23:30.794 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:23:30.794 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:30.794 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:23:30.794 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:23:30.794 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:30.794 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 41278 2024-09-13T11:23:30.794 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:30.794 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:23:30.794 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:30.794 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:23:30.794 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:16.819499+0000", 2024-09-13T11:23:30.795 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:30.795 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:30.795 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:30.795 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:23:30.795 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:16.819499+0000", 2024-09-13T11:23:30.795 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:30.795 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:30.795 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:30.795 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:23:30.795 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:16.819502+0000", 2024-09-13T11:23:30.795 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.1599999999999998e-06 2024-09-13T11:23:30.795 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:30.795 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:30.796 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:23:30.796 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:16.819530+0000", 2024-09-13T11:23:30.796 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.7613000000000001e-05 2024-09-13T11:23:30.796 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:30.796 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:30.796 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:23:30.796 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:16.819548+0000", 2024-09-13T11:23:30.796 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.8091e-05 2024-09-13T11:23:30.796 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:30.796 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:30.796 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:23:30.796 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:16.819557+0000", 2024-09-13T11:23:30.796 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.4050000000000006e-06 2024-09-13T11:23:30.797 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:30.797 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:30.797 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:23:30.797 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:16.819635+0000", 2024-09-13T11:23:30.797 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.7990000000000004e-05 2024-09-13T11:23:30.797 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:30.797 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:30.797 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:23:30.797 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:16.819813+0000", 2024-09-13T11:23:30.797 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00017749899999999999 2024-09-13T11:23:30.797 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:30.797 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:30.797 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 13", 2024-09-13T11:23:30.798 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:16.819984+0000", 2024-09-13T11:23:30.798 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00017193899999999999 2024-09-13T11:23:30.798 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:30.798 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:30.798 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:23:30.798 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:16.821659+0000", 2024-09-13T11:23:30.798 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0016748329999999999 2024-09-13T11:23:30.798 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:30.798 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:30.798 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:30.798 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:16.901504+0000", 2024-09-13T11:23:30.798 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.079845149000000004 2024-09-13T11:23:30.798 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:30.798 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:30.799 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:23:30.799 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:16.901547+0000", 2024-09-13T11:23:30.799 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.2553999999999999e-05 2024-09-13T11:23:30.799 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:30.799 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:30.799 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:23:30.799 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:16.901602+0000", 2024-09-13T11:23:30.799 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.5114999999999997e-05 2024-09-13T11:23:30.799 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:30.799 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:23:30.799 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:30.799 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:30.799 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:30.799 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop_reply(client.4665.0:41278 3.5 e65/30 ondisk, result = 0)", 2024-09-13T11:23:30.800 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:16.821657+0000", 2024-09-13T11:23:30.800 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 13.886630163, 2024-09-13T11:23:30.800 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.079952737999999995, 2024-09-13T11:23:30.800 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:23:30.800 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:23:30.800 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:23:30.800 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:23:30.800 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:30.800 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:23:30.800 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:16.821657+0000", 2024-09-13T11:23:30.800 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:30.800 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:30.800 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:30.800 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:23:30.800 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:16.821657+0000", 2024-09-13T11:23:30.801 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:30.801 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:30.801 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:30.801 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:23:30.801 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:16.821659+0000", 2024-09-13T11:23:30.801 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.801e-06 2024-09-13T11:23:30.801 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:30.801 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:30.801 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:23:30.801 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:16.821666+0000", 2024-09-13T11:23:30.801 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.6669999999999999e-06 2024-09-13T11:23:30.801 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:30.801 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:30.802 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:23:30.802 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:16.901428+0000", 2024-09-13T11:23:30.802 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.079762258000000003 2024-09-13T11:23:30.802 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:30.802 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:30.802 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:23:30.802 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:16.901436+0000", 2024-09-13T11:23:30.802 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.0460000000000006e-06 2024-09-13T11:23:30.802 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:30.802 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:30.802 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:23:30.802 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:16.901493+0000", 2024-09-13T11:23:30.802 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.6511999999999997e-05 2024-09-13T11:23:30.802 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:30.802 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:30.803 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:23:30.803 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:16.901500+0000", 2024-09-13T11:23:30.803 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.8990000000000001e-06 2024-09-13T11:23:30.803 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:30.803 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:30.803 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:23:30.803 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:16.901610+0000", 2024-09-13T11:23:30.803 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000109555 2024-09-13T11:23:30.803 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:30.803 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:23:30.803 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:30.803 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:30.803 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:30.804 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:41301 3.0 e65/30 3:02cfc2d3:::benchmark_data_smithi177_31116_object41300:head v 65'2613, mlcod=65'2610)", 2024-09-13T11:23:30.804 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:16.852266+0000", 2024-09-13T11:23:30.804 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 13.856021794, 2024-09-13T11:23:30.804 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.046276194, 2024-09-13T11:23:30.804 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:23:30.804 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:23:30.804 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:30.804 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:23:30.804 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:30.804 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:23:30.804 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:16.852266+0000", 2024-09-13T11:23:30.804 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:30.804 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:30.804 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:30.805 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:23:30.805 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:16.852266+0000", 2024-09-13T11:23:30.805 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:30.805 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:30.805 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:30.805 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:23:30.805 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:16.852267+0000", 2024-09-13T11:23:30.805 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5460000000000001e-06 2024-09-13T11:23:30.805 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:30.805 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:30.805 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:23:30.805 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:16.852316+0000", 2024-09-13T11:23:30.805 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.8584000000000001e-05 2024-09-13T11:23:30.805 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:30.805 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:30.806 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:23:30.806 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:16.896704+0000", 2024-09-13T11:23:30.806 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.044388351 2024-09-13T11:23:30.806 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:30.806 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:30.806 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:23:30.806 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:16.896714+0000", 2024-09-13T11:23:30.806 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.7289999999999995e-06 2024-09-13T11:23:30.806 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:30.806 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:30.806 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:23:30.806 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:16.896790+0000", 2024-09-13T11:23:30.806 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.6594999999999994e-05 2024-09-13T11:23:30.806 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:30.806 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:30.807 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:23:30.807 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:16.896810+0000", 2024-09-13T11:23:30.807 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.9423999999999999e-05 2024-09-13T11:23:30.807 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:30.807 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:30.807 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:23:30.807 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:16.898504+0000", 2024-09-13T11:23:30.807 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001694617 2024-09-13T11:23:30.807 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:30.807 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:30.807 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:23:30.807 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:16.898542+0000", 2024-09-13T11:23:30.807 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.7348000000000002e-05 2024-09-13T11:23:30.807 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:30.808 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:23:30.808 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:30.808 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:30.808 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:30.808 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:41541 3.5 3:a5fb3e8a:::benchmark_data_smithi177_31116_object41540:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e65)", 2024-09-13T11:23:30.808 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:17.081524+0000", 2024-09-13T11:23:30.808 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 13.62676364, 2024-09-13T11:23:30.808 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.054097655000000001, 2024-09-13T11:23:30.808 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:23:30.808 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:23:30.808 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:30.808 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:23:30.808 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:23:30.808 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:30.809 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 41541 2024-09-13T11:23:30.809 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:30.809 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:23:30.809 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:30.809 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:23:30.809 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:17.081524+0000", 2024-09-13T11:23:30.809 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:30.809 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:30.809 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:30.809 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:23:30.809 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:17.081524+0000", 2024-09-13T11:23:30.809 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:30.810 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:30.810 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:30.810 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:23:30.810 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:17.081527+0000", 2024-09-13T11:23:30.810 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.0280000000000001e-06 2024-09-13T11:23:30.810 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:30.810 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:30.810 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:23:30.810 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:17.081560+0000", 2024-09-13T11:23:30.810 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.3148999999999997e-05 2024-09-13T11:23:30.810 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:30.810 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:30.810 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:23:30.810 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:17.081581+0000", 2024-09-13T11:23:30.811 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.0905999999999999e-05 2024-09-13T11:23:30.811 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:30.811 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:30.811 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:23:30.811 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:17.081592+0000", 2024-09-13T11:23:30.811 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0879999999999999e-05 2024-09-13T11:23:30.811 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:30.811 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:30.811 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:23:30.811 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:17.081661+0000", 2024-09-13T11:23:30.811 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.8992000000000006e-05 2024-09-13T11:23:30.811 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:30.811 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:30.812 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:23:30.812 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:17.081782+0000", 2024-09-13T11:23:30.812 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000121159 2024-09-13T11:23:30.812 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:30.812 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:30.812 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 13", 2024-09-13T11:23:30.812 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:17.081926+0000", 2024-09-13T11:23:30.812 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000144454 2024-09-13T11:23:30.812 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:30.812 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:30.812 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:23:30.812 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:17.083717+0000", 2024-09-13T11:23:30.812 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001790343 2024-09-13T11:23:30.813 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:30.813 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:30.813 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:30.813 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:17.135532+0000", 2024-09-13T11:23:30.813 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.051815426999999997 2024-09-13T11:23:30.813 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:30.813 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:30.813 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:23:30.813 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:17.135572+0000", 2024-09-13T11:23:30.813 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.0136000000000003e-05 2024-09-13T11:23:30.813 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:30.813 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:30.813 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:23:30.813 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:17.135621+0000", 2024-09-13T11:23:30.814 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.9181000000000002e-05 2024-09-13T11:23:30.814 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:30.814 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:23:30.814 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:30.814 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:30.814 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:30.814 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:41562 3.5 3:a22c5a6f:::benchmark_data_smithi177_31116_object41561:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e65)", 2024-09-13T11:23:30.814 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:17.098774+0000", 2024-09-13T11:23:30.814 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 13.609513668, 2024-09-13T11:23:30.814 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.052671197000000003, 2024-09-13T11:23:30.814 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:23:30.814 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:23:30.814 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:30.815 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:23:30.815 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:23:30.815 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:30.815 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 41562 2024-09-13T11:23:30.815 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:30.815 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:23:30.815 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:30.815 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:23:30.815 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:17.098774+0000", 2024-09-13T11:23:30.815 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:30.815 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:30.815 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:30.815 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:23:30.816 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:17.098774+0000", 2024-09-13T11:23:30.816 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:30.816 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:30.816 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:30.816 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:23:30.816 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:17.098777+0000", 2024-09-13T11:23:30.816 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.5140000000000002e-06 2024-09-13T11:23:30.816 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:30.816 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:30.816 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:23:30.816 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:17.098816+0000", 2024-09-13T11:23:30.816 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.8911000000000001e-05 2024-09-13T11:23:30.816 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:30.817 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:30.817 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:23:30.817 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:17.098838+0000", 2024-09-13T11:23:30.817 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.1665e-05 2024-09-13T11:23:30.817 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:30.817 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:30.817 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:23:30.817 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:17.098850+0000", 2024-09-13T11:23:30.817 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.2091e-05 2024-09-13T11:23:30.817 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:30.817 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:30.817 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:23:30.817 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:17.098923+0000", 2024-09-13T11:23:30.818 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.3165999999999997e-05 2024-09-13T11:23:30.818 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:30.818 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:30.818 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:23:30.818 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:17.099032+0000", 2024-09-13T11:23:30.818 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000108964 2024-09-13T11:23:30.818 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:30.818 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:30.818 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 13", 2024-09-13T11:23:30.818 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:17.099143+0000", 2024-09-13T11:23:30.818 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00011065000000000001 2024-09-13T11:23:30.818 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:30.818 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:30.819 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:23:30.819 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:17.101083+0000", 2024-09-13T11:23:30.819 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0019405449999999999 2024-09-13T11:23:30.819 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:30.819 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:30.819 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:30.819 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:17.151206+0000", 2024-09-13T11:23:30.819 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.050122672 2024-09-13T11:23:30.819 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:30.819 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:30.819 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:23:30.819 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:17.151357+0000", 2024-09-13T11:23:30.819 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00015066100000000001 2024-09-13T11:23:30.819 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:30.819 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:30.820 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:23:30.820 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:17.151445+0000", 2024-09-13T11:23:30.820 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.8357999999999997e-05 2024-09-13T11:23:30.820 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:30.820 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:23:30.820 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:30.820 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:30.820 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:23:30.820 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-09-13T11:23:30.821 INFO:tasks.ceph.osd.1.smithi092.stderr:2024-09-13T11:23:30.820+0000 7fe7789c5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33718) UID: 0 2024-09-13T11:23:30.822 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-13T11:23:30.922 INFO:tasks.ceph.osd.2.smithi162.stderr:2024-09-13T11:23:30.917+0000 7fe92b35a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26313) UID: 0 2024-09-13T11:23:30.962 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-09-13T11:23:30.962 INFO:teuthology.orchestra.run.smithi092.stdout: "ops": [ 2024-09-13T11:23:30.962 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:30.962 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:54629 3.18 3.2120e58 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e74)", 2024-09-13T11:23:30.962 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:30.956152+0000", 2024-09-13T11:23:30.962 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 0.0051966210000000002, 2024-09-13T11:23:30.962 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0052159379999999998, 2024-09-13T11:23:30.962 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:23:30.962 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:23:30.962 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "queued for pg", 2024-09-13T11:23:30.962 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:23:30.963 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:23:30.963 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:30.963 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 54629 2024-09-13T11:23:30.963 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:30.963 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:23:30.963 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:30.963 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:23:30.963 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:30.956152+0000", 2024-09-13T11:23:30.963 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:30.963 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:30.963 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:30.963 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:23:30.963 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:30.956152+0000", 2024-09-13T11:23:30.964 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:30.964 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:30.964 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:30.964 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:23:30.964 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:30.956154+0000", 2024-09-13T11:23:30.964 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.2859999999999998e-06 2024-09-13T11:23:30.964 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:30.964 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:30.964 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:23:30.964 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:30.956196+0000", 2024-09-13T11:23:30.964 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.1526999999999997e-05 2024-09-13T11:23:30.964 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:30.964 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:30.965 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:23:30.965 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:30.956213+0000", 2024-09-13T11:23:30.965 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.7278e-05 2024-09-13T11:23:30.965 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:30.965 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:30.965 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:23:30.965 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:30.956222+0000", 2024-09-13T11:23:30.965 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.3790000000000006e-06 2024-09-13T11:23:30.965 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:30.965 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:23:30.965 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:30.965 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:30.965 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:30.965 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop(client.4665.0:54630 3.e e74/73 3:7eba6a09:::benchmark_data_smithi177_31116_object54629:head v 74'3518, mlcod=74'3516)", 2024-09-13T11:23:30.966 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:30.959325+0000", 2024-09-13T11:23:30.966 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 0.0020236529999999998, 2024-09-13T11:23:30.966 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0021122419999999999, 2024-09-13T11:23:30.966 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:23:30.966 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:23:30.966 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-09-13T11:23:30.966 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:23:30.966 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:30.966 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:23:30.966 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:30.959325+0000", 2024-09-13T11:23:30.966 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:30.966 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:30.966 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:30.967 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:23:30.967 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:30.959325+0000", 2024-09-13T11:23:30.967 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:30.967 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:30.967 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:30.967 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:23:30.967 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:30.959327+0000", 2024-09-13T11:23:30.967 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.677e-06 2024-09-13T11:23:30.967 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:30.967 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:30.967 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:23:30.967 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:30.959385+0000", 2024-09-13T11:23:30.967 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.8384000000000002e-05 2024-09-13T11:23:30.968 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:30.968 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:30.968 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:23:30.968 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:30.959416+0000", 2024-09-13T11:23:30.968 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.0859999999999999e-05 2024-09-13T11:23:30.968 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:30.968 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:30.968 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:23:30.968 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:30.959423+0000", 2024-09-13T11:23:30.968 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.1010000000000003e-06 2024-09-13T11:23:30.968 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:30.968 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:30.968 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:23:30.968 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:30.959474+0000", 2024-09-13T11:23:30.969 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.1075999999999997e-05 2024-09-13T11:23:30.969 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:30.969 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:30.969 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:23:30.969 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:30.959491+0000", 2024-09-13T11:23:30.969 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.7173999999999999e-05 2024-09-13T11:23:30.969 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:30.969 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:23:30.969 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:30.969 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:30.969 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:30.969 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:54634 3.18 3.ac5009f8 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e74)", 2024-09-13T11:23:30.969 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:30.960728+0000", 2024-09-13T11:23:30.969 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 0.00062085899999999997, 2024-09-13T11:23:30.970 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00075888799999999999, 2024-09-13T11:23:30.970 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:23:30.970 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:23:30.970 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "queued for pg", 2024-09-13T11:23:30.970 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:23:30.970 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:23:30.970 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:30.970 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 54634 2024-09-13T11:23:30.970 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:30.970 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:23:30.970 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:30.970 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:23:30.970 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:30.960728+0000", 2024-09-13T11:23:30.970 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:30.971 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:30.971 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:30.971 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:23:30.971 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:30.960728+0000", 2024-09-13T11:23:30.971 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:30.971 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:30.971 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:30.971 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:23:30.971 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:30.960729+0000", 2024-09-13T11:23:30.971 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.618e-06 2024-09-13T11:23:30.971 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:30.971 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:30.971 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:23:30.971 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:30.960775+0000", 2024-09-13T11:23:30.972 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.5507999999999999e-05 2024-09-13T11:23:30.972 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:30.972 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:30.972 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:23:30.972 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:30.960786+0000", 2024-09-13T11:23:30.972 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0941000000000001e-05 2024-09-13T11:23:30.972 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:30.972 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:30.972 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:23:30.972 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:30.960794+0000", 2024-09-13T11:23:30.972 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.5970000000000003e-06 2024-09-13T11:23:30.972 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:30.972 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:23:30.972 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:30.973 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:30.973 INFO:teuthology.orchestra.run.smithi092.stdout: ], 2024-09-13T11:23:30.973 INFO:teuthology.orchestra.run.smithi092.stdout: "num_ops": 3 2024-09-13T11:23:30.973 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-09-13T11:23:30.982 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-13T11:23:30.984 INFO:tasks.ceph.osd.11.smithi177.stderr:2024-09-13T11:23:30.977+0000 7f246bcf0640 -1 osd.11 74 handle_osd_map injecting map crc failure 2024-09-13T11:23:30.987 INFO:tasks.ceph.osd.8.smithi001.stderr:2024-09-13T11:23:30.980+0000 7fb9ab2df640 -1 osd.8 74 handle_osd_map injecting map crc failure 2024-09-13T11:23:31.004 INFO:tasks.radosbench.radosbench.0.smithi177.stdout: 64 16 54660 54644 53.3575 63.1875 0.0399491 0.0187284 2024-09-13T11:23:31.014 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-13T11:23:31.023 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-09-13T11:23:31.017+0000 7f97befc9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26315) UID: 0 2024-09-13T11:23:31.124 INFO:tasks.ceph.osd.9.smithi092.stderr:2024-09-13T11:23:31.120+0000 7ff5ab3d0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26348) UID: 0 2024-09-13T11:23:31.141 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-09-13T11:23:31.142 INFO:teuthology.orchestra.run.smithi092.stdout: "ops": [], 2024-09-13T11:23:31.142 INFO:teuthology.orchestra.run.smithi092.stdout: "complaint_time": 30, 2024-09-13T11:23:31.142 INFO:teuthology.orchestra.run.smithi092.stdout: "num_blocked_ops": 0 2024-09-13T11:23:31.142 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-09-13T11:23:31.160 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-13T11:23:31.226 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-09-13T11:23:31.221+0000 7f97befc9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26315) UID: 0 2024-09-13T11:23:31.326 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-13T11:23:31.320+0000 7f6a68775640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26307) UID: 0 2024-09-13T11:23:31.337 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-09-13T11:23:31.337 INFO:teuthology.orchestra.run.smithi092.stdout: "size": 20, 2024-09-13T11:23:31.337 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 600, 2024-09-13T11:23:31.337 INFO:teuthology.orchestra.run.smithi092.stdout: "ops": [ 2024-09-13T11:23:31.337 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:31.337 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:3 3.4 3.125e09e4 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e40)", 2024-09-13T11:23:31.337 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:46.304641+0000", 2024-09-13T11:23:31.338 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 45.03088777, 2024-09-13T11:23:31.338 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.374372119, 2024-09-13T11:23:31.338 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:23:31.338 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:23:31.338 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "reached pg", 2024-09-13T11:23:31.338 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:23:31.338 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:23:31.338 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:31.338 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 3 2024-09-13T11:23:31.338 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:31.338 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:23:31.338 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:31.339 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:23:31.339 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.304641+0000", 2024-09-13T11:23:31.339 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:31.339 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:31.339 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:31.339 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:23:31.339 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.304641+0000", 2024-09-13T11:23:31.339 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:31.339 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:31.339 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:31.339 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:23:31.339 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.304644+0000", 2024-09-13T11:23:31.339 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.5130000000000001e-06 2024-09-13T11:23:31.339 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:31.340 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:31.340 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:23:31.340 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305058+0000", 2024-09-13T11:23:31.340 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000414277 2024-09-13T11:23:31.340 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:31.340 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:31.340 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:23:31.340 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305124+0000", 2024-09-13T11:23:31.340 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.5813000000000002e-05 2024-09-13T11:23:31.340 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:31.340 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:31.340 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:23:31.341 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305156+0000", 2024-09-13T11:23:31.341 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.1489000000000003e-05 2024-09-13T11:23:31.341 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:31.341 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:31.341 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:23:31.341 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.678939+0000", 2024-09-13T11:23:31.341 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.37378339999999999 2024-09-13T11:23:31.341 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:31.341 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:31.341 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:23:31.341 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.679013+0000", 2024-09-13T11:23:31.341 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.3627000000000005e-05 2024-09-13T11:23:31.342 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:31.342 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:23:31.342 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:31.342 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:31.342 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:31.342 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:29 3.4 3.b7bbfbe4 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-09-13T11:23:31.342 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:46.305215+0000", 2024-09-13T11:23:31.342 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 45.030313466999999, 2024-09-13T11:23:31.342 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.37387964600000001, 2024-09-13T11:23:31.342 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:23:31.342 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:23:31.342 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "reached pg", 2024-09-13T11:23:31.342 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:23:31.343 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:23:31.343 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:31.343 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 29 2024-09-13T11:23:31.343 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:31.343 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:23:31.343 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:31.343 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:23:31.343 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305215+0000", 2024-09-13T11:23:31.343 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:31.343 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:31.343 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:31.343 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:23:31.343 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305215+0000", 2024-09-13T11:23:31.344 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:31.344 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:31.344 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:31.344 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:23:31.344 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305218+0000", 2024-09-13T11:23:31.344 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.7030000000000002e-06 2024-09-13T11:23:31.344 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:31.344 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:31.344 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:23:31.344 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305366+0000", 2024-09-13T11:23:31.344 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00014825400000000001 2024-09-13T11:23:31.345 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:31.345 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:31.345 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:23:31.345 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305386+0000", 2024-09-13T11:23:31.345 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.9689000000000001e-05 2024-09-13T11:23:31.345 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:31.345 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:31.345 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:23:31.345 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305403+0000", 2024-09-13T11:23:31.346 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.7600000000000001e-05 2024-09-13T11:23:31.346 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:31.346 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:31.346 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:23:31.346 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.679067+0000", 2024-09-13T11:23:31.346 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.37366339100000001 2024-09-13T11:23:31.346 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:31.346 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:31.346 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:23:31.346 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.679095+0000", 2024-09-13T11:23:31.346 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.8008999999999999e-05 2024-09-13T11:23:31.346 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:31.346 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:23:31.347 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:31.347 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:31.347 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:31.347 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:30 3.4 3.547ec0f4 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-09-13T11:23:31.347 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:46.305453+0000", 2024-09-13T11:23:31.347 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 45.030075343, 2024-09-13T11:23:31.347 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.373764914, 2024-09-13T11:23:31.347 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:23:31.347 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:23:31.347 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "reached pg", 2024-09-13T11:23:31.347 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:23:31.347 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:23:31.347 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:31.348 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 30 2024-09-13T11:23:31.348 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:31.348 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:23:31.348 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:31.348 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:23:31.348 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305453+0000", 2024-09-13T11:23:31.348 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:31.348 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:31.348 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:31.348 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:23:31.348 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305453+0000", 2024-09-13T11:23:31.348 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:31.348 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:31.349 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:31.349 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:23:31.349 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305456+0000", 2024-09-13T11:23:31.349 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.9560000000000002e-06 2024-09-13T11:23:31.349 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:31.349 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:31.349 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:23:31.349 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305597+0000", 2024-09-13T11:23:31.349 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00014122700000000001 2024-09-13T11:23:31.349 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:31.349 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:31.349 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:23:31.349 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305615+0000", 2024-09-13T11:23:31.350 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.8139999999999999e-05 2024-09-13T11:23:31.350 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:31.350 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:31.350 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:23:31.350 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305632+0000", 2024-09-13T11:23:31.350 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.6101999999999999e-05 2024-09-13T11:23:31.350 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:31.350 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:31.350 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:23:31.350 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.679195+0000", 2024-09-13T11:23:31.350 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.373563486 2024-09-13T11:23:31.350 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:31.350 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:31.351 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:23:31.351 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.679218+0000", 2024-09-13T11:23:31.351 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.3003e-05 2024-09-13T11:23:31.351 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:31.351 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:23:31.351 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:31.351 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:31.351 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:31.351 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:70 3.4 3.1404a604 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-09-13T11:23:31.351 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:46.305665+0000", 2024-09-13T11:23:31.351 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 45.029863194999997, 2024-09-13T11:23:31.351 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.37365550199999997, 2024-09-13T11:23:31.351 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:23:31.352 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:23:31.352 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "reached pg", 2024-09-13T11:23:31.352 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:23:31.352 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:23:31.352 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:31.352 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 70 2024-09-13T11:23:31.352 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:31.352 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:23:31.352 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:31.352 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:23:31.352 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305665+0000", 2024-09-13T11:23:31.352 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:31.352 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:31.353 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:31.353 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:23:31.353 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305665+0000", 2024-09-13T11:23:31.353 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:31.353 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:31.353 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:31.353 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:23:31.353 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305668+0000", 2024-09-13T11:23:31.353 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.5890000000000001e-06 2024-09-13T11:23:31.353 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:31.353 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:31.353 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:23:31.353 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305791+0000", 2024-09-13T11:23:31.354 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000123545 2024-09-13T11:23:31.354 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:31.354 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:31.354 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:23:31.354 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305808+0000", 2024-09-13T11:23:31.354 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.668e-05 2024-09-13T11:23:31.354 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:31.354 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:31.354 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:23:31.354 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305824+0000", 2024-09-13T11:23:31.354 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.6385e-05 2024-09-13T11:23:31.354 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:31.354 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:31.355 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:23:31.355 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.679305+0000", 2024-09-13T11:23:31.355 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.37348059900000002 2024-09-13T11:23:31.355 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:31.355 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:31.355 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:23:31.355 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.679321+0000", 2024-09-13T11:23:31.355 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.5704000000000001e-05 2024-09-13T11:23:31.355 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:31.355 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:23:31.355 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:31.355 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:31.356 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:31.356 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:25290 3.4 3:2c6b3a0e:::benchmark_data_smithi177_31116_object25289:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e55)", 2024-09-13T11:23:31.356 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:04.310470+0000", 2024-09-13T11:23:31.356 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 27.025058468000001, 2024-09-13T11:23:31.356 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.095019785999999995, 2024-09-13T11:23:31.356 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:23:31.356 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:23:31.356 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:31.356 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:23:31.356 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:23:31.356 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:31.356 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 25290 2024-09-13T11:23:31.356 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:31.357 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:23:31.357 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:31.357 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:23:31.357 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.310470+0000", 2024-09-13T11:23:31.357 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:31.357 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:31.357 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:31.357 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:23:31.357 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.310470+0000", 2024-09-13T11:23:31.357 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:31.357 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:31.357 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:31.357 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:23:31.357 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.310472+0000", 2024-09-13T11:23:31.358 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.481e-06 2024-09-13T11:23:31.358 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:31.358 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:31.358 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:23:31.358 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.310527+0000", 2024-09-13T11:23:31.358 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.4999e-05 2024-09-13T11:23:31.358 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:31.358 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:31.358 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:23:31.358 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.310540+0000", 2024-09-13T11:23:31.358 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.2206e-05 2024-09-13T11:23:31.358 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:31.358 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:31.359 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:23:31.359 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.310545+0000", 2024-09-13T11:23:31.359 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.7749999999999998e-06 2024-09-13T11:23:31.359 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:31.359 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:31.359 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:23:31.359 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.310598+0000", 2024-09-13T11:23:31.359 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.2095999999999997e-05 2024-09-13T11:23:31.359 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:31.359 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:31.359 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:23:31.359 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.310706+0000", 2024-09-13T11:23:31.359 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000108655 2024-09-13T11:23:31.360 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:31.360 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:31.360 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 12", 2024-09-13T11:23:31.360 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.310793+0000", 2024-09-13T11:23:31.360 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.7023999999999999e-05 2024-09-13T11:23:31.360 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:31.360 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:31.360 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:23:31.360 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.311820+0000", 2024-09-13T11:23:31.360 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.001027062 2024-09-13T11:23:31.360 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:31.360 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:31.360 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:31.361 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.405301+0000", 2024-09-13T11:23:31.361 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.093480806 2024-09-13T11:23:31.361 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:31.361 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:31.361 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:23:31.361 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.405400+0000", 2024-09-13T11:23:31.361 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.9279999999999998e-05 2024-09-13T11:23:31.361 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:31.361 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:31.361 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:23:31.361 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.405490+0000", 2024-09-13T11:23:31.361 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.9401999999999995e-05 2024-09-13T11:23:31.361 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:31.362 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:23:31.362 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:31.362 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:31.362 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:31.362 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:25408 3.4 3:294e23bd:::benchmark_data_smithi177_31116_object25407:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e56)", 2024-09-13T11:23:31.362 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:04.495455+0000", 2024-09-13T11:23:31.362 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 26.840073542999999, 2024-09-13T11:23:31.362 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.29897367600000002, 2024-09-13T11:23:31.362 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:23:31.362 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:23:31.362 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:31.362 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:23:31.362 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:23:31.363 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:31.363 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 25408 2024-09-13T11:23:31.363 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:31.363 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:23:31.363 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:31.363 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:23:31.363 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.495455+0000", 2024-09-13T11:23:31.363 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:31.363 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:31.363 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:31.363 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:23:31.363 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.495455+0000", 2024-09-13T11:23:31.363 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:31.364 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:31.364 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:31.364 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:23:31.364 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.495459+0000", 2024-09-13T11:23:31.364 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.5630000000000002e-06 2024-09-13T11:23:31.364 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:31.364 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:31.364 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:23:31.364 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.495545+0000", 2024-09-13T11:23:31.364 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.5704000000000005e-05 2024-09-13T11:23:31.364 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:31.364 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:31.364 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:23:31.364 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.495581+0000", 2024-09-13T11:23:31.365 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.5689000000000003e-05 2024-09-13T11:23:31.365 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:31.365 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:31.365 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:23:31.365 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.495597+0000", 2024-09-13T11:23:31.365 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.6053e-05 2024-09-13T11:23:31.365 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:31.365 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:31.365 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:23:31.365 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.495721+0000", 2024-09-13T11:23:31.365 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00012379099999999999 2024-09-13T11:23:31.365 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:31.365 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:31.365 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:23:31.366 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.495940+0000", 2024-09-13T11:23:31.366 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00021967699999999999 2024-09-13T11:23:31.366 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:31.366 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:31.366 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 12", 2024-09-13T11:23:31.366 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.496162+0000", 2024-09-13T11:23:31.366 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000221917 2024-09-13T11:23:31.366 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:31.366 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:31.366 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:23:31.366 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.498699+0000", 2024-09-13T11:23:31.366 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0025366690000000001 2024-09-13T11:23:31.366 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:31.367 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:31.367 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:31.367 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.794139+0000", 2024-09-13T11:23:31.367 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.295440025 2024-09-13T11:23:31.367 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:31.367 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:31.367 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:23:31.367 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.794306+0000", 2024-09-13T11:23:31.367 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00016689799999999999 2024-09-13T11:23:31.367 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:31.367 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:31.367 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:23:31.367 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.794429+0000", 2024-09-13T11:23:31.368 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00012269000000000001 2024-09-13T11:23:31.368 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:31.368 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:23:31.368 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:31.368 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:31.368 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:31.368 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop(client.4665.0:30353 3.b e59/40 3:da72bd4f:::benchmark_data_smithi177_31116_object30352:head v 59'1909, mlcod=59'1906)", 2024-09-13T11:23:31.368 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:09.165566+0000", 2024-09-13T11:23:31.368 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 22.169962456, 2024-09-13T11:23:31.368 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.090209561999999993, 2024-09-13T11:23:31.368 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:23:31.368 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:23:31.369 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:31.369 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:23:31.369 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:31.369 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:23:31.369 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:09.165566+0000", 2024-09-13T11:23:31.369 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:31.369 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:31.369 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:31.369 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:23:31.369 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:09.165566+0000", 2024-09-13T11:23:31.369 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:31.369 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:31.369 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:31.369 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:23:31.370 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:09.165568+0000", 2024-09-13T11:23:31.370 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.4449999999999999e-06 2024-09-13T11:23:31.370 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:31.370 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:31.370 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:23:31.370 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:09.165621+0000", 2024-09-13T11:23:31.370 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.2533e-05 2024-09-13T11:23:31.370 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:31.370 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:31.370 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:23:31.370 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:09.249936+0000", 2024-09-13T11:23:31.370 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.084315347999999998 2024-09-13T11:23:31.370 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:31.371 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:31.371 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:23:31.371 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:09.249954+0000", 2024-09-13T11:23:31.371 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.7479e-05 2024-09-13T11:23:31.371 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:31.371 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:31.371 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:23:31.371 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:09.250071+0000", 2024-09-13T11:23:31.371 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000116827 2024-09-13T11:23:31.371 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:31.371 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:31.371 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:23:31.371 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:09.250105+0000", 2024-09-13T11:23:31.372 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.4212999999999999e-05 2024-09-13T11:23:31.372 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:31.372 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:31.372 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:23:31.372 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:09.255685+0000", 2024-09-13T11:23:31.372 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.005580588 2024-09-13T11:23:31.372 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:31.372 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:31.372 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:23:31.372 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:09.255776+0000", 2024-09-13T11:23:31.372 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.0129000000000004e-05 2024-09-13T11:23:31.372 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:31.372 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:23:31.372 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:31.373 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:31.373 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:31.373 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop(client.4665.0:30855 3.b e59/40 3:dc59714f:::benchmark_data_smithi177_31116_object30854:head v 59'1930, mlcod=59'1927)", 2024-09-13T11:23:31.373 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:09.550713+0000", 2024-09-13T11:23:31.373 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 21.784815751, 2024-09-13T11:23:31.373 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.17337753, 2024-09-13T11:23:31.373 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:23:31.373 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:23:31.373 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:31.373 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:23:31.373 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:31.373 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:23:31.373 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:09.550713+0000", 2024-09-13T11:23:31.374 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:31.374 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:31.374 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:31.374 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:23:31.374 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:09.550713+0000", 2024-09-13T11:23:31.374 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:31.374 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:31.374 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:31.374 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:23:31.374 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:09.550716+0000", 2024-09-13T11:23:31.374 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.5099999999999999e-06 2024-09-13T11:23:31.375 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:31.375 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:31.375 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:23:31.375 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:09.550764+0000", 2024-09-13T11:23:31.375 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.7389000000000003e-05 2024-09-13T11:23:31.375 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:31.375 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:31.375 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:23:31.375 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:09.719062+0000", 2024-09-13T11:23:31.375 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.16829854499999999 2024-09-13T11:23:31.375 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:31.375 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:31.375 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:23:31.375 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:09.719075+0000", 2024-09-13T11:23:31.376 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.2969000000000001e-05 2024-09-13T11:23:31.376 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:31.376 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:31.376 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:23:31.376 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:09.719216+0000", 2024-09-13T11:23:31.376 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000141266 2024-09-13T11:23:31.376 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:31.376 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:31.376 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:23:31.376 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:09.719277+0000", 2024-09-13T11:23:31.376 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.0244999999999997e-05 2024-09-13T11:23:31.376 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:31.376 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:31.376 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:23:31.377 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:09.724019+0000", 2024-09-13T11:23:31.377 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0047426020000000003 2024-09-13T11:23:31.377 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:31.377 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:31.377 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:23:31.377 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:09.724090+0000", 2024-09-13T11:23:31.377 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.1004e-05 2024-09-13T11:23:31.377 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:31.377 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:23:31.377 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:31.377 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:31.377 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:31.377 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop(client.4665.0:41257 3.b e65/40 3:d342dab2:::benchmark_data_smithi177_31116_object41256:head v 65'2613, mlcod=65'2610)", 2024-09-13T11:23:31.377 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:16.800467+0000", 2024-09-13T11:23:31.378 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 14.535061505, 2024-09-13T11:23:31.378 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.095071575000000005, 2024-09-13T11:23:31.378 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:23:31.378 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:23:31.378 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:31.378 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:23:31.378 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:31.378 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:23:31.378 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:16.800467+0000", 2024-09-13T11:23:31.378 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:31.378 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:31.378 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:31.378 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:23:31.379 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:16.800467+0000", 2024-09-13T11:23:31.379 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:31.379 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:31.379 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:31.379 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:23:31.379 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:16.800469+0000", 2024-09-13T11:23:31.379 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.2340000000000001e-06 2024-09-13T11:23:31.379 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:31.379 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:31.379 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:23:31.379 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:16.800495+0000", 2024-09-13T11:23:31.379 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.5434000000000001e-05 2024-09-13T11:23:31.379 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:31.379 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:31.380 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:23:31.380 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:16.893496+0000", 2024-09-13T11:23:31.380 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.093001167999999995 2024-09-13T11:23:31.380 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:31.380 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:31.380 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:23:31.380 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:16.893503+0000", 2024-09-13T11:23:31.380 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.029e-06 2024-09-13T11:23:31.380 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:31.380 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:31.380 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:23:31.380 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:16.893569+0000", 2024-09-13T11:23:31.380 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.6291999999999995e-05 2024-09-13T11:23:31.380 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:31.381 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:31.381 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:23:31.381 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:16.893595+0000", 2024-09-13T11:23:31.381 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.5377000000000001e-05 2024-09-13T11:23:31.381 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:31.381 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:31.381 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:23:31.381 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:16.895501+0000", 2024-09-13T11:23:31.381 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.001906865 2024-09-13T11:23:31.381 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:31.381 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:31.381 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:23:31.381 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:16.895539+0000", 2024-09-13T11:23:31.382 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.7175999999999998e-05 2024-09-13T11:23:31.382 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:31.382 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:23:31.382 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:31.382 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:31.382 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:31.382 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop(client.4665.0:45770 3.b e68/40 3:d2ece503:::benchmark_data_smithi177_31116_object45769:head v 68'2878, mlcod=68'2875)", 2024-09-13T11:23:31.382 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:20.925073+0000", 2024-09-13T11:23:31.382 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 10.410455525, 2024-09-13T11:23:31.382 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.14884730400000001, 2024-09-13T11:23:31.382 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:23:31.382 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:23:31.382 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:31.383 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:23:31.383 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:31.383 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:23:31.383 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:20.925073+0000", 2024-09-13T11:23:31.383 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:31.383 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:31.383 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:31.383 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:23:31.383 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:20.925073+0000", 2024-09-13T11:23:31.383 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:31.383 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:31.383 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:31.383 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:23:31.384 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:20.925075+0000", 2024-09-13T11:23:31.384 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.9360000000000002e-06 2024-09-13T11:23:31.384 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:31.384 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:31.384 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:23:31.384 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:20.925100+0000", 2024-09-13T11:23:31.384 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.5602e-05 2024-09-13T11:23:31.384 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:31.384 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:31.384 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:23:31.384 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:21.070362+0000", 2024-09-13T11:23:31.384 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.14526132 2024-09-13T11:23:31.385 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:31.385 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:31.385 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:23:31.385 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:21.070375+0000", 2024-09-13T11:23:31.385 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.3636e-05 2024-09-13T11:23:31.385 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:31.385 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:31.385 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:23:31.385 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:21.070523+0000", 2024-09-13T11:23:31.386 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00014711999999999999 2024-09-13T11:23:31.386 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:31.386 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:31.386 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:23:31.386 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:21.070577+0000", 2024-09-13T11:23:31.386 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.3987000000000003e-05 2024-09-13T11:23:31.386 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:31.386 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:31.386 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:23:31.386 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:21.073840+0000", 2024-09-13T11:23:31.386 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0032633050000000002 2024-09-13T11:23:31.386 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:31.386 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:31.386 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:23:31.387 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:21.073920+0000", 2024-09-13T11:23:31.387 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.0397999999999998e-05 2024-09-13T11:23:31.387 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:31.387 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:23:31.387 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:31.387 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:31.387 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:31.387 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:48309 3.18 3:1a4870be:::benchmark_data_smithi177_31116_object48308:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e71)", 2024-09-13T11:23:31.388 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:24.915457+0000", 2024-09-13T11:23:31.388 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 6.4200714779999997, 2024-09-13T11:23:31.388 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.099135997000000003, 2024-09-13T11:23:31.388 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:23:31.388 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:23:31.388 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:31.388 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:23:31.388 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:23:31.388 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:31.388 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 48309 2024-09-13T11:23:31.388 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:31.388 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:23:31.389 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:31.389 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:23:31.389 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:24.915457+0000", 2024-09-13T11:23:31.389 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:31.389 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:31.389 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:31.389 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:23:31.389 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:24.915457+0000", 2024-09-13T11:23:31.389 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:31.389 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:31.389 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:31.389 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:23:31.389 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:24.915459+0000", 2024-09-13T11:23:31.390 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.6419999999999999e-06 2024-09-13T11:23:31.390 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:31.390 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:31.390 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:23:31.390 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:24.915509+0000", 2024-09-13T11:23:31.390 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.0065999999999998e-05 2024-09-13T11:23:31.390 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:31.390 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:31.390 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:23:31.390 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:24.915520+0000", 2024-09-13T11:23:31.390 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0868e-05 2024-09-13T11:23:31.390 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:31.390 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:31.391 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:23:31.391 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:24.915527+0000", 2024-09-13T11:23:31.391 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.5320000000000001e-06 2024-09-13T11:23:31.391 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:31.391 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:31.391 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:23:31.391 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:24.915572+0000", 2024-09-13T11:23:31.391 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.5315999999999998e-05 2024-09-13T11:23:31.391 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:31.391 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:31.391 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:23:31.391 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:24.915686+0000", 2024-09-13T11:23:31.391 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00011357999999999999 2024-09-13T11:23:31.391 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:31.392 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:31.392 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 0,10", 2024-09-13T11:23:31.392 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:24.915832+0000", 2024-09-13T11:23:31.392 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00014605400000000001 2024-09-13T11:23:31.392 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:31.392 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:31.392 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:23:31.392 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:24.917508+0000", 2024-09-13T11:23:31.392 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0016761219999999999 2024-09-13T11:23:31.392 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:31.392 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:31.392 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:31.392 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:24.919180+0000", 2024-09-13T11:23:31.393 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0016717380000000001 2024-09-13T11:23:31.393 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:31.393 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:31.393 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:31.393 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:25.013864+0000", 2024-09-13T11:23:31.393 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.09468385 2024-09-13T11:23:31.393 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:31.393 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:31.393 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:23:31.393 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:25.013928+0000", 2024-09-13T11:23:31.393 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.4450999999999997e-05 2024-09-13T11:23:31.393 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:31.394 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:31.394 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:23:31.394 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:25.014593+0000", 2024-09-13T11:23:31.394 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00066477799999999998 2024-09-13T11:23:31.394 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:31.394 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:23:31.394 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:31.394 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:31.394 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:31.394 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop_reply(client.4665.0:48309 3.18 e71/70 ondisk, result = 0)", 2024-09-13T11:23:31.394 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:24.917763+0000", 2024-09-13T11:23:31.394 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 6.4177659560000002, 2024-09-13T11:23:31.394 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.096835597999999995, 2024-09-13T11:23:31.394 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:23:31.395 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:23:31.395 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-09-13T11:23:31.395 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:23:31.395 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:31.395 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:23:31.395 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:24.917763+0000", 2024-09-13T11:23:31.395 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:31.395 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:31.395 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:31.395 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:23:31.395 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:24.917763+0000", 2024-09-13T11:23:31.395 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:31.395 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:31.396 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:31.396 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:23:31.396 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:24.917764+0000", 2024-09-13T11:23:31.396 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.2619999999999999e-06 2024-09-13T11:23:31.396 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:31.396 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:31.396 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:23:31.396 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:24.917770+0000", 2024-09-13T11:23:31.396 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.3069999999999999e-06 2024-09-13T11:23:31.396 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:31.396 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:31.396 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:23:31.396 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:25.013790+0000", 2024-09-13T11:23:31.397 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.096019541 2024-09-13T11:23:31.397 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:31.397 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:31.397 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:23:31.397 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:25.013796+0000", 2024-09-13T11:23:31.397 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.9209999999999997e-06 2024-09-13T11:23:31.397 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:31.397 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:31.397 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:23:31.397 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:25.013845+0000", 2024-09-13T11:23:31.397 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.9310000000000001e-05 2024-09-13T11:23:31.397 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:31.397 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:31.398 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:23:31.398 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:25.013857+0000", 2024-09-13T11:23:31.398 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.1681e-05 2024-09-13T11:23:31.398 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:31.398 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:31.398 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:23:31.398 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:25.014598+0000", 2024-09-13T11:23:31.398 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00074157599999999998 2024-09-13T11:23:31.398 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:31.398 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:23:31.398 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:31.398 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:31.398 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:31.399 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop(client.4665.0:48977 3.8 e71/67 3:1030343e:::benchmark_data_smithi177_31116_object48976:head v 71'2885, mlcod=71'2882)", 2024-09-13T11:23:31.399 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:25.518319+0000", 2024-09-13T11:23:31.399 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 5.8172093690000004, 2024-09-13T11:23:31.399 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.14420084699999999, 2024-09-13T11:23:31.399 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:23:31.399 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:23:31.399 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:31.399 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:23:31.399 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:31.399 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:23:31.399 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:25.518319+0000", 2024-09-13T11:23:31.399 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:31.399 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:31.400 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:31.400 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:23:31.400 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:25.518319+0000", 2024-09-13T11:23:31.400 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:31.400 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:31.400 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:31.400 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:23:31.400 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:25.518320+0000", 2024-09-13T11:23:31.400 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.215e-06 2024-09-13T11:23:31.400 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:31.400 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:31.400 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:23:31.400 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:25.518385+0000", 2024-09-13T11:23:31.400 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.5124999999999997e-05 2024-09-13T11:23:31.401 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:31.401 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:31.401 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:23:31.401 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:25.661380+0000", 2024-09-13T11:23:31.401 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.142994184 2024-09-13T11:23:31.401 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:31.401 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:31.401 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:23:31.401 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:25.661384+0000", 2024-09-13T11:23:31.401 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.3150000000000002e-06 2024-09-13T11:23:31.401 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:31.401 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:31.401 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:23:31.402 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:25.661433+0000", 2024-09-13T11:23:31.402 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.8789999999999999e-05 2024-09-13T11:23:31.402 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:31.402 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:31.402 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:23:31.402 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:25.661446+0000", 2024-09-13T11:23:31.402 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.2938e-05 2024-09-13T11:23:31.402 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:31.402 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:31.402 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:23:31.402 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:25.662490+0000", 2024-09-13T11:23:31.402 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.001044779 2024-09-13T11:23:31.402 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:31.402 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:31.403 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:23:31.403 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:25.662520+0000", 2024-09-13T11:23:31.403 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.9501000000000001e-05 2024-09-13T11:23:31.403 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:31.403 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:23:31.403 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:31.403 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:31.403 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:31.403 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:49963 3.18 3:19b86b73:::benchmark_data_smithi177_31116_object49962:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e72)", 2024-09-13T11:23:31.403 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:26.307557+0000", 2024-09-13T11:23:31.403 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 5.0279715679999999, 2024-09-13T11:23:31.403 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.22140597300000001, 2024-09-13T11:23:31.403 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:23:31.404 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:23:31.404 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:31.404 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:23:31.404 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:23:31.404 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:31.404 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 49963 2024-09-13T11:23:31.404 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:31.404 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:23:31.404 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:31.404 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:23:31.404 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:26.307557+0000", 2024-09-13T11:23:31.404 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:31.405 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:31.405 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:31.405 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:23:31.405 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:26.307557+0000", 2024-09-13T11:23:31.405 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:31.405 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:31.405 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:31.405 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:23:31.405 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:26.307558+0000", 2024-09-13T11:23:31.405 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.3740000000000001e-06 2024-09-13T11:23:31.405 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:31.405 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:31.405 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:23:31.406 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:26.516580+0000", 2024-09-13T11:23:31.406 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.209021596 2024-09-13T11:23:31.406 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:31.406 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:31.406 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:23:31.406 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:26.516614+0000", 2024-09-13T11:23:31.406 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.3858e-05 2024-09-13T11:23:31.406 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:31.406 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:31.406 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:23:31.406 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:26.516628+0000", 2024-09-13T11:23:31.406 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.4715999999999999e-05 2024-09-13T11:23:31.406 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:31.406 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:31.407 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:23:31.407 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:26.516790+0000", 2024-09-13T11:23:31.407 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00016138 2024-09-13T11:23:31.407 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:31.407 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:31.407 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:23:31.407 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:26.517123+0000", 2024-09-13T11:23:31.407 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00033296600000000002 2024-09-13T11:23:31.407 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:31.407 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:31.407 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 0,10", 2024-09-13T11:23:31.407 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:26.517367+0000", 2024-09-13T11:23:31.407 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00024416900000000003 2024-09-13T11:23:31.408 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:31.408 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:31.408 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:23:31.408 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:26.526939+0000", 2024-09-13T11:23:31.408 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0095718650000000006 2024-09-13T11:23:31.408 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:31.408 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:31.408 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:31.408 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:26.527095+0000", 2024-09-13T11:23:31.408 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00015643800000000001 2024-09-13T11:23:31.408 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:31.408 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:31.408 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:31.409 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:26.527254+0000", 2024-09-13T11:23:31.409 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000158383 2024-09-13T11:23:31.409 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:31.409 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:31.409 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:23:31.409 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:26.527405+0000", 2024-09-13T11:23:31.409 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000151731 2024-09-13T11:23:31.409 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:31.409 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:31.409 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:23:31.409 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:26.528963+0000", 2024-09-13T11:23:31.409 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0015574969999999999 2024-09-13T11:23:31.409 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:31.410 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:23:31.410 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:31.410 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:31.410 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:31.410 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "MOSDPGPush(3.e 72/57 [PushOp(3:7481641d:::benchmark_data_smithi177_31116_object23454:head, version: 55'1541, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:7481641d:::benchmark_data_smithi177_31116_object23454:head@55'1541, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:23:31.410 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:26.368394+0000", 2024-09-13T11:23:31.410 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 4.9671347900000002, 2024-09-13T11:23:31.410 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.16488114500000001, 2024-09-13T11:23:31.410 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:23:31.410 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:23:31.410 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-09-13T11:23:31.410 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:23:31.410 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:31.411 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:23:31.411 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:26.368394+0000", 2024-09-13T11:23:31.411 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:31.411 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:31.411 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:31.411 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:23:31.411 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:26.368394+0000", 2024-09-13T11:23:31.411 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:31.411 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:31.411 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:31.411 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:23:31.411 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:26.368395+0000", 2024-09-13T11:23:31.411 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.054e-06 2024-09-13T11:23:31.411 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:31.412 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:31.412 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:23:31.412 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:26.368453+0000", 2024-09-13T11:23:31.412 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.8621e-05 2024-09-13T11:23:31.412 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:31.412 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:31.412 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:23:31.412 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:26.528803+0000", 2024-09-13T11:23:31.412 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.16034983 2024-09-13T11:23:31.412 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:31.412 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:31.412 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:23:31.412 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:26.528834+0000", 2024-09-13T11:23:31.413 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.1288000000000003e-05 2024-09-13T11:23:31.413 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:31.413 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:31.413 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:23:31.413 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:26.529064+0000", 2024-09-13T11:23:31.413 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00022959 2024-09-13T11:23:31.413 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:31.413 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:31.413 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:23:31.413 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:26.529097+0000", 2024-09-13T11:23:31.413 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.3423999999999997e-05 2024-09-13T11:23:31.413 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:31.413 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:31.414 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:23:31.414 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:26.533275+0000", 2024-09-13T11:23:31.414 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0041773380000000001 2024-09-13T11:23:31.414 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:31.414 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:23:31.414 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:31.414 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:31.414 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:31.414 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "MOSDPGPush(3.19 72/70 [PushOp(3:9f24a12e:::benchmark_data_smithi177_31116_object6489:head, version: 46'406, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:9f24a12e:::benchmark_data_smithi177_31116_object6489:head@46'406, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:23:31.414 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:26.381103+0000", 2024-09-13T11:23:31.414 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 4.9544250410000004, 2024-09-13T11:23:31.414 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.15612552199999999, 2024-09-13T11:23:31.414 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:23:31.415 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:23:31.415 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-09-13T11:23:31.415 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:23:31.415 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:31.415 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:23:31.415 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:26.381103+0000", 2024-09-13T11:23:31.415 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:31.415 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:31.415 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:31.415 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:23:31.415 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:26.381103+0000", 2024-09-13T11:23:31.415 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:31.415 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:31.416 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:31.416 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:23:31.416 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:26.381105+0000", 2024-09-13T11:23:31.416 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.1769999999999999e-06 2024-09-13T11:23:31.416 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:31.416 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:31.416 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:23:31.416 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:26.381205+0000", 2024-09-13T11:23:31.416 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000100623 2024-09-13T11:23:31.416 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:31.416 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:31.416 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:23:31.417 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:26.535299+0000", 2024-09-13T11:23:31.417 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.15409379100000001 2024-09-13T11:23:31.417 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:31.417 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:31.417 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:23:31.417 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:26.535371+0000", 2024-09-13T11:23:31.417 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.1668000000000006e-05 2024-09-13T11:23:31.417 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:31.417 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:31.417 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:23:31.417 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:26.535612+0000", 2024-09-13T11:23:31.417 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00024121400000000001 2024-09-13T11:23:31.417 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:31.417 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:31.418 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:23:31.418 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:26.535639+0000", 2024-09-13T11:23:31.418 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.7324000000000001e-05 2024-09-13T11:23:31.418 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:31.418 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:31.418 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:23:31.418 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:26.537229+0000", 2024-09-13T11:23:31.418 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0015897249999999999 2024-09-13T11:23:31.418 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:31.418 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:23:31.418 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:31.418 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:31.418 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:31.418 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:50024 3.14 3:2e6ab6ee:::benchmark_data_smithi177_31116_object50023:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e72)", 2024-09-13T11:23:31.419 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:26.517317+0000", 2024-09-13T11:23:31.419 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 4.8182117460000002, 2024-09-13T11:23:31.419 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.119059073, 2024-09-13T11:23:31.419 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:23:31.419 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:23:31.419 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:31.419 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:23:31.419 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:23:31.419 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:31.419 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 50024 2024-09-13T11:23:31.419 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:31.419 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:23:31.419 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:31.420 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:23:31.420 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:26.517317+0000", 2024-09-13T11:23:31.420 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:31.420 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:31.420 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:31.420 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:23:31.420 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:26.517317+0000", 2024-09-13T11:23:31.420 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:31.420 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:31.420 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:31.420 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:23:31.420 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:26.517318+0000", 2024-09-13T11:23:31.420 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.1370000000000001e-06 2024-09-13T11:23:31.421 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:31.421 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:31.421 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:23:31.421 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:26.517488+0000", 2024-09-13T11:23:31.421 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00017060299999999999 2024-09-13T11:23:31.421 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:31.421 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:31.421 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:23:31.421 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:26.517508+0000", 2024-09-13T11:23:31.421 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.0021999999999999e-05 2024-09-13T11:23:31.421 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:31.421 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:31.421 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:23:31.422 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:26.517523+0000", 2024-09-13T11:23:31.422 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.4599e-05 2024-09-13T11:23:31.422 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:31.422 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:31.422 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:23:31.422 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:26.525069+0000", 2024-09-13T11:23:31.422 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0075455330000000001 2024-09-13T11:23:31.422 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:31.422 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:31.422 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:23:31.422 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:26.525372+0000", 2024-09-13T11:23:31.422 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00030381800000000002 2024-09-13T11:23:31.422 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:31.422 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:31.423 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 3,14", 2024-09-13T11:23:31.423 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:26.525677+0000", 2024-09-13T11:23:31.423 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00030414500000000002 2024-09-13T11:23:31.423 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:31.423 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:31.423 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:31.423 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:26.533341+0000", 2024-09-13T11:23:31.423 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0076645769999999997 2024-09-13T11:23:31.423 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:31.423 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:31.423 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:23:31.423 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:26.533389+0000", 2024-09-13T11:23:31.423 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.7351000000000001e-05 2024-09-13T11:23:31.423 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:31.424 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:31.424 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:31.424 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:26.634754+0000", 2024-09-13T11:23:31.424 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.101365305 2024-09-13T11:23:31.424 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:31.424 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:31.424 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:23:31.424 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:26.634846+0000", 2024-09-13T11:23:31.424 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.2625000000000002e-05 2024-09-13T11:23:31.424 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:31.424 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:31.424 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:23:31.424 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:26.636376+0000", 2024-09-13T11:23:31.424 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.001529358 2024-09-13T11:23:31.425 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:31.425 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:23:31.425 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:31.425 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:31.425 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:31.425 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:50070 3.14 3:2f61ab0c:::benchmark_data_smithi177_31116_object50069:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e72)", 2024-09-13T11:23:31.425 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:26.518044+0000", 2024-09-13T11:23:31.425 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 4.8174845030000002, 2024-09-13T11:23:31.425 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.119789226, 2024-09-13T11:23:31.425 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:23:31.426 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:23:31.426 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:31.426 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:23:31.426 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:23:31.426 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:31.426 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 50070 2024-09-13T11:23:31.426 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:31.426 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:23:31.426 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:31.426 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:23:31.426 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:26.518044+0000", 2024-09-13T11:23:31.426 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:31.426 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:31.426 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:31.427 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:23:31.427 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:26.518044+0000", 2024-09-13T11:23:31.427 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:31.427 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:31.427 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:31.427 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:23:31.427 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:26.518045+0000", 2024-09-13T11:23:31.427 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.144e-06 2024-09-13T11:23:31.427 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:31.427 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:31.427 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:23:31.427 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:26.518126+0000", 2024-09-13T11:23:31.427 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.0940999999999996e-05 2024-09-13T11:23:31.428 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:31.428 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:31.428 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:23:31.428 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:26.518140+0000", 2024-09-13T11:23:31.428 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.429e-05 2024-09-13T11:23:31.428 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:31.428 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:31.428 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:23:31.428 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:26.518148+0000", 2024-09-13T11:23:31.428 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.7659999999999998e-06 2024-09-13T11:23:31.428 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:31.428 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:31.428 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:23:31.428 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:26.527626+0000", 2024-09-13T11:23:31.429 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0094781640000000007 2024-09-13T11:23:31.429 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:31.429 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:31.429 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:23:31.429 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:26.527911+0000", 2024-09-13T11:23:31.429 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00028477299999999999 2024-09-13T11:23:31.429 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:31.429 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:31.429 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 3,14", 2024-09-13T11:23:31.429 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:26.528360+0000", 2024-09-13T11:23:31.429 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00044899300000000002 2024-09-13T11:23:31.429 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:31.429 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:31.430 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:31.430 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:26.533499+0000", 2024-09-13T11:23:31.430 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0051387849999999999 2024-09-13T11:23:31.430 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:31.430 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:31.430 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:23:31.430 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:26.537355+0000", 2024-09-13T11:23:31.430 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0038560320000000001 2024-09-13T11:23:31.430 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:31.430 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:31.430 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:31.430 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:26.636534+0000", 2024-09-13T11:23:31.430 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.099178793000000001 2024-09-13T11:23:31.430 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:31.431 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:31.431 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:23:31.431 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:26.636677+0000", 2024-09-13T11:23:31.431 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00014361699999999999 2024-09-13T11:23:31.431 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:31.431 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:31.431 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:23:31.431 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:26.637833+0000", 2024-09-13T11:23:31.431 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0011559280000000001 2024-09-13T11:23:31.431 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:31.431 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:23:31.431 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:31.431 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:31.431 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:31.432 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:50085 3.14 3:2bad6115:::benchmark_data_smithi177_31116_object50084:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e72)", 2024-09-13T11:23:31.432 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:26.518372+0000", 2024-09-13T11:23:31.432 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 4.8171563490000002, 2024-09-13T11:23:31.432 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.120828914, 2024-09-13T11:23:31.432 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:23:31.432 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:23:31.432 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:31.432 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:23:31.432 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:23:31.432 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:31.432 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 50085 2024-09-13T11:23:31.432 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:31.432 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:23:31.432 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:31.433 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:23:31.433 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:26.518372+0000", 2024-09-13T11:23:31.433 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:31.433 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:31.433 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:31.433 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:23:31.433 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:26.518372+0000", 2024-09-13T11:23:31.433 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:31.433 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:31.433 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:31.433 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:23:31.433 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:26.518373+0000", 2024-09-13T11:23:31.433 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.0699999999999996e-07 2024-09-13T11:23:31.434 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:31.434 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:31.434 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:23:31.434 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:26.518413+0000", 2024-09-13T11:23:31.434 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.9703999999999998e-05 2024-09-13T11:23:31.434 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:31.434 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:31.434 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:23:31.434 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:26.518421+0000", 2024-09-13T11:23:31.434 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.3270000000000001e-06 2024-09-13T11:23:31.434 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:31.434 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:31.434 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:23:31.434 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:26.518428+0000", 2024-09-13T11:23:31.435 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.1749999999999999e-06 2024-09-13T11:23:31.435 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:31.435 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:31.435 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:23:31.435 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:26.533583+0000", 2024-09-13T11:23:31.435 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.015154707 2024-09-13T11:23:31.435 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:31.435 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:31.435 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:23:31.435 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:26.533912+0000", 2024-09-13T11:23:31.435 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00032901099999999998 2024-09-13T11:23:31.435 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:31.435 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:31.436 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 3,14", 2024-09-13T11:23:31.436 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:26.534289+0000", 2024-09-13T11:23:31.436 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000377272 2024-09-13T11:23:31.436 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:31.436 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:31.436 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:31.436 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:26.540624+0000", 2024-09-13T11:23:31.436 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0063345950000000002 2024-09-13T11:23:31.436 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:31.436 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:31.436 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:23:31.436 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:26.540677+0000", 2024-09-13T11:23:31.436 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.3622999999999998e-05 2024-09-13T11:23:31.436 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:31.437 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:31.437 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:31.437 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:26.638002+0000", 2024-09-13T11:23:31.437 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.097324778000000001 2024-09-13T11:23:31.437 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:31.437 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:31.437 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:23:31.437 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:26.638165+0000", 2024-09-13T11:23:31.437 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00016326700000000001 2024-09-13T11:23:31.437 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:31.437 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:31.437 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:23:31.437 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:26.639201+0000", 2024-09-13T11:23:31.438 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0010355480000000001 2024-09-13T11:23:31.438 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:31.438 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:23:31.438 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:31.438 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:31.438 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:31.438 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:50090 3.14 3:2cc23e43:::benchmark_data_smithi177_31116_object50089:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e72)", 2024-09-13T11:23:31.438 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:26.528414+0000", 2024-09-13T11:23:31.438 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 4.8071145780000002, 2024-09-13T11:23:31.438 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.11229205, 2024-09-13T11:23:31.438 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:23:31.438 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:23:31.438 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:31.439 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:23:31.439 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:23:31.439 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:31.439 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 50090 2024-09-13T11:23:31.439 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:31.439 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:23:31.439 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:31.439 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:23:31.439 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:26.528414+0000", 2024-09-13T11:23:31.439 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:31.439 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:31.439 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:31.439 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:23:31.439 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:26.528414+0000", 2024-09-13T11:23:31.440 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:31.440 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:31.440 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:31.440 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:23:31.440 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:26.528418+0000", 2024-09-13T11:23:31.440 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.9389999999999999e-06 2024-09-13T11:23:31.440 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:31.440 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:31.440 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:23:31.440 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:26.528483+0000", 2024-09-13T11:23:31.440 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.4750000000000002e-05 2024-09-13T11:23:31.440 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:31.440 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:31.441 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:23:31.441 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:26.528515+0000", 2024-09-13T11:23:31.441 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.2194999999999999e-05 2024-09-13T11:23:31.441 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:31.441 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:31.441 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:23:31.441 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:26.528538+0000", 2024-09-13T11:23:31.441 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.2945000000000001e-05 2024-09-13T11:23:31.441 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:31.441 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:31.441 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:23:31.441 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:26.538451+0000", 2024-09-13T11:23:31.441 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0099133320000000004 2024-09-13T11:23:31.441 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:31.442 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:31.442 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:23:31.442 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:26.538740+0000", 2024-09-13T11:23:31.442 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00028853100000000002 2024-09-13T11:23:31.442 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:31.442 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:31.442 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 3,14", 2024-09-13T11:23:31.442 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:26.539027+0000", 2024-09-13T11:23:31.442 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00028764300000000002 2024-09-13T11:23:31.442 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:31.442 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:31.442 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:23:31.442 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:26.541083+0000", 2024-09-13T11:23:31.442 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.002056258 2024-09-13T11:23:31.443 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:31.443 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:31.443 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:31.443 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:26.543346+0000", 2024-09-13T11:23:31.443 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0022630039999999999 2024-09-13T11:23:31.443 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:31.443 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:31.443 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:31.443 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:26.639337+0000", 2024-09-13T11:23:31.443 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.095990991999999997 2024-09-13T11:23:31.443 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:31.443 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:31.443 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:23:31.444 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:26.639479+0000", 2024-09-13T11:23:31.444 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000141967 2024-09-13T11:23:31.444 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:31.444 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:31.444 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:23:31.444 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:26.640706+0000", 2024-09-13T11:23:31.444 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0012264940000000001 2024-09-13T11:23:31.444 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:31.444 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:23:31.444 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:31.444 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:31.444 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:23:31.444 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-09-13T11:23:31.446 INFO:tasks.ceph.osd.9.smithi092.stderr:2024-09-13T11:23:31.444+0000 7ff5ab3d0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26348) UID: 0 2024-09-13T11:23:31.446 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-13T11:23:31.547 INFO:tasks.ceph.osd.3.smithi177.stderr:2024-09-13T11:23:31.541+0000 7f10adba4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26574) UID: 0 2024-09-13T11:23:31.598 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-13T11:23:31.598 INFO:teuthology.orchestra.run.smithi001.stdout: "ops": [], 2024-09-13T11:23:31.598 INFO:teuthology.orchestra.run.smithi001.stdout: "num_ops": 0 2024-09-13T11:23:31.598 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-13T11:23:31.619 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-13T11:23:31.648 INFO:tasks.ceph.osd.2.smithi162.stderr:2024-09-13T11:23:31.641+0000 7fe92b35a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26313) UID: 0 2024-09-13T11:23:31.749 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-09-13T11:23:31.741+0000 7f97befc9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26315) UID: 0 2024-09-13T11:23:31.759 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-13T11:23:31.760 INFO:teuthology.orchestra.run.smithi001.stdout: "ops": [], 2024-09-13T11:23:31.760 INFO:teuthology.orchestra.run.smithi001.stdout: "complaint_time": 30, 2024-09-13T11:23:31.760 INFO:teuthology.orchestra.run.smithi001.stdout: "num_blocked_ops": 0 2024-09-13T11:23:31.760 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-13T11:23:31.778 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-13T11:23:31.850 INFO:tasks.ceph.osd.4.smithi001.stderr:2024-09-13T11:23:31.844+0000 7ff8e0218640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26310) UID: 0 2024-09-13T11:23:31.903 INFO:teuthology.orchestra.run.smithi001.stderr:noscrub is unset 2024-09-13T11:23:31.909 INFO:tasks.ceph.osd.12.smithi001.stderr:2024-09-13T11:23:31.900+0000 7f891f10f640 -1 osd.12 75 handle_osd_map injecting map crc failure 2024-09-13T11:23:31.928 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-13T11:23:31.928 INFO:teuthology.orchestra.run.smithi001.stdout: "size": 20, 2024-09-13T11:23:31.928 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 600, 2024-09-13T11:23:31.928 INFO:teuthology.orchestra.run.smithi001.stdout: "ops": [ 2024-09-13T11:23:31.928 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:31.928 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4665.0:30360 3.e e59/57 3:7373925b:::benchmark_data_smithi177_31116_object30359:head v 59'1966, mlcod=43'163)", 2024-09-13T11:23:31.928 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:09.174964+0000", 2024-09-13T11:23:31.928 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 22.749052971000001, 2024-09-13T11:23:31.928 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.063585121999999994, 2024-09-13T11:23:31.928 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:23:31.929 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:23:31.929 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:31.929 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:23:31.929 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:31.929 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:23:31.929 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.174964+0000", 2024-09-13T11:23:31.929 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:31.929 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:31.929 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:31.929 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:23:31.929 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.174964+0000", 2024-09-13T11:23:31.929 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:31.929 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:31.930 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:31.930 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:23:31.930 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.174966+0000", 2024-09-13T11:23:31.930 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.198e-06 2024-09-13T11:23:31.931 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:31.932 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:31.932 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:23:31.932 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.175027+0000", 2024-09-13T11:23:31.932 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.1058999999999999e-05 2024-09-13T11:23:31.932 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:31.932 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:31.932 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:23:31.932 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.235049+0000", 2024-09-13T11:23:31.932 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.060022631999999999 2024-09-13T11:23:31.932 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:31.932 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:31.932 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:23:31.932 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.235059+0000", 2024-09-13T11:23:31.933 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.6490000000000003e-06 2024-09-13T11:23:31.933 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:31.933 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:31.933 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:31.933 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.235127+0000", 2024-09-13T11:23:31.933 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.7543e-05 2024-09-13T11:23:31.933 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:31.933 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:31.933 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:23:31.933 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.235157+0000", 2024-09-13T11:23:31.933 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.0525000000000003e-05 2024-09-13T11:23:31.933 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:31.933 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:31.933 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:23:31.934 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.238499+0000", 2024-09-13T11:23:31.934 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0033422679999999998 2024-09-13T11:23:31.934 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:31.934 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:31.934 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:23:31.934 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.238550+0000", 2024-09-13T11:23:31.934 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.0247999999999997e-05 2024-09-13T11:23:31.934 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:31.934 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:31.934 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:31.934 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:31.934 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:31.934 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDPGPush(3.e 63/57 [PushOp(3:76ee7eee:::benchmark_data_smithi177_31116_object10827:head, version: 48'701, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:76ee7eee:::benchmark_data_smithi177_31116_object10827:head@48'701, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:23:31.935 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:15.264265+0000", 2024-09-13T11:23:31.935 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 16.659752921999999, 2024-09-13T11:23:31.935 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.053083765999999998, 2024-09-13T11:23:31.935 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:23:31.935 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:23:31.935 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:23:31.935 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:23:31.935 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:31.935 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:23:31.935 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:15.264265+0000", 2024-09-13T11:23:31.935 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:31.935 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:31.935 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:31.935 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:23:31.936 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:15.264265+0000", 2024-09-13T11:23:31.936 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:31.936 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:31.936 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:31.936 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:23:31.936 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:15.264267+0000", 2024-09-13T11:23:31.936 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.9289999999999998e-06 2024-09-13T11:23:31.936 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:31.936 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:31.936 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:23:31.936 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:15.264316+0000", 2024-09-13T11:23:31.936 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.8232000000000002e-05 2024-09-13T11:23:31.936 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:31.937 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:31.937 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:23:31.937 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:15.316008+0000", 2024-09-13T11:23:31.937 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.051692121000000001 2024-09-13T11:23:31.937 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:31.937 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:31.937 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:23:31.937 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:15.316021+0000", 2024-09-13T11:23:31.937 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.287e-05 2024-09-13T11:23:31.937 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:31.937 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:31.937 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:31.937 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:15.316140+0000", 2024-09-13T11:23:31.937 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00011913099999999999 2024-09-13T11:23:31.938 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:31.938 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:31.938 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:23:31.938 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:15.316190+0000", 2024-09-13T11:23:31.938 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.9719999999999998e-05 2024-09-13T11:23:31.938 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:31.938 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:31.938 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:23:31.938 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:15.317348+0000", 2024-09-13T11:23:31.938 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0011587629999999999 2024-09-13T11:23:31.938 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:31.938 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:31.938 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:31.938 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:31.939 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:31.939 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDPGPush(3.e 63/57 [PushOp(3:7ea7b183:::benchmark_data_smithi177_31116_object10829:head, version: 48'702, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:7ea7b183:::benchmark_data_smithi177_31116_object10829:head@48'702, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:23:31.939 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:15.324717+0000", 2024-09-13T11:23:31.939 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 16.599299963, 2024-09-13T11:23:31.939 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.038697948000000003, 2024-09-13T11:23:31.939 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:23:31.939 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:23:31.939 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:23:31.939 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:23:31.939 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:31.939 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:23:31.939 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:15.324717+0000", 2024-09-13T11:23:31.939 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:31.939 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:31.940 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:31.940 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:23:31.940 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:15.324717+0000", 2024-09-13T11:23:31.940 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:31.940 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:31.940 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:31.940 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:23:31.940 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:15.324719+0000", 2024-09-13T11:23:31.940 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.801e-06 2024-09-13T11:23:31.940 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:31.940 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:31.940 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:23:31.940 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:15.324754+0000", 2024-09-13T11:23:31.941 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.4418000000000001e-05 2024-09-13T11:23:31.941 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:31.941 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:31.941 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:23:31.941 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:15.362161+0000", 2024-09-13T11:23:31.941 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.037407045 2024-09-13T11:23:31.941 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:31.941 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:31.941 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:23:31.941 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:15.362180+0000", 2024-09-13T11:23:31.941 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.9709000000000001e-05 2024-09-13T11:23:31.941 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:31.941 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:31.941 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:31.941 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:15.362283+0000", 2024-09-13T11:23:31.942 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00010224 2024-09-13T11:23:31.942 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:31.942 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:31.942 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:23:31.942 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:15.362296+0000", 2024-09-13T11:23:31.942 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.3759e-05 2024-09-13T11:23:31.942 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:31.942 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:31.942 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:23:31.942 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:15.363415+0000", 2024-09-13T11:23:31.942 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.001118976 2024-09-13T11:23:31.942 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:31.942 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:31.943 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:31.943 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:31.943 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:31.943 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4665.0:39188 3.e e63/57 3:70f16f91:::benchmark_data_smithi177_31116_object39187:head v 63'2520, mlcod=48'701)", 2024-09-13T11:23:31.943 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:15.328821+0000", 2024-09-13T11:23:31.943 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 16.595196217000002, 2024-09-13T11:23:31.943 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.040352576000000001, 2024-09-13T11:23:31.943 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:23:31.943 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:23:31.943 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:31.943 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:23:31.943 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:31.943 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:23:31.943 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:15.328821+0000", 2024-09-13T11:23:31.944 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:31.944 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:31.944 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:31.944 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:23:31.944 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:15.328821+0000", 2024-09-13T11:23:31.944 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:31.944 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:31.944 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:31.944 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:23:31.944 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:15.328823+0000", 2024-09-13T11:23:31.944 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.48e-06 2024-09-13T11:23:31.944 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:31.944 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:31.945 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:23:31.945 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:15.328847+0000", 2024-09-13T11:23:31.945 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.3791e-05 2024-09-13T11:23:31.945 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:31.945 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:31.945 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:23:31.945 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:15.367474+0000", 2024-09-13T11:23:31.945 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.038627335999999998 2024-09-13T11:23:31.945 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:31.945 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:31.945 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:23:31.945 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:15.367480+0000", 2024-09-13T11:23:31.945 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.3520000000000003e-06 2024-09-13T11:23:31.945 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:31.946 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:31.946 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:31.946 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:15.367536+0000", 2024-09-13T11:23:31.946 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.5872000000000003e-05 2024-09-13T11:23:31.946 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:31.946 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:31.946 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:23:31.946 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:15.367563+0000", 2024-09-13T11:23:31.946 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.6755e-05 2024-09-13T11:23:31.946 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:31.946 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:31.946 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:23:31.946 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:15.369141+0000", 2024-09-13T11:23:31.947 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0015783730000000001 2024-09-13T11:23:31.947 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:31.947 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:31.947 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:23:31.947 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:15.369174+0000", 2024-09-13T11:23:31.947 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.2617000000000003e-05 2024-09-13T11:23:31.947 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:31.947 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:31.947 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:31.947 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:31.947 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:31.947 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDPGPush(3.e 64/57 [PushOp(3:7b9273cf:::benchmark_data_smithi177_31116_object11230:head, version: 48'726, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:7b9273cf:::benchmark_data_smithi177_31116_object11230:head@48'726, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:23:31.947 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:15.778774+0000", 2024-09-13T11:23:31.947 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 16.145243425, 2024-09-13T11:23:31.948 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.041611321, 2024-09-13T11:23:31.948 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:23:31.948 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:23:31.948 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:23:31.948 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:23:31.948 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:31.948 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:23:31.948 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:15.778774+0000", 2024-09-13T11:23:31.948 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:31.948 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:31.948 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:31.948 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:23:31.948 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:15.778774+0000", 2024-09-13T11:23:31.949 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:31.949 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:31.949 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:31.949 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:23:31.949 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:15.778775+0000", 2024-09-13T11:23:31.949 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.8599999999999996e-07 2024-09-13T11:23:31.949 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:31.949 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:31.949 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:23:31.949 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:15.778798+0000", 2024-09-13T11:23:31.949 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.2796999999999999e-05 2024-09-13T11:23:31.949 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:31.949 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:31.950 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:23:31.950 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:15.819362+0000", 2024-09-13T11:23:31.950 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.040564464000000001 2024-09-13T11:23:31.950 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:31.950 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:31.950 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:23:31.950 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:15.819376+0000", 2024-09-13T11:23:31.950 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.3628999999999999e-05 2024-09-13T11:23:31.950 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:31.950 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:31.950 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:31.950 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:15.819476+0000", 2024-09-13T11:23:31.950 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00010021100000000001 2024-09-13T11:23:31.950 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:31.951 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:31.951 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:23:31.951 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:15.819490+0000", 2024-09-13T11:23:31.951 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.3399e-05 2024-09-13T11:23:31.951 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:31.952 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:31.952 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:23:31.952 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:15.820385+0000", 2024-09-13T11:23:31.952 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00089583499999999999 2024-09-13T11:23:31.952 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:31.952 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:31.952 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:31.952 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:31.952 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:31.952 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDPGPush(3.e 64/57 [PushOp(3:71c4c7e7:::benchmark_data_smithi177_31116_object11233:head, version: 48'728, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:71c4c7e7:::benchmark_data_smithi177_31116_object11233:head@48'728, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:23:31.952 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:15.855532+0000", 2024-09-13T11:23:31.953 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 16.068485839000001, 2024-09-13T11:23:31.953 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.055507661, 2024-09-13T11:23:31.953 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:23:31.953 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:23:31.953 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:23:31.953 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:23:31.953 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:31.953 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:23:31.953 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:15.855532+0000", 2024-09-13T11:23:31.953 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:31.953 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:31.953 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:31.953 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:23:31.954 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:15.855532+0000", 2024-09-13T11:23:31.954 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:31.954 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:31.954 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:31.954 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:23:31.954 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:15.855533+0000", 2024-09-13T11:23:31.954 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.584e-06 2024-09-13T11:23:31.954 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:31.954 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:31.954 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:23:31.954 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:15.855567+0000", 2024-09-13T11:23:31.954 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.4220000000000001e-05 2024-09-13T11:23:31.954 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:31.954 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:31.955 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:23:31.955 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:15.908665+0000", 2024-09-13T11:23:31.955 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.053097345999999997 2024-09-13T11:23:31.955 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:31.955 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:31.955 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:23:31.955 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:15.908708+0000", 2024-09-13T11:23:31.955 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.3204000000000002e-05 2024-09-13T11:23:31.955 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:31.955 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:31.955 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:31.956 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:15.909011+0000", 2024-09-13T11:23:31.956 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000302928 2024-09-13T11:23:31.956 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:31.956 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:31.956 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:23:31.957 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:15.909054+0000", 2024-09-13T11:23:31.957 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.2781999999999998e-05 2024-09-13T11:23:31.957 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:31.957 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:31.957 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:23:31.957 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:15.911039+0000", 2024-09-13T11:23:31.957 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.001985597 2024-09-13T11:23:31.957 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:31.957 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:31.958 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:31.958 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:31.958 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:31.958 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDPGPush(3.e 68/57 [PushOp(3:74d482b2:::benchmark_data_smithi177_31116_object15976:head, version: 52'1069, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:74d482b2:::benchmark_data_smithi177_31116_object15976:head@52'1069, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:23:31.958 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:20.419244+0000", 2024-09-13T11:23:31.958 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 11.504773194, 2024-09-13T11:23:31.958 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.035798097000000001, 2024-09-13T11:23:31.958 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:23:31.958 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:23:31.958 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:23:31.958 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:23:31.958 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:31.958 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:23:31.958 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:20.419244+0000", 2024-09-13T11:23:31.959 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:31.959 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:31.959 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:31.959 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:23:31.959 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:20.419244+0000", 2024-09-13T11:23:31.959 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:31.959 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:31.959 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:31.959 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:23:31.959 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:20.419246+0000", 2024-09-13T11:23:31.959 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.6249999999999999e-06 2024-09-13T11:23:31.959 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:31.959 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:31.960 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:23:31.960 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:20.419274+0000", 2024-09-13T11:23:31.960 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.8441000000000001e-05 2024-09-13T11:23:31.960 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:31.960 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:31.960 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:23:31.960 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:20.453463+0000", 2024-09-13T11:23:31.960 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.034189169999999998 2024-09-13T11:23:31.960 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:31.960 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:31.960 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:23:31.960 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:20.453492+0000", 2024-09-13T11:23:31.960 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.8031999999999999e-05 2024-09-13T11:23:31.961 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:31.961 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:31.961 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:31.961 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:20.453589+0000", 2024-09-13T11:23:31.961 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.7484000000000004e-05 2024-09-13T11:23:31.961 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:31.961 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:31.961 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:23:31.961 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:20.453598+0000", 2024-09-13T11:23:31.961 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.2269999999999999e-06 2024-09-13T11:23:31.961 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:31.961 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:31.961 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:23:31.962 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:20.455042+0000", 2024-09-13T11:23:31.962 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0014441180000000001 2024-09-13T11:23:31.962 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:31.962 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:31.962 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:31.962 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:31.962 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:31.962 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4665.0:45323 3.e e68/57 3:7718dee6:::benchmark_data_smithi177_31116_object45322:head v 68'2914, mlcod=52'1073)", 2024-09-13T11:23:31.962 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:20.548858+0000", 2024-09-13T11:23:31.962 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 11.375159693000001, 2024-09-13T11:23:31.962 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.053200113, 2024-09-13T11:23:31.962 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:23:31.962 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:23:31.962 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:31.963 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:23:31.963 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:31.963 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:23:31.963 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:20.548858+0000", 2024-09-13T11:23:31.963 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:31.963 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:31.963 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:31.963 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:23:31.963 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:20.548858+0000", 2024-09-13T11:23:31.963 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:31.963 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:31.963 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:31.963 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:23:31.963 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:20.548860+0000", 2024-09-13T11:23:31.964 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.7439999999999999e-06 2024-09-13T11:23:31.964 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:31.964 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:31.964 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:23:31.964 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:20.548889+0000", 2024-09-13T11:23:31.964 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.9031e-05 2024-09-13T11:23:31.964 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:31.964 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:31.964 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:23:31.964 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:20.598745+0000", 2024-09-13T11:23:31.964 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.049856625000000002 2024-09-13T11:23:31.964 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:31.964 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:31.965 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:23:31.965 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:20.598755+0000", 2024-09-13T11:23:31.965 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.4760000000000004e-06 2024-09-13T11:23:31.965 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:31.965 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:31.965 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:31.965 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:20.598889+0000", 2024-09-13T11:23:31.965 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00013463700000000001 2024-09-13T11:23:31.965 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:31.965 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:31.965 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:23:31.965 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:20.598934+0000", 2024-09-13T11:23:31.965 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.5114000000000002e-05 2024-09-13T11:23:31.965 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:31.966 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:31.966 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:23:31.966 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:20.602012+0000", 2024-09-13T11:23:31.966 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0030772360000000001 2024-09-13T11:23:31.966 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:31.966 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:31.966 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:23:31.966 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:20.602058+0000", 2024-09-13T11:23:31.966 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.6249999999999999e-05 2024-09-13T11:23:31.966 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:31.966 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:31.966 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:31.966 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:31.966 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:31.967 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4665.0:45376 3.e e68/57 3:7b68282c:::benchmark_data_smithi177_31116_object45375:head v 68'2916, mlcod=52'1073)", 2024-09-13T11:23:31.967 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:20.610260+0000", 2024-09-13T11:23:31.967 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 11.313757674, 2024-09-13T11:23:31.967 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.040113640999999998, 2024-09-13T11:23:31.967 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:23:31.967 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:23:31.967 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:31.967 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:23:31.967 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:31.967 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:23:31.967 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:20.610260+0000", 2024-09-13T11:23:31.967 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:31.967 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:31.967 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:31.968 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:23:31.968 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:20.610260+0000", 2024-09-13T11:23:31.968 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:31.968 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:31.968 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:31.968 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:23:31.968 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:20.610261+0000", 2024-09-13T11:23:31.968 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.542e-06 2024-09-13T11:23:31.968 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:31.968 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:31.968 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:23:31.968 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:20.610293+0000", 2024-09-13T11:23:31.968 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.1399000000000002e-05 2024-09-13T11:23:31.968 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:31.969 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:31.969 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:23:31.969 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:20.647385+0000", 2024-09-13T11:23:31.969 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.037091956000000002 2024-09-13T11:23:31.969 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:31.969 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:31.969 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:23:31.969 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:20.647399+0000", 2024-09-13T11:23:31.969 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.4673e-05 2024-09-13T11:23:31.969 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:31.969 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:31.969 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:31.969 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:20.647497+0000", 2024-09-13T11:23:31.970 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.7424999999999996e-05 2024-09-13T11:23:31.970 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:31.970 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:31.970 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:23:31.970 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:20.647519+0000", 2024-09-13T11:23:31.970 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.2184e-05 2024-09-13T11:23:31.970 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:31.970 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:31.970 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:23:31.970 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:20.650320+0000", 2024-09-13T11:23:31.970 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.002801521 2024-09-13T11:23:31.970 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:31.970 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:31.971 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:23:31.971 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:20.650373+0000", 2024-09-13T11:23:31.971 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.2941e-05 2024-09-13T11:23:31.971 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:31.971 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:31.971 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:31.971 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:31.971 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:31.971 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4665.0:45746 3.e e68/57 3:720fe071:::benchmark_data_smithi177_31116_object45745:head v 68'2942, mlcod=52'1085)", 2024-09-13T11:23:31.971 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:20.907344+0000", 2024-09-13T11:23:31.971 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 11.016672984, 2024-09-13T11:23:31.971 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.038718234999999997, 2024-09-13T11:23:31.971 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:23:31.972 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:23:31.972 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:31.972 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:23:31.972 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:31.972 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:23:31.972 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:20.907344+0000", 2024-09-13T11:23:31.972 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:31.972 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:31.972 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:31.972 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:23:31.972 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:20.907344+0000", 2024-09-13T11:23:31.972 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:31.972 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:31.973 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:31.973 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:23:31.973 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:20.907346+0000", 2024-09-13T11:23:31.973 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.9369999999999998e-06 2024-09-13T11:23:31.973 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:31.973 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:31.973 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:23:31.973 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:20.907381+0000", 2024-09-13T11:23:31.973 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.4508999999999998e-05 2024-09-13T11:23:31.973 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:31.973 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:31.973 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:23:31.973 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:20.942849+0000", 2024-09-13T11:23:31.974 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.035468116000000001 2024-09-13T11:23:31.974 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:31.974 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:31.974 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:23:31.974 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:20.942856+0000", 2024-09-13T11:23:31.974 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.9450000000000004e-06 2024-09-13T11:23:31.974 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:31.974 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:31.974 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:31.974 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:20.942937+0000", 2024-09-13T11:23:31.974 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.0988000000000005e-05 2024-09-13T11:23:31.974 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:31.974 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:31.974 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:23:31.974 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:20.942958+0000", 2024-09-13T11:23:31.975 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.1382e-05 2024-09-13T11:23:31.975 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:31.975 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:31.975 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:23:31.975 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:20.946001+0000", 2024-09-13T11:23:31.975 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0030430269999999998 2024-09-13T11:23:31.975 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:31.975 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:31.975 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:23:31.975 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:20.946063+0000", 2024-09-13T11:23:31.975 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.1330999999999999e-05 2024-09-13T11:23:31.975 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:31.975 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:31.976 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:31.976 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:31.976 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:31.976 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDPGPush(3.e 69/57 [PushOp(3:7970c88c:::benchmark_data_smithi177_31116_object16245:head, version: 52'1088, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:7970c88c:::benchmark_data_smithi177_31116_object16245:head@52'1088, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:23:31.976 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:20.945743+0000", 2024-09-13T11:23:31.976 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 10.978274785, 2024-09-13T11:23:31.976 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.044762211000000003, 2024-09-13T11:23:31.976 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:23:31.976 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:23:31.976 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:23:31.976 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:23:31.976 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:31.976 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:23:31.977 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:20.945743+0000", 2024-09-13T11:23:31.977 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:31.977 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:31.977 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:31.977 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:23:31.977 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:20.945743+0000", 2024-09-13T11:23:31.977 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:31.977 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:31.977 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:31.977 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:23:31.977 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:20.945744+0000", 2024-09-13T11:23:31.977 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.6700000000000001e-06 2024-09-13T11:23:31.977 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:31.978 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:31.978 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:23:31.978 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:20.945796+0000", 2024-09-13T11:23:31.978 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.1826999999999997e-05 2024-09-13T11:23:31.978 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:31.978 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:31.978 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:23:31.978 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:20.989370+0000", 2024-09-13T11:23:31.978 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.043574195000000003 2024-09-13T11:23:31.978 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:31.978 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:31.978 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:23:31.978 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:20.989384+0000", 2024-09-13T11:23:31.978 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.3655000000000001e-05 2024-09-13T11:23:31.979 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:31.979 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:31.979 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:31.979 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:20.989496+0000", 2024-09-13T11:23:31.979 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00011205300000000001 2024-09-13T11:23:31.979 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:31.979 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:31.979 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:23:31.979 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:20.989506+0000", 2024-09-13T11:23:31.979 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0132999999999999e-05 2024-09-13T11:23:31.979 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:31.979 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:31.979 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:23:31.979 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:20.990505+0000", 2024-09-13T11:23:31.980 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00099867800000000002 2024-09-13T11:23:31.980 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:31.980 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:31.980 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:31.980 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:31.980 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:31.980 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDPGPush(3.15 71/70 [PushOp(3:a8e7ff00:::benchmark_data_smithi177_31116_object3532:head, version: 44'215, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:a8e7ff00:::benchmark_data_smithi177_31116_object3532:head@44'215, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:23:31.980 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:25.425119+0000", 2024-09-13T11:23:31.980 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 6.4988979640000002, 2024-09-13T11:23:31.980 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.095533526999999993, 2024-09-13T11:23:31.980 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:23:31.980 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:23:31.980 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:23:31.981 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:23:31.981 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:31.981 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:23:31.981 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:25.425119+0000", 2024-09-13T11:23:31.981 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:31.981 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:31.981 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:31.981 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:23:31.981 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:25.425119+0000", 2024-09-13T11:23:31.981 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:31.981 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:31.981 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:31.981 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:23:31.982 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:25.425121+0000", 2024-09-13T11:23:31.982 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.61e-06 2024-09-13T11:23:31.982 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:31.982 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:31.982 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:23:31.982 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:25.425266+0000", 2024-09-13T11:23:31.982 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000145184 2024-09-13T11:23:31.982 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:31.982 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:31.982 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:23:31.982 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:25.520033+0000", 2024-09-13T11:23:31.982 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.094766403999999999 2024-09-13T11:23:31.982 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:31.982 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:31.983 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:23:31.983 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:25.520066+0000", 2024-09-13T11:23:31.983 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.3169e-05 2024-09-13T11:23:31.983 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:31.983 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:31.983 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:31.983 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:25.520149+0000", 2024-09-13T11:23:31.983 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.3571000000000003e-05 2024-09-13T11:23:31.983 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:31.983 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:31.983 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:23:31.983 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:25.520157+0000", 2024-09-13T11:23:31.983 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.9650000000000008e-06 2024-09-13T11:23:31.984 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:31.984 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:31.984 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:23:31.984 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:25.520653+0000", 2024-09-13T11:23:31.984 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00049562399999999996 2024-09-13T11:23:31.984 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:31.984 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:31.984 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:31.984 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:31.984 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:31.984 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDPGPush(3.15 71/70 [PushOp(3:ac810159:::benchmark_data_smithi177_31116_object3553:head, version: 44'216, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:ac810159:::benchmark_data_smithi177_31116_object3553:head@44'216, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:23:31.984 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:25.529030+0000", 2024-09-13T11:23:31.984 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 6.394987532, 2024-09-13T11:23:31.984 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.035903248999999998, 2024-09-13T11:23:31.985 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:23:31.985 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:23:31.985 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:23:31.985 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:23:31.985 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:31.985 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:23:31.985 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:25.529030+0000", 2024-09-13T11:23:31.985 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:31.985 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:31.985 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:31.985 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:23:31.985 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:25.529030+0000", 2024-09-13T11:23:31.985 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:31.986 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:31.986 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:31.986 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:23:31.986 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:25.529031+0000", 2024-09-13T11:23:31.986 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.423e-06 2024-09-13T11:23:31.986 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:31.986 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:31.986 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:23:31.986 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:25.529155+0000", 2024-09-13T11:23:31.986 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00012360299999999999 2024-09-13T11:23:31.986 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:31.986 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:31.986 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:23:31.987 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:25.564164+0000", 2024-09-13T11:23:31.987 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.035008709999999998 2024-09-13T11:23:31.987 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:31.987 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:31.987 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:23:31.987 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:25.564180+0000", 2024-09-13T11:23:31.987 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.6059e-05 2024-09-13T11:23:31.987 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:31.987 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:31.987 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:31.987 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:25.564287+0000", 2024-09-13T11:23:31.987 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00010747900000000001 2024-09-13T11:23:31.987 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:31.988 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:31.988 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:23:31.988 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:25.564302+0000", 2024-09-13T11:23:31.988 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.4408e-05 2024-09-13T11:23:31.988 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:31.988 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:31.988 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:23:31.988 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:25.564933+0000", 2024-09-13T11:23:31.988 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00063156699999999998 2024-09-13T11:23:31.988 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:31.988 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:31.988 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:31.988 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:31.989 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:31.989 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4665.0:49041 3.e e71/57 3:7aa51d7d:::benchmark_data_smithi177_31116_object49040:head v 71'3150, mlcod=55'1456)", 2024-09-13T11:23:31.989 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:25.583357+0000", 2024-09-13T11:23:31.989 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 6.3406599620000001, 2024-09-13T11:23:31.989 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.037496603000000003, 2024-09-13T11:23:31.989 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:23:31.989 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:23:31.989 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:31.989 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:23:31.989 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:31.989 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:23:31.989 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:25.583357+0000", 2024-09-13T11:23:31.989 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:31.989 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:31.990 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:31.990 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:23:31.990 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:25.583357+0000", 2024-09-13T11:23:31.990 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:31.990 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:31.990 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:31.990 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:23:31.990 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:25.583359+0000", 2024-09-13T11:23:31.990 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.5409999999999999e-06 2024-09-13T11:23:31.990 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:31.990 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:31.990 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:23:31.990 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:25.583383+0000", 2024-09-13T11:23:31.990 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.4096000000000001e-05 2024-09-13T11:23:31.991 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:31.991 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:31.991 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:23:31.991 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:25.617621+0000", 2024-09-13T11:23:31.991 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.034237541000000003 2024-09-13T11:23:31.991 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:31.991 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:31.991 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:23:31.991 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:25.617628+0000", 2024-09-13T11:23:31.991 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.1040000000000001e-06 2024-09-13T11:23:31.991 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:31.991 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:31.991 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:31.992 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:25.617784+0000", 2024-09-13T11:23:31.992 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00015600999999999999 2024-09-13T11:23:31.992 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:31.992 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:31.992 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:23:31.992 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:25.617830+0000", 2024-09-13T11:23:31.992 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.6014999999999999e-05 2024-09-13T11:23:31.992 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:31.992 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:31.992 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:23:31.992 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:25.620811+0000", 2024-09-13T11:23:31.992 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0029816970000000002 2024-09-13T11:23:31.992 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:31.992 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:31.993 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:23:31.993 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:25.620854+0000", 2024-09-13T11:23:31.993 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.2599000000000003e-05 2024-09-13T11:23:31.993 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:31.993 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:31.993 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:31.993 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:31.993 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:31.993 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDPGPush(3.15 71/70 [PushOp(3:ab1bae1d:::benchmark_data_smithi177_31116_object3599:head, version: 44'222, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:ab1bae1d:::benchmark_data_smithi177_31116_object3599:head@44'222, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:23:31.993 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:25.648395+0000", 2024-09-13T11:23:31.993 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 6.2756228329999999, 2024-09-13T11:23:31.993 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.035696770000000003, 2024-09-13T11:23:31.993 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:23:31.993 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:23:31.994 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:23:31.994 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:23:31.994 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:31.994 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:23:31.994 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:25.648395+0000", 2024-09-13T11:23:31.994 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:31.994 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:31.994 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:31.994 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:23:31.994 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:25.648395+0000", 2024-09-13T11:23:31.994 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:31.994 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:31.994 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:31.995 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:23:31.995 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:25.648397+0000", 2024-09-13T11:23:31.995 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.1019999999999999e-06 2024-09-13T11:23:31.995 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:31.995 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:31.995 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:23:31.995 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:25.648505+0000", 2024-09-13T11:23:31.996 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00010809199999999999 2024-09-13T11:23:31.996 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:31.996 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:31.996 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:23:31.996 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:25.683164+0000", 2024-09-13T11:23:31.996 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.034659519999999999 2024-09-13T11:23:31.996 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:31.996 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:31.996 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:23:31.996 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:25.683184+0000", 2024-09-13T11:23:31.996 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.9706000000000001e-05 2024-09-13T11:23:31.997 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:31.997 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:31.997 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:31.997 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:25.683259+0000", 2024-09-13T11:23:31.997 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.4666000000000006e-05 2024-09-13T11:23:31.997 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:31.997 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:31.997 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:23:31.997 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:25.683268+0000", 2024-09-13T11:23:31.997 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.85e-06 2024-09-13T11:23:31.997 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:31.997 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:31.997 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:23:31.997 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:25.684091+0000", 2024-09-13T11:23:31.998 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00082383399999999996 2024-09-13T11:23:31.998 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:31.998 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:31.998 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:31.998 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:31.998 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:31.998 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDPGPush(3.15 71/70 [PushOp(3:ae0fc128:::benchmark_data_smithi177_31116_object3623:head, version: 44'224, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:ae0fc128:::benchmark_data_smithi177_31116_object3623:head@44'224, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:23:31.998 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:25.690389+0000", 2024-09-13T11:23:31.998 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 6.2336284519999996, 2024-09-13T11:23:31.998 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.039041207000000001, 2024-09-13T11:23:31.998 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:23:31.998 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:23:31.998 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:23:31.998 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:23:31.999 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:31.999 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:23:31.999 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:25.690389+0000", 2024-09-13T11:23:31.999 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:31.999 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:31.999 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:31.999 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:23:31.999 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:25.690389+0000", 2024-09-13T11:23:31.999 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:31.999 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:31.999 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:31.999 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:23:31.999 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:25.690391+0000", 2024-09-13T11:23:32.000 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.8980000000000001e-06 2024-09-13T11:23:32.000 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.000 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.000 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:23:32.000 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:25.690515+0000", 2024-09-13T11:23:32.000 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000123965 2024-09-13T11:23:32.000 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.000 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.000 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:23:32.000 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:25.728149+0000", 2024-09-13T11:23:32.000 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.037634473000000002 2024-09-13T11:23:32.000 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.000 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.000 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:23:32.001 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:25.728190+0000", 2024-09-13T11:23:32.001 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.0775000000000002e-05 2024-09-13T11:23:32.001 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.001 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.001 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:32.001 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:25.728291+0000", 2024-09-13T11:23:32.001 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00010094699999999999 2024-09-13T11:23:32.001 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.001 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.001 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:23:32.001 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:25.728303+0000", 2024-09-13T11:23:32.001 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.1586999999999999e-05 2024-09-13T11:23:32.001 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.001 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.002 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:23:32.002 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:25.729430+0000", 2024-09-13T11:23:32.002 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0011275619999999999 2024-09-13T11:23:32.002 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:32.002 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:32.002 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:32.002 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.002 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.002 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDPGPush(3.15 71/70 [PushOp(3:a97e9c9d:::benchmark_data_smithi177_31116_object3643:head, version: 44'227, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:a97e9c9d:::benchmark_data_smithi177_31116_object3643:head@44'227, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:23:32.002 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:25.791951+0000", 2024-09-13T11:23:32.002 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 6.1320664540000003, 2024-09-13T11:23:32.002 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.071600253000000003, 2024-09-13T11:23:32.002 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:23:32.003 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:23:32.003 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:23:32.003 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:23:32.003 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.003 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:23:32.003 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:25.791951+0000", 2024-09-13T11:23:32.003 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:32.003 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.003 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.003 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:23:32.003 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:25.791951+0000", 2024-09-13T11:23:32.003 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:32.003 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.003 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.004 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:23:32.004 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:25.791953+0000", 2024-09-13T11:23:32.004 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.609e-06 2024-09-13T11:23:32.004 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.004 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.004 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:23:32.004 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:25.791990+0000", 2024-09-13T11:23:32.004 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.7817000000000001e-05 2024-09-13T11:23:32.004 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.004 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.004 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:23:32.005 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:25.862623+0000", 2024-09-13T11:23:32.005 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.070632232000000003 2024-09-13T11:23:32.005 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.005 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.005 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:23:32.005 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:25.862638+0000", 2024-09-13T11:23:32.005 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.5577e-05 2024-09-13T11:23:32.005 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.005 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.005 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:32.005 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:25.862710+0000", 2024-09-13T11:23:32.005 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.1540999999999998e-05 2024-09-13T11:23:32.006 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.006 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.006 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:23:32.006 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:25.862717+0000", 2024-09-13T11:23:32.006 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.2099999999999996e-06 2024-09-13T11:23:32.006 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.006 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.006 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:23:32.006 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:25.863551+0000", 2024-09-13T11:23:32.006 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00083426699999999997 2024-09-13T11:23:32.006 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:32.006 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:32.006 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:32.006 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.007 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.007 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDPGPush(3.e 72/57 [PushOp(3:731e004d:::benchmark_data_smithi177_31116_object22930:head, version: 55'1516, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:731e004d:::benchmark_data_smithi177_31116_object22930:head@55'1516, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:23:32.007 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:26.052768+0000", 2024-09-13T11:23:32.007 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 5.8712494939999997, 2024-09-13T11:23:32.007 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.053461797999999998, 2024-09-13T11:23:32.007 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:23:32.007 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:23:32.007 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:23:32.007 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:23:32.007 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.007 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:23:32.007 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:26.052768+0000", 2024-09-13T11:23:32.007 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:32.008 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.008 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.008 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:23:32.008 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:26.052768+0000", 2024-09-13T11:23:32.008 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:32.008 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.008 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.008 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:23:32.008 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:26.052769+0000", 2024-09-13T11:23:32.008 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.279e-06 2024-09-13T11:23:32.008 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.008 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.008 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:23:32.009 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:26.052793+0000", 2024-09-13T11:23:32.009 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.3765e-05 2024-09-13T11:23:32.009 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.009 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.009 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:23:32.009 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:26.104876+0000", 2024-09-13T11:23:32.009 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.052083109000000002 2024-09-13T11:23:32.009 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.009 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.009 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:23:32.009 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:26.104887+0000", 2024-09-13T11:23:32.009 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.1358999999999999e-05 2024-09-13T11:23:32.009 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.009 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.010 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:32.010 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:26.104979+0000", 2024-09-13T11:23:32.010 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.1582000000000005e-05 2024-09-13T11:23:32.010 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.010 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.010 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:23:32.010 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:26.104993+0000", 2024-09-13T11:23:32.010 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.4063000000000001e-05 2024-09-13T11:23:32.010 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.010 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.010 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:23:32.010 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:26.106230+0000", 2024-09-13T11:23:32.010 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0012366409999999999 2024-09-13T11:23:32.011 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:32.011 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:32.011 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:32.011 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.011 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.011 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDPGPush(3.10 74/70 [PushOp(3:0bb0d12b:::benchmark_data_smithi177_31116_object11258:head, version: 48'710, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:0bb0d12b:::benchmark_data_smithi177_31116_object11258:head@48'710, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:23:32.011 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:30.077596+0000", 2024-09-13T11:23:32.011 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 1.846421726, 2024-09-13T11:23:32.011 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.035217726999999997, 2024-09-13T11:23:32.011 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:23:32.011 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:23:32.011 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:23:32.011 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:23:32.011 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.012 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:23:32.012 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:30.077596+0000", 2024-09-13T11:23:32.012 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:32.012 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.012 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.012 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:23:32.012 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:30.077596+0000", 2024-09-13T11:23:32.012 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:32.012 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.012 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.012 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:23:32.012 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:30.077597+0000", 2024-09-13T11:23:32.012 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.1829999999999999e-06 2024-09-13T11:23:32.013 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.013 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.013 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:23:32.013 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:30.077623+0000", 2024-09-13T11:23:32.013 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.567e-05 2024-09-13T11:23:32.013 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.013 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.013 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:23:32.013 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:30.111632+0000", 2024-09-13T11:23:32.013 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.034009427000000002 2024-09-13T11:23:32.013 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.013 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.013 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:23:32.013 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:30.111643+0000", 2024-09-13T11:23:32.014 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.1389999999999999e-05 2024-09-13T11:23:32.014 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.014 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.014 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:32.014 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:30.111760+0000", 2024-09-13T11:23:32.014 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000116439 2024-09-13T11:23:32.014 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.014 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.014 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:23:32.014 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:30.111776+0000", 2024-09-13T11:23:32.014 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.5716e-05 2024-09-13T11:23:32.014 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.014 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.015 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:23:32.015 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:30.112813+0000", 2024-09-13T11:23:32.015 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0010379020000000001 2024-09-13T11:23:32.015 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:32.015 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:32.015 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:32.015 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.015 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.015 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:54664 3.e 3.b3b490ce (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e75)", 2024-09-13T11:23:32.015 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:31.027497+0000", 2024-09-13T11:23:32.015 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 0.89652074900000001, 2024-09-13T11:23:32.015 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.058327592999999997, 2024-09-13T11:23:32.015 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:23:32.015 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:23:32.016 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-13T11:23:32.016 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:23:32.016 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:23:32.016 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:32.016 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 54664 2024-09-13T11:23:32.016 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.016 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:23:32.016 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.016 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:23:32.016 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.027497+0000", 2024-09-13T11:23:32.016 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:32.016 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.016 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.017 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:23:32.017 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.027497+0000", 2024-09-13T11:23:32.017 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:32.017 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.017 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.017 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:23:32.017 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.027505+0000", 2024-09-13T11:23:32.017 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.2740000000000007e-06 2024-09-13T11:23:32.017 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.017 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.017 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:23:32.017 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.027556+0000", 2024-09-13T11:23:32.017 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.0680000000000003e-05 2024-09-13T11:23:32.018 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.018 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.018 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:23:32.018 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.027588+0000", 2024-09-13T11:23:32.018 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.2081e-05 2024-09-13T11:23:32.018 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.018 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.018 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:23:32.018 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.027601+0000", 2024-09-13T11:23:32.018 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2801e-05 2024-09-13T11:23:32.018 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.018 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.018 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:32.018 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.085783+0000", 2024-09-13T11:23:32.019 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.058182506000000002 2024-09-13T11:23:32.019 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.019 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.019 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:23:32.019 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.085824+0000", 2024-09-13T11:23:32.019 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.1251000000000002e-05 2024-09-13T11:23:32.019 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:32.019 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:32.019 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:32.019 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:32.019 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:32.019 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-13T11:23:32.020 INFO:tasks.radosbench.radosbench.0.smithi177.stdout: 65 16 54768 54752 52.6404 6.75 0.0174137 0.0189722 2024-09-13T11:23:32.021 INFO:tasks.ceph.osd.7.smithi177.stderr:2024-09-13T11:23:32.017+0000 7f897b51d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26575) UID: 0 2024-09-13T11:23:32.022 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-13T11:23:32.121 INFO:tasks.ceph.osd.10.smithi162.stderr:2024-09-13T11:23:32.113+0000 7fb59c9f0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26317) UID: 0 2024-09-13T11:23:32.222 INFO:tasks.ceph.osd.4.smithi001.stderr:2024-09-13T11:23:32.216+0000 7ff8e0218640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26310) UID: 0 2024-09-13T11:23:32.236 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-13T11:23:32.236 INFO:teuthology.orchestra.run.smithi001.stdout: "ops": [], 2024-09-13T11:23:32.236 INFO:teuthology.orchestra.run.smithi001.stdout: "num_ops": 0 2024-09-13T11:23:32.236 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-13T11:23:32.257 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-13T11:23:32.322 INFO:tasks.ceph.osd.15.smithi177.stderr:2024-09-13T11:23:32.317+0000 7f9d6ce77640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26578) UID: 0 2024-09-13T11:23:32.424 INFO:tasks.ceph.osd.11.smithi177.stderr:2024-09-13T11:23:32.417+0000 7f24792c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26580) UID: 0 2024-09-13T11:23:32.438 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-13T11:23:32.439 INFO:teuthology.orchestra.run.smithi001.stdout: "ops": [], 2024-09-13T11:23:32.439 INFO:teuthology.orchestra.run.smithi001.stdout: "complaint_time": 30, 2024-09-13T11:23:32.439 INFO:teuthology.orchestra.run.smithi001.stdout: "num_blocked_ops": 0 2024-09-13T11:23:32.439 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-13T11:23:32.455 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-13T11:23:32.523 INFO:tasks.ceph.osd.7.smithi177.stderr:2024-09-13T11:23:32.517+0000 7f897b51d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26575) UID: 0 2024-09-13T11:23:32.589 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-13T11:23:32.590 INFO:teuthology.orchestra.run.smithi001.stdout: "size": 20, 2024-09-13T11:23:32.590 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 600, 2024-09-13T11:23:32.590 INFO:teuthology.orchestra.run.smithi001.stdout: "ops": [ 2024-09-13T11:23:32.590 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.590 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:45776 3.11 3.2ba9efd1 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e69)", 2024-09-13T11:23:32.590 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:20.932153+0000", 2024-09-13T11:23:32.590 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 11.653192896, 2024-09-13T11:23:32.590 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0091029370000000002, 2024-09-13T11:23:32.590 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:23:32.590 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:23:32.590 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-13T11:23:32.591 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:23:32.591 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:23:32.591 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:32.591 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 45776 2024-09-13T11:23:32.591 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.591 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:23:32.591 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.591 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:23:32.591 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:20.932153+0000", 2024-09-13T11:23:32.591 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:32.591 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.591 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.591 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:23:32.591 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:20.932153+0000", 2024-09-13T11:23:32.592 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:32.592 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.592 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.592 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:23:32.592 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:20.932156+0000", 2024-09-13T11:23:32.592 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.0230000000000001e-06 2024-09-13T11:23:32.592 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.592 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.592 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:23:32.592 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:20.932311+0000", 2024-09-13T11:23:32.592 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00015488599999999999 2024-09-13T11:23:32.592 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.592 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.592 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:23:32.593 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:20.932330+0000", 2024-09-13T11:23:32.593 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.9284e-05 2024-09-13T11:23:32.593 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.593 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.593 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:23:32.593 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:20.932343+0000", 2024-09-13T11:23:32.593 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.3125000000000001e-05 2024-09-13T11:23:32.593 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.593 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.593 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:32.593 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:20.941211+0000", 2024-09-13T11:23:32.593 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.008868183 2024-09-13T11:23:32.593 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.594 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.594 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:23:32.594 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:20.941256+0000", 2024-09-13T11:23:32.594 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.4435999999999999e-05 2024-09-13T11:23:32.594 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:32.594 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:32.594 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:32.594 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.594 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.594 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4665.0:48326 3.11 e71/70 3:892c8b48:::benchmark_data_smithi177_31116_object48325:head v 71'2881, mlcod=44'195)", 2024-09-13T11:23:32.594 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:24.932419+0000", 2024-09-13T11:23:32.594 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 7.6529267279999997, 2024-09-13T11:23:32.595 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.070889482000000004, 2024-09-13T11:23:32.595 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:23:32.595 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:23:32.595 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:32.595 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:23:32.595 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.595 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:23:32.595 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:24.932419+0000", 2024-09-13T11:23:32.595 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:32.595 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.595 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.595 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:23:32.595 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:24.932419+0000", 2024-09-13T11:23:32.595 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:32.596 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.596 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.596 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:23:32.596 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:24.932421+0000", 2024-09-13T11:23:32.596 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.4260000000000002e-06 2024-09-13T11:23:32.596 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.596 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.596 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:23:32.596 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:24.932492+0000", 2024-09-13T11:23:32.596 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.0553e-05 2024-09-13T11:23:32.596 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.596 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.596 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:23:32.596 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:25.001307+0000", 2024-09-13T11:23:32.597 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.068815692999999997 2024-09-13T11:23:32.597 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.597 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.597 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:23:32.597 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:25.001314+0000", 2024-09-13T11:23:32.597 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.319e-06 2024-09-13T11:23:32.597 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.597 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.597 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:32.597 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:25.001375+0000", 2024-09-13T11:23:32.597 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.1530000000000002e-05 2024-09-13T11:23:32.597 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.597 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.597 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:23:32.598 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:25.001406+0000", 2024-09-13T11:23:32.598 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.0559000000000003e-05 2024-09-13T11:23:32.598 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.598 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.598 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:23:32.598 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:25.003260+0000", 2024-09-13T11:23:32.598 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0018541720000000001 2024-09-13T11:23:32.598 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.598 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.598 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:23:32.599 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:25.003308+0000", 2024-09-13T11:23:32.599 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.8229999999999997e-05 2024-09-13T11:23:32.599 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:32.599 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:32.599 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:32.599 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.599 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.599 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDPGPush(3.11 71/70 [PushOp(3:8b9d23bc:::benchmark_data_smithi177_31116_object5582:head, version: 45'326, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:8b9d23bc:::benchmark_data_smithi177_31116_object5582:head@45'326, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:23:32.599 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:25.675060+0000", 2024-09-13T11:23:32.599 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 6.9102856299999997, 2024-09-13T11:23:32.599 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.022925826999999999, 2024-09-13T11:23:32.599 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:23:32.599 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:23:32.599 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:23:32.600 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:23:32.600 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.600 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:23:32.600 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:25.675060+0000", 2024-09-13T11:23:32.600 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:32.600 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.600 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.600 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:23:32.600 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:25.675060+0000", 2024-09-13T11:23:32.600 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:32.600 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.600 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.600 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:23:32.600 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:25.675061+0000", 2024-09-13T11:23:32.601 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.463e-06 2024-09-13T11:23:32.601 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.601 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.601 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:23:32.601 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:25.675117+0000", 2024-09-13T11:23:32.601 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.5754000000000002e-05 2024-09-13T11:23:32.601 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.601 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.601 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:23:32.601 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:25.696693+0000", 2024-09-13T11:23:32.601 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.021575555 2024-09-13T11:23:32.601 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.601 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.601 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:23:32.602 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:25.696711+0000", 2024-09-13T11:23:32.602 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.8808e-05 2024-09-13T11:23:32.602 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.602 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.602 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:32.602 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:25.696870+0000", 2024-09-13T11:23:32.602 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00015851099999999999 2024-09-13T11:23:32.602 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.602 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.602 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:23:32.602 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:25.696896+0000", 2024-09-13T11:23:32.602 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.6308999999999998e-05 2024-09-13T11:23:32.602 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.602 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.603 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:23:32.603 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:25.697986+0000", 2024-09-13T11:23:32.603 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.001089427 2024-09-13T11:23:32.603 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:32.603 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:32.603 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:32.603 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.603 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.603 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDPGPush(3.11 71/70 [PushOp(3:8ce47ede:::benchmark_data_smithi177_31116_object5592:head, version: 45'327, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:8ce47ede:::benchmark_data_smithi177_31116_object5592:head@45'327, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:23:32.603 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:25.702108+0000", 2024-09-13T11:23:32.603 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 6.8832373990000004, 2024-09-13T11:23:32.603 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.021996188, 2024-09-13T11:23:32.603 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:23:32.604 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:23:32.604 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:23:32.604 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:23:32.604 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.604 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:23:32.604 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:25.702108+0000", 2024-09-13T11:23:32.604 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:32.604 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.604 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.604 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:23:32.604 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:25.702108+0000", 2024-09-13T11:23:32.604 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:32.604 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.604 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.605 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:23:32.605 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:25.702110+0000", 2024-09-13T11:23:32.605 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.4419999999999999e-06 2024-09-13T11:23:32.605 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.605 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.605 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:23:32.605 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:25.702255+0000", 2024-09-13T11:23:32.605 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000145007 2024-09-13T11:23:32.605 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.605 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.605 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:23:32.605 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:25.723246+0000", 2024-09-13T11:23:32.605 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.020991586999999999 2024-09-13T11:23:32.605 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.606 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.606 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:23:32.606 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:25.723263+0000", 2024-09-13T11:23:32.606 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.7217000000000001e-05 2024-09-13T11:23:32.606 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.606 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.606 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:32.606 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:25.723369+0000", 2024-09-13T11:23:32.606 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000105173 2024-09-13T11:23:32.606 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.606 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.606 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:23:32.606 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:25.723382+0000", 2024-09-13T11:23:32.606 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.3619e-05 2024-09-13T11:23:32.607 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.607 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.607 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:23:32.607 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:25.724104+0000", 2024-09-13T11:23:32.607 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00072214299999999996 2024-09-13T11:23:32.607 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:32.607 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:32.607 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:32.607 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.607 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.607 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDPGPush(3.11 71/70 [PushOp(3:8a7dfb62:::benchmark_data_smithi177_31116_object5716:head, version: 45'331, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:8a7dfb62:::benchmark_data_smithi177_31116_object5716:head@45'331, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:23:32.607 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:25.728968+0000", 2024-09-13T11:23:32.607 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 6.8563778480000002, 2024-09-13T11:23:32.607 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.01838157, 2024-09-13T11:23:32.608 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:23:32.608 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:23:32.608 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:23:32.608 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:23:32.608 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.608 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:23:32.608 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:25.728968+0000", 2024-09-13T11:23:32.608 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:32.608 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.608 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.608 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:23:32.608 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:25.728968+0000", 2024-09-13T11:23:32.608 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:32.609 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.609 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.609 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:23:32.609 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:25.728970+0000", 2024-09-13T11:23:32.609 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.5380000000000001e-06 2024-09-13T11:23:32.609 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.609 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.609 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:23:32.609 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:25.729040+0000", 2024-09-13T11:23:32.609 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.9727999999999994e-05 2024-09-13T11:23:32.609 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.609 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.609 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:23:32.609 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:25.746427+0000", 2024-09-13T11:23:32.610 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.017386922999999999 2024-09-13T11:23:32.610 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.610 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.610 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:23:32.610 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:25.746448+0000", 2024-09-13T11:23:32.610 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.1568000000000001e-05 2024-09-13T11:23:32.610 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.610 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.610 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:32.610 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:25.746552+0000", 2024-09-13T11:23:32.610 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000104048 2024-09-13T11:23:32.610 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.610 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.610 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:23:32.611 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:25.746569+0000", 2024-09-13T11:23:32.611 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.6742999999999999e-05 2024-09-13T11:23:32.611 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.611 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.611 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:23:32.611 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:25.747349+0000", 2024-09-13T11:23:32.611 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00078002200000000001 2024-09-13T11:23:32.611 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:32.611 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:32.611 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:32.611 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.612 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.612 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDPGPush(3.11 71/70 [PushOp(3:8c4ce3e4:::benchmark_data_smithi177_31116_object5737:head, version: 45'333, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:8c4ce3e4:::benchmark_data_smithi177_31116_object5737:head@45'333, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:23:32.612 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:25.752459+0000", 2024-09-13T11:23:32.612 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 6.8328867009999996, 2024-09-13T11:23:32.612 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.011535642, 2024-09-13T11:23:32.612 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:23:32.612 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:23:32.613 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:23:32.613 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:23:32.613 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.613 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:23:32.614 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:25.752459+0000", 2024-09-13T11:23:32.614 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:32.614 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.614 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.614 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:23:32.614 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:25.752459+0000", 2024-09-13T11:23:32.614 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:32.614 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.614 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.614 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:23:32.614 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:25.752464+0000", 2024-09-13T11:23:32.614 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.215e-06 2024-09-13T11:23:32.614 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.614 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.615 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:23:32.615 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:25.752516+0000", 2024-09-13T11:23:32.615 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.1625000000000001e-05 2024-09-13T11:23:32.615 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.615 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.615 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:23:32.615 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:25.763146+0000", 2024-09-13T11:23:32.615 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.010630723999999999 2024-09-13T11:23:32.615 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.615 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.615 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:23:32.615 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:25.763160+0000", 2024-09-13T11:23:32.615 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.3752e-05 2024-09-13T11:23:32.615 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.616 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.616 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:32.616 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:25.763272+0000", 2024-09-13T11:23:32.616 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000111858 2024-09-13T11:23:32.616 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.616 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.616 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:23:32.616 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:25.763289+0000", 2024-09-13T11:23:32.616 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.6631999999999999e-05 2024-09-13T11:23:32.616 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.616 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.616 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:23:32.616 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:25.763994+0000", 2024-09-13T11:23:32.617 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00070583600000000005 2024-09-13T11:23:32.617 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:32.617 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:32.617 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:32.617 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.617 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.617 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDPGPush(3.11 71/70 [PushOp(3:8f196550:::benchmark_data_smithi177_31116_object5796:head, version: 45'338, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:8f196550:::benchmark_data_smithi177_31116_object5796:head@45'338, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:23:32.617 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:25.767591+0000", 2024-09-13T11:23:32.617 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 6.817754162, 2024-09-13T11:23:32.617 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.023213363000000001, 2024-09-13T11:23:32.617 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:23:32.617 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:23:32.617 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:23:32.617 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:23:32.618 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.618 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:23:32.618 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:25.767591+0000", 2024-09-13T11:23:32.618 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:32.618 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.618 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.618 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:23:32.618 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:25.767591+0000", 2024-09-13T11:23:32.618 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:32.618 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.618 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.618 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:23:32.618 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:25.767593+0000", 2024-09-13T11:23:32.619 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.4410000000000001e-06 2024-09-13T11:23:32.619 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.619 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.619 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:23:32.619 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:25.767675+0000", 2024-09-13T11:23:32.619 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.2316000000000003e-05 2024-09-13T11:23:32.619 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.619 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.619 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:23:32.619 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:25.789911+0000", 2024-09-13T11:23:32.619 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.022236359000000001 2024-09-13T11:23:32.619 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.619 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.619 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:23:32.620 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:25.789929+0000", 2024-09-13T11:23:32.620 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.7082999999999999e-05 2024-09-13T11:23:32.620 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.620 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.620 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:32.620 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:25.790044+0000", 2024-09-13T11:23:32.620 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00011573 2024-09-13T11:23:32.620 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.620 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.620 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:23:32.620 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:25.790057+0000", 2024-09-13T11:23:32.620 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2649999999999999e-05 2024-09-13T11:23:32.620 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.620 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.621 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:23:32.621 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:25.790805+0000", 2024-09-13T11:23:32.621 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00074778400000000004 2024-09-13T11:23:32.621 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:32.621 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:32.621 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:32.621 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.621 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.621 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4665.0:49275 3.11 e71/70 3:8a850d9d:::benchmark_data_smithi177_31116_object49274:head v 71'2907, mlcod=45'333)", 2024-09-13T11:23:32.621 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:25.791023+0000", 2024-09-13T11:23:32.621 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 6.7943220889999996, 2024-09-13T11:23:32.621 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0096330270000000006, 2024-09-13T11:23:32.621 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:23:32.622 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:23:32.622 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:32.622 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:23:32.622 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.622 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:23:32.622 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:25.791023+0000", 2024-09-13T11:23:32.622 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:32.622 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.622 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.622 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:23:32.622 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:25.791023+0000", 2024-09-13T11:23:32.622 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:32.622 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.623 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.623 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:23:32.623 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:25.791025+0000", 2024-09-13T11:23:32.623 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2720000000000001e-06 2024-09-13T11:23:32.623 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.623 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.623 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:23:32.624 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:25.791072+0000", 2024-09-13T11:23:32.624 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.71e-05 2024-09-13T11:23:32.624 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.624 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.624 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:23:32.624 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:25.798071+0000", 2024-09-13T11:23:32.624 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0069995550000000002 2024-09-13T11:23:32.624 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.624 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.625 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:23:32.625 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:25.798080+0000", 2024-09-13T11:23:32.625 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.3769999999999996e-06 2024-09-13T11:23:32.625 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.625 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.625 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:32.625 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:25.798172+0000", 2024-09-13T11:23:32.625 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.2281999999999995e-05 2024-09-13T11:23:32.625 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.625 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.626 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:23:32.626 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:25.798200+0000", 2024-09-13T11:23:32.626 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.7633000000000001e-05 2024-09-13T11:23:32.626 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.626 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.626 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:23:32.626 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:25.800597+0000", 2024-09-13T11:23:32.626 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.002396881 2024-09-13T11:23:32.626 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.626 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.626 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:23:32.626 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:25.800656+0000", 2024-09-13T11:23:32.626 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.9926999999999997e-05 2024-09-13T11:23:32.626 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:32.627 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:32.627 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:32.627 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.627 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.627 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4665.0:49284 3.11 e71/70 3:88f06924:::benchmark_data_smithi177_31116_object49283:head v 71'2908, mlcod=45'338)", 2024-09-13T11:23:32.627 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:25.798398+0000", 2024-09-13T11:23:32.627 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 6.7869475320000001, 2024-09-13T11:23:32.627 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0095383529999999994, 2024-09-13T11:23:32.627 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:23:32.627 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:23:32.627 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:32.628 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:23:32.628 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.628 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:23:32.628 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:25.798398+0000", 2024-09-13T11:23:32.628 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:32.628 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.628 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.628 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:23:32.628 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:25.798398+0000", 2024-09-13T11:23:32.628 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:32.628 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.628 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.628 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:23:32.629 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:25.798399+0000", 2024-09-13T11:23:32.629 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2839999999999999e-06 2024-09-13T11:23:32.629 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.629 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.629 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:23:32.629 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:25.798452+0000", 2024-09-13T11:23:32.629 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.253e-05 2024-09-13T11:23:32.629 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.629 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.629 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:23:32.629 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:25.806014+0000", 2024-09-13T11:23:32.629 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0075619199999999998 2024-09-13T11:23:32.629 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.630 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.630 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:23:32.630 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:25.806023+0000", 2024-09-13T11:23:32.630 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.5789999999999993e-06 2024-09-13T11:23:32.630 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.630 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.630 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:32.630 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:25.806087+0000", 2024-09-13T11:23:32.630 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.3454999999999995e-05 2024-09-13T11:23:32.630 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.631 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.631 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:23:32.631 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:25.806112+0000", 2024-09-13T11:23:32.631 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.5225e-05 2024-09-13T11:23:32.631 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.631 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.631 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:23:32.631 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:25.807901+0000", 2024-09-13T11:23:32.631 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0017891249999999999 2024-09-13T11:23:32.631 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.631 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.631 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:23:32.631 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:25.807936+0000", 2024-09-13T11:23:32.632 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.5234999999999997e-05 2024-09-13T11:23:32.632 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:32.632 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:32.632 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:32.632 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.632 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.632 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDPGPush(3.11 72/70 [PushOp(3:8ae6b6fc:::benchmark_data_smithi177_31116_object7617:head, version: 46'455, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:8ae6b6fc:::benchmark_data_smithi177_31116_object7617:head@46'455, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:23:32.632 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:26.176296+0000", 2024-09-13T11:23:32.632 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 6.409049284, 2024-09-13T11:23:32.632 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.13349134900000001, 2024-09-13T11:23:32.632 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:23:32.633 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:23:32.633 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:23:32.633 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:23:32.633 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.633 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:23:32.633 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:26.176296+0000", 2024-09-13T11:23:32.633 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:32.633 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.633 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.633 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:23:32.633 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:26.176296+0000", 2024-09-13T11:23:32.633 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:32.633 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.633 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.634 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:23:32.634 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:26.176297+0000", 2024-09-13T11:23:32.634 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2440000000000001e-06 2024-09-13T11:23:32.634 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.634 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.634 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:23:32.634 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:26.176433+0000", 2024-09-13T11:23:32.634 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00013554999999999999 2024-09-13T11:23:32.634 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.634 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.634 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:23:32.634 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:26.308747+0000", 2024-09-13T11:23:32.634 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.132314392 2024-09-13T11:23:32.634 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.635 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.635 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:23:32.635 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:26.308768+0000", 2024-09-13T11:23:32.635 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.0758999999999999e-05 2024-09-13T11:23:32.635 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.635 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.636 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:32.636 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:26.308874+0000", 2024-09-13T11:23:32.636 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000105334 2024-09-13T11:23:32.636 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.636 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.636 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:23:32.636 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:26.308890+0000", 2024-09-13T11:23:32.636 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.6367000000000001e-05 2024-09-13T11:23:32.636 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.636 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.636 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:23:32.636 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:26.309788+0000", 2024-09-13T11:23:32.637 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00089770300000000002 2024-09-13T11:23:32.637 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:32.637 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:32.638 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:32.638 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.638 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.638 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDPGPush(3.11 72/70 [PushOp(3:8f62e9f7:::benchmark_data_smithi177_31116_object7652:head, version: 46'458, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:8f62e9f7:::benchmark_data_smithi177_31116_object7652:head@46'458, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:23:32.638 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:26.326138+0000", 2024-09-13T11:23:32.638 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 6.2592079629999997, 2024-09-13T11:23:32.638 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.036052775000000002, 2024-09-13T11:23:32.638 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:23:32.638 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:23:32.638 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:23:32.638 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:23:32.638 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.638 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:23:32.639 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:26.326138+0000", 2024-09-13T11:23:32.639 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:32.639 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.639 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.639 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:23:32.639 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:26.326138+0000", 2024-09-13T11:23:32.639 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:32.639 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.639 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.639 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:23:32.639 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:26.326139+0000", 2024-09-13T11:23:32.639 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.5290000000000001e-06 2024-09-13T11:23:32.639 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.639 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.640 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:23:32.640 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:26.326194+0000", 2024-09-13T11:23:32.640 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.4833999999999997e-05 2024-09-13T11:23:32.640 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.640 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.640 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:23:32.640 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:26.361482+0000", 2024-09-13T11:23:32.640 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.035288370999999999 2024-09-13T11:23:32.640 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.640 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.640 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:23:32.640 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:26.361502+0000", 2024-09-13T11:23:32.640 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.9400999999999999e-05 2024-09-13T11:23:32.641 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.641 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.641 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:32.641 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:26.361595+0000", 2024-09-13T11:23:32.641 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.3535000000000005e-05 2024-09-13T11:23:32.641 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.641 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.641 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:23:32.641 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:26.361604+0000", 2024-09-13T11:23:32.641 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.4449999999999996e-06 2024-09-13T11:23:32.641 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.641 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.641 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:23:32.641 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:26.362190+0000", 2024-09-13T11:23:32.642 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00058666000000000002 2024-09-13T11:23:32.642 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:32.642 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:32.642 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:32.642 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.642 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.642 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDPGPush(3.11 72/70 [PushOp(3:8d1d2c46:::benchmark_data_smithi177_31116_object7682:head, version: 46'460, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:8d1d2c46:::benchmark_data_smithi177_31116_object7682:head@46'460, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:23:32.642 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:26.365915+0000", 2024-09-13T11:23:32.642 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 6.2194308879999998, 2024-09-13T11:23:32.642 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.023631208000000001, 2024-09-13T11:23:32.642 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:23:32.642 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:23:32.642 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:23:32.643 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:23:32.643 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.643 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:23:32.643 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:26.365915+0000", 2024-09-13T11:23:32.643 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:32.643 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.643 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.643 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:23:32.643 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:26.365915+0000", 2024-09-13T11:23:32.643 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:32.643 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.643 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.643 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:23:32.644 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:26.365917+0000", 2024-09-13T11:23:32.644 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.3879999999999998e-06 2024-09-13T11:23:32.644 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.644 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.644 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:23:32.644 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:26.365981+0000", 2024-09-13T11:23:32.644 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.3775000000000005e-05 2024-09-13T11:23:32.644 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.644 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.644 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:23:32.644 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:26.388918+0000", 2024-09-13T11:23:32.644 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.022937362999999999 2024-09-13T11:23:32.644 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.645 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.645 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:23:32.645 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:26.388938+0000", 2024-09-13T11:23:32.645 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.0021999999999999e-05 2024-09-13T11:23:32.645 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.645 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.645 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:32.645 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:26.389012+0000", 2024-09-13T11:23:32.645 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.3654999999999999e-05 2024-09-13T11:23:32.645 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.645 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.645 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:23:32.645 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:26.389023+0000", 2024-09-13T11:23:32.645 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.1177e-05 2024-09-13T11:23:32.646 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.646 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.646 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:23:32.646 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:26.389546+0000", 2024-09-13T11:23:32.646 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00052282800000000003 2024-09-13T11:23:32.646 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:32.646 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:32.646 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:32.646 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.646 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.646 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDPGPush(3.11 72/70 [PushOp(3:8da9c977:::benchmark_data_smithi177_31116_object7736:head, version: 46'461, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:8da9c977:::benchmark_data_smithi177_31116_object7736:head@46'461, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:23:32.646 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:26.393216+0000", 2024-09-13T11:23:32.647 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 6.1921294250000001, 2024-09-13T11:23:32.647 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.15683955499999999, 2024-09-13T11:23:32.647 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:23:32.647 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:23:32.647 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:23:32.647 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:23:32.647 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.647 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:23:32.647 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:26.393216+0000", 2024-09-13T11:23:32.647 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:32.647 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.647 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.647 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:23:32.648 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:26.393216+0000", 2024-09-13T11:23:32.648 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:32.648 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.648 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.648 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:23:32.648 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:26.393217+0000", 2024-09-13T11:23:32.648 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2410000000000001e-06 2024-09-13T11:23:32.648 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.648 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.648 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:23:32.648 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:26.393309+0000", 2024-09-13T11:23:32.648 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.2101000000000005e-05 2024-09-13T11:23:32.648 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.649 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.649 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:23:32.649 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:26.547936+0000", 2024-09-13T11:23:32.649 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.15462682699999999 2024-09-13T11:23:32.649 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.649 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.649 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:23:32.649 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:26.548021+0000", 2024-09-13T11:23:32.649 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.4529000000000003e-05 2024-09-13T11:23:32.649 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.649 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.649 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:32.649 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:26.548321+0000", 2024-09-13T11:23:32.649 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00029988399999999999 2024-09-13T11:23:32.650 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.650 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.650 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:23:32.650 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:26.548353+0000", 2024-09-13T11:23:32.650 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.2573000000000001e-05 2024-09-13T11:23:32.650 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.650 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.650 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:23:32.650 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:26.550056+0000", 2024-09-13T11:23:32.650 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0017024 2024-09-13T11:23:32.650 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:32.650 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:32.650 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:32.650 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.651 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.651 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4665.0:50103 3.11 e72/70 3:8afe6275:::benchmark_data_smithi177_31116_object50102:head v 72'2926, mlcod=46'460)", 2024-09-13T11:23:32.651 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:26.547745+0000", 2024-09-13T11:23:32.651 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 6.0376005810000004, 2024-09-13T11:23:32.651 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.014832488, 2024-09-13T11:23:32.651 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:23:32.651 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:23:32.651 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:32.651 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:23:32.651 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.651 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:23:32.651 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:26.547745+0000", 2024-09-13T11:23:32.651 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:32.652 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.652 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.652 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:23:32.652 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:26.547745+0000", 2024-09-13T11:23:32.652 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:32.652 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.652 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.652 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:23:32.652 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:26.547750+0000", 2024-09-13T11:23:32.652 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.1229999999999999e-06 2024-09-13T11:23:32.652 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.652 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.652 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:23:32.652 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:26.547854+0000", 2024-09-13T11:23:32.653 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00010354899999999999 2024-09-13T11:23:32.653 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.653 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.653 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:23:32.653 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:26.559803+0000", 2024-09-13T11:23:32.653 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.011949859 2024-09-13T11:23:32.653 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.653 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.653 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:23:32.653 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:26.559815+0000", 2024-09-13T11:23:32.653 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2034e-05 2024-09-13T11:23:32.653 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.653 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.653 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:32.654 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:26.559958+0000", 2024-09-13T11:23:32.654 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00014229800000000001 2024-09-13T11:23:32.654 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.654 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.654 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:23:32.654 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:26.560001+0000", 2024-09-13T11:23:32.654 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.2873000000000001e-05 2024-09-13T11:23:32.654 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.654 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.654 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:23:32.654 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:26.562461+0000", 2024-09-13T11:23:32.654 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0024605159999999998 2024-09-13T11:23:32.654 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.654 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.655 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:23:32.655 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:26.562577+0000", 2024-09-13T11:23:32.655 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000116236 2024-09-13T11:23:32.655 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:32.655 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:32.655 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:32.655 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.655 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.655 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDPGPush(3.11 72/70 [PushOp(3:88597a03:::benchmark_data_smithi177_31116_object7757:head, version: 46'463, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:88597a03:::benchmark_data_smithi177_31116_object7757:head@46'463, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:23:32.655 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:26.559694+0000", 2024-09-13T11:23:32.655 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 6.0256511489999998, 2024-09-13T11:23:32.655 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.038634953, 2024-09-13T11:23:32.655 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:23:32.655 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:23:32.656 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:23:32.656 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:23:32.656 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.656 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:23:32.656 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:26.559694+0000", 2024-09-13T11:23:32.656 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:32.656 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.656 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.656 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:23:32.656 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:26.559694+0000", 2024-09-13T11:23:32.656 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:32.656 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.656 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.656 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:23:32.657 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:26.559698+0000", 2024-09-13T11:23:32.657 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.2059999999999999e-06 2024-09-13T11:23:32.657 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.657 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.657 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:23:32.657 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:26.559771+0000", 2024-09-13T11:23:32.657 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.3705e-05 2024-09-13T11:23:32.657 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.657 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.657 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:23:32.657 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:26.595040+0000", 2024-09-13T11:23:32.657 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.035268769999999998 2024-09-13T11:23:32.657 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.657 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.658 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:23:32.658 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:26.595096+0000", 2024-09-13T11:23:32.658 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.5933000000000001e-05 2024-09-13T11:23:32.658 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.658 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.658 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:32.658 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:26.595365+0000", 2024-09-13T11:23:32.658 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00026861399999999997 2024-09-13T11:23:32.658 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.658 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.658 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:23:32.658 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:26.595408+0000", 2024-09-13T11:23:32.658 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.3866999999999999e-05 2024-09-13T11:23:32.658 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.659 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.659 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:23:32.659 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:26.598329+0000", 2024-09-13T11:23:32.659 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0029208580000000001 2024-09-13T11:23:32.659 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:32.659 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:32.659 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:32.659 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.659 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.659 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4665.0:50141 3.11 e72/70 3:890018ca:::benchmark_data_smithi177_31116_object50140:head v 72'2927, mlcod=46'461)", 2024-09-13T11:23:32.659 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:26.594874+0000", 2024-09-13T11:23:32.659 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 5.9904714229999998, 2024-09-13T11:23:32.659 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.014570400000000001, 2024-09-13T11:23:32.659 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:23:32.660 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:23:32.660 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:32.660 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:23:32.660 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.660 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:23:32.660 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:26.594874+0000", 2024-09-13T11:23:32.660 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:32.660 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.660 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.660 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:23:32.660 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:26.594874+0000", 2024-09-13T11:23:32.660 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:32.660 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.660 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.661 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:23:32.661 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:26.594879+0000", 2024-09-13T11:23:32.661 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.8369999999999996e-06 2024-09-13T11:23:32.661 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.661 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.661 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:23:32.661 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:26.594972+0000", 2024-09-13T11:23:32.661 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.3542000000000007e-05 2024-09-13T11:23:32.661 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.661 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.661 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:23:32.661 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:26.604336+0000", 2024-09-13T11:23:32.661 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0093638139999999998 2024-09-13T11:23:32.661 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.662 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.662 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:23:32.662 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:26.604350+0000", 2024-09-13T11:23:32.662 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.3746e-05 2024-09-13T11:23:32.662 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.662 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.662 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:32.662 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:26.604476+0000", 2024-09-13T11:23:32.662 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00012633699999999999 2024-09-13T11:23:32.662 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.662 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.662 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:23:32.662 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:26.604536+0000", 2024-09-13T11:23:32.662 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.9882e-05 2024-09-13T11:23:32.663 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.663 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.663 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:23:32.663 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:26.609333+0000", 2024-09-13T11:23:32.663 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0047971510000000004 2024-09-13T11:23:32.663 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.663 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.663 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:23:32.663 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:26.609444+0000", 2024-09-13T11:23:32.663 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000111091 2024-09-13T11:23:32.663 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:32.663 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:32.663 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:32.663 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.664 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.664 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4665.0:54052 3.11 e74/70 3:8c6be821:::benchmark_data_smithi177_31116_object54051:head v 74'3049, mlcod=55'1352)", 2024-09-13T11:23:32.664 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:30.314329+0000", 2024-09-13T11:23:32.664 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 2.2710160309999998, 2024-09-13T11:23:32.664 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.100070243, 2024-09-13T11:23:32.664 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:23:32.664 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:23:32.664 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:32.664 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:23:32.664 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.664 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:23:32.664 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:30.314329+0000", 2024-09-13T11:23:32.664 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:32.664 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.665 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.665 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:23:32.665 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:30.314329+0000", 2024-09-13T11:23:32.665 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:32.665 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.665 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.665 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:23:32.665 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:30.314332+0000", 2024-09-13T11:23:32.665 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.1900000000000002e-06 2024-09-13T11:23:32.665 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.665 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.665 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:23:32.665 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:30.314372+0000", 2024-09-13T11:23:32.665 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.0788000000000003e-05 2024-09-13T11:23:32.666 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.666 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.666 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:23:32.666 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:30.412014+0000", 2024-09-13T11:23:32.666 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.097641172999999998 2024-09-13T11:23:32.666 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.666 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.666 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:23:32.666 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:30.412026+0000", 2024-09-13T11:23:32.666 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2551999999999999e-05 2024-09-13T11:23:32.666 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.666 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.666 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:32.666 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:30.412134+0000", 2024-09-13T11:23:32.667 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00010763400000000001 2024-09-13T11:23:32.667 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.667 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.667 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:23:32.667 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:30.412172+0000", 2024-09-13T11:23:32.667 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.8333e-05 2024-09-13T11:23:32.667 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.667 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.667 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:23:32.667 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:30.414340+0000", 2024-09-13T11:23:32.667 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0021680879999999999 2024-09-13T11:23:32.667 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.667 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.668 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:23:32.668 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:30.414400+0000", 2024-09-13T11:23:32.668 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.9484999999999997e-05 2024-09-13T11:23:32.668 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:32.668 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:32.668 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:32.668 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.668 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.668 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4665.0:54158 3.11 e74/70 3:88b7f35c:::benchmark_data_smithi177_31116_object54157:head v 74'3050, mlcod=55'1352)", 2024-09-13T11:23:32.668 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:30.411893+0000", 2024-09-13T11:23:32.668 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 2.1734525370000002, 2024-09-13T11:23:32.668 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0093750039999999993, 2024-09-13T11:23:32.668 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:23:32.668 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:23:32.669 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:32.669 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:23:32.669 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.669 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:23:32.669 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:30.411893+0000", 2024-09-13T11:23:32.669 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:32.669 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.669 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.669 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:23:32.669 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:30.411893+0000", 2024-09-13T11:23:32.669 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:32.669 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.669 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.670 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:23:32.670 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:30.411897+0000", 2024-09-13T11:23:32.670 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.8430000000000003e-06 2024-09-13T11:23:32.670 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.670 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.670 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:23:32.670 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:30.411973+0000", 2024-09-13T11:23:32.670 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.6266000000000005e-05 2024-09-13T11:23:32.670 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.670 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.670 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:23:32.670 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:30.419164+0000", 2024-09-13T11:23:32.670 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0071908270000000003 2024-09-13T11:23:32.670 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.671 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.671 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:23:32.671 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:30.419174+0000", 2024-09-13T11:23:32.671 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0131e-05 2024-09-13T11:23:32.671 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.671 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.671 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:32.671 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:30.419244+0000", 2024-09-13T11:23:32.671 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.0017000000000004e-05 2024-09-13T11:23:32.671 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.671 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.671 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:23:32.671 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:30.419263+0000", 2024-09-13T11:23:32.672 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.8470000000000001e-05 2024-09-13T11:23:32.672 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.672 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.672 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:23:32.672 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:30.421208+0000", 2024-09-13T11:23:32.672 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0019455500000000001 2024-09-13T11:23:32.672 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.672 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.672 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:23:32.672 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:30.421268+0000", 2024-09-13T11:23:32.672 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.9899999999999999e-05 2024-09-13T11:23:32.672 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:32.672 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:32.672 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:32.673 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.673 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.673 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDPGPush(3.11 74/70 [PushOp(3:88fa18ae:::benchmark_data_smithi177_31116_object22472:head, version: 55'1361, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:88fa18ae:::benchmark_data_smithi177_31116_object22472:head@55'1361, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:23:32.673 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:30.419079+0000", 2024-09-13T11:23:32.673 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 2.1662664110000001, 2024-09-13T11:23:32.673 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.030140384999999999, 2024-09-13T11:23:32.673 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:23:32.673 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:23:32.673 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:23:32.673 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:23:32.673 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.673 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:23:32.673 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:30.419079+0000", 2024-09-13T11:23:32.674 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:32.674 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.674 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.674 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:23:32.674 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:30.419079+0000", 2024-09-13T11:23:32.674 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:32.674 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.674 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.674 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:23:32.674 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:30.419081+0000", 2024-09-13T11:23:32.674 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.2570000000000002e-06 2024-09-13T11:23:32.674 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.674 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.674 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:23:32.675 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:30.419136+0000", 2024-09-13T11:23:32.675 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.5022000000000003e-05 2024-09-13T11:23:32.675 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.675 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.675 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:23:32.675 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:30.448102+0000", 2024-09-13T11:23:32.675 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.028965359 2024-09-13T11:23:32.675 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.675 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.675 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:23:32.675 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:30.448132+0000", 2024-09-13T11:23:32.675 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.0111999999999999e-05 2024-09-13T11:23:32.675 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.675 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.676 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:32.676 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:30.448255+0000", 2024-09-13T11:23:32.676 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00012329299999999999 2024-09-13T11:23:32.676 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.676 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.676 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:23:32.676 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:30.448269+0000", 2024-09-13T11:23:32.676 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.4117000000000001e-05 2024-09-13T11:23:32.676 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.676 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.676 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:23:32.676 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:30.449219+0000", 2024-09-13T11:23:32.676 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00095022499999999996 2024-09-13T11:23:32.677 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:32.677 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:32.677 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:32.677 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.677 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.677 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4665.0:54193 3.11 e74/70 3:8c8e9c38:::benchmark_data_smithi177_31116_object54192:head v 74'3051, mlcod=55'1358)", 2024-09-13T11:23:32.677 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:30.447991+0000", 2024-09-13T11:23:32.677 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 2.137354524, 2024-09-13T11:23:32.677 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.038008423, 2024-09-13T11:23:32.677 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:23:32.677 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:23:32.677 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:32.677 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:23:32.678 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.678 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:23:32.678 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:30.447991+0000", 2024-09-13T11:23:32.678 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:32.678 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.678 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.678 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:23:32.678 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:30.447991+0000", 2024-09-13T11:23:32.678 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:32.678 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.678 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.678 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:23:32.678 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:30.447994+0000", 2024-09-13T11:23:32.678 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.9799999999999998e-06 2024-09-13T11:23:32.678 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.679 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.679 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:23:32.679 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:30.448055+0000", 2024-09-13T11:23:32.679 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.0748000000000002e-05 2024-09-13T11:23:32.679 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.679 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.679 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:23:32.679 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:30.484156+0000", 2024-09-13T11:23:32.679 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.036101543999999999 2024-09-13T11:23:32.679 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.679 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.679 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:23:32.679 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:30.484167+0000", 2024-09-13T11:23:32.679 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0268000000000001e-05 2024-09-13T11:23:32.680 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.680 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.680 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:32.680 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:30.484243+0000", 2024-09-13T11:23:32.680 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.6532999999999994e-05 2024-09-13T11:23:32.680 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.680 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.680 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:23:32.680 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:30.484273+0000", 2024-09-13T11:23:32.680 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.0199999999999999e-05 2024-09-13T11:23:32.680 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.680 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.680 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:23:32.681 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:30.485943+0000", 2024-09-13T11:23:32.681 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0016701369999999999 2024-09-13T11:23:32.681 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.681 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.681 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:23:32.681 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:30.485999+0000", 2024-09-13T11:23:32.681 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.6013000000000001e-05 2024-09-13T11:23:32.681 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:32.681 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:32.681 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:32.681 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:32.681 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:32.681 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-13T11:23:32.682 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T11:23:32.683 INFO:tasks.ceph.osd.13.smithi092.stderr:2024-09-13T11:23:32.680+0000 7f3e5afea640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26350) UID: 0 2024-09-13T11:23:32.684 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-09-13T11:23:32.784 INFO:tasks.ceph.osd.4.smithi001.stderr:2024-09-13T11:23:32.780+0000 7ff8e0218640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26310) UID: 0 2024-09-13T11:23:32.853 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-13T11:23:32.854 INFO:teuthology.orchestra.run.smithi001.stdout: "ops": [ 2024-09-13T11:23:32.854 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.854 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:54760 3.1e 3.8c32c13e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-13T11:23:32.854 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:31.999685+0000", 2024-09-13T11:23:32.854 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 0.85254070299999996, 2024-09-13T11:23:32.854 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.85256046699999999, 2024-09-13T11:23:32.854 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:23:32.854 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:23:32.854 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "waiting for peered", 2024-09-13T11:23:32.854 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:23:32.854 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:23:32.854 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:32.855 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 54760 2024-09-13T11:23:32.855 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.855 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:23:32.855 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.855 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:23:32.855 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.999685+0000", 2024-09-13T11:23:32.855 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:32.855 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.855 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.855 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:23:32.855 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.999685+0000", 2024-09-13T11:23:32.855 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:32.855 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.855 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.856 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:23:32.856 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.999690+0000", 2024-09-13T11:23:32.856 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.7879999999999997e-06 2024-09-13T11:23:32.856 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.856 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.856 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:23:32.856 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.999729+0000", 2024-09-13T11:23:32.856 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.9032000000000002e-05 2024-09-13T11:23:32.856 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.856 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.856 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:23:32.856 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.999741+0000", 2024-09-13T11:23:32.856 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2136999999999999e-05 2024-09-13T11:23:32.856 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.857 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.857 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:23:32.857 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.999748+0000", 2024-09-13T11:23:32.857 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.9120000000000001e-06 2024-09-13T11:23:32.857 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.857 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.857 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:32.857 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.999803+0000", 2024-09-13T11:23:32.857 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.5064e-05 2024-09-13T11:23:32.857 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.857 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.857 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for peered", 2024-09-13T11:23:32.857 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.999815+0000", 2024-09-13T11:23:32.858 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2041e-05 2024-09-13T11:23:32.858 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:32.858 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:32.858 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:32.858 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.858 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.858 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:54768 3.1e 3.6dad499e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-13T11:23:32.858 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:32.003482+0000", 2024-09-13T11:23:32.858 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 0.848744053, 2024-09-13T11:23:32.858 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.84880534799999996, 2024-09-13T11:23:32.858 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:23:32.858 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:23:32.858 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "waiting for peered", 2024-09-13T11:23:32.858 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:23:32.859 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:23:32.859 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:32.859 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 54768 2024-09-13T11:23:32.859 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.859 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:23:32.859 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.859 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:23:32.859 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.003482+0000", 2024-09-13T11:23:32.859 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:32.859 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.859 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.859 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:23:32.859 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.003482+0000", 2024-09-13T11:23:32.860 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:32.860 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.860 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.860 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:23:32.860 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.003486+0000", 2024-09-13T11:23:32.860 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.1030000000000004e-06 2024-09-13T11:23:32.860 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.860 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.860 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:23:32.860 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.003523+0000", 2024-09-13T11:23:32.860 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.6732e-05 2024-09-13T11:23:32.860 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.860 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.860 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:23:32.861 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.003535+0000", 2024-09-13T11:23:32.861 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2520999999999999e-05 2024-09-13T11:23:32.861 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.861 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.861 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:23:32.861 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.003541+0000", 2024-09-13T11:23:32.861 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.8690000000000001e-06 2024-09-13T11:23:32.861 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.861 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.861 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:32.861 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.003593+0000", 2024-09-13T11:23:32.861 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.1786000000000002e-05 2024-09-13T11:23:32.861 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.861 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.862 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for peered", 2024-09-13T11:23:32.862 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.003602+0000", 2024-09-13T11:23:32.862 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.9679999999999995e-06 2024-09-13T11:23:32.862 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:32.862 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:32.862 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:32.862 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.862 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.862 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:54828 3.1e 3.e920c11e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-13T11:23:32.862 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:32.063344+0000", 2024-09-13T11:23:32.862 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 0.78888145700000001, 2024-09-13T11:23:32.862 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.78897489499999995, 2024-09-13T11:23:32.862 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:23:32.863 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:23:32.863 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "waiting for peered", 2024-09-13T11:23:32.863 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:23:32.863 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:23:32.863 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:32.863 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 54828 2024-09-13T11:23:32.863 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.863 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:23:32.863 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.863 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:23:32.863 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.063344+0000", 2024-09-13T11:23:32.863 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:32.863 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.863 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.863 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:23:32.864 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.063344+0000", 2024-09-13T11:23:32.864 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:32.864 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.864 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.864 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:23:32.864 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.063347+0000", 2024-09-13T11:23:32.864 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.2299999999999998e-06 2024-09-13T11:23:32.864 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.864 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.864 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:23:32.864 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.063384+0000", 2024-09-13T11:23:32.864 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.7290999999999999e-05 2024-09-13T11:23:32.864 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.864 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.865 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:23:32.865 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.063397+0000", 2024-09-13T11:23:32.865 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2955999999999999e-05 2024-09-13T11:23:32.865 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.865 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.865 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:23:32.865 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.063404+0000", 2024-09-13T11:23:32.865 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.4889999999999998e-06 2024-09-13T11:23:32.865 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.865 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.865 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:32.865 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.063470+0000", 2024-09-13T11:23:32.865 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.5516000000000001e-05 2024-09-13T11:23:32.865 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.866 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.866 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for peered", 2024-09-13T11:23:32.866 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.063482+0000", 2024-09-13T11:23:32.866 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2065000000000001e-05 2024-09-13T11:23:32.866 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:32.866 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:32.866 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:32.866 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.866 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.866 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:54829 3.1e 3.192f62be (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-13T11:23:32.866 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:32.065642+0000", 2024-09-13T11:23:32.866 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 0.78658438900000005, 2024-09-13T11:23:32.866 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.78670823700000003, 2024-09-13T11:23:32.866 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:23:32.867 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:23:32.867 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "waiting for peered", 2024-09-13T11:23:32.867 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:23:32.867 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:23:32.867 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:32.867 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 54829 2024-09-13T11:23:32.867 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.867 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:23:32.867 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.867 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:23:32.867 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.065642+0000", 2024-09-13T11:23:32.867 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:32.867 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.867 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.868 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:23:32.868 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.065642+0000", 2024-09-13T11:23:32.868 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:32.868 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.868 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.868 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:23:32.868 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.065644+0000", 2024-09-13T11:23:32.868 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.1780000000000002e-06 2024-09-13T11:23:32.868 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.868 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.869 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:23:32.869 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.065695+0000", 2024-09-13T11:23:32.869 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.1632000000000003e-05 2024-09-13T11:23:32.869 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.869 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.869 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:23:32.869 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.065711+0000", 2024-09-13T11:23:32.869 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.5522000000000001e-05 2024-09-13T11:23:32.869 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.869 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.869 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:23:32.869 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.065719+0000", 2024-09-13T11:23:32.869 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.3480000000000004e-06 2024-09-13T11:23:32.870 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.870 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.870 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:32.870 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.065785+0000", 2024-09-13T11:23:32.870 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.5627e-05 2024-09-13T11:23:32.870 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.870 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.870 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for peered", 2024-09-13T11:23:32.870 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.065797+0000", 2024-09-13T11:23:32.870 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.1766000000000001e-05 2024-09-13T11:23:32.870 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:32.870 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:32.870 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:32.870 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.871 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.871 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:54830 3.1e 3.b31554de (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-13T11:23:32.871 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:32.068259+0000", 2024-09-13T11:23:32.871 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 0.78396652099999997, 2024-09-13T11:23:32.871 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.78412130599999996, 2024-09-13T11:23:32.871 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:23:32.871 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:23:32.871 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "waiting for peered", 2024-09-13T11:23:32.871 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:23:32.871 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:23:32.871 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:32.871 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 54830 2024-09-13T11:23:32.871 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.871 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:23:32.872 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.872 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:23:32.872 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.068259+0000", 2024-09-13T11:23:32.872 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:32.872 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.872 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.872 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:23:32.872 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.068259+0000", 2024-09-13T11:23:32.872 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:32.872 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.872 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.872 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:23:32.872 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.068262+0000", 2024-09-13T11:23:32.872 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.1799999999999999e-06 2024-09-13T11:23:32.873 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.873 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.873 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:23:32.873 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.068301+0000", 2024-09-13T11:23:32.873 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.9289999999999998e-05 2024-09-13T11:23:32.873 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.873 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.873 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:23:32.873 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.068314+0000", 2024-09-13T11:23:32.873 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.3502999999999999e-05 2024-09-13T11:23:32.873 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.873 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.873 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:23:32.874 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.068324+0000", 2024-09-13T11:23:32.874 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.1630000000000002e-06 2024-09-13T11:23:32.874 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.874 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.874 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:32.874 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.068375+0000", 2024-09-13T11:23:32.874 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.1805000000000003e-05 2024-09-13T11:23:32.874 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.874 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.874 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for peered", 2024-09-13T11:23:32.874 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.068382+0000", 2024-09-13T11:23:32.874 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.224e-06 2024-09-13T11:23:32.874 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:32.874 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:32.875 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:32.875 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.875 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.875 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:54785 3.1e 3.d32f1ede (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-13T11:23:32.875 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:32.016238+0000", 2024-09-13T11:23:32.875 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 0.83598784299999995, 2024-09-13T11:23:32.875 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.83617277999999995, 2024-09-13T11:23:32.875 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:23:32.875 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:23:32.875 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "waiting for peered", 2024-09-13T11:23:32.875 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:23:32.875 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:23:32.875 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:32.875 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 54785 2024-09-13T11:23:32.876 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.876 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:23:32.876 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.876 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:23:32.876 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.016238+0000", 2024-09-13T11:23:32.876 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:32.876 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.876 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.876 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:23:32.876 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.016238+0000", 2024-09-13T11:23:32.876 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:32.876 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.876 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.877 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:23:32.877 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.016241+0000", 2024-09-13T11:23:32.877 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.6469999999999998e-06 2024-09-13T11:23:32.877 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.877 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.877 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:23:32.877 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.016271+0000", 2024-09-13T11:23:32.877 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.0505999999999999e-05 2024-09-13T11:23:32.877 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.877 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.877 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:23:32.877 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.016287+0000", 2024-09-13T11:23:32.877 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.6073999999999999e-05 2024-09-13T11:23:32.877 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.878 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.878 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:23:32.878 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.016295+0000", 2024-09-13T11:23:32.878 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.0590000000000003e-06 2024-09-13T11:23:32.878 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.878 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.878 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:32.878 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.016373+0000", 2024-09-13T11:23:32.878 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.7634999999999998e-05 2024-09-13T11:23:32.878 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.878 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.878 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for peered", 2024-09-13T11:23:32.878 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.016383+0000", 2024-09-13T11:23:32.879 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0246e-05 2024-09-13T11:23:32.879 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:32.879 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:32.879 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:32.879 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.879 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.879 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:54888 3.1e 3.43e14ede (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-13T11:23:32.879 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:32.423064+0000", 2024-09-13T11:23:32.879 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 0.42916162600000002, 2024-09-13T11:23:32.879 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.42937635699999999, 2024-09-13T11:23:32.879 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:23:32.879 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:23:32.879 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "waiting for peered", 2024-09-13T11:23:32.879 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:23:32.880 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:23:32.880 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:32.880 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 54888 2024-09-13T11:23:32.880 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.880 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:23:32.880 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.880 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:23:32.880 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.423064+0000", 2024-09-13T11:23:32.880 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:32.880 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.880 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.880 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:23:32.881 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.423064+0000", 2024-09-13T11:23:32.881 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:32.881 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.881 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.881 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:23:32.881 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.423069+0000", 2024-09-13T11:23:32.881 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.7389999999999999e-06 2024-09-13T11:23:32.881 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.881 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.881 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:23:32.881 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.423115+0000", 2024-09-13T11:23:32.881 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.5847000000000003e-05 2024-09-13T11:23:32.881 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.881 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.882 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:23:32.882 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.423126+0000", 2024-09-13T11:23:32.882 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.1418e-05 2024-09-13T11:23:32.882 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.882 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.882 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:23:32.882 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.423132+0000", 2024-09-13T11:23:32.882 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.57e-06 2024-09-13T11:23:32.882 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.882 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.882 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:32.882 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.423191+0000", 2024-09-13T11:23:32.882 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.9250999999999999e-05 2024-09-13T11:23:32.882 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.882 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.883 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for peered", 2024-09-13T11:23:32.883 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.423198+0000", 2024-09-13T11:23:32.883 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.9179999999999997e-06 2024-09-13T11:23:32.883 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:32.883 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:32.883 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:32.883 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.883 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.883 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:54846 3.1e 3.742e049e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-13T11:23:32.883 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:32.153567+0000", 2024-09-13T11:23:32.883 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 0.69865929699999996, 2024-09-13T11:23:32.883 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.69890558899999999, 2024-09-13T11:23:32.883 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:23:32.883 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:23:32.883 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "waiting for peered", 2024-09-13T11:23:32.884 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:23:32.884 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:23:32.884 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:32.884 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 54846 2024-09-13T11:23:32.884 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.884 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:23:32.884 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.884 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:23:32.884 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.153567+0000", 2024-09-13T11:23:32.884 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:32.884 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.884 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.884 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:23:32.885 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.153567+0000", 2024-09-13T11:23:32.885 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:32.885 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.885 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.885 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:23:32.885 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.153569+0000", 2024-09-13T11:23:32.885 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.4660000000000002e-06 2024-09-13T11:23:32.885 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.885 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.885 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:23:32.885 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.153619+0000", 2024-09-13T11:23:32.885 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.9617000000000003e-05 2024-09-13T11:23:32.885 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.885 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.886 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:23:32.886 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.153634+0000", 2024-09-13T11:23:32.886 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.4904e-05 2024-09-13T11:23:32.886 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.886 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.886 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:23:32.886 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.153642+0000", 2024-09-13T11:23:32.886 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.5599999999999994e-06 2024-09-13T11:23:32.886 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.886 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.886 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:32.886 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.153721+0000", 2024-09-13T11:23:32.886 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.8603999999999995e-05 2024-09-13T11:23:32.886 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.887 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.887 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for peered", 2024-09-13T11:23:32.887 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.153730+0000", 2024-09-13T11:23:32.887 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.3840000000000003e-06 2024-09-13T11:23:32.887 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:32.887 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:32.887 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:32.887 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.887 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.887 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:54935 3.1e 3.42cf919e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-13T11:23:32.887 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:32.720640+0000", 2024-09-13T11:23:32.887 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 0.131585696, 2024-09-13T11:23:32.887 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.13186166799999999, 2024-09-13T11:23:32.887 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:23:32.888 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:23:32.888 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "waiting for peered", 2024-09-13T11:23:32.888 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:23:32.888 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:23:32.888 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:32.888 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 54935 2024-09-13T11:23:32.888 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.888 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:23:32.888 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.888 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:23:32.888 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.720640+0000", 2024-09-13T11:23:32.888 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:32.888 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.889 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.889 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:23:32.889 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.720640+0000", 2024-09-13T11:23:32.889 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:32.889 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.889 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.889 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:23:32.889 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.720644+0000", 2024-09-13T11:23:32.889 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.9530000000000001e-06 2024-09-13T11:23:32.889 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.889 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.889 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:23:32.889 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.720711+0000", 2024-09-13T11:23:32.889 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.7117000000000001e-05 2024-09-13T11:23:32.890 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.890 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.890 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:23:32.890 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.720746+0000", 2024-09-13T11:23:32.890 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.4829999999999997e-05 2024-09-13T11:23:32.890 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.890 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.890 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:23:32.890 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.720765+0000", 2024-09-13T11:23:32.890 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.9168999999999999e-05 2024-09-13T11:23:32.890 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.890 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.890 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:32.890 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.720854+0000", 2024-09-13T11:23:32.891 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.8257999999999994e-05 2024-09-13T11:23:32.891 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.891 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.891 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for peered", 2024-09-13T11:23:32.891 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.720865+0000", 2024-09-13T11:23:32.891 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.1858999999999999e-05 2024-09-13T11:23:32.891 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:32.891 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:32.891 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:32.891 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.891 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.891 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:54720 3.1e 3.a944303e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-13T11:23:32.891 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:31.973315+0000", 2024-09-13T11:23:32.891 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 0.87891073799999997, 2024-09-13T11:23:32.892 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.879216416, 2024-09-13T11:23:32.892 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:23:32.892 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:23:32.892 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "waiting for peered", 2024-09-13T11:23:32.892 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:23:32.892 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:23:32.892 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:32.892 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 54720 2024-09-13T11:23:32.892 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.892 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:23:32.892 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.892 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:23:32.892 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.973315+0000", 2024-09-13T11:23:32.893 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:32.893 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.893 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.893 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:23:32.893 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.973315+0000", 2024-09-13T11:23:32.893 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:32.893 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.893 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.893 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:23:32.893 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.973318+0000", 2024-09-13T11:23:32.893 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.5830000000000001e-06 2024-09-13T11:23:32.893 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.893 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.893 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:23:32.894 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.973465+0000", 2024-09-13T11:23:32.894 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000146816 2024-09-13T11:23:32.894 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.894 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.894 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:23:32.894 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.973478+0000", 2024-09-13T11:23:32.894 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2931e-05 2024-09-13T11:23:32.894 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.894 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.894 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:23:32.894 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.973483+0000", 2024-09-13T11:23:32.894 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.8969999999999996e-06 2024-09-13T11:23:32.894 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.894 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.894 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:32.895 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.973548+0000", 2024-09-13T11:23:32.895 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.4771999999999996e-05 2024-09-13T11:23:32.895 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.895 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.895 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for peered", 2024-09-13T11:23:32.895 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.973557+0000", 2024-09-13T11:23:32.895 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.0359999999999995e-06 2024-09-13T11:23:32.895 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:32.895 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:32.895 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:32.895 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.895 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.895 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:54811 3.1e 3.3754367e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-13T11:23:32.895 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:32.045424+0000", 2024-09-13T11:23:32.896 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 0.80680219200000003, 2024-09-13T11:23:32.896 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.80713807500000001, 2024-09-13T11:23:32.896 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:23:32.896 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:23:32.896 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "waiting for peered", 2024-09-13T11:23:32.896 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:23:32.896 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:23:32.896 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:32.896 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 54811 2024-09-13T11:23:32.896 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.896 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:23:32.896 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.896 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:23:32.896 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.045424+0000", 2024-09-13T11:23:32.897 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:32.897 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.897 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.897 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:23:32.897 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.045424+0000", 2024-09-13T11:23:32.897 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:32.897 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.897 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.897 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:23:32.897 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.045426+0000", 2024-09-13T11:23:32.897 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.0260000000000002e-06 2024-09-13T11:23:32.897 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.897 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.898 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:23:32.898 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.045556+0000", 2024-09-13T11:23:32.898 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00013003099999999999 2024-09-13T11:23:32.898 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.898 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.898 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:23:32.898 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.045568+0000", 2024-09-13T11:23:32.898 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.1727e-05 2024-09-13T11:23:32.898 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.898 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.898 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:23:32.898 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.045576+0000", 2024-09-13T11:23:32.899 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.6459999999999999e-06 2024-09-13T11:23:32.899 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.899 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.899 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:32.899 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.045650+0000", 2024-09-13T11:23:32.899 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.4000999999999998e-05 2024-09-13T11:23:32.899 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.899 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.900 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for peered", 2024-09-13T11:23:32.900 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.045658+0000", 2024-09-13T11:23:32.900 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.3869999999999997e-06 2024-09-13T11:23:32.900 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:32.900 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:32.900 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:32.900 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.900 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.900 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:54734 3.1e 3.d7f6f7e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-13T11:23:32.900 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:31.983342+0000", 2024-09-13T11:23:32.900 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 0.86888393500000005, 2024-09-13T11:23:32.900 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.86924988400000003, 2024-09-13T11:23:32.900 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:23:32.901 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:23:32.901 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "waiting for peered", 2024-09-13T11:23:32.901 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:23:32.901 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:23:32.901 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:32.901 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 54734 2024-09-13T11:23:32.901 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.901 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:23:32.901 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.901 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:23:32.901 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.983342+0000", 2024-09-13T11:23:32.901 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:32.901 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.902 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.902 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:23:32.902 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.983342+0000", 2024-09-13T11:23:32.902 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:32.902 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.902 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.902 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:23:32.902 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.983345+0000", 2024-09-13T11:23:32.902 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.6220000000000001e-06 2024-09-13T11:23:32.902 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.902 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.902 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:23:32.902 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.983438+0000", 2024-09-13T11:23:32.902 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.3517000000000006e-05 2024-09-13T11:23:32.903 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.903 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.903 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:23:32.903 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.983451+0000", 2024-09-13T11:23:32.903 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2527999999999999e-05 2024-09-13T11:23:32.903 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.903 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.903 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:23:32.903 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.983458+0000", 2024-09-13T11:23:32.903 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.6380000000000004e-06 2024-09-13T11:23:32.903 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.903 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.903 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:32.904 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.983521+0000", 2024-09-13T11:23:32.904 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.2729000000000002e-05 2024-09-13T11:23:32.904 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.904 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.904 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for peered", 2024-09-13T11:23:32.904 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.983548+0000", 2024-09-13T11:23:32.904 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.6537999999999999e-05 2024-09-13T11:23:32.904 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:32.904 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:32.904 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:32.904 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.904 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.904 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:54817 3.1e 3.36dd2fde (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-13T11:23:32.905 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:32.050108+0000", 2024-09-13T11:23:32.905 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 0.80211829999999995, 2024-09-13T11:23:32.905 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.80251393699999996, 2024-09-13T11:23:32.905 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:23:32.905 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:23:32.905 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "waiting for peered", 2024-09-13T11:23:32.905 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:23:32.905 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:23:32.905 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:32.905 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 54817 2024-09-13T11:23:32.905 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.905 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:23:32.905 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.905 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:23:32.906 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.050108+0000", 2024-09-13T11:23:32.906 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:32.906 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.906 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.906 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:23:32.906 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.050108+0000", 2024-09-13T11:23:32.906 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:32.906 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.906 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.906 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:23:32.906 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.050110+0000", 2024-09-13T11:23:32.906 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.0209999999999998e-06 2024-09-13T11:23:32.906 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.907 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.907 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:23:32.907 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.050166+0000", 2024-09-13T11:23:32.907 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.6026000000000002e-05 2024-09-13T11:23:32.907 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.907 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.907 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:23:32.907 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.050180+0000", 2024-09-13T11:23:32.907 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.4307e-05 2024-09-13T11:23:32.907 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.907 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.907 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:23:32.907 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.050188+0000", 2024-09-13T11:23:32.907 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.5410000000000003e-06 2024-09-13T11:23:32.907 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.908 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.908 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:32.908 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.050260+0000", 2024-09-13T11:23:32.908 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.2157999999999996e-05 2024-09-13T11:23:32.908 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.908 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.908 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for peered", 2024-09-13T11:23:32.908 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.050267+0000", 2024-09-13T11:23:32.908 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.0190000000000001e-06 2024-09-13T11:23:32.908 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:32.908 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:32.908 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:32.908 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.908 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.909 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:54656 3.1e 3.6a49163e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e75)", 2024-09-13T11:23:32.909 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:30.979993+0000", 2024-09-13T11:23:32.909 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 1.872232973, 2024-09-13T11:23:32.909 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.8726584470000001, 2024-09-13T11:23:32.909 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:23:32.909 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:23:32.909 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "waiting for peered", 2024-09-13T11:23:32.909 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:23:32.909 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:23:32.909 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:32.909 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 54656 2024-09-13T11:23:32.909 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.909 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:23:32.909 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.910 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:23:32.910 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:30.979993+0000", 2024-09-13T11:23:32.910 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:32.910 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.910 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.910 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:23:32.910 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:30.979993+0000", 2024-09-13T11:23:32.910 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:32.910 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.910 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.910 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:23:32.910 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:30.979997+0000", 2024-09-13T11:23:32.910 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.0319999999999997e-06 2024-09-13T11:23:32.910 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.911 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.911 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:23:32.911 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:30.980057+0000", 2024-09-13T11:23:32.911 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.0341999999999999e-05 2024-09-13T11:23:32.911 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.911 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.911 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:23:32.911 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:30.980094+0000", 2024-09-13T11:23:32.911 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.6770000000000002e-05 2024-09-13T11:23:32.911 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.911 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.911 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:23:32.911 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:30.980113+0000", 2024-09-13T11:23:32.911 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.8791999999999999e-05 2024-09-13T11:23:32.912 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.912 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.912 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:32.912 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.097773+0000", 2024-09-13T11:23:32.912 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.117660379 2024-09-13T11:23:32.912 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.912 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.912 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for peered", 2024-09-13T11:23:32.912 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.097784+0000", 2024-09-13T11:23:32.912 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.1025e-05 2024-09-13T11:23:32.912 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:32.912 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:32.912 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:32.913 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.913 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.913 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:54818 3.1e 3.5cfcc69e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-13T11:23:32.913 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:32.051374+0000", 2024-09-13T11:23:32.913 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 0.80085188399999996, 2024-09-13T11:23:32.913 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.80131083800000003, 2024-09-13T11:23:32.913 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:23:32.913 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:23:32.913 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "waiting for peered", 2024-09-13T11:23:32.913 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:23:32.913 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:23:32.913 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:32.913 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 54818 2024-09-13T11:23:32.913 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.913 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:23:32.914 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.914 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:23:32.914 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.051374+0000", 2024-09-13T11:23:32.914 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:32.914 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.914 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.914 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:23:32.914 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.051374+0000", 2024-09-13T11:23:32.914 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:32.914 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.914 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.914 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:23:32.914 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.051375+0000", 2024-09-13T11:23:32.914 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.4440000000000001e-06 2024-09-13T11:23:32.915 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.915 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.915 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:23:32.915 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.051426+0000", 2024-09-13T11:23:32.915 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.0809000000000001e-05 2024-09-13T11:23:32.915 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.915 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.915 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:23:32.915 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.051441+0000", 2024-09-13T11:23:32.915 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.4460999999999999e-05 2024-09-13T11:23:32.915 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.915 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.915 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:23:32.916 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.051449+0000", 2024-09-13T11:23:32.916 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.4880000000000007e-06 2024-09-13T11:23:32.916 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.916 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.916 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:32.916 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.051499+0000", 2024-09-13T11:23:32.916 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.9691999999999998e-05 2024-09-13T11:23:32.916 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:32.916 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:32.916 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for peered", 2024-09-13T11:23:32.916 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.051505+0000", 2024-09-13T11:23:32.916 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.2029999999999997e-06 2024-09-13T11:23:32.916 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:32.916 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:32.917 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:32.917 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:32.917 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-09-13T11:23:32.917 INFO:teuthology.orchestra.run.smithi001.stdout: "num_ops": 15 2024-09-13T11:23:32.917 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-13T11:23:32.918 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-09-13T11:23:32.919 INFO:teuthology.orchestra.run.smithi001.stdout:osd.0: {} 2024-09-13T11:23:32.919 INFO:teuthology.orchestra.run.smithi001.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T11:23:32.919 INFO:teuthology.orchestra.run.smithi001.stdout:osd.1: {} 2024-09-13T11:23:32.920 INFO:teuthology.orchestra.run.smithi001.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T11:23:32.920 INFO:tasks.ceph.osd.10.smithi162.stderr:2024-09-13T11:23:32.913+0000 7fb59c9f0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26317) UID: 0 2024-09-13T11:23:32.921 INFO:tasks.ceph.osd.9.smithi092.stderr:2024-09-13T11:23:32.920+0000 7ff59bdf5640 -1 osd.9 76 handle_osd_map injecting map crc failure 2024-09-13T11:23:32.924 INFO:tasks.ceph.osd.9.smithi092.stderr:2024-09-13T11:23:32.920+0000 7ff59bdf5640 -1 osd.9 76 handle_osd_map injecting map crc failure 2024-09-13T11:23:32.937 INFO:teuthology.orchestra.run.smithi001.stdout:osd.2: {} 2024-09-13T11:23:32.938 INFO:teuthology.orchestra.run.smithi001.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T11:23:32.975 INFO:teuthology.orchestra.run.smithi001.stdout:osd.3: {} 2024-09-13T11:23:32.975 INFO:teuthology.orchestra.run.smithi001.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T11:23:33.001 INFO:teuthology.orchestra.run.smithi001.stdout:osd.4: {} 2024-09-13T11:23:33.001 INFO:teuthology.orchestra.run.smithi001.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T11:23:33.004 INFO:tasks.radosbench.radosbench.0.smithi177.stdout: 66 16 54935 54919 52.001 10.4375 0.00309593 0.0189571 2024-09-13T11:23:33.019 INFO:tasks.ceph.osd.12.smithi001.stderr:2024-09-13T11:23:33.012+0000 7f892c6e6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26314) UID: 0 2024-09-13T11:23:33.026 INFO:teuthology.orchestra.run.smithi001.stdout:osd.5: {} 2024-09-13T11:23:33.026 INFO:teuthology.orchestra.run.smithi001.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T11:23:33.051 INFO:teuthology.orchestra.run.smithi001.stdout:osd.6: {} 2024-09-13T11:23:33.051 INFO:teuthology.orchestra.run.smithi001.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T11:23:33.085 INFO:teuthology.orchestra.run.smithi001.stdout:osd.7: {} 2024-09-13T11:23:33.085 INFO:teuthology.orchestra.run.smithi001.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T11:23:33.089 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-13T11:23:33.089 INFO:teuthology.orchestra.run.smithi001.stdout: "ops": [], 2024-09-13T11:23:33.089 INFO:teuthology.orchestra.run.smithi001.stdout: "complaint_time": 30, 2024-09-13T11:23:33.089 INFO:teuthology.orchestra.run.smithi001.stdout: "num_blocked_ops": 0 2024-09-13T11:23:33.089 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-13T11:23:33.106 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-09-13T11:23:33.107 INFO:teuthology.orchestra.run.smithi001.stdout:osd.8: {} 2024-09-13T11:23:33.108 INFO:teuthology.orchestra.run.smithi001.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-13T11:23:33.120 INFO:tasks.ceph.osd.3.smithi177.stderr:2024-09-13T11:23:33.117+0000 7f10adba4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26574) UID: 0 2024-09-13T11:23:33.135 INFO:teuthology.orchestra.run.smithi001.stdout:osd.9: {} 2024-09-13T11:23:33.136 INFO:teuthology.orchestra.run.smithi001.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-13T11:23:33.182 INFO:teuthology.orchestra.run.smithi001.stdout:osd.10: {} 2024-09-13T11:23:33.182 INFO:teuthology.orchestra.run.smithi001.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-13T11:23:33.205 INFO:teuthology.orchestra.run.smithi001.stdout:osd.11: {} 2024-09-13T11:23:33.206 INFO:teuthology.orchestra.run.smithi001.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-13T11:23:33.222 INFO:tasks.ceph.osd.10.smithi162.stderr:2024-09-13T11:23:33.213+0000 7fb59c9f0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26317) UID: 0 2024-09-13T11:23:33.241 INFO:teuthology.orchestra.run.smithi001.stdout:osd.12: {} 2024-09-13T11:23:33.241 INFO:teuthology.orchestra.run.smithi001.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-13T11:23:33.251 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-13T11:23:33.251 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:23:33.268 INFO:teuthology.orchestra.run.smithi001.stdout:osd.13: {} 2024-09-13T11:23:33.269 INFO:teuthology.orchestra.run.smithi001.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-13T11:23:33.270 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_ops_in_flight 2024-09-13T11:23:33.294 INFO:teuthology.orchestra.run.smithi001.stdout:osd.14: {} 2024-09-13T11:23:33.294 INFO:teuthology.orchestra.run.smithi001.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-13T11:23:33.319 INFO:teuthology.orchestra.run.smithi001.stdout:osd.15: {} 2024-09-13T11:23:33.319 INFO:teuthology.orchestra.run.smithi001.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-13T11:23:33.321 INFO:tasks.ceph.osd.3.smithi177.stderr:2024-09-13T11:23:33.317+0000 7f10adba4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26574) UID: 0 2024-09-13T11:23:33.423 INFO:tasks.ceph.osd.13.smithi092.stderr:2024-09-13T11:23:33.420+0000 7f3e5afea640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26350) UID: 0 2024-09-13T11:23:33.437 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-13T11:23:33.437 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:23:33.454 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_blocked_ops 2024-09-13T11:23:33.523 INFO:tasks.ceph.osd.10.smithi162.stderr:2024-09-13T11:23:33.517+0000 7fb59c9f0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26317) UID: 0 2024-09-13T11:23:33.590 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-13T11:23:33.590 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:23:33.607 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_historic_ops 2024-09-13T11:23:33.624 INFO:tasks.ceph.osd.14.smithi162.stderr:2024-09-13T11:23:33.617+0000 7f235c0db640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26319) UID: 0 2024-09-13T11:23:33.725 INFO:tasks.ceph.osd.2.smithi162.stderr:2024-09-13T11:23:33.717+0000 7fe92b35a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26313) UID: 0 2024-09-13T11:23:33.747 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-13T11:23:33.747 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:23:33.763 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-13T11:23:33.826 INFO:tasks.ceph.osd.7.smithi177.stderr:2024-09-13T11:23:33.821+0000 7f897b51d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26575) UID: 0 2024-09-13T11:23:33.912 INFO:teuthology.orchestra.run.smithi092.stdout:ERROR: (22) Invalid argument 2024-09-13T11:23:33.912 INFO:teuthology.orchestra.run.smithi092.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:23:33.912 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-13T11:23:33.904+0000 7f6a5f1a6640 -1 osd.0 77 handle_osd_map injecting map crc failure 2024-09-13T11:23:33.914 INFO:tasks.ceph.osd.15.smithi177.stderr:2024-09-13T11:23:33.909+0000 7f9d618a4640 -1 osd.15 77 handle_osd_map injecting map crc failure 2024-09-13T11:23:33.928 INFO:tasks.ceph.osd.13.smithi092.stderr:2024-09-13T11:23:33.924+0000 7f3e5afea640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26350) UID: 0 2024-09-13T11:23:33.936 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-13T11:23:34.004 INFO:tasks.radosbench.radosbench.0.smithi177.stdout: 67 16 54940 54924 51.2296 0.3125 0.0104147 0.0189631 2024-09-13T11:23:34.014 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15] out_osds: [] dead_osds: [] live_osds: [0, 4, 8, 12, 5, 9, 13, 2, 6, 10, 14, 3, 7, 11, 15, 1] 2024-09-13T11:23:34.014 INFO:tasks.thrashosds.thrasher:choose_action: min_in 2 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-13T11:23:34.014 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-09-13T11:23:34.014 INFO:tasks.thrashosds.thrasher:Setting osd 5 primary_affinity to 1.000000 2024-09-13T11:23:34.014 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 5 1 2024-09-13T11:23:34.022 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-13T11:23:34.028 INFO:tasks.ceph.osd.2.smithi162.stderr:2024-09-13T11:23:34.021+0000 7fe92b35a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26313) UID: 0 2024-09-13T11:23:34.127 INFO:teuthology.orchestra.run.smithi092.stdout:ERROR: (22) Invalid argument 2024-09-13T11:23:34.127 INFO:teuthology.orchestra.run.smithi092.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:23:34.129 INFO:tasks.ceph.osd.11.smithi177.stderr:2024-09-13T11:23:34.125+0000 7f24792c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26580) UID: 0 2024-09-13T11:23:34.145 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-13T11:23:34.229 INFO:tasks.ceph.osd.1.smithi092.stderr:2024-09-13T11:23:34.228+0000 7fe7789c5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33718) UID: 0 2024-09-13T11:23:34.281 INFO:teuthology.orchestra.run.smithi092.stdout:ERROR: (22) Invalid argument 2024-09-13T11:23:34.281 INFO:teuthology.orchestra.run.smithi092.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:23:34.301 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-09-13T11:23:34.345 INFO:tasks.ceph.osd.1.smithi092.stderr:2024-09-13T11:23:34.344+0000 7fe7789c5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33718) UID: 0 2024-09-13T11:23:34.431 INFO:tasks.ceph.osd.15.smithi177.stderr:2024-09-13T11:23:34.425+0000 7f9d6ce77640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26578) UID: 0 2024-09-13T11:23:34.486 INFO:teuthology.orchestra.run.smithi092.stdout:ERROR: (22) Invalid argument 2024-09-13T11:23:34.487 INFO:teuthology.orchestra.run.smithi092.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:23:34.505 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-09-13T11:23:34.532 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-09-13T11:23:34.525+0000 7f97befc9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26315) UID: 0 2024-09-13T11:23:34.633 INFO:tasks.ceph.osd.2.smithi162.stderr:2024-09-13T11:23:34.625+0000 7fe92b35a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26313) UID: 0 2024-09-13T11:23:34.653 INFO:teuthology.orchestra.run.smithi092.stdout:ERROR: (22) Invalid argument 2024-09-13T11:23:34.653 INFO:teuthology.orchestra.run.smithi092.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:23:34.674 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-09-13T11:23:34.734 INFO:tasks.ceph.osd.5.smithi092.stderr:2024-09-13T11:23:34.732+0000 7f6f0b705640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26346) UID: 0 2024-09-13T11:23:34.814 INFO:teuthology.orchestra.run.smithi092.stdout:ERROR: (22) Invalid argument 2024-09-13T11:23:34.814 INFO:teuthology.orchestra.run.smithi092.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:23:34.832 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_ops_in_flight 2024-09-13T11:23:34.835 INFO:tasks.ceph.osd.12.smithi001.stderr:2024-09-13T11:23:34.828+0000 7f892c6e6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26314) UID: 0 2024-09-13T11:23:34.909 INFO:teuthology.orchestra.run.smithi001.stderr:set osd.5 primary-affinity to 1 (8655362) 2024-09-13T11:23:34.909 INFO:teuthology.orchestra.run.smithi001.stderr:nodeep-scrub is unset 2024-09-13T11:23:34.923 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-09-13T11:23:34.917+0000 7f97b39f6640 -1 osd.6 78 handle_osd_map injecting map crc failure 2024-09-13T11:23:34.931 INFO:tasks.ceph.osd.13.smithi092.stderr:2024-09-13T11:23:34.928+0000 7f3e4ba0f640 -1 osd.13 78 handle_osd_map injecting map crc failure 2024-09-13T11:23:34.937 INFO:tasks.ceph.osd.7.smithi177.stderr:2024-09-13T11:23:34.933+0000 7f897b51d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26575) UID: 0 2024-09-13T11:23:35.004 INFO:tasks.radosbench.radosbench.0.smithi177.stdout: 68 16 54940 54924 50.4762 0 - 0.0189631 2024-09-13T11:23:35.040 INFO:tasks.ceph.osd.15.smithi177.stderr:2024-09-13T11:23:35.033+0000 7f9d6ce77640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26578) UID: 0 2024-09-13T11:23:35.049 INFO:teuthology.orchestra.run.smithi092.stdout:ERROR: (22) Invalid argument 2024-09-13T11:23:35.050 INFO:teuthology.orchestra.run.smithi092.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:23:35.070 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_blocked_ops 2024-09-13T11:23:35.138 INFO:tasks.ceph.osd.5.smithi092.stderr:2024-09-13T11:23:35.136+0000 7f6f0b705640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26346) UID: 0 2024-09-13T11:23:35.239 INFO:tasks.ceph.osd.2.smithi162.stderr:2024-09-13T11:23:35.233+0000 7fe92b35a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26313) UID: 0 2024-09-13T11:23:35.253 INFO:teuthology.orchestra.run.smithi092.stdout:ERROR: (22) Invalid argument 2024-09-13T11:23:35.253 INFO:teuthology.orchestra.run.smithi092.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:23:35.278 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_historic_ops 2024-09-13T11:23:35.340 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-09-13T11:23:35.333+0000 7f97befc9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26315) UID: 0 2024-09-13T11:23:35.341 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T11:23:35.441 INFO:tasks.ceph.osd.7.smithi177.stderr:2024-09-13T11:23:35.436+0000 7f897b51d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26575) UID: 0 2024-09-13T11:23:35.479 INFO:teuthology.orchestra.run.smithi092.stdout:ERROR: (22) Invalid argument 2024-09-13T11:23:35.479 INFO:teuthology.orchestra.run.smithi092.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:23:35.501 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-13T11:23:35.542 INFO:tasks.ceph.osd.5.smithi092.stderr:2024-09-13T11:23:35.540+0000 7f6f0b705640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26346) UID: 0 2024-09-13T11:23:35.550 INFO:teuthology.orchestra.run.smithi001.stdout:osd.0: {} 2024-09-13T11:23:35.550 INFO:teuthology.orchestra.run.smithi001.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T11:23:35.586 INFO:teuthology.orchestra.run.smithi001.stdout:osd.1: {} 2024-09-13T11:23:35.586 INFO:teuthology.orchestra.run.smithi001.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T11:23:35.617 INFO:teuthology.orchestra.run.smithi001.stdout:osd.2: {} 2024-09-13T11:23:35.618 INFO:teuthology.orchestra.run.smithi001.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T11:23:35.642 INFO:tasks.ceph.osd.8.smithi001.stderr:2024-09-13T11:23:35.636+0000 7fb9b48ae640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26312) UID: 0 2024-09-13T11:23:35.648 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-13T11:23:35.648 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [], 2024-09-13T11:23:35.648 INFO:teuthology.orchestra.run.smithi162.stdout: "num_ops": 0 2024-09-13T11:23:35.648 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-13T11:23:35.654 INFO:teuthology.orchestra.run.smithi001.stdout:osd.3: {} 2024-09-13T11:23:35.654 INFO:teuthology.orchestra.run.smithi001.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T11:23:35.668 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-13T11:23:35.687 INFO:teuthology.orchestra.run.smithi001.stdout:osd.4: {} 2024-09-13T11:23:35.687 INFO:teuthology.orchestra.run.smithi001.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T11:23:35.718 INFO:teuthology.orchestra.run.smithi001.stdout:osd.5: {} 2024-09-13T11:23:35.718 INFO:teuthology.orchestra.run.smithi001.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T11:23:35.744 INFO:tasks.ceph.osd.3.smithi177.stderr:2024-09-13T11:23:35.740+0000 7f10adba4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26574) UID: 0 2024-09-13T11:23:35.752 INFO:teuthology.orchestra.run.smithi001.stdout:osd.6: {} 2024-09-13T11:23:35.752 INFO:teuthology.orchestra.run.smithi001.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T11:23:35.784 INFO:teuthology.orchestra.run.smithi001.stdout:osd.7: {} 2024-09-13T11:23:35.785 INFO:teuthology.orchestra.run.smithi001.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T11:23:35.806 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-13T11:23:35.806 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [], 2024-09-13T11:23:35.806 INFO:teuthology.orchestra.run.smithi162.stdout: "complaint_time": 30, 2024-09-13T11:23:35.806 INFO:teuthology.orchestra.run.smithi162.stdout: "num_blocked_ops": 0 2024-09-13T11:23:35.806 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-13T11:23:35.814 INFO:teuthology.orchestra.run.smithi001.stdout:osd.8: {} 2024-09-13T11:23:35.814 INFO:teuthology.orchestra.run.smithi001.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-13T11:23:35.824 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-13T11:23:35.843 INFO:teuthology.orchestra.run.smithi001.stdout:osd.9: {} 2024-09-13T11:23:35.843 INFO:teuthology.orchestra.run.smithi001.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-13T11:23:35.844 INFO:tasks.ceph.osd.4.smithi001.stderr:2024-09-13T11:23:35.836+0000 7ff8e0218640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26310) UID: 0 2024-09-13T11:23:35.872 INFO:teuthology.orchestra.run.smithi001.stdout:osd.10: {} 2024-09-13T11:23:35.872 INFO:teuthology.orchestra.run.smithi001.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-13T11:23:35.902 INFO:teuthology.orchestra.run.smithi001.stdout:osd.11: {} 2024-09-13T11:23:35.902 INFO:teuthology.orchestra.run.smithi001.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-13T11:23:35.930 INFO:teuthology.orchestra.run.smithi001.stdout:osd.12: {} 2024-09-13T11:23:35.930 INFO:teuthology.orchestra.run.smithi001.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-13T11:23:35.945 INFO:tasks.ceph.osd.4.smithi001.stderr:2024-09-13T11:23:35.940+0000 7ff8e0218640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26310) UID: 0 2024-09-13T11:23:35.961 INFO:teuthology.orchestra.run.smithi001.stdout:osd.13: {} 2024-09-13T11:23:35.961 INFO:teuthology.orchestra.run.smithi001.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-13T11:23:35.963 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-13T11:23:35.963 INFO:teuthology.orchestra.run.smithi162.stdout: "size": 20, 2024-09-13T11:23:35.963 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 600, 2024-09-13T11:23:35.963 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [ 2024-09-13T11:23:35.963 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:35.963 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.4665.0:171 3.9 e41/30 3:9e1c7eeb:::benchmark_data_smithi177_31116_object170:head v 41'13, mlcod=41'9)", 2024-09-13T11:23:35.963 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:22:47.418602+0000", 2024-09-13T11:23:35.963 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 48.541538168000002, 2024-09-13T11:23:35.963 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.67235367300000004, 2024-09-13T11:23:35.963 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:23:35.964 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:23:35.964 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:35.964 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:23:35.964 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:35.964 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:23:35.964 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:47.418602+0000", 2024-09-13T11:23:35.964 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:35.964 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:35.964 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:35.964 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:23:35.964 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:47.418602+0000", 2024-09-13T11:23:35.965 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:35.965 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:35.965 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:35.965 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:23:35.965 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:47.418605+0000", 2024-09-13T11:23:35.965 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.2200000000000001e-06 2024-09-13T11:23:35.965 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:35.965 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:35.965 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:23:35.965 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:47.418668+0000", 2024-09-13T11:23:35.965 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.2921000000000003e-05 2024-09-13T11:23:35.965 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:35.965 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:35.966 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:23:35.966 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:48.089380+0000", 2024-09-13T11:23:35.966 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.67071191100000005 2024-09-13T11:23:35.966 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:35.966 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:35.966 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:23:35.966 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:48.089423+0000", 2024-09-13T11:23:35.966 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.2916e-05 2024-09-13T11:23:35.967 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:35.967 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:35.967 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:23:35.967 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:48.089533+0000", 2024-09-13T11:23:35.967 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000109996 2024-09-13T11:23:35.967 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:35.967 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:35.967 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:23:35.967 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:48.089570+0000", 2024-09-13T11:23:35.967 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.7314999999999998e-05 2024-09-13T11:23:35.967 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:35.967 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:35.967 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:23:35.968 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:48.090852+0000", 2024-09-13T11:23:35.968 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001281641 2024-09-13T11:23:35.968 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:35.968 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:35.968 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:23:35.968 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:48.090956+0000", 2024-09-13T11:23:35.968 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00010375299999999999 2024-09-13T11:23:35.968 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:35.968 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:23:35.968 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:35.968 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:35.968 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:35.968 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.4665.0:39960 3.9 e64/30 3:9ee3e6ef:::benchmark_data_smithi177_31116_object39959:head v 64'2487, mlcod=64'2482)", 2024-09-13T11:23:35.968 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:15.804811+0000", 2024-09-13T11:23:35.968 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 20.155329326, 2024-09-13T11:23:35.969 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.070576958999999995, 2024-09-13T11:23:35.969 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:23:35.969 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:23:35.969 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:35.969 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:23:35.969 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:35.969 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:23:35.969 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:15.804811+0000", 2024-09-13T11:23:35.969 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:35.969 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:35.969 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:35.969 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:23:35.969 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:15.804811+0000", 2024-09-13T11:23:35.969 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:35.970 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:35.970 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:35.970 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:23:35.970 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:15.804815+0000", 2024-09-13T11:23:35.970 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.1330000000000001e-06 2024-09-13T11:23:35.970 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:35.970 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:35.970 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:23:35.970 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:15.804890+0000", 2024-09-13T11:23:35.970 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.5096000000000001e-05 2024-09-13T11:23:35.970 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:35.970 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:35.970 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:23:35.970 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:15.872369+0000", 2024-09-13T11:23:35.971 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.067478350000000006 2024-09-13T11:23:35.971 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:35.971 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:35.971 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:23:35.971 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:15.872387+0000", 2024-09-13T11:23:35.971 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.8315e-05 2024-09-13T11:23:35.971 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:35.971 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:35.971 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:23:35.971 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:15.872516+0000", 2024-09-13T11:23:35.971 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00012954000000000001 2024-09-13T11:23:35.971 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:35.971 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:35.971 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:23:35.972 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:15.872556+0000", 2024-09-13T11:23:35.972 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.0055000000000002e-05 2024-09-13T11:23:35.972 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:35.972 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:35.972 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:23:35.972 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:15.875302+0000", 2024-09-13T11:23:35.972 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0027456149999999999 2024-09-13T11:23:35.972 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:35.972 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:35.972 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:23:35.972 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:15.875388+0000", 2024-09-13T11:23:35.972 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.5854999999999997e-05 2024-09-13T11:23:35.973 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:35.973 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:23:35.973 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:35.973 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:35.973 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:35.973 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.4665.0:48216 3.9 e71/67 3:927b9ff4:::benchmark_data_smithi177_31116_object48215:head v 71'2887, mlcod=71'2883)", 2024-09-13T11:23:35.973 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:24.835666+0000", 2024-09-13T11:23:35.973 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 11.124474529, 2024-09-13T11:23:35.973 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.085651284999999994, 2024-09-13T11:23:35.973 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:23:35.973 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:23:35.973 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:35.973 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:23:35.974 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:35.974 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:23:35.974 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:24.835666+0000", 2024-09-13T11:23:35.974 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:35.974 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:35.974 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:35.974 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:23:35.974 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:24.835666+0000", 2024-09-13T11:23:35.975 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:35.975 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:35.975 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:35.975 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:23:35.975 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:24.835668+0000", 2024-09-13T11:23:35.975 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.778e-06 2024-09-13T11:23:35.975 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:35.975 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:35.975 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:23:35.975 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:24.835723+0000", 2024-09-13T11:23:35.975 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.5433999999999998e-05 2024-09-13T11:23:35.975 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:35.976 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:35.976 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:23:35.976 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:24.919015+0000", 2024-09-13T11:23:35.976 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.083291986999999998 2024-09-13T11:23:35.976 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:35.976 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:35.976 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:23:35.977 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:24.919025+0000", 2024-09-13T11:23:35.977 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0234000000000001e-05 2024-09-13T11:23:35.977 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:35.977 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:35.977 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:23:35.977 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:24.919115+0000", 2024-09-13T11:23:35.977 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.9969000000000005e-05 2024-09-13T11:23:35.977 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:35.977 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:35.977 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:23:35.977 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:24.919151+0000", 2024-09-13T11:23:35.977 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.6170000000000001e-05 2024-09-13T11:23:35.977 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:35.978 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:35.978 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:23:35.978 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:24.921265+0000", 2024-09-13T11:23:35.978 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0021133380000000002 2024-09-13T11:23:35.978 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:35.978 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:35.978 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:23:35.978 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:24.921317+0000", 2024-09-13T11:23:35.978 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.2374999999999999e-05 2024-09-13T11:23:35.978 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:35.978 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:23:35.978 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:35.978 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:35.979 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:35.979 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPushReply(3.19 71/70 [PushReplyOp(3:998ae847:::benchmark_data_smithi177_31116_object4216:head)])", 2024-09-13T11:23:35.979 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:24.931374+0000", 2024-09-13T11:23:35.979 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 11.028766387999999, 2024-09-13T11:23:35.979 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.073131357999999994, 2024-09-13T11:23:35.979 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:23:35.979 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:23:35.979 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-09-13T11:23:35.979 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:23:35.979 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:35.979 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:23:35.979 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:24.931374+0000", 2024-09-13T11:23:35.980 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:35.980 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:35.980 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:35.980 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:23:35.980 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:24.931374+0000", 2024-09-13T11:23:35.980 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:35.980 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:35.980 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:35.980 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:23:35.980 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:24.931375+0000", 2024-09-13T11:23:35.980 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1820000000000001e-06 2024-09-13T11:23:35.980 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:35.980 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:35.980 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:23:35.981 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:24.931380+0000", 2024-09-13T11:23:35.981 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.0540000000000002e-06 2024-09-13T11:23:35.981 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:35.981 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:35.981 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:23:35.981 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.002979+0000", 2024-09-13T11:23:35.981 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.071598740999999994 2024-09-13T11:23:35.981 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:35.981 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:35.981 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:23:35.981 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.003001+0000", 2024-09-13T11:23:35.981 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.2402e-05 2024-09-13T11:23:35.981 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:35.982 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:35.982 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:23:35.982 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.003125+0000", 2024-09-13T11:23:35.982 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00012376499999999999 2024-09-13T11:23:35.982 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:35.982 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:35.982 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:23:35.982 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.004505+0000", 2024-09-13T11:23:35.982 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0013802140000000001 2024-09-13T11:23:35.982 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:35.982 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:23:35.982 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:35.982 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:35.982 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:35.983 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPushReply(3.19 71/70 [PushReplyOp(3:9f04fe53:::benchmark_data_smithi177_31116_object4225:head)])", 2024-09-13T11:23:35.983 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:25.012367+0000", 2024-09-13T11:23:35.983 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 10.947773046, 2024-09-13T11:23:35.983 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.072652915999999998, 2024-09-13T11:23:35.983 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:23:35.983 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:23:35.983 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-09-13T11:23:35.983 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:23:35.983 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:35.983 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:23:35.983 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.012367+0000", 2024-09-13T11:23:35.983 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:35.984 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:35.984 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:35.984 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:23:35.984 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.012367+0000", 2024-09-13T11:23:35.984 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:35.984 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:35.984 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:35.984 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:23:35.984 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.012370+0000", 2024-09-13T11:23:35.984 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.4760000000000001e-06 2024-09-13T11:23:35.984 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:35.984 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:35.984 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:23:35.984 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.012378+0000", 2024-09-13T11:23:35.985 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.2649999999999996e-06 2024-09-13T11:23:35.985 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:35.985 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:35.985 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:23:35.985 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.083643+0000", 2024-09-13T11:23:35.985 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.071265257999999998 2024-09-13T11:23:35.985 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:35.985 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:35.985 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:23:35.985 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.083655+0000", 2024-09-13T11:23:35.985 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2004999999999999e-05 2024-09-13T11:23:35.985 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:35.985 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:35.986 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:23:35.986 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.083753+0000", 2024-09-13T11:23:35.986 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.7572999999999996e-05 2024-09-13T11:23:35.986 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:35.986 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:35.986 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:23:35.986 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.085020+0000", 2024-09-13T11:23:35.986 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0012673389999999999 2024-09-13T11:23:35.986 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:35.986 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:23:35.986 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:35.986 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:35.986 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:35.987 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPushReply(3.19 71/70 [PushReplyOp(3:980058cd:::benchmark_data_smithi177_31116_object4309:head)])", 2024-09-13T11:23:35.987 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:25.123140+0000", 2024-09-13T11:23:35.987 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 10.837000488999999, 2024-09-13T11:23:35.987 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.16436381899999999, 2024-09-13T11:23:35.987 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:23:35.987 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:23:35.987 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-09-13T11:23:35.987 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:23:35.987 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:35.987 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:23:35.987 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.123140+0000", 2024-09-13T11:23:35.987 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:35.987 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:35.988 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:35.988 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:23:35.988 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.123140+0000", 2024-09-13T11:23:35.988 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:35.988 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:35.988 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:35.988 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:23:35.988 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.123141+0000", 2024-09-13T11:23:35.988 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.562e-06 2024-09-13T11:23:35.988 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:35.988 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:35.988 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:23:35.988 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.123151+0000", 2024-09-13T11:23:35.989 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.3030000000000005e-06 2024-09-13T11:23:35.989 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:35.989 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:35.989 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:23:35.989 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.285828+0000", 2024-09-13T11:23:35.989 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.16267689399999999 2024-09-13T11:23:35.989 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:35.989 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:35.989 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:23:35.989 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.285838+0000", 2024-09-13T11:23:35.989 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0016e-05 2024-09-13T11:23:35.989 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:35.989 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:35.990 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:23:35.990 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.285966+0000", 2024-09-13T11:23:35.990 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00012870500000000001 2024-09-13T11:23:35.990 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:35.990 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:35.990 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:23:35.990 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.287504+0000", 2024-09-13T11:23:35.990 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001537339 2024-09-13T11:23:35.990 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:35.990 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:23:35.990 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:35.990 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:35.990 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:35.991 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:48909 3.19 3:9ef23a04:::benchmark_data_smithi177_31116_object48908:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e71)", 2024-09-13T11:23:35.991 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:25.454486+0000", 2024-09-13T11:23:35.991 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 10.505654625, 2024-09-13T11:23:35.991 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.072962329000000006, 2024-09-13T11:23:35.991 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:23:35.991 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:23:35.991 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:35.991 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:23:35.991 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:23:35.991 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:35.991 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 48909 2024-09-13T11:23:35.991 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:35.991 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:23:35.991 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:35.992 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:23:35.992 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.454486+0000", 2024-09-13T11:23:35.992 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:35.992 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:35.992 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:35.992 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:23:35.992 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.454486+0000", 2024-09-13T11:23:35.992 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:35.992 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:35.992 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:35.992 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:23:35.992 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.454488+0000", 2024-09-13T11:23:35.992 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.723e-06 2024-09-13T11:23:35.993 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:35.993 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:35.993 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:23:35.993 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.454567+0000", 2024-09-13T11:23:35.993 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.8671000000000006e-05 2024-09-13T11:23:35.993 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:35.993 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:35.993 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:23:35.993 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.454590+0000", 2024-09-13T11:23:35.993 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.2483000000000001e-05 2024-09-13T11:23:35.993 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:35.993 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:35.993 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:23:35.994 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.454602+0000", 2024-09-13T11:23:35.994 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2709e-05 2024-09-13T11:23:35.994 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:35.994 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:35.994 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:23:35.994 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.454672+0000", 2024-09-13T11:23:35.994 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.9487999999999996e-05 2024-09-13T11:23:35.994 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:35.994 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:35.994 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:23:35.994 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.454816+0000", 2024-09-13T11:23:35.994 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00014411 2024-09-13T11:23:35.994 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:35.994 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:35.995 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 1,9", 2024-09-13T11:23:35.995 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.454952+0000", 2024-09-13T11:23:35.995 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00013627999999999999 2024-09-13T11:23:35.995 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:35.995 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:35.995 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:23:35.995 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.456063+0000", 2024-09-13T11:23:35.995 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001110411 2024-09-13T11:23:35.995 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:35.995 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:35.995 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:35.995 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.458692+0000", 2024-09-13T11:23:35.995 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00262962 2024-09-13T11:23:35.996 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:35.996 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:35.996 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:35.996 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.526869+0000", 2024-09-13T11:23:35.996 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.068176666999999996 2024-09-13T11:23:35.996 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:35.996 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:35.996 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:23:35.996 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.526930+0000", 2024-09-13T11:23:35.996 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.0822000000000001e-05 2024-09-13T11:23:35.996 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:35.996 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:35.996 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:23:35.996 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.527448+0000", 2024-09-13T11:23:35.997 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00051834500000000003 2024-09-13T11:23:35.997 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:35.997 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:23:35.997 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:35.997 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:35.997 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:35.997 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop_reply(client.4665.0:48909 3.19 e71/70 ondisk, result = 0)", 2024-09-13T11:23:35.997 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:25.458751+0000", 2024-09-13T11:23:35.997 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 10.501389236, 2024-09-13T11:23:35.997 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.068704475000000001, 2024-09-13T11:23:35.997 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:23:35.997 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:23:35.997 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:23:35.998 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:23:35.998 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:35.998 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:23:35.998 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.458751+0000", 2024-09-13T11:23:35.998 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:35.998 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:35.998 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:35.998 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:23:35.998 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.458751+0000", 2024-09-13T11:23:35.998 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:35.998 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:35.998 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:35.998 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:23:35.998 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.458752+0000", 2024-09-13T11:23:35.999 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.2200000000000003e-07 2024-09-13T11:23:35.999 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:35.999 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:35.999 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:23:35.999 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.458755+0000", 2024-09-13T11:23:35.999 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.377e-06 2024-09-13T11:23:35.999 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:35.999 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:35.999 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:23:35.999 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.526795+0000", 2024-09-13T11:23:35.999 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.068040001000000003 2024-09-13T11:23:35.999 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:35.999 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.000 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:23:36.000 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.526802+0000", 2024-09-13T11:23:36.000 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.3620000000000002e-06 2024-09-13T11:23:36.000 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.000 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.000 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:23:36.000 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.526857+0000", 2024-09-13T11:23:36.000 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.5087999999999998e-05 2024-09-13T11:23:36.000 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.000 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.000 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:23:36.000 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.526864+0000", 2024-09-13T11:23:36.001 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.6109999999999996e-06 2024-09-13T11:23:36.001 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.001 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.001 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:23:36.001 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.527456+0000", 2024-09-13T11:23:36.001 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00059121399999999995 2024-09-13T11:23:36.001 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:36.001 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:23:36.001 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:36.001 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.001 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.001 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPushReply(3.19 71/70 [PushReplyOp(3:9b41e30d:::benchmark_data_smithi177_31116_object4694:head)])", 2024-09-13T11:23:36.001 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:25.466329+0000", 2024-09-13T11:23:36.002 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 10.493810908, 2024-09-13T11:23:36.002 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.084015425000000005, 2024-09-13T11:23:36.002 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:23:36.002 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:23:36.002 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-09-13T11:23:36.002 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:23:36.002 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.002 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:23:36.002 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.466329+0000", 2024-09-13T11:23:36.002 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:36.002 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.002 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.002 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:23:36.002 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.466329+0000", 2024-09-13T11:23:36.003 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:36.003 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.003 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.003 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:23:36.003 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.466331+0000", 2024-09-13T11:23:36.003 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4500000000000001e-06 2024-09-13T11:23:36.003 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.003 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.003 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:23:36.003 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.466336+0000", 2024-09-13T11:23:36.003 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.5029999999999997e-06 2024-09-13T11:23:36.003 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.003 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.003 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:23:36.004 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.548447+0000", 2024-09-13T11:23:36.004 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.082110965999999994 2024-09-13T11:23:36.004 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.004 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.004 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:23:36.004 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.548469+0000", 2024-09-13T11:23:36.004 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.1628999999999999e-05 2024-09-13T11:23:36.004 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.004 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.004 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:23:36.004 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.548626+0000", 2024-09-13T11:23:36.004 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00015717699999999999 2024-09-13T11:23:36.004 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.005 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.005 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:23:36.005 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.550345+0000", 2024-09-13T11:23:36.005 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0017187000000000001 2024-09-13T11:23:36.005 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:36.005 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:23:36.005 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:36.005 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.005 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.005 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPushReply(3.19 72/70 [PushReplyOp(3:98989a53:::benchmark_data_smithi177_31116_object7141:head)])", 2024-09-13T11:23:36.005 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:27.177330+0000", 2024-09-13T11:23:36.005 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 8.7828099169999998, 2024-09-13T11:23:36.005 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.065907854000000002, 2024-09-13T11:23:36.005 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:23:36.006 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:23:36.006 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-09-13T11:23:36.006 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:23:36.006 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.006 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:23:36.006 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:27.177330+0000", 2024-09-13T11:23:36.006 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:36.006 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.006 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.006 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:23:36.006 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:27.177330+0000", 2024-09-13T11:23:36.006 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:36.006 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.006 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.007 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:23:36.007 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:27.177332+0000", 2024-09-13T11:23:36.007 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.8619999999999999e-06 2024-09-13T11:23:36.007 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.007 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.007 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:23:36.007 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:27.177338+0000", 2024-09-13T11:23:36.007 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.6459999999999998e-06 2024-09-13T11:23:36.007 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.007 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.007 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:23:36.007 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:27.242386+0000", 2024-09-13T11:23:36.007 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.065047802000000002 2024-09-13T11:23:36.007 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.008 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.008 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:23:36.008 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:27.242397+0000", 2024-09-13T11:23:36.008 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1141000000000001e-05 2024-09-13T11:23:36.008 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.008 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.008 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:23:36.008 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:27.242465+0000", 2024-09-13T11:23:36.008 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.7994e-05 2024-09-13T11:23:36.008 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.008 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.008 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:23:36.008 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:27.243238+0000", 2024-09-13T11:23:36.008 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00077340899999999997 2024-09-13T11:23:36.009 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:36.009 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:23:36.009 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:36.009 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.009 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.009 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:53947 3.19 3:9ff49d75:::benchmark_data_smithi177_31116_object53946:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e74)", 2024-09-13T11:23:36.009 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:30.219526+0000", 2024-09-13T11:23:36.009 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 5.7406140600000004, 2024-09-13T11:23:36.009 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.094435565999999999, 2024-09-13T11:23:36.009 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:23:36.009 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:23:36.009 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:36.009 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:23:36.010 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:23:36.010 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:36.010 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 53947 2024-09-13T11:23:36.010 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.010 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:23:36.010 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.010 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:23:36.010 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.219526+0000", 2024-09-13T11:23:36.010 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:36.010 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.010 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.010 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:23:36.010 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.219526+0000", 2024-09-13T11:23:36.010 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:36.011 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.011 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.011 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:23:36.011 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.219533+0000", 2024-09-13T11:23:36.011 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.8549999999999996e-06 2024-09-13T11:23:36.011 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.011 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.011 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:23:36.011 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.219594+0000", 2024-09-13T11:23:36.011 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.0979e-05 2024-09-13T11:23:36.011 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.011 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.011 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:23:36.011 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.219612+0000", 2024-09-13T11:23:36.012 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.7396000000000001e-05 2024-09-13T11:23:36.012 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.012 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.012 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:23:36.012 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.219620+0000", 2024-09-13T11:23:36.012 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.6009999999999995e-06 2024-09-13T11:23:36.012 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.012 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.012 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:23:36.012 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.223111+0000", 2024-09-13T11:23:36.012 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0034904319999999999 2024-09-13T11:23:36.012 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.012 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.013 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:23:36.013 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.223349+0000", 2024-09-13T11:23:36.013 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00023811500000000001 2024-09-13T11:23:36.013 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.013 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.013 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 1,9", 2024-09-13T11:23:36.013 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.223615+0000", 2024-09-13T11:23:36.013 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00026669600000000002 2024-09-13T11:23:36.013 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.013 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.013 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:23:36.013 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.225114+0000", 2024-09-13T11:23:36.013 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001499071 2024-09-13T11:23:36.013 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.014 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.014 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:36.014 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.226428+0000", 2024-09-13T11:23:36.014 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0013132230000000001 2024-09-13T11:23:36.014 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.014 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.014 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:36.014 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.313440+0000", 2024-09-13T11:23:36.014 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.087011863999999994 2024-09-13T11:23:36.014 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.014 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.014 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:23:36.014 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.313508+0000", 2024-09-13T11:23:36.015 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.8817000000000002e-05 2024-09-13T11:23:36.015 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.015 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.015 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:23:36.015 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.313962+0000", 2024-09-13T11:23:36.015 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00045351700000000002 2024-09-13T11:23:36.015 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:36.015 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:23:36.015 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:36.015 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.015 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.015 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPushReply(3.19 74/70 [PushReplyOp(3:9f1cbe13:::benchmark_data_smithi177_31116_object16686:head)])", 2024-09-13T11:23:36.016 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:30.225983+0000", 2024-09-13T11:23:36.016 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 5.7341578159999997, 2024-09-13T11:23:36.016 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.081856162999999996, 2024-09-13T11:23:36.016 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:23:36.016 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:23:36.016 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-09-13T11:23:36.016 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:23:36.016 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.016 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:23:36.016 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.225983+0000", 2024-09-13T11:23:36.016 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:36.016 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.016 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.017 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:23:36.017 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.225983+0000", 2024-09-13T11:23:36.017 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:36.017 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.017 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.017 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:23:36.017 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.225984+0000", 2024-09-13T11:23:36.017 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1999999999999999e-06 2024-09-13T11:23:36.017 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.017 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.017 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:23:36.017 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.225988+0000", 2024-09-13T11:23:36.017 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.4179999999999999e-06 2024-09-13T11:23:36.018 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.018 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.018 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:23:36.018 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.306757+0000", 2024-09-13T11:23:36.018 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.080768435 2024-09-13T11:23:36.018 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.018 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.018 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:23:36.018 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.306767+0000", 2024-09-13T11:23:36.018 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0093e-05 2024-09-13T11:23:36.018 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.018 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.018 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:23:36.019 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.306816+0000", 2024-09-13T11:23:36.019 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.9021999999999999e-05 2024-09-13T11:23:36.019 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.019 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.019 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:23:36.019 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.307839+0000", 2024-09-13T11:23:36.019 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0010229950000000001 2024-09-13T11:23:36.019 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:36.019 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:23:36.019 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:36.019 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.019 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.019 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop_reply(client.4665.0:53947 3.19 e74/70 ondisk, result = 0)", 2024-09-13T11:23:36.020 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:30.227904+0000", 2024-09-13T11:23:36.020 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 5.7322362550000001, 2024-09-13T11:23:36.020 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.086073851000000007, 2024-09-13T11:23:36.020 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:23:36.020 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:23:36.020 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:23:36.020 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:23:36.020 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.020 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:23:36.020 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.227904+0000", 2024-09-13T11:23:36.020 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:36.020 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.020 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.020 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:23:36.021 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.227904+0000", 2024-09-13T11:23:36.021 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:36.021 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.021 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.021 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:23:36.021 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.227906+0000", 2024-09-13T11:23:36.021 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5549999999999999e-06 2024-09-13T11:23:36.021 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.021 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.021 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:23:36.022 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.227911+0000", 2024-09-13T11:23:36.022 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.5940000000000001e-06 2024-09-13T11:23:36.022 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.022 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.022 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:23:36.022 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.313329+0000", 2024-09-13T11:23:36.022 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.08541791 2024-09-13T11:23:36.022 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.022 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.022 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:23:36.022 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.313340+0000", 2024-09-13T11:23:36.022 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1061e-05 2024-09-13T11:23:36.023 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.023 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.023 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:23:36.023 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.313412+0000", 2024-09-13T11:23:36.023 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.1581000000000005e-05 2024-09-13T11:23:36.023 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.023 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.023 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:23:36.023 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.313421+0000", 2024-09-13T11:23:36.023 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.8920000000000006e-06 2024-09-13T11:23:36.023 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.023 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.023 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:23:36.023 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.313978+0000", 2024-09-13T11:23:36.024 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00055725800000000004 2024-09-13T11:23:36.024 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:36.024 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:23:36.024 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:36.024 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.024 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.024 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPushReply(3.19 74/70 [PushReplyOp(3:9b6cad3a:::benchmark_data_smithi177_31116_object16698:head)])", 2024-09-13T11:23:36.024 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:30.313297+0000", 2024-09-13T11:23:36.024 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 5.6468438130000003, 2024-09-13T11:23:36.024 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.162355306, 2024-09-13T11:23:36.024 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:23:36.024 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:23:36.025 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-09-13T11:23:36.025 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:23:36.025 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.025 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:23:36.025 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.313297+0000", 2024-09-13T11:23:36.025 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:36.025 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.025 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.025 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:23:36.025 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.313297+0000", 2024-09-13T11:23:36.025 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:36.025 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.025 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.026 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:23:36.026 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.313298+0000", 2024-09-13T11:23:36.026 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.61e-06 2024-09-13T11:23:36.026 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.026 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.026 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:23:36.026 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.313306+0000", 2024-09-13T11:23:36.026 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.9950000000000005e-06 2024-09-13T11:23:36.026 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.026 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.026 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:23:36.026 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.473757+0000", 2024-09-13T11:23:36.026 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.16045083099999999 2024-09-13T11:23:36.027 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.027 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.027 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:23:36.027 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.473764+0000", 2024-09-13T11:23:36.027 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.4769999999999997e-06 2024-09-13T11:23:36.027 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.027 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.027 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:23:36.027 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.473870+0000", 2024-09-13T11:23:36.027 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00010529100000000001 2024-09-13T11:23:36.027 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.027 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.027 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:23:36.028 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.475652+0000", 2024-09-13T11:23:36.028 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001782102 2024-09-13T11:23:36.028 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:36.028 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:23:36.028 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:36.028 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.028 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.028 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:54073 3.19 3:9f968cf6:::benchmark_data_smithi177_31116_object54072:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e74)", 2024-09-13T11:23:36.028 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:30.331382+0000", 2024-09-13T11:23:36.028 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 5.6287581190000004, 2024-09-13T11:23:36.028 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.15619861600000001, 2024-09-13T11:23:36.028 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:23:36.028 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:23:36.029 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:36.029 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:23:36.029 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:23:36.029 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:36.029 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 54073 2024-09-13T11:23:36.029 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.029 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:23:36.029 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.029 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:23:36.029 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.331382+0000", 2024-09-13T11:23:36.029 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:36.029 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.029 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.029 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:23:36.030 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.331382+0000", 2024-09-13T11:23:36.030 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:36.030 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.030 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.030 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:23:36.030 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.331386+0000", 2024-09-13T11:23:36.030 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.5769999999999998e-06 2024-09-13T11:23:36.030 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.030 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.030 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:23:36.030 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.331430+0000", 2024-09-13T11:23:36.030 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.4172999999999999e-05 2024-09-13T11:23:36.030 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.031 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.031 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:23:36.031 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.331450+0000", 2024-09-13T11:23:36.031 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.9922999999999999e-05 2024-09-13T11:23:36.031 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.031 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.031 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:23:36.031 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.331460+0000", 2024-09-13T11:23:36.031 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0444e-05 2024-09-13T11:23:36.031 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.031 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.031 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:23:36.031 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.331590+0000", 2024-09-13T11:23:36.031 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00013000899999999999 2024-09-13T11:23:36.032 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.032 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.032 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:23:36.032 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.331929+0000", 2024-09-13T11:23:36.032 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00033835400000000002 2024-09-13T11:23:36.032 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.032 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.032 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 1,9", 2024-09-13T11:23:36.032 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.332198+0000", 2024-09-13T11:23:36.032 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00026941900000000002 2024-09-13T11:23:36.032 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.032 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.032 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:23:36.032 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.334200+0000", 2024-09-13T11:23:36.033 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0020020179999999999 2024-09-13T11:23:36.033 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.033 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.033 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:36.033 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.334826+0000", 2024-09-13T11:23:36.033 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00062540600000000001 2024-09-13T11:23:36.033 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.033 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.033 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:36.033 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.486479+0000", 2024-09-13T11:23:36.033 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.151653395 2024-09-13T11:23:36.033 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.034 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.034 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:23:36.034 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.486691+0000", 2024-09-13T11:23:36.034 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00021176499999999999 2024-09-13T11:23:36.034 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.034 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.034 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:23:36.034 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.487581+0000", 2024-09-13T11:23:36.034 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00089013299999999998 2024-09-13T11:23:36.034 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:36.034 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:23:36.034 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:36.034 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.035 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.035 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop_reply(client.4665.0:54073 3.19 e74/70 ondisk, result = 0)", 2024-09-13T11:23:36.035 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:30.334897+0000", 2024-09-13T11:23:36.035 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 5.6252433420000001, 2024-09-13T11:23:36.035 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.152699536, 2024-09-13T11:23:36.035 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:23:36.035 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:23:36.035 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:23:36.035 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:23:36.035 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.035 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:23:36.035 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.334897+0000", 2024-09-13T11:23:36.035 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:36.035 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.036 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.036 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:23:36.036 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.334897+0000", 2024-09-13T11:23:36.036 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:36.036 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.036 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.036 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:23:36.036 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.334898+0000", 2024-09-13T11:23:36.036 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.499e-06 2024-09-13T11:23:36.036 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.036 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.036 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:23:36.036 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.334905+0000", 2024-09-13T11:23:36.037 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.6470000000000001e-06 2024-09-13T11:23:36.037 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.037 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.037 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:23:36.037 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.486251+0000", 2024-09-13T11:23:36.037 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.151345432 2024-09-13T11:23:36.037 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.037 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.037 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:23:36.037 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.486262+0000", 2024-09-13T11:23:36.037 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1005000000000001e-05 2024-09-13T11:23:36.037 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.037 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.037 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:23:36.038 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.486411+0000", 2024-09-13T11:23:36.038 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00014960099999999999 2024-09-13T11:23:36.038 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.038 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.038 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:23:36.038 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.486434+0000", 2024-09-13T11:23:36.038 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.3034e-05 2024-09-13T11:23:36.038 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.038 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.038 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:23:36.038 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.487597+0000", 2024-09-13T11:23:36.038 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0011623180000000001 2024-09-13T11:23:36.038 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:36.039 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:23:36.039 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:36.039 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.039 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.039 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPushReply(3.19 74/70 [PushReplyOp(3:9d264015:::benchmark_data_smithi177_31116_object16883:head)])", 2024-09-13T11:23:36.039 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:30.637135+0000", 2024-09-13T11:23:36.039 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 5.3230055040000002, 2024-09-13T11:23:36.039 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.098447104999999993, 2024-09-13T11:23:36.039 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:23:36.039 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:23:36.039 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-09-13T11:23:36.039 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:23:36.039 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.040 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:23:36.040 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.637135+0000", 2024-09-13T11:23:36.040 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:36.040 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.040 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.040 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:23:36.040 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.637135+0000", 2024-09-13T11:23:36.040 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:36.040 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.040 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.040 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:23:36.040 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.637137+0000", 2024-09-13T11:23:36.040 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.885e-06 2024-09-13T11:23:36.041 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.041 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.041 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:23:36.041 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.637144+0000", 2024-09-13T11:23:36.041 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.3760000000000002e-06 2024-09-13T11:23:36.041 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.041 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.041 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:23:36.041 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.734363+0000", 2024-09-13T11:23:36.041 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.097218465000000004 2024-09-13T11:23:36.041 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.041 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.041 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:23:36.042 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.734373+0000", 2024-09-13T11:23:36.042 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0472000000000001e-05 2024-09-13T11:23:36.042 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.042 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.042 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:23:36.042 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.734437+0000", 2024-09-13T11:23:36.042 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.3547999999999995e-05 2024-09-13T11:23:36.042 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.042 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.042 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:23:36.042 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.735582+0000", 2024-09-13T11:23:36.042 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0011453590000000001 2024-09-13T11:23:36.042 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:36.043 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:23:36.043 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:36.043 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.043 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.043 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPushReply(3.19 74/70 [PushReplyOp(3:9edf939c:::benchmark_data_smithi177_31116_object17207:head)])", 2024-09-13T11:23:36.043 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:30.897971+0000", 2024-09-13T11:23:36.043 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 5.0621693170000004, 2024-09-13T11:23:36.043 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.40616071100000001, 2024-09-13T11:23:36.043 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:23:36.043 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:23:36.043 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-09-13T11:23:36.043 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:23:36.043 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.043 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:23:36.044 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.897971+0000", 2024-09-13T11:23:36.044 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:36.044 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.044 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.044 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:23:36.044 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.897971+0000", 2024-09-13T11:23:36.044 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:36.044 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.044 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.044 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:23:36.044 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.897972+0000", 2024-09-13T11:23:36.044 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1429999999999999e-06 2024-09-13T11:23:36.045 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.045 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.045 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:23:36.045 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.897977+0000", 2024-09-13T11:23:36.045 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.5199999999999999e-06 2024-09-13T11:23:36.045 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.045 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.045 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:23:36.045 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:31.302500+0000", 2024-09-13T11:23:36.045 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.40452312400000001 2024-09-13T11:23:36.045 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.045 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.045 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:23:36.046 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:31.302513+0000", 2024-09-13T11:23:36.046 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2753e-05 2024-09-13T11:23:36.046 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.046 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.046 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:23:36.046 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:31.302645+0000", 2024-09-13T11:23:36.046 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00013292100000000001 2024-09-13T11:23:36.046 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.046 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.046 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:23:36.046 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:31.304132+0000", 2024-09-13T11:23:36.046 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0014862499999999999 2024-09-13T11:23:36.046 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:36.047 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:23:36.047 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:36.047 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.047 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.047 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPushReply(3.19 75/70 [PushReplyOp(3:9d1489b1:::benchmark_data_smithi177_31116_object18698:head)])", 2024-09-13T11:23:36.047 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:31.697212+0000", 2024-09-13T11:23:36.047 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 4.262928316, 2024-09-13T11:23:36.047 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.22850633200000001, 2024-09-13T11:23:36.047 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:23:36.047 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:23:36.047 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-09-13T11:23:36.047 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:23:36.047 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.048 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:23:36.048 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:31.697212+0000", 2024-09-13T11:23:36.048 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:36.048 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.048 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.048 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:23:36.048 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:31.697212+0000", 2024-09-13T11:23:36.048 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:36.048 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.048 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.048 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:23:36.048 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:31.697214+0000", 2024-09-13T11:23:36.048 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.7439999999999999e-06 2024-09-13T11:23:36.049 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.049 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.049 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:23:36.049 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:31.697224+0000", 2024-09-13T11:23:36.049 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0173000000000001e-05 2024-09-13T11:23:36.049 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.049 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.049 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:23:36.049 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:31.923931+0000", 2024-09-13T11:23:36.049 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.226706926 2024-09-13T11:23:36.049 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.049 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.049 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:23:36.049 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:31.923950+0000", 2024-09-13T11:23:36.050 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.8899e-05 2024-09-13T11:23:36.050 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.050 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.050 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:23:36.050 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:31.924449+0000", 2024-09-13T11:23:36.050 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00049907800000000002 2024-09-13T11:23:36.050 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.050 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.050 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:23:36.050 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:31.925718+0000", 2024-09-13T11:23:36.050 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001269512 2024-09-13T11:23:36.050 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:36.050 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:23:36.051 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:36.051 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.051 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.051 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPushReply(3.19 76/70 [PushReplyOp(3:9faf6d5f:::benchmark_data_smithi177_31116_object18954:head)])", 2024-09-13T11:23:36.051 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:32.042874+0000", 2024-09-13T11:23:36.051 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 3.9172664470000003, 2024-09-13T11:23:36.051 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.40212404299999999, 2024-09-13T11:23:36.051 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:23:36.051 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:23:36.051 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-09-13T11:23:36.051 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:23:36.051 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.051 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:23:36.051 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.042874+0000", 2024-09-13T11:23:36.052 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:36.052 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.052 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.052 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:23:36.052 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.042874+0000", 2024-09-13T11:23:36.052 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:36.052 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.052 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.052 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:23:36.052 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.042875+0000", 2024-09-13T11:23:36.052 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.432e-06 2024-09-13T11:23:36.052 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.053 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.053 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:23:36.053 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.042886+0000", 2024-09-13T11:23:36.053 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0265000000000001e-05 2024-09-13T11:23:36.053 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.053 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.053 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:23:36.053 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.443455+0000", 2024-09-13T11:23:36.053 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.40056951400000002 2024-09-13T11:23:36.053 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.053 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.053 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:23:36.053 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.443480+0000", 2024-09-13T11:23:36.053 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.4929000000000001e-05 2024-09-13T11:23:36.054 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.054 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.054 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:23:36.054 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.443666+0000", 2024-09-13T11:23:36.054 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000186446 2024-09-13T11:23:36.054 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.054 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.054 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:23:36.054 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.444998+0000", 2024-09-13T11:23:36.054 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0013314570000000001 2024-09-13T11:23:36.054 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:36.054 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:23:36.054 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:36.055 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:36.055 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:23:36.055 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-13T11:23:36.056 INFO:tasks.radosbench.radosbench.0.smithi177.stdout: 69 16 55879 55863 50.5951 29.3438 0.015641 0.0197536 2024-09-13T11:23:36.056 INFO:teuthology.orchestra.run.smithi001.stdout:osd.14: {} 2024-09-13T11:23:36.057 INFO:teuthology.orchestra.run.smithi001.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-13T11:23:36.057 INFO:teuthology.orchestra.run.smithi001.stdout:osd.15: {} 2024-09-13T11:23:36.058 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-13T11:23:36.058 INFO:tasks.ceph.osd.10.smithi162.stderr:2024-09-13T11:23:36.049+0000 7fb59c9f0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26317) UID: 0 2024-09-13T11:23:36.059 INFO:teuthology.orchestra.run.smithi001.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-13T11:23:36.158 INFO:tasks.ceph.osd.7.smithi177.stderr:2024-09-13T11:23:36.152+0000 7f897b51d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26575) UID: 0 2024-09-13T11:23:36.244 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-13T11:23:36.244 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [], 2024-09-13T11:23:36.244 INFO:teuthology.orchestra.run.smithi162.stdout: "num_ops": 0 2024-09-13T11:23:36.244 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-13T11:23:36.257 INFO:tasks.ceph.osd.11.smithi177.stderr:2024-09-13T11:23:36.252+0000 7f24792c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26580) UID: 0 2024-09-13T11:23:36.263 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-13T11:23:36.358 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-13T11:23:36.352+0000 7f6a68775640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26307) UID: 0 2024-09-13T11:23:36.403 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-13T11:23:36.403 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [], 2024-09-13T11:23:36.403 INFO:teuthology.orchestra.run.smithi162.stdout: "complaint_time": 30, 2024-09-13T11:23:36.403 INFO:teuthology.orchestra.run.smithi162.stdout: "num_blocked_ops": 0 2024-09-13T11:23:36.404 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-13T11:23:36.420 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-13T11:23:36.459 INFO:tasks.ceph.osd.13.smithi092.stderr:2024-09-13T11:23:36.456+0000 7f3e5afea640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26350) UID: 0 2024-09-13T11:23:36.558 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-13T11:23:36.558 INFO:teuthology.orchestra.run.smithi162.stdout: "size": 20, 2024-09-13T11:23:36.558 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 600, 2024-09-13T11:23:36.558 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [ 2024-09-13T11:23:36.558 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.558 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4257.0:80 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 262144~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-13T11:23:36.559 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:22:12.516065+0000", 2024-09-13T11:23:36.559 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 84.039947377999994, 2024-09-13T11:23:36.559 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.46461997999999999, 2024-09-13T11:23:36.559 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:23:36.559 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:23:36.559 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:36.559 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:23:36.559 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4257", 2024-09-13T11:23:36.559 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/1488131527", 2024-09-13T11:23:36.559 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 80 2024-09-13T11:23:36.559 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.559 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:23:36.559 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.559 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:23:36.560 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.516065+0000", 2024-09-13T11:23:36.560 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:36.560 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.560 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.560 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:23:36.560 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.516065+0000", 2024-09-13T11:23:36.560 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:36.560 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.560 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.560 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:23:36.560 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.516066+0000", 2024-09-13T11:23:36.560 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0869999999999999e-06 2024-09-13T11:23:36.560 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.560 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.561 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:23:36.561 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.516112+0000", 2024-09-13T11:23:36.561 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.5503000000000001e-05 2024-09-13T11:23:36.561 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.561 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.561 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:23:36.561 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.516128+0000", 2024-09-13T11:23:36.561 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5891000000000001e-05 2024-09-13T11:23:36.561 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.561 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.561 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:23:36.561 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.516138+0000", 2024-09-13T11:23:36.561 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.8120000000000002e-06 2024-09-13T11:23:36.561 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.562 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.562 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:23:36.562 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.520398+0000", 2024-09-13T11:23:36.562 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0042605450000000001 2024-09-13T11:23:36.562 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.562 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.562 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:23:36.562 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.520526+0000", 2024-09-13T11:23:36.562 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000128355 2024-09-13T11:23:36.562 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.562 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.562 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 0", 2024-09-13T11:23:36.562 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.520730+0000", 2024-09-13T11:23:36.563 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00020337799999999999 2024-09-13T11:23:36.563 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.563 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.563 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:23:36.563 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.522417+0000", 2024-09-13T11:23:36.563 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001687184 2024-09-13T11:23:36.563 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.563 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.563 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:36.563 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.980408+0000", 2024-09-13T11:23:36.563 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.45799068399999998 2024-09-13T11:23:36.563 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.563 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.563 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:23:36.564 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.980565+0000", 2024-09-13T11:23:36.564 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00015760899999999999 2024-09-13T11:23:36.564 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.564 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.564 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:23:36.564 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.980685+0000", 2024-09-13T11:23:36.564 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000119932 2024-09-13T11:23:36.564 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:36.564 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:23:36.564 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:36.564 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.564 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.564 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop_reply(client.4257.0:80 1.0 e24/23 ondisk, result = 0)", 2024-09-13T11:23:36.565 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:22:12.522938+0000", 2024-09-13T11:23:36.565 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 84.033074345000003, 2024-09-13T11:23:36.565 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.45776420600000001, 2024-09-13T11:23:36.565 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:23:36.565 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:23:36.565 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:23:36.565 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:23:36.565 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.565 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:23:36.565 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.522938+0000", 2024-09-13T11:23:36.565 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:36.565 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.565 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.565 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:23:36.566 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.522938+0000", 2024-09-13T11:23:36.566 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:36.566 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.566 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.566 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:23:36.566 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.522941+0000", 2024-09-13T11:23:36.566 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.0290000000000001e-06 2024-09-13T11:23:36.566 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.566 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.566 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:23:36.566 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.522950+0000", 2024-09-13T11:23:36.566 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.7499999999999992e-06 2024-09-13T11:23:36.566 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.566 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.567 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:23:36.567 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.980215+0000", 2024-09-13T11:23:36.567 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.45726523800000002 2024-09-13T11:23:36.567 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.567 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.567 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:23:36.567 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.980229+0000", 2024-09-13T11:23:36.567 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3549e-05 2024-09-13T11:23:36.567 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.567 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.567 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:23:36.567 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.980365+0000", 2024-09-13T11:23:36.567 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00013608700000000001 2024-09-13T11:23:36.568 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.568 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.568 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:23:36.568 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.980388+0000", 2024-09-13T11:23:36.568 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.3112000000000002e-05 2024-09-13T11:23:36.568 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.568 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.568 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:23:36.568 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.980702+0000", 2024-09-13T11:23:36.568 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00031444099999999999 2024-09-13T11:23:36.568 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:36.568 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:23:36.568 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:36.568 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.569 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.569 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:25416 3.8 3:1a1fbdd1:::benchmark_data_smithi177_31116_object25415:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e56)", 2024-09-13T11:23:36.569 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:04.512188+0000", 2024-09-13T11:23:36.569 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 32.043824377999996, 2024-09-13T11:23:36.569 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.210208697, 2024-09-13T11:23:36.569 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:23:36.569 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:23:36.569 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:36.569 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:23:36.569 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:23:36.569 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:36.569 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 25416 2024-09-13T11:23:36.569 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.569 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:23:36.570 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.570 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:23:36.570 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.512188+0000", 2024-09-13T11:23:36.570 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:36.570 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.570 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.570 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:23:36.570 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.512188+0000", 2024-09-13T11:23:36.570 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:36.570 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.570 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.570 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:23:36.570 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.512192+0000", 2024-09-13T11:23:36.570 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.2660000000000003e-06 2024-09-13T11:23:36.571 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.571 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.571 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:23:36.571 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.512247+0000", 2024-09-13T11:23:36.571 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.4889000000000003e-05 2024-09-13T11:23:36.571 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.571 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.571 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:23:36.571 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.512279+0000", 2024-09-13T11:23:36.571 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.1146000000000003e-05 2024-09-13T11:23:36.571 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.571 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.571 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:23:36.571 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.512297+0000", 2024-09-13T11:23:36.572 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.825e-05 2024-09-13T11:23:36.572 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.572 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.572 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:23:36.572 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.512377+0000", 2024-09-13T11:23:36.572 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.9875000000000004e-05 2024-09-13T11:23:36.572 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.572 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.572 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:23:36.572 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.512570+0000", 2024-09-13T11:23:36.572 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00019312100000000001 2024-09-13T11:23:36.572 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.572 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.573 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 1", 2024-09-13T11:23:36.573 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.512798+0000", 2024-09-13T11:23:36.573 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00022847899999999999 2024-09-13T11:23:36.573 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.573 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.573 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:23:36.573 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.514975+0000", 2024-09-13T11:23:36.573 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0021769979999999999 2024-09-13T11:23:36.573 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.573 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.573 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:36.573 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.722207+0000", 2024-09-13T11:23:36.573 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.20723217099999999 2024-09-13T11:23:36.573 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.574 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.574 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:23:36.574 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.722314+0000", 2024-09-13T11:23:36.574 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00010648400000000001 2024-09-13T11:23:36.574 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.574 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.574 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:23:36.574 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.722397+0000", 2024-09-13T11:23:36.574 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.3017999999999997e-05 2024-09-13T11:23:36.574 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:36.574 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:23:36.574 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:36.574 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.574 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.574 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop_reply(client.4665.0:25416 3.8 e56/40 ondisk, result = 0)", 2024-09-13T11:23:36.575 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:04.515635+0000", 2024-09-13T11:23:36.575 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 32.040377726000003, 2024-09-13T11:23:36.575 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.20677600500000001, 2024-09-13T11:23:36.575 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:23:36.575 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:23:36.575 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:23:36.575 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:23:36.575 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.575 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:23:36.575 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.515635+0000", 2024-09-13T11:23:36.575 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:36.575 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.575 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.575 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:23:36.576 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.515635+0000", 2024-09-13T11:23:36.576 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:36.576 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.576 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.576 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:23:36.576 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.515639+0000", 2024-09-13T11:23:36.577 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.619e-06 2024-09-13T11:23:36.577 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.577 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.577 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:23:36.577 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.515650+0000", 2024-09-13T11:23:36.577 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.111e-05 2024-09-13T11:23:36.577 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.577 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.577 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:23:36.577 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.722004+0000", 2024-09-13T11:23:36.577 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.206354805 2024-09-13T11:23:36.577 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.577 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.577 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:23:36.578 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.722017+0000", 2024-09-13T11:23:36.578 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2840000000000001e-05 2024-09-13T11:23:36.578 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.578 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.578 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:23:36.578 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.722186+0000", 2024-09-13T11:23:36.578 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00016898299999999999 2024-09-13T11:23:36.578 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.578 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.578 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:23:36.578 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.722199+0000", 2024-09-13T11:23:36.578 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.289e-05 2024-09-13T11:23:36.578 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.578 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.579 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:23:36.579 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.722411+0000", 2024-09-13T11:23:36.579 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00021175799999999999 2024-09-13T11:23:36.579 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:36.579 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:23:36.579 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:36.579 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.579 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.579 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:25432 3.8 3:19671730:::benchmark_data_smithi177_31116_object25431:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e56)", 2024-09-13T11:23:36.579 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:04.717754+0000", 2024-09-13T11:23:36.579 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 31.838258711000002, 2024-09-13T11:23:36.579 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.102412884, 2024-09-13T11:23:36.579 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:23:36.579 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:23:36.580 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:36.580 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:23:36.580 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:23:36.580 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:36.580 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 25432 2024-09-13T11:23:36.580 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.581 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:23:36.581 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.581 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:23:36.581 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.717754+0000", 2024-09-13T11:23:36.581 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:36.581 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.581 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.581 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:23:36.581 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.717754+0000", 2024-09-13T11:23:36.581 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:36.581 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.581 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.581 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:23:36.581 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.717759+0000", 2024-09-13T11:23:36.582 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.04e-06 2024-09-13T11:23:36.582 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.582 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.582 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:23:36.582 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.717820+0000", 2024-09-13T11:23:36.582 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.0900999999999998e-05 2024-09-13T11:23:36.582 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.583 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.583 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:23:36.583 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.717854+0000", 2024-09-13T11:23:36.583 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.3797000000000002e-05 2024-09-13T11:23:36.583 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.583 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.583 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:23:36.583 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.717871+0000", 2024-09-13T11:23:36.583 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.6898999999999999e-05 2024-09-13T11:23:36.583 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.583 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.583 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:23:36.583 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.717993+0000", 2024-09-13T11:23:36.583 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00012248399999999999 2024-09-13T11:23:36.584 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.584 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.584 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:23:36.584 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.718222+0000", 2024-09-13T11:23:36.584 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00022848500000000001 2024-09-13T11:23:36.584 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.584 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.584 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 1", 2024-09-13T11:23:36.584 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.718447+0000", 2024-09-13T11:23:36.584 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00022588400000000001 2024-09-13T11:23:36.584 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.584 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.584 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:23:36.585 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.720604+0000", 2024-09-13T11:23:36.585 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0021561430000000001 2024-09-13T11:23:36.585 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.585 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.585 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:36.585 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.820012+0000", 2024-09-13T11:23:36.585 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.099408924999999995 2024-09-13T11:23:36.585 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.585 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.585 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:23:36.585 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.820097+0000", 2024-09-13T11:23:36.585 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.4147999999999995e-05 2024-09-13T11:23:36.585 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.585 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.586 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:23:36.586 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.820167+0000", 2024-09-13T11:23:36.586 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.0178000000000005e-05 2024-09-13T11:23:36.586 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:36.586 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:23:36.586 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:36.586 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.586 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.586 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop_reply(client.4665.0:25432 3.8 e56/40 ondisk, result = 0)", 2024-09-13T11:23:36.586 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:04.721952+0000", 2024-09-13T11:23:36.586 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 31.83406037, 2024-09-13T11:23:36.586 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.098227587000000005, 2024-09-13T11:23:36.586 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:23:36.586 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:23:36.587 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:23:36.587 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:23:36.587 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.587 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:23:36.587 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.721952+0000", 2024-09-13T11:23:36.587 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:36.587 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.587 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.587 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:23:36.587 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.721952+0000", 2024-09-13T11:23:36.587 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:36.587 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.587 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.587 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:23:36.588 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.721956+0000", 2024-09-13T11:23:36.588 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.8130000000000002e-06 2024-09-13T11:23:36.588 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.588 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.588 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:23:36.588 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.721967+0000", 2024-09-13T11:23:36.588 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1229e-05 2024-09-13T11:23:36.588 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.588 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.588 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:23:36.588 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.819880+0000", 2024-09-13T11:23:36.588 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.097912274999999993 2024-09-13T11:23:36.589 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.589 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.589 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:23:36.589 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.819894+0000", 2024-09-13T11:23:36.589 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.416e-05 2024-09-13T11:23:36.589 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.589 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.589 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:23:36.589 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.819993+0000", 2024-09-13T11:23:36.589 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.8859999999999999e-05 2024-09-13T11:23:36.589 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.589 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.589 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:23:36.589 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.820005+0000", 2024-09-13T11:23:36.590 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1960000000000001e-05 2024-09-13T11:23:36.590 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.590 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.590 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:23:36.590 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.820180+0000", 2024-09-13T11:23:36.590 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00017529000000000001 2024-09-13T11:23:36.590 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:36.590 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:23:36.590 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:36.590 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.590 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.590 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:30833 3.8 3:1c203f8d:::benchmark_data_smithi177_31116_object30832:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e59)", 2024-09-13T11:23:36.590 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:09.506656+0000", 2024-09-13T11:23:36.591 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 27.049356828000001, 2024-09-13T11:23:36.591 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.224862282, 2024-09-13T11:23:36.591 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:23:36.591 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:23:36.591 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:36.591 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:23:36.591 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:23:36.591 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:36.591 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 30833 2024-09-13T11:23:36.591 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.591 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:23:36.591 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.591 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:23:36.591 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.506656+0000", 2024-09-13T11:23:36.592 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:36.592 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.592 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.592 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:23:36.592 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.506656+0000", 2024-09-13T11:23:36.592 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:36.592 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.592 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.592 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:23:36.592 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.506660+0000", 2024-09-13T11:23:36.592 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.6e-06 2024-09-13T11:23:36.592 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.592 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.593 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:23:36.593 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.506717+0000", 2024-09-13T11:23:36.593 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.6131999999999997e-05 2024-09-13T11:23:36.593 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.593 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.593 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:23:36.593 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.506751+0000", 2024-09-13T11:23:36.593 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.4746000000000002e-05 2024-09-13T11:23:36.593 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.593 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.593 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:23:36.593 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.506774+0000", 2024-09-13T11:23:36.593 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.2636000000000001e-05 2024-09-13T11:23:36.593 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.594 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.594 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:23:36.594 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.506878+0000", 2024-09-13T11:23:36.594 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000104057 2024-09-13T11:23:36.594 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.594 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.594 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:23:36.594 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.507052+0000", 2024-09-13T11:23:36.594 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00017402800000000001 2024-09-13T11:23:36.594 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.594 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.594 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 1", 2024-09-13T11:23:36.594 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.507217+0000", 2024-09-13T11:23:36.595 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00016516 2024-09-13T11:23:36.595 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.595 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.595 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:23:36.595 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.508992+0000", 2024-09-13T11:23:36.595 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0017743990000000001 2024-09-13T11:23:36.595 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.595 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.595 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:36.595 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.731429+0000", 2024-09-13T11:23:36.595 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.22243718000000001 2024-09-13T11:23:36.595 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.595 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.595 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:23:36.596 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.731479+0000", 2024-09-13T11:23:36.596 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.0146999999999999e-05 2024-09-13T11:23:36.596 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.596 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.596 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:23:36.596 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.731518+0000", 2024-09-13T11:23:36.596 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.9196999999999998e-05 2024-09-13T11:23:36.596 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:36.596 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:23:36.596 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:36.596 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.596 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.596 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop_reply(client.4665.0:30833 3.8 e59/40 ondisk, result = 0)", 2024-09-13T11:23:36.597 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:09.508907+0000", 2024-09-13T11:23:36.597 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 27.047105733999999, 2024-09-13T11:23:36.597 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.222618862, 2024-09-13T11:23:36.597 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:23:36.597 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:23:36.597 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:23:36.597 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:23:36.597 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.597 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:23:36.597 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.508907+0000", 2024-09-13T11:23:36.597 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:36.597 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.597 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.597 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:23:36.598 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.508907+0000", 2024-09-13T11:23:36.598 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:36.598 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.598 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.598 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:23:36.598 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.508910+0000", 2024-09-13T11:23:36.598 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.993e-06 2024-09-13T11:23:36.598 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.598 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.598 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:23:36.598 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.508918+0000", 2024-09-13T11:23:36.598 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.7000000000000008e-06 2024-09-13T11:23:36.598 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.598 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.599 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:23:36.599 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.731341+0000", 2024-09-13T11:23:36.599 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.22242308699999999 2024-09-13T11:23:36.599 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.599 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.599 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:23:36.599 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.731350+0000", 2024-09-13T11:23:36.599 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.3179999999999996e-06 2024-09-13T11:23:36.599 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.599 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.599 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:23:36.599 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.731418+0000", 2024-09-13T11:23:36.599 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.7586e-05 2024-09-13T11:23:36.600 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.600 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.600 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:23:36.600 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.731424+0000", 2024-09-13T11:23:36.600 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.844e-06 2024-09-13T11:23:36.600 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.600 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.600 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:23:36.600 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.731526+0000", 2024-09-13T11:23:36.600 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000101334 2024-09-13T11:23:36.601 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:36.601 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:23:36.601 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:36.601 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.601 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.601 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:48977 3.8 3:1030343e:::benchmark_data_smithi177_31116_object48976:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e71)", 2024-09-13T11:23:36.601 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:25.517299+0000", 2024-09-13T11:23:36.601 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 11.03871344, 2024-09-13T11:23:36.601 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.14514738599999999, 2024-09-13T11:23:36.601 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:23:36.601 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:23:36.601 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:36.601 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:23:36.602 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:23:36.602 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:36.602 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 48977 2024-09-13T11:23:36.602 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.602 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:23:36.602 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.602 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:23:36.602 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.517299+0000", 2024-09-13T11:23:36.602 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:36.602 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.602 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.602 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:23:36.602 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.517299+0000", 2024-09-13T11:23:36.603 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:36.603 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.603 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.603 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:23:36.603 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.517301+0000", 2024-09-13T11:23:36.603 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.7060000000000001e-06 2024-09-13T11:23:36.603 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.603 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.603 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:23:36.603 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.517354+0000", 2024-09-13T11:23:36.603 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.3148e-05 2024-09-13T11:23:36.603 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.604 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.604 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:23:36.604 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.517365+0000", 2024-09-13T11:23:36.604 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1389000000000001e-05 2024-09-13T11:23:36.604 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.604 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.604 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:23:36.604 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.517374+0000", 2024-09-13T11:23:36.604 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.6740000000000003e-06 2024-09-13T11:23:36.604 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.604 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.604 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:23:36.605 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.517454+0000", 2024-09-13T11:23:36.605 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.9968000000000004e-05 2024-09-13T11:23:36.605 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.605 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.605 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:23:36.605 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.517561+0000", 2024-09-13T11:23:36.605 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00010655 2024-09-13T11:23:36.605 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.605 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.605 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 1", 2024-09-13T11:23:36.605 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.517664+0000", 2024-09-13T11:23:36.605 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000103409 2024-09-13T11:23:36.605 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.605 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.606 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:23:36.606 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.521381+0000", 2024-09-13T11:23:36.606 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0037174019999999999 2024-09-13T11:23:36.606 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.606 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.606 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:36.606 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.662307+0000", 2024-09-13T11:23:36.606 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.14092524000000001 2024-09-13T11:23:36.606 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.606 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.606 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:23:36.606 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.662382+0000", 2024-09-13T11:23:36.607 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.5331000000000001e-05 2024-09-13T11:23:36.607 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.607 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.607 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:23:36.607 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.662447+0000", 2024-09-13T11:23:36.607 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.4568999999999998e-05 2024-09-13T11:23:36.607 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:36.607 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:23:36.607 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:36.607 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.607 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.607 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:49109 3.8 3:1587435f:::benchmark_data_smithi177_31116_object49108:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e71)", 2024-09-13T11:23:36.607 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:25.659607+0000", 2024-09-13T11:23:36.607 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 10.896405356000001, 2024-09-13T11:23:36.608 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.077972397999999998, 2024-09-13T11:23:36.608 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:23:36.608 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:23:36.608 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:36.608 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:23:36.608 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:23:36.608 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:36.608 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 49109 2024-09-13T11:23:36.608 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.608 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:23:36.608 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.608 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:23:36.608 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.659607+0000", 2024-09-13T11:23:36.609 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:36.609 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.609 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.609 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:23:36.609 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.659607+0000", 2024-09-13T11:23:36.609 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:36.609 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.609 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.609 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:23:36.609 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.659612+0000", 2024-09-13T11:23:36.609 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.5630000000000002e-06 2024-09-13T11:23:36.609 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.609 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.610 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:23:36.610 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.659667+0000", 2024-09-13T11:23:36.610 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.4898000000000002e-05 2024-09-13T11:23:36.610 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.610 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.610 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:23:36.610 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.659696+0000", 2024-09-13T11:23:36.610 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.9776000000000001e-05 2024-09-13T11:23:36.610 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.610 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.610 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:23:36.610 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.659719+0000", 2024-09-13T11:23:36.610 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.2106000000000001e-05 2024-09-13T11:23:36.610 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.611 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.611 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:23:36.611 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.659883+0000", 2024-09-13T11:23:36.611 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00016433499999999999 2024-09-13T11:23:36.611 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.611 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.611 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:23:36.611 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.660207+0000", 2024-09-13T11:23:36.611 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00032428099999999999 2024-09-13T11:23:36.611 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.611 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.611 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 1", 2024-09-13T11:23:36.611 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.660564+0000", 2024-09-13T11:23:36.612 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00035704500000000001 2024-09-13T11:23:36.612 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.612 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.612 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:23:36.612 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.662883+0000", 2024-09-13T11:23:36.612 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0023183829999999998 2024-09-13T11:23:36.612 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.612 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.612 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:36.612 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.737361+0000", 2024-09-13T11:23:36.612 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.074478279999999994 2024-09-13T11:23:36.612 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.613 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.613 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:23:36.613 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.737486+0000", 2024-09-13T11:23:36.613 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00012487500000000001 2024-09-13T11:23:36.613 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.613 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.613 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:23:36.613 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.737580+0000", 2024-09-13T11:23:36.613 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.3856000000000004e-05 2024-09-13T11:23:36.613 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:36.613 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:23:36.613 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:36.613 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.613 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.614 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:49280 3.8 3:11086105:::benchmark_data_smithi177_31116_object49279:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e71)", 2024-09-13T11:23:36.614 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:25.791647+0000", 2024-09-13T11:23:36.614 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 10.764365431, 2024-09-13T11:23:36.614 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.083081314000000003, 2024-09-13T11:23:36.614 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:23:36.614 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:23:36.614 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:36.614 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:23:36.614 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:23:36.614 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:36.614 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 49280 2024-09-13T11:23:36.614 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.614 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:23:36.614 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.615 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:23:36.615 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.791647+0000", 2024-09-13T11:23:36.615 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:36.615 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.615 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.615 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:23:36.615 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.791647+0000", 2024-09-13T11:23:36.615 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:36.615 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.615 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.615 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:23:36.615 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.791652+0000", 2024-09-13T11:23:36.615 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.2220000000000001e-06 2024-09-13T11:23:36.615 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.616 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.616 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:23:36.616 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.791707+0000", 2024-09-13T11:23:36.616 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.5053999999999998e-05 2024-09-13T11:23:36.616 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.616 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.616 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:23:36.616 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.791731+0000", 2024-09-13T11:23:36.616 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.3261e-05 2024-09-13T11:23:36.616 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.616 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.616 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:23:36.616 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.791743+0000", 2024-09-13T11:23:36.616 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2037e-05 2024-09-13T11:23:36.617 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.617 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.617 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:23:36.617 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.793450+0000", 2024-09-13T11:23:36.617 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0017074830000000001 2024-09-13T11:23:36.617 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.617 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.617 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:23:36.617 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.793671+0000", 2024-09-13T11:23:36.617 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000220891 2024-09-13T11:23:36.617 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.617 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.617 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 1", 2024-09-13T11:23:36.618 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.793883+0000", 2024-09-13T11:23:36.618 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00021190200000000001 2024-09-13T11:23:36.618 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.618 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.618 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:23:36.618 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.795973+0000", 2024-09-13T11:23:36.618 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0020898380000000001 2024-09-13T11:23:36.618 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.618 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.619 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:36.619 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.874651+0000", 2024-09-13T11:23:36.619 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.078678389000000001 2024-09-13T11:23:36.619 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.619 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.619 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:23:36.619 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.874694+0000", 2024-09-13T11:23:36.619 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.3071999999999997e-05 2024-09-13T11:23:36.619 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.619 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.619 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:23:36.619 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.874728+0000", 2024-09-13T11:23:36.619 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.4165000000000002e-05 2024-09-13T11:23:36.619 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:36.620 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:23:36.620 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:36.620 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.620 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.620 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:49373 3.8 3:17a23476:::benchmark_data_smithi177_31116_object49372:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e71)", 2024-09-13T11:23:36.620 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:25.858495+0000", 2024-09-13T11:23:36.620 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 10.697517251000001, 2024-09-13T11:23:36.620 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.088819487000000003, 2024-09-13T11:23:36.620 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:23:36.620 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:23:36.620 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:36.620 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:23:36.620 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:23:36.621 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:36.621 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 49373 2024-09-13T11:23:36.621 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.621 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:23:36.621 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.621 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:23:36.621 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.858495+0000", 2024-09-13T11:23:36.621 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:36.621 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.621 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.621 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:23:36.621 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.858495+0000", 2024-09-13T11:23:36.621 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:36.622 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.622 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.622 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:23:36.622 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.858509+0000", 2024-09-13T11:23:36.622 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3195e-05 2024-09-13T11:23:36.622 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.622 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.622 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:23:36.622 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.858546+0000", 2024-09-13T11:23:36.622 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.7842999999999997e-05 2024-09-13T11:23:36.622 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.622 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.622 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:23:36.622 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.858562+0000", 2024-09-13T11:23:36.623 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.6030000000000001e-05 2024-09-13T11:23:36.623 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.623 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.623 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:23:36.623 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.858571+0000", 2024-09-13T11:23:36.623 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.4970000000000001e-06 2024-09-13T11:23:36.623 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.623 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.623 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:23:36.623 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.858724+0000", 2024-09-13T11:23:36.623 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00015293500000000001 2024-09-13T11:23:36.623 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.624 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.624 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:23:36.624 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.858934+0000", 2024-09-13T11:23:36.624 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00020973500000000001 2024-09-13T11:23:36.624 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.624 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.624 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 1", 2024-09-13T11:23:36.624 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.859197+0000", 2024-09-13T11:23:36.624 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00026385999999999998 2024-09-13T11:23:36.624 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.624 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.624 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:23:36.624 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.861435+0000", 2024-09-13T11:23:36.625 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.002237555 2024-09-13T11:23:36.625 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.625 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.625 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:36.625 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.947194+0000", 2024-09-13T11:23:36.625 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.085758501000000001 2024-09-13T11:23:36.625 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.625 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.625 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:23:36.625 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.947260+0000", 2024-09-13T11:23:36.625 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.6675999999999997e-05 2024-09-13T11:23:36.625 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.625 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.626 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:23:36.626 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.947315+0000", 2024-09-13T11:23:36.626 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.4660000000000002e-05 2024-09-13T11:23:36.626 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:36.626 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:23:36.626 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:36.626 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.626 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.626 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPush(3.10 72/70 [PushOp(3:0f8adc10:::benchmark_data_smithi177_31116_object5718:head, version: 45'362, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:0f8adc10:::benchmark_data_smithi177_31116_object5718:head@45'362, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:23:36.626 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:26.392100+0000", 2024-09-13T11:23:36.626 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 10.163912506999999, 2024-09-13T11:23:36.626 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.079916466000000005, 2024-09-13T11:23:36.626 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:23:36.626 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:23:36.627 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:23:36.627 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:23:36.627 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.627 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:23:36.627 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:26.392100+0000", 2024-09-13T11:23:36.627 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:36.627 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.627 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.627 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:23:36.627 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:26.392100+0000", 2024-09-13T11:23:36.627 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:36.627 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.627 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.628 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:23:36.628 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:26.392101+0000", 2024-09-13T11:23:36.628 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.226e-06 2024-09-13T11:23:36.628 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.628 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.628 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:23:36.628 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:26.392183+0000", 2024-09-13T11:23:36.628 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.1460999999999998e-05 2024-09-13T11:23:36.628 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.628 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.628 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:23:36.628 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:26.470887+0000", 2024-09-13T11:23:36.628 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.078704619000000003 2024-09-13T11:23:36.628 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.629 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.629 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:23:36.629 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:26.470901+0000", 2024-09-13T11:23:36.629 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3191e-05 2024-09-13T11:23:36.629 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.629 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.629 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:23:36.629 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:26.471023+0000", 2024-09-13T11:23:36.629 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00012229 2024-09-13T11:23:36.629 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.629 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.629 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:23:36.629 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:26.471045+0000", 2024-09-13T11:23:36.629 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.1670000000000001e-05 2024-09-13T11:23:36.630 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.630 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.630 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:23:36.630 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:26.472017+0000", 2024-09-13T11:23:36.630 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00097200900000000002 2024-09-13T11:23:36.630 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:36.630 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:23:36.630 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:36.630 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.630 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.630 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:54234 3.8 3:10c2ba28:::benchmark_data_smithi177_31116_object54233:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e74)", 2024-09-13T11:23:36.630 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:30.490281+0000", 2024-09-13T11:23:36.630 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 6.0657314649999998, 2024-09-13T11:23:36.631 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.088032962000000006, 2024-09-13T11:23:36.631 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:23:36.631 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:23:36.631 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:36.631 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:23:36.631 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:23:36.631 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:36.631 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 54234 2024-09-13T11:23:36.631 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.631 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:23:36.631 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.631 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:23:36.631 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.490281+0000", 2024-09-13T11:23:36.632 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:36.632 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.632 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.632 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:23:36.632 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.490281+0000", 2024-09-13T11:23:36.632 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:36.632 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.632 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.632 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:23:36.632 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.490283+0000", 2024-09-13T11:23:36.632 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.26e-06 2024-09-13T11:23:36.632 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.632 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.632 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:23:36.633 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.490360+0000", 2024-09-13T11:23:36.633 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.6124000000000005e-05 2024-09-13T11:23:36.633 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.633 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.633 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:23:36.633 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.490441+0000", 2024-09-13T11:23:36.633 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.1478999999999997e-05 2024-09-13T11:23:36.633 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.633 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.633 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:23:36.633 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.490468+0000", 2024-09-13T11:23:36.633 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.7390999999999999e-05 2024-09-13T11:23:36.633 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.634 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.634 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:23:36.634 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.490552+0000", 2024-09-13T11:23:36.634 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.3949999999999994e-05 2024-09-13T11:23:36.634 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.634 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.634 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:23:36.634 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.490677+0000", 2024-09-13T11:23:36.634 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000124924 2024-09-13T11:23:36.634 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.634 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.634 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 1", 2024-09-13T11:23:36.634 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.490875+0000", 2024-09-13T11:23:36.634 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00019803600000000001 2024-09-13T11:23:36.635 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.635 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.635 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:23:36.635 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.492928+0000", 2024-09-13T11:23:36.635 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0020524240000000002 2024-09-13T11:23:36.635 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.635 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.635 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:36.635 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.578160+0000", 2024-09-13T11:23:36.635 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.085232174999999993 2024-09-13T11:23:36.635 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.635 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.636 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:23:36.636 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.578245+0000", 2024-09-13T11:23:36.636 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.5032999999999997e-05 2024-09-13T11:23:36.636 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.636 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.636 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:23:36.636 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.578314+0000", 2024-09-13T11:23:36.636 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.9165999999999995e-05 2024-09-13T11:23:36.636 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:36.636 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:23:36.636 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:36.636 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.636 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.637 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.4665.0:54351 3.10 e74/70 3:085f283e:::benchmark_data_smithi177_31116_object54350:head v 74'3087, mlcod=49'780)", 2024-09-13T11:23:36.637 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:30.599667+0000", 2024-09-13T11:23:36.637 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 5.956345228, 2024-09-13T11:23:36.637 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.16617266999999999, 2024-09-13T11:23:36.637 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:23:36.637 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:23:36.637 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:36.637 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:23:36.637 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.637 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:23:36.637 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.599667+0000", 2024-09-13T11:23:36.637 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:36.637 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.638 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.638 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:23:36.638 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.599667+0000", 2024-09-13T11:23:36.638 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:36.638 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.638 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.638 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:23:36.638 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.599669+0000", 2024-09-13T11:23:36.638 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3599999999999999e-06 2024-09-13T11:23:36.638 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.638 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.638 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:23:36.638 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.599712+0000", 2024-09-13T11:23:36.639 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.3723000000000001e-05 2024-09-13T11:23:36.639 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.639 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.639 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:23:36.639 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.762241+0000", 2024-09-13T11:23:36.639 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.16252863400000001 2024-09-13T11:23:36.639 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.639 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.639 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:23:36.639 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.762255+0000", 2024-09-13T11:23:36.639 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3414999999999999e-05 2024-09-13T11:23:36.639 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.639 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.639 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:23:36.640 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.762329+0000", 2024-09-13T11:23:36.640 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.4735999999999997e-05 2024-09-13T11:23:36.640 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.640 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.640 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:23:36.640 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.762358+0000", 2024-09-13T11:23:36.640 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.8308000000000001e-05 2024-09-13T11:23:36.640 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.640 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.640 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:23:36.640 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.765770+0000", 2024-09-13T11:23:36.640 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0034126590000000002 2024-09-13T11:23:36.640 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.641 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.641 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:23:36.641 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.765840+0000", 2024-09-13T11:23:36.641 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.9834999999999998e-05 2024-09-13T11:23:36.641 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:36.641 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:23:36.641 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:36.641 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.641 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.641 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPush(3.10 74/70 [PushOp(3:0b3e58bf:::benchmark_data_smithi177_31116_object12372:head, version: 49'782, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:0b3e58bf:::benchmark_data_smithi177_31116_object12372:head@49'782, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:23:36.641 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:30.608892+0000", 2024-09-13T11:23:36.641 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 5.9471208740000003, 2024-09-13T11:23:36.641 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.15687285300000001, 2024-09-13T11:23:36.641 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:23:36.642 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:23:36.642 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:23:36.642 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:23:36.642 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.642 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:23:36.642 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.608892+0000", 2024-09-13T11:23:36.642 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:36.642 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.642 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.642 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:23:36.642 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.608892+0000", 2024-09-13T11:23:36.642 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:36.642 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.642 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.643 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:23:36.643 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.608893+0000", 2024-09-13T11:23:36.643 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.666e-06 2024-09-13T11:23:36.643 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.643 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.643 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:23:36.643 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.609008+0000", 2024-09-13T11:23:36.643 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00011411 2024-09-13T11:23:36.643 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.643 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.643 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:23:36.643 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.763870+0000", 2024-09-13T11:23:36.643 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.15486217799999999 2024-09-13T11:23:36.644 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.644 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.644 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:23:36.644 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.763904+0000", 2024-09-13T11:23:36.644 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.4082000000000003e-05 2024-09-13T11:23:36.644 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.644 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.644 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:23:36.644 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.764247+0000", 2024-09-13T11:23:36.644 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00034281900000000001 2024-09-13T11:23:36.644 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.644 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.644 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:23:36.645 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.764270+0000", 2024-09-13T11:23:36.645 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.3244e-05 2024-09-13T11:23:36.645 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.645 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.645 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:23:36.645 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.765765+0000", 2024-09-13T11:23:36.645 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001494754 2024-09-13T11:23:36.645 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:36.645 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:23:36.645 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:36.645 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.645 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.645 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPush(3.10 76/70 [PushOp(3:0e7d2700:::benchmark_data_smithi177_31116_object17406:head, version: 52'1075, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:0e7d2700:::benchmark_data_smithi177_31116_object17406:head@52'1075, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:23:36.646 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:32.206198+0000", 2024-09-13T11:23:36.646 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 4.3498150249999998, 2024-09-13T11:23:36.646 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.25193395000000002, 2024-09-13T11:23:36.646 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:23:36.646 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:23:36.646 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:23:36.646 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:23:36.646 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.646 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:23:36.646 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.206198+0000", 2024-09-13T11:23:36.646 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:36.646 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.647 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.647 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:23:36.647 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.206198+0000", 2024-09-13T11:23:36.647 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:36.647 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.647 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.647 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:23:36.647 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.206199+0000", 2024-09-13T11:23:36.647 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.46e-06 2024-09-13T11:23:36.647 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.647 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.647 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:23:36.647 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.206250+0000", 2024-09-13T11:23:36.648 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.113e-05 2024-09-13T11:23:36.648 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.648 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.648 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:23:36.648 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.456042+0000", 2024-09-13T11:23:36.648 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.24979182999999999 2024-09-13T11:23:36.648 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.648 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.648 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:23:36.648 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.456084+0000", 2024-09-13T11:23:36.648 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.1814000000000003e-05 2024-09-13T11:23:36.648 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.648 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.649 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:23:36.649 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.456342+0000", 2024-09-13T11:23:36.649 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000258404 2024-09-13T11:23:36.649 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.649 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.649 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:23:36.649 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.456367+0000", 2024-09-13T11:23:36.649 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.5099000000000001e-05 2024-09-13T11:23:36.649 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.649 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.649 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:23:36.649 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.458132+0000", 2024-09-13T11:23:36.649 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0017642129999999999 2024-09-13T11:23:36.650 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:36.650 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:23:36.650 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:36.650 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.650 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.650 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPush(3.10 76/70 [PushOp(3:0aaacffd:::benchmark_data_smithi177_31116_object17487:head, version: 53'1081, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:0aaacffd:::benchmark_data_smithi177_31116_object17487:head@53'1081, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:23:36.650 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:32.493092+0000", 2024-09-13T11:23:36.650 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 4.062920611, 2024-09-13T11:23:36.650 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.085504803000000004, 2024-09-13T11:23:36.650 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:23:36.650 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:23:36.650 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:23:36.651 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:23:36.651 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.651 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:23:36.651 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.493092+0000", 2024-09-13T11:23:36.651 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:36.651 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.651 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.651 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:23:36.651 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.493092+0000", 2024-09-13T11:23:36.651 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:36.651 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.651 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.651 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:23:36.652 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.493095+0000", 2024-09-13T11:23:36.652 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.7470000000000001e-06 2024-09-13T11:23:36.652 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.652 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.652 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:23:36.652 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.493154+0000", 2024-09-13T11:23:36.652 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.9552999999999997e-05 2024-09-13T11:23:36.652 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.652 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.652 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:23:36.652 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.576353+0000", 2024-09-13T11:23:36.652 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.083198421999999994 2024-09-13T11:23:36.652 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.653 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.653 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:23:36.653 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.576404+0000", 2024-09-13T11:23:36.653 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.0861e-05 2024-09-13T11:23:36.653 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.653 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.653 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:23:36.653 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.576664+0000", 2024-09-13T11:23:36.653 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000259981 2024-09-13T11:23:36.653 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.653 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.653 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:23:36.653 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.576699+0000", 2024-09-13T11:23:36.654 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.5104999999999997e-05 2024-09-13T11:23:36.654 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.654 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.654 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:23:36.654 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.578597+0000", 2024-09-13T11:23:36.654 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0018981340000000001 2024-09-13T11:23:36.654 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:36.654 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:23:36.654 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:36.654 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.654 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.654 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPush(3.10 76/70 [PushOp(3:0eee8bcf:::benchmark_data_smithi177_31116_object17589:head, version: 53'1086, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:0eee8bcf:::benchmark_data_smithi177_31116_object17589:head@53'1086, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:23:36.654 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:32.585183+0000", 2024-09-13T11:23:36.655 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 3.9708291309999999, 2024-09-13T11:23:36.655 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.16696202199999999, 2024-09-13T11:23:36.655 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:23:36.655 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:23:36.655 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:23:36.655 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:23:36.655 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.655 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:23:36.655 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.585183+0000", 2024-09-13T11:23:36.655 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:36.655 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.655 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.655 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:23:36.656 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.585183+0000", 2024-09-13T11:23:36.656 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:36.656 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.656 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.656 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:23:36.656 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.585187+0000", 2024-09-13T11:23:36.656 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.8480000000000003e-06 2024-09-13T11:23:36.656 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.656 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.656 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:23:36.656 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.585264+0000", 2024-09-13T11:23:36.656 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.6327000000000003e-05 2024-09-13T11:23:36.657 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.657 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.657 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:23:36.657 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.750459+0000", 2024-09-13T11:23:36.657 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.16519551900000001 2024-09-13T11:23:36.657 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.657 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.657 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:23:36.657 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.750480+0000", 2024-09-13T11:23:36.657 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.0984000000000001e-05 2024-09-13T11:23:36.657 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.657 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.657 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:23:36.658 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.750633+0000", 2024-09-13T11:23:36.658 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000153148 2024-09-13T11:23:36.658 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.658 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.658 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:23:36.658 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.750652+0000", 2024-09-13T11:23:36.658 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.8981e-05 2024-09-13T11:23:36.658 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.658 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.658 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:23:36.658 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.752145+0000", 2024-09-13T11:23:36.658 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001493215 2024-09-13T11:23:36.659 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:36.659 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:23:36.659 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:36.659 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.659 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.659 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPush(3.10 76/70 [PushOp(3:0e7bfa69:::benchmark_data_smithi177_31116_object17885:head, version: 53'1107, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:0e7bfa69:::benchmark_data_smithi177_31116_object17885:head@53'1107, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:23:36.659 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:32.834900+0000", 2024-09-13T11:23:36.659 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 3.7211129239999998, 2024-09-13T11:23:36.659 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.087748388999999996, 2024-09-13T11:23:36.659 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:23:36.659 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:23:36.659 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:23:36.659 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:23:36.660 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.660 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:23:36.660 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.834900+0000", 2024-09-13T11:23:36.660 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:36.660 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.660 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.660 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:23:36.660 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.834900+0000", 2024-09-13T11:23:36.660 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:36.660 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.660 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.660 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:23:36.660 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.834908+0000", 2024-09-13T11:23:36.661 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.9610000000000005e-06 2024-09-13T11:23:36.661 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.661 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.661 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:23:36.661 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.835005+0000", 2024-09-13T11:23:36.661 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.7621000000000006e-05 2024-09-13T11:23:36.661 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.661 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.661 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:23:36.661 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.920337+0000", 2024-09-13T11:23:36.661 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.085332080000000005 2024-09-13T11:23:36.661 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.661 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.662 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:23:36.662 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.920376+0000", 2024-09-13T11:23:36.662 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.8461999999999999e-05 2024-09-13T11:23:36.662 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.662 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.662 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:23:36.662 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.920596+0000", 2024-09-13T11:23:36.662 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000219707 2024-09-13T11:23:36.662 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.662 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.662 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:23:36.662 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.920631+0000", 2024-09-13T11:23:36.663 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.5837000000000003e-05 2024-09-13T11:23:36.663 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.663 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.663 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:23:36.663 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.922648+0000", 2024-09-13T11:23:36.663 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.002016721 2024-09-13T11:23:36.663 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:36.663 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:23:36.663 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:36.663 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:36.663 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:23:36.663 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-13T11:23:36.665 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-09-13T11:23:36.668 INFO:tasks.ceph.osd.3.smithi177.stderr:2024-09-13T11:23:36.660+0000 7f10adba4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26574) UID: 0 2024-09-13T11:23:36.766 INFO:tasks.ceph.osd.14.smithi162.stderr:2024-09-13T11:23:36.761+0000 7f235c0db640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26319) UID: 0 2024-09-13T11:23:36.824 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-13T11:23:36.824 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [ 2024-09-13T11:23:36.824 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.824 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPush(3.17 79/70 [PushOp(3:e84d09ba:::benchmark_data_smithi177_31116_object30546:head, version: 59'1839, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:e84d09ba:::benchmark_data_smithi177_31116_object30546:head@59'1839, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:23:36.825 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:36.821856+0000", 2024-09-13T11:23:36.825 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 0.0013203729999999999, 2024-09-13T11:23:36.825 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0013772719999999999, 2024-09-13T11:23:36.825 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:23:36.825 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:23:36.825 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:23:36.825 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:23:36.825 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.825 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:23:36.825 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:36.821856+0000", 2024-09-13T11:23:36.825 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:36.825 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.826 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.826 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:23:36.826 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:36.821856+0000", 2024-09-13T11:23:36.826 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:36.826 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.826 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.826 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:23:36.826 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:36.821858+0000", 2024-09-13T11:23:36.826 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.852e-06 2024-09-13T11:23:36.826 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.826 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.826 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:23:36.826 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:36.821920+0000", 2024-09-13T11:23:36.826 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.2135000000000001e-05 2024-09-13T11:23:36.827 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.827 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.827 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:23:36.827 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:36.821958+0000", 2024-09-13T11:23:36.827 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.7731000000000002e-05 2024-09-13T11:23:36.827 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.827 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.827 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:23:36.827 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:36.821970+0000", 2024-09-13T11:23:36.827 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2609000000000001e-05 2024-09-13T11:23:36.827 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.827 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.827 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:23:36.827 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:36.822050+0000", 2024-09-13T11:23:36.828 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.9177000000000004e-05 2024-09-13T11:23:36.828 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:36.828 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:36.828 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:23:36.828 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:36.822059+0000", 2024-09-13T11:23:36.828 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.5980000000000003e-06 2024-09-13T11:23:36.828 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:36.828 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:23:36.828 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:36.828 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:36.828 INFO:teuthology.orchestra.run.smithi162.stdout: ], 2024-09-13T11:23:36.828 INFO:teuthology.orchestra.run.smithi162.stdout: "num_ops": 1 2024-09-13T11:23:36.828 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-13T11:23:36.847 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-09-13T11:23:36.868 INFO:tasks.ceph.osd.15.smithi177.stderr:2024-09-13T11:23:36.860+0000 7f9d6ce77640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26578) UID: 0 2024-09-13T11:23:36.948 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-13T11:23:36.966 INFO:tasks.ceph.osd.7.smithi177.stderr:2024-09-13T11:23:36.960+0000 7f897b51d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26575) UID: 0 2024-09-13T11:23:37.005 INFO:tasks.radosbench.radosbench.0.smithi177.stdout: 70 16 56892 56876 50.7767 63.3125 0.0152167 0.0196865 2024-09-13T11:23:37.037 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-13T11:23:37.037 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [], 2024-09-13T11:23:37.037 INFO:teuthology.orchestra.run.smithi162.stdout: "complaint_time": 30, 2024-09-13T11:23:37.037 INFO:teuthology.orchestra.run.smithi162.stdout: "num_blocked_ops": 0 2024-09-13T11:23:37.037 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-13T11:23:37.057 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-09-13T11:23:37.067 INFO:tasks.ceph.osd.4.smithi001.stderr:2024-09-13T11:23:37.060+0000 7ff8e0218640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26310) UID: 0 2024-09-13T11:23:37.168 INFO:tasks.ceph.osd.2.smithi162.stderr:2024-09-13T11:23:37.161+0000 7fe92b35a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26313) UID: 0 2024-09-13T11:23:37.208 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-13T11:23:37.208 INFO:teuthology.orchestra.run.smithi162.stdout: "size": 20, 2024-09-13T11:23:37.208 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 600, 2024-09-13T11:23:37.208 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [ 2024-09-13T11:23:37.208 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.208 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:25395 3.3 3:c1df8ccf:::benchmark_data_smithi177_31116_object25394:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e55)", 2024-09-13T11:23:37.208 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:04.465425+0000", 2024-09-13T11:23:37.208 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 32.739379591999999, 2024-09-13T11:23:37.208 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.18641877200000001, 2024-09-13T11:23:37.208 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:23:37.208 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:23:37.209 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:37.209 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:23:37.209 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:23:37.209 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:37.209 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 25395 2024-09-13T11:23:37.209 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.209 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:23:37.209 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.209 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:23:37.209 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.465425+0000", 2024-09-13T11:23:37.209 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:37.209 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.209 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.210 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:23:37.210 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.465425+0000", 2024-09-13T11:23:37.210 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:37.210 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.210 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.210 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:23:37.210 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.465427+0000", 2024-09-13T11:23:37.210 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.9970000000000001e-06 2024-09-13T11:23:37.210 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.210 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.210 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:23:37.210 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.465516+0000", 2024-09-13T11:23:37.210 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.8181000000000002e-05 2024-09-13T11:23:37.211 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.211 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.211 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:23:37.211 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.465534+0000", 2024-09-13T11:23:37.211 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.8204000000000001e-05 2024-09-13T11:23:37.211 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.211 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.211 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:23:37.211 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.465548+0000", 2024-09-13T11:23:37.211 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4195999999999999e-05 2024-09-13T11:23:37.211 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.211 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.211 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:23:37.212 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.468249+0000", 2024-09-13T11:23:37.212 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0027014729999999998 2024-09-13T11:23:37.212 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.212 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.212 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:23:37.212 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.468626+0000", 2024-09-13T11:23:37.212 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000376721 2024-09-13T11:23:37.212 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.212 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.212 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 7", 2024-09-13T11:23:37.212 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.469016+0000", 2024-09-13T11:23:37.212 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00038975000000000001 2024-09-13T11:23:37.212 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.213 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.213 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:23:37.213 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.471140+0000", 2024-09-13T11:23:37.213 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0021243400000000002 2024-09-13T11:23:37.213 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.213 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.213 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:37.213 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.651698+0000", 2024-09-13T11:23:37.213 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.18055781700000001 2024-09-13T11:23:37.213 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.213 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.213 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:23:37.213 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.651781+0000", 2024-09-13T11:23:37.213 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.3350999999999995e-05 2024-09-13T11:23:37.214 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.214 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.214 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:23:37.214 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.651844+0000", 2024-09-13T11:23:37.214 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.2742000000000003e-05 2024-09-13T11:23:37.214 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:37.214 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:23:37.214 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:37.214 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.214 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.214 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop_reply(client.4665.0:25395 3.3 e55/30 ondisk, result = 0)", 2024-09-13T11:23:37.214 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:04.471832+0000", 2024-09-13T11:23:37.214 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 32.732973289999997, 2024-09-13T11:23:37.214 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.18002319, 2024-09-13T11:23:37.215 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:23:37.215 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:23:37.215 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:23:37.215 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:23:37.215 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.215 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:23:37.215 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.471832+0000", 2024-09-13T11:23:37.215 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:37.215 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.215 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.216 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:23:37.216 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.471832+0000", 2024-09-13T11:23:37.216 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:37.216 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.216 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.216 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:23:37.216 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.471840+0000", 2024-09-13T11:23:37.216 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.7900000000000005e-06 2024-09-13T11:23:37.216 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.216 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.216 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:23:37.216 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.471850+0000", 2024-09-13T11:23:37.217 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.3899999999999999e-06 2024-09-13T11:23:37.217 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.217 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.217 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:23:37.217 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.651528+0000", 2024-09-13T11:23:37.217 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.17967789100000001 2024-09-13T11:23:37.217 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.217 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.217 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:23:37.217 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.651540+0000", 2024-09-13T11:23:37.217 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2e-05 2024-09-13T11:23:37.217 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.217 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.218 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:23:37.218 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.651676+0000", 2024-09-13T11:23:37.218 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000136033 2024-09-13T11:23:37.218 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.218 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.218 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:23:37.218 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.651689+0000", 2024-09-13T11:23:37.218 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.364e-05 2024-09-13T11:23:37.218 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.218 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.218 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:23:37.218 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.651855+0000", 2024-09-13T11:23:37.218 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000165446 2024-09-13T11:23:37.218 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:37.219 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:23:37.219 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:37.219 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.219 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.219 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:31782 3.3 3:c80d1443:::benchmark_data_smithi177_31116_object31781:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e59)", 2024-09-13T11:23:37.219 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:10.372548+0000", 2024-09-13T11:23:37.219 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 26.832256863000001, 2024-09-13T11:23:37.219 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.058275848999999998, 2024-09-13T11:23:37.219 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:23:37.219 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:23:37.219 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:37.219 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:23:37.219 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:23:37.220 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:37.220 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 31782 2024-09-13T11:23:37.220 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.220 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:23:37.220 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.220 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:23:37.220 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:10.372548+0000", 2024-09-13T11:23:37.220 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:37.220 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.221 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.221 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:23:37.221 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:10.372548+0000", 2024-09-13T11:23:37.221 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:37.221 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.221 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.221 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:23:37.221 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:10.372551+0000", 2024-09-13T11:23:37.221 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.948e-06 2024-09-13T11:23:37.221 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.221 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.221 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:23:37.221 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:10.372612+0000", 2024-09-13T11:23:37.221 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.1328000000000006e-05 2024-09-13T11:23:37.222 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.222 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.222 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:23:37.222 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:10.372635+0000", 2024-09-13T11:23:37.222 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.2580999999999999e-05 2024-09-13T11:23:37.222 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.222 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.222 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:23:37.222 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:10.372647+0000", 2024-09-13T11:23:37.222 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1635e-05 2024-09-13T11:23:37.222 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.222 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.222 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:23:37.223 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:10.372729+0000", 2024-09-13T11:23:37.223 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.2544000000000001e-05 2024-09-13T11:23:37.223 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.223 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.223 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:23:37.223 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:10.372888+0000", 2024-09-13T11:23:37.223 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000159116 2024-09-13T11:23:37.223 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.223 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.223 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 7", 2024-09-13T11:23:37.223 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:10.373063+0000", 2024-09-13T11:23:37.223 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00017481800000000001 2024-09-13T11:23:37.224 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.224 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.224 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:23:37.224 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:10.375475+0000", 2024-09-13T11:23:37.224 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0024119440000000001 2024-09-13T11:23:37.224 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.224 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.224 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:37.224 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:10.430755+0000", 2024-09-13T11:23:37.224 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.055280442999999999 2024-09-13T11:23:37.224 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.224 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.224 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:23:37.225 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:10.430794+0000", 2024-09-13T11:23:37.225 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.8457000000000001e-05 2024-09-13T11:23:37.225 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.225 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.225 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:23:37.225 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:10.430824+0000", 2024-09-13T11:23:37.225 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.0035e-05 2024-09-13T11:23:37.225 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:37.225 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:23:37.225 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:37.225 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.225 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.226 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop_reply(client.4665.0:31782 3.3 e59/30 ondisk, result = 0)", 2024-09-13T11:23:37.226 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:10.377024+0000", 2024-09-13T11:23:37.226 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 26.827781437999999, 2024-09-13T11:23:37.226 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.053807298000000003, 2024-09-13T11:23:37.226 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:23:37.226 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:23:37.226 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:23:37.226 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:23:37.226 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.226 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:23:37.226 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:10.377024+0000", 2024-09-13T11:23:37.226 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:37.226 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.227 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.227 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:23:37.227 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:10.377024+0000", 2024-09-13T11:23:37.227 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:37.227 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.227 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.227 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:23:37.227 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:10.377026+0000", 2024-09-13T11:23:37.227 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.6340000000000001e-06 2024-09-13T11:23:37.227 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.227 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.227 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:23:37.227 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:10.377033+0000", 2024-09-13T11:23:37.227 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.3000000000000004e-06 2024-09-13T11:23:37.228 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.228 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.228 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:23:37.228 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:10.430682+0000", 2024-09-13T11:23:37.228 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.053648269999999998 2024-09-13T11:23:37.228 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.228 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.228 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:23:37.228 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:10.430690+0000", 2024-09-13T11:23:37.228 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.7800000000000006e-06 2024-09-13T11:23:37.228 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.229 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.229 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:23:37.229 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:10.430747+0000", 2024-09-13T11:23:37.229 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.6168000000000002e-05 2024-09-13T11:23:37.229 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.229 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.229 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:23:37.229 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:10.430752+0000", 2024-09-13T11:23:37.229 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.1510000000000003e-06 2024-09-13T11:23:37.229 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.229 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.230 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:23:37.230 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:10.430831+0000", 2024-09-13T11:23:37.230 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.8994999999999998e-05 2024-09-13T11:23:37.230 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:37.230 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:23:37.230 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:37.230 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.230 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.230 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:32023 3.3 3:c1d48a65:::benchmark_data_smithi177_31116_object32022:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e59)", 2024-09-13T11:23:37.230 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:10.509005+0000", 2024-09-13T11:23:37.230 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 26.695800139999999, 2024-09-13T11:23:37.230 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.066713914999999999, 2024-09-13T11:23:37.231 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:23:37.231 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:23:37.231 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:37.231 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:23:37.231 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:23:37.231 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:37.231 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 32023 2024-09-13T11:23:37.231 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.231 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:23:37.231 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.231 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:23:37.232 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:10.509005+0000", 2024-09-13T11:23:37.232 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:37.232 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.232 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.232 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:23:37.232 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:10.509005+0000", 2024-09-13T11:23:37.232 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:37.232 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.232 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.232 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:23:37.232 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:10.509014+0000", 2024-09-13T11:23:37.232 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.115e-06 2024-09-13T11:23:37.232 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.233 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.233 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:23:37.233 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:10.509052+0000", 2024-09-13T11:23:37.233 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.8312000000000002e-05 2024-09-13T11:23:37.233 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.233 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.233 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:23:37.233 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:10.509066+0000", 2024-09-13T11:23:37.233 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.36e-05 2024-09-13T11:23:37.233 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.233 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.233 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:23:37.234 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:10.509073+0000", 2024-09-13T11:23:37.234 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.3300000000000001e-06 2024-09-13T11:23:37.234 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.234 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.234 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:23:37.234 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:10.509138+0000", 2024-09-13T11:23:37.234 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.4659999999999994e-05 2024-09-13T11:23:37.234 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.234 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.234 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:23:37.234 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:10.509318+0000", 2024-09-13T11:23:37.235 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00018018799999999999 2024-09-13T11:23:37.235 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.235 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.235 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 7", 2024-09-13T11:23:37.235 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:10.509511+0000", 2024-09-13T11:23:37.235 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00019297300000000001 2024-09-13T11:23:37.235 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.235 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.235 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:23:37.235 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:10.511111+0000", 2024-09-13T11:23:37.235 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001599804 2024-09-13T11:23:37.235 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.236 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.236 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:37.236 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:10.575654+0000", 2024-09-13T11:23:37.236 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.064543152000000006 2024-09-13T11:23:37.236 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.236 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.236 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:23:37.236 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:10.575690+0000", 2024-09-13T11:23:37.236 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.5661999999999998e-05 2024-09-13T11:23:37.236 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.236 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.236 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:23:37.236 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:10.575719+0000", 2024-09-13T11:23:37.237 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.9119e-05 2024-09-13T11:23:37.237 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:37.237 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:23:37.237 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:37.237 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.237 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.237 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop_reply(client.4665.0:32023 3.3 e59/30 ondisk, result = 0)", 2024-09-13T11:23:37.237 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:10.511491+0000", 2024-09-13T11:23:37.237 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 26.693313805999999, 2024-09-13T11:23:37.237 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.064234826999999994, 2024-09-13T11:23:37.237 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:23:37.237 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:23:37.238 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:23:37.238 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:23:37.238 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.238 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:23:37.238 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:10.511491+0000", 2024-09-13T11:23:37.238 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:37.238 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.238 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.238 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:23:37.238 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:10.511491+0000", 2024-09-13T11:23:37.238 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:37.239 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.239 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.239 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:23:37.239 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:10.511493+0000", 2024-09-13T11:23:37.239 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.2280000000000001e-06 2024-09-13T11:23:37.239 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.239 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.239 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:23:37.239 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:10.511502+0000", 2024-09-13T11:23:37.239 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.4870000000000002e-06 2024-09-13T11:23:37.239 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.240 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.240 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:23:37.240 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:10.575605+0000", 2024-09-13T11:23:37.240 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.064102938999999998 2024-09-13T11:23:37.240 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.240 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.240 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:23:37.240 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:10.575610+0000", 2024-09-13T11:23:37.240 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.665e-06 2024-09-13T11:23:37.240 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.241 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.241 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:23:37.241 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:10.575646+0000", 2024-09-13T11:23:37.241 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.5815999999999997e-05 2024-09-13T11:23:37.241 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.241 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.241 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:23:37.241 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:10.575651+0000", 2024-09-13T11:23:37.241 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.4800000000000003e-06 2024-09-13T11:23:37.241 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.241 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.241 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:23:37.241 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:10.575726+0000", 2024-09-13T11:23:37.242 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.5211999999999997e-05 2024-09-13T11:23:37.242 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:37.242 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:23:37.242 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:37.242 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.242 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.242 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:39935 3.3 3:c6ae38b7:::benchmark_data_smithi177_31116_object39934:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-13T11:23:37.242 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:15.785349+0000", 2024-09-13T11:23:37.242 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 21.419455652, 2024-09-13T11:23:37.242 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.059974834999999997, 2024-09-13T11:23:37.242 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:23:37.242 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:23:37.242 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:37.243 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:23:37.243 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:23:37.243 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:37.243 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 39935 2024-09-13T11:23:37.243 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.243 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:23:37.243 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.243 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:23:37.243 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:15.785349+0000", 2024-09-13T11:23:37.243 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:37.243 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.243 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.243 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:23:37.243 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:15.785349+0000", 2024-09-13T11:23:37.244 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:37.244 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.244 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.244 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:23:37.244 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:15.785353+0000", 2024-09-13T11:23:37.244 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.9990000000000002e-06 2024-09-13T11:23:37.244 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.244 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.244 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:23:37.244 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:15.785420+0000", 2024-09-13T11:23:37.244 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.6284000000000004e-05 2024-09-13T11:23:37.244 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.245 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.245 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:23:37.245 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:15.785447+0000", 2024-09-13T11:23:37.245 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.7158e-05 2024-09-13T11:23:37.245 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.245 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.245 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:23:37.245 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:15.785461+0000", 2024-09-13T11:23:37.245 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4283e-05 2024-09-13T11:23:37.245 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.245 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.245 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:23:37.245 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:15.785549+0000", 2024-09-13T11:23:37.246 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.7510999999999996e-05 2024-09-13T11:23:37.246 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.246 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.246 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:23:37.246 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:15.785734+0000", 2024-09-13T11:23:37.246 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000185067 2024-09-13T11:23:37.246 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.246 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.246 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 7", 2024-09-13T11:23:37.246 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:15.786019+0000", 2024-09-13T11:23:37.246 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00028562299999999999 2024-09-13T11:23:37.246 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.246 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.247 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:23:37.247 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:15.788766+0000", 2024-09-13T11:23:37.247 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0027468879999999998 2024-09-13T11:23:37.247 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.247 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.247 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:37.247 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:15.845115+0000", 2024-09-13T11:23:37.247 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.056348913 2024-09-13T11:23:37.247 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.247 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.247 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:23:37.247 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:15.845245+0000", 2024-09-13T11:23:37.248 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00012998900000000001 2024-09-13T11:23:37.248 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.248 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.248 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:23:37.248 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:15.845324+0000", 2024-09-13T11:23:37.248 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.9120000000000001e-05 2024-09-13T11:23:37.248 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:37.248 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:23:37.248 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:37.248 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.248 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.248 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:45810 3.17 3.9bd13e57 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e69)", 2024-09-13T11:23:37.248 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:21.347960+0000", 2024-09-13T11:23:37.249 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 15.856844638, 2024-09-13T11:23:37.249 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.56814464200000003, 2024-09-13T11:23:37.249 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:23:37.249 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:23:37.249 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-09-13T11:23:37.249 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:23:37.249 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:23:37.249 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:37.249 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 45810 2024-09-13T11:23:37.249 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.249 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:23:37.249 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.250 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:23:37.250 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:21.347960+0000", 2024-09-13T11:23:37.250 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:37.250 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.250 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.250 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:23:37.250 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:21.347960+0000", 2024-09-13T11:23:37.250 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:37.250 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.250 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.250 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:23:37.250 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:21.347965+0000", 2024-09-13T11:23:37.250 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.6720000000000003e-06 2024-09-13T11:23:37.250 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.251 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.251 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:23:37.251 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:21.348019+0000", 2024-09-13T11:23:37.251 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.4391000000000002e-05 2024-09-13T11:23:37.251 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.251 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.251 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:23:37.251 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:21.348052+0000", 2024-09-13T11:23:37.251 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.2246000000000003e-05 2024-09-13T11:23:37.251 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.251 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.251 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:23:37.251 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:21.348072+0000", 2024-09-13T11:23:37.252 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.0622e-05 2024-09-13T11:23:37.252 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.252 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.252 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:23:37.252 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:21.348278+0000", 2024-09-13T11:23:37.252 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000205391 2024-09-13T11:23:37.252 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.252 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.252 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-09-13T11:23:37.252 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:21.348292+0000", 2024-09-13T11:23:37.252 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4759e-05 2024-09-13T11:23:37.252 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.252 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.253 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:23:37.253 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:21.916064+0000", 2024-09-13T11:23:37.253 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.56777128600000004 2024-09-13T11:23:37.253 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.253 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.253 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:23:37.253 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:21.916105+0000", 2024-09-13T11:23:37.253 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.1275e-05 2024-09-13T11:23:37.253 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:37.253 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:23:37.253 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:37.253 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.254 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.254 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPush(3.17 71/70 [PushOp(3:e9675271:::benchmark_data_smithi177_31116_object2875:head, version: 44'189, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:e9675271:::benchmark_data_smithi177_31116_object2875:head@44'189, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:23:37.254 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:25.071461+0000", 2024-09-13T11:23:37.254 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 12.13334349, 2024-09-13T11:23:37.254 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.059432921999999999, 2024-09-13T11:23:37.254 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:23:37.254 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:23:37.254 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:23:37.254 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:23:37.254 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.254 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:23:37.254 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.071461+0000", 2024-09-13T11:23:37.255 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:37.255 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.255 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.255 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:23:37.255 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.071461+0000", 2024-09-13T11:23:37.255 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:37.255 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.255 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.255 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:23:37.255 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.071465+0000", 2024-09-13T11:23:37.255 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.2219999999999998e-06 2024-09-13T11:23:37.255 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.255 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.255 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:23:37.256 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.071549+0000", 2024-09-13T11:23:37.256 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.4406000000000005e-05 2024-09-13T11:23:37.256 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.256 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.256 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:23:37.256 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.129768+0000", 2024-09-13T11:23:37.256 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.058218457000000001 2024-09-13T11:23:37.256 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.256 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.256 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:23:37.256 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.129790+0000", 2024-09-13T11:23:37.256 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.2333000000000001e-05 2024-09-13T11:23:37.256 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.257 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.257 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:23:37.257 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.129923+0000", 2024-09-13T11:23:37.257 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000132686 2024-09-13T11:23:37.257 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.257 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.257 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:23:37.257 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.129941+0000", 2024-09-13T11:23:37.257 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.8148e-05 2024-09-13T11:23:37.257 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.257 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.257 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:23:37.257 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.130894+0000", 2024-09-13T11:23:37.257 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00095367 2024-09-13T11:23:37.258 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:37.258 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:23:37.258 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:37.258 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.258 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.258 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPush(3.17 71/70 [PushOp(3:ebc04bd6:::benchmark_data_smithi177_31116_object2889:head, version: 44'191, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:ebc04bd6:::benchmark_data_smithi177_31116_object2889:head@44'191, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:23:37.259 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:25.180557+0000", 2024-09-13T11:23:37.259 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 12.024247967999999, 2024-09-13T11:23:37.259 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.058845474000000002, 2024-09-13T11:23:37.260 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:23:37.260 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:23:37.260 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:23:37.260 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:23:37.260 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.260 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:23:37.260 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.180557+0000", 2024-09-13T11:23:37.260 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:37.260 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.260 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.260 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:23:37.260 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.180557+0000", 2024-09-13T11:23:37.260 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:37.261 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.261 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.261 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:23:37.261 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.180560+0000", 2024-09-13T11:23:37.261 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.4699999999999998e-06 2024-09-13T11:23:37.261 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.261 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.261 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:23:37.261 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.180637+0000", 2024-09-13T11:23:37.261 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.6864000000000001e-05 2024-09-13T11:23:37.261 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.261 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.261 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:23:37.262 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.238340+0000", 2024-09-13T11:23:37.262 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.057702519000000001 2024-09-13T11:23:37.262 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.262 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.262 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:23:37.262 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.238356+0000", 2024-09-13T11:23:37.262 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.6504999999999999e-05 2024-09-13T11:23:37.262 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.262 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.262 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:23:37.262 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.238468+0000", 2024-09-13T11:23:37.262 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000111341 2024-09-13T11:23:37.262 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.262 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.263 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:23:37.263 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.238480+0000", 2024-09-13T11:23:37.263 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2527999999999999e-05 2024-09-13T11:23:37.263 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.263 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.263 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:23:37.263 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.239402+0000", 2024-09-13T11:23:37.263 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00092224699999999995 2024-09-13T11:23:37.263 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:37.263 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:23:37.263 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:37.263 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.264 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.264 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPush(3.17 71/70 [PushOp(3:ea85fdfb:::benchmark_data_smithi177_31116_object3073:head, version: 44'207, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:ea85fdfb:::benchmark_data_smithi177_31116_object3073:head@44'207, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:23:37.264 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:25.444180+0000", 2024-09-13T11:23:37.264 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 11.760624812, 2024-09-13T11:23:37.264 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.058502555999999997, 2024-09-13T11:23:37.264 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:23:37.264 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:23:37.264 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:23:37.264 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:23:37.264 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.264 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:23:37.264 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.444180+0000", 2024-09-13T11:23:37.264 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:37.265 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.265 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.265 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:23:37.265 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.444180+0000", 2024-09-13T11:23:37.265 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:37.265 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.265 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.265 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:23:37.265 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.444182+0000", 2024-09-13T11:23:37.265 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.716e-06 2024-09-13T11:23:37.265 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.265 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.265 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:23:37.266 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.444248+0000", 2024-09-13T11:23:37.266 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.6291000000000006e-05 2024-09-13T11:23:37.266 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.266 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.266 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:23:37.266 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.501765+0000", 2024-09-13T11:23:37.266 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.057516838000000001 2024-09-13T11:23:37.266 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.266 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.266 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:23:37.266 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.501782+0000", 2024-09-13T11:23:37.266 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.6560999999999999e-05 2024-09-13T11:23:37.266 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.266 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.267 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:23:37.267 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.501903+0000", 2024-09-13T11:23:37.267 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000120993 2024-09-13T11:23:37.267 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.267 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.267 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:23:37.267 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.501920+0000", 2024-09-13T11:23:37.267 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.7640000000000001e-05 2024-09-13T11:23:37.267 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.267 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.267 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:23:37.267 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.502683+0000", 2024-09-13T11:23:37.267 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00076251699999999999 2024-09-13T11:23:37.268 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:37.268 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:23:37.268 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:37.268 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.268 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.268 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPush(3.17 71/70 [PushOp(3:ec82217b:::benchmark_data_smithi177_31116_object3121:head, version: 44'215, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:ec82217b:::benchmark_data_smithi177_31116_object3121:head@44'215, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:23:37.268 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:25.596519+0000", 2024-09-13T11:23:37.268 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 11.608285843000001, 2024-09-13T11:23:37.268 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.081111691, 2024-09-13T11:23:37.268 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:23:37.268 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:23:37.268 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:23:37.269 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:23:37.269 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.269 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:23:37.269 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.596519+0000", 2024-09-13T11:23:37.269 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:37.269 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.269 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.269 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:23:37.269 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.596519+0000", 2024-09-13T11:23:37.269 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:37.269 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.269 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.269 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:23:37.270 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.596526+0000", 2024-09-13T11:23:37.270 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.2169999999999997e-06 2024-09-13T11:23:37.270 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.270 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.270 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:23:37.270 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.596592+0000", 2024-09-13T11:23:37.270 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.5203999999999994e-05 2024-09-13T11:23:37.270 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.270 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.270 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:23:37.270 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.675751+0000", 2024-09-13T11:23:37.270 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.079159859999999999 2024-09-13T11:23:37.270 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.270 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.271 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:23:37.271 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.675781+0000", 2024-09-13T11:23:37.271 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.9662000000000002e-05 2024-09-13T11:23:37.271 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.271 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.271 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:23:37.271 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.676016+0000", 2024-09-13T11:23:37.271 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000235119 2024-09-13T11:23:37.271 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.271 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.271 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:23:37.272 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.676042+0000", 2024-09-13T11:23:37.272 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.5510000000000001e-05 2024-09-13T11:23:37.272 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.272 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.272 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:23:37.272 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.677631+0000", 2024-09-13T11:23:37.272 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0015891189999999999 2024-09-13T11:23:37.272 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:37.272 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:23:37.272 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:37.272 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.272 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.273 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPush(3.17 71/70 [PushOp(3:ecdb4da9:::benchmark_data_smithi177_31116_object3198:head, version: 44'219, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:ecdb4da9:::benchmark_data_smithi177_31116_object3198:head@44'219, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:23:37.273 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:25.829215+0000", 2024-09-13T11:23:37.273 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 11.375590374, 2024-09-13T11:23:37.273 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.054892722999999997, 2024-09-13T11:23:37.273 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:23:37.273 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:23:37.273 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:23:37.273 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:23:37.273 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.273 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:23:37.273 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.829215+0000", 2024-09-13T11:23:37.273 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:37.274 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.274 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.274 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:23:37.274 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.829215+0000", 2024-09-13T11:23:37.274 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:37.274 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.274 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.274 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:23:37.274 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.829217+0000", 2024-09-13T11:23:37.274 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.057e-06 2024-09-13T11:23:37.274 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.274 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.275 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:23:37.275 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.829286+0000", 2024-09-13T11:23:37.275 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.9159999999999995e-05 2024-09-13T11:23:37.275 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.275 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.275 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:23:37.275 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.883029+0000", 2024-09-13T11:23:37.275 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.053742960999999999 2024-09-13T11:23:37.275 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.275 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.275 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:23:37.275 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.883054+0000", 2024-09-13T11:23:37.275 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.5724e-05 2024-09-13T11:23:37.276 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.276 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.276 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:23:37.276 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.883177+0000", 2024-09-13T11:23:37.276 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00012227000000000001 2024-09-13T11:23:37.276 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.276 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.276 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:23:37.276 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.883197+0000", 2024-09-13T11:23:37.276 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.0285e-05 2024-09-13T11:23:37.276 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.276 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.276 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:23:37.277 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.884107+0000", 2024-09-13T11:23:37.277 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00091026599999999996 2024-09-13T11:23:37.277 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:37.277 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:23:37.277 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:37.277 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.277 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.277 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPush(3.17 72/70 [PushOp(3:ecb2eb17:::benchmark_data_smithi177_31116_object4630:head, version: 44'317, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:ecb2eb17:::benchmark_data_smithi177_31116_object4630:head@44'317, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:23:37.277 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:26.434926+0000", 2024-09-13T11:23:37.277 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 10.769879135, 2024-09-13T11:23:37.277 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.155542491, 2024-09-13T11:23:37.277 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:23:37.277 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:23:37.278 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:23:37.278 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:23:37.278 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.278 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:23:37.278 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:26.434926+0000", 2024-09-13T11:23:37.278 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:37.278 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.278 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.278 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:23:37.278 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:26.434926+0000", 2024-09-13T11:23:37.278 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:37.278 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.278 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.279 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:23:37.279 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:26.434928+0000", 2024-09-13T11:23:37.279 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.7489999999999999e-06 2024-09-13T11:23:37.279 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.279 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.279 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:23:37.279 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:26.434983+0000", 2024-09-13T11:23:37.279 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.4988000000000003e-05 2024-09-13T11:23:37.279 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.279 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.279 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:23:37.279 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:26.589751+0000", 2024-09-13T11:23:37.279 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.15476837299999999 2024-09-13T11:23:37.279 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.280 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.280 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:23:37.280 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:26.589778+0000", 2024-09-13T11:23:37.280 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.7385e-05 2024-09-13T11:23:37.280 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.280 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.280 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:23:37.280 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:26.589891+0000", 2024-09-13T11:23:37.280 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000112849 2024-09-13T11:23:37.280 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.280 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.280 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:23:37.280 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:26.589900+0000", 2024-09-13T11:23:37.281 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.3440000000000007e-06 2024-09-13T11:23:37.281 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.281 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.281 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:23:37.281 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:26.590468+0000", 2024-09-13T11:23:37.281 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00056780299999999997 2024-09-13T11:23:37.281 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:37.281 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:23:37.281 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:37.281 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.282 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.282 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPush(3.17 72/70 [PushOp(3:ee101882:::benchmark_data_smithi177_31116_object5448:head, version: 45'353, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:ee101882:::benchmark_data_smithi177_31116_object5448:head@45'353, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:23:37.282 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:27.057731+0000", 2024-09-13T11:23:37.282 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 10.147073541999999, 2024-09-13T11:23:37.282 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.089951626000000007, 2024-09-13T11:23:37.282 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:23:37.282 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:23:37.282 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:23:37.282 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:23:37.282 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.282 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:23:37.282 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:27.057731+0000", 2024-09-13T11:23:37.282 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:37.283 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.283 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.283 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:23:37.283 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:27.057731+0000", 2024-09-13T11:23:37.283 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:37.283 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.283 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.283 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:23:37.283 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:27.057733+0000", 2024-09-13T11:23:37.283 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.978e-06 2024-09-13T11:23:37.283 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.283 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.283 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:23:37.283 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:27.057819+0000", 2024-09-13T11:23:37.284 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.5457999999999994e-05 2024-09-13T11:23:37.284 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.284 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.284 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:23:37.284 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:27.146807+0000", 2024-09-13T11:23:37.284 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.088988445999999999 2024-09-13T11:23:37.284 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.284 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.284 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:23:37.284 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:27.146820+0000", 2024-09-13T11:23:37.285 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3081000000000001e-05 2024-09-13T11:23:37.285 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.285 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.285 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:23:37.285 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:27.146900+0000", 2024-09-13T11:23:37.285 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.9251000000000004e-05 2024-09-13T11:23:37.285 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.285 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.285 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:23:37.285 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:27.146908+0000", 2024-09-13T11:23:37.286 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.0739999999999993e-06 2024-09-13T11:23:37.286 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.286 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.286 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:23:37.286 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:27.147683+0000", 2024-09-13T11:23:37.286 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00077533799999999998 2024-09-13T11:23:37.286 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:37.286 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:23:37.286 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:37.286 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.286 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.286 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.4665.0:50861 3.17 e72/70 3:ee280ac8:::benchmark_data_smithi177_31116_object50860:head v 72'2909, mlcod=45'348)", 2024-09-13T11:23:37.286 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:27.058825+0000", 2024-09-13T11:23:37.287 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 10.145979939, 2024-09-13T11:23:37.287 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.098196383999999998, 2024-09-13T11:23:37.287 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:23:37.287 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:23:37.287 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:37.287 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:23:37.287 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.287 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:23:37.287 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:27.058825+0000", 2024-09-13T11:23:37.287 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:37.287 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.287 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.287 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:23:37.287 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:27.058825+0000", 2024-09-13T11:23:37.288 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:37.288 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.288 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.288 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:23:37.288 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:27.058826+0000", 2024-09-13T11:23:37.288 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.079e-06 2024-09-13T11:23:37.288 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.288 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.288 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:23:37.288 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:27.058906+0000", 2024-09-13T11:23:37.288 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.9863000000000004e-05 2024-09-13T11:23:37.288 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.288 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.289 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:23:37.289 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:27.154716+0000", 2024-09-13T11:23:37.289 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.095809748 2024-09-13T11:23:37.289 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.289 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.289 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:23:37.289 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:27.154721+0000", 2024-09-13T11:23:37.289 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.5219999999999998e-06 2024-09-13T11:23:37.289 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.289 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.289 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:23:37.289 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:27.154781+0000", 2024-09-13T11:23:37.289 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.9732999999999999e-05 2024-09-13T11:23:37.290 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.290 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.290 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:23:37.290 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:27.154807+0000", 2024-09-13T11:23:37.290 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.6509e-05 2024-09-13T11:23:37.290 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.290 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.290 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:23:37.290 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:27.156976+0000", 2024-09-13T11:23:37.290 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0021687379999999999 2024-09-13T11:23:37.290 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.290 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.291 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:23:37.291 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:27.157021+0000", 2024-09-13T11:23:37.291 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.5191999999999997e-05 2024-09-13T11:23:37.291 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:37.291 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:23:37.291 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:37.291 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.291 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.291 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:54653 3.23 3:c7a54800:::benchmark_data_smithi177_31116_object54652:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e75)", 2024-09-13T11:23:37.291 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:30.979059+0000", 2024-09-13T11:23:37.291 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 6.225746139, 2024-09-13T11:23:37.291 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.97284747100000002, 2024-09-13T11:23:37.291 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:23:37.292 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:23:37.292 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-09-13T11:23:37.292 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:23:37.292 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:23:37.292 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:37.292 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 54653 2024-09-13T11:23:37.292 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.292 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:23:37.292 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.292 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:23:37.292 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.979059+0000", 2024-09-13T11:23:37.292 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:37.293 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.293 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.293 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:23:37.293 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.979059+0000", 2024-09-13T11:23:37.293 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:37.293 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.293 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.293 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:23:37.293 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.979061+0000", 2024-09-13T11:23:37.293 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.4279999999999999e-06 2024-09-13T11:23:37.293 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.293 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.294 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:23:37.294 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.979114+0000", 2024-09-13T11:23:37.294 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.3254000000000002e-05 2024-09-13T11:23:37.294 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.294 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.294 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:23:37.294 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.979157+0000", 2024-09-13T11:23:37.294 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.2018000000000003e-05 2024-09-13T11:23:37.294 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.294 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.294 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:23:37.294 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.979172+0000", 2024-09-13T11:23:37.295 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5636000000000001e-05 2024-09-13T11:23:37.295 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.295 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.295 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:23:37.295 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:31.045109+0000", 2024-09-13T11:23:37.295 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.065936982000000005 2024-09-13T11:23:37.295 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.295 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.295 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-09-13T11:23:37.295 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:31.045121+0000", 2024-09-13T11:23:37.295 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1608e-05 2024-09-13T11:23:37.296 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.296 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.296 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:23:37.296 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:31.951636+0000", 2024-09-13T11:23:37.296 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.90651503899999997 2024-09-13T11:23:37.296 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.296 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.296 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:23:37.296 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:31.951906+0000", 2024-09-13T11:23:37.296 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000270506 2024-09-13T11:23:37.296 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:37.296 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:23:37.297 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:37.297 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.297 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.297 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:54659 3.3 3.bc2763c3 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e75)", 2024-09-13T11:23:37.297 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:30.989047+0000", 2024-09-13T11:23:37.297 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 6.2157577689999997, 2024-09-13T11:23:37.297 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.055860816000000001, 2024-09-13T11:23:37.297 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:23:37.297 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:23:37.297 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-09-13T11:23:37.297 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:23:37.298 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:23:37.298 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:37.298 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 54659 2024-09-13T11:23:37.298 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.298 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:23:37.298 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.298 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:23:37.298 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.989047+0000", 2024-09-13T11:23:37.298 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:37.298 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.298 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.298 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:23:37.298 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.989047+0000", 2024-09-13T11:23:37.299 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:37.299 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.299 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.299 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:23:37.299 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.989050+0000", 2024-09-13T11:23:37.299 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.8890000000000002e-06 2024-09-13T11:23:37.299 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.299 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.299 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:23:37.299 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.989115+0000", 2024-09-13T11:23:37.299 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.5137999999999999e-05 2024-09-13T11:23:37.300 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.300 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.300 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:23:37.300 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.989147+0000", 2024-09-13T11:23:37.300 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.1853000000000001e-05 2024-09-13T11:23:37.300 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.300 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.300 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:23:37.300 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.989163+0000", 2024-09-13T11:23:37.300 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5628999999999999e-05 2024-09-13T11:23:37.300 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.300 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.301 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:23:37.301 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.989252+0000", 2024-09-13T11:23:37.301 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.8824000000000002e-05 2024-09-13T11:23:37.301 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.301 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.301 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op must wait for map", 2024-09-13T11:23:37.301 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.989259+0000", 2024-09-13T11:23:37.301 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.4420000000000001e-06 2024-09-13T11:23:37.301 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.301 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.301 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:23:37.301 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:31.044871+0000", 2024-09-13T11:23:37.301 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.055612502000000001 2024-09-13T11:23:37.302 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.302 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.302 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:23:37.302 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:31.044908+0000", 2024-09-13T11:23:37.302 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.6538999999999997e-05 2024-09-13T11:23:37.302 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:37.302 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:23:37.302 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:37.302 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.302 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.302 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPush(3.18 76/70 [PushOp(3:1a32c9cd:::benchmark_data_smithi177_31116_object12833:head, version: 49'818, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:1a32c9cd:::benchmark_data_smithi177_31116_object12833:head@49'818, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:23:37.302 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:32.522305+0000", 2024-09-13T11:23:37.303 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 4.6825000550000002, 2024-09-13T11:23:37.303 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.40201207100000003, 2024-09-13T11:23:37.303 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:23:37.303 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:23:37.303 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:23:37.303 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:23:37.303 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.303 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:23:37.303 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.522305+0000", 2024-09-13T11:23:37.303 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:37.303 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.303 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.304 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:23:37.304 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.522305+0000", 2024-09-13T11:23:37.304 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:37.304 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.304 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.304 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:23:37.304 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.522309+0000", 2024-09-13T11:23:37.304 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.9829999999999998e-06 2024-09-13T11:23:37.304 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.304 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.304 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:23:37.304 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.522377+0000", 2024-09-13T11:23:37.305 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.8069000000000002e-05 2024-09-13T11:23:37.305 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.305 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.305 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:23:37.305 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.921950+0000", 2024-09-13T11:23:37.305 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.399573438 2024-09-13T11:23:37.305 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.305 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.305 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:23:37.305 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.921983+0000", 2024-09-13T11:23:37.305 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.2858000000000003e-05 2024-09-13T11:23:37.305 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.305 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.305 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:23:37.306 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.922389+0000", 2024-09-13T11:23:37.306 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00040612999999999999 2024-09-13T11:23:37.306 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.306 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.306 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:23:37.306 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.922421+0000", 2024-09-13T11:23:37.306 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.1356999999999998e-05 2024-09-13T11:23:37.306 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.306 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.306 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:23:37.306 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.924317+0000", 2024-09-13T11:23:37.306 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0018962359999999999 2024-09-13T11:23:37.306 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:37.307 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:23:37.307 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:37.307 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.307 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.307 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPush(3.17 76/70 [PushOp(3:ebcbce67:::benchmark_data_smithi177_31116_object18268:head, version: 53'1102, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:ebcbce67:::benchmark_data_smithi177_31116_object18268:head@53'1102, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:23:37.307 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:32.638908+0000", 2024-09-13T11:23:37.307 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 4.565896607, 2024-09-13T11:23:37.307 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.229064031, 2024-09-13T11:23:37.307 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:23:37.307 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:23:37.307 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:23:37.307 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:23:37.307 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.308 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:23:37.308 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.638908+0000", 2024-09-13T11:23:37.308 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:37.308 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.308 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.308 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:23:37.308 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.638908+0000", 2024-09-13T11:23:37.308 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:37.308 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.308 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.308 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:23:37.308 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.638910+0000", 2024-09-13T11:23:37.309 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.7379999999999999e-06 2024-09-13T11:23:37.309 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.309 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.309 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:23:37.309 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.638975+0000", 2024-09-13T11:23:37.309 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.4937000000000005e-05 2024-09-13T11:23:37.309 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.309 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.309 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:23:37.309 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.865860+0000", 2024-09-13T11:23:37.309 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.22688509800000001 2024-09-13T11:23:37.309 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.310 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.310 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:23:37.310 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.865890+0000", 2024-09-13T11:23:37.310 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.0273e-05 2024-09-13T11:23:37.310 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.310 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.310 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:23:37.310 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.866143+0000", 2024-09-13T11:23:37.310 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00025212800000000001 2024-09-13T11:23:37.310 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.310 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.310 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:23:37.310 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.866181+0000", 2024-09-13T11:23:37.311 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.8945000000000001e-05 2024-09-13T11:23:37.311 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.311 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.311 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:23:37.311 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.867972+0000", 2024-09-13T11:23:37.311 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001790912 2024-09-13T11:23:37.311 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:37.311 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:23:37.311 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:37.311 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:37.311 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:23:37.311 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-13T11:23:37.313 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_ops_in_flight 2024-09-13T11:23:37.314 INFO:tasks.ceph.osd.10.smithi162.stderr:2024-09-13T11:23:37.305+0000 7fb59c9f0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26317) UID: 0 2024-09-13T11:23:37.413 INFO:tasks.ceph.osd.12.smithi001.stderr:2024-09-13T11:23:37.408+0000 7f892c6e6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26314) UID: 0 2024-09-13T11:23:37.501 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-13T11:23:37.501 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [ 2024-09-13T11:23:37.501 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.501 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.4665.0:57497 3.14 e80/70 3:2fea6cf9:::benchmark_data_smithi177_31116_object57496:head v 80'3131, mlcod=62'2206)", 2024-09-13T11:23:37.501 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:37.498857+0000", 2024-09-13T11:23:37.501 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 0.00080379099999999999, 2024-09-13T11:23:37.501 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00082144900000000003, 2024-09-13T11:23:37.501 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:23:37.502 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:23:37.502 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:23:37.502 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:23:37.502 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.502 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:23:37.502 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:37.498857+0000", 2024-09-13T11:23:37.502 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:37.502 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.502 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.502 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:23:37.502 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:37.498857+0000", 2024-09-13T11:23:37.502 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:37.502 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.502 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.503 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:23:37.503 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:37.498858+0000", 2024-09-13T11:23:37.503 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.78e-07 2024-09-13T11:23:37.503 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.503 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.503 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:23:37.503 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:37.498916+0000", 2024-09-13T11:23:37.503 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.8458000000000002e-05 2024-09-13T11:23:37.503 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.503 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.503 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:23:37.503 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:37.498936+0000", 2024-09-13T11:23:37.503 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.0013e-05 2024-09-13T11:23:37.503 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.504 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.504 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:23:37.504 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:37.498941+0000", 2024-09-13T11:23:37.504 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.0119999999999996e-06 2024-09-13T11:23:37.504 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.504 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.504 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:23:37.504 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:37.499373+0000", 2024-09-13T11:23:37.504 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00043197599999999999 2024-09-13T11:23:37.504 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.504 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.504 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:23:37.504 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:37.499402+0000", 2024-09-13T11:23:37.505 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.8554000000000001e-05 2024-09-13T11:23:37.505 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:37.505 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:23:37.505 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:37.505 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:37.505 INFO:teuthology.orchestra.run.smithi162.stdout: ], 2024-09-13T11:23:37.505 INFO:teuthology.orchestra.run.smithi162.stdout: "num_ops": 1 2024-09-13T11:23:37.505 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-13T11:23:37.515 INFO:tasks.ceph.osd.12.smithi001.stderr:2024-09-13T11:23:37.508+0000 7f892c6e6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26314) UID: 0 2024-09-13T11:23:37.521 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_blocked_ops 2024-09-13T11:23:37.615 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-13T11:23:37.608+0000 7f6a68775640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26307) UID: 0 2024-09-13T11:23:37.665 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-13T11:23:37.665 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [], 2024-09-13T11:23:37.665 INFO:teuthology.orchestra.run.smithi162.stdout: "complaint_time": 30, 2024-09-13T11:23:37.665 INFO:teuthology.orchestra.run.smithi162.stdout: "num_blocked_ops": 0 2024-09-13T11:23:37.665 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-13T11:23:37.685 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_historic_ops 2024-09-13T11:23:37.716 INFO:tasks.ceph.osd.15.smithi177.stderr:2024-09-13T11:23:37.712+0000 7f9d6ce77640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26578) UID: 0 2024-09-13T11:23:37.818 INFO:tasks.ceph.osd.4.smithi001.stderr:2024-09-13T11:23:37.812+0000 7ff8e0218640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26310) UID: 0 2024-09-13T11:23:37.823 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-13T11:23:37.823 INFO:teuthology.orchestra.run.smithi162.stdout: "size": 20, 2024-09-13T11:23:37.823 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 600, 2024-09-13T11:23:37.823 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [ 2024-09-13T11:23:37.823 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.823 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.4665.0:9905 3.6 e47/30 3:639d4f69:::benchmark_data_smithi177_31116_object9904:head v 47'621, mlcod=47'618)", 2024-09-13T11:23:37.824 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:22:54.301066+0000", 2024-09-13T11:23:37.824 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 43.519607596, 2024-09-13T11:23:37.824 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.11187942300000001, 2024-09-13T11:23:37.824 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:23:37.824 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:23:37.824 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:37.824 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:23:37.824 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.824 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:23:37.825 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:54.301066+0000", 2024-09-13T11:23:37.825 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:37.825 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.825 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.825 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:23:37.826 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:54.301066+0000", 2024-09-13T11:23:37.826 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:37.826 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.826 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.826 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:23:37.827 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:54.301077+0000", 2024-09-13T11:23:37.827 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1154e-05 2024-09-13T11:23:37.827 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.827 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.827 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:23:37.827 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:54.301136+0000", 2024-09-13T11:23:37.827 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.8480999999999998e-05 2024-09-13T11:23:37.827 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.827 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.828 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:23:37.828 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:54.410822+0000", 2024-09-13T11:23:37.828 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.109685649 2024-09-13T11:23:37.828 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.828 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.828 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:23:37.828 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:54.410829+0000", 2024-09-13T11:23:37.828 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.7470000000000005e-06 2024-09-13T11:23:37.828 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.828 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.828 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:23:37.829 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:54.410995+0000", 2024-09-13T11:23:37.829 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00016614200000000001 2024-09-13T11:23:37.829 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.829 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.829 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:23:37.829 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:54.411056+0000", 2024-09-13T11:23:37.829 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.0174999999999999e-05 2024-09-13T11:23:37.829 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.829 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.829 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:23:37.829 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:54.412869+0000", 2024-09-13T11:23:37.829 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001813423 2024-09-13T11:23:37.830 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.830 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.830 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:23:37.830 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:54.412946+0000", 2024-09-13T11:23:37.830 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.6651999999999997e-05 2024-09-13T11:23:37.830 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:37.830 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:23:37.830 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:37.830 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.830 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.830 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:10728 3.2 3:43fb4edc:::benchmark_data_smithi177_31116_object10727:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e48)", 2024-09-13T11:23:37.830 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:22:54.836511+0000", 2024-09-13T11:23:37.830 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 42.984162376999997, 2024-09-13T11:23:37.830 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.106223183, 2024-09-13T11:23:37.831 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:23:37.831 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:23:37.831 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:37.831 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:23:37.831 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:23:37.831 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:37.831 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 10728 2024-09-13T11:23:37.831 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.831 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:23:37.831 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.831 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:23:37.831 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:54.836511+0000", 2024-09-13T11:23:37.832 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:37.832 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.832 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.832 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:23:37.832 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:54.836511+0000", 2024-09-13T11:23:37.832 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:37.832 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.832 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.832 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:23:37.832 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:54.836514+0000", 2024-09-13T11:23:37.832 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.948e-06 2024-09-13T11:23:37.832 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.833 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.833 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:23:37.833 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:54.836564+0000", 2024-09-13T11:23:37.833 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.9129999999999999e-05 2024-09-13T11:23:37.833 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.833 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.833 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:23:37.833 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:54.836584+0000", 2024-09-13T11:23:37.833 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.0684000000000001e-05 2024-09-13T11:23:37.833 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.833 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.833 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:23:37.833 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:54.836599+0000", 2024-09-13T11:23:37.834 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4409e-05 2024-09-13T11:23:37.834 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.834 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.834 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:23:37.834 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:54.836738+0000", 2024-09-13T11:23:37.834 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00013955799999999999 2024-09-13T11:23:37.834 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.834 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.834 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:23:37.834 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:54.836985+0000", 2024-09-13T11:23:37.834 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000247159 2024-09-13T11:23:37.834 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.834 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.834 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 12", 2024-09-13T11:23:37.835 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:54.837238+0000", 2024-09-13T11:23:37.835 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00025263299999999999 2024-09-13T11:23:37.835 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.835 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.835 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:23:37.835 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:54.839339+0000", 2024-09-13T11:23:37.835 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0021009380000000001 2024-09-13T11:23:37.835 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.835 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.835 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:37.835 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:54.942575+0000", 2024-09-13T11:23:37.835 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.103236046 2024-09-13T11:23:37.836 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.836 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.836 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:23:37.836 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:54.942665+0000", 2024-09-13T11:23:37.836 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.9747000000000007e-05 2024-09-13T11:23:37.836 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.836 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.836 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:23:37.836 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:54.942735+0000", 2024-09-13T11:23:37.836 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.9931000000000005e-05 2024-09-13T11:23:37.836 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:37.836 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:23:37.836 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:37.837 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.837 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.837 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:48911 3.12 3:4a364a6a:::benchmark_data_smithi177_31116_object48910:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e71)", 2024-09-13T11:23:37.837 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:25.457776+0000", 2024-09-13T11:23:37.837 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 12.362897521000001, 2024-09-13T11:23:37.837 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.22561999599999999, 2024-09-13T11:23:37.837 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:23:37.837 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:23:37.837 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:37.837 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:23:37.837 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:23:37.837 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:37.838 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 48911 2024-09-13T11:23:37.838 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.838 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:23:37.838 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.838 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:23:37.838 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.457776+0000", 2024-09-13T11:23:37.839 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:37.839 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.839 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.839 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:23:37.839 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.457776+0000", 2024-09-13T11:23:37.840 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:37.840 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.840 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.840 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:23:37.840 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.457788+0000", 2024-09-13T11:23:37.840 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2084e-05 2024-09-13T11:23:37.840 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.840 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.841 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:23:37.841 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.457823+0000", 2024-09-13T11:23:37.841 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.5002000000000001e-05 2024-09-13T11:23:37.841 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.841 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.841 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:23:37.841 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.457836+0000", 2024-09-13T11:23:37.841 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2988e-05 2024-09-13T11:23:37.841 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.841 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.841 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:23:37.841 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.457843+0000", 2024-09-13T11:23:37.841 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.5320000000000003e-06 2024-09-13T11:23:37.842 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.842 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.842 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:23:37.842 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.457895+0000", 2024-09-13T11:23:37.842 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.1690000000000001e-05 2024-09-13T11:23:37.842 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.842 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.842 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:23:37.842 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.458001+0000", 2024-09-13T11:23:37.842 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000106024 2024-09-13T11:23:37.842 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.842 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.843 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 7", 2024-09-13T11:23:37.843 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.458101+0000", 2024-09-13T11:23:37.843 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00010060699999999999 2024-09-13T11:23:37.843 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.843 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.843 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:23:37.843 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.459250+0000", 2024-09-13T11:23:37.843 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0011485670000000001 2024-09-13T11:23:37.843 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.843 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.843 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:37.843 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.682166+0000", 2024-09-13T11:23:37.843 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.22291609500000001 2024-09-13T11:23:37.844 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.844 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.844 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:23:37.844 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.682271+0000", 2024-09-13T11:23:37.844 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000105179 2024-09-13T11:23:37.844 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.845 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.845 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:23:37.845 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.683396+0000", 2024-09-13T11:23:37.845 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001125228 2024-09-13T11:23:37.846 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:37.846 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:23:37.846 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:37.846 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.847 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.847 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:48923 3.12 3:4ae1afd7:::benchmark_data_smithi177_31116_object48922:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e71)", 2024-09-13T11:23:37.847 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:25.465697+0000", 2024-09-13T11:23:37.848 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 12.354976356, 2024-09-13T11:23:37.848 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.21931837100000001, 2024-09-13T11:23:37.848 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:23:37.848 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:23:37.849 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:37.849 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:23:37.849 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:23:37.849 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:37.850 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 48923 2024-09-13T11:23:37.850 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.850 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:23:37.850 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.850 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:23:37.850 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.465697+0000", 2024-09-13T11:23:37.851 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:37.851 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.851 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.851 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:23:37.851 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.465697+0000", 2024-09-13T11:23:37.851 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:37.852 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.852 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.852 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:23:37.852 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.465700+0000", 2024-09-13T11:23:37.852 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.041e-06 2024-09-13T11:23:37.852 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.853 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.853 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:23:37.853 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.465757+0000", 2024-09-13T11:23:37.853 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.7664999999999998e-05 2024-09-13T11:23:37.853 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.853 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.854 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:23:37.854 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.465769+0000", 2024-09-13T11:23:37.854 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.219e-05 2024-09-13T11:23:37.854 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.854 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.854 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:23:37.855 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.465775+0000", 2024-09-13T11:23:37.855 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.6779999999999997e-06 2024-09-13T11:23:37.855 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.855 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.855 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:23:37.855 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.465821+0000", 2024-09-13T11:23:37.855 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.5577000000000001e-05 2024-09-13T11:23:37.856 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.856 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.856 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:23:37.856 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.465928+0000", 2024-09-13T11:23:37.856 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000107502 2024-09-13T11:23:37.856 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.856 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.857 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 7", 2024-09-13T11:23:37.857 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.466022+0000", 2024-09-13T11:23:37.857 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.3802999999999996e-05 2024-09-13T11:23:37.857 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.857 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.857 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:23:37.857 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.466901+0000", 2024-09-13T11:23:37.857 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00087950699999999997 2024-09-13T11:23:37.857 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.857 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.857 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:37.857 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.683622+0000", 2024-09-13T11:23:37.857 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.216720462 2024-09-13T11:23:37.858 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.858 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.858 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:23:37.858 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.683771+0000", 2024-09-13T11:23:37.858 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00014900199999999999 2024-09-13T11:23:37.858 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.858 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.858 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:23:37.858 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.685016+0000", 2024-09-13T11:23:37.858 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001244944 2024-09-13T11:23:37.858 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:37.858 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:23:37.858 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:37.859 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.859 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.859 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:48955 3.12 3:4ef2991b:::benchmark_data_smithi177_31116_object48954:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e71)", 2024-09-13T11:23:37.859 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:25.494091+0000", 2024-09-13T11:23:37.859 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 12.326582718999999, 2024-09-13T11:23:37.859 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.19235933599999999, 2024-09-13T11:23:37.859 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:23:37.859 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:23:37.859 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:37.859 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:23:37.859 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:23:37.859 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:37.859 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 48955 2024-09-13T11:23:37.860 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.860 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:23:37.860 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.860 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:23:37.860 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.494091+0000", 2024-09-13T11:23:37.860 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:37.861 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.861 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.861 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:23:37.861 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.494091+0000", 2024-09-13T11:23:37.861 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:37.861 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.862 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.862 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:23:37.862 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.494093+0000", 2024-09-13T11:23:37.862 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.0770000000000002e-06 2024-09-13T11:23:37.862 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.862 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.863 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:23:37.863 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.494160+0000", 2024-09-13T11:23:37.863 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.6798999999999995e-05 2024-09-13T11:23:37.863 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.863 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.863 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:23:37.863 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.494174+0000", 2024-09-13T11:23:37.864 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4489e-05 2024-09-13T11:23:37.864 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.864 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.864 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:23:37.865 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.494183+0000", 2024-09-13T11:23:37.865 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.4460000000000001e-06 2024-09-13T11:23:37.865 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.865 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.866 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:23:37.866 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.494239+0000", 2024-09-13T11:23:37.866 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.6140999999999997e-05 2024-09-13T11:23:37.866 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.866 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.866 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:23:37.867 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.494385+0000", 2024-09-13T11:23:37.867 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00014546399999999999 2024-09-13T11:23:37.867 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.867 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.867 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 7", 2024-09-13T11:23:37.867 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.494506+0000", 2024-09-13T11:23:37.867 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000121328 2024-09-13T11:23:37.868 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.868 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.868 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:23:37.868 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.495528+0000", 2024-09-13T11:23:37.868 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0010226079999999999 2024-09-13T11:23:37.868 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.869 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.869 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:37.869 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.685177+0000", 2024-09-13T11:23:37.869 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.18964847500000001 2024-09-13T11:23:37.869 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.869 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.870 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:23:37.870 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.685337+0000", 2024-09-13T11:23:37.870 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00016055400000000001 2024-09-13T11:23:37.870 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.870 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.870 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:23:37.871 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.686450+0000", 2024-09-13T11:23:37.871 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0011129550000000001 2024-09-13T11:23:37.871 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:37.871 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:23:37.871 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:37.872 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.872 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.872 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:48967 3.12 3:4abf2346:::benchmark_data_smithi177_31116_object48966:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e71)", 2024-09-13T11:23:37.872 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:25.509687+0000", 2024-09-13T11:23:37.872 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 12.310986594999999, 2024-09-13T11:23:37.872 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.17798804900000001, 2024-09-13T11:23:37.873 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:23:37.873 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:23:37.875 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:37.876 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:23:37.876 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:23:37.876 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:37.876 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 48967 2024-09-13T11:23:37.876 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.876 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:23:37.877 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.877 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:23:37.877 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.509687+0000", 2024-09-13T11:23:37.877 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:37.877 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.877 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.877 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:23:37.877 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.509687+0000", 2024-09-13T11:23:37.877 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:37.877 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.877 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.877 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:23:37.877 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.509689+0000", 2024-09-13T11:23:37.878 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.9719999999999999e-06 2024-09-13T11:23:37.878 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.878 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.878 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:23:37.878 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.509754+0000", 2024-09-13T11:23:37.878 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.5031999999999997e-05 2024-09-13T11:23:37.878 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.878 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.878 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:23:37.878 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.509787+0000", 2024-09-13T11:23:37.878 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.2891000000000001e-05 2024-09-13T11:23:37.878 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.878 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.879 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:23:37.879 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.509793+0000", 2024-09-13T11:23:37.879 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.1500000000000004e-06 2024-09-13T11:23:37.879 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.879 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.879 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:23:37.879 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.509846+0000", 2024-09-13T11:23:37.879 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.2926000000000001e-05 2024-09-13T11:23:37.879 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.879 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.879 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:23:37.880 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.509957+0000", 2024-09-13T11:23:37.880 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000110734 2024-09-13T11:23:37.880 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.880 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.880 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 7", 2024-09-13T11:23:37.880 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.510083+0000", 2024-09-13T11:23:37.881 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00012628300000000001 2024-09-13T11:23:37.881 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.881 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.881 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:23:37.881 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.511047+0000", 2024-09-13T11:23:37.881 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00096376800000000005 2024-09-13T11:23:37.882 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.882 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.882 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:37.882 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.686579+0000", 2024-09-13T11:23:37.882 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.17553200999999999 2024-09-13T11:23:37.882 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.883 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.883 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:23:37.883 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.686846+0000", 2024-09-13T11:23:37.883 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000267039 2024-09-13T11:23:37.883 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.883 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.884 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:23:37.884 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.687675+0000", 2024-09-13T11:23:37.884 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00082924400000000003 2024-09-13T11:23:37.884 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:37.884 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:23:37.884 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:37.884 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.885 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.885 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:48987 3.12 3:4e1e6d0c:::benchmark_data_smithi177_31116_object48986:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e71)", 2024-09-13T11:23:37.885 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:25.525640+0000", 2024-09-13T11:23:37.885 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 12.295033951000001, 2024-09-13T11:23:37.885 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.163044945, 2024-09-13T11:23:37.885 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:23:37.886 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:23:37.886 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:37.886 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:23:37.886 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:23:37.886 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:37.886 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 48987 2024-09-13T11:23:37.887 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.887 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:23:37.887 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.887 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:23:37.887 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.525640+0000", 2024-09-13T11:23:37.887 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:37.888 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.888 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.888 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:23:37.888 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.525640+0000", 2024-09-13T11:23:37.888 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:37.888 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.889 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.889 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:23:37.889 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.525651+0000", 2024-09-13T11:23:37.889 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1092e-05 2024-09-13T11:23:37.889 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.890 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.890 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:23:37.890 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.525795+0000", 2024-09-13T11:23:37.890 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00014417400000000001 2024-09-13T11:23:37.890 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.890 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.890 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:23:37.891 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.525834+0000", 2024-09-13T11:23:37.891 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.9026000000000002e-05 2024-09-13T11:23:37.891 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.891 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.891 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:23:37.891 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.525857+0000", 2024-09-13T11:23:37.892 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.2904999999999998e-05 2024-09-13T11:23:37.892 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.892 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.892 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:23:37.892 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.525967+0000", 2024-09-13T11:23:37.893 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000110188 2024-09-13T11:23:37.893 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.893 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.893 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:23:37.893 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.526214+0000", 2024-09-13T11:23:37.893 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00024667800000000001 2024-09-13T11:23:37.894 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.894 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.894 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 7", 2024-09-13T11:23:37.894 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.526499+0000", 2024-09-13T11:23:37.894 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00028463799999999998 2024-09-13T11:23:37.894 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.894 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.894 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:23:37.895 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.528226+0000", 2024-09-13T11:23:37.895 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0017276679999999999 2024-09-13T11:23:37.895 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.895 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.895 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:37.895 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.687803+0000", 2024-09-13T11:23:37.895 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.15957658899999999 2024-09-13T11:23:37.895 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.895 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.895 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:23:37.895 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.687986+0000", 2024-09-13T11:23:37.896 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00018332400000000001 2024-09-13T11:23:37.896 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.896 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.896 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:23:37.896 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.688685+0000", 2024-09-13T11:23:37.897 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00069866299999999998 2024-09-13T11:23:37.897 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:37.897 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:23:37.897 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:37.898 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.898 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.898 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:49023 3.12 3:4a994401:::benchmark_data_smithi177_31116_object49022:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e71)", 2024-09-13T11:23:37.898 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:25.562627+0000", 2024-09-13T11:23:37.898 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 12.258046581, 2024-09-13T11:23:37.899 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.12709894099999999, 2024-09-13T11:23:37.899 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:23:37.899 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:23:37.899 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:37.899 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:23:37.899 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:23:37.900 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:37.900 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 49023 2024-09-13T11:23:37.900 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.900 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:23:37.900 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.901 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:23:37.901 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.562627+0000", 2024-09-13T11:23:37.901 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:37.902 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.902 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.902 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:23:37.902 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.562627+0000", 2024-09-13T11:23:37.902 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:37.902 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.902 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.902 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:23:37.902 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.562636+0000", 2024-09-13T11:23:37.902 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.2530000000000004e-06 2024-09-13T11:23:37.903 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.903 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.903 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:23:37.903 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.562685+0000", 2024-09-13T11:23:37.903 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.9870999999999997e-05 2024-09-13T11:23:37.903 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.903 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.904 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:23:37.904 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.562717+0000", 2024-09-13T11:23:37.904 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.1418999999999999e-05 2024-09-13T11:23:37.904 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.904 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.904 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:23:37.904 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.562739+0000", 2024-09-13T11:23:37.904 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.1756e-05 2024-09-13T11:23:37.904 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.904 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.905 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:23:37.905 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.562823+0000", 2024-09-13T11:23:37.905 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.4129999999999996e-05 2024-09-13T11:23:37.905 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.905 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.905 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:23:37.905 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.562999+0000", 2024-09-13T11:23:37.905 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00017660199999999999 2024-09-13T11:23:37.905 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.905 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.905 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 7", 2024-09-13T11:23:37.906 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.563179+0000", 2024-09-13T11:23:37.906 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00018009800000000001 2024-09-13T11:23:37.906 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.906 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.906 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:23:37.906 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.564940+0000", 2024-09-13T11:23:37.906 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0017609310000000001 2024-09-13T11:23:37.906 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.906 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.906 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:37.907 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.688830+0000", 2024-09-13T11:23:37.907 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.123889754 2024-09-13T11:23:37.907 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.907 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.907 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:23:37.907 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.688919+0000", 2024-09-13T11:23:37.907 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.9010000000000003e-05 2024-09-13T11:23:37.908 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.908 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.908 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:23:37.908 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.689726+0000", 2024-09-13T11:23:37.908 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00080711699999999999 2024-09-13T11:23:37.908 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:37.908 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:23:37.908 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:37.908 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.908 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.908 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:49036 3.12 3:4ce03469:::benchmark_data_smithi177_31116_object49035:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e71)", 2024-09-13T11:23:37.908 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:25.576814+0000", 2024-09-13T11:23:37.909 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 12.243859769, 2024-09-13T11:23:37.909 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.114098698, 2024-09-13T11:23:37.909 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:23:37.909 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:23:37.909 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:37.909 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:23:37.909 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:23:37.909 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:37.909 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 49036 2024-09-13T11:23:37.909 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.909 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:23:37.909 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.910 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:23:37.910 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.576814+0000", 2024-09-13T11:23:37.910 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:37.910 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.910 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.910 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:23:37.910 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.576814+0000", 2024-09-13T11:23:37.910 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:37.910 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.910 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.911 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:23:37.911 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.576822+0000", 2024-09-13T11:23:37.911 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.6399999999999997e-06 2024-09-13T11:23:37.911 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.911 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.911 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:23:37.911 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.576875+0000", 2024-09-13T11:23:37.911 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.3517000000000003e-05 2024-09-13T11:23:37.911 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.911 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.911 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:23:37.911 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.576900+0000", 2024-09-13T11:23:37.911 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.5233e-05 2024-09-13T11:23:37.911 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.912 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.912 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:23:37.912 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.576915+0000", 2024-09-13T11:23:37.912 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4093e-05 2024-09-13T11:23:37.912 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.912 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.912 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:23:37.912 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.577008+0000", 2024-09-13T11:23:37.912 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.3794000000000004e-05 2024-09-13T11:23:37.912 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.912 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.912 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:23:37.913 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.577290+0000", 2024-09-13T11:23:37.913 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00028117300000000001 2024-09-13T11:23:37.913 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.913 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.913 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 7", 2024-09-13T11:23:37.913 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.577519+0000", 2024-09-13T11:23:37.913 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000229008 2024-09-13T11:23:37.913 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.913 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.913 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:23:37.913 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.579594+0000", 2024-09-13T11:23:37.913 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0020753189999999999 2024-09-13T11:23:37.914 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.914 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.914 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:37.914 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.689873+0000", 2024-09-13T11:23:37.914 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.110279455 2024-09-13T11:23:37.914 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.914 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.914 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:23:37.914 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.690011+0000", 2024-09-13T11:23:37.914 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00013810299999999999 2024-09-13T11:23:37.914 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.914 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.914 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:23:37.914 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.690913+0000", 2024-09-13T11:23:37.915 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00090136299999999997 2024-09-13T11:23:37.915 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:37.915 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:23:37.915 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:37.915 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.915 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.915 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:54637 3.22 3:44450106:::benchmark_data_smithi177_31116_object54636:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e75)", 2024-09-13T11:23:37.915 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:30.978960+0000", 2024-09-13T11:23:37.915 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 6.8417135189999998, 2024-09-13T11:23:37.915 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.960188928, 2024-09-13T11:23:37.915 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:23:37.915 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:23:37.915 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-09-13T11:23:37.916 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:23:37.916 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:23:37.916 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:37.916 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 54637 2024-09-13T11:23:37.916 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.916 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:23:37.916 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.916 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:23:37.916 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.978960+0000", 2024-09-13T11:23:37.916 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:37.917 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.917 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.917 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:23:37.917 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.978960+0000", 2024-09-13T11:23:37.917 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:37.917 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.917 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.917 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:23:37.917 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.978962+0000", 2024-09-13T11:23:37.917 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.46e-06 2024-09-13T11:23:37.917 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.917 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.917 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:23:37.918 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.979024+0000", 2024-09-13T11:23:37.918 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.1779999999999995e-05 2024-09-13T11:23:37.918 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.918 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.918 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:23:37.918 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.979037+0000", 2024-09-13T11:23:37.918 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2948e-05 2024-09-13T11:23:37.919 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.919 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.919 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:23:37.919 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.979048+0000", 2024-09-13T11:23:37.919 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1241e-05 2024-09-13T11:23:37.919 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.919 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.919 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:23:37.919 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:31.032951+0000", 2024-09-13T11:23:37.919 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.053903286000000002 2024-09-13T11:23:37.919 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.919 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.919 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-09-13T11:23:37.920 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:31.032958+0000", 2024-09-13T11:23:37.920 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.8109999999999997e-06 2024-09-13T11:23:37.920 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.920 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.920 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:23:37.920 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:31.933462+0000", 2024-09-13T11:23:37.920 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.90050463999999997 2024-09-13T11:23:37.920 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.920 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.920 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for degraded object", 2024-09-13T11:23:37.920 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:31.934668+0000", 2024-09-13T11:23:37.920 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0012053249999999999 2024-09-13T11:23:37.920 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.921 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.921 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:23:37.921 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:31.938887+0000", 2024-09-13T11:23:37.921 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0042187120000000003 2024-09-13T11:23:37.921 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.921 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.921 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:23:37.921 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:31.939149+0000", 2024-09-13T11:23:37.921 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000262725 2024-09-13T11:23:37.921 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:37.921 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:23:37.921 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:37.922 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.922 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.922 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:54663 3.12 3:49ad4b1b:::benchmark_data_smithi177_31116_object54662:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e75)", 2024-09-13T11:23:37.922 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:31.017498+0000", 2024-09-13T11:23:37.922 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 6.8031756210000003, 2024-09-13T11:23:37.922 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.137773485, 2024-09-13T11:23:37.922 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:23:37.922 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:23:37.922 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:37.922 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:23:37.922 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:23:37.923 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:37.923 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 54663 2024-09-13T11:23:37.923 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.923 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:23:37.923 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.923 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:23:37.923 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:31.017498+0000", 2024-09-13T11:23:37.923 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:37.923 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.923 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.923 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:23:37.923 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:31.017498+0000", 2024-09-13T11:23:37.923 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:37.924 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.924 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.924 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:23:37.924 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:31.017501+0000", 2024-09-13T11:23:37.924 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.0730000000000001e-06 2024-09-13T11:23:37.924 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.924 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.924 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:23:37.924 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:31.017547+0000", 2024-09-13T11:23:37.924 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.5553000000000002e-05 2024-09-13T11:23:37.924 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.924 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.925 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:23:37.925 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:31.017563+0000", 2024-09-13T11:23:37.925 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.6041000000000001e-05 2024-09-13T11:23:37.925 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.925 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.925 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:23:37.925 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:31.017573+0000", 2024-09-13T11:23:37.925 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0573e-05 2024-09-13T11:23:37.925 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.925 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.925 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:23:37.925 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:31.032699+0000", 2024-09-13T11:23:37.925 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.015125748 2024-09-13T11:23:37.926 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.926 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.926 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:23:37.926 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:31.032866+0000", 2024-09-13T11:23:37.926 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00016644300000000001 2024-09-13T11:23:37.926 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.926 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.926 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 7", 2024-09-13T11:23:37.926 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:31.033011+0000", 2024-09-13T11:23:37.926 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00014518300000000001 2024-09-13T11:23:37.926 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.927 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.927 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:23:37.927 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:31.042895+0000", 2024-09-13T11:23:37.927 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0098840160000000007 2024-09-13T11:23:37.927 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.927 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.927 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:37.927 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:31.154503+0000", 2024-09-13T11:23:37.927 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.11160790900000001 2024-09-13T11:23:37.927 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.927 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.927 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:23:37.928 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:31.154595+0000", 2024-09-13T11:23:37.928 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.2386000000000006e-05 2024-09-13T11:23:37.928 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.928 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.928 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:23:37.928 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:31.155272+0000", 2024-09-13T11:23:37.928 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00067655999999999999 2024-09-13T11:23:37.928 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:37.928 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:23:37.928 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:37.928 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.928 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.929 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPushReply(3.12 75/69 [PushReplyOp(3:4db63d39:::benchmark_data_smithi177_31116_object32641:head)])", 2024-09-13T11:23:37.929 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:31.646428+0000", 2024-09-13T11:23:37.929 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 6.1742453949999998, 2024-09-13T11:23:37.929 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.28409169299999998, 2024-09-13T11:23:37.929 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:23:37.929 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:23:37.929 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-09-13T11:23:37.929 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:23:37.929 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.929 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:23:37.929 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:31.646428+0000", 2024-09-13T11:23:37.929 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:37.929 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.930 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.930 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:23:37.930 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:31.646428+0000", 2024-09-13T11:23:37.930 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:37.930 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.930 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.930 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:23:37.930 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:31.646431+0000", 2024-09-13T11:23:37.930 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.3149999999999999e-06 2024-09-13T11:23:37.930 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.930 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.930 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:23:37.930 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:31.646438+0000", 2024-09-13T11:23:37.930 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.7379999999999994e-06 2024-09-13T11:23:37.931 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.931 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.931 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:23:37.931 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:31.928787+0000", 2024-09-13T11:23:37.931 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.28234828699999998 2024-09-13T11:23:37.931 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.931 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.931 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:23:37.931 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:31.929066+0000", 2024-09-13T11:23:37.931 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00027956500000000001 2024-09-13T11:23:37.931 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.931 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.931 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:23:37.931 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:31.929216+0000", 2024-09-13T11:23:37.932 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00014963500000000001 2024-09-13T11:23:37.932 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.932 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.932 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:23:37.932 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:31.930520+0000", 2024-09-13T11:23:37.932 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001304153 2024-09-13T11:23:37.932 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:37.932 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:23:37.932 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:37.932 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.932 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.932 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPushReply(3.12 76/69 [PushReplyOp(3:4a5f7a67:::benchmark_data_smithi177_31116_object32993:head)])", 2024-09-13T11:23:37.932 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:32.069056+0000", 2024-09-13T11:23:37.933 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 5.7516181140000002, 2024-09-13T11:23:37.933 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.17843624299999999, 2024-09-13T11:23:37.933 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:23:37.933 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:23:37.933 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-09-13T11:23:37.933 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:23:37.933 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.933 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:23:37.933 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.069056+0000", 2024-09-13T11:23:37.933 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:37.933 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.933 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.933 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:23:37.934 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.069056+0000", 2024-09-13T11:23:37.934 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:37.934 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.934 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.934 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:23:37.934 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.069057+0000", 2024-09-13T11:23:37.934 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4699999999999999e-06 2024-09-13T11:23:37.934 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.934 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.934 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:23:37.934 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.069065+0000", 2024-09-13T11:23:37.934 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.9740000000000002e-06 2024-09-13T11:23:37.935 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.935 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.935 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:23:37.935 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.246435+0000", 2024-09-13T11:23:37.935 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.17736979 2024-09-13T11:23:37.935 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.935 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.935 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:23:37.935 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.246450+0000", 2024-09-13T11:23:37.935 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4582e-05 2024-09-13T11:23:37.935 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.935 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.935 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:23:37.936 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.246556+0000", 2024-09-13T11:23:37.936 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00010608200000000001 2024-09-13T11:23:37.936 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.936 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.936 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:23:37.936 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.247492+0000", 2024-09-13T11:23:37.936 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00093634499999999999 2024-09-13T11:23:37.936 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:37.936 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:23:37.936 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:37.936 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.937 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.937 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPush(3.14 76/70 [PushOp(3:297981b4:::benchmark_data_smithi177_31116_object16264:head, version: 52'1034, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:297981b4:::benchmark_data_smithi177_31116_object16264:head@52'1034, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:23:37.937 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:32.119983+0000", 2024-09-13T11:23:37.937 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 5.7006904110000001, 2024-09-13T11:23:37.937 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.107915715, 2024-09-13T11:23:37.937 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:23:37.937 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:23:37.937 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:23:37.937 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:23:37.937 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.937 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:23:37.938 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.119983+0000", 2024-09-13T11:23:37.938 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:37.938 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.938 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.938 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:23:37.938 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.119983+0000", 2024-09-13T11:23:37.938 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:37.938 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.938 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.938 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:23:37.938 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.119985+0000", 2024-09-13T11:23:37.938 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5540000000000001e-06 2024-09-13T11:23:37.938 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.938 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.939 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:23:37.939 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.120161+0000", 2024-09-13T11:23:37.939 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00017632800000000001 2024-09-13T11:23:37.939 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.939 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.939 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:23:37.939 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.226182+0000", 2024-09-13T11:23:37.939 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.106020504 2024-09-13T11:23:37.939 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.939 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.939 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:23:37.939 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.226209+0000", 2024-09-13T11:23:37.939 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.7673000000000001e-05 2024-09-13T11:23:37.940 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.940 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.940 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:23:37.940 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.226421+0000", 2024-09-13T11:23:37.940 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000211967 2024-09-13T11:23:37.940 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.940 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.940 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:23:37.940 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.226455+0000", 2024-09-13T11:23:37.940 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.3182000000000002e-05 2024-09-13T11:23:37.940 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.940 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.940 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:23:37.941 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.227899+0000", 2024-09-13T11:23:37.941 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001444507 2024-09-13T11:23:37.941 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:37.941 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:23:37.941 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:37.941 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.941 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.941 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPush(3.15 76/70 [PushOp(3:aab036a3:::benchmark_data_smithi177_31116_object22075:head, version: 55'1344, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:aab036a3:::benchmark_data_smithi177_31116_object22075:head@55'1344, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:23:37.941 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:32.223005+0000", 2024-09-13T11:23:37.941 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 5.5976690920000003, 2024-09-13T11:23:37.941 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.13015405599999999, 2024-09-13T11:23:37.941 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:23:37.942 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:23:37.942 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:23:37.942 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:23:37.942 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.942 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:23:37.942 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.223005+0000", 2024-09-13T11:23:37.942 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:37.942 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.942 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.942 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:23:37.942 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.223005+0000", 2024-09-13T11:23:37.942 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:37.942 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.942 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.943 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:23:37.943 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.223006+0000", 2024-09-13T11:23:37.943 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.7379999999999999e-06 2024-09-13T11:23:37.943 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.943 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.943 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:23:37.943 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.223060+0000", 2024-09-13T11:23:37.943 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.3050000000000002e-05 2024-09-13T11:23:37.943 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.943 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.943 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:23:37.944 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.351482+0000", 2024-09-13T11:23:37.944 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.12842293299999999 2024-09-13T11:23:37.944 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.944 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.944 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:23:37.944 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.351512+0000", 2024-09-13T11:23:37.944 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.9212000000000001e-05 2024-09-13T11:23:37.944 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.944 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.944 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:23:37.944 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.351685+0000", 2024-09-13T11:23:37.945 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00017326099999999999 2024-09-13T11:23:37.945 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.945 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.945 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:23:37.945 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.351709+0000", 2024-09-13T11:23:37.945 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.4105e-05 2024-09-13T11:23:37.945 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.945 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.945 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:23:37.945 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.353159+0000", 2024-09-13T11:23:37.946 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0014497570000000001 2024-09-13T11:23:37.946 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:37.946 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:23:37.946 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:37.946 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.946 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.946 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPushReply(3.12 76/69 [PushReplyOp(3:4d1753c5:::benchmark_data_smithi177_31116_object33024:head)])", 2024-09-13T11:23:37.946 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:32.277343+0000", 2024-09-13T11:23:37.946 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 5.5433308769999998, 2024-09-13T11:23:37.947 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.14475505799999999, 2024-09-13T11:23:37.947 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:23:37.947 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:23:37.947 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-09-13T11:23:37.947 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:23:37.947 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.947 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:23:37.947 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.277343+0000", 2024-09-13T11:23:37.947 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:37.947 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.947 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.947 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:23:37.947 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.277343+0000", 2024-09-13T11:23:37.948 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:37.948 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.948 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.948 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:23:37.948 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.277346+0000", 2024-09-13T11:23:37.948 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.7800000000000001e-06 2024-09-13T11:23:37.949 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.949 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.949 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:23:37.949 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.277357+0000", 2024-09-13T11:23:37.949 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0983e-05 2024-09-13T11:23:37.949 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.949 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.949 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:23:37.949 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.420933+0000", 2024-09-13T11:23:37.949 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.14357647800000001 2024-09-13T11:23:37.949 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.950 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.950 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:23:37.950 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.420954+0000", 2024-09-13T11:23:37.950 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.0322999999999998e-05 2024-09-13T11:23:37.950 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.950 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.950 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:23:37.950 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.421048+0000", 2024-09-13T11:23:37.950 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.4263000000000002e-05 2024-09-13T11:23:37.950 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.950 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.951 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:23:37.951 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.422098+0000", 2024-09-13T11:23:37.951 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001050231 2024-09-13T11:23:37.951 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:37.951 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:23:37.951 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:37.951 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.951 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.951 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPush(3.15 76/70 [PushOp(3:ac553b6c:::benchmark_data_smithi177_31116_object22092:head, version: 55'1348, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:ac553b6c:::benchmark_data_smithi177_31116_object22092:head@55'1348, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:23:37.951 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:32.370110+0000", 2024-09-13T11:23:37.951 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 5.450564097, 2024-09-13T11:23:37.952 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.134376467, 2024-09-13T11:23:37.952 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:23:37.952 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:23:37.952 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:23:37.952 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:23:37.952 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.952 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:23:37.952 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.370110+0000", 2024-09-13T11:23:37.952 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:37.952 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.952 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.952 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:23:37.952 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.370110+0000", 2024-09-13T11:23:37.953 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:37.953 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.953 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.953 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:23:37.953 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.370112+0000", 2024-09-13T11:23:37.953 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.384e-06 2024-09-13T11:23:37.953 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.953 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.953 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:23:37.953 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.370175+0000", 2024-09-13T11:23:37.953 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.2529999999999999e-05 2024-09-13T11:23:37.953 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.953 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.954 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:23:37.954 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.502612+0000", 2024-09-13T11:23:37.954 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.13243751400000001 2024-09-13T11:23:37.954 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.954 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.954 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:23:37.954 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.502642+0000", 2024-09-13T11:23:37.954 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.9770000000000001e-05 2024-09-13T11:23:37.954 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.954 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.954 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:23:37.954 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.502808+0000", 2024-09-13T11:23:37.954 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00016618399999999999 2024-09-13T11:23:37.955 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.955 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.955 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:23:37.955 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.502832+0000", 2024-09-13T11:23:37.955 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.4136000000000001e-05 2024-09-13T11:23:37.955 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.955 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.955 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:23:37.955 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.504486+0000", 2024-09-13T11:23:37.955 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001653949 2024-09-13T11:23:37.955 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:37.956 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:23:37.956 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:37.956 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.956 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.956 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPushReply(3.12 76/69 [PushReplyOp(3:48826e21:::benchmark_data_smithi177_31116_object33140:head)])", 2024-09-13T11:23:37.956 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:32.427958+0000", 2024-09-13T11:23:37.956 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 5.3927162629999996, 2024-09-13T11:23:37.956 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.115554986, 2024-09-13T11:23:37.956 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:23:37.957 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:23:37.957 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-09-13T11:23:37.957 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:23:37.957 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.957 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:23:37.957 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.427958+0000", 2024-09-13T11:23:37.957 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:37.958 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.958 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.958 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:23:37.958 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.427958+0000", 2024-09-13T11:23:37.958 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:37.958 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.959 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.959 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:23:37.959 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.427960+0000", 2024-09-13T11:23:37.959 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.04e-06 2024-09-13T11:23:37.959 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.960 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.960 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:23:37.960 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.427969+0000", 2024-09-13T11:23:37.960 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.5170000000000005e-06 2024-09-13T11:23:37.960 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.960 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.961 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:23:37.961 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.542294+0000", 2024-09-13T11:23:37.961 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.11432479 2024-09-13T11:23:37.961 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.961 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.961 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:23:37.962 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.542318+0000", 2024-09-13T11:23:37.962 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.3790000000000001e-05 2024-09-13T11:23:37.962 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.962 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.962 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:23:37.962 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.542456+0000", 2024-09-13T11:23:37.963 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000137911 2024-09-13T11:23:37.963 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.963 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.963 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:23:37.963 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.543513+0000", 2024-09-13T11:23:37.964 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0010569380000000001 2024-09-13T11:23:37.964 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:37.964 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:23:37.964 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:37.964 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.965 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.965 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPush(3.15 76/70 [PushOp(3:ab4c41a1:::benchmark_data_smithi177_31116_object23392:head, version: 55'1416, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:ab4c41a1:::benchmark_data_smithi177_31116_object23392:head@55'1416, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:23:37.965 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:32.723597+0000", 2024-09-13T11:23:37.965 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 5.0970768040000003, 2024-09-13T11:23:37.965 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.205507309, 2024-09-13T11:23:37.966 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:23:37.966 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:23:37.966 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:23:37.966 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:23:37.967 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.967 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:23:37.967 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.723597+0000", 2024-09-13T11:23:37.967 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:37.967 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.967 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.968 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:23:37.968 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.723597+0000", 2024-09-13T11:23:37.968 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:37.968 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.968 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.969 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:23:37.969 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.723599+0000", 2024-09-13T11:23:37.969 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.1639999999999999e-06 2024-09-13T11:23:37.969 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.969 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.969 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:23:37.970 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.723661+0000", 2024-09-13T11:23:37.970 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.1851000000000001e-05 2024-09-13T11:23:37.970 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.970 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.970 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:23:37.971 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.927067+0000", 2024-09-13T11:23:37.971 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.20340633899999999 2024-09-13T11:23:37.971 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.971 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.971 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:23:37.972 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.927102+0000", 2024-09-13T11:23:37.972 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.4178999999999999e-05 2024-09-13T11:23:37.972 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.973 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.973 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:23:37.973 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.927424+0000", 2024-09-13T11:23:37.973 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000322536 2024-09-13T11:23:37.973 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.973 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.973 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:23:37.973 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.927457+0000", 2024-09-13T11:23:37.973 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.3191000000000001e-05 2024-09-13T11:23:37.973 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.973 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.973 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:23:37.974 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.929104+0000", 2024-09-13T11:23:37.974 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001647049 2024-09-13T11:23:37.974 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:37.974 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:23:37.974 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:37.974 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.974 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.974 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop_reply(client.4665.0:55795 3.12 e79/69 ondisk, result = 0)", 2024-09-13T11:23:37.974 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:35.893777+0000", 2024-09-13T11:23:37.974 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 1.926897171, 2024-09-13T11:23:37.974 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.189196325, 2024-09-13T11:23:37.974 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:23:37.974 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:23:37.974 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:23:37.975 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:23:37.975 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.975 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:23:37.975 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:35.893777+0000", 2024-09-13T11:23:37.975 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:37.975 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.975 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.975 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:23:37.975 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:35.893777+0000", 2024-09-13T11:23:37.975 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:37.975 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.975 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.975 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:23:37.975 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:35.893778+0000", 2024-09-13T11:23:37.976 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.6670000000000001e-06 2024-09-13T11:23:37.976 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.976 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.976 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:23:37.976 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:35.893784+0000", 2024-09-13T11:23:37.976 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.1989999999999997e-06 2024-09-13T11:23:37.976 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.976 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.976 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:23:37.976 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:36.082661+0000", 2024-09-13T11:23:37.976 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.18887743600000001 2024-09-13T11:23:37.976 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.976 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.976 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:23:37.977 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:36.082666+0000", 2024-09-13T11:23:37.977 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.2270000000000001e-06 2024-09-13T11:23:37.977 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.977 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.977 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:23:37.977 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:36.082784+0000", 2024-09-13T11:23:37.977 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00011776400000000001 2024-09-13T11:23:37.977 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.977 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.977 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:23:37.977 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:36.082799+0000", 2024-09-13T11:23:37.977 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5058999999999999e-05 2024-09-13T11:23:37.977 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:37.977 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:37.978 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:23:37.978 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:36.082973+0000", 2024-09-13T11:23:37.978 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00017397300000000001 2024-09-13T11:23:37.978 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:37.978 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:23:37.978 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:37.978 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:37.978 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:23:37.978 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-13T11:23:37.979 INFO:tasks.ceph.osd.12.smithi001.stderr:2024-09-13T11:23:37.972+0000 7f892c6e6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26314) UID: 0 2024-09-13T11:23:37.980 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-13T11:23:38.005 INFO:tasks.radosbench.radosbench.0.smithi177.stdout: 71 16 57981 57965 51.02 68.0625 0.00657814 0.0195936 2024-09-13T11:23:38.022 INFO:teuthology.orchestra.run.smithi001.stderr:noscrub is set 2024-09-13T11:23:38.023 INFO:tasks.ceph.osd.9.smithi092.stderr:2024-09-13T11:23:38.020+0000 7ff59ddf9640 -1 osd.9 80 handle_osd_map injecting map crc failure 2024-09-13T11:23:38.028 INFO:tasks.ceph.osd.4.smithi001.stderr:2024-09-13T11:23:38.024+0000 7ff8d0c3d640 -1 osd.4 80 handle_osd_map injecting map crc failure 2024-09-13T11:23:38.036 INFO:tasks.ceph.osd.15.smithi177.stderr:2024-09-13T11:23:38.032+0000 7f9d618a4640 -1 osd.15 80 handle_osd_map injecting map crc failure 2024-09-13T11:23:38.060 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T11:23:38.080 INFO:tasks.ceph.osd.7.smithi177.stderr:2024-09-13T11:23:38.076+0000 7f897b51d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26575) UID: 0 2024-09-13T11:23:38.181 INFO:tasks.ceph.osd.10.smithi162.stderr:2024-09-13T11:23:38.173+0000 7fb59c9f0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26317) UID: 0 2024-09-13T11:23:38.192 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-09-13T11:23:38.192 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [], 2024-09-13T11:23:38.193 INFO:teuthology.orchestra.run.smithi177.stdout: "num_ops": 0 2024-09-13T11:23:38.193 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-09-13T11:23:38.218 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-13T11:23:38.272 INFO:teuthology.orchestra.run.smithi001.stdout:osd.0: {} 2024-09-13T11:23:38.272 INFO:teuthology.orchestra.run.smithi001.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T11:23:38.284 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-09-13T11:23:38.277+0000 7f97befc9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26315) UID: 0 2024-09-13T11:23:38.299 INFO:teuthology.orchestra.run.smithi001.stdout:osd.1: {} 2024-09-13T11:23:38.299 INFO:teuthology.orchestra.run.smithi001.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T11:23:38.325 INFO:teuthology.orchestra.run.smithi001.stdout:osd.2: {} 2024-09-13T11:23:38.325 INFO:teuthology.orchestra.run.smithi001.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T11:23:38.350 INFO:teuthology.orchestra.run.smithi001.stdout:osd.3: {} 2024-09-13T11:23:38.350 INFO:teuthology.orchestra.run.smithi001.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T11:23:38.373 INFO:teuthology.orchestra.run.smithi001.stdout:osd.4: {} 2024-09-13T11:23:38.373 INFO:teuthology.orchestra.run.smithi001.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T11:23:38.385 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-13T11:23:38.380+0000 7f6a68775640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26307) UID: 0 2024-09-13T11:23:38.396 INFO:teuthology.orchestra.run.smithi001.stdout:osd.5: {} 2024-09-13T11:23:38.396 INFO:teuthology.orchestra.run.smithi001.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T11:23:38.423 INFO:teuthology.orchestra.run.smithi001.stdout:osd.6: {} 2024-09-13T11:23:38.423 INFO:teuthology.orchestra.run.smithi001.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T11:23:38.427 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-09-13T11:23:38.427 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:23:38.448 INFO:teuthology.orchestra.run.smithi001.stdout:osd.7: {} 2024-09-13T11:23:38.448 INFO:teuthology.orchestra.run.smithi001.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T11:23:38.451 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-13T11:23:38.470 INFO:teuthology.orchestra.run.smithi001.stdout:osd.8: {} 2024-09-13T11:23:38.470 INFO:teuthology.orchestra.run.smithi001.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-13T11:23:38.486 INFO:tasks.ceph.osd.12.smithi001.stderr:2024-09-13T11:23:38.480+0000 7f892c6e6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26314) UID: 0 2024-09-13T11:23:38.492 INFO:teuthology.orchestra.run.smithi001.stdout:osd.9: {} 2024-09-13T11:23:38.492 INFO:teuthology.orchestra.run.smithi001.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-13T11:23:38.519 INFO:teuthology.orchestra.run.smithi001.stdout:osd.10: {} 2024-09-13T11:23:38.519 INFO:teuthology.orchestra.run.smithi001.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-13T11:23:38.544 INFO:teuthology.orchestra.run.smithi001.stdout:osd.11: {} 2024-09-13T11:23:38.544 INFO:teuthology.orchestra.run.smithi001.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-13T11:23:38.566 INFO:teuthology.orchestra.run.smithi001.stdout:osd.12: {} 2024-09-13T11:23:38.566 INFO:teuthology.orchestra.run.smithi001.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-13T11:23:38.587 INFO:tasks.ceph.osd.7.smithi177.stderr:2024-09-13T11:23:38.580+0000 7f897b51d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26575) UID: 0 2024-09-13T11:23:38.590 INFO:teuthology.orchestra.run.smithi001.stdout:osd.13: {} 2024-09-13T11:23:38.590 INFO:teuthology.orchestra.run.smithi001.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-13T11:23:38.612 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-09-13T11:23:38.612 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:23:38.617 INFO:teuthology.orchestra.run.smithi001.stdout:osd.14: {} 2024-09-13T11:23:38.617 INFO:teuthology.orchestra.run.smithi001.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-13T11:23:38.633 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-13T11:23:38.639 INFO:teuthology.orchestra.run.smithi001.stdout:osd.15: {} 2024-09-13T11:23:38.639 INFO:teuthology.orchestra.run.smithi001.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-13T11:23:38.696 INFO:tasks.ceph.osd.8.smithi001.stderr:2024-09-13T11:23:38.692+0000 7fb9b48ae640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26312) UID: 0 2024-09-13T11:23:38.789 INFO:tasks.ceph.osd.9.smithi092.stderr:2024-09-13T11:23:38.788+0000 7ff5ab3d0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26348) UID: 0 2024-09-13T11:23:38.813 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-09-13T11:23:38.813 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:23:38.836 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-13T11:23:38.890 INFO:tasks.ceph.osd.4.smithi001.stderr:2024-09-13T11:23:38.884+0000 7ff8e0218640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26310) UID: 0 2024-09-13T11:23:38.991 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-09-13T11:23:38.985+0000 7f97befc9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26315) UID: 0 2024-09-13T11:23:38.992 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-09-13T11:23:38.992 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:23:39.005 INFO:tasks.radosbench.radosbench.0.smithi177.stdout: 72 16 58451 58435 50.7194 29.375 0.0448938 0.0195938 2024-09-13T11:23:39.014 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-13T11:23:39.065 INFO:tasks.ceph.osd.10.smithi162.stderr:2024-09-13T11:23:39.057+0000 7fb59141d640 -1 osd.10 81 handle_osd_map injecting map crc failure 2024-09-13T11:23:39.066 INFO:tasks.ceph.osd.2.smithi162.stderr:2024-09-13T11:23:39.057+0000 7fe91fd87640 -1 osd.2 81 handle_osd_map injecting map crc failure 2024-09-13T11:23:39.073 INFO:tasks.ceph.osd.14.smithi162.stderr:2024-09-13T11:23:39.065+0000 7f2350b08640 -1 osd.14 81 handle_osd_map injecting map crc failure 2024-09-13T11:23:39.092 INFO:tasks.ceph.osd.3.smithi177.stderr:2024-09-13T11:23:39.088+0000 7f10adba4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26574) UID: 0 2024-09-13T11:23:39.196 INFO:tasks.ceph.osd.3.smithi177.stderr:2024-09-13T11:23:39.188+0000 7f10adba4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26574) UID: 0 2024-09-13T11:23:39.199 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-09-13T11:23:39.200 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:23:39.237 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-09-13T11:23:39.329 INFO:tasks.ceph.osd.15.smithi177.stderr:2024-09-13T11:23:39.324+0000 7f9d6ce77640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26578) UID: 0 2024-09-13T11:23:39.394 INFO:tasks.ceph.osd.11.smithi177.stderr:2024-09-13T11:23:39.388+0000 7f24792c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26580) UID: 0 2024-09-13T11:23:39.401 INFO:tasks.ceph.osd.5.smithi092.stderr:2024-09-13T11:23:39.396+0000 7f6f00132640 -1 osd.5 81 handle_osd_map injecting map crc failure 2024-09-13T11:23:39.464 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-09-13T11:23:39.464 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:23:39.488 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-09-13T11:23:39.496 INFO:tasks.ceph.osd.5.smithi092.stderr:2024-09-13T11:23:39.492+0000 7f6f0b705640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26346) UID: 0 2024-09-13T11:23:39.596 INFO:tasks.ceph.osd.10.smithi162.stderr:2024-09-13T11:23:39.589+0000 7fb59c9f0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26317) UID: 0 2024-09-13T11:23:39.694 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-09-13T11:23:39.694 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:23:39.696 INFO:tasks.ceph.osd.9.smithi092.stderr:2024-09-13T11:23:39.692+0000 7ff5ab3d0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26348) UID: 0 2024-09-13T11:23:39.721 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-09-13T11:23:39.798 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-09-13T11:23:39.789+0000 7f97befc9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26315) UID: 0 2024-09-13T11:23:39.899 INFO:tasks.ceph.osd.8.smithi001.stderr:2024-09-13T11:23:39.892+0000 7fb9b48ae640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26312) UID: 0 2024-09-13T11:23:39.899 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-09-13T11:23:39.900 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:23:39.926 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_ops_in_flight 2024-09-13T11:23:39.961 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15] out_osds: [] dead_osds: [] live_osds: [0, 4, 8, 12, 5, 9, 13, 2, 6, 10, 14, 3, 7, 11, 15, 1] 2024-09-13T11:23:39.962 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 0 1 2024-09-13T11:23:39.999 INFO:tasks.ceph.osd.10.smithi162.stderr:2024-09-13T11:23:39.993+0000 7fb59c9f0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26317) UID: 0 2024-09-13T11:23:40.005 INFO:tasks.radosbench.radosbench.0.smithi177.stdout: 73 16 59539 59523 50.956 68 0.00605741 0.0196171 2024-09-13T11:23:40.049 INFO:tasks.ceph.osd.9.smithi092.stderr:2024-09-13T11:23:40.044+0000 7ff59ddf9640 -1 osd.9 82 handle_osd_map injecting map crc failure 2024-09-13T11:23:40.050 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-13T11:23:40.050 INFO:tasks.ceph.osd.12.smithi001.stderr:2024-09-13T11:23:40.044+0000 7f891f10f640 -1 osd.12 82 handle_osd_map injecting map crc failure 2024-09-13T11:23:40.101 INFO:tasks.ceph.osd.13.smithi092.stderr:2024-09-13T11:23:40.096+0000 7f3e5afea640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26350) UID: 0 2024-09-13T11:23:40.122 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-09-13T11:23:40.122 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:23:40.153 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_blocked_ops 2024-09-13T11:23:40.207 INFO:tasks.ceph.osd.7.smithi177.stderr:2024-09-13T11:23:40.196+0000 7f897b51d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26575) UID: 0 2024-09-13T11:23:40.302 INFO:tasks.ceph.osd.7.smithi177.stderr:2024-09-13T11:23:40.296+0000 7f897b51d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26575) UID: 0 2024-09-13T11:23:40.320 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-09-13T11:23:40.320 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:23:40.344 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_historic_ops 2024-09-13T11:23:40.403 INFO:tasks.ceph.osd.13.smithi092.stderr:2024-09-13T11:23:40.400+0000 7f3e5afea640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26350) UID: 0 2024-09-13T11:23:40.504 INFO:tasks.ceph.osd.1.smithi092.stderr:2024-09-13T11:23:40.500+0000 7fe7789c5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33718) UID: 0 2024-09-13T11:23:40.531 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-09-13T11:23:40.531 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:23:40.550 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-13T11:23:40.605 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-13T11:23:40.600+0000 7f6a68775640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26307) UID: 0 2024-09-13T11:23:40.656 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T11:23:40.690 INFO:teuthology.orchestra.run.smithi092.stdout:ERROR: (22) Invalid argument 2024-09-13T11:23:40.690 INFO:teuthology.orchestra.run.smithi092.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:23:40.706 INFO:tasks.ceph.osd.4.smithi001.stderr:2024-09-13T11:23:40.700+0000 7ff8e0218640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26310) UID: 0 2024-09-13T11:23:40.710 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-13T11:23:40.808 INFO:tasks.ceph.osd.12.smithi001.stderr:2024-09-13T11:23:40.804+0000 7f892c6e6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26314) UID: 0 2024-09-13T11:23:40.827 INFO:teuthology.orchestra.run.smithi001.stdout:osd.0: {} 2024-09-13T11:23:40.827 INFO:teuthology.orchestra.run.smithi001.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T11:23:40.839 INFO:teuthology.orchestra.run.smithi092.stdout:ERROR: (22) Invalid argument 2024-09-13T11:23:40.839 INFO:teuthology.orchestra.run.smithi092.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:23:40.852 INFO:teuthology.orchestra.run.smithi001.stdout:osd.1: {} 2024-09-13T11:23:40.852 INFO:teuthology.orchestra.run.smithi001.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T11:23:40.855 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-13T11:23:40.881 INFO:teuthology.orchestra.run.smithi001.stdout:osd.2: {} 2024-09-13T11:23:40.881 INFO:teuthology.orchestra.run.smithi001.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T11:23:40.908 INFO:tasks.ceph.osd.4.smithi001.stderr:2024-09-13T11:23:40.904+0000 7ff8e0218640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26310) UID: 0 2024-09-13T11:23:40.912 INFO:teuthology.orchestra.run.smithi001.stdout:osd.3: {} 2024-09-13T11:23:40.912 INFO:teuthology.orchestra.run.smithi001.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T11:23:40.935 INFO:teuthology.orchestra.run.smithi001.stdout:osd.4: {} 2024-09-13T11:23:40.935 INFO:teuthology.orchestra.run.smithi001.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T11:23:40.959 INFO:teuthology.orchestra.run.smithi001.stdout:osd.5: {} 2024-09-13T11:23:40.959 INFO:teuthology.orchestra.run.smithi001.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T11:23:40.984 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-09-13T11:23:40.984 INFO:teuthology.orchestra.run.smithi092.stdout: "size": 20, 2024-09-13T11:23:40.984 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 600, 2024-09-13T11:23:40.984 INFO:teuthology.orchestra.run.smithi092.stdout: "ops": [ 2024-09-13T11:23:40.984 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:40.985 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:3 3.4 3.125e09e4 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e40)", 2024-09-13T11:23:40.985 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:46.304641+0000", 2024-09-13T11:23:40.985 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 54.677961480999997, 2024-09-13T11:23:40.985 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.374372119, 2024-09-13T11:23:40.985 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:23:40.985 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:23:40.985 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "reached pg", 2024-09-13T11:23:40.985 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:23:40.985 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:23:40.985 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:40.985 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 3 2024-09-13T11:23:40.985 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:40.985 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:23:40.986 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:40.986 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:23:40.986 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.304641+0000", 2024-09-13T11:23:40.986 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:40.986 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:40.986 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:40.986 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:23:40.986 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.304641+0000", 2024-09-13T11:23:40.986 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:40.986 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:40.986 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:40.986 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:23:40.986 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.304644+0000", 2024-09-13T11:23:40.986 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.5130000000000001e-06 2024-09-13T11:23:40.987 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:40.987 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:40.987 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:23:40.987 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305058+0000", 2024-09-13T11:23:40.987 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000414277 2024-09-13T11:23:40.987 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:40.987 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:40.987 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:23:40.987 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305124+0000", 2024-09-13T11:23:40.987 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.5813000000000002e-05 2024-09-13T11:23:40.987 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:40.987 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:40.987 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:23:40.988 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305156+0000", 2024-09-13T11:23:40.988 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.1489000000000003e-05 2024-09-13T11:23:40.988 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:40.988 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:40.988 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:23:40.988 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.678939+0000", 2024-09-13T11:23:40.988 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.37378339999999999 2024-09-13T11:23:40.988 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:40.988 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:40.988 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:23:40.988 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.679013+0000", 2024-09-13T11:23:40.988 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.3627000000000005e-05 2024-09-13T11:23:40.988 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:40.988 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:23:40.989 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:40.989 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:40.989 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:40.989 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:29 3.4 3.b7bbfbe4 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-09-13T11:23:40.989 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:46.305215+0000", 2024-09-13T11:23:40.989 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 54.677387178000004, 2024-09-13T11:23:40.989 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.37387964600000001, 2024-09-13T11:23:40.989 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:23:40.989 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:23:40.989 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "reached pg", 2024-09-13T11:23:40.989 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:23:40.989 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:23:40.989 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:40.990 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 29 2024-09-13T11:23:40.990 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:40.990 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:23:40.990 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:40.990 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:23:40.990 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305215+0000", 2024-09-13T11:23:40.990 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:40.990 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:40.990 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:40.990 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:23:40.990 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305215+0000", 2024-09-13T11:23:40.991 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:40.991 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:40.991 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:40.991 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:23:40.991 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305218+0000", 2024-09-13T11:23:40.991 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.7030000000000002e-06 2024-09-13T11:23:40.991 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:40.991 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:40.991 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:23:40.991 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305366+0000", 2024-09-13T11:23:40.991 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00014825400000000001 2024-09-13T11:23:40.991 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:40.991 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:40.992 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:23:40.992 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305386+0000", 2024-09-13T11:23:40.992 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.9689000000000001e-05 2024-09-13T11:23:40.992 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:40.992 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:40.992 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:23:40.992 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305403+0000", 2024-09-13T11:23:40.992 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.7600000000000001e-05 2024-09-13T11:23:40.992 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:40.992 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:40.992 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:23:40.992 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.679067+0000", 2024-09-13T11:23:40.992 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.37366339100000001 2024-09-13T11:23:40.993 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:40.993 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:40.993 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:23:40.993 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.679095+0000", 2024-09-13T11:23:40.993 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.8008999999999999e-05 2024-09-13T11:23:40.993 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:40.993 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:23:40.993 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:40.993 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:40.993 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:40.993 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:30 3.4 3.547ec0f4 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-09-13T11:23:40.993 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:46.305453+0000", 2024-09-13T11:23:40.993 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 54.677149053999997, 2024-09-13T11:23:40.994 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.373764914, 2024-09-13T11:23:40.994 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:23:40.994 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:23:40.994 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "reached pg", 2024-09-13T11:23:40.994 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:23:40.994 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:23:40.994 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:40.994 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 30 2024-09-13T11:23:40.994 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:40.994 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:23:40.994 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:40.994 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:23:40.994 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305453+0000", 2024-09-13T11:23:40.995 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:40.995 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:40.995 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:40.995 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:23:40.995 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305453+0000", 2024-09-13T11:23:40.995 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:40.995 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:40.995 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:40.995 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:23:40.995 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305456+0000", 2024-09-13T11:23:40.995 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.9560000000000002e-06 2024-09-13T11:23:40.995 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:40.996 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:40.996 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:23:40.996 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305597+0000", 2024-09-13T11:23:40.997 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00014122700000000001 2024-09-13T11:23:40.997 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:40.997 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:40.997 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:23:40.997 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305615+0000", 2024-09-13T11:23:40.997 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.8139999999999999e-05 2024-09-13T11:23:40.997 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:40.997 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:40.997 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:23:40.997 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305632+0000", 2024-09-13T11:23:40.997 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.6101999999999999e-05 2024-09-13T11:23:40.997 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:40.997 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:40.998 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:23:40.998 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.679195+0000", 2024-09-13T11:23:40.998 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.373563486 2024-09-13T11:23:40.998 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:40.998 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:40.998 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:23:40.998 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.679218+0000", 2024-09-13T11:23:40.998 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.3003e-05 2024-09-13T11:23:40.998 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:40.998 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:23:40.998 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:40.998 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:40.998 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:40.999 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:70 3.4 3.1404a604 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-09-13T11:23:40.999 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:46.305665+0000", 2024-09-13T11:23:40.999 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 54.676936906000002, 2024-09-13T11:23:40.999 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.37365550199999997, 2024-09-13T11:23:40.999 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:23:40.999 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:23:40.999 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "reached pg", 2024-09-13T11:23:40.999 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:23:40.999 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:23:40.999 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:40.999 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 70 2024-09-13T11:23:40.999 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:40.999 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:23:40.999 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:41.000 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:23:41.000 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305665+0000", 2024-09-13T11:23:41.000 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:41.000 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:41.000 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:41.000 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:23:41.000 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305665+0000", 2024-09-13T11:23:41.000 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:41.000 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:41.000 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:41.000 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:23:41.000 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305668+0000", 2024-09-13T11:23:41.000 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.5890000000000001e-06 2024-09-13T11:23:41.001 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:41.001 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:41.001 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:23:41.001 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305791+0000", 2024-09-13T11:23:41.001 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000123545 2024-09-13T11:23:41.001 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:41.001 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:41.001 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:23:41.001 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305808+0000", 2024-09-13T11:23:41.001 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.668e-05 2024-09-13T11:23:41.001 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:41.001 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:41.001 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:23:41.001 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305824+0000", 2024-09-13T11:23:41.002 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.6385e-05 2024-09-13T11:23:41.002 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:41.002 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:41.002 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:23:41.002 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.679305+0000", 2024-09-13T11:23:41.002 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.37348059900000002 2024-09-13T11:23:41.002 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:41.002 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:41.002 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:23:41.002 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.679321+0000", 2024-09-13T11:23:41.002 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.5704000000000001e-05 2024-09-13T11:23:41.002 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:41.002 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:23:41.002 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:41.003 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:41.003 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:41.003 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:25408 3.4 3:294e23bd:::benchmark_data_smithi177_31116_object25407:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e56)", 2024-09-13T11:23:41.003 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:04.495455+0000", 2024-09-13T11:23:41.003 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 36.487147254, 2024-09-13T11:23:41.003 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.29897367600000002, 2024-09-13T11:23:41.003 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:23:41.003 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:23:41.003 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:41.003 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:23:41.003 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:23:41.003 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:41.003 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 25408 2024-09-13T11:23:41.004 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:41.004 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:23:41.004 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:41.004 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:23:41.004 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.495455+0000", 2024-09-13T11:23:41.004 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:41.004 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:41.004 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:41.004 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:23:41.004 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.495455+0000", 2024-09-13T11:23:41.004 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:41.004 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:41.004 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:41.005 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:23:41.005 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.495459+0000", 2024-09-13T11:23:41.005 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.5630000000000002e-06 2024-09-13T11:23:41.005 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:41.005 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:41.005 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:23:41.005 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.495545+0000", 2024-09-13T11:23:41.005 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.5704000000000005e-05 2024-09-13T11:23:41.005 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:41.005 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:41.005 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:23:41.005 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.495581+0000", 2024-09-13T11:23:41.005 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.5689000000000003e-05 2024-09-13T11:23:41.006 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:41.006 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:41.006 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:23:41.006 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.495597+0000", 2024-09-13T11:23:41.006 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.6053e-05 2024-09-13T11:23:41.006 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:41.006 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:41.006 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:23:41.006 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.495721+0000", 2024-09-13T11:23:41.006 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00012379099999999999 2024-09-13T11:23:41.006 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:41.006 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:41.006 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:23:41.007 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.495940+0000", 2024-09-13T11:23:41.007 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00021967699999999999 2024-09-13T11:23:41.007 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:41.007 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:41.007 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 12", 2024-09-13T11:23:41.007 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.496162+0000", 2024-09-13T11:23:41.007 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000221917 2024-09-13T11:23:41.007 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:41.007 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:41.007 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:23:41.007 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.498699+0000", 2024-09-13T11:23:41.007 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0025366690000000001 2024-09-13T11:23:41.007 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:41.007 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:41.008 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:41.008 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.794139+0000", 2024-09-13T11:23:41.008 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.295440025 2024-09-13T11:23:41.008 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:41.008 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:41.008 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:23:41.008 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.794306+0000", 2024-09-13T11:23:41.008 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00016689799999999999 2024-09-13T11:23:41.008 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:41.008 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:41.008 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:23:41.008 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.794429+0000", 2024-09-13T11:23:41.008 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00012269000000000001 2024-09-13T11:23:41.009 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:41.009 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:23:41.009 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:41.009 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:41.009 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:41.009 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop(client.4665.0:30855 3.b e59/40 3:dc59714f:::benchmark_data_smithi177_31116_object30854:head v 59'1930, mlcod=59'1927)", 2024-09-13T11:23:41.009 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:09.550713+0000", 2024-09-13T11:23:41.009 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 31.431889462000001, 2024-09-13T11:23:41.009 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.17337753, 2024-09-13T11:23:41.009 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:23:41.009 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:23:41.009 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:41.009 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:23:41.009 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:41.010 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:23:41.010 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:09.550713+0000", 2024-09-13T11:23:41.010 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:41.010 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:41.010 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:41.010 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:23:41.010 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:09.550713+0000", 2024-09-13T11:23:41.010 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:41.010 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:41.010 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:41.010 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:23:41.010 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:09.550716+0000", 2024-09-13T11:23:41.010 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.5099999999999999e-06 2024-09-13T11:23:41.010 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:41.011 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:41.011 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:23:41.011 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:09.550764+0000", 2024-09-13T11:23:41.011 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.7389000000000003e-05 2024-09-13T11:23:41.011 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:41.011 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:41.011 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:23:41.011 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:09.719062+0000", 2024-09-13T11:23:41.011 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.16829854499999999 2024-09-13T11:23:41.011 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:41.011 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:41.011 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:23:41.011 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:09.719075+0000", 2024-09-13T11:23:41.012 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.2969000000000001e-05 2024-09-13T11:23:41.012 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:41.012 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:41.012 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:23:41.012 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:09.719216+0000", 2024-09-13T11:23:41.012 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000141266 2024-09-13T11:23:41.012 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:41.012 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:41.012 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:23:41.012 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:09.719277+0000", 2024-09-13T11:23:41.012 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.0244999999999997e-05 2024-09-13T11:23:41.012 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:41.012 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:41.013 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:23:41.013 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:09.724019+0000", 2024-09-13T11:23:41.013 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0047426020000000003 2024-09-13T11:23:41.013 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:41.013 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:41.013 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:23:41.013 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:09.724090+0000", 2024-09-13T11:23:41.013 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.1004e-05 2024-09-13T11:23:41.013 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:41.013 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:23:41.013 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:41.013 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:41.013 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:41.014 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop(client.4665.0:45770 3.b e68/40 3:d2ece503:::benchmark_data_smithi177_31116_object45769:head v 68'2878, mlcod=68'2875)", 2024-09-13T11:23:41.014 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:20.925073+0000", 2024-09-13T11:23:41.014 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 20.057529236000001, 2024-09-13T11:23:41.014 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.14884730400000001, 2024-09-13T11:23:41.014 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:23:41.014 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:23:41.014 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:41.014 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:23:41.014 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:41.014 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:23:41.014 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:20.925073+0000", 2024-09-13T11:23:41.014 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:41.014 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:41.015 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:41.015 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:23:41.015 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:20.925073+0000", 2024-09-13T11:23:41.015 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:41.015 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:41.015 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:41.015 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:23:41.015 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:20.925075+0000", 2024-09-13T11:23:41.015 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.9360000000000002e-06 2024-09-13T11:23:41.015 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:41.015 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:41.015 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:23:41.015 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:20.925100+0000", 2024-09-13T11:23:41.016 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.5602e-05 2024-09-13T11:23:41.016 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:41.016 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:41.016 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:23:41.016 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:21.070362+0000", 2024-09-13T11:23:41.016 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.14526132 2024-09-13T11:23:41.016 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:41.016 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:41.016 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:23:41.016 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:21.070375+0000", 2024-09-13T11:23:41.016 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.3636e-05 2024-09-13T11:23:41.016 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:41.016 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:41.017 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:23:41.017 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:21.070523+0000", 2024-09-13T11:23:41.017 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00014711999999999999 2024-09-13T11:23:41.017 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:41.017 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:41.017 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:23:41.017 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:21.070577+0000", 2024-09-13T11:23:41.017 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.3987000000000003e-05 2024-09-13T11:23:41.017 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:41.017 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:41.017 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:23:41.017 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:21.073840+0000", 2024-09-13T11:23:41.017 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0032633050000000002 2024-09-13T11:23:41.018 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:41.018 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:41.018 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:23:41.018 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:21.073920+0000", 2024-09-13T11:23:41.018 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.0397999999999998e-05 2024-09-13T11:23:41.018 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:41.018 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:23:41.018 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:41.018 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:41.018 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:41.018 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop(client.4665.0:48977 3.8 e71/67 3:1030343e:::benchmark_data_smithi177_31116_object48976:head v 71'2885, mlcod=71'2882)", 2024-09-13T11:23:41.019 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:25.518319+0000", 2024-09-13T11:23:41.019 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 15.46428308, 2024-09-13T11:23:41.019 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.14420084699999999, 2024-09-13T11:23:41.019 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:23:41.019 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:23:41.019 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:41.020 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:23:41.020 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:41.020 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:23:41.020 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:25.518319+0000", 2024-09-13T11:23:41.020 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:41.020 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:41.020 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:41.020 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:23:41.020 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:25.518319+0000", 2024-09-13T11:23:41.020 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:41.020 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:41.020 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:41.020 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:23:41.021 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:25.518320+0000", 2024-09-13T11:23:41.021 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.215e-06 2024-09-13T11:23:41.021 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:41.021 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:41.021 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:23:41.021 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:25.518385+0000", 2024-09-13T11:23:41.021 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.5124999999999997e-05 2024-09-13T11:23:41.021 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:41.021 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:41.021 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:23:41.021 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:25.661380+0000", 2024-09-13T11:23:41.021 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.142994184 2024-09-13T11:23:41.021 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:41.021 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:41.022 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:23:41.022 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:25.661384+0000", 2024-09-13T11:23:41.022 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.3150000000000002e-06 2024-09-13T11:23:41.022 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:41.022 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:41.022 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:23:41.022 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:25.661433+0000", 2024-09-13T11:23:41.022 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.8789999999999999e-05 2024-09-13T11:23:41.022 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:41.022 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:41.022 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:23:41.022 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:25.661446+0000", 2024-09-13T11:23:41.022 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.2938e-05 2024-09-13T11:23:41.023 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:41.023 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:41.023 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:23:41.023 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:25.662490+0000", 2024-09-13T11:23:41.023 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.001044779 2024-09-13T11:23:41.023 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:41.023 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:41.023 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:23:41.023 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:25.662520+0000", 2024-09-13T11:23:41.023 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.9501000000000001e-05 2024-09-13T11:23:41.023 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:41.023 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:23:41.023 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:41.023 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:41.024 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:41.024 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:49963 3.18 3:19b86b73:::benchmark_data_smithi177_31116_object49962:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e72)", 2024-09-13T11:23:41.024 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:26.307557+0000", 2024-09-13T11:23:41.024 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 14.675045279000001, 2024-09-13T11:23:41.024 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.22140597300000001, 2024-09-13T11:23:41.024 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:23:41.024 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:23:41.024 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:41.024 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:23:41.024 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:23:41.024 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:41.024 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 49963 2024-09-13T11:23:41.025 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:41.025 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:23:41.025 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:41.025 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:23:41.025 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:26.307557+0000", 2024-09-13T11:23:41.025 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:41.025 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:41.025 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:41.025 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:23:41.025 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:26.307557+0000", 2024-09-13T11:23:41.025 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:41.025 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:41.025 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:41.025 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:23:41.026 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:26.307558+0000", 2024-09-13T11:23:41.026 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.3740000000000001e-06 2024-09-13T11:23:41.026 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:41.026 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:41.026 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:23:41.026 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:26.516580+0000", 2024-09-13T11:23:41.026 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.209021596 2024-09-13T11:23:41.026 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:41.026 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:41.026 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:23:41.026 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:26.516614+0000", 2024-09-13T11:23:41.026 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.3858e-05 2024-09-13T11:23:41.026 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:41.026 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:41.027 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:23:41.027 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:26.516628+0000", 2024-09-13T11:23:41.027 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.4715999999999999e-05 2024-09-13T11:23:41.027 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:41.027 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:41.027 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:23:41.027 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:26.516790+0000", 2024-09-13T11:23:41.027 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00016138 2024-09-13T11:23:41.027 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:41.027 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:41.027 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:23:41.027 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:26.517123+0000", 2024-09-13T11:23:41.027 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00033296600000000002 2024-09-13T11:23:41.027 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:41.028 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:41.028 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 0,10", 2024-09-13T11:23:41.028 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:26.517367+0000", 2024-09-13T11:23:41.028 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00024416900000000003 2024-09-13T11:23:41.028 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:41.028 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:41.028 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:23:41.028 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:26.526939+0000", 2024-09-13T11:23:41.028 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0095718650000000006 2024-09-13T11:23:41.028 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:41.028 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:41.028 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:41.028 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:26.527095+0000", 2024-09-13T11:23:41.029 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00015643800000000001 2024-09-13T11:23:41.029 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:41.029 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:41.029 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:41.029 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:26.527254+0000", 2024-09-13T11:23:41.029 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000158383 2024-09-13T11:23:41.029 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:41.029 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:41.029 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:23:41.029 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:26.527405+0000", 2024-09-13T11:23:41.029 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000151731 2024-09-13T11:23:41.029 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:41.029 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:41.029 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:23:41.029 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:26.528963+0000", 2024-09-13T11:23:41.030 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0015574969999999999 2024-09-13T11:23:41.030 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:41.030 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:23:41.030 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:41.030 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:41.030 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:41.030 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "MOSDPGPush(3.e 72/57 [PushOp(3:7481641d:::benchmark_data_smithi177_31116_object23454:head, version: 55'1541, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:7481641d:::benchmark_data_smithi177_31116_object23454:head@55'1541, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:23:41.030 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:26.368394+0000", 2024-09-13T11:23:41.030 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 14.614208501, 2024-09-13T11:23:41.030 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.16488114500000001, 2024-09-13T11:23:41.030 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:23:41.030 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:23:41.030 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-09-13T11:23:41.031 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:23:41.031 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:41.031 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:23:41.031 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:26.368394+0000", 2024-09-13T11:23:41.031 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:41.031 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:41.031 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:41.031 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:23:41.031 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:26.368394+0000", 2024-09-13T11:23:41.031 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:41.031 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:41.031 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:41.031 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:23:41.031 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:26.368395+0000", 2024-09-13T11:23:41.032 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.054e-06 2024-09-13T11:23:41.032 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:41.032 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:41.032 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:23:41.032 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:26.368453+0000", 2024-09-13T11:23:41.032 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.8621e-05 2024-09-13T11:23:41.032 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:41.032 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:41.032 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:23:41.032 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:26.528803+0000", 2024-09-13T11:23:41.032 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.16034983 2024-09-13T11:23:41.033 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:41.033 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:41.033 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:23:41.033 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:26.528834+0000", 2024-09-13T11:23:41.033 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.1288000000000003e-05 2024-09-13T11:23:41.033 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:41.033 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:41.033 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:23:41.033 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:26.529064+0000", 2024-09-13T11:23:41.033 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00022959 2024-09-13T11:23:41.033 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:41.033 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:41.033 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:23:41.033 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:26.529097+0000", 2024-09-13T11:23:41.034 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.3423999999999997e-05 2024-09-13T11:23:41.034 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:41.034 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:41.034 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:23:41.034 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:26.533275+0000", 2024-09-13T11:23:41.034 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0041773380000000001 2024-09-13T11:23:41.034 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:41.034 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:23:41.034 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:41.034 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:41.034 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:41.034 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "MOSDPGPush(3.19 72/70 [PushOp(3:9f24a12e:::benchmark_data_smithi177_31116_object6489:head, version: 46'406, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:9f24a12e:::benchmark_data_smithi177_31116_object6489:head@46'406, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:23:41.034 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:26.381103+0000", 2024-09-13T11:23:41.034 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 14.601498751999999, 2024-09-13T11:23:41.035 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.15612552199999999, 2024-09-13T11:23:41.035 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:23:41.035 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:23:41.035 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-09-13T11:23:41.035 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:23:41.035 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:41.035 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:23:41.035 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:26.381103+0000", 2024-09-13T11:23:41.035 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:41.035 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:41.035 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:41.035 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:23:41.035 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:26.381103+0000", 2024-09-13T11:23:41.035 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:41.036 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:41.036 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:41.036 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:23:41.036 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:26.381105+0000", 2024-09-13T11:23:41.036 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.1769999999999999e-06 2024-09-13T11:23:41.036 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:41.036 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:41.036 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:23:41.036 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:26.381205+0000", 2024-09-13T11:23:41.036 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000100623 2024-09-13T11:23:41.036 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:41.036 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:41.036 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:23:41.037 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:26.535299+0000", 2024-09-13T11:23:41.037 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.15409379100000001 2024-09-13T11:23:41.037 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:41.037 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:41.037 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:23:41.037 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:26.535371+0000", 2024-09-13T11:23:41.037 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.1668000000000006e-05 2024-09-13T11:23:41.037 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:41.037 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:41.037 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:23:41.037 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:26.535612+0000", 2024-09-13T11:23:41.037 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00024121400000000001 2024-09-13T11:23:41.038 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:41.038 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:41.038 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:23:41.038 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:26.535639+0000", 2024-09-13T11:23:41.038 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.7324000000000001e-05 2024-09-13T11:23:41.038 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:41.038 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:41.038 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:23:41.038 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:26.537229+0000", 2024-09-13T11:23:41.038 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0015897249999999999 2024-09-13T11:23:41.038 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:41.038 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:23:41.038 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:41.038 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:41.039 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:41.039 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "MOSDPGPushReply(3.14 75/70 [PushReplyOp(3:2f14928a:::benchmark_data_smithi177_31116_object15123:head)])", 2024-09-13T11:23:41.039 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:31.569662+0000", 2024-09-13T11:23:41.039 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 9.4129404819999998, 2024-09-13T11:23:41.039 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.35759148299999999, 2024-09-13T11:23:41.039 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:23:41.039 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:23:41.039 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "reached pg", 2024-09-13T11:23:41.039 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:23:41.039 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:41.039 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:23:41.039 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:31.569662+0000", 2024-09-13T11:23:41.039 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:41.040 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:41.040 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:41.040 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:23:41.040 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:31.569662+0000", 2024-09-13T11:23:41.040 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:41.040 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:41.040 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:41.040 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:23:41.040 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:31.569663+0000", 2024-09-13T11:23:41.040 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.333e-06 2024-09-13T11:23:41.040 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:41.040 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:41.040 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:23:41.040 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:31.569667+0000", 2024-09-13T11:23:41.041 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.8929999999999998e-06 2024-09-13T11:23:41.041 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:41.041 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:41.041 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:23:41.041 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:31.923221+0000", 2024-09-13T11:23:41.041 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.35355413699999999 2024-09-13T11:23:41.041 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:41.041 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:41.041 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:23:41.041 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:31.923230+0000", 2024-09-13T11:23:41.041 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.6829999999999997e-06 2024-09-13T11:23:41.041 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:41.042 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:41.042 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:23:41.042 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:31.923807+0000", 2024-09-13T11:23:41.042 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00057686800000000002 2024-09-13T11:23:41.042 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:41.042 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:41.042 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:23:41.042 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:31.927253+0000", 2024-09-13T11:23:41.042 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.003446569 2024-09-13T11:23:41.042 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:41.042 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:23:41.042 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:41.042 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:41.042 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:41.043 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "MOSDPGPushReply(3.14 76/70 [PushReplyOp(3:297981b4:::benchmark_data_smithi177_31116_object16264:head)])", 2024-09-13T11:23:41.043 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:32.122153+0000", 2024-09-13T11:23:41.043 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 8.8604487929999998, 2024-09-13T11:23:41.043 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.20534369899999999, 2024-09-13T11:23:41.043 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:23:41.043 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:23:41.043 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "reached pg", 2024-09-13T11:23:41.043 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:23:41.043 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:41.043 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:23:41.043 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:32.122153+0000", 2024-09-13T11:23:41.043 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:41.043 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:41.044 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:41.044 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:23:41.044 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:32.122153+0000", 2024-09-13T11:23:41.044 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:41.044 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:41.044 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:41.044 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:23:41.044 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:32.122154+0000", 2024-09-13T11:23:41.044 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.5300000000000003e-07 2024-09-13T11:23:41.044 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:41.044 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:41.044 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:23:41.045 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:32.122158+0000", 2024-09-13T11:23:41.045 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.8770000000000003e-06 2024-09-13T11:23:41.045 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:41.045 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:41.045 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:23:41.045 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:32.326042+0000", 2024-09-13T11:23:41.045 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.20388401 2024-09-13T11:23:41.045 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:41.045 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:41.045 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:23:41.045 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:32.326049+0000", 2024-09-13T11:23:41.045 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.8369999999999999e-06 2024-09-13T11:23:41.045 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:41.046 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:41.046 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:23:41.046 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:32.326135+0000", 2024-09-13T11:23:41.046 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.6215e-05 2024-09-13T11:23:41.046 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:41.046 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:41.046 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:23:41.046 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:32.327497+0000", 2024-09-13T11:23:41.046 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0013619070000000001 2024-09-13T11:23:41.046 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:41.046 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:23:41.046 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:41.046 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:41.046 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:41.046 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "MOSDPGPushReply(3.14 79/70 [PushReplyOp(3:2af22440:::benchmark_data_smithi177_31116_object31881:head)])", 2024-09-13T11:23:41.047 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:35.510507+0000", 2024-09-13T11:23:41.047 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 5.4720954270000002, 2024-09-13T11:23:41.047 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.139790213, 2024-09-13T11:23:41.047 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:23:41.047 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:23:41.047 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "reached pg", 2024-09-13T11:23:41.047 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:23:41.047 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:41.047 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:23:41.047 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:35.510507+0000", 2024-09-13T11:23:41.047 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:41.047 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:41.047 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:41.047 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:23:41.048 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:35.510507+0000", 2024-09-13T11:23:41.048 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:41.048 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:41.048 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:41.048 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:23:41.048 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:35.510508+0000", 2024-09-13T11:23:41.048 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.561e-06 2024-09-13T11:23:41.048 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:41.048 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:41.048 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:23:41.048 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:35.510514+0000", 2024-09-13T11:23:41.048 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.587e-06 2024-09-13T11:23:41.048 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:41.049 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:41.049 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:23:41.049 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:35.649604+0000", 2024-09-13T11:23:41.049 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.13908973599999999 2024-09-13T11:23:41.049 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:41.049 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:41.049 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:23:41.049 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:35.649613+0000", 2024-09-13T11:23:41.049 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.1819999999999995e-06 2024-09-13T11:23:41.049 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:41.049 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:41.049 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:23:41.049 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:35.649670+0000", 2024-09-13T11:23:41.049 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.7176999999999998e-05 2024-09-13T11:23:41.050 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:41.050 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:41.050 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:23:41.050 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:35.650297+0000", 2024-09-13T11:23:41.050 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00062697000000000002 2024-09-13T11:23:41.050 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:41.050 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:23:41.050 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:41.050 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:41.050 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:41.050 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:55670 3.14 3:2e43484c:::benchmark_data_smithi177_31116_object55669:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e79)", 2024-09-13T11:23:41.050 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:35.763374+0000", 2024-09-13T11:23:41.050 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 5.2192285700000003, 2024-09-13T11:23:41.051 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.18805201299999999, 2024-09-13T11:23:41.051 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:23:41.051 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:23:41.051 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:41.051 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:23:41.051 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:23:41.051 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:41.051 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 55670 2024-09-13T11:23:41.051 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:41.051 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:23:41.051 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:41.051 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:23:41.051 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:35.763374+0000", 2024-09-13T11:23:41.051 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:41.052 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:41.052 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:41.052 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:23:41.052 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:35.763374+0000", 2024-09-13T11:23:41.052 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:41.052 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:41.052 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:41.052 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:23:41.052 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:35.763376+0000", 2024-09-13T11:23:41.052 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.1239999999999999e-06 2024-09-13T11:23:41.052 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:41.052 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:41.052 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:23:41.053 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:35.763428+0000", 2024-09-13T11:23:41.053 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.2553000000000003e-05 2024-09-13T11:23:41.053 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:41.053 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:41.053 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:23:41.053 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:35.763437+0000", 2024-09-13T11:23:41.053 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.4649999999999994e-06 2024-09-13T11:23:41.053 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:41.053 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:41.053 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:23:41.053 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:35.763442+0000", 2024-09-13T11:23:41.053 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.5310000000000001e-06 2024-09-13T11:23:41.054 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:41.054 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:41.054 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:23:41.054 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:35.763498+0000", 2024-09-13T11:23:41.054 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.5591999999999999e-05 2024-09-13T11:23:41.054 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:41.054 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:41.054 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:23:41.054 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:35.763660+0000", 2024-09-13T11:23:41.054 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000161965 2024-09-13T11:23:41.054 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:41.054 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:41.054 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 3,14", 2024-09-13T11:23:41.055 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:35.763850+0000", 2024-09-13T11:23:41.055 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00019013999999999999 2024-09-13T11:23:41.055 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:41.055 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:41.055 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:23:41.055 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:35.768339+0000", 2024-09-13T11:23:41.055 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0044892739999999997 2024-09-13T11:23:41.055 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:41.055 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:41.055 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:41.055 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:35.770403+0000", 2024-09-13T11:23:41.055 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0020638150000000001 2024-09-13T11:23:41.056 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:41.056 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:41.056 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:41.056 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:35.951091+0000", 2024-09-13T11:23:41.056 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.18068817500000001 2024-09-13T11:23:41.056 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:41.056 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:41.056 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:23:41.056 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:35.951146+0000", 2024-09-13T11:23:41.056 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.4537000000000003e-05 2024-09-13T11:23:41.056 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:41.056 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:41.056 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:23:41.056 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:35.951426+0000", 2024-09-13T11:23:41.057 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00027984199999999998 2024-09-13T11:23:41.057 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:41.057 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:23:41.057 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:41.057 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:41.057 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:41.057 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "MOSDPGPushReply(3.14 79/70 [PushReplyOp(3:2c86f72b:::benchmark_data_smithi177_31116_object32132:head)])", 2024-09-13T11:23:41.057 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:35.770311+0000", 2024-09-13T11:23:41.057 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 5.2122911260000002, 2024-09-13T11:23:41.057 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.17980474099999999, 2024-09-13T11:23:41.057 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:23:41.057 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:23:41.057 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "reached pg", 2024-09-13T11:23:41.058 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:23:41.058 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:41.058 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:23:41.058 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:35.770311+0000", 2024-09-13T11:23:41.058 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:41.058 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:41.058 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:41.058 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:23:41.058 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:35.770311+0000", 2024-09-13T11:23:41.058 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:41.058 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:41.058 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:41.058 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:23:41.059 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:35.770312+0000", 2024-09-13T11:23:41.059 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.9399999999999993e-07 2024-09-13T11:23:41.059 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:41.059 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:41.059 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:23:41.059 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:35.770319+0000", 2024-09-13T11:23:41.059 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.635e-06 2024-09-13T11:23:41.059 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:41.059 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:41.059 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:23:41.059 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:35.949590+0000", 2024-09-13T11:23:41.059 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.17927082799999999 2024-09-13T11:23:41.059 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:41.059 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:41.060 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:23:41.060 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:35.949596+0000", 2024-09-13T11:23:41.060 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.3969999999999999e-06 2024-09-13T11:23:41.060 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:41.060 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:41.060 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:23:41.060 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:35.949666+0000", 2024-09-13T11:23:41.060 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.0087999999999997e-05 2024-09-13T11:23:41.060 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:41.060 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:41.060 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:23:41.060 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:35.950116+0000", 2024-09-13T11:23:41.060 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00044979900000000002 2024-09-13T11:23:41.061 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:41.061 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:23:41.061 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:41.061 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:41.061 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:41.061 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop_reply(client.4665.0:55670 3.14 e79/70 ondisk, result = 0)", 2024-09-13T11:23:41.061 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:35.774267+0000", 2024-09-13T11:23:41.061 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 5.2083350169999996, 2024-09-13T11:23:41.061 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.177164408, 2024-09-13T11:23:41.061 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:23:41.061 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:23:41.061 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-09-13T11:23:41.061 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:23:41.062 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:41.062 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:23:41.062 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:35.774267+0000", 2024-09-13T11:23:41.062 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:41.062 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:41.062 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:41.062 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:23:41.062 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:35.774267+0000", 2024-09-13T11:23:41.062 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:41.062 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:41.062 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:41.062 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:23:41.063 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:35.774269+0000", 2024-09-13T11:23:41.063 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.3820000000000001e-06 2024-09-13T11:23:41.063 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:41.063 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:41.063 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:23:41.063 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:35.774276+0000", 2024-09-13T11:23:41.063 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.0520000000000004e-06 2024-09-13T11:23:41.063 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:41.063 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:41.063 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:23:41.063 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:35.951004+0000", 2024-09-13T11:23:41.063 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.17672813800000001 2024-09-13T11:23:41.063 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:41.064 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:41.064 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:23:41.064 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:35.951013+0000", 2024-09-13T11:23:41.064 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.5480000000000007e-06 2024-09-13T11:23:41.064 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:41.064 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:41.064 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:23:41.064 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:35.951076+0000", 2024-09-13T11:23:41.064 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.3046999999999995e-05 2024-09-13T11:23:41.064 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:41.064 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:41.064 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:23:41.064 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:35.951085+0000", 2024-09-13T11:23:41.065 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.3950000000000001e-06 2024-09-13T11:23:41.065 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:41.065 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:41.065 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:23:41.065 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:35.951432+0000", 2024-09-13T11:23:41.065 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00034684599999999999 2024-09-13T11:23:41.065 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:41.065 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:23:41.065 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:41.065 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:41.065 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:41.065 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "MOSDPGPush(3.19 79/70 [PushOp(3:99a0d211:::benchmark_data_smithi177_31116_object31006:head, version: 59'1942, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:99a0d211:::benchmark_data_smithi177_31116_object31006:head@59'1942, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:23:41.065 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:36.285317+0000", 2024-09-13T11:23:41.066 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 4.6972856580000002, 2024-09-13T11:23:41.066 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.14587452100000001, 2024-09-13T11:23:41.066 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:23:41.066 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:23:41.066 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-09-13T11:23:41.066 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:23:41.066 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:41.066 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:23:41.066 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:36.285317+0000", 2024-09-13T11:23:41.066 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:41.066 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:41.066 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:41.066 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:23:41.067 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:36.285317+0000", 2024-09-13T11:23:41.067 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:41.067 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:41.067 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:41.067 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:23:41.067 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:36.285318+0000", 2024-09-13T11:23:41.067 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.096e-06 2024-09-13T11:23:41.067 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:41.067 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:41.067 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:23:41.067 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:36.285398+0000", 2024-09-13T11:23:41.067 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.0436e-05 2024-09-13T11:23:41.067 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:41.068 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:41.068 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:23:41.068 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:36.430111+0000", 2024-09-13T11:23:41.068 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.14471340799999999 2024-09-13T11:23:41.068 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:41.068 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:41.068 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:23:41.068 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:36.430144+0000", 2024-09-13T11:23:41.068 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.2725999999999998e-05 2024-09-13T11:23:41.068 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:41.068 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:41.068 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:23:41.068 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:36.430336+0000", 2024-09-13T11:23:41.069 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000192232 2024-09-13T11:23:41.069 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:41.069 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:41.069 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:23:41.069 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:36.430349+0000", 2024-09-13T11:23:41.069 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.2529e-05 2024-09-13T11:23:41.069 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:41.069 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:41.069 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:23:41.069 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:36.431191+0000", 2024-09-13T11:23:41.069 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00084209400000000005 2024-09-13T11:23:41.069 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:41.069 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:23:41.070 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:41.070 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:41.070 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:41.070 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "MOSDPGPushReply(3.1b 80/78 [PushReplyOp(3:d933f51a:::benchmark_data_smithi177_31116_object4707:head)])", 2024-09-13T11:23:41.070 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:37.792024+0000", 2024-09-13T11:23:41.070 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 3.1905777190000002, 2024-09-13T11:23:41.070 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.1532374, 2024-09-13T11:23:41.070 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:23:41.070 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:23:41.070 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "reached pg", 2024-09-13T11:23:41.070 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:23:41.070 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:41.070 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:23:41.070 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:37.792024+0000", 2024-09-13T11:23:41.071 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:41.071 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:41.071 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:41.071 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:23:41.071 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:37.792024+0000", 2024-09-13T11:23:41.071 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:41.071 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:41.071 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:41.071 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:23:41.071 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:37.792026+0000", 2024-09-13T11:23:41.071 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.4610000000000001e-06 2024-09-13T11:23:41.071 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:41.071 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:41.071 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:23:41.072 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:37.792035+0000", 2024-09-13T11:23:41.072 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.7520000000000002e-06 2024-09-13T11:23:41.072 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:41.072 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:41.072 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:23:41.072 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:37.943665+0000", 2024-09-13T11:23:41.072 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.151630442 2024-09-13T11:23:41.072 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:41.072 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:41.072 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:23:41.072 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:37.943671+0000", 2024-09-13T11:23:41.072 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.3199999999999996e-06 2024-09-13T11:23:41.073 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:41.073 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:41.073 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:23:41.073 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:37.943717+0000", 2024-09-13T11:23:41.073 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.5358999999999997e-05 2024-09-13T11:23:41.073 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:41.073 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:41.073 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:23:41.073 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:37.945262+0000", 2024-09-13T11:23:41.073 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0015450660000000001 2024-09-13T11:23:41.073 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:41.073 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:23:41.074 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:41.074 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:41.074 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:41.074 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "MOSDPGPush(3.19 83/70 [PushOp(3:9863b4ba:::benchmark_data_smithi177_31116_object45096:head, version: 68'2794, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:9863b4ba:::benchmark_data_smithi177_31116_object45096:head@68'2794, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:23:41.074 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:40.671028+0000", 2024-09-13T11:23:41.074 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 0.31157388600000002, 2024-09-13T11:23:41.074 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.22869726100000001, 2024-09-13T11:23:41.074 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:23:41.074 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:23:41.074 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-09-13T11:23:41.074 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:23:41.074 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:41.075 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:23:41.075 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:40.671028+0000", 2024-09-13T11:23:41.075 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:41.075 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:41.075 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:41.075 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:23:41.075 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:40.671028+0000", 2024-09-13T11:23:41.075 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:41.075 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:41.075 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:41.075 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:23:41.075 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:40.671029+0000", 2024-09-13T11:23:41.075 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.2200000000000003e-07 2024-09-13T11:23:41.076 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:41.076 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:41.076 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:23:41.076 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:40.671098+0000", 2024-09-13T11:23:41.076 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.9241999999999999e-05 2024-09-13T11:23:41.076 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:41.076 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:41.076 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:23:41.076 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:40.898838+0000", 2024-09-13T11:23:41.076 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.22773932699999999 2024-09-13T11:23:41.076 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:41.076 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:41.076 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:23:41.076 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:40.898848+0000", 2024-09-13T11:23:41.077 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0373999999999999e-05 2024-09-13T11:23:41.077 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:41.077 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:41.077 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:23:41.077 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:40.898930+0000", 2024-09-13T11:23:41.077 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.1865000000000003e-05 2024-09-13T11:23:41.077 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:41.077 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:41.077 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:23:41.077 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:40.898938+0000", 2024-09-13T11:23:41.077 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.6109999999999996e-06 2024-09-13T11:23:41.077 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:41.077 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:41.077 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:23:41.078 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:40.899726+0000", 2024-09-13T11:23:41.078 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00078801999999999995 2024-09-13T11:23:41.078 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:41.078 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:23:41.078 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:41.078 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:41.078 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:23:41.078 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-09-13T11:23:41.079 INFO:tasks.radosbench.radosbench.0.smithi177.stdout: 74 16 60084 60068 50.7277 34.0625 0.0118024 0.0195565 2024-09-13T11:23:41.079 INFO:teuthology.orchestra.run.smithi001.stdout:osd.6: {} 2024-09-13T11:23:41.080 INFO:teuthology.orchestra.run.smithi001.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T11:23:41.080 INFO:teuthology.orchestra.run.smithi001.stdout:osd.7: {} 2024-09-13T11:23:41.081 INFO:teuthology.orchestra.run.smithi001.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T11:23:41.081 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-13T11:23:41.082 INFO:teuthology.orchestra.run.smithi001.stdout:osd.8: {} 2024-09-13T11:23:41.082 INFO:teuthology.orchestra.run.smithi001.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-13T11:23:41.082 INFO:teuthology.orchestra.run.smithi001.stderr:nodeep-scrub is set 2024-09-13T11:23:41.083 INFO:teuthology.orchestra.run.smithi001.stderr:reweighted osd.0 to 1 (10000) 2024-09-13T11:23:41.083 INFO:teuthology.orchestra.run.smithi001.stdout:osd.9: {} 2024-09-13T11:23:41.083 INFO:teuthology.orchestra.run.smithi001.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-13T11:23:41.084 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-09-13T11:23:41.073+0000 7f97befc9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26315) UID: 0 2024-09-13T11:23:41.101 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 1 1 2024-09-13T11:23:41.113 INFO:teuthology.orchestra.run.smithi001.stdout:osd.10: {} 2024-09-13T11:23:41.113 INFO:teuthology.orchestra.run.smithi001.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-13T11:23:41.180 INFO:tasks.ceph.osd.13.smithi092.stderr:2024-09-13T11:23:41.176+0000 7f3e5afea640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26350) UID: 0 2024-09-13T11:23:41.183 INFO:teuthology.orchestra.run.smithi001.stdout:osd.11: {} 2024-09-13T11:23:41.183 INFO:teuthology.orchestra.run.smithi001.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-13T11:23:41.217 INFO:teuthology.orchestra.run.smithi001.stdout:osd.12: {} 2024-09-13T11:23:41.218 INFO:teuthology.orchestra.run.smithi001.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-13T11:23:41.255 INFO:teuthology.orchestra.run.smithi001.stdout:osd.13: {} 2024-09-13T11:23:41.256 INFO:teuthology.orchestra.run.smithi001.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-13T11:23:41.256 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-13T11:23:41.256 INFO:teuthology.orchestra.run.smithi001.stdout: "ops": [], 2024-09-13T11:23:41.256 INFO:teuthology.orchestra.run.smithi001.stdout: "num_ops": 0 2024-09-13T11:23:41.256 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-13T11:23:41.279 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-13T11:23:41.280 INFO:tasks.ceph.osd.7.smithi177.stderr:2024-09-13T11:23:41.276+0000 7f897b51d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26575) UID: 0 2024-09-13T11:23:41.286 INFO:teuthology.orchestra.run.smithi001.stdout:osd.14: {} 2024-09-13T11:23:41.286 INFO:teuthology.orchestra.run.smithi001.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-13T11:23:41.332 INFO:teuthology.orchestra.run.smithi001.stdout:osd.15: {} 2024-09-13T11:23:41.333 INFO:teuthology.orchestra.run.smithi001.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-13T11:23:41.408 INFO:tasks.ceph.osd.4.smithi001.stderr:2024-09-13T11:23:41.404+0000 7ff8e0218640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26310) UID: 0 2024-09-13T11:23:41.440 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-13T11:23:41.440 INFO:teuthology.orchestra.run.smithi001.stdout: "ops": [], 2024-09-13T11:23:41.440 INFO:teuthology.orchestra.run.smithi001.stdout: "complaint_time": 30, 2024-09-13T11:23:41.440 INFO:teuthology.orchestra.run.smithi001.stdout: "num_blocked_ops": 0 2024-09-13T11:23:41.440 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-13T11:23:41.460 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-13T11:23:41.482 INFO:tasks.ceph.osd.15.smithi177.stderr:2024-09-13T11:23:41.476+0000 7f9d6ce77640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26578) UID: 0 2024-09-13T11:23:41.584 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-13T11:23:41.580+0000 7f6a68775640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26307) UID: 0 2024-09-13T11:23:41.653 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-13T11:23:41.653 INFO:teuthology.orchestra.run.smithi001.stdout: "size": 20, 2024-09-13T11:23:41.653 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 600, 2024-09-13T11:23:41.653 INFO:teuthology.orchestra.run.smithi001.stdout: "ops": [ 2024-09-13T11:23:41.653 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:41.653 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDPGPush(3.15 71/70 [PushOp(3:a8e7ff00:::benchmark_data_smithi177_31116_object3532:head, version: 44'215, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:a8e7ff00:::benchmark_data_smithi177_31116_object3532:head@44'215, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:23:41.654 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:25.425119+0000", 2024-09-13T11:23:41.654 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 16.226018569000001, 2024-09-13T11:23:41.654 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.095533526999999993, 2024-09-13T11:23:41.654 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:23:41.654 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:23:41.654 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:23:41.654 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:23:41.654 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:41.654 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:23:41.654 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:25.425119+0000", 2024-09-13T11:23:41.654 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:41.654 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:41.655 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:41.655 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:23:41.655 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:25.425119+0000", 2024-09-13T11:23:41.655 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:41.655 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:41.655 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:41.655 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:23:41.655 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:25.425121+0000", 2024-09-13T11:23:41.655 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.61e-06 2024-09-13T11:23:41.655 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:41.655 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:41.655 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:23:41.655 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:25.425266+0000", 2024-09-13T11:23:41.656 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000145184 2024-09-13T11:23:41.656 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:41.656 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:41.656 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:23:41.656 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:25.520033+0000", 2024-09-13T11:23:41.656 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.094766403999999999 2024-09-13T11:23:41.656 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:41.656 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:41.656 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:23:41.656 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:25.520066+0000", 2024-09-13T11:23:41.656 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.3169e-05 2024-09-13T11:23:41.656 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:41.656 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:41.657 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:41.657 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:25.520149+0000", 2024-09-13T11:23:41.657 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.3571000000000003e-05 2024-09-13T11:23:41.657 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:41.657 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:41.657 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:23:41.657 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:25.520157+0000", 2024-09-13T11:23:41.657 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.9650000000000008e-06 2024-09-13T11:23:41.657 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:41.657 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:41.657 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:23:41.657 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:25.520653+0000", 2024-09-13T11:23:41.657 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00049562399999999996 2024-09-13T11:23:41.657 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:41.658 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:41.658 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:41.658 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:41.658 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:41.658 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDPGPush(3.18 75/70 [PushOp(3:1fba5393:::benchmark_data_smithi177_31116_object10005:head, version: 48'652, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:1fba5393:::benchmark_data_smithi177_31116_object10005:head@48'652, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:23:41.658 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:31.800150+0000", 2024-09-13T11:23:41.658 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 9.8509884799999998, 2024-09-13T11:23:41.658 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.11023854499999999, 2024-09-13T11:23:41.658 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:23:41.658 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:23:41.658 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:23:41.658 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:23:41.658 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:41.658 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:23:41.659 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.800150+0000", 2024-09-13T11:23:41.659 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:41.659 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:41.659 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:41.659 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:23:41.659 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.800150+0000", 2024-09-13T11:23:41.659 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:41.659 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:41.659 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:41.659 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:23:41.659 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.800151+0000", 2024-09-13T11:23:41.659 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.454e-06 2024-09-13T11:23:41.659 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:41.660 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:41.660 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:23:41.660 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.800198+0000", 2024-09-13T11:23:41.660 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.6489000000000002e-05 2024-09-13T11:23:41.660 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:41.660 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:41.660 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:23:41.660 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.909264+0000", 2024-09-13T11:23:41.660 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.109065992 2024-09-13T11:23:41.660 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:41.660 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:41.660 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:23:41.660 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.909281+0000", 2024-09-13T11:23:41.660 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.7844999999999999e-05 2024-09-13T11:23:41.661 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:41.661 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:41.661 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:41.661 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.909433+0000", 2024-09-13T11:23:41.661 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00015173000000000001 2024-09-13T11:23:41.661 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:41.661 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:41.661 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:23:41.661 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.909448+0000", 2024-09-13T11:23:41.661 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.4925e-05 2024-09-13T11:23:41.661 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:41.661 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:41.661 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:23:41.662 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.910388+0000", 2024-09-13T11:23:41.662 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00094010999999999997 2024-09-13T11:23:41.662 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:41.662 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:41.662 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:41.662 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:41.662 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:41.662 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4665.0:55383 3.21 e79/78 3:87f90f84:::benchmark_data_smithi177_31116_object55382:head v 79'3083, mlcod=0'0)", 2024-09-13T11:23:41.662 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:35.477154+0000", 2024-09-13T11:23:41.662 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 6.1739844259999996, 2024-09-13T11:23:41.662 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.116974571, 2024-09-13T11:23:41.662 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:23:41.662 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:23:41.663 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:41.663 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:23:41.663 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:41.663 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:23:41.663 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:35.477154+0000", 2024-09-13T11:23:41.663 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:41.663 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:41.663 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:41.663 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:23:41.663 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:35.477154+0000", 2024-09-13T11:23:41.663 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:41.663 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:41.663 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:41.663 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:23:41.664 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:35.477155+0000", 2024-09-13T11:23:41.664 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.5710000000000001e-06 2024-09-13T11:23:41.664 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:41.664 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:41.664 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:23:41.664 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:35.477202+0000", 2024-09-13T11:23:41.664 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.6489999999999997e-05 2024-09-13T11:23:41.664 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:41.664 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:41.664 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:23:41.664 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:35.592075+0000", 2024-09-13T11:23:41.664 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.11487309599999999 2024-09-13T11:23:41.664 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:41.664 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:41.664 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:23:41.665 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:35.592081+0000", 2024-09-13T11:23:41.665 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.4239999999999998e-06 2024-09-13T11:23:41.665 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:41.665 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:41.665 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:41.665 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:35.592154+0000", 2024-09-13T11:23:41.665 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.2502000000000005e-05 2024-09-13T11:23:41.665 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:41.665 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:41.665 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:23:41.665 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:35.592179+0000", 2024-09-13T11:23:41.665 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.5650999999999999e-05 2024-09-13T11:23:41.665 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:41.666 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:41.666 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:23:41.666 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:35.594087+0000", 2024-09-13T11:23:41.666 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.001908098 2024-09-13T11:23:41.666 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:41.666 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:41.666 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:23:41.666 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:35.594128+0000", 2024-09-13T11:23:41.666 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.0738999999999997e-05 2024-09-13T11:23:41.666 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:41.666 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:41.666 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:41.666 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:41.667 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:41.667 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4665.0:55504 3.21 e79/78 3:8412c79a:::benchmark_data_smithi177_31116_object55503:head v 79'3084, mlcod=0'0)", 2024-09-13T11:23:41.667 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:35.591951+0000", 2024-09-13T11:23:41.667 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 6.0591874289999996, 2024-09-13T11:23:41.667 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.118668976, 2024-09-13T11:23:41.667 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:23:41.667 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:23:41.667 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:41.667 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:23:41.667 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:41.667 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:23:41.667 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:35.591951+0000", 2024-09-13T11:23:41.667 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:41.668 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:41.668 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:41.668 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:23:41.668 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:35.591951+0000", 2024-09-13T11:23:41.668 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:41.668 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:41.668 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:41.668 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:23:41.668 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:35.591952+0000", 2024-09-13T11:23:41.668 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.296e-06 2024-09-13T11:23:41.668 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:41.668 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:41.668 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:23:41.668 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:35.592060+0000", 2024-09-13T11:23:41.669 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000108218 2024-09-13T11:23:41.669 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:41.669 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:41.669 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:23:41.669 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:35.709102+0000", 2024-09-13T11:23:41.669 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.117041834 2024-09-13T11:23:41.669 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:41.669 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:41.669 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:23:41.669 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:35.709109+0000", 2024-09-13T11:23:41.669 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.7689999999999999e-06 2024-09-13T11:23:41.669 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:41.669 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:41.670 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:41.670 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:35.709171+0000", 2024-09-13T11:23:41.670 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.2130000000000003e-05 2024-09-13T11:23:41.670 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:41.670 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:41.670 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:23:41.670 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:35.709198+0000", 2024-09-13T11:23:41.670 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.6852e-05 2024-09-13T11:23:41.670 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:41.670 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:41.670 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:23:41.670 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:35.710569+0000", 2024-09-13T11:23:41.670 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0013707909999999999 2024-09-13T11:23:41.670 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:41.671 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:41.671 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:23:41.671 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:35.710620+0000", 2024-09-13T11:23:41.671 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.1085999999999998e-05 2024-09-13T11:23:41.671 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:41.671 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:41.671 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:41.671 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:41.671 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:41.671 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4665.0:55661 3.21 e79/78 3:8450c44f:::benchmark_data_smithi177_31116_object55660:head v 79'3086, mlcod=0'0)", 2024-09-13T11:23:41.671 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:35.752238+0000", 2024-09-13T11:23:41.671 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 5.8989003929999999, 2024-09-13T11:23:41.671 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.082686490000000001, 2024-09-13T11:23:41.671 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:23:41.672 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:23:41.672 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:41.672 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:23:41.672 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:41.672 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:23:41.672 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:35.752238+0000", 2024-09-13T11:23:41.672 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:41.672 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:41.672 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:41.672 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:23:41.673 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:35.752238+0000", 2024-09-13T11:23:41.673 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:41.673 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:41.673 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:41.673 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:23:41.673 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:35.752239+0000", 2024-09-13T11:23:41.673 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.5e-06 2024-09-13T11:23:41.673 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:41.673 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:41.673 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:23:41.673 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:35.752308+0000", 2024-09-13T11:23:41.673 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.8956999999999997e-05 2024-09-13T11:23:41.673 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:41.674 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:41.674 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:23:41.674 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:35.833314+0000", 2024-09-13T11:23:41.674 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.081005670000000002 2024-09-13T11:23:41.674 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:41.674 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:41.674 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:23:41.674 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:35.833321+0000", 2024-09-13T11:23:41.674 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.8379999999999996e-06 2024-09-13T11:23:41.674 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:41.674 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:41.674 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:41.674 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:35.833395+0000", 2024-09-13T11:23:41.674 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.3998999999999994e-05 2024-09-13T11:23:41.675 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:41.675 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:41.675 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:23:41.675 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:35.833422+0000", 2024-09-13T11:23:41.675 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.7267000000000002e-05 2024-09-13T11:23:41.675 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:41.675 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:41.675 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:23:41.675 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:35.834883+0000", 2024-09-13T11:23:41.675 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0014614999999999999 2024-09-13T11:23:41.675 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:41.675 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:41.675 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:23:41.675 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:35.834924+0000", 2024-09-13T11:23:41.676 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.0759000000000001e-05 2024-09-13T11:23:41.676 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:41.676 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:41.676 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:41.676 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:41.676 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:41.676 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:56086 3.1e 3:7dc07268:::benchmark_data_smithi177_31116_object56085:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e79)", 2024-09-13T11:23:41.676 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:36.214326+0000", 2024-09-13T11:23:41.676 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 5.4368115860000001, 2024-09-13T11:23:41.676 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.082486591999999997, 2024-09-13T11:23:41.676 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:23:41.676 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:23:41.676 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:41.677 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:23:41.677 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:23:41.677 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:41.677 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 56086 2024-09-13T11:23:41.677 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:41.677 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:23:41.677 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:41.677 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:23:41.677 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:36.214326+0000", 2024-09-13T11:23:41.677 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:41.677 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:41.677 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:41.677 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:23:41.677 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:36.214326+0000", 2024-09-13T11:23:41.678 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:41.678 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:41.678 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:41.678 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:23:41.678 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:36.214329+0000", 2024-09-13T11:23:41.678 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.17e-06 2024-09-13T11:23:41.678 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:41.678 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:41.678 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:23:41.678 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:36.214371+0000", 2024-09-13T11:23:41.678 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.2577999999999998e-05 2024-09-13T11:23:41.678 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:41.678 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:41.679 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:23:41.679 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:36.214385+0000", 2024-09-13T11:23:41.679 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.4042e-05 2024-09-13T11:23:41.679 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:41.679 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:41.679 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:23:41.679 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:36.214393+0000", 2024-09-13T11:23:41.679 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.0520000000000002e-06 2024-09-13T11:23:41.679 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:41.679 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:41.679 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:41.679 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:36.214447+0000", 2024-09-13T11:23:41.679 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.3465999999999999e-05 2024-09-13T11:23:41.679 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:41.679 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:41.680 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:23:41.680 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:36.214603+0000", 2024-09-13T11:23:41.680 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00015607800000000001 2024-09-13T11:23:41.680 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:41.680 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:41.680 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 4,5", 2024-09-13T11:23:41.680 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:36.214763+0000", 2024-09-13T11:23:41.680 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00015975200000000001 2024-09-13T11:23:41.680 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:41.680 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:41.680 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:23:41.680 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:36.216041+0000", 2024-09-13T11:23:41.680 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0012783639999999999 2024-09-13T11:23:41.681 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:41.681 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:41.681 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:41.681 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:36.228133+0000", 2024-09-13T11:23:41.681 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.012091737999999999 2024-09-13T11:23:41.681 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:41.681 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:41.681 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:41.681 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:36.296386+0000", 2024-09-13T11:23:41.681 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.068253674 2024-09-13T11:23:41.681 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:41.681 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:41.681 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:23:41.682 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:36.296424+0000", 2024-09-13T11:23:41.682 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.7481999999999998e-05 2024-09-13T11:23:41.682 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:41.682 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:41.682 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:23:41.682 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:36.296813+0000", 2024-09-13T11:23:41.682 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00038919600000000002 2024-09-13T11:23:41.682 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:41.682 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:41.682 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:41.682 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:41.682 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:41.682 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDPGPushReply(3.1e 79/78 [PushReplyOp(3:78b61fd3:::benchmark_data_smithi177_31116_object2494:head)])", 2024-09-13T11:23:41.683 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:36.228411+0000", 2024-09-13T11:23:41.683 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 5.4227271879999996, 2024-09-13T11:23:41.683 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.13316029400000001, 2024-09-13T11:23:41.683 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:23:41.683 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:23:41.683 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-13T11:23:41.683 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:23:41.683 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:41.683 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:23:41.683 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:36.228411+0000", 2024-09-13T11:23:41.683 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:41.683 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:41.683 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:41.683 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:23:41.684 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:36.228411+0000", 2024-09-13T11:23:41.684 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:41.684 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:41.684 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:41.684 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:23:41.684 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:36.228412+0000", 2024-09-13T11:23:41.684 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.071e-06 2024-09-13T11:23:41.684 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:41.684 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:41.684 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:23:41.684 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:36.228428+0000", 2024-09-13T11:23:41.684 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.6055000000000001e-05 2024-09-13T11:23:41.684 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:41.685 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:41.685 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:23:41.685 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:36.360051+0000", 2024-09-13T11:23:41.685 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.131622989 2024-09-13T11:23:41.685 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:41.685 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:41.685 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:23:41.685 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:36.360059+0000", 2024-09-13T11:23:41.685 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.729e-06 2024-09-13T11:23:41.685 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:41.685 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:41.685 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:41.685 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:36.360124+0000", 2024-09-13T11:23:41.686 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.5662999999999998e-05 2024-09-13T11:23:41.686 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:41.686 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:41.686 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:23:41.686 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:36.361571+0000", 2024-09-13T11:23:41.686 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0014467869999999999 2024-09-13T11:23:41.686 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:41.686 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:41.686 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:41.686 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:41.686 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:41.686 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:56237 3.1e 3:7ba56f65:::benchmark_data_smithi177_31116_object56236:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e79)", 2024-09-13T11:23:41.686 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:36.400007+0000", 2024-09-13T11:23:41.687 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 5.2511307169999997, 2024-09-13T11:23:41.687 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.080442219999999995, 2024-09-13T11:23:41.687 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:23:41.687 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:23:41.687 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:41.687 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:23:41.687 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:23:41.687 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:41.687 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 56237 2024-09-13T11:23:41.687 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:41.687 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:23:41.687 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:41.687 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:23:41.687 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:36.400007+0000", 2024-09-13T11:23:41.688 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:41.688 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:41.688 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:41.688 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:23:41.688 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:36.400007+0000", 2024-09-13T11:23:41.688 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:41.688 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:41.688 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:41.688 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:23:41.688 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:36.400009+0000", 2024-09-13T11:23:41.688 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.7749999999999999e-06 2024-09-13T11:23:41.688 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:41.688 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:41.688 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:23:41.689 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:36.400102+0000", 2024-09-13T11:23:41.689 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.2536999999999998e-05 2024-09-13T11:23:41.689 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:41.689 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:41.689 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:23:41.689 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:36.400112+0000", 2024-09-13T11:23:41.689 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0543999999999999e-05 2024-09-13T11:23:41.689 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:41.689 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:41.689 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:23:41.689 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:36.400122+0000", 2024-09-13T11:23:41.689 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0029e-05 2024-09-13T11:23:41.689 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:41.690 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:41.690 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:41.690 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:36.400172+0000", 2024-09-13T11:23:41.690 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.9716999999999998e-05 2024-09-13T11:23:41.690 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:41.690 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:41.690 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:23:41.690 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:36.400298+0000", 2024-09-13T11:23:41.690 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00012615099999999999 2024-09-13T11:23:41.690 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:41.690 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:41.690 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 4,5", 2024-09-13T11:23:41.690 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:36.400452+0000", 2024-09-13T11:23:41.690 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000153406 2024-09-13T11:23:41.691 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:41.691 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:41.691 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:23:41.691 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:36.401546+0000", 2024-09-13T11:23:41.691 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00109493 2024-09-13T11:23:41.691 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:41.691 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:41.691 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:41.691 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:36.479005+0000", 2024-09-13T11:23:41.691 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.077458250000000006 2024-09-13T11:23:41.691 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:41.691 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:41.691 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:41.691 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:36.479705+0000", 2024-09-13T11:23:41.692 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00070010199999999997 2024-09-13T11:23:41.692 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:41.692 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:41.692 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:23:41.692 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:36.479746+0000", 2024-09-13T11:23:41.692 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.1542999999999998e-05 2024-09-13T11:23:41.692 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:41.692 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:41.692 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:23:41.692 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:36.480450+0000", 2024-09-13T11:23:41.692 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00070323599999999999 2024-09-13T11:23:41.692 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:41.692 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:41.692 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:41.693 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:41.693 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:41.693 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDPGPushReply(3.1e 79/78 [PushReplyOp(3:7a3a85df:::benchmark_data_smithi177_31116_object2607:head)])", 2024-09-13T11:23:41.693 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:36.409945+0000", 2024-09-13T11:23:41.693 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 5.2411932620000004, 2024-09-13T11:23:41.693 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.115330801, 2024-09-13T11:23:41.693 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:23:41.693 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:23:41.693 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-13T11:23:41.693 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:23:41.693 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:41.693 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:23:41.693 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:36.409945+0000", 2024-09-13T11:23:41.693 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:41.694 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:41.694 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:41.694 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:23:41.694 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:36.409945+0000", 2024-09-13T11:23:41.694 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:41.694 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:41.694 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:41.694 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:23:41.694 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:36.409946+0000", 2024-09-13T11:23:41.694 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2580000000000001e-06 2024-09-13T11:23:41.694 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:41.694 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:41.694 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:23:41.694 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:36.409951+0000", 2024-09-13T11:23:41.695 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.9559999999999996e-06 2024-09-13T11:23:41.695 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:41.695 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:41.695 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:23:41.695 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:36.524389+0000", 2024-09-13T11:23:41.695 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.11443774299999999 2024-09-13T11:23:41.695 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:41.695 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:41.695 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:23:41.695 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:36.524395+0000", 2024-09-13T11:23:41.695 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.5069999999999997e-06 2024-09-13T11:23:41.695 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:41.696 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:41.696 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:41.696 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:36.524456+0000", 2024-09-13T11:23:41.696 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.0739999999999998e-05 2024-09-13T11:23:41.696 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:41.696 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:41.696 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:23:41.696 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:36.525276+0000", 2024-09-13T11:23:41.696 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00081959700000000003 2024-09-13T11:23:41.697 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:41.697 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:41.697 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:41.697 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:41.697 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:41.697 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDPGPushReply(3.1e 79/78 [PushReplyOp(3:7a3a85df:::benchmark_data_smithi177_31116_object2607:head)])", 2024-09-13T11:23:41.697 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:36.410722+0000", 2024-09-13T11:23:41.697 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 5.2404158709999997, 2024-09-13T11:23:41.697 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.15957022300000001, 2024-09-13T11:23:41.697 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:23:41.697 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:23:41.697 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-13T11:23:41.697 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:23:41.698 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:41.698 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:23:41.698 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:36.410722+0000", 2024-09-13T11:23:41.698 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:41.698 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:41.698 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:41.698 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:23:41.698 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:36.410722+0000", 2024-09-13T11:23:41.698 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:41.698 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:41.698 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:41.698 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:23:41.698 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:36.410724+0000", 2024-09-13T11:23:41.698 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.367e-06 2024-09-13T11:23:41.699 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:41.699 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:41.699 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:23:41.699 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:36.410729+0000", 2024-09-13T11:23:41.699 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.4870000000000002e-06 2024-09-13T11:23:41.699 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:41.699 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:41.699 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:23:41.699 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:36.568151+0000", 2024-09-13T11:23:41.699 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.15742154999999999 2024-09-13T11:23:41.699 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:41.699 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:41.699 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:23:41.699 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:36.568162+0000", 2024-09-13T11:23:41.700 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.117e-05 2024-09-13T11:23:41.700 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:41.700 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:41.700 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:41.700 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:36.568214+0000", 2024-09-13T11:23:41.700 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.2187e-05 2024-09-13T11:23:41.700 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:41.700 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:41.700 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:23:41.700 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:36.570292+0000", 2024-09-13T11:23:41.700 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0020784620000000001 2024-09-13T11:23:41.700 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:41.700 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:41.701 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:41.701 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:41.701 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:41.701 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:56305 3.1e 3:7fefa870:::benchmark_data_smithi177_31116_object56304:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e79)", 2024-09-13T11:23:41.701 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:36.476352+0000", 2024-09-13T11:23:41.701 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 5.1747860059999997, 2024-09-13T11:23:41.701 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.094779549000000005, 2024-09-13T11:23:41.701 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:23:41.701 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:23:41.701 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:41.701 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:23:41.701 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:23:41.701 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:41.702 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 56305 2024-09-13T11:23:41.702 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:41.702 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:23:41.702 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:41.702 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:23:41.702 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:36.476352+0000", 2024-09-13T11:23:41.702 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:41.702 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:41.702 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:41.702 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:23:41.702 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:36.476352+0000", 2024-09-13T11:23:41.702 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:41.702 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:41.702 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:41.703 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:23:41.703 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:36.476354+0000", 2024-09-13T11:23:41.703 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.7999999999999999e-06 2024-09-13T11:23:41.703 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:41.703 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:41.703 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:23:41.703 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:36.476401+0000", 2024-09-13T11:23:41.703 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.7215000000000001e-05 2024-09-13T11:23:41.703 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:41.703 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:41.703 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:23:41.703 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:36.476417+0000", 2024-09-13T11:23:41.703 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.5590000000000002e-05 2024-09-13T11:23:41.703 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:41.704 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:41.704 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:23:41.704 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:36.476426+0000", 2024-09-13T11:23:41.704 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.0410000000000003e-06 2024-09-13T11:23:41.704 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:41.704 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:41.704 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:41.704 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:36.476487+0000", 2024-09-13T11:23:41.704 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.0862000000000001e-05 2024-09-13T11:23:41.704 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:41.704 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:41.704 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:23:41.704 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:36.476646+0000", 2024-09-13T11:23:41.704 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00015895800000000001 2024-09-13T11:23:41.705 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:41.705 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:41.705 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 4,5", 2024-09-13T11:23:41.705 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:36.476815+0000", 2024-09-13T11:23:41.705 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00016975700000000001 2024-09-13T11:23:41.705 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:41.705 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:41.705 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:23:41.705 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:36.478015+0000", 2024-09-13T11:23:41.705 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.001199486 2024-09-13T11:23:41.705 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:41.705 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:41.705 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:41.705 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:36.544475+0000", 2024-09-13T11:23:41.706 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.066460068999999997 2024-09-13T11:23:41.706 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:41.706 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:41.706 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:41.706 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:36.570362+0000", 2024-09-13T11:23:41.706 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.025887107999999999 2024-09-13T11:23:41.706 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:41.706 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:41.706 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:23:41.706 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:36.570440+0000", 2024-09-13T11:23:41.706 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.7979999999999995e-05 2024-09-13T11:23:41.706 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:41.706 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:41.706 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:23:41.706 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:36.571132+0000", 2024-09-13T11:23:41.707 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00069168300000000001 2024-09-13T11:23:41.707 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:41.707 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:41.707 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:41.707 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:41.707 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:41.707 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop_reply(client.4665.0:56305 3.1e e79/78 ondisk, result = 0)", 2024-09-13T11:23:41.707 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:36.479586+0000", 2024-09-13T11:23:41.707 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 5.1715519130000001, 2024-09-13T11:23:41.707 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.091556449999999998, 2024-09-13T11:23:41.707 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:23:41.707 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:23:41.707 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:23:41.707 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:23:41.708 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:41.708 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:23:41.708 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:36.479586+0000", 2024-09-13T11:23:41.708 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:41.708 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:41.708 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:41.708 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:23:41.708 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:36.479586+0000", 2024-09-13T11:23:41.708 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:41.708 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:41.708 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:41.708 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:23:41.708 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:36.479589+0000", 2024-09-13T11:23:41.708 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.3690000000000001e-06 2024-09-13T11:23:41.709 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:41.709 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:41.709 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:23:41.709 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:36.479594+0000", 2024-09-13T11:23:41.709 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.7679999999999997e-06 2024-09-13T11:23:41.709 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:41.709 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:41.709 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:23:41.709 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:36.569314+0000", 2024-09-13T11:23:41.709 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.089719861999999997 2024-09-13T11:23:41.709 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:41.709 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:41.709 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:23:41.710 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:36.569322+0000", 2024-09-13T11:23:41.710 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.2220000000000002e-06 2024-09-13T11:23:41.710 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:41.710 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:41.710 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:41.710 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:36.570343+0000", 2024-09-13T11:23:41.710 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0010204050000000001 2024-09-13T11:23:41.710 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:41.710 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:41.710 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:23:41.710 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:36.570354+0000", 2024-09-13T11:23:41.710 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.1005999999999999e-05 2024-09-13T11:23:41.710 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:41.710 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:41.711 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:23:41.711 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:36.571143+0000", 2024-09-13T11:23:41.711 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00078881800000000005 2024-09-13T11:23:41.711 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:41.711 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:41.711 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:41.711 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:41.711 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:41.711 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:56461 3.1e 3:7d9fed89:::benchmark_data_smithi177_31116_object56460:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e79)", 2024-09-13T11:23:41.711 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:36.623197+0000", 2024-09-13T11:23:41.711 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 5.0279414669999998, 2024-09-13T11:23:41.711 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.113432371, 2024-09-13T11:23:41.711 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:23:41.711 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:23:41.712 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:41.712 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:23:41.712 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:23:41.712 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:41.712 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 56461 2024-09-13T11:23:41.712 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:41.712 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:23:41.712 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:41.712 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:23:41.712 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:36.623197+0000", 2024-09-13T11:23:41.712 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:41.712 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:41.712 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:41.712 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:23:41.713 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:36.623197+0000", 2024-09-13T11:23:41.713 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:41.713 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:41.713 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:41.713 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:23:41.713 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:36.623199+0000", 2024-09-13T11:23:41.713 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.2919999999999998e-06 2024-09-13T11:23:41.713 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:41.713 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:41.713 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:23:41.713 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:36.623256+0000", 2024-09-13T11:23:41.713 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.6857000000000001e-05 2024-09-13T11:23:41.713 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:41.713 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:41.714 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:23:41.714 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:36.623267+0000", 2024-09-13T11:23:41.714 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.1231e-05 2024-09-13T11:23:41.714 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:41.714 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:41.714 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:23:41.714 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:36.623274+0000", 2024-09-13T11:23:41.714 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.2790000000000001e-06 2024-09-13T11:23:41.714 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:41.714 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:41.714 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:41.714 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:36.626059+0000", 2024-09-13T11:23:41.714 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0027848339999999999 2024-09-13T11:23:41.714 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:41.715 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:41.715 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:23:41.715 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:36.626218+0000", 2024-09-13T11:23:41.715 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000159316 2024-09-13T11:23:41.715 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:41.715 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:41.715 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 4,5", 2024-09-13T11:23:41.715 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:36.626389+0000", 2024-09-13T11:23:41.715 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000170328 2024-09-13T11:23:41.715 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:41.715 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:41.715 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:23:41.715 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:36.628371+0000", 2024-09-13T11:23:41.715 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0019818829999999998 2024-09-13T11:23:41.716 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:41.716 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:41.716 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:41.716 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:36.732757+0000", 2024-09-13T11:23:41.716 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.10438676700000001 2024-09-13T11:23:41.716 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:41.716 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:41.716 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:41.716 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:36.735680+0000", 2024-09-13T11:23:41.716 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0029226429999999999 2024-09-13T11:23:41.716 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:41.716 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:41.716 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:23:41.716 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:36.735757+0000", 2024-09-13T11:23:41.717 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.6531000000000003e-05 2024-09-13T11:23:41.717 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:41.717 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:41.717 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:23:41.717 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:36.736629+0000", 2024-09-13T11:23:41.717 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00087241 2024-09-13T11:23:41.717 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:41.717 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:41.717 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:41.717 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:41.717 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:41.717 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDPGPushReply(3.1e 79/78 [PushReplyOp(3:7da6b219:::benchmark_data_smithi177_31116_object2705:head)])", 2024-09-13T11:23:41.717 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:36.626701+0000", 2024-09-13T11:23:41.717 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 5.0244371149999996, 2024-09-13T11:23:41.718 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.099053025000000003, 2024-09-13T11:23:41.718 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:23:41.718 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:23:41.718 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-13T11:23:41.718 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:23:41.718 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:41.718 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:23:41.718 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:36.626701+0000", 2024-09-13T11:23:41.718 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:41.718 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:41.718 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:41.718 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:23:41.718 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:36.626701+0000", 2024-09-13T11:23:41.718 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:41.719 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:41.719 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:41.719 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:23:41.719 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:36.626702+0000", 2024-09-13T11:23:41.719 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.4479999999999999e-06 2024-09-13T11:23:41.719 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:41.719 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:41.719 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:23:41.719 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:36.626708+0000", 2024-09-13T11:23:41.719 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.6810000000000003e-06 2024-09-13T11:23:41.719 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:41.719 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:41.719 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:23:41.719 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:36.724501+0000", 2024-09-13T11:23:41.719 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.097793396000000005 2024-09-13T11:23:41.720 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:41.720 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:41.720 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:23:41.720 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:36.724507+0000", 2024-09-13T11:23:41.720 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.536e-06 2024-09-13T11:23:41.720 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:41.720 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:41.720 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:41.720 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:36.724566+0000", 2024-09-13T11:23:41.720 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.8941999999999999e-05 2024-09-13T11:23:41.720 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:41.720 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:41.720 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:23:41.720 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:36.725754+0000", 2024-09-13T11:23:41.721 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0011880219999999999 2024-09-13T11:23:41.721 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:41.721 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:41.721 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:41.721 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:41.721 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:41.721 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop_reply(client.4665.0:56461 3.1e e79/78 ondisk, result = 0)", 2024-09-13T11:23:41.721 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:36.628759+0000", 2024-09-13T11:23:41.721 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 5.0223791179999999, 2024-09-13T11:23:41.721 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.107879138, 2024-09-13T11:23:41.721 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:23:41.721 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:23:41.721 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:23:41.721 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:23:41.722 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:41.722 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:23:41.722 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:36.628759+0000", 2024-09-13T11:23:41.722 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:41.722 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:41.722 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:41.722 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:23:41.722 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:36.628759+0000", 2024-09-13T11:23:41.722 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:41.722 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:41.722 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:41.722 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:23:41.722 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:36.628760+0000", 2024-09-13T11:23:41.722 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.3430000000000001e-06 2024-09-13T11:23:41.723 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:41.723 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:41.723 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:23:41.723 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:36.628765+0000", 2024-09-13T11:23:41.723 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.9579999999999998e-06 2024-09-13T11:23:41.723 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:41.723 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:41.723 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:23:41.723 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:36.735600+0000", 2024-09-13T11:23:41.723 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.106834922 2024-09-13T11:23:41.723 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:41.723 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:41.723 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:23:41.724 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:36.735606+0000", 2024-09-13T11:23:41.724 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.7100000000000004e-06 2024-09-13T11:23:41.724 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:41.724 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:41.724 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:41.724 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:36.735660+0000", 2024-09-13T11:23:41.724 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.3891000000000003e-05 2024-09-13T11:23:41.724 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:41.724 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:41.724 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:23:41.724 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:36.735671+0000", 2024-09-13T11:23:41.724 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.1382999999999999e-05 2024-09-13T11:23:41.724 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:41.724 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:41.725 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:23:41.725 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:36.736638+0000", 2024-09-13T11:23:41.725 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00096693099999999998 2024-09-13T11:23:41.725 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:41.725 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:41.725 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:41.725 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:41.725 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:41.725 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop_reply(client.4665.0:56461 3.1e e79/78 ondisk, result = 0)", 2024-09-13T11:23:41.725 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:36.629098+0000", 2024-09-13T11:23:41.725 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 5.0220404209999998, 2024-09-13T11:23:41.725 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.103664887, 2024-09-13T11:23:41.726 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:23:41.726 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:23:41.726 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:23:41.726 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:23:41.726 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:41.726 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:23:41.726 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:36.629098+0000", 2024-09-13T11:23:41.726 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:41.726 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:41.726 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:41.726 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:23:41.726 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:36.629098+0000", 2024-09-13T11:23:41.726 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:41.727 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:41.727 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:41.727 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:23:41.727 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:36.629099+0000", 2024-09-13T11:23:41.727 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.7400000000000002e-07 2024-09-13T11:23:41.727 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:41.727 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:41.727 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:23:41.727 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:36.629102+0000", 2024-09-13T11:23:41.727 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.1609999999999999e-06 2024-09-13T11:23:41.727 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:41.727 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:41.727 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:23:41.728 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:36.732693+0000", 2024-09-13T11:23:41.728 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.10359096800000001 2024-09-13T11:23:41.728 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:41.728 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:41.728 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:23:41.728 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:36.732698+0000", 2024-09-13T11:23:41.728 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.074e-06 2024-09-13T11:23:41.728 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:41.728 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:41.728 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:41.728 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:36.732747+0000", 2024-09-13T11:23:41.728 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.8789999999999999e-05 2024-09-13T11:23:41.728 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:41.729 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:41.729 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:23:41.729 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:36.732753+0000", 2024-09-13T11:23:41.729 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.5579999999999997e-06 2024-09-13T11:23:41.729 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:41.729 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:41.729 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:23:41.729 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:36.732763+0000", 2024-09-13T11:23:41.729 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.4620000000000002e-06 2024-09-13T11:23:41.729 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:41.729 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:41.729 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:41.730 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:41.730 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:41.730 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDPGPushReply(3.1e 79/78 [PushReplyOp(3:7fe28141:::benchmark_data_smithi177_31116_object2757:head)])", 2024-09-13T11:23:41.730 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:36.735579+0000", 2024-09-13T11:23:41.730 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 4.9155592229999998, 2024-09-13T11:23:41.730 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.087382827999999996, 2024-09-13T11:23:41.730 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:23:41.730 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:23:41.730 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-13T11:23:41.730 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:23:41.730 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:41.730 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:23:41.730 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:36.735579+0000", 2024-09-13T11:23:41.730 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:41.731 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:41.731 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:41.731 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:23:41.731 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:36.735579+0000", 2024-09-13T11:23:41.731 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:41.731 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:41.731 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:41.731 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:23:41.731 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:36.735580+0000", 2024-09-13T11:23:41.731 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.204e-06 2024-09-13T11:23:41.731 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:41.731 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:41.731 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:23:41.731 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:36.735587+0000", 2024-09-13T11:23:41.732 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.8410000000000002e-06 2024-09-13T11:23:41.732 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:41.732 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:41.732 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:23:41.732 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:36.821376+0000", 2024-09-13T11:23:41.732 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.085789283999999993 2024-09-13T11:23:41.732 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:41.732 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:41.732 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:23:41.732 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:36.821385+0000", 2024-09-13T11:23:41.732 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.5140000000000001e-06 2024-09-13T11:23:41.732 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:41.732 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:41.732 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:41.733 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:36.821468+0000", 2024-09-13T11:23:41.733 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.2913999999999999e-05 2024-09-13T11:23:41.733 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:41.733 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:41.733 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:23:41.733 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:36.822962+0000", 2024-09-13T11:23:41.733 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.001494071 2024-09-13T11:23:41.733 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:41.733 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:41.733 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:41.733 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:41.733 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:41.733 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDPGPush(3.15 79/70 [PushOp(3:af23dbef:::benchmark_data_smithi177_31116_object35019:head, version: 61'2146, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:af23dbef:::benchmark_data_smithi177_31116_object35019:head@61'2146, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:23:41.734 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:36.740285+0000", 2024-09-13T11:23:41.734 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 4.910853296, 2024-09-13T11:23:41.734 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.090955465999999999, 2024-09-13T11:23:41.734 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:23:41.734 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:23:41.734 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:23:41.734 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:23:41.734 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:41.734 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:23:41.734 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:36.740285+0000", 2024-09-13T11:23:41.734 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:41.734 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:41.734 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:41.734 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:23:41.735 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:36.740285+0000", 2024-09-13T11:23:41.735 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:41.735 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:41.735 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:41.735 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:23:41.735 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:36.740286+0000", 2024-09-13T11:23:41.735 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2580000000000001e-06 2024-09-13T11:23:41.735 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:41.735 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:41.735 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:23:41.735 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:36.740416+0000", 2024-09-13T11:23:41.735 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000130211 2024-09-13T11:23:41.735 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:41.735 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:41.735 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:23:41.736 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:36.830509+0000", 2024-09-13T11:23:41.736 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.090092285999999994 2024-09-13T11:23:41.736 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:41.736 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:41.736 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:23:41.736 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:36.830528+0000", 2024-09-13T11:23:41.736 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.9417000000000001e-05 2024-09-13T11:23:41.736 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:41.736 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:41.736 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:41.736 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:36.830616+0000", 2024-09-13T11:23:41.736 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.7866000000000002e-05 2024-09-13T11:23:41.736 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:41.736 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:41.737 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:23:41.737 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:36.830627+0000", 2024-09-13T11:23:41.737 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.1659e-05 2024-09-13T11:23:41.737 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:41.737 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:41.737 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:23:41.737 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:36.831240+0000", 2024-09-13T11:23:41.737 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00061276899999999999 2024-09-13T11:23:41.737 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:41.737 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:41.737 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:41.737 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:41.737 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:41.737 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDPGPush(3.15 79/70 [PushOp(3:ae730848:::benchmark_data_smithi177_31116_object35058:head, version: 62'2147, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:ae730848:::benchmark_data_smithi177_31116_object35058:head@62'2147, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:23:41.738 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:36.834397+0000", 2024-09-13T11:23:41.738 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 4.8167411329999998, 2024-09-13T11:23:41.738 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.079624083999999998, 2024-09-13T11:23:41.738 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:23:41.738 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:23:41.738 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:23:41.738 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:23:41.738 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:41.738 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:23:41.738 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:36.834397+0000", 2024-09-13T11:23:41.738 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:41.738 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:41.738 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:41.738 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:23:41.739 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:36.834397+0000", 2024-09-13T11:23:41.739 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:41.739 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:41.739 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:41.739 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:23:41.739 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:36.834398+0000", 2024-09-13T11:23:41.739 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.412e-06 2024-09-13T11:23:41.739 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:41.739 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:41.739 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:23:41.739 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:36.834506+0000", 2024-09-13T11:23:41.739 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000107506 2024-09-13T11:23:41.739 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:41.739 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:41.740 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:23:41.740 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:36.912850+0000", 2024-09-13T11:23:41.740 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.078344618000000005 2024-09-13T11:23:41.740 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:41.740 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:41.740 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:23:41.740 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:36.912872+0000", 2024-09-13T11:23:41.740 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.1831000000000002e-05 2024-09-13T11:23:41.740 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:41.740 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:41.740 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:41.740 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:36.912971+0000", 2024-09-13T11:23:41.740 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.8929000000000001e-05 2024-09-13T11:23:41.740 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:41.741 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:41.741 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:23:41.741 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:36.912983+0000", 2024-09-13T11:23:41.741 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2106999999999999e-05 2024-09-13T11:23:41.741 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:41.741 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:41.741 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:23:41.741 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:36.914021+0000", 2024-09-13T11:23:41.741 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0010376809999999999 2024-09-13T11:23:41.741 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:41.741 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:41.741 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:41.741 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:41.741 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:41.741 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDPGPush(3.20 83/77 [PushOp(3:05148b46:::benchmark_data_smithi177_31116_object1401:head, version: 43'74, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:05148b46:::benchmark_data_smithi177_31116_object1401:head@43'74, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:23:41.742 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:40.912122+0000", 2024-09-13T11:23:41.742 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 0.73901604399999998, 2024-09-13T11:23:41.742 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.158959773, 2024-09-13T11:23:41.742 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:23:41.742 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:23:41.742 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:23:41.742 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:23:41.742 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:41.742 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:23:41.742 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:40.912122+0000", 2024-09-13T11:23:41.742 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:41.742 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:41.742 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:41.742 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:23:41.743 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:40.912122+0000", 2024-09-13T11:23:41.743 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:41.743 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:41.743 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:41.743 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:23:41.743 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:40.912123+0000", 2024-09-13T11:23:41.743 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0589999999999999e-06 2024-09-13T11:23:41.743 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:41.743 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:41.743 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:23:41.743 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:40.912176+0000", 2024-09-13T11:23:41.743 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.2431e-05 2024-09-13T11:23:41.743 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:41.743 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:41.744 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:23:41.744 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:41.070227+0000", 2024-09-13T11:23:41.744 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.15805137499999999 2024-09-13T11:23:41.744 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:41.744 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:41.744 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:23:41.744 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:41.070243+0000", 2024-09-13T11:23:41.744 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.5948999999999999e-05 2024-09-13T11:23:41.744 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:41.744 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:41.744 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:41.744 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:41.070360+0000", 2024-09-13T11:23:41.744 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000117267 2024-09-13T11:23:41.744 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:41.745 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:41.745 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:23:41.745 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:41.070373+0000", 2024-09-13T11:23:41.745 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2665e-05 2024-09-13T11:23:41.745 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:41.745 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:41.745 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:23:41.745 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:41.071082+0000", 2024-09-13T11:23:41.745 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00070902699999999999 2024-09-13T11:23:41.745 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:41.745 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:41.745 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:41.745 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:41.745 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:41.745 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-13T11:23:41.747 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-13T11:23:41.748 INFO:tasks.ceph.osd.10.smithi162.stderr:2024-09-13T11:23:41.741+0000 7fb59c9f0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26317) UID: 0 2024-09-13T11:23:41.848 INFO:tasks.ceph.osd.15.smithi177.stderr:2024-09-13T11:23:41.840+0000 7f9d6ce77640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26578) UID: 0 2024-09-13T11:23:41.913 INFO:tasks.ceph.osd.15.smithi177.stderr:2024-09-13T11:23:41.908+0000 7f9d618a4640 -1 osd.15 84 handle_osd_map injecting map crc failure 2024-09-13T11:23:41.916 INFO:teuthology.orchestra.run.smithi001.stderr:reweighted osd.1 to 1 (10000) 2024-09-13T11:23:41.922 INFO:tasks.ceph.osd.4.smithi001.stderr:2024-09-13T11:23:41.916+0000 7ff8d0c3d640 -1 osd.4 84 handle_osd_map injecting map crc failure 2024-09-13T11:23:41.930 INFO:tasks.ceph.osd.7.smithi177.stderr:2024-09-13T11:23:41.924+0000 7f896ff4a640 -1 osd.7 84 handle_osd_map injecting map crc failure 2024-09-13T11:23:41.943 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-13T11:23:41.943 INFO:teuthology.orchestra.run.smithi001.stdout: "ops": [], 2024-09-13T11:23:41.944 INFO:teuthology.orchestra.run.smithi001.stdout: "num_ops": 0 2024-09-13T11:23:41.944 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-13T11:23:41.950 INFO:tasks.ceph.osd.11.smithi177.stderr:2024-09-13T11:23:41.944+0000 7f24792c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26580) UID: 0 2024-09-13T11:23:41.958 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 2 1 2024-09-13T11:23:41.974 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-13T11:23:42.005 INFO:tasks.radosbench.radosbench.0.smithi177.stdout: 75 16 60257 60241 50.1955 10.8125 0.00619432 0.019915 2024-09-13T11:23:42.051 INFO:tasks.ceph.osd.3.smithi177.stderr:2024-09-13T11:23:42.044+0000 7f10adba4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26574) UID: 0 2024-09-13T11:23:42.151 INFO:tasks.ceph.osd.7.smithi177.stderr:2024-09-13T11:23:42.144+0000 7f897b51d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26575) UID: 0 2024-09-13T11:23:42.185 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-13T11:23:42.186 INFO:teuthology.orchestra.run.smithi001.stdout: "ops": [], 2024-09-13T11:23:42.186 INFO:teuthology.orchestra.run.smithi001.stdout: "complaint_time": 30, 2024-09-13T11:23:42.186 INFO:teuthology.orchestra.run.smithi001.stdout: "num_blocked_ops": 0 2024-09-13T11:23:42.186 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-13T11:23:42.204 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-13T11:23:42.252 INFO:tasks.ceph.osd.13.smithi092.stderr:2024-09-13T11:23:42.248+0000 7f3e5afea640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26350) UID: 0 2024-09-13T11:23:42.353 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-13T11:23:42.353 INFO:teuthology.orchestra.run.smithi001.stdout: "size": 20, 2024-09-13T11:23:42.354 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 600, 2024-09-13T11:23:42.354 INFO:teuthology.orchestra.run.smithi001.stdout: "ops": [ 2024-09-13T11:23:42.354 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.354 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4665.0:48326 3.11 e71/70 3:892c8b48:::benchmark_data_smithi177_31116_object48325:head v 71'2881, mlcod=44'195)", 2024-09-13T11:23:42.354 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:24.932419+0000", 2024-09-13T11:23:42.354 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 17.418446094, 2024-09-13T11:23:42.354 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.070889482000000004, 2024-09-13T11:23:42.354 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:23:42.354 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:23:42.354 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:42.355 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:23:42.355 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.355 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:23:42.355 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:24.932419+0000", 2024-09-13T11:23:42.355 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:42.355 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.355 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.355 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:23:42.355 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:24.932419+0000", 2024-09-13T11:23:42.356 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:42.356 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.356 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.356 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:23:42.356 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:24.932421+0000", 2024-09-13T11:23:42.356 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.4260000000000002e-06 2024-09-13T11:23:42.356 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.356 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.356 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:23:42.356 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:24.932492+0000", 2024-09-13T11:23:42.356 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.0553e-05 2024-09-13T11:23:42.356 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.356 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.357 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:23:42.357 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:25.001307+0000", 2024-09-13T11:23:42.357 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.068815692999999997 2024-09-13T11:23:42.357 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.357 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.357 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:23:42.357 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:25.001314+0000", 2024-09-13T11:23:42.357 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.319e-06 2024-09-13T11:23:42.357 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.357 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.357 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:42.357 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:25.001375+0000", 2024-09-13T11:23:42.358 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.1530000000000002e-05 2024-09-13T11:23:42.358 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.358 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.358 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:23:42.358 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:25.001406+0000", 2024-09-13T11:23:42.358 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.0559000000000003e-05 2024-09-13T11:23:42.358 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.358 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.358 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:23:42.358 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:25.003260+0000", 2024-09-13T11:23:42.359 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0018541720000000001 2024-09-13T11:23:42.359 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.359 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.359 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:23:42.359 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:25.003308+0000", 2024-09-13T11:23:42.359 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.8229999999999997e-05 2024-09-13T11:23:42.359 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:42.359 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:42.359 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:42.359 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.359 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.359 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDPGPush(3.11 72/70 [PushOp(3:8ae6b6fc:::benchmark_data_smithi177_31116_object7617:head, version: 46'455, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:8ae6b6fc:::benchmark_data_smithi177_31116_object7617:head@46'455, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:23:42.360 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:26.176296+0000", 2024-09-13T11:23:42.360 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 16.174568650000001, 2024-09-13T11:23:42.360 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.13349134900000001, 2024-09-13T11:23:42.360 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:23:42.360 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:23:42.360 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:23:42.360 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:23:42.360 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.360 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:23:42.360 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:26.176296+0000", 2024-09-13T11:23:42.360 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:42.361 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.361 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.361 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:23:42.361 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:26.176296+0000", 2024-09-13T11:23:42.361 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:42.361 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.361 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.361 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:23:42.361 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:26.176297+0000", 2024-09-13T11:23:42.361 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2440000000000001e-06 2024-09-13T11:23:42.362 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.362 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.362 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:23:42.362 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:26.176433+0000", 2024-09-13T11:23:42.362 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00013554999999999999 2024-09-13T11:23:42.362 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.362 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.362 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:23:42.362 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:26.308747+0000", 2024-09-13T11:23:42.362 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.132314392 2024-09-13T11:23:42.362 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.362 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.363 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:23:42.363 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:26.308768+0000", 2024-09-13T11:23:42.363 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.0758999999999999e-05 2024-09-13T11:23:42.363 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.363 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.363 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:42.363 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:26.308874+0000", 2024-09-13T11:23:42.363 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000105334 2024-09-13T11:23:42.363 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.363 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.363 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:23:42.363 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:26.308890+0000", 2024-09-13T11:23:42.363 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.6367000000000001e-05 2024-09-13T11:23:42.364 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.364 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.364 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:23:42.364 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:26.309788+0000", 2024-09-13T11:23:42.364 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00089770300000000002 2024-09-13T11:23:42.364 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:42.364 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:42.364 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:42.364 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.364 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.364 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDPGPush(3.11 72/70 [PushOp(3:8f62e9f7:::benchmark_data_smithi177_31116_object7652:head, version: 46'458, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:8f62e9f7:::benchmark_data_smithi177_31116_object7652:head@46'458, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:23:42.365 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:26.326138+0000", 2024-09-13T11:23:42.365 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 16.024727329000001, 2024-09-13T11:23:42.365 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.036052775000000002, 2024-09-13T11:23:42.365 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:23:42.365 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:23:42.365 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:23:42.365 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:23:42.365 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.365 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:23:42.365 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:26.326138+0000", 2024-09-13T11:23:42.366 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:42.366 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.366 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.366 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:23:42.366 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:26.326138+0000", 2024-09-13T11:23:42.366 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:42.366 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.366 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.366 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:23:42.366 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:26.326139+0000", 2024-09-13T11:23:42.366 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.5290000000000001e-06 2024-09-13T11:23:42.367 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.367 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.367 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:23:42.367 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:26.326194+0000", 2024-09-13T11:23:42.367 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.4833999999999997e-05 2024-09-13T11:23:42.367 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.367 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.367 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:23:42.367 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:26.361482+0000", 2024-09-13T11:23:42.367 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.035288370999999999 2024-09-13T11:23:42.367 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.367 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.368 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:23:42.368 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:26.361502+0000", 2024-09-13T11:23:42.368 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.9400999999999999e-05 2024-09-13T11:23:42.368 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.368 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.368 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:42.368 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:26.361595+0000", 2024-09-13T11:23:42.368 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.3535000000000005e-05 2024-09-13T11:23:42.368 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.368 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.368 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:23:42.368 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:26.361604+0000", 2024-09-13T11:23:42.369 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.4449999999999996e-06 2024-09-13T11:23:42.369 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.369 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.369 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:23:42.369 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:26.362190+0000", 2024-09-13T11:23:42.369 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00058666000000000002 2024-09-13T11:23:42.369 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:42.369 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:42.369 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:42.369 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.369 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.369 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDPGPush(3.11 72/70 [PushOp(3:8da9c977:::benchmark_data_smithi177_31116_object7736:head, version: 46'461, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:8da9c977:::benchmark_data_smithi177_31116_object7736:head@46'461, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:23:42.370 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:26.393216+0000", 2024-09-13T11:23:42.370 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 15.957648791, 2024-09-13T11:23:42.370 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.15683955499999999, 2024-09-13T11:23:42.370 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:23:42.370 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:23:42.370 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:23:42.370 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:23:42.370 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.370 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:23:42.370 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:26.393216+0000", 2024-09-13T11:23:42.370 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:42.370 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.370 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.371 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:23:42.371 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:26.393216+0000", 2024-09-13T11:23:42.371 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:42.371 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.371 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.371 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:23:42.371 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:26.393217+0000", 2024-09-13T11:23:42.371 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2410000000000001e-06 2024-09-13T11:23:42.371 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.371 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.371 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:23:42.371 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:26.393309+0000", 2024-09-13T11:23:42.372 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.2101000000000005e-05 2024-09-13T11:23:42.372 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.372 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.372 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:23:42.372 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:26.547936+0000", 2024-09-13T11:23:42.372 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.15462682699999999 2024-09-13T11:23:42.372 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.372 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.372 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:23:42.372 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:26.548021+0000", 2024-09-13T11:23:42.372 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.4529000000000003e-05 2024-09-13T11:23:42.372 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.373 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.373 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:42.373 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:26.548321+0000", 2024-09-13T11:23:42.373 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00029988399999999999 2024-09-13T11:23:42.373 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.373 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.373 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:23:42.373 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:26.548353+0000", 2024-09-13T11:23:42.373 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.2573000000000001e-05 2024-09-13T11:23:42.373 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.373 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.373 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:23:42.374 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:26.550056+0000", 2024-09-13T11:23:42.374 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0017024 2024-09-13T11:23:42.374 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:42.374 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:42.374 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:42.374 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.374 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.374 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDPGPush(3.11 72/70 [PushOp(3:88597a03:::benchmark_data_smithi177_31116_object7757:head, version: 46'463, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:88597a03:::benchmark_data_smithi177_31116_object7757:head@46'463, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:23:42.374 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:26.559694+0000", 2024-09-13T11:23:42.374 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 15.791170514999999, 2024-09-13T11:23:42.374 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.038634953, 2024-09-13T11:23:42.374 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:23:42.375 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:23:42.375 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:23:42.375 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:23:42.375 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.375 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:23:42.375 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:26.559694+0000", 2024-09-13T11:23:42.375 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:42.375 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.375 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.375 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:23:42.375 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:26.559694+0000", 2024-09-13T11:23:42.375 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:42.376 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.376 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.376 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:23:42.376 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:26.559698+0000", 2024-09-13T11:23:42.376 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.2059999999999999e-06 2024-09-13T11:23:42.376 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.376 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.376 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:23:42.376 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:26.559771+0000", 2024-09-13T11:23:42.376 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.3705e-05 2024-09-13T11:23:42.376 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.377 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.377 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:23:42.377 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:26.595040+0000", 2024-09-13T11:23:42.377 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.035268769999999998 2024-09-13T11:23:42.377 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.377 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.377 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:23:42.377 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:26.595096+0000", 2024-09-13T11:23:42.377 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.5933000000000001e-05 2024-09-13T11:23:42.377 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.378 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.378 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:42.378 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:26.595365+0000", 2024-09-13T11:23:42.378 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00026861399999999997 2024-09-13T11:23:42.378 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.378 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.378 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:23:42.378 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:26.595408+0000", 2024-09-13T11:23:42.378 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.3866999999999999e-05 2024-09-13T11:23:42.378 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.378 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.378 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:23:42.379 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:26.598329+0000", 2024-09-13T11:23:42.379 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0029208580000000001 2024-09-13T11:23:42.379 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:42.379 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:42.379 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:42.379 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.379 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.379 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4665.0:54052 3.11 e74/70 3:8c6be821:::benchmark_data_smithi177_31116_object54051:head v 74'3049, mlcod=55'1352)", 2024-09-13T11:23:42.379 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:30.314329+0000", 2024-09-13T11:23:42.379 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 12.036535397, 2024-09-13T11:23:42.379 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.100070243, 2024-09-13T11:23:42.379 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:23:42.379 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:23:42.380 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:42.380 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:23:42.380 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.380 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:23:42.380 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:30.314329+0000", 2024-09-13T11:23:42.380 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:42.380 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.380 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.380 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:23:42.380 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:30.314329+0000", 2024-09-13T11:23:42.380 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:42.380 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.380 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.380 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:23:42.381 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:30.314332+0000", 2024-09-13T11:23:42.381 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.1900000000000002e-06 2024-09-13T11:23:42.381 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.381 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.381 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:23:42.381 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:30.314372+0000", 2024-09-13T11:23:42.381 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.0788000000000003e-05 2024-09-13T11:23:42.381 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.381 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.381 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:23:42.381 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:30.412014+0000", 2024-09-13T11:23:42.381 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.097641172999999998 2024-09-13T11:23:42.381 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.381 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.382 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:23:42.382 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:30.412026+0000", 2024-09-13T11:23:42.382 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2551999999999999e-05 2024-09-13T11:23:42.382 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.382 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.382 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:42.382 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:30.412134+0000", 2024-09-13T11:23:42.382 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00010763400000000001 2024-09-13T11:23:42.382 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.382 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.382 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:23:42.382 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:30.412172+0000", 2024-09-13T11:23:42.382 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.8333e-05 2024-09-13T11:23:42.383 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.383 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.383 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:23:42.383 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:30.414340+0000", 2024-09-13T11:23:42.383 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0021680879999999999 2024-09-13T11:23:42.383 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.383 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.383 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:23:42.383 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:30.414400+0000", 2024-09-13T11:23:42.383 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.9484999999999997e-05 2024-09-13T11:23:42.383 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:42.383 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:42.384 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:42.384 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.384 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.384 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4665.0:54193 3.11 e74/70 3:8c8e9c38:::benchmark_data_smithi177_31116_object54192:head v 74'3051, mlcod=55'1358)", 2024-09-13T11:23:42.384 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:30.447991+0000", 2024-09-13T11:23:42.384 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 11.90287389, 2024-09-13T11:23:42.384 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.038008423, 2024-09-13T11:23:42.384 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:23:42.384 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:23:42.384 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:42.384 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:23:42.384 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.385 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:23:42.385 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:30.447991+0000", 2024-09-13T11:23:42.385 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:42.385 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.385 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.385 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:23:42.385 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:30.447991+0000", 2024-09-13T11:23:42.385 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:42.385 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.385 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.386 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:23:42.386 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:30.447994+0000", 2024-09-13T11:23:42.386 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.9799999999999998e-06 2024-09-13T11:23:42.388 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.388 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.388 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:23:42.388 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:30.448055+0000", 2024-09-13T11:23:42.388 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.0748000000000002e-05 2024-09-13T11:23:42.388 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.388 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.388 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:23:42.388 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:30.484156+0000", 2024-09-13T11:23:42.388 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.036101543999999999 2024-09-13T11:23:42.389 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.389 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.389 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:23:42.389 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:30.484167+0000", 2024-09-13T11:23:42.389 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0268000000000001e-05 2024-09-13T11:23:42.389 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.389 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.389 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:42.389 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:30.484243+0000", 2024-09-13T11:23:42.389 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.6532999999999994e-05 2024-09-13T11:23:42.389 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.389 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.390 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:23:42.390 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:30.484273+0000", 2024-09-13T11:23:42.390 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.0199999999999999e-05 2024-09-13T11:23:42.390 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.390 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.390 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:23:42.390 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:30.485943+0000", 2024-09-13T11:23:42.390 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0016701369999999999 2024-09-13T11:23:42.390 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.390 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.390 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:23:42.390 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:30.485999+0000", 2024-09-13T11:23:42.390 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.6013000000000001e-05 2024-09-13T11:23:42.391 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:42.391 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:42.391 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:42.391 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.391 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.391 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDPGPush(3.1a 79/78 [PushOp(3:5836c02d:::benchmark_data_smithi177_31116_object722:head, version: 43'43, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:5836c02d:::benchmark_data_smithi177_31116_object722:head@43'43, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:23:42.391 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:35.713273+0000", 2024-09-13T11:23:42.391 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 6.6375919789999998, 2024-09-13T11:23:42.391 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.056843473999999998, 2024-09-13T11:23:42.391 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:23:42.391 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:23:42.391 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:23:42.391 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:23:42.392 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.392 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:23:42.392 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:35.713273+0000", 2024-09-13T11:23:42.392 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:42.392 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.392 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.392 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:23:42.392 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:35.713273+0000", 2024-09-13T11:23:42.392 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:42.392 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.392 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.392 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:23:42.392 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:35.713275+0000", 2024-09-13T11:23:42.393 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.612e-06 2024-09-13T11:23:42.393 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.393 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.393 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:23:42.393 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:35.713402+0000", 2024-09-13T11:23:42.393 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00012723500000000001 2024-09-13T11:23:42.393 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.393 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.393 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:23:42.393 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:35.769307+0000", 2024-09-13T11:23:42.393 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.055904843000000003 2024-09-13T11:23:42.394 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.394 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.394 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:23:42.394 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:35.769318+0000", 2024-09-13T11:23:42.394 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.1691e-05 2024-09-13T11:23:42.394 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.394 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.394 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:42.394 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:35.769414+0000", 2024-09-13T11:23:42.394 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.6178e-05 2024-09-13T11:23:42.394 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.394 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.394 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:23:42.395 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:35.769429+0000", 2024-09-13T11:23:42.395 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.4759999999999999e-05 2024-09-13T11:23:42.395 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.395 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.395 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:23:42.395 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:35.770116+0000", 2024-09-13T11:23:42.395 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00068715499999999995 2024-09-13T11:23:42.395 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:42.396 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:42.396 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:42.396 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.396 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.396 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDPGPush(3.1a 79/78 [PushOp(3:5add3bb0:::benchmark_data_smithi177_31116_object752:head, version: 43'45, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:5add3bb0:::benchmark_data_smithi177_31116_object752:head@43'45, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:23:42.396 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:35.794551+0000", 2024-09-13T11:23:42.396 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 6.5563138800000003, 2024-09-13T11:23:42.396 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.053825619999999998, 2024-09-13T11:23:42.396 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:23:42.396 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:23:42.397 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:23:42.397 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:23:42.397 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.397 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:23:42.397 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:35.794551+0000", 2024-09-13T11:23:42.397 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:42.397 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.397 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.397 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:23:42.397 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:35.794551+0000", 2024-09-13T11:23:42.397 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:42.397 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.398 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.398 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:23:42.398 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:35.794553+0000", 2024-09-13T11:23:42.398 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.2239999999999998e-06 2024-09-13T11:23:42.398 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.398 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.398 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:23:42.398 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:35.794606+0000", 2024-09-13T11:23:42.398 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.2614000000000001e-05 2024-09-13T11:23:42.399 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.399 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.399 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:23:42.399 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:35.847786+0000", 2024-09-13T11:23:42.399 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.053179892999999999 2024-09-13T11:23:42.399 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.399 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.399 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:23:42.399 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:35.847803+0000", 2024-09-13T11:23:42.399 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.7323999999999999e-05 2024-09-13T11:23:42.399 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.399 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.399 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:42.400 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:35.847884+0000", 2024-09-13T11:23:42.400 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.1332e-05 2024-09-13T11:23:42.400 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.400 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.400 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:23:42.400 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:35.847893+0000", 2024-09-13T11:23:42.400 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.9479999999999997e-06 2024-09-13T11:23:42.400 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.400 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.400 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:23:42.400 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:35.848377+0000", 2024-09-13T11:23:42.401 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00048328500000000002 2024-09-13T11:23:42.401 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:42.401 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:42.401 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:42.401 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.401 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.401 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDPGPush(3.1a 79/78 [PushOp(3:5852224f:::benchmark_data_smithi177_31116_object768:head, version: 43'47, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:5852224f:::benchmark_data_smithi177_31116_object768:head@43'47, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:23:42.401 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:35.859184+0000", 2024-09-13T11:23:42.401 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 6.4916812909999999, 2024-09-13T11:23:42.401 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.069563262000000001, 2024-09-13T11:23:42.401 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:23:42.402 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:23:42.402 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:23:42.402 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:23:42.402 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.402 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:23:42.402 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:35.859184+0000", 2024-09-13T11:23:42.402 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:42.402 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.402 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.402 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:23:42.402 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:35.859184+0000", 2024-09-13T11:23:42.402 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:42.402 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.403 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.403 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:23:42.403 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:35.859188+0000", 2024-09-13T11:23:42.403 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.5709999999999999e-06 2024-09-13T11:23:42.403 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.403 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.403 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:23:42.403 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:35.859304+0000", 2024-09-13T11:23:42.403 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000116072 2024-09-13T11:23:42.403 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.403 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.403 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:23:42.403 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:35.927936+0000", 2024-09-13T11:23:42.403 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.068631409000000004 2024-09-13T11:23:42.404 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.404 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.404 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:23:42.404 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:35.927952+0000", 2024-09-13T11:23:42.404 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.6433e-05 2024-09-13T11:23:42.404 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.404 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.404 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:42.404 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:35.928028+0000", 2024-09-13T11:23:42.404 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.5828999999999995e-05 2024-09-13T11:23:42.404 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.404 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.404 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:23:42.405 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:35.928037+0000", 2024-09-13T11:23:42.405 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.7360000000000007e-06 2024-09-13T11:23:42.405 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.405 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.405 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:23:42.405 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:35.928747+0000", 2024-09-13T11:23:42.405 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000710212 2024-09-13T11:23:42.405 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:42.405 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:42.405 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:42.405 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.405 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.405 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDPGPush(3.1a 79/78 [PushOp(3:58bd233f:::benchmark_data_smithi177_31116_object771:head, version: 43'48, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:58bd233f:::benchmark_data_smithi177_31116_object771:head@43'48, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:23:42.406 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:35.939279+0000", 2024-09-13T11:23:42.406 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 6.4115861250000004, 2024-09-13T11:23:42.406 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.068592851999999996, 2024-09-13T11:23:42.406 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:23:42.406 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:23:42.406 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:23:42.406 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:23:42.406 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.406 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:23:42.406 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:35.939279+0000", 2024-09-13T11:23:42.406 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:42.406 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.406 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.406 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:23:42.407 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:35.939279+0000", 2024-09-13T11:23:42.407 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:42.407 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.407 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.407 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:23:42.407 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:35.939280+0000", 2024-09-13T11:23:42.407 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.187e-06 2024-09-13T11:23:42.407 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.407 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.407 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:23:42.407 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:35.939377+0000", 2024-09-13T11:23:42.407 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.7501e-05 2024-09-13T11:23:42.407 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.407 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.408 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:23:42.408 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:36.007284+0000", 2024-09-13T11:23:42.408 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.067906562000000004 2024-09-13T11:23:42.408 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.408 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.408 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:23:42.408 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:36.007301+0000", 2024-09-13T11:23:42.408 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.7163000000000002e-05 2024-09-13T11:23:42.408 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.408 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.408 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:42.408 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:36.007374+0000", 2024-09-13T11:23:42.408 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.2952000000000002e-05 2024-09-13T11:23:42.409 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.409 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.409 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:23:42.409 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:36.007382+0000", 2024-09-13T11:23:42.409 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.2500000000000006e-06 2024-09-13T11:23:42.409 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.409 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.409 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:23:42.409 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:36.007872+0000", 2024-09-13T11:23:42.409 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00048923700000000001 2024-09-13T11:23:42.409 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:42.409 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:42.409 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:42.409 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.410 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.410 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDPGPush(3.1a 79/78 [PushOp(3:5e26034b:::benchmark_data_smithi177_31116_object830:head, version: 43'56, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:5e26034b:::benchmark_data_smithi177_31116_object830:head@43'56, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:23:42.410 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:36.164241+0000", 2024-09-13T11:23:42.410 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 6.1866234990000004, 2024-09-13T11:23:42.410 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.039633759999999997, 2024-09-13T11:23:42.410 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:23:42.410 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:23:42.410 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:23:42.410 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:23:42.410 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.410 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:23:42.410 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:36.164241+0000", 2024-09-13T11:23:42.410 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:42.411 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.411 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.411 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:23:42.411 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:36.164241+0000", 2024-09-13T11:23:42.411 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:42.411 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.411 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.411 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:23:42.411 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:36.164253+0000", 2024-09-13T11:23:42.411 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.1240000000000001e-05 2024-09-13T11:23:42.411 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.411 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.411 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:23:42.411 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:36.164292+0000", 2024-09-13T11:23:42.412 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.9208000000000001e-05 2024-09-13T11:23:42.412 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.412 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.412 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:23:42.412 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:36.203116+0000", 2024-09-13T11:23:42.412 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.038823713000000003 2024-09-13T11:23:42.412 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.412 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.412 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:23:42.412 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:36.203126+0000", 2024-09-13T11:23:42.412 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0794e-05 2024-09-13T11:23:42.412 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.412 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.412 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:42.413 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:36.203204+0000", 2024-09-13T11:23:42.413 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.7871e-05 2024-09-13T11:23:42.413 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.413 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.413 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:23:42.413 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:36.203213+0000", 2024-09-13T11:23:42.413 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.6989999999999992e-06 2024-09-13T11:23:42.413 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.413 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.413 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:23:42.413 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:36.203875+0000", 2024-09-13T11:23:42.413 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00066223499999999995 2024-09-13T11:23:42.413 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:42.414 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:42.414 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:42.414 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.414 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.414 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDPGPush(3.11 79/70 [PushOp(3:8b1be12d:::benchmark_data_smithi177_31116_object43231:head, version: 66'2661, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:8b1be12d:::benchmark_data_smithi177_31116_object43231:head@66'2661, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:23:42.414 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:36.378666+0000", 2024-09-13T11:23:42.414 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 5.9721988240000004, 2024-09-13T11:23:42.414 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.035642113000000003, 2024-09-13T11:23:42.414 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:23:42.414 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:23:42.414 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:23:42.414 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:23:42.414 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.414 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:23:42.415 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:36.378666+0000", 2024-09-13T11:23:42.415 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:42.415 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.415 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.415 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:23:42.415 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:36.378666+0000", 2024-09-13T11:23:42.415 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:42.415 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.415 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.415 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:23:42.415 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:36.378668+0000", 2024-09-13T11:23:42.415 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.897e-06 2024-09-13T11:23:42.415 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.416 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.416 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:23:42.416 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:36.378721+0000", 2024-09-13T11:23:42.416 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.3499999999999999e-05 2024-09-13T11:23:42.416 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.416 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.416 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:23:42.416 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:36.413398+0000", 2024-09-13T11:23:42.416 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.034677028999999998 2024-09-13T11:23:42.416 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.416 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.416 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:23:42.416 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:36.413410+0000", 2024-09-13T11:23:42.416 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.1191e-05 2024-09-13T11:23:42.417 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.417 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.417 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:42.417 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:36.413489+0000", 2024-09-13T11:23:42.417 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.9535000000000003e-05 2024-09-13T11:23:42.417 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.417 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.417 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:23:42.417 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:36.413500+0000", 2024-09-13T11:23:42.417 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0597999999999999e-05 2024-09-13T11:23:42.417 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.417 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.417 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:23:42.417 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:36.414308+0000", 2024-09-13T11:23:42.418 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00080836299999999999 2024-09-13T11:23:42.418 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:42.418 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:42.418 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:42.418 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.418 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.418 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDPGPush(3.11 79/70 [PushOp(3:8c0cd36e:::benchmark_data_smithi177_31116_object43288:head, version: 66'2664, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:8c0cd36e:::benchmark_data_smithi177_31116_object43288:head@66'2664, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:23:42.418 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:36.416178+0000", 2024-09-13T11:23:42.418 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 5.9346869870000001, 2024-09-13T11:23:42.418 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.051627675999999997, 2024-09-13T11:23:42.418 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:23:42.418 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:23:42.418 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:23:42.418 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:23:42.419 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.419 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:23:42.419 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:36.416178+0000", 2024-09-13T11:23:42.419 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:42.419 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.419 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.419 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:23:42.419 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:36.416178+0000", 2024-09-13T11:23:42.419 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:42.419 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.419 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.419 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:23:42.419 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:36.416179+0000", 2024-09-13T11:23:42.419 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.251e-06 2024-09-13T11:23:42.420 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.420 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.420 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:23:42.420 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:36.416272+0000", 2024-09-13T11:23:42.420 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.3123999999999998e-05 2024-09-13T11:23:42.420 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.420 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.420 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:23:42.420 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:36.466753+0000", 2024-09-13T11:23:42.420 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.050480766000000003 2024-09-13T11:23:42.420 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.420 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.420 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:23:42.421 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:36.466771+0000", 2024-09-13T11:23:42.421 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.84e-05 2024-09-13T11:23:42.421 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.421 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.421 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:42.421 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:36.466901+0000", 2024-09-13T11:23:42.421 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00012914 2024-09-13T11:23:42.421 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.421 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.421 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:23:42.421 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:36.466916+0000", 2024-09-13T11:23:42.421 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.5855999999999999e-05 2024-09-13T11:23:42.421 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.421 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.422 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:23:42.422 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:36.467806+0000", 2024-09-13T11:23:42.422 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00088913900000000005 2024-09-13T11:23:42.422 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:42.422 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:42.422 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:42.422 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.422 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.422 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDPGPush(3.11 79/70 [PushOp(3:8c9d203a:::benchmark_data_smithi177_31116_object43434:head, version: 66'2673, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:8c9d203a:::benchmark_data_smithi177_31116_object43434:head@66'2673, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:23:42.422 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:36.536956+0000", 2024-09-13T11:23:42.422 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 5.8139093219999998, 2024-09-13T11:23:42.422 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.061170707999999997, 2024-09-13T11:23:42.422 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:23:42.423 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:23:42.423 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:23:42.423 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:23:42.423 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.423 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:23:42.423 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:36.536956+0000", 2024-09-13T11:23:42.423 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:42.423 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.423 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.423 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:23:42.423 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:36.536956+0000", 2024-09-13T11:23:42.423 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:42.423 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.424 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.424 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:23:42.424 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:36.536957+0000", 2024-09-13T11:23:42.424 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.296e-06 2024-09-13T11:23:42.424 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.424 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.424 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:23:42.424 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:36.537008+0000", 2024-09-13T11:23:42.424 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.1133e-05 2024-09-13T11:23:42.424 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.424 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.424 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:23:42.424 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:36.596934+0000", 2024-09-13T11:23:42.425 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.05992608 2024-09-13T11:23:42.425 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.425 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.425 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:23:42.425 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:36.596959+0000", 2024-09-13T11:23:42.425 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.4723e-05 2024-09-13T11:23:42.425 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.425 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.425 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:42.425 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:36.597107+0000", 2024-09-13T11:23:42.425 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00014771599999999999 2024-09-13T11:23:42.425 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.425 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.425 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:23:42.426 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:36.597124+0000", 2024-09-13T11:23:42.426 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.7909000000000001e-05 2024-09-13T11:23:42.426 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.426 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.426 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:23:42.426 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:36.598126+0000", 2024-09-13T11:23:42.426 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.001001851 2024-09-13T11:23:42.426 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:42.426 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:42.426 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:42.426 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.426 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.426 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDPGPush(3.11 79/70 [PushOp(3:8d123873:::benchmark_data_smithi177_31116_object43459:head, version: 66'2676, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:8d123873:::benchmark_data_smithi177_31116_object43459:head@66'2676, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:23:42.427 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:36.632222+0000", 2024-09-13T11:23:42.427 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 5.7186428060000001, 2024-09-13T11:23:42.427 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.054211717999999999, 2024-09-13T11:23:42.427 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:23:42.427 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:23:42.427 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:23:42.427 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:23:42.427 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.427 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:23:42.427 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:36.632222+0000", 2024-09-13T11:23:42.427 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:42.427 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.427 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.427 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:23:42.428 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:36.632222+0000", 2024-09-13T11:23:42.428 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:42.428 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.428 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.428 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:23:42.428 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:36.632223+0000", 2024-09-13T11:23:42.428 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.3289999999999999e-06 2024-09-13T11:23:42.428 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.428 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.428 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:23:42.428 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:36.632314+0000", 2024-09-13T11:23:42.428 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.0409999999999997e-05 2024-09-13T11:23:42.428 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.429 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.429 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:23:42.429 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:36.685410+0000", 2024-09-13T11:23:42.429 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.053096078999999997 2024-09-13T11:23:42.429 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.429 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.429 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:23:42.429 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:36.685426+0000", 2024-09-13T11:23:42.429 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.5909e-05 2024-09-13T11:23:42.429 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.429 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.429 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:42.429 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:36.685527+0000", 2024-09-13T11:23:42.429 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000101514 2024-09-13T11:23:42.430 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.430 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.430 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:23:42.430 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:36.685541+0000", 2024-09-13T11:23:42.430 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.3633e-05 2024-09-13T11:23:42.430 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.430 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.430 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:23:42.430 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:36.686434+0000", 2024-09-13T11:23:42.430 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00089284399999999995 2024-09-13T11:23:42.430 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:42.430 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:42.430 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:42.431 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.431 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.431 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDPGPush(3.11 79/70 [PushOp(3:8fb17a1a:::benchmark_data_smithi177_31116_object43489:head, version: 66'2680, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:8fb17a1a:::benchmark_data_smithi177_31116_object43489:head@66'2680, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:23:42.431 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:36.689090+0000", 2024-09-13T11:23:42.431 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 5.6617744999999999, 2024-09-13T11:23:42.431 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.052326250999999997, 2024-09-13T11:23:42.431 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:23:42.431 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:23:42.431 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:23:42.431 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:23:42.431 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.431 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:23:42.431 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:36.689090+0000", 2024-09-13T11:23:42.431 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:42.432 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.432 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.432 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:23:42.432 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:36.689090+0000", 2024-09-13T11:23:42.432 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:42.432 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.432 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.432 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:23:42.432 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:36.689092+0000", 2024-09-13T11:23:42.432 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.294e-06 2024-09-13T11:23:42.432 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.432 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.432 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:23:42.432 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:36.689158+0000", 2024-09-13T11:23:42.433 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.6116000000000002e-05 2024-09-13T11:23:42.433 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.433 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.433 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:23:42.433 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:36.740439+0000", 2024-09-13T11:23:42.433 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.051281302000000001 2024-09-13T11:23:42.433 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.433 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.433 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:23:42.433 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:36.740458+0000", 2024-09-13T11:23:42.433 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.8649999999999999e-05 2024-09-13T11:23:42.433 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.433 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.434 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:42.434 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:36.740562+0000", 2024-09-13T11:23:42.434 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000103769 2024-09-13T11:23:42.434 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.434 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.434 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:23:42.434 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:36.740576+0000", 2024-09-13T11:23:42.434 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.4243000000000001e-05 2024-09-13T11:23:42.434 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.434 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.434 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:23:42.434 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:36.741417+0000", 2024-09-13T11:23:42.434 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00084087699999999997 2024-09-13T11:23:42.435 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:42.435 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:42.435 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:42.435 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.435 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.435 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDPGPush(3.11 79/70 [PushOp(3:8a1fe564:::benchmark_data_smithi177_31116_object43550:head, version: 66'2685, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:8a1fe564:::benchmark_data_smithi177_31116_object43550:head@66'2685, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:23:42.435 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:36.748870+0000", 2024-09-13T11:23:42.435 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 5.6019946850000002, 2024-09-13T11:23:42.435 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.055968057000000002, 2024-09-13T11:23:42.435 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:23:42.435 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:23:42.435 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:23:42.435 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:23:42.436 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.436 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:23:42.436 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:36.748870+0000", 2024-09-13T11:23:42.436 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:42.436 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.436 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.436 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:23:42.436 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:36.748870+0000", 2024-09-13T11:23:42.436 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:42.436 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.436 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.436 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:23:42.436 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:36.748881+0000", 2024-09-13T11:23:42.436 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.1268e-05 2024-09-13T11:23:42.437 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.437 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.437 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:23:42.437 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:36.748922+0000", 2024-09-13T11:23:42.437 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.0719000000000001e-05 2024-09-13T11:23:42.437 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.437 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.437 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:23:42.437 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:36.803947+0000", 2024-09-13T11:23:42.437 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.055025253000000003 2024-09-13T11:23:42.437 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.437 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.437 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:23:42.437 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:36.803967+0000", 2024-09-13T11:23:42.438 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.9426e-05 2024-09-13T11:23:42.438 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.438 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.438 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:42.438 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:36.804047+0000", 2024-09-13T11:23:42.438 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.0512000000000004e-05 2024-09-13T11:23:42.438 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.438 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.438 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:23:42.438 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:36.804056+0000", 2024-09-13T11:23:42.438 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.1359999999999997e-06 2024-09-13T11:23:42.438 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.438 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.439 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:23:42.439 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:36.804838+0000", 2024-09-13T11:23:42.439 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00078274300000000002 2024-09-13T11:23:42.439 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:42.439 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:42.439 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:42.439 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.439 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.439 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDPGPush(3.11 79/70 [PushOp(3:8d2fd49c:::benchmark_data_smithi177_31116_object43641:head, version: 66'2691, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:8d2fd49c:::benchmark_data_smithi177_31116_object43641:head@66'2691, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:23:42.439 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:36.840640+0000", 2024-09-13T11:23:42.439 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 5.510225342, 2024-09-13T11:23:42.439 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.051580590000000003, 2024-09-13T11:23:42.439 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:23:42.439 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:23:42.440 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:23:42.440 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:23:42.440 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.440 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:23:42.440 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:36.840640+0000", 2024-09-13T11:23:42.440 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:42.440 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.440 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.440 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:23:42.440 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:36.840640+0000", 2024-09-13T11:23:42.440 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:42.440 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.440 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.441 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:23:42.441 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:36.840641+0000", 2024-09-13T11:23:42.441 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.547e-06 2024-09-13T11:23:42.441 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.441 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.441 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:23:42.441 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:36.840693+0000", 2024-09-13T11:23:42.441 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.1690000000000001e-05 2024-09-13T11:23:42.441 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.441 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.441 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:23:42.441 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:36.891210+0000", 2024-09-13T11:23:42.441 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.050517339000000001 2024-09-13T11:23:42.442 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.442 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.442 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:23:42.442 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:36.891228+0000", 2024-09-13T11:23:42.442 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.7937000000000001e-05 2024-09-13T11:23:42.442 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.442 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.442 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:42.442 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:36.891347+0000", 2024-09-13T11:23:42.442 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000118733 2024-09-13T11:23:42.442 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.442 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.442 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:23:42.442 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:36.891367+0000", 2024-09-13T11:23:42.443 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.0514999999999999e-05 2024-09-13T11:23:42.443 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.443 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.443 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:23:42.443 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:36.892220+0000", 2024-09-13T11:23:42.443 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00085282900000000004 2024-09-13T11:23:42.443 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:42.443 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:42.443 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:42.443 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.443 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.443 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDPGPush(3.1e 80/78 [PushOp(3:7c63aba7:::benchmark_data_smithi177_31116_object2869:head, version: 44'199, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:7c63aba7:::benchmark_data_smithi177_31116_object2869:head@44'199, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:23:42.443 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:36.987368+0000", 2024-09-13T11:23:42.444 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 5.3634964529999998, 2024-09-13T11:23:42.444 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.078572671999999996, 2024-09-13T11:23:42.444 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:23:42.444 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:23:42.444 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:23:42.444 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:23:42.444 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.444 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:23:42.444 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:36.987368+0000", 2024-09-13T11:23:42.444 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:42.444 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.444 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.444 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:23:42.445 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:36.987368+0000", 2024-09-13T11:23:42.445 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:42.445 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.445 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.445 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:23:42.445 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:36.987370+0000", 2024-09-13T11:23:42.445 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.66e-06 2024-09-13T11:23:42.445 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.445 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.445 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:23:42.445 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:36.987402+0000", 2024-09-13T11:23:42.445 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.2406000000000001e-05 2024-09-13T11:23:42.445 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.445 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.446 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:23:42.446 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:37.065145+0000", 2024-09-13T11:23:42.446 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.077742364999999994 2024-09-13T11:23:42.446 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.446 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.446 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:23:42.446 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:37.065164+0000", 2024-09-13T11:23:42.446 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.8706999999999999e-05 2024-09-13T11:23:42.446 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.446 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.446 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:42.446 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:37.065273+0000", 2024-09-13T11:23:42.446 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000109788 2024-09-13T11:23:42.446 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.447 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.447 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:23:42.447 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:37.065288+0000", 2024-09-13T11:23:42.447 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.4649e-05 2024-09-13T11:23:42.447 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.447 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.447 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:23:42.447 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:37.065941+0000", 2024-09-13T11:23:42.447 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00065309700000000005 2024-09-13T11:23:42.447 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:42.447 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:42.447 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:42.447 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:42.447 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:42.448 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-13T11:23:42.448 INFO:tasks.ceph.osd.12.smithi001.stderr:2024-09-13T11:23:42.348+0000 7f892c6e6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26314) UID: 0 2024-09-13T11:23:42.449 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-09-13T11:23:42.453 INFO:tasks.ceph.osd.11.smithi177.stderr:2024-09-13T11:23:42.448+0000 7f24792c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26580) UID: 0 2024-09-13T11:23:42.555 INFO:tasks.ceph.osd.7.smithi177.stderr:2024-09-13T11:23:42.548+0000 7f897b51d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26575) UID: 0 2024-09-13T11:23:42.620 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-13T11:23:42.621 INFO:teuthology.orchestra.run.smithi001.stdout: "ops": [], 2024-09-13T11:23:42.621 INFO:teuthology.orchestra.run.smithi001.stdout: "num_ops": 0 2024-09-13T11:23:42.621 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-13T11:23:42.638 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-09-13T11:23:42.688 INFO:tasks.ceph.osd.4.smithi001.stderr:2024-09-13T11:23:42.684+0000 7ff8e0218640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26310) UID: 0 2024-09-13T11:23:42.757 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-13T11:23:42.757 INFO:teuthology.orchestra.run.smithi001.stdout: "ops": [], 2024-09-13T11:23:42.757 INFO:teuthology.orchestra.run.smithi001.stdout: "complaint_time": 30, 2024-09-13T11:23:42.757 INFO:teuthology.orchestra.run.smithi001.stdout: "num_blocked_ops": 0 2024-09-13T11:23:42.757 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-13T11:23:42.757 INFO:tasks.ceph.osd.4.smithi001.stderr:2024-09-13T11:23:42.752+0000 7ff8e0218640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26310) UID: 0 2024-09-13T11:23:42.771 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-09-13T11:23:42.860 INFO:tasks.ceph.osd.4.smithi001.stderr:2024-09-13T11:23:42.856+0000 7ff8e0218640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26310) UID: 0 2024-09-13T11:23:42.920 INFO:tasks.ceph.osd.8.smithi001.stderr:2024-09-13T11:23:42.916+0000 7fb9ab2df640 -1 osd.8 85 handle_osd_map injecting map crc failure 2024-09-13T11:23:42.926 INFO:teuthology.orchestra.run.smithi001.stderr:reweighted osd.2 to 1 (10000) 2024-09-13T11:23:42.933 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-13T11:23:42.933 INFO:teuthology.orchestra.run.smithi001.stdout: "size": 20, 2024-09-13T11:23:42.933 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 600, 2024-09-13T11:23:42.934 INFO:teuthology.orchestra.run.smithi001.stdout: "ops": [ 2024-09-13T11:23:42.934 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.934 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:30779 3.e 3:77c07d05:::benchmark_data_smithi177_31116_object30778:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e59)", 2024-09-13T11:23:42.934 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:09.473906+0000", 2024-09-13T11:23:42.934 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 33.457321919000002, 2024-09-13T11:23:42.934 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.224949224, 2024-09-13T11:23:42.934 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:23:42.934 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:23:42.934 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:42.934 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:23:42.935 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:23:42.935 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:42.935 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 30779 2024-09-13T11:23:42.935 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.935 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:23:42.935 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.935 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:23:42.935 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.473906+0000", 2024-09-13T11:23:42.935 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:42.935 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.935 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.935 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:23:42.935 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.473906+0000", 2024-09-13T11:23:42.935 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:42.936 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.936 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.936 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:23:42.936 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.473908+0000", 2024-09-13T11:23:42.936 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.6230000000000001e-06 2024-09-13T11:23:42.936 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.936 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.936 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:23:42.936 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.473961+0000", 2024-09-13T11:23:42.936 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.2374999999999999e-05 2024-09-13T11:23:42.936 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.936 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.936 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:23:42.937 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.473976+0000", 2024-09-13T11:23:42.937 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.56e-05 2024-09-13T11:23:42.937 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.937 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.937 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:23:42.937 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.473987+0000", 2024-09-13T11:23:42.937 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0394000000000001e-05 2024-09-13T11:23:42.937 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.937 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.937 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:42.937 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.474275+0000", 2024-09-13T11:23:42.937 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00028800800000000002 2024-09-13T11:23:42.937 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.937 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.938 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:23:42.938 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.474466+0000", 2024-09-13T11:23:42.938 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000191499 2024-09-13T11:23:42.938 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.938 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.938 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 0,1", 2024-09-13T11:23:42.938 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.474666+0000", 2024-09-13T11:23:42.938 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00019994899999999999 2024-09-13T11:23:42.938 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.938 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.938 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:42.938 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.478354+0000", 2024-09-13T11:23:42.938 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0036876779999999998 2024-09-13T11:23:42.939 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.939 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.939 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:23:42.939 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.483128+0000", 2024-09-13T11:23:42.939 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0047739410000000003 2024-09-13T11:23:42.939 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.939 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.939 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:42.939 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.697695+0000", 2024-09-13T11:23:42.939 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.214566955 2024-09-13T11:23:42.939 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.939 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.939 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:23:42.939 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.697786+0000", 2024-09-13T11:23:42.940 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.1310000000000005e-05 2024-09-13T11:23:42.940 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.940 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.940 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:23:42.940 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.698855+0000", 2024-09-13T11:23:42.940 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0010688920000000001 2024-09-13T11:23:42.940 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:42.940 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:42.940 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:42.940 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.940 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.940 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:30801 3.e 3:73b0f21a:::benchmark_data_smithi177_31116_object30800:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e59)", 2024-09-13T11:23:42.940 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:09.485629+0000", 2024-09-13T11:23:42.941 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 33.445599117, 2024-09-13T11:23:42.941 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.22604919900000001, 2024-09-13T11:23:42.941 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:23:42.941 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:23:42.941 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:42.941 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:23:42.941 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:23:42.941 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:42.941 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 30801 2024-09-13T11:23:42.941 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.941 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:23:42.941 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.941 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:23:42.942 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.485629+0000", 2024-09-13T11:23:42.942 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:42.942 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.942 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.942 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:23:42.942 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.485629+0000", 2024-09-13T11:23:42.942 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:42.942 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.942 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.942 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:23:42.942 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.485631+0000", 2024-09-13T11:23:42.942 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.9989999999999998e-06 2024-09-13T11:23:42.942 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.942 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.943 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:23:42.943 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.485664+0000", 2024-09-13T11:23:42.943 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.3896000000000002e-05 2024-09-13T11:23:42.943 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.943 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.943 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:23:42.943 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.485679+0000", 2024-09-13T11:23:42.943 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.4438e-05 2024-09-13T11:23:42.943 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.943 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.943 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:23:42.943 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.485688+0000", 2024-09-13T11:23:42.943 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.0180000000000007e-06 2024-09-13T11:23:42.943 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.944 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.944 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:42.944 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.485742+0000", 2024-09-13T11:23:42.944 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.4026000000000001e-05 2024-09-13T11:23:42.944 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.944 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.944 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:23:42.944 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.485841+0000", 2024-09-13T11:23:42.944 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.8875000000000005e-05 2024-09-13T11:23:42.944 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.944 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.944 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 0,1", 2024-09-13T11:23:42.944 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.486005+0000", 2024-09-13T11:23:42.945 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000163747 2024-09-13T11:23:42.945 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.945 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.945 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:23:42.945 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.487867+0000", 2024-09-13T11:23:42.945 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.001862945 2024-09-13T11:23:42.945 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.945 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.945 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:42.945 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.488570+0000", 2024-09-13T11:23:42.945 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00070255700000000003 2024-09-13T11:23:42.945 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.945 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.946 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:42.946 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.711140+0000", 2024-09-13T11:23:42.946 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.222570348 2024-09-13T11:23:42.946 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.946 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.946 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:23:42.946 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.711180+0000", 2024-09-13T11:23:42.946 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.9273999999999997e-05 2024-09-13T11:23:42.946 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.946 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.946 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:23:42.946 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.711678+0000", 2024-09-13T11:23:42.946 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00049807599999999997 2024-09-13T11:23:42.947 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:42.947 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:42.947 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:42.947 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.947 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.947 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:54656 3.1e 3.6a49163e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e75)", 2024-09-13T11:23:42.947 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:30.979993+0000", 2024-09-13T11:23:42.947 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 11.951234684999999, 2024-09-13T11:23:42.947 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.9269557150000001, 2024-09-13T11:23:42.947 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:23:42.947 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:23:42.947 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-13T11:23:42.947 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:23:42.947 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:23:42.947 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:42.948 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 54656 2024-09-13T11:23:42.948 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.948 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:23:42.948 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.948 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:23:42.948 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:30.979993+0000", 2024-09-13T11:23:42.948 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:42.948 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.948 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.948 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:23:42.948 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:30.979993+0000", 2024-09-13T11:23:42.948 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:42.949 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.949 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.949 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:23:42.949 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:30.979997+0000", 2024-09-13T11:23:42.949 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.0319999999999997e-06 2024-09-13T11:23:42.949 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.949 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.949 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:23:42.949 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:30.980057+0000", 2024-09-13T11:23:42.949 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.0341999999999999e-05 2024-09-13T11:23:42.949 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.949 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.949 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:23:42.949 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:30.980094+0000", 2024-09-13T11:23:42.950 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.6770000000000002e-05 2024-09-13T11:23:42.950 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.950 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.950 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:23:42.950 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:30.980113+0000", 2024-09-13T11:23:42.950 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.8791999999999999e-05 2024-09-13T11:23:42.950 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.950 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.950 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:42.950 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.097773+0000", 2024-09-13T11:23:42.950 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.117660379 2024-09-13T11:23:42.950 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.950 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.951 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for peered", 2024-09-13T11:23:42.951 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.097784+0000", 2024-09-13T11:23:42.951 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.1025e-05 2024-09-13T11:23:42.951 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.951 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.951 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:42.951 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.906937+0000", 2024-09-13T11:23:42.951 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.8091523789999999 2024-09-13T11:23:42.951 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.951 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.951 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:23:42.951 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.906949+0000", 2024-09-13T11:23:42.951 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.1996e-05 2024-09-13T11:23:42.951 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:42.952 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:42.952 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:42.952 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.952 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.952 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDPGPushReply(3.17 75/70 [PushReplyOp(3:e82800c8:::benchmark_data_smithi177_31116_object13909:head)])", 2024-09-13T11:23:42.952 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:31.632344+0000", 2024-09-13T11:23:42.952 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 11.298883829999999, 2024-09-13T11:23:42.952 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.28592102899999999, 2024-09-13T11:23:42.952 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:23:42.952 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:23:42.952 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-13T11:23:42.952 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:23:42.952 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.953 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:23:42.953 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.632344+0000", 2024-09-13T11:23:42.953 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:42.953 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.953 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.953 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:23:42.953 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.632344+0000", 2024-09-13T11:23:42.953 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:42.953 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.953 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.953 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:23:42.953 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.632346+0000", 2024-09-13T11:23:42.953 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.8819999999999999e-06 2024-09-13T11:23:42.954 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.954 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.954 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:23:42.954 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.632353+0000", 2024-09-13T11:23:42.954 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.5059999999999998e-06 2024-09-13T11:23:42.954 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.954 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.954 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:23:42.954 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.917361+0000", 2024-09-13T11:23:42.954 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.285008231 2024-09-13T11:23:42.954 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.954 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.954 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:23:42.955 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.917372+0000", 2024-09-13T11:23:42.955 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0764999999999999e-05 2024-09-13T11:23:42.955 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.955 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.955 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:42.955 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.917458+0000", 2024-09-13T11:23:42.955 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.6136000000000003e-05 2024-09-13T11:23:42.955 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.955 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.955 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:23:42.955 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.918265+0000", 2024-09-13T11:23:42.955 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00080650899999999996 2024-09-13T11:23:42.955 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:42.955 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:42.956 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:42.956 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.956 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.956 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:54720 3.1e 3.a944303e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-13T11:23:42.956 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:31.973315+0000", 2024-09-13T11:23:42.956 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 10.95791245, 2024-09-13T11:23:42.956 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.93366527200000005, 2024-09-13T11:23:42.956 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:23:42.956 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:23:42.956 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-13T11:23:42.956 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:23:42.956 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:23:42.956 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:42.957 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 54720 2024-09-13T11:23:42.957 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.957 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:23:42.957 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.957 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:23:42.957 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.973315+0000", 2024-09-13T11:23:42.957 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:42.957 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.957 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.957 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:23:42.957 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.973315+0000", 2024-09-13T11:23:42.957 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:42.957 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.957 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.958 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:23:42.958 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.973318+0000", 2024-09-13T11:23:42.960 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.5830000000000001e-06 2024-09-13T11:23:42.960 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.960 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.960 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:23:42.960 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.973465+0000", 2024-09-13T11:23:42.960 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000146816 2024-09-13T11:23:42.960 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.960 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.960 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:23:42.960 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.973478+0000", 2024-09-13T11:23:42.960 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2931e-05 2024-09-13T11:23:42.960 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.960 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.961 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:23:42.961 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.973483+0000", 2024-09-13T11:23:42.961 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.8969999999999996e-06 2024-09-13T11:23:42.961 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.961 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.961 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:42.961 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.973548+0000", 2024-09-13T11:23:42.961 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.4771999999999996e-05 2024-09-13T11:23:42.961 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.961 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.961 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for peered", 2024-09-13T11:23:42.961 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.973557+0000", 2024-09-13T11:23:42.961 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.0359999999999995e-06 2024-09-13T11:23:42.962 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.962 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.962 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:42.962 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.906974+0000", 2024-09-13T11:23:42.962 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.93341706000000002 2024-09-13T11:23:42.962 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.962 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.962 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:23:42.962 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.906980+0000", 2024-09-13T11:23:42.962 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.1770000000000003e-06 2024-09-13T11:23:42.962 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:42.962 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:42.962 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:42.962 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.963 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.963 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:54734 3.1e 3.d7f6f7e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-13T11:23:42.963 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:31.983342+0000", 2024-09-13T11:23:42.963 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 10.947885647, 2024-09-13T11:23:42.963 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.92367058599999996, 2024-09-13T11:23:42.963 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:23:42.964 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:23:42.964 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-13T11:23:42.964 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:23:42.964 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:23:42.964 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:42.964 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 54734 2024-09-13T11:23:42.965 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.965 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:23:42.965 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.965 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:23:42.965 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.983342+0000", 2024-09-13T11:23:42.966 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:42.966 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.966 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.966 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:23:42.966 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.983342+0000", 2024-09-13T11:23:42.966 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:42.967 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.967 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.967 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:23:42.967 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.983345+0000", 2024-09-13T11:23:42.967 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.6220000000000001e-06 2024-09-13T11:23:42.967 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.967 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.967 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:23:42.967 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.983438+0000", 2024-09-13T11:23:42.967 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.3517000000000006e-05 2024-09-13T11:23:42.967 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.968 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.968 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:23:42.968 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.983451+0000", 2024-09-13T11:23:42.968 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2527999999999999e-05 2024-09-13T11:23:42.968 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.968 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.968 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:23:42.968 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.983458+0000", 2024-09-13T11:23:42.968 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.6380000000000004e-06 2024-09-13T11:23:42.968 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.968 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.968 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:42.968 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.983521+0000", 2024-09-13T11:23:42.969 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.2729000000000002e-05 2024-09-13T11:23:42.969 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.969 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.969 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for peered", 2024-09-13T11:23:42.969 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.983548+0000", 2024-09-13T11:23:42.969 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.6537999999999999e-05 2024-09-13T11:23:42.969 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.969 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.969 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:42.969 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.907005+0000", 2024-09-13T11:23:42.969 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.923456991 2024-09-13T11:23:42.969 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.969 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.970 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:23:42.970 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.907013+0000", 2024-09-13T11:23:42.970 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.0229999999999992e-06 2024-09-13T11:23:42.970 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:42.970 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:42.970 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:42.970 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.970 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.970 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:54760 3.1e 3.8c32c13e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-13T11:23:42.970 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:31.999685+0000", 2024-09-13T11:23:42.970 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 10.931542414999999, 2024-09-13T11:23:42.970 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.90735639599999995, 2024-09-13T11:23:42.970 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:23:42.971 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:23:42.971 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-13T11:23:42.971 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:23:42.971 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:23:42.971 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:42.971 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 54760 2024-09-13T11:23:42.971 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.971 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:23:42.971 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.971 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:23:42.971 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.999685+0000", 2024-09-13T11:23:42.971 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:42.971 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.971 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.972 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:23:42.972 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.999685+0000", 2024-09-13T11:23:42.972 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:42.972 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.972 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.972 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:23:42.972 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.999690+0000", 2024-09-13T11:23:42.972 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.7879999999999997e-06 2024-09-13T11:23:42.972 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.972 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.972 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:23:42.972 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.999729+0000", 2024-09-13T11:23:42.972 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.9032000000000002e-05 2024-09-13T11:23:42.973 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.973 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.973 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:23:42.973 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.999741+0000", 2024-09-13T11:23:42.973 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2136999999999999e-05 2024-09-13T11:23:42.973 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.973 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.973 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:23:42.973 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.999748+0000", 2024-09-13T11:23:42.973 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.9120000000000001e-06 2024-09-13T11:23:42.973 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.973 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.974 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:42.974 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.999803+0000", 2024-09-13T11:23:42.974 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.5064e-05 2024-09-13T11:23:42.974 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.974 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.974 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for peered", 2024-09-13T11:23:42.974 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.999815+0000", 2024-09-13T11:23:42.974 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2041e-05 2024-09-13T11:23:42.974 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.974 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.974 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:42.974 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.907034+0000", 2024-09-13T11:23:42.974 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.90721858200000005 2024-09-13T11:23:42.974 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.975 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.975 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:23:42.975 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.907042+0000", 2024-09-13T11:23:42.975 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.8399999999999995e-06 2024-09-13T11:23:42.975 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:42.975 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:42.975 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:42.975 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.975 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.975 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:54768 3.1e 3.6dad499e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-13T11:23:42.975 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:32.003482+0000", 2024-09-13T11:23:42.975 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 10.927745764999999, 2024-09-13T11:23:42.975 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.90358804699999995, 2024-09-13T11:23:42.975 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:23:42.976 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:23:42.976 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-13T11:23:42.976 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:23:42.976 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:23:42.976 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:42.976 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 54768 2024-09-13T11:23:42.976 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.976 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:23:42.976 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.976 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:23:42.976 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.003482+0000", 2024-09-13T11:23:42.976 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:42.976 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.977 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.977 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:23:42.977 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.003482+0000", 2024-09-13T11:23:42.977 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:42.977 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.977 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.977 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:23:42.977 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.003486+0000", 2024-09-13T11:23:42.977 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.1030000000000004e-06 2024-09-13T11:23:42.977 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.977 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.977 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:23:42.977 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.003523+0000", 2024-09-13T11:23:42.977 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.6732e-05 2024-09-13T11:23:42.978 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.978 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.978 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:23:42.978 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.003535+0000", 2024-09-13T11:23:42.978 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2520999999999999e-05 2024-09-13T11:23:42.978 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.978 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.978 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:23:42.978 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.003541+0000", 2024-09-13T11:23:42.978 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.8690000000000001e-06 2024-09-13T11:23:42.978 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.978 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.978 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:42.978 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.003593+0000", 2024-09-13T11:23:42.979 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.1786000000000002e-05 2024-09-13T11:23:42.979 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.979 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.979 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for peered", 2024-09-13T11:23:42.979 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.003602+0000", 2024-09-13T11:23:42.979 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.9679999999999995e-06 2024-09-13T11:23:42.979 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.979 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.979 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:42.979 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.907063+0000", 2024-09-13T11:23:42.979 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.90346116600000004 2024-09-13T11:23:42.979 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.979 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.979 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:23:42.980 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.907070+0000", 2024-09-13T11:23:42.980 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.9020000000000002e-06 2024-09-13T11:23:42.980 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:42.980 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:42.980 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:42.980 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.980 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.980 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:54785 3.1e 3.d32f1ede (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-13T11:23:42.980 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:32.016238+0000", 2024-09-13T11:23:42.980 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 10.914989555, 2024-09-13T11:23:42.980 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.89086243399999998, 2024-09-13T11:23:42.980 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:23:42.980 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:23:42.981 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-13T11:23:42.981 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:23:42.981 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:23:42.981 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:42.981 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 54785 2024-09-13T11:23:42.981 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.981 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:23:42.981 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.981 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:23:42.981 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.016238+0000", 2024-09-13T11:23:42.981 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:42.981 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.981 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.981 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:23:42.982 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.016238+0000", 2024-09-13T11:23:42.982 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:42.982 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.982 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.982 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:23:42.982 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.016241+0000", 2024-09-13T11:23:42.982 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.6469999999999998e-06 2024-09-13T11:23:42.982 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.982 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.982 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:23:42.982 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.016271+0000", 2024-09-13T11:23:42.982 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.0505999999999999e-05 2024-09-13T11:23:42.982 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.983 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.983 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:23:42.983 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.016287+0000", 2024-09-13T11:23:42.983 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.6073999999999999e-05 2024-09-13T11:23:42.983 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.983 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.983 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:23:42.983 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.016295+0000", 2024-09-13T11:23:42.983 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.0590000000000003e-06 2024-09-13T11:23:42.983 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.983 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.983 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:42.983 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.016373+0000", 2024-09-13T11:23:42.983 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.7634999999999998e-05 2024-09-13T11:23:42.984 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.984 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.984 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for peered", 2024-09-13T11:23:42.984 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.016383+0000", 2024-09-13T11:23:42.984 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0246e-05 2024-09-13T11:23:42.984 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.984 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.984 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:42.984 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.907092+0000", 2024-09-13T11:23:42.984 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.89070849200000002 2024-09-13T11:23:42.984 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.984 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.985 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:23:42.985 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.907101+0000", 2024-09-13T11:23:42.985 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.7749999999999998e-06 2024-09-13T11:23:42.985 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:42.985 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:42.985 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:42.985 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.985 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.985 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:54811 3.1e 3.3754367e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-13T11:23:42.985 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:32.045424+0000", 2024-09-13T11:23:42.985 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 10.885803903999999, 2024-09-13T11:23:42.985 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.86170719399999995, 2024-09-13T11:23:42.985 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:23:42.985 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:23:42.986 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-13T11:23:42.986 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:23:42.986 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:23:42.986 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:42.986 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 54811 2024-09-13T11:23:42.986 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.986 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:23:42.986 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.986 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:23:42.986 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.045424+0000", 2024-09-13T11:23:42.986 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:42.986 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.986 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.987 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:23:42.987 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.045424+0000", 2024-09-13T11:23:42.987 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:42.987 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.987 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.987 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:23:42.987 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.045426+0000", 2024-09-13T11:23:42.987 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.0260000000000002e-06 2024-09-13T11:23:42.987 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.987 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.987 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:23:42.987 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.045556+0000", 2024-09-13T11:23:42.987 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00013003099999999999 2024-09-13T11:23:42.988 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.988 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.988 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:23:42.988 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.045568+0000", 2024-09-13T11:23:42.988 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.1727e-05 2024-09-13T11:23:42.988 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.988 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.988 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:23:42.988 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.045576+0000", 2024-09-13T11:23:42.988 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.6459999999999999e-06 2024-09-13T11:23:42.988 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.988 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.988 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:42.988 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.045650+0000", 2024-09-13T11:23:42.989 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.4000999999999998e-05 2024-09-13T11:23:42.989 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.989 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.989 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for peered", 2024-09-13T11:23:42.989 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.045658+0000", 2024-09-13T11:23:42.989 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.3869999999999997e-06 2024-09-13T11:23:42.989 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.989 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.989 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:42.989 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.907123+0000", 2024-09-13T11:23:42.989 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.86146508399999999 2024-09-13T11:23:42.989 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.989 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.990 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:23:42.990 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.907131+0000", 2024-09-13T11:23:42.990 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.2919999999999996e-06 2024-09-13T11:23:42.990 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:42.990 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:42.990 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:42.990 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.990 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.990 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:54817 3.1e 3.36dd2fde (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-13T11:23:42.990 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:32.050108+0000", 2024-09-13T11:23:42.990 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 10.881120012, 2024-09-13T11:23:42.990 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.85705549199999997, 2024-09-13T11:23:42.990 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:23:42.990 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:23:42.991 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-13T11:23:42.991 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:23:42.991 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:23:42.991 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:42.991 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 54817 2024-09-13T11:23:42.991 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.991 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:23:42.991 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.991 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:23:42.991 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.050108+0000", 2024-09-13T11:23:42.991 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:42.991 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.991 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.991 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:23:42.992 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.050108+0000", 2024-09-13T11:23:42.992 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:42.992 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.992 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.992 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:23:42.992 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.050110+0000", 2024-09-13T11:23:42.992 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.0209999999999998e-06 2024-09-13T11:23:42.992 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.992 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.992 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:23:42.992 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.050166+0000", 2024-09-13T11:23:42.992 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.6026000000000002e-05 2024-09-13T11:23:42.992 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.992 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.993 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:23:42.993 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.050180+0000", 2024-09-13T11:23:42.993 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.4307e-05 2024-09-13T11:23:42.993 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.993 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.993 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:23:42.993 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.050188+0000", 2024-09-13T11:23:42.993 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.5410000000000003e-06 2024-09-13T11:23:42.993 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.993 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.993 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:42.994 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.050260+0000", 2024-09-13T11:23:42.994 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.2157999999999996e-05 2024-09-13T11:23:42.994 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.994 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.994 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for peered", 2024-09-13T11:23:42.994 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.050267+0000", 2024-09-13T11:23:42.994 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.0190000000000001e-06 2024-09-13T11:23:42.994 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.994 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.994 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:42.994 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.907155+0000", 2024-09-13T11:23:42.994 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.85688824399999997 2024-09-13T11:23:42.994 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.994 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.995 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:23:42.995 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.907163+0000", 2024-09-13T11:23:42.995 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.1759999999999993e-06 2024-09-13T11:23:42.995 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:42.995 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:42.995 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:42.995 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.995 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.995 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:54818 3.1e 3.5cfcc69e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-13T11:23:42.995 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:32.051374+0000", 2024-09-13T11:23:42.995 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 10.879853596, 2024-09-13T11:23:42.995 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.85581781000000001, 2024-09-13T11:23:42.995 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:23:42.996 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:23:42.996 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-13T11:23:42.996 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:23:42.996 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:23:42.996 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:42.996 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 54818 2024-09-13T11:23:42.996 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.996 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:23:42.996 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.996 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:23:42.996 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.051374+0000", 2024-09-13T11:23:42.996 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:42.996 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.997 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.997 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:23:42.997 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.051374+0000", 2024-09-13T11:23:42.997 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:42.997 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.997 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.997 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:23:42.997 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.051375+0000", 2024-09-13T11:23:42.997 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.4440000000000001e-06 2024-09-13T11:23:42.997 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.997 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.997 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:23:42.998 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.051426+0000", 2024-09-13T11:23:42.998 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.0809000000000001e-05 2024-09-13T11:23:42.998 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:42.998 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:42.998 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:23:42.998 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.051441+0000", 2024-09-13T11:23:42.998 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.4460999999999999e-05 2024-09-13T11:23:43.000 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:43.000 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.000 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:23:43.000 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.051449+0000", 2024-09-13T11:23:43.000 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.4880000000000007e-06 2024-09-13T11:23:43.000 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:43.000 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.000 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:43.001 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.051499+0000", 2024-09-13T11:23:43.001 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.9691999999999998e-05 2024-09-13T11:23:43.001 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:43.001 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.001 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for peered", 2024-09-13T11:23:43.001 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.051505+0000", 2024-09-13T11:23:43.001 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.2029999999999997e-06 2024-09-13T11:23:43.001 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:43.001 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.001 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:43.001 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.907184+0000", 2024-09-13T11:23:43.001 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.85567906599999999 2024-09-13T11:23:43.001 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:43.002 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.002 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:23:43.002 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.907192+0000", 2024-09-13T11:23:43.002 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.6469999999999998e-06 2024-09-13T11:23:43.002 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:43.002 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:43.002 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:43.002 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:43.002 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.002 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:54828 3.1e 3.e920c11e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-13T11:23:43.002 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:32.063344+0000", 2024-09-13T11:23:43.002 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 10.867883169000001, 2024-09-13T11:23:43.002 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.84387899099999997, 2024-09-13T11:23:43.003 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:23:43.003 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:23:43.003 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-13T11:23:43.003 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:23:43.003 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:23:43.003 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:43.003 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 54828 2024-09-13T11:23:43.003 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:43.003 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:23:43.003 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.003 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:23:43.003 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.063344+0000", 2024-09-13T11:23:43.003 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:43.004 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:43.004 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.004 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:23:43.004 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.063344+0000", 2024-09-13T11:23:43.004 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:43.004 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:43.004 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.004 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:23:43.004 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.063347+0000", 2024-09-13T11:23:43.004 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.2299999999999998e-06 2024-09-13T11:23:43.004 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:43.004 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.004 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:23:43.005 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.063384+0000", 2024-09-13T11:23:43.005 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.7290999999999999e-05 2024-09-13T11:23:43.005 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:43.005 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.005 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:23:43.005 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.063397+0000", 2024-09-13T11:23:43.005 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2955999999999999e-05 2024-09-13T11:23:43.005 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:43.005 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.005 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:23:43.005 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.063404+0000", 2024-09-13T11:23:43.005 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.4889999999999998e-06 2024-09-13T11:23:43.005 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:43.006 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.006 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:43.006 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.063470+0000", 2024-09-13T11:23:43.006 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.5516000000000001e-05 2024-09-13T11:23:43.006 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:43.006 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.006 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for peered", 2024-09-13T11:23:43.006 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.063482+0000", 2024-09-13T11:23:43.006 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2065000000000001e-05 2024-09-13T11:23:43.006 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:43.006 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.006 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:43.006 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.907216+0000", 2024-09-13T11:23:43.006 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.84373388900000001 2024-09-13T11:23:43.007 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:43.007 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.007 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:23:43.007 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.907223+0000", 2024-09-13T11:23:43.007 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.5549999999999997e-06 2024-09-13T11:23:43.007 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:43.007 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:43.007 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:43.007 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:43.007 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.007 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:54829 3.1e 3.192f62be (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-13T11:23:43.007 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:32.065642+0000", 2024-09-13T11:23:43.007 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 10.865586101, 2024-09-13T11:23:43.008 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.84161483100000001, 2024-09-13T11:23:43.008 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:23:43.008 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:23:43.008 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-13T11:23:43.008 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:23:43.008 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:23:43.008 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:43.008 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 54829 2024-09-13T11:23:43.008 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:43.008 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:23:43.008 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.008 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:23:43.008 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.065642+0000", 2024-09-13T11:23:43.009 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:43.009 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:43.009 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.009 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:23:43.009 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.065642+0000", 2024-09-13T11:23:43.009 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:43.009 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:43.009 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.009 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:23:43.009 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.065644+0000", 2024-09-13T11:23:43.009 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.1780000000000002e-06 2024-09-13T11:23:43.009 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:43.009 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.010 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:23:43.010 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.065695+0000", 2024-09-13T11:23:43.010 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.1632000000000003e-05 2024-09-13T11:23:43.010 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:43.010 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.010 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:23:43.010 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.065711+0000", 2024-09-13T11:23:43.010 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.5522000000000001e-05 2024-09-13T11:23:43.010 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:43.010 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.010 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:23:43.010 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.065719+0000", 2024-09-13T11:23:43.010 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.3480000000000004e-06 2024-09-13T11:23:43.011 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:43.011 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.011 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:43.011 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.065785+0000", 2024-09-13T11:23:43.011 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.5627e-05 2024-09-13T11:23:43.011 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:43.011 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.011 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for peered", 2024-09-13T11:23:43.011 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.065797+0000", 2024-09-13T11:23:43.011 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.1766000000000001e-05 2024-09-13T11:23:43.011 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:43.011 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.011 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:43.011 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.907248+0000", 2024-09-13T11:23:43.012 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.84145150300000005 2024-09-13T11:23:43.012 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:43.012 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.012 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:23:43.012 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.907256+0000", 2024-09-13T11:23:43.012 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.2549999999999997e-06 2024-09-13T11:23:43.012 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:43.012 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:43.012 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:43.012 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:43.012 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.012 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:54830 3.1e 3.b31554de (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-13T11:23:43.012 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:32.068259+0000", 2024-09-13T11:23:43.013 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 10.862968233, 2024-09-13T11:23:43.013 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.83902489599999996, 2024-09-13T11:23:43.013 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:23:43.013 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:23:43.013 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-13T11:23:43.013 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:23:43.013 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:23:43.013 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:43.013 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 54830 2024-09-13T11:23:43.013 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:43.013 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:23:43.013 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.013 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:23:43.014 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.068259+0000", 2024-09-13T11:23:43.014 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:43.014 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:43.014 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.014 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:23:43.014 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.068259+0000", 2024-09-13T11:23:43.014 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:43.014 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:43.014 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.014 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:23:43.014 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.068262+0000", 2024-09-13T11:23:43.014 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.1799999999999999e-06 2024-09-13T11:23:43.014 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:43.015 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.015 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:23:43.015 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.068301+0000", 2024-09-13T11:23:43.015 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.9289999999999998e-05 2024-09-13T11:23:43.015 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:43.015 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.015 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:23:43.015 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.068314+0000", 2024-09-13T11:23:43.015 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.3502999999999999e-05 2024-09-13T11:23:43.015 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:43.015 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.015 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:23:43.015 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.068324+0000", 2024-09-13T11:23:43.016 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.1630000000000002e-06 2024-09-13T11:23:43.016 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:43.016 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.016 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:43.016 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.068375+0000", 2024-09-13T11:23:43.016 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.1805000000000003e-05 2024-09-13T11:23:43.016 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:43.016 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.016 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for peered", 2024-09-13T11:23:43.016 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.068382+0000", 2024-09-13T11:23:43.016 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.224e-06 2024-09-13T11:23:43.016 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:43.016 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.016 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:43.017 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.907277+0000", 2024-09-13T11:23:43.017 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.83889509500000004 2024-09-13T11:23:43.017 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:43.017 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.017 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:23:43.017 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.907284+0000", 2024-09-13T11:23:43.017 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.6359999999999994e-06 2024-09-13T11:23:43.017 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:43.017 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:43.017 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:43.017 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:43.017 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.017 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:54846 3.1e 3.742e049e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-13T11:23:43.018 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:32.153567+0000", 2024-09-13T11:23:43.018 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 10.777661008999999, 2024-09-13T11:23:43.018 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.75378642399999995, 2024-09-13T11:23:43.018 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:23:43.018 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:23:43.018 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-13T11:23:43.018 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:23:43.018 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:23:43.018 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:43.018 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 54846 2024-09-13T11:23:43.018 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:43.018 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:23:43.018 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.019 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:23:43.019 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.153567+0000", 2024-09-13T11:23:43.019 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:43.019 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:43.019 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.019 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:23:43.019 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.153567+0000", 2024-09-13T11:23:43.019 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:43.019 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:43.019 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.019 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:23:43.019 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.153569+0000", 2024-09-13T11:23:43.019 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.4660000000000002e-06 2024-09-13T11:23:43.019 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:43.020 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.020 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:23:43.020 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.153619+0000", 2024-09-13T11:23:43.020 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.9617000000000003e-05 2024-09-13T11:23:43.020 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:43.020 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.020 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:23:43.020 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.153634+0000", 2024-09-13T11:23:43.020 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.4904e-05 2024-09-13T11:23:43.020 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:43.020 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.020 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:23:43.020 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.153642+0000", 2024-09-13T11:23:43.021 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.5599999999999994e-06 2024-09-13T11:23:43.021 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:43.021 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.021 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:43.021 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.153721+0000", 2024-09-13T11:23:43.021 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.8603999999999995e-05 2024-09-13T11:23:43.021 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:43.021 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.021 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for peered", 2024-09-13T11:23:43.021 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.153730+0000", 2024-09-13T11:23:43.021 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.3840000000000003e-06 2024-09-13T11:23:43.021 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:43.021 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.021 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:43.022 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.907346+0000", 2024-09-13T11:23:43.022 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.75361563099999995 2024-09-13T11:23:43.022 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:43.022 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.022 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:23:43.022 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.907353+0000", 2024-09-13T11:23:43.022 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.2579999999999998e-06 2024-09-13T11:23:43.022 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:43.022 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:43.022 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:43.022 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:43.022 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.022 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:54888 3.1e 3.43e14ede (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-13T11:23:43.023 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:32.423064+0000", 2024-09-13T11:23:43.023 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 10.508163337999999, 2024-09-13T11:23:43.023 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.48433585299999998, 2024-09-13T11:23:43.023 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:23:43.023 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:23:43.023 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-13T11:23:43.023 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:23:43.023 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:23:43.023 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:43.023 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 54888 2024-09-13T11:23:43.023 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:43.023 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:23:43.023 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.023 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:23:43.024 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.423064+0000", 2024-09-13T11:23:43.024 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:43.024 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:43.024 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.024 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:23:43.024 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.423064+0000", 2024-09-13T11:23:43.024 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:43.024 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:43.024 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.024 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:23:43.024 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.423069+0000", 2024-09-13T11:23:43.024 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.7389999999999999e-06 2024-09-13T11:23:43.024 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:43.024 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.025 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:23:43.025 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.423115+0000", 2024-09-13T11:23:43.025 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.5847000000000003e-05 2024-09-13T11:23:43.025 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:43.025 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.025 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:23:43.025 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.423126+0000", 2024-09-13T11:23:43.025 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.1418e-05 2024-09-13T11:23:43.025 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:43.025 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.025 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:23:43.025 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.423132+0000", 2024-09-13T11:23:43.025 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.57e-06 2024-09-13T11:23:43.026 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:43.026 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.026 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:43.026 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.423191+0000", 2024-09-13T11:23:43.026 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.9250999999999999e-05 2024-09-13T11:23:43.026 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:43.026 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.026 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for peered", 2024-09-13T11:23:43.026 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.423198+0000", 2024-09-13T11:23:43.026 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.9179999999999997e-06 2024-09-13T11:23:43.026 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:43.026 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.026 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:43.027 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.907392+0000", 2024-09-13T11:23:43.027 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.48419437500000001 2024-09-13T11:23:43.027 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:43.027 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.027 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:23:43.027 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.907400+0000", 2024-09-13T11:23:43.027 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.7349999999999996e-06 2024-09-13T11:23:43.027 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:43.027 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:43.027 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:43.027 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:43.027 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.027 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDPGPushReply(3.17 76/70 [PushReplyOp(3:eb8d2fae:::benchmark_data_smithi177_31116_object18267:head)])", 2024-09-13T11:23:43.027 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:32.636274+0000", 2024-09-13T11:23:43.028 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 10.294953745000001, 2024-09-13T11:23:43.028 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.269081036, 2024-09-13T11:23:43.028 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:23:43.028 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:23:43.028 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-13T11:23:43.028 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:23:43.028 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.028 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:23:43.028 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.636274+0000", 2024-09-13T11:23:43.028 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:43.028 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:43.028 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.029 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:23:43.029 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.636274+0000", 2024-09-13T11:23:43.029 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:43.029 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:43.029 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.029 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:23:43.029 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.636276+0000", 2024-09-13T11:23:43.029 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.1679999999999998e-06 2024-09-13T11:23:43.029 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:43.029 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.029 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:23:43.029 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.636284+0000", 2024-09-13T11:23:43.029 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.7440000000000007e-06 2024-09-13T11:23:43.030 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:43.030 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.030 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:23:43.030 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.904155+0000", 2024-09-13T11:23:43.030 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.26787080800000002 2024-09-13T11:23:43.030 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:43.030 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.030 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:23:43.030 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.904165+0000", 2024-09-13T11:23:43.030 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0750999999999999e-05 2024-09-13T11:23:43.030 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:43.030 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.030 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:43.031 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.904238+0000", 2024-09-13T11:23:43.031 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.2335000000000005e-05 2024-09-13T11:23:43.031 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:43.031 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.031 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:23:43.031 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.905355+0000", 2024-09-13T11:23:43.031 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0011172300000000001 2024-09-13T11:23:43.031 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:43.031 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:43.031 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:43.031 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:43.031 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.031 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDPGPushReply(3.17 76/70 [PushReplyOp(3:ebcbce67:::benchmark_data_smithi177_31116_object18268:head)])", 2024-09-13T11:23:43.032 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:32.641093+0000", 2024-09-13T11:23:43.032 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 10.290134197, 2024-09-13T11:23:43.032 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.27471406799999998, 2024-09-13T11:23:43.032 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:23:43.032 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:23:43.032 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-13T11:23:43.032 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:23:43.032 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.032 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:23:43.032 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.641093+0000", 2024-09-13T11:23:43.032 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:43.032 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:43.032 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.032 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:23:43.033 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.641093+0000", 2024-09-13T11:23:43.033 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:43.033 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:43.033 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.033 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:23:43.033 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.641094+0000", 2024-09-13T11:23:43.033 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.0299999999999999e-07 2024-09-13T11:23:43.033 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:43.033 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.033 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:23:43.033 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.641098+0000", 2024-09-13T11:23:43.033 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.2200000000000003e-06 2024-09-13T11:23:43.033 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:43.034 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.034 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:23:43.034 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.910434+0000", 2024-09-13T11:23:43.034 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.26933570600000001 2024-09-13T11:23:43.034 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:43.034 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.034 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:23:43.034 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.910442+0000", 2024-09-13T11:23:43.034 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.2700000000000004e-06 2024-09-13T11:23:43.034 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:43.034 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.034 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:43.034 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.914989+0000", 2024-09-13T11:23:43.034 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0045465059999999996 2024-09-13T11:23:43.035 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:43.035 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.035 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:23:43.035 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.915808+0000", 2024-09-13T11:23:43.035 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00081876300000000002 2024-09-13T11:23:43.035 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:43.035 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:43.035 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:43.035 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:43.035 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.035 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4665.0:59977 3.21 e83/78 3:86b4d8f6:::benchmark_data_smithi177_31116_object59976:head v 83'3136, mlcod=0'0)", 2024-09-13T11:23:43.035 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:40.571147+0000", 2024-09-13T11:23:43.035 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 2.3600808889999998, 2024-09-13T11:23:43.035 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.51907542799999995, 2024-09-13T11:23:43.036 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:23:43.036 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:23:43.036 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:43.036 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:23:43.036 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.036 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:23:43.036 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:40.571147+0000", 2024-09-13T11:23:43.036 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:43.036 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:43.036 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.036 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:23:43.036 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:40.571147+0000", 2024-09-13T11:23:43.036 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:43.036 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:43.037 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.037 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:23:43.037 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:40.571151+0000", 2024-09-13T11:23:43.037 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.5650000000000003e-06 2024-09-13T11:23:43.037 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:43.037 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.037 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:23:43.037 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:40.571204+0000", 2024-09-13T11:23:43.037 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.2720999999999999e-05 2024-09-13T11:23:43.037 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:43.037 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.037 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:23:43.037 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:41.088332+0000", 2024-09-13T11:23:43.037 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.51712822000000003 2024-09-13T11:23:43.038 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:43.038 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.038 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:23:43.038 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:41.088341+0000", 2024-09-13T11:23:43.038 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.3100000000000001e-06 2024-09-13T11:23:43.038 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:43.038 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.038 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:43.038 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:41.088444+0000", 2024-09-13T11:23:43.038 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000103708 2024-09-13T11:23:43.038 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:43.038 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.038 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:23:43.039 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:41.088475+0000", 2024-09-13T11:23:43.039 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.0497e-05 2024-09-13T11:23:43.039 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:43.039 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.039 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:23:43.039 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:41.090192+0000", 2024-09-13T11:23:43.039 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.001717288 2024-09-13T11:23:43.039 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:43.039 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.039 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:23:43.039 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:41.090222+0000", 2024-09-13T11:23:43.039 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.0119000000000001e-05 2024-09-13T11:23:43.039 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:43.039 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:43.039 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:43.040 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:43.040 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:43.040 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-13T11:23:43.041 INFO:tasks.radosbench.radosbench.0.smithi177.stdout: 76 16 60529 60513 49.7587 17 0.0390154 0.0200769 2024-09-13T11:23:43.042 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 3 1 2024-09-13T11:23:43.043 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_ops_in_flight 2024-09-13T11:23:43.043 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-13T11:23:43.036+0000 7f6a68775640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26307) UID: 0 2024-09-13T11:23:43.096 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-13T11:23:43.141 INFO:tasks.ceph.osd.9.smithi092.stderr:2024-09-13T11:23:43.140+0000 7ff5ab3d0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26348) UID: 0 2024-09-13T11:23:43.242 INFO:tasks.ceph.osd.7.smithi177.stderr:2024-09-13T11:23:43.236+0000 7f897b51d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26575) UID: 0 2024-09-13T11:23:43.282 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-13T11:23:43.282 INFO:teuthology.orchestra.run.smithi001.stdout: "ops": [ 2024-09-13T11:23:43.282 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.282 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:60930 3.9 3:95c5439b:::benchmark_data_smithi177_31116_object60929:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86)", 2024-09-13T11:23:43.282 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:43.280444+0000", 2024-09-13T11:23:43.282 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 0.00057525700000000003, 2024-09-13T11:23:43.282 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00060726300000000003, 2024-09-13T11:23:43.282 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:23:43.282 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:23:43.282 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "waiting for sub ops", 2024-09-13T11:23:43.283 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:23:43.283 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:23:43.283 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:43.283 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 60930 2024-09-13T11:23:43.283 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:43.283 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:23:43.283 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.283 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:23:43.283 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:43.280444+0000", 2024-09-13T11:23:43.283 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:43.283 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:43.283 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.283 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:23:43.283 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:43.280444+0000", 2024-09-13T11:23:43.284 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:43.284 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:43.284 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.284 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:23:43.284 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:43.280446+0000", 2024-09-13T11:23:43.284 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.125e-06 2024-09-13T11:23:43.284 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:43.284 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.284 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:23:43.284 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:43.280478+0000", 2024-09-13T11:23:43.284 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.1899000000000001e-05 2024-09-13T11:23:43.284 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:43.284 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.285 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:23:43.285 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:43.280496+0000", 2024-09-13T11:23:43.285 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.8428999999999999e-05 2024-09-13T11:23:43.285 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:43.285 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.285 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:23:43.285 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:43.280504+0000", 2024-09-13T11:23:43.285 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.3149999999999992e-06 2024-09-13T11:23:43.285 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:43.285 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.285 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:43.285 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:43.280559+0000", 2024-09-13T11:23:43.285 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.4236e-05 2024-09-13T11:23:43.286 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:43.286 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.286 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:23:43.286 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:43.280665+0000", 2024-09-13T11:23:43.286 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00010630599999999999 2024-09-13T11:23:43.286 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:43.286 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.286 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 2", 2024-09-13T11:23:43.286 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:43.280752+0000", 2024-09-13T11:23:43.287 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.7094000000000003e-05 2024-09-13T11:23:43.287 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:43.287 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:43.287 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:43.287 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:43.287 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-09-13T11:23:43.287 INFO:teuthology.orchestra.run.smithi001.stdout: "num_ops": 1 2024-09-13T11:23:43.287 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-13T11:23:43.317 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_blocked_ops 2024-09-13T11:23:43.360 INFO:tasks.ceph.osd.4.smithi001.stderr:2024-09-13T11:23:43.356+0000 7ff8e0218640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26310) UID: 0 2024-09-13T11:23:43.366 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T11:23:43.444 INFO:tasks.ceph.osd.14.smithi162.stderr:2024-09-13T11:23:43.441+0000 7f235c0db640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26319) UID: 0 2024-09-13T11:23:43.503 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-13T11:23:43.503 INFO:teuthology.orchestra.run.smithi001.stdout: "ops": [], 2024-09-13T11:23:43.503 INFO:teuthology.orchestra.run.smithi001.stdout: "complaint_time": 30, 2024-09-13T11:23:43.503 INFO:teuthology.orchestra.run.smithi001.stdout: "num_blocked_ops": 0 2024-09-13T11:23:43.504 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-13T11:23:43.530 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_historic_ops 2024-09-13T11:23:43.545 INFO:tasks.ceph.osd.1.smithi092.stderr:2024-09-13T11:23:43.544+0000 7fe7789c5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33718) UID: 0 2024-09-13T11:23:43.646 INFO:tasks.ceph.osd.14.smithi162.stderr:2024-09-13T11:23:43.645+0000 7f235c0db640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26319) UID: 0 2024-09-13T11:23:43.667 INFO:teuthology.orchestra.run.smithi001.stdout:osd.0: {} 2024-09-13T11:23:43.668 INFO:teuthology.orchestra.run.smithi001.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T11:23:43.706 INFO:teuthology.orchestra.run.smithi001.stdout:osd.1: {} 2024-09-13T11:23:43.707 INFO:teuthology.orchestra.run.smithi001.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T11:23:43.708 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-13T11:23:43.709 INFO:teuthology.orchestra.run.smithi001.stdout: "size": 20, 2024-09-13T11:23:43.709 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 600, 2024-09-13T11:23:43.709 INFO:teuthology.orchestra.run.smithi001.stdout: "ops": [ 2024-09-13T11:23:43.709 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.709 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop_reply(client.4665.0:167 3.9 e41/30 ondisk, result = 0)", 2024-09-13T11:23:43.709 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:22:47.419696+0000", 2024-09-13T11:23:43.709 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 56.286994538000002, 2024-09-13T11:23:43.709 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.64615778199999996, 2024-09-13T11:23:43.709 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:23:43.709 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:23:43.710 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:23:43.710 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:23:43.710 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.710 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:23:43.710 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:47.419696+0000", 2024-09-13T11:23:43.710 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:43.710 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:43.710 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.710 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:23:43.710 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:47.419696+0000", 2024-09-13T11:23:43.710 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:43.710 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:43.711 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.711 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:23:43.711 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:47.419697+0000", 2024-09-13T11:23:43.711 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.15e-06 2024-09-13T11:23:43.711 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:43.711 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.711 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:23:43.711 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:47.419700+0000", 2024-09-13T11:23:43.711 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.2449999999999998e-06 2024-09-13T11:23:43.711 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:43.711 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.711 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:23:43.712 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:48.065637+0000", 2024-09-13T11:23:43.712 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.64593674300000004 2024-09-13T11:23:43.712 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:43.712 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.712 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:23:43.712 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:48.065648+0000", 2024-09-13T11:23:43.712 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0865e-05 2024-09-13T11:23:43.712 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:43.712 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.712 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:43.712 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:48.065720+0000", 2024-09-13T11:23:43.713 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.2020999999999994e-05 2024-09-13T11:23:43.713 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:43.713 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.713 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:23:43.713 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:48.065728+0000", 2024-09-13T11:23:43.713 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.7660000000000004e-06 2024-09-13T11:23:43.713 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:43.713 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.713 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:23:43.713 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:48.065853+0000", 2024-09-13T11:23:43.713 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00012499200000000001 2024-09-13T11:23:43.714 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:43.714 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:43.714 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:43.714 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:43.714 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.714 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:10697 3.0 3:08f06ac6:::benchmark_data_smithi177_31116_object10696:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e48)", 2024-09-13T11:23:43.714 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:22:54.806675+0000", 2024-09-13T11:23:43.714 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 48.900015519999997, 2024-09-13T11:23:43.714 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.102587161, 2024-09-13T11:23:43.714 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:23:43.714 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:23:43.714 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:43.715 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:23:43.715 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:23:43.715 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:43.715 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 10697 2024-09-13T11:23:43.715 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:43.715 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:23:43.715 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.715 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:23:43.715 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:54.806675+0000", 2024-09-13T11:23:43.715 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:43.715 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:43.716 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.716 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:23:43.716 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:54.806675+0000", 2024-09-13T11:23:43.716 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:43.716 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:43.716 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.716 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:23:43.716 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:54.806680+0000", 2024-09-13T11:23:43.716 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.2970000000000003e-06 2024-09-13T11:23:43.716 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:43.716 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.716 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:23:43.717 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:54.806731+0000", 2024-09-13T11:23:43.717 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.1048999999999999e-05 2024-09-13T11:23:43.717 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:43.717 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.717 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:23:43.717 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:54.806743+0000", 2024-09-13T11:23:43.717 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2264e-05 2024-09-13T11:23:43.717 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:43.717 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.717 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:23:43.717 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:54.806752+0000", 2024-09-13T11:23:43.717 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.3259999999999996e-06 2024-09-13T11:23:43.717 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:43.718 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.718 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:43.718 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:54.806812+0000", 2024-09-13T11:23:43.718 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.0433000000000002e-05 2024-09-13T11:23:43.718 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:43.718 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.718 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:23:43.718 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:54.806930+0000", 2024-09-13T11:23:43.718 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000117805 2024-09-13T11:23:43.718 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:43.719 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.719 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 15", 2024-09-13T11:23:43.719 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:54.807042+0000", 2024-09-13T11:23:43.719 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000112528 2024-09-13T11:23:43.719 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:43.719 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.719 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:23:43.719 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:54.807872+0000", 2024-09-13T11:23:43.719 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00082981600000000004 2024-09-13T11:23:43.719 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:43.719 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.719 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:43.719 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:54.908964+0000", 2024-09-13T11:23:43.720 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.101092056 2024-09-13T11:23:43.720 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:43.720 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.720 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:23:43.720 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:54.909137+0000", 2024-09-13T11:23:43.720 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00017322799999999999 2024-09-13T11:23:43.720 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:43.720 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.721 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:23:43.721 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:54.909262+0000", 2024-09-13T11:23:43.721 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00012435899999999999 2024-09-13T11:23:43.721 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:43.721 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:43.721 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:43.721 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:43.721 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.721 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop_reply(client.4665.0:10697 3.0 e48/30 ondisk, result = 0)", 2024-09-13T11:23:43.722 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:22:54.808193+0000", 2024-09-13T11:23:43.722 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 48.898497484000004, 2024-09-13T11:23:43.722 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.10108661100000001, 2024-09-13T11:23:43.722 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:23:43.722 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:23:43.722 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:23:43.722 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:23:43.722 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.722 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:23:43.722 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:54.808193+0000", 2024-09-13T11:23:43.722 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:43.722 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:43.723 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.723 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:23:43.723 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:54.808193+0000", 2024-09-13T11:23:43.723 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:43.723 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:43.723 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.723 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:23:43.723 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:54.808194+0000", 2024-09-13T11:23:43.723 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.392e-06 2024-09-13T11:23:43.723 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:43.723 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.724 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:23:43.724 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:54.808198+0000", 2024-09-13T11:23:43.724 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.9099999999999998e-06 2024-09-13T11:23:43.724 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:43.724 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.724 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:23:43.724 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:54.908728+0000", 2024-09-13T11:23:43.724 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.100530419 2024-09-13T11:23:43.724 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:43.724 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.725 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:23:43.725 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:54.908775+0000", 2024-09-13T11:23:43.725 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.6479e-05 2024-09-13T11:23:43.725 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:43.725 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.725 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:43.725 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:54.908920+0000", 2024-09-13T11:23:43.725 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00014498500000000001 2024-09-13T11:23:43.725 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:43.725 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.725 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:23:43.726 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:54.908944+0000", 2024-09-13T11:23:43.726 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.4493000000000001e-05 2024-09-13T11:23:43.726 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:43.726 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.726 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:23:43.726 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:54.909279+0000", 2024-09-13T11:23:43.726 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00033493299999999999 2024-09-13T11:23:43.726 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:43.726 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:43.726 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:43.726 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:43.727 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.727 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4665.0:25290 3.4 e55/40 3:2c6b3a0e:::benchmark_data_smithi177_31116_object25289:head v 55'1638, mlcod=55'1635)", 2024-09-13T11:23:43.727 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:04.309280+0000", 2024-09-13T11:23:43.727 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 39.397409809000003, 2024-09-13T11:23:43.727 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.094030580000000002, 2024-09-13T11:23:43.727 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:23:43.727 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:23:43.727 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:43.727 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:23:43.727 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.727 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:23:43.727 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.309280+0000", 2024-09-13T11:23:43.727 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:43.728 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:43.728 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.728 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:23:43.728 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.309280+0000", 2024-09-13T11:23:43.728 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:43.728 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:43.728 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.728 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:23:43.728 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.309283+0000", 2024-09-13T11:23:43.728 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.7800000000000001e-06 2024-09-13T11:23:43.728 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:43.728 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.729 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:23:43.729 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.309334+0000", 2024-09-13T11:23:43.729 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.0510000000000003e-05 2024-09-13T11:23:43.729 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:43.729 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.729 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:23:43.729 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.400659+0000", 2024-09-13T11:23:43.729 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.091325274999999997 2024-09-13T11:23:43.729 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:43.729 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.729 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:23:43.729 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.400676+0000", 2024-09-13T11:23:43.730 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.7537000000000002e-05 2024-09-13T11:23:43.730 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:43.730 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.730 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:43.730 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.400785+0000", 2024-09-13T11:23:43.730 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000108756 2024-09-13T11:23:43.730 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:43.730 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.730 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:23:43.730 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.400818+0000", 2024-09-13T11:23:43.730 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.2635999999999997e-05 2024-09-13T11:23:43.730 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:43.731 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.731 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:23:43.731 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.403208+0000", 2024-09-13T11:23:43.731 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0023897900000000001 2024-09-13T11:23:43.731 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:43.731 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.731 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:23:43.731 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.403311+0000", 2024-09-13T11:23:43.731 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00010329599999999999 2024-09-13T11:23:43.731 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:43.731 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:43.731 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:43.732 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:43.732 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.732 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:25383 3.0 3:0a6bb869:::benchmark_data_smithi177_31116_object25382:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e55)", 2024-09-13T11:23:43.732 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:04.447376+0000", 2024-09-13T11:23:43.732 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 39.259313869000003, 2024-09-13T11:23:43.732 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.21239864, 2024-09-13T11:23:43.732 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:23:43.732 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:23:43.732 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:43.732 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:23:43.732 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:23:43.733 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:43.733 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 25383 2024-09-13T11:23:43.733 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:43.733 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:23:43.733 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.733 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:23:43.733 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.447376+0000", 2024-09-13T11:23:43.733 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:43.733 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:43.733 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.733 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:23:43.734 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.447376+0000", 2024-09-13T11:23:43.734 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:43.734 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:43.734 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.734 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:23:43.734 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.447378+0000", 2024-09-13T11:23:43.734 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.6729999999999999e-06 2024-09-13T11:23:43.734 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:43.734 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.734 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:23:43.734 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.447433+0000", 2024-09-13T11:23:43.734 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.5371000000000002e-05 2024-09-13T11:23:43.734 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:43.735 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.735 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:23:43.735 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.447446+0000", 2024-09-13T11:23:43.735 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.3179999999999999e-05 2024-09-13T11:23:43.735 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:43.735 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.735 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:23:43.735 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.447456+0000", 2024-09-13T11:23:43.735 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0008e-05 2024-09-13T11:23:43.735 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:43.735 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.735 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:43.736 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.447514+0000", 2024-09-13T11:23:43.736 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.8023999999999999e-05 2024-09-13T11:23:43.736 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:43.736 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.736 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:23:43.736 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.447642+0000", 2024-09-13T11:23:43.736 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000127296 2024-09-13T11:23:43.736 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:43.736 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.736 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 15", 2024-09-13T11:23:43.736 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.447767+0000", 2024-09-13T11:23:43.736 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000125547 2024-09-13T11:23:43.737 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:43.737 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.737 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:23:43.737 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.449188+0000", 2024-09-13T11:23:43.737 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.001420604 2024-09-13T11:23:43.737 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:43.737 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.737 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:43.737 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.659668+0000", 2024-09-13T11:23:43.737 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.210479685 2024-09-13T11:23:43.737 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:43.738 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.738 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:23:43.738 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.659716+0000", 2024-09-13T11:23:43.738 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.8609999999999997e-05 2024-09-13T11:23:43.738 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:43.738 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.738 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:23:43.738 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.659775+0000", 2024-09-13T11:23:43.738 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.8641999999999999e-05 2024-09-13T11:23:43.738 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:43.738 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:43.738 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:43.739 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:43.739 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.739 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4665.0:25408 3.4 e56/40 3:294e23bd:::benchmark_data_smithi177_31116_object25407:head v 56'1645, mlcod=55'1642)", 2024-09-13T11:23:43.739 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:04.494686+0000", 2024-09-13T11:23:43.739 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 39.212004210000003, 2024-09-13T11:23:43.739 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.29725566399999998, 2024-09-13T11:23:43.739 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:23:43.739 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:23:43.739 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:43.739 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:23:43.739 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.739 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:23:43.740 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.494686+0000", 2024-09-13T11:23:43.740 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:43.740 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:43.740 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.740 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:23:43.740 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.494686+0000", 2024-09-13T11:23:43.740 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:43.740 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:43.740 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.740 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:23:43.741 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.494686+0000", 2024-09-13T11:23:43.741 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.2399999999999999e-07 2024-09-13T11:23:43.741 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:43.741 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.741 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:23:43.741 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.494809+0000", 2024-09-13T11:23:43.741 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00012308600000000001 2024-09-13T11:23:43.741 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:43.741 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.741 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:23:43.741 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.789410+0000", 2024-09-13T11:23:43.742 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.29460066899999998 2024-09-13T11:23:43.742 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:43.742 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.742 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:23:43.742 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.789425+0000", 2024-09-13T11:23:43.742 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.4600000000000001e-05 2024-09-13T11:23:43.742 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:43.742 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.742 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:43.742 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.789507+0000", 2024-09-13T11:23:43.742 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.2785000000000001e-05 2024-09-13T11:23:43.743 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:43.743 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.743 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:23:43.743 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.789525+0000", 2024-09-13T11:23:43.743 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.8117999999999998e-05 2024-09-13T11:23:43.743 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:43.743 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.743 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:23:43.743 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.791852+0000", 2024-09-13T11:23:43.743 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0023264459999999998 2024-09-13T11:23:43.743 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:43.744 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.744 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:23:43.744 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.791942+0000", 2024-09-13T11:23:43.744 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.9635999999999994e-05 2024-09-13T11:23:43.744 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:43.744 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:43.744 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:43.744 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:43.744 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.744 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4665.0:30815 3.f e59/30 3:fde17cb1:::benchmark_data_smithi177_31116_object30814:head v 59'1919, mlcod=59'1914)", 2024-09-13T11:23:43.744 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:09.494342+0000", 2024-09-13T11:23:43.745 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 34.212348224000003, 2024-09-13T11:23:43.745 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.22787855700000001, 2024-09-13T11:23:43.745 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:23:43.745 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:23:43.745 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:43.745 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:23:43.745 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.745 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:23:43.745 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.494342+0000", 2024-09-13T11:23:43.745 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:43.745 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:43.745 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.746 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:23:43.746 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.494342+0000", 2024-09-13T11:23:43.746 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:43.746 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:43.746 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.746 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:23:43.746 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.494343+0000", 2024-09-13T11:23:43.746 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.251e-06 2024-09-13T11:23:43.746 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:43.746 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.746 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:23:43.747 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.494387+0000", 2024-09-13T11:23:43.747 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.4276000000000001e-05 2024-09-13T11:23:43.747 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:43.747 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.747 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:23:43.747 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.720454+0000", 2024-09-13T11:23:43.747 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.226066614 2024-09-13T11:23:43.747 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:43.747 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.747 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:23:43.747 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.720462+0000", 2024-09-13T11:23:43.747 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.9240000000000007e-06 2024-09-13T11:23:43.748 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:43.748 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.748 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:43.748 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.720537+0000", 2024-09-13T11:23:43.748 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.4967000000000002e-05 2024-09-13T11:23:43.748 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:43.748 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.748 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:23:43.748 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.720563+0000", 2024-09-13T11:23:43.748 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.5630999999999999e-05 2024-09-13T11:23:43.748 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:43.749 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.749 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:23:43.749 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.722183+0000", 2024-09-13T11:23:43.749 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0016209670000000001 2024-09-13T11:23:43.749 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:43.749 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.749 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:23:43.749 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.722220+0000", 2024-09-13T11:23:43.749 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.6927000000000001e-05 2024-09-13T11:23:43.749 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:43.749 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:43.749 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:43.750 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:43.750 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.750 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4665.0:30828 3.f e59/30 3:f3a9c6a9:::benchmark_data_smithi177_31116_object30827:head v 59'1920, mlcod=59'1916)", 2024-09-13T11:23:43.750 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:09.502607+0000", 2024-09-13T11:23:43.750 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 34.204083167, 2024-09-13T11:23:43.750 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.26374408300000002, 2024-09-13T11:23:43.750 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:23:43.750 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:23:43.750 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:43.750 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:23:43.750 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.750 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:23:43.751 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.502607+0000", 2024-09-13T11:23:43.751 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:43.751 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:43.751 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.751 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:23:43.751 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.502607+0000", 2024-09-13T11:23:43.751 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:43.751 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:43.751 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.751 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:23:43.751 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.502609+0000", 2024-09-13T11:23:43.751 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.84e-06 2024-09-13T11:23:43.751 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:43.752 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.752 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:23:43.752 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.502663+0000", 2024-09-13T11:23:43.752 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.3971000000000002e-05 2024-09-13T11:23:43.752 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:43.752 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.752 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:23:43.752 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.763912+0000", 2024-09-13T11:23:43.752 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.26124904300000001 2024-09-13T11:23:43.752 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:43.752 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.752 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:23:43.753 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.763920+0000", 2024-09-13T11:23:43.753 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.1149999999999994e-06 2024-09-13T11:23:43.753 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:43.753 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.753 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:43.753 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.764024+0000", 2024-09-13T11:23:43.753 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00010380899999999999 2024-09-13T11:23:43.753 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:43.753 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.753 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:23:43.753 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.764068+0000", 2024-09-13T11:23:43.753 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.4638999999999997e-05 2024-09-13T11:23:43.753 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:43.754 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.754 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:23:43.754 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.766308+0000", 2024-09-13T11:23:43.754 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0022392039999999998 2024-09-13T11:23:43.754 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:43.754 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.754 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:23:43.754 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.766351+0000", 2024-09-13T11:23:43.754 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.3461999999999998e-05 2024-09-13T11:23:43.754 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:43.754 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:43.754 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:43.755 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:43.755 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.755 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:39894 3.9 3:9190c04c:::benchmark_data_smithi177_31116_object39893:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-13T11:23:43.755 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:15.746095+0000", 2024-09-13T11:23:43.755 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 27.960595542, 2024-09-13T11:23:43.755 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.129842069, 2024-09-13T11:23:43.755 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:23:43.755 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:23:43.755 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:43.755 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:23:43.755 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:23:43.755 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:43.756 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 39894 2024-09-13T11:23:43.756 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:43.756 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:23:43.756 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.756 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:23:43.756 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:15.746095+0000", 2024-09-13T11:23:43.756 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:43.756 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:43.756 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.756 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:23:43.756 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:15.746095+0000", 2024-09-13T11:23:43.756 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:43.757 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:43.757 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.757 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:23:43.757 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:15.746096+0000", 2024-09-13T11:23:43.757 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.9800000000000002e-07 2024-09-13T11:23:43.757 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:43.757 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.757 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:23:43.757 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:15.746154+0000", 2024-09-13T11:23:43.757 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.8734999999999999e-05 2024-09-13T11:23:43.757 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:43.757 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.757 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:23:43.758 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:15.746163+0000", 2024-09-13T11:23:43.758 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.6780000000000006e-06 2024-09-13T11:23:43.758 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:43.758 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.758 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:23:43.758 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:15.746168+0000", 2024-09-13T11:23:43.758 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.622e-06 2024-09-13T11:23:43.758 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:43.758 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.758 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:43.758 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:15.747109+0000", 2024-09-13T11:23:43.758 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00094147399999999998 2024-09-13T11:23:43.758 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:43.758 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.759 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:23:43.759 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:15.747196+0000", 2024-09-13T11:23:43.759 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.6537000000000002e-05 2024-09-13T11:23:43.759 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:43.759 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.759 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 2", 2024-09-13T11:23:43.759 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:15.747304+0000", 2024-09-13T11:23:43.759 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000108208 2024-09-13T11:23:43.759 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:43.759 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.759 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:23:43.759 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:15.748640+0000", 2024-09-13T11:23:43.760 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0013365339999999999 2024-09-13T11:23:43.760 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:43.760 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.760 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:43.760 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:15.875866+0000", 2024-09-13T11:23:43.760 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.12722560399999999 2024-09-13T11:23:43.760 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:43.760 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.760 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:23:43.760 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:15.875910+0000", 2024-09-13T11:23:43.760 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.4409000000000001e-05 2024-09-13T11:23:43.761 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:43.761 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.761 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:23:43.761 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:15.875937+0000", 2024-09-13T11:23:43.761 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.6270000000000001e-05 2024-09-13T11:23:43.761 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:43.761 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:43.761 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:43.761 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:43.761 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.761 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:39934 3.9 3:9dc03fbd:::benchmark_data_smithi177_31116_object39933:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-13T11:23:43.761 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:15.784769+0000", 2024-09-13T11:23:43.762 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 27.921921583, 2024-09-13T11:23:43.762 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.094845225000000005, 2024-09-13T11:23:43.762 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:23:43.762 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:23:43.762 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:43.762 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:23:43.762 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:23:43.762 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:43.762 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 39934 2024-09-13T11:23:43.762 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:43.762 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:23:43.762 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.762 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:23:43.763 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:15.784769+0000", 2024-09-13T11:23:43.763 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:43.763 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:43.763 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.763 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:23:43.763 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:15.784769+0000", 2024-09-13T11:23:43.763 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:43.763 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:43.763 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.763 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:23:43.763 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:15.784770+0000", 2024-09-13T11:23:43.763 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.9400000000000001e-06 2024-09-13T11:23:43.763 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:43.764 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.764 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:23:43.764 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:15.784802+0000", 2024-09-13T11:23:43.764 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.1721999999999999e-05 2024-09-13T11:23:43.764 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:43.764 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.764 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:23:43.764 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:15.784813+0000", 2024-09-13T11:23:43.764 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0603e-05 2024-09-13T11:23:43.764 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:43.764 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.765 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:23:43.765 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:15.784818+0000", 2024-09-13T11:23:43.765 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.5860000000000004e-06 2024-09-13T11:23:43.765 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:43.765 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.765 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:43.765 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:15.784865+0000", 2024-09-13T11:23:43.765 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.7023e-05 2024-09-13T11:23:43.765 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:43.765 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.765 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:23:43.766 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:15.784969+0000", 2024-09-13T11:23:43.766 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000103514 2024-09-13T11:23:43.766 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:43.766 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.766 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 2", 2024-09-13T11:23:43.766 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:15.785062+0000", 2024-09-13T11:23:43.766 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.2666999999999999e-05 2024-09-13T11:23:43.766 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:43.766 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.766 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:23:43.766 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:15.786446+0000", 2024-09-13T11:23:43.766 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.001384056 2024-09-13T11:23:43.766 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:43.767 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.767 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:43.767 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:15.879559+0000", 2024-09-13T11:23:43.767 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.093113589999999996 2024-09-13T11:23:43.767 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:43.767 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.767 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:23:43.767 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:15.879589+0000", 2024-09-13T11:23:43.767 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.9691000000000001e-05 2024-09-13T11:23:43.767 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:43.767 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.767 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:23:43.768 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:15.879614+0000", 2024-09-13T11:23:43.768 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.4833000000000001e-05 2024-09-13T11:23:43.768 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:43.768 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:43.768 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:43.768 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:43.768 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.768 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDPGPushReply(3.10 72/70 [PushReplyOp(3:081f1665:::benchmark_data_smithi177_31116_object7063:head)])", 2024-09-13T11:23:43.768 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:27.369169+0000", 2024-09-13T11:23:43.768 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 16.337521458000001, 2024-09-13T11:23:43.768 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.095842116000000005, 2024-09-13T11:23:43.769 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:23:43.769 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:23:43.769 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-13T11:23:43.769 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:23:43.769 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.769 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:23:43.769 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:27.369169+0000", 2024-09-13T11:23:43.769 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:43.769 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:43.769 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.769 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:23:43.769 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:27.369169+0000", 2024-09-13T11:23:43.770 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:43.770 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:43.770 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.770 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:23:43.770 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:27.369170+0000", 2024-09-13T11:23:43.770 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.5939999999999999e-06 2024-09-13T11:23:43.770 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:43.770 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.770 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:23:43.770 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:27.369176+0000", 2024-09-13T11:23:43.770 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.9920000000000004e-06 2024-09-13T11:23:43.770 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:43.771 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.771 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:23:43.771 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:27.463769+0000", 2024-09-13T11:23:43.771 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.094593079999999996 2024-09-13T11:23:43.771 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:43.771 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.771 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:23:43.771 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:27.463778+0000", 2024-09-13T11:23:43.771 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.303e-06 2024-09-13T11:23:43.771 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:43.771 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.771 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:43.771 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:27.463855+0000", 2024-09-13T11:23:43.771 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.7826999999999999e-05 2024-09-13T11:23:43.772 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:43.772 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.772 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:23:43.772 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:27.465011+0000", 2024-09-13T11:23:43.772 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00115532 2024-09-13T11:23:43.772 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:43.772 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:43.772 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:43.772 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:43.772 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.772 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:54351 3.10 3:085f283e:::benchmark_data_smithi177_31116_object54350:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e74)", 2024-09-13T11:23:43.773 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:30.599125+0000", 2024-09-13T11:23:43.773 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 13.107565462, 2024-09-13T11:23:43.773 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.167573261, 2024-09-13T11:23:43.773 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:23:43.773 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:23:43.773 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:43.773 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:23:43.773 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:23:43.773 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:43.773 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 54351 2024-09-13T11:23:43.773 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:43.773 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:23:43.773 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.774 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:23:43.774 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:30.599125+0000", 2024-09-13T11:23:43.774 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:43.774 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:43.774 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.774 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:23:43.774 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:30.599125+0000", 2024-09-13T11:23:43.774 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:43.774 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:43.774 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.774 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:23:43.774 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:30.599127+0000", 2024-09-13T11:23:43.775 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.847e-06 2024-09-13T11:23:43.775 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:43.775 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.775 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:23:43.775 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:30.599164+0000", 2024-09-13T11:23:43.775 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.6758999999999998e-05 2024-09-13T11:23:43.775 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:43.775 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.775 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:23:43.775 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:30.599184+0000", 2024-09-13T11:23:43.775 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.9928e-05 2024-09-13T11:23:43.775 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:43.775 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.776 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:23:43.776 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:30.599194+0000", 2024-09-13T11:23:43.776 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0035e-05 2024-09-13T11:23:43.776 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:43.776 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.776 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:43.776 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:30.599259+0000", 2024-09-13T11:23:43.776 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.5147000000000005e-05 2024-09-13T11:23:43.776 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:43.776 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.777 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:23:43.777 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:30.599381+0000", 2024-09-13T11:23:43.777 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000121876 2024-09-13T11:23:43.777 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:43.777 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.777 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 0,6", 2024-09-13T11:23:43.777 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:30.599496+0000", 2024-09-13T11:23:43.777 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000114307 2024-09-13T11:23:43.777 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:43.777 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.777 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:23:43.778 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:30.601094+0000", 2024-09-13T11:23:43.778 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.001598857 2024-09-13T11:23:43.778 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:43.778 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.778 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:43.778 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:30.601896+0000", 2024-09-13T11:23:43.778 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00080183800000000003 2024-09-13T11:23:43.778 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:43.778 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.778 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:43.778 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:30.766040+0000", 2024-09-13T11:23:43.778 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.16414414099999999 2024-09-13T11:23:43.779 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:43.779 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.779 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:23:43.779 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:30.766092+0000", 2024-09-13T11:23:43.779 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.1230000000000002e-05 2024-09-13T11:23:43.779 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:43.779 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.779 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:23:43.779 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:30.766698+0000", 2024-09-13T11:23:43.779 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00060629599999999998 2024-09-13T11:23:43.779 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:43.779 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:43.780 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:43.780 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:43.780 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.780 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4665.0:54570 3.f e74/30 3:ff206a5c:::benchmark_data_smithi177_31116_object54569:head v 74'3411, mlcod=74'3408)", 2024-09-13T11:23:43.780 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:30.829251+0000", 2024-09-13T11:23:43.780 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 12.877438733, 2024-09-13T11:23:43.780 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.12680533199999999, 2024-09-13T11:23:43.780 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:23:43.780 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:23:43.780 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:43.780 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:23:43.781 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.781 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:23:43.781 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:30.829251+0000", 2024-09-13T11:23:43.781 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:43.781 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:43.781 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.781 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:23:43.781 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:30.829251+0000", 2024-09-13T11:23:43.781 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:43.781 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:43.781 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.782 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:23:43.782 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:30.829253+0000", 2024-09-13T11:23:43.782 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.5629999999999999e-06 2024-09-13T11:23:43.782 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:43.782 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.782 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:23:43.782 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:30.829313+0000", 2024-09-13T11:23:43.784 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.9864000000000001e-05 2024-09-13T11:23:43.784 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:43.784 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.784 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:23:43.784 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:30.952404+0000", 2024-09-13T11:23:43.784 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.123091229 2024-09-13T11:23:43.784 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:43.784 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.784 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:23:43.784 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:30.952440+0000", 2024-09-13T11:23:43.785 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.6161999999999997e-05 2024-09-13T11:23:43.785 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:43.785 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.785 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:43.785 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:30.952544+0000", 2024-09-13T11:23:43.785 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000103742 2024-09-13T11:23:43.785 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:43.785 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.785 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:23:43.785 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:30.952566+0000", 2024-09-13T11:23:43.785 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.2481e-05 2024-09-13T11:23:43.785 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:43.785 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.786 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:23:43.786 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:30.955979+0000", 2024-09-13T11:23:43.786 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0034129999999999998 2024-09-13T11:23:43.786 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:43.786 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.786 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:23:43.786 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:30.956057+0000", 2024-09-13T11:23:43.786 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.7291000000000002e-05 2024-09-13T11:23:43.786 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:43.786 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:43.786 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:43.786 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:43.786 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.787 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4665.0:54637 3.2 e74/67)", 2024-09-13T11:23:43.787 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:30.962888+0000", 2024-09-13T11:23:43.787 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 12.743802408000001, 2024-09-13T11:23:43.787 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.099275611, 2024-09-13T11:23:43.787 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:23:43.787 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:23:43.787 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-13T11:23:43.787 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:23:43.787 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.787 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:23:43.787 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:30.962888+0000", 2024-09-13T11:23:43.787 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:43.787 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:43.787 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.788 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:23:43.788 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:30.962888+0000", 2024-09-13T11:23:43.788 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:43.788 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:43.788 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.788 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:23:43.788 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:30.962891+0000", 2024-09-13T11:23:43.788 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.6969999999999999e-06 2024-09-13T11:23:43.788 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:43.788 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.788 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:23:43.788 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:30.962975+0000", 2024-09-13T11:23:43.788 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.4122000000000005e-05 2024-09-13T11:23:43.789 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:43.789 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.789 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:23:43.789 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.014360+0000", 2024-09-13T11:23:43.789 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.051384594999999998 2024-09-13T11:23:43.789 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:43.789 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.789 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:23:43.789 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.014372+0000", 2024-09-13T11:23:43.789 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.1549000000000001e-05 2024-09-13T11:23:43.789 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:43.789 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.789 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:43.789 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.062139+0000", 2024-09-13T11:23:43.790 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.047767782000000002 2024-09-13T11:23:43.790 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:43.790 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.790 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:23:43.790 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.062163+0000", 2024-09-13T11:23:43.790 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.3866000000000002e-05 2024-09-13T11:23:43.790 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:43.790 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:43.790 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:43.790 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:43.790 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.790 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4665.0:54651 3.f e74/30 3:f1b76062:::benchmark_data_smithi177_31116_object54650:head v 74'3413, mlcod=74'3410)", 2024-09-13T11:23:43.790 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:30.978338+0000", 2024-09-13T11:23:43.790 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 12.728352355, 2024-09-13T11:23:43.791 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.11551019999999999, 2024-09-13T11:23:43.791 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:23:43.791 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:23:43.791 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:23:43.791 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:23:43.791 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.791 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:23:43.791 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:30.978338+0000", 2024-09-13T11:23:43.791 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:43.791 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:43.791 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.791 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:23:43.791 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:30.978338+0000", 2024-09-13T11:23:43.792 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:43.792 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:43.792 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.792 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:23:43.792 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:30.978342+0000", 2024-09-13T11:23:43.792 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.89e-06 2024-09-13T11:23:43.792 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:43.792 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.792 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:23:43.792 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:30.978428+0000", 2024-09-13T11:23:43.792 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.6235000000000004e-05 2024-09-13T11:23:43.792 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:43.792 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.793 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:23:43.793 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:30.998757+0000", 2024-09-13T11:23:43.793 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.020329122000000002 2024-09-13T11:23:43.793 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:43.793 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.793 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:23:43.793 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:30.998776+0000", 2024-09-13T11:23:43.793 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.9378e-05 2024-09-13T11:23:43.793 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:43.793 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.793 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:43.793 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:30.998894+0000", 2024-09-13T11:23:43.793 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00011770899999999999 2024-09-13T11:23:43.793 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:43.794 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.794 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:23:43.794 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:30.998931+0000", 2024-09-13T11:23:43.794 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.7101000000000003e-05 2024-09-13T11:23:43.794 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:43.794 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.794 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:23:43.794 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.093848+0000", 2024-09-13T11:23:43.794 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.094916765 2024-09-13T11:23:43.794 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:43.795 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:43.795 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:43.795 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:43.795 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.795 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDPGPushReply(3.10 77/70 [PushReplyOp(3:0b779822:::benchmark_data_smithi177_31116_object18436:head)])", 2024-09-13T11:23:43.795 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:32.995348+0000", 2024-09-13T11:23:43.795 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 10.71134249, 2024-09-13T11:23:43.795 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.12346877000000001, 2024-09-13T11:23:43.795 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:23:43.795 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:23:43.795 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-13T11:23:43.795 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:23:43.795 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.796 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:23:43.796 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.995348+0000", 2024-09-13T11:23:43.796 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:43.796 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:43.796 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.796 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:23:43.796 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.995348+0000", 2024-09-13T11:23:43.796 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:43.796 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:43.796 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.796 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:23:43.796 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.995349+0000", 2024-09-13T11:23:43.796 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.5e-06 2024-09-13T11:23:43.797 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:43.797 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.797 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:23:43.797 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.995355+0000", 2024-09-13T11:23:43.797 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.4480000000000002e-06 2024-09-13T11:23:43.797 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:43.797 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.797 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:23:43.797 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:33.117887+0000", 2024-09-13T11:23:43.797 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.122532616 2024-09-13T11:23:43.797 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:43.798 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.798 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:23:43.798 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:33.117898+0000", 2024-09-13T11:23:43.798 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0909e-05 2024-09-13T11:23:43.798 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:43.798 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.798 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:43.798 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:33.117972+0000", 2024-09-13T11:23:43.798 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.3683000000000006e-05 2024-09-13T11:23:43.798 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:43.798 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.798 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:23:43.798 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:33.118816+0000", 2024-09-13T11:23:43.798 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00084461400000000004 2024-09-13T11:23:43.799 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:43.799 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:43.799 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:43.799 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:43.799 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.799 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDPGPushReply(3.10 79/70 [PushReplyOp(3:0a2ed60b:::benchmark_data_smithi177_31116_object30965:head)])", 2024-09-13T11:23:43.799 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:36.399588+0000", 2024-09-13T11:23:43.799 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 7.3071025829999998, 2024-09-13T11:23:43.799 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.100187135, 2024-09-13T11:23:43.799 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:23:43.799 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:23:43.799 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-13T11:23:43.799 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:23:43.800 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.800 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:23:43.800 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:36.399588+0000", 2024-09-13T11:23:43.800 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:43.800 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:43.800 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.800 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:23:43.800 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:36.399588+0000", 2024-09-13T11:23:43.800 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:43.800 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:43.800 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.800 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:23:43.801 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:36.399589+0000", 2024-09-13T11:23:43.801 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.283e-06 2024-09-13T11:23:43.801 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:43.801 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.801 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:23:43.801 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:36.399594+0000", 2024-09-13T11:23:43.801 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.3299999999999998e-06 2024-09-13T11:23:43.801 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:43.801 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.801 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:23:43.801 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:36.499052+0000", 2024-09-13T11:23:43.801 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.099457397000000003 2024-09-13T11:23:43.801 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:43.801 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.802 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:23:43.802 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:36.499059+0000", 2024-09-13T11:23:43.802 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.0149999999999998e-06 2024-09-13T11:23:43.802 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:43.802 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.802 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:43.802 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:36.499125+0000", 2024-09-13T11:23:43.802 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.6251e-05 2024-09-13T11:23:43.802 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:43.802 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.802 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:23:43.802 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:36.499775+0000", 2024-09-13T11:23:43.802 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00064985900000000003 2024-09-13T11:23:43.803 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:43.803 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:43.803 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:43.803 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:43.803 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.803 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDPGPush(3.1b 84/78 [PushOp(3:d9f9d9fe:::benchmark_data_smithi177_31116_object10323:head, version: 48'670, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:d9f9d9fe:::benchmark_data_smithi177_31116_object10323:head@48'670, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:23:43.803 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:41.277355+0000", 2024-09-13T11:23:43.803 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 2.429334817, 2024-09-13T11:23:43.803 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.094370475999999995, 2024-09-13T11:23:43.803 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:23:43.803 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:23:43.803 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:23:43.803 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:23:43.804 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.804 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:23:43.804 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:41.277355+0000", 2024-09-13T11:23:43.804 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:43.804 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:43.804 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.804 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:23:43.804 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:41.277355+0000", 2024-09-13T11:23:43.804 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:43.804 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:43.804 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.804 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:23:43.804 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:41.277357+0000", 2024-09-13T11:23:43.804 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.9599999999999999e-06 2024-09-13T11:23:43.805 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:43.805 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.805 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:23:43.805 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:41.277436+0000", 2024-09-13T11:23:43.805 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.8584999999999994e-05 2024-09-13T11:23:43.805 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:43.805 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.805 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:23:43.805 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:41.370667+0000", 2024-09-13T11:23:43.805 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.093230775000000002 2024-09-13T11:23:43.805 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:43.805 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.806 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:23:43.806 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:41.370686+0000", 2024-09-13T11:23:43.806 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.9015e-05 2024-09-13T11:23:43.806 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:43.806 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.806 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:43.806 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:41.370802+0000", 2024-09-13T11:23:43.806 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00011681299999999999 2024-09-13T11:23:43.806 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:43.806 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.806 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:23:43.806 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:41.370815+0000", 2024-09-13T11:23:43.807 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2670000000000001e-05 2024-09-13T11:23:43.807 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:43.807 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.807 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:23:43.807 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:41.371726+0000", 2024-09-13T11:23:43.807 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00091065799999999995 2024-09-13T11:23:43.807 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:43.807 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:43.807 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:43.807 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:43.807 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.808 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDPGPush(3.1b 84/78 [PushOp(3:dcddd139:::benchmark_data_smithi177_31116_object10384:head, version: 48'673, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:dcddd139:::benchmark_data_smithi177_31116_object10384:head@48'673, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:23:43.808 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:41.400602+0000", 2024-09-13T11:23:43.808 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 2.3060876820000002, 2024-09-13T11:23:43.808 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.43365821999999998, 2024-09-13T11:23:43.808 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:23:43.808 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:23:43.808 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:23:43.808 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:23:43.808 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.808 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:23:43.808 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:41.400602+0000", 2024-09-13T11:23:43.809 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:43.809 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:43.809 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.809 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:23:43.809 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:41.400602+0000", 2024-09-13T11:23:43.809 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:43.809 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:43.809 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.809 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:23:43.809 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:41.400604+0000", 2024-09-13T11:23:43.809 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.99e-06 2024-09-13T11:23:43.810 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:43.810 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.810 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:23:43.810 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:41.400653+0000", 2024-09-13T11:23:43.810 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.8195000000000002e-05 2024-09-13T11:23:43.810 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:43.810 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.810 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:23:43.810 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:41.833180+0000", 2024-09-13T11:23:43.810 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.43252769200000002 2024-09-13T11:23:43.810 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:43.811 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.811 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:23:43.811 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:41.833213+0000", 2024-09-13T11:23:43.811 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.2737999999999997e-05 2024-09-13T11:23:43.811 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:43.811 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.811 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:43.811 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:41.833408+0000", 2024-09-13T11:23:43.811 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00019473799999999999 2024-09-13T11:23:43.811 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:43.811 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.812 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:23:43.812 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:41.833417+0000", 2024-09-13T11:23:43.812 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.5349999999999993e-06 2024-09-13T11:23:43.812 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:43.812 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.812 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:23:43.812 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:41.834261+0000", 2024-09-13T11:23:43.812 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00084333200000000002 2024-09-13T11:23:43.812 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:43.812 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:43.813 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:43.813 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:43.813 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.813 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDPGPushReply(3.20 85/77 [PushReplyOp(3:066982a6:::benchmark_data_smithi177_31116_object16400:head)])", 2024-09-13T11:23:43.813 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:42.671551+0000", 2024-09-13T11:23:43.813 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 1.0351388589999999, 2024-09-13T11:23:43.813 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.28249085499999999, 2024-09-13T11:23:43.813 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:23:43.813 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:23:43.813 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-13T11:23:43.813 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:23:43.814 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.814 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:23:43.814 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.671551+0000", 2024-09-13T11:23:43.814 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:43.814 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:43.814 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.814 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:23:43.814 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.671551+0000", 2024-09-13T11:23:43.814 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:43.814 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:43.815 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.815 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:23:43.815 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.671552+0000", 2024-09-13T11:23:43.815 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.097e-06 2024-09-13T11:23:43.815 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:43.815 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.815 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:23:43.815 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.671556+0000", 2024-09-13T11:23:43.815 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.8519999999999997e-06 2024-09-13T11:23:43.815 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:43.816 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.816 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:23:43.816 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.951769+0000", 2024-09-13T11:23:43.816 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.28021293000000003 2024-09-13T11:23:43.816 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:43.816 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.816 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:23:43.816 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.951780+0000", 2024-09-13T11:23:43.816 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.1059e-05 2024-09-13T11:23:43.816 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:43.816 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.817 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:43.817 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.951992+0000", 2024-09-13T11:23:43.817 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000211478 2024-09-13T11:23:43.817 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:43.817 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:43.817 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:23:43.817 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.954042+0000", 2024-09-13T11:23:43.817 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0020504389999999998 2024-09-13T11:23:43.817 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:43.817 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:43.817 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:43.817 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:43.818 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:43.818 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-13T11:23:43.821 INFO:tasks.ceph.osd.10.smithi162.stderr:2024-09-13T11:23:43.817+0000 7fb59c9f0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26317) UID: 0 2024-09-13T11:23:43.823 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-13T11:23:43.823 INFO:teuthology.orchestra.run.smithi001.stdout:osd.2: {} 2024-09-13T11:23:43.824 INFO:teuthology.orchestra.run.smithi001.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T11:23:43.824 INFO:teuthology.orchestra.run.smithi001.stdout:osd.3: {} 2024-09-13T11:23:43.825 INFO:teuthology.orchestra.run.smithi001.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T11:23:43.825 INFO:teuthology.orchestra.run.smithi001.stdout:osd.4: {} 2024-09-13T11:23:43.825 INFO:teuthology.orchestra.run.smithi001.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T11:23:43.843 INFO:teuthology.orchestra.run.smithi001.stdout:osd.5: {} 2024-09-13T11:23:43.843 INFO:teuthology.orchestra.run.smithi001.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T11:23:43.877 INFO:teuthology.orchestra.run.smithi001.stdout:osd.6: {} 2024-09-13T11:23:43.877 INFO:teuthology.orchestra.run.smithi001.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T11:23:43.907 INFO:teuthology.orchestra.run.smithi001.stdout:osd.7: {} 2024-09-13T11:23:43.908 INFO:teuthology.orchestra.run.smithi001.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T11:23:43.921 INFO:tasks.ceph.osd.11.smithi177.stderr:2024-09-13T11:23:43.916+0000 7f24792c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26580) UID: 0 2024-09-13T11:23:43.935 INFO:teuthology.orchestra.run.smithi001.stdout:osd.8: {} 2024-09-13T11:23:43.935 INFO:teuthology.orchestra.run.smithi001.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-13T11:23:43.968 INFO:teuthology.orchestra.run.smithi001.stdout:osd.9: {} 2024-09-13T11:23:43.968 INFO:teuthology.orchestra.run.smithi001.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-13T11:23:43.977 INFO:teuthology.orchestra.run.smithi092.stdout:ERROR: (22) Invalid argument 2024-09-13T11:23:43.977 INFO:teuthology.orchestra.run.smithi092.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:23:43.997 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-13T11:23:44.002 INFO:teuthology.orchestra.run.smithi001.stdout:osd.10: {} 2024-09-13T11:23:44.002 INFO:teuthology.orchestra.run.smithi001.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-13T11:23:44.005 INFO:tasks.radosbench.radosbench.0.smithi177.stdout: 77 16 61748 61732 50.1018 76.1875 0.00958934 0.019952 2024-09-13T11:23:44.023 INFO:tasks.ceph.osd.15.smithi177.stderr:2024-09-13T11:23:44.016+0000 7f9d6ce77640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26578) UID: 0 2024-09-13T11:23:44.032 INFO:teuthology.orchestra.run.smithi001.stdout:osd.11: {} 2024-09-13T11:23:44.032 INFO:teuthology.orchestra.run.smithi001.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-13T11:23:44.063 INFO:teuthology.orchestra.run.smithi001.stdout:osd.12: {} 2024-09-13T11:23:44.063 INFO:teuthology.orchestra.run.smithi001.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-13T11:23:44.079 INFO:tasks.ceph.osd.11.smithi177.stderr:2024-09-13T11:23:44.072+0000 7f2469cec640 -1 osd.11 86 handle_osd_map injecting map crc failure 2024-09-13T11:23:44.081 INFO:teuthology.orchestra.run.smithi001.stderr:reweighted osd.3 to 1 (10000) 2024-09-13T11:23:44.081 INFO:teuthology.orchestra.run.smithi001.stderr:noscrub is unset 2024-09-13T11:23:44.094 INFO:teuthology.orchestra.run.smithi001.stdout:osd.13: {} 2024-09-13T11:23:44.095 INFO:teuthology.orchestra.run.smithi001.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-13T11:23:44.123 INFO:tasks.ceph.osd.1.smithi092.stderr:2024-09-13T11:23:44.120+0000 7fe7789c5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33718) UID: 0 2024-09-13T11:23:44.125 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 4 1 2024-09-13T11:23:44.137 INFO:teuthology.orchestra.run.smithi001.stdout:osd.14: {} 2024-09-13T11:23:44.137 INFO:teuthology.orchestra.run.smithi001.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-13T11:23:44.149 INFO:teuthology.orchestra.run.smithi092.stdout:ERROR: (22) Invalid argument 2024-09-13T11:23:44.149 INFO:teuthology.orchestra.run.smithi092.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:23:44.169 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-13T11:23:44.177 INFO:teuthology.orchestra.run.smithi001.stdout:osd.15: {} 2024-09-13T11:23:44.177 INFO:teuthology.orchestra.run.smithi001.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-13T11:23:44.261 INFO:tasks.ceph.osd.13.smithi092.stderr:2024-09-13T11:23:44.260+0000 7f3e5afea640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26350) UID: 0 2024-09-13T11:23:44.324 INFO:tasks.ceph.osd.2.smithi162.stderr:2024-09-13T11:23:44.321+0000 7fe92b35a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26313) UID: 0 2024-09-13T11:23:44.369 INFO:teuthology.orchestra.run.smithi092.stdout:ERROR: (22) Invalid argument 2024-09-13T11:23:44.369 INFO:teuthology.orchestra.run.smithi092.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:23:44.392 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-09-13T11:23:44.425 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-13T11:23:44.420+0000 7f6a68775640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26307) UID: 0 2024-09-13T11:23:44.526 INFO:tasks.ceph.osd.15.smithi177.stderr:2024-09-13T11:23:44.520+0000 7f9d6ce77640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26578) UID: 0 2024-09-13T11:23:44.585 INFO:teuthology.orchestra.run.smithi092.stdout:ERROR: (22) Invalid argument 2024-09-13T11:23:44.586 INFO:teuthology.orchestra.run.smithi092.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:23:44.605 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-09-13T11:23:44.627 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-13T11:23:44.620+0000 7f6a68775640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26307) UID: 0 2024-09-13T11:23:44.728 INFO:tasks.ceph.osd.7.smithi177.stderr:2024-09-13T11:23:44.724+0000 7f897b51d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26575) UID: 0 2024-09-13T11:23:44.747 INFO:teuthology.orchestra.run.smithi092.stdout:ERROR: (22) Invalid argument 2024-09-13T11:23:44.747 INFO:teuthology.orchestra.run.smithi092.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:23:44.765 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-09-13T11:23:44.829 INFO:tasks.ceph.osd.10.smithi162.stderr:2024-09-13T11:23:44.825+0000 7fb59c9f0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26317) UID: 0 2024-09-13T11:23:44.906 INFO:teuthology.orchestra.run.smithi092.stdout:ERROR: (22) Invalid argument 2024-09-13T11:23:44.906 INFO:teuthology.orchestra.run.smithi092.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:23:44.926 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_ops_in_flight 2024-09-13T11:23:44.930 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-13T11:23:44.924+0000 7f6a68775640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26307) UID: 0 2024-09-13T11:23:45.006 INFO:tasks.radosbench.radosbench.0.smithi177.stdout: 78 16 63134 63118 50.57 86.625 0.00784099 0.0197661 2024-09-13T11:23:45.031 INFO:tasks.ceph.osd.7.smithi177.stderr:2024-09-13T11:23:45.024+0000 7f897b51d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26575) UID: 0 2024-09-13T11:23:45.061 INFO:teuthology.orchestra.run.smithi092.stdout:ERROR: (22) Invalid argument 2024-09-13T11:23:45.061 INFO:teuthology.orchestra.run.smithi092.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:23:45.076 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_blocked_ops 2024-09-13T11:23:45.089 INFO:teuthology.orchestra.run.smithi001.stderr:reweighted osd.4 to 1 (10000) 2024-09-13T11:23:45.090 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-09-13T11:23:45.089+0000 7f97b59fa640 -1 osd.6 87 handle_osd_map injecting map crc failure 2024-09-13T11:23:45.114 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 5 1 2024-09-13T11:23:45.132 INFO:tasks.ceph.osd.10.smithi162.stderr:2024-09-13T11:23:45.129+0000 7fb59c9f0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26317) UID: 0 2024-09-13T11:23:45.216 INFO:teuthology.orchestra.run.smithi092.stdout:ERROR: (22) Invalid argument 2024-09-13T11:23:45.216 INFO:teuthology.orchestra.run.smithi092.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:23:45.231 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_historic_ops 2024-09-13T11:23:45.232 INFO:tasks.ceph.osd.8.smithi001.stderr:2024-09-13T11:23:45.228+0000 7fb9b48ae640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26312) UID: 0 2024-09-13T11:23:45.334 INFO:tasks.ceph.osd.4.smithi001.stderr:2024-09-13T11:23:45.328+0000 7ff8e0218640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26310) UID: 0 2024-09-13T11:23:45.361 INFO:teuthology.orchestra.run.smithi092.stdout:ERROR: (22) Invalid argument 2024-09-13T11:23:45.361 INFO:teuthology.orchestra.run.smithi092.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:23:45.374 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-13T11:23:45.434 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-09-13T11:23:45.433+0000 7f97befc9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26315) UID: 0 2024-09-13T11:23:45.522 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-13T11:23:45.523 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:23:45.535 INFO:tasks.ceph.osd.2.smithi162.stderr:2024-09-13T11:23:45.533+0000 7fe92b35a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26313) UID: 0 2024-09-13T11:23:45.540 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-13T11:23:45.635 INFO:tasks.ceph.osd.8.smithi001.stderr:2024-09-13T11:23:45.628+0000 7fb9b48ae640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26312) UID: 0 2024-09-13T11:23:45.717 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-13T11:23:45.717 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:23:45.733 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-13T11:23:45.736 INFO:tasks.ceph.osd.9.smithi092.stderr:2024-09-13T11:23:45.732+0000 7ff5ab3d0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26348) UID: 0 2024-09-13T11:23:45.836 INFO:tasks.ceph.osd.7.smithi177.stderr:2024-09-13T11:23:45.832+0000 7f897b51d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26575) UID: 0 2024-09-13T11:23:45.865 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-13T11:23:45.866 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:23:45.881 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-13T11:23:45.937 INFO:tasks.ceph.osd.10.smithi162.stderr:2024-09-13T11:23:45.933+0000 7fb59c9f0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26317) UID: 0 2024-09-13T11:23:46.006 INFO:tasks.radosbench.radosbench.0.smithi177.stdout: 79 16 63592 63576 50.2922 28.625 0.0159093 0.0197708 2024-09-13T11:23:46.011 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-13T11:23:46.012 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:23:46.026 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-13T11:23:46.038 INFO:tasks.ceph.osd.4.smithi001.stderr:2024-09-13T11:23:46.032+0000 7ff8e0218640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26310) UID: 0 2024-09-13T11:23:46.100 INFO:teuthology.orchestra.run.smithi001.stderr:reweighted osd.5 to 1 (10000) 2024-09-13T11:23:46.111 INFO:tasks.ceph.osd.3.smithi177.stderr:2024-09-13T11:23:46.104+0000 7f10a25d1640 -1 osd.3 88 handle_osd_map injecting map crc failure 2024-09-13T11:23:46.118 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-13T11:23:46.139 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 6 1 2024-09-13T11:23:46.162 INFO:tasks.ceph.osd.9.smithi092.stderr:2024-09-13T11:23:46.156+0000 7ff59ddf9640 -1 osd.9 88 handle_osd_map injecting map crc failure 2024-09-13T11:23:46.180 INFO:tasks.ceph.osd.8.smithi001.stderr:2024-09-13T11:23:46.176+0000 7fb9b48ae640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26312) UID: 0 2024-09-13T11:23:46.210 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-13T11:23:46.210 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:23:46.214 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T11:23:46.229 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-13T11:23:46.241 INFO:tasks.ceph.osd.11.smithi177.stderr:2024-09-13T11:23:46.236+0000 7f24792c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26580) UID: 0 2024-09-13T11:23:46.343 INFO:tasks.ceph.osd.9.smithi092.stderr:2024-09-13T11:23:46.340+0000 7ff5ab3d0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26348) UID: 0 2024-09-13T11:23:46.376 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-13T11:23:46.376 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:23:46.395 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-09-13T11:23:46.443 INFO:tasks.ceph.osd.9.smithi092.stderr:2024-09-13T11:23:46.440+0000 7ff5ab3d0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26348) UID: 0 2024-09-13T11:23:46.451 INFO:teuthology.orchestra.run.smithi001.stdout:osd.0: {} 2024-09-13T11:23:46.451 INFO:teuthology.orchestra.run.smithi001.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T11:23:46.488 INFO:teuthology.orchestra.run.smithi001.stdout:osd.1: {} 2024-09-13T11:23:46.489 INFO:teuthology.orchestra.run.smithi001.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T11:23:46.514 INFO:teuthology.orchestra.run.smithi001.stdout:osd.2: {} 2024-09-13T11:23:46.514 INFO:teuthology.orchestra.run.smithi001.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T11:23:46.527 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-13T11:23:46.527 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:23:46.542 INFO:teuthology.orchestra.run.smithi001.stdout:osd.3: {} 2024-09-13T11:23:46.542 INFO:teuthology.orchestra.run.smithi001.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T11:23:46.545 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-09-13T11:23:46.546 INFO:tasks.ceph.osd.5.smithi092.stderr:2024-09-13T11:23:46.540+0000 7f6f0b705640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26346) UID: 0 2024-09-13T11:23:46.567 INFO:teuthology.orchestra.run.smithi001.stdout:osd.4: {} 2024-09-13T11:23:46.567 INFO:teuthology.orchestra.run.smithi001.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T11:23:46.596 INFO:teuthology.orchestra.run.smithi001.stdout:osd.5: {} 2024-09-13T11:23:46.596 INFO:teuthology.orchestra.run.smithi001.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T11:23:46.621 INFO:teuthology.orchestra.run.smithi001.stdout:osd.6: {} 2024-09-13T11:23:46.622 INFO:teuthology.orchestra.run.smithi001.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T11:23:46.644 INFO:teuthology.orchestra.run.smithi001.stdout:osd.7: {} 2024-09-13T11:23:46.645 INFO:teuthology.orchestra.run.smithi001.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T11:23:46.645 INFO:tasks.ceph.osd.12.smithi001.stderr:2024-09-13T11:23:46.640+0000 7f892c6e6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26314) UID: 0 2024-09-13T11:23:46.667 INFO:teuthology.orchestra.run.smithi001.stdout:osd.8: {} 2024-09-13T11:23:46.667 INFO:teuthology.orchestra.run.smithi001.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-13T11:23:46.668 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-13T11:23:46.669 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:23:46.685 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-09-13T11:23:46.692 INFO:teuthology.orchestra.run.smithi001.stdout:osd.9: {} 2024-09-13T11:23:46.692 INFO:teuthology.orchestra.run.smithi001.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-13T11:23:46.715 INFO:teuthology.orchestra.run.smithi001.stdout:osd.10: {} 2024-09-13T11:23:46.715 INFO:teuthology.orchestra.run.smithi001.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-13T11:23:46.736 INFO:teuthology.orchestra.run.smithi001.stdout:osd.11: {} 2024-09-13T11:23:46.737 INFO:teuthology.orchestra.run.smithi001.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-13T11:23:46.745 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-13T11:23:46.740+0000 7f6a68775640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26307) UID: 0 2024-09-13T11:23:46.757 INFO:teuthology.orchestra.run.smithi001.stdout:osd.12: {} 2024-09-13T11:23:46.757 INFO:teuthology.orchestra.run.smithi001.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-13T11:23:46.781 INFO:teuthology.orchestra.run.smithi001.stdout:osd.13: {} 2024-09-13T11:23:46.782 INFO:teuthology.orchestra.run.smithi001.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-13T11:23:46.803 INFO:teuthology.orchestra.run.smithi001.stdout:osd.14: {} 2024-09-13T11:23:46.804 INFO:teuthology.orchestra.run.smithi001.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-13T11:23:46.805 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-13T11:23:46.805 INFO:teuthology.orchestra.run.smithi162.stdout: "size": 20, 2024-09-13T11:23:46.805 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 600, 2024-09-13T11:23:46.805 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [ 2024-09-13T11:23:46.806 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:46.806 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:25395 3.3 3:c1df8ccf:::benchmark_data_smithi177_31116_object25394:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e55)", 2024-09-13T11:23:46.806 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:04.465425+0000", 2024-09-13T11:23:46.806 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 42.335230809000002, 2024-09-13T11:23:46.806 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.18641877200000001, 2024-09-13T11:23:46.806 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:23:46.806 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:23:46.806 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:46.806 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:23:46.806 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:23:46.807 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:46.807 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 25395 2024-09-13T11:23:46.807 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:46.807 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:23:46.808 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:46.808 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:23:46.808 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.465425+0000", 2024-09-13T11:23:46.808 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:46.808 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:46.809 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:46.809 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:23:46.809 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.465425+0000", 2024-09-13T11:23:46.809 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:46.809 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:46.809 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:46.809 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:23:46.809 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.465427+0000", 2024-09-13T11:23:46.809 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.9970000000000001e-06 2024-09-13T11:23:46.809 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:46.809 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:46.809 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:23:46.810 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.465516+0000", 2024-09-13T11:23:46.810 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.8181000000000002e-05 2024-09-13T11:23:46.810 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:46.810 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:46.810 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:23:46.810 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.465534+0000", 2024-09-13T11:23:46.810 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.8204000000000001e-05 2024-09-13T11:23:46.810 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:46.810 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:46.811 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:23:46.811 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.465548+0000", 2024-09-13T11:23:46.811 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4195999999999999e-05 2024-09-13T11:23:46.811 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:46.811 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:46.811 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:23:46.812 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.468249+0000", 2024-09-13T11:23:46.812 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0027014729999999998 2024-09-13T11:23:46.812 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:46.812 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:46.812 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:23:46.812 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.468626+0000", 2024-09-13T11:23:46.812 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000376721 2024-09-13T11:23:46.812 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:46.812 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:46.812 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 7", 2024-09-13T11:23:46.812 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.469016+0000", 2024-09-13T11:23:46.812 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00038975000000000001 2024-09-13T11:23:46.812 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:46.813 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:46.813 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:23:46.813 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.471140+0000", 2024-09-13T11:23:46.813 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0021243400000000002 2024-09-13T11:23:46.816 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:46.816 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:46.816 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:46.816 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.651698+0000", 2024-09-13T11:23:46.817 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.18055781700000001 2024-09-13T11:23:46.817 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:46.817 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:46.817 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:23:46.817 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.651781+0000", 2024-09-13T11:23:46.817 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.3350999999999995e-05 2024-09-13T11:23:46.817 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:46.817 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:46.817 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:23:46.817 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.651844+0000", 2024-09-13T11:23:46.817 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.2742000000000003e-05 2024-09-13T11:23:46.817 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:46.817 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:23:46.818 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:46.818 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:46.818 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:46.818 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop_reply(client.4665.0:25395 3.3 e55/30 ondisk, result = 0)", 2024-09-13T11:23:46.818 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:04.471832+0000", 2024-09-13T11:23:46.818 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 42.328824507, 2024-09-13T11:23:46.818 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.18002319, 2024-09-13T11:23:46.818 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:23:46.818 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:23:46.819 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:23:46.819 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:23:46.819 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:46.819 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:23:46.819 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.471832+0000", 2024-09-13T11:23:46.819 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:46.819 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:46.819 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:46.819 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:23:46.819 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.471832+0000", 2024-09-13T11:23:46.819 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:46.819 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:46.819 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:46.820 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:23:46.820 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.471840+0000", 2024-09-13T11:23:46.820 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.7900000000000005e-06 2024-09-13T11:23:46.820 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:46.820 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:46.820 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:23:46.820 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.471850+0000", 2024-09-13T11:23:46.820 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.3899999999999999e-06 2024-09-13T11:23:46.821 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:46.821 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:46.821 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:23:46.821 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.651528+0000", 2024-09-13T11:23:46.821 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.17967789100000001 2024-09-13T11:23:46.821 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:46.821 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:46.821 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:23:46.821 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.651540+0000", 2024-09-13T11:23:46.821 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2e-05 2024-09-13T11:23:46.821 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:46.821 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:46.821 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:23:46.822 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.651676+0000", 2024-09-13T11:23:46.822 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000136033 2024-09-13T11:23:46.822 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:46.822 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:46.822 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:23:46.822 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.651689+0000", 2024-09-13T11:23:46.822 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.364e-05 2024-09-13T11:23:46.822 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:46.822 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:46.822 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:23:46.822 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.651855+0000", 2024-09-13T11:23:46.822 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000165446 2024-09-13T11:23:46.822 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:46.822 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:23:46.823 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:46.823 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:46.823 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:46.823 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:45810 3.17 3.9bd13e57 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e69)", 2024-09-13T11:23:46.823 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:21.347960+0000", 2024-09-13T11:23:46.823 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 25.452695855000002, 2024-09-13T11:23:46.823 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.56814464200000003, 2024-09-13T11:23:46.823 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:23:46.823 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:23:46.823 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-09-13T11:23:46.823 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:23:46.823 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:23:46.823 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:46.824 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 45810 2024-09-13T11:23:46.824 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:46.824 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:23:46.824 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:46.824 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:23:46.824 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:21.347960+0000", 2024-09-13T11:23:46.824 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:46.824 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:46.824 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:46.824 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:23:46.824 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:21.347960+0000", 2024-09-13T11:23:46.824 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:46.825 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:46.825 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:46.825 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:23:46.825 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:21.347965+0000", 2024-09-13T11:23:46.825 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.6720000000000003e-06 2024-09-13T11:23:46.825 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:46.825 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:46.825 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:23:46.825 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:21.348019+0000", 2024-09-13T11:23:46.825 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.4391000000000002e-05 2024-09-13T11:23:46.825 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:46.825 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:46.825 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:23:46.826 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:21.348052+0000", 2024-09-13T11:23:46.826 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.2246000000000003e-05 2024-09-13T11:23:46.826 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:46.826 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:46.826 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:23:46.826 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:21.348072+0000", 2024-09-13T11:23:46.826 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.0622e-05 2024-09-13T11:23:46.826 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:46.826 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:46.826 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:23:46.826 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:21.348278+0000", 2024-09-13T11:23:46.826 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000205391 2024-09-13T11:23:46.826 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:46.826 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:46.827 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-09-13T11:23:46.827 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:21.348292+0000", 2024-09-13T11:23:46.827 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4759e-05 2024-09-13T11:23:46.827 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:46.827 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:46.827 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:23:46.827 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:21.916064+0000", 2024-09-13T11:23:46.827 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.56777128600000004 2024-09-13T11:23:46.827 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:46.827 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:46.827 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:23:46.827 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:21.916105+0000", 2024-09-13T11:23:46.827 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.1275e-05 2024-09-13T11:23:46.828 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:46.828 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:23:46.828 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:46.828 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:46.828 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:46.828 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPush(3.17 71/70 [PushOp(3:ec82217b:::benchmark_data_smithi177_31116_object3121:head, version: 44'215, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:ec82217b:::benchmark_data_smithi177_31116_object3121:head@44'215, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:23:46.828 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:25.596519+0000", 2024-09-13T11:23:46.828 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 21.204137060000001, 2024-09-13T11:23:46.828 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.081111691, 2024-09-13T11:23:46.828 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:23:46.828 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:23:46.829 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:23:46.829 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:23:46.829 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:46.829 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:23:46.829 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.596519+0000", 2024-09-13T11:23:46.829 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:46.829 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:46.829 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:46.829 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:23:46.829 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.596519+0000", 2024-09-13T11:23:46.830 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:46.830 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:46.830 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:46.830 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:23:46.830 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.596526+0000", 2024-09-13T11:23:46.830 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.2169999999999997e-06 2024-09-13T11:23:46.830 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:46.830 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:46.830 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:23:46.831 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.596592+0000", 2024-09-13T11:23:46.831 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.5203999999999994e-05 2024-09-13T11:23:46.831 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:46.831 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:46.831 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:23:46.832 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.675751+0000", 2024-09-13T11:23:46.832 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.079159859999999999 2024-09-13T11:23:46.833 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:46.833 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:46.833 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:23:46.833 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.675781+0000", 2024-09-13T11:23:46.833 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.9662000000000002e-05 2024-09-13T11:23:46.833 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:46.833 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:46.833 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:23:46.833 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.676016+0000", 2024-09-13T11:23:46.833 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000235119 2024-09-13T11:23:46.834 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:46.834 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:46.834 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:23:46.834 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.676042+0000", 2024-09-13T11:23:46.834 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.5510000000000001e-05 2024-09-13T11:23:46.834 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:46.834 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:46.834 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:23:46.834 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.677631+0000", 2024-09-13T11:23:46.834 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0015891189999999999 2024-09-13T11:23:46.834 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:46.834 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:23:46.834 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:46.834 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:46.835 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:46.835 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPush(3.17 72/70 [PushOp(3:ecb2eb17:::benchmark_data_smithi177_31116_object4630:head, version: 44'317, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:ecb2eb17:::benchmark_data_smithi177_31116_object4630:head@44'317, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:23:46.835 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:26.434926+0000", 2024-09-13T11:23:46.835 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 20.365730352, 2024-09-13T11:23:46.835 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.155542491, 2024-09-13T11:23:46.835 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:23:46.835 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:23:46.835 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:23:46.835 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:23:46.835 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:46.835 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:23:46.835 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:26.434926+0000", 2024-09-13T11:23:46.835 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:46.836 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:46.836 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:46.836 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:23:46.836 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:26.434926+0000", 2024-09-13T11:23:46.836 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:46.836 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:46.836 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:46.836 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:23:46.836 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:26.434928+0000", 2024-09-13T11:23:46.836 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.7489999999999999e-06 2024-09-13T11:23:46.836 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:46.836 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:46.836 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:23:46.837 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:26.434983+0000", 2024-09-13T11:23:46.837 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.4988000000000003e-05 2024-09-13T11:23:46.837 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:46.837 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:46.837 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:23:46.837 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:26.589751+0000", 2024-09-13T11:23:46.837 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.15476837299999999 2024-09-13T11:23:46.837 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:46.837 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:46.837 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:23:46.837 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:26.589778+0000", 2024-09-13T11:23:46.837 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.7385e-05 2024-09-13T11:23:46.837 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:46.837 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:46.838 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:23:46.838 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:26.589891+0000", 2024-09-13T11:23:46.838 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000112849 2024-09-13T11:23:46.838 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:46.838 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:46.838 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:23:46.838 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:26.589900+0000", 2024-09-13T11:23:46.838 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.3440000000000007e-06 2024-09-13T11:23:46.838 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:46.838 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:46.838 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:23:46.838 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:26.590468+0000", 2024-09-13T11:23:46.838 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00056780299999999997 2024-09-13T11:23:46.838 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:46.839 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:23:46.839 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:46.839 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:46.839 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:46.839 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPush(3.17 72/70 [PushOp(3:ee101882:::benchmark_data_smithi177_31116_object5448:head, version: 45'353, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:ee101882:::benchmark_data_smithi177_31116_object5448:head@45'353, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:23:46.839 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:27.057731+0000", 2024-09-13T11:23:46.839 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 19.742924759000001, 2024-09-13T11:23:46.839 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.089951626000000007, 2024-09-13T11:23:46.839 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:23:46.839 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:23:46.839 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:23:46.839 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:23:46.839 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:46.839 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:23:46.840 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:27.057731+0000", 2024-09-13T11:23:46.840 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:46.840 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:46.840 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:46.840 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:23:46.840 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:27.057731+0000", 2024-09-13T11:23:46.840 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:46.840 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:46.840 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:46.840 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:23:46.841 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:27.057733+0000", 2024-09-13T11:23:46.841 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.978e-06 2024-09-13T11:23:46.841 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:46.841 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:46.841 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:23:46.841 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:27.057819+0000", 2024-09-13T11:23:46.841 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.5457999999999994e-05 2024-09-13T11:23:46.841 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:46.841 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:46.841 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:23:46.841 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:27.146807+0000", 2024-09-13T11:23:46.841 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.088988445999999999 2024-09-13T11:23:46.842 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:46.842 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:46.842 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:23:46.842 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:27.146820+0000", 2024-09-13T11:23:46.842 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3081000000000001e-05 2024-09-13T11:23:46.842 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:46.842 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:46.842 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:23:46.842 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:27.146900+0000", 2024-09-13T11:23:46.842 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.9251000000000004e-05 2024-09-13T11:23:46.842 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:46.842 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:46.842 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:23:46.843 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:27.146908+0000", 2024-09-13T11:23:46.843 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.0739999999999993e-06 2024-09-13T11:23:46.843 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:46.843 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:46.843 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:23:46.843 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:27.147683+0000", 2024-09-13T11:23:46.843 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00077533799999999998 2024-09-13T11:23:46.843 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:46.843 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:23:46.843 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:46.843 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:46.843 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:46.844 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.4665.0:50861 3.17 e72/70 3:ee280ac8:::benchmark_data_smithi177_31116_object50860:head v 72'2909, mlcod=45'348)", 2024-09-13T11:23:46.844 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:27.058825+0000", 2024-09-13T11:23:46.844 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 19.741831156, 2024-09-13T11:23:46.844 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.098196383999999998, 2024-09-13T11:23:46.844 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:23:46.844 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:23:46.844 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:46.844 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:23:46.844 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:46.844 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:23:46.844 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:27.058825+0000", 2024-09-13T11:23:46.844 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:46.845 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:46.845 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:46.845 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:23:46.845 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:27.058825+0000", 2024-09-13T11:23:46.845 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:46.845 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:46.845 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:46.845 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:23:46.845 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:27.058826+0000", 2024-09-13T11:23:46.845 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.079e-06 2024-09-13T11:23:46.845 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:46.845 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:46.846 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:23:46.846 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:27.058906+0000", 2024-09-13T11:23:46.846 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.9863000000000004e-05 2024-09-13T11:23:46.846 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:46.846 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:46.846 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:23:46.846 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:27.154716+0000", 2024-09-13T11:23:46.846 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.095809748 2024-09-13T11:23:46.846 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:46.846 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:46.846 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:23:46.846 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:27.154721+0000", 2024-09-13T11:23:46.847 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.5219999999999998e-06 2024-09-13T11:23:46.847 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:46.847 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:46.847 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:23:46.847 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:27.154781+0000", 2024-09-13T11:23:46.847 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.9732999999999999e-05 2024-09-13T11:23:46.847 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:46.847 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:46.847 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:23:46.847 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:27.154807+0000", 2024-09-13T11:23:46.847 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.6509e-05 2024-09-13T11:23:46.847 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:46.847 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:46.848 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:23:46.848 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:27.156976+0000", 2024-09-13T11:23:46.848 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0021687379999999999 2024-09-13T11:23:46.848 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:46.848 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:46.848 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:23:46.848 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:27.157021+0000", 2024-09-13T11:23:46.848 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.5191999999999997e-05 2024-09-13T11:23:46.848 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:46.848 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:23:46.848 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:46.848 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:46.849 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:46.849 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:54653 3.23 3:c7a54800:::benchmark_data_smithi177_31116_object54652:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e75)", 2024-09-13T11:23:46.849 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:30.979059+0000", 2024-09-13T11:23:46.849 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 15.821597356, 2024-09-13T11:23:46.849 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.97284747100000002, 2024-09-13T11:23:46.849 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:23:46.849 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:23:46.849 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-09-13T11:23:46.849 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:23:46.849 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:23:46.849 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:46.849 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 54653 2024-09-13T11:23:46.850 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:46.850 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:23:46.850 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:46.850 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:23:46.850 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.979059+0000", 2024-09-13T11:23:46.850 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:46.850 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:46.850 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:46.850 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:23:46.850 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.979059+0000", 2024-09-13T11:23:46.850 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:46.850 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:46.850 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:46.851 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:23:46.851 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.979061+0000", 2024-09-13T11:23:46.851 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.4279999999999999e-06 2024-09-13T11:23:46.851 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:46.851 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:46.851 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:23:46.851 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.979114+0000", 2024-09-13T11:23:46.851 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.3254000000000002e-05 2024-09-13T11:23:46.851 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:46.851 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:46.851 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:23:46.851 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.979157+0000", 2024-09-13T11:23:46.851 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.2018000000000003e-05 2024-09-13T11:23:46.851 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:46.852 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:46.852 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:23:46.852 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.979172+0000", 2024-09-13T11:23:46.852 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5636000000000001e-05 2024-09-13T11:23:46.852 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:46.852 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:46.852 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:23:46.852 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:31.045109+0000", 2024-09-13T11:23:46.852 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.065936982000000005 2024-09-13T11:23:46.852 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:46.852 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:46.852 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-09-13T11:23:46.852 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:31.045121+0000", 2024-09-13T11:23:46.853 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1608e-05 2024-09-13T11:23:46.853 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:46.853 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:46.853 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:23:46.853 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:31.951636+0000", 2024-09-13T11:23:46.853 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.90651503899999997 2024-09-13T11:23:46.853 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:46.853 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:46.853 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:23:46.853 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:31.951906+0000", 2024-09-13T11:23:46.853 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000270506 2024-09-13T11:23:46.853 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:46.853 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:23:46.853 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:46.854 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:46.854 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:46.854 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPush(3.18 76/70 [PushOp(3:1a32c9cd:::benchmark_data_smithi177_31116_object12833:head, version: 49'818, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:1a32c9cd:::benchmark_data_smithi177_31116_object12833:head@49'818, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:23:46.854 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:32.522305+0000", 2024-09-13T11:23:46.854 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 14.278351272, 2024-09-13T11:23:46.854 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.40201207100000003, 2024-09-13T11:23:46.854 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:23:46.854 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:23:46.854 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:23:46.854 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:23:46.854 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:46.854 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:23:46.854 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.522305+0000", 2024-09-13T11:23:46.855 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:46.855 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:46.855 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:46.855 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:23:46.855 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.522305+0000", 2024-09-13T11:23:46.855 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:46.855 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:46.855 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:46.855 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:23:46.855 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.522309+0000", 2024-09-13T11:23:46.855 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.9829999999999998e-06 2024-09-13T11:23:46.855 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:46.855 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:46.856 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:23:46.856 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.522377+0000", 2024-09-13T11:23:46.856 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.8069000000000002e-05 2024-09-13T11:23:46.856 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:46.856 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:46.856 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:23:46.856 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.921950+0000", 2024-09-13T11:23:46.856 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.399573438 2024-09-13T11:23:46.856 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:46.856 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:46.856 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:23:46.856 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.921983+0000", 2024-09-13T11:23:46.857 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.2858000000000003e-05 2024-09-13T11:23:46.857 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:46.857 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:46.857 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:23:46.857 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.922389+0000", 2024-09-13T11:23:46.857 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00040612999999999999 2024-09-13T11:23:46.857 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:46.857 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:46.857 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:23:46.857 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.922421+0000", 2024-09-13T11:23:46.857 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.1356999999999998e-05 2024-09-13T11:23:46.857 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:46.857 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:46.858 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:23:46.858 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.924317+0000", 2024-09-13T11:23:46.858 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0018962359999999999 2024-09-13T11:23:46.858 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:46.858 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:23:46.858 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:46.858 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:46.858 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:46.858 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPush(3.17 76/70 [PushOp(3:ebcbce67:::benchmark_data_smithi177_31116_object18268:head, version: 53'1102, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:ebcbce67:::benchmark_data_smithi177_31116_object18268:head@53'1102, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:23:46.858 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:32.638908+0000", 2024-09-13T11:23:46.858 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 14.161747824000001, 2024-09-13T11:23:46.858 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.229064031, 2024-09-13T11:23:46.859 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:23:46.859 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:23:46.859 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:23:46.859 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:23:46.859 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:46.859 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:23:46.859 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.638908+0000", 2024-09-13T11:23:46.859 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:46.859 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:46.859 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:46.859 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:23:46.859 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.638908+0000", 2024-09-13T11:23:46.859 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:46.860 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:46.860 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:46.860 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:23:46.860 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.638910+0000", 2024-09-13T11:23:46.860 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.7379999999999999e-06 2024-09-13T11:23:46.860 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:46.860 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:46.860 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:23:46.860 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.638975+0000", 2024-09-13T11:23:46.860 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.4937000000000005e-05 2024-09-13T11:23:46.860 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:46.861 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:46.861 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:23:46.861 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.865860+0000", 2024-09-13T11:23:46.861 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.22688509800000001 2024-09-13T11:23:46.861 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:46.861 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:46.861 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:23:46.861 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.865890+0000", 2024-09-13T11:23:46.861 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.0273e-05 2024-09-13T11:23:46.861 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:46.861 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:46.861 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:23:46.861 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.866143+0000", 2024-09-13T11:23:46.862 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00025212800000000001 2024-09-13T11:23:46.862 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:46.862 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:46.862 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:23:46.862 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.866181+0000", 2024-09-13T11:23:46.862 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.8945000000000001e-05 2024-09-13T11:23:46.862 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:46.862 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:46.862 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:23:46.862 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.867972+0000", 2024-09-13T11:23:46.862 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001790912 2024-09-13T11:23:46.862 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:46.862 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:23:46.862 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:46.863 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:46.863 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:46.863 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:57435 3.3 3:c0db4777:::benchmark_data_smithi177_31116_object57434:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e80)", 2024-09-13T11:23:46.863 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:37.436034+0000", 2024-09-13T11:23:46.863 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 9.3646223640000006, 2024-09-13T11:23:46.863 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.35949142000000001, 2024-09-13T11:23:46.863 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:23:46.863 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:23:46.863 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:46.863 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:23:46.863 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:23:46.863 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:46.863 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 57435 2024-09-13T11:23:46.864 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:46.864 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:23:46.864 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:46.864 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:23:46.864 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:37.436034+0000", 2024-09-13T11:23:46.864 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:46.864 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:46.864 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:46.864 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:23:46.864 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:37.436034+0000", 2024-09-13T11:23:46.864 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:46.864 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:46.864 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:46.865 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:23:46.865 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:37.436037+0000", 2024-09-13T11:23:46.865 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.7829999999999999e-06 2024-09-13T11:23:46.865 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:46.865 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:46.865 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:23:46.865 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:37.436090+0000", 2024-09-13T11:23:46.865 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.3665999999999997e-05 2024-09-13T11:23:46.865 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:46.865 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:46.865 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:23:46.865 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:37.436112+0000", 2024-09-13T11:23:46.865 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.1556000000000002e-05 2024-09-13T11:23:46.866 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:46.866 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:46.866 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:23:46.866 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:37.436122+0000", 2024-09-13T11:23:46.866 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0098000000000001e-05 2024-09-13T11:23:46.866 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:46.866 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:46.866 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:23:46.866 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:37.436187+0000", 2024-09-13T11:23:46.866 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.4857999999999995e-05 2024-09-13T11:23:46.866 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:46.866 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:46.866 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:23:46.866 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:37.436285+0000", 2024-09-13T11:23:46.867 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.8643999999999999e-05 2024-09-13T11:23:46.867 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:46.867 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:46.867 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 7", 2024-09-13T11:23:46.867 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:37.436423+0000", 2024-09-13T11:23:46.867 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00013809299999999999 2024-09-13T11:23:46.867 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:46.867 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:46.867 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:23:46.867 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:37.440125+0000", 2024-09-13T11:23:46.867 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0037015239999999999 2024-09-13T11:23:46.867 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:46.867 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:46.868 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:46.868 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:37.795397+0000", 2024-09-13T11:23:46.868 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.35527234400000002 2024-09-13T11:23:46.868 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:46.868 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:46.868 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:23:46.868 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:37.795485+0000", 2024-09-13T11:23:46.868 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.7538000000000001e-05 2024-09-13T11:23:46.868 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:46.868 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:46.868 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:23:46.868 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:37.795525+0000", 2024-09-13T11:23:46.868 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.0315999999999998e-05 2024-09-13T11:23:46.868 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:46.869 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:23:46.869 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:46.869 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:46.869 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:46.869 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPush(3.1f 80/78 [PushOp(3:fc3ed0c7:::benchmark_data_smithi177_31116_object672:head, version: 43'36, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:fc3ed0c7:::benchmark_data_smithi177_31116_object672:head@43'36, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:23:46.869 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:37.801754+0000", 2024-09-13T11:23:46.869 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 8.9989020039999996, 2024-09-13T11:23:46.869 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.209613519, 2024-09-13T11:23:46.869 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:23:46.869 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:23:46.869 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:23:46.869 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:23:46.869 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:46.870 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:23:46.870 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:37.801754+0000", 2024-09-13T11:23:46.870 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:46.870 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:46.870 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:46.870 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:23:46.870 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:37.801754+0000", 2024-09-13T11:23:46.870 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:46.870 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:46.870 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:46.870 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:23:46.871 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:37.801756+0000", 2024-09-13T11:23:46.871 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.804e-06 2024-09-13T11:23:46.871 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:46.871 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:46.871 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:23:46.871 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:37.801947+0000", 2024-09-13T11:23:46.871 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000190636 2024-09-13T11:23:46.871 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:46.871 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:46.871 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:23:46.871 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:38.010478+0000", 2024-09-13T11:23:46.871 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.20853091100000001 2024-09-13T11:23:46.871 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:46.872 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:46.872 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:23:46.872 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:38.010506+0000", 2024-09-13T11:23:46.872 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.8793999999999999e-05 2024-09-13T11:23:46.872 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:46.872 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:46.872 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:23:46.872 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:38.010601+0000", 2024-09-13T11:23:46.872 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.4786999999999999e-05 2024-09-13T11:23:46.872 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:46.872 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:46.872 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:23:46.872 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:38.010617+0000", 2024-09-13T11:23:46.873 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5790999999999998e-05 2024-09-13T11:23:46.873 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:46.873 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:46.873 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:23:46.873 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:38.011368+0000", 2024-09-13T11:23:46.873 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00075079599999999997 2024-09-13T11:23:46.873 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:46.873 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:23:46.873 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:46.873 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:46.873 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:46.873 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:57882 3.3 3:c17a2efd:::benchmark_data_smithi177_31116_object57881:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e80)", 2024-09-13T11:23:46.874 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:37.902902+0000", 2024-09-13T11:23:46.874 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 8.8977546049999994, 2024-09-13T11:23:46.874 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.099182874000000004, 2024-09-13T11:23:46.874 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:23:46.874 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:23:46.874 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:46.874 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:23:46.874 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:23:46.874 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:46.874 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 57882 2024-09-13T11:23:46.874 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:46.874 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:23:46.874 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:46.874 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:23:46.875 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:37.902902+0000", 2024-09-13T11:23:46.875 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:46.875 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:46.875 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:46.875 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:23:46.875 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:37.902902+0000", 2024-09-13T11:23:46.875 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:46.875 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:46.875 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:46.875 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:23:46.875 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:37.902904+0000", 2024-09-13T11:23:46.875 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.6709999999999999e-06 2024-09-13T11:23:46.875 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:46.875 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:46.875 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:23:46.876 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:37.902980+0000", 2024-09-13T11:23:46.876 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.5363000000000003e-05 2024-09-13T11:23:46.876 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:46.876 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:46.876 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:23:46.876 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:37.902993+0000", 2024-09-13T11:23:46.876 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3365e-05 2024-09-13T11:23:46.876 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:46.876 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:46.876 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:23:46.876 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:37.903001+0000", 2024-09-13T11:23:46.876 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.2430000000000005e-06 2024-09-13T11:23:46.876 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:46.877 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:46.877 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:23:46.877 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:37.903071+0000", 2024-09-13T11:23:46.877 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.9521000000000001e-05 2024-09-13T11:23:46.877 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:46.877 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:46.877 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:23:46.877 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:37.903224+0000", 2024-09-13T11:23:46.877 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00015312999999999999 2024-09-13T11:23:46.877 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:46.877 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:46.877 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 7", 2024-09-13T11:23:46.878 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:37.903324+0000", 2024-09-13T11:23:46.878 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00010022900000000001 2024-09-13T11:23:46.878 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:46.878 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:46.878 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:23:46.878 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:37.904286+0000", 2024-09-13T11:23:46.878 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00096145400000000004 2024-09-13T11:23:46.878 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:46.878 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:46.878 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:46.878 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:38.002002+0000", 2024-09-13T11:23:46.878 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.097716234999999999 2024-09-13T11:23:46.878 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:46.878 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:46.879 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:23:46.879 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:38.002040+0000", 2024-09-13T11:23:46.879 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.7957000000000003e-05 2024-09-13T11:23:46.879 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:46.879 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:46.879 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:23:46.879 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:38.002084+0000", 2024-09-13T11:23:46.879 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.4706000000000002e-05 2024-09-13T11:23:46.879 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:46.879 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:23:46.879 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:46.879 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:46.879 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:46.879 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:57985 3.3 3:c330bb03:::benchmark_data_smithi177_31116_object57984:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e80)", 2024-09-13T11:23:46.880 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:38.006112+0000", 2024-09-13T11:23:46.880 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 8.7945441360000007, 2024-09-13T11:23:46.880 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.18277887900000001, 2024-09-13T11:23:46.880 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:23:46.880 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:23:46.880 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:46.880 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:23:46.880 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:23:46.880 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:46.880 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 57985 2024-09-13T11:23:46.880 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:46.880 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:23:46.880 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:46.881 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:23:46.881 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:38.006112+0000", 2024-09-13T11:23:46.881 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:46.881 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:46.881 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:46.881 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:23:46.881 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:38.006112+0000", 2024-09-13T11:23:46.881 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:46.881 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:46.881 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:46.881 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:23:46.881 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:38.006115+0000", 2024-09-13T11:23:46.881 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.6400000000000001e-06 2024-09-13T11:23:46.882 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:46.882 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:46.882 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:23:46.882 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:38.006162+0000", 2024-09-13T11:23:46.882 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.7599000000000003e-05 2024-09-13T11:23:46.882 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:46.882 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:46.882 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:23:46.882 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:38.006182+0000", 2024-09-13T11:23:46.882 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.9281999999999999e-05 2024-09-13T11:23:46.882 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:46.882 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:46.882 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:23:46.883 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:38.006191+0000", 2024-09-13T11:23:46.883 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.6490000000000003e-06 2024-09-13T11:23:46.883 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:46.883 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:46.883 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:23:46.883 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:38.006243+0000", 2024-09-13T11:23:46.883 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.1511999999999997e-05 2024-09-13T11:23:46.883 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:46.883 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:46.883 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:23:46.883 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:38.006333+0000", 2024-09-13T11:23:46.883 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.9843e-05 2024-09-13T11:23:46.884 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:46.884 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:46.884 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 7", 2024-09-13T11:23:46.884 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:38.006480+0000", 2024-09-13T11:23:46.884 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00014701699999999999 2024-09-13T11:23:46.884 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:46.884 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:46.884 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:23:46.884 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:38.008806+0000", 2024-09-13T11:23:46.884 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.002326172 2024-09-13T11:23:46.884 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:46.885 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:46.885 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:46.885 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:38.188815+0000", 2024-09-13T11:23:46.885 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.18000961100000001 2024-09-13T11:23:46.885 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:46.885 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:46.885 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:23:46.885 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:38.188857+0000", 2024-09-13T11:23:46.885 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.1591999999999998e-05 2024-09-13T11:23:46.885 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:46.885 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:46.885 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:23:46.886 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:38.188891+0000", 2024-09-13T11:23:46.886 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.3961999999999998e-05 2024-09-13T11:23:46.886 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:46.886 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:23:46.886 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:46.886 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:46.886 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:46.886 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:60433 3.3 3:c01f7ddc:::benchmark_data_smithi177_31116_object60432:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e85)", 2024-09-13T11:23:46.886 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:42.199382+0000", 2024-09-13T11:23:46.886 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 4.6012744730000001, 2024-09-13T11:23:46.886 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.33918965200000001, 2024-09-13T11:23:46.886 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:23:46.886 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:23:46.887 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:46.887 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:23:46.887 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:23:46.887 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:46.887 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 60433 2024-09-13T11:23:46.887 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:46.887 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:23:46.887 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:46.887 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:23:46.887 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.199382+0000", 2024-09-13T11:23:46.887 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:46.887 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:46.887 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:46.888 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:23:46.888 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.199382+0000", 2024-09-13T11:23:46.888 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:46.888 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:46.888 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:46.888 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:23:46.888 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.199386+0000", 2024-09-13T11:23:46.888 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.8759999999999998e-06 2024-09-13T11:23:46.888 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:46.888 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:46.888 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:23:46.888 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.199455+0000", 2024-09-13T11:23:46.889 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.9305000000000001e-05 2024-09-13T11:23:46.889 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:46.889 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:46.889 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:23:46.889 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.199482+0000", 2024-09-13T11:23:46.889 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.6749000000000001e-05 2024-09-13T11:23:46.889 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:46.889 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:46.889 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:23:46.889 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.199503+0000", 2024-09-13T11:23:46.889 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.1137000000000001e-05 2024-09-13T11:23:46.889 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:46.890 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:46.890 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:23:46.890 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.199602+0000", 2024-09-13T11:23:46.890 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.8743e-05 2024-09-13T11:23:46.890 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:46.890 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:46.890 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:23:46.890 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.199784+0000", 2024-09-13T11:23:46.890 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00018265100000000001 2024-09-13T11:23:46.890 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:46.890 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:46.890 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 7", 2024-09-13T11:23:46.890 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.199955+0000", 2024-09-13T11:23:46.891 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00017067100000000001 2024-09-13T11:23:46.891 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:46.891 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:46.891 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:23:46.891 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.202075+0000", 2024-09-13T11:23:46.891 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0021206749999999998 2024-09-13T11:23:46.891 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:46.891 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:46.891 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:46.891 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.538395+0000", 2024-09-13T11:23:46.891 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.33631963599999998 2024-09-13T11:23:46.891 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:46.891 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:46.892 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:23:46.892 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.538498+0000", 2024-09-13T11:23:46.892 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00010267099999999999 2024-09-13T11:23:46.892 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:46.892 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:46.892 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:23:46.892 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.538571+0000", 2024-09-13T11:23:46.892 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.3538e-05 2024-09-13T11:23:46.892 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:46.892 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:23:46.892 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:46.892 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:46.892 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:46.893 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop_reply(client.4665.0:60433 3.3 e85/75 ondisk, result = 0)", 2024-09-13T11:23:46.893 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:42.203186+0000", 2024-09-13T11:23:46.893 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 4.5974702949999999, 2024-09-13T11:23:46.893 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.33540096699999999, 2024-09-13T11:23:46.893 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:23:46.893 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:23:46.893 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:23:46.893 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:23:46.893 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:46.893 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:23:46.893 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.203186+0000", 2024-09-13T11:23:46.893 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:46.893 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:46.893 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:46.894 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:23:46.894 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.203186+0000", 2024-09-13T11:23:46.894 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:46.894 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:46.894 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:46.894 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:23:46.894 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.203194+0000", 2024-09-13T11:23:46.894 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.0020000000000006e-06 2024-09-13T11:23:46.894 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:46.894 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:46.894 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:23:46.894 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.203202+0000", 2024-09-13T11:23:46.894 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.9829999999999996e-06 2024-09-13T11:23:46.895 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:46.895 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:46.895 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:23:46.895 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.538182+0000", 2024-09-13T11:23:46.895 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.33498029899999998 2024-09-13T11:23:46.895 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:46.895 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:46.895 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:23:46.895 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.538196+0000", 2024-09-13T11:23:46.895 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3807e-05 2024-09-13T11:23:46.895 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:46.895 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:46.896 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:23:46.896 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.538371+0000", 2024-09-13T11:23:46.896 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00017541200000000001 2024-09-13T11:23:46.896 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:46.896 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:46.896 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:23:46.896 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.538386+0000", 2024-09-13T11:23:46.896 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4833e-05 2024-09-13T11:23:46.896 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:46.896 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:46.896 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:23:46.897 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.538587+0000", 2024-09-13T11:23:46.897 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000200631 2024-09-13T11:23:46.897 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:46.897 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:23:46.897 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:46.897 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:46.897 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:46.897 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:61181 3.3 3:c321c36b:::benchmark_data_smithi177_31116_object61180:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86)", 2024-09-13T11:23:46.897 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:43.472106+0000", 2024-09-13T11:23:46.897 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 3.3285498649999998, 2024-09-13T11:23:46.897 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.085286808000000006, 2024-09-13T11:23:46.897 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:23:46.897 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:23:46.897 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:46.898 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:23:46.898 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:23:46.898 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:46.898 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 61181 2024-09-13T11:23:46.898 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:46.898 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:23:46.898 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:46.898 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:23:46.898 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:43.472106+0000", 2024-09-13T11:23:46.898 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:46.898 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:46.898 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:46.899 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:23:46.899 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:43.472106+0000", 2024-09-13T11:23:46.899 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:46.899 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:46.899 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:46.899 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:23:46.899 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:43.472110+0000", 2024-09-13T11:23:46.899 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.8600000000000003e-06 2024-09-13T11:23:46.899 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:46.899 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:46.899 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:23:46.899 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:43.472235+0000", 2024-09-13T11:23:46.899 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00012514199999999999 2024-09-13T11:23:46.899 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:46.900 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:46.900 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:23:46.900 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:43.472255+0000", 2024-09-13T11:23:46.900 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.9783999999999999e-05 2024-09-13T11:23:46.900 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:46.900 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:46.900 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:23:46.900 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:43.472271+0000", 2024-09-13T11:23:46.900 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5597e-05 2024-09-13T11:23:46.900 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:46.900 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:46.900 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:23:46.900 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:43.472357+0000", 2024-09-13T11:23:46.901 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.6817999999999994e-05 2024-09-13T11:23:46.901 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:46.901 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:46.901 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:23:46.901 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:43.472520+0000", 2024-09-13T11:23:46.901 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000162728 2024-09-13T11:23:46.901 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:46.901 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:46.901 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 7", 2024-09-13T11:23:46.901 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:43.472674+0000", 2024-09-13T11:23:46.901 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000153306 2024-09-13T11:23:46.901 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:46.902 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:46.902 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:23:46.902 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:43.474647+0000", 2024-09-13T11:23:46.902 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0019732069999999998 2024-09-13T11:23:46.902 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:46.902 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:46.902 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:46.902 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:43.557294+0000", 2024-09-13T11:23:46.902 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.082646837000000001 2024-09-13T11:23:46.902 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:46.902 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:46.902 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:23:46.902 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:43.557348+0000", 2024-09-13T11:23:46.902 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.4830000000000002e-05 2024-09-13T11:23:46.903 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:46.903 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:46.903 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:23:46.903 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:43.557393+0000", 2024-09-13T11:23:46.903 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.4699e-05 2024-09-13T11:23:46.903 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:46.903 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:23:46.903 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:46.903 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:46.903 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:46.903 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPush(3.1f 86/78 [PushOp(3:fc008368:::benchmark_data_smithi177_31116_object9243:head, version: 47'619, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:fc008368:::benchmark_data_smithi177_31116_object9243:head@47'619, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:23:46.903 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:43.771116+0000", 2024-09-13T11:23:46.903 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 3.0295400830000001, 2024-09-13T11:23:46.903 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.089632000000000003, 2024-09-13T11:23:46.903 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:23:46.904 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:23:46.904 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:23:46.904 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:23:46.904 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:46.904 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:23:46.904 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:43.771116+0000", 2024-09-13T11:23:46.904 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:46.904 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:46.904 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:46.904 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:23:46.904 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:43.771116+0000", 2024-09-13T11:23:46.904 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:46.904 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:46.905 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:46.905 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:23:46.905 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:43.771123+0000", 2024-09-13T11:23:46.905 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.1300000000000003e-06 2024-09-13T11:23:46.905 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:46.905 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:46.905 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:23:46.905 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:43.771194+0000", 2024-09-13T11:23:46.905 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.0753999999999994e-05 2024-09-13T11:23:46.905 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:46.905 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:46.905 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:23:46.905 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:43.858670+0000", 2024-09-13T11:23:46.906 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.087476133999999997 2024-09-13T11:23:46.906 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:46.906 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:46.906 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:23:46.906 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:43.858715+0000", 2024-09-13T11:23:46.906 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.4935999999999998e-05 2024-09-13T11:23:46.906 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:46.906 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:46.906 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:23:46.906 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:43.858937+0000", 2024-09-13T11:23:46.906 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000221683 2024-09-13T11:23:46.906 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:46.906 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:46.907 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:23:46.907 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:43.858963+0000", 2024-09-13T11:23:46.907 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.6587000000000001e-05 2024-09-13T11:23:46.907 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:46.907 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:46.907 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:23:46.907 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:43.860748+0000", 2024-09-13T11:23:46.907 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0017847760000000001 2024-09-13T11:23:46.907 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:46.907 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:23:46.907 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:46.907 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:46.907 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:46.908 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:61505 3.17 3:ec796459:::benchmark_data_smithi177_31116_object61504:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86)", 2024-09-13T11:23:46.908 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:43.786832+0000", 2024-09-13T11:23:46.908 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 3.0138244260000002, 2024-09-13T11:23:46.908 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.13516969700000001, 2024-09-13T11:23:46.908 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:23:46.908 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:23:46.908 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:46.908 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:23:46.908 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:23:46.908 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:46.908 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 61505 2024-09-13T11:23:46.908 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:46.909 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:23:46.909 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:46.909 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:23:46.909 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:43.786832+0000", 2024-09-13T11:23:46.909 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:46.909 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:46.909 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:46.909 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:23:46.909 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:43.786832+0000", 2024-09-13T11:23:46.909 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:46.909 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:46.909 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:46.910 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:23:46.910 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:43.786835+0000", 2024-09-13T11:23:46.910 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.2169999999999999e-06 2024-09-13T11:23:46.910 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:46.910 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:46.910 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:23:46.910 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:43.786910+0000", 2024-09-13T11:23:46.910 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.5129000000000005e-05 2024-09-13T11:23:46.910 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:46.910 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:46.910 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:23:46.910 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:43.786929+0000", 2024-09-13T11:23:46.910 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.8456000000000001e-05 2024-09-13T11:23:46.910 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:46.911 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:46.911 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:23:46.911 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:43.786942+0000", 2024-09-13T11:23:46.911 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3064000000000001e-05 2024-09-13T11:23:46.911 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:46.911 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:46.911 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:23:46.911 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:43.787039+0000", 2024-09-13T11:23:46.911 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.7193000000000003e-05 2024-09-13T11:23:46.911 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:46.911 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:46.911 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:23:46.911 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:43.787261+0000", 2024-09-13T11:23:46.912 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00022231200000000001 2024-09-13T11:23:46.912 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:46.912 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:46.912 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 3", 2024-09-13T11:23:46.912 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:43.787480+0000", 2024-09-13T11:23:46.912 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000218989 2024-09-13T11:23:46.912 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:46.912 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:46.912 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:23:46.912 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:43.789456+0000", 2024-09-13T11:23:46.912 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001975419 2024-09-13T11:23:46.912 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:46.912 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:46.913 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:46.913 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:43.921787+0000", 2024-09-13T11:23:46.913 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.132331755 2024-09-13T11:23:46.913 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:46.913 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:46.913 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:23:46.913 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:43.921902+0000", 2024-09-13T11:23:46.913 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000114269 2024-09-13T11:23:46.913 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:46.913 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:46.913 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:23:46.913 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:43.922001+0000", 2024-09-13T11:23:46.913 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.9894000000000003e-05 2024-09-13T11:23:46.914 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:46.914 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:23:46.914 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:46.914 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:46.914 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:46.914 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPush(3.1f 86/78 [PushOp(3:fa07f2f8:::benchmark_data_smithi177_31116_object9280:head, version: 47'620, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:fa07f2f8:::benchmark_data_smithi177_31116_object9280:head@47'620, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:23:46.914 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:43.869450+0000", 2024-09-13T11:23:46.914 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 2.9312058099999998, 2024-09-13T11:23:46.914 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.10950373200000001, 2024-09-13T11:23:46.914 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:23:46.914 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:23:46.914 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:23:46.914 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:23:46.915 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:46.915 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:23:46.915 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:43.869450+0000", 2024-09-13T11:23:46.915 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:46.915 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:46.915 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:46.915 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:23:46.915 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:43.869450+0000", 2024-09-13T11:23:46.915 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:46.915 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:46.915 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:46.915 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:23:46.915 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:43.869459+0000", 2024-09-13T11:23:46.916 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.5879999999999998e-06 2024-09-13T11:23:46.916 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:46.916 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:46.916 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:23:46.916 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:43.869545+0000", 2024-09-13T11:23:46.916 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.5869000000000001e-05 2024-09-13T11:23:46.916 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:46.916 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:46.916 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:23:46.916 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:43.976823+0000", 2024-09-13T11:23:46.916 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.107278659 2024-09-13T11:23:46.916 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:46.916 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:46.917 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:23:46.917 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:43.976864+0000", 2024-09-13T11:23:46.917 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.0775999999999997e-05 2024-09-13T11:23:46.917 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:46.917 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:46.917 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:23:46.917 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:43.977087+0000", 2024-09-13T11:23:46.917 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00022266500000000001 2024-09-13T11:23:46.917 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:46.917 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:46.917 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:23:46.917 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:43.977123+0000", 2024-09-13T11:23:46.917 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.6529999999999998e-05 2024-09-13T11:23:46.918 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:46.918 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:46.918 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:23:46.918 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:43.978954+0000", 2024-09-13T11:23:46.918 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0018306450000000001 2024-09-13T11:23:46.918 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:46.918 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:23:46.918 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:46.918 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:46.918 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:23:46.918 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-13T11:23:46.919 INFO:teuthology.orchestra.run.smithi001.stdout:osd.15: {} 2024-09-13T11:23:46.921 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_ops_in_flight 2024-09-13T11:23:46.921 INFO:tasks.ceph.osd.14.smithi162.stderr:2024-09-13T11:23:46.913+0000 7f2350b08640 -1 osd.14 89 handle_osd_map injecting map crc failure 2024-09-13T11:23:46.922 INFO:teuthology.orchestra.run.smithi001.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-13T11:23:46.922 INFO:teuthology.orchestra.run.smithi001.stderr:reweighted osd.6 to 1 (10000) 2024-09-13T11:23:46.922 INFO:teuthology.orchestra.run.smithi001.stderr:nodeep-scrub is unset 2024-09-13T11:23:46.923 INFO:tasks.ceph.osd.1.smithi092.stderr:2024-09-13T11:23:46.915+0000 7fe7789c5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33718) UID: 0 2024-09-13T11:23:46.935 INFO:tasks.ceph.osd.11.smithi177.stderr:2024-09-13T11:23:46.928+0000 7f246bcf0640 -1 osd.11 89 handle_osd_map injecting map crc failure 2024-09-13T11:23:46.941 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 7 1 2024-09-13T11:23:47.006 INFO:tasks.radosbench.radosbench.0.smithi177.stdout:2024-09-13T11:23:47.006003+0000 min lat: 0.00139681 max lat: 22.1835 avg lat: 0.0199495 2024-09-13T11:23:47.006 INFO:tasks.radosbench.radosbench.0.smithi177.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-13T11:23:47.006 INFO:tasks.radosbench.radosbench.0.smithi177.stdout: 80 16 64159 64143 50.1064 35.4375 0.00678632 0.0199495 2024-09-13T11:23:47.020 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-09-13T11:23:47.017+0000 7f97befc9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26315) UID: 0 2024-09-13T11:23:47.074 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-13T11:23:47.074 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [], 2024-09-13T11:23:47.075 INFO:teuthology.orchestra.run.smithi162.stdout: "num_ops": 0 2024-09-13T11:23:47.075 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-13T11:23:47.093 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_blocked_ops 2024-09-13T11:23:47.120 INFO:tasks.ceph.osd.4.smithi001.stderr:2024-09-13T11:23:47.116+0000 7ff8e0218640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26310) UID: 0 2024-09-13T11:23:47.221 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-09-13T11:23:47.217+0000 7f97befc9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26315) UID: 0 2024-09-13T11:23:47.268 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-13T11:23:47.268 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [], 2024-09-13T11:23:47.268 INFO:teuthology.orchestra.run.smithi162.stdout: "complaint_time": 30, 2024-09-13T11:23:47.268 INFO:teuthology.orchestra.run.smithi162.stdout: "num_blocked_ops": 0 2024-09-13T11:23:47.268 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-13T11:23:47.286 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_historic_ops 2024-09-13T11:23:47.322 INFO:tasks.ceph.osd.7.smithi177.stderr:2024-09-13T11:23:47.316+0000 7f897b51d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26575) UID: 0 2024-09-13T11:23:47.424 INFO:tasks.ceph.osd.3.smithi177.stderr:2024-09-13T11:23:47.420+0000 7f10adba4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26574) UID: 0 2024-09-13T11:23:47.466 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-13T11:23:47.466 INFO:teuthology.orchestra.run.smithi162.stdout: "size": 20, 2024-09-13T11:23:47.466 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 600, 2024-09-13T11:23:47.466 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [ 2024-09-13T11:23:47.466 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:47.466 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.4665.0:9905 3.6 e47/30 3:639d4f69:::benchmark_data_smithi177_31116_object9904:head v 47'621, mlcod=47'618)", 2024-09-13T11:23:47.466 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:22:54.301066+0000", 2024-09-13T11:23:47.466 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 53.161310225999998, 2024-09-13T11:23:47.466 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.11187942300000001, 2024-09-13T11:23:47.467 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:23:47.467 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:23:47.467 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:47.467 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:23:47.467 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:47.467 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:23:47.467 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:54.301066+0000", 2024-09-13T11:23:47.467 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:47.467 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:47.467 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:47.467 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:23:47.467 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:54.301066+0000", 2024-09-13T11:23:47.468 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:47.468 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:47.468 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:47.468 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:23:47.468 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:54.301077+0000", 2024-09-13T11:23:47.468 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1154e-05 2024-09-13T11:23:47.468 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:47.468 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:47.468 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:23:47.468 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:54.301136+0000", 2024-09-13T11:23:47.468 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.8480999999999998e-05 2024-09-13T11:23:47.468 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:47.468 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:47.469 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:23:47.469 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:54.410822+0000", 2024-09-13T11:23:47.469 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.109685649 2024-09-13T11:23:47.469 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:47.469 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:47.469 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:23:47.469 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:54.410829+0000", 2024-09-13T11:23:47.469 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.7470000000000005e-06 2024-09-13T11:23:47.469 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:47.469 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:47.469 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:23:47.469 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:54.410995+0000", 2024-09-13T11:23:47.469 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00016614200000000001 2024-09-13T11:23:47.470 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:47.470 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:47.470 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:23:47.470 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:54.411056+0000", 2024-09-13T11:23:47.470 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.0174999999999999e-05 2024-09-13T11:23:47.470 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:47.470 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:47.470 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:23:47.470 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:54.412869+0000", 2024-09-13T11:23:47.470 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001813423 2024-09-13T11:23:47.470 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:47.470 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:47.470 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:23:47.470 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:54.412946+0000", 2024-09-13T11:23:47.471 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.6651999999999997e-05 2024-09-13T11:23:47.471 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:47.471 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:23:47.471 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:47.471 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:47.471 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:47.471 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:48911 3.12 3:4a364a6a:::benchmark_data_smithi177_31116_object48910:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e71)", 2024-09-13T11:23:47.471 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:25.457776+0000", 2024-09-13T11:23:47.471 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 22.004600151000002, 2024-09-13T11:23:47.471 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.22561999599999999, 2024-09-13T11:23:47.471 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:23:47.471 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:23:47.471 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:47.472 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:23:47.472 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:23:47.472 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:47.472 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 48911 2024-09-13T11:23:47.472 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:47.472 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:23:47.472 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:47.472 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:23:47.472 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.457776+0000", 2024-09-13T11:23:47.472 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:47.472 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:47.472 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:47.472 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:23:47.473 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.457776+0000", 2024-09-13T11:23:47.473 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:47.473 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:47.473 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:47.473 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:23:47.473 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.457788+0000", 2024-09-13T11:23:47.473 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2084e-05 2024-09-13T11:23:47.473 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:47.473 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:47.473 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:23:47.473 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.457823+0000", 2024-09-13T11:23:47.473 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.5002000000000001e-05 2024-09-13T11:23:47.473 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:47.474 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:47.474 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:23:47.474 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.457836+0000", 2024-09-13T11:23:47.474 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2988e-05 2024-09-13T11:23:47.474 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:47.474 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:47.474 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:23:47.474 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.457843+0000", 2024-09-13T11:23:47.474 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.5320000000000003e-06 2024-09-13T11:23:47.474 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:47.474 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:47.474 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:23:47.474 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.457895+0000", 2024-09-13T11:23:47.474 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.1690000000000001e-05 2024-09-13T11:23:47.475 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:47.475 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:47.475 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:23:47.475 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.458001+0000", 2024-09-13T11:23:47.475 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000106024 2024-09-13T11:23:47.475 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:47.475 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:47.475 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 7", 2024-09-13T11:23:47.475 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.458101+0000", 2024-09-13T11:23:47.475 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00010060699999999999 2024-09-13T11:23:47.475 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:47.475 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:47.475 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:23:47.475 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.459250+0000", 2024-09-13T11:23:47.475 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0011485670000000001 2024-09-13T11:23:47.476 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:47.476 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:47.476 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:47.476 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.682166+0000", 2024-09-13T11:23:47.476 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.22291609500000001 2024-09-13T11:23:47.476 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:47.476 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:47.476 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:23:47.476 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.682271+0000", 2024-09-13T11:23:47.476 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000105179 2024-09-13T11:23:47.477 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:47.477 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:47.477 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:23:47.477 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.683396+0000", 2024-09-13T11:23:47.477 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001125228 2024-09-13T11:23:47.477 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:47.477 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:23:47.477 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:47.477 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:47.477 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:47.477 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:48923 3.12 3:4ae1afd7:::benchmark_data_smithi177_31116_object48922:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e71)", 2024-09-13T11:23:47.477 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:25.465697+0000", 2024-09-13T11:23:47.477 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 21.996678985999999, 2024-09-13T11:23:47.478 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.21931837100000001, 2024-09-13T11:23:47.478 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:23:47.478 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:23:47.478 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:47.478 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:23:47.478 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:23:47.478 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:47.478 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 48923 2024-09-13T11:23:47.478 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:47.478 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:23:47.478 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:47.478 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:23:47.478 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.465697+0000", 2024-09-13T11:23:47.478 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:47.479 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:47.479 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:47.479 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:23:47.479 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.465697+0000", 2024-09-13T11:23:47.479 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:47.479 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:47.479 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:47.479 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:23:47.479 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.465700+0000", 2024-09-13T11:23:47.479 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.041e-06 2024-09-13T11:23:47.479 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:47.479 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:47.479 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:23:47.480 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.465757+0000", 2024-09-13T11:23:47.480 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.7664999999999998e-05 2024-09-13T11:23:47.480 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:47.480 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:47.480 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:23:47.480 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.465769+0000", 2024-09-13T11:23:47.480 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.219e-05 2024-09-13T11:23:47.480 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:47.480 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:47.480 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:23:47.480 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.465775+0000", 2024-09-13T11:23:47.480 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.6779999999999997e-06 2024-09-13T11:23:47.480 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:47.481 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:47.481 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:23:47.481 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.465821+0000", 2024-09-13T11:23:47.481 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.5577000000000001e-05 2024-09-13T11:23:47.481 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:47.481 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:47.481 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:23:47.481 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.465928+0000", 2024-09-13T11:23:47.481 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000107502 2024-09-13T11:23:47.481 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:47.481 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:47.481 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 7", 2024-09-13T11:23:47.481 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.466022+0000", 2024-09-13T11:23:47.481 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.3802999999999996e-05 2024-09-13T11:23:47.482 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:47.482 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:47.482 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:23:47.482 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.466901+0000", 2024-09-13T11:23:47.482 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00087950699999999997 2024-09-13T11:23:47.482 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:47.482 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:47.482 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:47.482 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.683622+0000", 2024-09-13T11:23:47.482 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.216720462 2024-09-13T11:23:47.482 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:47.482 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:47.482 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:23:47.483 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.683771+0000", 2024-09-13T11:23:47.483 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00014900199999999999 2024-09-13T11:23:47.483 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:47.483 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:47.483 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:23:47.483 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.685016+0000", 2024-09-13T11:23:47.483 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001244944 2024-09-13T11:23:47.483 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:47.483 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:23:47.483 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:47.483 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:47.483 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:47.483 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:48955 3.12 3:4ef2991b:::benchmark_data_smithi177_31116_object48954:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e71)", 2024-09-13T11:23:47.484 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:25.494091+0000", 2024-09-13T11:23:47.484 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 21.968285348999999, 2024-09-13T11:23:47.484 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.19235933599999999, 2024-09-13T11:23:47.484 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:23:47.484 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:23:47.484 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:47.484 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:23:47.484 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:23:47.484 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:47.484 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 48955 2024-09-13T11:23:47.484 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:47.484 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:23:47.484 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:47.485 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:23:47.485 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.494091+0000", 2024-09-13T11:23:47.485 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:47.485 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:47.485 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:47.485 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:23:47.485 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.494091+0000", 2024-09-13T11:23:47.485 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:47.485 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:47.485 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:47.485 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:23:47.485 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.494093+0000", 2024-09-13T11:23:47.485 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.0770000000000002e-06 2024-09-13T11:23:47.485 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:47.486 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:47.486 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:23:47.486 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.494160+0000", 2024-09-13T11:23:47.486 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.6798999999999995e-05 2024-09-13T11:23:47.486 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:47.486 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:47.486 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:23:47.486 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.494174+0000", 2024-09-13T11:23:47.486 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4489e-05 2024-09-13T11:23:47.486 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:47.487 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:47.487 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:23:47.487 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.494183+0000", 2024-09-13T11:23:47.487 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.4460000000000001e-06 2024-09-13T11:23:47.487 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:47.487 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:47.487 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:23:47.487 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.494239+0000", 2024-09-13T11:23:47.487 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.6140999999999997e-05 2024-09-13T11:23:47.487 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:47.487 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:47.487 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:23:47.487 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.494385+0000", 2024-09-13T11:23:47.488 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00014546399999999999 2024-09-13T11:23:47.488 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:47.488 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:47.488 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 7", 2024-09-13T11:23:47.488 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.494506+0000", 2024-09-13T11:23:47.488 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000121328 2024-09-13T11:23:47.488 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:47.488 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:47.488 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:23:47.488 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.495528+0000", 2024-09-13T11:23:47.488 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0010226079999999999 2024-09-13T11:23:47.488 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:47.488 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:47.489 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:47.489 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.685177+0000", 2024-09-13T11:23:47.489 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.18964847500000001 2024-09-13T11:23:47.489 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:47.489 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:47.489 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:23:47.489 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.685337+0000", 2024-09-13T11:23:47.489 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00016055400000000001 2024-09-13T11:23:47.489 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:47.489 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:47.489 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:23:47.489 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.686450+0000", 2024-09-13T11:23:47.489 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0011129550000000001 2024-09-13T11:23:47.490 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:47.490 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:23:47.490 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:47.490 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:47.490 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:47.490 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:48967 3.12 3:4abf2346:::benchmark_data_smithi177_31116_object48966:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e71)", 2024-09-13T11:23:47.490 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:25.509687+0000", 2024-09-13T11:23:47.490 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 21.952689225, 2024-09-13T11:23:47.490 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.17798804900000001, 2024-09-13T11:23:47.490 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:23:47.490 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:23:47.490 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:47.490 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:23:47.491 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:23:47.491 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:47.491 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 48967 2024-09-13T11:23:47.491 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:47.491 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:23:47.491 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:47.491 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:23:47.491 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.509687+0000", 2024-09-13T11:23:47.491 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:47.491 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:47.491 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:47.491 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:23:47.491 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.509687+0000", 2024-09-13T11:23:47.491 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:47.492 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:47.492 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:47.492 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:23:47.492 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.509689+0000", 2024-09-13T11:23:47.492 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.9719999999999999e-06 2024-09-13T11:23:47.492 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:47.492 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:47.492 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:23:47.492 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.509754+0000", 2024-09-13T11:23:47.492 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.5031999999999997e-05 2024-09-13T11:23:47.492 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:47.492 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:47.492 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:23:47.492 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.509787+0000", 2024-09-13T11:23:47.493 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.2891000000000001e-05 2024-09-13T11:23:47.493 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:47.493 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:47.493 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:23:47.493 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.509793+0000", 2024-09-13T11:23:47.493 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.1500000000000004e-06 2024-09-13T11:23:47.493 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:47.493 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:47.493 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:23:47.493 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.509846+0000", 2024-09-13T11:23:47.493 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.2926000000000001e-05 2024-09-13T11:23:47.493 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:47.493 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:47.494 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:23:47.494 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.509957+0000", 2024-09-13T11:23:47.494 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000110734 2024-09-13T11:23:47.494 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:47.494 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:47.494 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 7", 2024-09-13T11:23:47.494 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.510083+0000", 2024-09-13T11:23:47.494 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00012628300000000001 2024-09-13T11:23:47.495 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:47.495 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:47.495 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:23:47.495 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.511047+0000", 2024-09-13T11:23:47.495 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00096376800000000005 2024-09-13T11:23:47.495 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:47.495 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:47.495 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:47.495 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.686579+0000", 2024-09-13T11:23:47.495 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.17553200999999999 2024-09-13T11:23:47.495 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:47.495 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:47.496 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:23:47.496 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.686846+0000", 2024-09-13T11:23:47.496 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000267039 2024-09-13T11:23:47.496 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:47.496 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:47.496 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:23:47.496 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.687675+0000", 2024-09-13T11:23:47.496 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00082924400000000003 2024-09-13T11:23:47.496 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:47.496 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:23:47.496 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:47.496 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:47.496 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:47.497 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:48987 3.12 3:4e1e6d0c:::benchmark_data_smithi177_31116_object48986:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e71)", 2024-09-13T11:23:47.497 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:25.525640+0000", 2024-09-13T11:23:47.497 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 21.936736581000002, 2024-09-13T11:23:47.497 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.163044945, 2024-09-13T11:23:47.497 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:23:47.497 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:23:47.500 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:47.500 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:23:47.500 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:23:47.500 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:47.500 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 48987 2024-09-13T11:23:47.500 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:47.500 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:23:47.500 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:47.500 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:23:47.500 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.525640+0000", 2024-09-13T11:23:47.500 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:47.501 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:47.501 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:47.501 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:23:47.501 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.525640+0000", 2024-09-13T11:23:47.501 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:47.501 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:47.501 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:47.501 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:23:47.501 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.525651+0000", 2024-09-13T11:23:47.501 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1092e-05 2024-09-13T11:23:47.501 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:47.501 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:47.501 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:23:47.501 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.525795+0000", 2024-09-13T11:23:47.502 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00014417400000000001 2024-09-13T11:23:47.502 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:47.502 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:47.502 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:23:47.502 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.525834+0000", 2024-09-13T11:23:47.502 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.9026000000000002e-05 2024-09-13T11:23:47.502 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:47.502 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:47.502 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:23:47.502 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.525857+0000", 2024-09-13T11:23:47.502 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.2904999999999998e-05 2024-09-13T11:23:47.502 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:47.503 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:47.503 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:23:47.503 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.525967+0000", 2024-09-13T11:23:47.503 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000110188 2024-09-13T11:23:47.503 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:47.503 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:47.503 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:23:47.503 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.526214+0000", 2024-09-13T11:23:47.503 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00024667800000000001 2024-09-13T11:23:47.503 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:47.504 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:47.504 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 7", 2024-09-13T11:23:47.504 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.526499+0000", 2024-09-13T11:23:47.504 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00028463799999999998 2024-09-13T11:23:47.504 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:47.504 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:47.504 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:23:47.504 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.528226+0000", 2024-09-13T11:23:47.504 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0017276679999999999 2024-09-13T11:23:47.504 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:47.504 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:47.505 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:47.505 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.687803+0000", 2024-09-13T11:23:47.505 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.15957658899999999 2024-09-13T11:23:47.505 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:47.505 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:47.505 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:23:47.505 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.687986+0000", 2024-09-13T11:23:47.505 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00018332400000000001 2024-09-13T11:23:47.505 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:47.505 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:47.505 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:23:47.505 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.688685+0000", 2024-09-13T11:23:47.506 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00069866299999999998 2024-09-13T11:23:47.506 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:47.506 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:23:47.506 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:47.506 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:47.506 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:47.506 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:49023 3.12 3:4a994401:::benchmark_data_smithi177_31116_object49022:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e71)", 2024-09-13T11:23:47.506 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:25.562627+0000", 2024-09-13T11:23:47.506 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 21.899749211, 2024-09-13T11:23:47.506 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.12709894099999999, 2024-09-13T11:23:47.506 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:23:47.506 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:23:47.506 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:47.507 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:23:47.507 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:23:47.507 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:47.507 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 49023 2024-09-13T11:23:47.507 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:47.507 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:23:47.507 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:47.507 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:23:47.507 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.562627+0000", 2024-09-13T11:23:47.507 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:47.507 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:47.507 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:47.508 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:23:47.508 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.562627+0000", 2024-09-13T11:23:47.508 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:47.508 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:47.508 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:47.508 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:23:47.508 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.562636+0000", 2024-09-13T11:23:47.508 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.2530000000000004e-06 2024-09-13T11:23:47.508 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:47.508 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:47.508 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:23:47.508 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.562685+0000", 2024-09-13T11:23:47.508 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.9870999999999997e-05 2024-09-13T11:23:47.509 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:47.509 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:47.509 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:23:47.509 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.562717+0000", 2024-09-13T11:23:47.509 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.1418999999999999e-05 2024-09-13T11:23:47.509 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:47.509 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:47.509 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:23:47.509 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.562739+0000", 2024-09-13T11:23:47.509 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.1756e-05 2024-09-13T11:23:47.509 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:47.509 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:47.509 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:23:47.509 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.562823+0000", 2024-09-13T11:23:47.510 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.4129999999999996e-05 2024-09-13T11:23:47.510 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:47.510 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:47.510 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:23:47.510 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.562999+0000", 2024-09-13T11:23:47.510 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00017660199999999999 2024-09-13T11:23:47.510 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:47.510 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:47.510 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 7", 2024-09-13T11:23:47.510 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.563179+0000", 2024-09-13T11:23:47.510 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00018009800000000001 2024-09-13T11:23:47.510 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:47.510 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:47.511 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:23:47.511 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.564940+0000", 2024-09-13T11:23:47.511 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0017609310000000001 2024-09-13T11:23:47.511 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:47.511 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:47.511 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:47.511 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.688830+0000", 2024-09-13T11:23:47.511 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.123889754 2024-09-13T11:23:47.511 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:47.511 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:47.511 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:23:47.511 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.688919+0000", 2024-09-13T11:23:47.511 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.9010000000000003e-05 2024-09-13T11:23:47.512 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:47.512 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:47.512 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:23:47.512 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.689726+0000", 2024-09-13T11:23:47.512 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00080711699999999999 2024-09-13T11:23:47.512 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:47.512 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:23:47.512 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:47.512 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:47.512 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:47.512 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:49036 3.12 3:4ce03469:::benchmark_data_smithi177_31116_object49035:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e71)", 2024-09-13T11:23:47.513 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:25.576814+0000", 2024-09-13T11:23:47.513 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 21.885562399000001, 2024-09-13T11:23:47.513 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.114098698, 2024-09-13T11:23:47.513 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:23:47.513 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:23:47.513 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:47.513 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:23:47.513 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:23:47.513 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:47.513 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 49036 2024-09-13T11:23:47.513 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:47.514 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:23:47.514 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:47.514 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:23:47.514 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.576814+0000", 2024-09-13T11:23:47.514 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:47.514 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:47.514 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:47.514 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:23:47.514 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.576814+0000", 2024-09-13T11:23:47.514 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:47.514 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:47.515 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:47.515 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:23:47.515 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.576822+0000", 2024-09-13T11:23:47.515 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.6399999999999997e-06 2024-09-13T11:23:47.515 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:47.515 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:47.515 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:23:47.515 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.576875+0000", 2024-09-13T11:23:47.515 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.3517000000000003e-05 2024-09-13T11:23:47.515 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:47.515 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:47.515 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:23:47.515 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.576900+0000", 2024-09-13T11:23:47.516 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.5233e-05 2024-09-13T11:23:47.516 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:47.516 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:47.516 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:23:47.516 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.576915+0000", 2024-09-13T11:23:47.516 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4093e-05 2024-09-13T11:23:47.516 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:47.516 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:47.516 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:23:47.516 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.577008+0000", 2024-09-13T11:23:47.516 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.3794000000000004e-05 2024-09-13T11:23:47.516 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:47.517 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:47.517 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:23:47.517 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.577290+0000", 2024-09-13T11:23:47.517 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00028117300000000001 2024-09-13T11:23:47.517 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:47.517 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:47.517 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 7", 2024-09-13T11:23:47.517 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.577519+0000", 2024-09-13T11:23:47.517 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000229008 2024-09-13T11:23:47.517 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:47.517 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:47.517 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:23:47.518 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.579594+0000", 2024-09-13T11:23:47.518 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0020753189999999999 2024-09-13T11:23:47.518 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:47.518 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:47.518 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:47.518 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.689873+0000", 2024-09-13T11:23:47.518 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.110279455 2024-09-13T11:23:47.518 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:47.518 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:47.518 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:23:47.518 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.690011+0000", 2024-09-13T11:23:47.518 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00013810299999999999 2024-09-13T11:23:47.518 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:47.518 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:47.519 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:23:47.519 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.690913+0000", 2024-09-13T11:23:47.519 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00090136299999999997 2024-09-13T11:23:47.519 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:47.519 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:23:47.519 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:47.519 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:47.519 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:47.519 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:54637 3.22 3:44450106:::benchmark_data_smithi177_31116_object54636:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e75)", 2024-09-13T11:23:47.519 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:30.978960+0000", 2024-09-13T11:23:47.519 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 16.483416149, 2024-09-13T11:23:47.519 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.960188928, 2024-09-13T11:23:47.519 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:23:47.520 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:23:47.520 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-09-13T11:23:47.520 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:23:47.520 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:23:47.520 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:47.520 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 54637 2024-09-13T11:23:47.520 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:47.520 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:23:47.520 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:47.520 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:23:47.520 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.978960+0000", 2024-09-13T11:23:47.520 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:47.520 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:47.521 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:47.521 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:23:47.521 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.978960+0000", 2024-09-13T11:23:47.521 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:47.521 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:47.521 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:47.521 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:23:47.521 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.978962+0000", 2024-09-13T11:23:47.521 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.46e-06 2024-09-13T11:23:47.521 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:47.522 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:47.522 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:23:47.522 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.979024+0000", 2024-09-13T11:23:47.522 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.1779999999999995e-05 2024-09-13T11:23:47.522 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:47.522 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:47.522 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:23:47.522 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.979037+0000", 2024-09-13T11:23:47.522 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2948e-05 2024-09-13T11:23:47.522 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:47.522 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:47.522 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:23:47.522 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.979048+0000", 2024-09-13T11:23:47.523 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1241e-05 2024-09-13T11:23:47.523 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:47.523 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:47.523 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:23:47.523 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:31.032951+0000", 2024-09-13T11:23:47.523 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.053903286000000002 2024-09-13T11:23:47.523 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:47.523 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:47.523 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-09-13T11:23:47.523 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:31.032958+0000", 2024-09-13T11:23:47.523 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.8109999999999997e-06 2024-09-13T11:23:47.523 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:47.523 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:47.524 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:23:47.524 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:31.933462+0000", 2024-09-13T11:23:47.524 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.90050463999999997 2024-09-13T11:23:47.524 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:47.524 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:47.524 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for degraded object", 2024-09-13T11:23:47.524 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:31.934668+0000", 2024-09-13T11:23:47.524 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0012053249999999999 2024-09-13T11:23:47.524 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:47.524 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:47.524 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:23:47.525 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:31.938887+0000", 2024-09-13T11:23:47.525 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0042187120000000003 2024-09-13T11:23:47.525 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:47.525 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:47.525 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:23:47.525 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:31.939149+0000", 2024-09-13T11:23:47.525 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000262725 2024-09-13T11:23:47.525 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:47.525 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:23:47.525 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:47.525 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:47.525 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:47.525 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:54663 3.12 3:49ad4b1b:::benchmark_data_smithi177_31116_object54662:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e75)", 2024-09-13T11:23:47.526 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:31.017498+0000", 2024-09-13T11:23:47.526 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 16.444878250999999, 2024-09-13T11:23:47.526 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.137773485, 2024-09-13T11:23:47.526 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:23:47.526 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:23:47.526 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:47.526 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:23:47.526 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:23:47.526 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:47.526 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 54663 2024-09-13T11:23:47.526 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:47.526 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:23:47.526 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:47.526 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:23:47.527 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:31.017498+0000", 2024-09-13T11:23:47.527 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:47.527 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:47.527 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:47.527 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:23:47.527 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:31.017498+0000", 2024-09-13T11:23:47.527 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:47.527 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:47.527 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:47.527 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:23:47.528 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:31.017501+0000", 2024-09-13T11:23:47.528 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.0730000000000001e-06 2024-09-13T11:23:47.528 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:47.528 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:47.528 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:23:47.528 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:31.017547+0000", 2024-09-13T11:23:47.528 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.5553000000000002e-05 2024-09-13T11:23:47.528 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:47.528 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:47.528 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:23:47.528 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:31.017563+0000", 2024-09-13T11:23:47.528 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.6041000000000001e-05 2024-09-13T11:23:47.528 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:47.528 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:47.529 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:23:47.529 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:31.017573+0000", 2024-09-13T11:23:47.529 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0573e-05 2024-09-13T11:23:47.529 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:47.529 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:47.529 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:23:47.529 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:31.032699+0000", 2024-09-13T11:23:47.529 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.015125748 2024-09-13T11:23:47.529 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:47.529 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:47.530 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:23:47.530 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:31.032866+0000", 2024-09-13T11:23:47.530 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00016644300000000001 2024-09-13T11:23:47.530 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:47.530 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:47.530 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 7", 2024-09-13T11:23:47.530 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:31.033011+0000", 2024-09-13T11:23:47.530 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00014518300000000001 2024-09-13T11:23:47.530 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:47.530 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:47.530 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:23:47.530 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:31.042895+0000", 2024-09-13T11:23:47.530 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0098840160000000007 2024-09-13T11:23:47.531 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:47.531 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:47.531 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:47.531 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:31.154503+0000", 2024-09-13T11:23:47.531 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.11160790900000001 2024-09-13T11:23:47.531 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:47.531 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:47.531 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:23:47.531 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:31.154595+0000", 2024-09-13T11:23:47.531 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.2386000000000006e-05 2024-09-13T11:23:47.531 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:47.531 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:47.531 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:23:47.532 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:31.155272+0000", 2024-09-13T11:23:47.532 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00067655999999999999 2024-09-13T11:23:47.532 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:47.532 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:23:47.532 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:47.532 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:47.532 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:47.532 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPushReply(3.12 75/69 [PushReplyOp(3:4db63d39:::benchmark_data_smithi177_31116_object32641:head)])", 2024-09-13T11:23:47.532 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:31.646428+0000", 2024-09-13T11:23:47.532 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 15.815948025000001, 2024-09-13T11:23:47.532 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.28409169299999998, 2024-09-13T11:23:47.532 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:23:47.532 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:23:47.533 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-09-13T11:23:47.533 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:23:47.533 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:47.533 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:23:47.533 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:31.646428+0000", 2024-09-13T11:23:47.533 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:47.533 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:47.533 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:47.533 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:23:47.533 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:31.646428+0000", 2024-09-13T11:23:47.533 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:47.533 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:47.534 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:47.534 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:23:47.534 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:31.646431+0000", 2024-09-13T11:23:47.534 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.3149999999999999e-06 2024-09-13T11:23:47.534 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:47.534 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:47.534 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:23:47.534 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:31.646438+0000", 2024-09-13T11:23:47.534 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.7379999999999994e-06 2024-09-13T11:23:47.534 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:47.534 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:47.534 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:23:47.534 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:31.928787+0000", 2024-09-13T11:23:47.535 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.28234828699999998 2024-09-13T11:23:47.535 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:47.535 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:47.535 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:23:47.535 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:31.929066+0000", 2024-09-13T11:23:47.535 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00027956500000000001 2024-09-13T11:23:47.535 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:47.535 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:47.535 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:23:47.535 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:31.929216+0000", 2024-09-13T11:23:47.535 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00014963500000000001 2024-09-13T11:23:47.535 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:47.535 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:47.536 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:23:47.536 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:31.930520+0000", 2024-09-13T11:23:47.536 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001304153 2024-09-13T11:23:47.536 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:47.536 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:23:47.536 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:47.536 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:47.536 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:47.536 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPushReply(3.12 76/69 [PushReplyOp(3:4a5f7a67:::benchmark_data_smithi177_31116_object32993:head)])", 2024-09-13T11:23:47.536 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:32.069056+0000", 2024-09-13T11:23:47.536 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 15.393320744, 2024-09-13T11:23:47.536 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.17843624299999999, 2024-09-13T11:23:47.537 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:23:47.537 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:23:47.537 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-09-13T11:23:47.537 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:23:47.537 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:47.537 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:23:47.537 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.069056+0000", 2024-09-13T11:23:47.537 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:47.537 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:47.537 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:47.537 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:23:47.538 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.069056+0000", 2024-09-13T11:23:47.538 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:47.538 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:47.538 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:47.538 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:23:47.538 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.069057+0000", 2024-09-13T11:23:47.538 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4699999999999999e-06 2024-09-13T11:23:47.538 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:47.538 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:47.538 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:23:47.538 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.069065+0000", 2024-09-13T11:23:47.538 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.9740000000000002e-06 2024-09-13T11:23:47.538 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:47.539 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:47.539 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:23:47.539 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.246435+0000", 2024-09-13T11:23:47.539 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.17736979 2024-09-13T11:23:47.539 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:47.539 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:47.539 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:23:47.539 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.246450+0000", 2024-09-13T11:23:47.539 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4582e-05 2024-09-13T11:23:47.539 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:47.539 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:47.539 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:23:47.539 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.246556+0000", 2024-09-13T11:23:47.540 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00010608200000000001 2024-09-13T11:23:47.540 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:47.540 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:47.540 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:23:47.540 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.247492+0000", 2024-09-13T11:23:47.540 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00093634499999999999 2024-09-13T11:23:47.540 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:47.540 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:23:47.540 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:47.540 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:47.540 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:47.540 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPush(3.14 76/70 [PushOp(3:297981b4:::benchmark_data_smithi177_31116_object16264:head, version: 52'1034, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:297981b4:::benchmark_data_smithi177_31116_object16264:head@52'1034, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:23:47.541 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:32.119983+0000", 2024-09-13T11:23:47.541 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 15.342393040999999, 2024-09-13T11:23:47.541 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.107915715, 2024-09-13T11:23:47.541 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:23:47.541 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:23:47.541 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:23:47.541 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:23:47.541 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:47.541 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:23:47.541 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.119983+0000", 2024-09-13T11:23:47.541 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:47.541 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:47.542 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:47.542 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:23:47.542 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.119983+0000", 2024-09-13T11:23:47.542 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:47.542 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:47.542 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:47.542 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:23:47.542 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.119985+0000", 2024-09-13T11:23:47.542 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5540000000000001e-06 2024-09-13T11:23:47.542 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:47.542 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:47.542 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:23:47.542 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.120161+0000", 2024-09-13T11:23:47.543 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00017632800000000001 2024-09-13T11:23:47.543 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:47.543 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:47.543 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:23:47.543 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.226182+0000", 2024-09-13T11:23:47.543 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.106020504 2024-09-13T11:23:47.543 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:47.543 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:47.543 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:23:47.543 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.226209+0000", 2024-09-13T11:23:47.543 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.7673000000000001e-05 2024-09-13T11:23:47.543 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:47.543 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:47.544 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:23:47.544 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.226421+0000", 2024-09-13T11:23:47.544 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000211967 2024-09-13T11:23:47.544 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:47.544 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:47.544 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:23:47.544 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.226455+0000", 2024-09-13T11:23:47.544 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.3182000000000002e-05 2024-09-13T11:23:47.544 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:47.544 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:47.544 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:23:47.544 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.227899+0000", 2024-09-13T11:23:47.544 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001444507 2024-09-13T11:23:47.545 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:47.545 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:23:47.545 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:47.545 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:47.545 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:47.545 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPush(3.15 76/70 [PushOp(3:aab036a3:::benchmark_data_smithi177_31116_object22075:head, version: 55'1344, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:aab036a3:::benchmark_data_smithi177_31116_object22075:head@55'1344, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:23:47.545 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:32.223005+0000", 2024-09-13T11:23:47.545 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 15.239371722, 2024-09-13T11:23:47.545 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.13015405599999999, 2024-09-13T11:23:47.545 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:23:47.545 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:23:47.545 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:23:47.545 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:23:47.546 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:47.546 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:23:47.546 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.223005+0000", 2024-09-13T11:23:47.546 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:47.546 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:47.546 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:47.546 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:23:47.546 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.223005+0000", 2024-09-13T11:23:47.546 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:47.546 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:47.546 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:47.546 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:23:47.546 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.223006+0000", 2024-09-13T11:23:47.546 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.7379999999999999e-06 2024-09-13T11:23:47.547 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:47.547 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:47.547 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:23:47.547 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.223060+0000", 2024-09-13T11:23:47.547 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.3050000000000002e-05 2024-09-13T11:23:47.547 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:47.547 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:47.547 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:23:47.547 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.351482+0000", 2024-09-13T11:23:47.547 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.12842293299999999 2024-09-13T11:23:47.547 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:47.547 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:47.547 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:23:47.547 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.351512+0000", 2024-09-13T11:23:47.548 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.9212000000000001e-05 2024-09-13T11:23:47.548 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:47.548 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:47.548 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:23:47.548 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.351685+0000", 2024-09-13T11:23:47.548 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00017326099999999999 2024-09-13T11:23:47.548 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:47.548 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:47.548 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:23:47.548 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.351709+0000", 2024-09-13T11:23:47.548 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.4105e-05 2024-09-13T11:23:47.549 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:47.549 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:47.549 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:23:47.549 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.353159+0000", 2024-09-13T11:23:47.549 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0014497570000000001 2024-09-13T11:23:47.549 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:47.549 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:23:47.549 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:47.549 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:47.549 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:47.549 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPushReply(3.12 76/69 [PushReplyOp(3:4d1753c5:::benchmark_data_smithi177_31116_object33024:head)])", 2024-09-13T11:23:47.549 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:32.277343+0000", 2024-09-13T11:23:47.549 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 15.185033507, 2024-09-13T11:23:47.549 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.14475505799999999, 2024-09-13T11:23:47.550 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:23:47.550 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:23:47.550 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-09-13T11:23:47.550 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:23:47.550 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:47.550 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:23:47.550 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.277343+0000", 2024-09-13T11:23:47.550 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:47.550 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:47.550 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:47.550 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:23:47.550 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.277343+0000", 2024-09-13T11:23:47.550 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:47.551 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:47.551 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:47.551 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:23:47.551 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.277346+0000", 2024-09-13T11:23:47.551 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.7800000000000001e-06 2024-09-13T11:23:47.551 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:47.551 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:47.551 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:23:47.551 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.277357+0000", 2024-09-13T11:23:47.551 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0983e-05 2024-09-13T11:23:47.551 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:47.551 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:47.551 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:23:47.552 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.420933+0000", 2024-09-13T11:23:47.552 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.14357647800000001 2024-09-13T11:23:47.552 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:47.552 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:47.552 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:23:47.552 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.420954+0000", 2024-09-13T11:23:47.552 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.0322999999999998e-05 2024-09-13T11:23:47.552 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:47.552 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:47.552 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:23:47.552 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.421048+0000", 2024-09-13T11:23:47.552 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.4263000000000002e-05 2024-09-13T11:23:47.553 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:47.553 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:47.553 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:23:47.553 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.422098+0000", 2024-09-13T11:23:47.553 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001050231 2024-09-13T11:23:47.553 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:47.553 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:23:47.553 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:47.553 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:47.553 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:47.553 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPush(3.15 76/70 [PushOp(3:ac553b6c:::benchmark_data_smithi177_31116_object22092:head, version: 55'1348, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:ac553b6c:::benchmark_data_smithi177_31116_object22092:head@55'1348, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:23:47.553 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:32.370110+0000", 2024-09-13T11:23:47.554 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 15.092266727, 2024-09-13T11:23:47.554 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.134376467, 2024-09-13T11:23:47.554 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:23:47.554 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:23:47.554 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:23:47.554 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:23:47.554 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:47.554 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:23:47.554 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.370110+0000", 2024-09-13T11:23:47.554 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:47.554 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:47.554 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:47.554 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:23:47.555 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.370110+0000", 2024-09-13T11:23:47.555 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:47.555 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:47.555 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:47.555 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:23:47.555 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.370112+0000", 2024-09-13T11:23:47.555 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.384e-06 2024-09-13T11:23:47.555 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:47.555 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:47.555 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:23:47.555 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.370175+0000", 2024-09-13T11:23:47.555 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.2529999999999999e-05 2024-09-13T11:23:47.555 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:47.556 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:47.556 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:23:47.556 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.502612+0000", 2024-09-13T11:23:47.556 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.13243751400000001 2024-09-13T11:23:47.556 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:47.556 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:47.556 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:23:47.556 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.502642+0000", 2024-09-13T11:23:47.556 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.9770000000000001e-05 2024-09-13T11:23:47.556 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:47.556 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:47.557 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:23:47.557 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.502808+0000", 2024-09-13T11:23:47.557 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00016618399999999999 2024-09-13T11:23:47.557 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:47.557 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:47.557 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:23:47.557 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.502832+0000", 2024-09-13T11:23:47.557 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.4136000000000001e-05 2024-09-13T11:23:47.557 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:47.557 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:47.557 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:23:47.557 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.504486+0000", 2024-09-13T11:23:47.557 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001653949 2024-09-13T11:23:47.558 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:47.558 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:23:47.558 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:47.558 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:47.558 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:47.558 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPushReply(3.12 76/69 [PushReplyOp(3:48826e21:::benchmark_data_smithi177_31116_object33140:head)])", 2024-09-13T11:23:47.558 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:32.427958+0000", 2024-09-13T11:23:47.558 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 15.034418893, 2024-09-13T11:23:47.558 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.115554986, 2024-09-13T11:23:47.558 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:23:47.558 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:23:47.558 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-09-13T11:23:47.559 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:23:47.559 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:47.559 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:23:47.559 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.427958+0000", 2024-09-13T11:23:47.559 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:47.559 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:47.559 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:47.559 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:23:47.559 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.427958+0000", 2024-09-13T11:23:47.559 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:47.559 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:47.559 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:47.560 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:23:47.560 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.427960+0000", 2024-09-13T11:23:47.560 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.04e-06 2024-09-13T11:23:47.560 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:47.560 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:47.560 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:23:47.560 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.427969+0000", 2024-09-13T11:23:47.560 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.5170000000000005e-06 2024-09-13T11:23:47.560 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:47.560 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:47.560 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:23:47.560 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.542294+0000", 2024-09-13T11:23:47.560 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.11432479 2024-09-13T11:23:47.561 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:47.561 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:47.561 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:23:47.561 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.542318+0000", 2024-09-13T11:23:47.561 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.3790000000000001e-05 2024-09-13T11:23:47.561 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:47.561 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:47.561 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:23:47.561 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.542456+0000", 2024-09-13T11:23:47.561 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000137911 2024-09-13T11:23:47.561 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:47.561 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:47.561 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:23:47.561 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.543513+0000", 2024-09-13T11:23:47.562 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0010569380000000001 2024-09-13T11:23:47.562 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:47.562 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:23:47.562 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:47.562 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:47.562 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:47.562 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPush(3.15 76/70 [PushOp(3:ab4c41a1:::benchmark_data_smithi177_31116_object23392:head, version: 55'1416, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:ab4c41a1:::benchmark_data_smithi177_31116_object23392:head@55'1416, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:23:47.562 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:32.723597+0000", 2024-09-13T11:23:47.562 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 14.738779434, 2024-09-13T11:23:47.562 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.205507309, 2024-09-13T11:23:47.562 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:23:47.562 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:23:47.562 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:23:47.563 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:23:47.563 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:47.563 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:23:47.563 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.723597+0000", 2024-09-13T11:23:47.563 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:47.563 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:47.563 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:47.563 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:23:47.563 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.723597+0000", 2024-09-13T11:23:47.563 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:47.563 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:47.563 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:47.563 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:23:47.564 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.723599+0000", 2024-09-13T11:23:47.564 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.1639999999999999e-06 2024-09-13T11:23:47.564 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:47.564 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:47.564 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:23:47.564 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.723661+0000", 2024-09-13T11:23:47.564 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.1851000000000001e-05 2024-09-13T11:23:47.564 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:47.564 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:47.564 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:23:47.564 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.927067+0000", 2024-09-13T11:23:47.564 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.20340633899999999 2024-09-13T11:23:47.564 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:47.565 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:47.565 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:23:47.565 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.927102+0000", 2024-09-13T11:23:47.565 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.4178999999999999e-05 2024-09-13T11:23:47.565 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:47.565 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:47.565 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:23:47.565 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.927424+0000", 2024-09-13T11:23:47.565 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000322536 2024-09-13T11:23:47.565 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:47.565 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:47.565 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:23:47.565 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.927457+0000", 2024-09-13T11:23:47.566 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.3191000000000001e-05 2024-09-13T11:23:47.566 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:47.566 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:47.566 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:23:47.566 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.929104+0000", 2024-09-13T11:23:47.566 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001647049 2024-09-13T11:23:47.566 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:47.566 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:23:47.566 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:47.566 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:47.566 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:47.566 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop_reply(client.4665.0:55795 3.12 e79/69 ondisk, result = 0)", 2024-09-13T11:23:47.566 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:35.893777+0000", 2024-09-13T11:23:47.567 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 11.568599801, 2024-09-13T11:23:47.567 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.189196325, 2024-09-13T11:23:47.567 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:23:47.567 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:23:47.567 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:23:47.567 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:23:47.567 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:47.567 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:23:47.567 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:35.893777+0000", 2024-09-13T11:23:47.567 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:47.567 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:47.567 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:47.567 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:23:47.568 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:35.893777+0000", 2024-09-13T11:23:47.568 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:47.568 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:47.568 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:47.568 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:23:47.568 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:35.893778+0000", 2024-09-13T11:23:47.568 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.6670000000000001e-06 2024-09-13T11:23:47.568 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:47.568 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:47.568 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:23:47.568 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:35.893784+0000", 2024-09-13T11:23:47.568 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.1989999999999997e-06 2024-09-13T11:23:47.569 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:47.569 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:47.569 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:23:47.569 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:36.082661+0000", 2024-09-13T11:23:47.569 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.18887743600000001 2024-09-13T11:23:47.569 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:47.569 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:47.569 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:23:47.569 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:36.082666+0000", 2024-09-13T11:23:47.569 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.2270000000000001e-06 2024-09-13T11:23:47.569 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:47.569 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:47.569 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:23:47.569 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:36.082784+0000", 2024-09-13T11:23:47.570 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00011776400000000001 2024-09-13T11:23:47.570 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:47.570 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:47.570 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:23:47.570 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:36.082799+0000", 2024-09-13T11:23:47.570 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5058999999999999e-05 2024-09-13T11:23:47.570 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:47.570 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:47.570 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:23:47.570 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:36.082973+0000", 2024-09-13T11:23:47.570 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00017397300000000001 2024-09-13T11:23:47.570 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:47.570 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:23:47.571 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:47.571 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:47.571 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:47.571 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPush(3.1f 84/78 [PushOp(3:fd5621e0:::benchmark_data_smithi177_31116_object7011:head, version: 46'483, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:fd5621e0:::benchmark_data_smithi177_31116_object7011:head@46'483, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:23:47.571 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:41.510296+0000", 2024-09-13T11:23:47.571 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 5.9520803180000001, 2024-09-13T11:23:47.571 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.41419089999999997, 2024-09-13T11:23:47.571 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:23:47.571 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:23:47.571 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:23:47.571 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:23:47.571 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:47.571 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:23:47.572 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:41.510296+0000", 2024-09-13T11:23:47.572 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:47.572 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:47.572 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:47.572 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:23:47.572 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:41.510296+0000", 2024-09-13T11:23:47.572 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:23:47.572 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:47.572 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:47.572 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:23:47.572 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:41.510305+0000", 2024-09-13T11:23:47.572 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.9700000000000005e-06 2024-09-13T11:23:47.572 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:47.573 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:47.573 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:23:47.573 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:41.510458+0000", 2024-09-13T11:23:47.573 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000152418 2024-09-13T11:23:47.573 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:47.573 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:47.573 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:23:47.573 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:41.918225+0000", 2024-09-13T11:23:47.573 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.40776698 2024-09-13T11:23:47.573 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:47.573 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:47.573 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:23:47.573 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:41.918251+0000", 2024-09-13T11:23:47.574 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.6531000000000001e-05 2024-09-13T11:23:47.574 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:47.574 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:47.574 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:23:47.574 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:41.922211+0000", 2024-09-13T11:23:47.574 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0039603040000000004 2024-09-13T11:23:47.574 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:47.574 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:47.574 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:23:47.574 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:41.922253+0000", 2024-09-13T11:23:47.574 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.1155999999999997e-05 2024-09-13T11:23:47.574 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:23:47.574 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:23:47.575 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:23:47.575 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:41.924487+0000", 2024-09-13T11:23:47.575 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.002234541 2024-09-13T11:23:47.575 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:47.575 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:23:47.575 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:47.575 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:23:47.575 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:23:47.575 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-13T11:23:47.576 INFO:tasks.ceph.osd.1.smithi092.stderr:2024-09-13T11:23:47.571+0000 7fe7789c5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33718) UID: 0 2024-09-13T11:23:47.577 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-13T11:23:47.678 INFO:tasks.ceph.osd.5.smithi092.stderr:2024-09-13T11:23:47.675+0000 7f6f0b705640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26346) UID: 0 2024-09-13T11:23:47.729 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-09-13T11:23:47.729 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [], 2024-09-13T11:23:47.729 INFO:teuthology.orchestra.run.smithi177.stdout: "num_ops": 0 2024-09-13T11:23:47.730 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-09-13T11:23:47.750 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-13T11:23:47.800 INFO:tasks.ceph.osd.11.smithi177.stderr:2024-09-13T11:23:47.796+0000 7f24792c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26580) UID: 0 2024-09-13T11:23:47.879 INFO:tasks.ceph.osd.14.smithi162.stderr:2024-09-13T11:23:47.877+0000 7f235c0db640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26319) UID: 0 2024-09-13T11:23:47.896 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-09-13T11:23:47.897 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [], 2024-09-13T11:23:47.897 INFO:teuthology.orchestra.run.smithi177.stdout: "complaint_time": 30, 2024-09-13T11:23:47.897 INFO:teuthology.orchestra.run.smithi177.stdout: "num_blocked_ops": 0 2024-09-13T11:23:47.897 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-09-13T11:23:47.910 INFO:teuthology.orchestra.run.smithi001.stderr:reweighted osd.7 to 1 (10000) 2024-09-13T11:23:47.919 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-13T11:23:47.938 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 8 1 2024-09-13T11:23:48.006 INFO:tasks.radosbench.radosbench.0.smithi177.stdout: 81 16 64994 64978 50.132 52.1875 0.0111411 0.0199376 2024-09-13T11:23:48.006 INFO:tasks.ceph.osd.11.smithi177.stderr:2024-09-13T11:23:48.000+0000 7f24792c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26580) UID: 0 2024-09-13T11:23:48.082 INFO:tasks.ceph.osd.3.smithi177.stderr:2024-09-13T11:23:48.076+0000 7f10adba4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26574) UID: 0 2024-09-13T11:23:48.142 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-09-13T11:23:48.142 INFO:teuthology.orchestra.run.smithi177.stdout: "size": 20, 2024-09-13T11:23:48.142 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 600, 2024-09-13T11:23:48.142 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [ 2024-09-13T11:23:48.142 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.142 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:30852 3.a 3:522fb0d3:::benchmark_data_smithi177_31116_object30851:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e59)", 2024-09-13T11:23:48.142 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:09.538712+0000", 2024-09-13T11:23:48.142 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 38.601299005999998, 2024-09-13T11:23:48.142 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.19661263300000001, 2024-09-13T11:23:48.143 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:23:48.143 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:23:48.143 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:48.143 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:23:48.143 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:23:48.143 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:48.143 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 30852 2024-09-13T11:23:48.143 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.143 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:23:48.143 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.143 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:23:48.143 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.538712+0000", 2024-09-13T11:23:48.143 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:48.143 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.144 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.144 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:23:48.144 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.538712+0000", 2024-09-13T11:23:48.144 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:48.144 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.144 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.144 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:23:48.144 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.538714+0000", 2024-09-13T11:23:48.144 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.658e-06 2024-09-13T11:23:48.144 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.144 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.144 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:23:48.144 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.538735+0000", 2024-09-13T11:23:48.145 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.141e-05 2024-09-13T11:23:48.145 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.145 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.145 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:23:48.145 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.538747+0000", 2024-09-13T11:23:48.145 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.2258e-05 2024-09-13T11:23:48.145 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.145 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.145 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:23:48.145 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.538754+0000", 2024-09-13T11:23:48.146 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.2859999999999996e-06 2024-09-13T11:23:48.146 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.146 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.146 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:23:48.146 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.538798+0000", 2024-09-13T11:23:48.146 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.4258000000000002e-05 2024-09-13T11:23:48.146 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.146 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.146 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:23:48.146 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.538890+0000", 2024-09-13T11:23:48.146 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.1676999999999997e-05 2024-09-13T11:23:48.146 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.146 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.147 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 7", 2024-09-13T11:23:48.147 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.538991+0000", 2024-09-13T11:23:48.147 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00010173700000000001 2024-09-13T11:23:48.147 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.147 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.147 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:23:48.147 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.540171+0000", 2024-09-13T11:23:48.147 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0011800479999999999 2024-09-13T11:23:48.147 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.147 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.147 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:48.147 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.735031+0000", 2024-09-13T11:23:48.147 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.194859798 2024-09-13T11:23:48.148 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.148 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.148 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:23:48.148 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.735197+0000", 2024-09-13T11:23:48.148 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00016569499999999999 2024-09-13T11:23:48.148 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.148 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.148 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:23:48.148 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.735325+0000", 2024-09-13T11:23:48.148 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00012780800000000001 2024-09-13T11:23:48.148 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:48.149 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:23:48.149 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:48.149 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.149 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.149 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop_reply(client.4665.0:30852 3.a e59/30 ondisk, result = 0)", 2024-09-13T11:23:48.149 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:09.540458+0000", 2024-09-13T11:23:48.149 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 38.599553518, 2024-09-13T11:23:48.149 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.194887, 2024-09-13T11:23:48.149 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:23:48.149 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:23:48.149 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:23:48.149 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:23:48.149 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.149 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:23:48.150 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.540458+0000", 2024-09-13T11:23:48.150 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:48.150 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.150 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.150 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:23:48.150 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.540458+0000", 2024-09-13T11:23:48.150 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:48.150 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.150 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.150 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:23:48.150 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.540459+0000", 2024-09-13T11:23:48.150 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.2699999999999999e-06 2024-09-13T11:23:48.150 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.151 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.151 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:23:48.151 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.540463+0000", 2024-09-13T11:23:48.151 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.8500000000000004e-06 2024-09-13T11:23:48.151 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.151 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.151 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:23:48.151 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.734813+0000", 2024-09-13T11:23:48.151 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.194350047 2024-09-13T11:23:48.151 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.151 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.151 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:23:48.151 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.734832+0000", 2024-09-13T11:23:48.151 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.9610999999999999e-05 2024-09-13T11:23:48.152 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.152 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.152 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:23:48.152 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.734989+0000", 2024-09-13T11:23:48.152 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00015694799999999999 2024-09-13T11:23:48.152 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.152 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.152 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:23:48.152 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.735015+0000", 2024-09-13T11:23:48.153 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.6165000000000001e-05 2024-09-13T11:23:48.153 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.153 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.153 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:23:48.153 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.735345+0000", 2024-09-13T11:23:48.154 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00032910900000000001 2024-09-13T11:23:48.154 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:48.154 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:23:48.154 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:48.154 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.154 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.154 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPush(3.17 75/70 [PushOp(3:eda94455:::benchmark_data_smithi177_31116_object13017:head, version: 50'788, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:eda94455:::benchmark_data_smithi177_31116_object13017:head@50'788, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:23:48.154 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:31.038029+0000", 2024-09-13T11:23:48.154 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 17.101982453000002, 2024-09-13T11:23:48.155 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.42064623499999998, 2024-09-13T11:23:48.155 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:23:48.155 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:23:48.155 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:23:48.155 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:23:48.155 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.155 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:23:48.155 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.038029+0000", 2024-09-13T11:23:48.155 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:48.155 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.155 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.155 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:23:48.155 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.038029+0000", 2024-09-13T11:23:48.155 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:48.156 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.156 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.156 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:23:48.156 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.038030+0000", 2024-09-13T11:23:48.156 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.499e-06 2024-09-13T11:23:48.156 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.156 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.156 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:23:48.156 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.038088+0000", 2024-09-13T11:23:48.156 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.7771e-05 2024-09-13T11:23:48.156 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.156 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.156 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:23:48.157 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.457025+0000", 2024-09-13T11:23:48.157 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.41893754900000002 2024-09-13T11:23:48.157 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.157 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.157 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:23:48.157 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.457073+0000", 2024-09-13T11:23:48.157 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.7450999999999997e-05 2024-09-13T11:23:48.157 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.157 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.157 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:23:48.157 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.457304+0000", 2024-09-13T11:23:48.157 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000231351 2024-09-13T11:23:48.157 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.158 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.158 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:23:48.158 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.457331+0000", 2024-09-13T11:23:48.158 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.6372000000000001e-05 2024-09-13T11:23:48.158 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.158 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.158 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:23:48.158 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.458675+0000", 2024-09-13T11:23:48.158 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001344242 2024-09-13T11:23:48.159 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:48.159 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:23:48.159 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:48.159 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.159 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.159 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "rep_scrubmap(3.a e75 from shard 7)", 2024-09-13T11:23:48.159 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:32.151191+0000", 2024-09-13T11:23:48.159 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 15.9888204, 2024-09-13T11:23:48.159 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.164608173, 2024-09-13T11:23:48.159 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:23:48.159 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:23:48.159 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:23:48.159 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:23:48.159 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.160 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:23:48.160 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:32.151191+0000", 2024-09-13T11:23:48.160 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:48.160 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.160 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.160 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:23:48.160 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:32.151191+0000", 2024-09-13T11:23:48.160 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:48.160 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.160 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.160 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:23:48.160 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:32.151192+0000", 2024-09-13T11:23:48.161 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.28e-06 2024-09-13T11:23:48.161 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.161 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.161 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:23:48.161 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:32.151220+0000", 2024-09-13T11:23:48.161 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.8045e-05 2024-09-13T11:23:48.161 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.161 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.161 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:23:48.161 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:32.315480+0000", 2024-09-13T11:23:48.161 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.16425959000000001 2024-09-13T11:23:48.161 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.161 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.162 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:23:48.162 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:32.315488+0000", 2024-09-13T11:23:48.162 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.0819999999999999e-06 2024-09-13T11:23:48.162 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.162 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.162 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:23:48.162 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:32.315558+0000", 2024-09-13T11:23:48.162 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.0500999999999995e-05 2024-09-13T11:23:48.162 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.162 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.162 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:23:48.162 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:32.315570+0000", 2024-09-13T11:23:48.162 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.2238999999999999e-05 2024-09-13T11:23:48.163 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.163 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.163 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:23:48.163 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:32.315799+0000", 2024-09-13T11:23:48.163 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00022843599999999999 2024-09-13T11:23:48.163 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:48.163 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:23:48.163 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:48.163 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.163 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.163 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPush(3.14 79/70 [PushOp(3:2c86f72b:::benchmark_data_smithi177_31116_object32132:head, version: 60'2019, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:2c86f72b:::benchmark_data_smithi177_31116_object32132:head@60'2019, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:23:48.163 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:35.768489+0000", 2024-09-13T11:23:48.163 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 12.371522236000001, 2024-09-13T11:23:48.163 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.179450581, 2024-09-13T11:23:48.164 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:23:48.164 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:23:48.164 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:23:48.164 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:23:48.164 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.164 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:23:48.164 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:35.768489+0000", 2024-09-13T11:23:48.164 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:48.164 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.164 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.164 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:23:48.164 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:35.768489+0000", 2024-09-13T11:23:48.164 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:48.165 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.165 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.165 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:23:48.165 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:35.768490+0000", 2024-09-13T11:23:48.165 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5999999999999999e-06 2024-09-13T11:23:48.165 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.165 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.165 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:23:48.165 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:35.768592+0000", 2024-09-13T11:23:48.165 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00010194799999999999 2024-09-13T11:23:48.165 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.165 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.165 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:23:48.165 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:35.946933+0000", 2024-09-13T11:23:48.166 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.17834060800000001 2024-09-13T11:23:48.166 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.166 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.166 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:23:48.166 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:35.946953+0000", 2024-09-13T11:23:48.166 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.0347e-05 2024-09-13T11:23:48.166 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.166 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.166 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:23:48.166 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:35.947056+0000", 2024-09-13T11:23:48.166 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000103164 2024-09-13T11:23:48.166 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.166 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.166 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:23:48.167 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:35.947071+0000", 2024-09-13T11:23:48.167 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.4854e-05 2024-09-13T11:23:48.167 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.167 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.167 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:23:48.167 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:35.947939+0000", 2024-09-13T11:23:48.167 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00086806000000000003 2024-09-13T11:23:48.167 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:48.167 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:23:48.167 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:48.167 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.167 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.167 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPushReply(3.1d 79/78 [PushReplyOp(3:bbdea640:::benchmark_data_smithi177_31116_object1641:head)])", 2024-09-13T11:23:48.167 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:36.125019+0000", 2024-09-13T11:23:48.168 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 12.014991896, 2024-09-13T11:23:48.168 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.129235131, 2024-09-13T11:23:48.168 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:23:48.168 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:23:48.168 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-13T11:23:48.168 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:23:48.168 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.168 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:23:48.168 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:36.125019+0000", 2024-09-13T11:23:48.168 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:48.168 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.168 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.168 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:23:48.169 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:36.125019+0000", 2024-09-13T11:23:48.169 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:48.169 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.169 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.169 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:23:48.169 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:36.125021+0000", 2024-09-13T11:23:48.169 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.7290000000000001e-06 2024-09-13T11:23:48.169 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.169 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.169 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:23:48.169 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:36.125027+0000", 2024-09-13T11:23:48.169 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.4570000000000002e-06 2024-09-13T11:23:48.169 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.169 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.170 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:23:48.170 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:36.252505+0000", 2024-09-13T11:23:48.170 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.127477589 2024-09-13T11:23:48.170 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.170 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.170 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:23:48.170 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:36.252511+0000", 2024-09-13T11:23:48.170 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.5390000000000004e-06 2024-09-13T11:23:48.170 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.170 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.170 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:23:48.170 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:36.252569+0000", 2024-09-13T11:23:48.170 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.7621000000000003e-05 2024-09-13T11:23:48.170 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.170 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.171 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:23:48.171 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:36.254254+0000", 2024-09-13T11:23:48.171 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0016851959999999999 2024-09-13T11:23:48.171 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:48.171 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:23:48.171 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:48.171 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.171 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.171 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:56066 3.a 3:561a1dd2:::benchmark_data_smithi177_31116_object56065:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e79)", 2024-09-13T11:23:48.171 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:36.193843+0000", 2024-09-13T11:23:48.171 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 11.946168243000001, 2024-09-13T11:23:48.171 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.13659179299999999, 2024-09-13T11:23:48.171 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:23:48.172 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:23:48.172 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:48.172 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:23:48.172 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:23:48.172 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:48.172 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 56066 2024-09-13T11:23:48.172 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.172 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:23:48.172 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.172 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:23:48.172 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:36.193843+0000", 2024-09-13T11:23:48.172 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:48.172 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.173 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.173 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:23:48.173 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:36.193843+0000", 2024-09-13T11:23:48.173 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:48.173 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.173 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.173 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:23:48.173 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:36.193846+0000", 2024-09-13T11:23:48.173 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.0299999999999998e-06 2024-09-13T11:23:48.173 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.173 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.173 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:23:48.173 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:36.193873+0000", 2024-09-13T11:23:48.173 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.7464e-05 2024-09-13T11:23:48.174 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.174 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.174 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:23:48.174 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:36.193895+0000", 2024-09-13T11:23:48.174 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.2152000000000001e-05 2024-09-13T11:23:48.174 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.174 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.174 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:23:48.174 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:36.193906+0000", 2024-09-13T11:23:48.174 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0328e-05 2024-09-13T11:23:48.174 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.174 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.174 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:23:48.174 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:36.193971+0000", 2024-09-13T11:23:48.174 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.5284999999999996e-05 2024-09-13T11:23:48.175 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.175 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.175 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:23:48.175 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:36.194112+0000", 2024-09-13T11:23:48.175 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00014097400000000001 2024-09-13T11:23:48.175 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.175 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.175 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 7", 2024-09-13T11:23:48.175 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:36.194235+0000", 2024-09-13T11:23:48.175 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000122724 2024-09-13T11:23:48.175 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.175 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.175 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:23:48.176 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:36.196065+0000", 2024-09-13T11:23:48.176 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0018305000000000001 2024-09-13T11:23:48.176 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.176 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.176 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:48.176 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:36.330327+0000", 2024-09-13T11:23:48.176 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.13426153900000001 2024-09-13T11:23:48.176 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.176 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.176 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:23:48.176 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:36.330387+0000", 2024-09-13T11:23:48.176 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.0681999999999999e-05 2024-09-13T11:23:48.176 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.177 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.177 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:23:48.177 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:36.330435+0000", 2024-09-13T11:23:48.177 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.7114999999999999e-05 2024-09-13T11:23:48.177 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:48.177 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:23:48.177 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:48.177 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.177 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.177 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:57499 3.a 3:53f09f43:::benchmark_data_smithi177_31116_object57498:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e80)", 2024-09-13T11:23:48.177 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:37.500054+0000", 2024-09-13T11:23:48.177 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 10.639957218999999, 2024-09-13T11:23:48.177 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.12624653999999999, 2024-09-13T11:23:48.177 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:23:48.178 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:23:48.178 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:48.178 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:23:48.178 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:23:48.178 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:48.178 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 57499 2024-09-13T11:23:48.178 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.178 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:23:48.178 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.178 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:23:48.178 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:37.500054+0000", 2024-09-13T11:23:48.178 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:48.178 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.178 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.179 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:23:48.179 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:37.500054+0000", 2024-09-13T11:23:48.179 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:48.179 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.179 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.179 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:23:48.179 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:37.500056+0000", 2024-09-13T11:23:48.179 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.419e-06 2024-09-13T11:23:48.179 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.179 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.179 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:23:48.179 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:37.500079+0000", 2024-09-13T11:23:48.179 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.2719000000000001e-05 2024-09-13T11:23:48.179 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.180 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.180 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:23:48.180 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:37.500089+0000", 2024-09-13T11:23:48.180 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0441999999999999e-05 2024-09-13T11:23:48.180 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.180 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.180 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:23:48.180 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:37.500103+0000", 2024-09-13T11:23:48.180 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.4056000000000001e-05 2024-09-13T11:23:48.180 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.180 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.180 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:23:48.180 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:37.500158+0000", 2024-09-13T11:23:48.181 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.4509999999999998e-05 2024-09-13T11:23:48.181 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.181 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.181 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:23:48.181 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:37.500288+0000", 2024-09-13T11:23:48.181 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00013026199999999999 2024-09-13T11:23:48.181 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.181 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.181 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 7", 2024-09-13T11:23:48.181 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:37.500425+0000", 2024-09-13T11:23:48.181 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00013640500000000001 2024-09-13T11:23:48.181 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.181 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.181 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:23:48.181 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:37.502593+0000", 2024-09-13T11:23:48.182 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0021686000000000001 2024-09-13T11:23:48.182 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.182 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.182 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:48.182 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:37.626231+0000", 2024-09-13T11:23:48.182 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.12363742699999999 2024-09-13T11:23:48.182 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.182 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.182 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:23:48.182 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:37.626263+0000", 2024-09-13T11:23:48.182 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.1841000000000002e-05 2024-09-13T11:23:48.182 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.182 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.182 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:23:48.183 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:37.626300+0000", 2024-09-13T11:23:48.183 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.7858999999999998e-05 2024-09-13T11:23:48.183 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:48.183 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:23:48.183 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:48.183 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.183 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.183 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPush(3.17 83/70 [PushOp(3:eb574268:::benchmark_data_smithi177_31116_object40622:head, version: 64'2475, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:eb574268:::benchmark_data_smithi177_31116_object40622:head@64'2475, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:23:48.183 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:40.760042+0000", 2024-09-13T11:23:48.183 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 7.3799687360000004, 2024-09-13T11:23:48.183 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.30544642100000002, 2024-09-13T11:23:48.183 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:23:48.183 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:23:48.183 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:23:48.184 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:23:48.184 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.184 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:23:48.184 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:40.760042+0000", 2024-09-13T11:23:48.184 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:48.184 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.184 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.184 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:23:48.184 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:40.760042+0000", 2024-09-13T11:23:48.184 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:48.184 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.184 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.184 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:23:48.185 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:40.760044+0000", 2024-09-13T11:23:48.185 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.511e-06 2024-09-13T11:23:48.185 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.185 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.185 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:23:48.185 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:40.760149+0000", 2024-09-13T11:23:48.185 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000105473 2024-09-13T11:23:48.185 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.185 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.185 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:23:48.185 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.064465+0000", 2024-09-13T11:23:48.185 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.30431613800000001 2024-09-13T11:23:48.185 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.186 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.186 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:23:48.186 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.064479+0000", 2024-09-13T11:23:48.186 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3586e-05 2024-09-13T11:23:48.186 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.186 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.186 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:23:48.186 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.064553+0000", 2024-09-13T11:23:48.186 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.3646000000000006e-05 2024-09-13T11:23:48.186 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.186 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.186 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:23:48.186 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.064568+0000", 2024-09-13T11:23:48.186 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.52e-05 2024-09-13T11:23:48.186 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.187 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.187 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:23:48.187 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.065489+0000", 2024-09-13T11:23:48.187 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00092086700000000002 2024-09-13T11:23:48.187 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:48.187 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:23:48.187 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:48.187 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.187 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.187 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPushReply(3.1a 83/78 [PushReplyOp(3:5c3834bd:::benchmark_data_smithi177_31116_object8669:head)])", 2024-09-13T11:23:48.187 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:40.798236+0000", 2024-09-13T11:23:48.187 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 7.3417753059999997, 2024-09-13T11:23:48.187 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.29701520999999997, 2024-09-13T11:23:48.188 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:23:48.188 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:23:48.188 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-13T11:23:48.188 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:23:48.188 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.188 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:23:48.188 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:40.798236+0000", 2024-09-13T11:23:48.188 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:48.188 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.188 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.189 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:23:48.189 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:40.798236+0000", 2024-09-13T11:23:48.189 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:48.189 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.189 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.189 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:23:48.189 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:40.798237+0000", 2024-09-13T11:23:48.189 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.0500000000000002e-07 2024-09-13T11:23:48.189 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.189 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.189 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:23:48.189 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:40.798240+0000", 2024-09-13T11:23:48.189 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.4819999999999999e-06 2024-09-13T11:23:48.190 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.190 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.190 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:23:48.190 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.093360+0000", 2024-09-13T11:23:48.190 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.29512009 2024-09-13T11:23:48.190 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.190 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.190 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:23:48.190 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.093372+0000", 2024-09-13T11:23:48.190 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.2221e-05 2024-09-13T11:23:48.190 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.190 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.190 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:23:48.190 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.093482+0000", 2024-09-13T11:23:48.191 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000109571 2024-09-13T11:23:48.191 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.191 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.191 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:23:48.191 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.095251+0000", 2024-09-13T11:23:48.191 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001768941 2024-09-13T11:23:48.191 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:48.191 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:23:48.191 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:48.191 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.191 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.191 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPushReply(3.1a 83/78 [PushReplyOp(3:58c7ffc0:::benchmark_data_smithi177_31116_object8738:head)])", 2024-09-13T11:23:48.191 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:40.805941+0000", 2024-09-13T11:23:48.191 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 7.3340700520000004, 2024-09-13T11:23:48.191 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.30315678499999998, 2024-09-13T11:23:48.192 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:23:48.192 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:23:48.192 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-13T11:23:48.192 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:23:48.192 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.192 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:23:48.192 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:40.805941+0000", 2024-09-13T11:23:48.192 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:48.192 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.192 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.192 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:23:48.192 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:40.805941+0000", 2024-09-13T11:23:48.192 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:48.192 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.193 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.193 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:23:48.193 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:40.805942+0000", 2024-09-13T11:23:48.193 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3769999999999999e-06 2024-09-13T11:23:48.193 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.193 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.193 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:23:48.193 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:40.805947+0000", 2024-09-13T11:23:48.193 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.8980000000000004e-06 2024-09-13T11:23:48.193 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.193 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.193 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:23:48.193 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.106505+0000", 2024-09-13T11:23:48.193 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.30055777700000003 2024-09-13T11:23:48.194 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.194 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.194 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:23:48.194 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.106512+0000", 2024-09-13T11:23:48.194 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.3749999999999997e-06 2024-09-13T11:23:48.194 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.194 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.194 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:23:48.194 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.106566+0000", 2024-09-13T11:23:48.194 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.3103999999999998e-05 2024-09-13T11:23:48.194 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.194 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.194 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:23:48.194 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.109098+0000", 2024-09-13T11:23:48.194 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0025322539999999998 2024-09-13T11:23:48.195 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:48.195 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:23:48.195 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:48.195 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.195 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.195 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPushReply(3.1a 83/78 [PushReplyOp(3:58c7ffc0:::benchmark_data_smithi177_31116_object8738:head)])", 2024-09-13T11:23:48.195 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:40.805973+0000", 2024-09-13T11:23:48.195 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 7.3340380490000001, 2024-09-13T11:23:48.195 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.28729672099999998, 2024-09-13T11:23:48.195 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:23:48.195 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:23:48.195 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-13T11:23:48.195 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:23:48.196 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.196 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:23:48.196 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:40.805973+0000", 2024-09-13T11:23:48.196 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:48.196 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.196 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.196 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:23:48.196 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:40.805973+0000", 2024-09-13T11:23:48.196 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:48.196 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.196 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.196 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:23:48.196 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:40.805974+0000", 2024-09-13T11:23:48.196 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.8599999999999998e-07 2024-09-13T11:23:48.197 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.197 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.197 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:23:48.197 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:40.805976+0000", 2024-09-13T11:23:48.197 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.2809999999999998e-06 2024-09-13T11:23:48.197 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.197 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.197 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:23:48.197 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.091588+0000", 2024-09-13T11:23:48.197 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.28561173899999998 2024-09-13T11:23:48.197 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.197 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.197 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:23:48.197 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.091596+0000", 2024-09-13T11:23:48.198 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.0520000000000002e-06 2024-09-13T11:23:48.198 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.198 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.198 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:23:48.198 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.091907+0000", 2024-09-13T11:23:48.198 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00031179599999999998 2024-09-13T11:23:48.198 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.198 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.198 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:23:48.198 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.093270+0000", 2024-09-13T11:23:48.198 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0013622669999999999 2024-09-13T11:23:48.198 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:48.198 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:23:48.198 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:48.199 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.199 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.199 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPushReply(3.1a 84/78 [PushReplyOp(3:581ea97e:::benchmark_data_smithi177_31116_object10255:head)])", 2024-09-13T11:23:48.199 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:41.514228+0000", 2024-09-13T11:23:48.199 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 6.6257825610000003, 2024-09-13T11:23:48.199 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.40695564200000001, 2024-09-13T11:23:48.199 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:23:48.199 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:23:48.199 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-13T11:23:48.199 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:23:48.199 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.199 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:23:48.199 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.514228+0000", 2024-09-13T11:23:48.200 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:48.200 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.200 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.200 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:23:48.200 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.514228+0000", 2024-09-13T11:23:48.200 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:48.200 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.200 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.200 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:23:48.200 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.514230+0000", 2024-09-13T11:23:48.200 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.499e-06 2024-09-13T11:23:48.200 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.200 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.201 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:23:48.201 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.514236+0000", 2024-09-13T11:23:48.201 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.5389999999999998e-06 2024-09-13T11:23:48.201 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.201 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.201 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:23:48.201 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.919787+0000", 2024-09-13T11:23:48.201 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.40555152999999999 2024-09-13T11:23:48.201 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.201 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.201 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:23:48.201 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.919796+0000", 2024-09-13T11:23:48.201 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.4829999999999999e-06 2024-09-13T11:23:48.201 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.202 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.202 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:23:48.202 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.920366+0000", 2024-09-13T11:23:48.202 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00057061700000000004 2024-09-13T11:23:48.202 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.202 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.202 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:23:48.202 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.921184+0000", 2024-09-13T11:23:48.202 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00081797399999999996 2024-09-13T11:23:48.202 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:48.202 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:23:48.202 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:48.202 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.203 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.203 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:60312 3.14 3:2e846241:::benchmark_data_smithi177_31116_object60311:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e85)", 2024-09-13T11:23:48.203 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:42.065725+0000", 2024-09-13T11:23:48.203 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 6.0742859180000002, 2024-09-13T11:23:48.203 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.211323176, 2024-09-13T11:23:48.203 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:23:48.203 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:23:48.203 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:48.203 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:23:48.203 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:23:48.203 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:48.203 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 60312 2024-09-13T11:23:48.203 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.203 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:23:48.204 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.204 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:23:48.204 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.065725+0000", 2024-09-13T11:23:48.204 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:48.204 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.204 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.204 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:23:48.204 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.065725+0000", 2024-09-13T11:23:48.204 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:48.204 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.204 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.204 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:23:48.204 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.065728+0000", 2024-09-13T11:23:48.205 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.3400000000000002e-06 2024-09-13T11:23:48.205 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.205 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.205 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:23:48.205 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.065759+0000", 2024-09-13T11:23:48.205 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.0074e-05 2024-09-13T11:23:48.205 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.205 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.205 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:23:48.205 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.065771+0000", 2024-09-13T11:23:48.205 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.2869e-05 2024-09-13T11:23:48.205 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.205 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.205 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:23:48.206 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.065780+0000", 2024-09-13T11:23:48.206 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.3489999999999992e-06 2024-09-13T11:23:48.206 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.206 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.206 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:23:48.206 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.065837+0000", 2024-09-13T11:23:48.206 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.7444000000000001e-05 2024-09-13T11:23:48.206 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.206 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.206 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:23:48.206 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.065966+0000", 2024-09-13T11:23:48.206 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000128973 2024-09-13T11:23:48.206 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.206 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.207 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 14", 2024-09-13T11:23:48.207 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.066114+0000", 2024-09-13T11:23:48.207 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00014788299999999999 2024-09-13T11:23:48.207 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.207 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.207 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:23:48.207 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.067604+0000", 2024-09-13T11:23:48.207 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001489637 2024-09-13T11:23:48.207 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.207 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.207 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:48.207 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.276963+0000", 2024-09-13T11:23:48.207 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.20935888499999999 2024-09-13T11:23:48.208 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.208 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.208 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:23:48.208 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.277003+0000", 2024-09-13T11:23:48.208 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.0241000000000003e-05 2024-09-13T11:23:48.208 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.208 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.208 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:23:48.208 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.277048+0000", 2024-09-13T11:23:48.208 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.5481e-05 2024-09-13T11:23:48.208 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:48.208 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:23:48.208 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:48.208 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.209 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.209 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop_reply(client.4665.0:60312 3.14 e85/84 ondisk, result = 0)", 2024-09-13T11:23:48.209 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:42.069120+0000", 2024-09-13T11:23:48.209 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 6.0708907979999998, 2024-09-13T11:23:48.209 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.20804031000000001, 2024-09-13T11:23:48.209 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:23:48.209 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:23:48.209 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:23:48.209 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:23:48.209 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.209 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:23:48.209 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.069120+0000", 2024-09-13T11:23:48.209 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:48.209 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.210 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.210 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:23:48.210 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.069120+0000", 2024-09-13T11:23:48.210 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:48.210 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.210 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.210 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:23:48.210 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.069123+0000", 2024-09-13T11:23:48.210 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.289e-06 2024-09-13T11:23:48.210 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.210 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.210 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:23:48.210 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.069128+0000", 2024-09-13T11:23:48.211 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.3530000000000002e-06 2024-09-13T11:23:48.211 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.211 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.211 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:23:48.211 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.276885+0000", 2024-09-13T11:23:48.211 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.207756989 2024-09-13T11:23:48.211 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.211 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.211 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:23:48.212 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.276892+0000", 2024-09-13T11:23:48.212 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.922e-06 2024-09-13T11:23:48.212 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.212 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.212 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:23:48.212 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.276953+0000", 2024-09-13T11:23:48.212 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.0890000000000001e-05 2024-09-13T11:23:48.212 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.212 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.212 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:23:48.212 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.276959+0000", 2024-09-13T11:23:48.212 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.9179999999999999e-06 2024-09-13T11:23:48.212 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.213 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.213 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:23:48.213 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.277161+0000", 2024-09-13T11:23:48.213 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00020194899999999999 2024-09-13T11:23:48.213 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:48.213 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:23:48.213 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:48.213 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.213 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.213 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:60427 3.14 3:2afea229:::benchmark_data_smithi177_31116_object60426:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e85)", 2024-09-13T11:23:48.213 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:42.187231+0000", 2024-09-13T11:23:48.213 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 5.952779692, 2024-09-13T11:23:48.213 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.83900596999999999, 2024-09-13T11:23:48.213 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:23:48.214 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:23:48.214 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:48.214 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:23:48.214 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:23:48.214 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:48.214 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 60427 2024-09-13T11:23:48.214 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.214 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:23:48.214 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.214 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:23:48.214 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.187231+0000", 2024-09-13T11:23:48.214 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:48.214 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.214 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.215 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:23:48.215 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.187231+0000", 2024-09-13T11:23:48.215 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:48.215 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.215 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.215 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:23:48.215 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.187235+0000", 2024-09-13T11:23:48.215 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.3579999999999999e-06 2024-09-13T11:23:48.215 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.215 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.215 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:23:48.215 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.187261+0000", 2024-09-13T11:23:48.215 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.6089e-05 2024-09-13T11:23:48.215 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.215 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.216 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:23:48.216 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.187275+0000", 2024-09-13T11:23:48.216 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.4653e-05 2024-09-13T11:23:48.216 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.216 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.216 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:23:48.216 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.187282+0000", 2024-09-13T11:23:48.216 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.8870000000000003e-06 2024-09-13T11:23:48.216 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.216 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.216 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:23:48.216 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.187336+0000", 2024-09-13T11:23:48.216 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.3875999999999997e-05 2024-09-13T11:23:48.216 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.217 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.217 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:23:48.217 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.187480+0000", 2024-09-13T11:23:48.217 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00014330899999999999 2024-09-13T11:23:48.217 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.217 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.217 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 14", 2024-09-13T11:23:48.217 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.187658+0000", 2024-09-13T11:23:48.217 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00017861900000000001 2024-09-13T11:23:48.217 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.217 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.217 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:23:48.217 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.189353+0000", 2024-09-13T11:23:48.218 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001695144 2024-09-13T11:23:48.218 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.218 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.218 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:48.218 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:43.026081+0000", 2024-09-13T11:23:48.218 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.83672736199999997 2024-09-13T11:23:48.218 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.218 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.218 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:23:48.218 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:43.026169+0000", 2024-09-13T11:23:48.218 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.8411000000000004e-05 2024-09-13T11:23:48.218 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.218 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.219 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:23:48.219 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:43.026237+0000", 2024-09-13T11:23:48.219 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.8262000000000005e-05 2024-09-13T11:23:48.219 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:48.219 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:23:48.219 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:48.219 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.219 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.219 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop_reply(client.4665.0:60427 3.14 e85/84 ondisk, result = 0)", 2024-09-13T11:23:48.219 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:42.190914+0000", 2024-09-13T11:23:48.219 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 5.9490970279999997, 2024-09-13T11:23:48.219 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.83534071499999996, 2024-09-13T11:23:48.219 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:23:48.219 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:23:48.220 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:23:48.220 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:23:48.220 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.220 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:23:48.220 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.190914+0000", 2024-09-13T11:23:48.220 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:48.220 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.220 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.220 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:23:48.220 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.190914+0000", 2024-09-13T11:23:48.220 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:48.220 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.220 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.221 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:23:48.221 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.190916+0000", 2024-09-13T11:23:48.221 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.84e-06 2024-09-13T11:23:48.221 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.221 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.221 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:23:48.221 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.190921+0000", 2024-09-13T11:23:48.221 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.2039999999999996e-06 2024-09-13T11:23:48.221 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.221 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.221 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:23:48.221 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:43.025926+0000", 2024-09-13T11:23:48.221 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.83500542 2024-09-13T11:23:48.222 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.222 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.222 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:23:48.222 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:43.025937+0000", 2024-09-13T11:23:48.222 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0597e-05 2024-09-13T11:23:48.222 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.222 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.222 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:23:48.222 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:43.026052+0000", 2024-09-13T11:23:48.222 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000114838 2024-09-13T11:23:48.222 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.222 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.222 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:23:48.222 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:43.026070+0000", 2024-09-13T11:23:48.223 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.8131e-05 2024-09-13T11:23:48.223 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.223 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.223 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:23:48.223 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:43.026255+0000", 2024-09-13T11:23:48.223 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000184685 2024-09-13T11:23:48.223 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:48.223 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:23:48.223 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:48.223 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.223 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.223 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPushReply(3.1a 86/78 [PushReplyOp(3:5df655d4:::benchmark_data_smithi177_31116_object12486:head)])", 2024-09-13T11:23:48.223 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:43.457413+0000", 2024-09-13T11:23:48.223 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 4.6825980529999995, 2024-09-13T11:23:48.224 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.13683013999999999, 2024-09-13T11:23:48.224 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:23:48.224 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:23:48.224 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-13T11:23:48.224 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:23:48.224 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.224 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:23:48.224 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:43.457413+0000", 2024-09-13T11:23:48.224 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:48.224 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.224 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.224 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:23:48.224 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:43.457413+0000", 2024-09-13T11:23:48.224 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:48.225 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.225 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.225 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:23:48.225 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:43.457415+0000", 2024-09-13T11:23:48.225 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.632e-06 2024-09-13T11:23:48.225 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.225 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.225 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:23:48.225 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:43.457422+0000", 2024-09-13T11:23:48.225 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.9739999999999996e-06 2024-09-13T11:23:48.225 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.225 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.225 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:23:48.226 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:43.593264+0000", 2024-09-13T11:23:48.226 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.13584195700000001 2024-09-13T11:23:48.226 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.226 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.226 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:23:48.226 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:43.593271+0000", 2024-09-13T11:23:48.226 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.3989999999999998e-06 2024-09-13T11:23:48.226 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.226 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.226 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:23:48.226 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:43.593323+0000", 2024-09-13T11:23:48.226 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.2170999999999999e-05 2024-09-13T11:23:48.226 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.226 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.226 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:23:48.227 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:43.594243+0000", 2024-09-13T11:23:48.227 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00092000700000000003 2024-09-13T11:23:48.227 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:48.227 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:23:48.227 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:48.227 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.227 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.227 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPushReply(3.1d 89/78 [PushReplyOp(3:bc1b7cfb:::benchmark_data_smithi177_31116_object22629:head)])", 2024-09-13T11:23:48.227 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:46.663106+0000", 2024-09-13T11:23:48.227 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 1.476904845, 2024-09-13T11:23:48.227 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.26645533199999999, 2024-09-13T11:23:48.227 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:23:48.227 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:23:48.227 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-13T11:23:48.227 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:23:48.228 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.228 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:23:48.228 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.663106+0000", 2024-09-13T11:23:48.228 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:48.228 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.228 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.228 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:23:48.228 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.663106+0000", 2024-09-13T11:23:48.228 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:48.229 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.229 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.229 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:23:48.229 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.663108+0000", 2024-09-13T11:23:48.229 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.0140000000000001e-06 2024-09-13T11:23:48.229 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.229 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.229 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:23:48.229 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.663116+0000", 2024-09-13T11:23:48.229 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.5440000000000001e-06 2024-09-13T11:23:48.229 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.229 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.229 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:23:48.229 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.928111+0000", 2024-09-13T11:23:48.230 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.26499566899999999 2024-09-13T11:23:48.230 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.230 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.230 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:23:48.230 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.928135+0000", 2024-09-13T11:23:48.230 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.4003e-05 2024-09-13T11:23:48.230 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.230 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.230 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:23:48.230 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.928559+0000", 2024-09-13T11:23:48.230 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000423318 2024-09-13T11:23:48.230 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.230 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.230 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:23:48.231 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.929562+0000", 2024-09-13T11:23:48.231 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0010027840000000001 2024-09-13T11:23:48.231 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:48.231 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:23:48.231 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:48.231 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.231 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.231 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:64349 3.17 e90/85 3:ef87a22f:::benchmark_data_smithi177_31116_object64348:head v 90'3298, mlcod=90'3295)", 2024-09-13T11:23:48.231 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:47.159683+0000", 2024-09-13T11:23:48.231 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 0.980328329, 2024-09-13T11:23:48.231 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.153974149, 2024-09-13T11:23:48.231 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:23:48.231 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:23:48.231 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:48.232 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:23:48.232 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.232 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:23:48.232 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:47.159683+0000", 2024-09-13T11:23:48.232 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:48.232 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.232 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.232 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:23:48.232 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:47.159683+0000", 2024-09-13T11:23:48.232 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:48.232 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.232 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.232 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:23:48.232 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:47.159684+0000", 2024-09-13T11:23:48.233 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.6649999999999999e-06 2024-09-13T11:23:48.233 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.233 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.233 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:23:48.233 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:47.159742+0000", 2024-09-13T11:23:48.233 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.8004999999999998e-05 2024-09-13T11:23:48.233 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.233 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.233 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:23:48.233 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:47.312341+0000", 2024-09-13T11:23:48.233 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.15259900300000001 2024-09-13T11:23:48.233 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.233 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.233 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:23:48.234 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:47.312348+0000", 2024-09-13T11:23:48.234 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.7449999999999998e-06 2024-09-13T11:23:48.234 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.234 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.234 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:23:48.234 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:47.312395+0000", 2024-09-13T11:23:48.234 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.7123999999999998e-05 2024-09-13T11:23:48.234 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.234 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.234 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:23:48.234 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:47.312410+0000", 2024-09-13T11:23:48.234 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.504e-05 2024-09-13T11:23:48.234 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.235 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.235 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:23:48.235 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:47.313619+0000", 2024-09-13T11:23:48.235 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001208558 2024-09-13T11:23:48.235 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.235 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.235 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:23:48.235 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:47.313657+0000", 2024-09-13T11:23:48.235 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.8009000000000001e-05 2024-09-13T11:23:48.235 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:48.235 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:23:48.235 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:48.235 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:48.235 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:23:48.235 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-09-13T11:23:48.237 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-13T11:23:48.238 INFO:tasks.ceph.osd.5.smithi092.stderr:2024-09-13T11:23:48.231+0000 7f6f0b705640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26346) UID: 0 2024-09-13T11:23:48.338 INFO:tasks.ceph.osd.9.smithi092.stderr:2024-09-13T11:23:48.335+0000 7ff5ab3d0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26348) UID: 0 2024-09-13T11:23:48.422 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-09-13T11:23:48.422 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [], 2024-09-13T11:23:48.422 INFO:teuthology.orchestra.run.smithi177.stdout: "num_ops": 0 2024-09-13T11:23:48.422 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-09-13T11:23:48.437 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-13T11:23:48.438 INFO:tasks.ceph.osd.5.smithi092.stderr:2024-09-13T11:23:48.435+0000 7f6f0b705640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26346) UID: 0 2024-09-13T11:23:48.539 INFO:tasks.ceph.osd.12.smithi001.stderr:2024-09-13T11:23:48.532+0000 7f892c6e6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26314) UID: 0 2024-09-13T11:23:48.565 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-09-13T11:23:48.565 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [], 2024-09-13T11:23:48.565 INFO:teuthology.orchestra.run.smithi177.stdout: "complaint_time": 30, 2024-09-13T11:23:48.565 INFO:teuthology.orchestra.run.smithi177.stdout: "num_blocked_ops": 0 2024-09-13T11:23:48.565 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-09-13T11:23:48.582 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-13T11:23:48.640 INFO:tasks.ceph.osd.1.smithi092.stderr:2024-09-13T11:23:48.635+0000 7fe7789c5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33718) UID: 0 2024-09-13T11:23:48.718 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-09-13T11:23:48.718 INFO:teuthology.orchestra.run.smithi177.stdout: "size": 20, 2024-09-13T11:23:48.718 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 600, 2024-09-13T11:23:48.718 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [ 2024-09-13T11:23:48.718 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.718 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPush(3.12 74/69 [PushOp(3:4f110bb2:::benchmark_data_smithi177_31116_object30564:head, version: 59'1932, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:4f110bb2:::benchmark_data_smithi177_31116_object30564:head@59'1932, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:23:48.718 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:30.835784+0000", 2024-09-13T11:23:48.718 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 17.880403203, 2024-09-13T11:23:48.718 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.12151646200000001, 2024-09-13T11:23:48.718 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:23:48.719 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:23:48.719 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:23:48.719 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:23:48.719 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.719 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:23:48.719 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.835784+0000", 2024-09-13T11:23:48.719 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:48.719 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.719 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.719 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:23:48.719 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.835784+0000", 2024-09-13T11:23:48.719 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:48.719 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.720 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.720 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:23:48.720 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.835785+0000", 2024-09-13T11:23:48.720 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.734e-06 2024-09-13T11:23:48.720 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.720 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.720 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:23:48.720 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.835824+0000", 2024-09-13T11:23:48.720 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.8804999999999999e-05 2024-09-13T11:23:48.720 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.720 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.720 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:23:48.720 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.955218+0000", 2024-09-13T11:23:48.721 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.11939419800000001 2024-09-13T11:23:48.721 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.721 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.721 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:23:48.721 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.955253+0000", 2024-09-13T11:23:48.721 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.4254000000000001e-05 2024-09-13T11:23:48.721 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.721 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.721 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:23:48.721 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.955436+0000", 2024-09-13T11:23:48.721 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000183411 2024-09-13T11:23:48.721 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.721 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.722 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:23:48.722 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.955458+0000", 2024-09-13T11:23:48.722 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.2062999999999999e-05 2024-09-13T11:23:48.722 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.722 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.722 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:23:48.722 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.957300+0000", 2024-09-13T11:23:48.722 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0018419969999999999 2024-09-13T11:23:48.722 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:48.722 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:23:48.722 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:48.722 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.722 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.722 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:54586 3.a e74/30 3:55e4d255:::benchmark_data_smithi177_31116_object54585:head v 74'3410, mlcod=74'3407)", 2024-09-13T11:23:48.723 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:30.851720+0000", 2024-09-13T11:23:48.723 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 17.864466371999999, 2024-09-13T11:23:48.723 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.097295277999999999, 2024-09-13T11:23:48.723 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:23:48.723 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:23:48.723 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:48.723 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:23:48.723 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.723 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:23:48.723 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.851720+0000", 2024-09-13T11:23:48.723 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:48.723 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.724 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.724 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:23:48.724 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.851720+0000", 2024-09-13T11:23:48.724 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:48.724 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.724 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.724 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:23:48.724 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.851721+0000", 2024-09-13T11:23:48.724 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.72e-07 2024-09-13T11:23:48.724 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.724 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.724 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:23:48.724 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.851738+0000", 2024-09-13T11:23:48.725 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.6792000000000001e-05 2024-09-13T11:23:48.725 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.725 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.725 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:23:48.725 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.851780+0000", 2024-09-13T11:23:48.725 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.2116000000000001e-05 2024-09-13T11:23:48.725 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.725 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.725 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:23:48.725 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.851785+0000", 2024-09-13T11:23:48.725 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.8840000000000002e-06 2024-09-13T11:23:48.725 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.725 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.725 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:23:48.726 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.942807+0000", 2024-09-13T11:23:48.726 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.091022163000000003 2024-09-13T11:23:48.726 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.726 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.726 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:23:48.726 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.942840+0000", 2024-09-13T11:23:48.726 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.3232999999999998e-05 2024-09-13T11:23:48.726 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.726 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.728 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:23:48.728 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.948917+0000", 2024-09-13T11:23:48.728 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0060771790000000003 2024-09-13T11:23:48.728 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.728 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.728 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:23:48.728 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.949016+0000", 2024-09-13T11:23:48.728 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.8539e-05 2024-09-13T11:23:48.728 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:48.728 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:23:48.728 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:48.728 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.728 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.728 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:54598 3.13 3:cb6b60f3:::benchmark_data_smithi177_31116_object54597:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e74)", 2024-09-13T11:23:48.729 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:30.859951+0000", 2024-09-13T11:23:48.729 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 17.856235326, 2024-09-13T11:23:48.729 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.10132959900000001, 2024-09-13T11:23:48.729 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:23:48.729 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:23:48.729 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:48.729 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:23:48.729 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:23:48.729 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:48.729 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 54598 2024-09-13T11:23:48.729 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.729 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:23:48.729 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.730 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:23:48.730 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.859951+0000", 2024-09-13T11:23:48.730 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:48.730 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.730 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.730 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:23:48.730 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.859951+0000", 2024-09-13T11:23:48.730 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:48.730 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.730 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.730 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:23:48.730 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.859953+0000", 2024-09-13T11:23:48.730 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.573e-06 2024-09-13T11:23:48.730 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.731 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.731 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:23:48.731 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.859972+0000", 2024-09-13T11:23:48.731 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.8975e-05 2024-09-13T11:23:48.731 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.731 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.731 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:23:48.731 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.939205+0000", 2024-09-13T11:23:48.731 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.079233245999999993 2024-09-13T11:23:48.731 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.731 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.731 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:23:48.731 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.939242+0000", 2024-09-13T11:23:48.731 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.7218000000000002e-05 2024-09-13T11:23:48.732 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.732 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.732 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:23:48.732 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.939896+0000", 2024-09-13T11:23:48.732 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00065337499999999998 2024-09-13T11:23:48.732 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.732 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.732 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:23:48.732 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.940165+0000", 2024-09-13T11:23:48.732 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00026946700000000001 2024-09-13T11:23:48.732 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.732 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.732 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 11", 2024-09-13T11:23:48.732 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.940608+0000", 2024-09-13T11:23:48.733 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00044320500000000001 2024-09-13T11:23:48.733 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.733 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.733 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:23:48.733 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.944414+0000", 2024-09-13T11:23:48.733 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0038051869999999998 2024-09-13T11:23:48.733 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.733 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.733 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:48.733 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.960399+0000", 2024-09-13T11:23:48.733 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.015985141000000001 2024-09-13T11:23:48.733 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.733 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.734 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:23:48.734 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.960482+0000", 2024-09-13T11:23:48.734 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.3398000000000003e-05 2024-09-13T11:23:48.734 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.734 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.734 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:23:48.734 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.961281+0000", 2024-09-13T11:23:48.734 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00079881400000000001 2024-09-13T11:23:48.734 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:48.734 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:23:48.734 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:48.734 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.734 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.735 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:54663 3.12 e75/69 3:49ad4b1b:::benchmark_data_smithi177_31116_object54662:head v 75'3095, mlcod=59'1933)", 2024-09-13T11:23:48.735 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:31.033716+0000", 2024-09-13T11:23:48.735 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 17.682471141000001, 2024-09-13T11:23:48.735 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.121029233, 2024-09-13T11:23:48.735 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:23:48.735 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:23:48.735 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:48.735 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:23:48.735 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.735 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:23:48.735 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.033716+0000", 2024-09-13T11:23:48.735 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:48.735 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.735 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.736 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:23:48.736 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.033716+0000", 2024-09-13T11:23:48.736 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:48.736 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.736 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.736 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:23:48.736 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.033724+0000", 2024-09-13T11:23:48.736 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.2910000000000007e-06 2024-09-13T11:23:48.736 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.736 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.736 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:23:48.736 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.033783+0000", 2024-09-13T11:23:48.736 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.8826999999999998e-05 2024-09-13T11:23:48.736 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.737 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.737 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:23:48.737 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.138806+0000", 2024-09-13T11:23:48.737 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.105023701 2024-09-13T11:23:48.737 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.737 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.737 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:23:48.737 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.138814+0000", 2024-09-13T11:23:48.737 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.5340000000000002e-06 2024-09-13T11:23:48.737 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.737 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.737 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:23:48.737 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.138871+0000", 2024-09-13T11:23:48.738 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.6988999999999999e-05 2024-09-13T11:23:48.738 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.738 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.738 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:23:48.738 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.138895+0000", 2024-09-13T11:23:48.738 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.3887e-05 2024-09-13T11:23:48.738 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.738 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.738 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:23:48.738 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.154684+0000", 2024-09-13T11:23:48.738 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.015788710000000001 2024-09-13T11:23:48.738 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.739 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.739 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:23:48.739 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.154745+0000", 2024-09-13T11:23:48.739 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.1294e-05 2024-09-13T11:23:48.739 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:48.739 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:23:48.739 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:48.739 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.739 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.739 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPushReply(3.13 75/70 [PushReplyOp(3:cfab03d6:::benchmark_data_smithi177_31116_object16599:head)])", 2024-09-13T11:23:48.739 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:31.163480+0000", 2024-09-13T11:23:48.739 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 17.552706779000001, 2024-09-13T11:23:48.739 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.63009029299999997, 2024-09-13T11:23:48.739 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:23:48.740 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:23:48.740 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-13T11:23:48.740 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:23:48.740 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.740 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:23:48.740 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.163480+0000", 2024-09-13T11:23:48.740 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:48.740 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.740 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.740 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:23:48.740 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.163480+0000", 2024-09-13T11:23:48.740 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:48.740 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.741 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.741 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:23:48.741 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.163481+0000", 2024-09-13T11:23:48.741 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0410000000000001e-06 2024-09-13T11:23:48.741 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.741 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.741 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:23:48.741 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.163485+0000", 2024-09-13T11:23:48.741 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.1629999999999998e-06 2024-09-13T11:23:48.741 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.741 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.741 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:23:48.741 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.792166+0000", 2024-09-13T11:23:48.741 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.62868067400000005 2024-09-13T11:23:48.742 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.742 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.742 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:23:48.742 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.792184+0000", 2024-09-13T11:23:48.742 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.7898999999999999e-05 2024-09-13T11:23:48.742 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.742 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.742 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:23:48.742 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.792312+0000", 2024-09-13T11:23:48.742 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00012871000000000001 2024-09-13T11:23:48.742 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.742 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.742 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:23:48.742 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.793570+0000", 2024-09-13T11:23:48.743 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001257806 2024-09-13T11:23:48.743 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:48.743 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:23:48.743 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:48.743 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.743 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.743 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPush(3.12 75/69 [PushOp(3:4a36ca6d:::benchmark_data_smithi177_31116_object31707:head, version: 59'1993, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:4a36ca6d:::benchmark_data_smithi177_31116_object31707:head@59'1993, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:23:48.743 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:31.309623+0000", 2024-09-13T11:23:48.743 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 17.406563823999999, 2024-09-13T11:23:48.743 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.165459152, 2024-09-13T11:23:48.743 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:23:48.743 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:23:48.744 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:23:48.744 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:23:48.744 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.744 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:23:48.744 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.309623+0000", 2024-09-13T11:23:48.744 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:48.744 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.744 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.744 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:23:48.744 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.309623+0000", 2024-09-13T11:23:48.744 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:48.744 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.744 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.745 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:23:48.745 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.309625+0000", 2024-09-13T11:23:48.745 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.3860000000000001e-06 2024-09-13T11:23:48.745 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.745 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.745 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:23:48.745 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.309680+0000", 2024-09-13T11:23:48.745 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.4388000000000002e-05 2024-09-13T11:23:48.745 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.745 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.745 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:23:48.745 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.473157+0000", 2024-09-13T11:23:48.745 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.163476813 2024-09-13T11:23:48.746 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.746 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.746 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:23:48.746 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.473189+0000", 2024-09-13T11:23:48.746 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.2975000000000002e-05 2024-09-13T11:23:48.746 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.746 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.746 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:23:48.746 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.473395+0000", 2024-09-13T11:23:48.746 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000206009 2024-09-13T11:23:48.746 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.746 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.746 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:23:48.747 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.473424+0000", 2024-09-13T11:23:48.747 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.8255e-05 2024-09-13T11:23:48.747 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.747 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.747 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:23:48.747 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.475082+0000", 2024-09-13T11:23:48.747 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0016583259999999999 2024-09-13T11:23:48.747 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:48.747 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:23:48.747 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:48.747 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.747 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.747 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPushReply(3.13 76/70 [PushReplyOp(3:c9d43a82:::benchmark_data_smithi177_31116_object20026:head)])", 2024-09-13T11:23:48.748 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:32.427709+0000", 2024-09-13T11:23:48.748 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 16.288477264000001, 2024-09-13T11:23:48.748 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.46547316300000002, 2024-09-13T11:23:48.748 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:23:48.748 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:23:48.748 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-13T11:23:48.748 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:23:48.748 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.748 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:23:48.748 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:32.427709+0000", 2024-09-13T11:23:48.748 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:48.748 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.748 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.748 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:23:48.749 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:32.427709+0000", 2024-09-13T11:23:48.749 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:48.749 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.749 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.749 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:23:48.749 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:32.427711+0000", 2024-09-13T11:23:48.749 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5099999999999999e-06 2024-09-13T11:23:48.749 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.749 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.749 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:23:48.749 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:32.427717+0000", 2024-09-13T11:23:48.749 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.1569999999999997e-06 2024-09-13T11:23:48.749 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.749 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.750 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:23:48.750 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:32.892390+0000", 2024-09-13T11:23:48.750 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.46467251100000001 2024-09-13T11:23:48.750 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.750 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.750 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:23:48.750 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:32.892405+0000", 2024-09-13T11:23:48.750 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5296999999999999e-05 2024-09-13T11:23:48.750 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.750 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.750 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:23:48.750 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:32.892503+0000", 2024-09-13T11:23:48.750 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.7820999999999997e-05 2024-09-13T11:23:48.750 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.751 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.751 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:23:48.751 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:32.893183+0000", 2024-09-13T11:23:48.751 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00067986700000000001 2024-09-13T11:23:48.751 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:48.751 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:23:48.751 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:48.751 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.751 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.751 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:56066 3.a e79/75 3:561a1dd2:::benchmark_data_smithi177_31116_object56065:head v 79'3452, mlcod=79'3449)", 2024-09-13T11:23:48.751 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:36.194429+0000", 2024-09-13T11:23:48.751 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 12.521757379, 2024-09-13T11:23:48.752 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.13436872499999999, 2024-09-13T11:23:48.752 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:23:48.752 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:23:48.752 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:48.752 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:23:48.752 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.752 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:23:48.752 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:36.194429+0000", 2024-09-13T11:23:48.752 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:48.752 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.752 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.752 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:23:48.753 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:36.194429+0000", 2024-09-13T11:23:48.753 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:48.753 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.753 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.753 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:23:48.753 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:36.194431+0000", 2024-09-13T11:23:48.753 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.4279999999999999e-06 2024-09-13T11:23:48.753 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.753 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.753 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:23:48.753 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:36.194467+0000", 2024-09-13T11:23:48.753 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.6402000000000001e-05 2024-09-13T11:23:48.753 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.753 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.754 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:23:48.754 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:36.327354+0000", 2024-09-13T11:23:48.754 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.13288720800000001 2024-09-13T11:23:48.754 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.754 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.754 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:23:48.754 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:36.327361+0000", 2024-09-13T11:23:48.754 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.302e-06 2024-09-13T11:23:48.754 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.754 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.754 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:23:48.754 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:36.327441+0000", 2024-09-13T11:23:48.754 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.0406000000000002e-05 2024-09-13T11:23:48.755 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.755 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.755 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:23:48.755 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:36.327466+0000", 2024-09-13T11:23:48.755 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.5367e-05 2024-09-13T11:23:48.755 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.755 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.755 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:23:48.755 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:36.328754+0000", 2024-09-13T11:23:48.755 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001287627 2024-09-13T11:23:48.755 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.755 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.755 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:23:48.756 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:36.328798+0000", 2024-09-13T11:23:48.756 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.3985e-05 2024-09-13T11:23:48.756 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:48.756 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:23:48.756 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:48.756 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.756 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.756 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:56096 3.d 3:b305b87e:::benchmark_data_smithi177_31116_object56095:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e79)", 2024-09-13T11:23:48.756 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:36.226921+0000", 2024-09-13T11:23:48.756 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 12.489266024000001, 2024-09-13T11:23:48.756 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.11682587899999999, 2024-09-13T11:23:48.756 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:23:48.757 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:23:48.757 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:48.757 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:23:48.757 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:23:48.757 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:48.757 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 56096 2024-09-13T11:23:48.757 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.757 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:23:48.757 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.757 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:23:48.757 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:36.226921+0000", 2024-09-13T11:23:48.757 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:48.757 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.758 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.758 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:23:48.758 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:36.226921+0000", 2024-09-13T11:23:48.758 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:48.758 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.758 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.758 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:23:48.758 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:36.226923+0000", 2024-09-13T11:23:48.758 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.843e-06 2024-09-13T11:23:48.758 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.758 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.758 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:23:48.758 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:36.226944+0000", 2024-09-13T11:23:48.759 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.1571999999999999e-05 2024-09-13T11:23:48.759 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.759 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.759 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:23:48.759 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:36.226954+0000", 2024-09-13T11:23:48.759 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.8970000000000003e-06 2024-09-13T11:23:48.759 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.759 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.759 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:23:48.759 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:36.226959+0000", 2024-09-13T11:23:48.759 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.4210000000000003e-06 2024-09-13T11:23:48.759 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.759 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.759 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:23:48.760 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:36.227012+0000", 2024-09-13T11:23:48.760 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.2578999999999999e-05 2024-09-13T11:23:48.760 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.760 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.760 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:23:48.760 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:36.227112+0000", 2024-09-13T11:23:48.760 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000100231 2024-09-13T11:23:48.760 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.760 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.760 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 3", 2024-09-13T11:23:48.760 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:36.227352+0000", 2024-09-13T11:23:48.760 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00023945999999999999 2024-09-13T11:23:48.760 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.760 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.761 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:23:48.761 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:36.228973+0000", 2024-09-13T11:23:48.761 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001621003 2024-09-13T11:23:48.761 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.761 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.761 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:48.761 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:36.343652+0000", 2024-09-13T11:23:48.761 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.114678868 2024-09-13T11:23:48.761 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.761 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.761 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:23:48.761 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:36.343706+0000", 2024-09-13T11:23:48.762 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.4505000000000001e-05 2024-09-13T11:23:48.762 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.762 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.762 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:23:48.762 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:36.343747+0000", 2024-09-13T11:23:48.762 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.0500000000000002e-05 2024-09-13T11:23:48.762 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:48.762 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:23:48.762 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:48.762 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.762 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.762 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPushReply(3.23 80/78 [PushReplyOp(3:c54a2e13:::benchmark_data_smithi177_31116_object16923:head)])", 2024-09-13T11:23:48.762 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:37.275242+0000", 2024-09-13T11:23:48.763 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 11.440945052, 2024-09-13T11:23:48.763 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.098976975999999994, 2024-09-13T11:23:48.763 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:23:48.763 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:23:48.763 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-13T11:23:48.763 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:23:48.763 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.763 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:23:48.763 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:37.275242+0000", 2024-09-13T11:23:48.763 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:48.763 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.763 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.764 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:23:48.764 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:37.275242+0000", 2024-09-13T11:23:48.764 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:48.764 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.764 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.764 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:23:48.764 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:37.275244+0000", 2024-09-13T11:23:48.764 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.813e-06 2024-09-13T11:23:48.764 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.764 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.764 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:23:48.764 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:37.275251+0000", 2024-09-13T11:23:48.765 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.678e-06 2024-09-13T11:23:48.765 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.765 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.765 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:23:48.765 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:37.373653+0000", 2024-09-13T11:23:48.765 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.098401605000000003 2024-09-13T11:23:48.765 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.765 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.765 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:23:48.765 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:37.373658+0000", 2024-09-13T11:23:48.765 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.6799999999999998e-06 2024-09-13T11:23:48.765 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.765 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.765 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:23:48.766 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:37.373702+0000", 2024-09-13T11:23:48.766 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.3996999999999999e-05 2024-09-13T11:23:48.766 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.766 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.766 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:23:48.766 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:37.374219+0000", 2024-09-13T11:23:48.766 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00051620300000000002 2024-09-13T11:23:48.766 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:48.766 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:23:48.766 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:48.766 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.766 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.766 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:57435 3.3 e80/75 3:c0db4777:::benchmark_data_smithi177_31116_object57434:head v 80'3155, mlcod=80'3153)", 2024-09-13T11:23:48.766 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:37.437070+0000", 2024-09-13T11:23:48.767 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 11.279116447, 2024-09-13T11:23:48.767 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.35861845999999997, 2024-09-13T11:23:48.767 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:23:48.767 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:23:48.767 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:48.767 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:23:48.767 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.767 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:23:48.767 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:37.437070+0000", 2024-09-13T11:23:48.767 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:48.767 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.767 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.767 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:23:48.767 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:37.437070+0000", 2024-09-13T11:23:48.768 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:48.768 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.768 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.768 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:23:48.768 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:37.437072+0000", 2024-09-13T11:23:48.768 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5549999999999999e-06 2024-09-13T11:23:48.768 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.768 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.768 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:23:48.768 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:37.437141+0000", 2024-09-13T11:23:48.768 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.9494999999999998e-05 2024-09-13T11:23:48.768 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.769 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.769 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:23:48.769 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:37.793400+0000", 2024-09-13T11:23:48.769 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.35625817999999998 2024-09-13T11:23:48.769 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.769 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.769 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:23:48.769 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:37.793406+0000", 2024-09-13T11:23:48.769 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.6830000000000002e-06 2024-09-13T11:23:48.769 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.769 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.769 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:23:48.769 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:37.793472+0000", 2024-09-13T11:23:48.769 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.5671000000000002e-05 2024-09-13T11:23:48.770 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.770 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.770 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:23:48.770 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:37.793492+0000", 2024-09-13T11:23:48.770 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.0051000000000002e-05 2024-09-13T11:23:48.770 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.770 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.770 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:23:48.770 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:37.795626+0000", 2024-09-13T11:23:48.770 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0021344990000000002 2024-09-13T11:23:48.770 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.770 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.770 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:23:48.770 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:37.795689+0000", 2024-09-13T11:23:48.771 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.2325999999999999e-05 2024-09-13T11:23:48.771 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:48.771 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:23:48.771 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:48.771 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.771 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.771 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:57882 3.3 e80/75 3:c17a2efd:::benchmark_data_smithi177_31116_object57881:head v 80'3160, mlcod=80'3157)", 2024-09-13T11:23:48.771 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:37.903922+0000", 2024-09-13T11:23:48.771 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 10.812264527, 2024-09-13T11:23:48.771 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.098400796999999998, 2024-09-13T11:23:48.771 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:23:48.771 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:23:48.771 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:48.771 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:23:48.772 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.772 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:23:48.772 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:37.903922+0000", 2024-09-13T11:23:48.772 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:48.772 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.772 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.772 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:23:48.772 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:37.903922+0000", 2024-09-13T11:23:48.772 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:48.772 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.772 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.772 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:23:48.772 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:37.903924+0000", 2024-09-13T11:23:48.773 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5519999999999999e-06 2024-09-13T11:23:48.773 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.773 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.773 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:23:48.773 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:37.903991+0000", 2024-09-13T11:23:48.773 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.737e-05 2024-09-13T11:23:48.773 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.773 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.773 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:23:48.773 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:37.999423+0000", 2024-09-13T11:23:48.773 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.095432197999999996 2024-09-13T11:23:48.773 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.773 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.774 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:23:48.774 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:37.999430+0000", 2024-09-13T11:23:48.774 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.7750000000000004e-06 2024-09-13T11:23:48.774 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.774 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.774 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:23:48.774 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:37.999616+0000", 2024-09-13T11:23:48.774 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000185978 2024-09-13T11:23:48.774 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.774 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.774 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:23:48.774 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:37.999635+0000", 2024-09-13T11:23:48.774 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.9029999999999999e-05 2024-09-13T11:23:48.774 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.775 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.775 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:23:48.775 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:38.002276+0000", 2024-09-13T11:23:48.775 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0026407319999999998 2024-09-13T11:23:48.775 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.775 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.775 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:23:48.775 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:38.002323+0000", 2024-09-13T11:23:48.775 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.7162e-05 2024-09-13T11:23:48.775 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:48.775 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:23:48.775 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:48.775 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.775 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.776 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:57985 3.3 e80/75 3:c330bb03:::benchmark_data_smithi177_31116_object57984:head v 80'3162, mlcod=80'3159)", 2024-09-13T11:23:48.776 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:38.007076+0000", 2024-09-13T11:23:48.776 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 10.709111095000001, 2024-09-13T11:23:48.776 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.18211438199999999, 2024-09-13T11:23:48.776 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:23:48.776 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:23:48.776 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:48.776 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:23:48.776 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.776 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:23:48.776 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:38.007076+0000", 2024-09-13T11:23:48.776 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:48.776 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.777 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.777 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:23:48.777 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:38.007076+0000", 2024-09-13T11:23:48.777 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:48.777 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.777 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.777 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:23:48.777 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:38.007077+0000", 2024-09-13T11:23:48.777 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.6759999999999999e-06 2024-09-13T11:23:48.777 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.777 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.777 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:23:48.777 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:38.007221+0000", 2024-09-13T11:23:48.778 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00014344299999999999 2024-09-13T11:23:48.778 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.778 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.778 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:23:48.778 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:38.186808+0000", 2024-09-13T11:23:48.778 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.17958726699999999 2024-09-13T11:23:48.778 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.778 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.778 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:23:48.778 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:38.186815+0000", 2024-09-13T11:23:48.778 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.0890000000000002e-06 2024-09-13T11:23:48.779 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.779 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.779 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:23:48.779 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:38.186887+0000", 2024-09-13T11:23:48.779 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.2006000000000001e-05 2024-09-13T11:23:48.779 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.779 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.779 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:23:48.779 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:38.186913+0000", 2024-09-13T11:23:48.779 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.5962e-05 2024-09-13T11:23:48.779 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.779 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.779 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:23:48.779 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:38.189123+0000", 2024-09-13T11:23:48.780 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0022099649999999999 2024-09-13T11:23:48.780 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.780 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.780 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:23:48.780 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:38.189190+0000", 2024-09-13T11:23:48.780 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.6973999999999999e-05 2024-09-13T11:23:48.780 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:48.780 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:23:48.780 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:48.780 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.780 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.780 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPush(3.1c 84/77 [PushOp(3:3df7cc9e:::benchmark_data_smithi177_31116_object19949:head, version: 54'1255, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:3df7cc9e:::benchmark_data_smithi177_31116_object19949:head@54'1255, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:23:48.780 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:41.278800+0000", 2024-09-13T11:23:48.781 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 7.4373866250000003, 2024-09-13T11:23:48.781 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.10680685700000001, 2024-09-13T11:23:48.781 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:23:48.781 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:23:48.781 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:23:48.781 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:23:48.781 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.781 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:23:48.781 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.278800+0000", 2024-09-13T11:23:48.781 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:48.781 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.781 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.781 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:23:48.781 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.278800+0000", 2024-09-13T11:23:48.782 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:48.782 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.782 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.782 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:23:48.782 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.278802+0000", 2024-09-13T11:23:48.782 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.779e-06 2024-09-13T11:23:48.782 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.782 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.782 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:23:48.782 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.278832+0000", 2024-09-13T11:23:48.782 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.9989999999999999e-05 2024-09-13T11:23:48.782 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.782 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.782 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:23:48.783 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.384356+0000", 2024-09-13T11:23:48.783 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.105524303 2024-09-13T11:23:48.783 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.783 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.783 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:23:48.783 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.384376+0000", 2024-09-13T11:23:48.783 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.0049000000000001e-05 2024-09-13T11:23:48.783 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.783 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.783 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:23:48.783 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.384483+0000", 2024-09-13T11:23:48.783 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000106271 2024-09-13T11:23:48.783 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.783 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.783 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:23:48.784 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.384498+0000", 2024-09-13T11:23:48.784 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5109000000000001e-05 2024-09-13T11:23:48.784 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.784 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.784 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:23:48.784 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.385607+0000", 2024-09-13T11:23:48.784 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0011093559999999999 2024-09-13T11:23:48.784 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:48.784 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:23:48.784 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:48.784 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.784 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.784 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPushReply(3.23 84/78 [PushReplyOp(3:c7e44616:::benchmark_data_smithi177_31116_object32371:head)])", 2024-09-13T11:23:48.784 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:41.338210+0000", 2024-09-13T11:23:48.785 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 7.3779763760000003, 2024-09-13T11:23:48.785 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.41118492299999998, 2024-09-13T11:23:48.785 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:23:48.785 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:23:48.785 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-13T11:23:48.785 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:23:48.785 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.785 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:23:48.785 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.338210+0000", 2024-09-13T11:23:48.785 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:48.785 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.785 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.785 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:23:48.786 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.338210+0000", 2024-09-13T11:23:48.786 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:48.786 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.786 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.786 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:23:48.786 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.338212+0000", 2024-09-13T11:23:48.786 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.314e-06 2024-09-13T11:23:48.786 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.786 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.786 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:23:48.786 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.338219+0000", 2024-09-13T11:23:48.786 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.6469999999999998e-06 2024-09-13T11:23:48.786 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.786 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.787 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:23:48.787 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.748675+0000", 2024-09-13T11:23:48.787 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.41045567500000002 2024-09-13T11:23:48.787 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.787 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.787 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:23:48.787 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.748684+0000", 2024-09-13T11:23:48.787 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.2150000000000007e-06 2024-09-13T11:23:48.787 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.787 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.787 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:23:48.787 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.748739+0000", 2024-09-13T11:23:48.787 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.4543000000000003e-05 2024-09-13T11:23:48.787 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.788 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.788 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:23:48.788 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.749395+0000", 2024-09-13T11:23:48.788 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00065652899999999999 2024-09-13T11:23:48.788 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:48.788 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:23:48.788 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:48.788 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.788 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.788 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPushReply(3.23 84/78 [PushReplyOp(3:c57c3ccc:::benchmark_data_smithi177_31116_object33278:head)])", 2024-09-13T11:23:48.788 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:41.825672+0000", 2024-09-13T11:23:48.788 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 6.8905150700000002, 2024-09-13T11:23:48.789 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.11608001499999999, 2024-09-13T11:23:48.789 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:23:48.789 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:23:48.789 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-13T11:23:48.789 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:23:48.789 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.789 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:23:48.789 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.825672+0000", 2024-09-13T11:23:48.789 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:48.789 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.789 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.789 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:23:48.790 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.825672+0000", 2024-09-13T11:23:48.790 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:48.790 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.790 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.790 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:23:48.790 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.825673+0000", 2024-09-13T11:23:48.790 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0300000000000001e-06 2024-09-13T11:23:48.790 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.790 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.790 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:23:48.790 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.825680+0000", 2024-09-13T11:23:48.790 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.1520000000000003e-06 2024-09-13T11:23:48.791 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.791 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.791 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:23:48.791 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.940925+0000", 2024-09-13T11:23:48.791 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.115244816 2024-09-13T11:23:48.791 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.791 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.791 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:23:48.791 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.940932+0000", 2024-09-13T11:23:48.791 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.7319999999999998e-06 2024-09-13T11:23:48.791 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.791 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.792 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:23:48.792 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.941218+0000", 2024-09-13T11:23:48.792 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00028532699999999999 2024-09-13T11:23:48.792 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.792 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.792 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:23:48.792 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.941752+0000", 2024-09-13T11:23:48.792 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00053395799999999996 2024-09-13T11:23:48.792 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:48.792 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:23:48.792 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:48.792 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.792 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.793 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPushReply(3.23 85/78 [PushReplyOp(3:c5c6dce1:::benchmark_data_smithi177_31116_object37293:head)])", 2024-09-13T11:23:48.793 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:42.377862+0000", 2024-09-13T11:23:48.793 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 6.3383249169999996, 2024-09-13T11:23:48.793 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.62538808599999995, 2024-09-13T11:23:48.793 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:23:48.793 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:23:48.793 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-13T11:23:48.793 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:23:48.793 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.793 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:23:48.793 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.377862+0000", 2024-09-13T11:23:48.793 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:48.793 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.794 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.794 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:23:48.794 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.377862+0000", 2024-09-13T11:23:48.794 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:48.794 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.794 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.794 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:23:48.794 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.377863+0000", 2024-09-13T11:23:48.794 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.6899999999999996e-07 2024-09-13T11:23:48.794 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.794 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.794 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:23:48.794 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.377866+0000", 2024-09-13T11:23:48.794 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.219e-06 2024-09-13T11:23:48.795 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.795 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.795 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:23:48.795 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:43.001857+0000", 2024-09-13T11:23:48.795 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.62399084299999996 2024-09-13T11:23:48.795 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.795 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.795 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:23:48.795 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:43.001869+0000", 2024-09-13T11:23:48.795 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.2415999999999999e-05 2024-09-13T11:23:48.795 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.795 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.795 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:23:48.796 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:43.002194+0000", 2024-09-13T11:23:48.796 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000324774 2024-09-13T11:23:48.796 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.796 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.796 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:23:48.796 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:43.003250+0000", 2024-09-13T11:23:48.796 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001055865 2024-09-13T11:23:48.796 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:48.796 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:23:48.796 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:48.796 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.796 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.796 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPushReply(3.23 86/78 [PushReplyOp(3:c4ca4518:::benchmark_data_smithi177_31116_object39787:head)])", 2024-09-13T11:23:48.797 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:43.580816+0000", 2024-09-13T11:23:48.797 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 5.1353712260000002, 2024-09-13T11:23:48.797 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.14999331199999999, 2024-09-13T11:23:48.797 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:23:48.797 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:23:48.797 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-13T11:23:48.797 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:23:48.797 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.797 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:23:48.797 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:43.580816+0000", 2024-09-13T11:23:48.797 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:48.797 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.797 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.798 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:23:48.798 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:43.580816+0000", 2024-09-13T11:23:48.798 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:48.798 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.798 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.798 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:23:48.798 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:43.580818+0000", 2024-09-13T11:23:48.798 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.7250000000000002e-06 2024-09-13T11:23:48.798 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.798 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.798 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:23:48.798 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:43.580830+0000", 2024-09-13T11:23:48.798 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.2247e-05 2024-09-13T11:23:48.798 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.799 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.799 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:23:48.799 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:43.730204+0000", 2024-09-13T11:23:48.799 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.149373531 2024-09-13T11:23:48.799 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.799 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.799 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:23:48.799 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:43.730213+0000", 2024-09-13T11:23:48.799 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.6989999999999992e-06 2024-09-13T11:23:48.799 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.799 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.799 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:23:48.800 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:43.730274+0000", 2024-09-13T11:23:48.800 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.1419000000000003e-05 2024-09-13T11:23:48.800 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.800 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.800 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:23:48.800 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:43.730809+0000", 2024-09-13T11:23:48.800 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00053469100000000001 2024-09-13T11:23:48.800 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:48.800 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:23:48.800 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:48.800 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.800 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.800 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPushReply(3.23 86/78 [PushReplyOp(3:c6ae38b7:::benchmark_data_smithi177_31116_object39934:head)])", 2024-09-13T11:23:48.800 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:43.748700+0000", 2024-09-13T11:23:48.801 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 4.9674870530000002, 2024-09-13T11:23:48.801 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.11236333, 2024-09-13T11:23:48.801 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:23:48.801 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:23:48.801 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-13T11:23:48.801 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:23:48.801 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.801 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:23:48.801 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:43.748700+0000", 2024-09-13T11:23:48.801 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:48.801 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.801 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.801 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:23:48.802 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:43.748700+0000", 2024-09-13T11:23:48.802 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:48.802 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.802 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.802 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:23:48.802 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:43.748701+0000", 2024-09-13T11:23:48.802 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.398e-06 2024-09-13T11:23:48.802 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.802 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.802 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:23:48.802 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:43.748707+0000", 2024-09-13T11:23:48.802 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.993e-06 2024-09-13T11:23:48.802 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.803 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.803 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:23:48.803 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:43.860552+0000", 2024-09-13T11:23:48.803 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.11184472099999999 2024-09-13T11:23:48.803 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.803 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.803 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:23:48.803 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:43.860560+0000", 2024-09-13T11:23:48.803 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.5949999999999999e-06 2024-09-13T11:23:48.803 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.803 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.803 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:23:48.803 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:43.860619+0000", 2024-09-13T11:23:48.804 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.8455999999999997e-05 2024-09-13T11:23:48.804 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.804 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.804 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:23:48.804 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:43.861063+0000", 2024-09-13T11:23:48.804 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00044416699999999998 2024-09-13T11:23:48.804 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:48.804 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:23:48.804 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:48.804 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.804 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.804 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPush(3.1c 89/77 [PushOp(3:38bcbc71:::benchmark_data_smithi177_31116_object31448:head, version: 59'1946, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:38bcbc71:::benchmark_data_smithi177_31116_object31448:head@59'1946, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:23:48.804 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:46.670723+0000", 2024-09-13T11:23:48.805 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 2.0454636929999999, 2024-09-13T11:23:48.805 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.27576065, 2024-09-13T11:23:48.805 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:23:48.805 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:23:48.805 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:23:48.805 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:23:48.805 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.805 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:23:48.805 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.670723+0000", 2024-09-13T11:23:48.805 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:48.805 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.805 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.806 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:23:48.806 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.670723+0000", 2024-09-13T11:23:48.806 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:48.806 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.806 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.806 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:23:48.806 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.670724+0000", 2024-09-13T11:23:48.806 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.4389999999999999e-06 2024-09-13T11:23:48.807 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.807 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.807 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:23:48.807 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.670754+0000", 2024-09-13T11:23:48.807 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.9224999999999999e-05 2024-09-13T11:23:48.807 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.807 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.807 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:23:48.807 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.944369+0000", 2024-09-13T11:23:48.807 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.27361529800000001 2024-09-13T11:23:48.807 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.807 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.808 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:23:48.808 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.944422+0000", 2024-09-13T11:23:48.808 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.3318e-05 2024-09-13T11:23:48.808 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.808 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.808 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:23:48.808 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.944711+0000", 2024-09-13T11:23:48.808 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00028912399999999998 2024-09-13T11:23:48.808 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.808 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.808 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:23:48.808 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.944735+0000", 2024-09-13T11:23:48.809 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.3366999999999998e-05 2024-09-13T11:23:48.809 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:48.809 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:48.809 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:23:48.809 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.946484+0000", 2024-09-13T11:23:48.809 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0017488790000000001 2024-09-13T11:23:48.809 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:48.809 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:23:48.809 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:48.809 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:48.809 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:23:48.809 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-09-13T11:23:48.811 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-09-13T11:23:48.812 INFO:tasks.ceph.osd.8.smithi001.stderr:2024-09-13T11:23:48.804+0000 7fb9b48ae640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26312) UID: 0 2024-09-13T11:23:48.912 INFO:tasks.ceph.osd.14.smithi162.stderr:2024-09-13T11:23:48.909+0000 7f235c0db640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26319) UID: 0 2024-09-13T11:23:48.914 INFO:teuthology.orchestra.run.smithi001.stderr:reweighted osd.8 to 1 (10000) 2024-09-13T11:23:48.921 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-13T11:23:48.916+0000 7f6a5d1a2640 -1 osd.0 91 handle_osd_map injecting map crc failure 2024-09-13T11:23:48.923 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T11:23:48.926 INFO:tasks.ceph.osd.7.smithi177.stderr:2024-09-13T11:23:48.920+0000 7f896ff4a640 -1 osd.7 91 handle_osd_map injecting map crc failure 2024-09-13T11:23:48.946 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-13T11:23:49.026 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-09-13T11:23:49.026 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [ 2024-09-13T11:23:49.026 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:49.026 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:65424 3.1f 3:faa3f144:::benchmark_data_smithi177_31116_object65423:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92)", 2024-09-13T11:23:49.026 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:48.945898+0000", 2024-09-13T11:23:49.027 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 0.0086988229999999996, 2024-09-13T11:23:49.027 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0087220960000000004, 2024-09-13T11:23:49.027 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:23:49.027 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:23:49.027 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "waiting for sub ops", 2024-09-13T11:23:49.028 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:23:49.028 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:23:49.028 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:49.028 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 65424 2024-09-13T11:23:49.028 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:49.028 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:23:49.029 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:49.029 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:23:49.029 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.945898+0000", 2024-09-13T11:23:49.029 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:49.029 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:49.030 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:49.030 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:23:49.030 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.945898+0000", 2024-09-13T11:23:49.030 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:49.030 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:49.030 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:49.031 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:23:49.031 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.945900+0000", 2024-09-13T11:23:49.031 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.886e-06 2024-09-13T11:23:49.031 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:49.031 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:49.032 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:23:49.032 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.945922+0000", 2024-09-13T11:23:49.032 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.1403000000000002e-05 2024-09-13T11:23:49.032 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:49.032 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:49.032 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:23:49.033 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.945935+0000", 2024-09-13T11:23:49.033 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3786e-05 2024-09-13T11:23:49.033 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:49.033 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:49.033 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:23:49.034 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.945943+0000", 2024-09-13T11:23:49.034 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.8609999999999998e-06 2024-09-13T11:23:49.034 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:49.034 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:49.034 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:23:49.034 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.946879+0000", 2024-09-13T11:23:49.035 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00093595500000000003 2024-09-13T11:23:49.035 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:49.035 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:49.035 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:23:49.035 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.947017+0000", 2024-09-13T11:23:49.035 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00013761700000000001 2024-09-13T11:23:49.036 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:49.036 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:49.036 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 10,14", 2024-09-13T11:23:49.036 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.947130+0000", 2024-09-13T11:23:49.036 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00011307999999999999 2024-09-13T11:23:49.037 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:49.037 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:49.037 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:23:49.037 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.951969+0000", 2024-09-13T11:23:49.037 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0048394500000000003 2024-09-13T11:23:49.037 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:49.038 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:23:49.038 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:49.038 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:49.038 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:49.038 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPushReply(3.1f 92/78 [PushReplyOp(3:fa805cea:::benchmark_data_smithi177_31116_object18286:head)])", 2024-09-13T11:23:49.039 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:48.954446+0000", 2024-09-13T11:23:49.039 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 0.00015147399999999999, 2024-09-13T11:23:49.039 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000253642, 2024-09-13T11:23:49.039 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:23:49.039 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:23:49.039 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "queued for pg", 2024-09-13T11:23:49.040 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:23:49.040 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:49.040 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:23:49.040 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.954446+0000", 2024-09-13T11:23:49.040 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:49.040 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:49.041 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:49.041 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:23:49.041 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.954446+0000", 2024-09-13T11:23:49.041 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:49.041 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:49.042 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:49.042 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:23:49.042 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.954447+0000", 2024-09-13T11:23:49.042 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.1629999999999999e-06 2024-09-13T11:23:49.042 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:49.042 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:49.042 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:23:49.042 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.954456+0000", 2024-09-13T11:23:49.042 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.7800000000000006e-06 2024-09-13T11:23:49.042 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:49.043 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:49.043 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:23:49.043 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.954488+0000", 2024-09-13T11:23:49.043 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.2036000000000003e-05 2024-09-13T11:23:49.043 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:49.043 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:49.043 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:23:49.043 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.954495+0000", 2024-09-13T11:23:49.043 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.712e-06 2024-09-13T11:23:49.043 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:49.043 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:23:49.043 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:49.043 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:49.043 INFO:teuthology.orchestra.run.smithi177.stdout: ], 2024-09-13T11:23:49.044 INFO:teuthology.orchestra.run.smithi177.stdout: "num_ops": 2 2024-09-13T11:23:49.044 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-09-13T11:23:49.044 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 9 1 2024-09-13T11:23:49.045 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-09-13T11:23:49.045 INFO:tasks.radosbench.radosbench.0.smithi177.stdout: 82 16 65511 65495 49.9147 32.3125 0.0096121 0.0200291 2024-09-13T11:23:49.046 INFO:tasks.ceph.osd.15.smithi177.stderr:2024-09-13T11:23:49.016+0000 7f9d6ce77640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26578) UID: 0 2024-09-13T11:23:49.120 INFO:teuthology.orchestra.run.smithi001.stdout:osd.0: {} 2024-09-13T11:23:49.120 INFO:teuthology.orchestra.run.smithi001.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T11:23:49.124 INFO:tasks.ceph.osd.8.smithi001.stderr:2024-09-13T11:23:49.120+0000 7fb9b48ae640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26312) UID: 0 2024-09-13T11:23:49.151 INFO:teuthology.orchestra.run.smithi001.stdout:osd.1: {} 2024-09-13T11:23:49.152 INFO:teuthology.orchestra.run.smithi001.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T11:23:49.195 INFO:teuthology.orchestra.run.smithi001.stdout:osd.2: {} 2024-09-13T11:23:49.195 INFO:teuthology.orchestra.run.smithi001.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T11:23:49.225 INFO:tasks.ceph.osd.7.smithi177.stderr:2024-09-13T11:23:49.220+0000 7f897b51d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26575) UID: 0 2024-09-13T11:23:49.234 INFO:teuthology.orchestra.run.smithi001.stdout:osd.3: {} 2024-09-13T11:23:49.235 INFO:teuthology.orchestra.run.smithi001.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T11:23:49.242 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-09-13T11:23:49.243 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [], 2024-09-13T11:23:49.243 INFO:teuthology.orchestra.run.smithi177.stdout: "complaint_time": 30, 2024-09-13T11:23:49.243 INFO:teuthology.orchestra.run.smithi177.stdout: "num_blocked_ops": 0 2024-09-13T11:23:49.243 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-09-13T11:23:49.266 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-09-13T11:23:49.271 INFO:teuthology.orchestra.run.smithi001.stdout:osd.4: {} 2024-09-13T11:23:49.271 INFO:teuthology.orchestra.run.smithi001.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T11:23:49.318 INFO:teuthology.orchestra.run.smithi001.stdout:osd.5: {} 2024-09-13T11:23:49.318 INFO:teuthology.orchestra.run.smithi001.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T11:23:49.352 INFO:tasks.ceph.osd.3.smithi177.stderr:2024-09-13T11:23:49.348+0000 7f10adba4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26574) UID: 0 2024-09-13T11:23:49.357 INFO:teuthology.orchestra.run.smithi001.stdout:osd.6: {} 2024-09-13T11:23:49.357 INFO:teuthology.orchestra.run.smithi001.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T11:23:49.395 INFO:teuthology.orchestra.run.smithi001.stdout:osd.7: {} 2024-09-13T11:23:49.396 INFO:teuthology.orchestra.run.smithi001.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T11:23:49.426 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-09-13T11:23:49.425+0000 7f97befc9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26315) UID: 0 2024-09-13T11:23:49.430 INFO:teuthology.orchestra.run.smithi001.stdout:osd.8: {} 2024-09-13T11:23:49.430 INFO:teuthology.orchestra.run.smithi001.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-13T11:23:49.459 INFO:teuthology.orchestra.run.smithi001.stdout:osd.9: {} 2024-09-13T11:23:49.459 INFO:teuthology.orchestra.run.smithi001.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-13T11:23:49.479 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-09-13T11:23:49.479 INFO:teuthology.orchestra.run.smithi177.stdout: "size": 20, 2024-09-13T11:23:49.479 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 600, 2024-09-13T11:23:49.479 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [ 2024-09-13T11:23:49.479 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:49.479 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:79 3.1 e32/30 3:841df5b5:::benchmark_data_smithi177_31116_object78:head v 32'6, mlcod=32'4)", 2024-09-13T11:23:49.480 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:22:27.057292+0000", 2024-09-13T11:23:49.480 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 82.419902961999995, 2024-09-13T11:23:49.480 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.89296786800000005, 2024-09-13T11:23:49.480 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:23:49.480 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:23:49.480 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:49.480 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:23:49.480 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:49.480 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:23:49.480 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.057292+0000", 2024-09-13T11:23:49.480 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:49.480 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:49.480 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:49.481 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:23:49.481 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.057292+0000", 2024-09-13T11:23:49.481 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:49.481 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:49.481 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:49.481 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:23:49.481 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.057293+0000", 2024-09-13T11:23:49.481 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.6109999999999999e-06 2024-09-13T11:23:49.481 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:49.481 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:49.481 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:23:49.481 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.057330+0000", 2024-09-13T11:23:49.481 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.6721999999999998e-05 2024-09-13T11:23:49.481 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:49.482 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:49.482 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:23:49.482 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.948197+0000", 2024-09-13T11:23:49.482 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.890867042 2024-09-13T11:23:49.482 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:49.482 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:49.482 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:23:49.482 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.948213+0000", 2024-09-13T11:23:49.482 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5826000000000001e-05 2024-09-13T11:23:49.482 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:49.482 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:49.482 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:23:49.482 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.948377+0000", 2024-09-13T11:23:49.482 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00016348600000000001 2024-09-13T11:23:49.483 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:49.483 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:49.483 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:23:49.483 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.948447+0000", 2024-09-13T11:23:49.483 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.0891999999999998e-05 2024-09-13T11:23:49.483 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:49.483 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:49.483 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:23:49.483 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.950150+0000", 2024-09-13T11:23:49.483 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001702329 2024-09-13T11:23:49.483 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:49.483 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:49.483 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:23:49.483 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.950260+0000", 2024-09-13T11:23:49.484 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00010996 2024-09-13T11:23:49.484 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:49.484 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:23:49.484 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:49.484 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:49.484 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:49.484 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:30815 3.f 3:fde17cb1:::benchmark_data_smithi177_31116_object30814:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e59)", 2024-09-13T11:23:49.484 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:09.495035+0000", 2024-09-13T11:23:49.484 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 39.982159797000001, 2024-09-13T11:23:49.484 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.229307711, 2024-09-13T11:23:49.484 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:23:49.484 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:23:49.484 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:49.485 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:23:49.485 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:23:49.485 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:49.485 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 30815 2024-09-13T11:23:49.485 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:49.485 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:23:49.485 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:49.485 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:23:49.485 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.495035+0000", 2024-09-13T11:23:49.485 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:49.485 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:49.485 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:49.485 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:23:49.486 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.495035+0000", 2024-09-13T11:23:49.486 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:49.486 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:49.486 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:49.486 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:23:49.486 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.495037+0000", 2024-09-13T11:23:49.486 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.871e-06 2024-09-13T11:23:49.486 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:49.486 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:49.486 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:23:49.486 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.495062+0000", 2024-09-13T11:23:49.486 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.5164000000000002e-05 2024-09-13T11:23:49.486 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:49.486 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:49.487 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:23:49.487 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.495075+0000", 2024-09-13T11:23:49.487 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3369e-05 2024-09-13T11:23:49.487 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:49.487 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:49.487 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:23:49.487 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.495084+0000", 2024-09-13T11:23:49.487 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.3510000000000002e-06 2024-09-13T11:23:49.487 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:49.487 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:49.487 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:23:49.487 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.495346+0000", 2024-09-13T11:23:49.487 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00026264000000000002 2024-09-13T11:23:49.487 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:49.488 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:49.488 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:23:49.488 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.495523+0000", 2024-09-13T11:23:49.488 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00017640000000000001 2024-09-13T11:23:49.488 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:49.488 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:49.488 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 12", 2024-09-13T11:23:49.488 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.495699+0000", 2024-09-13T11:23:49.488 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00017653600000000001 2024-09-13T11:23:49.488 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:49.488 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:49.488 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:23:49.488 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.497040+0000", 2024-09-13T11:23:49.488 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001340473 2024-09-13T11:23:49.489 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:49.489 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:49.489 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:49.489 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.724049+0000", 2024-09-13T11:23:49.489 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.22700931799999999 2024-09-13T11:23:49.489 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:49.489 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:49.489 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:23:49.489 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.724215+0000", 2024-09-13T11:23:49.489 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000165988 2024-09-13T11:23:49.489 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:49.489 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:49.489 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:23:49.490 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.724343+0000", 2024-09-13T11:23:49.490 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000127601 2024-09-13T11:23:49.490 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:49.490 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:23:49.490 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:49.490 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:49.490 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:49.490 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:30828 3.f 3:f3a9c6a9:::benchmark_data_smithi177_31116_object30827:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e59)", 2024-09-13T11:23:49.490 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:09.503661+0000", 2024-09-13T11:23:49.490 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 39.973533613000001, 2024-09-13T11:23:49.490 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.26477667100000002, 2024-09-13T11:23:49.490 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:23:49.490 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:23:49.490 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:49.490 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:23:49.491 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:23:49.491 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:49.491 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 30828 2024-09-13T11:23:49.491 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:49.491 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:23:49.491 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:49.491 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:23:49.491 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.503661+0000", 2024-09-13T11:23:49.491 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:49.491 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:49.491 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:49.491 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:23:49.491 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.503661+0000", 2024-09-13T11:23:49.491 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:49.492 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:49.492 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:49.492 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:23:49.492 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.503663+0000", 2024-09-13T11:23:49.492 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.968e-06 2024-09-13T11:23:49.492 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:49.492 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:49.492 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:23:49.492 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.503687+0000", 2024-09-13T11:23:49.492 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.3983999999999999e-05 2024-09-13T11:23:49.493 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:49.493 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:49.493 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:23:49.493 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.503701+0000", 2024-09-13T11:23:49.493 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3416e-05 2024-09-13T11:23:49.493 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:49.493 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:49.494 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:23:49.494 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.503708+0000", 2024-09-13T11:23:49.494 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.9530000000000002e-06 2024-09-13T11:23:49.494 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:49.494 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:49.494 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:23:49.494 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.503759+0000", 2024-09-13T11:23:49.494 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.1075999999999997e-05 2024-09-13T11:23:49.494 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:49.495 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:49.495 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:23:49.495 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.503861+0000", 2024-09-13T11:23:49.495 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000102372 2024-09-13T11:23:49.495 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:49.495 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:49.495 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 12", 2024-09-13T11:23:49.495 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.503952+0000", 2024-09-13T11:23:49.496 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.1135000000000001e-05 2024-09-13T11:23:49.496 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:49.496 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:49.496 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:23:49.496 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.505340+0000", 2024-09-13T11:23:49.496 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0013877169999999999 2024-09-13T11:23:49.496 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:49.496 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:49.497 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:49.497 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.768281+0000", 2024-09-13T11:23:49.497 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.26294092400000002 2024-09-13T11:23:49.497 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:49.497 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:49.497 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:23:49.497 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.768369+0000", 2024-09-13T11:23:49.497 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.7818999999999994e-05 2024-09-13T11:23:49.497 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:49.498 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:49.498 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:23:49.498 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.768438+0000", 2024-09-13T11:23:49.498 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.9307000000000006e-05 2024-09-13T11:23:49.498 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:49.498 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:23:49.498 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:49.498 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:49.499 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:49.499 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:45783 3.16 3.17c10a76 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e69)", 2024-09-13T11:23:49.499 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:20.947774+0000", 2024-09-13T11:23:49.499 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 28.529420711, 2024-09-13T11:23:49.499 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.96682922400000004, 2024-09-13T11:23:49.499 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:23:49.499 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:23:49.499 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-13T11:23:49.499 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:23:49.500 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:23:49.500 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:49.500 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 45783 2024-09-13T11:23:49.500 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:49.500 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:23:49.500 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:49.500 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:23:49.500 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:20.947774+0000", 2024-09-13T11:23:49.500 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:49.501 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:49.501 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:49.501 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:23:49.501 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:20.947774+0000", 2024-09-13T11:23:49.501 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:49.501 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:49.501 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:49.501 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:23:49.502 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:20.947777+0000", 2024-09-13T11:23:49.502 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.57e-06 2024-09-13T11:23:49.502 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:49.502 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:49.502 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:23:49.502 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:20.947802+0000", 2024-09-13T11:23:49.502 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.5528999999999999e-05 2024-09-13T11:23:49.502 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:49.502 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:49.503 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:23:49.503 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:20.947813+0000", 2024-09-13T11:23:49.503 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0473e-05 2024-09-13T11:23:49.503 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:49.503 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:49.503 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:23:49.503 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:20.947821+0000", 2024-09-13T11:23:49.503 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.3440000000000001e-06 2024-09-13T11:23:49.504 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:49.504 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:49.504 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:23:49.504 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:20.947933+0000", 2024-09-13T11:23:49.504 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00011210200000000001 2024-09-13T11:23:49.504 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:49.504 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:49.504 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for peered", 2024-09-13T11:23:49.504 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:20.947942+0000", 2024-09-13T11:23:49.505 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.6610000000000006e-06 2024-09-13T11:23:49.505 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:49.505 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:49.505 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:23:49.505 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:21.914579+0000", 2024-09-13T11:23:49.505 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.96663672300000003 2024-09-13T11:23:49.505 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:49.505 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:49.506 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:23:49.506 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:21.914603+0000", 2024-09-13T11:23:49.506 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.4822000000000001e-05 2024-09-13T11:23:49.506 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:49.506 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:23:49.506 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:49.506 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:49.506 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:49.506 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPush(3.13 72/70 [PushOp(3:cd7b7b57:::benchmark_data_smithi177_31116_object7402:head, version: 46'454, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:cd7b7b57:::benchmark_data_smithi177_31116_object7402:head@46'454, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:23:49.507 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:26.312967+0000", 2024-09-13T11:23:49.507 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 23.164227440000001, 2024-09-13T11:23:49.507 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.16922120399999999, 2024-09-13T11:23:49.507 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:23:49.507 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:23:49.507 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:23:49.507 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:23:49.507 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:49.507 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:23:49.507 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:26.312967+0000", 2024-09-13T11:23:49.507 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:49.508 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:49.508 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:49.508 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:23:49.508 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:26.312967+0000", 2024-09-13T11:23:49.508 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:49.508 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:49.508 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:49.508 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:23:49.508 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:26.312969+0000", 2024-09-13T11:23:49.508 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3659999999999999e-06 2024-09-13T11:23:49.508 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:49.508 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:49.508 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:23:49.509 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:26.312993+0000", 2024-09-13T11:23:49.509 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.4593999999999999e-05 2024-09-13T11:23:49.509 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:49.509 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:49.509 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:23:49.509 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:26.480625+0000", 2024-09-13T11:23:49.509 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.167631369 2024-09-13T11:23:49.509 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:49.509 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:49.509 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:23:49.509 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:26.480636+0000", 2024-09-13T11:23:49.509 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.1643e-05 2024-09-13T11:23:49.509 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:49.510 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:49.510 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:23:49.510 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:26.480867+0000", 2024-09-13T11:23:49.510 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000230435 2024-09-13T11:23:49.510 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:49.510 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:49.510 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:23:49.510 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:26.480890+0000", 2024-09-13T11:23:49.510 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.2895999999999999e-05 2024-09-13T11:23:49.510 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:49.510 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:49.510 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:23:49.510 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:26.482189+0000", 2024-09-13T11:23:49.510 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0012989010000000001 2024-09-13T11:23:49.511 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:49.511 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:23:49.511 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:49.511 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:49.511 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:49.511 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:54651 3.f 3:f1b76062:::benchmark_data_smithi177_31116_object54650:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e74)", 2024-09-13T11:23:49.511 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:30.975158+0000", 2024-09-13T11:23:49.511 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 18.502036871000001, 2024-09-13T11:23:49.511 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.14666372499999999, 2024-09-13T11:23:49.512 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:23:49.512 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:23:49.512 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-13T11:23:49.512 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:23:49.512 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:23:49.512 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:49.512 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 54651 2024-09-13T11:23:49.512 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:49.512 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:23:49.512 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:49.512 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:23:49.512 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.975158+0000", 2024-09-13T11:23:49.512 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:49.512 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:49.513 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:49.513 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:23:49.513 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.975158+0000", 2024-09-13T11:23:49.513 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:49.513 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:49.513 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:49.513 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:23:49.513 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.975161+0000", 2024-09-13T11:23:49.513 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.5359999999999999e-06 2024-09-13T11:23:49.513 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:49.513 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:49.513 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:23:49.513 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.975217+0000", 2024-09-13T11:23:49.514 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.6555000000000003e-05 2024-09-13T11:23:49.514 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:49.514 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:49.514 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:23:49.514 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.975233+0000", 2024-09-13T11:23:49.514 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.6239000000000002e-05 2024-09-13T11:23:49.514 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:49.514 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:49.514 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:23:49.514 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.975246+0000", 2024-09-13T11:23:49.514 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.2476e-05 2024-09-13T11:23:49.514 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:49.514 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:49.514 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:23:49.515 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.975414+0000", 2024-09-13T11:23:49.515 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00016866299999999999 2024-09-13T11:23:49.515 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:49.515 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:49.515 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:23:49.515 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.975626+0000", 2024-09-13T11:23:49.515 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000211366 2024-09-13T11:23:49.515 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:49.515 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:49.515 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 12", 2024-09-13T11:23:49.515 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.975897+0000", 2024-09-13T11:23:49.515 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000271308 2024-09-13T11:23:49.515 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:49.515 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:49.515 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:23:49.516 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.980197+0000", 2024-09-13T11:23:49.516 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0043000499999999997 2024-09-13T11:23:49.516 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:49.516 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:49.516 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:23:49.516 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.121811+0000", 2024-09-13T11:23:49.516 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.14161378099999999 2024-09-13T11:23:49.516 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:49.516 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:49.516 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:23:49.516 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.121822+0000", 2024-09-13T11:23:49.516 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0750999999999999e-05 2024-09-13T11:23:49.516 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:49.517 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:23:49.517 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:49.517 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:49.517 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:49.517 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:54625 3.1f 3:f812cdfb:::benchmark_data_smithi177_31116_object54624:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e75)", 2024-09-13T11:23:49.517 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:30.979327+0000", 2024-09-13T11:23:49.517 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 18.497867922000001, 2024-09-13T11:23:49.517 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.96163670899999998, 2024-09-13T11:23:49.517 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:23:49.517 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:23:49.517 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-13T11:23:49.517 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:23:49.517 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:23:49.517 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:49.518 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 54625 2024-09-13T11:23:49.518 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:49.518 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:23:49.518 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:49.518 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:23:49.518 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.979327+0000", 2024-09-13T11:23:49.518 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:49.518 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:49.518 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:49.518 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:23:49.518 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.979327+0000", 2024-09-13T11:23:49.518 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:49.518 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:49.518 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:49.519 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:23:49.519 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.979329+0000", 2024-09-13T11:23:49.519 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.686e-06 2024-09-13T11:23:49.519 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:49.519 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:49.519 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:23:49.519 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.979364+0000", 2024-09-13T11:23:49.519 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.5750000000000002e-05 2024-09-13T11:23:49.519 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:49.519 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:49.519 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:23:49.519 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.979408+0000", 2024-09-13T11:23:49.519 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.3795999999999999e-05 2024-09-13T11:23:49.519 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:49.520 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:49.520 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:23:49.520 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.979427+0000", 2024-09-13T11:23:49.520 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.9040999999999999e-05 2024-09-13T11:23:49.520 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:49.520 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:49.520 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:23:49.520 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.122291+0000", 2024-09-13T11:23:49.520 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.14286373599999999 2024-09-13T11:23:49.520 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:49.520 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:49.520 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for peered", 2024-09-13T11:23:49.520 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.122298+0000", 2024-09-13T11:23:49.520 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.3719999999999999e-06 2024-09-13T11:23:49.521 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:49.521 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:49.521 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:23:49.521 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.940341+0000", 2024-09-13T11:23:49.521 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.81804249699999998 2024-09-13T11:23:49.521 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:49.521 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:49.521 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:23:49.521 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.940964+0000", 2024-09-13T11:23:49.521 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00062283100000000001 2024-09-13T11:23:49.521 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:49.521 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:23:49.521 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:49.522 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:49.522 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:49.522 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:54649 3.1c 3:3a457a93:::benchmark_data_smithi177_31116_object54648:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e75)", 2024-09-13T11:23:49.522 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:30.979460+0000", 2024-09-13T11:23:49.522 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 18.497734732000001, 2024-09-13T11:23:49.522 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.95579424000000002, 2024-09-13T11:23:49.522 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:23:49.522 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:23:49.522 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-13T11:23:49.522 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:23:49.522 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:23:49.522 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:49.522 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 54649 2024-09-13T11:23:49.522 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:49.523 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:23:49.523 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:49.523 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:23:49.523 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.979460+0000", 2024-09-13T11:23:49.523 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:49.523 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:49.523 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:49.523 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:23:49.523 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.979460+0000", 2024-09-13T11:23:49.523 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:49.523 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:49.524 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:49.524 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:23:49.524 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.979462+0000", 2024-09-13T11:23:49.524 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5799999999999999e-06 2024-09-13T11:23:49.524 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:49.524 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:49.524 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:23:49.524 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.979524+0000", 2024-09-13T11:23:49.524 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.2521999999999995e-05 2024-09-13T11:23:49.524 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:49.524 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:49.524 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:23:49.524 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.979561+0000", 2024-09-13T11:23:49.524 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.6349e-05 2024-09-13T11:23:49.525 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:49.525 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:49.525 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:23:49.525 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.979571+0000", 2024-09-13T11:23:49.525 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0457e-05 2024-09-13T11:23:49.525 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:49.525 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:49.525 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:23:49.525 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.121029+0000", 2024-09-13T11:23:49.525 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.14145766200000001 2024-09-13T11:23:49.525 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:49.525 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:49.525 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for peered", 2024-09-13T11:23:49.526 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.121039+0000", 2024-09-13T11:23:49.526 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0084e-05 2024-09-13T11:23:49.526 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:49.526 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:49.526 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:23:49.526 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.935157+0000", 2024-09-13T11:23:49.526 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.81411807899999999 2024-09-13T11:23:49.526 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:49.526 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:49.526 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:23:49.526 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.935254+0000", 2024-09-13T11:23:49.526 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.7507e-05 2024-09-13T11:23:49.526 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:49.526 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:23:49.527 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:49.527 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:49.527 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:49.527 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:54651 3.f 3.4606ed8f (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e75)", 2024-09-13T11:23:49.527 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:30.979590+0000", 2024-09-13T11:23:49.527 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 18.497605324999999, 2024-09-13T11:23:49.527 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.142765428, 2024-09-13T11:23:49.527 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:23:49.527 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:23:49.527 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-13T11:23:49.527 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:23:49.527 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:23:49.527 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:49.527 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 54651 2024-09-13T11:23:49.528 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:49.528 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:23:49.528 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:49.528 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:23:49.528 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.979590+0000", 2024-09-13T11:23:49.528 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:49.528 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:49.528 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:49.528 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:23:49.528 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.979590+0000", 2024-09-13T11:23:49.528 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:49.528 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:49.528 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:49.529 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:23:49.529 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.979591+0000", 2024-09-13T11:23:49.529 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5680000000000001e-06 2024-09-13T11:23:49.529 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:49.529 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:49.529 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:23:49.529 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.979622+0000", 2024-09-13T11:23:49.529 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.1233999999999999e-05 2024-09-13T11:23:49.529 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:49.529 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:49.529 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:23:49.529 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.979633+0000", 2024-09-13T11:23:49.529 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.1066e-05 2024-09-13T11:23:49.530 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:49.530 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:49.530 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:23:49.530 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.979643+0000", 2024-09-13T11:23:49.530 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.6509999999999996e-06 2024-09-13T11:23:49.530 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:49.530 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:49.530 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:23:49.530 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.979906+0000", 2024-09-13T11:23:49.530 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00026331400000000001 2024-09-13T11:23:49.530 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:49.530 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:49.530 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op must wait for map", 2024-09-13T11:23:49.530 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.979915+0000", 2024-09-13T11:23:49.531 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.6039999999999993e-06 2024-09-13T11:23:49.531 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:49.531 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:49.531 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:23:49.531 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.122322+0000", 2024-09-13T11:23:49.531 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.142407056 2024-09-13T11:23:49.531 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:49.531 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:49.531 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:23:49.531 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.122355+0000", 2024-09-13T11:23:49.531 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.2935000000000002e-05 2024-09-13T11:23:49.531 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:49.531 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:23:49.531 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:49.531 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:49.532 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:49.532 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:54654 3.1c 3:3f0bb31a:::benchmark_data_smithi177_31116_object54653:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e75)", 2024-09-13T11:23:49.532 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:30.979665+0000", 2024-09-13T11:23:49.532 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 18.497529567000001, 2024-09-13T11:23:49.532 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.95623908099999999, 2024-09-13T11:23:49.532 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:23:49.532 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:23:49.532 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-13T11:23:49.532 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:23:49.532 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:23:49.532 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:49.532 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 54654 2024-09-13T11:23:49.532 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:49.533 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:23:49.533 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:49.533 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:23:49.533 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.979665+0000", 2024-09-13T11:23:49.533 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:49.533 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:49.533 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:49.533 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:23:49.533 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.979665+0000", 2024-09-13T11:23:49.533 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:49.533 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:49.533 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:49.533 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:23:49.533 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.979667+0000", 2024-09-13T11:23:49.534 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3540000000000001e-06 2024-09-13T11:23:49.534 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:49.534 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:49.534 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:23:49.534 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.979703+0000", 2024-09-13T11:23:49.534 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.6758000000000003e-05 2024-09-13T11:23:49.534 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:49.534 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:49.534 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:23:49.534 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.979714+0000", 2024-09-13T11:23:49.534 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0804e-05 2024-09-13T11:23:49.535 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:49.535 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:49.535 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:23:49.535 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.979724+0000", 2024-09-13T11:23:49.535 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0040999999999999e-05 2024-09-13T11:23:49.535 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:49.535 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:49.535 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:23:49.535 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.121061+0000", 2024-09-13T11:23:49.535 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.141336713 2024-09-13T11:23:49.535 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:49.535 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:49.535 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for peered", 2024-09-13T11:23:49.535 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.121067+0000", 2024-09-13T11:23:49.536 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.4150000000000004e-06 2024-09-13T11:23:49.536 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:49.536 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:49.536 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:23:49.536 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.935776+0000", 2024-09-13T11:23:49.536 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.81470863100000002 2024-09-13T11:23:49.536 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:49.536 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:49.536 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:23:49.536 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.935904+0000", 2024-09-13T11:23:49.536 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00012836499999999999 2024-09-13T11:23:49.536 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:49.536 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:23:49.537 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:49.537 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:49.537 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:49.537 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPushReply(3.11 75/70 [PushReplyOp(3:8df10771:::benchmark_data_smithi177_31116_object24734:head)])", 2024-09-13T11:23:49.537 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:31.149798+0000", 2024-09-13T11:23:49.537 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 18.327396557, 2024-09-13T11:23:49.537 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.32402702500000002, 2024-09-13T11:23:49.537 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:23:49.537 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:23:49.537 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-13T11:23:49.537 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:23:49.537 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:49.537 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:23:49.538 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.149798+0000", 2024-09-13T11:23:49.538 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:49.538 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:49.538 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:49.538 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:23:49.538 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.149798+0000", 2024-09-13T11:23:49.538 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:49.538 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:49.538 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:49.538 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:23:49.538 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.149799+0000", 2024-09-13T11:23:49.538 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.5700000000000001e-07 2024-09-13T11:23:49.538 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:49.539 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:49.539 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:23:49.539 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.149803+0000", 2024-09-13T11:23:49.539 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.0040000000000001e-06 2024-09-13T11:23:49.539 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:49.539 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:49.539 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:23:49.539 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.472172+0000", 2024-09-13T11:23:49.539 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.32236856699999999 2024-09-13T11:23:49.539 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:49.539 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:49.539 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:23:49.539 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.472187+0000", 2024-09-13T11:23:49.539 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5016e-05 2024-09-13T11:23:49.539 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:49.540 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:49.540 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:23:49.540 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.472302+0000", 2024-09-13T11:23:49.540 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000115121 2024-09-13T11:23:49.540 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:49.540 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:49.540 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:23:49.540 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.473825+0000", 2024-09-13T11:23:49.540 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0015234599999999999 2024-09-13T11:23:49.540 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:49.540 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:23:49.540 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:49.540 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:49.540 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:49.541 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:54668 3.1c 3:3d2c5e40:::benchmark_data_smithi177_31116_object54667:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e75)", 2024-09-13T11:23:49.541 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:31.245525+0000", 2024-09-13T11:23:49.541 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 18.231670300000001, 2024-09-13T11:23:49.541 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.69828973000000005, 2024-09-13T11:23:49.541 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:23:49.541 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:23:49.541 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:49.541 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:23:49.541 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:23:49.541 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:49.541 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 54668 2024-09-13T11:23:49.542 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:49.542 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:23:49.542 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:49.542 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:23:49.542 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.245525+0000", 2024-09-13T11:23:49.542 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:49.542 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:49.542 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:49.542 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:23:49.542 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.245525+0000", 2024-09-13T11:23:49.542 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:49.544 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:49.544 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:49.544 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:23:49.544 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.245526+0000", 2024-09-13T11:23:49.544 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.719e-06 2024-09-13T11:23:49.544 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:49.544 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:49.544 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:23:49.544 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.245547+0000", 2024-09-13T11:23:49.544 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.1084999999999999e-05 2024-09-13T11:23:49.544 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:49.544 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:49.545 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:23:49.545 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.245558+0000", 2024-09-13T11:23:49.545 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0934000000000001e-05 2024-09-13T11:23:49.545 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:49.545 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:49.545 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:23:49.545 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.245564+0000", 2024-09-13T11:23:49.545 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.6269999999999997e-06 2024-09-13T11:23:49.545 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:49.545 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:49.545 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:23:49.545 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.245620+0000", 2024-09-13T11:23:49.545 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.6289999999999998e-05 2024-09-13T11:23:49.545 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:49.546 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:49.546 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for peered", 2024-09-13T11:23:49.546 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.245632+0000", 2024-09-13T11:23:49.546 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.1748e-05 2024-09-13T11:23:49.546 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:49.546 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:49.546 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:23:49.546 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.935976+0000", 2024-09-13T11:23:49.546 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.69034455299999997 2024-09-13T11:23:49.546 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:49.546 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:49.546 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:23:49.546 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.936302+0000", 2024-09-13T11:23:49.547 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000325397 2024-09-13T11:23:49.547 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:49.547 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:49.547 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 13", 2024-09-13T11:23:49.547 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.936459+0000", 2024-09-13T11:23:49.547 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00015754300000000001 2024-09-13T11:23:49.547 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:49.547 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:49.547 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:23:49.547 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.938813+0000", 2024-09-13T11:23:49.547 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0023530999999999999 2024-09-13T11:23:49.547 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:49.547 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:49.547 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:49.548 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.943695+0000", 2024-09-13T11:23:49.548 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.004882506 2024-09-13T11:23:49.548 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:49.548 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:49.548 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:23:49.548 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.943767+0000", 2024-09-13T11:23:49.548 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.2454999999999997e-05 2024-09-13T11:23:49.548 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:49.549 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:49.549 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:23:49.549 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.943814+0000", 2024-09-13T11:23:49.549 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.6773000000000001e-05 2024-09-13T11:23:49.549 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:49.549 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:23:49.549 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:49.549 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:49.549 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:49.549 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPushReply(3.11 76/70 [PushReplyOp(3:8a23be1f:::benchmark_data_smithi177_31116_object31379:head)])", 2024-09-13T11:23:49.549 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:32.588647+0000", 2024-09-13T11:23:49.549 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 16.888547751000001, 2024-09-13T11:23:49.549 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.20841558099999999, 2024-09-13T11:23:49.549 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:23:49.550 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:23:49.550 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-13T11:23:49.550 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:23:49.550 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:49.550 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:23:49.550 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:32.588647+0000", 2024-09-13T11:23:49.550 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:49.550 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:49.550 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:49.550 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:23:49.550 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:32.588647+0000", 2024-09-13T11:23:49.550 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:49.550 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:49.551 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:49.551 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:23:49.551 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:32.588648+0000", 2024-09-13T11:23:49.551 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.265e-06 2024-09-13T11:23:49.551 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:49.551 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:49.551 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:23:49.551 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:32.588652+0000", 2024-09-13T11:23:49.551 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.1069999999999998e-06 2024-09-13T11:23:49.551 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:49.551 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:49.551 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:23:49.552 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:32.796501+0000", 2024-09-13T11:23:49.552 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.20784898199999999 2024-09-13T11:23:49.552 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:49.552 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:49.552 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:23:49.552 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:32.796509+0000", 2024-09-13T11:23:49.552 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.8099999999999998e-06 2024-09-13T11:23:49.552 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:49.552 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:49.552 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:23:49.552 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:32.796572+0000", 2024-09-13T11:23:49.552 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.2352000000000002e-05 2024-09-13T11:23:49.552 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:49.552 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:49.553 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:23:49.553 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:32.797063+0000", 2024-09-13T11:23:49.553 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00049106499999999995 2024-09-13T11:23:49.553 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:49.553 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:23:49.553 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:49.553 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:49.553 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:49.553 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPush(3.16 84/70 [PushOp(3:6f7be541:::benchmark_data_smithi177_31116_object43332:head, version: 66'2788, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:6f7be541:::benchmark_data_smithi177_31116_object43332:head@66'2788, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:23:49.553 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:41.207515+0000", 2024-09-13T11:23:49.553 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 8.2696795289999994, 2024-09-13T11:23:49.553 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.52643771500000003, 2024-09-13T11:23:49.553 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:23:49.553 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:23:49.554 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:23:49.554 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:23:49.554 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:49.554 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:23:49.554 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.207515+0000", 2024-09-13T11:23:49.554 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:49.554 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:49.554 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:49.554 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:23:49.554 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.207515+0000", 2024-09-13T11:23:49.554 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:49.554 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:49.554 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:49.554 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:23:49.554 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.207516+0000", 2024-09-13T11:23:49.555 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.096e-06 2024-09-13T11:23:49.555 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:49.555 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:49.555 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:23:49.555 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.207717+0000", 2024-09-13T11:23:49.555 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000200199 2024-09-13T11:23:49.555 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:49.555 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:49.555 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:23:49.555 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.732785+0000", 2024-09-13T11:23:49.555 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.52506869599999995 2024-09-13T11:23:49.555 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:49.556 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:49.556 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:23:49.556 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.732801+0000", 2024-09-13T11:23:49.556 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5305999999999999e-05 2024-09-13T11:23:49.556 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:49.556 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:49.556 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:23:49.556 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.732889+0000", 2024-09-13T11:23:49.556 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.8157000000000003e-05 2024-09-13T11:23:49.556 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:49.556 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:49.556 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:23:49.556 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.732901+0000", 2024-09-13T11:23:49.556 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.1866e-05 2024-09-13T11:23:49.557 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:49.557 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:49.557 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:23:49.557 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.733953+0000", 2024-09-13T11:23:49.557 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001052395 2024-09-13T11:23:49.557 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:49.557 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:23:49.557 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:49.557 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:49.557 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:49.557 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPushReply(3.1f 85/78 [PushReplyOp(3:fd221d21:::benchmark_data_smithi177_31116_object7639:head)])", 2024-09-13T11:23:49.557 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:42.198715+0000", 2024-09-13T11:23:49.557 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 7.2784799600000003, 2024-09-13T11:23:49.558 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.48930391099999998, 2024-09-13T11:23:49.558 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:23:49.558 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:23:49.558 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-13T11:23:49.558 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:23:49.558 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:49.558 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:23:49.558 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.198715+0000", 2024-09-13T11:23:49.558 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:49.558 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:49.558 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:49.558 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:23:49.558 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.198715+0000", 2024-09-13T11:23:49.559 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:49.559 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:49.559 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:49.559 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:23:49.559 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.198716+0000", 2024-09-13T11:23:49.559 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.3500000000000005e-07 2024-09-13T11:23:49.559 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:49.559 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:49.559 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:23:49.559 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.198719+0000", 2024-09-13T11:23:49.559 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.4539999999999999e-06 2024-09-13T11:23:49.559 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:49.560 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:49.560 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:23:49.560 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.684732+0000", 2024-09-13T11:23:49.560 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.48601262099999998 2024-09-13T11:23:49.560 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:49.560 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:49.560 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:23:49.560 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.684746+0000", 2024-09-13T11:23:49.560 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.4518999999999999e-05 2024-09-13T11:23:49.560 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:49.560 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:49.560 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:23:49.560 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.684911+0000", 2024-09-13T11:23:49.561 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000164728 2024-09-13T11:23:49.561 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:49.561 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:49.561 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:23:49.561 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.688019+0000", 2024-09-13T11:23:49.561 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0031077539999999999 2024-09-13T11:23:49.561 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:49.561 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:23:49.561 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:49.561 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:49.561 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:49.561 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPushReply(3.1c 85/77 [PushReplyOp(3:3a669437:::benchmark_data_smithi177_31116_object25280:head)])", 2024-09-13T11:23:49.561 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:42.352466+0000", 2024-09-13T11:23:49.561 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 7.1247290660000004, 2024-09-13T11:23:49.562 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.36670725300000001, 2024-09-13T11:23:49.562 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:23:49.562 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:23:49.562 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-13T11:23:49.562 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:23:49.562 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:49.562 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:23:49.562 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.352466+0000", 2024-09-13T11:23:49.562 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:49.562 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:49.562 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:49.562 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:23:49.562 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.352466+0000", 2024-09-13T11:23:49.562 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:49.563 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:49.563 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:49.563 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:23:49.563 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.352467+0000", 2024-09-13T11:23:49.563 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.1429999999999999e-06 2024-09-13T11:23:49.563 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:49.563 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:49.563 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:23:49.563 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.352474+0000", 2024-09-13T11:23:49.563 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.6100000000000002e-06 2024-09-13T11:23:49.564 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:49.564 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:49.564 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:23:49.564 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.717011+0000", 2024-09-13T11:23:49.564 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.36453790200000002 2024-09-13T11:23:49.564 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:49.564 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:49.564 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:23:49.564 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.717032+0000", 2024-09-13T11:23:49.564 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.0999e-05 2024-09-13T11:23:49.564 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:49.564 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:49.564 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:23:49.565 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.717162+0000", 2024-09-13T11:23:49.565 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000129159 2024-09-13T11:23:49.565 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:49.565 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:49.565 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:23:49.565 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.719173+0000", 2024-09-13T11:23:49.565 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0020114400000000002 2024-09-13T11:23:49.565 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:49.565 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:23:49.565 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:49.565 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:49.565 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:49.565 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPush(3.16 85/70 [PushOp(3:6aef2f27:::benchmark_data_smithi177_31116_object44090:head, version: 68'2844, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:6aef2f27:::benchmark_data_smithi177_31116_object44090:head@68'2844, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:23:49.565 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:42.574221+0000", 2024-09-13T11:23:49.566 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 6.9029739580000005, 2024-09-13T11:23:49.566 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.25345585700000001, 2024-09-13T11:23:49.566 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:23:49.566 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:23:49.566 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:23:49.566 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:23:49.566 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:49.566 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:23:49.566 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.574221+0000", 2024-09-13T11:23:49.566 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:49.566 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:49.566 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:49.566 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:23:49.566 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.574221+0000", 2024-09-13T11:23:49.567 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:49.567 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:49.567 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:49.567 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:23:49.567 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.574252+0000", 2024-09-13T11:23:49.567 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.0973000000000003e-05 2024-09-13T11:23:49.567 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:49.567 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:49.567 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:23:49.567 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.574434+0000", 2024-09-13T11:23:49.567 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000182265 2024-09-13T11:23:49.567 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:49.567 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:49.567 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:23:49.567 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.825391+0000", 2024-09-13T11:23:49.568 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.250957023 2024-09-13T11:23:49.568 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:49.568 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:49.568 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:23:49.568 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.825442+0000", 2024-09-13T11:23:49.568 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.1267999999999998e-05 2024-09-13T11:23:49.568 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:49.568 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:49.568 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:23:49.569 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.825692+0000", 2024-09-13T11:23:49.569 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00024924700000000002 2024-09-13T11:23:49.569 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:49.569 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:49.569 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:23:49.569 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.825715+0000", 2024-09-13T11:23:49.569 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.3176e-05 2024-09-13T11:23:49.569 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:49.569 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:49.569 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:23:49.570 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.827677+0000", 2024-09-13T11:23:49.570 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0019619049999999999 2024-09-13T11:23:49.570 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:49.570 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:23:49.570 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:49.570 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:49.570 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:49.570 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPushReply(3.1f 85/78 [PushReplyOp(3:fecb0d6a:::benchmark_data_smithi177_31116_object7644:head)])", 2024-09-13T11:23:49.570 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:42.705351+0000", 2024-09-13T11:23:49.570 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 6.7718442239999996, 2024-09-13T11:23:49.570 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.290549681, 2024-09-13T11:23:49.570 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:23:49.570 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:23:49.570 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-13T11:23:49.571 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:23:49.571 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:49.571 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:23:49.571 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.705351+0000", 2024-09-13T11:23:49.571 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:49.571 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:49.571 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:49.571 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:23:49.571 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.705351+0000", 2024-09-13T11:23:49.571 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:49.571 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:49.571 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:49.572 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:23:49.572 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.705353+0000", 2024-09-13T11:23:49.572 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.4169999999999999e-06 2024-09-13T11:23:49.572 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:49.572 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:49.572 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:23:49.572 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.705365+0000", 2024-09-13T11:23:49.572 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.1894e-05 2024-09-13T11:23:49.572 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:49.572 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:49.572 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:23:49.572 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.992081+0000", 2024-09-13T11:23:49.572 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.28671641199999998 2024-09-13T11:23:49.573 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:49.573 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:49.573 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:23:49.573 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.992111+0000", 2024-09-13T11:23:49.573 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.9427000000000001e-05 2024-09-13T11:23:49.573 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:49.573 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:49.573 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:23:49.573 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.993573+0000", 2024-09-13T11:23:49.573 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0014623590000000001 2024-09-13T11:23:49.573 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:49.573 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:49.573 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:23:49.573 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.995900+0000", 2024-09-13T11:23:49.574 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0023271720000000002 2024-09-13T11:23:49.574 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:49.574 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:23:49.574 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:49.574 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:49.574 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:49.574 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPushReply(3.21 89/78 [PushReplyOp(3:8534d087:::benchmark_data_smithi177_31116_object5474:head)])", 2024-09-13T11:23:49.574 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:46.632834+0000", 2024-09-13T11:23:49.574 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 2.844360784, 2024-09-13T11:23:49.574 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.324805556, 2024-09-13T11:23:49.574 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:23:49.574 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:23:49.574 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-13T11:23:49.574 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:23:49.575 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:49.575 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:23:49.575 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.632834+0000", 2024-09-13T11:23:49.575 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:49.575 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:49.575 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:49.575 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:23:49.575 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.632834+0000", 2024-09-13T11:23:49.575 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:49.575 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:49.575 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:49.575 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:23:49.575 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.632835+0000", 2024-09-13T11:23:49.575 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.4270000000000001e-06 2024-09-13T11:23:49.576 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:49.576 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:49.576 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:23:49.576 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.632840+0000", 2024-09-13T11:23:49.576 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.7809999999999996e-06 2024-09-13T11:23:49.576 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:49.576 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:49.576 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:23:49.576 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.956282+0000", 2024-09-13T11:23:49.576 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.323441639 2024-09-13T11:23:49.576 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:49.576 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:49.576 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:23:49.577 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.956293+0000", 2024-09-13T11:23:49.577 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.083e-05 2024-09-13T11:23:49.577 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:49.577 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:49.577 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:23:49.577 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.956395+0000", 2024-09-13T11:23:49.577 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00010184200000000001 2024-09-13T11:23:49.577 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:49.577 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:49.577 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:23:49.577 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.957640+0000", 2024-09-13T11:23:49.577 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0012450370000000001 2024-09-13T11:23:49.577 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:49.577 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:23:49.578 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:49.578 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:49.578 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:49.578 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPushReply(3.1f 89/78 [PushReplyOp(3:ff150976:::benchmark_data_smithi177_31116_object12907:head)])", 2024-09-13T11:23:49.578 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:46.697504+0000", 2024-09-13T11:23:49.578 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 2.7796911849999999, 2024-09-13T11:23:49.578 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.26836818099999998, 2024-09-13T11:23:49.578 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:23:49.578 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:23:49.578 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-13T11:23:49.578 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:23:49.578 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:49.578 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:23:49.578 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.697504+0000", 2024-09-13T11:23:49.579 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:49.579 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:49.579 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:49.579 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:23:49.579 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.697504+0000", 2024-09-13T11:23:49.579 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:49.579 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:49.579 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:49.579 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:23:49.579 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.697505+0000", 2024-09-13T11:23:49.579 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0529999999999999e-06 2024-09-13T11:23:49.579 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:49.579 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:49.579 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:23:49.579 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.697509+0000", 2024-09-13T11:23:49.580 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.9249999999999997e-06 2024-09-13T11:23:49.580 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:49.580 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:49.580 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:23:49.580 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.964086+0000", 2024-09-13T11:23:49.580 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.26657776100000002 2024-09-13T11:23:49.580 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:49.580 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:49.580 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:23:49.580 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.964110+0000", 2024-09-13T11:23:49.580 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.3189000000000001e-05 2024-09-13T11:23:49.580 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:49.580 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:49.581 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:23:49.581 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.964354+0000", 2024-09-13T11:23:49.581 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00024481399999999999 2024-09-13T11:23:49.581 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:49.581 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:49.581 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:23:49.581 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.965872+0000", 2024-09-13T11:23:49.581 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001517439 2024-09-13T11:23:49.581 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:49.581 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:23:49.581 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:49.581 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:49.581 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:23:49.581 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-09-13T11:23:49.583 INFO:teuthology.orchestra.run.smithi001.stdout:osd.10: {} 2024-09-13T11:23:49.583 INFO:teuthology.orchestra.run.smithi001.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-13T11:23:49.584 INFO:teuthology.orchestra.run.smithi001.stdout:osd.11: {} 2024-09-13T11:23:49.584 INFO:teuthology.orchestra.run.smithi001.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-13T11:23:49.584 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_ops_in_flight 2024-09-13T11:23:49.585 INFO:tasks.ceph.osd.11.smithi177.stderr:2024-09-13T11:23:49.576+0000 7f24792c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26580) UID: 0 2024-09-13T11:23:49.585 INFO:teuthology.orchestra.run.smithi001.stdout:osd.12: {} 2024-09-13T11:23:49.585 INFO:teuthology.orchestra.run.smithi001.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-13T11:23:49.596 INFO:teuthology.orchestra.run.smithi001.stdout:osd.13: {} 2024-09-13T11:23:49.596 INFO:teuthology.orchestra.run.smithi001.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-13T11:23:49.620 INFO:teuthology.orchestra.run.smithi001.stdout:osd.14: {} 2024-09-13T11:23:49.620 INFO:teuthology.orchestra.run.smithi001.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-13T11:23:49.645 INFO:teuthology.orchestra.run.smithi001.stdout:osd.15: {} 2024-09-13T11:23:49.645 INFO:teuthology.orchestra.run.smithi001.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-13T11:23:49.684 INFO:tasks.ceph.osd.13.smithi092.stderr:2024-09-13T11:23:49.679+0000 7f3e5afea640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26350) UID: 0 2024-09-13T11:23:49.778 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-09-13T11:23:49.779 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:23:49.785 INFO:tasks.ceph.osd.14.smithi162.stderr:2024-09-13T11:23:49.781+0000 7f235c0db640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26319) UID: 0 2024-09-13T11:23:49.796 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_blocked_ops 2024-09-13T11:23:49.885 INFO:tasks.ceph.osd.3.smithi177.stderr:2024-09-13T11:23:49.880+0000 7f10adba4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26574) UID: 0 2024-09-13T11:23:49.933 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-09-13T11:23:49.933 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:23:49.953 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_historic_ops 2024-09-13T11:23:49.987 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-09-13T11:23:49.985+0000 7f97befc9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26315) UID: 0 2024-09-13T11:23:50.006 INFO:tasks.radosbench.radosbench.0.smithi177.stdout: 83 16 66795 66779 50.2801 80.25 0.0471766 0.019879 2024-09-13T11:23:50.088 INFO:tasks.ceph.osd.4.smithi001.stderr:2024-09-13T11:23:50.080+0000 7ff8e0218640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26310) UID: 0 2024-09-13T11:23:50.134 INFO:teuthology.orchestra.run.smithi001.stderr:noscrub is set 2024-09-13T11:23:50.134 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-09-13T11:23:50.134 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:23:50.138 INFO:teuthology.orchestra.run.smithi001.stderr:reweighted osd.9 to 1 (10000) 2024-09-13T11:23:50.157 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-13T11:23:50.169 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 10 1 2024-09-13T11:23:50.188 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-09-13T11:23:50.185+0000 7f97befc9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26315) UID: 0 2024-09-13T11:23:50.289 INFO:tasks.ceph.osd.9.smithi092.stderr:2024-09-13T11:23:50.283+0000 7ff5ab3d0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26348) UID: 0 2024-09-13T11:23:50.317 INFO:teuthology.orchestra.run.smithi092.stdout:ERROR: (22) Invalid argument 2024-09-13T11:23:50.317 INFO:teuthology.orchestra.run.smithi092.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:23:50.339 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-13T11:23:50.425 INFO:tasks.ceph.osd.13.smithi092.stderr:2024-09-13T11:23:50.423+0000 7f3e5afea640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26350) UID: 0 2024-09-13T11:23:50.490 INFO:tasks.ceph.osd.12.smithi001.stderr:2024-09-13T11:23:50.484+0000 7f892c6e6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26314) UID: 0 2024-09-13T11:23:50.522 INFO:teuthology.orchestra.run.smithi092.stdout:ERROR: (22) Invalid argument 2024-09-13T11:23:50.522 INFO:teuthology.orchestra.run.smithi092.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:23:50.539 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-13T11:23:50.592 INFO:tasks.ceph.osd.11.smithi177.stderr:2024-09-13T11:23:50.584+0000 7f24792c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26580) UID: 0 2024-09-13T11:23:50.693 INFO:tasks.ceph.osd.2.smithi162.stderr:2024-09-13T11:23:50.689+0000 7fe92b35a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26313) UID: 0 2024-09-13T11:23:50.732 INFO:teuthology.orchestra.run.smithi092.stdout:ERROR: (22) Invalid argument 2024-09-13T11:23:50.732 INFO:teuthology.orchestra.run.smithi092.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:23:50.753 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-13T11:23:50.800 INFO:tasks.ceph.osd.8.smithi001.stderr:2024-09-13T11:23:50.796+0000 7fb9b48ae640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26312) UID: 0 2024-09-13T11:23:50.892 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-13T11:23:50.893 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:23:50.896 INFO:tasks.ceph.osd.11.smithi177.stderr:2024-09-13T11:23:50.888+0000 7f24792c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26580) UID: 0 2024-09-13T11:23:50.908 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-13T11:23:51.000 INFO:tasks.ceph.osd.8.smithi001.stderr:2024-09-13T11:23:50.996+0000 7fb9b48ae640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26312) UID: 0 2024-09-13T11:23:51.006 INFO:tasks.radosbench.radosbench.0.smithi177.stdout: 84 16 68169 68153 50.7037 85.875 0.0296036 0.0197161 2024-09-13T11:23:51.097 INFO:tasks.ceph.osd.3.smithi177.stderr:2024-09-13T11:23:51.092+0000 7f10adba4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26574) UID: 0 2024-09-13T11:23:51.099 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-13T11:23:51.099 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:23:51.116 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-13T11:23:51.143 INFO:teuthology.orchestra.run.smithi001.stderr:reweighted osd.10 to 1 (10000) 2024-09-13T11:23:51.148 INFO:tasks.ceph.osd.2.smithi162.stderr:2024-09-13T11:23:51.145+0000 7fe91fd87640 -1 osd.2 93 handle_osd_map injecting map crc failure 2024-09-13T11:23:51.150 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-09-13T11:23:51.149+0000 7f97b59fa640 -1 osd.6 93 handle_osd_map injecting map crc failure 2024-09-13T11:23:51.166 INFO:tasks.ceph.osd.4.smithi001.stderr:2024-09-13T11:23:51.160+0000 7ff8d2c41640 -1 osd.4 93 handle_osd_map injecting map crc failure 2024-09-13T11:23:51.175 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 11 1 2024-09-13T11:23:51.198 INFO:tasks.ceph.osd.15.smithi177.stderr:2024-09-13T11:23:51.192+0000 7f9d6ce77640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26578) UID: 0 2024-09-13T11:23:51.290 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-13T11:23:51.290 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:23:51.299 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-13T11:23:51.292+0000 7f6a68775640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26307) UID: 0 2024-09-13T11:23:51.312 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-13T11:23:51.404 INFO:tasks.ceph.osd.12.smithi001.stderr:2024-09-13T11:23:51.400+0000 7f892c6e6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26314) UID: 0 2024-09-13T11:23:51.501 INFO:tasks.ceph.osd.11.smithi177.stderr:2024-09-13T11:23:51.496+0000 7f24792c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26580) UID: 0 2024-09-13T11:23:51.503 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-13T11:23:51.503 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:23:51.521 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-13T11:23:51.608 INFO:tasks.ceph.osd.12.smithi001.stderr:2024-09-13T11:23:51.604+0000 7f892c6e6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26314) UID: 0 2024-09-13T11:23:51.662 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T11:23:51.688 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-13T11:23:51.689 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:23:51.702 INFO:tasks.ceph.osd.2.smithi162.stderr:2024-09-13T11:23:51.701+0000 7fe92b35a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26313) UID: 0 2024-09-13T11:23:51.706 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-13T11:23:51.803 INFO:tasks.ceph.osd.11.smithi177.stderr:2024-09-13T11:23:51.796+0000 7f24792c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26580) UID: 0 2024-09-13T11:23:51.828 INFO:teuthology.orchestra.run.smithi001.stdout:osd.0: {} 2024-09-13T11:23:51.828 INFO:teuthology.orchestra.run.smithi001.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T11:23:51.860 INFO:teuthology.orchestra.run.smithi001.stdout:osd.1: {} 2024-09-13T11:23:51.860 INFO:teuthology.orchestra.run.smithi001.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T11:23:51.895 INFO:teuthology.orchestra.run.smithi001.stdout:osd.2: {} 2024-09-13T11:23:51.895 INFO:teuthology.orchestra.run.smithi001.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T11:23:51.899 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-13T11:23:51.899 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:23:51.903 INFO:tasks.ceph.osd.1.smithi092.stderr:2024-09-13T11:23:51.899+0000 7fe7789c5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33718) UID: 0 2024-09-13T11:23:51.918 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-09-13T11:23:51.921 INFO:teuthology.orchestra.run.smithi001.stdout:osd.3: {} 2024-09-13T11:23:51.922 INFO:teuthology.orchestra.run.smithi001.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T11:23:51.949 INFO:teuthology.orchestra.run.smithi001.stdout:osd.4: {} 2024-09-13T11:23:51.949 INFO:teuthology.orchestra.run.smithi001.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T11:23:51.980 INFO:teuthology.orchestra.run.smithi001.stdout:osd.5: {} 2024-09-13T11:23:51.980 INFO:teuthology.orchestra.run.smithi001.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T11:23:52.004 INFO:tasks.ceph.osd.11.smithi177.stderr:2024-09-13T11:23:52.000+0000 7f24792c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26580) UID: 0 2024-09-13T11:23:52.006 INFO:tasks.radosbench.radosbench.0.smithi177.stdout: 85 15 69432 69417 51.0366 79 0.00849652 0.019587 2024-09-13T11:23:52.010 INFO:teuthology.orchestra.run.smithi001.stdout:osd.6: {} 2024-09-13T11:23:52.010 INFO:teuthology.orchestra.run.smithi001.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T11:23:52.038 INFO:teuthology.orchestra.run.smithi001.stdout:osd.7: {} 2024-09-13T11:23:52.038 INFO:teuthology.orchestra.run.smithi001.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T11:23:52.064 INFO:teuthology.orchestra.run.smithi001.stdout:osd.8: {} 2024-09-13T11:23:52.064 INFO:teuthology.orchestra.run.smithi001.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-13T11:23:52.065 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-13T11:23:52.065 INFO:teuthology.orchestra.run.smithi001.stdout: "ops": [], 2024-09-13T11:23:52.065 INFO:teuthology.orchestra.run.smithi001.stdout: "num_ops": 0 2024-09-13T11:23:52.065 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-13T11:23:52.084 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-09-13T11:23:52.096 INFO:teuthology.orchestra.run.smithi001.stdout:osd.9: {} 2024-09-13T11:23:52.097 INFO:teuthology.orchestra.run.smithi001.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-13T11:23:52.105 INFO:tasks.ceph.osd.2.smithi162.stderr:2024-09-13T11:23:52.101+0000 7fe92b35a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26313) UID: 0 2024-09-13T11:23:52.120 INFO:teuthology.orchestra.run.smithi001.stdout:osd.10: {} 2024-09-13T11:23:52.121 INFO:teuthology.orchestra.run.smithi001.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-13T11:23:52.143 INFO:teuthology.orchestra.run.smithi001.stdout:osd.11: {} 2024-09-13T11:23:52.144 INFO:teuthology.orchestra.run.smithi001.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-13T11:23:52.160 INFO:teuthology.orchestra.run.smithi001.stderr:reweighted osd.11 to 1 (10000) 2024-09-13T11:23:52.167 INFO:teuthology.orchestra.run.smithi001.stdout:osd.12: {} 2024-09-13T11:23:52.167 INFO:teuthology.orchestra.run.smithi001.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-13T11:23:52.177 INFO:tasks.ceph.osd.12.smithi001.stderr:2024-09-13T11:23:52.172+0000 7f891d10b640 -1 osd.12 94 handle_osd_map injecting map crc failure 2024-09-13T11:23:52.184 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-13T11:23:52.188 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 12 1 2024-09-13T11:23:52.202 INFO:teuthology.orchestra.run.smithi001.stdout:osd.13: {} 2024-09-13T11:23:52.203 INFO:teuthology.orchestra.run.smithi001.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-13T11:23:52.206 INFO:tasks.ceph.osd.15.smithi177.stderr:2024-09-13T11:23:52.200+0000 7f9d6ce77640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26578) UID: 0 2024-09-13T11:23:52.233 INFO:teuthology.orchestra.run.smithi001.stdout:osd.14: {} 2024-09-13T11:23:52.234 INFO:teuthology.orchestra.run.smithi001.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-13T11:23:52.237 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-13T11:23:52.237 INFO:teuthology.orchestra.run.smithi001.stdout: "ops": [], 2024-09-13T11:23:52.237 INFO:teuthology.orchestra.run.smithi001.stdout: "complaint_time": 30, 2024-09-13T11:23:52.237 INFO:teuthology.orchestra.run.smithi001.stdout: "num_blocked_ops": 0 2024-09-13T11:23:52.237 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-13T11:23:52.257 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-09-13T11:23:52.271 INFO:teuthology.orchestra.run.smithi001.stdout:osd.15: {} 2024-09-13T11:23:52.271 INFO:teuthology.orchestra.run.smithi001.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-13T11:23:52.306 INFO:tasks.ceph.osd.7.smithi177.stderr:2024-09-13T11:23:52.300+0000 7f897b51d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26575) UID: 0 2024-09-13T11:23:52.408 INFO:tasks.ceph.osd.4.smithi001.stderr:2024-09-13T11:23:52.400+0000 7ff8e0218640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26310) UID: 0 2024-09-13T11:23:52.443 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-13T11:23:52.443 INFO:teuthology.orchestra.run.smithi001.stdout: "size": 20, 2024-09-13T11:23:52.443 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 600, 2024-09-13T11:23:52.444 INFO:teuthology.orchestra.run.smithi001.stdout: "ops": [ 2024-09-13T11:23:52.444 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:52.444 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:30779 3.e 3:77c07d05:::benchmark_data_smithi177_31116_object30778:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e59)", 2024-09-13T11:23:52.444 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:09.473906+0000", 2024-09-13T11:23:52.444 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 42.966568721999998, 2024-09-13T11:23:52.444 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.224949224, 2024-09-13T11:23:52.444 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:23:52.444 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:23:52.444 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:52.444 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:23:52.444 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:23:52.444 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:52.444 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 30779 2024-09-13T11:23:52.444 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:52.445 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:23:52.445 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:52.445 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:23:52.445 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.473906+0000", 2024-09-13T11:23:52.445 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:52.445 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:52.445 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:52.445 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:23:52.445 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.473906+0000", 2024-09-13T11:23:52.445 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:52.445 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:52.445 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:52.445 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:23:52.446 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.473908+0000", 2024-09-13T11:23:52.446 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.6230000000000001e-06 2024-09-13T11:23:52.446 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:52.446 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:52.446 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:23:52.446 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.473961+0000", 2024-09-13T11:23:52.446 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.2374999999999999e-05 2024-09-13T11:23:52.446 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:52.446 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:52.446 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:23:52.446 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.473976+0000", 2024-09-13T11:23:52.446 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.56e-05 2024-09-13T11:23:52.446 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:52.446 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:52.447 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:23:52.447 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.473987+0000", 2024-09-13T11:23:52.447 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0394000000000001e-05 2024-09-13T11:23:52.447 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:52.447 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:52.447 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:52.447 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.474275+0000", 2024-09-13T11:23:52.447 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00028800800000000002 2024-09-13T11:23:52.447 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:52.447 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:52.447 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:23:52.447 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.474466+0000", 2024-09-13T11:23:52.447 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000191499 2024-09-13T11:23:52.447 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:52.448 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:52.448 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 0,1", 2024-09-13T11:23:52.448 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.474666+0000", 2024-09-13T11:23:52.448 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00019994899999999999 2024-09-13T11:23:52.448 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:52.448 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:52.448 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:52.448 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.478354+0000", 2024-09-13T11:23:52.448 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0036876779999999998 2024-09-13T11:23:52.448 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:52.448 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:52.448 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:23:52.448 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.483128+0000", 2024-09-13T11:23:52.449 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0047739410000000003 2024-09-13T11:23:52.449 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:52.449 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:52.449 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:52.449 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.697695+0000", 2024-09-13T11:23:52.449 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.214566955 2024-09-13T11:23:52.449 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:52.449 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:52.449 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:23:52.449 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.697786+0000", 2024-09-13T11:23:52.449 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.1310000000000005e-05 2024-09-13T11:23:52.449 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:52.449 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:52.449 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:23:52.450 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.698855+0000", 2024-09-13T11:23:52.450 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0010688920000000001 2024-09-13T11:23:52.450 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:52.450 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:52.450 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:52.450 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:52.450 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:52.450 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:30801 3.e 3:73b0f21a:::benchmark_data_smithi177_31116_object30800:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e59)", 2024-09-13T11:23:52.450 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:09.485629+0000", 2024-09-13T11:23:52.450 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 42.954845919999997, 2024-09-13T11:23:52.450 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.22604919900000001, 2024-09-13T11:23:52.450 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:23:52.450 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:23:52.450 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:52.451 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:23:52.451 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:23:52.451 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:52.451 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 30801 2024-09-13T11:23:52.451 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:52.451 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:23:52.451 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:52.451 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:23:52.451 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.485629+0000", 2024-09-13T11:23:52.451 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:52.451 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:52.451 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:52.451 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:23:52.451 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.485629+0000", 2024-09-13T11:23:52.452 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:52.452 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:52.452 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:52.452 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:23:52.452 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.485631+0000", 2024-09-13T11:23:52.452 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.9989999999999998e-06 2024-09-13T11:23:52.452 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:52.452 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:52.452 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:23:52.452 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.485664+0000", 2024-09-13T11:23:52.452 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.3896000000000002e-05 2024-09-13T11:23:52.452 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:52.452 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:52.452 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:23:52.453 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.485679+0000", 2024-09-13T11:23:52.453 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.4438e-05 2024-09-13T11:23:52.453 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:52.453 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:52.453 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:23:52.453 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.485688+0000", 2024-09-13T11:23:52.453 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.0180000000000007e-06 2024-09-13T11:23:52.453 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:52.453 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:52.453 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:52.453 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.485742+0000", 2024-09-13T11:23:52.453 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.4026000000000001e-05 2024-09-13T11:23:52.453 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:52.453 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:52.454 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:23:52.454 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.485841+0000", 2024-09-13T11:23:52.454 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.8875000000000005e-05 2024-09-13T11:23:52.454 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:52.454 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:52.454 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 0,1", 2024-09-13T11:23:52.454 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.486005+0000", 2024-09-13T11:23:52.454 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000163747 2024-09-13T11:23:52.454 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:52.454 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:52.454 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:23:52.454 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.487867+0000", 2024-09-13T11:23:52.454 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.001862945 2024-09-13T11:23:52.454 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:52.454 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:52.455 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:52.455 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.488570+0000", 2024-09-13T11:23:52.455 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00070255700000000003 2024-09-13T11:23:52.455 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:52.455 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:52.455 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:52.455 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.711140+0000", 2024-09-13T11:23:52.455 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.222570348 2024-09-13T11:23:52.455 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:52.455 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:52.455 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:23:52.455 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.711180+0000", 2024-09-13T11:23:52.455 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.9273999999999997e-05 2024-09-13T11:23:52.455 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:52.456 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:52.456 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:23:52.456 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.711678+0000", 2024-09-13T11:23:52.456 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00049807599999999997 2024-09-13T11:23:52.456 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:52.456 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:52.456 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:52.456 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:52.456 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:52.456 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:54656 3.1e 3.6a49163e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e75)", 2024-09-13T11:23:52.456 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:30.979993+0000", 2024-09-13T11:23:52.456 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 21.460481487999999, 2024-09-13T11:23:52.456 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.9269557150000001, 2024-09-13T11:23:52.456 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:23:52.457 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:23:52.457 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-13T11:23:52.457 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:23:52.457 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:23:52.457 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:52.457 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 54656 2024-09-13T11:23:52.457 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:52.457 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:23:52.457 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:52.457 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:23:52.457 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:30.979993+0000", 2024-09-13T11:23:52.457 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:52.457 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:52.457 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:52.458 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:23:52.458 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:30.979993+0000", 2024-09-13T11:23:52.458 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:52.458 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:52.458 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:52.458 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:23:52.458 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:30.979997+0000", 2024-09-13T11:23:52.458 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.0319999999999997e-06 2024-09-13T11:23:52.458 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:52.458 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:52.458 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:23:52.458 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:30.980057+0000", 2024-09-13T11:23:52.458 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.0341999999999999e-05 2024-09-13T11:23:52.458 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:52.459 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:52.459 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:23:52.459 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:30.980094+0000", 2024-09-13T11:23:52.459 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.6770000000000002e-05 2024-09-13T11:23:52.459 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:52.459 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:52.459 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:23:52.459 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:30.980113+0000", 2024-09-13T11:23:52.459 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.8791999999999999e-05 2024-09-13T11:23:52.459 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:52.459 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:52.459 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:52.459 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.097773+0000", 2024-09-13T11:23:52.459 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.117660379 2024-09-13T11:23:52.460 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:52.460 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:52.460 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for peered", 2024-09-13T11:23:52.460 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.097784+0000", 2024-09-13T11:23:52.460 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.1025e-05 2024-09-13T11:23:52.460 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:52.460 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:52.460 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:52.460 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.906937+0000", 2024-09-13T11:23:52.460 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.8091523789999999 2024-09-13T11:23:52.460 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:52.460 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:52.460 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:23:52.460 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.906949+0000", 2024-09-13T11:23:52.461 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.1996e-05 2024-09-13T11:23:52.461 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:52.461 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:52.461 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:52.461 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:52.461 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:52.461 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDPGPushReply(3.17 75/70 [PushReplyOp(3:e82800c8:::benchmark_data_smithi177_31116_object13909:head)])", 2024-09-13T11:23:52.461 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:31.632344+0000", 2024-09-13T11:23:52.461 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 20.808130633000001, 2024-09-13T11:23:52.461 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.28592102899999999, 2024-09-13T11:23:52.461 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:23:52.461 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:23:52.461 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-13T11:23:52.462 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:23:52.462 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:52.462 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:23:52.462 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.632344+0000", 2024-09-13T11:23:52.462 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:52.462 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:52.462 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:52.462 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:23:52.462 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.632344+0000", 2024-09-13T11:23:52.462 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:52.462 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:52.462 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:52.462 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:23:52.462 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.632346+0000", 2024-09-13T11:23:52.463 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.8819999999999999e-06 2024-09-13T11:23:52.463 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:52.463 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:52.463 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:23:52.463 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.632353+0000", 2024-09-13T11:23:52.463 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.5059999999999998e-06 2024-09-13T11:23:52.463 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:52.463 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:52.463 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:23:52.463 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.917361+0000", 2024-09-13T11:23:52.463 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.285008231 2024-09-13T11:23:52.463 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:52.463 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:52.463 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:23:52.464 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.917372+0000", 2024-09-13T11:23:52.464 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0764999999999999e-05 2024-09-13T11:23:52.464 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:52.464 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:52.464 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:52.464 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.917458+0000", 2024-09-13T11:23:52.464 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.6136000000000003e-05 2024-09-13T11:23:52.464 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:52.464 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:52.464 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:23:52.464 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.918265+0000", 2024-09-13T11:23:52.464 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00080650899999999996 2024-09-13T11:23:52.464 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:52.464 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:52.465 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:52.465 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:52.465 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:52.465 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:54720 3.1e 3.a944303e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-13T11:23:52.465 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:31.973315+0000", 2024-09-13T11:23:52.465 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 20.467159252999998, 2024-09-13T11:23:52.465 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.93366527200000005, 2024-09-13T11:23:52.465 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:23:52.465 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:23:52.465 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-13T11:23:52.465 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:23:52.465 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:23:52.465 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:52.465 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 54720 2024-09-13T11:23:52.466 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:52.466 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:23:52.466 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:52.466 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:23:52.466 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.973315+0000", 2024-09-13T11:23:52.466 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:52.466 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:52.466 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:52.466 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:23:52.466 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.973315+0000", 2024-09-13T11:23:52.466 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:52.466 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:52.466 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:52.467 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:23:52.467 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.973318+0000", 2024-09-13T11:23:52.467 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.5830000000000001e-06 2024-09-13T11:23:52.467 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:52.467 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:52.467 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:23:52.467 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.973465+0000", 2024-09-13T11:23:52.467 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000146816 2024-09-13T11:23:52.467 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:52.467 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:52.467 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:23:52.467 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.973478+0000", 2024-09-13T11:23:52.467 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2931e-05 2024-09-13T11:23:52.467 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:52.468 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:52.468 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:23:52.468 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.973483+0000", 2024-09-13T11:23:52.468 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.8969999999999996e-06 2024-09-13T11:23:52.468 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:52.468 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:52.468 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:52.468 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.973548+0000", 2024-09-13T11:23:52.468 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.4771999999999996e-05 2024-09-13T11:23:52.468 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:52.468 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:52.469 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for peered", 2024-09-13T11:23:52.469 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.973557+0000", 2024-09-13T11:23:52.469 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.0359999999999995e-06 2024-09-13T11:23:52.469 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:52.469 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:52.469 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:52.469 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.906974+0000", 2024-09-13T11:23:52.469 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.93341706000000002 2024-09-13T11:23:52.469 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:52.469 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:52.469 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:23:52.469 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.906980+0000", 2024-09-13T11:23:52.469 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.1770000000000003e-06 2024-09-13T11:23:52.470 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:52.470 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:52.470 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:52.470 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:52.470 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:52.470 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:54734 3.1e 3.d7f6f7e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-13T11:23:52.470 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:31.983342+0000", 2024-09-13T11:23:52.470 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 20.45713245, 2024-09-13T11:23:52.470 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.92367058599999996, 2024-09-13T11:23:52.470 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:23:52.470 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:23:52.470 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-13T11:23:52.470 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:23:52.470 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:23:52.471 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:52.471 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 54734 2024-09-13T11:23:52.471 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:52.471 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:23:52.471 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:52.471 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:23:52.471 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.983342+0000", 2024-09-13T11:23:52.471 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:52.471 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:52.471 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:52.471 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:23:52.471 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.983342+0000", 2024-09-13T11:23:52.471 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:52.471 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:52.471 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:52.472 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:23:52.472 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.983345+0000", 2024-09-13T11:23:52.472 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.6220000000000001e-06 2024-09-13T11:23:52.472 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:52.472 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:52.472 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:23:52.472 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.983438+0000", 2024-09-13T11:23:52.472 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.3517000000000006e-05 2024-09-13T11:23:52.472 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:52.472 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:52.472 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:23:52.472 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.983451+0000", 2024-09-13T11:23:52.472 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2527999999999999e-05 2024-09-13T11:23:52.472 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:52.473 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:52.473 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:23:52.473 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.983458+0000", 2024-09-13T11:23:52.473 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.6380000000000004e-06 2024-09-13T11:23:52.473 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:52.473 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:52.473 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:52.473 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.983521+0000", 2024-09-13T11:23:52.473 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.2729000000000002e-05 2024-09-13T11:23:52.473 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:52.473 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:52.473 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for peered", 2024-09-13T11:23:52.473 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.983548+0000", 2024-09-13T11:23:52.473 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.6537999999999999e-05 2024-09-13T11:23:52.474 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:52.474 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:52.474 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:52.474 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.907005+0000", 2024-09-13T11:23:52.474 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.923456991 2024-09-13T11:23:52.474 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:52.474 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:52.474 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:23:52.474 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.907013+0000", 2024-09-13T11:23:52.474 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.0229999999999992e-06 2024-09-13T11:23:52.474 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:52.474 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:52.474 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:52.474 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:52.475 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:52.475 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:54760 3.1e 3.8c32c13e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-13T11:23:52.475 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:31.999685+0000", 2024-09-13T11:23:52.475 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 20.440789217999999, 2024-09-13T11:23:52.475 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.90735639599999995, 2024-09-13T11:23:52.475 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:23:52.475 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:23:52.475 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-13T11:23:52.475 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:23:52.475 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:23:52.475 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:52.475 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 54760 2024-09-13T11:23:52.475 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:52.475 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:23:52.476 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:52.476 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:23:52.476 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.999685+0000", 2024-09-13T11:23:52.476 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:52.476 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:52.476 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:52.476 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:23:52.476 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.999685+0000", 2024-09-13T11:23:52.476 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:52.476 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:52.476 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:52.476 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:23:52.476 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.999690+0000", 2024-09-13T11:23:52.476 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.7879999999999997e-06 2024-09-13T11:23:52.477 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:52.477 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:52.477 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:23:52.477 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.999729+0000", 2024-09-13T11:23:52.477 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.9032000000000002e-05 2024-09-13T11:23:52.478 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:52.478 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:52.478 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:23:52.478 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.999741+0000", 2024-09-13T11:23:52.478 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2136999999999999e-05 2024-09-13T11:23:52.478 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:52.478 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:52.478 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:23:52.478 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.999748+0000", 2024-09-13T11:23:52.479 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.9120000000000001e-06 2024-09-13T11:23:52.479 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:52.479 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:52.479 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:52.479 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.999803+0000", 2024-09-13T11:23:52.479 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.5064e-05 2024-09-13T11:23:52.479 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:52.479 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:52.479 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for peered", 2024-09-13T11:23:52.479 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.999815+0000", 2024-09-13T11:23:52.479 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2041e-05 2024-09-13T11:23:52.479 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:52.479 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:52.479 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:52.480 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.907034+0000", 2024-09-13T11:23:52.480 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.90721858200000005 2024-09-13T11:23:52.480 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:52.480 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:52.480 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:23:52.480 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.907042+0000", 2024-09-13T11:23:52.480 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.8399999999999995e-06 2024-09-13T11:23:52.480 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:52.480 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:52.480 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:52.480 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:52.480 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:52.480 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:54768 3.1e 3.6dad499e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-13T11:23:52.481 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:32.003482+0000", 2024-09-13T11:23:52.481 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 20.436992568000001, 2024-09-13T11:23:52.481 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.90358804699999995, 2024-09-13T11:23:52.481 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:23:52.481 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:23:52.481 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-13T11:23:52.481 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:23:52.481 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:23:52.481 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:52.481 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 54768 2024-09-13T11:23:52.481 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:52.481 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:23:52.481 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:52.481 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:23:52.482 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.003482+0000", 2024-09-13T11:23:52.482 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:52.482 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:52.482 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:52.482 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:23:52.482 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.003482+0000", 2024-09-13T11:23:52.482 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:52.482 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:52.482 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:52.482 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:23:52.482 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.003486+0000", 2024-09-13T11:23:52.482 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.1030000000000004e-06 2024-09-13T11:23:52.482 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:52.482 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:52.483 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:23:52.483 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.003523+0000", 2024-09-13T11:23:52.483 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.6732e-05 2024-09-13T11:23:52.483 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:52.483 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:52.483 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:23:52.483 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.003535+0000", 2024-09-13T11:23:52.483 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2520999999999999e-05 2024-09-13T11:23:52.483 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:52.483 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:52.483 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:23:52.483 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.003541+0000", 2024-09-13T11:23:52.483 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.8690000000000001e-06 2024-09-13T11:23:52.484 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:52.484 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:52.484 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:52.484 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.003593+0000", 2024-09-13T11:23:52.484 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.1786000000000002e-05 2024-09-13T11:23:52.484 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:52.484 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:52.484 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for peered", 2024-09-13T11:23:52.484 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.003602+0000", 2024-09-13T11:23:52.484 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.9679999999999995e-06 2024-09-13T11:23:52.484 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:52.484 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:52.484 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:52.484 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.907063+0000", 2024-09-13T11:23:52.484 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.90346116600000004 2024-09-13T11:23:52.485 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:52.485 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:52.485 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:23:52.485 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.907070+0000", 2024-09-13T11:23:52.485 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.9020000000000002e-06 2024-09-13T11:23:52.485 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:52.485 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:52.485 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:52.485 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:52.485 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:52.485 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:54785 3.1e 3.d32f1ede (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-13T11:23:52.485 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:32.016238+0000", 2024-09-13T11:23:52.485 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 20.424236358000002, 2024-09-13T11:23:52.485 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.89086243399999998, 2024-09-13T11:23:52.486 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:23:52.486 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:23:52.486 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-13T11:23:52.486 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:23:52.486 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:23:52.486 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:52.486 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 54785 2024-09-13T11:23:52.486 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:52.486 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:23:52.486 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:52.486 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:23:52.486 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.016238+0000", 2024-09-13T11:23:52.486 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:52.487 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:52.487 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:52.487 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:23:52.487 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.016238+0000", 2024-09-13T11:23:52.487 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:52.487 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:52.487 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:52.487 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:23:52.487 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.016241+0000", 2024-09-13T11:23:52.487 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.6469999999999998e-06 2024-09-13T11:23:52.487 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:52.487 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:52.487 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:23:52.488 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.016271+0000", 2024-09-13T11:23:52.488 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.0505999999999999e-05 2024-09-13T11:23:52.488 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:52.488 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:52.488 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:23:52.488 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.016287+0000", 2024-09-13T11:23:52.488 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.6073999999999999e-05 2024-09-13T11:23:52.488 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:52.488 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:52.488 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:23:52.488 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.016295+0000", 2024-09-13T11:23:52.488 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.0590000000000003e-06 2024-09-13T11:23:52.488 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:52.488 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:52.489 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:52.489 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.016373+0000", 2024-09-13T11:23:52.489 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.7634999999999998e-05 2024-09-13T11:23:52.489 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:52.489 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:52.489 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for peered", 2024-09-13T11:23:52.489 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.016383+0000", 2024-09-13T11:23:52.489 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0246e-05 2024-09-13T11:23:52.489 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:52.489 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:52.489 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:52.489 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.907092+0000", 2024-09-13T11:23:52.489 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.89070849200000002 2024-09-13T11:23:52.489 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:52.490 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:52.490 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:23:52.490 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.907101+0000", 2024-09-13T11:23:52.490 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.7749999999999998e-06 2024-09-13T11:23:52.490 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:52.490 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:52.490 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:52.490 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:52.490 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:52.490 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:54811 3.1e 3.3754367e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-13T11:23:52.490 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:32.045424+0000", 2024-09-13T11:23:52.490 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 20.395050706999999, 2024-09-13T11:23:52.490 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.86170719399999995, 2024-09-13T11:23:52.491 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:23:52.491 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:23:52.491 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-13T11:23:52.491 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:23:52.491 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:23:52.491 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:52.491 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 54811 2024-09-13T11:23:52.491 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:52.491 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:23:52.491 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:52.491 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:23:52.491 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.045424+0000", 2024-09-13T11:23:52.491 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:52.491 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:52.492 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:52.492 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:23:52.492 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.045424+0000", 2024-09-13T11:23:52.492 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:52.492 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:52.492 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:52.492 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:23:52.492 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.045426+0000", 2024-09-13T11:23:52.492 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.0260000000000002e-06 2024-09-13T11:23:52.492 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:52.492 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:52.492 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:23:52.492 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.045556+0000", 2024-09-13T11:23:52.493 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00013003099999999999 2024-09-13T11:23:52.493 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:52.493 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:52.493 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:23:52.493 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.045568+0000", 2024-09-13T11:23:52.493 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.1727e-05 2024-09-13T11:23:52.493 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:52.493 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:52.493 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:23:52.493 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.045576+0000", 2024-09-13T11:23:52.493 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.6459999999999999e-06 2024-09-13T11:23:52.493 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:52.493 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:52.494 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:52.494 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.045650+0000", 2024-09-13T11:23:52.494 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.4000999999999998e-05 2024-09-13T11:23:52.494 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:52.494 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:52.494 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for peered", 2024-09-13T11:23:52.494 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.045658+0000", 2024-09-13T11:23:52.494 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.3869999999999997e-06 2024-09-13T11:23:52.494 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:52.494 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:52.494 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:52.494 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.907123+0000", 2024-09-13T11:23:52.494 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.86146508399999999 2024-09-13T11:23:52.494 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:52.495 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:52.495 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:23:52.495 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.907131+0000", 2024-09-13T11:23:52.495 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.2919999999999996e-06 2024-09-13T11:23:52.495 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:52.495 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:52.495 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:52.495 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:52.495 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:52.495 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:54817 3.1e 3.36dd2fde (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-13T11:23:52.495 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:32.050108+0000", 2024-09-13T11:23:52.495 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 20.390366815, 2024-09-13T11:23:52.495 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.85705549199999997, 2024-09-13T11:23:52.495 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:23:52.496 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:23:52.496 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-13T11:23:52.496 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:23:52.496 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:23:52.496 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:52.496 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 54817 2024-09-13T11:23:52.496 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:52.496 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:23:52.496 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:52.496 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:23:52.496 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.050108+0000", 2024-09-13T11:23:52.496 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:52.496 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:52.497 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:52.497 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:23:52.497 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.050108+0000", 2024-09-13T11:23:52.497 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:52.497 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:52.497 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:52.497 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:23:52.497 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.050110+0000", 2024-09-13T11:23:52.497 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.0209999999999998e-06 2024-09-13T11:23:52.497 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:52.497 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:52.497 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:23:52.497 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.050166+0000", 2024-09-13T11:23:52.498 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.6026000000000002e-05 2024-09-13T11:23:52.498 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:52.498 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:52.498 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:23:52.498 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.050180+0000", 2024-09-13T11:23:52.498 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.4307e-05 2024-09-13T11:23:52.498 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:52.498 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:52.498 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:23:52.498 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.050188+0000", 2024-09-13T11:23:52.498 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.5410000000000003e-06 2024-09-13T11:23:52.498 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:52.498 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:52.498 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:52.499 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.050260+0000", 2024-09-13T11:23:52.499 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.2157999999999996e-05 2024-09-13T11:23:52.499 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:52.499 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:52.499 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for peered", 2024-09-13T11:23:52.499 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.050267+0000", 2024-09-13T11:23:52.499 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.0190000000000001e-06 2024-09-13T11:23:52.499 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:52.499 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:52.499 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:52.499 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.907155+0000", 2024-09-13T11:23:52.499 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.85688824399999997 2024-09-13T11:23:52.499 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:52.499 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:52.500 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:23:52.500 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.907163+0000", 2024-09-13T11:23:52.500 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.1759999999999993e-06 2024-09-13T11:23:52.500 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:52.500 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:52.500 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:52.500 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:52.500 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:52.500 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:54818 3.1e 3.5cfcc69e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-13T11:23:52.500 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:32.051374+0000", 2024-09-13T11:23:52.500 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 20.389100399, 2024-09-13T11:23:52.500 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.85581781000000001, 2024-09-13T11:23:52.500 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:23:52.500 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:23:52.501 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-13T11:23:52.501 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:23:52.501 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:23:52.501 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:52.501 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 54818 2024-09-13T11:23:52.501 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:52.501 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:23:52.501 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:52.501 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:23:52.501 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.051374+0000", 2024-09-13T11:23:52.501 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:52.501 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:52.501 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:52.501 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:23:52.502 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.051374+0000", 2024-09-13T11:23:52.502 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:52.502 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:52.502 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:52.502 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:23:52.502 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.051375+0000", 2024-09-13T11:23:52.502 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.4440000000000001e-06 2024-09-13T11:23:52.502 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:52.502 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:52.502 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:23:52.502 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.051426+0000", 2024-09-13T11:23:52.502 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.0809000000000001e-05 2024-09-13T11:23:52.502 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:52.502 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:52.503 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:23:52.503 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.051441+0000", 2024-09-13T11:23:52.503 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.4460999999999999e-05 2024-09-13T11:23:52.503 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:52.503 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:52.503 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:23:52.503 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.051449+0000", 2024-09-13T11:23:52.503 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.4880000000000007e-06 2024-09-13T11:23:52.503 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:52.503 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:52.503 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:52.503 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.051499+0000", 2024-09-13T11:23:52.503 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.9691999999999998e-05 2024-09-13T11:23:52.503 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:52.504 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:52.504 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for peered", 2024-09-13T11:23:52.504 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.051505+0000", 2024-09-13T11:23:52.504 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.2029999999999997e-06 2024-09-13T11:23:52.504 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:52.504 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:52.504 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:52.504 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.907184+0000", 2024-09-13T11:23:52.504 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.85567906599999999 2024-09-13T11:23:52.504 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:52.504 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:52.504 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:23:52.504 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.907192+0000", 2024-09-13T11:23:52.504 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.6469999999999998e-06 2024-09-13T11:23:52.505 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:52.505 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:52.505 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:52.505 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:52.505 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:52.505 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:54828 3.1e 3.e920c11e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-13T11:23:52.505 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:32.063344+0000", 2024-09-13T11:23:52.505 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 20.377129971999999, 2024-09-13T11:23:52.505 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.84387899099999997, 2024-09-13T11:23:52.505 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:23:52.505 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:23:52.505 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-13T11:23:52.505 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:23:52.505 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:23:52.506 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:52.506 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 54828 2024-09-13T11:23:52.506 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:52.506 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:23:52.506 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:52.506 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:23:52.506 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.063344+0000", 2024-09-13T11:23:52.506 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:52.506 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:52.506 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:52.506 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:23:52.506 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.063344+0000", 2024-09-13T11:23:52.506 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:52.506 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:52.507 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:52.507 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:23:52.507 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.063347+0000", 2024-09-13T11:23:52.507 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.2299999999999998e-06 2024-09-13T11:23:52.507 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:52.507 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:52.507 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:23:52.507 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.063384+0000", 2024-09-13T11:23:52.507 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.7290999999999999e-05 2024-09-13T11:23:52.507 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:52.507 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:52.507 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:23:52.507 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.063397+0000", 2024-09-13T11:23:52.507 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2955999999999999e-05 2024-09-13T11:23:52.508 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:52.508 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:52.508 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:23:52.508 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.063404+0000", 2024-09-13T11:23:52.508 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.4889999999999998e-06 2024-09-13T11:23:52.508 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:52.508 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:52.508 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:52.508 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.063470+0000", 2024-09-13T11:23:52.508 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.5516000000000001e-05 2024-09-13T11:23:52.508 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:52.508 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:52.508 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for peered", 2024-09-13T11:23:52.509 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.063482+0000", 2024-09-13T11:23:52.509 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2065000000000001e-05 2024-09-13T11:23:52.509 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:52.509 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:52.509 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:52.509 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.907216+0000", 2024-09-13T11:23:52.509 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.84373388900000001 2024-09-13T11:23:52.509 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:52.509 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:52.509 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:23:52.509 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.907223+0000", 2024-09-13T11:23:52.509 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.5549999999999997e-06 2024-09-13T11:23:52.509 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:52.509 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:52.510 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:52.510 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:52.510 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:52.510 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:54829 3.1e 3.192f62be (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-13T11:23:52.510 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:32.065642+0000", 2024-09-13T11:23:52.510 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 20.374832904000002, 2024-09-13T11:23:52.510 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.84161483100000001, 2024-09-13T11:23:52.510 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:23:52.510 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:23:52.510 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-13T11:23:52.510 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:23:52.510 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:23:52.510 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:52.510 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 54829 2024-09-13T11:23:52.511 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:52.511 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:23:52.511 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:52.511 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:23:52.511 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.065642+0000", 2024-09-13T11:23:52.511 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:52.511 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:52.511 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:52.511 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:23:52.511 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.065642+0000", 2024-09-13T11:23:52.511 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:52.511 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:52.511 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:52.511 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:23:52.512 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.065644+0000", 2024-09-13T11:23:52.512 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.1780000000000002e-06 2024-09-13T11:23:52.512 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:52.512 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:52.512 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:23:52.512 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.065695+0000", 2024-09-13T11:23:52.512 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.1632000000000003e-05 2024-09-13T11:23:52.512 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:52.512 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:52.512 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:23:52.512 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.065711+0000", 2024-09-13T11:23:52.512 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.5522000000000001e-05 2024-09-13T11:23:52.512 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:52.512 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:52.513 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:23:52.513 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.065719+0000", 2024-09-13T11:23:52.513 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.3480000000000004e-06 2024-09-13T11:23:52.513 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:52.513 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:52.513 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:52.513 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.065785+0000", 2024-09-13T11:23:52.513 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.5627e-05 2024-09-13T11:23:52.513 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:52.513 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:52.513 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for peered", 2024-09-13T11:23:52.513 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.065797+0000", 2024-09-13T11:23:52.513 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.1766000000000001e-05 2024-09-13T11:23:52.513 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:52.514 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:52.514 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:52.514 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.907248+0000", 2024-09-13T11:23:52.514 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.84145150300000005 2024-09-13T11:23:52.514 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:52.514 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:52.514 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:23:52.514 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.907256+0000", 2024-09-13T11:23:52.514 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.2549999999999997e-06 2024-09-13T11:23:52.514 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:52.514 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:52.514 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:52.514 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:52.514 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:52.514 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:54830 3.1e 3.b31554de (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-13T11:23:52.515 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:32.068259+0000", 2024-09-13T11:23:52.515 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 20.372215036, 2024-09-13T11:23:52.515 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.83902489599999996, 2024-09-13T11:23:52.515 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:23:52.515 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:23:52.515 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-13T11:23:52.515 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:23:52.515 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:23:52.515 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:52.515 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 54830 2024-09-13T11:23:52.515 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:52.515 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:23:52.515 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:52.515 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:23:52.516 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.068259+0000", 2024-09-13T11:23:52.516 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:52.516 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:52.516 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:52.516 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:23:52.516 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.068259+0000", 2024-09-13T11:23:52.516 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:52.516 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:52.516 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:52.516 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:23:52.516 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.068262+0000", 2024-09-13T11:23:52.516 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.1799999999999999e-06 2024-09-13T11:23:52.516 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:52.516 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:52.517 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:23:52.517 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.068301+0000", 2024-09-13T11:23:52.517 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.9289999999999998e-05 2024-09-13T11:23:52.517 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:52.517 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:52.517 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:23:52.517 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.068314+0000", 2024-09-13T11:23:52.517 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.3502999999999999e-05 2024-09-13T11:23:52.517 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:52.517 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:52.517 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:23:52.517 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.068324+0000", 2024-09-13T11:23:52.517 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.1630000000000002e-06 2024-09-13T11:23:52.518 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:52.518 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:52.518 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:52.518 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.068375+0000", 2024-09-13T11:23:52.518 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.1805000000000003e-05 2024-09-13T11:23:52.518 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:52.518 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:52.518 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for peered", 2024-09-13T11:23:52.518 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.068382+0000", 2024-09-13T11:23:52.518 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.224e-06 2024-09-13T11:23:52.518 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:52.518 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:52.518 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:52.519 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.907277+0000", 2024-09-13T11:23:52.519 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.83889509500000004 2024-09-13T11:23:52.519 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:52.519 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:52.519 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:23:52.519 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.907284+0000", 2024-09-13T11:23:52.519 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.6359999999999994e-06 2024-09-13T11:23:52.519 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:52.519 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:52.519 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:52.519 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:52.519 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:52.519 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:54846 3.1e 3.742e049e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-13T11:23:52.519 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:32.153567+0000", 2024-09-13T11:23:52.520 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 20.286907811999999, 2024-09-13T11:23:52.520 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.75378642399999995, 2024-09-13T11:23:52.520 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:23:52.520 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:23:52.520 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-13T11:23:52.520 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:23:52.520 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:23:52.520 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:52.520 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 54846 2024-09-13T11:23:52.520 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:52.520 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:23:52.520 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:52.520 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:23:52.521 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.153567+0000", 2024-09-13T11:23:52.521 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:52.521 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:52.521 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:52.521 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:23:52.521 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.153567+0000", 2024-09-13T11:23:52.521 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:52.521 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:52.521 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:52.521 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:23:52.521 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.153569+0000", 2024-09-13T11:23:52.521 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.4660000000000002e-06 2024-09-13T11:23:52.521 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:52.521 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:52.522 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:23:52.522 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.153619+0000", 2024-09-13T11:23:52.522 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.9617000000000003e-05 2024-09-13T11:23:52.522 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:52.522 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:52.522 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:23:52.522 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.153634+0000", 2024-09-13T11:23:52.522 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.4904e-05 2024-09-13T11:23:52.522 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:52.522 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:52.522 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:23:52.522 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.153642+0000", 2024-09-13T11:23:52.522 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.5599999999999994e-06 2024-09-13T11:23:52.522 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:52.523 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:52.523 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:52.523 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.153721+0000", 2024-09-13T11:23:52.523 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.8603999999999995e-05 2024-09-13T11:23:52.523 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:52.523 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:52.523 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for peered", 2024-09-13T11:23:52.523 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.153730+0000", 2024-09-13T11:23:52.523 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.3840000000000003e-06 2024-09-13T11:23:52.523 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:52.523 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:52.523 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:52.523 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.907346+0000", 2024-09-13T11:23:52.523 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.75361563099999995 2024-09-13T11:23:52.524 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:52.524 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:52.524 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:23:52.524 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.907353+0000", 2024-09-13T11:23:52.524 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.2579999999999998e-06 2024-09-13T11:23:52.524 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:52.524 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:52.524 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:52.524 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:52.524 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:52.524 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:54888 3.1e 3.43e14ede (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-13T11:23:52.524 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:32.423064+0000", 2024-09-13T11:23:52.524 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 20.017410140999999, 2024-09-13T11:23:52.524 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.48433585299999998, 2024-09-13T11:23:52.525 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:23:52.525 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:23:52.525 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-13T11:23:52.525 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:23:52.525 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:23:52.525 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:52.525 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 54888 2024-09-13T11:23:52.525 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:52.525 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:23:52.525 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:52.525 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:23:52.525 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.423064+0000", 2024-09-13T11:23:52.525 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:52.525 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:52.525 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:52.526 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:23:52.526 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.423064+0000", 2024-09-13T11:23:52.526 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:52.526 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:52.526 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:52.526 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:23:52.526 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.423069+0000", 2024-09-13T11:23:52.526 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.7389999999999999e-06 2024-09-13T11:23:52.526 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:52.526 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:52.526 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:23:52.526 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.423115+0000", 2024-09-13T11:23:52.526 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.5847000000000003e-05 2024-09-13T11:23:52.526 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:52.527 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:52.527 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:23:52.527 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.423126+0000", 2024-09-13T11:23:52.527 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.1418e-05 2024-09-13T11:23:52.527 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:52.527 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:52.527 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:23:52.527 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.423132+0000", 2024-09-13T11:23:52.527 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.57e-06 2024-09-13T11:23:52.527 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:52.527 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:52.527 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:52.527 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.423191+0000", 2024-09-13T11:23:52.527 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.9250999999999999e-05 2024-09-13T11:23:52.527 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:52.528 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:52.528 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for peered", 2024-09-13T11:23:52.528 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.423198+0000", 2024-09-13T11:23:52.528 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.9179999999999997e-06 2024-09-13T11:23:52.528 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:52.528 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:52.528 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:52.528 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.907392+0000", 2024-09-13T11:23:52.528 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.48419437500000001 2024-09-13T11:23:52.528 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:52.528 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:52.528 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:23:52.528 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.907400+0000", 2024-09-13T11:23:52.528 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.7349999999999996e-06 2024-09-13T11:23:52.529 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:52.529 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:52.529 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:52.529 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:52.529 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:52.529 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDPGPushReply(3.17 76/70 [PushReplyOp(3:eb8d2fae:::benchmark_data_smithi177_31116_object18267:head)])", 2024-09-13T11:23:52.529 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:32.636274+0000", 2024-09-13T11:23:52.529 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 19.804200548000001, 2024-09-13T11:23:52.529 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.269081036, 2024-09-13T11:23:52.529 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:23:52.529 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:23:52.529 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-13T11:23:52.529 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:23:52.530 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:52.530 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:23:52.530 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.636274+0000", 2024-09-13T11:23:52.530 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:52.530 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:52.530 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:52.530 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:23:52.530 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.636274+0000", 2024-09-13T11:23:52.530 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:52.530 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:52.530 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:52.530 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:23:52.530 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.636276+0000", 2024-09-13T11:23:52.530 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.1679999999999998e-06 2024-09-13T11:23:52.531 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:52.531 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:52.531 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:23:52.531 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.636284+0000", 2024-09-13T11:23:52.531 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.7440000000000007e-06 2024-09-13T11:23:52.531 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:52.531 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:52.531 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:23:52.531 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.904155+0000", 2024-09-13T11:23:52.531 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.26787080800000002 2024-09-13T11:23:52.531 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:52.531 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:52.531 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:23:52.531 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.904165+0000", 2024-09-13T11:23:52.532 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0750999999999999e-05 2024-09-13T11:23:52.532 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:52.532 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:52.532 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:52.532 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.904238+0000", 2024-09-13T11:23:52.532 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.2335000000000005e-05 2024-09-13T11:23:52.532 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:52.532 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:52.532 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:23:52.532 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.905355+0000", 2024-09-13T11:23:52.532 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0011172300000000001 2024-09-13T11:23:52.532 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:52.532 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:52.533 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:52.533 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:52.533 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:52.533 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDPGPushReply(3.17 76/70 [PushReplyOp(3:ebcbce67:::benchmark_data_smithi177_31116_object18268:head)])", 2024-09-13T11:23:52.533 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:32.641093+0000", 2024-09-13T11:23:52.533 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 19.799381, 2024-09-13T11:23:52.533 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.27471406799999998, 2024-09-13T11:23:52.533 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:23:52.533 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:23:52.533 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-13T11:23:52.533 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:23:52.533 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:52.533 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:23:52.533 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.641093+0000", 2024-09-13T11:23:52.533 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:52.534 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:52.534 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:52.534 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:23:52.534 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.641093+0000", 2024-09-13T11:23:52.534 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:52.534 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:52.534 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:52.534 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:23:52.534 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.641094+0000", 2024-09-13T11:23:52.534 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.0299999999999999e-07 2024-09-13T11:23:52.534 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:52.534 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:52.534 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:23:52.534 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.641098+0000", 2024-09-13T11:23:52.535 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.2200000000000003e-06 2024-09-13T11:23:52.535 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:52.535 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:52.535 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:23:52.535 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.910434+0000", 2024-09-13T11:23:52.535 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.26933570600000001 2024-09-13T11:23:52.535 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:52.535 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:52.535 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:23:52.535 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.910442+0000", 2024-09-13T11:23:52.535 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.2700000000000004e-06 2024-09-13T11:23:52.535 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:52.535 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:52.535 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:52.536 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.914989+0000", 2024-09-13T11:23:52.536 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0045465059999999996 2024-09-13T11:23:52.536 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:52.536 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:52.536 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:23:52.536 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.915808+0000", 2024-09-13T11:23:52.536 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00081876300000000002 2024-09-13T11:23:52.536 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:52.536 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:52.536 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:52.536 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:52.536 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:52.536 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4665.0:59977 3.21 e83/78 3:86b4d8f6:::benchmark_data_smithi177_31116_object59976:head v 83'3136, mlcod=0'0)", 2024-09-13T11:23:52.536 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:40.571147+0000", 2024-09-13T11:23:52.537 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 11.869327692000001, 2024-09-13T11:23:52.537 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.51907542799999995, 2024-09-13T11:23:52.537 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:23:52.537 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:23:52.537 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:52.537 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:23:52.537 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:52.537 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:23:52.537 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:40.571147+0000", 2024-09-13T11:23:52.537 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:52.537 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:52.537 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:52.537 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:23:52.537 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:40.571147+0000", 2024-09-13T11:23:52.538 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:52.538 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:52.538 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:52.538 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:23:52.538 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:40.571151+0000", 2024-09-13T11:23:52.538 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.5650000000000003e-06 2024-09-13T11:23:52.538 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:52.538 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:52.538 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:23:52.538 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:40.571204+0000", 2024-09-13T11:23:52.538 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.2720999999999999e-05 2024-09-13T11:23:52.538 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:52.538 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:52.538 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:23:52.539 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:41.088332+0000", 2024-09-13T11:23:52.539 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.51712822000000003 2024-09-13T11:23:52.539 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:52.539 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:52.539 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:23:52.539 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:41.088341+0000", 2024-09-13T11:23:52.539 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.3100000000000001e-06 2024-09-13T11:23:52.539 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:52.539 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:52.539 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:52.539 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:41.088444+0000", 2024-09-13T11:23:52.539 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000103708 2024-09-13T11:23:52.539 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:52.539 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:52.539 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:23:52.540 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:41.088475+0000", 2024-09-13T11:23:52.540 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.0497e-05 2024-09-13T11:23:52.540 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:52.540 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:52.540 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:23:52.540 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:41.090192+0000", 2024-09-13T11:23:52.540 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.001717288 2024-09-13T11:23:52.540 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:52.540 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:52.540 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:23:52.540 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:41.090222+0000", 2024-09-13T11:23:52.540 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.0119000000000001e-05 2024-09-13T11:23:52.540 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:52.540 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:52.541 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:52.541 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:52.541 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:52.541 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-13T11:23:52.542 INFO:tasks.ceph.osd.15.smithi177.stderr:2024-09-13T11:23:52.536+0000 7f9d6ce77640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26578) UID: 0 2024-09-13T11:23:52.543 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_ops_in_flight 2024-09-13T11:23:52.642 INFO:tasks.ceph.osd.4.smithi001.stderr:2024-09-13T11:23:52.636+0000 7ff8e0218640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26310) UID: 0 2024-09-13T11:23:52.717 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-13T11:23:52.717 INFO:teuthology.orchestra.run.smithi001.stdout: "ops": [ 2024-09-13T11:23:52.717 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:52.717 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:70234 3.20 3:04c25419:::benchmark_data_smithi177_31116_object70233:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95)", 2024-09-13T11:23:52.717 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:52.706238+0000", 2024-09-13T11:23:52.718 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 0.010380172, 2024-09-13T11:23:52.718 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.01040017, 2024-09-13T11:23:52.718 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:23:52.718 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:23:52.718 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "waiting for sub ops", 2024-09-13T11:23:52.718 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:23:52.718 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:23:52.718 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:52.718 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 70234 2024-09-13T11:23:52.718 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:52.718 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:23:52.718 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:52.718 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:23:52.718 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:52.706238+0000", 2024-09-13T11:23:52.719 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:52.719 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:52.719 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:52.719 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:23:52.719 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:52.706238+0000", 2024-09-13T11:23:52.719 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:52.719 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:52.719 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:52.719 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:23:52.719 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:52.706241+0000", 2024-09-13T11:23:52.719 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.2699999999999999e-06 2024-09-13T11:23:52.719 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:52.719 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:52.719 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:23:52.719 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:52.706283+0000", 2024-09-13T11:23:52.720 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.1891999999999998e-05 2024-09-13T11:23:52.720 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:52.720 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:52.720 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:23:52.720 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:52.706293+0000", 2024-09-13T11:23:52.720 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0105999999999999e-05 2024-09-13T11:23:52.720 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:52.720 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:52.720 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:23:52.720 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:52.706298+0000", 2024-09-13T11:23:52.720 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.0390000000000004e-06 2024-09-13T11:23:52.720 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:52.720 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:52.721 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:52.721 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:52.706348+0000", 2024-09-13T11:23:52.721 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.0108999999999997e-05 2024-09-13T11:23:52.721 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:52.721 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:52.721 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:23:52.721 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:52.706449+0000", 2024-09-13T11:23:52.721 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000101424 2024-09-13T11:23:52.721 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:52.721 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:52.721 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 0", 2024-09-13T11:23:52.721 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:52.706538+0000", 2024-09-13T11:23:52.721 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.8516000000000004e-05 2024-09-13T11:23:52.721 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:52.722 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:52.722 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:23:52.722 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:52.707490+0000", 2024-09-13T11:23:52.722 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00095207099999999997 2024-09-13T11:23:52.722 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:52.722 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:52.722 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:52.722 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:52.722 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-09-13T11:23:52.722 INFO:teuthology.orchestra.run.smithi001.stdout: "num_ops": 1 2024-09-13T11:23:52.722 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-13T11:23:52.732 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_blocked_ops 2024-09-13T11:23:52.743 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-09-13T11:23:52.741+0000 7f97befc9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26315) UID: 0 2024-09-13T11:23:52.844 INFO:tasks.ceph.osd.12.smithi001.stderr:2024-09-13T11:23:52.840+0000 7f892c6e6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26314) UID: 0 2024-09-13T11:23:52.907 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-13T11:23:52.907 INFO:teuthology.orchestra.run.smithi001.stdout: "ops": [], 2024-09-13T11:23:52.907 INFO:teuthology.orchestra.run.smithi001.stdout: "complaint_time": 30, 2024-09-13T11:23:52.907 INFO:teuthology.orchestra.run.smithi001.stdout: "num_blocked_ops": 0 2024-09-13T11:23:52.907 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-13T11:23:52.921 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_historic_ops 2024-09-13T11:23:52.968 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-13T11:23:52.964+0000 7f6a68775640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26307) UID: 0 2024-09-13T11:23:53.006 INFO:tasks.radosbench.radosbench.0.smithi177.stdout: 86 16 70536 70520 51.2446 68.9375 0.00379475 0.0195062 2024-09-13T11:23:53.049 INFO:tasks.ceph.osd.15.smithi177.stderr:2024-09-13T11:23:53.044+0000 7f9d6ce77640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26578) UID: 0 2024-09-13T11:23:53.106 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-13T11:23:53.106 INFO:teuthology.orchestra.run.smithi001.stdout: "size": 20, 2024-09-13T11:23:53.106 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 600, 2024-09-13T11:23:53.106 INFO:teuthology.orchestra.run.smithi001.stdout: "ops": [ 2024-09-13T11:23:53.106 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:53.106 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop_reply(client.4665.0:167 3.9 e41/30 ondisk, result = 0)", 2024-09-13T11:23:53.106 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:22:47.419696+0000", 2024-09-13T11:23:53.106 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 65.683845512000005, 2024-09-13T11:23:53.107 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.64615778199999996, 2024-09-13T11:23:53.107 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:23:53.107 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:23:53.107 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:23:53.107 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:23:53.107 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:53.107 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:23:53.107 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:47.419696+0000", 2024-09-13T11:23:53.107 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:53.107 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:53.107 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:53.107 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:23:53.107 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:47.419696+0000", 2024-09-13T11:23:53.107 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:53.108 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:53.108 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:53.108 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:23:53.108 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:47.419697+0000", 2024-09-13T11:23:53.108 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.15e-06 2024-09-13T11:23:53.108 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:53.108 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:53.108 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:23:53.108 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:47.419700+0000", 2024-09-13T11:23:53.108 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.2449999999999998e-06 2024-09-13T11:23:53.108 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:53.108 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:53.108 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:23:53.108 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:48.065637+0000", 2024-09-13T11:23:53.109 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.64593674300000004 2024-09-13T11:23:53.109 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:53.109 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:53.109 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:23:53.109 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:48.065648+0000", 2024-09-13T11:23:53.109 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0865e-05 2024-09-13T11:23:53.109 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:53.109 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:53.109 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:53.109 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:48.065720+0000", 2024-09-13T11:23:53.109 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.2020999999999994e-05 2024-09-13T11:23:53.109 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:53.109 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:53.109 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:23:53.110 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:48.065728+0000", 2024-09-13T11:23:53.110 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.7660000000000004e-06 2024-09-13T11:23:53.110 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:53.110 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:53.110 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:23:53.110 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:48.065853+0000", 2024-09-13T11:23:53.110 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00012499200000000001 2024-09-13T11:23:53.110 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:53.110 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:53.110 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:53.110 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:53.110 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:53.110 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:25383 3.0 3:0a6bb869:::benchmark_data_smithi177_31116_object25382:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e55)", 2024-09-13T11:23:53.110 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:04.447376+0000", 2024-09-13T11:23:53.111 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 48.656164842999999, 2024-09-13T11:23:53.111 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.21239864, 2024-09-13T11:23:53.111 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:23:53.111 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:23:53.111 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:53.111 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:23:53.111 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:23:53.111 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:53.111 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 25383 2024-09-13T11:23:53.111 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:53.111 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:23:53.111 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:53.111 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:23:53.112 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.447376+0000", 2024-09-13T11:23:53.112 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:53.112 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:53.112 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:53.112 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:23:53.112 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.447376+0000", 2024-09-13T11:23:53.112 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:53.112 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:53.112 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:53.112 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:23:53.112 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.447378+0000", 2024-09-13T11:23:53.112 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.6729999999999999e-06 2024-09-13T11:23:53.112 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:53.112 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:53.113 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:23:53.113 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.447433+0000", 2024-09-13T11:23:53.113 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.5371000000000002e-05 2024-09-13T11:23:53.113 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:53.113 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:53.113 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:23:53.113 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.447446+0000", 2024-09-13T11:23:53.113 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.3179999999999999e-05 2024-09-13T11:23:53.113 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:53.113 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:53.113 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:23:53.113 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.447456+0000", 2024-09-13T11:23:53.113 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0008e-05 2024-09-13T11:23:53.113 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:53.114 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:53.114 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:53.114 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.447514+0000", 2024-09-13T11:23:53.114 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.8023999999999999e-05 2024-09-13T11:23:53.114 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:53.114 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:53.114 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:23:53.114 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.447642+0000", 2024-09-13T11:23:53.114 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000127296 2024-09-13T11:23:53.114 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:53.114 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:53.114 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 15", 2024-09-13T11:23:53.114 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.447767+0000", 2024-09-13T11:23:53.114 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000125547 2024-09-13T11:23:53.115 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:53.115 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:53.115 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:23:53.115 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.449188+0000", 2024-09-13T11:23:53.115 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.001420604 2024-09-13T11:23:53.115 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:53.115 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:53.115 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:53.115 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.659668+0000", 2024-09-13T11:23:53.115 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.210479685 2024-09-13T11:23:53.115 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:53.115 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:53.115 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:23:53.115 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.659716+0000", 2024-09-13T11:23:53.116 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.8609999999999997e-05 2024-09-13T11:23:53.116 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:53.116 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:53.116 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:23:53.116 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.659775+0000", 2024-09-13T11:23:53.116 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.8641999999999999e-05 2024-09-13T11:23:53.116 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:53.116 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:53.116 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:53.116 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:53.116 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:53.116 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4665.0:25408 3.4 e56/40 3:294e23bd:::benchmark_data_smithi177_31116_object25407:head v 56'1645, mlcod=55'1642)", 2024-09-13T11:23:53.116 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:04.494686+0000", 2024-09-13T11:23:53.116 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 48.608855183999999, 2024-09-13T11:23:53.117 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.29725566399999998, 2024-09-13T11:23:53.117 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:23:53.117 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:23:53.117 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:53.117 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:23:53.117 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:53.117 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:23:53.117 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.494686+0000", 2024-09-13T11:23:53.117 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:53.117 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:53.117 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:53.117 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:23:53.117 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.494686+0000", 2024-09-13T11:23:53.117 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:53.117 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:53.118 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:53.118 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:23:53.118 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.494686+0000", 2024-09-13T11:23:53.118 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.2399999999999999e-07 2024-09-13T11:23:53.118 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:53.118 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:53.118 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:23:53.118 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.494809+0000", 2024-09-13T11:23:53.118 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00012308600000000001 2024-09-13T11:23:53.118 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:53.118 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:53.118 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:23:53.118 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.789410+0000", 2024-09-13T11:23:53.118 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.29460066899999998 2024-09-13T11:23:53.119 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:53.119 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:53.119 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:23:53.119 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.789425+0000", 2024-09-13T11:23:53.119 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.4600000000000001e-05 2024-09-13T11:23:53.119 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:53.119 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:53.119 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:53.119 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.789507+0000", 2024-09-13T11:23:53.119 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.2785000000000001e-05 2024-09-13T11:23:53.119 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:53.119 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:53.119 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:23:53.119 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.789525+0000", 2024-09-13T11:23:53.119 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.8117999999999998e-05 2024-09-13T11:23:53.120 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:53.120 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:53.120 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:23:53.120 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.791852+0000", 2024-09-13T11:23:53.120 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0023264459999999998 2024-09-13T11:23:53.120 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:53.120 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:53.120 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:23:53.120 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.791942+0000", 2024-09-13T11:23:53.120 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.9635999999999994e-05 2024-09-13T11:23:53.120 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:53.120 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:53.121 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:53.121 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:53.121 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:53.121 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4665.0:30815 3.f e59/30 3:fde17cb1:::benchmark_data_smithi177_31116_object30814:head v 59'1919, mlcod=59'1914)", 2024-09-13T11:23:53.121 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:09.494342+0000", 2024-09-13T11:23:53.121 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 43.609199197999999, 2024-09-13T11:23:53.121 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.22787855700000001, 2024-09-13T11:23:53.121 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:23:53.121 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:23:53.121 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:53.121 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:23:53.121 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:53.121 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:23:53.121 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.494342+0000", 2024-09-13T11:23:53.122 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:53.122 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:53.122 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:53.122 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:23:53.122 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.494342+0000", 2024-09-13T11:23:53.122 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:53.122 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:53.122 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:53.122 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:23:53.122 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.494343+0000", 2024-09-13T11:23:53.122 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.251e-06 2024-09-13T11:23:53.122 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:53.122 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:53.122 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:23:53.123 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.494387+0000", 2024-09-13T11:23:53.123 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.4276000000000001e-05 2024-09-13T11:23:53.123 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:53.123 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:53.123 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:23:53.123 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.720454+0000", 2024-09-13T11:23:53.123 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.226066614 2024-09-13T11:23:53.123 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:53.123 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:53.123 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:23:53.123 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.720462+0000", 2024-09-13T11:23:53.123 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.9240000000000007e-06 2024-09-13T11:23:53.123 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:53.123 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:53.124 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:53.124 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.720537+0000", 2024-09-13T11:23:53.124 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.4967000000000002e-05 2024-09-13T11:23:53.124 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:53.124 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:53.124 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:23:53.124 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.720563+0000", 2024-09-13T11:23:53.124 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.5630999999999999e-05 2024-09-13T11:23:53.124 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:53.124 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:53.124 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:23:53.124 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.722183+0000", 2024-09-13T11:23:53.124 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0016209670000000001 2024-09-13T11:23:53.125 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:53.125 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:53.125 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:23:53.125 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.722220+0000", 2024-09-13T11:23:53.125 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.6927000000000001e-05 2024-09-13T11:23:53.125 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:53.125 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:53.125 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:53.125 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:53.125 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:53.125 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4665.0:30828 3.f e59/30 3:f3a9c6a9:::benchmark_data_smithi177_31116_object30827:head v 59'1920, mlcod=59'1916)", 2024-09-13T11:23:53.125 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:09.502607+0000", 2024-09-13T11:23:53.125 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 43.600934141000003, 2024-09-13T11:23:53.125 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.26374408300000002, 2024-09-13T11:23:53.126 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:23:53.126 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:23:53.126 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:53.126 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:23:53.126 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:53.126 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:23:53.126 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.502607+0000", 2024-09-13T11:23:53.126 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:53.126 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:53.126 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:53.126 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:23:53.126 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.502607+0000", 2024-09-13T11:23:53.126 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:53.126 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:53.127 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:53.127 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:23:53.127 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.502609+0000", 2024-09-13T11:23:53.127 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.84e-06 2024-09-13T11:23:53.127 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:53.127 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:53.127 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:23:53.127 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.502663+0000", 2024-09-13T11:23:53.127 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.3971000000000002e-05 2024-09-13T11:23:53.127 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:53.127 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:53.127 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:23:53.127 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.763912+0000", 2024-09-13T11:23:53.127 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.26124904300000001 2024-09-13T11:23:53.128 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:53.128 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:53.128 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:23:53.128 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.763920+0000", 2024-09-13T11:23:53.128 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.1149999999999994e-06 2024-09-13T11:23:53.128 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:53.128 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:53.128 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:53.128 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.764024+0000", 2024-09-13T11:23:53.128 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00010380899999999999 2024-09-13T11:23:53.128 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:53.128 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:53.128 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:23:53.128 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.764068+0000", 2024-09-13T11:23:53.129 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.4638999999999997e-05 2024-09-13T11:23:53.129 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:53.129 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:53.129 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:23:53.129 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.766308+0000", 2024-09-13T11:23:53.129 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0022392039999999998 2024-09-13T11:23:53.129 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:53.129 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:53.129 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:23:53.129 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.766351+0000", 2024-09-13T11:23:53.129 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.3461999999999998e-05 2024-09-13T11:23:53.129 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:53.129 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:53.129 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:53.130 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:53.130 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:53.130 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:39894 3.9 3:9190c04c:::benchmark_data_smithi177_31116_object39893:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-13T11:23:53.130 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:15.746095+0000", 2024-09-13T11:23:53.130 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 37.357446516000003, 2024-09-13T11:23:53.130 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.129842069, 2024-09-13T11:23:53.130 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:23:53.130 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:23:53.130 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:53.130 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:23:53.130 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:23:53.130 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:53.130 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 39894 2024-09-13T11:23:53.130 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:53.130 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:23:53.131 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:53.131 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:23:53.131 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:15.746095+0000", 2024-09-13T11:23:53.131 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:53.131 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:53.131 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:53.131 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:23:53.131 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:15.746095+0000", 2024-09-13T11:23:53.131 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:53.131 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:53.131 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:53.131 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:23:53.131 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:15.746096+0000", 2024-09-13T11:23:53.131 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.9800000000000002e-07 2024-09-13T11:23:53.132 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:53.132 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:53.132 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:23:53.132 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:15.746154+0000", 2024-09-13T11:23:53.132 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.8734999999999999e-05 2024-09-13T11:23:53.132 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:53.132 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:53.132 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:23:53.132 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:15.746163+0000", 2024-09-13T11:23:53.132 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.6780000000000006e-06 2024-09-13T11:23:53.132 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:53.132 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:53.132 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:23:53.132 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:15.746168+0000", 2024-09-13T11:23:53.133 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.622e-06 2024-09-13T11:23:53.133 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:53.133 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:53.133 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:53.133 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:15.747109+0000", 2024-09-13T11:23:53.133 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00094147399999999998 2024-09-13T11:23:53.133 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:53.133 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:53.133 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:23:53.133 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:15.747196+0000", 2024-09-13T11:23:53.133 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.6537000000000002e-05 2024-09-13T11:23:53.133 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:53.133 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:53.133 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 2", 2024-09-13T11:23:53.134 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:15.747304+0000", 2024-09-13T11:23:53.134 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000108208 2024-09-13T11:23:53.134 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:53.134 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:53.134 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:23:53.134 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:15.748640+0000", 2024-09-13T11:23:53.134 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0013365339999999999 2024-09-13T11:23:53.134 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:53.134 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:53.134 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:53.134 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:15.875866+0000", 2024-09-13T11:23:53.134 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.12722560399999999 2024-09-13T11:23:53.134 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:53.134 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:53.135 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:23:53.135 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:15.875910+0000", 2024-09-13T11:23:53.135 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.4409000000000001e-05 2024-09-13T11:23:53.135 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:53.135 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:53.135 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:23:53.135 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:15.875937+0000", 2024-09-13T11:23:53.135 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.6270000000000001e-05 2024-09-13T11:23:53.135 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:53.135 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:53.135 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:53.135 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:53.135 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:53.135 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:54351 3.10 3:085f283e:::benchmark_data_smithi177_31116_object54350:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e74)", 2024-09-13T11:23:53.136 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:30.599125+0000", 2024-09-13T11:23:53.136 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 22.504416436, 2024-09-13T11:23:53.136 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.167573261, 2024-09-13T11:23:53.136 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:23:53.136 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:23:53.136 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:53.136 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:23:53.136 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:23:53.136 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:53.136 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 54351 2024-09-13T11:23:53.136 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:53.136 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:23:53.136 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:53.136 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:23:53.137 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:30.599125+0000", 2024-09-13T11:23:53.137 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:53.137 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:53.137 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:53.137 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:23:53.137 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:30.599125+0000", 2024-09-13T11:23:53.137 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:53.137 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:53.137 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:53.137 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:23:53.137 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:30.599127+0000", 2024-09-13T11:23:53.137 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.847e-06 2024-09-13T11:23:53.137 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:53.137 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:53.138 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:23:53.138 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:30.599164+0000", 2024-09-13T11:23:53.138 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.6758999999999998e-05 2024-09-13T11:23:53.138 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:53.138 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:53.138 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:23:53.138 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:30.599184+0000", 2024-09-13T11:23:53.138 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.9928e-05 2024-09-13T11:23:53.138 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:53.138 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:53.138 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:23:53.138 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:30.599194+0000", 2024-09-13T11:23:53.138 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0035e-05 2024-09-13T11:23:53.138 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:53.139 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:53.139 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:53.139 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:30.599259+0000", 2024-09-13T11:23:53.139 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.5147000000000005e-05 2024-09-13T11:23:53.139 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:53.139 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:53.139 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:23:53.139 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:30.599381+0000", 2024-09-13T11:23:53.139 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000121876 2024-09-13T11:23:53.139 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:53.139 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:53.139 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 0,6", 2024-09-13T11:23:53.139 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:30.599496+0000", 2024-09-13T11:23:53.139 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000114307 2024-09-13T11:23:53.139 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:53.140 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:53.140 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:23:53.140 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:30.601094+0000", 2024-09-13T11:23:53.140 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.001598857 2024-09-13T11:23:53.140 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:53.140 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:53.140 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:53.140 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:30.601896+0000", 2024-09-13T11:23:53.140 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00080183800000000003 2024-09-13T11:23:53.140 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:53.140 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:53.140 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:53.140 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:30.766040+0000", 2024-09-13T11:23:53.141 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.16414414099999999 2024-09-13T11:23:53.141 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:53.141 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:53.141 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:23:53.141 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:30.766092+0000", 2024-09-13T11:23:53.141 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.1230000000000002e-05 2024-09-13T11:23:53.141 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:53.141 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:53.141 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:23:53.141 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:30.766698+0000", 2024-09-13T11:23:53.141 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00060629599999999998 2024-09-13T11:23:53.141 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:53.141 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:53.141 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:53.141 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:53.142 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:53.142 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4665.0:54570 3.f e74/30 3:ff206a5c:::benchmark_data_smithi177_31116_object54569:head v 74'3411, mlcod=74'3408)", 2024-09-13T11:23:53.142 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:30.829251+0000", 2024-09-13T11:23:53.142 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 22.274289707000001, 2024-09-13T11:23:53.142 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.12680533199999999, 2024-09-13T11:23:53.142 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:23:53.142 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:23:53.142 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:53.142 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:23:53.142 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:53.142 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:23:53.142 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:30.829251+0000", 2024-09-13T11:23:53.142 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:53.142 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:53.143 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:53.143 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:23:53.143 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:30.829251+0000", 2024-09-13T11:23:53.143 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:53.143 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:53.143 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:53.143 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:23:53.143 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:30.829253+0000", 2024-09-13T11:23:53.143 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.5629999999999999e-06 2024-09-13T11:23:53.143 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:53.143 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:53.143 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:23:53.143 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:30.829313+0000", 2024-09-13T11:23:53.143 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.9864000000000001e-05 2024-09-13T11:23:53.144 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:53.144 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:53.144 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:23:53.144 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:30.952404+0000", 2024-09-13T11:23:53.144 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.123091229 2024-09-13T11:23:53.144 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:53.144 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:53.144 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:23:53.144 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:30.952440+0000", 2024-09-13T11:23:53.144 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.6161999999999997e-05 2024-09-13T11:23:53.144 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:53.144 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:53.144 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:53.144 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:30.952544+0000", 2024-09-13T11:23:53.145 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000103742 2024-09-13T11:23:53.145 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:53.145 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:53.145 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:23:53.145 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:30.952566+0000", 2024-09-13T11:23:53.145 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.2481e-05 2024-09-13T11:23:53.145 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:53.145 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:53.145 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:23:53.145 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:30.955979+0000", 2024-09-13T11:23:53.145 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0034129999999999998 2024-09-13T11:23:53.145 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:53.145 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:53.145 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:23:53.146 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:30.956057+0000", 2024-09-13T11:23:53.146 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.7291000000000002e-05 2024-09-13T11:23:53.146 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:53.146 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:53.146 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:53.146 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:53.146 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:53.146 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4665.0:54651 3.f e74/30 3:f1b76062:::benchmark_data_smithi177_31116_object54650:head v 74'3413, mlcod=74'3410)", 2024-09-13T11:23:53.146 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:30.978338+0000", 2024-09-13T11:23:53.146 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 22.125203329000001, 2024-09-13T11:23:53.146 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.11551019999999999, 2024-09-13T11:23:53.146 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:23:53.146 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:23:53.146 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:23:53.146 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:23:53.147 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:53.147 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:23:53.147 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:30.978338+0000", 2024-09-13T11:23:53.147 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:53.147 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:53.147 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:53.147 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:23:53.147 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:30.978338+0000", 2024-09-13T11:23:53.147 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:53.147 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:53.147 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:53.147 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:23:53.147 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:30.978342+0000", 2024-09-13T11:23:53.147 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.89e-06 2024-09-13T11:23:53.148 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:53.148 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:53.148 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:23:53.148 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:30.978428+0000", 2024-09-13T11:23:53.148 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.6235000000000004e-05 2024-09-13T11:23:53.148 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:53.148 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:53.148 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:23:53.148 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:30.998757+0000", 2024-09-13T11:23:53.148 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.020329122000000002 2024-09-13T11:23:53.148 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:53.148 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:53.148 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:23:53.149 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:30.998776+0000", 2024-09-13T11:23:53.149 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.9378e-05 2024-09-13T11:23:53.149 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:53.149 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:53.149 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:53.149 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:30.998894+0000", 2024-09-13T11:23:53.149 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00011770899999999999 2024-09-13T11:23:53.149 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:53.149 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:53.149 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:23:53.149 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:30.998931+0000", 2024-09-13T11:23:53.149 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.7101000000000003e-05 2024-09-13T11:23:53.149 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:53.149 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:53.150 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:23:53.150 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.093848+0000", 2024-09-13T11:23:53.150 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.094916765 2024-09-13T11:23:53.150 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:53.150 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:53.150 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:53.150 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:53.150 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:53.150 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDPGPushReply(3.10 77/70 [PushReplyOp(3:0b779822:::benchmark_data_smithi177_31116_object18436:head)])", 2024-09-13T11:23:53.150 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:32.995348+0000", 2024-09-13T11:23:53.150 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 20.108193463999999, 2024-09-13T11:23:53.150 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.12346877000000001, 2024-09-13T11:23:53.150 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:23:53.150 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:23:53.150 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-13T11:23:53.151 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:23:53.151 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:53.151 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:23:53.151 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.995348+0000", 2024-09-13T11:23:53.151 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:53.151 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:53.151 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:53.151 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:23:53.151 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.995348+0000", 2024-09-13T11:23:53.151 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:53.151 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:53.151 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:53.151 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:23:53.151 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.995349+0000", 2024-09-13T11:23:53.152 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.5e-06 2024-09-13T11:23:53.152 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:53.152 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:53.152 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:23:53.152 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.995355+0000", 2024-09-13T11:23:53.152 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.4480000000000002e-06 2024-09-13T11:23:53.152 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:53.152 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:53.152 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:23:53.152 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:33.117887+0000", 2024-09-13T11:23:53.152 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.122532616 2024-09-13T11:23:53.152 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:53.152 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:53.152 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:23:53.153 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:33.117898+0000", 2024-09-13T11:23:53.153 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0909e-05 2024-09-13T11:23:53.153 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:53.153 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:53.153 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:53.153 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:33.117972+0000", 2024-09-13T11:23:53.153 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.3683000000000006e-05 2024-09-13T11:23:53.153 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:53.153 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:53.153 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:23:53.153 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:33.118816+0000", 2024-09-13T11:23:53.153 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00084461400000000004 2024-09-13T11:23:53.153 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:53.153 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:53.154 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:53.154 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:53.154 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:53.154 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDPGPush(3.1b 84/78 [PushOp(3:dcddd139:::benchmark_data_smithi177_31116_object10384:head, version: 48'673, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:dcddd139:::benchmark_data_smithi177_31116_object10384:head@48'673, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:23:53.154 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:41.400602+0000", 2024-09-13T11:23:53.154 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 11.702938656000001, 2024-09-13T11:23:53.154 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.43365821999999998, 2024-09-13T11:23:53.154 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:23:53.154 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:23:53.154 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:23:53.154 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:23:53.154 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:53.154 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:23:53.154 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:41.400602+0000", 2024-09-13T11:23:53.155 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:53.155 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:53.155 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:53.155 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:23:53.155 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:41.400602+0000", 2024-09-13T11:23:53.155 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:53.155 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:53.155 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:53.155 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:23:53.155 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:41.400604+0000", 2024-09-13T11:23:53.155 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.99e-06 2024-09-13T11:23:53.155 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:53.155 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:53.155 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:23:53.156 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:41.400653+0000", 2024-09-13T11:23:53.156 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.8195000000000002e-05 2024-09-13T11:23:53.156 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:53.156 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:53.156 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:23:53.156 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:41.833180+0000", 2024-09-13T11:23:53.156 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.43252769200000002 2024-09-13T11:23:53.156 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:53.156 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:53.156 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:23:53.156 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:41.833213+0000", 2024-09-13T11:23:53.156 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.2737999999999997e-05 2024-09-13T11:23:53.156 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:53.157 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:53.157 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:53.157 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:41.833408+0000", 2024-09-13T11:23:53.157 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00019473799999999999 2024-09-13T11:23:53.157 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:53.157 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:53.157 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:23:53.157 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:41.833417+0000", 2024-09-13T11:23:53.157 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.5349999999999993e-06 2024-09-13T11:23:53.157 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:53.157 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:53.157 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:23:53.157 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:41.834261+0000", 2024-09-13T11:23:53.157 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00084333200000000002 2024-09-13T11:23:53.157 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:53.158 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:53.158 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:53.158 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:53.158 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:53.158 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDPGPushReply(3.20 85/77 [PushReplyOp(3:066982a6:::benchmark_data_smithi177_31116_object16400:head)])", 2024-09-13T11:23:53.158 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:42.671551+0000", 2024-09-13T11:23:53.158 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 10.431989832999999, 2024-09-13T11:23:53.158 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.28249085499999999, 2024-09-13T11:23:53.158 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:23:53.158 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:23:53.158 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-13T11:23:53.158 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:23:53.158 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:53.158 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:23:53.159 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.671551+0000", 2024-09-13T11:23:53.159 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:53.159 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:53.159 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:53.159 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:23:53.159 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.671551+0000", 2024-09-13T11:23:53.159 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:53.159 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:53.159 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:53.159 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:23:53.159 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.671552+0000", 2024-09-13T11:23:53.159 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.097e-06 2024-09-13T11:23:53.159 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:53.159 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:53.160 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:23:53.160 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.671556+0000", 2024-09-13T11:23:53.160 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.8519999999999997e-06 2024-09-13T11:23:53.160 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:53.160 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:53.160 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:23:53.160 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.951769+0000", 2024-09-13T11:23:53.160 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.28021293000000003 2024-09-13T11:23:53.160 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:53.160 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:53.160 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:23:53.160 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.951780+0000", 2024-09-13T11:23:53.160 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.1059e-05 2024-09-13T11:23:53.160 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:53.161 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:53.161 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:53.161 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.951992+0000", 2024-09-13T11:23:53.161 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000211478 2024-09-13T11:23:53.161 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:53.161 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:53.161 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:23:53.161 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.954042+0000", 2024-09-13T11:23:53.161 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0020504389999999998 2024-09-13T11:23:53.161 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:53.161 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:53.161 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:53.161 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:53.161 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:53.162 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:61619 3.22 3:455935c5:::benchmark_data_smithi177_31116_object61618:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86)", 2024-09-13T11:23:53.162 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:43.891076+0000", 2024-09-13T11:23:53.162 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 9.2124646069999994, 2024-09-13T11:23:53.162 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.10460481100000001, 2024-09-13T11:23:53.162 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:23:53.162 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:23:53.162 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:53.162 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:23:53.162 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:23:53.162 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:53.162 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 61619 2024-09-13T11:23:53.162 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:53.162 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:23:53.162 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:53.163 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:23:53.163 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:43.891076+0000", 2024-09-13T11:23:53.163 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:53.163 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:53.163 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:53.163 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:23:53.163 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:43.891076+0000", 2024-09-13T11:23:53.163 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:53.163 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:53.163 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:53.163 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:23:53.163 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:43.891079+0000", 2024-09-13T11:23:53.163 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.1270000000000001e-06 2024-09-13T11:23:53.163 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:53.163 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:53.164 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:23:53.164 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:43.891133+0000", 2024-09-13T11:23:53.164 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.4660000000000002e-05 2024-09-13T11:23:53.164 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:53.164 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:53.164 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:23:53.164 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:43.891143+0000", 2024-09-13T11:23:53.164 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.7089999999999997e-06 2024-09-13T11:23:53.164 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:53.164 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:53.164 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:23:53.164 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:43.891148+0000", 2024-09-13T11:23:53.164 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.3959999999999997e-06 2024-09-13T11:23:53.165 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:53.165 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:53.165 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:53.165 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:43.891473+0000", 2024-09-13T11:23:53.165 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00032423400000000001 2024-09-13T11:23:53.165 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:53.165 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:53.165 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:23:53.165 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:43.891625+0000", 2024-09-13T11:23:53.165 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00015278899999999999 2024-09-13T11:23:53.165 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:53.165 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:53.165 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 0,14", 2024-09-13T11:23:53.165 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:43.891784+0000", 2024-09-13T11:23:53.165 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00015875500000000001 2024-09-13T11:23:53.166 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:53.166 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:53.166 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:23:53.166 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:43.893062+0000", 2024-09-13T11:23:53.166 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0012780790000000001 2024-09-13T11:23:53.166 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:53.166 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:53.166 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:53.166 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:43.894487+0000", 2024-09-13T11:23:53.166 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0014246910000000001 2024-09-13T11:23:53.166 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:53.166 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:53.167 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:53.167 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:43.995303+0000", 2024-09-13T11:23:53.167 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.10081649500000001 2024-09-13T11:23:53.167 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:53.167 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:53.167 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:23:53.167 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:43.995359+0000", 2024-09-13T11:23:53.167 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.5804999999999998e-05 2024-09-13T11:23:53.167 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:53.167 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:53.167 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:23:53.167 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:43.995681+0000", 2024-09-13T11:23:53.167 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000322071 2024-09-13T11:23:53.167 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:53.168 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:53.168 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:53.168 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:53.168 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:53.168 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDPGPush(3.1b 89/78 [PushOp(3:dc4c79e7:::benchmark_data_smithi177_31116_object19857:head, version: 54'1266, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:dc4c79e7:::benchmark_data_smithi177_31116_object19857:head@54'1266, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:23:53.168 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:46.649128+0000", 2024-09-13T11:23:53.168 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 6.4544127250000001, 2024-09-13T11:23:53.168 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.29963603, 2024-09-13T11:23:53.168 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:23:53.168 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:23:53.168 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:23:53.168 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:23:53.168 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:53.169 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:23:53.169 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.649128+0000", 2024-09-13T11:23:53.169 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:53.169 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:53.169 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:53.169 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:23:53.169 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.649128+0000", 2024-09-13T11:23:53.169 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:53.169 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:53.169 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:53.169 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:23:53.169 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.649130+0000", 2024-09-13T11:23:53.169 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2950000000000001e-06 2024-09-13T11:23:53.169 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:53.169 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:53.170 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:23:53.170 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.649197+0000", 2024-09-13T11:23:53.170 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.7033999999999995e-05 2024-09-13T11:23:53.170 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:53.170 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:53.170 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:23:53.170 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.947691+0000", 2024-09-13T11:23:53.170 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.29849387300000002 2024-09-13T11:23:53.170 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:53.170 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:53.171 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:23:53.171 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.947702+0000", 2024-09-13T11:23:53.171 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.1239e-05 2024-09-13T11:23:53.171 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:53.171 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:53.171 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:53.171 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.947799+0000", 2024-09-13T11:23:53.171 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.7283999999999999e-05 2024-09-13T11:23:53.171 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:53.171 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:53.171 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:23:53.171 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.947809+0000", 2024-09-13T11:23:53.171 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.008e-05 2024-09-13T11:23:53.171 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:53.172 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:53.172 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:23:53.172 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.948764+0000", 2024-09-13T11:23:53.172 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00095522499999999998 2024-09-13T11:23:53.172 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:53.172 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:53.172 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:53.172 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:53.172 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:53.172 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:64407 3.20 3:0477cf3a:::benchmark_data_smithi177_31116_object64406:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90)", 2024-09-13T11:23:53.172 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:47.219673+0000", 2024-09-13T11:23:53.172 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 5.8838678959999999, 2024-09-13T11:23:53.172 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.126665629, 2024-09-13T11:23:53.172 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:23:53.173 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:23:53.173 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:53.173 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:23:53.173 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:23:53.173 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:53.173 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 64407 2024-09-13T11:23:53.173 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:53.173 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:23:53.173 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:53.173 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:23:53.173 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:47.219673+0000", 2024-09-13T11:23:53.173 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:53.173 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:53.174 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:53.174 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:23:53.174 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:47.219673+0000", 2024-09-13T11:23:53.174 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:53.174 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:53.174 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:53.174 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:23:53.174 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:47.219675+0000", 2024-09-13T11:23:53.174 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.148e-06 2024-09-13T11:23:53.174 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:53.174 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:53.174 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:23:53.174 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:47.219721+0000", 2024-09-13T11:23:53.174 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.5555e-05 2024-09-13T11:23:53.174 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:53.175 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:53.175 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:23:53.175 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:47.219733+0000", 2024-09-13T11:23:53.175 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2098e-05 2024-09-13T11:23:53.175 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:53.175 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:53.175 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:23:53.175 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:47.219741+0000", 2024-09-13T11:23:53.175 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.5549999999999997e-06 2024-09-13T11:23:53.175 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:53.175 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:53.175 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:53.175 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:47.229374+0000", 2024-09-13T11:23:53.175 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0096331850000000007 2024-09-13T11:23:53.176 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:53.176 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:53.176 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:23:53.176 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:47.229554+0000", 2024-09-13T11:23:53.176 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00018005899999999999 2024-09-13T11:23:53.176 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:53.176 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:53.176 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 0", 2024-09-13T11:23:53.176 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:47.229742+0000", 2024-09-13T11:23:53.176 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000188469 2024-09-13T11:23:53.176 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:53.176 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:53.176 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:23:53.177 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:47.231370+0000", 2024-09-13T11:23:53.177 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.001627451 2024-09-13T11:23:53.177 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:53.177 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:53.177 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:53.177 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:47.346217+0000", 2024-09-13T11:23:53.177 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.114847115 2024-09-13T11:23:53.177 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:53.177 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:53.177 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:23:53.177 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:47.346285+0000", 2024-09-13T11:23:53.177 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.7891000000000004e-05 2024-09-13T11:23:53.177 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:53.177 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:53.178 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:23:53.178 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:47.346339+0000", 2024-09-13T11:23:53.178 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.4103e-05 2024-09-13T11:23:53.178 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:53.178 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:53.178 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:53.178 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:53.178 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:53.178 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:70082 3.20 3:073afe11:::benchmark_data_smithi177_31116_object70081:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95)", 2024-09-13T11:23:53.178 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:52.594278+0000", 2024-09-13T11:23:53.178 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 0.50926275099999996, 2024-09-13T11:23:53.178 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.11415911300000001, 2024-09-13T11:23:53.178 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:23:53.178 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:23:53.178 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:53.179 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:23:53.179 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:23:53.179 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:53.179 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 70082 2024-09-13T11:23:53.179 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:53.179 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:23:53.179 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:53.179 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:23:53.179 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:52.594278+0000", 2024-09-13T11:23:53.179 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:53.179 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:53.179 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:53.179 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:23:53.179 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:52.594278+0000", 2024-09-13T11:23:53.180 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:53.180 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:53.180 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:53.180 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:23:53.180 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:52.594281+0000", 2024-09-13T11:23:53.180 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.9979999999999999e-06 2024-09-13T11:23:53.180 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:53.180 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:53.180 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:23:53.180 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:52.594315+0000", 2024-09-13T11:23:53.180 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.3466000000000001e-05 2024-09-13T11:23:53.180 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:53.180 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:53.181 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:23:53.181 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:52.594331+0000", 2024-09-13T11:23:53.181 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.5956000000000001e-05 2024-09-13T11:23:53.181 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:53.181 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:53.181 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:23:53.181 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:52.594340+0000", 2024-09-13T11:23:53.181 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.4080000000000004e-06 2024-09-13T11:23:53.181 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:53.181 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:53.181 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:53.181 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:52.594418+0000", 2024-09-13T11:23:53.181 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.7439000000000002e-05 2024-09-13T11:23:53.181 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:53.182 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:53.182 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:23:53.182 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:52.594635+0000", 2024-09-13T11:23:53.182 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00021716300000000001 2024-09-13T11:23:53.182 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:53.182 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:53.182 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 0", 2024-09-13T11:23:53.182 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:52.594781+0000", 2024-09-13T11:23:53.182 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00014598000000000001 2024-09-13T11:23:53.182 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:53.182 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:53.182 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:23:53.182 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:52.596318+0000", 2024-09-13T11:23:53.182 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.001537294 2024-09-13T11:23:53.183 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:53.183 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:53.183 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:53.183 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:52.708336+0000", 2024-09-13T11:23:53.183 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.112017859 2024-09-13T11:23:53.183 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:53.183 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:53.183 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:23:53.183 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:52.708397+0000", 2024-09-13T11:23:53.183 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.0791000000000001e-05 2024-09-13T11:23:53.183 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:53.183 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:53.183 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:23:53.183 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:52.708437+0000", 2024-09-13T11:23:53.183 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.0759000000000001e-05 2024-09-13T11:23:53.184 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:53.184 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:53.184 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:53.184 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:53.184 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:53.184 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop_reply(client.4665.0:70082 3.20 e95/77 ondisk, result = 0)", 2024-09-13T11:23:53.184 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:52.596380+0000", 2024-09-13T11:23:53.184 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 0.50716121599999997, 2024-09-13T11:23:53.184 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.112064645, 2024-09-13T11:23:53.184 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:23:53.184 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:23:53.184 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:23:53.184 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:23:53.184 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:53.185 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:23:53.185 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:52.596380+0000", 2024-09-13T11:23:53.185 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:53.185 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:53.185 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:53.185 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:23:53.185 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:52.596380+0000", 2024-09-13T11:23:53.185 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:53.185 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:53.185 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:53.185 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:23:53.185 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:52.596381+0000", 2024-09-13T11:23:53.185 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.395e-06 2024-09-13T11:23:53.185 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:53.186 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:53.186 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:23:53.186 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:52.596387+0000", 2024-09-13T11:23:53.186 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.3059999999999997e-06 2024-09-13T11:23:53.186 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:53.186 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:53.186 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:23:53.186 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:52.708263+0000", 2024-09-13T11:23:53.186 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.111876748 2024-09-13T11:23:53.186 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:53.186 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:53.186 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:23:53.186 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:52.708271+0000", 2024-09-13T11:23:53.186 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.5820000000000004e-06 2024-09-13T11:23:53.186 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:53.187 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:53.187 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:53.187 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:52.708323+0000", 2024-09-13T11:23:53.187 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.2494000000000003e-05 2024-09-13T11:23:53.187 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:53.187 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:53.187 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:23:53.187 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:52.708330+0000", 2024-09-13T11:23:53.187 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.9099999999999999e-06 2024-09-13T11:23:53.187 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:53.187 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:53.187 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:23:53.187 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:52.708444+0000", 2024-09-13T11:23:53.188 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00011421000000000001 2024-09-13T11:23:53.188 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:53.188 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:53.188 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:53.188 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:53.188 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:53.188 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4665.0:70358 3.f e95/75 3:f676477d:::benchmark_data_smithi177_31116_object70357:head v 95'3868, mlcod=95'3864)", 2024-09-13T11:23:53.188 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:52.802471+0000", 2024-09-13T11:23:53.188 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 0.30106973300000001, 2024-09-13T11:23:53.188 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.144774187, 2024-09-13T11:23:53.188 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:23:53.188 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:23:53.188 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:53.188 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:23:53.189 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:53.189 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:23:53.189 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:52.802471+0000", 2024-09-13T11:23:53.189 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:53.189 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:53.189 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:53.189 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:23:53.189 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:52.802471+0000", 2024-09-13T11:23:53.189 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:53.189 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:53.189 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:53.189 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:23:53.189 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:52.802473+0000", 2024-09-13T11:23:53.189 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.8610000000000001e-06 2024-09-13T11:23:53.189 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:53.190 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:53.190 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:23:53.190 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:52.802559+0000", 2024-09-13T11:23:53.190 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.5884000000000006e-05 2024-09-13T11:23:53.190 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:53.190 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:53.190 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:23:53.190 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:52.945854+0000", 2024-09-13T11:23:53.190 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.14329460199999999 2024-09-13T11:23:53.190 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:53.190 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:53.190 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:23:53.190 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:52.945858+0000", 2024-09-13T11:23:53.190 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.42e-06 2024-09-13T11:23:53.191 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:53.191 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:53.191 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:53.191 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:52.945904+0000", 2024-09-13T11:23:53.191 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.5719000000000001e-05 2024-09-13T11:23:53.191 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:53.191 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:53.191 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:23:53.191 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:52.945917+0000", 2024-09-13T11:23:53.191 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.3002e-05 2024-09-13T11:23:53.191 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:53.191 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:53.191 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:23:53.191 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:52.947220+0000", 2024-09-13T11:23:53.192 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.001302903 2024-09-13T11:23:53.192 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:53.192 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:53.192 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:23:53.192 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:52.947246+0000", 2024-09-13T11:23:53.192 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.5795999999999998e-05 2024-09-13T11:23:53.192 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:53.192 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:53.192 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:53.192 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:53.192 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:53.193 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:70459 3.20 3:055da73a:::benchmark_data_smithi177_31116_object70458:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95)", 2024-09-13T11:23:53.193 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:52.920422+0000", 2024-09-13T11:23:53.193 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 0.18311873300000001, 2024-09-13T11:23:53.193 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.132136635, 2024-09-13T11:23:53.193 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:23:53.193 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:23:53.193 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:53.193 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:23:53.193 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:23:53.193 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:53.193 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 70459 2024-09-13T11:23:53.193 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:53.193 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:23:53.193 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:53.194 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:23:53.194 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:52.920422+0000", 2024-09-13T11:23:53.194 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:53.194 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:53.194 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:53.194 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:23:53.194 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:52.920422+0000", 2024-09-13T11:23:53.194 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:53.194 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:53.194 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:53.194 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:23:53.194 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:52.920431+0000", 2024-09-13T11:23:53.194 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.7329999999999992e-06 2024-09-13T11:23:53.194 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:53.195 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:53.195 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:23:53.195 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:52.920532+0000", 2024-09-13T11:23:53.195 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00010066899999999999 2024-09-13T11:23:53.195 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:53.195 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:53.195 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:23:53.195 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:52.920566+0000", 2024-09-13T11:23:53.195 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.4452999999999997e-05 2024-09-13T11:23:53.195 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:53.195 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:53.195 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:23:53.195 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:52.920583+0000", 2024-09-13T11:23:53.195 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.7166000000000001e-05 2024-09-13T11:23:53.195 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:53.196 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:53.196 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:53.196 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:52.920687+0000", 2024-09-13T11:23:53.196 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000103463 2024-09-13T11:23:53.196 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:53.196 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:53.196 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:23:53.196 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:52.920816+0000", 2024-09-13T11:23:53.196 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00012956799999999999 2024-09-13T11:23:53.196 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:53.196 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:53.196 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 0", 2024-09-13T11:23:53.196 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:52.920916+0000", 2024-09-13T11:23:53.196 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.9803999999999995e-05 2024-09-13T11:23:53.197 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:53.197 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:53.197 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:23:53.197 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:52.921989+0000", 2024-09-13T11:23:53.197 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0010728560000000001 2024-09-13T11:23:53.197 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:53.197 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:53.197 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:53.197 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:53.052432+0000", 2024-09-13T11:23:53.197 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.13044277400000001 2024-09-13T11:23:53.197 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:53.197 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:53.197 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:23:53.197 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:53.052503+0000", 2024-09-13T11:23:53.198 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.1503999999999998e-05 2024-09-13T11:23:53.198 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:53.198 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:53.198 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:23:53.198 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:53.052559+0000", 2024-09-13T11:23:53.198 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.5645e-05 2024-09-13T11:23:53.198 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:53.198 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:53.198 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:53.198 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:53.198 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:53.198 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop_reply(client.4665.0:70459 3.20 e95/77 ondisk, result = 0)", 2024-09-13T11:23:53.198 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:52.922483+0000", 2024-09-13T11:23:53.198 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 0.18105848999999999, 2024-09-13T11:23:53.199 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.13011756199999999, 2024-09-13T11:23:53.199 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:23:53.199 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:23:53.199 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:23:53.199 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:23:53.199 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:53.199 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:23:53.199 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:52.922483+0000", 2024-09-13T11:23:53.199 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:53.199 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:53.199 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:53.199 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:23:53.199 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:52.922483+0000", 2024-09-13T11:23:53.199 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:23:53.199 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:53.200 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:53.200 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:23:53.200 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:52.922484+0000", 2024-09-13T11:23:53.200 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.4449999999999999e-06 2024-09-13T11:23:53.200 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:53.200 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:53.200 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:23:53.200 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:52.922488+0000", 2024-09-13T11:23:53.200 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.2860000000000001e-06 2024-09-13T11:23:53.200 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:53.200 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:53.201 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:23:53.201 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:53.052355+0000", 2024-09-13T11:23:53.201 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.129866694 2024-09-13T11:23:53.201 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:53.201 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:53.201 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:23:53.201 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:53.052360+0000", 2024-09-13T11:23:53.201 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.6290000000000001e-06 2024-09-13T11:23:53.201 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:53.201 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:53.201 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:23:53.201 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:53.052414+0000", 2024-09-13T11:23:53.201 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.4521999999999997e-05 2024-09-13T11:23:53.201 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:53.202 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:53.202 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:23:53.202 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:53.052424+0000", 2024-09-13T11:23:53.202 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0081000000000001e-05 2024-09-13T11:23:53.202 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:23:53.202 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:23:53.202 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:23:53.202 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:53.052600+0000", 2024-09-13T11:23:53.202 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00017590499999999999 2024-09-13T11:23:53.202 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:53.202 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:53.202 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:53.202 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:23:53.202 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:23:53.202 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-13T11:23:53.204 INFO:tasks.ceph.osd.2.smithi162.stderr:2024-09-13T11:23:53.177+0000 7fe91fd87640 -1 osd.2 95 handle_osd_map injecting map crc failure 2024-09-13T11:23:53.204 INFO:tasks.ceph.osd.1.smithi092.stderr:2024-09-13T11:23:53.175+0000 7fe76d3f2640 -1 osd.1 95 handle_osd_map injecting map crc failure 2024-09-13T11:23:53.204 INFO:tasks.ceph.osd.13.smithi092.stderr:2024-09-13T11:23:53.199+0000 7f3e5afea640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26350) UID: 0 2024-09-13T11:23:53.205 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-13T11:23:53.205 INFO:teuthology.orchestra.run.smithi001.stderr:reweighted osd.12 to 1 (10000) 2024-09-13T11:23:53.206 INFO:teuthology.orchestra.run.smithi001.stderr:nodeep-scrub is set 2024-09-13T11:23:53.206 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-13T11:23:53.176+0000 7f6a5d1a2640 -1 osd.0 95 handle_osd_map injecting map crc failure 2024-09-13T11:23:53.207 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 13 1 2024-09-13T11:23:53.305 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-09-13T11:23:53.301+0000 7f97befc9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26315) UID: 0 2024-09-13T11:23:53.338 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-09-13T11:23:53.338 INFO:teuthology.orchestra.run.smithi092.stdout: "ops": [], 2024-09-13T11:23:53.338 INFO:teuthology.orchestra.run.smithi092.stdout: "num_ops": 0 2024-09-13T11:23:53.338 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-09-13T11:23:53.354 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-13T11:23:53.406 INFO:tasks.ceph.osd.3.smithi177.stderr:2024-09-13T11:23:53.400+0000 7f10adba4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26574) UID: 0 2024-09-13T11:23:53.499 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-09-13T11:23:53.500 INFO:teuthology.orchestra.run.smithi092.stdout: "ops": [], 2024-09-13T11:23:53.500 INFO:teuthology.orchestra.run.smithi092.stdout: "complaint_time": 30, 2024-09-13T11:23:53.500 INFO:teuthology.orchestra.run.smithi092.stdout: "num_blocked_ops": 0 2024-09-13T11:23:53.500 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-09-13T11:23:53.507 INFO:tasks.ceph.osd.14.smithi162.stderr:2024-09-13T11:23:53.505+0000 7f235c0db640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26319) UID: 0 2024-09-13T11:23:53.520 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-13T11:23:53.608 INFO:tasks.ceph.osd.15.smithi177.stderr:2024-09-13T11:23:53.604+0000 7f9d6ce77640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26578) UID: 0 2024-09-13T11:23:53.667 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-09-13T11:23:53.667 INFO:teuthology.orchestra.run.smithi092.stdout: "size": 20, 2024-09-13T11:23:53.667 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 600, 2024-09-13T11:23:53.667 INFO:teuthology.orchestra.run.smithi092.stdout: "ops": [ 2024-09-13T11:23:53.667 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:53.667 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "MOSDPGPush(3.1e 80/78 [PushOp(3:7d369a0c:::benchmark_data_smithi177_31116_object5429:head, version: 45'348, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:7d369a0c:::benchmark_data_smithi177_31116_object5429:head@45'348, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:23:53.668 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:37.973842+0000", 2024-09-13T11:23:53.668 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 15.691047457, 2024-09-13T11:23:53.668 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.032812581, 2024-09-13T11:23:53.668 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:23:53.668 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:23:53.668 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-09-13T11:23:53.668 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:23:53.668 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:53.668 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:23:53.668 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:37.973842+0000", 2024-09-13T11:23:53.668 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:53.668 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:53.668 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:53.668 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:23:53.669 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:37.973842+0000", 2024-09-13T11:23:53.669 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:53.669 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:53.669 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:53.669 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:23:53.669 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:37.973844+0000", 2024-09-13T11:23:53.669 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.0679999999999999e-06 2024-09-13T11:23:53.669 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:53.669 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:53.669 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:23:53.669 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:37.973911+0000", 2024-09-13T11:23:53.669 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.7065999999999998e-05 2024-09-13T11:23:53.669 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:53.670 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:53.670 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:23:53.670 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:38.005936+0000", 2024-09-13T11:23:53.670 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.032024229000000001 2024-09-13T11:23:53.670 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:53.670 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:53.670 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:23:53.670 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:38.005956+0000", 2024-09-13T11:23:53.671 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.0747999999999999e-05 2024-09-13T11:23:53.671 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:53.671 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:53.671 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:23:53.671 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:38.006055+0000", 2024-09-13T11:23:53.671 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.8861000000000001e-05 2024-09-13T11:23:53.671 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:53.671 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:53.671 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:23:53.671 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:38.006065+0000", 2024-09-13T11:23:53.671 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0003999999999999e-05 2024-09-13T11:23:53.671 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:53.671 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:53.671 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:23:53.672 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:38.006655+0000", 2024-09-13T11:23:53.672 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00058960499999999999 2024-09-13T11:23:53.672 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:53.672 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:23:53.672 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:53.672 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:53.672 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:53.672 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "MOSDPGPush(3.1e 81/78 [PushOp(3:7fb76525:::benchmark_data_smithi177_31116_object5788:head, version: 45'372, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:7fb76525:::benchmark_data_smithi177_31116_object5788:head@45'372, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:23:53.672 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:38.165452+0000", 2024-09-13T11:23:53.672 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 15.499437181999999, 2024-09-13T11:23:53.672 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.045132691000000003, 2024-09-13T11:23:53.672 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:23:53.672 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:23:53.673 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-09-13T11:23:53.673 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:23:53.673 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:53.673 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:23:53.673 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:38.165452+0000", 2024-09-13T11:23:53.673 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:53.673 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:53.673 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:53.673 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:23:53.673 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:38.165452+0000", 2024-09-13T11:23:53.673 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:53.673 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:53.674 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:53.674 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:23:53.674 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:38.165454+0000", 2024-09-13T11:23:53.674 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.9539999999999998e-06 2024-09-13T11:23:53.674 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:53.674 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:53.674 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:23:53.674 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:38.165502+0000", 2024-09-13T11:23:53.674 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.7234999999999998e-05 2024-09-13T11:23:53.674 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:53.675 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:53.675 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:23:53.675 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:38.209677+0000", 2024-09-13T11:23:53.675 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.044175752999999998 2024-09-13T11:23:53.675 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:53.675 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:53.675 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:23:53.675 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:38.209696+0000", 2024-09-13T11:23:53.675 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.8252000000000001e-05 2024-09-13T11:23:53.675 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:53.675 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:53.675 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:23:53.675 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:38.209799+0000", 2024-09-13T11:23:53.675 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000103131 2024-09-13T11:23:53.676 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:53.676 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:53.676 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:23:53.676 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:38.209809+0000", 2024-09-13T11:23:53.676 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.961e-06 2024-09-13T11:23:53.676 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:53.676 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:53.676 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:23:53.676 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:38.210585+0000", 2024-09-13T11:23:53.677 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00077640499999999998 2024-09-13T11:23:53.677 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:53.677 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:23:53.677 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:53.677 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:53.677 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:53.677 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "MOSDPGPush(3.1e 81/78 [PushOp(3:7d4aed33:::benchmark_data_smithi177_31116_object5867:head, version: 45'379, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:7d4aed33:::benchmark_data_smithi177_31116_object5867:head@45'379, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:23:53.677 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:38.298563+0000", 2024-09-13T11:23:53.677 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 15.366326214000001, 2024-09-13T11:23:53.677 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.041296825000000002, 2024-09-13T11:23:53.677 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:23:53.677 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:23:53.677 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-09-13T11:23:53.677 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:23:53.678 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:53.678 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:23:53.678 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:38.298563+0000", 2024-09-13T11:23:53.678 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:53.678 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:53.678 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:53.678 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:23:53.678 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:38.298563+0000", 2024-09-13T11:23:53.678 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:53.678 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:53.678 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:53.679 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:23:53.679 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:38.298565+0000", 2024-09-13T11:23:53.679 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.407e-06 2024-09-13T11:23:53.679 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:53.679 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:53.679 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:23:53.679 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:38.298619+0000", 2024-09-13T11:23:53.679 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.4265999999999998e-05 2024-09-13T11:23:53.679 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:53.679 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:53.679 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:23:53.679 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:38.339181+0000", 2024-09-13T11:23:53.679 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.040562355000000001 2024-09-13T11:23:53.680 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:53.680 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:53.680 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:23:53.680 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:38.339198+0000", 2024-09-13T11:23:53.680 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.6971000000000001e-05 2024-09-13T11:23:53.680 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:53.680 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:53.680 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:23:53.680 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:38.339269+0000", 2024-09-13T11:23:53.680 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.1116000000000001e-05 2024-09-13T11:23:53.680 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:53.680 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:53.680 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:23:53.681 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:38.339278+0000", 2024-09-13T11:23:53.681 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.8389999999999996e-06 2024-09-13T11:23:53.681 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:53.681 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:53.681 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:23:53.681 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:38.339860+0000", 2024-09-13T11:23:53.681 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00058187099999999997 2024-09-13T11:23:53.681 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:53.681 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:23:53.681 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:53.681 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:53.681 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:53.681 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "MOSDPGPush(3.1e 81/78 [PushOp(3:7a202f89:::benchmark_data_smithi177_31116_object5883:head, version: 45'381, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:7a202f89:::benchmark_data_smithi177_31116_object5883:head@45'381, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:23:53.682 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:38.345526+0000", 2024-09-13T11:23:53.682 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 15.319363254000001, 2024-09-13T11:23:53.682 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.029400735000000001, 2024-09-13T11:23:53.682 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:23:53.682 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:23:53.682 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-09-13T11:23:53.682 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:23:53.682 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:53.682 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:23:53.682 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:38.345526+0000", 2024-09-13T11:23:53.682 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:53.682 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:53.682 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:53.682 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:23:53.683 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:38.345526+0000", 2024-09-13T11:23:53.683 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:53.683 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:53.683 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:53.683 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:23:53.683 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:38.345528+0000", 2024-09-13T11:23:53.683 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.5200000000000001e-06 2024-09-13T11:23:53.683 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:53.683 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:53.683 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:23:53.683 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:38.345634+0000", 2024-09-13T11:23:53.683 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000105802 2024-09-13T11:23:53.683 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:53.683 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:53.684 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:23:53.684 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:38.373882+0000", 2024-09-13T11:23:53.684 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.028248223999999999 2024-09-13T11:23:53.684 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:53.684 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:53.684 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:23:53.684 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:38.373898+0000", 2024-09-13T11:23:53.684 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.6147999999999999e-05 2024-09-13T11:23:53.684 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:53.684 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:53.684 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:23:53.684 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:38.374004+0000", 2024-09-13T11:23:53.684 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000105646 2024-09-13T11:23:53.684 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:53.685 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:53.685 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:23:53.685 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:38.374016+0000", 2024-09-13T11:23:53.685 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.2235000000000001e-05 2024-09-13T11:23:53.685 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:53.685 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:53.685 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:23:53.686 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:38.374927+0000", 2024-09-13T11:23:53.686 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00091116000000000005 2024-09-13T11:23:53.686 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:53.686 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:23:53.686 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:53.686 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:53.686 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:53.686 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "MOSDPGPush(3.23 83/78 [PushOp(3:c61f6bab:::benchmark_data_smithi177_31116_object29479:head, version: 59'1860, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:c61f6bab:::benchmark_data_smithi177_31116_object29479:head@59'1860, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:23:53.686 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:40.770864+0000", 2024-09-13T11:23:53.686 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 12.894025716, 2024-09-13T11:23:53.687 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.31229662800000002, 2024-09-13T11:23:53.687 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:23:53.687 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:23:53.687 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-09-13T11:23:53.687 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:23:53.687 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:53.687 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:23:53.687 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:40.770864+0000", 2024-09-13T11:23:53.687 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:53.687 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:53.687 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:53.688 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:23:53.688 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:40.770864+0000", 2024-09-13T11:23:53.688 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:53.688 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:53.688 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:53.688 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:23:53.688 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:40.770865+0000", 2024-09-13T11:23:53.688 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.288e-06 2024-09-13T11:23:53.688 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:53.688 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:53.688 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:23:53.688 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:40.770955+0000", 2024-09-13T11:23:53.688 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.9871999999999996e-05 2024-09-13T11:23:53.689 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:53.689 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:53.689 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:23:53.689 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:41.082105+0000", 2024-09-13T11:23:53.689 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.31114963400000001 2024-09-13T11:23:53.689 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:53.689 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:53.689 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:23:53.689 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:41.082136+0000", 2024-09-13T11:23:53.689 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.0972000000000001e-05 2024-09-13T11:23:53.689 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:53.689 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:53.689 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:23:53.689 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:41.082404+0000", 2024-09-13T11:23:53.690 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00026844800000000002 2024-09-13T11:23:53.690 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:53.690 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:53.690 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:23:53.690 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:41.082429+0000", 2024-09-13T11:23:53.690 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.5021e-05 2024-09-13T11:23:53.690 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:53.690 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:53.690 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:23:53.690 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:41.083161+0000", 2024-09-13T11:23:53.690 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000731393 2024-09-13T11:23:53.690 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:53.690 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:23:53.690 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:53.690 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:53.691 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:53.691 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "MOSDPGPush(3.1e 85/78 [PushOp(3:7942cf28:::benchmark_data_smithi177_31116_object14056:head, version: 50'925, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:7942cf28:::benchmark_data_smithi177_31116_object14056:head@50'925, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:23:53.691 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:42.656796+0000", 2024-09-13T11:23:53.691 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 11.008093871, 2024-09-13T11:23:53.691 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.086013249, 2024-09-13T11:23:53.691 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:23:53.691 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:23:53.691 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-09-13T11:23:53.691 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:23:53.691 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:53.691 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:23:53.691 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.656796+0000", 2024-09-13T11:23:53.691 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:53.691 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:53.692 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:53.692 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:23:53.692 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.656796+0000", 2024-09-13T11:23:53.692 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:53.692 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:53.692 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:53.692 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:23:53.692 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.656799+0000", 2024-09-13T11:23:53.692 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.693e-06 2024-09-13T11:23:53.692 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:53.692 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:53.692 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:23:53.692 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.656899+0000", 2024-09-13T11:23:53.693 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.9568999999999995e-05 2024-09-13T11:23:53.693 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:53.693 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:53.693 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:23:53.693 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.741110+0000", 2024-09-13T11:23:53.693 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.084210544999999998 2024-09-13T11:23:53.693 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:53.693 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:53.693 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:23:53.693 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.741140+0000", 2024-09-13T11:23:53.693 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.0306000000000001e-05 2024-09-13T11:23:53.693 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:53.693 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:53.694 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:23:53.694 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.741362+0000", 2024-09-13T11:23:53.694 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000221948 2024-09-13T11:23:53.694 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:53.694 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:53.694 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:23:53.694 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.741385+0000", 2024-09-13T11:23:53.694 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.3632999999999999e-05 2024-09-13T11:23:53.694 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:53.694 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:53.694 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:23:53.694 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.742809+0000", 2024-09-13T11:23:53.695 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.001423555 2024-09-13T11:23:53.695 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:53.695 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:23:53.695 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:53.695 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:53.695 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:53.695 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "MOSDPGPush(3.1e 85/78 [PushOp(3:7f40e925:::benchmark_data_smithi177_31116_object14084:head, version: 50'927, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:7f40e925:::benchmark_data_smithi177_31116_object14084:head@50'927, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:23:53.695 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:42.751000+0000", 2024-09-13T11:23:53.695 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 10.913889408999999, 2024-09-13T11:23:53.695 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.19208357500000001, 2024-09-13T11:23:53.695 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:23:53.695 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:23:53.695 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-09-13T11:23:53.695 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:23:53.696 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:53.696 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:23:53.696 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.751000+0000", 2024-09-13T11:23:53.696 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:53.696 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:53.696 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:53.696 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:23:53.696 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.751000+0000", 2024-09-13T11:23:53.696 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:53.696 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:53.696 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:53.696 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:23:53.696 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.751003+0000", 2024-09-13T11:23:53.696 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.0800000000000002e-06 2024-09-13T11:23:53.697 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:53.697 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:53.697 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:23:53.697 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.751069+0000", 2024-09-13T11:23:53.697 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.5417000000000001e-05 2024-09-13T11:23:53.697 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:53.697 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:53.697 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:23:53.697 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.941261+0000", 2024-09-13T11:23:53.697 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.19019233899999999 2024-09-13T11:23:53.697 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:53.697 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:53.697 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:23:53.697 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.941300+0000", 2024-09-13T11:23:53.698 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.9091999999999998e-05 2024-09-13T11:23:53.698 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:53.698 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:53.698 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:23:53.698 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.941519+0000", 2024-09-13T11:23:53.698 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000218521 2024-09-13T11:23:53.698 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:53.698 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:53.698 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:23:53.698 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.941544+0000", 2024-09-13T11:23:53.698 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.5412999999999999e-05 2024-09-13T11:23:53.698 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:53.698 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:53.699 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:23:53.699 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.943084+0000", 2024-09-13T11:23:53.699 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0015397130000000001 2024-09-13T11:23:53.699 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:53.699 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:23:53.699 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:53.699 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:53.699 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:53.699 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "MOSDPGPush(3.1e 86/78 [PushOp(3:7c567110:::benchmark_data_smithi177_31116_object14516:head, version: 51'957, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:7c567110:::benchmark_data_smithi177_31116_object14516:head@51'957, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:23:53.699 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:43.188372+0000", 2024-09-13T11:23:53.699 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 10.476517858999999, 2024-09-13T11:23:53.699 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.035417856999999997, 2024-09-13T11:23:53.699 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:23:53.699 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:23:53.700 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-09-13T11:23:53.700 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:23:53.700 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:53.700 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:23:53.700 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:43.188372+0000", 2024-09-13T11:23:53.700 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:53.700 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:53.700 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:53.700 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:23:53.700 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:43.188372+0000", 2024-09-13T11:23:53.700 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:53.700 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:53.701 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:53.701 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:23:53.701 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:43.188374+0000", 2024-09-13T11:23:53.701 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.8729999999999999e-06 2024-09-13T11:23:53.701 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:53.701 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:53.701 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:23:53.701 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:43.188409+0000", 2024-09-13T11:23:53.701 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.5024000000000002e-05 2024-09-13T11:23:53.701 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:53.701 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:53.702 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:23:53.702 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:43.222885+0000", 2024-09-13T11:23:53.702 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.034476013 2024-09-13T11:23:53.702 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:53.702 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:53.702 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:23:53.702 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:43.222905+0000", 2024-09-13T11:23:53.702 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.0186999999999998e-05 2024-09-13T11:23:53.702 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:53.702 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:53.702 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:23:53.702 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:43.223013+0000", 2024-09-13T11:23:53.702 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000108544 2024-09-13T11:23:53.703 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:53.703 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:53.703 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:23:53.703 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:43.223030+0000", 2024-09-13T11:23:53.703 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.696e-05 2024-09-13T11:23:53.703 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:53.703 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:53.703 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:23:53.703 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:43.223790+0000", 2024-09-13T11:23:53.703 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00075925600000000004 2024-09-13T11:23:53.703 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:53.703 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:23:53.703 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:53.703 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:53.704 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:53.704 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "MOSDPGPush(3.1e 86/78 [PushOp(3:7aada7c3:::benchmark_data_smithi177_31116_object14640:head, version: 51'967, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:7aada7c3:::benchmark_data_smithi177_31116_object14640:head@51'967, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:23:53.704 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:43.291230+0000", 2024-09-13T11:23:53.704 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 10.373659814, 2024-09-13T11:23:53.704 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.058273907999999999, 2024-09-13T11:23:53.704 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:23:53.704 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:23:53.704 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-09-13T11:23:53.704 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:23:53.704 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:53.704 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:23:53.705 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:43.291230+0000", 2024-09-13T11:23:53.705 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:53.705 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:53.705 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:53.705 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:23:53.705 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:43.291230+0000", 2024-09-13T11:23:53.705 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:53.705 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:53.705 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:53.705 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:23:53.705 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:43.291231+0000", 2024-09-13T11:23:53.705 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.5969999999999999e-06 2024-09-13T11:23:53.705 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:53.706 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:53.706 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:23:53.706 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:43.291296+0000", 2024-09-13T11:23:53.706 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.4956999999999995e-05 2024-09-13T11:23:53.706 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:53.706 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:53.706 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:23:53.706 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:43.348751+0000", 2024-09-13T11:23:53.706 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.057454572000000002 2024-09-13T11:23:53.706 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:53.706 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:53.706 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:23:53.706 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:43.348763+0000", 2024-09-13T11:23:53.706 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.2296000000000001e-05 2024-09-13T11:23:53.707 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:53.707 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:53.707 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:23:53.707 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:43.348833+0000", 2024-09-13T11:23:53.707 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.0197000000000006e-05 2024-09-13T11:23:53.707 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:53.707 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:53.707 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:23:53.707 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:43.348843+0000", 2024-09-13T11:23:53.707 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.1640000000000007e-06 2024-09-13T11:23:53.707 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:53.707 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:53.707 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:23:53.707 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:43.349504+0000", 2024-09-13T11:23:53.708 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00066112500000000004 2024-09-13T11:23:53.708 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:53.708 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:23:53.708 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:53.708 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:53.708 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:53.708 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "MOSDPGPush(3.1e 90/78 [PushOp(3:7f342246:::benchmark_data_smithi177_31116_object22523:head, version: 55'1491, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:7f342246:::benchmark_data_smithi177_31116_object22523:head@55'1491, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:23:53.708 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:47.356265+0000", 2024-09-13T11:23:53.708 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 6.3086241330000004, 2024-09-13T11:23:53.708 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.058328110000000002, 2024-09-13T11:23:53.708 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:23:53.708 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:23:53.709 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-09-13T11:23:53.709 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:23:53.709 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:53.709 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:23:53.709 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:47.356265+0000", 2024-09-13T11:23:53.709 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:53.709 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:53.709 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:53.709 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:23:53.710 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:47.356265+0000", 2024-09-13T11:23:53.710 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:53.710 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:53.710 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:53.710 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:23:53.710 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:47.356268+0000", 2024-09-13T11:23:53.710 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.4439999999999998e-06 2024-09-13T11:23:53.710 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:53.710 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:53.710 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:23:53.710 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:47.356335+0000", 2024-09-13T11:23:53.710 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.7006000000000002e-05 2024-09-13T11:23:53.710 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:53.711 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:53.711 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:23:53.711 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:47.412193+0000", 2024-09-13T11:23:53.711 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.055858374000000002 2024-09-13T11:23:53.711 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:53.711 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:53.711 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:23:53.711 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:47.412231+0000", 2024-09-13T11:23:53.711 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.7267000000000001e-05 2024-09-13T11:23:53.711 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:53.711 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:53.711 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:23:53.711 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:47.412451+0000", 2024-09-13T11:23:53.711 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000220836 2024-09-13T11:23:53.712 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:53.712 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:53.712 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:23:53.712 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:47.412483+0000", 2024-09-13T11:23:53.712 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.1751000000000001e-05 2024-09-13T11:23:53.712 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:53.712 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:53.712 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:23:53.712 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:47.414594+0000", 2024-09-13T11:23:53.712 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0021104320000000002 2024-09-13T11:23:53.712 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:53.712 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:23:53.713 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:53.713 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:53.713 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:53.713 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "MOSDPGPush(3.1e 90/78 [PushOp(3:7b6706a1:::benchmark_data_smithi177_31116_object22539:head, version: 55'1492, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:7b6706a1:::benchmark_data_smithi177_31116_object22539:head@55'1492, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:23:53.713 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:47.420832+0000", 2024-09-13T11:23:53.713 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 6.2440578970000002, 2024-09-13T11:23:53.713 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.035004834999999998, 2024-09-13T11:23:53.713 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:23:53.713 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:23:53.713 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-09-13T11:23:53.713 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:23:53.713 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:53.714 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:23:53.714 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:47.420832+0000", 2024-09-13T11:23:53.714 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:53.714 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:53.714 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:53.714 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:23:53.714 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:47.420832+0000", 2024-09-13T11:23:53.714 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:53.714 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:53.714 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:53.714 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:23:53.714 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:47.420834+0000", 2024-09-13T11:23:53.714 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.3879999999999998e-06 2024-09-13T11:23:53.714 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:53.715 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:53.715 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:23:53.715 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:47.420888+0000", 2024-09-13T11:23:53.715 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.3764999999999998e-05 2024-09-13T11:23:53.715 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:53.715 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:53.715 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:23:53.715 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:47.454433+0000", 2024-09-13T11:23:53.715 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.033544976999999997 2024-09-13T11:23:53.715 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:53.715 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:53.715 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:23:53.715 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:47.454452+0000", 2024-09-13T11:23:53.715 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.9293999999999999e-05 2024-09-13T11:23:53.716 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:53.716 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:53.716 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:23:53.716 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:47.454576+0000", 2024-09-13T11:23:53.716 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000123452 2024-09-13T11:23:53.716 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:53.716 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:53.716 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:23:53.716 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:47.454592+0000", 2024-09-13T11:23:53.716 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.6064000000000001e-05 2024-09-13T11:23:53.716 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:53.716 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:53.716 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:23:53.716 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:47.455837+0000", 2024-09-13T11:23:53.717 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.001244895 2024-09-13T11:23:53.717 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:53.717 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:23:53.717 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:53.717 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:53.717 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:53.717 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "MOSDPGPush(3.1e 90/78 [PushOp(3:782a0814:::benchmark_data_smithi177_31116_object22606:head, version: 55'1494, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:782a0814:::benchmark_data_smithi177_31116_object22606:head@55'1494, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:23:53.717 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:47.480379+0000", 2024-09-13T11:23:53.717 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 6.184510628, 2024-09-13T11:23:53.717 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.061871597, 2024-09-13T11:23:53.717 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:23:53.717 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:23:53.717 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-09-13T11:23:53.718 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:23:53.718 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:53.718 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:23:53.718 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:47.480379+0000", 2024-09-13T11:23:53.718 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:53.718 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:53.718 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:53.718 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:23:53.718 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:47.480379+0000", 2024-09-13T11:23:53.718 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:53.718 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:53.718 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:53.718 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:23:53.718 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:47.480381+0000", 2024-09-13T11:23:53.719 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.1430000000000001e-06 2024-09-13T11:23:53.719 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:53.719 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:53.719 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:23:53.719 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:47.480431+0000", 2024-09-13T11:23:53.719 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.9812999999999999e-05 2024-09-13T11:23:53.719 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:53.719 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:53.719 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:23:53.719 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:47.541318+0000", 2024-09-13T11:23:53.719 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.060886719999999998 2024-09-13T11:23:53.719 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:53.719 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:53.720 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:23:53.720 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:47.541334+0000", 2024-09-13T11:23:53.720 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.6001000000000001e-05 2024-09-13T11:23:53.720 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:53.720 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:53.720 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:23:53.720 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:47.541439+0000", 2024-09-13T11:23:53.720 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000105816 2024-09-13T11:23:53.720 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:53.720 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:53.720 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:23:53.720 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:47.541451+0000", 2024-09-13T11:23:53.720 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.202e-05 2024-09-13T11:23:53.721 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:53.721 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:53.721 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:23:53.721 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:47.542251+0000", 2024-09-13T11:23:53.721 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00079908400000000004 2024-09-13T11:23:53.721 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:53.721 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:23:53.721 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:53.721 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:53.721 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:53.721 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "MOSDPGPush(3.1e 90/78 [PushOp(3:7c65c13b:::benchmark_data_smithi177_31116_object22761:head, version: 55'1503, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:7c65c13b:::benchmark_data_smithi177_31116_object22761:head@55'1503, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:23:53.721 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:47.705735+0000", 2024-09-13T11:23:53.721 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 5.9591543040000001, 2024-09-13T11:23:53.721 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.107567331, 2024-09-13T11:23:53.721 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:23:53.722 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:23:53.722 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-09-13T11:23:53.722 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:23:53.722 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:53.722 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:23:53.722 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:47.705735+0000", 2024-09-13T11:23:53.722 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:53.722 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:53.722 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:53.722 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:23:53.722 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:47.705735+0000", 2024-09-13T11:23:53.722 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:53.722 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:53.723 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:53.723 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:23:53.723 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:47.705738+0000", 2024-09-13T11:23:53.723 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.5940000000000001e-06 2024-09-13T11:23:53.723 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:53.723 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:53.723 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:23:53.723 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:47.705798+0000", 2024-09-13T11:23:53.723 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.9848e-05 2024-09-13T11:23:53.723 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:53.723 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:53.723 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:23:53.723 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:47.811473+0000", 2024-09-13T11:23:53.724 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.10567505100000001 2024-09-13T11:23:53.724 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:53.724 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:53.724 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:23:53.724 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:47.811488+0000", 2024-09-13T11:23:53.724 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.5311999999999998e-05 2024-09-13T11:23:53.724 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:53.724 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:53.724 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:23:53.724 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:47.811684+0000", 2024-09-13T11:23:53.724 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00019606100000000001 2024-09-13T11:23:53.724 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:53.724 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:53.725 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:23:53.725 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:47.811709+0000", 2024-09-13T11:23:53.725 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.5066e-05 2024-09-13T11:23:53.725 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:53.725 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:53.725 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:23:53.725 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:47.813303+0000", 2024-09-13T11:23:53.725 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0015933989999999999 2024-09-13T11:23:53.725 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:53.725 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:23:53.725 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:53.725 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:53.725 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:53.725 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "MOSDPGPush(3.1e 90/78 [PushOp(3:7d0d3b0f:::benchmark_data_smithi177_31116_object22844:head, version: 55'1508, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:7d0d3b0f:::benchmark_data_smithi177_31116_object22844:head@55'1508, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:23:53.726 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:47.847160+0000", 2024-09-13T11:23:53.726 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 5.8177296590000003, 2024-09-13T11:23:53.726 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.077491803999999997, 2024-09-13T11:23:53.726 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:23:53.726 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:23:53.726 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-09-13T11:23:53.726 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:23:53.726 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:53.726 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:23:53.726 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:47.847160+0000", 2024-09-13T11:23:53.726 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:53.726 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:53.726 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:53.726 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:23:53.727 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:47.847160+0000", 2024-09-13T11:23:53.727 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:53.727 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:53.727 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:53.727 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:23:53.727 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:47.847162+0000", 2024-09-13T11:23:53.727 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.7099999999999999e-06 2024-09-13T11:23:53.727 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:53.727 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:53.727 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:23:53.727 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:47.847233+0000", 2024-09-13T11:23:53.727 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.1824999999999998e-05 2024-09-13T11:23:53.727 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:53.728 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:53.728 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:23:53.728 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:47.919816+0000", 2024-09-13T11:23:53.728 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.072582926000000006 2024-09-13T11:23:53.728 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:53.728 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:53.728 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:23:53.728 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:47.919905+0000", 2024-09-13T11:23:53.728 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.8116999999999997e-05 2024-09-13T11:23:53.728 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:53.728 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:53.728 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:23:53.728 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:47.921939+0000", 2024-09-13T11:23:53.729 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0020349140000000001 2024-09-13T11:23:53.729 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:53.729 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:53.729 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:23:53.729 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:47.922021+0000", 2024-09-13T11:23:53.729 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.1694999999999996e-05 2024-09-13T11:23:53.729 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:53.729 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:53.729 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:23:53.729 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:47.924652+0000", 2024-09-13T11:23:53.729 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.002630617 2024-09-13T11:23:53.729 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:53.729 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:23:53.730 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:53.730 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:53.730 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:53.730 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:64721 3.23 3:c7f1bcd7:::benchmark_data_smithi177_31116_object64720:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91)", 2024-09-13T11:23:53.730 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:47.920488+0000", 2024-09-13T11:23:53.730 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 5.7444019009999998, 2024-09-13T11:23:53.730 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.028039516, 2024-09-13T11:23:53.730 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:23:53.730 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:23:53.730 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:53.730 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:23:53.730 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:23:53.730 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:53.730 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 64721 2024-09-13T11:23:53.731 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:53.731 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:23:53.731 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:53.731 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:23:53.731 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:47.920488+0000", 2024-09-13T11:23:53.731 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:53.731 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:53.731 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:53.731 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:23:53.731 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:47.920488+0000", 2024-09-13T11:23:53.731 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:53.731 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:53.731 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:53.732 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:23:53.732 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:47.920489+0000", 2024-09-13T11:23:53.732 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.7570000000000001e-06 2024-09-13T11:23:53.732 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:53.732 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:53.732 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:23:53.732 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:47.920629+0000", 2024-09-13T11:23:53.732 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000139865 2024-09-13T11:23:53.732 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:53.732 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:53.732 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:23:53.732 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:47.920651+0000", 2024-09-13T11:23:53.732 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.1716e-05 2024-09-13T11:23:53.732 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:53.733 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:53.733 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:23:53.733 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:47.920667+0000", 2024-09-13T11:23:53.733 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.5495999999999999e-05 2024-09-13T11:23:53.733 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:53.733 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:53.733 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:23:53.733 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:47.922999+0000", 2024-09-13T11:23:53.733 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0023320319999999999 2024-09-13T11:23:53.733 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:53.733 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:53.733 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-09-13T11:23:53.733 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:47.923014+0000", 2024-09-13T11:23:53.734 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.5245000000000001e-05 2024-09-13T11:23:53.734 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:53.734 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:53.734 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:23:53.734 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:47.937589+0000", 2024-09-13T11:23:53.734 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.014574871999999999 2024-09-13T11:23:53.734 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:53.734 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:53.734 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:23:53.734 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:47.937891+0000", 2024-09-13T11:23:53.734 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00030227800000000001 2024-09-13T11:23:53.734 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:53.734 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:53.735 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 9", 2024-09-13T11:23:53.735 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:47.938232+0000", 2024-09-13T11:23:53.735 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000340646 2024-09-13T11:23:53.735 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:53.735 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:53.735 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:23:53.735 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:47.947838+0000", 2024-09-13T11:23:53.735 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0096065249999999994 2024-09-13T11:23:53.735 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:53.735 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:53.735 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:53.735 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:47.948408+0000", 2024-09-13T11:23:53.735 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00057026199999999998 2024-09-13T11:23:53.736 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:53.736 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:53.736 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:23:53.736 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:47.948478+0000", 2024-09-13T11:23:53.736 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.9945999999999997e-05 2024-09-13T11:23:53.736 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:53.736 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:53.736 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:23:53.736 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:47.948527+0000", 2024-09-13T11:23:53.736 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.8875999999999998e-05 2024-09-13T11:23:53.736 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:53.736 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:23:53.736 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:53.737 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:53.737 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:53.737 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:64824 3.23 3:c58d0b1d:::benchmark_data_smithi177_31116_object64823:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91)", 2024-09-13T11:23:53.737 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:47.921091+0000", 2024-09-13T11:23:53.737 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 5.7437981449999995, 2024-09-13T11:23:53.737 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.02909345, 2024-09-13T11:23:53.737 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:23:53.737 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:23:53.737 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:53.737 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:23:53.737 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:23:53.737 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:53.737 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 64824 2024-09-13T11:23:53.738 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:53.738 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:23:53.738 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:53.738 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:23:53.738 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:47.921091+0000", 2024-09-13T11:23:53.738 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:53.738 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:53.738 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:53.738 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:23:53.738 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:47.921091+0000", 2024-09-13T11:23:53.738 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:53.738 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:53.738 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:53.738 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:23:53.739 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:47.921093+0000", 2024-09-13T11:23:53.739 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.8470000000000001e-06 2024-09-13T11:23:53.739 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:53.739 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:53.739 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:23:53.739 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:47.921238+0000", 2024-09-13T11:23:53.739 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00014429800000000001 2024-09-13T11:23:53.739 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:53.739 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:53.739 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:23:53.739 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:47.921263+0000", 2024-09-13T11:23:53.739 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.5143e-05 2024-09-13T11:23:53.739 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:53.739 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:53.740 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:23:53.740 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:47.921279+0000", 2024-09-13T11:23:53.740 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.6589999999999999e-05 2024-09-13T11:23:53.740 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:53.740 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:53.740 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:23:53.740 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:47.923306+0000", 2024-09-13T11:23:53.740 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.002026361 2024-09-13T11:23:53.740 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:53.740 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:53.740 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-09-13T11:23:53.740 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:47.923321+0000", 2024-09-13T11:23:53.740 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.5722999999999998e-05 2024-09-13T11:23:53.740 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:53.741 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:53.741 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:23:53.741 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:47.944025+0000", 2024-09-13T11:23:53.741 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.020703533999999999 2024-09-13T11:23:53.741 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:53.741 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:53.741 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:23:53.741 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:47.944144+0000", 2024-09-13T11:23:53.741 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000118663 2024-09-13T11:23:53.741 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:53.741 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:53.741 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 9", 2024-09-13T11:23:53.741 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:47.944290+0000", 2024-09-13T11:23:53.742 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000146506 2024-09-13T11:23:53.742 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:53.742 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:53.742 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:23:53.742 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:47.947858+0000", 2024-09-13T11:23:53.742 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0035677640000000002 2024-09-13T11:23:53.742 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:53.742 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:53.742 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:53.742 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:47.950061+0000", 2024-09-13T11:23:53.742 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00220294 2024-09-13T11:23:53.742 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:53.742 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:53.742 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:23:53.743 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:47.950137+0000", 2024-09-13T11:23:53.743 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.6036000000000002e-05 2024-09-13T11:23:53.743 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:53.743 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:53.743 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:23:53.743 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:47.950185+0000", 2024-09-13T11:23:53.743 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.8044999999999998e-05 2024-09-13T11:23:53.743 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:53.743 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:23:53.743 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:53.743 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:53.743 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:53.743 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:64879 3.23 3:c468a4a7:::benchmark_data_smithi177_31116_object64878:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91)", 2024-09-13T11:23:53.744 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:47.921304+0000", 2024-09-13T11:23:53.744 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 5.7435854519999996, 2024-09-13T11:23:53.744 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.029912841999999999, 2024-09-13T11:23:53.744 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:23:53.744 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:23:53.744 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:53.744 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:23:53.744 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:23:53.744 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:53.744 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 64879 2024-09-13T11:23:53.744 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:53.744 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:23:53.745 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:53.745 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:23:53.745 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:47.921304+0000", 2024-09-13T11:23:53.745 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:53.745 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:53.745 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:53.745 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:23:53.745 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:47.921304+0000", 2024-09-13T11:23:53.745 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:53.745 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:53.745 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:53.745 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:23:53.745 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:47.921306+0000", 2024-09-13T11:23:53.746 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.1169999999999998e-06 2024-09-13T11:23:53.746 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:53.746 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:53.746 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:23:53.746 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:47.921472+0000", 2024-09-13T11:23:53.746 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000166181 2024-09-13T11:23:53.746 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:53.746 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:53.746 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:23:53.746 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:47.921495+0000", 2024-09-13T11:23:53.746 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.2416e-05 2024-09-13T11:23:53.746 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:53.746 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:53.746 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:23:53.747 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:47.921508+0000", 2024-09-13T11:23:53.747 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.3506000000000001e-05 2024-09-13T11:23:53.747 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:53.747 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:53.747 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:23:53.747 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:47.923361+0000", 2024-09-13T11:23:53.747 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0018523210000000001 2024-09-13T11:23:53.747 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:53.747 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:53.747 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-09-13T11:23:53.747 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:47.923368+0000", 2024-09-13T11:23:53.747 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.5360000000000004e-06 2024-09-13T11:23:53.747 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:53.748 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:53.748 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:23:53.748 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:47.945700+0000", 2024-09-13T11:23:53.748 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.022331747999999998 2024-09-13T11:23:53.748 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:53.748 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:53.748 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:23:53.748 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:47.945856+0000", 2024-09-13T11:23:53.748 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00015647600000000001 2024-09-13T11:23:53.748 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:53.748 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:53.748 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 9", 2024-09-13T11:23:53.748 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:47.946016+0000", 2024-09-13T11:23:53.749 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00015942300000000001 2024-09-13T11:23:53.749 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:53.749 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:53.749 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:23:53.749 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:47.947864+0000", 2024-09-13T11:23:53.749 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0018477179999999999 2024-09-13T11:23:53.749 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:53.749 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:53.749 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:53.749 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:47.950969+0000", 2024-09-13T11:23:53.749 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0031058320000000002 2024-09-13T11:23:53.749 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:53.749 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:53.750 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:23:53.750 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:47.951129+0000", 2024-09-13T11:23:53.750 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000159691 2024-09-13T11:23:53.750 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:53.750 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:53.750 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:23:53.750 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:47.951217+0000", 2024-09-13T11:23:53.750 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.7876999999999999e-05 2024-09-13T11:23:53.750 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:53.750 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:23:53.750 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:53.750 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:53.750 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:53.750 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:64901 3.23 3:c5ae0d57:::benchmark_data_smithi177_31116_object64900:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91)", 2024-09-13T11:23:53.751 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:47.921540+0000", 2024-09-13T11:23:53.751 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 5.7433493660000003, 2024-09-13T11:23:53.751 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.029975383000000001, 2024-09-13T11:23:53.751 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:23:53.751 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:23:53.751 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:53.751 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:23:53.751 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:23:53.751 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:53.751 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 64901 2024-09-13T11:23:53.751 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:53.751 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:23:53.751 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:53.751 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:23:53.752 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:47.921540+0000", 2024-09-13T11:23:53.752 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:53.752 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:53.752 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:53.752 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:23:53.752 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:47.921540+0000", 2024-09-13T11:23:53.752 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:53.752 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:53.752 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:53.752 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:23:53.752 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:47.921543+0000", 2024-09-13T11:23:53.752 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.34e-06 2024-09-13T11:23:53.752 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:53.753 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:53.753 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:23:53.753 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:47.921689+0000", 2024-09-13T11:23:53.753 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00014648100000000001 2024-09-13T11:23:53.753 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:53.753 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:53.753 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:23:53.753 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:47.921712+0000", 2024-09-13T11:23:53.753 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.2722999999999999e-05 2024-09-13T11:23:53.753 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:53.753 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:53.753 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:23:53.754 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:47.921724+0000", 2024-09-13T11:23:53.754 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.2670000000000001e-05 2024-09-13T11:23:53.754 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:53.754 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:53.754 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:23:53.754 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:47.923492+0000", 2024-09-13T11:23:53.754 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0017676230000000001 2024-09-13T11:23:53.754 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:53.754 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:53.754 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-09-13T11:23:53.754 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:47.923516+0000", 2024-09-13T11:23:53.754 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.4256e-05 2024-09-13T11:23:53.754 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:53.754 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:53.755 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:23:53.755 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:47.946765+0000", 2024-09-13T11:23:53.755 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.023249041000000002 2024-09-13T11:23:53.755 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:53.755 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:53.755 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:23:53.755 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:47.946909+0000", 2024-09-13T11:23:53.755 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00014331399999999999 2024-09-13T11:23:53.755 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:53.755 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:53.755 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 9", 2024-09-13T11:23:53.755 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:47.947067+0000", 2024-09-13T11:23:53.755 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00015845900000000001 2024-09-13T11:23:53.756 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:53.756 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:53.756 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:23:53.756 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:47.950492+0000", 2024-09-13T11:23:53.756 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0034244330000000002 2024-09-13T11:23:53.756 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:53.756 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:53.756 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:53.756 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:47.951369+0000", 2024-09-13T11:23:53.756 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000877478 2024-09-13T11:23:53.756 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:53.756 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:53.756 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:23:53.756 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:47.951451+0000", 2024-09-13T11:23:53.757 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.1973999999999998e-05 2024-09-13T11:23:53.757 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:53.757 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:53.757 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:23:53.757 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:47.951516+0000", 2024-09-13T11:23:53.757 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.4591000000000006e-05 2024-09-13T11:23:53.757 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:53.757 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:23:53.757 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:53.757 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:53.757 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:53.757 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "MOSDPGPush(3.1e 91/78 [PushOp(3:7db197f7:::benchmark_data_smithi177_31116_object23201:head, version: 55'1528, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:7db197f7:::benchmark_data_smithi177_31116_object23201:head@55'1528, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:23:53.758 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:48.004266+0000", 2024-09-13T11:23:53.758 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 5.6606240919999999, 2024-09-13T11:23:53.758 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.031291713999999998, 2024-09-13T11:23:53.758 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:23:53.758 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:23:53.758 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-09-13T11:23:53.758 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:23:53.758 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:53.758 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:23:53.758 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:48.004266+0000", 2024-09-13T11:23:53.758 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:53.758 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:53.759 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:53.759 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:23:53.759 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:48.004266+0000", 2024-09-13T11:23:53.759 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:53.759 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:53.759 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:53.759 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:23:53.759 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:48.004268+0000", 2024-09-13T11:23:53.759 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.4339999999999999e-06 2024-09-13T11:23:53.759 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:53.759 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:53.759 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:23:53.759 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:48.004376+0000", 2024-09-13T11:23:53.760 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000108424 2024-09-13T11:23:53.760 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:53.760 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:53.760 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:23:53.760 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:48.034297+0000", 2024-09-13T11:23:53.760 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.029920237999999998 2024-09-13T11:23:53.760 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:53.760 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:53.760 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:23:53.760 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:48.034320+0000", 2024-09-13T11:23:53.760 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.2968000000000001e-05 2024-09-13T11:23:53.760 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:53.760 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:53.760 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:23:53.761 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:48.034435+0000", 2024-09-13T11:23:53.761 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000115035 2024-09-13T11:23:53.761 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:53.761 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:53.761 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:23:53.761 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:48.034445+0000", 2024-09-13T11:23:53.761 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0275000000000001e-05 2024-09-13T11:23:53.761 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:53.761 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:53.761 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:23:53.761 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:48.035557+0000", 2024-09-13T11:23:53.761 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0011123400000000001 2024-09-13T11:23:53.761 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:53.762 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:23:53.762 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:53.762 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:53.762 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:53.762 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "MOSDPGPush(3.1e 94/78 [PushOp(3:7e3b05ed:::benchmark_data_smithi177_31116_object30052:head, version: 59'1945, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:7e3b05ed:::benchmark_data_smithi177_31116_object30052:head@59'1945, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:23:53.762 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:52.086403+0000", 2024-09-13T11:23:53.762 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 1.5784867930000002, 2024-09-13T11:23:53.762 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.088501402000000007, 2024-09-13T11:23:53.762 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:23:53.762 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:23:53.762 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-09-13T11:23:53.762 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:23:53.762 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:53.763 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:23:53.763 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:52.086403+0000", 2024-09-13T11:23:53.763 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:53.763 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:53.763 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:53.763 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:23:53.763 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:52.086403+0000", 2024-09-13T11:23:53.763 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:53.763 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:53.763 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:53.763 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:23:53.763 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:52.086405+0000", 2024-09-13T11:23:53.764 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.5620000000000002e-06 2024-09-13T11:23:53.764 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:53.764 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:53.764 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:23:53.764 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:52.086472+0000", 2024-09-13T11:23:53.764 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.7053999999999999e-05 2024-09-13T11:23:53.764 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:53.764 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:53.764 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:23:53.764 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:52.172902+0000", 2024-09-13T11:23:53.764 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.086430050999999994 2024-09-13T11:23:53.764 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:53.765 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:53.765 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:23:53.765 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:52.172928+0000", 2024-09-13T11:23:53.765 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.5106999999999999e-05 2024-09-13T11:23:53.765 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:53.765 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:53.765 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:23:53.765 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:52.173094+0000", 2024-09-13T11:23:53.765 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00016682200000000001 2024-09-13T11:23:53.765 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:53.765 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:53.765 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:23:53.766 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:52.173117+0000", 2024-09-13T11:23:53.766 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.2459e-05 2024-09-13T11:23:53.766 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:53.766 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:53.766 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:23:53.766 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:52.174904+0000", 2024-09-13T11:23:53.766 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0017873469999999999 2024-09-13T11:23:53.766 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:53.766 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:23:53.766 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:53.766 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:53.766 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:23:53.767 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-09-13T11:23:53.768 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-09-13T11:23:53.769 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-09-13T11:23:53.765+0000 7f97befc9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26315) UID: 0 2024-09-13T11:23:53.869 INFO:tasks.ceph.osd.4.smithi001.stderr:2024-09-13T11:23:53.864+0000 7ff8e0218640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26310) UID: 0 2024-09-13T11:23:53.912 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-09-13T11:23:53.913 INFO:teuthology.orchestra.run.smithi092.stdout: "ops": [ 2024-09-13T11:23:53.913 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:53.913 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:71611 3.19 3:9d9fa5ff:::benchmark_data_smithi177_31116_object71610:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96)", 2024-09-13T11:23:53.913 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:53.892604+0000", 2024-09-13T11:23:53.913 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 0.019516629000000001, 2024-09-13T11:23:53.913 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.019543430000000001, 2024-09-13T11:23:53.913 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:23:53.913 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:23:53.913 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "waiting for sub ops", 2024-09-13T11:23:53.913 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:23:53.913 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:23:53.914 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:53.914 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 71611 2024-09-13T11:23:53.914 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:53.914 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:23:53.914 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:53.914 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:23:53.914 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:53.892604+0000", 2024-09-13T11:23:53.914 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:53.914 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:53.914 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:53.914 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:23:53.914 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:53.892604+0000", 2024-09-13T11:23:53.914 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:53.915 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:53.915 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:53.915 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:23:53.915 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:53.892615+0000", 2024-09-13T11:23:53.915 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.1439e-05 2024-09-13T11:23:53.915 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:53.915 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:53.915 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:23:53.915 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:53.892654+0000", 2024-09-13T11:23:53.915 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.8151000000000001e-05 2024-09-13T11:23:53.915 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:53.915 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:53.915 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:23:53.916 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:53.892668+0000", 2024-09-13T11:23:53.916 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.4939e-05 2024-09-13T11:23:53.916 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:53.916 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:53.916 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:23:53.916 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:53.892678+0000", 2024-09-13T11:23:53.916 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.7589999999999992e-06 2024-09-13T11:23:53.916 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:53.916 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:53.916 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:23:53.916 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:53.892730+0000", 2024-09-13T11:23:53.916 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.2206000000000001e-05 2024-09-13T11:23:53.916 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:53.917 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:53.917 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:23:53.917 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:53.892841+0000", 2024-09-13T11:23:53.917 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000110429 2024-09-13T11:23:53.917 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:53.917 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:53.917 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1", 2024-09-13T11:23:53.917 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:53.892935+0000", 2024-09-13T11:23:53.917 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.4069999999999999e-05 2024-09-13T11:23:53.917 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:53.917 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:53.917 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:23:53.917 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:53.894292+0000", 2024-09-13T11:23:53.918 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0013572370000000001 2024-09-13T11:23:53.918 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:53.918 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:23:53.918 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:53.918 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:53.918 INFO:teuthology.orchestra.run.smithi092.stdout: ], 2024-09-13T11:23:53.918 INFO:teuthology.orchestra.run.smithi092.stdout: "num_ops": 1 2024-09-13T11:23:53.918 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-09-13T11:23:53.929 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-09-13T11:23:53.970 INFO:tasks.ceph.osd.8.smithi001.stderr:2024-09-13T11:23:53.964+0000 7fb9b48ae640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26312) UID: 0 2024-09-13T11:23:54.006 INFO:tasks.radosbench.radosbench.0.smithi177.stdout: 87 16 71768 71752 51.5406 77 0.00325769 0.0193918 2024-09-13T11:23:54.069 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-09-13T11:23:54.069 INFO:teuthology.orchestra.run.smithi092.stdout: "ops": [], 2024-09-13T11:23:54.069 INFO:teuthology.orchestra.run.smithi092.stdout: "complaint_time": 30, 2024-09-13T11:23:54.070 INFO:teuthology.orchestra.run.smithi092.stdout: "num_blocked_ops": 0 2024-09-13T11:23:54.070 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-09-13T11:23:54.071 INFO:tasks.ceph.osd.9.smithi092.stderr:2024-09-13T11:23:54.067+0000 7ff5ab3d0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26348) UID: 0 2024-09-13T11:23:54.087 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-09-13T11:23:54.173 INFO:tasks.ceph.osd.5.smithi092.stderr:2024-09-13T11:23:54.171+0000 7f6f0b705640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26346) UID: 0 2024-09-13T11:23:54.175 INFO:teuthology.orchestra.run.smithi001.stderr:reweighted osd.13 to 1 (10000) 2024-09-13T11:23:54.203 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 14 1 2024-09-13T11:23:54.248 INFO:tasks.ceph.osd.5.smithi092.stderr:2024-09-13T11:23:54.211+0000 7f6f00132640 -1 osd.5 96 handle_osd_map injecting map crc failure 2024-09-13T11:23:54.262 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-09-13T11:23:54.262 INFO:teuthology.orchestra.run.smithi092.stdout: "size": 20, 2024-09-13T11:23:54.262 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 600, 2024-09-13T11:23:54.262 INFO:teuthology.orchestra.run.smithi092.stdout: "ops": [ 2024-09-13T11:23:54.262 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.262 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "MOSDPGPush(3.19 72/70 [PushOp(3:9d4b3c3f:::benchmark_data_smithi177_31116_object6607:head, version: 46'418, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:9d4b3c3f:::benchmark_data_smithi177_31116_object6607:head@46'418, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:23:54.262 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:26.723924+0000", 2024-09-13T11:23:54.263 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 27.535700407, 2024-09-13T11:23:54.263 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.076312135000000003, 2024-09-13T11:23:54.263 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:23:54.263 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:23:54.263 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-09-13T11:23:54.263 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:23:54.263 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.263 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:23:54.263 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:26.723924+0000", 2024-09-13T11:23:54.263 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:54.263 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.263 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.263 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:23:54.264 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:26.723924+0000", 2024-09-13T11:23:54.264 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:54.264 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.264 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.264 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:23:54.264 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:26.723927+0000", 2024-09-13T11:23:54.264 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.1520000000000001e-06 2024-09-13T11:23:54.264 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.264 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.264 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:23:54.264 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:26.723993+0000", 2024-09-13T11:23:54.264 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.5747000000000006e-05 2024-09-13T11:23:54.264 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.265 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.265 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:23:54.265 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:26.798286+0000", 2024-09-13T11:23:54.265 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.074292320999999995 2024-09-13T11:23:54.265 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.265 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.265 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:23:54.265 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:26.798308+0000", 2024-09-13T11:23:54.265 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.1993000000000001e-05 2024-09-13T11:23:54.265 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.265 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.265 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:23:54.265 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:26.799573+0000", 2024-09-13T11:23:54.266 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0012656189999999999 2024-09-13T11:23:54.266 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.266 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.266 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:23:54.266 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:26.799584+0000", 2024-09-13T11:23:54.266 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0961999999999999e-05 2024-09-13T11:23:54.266 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.266 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.266 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:23:54.266 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:26.800236+0000", 2024-09-13T11:23:54.266 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00065234099999999997 2024-09-13T11:23:54.266 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:54.267 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:23:54.267 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:54.267 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.267 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.267 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "MOSDPGPush(3.19 76/70 [PushOp(3:9eacf3d0:::benchmark_data_smithi177_31116_object19747:head, version: 54'1236, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:9eacf3d0:::benchmark_data_smithi177_31116_object19747:head@54'1236, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:23:54.267 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:32.704676+0000", 2024-09-13T11:23:54.267 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 21.554949088000001, 2024-09-13T11:23:54.267 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.23525014899999999, 2024-09-13T11:23:54.267 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:23:54.267 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:23:54.268 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-09-13T11:23:54.268 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:23:54.268 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.268 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:23:54.268 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:32.704676+0000", 2024-09-13T11:23:54.268 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:54.268 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.268 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.268 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:23:54.268 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:32.704676+0000", 2024-09-13T11:23:54.268 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:54.268 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.268 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.268 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:23:54.269 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:32.704677+0000", 2024-09-13T11:23:54.269 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.531e-06 2024-09-13T11:23:54.269 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.269 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.269 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:23:54.269 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:32.704749+0000", 2024-09-13T11:23:54.269 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.1451000000000004e-05 2024-09-13T11:23:54.269 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.269 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.270 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:23:54.270 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:32.938053+0000", 2024-09-13T11:23:54.270 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.233304752 2024-09-13T11:23:54.270 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.270 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.270 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:23:54.270 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:32.938087+0000", 2024-09-13T11:23:54.272 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.4016000000000001e-05 2024-09-13T11:23:54.273 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.273 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.273 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:23:54.273 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:32.938351+0000", 2024-09-13T11:23:54.273 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00026397500000000001 2024-09-13T11:23:54.273 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.273 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.273 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:23:54.273 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:32.938381+0000", 2024-09-13T11:23:54.273 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.0001999999999998e-05 2024-09-13T11:23:54.273 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.273 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.273 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:23:54.274 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:32.939926+0000", 2024-09-13T11:23:54.274 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0015444219999999999 2024-09-13T11:23:54.274 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:54.274 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:23:54.274 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:54.274 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.274 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.274 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "MOSDPGPush(3.23 80/78 [PushOp(3:c54a2e13:::benchmark_data_smithi177_31116_object16923:head, version: 52'1050, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:c54a2e13:::benchmark_data_smithi177_31116_object16923:head@52'1050, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:23:54.274 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:37.274061+0000", 2024-09-13T11:23:54.274 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 16.985563368000001, 2024-09-13T11:23:54.274 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.099925987999999993, 2024-09-13T11:23:54.274 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:23:54.274 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:23:54.274 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-09-13T11:23:54.275 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:23:54.275 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.275 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:23:54.275 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:37.274061+0000", 2024-09-13T11:23:54.275 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:54.275 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.275 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.275 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:23:54.275 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:37.274061+0000", 2024-09-13T11:23:54.275 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:54.275 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.275 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.275 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:23:54.275 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:37.274064+0000", 2024-09-13T11:23:54.276 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.3439999999999999e-06 2024-09-13T11:23:54.276 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.276 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.277 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:23:54.277 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:37.274135+0000", 2024-09-13T11:23:54.277 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.1716000000000003e-05 2024-09-13T11:23:54.277 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.277 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.277 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:23:54.277 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:37.372114+0000", 2024-09-13T11:23:54.277 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.097978407000000003 2024-09-13T11:23:54.277 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.277 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.277 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:23:54.277 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:37.372133+0000", 2024-09-13T11:23:54.278 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.8983000000000001e-05 2024-09-13T11:23:54.278 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.278 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.278 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:23:54.278 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:37.373424+0000", 2024-09-13T11:23:54.278 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0012912220000000001 2024-09-13T11:23:54.278 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.278 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.278 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:23:54.278 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:37.373437+0000", 2024-09-13T11:23:54.278 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.2608e-05 2024-09-13T11:23:54.278 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.278 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.278 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:23:54.279 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:37.373987+0000", 2024-09-13T11:23:54.279 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00055070800000000001 2024-09-13T11:23:54.279 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:54.279 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:23:54.279 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:54.279 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.279 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.279 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "MOSDPGPush(3.23 80/78 [PushOp(3:c59ecc6d:::benchmark_data_smithi177_31116_object17005:head, version: 52'1054, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:c59ecc6d:::benchmark_data_smithi177_31116_object17005:head@52'1054, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:23:54.279 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:37.433040+0000", 2024-09-13T11:23:54.279 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 16.826585112, 2024-09-13T11:23:54.279 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.26658219100000002, 2024-09-13T11:23:54.279 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:23:54.279 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:23:54.280 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-09-13T11:23:54.280 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:23:54.280 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.280 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:23:54.280 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:37.433040+0000", 2024-09-13T11:23:54.280 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:54.280 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.280 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.280 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:23:54.280 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:37.433040+0000", 2024-09-13T11:23:54.280 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:54.280 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.280 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.281 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:23:54.281 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:37.433042+0000", 2024-09-13T11:23:54.281 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.3999999999999999e-06 2024-09-13T11:23:54.281 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.281 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.281 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:23:54.281 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:37.433090+0000", 2024-09-13T11:23:54.281 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.8065999999999997e-05 2024-09-13T11:23:54.281 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.281 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.281 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:23:54.281 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:37.698797+0000", 2024-09-13T11:23:54.281 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.26570674500000002 2024-09-13T11:23:54.281 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.282 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.282 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:23:54.282 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:37.698827+0000", 2024-09-13T11:23:54.282 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.0505e-05 2024-09-13T11:23:54.282 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.282 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.282 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:23:54.282 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:37.698941+0000", 2024-09-13T11:23:54.282 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00011403799999999999 2024-09-13T11:23:54.282 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.282 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.282 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:23:54.282 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:37.698956+0000", 2024-09-13T11:23:54.283 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.4765e-05 2024-09-13T11:23:54.283 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.283 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.283 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:23:54.283 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:37.699622+0000", 2024-09-13T11:23:54.283 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00066567199999999997 2024-09-13T11:23:54.283 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:54.283 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:23:54.283 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:54.283 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.283 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.283 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "MOSDPGPush(3.23 80/78 [PushOp(3:c58155d4:::benchmark_data_smithi177_31116_object17021:head, version: 52'1056, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:c58155d4:::benchmark_data_smithi177_31116_object17021:head@52'1056, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:23:54.283 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:37.704555+0000", 2024-09-13T11:23:54.284 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 16.555069554999999, 2024-09-13T11:23:54.284 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.070200804000000006, 2024-09-13T11:23:54.284 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:23:54.284 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:23:54.284 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-09-13T11:23:54.284 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:23:54.284 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.284 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:23:54.284 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:37.704555+0000", 2024-09-13T11:23:54.284 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:54.284 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.284 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.284 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:23:54.285 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:37.704555+0000", 2024-09-13T11:23:54.285 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:54.285 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.285 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.285 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:23:54.285 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:37.704557+0000", 2024-09-13T11:23:54.285 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.9810000000000002e-06 2024-09-13T11:23:54.285 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.285 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.285 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:23:54.285 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:37.704617+0000", 2024-09-13T11:23:54.285 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.0096000000000002e-05 2024-09-13T11:23:54.285 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.285 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.286 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:23:54.286 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:37.774286+0000", 2024-09-13T11:23:54.286 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.069668616000000003 2024-09-13T11:23:54.286 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.286 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.286 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:23:54.286 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:37.774303+0000", 2024-09-13T11:23:54.286 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.7496999999999999e-05 2024-09-13T11:23:54.286 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.286 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.286 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:23:54.286 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:37.774372+0000", 2024-09-13T11:23:54.286 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.8732999999999994e-05 2024-09-13T11:23:54.286 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.287 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.287 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:23:54.287 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:37.774380+0000", 2024-09-13T11:23:54.287 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.9379999999999992e-06 2024-09-13T11:23:54.287 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.287 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.287 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:23:54.287 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:37.774756+0000", 2024-09-13T11:23:54.287 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000375943 2024-09-13T11:23:54.287 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:54.287 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:23:54.287 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:54.287 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.287 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.287 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "MOSDPGPush(3.1a 81/78 [PushOp(3:5a9c05b8:::benchmark_data_smithi177_31116_object4600:head, version: 44'280, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:5a9c05b8:::benchmark_data_smithi177_31116_object4600:head@44'280, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:23:54.288 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:38.261429+0000", 2024-09-13T11:23:54.288 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 15.998195676, 2024-09-13T11:23:54.288 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.13814789799999999, 2024-09-13T11:23:54.288 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:23:54.288 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:23:54.288 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-09-13T11:23:54.288 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:23:54.288 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.288 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:23:54.289 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:38.261429+0000", 2024-09-13T11:23:54.289 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:54.289 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.289 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.289 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:23:54.289 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:38.261429+0000", 2024-09-13T11:23:54.289 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:54.289 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.289 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.289 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:23:54.289 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:38.261430+0000", 2024-09-13T11:23:54.289 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.376e-06 2024-09-13T11:23:54.289 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.290 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.290 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:23:54.290 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:38.261501+0000", 2024-09-13T11:23:54.290 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.0724999999999998e-05 2024-09-13T11:23:54.290 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.290 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.290 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:23:54.290 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:38.398898+0000", 2024-09-13T11:23:54.290 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.13739662499999999 2024-09-13T11:23:54.290 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.290 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.290 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:23:54.290 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:38.398909+0000", 2024-09-13T11:23:54.291 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0926e-05 2024-09-13T11:23:54.291 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.291 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.291 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:23:54.291 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:38.398974+0000", 2024-09-13T11:23:54.291 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.5467000000000002e-05 2024-09-13T11:23:54.291 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.291 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.291 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:23:54.291 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:38.398982+0000", 2024-09-13T11:23:54.291 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.841e-06 2024-09-13T11:23:54.291 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.291 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.291 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:23:54.292 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:38.399577+0000", 2024-09-13T11:23:54.292 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00059493800000000004 2024-09-13T11:23:54.292 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:54.292 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:23:54.292 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:54.292 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.292 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.292 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "MOSDPGPush(3.19 81/70 [PushOp(3:9deaeb38:::benchmark_data_smithi177_31116_object38227:head, version: 63'2388, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:9deaeb38:::benchmark_data_smithi177_31116_object38227:head@63'2388, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:23:54.292 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:38.373563+0000", 2024-09-13T11:23:54.292 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 15.886061646, 2024-09-13T11:23:54.292 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.087759459999999997, 2024-09-13T11:23:54.292 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:23:54.292 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:23:54.293 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-09-13T11:23:54.293 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:23:54.293 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.293 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:23:54.293 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:38.373563+0000", 2024-09-13T11:23:54.293 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:54.293 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.293 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.293 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:23:54.293 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:38.373563+0000", 2024-09-13T11:23:54.293 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:54.293 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.293 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.294 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:23:54.294 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:38.373565+0000", 2024-09-13T11:23:54.294 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.528e-06 2024-09-13T11:23:54.294 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.294 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.294 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:23:54.294 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:38.373626+0000", 2024-09-13T11:23:54.294 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.1186999999999995e-05 2024-09-13T11:23:54.294 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.294 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.294 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:23:54.294 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:38.460191+0000", 2024-09-13T11:23:54.294 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.086564957999999997 2024-09-13T11:23:54.295 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.295 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.295 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:23:54.295 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:38.460203+0000", 2024-09-13T11:23:54.295 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.2103000000000001e-05 2024-09-13T11:23:54.295 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.295 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.295 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:23:54.295 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:38.460299+0000", 2024-09-13T11:23:54.295 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.5727e-05 2024-09-13T11:23:54.295 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.296 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.296 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:23:54.296 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:38.460314+0000", 2024-09-13T11:23:54.296 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.5048000000000001e-05 2024-09-13T11:23:54.296 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.296 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.296 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:23:54.296 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:38.461323+0000", 2024-09-13T11:23:54.296 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0010089090000000001 2024-09-13T11:23:54.296 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:54.296 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:23:54.296 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:54.296 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.297 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.297 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "MOSDPGPush(3.1a 85/78 [PushOp(3:5dc85525:::benchmark_data_smithi177_31116_object11755:head, version: 49'723, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:5dc85525:::benchmark_data_smithi177_31116_object11755:head@49'723, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:23:54.297 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:42.422061+0000", 2024-09-13T11:23:54.297 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 11.837563659000001, 2024-09-13T11:23:54.297 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.52875617500000005, 2024-09-13T11:23:54.297 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:23:54.297 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:23:54.297 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-09-13T11:23:54.297 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:23:54.297 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.297 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:23:54.297 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.422061+0000", 2024-09-13T11:23:54.297 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:54.298 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.298 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.298 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:23:54.298 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.422061+0000", 2024-09-13T11:23:54.298 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:54.298 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.298 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.298 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:23:54.298 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.422064+0000", 2024-09-13T11:23:54.298 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.5409999999999999e-06 2024-09-13T11:23:54.298 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.298 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.298 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:23:54.299 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.422139+0000", 2024-09-13T11:23:54.299 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.5488999999999995e-05 2024-09-13T11:23:54.299 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.299 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.299 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:23:54.299 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.948781+0000", 2024-09-13T11:23:54.299 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.52664230999999995 2024-09-13T11:23:54.299 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.299 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.299 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:23:54.299 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.948841+0000", 2024-09-13T11:23:54.299 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.9450999999999997e-05 2024-09-13T11:23:54.299 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.299 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.300 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:23:54.300 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.949035+0000", 2024-09-13T11:23:54.300 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00019380600000000001 2024-09-13T11:23:54.300 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.300 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.300 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:23:54.300 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.949060+0000", 2024-09-13T11:23:54.300 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.5381999999999999e-05 2024-09-13T11:23:54.300 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.300 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.300 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:23:54.300 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.950817+0000", 2024-09-13T11:23:54.300 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0017571959999999999 2024-09-13T11:23:54.300 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:54.301 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:23:54.301 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:54.301 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.301 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.301 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "MOSDPGPush(3.1a 89/78 [PushOp(3:59270734:::benchmark_data_smithi177_31116_object20039:head, version: 54'1256, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:59270734:::benchmark_data_smithi177_31116_object20039:head@54'1256, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:23:54.301 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:46.607562+0000", 2024-09-13T11:23:54.301 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 7.6520628820000001, 2024-09-13T11:23:54.301 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.32123945300000001, 2024-09-13T11:23:54.301 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:23:54.301 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:23:54.301 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-09-13T11:23:54.301 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:23:54.301 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.302 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:23:54.302 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.607562+0000", 2024-09-13T11:23:54.302 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:54.302 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.302 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.302 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:23:54.302 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.607562+0000", 2024-09-13T11:23:54.302 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:54.302 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.302 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.302 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:23:54.303 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.607564+0000", 2024-09-13T11:23:54.303 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.8610000000000001e-06 2024-09-13T11:23:54.303 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.303 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.303 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:23:54.303 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.607663+0000", 2024-09-13T11:23:54.303 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.9210999999999996e-05 2024-09-13T11:23:54.303 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.303 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.303 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:23:54.303 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.925193+0000", 2024-09-13T11:23:54.303 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.317530586 2024-09-13T11:23:54.303 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.303 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.304 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:23:54.304 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.925250+0000", 2024-09-13T11:23:54.304 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.6925000000000001e-05 2024-09-13T11:23:54.304 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.304 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.304 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:23:54.304 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.927013+0000", 2024-09-13T11:23:54.304 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.001762853 2024-09-13T11:23:54.304 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.304 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.304 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:23:54.304 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.927067+0000", 2024-09-13T11:23:54.304 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.3601999999999999e-05 2024-09-13T11:23:54.305 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.305 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.305 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:23:54.305 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.928801+0000", 2024-09-13T11:23:54.305 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.001734415 2024-09-13T11:23:54.305 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:54.305 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:23:54.305 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:54.305 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.305 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.305 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "MOSDPGPush(3.1a 90/78 [PushOp(3:5d2269a4:::benchmark_data_smithi177_31116_object20650:head, version: 54'1284, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:5d2269a4:::benchmark_data_smithi177_31116_object20650:head@54'1284, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:23:54.305 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:47.156435+0000", 2024-09-13T11:23:54.305 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 7.103189253, 2024-09-13T11:23:54.306 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.098664977000000001, 2024-09-13T11:23:54.306 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:23:54.306 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:23:54.306 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-09-13T11:23:54.306 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:23:54.306 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.306 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:23:54.306 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:47.156435+0000", 2024-09-13T11:23:54.306 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:54.306 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.306 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.306 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:23:54.306 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:47.156435+0000", 2024-09-13T11:23:54.306 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:54.307 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.307 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.307 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:23:54.307 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:47.156438+0000", 2024-09-13T11:23:54.307 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.492e-06 2024-09-13T11:23:54.307 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.307 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.307 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:23:54.307 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:47.156508+0000", 2024-09-13T11:23:54.307 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.0364000000000006e-05 2024-09-13T11:23:54.307 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.307 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.307 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:23:54.308 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:47.254084+0000", 2024-09-13T11:23:54.308 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.097576044000000001 2024-09-13T11:23:54.308 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.308 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.308 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:23:54.308 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:47.254113+0000", 2024-09-13T11:23:54.308 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.8384000000000001e-05 2024-09-13T11:23:54.308 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.308 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.308 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:23:54.309 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:47.254246+0000", 2024-09-13T11:23:54.309 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000133662 2024-09-13T11:23:54.309 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.309 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.309 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:23:54.309 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:47.254259+0000", 2024-09-13T11:23:54.309 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.2673999999999999e-05 2024-09-13T11:23:54.309 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.309 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.309 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:23:54.309 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:47.255100+0000", 2024-09-13T11:23:54.309 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00084135700000000004 2024-09-13T11:23:54.309 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:54.309 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:23:54.310 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:54.310 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.310 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.310 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:64389 3.19 3:9e2bf991:::benchmark_data_smithi177_31116_object64388:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90)", 2024-09-13T11:23:54.310 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:47.195560+0000", 2024-09-13T11:23:54.310 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 7.0640648549999998, 2024-09-13T11:23:54.310 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.108379857, 2024-09-13T11:23:54.310 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:23:54.310 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:23:54.310 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:54.310 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:23:54.310 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:23:54.310 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:54.310 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 64389 2024-09-13T11:23:54.311 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.311 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:23:54.311 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.311 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:23:54.311 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:47.195560+0000", 2024-09-13T11:23:54.311 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:54.311 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.311 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.311 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:23:54.311 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:47.195560+0000", 2024-09-13T11:23:54.311 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:54.311 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.311 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.312 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:23:54.312 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:47.195561+0000", 2024-09-13T11:23:54.312 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.4300000000000001e-06 2024-09-13T11:23:54.312 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.312 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.312 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:23:54.312 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:47.195649+0000", 2024-09-13T11:23:54.312 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.7379000000000005e-05 2024-09-13T11:23:54.312 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.312 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.312 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:23:54.312 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:47.195663+0000", 2024-09-13T11:23:54.312 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.4012999999999999e-05 2024-09-13T11:23:54.313 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.313 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.313 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:23:54.313 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:47.195673+0000", 2024-09-13T11:23:54.313 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0045e-05 2024-09-13T11:23:54.313 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.313 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.313 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:23:54.313 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:47.195739+0000", 2024-09-13T11:23:54.313 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.6651999999999998e-05 2024-09-13T11:23:54.313 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.313 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.313 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:23:54.313 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:47.195908+0000", 2024-09-13T11:23:54.314 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00016875700000000001 2024-09-13T11:23:54.314 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.314 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.314 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1", 2024-09-13T11:23:54.314 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:47.196082+0000", 2024-09-13T11:23:54.314 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00017355199999999999 2024-09-13T11:23:54.314 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.314 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.314 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:23:54.314 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:47.197432+0000", 2024-09-13T11:23:54.314 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0013507700000000001 2024-09-13T11:23:54.314 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.314 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.314 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:54.315 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:47.303796+0000", 2024-09-13T11:23:54.315 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.106363729 2024-09-13T11:23:54.315 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.315 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.315 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:23:54.315 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:47.303875+0000", 2024-09-13T11:23:54.315 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.9092999999999996e-05 2024-09-13T11:23:54.315 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.315 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.315 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:23:54.315 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:47.303940+0000", 2024-09-13T11:23:54.315 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.4437000000000007e-05 2024-09-13T11:23:54.315 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:54.315 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:23:54.316 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:54.316 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.316 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.316 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:64474 3.19 3:9a522a61:::benchmark_data_smithi177_31116_object64473:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90)", 2024-09-13T11:23:54.316 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:47.318675+0000", 2024-09-13T11:23:54.316 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 6.9409494980000002, 2024-09-13T11:23:54.316 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.072523947000000005, 2024-09-13T11:23:54.316 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:23:54.316 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:23:54.316 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:54.316 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:23:54.316 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:23:54.316 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:54.317 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 64474 2024-09-13T11:23:54.317 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.317 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:23:54.317 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.317 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:23:54.317 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:47.318675+0000", 2024-09-13T11:23:54.317 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:54.317 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.318 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.318 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:23:54.318 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:47.318675+0000", 2024-09-13T11:23:54.318 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:54.318 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.318 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.318 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:23:54.318 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:47.318679+0000", 2024-09-13T11:23:54.318 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.8680000000000001e-06 2024-09-13T11:23:54.318 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.318 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.318 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:23:54.319 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:47.318746+0000", 2024-09-13T11:23:54.319 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.7046999999999997e-05 2024-09-13T11:23:54.319 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.319 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.319 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:23:54.319 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:47.318780+0000", 2024-09-13T11:23:54.319 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.3704000000000001e-05 2024-09-13T11:23:54.319 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.319 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.319 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:23:54.319 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:47.318803+0000", 2024-09-13T11:23:54.319 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.3308000000000001e-05 2024-09-13T11:23:54.319 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.320 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.320 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:23:54.320 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:47.318967+0000", 2024-09-13T11:23:54.320 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00016390799999999999 2024-09-13T11:23:54.320 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.320 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.320 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:23:54.320 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:47.319161+0000", 2024-09-13T11:23:54.320 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00019372200000000001 2024-09-13T11:23:54.320 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.320 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.320 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1", 2024-09-13T11:23:54.320 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:47.319265+0000", 2024-09-13T11:23:54.320 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000103842 2024-09-13T11:23:54.321 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.321 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.321 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:23:54.321 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:47.320987+0000", 2024-09-13T11:23:54.321 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0017222139999999999 2024-09-13T11:23:54.321 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.321 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.321 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:54.321 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:47.391118+0000", 2024-09-13T11:23:54.321 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.070131008999999994 2024-09-13T11:23:54.321 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.321 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.321 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:23:54.322 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:47.391162+0000", 2024-09-13T11:23:54.322 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.3807000000000003e-05 2024-09-13T11:23:54.322 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.322 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.322 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:23:54.322 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:47.391199+0000", 2024-09-13T11:23:54.322 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.7518000000000002e-05 2024-09-13T11:23:54.322 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:54.322 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:23:54.322 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:54.322 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.322 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.323 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:64694 3.19 3:9b336b44:::benchmark_data_smithi177_31116_object64693:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90)", 2024-09-13T11:23:54.323 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:47.542480+0000", 2024-09-13T11:23:54.323 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 6.7171447979999996, 2024-09-13T11:23:54.323 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.068176809000000005, 2024-09-13T11:23:54.323 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:23:54.323 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:23:54.323 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:54.323 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:23:54.323 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:23:54.323 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:54.323 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 64694 2024-09-13T11:23:54.323 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.323 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:23:54.324 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.324 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:23:54.324 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:47.542480+0000", 2024-09-13T11:23:54.324 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:54.324 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.324 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.324 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:23:54.324 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:47.542480+0000", 2024-09-13T11:23:54.324 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:54.324 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.324 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.324 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:23:54.324 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:47.542481+0000", 2024-09-13T11:23:54.325 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.4500000000000001e-06 2024-09-13T11:23:54.325 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.325 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.325 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:23:54.325 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:47.542557+0000", 2024-09-13T11:23:54.325 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.5320000000000004e-05 2024-09-13T11:23:54.325 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.325 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.325 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:23:54.325 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:47.542568+0000", 2024-09-13T11:23:54.325 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0895e-05 2024-09-13T11:23:54.325 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.325 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.325 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:23:54.326 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:47.542576+0000", 2024-09-13T11:23:54.326 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.1499999999999999e-06 2024-09-13T11:23:54.326 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.326 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.326 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:23:54.326 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:47.543375+0000", 2024-09-13T11:23:54.326 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00079929100000000004 2024-09-13T11:23:54.326 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.326 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.326 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:23:54.326 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:47.543479+0000", 2024-09-13T11:23:54.326 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000104318 2024-09-13T11:23:54.326 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.327 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.327 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1", 2024-09-13T11:23:54.327 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:47.543581+0000", 2024-09-13T11:23:54.327 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000101603 2024-09-13T11:23:54.327 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.327 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.327 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:23:54.327 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:47.545853+0000", 2024-09-13T11:23:54.327 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0022720449999999999 2024-09-13T11:23:54.327 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.327 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.327 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:54.328 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:47.610483+0000", 2024-09-13T11:23:54.328 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.06462996 2024-09-13T11:23:54.328 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.328 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.328 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:23:54.328 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:47.610589+0000", 2024-09-13T11:23:54.328 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000105901 2024-09-13T11:23:54.328 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.328 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.328 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:23:54.328 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:47.610657+0000", 2024-09-13T11:23:54.328 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.7875999999999999e-05 2024-09-13T11:23:54.328 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:54.329 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:23:54.329 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:54.329 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.329 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.329 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "MOSDPGPush(3.1a 91/78 [PushOp(3:5e5010d2:::benchmark_data_smithi177_31116_object25733:head, version: 58'1609, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:5e5010d2:::benchmark_data_smithi177_31116_object25733:head@58'1609, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:23:54.329 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:48.608485+0000", 2024-09-13T11:23:54.329 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 5.6511397639999998, 2024-09-13T11:23:54.329 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.35035599699999997, 2024-09-13T11:23:54.329 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:23:54.329 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:23:54.329 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-09-13T11:23:54.329 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:23:54.329 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.330 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:23:54.330 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:48.608485+0000", 2024-09-13T11:23:54.330 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:54.330 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.330 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.330 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:23:54.330 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:48.608485+0000", 2024-09-13T11:23:54.330 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:54.330 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.330 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.330 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:23:54.330 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:48.608488+0000", 2024-09-13T11:23:54.330 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.9399999999999998e-06 2024-09-13T11:23:54.331 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.331 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.331 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:23:54.331 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:48.608552+0000", 2024-09-13T11:23:54.331 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.3731000000000004e-05 2024-09-13T11:23:54.331 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.331 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.331 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:23:54.331 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:48.957330+0000", 2024-09-13T11:23:54.331 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.34877806099999997 2024-09-13T11:23:54.331 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.331 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.331 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:23:54.331 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:48.957361+0000", 2024-09-13T11:23:54.332 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.0852000000000002e-05 2024-09-13T11:23:54.332 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.332 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.332 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:23:54.332 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:48.957544+0000", 2024-09-13T11:23:54.332 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000183537 2024-09-13T11:23:54.332 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.332 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.332 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:23:54.332 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:48.957582+0000", 2024-09-13T11:23:54.332 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.7472999999999998e-05 2024-09-13T11:23:54.332 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.332 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.332 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:23:54.333 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:48.958841+0000", 2024-09-13T11:23:54.333 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0012594030000000001 2024-09-13T11:23:54.333 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:54.333 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:23:54.333 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:54.333 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.333 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.333 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "MOSDPGPush(3.1a 92/78 [PushOp(3:5d12edf6:::benchmark_data_smithi177_31116_object26425:head, version: 58'1647, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:5d12edf6:::benchmark_data_smithi177_31116_object26425:head@58'1647, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:23:54.333 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:49.221652+0000", 2024-09-13T11:23:54.333 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 5.0379727909999996, 2024-09-13T11:23:54.333 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.097120007999999994, 2024-09-13T11:23:54.333 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:23:54.333 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:23:54.334 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-09-13T11:23:54.334 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:23:54.334 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.334 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:23:54.334 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:49.221652+0000", 2024-09-13T11:23:54.334 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:54.334 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.334 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.334 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:23:54.334 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:49.221652+0000", 2024-09-13T11:23:54.334 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:54.334 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.334 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.334 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:23:54.335 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:49.221653+0000", 2024-09-13T11:23:54.335 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.279e-06 2024-09-13T11:23:54.335 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.335 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.335 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:23:54.335 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:49.221719+0000", 2024-09-13T11:23:54.335 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.5432999999999995e-05 2024-09-13T11:23:54.335 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.335 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.335 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:23:54.335 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:49.317753+0000", 2024-09-13T11:23:54.335 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.096033952000000006 2024-09-13T11:23:54.335 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.336 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.336 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:23:54.336 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:49.317774+0000", 2024-09-13T11:23:54.336 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.1234999999999999e-05 2024-09-13T11:23:54.336 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.336 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.336 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:23:54.336 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:49.317868+0000", 2024-09-13T11:23:54.336 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.4630999999999996e-05 2024-09-13T11:23:54.336 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.336 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.336 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:23:54.336 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:49.317878+0000", 2024-09-13T11:23:54.336 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0008e-05 2024-09-13T11:23:54.337 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.337 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.337 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:23:54.337 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:49.318772+0000", 2024-09-13T11:23:54.337 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00089347000000000005 2024-09-13T11:23:54.337 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:54.337 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:23:54.337 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:54.337 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.337 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.337 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "MOSDPGPush(3.1a 95/78 [PushOp(3:5f9270ec:::benchmark_data_smithi177_31116_object33007:head, version: 60'2076, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:5f9270ec:::benchmark_data_smithi177_31116_object33007:head@60'2076, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:23:54.337 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:52.817095+0000", 2024-09-13T11:23:54.337 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 1.4425300619999999, 2024-09-13T11:23:54.337 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.097724647999999997, 2024-09-13T11:23:54.338 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:23:54.338 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:23:54.338 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-09-13T11:23:54.338 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:23:54.338 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.338 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:23:54.338 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:52.817095+0000", 2024-09-13T11:23:54.338 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:54.338 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.338 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.338 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:23:54.338 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:52.817095+0000", 2024-09-13T11:23:54.338 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:54.338 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.339 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.339 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:23:54.339 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:52.817097+0000", 2024-09-13T11:23:54.339 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.4899999999999999e-06 2024-09-13T11:23:54.339 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.339 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.339 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:23:54.339 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:52.817175+0000", 2024-09-13T11:23:54.339 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.8320999999999998e-05 2024-09-13T11:23:54.339 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.339 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.339 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:23:54.339 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:52.913313+0000", 2024-09-13T11:23:54.339 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.096137632000000001 2024-09-13T11:23:54.340 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.340 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.340 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:23:54.340 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:52.913346+0000", 2024-09-13T11:23:54.340 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.2835e-05 2024-09-13T11:23:54.340 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.340 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.340 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:23:54.340 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:52.913580+0000", 2024-09-13T11:23:54.340 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000234579 2024-09-13T11:23:54.340 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.340 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.340 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:23:54.340 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:52.913598+0000", 2024-09-13T11:23:54.341 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.7447000000000001e-05 2024-09-13T11:23:54.341 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.341 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.341 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:23:54.341 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:52.914819+0000", 2024-09-13T11:23:54.341 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0012213440000000001 2024-09-13T11:23:54.341 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:54.341 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:23:54.341 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:54.341 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.341 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.341 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop(client.4665.0:70536 3.1a e95/78 3:5b699d68:::benchmark_data_smithi177_31116_object70535:head v 95'3856, mlcod=60'2077)", 2024-09-13T11:23:54.341 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:53.006730+0000", 2024-09-13T11:23:54.342 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 1.2528950409999999, 2024-09-13T11:23:54.342 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.068965369999999998, 2024-09-13T11:23:54.342 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:23:54.342 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:23:54.342 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:54.342 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:23:54.342 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.342 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:23:54.342 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:53.006730+0000", 2024-09-13T11:23:54.342 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:54.342 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.342 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.342 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:23:54.342 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:53.006730+0000", 2024-09-13T11:23:54.343 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:54.343 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.343 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.343 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:23:54.343 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:53.006732+0000", 2024-09-13T11:23:54.343 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.295e-06 2024-09-13T11:23:54.343 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.343 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.343 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:23:54.343 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:53.006790+0000", 2024-09-13T11:23:54.343 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.7606999999999999e-05 2024-09-13T11:23:54.343 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.343 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.343 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:23:54.344 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:53.070743+0000", 2024-09-13T11:23:54.344 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.063953768999999994 2024-09-13T11:23:54.344 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.344 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.344 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:23:54.344 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:53.070755+0000", 2024-09-13T11:23:54.344 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.1197e-05 2024-09-13T11:23:54.344 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.344 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.344 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:23:54.344 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:53.070867+0000", 2024-09-13T11:23:54.344 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000112191 2024-09-13T11:23:54.345 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.345 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.345 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:23:54.345 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:53.070917+0000", 2024-09-13T11:23:54.345 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.9861000000000003e-05 2024-09-13T11:23:54.345 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.345 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.345 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:23:54.345 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:53.075646+0000", 2024-09-13T11:23:54.345 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0047293810000000004 2024-09-13T11:23:54.345 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.345 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.345 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:23:54.345 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:53.075695+0000", 2024-09-13T11:23:54.346 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.9069000000000001e-05 2024-09-13T11:23:54.346 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:54.346 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:23:54.346 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:54.346 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.346 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.346 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "MOSDPGPush(3.1a 95/78 [PushOp(3:58a56058:::benchmark_data_smithi177_31116_object33115:head, version: 60'2079, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:58a56058:::benchmark_data_smithi177_31116_object33115:head@60'2079, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:23:54.346 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:53.080252+0000", 2024-09-13T11:23:54.346 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 1.1793729399999999, 2024-09-13T11:23:54.346 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.10589196200000001, 2024-09-13T11:23:54.346 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:23:54.346 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:23:54.346 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-09-13T11:23:54.347 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:23:54.347 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.347 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:23:54.347 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:53.080252+0000", 2024-09-13T11:23:54.347 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:54.347 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.347 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.347 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:23:54.347 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:53.080252+0000", 2024-09-13T11:23:54.347 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:54.347 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.347 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.347 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:23:54.347 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:53.080254+0000", 2024-09-13T11:23:54.348 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.2639999999999998e-06 2024-09-13T11:23:54.348 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.348 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.348 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:23:54.348 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:53.080316+0000", 2024-09-13T11:23:54.348 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.2283e-05 2024-09-13T11:23:54.348 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.348 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.348 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:23:54.348 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:53.183273+0000", 2024-09-13T11:23:54.348 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.102956333 2024-09-13T11:23:54.348 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.348 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.349 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:23:54.349 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:53.183295+0000", 2024-09-13T11:23:54.349 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.2459e-05 2024-09-13T11:23:54.349 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.349 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.349 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:23:54.349 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:53.185153+0000", 2024-09-13T11:23:54.349 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.001857893 2024-09-13T11:23:54.349 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.349 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.349 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:23:54.349 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:53.185177+0000", 2024-09-13T11:23:54.349 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.3923000000000001e-05 2024-09-13T11:23:54.349 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.350 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.350 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:23:54.350 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:53.186144+0000", 2024-09-13T11:23:54.350 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00096680699999999997 2024-09-13T11:23:54.350 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:54.350 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:23:54.350 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:54.350 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.350 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.350 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:71408 3.19 3:9c34ac14:::benchmark_data_smithi177_31116_object71407:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96)", 2024-09-13T11:23:54.350 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:53.764940+0000", 2024-09-13T11:23:54.350 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 0.49468491599999997, 2024-09-13T11:23:54.350 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.068045372000000007, 2024-09-13T11:23:54.350 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:23:54.351 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:23:54.351 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:54.351 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:23:54.351 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:23:54.351 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:54.351 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 71408 2024-09-13T11:23:54.351 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.351 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:23:54.351 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.351 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:23:54.351 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:53.764940+0000", 2024-09-13T11:23:54.351 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:54.351 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.351 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.352 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:23:54.352 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:53.764940+0000", 2024-09-13T11:23:54.352 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:54.352 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.352 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.352 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:23:54.352 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:53.764942+0000", 2024-09-13T11:23:54.352 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.221e-06 2024-09-13T11:23:54.352 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.352 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.352 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:23:54.352 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:53.765047+0000", 2024-09-13T11:23:54.352 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00010527900000000001 2024-09-13T11:23:54.352 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.353 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.353 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:23:54.353 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:53.765061+0000", 2024-09-13T11:23:54.353 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.3474e-05 2024-09-13T11:23:54.353 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.353 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.353 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:23:54.353 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:53.765068+0000", 2024-09-13T11:23:54.353 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.7819999999999996e-06 2024-09-13T11:23:54.353 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.353 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.353 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:23:54.353 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:53.765135+0000", 2024-09-13T11:23:54.354 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.7922999999999993e-05 2024-09-13T11:23:54.354 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.354 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.354 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:23:54.354 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:53.765278+0000", 2024-09-13T11:23:54.354 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000142516 2024-09-13T11:23:54.354 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.354 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.354 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1", 2024-09-13T11:23:54.354 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:53.765446+0000", 2024-09-13T11:23:54.354 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000167677 2024-09-13T11:23:54.354 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.354 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.355 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:23:54.355 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:53.767027+0000", 2024-09-13T11:23:54.355 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.001581678 2024-09-13T11:23:54.355 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.355 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.355 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:54.355 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:53.832868+0000", 2024-09-13T11:23:54.355 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.065840746000000006 2024-09-13T11:23:54.355 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.355 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.355 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:23:54.355 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:53.832937+0000", 2024-09-13T11:23:54.355 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.8607000000000002e-05 2024-09-13T11:23:54.356 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.356 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.356 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:23:54.356 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:53.832985+0000", 2024-09-13T11:23:54.356 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.8469e-05 2024-09-13T11:23:54.356 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:54.356 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:23:54.356 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:54.356 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.356 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.356 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:71611 3.19 3:9d9fa5ff:::benchmark_data_smithi177_31116_object71610:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96)", 2024-09-13T11:23:54.356 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:53.892604+0000", 2024-09-13T11:23:54.356 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 0.36702074299999998, 2024-09-13T11:23:54.356 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.071128778000000004, 2024-09-13T11:23:54.357 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:23:54.357 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:23:54.357 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:54.357 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:23:54.357 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:23:54.357 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:54.357 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 71611 2024-09-13T11:23:54.357 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.357 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:23:54.357 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.357 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:23:54.357 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:53.892604+0000", 2024-09-13T11:23:54.357 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:54.358 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.358 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.358 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:23:54.358 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:53.892604+0000", 2024-09-13T11:23:54.358 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:54.358 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.358 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.358 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:23:54.358 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:53.892615+0000", 2024-09-13T11:23:54.358 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.1439e-05 2024-09-13T11:23:54.358 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.358 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.358 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:23:54.359 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:53.892654+0000", 2024-09-13T11:23:54.359 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.8151000000000001e-05 2024-09-13T11:23:54.359 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.359 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.359 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:23:54.359 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:53.892668+0000", 2024-09-13T11:23:54.359 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.4939e-05 2024-09-13T11:23:54.359 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.359 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.359 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:23:54.359 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:53.892678+0000", 2024-09-13T11:23:54.359 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.7589999999999992e-06 2024-09-13T11:23:54.359 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.359 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.360 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:23:54.360 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:53.892730+0000", 2024-09-13T11:23:54.360 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.2206000000000001e-05 2024-09-13T11:23:54.360 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.360 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.360 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:23:54.360 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:53.892841+0000", 2024-09-13T11:23:54.360 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000110429 2024-09-13T11:23:54.360 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.360 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.360 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1", 2024-09-13T11:23:54.360 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:53.892935+0000", 2024-09-13T11:23:54.360 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.4069999999999999e-05 2024-09-13T11:23:54.361 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.361 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.361 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:23:54.361 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:53.894292+0000", 2024-09-13T11:23:54.361 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0013572370000000001 2024-09-13T11:23:54.361 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.361 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.361 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:54.361 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:53.963636+0000", 2024-09-13T11:23:54.361 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.069343658000000002 2024-09-13T11:23:54.361 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.362 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.362 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:23:54.362 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:53.963692+0000", 2024-09-13T11:23:54.362 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.5699000000000003e-05 2024-09-13T11:23:54.362 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.362 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.362 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:23:54.362 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:53.963733+0000", 2024-09-13T11:23:54.362 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.1190999999999999e-05 2024-09-13T11:23:54.362 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:54.362 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:23:54.362 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:54.362 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:54.362 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:23:54.363 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-09-13T11:23:54.363 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T11:23:54.365 INFO:tasks.ceph.osd.11.smithi177.stderr:2024-09-13T11:23:54.356+0000 7f24792c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26580) UID: 0 2024-09-13T11:23:54.365 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_ops_in_flight 2024-09-13T11:23:54.465 INFO:tasks.ceph.osd.14.smithi162.stderr:2024-09-13T11:23:54.461+0000 7f235c0db640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26319) UID: 0 2024-09-13T11:23:54.552 INFO:teuthology.orchestra.run.smithi001.stdout:osd.0: {} 2024-09-13T11:23:54.553 INFO:teuthology.orchestra.run.smithi001.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T11:23:54.554 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-09-13T11:23:54.554 INFO:teuthology.orchestra.run.smithi092.stdout: "ops": [ 2024-09-13T11:23:54.554 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.554 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:72272 3.7 3:e1b29c8c:::benchmark_data_smithi177_31116_object72271:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2024-09-13T11:23:54.555 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:54.534904+0000", 2024-09-13T11:23:54.555 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 0.017728351999999999, 2024-09-13T11:23:54.555 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.017762474, 2024-09-13T11:23:54.555 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:23:54.555 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:23:54.555 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "waiting for sub ops", 2024-09-13T11:23:54.556 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:23:54.556 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:23:54.556 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:54.556 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 72272 2024-09-13T11:23:54.556 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.556 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:23:54.557 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.557 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:23:54.557 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:54.534904+0000", 2024-09-13T11:23:54.557 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:54.557 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.557 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.558 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:23:54.558 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:54.534904+0000", 2024-09-13T11:23:54.558 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:54.558 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.558 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.558 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:23:54.559 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:54.534906+0000", 2024-09-13T11:23:54.559 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.357e-06 2024-09-13T11:23:54.559 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.559 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.559 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:23:54.560 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:54.534940+0000", 2024-09-13T11:23:54.560 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.3985000000000001e-05 2024-09-13T11:23:54.560 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.560 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.560 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:23:54.560 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:54.534954+0000", 2024-09-13T11:23:54.560 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.4052e-05 2024-09-13T11:23:54.560 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.560 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.561 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:23:54.561 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:54.534963+0000", 2024-09-13T11:23:54.561 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.2760000000000006e-06 2024-09-13T11:23:54.561 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.561 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.561 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:23:54.562 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:54.535017+0000", 2024-09-13T11:23:54.562 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.3482000000000001e-05 2024-09-13T11:23:54.562 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.562 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.562 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:23:54.562 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:54.535124+0000", 2024-09-13T11:23:54.563 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000106993 2024-09-13T11:23:54.563 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.563 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.563 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 8", 2024-09-13T11:23:54.563 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:54.535214+0000", 2024-09-13T11:23:54.563 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.9851000000000004e-05 2024-09-13T11:23:54.564 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.564 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.564 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:23:54.564 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:54.537065+0000", 2024-09-13T11:23:54.564 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0018511300000000001 2024-09-13T11:23:54.564 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:54.565 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:23:54.565 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:54.565 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:54.565 INFO:teuthology.orchestra.run.smithi092.stdout: ], 2024-09-13T11:23:54.565 INFO:teuthology.orchestra.run.smithi092.stdout: "num_ops": 1 2024-09-13T11:23:54.565 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-09-13T11:23:54.567 INFO:tasks.ceph.osd.12.smithi001.stderr:2024-09-13T11:23:54.560+0000 7f892c6e6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26314) UID: 0 2024-09-13T11:23:54.574 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_blocked_ops 2024-09-13T11:23:54.581 INFO:teuthology.orchestra.run.smithi001.stdout:osd.1: {} 2024-09-13T11:23:54.581 INFO:teuthology.orchestra.run.smithi001.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T11:23:54.609 INFO:teuthology.orchestra.run.smithi001.stdout:osd.2: {} 2024-09-13T11:23:54.610 INFO:teuthology.orchestra.run.smithi001.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T11:23:54.638 INFO:teuthology.orchestra.run.smithi001.stdout:osd.3: {} 2024-09-13T11:23:54.638 INFO:teuthology.orchestra.run.smithi001.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T11:23:54.664 INFO:teuthology.orchestra.run.smithi001.stdout:osd.4: {} 2024-09-13T11:23:54.664 INFO:teuthology.orchestra.run.smithi001.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T11:23:54.668 INFO:tasks.ceph.osd.13.smithi092.stderr:2024-09-13T11:23:54.663+0000 7f3e5afea640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26350) UID: 0 2024-09-13T11:23:54.701 INFO:teuthology.orchestra.run.smithi001.stdout:osd.5: {} 2024-09-13T11:23:54.701 INFO:teuthology.orchestra.run.smithi001.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T11:23:54.711 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-09-13T11:23:54.711 INFO:teuthology.orchestra.run.smithi092.stdout: "ops": [], 2024-09-13T11:23:54.711 INFO:teuthology.orchestra.run.smithi092.stdout: "complaint_time": 30, 2024-09-13T11:23:54.712 INFO:teuthology.orchestra.run.smithi092.stdout: "num_blocked_ops": 0 2024-09-13T11:23:54.712 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-09-13T11:23:54.729 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_historic_ops 2024-09-13T11:23:54.768 INFO:tasks.ceph.osd.2.smithi162.stderr:2024-09-13T11:23:54.765+0000 7fe92b35a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26313) UID: 0 2024-09-13T11:23:54.869 INFO:tasks.ceph.osd.2.smithi162.stderr:2024-09-13T11:23:54.865+0000 7fe92b35a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26313) UID: 0 2024-09-13T11:23:54.913 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-09-13T11:23:54.913 INFO:teuthology.orchestra.run.smithi092.stdout: "size": 20, 2024-09-13T11:23:54.913 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 600, 2024-09-13T11:23:54.913 INFO:teuthology.orchestra.run.smithi092.stdout: "ops": [ 2024-09-13T11:23:54.914 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.914 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:79 3.1 3:841df5b5:::benchmark_data_smithi177_31116_object78:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-09-13T11:23:54.914 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:27.056083+0000", 2024-09-13T11:23:54.914 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 87.854286688000002, 2024-09-13T11:23:54.914 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.89469730199999997, 2024-09-13T11:23:54.914 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:23:54.914 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:23:54.914 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:54.914 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:23:54.914 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:23:54.914 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:54.915 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 79 2024-09-13T11:23:54.915 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.915 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:23:54.915 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.915 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:23:54.915 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.056083+0000", 2024-09-13T11:23:54.915 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:54.915 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.915 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.915 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:23:54.915 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.056083+0000", 2024-09-13T11:23:54.915 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:54.916 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.916 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.916 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:23:54.916 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.056087+0000", 2024-09-13T11:23:54.916 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.0659999999999997e-06 2024-09-13T11:23:54.916 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.916 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.916 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:23:54.916 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.056148+0000", 2024-09-13T11:23:54.916 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.1376000000000003e-05 2024-09-13T11:23:54.917 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.917 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.917 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:23:54.917 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.056180+0000", 2024-09-13T11:23:54.917 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.2370999999999999e-05 2024-09-13T11:23:54.917 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.918 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.918 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:23:54.918 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.056195+0000", 2024-09-13T11:23:54.918 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.5119e-05 2024-09-13T11:23:54.918 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.918 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.918 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:23:54.918 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.056304+0000", 2024-09-13T11:23:54.918 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000108365 2024-09-13T11:23:54.918 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.918 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.918 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:23:54.918 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.056478+0000", 2024-09-13T11:23:54.918 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000174542 2024-09-13T11:23:54.919 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.919 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.919 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 11", 2024-09-13T11:23:54.919 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.056690+0000", 2024-09-13T11:23:54.919 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00021167500000000001 2024-09-13T11:23:54.919 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.919 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.919 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:23:54.919 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.058063+0000", 2024-09-13T11:23:54.919 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0013731819999999999 2024-09-13T11:23:54.919 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.919 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.919 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:54.919 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.950477+0000", 2024-09-13T11:23:54.920 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.89241369400000004 2024-09-13T11:23:54.920 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.920 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.920 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:23:54.920 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.950652+0000", 2024-09-13T11:23:54.920 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00017526400000000001 2024-09-13T11:23:54.920 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.920 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.920 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:23:54.920 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.950780+0000", 2024-09-13T11:23:54.920 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00012764800000000001 2024-09-13T11:23:54.920 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:54.921 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:23:54.921 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:54.921 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.921 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.921 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:7 3.b 3:de78393b:::benchmark_data_smithi177_31116_object6:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T11:23:54.921 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:48.313952+0000", 2024-09-13T11:23:54.921 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 66.596416938999994, 2024-09-13T11:23:54.921 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.86421288600000001, 2024-09-13T11:23:54.921 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:23:54.921 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:23:54.921 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:54.921 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:23:54.921 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:23:54.922 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:54.922 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 7 2024-09-13T11:23:54.922 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.922 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:23:54.922 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.922 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:23:54.922 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.313952+0000", 2024-09-13T11:23:54.922 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:54.922 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.922 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.922 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:23:54.922 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.313952+0000", 2024-09-13T11:23:54.922 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:54.922 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.922 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.923 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:23:54.923 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.313957+0000", 2024-09-13T11:23:54.923 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.3599999999999998e-06 2024-09-13T11:23:54.923 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.923 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.923 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:23:54.923 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314043+0000", 2024-09-13T11:23:54.923 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.6766000000000002e-05 2024-09-13T11:23:54.923 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.923 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.923 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:23:54.923 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314065+0000", 2024-09-13T11:23:54.923 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.1416e-05 2024-09-13T11:23:54.924 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.925 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.925 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:23:54.925 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314080+0000", 2024-09-13T11:23:54.925 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.5519000000000002e-05 2024-09-13T11:23:54.925 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.925 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.925 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:23:54.925 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315478+0000", 2024-09-13T11:23:54.925 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.001397765 2024-09-13T11:23:54.925 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.925 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.925 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-09-13T11:23:54.926 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315507+0000", 2024-09-13T11:23:54.926 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.9128999999999998e-05 2024-09-13T11:23:54.926 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.926 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.926 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:23:54.926 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.167978+0000", 2024-09-13T11:23:54.926 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.85247068400000003 2024-09-13T11:23:54.927 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.927 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.927 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:23:54.927 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.168307+0000", 2024-09-13T11:23:54.927 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00032886699999999999 2024-09-13T11:23:54.927 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.927 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.927 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1", 2024-09-13T11:23:54.927 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.168647+0000", 2024-09-13T11:23:54.927 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00033998399999999999 2024-09-13T11:23:54.927 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.927 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.927 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:54.928 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.178011+0000", 2024-09-13T11:23:54.928 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0093639169999999994 2024-09-13T11:23:54.928 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.928 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.928 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:23:54.928 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.178026+0000", 2024-09-13T11:23:54.928 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.5274999999999999e-05 2024-09-13T11:23:54.928 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.928 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.929 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:23:54.929 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.178080+0000", 2024-09-13T11:23:54.929 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.3601999999999999e-05 2024-09-13T11:23:54.929 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.929 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.929 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:23:54.929 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.178165+0000", 2024-09-13T11:23:54.929 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.5601999999999998e-05 2024-09-13T11:23:54.929 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:54.929 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:23:54.929 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:54.929 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.929 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.929 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:9 3.b 3:dafadff6:::benchmark_data_smithi177_31116_object8:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T11:23:54.930 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:48.314114+0000", 2024-09-13T11:23:54.930 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 66.596255124999999, 2024-09-13T11:23:54.930 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.86613615899999996, 2024-09-13T11:23:54.930 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:23:54.930 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:23:54.930 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:54.930 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:23:54.930 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:23:54.930 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:54.930 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 9 2024-09-13T11:23:54.930 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.930 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:23:54.931 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.931 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:23:54.931 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314114+0000", 2024-09-13T11:23:54.931 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:54.931 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.931 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.931 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:23:54.931 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314114+0000", 2024-09-13T11:23:54.931 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:54.931 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.931 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.931 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:23:54.931 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314117+0000", 2024-09-13T11:23:54.932 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.3860000000000001e-06 2024-09-13T11:23:54.932 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.932 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.932 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:23:54.932 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314215+0000", 2024-09-13T11:23:54.932 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.8900000000000005e-05 2024-09-13T11:23:54.932 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.932 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.933 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:23:54.933 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314229+0000", 2024-09-13T11:23:54.933 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.3444999999999999e-05 2024-09-13T11:23:54.933 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.933 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.933 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:23:54.933 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314239+0000", 2024-09-13T11:23:54.933 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.8470000000000008e-06 2024-09-13T11:23:54.933 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.933 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.933 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:23:54.933 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315648+0000", 2024-09-13T11:23:54.934 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.001409291 2024-09-13T11:23:54.934 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.934 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.934 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-09-13T11:23:54.934 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315671+0000", 2024-09-13T11:23:54.934 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.3169000000000001e-05 2024-09-13T11:23:54.934 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.934 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.934 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:23:54.934 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.176036+0000", 2024-09-13T11:23:54.934 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.86036497499999998 2024-09-13T11:23:54.934 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.934 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.935 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:23:54.935 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.176189+0000", 2024-09-13T11:23:54.935 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00015271100000000001 2024-09-13T11:23:54.935 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.935 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.935 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1", 2024-09-13T11:23:54.935 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.176339+0000", 2024-09-13T11:23:54.935 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00015057499999999999 2024-09-13T11:23:54.935 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.935 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.935 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:54.935 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.180148+0000", 2024-09-13T11:23:54.935 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.003808891 2024-09-13T11:23:54.935 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.936 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.936 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:23:54.936 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.180168+0000", 2024-09-13T11:23:54.936 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.9507000000000001e-05 2024-09-13T11:23:54.936 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.936 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.936 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:23:54.936 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.180213+0000", 2024-09-13T11:23:54.936 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.5504999999999999e-05 2024-09-13T11:23:54.936 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.936 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.936 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:23:54.937 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.180250+0000", 2024-09-13T11:23:54.937 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.6956999999999999e-05 2024-09-13T11:23:54.937 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:54.937 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:23:54.937 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:54.937 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.937 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.937 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:10 3.b 3:dbe028a8:::benchmark_data_smithi177_31116_object9:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T11:23:54.937 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:48.314259+0000", 2024-09-13T11:23:54.937 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 66.596109995000006, 2024-09-13T11:23:54.937 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.86637313599999999, 2024-09-13T11:23:54.937 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:23:54.937 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:23:54.937 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:54.938 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:23:54.938 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:23:54.938 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:54.938 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 10 2024-09-13T11:23:54.938 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.938 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:23:54.938 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.938 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:23:54.938 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314259+0000", 2024-09-13T11:23:54.938 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:54.938 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.938 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.938 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:23:54.939 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314259+0000", 2024-09-13T11:23:54.939 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:54.939 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.939 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.939 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:23:54.939 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314261+0000", 2024-09-13T11:23:54.939 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.1500000000000002e-06 2024-09-13T11:23:54.939 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.939 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.939 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:23:54.939 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314349+0000", 2024-09-13T11:23:54.939 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.8070000000000002e-05 2024-09-13T11:23:54.939 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.939 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.940 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:23:54.940 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314365+0000", 2024-09-13T11:23:54.940 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.5835999999999999e-05 2024-09-13T11:23:54.940 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.940 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.940 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:23:54.940 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314377+0000", 2024-09-13T11:23:54.940 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.1589e-05 2024-09-13T11:23:54.940 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.940 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.940 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:23:54.940 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315788+0000", 2024-09-13T11:23:54.940 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0014109579999999999 2024-09-13T11:23:54.941 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.941 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.941 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-09-13T11:23:54.941 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315809+0000", 2024-09-13T11:23:54.941 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.0985999999999999e-05 2024-09-13T11:23:54.941 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.941 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.941 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:23:54.941 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.176822+0000", 2024-09-13T11:23:54.941 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.86101310799999997 2024-09-13T11:23:54.942 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.942 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.942 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:23:54.942 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.177021+0000", 2024-09-13T11:23:54.942 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00019911199999999999 2024-09-13T11:23:54.942 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.942 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.942 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1", 2024-09-13T11:23:54.942 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.177226+0000", 2024-09-13T11:23:54.942 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00020516600000000001 2024-09-13T11:23:54.942 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.943 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.943 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:23:54.943 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.180264+0000", 2024-09-13T11:23:54.943 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0030381739999999998 2024-09-13T11:23:54.943 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.943 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.943 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:54.943 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.180387+0000", 2024-09-13T11:23:54.943 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00012257699999999999 2024-09-13T11:23:54.943 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.943 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.943 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:23:54.944 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.180518+0000", 2024-09-13T11:23:54.944 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00013135800000000001 2024-09-13T11:23:54.944 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.944 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.944 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:23:54.944 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.180632+0000", 2024-09-13T11:23:54.944 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000114052 2024-09-13T11:23:54.944 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:54.944 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:23:54.944 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:54.944 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.944 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.944 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:39 3.b 3:d91854b3:::benchmark_data_smithi177_31116_object38:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T11:23:54.944 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:48.314393+0000", 2024-09-13T11:23:54.945 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 66.595976453000006, 2024-09-13T11:23:54.945 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.86850121700000005, 2024-09-13T11:23:54.945 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:23:54.945 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:23:54.945 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:54.945 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:23:54.945 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:23:54.945 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:54.945 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 39 2024-09-13T11:23:54.945 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.945 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:23:54.945 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.945 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:23:54.946 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314393+0000", 2024-09-13T11:23:54.946 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:54.946 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.946 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.946 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:23:54.946 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314393+0000", 2024-09-13T11:23:54.946 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:54.946 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.946 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.946 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:23:54.946 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314394+0000", 2024-09-13T11:23:54.946 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.513e-06 2024-09-13T11:23:54.946 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.946 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.947 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:23:54.947 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314463+0000", 2024-09-13T11:23:54.947 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.8261000000000003e-05 2024-09-13T11:23:54.947 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.947 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.947 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:23:54.947 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314477+0000", 2024-09-13T11:23:54.947 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.4619e-05 2024-09-13T11:23:54.947 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.947 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.947 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:23:54.947 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314485+0000", 2024-09-13T11:23:54.947 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.1699999999999997e-06 2024-09-13T11:23:54.947 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.948 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.948 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:23:54.948 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315917+0000", 2024-09-13T11:23:54.948 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0014316089999999999 2024-09-13T11:23:54.948 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.948 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.948 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-09-13T11:23:54.948 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315933+0000", 2024-09-13T11:23:54.948 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.5999999999999999e-05 2024-09-13T11:23:54.948 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.948 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.948 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:23:54.948 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.178205+0000", 2024-09-13T11:23:54.949 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.86227247100000004 2024-09-13T11:23:54.949 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.949 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.949 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:23:54.949 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.178316+0000", 2024-09-13T11:23:54.949 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000110788 2024-09-13T11:23:54.949 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.949 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.949 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1", 2024-09-13T11:23:54.949 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.178448+0000", 2024-09-13T11:23:54.949 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000132056 2024-09-13T11:23:54.949 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.949 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.950 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:23:54.950 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.182721+0000", 2024-09-13T11:23:54.950 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0042729780000000002 2024-09-13T11:23:54.950 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.950 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.950 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:54.950 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.182801+0000", 2024-09-13T11:23:54.950 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.0209000000000004e-05 2024-09-13T11:23:54.950 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.950 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.950 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:23:54.950 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.182855+0000", 2024-09-13T11:23:54.950 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.3749000000000003e-05 2024-09-13T11:23:54.951 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.951 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.951 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:23:54.951 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.182894+0000", 2024-09-13T11:23:54.951 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.8794000000000002e-05 2024-09-13T11:23:54.951 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:54.951 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:23:54.951 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:54.951 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.951 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.951 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:50 3.b 3:d1c144b8:::benchmark_data_smithi177_31116_object49:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T11:23:54.951 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:48.314498+0000", 2024-09-13T11:23:54.952 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 66.595871037999999, 2024-09-13T11:23:54.952 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.86878673299999998, 2024-09-13T11:23:54.952 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:23:54.952 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:23:54.952 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:54.952 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:23:54.952 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:23:54.952 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:54.952 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 50 2024-09-13T11:23:54.952 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.952 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:23:54.952 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.952 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:23:54.952 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314498+0000", 2024-09-13T11:23:54.953 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:54.953 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.953 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.953 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:23:54.953 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314498+0000", 2024-09-13T11:23:54.953 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:54.953 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.953 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.953 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:23:54.953 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314499+0000", 2024-09-13T11:23:54.953 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.1969999999999999e-06 2024-09-13T11:23:54.953 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.953 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.954 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:23:54.954 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314565+0000", 2024-09-13T11:23:54.954 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.5628000000000003e-05 2024-09-13T11:23:54.954 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.954 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.954 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:23:54.954 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314579+0000", 2024-09-13T11:23:54.954 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.4222e-05 2024-09-13T11:23:54.954 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.954 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.954 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:23:54.954 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314590+0000", 2024-09-13T11:23:54.954 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.1240000000000001e-05 2024-09-13T11:23:54.955 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.955 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.955 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:23:54.955 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316060+0000", 2024-09-13T11:23:54.955 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.001469515 2024-09-13T11:23:54.955 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.955 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.955 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-09-13T11:23:54.955 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316076+0000", 2024-09-13T11:23:54.955 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.5608999999999999e-05 2024-09-13T11:23:54.955 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.955 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.955 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:23:54.955 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.178857+0000", 2024-09-13T11:23:54.956 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.86278099900000005 2024-09-13T11:23:54.956 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.956 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.956 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:23:54.956 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.178961+0000", 2024-09-13T11:23:54.956 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000104664 2024-09-13T11:23:54.956 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.956 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.956 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1", 2024-09-13T11:23:54.956 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.179123+0000", 2024-09-13T11:23:54.956 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000161738 2024-09-13T11:23:54.956 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.956 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.957 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:23:54.957 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.182755+0000", 2024-09-13T11:23:54.957 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0036323200000000001 2024-09-13T11:23:54.957 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.957 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.957 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:54.957 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.183023+0000", 2024-09-13T11:23:54.957 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00026749100000000002 2024-09-13T11:23:54.957 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.957 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.957 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:23:54.957 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.183168+0000", 2024-09-13T11:23:54.957 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00014499099999999999 2024-09-13T11:23:54.958 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.958 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.958 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:23:54.958 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.183285+0000", 2024-09-13T11:23:54.958 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000117119 2024-09-13T11:23:54.958 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:54.958 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:23:54.958 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:54.958 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.958 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.958 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:60 3.b 3:d67218b0:::benchmark_data_smithi177_31116_object59:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T11:23:54.958 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:48.314603+0000", 2024-09-13T11:23:54.958 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 66.595766475000005, 2024-09-13T11:23:54.958 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.87106201100000002, 2024-09-13T11:23:54.959 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:23:54.959 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:23:54.959 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:54.959 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:23:54.959 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:23:54.959 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:54.959 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 60 2024-09-13T11:23:54.959 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.959 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:23:54.959 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.959 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:23:54.959 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314603+0000", 2024-09-13T11:23:54.960 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:54.960 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.960 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.960 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:23:54.960 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314603+0000", 2024-09-13T11:23:54.960 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:54.960 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.960 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.960 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:23:54.960 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314604+0000", 2024-09-13T11:23:54.960 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.2359999999999999e-06 2024-09-13T11:23:54.960 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.960 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.961 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:23:54.961 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314667+0000", 2024-09-13T11:23:54.961 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.3077999999999995e-05 2024-09-13T11:23:54.961 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.961 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.961 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:23:54.961 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314699+0000", 2024-09-13T11:23:54.961 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.1965999999999999e-05 2024-09-13T11:23:54.961 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.961 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.961 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:23:54.961 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314710+0000", 2024-09-13T11:23:54.961 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.1148999999999999e-05 2024-09-13T11:23:54.962 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.962 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.962 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:23:54.962 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316291+0000", 2024-09-13T11:23:54.962 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0015804440000000001 2024-09-13T11:23:54.962 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.962 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.962 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-09-13T11:23:54.962 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316310+0000", 2024-09-13T11:23:54.962 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.9799000000000001e-05 2024-09-13T11:23:54.962 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.962 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.962 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:23:54.963 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.179555+0000", 2024-09-13T11:23:54.963 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.86324469500000001 2024-09-13T11:23:54.963 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.963 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.963 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:23:54.963 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.179666+0000", 2024-09-13T11:23:54.963 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000110443 2024-09-13T11:23:54.963 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.963 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.963 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1", 2024-09-13T11:23:54.963 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.179791+0000", 2024-09-13T11:23:54.963 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00012560499999999999 2024-09-13T11:23:54.963 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.963 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.964 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:23:54.964 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.182761+0000", 2024-09-13T11:23:54.964 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0029697019999999998 2024-09-13T11:23:54.964 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.964 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.964 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:54.964 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.185572+0000", 2024-09-13T11:23:54.964 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.002811512 2024-09-13T11:23:54.964 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.964 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.964 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:23:54.964 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.185625+0000", 2024-09-13T11:23:54.964 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.2716000000000001e-05 2024-09-13T11:23:54.965 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.965 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.965 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:23:54.965 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.185665+0000", 2024-09-13T11:23:54.965 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.9666000000000003e-05 2024-09-13T11:23:54.965 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:54.965 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:23:54.965 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:54.965 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.965 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.965 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:62 3.b 3:d20ae5a8:::benchmark_data_smithi177_31116_object61:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T11:23:54.965 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:48.314723+0000", 2024-09-13T11:23:54.965 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 66.595645933, 2024-09-13T11:23:54.965 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.87338985099999999, 2024-09-13T11:23:54.966 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:23:54.966 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:23:54.966 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:54.966 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:23:54.966 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:23:54.966 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:54.966 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 62 2024-09-13T11:23:54.966 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.966 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:23:54.966 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.966 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:23:54.966 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314723+0000", 2024-09-13T11:23:54.966 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:54.966 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.967 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.967 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:23:54.967 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314723+0000", 2024-09-13T11:23:54.967 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:54.967 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.967 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.967 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:23:54.967 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314725+0000", 2024-09-13T11:23:54.967 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.3069999999999999e-06 2024-09-13T11:23:54.967 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.967 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.967 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:23:54.967 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314797+0000", 2024-09-13T11:23:54.967 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.2652999999999997e-05 2024-09-13T11:23:54.968 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.968 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.968 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:23:54.968 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314809+0000", 2024-09-13T11:23:54.968 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.1496e-05 2024-09-13T11:23:54.968 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.968 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.968 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:23:54.968 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314816+0000", 2024-09-13T11:23:54.968 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.6639999999999998e-06 2024-09-13T11:23:54.968 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.968 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.968 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:23:54.969 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316424+0000", 2024-09-13T11:23:54.969 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0016077559999999999 2024-09-13T11:23:54.969 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.969 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.969 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-09-13T11:23:54.969 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316441+0000", 2024-09-13T11:23:54.969 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.7305000000000001e-05 2024-09-13T11:23:54.970 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.970 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.970 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:23:54.970 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.180688+0000", 2024-09-13T11:23:54.970 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.86424692800000003 2024-09-13T11:23:54.970 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.970 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.970 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:23:54.970 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.180807+0000", 2024-09-13T11:23:54.970 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000118977 2024-09-13T11:23:54.970 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.970 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.971 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1", 2024-09-13T11:23:54.971 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.180938+0000", 2024-09-13T11:23:54.971 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00013049700000000001 2024-09-13T11:23:54.971 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.971 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.971 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:23:54.971 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.185680+0000", 2024-09-13T11:23:54.971 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0047417029999999999 2024-09-13T11:23:54.971 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.971 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.971 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:54.971 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.187856+0000", 2024-09-13T11:23:54.972 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0021760550000000001 2024-09-13T11:23:54.972 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.972 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.972 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:23:54.972 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.188007+0000", 2024-09-13T11:23:54.972 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00015185099999999999 2024-09-13T11:23:54.972 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.972 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.972 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:23:54.972 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.188113+0000", 2024-09-13T11:23:54.972 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000105659 2024-09-13T11:23:54.972 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:54.973 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:23:54.973 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:54.973 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.973 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.973 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:73 3.b 3:d0060d5f:::benchmark_data_smithi177_31116_object72:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T11:23:54.973 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:48.314829+0000", 2024-09-13T11:23:54.973 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 66.595540584000005, 2024-09-13T11:23:54.973 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.87364757199999998, 2024-09-13T11:23:54.973 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:23:54.973 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:23:54.973 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:54.973 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:23:54.974 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:23:54.974 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:54.974 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 73 2024-09-13T11:23:54.974 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.974 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:23:54.974 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.974 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:23:54.974 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314829+0000", 2024-09-13T11:23:54.974 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:54.974 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.974 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.974 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:23:54.974 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314829+0000", 2024-09-13T11:23:54.975 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:54.975 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.975 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.975 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:23:54.975 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314830+0000", 2024-09-13T11:23:54.975 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.243e-06 2024-09-13T11:23:54.975 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.975 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.975 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:23:54.975 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314965+0000", 2024-09-13T11:23:54.975 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00013470800000000001 2024-09-13T11:23:54.975 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.975 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.976 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:23:54.976 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314979+0000", 2024-09-13T11:23:54.976 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.4375999999999999e-05 2024-09-13T11:23:54.976 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.976 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.976 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:23:54.976 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314988+0000", 2024-09-13T11:23:54.976 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.8570000000000001e-06 2024-09-13T11:23:54.976 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.976 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.976 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:23:54.976 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316545+0000", 2024-09-13T11:23:54.976 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.001556878 2024-09-13T11:23:54.977 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.977 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.977 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-09-13T11:23:54.977 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316556+0000", 2024-09-13T11:23:54.977 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.1453e-05 2024-09-13T11:23:54.977 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.977 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.977 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:23:54.977 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.182124+0000", 2024-09-13T11:23:54.977 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.86556744200000002 2024-09-13T11:23:54.977 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.977 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.978 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:23:54.978 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.182246+0000", 2024-09-13T11:23:54.978 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00012210500000000001 2024-09-13T11:23:54.978 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.978 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.978 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1", 2024-09-13T11:23:54.978 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.182375+0000", 2024-09-13T11:23:54.978 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00012920999999999999 2024-09-13T11:23:54.978 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.978 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.978 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:23:54.978 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.185684+0000", 2024-09-13T11:23:54.978 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0033093969999999999 2024-09-13T11:23:54.979 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.979 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.979 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:54.979 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.188244+0000", 2024-09-13T11:23:54.979 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0025601550000000002 2024-09-13T11:23:54.979 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.979 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.979 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:23:54.979 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.188379+0000", 2024-09-13T11:23:54.979 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00013463400000000001 2024-09-13T11:23:54.979 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.979 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.979 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:23:54.979 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.188476+0000", 2024-09-13T11:23:54.980 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.7114000000000006e-05 2024-09-13T11:23:54.980 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:54.980 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:23:54.980 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:54.980 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.980 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.980 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:81 3.b 3:d705fd70:::benchmark_data_smithi177_31116_object80:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T11:23:54.980 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:48.315000+0000", 2024-09-13T11:23:54.980 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 66.595368746999995, 2024-09-13T11:23:54.980 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.87561418300000005, 2024-09-13T11:23:54.980 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:23:54.980 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:23:54.980 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:54.981 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:23:54.981 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:23:54.981 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:54.981 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 81 2024-09-13T11:23:54.981 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.981 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:23:54.981 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.981 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:23:54.981 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315000+0000", 2024-09-13T11:23:54.981 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:54.981 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.981 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.982 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:23:54.982 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315000+0000", 2024-09-13T11:23:54.982 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:54.982 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.982 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.982 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:23:54.982 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315002+0000", 2024-09-13T11:23:54.982 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.3799999999999999e-06 2024-09-13T11:23:54.982 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.982 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.982 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:23:54.982 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315119+0000", 2024-09-13T11:23:54.982 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000116925 2024-09-13T11:23:54.982 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.983 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.983 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:23:54.983 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315133+0000", 2024-09-13T11:23:54.983 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.3971999999999999e-05 2024-09-13T11:23:54.983 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.983 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.983 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:23:54.983 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315142+0000", 2024-09-13T11:23:54.983 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.4399999999999994e-06 2024-09-13T11:23:54.983 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.983 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.983 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:23:54.983 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316652+0000", 2024-09-13T11:23:54.983 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.001509907 2024-09-13T11:23:54.984 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.984 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.984 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-09-13T11:23:54.984 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316668+0000", 2024-09-13T11:23:54.984 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.5945999999999999e-05 2024-09-13T11:23:54.984 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.984 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.984 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:23:54.984 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.183405+0000", 2024-09-13T11:23:54.984 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.866737384 2024-09-13T11:23:54.984 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.984 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.984 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:23:54.984 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.183701+0000", 2024-09-13T11:23:54.985 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00029511099999999997 2024-09-13T11:23:54.985 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.985 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.985 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1", 2024-09-13T11:23:54.985 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.184078+0000", 2024-09-13T11:23:54.985 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00037741000000000001 2024-09-13T11:23:54.985 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.985 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.985 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:23:54.985 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.188504+0000", 2024-09-13T11:23:54.985 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0044260280000000003 2024-09-13T11:23:54.985 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.986 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.986 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:54.986 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.190562+0000", 2024-09-13T11:23:54.986 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0020580099999999999 2024-09-13T11:23:54.986 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.986 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.986 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:23:54.986 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.190592+0000", 2024-09-13T11:23:54.986 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.0057999999999999e-05 2024-09-13T11:23:54.986 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.986 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.986 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:23:54.986 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.190615+0000", 2024-09-13T11:23:54.986 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.2612e-05 2024-09-13T11:23:54.986 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:54.987 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:23:54.987 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:54.987 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.987 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.987 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:87 3.b 3:dd7de0ad:::benchmark_data_smithi177_31116_object86:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T11:23:54.987 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:48.315155+0000", 2024-09-13T11:23:54.987 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 66.595213950000002, 2024-09-13T11:23:54.987 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.87555916099999997, 2024-09-13T11:23:54.987 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:23:54.987 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:23:54.987 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:54.987 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:23:54.987 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:23:54.987 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:54.988 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 87 2024-09-13T11:23:54.988 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.988 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:23:54.988 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.988 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:23:54.988 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315155+0000", 2024-09-13T11:23:54.988 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:54.988 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.988 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.988 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:23:54.988 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315155+0000", 2024-09-13T11:23:54.988 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:54.988 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.989 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.989 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:23:54.989 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315157+0000", 2024-09-13T11:23:54.989 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.3829999999999999e-06 2024-09-13T11:23:54.989 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.989 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.989 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:23:54.989 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315271+0000", 2024-09-13T11:23:54.989 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000114539 2024-09-13T11:23:54.989 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.989 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.989 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:23:54.989 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315285+0000", 2024-09-13T11:23:54.990 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.4042e-05 2024-09-13T11:23:54.990 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.990 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.990 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:23:54.990 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315295+0000", 2024-09-13T11:23:54.990 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.2739999999999996e-06 2024-09-13T11:23:54.990 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.990 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.990 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:23:54.990 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316739+0000", 2024-09-13T11:23:54.990 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0014446960000000001 2024-09-13T11:23:54.990 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.990 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.991 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-09-13T11:23:54.991 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316747+0000", 2024-09-13T11:23:54.991 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.0790000000000001e-06 2024-09-13T11:23:54.991 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.991 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.991 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:23:54.991 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.184922+0000", 2024-09-13T11:23:54.991 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.86817446399999998 2024-09-13T11:23:54.991 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.991 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.991 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:23:54.991 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.185037+0000", 2024-09-13T11:23:54.991 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00011565099999999999 2024-09-13T11:23:54.991 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.992 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.992 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1", 2024-09-13T11:23:54.992 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.185171+0000", 2024-09-13T11:23:54.992 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00013316900000000001 2024-09-13T11:23:54.992 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.992 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.992 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:23:54.992 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.188520+0000", 2024-09-13T11:23:54.992 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0033489150000000001 2024-09-13T11:23:54.992 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.992 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.992 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:54.992 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.190652+0000", 2024-09-13T11:23:54.993 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0021325649999999999 2024-09-13T11:23:54.993 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.993 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.993 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:23:54.993 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.190686+0000", 2024-09-13T11:23:54.993 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.3827e-05 2024-09-13T11:23:54.993 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.993 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.993 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:23:54.993 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.190714+0000", 2024-09-13T11:23:54.993 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.8557000000000001e-05 2024-09-13T11:23:54.993 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:54.993 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:23:54.993 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:54.993 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.994 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.994 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:99 3.b 3:d011b600:::benchmark_data_smithi177_31116_object98:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T11:23:54.994 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:48.315307+0000", 2024-09-13T11:23:54.994 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 66.595062123000005, 2024-09-13T11:23:54.994 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.87551770799999995, 2024-09-13T11:23:54.994 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:23:54.994 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:23:54.994 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:54.994 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:23:54.994 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:23:54.994 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:54.994 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 99 2024-09-13T11:23:54.994 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.995 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:23:54.995 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.995 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:23:54.995 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315307+0000", 2024-09-13T11:23:54.995 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:54.995 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.995 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.995 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:23:54.995 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315307+0000", 2024-09-13T11:23:54.995 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:54.995 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.995 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.995 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:23:54.995 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315308+0000", 2024-09-13T11:23:54.995 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.316e-06 2024-09-13T11:23:54.996 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.996 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.996 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:23:54.996 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315453+0000", 2024-09-13T11:23:54.996 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00014473200000000001 2024-09-13T11:23:54.996 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.996 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.996 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:23:54.996 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315471+0000", 2024-09-13T11:23:54.996 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.8019000000000001e-05 2024-09-13T11:23:54.996 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.996 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.996 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:23:54.997 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315489+0000", 2024-09-13T11:23:54.997 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.7861999999999999e-05 2024-09-13T11:23:54.997 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.997 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.997 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:23:54.997 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316837+0000", 2024-09-13T11:23:54.997 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.001347787 2024-09-13T11:23:54.997 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.997 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.997 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-09-13T11:23:54.997 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316858+0000", 2024-09-13T11:23:54.997 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.1103000000000001e-05 2024-09-13T11:23:54.997 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.997 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.998 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:23:54.998 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.185713+0000", 2024-09-13T11:23:54.998 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.86885540800000005 2024-09-13T11:23:54.998 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.998 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.998 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:23:54.998 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.185820+0000", 2024-09-13T11:23:54.998 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000107081 2024-09-13T11:23:54.998 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.998 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.998 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1", 2024-09-13T11:23:54.998 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.185948+0000", 2024-09-13T11:23:54.998 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00012775800000000001 2024-09-13T11:23:54.999 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.999 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.999 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:23:54.999 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.190527+0000", 2024-09-13T11:23:54.999 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0045783910000000002 2024-09-13T11:23:54.999 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.999 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.999 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:54.999 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.190759+0000", 2024-09-13T11:23:54.999 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00023266199999999999 2024-09-13T11:23:54.999 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:54.999 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:54.999 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:23:54.999 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.190789+0000", 2024-09-13T11:23:54.999 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.9666999999999999e-05 2024-09-13T11:23:55.000 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:55.000 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:55.000 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:23:55.000 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.190825+0000", 2024-09-13T11:23:55.000 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.5921999999999999e-05 2024-09-13T11:23:55.000 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:55.000 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:23:55.000 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:55.000 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:55.000 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:55.000 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:108 3.b 3:d5c90217:::benchmark_data_smithi177_31116_object107:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T11:23:55.000 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:48.315513+0000", 2024-09-13T11:23:55.000 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 66.594856418000006, 2024-09-13T11:23:55.001 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.87544772500000001, 2024-09-13T11:23:55.001 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:23:55.001 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:23:55.001 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:55.001 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:23:55.001 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:23:55.001 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:55.001 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 108 2024-09-13T11:23:55.001 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:55.001 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:23:55.001 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:55.001 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:23:55.001 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315513+0000", 2024-09-13T11:23:55.001 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:55.002 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:55.002 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:55.002 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:23:55.002 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315513+0000", 2024-09-13T11:23:55.002 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:55.002 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:55.002 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:55.002 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:23:55.002 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315515+0000", 2024-09-13T11:23:55.002 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.9369999999999998e-06 2024-09-13T11:23:55.002 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:55.002 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:55.002 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:23:55.002 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315644+0000", 2024-09-13T11:23:55.002 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000129011 2024-09-13T11:23:55.003 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:55.003 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:55.003 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:23:55.003 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315657+0000", 2024-09-13T11:23:55.003 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.3689999999999999e-05 2024-09-13T11:23:55.003 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:55.003 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:55.003 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:23:55.003 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315668+0000", 2024-09-13T11:23:55.003 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0589e-05 2024-09-13T11:23:55.003 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:55.003 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:55.003 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:23:55.003 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316925+0000", 2024-09-13T11:23:55.003 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0012568340000000001 2024-09-13T11:23:55.004 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:55.004 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:55.004 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-09-13T11:23:55.004 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316932+0000", 2024-09-13T11:23:55.004 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.2230000000000001e-06 2024-09-13T11:23:55.004 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:55.004 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:55.004 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:23:55.004 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.186619+0000", 2024-09-13T11:23:55.004 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.86968679599999998 2024-09-13T11:23:55.004 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:55.004 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:55.004 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:23:55.004 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.186914+0000", 2024-09-13T11:23:55.005 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00029528900000000001 2024-09-13T11:23:55.005 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:55.005 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:55.005 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1", 2024-09-13T11:23:55.005 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.187217+0000", 2024-09-13T11:23:55.005 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00030232099999999998 2024-09-13T11:23:55.005 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:55.005 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:55.005 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:23:55.005 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.190532+0000", 2024-09-13T11:23:55.005 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.003315391 2024-09-13T11:23:55.005 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:55.005 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:55.006 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:55.006 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.190875+0000", 2024-09-13T11:23:55.006 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00034326300000000001 2024-09-13T11:23:55.006 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:55.006 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:55.006 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:23:55.006 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.190925+0000", 2024-09-13T11:23:55.006 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.9429999999999999e-05 2024-09-13T11:23:55.006 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:55.006 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:55.006 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:23:55.006 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.190961+0000", 2024-09-13T11:23:55.006 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.5951000000000002e-05 2024-09-13T11:23:55.007 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:55.007 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:23:55.007 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:55.007 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:55.007 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:55.007 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:172 3.b 3:dbec1eeb:::benchmark_data_smithi177_31116_object171:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T11:23:55.007 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:48.315684+0000", 2024-09-13T11:23:55.007 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 66.594685260999995, 2024-09-13T11:23:55.007 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.87615312999999995, 2024-09-13T11:23:55.007 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:23:55.007 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:23:55.007 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:55.007 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:23:55.007 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:23:55.008 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:55.008 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 172 2024-09-13T11:23:55.008 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:55.008 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:23:55.008 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:55.008 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:23:55.008 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315684+0000", 2024-09-13T11:23:55.008 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:55.008 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:55.008 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:55.008 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:23:55.008 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315684+0000", 2024-09-13T11:23:55.008 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:55.009 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:55.009 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:55.009 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:23:55.009 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315686+0000", 2024-09-13T11:23:55.009 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.886e-06 2024-09-13T11:23:55.009 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:55.009 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:55.009 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:23:55.009 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315820+0000", 2024-09-13T11:23:55.010 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00013388500000000001 2024-09-13T11:23:55.010 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:55.010 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:55.010 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:23:55.010 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315837+0000", 2024-09-13T11:23:55.010 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.6756e-05 2024-09-13T11:23:55.010 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:55.010 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:55.010 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:23:55.010 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315851+0000", 2024-09-13T11:23:55.010 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.4566000000000001e-05 2024-09-13T11:23:55.010 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:55.010 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:55.011 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:23:55.011 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.317021+0000", 2024-09-13T11:23:55.011 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0011701769999999999 2024-09-13T11:23:55.011 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:55.011 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:55.011 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-09-13T11:23:55.011 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.317037+0000", 2024-09-13T11:23:55.011 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.609e-05 2024-09-13T11:23:55.011 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:55.011 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:55.011 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:23:55.011 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.188634+0000", 2024-09-13T11:23:55.011 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.87159711900000003 2024-09-13T11:23:55.011 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:55.012 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:55.012 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:23:55.012 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.188974+0000", 2024-09-13T11:23:55.012 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00033922299999999999 2024-09-13T11:23:55.012 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:55.012 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:55.012 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1", 2024-09-13T11:23:55.012 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.189510+0000", 2024-09-13T11:23:55.012 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00053618999999999997 2024-09-13T11:23:55.012 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:55.012 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:55.012 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:55.012 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.191027+0000", 2024-09-13T11:23:55.012 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0015168390000000001 2024-09-13T11:23:55.013 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:55.013 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:55.013 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:23:55.013 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.191766+0000", 2024-09-13T11:23:55.013 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000739064 2024-09-13T11:23:55.013 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:55.013 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:55.013 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:23:55.013 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.191810+0000", 2024-09-13T11:23:55.013 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.3755999999999999e-05 2024-09-13T11:23:55.013 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:55.013 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:55.013 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:23:55.013 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.191837+0000", 2024-09-13T11:23:55.014 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.7579000000000001e-05 2024-09-13T11:23:55.014 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:55.014 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:23:55.014 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:55.014 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:55.014 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:55.014 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:177 3.b 3:de89c779:::benchmark_data_smithi177_31116_object176:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T11:23:55.014 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:48.315898+0000", 2024-09-13T11:23:55.014 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 66.594470946000001, 2024-09-13T11:23:55.014 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.87600712599999997, 2024-09-13T11:23:55.014 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:23:55.014 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:23:55.014 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:55.014 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:23:55.014 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:23:55.015 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:55.015 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 177 2024-09-13T11:23:55.015 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:55.015 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:23:55.015 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:55.015 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:23:55.015 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315898+0000", 2024-09-13T11:23:55.015 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:55.015 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:55.015 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:55.015 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:23:55.015 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315898+0000", 2024-09-13T11:23:55.015 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:55.015 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:55.015 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:55.016 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:23:55.016 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315900+0000", 2024-09-13T11:23:55.016 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.4929999999999999e-06 2024-09-13T11:23:55.016 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:55.016 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:55.016 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:23:55.016 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316027+0000", 2024-09-13T11:23:55.016 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000127568 2024-09-13T11:23:55.016 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:55.016 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:55.016 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:23:55.016 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316048+0000", 2024-09-13T11:23:55.016 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.0795999999999999e-05 2024-09-13T11:23:55.017 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:55.017 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:55.017 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:23:55.017 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316061+0000", 2024-09-13T11:23:55.017 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.3283e-05 2024-09-13T11:23:55.017 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:55.017 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:55.017 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:23:55.017 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.317107+0000", 2024-09-13T11:23:55.017 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0010452949999999999 2024-09-13T11:23:55.017 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:55.017 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:55.017 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-09-13T11:23:55.018 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.317114+0000", 2024-09-13T11:23:55.018 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.4569999999999999e-06 2024-09-13T11:23:55.018 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:55.018 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:55.018 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:23:55.018 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.190017+0000", 2024-09-13T11:23:55.018 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.87290258499999995 2024-09-13T11:23:55.018 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:55.018 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:55.018 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:23:55.018 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.190138+0000", 2024-09-13T11:23:55.018 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000121418 2024-09-13T11:23:55.018 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:55.018 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:55.018 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1", 2024-09-13T11:23:55.019 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.190216+0000", 2024-09-13T11:23:55.019 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.7410999999999995e-05 2024-09-13T11:23:55.019 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:55.019 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:55.019 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:55.019 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.191755+0000", 2024-09-13T11:23:55.019 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0015395420000000001 2024-09-13T11:23:55.019 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:55.019 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:55.019 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:23:55.019 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.191843+0000", 2024-09-13T11:23:55.019 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.8022999999999994e-05 2024-09-13T11:23:55.019 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:55.019 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:55.020 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:23:55.020 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.191876+0000", 2024-09-13T11:23:55.020 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.2619000000000001e-05 2024-09-13T11:23:55.020 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:55.020 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:55.020 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:23:55.020 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.191905+0000", 2024-09-13T11:23:55.020 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.9635999999999999e-05 2024-09-13T11:23:55.020 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:55.020 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:23:55.020 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:55.020 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:55.020 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:55.021 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:187 3.b 3:da03c8dd:::benchmark_data_smithi177_31116_object186:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T11:23:55.021 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:48.316083+0000", 2024-09-13T11:23:55.021 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 66.594285990000003, 2024-09-13T11:23:55.021 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.87835499500000003, 2024-09-13T11:23:55.021 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:23:55.021 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:23:55.021 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:55.021 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:23:55.021 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:23:55.021 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:55.021 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 187 2024-09-13T11:23:55.021 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:55.021 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:23:55.021 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:55.022 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:23:55.022 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316083+0000", 2024-09-13T11:23:55.022 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:55.022 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:55.022 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:55.022 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:23:55.022 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316083+0000", 2024-09-13T11:23:55.022 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:55.022 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:55.022 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:55.022 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:23:55.022 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316086+0000", 2024-09-13T11:23:55.022 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.5560000000000002e-06 2024-09-13T11:23:55.023 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:55.023 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:55.023 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:23:55.023 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316199+0000", 2024-09-13T11:23:55.023 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000113522 2024-09-13T11:23:55.023 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:55.023 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:55.023 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:23:55.023 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316215+0000", 2024-09-13T11:23:55.023 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.5265999999999999e-05 2024-09-13T11:23:55.023 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:55.023 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:55.023 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:23:55.023 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316226+0000", 2024-09-13T11:23:55.024 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0967e-05 2024-09-13T11:23:55.024 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:55.024 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:55.024 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:23:55.024 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.317198+0000", 2024-09-13T11:23:55.024 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00097194100000000004 2024-09-13T11:23:55.024 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:55.024 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:55.024 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-09-13T11:23:55.024 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.317213+0000", 2024-09-13T11:23:55.024 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.5472999999999999e-05 2024-09-13T11:23:55.024 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:55.024 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:55.025 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:23:55.025 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.191077+0000", 2024-09-13T11:23:55.025 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.87386435200000001 2024-09-13T11:23:55.025 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:55.025 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:55.025 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:23:55.025 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.191181+0000", 2024-09-13T11:23:55.025 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000103618 2024-09-13T11:23:55.025 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:55.025 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:55.025 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1", 2024-09-13T11:23:55.025 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.191305+0000", 2024-09-13T11:23:55.025 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00012358599999999999 2024-09-13T11:23:55.025 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:55.025 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:55.026 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:55.026 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.194279+0000", 2024-09-13T11:23:55.026 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0029746970000000001 2024-09-13T11:23:55.026 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:55.026 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:55.026 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:23:55.026 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.194336+0000", 2024-09-13T11:23:55.026 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.7244000000000003e-05 2024-09-13T11:23:55.026 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:55.026 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:55.026 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:23:55.026 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.194397+0000", 2024-09-13T11:23:55.026 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.0278000000000001e-05 2024-09-13T11:23:55.027 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:55.027 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:55.027 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:23:55.027 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.194438+0000", 2024-09-13T11:23:55.027 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.1495000000000002e-05 2024-09-13T11:23:55.027 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:55.027 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:23:55.027 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:55.027 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:55.027 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:55.027 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:200 3.b 3:d90039f1:::benchmark_data_smithi177_31116_object199:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T11:23:55.027 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:48.316280+0000", 2024-09-13T11:23:55.027 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 66.594089525000001, 2024-09-13T11:23:55.027 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.87879273000000002, 2024-09-13T11:23:55.028 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:23:55.028 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:23:55.028 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:55.028 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:23:55.028 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:23:55.028 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:55.028 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 200 2024-09-13T11:23:55.028 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:55.028 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:23:55.028 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:55.028 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:23:55.028 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316280+0000", 2024-09-13T11:23:55.028 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:55.029 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:55.029 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:55.029 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:23:55.029 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316280+0000", 2024-09-13T11:23:55.029 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:55.029 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:55.029 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:55.029 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:23:55.029 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316282+0000", 2024-09-13T11:23:55.029 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.497e-06 2024-09-13T11:23:55.029 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:55.029 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:55.029 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:23:55.030 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316411+0000", 2024-09-13T11:23:55.030 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00012924799999999999 2024-09-13T11:23:55.030 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:55.030 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:55.030 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:23:55.030 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316426+0000", 2024-09-13T11:23:55.030 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.4183e-05 2024-09-13T11:23:55.030 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:55.030 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:55.030 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:23:55.030 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316435+0000", 2024-09-13T11:23:55.030 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.0990000000000005e-06 2024-09-13T11:23:55.030 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:55.030 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:55.031 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:23:55.031 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.317250+0000", 2024-09-13T11:23:55.031 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00081501100000000003 2024-09-13T11:23:55.031 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:55.031 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:55.031 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-09-13T11:23:55.031 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.317257+0000", 2024-09-13T11:23:55.031 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.2169999999999997e-06 2024-09-13T11:23:55.031 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:55.031 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:55.031 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:23:55.031 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.191936+0000", 2024-09-13T11:23:55.031 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.87467922099999995 2024-09-13T11:23:55.032 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:55.032 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:55.032 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:23:55.032 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.192056+0000", 2024-09-13T11:23:55.032 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000119638 2024-09-13T11:23:55.032 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:55.032 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:55.032 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1", 2024-09-13T11:23:55.032 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.192184+0000", 2024-09-13T11:23:55.032 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00012813399999999999 2024-09-13T11:23:55.032 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:55.032 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:55.032 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:55.033 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.194324+0000", 2024-09-13T11:23:55.033 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0021399420000000001 2024-09-13T11:23:55.033 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:55.033 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:55.033 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:23:55.033 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.194802+0000", 2024-09-13T11:23:55.033 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00047798799999999999 2024-09-13T11:23:55.033 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:55.033 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:55.033 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:23:55.033 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.194951+0000", 2024-09-13T11:23:55.033 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0001494 2024-09-13T11:23:55.033 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:55.033 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:55.034 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:23:55.034 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.195072+0000", 2024-09-13T11:23:55.034 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00012115199999999999 2024-09-13T11:23:55.034 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:55.034 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:23:55.034 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:55.034 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:55.034 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:55.034 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:25525 3.1 3:8e5634af:::benchmark_data_smithi177_31116_object25524:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e56)", 2024-09-13T11:23:55.034 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:04.981590+0000", 2024-09-13T11:23:55.034 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 49.928779247999998, 2024-09-13T11:23:55.034 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.23642401599999999, 2024-09-13T11:23:55.034 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:23:55.035 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:23:55.035 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:55.035 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:23:55.035 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:23:55.035 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:55.035 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 25525 2024-09-13T11:23:55.035 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:55.035 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:23:55.035 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:55.035 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:23:55.035 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.981590+0000", 2024-09-13T11:23:55.035 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:55.035 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:55.035 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:55.036 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:23:55.036 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.981590+0000", 2024-09-13T11:23:55.036 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:55.036 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:55.036 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:55.036 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:23:55.036 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.981598+0000", 2024-09-13T11:23:55.036 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.4230000000000005e-06 2024-09-13T11:23:55.036 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:55.036 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:55.036 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:23:55.036 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.981652+0000", 2024-09-13T11:23:55.036 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.3226000000000002e-05 2024-09-13T11:23:55.037 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:55.037 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:55.037 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:23:55.037 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.981680+0000", 2024-09-13T11:23:55.037 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.8348999999999999e-05 2024-09-13T11:23:55.037 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:55.037 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:55.037 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:23:55.037 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.981695+0000", 2024-09-13T11:23:55.037 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.5085999999999999e-05 2024-09-13T11:23:55.037 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:55.037 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:55.037 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:23:55.037 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.981809+0000", 2024-09-13T11:23:55.038 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00011382700000000001 2024-09-13T11:23:55.038 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:55.038 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:55.038 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:23:55.038 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.982042+0000", 2024-09-13T11:23:55.038 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000232697 2024-09-13T11:23:55.038 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:55.038 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:55.038 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 11", 2024-09-13T11:23:55.038 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.982261+0000", 2024-09-13T11:23:55.038 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00021918400000000001 2024-09-13T11:23:55.038 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:55.038 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:55.038 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:23:55.038 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.984553+0000", 2024-09-13T11:23:55.039 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0022918460000000002 2024-09-13T11:23:55.039 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:55.039 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:55.039 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:55.039 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:05.217842+0000", 2024-09-13T11:23:55.039 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.233289106 2024-09-13T11:23:55.039 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:55.039 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:55.039 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:23:55.039 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:05.217940+0000", 2024-09-13T11:23:55.039 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.7936999999999994e-05 2024-09-13T11:23:55.039 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:55.039 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:55.039 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:23:55.040 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:05.218014+0000", 2024-09-13T11:23:55.040 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.4334999999999999e-05 2024-09-13T11:23:55.040 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:55.040 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:23:55.040 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:55.040 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:55.040 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:55.040 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop_reply(client.4665.0:25525 3.1 e56/30 ondisk, result = 0)", 2024-09-13T11:23:55.040 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:04.985699+0000", 2024-09-13T11:23:55.040 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 49.924670517000003, 2024-09-13T11:23:55.040 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.23233051800000001, 2024-09-13T11:23:55.040 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:23:55.040 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:23:55.040 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-09-13T11:23:55.041 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:23:55.041 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:55.041 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:23:55.041 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.985699+0000", 2024-09-13T11:23:55.041 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:55.041 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:55.041 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:55.041 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:23:55.041 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.985699+0000", 2024-09-13T11:23:55.041 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:55.041 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:55.041 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:55.041 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:23:55.041 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.985702+0000", 2024-09-13T11:23:55.042 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.0570000000000001e-06 2024-09-13T11:23:55.042 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:55.042 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:55.042 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:23:55.042 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.985711+0000", 2024-09-13T11:23:55.042 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.1060000000000006e-06 2024-09-13T11:23:55.042 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:55.042 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:55.042 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:23:55.042 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:05.217667+0000", 2024-09-13T11:23:55.042 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.231955774 2024-09-13T11:23:55.042 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:55.042 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:55.042 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:23:55.043 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:05.217685+0000", 2024-09-13T11:23:55.043 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.791e-05 2024-09-13T11:23:55.043 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:55.043 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:55.043 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:23:55.043 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:05.217818+0000", 2024-09-13T11:23:55.043 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000133036 2024-09-13T11:23:55.043 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:55.043 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:55.043 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:23:55.043 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:05.217833+0000", 2024-09-13T11:23:55.043 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.519e-05 2024-09-13T11:23:55.043 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:55.044 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:55.044 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:23:55.044 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:05.218029+0000", 2024-09-13T11:23:55.044 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00019644500000000001 2024-09-13T11:23:55.044 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:55.044 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:23:55.044 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:55.044 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:55.044 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:55.044 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "MOSDPGPushReply(3.16 76/70 [PushReplyOp(3:6efa689b:::benchmark_data_smithi177_31116_object24598:head)])", 2024-09-13T11:23:55.044 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:32.699090+0000", 2024-09-13T11:23:55.044 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 22.211278837999998, 2024-09-13T11:23:55.044 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.232999069, 2024-09-13T11:23:55.044 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:23:55.045 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:23:55.045 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "reached pg", 2024-09-13T11:23:55.045 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:23:55.045 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:55.045 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:23:55.045 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:32.699090+0000", 2024-09-13T11:23:55.045 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:55.045 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:55.045 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:55.045 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:23:55.045 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:32.699090+0000", 2024-09-13T11:23:55.045 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:55.045 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:55.046 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:55.046 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:23:55.046 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:32.699092+0000", 2024-09-13T11:23:55.046 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.4559999999999999e-06 2024-09-13T11:23:55.046 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:55.046 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:55.046 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:23:55.046 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:32.699100+0000", 2024-09-13T11:23:55.046 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.1899999999999995e-06 2024-09-13T11:23:55.046 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:55.046 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:55.046 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:23:55.046 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:32.931118+0000", 2024-09-13T11:23:55.046 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.23201782900000001 2024-09-13T11:23:55.047 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:55.047 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:55.047 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:23:55.047 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:32.931132+0000", 2024-09-13T11:23:55.047 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.4212999999999999e-05 2024-09-13T11:23:55.047 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:55.047 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:55.047 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:23:55.047 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:32.931223+0000", 2024-09-13T11:23:55.047 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.1058999999999997e-05 2024-09-13T11:23:55.047 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:55.047 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:55.047 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:23:55.047 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:32.932089+0000", 2024-09-13T11:23:55.048 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00086632199999999999 2024-09-13T11:23:55.048 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:55.048 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:23:55.048 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:55.048 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:55.048 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:23:55.048 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-09-13T11:23:55.049 INFO:tasks.radosbench.radosbench.0.smithi177.stdout: 88 15 72879 72864 51.7446 69.5 0.0211235 0.0193193 2024-09-13T11:23:55.049 INFO:tasks.ceph.osd.7.smithi177.stderr:2024-09-13T11:23:55.044+0000 7f897b51d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26575) UID: 0 2024-09-13T11:23:55.050 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-13T11:23:55.150 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-13T11:23:55.144+0000 7f6a68775640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26307) UID: 0 2024-09-13T11:23:55.184 INFO:teuthology.orchestra.run.smithi001.stderr:reweighted osd.14 to 1 (10000) 2024-09-13T11:23:55.190 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-13T11:23:55.190 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:23:55.208 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-13T11:23:55.208 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-13T11:23:55.220 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 15 1 2024-09-13T11:23:55.267 INFO:tasks.ceph.osd.8.smithi001.stderr:2024-09-13T11:23:55.260+0000 7fb9b48ae640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26312) UID: 0 2024-09-13T11:23:55.352 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-09-13T11:23:55.349+0000 7f97befc9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26315) UID: 0 2024-09-13T11:23:55.354 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-13T11:23:55.354 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:23:55.371 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-13T11:23:55.458 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-09-13T11:23:55.457+0000 7f97befc9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26315) UID: 0 2024-09-13T11:23:55.529 INFO:teuthology.orchestra.run.smithi001.stdout:osd.6: {} 2024-09-13T11:23:55.529 INFO:teuthology.orchestra.run.smithi001.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T11:23:55.553 INFO:tasks.ceph.osd.7.smithi177.stderr:2024-09-13T11:23:55.548+0000 7f897b51d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26575) UID: 0 2024-09-13T11:23:55.556 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-13T11:23:55.557 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:23:55.559 INFO:teuthology.orchestra.run.smithi001.stdout:osd.7: {} 2024-09-13T11:23:55.559 INFO:teuthology.orchestra.run.smithi001.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T11:23:55.574 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-13T11:23:55.593 INFO:teuthology.orchestra.run.smithi001.stdout:osd.8: {} 2024-09-13T11:23:55.594 INFO:teuthology.orchestra.run.smithi001.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-13T11:23:55.630 INFO:teuthology.orchestra.run.smithi001.stdout:osd.9: {} 2024-09-13T11:23:55.630 INFO:teuthology.orchestra.run.smithi001.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-13T11:23:55.654 INFO:tasks.ceph.osd.9.smithi092.stderr:2024-09-13T11:23:55.651+0000 7ff5ab3d0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26348) UID: 0 2024-09-13T11:23:55.664 INFO:teuthology.orchestra.run.smithi001.stdout:osd.10: {} 2024-09-13T11:23:55.664 INFO:teuthology.orchestra.run.smithi001.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-13T11:23:55.707 INFO:teuthology.orchestra.run.smithi001.stdout:osd.11: {} 2024-09-13T11:23:55.707 INFO:teuthology.orchestra.run.smithi001.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-13T11:23:55.745 INFO:teuthology.orchestra.run.smithi001.stdout:osd.12: {} 2024-09-13T11:23:55.745 INFO:teuthology.orchestra.run.smithi001.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-13T11:23:55.755 INFO:tasks.ceph.osd.11.smithi177.stderr:2024-09-13T11:23:55.748+0000 7f24792c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26580) UID: 0 2024-09-13T11:23:55.765 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-13T11:23:55.765 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:23:55.777 INFO:teuthology.orchestra.run.smithi001.stdout:osd.13: {} 2024-09-13T11:23:55.777 INFO:teuthology.orchestra.run.smithi001.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-13T11:23:55.788 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-13T11:23:55.808 INFO:teuthology.orchestra.run.smithi001.stdout:osd.14: {} 2024-09-13T11:23:55.808 INFO:teuthology.orchestra.run.smithi001.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-13T11:23:55.840 INFO:teuthology.orchestra.run.smithi001.stdout:osd.15: {} 2024-09-13T11:23:55.840 INFO:teuthology.orchestra.run.smithi001.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-13T11:23:55.856 INFO:tasks.ceph.osd.10.smithi162.stderr:2024-09-13T11:23:55.853+0000 7fb59c9f0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26317) UID: 0 2024-09-13T11:23:55.937 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-13T11:23:55.937 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:23:55.956 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-13T11:23:55.957 INFO:tasks.ceph.osd.12.smithi001.stderr:2024-09-13T11:23:55.952+0000 7f892c6e6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26314) UID: 0 2024-09-13T11:23:56.006 INFO:tasks.radosbench.radosbench.0.smithi177.stdout: 89 16 74210 74194 52.0971 83.125 0.00705007 0.0191882 2024-09-13T11:23:56.058 INFO:tasks.ceph.osd.12.smithi001.stderr:2024-09-13T11:23:56.052+0000 7f892c6e6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26314) UID: 0 2024-09-13T11:23:56.140 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-13T11:23:56.140 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:23:56.159 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-13T11:23:56.152+0000 7f6a68775640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26307) UID: 0 2024-09-13T11:23:56.161 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-09-13T11:23:56.207 INFO:tasks.ceph.osd.12.smithi001.stderr:2024-09-13T11:23:56.200+0000 7f891f10f640 -1 osd.12 98 handle_osd_map injecting map crc failure 2024-09-13T11:23:56.215 INFO:teuthology.orchestra.run.smithi001.stderr:reweighted osd.15 to 1 (10000) 2024-09-13T11:23:56.215 INFO:teuthology.orchestra.run.smithi001.stderr:noscrub is unset 2024-09-13T11:23:56.242 INFO:tasks.ceph.ceph_manager.ceph:waiting for recovery to complete 2024-09-13T11:23:56.242 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-13T11:23:56.259 INFO:tasks.ceph.osd.5.smithi092.stderr:2024-09-13T11:23:56.255+0000 7f6f0b705640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26346) UID: 0 2024-09-13T11:23:56.301 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-13T11:23:56.302 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:23:56.323 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-09-13T11:23:56.360 INFO:tasks.ceph.osd.12.smithi001.stderr:2024-09-13T11:23:56.356+0000 7f892c6e6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26314) UID: 0 2024-09-13T11:23:56.461 INFO:tasks.ceph.osd.15.smithi177.stderr:2024-09-13T11:23:56.456+0000 7f9d6ce77640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26578) UID: 0 2024-09-13T11:23:56.462 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-13T11:23:56.462 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:23:56.480 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-09-13T11:23:56.562 INFO:tasks.ceph.osd.3.smithi177.stderr:2024-09-13T11:23:56.556+0000 7f10adba4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26574) UID: 0 2024-09-13T11:23:56.612 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-13T11:23:56.612 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:23:56.632 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_ops_in_flight 2024-09-13T11:23:56.659 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-13T11:23:56.660 INFO:teuthology.orchestra.run.smithi001.stderr:dumped all 2024-09-13T11:23:56.678 INFO:tasks.ceph.osd.10.smithi162.stderr:2024-09-13T11:23:56.677+0000 7fb59c9f0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26317) UID: 0 2024-09-13T11:23:56.679 INFO:teuthology.orchestra.run.smithi001.stdout:{"pg_ready":true,"pg_map":{"version":138,"stamp":"2024-09-13T11:23:56.207424+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4738744883,"num_objects":72304,"num_object_clones":0,"num_object_copies":144608,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6760,"num_objects_unfound":0,"num_objects_dirty":72304,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":28462,"num_write_kb":1814738,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9885,"num_bytes_recovered":647725075,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":132856,"ondisk_log_size":132856,"up":90,"acting":90,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":119,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":16,"kb":1509949440,"kb_used":12220340,"kb_used_data":11386036,"kb_used_omap":27,"kb_used_meta":834020,"kb_avail":1497729100,"statfs":{"total":1546188226560,"available":1533674598400,"internally_reserved":0,"allocated":11659300864,"data_stored":11652542616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28358,"internal_metadata":854036794},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,1,1,2,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":247726080,"num_objects":3780,"num_object_clones":0,"num_object_copies":7560,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-3061,"num_objects_unfound":0,"num_objects_dirty":3780,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3780,"num_write_kb":241920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1531,"num_bytes_recovered":100335616,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.051370"},"pg_stats":[{"pgid":"3.23","version":"98'3432","reported_seq":6144,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-09-13T11:23:55.908873+0000","last_change":"2024-09-13T11:23:48.133184+0000","last_active":"2024-09-13T11:23:55.908873+0000","last_peered":"2024-09-13T11:23:55.908873+0000","last_clean":"2024-09-13T11:23:55.908873+0000","last_became_active":"2024-09-13T11:23:47.926677+0000","last_became_peered":"2024-09-13T11:23:47.926677+0000","last_unstale":"2024-09-13T11:23:55.908873+0000","last_undegraded":"2024-09-13T11:23:55.908873+0000","last_fullsized":"2024-09-13T11:23:55.908873+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":6,"last_scrub":"91'3278","last_scrub_stamp":"2024-09-13T11:23:48.133084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:48.133084+0000","objects_scrubbed":1029,"log_size":3432,"log_dups_size":0,"ondisk_log_size":3432,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:12.874134+0000","scrub_duration":139,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":77594624,"num_objects":1184,"num_object_clones":0,"num_object_copies":2368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1184,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":155,"num_write_kb":9920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9],"acting":[5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.22","version":"98'3408","reported_seq":5307,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-09-13T11:23:56.014918+0000","last_change":"2024-09-13T11:23:47.300446+0000","last_active":"2024-09-13T11:23:56.014918+0000","last_peered":"2024-09-13T11:23:56.014918+0000","last_clean":"2024-09-13T11:23:56.014918+0000","last_became_active":"2024-09-13T11:23:46.925318+0000","last_became_peered":"2024-09-13T11:23:46.925318+0000","last_unstale":"2024-09-13T11:23:56.014918+0000","last_undegraded":"2024-09-13T11:23:56.014918+0000","last_fullsized":"2024-09-13T11:23:56.014918+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":6,"last_scrub":"90'3246","last_scrub_stamp":"2024-09-13T11:23:47.300339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:47.300339+0000","objects_scrubbed":1033,"log_size":3408,"log_dups_size":0,"ondisk_log_size":3408,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:24:49.780485+0000","scrub_duration":203,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":78446592,"num_objects":1197,"num_object_clones":0,"num_object_copies":2394,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1197,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":164,"num_write_kb":10496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,14],"acting":[0,14],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.21","version":"94'3286","reported_seq":4757,"reported_epoch":94,"state":"active+recovering+remapped","last_fresh":"2024-09-13T11:23:52.020283+0000","last_change":"2024-09-13T11:23:45.861514+0000","last_active":"2024-09-13T11:23:52.020283+0000","last_peered":"2024-09-13T11:23:52.020283+0000","last_clean":"2024-09-13T11:23:32.050222+0000","last_became_active":"2024-09-13T11:23:35.026171+0000","last_became_peered":"2024-09-13T11:23:35.026171+0000","last_unstale":"2024-09-13T11:23:52.020283+0000","last_undegraded":"2024-09-13T11:23:52.020283+0000","last_fullsized":"2024-09-13T11:23:52.020283+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":6,"last_scrub":"69'2784","last_scrub_stamp":"2024-09-13T11:23:21.595419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:21.595419+0000","objects_scrubbed":1386,"log_size":3286,"log_dups_size":0,"ondisk_log_size":3286,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":311,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":69074944,"num_objects":1054,"num_object_clones":0,"num_object_copies":2108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":810,"num_objects_unfound":0,"num_objects_dirty":1054,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":361,"num_write_kb":23104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":431,"num_bytes_recovered":28246016,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8],"acting":[11,0],"avail_no_missing":["11","13"],"object_location_counts":[{"shards":"0,11","objects":649},{"shards":"11,13","objects":405}],"blocked_by":[],"up_primary":0,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.20","version":"98'3426","reported_seq":5440,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-09-13T11:23:56.029766+0000","last_change":"2024-09-13T11:23:47.969503+0000","last_active":"2024-09-13T11:23:56.029766+0000","last_peered":"2024-09-13T11:23:56.029766+0000","last_clean":"2024-09-13T11:23:56.029766+0000","last_became_active":"2024-09-13T11:23:33.954946+0000","last_became_peered":"2024-09-13T11:23:33.954946+0000","last_unstale":"2024-09-13T11:23:56.029766+0000","last_undegraded":"2024-09-13T11:23:56.029766+0000","last_fullsized":"2024-09-13T11:23:56.029766+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":6,"last_scrub":"91'3270","last_scrub_stamp":"2024-09-13T11:23:47.969372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:47.969372+0000","objects_scrubbed":1061,"log_size":3426,"log_dups_size":0,"ondisk_log_size":3426,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:24:48.255263+0000","scrub_duration":1327,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":80084992,"num_objects":1222,"num_object_clones":0,"num_object_copies":2444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1222,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":161,"num_write_kb":10304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,0],"acting":[12,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[]},{"pgid":"3.1f","version":"94'3850","reported_seq":6144,"reported_epoch":94,"state":"active+recovering+remapped","last_fresh":"2024-09-13T11:23:52.060704+0000","last_change":"2024-09-13T11:23:37.329722+0000","last_active":"2024-09-13T11:23:52.060704+0000","last_peered":"2024-09-13T11:23:52.060704+0000","last_clean":"2024-09-13T11:23:32.316727+0000","last_became_active":"2024-09-13T11:23:35.053803+0000","last_became_peered":"2024-09-13T11:23:35.053803+0000","last_unstale":"2024-09-13T11:23:52.060704+0000","last_undegraded":"2024-09-13T11:23:52.060704+0000","last_fullsized":"2024-09-13T11:23:52.060704+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:22:24.896055+0000","objects_scrubbed":0,"log_size":3850,"log_dups_size":0,"ondisk_log_size":3850,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":140476416,"num_objects":2144,"num_object_clones":0,"num_object_copies":4288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1872,"num_objects_unfound":0,"num_objects_dirty":2144,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2144,"num_write_kb":137184,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":715,"num_bytes_recovered":46858240,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,14],"acting":[11,10],"avail_no_missing":["11","12"],"object_location_counts":[{"shards":"10,11","objects":1208},{"shards":"11,12","objects":936}],"blocked_by":[],"up_primary":10,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1e","version":"98'4129","reported_seq":12953,"reported_epoch":98,"state":"active+recovering+remapped","last_fresh":"2024-09-13T11:23:56.078942+0000","last_change":"2024-09-13T11:23:35.106919+0000","last_active":"2024-09-13T11:23:56.078942+0000","last_peered":"2024-09-13T11:23:56.078942+0000","last_clean":"2024-09-13T11:23:30.974769+0000","last_became_active":"2024-09-13T11:23:35.075137+0000","last_became_peered":"2024-09-13T11:23:35.075137+0000","last_unstale":"2024-09-13T11:23:56.078942+0000","last_undegraded":"2024-09-13T11:23:56.078942+0000","last_fullsized":"2024-09-13T11:23:56.078942+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:22:24.896055+0000","objects_scrubbed":0,"log_size":4129,"log_dups_size":0,"ondisk_log_size":4129,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":155254784,"num_objects":2369,"num_object_clones":0,"num_object_copies":4738,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1005,"num_objects_unfound":0,"num_objects_dirty":2369,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2369,"num_write_kb":151616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2070,"num_bytes_recovered":135626752,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[0,4],"avail_no_missing":["0","1","8","14"],"object_location_counts":[{"shards":"0,1","objects":502},{"shards":"0,1,4","objects":1},{"shards":"0,4","objects":1866}],"blocked_by":[],"up_primary":5,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1d","version":"97'3925","reported_seq":7866,"reported_epoch":97,"state":"active+recovering+remapped","last_fresh":"2024-09-13T11:23:55.002071+0000","last_change":"2024-09-13T11:23:35.096108+0000","last_active":"2024-09-13T11:23:55.002071+0000","last_peered":"2024-09-13T11:23:55.002071+0000","last_clean":"2024-09-13T11:23:32.393844+0000","last_became_active":"2024-09-13T11:23:35.080149+0000","last_became_peered":"2024-09-13T11:23:35.080149+0000","last_unstale":"2024-09-13T11:23:55.002071+0000","last_undegraded":"2024-09-13T11:23:55.002071+0000","last_fullsized":"2024-09-13T11:23:55.002071+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:22:24.896055+0000","objects_scrubbed":0,"log_size":3925,"log_dups_size":0,"ondisk_log_size":3925,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":146407424,"num_objects":2234,"num_object_clones":0,"num_object_copies":4468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":876,"num_objects_unfound":0,"num_objects_dirty":2234,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2234,"num_write_kb":142976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1262,"num_bytes_recovered":82706432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,15],"acting":[3,2],"avail_no_missing":["3","7"],"object_location_counts":[{"shards":"2,3","objects":1796},{"shards":"3,7","objects":438}],"blocked_by":[],"up_primary":2,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1c","version":"94'3954","reported_seq":7192,"reported_epoch":94,"state":"active+recovering","last_fresh":"2024-09-13T11:23:52.064402+0000","last_change":"2024-09-13T11:23:33.998502+0000","last_active":"2024-09-13T11:23:52.064402+0000","last_peered":"2024-09-13T11:23:52.064402+0000","last_clean":"2024-09-13T11:23:32.339821+0000","last_became_active":"2024-09-13T11:23:33.989492+0000","last_became_peered":"2024-09-13T11:23:33.989492+0000","last_unstale":"2024-09-13T11:23:52.064402+0000","last_undegraded":"2024-09-13T11:23:52.064402+0000","last_fullsized":"2024-09-13T11:23:52.064402+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:22:24.896055+0000","objects_scrubbed":0,"log_size":3954,"log_dups_size":0,"ondisk_log_size":3954,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":145981440,"num_objects":2228,"num_object_clones":0,"num_object_copies":4456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":189,"num_objects_unfound":0,"num_objects_dirty":2228,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2228,"num_write_kb":142560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1544,"num_bytes_recovered":101187584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7],"acting":[11,7],"avail_no_missing":["11","13"],"object_location_counts":[{"shards":"7,11","objects":2039},{"shards":"11,13","objects":189}],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1b","version":"98'4007","reported_seq":7954,"reported_epoch":98,"state":"active+recovering+remapped","last_fresh":"2024-09-13T11:23:55.341018+0000","last_change":"2024-09-13T11:23:35.097804+0000","last_active":"2024-09-13T11:23:55.341018+0000","last_peered":"2024-09-13T11:23:55.341018+0000","last_clean":"2024-09-13T11:23:32.309147+0000","last_became_active":"2024-09-13T11:23:35.080340+0000","last_became_peered":"2024-09-13T11:23:35.080340+0000","last_unstale":"2024-09-13T11:23:55.341018+0000","last_undegraded":"2024-09-13T11:23:55.341018+0000","last_fullsized":"2024-09-13T11:23:55.341018+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:22:24.896055+0000","objects_scrubbed":0,"log_size":4007,"log_dups_size":0,"ondisk_log_size":4007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":149192704,"num_objects":2277,"num_object_clones":0,"num_object_copies":4554,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":994,"num_objects_unfound":0,"num_objects_dirty":2277,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2277,"num_write_kb":145696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1263,"num_bytes_recovered":82771968,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,15],"acting":[1,12],"avail_no_missing":["1","13"],"object_location_counts":[{"shards":"1,12","objects":1780},{"shards":"1,13","objects":497}],"blocked_by":[],"up_primary":12,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1a","version":"97'3923","reported_seq":7501,"reported_epoch":97,"state":"active+recovering+remapped","last_fresh":"2024-09-13T11:23:55.050047+0000","last_change":"2024-09-13T11:23:35.089105+0000","last_active":"2024-09-13T11:23:55.050047+0000","last_peered":"2024-09-13T11:23:55.050047+0000","last_clean":"2024-09-13T11:23:32.455463+0000","last_became_active":"2024-09-13T11:23:35.072595+0000","last_became_peered":"2024-09-13T11:23:35.072595+0000","last_unstale":"2024-09-13T11:23:55.050047+0000","last_undegraded":"2024-09-13T11:23:55.050047+0000","last_fullsized":"2024-09-13T11:23:55.050047+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:22:24.896055+0000","objects_scrubbed":0,"log_size":3923,"log_dups_size":0,"ondisk_log_size":3923,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":145293312,"num_objects":2217,"num_object_clones":0,"num_object_copies":4434,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1014,"num_objects_unfound":0,"num_objects_dirty":2217,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2217,"num_write_kb":141888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1141,"num_bytes_recovered":74776576,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9],"acting":[3,4],"avail_no_missing":["3","7"],"object_location_counts":[{"shards":"3,4","objects":1710},{"shards":"3,7","objects":507}],"blocked_by":[],"up_primary":4,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.19","version":"94'3533","reported_seq":7812,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-13T11:23:51.943889+0000","last_change":"2024-09-13T11:23:49.429522+0000","last_active":"2024-09-13T11:23:51.943889+0000","last_peered":"2024-09-13T11:23:51.943889+0000","last_clean":"2024-09-13T11:23:51.943889+0000","last_became_active":"2024-09-13T11:23:41.924941+0000","last_became_peered":"2024-09-13T11:23:41.924941+0000","last_unstale":"2024-09-13T11:23:51.943889+0000","last_undegraded":"2024-09-13T11:23:51.943889+0000","last_fullsized":"2024-09-13T11:23:51.943889+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":5,"last_scrub":"92'3415","last_scrub_stamp":"2024-09-13T11:23:49.429429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:49.429429+0000","objects_scrubbed":1979,"log_size":3533,"log_dups_size":0,"ondisk_log_size":3533,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:04.629542+0000","scrub_duration":587,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":138280960,"num_objects":2110,"num_object_clones":0,"num_object_copies":4220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2110,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":131,"num_write_kb":8384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1],"acting":[9,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.18","version":"98'3727","reported_seq":8276,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-09-13T11:23:55.981307+0000","last_change":"2024-09-13T11:23:50.145341+0000","last_active":"2024-09-13T11:23:55.981307+0000","last_peered":"2024-09-13T11:23:55.981307+0000","last_clean":"2024-09-13T11:23:55.981307+0000","last_became_active":"2024-09-13T11:23:48.926014+0000","last_became_peered":"2024-09-13T11:23:48.926014+0000","last_unstale":"2024-09-13T11:23:55.981307+0000","last_undegraded":"2024-09-13T11:23:55.981307+0000","last_fullsized":"2024-09-13T11:23:55.981307+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:22:24.896055+0000","objects_scrubbed":949,"log_size":3727,"log_dups_size":0,"ondisk_log_size":3727,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154664960,"num_objects":2360,"num_object_clones":0,"num_object_copies":4720,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2360,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2360,"num_write_kb":151040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1458,"num_bytes_recovered":95551488,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0],"acting":[10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.17","version":"98'3609","reported_seq":7631,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-09-13T11:23:56.009818+0000","last_change":"2024-09-13T11:23:49.349241+0000","last_active":"2024-09-13T11:23:56.009818+0000","last_peered":"2024-09-13T11:23:56.009818+0000","last_clean":"2024-09-13T11:23:56.009818+0000","last_became_active":"2024-09-13T11:23:42.935601+0000","last_became_peered":"2024-09-13T11:23:42.935601+0000","last_unstale":"2024-09-13T11:23:56.009818+0000","last_undegraded":"2024-09-13T11:23:56.009818+0000","last_fullsized":"2024-09-13T11:23:56.009818+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":5,"last_scrub":"92'3359","last_scrub_stamp":"2024-09-13T11:23:49.349167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:49.349167+0000","objects_scrubbed":2004,"log_size":3609,"log_dups_size":0,"ondisk_log_size":3609,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:24:59.401302+0000","scrub_duration":363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":148242432,"num_objects":2262,"num_object_clones":0,"num_object_copies":4524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2262,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":258,"num_write_kb":16512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3],"acting":[10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"94'3677","reported_seq":8318,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-13T11:23:52.012337+0000","last_change":"2024-09-13T11:23:46.510544+0000","last_active":"2024-09-13T11:23:52.012337+0000","last_peered":"2024-09-13T11:23:52.012337+0000","last_clean":"2024-09-13T11:23:52.012337+0000","last_became_active":"2024-09-13T11:23:46.105237+0000","last_became_peered":"2024-09-13T11:23:46.105237+0000","last_unstale":"2024-09-13T11:23:52.012337+0000","last_undegraded":"2024-09-13T11:23:52.012337+0000","last_fullsized":"2024-09-13T11:23:52.012337+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":5,"last_scrub":"89'3478","last_scrub_stamp":"2024-09-13T11:23:46.510384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:46.510384+0000","objects_scrubbed":2054,"log_size":3677,"log_dups_size":0,"ondisk_log_size":3677,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:09.149878+0000","scrub_duration":363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":147849216,"num_objects":2256,"num_object_clones":0,"num_object_copies":4512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2256,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":202,"num_write_kb":12928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,15],"acting":[11,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.15","version":"98'3655","reported_seq":8066,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-09-13T11:23:56.050031+0000","last_change":"2024-09-13T11:23:48.332790+0000","last_active":"2024-09-13T11:23:56.050031+0000","last_peered":"2024-09-13T11:23:56.050031+0000","last_clean":"2024-09-13T11:23:56.050031+0000","last_became_active":"2024-09-13T11:23:42.931149+0000","last_became_peered":"2024-09-13T11:23:42.931149+0000","last_unstale":"2024-09-13T11:23:56.050031+0000","last_undegraded":"2024-09-13T11:23:56.050031+0000","last_fullsized":"2024-09-13T11:23:56.050031+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":5,"last_scrub":"91'3384","last_scrub_stamp":"2024-09-13T11:23:48.332600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:48.332600+0000","objects_scrubbed":2032,"log_size":3655,"log_dups_size":0,"ondisk_log_size":3655,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:24:58.239508+0000","scrub_duration":235,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":150994944,"num_objects":2304,"num_object_clones":0,"num_object_copies":4608,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2304,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":272,"num_write_kb":17408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,0],"acting":[14,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"3.14","version":"97'3624","reported_seq":7853,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-09-13T11:23:55.014100+0000","last_change":"2024-09-13T11:23:48.697100+0000","last_active":"2024-09-13T11:23:55.014100+0000","last_peered":"2024-09-13T11:23:55.014100+0000","last_clean":"2024-09-13T11:23:55.014100+0000","last_became_active":"2024-09-13T11:23:41.920989+0000","last_became_peered":"2024-09-13T11:23:41.920989+0000","last_unstale":"2024-09-13T11:23:55.014100+0000","last_undegraded":"2024-09-13T11:23:55.014100+0000","last_fullsized":"2024-09-13T11:23:55.014100+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":5,"last_scrub":"91'3371","last_scrub_stamp":"2024-09-13T11:23:48.696960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:48.696960+0000","objects_scrubbed":1935,"log_size":3624,"log_dups_size":0,"ondisk_log_size":3624,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:02.861530+0000","scrub_duration":735,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":144048128,"num_objects":2198,"num_object_clones":0,"num_object_copies":4396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2198,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":263,"num_write_kb":16832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,14],"acting":[3,14],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":149,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-13T11:23:50.149617+0000","last_change":"2024-09-13T11:22:16.889130+0000","last_active":"2024-09-13T11:23:50.149617+0000","last_peered":"2024-09-13T11:23:50.149617+0000","last_clean":"2024-09-13T11:23:50.149617+0000","last_became_active":"2024-09-13T11:22:14.192897+0000","last_became_peered":"2024-09-13T11:22:14.192897+0000","last_unstale":"2024-09-13T11:23:50.149617+0000","last_undegraded":"2024-09-13T11:23:50.149617+0000","last_fullsized":"2024-09-13T11:23:50.149617+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:22:13.113563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00084655899999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,10],"acting":[15,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"3.6","version":"94'3675","reported_seq":3844,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-13T11:23:51.971505+0000","last_change":"2024-09-13T11:23:22.656608+0000","last_active":"2024-09-13T11:23:51.971505+0000","last_peered":"2024-09-13T11:23:51.971505+0000","last_clean":"2024-09-13T11:23:51.971505+0000","last_became_active":"2024-09-13T11:23:19.939432+0000","last_became_peered":"2024-09-13T11:23:19.939432+0000","last_unstale":"2024-09-13T11:23:51.971505+0000","last_undegraded":"2024-09-13T11:23:51.971505+0000","last_fullsized":"2024-09-13T11:23:51.971505+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'2909","last_scrub_stamp":"2024-09-13T11:23:22.656323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:22.656323+0000","objects_scrubbed":1487,"log_size":3675,"log_dups_size":0,"ondisk_log_size":3675,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:24:31.905185+0000","scrub_duration":323,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":147652608,"num_objects":2253,"num_object_clones":0,"num_object_copies":4506,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2253,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":766,"num_write_kb":49024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,14],"acting":[13,14],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":158,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-13T11:23:51.161899+0000","last_change":"2024-09-13T11:23:21.875855+0000","last_active":"2024-09-13T11:23:51.161899+0000","last_peered":"2024-09-13T11:23:51.161899+0000","last_clean":"2024-09-13T11:23:51.161899+0000","last_became_active":"2024-09-13T11:22:14.211923+0000","last_became_peered":"2024-09-13T11:22:14.211923+0000","last_unstale":"2024-09-13T11:23:51.161899+0000","last_undegraded":"2024-09-13T11:23:51.161899+0000","last_fullsized":"2024-09-13T11:23:51.161899+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:23:21.875683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:23:21.875683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:24:40.678136+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0011689529999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0],"acting":[9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"94'3499","reported_seq":3668,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-13T11:23:52.049463+0000","last_change":"2024-09-13T11:23:20.443418+0000","last_active":"2024-09-13T11:23:52.049463+0000","last_peered":"2024-09-13T11:23:52.049463+0000","last_clean":"2024-09-13T11:23:52.049463+0000","last_became_active":"2024-09-13T11:23:19.927680+0000","last_became_peered":"2024-09-13T11:23:19.927680+0000","last_unstale":"2024-09-13T11:23:52.049463+0000","last_undegraded":"2024-09-13T11:23:52.049463+0000","last_fullsized":"2024-09-13T11:23:52.049463+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'2736","last_scrub_stamp":"2024-09-13T11:23:20.443329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:20.443329+0000","objects_scrubbed":1369,"log_size":3499,"log_dups_size":0,"ondisk_log_size":3499,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:24:48.147625+0000","scrub_duration":183,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":139919360,"num_objects":2135,"num_object_clones":0,"num_object_copies":4270,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2135,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":766,"num_write_kb":49024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,8],"acting":[13,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":159,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-09-13T11:23:55.187050+0000","last_change":"2024-09-13T11:22:16.504060+0000","last_active":"2024-09-13T11:23:55.187050+0000","last_peered":"2024-09-13T11:23:55.187050+0000","last_clean":"2024-09-13T11:23:55.187050+0000","last_became_active":"2024-09-13T11:22:14.581458+0000","last_became_peered":"2024-09-13T11:22:14.581458+0000","last_unstale":"2024-09-13T11:23:55.187050+0000","last_undegraded":"2024-09-13T11:23:55.187050+0000","last_fullsized":"2024-09-13T11:23:55.187050+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:22:13.113563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001026047,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5],"acting":[8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"98'3660","reported_seq":3818,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-09-13T11:23:55.280832+0000","last_change":"2024-09-13T11:23:20.254213+0000","last_active":"2024-09-13T11:23:55.280832+0000","last_peered":"2024-09-13T11:23:55.280832+0000","last_clean":"2024-09-13T11:23:55.280832+0000","last_became_active":"2024-09-13T11:23:19.925399+0000","last_became_peered":"2024-09-13T11:23:19.925399+0000","last_unstale":"2024-09-13T11:23:55.280832+0000","last_undegraded":"2024-09-13T11:23:55.280832+0000","last_fullsized":"2024-09-13T11:23:55.280832+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'2755","last_scrub_stamp":"2024-09-13T11:23:20.254124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:20.254124+0000","objects_scrubbed":1437,"log_size":3660,"log_dups_size":0,"ondisk_log_size":3660,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:24:49.993543+0000","scrub_duration":187,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":153812992,"num_objects":2347,"num_object_clones":0,"num_object_copies":4694,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2347,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":910,"num_write_kb":58240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,12],"acting":[1,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":147,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-09-13T11:23:55.188568+0000","last_change":"2024-09-13T11:22:47.720313+0000","last_active":"2024-09-13T11:23:55.188568+0000","last_peered":"2024-09-13T11:23:55.188568+0000","last_clean":"2024-09-13T11:23:55.188568+0000","last_became_active":"2024-09-13T11:22:47.719636+0000","last_became_peered":"2024-09-13T11:22:47.719636+0000","last_unstale":"2024-09-13T11:23:55.188568+0000","last_undegraded":"2024-09-13T11:23:55.188568+0000","last_fullsized":"2024-09-13T11:23:55.188568+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:22:13.113563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00087476000000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,9],"acting":[1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"93'3470","reported_seq":3638,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-13T11:23:51.055058+0000","last_change":"2024-09-13T11:23:20.686446+0000","last_active":"2024-09-13T11:23:51.055058+0000","last_peered":"2024-09-13T11:23:51.055058+0000","last_clean":"2024-09-13T11:23:51.055058+0000","last_became_active":"2024-09-13T11:23:19.939695+0000","last_became_peered":"2024-09-13T11:23:19.939695+0000","last_unstale":"2024-09-13T11:23:51.055058+0000","last_undegraded":"2024-09-13T11:23:51.055058+0000","last_fullsized":"2024-09-13T11:23:51.055058+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'2739","last_scrub_stamp":"2024-09-13T11:23:20.686358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:20.686358+0000","objects_scrubbed":1386,"log_size":3470,"log_dups_size":0,"ondisk_log_size":3470,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:24:38.891737+0000","scrub_duration":271,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":138805248,"num_objects":2118,"num_object_clones":0,"num_object_copies":4236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2118,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":732,"num_write_kb":46848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,13],"acting":[15,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"3.3","version":"98'3381","reported_seq":3582,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-09-13T11:23:55.995850+0000","last_change":"2024-09-13T11:23:32.147115+0000","last_active":"2024-09-13T11:23:55.995850+0000","last_peered":"2024-09-13T11:23:55.995850+0000","last_clean":"2024-09-13T11:23:55.995850+0000","last_became_active":"2024-09-13T11:23:31.930448+0000","last_became_peered":"2024-09-13T11:23:31.930448+0000","last_unstale":"2024-09-13T11:23:55.995850+0000","last_undegraded":"2024-09-13T11:23:55.995850+0000","last_fullsized":"2024-09-13T11:23:55.995850+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'3122","last_scrub_stamp":"2024-09-13T11:23:32.147021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:32.147021+0000","objects_scrubbed":876,"log_size":3381,"log_dups_size":0,"ondisk_log_size":3381,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:24:48.175541+0000","scrub_duration":99,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":74383360,"num_objects":1135,"num_object_clones":0,"num_object_copies":2270,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1135,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":259,"num_write_kb":16576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,7],"acting":[10,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"28'2","reported_seq":89,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-09-13T11:23:55.186561+0000","last_change":"2024-09-13T11:23:12.560311+0000","last_active":"2024-09-13T11:23:55.186561+0000","last_peered":"2024-09-13T11:23:55.186561+0000","last_clean":"2024-09-13T11:23:55.186561+0000","last_became_active":"2024-09-13T11:23:12.546348+0000","last_became_peered":"2024-09-13T11:23:12.546348+0000","last_unstale":"2024-09-13T11:23:55.186561+0000","last_undegraded":"2024-09-13T11:23:55.186561+0000","last_fullsized":"2024-09-13T11:23:55.186561+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:22:13.113563+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0025826730000000002,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8],"acting":[2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"98'3393","reported_seq":3592,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-09-13T11:23:56.027820+0000","last_change":"2024-09-13T11:23:32.749651+0000","last_active":"2024-09-13T11:23:56.027820+0000","last_peered":"2024-09-13T11:23:56.027820+0000","last_clean":"2024-09-13T11:23:56.027820+0000","last_became_active":"2024-09-13T11:23:31.921388+0000","last_became_peered":"2024-09-13T11:23:31.921388+0000","last_unstale":"2024-09-13T11:23:56.027820+0000","last_undegraded":"2024-09-13T11:23:56.027820+0000","last_fullsized":"2024-09-13T11:23:56.027820+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'3106","last_scrub_stamp":"2024-09-13T11:23:32.749487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:32.749487+0000","objects_scrubbed":856,"log_size":3393,"log_dups_size":0,"ondisk_log_size":3393,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:24:34.709501+0000","scrub_duration":91,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":74907648,"num_objects":1143,"num_object_clones":0,"num_object_copies":2286,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1143,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":287,"num_write_kb":18368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,15],"acting":[12,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":169,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-09-13T11:23:55.208037+0000","last_change":"2024-09-13T11:22:47.298206+0000","last_active":"2024-09-13T11:23:55.208037+0000","last_peered":"2024-09-13T11:23:55.208037+0000","last_clean":"2024-09-13T11:23:55.208037+0000","last_became_active":"2024-09-13T11:22:47.297319+0000","last_became_peered":"2024-09-13T11:22:47.297319+0000","last_unstale":"2024-09-13T11:23:55.208037+0000","last_undegraded":"2024-09-13T11:23:55.208037+0000","last_fullsized":"2024-09-13T11:23:55.208037+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:22:13.113563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00081225399999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,1],"acting":[14,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"3.1","version":"94'3295","reported_seq":3486,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-13T11:23:51.940920+0000","last_change":"2024-09-13T11:23:33.604966+0000","last_active":"2024-09-13T11:23:51.940920+0000","last_peered":"2024-09-13T11:23:51.940920+0000","last_clean":"2024-09-13T11:23:51.940920+0000","last_became_active":"2024-09-13T11:23:31.926528+0000","last_became_peered":"2024-09-13T11:23:31.926528+0000","last_unstale":"2024-09-13T11:23:51.940920+0000","last_undegraded":"2024-09-13T11:23:51.940920+0000","last_fullsized":"2024-09-13T11:23:51.940920+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'3070","last_scrub_stamp":"2024-09-13T11:23:33.604792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:33.604792+0000","objects_scrubbed":842,"log_size":3295,"log_dups_size":0,"ondisk_log_size":3295,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:24:43.880038+0000","scrub_duration":119,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":69926912,"num_objects":1067,"num_object_clones":0,"num_object_copies":2134,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1067,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":225,"num_write_kb":14400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,11],"acting":[13,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":157,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-09-13T11:23:54.178449+0000","last_change":"2024-09-13T11:22:16.267999+0000","last_active":"2024-09-13T11:23:54.178449+0000","last_peered":"2024-09-13T11:23:54.178449+0000","last_clean":"2024-09-13T11:23:54.178449+0000","last_became_active":"2024-09-13T11:22:14.211435+0000","last_became_peered":"2024-09-13T11:22:14.211435+0000","last_unstale":"2024-09-13T11:23:54.178449+0000","last_undegraded":"2024-09-13T11:23:54.178449+0000","last_fullsized":"2024-09-13T11:23:54.178449+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:22:13.113563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010010360000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2","version":"98'3382","reported_seq":3582,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-09-13T11:23:55.956606+0000","last_change":"2024-09-13T11:23:32.977666+0000","last_active":"2024-09-13T11:23:55.956606+0000","last_peered":"2024-09-13T11:23:55.956606+0000","last_clean":"2024-09-13T11:23:55.956606+0000","last_became_active":"2024-09-13T11:23:31.919828+0000","last_became_peered":"2024-09-13T11:23:31.919828+0000","last_unstale":"2024-09-13T11:23:55.956606+0000","last_undegraded":"2024-09-13T11:23:55.956606+0000","last_fullsized":"2024-09-13T11:23:55.956606+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'3057","last_scrub_stamp":"2024-09-13T11:23:32.977559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:32.977559+0000","objects_scrubbed":841,"log_size":3382,"log_dups_size":0,"ondisk_log_size":3382,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:24:52.169339+0000","scrub_duration":151,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76414976,"num_objects":1166,"num_object_clones":0,"num_object_copies":2332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1166,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":325,"num_write_kb":20800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,12],"acting":[14,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"2.3","version":"26'1","reported_seq":161,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-09-13T11:23:55.208262+0000","last_change":"2024-09-13T11:22:16.547691+0000","last_active":"2024-09-13T11:23:55.208262+0000","last_peered":"2024-09-13T11:23:55.208262+0000","last_clean":"2024-09-13T11:23:55.208262+0000","last_became_active":"2024-09-13T11:22:14.146507+0000","last_became_peered":"2024-09-13T11:22:14.146507+0000","last_unstale":"2024-09-13T11:23:55.208262+0000","last_undegraded":"2024-09-13T11:23:55.208262+0000","last_fullsized":"2024-09-13T11:23:55.208262+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:22:13.113563+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00085610999999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,15],"acting":[14,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"1.0","version":"26'112","reported_seq":301,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-09-13T11:23:55.197912+0000","last_change":"2024-09-13T11:23:49.325249+0000","last_active":"2024-09-13T11:23:55.197912+0000","last_peered":"2024-09-13T11:23:55.197912+0000","last_clean":"2024-09-13T11:23:55.197912+0000","last_became_active":"2024-09-13T11:22:12.129721+0000","last_became_peered":"2024-09-13T11:22:12.129721+0000","last_unstale":"2024-09-13T11:23:55.197912+0000","last_undegraded":"2024-09-13T11:23:55.197912+0000","last_fullsized":"2024-09-13T11:23:55.197912+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"26'112","last_scrub_stamp":"2024-09-13T11:23:49.325175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:11.098388+0000","last_clean_scrub_stamp":"2024-09-13T11:23:49.325175+0000","objects_scrubbed":2,"log_size":112,"log_dups_size":0,"ondisk_log_size":112,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:14.422287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.8","version":"98'3647","reported_seq":3826,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-09-13T11:23:55.962021+0000","last_change":"2024-09-13T11:23:20.644952+0000","last_active":"2024-09-13T11:23:55.962021+0000","last_peered":"2024-09-13T11:23:55.962021+0000","last_clean":"2024-09-13T11:23:55.962021+0000","last_became_active":"2024-09-13T11:23:19.925802+0000","last_became_peered":"2024-09-13T11:23:19.925802+0000","last_unstale":"2024-09-13T11:23:55.962021+0000","last_undegraded":"2024-09-13T11:23:55.962021+0000","last_fullsized":"2024-09-13T11:23:55.962021+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'2771","last_scrub_stamp":"2024-09-13T11:23:20.644747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:20.644747+0000","objects_scrubbed":1356,"log_size":3647,"log_dups_size":0,"ondisk_log_size":3647,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:24:38.144523+0000","scrub_duration":411,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":147062784,"num_objects":2244,"num_object_clones":0,"num_object_copies":4488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2244,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":888,"num_write_kb":56832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.9","version":"98'3706","reported_seq":3882,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-09-13T11:23:56.067044+0000","last_change":"2024-09-13T11:23:22.033678+0000","last_active":"2024-09-13T11:23:56.067044+0000","last_peered":"2024-09-13T11:23:56.067044+0000","last_clean":"2024-09-13T11:23:56.067044+0000","last_became_active":"2024-09-13T11:23:19.935114+0000","last_became_peered":"2024-09-13T11:23:19.935114+0000","last_unstale":"2024-09-13T11:23:56.067044+0000","last_undegraded":"2024-09-13T11:23:56.067044+0000","last_fullsized":"2024-09-13T11:23:56.067044+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'2808","last_scrub_stamp":"2024-09-13T11:23:22.033572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:22.033572+0000","objects_scrubbed":1471,"log_size":3706,"log_dups_size":0,"ondisk_log_size":3706,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:24:46.595859+0000","scrub_duration":287,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":155254784,"num_objects":2369,"num_object_clones":0,"num_object_copies":4738,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2369,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":898,"num_write_kb":57472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,2],"acting":[12,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[]},{"pgid":"3.a","version":"97'3984","reported_seq":4164,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-09-13T11:23:54.989728+0000","last_change":"2024-09-13T11:23:32.526150+0000","last_active":"2024-09-13T11:23:54.989728+0000","last_peered":"2024-09-13T11:23:54.989728+0000","last_clean":"2024-09-13T11:23:54.989728+0000","last_became_active":"2024-09-13T11:23:31.934170+0000","last_became_peered":"2024-09-13T11:23:31.934170+0000","last_unstale":"2024-09-13T11:23:54.989728+0000","last_undegraded":"2024-09-13T11:23:54.989728+0000","last_fullsized":"2024-09-13T11:23:54.989728+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'3418","last_scrub_stamp":"2024-09-13T11:23:32.526057+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:32.526057+0000","objects_scrubbed":1774,"log_size":3984,"log_dups_size":0,"ondisk_log_size":3984,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:24:34.770161+0000","scrub_duration":507,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":153354240,"num_objects":2340,"num_object_clones":0,"num_object_copies":4680,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2340,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":566,"num_write_kb":36224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.b","version":"94'3905","reported_seq":4078,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-13T11:23:51.951796+0000","last_change":"2024-09-13T11:23:32.681343+0000","last_active":"2024-09-13T11:23:51.951796+0000","last_peered":"2024-09-13T11:23:51.951796+0000","last_clean":"2024-09-13T11:23:51.951796+0000","last_became_active":"2024-09-13T11:23:31.923415+0000","last_became_peered":"2024-09-13T11:23:31.923415+0000","last_unstale":"2024-09-13T11:23:51.951796+0000","last_undegraded":"2024-09-13T11:23:51.951796+0000","last_fullsized":"2024-09-13T11:23:51.951796+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'3471","last_scrub_stamp":"2024-09-13T11:23:32.681257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:32.681257+0000","objects_scrubbed":1720,"log_size":3905,"log_dups_size":0,"ondisk_log_size":3905,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:24:58.857812+0000","scrub_duration":199,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":141164544,"num_objects":2154,"num_object_clones":0,"num_object_copies":4308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2154,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":434,"num_write_kb":27776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,1],"acting":[13,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"3.c","version":"94'3888","reported_seq":4059,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-13T11:23:51.996813+0000","last_change":"2024-09-13T11:23:32.524870+0000","last_active":"2024-09-13T11:23:51.996813+0000","last_peered":"2024-09-13T11:23:51.996813+0000","last_clean":"2024-09-13T11:23:51.996813+0000","last_became_active":"2024-09-13T11:23:31.922620+0000","last_became_peered":"2024-09-13T11:23:31.922620+0000","last_unstale":"2024-09-13T11:23:51.996813+0000","last_undegraded":"2024-09-13T11:23:51.996813+0000","last_fullsized":"2024-09-13T11:23:51.996813+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'3463","last_scrub_stamp":"2024-09-13T11:23:32.524776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:32.524776+0000","objects_scrubbed":1741,"log_size":3888,"log_dups_size":0,"ondisk_log_size":3888,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:24:37.745747+0000","scrub_duration":235,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":141950976,"num_objects":2166,"num_object_clones":0,"num_object_copies":4332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2166,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":425,"num_write_kb":27200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,13],"acting":[11,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.d","version":"98'4003","reported_seq":4182,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-09-13T11:23:56.143293+0000","last_change":"2024-09-13T11:23:32.776835+0000","last_active":"2024-09-13T11:23:56.143293+0000","last_peered":"2024-09-13T11:23:56.143293+0000","last_clean":"2024-09-13T11:23:56.143293+0000","last_became_active":"2024-09-13T11:23:31.939246+0000","last_became_peered":"2024-09-13T11:23:31.939246+0000","last_unstale":"2024-09-13T11:23:56.143293+0000","last_undegraded":"2024-09-13T11:23:56.143293+0000","last_fullsized":"2024-09-13T11:23:56.143293+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'3395","last_scrub_stamp":"2024-09-13T11:23:32.776747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:32.776747+0000","objects_scrubbed":1704,"log_size":4003,"log_dups_size":0,"ondisk_log_size":4003,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:24:56.402862+0000","scrub_duration":179,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":151519232,"num_objects":2312,"num_object_clones":0,"num_object_copies":4624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2312,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":608,"num_write_kb":38912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.e","version":"98'4131","reported_seq":9240,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-09-13T11:23:56.036465+0000","last_change":"2024-09-13T11:23:32.368856+0000","last_active":"2024-09-13T11:23:56.036465+0000","last_peered":"2024-09-13T11:23:56.036465+0000","last_clean":"2024-09-13T11:23:56.036465+0000","last_became_active":"2024-09-13T11:23:31.925869+0000","last_became_peered":"2024-09-13T11:23:31.925869+0000","last_unstale":"2024-09-13T11:23:56.036465+0000","last_undegraded":"2024-09-13T11:23:56.036465+0000","last_fullsized":"2024-09-13T11:23:56.036465+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'3527","last_scrub_stamp":"2024-09-13T11:23:32.368727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:32.368727+0000","objects_scrubbed":1774,"log_size":4131,"log_dups_size":0,"ondisk_log_size":4131,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:24:57.811779+0000","scrub_duration":219,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":155910144,"num_objects":2379,"num_object_clones":0,"num_object_copies":4758,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2379,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":605,"num_write_kb":38720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"94'3835","reported_seq":4007,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-13T11:23:52.022840+0000","last_change":"2024-09-13T11:23:33.492630+0000","last_active":"2024-09-13T11:23:52.022840+0000","last_peered":"2024-09-13T11:23:52.022840+0000","last_clean":"2024-09-13T11:23:52.022840+0000","last_became_active":"2024-09-13T11:23:31.935239+0000","last_became_peered":"2024-09-13T11:23:31.935239+0000","last_unstale":"2024-09-13T11:23:52.022840+0000","last_undegraded":"2024-09-13T11:23:52.022840+0000","last_fullsized":"2024-09-13T11:23:52.022840+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'3420","last_scrub_stamp":"2024-09-13T11:23:33.492488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:33.492488+0000","objects_scrubbed":1776,"log_size":3835,"log_dups_size":0,"ondisk_log_size":3835,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:24:44.851261+0000","scrub_duration":215,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":143589376,"num_objects":2191,"num_object_clones":0,"num_object_copies":4382,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2191,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":415,"num_write_kb":26560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,12],"acting":[11,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.10","version":"98'3756","reported_seq":8091,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-09-13T11:23:56.020141+0000","last_change":"2024-09-13T11:23:48.519654+0000","last_active":"2024-09-13T11:23:56.020141+0000","last_peered":"2024-09-13T11:23:56.020141+0000","last_clean":"2024-09-13T11:23:56.020141+0000","last_became_active":"2024-09-13T11:23:41.931183+0000","last_became_peered":"2024-09-13T11:23:41.931183+0000","last_unstale":"2024-09-13T11:23:56.020141+0000","last_undegraded":"2024-09-13T11:23:56.020141+0000","last_fullsized":"2024-09-13T11:23:56.020141+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":5,"last_scrub":"91'3468","last_scrub_stamp":"2024-09-13T11:23:48.519574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:48.519574+0000","objects_scrubbed":2031,"log_size":3756,"log_dups_size":0,"ondisk_log_size":3756,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:24:57.342213+0000","scrub_duration":215,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":152109056,"num_objects":2321,"num_object_clones":0,"num_object_copies":4642,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2321,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":290,"num_write_kb":18560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.11","version":"98'3682","reported_seq":6795,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-09-13T11:23:55.998860+0000","last_change":"2024-09-13T11:23:46.102466+0000","last_active":"2024-09-13T11:23:55.998860+0000","last_peered":"2024-09-13T11:23:55.998860+0000","last_clean":"2024-09-13T11:23:55.998860+0000","last_became_active":"2024-09-13T11:23:39.063172+0000","last_became_peered":"2024-09-13T11:23:39.063172+0000","last_unstale":"2024-09-13T11:23:55.998860+0000","last_undegraded":"2024-09-13T11:23:55.998860+0000","last_fullsized":"2024-09-13T11:23:55.998860+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":5,"last_scrub":"88'3345","last_scrub_stamp":"2024-09-13T11:23:46.101973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:46.101973+0000","objects_scrubbed":2011,"log_size":3682,"log_dups_size":0,"ondisk_log_size":3682,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:08.671350+0000","scrub_duration":955,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154140672,"num_objects":2352,"num_object_clones":0,"num_object_copies":4704,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2352,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":341,"num_write_kb":21824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11],"acting":[4,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.12","version":"98'3714","reported_seq":6750,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-09-13T11:23:56.015916+0000","last_change":"2024-09-13T11:23:35.111409+0000","last_active":"2024-09-13T11:23:56.015916+0000","last_peered":"2024-09-13T11:23:56.015916+0000","last_clean":"2024-09-13T11:23:56.015916+0000","last_became_active":"2024-09-13T11:23:21.998877+0000","last_became_peered":"2024-09-13T11:23:21.998877+0000","last_unstale":"2024-09-13T11:23:56.015916+0000","last_undegraded":"2024-09-13T11:23:56.015916+0000","last_fullsized":"2024-09-13T11:23:56.015916+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'3109","last_scrub_stamp":"2024-09-13T11:23:35.111320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:35.111320+0000","objects_scrubbed":1732,"log_size":3714,"log_dups_size":0,"ondisk_log_size":3714,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:03.371433+0000","scrub_duration":239,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":153157632,"num_objects":2337,"num_object_clones":0,"num_object_copies":4674,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2337,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":605,"num_write_kb":38720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,7],"acting":[14,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"3.13","version":"94'3570","reported_seq":6601,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-13T11:23:52.055839+0000","last_change":"2024-09-13T11:23:47.565094+0000","last_active":"2024-09-13T11:23:52.055839+0000","last_peered":"2024-09-13T11:23:52.055839+0000","last_clean":"2024-09-13T11:23:52.055839+0000","last_became_active":"2024-09-13T11:23:41.082433+0000","last_became_peered":"2024-09-13T11:23:41.082433+0000","last_unstale":"2024-09-13T11:23:52.055839+0000","last_undegraded":"2024-09-13T11:23:52.055839+0000","last_fullsized":"2024-09-13T11:23:52.055839+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'3408","last_scrub_stamp":"2024-09-13T11:23:47.564931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:47.564931+0000","objects_scrubbed":1981,"log_size":3570,"log_dups_size":0,"ondisk_log_size":3570,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:04.258231+0000","scrub_duration":471,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":141230080,"num_objects":2155,"num_object_clones":0,"num_object_copies":4310,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2155,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":174,"num_write_kb":11136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7],"acting":[11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":36,"stat_sum":{"num_bytes":4738154496,"num_objects":72300,"num_object_clones":0,"num_object_copies":144600,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6760,"num_objects_unfound":0,"num_objects_dirty":72300,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":28315,"num_write_kb":1812064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9884,"num_bytes_recovered":647725056,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":11732451328,"data_stored":11732451328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":132741,"ondisk_log_size":132741,"up":72,"acting":72,"num_store_stats":16},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":10},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":112,"ondisk_log_size":112,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":13,"up_from":23,"seq":98784247832,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1097252,"kb_used_data":1034132,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":93274588,"statfs":{"total":96636764160,"available":95513178112,"internally_reserved":0,"allocated":1058951168,"data_stored":1058533381,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[0,1,3,4,6,8,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":9,"up_from":23,"seq":98784247831,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":323796,"kb_used_data":292612,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94048044,"statfs":{"total":96636764160,"available":96305197056,"internally_reserved":0,"allocated":299634688,"data_stored":299232379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,8,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":40,"seq":171798691855,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1260772,"kb_used_data":1192660,"kb_used_omap":4,"kb_used_meta":68091,"kb_avail":93111068,"statfs":{"total":96636764160,"available":95345733632,"internally_reserved":0,"allocated":1221283840,"data_stored":1220832136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4433,"internal_metadata":69725871},"hb_peers":[0,2,3,6,8,9,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":21,"seq":90194313240,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":901600,"kb_used_data":856528,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":93470240,"statfs":{"total":96636764160,"available":95713525760,"internally_reserved":0,"allocated":877084672,"data_stored":876634785,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":46135751},"hb_peers":[0,2,4,7,9,10,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":21,"seq":90194313243,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":234152,"kb_used_data":194648,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94137688,"statfs":{"total":96636764160,"available":96396992512,"internally_reserved":0,"allocated":199319552,"data_stored":198865634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40434119},"hb_peers":[0,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":22,"seq":94489280535,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1015504,"kb_used_data":944064,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":93356336,"statfs":{"total":96636764160,"available":95596888064,"internally_reserved":0,"allocated":966721536,"data_stored":966345025,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73136581},"hb_peers":[1,3,4,5,8,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":22,"seq":94489280536,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":395228,"kb_used_data":359884,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93976612,"statfs":{"total":96636764160,"available":96232050688,"internally_reserved":0,"allocated":368521216,"data_stored":368083624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,7,8,9,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":15,"up_from":22,"seq":94489280535,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":751456,"kb_used_data":693904,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":93620384,"statfs":{"total":96636764160,"available":95867273216,"internally_reserved":0,"allocated":710557696,"data_stored":710200511,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58915270},"hb_peers":[0,3,4,5,6,7,10,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":22,"seq":94489280536,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":422956,"kb_used_data":383452,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":93948884,"statfs":{"total":96636764160,"available":96203657216,"internally_reserved":0,"allocated":392654848,"data_stored":392202303,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[3,5,7,9,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":22,"seq":94489280536,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":404284,"kb_used_data":368940,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93967556,"statfs":{"total":96636764160,"available":96222777344,"internally_reserved":0,"allocated":377794560,"data_stored":377384292,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,3,5,7,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":22,"seq":94489280536,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1161328,"kb_used_data":1081568,"kb_used_omap":1,"kb_used_meta":79742,"kb_avail":93210512,"statfs":{"total":96636764160,"available":95447564288,"internally_reserved":0,"allocated":1107525632,"data_stored":1107125067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":81656266},"hb_peers":[0,3,5,6,8,9,10,11,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":22,"seq":94489280536,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":485428,"kb_used_data":458404,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93886412,"statfs":{"total":96636764160,"available":96139685888,"internally_reserved":0,"allocated":469405696,"data_stored":468941270,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,5,7,9,10,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":22,"seq":94489280535,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":588460,"kb_used_data":557276,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93783380,"statfs":{"total":96636764160,"available":96034181120,"internally_reserved":0,"allocated":570650624,"data_stored":570248918,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,3,6,7,9,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":11,"up_from":22,"seq":94489280536,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1159964,"kb_used_data":1096844,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":93211876,"statfs":{"total":96636764160,"available":95448961024,"internally_reserved":0,"allocated":1123168256,"data_stored":1122761730,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":64616900},"hb_peers":[0,2,4,7,8,10,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,0,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":12,"up_from":22,"seq":94489280536,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":970648,"kb_used_data":899208,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":93401192,"statfs":{"total":96636764160,"available":95642820608,"internally_reserved":0,"allocated":920788992,"data_stored":920352163,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73136581},"hb_peers":[0,1,2,3,5,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":14,"up_from":22,"seq":94489280536,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1047512,"kb_used_data":971912,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":93324328,"statfs":{"total":96636764160,"available":95564111872,"internally_reserved":0,"allocated":995237888,"data_stored":994799398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,1,5,6,7,8,9,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":986972160,"data_stored":986972160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1222311936,"data_stored":1222311936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":367132672,"data_stored":367132672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":878641152,"data_stored":878641152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":391577600,"data_stored":391577600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":199098368,"data_stored":199098368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":379125760,"data_stored":379125760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1111490560,"data_stored":1111490560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":468844544,"data_stored":468844544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":299565056,"data_stored":299565056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":578420736,"data_stored":578420736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":1140719616,"data_stored":1140719616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":928645120,"data_stored":928645120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":1066074112,"data_stored":1066074112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":993460224,"data_stored":993460224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":720371712,"data_stored":720371712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T11:23:56.682 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-13T11:23:56.765 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-09-13T11:23:56.761+0000 7f97befc9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26315) UID: 0 2024-09-13T11:23:56.772 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-13T11:23:56.773 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:23:56.791 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_blocked_ops 2024-09-13T11:23:56.878 INFO:tasks.ceph.osd.10.smithi162.stderr:2024-09-13T11:23:56.877+0000 7fb59c9f0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26317) UID: 0 2024-09-13T11:23:56.912 INFO:tasks.ceph.osd.3.smithi177.stderr:2024-09-13T11:23:56.904+0000 7f10a45d5640 -1 osd.3 99 handle_osd_map injecting map crc failure 2024-09-13T11:23:56.912 INFO:tasks.ceph.osd.2.smithi162.stderr:2024-09-13T11:23:56.909+0000 7fe921d8b640 -1 osd.2 99 handle_osd_map injecting map crc failure 2024-09-13T11:23:56.918 INFO:tasks.ceph.osd.14.smithi162.stderr:2024-09-13T11:23:56.917+0000 7f2350b08640 -1 osd.14 99 handle_osd_map injecting map crc failure 2024-09-13T11:23:56.967 INFO:tasks.ceph.osd.1.smithi092.stderr:2024-09-13T11:23:56.963+0000 7fe7789c5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33718) UID: 0 2024-09-13T11:23:56.968 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-13T11:23:56.968 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:23:56.984 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_historic_ops 2024-09-13T11:23:57.007 INFO:tasks.radosbench.radosbench.0.smithi177.stdout: 90 16 75457 75441 52.3842 77.9375 0.00301512 0.0190837 2024-09-13T11:23:57.068 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-13T11:23:57.060+0000 7f6a68775640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26307) UID: 0 2024-09-13T11:23:57.119 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-13T11:23:57.120 INFO:teuthology.orchestra.run.smithi001.stderr:dumped all 2024-09-13T11:23:57.143 INFO:teuthology.orchestra.run.smithi001.stdout:{"pg_ready":true,"pg_map":{"version":139,"stamp":"2024-09-13T11:23:56.911242+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4752704051,"num_objects":72517,"num_object_clones":0,"num_object_copies":145034,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6760,"num_objects_unfound":0,"num_objects_dirty":72517,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":28675,"num_write_kb":1828370,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9885,"num_bytes_recovered":647725075,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":133069,"ondisk_log_size":133069,"up":90,"acting":90,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":119,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":16,"kb":1509949440,"kb_used":12306928,"kb_used_data":11467056,"kb_used_omap":27,"kb_used_meta":839588,"kb_avail":1497642512,"statfs":{"total":1546188226560,"available":1533585932288,"internally_reserved":0,"allocated":11742265344,"data_stored":11735483421,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28358,"internal_metadata":859738426},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,2,1,2,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":261685248,"num_objects":3993,"num_object_clones":0,"num_object_copies":7986,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-3061,"num_objects_unfound":0,"num_objects_dirty":3993,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3993,"num_write_kb":255552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1531,"num_bytes_recovered":100335616,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"3.738414"},"pg_stats":[{"pgid":"3.23","version":"98'3432","reported_seq":6144,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-09-13T11:23:55.908873+0000","last_change":"2024-09-13T11:23:48.133184+0000","last_active":"2024-09-13T11:23:55.908873+0000","last_peered":"2024-09-13T11:23:55.908873+0000","last_clean":"2024-09-13T11:23:55.908873+0000","last_became_active":"2024-09-13T11:23:47.926677+0000","last_became_peered":"2024-09-13T11:23:47.926677+0000","last_unstale":"2024-09-13T11:23:55.908873+0000","last_undegraded":"2024-09-13T11:23:55.908873+0000","last_fullsized":"2024-09-13T11:23:55.908873+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":6,"last_scrub":"91'3278","last_scrub_stamp":"2024-09-13T11:23:48.133084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:48.133084+0000","objects_scrubbed":1029,"log_size":3432,"log_dups_size":0,"ondisk_log_size":3432,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:12.874134+0000","scrub_duration":139,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":77594624,"num_objects":1184,"num_object_clones":0,"num_object_copies":2368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1184,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":155,"num_write_kb":9920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9],"acting":[5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.22","version":"98'3408","reported_seq":5307,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-09-13T11:23:56.014918+0000","last_change":"2024-09-13T11:23:47.300446+0000","last_active":"2024-09-13T11:23:56.014918+0000","last_peered":"2024-09-13T11:23:56.014918+0000","last_clean":"2024-09-13T11:23:56.014918+0000","last_became_active":"2024-09-13T11:23:46.925318+0000","last_became_peered":"2024-09-13T11:23:46.925318+0000","last_unstale":"2024-09-13T11:23:56.014918+0000","last_undegraded":"2024-09-13T11:23:56.014918+0000","last_fullsized":"2024-09-13T11:23:56.014918+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":6,"last_scrub":"90'3246","last_scrub_stamp":"2024-09-13T11:23:47.300339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:47.300339+0000","objects_scrubbed":1033,"log_size":3408,"log_dups_size":0,"ondisk_log_size":3408,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:24:49.780485+0000","scrub_duration":203,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":78446592,"num_objects":1197,"num_object_clones":0,"num_object_copies":2394,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1197,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":164,"num_write_kb":10496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,14],"acting":[0,14],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.21","version":"94'3286","reported_seq":4757,"reported_epoch":94,"state":"active+recovering+remapped","last_fresh":"2024-09-13T11:23:52.020283+0000","last_change":"2024-09-13T11:23:45.861514+0000","last_active":"2024-09-13T11:23:52.020283+0000","last_peered":"2024-09-13T11:23:52.020283+0000","last_clean":"2024-09-13T11:23:32.050222+0000","last_became_active":"2024-09-13T11:23:35.026171+0000","last_became_peered":"2024-09-13T11:23:35.026171+0000","last_unstale":"2024-09-13T11:23:52.020283+0000","last_undegraded":"2024-09-13T11:23:52.020283+0000","last_fullsized":"2024-09-13T11:23:52.020283+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":6,"last_scrub":"69'2784","last_scrub_stamp":"2024-09-13T11:23:21.595419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:21.595419+0000","objects_scrubbed":1386,"log_size":3286,"log_dups_size":0,"ondisk_log_size":3286,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":311,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":69074944,"num_objects":1054,"num_object_clones":0,"num_object_copies":2108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":810,"num_objects_unfound":0,"num_objects_dirty":1054,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":361,"num_write_kb":23104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":431,"num_bytes_recovered":28246016,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8],"acting":[11,0],"avail_no_missing":["11","13"],"object_location_counts":[{"shards":"0,11","objects":649},{"shards":"11,13","objects":405}],"blocked_by":[],"up_primary":0,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.20","version":"98'3426","reported_seq":5440,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-09-13T11:23:56.029766+0000","last_change":"2024-09-13T11:23:47.969503+0000","last_active":"2024-09-13T11:23:56.029766+0000","last_peered":"2024-09-13T11:23:56.029766+0000","last_clean":"2024-09-13T11:23:56.029766+0000","last_became_active":"2024-09-13T11:23:33.954946+0000","last_became_peered":"2024-09-13T11:23:33.954946+0000","last_unstale":"2024-09-13T11:23:56.029766+0000","last_undegraded":"2024-09-13T11:23:56.029766+0000","last_fullsized":"2024-09-13T11:23:56.029766+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":6,"last_scrub":"91'3270","last_scrub_stamp":"2024-09-13T11:23:47.969372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:47.969372+0000","objects_scrubbed":1061,"log_size":3426,"log_dups_size":0,"ondisk_log_size":3426,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:24:48.255263+0000","scrub_duration":1327,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":80084992,"num_objects":1222,"num_object_clones":0,"num_object_copies":2444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1222,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":161,"num_write_kb":10304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,0],"acting":[12,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[]},{"pgid":"3.1f","version":"94'3850","reported_seq":6144,"reported_epoch":94,"state":"active+recovering+remapped","last_fresh":"2024-09-13T11:23:52.060704+0000","last_change":"2024-09-13T11:23:37.329722+0000","last_active":"2024-09-13T11:23:52.060704+0000","last_peered":"2024-09-13T11:23:52.060704+0000","last_clean":"2024-09-13T11:23:32.316727+0000","last_became_active":"2024-09-13T11:23:35.053803+0000","last_became_peered":"2024-09-13T11:23:35.053803+0000","last_unstale":"2024-09-13T11:23:52.060704+0000","last_undegraded":"2024-09-13T11:23:52.060704+0000","last_fullsized":"2024-09-13T11:23:52.060704+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:22:24.896055+0000","objects_scrubbed":0,"log_size":3850,"log_dups_size":0,"ondisk_log_size":3850,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":140476416,"num_objects":2144,"num_object_clones":0,"num_object_copies":4288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1872,"num_objects_unfound":0,"num_objects_dirty":2144,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2144,"num_write_kb":137184,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":715,"num_bytes_recovered":46858240,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,14],"acting":[11,10],"avail_no_missing":["11","12"],"object_location_counts":[{"shards":"10,11","objects":1208},{"shards":"11,12","objects":936}],"blocked_by":[],"up_primary":10,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1e","version":"98'4129","reported_seq":12953,"reported_epoch":98,"state":"active+recovering+remapped","last_fresh":"2024-09-13T11:23:56.078942+0000","last_change":"2024-09-13T11:23:35.106919+0000","last_active":"2024-09-13T11:23:56.078942+0000","last_peered":"2024-09-13T11:23:56.078942+0000","last_clean":"2024-09-13T11:23:30.974769+0000","last_became_active":"2024-09-13T11:23:35.075137+0000","last_became_peered":"2024-09-13T11:23:35.075137+0000","last_unstale":"2024-09-13T11:23:56.078942+0000","last_undegraded":"2024-09-13T11:23:56.078942+0000","last_fullsized":"2024-09-13T11:23:56.078942+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:22:24.896055+0000","objects_scrubbed":0,"log_size":4129,"log_dups_size":0,"ondisk_log_size":4129,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":155254784,"num_objects":2369,"num_object_clones":0,"num_object_copies":4738,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1005,"num_objects_unfound":0,"num_objects_dirty":2369,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2369,"num_write_kb":151616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2070,"num_bytes_recovered":135626752,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[0,4],"avail_no_missing":["0","1","8","14"],"object_location_counts":[{"shards":"0,1","objects":502},{"shards":"0,1,4","objects":1},{"shards":"0,4","objects":1866}],"blocked_by":[],"up_primary":5,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1d","version":"97'3925","reported_seq":7866,"reported_epoch":97,"state":"active+recovering+remapped","last_fresh":"2024-09-13T11:23:55.002071+0000","last_change":"2024-09-13T11:23:35.096108+0000","last_active":"2024-09-13T11:23:55.002071+0000","last_peered":"2024-09-13T11:23:55.002071+0000","last_clean":"2024-09-13T11:23:32.393844+0000","last_became_active":"2024-09-13T11:23:35.080149+0000","last_became_peered":"2024-09-13T11:23:35.080149+0000","last_unstale":"2024-09-13T11:23:55.002071+0000","last_undegraded":"2024-09-13T11:23:55.002071+0000","last_fullsized":"2024-09-13T11:23:55.002071+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:22:24.896055+0000","objects_scrubbed":0,"log_size":3925,"log_dups_size":0,"ondisk_log_size":3925,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":146407424,"num_objects":2234,"num_object_clones":0,"num_object_copies":4468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":876,"num_objects_unfound":0,"num_objects_dirty":2234,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2234,"num_write_kb":142976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1262,"num_bytes_recovered":82706432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,15],"acting":[3,2],"avail_no_missing":["3","7"],"object_location_counts":[{"shards":"2,3","objects":1796},{"shards":"3,7","objects":438}],"blocked_by":[],"up_primary":2,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1c","version":"94'3954","reported_seq":7192,"reported_epoch":94,"state":"active+recovering","last_fresh":"2024-09-13T11:23:52.064402+0000","last_change":"2024-09-13T11:23:33.998502+0000","last_active":"2024-09-13T11:23:52.064402+0000","last_peered":"2024-09-13T11:23:52.064402+0000","last_clean":"2024-09-13T11:23:32.339821+0000","last_became_active":"2024-09-13T11:23:33.989492+0000","last_became_peered":"2024-09-13T11:23:33.989492+0000","last_unstale":"2024-09-13T11:23:52.064402+0000","last_undegraded":"2024-09-13T11:23:52.064402+0000","last_fullsized":"2024-09-13T11:23:52.064402+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:22:24.896055+0000","objects_scrubbed":0,"log_size":3954,"log_dups_size":0,"ondisk_log_size":3954,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":145981440,"num_objects":2228,"num_object_clones":0,"num_object_copies":4456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":189,"num_objects_unfound":0,"num_objects_dirty":2228,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2228,"num_write_kb":142560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1544,"num_bytes_recovered":101187584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7],"acting":[11,7],"avail_no_missing":["11","13"],"object_location_counts":[{"shards":"7,11","objects":2039},{"shards":"11,13","objects":189}],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1b","version":"98'4007","reported_seq":7954,"reported_epoch":98,"state":"active+recovering+remapped","last_fresh":"2024-09-13T11:23:55.341018+0000","last_change":"2024-09-13T11:23:35.097804+0000","last_active":"2024-09-13T11:23:55.341018+0000","last_peered":"2024-09-13T11:23:55.341018+0000","last_clean":"2024-09-13T11:23:32.309147+0000","last_became_active":"2024-09-13T11:23:35.080340+0000","last_became_peered":"2024-09-13T11:23:35.080340+0000","last_unstale":"2024-09-13T11:23:55.341018+0000","last_undegraded":"2024-09-13T11:23:55.341018+0000","last_fullsized":"2024-09-13T11:23:55.341018+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:22:24.896055+0000","objects_scrubbed":0,"log_size":4007,"log_dups_size":0,"ondisk_log_size":4007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":149192704,"num_objects":2277,"num_object_clones":0,"num_object_copies":4554,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":994,"num_objects_unfound":0,"num_objects_dirty":2277,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2277,"num_write_kb":145696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1263,"num_bytes_recovered":82771968,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,15],"acting":[1,12],"avail_no_missing":["1","13"],"object_location_counts":[{"shards":"1,12","objects":1780},{"shards":"1,13","objects":497}],"blocked_by":[],"up_primary":12,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1a","version":"97'3923","reported_seq":7501,"reported_epoch":97,"state":"active+recovering+remapped","last_fresh":"2024-09-13T11:23:55.050047+0000","last_change":"2024-09-13T11:23:35.089105+0000","last_active":"2024-09-13T11:23:55.050047+0000","last_peered":"2024-09-13T11:23:55.050047+0000","last_clean":"2024-09-13T11:23:32.455463+0000","last_became_active":"2024-09-13T11:23:35.072595+0000","last_became_peered":"2024-09-13T11:23:35.072595+0000","last_unstale":"2024-09-13T11:23:55.050047+0000","last_undegraded":"2024-09-13T11:23:55.050047+0000","last_fullsized":"2024-09-13T11:23:55.050047+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:22:24.896055+0000","objects_scrubbed":0,"log_size":3923,"log_dups_size":0,"ondisk_log_size":3923,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":145293312,"num_objects":2217,"num_object_clones":0,"num_object_copies":4434,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1014,"num_objects_unfound":0,"num_objects_dirty":2217,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2217,"num_write_kb":141888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1141,"num_bytes_recovered":74776576,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9],"acting":[3,4],"avail_no_missing":["3","7"],"object_location_counts":[{"shards":"3,4","objects":1710},{"shards":"3,7","objects":507}],"blocked_by":[],"up_primary":4,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.19","version":"94'3533","reported_seq":7812,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-13T11:23:51.943889+0000","last_change":"2024-09-13T11:23:49.429522+0000","last_active":"2024-09-13T11:23:51.943889+0000","last_peered":"2024-09-13T11:23:51.943889+0000","last_clean":"2024-09-13T11:23:51.943889+0000","last_became_active":"2024-09-13T11:23:41.924941+0000","last_became_peered":"2024-09-13T11:23:41.924941+0000","last_unstale":"2024-09-13T11:23:51.943889+0000","last_undegraded":"2024-09-13T11:23:51.943889+0000","last_fullsized":"2024-09-13T11:23:51.943889+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":5,"last_scrub":"92'3415","last_scrub_stamp":"2024-09-13T11:23:49.429429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:49.429429+0000","objects_scrubbed":1979,"log_size":3533,"log_dups_size":0,"ondisk_log_size":3533,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:04.629542+0000","scrub_duration":587,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":138280960,"num_objects":2110,"num_object_clones":0,"num_object_copies":4220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2110,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":131,"num_write_kb":8384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1],"acting":[9,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.18","version":"98'3727","reported_seq":8276,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-09-13T11:23:55.981307+0000","last_change":"2024-09-13T11:23:50.145341+0000","last_active":"2024-09-13T11:23:55.981307+0000","last_peered":"2024-09-13T11:23:55.981307+0000","last_clean":"2024-09-13T11:23:55.981307+0000","last_became_active":"2024-09-13T11:23:48.926014+0000","last_became_peered":"2024-09-13T11:23:48.926014+0000","last_unstale":"2024-09-13T11:23:55.981307+0000","last_undegraded":"2024-09-13T11:23:55.981307+0000","last_fullsized":"2024-09-13T11:23:55.981307+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:22:24.896055+0000","objects_scrubbed":949,"log_size":3727,"log_dups_size":0,"ondisk_log_size":3727,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154664960,"num_objects":2360,"num_object_clones":0,"num_object_copies":4720,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2360,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2360,"num_write_kb":151040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1458,"num_bytes_recovered":95551488,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0],"acting":[10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.17","version":"98'3609","reported_seq":7631,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-09-13T11:23:56.009818+0000","last_change":"2024-09-13T11:23:49.349241+0000","last_active":"2024-09-13T11:23:56.009818+0000","last_peered":"2024-09-13T11:23:56.009818+0000","last_clean":"2024-09-13T11:23:56.009818+0000","last_became_active":"2024-09-13T11:23:42.935601+0000","last_became_peered":"2024-09-13T11:23:42.935601+0000","last_unstale":"2024-09-13T11:23:56.009818+0000","last_undegraded":"2024-09-13T11:23:56.009818+0000","last_fullsized":"2024-09-13T11:23:56.009818+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":5,"last_scrub":"92'3359","last_scrub_stamp":"2024-09-13T11:23:49.349167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:49.349167+0000","objects_scrubbed":2004,"log_size":3609,"log_dups_size":0,"ondisk_log_size":3609,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:24:59.401302+0000","scrub_duration":363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":148242432,"num_objects":2262,"num_object_clones":0,"num_object_copies":4524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2262,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":258,"num_write_kb":16512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3],"acting":[10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"94'3677","reported_seq":8318,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-13T11:23:52.012337+0000","last_change":"2024-09-13T11:23:46.510544+0000","last_active":"2024-09-13T11:23:52.012337+0000","last_peered":"2024-09-13T11:23:52.012337+0000","last_clean":"2024-09-13T11:23:52.012337+0000","last_became_active":"2024-09-13T11:23:46.105237+0000","last_became_peered":"2024-09-13T11:23:46.105237+0000","last_unstale":"2024-09-13T11:23:52.012337+0000","last_undegraded":"2024-09-13T11:23:52.012337+0000","last_fullsized":"2024-09-13T11:23:52.012337+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":5,"last_scrub":"89'3478","last_scrub_stamp":"2024-09-13T11:23:46.510384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:46.510384+0000","objects_scrubbed":2054,"log_size":3677,"log_dups_size":0,"ondisk_log_size":3677,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:09.149878+0000","scrub_duration":363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":147849216,"num_objects":2256,"num_object_clones":0,"num_object_copies":4512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2256,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":202,"num_write_kb":12928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,15],"acting":[11,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.15","version":"98'3655","reported_seq":8066,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-09-13T11:23:56.050031+0000","last_change":"2024-09-13T11:23:48.332790+0000","last_active":"2024-09-13T11:23:56.050031+0000","last_peered":"2024-09-13T11:23:56.050031+0000","last_clean":"2024-09-13T11:23:56.050031+0000","last_became_active":"2024-09-13T11:23:42.931149+0000","last_became_peered":"2024-09-13T11:23:42.931149+0000","last_unstale":"2024-09-13T11:23:56.050031+0000","last_undegraded":"2024-09-13T11:23:56.050031+0000","last_fullsized":"2024-09-13T11:23:56.050031+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":5,"last_scrub":"91'3384","last_scrub_stamp":"2024-09-13T11:23:48.332600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:48.332600+0000","objects_scrubbed":2032,"log_size":3655,"log_dups_size":0,"ondisk_log_size":3655,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:24:58.239508+0000","scrub_duration":235,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":150994944,"num_objects":2304,"num_object_clones":0,"num_object_copies":4608,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2304,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":272,"num_write_kb":17408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,0],"acting":[14,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"3.14","version":"97'3624","reported_seq":7853,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-09-13T11:23:55.014100+0000","last_change":"2024-09-13T11:23:48.697100+0000","last_active":"2024-09-13T11:23:55.014100+0000","last_peered":"2024-09-13T11:23:55.014100+0000","last_clean":"2024-09-13T11:23:55.014100+0000","last_became_active":"2024-09-13T11:23:41.920989+0000","last_became_peered":"2024-09-13T11:23:41.920989+0000","last_unstale":"2024-09-13T11:23:55.014100+0000","last_undegraded":"2024-09-13T11:23:55.014100+0000","last_fullsized":"2024-09-13T11:23:55.014100+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":5,"last_scrub":"91'3371","last_scrub_stamp":"2024-09-13T11:23:48.696960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:48.696960+0000","objects_scrubbed":1935,"log_size":3624,"log_dups_size":0,"ondisk_log_size":3624,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:02.861530+0000","scrub_duration":735,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":144048128,"num_objects":2198,"num_object_clones":0,"num_object_copies":4396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2198,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":263,"num_write_kb":16832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,14],"acting":[3,14],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":159,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-09-13T11:23:55.198217+0000","last_change":"2024-09-13T11:22:16.889130+0000","last_active":"2024-09-13T11:23:55.198217+0000","last_peered":"2024-09-13T11:23:55.198217+0000","last_clean":"2024-09-13T11:23:55.198217+0000","last_became_active":"2024-09-13T11:22:14.192897+0000","last_became_peered":"2024-09-13T11:22:14.192897+0000","last_unstale":"2024-09-13T11:23:55.198217+0000","last_undegraded":"2024-09-13T11:23:55.198217+0000","last_fullsized":"2024-09-13T11:23:55.198217+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:22:13.113563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00084655899999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,10],"acting":[15,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"3.6","version":"94'3675","reported_seq":3844,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-13T11:23:51.971505+0000","last_change":"2024-09-13T11:23:22.656608+0000","last_active":"2024-09-13T11:23:51.971505+0000","last_peered":"2024-09-13T11:23:51.971505+0000","last_clean":"2024-09-13T11:23:51.971505+0000","last_became_active":"2024-09-13T11:23:19.939432+0000","last_became_peered":"2024-09-13T11:23:19.939432+0000","last_unstale":"2024-09-13T11:23:51.971505+0000","last_undegraded":"2024-09-13T11:23:51.971505+0000","last_fullsized":"2024-09-13T11:23:51.971505+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'2909","last_scrub_stamp":"2024-09-13T11:23:22.656323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:22.656323+0000","objects_scrubbed":1487,"log_size":3675,"log_dups_size":0,"ondisk_log_size":3675,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:24:31.905185+0000","scrub_duration":323,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":147652608,"num_objects":2253,"num_object_clones":0,"num_object_copies":4506,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2253,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":766,"num_write_kb":49024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,14],"acting":[13,14],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":158,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-13T11:23:51.161899+0000","last_change":"2024-09-13T11:23:21.875855+0000","last_active":"2024-09-13T11:23:51.161899+0000","last_peered":"2024-09-13T11:23:51.161899+0000","last_clean":"2024-09-13T11:23:51.161899+0000","last_became_active":"2024-09-13T11:22:14.211923+0000","last_became_peered":"2024-09-13T11:22:14.211923+0000","last_unstale":"2024-09-13T11:23:51.161899+0000","last_undegraded":"2024-09-13T11:23:51.161899+0000","last_fullsized":"2024-09-13T11:23:51.161899+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:23:21.875683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:23:21.875683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:24:40.678136+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0011689529999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0],"acting":[9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"94'3499","reported_seq":3668,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-13T11:23:52.049463+0000","last_change":"2024-09-13T11:23:20.443418+0000","last_active":"2024-09-13T11:23:52.049463+0000","last_peered":"2024-09-13T11:23:52.049463+0000","last_clean":"2024-09-13T11:23:52.049463+0000","last_became_active":"2024-09-13T11:23:19.927680+0000","last_became_peered":"2024-09-13T11:23:19.927680+0000","last_unstale":"2024-09-13T11:23:52.049463+0000","last_undegraded":"2024-09-13T11:23:52.049463+0000","last_fullsized":"2024-09-13T11:23:52.049463+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'2736","last_scrub_stamp":"2024-09-13T11:23:20.443329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:20.443329+0000","objects_scrubbed":1369,"log_size":3499,"log_dups_size":0,"ondisk_log_size":3499,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:24:48.147625+0000","scrub_duration":183,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":139919360,"num_objects":2135,"num_object_clones":0,"num_object_copies":4270,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2135,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":766,"num_write_kb":49024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,8],"acting":[13,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":159,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-09-13T11:23:55.187050+0000","last_change":"2024-09-13T11:22:16.504060+0000","last_active":"2024-09-13T11:23:55.187050+0000","last_peered":"2024-09-13T11:23:55.187050+0000","last_clean":"2024-09-13T11:23:55.187050+0000","last_became_active":"2024-09-13T11:22:14.581458+0000","last_became_peered":"2024-09-13T11:22:14.581458+0000","last_unstale":"2024-09-13T11:23:55.187050+0000","last_undegraded":"2024-09-13T11:23:55.187050+0000","last_fullsized":"2024-09-13T11:23:55.187050+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:22:13.113563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001026047,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5],"acting":[8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"98'3660","reported_seq":3818,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-09-13T11:23:55.280832+0000","last_change":"2024-09-13T11:23:20.254213+0000","last_active":"2024-09-13T11:23:55.280832+0000","last_peered":"2024-09-13T11:23:55.280832+0000","last_clean":"2024-09-13T11:23:55.280832+0000","last_became_active":"2024-09-13T11:23:19.925399+0000","last_became_peered":"2024-09-13T11:23:19.925399+0000","last_unstale":"2024-09-13T11:23:55.280832+0000","last_undegraded":"2024-09-13T11:23:55.280832+0000","last_fullsized":"2024-09-13T11:23:55.280832+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'2755","last_scrub_stamp":"2024-09-13T11:23:20.254124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:20.254124+0000","objects_scrubbed":1437,"log_size":3660,"log_dups_size":0,"ondisk_log_size":3660,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:24:49.993543+0000","scrub_duration":187,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":153812992,"num_objects":2347,"num_object_clones":0,"num_object_copies":4694,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2347,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":910,"num_write_kb":58240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,12],"acting":[1,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":147,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-09-13T11:23:55.188568+0000","last_change":"2024-09-13T11:22:47.720313+0000","last_active":"2024-09-13T11:23:55.188568+0000","last_peered":"2024-09-13T11:23:55.188568+0000","last_clean":"2024-09-13T11:23:55.188568+0000","last_became_active":"2024-09-13T11:22:47.719636+0000","last_became_peered":"2024-09-13T11:22:47.719636+0000","last_unstale":"2024-09-13T11:23:55.188568+0000","last_undegraded":"2024-09-13T11:23:55.188568+0000","last_fullsized":"2024-09-13T11:23:55.188568+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:22:13.113563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00087476000000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,9],"acting":[1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"98'3683","reported_seq":3860,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-09-13T11:23:56.040162+0000","last_change":"2024-09-13T11:23:20.686446+0000","last_active":"2024-09-13T11:23:56.040162+0000","last_peered":"2024-09-13T11:23:56.040162+0000","last_clean":"2024-09-13T11:23:56.040162+0000","last_became_active":"2024-09-13T11:23:19.939695+0000","last_became_peered":"2024-09-13T11:23:19.939695+0000","last_unstale":"2024-09-13T11:23:56.040162+0000","last_undegraded":"2024-09-13T11:23:56.040162+0000","last_fullsized":"2024-09-13T11:23:56.040162+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'2739","last_scrub_stamp":"2024-09-13T11:23:20.686358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:20.686358+0000","objects_scrubbed":1386,"log_size":3683,"log_dups_size":0,"ondisk_log_size":3683,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:24:38.891737+0000","scrub_duration":271,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":152764416,"num_objects":2331,"num_object_clones":0,"num_object_copies":4662,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2331,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":945,"num_write_kb":60480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,13],"acting":[15,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"3.3","version":"98'3381","reported_seq":3582,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-09-13T11:23:55.995850+0000","last_change":"2024-09-13T11:23:32.147115+0000","last_active":"2024-09-13T11:23:55.995850+0000","last_peered":"2024-09-13T11:23:55.995850+0000","last_clean":"2024-09-13T11:23:55.995850+0000","last_became_active":"2024-09-13T11:23:31.930448+0000","last_became_peered":"2024-09-13T11:23:31.930448+0000","last_unstale":"2024-09-13T11:23:55.995850+0000","last_undegraded":"2024-09-13T11:23:55.995850+0000","last_fullsized":"2024-09-13T11:23:55.995850+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'3122","last_scrub_stamp":"2024-09-13T11:23:32.147021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:32.147021+0000","objects_scrubbed":876,"log_size":3381,"log_dups_size":0,"ondisk_log_size":3381,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:24:48.175541+0000","scrub_duration":99,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":74383360,"num_objects":1135,"num_object_clones":0,"num_object_copies":2270,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1135,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":259,"num_write_kb":16576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,7],"acting":[10,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"28'2","reported_seq":89,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-09-13T11:23:55.186561+0000","last_change":"2024-09-13T11:23:12.560311+0000","last_active":"2024-09-13T11:23:55.186561+0000","last_peered":"2024-09-13T11:23:55.186561+0000","last_clean":"2024-09-13T11:23:55.186561+0000","last_became_active":"2024-09-13T11:23:12.546348+0000","last_became_peered":"2024-09-13T11:23:12.546348+0000","last_unstale":"2024-09-13T11:23:55.186561+0000","last_undegraded":"2024-09-13T11:23:55.186561+0000","last_fullsized":"2024-09-13T11:23:55.186561+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:22:13.113563+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0025826730000000002,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8],"acting":[2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"98'3393","reported_seq":3592,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-09-13T11:23:56.027820+0000","last_change":"2024-09-13T11:23:32.749651+0000","last_active":"2024-09-13T11:23:56.027820+0000","last_peered":"2024-09-13T11:23:56.027820+0000","last_clean":"2024-09-13T11:23:56.027820+0000","last_became_active":"2024-09-13T11:23:31.921388+0000","last_became_peered":"2024-09-13T11:23:31.921388+0000","last_unstale":"2024-09-13T11:23:56.027820+0000","last_undegraded":"2024-09-13T11:23:56.027820+0000","last_fullsized":"2024-09-13T11:23:56.027820+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'3106","last_scrub_stamp":"2024-09-13T11:23:32.749487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:32.749487+0000","objects_scrubbed":856,"log_size":3393,"log_dups_size":0,"ondisk_log_size":3393,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:24:34.709501+0000","scrub_duration":91,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":74907648,"num_objects":1143,"num_object_clones":0,"num_object_copies":2286,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1143,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":287,"num_write_kb":18368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,15],"acting":[12,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":169,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-09-13T11:23:55.208037+0000","last_change":"2024-09-13T11:22:47.298206+0000","last_active":"2024-09-13T11:23:55.208037+0000","last_peered":"2024-09-13T11:23:55.208037+0000","last_clean":"2024-09-13T11:23:55.208037+0000","last_became_active":"2024-09-13T11:22:47.297319+0000","last_became_peered":"2024-09-13T11:22:47.297319+0000","last_unstale":"2024-09-13T11:23:55.208037+0000","last_undegraded":"2024-09-13T11:23:55.208037+0000","last_fullsized":"2024-09-13T11:23:55.208037+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:22:13.113563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00081225399999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,1],"acting":[14,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"3.1","version":"94'3295","reported_seq":3486,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-13T11:23:51.940920+0000","last_change":"2024-09-13T11:23:33.604966+0000","last_active":"2024-09-13T11:23:51.940920+0000","last_peered":"2024-09-13T11:23:51.940920+0000","last_clean":"2024-09-13T11:23:51.940920+0000","last_became_active":"2024-09-13T11:23:31.926528+0000","last_became_peered":"2024-09-13T11:23:31.926528+0000","last_unstale":"2024-09-13T11:23:51.940920+0000","last_undegraded":"2024-09-13T11:23:51.940920+0000","last_fullsized":"2024-09-13T11:23:51.940920+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'3070","last_scrub_stamp":"2024-09-13T11:23:33.604792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:33.604792+0000","objects_scrubbed":842,"log_size":3295,"log_dups_size":0,"ondisk_log_size":3295,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:24:43.880038+0000","scrub_duration":119,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":69926912,"num_objects":1067,"num_object_clones":0,"num_object_copies":2134,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1067,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":225,"num_write_kb":14400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,11],"acting":[13,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":157,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-09-13T11:23:54.178449+0000","last_change":"2024-09-13T11:22:16.267999+0000","last_active":"2024-09-13T11:23:54.178449+0000","last_peered":"2024-09-13T11:23:54.178449+0000","last_clean":"2024-09-13T11:23:54.178449+0000","last_became_active":"2024-09-13T11:22:14.211435+0000","last_became_peered":"2024-09-13T11:22:14.211435+0000","last_unstale":"2024-09-13T11:23:54.178449+0000","last_undegraded":"2024-09-13T11:23:54.178449+0000","last_fullsized":"2024-09-13T11:23:54.178449+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:22:13.113563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010010360000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2","version":"98'3382","reported_seq":3582,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-09-13T11:23:55.956606+0000","last_change":"2024-09-13T11:23:32.977666+0000","last_active":"2024-09-13T11:23:55.956606+0000","last_peered":"2024-09-13T11:23:55.956606+0000","last_clean":"2024-09-13T11:23:55.956606+0000","last_became_active":"2024-09-13T11:23:31.919828+0000","last_became_peered":"2024-09-13T11:23:31.919828+0000","last_unstale":"2024-09-13T11:23:55.956606+0000","last_undegraded":"2024-09-13T11:23:55.956606+0000","last_fullsized":"2024-09-13T11:23:55.956606+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'3057","last_scrub_stamp":"2024-09-13T11:23:32.977559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:32.977559+0000","objects_scrubbed":841,"log_size":3382,"log_dups_size":0,"ondisk_log_size":3382,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:24:52.169339+0000","scrub_duration":151,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76414976,"num_objects":1166,"num_object_clones":0,"num_object_copies":2332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1166,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":325,"num_write_kb":20800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,12],"acting":[14,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"2.3","version":"26'1","reported_seq":161,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-09-13T11:23:55.208262+0000","last_change":"2024-09-13T11:22:16.547691+0000","last_active":"2024-09-13T11:23:55.208262+0000","last_peered":"2024-09-13T11:23:55.208262+0000","last_clean":"2024-09-13T11:23:55.208262+0000","last_became_active":"2024-09-13T11:22:14.146507+0000","last_became_peered":"2024-09-13T11:22:14.146507+0000","last_unstale":"2024-09-13T11:23:55.208262+0000","last_undegraded":"2024-09-13T11:23:55.208262+0000","last_fullsized":"2024-09-13T11:23:55.208262+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:22:13.113563+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00085610999999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,15],"acting":[14,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"1.0","version":"26'112","reported_seq":301,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-09-13T11:23:55.197912+0000","last_change":"2024-09-13T11:23:49.325249+0000","last_active":"2024-09-13T11:23:55.197912+0000","last_peered":"2024-09-13T11:23:55.197912+0000","last_clean":"2024-09-13T11:23:55.197912+0000","last_became_active":"2024-09-13T11:22:12.129721+0000","last_became_peered":"2024-09-13T11:22:12.129721+0000","last_unstale":"2024-09-13T11:23:55.197912+0000","last_undegraded":"2024-09-13T11:23:55.197912+0000","last_fullsized":"2024-09-13T11:23:55.197912+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"26'112","last_scrub_stamp":"2024-09-13T11:23:49.325175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:11.098388+0000","last_clean_scrub_stamp":"2024-09-13T11:23:49.325175+0000","objects_scrubbed":2,"log_size":112,"log_dups_size":0,"ondisk_log_size":112,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:14.422287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.8","version":"98'3647","reported_seq":3826,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-09-13T11:23:55.962021+0000","last_change":"2024-09-13T11:23:20.644952+0000","last_active":"2024-09-13T11:23:55.962021+0000","last_peered":"2024-09-13T11:23:55.962021+0000","last_clean":"2024-09-13T11:23:55.962021+0000","last_became_active":"2024-09-13T11:23:19.925802+0000","last_became_peered":"2024-09-13T11:23:19.925802+0000","last_unstale":"2024-09-13T11:23:55.962021+0000","last_undegraded":"2024-09-13T11:23:55.962021+0000","last_fullsized":"2024-09-13T11:23:55.962021+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'2771","last_scrub_stamp":"2024-09-13T11:23:20.644747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:20.644747+0000","objects_scrubbed":1356,"log_size":3647,"log_dups_size":0,"ondisk_log_size":3647,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:24:38.144523+0000","scrub_duration":411,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":147062784,"num_objects":2244,"num_object_clones":0,"num_object_copies":4488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2244,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":888,"num_write_kb":56832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.9","version":"98'3706","reported_seq":3882,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-09-13T11:23:56.067044+0000","last_change":"2024-09-13T11:23:22.033678+0000","last_active":"2024-09-13T11:23:56.067044+0000","last_peered":"2024-09-13T11:23:56.067044+0000","last_clean":"2024-09-13T11:23:56.067044+0000","last_became_active":"2024-09-13T11:23:19.935114+0000","last_became_peered":"2024-09-13T11:23:19.935114+0000","last_unstale":"2024-09-13T11:23:56.067044+0000","last_undegraded":"2024-09-13T11:23:56.067044+0000","last_fullsized":"2024-09-13T11:23:56.067044+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'2808","last_scrub_stamp":"2024-09-13T11:23:22.033572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:22.033572+0000","objects_scrubbed":1471,"log_size":3706,"log_dups_size":0,"ondisk_log_size":3706,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:24:46.595859+0000","scrub_duration":287,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":155254784,"num_objects":2369,"num_object_clones":0,"num_object_copies":4738,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2369,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":898,"num_write_kb":57472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,2],"acting":[12,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[]},{"pgid":"3.a","version":"97'3984","reported_seq":4164,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-09-13T11:23:54.989728+0000","last_change":"2024-09-13T11:23:32.526150+0000","last_active":"2024-09-13T11:23:54.989728+0000","last_peered":"2024-09-13T11:23:54.989728+0000","last_clean":"2024-09-13T11:23:54.989728+0000","last_became_active":"2024-09-13T11:23:31.934170+0000","last_became_peered":"2024-09-13T11:23:31.934170+0000","last_unstale":"2024-09-13T11:23:54.989728+0000","last_undegraded":"2024-09-13T11:23:54.989728+0000","last_fullsized":"2024-09-13T11:23:54.989728+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'3418","last_scrub_stamp":"2024-09-13T11:23:32.526057+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:32.526057+0000","objects_scrubbed":1774,"log_size":3984,"log_dups_size":0,"ondisk_log_size":3984,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:24:34.770161+0000","scrub_duration":507,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":153354240,"num_objects":2340,"num_object_clones":0,"num_object_copies":4680,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2340,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":566,"num_write_kb":36224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.b","version":"94'3905","reported_seq":4078,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-13T11:23:51.951796+0000","last_change":"2024-09-13T11:23:32.681343+0000","last_active":"2024-09-13T11:23:51.951796+0000","last_peered":"2024-09-13T11:23:51.951796+0000","last_clean":"2024-09-13T11:23:51.951796+0000","last_became_active":"2024-09-13T11:23:31.923415+0000","last_became_peered":"2024-09-13T11:23:31.923415+0000","last_unstale":"2024-09-13T11:23:51.951796+0000","last_undegraded":"2024-09-13T11:23:51.951796+0000","last_fullsized":"2024-09-13T11:23:51.951796+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'3471","last_scrub_stamp":"2024-09-13T11:23:32.681257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:32.681257+0000","objects_scrubbed":1720,"log_size":3905,"log_dups_size":0,"ondisk_log_size":3905,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:24:58.857812+0000","scrub_duration":199,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":141164544,"num_objects":2154,"num_object_clones":0,"num_object_copies":4308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2154,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":434,"num_write_kb":27776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,1],"acting":[13,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"3.c","version":"94'3888","reported_seq":4059,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-13T11:23:51.996813+0000","last_change":"2024-09-13T11:23:32.524870+0000","last_active":"2024-09-13T11:23:51.996813+0000","last_peered":"2024-09-13T11:23:51.996813+0000","last_clean":"2024-09-13T11:23:51.996813+0000","last_became_active":"2024-09-13T11:23:31.922620+0000","last_became_peered":"2024-09-13T11:23:31.922620+0000","last_unstale":"2024-09-13T11:23:51.996813+0000","last_undegraded":"2024-09-13T11:23:51.996813+0000","last_fullsized":"2024-09-13T11:23:51.996813+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'3463","last_scrub_stamp":"2024-09-13T11:23:32.524776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:32.524776+0000","objects_scrubbed":1741,"log_size":3888,"log_dups_size":0,"ondisk_log_size":3888,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:24:37.745747+0000","scrub_duration":235,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":141950976,"num_objects":2166,"num_object_clones":0,"num_object_copies":4332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2166,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":425,"num_write_kb":27200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,13],"acting":[11,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.d","version":"98'4003","reported_seq":4182,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-09-13T11:23:56.143293+0000","last_change":"2024-09-13T11:23:32.776835+0000","last_active":"2024-09-13T11:23:56.143293+0000","last_peered":"2024-09-13T11:23:56.143293+0000","last_clean":"2024-09-13T11:23:56.143293+0000","last_became_active":"2024-09-13T11:23:31.939246+0000","last_became_peered":"2024-09-13T11:23:31.939246+0000","last_unstale":"2024-09-13T11:23:56.143293+0000","last_undegraded":"2024-09-13T11:23:56.143293+0000","last_fullsized":"2024-09-13T11:23:56.143293+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'3395","last_scrub_stamp":"2024-09-13T11:23:32.776747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:32.776747+0000","objects_scrubbed":1704,"log_size":4003,"log_dups_size":0,"ondisk_log_size":4003,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:24:56.402862+0000","scrub_duration":179,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":151519232,"num_objects":2312,"num_object_clones":0,"num_object_copies":4624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2312,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":608,"num_write_kb":38912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.e","version":"98'4131","reported_seq":9240,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-09-13T11:23:56.036465+0000","last_change":"2024-09-13T11:23:32.368856+0000","last_active":"2024-09-13T11:23:56.036465+0000","last_peered":"2024-09-13T11:23:56.036465+0000","last_clean":"2024-09-13T11:23:56.036465+0000","last_became_active":"2024-09-13T11:23:31.925869+0000","last_became_peered":"2024-09-13T11:23:31.925869+0000","last_unstale":"2024-09-13T11:23:56.036465+0000","last_undegraded":"2024-09-13T11:23:56.036465+0000","last_fullsized":"2024-09-13T11:23:56.036465+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'3527","last_scrub_stamp":"2024-09-13T11:23:32.368727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:32.368727+0000","objects_scrubbed":1774,"log_size":4131,"log_dups_size":0,"ondisk_log_size":4131,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:24:57.811779+0000","scrub_duration":219,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":155910144,"num_objects":2379,"num_object_clones":0,"num_object_copies":4758,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2379,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":605,"num_write_kb":38720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"94'3835","reported_seq":4007,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-13T11:23:52.022840+0000","last_change":"2024-09-13T11:23:33.492630+0000","last_active":"2024-09-13T11:23:52.022840+0000","last_peered":"2024-09-13T11:23:52.022840+0000","last_clean":"2024-09-13T11:23:52.022840+0000","last_became_active":"2024-09-13T11:23:31.935239+0000","last_became_peered":"2024-09-13T11:23:31.935239+0000","last_unstale":"2024-09-13T11:23:52.022840+0000","last_undegraded":"2024-09-13T11:23:52.022840+0000","last_fullsized":"2024-09-13T11:23:52.022840+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'3420","last_scrub_stamp":"2024-09-13T11:23:33.492488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:33.492488+0000","objects_scrubbed":1776,"log_size":3835,"log_dups_size":0,"ondisk_log_size":3835,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:24:44.851261+0000","scrub_duration":215,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":143589376,"num_objects":2191,"num_object_clones":0,"num_object_copies":4382,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2191,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":415,"num_write_kb":26560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,12],"acting":[11,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.10","version":"98'3756","reported_seq":8091,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-09-13T11:23:56.020141+0000","last_change":"2024-09-13T11:23:48.519654+0000","last_active":"2024-09-13T11:23:56.020141+0000","last_peered":"2024-09-13T11:23:56.020141+0000","last_clean":"2024-09-13T11:23:56.020141+0000","last_became_active":"2024-09-13T11:23:41.931183+0000","last_became_peered":"2024-09-13T11:23:41.931183+0000","last_unstale":"2024-09-13T11:23:56.020141+0000","last_undegraded":"2024-09-13T11:23:56.020141+0000","last_fullsized":"2024-09-13T11:23:56.020141+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":5,"last_scrub":"91'3468","last_scrub_stamp":"2024-09-13T11:23:48.519574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:48.519574+0000","objects_scrubbed":2031,"log_size":3756,"log_dups_size":0,"ondisk_log_size":3756,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:24:57.342213+0000","scrub_duration":215,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":152109056,"num_objects":2321,"num_object_clones":0,"num_object_copies":4642,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2321,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":290,"num_write_kb":18560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.11","version":"98'3682","reported_seq":6795,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-09-13T11:23:55.998860+0000","last_change":"2024-09-13T11:23:46.102466+0000","last_active":"2024-09-13T11:23:55.998860+0000","last_peered":"2024-09-13T11:23:55.998860+0000","last_clean":"2024-09-13T11:23:55.998860+0000","last_became_active":"2024-09-13T11:23:39.063172+0000","last_became_peered":"2024-09-13T11:23:39.063172+0000","last_unstale":"2024-09-13T11:23:55.998860+0000","last_undegraded":"2024-09-13T11:23:55.998860+0000","last_fullsized":"2024-09-13T11:23:55.998860+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":5,"last_scrub":"88'3345","last_scrub_stamp":"2024-09-13T11:23:46.101973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:46.101973+0000","objects_scrubbed":2011,"log_size":3682,"log_dups_size":0,"ondisk_log_size":3682,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:08.671350+0000","scrub_duration":955,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154140672,"num_objects":2352,"num_object_clones":0,"num_object_copies":4704,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2352,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":341,"num_write_kb":21824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11],"acting":[4,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.12","version":"98'3714","reported_seq":6750,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-09-13T11:23:56.015916+0000","last_change":"2024-09-13T11:23:35.111409+0000","last_active":"2024-09-13T11:23:56.015916+0000","last_peered":"2024-09-13T11:23:56.015916+0000","last_clean":"2024-09-13T11:23:56.015916+0000","last_became_active":"2024-09-13T11:23:21.998877+0000","last_became_peered":"2024-09-13T11:23:21.998877+0000","last_unstale":"2024-09-13T11:23:56.015916+0000","last_undegraded":"2024-09-13T11:23:56.015916+0000","last_fullsized":"2024-09-13T11:23:56.015916+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'3109","last_scrub_stamp":"2024-09-13T11:23:35.111320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:35.111320+0000","objects_scrubbed":1732,"log_size":3714,"log_dups_size":0,"ondisk_log_size":3714,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:03.371433+0000","scrub_duration":239,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":153157632,"num_objects":2337,"num_object_clones":0,"num_object_copies":4674,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2337,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":605,"num_write_kb":38720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,7],"acting":[14,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"3.13","version":"94'3570","reported_seq":6601,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-13T11:23:52.055839+0000","last_change":"2024-09-13T11:23:47.565094+0000","last_active":"2024-09-13T11:23:52.055839+0000","last_peered":"2024-09-13T11:23:52.055839+0000","last_clean":"2024-09-13T11:23:52.055839+0000","last_became_active":"2024-09-13T11:23:41.082433+0000","last_became_peered":"2024-09-13T11:23:41.082433+0000","last_unstale":"2024-09-13T11:23:52.055839+0000","last_undegraded":"2024-09-13T11:23:52.055839+0000","last_fullsized":"2024-09-13T11:23:52.055839+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'3408","last_scrub_stamp":"2024-09-13T11:23:47.564931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:47.564931+0000","objects_scrubbed":1981,"log_size":3570,"log_dups_size":0,"ondisk_log_size":3570,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:04.258231+0000","scrub_duration":471,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":141230080,"num_objects":2155,"num_object_clones":0,"num_object_copies":4310,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2155,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":174,"num_write_kb":11136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7],"acting":[11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":36,"stat_sum":{"num_bytes":4752113664,"num_objects":72513,"num_object_clones":0,"num_object_copies":145026,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6760,"num_objects_unfound":0,"num_objects_dirty":72513,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":28528,"num_write_kb":1825696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9884,"num_bytes_recovered":647725056,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":11813322752,"data_stored":11813322752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":132954,"ondisk_log_size":132954,"up":72,"acting":72,"num_store_stats":16},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":10},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":112,"ondisk_log_size":112,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":13,"up_from":23,"seq":98784247832,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1097252,"kb_used_data":1034132,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":93274588,"statfs":{"total":96636764160,"available":95513178112,"internally_reserved":0,"allocated":1058951168,"data_stored":1058533381,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[0,1,3,4,6,8,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":9,"up_from":23,"seq":98784247831,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":323796,"kb_used_data":292612,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94048044,"statfs":{"total":96636764160,"available":96305197056,"internally_reserved":0,"allocated":299634688,"data_stored":299232379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,8,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":40,"seq":171798691855,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1260772,"kb_used_data":1192660,"kb_used_omap":4,"kb_used_meta":68091,"kb_avail":93111068,"statfs":{"total":96636764160,"available":95345733632,"internally_reserved":0,"allocated":1221283840,"data_stored":1220832136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4433,"internal_metadata":69725871},"hb_peers":[0,2,3,6,8,9,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":21,"seq":90194313240,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":901600,"kb_used_data":856528,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":93470240,"statfs":{"total":96636764160,"available":95713525760,"internally_reserved":0,"allocated":877084672,"data_stored":876634785,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":46135751},"hb_peers":[0,2,4,7,9,10,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":21,"seq":90194313243,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":234152,"kb_used_data":194648,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94137688,"statfs":{"total":96636764160,"available":96396992512,"internally_reserved":0,"allocated":199319552,"data_stored":198865634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40434119},"hb_peers":[0,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":22,"seq":94489280535,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1015504,"kb_used_data":944064,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":93356336,"statfs":{"total":96636764160,"available":95596888064,"internally_reserved":0,"allocated":966721536,"data_stored":966345025,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73136581},"hb_peers":[1,3,4,5,8,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":22,"seq":94489280536,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":395228,"kb_used_data":359884,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93976612,"statfs":{"total":96636764160,"available":96232050688,"internally_reserved":0,"allocated":368521216,"data_stored":368083624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,7,8,9,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":15,"up_from":22,"seq":94489280536,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":838044,"kb_used_data":774924,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":93533796,"statfs":{"total":96636764160,"available":95778607104,"internally_reserved":0,"allocated":793522176,"data_stored":793141316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":64616902},"hb_peers":[0,3,4,5,6,7,10,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":22,"seq":94489280536,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":422956,"kb_used_data":383452,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":93948884,"statfs":{"total":96636764160,"available":96203657216,"internally_reserved":0,"allocated":392654848,"data_stored":392202303,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[3,5,7,9,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":22,"seq":94489280536,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":404284,"kb_used_data":368940,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93967556,"statfs":{"total":96636764160,"available":96222777344,"internally_reserved":0,"allocated":377794560,"data_stored":377384292,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,3,5,7,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":22,"seq":94489280536,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1161328,"kb_used_data":1081568,"kb_used_omap":1,"kb_used_meta":79742,"kb_avail":93210512,"statfs":{"total":96636764160,"available":95447564288,"internally_reserved":0,"allocated":1107525632,"data_stored":1107125067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":81656266},"hb_peers":[0,3,5,6,8,9,10,11,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":22,"seq":94489280536,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":485428,"kb_used_data":458404,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93886412,"statfs":{"total":96636764160,"available":96139685888,"internally_reserved":0,"allocated":469405696,"data_stored":468941270,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,5,7,9,10,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":22,"seq":94489280535,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":588460,"kb_used_data":557276,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93783380,"statfs":{"total":96636764160,"available":96034181120,"internally_reserved":0,"allocated":570650624,"data_stored":570248918,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,3,6,7,9,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":11,"up_from":22,"seq":94489280536,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1159964,"kb_used_data":1096844,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":93211876,"statfs":{"total":96636764160,"available":95448961024,"internally_reserved":0,"allocated":1123168256,"data_stored":1122761730,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":64616900},"hb_peers":[0,2,4,7,8,10,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,0,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":12,"up_from":22,"seq":94489280536,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":970648,"kb_used_data":899208,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":93401192,"statfs":{"total":96636764160,"available":95642820608,"internally_reserved":0,"allocated":920788992,"data_stored":920352163,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73136581},"hb_peers":[0,1,2,3,5,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":14,"up_from":22,"seq":94489280536,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1047512,"kb_used_data":971912,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":93324328,"statfs":{"total":96636764160,"available":95564111872,"internally_reserved":0,"allocated":995237888,"data_stored":994799398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,1,5,6,7,8,9,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":986972160,"data_stored":986972160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1222311936,"data_stored":1222311936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":367132672,"data_stored":367132672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":878641152,"data_stored":878641152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":391577600,"data_stored":391577600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":199098368,"data_stored":199098368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":379125760,"data_stored":379125760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1111490560,"data_stored":1111490560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":468844544,"data_stored":468844544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":299565056,"data_stored":299565056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":578420736,"data_stored":578420736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":1140719616,"data_stored":1140719616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":928645120,"data_stored":928645120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":1066074112,"data_stored":1066074112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":993460224,"data_stored":993460224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":801243136,"data_stored":801243136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T11:23:57.145 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-13T11:23:57.174 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-13T11:23:57.175 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:23:57.184 INFO:tasks.ceph.osd.8.smithi001.stderr:2024-09-13T11:23:57.180+0000 7fb9b48ae640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26312) UID: 0 2024-09-13T11:23:57.193 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-13T11:23:57.269 INFO:tasks.ceph.osd.5.smithi092.stderr:2024-09-13T11:23:57.267+0000 7f6f0b705640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26346) UID: 0 2024-09-13T11:23:57.353 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-09-13T11:23:57.353 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:23:57.370 INFO:tasks.ceph.osd.2.smithi162.stderr:2024-09-13T11:23:57.369+0000 7fe92b35a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26313) UID: 0 2024-09-13T11:23:57.373 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-13T11:23:57.471 INFO:tasks.ceph.osd.14.smithi162.stderr:2024-09-13T11:23:57.469+0000 7f235c0db640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26319) UID: 0 2024-09-13T11:23:57.519 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-09-13T11:23:57.519 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:23:57.541 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-13T11:23:57.584 INFO:tasks.ceph.osd.15.smithi177.stderr:2024-09-13T11:23:57.580+0000 7f9d6ce77640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26578) UID: 0 2024-09-13T11:23:57.620 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-13T11:23:57.621 INFO:teuthology.orchestra.run.smithi001.stdout:{"fsid":"2fd68ac7-5581-4d90-80a4-30bcf5252a7a","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"nodeep-scrub flag(s) set","count":12},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":120,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":100,"num_osds":16,"num_up_osds":16,"osd_up_since":1726226566,"num_in_osds":16,"osd_in_since":1726226521,"num_remapped_pgs":6},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":38},{"state_name":"active+recovering+remapped","count":6},{"state_name":"active+recovering","count":1}],"num_pgs":45,"num_pools":3,"num_objects":72101,"data_bytes":4725441075,"bytes_used":12417884160,"bytes_avail":1533770342400,"bytes_total":1546188226560,"misplaced_objects":6760,"misplaced_total":144202,"misplaced_ratio":0.04687868406818213,"recovering_objects_per_sec":382,"recovering_bytes_per_sec":25079252,"recovering_keys_per_sec":0,"num_objects_recovered":1531,"num_bytes_recovered":100335616,"num_keys_recovered":0,"write_bytes_sec":58594699,"read_op_per_sec":0,"write_op_per_sec":894},"fsmap":{"epoch":1,"btime":"2024-09-13T11:21:57:222574+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":5,"modified":"2024-09-13T11:23:46.137611+0000","services":{"osd":{"daemons":{"summary":"","5":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-09-13T11:23:57.639 INFO:tasks.ceph.ceph_manager.ceph:making progress, resetting timeout 2024-09-13T11:23:57.639 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-13T11:23:57.673 INFO:tasks.ceph.osd.14.smithi162.stderr:2024-09-13T11:23:57.669+0000 7f235c0db640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26319) UID: 0 2024-09-13T11:23:57.674 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-09-13T11:23:57.675 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:23:57.688 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-13T11:23:57.776 INFO:tasks.ceph.osd.7.smithi177.stderr:2024-09-13T11:23:57.772+0000 7f897b51d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26575) UID: 0 2024-09-13T11:23:57.855 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-09-13T11:23:57.856 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:23:57.869 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T11:23:57.873 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-13T11:23:57.916 INFO:tasks.ceph.osd.8.smithi001.stderr:2024-09-13T11:23:57.912+0000 7fb9b48ae640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26312) UID: 0 2024-09-13T11:23:57.975 INFO:tasks.ceph.osd.11.smithi177.stderr:2024-09-13T11:23:57.968+0000 7f24792c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26580) UID: 0 2024-09-13T11:23:57.990 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-13T11:23:57.990 INFO:teuthology.orchestra.run.smithi001.stderr:dumped all 2024-09-13T11:23:58.007 INFO:tasks.radosbench.radosbench.0.smithi177.stdout: 91 16 76560 76544 52.566 68.9375 0.0161638 0.0190166 2024-09-13T11:23:58.015 INFO:teuthology.orchestra.run.smithi001.stdout:{"pg_ready":true,"pg_map":{"version":139,"stamp":"2024-09-13T11:23:56.911242+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4752704051,"num_objects":72517,"num_object_clones":0,"num_object_copies":145034,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6760,"num_objects_unfound":0,"num_objects_dirty":72517,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":28675,"num_write_kb":1828370,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9885,"num_bytes_recovered":647725075,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":133069,"ondisk_log_size":133069,"up":90,"acting":90,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":119,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":16,"kb":1509949440,"kb_used":12306928,"kb_used_data":11467056,"kb_used_omap":27,"kb_used_meta":839588,"kb_avail":1497642512,"statfs":{"total":1546188226560,"available":1533585932288,"internally_reserved":0,"allocated":11742265344,"data_stored":11735483421,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28358,"internal_metadata":859738426},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,2,1,2,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":261685248,"num_objects":3993,"num_object_clones":0,"num_object_copies":7986,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-3061,"num_objects_unfound":0,"num_objects_dirty":3993,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3993,"num_write_kb":255552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1531,"num_bytes_recovered":100335616,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"3.738414"},"pg_stats":[{"pgid":"3.23","version":"98'3432","reported_seq":6144,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-09-13T11:23:55.908873+0000","last_change":"2024-09-13T11:23:48.133184+0000","last_active":"2024-09-13T11:23:55.908873+0000","last_peered":"2024-09-13T11:23:55.908873+0000","last_clean":"2024-09-13T11:23:55.908873+0000","last_became_active":"2024-09-13T11:23:47.926677+0000","last_became_peered":"2024-09-13T11:23:47.926677+0000","last_unstale":"2024-09-13T11:23:55.908873+0000","last_undegraded":"2024-09-13T11:23:55.908873+0000","last_fullsized":"2024-09-13T11:23:55.908873+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":6,"last_scrub":"91'3278","last_scrub_stamp":"2024-09-13T11:23:48.133084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:48.133084+0000","objects_scrubbed":1029,"log_size":3432,"log_dups_size":0,"ondisk_log_size":3432,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:12.874134+0000","scrub_duration":139,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":77594624,"num_objects":1184,"num_object_clones":0,"num_object_copies":2368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1184,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":155,"num_write_kb":9920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9],"acting":[5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.22","version":"98'3408","reported_seq":5307,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-09-13T11:23:56.014918+0000","last_change":"2024-09-13T11:23:47.300446+0000","last_active":"2024-09-13T11:23:56.014918+0000","last_peered":"2024-09-13T11:23:56.014918+0000","last_clean":"2024-09-13T11:23:56.014918+0000","last_became_active":"2024-09-13T11:23:46.925318+0000","last_became_peered":"2024-09-13T11:23:46.925318+0000","last_unstale":"2024-09-13T11:23:56.014918+0000","last_undegraded":"2024-09-13T11:23:56.014918+0000","last_fullsized":"2024-09-13T11:23:56.014918+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":6,"last_scrub":"90'3246","last_scrub_stamp":"2024-09-13T11:23:47.300339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:47.300339+0000","objects_scrubbed":1033,"log_size":3408,"log_dups_size":0,"ondisk_log_size":3408,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:24:49.780485+0000","scrub_duration":203,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":78446592,"num_objects":1197,"num_object_clones":0,"num_object_copies":2394,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1197,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":164,"num_write_kb":10496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,14],"acting":[0,14],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.21","version":"94'3286","reported_seq":4757,"reported_epoch":94,"state":"active+recovering+remapped","last_fresh":"2024-09-13T11:23:52.020283+0000","last_change":"2024-09-13T11:23:45.861514+0000","last_active":"2024-09-13T11:23:52.020283+0000","last_peered":"2024-09-13T11:23:52.020283+0000","last_clean":"2024-09-13T11:23:32.050222+0000","last_became_active":"2024-09-13T11:23:35.026171+0000","last_became_peered":"2024-09-13T11:23:35.026171+0000","last_unstale":"2024-09-13T11:23:52.020283+0000","last_undegraded":"2024-09-13T11:23:52.020283+0000","last_fullsized":"2024-09-13T11:23:52.020283+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":6,"last_scrub":"69'2784","last_scrub_stamp":"2024-09-13T11:23:21.595419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:21.595419+0000","objects_scrubbed":1386,"log_size":3286,"log_dups_size":0,"ondisk_log_size":3286,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":311,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":69074944,"num_objects":1054,"num_object_clones":0,"num_object_copies":2108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":810,"num_objects_unfound":0,"num_objects_dirty":1054,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":361,"num_write_kb":23104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":431,"num_bytes_recovered":28246016,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8],"acting":[11,0],"avail_no_missing":["11","13"],"object_location_counts":[{"shards":"0,11","objects":649},{"shards":"11,13","objects":405}],"blocked_by":[],"up_primary":0,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.20","version":"98'3426","reported_seq":5440,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-09-13T11:23:56.029766+0000","last_change":"2024-09-13T11:23:47.969503+0000","last_active":"2024-09-13T11:23:56.029766+0000","last_peered":"2024-09-13T11:23:56.029766+0000","last_clean":"2024-09-13T11:23:56.029766+0000","last_became_active":"2024-09-13T11:23:33.954946+0000","last_became_peered":"2024-09-13T11:23:33.954946+0000","last_unstale":"2024-09-13T11:23:56.029766+0000","last_undegraded":"2024-09-13T11:23:56.029766+0000","last_fullsized":"2024-09-13T11:23:56.029766+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":6,"last_scrub":"91'3270","last_scrub_stamp":"2024-09-13T11:23:47.969372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:47.969372+0000","objects_scrubbed":1061,"log_size":3426,"log_dups_size":0,"ondisk_log_size":3426,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:24:48.255263+0000","scrub_duration":1327,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":80084992,"num_objects":1222,"num_object_clones":0,"num_object_copies":2444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1222,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":161,"num_write_kb":10304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,0],"acting":[12,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[]},{"pgid":"3.1f","version":"94'3850","reported_seq":6144,"reported_epoch":94,"state":"active+recovering+remapped","last_fresh":"2024-09-13T11:23:52.060704+0000","last_change":"2024-09-13T11:23:37.329722+0000","last_active":"2024-09-13T11:23:52.060704+0000","last_peered":"2024-09-13T11:23:52.060704+0000","last_clean":"2024-09-13T11:23:32.316727+0000","last_became_active":"2024-09-13T11:23:35.053803+0000","last_became_peered":"2024-09-13T11:23:35.053803+0000","last_unstale":"2024-09-13T11:23:52.060704+0000","last_undegraded":"2024-09-13T11:23:52.060704+0000","last_fullsized":"2024-09-13T11:23:52.060704+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:22:24.896055+0000","objects_scrubbed":0,"log_size":3850,"log_dups_size":0,"ondisk_log_size":3850,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":140476416,"num_objects":2144,"num_object_clones":0,"num_object_copies":4288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1872,"num_objects_unfound":0,"num_objects_dirty":2144,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2144,"num_write_kb":137184,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":715,"num_bytes_recovered":46858240,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,14],"acting":[11,10],"avail_no_missing":["11","12"],"object_location_counts":[{"shards":"10,11","objects":1208},{"shards":"11,12","objects":936}],"blocked_by":[],"up_primary":10,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1e","version":"98'4129","reported_seq":12953,"reported_epoch":98,"state":"active+recovering+remapped","last_fresh":"2024-09-13T11:23:56.078942+0000","last_change":"2024-09-13T11:23:35.106919+0000","last_active":"2024-09-13T11:23:56.078942+0000","last_peered":"2024-09-13T11:23:56.078942+0000","last_clean":"2024-09-13T11:23:30.974769+0000","last_became_active":"2024-09-13T11:23:35.075137+0000","last_became_peered":"2024-09-13T11:23:35.075137+0000","last_unstale":"2024-09-13T11:23:56.078942+0000","last_undegraded":"2024-09-13T11:23:56.078942+0000","last_fullsized":"2024-09-13T11:23:56.078942+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:22:24.896055+0000","objects_scrubbed":0,"log_size":4129,"log_dups_size":0,"ondisk_log_size":4129,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":155254784,"num_objects":2369,"num_object_clones":0,"num_object_copies":4738,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1005,"num_objects_unfound":0,"num_objects_dirty":2369,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2369,"num_write_kb":151616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2070,"num_bytes_recovered":135626752,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[0,4],"avail_no_missing":["0","1","8","14"],"object_location_counts":[{"shards":"0,1","objects":502},{"shards":"0,1,4","objects":1},{"shards":"0,4","objects":1866}],"blocked_by":[],"up_primary":5,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1d","version":"97'3925","reported_seq":7866,"reported_epoch":97,"state":"active+recovering+remapped","last_fresh":"2024-09-13T11:23:55.002071+0000","last_change":"2024-09-13T11:23:35.096108+0000","last_active":"2024-09-13T11:23:55.002071+0000","last_peered":"2024-09-13T11:23:55.002071+0000","last_clean":"2024-09-13T11:23:32.393844+0000","last_became_active":"2024-09-13T11:23:35.080149+0000","last_became_peered":"2024-09-13T11:23:35.080149+0000","last_unstale":"2024-09-13T11:23:55.002071+0000","last_undegraded":"2024-09-13T11:23:55.002071+0000","last_fullsized":"2024-09-13T11:23:55.002071+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:22:24.896055+0000","objects_scrubbed":0,"log_size":3925,"log_dups_size":0,"ondisk_log_size":3925,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":146407424,"num_objects":2234,"num_object_clones":0,"num_object_copies":4468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":876,"num_objects_unfound":0,"num_objects_dirty":2234,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2234,"num_write_kb":142976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1262,"num_bytes_recovered":82706432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,15],"acting":[3,2],"avail_no_missing":["3","7"],"object_location_counts":[{"shards":"2,3","objects":1796},{"shards":"3,7","objects":438}],"blocked_by":[],"up_primary":2,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1c","version":"94'3954","reported_seq":7192,"reported_epoch":94,"state":"active+recovering","last_fresh":"2024-09-13T11:23:52.064402+0000","last_change":"2024-09-13T11:23:33.998502+0000","last_active":"2024-09-13T11:23:52.064402+0000","last_peered":"2024-09-13T11:23:52.064402+0000","last_clean":"2024-09-13T11:23:32.339821+0000","last_became_active":"2024-09-13T11:23:33.989492+0000","last_became_peered":"2024-09-13T11:23:33.989492+0000","last_unstale":"2024-09-13T11:23:52.064402+0000","last_undegraded":"2024-09-13T11:23:52.064402+0000","last_fullsized":"2024-09-13T11:23:52.064402+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:22:24.896055+0000","objects_scrubbed":0,"log_size":3954,"log_dups_size":0,"ondisk_log_size":3954,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":145981440,"num_objects":2228,"num_object_clones":0,"num_object_copies":4456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":189,"num_objects_unfound":0,"num_objects_dirty":2228,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2228,"num_write_kb":142560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1544,"num_bytes_recovered":101187584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7],"acting":[11,7],"avail_no_missing":["11","13"],"object_location_counts":[{"shards":"7,11","objects":2039},{"shards":"11,13","objects":189}],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1b","version":"98'4007","reported_seq":7954,"reported_epoch":98,"state":"active+recovering+remapped","last_fresh":"2024-09-13T11:23:55.341018+0000","last_change":"2024-09-13T11:23:35.097804+0000","last_active":"2024-09-13T11:23:55.341018+0000","last_peered":"2024-09-13T11:23:55.341018+0000","last_clean":"2024-09-13T11:23:32.309147+0000","last_became_active":"2024-09-13T11:23:35.080340+0000","last_became_peered":"2024-09-13T11:23:35.080340+0000","last_unstale":"2024-09-13T11:23:55.341018+0000","last_undegraded":"2024-09-13T11:23:55.341018+0000","last_fullsized":"2024-09-13T11:23:55.341018+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:22:24.896055+0000","objects_scrubbed":0,"log_size":4007,"log_dups_size":0,"ondisk_log_size":4007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":149192704,"num_objects":2277,"num_object_clones":0,"num_object_copies":4554,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":994,"num_objects_unfound":0,"num_objects_dirty":2277,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2277,"num_write_kb":145696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1263,"num_bytes_recovered":82771968,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,15],"acting":[1,12],"avail_no_missing":["1","13"],"object_location_counts":[{"shards":"1,12","objects":1780},{"shards":"1,13","objects":497}],"blocked_by":[],"up_primary":12,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1a","version":"97'3923","reported_seq":7501,"reported_epoch":97,"state":"active+recovering+remapped","last_fresh":"2024-09-13T11:23:55.050047+0000","last_change":"2024-09-13T11:23:35.089105+0000","last_active":"2024-09-13T11:23:55.050047+0000","last_peered":"2024-09-13T11:23:55.050047+0000","last_clean":"2024-09-13T11:23:32.455463+0000","last_became_active":"2024-09-13T11:23:35.072595+0000","last_became_peered":"2024-09-13T11:23:35.072595+0000","last_unstale":"2024-09-13T11:23:55.050047+0000","last_undegraded":"2024-09-13T11:23:55.050047+0000","last_fullsized":"2024-09-13T11:23:55.050047+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:22:24.896055+0000","objects_scrubbed":0,"log_size":3923,"log_dups_size":0,"ondisk_log_size":3923,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":145293312,"num_objects":2217,"num_object_clones":0,"num_object_copies":4434,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1014,"num_objects_unfound":0,"num_objects_dirty":2217,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2217,"num_write_kb":141888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1141,"num_bytes_recovered":74776576,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9],"acting":[3,4],"avail_no_missing":["3","7"],"object_location_counts":[{"shards":"3,4","objects":1710},{"shards":"3,7","objects":507}],"blocked_by":[],"up_primary":4,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.19","version":"94'3533","reported_seq":7812,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-13T11:23:51.943889+0000","last_change":"2024-09-13T11:23:49.429522+0000","last_active":"2024-09-13T11:23:51.943889+0000","last_peered":"2024-09-13T11:23:51.943889+0000","last_clean":"2024-09-13T11:23:51.943889+0000","last_became_active":"2024-09-13T11:23:41.924941+0000","last_became_peered":"2024-09-13T11:23:41.924941+0000","last_unstale":"2024-09-13T11:23:51.943889+0000","last_undegraded":"2024-09-13T11:23:51.943889+0000","last_fullsized":"2024-09-13T11:23:51.943889+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":5,"last_scrub":"92'3415","last_scrub_stamp":"2024-09-13T11:23:49.429429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:49.429429+0000","objects_scrubbed":1979,"log_size":3533,"log_dups_size":0,"ondisk_log_size":3533,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:04.629542+0000","scrub_duration":587,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":138280960,"num_objects":2110,"num_object_clones":0,"num_object_copies":4220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2110,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":131,"num_write_kb":8384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1],"acting":[9,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.18","version":"98'3727","reported_seq":8276,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-09-13T11:23:55.981307+0000","last_change":"2024-09-13T11:23:50.145341+0000","last_active":"2024-09-13T11:23:55.981307+0000","last_peered":"2024-09-13T11:23:55.981307+0000","last_clean":"2024-09-13T11:23:55.981307+0000","last_became_active":"2024-09-13T11:23:48.926014+0000","last_became_peered":"2024-09-13T11:23:48.926014+0000","last_unstale":"2024-09-13T11:23:55.981307+0000","last_undegraded":"2024-09-13T11:23:55.981307+0000","last_fullsized":"2024-09-13T11:23:55.981307+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:22:24.896055+0000","objects_scrubbed":949,"log_size":3727,"log_dups_size":0,"ondisk_log_size":3727,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154664960,"num_objects":2360,"num_object_clones":0,"num_object_copies":4720,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2360,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2360,"num_write_kb":151040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1458,"num_bytes_recovered":95551488,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0],"acting":[10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.17","version":"98'3609","reported_seq":7631,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-09-13T11:23:56.009818+0000","last_change":"2024-09-13T11:23:49.349241+0000","last_active":"2024-09-13T11:23:56.009818+0000","last_peered":"2024-09-13T11:23:56.009818+0000","last_clean":"2024-09-13T11:23:56.009818+0000","last_became_active":"2024-09-13T11:23:42.935601+0000","last_became_peered":"2024-09-13T11:23:42.935601+0000","last_unstale":"2024-09-13T11:23:56.009818+0000","last_undegraded":"2024-09-13T11:23:56.009818+0000","last_fullsized":"2024-09-13T11:23:56.009818+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":5,"last_scrub":"92'3359","last_scrub_stamp":"2024-09-13T11:23:49.349167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:49.349167+0000","objects_scrubbed":2004,"log_size":3609,"log_dups_size":0,"ondisk_log_size":3609,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:24:59.401302+0000","scrub_duration":363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":148242432,"num_objects":2262,"num_object_clones":0,"num_object_copies":4524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2262,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":258,"num_write_kb":16512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3],"acting":[10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"94'3677","reported_seq":8318,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-13T11:23:52.012337+0000","last_change":"2024-09-13T11:23:46.510544+0000","last_active":"2024-09-13T11:23:52.012337+0000","last_peered":"2024-09-13T11:23:52.012337+0000","last_clean":"2024-09-13T11:23:52.012337+0000","last_became_active":"2024-09-13T11:23:46.105237+0000","last_became_peered":"2024-09-13T11:23:46.105237+0000","last_unstale":"2024-09-13T11:23:52.012337+0000","last_undegraded":"2024-09-13T11:23:52.012337+0000","last_fullsized":"2024-09-13T11:23:52.012337+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":5,"last_scrub":"89'3478","last_scrub_stamp":"2024-09-13T11:23:46.510384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:46.510384+0000","objects_scrubbed":2054,"log_size":3677,"log_dups_size":0,"ondisk_log_size":3677,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:09.149878+0000","scrub_duration":363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":147849216,"num_objects":2256,"num_object_clones":0,"num_object_copies":4512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2256,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":202,"num_write_kb":12928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,15],"acting":[11,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.15","version":"98'3655","reported_seq":8066,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-09-13T11:23:56.050031+0000","last_change":"2024-09-13T11:23:48.332790+0000","last_active":"2024-09-13T11:23:56.050031+0000","last_peered":"2024-09-13T11:23:56.050031+0000","last_clean":"2024-09-13T11:23:56.050031+0000","last_became_active":"2024-09-13T11:23:42.931149+0000","last_became_peered":"2024-09-13T11:23:42.931149+0000","last_unstale":"2024-09-13T11:23:56.050031+0000","last_undegraded":"2024-09-13T11:23:56.050031+0000","last_fullsized":"2024-09-13T11:23:56.050031+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":5,"last_scrub":"91'3384","last_scrub_stamp":"2024-09-13T11:23:48.332600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:48.332600+0000","objects_scrubbed":2032,"log_size":3655,"log_dups_size":0,"ondisk_log_size":3655,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:24:58.239508+0000","scrub_duration":235,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":150994944,"num_objects":2304,"num_object_clones":0,"num_object_copies":4608,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2304,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":272,"num_write_kb":17408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,0],"acting":[14,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"3.14","version":"97'3624","reported_seq":7853,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-09-13T11:23:55.014100+0000","last_change":"2024-09-13T11:23:48.697100+0000","last_active":"2024-09-13T11:23:55.014100+0000","last_peered":"2024-09-13T11:23:55.014100+0000","last_clean":"2024-09-13T11:23:55.014100+0000","last_became_active":"2024-09-13T11:23:41.920989+0000","last_became_peered":"2024-09-13T11:23:41.920989+0000","last_unstale":"2024-09-13T11:23:55.014100+0000","last_undegraded":"2024-09-13T11:23:55.014100+0000","last_fullsized":"2024-09-13T11:23:55.014100+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":5,"last_scrub":"91'3371","last_scrub_stamp":"2024-09-13T11:23:48.696960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:48.696960+0000","objects_scrubbed":1935,"log_size":3624,"log_dups_size":0,"ondisk_log_size":3624,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:02.861530+0000","scrub_duration":735,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":144048128,"num_objects":2198,"num_object_clones":0,"num_object_copies":4396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2198,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":263,"num_write_kb":16832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,14],"acting":[3,14],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":159,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-09-13T11:23:55.198217+0000","last_change":"2024-09-13T11:22:16.889130+0000","last_active":"2024-09-13T11:23:55.198217+0000","last_peered":"2024-09-13T11:23:55.198217+0000","last_clean":"2024-09-13T11:23:55.198217+0000","last_became_active":"2024-09-13T11:22:14.192897+0000","last_became_peered":"2024-09-13T11:22:14.192897+0000","last_unstale":"2024-09-13T11:23:55.198217+0000","last_undegraded":"2024-09-13T11:23:55.198217+0000","last_fullsized":"2024-09-13T11:23:55.198217+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:22:13.113563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00084655899999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,10],"acting":[15,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"3.6","version":"94'3675","reported_seq":3844,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-13T11:23:51.971505+0000","last_change":"2024-09-13T11:23:22.656608+0000","last_active":"2024-09-13T11:23:51.971505+0000","last_peered":"2024-09-13T11:23:51.971505+0000","last_clean":"2024-09-13T11:23:51.971505+0000","last_became_active":"2024-09-13T11:23:19.939432+0000","last_became_peered":"2024-09-13T11:23:19.939432+0000","last_unstale":"2024-09-13T11:23:51.971505+0000","last_undegraded":"2024-09-13T11:23:51.971505+0000","last_fullsized":"2024-09-13T11:23:51.971505+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'2909","last_scrub_stamp":"2024-09-13T11:23:22.656323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:22.656323+0000","objects_scrubbed":1487,"log_size":3675,"log_dups_size":0,"ondisk_log_size":3675,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:24:31.905185+0000","scrub_duration":323,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":147652608,"num_objects":2253,"num_object_clones":0,"num_object_copies":4506,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2253,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":766,"num_write_kb":49024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,14],"acting":[13,14],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":158,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-13T11:23:51.161899+0000","last_change":"2024-09-13T11:23:21.875855+0000","last_active":"2024-09-13T11:23:51.161899+0000","last_peered":"2024-09-13T11:23:51.161899+0000","last_clean":"2024-09-13T11:23:51.161899+0000","last_became_active":"2024-09-13T11:22:14.211923+0000","last_became_peered":"2024-09-13T11:22:14.211923+0000","last_unstale":"2024-09-13T11:23:51.161899+0000","last_undegraded":"2024-09-13T11:23:51.161899+0000","last_fullsized":"2024-09-13T11:23:51.161899+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:23:21.875683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:23:21.875683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:24:40.678136+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0011689529999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0],"acting":[9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"94'3499","reported_seq":3668,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-13T11:23:52.049463+0000","last_change":"2024-09-13T11:23:20.443418+0000","last_active":"2024-09-13T11:23:52.049463+0000","last_peered":"2024-09-13T11:23:52.049463+0000","last_clean":"2024-09-13T11:23:52.049463+0000","last_became_active":"2024-09-13T11:23:19.927680+0000","last_became_peered":"2024-09-13T11:23:19.927680+0000","last_unstale":"2024-09-13T11:23:52.049463+0000","last_undegraded":"2024-09-13T11:23:52.049463+0000","last_fullsized":"2024-09-13T11:23:52.049463+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'2736","last_scrub_stamp":"2024-09-13T11:23:20.443329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:20.443329+0000","objects_scrubbed":1369,"log_size":3499,"log_dups_size":0,"ondisk_log_size":3499,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:24:48.147625+0000","scrub_duration":183,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":139919360,"num_objects":2135,"num_object_clones":0,"num_object_copies":4270,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2135,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":766,"num_write_kb":49024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,8],"acting":[13,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":159,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-09-13T11:23:55.187050+0000","last_change":"2024-09-13T11:22:16.504060+0000","last_active":"2024-09-13T11:23:55.187050+0000","last_peered":"2024-09-13T11:23:55.187050+0000","last_clean":"2024-09-13T11:23:55.187050+0000","last_became_active":"2024-09-13T11:22:14.581458+0000","last_became_peered":"2024-09-13T11:22:14.581458+0000","last_unstale":"2024-09-13T11:23:55.187050+0000","last_undegraded":"2024-09-13T11:23:55.187050+0000","last_fullsized":"2024-09-13T11:23:55.187050+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:22:13.113563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001026047,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5],"acting":[8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"98'3660","reported_seq":3818,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-09-13T11:23:55.280832+0000","last_change":"2024-09-13T11:23:20.254213+0000","last_active":"2024-09-13T11:23:55.280832+0000","last_peered":"2024-09-13T11:23:55.280832+0000","last_clean":"2024-09-13T11:23:55.280832+0000","last_became_active":"2024-09-13T11:23:19.925399+0000","last_became_peered":"2024-09-13T11:23:19.925399+0000","last_unstale":"2024-09-13T11:23:55.280832+0000","last_undegraded":"2024-09-13T11:23:55.280832+0000","last_fullsized":"2024-09-13T11:23:55.280832+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'2755","last_scrub_stamp":"2024-09-13T11:23:20.254124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:20.254124+0000","objects_scrubbed":1437,"log_size":3660,"log_dups_size":0,"ondisk_log_size":3660,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:24:49.993543+0000","scrub_duration":187,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":153812992,"num_objects":2347,"num_object_clones":0,"num_object_copies":4694,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2347,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":910,"num_write_kb":58240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,12],"acting":[1,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":147,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-09-13T11:23:55.188568+0000","last_change":"2024-09-13T11:22:47.720313+0000","last_active":"2024-09-13T11:23:55.188568+0000","last_peered":"2024-09-13T11:23:55.188568+0000","last_clean":"2024-09-13T11:23:55.188568+0000","last_became_active":"2024-09-13T11:22:47.719636+0000","last_became_peered":"2024-09-13T11:22:47.719636+0000","last_unstale":"2024-09-13T11:23:55.188568+0000","last_undegraded":"2024-09-13T11:23:55.188568+0000","last_fullsized":"2024-09-13T11:23:55.188568+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:22:13.113563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00087476000000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,9],"acting":[1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"98'3683","reported_seq":3860,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-09-13T11:23:56.040162+0000","last_change":"2024-09-13T11:23:20.686446+0000","last_active":"2024-09-13T11:23:56.040162+0000","last_peered":"2024-09-13T11:23:56.040162+0000","last_clean":"2024-09-13T11:23:56.040162+0000","last_became_active":"2024-09-13T11:23:19.939695+0000","last_became_peered":"2024-09-13T11:23:19.939695+0000","last_unstale":"2024-09-13T11:23:56.040162+0000","last_undegraded":"2024-09-13T11:23:56.040162+0000","last_fullsized":"2024-09-13T11:23:56.040162+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'2739","last_scrub_stamp":"2024-09-13T11:23:20.686358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:20.686358+0000","objects_scrubbed":1386,"log_size":3683,"log_dups_size":0,"ondisk_log_size":3683,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:24:38.891737+0000","scrub_duration":271,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":152764416,"num_objects":2331,"num_object_clones":0,"num_object_copies":4662,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2331,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":945,"num_write_kb":60480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,13],"acting":[15,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"3.3","version":"98'3381","reported_seq":3582,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-09-13T11:23:55.995850+0000","last_change":"2024-09-13T11:23:32.147115+0000","last_active":"2024-09-13T11:23:55.995850+0000","last_peered":"2024-09-13T11:23:55.995850+0000","last_clean":"2024-09-13T11:23:55.995850+0000","last_became_active":"2024-09-13T11:23:31.930448+0000","last_became_peered":"2024-09-13T11:23:31.930448+0000","last_unstale":"2024-09-13T11:23:55.995850+0000","last_undegraded":"2024-09-13T11:23:55.995850+0000","last_fullsized":"2024-09-13T11:23:55.995850+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'3122","last_scrub_stamp":"2024-09-13T11:23:32.147021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:32.147021+0000","objects_scrubbed":876,"log_size":3381,"log_dups_size":0,"ondisk_log_size":3381,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:24:48.175541+0000","scrub_duration":99,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":74383360,"num_objects":1135,"num_object_clones":0,"num_object_copies":2270,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1135,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":259,"num_write_kb":16576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,7],"acting":[10,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"28'2","reported_seq":89,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-09-13T11:23:55.186561+0000","last_change":"2024-09-13T11:23:12.560311+0000","last_active":"2024-09-13T11:23:55.186561+0000","last_peered":"2024-09-13T11:23:55.186561+0000","last_clean":"2024-09-13T11:23:55.186561+0000","last_became_active":"2024-09-13T11:23:12.546348+0000","last_became_peered":"2024-09-13T11:23:12.546348+0000","last_unstale":"2024-09-13T11:23:55.186561+0000","last_undegraded":"2024-09-13T11:23:55.186561+0000","last_fullsized":"2024-09-13T11:23:55.186561+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:22:13.113563+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0025826730000000002,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8],"acting":[2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"98'3393","reported_seq":3592,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-09-13T11:23:56.027820+0000","last_change":"2024-09-13T11:23:32.749651+0000","last_active":"2024-09-13T11:23:56.027820+0000","last_peered":"2024-09-13T11:23:56.027820+0000","last_clean":"2024-09-13T11:23:56.027820+0000","last_became_active":"2024-09-13T11:23:31.921388+0000","last_became_peered":"2024-09-13T11:23:31.921388+0000","last_unstale":"2024-09-13T11:23:56.027820+0000","last_undegraded":"2024-09-13T11:23:56.027820+0000","last_fullsized":"2024-09-13T11:23:56.027820+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'3106","last_scrub_stamp":"2024-09-13T11:23:32.749487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:32.749487+0000","objects_scrubbed":856,"log_size":3393,"log_dups_size":0,"ondisk_log_size":3393,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:24:34.709501+0000","scrub_duration":91,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":74907648,"num_objects":1143,"num_object_clones":0,"num_object_copies":2286,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1143,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":287,"num_write_kb":18368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,15],"acting":[12,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":169,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-09-13T11:23:55.208037+0000","last_change":"2024-09-13T11:22:47.298206+0000","last_active":"2024-09-13T11:23:55.208037+0000","last_peered":"2024-09-13T11:23:55.208037+0000","last_clean":"2024-09-13T11:23:55.208037+0000","last_became_active":"2024-09-13T11:22:47.297319+0000","last_became_peered":"2024-09-13T11:22:47.297319+0000","last_unstale":"2024-09-13T11:23:55.208037+0000","last_undegraded":"2024-09-13T11:23:55.208037+0000","last_fullsized":"2024-09-13T11:23:55.208037+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:22:13.113563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00081225399999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,1],"acting":[14,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"3.1","version":"94'3295","reported_seq":3486,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-13T11:23:51.940920+0000","last_change":"2024-09-13T11:23:33.604966+0000","last_active":"2024-09-13T11:23:51.940920+0000","last_peered":"2024-09-13T11:23:51.940920+0000","last_clean":"2024-09-13T11:23:51.940920+0000","last_became_active":"2024-09-13T11:23:31.926528+0000","last_became_peered":"2024-09-13T11:23:31.926528+0000","last_unstale":"2024-09-13T11:23:51.940920+0000","last_undegraded":"2024-09-13T11:23:51.940920+0000","last_fullsized":"2024-09-13T11:23:51.940920+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'3070","last_scrub_stamp":"2024-09-13T11:23:33.604792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:33.604792+0000","objects_scrubbed":842,"log_size":3295,"log_dups_size":0,"ondisk_log_size":3295,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:24:43.880038+0000","scrub_duration":119,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":69926912,"num_objects":1067,"num_object_clones":0,"num_object_copies":2134,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1067,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":225,"num_write_kb":14400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,11],"acting":[13,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":157,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-09-13T11:23:54.178449+0000","last_change":"2024-09-13T11:22:16.267999+0000","last_active":"2024-09-13T11:23:54.178449+0000","last_peered":"2024-09-13T11:23:54.178449+0000","last_clean":"2024-09-13T11:23:54.178449+0000","last_became_active":"2024-09-13T11:22:14.211435+0000","last_became_peered":"2024-09-13T11:22:14.211435+0000","last_unstale":"2024-09-13T11:23:54.178449+0000","last_undegraded":"2024-09-13T11:23:54.178449+0000","last_fullsized":"2024-09-13T11:23:54.178449+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:22:13.113563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010010360000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2","version":"98'3382","reported_seq":3582,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-09-13T11:23:55.956606+0000","last_change":"2024-09-13T11:23:32.977666+0000","last_active":"2024-09-13T11:23:55.956606+0000","last_peered":"2024-09-13T11:23:55.956606+0000","last_clean":"2024-09-13T11:23:55.956606+0000","last_became_active":"2024-09-13T11:23:31.919828+0000","last_became_peered":"2024-09-13T11:23:31.919828+0000","last_unstale":"2024-09-13T11:23:55.956606+0000","last_undegraded":"2024-09-13T11:23:55.956606+0000","last_fullsized":"2024-09-13T11:23:55.956606+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'3057","last_scrub_stamp":"2024-09-13T11:23:32.977559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:32.977559+0000","objects_scrubbed":841,"log_size":3382,"log_dups_size":0,"ondisk_log_size":3382,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:24:52.169339+0000","scrub_duration":151,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76414976,"num_objects":1166,"num_object_clones":0,"num_object_copies":2332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1166,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":325,"num_write_kb":20800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,12],"acting":[14,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"2.3","version":"26'1","reported_seq":161,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-09-13T11:23:55.208262+0000","last_change":"2024-09-13T11:22:16.547691+0000","last_active":"2024-09-13T11:23:55.208262+0000","last_peered":"2024-09-13T11:23:55.208262+0000","last_clean":"2024-09-13T11:23:55.208262+0000","last_became_active":"2024-09-13T11:22:14.146507+0000","last_became_peered":"2024-09-13T11:22:14.146507+0000","last_unstale":"2024-09-13T11:23:55.208262+0000","last_undegraded":"2024-09-13T11:23:55.208262+0000","last_fullsized":"2024-09-13T11:23:55.208262+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:22:13.113563+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00085610999999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,15],"acting":[14,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"1.0","version":"26'112","reported_seq":301,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-09-13T11:23:55.197912+0000","last_change":"2024-09-13T11:23:49.325249+0000","last_active":"2024-09-13T11:23:55.197912+0000","last_peered":"2024-09-13T11:23:55.197912+0000","last_clean":"2024-09-13T11:23:55.197912+0000","last_became_active":"2024-09-13T11:22:12.129721+0000","last_became_peered":"2024-09-13T11:22:12.129721+0000","last_unstale":"2024-09-13T11:23:55.197912+0000","last_undegraded":"2024-09-13T11:23:55.197912+0000","last_fullsized":"2024-09-13T11:23:55.197912+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"26'112","last_scrub_stamp":"2024-09-13T11:23:49.325175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:11.098388+0000","last_clean_scrub_stamp":"2024-09-13T11:23:49.325175+0000","objects_scrubbed":2,"log_size":112,"log_dups_size":0,"ondisk_log_size":112,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:14.422287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.8","version":"98'3647","reported_seq":3826,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-09-13T11:23:55.962021+0000","last_change":"2024-09-13T11:23:20.644952+0000","last_active":"2024-09-13T11:23:55.962021+0000","last_peered":"2024-09-13T11:23:55.962021+0000","last_clean":"2024-09-13T11:23:55.962021+0000","last_became_active":"2024-09-13T11:23:19.925802+0000","last_became_peered":"2024-09-13T11:23:19.925802+0000","last_unstale":"2024-09-13T11:23:55.962021+0000","last_undegraded":"2024-09-13T11:23:55.962021+0000","last_fullsized":"2024-09-13T11:23:55.962021+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'2771","last_scrub_stamp":"2024-09-13T11:23:20.644747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:20.644747+0000","objects_scrubbed":1356,"log_size":3647,"log_dups_size":0,"ondisk_log_size":3647,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:24:38.144523+0000","scrub_duration":411,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":147062784,"num_objects":2244,"num_object_clones":0,"num_object_copies":4488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2244,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":888,"num_write_kb":56832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.9","version":"98'3706","reported_seq":3882,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-09-13T11:23:56.067044+0000","last_change":"2024-09-13T11:23:22.033678+0000","last_active":"2024-09-13T11:23:56.067044+0000","last_peered":"2024-09-13T11:23:56.067044+0000","last_clean":"2024-09-13T11:23:56.067044+0000","last_became_active":"2024-09-13T11:23:19.935114+0000","last_became_peered":"2024-09-13T11:23:19.935114+0000","last_unstale":"2024-09-13T11:23:56.067044+0000","last_undegraded":"2024-09-13T11:23:56.067044+0000","last_fullsized":"2024-09-13T11:23:56.067044+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'2808","last_scrub_stamp":"2024-09-13T11:23:22.033572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:22.033572+0000","objects_scrubbed":1471,"log_size":3706,"log_dups_size":0,"ondisk_log_size":3706,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:24:46.595859+0000","scrub_duration":287,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":155254784,"num_objects":2369,"num_object_clones":0,"num_object_copies":4738,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2369,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":898,"num_write_kb":57472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,2],"acting":[12,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[]},{"pgid":"3.a","version":"97'3984","reported_seq":4164,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-09-13T11:23:54.989728+0000","last_change":"2024-09-13T11:23:32.526150+0000","last_active":"2024-09-13T11:23:54.989728+0000","last_peered":"2024-09-13T11:23:54.989728+0000","last_clean":"2024-09-13T11:23:54.989728+0000","last_became_active":"2024-09-13T11:23:31.934170+0000","last_became_peered":"2024-09-13T11:23:31.934170+0000","last_unstale":"2024-09-13T11:23:54.989728+0000","last_undegraded":"2024-09-13T11:23:54.989728+0000","last_fullsized":"2024-09-13T11:23:54.989728+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'3418","last_scrub_stamp":"2024-09-13T11:23:32.526057+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:32.526057+0000","objects_scrubbed":1774,"log_size":3984,"log_dups_size":0,"ondisk_log_size":3984,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:24:34.770161+0000","scrub_duration":507,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":153354240,"num_objects":2340,"num_object_clones":0,"num_object_copies":4680,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2340,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":566,"num_write_kb":36224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.b","version":"94'3905","reported_seq":4078,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-13T11:23:51.951796+0000","last_change":"2024-09-13T11:23:32.681343+0000","last_active":"2024-09-13T11:23:51.951796+0000","last_peered":"2024-09-13T11:23:51.951796+0000","last_clean":"2024-09-13T11:23:51.951796+0000","last_became_active":"2024-09-13T11:23:31.923415+0000","last_became_peered":"2024-09-13T11:23:31.923415+0000","last_unstale":"2024-09-13T11:23:51.951796+0000","last_undegraded":"2024-09-13T11:23:51.951796+0000","last_fullsized":"2024-09-13T11:23:51.951796+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'3471","last_scrub_stamp":"2024-09-13T11:23:32.681257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:32.681257+0000","objects_scrubbed":1720,"log_size":3905,"log_dups_size":0,"ondisk_log_size":3905,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:24:58.857812+0000","scrub_duration":199,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":141164544,"num_objects":2154,"num_object_clones":0,"num_object_copies":4308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2154,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":434,"num_write_kb":27776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,1],"acting":[13,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"3.c","version":"94'3888","reported_seq":4059,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-13T11:23:51.996813+0000","last_change":"2024-09-13T11:23:32.524870+0000","last_active":"2024-09-13T11:23:51.996813+0000","last_peered":"2024-09-13T11:23:51.996813+0000","last_clean":"2024-09-13T11:23:51.996813+0000","last_became_active":"2024-09-13T11:23:31.922620+0000","last_became_peered":"2024-09-13T11:23:31.922620+0000","last_unstale":"2024-09-13T11:23:51.996813+0000","last_undegraded":"2024-09-13T11:23:51.996813+0000","last_fullsized":"2024-09-13T11:23:51.996813+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'3463","last_scrub_stamp":"2024-09-13T11:23:32.524776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:32.524776+0000","objects_scrubbed":1741,"log_size":3888,"log_dups_size":0,"ondisk_log_size":3888,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:24:37.745747+0000","scrub_duration":235,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":141950976,"num_objects":2166,"num_object_clones":0,"num_object_copies":4332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2166,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":425,"num_write_kb":27200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,13],"acting":[11,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.d","version":"98'4003","reported_seq":4182,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-09-13T11:23:56.143293+0000","last_change":"2024-09-13T11:23:32.776835+0000","last_active":"2024-09-13T11:23:56.143293+0000","last_peered":"2024-09-13T11:23:56.143293+0000","last_clean":"2024-09-13T11:23:56.143293+0000","last_became_active":"2024-09-13T11:23:31.939246+0000","last_became_peered":"2024-09-13T11:23:31.939246+0000","last_unstale":"2024-09-13T11:23:56.143293+0000","last_undegraded":"2024-09-13T11:23:56.143293+0000","last_fullsized":"2024-09-13T11:23:56.143293+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'3395","last_scrub_stamp":"2024-09-13T11:23:32.776747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:32.776747+0000","objects_scrubbed":1704,"log_size":4003,"log_dups_size":0,"ondisk_log_size":4003,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:24:56.402862+0000","scrub_duration":179,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":151519232,"num_objects":2312,"num_object_clones":0,"num_object_copies":4624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2312,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":608,"num_write_kb":38912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.e","version":"98'4131","reported_seq":9240,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-09-13T11:23:56.036465+0000","last_change":"2024-09-13T11:23:32.368856+0000","last_active":"2024-09-13T11:23:56.036465+0000","last_peered":"2024-09-13T11:23:56.036465+0000","last_clean":"2024-09-13T11:23:56.036465+0000","last_became_active":"2024-09-13T11:23:31.925869+0000","last_became_peered":"2024-09-13T11:23:31.925869+0000","last_unstale":"2024-09-13T11:23:56.036465+0000","last_undegraded":"2024-09-13T11:23:56.036465+0000","last_fullsized":"2024-09-13T11:23:56.036465+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'3527","last_scrub_stamp":"2024-09-13T11:23:32.368727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:32.368727+0000","objects_scrubbed":1774,"log_size":4131,"log_dups_size":0,"ondisk_log_size":4131,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:24:57.811779+0000","scrub_duration":219,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":155910144,"num_objects":2379,"num_object_clones":0,"num_object_copies":4758,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2379,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":605,"num_write_kb":38720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"94'3835","reported_seq":4007,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-13T11:23:52.022840+0000","last_change":"2024-09-13T11:23:33.492630+0000","last_active":"2024-09-13T11:23:52.022840+0000","last_peered":"2024-09-13T11:23:52.022840+0000","last_clean":"2024-09-13T11:23:52.022840+0000","last_became_active":"2024-09-13T11:23:31.935239+0000","last_became_peered":"2024-09-13T11:23:31.935239+0000","last_unstale":"2024-09-13T11:23:52.022840+0000","last_undegraded":"2024-09-13T11:23:52.022840+0000","last_fullsized":"2024-09-13T11:23:52.022840+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'3420","last_scrub_stamp":"2024-09-13T11:23:33.492488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:33.492488+0000","objects_scrubbed":1776,"log_size":3835,"log_dups_size":0,"ondisk_log_size":3835,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:24:44.851261+0000","scrub_duration":215,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":143589376,"num_objects":2191,"num_object_clones":0,"num_object_copies":4382,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2191,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":415,"num_write_kb":26560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,12],"acting":[11,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.10","version":"98'3756","reported_seq":8091,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-09-13T11:23:56.020141+0000","last_change":"2024-09-13T11:23:48.519654+0000","last_active":"2024-09-13T11:23:56.020141+0000","last_peered":"2024-09-13T11:23:56.020141+0000","last_clean":"2024-09-13T11:23:56.020141+0000","last_became_active":"2024-09-13T11:23:41.931183+0000","last_became_peered":"2024-09-13T11:23:41.931183+0000","last_unstale":"2024-09-13T11:23:56.020141+0000","last_undegraded":"2024-09-13T11:23:56.020141+0000","last_fullsized":"2024-09-13T11:23:56.020141+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":5,"last_scrub":"91'3468","last_scrub_stamp":"2024-09-13T11:23:48.519574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:48.519574+0000","objects_scrubbed":2031,"log_size":3756,"log_dups_size":0,"ondisk_log_size":3756,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:24:57.342213+0000","scrub_duration":215,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":152109056,"num_objects":2321,"num_object_clones":0,"num_object_copies":4642,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2321,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":290,"num_write_kb":18560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.11","version":"98'3682","reported_seq":6795,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-09-13T11:23:55.998860+0000","last_change":"2024-09-13T11:23:46.102466+0000","last_active":"2024-09-13T11:23:55.998860+0000","last_peered":"2024-09-13T11:23:55.998860+0000","last_clean":"2024-09-13T11:23:55.998860+0000","last_became_active":"2024-09-13T11:23:39.063172+0000","last_became_peered":"2024-09-13T11:23:39.063172+0000","last_unstale":"2024-09-13T11:23:55.998860+0000","last_undegraded":"2024-09-13T11:23:55.998860+0000","last_fullsized":"2024-09-13T11:23:55.998860+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":5,"last_scrub":"88'3345","last_scrub_stamp":"2024-09-13T11:23:46.101973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:46.101973+0000","objects_scrubbed":2011,"log_size":3682,"log_dups_size":0,"ondisk_log_size":3682,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:08.671350+0000","scrub_duration":955,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154140672,"num_objects":2352,"num_object_clones":0,"num_object_copies":4704,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2352,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":341,"num_write_kb":21824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11],"acting":[4,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.12","version":"98'3714","reported_seq":6750,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-09-13T11:23:56.015916+0000","last_change":"2024-09-13T11:23:35.111409+0000","last_active":"2024-09-13T11:23:56.015916+0000","last_peered":"2024-09-13T11:23:56.015916+0000","last_clean":"2024-09-13T11:23:56.015916+0000","last_became_active":"2024-09-13T11:23:21.998877+0000","last_became_peered":"2024-09-13T11:23:21.998877+0000","last_unstale":"2024-09-13T11:23:56.015916+0000","last_undegraded":"2024-09-13T11:23:56.015916+0000","last_fullsized":"2024-09-13T11:23:56.015916+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'3109","last_scrub_stamp":"2024-09-13T11:23:35.111320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:35.111320+0000","objects_scrubbed":1732,"log_size":3714,"log_dups_size":0,"ondisk_log_size":3714,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:03.371433+0000","scrub_duration":239,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":153157632,"num_objects":2337,"num_object_clones":0,"num_object_copies":4674,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2337,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":605,"num_write_kb":38720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,7],"acting":[14,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"3.13","version":"94'3570","reported_seq":6601,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-13T11:23:52.055839+0000","last_change":"2024-09-13T11:23:47.565094+0000","last_active":"2024-09-13T11:23:52.055839+0000","last_peered":"2024-09-13T11:23:52.055839+0000","last_clean":"2024-09-13T11:23:52.055839+0000","last_became_active":"2024-09-13T11:23:41.082433+0000","last_became_peered":"2024-09-13T11:23:41.082433+0000","last_unstale":"2024-09-13T11:23:52.055839+0000","last_undegraded":"2024-09-13T11:23:52.055839+0000","last_fullsized":"2024-09-13T11:23:52.055839+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'3408","last_scrub_stamp":"2024-09-13T11:23:47.564931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:47.564931+0000","objects_scrubbed":1981,"log_size":3570,"log_dups_size":0,"ondisk_log_size":3570,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:04.258231+0000","scrub_duration":471,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":141230080,"num_objects":2155,"num_object_clones":0,"num_object_copies":4310,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2155,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":174,"num_write_kb":11136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7],"acting":[11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":36,"stat_sum":{"num_bytes":4752113664,"num_objects":72513,"num_object_clones":0,"num_object_copies":145026,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6760,"num_objects_unfound":0,"num_objects_dirty":72513,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":28528,"num_write_kb":1825696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9884,"num_bytes_recovered":647725056,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":11813322752,"data_stored":11813322752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":132954,"ondisk_log_size":132954,"up":72,"acting":72,"num_store_stats":16},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":10},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":112,"ondisk_log_size":112,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":13,"up_from":23,"seq":98784247832,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1097252,"kb_used_data":1034132,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":93274588,"statfs":{"total":96636764160,"available":95513178112,"internally_reserved":0,"allocated":1058951168,"data_stored":1058533381,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[0,1,3,4,6,8,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":9,"up_from":23,"seq":98784247831,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":323796,"kb_used_data":292612,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94048044,"statfs":{"total":96636764160,"available":96305197056,"internally_reserved":0,"allocated":299634688,"data_stored":299232379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,8,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":40,"seq":171798691855,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1260772,"kb_used_data":1192660,"kb_used_omap":4,"kb_used_meta":68091,"kb_avail":93111068,"statfs":{"total":96636764160,"available":95345733632,"internally_reserved":0,"allocated":1221283840,"data_stored":1220832136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4433,"internal_metadata":69725871},"hb_peers":[0,2,3,6,8,9,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":21,"seq":90194313240,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":901600,"kb_used_data":856528,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":93470240,"statfs":{"total":96636764160,"available":95713525760,"internally_reserved":0,"allocated":877084672,"data_stored":876634785,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":46135751},"hb_peers":[0,2,4,7,9,10,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":21,"seq":90194313243,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":234152,"kb_used_data":194648,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94137688,"statfs":{"total":96636764160,"available":96396992512,"internally_reserved":0,"allocated":199319552,"data_stored":198865634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40434119},"hb_peers":[0,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":22,"seq":94489280535,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1015504,"kb_used_data":944064,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":93356336,"statfs":{"total":96636764160,"available":95596888064,"internally_reserved":0,"allocated":966721536,"data_stored":966345025,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73136581},"hb_peers":[1,3,4,5,8,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":22,"seq":94489280536,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":395228,"kb_used_data":359884,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93976612,"statfs":{"total":96636764160,"available":96232050688,"internally_reserved":0,"allocated":368521216,"data_stored":368083624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,7,8,9,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":15,"up_from":22,"seq":94489280536,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":838044,"kb_used_data":774924,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":93533796,"statfs":{"total":96636764160,"available":95778607104,"internally_reserved":0,"allocated":793522176,"data_stored":793141316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":64616902},"hb_peers":[0,3,4,5,6,7,10,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":22,"seq":94489280536,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":422956,"kb_used_data":383452,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":93948884,"statfs":{"total":96636764160,"available":96203657216,"internally_reserved":0,"allocated":392654848,"data_stored":392202303,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[3,5,7,9,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":22,"seq":94489280536,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":404284,"kb_used_data":368940,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93967556,"statfs":{"total":96636764160,"available":96222777344,"internally_reserved":0,"allocated":377794560,"data_stored":377384292,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,3,5,7,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":22,"seq":94489280536,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1161328,"kb_used_data":1081568,"kb_used_omap":1,"kb_used_meta":79742,"kb_avail":93210512,"statfs":{"total":96636764160,"available":95447564288,"internally_reserved":0,"allocated":1107525632,"data_stored":1107125067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":81656266},"hb_peers":[0,3,5,6,8,9,10,11,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":22,"seq":94489280536,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":485428,"kb_used_data":458404,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93886412,"statfs":{"total":96636764160,"available":96139685888,"internally_reserved":0,"allocated":469405696,"data_stored":468941270,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,5,7,9,10,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":22,"seq":94489280535,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":588460,"kb_used_data":557276,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93783380,"statfs":{"total":96636764160,"available":96034181120,"internally_reserved":0,"allocated":570650624,"data_stored":570248918,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,3,6,7,9,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":11,"up_from":22,"seq":94489280536,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1159964,"kb_used_data":1096844,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":93211876,"statfs":{"total":96636764160,"available":95448961024,"internally_reserved":0,"allocated":1123168256,"data_stored":1122761730,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":64616900},"hb_peers":[0,2,4,7,8,10,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,0,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":12,"up_from":22,"seq":94489280536,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":970648,"kb_used_data":899208,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":93401192,"statfs":{"total":96636764160,"available":95642820608,"internally_reserved":0,"allocated":920788992,"data_stored":920352163,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73136581},"hb_peers":[0,1,2,3,5,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":14,"up_from":22,"seq":94489280536,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1047512,"kb_used_data":971912,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":93324328,"statfs":{"total":96636764160,"available":95564111872,"internally_reserved":0,"allocated":995237888,"data_stored":994799398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,1,5,6,7,8,9,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":986972160,"data_stored":986972160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1222311936,"data_stored":1222311936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":367132672,"data_stored":367132672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":878641152,"data_stored":878641152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":391577600,"data_stored":391577600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":199098368,"data_stored":199098368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":379125760,"data_stored":379125760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1111490560,"data_stored":1111490560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":468844544,"data_stored":468844544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":299565056,"data_stored":299565056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":578420736,"data_stored":578420736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":1140719616,"data_stored":1140719616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":928645120,"data_stored":928645120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":1066074112,"data_stored":1066074112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":993460224,"data_stored":993460224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":801243136,"data_stored":801243136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T11:23:58.042 INFO:teuthology.orchestra.run.smithi001.stdout:osd.0: {} 2024-09-13T11:23:58.042 INFO:teuthology.orchestra.run.smithi001.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T11:23:58.057 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-09-13T11:23:58.057 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:23:58.072 INFO:teuthology.orchestra.run.smithi001.stdout:osd.1: {} 2024-09-13T11:23:58.072 INFO:teuthology.orchestra.run.smithi001.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T11:23:58.076 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-09-13T11:23:58.073+0000 7f97befc9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26315) UID: 0 2024-09-13T11:23:58.080 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-13T11:23:58.102 INFO:teuthology.orchestra.run.smithi001.stdout:osd.2: {} 2024-09-13T11:23:58.102 INFO:teuthology.orchestra.run.smithi001.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T11:23:58.131 INFO:teuthology.orchestra.run.smithi001.stdout:osd.3: {} 2024-09-13T11:23:58.132 INFO:teuthology.orchestra.run.smithi001.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T11:23:58.161 INFO:teuthology.orchestra.run.smithi001.stdout:osd.4: {} 2024-09-13T11:23:58.161 INFO:teuthology.orchestra.run.smithi001.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T11:23:58.177 INFO:tasks.ceph.osd.5.smithi092.stderr:2024-09-13T11:23:58.175+0000 7f6f0b705640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26346) UID: 0 2024-09-13T11:23:58.191 INFO:teuthology.orchestra.run.smithi001.stdout:osd.5: {} 2024-09-13T11:23:58.192 INFO:teuthology.orchestra.run.smithi001.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T11:23:58.222 INFO:teuthology.orchestra.run.smithi001.stdout:osd.6: {} 2024-09-13T11:23:58.222 INFO:teuthology.orchestra.run.smithi001.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T11:23:58.244 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-13T11:23:58.249 INFO:teuthology.orchestra.run.smithi001.stdout:osd.7: {} 2024-09-13T11:23:58.249 INFO:teuthology.orchestra.run.smithi001.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T11:23:58.268 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-09-13T11:23:58.268 INFO:teuthology.orchestra.run.smithi177.stdout: "size": 20, 2024-09-13T11:23:58.268 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 600, 2024-09-13T11:23:58.268 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [ 2024-09-13T11:23:58.268 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:58.268 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPush(3.12 74/69 [PushOp(3:4f110bb2:::benchmark_data_smithi177_31116_object30564:head, version: 59'1932, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:4f110bb2:::benchmark_data_smithi177_31116_object30564:head@59'1932, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:23:58.268 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:30.835784+0000", 2024-09-13T11:23:58.268 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 27.430487669000001, 2024-09-13T11:23:58.269 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.12151646200000001, 2024-09-13T11:23:58.269 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:23:58.269 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:23:58.269 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:23:58.269 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:23:58.269 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:58.269 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:23:58.269 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.835784+0000", 2024-09-13T11:23:58.269 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:58.269 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:58.269 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:58.269 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:23:58.270 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.835784+0000", 2024-09-13T11:23:58.270 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:58.270 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:58.270 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:58.270 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:23:58.270 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.835785+0000", 2024-09-13T11:23:58.270 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.734e-06 2024-09-13T11:23:58.270 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:58.270 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:58.271 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:23:58.271 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.835824+0000", 2024-09-13T11:23:58.271 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.8804999999999999e-05 2024-09-13T11:23:58.271 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:58.271 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:58.271 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:23:58.271 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.955218+0000", 2024-09-13T11:23:58.271 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.11939419800000001 2024-09-13T11:23:58.272 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:58.272 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:58.272 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:23:58.272 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.955253+0000", 2024-09-13T11:23:58.272 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.4254000000000001e-05 2024-09-13T11:23:58.272 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:58.272 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:58.272 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:23:58.272 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.955436+0000", 2024-09-13T11:23:58.272 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000183411 2024-09-13T11:23:58.272 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:58.273 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:58.273 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:23:58.273 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.955458+0000", 2024-09-13T11:23:58.273 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.2062999999999999e-05 2024-09-13T11:23:58.273 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:58.273 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:58.273 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:23:58.273 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.957300+0000", 2024-09-13T11:23:58.273 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0018419969999999999 2024-09-13T11:23:58.273 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:58.273 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:23:58.273 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:58.274 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:58.274 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:58.274 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:54598 3.13 3:cb6b60f3:::benchmark_data_smithi177_31116_object54597:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e74)", 2024-09-13T11:23:58.274 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:30.859951+0000", 2024-09-13T11:23:58.274 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 27.406319792000001, 2024-09-13T11:23:58.274 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.10132959900000001, 2024-09-13T11:23:58.274 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:23:58.274 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:23:58.274 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:58.274 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:23:58.274 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:23:58.274 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:58.275 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 54598 2024-09-13T11:23:58.275 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:58.275 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:23:58.275 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:58.275 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:23:58.275 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.859951+0000", 2024-09-13T11:23:58.275 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:58.275 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:58.275 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:58.275 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:23:58.275 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.859951+0000", 2024-09-13T11:23:58.275 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:58.276 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:58.276 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:58.276 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:23:58.276 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.859953+0000", 2024-09-13T11:23:58.276 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.573e-06 2024-09-13T11:23:58.276 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:58.276 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:58.276 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:23:58.276 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.859972+0000", 2024-09-13T11:23:58.276 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.8975e-05 2024-09-13T11:23:58.276 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:58.277 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:58.277 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:23:58.277 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.939205+0000", 2024-09-13T11:23:58.277 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.079233245999999993 2024-09-13T11:23:58.277 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:58.277 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:58.277 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:23:58.277 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.939242+0000", 2024-09-13T11:23:58.277 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.7218000000000002e-05 2024-09-13T11:23:58.277 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:58.277 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:58.277 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:23:58.278 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.939896+0000", 2024-09-13T11:23:58.278 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00065337499999999998 2024-09-13T11:23:58.278 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:58.278 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:58.278 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:23:58.278 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.940165+0000", 2024-09-13T11:23:58.278 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00026946700000000001 2024-09-13T11:23:58.278 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:58.278 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:58.278 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 11", 2024-09-13T11:23:58.278 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.940608+0000", 2024-09-13T11:23:58.278 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00044320500000000001 2024-09-13T11:23:58.279 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:58.279 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:58.279 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:23:58.279 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.944414+0000", 2024-09-13T11:23:58.279 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0038051869999999998 2024-09-13T11:23:58.279 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:58.279 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:58.279 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:58.279 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.960399+0000", 2024-09-13T11:23:58.279 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.015985141000000001 2024-09-13T11:23:58.279 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:58.279 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:58.280 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:23:58.280 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.960482+0000", 2024-09-13T11:23:58.280 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.3398000000000003e-05 2024-09-13T11:23:58.280 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:58.280 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:58.280 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:23:58.280 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.961281+0000", 2024-09-13T11:23:58.280 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00079881400000000001 2024-09-13T11:23:58.280 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:58.280 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:23:58.280 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:58.280 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:58.281 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:58.281 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:54663 3.12 e75/69 3:49ad4b1b:::benchmark_data_smithi177_31116_object54662:head v 75'3095, mlcod=59'1933)", 2024-09-13T11:23:58.281 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:31.033716+0000", 2024-09-13T11:23:58.281 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 27.232555606999998, 2024-09-13T11:23:58.281 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.121029233, 2024-09-13T11:23:58.281 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:23:58.281 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:23:58.281 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:58.281 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:23:58.281 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:58.281 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:23:58.281 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.033716+0000", 2024-09-13T11:23:58.282 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:58.282 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:58.282 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:58.282 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:23:58.282 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.033716+0000", 2024-09-13T11:23:58.282 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:58.282 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:58.282 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:58.282 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:23:58.282 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.033724+0000", 2024-09-13T11:23:58.282 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.2910000000000007e-06 2024-09-13T11:23:58.283 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:58.283 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:58.283 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:23:58.283 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.033783+0000", 2024-09-13T11:23:58.283 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.8826999999999998e-05 2024-09-13T11:23:58.283 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:58.283 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:58.283 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:23:58.283 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.138806+0000", 2024-09-13T11:23:58.283 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.105023701 2024-09-13T11:23:58.283 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:58.283 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:58.284 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:23:58.284 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.138814+0000", 2024-09-13T11:23:58.284 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.5340000000000002e-06 2024-09-13T11:23:58.284 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:58.284 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:58.284 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:23:58.284 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.138871+0000", 2024-09-13T11:23:58.284 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.6988999999999999e-05 2024-09-13T11:23:58.284 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:58.284 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:58.285 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:23:58.285 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.138895+0000", 2024-09-13T11:23:58.285 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.3887e-05 2024-09-13T11:23:58.285 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:58.285 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:58.285 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:23:58.285 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.154684+0000", 2024-09-13T11:23:58.285 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.015788710000000001 2024-09-13T11:23:58.285 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:58.286 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:58.286 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:23:58.286 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.154745+0000", 2024-09-13T11:23:58.286 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.1294e-05 2024-09-13T11:23:58.286 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:58.286 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:23:58.286 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:58.286 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:58.286 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:58.286 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPushReply(3.13 75/70 [PushReplyOp(3:cfab03d6:::benchmark_data_smithi177_31116_object16599:head)])", 2024-09-13T11:23:58.286 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:31.163480+0000", 2024-09-13T11:23:58.286 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 27.102791244999999, 2024-09-13T11:23:58.287 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.63009029299999997, 2024-09-13T11:23:58.287 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:23:58.287 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:23:58.287 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-13T11:23:58.287 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:23:58.287 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:58.287 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:23:58.287 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.163480+0000", 2024-09-13T11:23:58.287 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:58.287 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:58.287 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:58.287 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:23:58.287 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.163480+0000", 2024-09-13T11:23:58.288 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:58.288 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:58.288 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:58.288 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:23:58.288 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.163481+0000", 2024-09-13T11:23:58.288 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0410000000000001e-06 2024-09-13T11:23:58.288 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:58.288 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:58.288 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:23:58.288 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.163485+0000", 2024-09-13T11:23:58.288 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.1629999999999998e-06 2024-09-13T11:23:58.289 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:58.289 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:58.289 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:23:58.289 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.792166+0000", 2024-09-13T11:23:58.289 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.62868067400000005 2024-09-13T11:23:58.289 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:58.289 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:58.289 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:23:58.289 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.792184+0000", 2024-09-13T11:23:58.289 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.7898999999999999e-05 2024-09-13T11:23:58.289 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:58.290 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:58.290 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:23:58.290 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.792312+0000", 2024-09-13T11:23:58.290 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00012871000000000001 2024-09-13T11:23:58.290 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:58.290 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:58.290 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:23:58.290 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.793570+0000", 2024-09-13T11:23:58.290 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001257806 2024-09-13T11:23:58.290 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:58.290 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:23:58.290 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:58.291 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:58.291 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:58.291 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPush(3.12 75/69 [PushOp(3:4a36ca6d:::benchmark_data_smithi177_31116_object31707:head, version: 59'1993, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:4a36ca6d:::benchmark_data_smithi177_31116_object31707:head@59'1993, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:23:58.291 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:31.309623+0000", 2024-09-13T11:23:58.291 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 26.95664829, 2024-09-13T11:23:58.291 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.165459152, 2024-09-13T11:23:58.291 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:23:58.291 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:23:58.292 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:23:58.292 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:23:58.292 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:58.292 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:23:58.292 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.309623+0000", 2024-09-13T11:23:58.292 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:58.292 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:58.292 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:58.292 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:23:58.292 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.309623+0000", 2024-09-13T11:23:58.293 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:58.293 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:58.293 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:58.293 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:23:58.293 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.309625+0000", 2024-09-13T11:23:58.293 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.3860000000000001e-06 2024-09-13T11:23:58.293 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:58.293 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:58.293 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:23:58.293 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.309680+0000", 2024-09-13T11:23:58.293 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.4388000000000002e-05 2024-09-13T11:23:58.294 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:58.294 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:58.294 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:23:58.294 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.473157+0000", 2024-09-13T11:23:58.294 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.163476813 2024-09-13T11:23:58.294 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:58.294 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:58.294 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:23:58.294 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.473189+0000", 2024-09-13T11:23:58.294 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.2975000000000002e-05 2024-09-13T11:23:58.294 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:58.294 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:58.295 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:23:58.295 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.473395+0000", 2024-09-13T11:23:58.295 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000206009 2024-09-13T11:23:58.295 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:58.295 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:58.295 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:23:58.295 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.473424+0000", 2024-09-13T11:23:58.295 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.8255e-05 2024-09-13T11:23:58.295 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:58.295 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:58.295 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:23:58.296 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.475082+0000", 2024-09-13T11:23:58.296 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0016583259999999999 2024-09-13T11:23:58.296 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:58.296 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:23:58.296 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:58.296 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:58.296 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:58.296 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPushReply(3.13 76/70 [PushReplyOp(3:c9d43a82:::benchmark_data_smithi177_31116_object20026:head)])", 2024-09-13T11:23:58.296 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:32.427709+0000", 2024-09-13T11:23:58.296 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 25.838561729999999, 2024-09-13T11:23:58.296 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.46547316300000002, 2024-09-13T11:23:58.297 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:23:58.297 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:23:58.297 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-13T11:23:58.297 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:23:58.297 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:58.297 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:23:58.297 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:32.427709+0000", 2024-09-13T11:23:58.297 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:58.297 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:58.297 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:58.297 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:23:58.298 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:32.427709+0000", 2024-09-13T11:23:58.298 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:58.298 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:58.298 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:58.298 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:23:58.298 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:32.427711+0000", 2024-09-13T11:23:58.298 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5099999999999999e-06 2024-09-13T11:23:58.298 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:58.298 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:58.298 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:23:58.298 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:32.427717+0000", 2024-09-13T11:23:58.299 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.1569999999999997e-06 2024-09-13T11:23:58.299 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:58.299 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:58.299 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:23:58.299 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:32.892390+0000", 2024-09-13T11:23:58.299 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.46467251100000001 2024-09-13T11:23:58.299 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:58.299 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:58.299 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:23:58.299 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:32.892405+0000", 2024-09-13T11:23:58.299 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5296999999999999e-05 2024-09-13T11:23:58.300 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:58.300 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:58.300 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:23:58.300 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:32.892503+0000", 2024-09-13T11:23:58.300 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.7820999999999997e-05 2024-09-13T11:23:58.300 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:58.300 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:58.300 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:23:58.300 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:32.893183+0000", 2024-09-13T11:23:58.300 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00067986700000000001 2024-09-13T11:23:58.301 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:58.301 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:23:58.301 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:58.301 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:58.301 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:58.301 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:56066 3.a e79/75 3:561a1dd2:::benchmark_data_smithi177_31116_object56065:head v 79'3452, mlcod=79'3449)", 2024-09-13T11:23:58.301 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:36.194429+0000", 2024-09-13T11:23:58.301 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 22.071841845000002, 2024-09-13T11:23:58.301 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.13436872499999999, 2024-09-13T11:23:58.301 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:23:58.301 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:23:58.302 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:58.302 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:23:58.302 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:58.302 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:23:58.302 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:36.194429+0000", 2024-09-13T11:23:58.302 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:58.302 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:58.302 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:58.302 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:23:58.302 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:36.194429+0000", 2024-09-13T11:23:58.302 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:58.302 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:58.302 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:58.303 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:23:58.303 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:36.194431+0000", 2024-09-13T11:23:58.303 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.4279999999999999e-06 2024-09-13T11:23:58.303 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:58.303 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:58.303 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:23:58.303 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:36.194467+0000", 2024-09-13T11:23:58.303 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.6402000000000001e-05 2024-09-13T11:23:58.303 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:58.303 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:58.303 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:23:58.303 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:36.327354+0000", 2024-09-13T11:23:58.304 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.13288720800000001 2024-09-13T11:23:58.304 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:58.304 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:58.304 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:23:58.304 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:36.327361+0000", 2024-09-13T11:23:58.304 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.302e-06 2024-09-13T11:23:58.304 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:58.304 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:58.304 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:23:58.304 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:36.327441+0000", 2024-09-13T11:23:58.304 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.0406000000000002e-05 2024-09-13T11:23:58.304 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:58.305 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:58.305 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:23:58.305 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:36.327466+0000", 2024-09-13T11:23:58.305 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.5367e-05 2024-09-13T11:23:58.305 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:58.305 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:58.305 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:23:58.305 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:36.328754+0000", 2024-09-13T11:23:58.305 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001287627 2024-09-13T11:23:58.305 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:58.305 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:58.305 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:23:58.306 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:36.328798+0000", 2024-09-13T11:23:58.306 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.3985e-05 2024-09-13T11:23:58.306 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:58.306 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:23:58.306 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:58.306 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:58.306 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:58.306 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:56096 3.d 3:b305b87e:::benchmark_data_smithi177_31116_object56095:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e79)", 2024-09-13T11:23:58.306 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:36.226921+0000", 2024-09-13T11:23:58.306 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 22.03935049, 2024-09-13T11:23:58.306 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.11682587899999999, 2024-09-13T11:23:58.306 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:23:58.307 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:23:58.307 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:58.307 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:23:58.307 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:23:58.307 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:58.307 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 56096 2024-09-13T11:23:58.307 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:58.307 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:23:58.307 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:58.307 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:23:58.307 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:36.226921+0000", 2024-09-13T11:23:58.308 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:58.308 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:58.308 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:58.308 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:23:58.308 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:36.226921+0000", 2024-09-13T11:23:58.308 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:58.308 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:58.308 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:58.308 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:23:58.308 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:36.226923+0000", 2024-09-13T11:23:58.308 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.843e-06 2024-09-13T11:23:58.309 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:58.309 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:58.309 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:23:58.309 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:36.226944+0000", 2024-09-13T11:23:58.309 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.1571999999999999e-05 2024-09-13T11:23:58.309 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:58.309 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:58.309 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:23:58.309 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:36.226954+0000", 2024-09-13T11:23:58.309 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.8970000000000003e-06 2024-09-13T11:23:58.309 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:58.309 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:58.310 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:23:58.310 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:36.226959+0000", 2024-09-13T11:23:58.310 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.4210000000000003e-06 2024-09-13T11:23:58.310 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:58.310 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:58.310 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:23:58.310 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:36.227012+0000", 2024-09-13T11:23:58.310 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.2578999999999999e-05 2024-09-13T11:23:58.310 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:58.310 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:58.310 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:23:58.310 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:36.227112+0000", 2024-09-13T11:23:58.310 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000100231 2024-09-13T11:23:58.311 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:58.311 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:58.311 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 3", 2024-09-13T11:23:58.311 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:36.227352+0000", 2024-09-13T11:23:58.311 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00023945999999999999 2024-09-13T11:23:58.311 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:58.311 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:58.311 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:23:58.311 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:36.228973+0000", 2024-09-13T11:23:58.311 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001621003 2024-09-13T11:23:58.311 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:58.311 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:58.311 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:58.312 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:36.343652+0000", 2024-09-13T11:23:58.312 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.114678868 2024-09-13T11:23:58.312 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:58.312 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:58.312 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:23:58.312 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:36.343706+0000", 2024-09-13T11:23:58.312 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.4505000000000001e-05 2024-09-13T11:23:58.312 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:58.312 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:58.312 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:23:58.312 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:36.343747+0000", 2024-09-13T11:23:58.313 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.0500000000000002e-05 2024-09-13T11:23:58.313 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:58.313 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:23:58.313 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:58.313 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:58.313 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:58.313 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPushReply(3.23 80/78 [PushReplyOp(3:c54a2e13:::benchmark_data_smithi177_31116_object16923:head)])", 2024-09-13T11:23:58.313 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:37.275242+0000", 2024-09-13T11:23:58.313 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 20.991029518000001, 2024-09-13T11:23:58.313 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.098976975999999994, 2024-09-13T11:23:58.313 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:23:58.313 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:23:58.314 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-13T11:23:58.314 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:23:58.314 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:58.314 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:23:58.314 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:37.275242+0000", 2024-09-13T11:23:58.314 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:58.314 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:58.314 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:58.314 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:23:58.314 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:37.275242+0000", 2024-09-13T11:23:58.314 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:58.314 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:58.315 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:58.315 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:23:58.315 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:37.275244+0000", 2024-09-13T11:23:58.315 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.813e-06 2024-09-13T11:23:58.315 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:58.315 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:58.315 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:23:58.315 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:37.275251+0000", 2024-09-13T11:23:58.315 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.678e-06 2024-09-13T11:23:58.315 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:58.315 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:58.316 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:23:58.316 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:37.373653+0000", 2024-09-13T11:23:58.316 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.098401605000000003 2024-09-13T11:23:58.316 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:58.316 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:58.316 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:23:58.316 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:37.373658+0000", 2024-09-13T11:23:58.316 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.6799999999999998e-06 2024-09-13T11:23:58.316 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:58.316 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:58.316 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:23:58.316 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:37.373702+0000", 2024-09-13T11:23:58.317 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.3996999999999999e-05 2024-09-13T11:23:58.317 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:58.317 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:58.317 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:23:58.317 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:37.374219+0000", 2024-09-13T11:23:58.317 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00051620300000000002 2024-09-13T11:23:58.317 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:58.317 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:23:58.318 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:58.318 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:58.318 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:58.318 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:57435 3.3 e80/75 3:c0db4777:::benchmark_data_smithi177_31116_object57434:head v 80'3155, mlcod=80'3153)", 2024-09-13T11:23:58.318 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:37.437070+0000", 2024-09-13T11:23:58.318 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 20.829200913000001, 2024-09-13T11:23:58.318 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.35861845999999997, 2024-09-13T11:23:58.318 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:23:58.318 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:23:58.318 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:58.319 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:23:58.319 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:58.319 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:23:58.319 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:37.437070+0000", 2024-09-13T11:23:58.319 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:58.319 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:58.319 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:58.319 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:23:58.319 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:37.437070+0000", 2024-09-13T11:23:58.319 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:58.319 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:58.319 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:58.320 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:23:58.320 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:37.437072+0000", 2024-09-13T11:23:58.320 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5549999999999999e-06 2024-09-13T11:23:58.320 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:58.320 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:58.320 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:23:58.320 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:37.437141+0000", 2024-09-13T11:23:58.320 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.9494999999999998e-05 2024-09-13T11:23:58.320 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:58.320 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:58.320 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:23:58.321 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:37.793400+0000", 2024-09-13T11:23:58.321 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.35625817999999998 2024-09-13T11:23:58.321 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:58.321 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:58.321 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:23:58.321 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:37.793406+0000", 2024-09-13T11:23:58.321 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.6830000000000002e-06 2024-09-13T11:23:58.321 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:58.321 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:58.321 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:23:58.321 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:37.793472+0000", 2024-09-13T11:23:58.322 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.5671000000000002e-05 2024-09-13T11:23:58.322 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:58.322 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:58.322 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:23:58.322 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:37.793492+0000", 2024-09-13T11:23:58.322 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.0051000000000002e-05 2024-09-13T11:23:58.322 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:58.322 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:58.322 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:23:58.322 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:37.795626+0000", 2024-09-13T11:23:58.322 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0021344990000000002 2024-09-13T11:23:58.323 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:58.323 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:58.323 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:23:58.323 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:37.795689+0000", 2024-09-13T11:23:58.323 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.2325999999999999e-05 2024-09-13T11:23:58.323 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:58.323 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:23:58.323 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:58.323 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:58.323 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:58.323 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:57985 3.3 e80/75 3:c330bb03:::benchmark_data_smithi177_31116_object57984:head v 80'3162, mlcod=80'3159)", 2024-09-13T11:23:58.323 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:38.007076+0000", 2024-09-13T11:23:58.324 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 20.259195560999999, 2024-09-13T11:23:58.324 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.18211438199999999, 2024-09-13T11:23:58.324 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:23:58.324 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:23:58.324 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:58.324 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:23:58.324 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:58.324 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:23:58.324 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:38.007076+0000", 2024-09-13T11:23:58.324 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:58.324 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:58.325 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:58.325 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:23:58.325 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:38.007076+0000", 2024-09-13T11:23:58.325 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:58.325 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:58.325 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:58.325 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:23:58.325 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:38.007077+0000", 2024-09-13T11:23:58.325 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.6759999999999999e-06 2024-09-13T11:23:58.325 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:58.325 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:58.326 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:23:58.326 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:38.007221+0000", 2024-09-13T11:23:58.326 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00014344299999999999 2024-09-13T11:23:58.326 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:58.326 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:58.326 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:23:58.326 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:38.186808+0000", 2024-09-13T11:23:58.326 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.17958726699999999 2024-09-13T11:23:58.326 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:58.326 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:58.326 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:23:58.326 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:38.186815+0000", 2024-09-13T11:23:58.327 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.0890000000000002e-06 2024-09-13T11:23:58.327 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:58.327 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:58.327 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:23:58.327 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:38.186887+0000", 2024-09-13T11:23:58.327 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.2006000000000001e-05 2024-09-13T11:23:58.327 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:58.327 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:58.327 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:23:58.327 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:38.186913+0000", 2024-09-13T11:23:58.327 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.5962e-05 2024-09-13T11:23:58.327 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:58.328 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:58.328 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:23:58.328 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:38.189123+0000", 2024-09-13T11:23:58.328 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0022099649999999999 2024-09-13T11:23:58.328 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:58.328 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:58.328 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:23:58.328 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:38.189190+0000", 2024-09-13T11:23:58.328 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.6973999999999999e-05 2024-09-13T11:23:58.328 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:58.328 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:23:58.329 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:58.329 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:58.329 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:58.329 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPush(3.1c 84/77 [PushOp(3:3df7cc9e:::benchmark_data_smithi177_31116_object19949:head, version: 54'1255, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:3df7cc9e:::benchmark_data_smithi177_31116_object19949:head@54'1255, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:23:58.329 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:41.278800+0000", 2024-09-13T11:23:58.329 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 16.987471091, 2024-09-13T11:23:58.329 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.10680685700000001, 2024-09-13T11:23:58.329 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:23:58.329 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:23:58.329 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:23:58.329 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:23:58.329 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:58.330 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:23:58.330 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.278800+0000", 2024-09-13T11:23:58.330 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:58.330 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:58.330 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:58.330 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:23:58.330 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.278800+0000", 2024-09-13T11:23:58.330 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:58.330 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:58.330 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:58.330 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:23:58.330 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.278802+0000", 2024-09-13T11:23:58.331 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.779e-06 2024-09-13T11:23:58.331 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:58.331 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:58.331 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:23:58.331 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.278832+0000", 2024-09-13T11:23:58.331 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.9989999999999999e-05 2024-09-13T11:23:58.331 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:58.331 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:58.331 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:23:58.331 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.384356+0000", 2024-09-13T11:23:58.331 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.105524303 2024-09-13T11:23:58.331 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:58.332 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:58.332 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:23:58.332 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.384376+0000", 2024-09-13T11:23:58.332 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.0049000000000001e-05 2024-09-13T11:23:58.332 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:58.332 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:58.332 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:23:58.332 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.384483+0000", 2024-09-13T11:23:58.332 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000106271 2024-09-13T11:23:58.332 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:58.332 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:58.332 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:23:58.333 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.384498+0000", 2024-09-13T11:23:58.333 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5109000000000001e-05 2024-09-13T11:23:58.333 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:58.333 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:58.333 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:23:58.333 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.385607+0000", 2024-09-13T11:23:58.333 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0011093559999999999 2024-09-13T11:23:58.333 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:58.333 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:23:58.333 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:58.333 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:58.333 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:58.334 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPushReply(3.23 84/78 [PushReplyOp(3:c7e44616:::benchmark_data_smithi177_31116_object32371:head)])", 2024-09-13T11:23:58.334 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:41.338210+0000", 2024-09-13T11:23:58.334 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 16.928060842000001, 2024-09-13T11:23:58.334 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.41118492299999998, 2024-09-13T11:23:58.334 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:23:58.334 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:23:58.334 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-13T11:23:58.334 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:23:58.334 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:58.334 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:23:58.334 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.338210+0000", 2024-09-13T11:23:58.335 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:58.335 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:58.335 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:58.335 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:23:58.335 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.338210+0000", 2024-09-13T11:23:58.335 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:58.335 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:58.335 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:58.335 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:23:58.335 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.338212+0000", 2024-09-13T11:23:58.335 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.314e-06 2024-09-13T11:23:58.335 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:58.336 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:58.336 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:23:58.336 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.338219+0000", 2024-09-13T11:23:58.336 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.6469999999999998e-06 2024-09-13T11:23:58.336 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:58.336 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:58.336 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:23:58.336 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.748675+0000", 2024-09-13T11:23:58.336 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.41045567500000002 2024-09-13T11:23:58.336 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:58.336 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:58.336 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:23:58.337 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.748684+0000", 2024-09-13T11:23:58.337 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.2150000000000007e-06 2024-09-13T11:23:58.337 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:58.337 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:58.337 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:23:58.337 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.748739+0000", 2024-09-13T11:23:58.337 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.4543000000000003e-05 2024-09-13T11:23:58.337 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:58.337 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:58.337 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:23:58.337 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.749395+0000", 2024-09-13T11:23:58.338 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00065652899999999999 2024-09-13T11:23:58.338 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:58.338 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:23:58.338 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:58.338 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:58.338 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:58.338 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPushReply(3.23 84/78 [PushReplyOp(3:c57c3ccc:::benchmark_data_smithi177_31116_object33278:head)])", 2024-09-13T11:23:58.338 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:41.825672+0000", 2024-09-13T11:23:58.338 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 16.440599536000001, 2024-09-13T11:23:58.338 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.11608001499999999, 2024-09-13T11:23:58.338 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:23:58.338 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:23:58.339 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-13T11:23:58.339 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:23:58.339 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:58.339 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:23:58.339 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.825672+0000", 2024-09-13T11:23:58.339 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:58.339 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:58.339 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:58.339 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:23:58.339 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.825672+0000", 2024-09-13T11:23:58.339 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:58.340 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:58.340 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:58.340 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:23:58.340 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.825673+0000", 2024-09-13T11:23:58.340 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0300000000000001e-06 2024-09-13T11:23:58.340 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:58.340 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:58.340 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:23:58.340 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.825680+0000", 2024-09-13T11:23:58.340 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.1520000000000003e-06 2024-09-13T11:23:58.340 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:58.341 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:58.341 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:23:58.341 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.940925+0000", 2024-09-13T11:23:58.341 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.115244816 2024-09-13T11:23:58.341 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:58.341 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:58.341 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:23:58.341 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.940932+0000", 2024-09-13T11:23:58.341 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.7319999999999998e-06 2024-09-13T11:23:58.341 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:58.341 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:58.342 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:23:58.342 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.941218+0000", 2024-09-13T11:23:58.342 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00028532699999999999 2024-09-13T11:23:58.342 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:58.342 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:58.342 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:23:58.342 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.941752+0000", 2024-09-13T11:23:58.342 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00053395799999999996 2024-09-13T11:23:58.342 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:58.342 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:23:58.342 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:58.342 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:58.342 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:58.342 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPushReply(3.23 85/78 [PushReplyOp(3:c5c6dce1:::benchmark_data_smithi177_31116_object37293:head)])", 2024-09-13T11:23:58.343 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:42.377862+0000", 2024-09-13T11:23:58.343 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 15.888409383000001, 2024-09-13T11:23:58.343 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.62538808599999995, 2024-09-13T11:23:58.343 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:23:58.343 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:23:58.343 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-13T11:23:58.343 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:23:58.343 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:58.343 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:23:58.343 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.377862+0000", 2024-09-13T11:23:58.343 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:58.343 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:58.344 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:58.344 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:23:58.344 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.377862+0000", 2024-09-13T11:23:58.344 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:58.344 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:58.344 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:58.344 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:23:58.344 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.377863+0000", 2024-09-13T11:23:58.344 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.6899999999999996e-07 2024-09-13T11:23:58.344 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:58.344 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:58.344 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:23:58.344 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.377866+0000", 2024-09-13T11:23:58.345 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.219e-06 2024-09-13T11:23:58.345 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:58.345 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:58.345 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:23:58.345 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:43.001857+0000", 2024-09-13T11:23:58.345 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.62399084299999996 2024-09-13T11:23:58.345 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:58.345 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:58.345 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:23:58.345 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:43.001869+0000", 2024-09-13T11:23:58.345 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.2415999999999999e-05 2024-09-13T11:23:58.346 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:58.346 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:58.346 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:23:58.346 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:43.002194+0000", 2024-09-13T11:23:58.346 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000324774 2024-09-13T11:23:58.346 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:58.346 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:58.346 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:23:58.346 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:43.003250+0000", 2024-09-13T11:23:58.346 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001055865 2024-09-13T11:23:58.346 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:58.346 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:23:58.347 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:58.347 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:58.347 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:58.347 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPushReply(3.23 86/78 [PushReplyOp(3:c4ca4518:::benchmark_data_smithi177_31116_object39787:head)])", 2024-09-13T11:23:58.347 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:43.580816+0000", 2024-09-13T11:23:58.347 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 14.685455692, 2024-09-13T11:23:58.347 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.14999331199999999, 2024-09-13T11:23:58.347 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:23:58.347 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:23:58.347 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-13T11:23:58.347 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:23:58.347 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:58.348 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:23:58.348 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:43.580816+0000", 2024-09-13T11:23:58.348 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:58.348 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:58.348 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:58.348 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:23:58.348 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:43.580816+0000", 2024-09-13T11:23:58.348 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:58.348 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:58.348 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:58.348 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:23:58.349 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:43.580818+0000", 2024-09-13T11:23:58.349 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.7250000000000002e-06 2024-09-13T11:23:58.349 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:58.349 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:58.349 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:23:58.349 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:43.580830+0000", 2024-09-13T11:23:58.349 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.2247e-05 2024-09-13T11:23:58.349 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:58.349 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:58.349 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:23:58.349 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:43.730204+0000", 2024-09-13T11:23:58.349 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.149373531 2024-09-13T11:23:58.350 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:58.350 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:58.350 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:23:58.350 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:43.730213+0000", 2024-09-13T11:23:58.350 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.6989999999999992e-06 2024-09-13T11:23:58.350 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:58.350 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:58.350 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:23:58.350 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:43.730274+0000", 2024-09-13T11:23:58.350 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.1419000000000003e-05 2024-09-13T11:23:58.350 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:58.351 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:58.351 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:23:58.351 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:43.730809+0000", 2024-09-13T11:23:58.351 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00053469100000000001 2024-09-13T11:23:58.351 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:58.351 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:23:58.351 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:58.351 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:58.351 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:58.351 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPushReply(3.23 86/78 [PushReplyOp(3:c6ae38b7:::benchmark_data_smithi177_31116_object39934:head)])", 2024-09-13T11:23:58.351 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:43.748700+0000", 2024-09-13T11:23:58.352 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 14.517571519000001, 2024-09-13T11:23:58.352 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.11236333, 2024-09-13T11:23:58.352 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:23:58.352 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:23:58.352 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-13T11:23:58.352 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:23:58.352 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:58.352 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:23:58.352 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:43.748700+0000", 2024-09-13T11:23:58.352 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:58.352 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:58.352 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:58.353 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:23:58.353 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:43.748700+0000", 2024-09-13T11:23:58.353 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:58.353 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:58.353 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:58.353 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:23:58.353 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:43.748701+0000", 2024-09-13T11:23:58.353 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.398e-06 2024-09-13T11:23:58.353 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:58.353 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:58.356 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:23:58.356 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:43.748707+0000", 2024-09-13T11:23:58.356 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.993e-06 2024-09-13T11:23:58.356 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:58.356 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:58.356 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:23:58.356 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:43.860552+0000", 2024-09-13T11:23:58.356 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.11184472099999999 2024-09-13T11:23:58.357 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:58.357 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:58.357 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:23:58.357 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:43.860560+0000", 2024-09-13T11:23:58.357 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.5949999999999999e-06 2024-09-13T11:23:58.357 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:58.357 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:58.357 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:23:58.357 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:43.860619+0000", 2024-09-13T11:23:58.357 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.8455999999999997e-05 2024-09-13T11:23:58.357 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:58.357 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:58.358 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:23:58.358 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:43.861063+0000", 2024-09-13T11:23:58.358 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00044416699999999998 2024-09-13T11:23:58.358 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:58.358 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:23:58.358 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:58.358 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:58.358 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:58.358 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPush(3.1c 89/77 [PushOp(3:38bcbc71:::benchmark_data_smithi177_31116_object31448:head, version: 59'1946, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:38bcbc71:::benchmark_data_smithi177_31116_object31448:head@59'1946, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:23:58.358 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:46.670723+0000", 2024-09-13T11:23:58.358 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 11.595548159, 2024-09-13T11:23:58.359 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.27576065, 2024-09-13T11:23:58.359 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:23:58.359 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:23:58.359 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:23:58.359 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:23:58.359 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:58.359 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:23:58.359 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.670723+0000", 2024-09-13T11:23:58.359 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:58.359 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:58.359 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:58.359 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:23:58.360 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.670723+0000", 2024-09-13T11:23:58.360 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:58.360 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:58.360 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:58.360 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:23:58.360 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.670724+0000", 2024-09-13T11:23:58.360 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.4389999999999999e-06 2024-09-13T11:23:58.360 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:58.360 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:58.360 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:23:58.360 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.670754+0000", 2024-09-13T11:23:58.360 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.9224999999999999e-05 2024-09-13T11:23:58.361 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:58.361 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:58.361 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:23:58.361 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.944369+0000", 2024-09-13T11:23:58.361 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.27361529800000001 2024-09-13T11:23:58.361 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:58.361 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:58.361 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:23:58.361 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.944422+0000", 2024-09-13T11:23:58.361 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.3318e-05 2024-09-13T11:23:58.361 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:58.361 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:58.362 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:23:58.362 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.944711+0000", 2024-09-13T11:23:58.362 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00028912399999999998 2024-09-13T11:23:58.362 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:58.362 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:58.362 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:23:58.362 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.944735+0000", 2024-09-13T11:23:58.362 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.3366999999999998e-05 2024-09-13T11:23:58.362 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:58.362 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:58.362 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:23:58.363 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.946484+0000", 2024-09-13T11:23:58.363 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0017488790000000001 2024-09-13T11:23:58.363 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:58.363 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:23:58.363 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:58.363 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:58.363 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:58.363 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPush(3.1c 91/77 [PushOp(3:3e18abc2:::benchmark_data_smithi177_31116_object35874:head, version: 62'2235, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:3e18abc2:::benchmark_data_smithi177_31116_object35874:head@62'2235, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:23:58.363 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:48.350968+0000", 2024-09-13T11:23:58.363 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 9.9153028459999994, 2024-09-13T11:23:58.363 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.19257698000000001, 2024-09-13T11:23:58.363 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:23:58.364 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:23:58.364 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:23:58.364 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:23:58.364 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:58.364 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:23:58.364 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.350968+0000", 2024-09-13T11:23:58.364 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:58.364 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:58.364 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:58.364 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:23:58.364 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.350968+0000", 2024-09-13T11:23:58.365 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:58.365 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:58.365 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:58.365 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:23:58.365 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.350970+0000", 2024-09-13T11:23:58.365 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.796e-06 2024-09-13T11:23:58.365 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:58.365 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:58.365 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:23:58.365 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.351003+0000", 2024-09-13T11:23:58.365 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.2631e-05 2024-09-13T11:23:58.365 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:58.366 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:58.366 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:23:58.366 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.542140+0000", 2024-09-13T11:23:58.366 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.191136737 2024-09-13T11:23:58.366 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:58.366 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:58.366 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:23:58.366 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.542158+0000", 2024-09-13T11:23:58.366 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.8896999999999999e-05 2024-09-13T11:23:58.366 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:58.366 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:58.366 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:23:58.367 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.542263+0000", 2024-09-13T11:23:58.367 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000104158 2024-09-13T11:23:58.367 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:58.367 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:58.367 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:23:58.367 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.542279+0000", 2024-09-13T11:23:58.367 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5926999999999998e-05 2024-09-13T11:23:58.367 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:58.367 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:58.367 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:23:58.367 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.543545+0000", 2024-09-13T11:23:58.367 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0012668340000000001 2024-09-13T11:23:58.368 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:58.368 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:23:58.368 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:58.368 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:58.368 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:58.368 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPush(3.1c 91/77 [PushOp(3:39c44f2b:::benchmark_data_smithi177_31116_object37481:head, version: 63'2324, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:39c44f2b:::benchmark_data_smithi177_31116_object37481:head@63'2324, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:23:58.368 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:48.735009+0000", 2024-09-13T11:23:58.368 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 9.5312620770000009, 2024-09-13T11:23:58.368 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.104813673, 2024-09-13T11:23:58.368 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:23:58.368 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:23:58.368 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:23:58.369 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:23:58.369 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:58.369 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:23:58.369 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.735009+0000", 2024-09-13T11:23:58.369 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:58.369 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:58.369 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:58.369 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:23:58.369 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.735009+0000", 2024-09-13T11:23:58.369 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:58.369 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:58.369 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:58.370 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:23:58.370 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.735010+0000", 2024-09-13T11:23:58.370 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.2589999999999999e-06 2024-09-13T11:23:58.370 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:58.370 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:58.370 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:23:58.370 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.735032+0000", 2024-09-13T11:23:58.370 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.1923999999999999e-05 2024-09-13T11:23:58.370 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:58.370 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:58.370 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:23:58.370 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.838235+0000", 2024-09-13T11:23:58.371 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.10320252000000001 2024-09-13T11:23:58.371 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:58.371 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:58.371 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:23:58.371 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.838252+0000", 2024-09-13T11:23:58.371 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.7088999999999999e-05 2024-09-13T11:23:58.371 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:58.371 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:58.372 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:23:58.372 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.838366+0000", 2024-09-13T11:23:58.372 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000113824 2024-09-13T11:23:58.372 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:58.372 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:58.372 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:23:58.372 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.838381+0000", 2024-09-13T11:23:58.372 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.4839e-05 2024-09-13T11:23:58.372 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:58.372 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:58.373 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:23:58.373 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.839823+0000", 2024-09-13T11:23:58.373 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0014422180000000001 2024-09-13T11:23:58.373 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:58.373 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:23:58.373 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:58.373 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:58.373 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:23:58.373 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-09-13T11:23:58.375 INFO:teuthology.orchestra.run.smithi001.stdout:osd.8: {} 2024-09-13T11:23:58.375 INFO:tasks.ceph.osd.5.smithi092.stderr:2024-09-13T11:23:58.371+0000 7f6f0b705640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26346) UID: 0 2024-09-13T11:23:58.375 INFO:teuthology.orchestra.run.smithi001.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-13T11:23:58.376 INFO:teuthology.orchestra.run.smithi001.stdout:osd.9: {} 2024-09-13T11:23:58.376 INFO:teuthology.orchestra.run.smithi001.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-13T11:23:58.377 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-09-13T11:23:58.377 INFO:teuthology.orchestra.run.smithi001.stdout:osd.10: {} 2024-09-13T11:23:58.377 INFO:teuthology.orchestra.run.smithi001.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-13T11:23:58.378 INFO:teuthology.orchestra.run.smithi001.stdout:osd.11: {} 2024-09-13T11:23:58.378 INFO:teuthology.orchestra.run.smithi001.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-13T11:23:58.405 INFO:teuthology.orchestra.run.smithi001.stdout:osd.12: {} 2024-09-13T11:23:58.405 INFO:teuthology.orchestra.run.smithi001.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-13T11:23:58.435 INFO:teuthology.orchestra.run.smithi001.stdout:osd.13: {} 2024-09-13T11:23:58.435 INFO:teuthology.orchestra.run.smithi001.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-13T11:23:58.461 INFO:teuthology.orchestra.run.smithi001.stdout:osd.14: {} 2024-09-13T11:23:58.461 INFO:teuthology.orchestra.run.smithi001.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-13T11:23:58.475 INFO:tasks.ceph.osd.15.smithi177.stderr:2024-09-13T11:23:58.468+0000 7f9d6ce77640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26578) UID: 0 2024-09-13T11:23:58.489 INFO:teuthology.orchestra.run.smithi001.stdout:osd.15: {} 2024-09-13T11:23:58.489 INFO:teuthology.orchestra.run.smithi001.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-13T11:23:58.546 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-09-13T11:23:58.546 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [ 2024-09-13T11:23:58.546 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:58.546 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:77177 3.1c 3:38495bae:::benchmark_data_smithi177_31116_object77176:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-09-13T11:23:58.546 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:58.533909+0000", 2024-09-13T11:23:58.546 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 0.011509426999999999, 2024-09-13T11:23:58.546 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.011528935000000001, 2024-09-13T11:23:58.546 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:23:58.546 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:23:58.546 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "waiting for sub ops", 2024-09-13T11:23:58.546 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:23:58.546 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:23:58.547 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:58.547 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 77177 2024-09-13T11:23:58.547 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:58.547 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:23:58.547 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:58.547 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:23:58.547 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:58.533909+0000", 2024-09-13T11:23:58.547 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:58.547 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:58.547 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:58.547 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:23:58.547 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:58.533909+0000", 2024-09-13T11:23:58.547 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:58.548 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:58.548 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:58.548 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:23:58.548 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:58.533911+0000", 2024-09-13T11:23:58.548 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.8360000000000001e-06 2024-09-13T11:23:58.548 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:58.548 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:58.548 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:23:58.548 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:58.533934+0000", 2024-09-13T11:23:58.548 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.2969e-05 2024-09-13T11:23:58.548 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:58.548 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:58.549 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:23:58.549 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:58.533943+0000", 2024-09-13T11:23:58.549 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.4010000000000003e-06 2024-09-13T11:23:58.549 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:58.549 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:58.549 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:23:58.549 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:58.533950+0000", 2024-09-13T11:23:58.549 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.1930000000000004e-06 2024-09-13T11:23:58.549 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:58.549 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:58.549 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:23:58.549 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:58.534005+0000", 2024-09-13T11:23:58.550 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.5127999999999998e-05 2024-09-13T11:23:58.550 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:58.550 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:58.550 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:23:58.550 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:58.534105+0000", 2024-09-13T11:23:58.550 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.9974000000000002e-05 2024-09-13T11:23:58.550 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:58.550 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:58.550 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 7", 2024-09-13T11:23:58.550 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:58.534223+0000", 2024-09-13T11:23:58.550 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00011819099999999999 2024-09-13T11:23:58.550 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:58.550 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:58.550 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:23:58.551 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:58.537842+0000", 2024-09-13T11:23:58.551 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.003618553 2024-09-13T11:23:58.551 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:58.551 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:23:58.551 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:58.551 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:58.551 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:58.551 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:77153 3.f 3:f51d3253:::benchmark_data_smithi177_31116_object77152:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-09-13T11:23:58.551 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:58.516249+0000", 2024-09-13T11:23:58.551 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 0.029168981, 2024-09-13T11:23:58.551 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.029248929, 2024-09-13T11:23:58.551 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:23:58.551 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:23:58.552 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "waiting for sub ops", 2024-09-13T11:23:58.552 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:23:58.552 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:23:58.552 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:58.552 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 77153 2024-09-13T11:23:58.552 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:58.552 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:23:58.552 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:58.552 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:23:58.552 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:58.516249+0000", 2024-09-13T11:23:58.552 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:58.552 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:58.553 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:58.553 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:23:58.553 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:58.516249+0000", 2024-09-13T11:23:58.553 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:58.553 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:58.553 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:58.553 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:23:58.553 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:58.516252+0000", 2024-09-13T11:23:58.553 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.3120000000000001e-06 2024-09-13T11:23:58.553 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:58.553 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:58.553 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:23:58.554 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:58.516278+0000", 2024-09-13T11:23:58.554 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.6248e-05 2024-09-13T11:23:58.554 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:58.554 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:58.554 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:23:58.554 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:58.516292+0000", 2024-09-13T11:23:58.554 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.4538e-05 2024-09-13T11:23:58.554 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:58.554 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:58.554 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:23:58.554 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:58.516308+0000", 2024-09-13T11:23:58.554 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5444e-05 2024-09-13T11:23:58.554 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:58.555 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:58.555 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:23:58.555 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:58.516470+0000", 2024-09-13T11:23:58.555 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00016181400000000001 2024-09-13T11:23:58.555 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:58.555 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:58.555 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:23:58.555 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:58.516629+0000", 2024-09-13T11:23:58.555 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000159543 2024-09-13T11:23:58.555 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:58.555 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:58.555 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 12", 2024-09-13T11:23:58.556 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:58.516792+0000", 2024-09-13T11:23:58.556 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000163349 2024-09-13T11:23:58.556 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:58.556 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:58.556 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:23:58.556 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:58.520774+0000", 2024-09-13T11:23:58.556 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0039817790000000004 2024-09-13T11:23:58.556 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:58.556 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:23:58.556 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:58.556 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:58.556 INFO:teuthology.orchestra.run.smithi177.stdout: ], 2024-09-13T11:23:58.557 INFO:teuthology.orchestra.run.smithi177.stdout: "num_ops": 2 2024-09-13T11:23:58.557 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-09-13T11:23:58.567 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-09-13T11:23:58.612 INFO:tasks.ceph.osd.7.smithi177.stderr:2024-09-13T11:23:58.608+0000 7f897b51d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26575) UID: 0 2024-09-13T11:23:58.677 INFO:tasks.ceph.osd.3.smithi177.stderr:2024-09-13T11:23:58.672+0000 7f10adba4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26574) UID: 0 2024-09-13T11:23:58.758 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-09-13T11:23:58.758 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [], 2024-09-13T11:23:58.758 INFO:teuthology.orchestra.run.smithi177.stdout: "complaint_time": 30, 2024-09-13T11:23:58.758 INFO:teuthology.orchestra.run.smithi177.stdout: "num_blocked_ops": 0 2024-09-13T11:23:58.758 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-09-13T11:23:58.778 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-09-13T11:23:58.779 INFO:tasks.ceph.osd.4.smithi001.stderr:2024-09-13T11:23:58.772+0000 7ff8e0218640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26310) UID: 0 2024-09-13T11:23:58.879 INFO:tasks.ceph.osd.7.smithi177.stderr:2024-09-13T11:23:58.872+0000 7f897b51d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26575) UID: 0 2024-09-13T11:23:58.979 INFO:tasks.ceph.osd.1.smithi092.stderr:2024-09-13T11:23:58.975+0000 7fe7789c5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33718) UID: 0 2024-09-13T11:23:58.986 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-09-13T11:23:58.986 INFO:teuthology.orchestra.run.smithi177.stdout: "size": 20, 2024-09-13T11:23:58.986 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 600, 2024-09-13T11:23:58.986 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [ 2024-09-13T11:23:58.986 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:58.986 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:79 3.1 e32/30 3:841df5b5:::benchmark_data_smithi177_31116_object78:head v 32'6, mlcod=32'4)", 2024-09-13T11:23:58.986 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:22:27.057292+0000", 2024-09-13T11:23:58.987 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 91.926687795000007, 2024-09-13T11:23:58.987 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.89296786800000005, 2024-09-13T11:23:58.987 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:23:58.987 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:23:58.987 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:58.987 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:23:58.987 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:58.987 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:23:58.987 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.057292+0000", 2024-09-13T11:23:58.987 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:58.987 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:58.987 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:58.987 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:23:58.988 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.057292+0000", 2024-09-13T11:23:58.988 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:58.988 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:58.988 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:58.988 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:23:58.988 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.057293+0000", 2024-09-13T11:23:58.988 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.6109999999999999e-06 2024-09-13T11:23:58.988 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:58.988 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:58.988 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:23:58.988 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.057330+0000", 2024-09-13T11:23:58.988 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.6721999999999998e-05 2024-09-13T11:23:58.988 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:58.989 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:58.989 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:23:58.989 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.948197+0000", 2024-09-13T11:23:58.989 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.890867042 2024-09-13T11:23:58.989 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:58.989 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:58.989 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:23:58.989 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.948213+0000", 2024-09-13T11:23:58.989 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5826000000000001e-05 2024-09-13T11:23:58.989 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:58.989 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:58.989 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:23:58.990 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.948377+0000", 2024-09-13T11:23:58.990 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00016348600000000001 2024-09-13T11:23:58.990 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:58.990 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:58.990 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:23:58.990 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.948447+0000", 2024-09-13T11:23:58.990 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.0891999999999998e-05 2024-09-13T11:23:58.990 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:58.990 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:58.990 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:23:58.990 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.950150+0000", 2024-09-13T11:23:58.990 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001702329 2024-09-13T11:23:58.990 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:58.990 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:58.991 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:23:58.991 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.950260+0000", 2024-09-13T11:23:58.991 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00010996 2024-09-13T11:23:58.991 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:58.991 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:23:58.991 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:58.991 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:58.991 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:58.991 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:30815 3.f 3:fde17cb1:::benchmark_data_smithi177_31116_object30814:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e59)", 2024-09-13T11:23:58.991 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:09.495035+0000", 2024-09-13T11:23:58.991 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 49.488944629999999, 2024-09-13T11:23:58.991 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.229307711, 2024-09-13T11:23:58.991 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:23:58.992 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:23:58.992 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:58.992 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:23:58.992 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:23:58.992 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:58.992 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 30815 2024-09-13T11:23:58.992 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:58.992 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:23:58.992 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:58.992 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:23:58.992 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.495035+0000", 2024-09-13T11:23:58.992 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:58.993 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:58.993 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:58.993 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:23:58.993 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.495035+0000", 2024-09-13T11:23:58.993 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:58.993 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:58.993 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:58.993 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:23:58.993 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.495037+0000", 2024-09-13T11:23:58.993 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.871e-06 2024-09-13T11:23:58.993 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:58.993 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:58.993 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:23:58.994 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.495062+0000", 2024-09-13T11:23:58.994 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.5164000000000002e-05 2024-09-13T11:23:58.994 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:58.994 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:58.994 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:23:58.994 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.495075+0000", 2024-09-13T11:23:58.994 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3369e-05 2024-09-13T11:23:58.994 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:58.994 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:58.995 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:23:58.995 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.495084+0000", 2024-09-13T11:23:58.995 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.3510000000000002e-06 2024-09-13T11:23:58.995 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:58.995 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:58.995 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:23:58.995 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.495346+0000", 2024-09-13T11:23:58.995 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00026264000000000002 2024-09-13T11:23:58.995 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:58.995 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:58.995 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:23:58.996 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.495523+0000", 2024-09-13T11:23:58.996 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00017640000000000001 2024-09-13T11:23:58.996 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:58.996 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:58.996 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 12", 2024-09-13T11:23:58.996 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.495699+0000", 2024-09-13T11:23:58.996 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00017653600000000001 2024-09-13T11:23:58.996 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:58.996 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:58.996 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:23:58.996 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.497040+0000", 2024-09-13T11:23:58.996 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001340473 2024-09-13T11:23:58.997 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:58.997 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:58.997 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:58.997 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.724049+0000", 2024-09-13T11:23:58.997 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.22700931799999999 2024-09-13T11:23:58.997 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:58.997 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:58.997 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:23:58.997 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.724215+0000", 2024-09-13T11:23:58.997 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000165988 2024-09-13T11:23:58.997 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:58.997 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:58.997 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:23:58.998 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.724343+0000", 2024-09-13T11:23:58.998 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000127601 2024-09-13T11:23:58.998 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:58.998 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:23:58.998 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:58.998 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:58.998 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:58.998 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:30828 3.f 3:f3a9c6a9:::benchmark_data_smithi177_31116_object30827:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e59)", 2024-09-13T11:23:58.998 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:09.503661+0000", 2024-09-13T11:23:58.998 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 49.480318445999998, 2024-09-13T11:23:58.998 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.26477667100000002, 2024-09-13T11:23:58.998 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:23:58.999 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:23:58.999 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:59.000 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:23:59.000 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:23:59.000 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:59.000 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 30828 2024-09-13T11:23:59.000 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.000 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:23:59.000 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.000 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:23:59.000 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.503661+0000", 2024-09-13T11:23:59.000 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:59.001 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.001 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.001 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:23:59.001 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.503661+0000", 2024-09-13T11:23:59.001 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:59.001 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.001 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.001 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:23:59.001 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.503663+0000", 2024-09-13T11:23:59.001 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.968e-06 2024-09-13T11:23:59.001 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.001 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.001 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:23:59.001 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.503687+0000", 2024-09-13T11:23:59.002 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.3983999999999999e-05 2024-09-13T11:23:59.002 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.002 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.002 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:23:59.002 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.503701+0000", 2024-09-13T11:23:59.002 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3416e-05 2024-09-13T11:23:59.002 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.002 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.002 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:23:59.002 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.503708+0000", 2024-09-13T11:23:59.002 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.9530000000000002e-06 2024-09-13T11:23:59.002 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.003 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.003 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:23:59.003 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.503759+0000", 2024-09-13T11:23:59.003 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.1075999999999997e-05 2024-09-13T11:23:59.003 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.003 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.003 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:23:59.003 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.503861+0000", 2024-09-13T11:23:59.003 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000102372 2024-09-13T11:23:59.003 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.003 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.003 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 12", 2024-09-13T11:23:59.004 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.503952+0000", 2024-09-13T11:23:59.004 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.1135000000000001e-05 2024-09-13T11:23:59.004 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.004 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.004 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:23:59.004 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.505340+0000", 2024-09-13T11:23:59.004 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0013877169999999999 2024-09-13T11:23:59.004 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.004 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.004 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:59.004 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.768281+0000", 2024-09-13T11:23:59.004 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.26294092400000002 2024-09-13T11:23:59.004 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.005 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.005 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:23:59.005 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.768369+0000", 2024-09-13T11:23:59.005 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.7818999999999994e-05 2024-09-13T11:23:59.005 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.005 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.005 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:23:59.005 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.768438+0000", 2024-09-13T11:23:59.005 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.9307000000000006e-05 2024-09-13T11:23:59.005 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:59.005 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:23:59.005 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:59.005 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.005 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.006 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:45783 3.16 3.17c10a76 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e69)", 2024-09-13T11:23:59.006 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:20.947774+0000", 2024-09-13T11:23:59.006 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 38.036205543999998, 2024-09-13T11:23:59.006 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.96682922400000004, 2024-09-13T11:23:59.006 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:23:59.006 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:23:59.006 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-13T11:23:59.006 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:23:59.006 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:23:59.006 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:59.006 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 45783 2024-09-13T11:23:59.006 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.006 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:23:59.007 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.007 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:23:59.007 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:20.947774+0000", 2024-09-13T11:23:59.007 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:59.007 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.007 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.007 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:23:59.007 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:20.947774+0000", 2024-09-13T11:23:59.007 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:59.007 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.007 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.007 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:23:59.007 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:20.947777+0000", 2024-09-13T11:23:59.007 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.57e-06 2024-09-13T11:23:59.008 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.008 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.008 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:23:59.008 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:20.947802+0000", 2024-09-13T11:23:59.008 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.5528999999999999e-05 2024-09-13T11:23:59.008 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.008 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.008 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:23:59.008 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:20.947813+0000", 2024-09-13T11:23:59.008 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0473e-05 2024-09-13T11:23:59.008 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.008 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.008 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:23:59.008 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:20.947821+0000", 2024-09-13T11:23:59.009 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.3440000000000001e-06 2024-09-13T11:23:59.009 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.009 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.009 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:23:59.009 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:20.947933+0000", 2024-09-13T11:23:59.009 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00011210200000000001 2024-09-13T11:23:59.009 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.009 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.009 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for peered", 2024-09-13T11:23:59.009 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:20.947942+0000", 2024-09-13T11:23:59.009 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.6610000000000006e-06 2024-09-13T11:23:59.009 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.009 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.010 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:23:59.010 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:21.914579+0000", 2024-09-13T11:23:59.010 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.96663672300000003 2024-09-13T11:23:59.010 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.010 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.010 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:23:59.010 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:21.914603+0000", 2024-09-13T11:23:59.010 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.4822000000000001e-05 2024-09-13T11:23:59.010 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:59.010 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:23:59.010 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:59.010 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.010 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.011 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:54625 3.1f 3:f812cdfb:::benchmark_data_smithi177_31116_object54624:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e75)", 2024-09-13T11:23:59.011 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:30.979327+0000", 2024-09-13T11:23:59.011 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 28.004652754999999, 2024-09-13T11:23:59.011 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.96163670899999998, 2024-09-13T11:23:59.011 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:23:59.011 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:23:59.011 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-13T11:23:59.011 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:23:59.011 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:23:59.011 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:59.011 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 54625 2024-09-13T11:23:59.011 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.011 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:23:59.011 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.012 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:23:59.012 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.979327+0000", 2024-09-13T11:23:59.012 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:59.012 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.012 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.012 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:23:59.012 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.979327+0000", 2024-09-13T11:23:59.012 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:59.012 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.012 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.012 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:23:59.012 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.979329+0000", 2024-09-13T11:23:59.012 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.686e-06 2024-09-13T11:23:59.013 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.013 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.013 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:23:59.013 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.979364+0000", 2024-09-13T11:23:59.013 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.5750000000000002e-05 2024-09-13T11:23:59.013 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.013 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.013 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:23:59.013 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.979408+0000", 2024-09-13T11:23:59.013 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.3795999999999999e-05 2024-09-13T11:23:59.013 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.013 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.014 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:23:59.014 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.979427+0000", 2024-09-13T11:23:59.014 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.9040999999999999e-05 2024-09-13T11:23:59.014 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.014 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.014 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:23:59.014 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.122291+0000", 2024-09-13T11:23:59.014 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.14286373599999999 2024-09-13T11:23:59.014 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.014 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.014 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for peered", 2024-09-13T11:23:59.014 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.122298+0000", 2024-09-13T11:23:59.015 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.3719999999999999e-06 2024-09-13T11:23:59.015 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.015 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.015 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:23:59.015 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.940341+0000", 2024-09-13T11:23:59.015 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.81804249699999998 2024-09-13T11:23:59.015 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.015 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.015 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:23:59.015 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.940964+0000", 2024-09-13T11:23:59.015 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00062283100000000001 2024-09-13T11:23:59.015 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:59.015 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:23:59.016 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:59.016 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.016 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.016 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:54649 3.1c 3:3a457a93:::benchmark_data_smithi177_31116_object54648:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e75)", 2024-09-13T11:23:59.016 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:30.979460+0000", 2024-09-13T11:23:59.016 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 28.004519564999999, 2024-09-13T11:23:59.016 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.95579424000000002, 2024-09-13T11:23:59.016 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:23:59.016 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:23:59.016 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-13T11:23:59.016 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:23:59.016 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:23:59.016 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:59.017 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 54649 2024-09-13T11:23:59.017 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.017 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:23:59.017 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.017 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:23:59.017 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.979460+0000", 2024-09-13T11:23:59.017 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:59.017 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.017 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.017 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:23:59.017 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.979460+0000", 2024-09-13T11:23:59.017 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:59.017 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.017 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.018 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:23:59.018 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.979462+0000", 2024-09-13T11:23:59.018 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5799999999999999e-06 2024-09-13T11:23:59.018 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.018 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.018 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:23:59.018 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.979524+0000", 2024-09-13T11:23:59.018 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.2521999999999995e-05 2024-09-13T11:23:59.018 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.018 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.018 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:23:59.018 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.979561+0000", 2024-09-13T11:23:59.018 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.6349e-05 2024-09-13T11:23:59.019 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.019 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.019 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:23:59.019 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.979571+0000", 2024-09-13T11:23:59.019 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0457e-05 2024-09-13T11:23:59.019 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.019 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.019 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:23:59.019 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.121029+0000", 2024-09-13T11:23:59.019 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.14145766200000001 2024-09-13T11:23:59.019 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.019 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.019 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for peered", 2024-09-13T11:23:59.020 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.121039+0000", 2024-09-13T11:23:59.020 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0084e-05 2024-09-13T11:23:59.020 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.020 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.020 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:23:59.020 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.935157+0000", 2024-09-13T11:23:59.020 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.81411807899999999 2024-09-13T11:23:59.020 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.020 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.020 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:23:59.020 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.935254+0000", 2024-09-13T11:23:59.020 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.7507e-05 2024-09-13T11:23:59.020 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:59.021 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:23:59.021 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:59.021 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.021 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.021 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:54654 3.1c 3:3f0bb31a:::benchmark_data_smithi177_31116_object54653:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e75)", 2024-09-13T11:23:59.021 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:30.979665+0000", 2024-09-13T11:23:59.021 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 28.004314399999998, 2024-09-13T11:23:59.021 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.95623908099999999, 2024-09-13T11:23:59.021 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:23:59.021 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:23:59.021 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-13T11:23:59.021 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:23:59.021 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:23:59.021 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:59.022 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 54654 2024-09-13T11:23:59.022 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.022 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:23:59.022 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.022 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:23:59.022 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.979665+0000", 2024-09-13T11:23:59.022 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:59.022 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.022 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.022 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:23:59.022 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.979665+0000", 2024-09-13T11:23:59.022 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:59.022 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.023 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.023 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:23:59.023 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.979667+0000", 2024-09-13T11:23:59.023 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3540000000000001e-06 2024-09-13T11:23:59.023 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.023 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.023 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:23:59.023 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.979703+0000", 2024-09-13T11:23:59.023 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.6758000000000003e-05 2024-09-13T11:23:59.023 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.023 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.023 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:23:59.023 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.979714+0000", 2024-09-13T11:23:59.023 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0804e-05 2024-09-13T11:23:59.024 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.024 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.024 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:23:59.024 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.979724+0000", 2024-09-13T11:23:59.024 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0040999999999999e-05 2024-09-13T11:23:59.024 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.024 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.024 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:23:59.024 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.121061+0000", 2024-09-13T11:23:59.024 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.141336713 2024-09-13T11:23:59.024 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.024 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.025 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for peered", 2024-09-13T11:23:59.025 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.121067+0000", 2024-09-13T11:23:59.025 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.4150000000000004e-06 2024-09-13T11:23:59.025 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.025 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.025 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:23:59.025 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.935776+0000", 2024-09-13T11:23:59.025 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.81470863100000002 2024-09-13T11:23:59.025 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.025 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.025 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:23:59.025 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.935904+0000", 2024-09-13T11:23:59.025 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00012836499999999999 2024-09-13T11:23:59.026 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:59.026 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:23:59.026 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:59.026 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.026 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.026 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPushReply(3.11 75/70 [PushReplyOp(3:8df10771:::benchmark_data_smithi177_31116_object24734:head)])", 2024-09-13T11:23:59.026 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:31.149798+0000", 2024-09-13T11:23:59.026 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 27.834181390000001, 2024-09-13T11:23:59.026 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.32402702500000002, 2024-09-13T11:23:59.026 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:23:59.026 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:23:59.026 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-13T11:23:59.026 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:23:59.026 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.027 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:23:59.027 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.149798+0000", 2024-09-13T11:23:59.027 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:59.027 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.027 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.027 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:23:59.027 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.149798+0000", 2024-09-13T11:23:59.027 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:59.027 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.027 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.027 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:23:59.027 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.149799+0000", 2024-09-13T11:23:59.027 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.5700000000000001e-07 2024-09-13T11:23:59.028 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.028 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.028 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:23:59.028 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.149803+0000", 2024-09-13T11:23:59.028 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.0040000000000001e-06 2024-09-13T11:23:59.028 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.028 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.028 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:23:59.028 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.472172+0000", 2024-09-13T11:23:59.028 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.32236856699999999 2024-09-13T11:23:59.028 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.028 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.029 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:23:59.029 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.472187+0000", 2024-09-13T11:23:59.029 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5016e-05 2024-09-13T11:23:59.029 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.029 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.029 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:23:59.029 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.472302+0000", 2024-09-13T11:23:59.029 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000115121 2024-09-13T11:23:59.029 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.029 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.029 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:23:59.030 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.473825+0000", 2024-09-13T11:23:59.030 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0015234599999999999 2024-09-13T11:23:59.030 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:59.030 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:23:59.030 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:59.030 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.030 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.030 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:54668 3.1c 3:3d2c5e40:::benchmark_data_smithi177_31116_object54667:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e75)", 2024-09-13T11:23:59.030 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:31.245525+0000", 2024-09-13T11:23:59.031 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 27.738455132999999, 2024-09-13T11:23:59.031 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.69828973000000005, 2024-09-13T11:23:59.031 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:23:59.031 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:23:59.031 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:59.031 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:23:59.031 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:23:59.031 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:59.031 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 54668 2024-09-13T11:23:59.031 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.031 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:23:59.031 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.031 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:23:59.032 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.245525+0000", 2024-09-13T11:23:59.032 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:59.032 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.032 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.032 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:23:59.032 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.245525+0000", 2024-09-13T11:23:59.032 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:59.032 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.032 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.032 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:23:59.032 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.245526+0000", 2024-09-13T11:23:59.032 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.719e-06 2024-09-13T11:23:59.032 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.033 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.033 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:23:59.033 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.245547+0000", 2024-09-13T11:23:59.033 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.1084999999999999e-05 2024-09-13T11:23:59.033 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.033 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.033 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:23:59.033 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.245558+0000", 2024-09-13T11:23:59.033 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0934000000000001e-05 2024-09-13T11:23:59.033 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.033 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.033 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:23:59.033 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.245564+0000", 2024-09-13T11:23:59.034 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.6269999999999997e-06 2024-09-13T11:23:59.034 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.034 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.034 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:23:59.034 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.245620+0000", 2024-09-13T11:23:59.034 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.6289999999999998e-05 2024-09-13T11:23:59.034 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.034 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.034 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for peered", 2024-09-13T11:23:59.034 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.245632+0000", 2024-09-13T11:23:59.034 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.1748e-05 2024-09-13T11:23:59.034 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.034 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.035 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:23:59.035 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.935976+0000", 2024-09-13T11:23:59.035 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.69034455299999997 2024-09-13T11:23:59.035 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.035 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.035 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:23:59.035 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.936302+0000", 2024-09-13T11:23:59.035 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000325397 2024-09-13T11:23:59.035 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.035 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.035 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 13", 2024-09-13T11:23:59.035 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.936459+0000", 2024-09-13T11:23:59.035 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00015754300000000001 2024-09-13T11:23:59.036 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.036 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.036 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:23:59.036 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.938813+0000", 2024-09-13T11:23:59.036 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0023530999999999999 2024-09-13T11:23:59.036 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.036 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.036 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:59.036 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.943695+0000", 2024-09-13T11:23:59.036 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.004882506 2024-09-13T11:23:59.036 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.036 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.036 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:23:59.037 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.943767+0000", 2024-09-13T11:23:59.037 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.2454999999999997e-05 2024-09-13T11:23:59.037 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.037 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.037 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:23:59.037 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.943814+0000", 2024-09-13T11:23:59.037 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.6773000000000001e-05 2024-09-13T11:23:59.037 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:59.037 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:23:59.037 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:59.037 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.037 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.037 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPush(3.16 84/70 [PushOp(3:6f7be541:::benchmark_data_smithi177_31116_object43332:head, version: 66'2788, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:6f7be541:::benchmark_data_smithi177_31116_object43332:head@66'2788, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:23:59.038 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:41.207515+0000", 2024-09-13T11:23:59.038 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 17.776464361999999, 2024-09-13T11:23:59.038 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.52643771500000003, 2024-09-13T11:23:59.038 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:23:59.038 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:23:59.038 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:23:59.038 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:23:59.038 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.038 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:23:59.038 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.207515+0000", 2024-09-13T11:23:59.038 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:59.038 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.038 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.039 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:23:59.039 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.207515+0000", 2024-09-13T11:23:59.039 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:59.039 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.039 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.039 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:23:59.039 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.207516+0000", 2024-09-13T11:23:59.039 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.096e-06 2024-09-13T11:23:59.039 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.039 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.039 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:23:59.039 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.207717+0000", 2024-09-13T11:23:59.039 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000200199 2024-09-13T11:23:59.039 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.040 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.040 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:23:59.040 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.732785+0000", 2024-09-13T11:23:59.040 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.52506869599999995 2024-09-13T11:23:59.040 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.040 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.040 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:23:59.040 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.732801+0000", 2024-09-13T11:23:59.040 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5305999999999999e-05 2024-09-13T11:23:59.040 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.040 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.040 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:23:59.041 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.732889+0000", 2024-09-13T11:23:59.041 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.8157000000000003e-05 2024-09-13T11:23:59.041 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.041 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.041 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:23:59.041 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.732901+0000", 2024-09-13T11:23:59.041 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.1866e-05 2024-09-13T11:23:59.041 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.041 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.041 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:23:59.042 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.733953+0000", 2024-09-13T11:23:59.042 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001052395 2024-09-13T11:23:59.042 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:59.042 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:23:59.042 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:59.042 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.042 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.042 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPushReply(3.1f 85/78 [PushReplyOp(3:fd221d21:::benchmark_data_smithi177_31116_object7639:head)])", 2024-09-13T11:23:59.042 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:42.198715+0000", 2024-09-13T11:23:59.042 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 16.785264793, 2024-09-13T11:23:59.042 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.48930391099999998, 2024-09-13T11:23:59.042 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:23:59.042 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:23:59.042 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-13T11:23:59.042 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:23:59.043 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.043 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:23:59.043 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.198715+0000", 2024-09-13T11:23:59.043 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:59.043 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.043 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.043 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:23:59.043 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.198715+0000", 2024-09-13T11:23:59.043 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:59.043 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.043 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.043 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:23:59.043 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.198716+0000", 2024-09-13T11:23:59.044 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.3500000000000005e-07 2024-09-13T11:23:59.044 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.044 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.044 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:23:59.044 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.198719+0000", 2024-09-13T11:23:59.044 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.4539999999999999e-06 2024-09-13T11:23:59.044 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.044 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.044 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:23:59.044 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.684732+0000", 2024-09-13T11:23:59.044 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.48601262099999998 2024-09-13T11:23:59.044 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.044 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.044 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:23:59.045 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.684746+0000", 2024-09-13T11:23:59.045 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.4518999999999999e-05 2024-09-13T11:23:59.045 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.045 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.045 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:23:59.045 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.684911+0000", 2024-09-13T11:23:59.045 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000164728 2024-09-13T11:23:59.045 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.045 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.045 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:23:59.045 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.688019+0000", 2024-09-13T11:23:59.045 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0031077539999999999 2024-09-13T11:23:59.045 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:59.045 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:23:59.046 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:59.046 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.046 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.046 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPushReply(3.1c 85/77 [PushReplyOp(3:3a669437:::benchmark_data_smithi177_31116_object25280:head)])", 2024-09-13T11:23:59.046 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:42.352466+0000", 2024-09-13T11:23:59.046 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 16.631513899000002, 2024-09-13T11:23:59.046 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.36670725300000001, 2024-09-13T11:23:59.046 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:23:59.046 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:23:59.046 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-13T11:23:59.046 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:23:59.046 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.046 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:23:59.047 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.352466+0000", 2024-09-13T11:23:59.047 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:59.047 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.047 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.047 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:23:59.047 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.352466+0000", 2024-09-13T11:23:59.047 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:59.047 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.047 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.047 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:23:59.047 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.352467+0000", 2024-09-13T11:23:59.047 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.1429999999999999e-06 2024-09-13T11:23:59.047 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.047 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.048 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:23:59.048 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.352474+0000", 2024-09-13T11:23:59.048 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.6100000000000002e-06 2024-09-13T11:23:59.048 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.048 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.048 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:23:59.048 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.717011+0000", 2024-09-13T11:23:59.048 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.36453790200000002 2024-09-13T11:23:59.048 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.048 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.048 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:23:59.048 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.717032+0000", 2024-09-13T11:23:59.048 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.0999e-05 2024-09-13T11:23:59.049 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.049 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.049 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:23:59.049 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.717162+0000", 2024-09-13T11:23:59.049 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000129159 2024-09-13T11:23:59.049 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.049 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.049 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:23:59.049 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.719173+0000", 2024-09-13T11:23:59.049 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0020114400000000002 2024-09-13T11:23:59.049 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:59.049 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:23:59.049 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:59.049 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.050 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.050 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPush(3.16 85/70 [PushOp(3:6aef2f27:::benchmark_data_smithi177_31116_object44090:head, version: 68'2844, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:6aef2f27:::benchmark_data_smithi177_31116_object44090:head@68'2844, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:23:59.050 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:42.574221+0000", 2024-09-13T11:23:59.050 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 16.409758791000002, 2024-09-13T11:23:59.050 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.25345585700000001, 2024-09-13T11:23:59.050 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:23:59.050 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:23:59.050 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:23:59.050 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:23:59.050 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.050 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:23:59.050 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.574221+0000", 2024-09-13T11:23:59.050 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:59.051 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.051 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.051 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:23:59.051 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.574221+0000", 2024-09-13T11:23:59.051 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:59.051 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.051 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.051 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:23:59.051 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.574252+0000", 2024-09-13T11:23:59.051 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.0973000000000003e-05 2024-09-13T11:23:59.051 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.051 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.051 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:23:59.051 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.574434+0000", 2024-09-13T11:23:59.052 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000182265 2024-09-13T11:23:59.052 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.052 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.052 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:23:59.052 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.825391+0000", 2024-09-13T11:23:59.052 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.250957023 2024-09-13T11:23:59.052 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.052 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.052 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:23:59.052 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.825442+0000", 2024-09-13T11:23:59.052 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.1267999999999998e-05 2024-09-13T11:23:59.052 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.052 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.053 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:23:59.053 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.825692+0000", 2024-09-13T11:23:59.053 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00024924700000000002 2024-09-13T11:23:59.053 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.053 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.053 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:23:59.053 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.825715+0000", 2024-09-13T11:23:59.053 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.3176e-05 2024-09-13T11:23:59.053 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.053 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.053 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:23:59.053 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.827677+0000", 2024-09-13T11:23:59.053 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0019619049999999999 2024-09-13T11:23:59.053 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:59.054 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:23:59.054 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:59.054 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.054 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.054 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPushReply(3.1f 85/78 [PushReplyOp(3:fecb0d6a:::benchmark_data_smithi177_31116_object7644:head)])", 2024-09-13T11:23:59.054 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:42.705351+0000", 2024-09-13T11:23:59.054 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 16.278629057, 2024-09-13T11:23:59.054 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.290549681, 2024-09-13T11:23:59.054 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:23:59.054 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:23:59.054 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-13T11:23:59.054 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:23:59.054 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.054 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:23:59.055 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.705351+0000", 2024-09-13T11:23:59.055 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:59.055 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.055 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.055 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:23:59.055 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.705351+0000", 2024-09-13T11:23:59.055 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:59.055 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.055 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.055 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:23:59.055 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.705353+0000", 2024-09-13T11:23:59.055 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.4169999999999999e-06 2024-09-13T11:23:59.055 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.055 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.056 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:23:59.056 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.705365+0000", 2024-09-13T11:23:59.056 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.1894e-05 2024-09-13T11:23:59.056 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.056 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.056 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:23:59.056 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.992081+0000", 2024-09-13T11:23:59.056 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.28671641199999998 2024-09-13T11:23:59.056 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.056 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.056 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:23:59.056 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.992111+0000", 2024-09-13T11:23:59.056 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.9427000000000001e-05 2024-09-13T11:23:59.057 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.057 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.057 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:23:59.057 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.993573+0000", 2024-09-13T11:23:59.057 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0014623590000000001 2024-09-13T11:23:59.057 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.057 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.057 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:23:59.057 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.995900+0000", 2024-09-13T11:23:59.057 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0023271720000000002 2024-09-13T11:23:59.057 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:59.057 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:23:59.057 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:59.058 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.058 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.058 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPushReply(3.21 89/78 [PushReplyOp(3:8534d087:::benchmark_data_smithi177_31116_object5474:head)])", 2024-09-13T11:23:59.058 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:46.632834+0000", 2024-09-13T11:23:59.058 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 12.351145617, 2024-09-13T11:23:59.058 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.324805556, 2024-09-13T11:23:59.058 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:23:59.058 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:23:59.058 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-13T11:23:59.058 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:23:59.058 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.058 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:23:59.058 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.632834+0000", 2024-09-13T11:23:59.059 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:59.059 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.059 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.059 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:23:59.059 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.632834+0000", 2024-09-13T11:23:59.059 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:59.059 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.059 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.059 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:23:59.059 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.632835+0000", 2024-09-13T11:23:59.059 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.4270000000000001e-06 2024-09-13T11:23:59.059 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.059 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.059 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:23:59.060 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.632840+0000", 2024-09-13T11:23:59.060 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.7809999999999996e-06 2024-09-13T11:23:59.060 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.060 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.060 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:23:59.060 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.956282+0000", 2024-09-13T11:23:59.060 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.323441639 2024-09-13T11:23:59.060 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.060 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.060 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:23:59.060 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.956293+0000", 2024-09-13T11:23:59.060 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.083e-05 2024-09-13T11:23:59.060 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.061 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.061 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:23:59.061 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.956395+0000", 2024-09-13T11:23:59.061 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00010184200000000001 2024-09-13T11:23:59.061 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.061 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.061 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:23:59.061 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.957640+0000", 2024-09-13T11:23:59.061 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0012450370000000001 2024-09-13T11:23:59.061 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:59.061 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:23:59.061 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:59.061 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.062 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.062 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPushReply(3.1f 89/78 [PushReplyOp(3:ff150976:::benchmark_data_smithi177_31116_object12907:head)])", 2024-09-13T11:23:59.062 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:46.697504+0000", 2024-09-13T11:23:59.062 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 12.286476018, 2024-09-13T11:23:59.062 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.26836818099999998, 2024-09-13T11:23:59.062 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:23:59.062 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:23:59.062 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-13T11:23:59.062 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:23:59.062 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.062 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:23:59.062 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.697504+0000", 2024-09-13T11:23:59.062 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:59.062 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.063 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.063 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:23:59.063 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.697504+0000", 2024-09-13T11:23:59.063 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:59.063 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.063 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.063 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:23:59.063 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.697505+0000", 2024-09-13T11:23:59.063 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0529999999999999e-06 2024-09-13T11:23:59.063 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.063 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.063 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:23:59.063 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.697509+0000", 2024-09-13T11:23:59.063 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.9249999999999997e-06 2024-09-13T11:23:59.064 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.064 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.064 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:23:59.064 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.964086+0000", 2024-09-13T11:23:59.064 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.26657776100000002 2024-09-13T11:23:59.064 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.064 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.064 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:23:59.064 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.964110+0000", 2024-09-13T11:23:59.064 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.3189000000000001e-05 2024-09-13T11:23:59.064 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.064 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.064 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:23:59.065 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.964354+0000", 2024-09-13T11:23:59.065 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00024481399999999999 2024-09-13T11:23:59.065 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.065 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.065 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:23:59.065 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.965872+0000", 2024-09-13T11:23:59.065 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001517439 2024-09-13T11:23:59.065 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:59.065 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:23:59.065 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:59.065 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.065 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.065 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:71584 3.21 3:8704127d:::benchmark_data_smithi177_31116_object71583:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96)", 2024-09-13T11:23:59.066 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:53.874886+0000", 2024-09-13T11:23:59.066 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 5.1090935970000002, 2024-09-13T11:23:59.066 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.221088965, 2024-09-13T11:23:59.066 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:23:59.066 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:23:59.066 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:59.066 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:23:59.066 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:23:59.066 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:59.066 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 71584 2024-09-13T11:23:59.066 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.066 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:23:59.066 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.066 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:23:59.067 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:53.874886+0000", 2024-09-13T11:23:59.067 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:59.067 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.067 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.067 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:23:59.067 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:53.874886+0000", 2024-09-13T11:23:59.067 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:59.067 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.067 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.067 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:23:59.067 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:53.874887+0000", 2024-09-13T11:23:59.067 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.3900000000000003e-07 2024-09-13T11:23:59.067 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.068 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.068 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:23:59.068 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:53.874913+0000", 2024-09-13T11:23:59.068 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.5987999999999999e-05 2024-09-13T11:23:59.068 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.068 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.068 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:23:59.068 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:53.874921+0000", 2024-09-13T11:23:59.068 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.7419999999999997e-06 2024-09-13T11:23:59.068 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.068 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.068 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:23:59.068 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:53.874929+0000", 2024-09-13T11:23:59.069 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.8599999999999993e-06 2024-09-13T11:23:59.069 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.069 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.069 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:23:59.069 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:53.874986+0000", 2024-09-13T11:23:59.069 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.7234000000000001e-05 2024-09-13T11:23:59.069 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.069 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.069 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:23:59.069 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:53.875149+0000", 2024-09-13T11:23:59.069 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00016321900000000001 2024-09-13T11:23:59.069 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.069 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.070 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 0,8", 2024-09-13T11:23:59.070 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:53.875337+0000", 2024-09-13T11:23:59.070 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00018800300000000001 2024-09-13T11:23:59.070 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.070 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.070 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:23:59.070 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:53.877955+0000", 2024-09-13T11:23:59.070 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00261794 2024-09-13T11:23:59.070 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.070 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.070 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:59.070 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:53.878422+0000", 2024-09-13T11:23:59.070 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00046655699999999999 2024-09-13T11:23:59.071 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.071 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.071 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:59.071 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.095663+0000", 2024-09-13T11:23:59.071 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.21724190099999999 2024-09-13T11:23:59.071 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.071 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.071 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:23:59.071 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.095702+0000", 2024-09-13T11:23:59.071 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.8438e-05 2024-09-13T11:23:59.071 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.071 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.071 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:23:59.071 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.095975+0000", 2024-09-13T11:23:59.072 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00027314400000000001 2024-09-13T11:23:59.072 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:59.072 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:23:59.072 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:59.072 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.072 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.072 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:71589 3.21 3:85f71b95:::benchmark_data_smithi177_31116_object71588:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96)", 2024-09-13T11:23:59.072 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:53.876584+0000", 2024-09-13T11:23:59.072 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 5.107395565, 2024-09-13T11:23:59.072 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.24303730800000001, 2024-09-13T11:23:59.072 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:23:59.072 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:23:59.073 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:59.073 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:23:59.073 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:23:59.073 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:59.073 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 71589 2024-09-13T11:23:59.073 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.073 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:23:59.073 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.073 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:23:59.074 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:53.876584+0000", 2024-09-13T11:23:59.074 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:59.074 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.074 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.074 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:23:59.074 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:53.876584+0000", 2024-09-13T11:23:59.074 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:59.074 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.074 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.074 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:23:59.074 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:53.876586+0000", 2024-09-13T11:23:59.074 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.4100000000000001e-06 2024-09-13T11:23:59.074 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.075 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.075 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:23:59.075 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:53.876604+0000", 2024-09-13T11:23:59.075 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.8389e-05 2024-09-13T11:23:59.075 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.075 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.075 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:23:59.075 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:53.876613+0000", 2024-09-13T11:23:59.075 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.8789999999999992e-06 2024-09-13T11:23:59.075 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.075 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.075 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:23:59.075 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:53.876619+0000", 2024-09-13T11:23:59.076 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.9699999999999996e-06 2024-09-13T11:23:59.076 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.076 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.076 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:23:59.076 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:53.876681+0000", 2024-09-13T11:23:59.076 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.2699000000000004e-05 2024-09-13T11:23:59.076 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.076 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.076 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:23:59.076 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:53.876849+0000", 2024-09-13T11:23:59.076 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00016714799999999999 2024-09-13T11:23:59.076 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.076 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.077 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 0,8", 2024-09-13T11:23:59.077 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:53.877032+0000", 2024-09-13T11:23:59.077 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00018387199999999999 2024-09-13T11:23:59.077 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.077 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.077 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:23:59.077 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:53.879473+0000", 2024-09-13T11:23:59.077 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0024403789999999999 2024-09-13T11:23:59.077 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.077 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.077 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:59.078 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:53.880046+0000", 2024-09-13T11:23:59.078 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00057281200000000002 2024-09-13T11:23:59.078 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.078 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.078 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:59.078 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.119299+0000", 2024-09-13T11:23:59.078 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.23925310399999999 2024-09-13T11:23:59.078 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.078 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.078 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:23:59.078 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.119337+0000", 2024-09-13T11:23:59.078 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.8109999999999999e-05 2024-09-13T11:23:59.078 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.078 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.079 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:23:59.079 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.119621+0000", 2024-09-13T11:23:59.079 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000284536 2024-09-13T11:23:59.079 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:59.079 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:23:59.079 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:59.079 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.079 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.079 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop_reply(client.4665.0:71584 3.21 e96/78 ondisk, result = 0)", 2024-09-13T11:23:59.079 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:53.880127+0000", 2024-09-13T11:23:59.079 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 5.1038525979999996, 2024-09-13T11:23:59.079 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.21585246699999999, 2024-09-13T11:23:59.079 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:23:59.080 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:23:59.080 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:23:59.080 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:23:59.080 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.080 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:23:59.080 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:53.880127+0000", 2024-09-13T11:23:59.080 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:59.080 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.080 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.080 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:23:59.080 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:53.880127+0000", 2024-09-13T11:23:59.080 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:59.080 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.081 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.081 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:23:59.081 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:53.880129+0000", 2024-09-13T11:23:59.081 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.573e-06 2024-09-13T11:23:59.081 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.081 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.081 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:23:59.081 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:53.880136+0000", 2024-09-13T11:23:59.081 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.746e-06 2024-09-13T11:23:59.081 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.081 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.081 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:23:59.081 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.095614+0000", 2024-09-13T11:23:59.081 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.215477631 2024-09-13T11:23:59.082 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.082 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.082 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:23:59.082 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.095619+0000", 2024-09-13T11:23:59.082 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.9300000000000002e-06 2024-09-13T11:23:59.082 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.082 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.082 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:23:59.082 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.095654+0000", 2024-09-13T11:23:59.082 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.523e-05 2024-09-13T11:23:59.082 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.082 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.082 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:23:59.083 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.095660+0000", 2024-09-13T11:23:59.083 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.3290000000000001e-06 2024-09-13T11:23:59.083 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.083 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.083 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:23:59.083 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.095980+0000", 2024-09-13T11:23:59.083 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00032002799999999998 2024-09-13T11:23:59.083 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:59.083 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:23:59.083 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:59.083 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.083 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.083 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop_reply(client.4665.0:71589 3.21 e96/78 ondisk, result = 0)", 2024-09-13T11:23:59.084 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:53.887108+0000", 2024-09-13T11:23:59.084 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 5.0968719650000001, 2024-09-13T11:23:59.084 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.232519532, 2024-09-13T11:23:59.084 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:23:59.084 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:23:59.084 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:23:59.084 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:23:59.084 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.084 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:23:59.084 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:53.887108+0000", 2024-09-13T11:23:59.084 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:59.084 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.084 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.084 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:23:59.085 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:53.887108+0000", 2024-09-13T11:23:59.085 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:59.085 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.085 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.085 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:23:59.085 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:53.887109+0000", 2024-09-13T11:23:59.085 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3370000000000001e-06 2024-09-13T11:23:59.085 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.085 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.085 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:23:59.085 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:53.887113+0000", 2024-09-13T11:23:59.085 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.4630000000000001e-06 2024-09-13T11:23:59.085 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.085 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.086 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:23:59.086 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.119248+0000", 2024-09-13T11:23:59.086 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.232135434 2024-09-13T11:23:59.086 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.086 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.086 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:23:59.086 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.119252+0000", 2024-09-13T11:23:59.086 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.1930000000000003e-06 2024-09-13T11:23:59.086 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.086 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.086 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:23:59.086 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.119289+0000", 2024-09-13T11:23:59.086 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.701e-05 2024-09-13T11:23:59.087 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.087 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.087 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:23:59.087 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.119295+0000", 2024-09-13T11:23:59.087 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.7250000000000002e-06 2024-09-13T11:23:59.087 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.087 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.087 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:23:59.087 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.119627+0000", 2024-09-13T11:23:59.087 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00033237000000000002 2024-09-13T11:23:59.087 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:59.087 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:23:59.087 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:59.088 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:59.088 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:23:59.088 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-09-13T11:23:59.089 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_ops_in_flight 2024-09-13T11:23:59.090 INFO:tasks.radosbench.radosbench.0.smithi177.stdout: 92 16 77943 77927 52.9341 86.4375 0.00474353 0.0188836 2024-09-13T11:23:59.090 INFO:tasks.ceph.osd.10.smithi162.stderr:2024-09-13T11:23:59.085+0000 7fb59c9f0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26317) UID: 0 2024-09-13T11:23:59.190 INFO:tasks.ceph.osd.10.smithi162.stderr:2024-09-13T11:23:59.189+0000 7fb59c9f0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26317) UID: 0 2024-09-13T11:23:59.225 INFO:teuthology.orchestra.run.smithi001.stderr:nodeep-scrub is unset 2024-09-13T11:23:59.226 INFO:tasks.ceph.osd.5.smithi092.stderr:2024-09-13T11:23:59.223+0000 7f6f02136640 -1 osd.5 100 handle_osd_map injecting map crc failure 2024-09-13T11:23:59.239 INFO:tasks.ceph.osd.3.smithi177.stderr:2024-09-13T11:23:59.232+0000 7f10a45d5640 -1 osd.3 100 handle_osd_map injecting map crc failure 2024-09-13T11:23:59.241 INFO:tasks.ceph.osd.14.smithi162.stderr:2024-09-13T11:23:59.237+0000 7f2352b0c640 -1 osd.14 100 handle_osd_map injecting map crc failure 2024-09-13T11:23:59.273 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-09-13T11:23:59.273 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [ 2024-09-13T11:23:59.273 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.273 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:78227 3.16 e101/88 3:68f06b4f:::benchmark_data_smithi177_31116_object78226:head v 101'3969, mlcod=101'3966)", 2024-09-13T11:23:59.273 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:59.270322+0000", 2024-09-13T11:23:59.273 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 0.0022250640000000001, 2024-09-13T11:23:59.273 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0022439999999999999, 2024-09-13T11:23:59.273 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:23:59.274 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:23:59.274 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:23:59.274 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:23:59.274 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.274 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:23:59.274 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:59.270322+0000", 2024-09-13T11:23:59.274 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:59.274 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.274 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.274 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:23:59.274 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:59.270322+0000", 2024-09-13T11:23:59.274 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:59.274 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.275 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.275 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:23:59.275 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:59.270324+0000", 2024-09-13T11:23:59.275 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5039999999999999e-06 2024-09-13T11:23:59.275 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.275 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.275 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:23:59.275 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:59.270392+0000", 2024-09-13T11:23:59.275 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.8659999999999997e-05 2024-09-13T11:23:59.275 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.275 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.275 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:23:59.275 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:59.270425+0000", 2024-09-13T11:23:59.275 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.2030000000000003e-05 2024-09-13T11:23:59.276 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.276 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.276 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:23:59.276 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:59.270431+0000", 2024-09-13T11:23:59.276 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.1149999999999999e-06 2024-09-13T11:23:59.276 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.276 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.276 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:23:59.276 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:59.270474+0000", 2024-09-13T11:23:59.276 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.3232999999999998e-05 2024-09-13T11:23:59.276 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.276 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.276 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:23:59.276 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:59.270492+0000", 2024-09-13T11:23:59.277 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.7880000000000002e-05 2024-09-13T11:23:59.277 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:59.277 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:23:59.277 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:59.277 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:59.277 INFO:teuthology.orchestra.run.smithi177.stdout: ], 2024-09-13T11:23:59.277 INFO:teuthology.orchestra.run.smithi177.stdout: "num_ops": 1 2024-09-13T11:23:59.277 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-09-13T11:23:59.291 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-09-13T11:23:59.289+0000 7f97befc9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26315) UID: 0 2024-09-13T11:23:59.294 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_blocked_ops 2024-09-13T11:23:59.394 INFO:tasks.ceph.osd.3.smithi177.stderr:2024-09-13T11:23:59.388+0000 7f10adba4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26574) UID: 0 2024-09-13T11:23:59.450 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-09-13T11:23:59.450 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [], 2024-09-13T11:23:59.450 INFO:teuthology.orchestra.run.smithi177.stdout: "complaint_time": 30, 2024-09-13T11:23:59.450 INFO:teuthology.orchestra.run.smithi177.stdout: "num_blocked_ops": 0 2024-09-13T11:23:59.450 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-09-13T11:23:59.470 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_historic_ops 2024-09-13T11:23:59.493 INFO:tasks.ceph.osd.4.smithi001.stderr:2024-09-13T11:23:59.488+0000 7ff8e0218640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26310) UID: 0 2024-09-13T11:23:59.594 INFO:tasks.ceph.osd.9.smithi092.stderr:2024-09-13T11:23:59.591+0000 7ff5ab3d0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26348) UID: 0 2024-09-13T11:23:59.668 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-09-13T11:23:59.669 INFO:teuthology.orchestra.run.smithi177.stdout: "size": 20, 2024-09-13T11:23:59.669 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 600, 2024-09-13T11:23:59.669 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [ 2024-09-13T11:23:59.669 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.669 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:25383 3.0 e55/30 3:0a6bb869:::benchmark_data_smithi177_31116_object25382:head v 55'1620, mlcod=55'1617)", 2024-09-13T11:23:59.669 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:04.449956+0000", 2024-09-13T11:23:59.669 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 55.216763559, 2024-09-13T11:23:59.669 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.21143479600000001, 2024-09-13T11:23:59.669 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:23:59.669 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:23:59.669 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:59.669 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:23:59.670 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.670 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:23:59.670 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.449956+0000", 2024-09-13T11:23:59.670 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:59.670 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.670 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.670 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:23:59.670 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.449956+0000", 2024-09-13T11:23:59.670 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:59.670 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.670 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.670 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:23:59.670 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.449959+0000", 2024-09-13T11:23:59.671 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.393e-06 2024-09-13T11:23:59.671 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.671 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.671 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:23:59.671 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.450014+0000", 2024-09-13T11:23:59.671 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.4200000000000003e-05 2024-09-13T11:23:59.671 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.671 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.671 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:23:59.671 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.658637+0000", 2024-09-13T11:23:59.671 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.20862334799999999 2024-09-13T11:23:59.671 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.671 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.672 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:23:59.672 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.658650+0000", 2024-09-13T11:23:59.672 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3148e-05 2024-09-13T11:23:59.672 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.672 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.672 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:23:59.672 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.658867+0000", 2024-09-13T11:23:59.672 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00021673499999999999 2024-09-13T11:23:59.672 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.672 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.672 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:23:59.672 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.658931+0000", 2024-09-13T11:23:59.672 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.4045000000000001e-05 2024-09-13T11:23:59.672 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.673 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.673 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:23:59.673 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.661317+0000", 2024-09-13T11:23:59.673 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.002385823 2024-09-13T11:23:59.673 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.673 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.673 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:23:59.673 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.661391+0000", 2024-09-13T11:23:59.673 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.4103999999999994e-05 2024-09-13T11:23:59.673 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:59.673 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:23:59.673 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:59.673 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.674 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.674 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPush(3.1d 79/78 [PushOp(3:bbdea640:::benchmark_data_smithi177_31116_object1641:head, version: 43'98, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:bbdea640:::benchmark_data_smithi177_31116_object1641:head@43'98, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:23:59.674 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:36.123258+0000", 2024-09-13T11:23:59.674 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 23.543462112, 2024-09-13T11:23:59.674 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.107976952, 2024-09-13T11:23:59.674 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:23:59.674 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:23:59.674 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:23:59.674 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:23:59.674 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.674 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:23:59.674 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:36.123258+0000", 2024-09-13T11:23:59.674 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:59.674 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.675 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.675 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:23:59.675 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:36.123258+0000", 2024-09-13T11:23:59.675 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:59.675 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.675 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.675 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:23:59.675 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:36.123259+0000", 2024-09-13T11:23:59.675 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.7099999999999999e-06 2024-09-13T11:23:59.675 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.675 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.675 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:23:59.675 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:36.123316+0000", 2024-09-13T11:23:59.675 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.6694999999999998e-05 2024-09-13T11:23:59.676 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.676 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.676 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:23:59.676 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:36.230197+0000", 2024-09-13T11:23:59.676 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.10688091600000001 2024-09-13T11:23:59.676 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.676 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.676 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:23:59.676 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:36.230214+0000", 2024-09-13T11:23:59.676 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.7193999999999998e-05 2024-09-13T11:23:59.676 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.676 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.676 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:23:59.676 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:36.230330+0000", 2024-09-13T11:23:59.677 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00011553999999999999 2024-09-13T11:23:59.677 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.677 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.677 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:23:59.677 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:36.230345+0000", 2024-09-13T11:23:59.677 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5713999999999999e-05 2024-09-13T11:23:59.677 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.677 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.677 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:23:59.677 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:36.231234+0000", 2024-09-13T11:23:59.677 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00088918299999999999 2024-09-13T11:23:59.677 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:59.677 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:23:59.677 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:59.678 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.678 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.678 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPush(3.1d 79/78 [PushOp(3:bc13ccaa:::benchmark_data_smithi177_31116_object1745:head, version: 43'107, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:bc13ccaa:::benchmark_data_smithi177_31116_object1745:head@43'107, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:23:59.678 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:36.427991+0000", 2024-09-13T11:23:59.678 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 23.238728654999999, 2024-09-13T11:23:59.678 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.11643592699999999, 2024-09-13T11:23:59.678 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:23:59.678 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:23:59.678 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:23:59.678 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:23:59.678 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.678 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:23:59.678 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:36.427991+0000", 2024-09-13T11:23:59.678 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:59.679 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.679 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.679 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:23:59.679 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:36.427991+0000", 2024-09-13T11:23:59.679 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:59.679 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.679 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.679 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:23:59.679 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:36.427993+0000", 2024-09-13T11:23:59.679 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.7290000000000001e-06 2024-09-13T11:23:59.679 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.679 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.679 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:23:59.680 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:36.428024+0000", 2024-09-13T11:23:59.680 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.1328000000000002e-05 2024-09-13T11:23:59.680 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.680 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.680 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:23:59.680 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:36.543387+0000", 2024-09-13T11:23:59.680 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.115363178 2024-09-13T11:23:59.680 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.680 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.680 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:23:59.680 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:36.543406+0000", 2024-09-13T11:23:59.680 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.8964e-05 2024-09-13T11:23:59.680 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.680 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.681 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:23:59.681 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:36.543504+0000", 2024-09-13T11:23:59.681 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.8317000000000001e-05 2024-09-13T11:23:59.681 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.681 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.681 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:23:59.681 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:36.543518+0000", 2024-09-13T11:23:59.681 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3448999999999999e-05 2024-09-13T11:23:59.681 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.681 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.681 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:23:59.681 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:36.544427+0000", 2024-09-13T11:23:59.682 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00090896200000000003 2024-09-13T11:23:59.682 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:59.682 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:23:59.682 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:59.682 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.682 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.682 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPush(3.1d 79/78 [PushOp(3:bb428951:::benchmark_data_smithi177_31116_object1827:head, version: 43'113, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:bb428951:::benchmark_data_smithi177_31116_object1827:head@43'113, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:23:59.682 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:36.578754+0000", 2024-09-13T11:23:59.682 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 23.087965428, 2024-09-13T11:23:59.682 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.096471424, 2024-09-13T11:23:59.682 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:23:59.683 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:23:59.683 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:23:59.683 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:23:59.683 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.683 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:23:59.683 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:36.578754+0000", 2024-09-13T11:23:59.683 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:59.683 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.683 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.683 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:23:59.683 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:36.578754+0000", 2024-09-13T11:23:59.683 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:59.683 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.684 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.684 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:23:59.684 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:36.578756+0000", 2024-09-13T11:23:59.684 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.517e-06 2024-09-13T11:23:59.684 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.684 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.684 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:23:59.684 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:36.578780+0000", 2024-09-13T11:23:59.684 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.4723e-05 2024-09-13T11:23:59.684 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.684 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.684 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:23:59.684 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:36.674233+0000", 2024-09-13T11:23:59.684 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.095453024999999997 2024-09-13T11:23:59.685 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.685 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.685 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:23:59.685 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:36.674251+0000", 2024-09-13T11:23:59.685 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.7713000000000001e-05 2024-09-13T11:23:59.685 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.685 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.685 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:23:59.685 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:36.674358+0000", 2024-09-13T11:23:59.685 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000107206 2024-09-13T11:23:59.685 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.685 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.685 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:23:59.685 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:36.674371+0000", 2024-09-13T11:23:59.686 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3005e-05 2024-09-13T11:23:59.686 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.686 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.686 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:23:59.686 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:36.675226+0000", 2024-09-13T11:23:59.686 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00085423499999999996 2024-09-13T11:23:59.686 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:59.686 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:23:59.686 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:59.686 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.686 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.686 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPush(3.16 84/70 [PushOp(3:687b0a06:::benchmark_data_smithi177_31116_object43649:head, version: 66'2810, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:687b0a06:::benchmark_data_smithi177_31116_object43649:head@66'2810, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:23:59.686 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:41.802280+0000", 2024-09-13T11:23:59.686 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 17.864439633, 2024-09-13T11:23:59.687 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.12130893299999999, 2024-09-13T11:23:59.687 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:23:59.687 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:23:59.687 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:23:59.687 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:23:59.687 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.687 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:23:59.687 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.802280+0000", 2024-09-13T11:23:59.687 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:59.687 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.687 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.687 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:23:59.687 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.802280+0000", 2024-09-13T11:23:59.687 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:59.688 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.688 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.688 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:23:59.688 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.802282+0000", 2024-09-13T11:23:59.688 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.72e-06 2024-09-13T11:23:59.688 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.688 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.688 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:23:59.688 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.802355+0000", 2024-09-13T11:23:59.688 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.2960999999999995e-05 2024-09-13T11:23:59.688 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.688 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.688 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:23:59.689 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.921995+0000", 2024-09-13T11:23:59.689 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.11963987299999999 2024-09-13T11:23:59.689 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.689 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.689 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:23:59.689 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.922012+0000", 2024-09-13T11:23:59.689 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.7037e-05 2024-09-13T11:23:59.689 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.689 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.689 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:23:59.689 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.922131+0000", 2024-09-13T11:23:59.689 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000119843 2024-09-13T11:23:59.689 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.689 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.690 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:23:59.690 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.922146+0000", 2024-09-13T11:23:59.690 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.4090000000000001e-05 2024-09-13T11:23:59.690 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.690 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.690 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:23:59.690 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.923589+0000", 2024-09-13T11:23:59.690 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0014434090000000001 2024-09-13T11:23:59.690 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:59.690 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:23:59.690 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:59.690 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.690 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.690 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPush(3.16 85/70 [PushOp(3:6ac421eb:::benchmark_data_smithi177_31116_object43910:head, version: 66'2836, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:6ac421eb:::benchmark_data_smithi177_31116_object43910:head@66'2836, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:23:59.691 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:42.145291+0000", 2024-09-13T11:23:59.691 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 17.521428353000001, 2024-09-13T11:23:59.691 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.148514919, 2024-09-13T11:23:59.691 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:23:59.691 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:23:59.691 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:23:59.691 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:23:59.691 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.691 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:23:59.691 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.145291+0000", 2024-09-13T11:23:59.691 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:59.691 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.691 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.691 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:23:59.692 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.145291+0000", 2024-09-13T11:23:59.692 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:59.692 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.692 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.692 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:23:59.692 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.145293+0000", 2024-09-13T11:23:59.692 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.699e-06 2024-09-13T11:23:59.692 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.692 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.692 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:23:59.692 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.145417+0000", 2024-09-13T11:23:59.692 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00012415200000000001 2024-09-13T11:23:59.692 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.693 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.693 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:23:59.693 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.292964+0000", 2024-09-13T11:23:59.693 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.14754715400000001 2024-09-13T11:23:59.693 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.693 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.693 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:23:59.693 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.292981+0000", 2024-09-13T11:23:59.693 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.6790999999999999e-05 2024-09-13T11:23:59.693 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.693 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.693 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:23:59.693 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.293083+0000", 2024-09-13T11:23:59.694 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00010199799999999999 2024-09-13T11:23:59.694 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.694 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.694 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:23:59.694 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.293098+0000", 2024-09-13T11:23:59.694 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.4612e-05 2024-09-13T11:23:59.694 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.694 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.694 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:23:59.694 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.293806+0000", 2024-09-13T11:23:59.694 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00070851300000000002 2024-09-13T11:23:59.694 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:59.694 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:23:59.694 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:59.695 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.695 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.695 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:60390 3.16 e85/70 3:6a388186:::benchmark_data_smithi177_31116_object60389:head v 85'3357, mlcod=66'2835)", 2024-09-13T11:23:59.695 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:42.149102+0000", 2024-09-13T11:23:59.695 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 17.517617296000001, 2024-09-13T11:23:59.695 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.14821161799999999, 2024-09-13T11:23:59.695 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:23:59.695 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:23:59.695 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:59.695 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:23:59.695 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.695 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:23:59.695 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.149102+0000", 2024-09-13T11:23:59.695 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:59.695 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.696 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.696 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:23:59.696 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.149102+0000", 2024-09-13T11:23:59.696 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:59.696 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.696 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.696 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:23:59.696 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.149104+0000", 2024-09-13T11:23:59.696 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.708e-06 2024-09-13T11:23:59.696 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.696 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.696 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:23:59.696 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.149160+0000", 2024-09-13T11:23:59.697 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.5618000000000002e-05 2024-09-13T11:23:59.697 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.697 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.697 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:23:59.697 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.296071+0000", 2024-09-13T11:23:59.697 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.14691148600000001 2024-09-13T11:23:59.697 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.697 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.697 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:23:59.697 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.296076+0000", 2024-09-13T11:23:59.697 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.5580000000000002e-06 2024-09-13T11:23:59.697 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.697 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.697 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:23:59.698 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.296127+0000", 2024-09-13T11:23:59.698 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.1405e-05 2024-09-13T11:23:59.698 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.698 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.698 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:23:59.698 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.296150+0000", 2024-09-13T11:23:59.698 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.3113e-05 2024-09-13T11:23:59.698 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.698 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.698 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:23:59.698 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.297274+0000", 2024-09-13T11:23:59.698 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0011239900000000001 2024-09-13T11:23:59.698 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.698 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.699 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:23:59.699 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.297314+0000", 2024-09-13T11:23:59.699 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.9740000000000002e-05 2024-09-13T11:23:59.699 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:59.699 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:23:59.699 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:59.699 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.699 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.699 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPush(3.16 85/70 [PushOp(3:69aac9fd:::benchmark_data_smithi177_31116_object43914:head, version: 66'2837, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:69aac9fd:::benchmark_data_smithi177_31116_object43914:head@66'2837, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:23:59.699 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:42.295995+0000", 2024-09-13T11:23:59.699 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 17.370724239000001, 2024-09-13T11:23:59.699 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.215089431, 2024-09-13T11:23:59.700 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:23:59.700 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:23:59.700 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:23:59.700 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:23:59.700 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.700 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:23:59.700 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.295995+0000", 2024-09-13T11:23:59.700 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:59.700 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.700 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.700 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:23:59.700 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.295995+0000", 2024-09-13T11:23:59.700 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:59.700 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.701 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.701 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:23:59.701 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.295997+0000", 2024-09-13T11:23:59.701 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3170000000000001e-06 2024-09-13T11:23:59.701 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.701 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.701 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:23:59.701 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.296058+0000", 2024-09-13T11:23:59.701 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.1690000000000001e-05 2024-09-13T11:23:59.701 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.701 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.701 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:23:59.701 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.509606+0000", 2024-09-13T11:23:59.701 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.21354748300000001 2024-09-13T11:23:59.702 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.702 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.702 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:23:59.702 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.509638+0000", 2024-09-13T11:23:59.702 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.1644999999999999e-05 2024-09-13T11:23:59.702 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.702 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.702 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:23:59.702 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.509814+0000", 2024-09-13T11:23:59.702 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00017679499999999999 2024-09-13T11:23:59.703 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.703 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.703 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:23:59.703 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.509840+0000", 2024-09-13T11:23:59.703 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.5871e-05 2024-09-13T11:23:59.703 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.703 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.703 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:23:59.703 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.511085+0000", 2024-09-13T11:23:59.703 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00124463 2024-09-13T11:23:59.703 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:59.703 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:23:59.703 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:59.703 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.704 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.704 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPush(3.1d 91/78 [PushOp(3:bb0a08cd:::benchmark_data_smithi177_31116_object25660:head, version: 58'1568, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:bb0a08cd:::benchmark_data_smithi177_31116_object25660:head@58'1568, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:23:59.704 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:48.351121+0000", 2024-09-13T11:23:59.704 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 11.315598945, 2024-09-13T11:23:59.704 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.12585117900000001, 2024-09-13T11:23:59.704 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:23:59.704 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:23:59.704 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:23:59.704 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:23:59.704 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.704 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:23:59.704 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.351121+0000", 2024-09-13T11:23:59.704 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:59.705 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.705 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.705 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:23:59.705 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.351121+0000", 2024-09-13T11:23:59.705 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:59.705 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.705 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.705 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:23:59.705 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.351122+0000", 2024-09-13T11:23:59.705 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3009999999999999e-06 2024-09-13T11:23:59.705 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.705 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.705 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:23:59.705 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.351147+0000", 2024-09-13T11:23:59.706 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.4967999999999999e-05 2024-09-13T11:23:59.706 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.706 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.706 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:23:59.706 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.475788+0000", 2024-09-13T11:23:59.706 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.12464135799999999 2024-09-13T11:23:59.706 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.706 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.706 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:23:59.706 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.475808+0000", 2024-09-13T11:23:59.706 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.0143999999999999e-05 2024-09-13T11:23:59.706 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.706 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.706 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:23:59.707 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.475908+0000", 2024-09-13T11:23:59.707 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.9134000000000003e-05 2024-09-13T11:23:59.707 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.707 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.707 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:23:59.707 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.475922+0000", 2024-09-13T11:23:59.707 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.469e-05 2024-09-13T11:23:59.707 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.707 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.707 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:23:59.707 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.476972+0000", 2024-09-13T11:23:59.707 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0010495840000000001 2024-09-13T11:23:59.707 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:59.707 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:23:59.708 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:59.708 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.708 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.708 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:65403 3.16 e91/88 3:6c3dca05:::benchmark_data_smithi177_31116_object65402:head v 91'3529, mlcod=91'3526)", 2024-09-13T11:23:59.708 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:48.354117+0000", 2024-09-13T11:23:59.708 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 11.312602697999999, 2024-09-13T11:23:59.708 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.099921708999999997, 2024-09-13T11:23:59.708 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:23:59.708 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:23:59.708 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:59.708 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:23:59.708 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.708 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:23:59.709 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.354117+0000", 2024-09-13T11:23:59.709 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:59.709 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.709 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.709 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:23:59.709 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.354117+0000", 2024-09-13T11:23:59.709 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:59.709 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.709 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.709 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:23:59.709 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.354118+0000", 2024-09-13T11:23:59.709 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.159e-06 2024-09-13T11:23:59.709 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.709 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.710 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:23:59.710 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.354138+0000", 2024-09-13T11:23:59.710 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.9760000000000001e-05 2024-09-13T11:23:59.710 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.710 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.711 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:23:59.711 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.452655+0000", 2024-09-13T11:23:59.711 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.098517473999999994 2024-09-13T11:23:59.711 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.711 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.711 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:23:59.711 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.452664+0000", 2024-09-13T11:23:59.711 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.2810000000000008e-06 2024-09-13T11:23:59.711 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.711 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.711 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:23:59.711 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.452723+0000", 2024-09-13T11:23:59.711 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.9305999999999998e-05 2024-09-13T11:23:59.712 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.712 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.712 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:23:59.712 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.452749+0000", 2024-09-13T11:23:59.712 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.641e-05 2024-09-13T11:23:59.712 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.712 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.712 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:23:59.712 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.454008+0000", 2024-09-13T11:23:59.712 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0012589109999999999 2024-09-13T11:23:59.712 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.712 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.712 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:23:59.713 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.454039+0000", 2024-09-13T11:23:59.713 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.0408000000000001e-05 2024-09-13T11:23:59.713 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:59.713 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:23:59.713 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:59.713 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.713 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.713 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPush(3.1d 91/78 [PushOp(3:be6a0ae4:::benchmark_data_smithi177_31116_object25671:head, version: 58'1569, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:be6a0ae4:::benchmark_data_smithi177_31116_object25671:head@58'1569, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:23:59.713 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:48.481971+0000", 2024-09-13T11:23:59.713 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 11.184748994, 2024-09-13T11:23:59.713 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.47945454199999998, 2024-09-13T11:23:59.713 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:23:59.713 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:23:59.713 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:23:59.714 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:23:59.714 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.714 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:23:59.714 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.481971+0000", 2024-09-13T11:23:59.714 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:59.714 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.714 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.714 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:23:59.714 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.481971+0000", 2024-09-13T11:23:59.714 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:59.714 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.714 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.714 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:23:59.714 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.481972+0000", 2024-09-13T11:23:59.715 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5179999999999999e-06 2024-09-13T11:23:59.715 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.715 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.715 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:23:59.715 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.482000+0000", 2024-09-13T11:23:59.715 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.8249e-05 2024-09-13T11:23:59.715 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.715 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.715 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:23:59.715 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.959926+0000", 2024-09-13T11:23:59.715 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.47792585100000001 2024-09-13T11:23:59.716 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.716 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.716 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:23:59.716 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.959944+0000", 2024-09-13T11:23:59.716 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.7620000000000001e-05 2024-09-13T11:23:59.716 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.716 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.716 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:23:59.716 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.960059+0000", 2024-09-13T11:23:59.716 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000115188 2024-09-13T11:23:59.716 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.716 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.716 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:23:59.717 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.960074+0000", 2024-09-13T11:23:59.717 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.4738e-05 2024-09-13T11:23:59.717 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.717 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.717 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:23:59.717 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.961425+0000", 2024-09-13T11:23:59.717 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0013513780000000001 2024-09-13T11:23:59.717 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:59.717 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:23:59.717 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:59.717 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.717 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.717 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:69582 3.16 e95/88 3:6c6c1ec5:::benchmark_data_smithi177_31116_object69581:head v 95'3683, mlcod=94'3680)", 2024-09-13T11:23:59.717 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:52.181484+0000", 2024-09-13T11:23:59.718 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 7.485235437, 2024-09-13T11:23:59.718 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.101349655, 2024-09-13T11:23:59.718 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:23:59.718 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:23:59.718 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:59.718 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:23:59.718 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.718 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:23:59.718 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:52.181484+0000", 2024-09-13T11:23:59.718 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:59.718 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.718 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.718 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:23:59.718 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:52.181484+0000", 2024-09-13T11:23:59.719 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:59.719 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.719 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.719 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:23:59.719 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:52.181486+0000", 2024-09-13T11:23:59.719 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5719999999999999e-06 2024-09-13T11:23:59.719 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.719 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.719 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:23:59.719 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:52.181506+0000", 2024-09-13T11:23:59.719 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.0152e-05 2024-09-13T11:23:59.719 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.719 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.720 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:23:59.720 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:52.281315+0000", 2024-09-13T11:23:59.720 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.099809013000000002 2024-09-13T11:23:59.720 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.720 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.720 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:23:59.720 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:52.281322+0000", 2024-09-13T11:23:59.720 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.221e-06 2024-09-13T11:23:59.720 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.720 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.720 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:23:59.720 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:52.281379+0000", 2024-09-13T11:23:59.720 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.7181e-05 2024-09-13T11:23:59.721 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.721 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.721 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:23:59.721 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:52.281400+0000", 2024-09-13T11:23:59.721 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.0843999999999999e-05 2024-09-13T11:23:59.721 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.721 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.721 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:23:59.721 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:52.282791+0000", 2024-09-13T11:23:59.721 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0013904779999999999 2024-09-13T11:23:59.721 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.721 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.721 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:23:59.722 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:52.282834+0000", 2024-09-13T11:23:59.722 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.3194e-05 2024-09-13T11:23:59.722 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:59.722 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:23:59.722 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:59.722 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.722 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.722 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPush(3.1b 95/78 [PushOp(3:dc611864:::benchmark_data_smithi177_31116_object32124:head, version: 60'2016, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:dc611864:::benchmark_data_smithi177_31116_object32124:head@60'2016, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:23:59.722 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:52.740114+0000", 2024-09-13T11:23:59.722 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 6.9266055870000001, 2024-09-13T11:23:59.722 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.111506535, 2024-09-13T11:23:59.722 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:23:59.722 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:23:59.722 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:23:59.722 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:23:59.723 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.723 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:23:59.723 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:52.740114+0000", 2024-09-13T11:23:59.723 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:59.723 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.723 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.723 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:23:59.723 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:52.740114+0000", 2024-09-13T11:23:59.723 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:59.723 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.723 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.723 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:23:59.723 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:52.740116+0000", 2024-09-13T11:23:59.724 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.4950000000000001e-06 2024-09-13T11:23:59.724 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.724 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.724 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:23:59.724 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:52.740174+0000", 2024-09-13T11:23:59.724 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.8865e-05 2024-09-13T11:23:59.724 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.724 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.724 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:23:59.724 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:52.850840+0000", 2024-09-13T11:23:59.724 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.11066557 2024-09-13T11:23:59.724 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.724 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.724 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:23:59.725 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:52.850851+0000", 2024-09-13T11:23:59.725 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0733999999999999e-05 2024-09-13T11:23:59.725 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.725 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.725 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:23:59.725 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:52.850921+0000", 2024-09-13T11:23:59.725 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.0536000000000004e-05 2024-09-13T11:23:59.725 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.725 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.725 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:23:59.725 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:52.850929+0000", 2024-09-13T11:23:59.725 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.1990000000000006e-06 2024-09-13T11:23:59.725 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.726 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.726 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:23:59.726 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:52.851621+0000", 2024-09-13T11:23:59.726 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00069113599999999996 2024-09-13T11:23:59.726 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:59.726 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:23:59.726 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:59.726 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.726 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.726 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:70316 3.1b e95/78 3:da14819e:::benchmark_data_smithi177_31116_object70315:head v 95'3906, mlcod=60'2014)", 2024-09-13T11:23:59.726 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:52.768181+0000", 2024-09-13T11:23:59.726 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 6.8985385279999996, 2024-09-13T11:23:59.726 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.093599618999999995, 2024-09-13T11:23:59.726 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:23:59.726 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:23:59.727 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:59.727 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:23:59.727 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.727 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:23:59.727 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:52.768181+0000", 2024-09-13T11:23:59.727 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:59.727 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.727 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.727 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:23:59.727 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:52.768181+0000", 2024-09-13T11:23:59.727 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:59.727 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.727 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.728 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:23:59.728 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:52.768183+0000", 2024-09-13T11:23:59.728 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.716e-06 2024-09-13T11:23:59.728 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.728 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.728 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:23:59.728 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:52.768278+0000", 2024-09-13T11:23:59.728 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.5012000000000004e-05 2024-09-13T11:23:59.728 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.728 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.728 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:23:59.728 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:52.859200+0000", 2024-09-13T11:23:59.728 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.090922038999999996 2024-09-13T11:23:59.728 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.729 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.729 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:23:59.729 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:52.859209+0000", 2024-09-13T11:23:59.729 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.884e-06 2024-09-13T11:23:59.729 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.729 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.729 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:23:59.729 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:52.859269+0000", 2024-09-13T11:23:59.729 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.9908000000000003e-05 2024-09-13T11:23:59.729 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.729 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.729 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:23:59.729 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:52.859296+0000", 2024-09-13T11:23:59.729 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.7058000000000001e-05 2024-09-13T11:23:59.730 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.730 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.730 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:23:59.730 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:52.861744+0000", 2024-09-13T11:23:59.730 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.002448043 2024-09-13T11:23:59.730 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.730 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.730 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:23:59.730 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:52.861781+0000", 2024-09-13T11:23:59.730 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.6959000000000003e-05 2024-09-13T11:23:59.730 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:59.730 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:23:59.730 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:59.730 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.730 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.731 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:71884 3.16 e96/88 3:6ca846b2:::benchmark_data_smithi177_31116_object71883:head v 96'3755, mlcod=96'3753)", 2024-09-13T11:23:59.731 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:54.108299+0000", 2024-09-13T11:23:59.731 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 5.5584208210000003, 2024-09-13T11:23:59.731 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.20685140099999999, 2024-09-13T11:23:59.731 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:23:59.731 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:23:59.731 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:59.731 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:23:59.731 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.731 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:23:59.731 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.108299+0000", 2024-09-13T11:23:59.731 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:59.731 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.731 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.732 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:23:59.732 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.108299+0000", 2024-09-13T11:23:59.732 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:59.732 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.732 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.732 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:23:59.732 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.108300+0000", 2024-09-13T11:23:59.732 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.173e-06 2024-09-13T11:23:59.732 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.732 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.732 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:23:59.732 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.108323+0000", 2024-09-13T11:23:59.732 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.2591000000000001e-05 2024-09-13T11:23:59.732 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.733 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.733 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:23:59.733 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.312377+0000", 2024-09-13T11:23:59.733 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.20405456599999999 2024-09-13T11:23:59.733 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.733 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.733 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:23:59.733 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.312382+0000", 2024-09-13T11:23:59.733 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.1429999999999997e-06 2024-09-13T11:23:59.733 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.733 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.733 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:23:59.734 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.312480+0000", 2024-09-13T11:23:59.734 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.7742e-05 2024-09-13T11:23:59.734 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.734 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.734 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:23:59.734 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.312521+0000", 2024-09-13T11:23:59.734 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.1369000000000003e-05 2024-09-13T11:23:59.734 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.734 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.734 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:23:59.734 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.315105+0000", 2024-09-13T11:23:59.734 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0025835060000000002 2024-09-13T11:23:59.734 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.735 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.735 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:23:59.735 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.315150+0000", 2024-09-13T11:23:59.735 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.5311e-05 2024-09-13T11:23:59.735 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:59.735 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:23:59.735 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:59.735 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.735 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.735 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:71927 3.0 e97/75 3:0197244a:::benchmark_data_smithi177_31116_object71926:head v 97'3351, mlcod=96'3348)", 2024-09-13T11:23:59.735 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:54.200161+0000", 2024-09-13T11:23:59.735 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 5.4665585930000002, 2024-09-13T11:23:59.735 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.26691738999999998, 2024-09-13T11:23:59.735 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:23:59.736 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:23:59.736 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:59.736 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:23:59.736 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.736 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:23:59.736 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.200161+0000", 2024-09-13T11:23:59.736 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:59.736 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.736 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.736 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:23:59.736 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.200161+0000", 2024-09-13T11:23:59.736 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:59.736 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.737 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.737 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:23:59.737 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.200162+0000", 2024-09-13T11:23:59.737 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.78e-07 2024-09-13T11:23:59.737 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.737 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.737 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:23:59.737 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.200200+0000", 2024-09-13T11:23:59.737 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.8121000000000003e-05 2024-09-13T11:23:59.737 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.737 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.737 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:23:59.737 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.465397+0000", 2024-09-13T11:23:59.737 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.26519681099999998 2024-09-13T11:23:59.738 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.738 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.738 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:23:59.738 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.465404+0000", 2024-09-13T11:23:59.738 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.7399999999999998e-06 2024-09-13T11:23:59.738 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.738 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.738 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:23:59.738 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.465456+0000", 2024-09-13T11:23:59.738 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.2626000000000001e-05 2024-09-13T11:23:59.738 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.738 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.738 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:23:59.738 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.465482+0000", 2024-09-13T11:23:59.738 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.6086000000000001e-05 2024-09-13T11:23:59.739 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.739 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.739 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:23:59.739 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.467035+0000", 2024-09-13T11:23:59.739 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0015523729999999999 2024-09-13T11:23:59.739 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.739 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.739 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:23:59.739 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.467078+0000", 2024-09-13T11:23:59.739 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.3754999999999997e-05 2024-09-13T11:23:59.739 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:59.739 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:23:59.739 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:59.739 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.739 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.740 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:72250 3.0 e97/75 3:010c7171:::benchmark_data_smithi177_31116_object72249:head v 97'3353, mlcod=96'3350)", 2024-09-13T11:23:59.740 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:54.516381+0000", 2024-09-13T11:23:59.740 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 5.1503390800000002, 2024-09-13T11:23:59.740 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.13663347300000001, 2024-09-13T11:23:59.740 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:23:59.740 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:23:59.740 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:59.740 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:23:59.740 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.740 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:23:59.740 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.516381+0000", 2024-09-13T11:23:59.740 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:59.740 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.740 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.741 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:23:59.741 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.516381+0000", 2024-09-13T11:23:59.741 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:59.741 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.741 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.741 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:23:59.741 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.516382+0000", 2024-09-13T11:23:59.741 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3239999999999999e-06 2024-09-13T11:23:59.741 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.741 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.741 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:23:59.741 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.516437+0000", 2024-09-13T11:23:59.741 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.5408999999999998e-05 2024-09-13T11:23:59.742 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.742 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.742 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:23:59.742 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.651731+0000", 2024-09-13T11:23:59.742 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.135293583 2024-09-13T11:23:59.742 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.742 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.742 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:23:59.742 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.651736+0000", 2024-09-13T11:23:59.742 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.3410000000000002e-06 2024-09-13T11:23:59.742 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.742 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.742 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:23:59.742 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.651791+0000", 2024-09-13T11:23:59.743 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.4758e-05 2024-09-13T11:23:59.743 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.743 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.743 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:23:59.743 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.651814+0000", 2024-09-13T11:23:59.743 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.3283999999999999e-05 2024-09-13T11:23:59.743 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.743 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.743 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:23:59.743 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.652975+0000", 2024-09-13T11:23:59.743 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0011604370000000001 2024-09-13T11:23:59.743 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.743 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.744 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:23:59.744 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.653014+0000", 2024-09-13T11:23:59.744 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.9337e-05 2024-09-13T11:23:59.744 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:59.744 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:23:59.744 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:59.744 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.744 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.744 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:72993 3.0 e97/75 3:02747b8f:::benchmark_data_smithi177_31116_object72992:head v 97'3366, mlcod=97'3363)", 2024-09-13T11:23:59.744 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:55.116009+0000", 2024-09-13T11:23:59.744 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 4.5507104739999997, 2024-09-13T11:23:59.744 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.13962660099999999, 2024-09-13T11:23:59.744 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:23:59.744 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:23:59.745 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:59.745 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:23:59.745 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.745 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:23:59.745 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:55.116009+0000", 2024-09-13T11:23:59.745 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:59.745 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.745 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.745 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:23:59.745 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:55.116009+0000", 2024-09-13T11:23:59.745 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:59.745 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.745 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.745 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:23:59.746 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:55.116010+0000", 2024-09-13T11:23:59.746 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.291e-06 2024-09-13T11:23:59.746 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.746 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.746 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:23:59.746 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:55.116062+0000", 2024-09-13T11:23:59.746 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.1949e-05 2024-09-13T11:23:59.746 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.746 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.746 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:23:59.746 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:55.254100+0000", 2024-09-13T11:23:59.746 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.13803723000000001 2024-09-13T11:23:59.746 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.747 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.747 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:23:59.747 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:55.254106+0000", 2024-09-13T11:23:59.747 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.5939999999999999e-06 2024-09-13T11:23:59.747 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.747 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.747 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:23:59.747 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:55.254196+0000", 2024-09-13T11:23:59.747 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.9567999999999994e-05 2024-09-13T11:23:59.747 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.747 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.747 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:23:59.747 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:55.254219+0000", 2024-09-13T11:23:59.748 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.3266000000000001e-05 2024-09-13T11:23:59.748 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.748 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.748 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:23:59.748 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:55.255596+0000", 2024-09-13T11:23:59.748 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0013767409999999999 2024-09-13T11:23:59.748 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.748 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.748 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:23:59.748 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:55.255636+0000", 2024-09-13T11:23:59.748 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.9962000000000001e-05 2024-09-13T11:23:59.748 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:59.748 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:23:59.748 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:59.749 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.749 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.749 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:72999 3.5 3:a297a429:::benchmark_data_smithi177_31116_object72998:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2024-09-13T11:23:59.749 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:55.121503+0000", 2024-09-13T11:23:59.749 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 4.5452170389999997, 2024-09-13T11:23:59.749 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.095355256999999999, 2024-09-13T11:23:59.749 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:23:59.749 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:23:59.749 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:59.749 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:23:59.749 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:23:59.749 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:59.749 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 72999 2024-09-13T11:23:59.749 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.750 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:23:59.750 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.750 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:23:59.750 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:55.121503+0000", 2024-09-13T11:23:59.750 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:59.750 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.750 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.750 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:23:59.750 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:55.121503+0000", 2024-09-13T11:23:59.750 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:59.750 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.750 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.750 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:23:59.751 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:55.121505+0000", 2024-09-13T11:23:59.751 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.5210000000000001e-06 2024-09-13T11:23:59.751 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.751 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.751 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:23:59.751 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:55.121529+0000", 2024-09-13T11:23:59.751 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.3737e-05 2024-09-13T11:23:59.751 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.751 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.751 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:23:59.751 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:55.121544+0000", 2024-09-13T11:23:59.751 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.4983e-05 2024-09-13T11:23:59.751 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.751 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.752 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:23:59.752 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:55.121552+0000", 2024-09-13T11:23:59.752 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.0830000000000004e-06 2024-09-13T11:23:59.752 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.752 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.752 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:23:59.752 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:55.121613+0000", 2024-09-13T11:23:59.752 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.0903999999999998e-05 2024-09-13T11:23:59.752 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.752 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.752 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:23:59.752 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:55.121744+0000", 2024-09-13T11:23:59.752 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000130782 2024-09-13T11:23:59.752 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.753 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.753 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 13", 2024-09-13T11:23:59.753 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:55.121865+0000", 2024-09-13T11:23:59.753 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000121851 2024-09-13T11:23:59.753 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.753 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.753 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:23:59.753 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:55.123453+0000", 2024-09-13T11:23:59.753 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001587455 2024-09-13T11:23:59.753 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.753 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.753 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:23:59.753 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:55.216796+0000", 2024-09-13T11:23:59.753 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.093342772000000004 2024-09-13T11:23:59.754 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.754 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.754 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:23:59.754 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:55.216829+0000", 2024-09-13T11:23:59.754 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.3668999999999999e-05 2024-09-13T11:23:59.754 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.754 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.754 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:23:59.754 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:55.216858+0000", 2024-09-13T11:23:59.754 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.8500000000000002e-05 2024-09-13T11:23:59.754 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:59.754 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:23:59.754 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:59.755 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.755 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.755 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:73523 3.0 e98/75 3:004115ff:::benchmark_data_smithi177_31116_object73522:head v 98'3382, mlcod=98'3378)", 2024-09-13T11:23:59.755 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:55.538309+0000", 2024-09-13T11:23:59.755 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 4.128410476, 2024-09-13T11:23:59.755 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.127489824, 2024-09-13T11:23:59.755 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:23:59.755 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:23:59.755 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:23:59.755 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:23:59.755 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.755 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:23:59.755 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:55.538309+0000", 2024-09-13T11:23:59.755 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:59.756 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.756 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.756 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:23:59.756 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:55.538309+0000", 2024-09-13T11:23:59.756 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:23:59.756 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.756 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.756 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:23:59.756 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:55.538310+0000", 2024-09-13T11:23:59.756 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.237e-06 2024-09-13T11:23:59.756 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.756 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.756 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:23:59.756 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:55.538343+0000", 2024-09-13T11:23:59.757 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.2480000000000001e-05 2024-09-13T11:23:59.757 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.757 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.757 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:23:59.757 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:55.663154+0000", 2024-09-13T11:23:59.757 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.124810988 2024-09-13T11:23:59.757 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.757 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.757 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:23:59.757 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:55.663161+0000", 2024-09-13T11:23:59.757 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.2389999999999996e-06 2024-09-13T11:23:59.757 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.757 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.758 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:23:59.758 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:55.663215+0000", 2024-09-13T11:23:59.758 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.3893000000000001e-05 2024-09-13T11:23:59.758 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.758 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.758 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:23:59.758 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:55.663235+0000", 2024-09-13T11:23:59.758 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.9624e-05 2024-09-13T11:23:59.758 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.758 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.758 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:23:59.758 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:55.665764+0000", 2024-09-13T11:23:59.758 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0025292859999999999 2024-09-13T11:23:59.758 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:23:59.759 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:23:59.759 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:23:59.759 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:55.665799+0000", 2024-09-13T11:23:59.759 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.5077000000000003e-05 2024-09-13T11:23:59.759 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:59.759 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:23:59.759 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:59.759 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:23:59.759 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:23:59.759 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-09-13T11:23:59.761 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-13T11:23:59.761 INFO:tasks.ceph.osd.10.smithi162.stderr:2024-09-13T11:23:59.757+0000 7fb59c9f0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26317) UID: 0 2024-09-13T11:23:59.861 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-09-13T11:23:59.857+0000 7f97befc9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26315) UID: 0 2024-09-13T11:23:59.909 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-09-13T11:23:59.909 INFO:teuthology.orchestra.run.smithi092.stdout: "ops": [ 2024-09-13T11:23:59.909 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:59.909 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:79158 3.4 3:279da32a:::benchmark_data_smithi177_31116_object79157:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101)", 2024-09-13T11:23:59.910 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:59.865495+0000", 2024-09-13T11:23:59.910 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 0.043306928000000001, 2024-09-13T11:23:59.910 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.043328428000000002, 2024-09-13T11:23:59.910 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:23:59.910 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:23:59.910 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "waiting for sub ops", 2024-09-13T11:23:59.910 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:23:59.910 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:23:59.910 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:23:59.910 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 79158 2024-09-13T11:23:59.910 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:59.911 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:23:59.911 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:59.911 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:23:59.911 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:59.865495+0000", 2024-09-13T11:23:59.911 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:59.911 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:59.911 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:59.911 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:23:59.911 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:59.865495+0000", 2024-09-13T11:23:59.911 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:23:59.911 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:59.911 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:59.911 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:23:59.911 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:59.865497+0000", 2024-09-13T11:23:59.911 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.3920000000000001e-06 2024-09-13T11:23:59.912 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:59.912 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:59.912 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:23:59.912 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:59.865557+0000", 2024-09-13T11:23:59.912 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.0386000000000001e-05 2024-09-13T11:23:59.912 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:59.912 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:59.912 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:23:59.912 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:59.865574+0000", 2024-09-13T11:23:59.912 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.6147999999999999e-05 2024-09-13T11:23:59.912 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:59.912 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:59.912 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:23:59.912 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:59.865584+0000", 2024-09-13T11:23:59.913 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0738e-05 2024-09-13T11:23:59.913 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:59.913 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:59.913 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:23:59.913 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:59.865645+0000", 2024-09-13T11:23:59.913 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.0346000000000001e-05 2024-09-13T11:23:59.913 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:59.913 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:59.913 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:23:59.913 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:59.865813+0000", 2024-09-13T11:23:59.913 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000168567 2024-09-13T11:23:59.913 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:59.913 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:59.914 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 12", 2024-09-13T11:23:59.914 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:59.865974+0000", 2024-09-13T11:23:59.914 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00016034900000000001 2024-09-13T11:23:59.914 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:23:59.914 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:23:59.914 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:23:59.914 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:59.867669+0000", 2024-09-13T11:23:59.914 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.001695883 2024-09-13T11:23:59.914 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:59.914 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:23:59.914 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:59.915 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:23:59.915 INFO:teuthology.orchestra.run.smithi092.stdout: ], 2024-09-13T11:23:59.915 INFO:teuthology.orchestra.run.smithi092.stdout: "num_ops": 1 2024-09-13T11:23:59.915 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-09-13T11:23:59.930 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-13T11:23:59.963 INFO:tasks.ceph.osd.11.smithi177.stderr:2024-09-13T11:23:59.956+0000 7f24792c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26580) UID: 0 2024-09-13T11:24:00.008 INFO:tasks.radosbench.radosbench.0.smithi177.stdout: 93 16 79411 79395 53.3514 91.75 0.0103911 0.0187364 2024-09-13T11:24:00.063 INFO:tasks.ceph.osd.4.smithi001.stderr:2024-09-13T11:24:00.056+0000 7ff8e0218640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26310) UID: 0 2024-09-13T11:24:00.081 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-09-13T11:24:00.081 INFO:teuthology.orchestra.run.smithi092.stdout: "ops": [], 2024-09-13T11:24:00.082 INFO:teuthology.orchestra.run.smithi092.stdout: "complaint_time": 30, 2024-09-13T11:24:00.082 INFO:teuthology.orchestra.run.smithi092.stdout: "num_blocked_ops": 0 2024-09-13T11:24:00.082 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-09-13T11:24:00.102 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-13T11:24:00.163 INFO:tasks.ceph.osd.5.smithi092.stderr:2024-09-13T11:24:00.159+0000 7f6f0b705640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26346) UID: 0 2024-09-13T11:24:00.239 INFO:tasks.ceph.osd.14.smithi162.stderr:2024-09-13T11:24:00.237+0000 7f2350b08640 -1 osd.14 101 handle_osd_map injecting map crc failure 2024-09-13T11:24:00.240 INFO:tasks.ceph.osd.10.smithi162.stderr:2024-09-13T11:24:00.237+0000 7fb59141d640 -1 osd.10 101 handle_osd_map injecting map crc failure 2024-09-13T11:24:00.242 INFO:tasks.ceph.osd.15.smithi177.stderr:2024-09-13T11:24:00.236+0000 7f9d618a4640 -1 osd.15 101 handle_osd_map injecting map crc failure 2024-09-13T11:24:00.243 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-09-13T11:24:00.243 INFO:teuthology.orchestra.run.smithi092.stdout: "size": 20, 2024-09-13T11:24:00.243 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 600, 2024-09-13T11:24:00.244 INFO:teuthology.orchestra.run.smithi092.stdout: "ops": [ 2024-09-13T11:24:00.244 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:00.244 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:3 3.4 3.125e09e4 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e40)", 2024-09-13T11:24:00.244 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:46.304641+0000", 2024-09-13T11:24:00.244 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 73.936718776000006, 2024-09-13T11:24:00.244 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.374372119, 2024-09-13T11:24:00.244 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:24:00.244 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:24:00.244 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "reached pg", 2024-09-13T11:24:00.244 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:24:00.244 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:24:00.244 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:00.244 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 3 2024-09-13T11:24:00.245 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:00.245 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:24:00.245 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:00.245 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:24:00.245 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.304641+0000", 2024-09-13T11:24:00.245 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:00.245 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:00.245 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:00.245 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:24:00.245 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.304641+0000", 2024-09-13T11:24:00.245 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:00.245 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:00.245 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:00.245 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:24:00.246 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.304644+0000", 2024-09-13T11:24:00.246 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.5130000000000001e-06 2024-09-13T11:24:00.246 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:00.246 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:00.246 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:24:00.246 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305058+0000", 2024-09-13T11:24:00.246 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000414277 2024-09-13T11:24:00.246 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:00.246 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:00.246 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:24:00.246 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305124+0000", 2024-09-13T11:24:00.246 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.5813000000000002e-05 2024-09-13T11:24:00.246 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:00.246 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:00.247 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:24:00.247 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305156+0000", 2024-09-13T11:24:00.247 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.1489000000000003e-05 2024-09-13T11:24:00.247 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:00.247 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:00.247 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:24:00.247 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.678939+0000", 2024-09-13T11:24:00.247 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.37378339999999999 2024-09-13T11:24:00.247 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:00.247 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:00.247 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:24:00.247 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.679013+0000", 2024-09-13T11:24:00.247 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.3627000000000005e-05 2024-09-13T11:24:00.247 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:00.248 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:24:00.248 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:00.248 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:00.248 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:00.248 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:29 3.4 3.b7bbfbe4 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-09-13T11:24:00.248 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:46.305215+0000", 2024-09-13T11:24:00.248 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 73.936144472999999, 2024-09-13T11:24:00.248 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.37387964600000001, 2024-09-13T11:24:00.248 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:24:00.248 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:24:00.248 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "reached pg", 2024-09-13T11:24:00.249 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:24:00.249 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:24:00.249 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:00.249 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 29 2024-09-13T11:24:00.249 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:00.249 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:24:00.249 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:00.249 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:24:00.249 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305215+0000", 2024-09-13T11:24:00.249 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:00.249 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:00.249 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:00.250 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:24:00.250 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305215+0000", 2024-09-13T11:24:00.250 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:00.250 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:00.250 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:00.250 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:24:00.250 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305218+0000", 2024-09-13T11:24:00.250 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.7030000000000002e-06 2024-09-13T11:24:00.250 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:00.250 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:00.250 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:24:00.250 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305366+0000", 2024-09-13T11:24:00.250 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00014825400000000001 2024-09-13T11:24:00.250 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:00.251 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:00.251 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:24:00.251 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305386+0000", 2024-09-13T11:24:00.251 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.9689000000000001e-05 2024-09-13T11:24:00.251 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:00.251 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:00.251 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:24:00.251 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305403+0000", 2024-09-13T11:24:00.251 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.7600000000000001e-05 2024-09-13T11:24:00.251 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:00.251 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:00.251 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:24:00.251 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.679067+0000", 2024-09-13T11:24:00.251 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.37366339100000001 2024-09-13T11:24:00.252 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:00.252 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:00.252 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:24:00.252 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.679095+0000", 2024-09-13T11:24:00.252 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.8008999999999999e-05 2024-09-13T11:24:00.252 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:00.252 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:24:00.252 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:00.252 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:00.252 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:00.252 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:30 3.4 3.547ec0f4 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-09-13T11:24:00.252 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:46.305453+0000", 2024-09-13T11:24:00.252 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 73.935906349000007, 2024-09-13T11:24:00.252 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.373764914, 2024-09-13T11:24:00.253 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:24:00.253 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:24:00.253 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "reached pg", 2024-09-13T11:24:00.253 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:24:00.253 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:24:00.253 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:00.253 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 30 2024-09-13T11:24:00.253 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:00.253 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:24:00.253 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:00.254 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:24:00.254 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305453+0000", 2024-09-13T11:24:00.254 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:00.254 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:00.254 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:00.254 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:24:00.254 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305453+0000", 2024-09-13T11:24:00.254 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:00.254 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:00.254 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:00.255 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:24:00.255 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305456+0000", 2024-09-13T11:24:00.255 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.9560000000000002e-06 2024-09-13T11:24:00.255 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:00.255 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:00.255 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:24:00.255 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305597+0000", 2024-09-13T11:24:00.255 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00014122700000000001 2024-09-13T11:24:00.255 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:00.255 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:00.255 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:24:00.255 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305615+0000", 2024-09-13T11:24:00.255 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.8139999999999999e-05 2024-09-13T11:24:00.256 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:00.256 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:00.256 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:24:00.256 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305632+0000", 2024-09-13T11:24:00.256 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.6101999999999999e-05 2024-09-13T11:24:00.256 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:00.256 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:00.256 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:24:00.256 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.679195+0000", 2024-09-13T11:24:00.256 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.373563486 2024-09-13T11:24:00.256 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:00.256 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:00.256 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:24:00.256 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.679218+0000", 2024-09-13T11:24:00.257 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.3003e-05 2024-09-13T11:24:00.257 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:00.257 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:24:00.257 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:00.257 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:00.257 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:00.257 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:70 3.4 3.1404a604 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-09-13T11:24:00.257 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:46.305665+0000", 2024-09-13T11:24:00.257 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 73.935694201000004, 2024-09-13T11:24:00.257 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.37365550199999997, 2024-09-13T11:24:00.257 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:24:00.257 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:24:00.257 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "reached pg", 2024-09-13T11:24:00.258 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:24:00.258 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:24:00.258 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:00.258 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 70 2024-09-13T11:24:00.258 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:00.258 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:24:00.258 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:00.258 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:24:00.258 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305665+0000", 2024-09-13T11:24:00.258 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:00.258 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:00.258 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:00.258 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:24:00.258 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305665+0000", 2024-09-13T11:24:00.258 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:00.259 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:00.259 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:00.259 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:24:00.259 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305668+0000", 2024-09-13T11:24:00.259 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.5890000000000001e-06 2024-09-13T11:24:00.259 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:00.259 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:00.259 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:24:00.259 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305791+0000", 2024-09-13T11:24:00.259 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000123545 2024-09-13T11:24:00.259 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:00.259 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:00.259 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:24:00.260 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305808+0000", 2024-09-13T11:24:00.260 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.668e-05 2024-09-13T11:24:00.260 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:00.260 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:00.260 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:24:00.260 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305824+0000", 2024-09-13T11:24:00.260 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.6385e-05 2024-09-13T11:24:00.260 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:00.260 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:00.260 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:24:00.260 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.679305+0000", 2024-09-13T11:24:00.260 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.37348059900000002 2024-09-13T11:24:00.260 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:00.260 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:00.261 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:24:00.261 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.679321+0000", 2024-09-13T11:24:00.261 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.5704000000000001e-05 2024-09-13T11:24:00.261 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:00.261 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:24:00.261 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:00.261 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:00.261 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:00.261 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:25408 3.4 3:294e23bd:::benchmark_data_smithi177_31116_object25407:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e56)", 2024-09-13T11:24:00.261 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:04.495455+0000", 2024-09-13T11:24:00.261 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 55.745904549000002, 2024-09-13T11:24:00.261 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.29897367600000002, 2024-09-13T11:24:00.261 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:24:00.261 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:24:00.262 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:00.262 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:24:00.262 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:24:00.262 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:00.262 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 25408 2024-09-13T11:24:00.262 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:00.262 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:24:00.262 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:00.262 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:24:00.262 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.495455+0000", 2024-09-13T11:24:00.262 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:00.262 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:00.262 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:00.262 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:24:00.263 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.495455+0000", 2024-09-13T11:24:00.263 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:00.263 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:00.263 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:00.263 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:24:00.263 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.495459+0000", 2024-09-13T11:24:00.263 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.5630000000000002e-06 2024-09-13T11:24:00.263 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:00.264 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:00.264 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:24:00.264 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.495545+0000", 2024-09-13T11:24:00.264 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.5704000000000005e-05 2024-09-13T11:24:00.264 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:00.264 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:00.264 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:24:00.264 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.495581+0000", 2024-09-13T11:24:00.264 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.5689000000000003e-05 2024-09-13T11:24:00.264 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:00.264 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:00.264 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:24:00.264 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.495597+0000", 2024-09-13T11:24:00.265 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.6053e-05 2024-09-13T11:24:00.265 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:00.265 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:00.265 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:24:00.265 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.495721+0000", 2024-09-13T11:24:00.265 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00012379099999999999 2024-09-13T11:24:00.265 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:00.265 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:00.265 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:24:00.265 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.495940+0000", 2024-09-13T11:24:00.265 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00021967699999999999 2024-09-13T11:24:00.266 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:00.266 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:00.266 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 12", 2024-09-13T11:24:00.266 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.496162+0000", 2024-09-13T11:24:00.266 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000221917 2024-09-13T11:24:00.266 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:00.266 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:00.266 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:24:00.266 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.498699+0000", 2024-09-13T11:24:00.266 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0025366690000000001 2024-09-13T11:24:00.267 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:00.267 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:00.267 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:00.267 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.794139+0000", 2024-09-13T11:24:00.267 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.295440025 2024-09-13T11:24:00.267 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:00.267 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:00.267 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:24:00.267 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.794306+0000", 2024-09-13T11:24:00.267 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00016689799999999999 2024-09-13T11:24:00.267 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:00.267 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:00.267 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:24:00.267 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.794429+0000", 2024-09-13T11:24:00.268 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00012269000000000001 2024-09-13T11:24:00.268 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:00.268 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:24:00.268 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:00.268 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:00.268 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:00.268 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop(client.4665.0:30855 3.b e59/40 3:dc59714f:::benchmark_data_smithi177_31116_object30854:head v 59'1930, mlcod=59'1927)", 2024-09-13T11:24:00.268 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:09.550713+0000", 2024-09-13T11:24:00.268 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 50.690646757000003, 2024-09-13T11:24:00.268 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.17337753, 2024-09-13T11:24:00.268 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:24:00.268 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:24:00.268 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:00.269 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:24:00.269 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:00.269 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:24:00.269 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:09.550713+0000", 2024-09-13T11:24:00.269 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:00.269 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:00.269 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:00.269 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:24:00.269 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:09.550713+0000", 2024-09-13T11:24:00.269 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:00.269 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:00.269 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:00.269 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:24:00.269 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:09.550716+0000", 2024-09-13T11:24:00.270 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.5099999999999999e-06 2024-09-13T11:24:00.270 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:00.270 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:00.270 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:24:00.270 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:09.550764+0000", 2024-09-13T11:24:00.270 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.7389000000000003e-05 2024-09-13T11:24:00.270 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:00.270 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:00.270 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:24:00.270 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:09.719062+0000", 2024-09-13T11:24:00.270 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.16829854499999999 2024-09-13T11:24:00.270 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:00.270 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:00.270 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:24:00.270 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:09.719075+0000", 2024-09-13T11:24:00.271 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.2969000000000001e-05 2024-09-13T11:24:00.271 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:00.271 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:00.271 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:24:00.271 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:09.719216+0000", 2024-09-13T11:24:00.271 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000141266 2024-09-13T11:24:00.271 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:00.271 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:00.271 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:24:00.271 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:09.719277+0000", 2024-09-13T11:24:00.271 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.0244999999999997e-05 2024-09-13T11:24:00.271 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:00.271 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:00.271 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:24:00.272 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:09.724019+0000", 2024-09-13T11:24:00.272 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0047426020000000003 2024-09-13T11:24:00.272 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:00.272 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:00.272 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:24:00.272 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:09.724090+0000", 2024-09-13T11:24:00.272 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.1004e-05 2024-09-13T11:24:00.272 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:00.272 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:24:00.272 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:00.272 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:00.272 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:00.272 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:49963 3.18 3:19b86b73:::benchmark_data_smithi177_31116_object49962:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e72)", 2024-09-13T11:24:00.272 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:26.307557+0000", 2024-09-13T11:24:00.273 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 33.933802573999998, 2024-09-13T11:24:00.273 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.22140597300000001, 2024-09-13T11:24:00.273 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:24:00.273 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:24:00.273 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:00.273 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:24:00.273 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:24:00.273 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:00.273 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 49963 2024-09-13T11:24:00.273 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:00.273 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:24:00.273 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:00.273 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:24:00.273 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:26.307557+0000", 2024-09-13T11:24:00.274 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:00.274 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:00.274 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:00.274 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:24:00.274 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:26.307557+0000", 2024-09-13T11:24:00.274 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:00.274 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:00.274 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:00.274 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:24:00.274 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:26.307558+0000", 2024-09-13T11:24:00.274 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.3740000000000001e-06 2024-09-13T11:24:00.274 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:00.274 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:00.274 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:24:00.275 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:26.516580+0000", 2024-09-13T11:24:00.275 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.209021596 2024-09-13T11:24:00.275 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:00.275 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:00.275 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:24:00.275 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:26.516614+0000", 2024-09-13T11:24:00.275 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.3858e-05 2024-09-13T11:24:00.275 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:00.275 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:00.275 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:24:00.275 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:26.516628+0000", 2024-09-13T11:24:00.275 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.4715999999999999e-05 2024-09-13T11:24:00.275 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:00.275 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:00.276 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:24:00.276 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:26.516790+0000", 2024-09-13T11:24:00.276 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00016138 2024-09-13T11:24:00.276 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:00.276 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:00.276 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:24:00.276 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:26.517123+0000", 2024-09-13T11:24:00.276 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00033296600000000002 2024-09-13T11:24:00.276 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:00.276 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:00.276 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 0,10", 2024-09-13T11:24:00.276 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:26.517367+0000", 2024-09-13T11:24:00.276 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00024416900000000003 2024-09-13T11:24:00.276 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:00.277 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:00.277 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:24:00.277 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:26.526939+0000", 2024-09-13T11:24:00.277 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0095718650000000006 2024-09-13T11:24:00.277 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:00.277 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:00.277 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:00.277 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:26.527095+0000", 2024-09-13T11:24:00.277 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00015643800000000001 2024-09-13T11:24:00.277 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:00.277 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:00.277 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:00.277 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:26.527254+0000", 2024-09-13T11:24:00.277 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000158383 2024-09-13T11:24:00.278 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:00.278 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:00.278 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:24:00.278 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:26.527405+0000", 2024-09-13T11:24:00.278 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000151731 2024-09-13T11:24:00.278 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:00.278 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:00.278 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:24:00.278 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:26.528963+0000", 2024-09-13T11:24:00.278 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0015574969999999999 2024-09-13T11:24:00.278 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:00.278 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:24:00.278 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:00.278 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:00.279 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:00.279 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "MOSDPGPush(3.e 72/57 [PushOp(3:7481641d:::benchmark_data_smithi177_31116_object23454:head, version: 55'1541, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:7481641d:::benchmark_data_smithi177_31116_object23454:head@55'1541, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:24:00.279 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:26.368394+0000", 2024-09-13T11:24:00.279 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 33.872965796000003, 2024-09-13T11:24:00.279 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.16488114500000001, 2024-09-13T11:24:00.279 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:24:00.279 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:24:00.279 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-09-13T11:24:00.279 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:24:00.279 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:00.279 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:24:00.279 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:26.368394+0000", 2024-09-13T11:24:00.279 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:00.279 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:00.280 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:00.280 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:24:00.280 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:26.368394+0000", 2024-09-13T11:24:00.280 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:00.280 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:00.280 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:00.280 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:24:00.280 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:26.368395+0000", 2024-09-13T11:24:00.280 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.054e-06 2024-09-13T11:24:00.280 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:00.280 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:00.280 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:24:00.281 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:26.368453+0000", 2024-09-13T11:24:00.281 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.8621e-05 2024-09-13T11:24:00.281 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:00.281 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:00.281 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:24:00.281 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:26.528803+0000", 2024-09-13T11:24:00.281 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.16034983 2024-09-13T11:24:00.281 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:00.281 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:00.281 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:24:00.281 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:26.528834+0000", 2024-09-13T11:24:00.282 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.1288000000000003e-05 2024-09-13T11:24:00.282 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:00.282 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:00.282 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:24:00.282 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:26.529064+0000", 2024-09-13T11:24:00.282 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00022959 2024-09-13T11:24:00.282 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:00.282 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:00.282 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:24:00.282 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:26.529097+0000", 2024-09-13T11:24:00.282 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.3423999999999997e-05 2024-09-13T11:24:00.282 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:00.282 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:00.283 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:24:00.283 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:26.533275+0000", 2024-09-13T11:24:00.283 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0041773380000000001 2024-09-13T11:24:00.283 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:00.283 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:24:00.283 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:00.283 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:00.283 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:00.283 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "MOSDPGPushReply(3.14 75/70 [PushReplyOp(3:2f14928a:::benchmark_data_smithi177_31116_object15123:head)])", 2024-09-13T11:24:00.283 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:31.569662+0000", 2024-09-13T11:24:00.283 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 28.671697776999999, 2024-09-13T11:24:00.283 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.35759148299999999, 2024-09-13T11:24:00.283 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:24:00.283 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:24:00.284 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "reached pg", 2024-09-13T11:24:00.284 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:24:00.284 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:00.284 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:24:00.284 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:31.569662+0000", 2024-09-13T11:24:00.284 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:00.284 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:00.284 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:00.284 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:24:00.284 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:31.569662+0000", 2024-09-13T11:24:00.284 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:00.284 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:00.284 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:00.284 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:24:00.285 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:31.569663+0000", 2024-09-13T11:24:00.285 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.333e-06 2024-09-13T11:24:00.285 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:00.285 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:00.285 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:24:00.285 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:31.569667+0000", 2024-09-13T11:24:00.285 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.8929999999999998e-06 2024-09-13T11:24:00.285 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:00.285 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:00.285 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:24:00.286 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:31.923221+0000", 2024-09-13T11:24:00.286 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.35355413699999999 2024-09-13T11:24:00.286 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:00.286 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:00.286 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:24:00.286 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:31.923230+0000", 2024-09-13T11:24:00.286 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.6829999999999997e-06 2024-09-13T11:24:00.286 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:00.286 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:00.286 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:24:00.286 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:31.923807+0000", 2024-09-13T11:24:00.286 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00057686800000000002 2024-09-13T11:24:00.286 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:00.286 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:00.287 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:24:00.287 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:31.927253+0000", 2024-09-13T11:24:00.287 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.003446569 2024-09-13T11:24:00.287 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:00.287 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:24:00.287 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:00.287 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:00.287 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:00.287 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "MOSDPGPushReply(3.14 76/70 [PushReplyOp(3:297981b4:::benchmark_data_smithi177_31116_object16264:head)])", 2024-09-13T11:24:00.287 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:32.122153+0000", 2024-09-13T11:24:00.287 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 28.119206087999999, 2024-09-13T11:24:00.287 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.20534369899999999, 2024-09-13T11:24:00.287 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:24:00.287 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:24:00.288 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "reached pg", 2024-09-13T11:24:00.288 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:24:00.288 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:00.288 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:24:00.288 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:32.122153+0000", 2024-09-13T11:24:00.288 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:00.288 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:00.288 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:00.288 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:24:00.288 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:32.122153+0000", 2024-09-13T11:24:00.288 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:00.288 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:00.288 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:00.289 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:24:00.289 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:32.122154+0000", 2024-09-13T11:24:00.289 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.5300000000000003e-07 2024-09-13T11:24:00.289 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:00.289 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:00.289 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:24:00.289 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:32.122158+0000", 2024-09-13T11:24:00.289 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.8770000000000003e-06 2024-09-13T11:24:00.289 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:00.289 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:00.289 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:24:00.289 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:32.326042+0000", 2024-09-13T11:24:00.289 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.20388401 2024-09-13T11:24:00.289 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:00.290 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:00.290 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:24:00.290 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:32.326049+0000", 2024-09-13T11:24:00.290 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.8369999999999999e-06 2024-09-13T11:24:00.290 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:00.290 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:00.290 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:24:00.290 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:32.326135+0000", 2024-09-13T11:24:00.291 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.6215e-05 2024-09-13T11:24:00.291 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:00.291 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:00.291 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:24:00.291 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:32.327497+0000", 2024-09-13T11:24:00.291 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0013619070000000001 2024-09-13T11:24:00.291 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:00.291 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:24:00.291 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:00.291 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:00.291 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:00.291 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:55670 3.14 3:2e43484c:::benchmark_data_smithi177_31116_object55669:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e79)", 2024-09-13T11:24:00.291 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:35.763374+0000", 2024-09-13T11:24:00.291 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 24.477985865000001, 2024-09-13T11:24:00.292 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.18805201299999999, 2024-09-13T11:24:00.292 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:24:00.292 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:24:00.292 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:00.292 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:24:00.292 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:24:00.292 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:00.292 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 55670 2024-09-13T11:24:00.292 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:00.292 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:24:00.292 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:00.293 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:24:00.293 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:35.763374+0000", 2024-09-13T11:24:00.293 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:00.293 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:00.293 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:00.293 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:24:00.293 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:35.763374+0000", 2024-09-13T11:24:00.293 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:00.293 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:00.294 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:00.294 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:24:00.294 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:35.763376+0000", 2024-09-13T11:24:00.294 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.1239999999999999e-06 2024-09-13T11:24:00.294 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:00.294 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:00.294 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:24:00.294 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:35.763428+0000", 2024-09-13T11:24:00.294 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.2553000000000003e-05 2024-09-13T11:24:00.294 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:00.294 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:00.294 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:24:00.294 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:35.763437+0000", 2024-09-13T11:24:00.295 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.4649999999999994e-06 2024-09-13T11:24:00.295 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:00.295 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:00.295 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:24:00.295 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:35.763442+0000", 2024-09-13T11:24:00.295 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.5310000000000001e-06 2024-09-13T11:24:00.295 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:00.295 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:00.295 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:24:00.295 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:35.763498+0000", 2024-09-13T11:24:00.295 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.5591999999999999e-05 2024-09-13T11:24:00.295 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:00.295 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:00.295 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:24:00.296 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:35.763660+0000", 2024-09-13T11:24:00.296 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000161965 2024-09-13T11:24:00.296 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:00.296 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:00.296 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 3,14", 2024-09-13T11:24:00.296 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:35.763850+0000", 2024-09-13T11:24:00.296 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00019013999999999999 2024-09-13T11:24:00.296 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:00.296 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:00.296 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:24:00.297 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:35.768339+0000", 2024-09-13T11:24:00.297 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0044892739999999997 2024-09-13T11:24:00.297 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:00.297 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:00.297 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:00.297 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:35.770403+0000", 2024-09-13T11:24:00.297 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0020638150000000001 2024-09-13T11:24:00.297 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:00.297 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:00.297 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:00.297 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:35.951091+0000", 2024-09-13T11:24:00.297 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.18068817500000001 2024-09-13T11:24:00.297 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:00.297 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:00.298 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:24:00.298 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:35.951146+0000", 2024-09-13T11:24:00.298 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.4537000000000003e-05 2024-09-13T11:24:00.298 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:00.298 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:00.298 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:24:00.298 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:35.951426+0000", 2024-09-13T11:24:00.298 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00027984199999999998 2024-09-13T11:24:00.298 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:00.298 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:24:00.298 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:00.298 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:00.298 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:00.298 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "MOSDPGPushReply(3.14 79/70 [PushReplyOp(3:2c86f72b:::benchmark_data_smithi177_31116_object32132:head)])", 2024-09-13T11:24:00.299 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:35.770311+0000", 2024-09-13T11:24:00.299 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 24.471048420999999, 2024-09-13T11:24:00.299 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.17980474099999999, 2024-09-13T11:24:00.299 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:24:00.299 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:24:00.299 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "reached pg", 2024-09-13T11:24:00.299 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:24:00.299 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:00.299 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:24:00.299 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:35.770311+0000", 2024-09-13T11:24:00.300 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:00.300 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:00.300 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:00.300 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:24:00.300 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:35.770311+0000", 2024-09-13T11:24:00.300 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:00.300 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:00.300 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:00.300 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:24:00.300 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:35.770312+0000", 2024-09-13T11:24:00.300 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.9399999999999993e-07 2024-09-13T11:24:00.300 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:00.301 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:00.301 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:24:00.301 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:35.770319+0000", 2024-09-13T11:24:00.301 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.635e-06 2024-09-13T11:24:00.301 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:00.301 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:00.301 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:24:00.301 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:35.949590+0000", 2024-09-13T11:24:00.301 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.17927082799999999 2024-09-13T11:24:00.301 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:00.301 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:00.301 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:24:00.301 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:35.949596+0000", 2024-09-13T11:24:00.301 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.3969999999999999e-06 2024-09-13T11:24:00.302 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:00.302 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:00.302 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:24:00.302 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:35.949666+0000", 2024-09-13T11:24:00.302 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.0087999999999997e-05 2024-09-13T11:24:00.302 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:00.302 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:00.302 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:24:00.302 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:35.950116+0000", 2024-09-13T11:24:00.302 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00044979900000000002 2024-09-13T11:24:00.302 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:00.302 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:24:00.302 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:00.302 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:00.303 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:00.303 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop_reply(client.4665.0:55670 3.14 e79/70 ondisk, result = 0)", 2024-09-13T11:24:00.303 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:35.774267+0000", 2024-09-13T11:24:00.303 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 24.467092311999998, 2024-09-13T11:24:00.303 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.177164408, 2024-09-13T11:24:00.303 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:24:00.303 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:24:00.303 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-09-13T11:24:00.303 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:24:00.303 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:00.303 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:24:00.303 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:35.774267+0000", 2024-09-13T11:24:00.303 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:00.304 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:00.304 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:00.304 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:24:00.304 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:35.774267+0000", 2024-09-13T11:24:00.304 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:00.304 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:00.304 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:00.304 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:24:00.304 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:35.774269+0000", 2024-09-13T11:24:00.304 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.3820000000000001e-06 2024-09-13T11:24:00.304 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:00.304 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:00.304 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:24:00.304 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:35.774276+0000", 2024-09-13T11:24:00.305 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.0520000000000004e-06 2024-09-13T11:24:00.305 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:00.305 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:00.305 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:24:00.305 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:35.951004+0000", 2024-09-13T11:24:00.305 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.17672813800000001 2024-09-13T11:24:00.305 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:00.305 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:00.305 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:24:00.305 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:35.951013+0000", 2024-09-13T11:24:00.305 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.5480000000000007e-06 2024-09-13T11:24:00.305 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:00.305 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:00.305 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:24:00.306 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:35.951076+0000", 2024-09-13T11:24:00.306 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.3046999999999995e-05 2024-09-13T11:24:00.306 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:00.306 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:00.306 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:24:00.307 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:35.951085+0000", 2024-09-13T11:24:00.307 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.3950000000000001e-06 2024-09-13T11:24:00.307 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:00.307 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:00.307 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:24:00.307 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:35.951432+0000", 2024-09-13T11:24:00.307 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00034684599999999999 2024-09-13T11:24:00.307 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:00.307 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:24:00.308 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:00.308 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:00.308 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:00.308 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "MOSDPGPush(3.19 83/70 [PushOp(3:9863b4ba:::benchmark_data_smithi177_31116_object45096:head, version: 68'2794, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:9863b4ba:::benchmark_data_smithi177_31116_object45096:head@68'2794, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:24:00.308 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:40.671028+0000", 2024-09-13T11:24:00.308 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 19.570331181, 2024-09-13T11:24:00.308 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.22869726100000001, 2024-09-13T11:24:00.308 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:24:00.308 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:24:00.308 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-09-13T11:24:00.308 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:24:00.308 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:00.308 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:24:00.309 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:40.671028+0000", 2024-09-13T11:24:00.309 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:00.309 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:00.309 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:00.309 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:24:00.309 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:40.671028+0000", 2024-09-13T11:24:00.309 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:00.309 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:00.309 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:00.309 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:24:00.309 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:40.671029+0000", 2024-09-13T11:24:00.309 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.2200000000000003e-07 2024-09-13T11:24:00.309 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:00.309 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:00.310 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:24:00.310 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:40.671098+0000", 2024-09-13T11:24:00.310 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.9241999999999999e-05 2024-09-13T11:24:00.310 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:00.310 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:00.310 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:24:00.310 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:40.898838+0000", 2024-09-13T11:24:00.310 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.22773932699999999 2024-09-13T11:24:00.310 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:00.310 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:00.310 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:24:00.310 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:40.898848+0000", 2024-09-13T11:24:00.310 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0373999999999999e-05 2024-09-13T11:24:00.310 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:00.311 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:00.311 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:24:00.311 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:40.898930+0000", 2024-09-13T11:24:00.311 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.1865000000000003e-05 2024-09-13T11:24:00.311 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:00.311 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:00.311 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:24:00.311 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:40.898938+0000", 2024-09-13T11:24:00.311 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.6109999999999996e-06 2024-09-13T11:24:00.311 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:00.311 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:00.311 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:24:00.311 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:40.899726+0000", 2024-09-13T11:24:00.311 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00078801999999999995 2024-09-13T11:24:00.312 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:00.312 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:24:00.312 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:00.312 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:00.312 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:00.312 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "MOSDPGPushReply(3.18 85/70 [PushReplyOp(3:1e410119:::benchmark_data_smithi177_31116_object35700:head)])", 2024-09-13T11:24:00.312 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:42.216614+0000", 2024-09-13T11:24:00.312 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 18.024745433, 2024-09-13T11:24:00.312 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.54260090699999997, 2024-09-13T11:24:00.312 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:24:00.312 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:24:00.312 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "reached pg", 2024-09-13T11:24:00.312 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:24:00.312 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:00.313 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:24:00.313 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.216614+0000", 2024-09-13T11:24:00.313 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:00.313 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:00.313 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:00.313 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:24:00.313 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.216614+0000", 2024-09-13T11:24:00.313 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:00.313 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:00.313 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:00.313 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:24:00.313 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.216615+0000", 2024-09-13T11:24:00.313 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.0200000000000001e-07 2024-09-13T11:24:00.313 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:00.314 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:00.314 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:24:00.314 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.216619+0000", 2024-09-13T11:24:00.314 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.805e-06 2024-09-13T11:24:00.314 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:00.314 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:00.314 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:24:00.314 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.757476+0000", 2024-09-13T11:24:00.314 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.54085762000000004 2024-09-13T11:24:00.314 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:00.314 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:00.314 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:24:00.314 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.757498+0000", 2024-09-13T11:24:00.314 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.1936000000000001e-05 2024-09-13T11:24:00.314 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:00.315 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:00.315 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:24:00.315 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.757678+0000", 2024-09-13T11:24:00.315 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00018006599999999999 2024-09-13T11:24:00.315 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:00.315 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:00.315 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:24:00.315 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.759215+0000", 2024-09-13T11:24:00.315 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0015366780000000001 2024-09-13T11:24:00.315 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:00.315 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:24:00.315 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:00.315 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:00.315 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:00.316 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "MOSDPGPushReply(3.1b 85/78 [PushReplyOp(3:dabda1cc:::benchmark_data_smithi177_31116_object12988:head)])", 2024-09-13T11:24:00.316 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:42.485171+0000", 2024-09-13T11:24:00.316 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 17.756188524999999, 2024-09-13T11:24:00.316 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.308420427, 2024-09-13T11:24:00.316 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:24:00.316 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:24:00.316 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "reached pg", 2024-09-13T11:24:00.316 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:24:00.316 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:00.316 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:24:00.316 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.485171+0000", 2024-09-13T11:24:00.316 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:00.316 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:00.316 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:00.317 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:24:00.317 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.485171+0000", 2024-09-13T11:24:00.317 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:00.317 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:00.317 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:00.317 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:24:00.317 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.485173+0000", 2024-09-13T11:24:00.317 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.1569999999999998e-06 2024-09-13T11:24:00.317 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:00.317 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:00.317 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:24:00.317 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.485182+0000", 2024-09-13T11:24:00.318 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.6759999999999996e-06 2024-09-13T11:24:00.318 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:00.318 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:00.318 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:24:00.318 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.791336+0000", 2024-09-13T11:24:00.318 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.30615446499999999 2024-09-13T11:24:00.318 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:00.318 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:00.318 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:24:00.318 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.791369+0000", 2024-09-13T11:24:00.318 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.3155999999999999e-05 2024-09-13T11:24:00.318 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:00.319 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:00.319 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:24:00.319 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.791472+0000", 2024-09-13T11:24:00.319 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00010247900000000001 2024-09-13T11:24:00.319 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:00.319 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:00.319 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:24:00.319 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.793591+0000", 2024-09-13T11:24:00.319 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.002119494 2024-09-13T11:24:00.319 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:00.319 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:24:00.319 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:00.319 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:00.319 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:00.320 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:64066 3.1b 3:dd84e0ec:::benchmark_data_smithi177_31116_object64065:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89)", 2024-09-13T11:24:00.320 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:46.633608+0000", 2024-09-13T11:24:00.320 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 13.607751086, 2024-09-13T11:24:00.320 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.32564315399999999, 2024-09-13T11:24:00.320 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:24:00.320 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:24:00.320 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:00.320 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:24:00.320 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:24:00.320 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:00.320 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 64066 2024-09-13T11:24:00.320 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:00.320 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:24:00.320 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:00.321 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:24:00.321 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.633608+0000", 2024-09-13T11:24:00.321 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:00.321 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:00.321 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:00.321 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:24:00.321 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.633608+0000", 2024-09-13T11:24:00.321 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:00.321 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:00.321 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:00.321 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:24:00.321 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.633612+0000", 2024-09-13T11:24:00.321 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.856e-06 2024-09-13T11:24:00.321 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:00.322 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:00.322 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:24:00.322 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.633683+0000", 2024-09-13T11:24:00.322 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.1000999999999993e-05 2024-09-13T11:24:00.322 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:00.322 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:00.322 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:24:00.322 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.633711+0000", 2024-09-13T11:24:00.322 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.8204999999999998e-05 2024-09-13T11:24:00.322 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:00.322 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:00.322 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:24:00.322 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.633732+0000", 2024-09-13T11:24:00.322 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.0662999999999999e-05 2024-09-13T11:24:00.323 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:00.323 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:00.323 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:24:00.323 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.635573+0000", 2024-09-13T11:24:00.323 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.001841365 2024-09-13T11:24:00.323 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:00.323 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:00.323 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:24:00.323 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.635995+0000", 2024-09-13T11:24:00.323 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00042113699999999998 2024-09-13T11:24:00.323 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:00.323 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:00.323 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 12,15", 2024-09-13T11:24:00.323 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.636475+0000", 2024-09-13T11:24:00.324 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00048067699999999999 2024-09-13T11:24:00.324 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:00.324 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:00.324 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:00.324 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.640247+0000", 2024-09-13T11:24:00.324 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0037716450000000001 2024-09-13T11:24:00.324 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:00.324 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:00.324 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:24:00.324 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.649308+0000", 2024-09-13T11:24:00.324 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0090613240000000008 2024-09-13T11:24:00.324 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:00.324 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:00.325 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:00.325 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.956912+0000", 2024-09-13T11:24:00.325 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.30760338500000001 2024-09-13T11:24:00.325 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:00.325 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:00.325 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:24:00.325 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.957156+0000", 2024-09-13T11:24:00.325 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000244031 2024-09-13T11:24:00.325 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:00.325 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:00.325 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:24:00.325 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.959252+0000", 2024-09-13T11:24:00.325 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0020958650000000001 2024-09-13T11:24:00.325 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:00.326 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:24:00.326 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:00.326 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:00.326 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:00.326 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop_reply(client.4665.0:64066 3.1b e89/78 ondisk, result = 0)", 2024-09-13T11:24:00.326 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:46.651596+0000", 2024-09-13T11:24:00.326 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 13.589763254999999, 2024-09-13T11:24:00.326 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.30767612599999999, 2024-09-13T11:24:00.326 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:24:00.326 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:24:00.326 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-09-13T11:24:00.326 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:24:00.326 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:00.326 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:24:00.327 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.651596+0000", 2024-09-13T11:24:00.327 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:00.327 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:00.327 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:00.327 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:24:00.327 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.651596+0000", 2024-09-13T11:24:00.327 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:00.327 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:00.327 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:00.327 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:24:00.327 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.651600+0000", 2024-09-13T11:24:00.327 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.6969999999999999e-06 2024-09-13T11:24:00.327 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:00.327 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:00.327 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:24:00.328 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.651611+0000", 2024-09-13T11:24:00.328 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.1516e-05 2024-09-13T11:24:00.328 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:00.328 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:00.328 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:24:00.328 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.948118+0000", 2024-09-13T11:24:00.328 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.29650663500000002 2024-09-13T11:24:00.328 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:00.328 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:00.328 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:24:00.328 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.948137+0000", 2024-09-13T11:24:00.328 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.8529000000000002e-05 2024-09-13T11:24:00.328 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:00.329 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:00.329 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:24:00.329 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.956870+0000", 2024-09-13T11:24:00.329 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0087336380000000002 2024-09-13T11:24:00.329 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:00.329 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:00.329 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:24:00.329 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.956893+0000", 2024-09-13T11:24:00.329 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.317e-05 2024-09-13T11:24:00.329 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:00.329 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:00.329 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:24:00.329 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.959272+0000", 2024-09-13T11:24:00.329 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0023789409999999999 2024-09-13T11:24:00.330 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:00.330 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:24:00.330 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:00.330 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:00.330 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:00.330 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "MOSDPGPushReply(3.1b 91/78 [PushReplyOp(3:d9ac6889:::benchmark_data_smithi177_31116_object22254:head)])", 2024-09-13T11:24:00.330 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:48.392729+0000", 2024-09-13T11:24:00.330 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 11.84863034, 2024-09-13T11:24:00.330 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.30924993099999998, 2024-09-13T11:24:00.330 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:24:00.330 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:24:00.330 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "reached pg", 2024-09-13T11:24:00.330 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:24:00.330 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:00.331 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:24:00.331 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:48.392729+0000", 2024-09-13T11:24:00.331 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:00.331 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:00.331 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:00.331 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:24:00.331 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:48.392729+0000", 2024-09-13T11:24:00.331 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:00.331 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:00.331 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:00.331 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:24:00.331 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:48.392730+0000", 2024-09-13T11:24:00.331 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.336e-06 2024-09-13T11:24:00.332 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:00.332 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:00.332 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:24:00.332 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:48.392739+0000", 2024-09-13T11:24:00.332 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.8580000000000006e-06 2024-09-13T11:24:00.332 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:00.332 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:00.332 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:24:00.332 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:48.700180+0000", 2024-09-13T11:24:00.332 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.30744035600000003 2024-09-13T11:24:00.332 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:00.332 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:00.332 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:24:00.333 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:48.700197+0000", 2024-09-13T11:24:00.333 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.7649999999999999e-05 2024-09-13T11:24:00.333 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:00.333 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:00.333 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:24:00.333 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:48.700398+0000", 2024-09-13T11:24:00.333 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000201113 2024-09-13T11:24:00.333 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:00.333 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:00.333 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:24:00.333 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:48.701979+0000", 2024-09-13T11:24:00.333 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.001580618 2024-09-13T11:24:00.333 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:00.333 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:24:00.334 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:00.334 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:00.334 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:00.334 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop(client.4665.0:69487 3.e e94/75 3:73f7f990:::benchmark_data_smithi177_31116_object69486:head v 94'3977, mlcod=94'3973)", 2024-09-13T11:24:00.334 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:52.064811+0000", 2024-09-13T11:24:00.334 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 8.1765486450000004, 2024-09-13T11:24:00.334 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.20068219600000001, 2024-09-13T11:24:00.334 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:24:00.334 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:24:00.334 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:00.334 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:24:00.334 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:00.334 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:24:00.334 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:52.064811+0000", 2024-09-13T11:24:00.335 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:00.335 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:00.335 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:00.335 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:24:00.335 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:52.064811+0000", 2024-09-13T11:24:00.335 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:00.335 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:00.335 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:00.335 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:24:00.335 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:52.064814+0000", 2024-09-13T11:24:00.335 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.9280000000000002e-06 2024-09-13T11:24:00.335 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:00.335 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:00.335 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:24:00.335 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:52.064877+0000", 2024-09-13T11:24:00.336 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.3214999999999997e-05 2024-09-13T11:24:00.336 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:00.336 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:00.336 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:24:00.336 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:52.262874+0000", 2024-09-13T11:24:00.336 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.19799718499999999 2024-09-13T11:24:00.336 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:00.336 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:00.336 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:24:00.336 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:52.262883+0000", 2024-09-13T11:24:00.336 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.9120000000000004e-06 2024-09-13T11:24:00.336 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:00.336 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:00.336 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:24:00.337 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:52.262994+0000", 2024-09-13T11:24:00.337 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000110952 2024-09-13T11:24:00.337 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:00.337 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:00.337 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:24:00.337 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:52.263021+0000", 2024-09-13T11:24:00.337 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.6755999999999999e-05 2024-09-13T11:24:00.337 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:00.337 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:00.337 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:24:00.337 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:52.265387+0000", 2024-09-13T11:24:00.337 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.002366085 2024-09-13T11:24:00.337 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:00.337 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:00.338 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:24:00.338 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:52.265493+0000", 2024-09-13T11:24:00.338 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00010616300000000001 2024-09-13T11:24:00.338 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:00.338 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:24:00.338 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:00.338 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:00.338 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:24:00.338 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-09-13T11:24:00.339 INFO:tasks.ceph.osd.9.smithi092.stderr:2024-09-13T11:24:00.239+0000 7ff59bdf5640 -1 osd.9 101 handle_osd_map injecting map crc failure 2024-09-13T11:24:00.340 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-13T11:24:00.341 INFO:tasks.ceph.osd.11.smithi177.stderr:2024-09-13T11:24:00.332+0000 7f24792c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26580) UID: 0 2024-09-13T11:24:00.441 INFO:tasks.ceph.osd.12.smithi001.stderr:2024-09-13T11:24:00.436+0000 7f892c6e6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26314) UID: 0 2024-09-13T11:24:00.473 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-13T11:24:00.473 INFO:teuthology.orchestra.run.smithi001.stdout: "ops": [], 2024-09-13T11:24:00.473 INFO:teuthology.orchestra.run.smithi001.stdout: "num_ops": 0 2024-09-13T11:24:00.473 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-13T11:24:00.488 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-13T11:24:00.507 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T11:24:00.542 INFO:tasks.ceph.osd.10.smithi162.stderr:2024-09-13T11:24:00.537+0000 7fb59c9f0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26317) UID: 0 2024-09-13T11:24:00.643 INFO:tasks.ceph.osd.5.smithi092.stderr:2024-09-13T11:24:00.639+0000 7f6f0b705640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26346) UID: 0 2024-09-13T11:24:00.654 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-13T11:24:00.654 INFO:teuthology.orchestra.run.smithi001.stdout: "ops": [], 2024-09-13T11:24:00.654 INFO:teuthology.orchestra.run.smithi001.stdout: "complaint_time": 30, 2024-09-13T11:24:00.654 INFO:teuthology.orchestra.run.smithi001.stdout: "num_blocked_ops": 0 2024-09-13T11:24:00.654 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-13T11:24:00.671 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-13T11:24:00.687 INFO:teuthology.orchestra.run.smithi001.stdout:osd.0: {} 2024-09-13T11:24:00.687 INFO:teuthology.orchestra.run.smithi001.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T11:24:00.712 INFO:teuthology.orchestra.run.smithi001.stdout:osd.1: {} 2024-09-13T11:24:00.713 INFO:teuthology.orchestra.run.smithi001.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T11:24:00.739 INFO:teuthology.orchestra.run.smithi001.stdout:osd.2: {} 2024-09-13T11:24:00.739 INFO:teuthology.orchestra.run.smithi001.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T11:24:00.743 INFO:tasks.ceph.osd.12.smithi001.stderr:2024-09-13T11:24:00.736+0000 7f892c6e6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26314) UID: 0 2024-09-13T11:24:00.764 INFO:teuthology.orchestra.run.smithi001.stdout:osd.3: {} 2024-09-13T11:24:00.764 INFO:teuthology.orchestra.run.smithi001.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T11:24:00.787 INFO:teuthology.orchestra.run.smithi001.stdout:osd.4: {} 2024-09-13T11:24:00.788 INFO:teuthology.orchestra.run.smithi001.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T11:24:00.792 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-13T11:24:00.792 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:24:00.807 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-13T11:24:00.815 INFO:teuthology.orchestra.run.smithi001.stdout:osd.5: {} 2024-09-13T11:24:00.815 INFO:teuthology.orchestra.run.smithi001.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T11:24:00.843 INFO:teuthology.orchestra.run.smithi001.stdout:osd.6: {} 2024-09-13T11:24:00.844 INFO:teuthology.orchestra.run.smithi001.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T11:24:00.844 INFO:tasks.ceph.osd.1.smithi092.stderr:2024-09-13T11:24:00.839+0000 7fe7789c5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33718) UID: 0 2024-09-13T11:24:00.869 INFO:teuthology.orchestra.run.smithi001.stdout:osd.7: {} 2024-09-13T11:24:00.869 INFO:teuthology.orchestra.run.smithi001.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T11:24:00.891 INFO:teuthology.orchestra.run.smithi001.stdout:osd.8: {} 2024-09-13T11:24:00.891 INFO:teuthology.orchestra.run.smithi001.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-13T11:24:00.917 INFO:teuthology.orchestra.run.smithi001.stdout:osd.9: {} 2024-09-13T11:24:00.917 INFO:teuthology.orchestra.run.smithi001.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-13T11:24:00.936 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-13T11:24:00.936 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:24:00.944 INFO:tasks.ceph.osd.4.smithi001.stderr:2024-09-13T11:24:00.940+0000 7ff8e0218640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26310) UID: 0 2024-09-13T11:24:00.947 INFO:teuthology.orchestra.run.smithi001.stdout:osd.10: {} 2024-09-13T11:24:00.947 INFO:teuthology.orchestra.run.smithi001.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-13T11:24:00.951 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-13T11:24:00.977 INFO:teuthology.orchestra.run.smithi001.stdout:osd.11: {} 2024-09-13T11:24:00.977 INFO:teuthology.orchestra.run.smithi001.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-13T11:24:01.007 INFO:tasks.radosbench.radosbench.0.smithi177.stdout: 94 16 79857 79841 53.0803 27.875 0.00483039 0.0187218 2024-09-13T11:24:01.009 INFO:teuthology.orchestra.run.smithi001.stdout:osd.12: {} 2024-09-13T11:24:01.009 INFO:teuthology.orchestra.run.smithi001.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-13T11:24:01.018 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-13T11:24:01.039 INFO:teuthology.orchestra.run.smithi001.stdout:osd.13: {} 2024-09-13T11:24:01.039 INFO:teuthology.orchestra.run.smithi001.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-13T11:24:01.091 INFO:teuthology.orchestra.run.smithi001.stdout:osd.14: {} 2024-09-13T11:24:01.091 INFO:teuthology.orchestra.run.smithi001.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-13T11:24:01.097 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-13T11:24:01.097 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:24:01.103 INFO:tasks.ceph.osd.5.smithi092.stderr:2024-09-13T11:24:01.043+0000 7f6f0b705640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26346) UID: 0 2024-09-13T11:24:01.113 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-13T11:24:01.123 INFO:teuthology.orchestra.run.smithi001.stdout:osd.15: {} 2024-09-13T11:24:01.123 INFO:teuthology.orchestra.run.smithi001.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-13T11:24:01.146 INFO:tasks.ceph.osd.4.smithi001.stderr:2024-09-13T11:24:01.140+0000 7ff8e0218640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26310) UID: 0 2024-09-13T11:24:01.237 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-13T11:24:01.237 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:24:01.247 INFO:tasks.ceph.osd.8.smithi001.stderr:2024-09-13T11:24:01.240+0000 7fb9b48ae640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26312) UID: 0 2024-09-13T11:24:01.252 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-09-13T11:24:01.259 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-13T11:24:01.348 INFO:tasks.ceph.osd.12.smithi001.stderr:2024-09-13T11:24:01.344+0000 7f892c6e6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26314) UID: 0 2024-09-13T11:24:01.352 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-13T11:24:01.353 INFO:teuthology.orchestra.run.smithi001.stderr:dumped all 2024-09-13T11:24:01.385 INFO:teuthology.orchestra.run.smithi001.stdout:{"pg_ready":true,"pg_map":{"version":144,"stamp":"2024-09-13T11:24:01.240894+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":5141004851,"num_objects":78442,"num_object_clones":0,"num_object_copies":156884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2074,"num_objects_unfound":0,"num_objects_dirty":78442,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":34600,"num_write_kb":2207570,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12322,"num_bytes_recovered":807436307,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":138994,"ondisk_log_size":138994,"up":90,"acting":90,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":118,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":16,"kb":1509949440,"kb_used":13221012,"kb_used_data":12346388,"kb_used_omap":27,"kb_used_meta":874276,"kb_avail":1496728428,"statfs":{"total":1546188226560,"available":1532649910272,"internally_reserved":0,"allocated":12642701312,"data_stored":12635642938,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28358,"internal_metadata":895258938},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[6,5,1,2,0,0,0,1,0,1],"upper_bound":1024},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":402259968,"num_objects":6138,"num_object_clones":0,"num_object_copies":12276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-4686,"num_objects_unfound":0,"num_objects_dirty":6138,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6138,"num_write_kb":392832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2437,"num_bytes_recovered":159711232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.033469"},"pg_stats":[{"pgid":"3.23","version":"102'3545","reported_seq":6265,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-13T11:24:00.318559+0000","last_change":"2024-09-13T11:23:48.133184+0000","last_active":"2024-09-13T11:24:00.318559+0000","last_peered":"2024-09-13T11:24:00.318559+0000","last_clean":"2024-09-13T11:24:00.318559+0000","last_became_active":"2024-09-13T11:23:47.926677+0000","last_became_peered":"2024-09-13T11:23:47.926677+0000","last_unstale":"2024-09-13T11:24:00.318559+0000","last_undegraded":"2024-09-13T11:24:00.318559+0000","last_fullsized":"2024-09-13T11:24:00.318559+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":6,"last_scrub":"91'3278","last_scrub_stamp":"2024-09-13T11:23:48.133084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:48.133084+0000","objects_scrubbed":1029,"log_size":3545,"log_dups_size":0,"ondisk_log_size":3545,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:12.874134+0000","scrub_duration":139,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":85000192,"num_objects":1297,"num_object_clones":0,"num_object_copies":2594,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1297,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":268,"num_write_kb":17152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9],"acting":[5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.22","version":"102'3485","reported_seq":5392,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-13T11:24:00.410951+0000","last_change":"2024-09-13T11:23:47.300446+0000","last_active":"2024-09-13T11:24:00.410951+0000","last_peered":"2024-09-13T11:24:00.410951+0000","last_clean":"2024-09-13T11:24:00.410951+0000","last_became_active":"2024-09-13T11:23:46.925318+0000","last_became_peered":"2024-09-13T11:23:46.925318+0000","last_unstale":"2024-09-13T11:24:00.410951+0000","last_undegraded":"2024-09-13T11:24:00.410951+0000","last_fullsized":"2024-09-13T11:24:00.410951+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":6,"last_scrub":"90'3246","last_scrub_stamp":"2024-09-13T11:23:47.300339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:47.300339+0000","objects_scrubbed":1033,"log_size":3485,"log_dups_size":0,"ondisk_log_size":3485,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:24:49.780485+0000","scrub_duration":203,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":83492864,"num_objects":1274,"num_object_clones":0,"num_object_copies":2548,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1274,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":241,"num_write_kb":15424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,14],"acting":[0,14],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.21","version":"101'3436","reported_seq":6136,"reported_epoch":102,"state":"peering","last_fresh":"2024-09-13T11:24:00.239108+0000","last_change":"2024-09-13T11:24:00.239108+0000","last_active":"2024-09-13T11:24:00.239108+0000","last_peered":"2024-09-13T11:24:00.144417+0000","last_clean":"2024-09-13T11:23:32.050222+0000","last_became_active":"2024-09-13T11:23:35.026171+0000","last_became_peered":"2024-09-13T11:23:35.026171+0000","last_unstale":"2024-09-13T11:24:00.239108+0000","last_undegraded":"2024-09-13T11:24:00.239108+0000","last_fullsized":"2024-09-13T11:24:00.239108+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":6,"last_scrub":"69'2784","last_scrub_stamp":"2024-09-13T11:23:21.595419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:21.595419+0000","objects_scrubbed":1386,"log_size":3436,"log_dups_size":0,"ondisk_log_size":3436,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":311,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":78905344,"num_objects":1204,"num_object_clones":0,"num_object_copies":2408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1204,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":511,"num_write_kb":32704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":836,"num_bytes_recovered":54788096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8],"acting":[0,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.20","version":"102'3514","reported_seq":5536,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-13T11:24:00.246567+0000","last_change":"2024-09-13T11:23:47.969503+0000","last_active":"2024-09-13T11:24:00.246567+0000","last_peered":"2024-09-13T11:24:00.246567+0000","last_clean":"2024-09-13T11:24:00.246567+0000","last_became_active":"2024-09-13T11:23:33.954946+0000","last_became_peered":"2024-09-13T11:23:33.954946+0000","last_unstale":"2024-09-13T11:24:00.246567+0000","last_undegraded":"2024-09-13T11:24:00.246567+0000","last_fullsized":"2024-09-13T11:24:00.246567+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":6,"last_scrub":"91'3270","last_scrub_stamp":"2024-09-13T11:23:47.969372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:47.969372+0000","objects_scrubbed":1061,"log_size":3514,"log_dups_size":0,"ondisk_log_size":3514,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:24:48.255263+0000","scrub_duration":1327,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":85852160,"num_objects":1310,"num_object_clones":0,"num_object_copies":2620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1310,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":249,"num_write_kb":15936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,0],"acting":[12,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[]},{"pgid":"3.1f","version":"100'4032","reported_seq":7143,"reported_epoch":100,"state":"active+recovering+remapped","last_fresh":"2024-09-13T11:23:57.059107+0000","last_change":"2024-09-13T11:23:37.329722+0000","last_active":"2024-09-13T11:23:57.059107+0000","last_peered":"2024-09-13T11:23:57.059107+0000","last_clean":"2024-09-13T11:23:32.316727+0000","last_became_active":"2024-09-13T11:23:35.053803+0000","last_became_peered":"2024-09-13T11:23:35.053803+0000","last_unstale":"2024-09-13T11:23:57.059107+0000","last_undegraded":"2024-09-13T11:23:57.059107+0000","last_fullsized":"2024-09-13T11:23:57.059107+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:22:24.896055+0000","objects_scrubbed":0,"log_size":4032,"log_dups_size":0,"ondisk_log_size":4032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":152403968,"num_objects":2326,"num_object_clones":0,"num_object_copies":4652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1336,"num_objects_unfound":0,"num_objects_dirty":2326,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2326,"num_write_kb":148832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":983,"num_bytes_recovered":64421888,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,14],"acting":[11,10],"avail_no_missing":["11","12"],"object_location_counts":[{"shards":"10,11","objects":1658},{"shards":"11,12","objects":668}],"blocked_by":[],"up_primary":10,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1e","version":"101'4289","reported_seq":14630,"reported_epoch":102,"state":"creating+peering","last_fresh":"2024-09-13T11:24:00.236853+0000","last_change":"2024-09-13T11:24:01.002570+0000","last_active":"2024-09-13T11:24:01.002570+0000","last_peered":"2024-09-13T11:24:00.236853+0000","last_clean":"2024-09-13T11:23:30.974769+0000","last_became_active":"2024-09-13T11:23:35.075137+0000","last_became_peered":"2024-09-13T11:23:35.075137+0000","last_unstale":"2024-09-13T11:24:00.236853+0000","last_undegraded":"2024-09-13T11:24:00.236853+0000","last_fullsized":"2024-09-13T11:24:00.236853+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:22:24.896055+0000","objects_scrubbed":0,"log_size":4289,"log_dups_size":0,"ondisk_log_size":4289,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":165740544,"num_objects":2529,"num_object_clones":0,"num_object_copies":5058,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2529,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2529,"num_write_kb":161856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2573,"num_bytes_recovered":168591360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"101'4115","reported_seq":9379,"reported_epoch":102,"state":"peering","last_fresh":"2024-09-13T11:24:00.240669+0000","last_change":"2024-09-13T11:24:00.240669+0000","last_active":"2024-09-13T11:24:00.240669+0000","last_peered":"2024-09-13T11:24:00.209888+0000","last_clean":"2024-09-13T11:23:32.393844+0000","last_became_active":"2024-09-13T11:23:35.080149+0000","last_became_peered":"2024-09-13T11:23:35.080149+0000","last_unstale":"2024-09-13T11:24:00.240669+0000","last_undegraded":"2024-09-13T11:24:00.240669+0000","last_fullsized":"2024-09-13T11:24:00.240669+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:22:24.896055+0000","objects_scrubbed":0,"log_size":4115,"log_dups_size":0,"ondisk_log_size":4115,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":158859264,"num_objects":2424,"num_object_clones":0,"num_object_copies":4848,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2424,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2424,"num_write_kb":155136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1700,"num_bytes_recovered":111411200,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,15],"acting":[2,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1c","version":"100'4121","reported_seq":7755,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-09-13T11:23:57.015597+0000","last_change":"2024-09-13T11:23:55.224527+0000","last_active":"2024-09-13T11:23:57.015597+0000","last_peered":"2024-09-13T11:23:57.015597+0000","last_clean":"2024-09-13T11:23:57.015597+0000","last_became_active":"2024-09-13T11:23:33.989492+0000","last_became_peered":"2024-09-13T11:23:33.989492+0000","last_unstale":"2024-09-13T11:23:57.015597+0000","last_undegraded":"2024-09-13T11:23:57.015597+0000","last_fullsized":"2024-09-13T11:23:57.015597+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:22:24.896055+0000","objects_scrubbed":0,"log_size":4121,"log_dups_size":0,"ondisk_log_size":4121,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-13T11:24:00.213643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":156925952,"num_objects":2395,"num_object_clones":0,"num_object_copies":4790,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2395,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2395,"num_write_kb":153248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1733,"num_bytes_recovered":113573888,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7],"acting":[11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1b","version":"102'4216","reported_seq":9052,"reported_epoch":102,"state":"active+recovering+remapped","last_fresh":"2024-09-13T11:24:00.295969+0000","last_change":"2024-09-13T11:23:35.097804+0000","last_active":"2024-09-13T11:24:00.295969+0000","last_peered":"2024-09-13T11:24:00.295969+0000","last_clean":"2024-09-13T11:23:32.309147+0000","last_became_active":"2024-09-13T11:23:35.080340+0000","last_became_peered":"2024-09-13T11:23:35.080340+0000","last_unstale":"2024-09-13T11:24:00.295969+0000","last_undegraded":"2024-09-13T11:24:00.295969+0000","last_fullsized":"2024-09-13T11:24:00.295969+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:22:24.896055+0000","objects_scrubbed":0,"log_size":4216,"log_dups_size":0,"ondisk_log_size":4216,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":162889728,"num_objects":2486,"num_object_clones":0,"num_object_copies":4972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":407,"num_objects_unfound":0,"num_objects_dirty":2486,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2486,"num_write_kb":159072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1556,"num_bytes_recovered":101974016,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,15],"acting":[1,12],"avail_no_missing":["1","13"],"object_location_counts":[{"shards":"1,12","objects":2282},{"shards":"1,12,13","objects":1},{"shards":"1,13","objects":203}],"blocked_by":[],"up_primary":12,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1a","version":"101'4113","reported_seq":8724,"reported_epoch":101,"state":"active+recovering+remapped","last_fresh":"2024-09-13T11:24:00.027277+0000","last_change":"2024-09-13T11:23:35.089105+0000","last_active":"2024-09-13T11:24:00.027277+0000","last_peered":"2024-09-13T11:24:00.027277+0000","last_clean":"2024-09-13T11:23:32.455463+0000","last_became_active":"2024-09-13T11:23:35.072595+0000","last_became_peered":"2024-09-13T11:23:35.072595+0000","last_unstale":"2024-09-13T11:24:00.027277+0000","last_undegraded":"2024-09-13T11:24:00.027277+0000","last_fullsized":"2024-09-13T11:24:00.027277+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:22:24.896055+0000","objects_scrubbed":0,"log_size":4113,"log_dups_size":0,"ondisk_log_size":4113,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":157745152,"num_objects":2407,"num_object_clones":0,"num_object_copies":4814,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":331,"num_objects_unfound":0,"num_objects_dirty":2407,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2407,"num_write_kb":154048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1482,"num_bytes_recovered":97124352,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9],"acting":[3,4],"avail_no_missing":["3","7"],"object_location_counts":[{"shards":"3,4","objects":2241},{"shards":"3,7","objects":165},{"shards":"3,7,9","objects":1}],"blocked_by":[],"up_primary":4,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.19","version":"100'3700","reported_seq":7991,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-09-13T11:23:57.006328+0000","last_change":"2024-09-13T11:23:49.429522+0000","last_active":"2024-09-13T11:23:57.006328+0000","last_peered":"2024-09-13T11:23:57.006328+0000","last_clean":"2024-09-13T11:23:57.006328+0000","last_became_active":"2024-09-13T11:23:41.924941+0000","last_became_peered":"2024-09-13T11:23:41.924941+0000","last_unstale":"2024-09-13T11:23:57.006328+0000","last_undegraded":"2024-09-13T11:23:57.006328+0000","last_fullsized":"2024-09-13T11:23:57.006328+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":5,"last_scrub":"92'3415","last_scrub_stamp":"2024-09-13T11:23:49.429429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:49.429429+0000","objects_scrubbed":1979,"log_size":3700,"log_dups_size":0,"ondisk_log_size":3700,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:04.629542+0000","scrub_duration":587,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":149225472,"num_objects":2277,"num_object_clones":0,"num_object_copies":4554,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2277,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":298,"num_write_kb":19072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1],"acting":[9,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.18","version":"102'3900","reported_seq":8460,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-13T11:24:00.656686+0000","last_change":"2024-09-13T11:23:56.067740+0000","last_active":"2024-09-13T11:24:00.656686+0000","last_peered":"2024-09-13T11:24:00.656686+0000","last_clean":"2024-09-13T11:24:00.656686+0000","last_became_active":"2024-09-13T11:23:48.926014+0000","last_became_peered":"2024-09-13T11:23:48.926014+0000","last_unstale":"2024-09-13T11:24:00.656686+0000","last_undegraded":"2024-09-13T11:24:00.656686+0000","last_fullsized":"2024-09-13T11:24:00.656686+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:22:24.896055+0000","objects_scrubbed":0,"log_size":3900,"log_dups_size":0,"ondisk_log_size":3900,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-13T11:24:01.063698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":166002688,"num_objects":2533,"num_object_clones":0,"num_object_copies":5066,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2533,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2533,"num_write_kb":162112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1458,"num_bytes_recovered":95551488,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0],"acting":[10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.17","version":"102'3796","reported_seq":7826,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-13T11:24:00.298569+0000","last_change":"2024-09-13T11:23:49.349241+0000","last_active":"2024-09-13T11:24:00.298569+0000","last_peered":"2024-09-13T11:24:00.298569+0000","last_clean":"2024-09-13T11:24:00.298569+0000","last_became_active":"2024-09-13T11:23:42.935601+0000","last_became_peered":"2024-09-13T11:23:42.935601+0000","last_unstale":"2024-09-13T11:24:00.298569+0000","last_undegraded":"2024-09-13T11:24:00.298569+0000","last_fullsized":"2024-09-13T11:24:00.298569+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":5,"last_scrub":"92'3359","last_scrub_stamp":"2024-09-13T11:23:49.349167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:49.349167+0000","objects_scrubbed":2004,"log_size":3796,"log_dups_size":0,"ondisk_log_size":3796,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:24:59.401302+0000","scrub_duration":363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":160497664,"num_objects":2449,"num_object_clones":0,"num_object_copies":4898,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2449,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":445,"num_write_kb":28480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3],"acting":[10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"100'3873","reported_seq":8528,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-09-13T11:23:56.979860+0000","last_change":"2024-09-13T11:23:46.510544+0000","last_active":"2024-09-13T11:23:56.979860+0000","last_peered":"2024-09-13T11:23:56.979860+0000","last_clean":"2024-09-13T11:23:56.979860+0000","last_became_active":"2024-09-13T11:23:46.105237+0000","last_became_peered":"2024-09-13T11:23:46.105237+0000","last_unstale":"2024-09-13T11:23:56.979860+0000","last_undegraded":"2024-09-13T11:23:56.979860+0000","last_fullsized":"2024-09-13T11:23:56.979860+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":5,"last_scrub":"89'3478","last_scrub_stamp":"2024-09-13T11:23:46.510384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:46.510384+0000","objects_scrubbed":2054,"log_size":3873,"log_dups_size":0,"ondisk_log_size":3873,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:09.149878+0000","scrub_duration":363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":160694272,"num_objects":2452,"num_object_clones":0,"num_object_copies":4904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2452,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":398,"num_write_kb":25472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,15],"acting":[11,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.15","version":"102'3821","reported_seq":8240,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-13T11:24:00.357761+0000","last_change":"2024-09-13T11:23:48.332790+0000","last_active":"2024-09-13T11:24:00.357761+0000","last_peered":"2024-09-13T11:24:00.357761+0000","last_clean":"2024-09-13T11:24:00.357761+0000","last_became_active":"2024-09-13T11:23:42.931149+0000","last_became_peered":"2024-09-13T11:23:42.931149+0000","last_unstale":"2024-09-13T11:24:00.357761+0000","last_undegraded":"2024-09-13T11:24:00.357761+0000","last_fullsized":"2024-09-13T11:24:00.357761+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":5,"last_scrub":"91'3384","last_scrub_stamp":"2024-09-13T11:23:48.332600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:48.332600+0000","objects_scrubbed":2032,"log_size":3821,"log_dups_size":0,"ondisk_log_size":3821,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:24:58.239508+0000","scrub_duration":235,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":161873920,"num_objects":2470,"num_object_clones":0,"num_object_copies":4940,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2470,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":438,"num_write_kb":28032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,0],"acting":[14,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"3.14","version":"101'3838","reported_seq":8075,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-09-13T11:24:00.009923+0000","last_change":"2024-09-13T11:23:48.697100+0000","last_active":"2024-09-13T11:24:00.009923+0000","last_peered":"2024-09-13T11:24:00.009923+0000","last_clean":"2024-09-13T11:24:00.009923+0000","last_became_active":"2024-09-13T11:23:41.920989+0000","last_became_peered":"2024-09-13T11:23:41.920989+0000","last_unstale":"2024-09-13T11:24:00.009923+0000","last_undegraded":"2024-09-13T11:24:00.009923+0000","last_fullsized":"2024-09-13T11:24:00.009923+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":5,"last_scrub":"91'3371","last_scrub_stamp":"2024-09-13T11:23:48.696960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:48.696960+0000","objects_scrubbed":1935,"log_size":3838,"log_dups_size":0,"ondisk_log_size":3838,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:02.861530+0000","scrub_duration":735,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":158072832,"num_objects":2412,"num_object_clones":0,"num_object_copies":4824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2412,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":477,"num_write_kb":30528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,14],"acting":[3,14],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":167,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-13T11:24:00.248208+0000","last_change":"2024-09-13T11:22:16.889130+0000","last_active":"2024-09-13T11:24:00.248208+0000","last_peered":"2024-09-13T11:24:00.248208+0000","last_clean":"2024-09-13T11:24:00.248208+0000","last_became_active":"2024-09-13T11:22:14.192897+0000","last_became_peered":"2024-09-13T11:22:14.192897+0000","last_unstale":"2024-09-13T11:24:00.248208+0000","last_undegraded":"2024-09-13T11:24:00.248208+0000","last_fullsized":"2024-09-13T11:24:00.248208+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:22:13.113563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00084655899999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,10],"acting":[15,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"3.6","version":"100'3859","reported_seq":4039,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-09-13T11:23:57.026575+0000","last_change":"2024-09-13T11:23:22.656608+0000","last_active":"2024-09-13T11:23:57.026575+0000","last_peered":"2024-09-13T11:23:57.026575+0000","last_clean":"2024-09-13T11:23:57.026575+0000","last_became_active":"2024-09-13T11:23:19.939432+0000","last_became_peered":"2024-09-13T11:23:19.939432+0000","last_unstale":"2024-09-13T11:23:57.026575+0000","last_undegraded":"2024-09-13T11:23:57.026575+0000","last_fullsized":"2024-09-13T11:23:57.026575+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'2909","last_scrub_stamp":"2024-09-13T11:23:22.656323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:22.656323+0000","objects_scrubbed":1487,"log_size":3859,"log_dups_size":0,"ondisk_log_size":3859,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:24:31.905185+0000","scrub_duration":323,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":159711232,"num_objects":2437,"num_object_clones":0,"num_object_copies":4874,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2437,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":950,"num_write_kb":60800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,14],"acting":[13,14],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":170,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-09-13T11:23:56.924115+0000","last_change":"2024-09-13T11:23:21.875855+0000","last_active":"2024-09-13T11:23:56.924115+0000","last_peered":"2024-09-13T11:23:56.924115+0000","last_clean":"2024-09-13T11:23:56.924115+0000","last_became_active":"2024-09-13T11:22:14.211923+0000","last_became_peered":"2024-09-13T11:22:14.211923+0000","last_unstale":"2024-09-13T11:23:56.924115+0000","last_undegraded":"2024-09-13T11:23:56.924115+0000","last_fullsized":"2024-09-13T11:23:56.924115+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:23:21.875683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:23:21.875683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:24:40.678136+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0011689529999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0],"acting":[9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"100'3691","reported_seq":3873,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-09-13T11:23:57.026051+0000","last_change":"2024-09-13T11:23:20.443418+0000","last_active":"2024-09-13T11:23:57.026051+0000","last_peered":"2024-09-13T11:23:57.026051+0000","last_clean":"2024-09-13T11:23:57.026051+0000","last_became_active":"2024-09-13T11:23:19.927680+0000","last_became_peered":"2024-09-13T11:23:19.927680+0000","last_unstale":"2024-09-13T11:23:57.026051+0000","last_undegraded":"2024-09-13T11:23:57.026051+0000","last_fullsized":"2024-09-13T11:23:57.026051+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'2736","last_scrub_stamp":"2024-09-13T11:23:20.443329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:20.443329+0000","objects_scrubbed":1369,"log_size":3691,"log_dups_size":0,"ondisk_log_size":3691,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:24:48.147625+0000","scrub_duration":183,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":152502272,"num_objects":2327,"num_object_clones":0,"num_object_copies":4654,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2327,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":958,"num_write_kb":61312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,8],"acting":[13,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":175,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-13T11:24:00.241368+0000","last_change":"2024-09-13T11:23:59.530794+0000","last_active":"2024-09-13T11:24:00.241368+0000","last_peered":"2024-09-13T11:24:00.241368+0000","last_clean":"2024-09-13T11:24:00.241368+0000","last_became_active":"2024-09-13T11:22:14.581458+0000","last_became_peered":"2024-09-13T11:22:14.581458+0000","last_unstale":"2024-09-13T11:24:00.241368+0000","last_undegraded":"2024-09-13T11:24:00.241368+0000","last_fullsized":"2024-09-13T11:24:00.241368+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:23:59.530670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:23:59.530670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:02.655631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001026047,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5],"acting":[8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"102'3860","reported_seq":4026,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-13T11:24:00.240209+0000","last_change":"2024-09-13T11:23:20.254213+0000","last_active":"2024-09-13T11:24:00.240209+0000","last_peered":"2024-09-13T11:24:00.240209+0000","last_clean":"2024-09-13T11:24:00.240209+0000","last_became_active":"2024-09-13T11:23:19.925399+0000","last_became_peered":"2024-09-13T11:23:19.925399+0000","last_unstale":"2024-09-13T11:24:00.240209+0000","last_undegraded":"2024-09-13T11:24:00.240209+0000","last_fullsized":"2024-09-13T11:24:00.240209+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'2755","last_scrub_stamp":"2024-09-13T11:23:20.254124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:20.254124+0000","objects_scrubbed":1437,"log_size":3860,"log_dups_size":0,"ondisk_log_size":3860,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:24:49.993543+0000","scrub_duration":187,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":166920192,"num_objects":2547,"num_object_clones":0,"num_object_copies":5094,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2547,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1110,"num_write_kb":71040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,12],"acting":[1,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":155,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-13T11:24:00.240262+0000","last_change":"2024-09-13T11:22:47.720313+0000","last_active":"2024-09-13T11:24:00.240262+0000","last_peered":"2024-09-13T11:24:00.240262+0000","last_clean":"2024-09-13T11:24:00.240262+0000","last_became_active":"2024-09-13T11:22:47.719636+0000","last_became_peered":"2024-09-13T11:22:47.719636+0000","last_unstale":"2024-09-13T11:24:00.240262+0000","last_undegraded":"2024-09-13T11:24:00.240262+0000","last_fullsized":"2024-09-13T11:24:00.240262+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:22:13.113563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00087476000000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,9],"acting":[1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"102'3851","reported_seq":4037,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-13T11:24:00.407158+0000","last_change":"2024-09-13T11:23:20.686446+0000","last_active":"2024-09-13T11:24:00.407158+0000","last_peered":"2024-09-13T11:24:00.407158+0000","last_clean":"2024-09-13T11:24:00.407158+0000","last_became_active":"2024-09-13T11:23:19.939695+0000","last_became_peered":"2024-09-13T11:23:19.939695+0000","last_unstale":"2024-09-13T11:24:00.407158+0000","last_undegraded":"2024-09-13T11:24:00.407158+0000","last_fullsized":"2024-09-13T11:24:00.407158+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'2739","last_scrub_stamp":"2024-09-13T11:23:20.686358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:20.686358+0000","objects_scrubbed":1386,"log_size":3851,"log_dups_size":0,"ondisk_log_size":3851,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:24:38.891737+0000","scrub_duration":271,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":163774464,"num_objects":2499,"num_object_clones":0,"num_object_copies":4998,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2499,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1113,"num_write_kb":71232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,13],"acting":[15,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"3.3","version":"102'3470","reported_seq":3679,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-13T11:24:00.365009+0000","last_change":"2024-09-13T11:23:32.147115+0000","last_active":"2024-09-13T11:24:00.365009+0000","last_peered":"2024-09-13T11:24:00.365009+0000","last_clean":"2024-09-13T11:24:00.365009+0000","last_became_active":"2024-09-13T11:23:31.930448+0000","last_became_peered":"2024-09-13T11:23:31.930448+0000","last_unstale":"2024-09-13T11:24:00.365009+0000","last_undegraded":"2024-09-13T11:24:00.365009+0000","last_fullsized":"2024-09-13T11:24:00.365009+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'3122","last_scrub_stamp":"2024-09-13T11:23:32.147021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:32.147021+0000","objects_scrubbed":876,"log_size":3470,"log_dups_size":0,"ondisk_log_size":3470,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:24:48.175541+0000","scrub_duration":99,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":80216064,"num_objects":1224,"num_object_clones":0,"num_object_copies":2448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1224,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":348,"num_write_kb":22272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,7],"acting":[10,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"28'2","reported_seq":105,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-13T11:24:00.996197+0000","last_change":"2024-09-13T11:24:00.996197+0000","last_active":"2024-09-13T11:24:00.996197+0000","last_peered":"2024-09-13T11:24:00.996197+0000","last_clean":"2024-09-13T11:24:00.996197+0000","last_became_active":"2024-09-13T11:23:12.546348+0000","last_became_peered":"2024-09-13T11:23:12.546348+0000","last_unstale":"2024-09-13T11:24:00.996197+0000","last_undegraded":"2024-09-13T11:24:00.996197+0000","last_fullsized":"2024-09-13T11:24:00.996197+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2","last_scrub_stamp":"2024-09-13T11:24:00.996028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:24:00.996028+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:12.111097+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0025826730000000002,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8],"acting":[2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"101'3481","reported_seq":3688,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-13T11:24:00.246508+0000","last_change":"2024-09-13T11:23:32.749651+0000","last_active":"2024-09-13T11:24:00.246508+0000","last_peered":"2024-09-13T11:24:00.246508+0000","last_clean":"2024-09-13T11:24:00.246508+0000","last_became_active":"2024-09-13T11:23:31.921388+0000","last_became_peered":"2024-09-13T11:23:31.921388+0000","last_unstale":"2024-09-13T11:24:00.246508+0000","last_undegraded":"2024-09-13T11:24:00.246508+0000","last_fullsized":"2024-09-13T11:24:00.246508+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'3106","last_scrub_stamp":"2024-09-13T11:23:32.749487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:32.749487+0000","objects_scrubbed":856,"log_size":3481,"log_dups_size":0,"ondisk_log_size":3481,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:24:34.709501+0000","scrub_duration":91,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":80674816,"num_objects":1231,"num_object_clones":0,"num_object_copies":2462,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1231,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":375,"num_write_kb":24000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,15],"acting":[12,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":177,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-13T11:24:00.242697+0000","last_change":"2024-09-13T11:22:47.298206+0000","last_active":"2024-09-13T11:24:00.242697+0000","last_peered":"2024-09-13T11:24:00.242697+0000","last_clean":"2024-09-13T11:24:00.242697+0000","last_became_active":"2024-09-13T11:22:47.297319+0000","last_became_peered":"2024-09-13T11:22:47.297319+0000","last_unstale":"2024-09-13T11:24:00.242697+0000","last_undegraded":"2024-09-13T11:24:00.242697+0000","last_fullsized":"2024-09-13T11:24:00.242697+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:22:13.113563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00081225399999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,1],"acting":[14,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"3.1","version":"100'3390","reported_seq":3593,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-09-13T11:23:57.003280+0000","last_change":"2024-09-13T11:23:33.604966+0000","last_active":"2024-09-13T11:23:57.003280+0000","last_peered":"2024-09-13T11:23:57.003280+0000","last_clean":"2024-09-13T11:23:57.003280+0000","last_became_active":"2024-09-13T11:23:31.926528+0000","last_became_peered":"2024-09-13T11:23:31.926528+0000","last_unstale":"2024-09-13T11:23:57.003280+0000","last_undegraded":"2024-09-13T11:23:57.003280+0000","last_fullsized":"2024-09-13T11:23:57.003280+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'3070","last_scrub_stamp":"2024-09-13T11:23:33.604792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:33.604792+0000","objects_scrubbed":842,"log_size":3390,"log_dups_size":0,"ondisk_log_size":3390,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:24:43.880038+0000","scrub_duration":119,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76152832,"num_objects":1162,"num_object_clones":0,"num_object_copies":2324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1162,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":320,"num_write_kb":20480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,11],"acting":[13,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":165,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-09-13T11:23:59.241768+0000","last_change":"2024-09-13T11:22:16.267999+0000","last_active":"2024-09-13T11:23:59.241768+0000","last_peered":"2024-09-13T11:23:59.241768+0000","last_clean":"2024-09-13T11:23:59.241768+0000","last_became_active":"2024-09-13T11:22:14.211435+0000","last_became_peered":"2024-09-13T11:22:14.211435+0000","last_unstale":"2024-09-13T11:23:59.241768+0000","last_undegraded":"2024-09-13T11:23:59.241768+0000","last_fullsized":"2024-09-13T11:23:59.241768+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:22:13.113563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010010360000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2","version":"102'3473","reported_seq":3681,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-13T11:24:00.385197+0000","last_change":"2024-09-13T11:23:32.977666+0000","last_active":"2024-09-13T11:24:00.385197+0000","last_peered":"2024-09-13T11:24:00.385197+0000","last_clean":"2024-09-13T11:24:00.385197+0000","last_became_active":"2024-09-13T11:23:31.919828+0000","last_became_peered":"2024-09-13T11:23:31.919828+0000","last_unstale":"2024-09-13T11:24:00.385197+0000","last_undegraded":"2024-09-13T11:24:00.385197+0000","last_fullsized":"2024-09-13T11:24:00.385197+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'3057","last_scrub_stamp":"2024-09-13T11:23:32.977559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:32.977559+0000","objects_scrubbed":841,"log_size":3473,"log_dups_size":0,"ondisk_log_size":3473,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:24:52.169339+0000","scrub_duration":151,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":82378752,"num_objects":1257,"num_object_clones":0,"num_object_copies":2514,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1257,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":416,"num_write_kb":26624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,12],"acting":[14,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"2.3","version":"26'1","reported_seq":169,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-13T11:24:00.243013+0000","last_change":"2024-09-13T11:22:16.547691+0000","last_active":"2024-09-13T11:24:00.243013+0000","last_peered":"2024-09-13T11:24:00.243013+0000","last_clean":"2024-09-13T11:24:00.243013+0000","last_became_active":"2024-09-13T11:22:14.146507+0000","last_became_peered":"2024-09-13T11:22:14.146507+0000","last_unstale":"2024-09-13T11:24:00.243013+0000","last_undegraded":"2024-09-13T11:24:00.243013+0000","last_fullsized":"2024-09-13T11:24:00.243013+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:22:13.113563+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00085610999999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,15],"acting":[14,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"1.0","version":"26'112","reported_seq":309,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-13T11:24:00.239283+0000","last_change":"2024-09-13T11:23:49.325249+0000","last_active":"2024-09-13T11:24:00.239283+0000","last_peered":"2024-09-13T11:24:00.239283+0000","last_clean":"2024-09-13T11:24:00.239283+0000","last_became_active":"2024-09-13T11:22:12.129721+0000","last_became_peered":"2024-09-13T11:22:12.129721+0000","last_unstale":"2024-09-13T11:24:00.239283+0000","last_undegraded":"2024-09-13T11:24:00.239283+0000","last_fullsized":"2024-09-13T11:24:00.239283+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"26'112","last_scrub_stamp":"2024-09-13T11:23:49.325175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:11.098388+0000","last_clean_scrub_stamp":"2024-09-13T11:23:49.325175+0000","objects_scrubbed":2,"log_size":112,"log_dups_size":0,"ondisk_log_size":112,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:14.422287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.8","version":"102'3815","reported_seq":4002,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-13T11:24:00.333927+0000","last_change":"2024-09-13T11:23:20.644952+0000","last_active":"2024-09-13T11:24:00.333927+0000","last_peered":"2024-09-13T11:24:00.333927+0000","last_clean":"2024-09-13T11:24:00.333927+0000","last_became_active":"2024-09-13T11:23:19.925802+0000","last_became_peered":"2024-09-13T11:23:19.925802+0000","last_unstale":"2024-09-13T11:24:00.333927+0000","last_undegraded":"2024-09-13T11:24:00.333927+0000","last_fullsized":"2024-09-13T11:24:00.333927+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'2771","last_scrub_stamp":"2024-09-13T11:23:20.644747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:20.644747+0000","objects_scrubbed":1356,"log_size":3815,"log_dups_size":0,"ondisk_log_size":3815,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:24:38.144523+0000","scrub_duration":411,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":158072832,"num_objects":2412,"num_object_clones":0,"num_object_copies":4824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2412,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1056,"num_write_kb":67584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.9","version":"102'3899","reported_seq":4084,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-13T11:24:00.701258+0000","last_change":"2024-09-13T11:23:22.033678+0000","last_active":"2024-09-13T11:24:00.701258+0000","last_peered":"2024-09-13T11:24:00.701258+0000","last_clean":"2024-09-13T11:24:00.701258+0000","last_became_active":"2024-09-13T11:23:19.935114+0000","last_became_peered":"2024-09-13T11:23:19.935114+0000","last_unstale":"2024-09-13T11:24:00.701258+0000","last_undegraded":"2024-09-13T11:24:00.701258+0000","last_fullsized":"2024-09-13T11:24:00.701258+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'2808","last_scrub_stamp":"2024-09-13T11:23:22.033572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:22.033572+0000","objects_scrubbed":1471,"log_size":3899,"log_dups_size":0,"ondisk_log_size":3899,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:24:46.595859+0000","scrub_duration":287,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":167903232,"num_objects":2562,"num_object_clones":0,"num_object_copies":5124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2562,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1091,"num_write_kb":69824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,2],"acting":[12,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[]},{"pgid":"3.a","version":"101'4184","reported_seq":4372,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-09-13T11:24:00.050180+0000","last_change":"2024-09-13T11:23:32.526150+0000","last_active":"2024-09-13T11:24:00.050180+0000","last_peered":"2024-09-13T11:24:00.050180+0000","last_clean":"2024-09-13T11:24:00.050180+0000","last_became_active":"2024-09-13T11:23:31.934170+0000","last_became_peered":"2024-09-13T11:23:31.934170+0000","last_unstale":"2024-09-13T11:24:00.050180+0000","last_undegraded":"2024-09-13T11:24:00.050180+0000","last_fullsized":"2024-09-13T11:24:00.050180+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'3418","last_scrub_stamp":"2024-09-13T11:23:32.526057+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:32.526057+0000","objects_scrubbed":1774,"log_size":4184,"log_dups_size":0,"ondisk_log_size":4184,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:24:34.770161+0000","scrub_duration":507,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":166461440,"num_objects":2540,"num_object_clones":0,"num_object_copies":5080,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2540,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":766,"num_write_kb":49024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.b","version":"100'4087","reported_seq":4272,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-09-13T11:23:56.974691+0000","last_change":"2024-09-13T11:23:32.681343+0000","last_active":"2024-09-13T11:23:56.974691+0000","last_peered":"2024-09-13T11:23:56.974691+0000","last_clean":"2024-09-13T11:23:56.974691+0000","last_became_active":"2024-09-13T11:23:31.923415+0000","last_became_peered":"2024-09-13T11:23:31.923415+0000","last_unstale":"2024-09-13T11:23:56.974691+0000","last_undegraded":"2024-09-13T11:23:56.974691+0000","last_fullsized":"2024-09-13T11:23:56.974691+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'3471","last_scrub_stamp":"2024-09-13T11:23:32.681257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:32.681257+0000","objects_scrubbed":1720,"log_size":4087,"log_dups_size":0,"ondisk_log_size":4087,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:24:58.857812+0000","scrub_duration":199,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":153092096,"num_objects":2336,"num_object_clones":0,"num_object_copies":4672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2336,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":616,"num_write_kb":39424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,1],"acting":[13,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"3.c","version":"100'4072","reported_seq":4255,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-09-13T11:23:56.933592+0000","last_change":"2024-09-13T11:23:32.524870+0000","last_active":"2024-09-13T11:23:56.933592+0000","last_peered":"2024-09-13T11:23:56.933592+0000","last_clean":"2024-09-13T11:23:56.933592+0000","last_became_active":"2024-09-13T11:23:31.922620+0000","last_became_peered":"2024-09-13T11:23:31.922620+0000","last_unstale":"2024-09-13T11:23:56.933592+0000","last_undegraded":"2024-09-13T11:23:56.933592+0000","last_fullsized":"2024-09-13T11:23:56.933592+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'3463","last_scrub_stamp":"2024-09-13T11:23:32.524776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:32.524776+0000","objects_scrubbed":1741,"log_size":4072,"log_dups_size":0,"ondisk_log_size":4072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:24:37.745747+0000","scrub_duration":235,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154009600,"num_objects":2350,"num_object_clones":0,"num_object_copies":4700,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2350,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":609,"num_write_kb":38976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,13],"acting":[11,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.d","version":"102'4172","reported_seq":4360,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-13T11:24:00.790903+0000","last_change":"2024-09-13T11:23:32.776835+0000","last_active":"2024-09-13T11:24:00.790903+0000","last_peered":"2024-09-13T11:24:00.790903+0000","last_clean":"2024-09-13T11:24:00.790903+0000","last_became_active":"2024-09-13T11:23:31.939246+0000","last_became_peered":"2024-09-13T11:23:31.939246+0000","last_unstale":"2024-09-13T11:24:00.790903+0000","last_undegraded":"2024-09-13T11:24:00.790903+0000","last_fullsized":"2024-09-13T11:24:00.790903+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'3395","last_scrub_stamp":"2024-09-13T11:23:32.776747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:32.776747+0000","objects_scrubbed":1704,"log_size":4172,"log_dups_size":0,"ondisk_log_size":4172,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:24:56.402862+0000","scrub_duration":179,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":162594816,"num_objects":2481,"num_object_clones":0,"num_object_copies":4962,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2481,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":777,"num_write_kb":49728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.e","version":"102'4304","reported_seq":9421,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-13T11:24:00.731742+0000","last_change":"2024-09-13T11:23:32.368856+0000","last_active":"2024-09-13T11:24:00.731742+0000","last_peered":"2024-09-13T11:24:00.731742+0000","last_clean":"2024-09-13T11:24:00.731742+0000","last_became_active":"2024-09-13T11:23:31.925869+0000","last_became_peered":"2024-09-13T11:23:31.925869+0000","last_unstale":"2024-09-13T11:24:00.731742+0000","last_undegraded":"2024-09-13T11:24:00.731742+0000","last_fullsized":"2024-09-13T11:24:00.731742+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'3527","last_scrub_stamp":"2024-09-13T11:23:32.368727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:32.368727+0000","objects_scrubbed":1774,"log_size":4304,"log_dups_size":0,"ondisk_log_size":4304,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:24:57.811779+0000","scrub_duration":219,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":167247872,"num_objects":2552,"num_object_clones":0,"num_object_copies":5104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2552,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":778,"num_write_kb":49792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"100'4008","reported_seq":4192,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-09-13T11:23:57.058981+0000","last_change":"2024-09-13T11:23:33.492630+0000","last_active":"2024-09-13T11:23:57.058981+0000","last_peered":"2024-09-13T11:23:57.058981+0000","last_clean":"2024-09-13T11:23:57.058981+0000","last_became_active":"2024-09-13T11:23:31.935239+0000","last_became_peered":"2024-09-13T11:23:31.935239+0000","last_unstale":"2024-09-13T11:23:57.058981+0000","last_undegraded":"2024-09-13T11:23:57.058981+0000","last_fullsized":"2024-09-13T11:23:57.058981+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'3420","last_scrub_stamp":"2024-09-13T11:23:33.492488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:33.492488+0000","objects_scrubbed":1776,"log_size":4008,"log_dups_size":0,"ondisk_log_size":4008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:24:44.851261+0000","scrub_duration":215,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154927104,"num_objects":2364,"num_object_clones":0,"num_object_copies":4728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2364,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":588,"num_write_kb":37632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,12],"acting":[11,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.10","version":"102'3954","reported_seq":8298,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-13T11:24:00.735862+0000","last_change":"2024-09-13T11:23:48.519654+0000","last_active":"2024-09-13T11:24:00.735862+0000","last_peered":"2024-09-13T11:24:00.735862+0000","last_clean":"2024-09-13T11:24:00.735862+0000","last_became_active":"2024-09-13T11:23:41.931183+0000","last_became_peered":"2024-09-13T11:23:41.931183+0000","last_unstale":"2024-09-13T11:24:00.735862+0000","last_undegraded":"2024-09-13T11:24:00.735862+0000","last_fullsized":"2024-09-13T11:24:00.735862+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":5,"last_scrub":"91'3468","last_scrub_stamp":"2024-09-13T11:23:48.519574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:48.519574+0000","objects_scrubbed":2031,"log_size":3954,"log_dups_size":0,"ondisk_log_size":3954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:24:57.342213+0000","scrub_duration":215,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":165085184,"num_objects":2519,"num_object_clones":0,"num_object_copies":5038,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2519,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":488,"num_write_kb":31232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.11","version":"102'3863","reported_seq":6985,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-13T11:24:00.234369+0000","last_change":"2024-09-13T11:23:46.102466+0000","last_active":"2024-09-13T11:24:00.234369+0000","last_peered":"2024-09-13T11:24:00.234369+0000","last_clean":"2024-09-13T11:24:00.234369+0000","last_became_active":"2024-09-13T11:23:39.063172+0000","last_became_peered":"2024-09-13T11:23:39.063172+0000","last_unstale":"2024-09-13T11:24:00.234369+0000","last_undegraded":"2024-09-13T11:24:00.234369+0000","last_fullsized":"2024-09-13T11:24:00.234369+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":5,"last_scrub":"88'3345","last_scrub_stamp":"2024-09-13T11:23:46.101973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:46.101973+0000","objects_scrubbed":2011,"log_size":3863,"log_dups_size":0,"ondisk_log_size":3863,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:08.671350+0000","scrub_duration":955,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":166002688,"num_objects":2533,"num_object_clones":0,"num_object_copies":5066,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2533,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":522,"num_write_kb":33408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11],"acting":[4,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.12","version":"102'3871","reported_seq":6917,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-13T11:24:00.530684+0000","last_change":"2024-09-13T11:23:35.111409+0000","last_active":"2024-09-13T11:24:00.530684+0000","last_peered":"2024-09-13T11:24:00.530684+0000","last_clean":"2024-09-13T11:24:00.530684+0000","last_became_active":"2024-09-13T11:23:21.998877+0000","last_became_peered":"2024-09-13T11:23:21.998877+0000","last_unstale":"2024-09-13T11:24:00.530684+0000","last_undegraded":"2024-09-13T11:24:00.530684+0000","last_fullsized":"2024-09-13T11:24:00.530684+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'3109","last_scrub_stamp":"2024-09-13T11:23:35.111320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:35.111320+0000","objects_scrubbed":1732,"log_size":3871,"log_dups_size":0,"ondisk_log_size":3871,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:03.371433+0000","scrub_duration":239,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":163446784,"num_objects":2494,"num_object_clones":0,"num_object_copies":4988,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2494,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":762,"num_write_kb":48768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,7],"acting":[14,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"3.13","version":"100'3781","reported_seq":6824,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-09-13T11:23:57.005931+0000","last_change":"2024-09-13T11:23:47.565094+0000","last_active":"2024-09-13T11:23:57.005931+0000","last_peered":"2024-09-13T11:23:57.005931+0000","last_clean":"2024-09-13T11:23:57.005931+0000","last_became_active":"2024-09-13T11:23:41.082433+0000","last_became_peered":"2024-09-13T11:23:41.082433+0000","last_unstale":"2024-09-13T11:23:57.005931+0000","last_undegraded":"2024-09-13T11:23:57.005931+0000","last_fullsized":"2024-09-13T11:23:57.005931+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'3408","last_scrub_stamp":"2024-09-13T11:23:47.564931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:47.564931+0000","objects_scrubbed":1981,"log_size":3781,"log_dups_size":0,"ondisk_log_size":3781,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:04.258231+0000","scrub_duration":471,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":155058176,"num_objects":2366,"num_object_clones":0,"num_object_copies":4732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2366,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":385,"num_write_kb":24640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7],"acting":[11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":36,"stat_sum":{"num_bytes":5140414464,"num_objects":78438,"num_object_clones":0,"num_object_copies":156876,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2074,"num_objects_unfound":0,"num_objects_dirty":78438,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":34453,"num_write_kb":2204896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12321,"num_bytes_recovered":807436288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12652445696,"data_stored":12652445696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":138879,"ondisk_log_size":138879,"up":72,"acting":72,"num_store_stats":16},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":10},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":112,"ondisk_log_size":112,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":13,"up_from":23,"seq":98784247833,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1056096,"kb_used_data":988816,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":93315744,"statfs":{"total":96636764160,"available":95555321856,"internally_reserved":0,"allocated":1012547584,"data_stored":1012106122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,1,3,4,6,8,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2],"upper_bound":2},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":9,"up_from":23,"seq":98784247832,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":374160,"kb_used_data":342976,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93997680,"statfs":{"total":96636764160,"available":96253624320,"internally_reserved":0,"allocated":351207424,"data_stored":350781440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,8,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":40,"seq":171798691856,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1328072,"kb_used_data":1251576,"kb_used_omap":4,"kb_used_meta":76411,"kb_avail":93043768,"statfs":{"total":96636764160,"available":95276818432,"internally_reserved":0,"allocated":1281613824,"data_stored":1281147899,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4433,"internal_metadata":78245551},"hb_peers":[0,2,3,6,8,9,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":21,"seq":90194313241,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":981260,"kb_used_data":932028,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":93390580,"statfs":{"total":96636764160,"available":95631953920,"internally_reserved":0,"allocated":954396672,"data_stored":953931681,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":50395591},"hb_peers":[0,2,4,7,9,10,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":21,"seq":90194313244,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":284948,"kb_used_data":245444,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94086892,"statfs":{"total":96636764160,"available":96344977408,"internally_reserved":0,"allocated":251334656,"data_stored":250865626,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40434119},"hb_peers":[0,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":22,"seq":94489280536,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1129088,"kb_used_data":1049328,"kb_used_omap":1,"kb_used_meta":79742,"kb_avail":93242752,"statfs":{"total":96636764160,"available":95480578048,"internally_reserved":0,"allocated":1074511872,"data_stored":1074116405,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":81656261},"hb_peers":[1,3,4,5,8,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":22,"seq":94489280537,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":433732,"kb_used_data":398388,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93938108,"statfs":{"total":96636764160,"available":96192622592,"internally_reserved":0,"allocated":407949312,"data_stored":407500452,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,7,8,9,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":15,"up_from":22,"seq":94489280537,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919304,"kb_used_data":847864,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":93452536,"statfs":{"total":96636764160,"available":95695396864,"internally_reserved":0,"allocated":868212736,"data_stored":867816677,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":73136582},"hb_peers":[0,3,4,5,6,7,10,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,1],"upper_bound":1024},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":22,"seq":94489280537,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":496284,"kb_used_data":469260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93875556,"statfs":{"total":96636764160,"available":96128569344,"internally_reserved":0,"allocated":480522240,"data_stored":480050731,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[3,5,7,9,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":22,"seq":94489280537,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":421856,"kb_used_data":386512,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93949984,"statfs":{"total":96636764160,"available":96204783616,"internally_reserved":0,"allocated":395788288,"data_stored":395363799,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,3,5,7,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":22,"seq":94489280537,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1205024,"kb_used_data":1137744,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":93166816,"statfs":{"total":96636764160,"available":95402819584,"internally_reserved":0,"allocated":1165049856,"data_stored":1164630327,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":68876746},"hb_peers":[0,3,5,6,8,9,10,11,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":22,"seq":94489280537,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":505444,"kb_used_data":474260,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93866396,"statfs":{"total":96636764160,"available":96119189504,"internally_reserved":0,"allocated":485642240,"data_stored":485158850,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,5,7,9,10,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":22,"seq":94489280536,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":656540,"kb_used_data":603148,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":93715300,"statfs":{"total":96636764160,"available":95964467200,"internally_reserved":0,"allocated":617623552,"data_stored":617202890,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":54655424},"hb_peers":[0,1,3,6,7,9,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":11,"up_from":22,"seq":94489280537,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1258828,"kb_used_data":1191548,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":93113012,"statfs":{"total":96636764160,"available":95347724288,"internally_reserved":0,"allocated":1220145152,"data_stored":1219719686,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":68876740},"hb_peers":[0,2,4,7,8,10,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":12,"up_from":22,"seq":94489280537,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1060804,"kb_used_data":985204,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":93311036,"statfs":{"total":96636764160,"available":95550500864,"internally_reserved":0,"allocated":1008848896,"data_stored":1008396947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,1,2,3,5,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":14,"up_from":22,"seq":94489280537,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1109572,"kb_used_data":1042292,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":93262268,"statfs":{"total":96636764160,"available":95500562432,"internally_reserved":0,"allocated":1067307008,"data_stored":1066853406,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,1,5,6,7,8,9,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1075773440,"data_stored":1075773440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1282670592,"data_stored":1282670592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":404881408,"data_stored":404881408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":958136320,"data_stored":958136320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":479461376,"data_stored":479461376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":250216448,"data_stored":250216448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":393216000,"data_stored":393216000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1162280960,"data_stored":1162280960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":480575488,"data_stored":480575488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":352190464,"data_stored":352190464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":618397696,"data_stored":618397696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":1237188608,"data_stored":1237188608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":1007812608,"data_stored":1007812608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":1021509632,"data_stored":1021509632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":1066336256,"data_stored":1066336256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":861798400,"data_stored":861798400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T11:24:01.388 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-13T11:24:01.448 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-13T11:24:01.448 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:24:01.450 INFO:tasks.ceph.osd.7.smithi177.stderr:2024-09-13T11:24:01.444+0000 7f897b51d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26575) UID: 0 2024-09-13T11:24:01.471 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-09-13T11:24:01.550 INFO:tasks.ceph.osd.13.smithi092.stderr:2024-09-13T11:24:01.547+0000 7f3e5afea640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26350) UID: 0 2024-09-13T11:24:01.651 INFO:tasks.ceph.osd.2.smithi162.stderr:2024-09-13T11:24:01.649+0000 7fe92b35a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26313) UID: 0 2024-09-13T11:24:01.658 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-13T11:24:01.658 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:24:01.678 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-09-13T11:24:01.752 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-13T11:24:01.748+0000 7f6a68775640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26307) UID: 0 2024-09-13T11:24:01.822 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-13T11:24:01.822 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:24:01.838 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_ops_in_flight 2024-09-13T11:24:01.880 INFO:tasks.ceph.osd.12.smithi001.stderr:2024-09-13T11:24:01.876+0000 7f892c6e6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26314) UID: 0 2024-09-13T11:24:01.908 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-13T11:24:01.908 INFO:teuthology.orchestra.run.smithi001.stdout:{"fsid":"2fd68ac7-5581-4d90-80a4-30bcf5252a7a","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":124,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":103,"num_osds":16,"num_up_osds":16,"osd_up_since":1726226566,"num_in_osds":16,"osd_in_since":1726226521,"num_remapped_pgs":3},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":39},{"state_name":"active+recovering+remapped","count":5},{"state_name":"active+remapped","count":1}],"num_pgs":45,"num_pools":3,"num_objects":75327,"data_bytes":4936860211,"bytes_used":12794531840,"bytes_avail":1533393694720,"bytes_total":1546188226560,"misplaced_objects":4019,"misplaced_total":150654,"misplaced_ratio":0.026677021519508279,"recovering_objects_per_sec":455,"recovering_bytes_per_sec":29883349,"recovering_keys_per_sec":0,"num_objects_recovered":2259,"num_bytes_recovered":148045824,"num_keys_recovered":0,"write_bytes_sec":80244532,"read_op_per_sec":0,"write_op_per_sec":1224},"fsmap":{"epoch":1,"btime":"2024-09-13T11:21:57:222574+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":5,"modified":"2024-09-13T11:23:46.137611+0000","services":{"osd":{"daemons":{"summary":"","5":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-09-13T11:24:01.918 INFO:teuthology.orchestra.run.smithi001.stderr:noscrub is set 2024-09-13T11:24:01.934 INFO:tasks.ceph.ceph_manager.ceph:making progress, resetting timeout 2024-09-13T11:24:01.934 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-13T11:24:01.954 INFO:tasks.ceph.osd.1.smithi092.stderr:2024-09-13T11:24:01.951+0000 7fe7789c5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33718) UID: 0 2024-09-13T11:24:02.007 INFO:tasks.radosbench.radosbench.0.smithi177.stdout: 95 16 81178 81162 53.3906 82.5625 0.00346157 0.0187207 2024-09-13T11:24:02.020 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-13T11:24:02.020 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:24:02.037 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_blocked_ops 2024-09-13T11:24:02.055 INFO:tasks.ceph.osd.2.smithi162.stderr:2024-09-13T11:24:02.053+0000 7fe92b35a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26313) UID: 0 2024-09-13T11:24:02.155 INFO:tasks.ceph.osd.11.smithi177.stderr:2024-09-13T11:24:02.152+0000 7f24792c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26580) UID: 0 2024-09-13T11:24:02.206 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-13T11:24:02.206 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:24:02.221 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_historic_ops 2024-09-13T11:24:02.257 INFO:tasks.ceph.osd.5.smithi092.stderr:2024-09-13T11:24:02.251+0000 7f6f0b705640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26346) UID: 0 2024-09-13T11:24:02.345 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-13T11:24:02.345 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:24:02.346 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-13T11:24:02.347 INFO:teuthology.orchestra.run.smithi001.stderr:dumped all 2024-09-13T11:24:02.358 INFO:tasks.ceph.osd.1.smithi092.stderr:2024-09-13T11:24:02.355+0000 7fe7789c5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33718) UID: 0 2024-09-13T11:24:02.361 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-13T11:24:02.365 INFO:teuthology.orchestra.run.smithi001.stdout:{"pg_ready":true,"pg_map":{"version":146,"stamp":"2024-09-13T11:24:02.141672+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":5264900659,"num_objects":80332,"num_object_clones":0,"num_object_copies":160664,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1468,"num_objects_unfound":0,"num_objects_dirty":80332,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":33946,"num_write_kb":2165746,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10892,"num_bytes_recovered":713719827,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":140889,"ondisk_log_size":140889,"up":90,"acting":90,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":118,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":16,"kb":1509949440,"kb_used":13414956,"kb_used_data":12523692,"kb_used_omap":27,"kb_used_meta":890916,"kb_avail":1496534484,"statfs":{"total":1546188226560,"available":1532451311616,"internally_reserved":0,"allocated":12824260608,"data_stored":12817144159,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28358,"internal_metadata":912298298},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,4,1,3,3,0,1,1,0,1],"upper_bound":1024},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":379617280,"num_objects":5792,"num_object_clones":0,"num_object_copies":11584,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-4110,"num_objects_unfound":0,"num_objects_dirty":5792,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3248,"num_write_kb":207904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":322,"num_bytes_recovered":21102592,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000753"},"pg_stats":[{"pgid":"3.23","version":"102'3545","reported_seq":6265,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-13T11:24:00.318559+0000","last_change":"2024-09-13T11:23:48.133184+0000","last_active":"2024-09-13T11:24:00.318559+0000","last_peered":"2024-09-13T11:24:00.318559+0000","last_clean":"2024-09-13T11:24:00.318559+0000","last_became_active":"2024-09-13T11:23:47.926677+0000","last_became_peered":"2024-09-13T11:23:47.926677+0000","last_unstale":"2024-09-13T11:24:00.318559+0000","last_undegraded":"2024-09-13T11:24:00.318559+0000","last_fullsized":"2024-09-13T11:24:00.318559+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":6,"last_scrub":"91'3278","last_scrub_stamp":"2024-09-13T11:23:48.133084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:48.133084+0000","objects_scrubbed":1029,"log_size":3545,"log_dups_size":0,"ondisk_log_size":3545,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:12.874134+0000","scrub_duration":139,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":85000192,"num_objects":1297,"num_object_clones":0,"num_object_copies":2594,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1297,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":268,"num_write_kb":17152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9],"acting":[5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.22","version":"102'3485","reported_seq":5392,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-13T11:24:00.410951+0000","last_change":"2024-09-13T11:23:47.300446+0000","last_active":"2024-09-13T11:24:00.410951+0000","last_peered":"2024-09-13T11:24:00.410951+0000","last_clean":"2024-09-13T11:24:00.410951+0000","last_became_active":"2024-09-13T11:23:46.925318+0000","last_became_peered":"2024-09-13T11:23:46.925318+0000","last_unstale":"2024-09-13T11:24:00.410951+0000","last_undegraded":"2024-09-13T11:24:00.410951+0000","last_fullsized":"2024-09-13T11:24:00.410951+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":6,"last_scrub":"90'3246","last_scrub_stamp":"2024-09-13T11:23:47.300339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:47.300339+0000","objects_scrubbed":1033,"log_size":3485,"log_dups_size":0,"ondisk_log_size":3485,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:24:49.780485+0000","scrub_duration":203,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":83492864,"num_objects":1274,"num_object_clones":0,"num_object_copies":2548,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1274,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":241,"num_write_kb":15424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,14],"acting":[0,14],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.21","version":"101'3436","reported_seq":6136,"reported_epoch":102,"state":"peering","last_fresh":"2024-09-13T11:24:00.239108+0000","last_change":"2024-09-13T11:24:00.239108+0000","last_active":"2024-09-13T11:24:00.239108+0000","last_peered":"2024-09-13T11:24:00.144417+0000","last_clean":"2024-09-13T11:23:32.050222+0000","last_became_active":"2024-09-13T11:23:35.026171+0000","last_became_peered":"2024-09-13T11:23:35.026171+0000","last_unstale":"2024-09-13T11:24:00.239108+0000","last_undegraded":"2024-09-13T11:24:00.239108+0000","last_fullsized":"2024-09-13T11:24:00.239108+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":6,"last_scrub":"69'2784","last_scrub_stamp":"2024-09-13T11:23:21.595419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:21.595419+0000","objects_scrubbed":1386,"log_size":3436,"log_dups_size":0,"ondisk_log_size":3436,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":311,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":78905344,"num_objects":1204,"num_object_clones":0,"num_object_copies":2408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1204,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":511,"num_write_kb":32704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":836,"num_bytes_recovered":54788096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8],"acting":[0,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.20","version":"102'3514","reported_seq":5536,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-13T11:24:00.246567+0000","last_change":"2024-09-13T11:23:47.969503+0000","last_active":"2024-09-13T11:24:00.246567+0000","last_peered":"2024-09-13T11:24:00.246567+0000","last_clean":"2024-09-13T11:24:00.246567+0000","last_became_active":"2024-09-13T11:23:33.954946+0000","last_became_peered":"2024-09-13T11:23:33.954946+0000","last_unstale":"2024-09-13T11:24:00.246567+0000","last_undegraded":"2024-09-13T11:24:00.246567+0000","last_fullsized":"2024-09-13T11:24:00.246567+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":6,"last_scrub":"91'3270","last_scrub_stamp":"2024-09-13T11:23:47.969372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:47.969372+0000","objects_scrubbed":1061,"log_size":3514,"log_dups_size":0,"ondisk_log_size":3514,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:24:48.255263+0000","scrub_duration":1327,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":85852160,"num_objects":1310,"num_object_clones":0,"num_object_copies":2620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1310,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":249,"num_write_kb":15936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,0],"acting":[12,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[]},{"pgid":"3.1f","version":"104'4210","reported_seq":8237,"reported_epoch":104,"state":"active+recovering+remapped","last_fresh":"2024-09-13T11:24:02.066916+0000","last_change":"2024-09-13T11:23:37.329722+0000","last_active":"2024-09-13T11:24:02.066916+0000","last_peered":"2024-09-13T11:24:02.066916+0000","last_clean":"2024-09-13T11:23:32.316727+0000","last_became_active":"2024-09-13T11:23:35.053803+0000","last_became_peered":"2024-09-13T11:23:35.053803+0000","last_unstale":"2024-09-13T11:24:02.066916+0000","last_undegraded":"2024-09-13T11:24:02.066916+0000","last_fullsized":"2024-09-13T11:24:02.066916+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:22:24.896055+0000","objects_scrubbed":0,"log_size":4210,"log_dups_size":0,"ondisk_log_size":4210,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":164069376,"num_objects":2504,"num_object_clones":0,"num_object_copies":5008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":730,"num_objects_unfound":0,"num_objects_dirty":2504,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2504,"num_write_kb":160224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1286,"num_bytes_recovered":84279296,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,14],"acting":[11,10],"avail_no_missing":["11","12"],"object_location_counts":[{"shards":"10,11","objects":2139},{"shards":"11,12","objects":365}],"blocked_by":[],"up_primary":10,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1e","version":"101'4289","reported_seq":14630,"reported_epoch":102,"state":"creating+peering","last_fresh":"2024-09-13T11:24:00.236853+0000","last_change":"2024-09-13T11:24:01.002570+0000","last_active":"2024-09-13T11:24:01.002570+0000","last_peered":"2024-09-13T11:24:00.236853+0000","last_clean":"2024-09-13T11:23:30.974769+0000","last_became_active":"2024-09-13T11:23:35.075137+0000","last_became_peered":"2024-09-13T11:23:35.075137+0000","last_unstale":"2024-09-13T11:24:00.236853+0000","last_undegraded":"2024-09-13T11:24:00.236853+0000","last_fullsized":"2024-09-13T11:24:00.236853+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:22:24.896055+0000","objects_scrubbed":0,"log_size":4289,"log_dups_size":0,"ondisk_log_size":4289,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":165740544,"num_objects":2529,"num_object_clones":0,"num_object_copies":5058,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2529,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2529,"num_write_kb":161856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2573,"num_bytes_recovered":168591360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"101'4115","reported_seq":9379,"reported_epoch":102,"state":"peering","last_fresh":"2024-09-13T11:24:00.240669+0000","last_change":"2024-09-13T11:24:00.240669+0000","last_active":"2024-09-13T11:24:00.240669+0000","last_peered":"2024-09-13T11:24:00.209888+0000","last_clean":"2024-09-13T11:23:32.393844+0000","last_became_active":"2024-09-13T11:23:35.080149+0000","last_became_peered":"2024-09-13T11:23:35.080149+0000","last_unstale":"2024-09-13T11:24:00.240669+0000","last_undegraded":"2024-09-13T11:24:00.240669+0000","last_fullsized":"2024-09-13T11:24:00.240669+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:22:24.896055+0000","objects_scrubbed":0,"log_size":4115,"log_dups_size":0,"ondisk_log_size":4115,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":158859264,"num_objects":2424,"num_object_clones":0,"num_object_copies":4848,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2424,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2424,"num_write_kb":155136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1700,"num_bytes_recovered":111411200,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,15],"acting":[2,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1c","version":"104'4310","reported_seq":7986,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-13T11:24:01.961650+0000","last_change":"2024-09-13T11:24:00.529312+0000","last_active":"2024-09-13T11:24:01.961650+0000","last_peered":"2024-09-13T11:24:01.961650+0000","last_clean":"2024-09-13T11:24:01.961650+0000","last_became_active":"2024-09-13T11:23:33.989492+0000","last_became_peered":"2024-09-13T11:23:33.989492+0000","last_unstale":"2024-09-13T11:24:01.961650+0000","last_undegraded":"2024-09-13T11:24:01.961650+0000","last_fullsized":"2024-09-13T11:24:01.961650+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":5,"last_scrub":"102'4277","last_scrub_stamp":"2024-09-13T11:24:00.529243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:24:00.529243+0000","objects_scrubbed":2544,"log_size":4310,"log_dups_size":0,"ondisk_log_size":4310,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:15.595052+0000","scrub_duration":259,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":169017344,"num_objects":2579,"num_object_clones":0,"num_object_copies":5158,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2579,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":35,"num_write_kb":2240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7],"acting":[11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1b","version":"102'4216","reported_seq":9052,"reported_epoch":102,"state":"active+recovering+remapped","last_fresh":"2024-09-13T11:24:00.295969+0000","last_change":"2024-09-13T11:23:35.097804+0000","last_active":"2024-09-13T11:24:00.295969+0000","last_peered":"2024-09-13T11:24:00.295969+0000","last_clean":"2024-09-13T11:23:32.309147+0000","last_became_active":"2024-09-13T11:23:35.080340+0000","last_became_peered":"2024-09-13T11:23:35.080340+0000","last_unstale":"2024-09-13T11:24:00.295969+0000","last_undegraded":"2024-09-13T11:24:00.295969+0000","last_fullsized":"2024-09-13T11:24:00.295969+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:22:24.896055+0000","objects_scrubbed":0,"log_size":4216,"log_dups_size":0,"ondisk_log_size":4216,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":162889728,"num_objects":2486,"num_object_clones":0,"num_object_copies":4972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":407,"num_objects_unfound":0,"num_objects_dirty":2486,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2486,"num_write_kb":159072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1556,"num_bytes_recovered":101974016,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,15],"acting":[1,12],"avail_no_missing":["1","13"],"object_location_counts":[{"shards":"1,12","objects":2282},{"shards":"1,12,13","objects":1},{"shards":"1,13","objects":203}],"blocked_by":[],"up_primary":12,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1a","version":"101'4113","reported_seq":8724,"reported_epoch":101,"state":"active+recovering+remapped","last_fresh":"2024-09-13T11:24:00.027277+0000","last_change":"2024-09-13T11:23:35.089105+0000","last_active":"2024-09-13T11:24:00.027277+0000","last_peered":"2024-09-13T11:24:00.027277+0000","last_clean":"2024-09-13T11:23:32.455463+0000","last_became_active":"2024-09-13T11:23:35.072595+0000","last_became_peered":"2024-09-13T11:23:35.072595+0000","last_unstale":"2024-09-13T11:24:00.027277+0000","last_undegraded":"2024-09-13T11:24:00.027277+0000","last_fullsized":"2024-09-13T11:24:00.027277+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:22:24.896055+0000","objects_scrubbed":0,"log_size":4113,"log_dups_size":0,"ondisk_log_size":4113,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":157745152,"num_objects":2407,"num_object_clones":0,"num_object_copies":4814,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":331,"num_objects_unfound":0,"num_objects_dirty":2407,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2407,"num_write_kb":154048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1482,"num_bytes_recovered":97124352,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9],"acting":[3,4],"avail_no_missing":["3","7"],"object_location_counts":[{"shards":"3,4","objects":2241},{"shards":"3,7","objects":165},{"shards":"3,7,9","objects":1}],"blocked_by":[],"up_primary":4,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.19","version":"104'3901","reported_seq":8200,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-13T11:24:01.930513+0000","last_change":"2024-09-13T11:23:49.429522+0000","last_active":"2024-09-13T11:24:01.930513+0000","last_peered":"2024-09-13T11:24:01.930513+0000","last_clean":"2024-09-13T11:24:01.930513+0000","last_became_active":"2024-09-13T11:23:41.924941+0000","last_became_peered":"2024-09-13T11:23:41.924941+0000","last_unstale":"2024-09-13T11:24:01.930513+0000","last_undegraded":"2024-09-13T11:24:01.930513+0000","last_fullsized":"2024-09-13T11:24:01.930513+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":5,"last_scrub":"92'3415","last_scrub_stamp":"2024-09-13T11:23:49.429429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:49.429429+0000","objects_scrubbed":1979,"log_size":3901,"log_dups_size":0,"ondisk_log_size":3901,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:04.629542+0000","scrub_duration":587,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":162398208,"num_objects":2478,"num_object_clones":0,"num_object_copies":4956,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2478,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":499,"num_write_kb":31936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1],"acting":[9,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.18","version":"102'3900","reported_seq":8460,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-13T11:24:00.656686+0000","last_change":"2024-09-13T11:23:56.067740+0000","last_active":"2024-09-13T11:24:00.656686+0000","last_peered":"2024-09-13T11:24:00.656686+0000","last_clean":"2024-09-13T11:24:00.656686+0000","last_became_active":"2024-09-13T11:23:48.926014+0000","last_became_peered":"2024-09-13T11:23:48.926014+0000","last_unstale":"2024-09-13T11:24:00.656686+0000","last_undegraded":"2024-09-13T11:24:00.656686+0000","last_fullsized":"2024-09-13T11:24:00.656686+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:22:24.896055+0000","objects_scrubbed":0,"log_size":3900,"log_dups_size":0,"ondisk_log_size":3900,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-13T11:24:01.063698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":166002688,"num_objects":2533,"num_object_clones":0,"num_object_copies":5066,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2533,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2533,"num_write_kb":162112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1458,"num_bytes_recovered":95551488,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0],"acting":[10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.17","version":"102'3796","reported_seq":7826,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-13T11:24:00.298569+0000","last_change":"2024-09-13T11:23:49.349241+0000","last_active":"2024-09-13T11:24:00.298569+0000","last_peered":"2024-09-13T11:24:00.298569+0000","last_clean":"2024-09-13T11:24:00.298569+0000","last_became_active":"2024-09-13T11:23:42.935601+0000","last_became_peered":"2024-09-13T11:23:42.935601+0000","last_unstale":"2024-09-13T11:24:00.298569+0000","last_undegraded":"2024-09-13T11:24:00.298569+0000","last_fullsized":"2024-09-13T11:24:00.298569+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":5,"last_scrub":"92'3359","last_scrub_stamp":"2024-09-13T11:23:49.349167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:49.349167+0000","objects_scrubbed":2004,"log_size":3796,"log_dups_size":0,"ondisk_log_size":3796,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:24:59.401302+0000","scrub_duration":363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":160497664,"num_objects":2449,"num_object_clones":0,"num_object_copies":4898,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2449,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":445,"num_write_kb":28480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3],"acting":[10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"104'4063","reported_seq":8724,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-13T11:24:01.937217+0000","last_change":"2024-09-13T11:23:46.510544+0000","last_active":"2024-09-13T11:24:01.937217+0000","last_peered":"2024-09-13T11:24:01.937217+0000","last_clean":"2024-09-13T11:24:01.937217+0000","last_became_active":"2024-09-13T11:23:46.105237+0000","last_became_peered":"2024-09-13T11:23:46.105237+0000","last_unstale":"2024-09-13T11:24:01.937217+0000","last_undegraded":"2024-09-13T11:24:01.937217+0000","last_fullsized":"2024-09-13T11:24:01.937217+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":5,"last_scrub":"89'3478","last_scrub_stamp":"2024-09-13T11:23:46.510384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:46.510384+0000","objects_scrubbed":2054,"log_size":4063,"log_dups_size":0,"ondisk_log_size":4063,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:09.149878+0000","scrub_duration":363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":173146112,"num_objects":2642,"num_object_clones":0,"num_object_copies":5284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2642,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":588,"num_write_kb":37632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,15],"acting":[11,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.15","version":"102'3821","reported_seq":8240,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-13T11:24:00.357761+0000","last_change":"2024-09-13T11:23:48.332790+0000","last_active":"2024-09-13T11:24:00.357761+0000","last_peered":"2024-09-13T11:24:00.357761+0000","last_clean":"2024-09-13T11:24:00.357761+0000","last_became_active":"2024-09-13T11:23:42.931149+0000","last_became_peered":"2024-09-13T11:23:42.931149+0000","last_unstale":"2024-09-13T11:24:00.357761+0000","last_undegraded":"2024-09-13T11:24:00.357761+0000","last_fullsized":"2024-09-13T11:24:00.357761+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":5,"last_scrub":"91'3384","last_scrub_stamp":"2024-09-13T11:23:48.332600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:48.332600+0000","objects_scrubbed":2032,"log_size":3821,"log_dups_size":0,"ondisk_log_size":3821,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:24:58.239508+0000","scrub_duration":235,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":161873920,"num_objects":2470,"num_object_clones":0,"num_object_copies":4940,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2470,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":438,"num_write_kb":28032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,0],"acting":[14,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"3.14","version":"101'3838","reported_seq":8075,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-09-13T11:24:00.009923+0000","last_change":"2024-09-13T11:23:48.697100+0000","last_active":"2024-09-13T11:24:00.009923+0000","last_peered":"2024-09-13T11:24:00.009923+0000","last_clean":"2024-09-13T11:24:00.009923+0000","last_became_active":"2024-09-13T11:23:41.920989+0000","last_became_peered":"2024-09-13T11:23:41.920989+0000","last_unstale":"2024-09-13T11:24:00.009923+0000","last_undegraded":"2024-09-13T11:24:00.009923+0000","last_fullsized":"2024-09-13T11:24:00.009923+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":5,"last_scrub":"91'3371","last_scrub_stamp":"2024-09-13T11:23:48.696960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:48.696960+0000","objects_scrubbed":1935,"log_size":3838,"log_dups_size":0,"ondisk_log_size":3838,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:02.861530+0000","scrub_duration":735,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":158072832,"num_objects":2412,"num_object_clones":0,"num_object_copies":4824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2412,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":477,"num_write_kb":30528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,14],"acting":[3,14],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":167,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-13T11:24:00.248208+0000","last_change":"2024-09-13T11:22:16.889130+0000","last_active":"2024-09-13T11:24:00.248208+0000","last_peered":"2024-09-13T11:24:00.248208+0000","last_clean":"2024-09-13T11:24:00.248208+0000","last_became_active":"2024-09-13T11:22:14.192897+0000","last_became_peered":"2024-09-13T11:22:14.192897+0000","last_unstale":"2024-09-13T11:24:00.248208+0000","last_undegraded":"2024-09-13T11:24:00.248208+0000","last_fullsized":"2024-09-13T11:24:00.248208+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:22:13.113563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00084655899999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,10],"acting":[15,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"3.6","version":"104'4010","reported_seq":4199,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-13T11:24:02.034252+0000","last_change":"2024-09-13T11:23:22.656608+0000","last_active":"2024-09-13T11:24:02.034252+0000","last_peered":"2024-09-13T11:24:02.034252+0000","last_clean":"2024-09-13T11:24:02.034252+0000","last_became_active":"2024-09-13T11:23:19.939432+0000","last_became_peered":"2024-09-13T11:23:19.939432+0000","last_unstale":"2024-09-13T11:24:02.034252+0000","last_undegraded":"2024-09-13T11:24:02.034252+0000","last_fullsized":"2024-09-13T11:24:02.034252+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'2909","last_scrub_stamp":"2024-09-13T11:23:22.656323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:22.656323+0000","objects_scrubbed":1487,"log_size":4010,"log_dups_size":0,"ondisk_log_size":4010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:24:31.905185+0000","scrub_duration":323,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":169607168,"num_objects":2588,"num_object_clones":0,"num_object_copies":5176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2588,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1101,"num_write_kb":70464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,14],"acting":[13,14],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":178,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-13T11:24:01.930648+0000","last_change":"2024-09-13T11:23:21.875855+0000","last_active":"2024-09-13T11:24:01.930648+0000","last_peered":"2024-09-13T11:24:01.930648+0000","last_clean":"2024-09-13T11:24:01.930648+0000","last_became_active":"2024-09-13T11:22:14.211923+0000","last_became_peered":"2024-09-13T11:22:14.211923+0000","last_unstale":"2024-09-13T11:24:01.930648+0000","last_undegraded":"2024-09-13T11:24:01.930648+0000","last_fullsized":"2024-09-13T11:24:01.930648+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:23:21.875683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:23:21.875683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:24:40.678136+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0011689529999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0],"acting":[9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"104'3857","reported_seq":4047,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-13T11:24:02.041621+0000","last_change":"2024-09-13T11:23:20.443418+0000","last_active":"2024-09-13T11:24:02.041621+0000","last_peered":"2024-09-13T11:24:02.041621+0000","last_clean":"2024-09-13T11:24:02.041621+0000","last_became_active":"2024-09-13T11:23:19.927680+0000","last_became_peered":"2024-09-13T11:23:19.927680+0000","last_unstale":"2024-09-13T11:24:02.041621+0000","last_undegraded":"2024-09-13T11:24:02.041621+0000","last_fullsized":"2024-09-13T11:24:02.041621+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'2736","last_scrub_stamp":"2024-09-13T11:23:20.443329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:20.443329+0000","objects_scrubbed":1369,"log_size":3857,"log_dups_size":0,"ondisk_log_size":3857,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:24:48.147625+0000","scrub_duration":183,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":163381248,"num_objects":2493,"num_object_clones":0,"num_object_copies":4986,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2493,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1124,"num_write_kb":71936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,8],"acting":[13,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":175,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-13T11:24:00.241368+0000","last_change":"2024-09-13T11:23:59.530794+0000","last_active":"2024-09-13T11:24:00.241368+0000","last_peered":"2024-09-13T11:24:00.241368+0000","last_clean":"2024-09-13T11:24:00.241368+0000","last_became_active":"2024-09-13T11:22:14.581458+0000","last_became_peered":"2024-09-13T11:22:14.581458+0000","last_unstale":"2024-09-13T11:24:00.241368+0000","last_undegraded":"2024-09-13T11:24:00.241368+0000","last_fullsized":"2024-09-13T11:24:00.241368+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:23:59.530670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:23:59.530670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:02.655631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001026047,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5],"acting":[8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"102'3860","reported_seq":4026,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-13T11:24:00.240209+0000","last_change":"2024-09-13T11:23:20.254213+0000","last_active":"2024-09-13T11:24:00.240209+0000","last_peered":"2024-09-13T11:24:00.240209+0000","last_clean":"2024-09-13T11:24:00.240209+0000","last_became_active":"2024-09-13T11:23:19.925399+0000","last_became_peered":"2024-09-13T11:23:19.925399+0000","last_unstale":"2024-09-13T11:24:00.240209+0000","last_undegraded":"2024-09-13T11:24:00.240209+0000","last_fullsized":"2024-09-13T11:24:00.240209+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'2755","last_scrub_stamp":"2024-09-13T11:23:20.254124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:20.254124+0000","objects_scrubbed":1437,"log_size":3860,"log_dups_size":0,"ondisk_log_size":3860,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:24:49.993543+0000","scrub_duration":187,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":166920192,"num_objects":2547,"num_object_clones":0,"num_object_copies":5094,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2547,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1110,"num_write_kb":71040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,12],"acting":[1,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":155,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-13T11:24:00.240262+0000","last_change":"2024-09-13T11:22:47.720313+0000","last_active":"2024-09-13T11:24:00.240262+0000","last_peered":"2024-09-13T11:24:00.240262+0000","last_clean":"2024-09-13T11:24:00.240262+0000","last_became_active":"2024-09-13T11:22:47.719636+0000","last_became_peered":"2024-09-13T11:22:47.719636+0000","last_unstale":"2024-09-13T11:24:00.240262+0000","last_undegraded":"2024-09-13T11:24:00.240262+0000","last_fullsized":"2024-09-13T11:24:00.240262+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:22:13.113563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00087476000000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,9],"acting":[1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"102'3851","reported_seq":4037,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-13T11:24:00.407158+0000","last_change":"2024-09-13T11:23:20.686446+0000","last_active":"2024-09-13T11:24:00.407158+0000","last_peered":"2024-09-13T11:24:00.407158+0000","last_clean":"2024-09-13T11:24:00.407158+0000","last_became_active":"2024-09-13T11:23:19.939695+0000","last_became_peered":"2024-09-13T11:23:19.939695+0000","last_unstale":"2024-09-13T11:24:00.407158+0000","last_undegraded":"2024-09-13T11:24:00.407158+0000","last_fullsized":"2024-09-13T11:24:00.407158+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'2739","last_scrub_stamp":"2024-09-13T11:23:20.686358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:20.686358+0000","objects_scrubbed":1386,"log_size":3851,"log_dups_size":0,"ondisk_log_size":3851,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:24:38.891737+0000","scrub_duration":271,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":163774464,"num_objects":2499,"num_object_clones":0,"num_object_copies":4998,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2499,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1113,"num_write_kb":71232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,13],"acting":[15,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"3.3","version":"102'3470","reported_seq":3679,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-13T11:24:00.365009+0000","last_change":"2024-09-13T11:23:32.147115+0000","last_active":"2024-09-13T11:24:00.365009+0000","last_peered":"2024-09-13T11:24:00.365009+0000","last_clean":"2024-09-13T11:24:00.365009+0000","last_became_active":"2024-09-13T11:23:31.930448+0000","last_became_peered":"2024-09-13T11:23:31.930448+0000","last_unstale":"2024-09-13T11:24:00.365009+0000","last_undegraded":"2024-09-13T11:24:00.365009+0000","last_fullsized":"2024-09-13T11:24:00.365009+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'3122","last_scrub_stamp":"2024-09-13T11:23:32.147021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:32.147021+0000","objects_scrubbed":876,"log_size":3470,"log_dups_size":0,"ondisk_log_size":3470,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:24:48.175541+0000","scrub_duration":99,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":80216064,"num_objects":1224,"num_object_clones":0,"num_object_copies":2448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1224,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":348,"num_write_kb":22272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,7],"acting":[10,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"28'2","reported_seq":105,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-13T11:24:00.996197+0000","last_change":"2024-09-13T11:24:00.996197+0000","last_active":"2024-09-13T11:24:00.996197+0000","last_peered":"2024-09-13T11:24:00.996197+0000","last_clean":"2024-09-13T11:24:00.996197+0000","last_became_active":"2024-09-13T11:23:12.546348+0000","last_became_peered":"2024-09-13T11:23:12.546348+0000","last_unstale":"2024-09-13T11:24:00.996197+0000","last_undegraded":"2024-09-13T11:24:00.996197+0000","last_fullsized":"2024-09-13T11:24:00.996197+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2","last_scrub_stamp":"2024-09-13T11:24:00.996028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:24:00.996028+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:12.111097+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0025826730000000002,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8],"acting":[2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"101'3481","reported_seq":3688,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-13T11:24:00.246508+0000","last_change":"2024-09-13T11:23:32.749651+0000","last_active":"2024-09-13T11:24:00.246508+0000","last_peered":"2024-09-13T11:24:00.246508+0000","last_clean":"2024-09-13T11:24:00.246508+0000","last_became_active":"2024-09-13T11:23:31.921388+0000","last_became_peered":"2024-09-13T11:23:31.921388+0000","last_unstale":"2024-09-13T11:24:00.246508+0000","last_undegraded":"2024-09-13T11:24:00.246508+0000","last_fullsized":"2024-09-13T11:24:00.246508+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'3106","last_scrub_stamp":"2024-09-13T11:23:32.749487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:32.749487+0000","objects_scrubbed":856,"log_size":3481,"log_dups_size":0,"ondisk_log_size":3481,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:24:34.709501+0000","scrub_duration":91,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":80674816,"num_objects":1231,"num_object_clones":0,"num_object_copies":2462,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1231,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":375,"num_write_kb":24000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,15],"acting":[12,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":177,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-13T11:24:00.242697+0000","last_change":"2024-09-13T11:22:47.298206+0000","last_active":"2024-09-13T11:24:00.242697+0000","last_peered":"2024-09-13T11:24:00.242697+0000","last_clean":"2024-09-13T11:24:00.242697+0000","last_became_active":"2024-09-13T11:22:47.297319+0000","last_became_peered":"2024-09-13T11:22:47.297319+0000","last_unstale":"2024-09-13T11:24:00.242697+0000","last_undegraded":"2024-09-13T11:24:00.242697+0000","last_fullsized":"2024-09-13T11:24:00.242697+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:22:13.113563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00081225399999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,1],"acting":[14,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"3.1","version":"104'3482","reported_seq":3693,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-13T11:24:02.051997+0000","last_change":"2024-09-13T11:23:33.604966+0000","last_active":"2024-09-13T11:24:02.051997+0000","last_peered":"2024-09-13T11:24:02.051997+0000","last_clean":"2024-09-13T11:24:02.051997+0000","last_became_active":"2024-09-13T11:23:31.926528+0000","last_became_peered":"2024-09-13T11:23:31.926528+0000","last_unstale":"2024-09-13T11:24:02.051997+0000","last_undegraded":"2024-09-13T11:24:02.051997+0000","last_fullsized":"2024-09-13T11:24:02.051997+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'3070","last_scrub_stamp":"2024-09-13T11:23:33.604792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:33.604792+0000","objects_scrubbed":842,"log_size":3482,"log_dups_size":0,"ondisk_log_size":3482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:24:43.880038+0000","scrub_duration":119,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":82182144,"num_objects":1254,"num_object_clones":0,"num_object_copies":2508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1254,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":412,"num_write_kb":26368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,11],"acting":[13,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":165,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-09-13T11:23:59.241768+0000","last_change":"2024-09-13T11:22:16.267999+0000","last_active":"2024-09-13T11:23:59.241768+0000","last_peered":"2024-09-13T11:23:59.241768+0000","last_clean":"2024-09-13T11:23:59.241768+0000","last_became_active":"2024-09-13T11:22:14.211435+0000","last_became_peered":"2024-09-13T11:22:14.211435+0000","last_unstale":"2024-09-13T11:23:59.241768+0000","last_undegraded":"2024-09-13T11:23:59.241768+0000","last_fullsized":"2024-09-13T11:23:59.241768+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:22:13.113563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010010360000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2","version":"102'3473","reported_seq":3681,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-13T11:24:00.385197+0000","last_change":"2024-09-13T11:23:32.977666+0000","last_active":"2024-09-13T11:24:00.385197+0000","last_peered":"2024-09-13T11:24:00.385197+0000","last_clean":"2024-09-13T11:24:00.385197+0000","last_became_active":"2024-09-13T11:23:31.919828+0000","last_became_peered":"2024-09-13T11:23:31.919828+0000","last_unstale":"2024-09-13T11:24:00.385197+0000","last_undegraded":"2024-09-13T11:24:00.385197+0000","last_fullsized":"2024-09-13T11:24:00.385197+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'3057","last_scrub_stamp":"2024-09-13T11:23:32.977559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:32.977559+0000","objects_scrubbed":841,"log_size":3473,"log_dups_size":0,"ondisk_log_size":3473,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:24:52.169339+0000","scrub_duration":151,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":82378752,"num_objects":1257,"num_object_clones":0,"num_object_copies":2514,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1257,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":416,"num_write_kb":26624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,12],"acting":[14,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"2.3","version":"26'1","reported_seq":169,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-13T11:24:00.243013+0000","last_change":"2024-09-13T11:22:16.547691+0000","last_active":"2024-09-13T11:24:00.243013+0000","last_peered":"2024-09-13T11:24:00.243013+0000","last_clean":"2024-09-13T11:24:00.243013+0000","last_became_active":"2024-09-13T11:22:14.146507+0000","last_became_peered":"2024-09-13T11:22:14.146507+0000","last_unstale":"2024-09-13T11:24:00.243013+0000","last_undegraded":"2024-09-13T11:24:00.243013+0000","last_fullsized":"2024-09-13T11:24:00.243013+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:22:13.113563+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00085610999999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,15],"acting":[14,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"1.0","version":"26'112","reported_seq":309,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-13T11:24:00.239283+0000","last_change":"2024-09-13T11:23:49.325249+0000","last_active":"2024-09-13T11:24:00.239283+0000","last_peered":"2024-09-13T11:24:00.239283+0000","last_clean":"2024-09-13T11:24:00.239283+0000","last_became_active":"2024-09-13T11:22:12.129721+0000","last_became_peered":"2024-09-13T11:22:12.129721+0000","last_unstale":"2024-09-13T11:24:00.239283+0000","last_undegraded":"2024-09-13T11:24:00.239283+0000","last_fullsized":"2024-09-13T11:24:00.239283+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"26'112","last_scrub_stamp":"2024-09-13T11:23:49.325175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:11.098388+0000","last_clean_scrub_stamp":"2024-09-13T11:23:49.325175+0000","objects_scrubbed":2,"log_size":112,"log_dups_size":0,"ondisk_log_size":112,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:14.422287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.8","version":"102'3815","reported_seq":4002,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-13T11:24:00.333927+0000","last_change":"2024-09-13T11:23:20.644952+0000","last_active":"2024-09-13T11:24:00.333927+0000","last_peered":"2024-09-13T11:24:00.333927+0000","last_clean":"2024-09-13T11:24:00.333927+0000","last_became_active":"2024-09-13T11:23:19.925802+0000","last_became_peered":"2024-09-13T11:23:19.925802+0000","last_unstale":"2024-09-13T11:24:00.333927+0000","last_undegraded":"2024-09-13T11:24:00.333927+0000","last_fullsized":"2024-09-13T11:24:00.333927+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'2771","last_scrub_stamp":"2024-09-13T11:23:20.644747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:20.644747+0000","objects_scrubbed":1356,"log_size":3815,"log_dups_size":0,"ondisk_log_size":3815,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:24:38.144523+0000","scrub_duration":411,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":158072832,"num_objects":2412,"num_object_clones":0,"num_object_copies":4824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2412,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1056,"num_write_kb":67584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.9","version":"102'3899","reported_seq":4084,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-13T11:24:00.701258+0000","last_change":"2024-09-13T11:23:22.033678+0000","last_active":"2024-09-13T11:24:00.701258+0000","last_peered":"2024-09-13T11:24:00.701258+0000","last_clean":"2024-09-13T11:24:00.701258+0000","last_became_active":"2024-09-13T11:23:19.935114+0000","last_became_peered":"2024-09-13T11:23:19.935114+0000","last_unstale":"2024-09-13T11:24:00.701258+0000","last_undegraded":"2024-09-13T11:24:00.701258+0000","last_fullsized":"2024-09-13T11:24:00.701258+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'2808","last_scrub_stamp":"2024-09-13T11:23:22.033572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:22.033572+0000","objects_scrubbed":1471,"log_size":3899,"log_dups_size":0,"ondisk_log_size":3899,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:24:46.595859+0000","scrub_duration":287,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":167903232,"num_objects":2562,"num_object_clones":0,"num_object_copies":5124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2562,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1091,"num_write_kb":69824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,2],"acting":[12,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[]},{"pgid":"3.a","version":"101'4184","reported_seq":4372,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-09-13T11:24:00.050180+0000","last_change":"2024-09-13T11:23:32.526150+0000","last_active":"2024-09-13T11:24:00.050180+0000","last_peered":"2024-09-13T11:24:00.050180+0000","last_clean":"2024-09-13T11:24:00.050180+0000","last_became_active":"2024-09-13T11:23:31.934170+0000","last_became_peered":"2024-09-13T11:23:31.934170+0000","last_unstale":"2024-09-13T11:24:00.050180+0000","last_undegraded":"2024-09-13T11:24:00.050180+0000","last_fullsized":"2024-09-13T11:24:00.050180+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'3418","last_scrub_stamp":"2024-09-13T11:23:32.526057+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:32.526057+0000","objects_scrubbed":1774,"log_size":4184,"log_dups_size":0,"ondisk_log_size":4184,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:24:34.770161+0000","scrub_duration":507,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":166461440,"num_objects":2540,"num_object_clones":0,"num_object_copies":5080,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2540,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":766,"num_write_kb":49024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.b","version":"104'4264","reported_seq":4457,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-13T11:24:02.000298+0000","last_change":"2024-09-13T11:23:32.681343+0000","last_active":"2024-09-13T11:24:02.000298+0000","last_peered":"2024-09-13T11:24:02.000298+0000","last_clean":"2024-09-13T11:24:02.000298+0000","last_became_active":"2024-09-13T11:23:31.923415+0000","last_became_peered":"2024-09-13T11:23:31.923415+0000","last_unstale":"2024-09-13T11:24:02.000298+0000","last_undegraded":"2024-09-13T11:24:02.000298+0000","last_fullsized":"2024-09-13T11:24:02.000298+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'3471","last_scrub_stamp":"2024-09-13T11:23:32.681257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:32.681257+0000","objects_scrubbed":1720,"log_size":4264,"log_dups_size":0,"ondisk_log_size":4264,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:24:58.857812+0000","scrub_duration":199,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":164691968,"num_objects":2513,"num_object_clones":0,"num_object_copies":5026,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2513,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":793,"num_write_kb":50752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,1],"acting":[13,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"3.c","version":"104'4251","reported_seq":4443,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-13T11:24:02.035553+0000","last_change":"2024-09-13T11:23:32.524870+0000","last_active":"2024-09-13T11:24:02.035553+0000","last_peered":"2024-09-13T11:24:02.035553+0000","last_clean":"2024-09-13T11:24:02.035553+0000","last_became_active":"2024-09-13T11:23:31.922620+0000","last_became_peered":"2024-09-13T11:23:31.922620+0000","last_unstale":"2024-09-13T11:24:02.035553+0000","last_undegraded":"2024-09-13T11:24:02.035553+0000","last_fullsized":"2024-09-13T11:24:02.035553+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'3463","last_scrub_stamp":"2024-09-13T11:23:32.524776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:32.524776+0000","objects_scrubbed":1741,"log_size":4251,"log_dups_size":0,"ondisk_log_size":4251,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:24:37.745747+0000","scrub_duration":235,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":165740544,"num_objects":2529,"num_object_clones":0,"num_object_copies":5058,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2529,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":788,"num_write_kb":50432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,13],"acting":[11,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.d","version":"102'4172","reported_seq":4360,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-13T11:24:00.790903+0000","last_change":"2024-09-13T11:23:32.776835+0000","last_active":"2024-09-13T11:24:00.790903+0000","last_peered":"2024-09-13T11:24:00.790903+0000","last_clean":"2024-09-13T11:24:00.790903+0000","last_became_active":"2024-09-13T11:23:31.939246+0000","last_became_peered":"2024-09-13T11:23:31.939246+0000","last_unstale":"2024-09-13T11:24:00.790903+0000","last_undegraded":"2024-09-13T11:24:00.790903+0000","last_fullsized":"2024-09-13T11:24:00.790903+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'3395","last_scrub_stamp":"2024-09-13T11:23:32.776747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:32.776747+0000","objects_scrubbed":1704,"log_size":4172,"log_dups_size":0,"ondisk_log_size":4172,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:24:56.402862+0000","scrub_duration":179,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":162594816,"num_objects":2481,"num_object_clones":0,"num_object_copies":4962,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2481,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":777,"num_write_kb":49728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.e","version":"102'4304","reported_seq":9421,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-13T11:24:00.731742+0000","last_change":"2024-09-13T11:23:32.368856+0000","last_active":"2024-09-13T11:24:00.731742+0000","last_peered":"2024-09-13T11:24:00.731742+0000","last_clean":"2024-09-13T11:24:00.731742+0000","last_became_active":"2024-09-13T11:23:31.925869+0000","last_became_peered":"2024-09-13T11:23:31.925869+0000","last_unstale":"2024-09-13T11:24:00.731742+0000","last_undegraded":"2024-09-13T11:24:00.731742+0000","last_fullsized":"2024-09-13T11:24:00.731742+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'3527","last_scrub_stamp":"2024-09-13T11:23:32.368727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:32.368727+0000","objects_scrubbed":1774,"log_size":4304,"log_dups_size":0,"ondisk_log_size":4304,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:24:57.811779+0000","scrub_duration":219,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":167247872,"num_objects":2552,"num_object_clones":0,"num_object_copies":5104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2552,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":778,"num_write_kb":49792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"104'4198","reported_seq":4390,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-13T11:24:02.002768+0000","last_change":"2024-09-13T11:23:33.492630+0000","last_active":"2024-09-13T11:24:02.002768+0000","last_peered":"2024-09-13T11:24:02.002768+0000","last_clean":"2024-09-13T11:24:02.002768+0000","last_became_active":"2024-09-13T11:23:31.935239+0000","last_became_peered":"2024-09-13T11:23:31.935239+0000","last_unstale":"2024-09-13T11:24:02.002768+0000","last_undegraded":"2024-09-13T11:24:02.002768+0000","last_fullsized":"2024-09-13T11:24:02.002768+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'3420","last_scrub_stamp":"2024-09-13T11:23:33.492488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:33.492488+0000","objects_scrubbed":1776,"log_size":4198,"log_dups_size":0,"ondisk_log_size":4198,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:24:44.851261+0000","scrub_duration":215,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":167378944,"num_objects":2554,"num_object_clones":0,"num_object_copies":5108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2554,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":778,"num_write_kb":49792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,12],"acting":[11,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.10","version":"102'3954","reported_seq":8298,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-13T11:24:00.735862+0000","last_change":"2024-09-13T11:23:48.519654+0000","last_active":"2024-09-13T11:24:00.735862+0000","last_peered":"2024-09-13T11:24:00.735862+0000","last_clean":"2024-09-13T11:24:00.735862+0000","last_became_active":"2024-09-13T11:23:41.931183+0000","last_became_peered":"2024-09-13T11:23:41.931183+0000","last_unstale":"2024-09-13T11:24:00.735862+0000","last_undegraded":"2024-09-13T11:24:00.735862+0000","last_fullsized":"2024-09-13T11:24:00.735862+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":5,"last_scrub":"91'3468","last_scrub_stamp":"2024-09-13T11:23:48.519574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:48.519574+0000","objects_scrubbed":2031,"log_size":3954,"log_dups_size":0,"ondisk_log_size":3954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:24:57.342213+0000","scrub_duration":215,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":165085184,"num_objects":2519,"num_object_clones":0,"num_object_copies":5038,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2519,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":488,"num_write_kb":31232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.11","version":"102'3863","reported_seq":6985,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-13T11:24:00.234369+0000","last_change":"2024-09-13T11:23:46.102466+0000","last_active":"2024-09-13T11:24:00.234369+0000","last_peered":"2024-09-13T11:24:00.234369+0000","last_clean":"2024-09-13T11:24:00.234369+0000","last_became_active":"2024-09-13T11:23:39.063172+0000","last_became_peered":"2024-09-13T11:23:39.063172+0000","last_unstale":"2024-09-13T11:24:00.234369+0000","last_undegraded":"2024-09-13T11:24:00.234369+0000","last_fullsized":"2024-09-13T11:24:00.234369+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":5,"last_scrub":"88'3345","last_scrub_stamp":"2024-09-13T11:23:46.101973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:46.101973+0000","objects_scrubbed":2011,"log_size":3863,"log_dups_size":0,"ondisk_log_size":3863,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:08.671350+0000","scrub_duration":955,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":166002688,"num_objects":2533,"num_object_clones":0,"num_object_copies":5066,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2533,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":522,"num_write_kb":33408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11],"acting":[4,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.12","version":"102'3871","reported_seq":6917,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-13T11:24:00.530684+0000","last_change":"2024-09-13T11:23:35.111409+0000","last_active":"2024-09-13T11:24:00.530684+0000","last_peered":"2024-09-13T11:24:00.530684+0000","last_clean":"2024-09-13T11:24:00.530684+0000","last_became_active":"2024-09-13T11:23:21.998877+0000","last_became_peered":"2024-09-13T11:23:21.998877+0000","last_unstale":"2024-09-13T11:24:00.530684+0000","last_undegraded":"2024-09-13T11:24:00.530684+0000","last_fullsized":"2024-09-13T11:24:00.530684+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'3109","last_scrub_stamp":"2024-09-13T11:23:35.111320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:35.111320+0000","objects_scrubbed":1732,"log_size":3871,"log_dups_size":0,"ondisk_log_size":3871,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:03.371433+0000","scrub_duration":239,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":163446784,"num_objects":2494,"num_object_clones":0,"num_object_copies":4988,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2494,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":762,"num_write_kb":48768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,7],"acting":[14,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"3.13","version":"104'3963","reported_seq":7013,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-13T11:24:02.020450+0000","last_change":"2024-09-13T11:23:47.565094+0000","last_active":"2024-09-13T11:24:02.020450+0000","last_peered":"2024-09-13T11:24:02.020450+0000","last_clean":"2024-09-13T11:24:02.020450+0000","last_became_active":"2024-09-13T11:23:41.082433+0000","last_became_peered":"2024-09-13T11:23:41.082433+0000","last_unstale":"2024-09-13T11:24:02.020450+0000","last_undegraded":"2024-09-13T11:24:02.020450+0000","last_fullsized":"2024-09-13T11:24:02.020450+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'3408","last_scrub_stamp":"2024-09-13T11:23:47.564931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:47.564931+0000","objects_scrubbed":1981,"log_size":3963,"log_dups_size":0,"ondisk_log_size":3963,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:04.258231+0000","scrub_duration":471,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":166985728,"num_objects":2548,"num_object_clones":0,"num_object_copies":5096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":567,"num_write_kb":36288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7],"acting":[11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":36,"stat_sum":{"num_bytes":5264310272,"num_objects":80328,"num_object_clones":0,"num_object_copies":160656,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1468,"num_objects_unfound":0,"num_objects_dirty":80328,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":33799,"num_write_kb":2163072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10891,"num_bytes_recovered":713719808,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12726239232,"data_stored":12726239232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":140774,"ondisk_log_size":140774,"up":72,"acting":72,"num_store_stats":16},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":10},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":112,"ondisk_log_size":112,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":13,"up_from":23,"seq":98784247834,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1106512,"kb_used_data":1035072,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":93265328,"statfs":{"total":96636764160,"available":95503695872,"internally_reserved":0,"allocated":1059913728,"data_stored":1059453231,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73136581},"hb_peers":[0,1,3,4,6,8,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":9,"up_from":23,"seq":98784247833,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":427964,"kb_used_data":392620,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93943876,"statfs":{"total":96636764160,"available":96198529024,"internally_reserved":0,"allocated":402042880,"data_stored":401601626,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,8,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":40,"seq":171798691856,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1328072,"kb_used_data":1251576,"kb_used_omap":4,"kb_used_meta":76411,"kb_avail":93043768,"statfs":{"total":96636764160,"available":95276818432,"internally_reserved":0,"allocated":1281613824,"data_stored":1281147899,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4433,"internal_metadata":78245551},"hb_peers":[0,2,3,6,8,9,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":21,"seq":90194313241,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":981260,"kb_used_data":932028,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":93390580,"statfs":{"total":96636764160,"available":95631953920,"internally_reserved":0,"allocated":954396672,"data_stored":953931681,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":50395591},"hb_peers":[0,2,4,7,9,10,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":21,"seq":90194313244,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":284948,"kb_used_data":245444,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94086892,"statfs":{"total":96636764160,"available":96344977408,"internally_reserved":0,"allocated":251334656,"data_stored":250865626,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40434119},"hb_peers":[0,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":22,"seq":94489280536,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1129088,"kb_used_data":1049328,"kb_used_omap":1,"kb_used_meta":79742,"kb_avail":93242752,"statfs":{"total":96636764160,"available":95480578048,"internally_reserved":0,"allocated":1074511872,"data_stored":1074116405,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":81656261},"hb_peers":[1,3,4,5,8,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":22,"seq":94489280537,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":433732,"kb_used_data":398388,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93938108,"statfs":{"total":96636764160,"available":96192622592,"internally_reserved":0,"allocated":407949312,"data_stored":407500452,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,7,8,9,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":15,"up_from":22,"seq":94489280537,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919304,"kb_used_data":847864,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":93452536,"statfs":{"total":96636764160,"available":95695396864,"internally_reserved":0,"allocated":868212736,"data_stored":867816677,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":73136582},"hb_peers":[0,3,4,5,6,7,10,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,1],"upper_bound":1024},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":22,"seq":94489280537,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":496284,"kb_used_data":469260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93875556,"statfs":{"total":96636764160,"available":96128569344,"internally_reserved":0,"allocated":480522240,"data_stored":480050731,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[3,5,7,9,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":22,"seq":94489280537,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":421856,"kb_used_data":386512,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93949984,"statfs":{"total":96636764160,"available":96204783616,"internally_reserved":0,"allocated":395788288,"data_stored":395363799,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,3,5,7,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":22,"seq":94489280537,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1205024,"kb_used_data":1137744,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":93166816,"statfs":{"total":96636764160,"available":95402819584,"internally_reserved":0,"allocated":1165049856,"data_stored":1164630327,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":68876746},"hb_peers":[0,3,5,6,8,9,10,11,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":22,"seq":94489280537,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":505444,"kb_used_data":474260,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93866396,"statfs":{"total":96636764160,"available":96119189504,"internally_reserved":0,"allocated":485642240,"data_stored":485158850,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,5,7,9,10,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":22,"seq":94489280536,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":656540,"kb_used_data":603148,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":93715300,"statfs":{"total":96636764160,"available":95964467200,"internally_reserved":0,"allocated":617623552,"data_stored":617202890,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":54655424},"hb_peers":[0,1,3,6,7,9,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":11,"up_from":22,"seq":94489280538,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1348552,"kb_used_data":1272952,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":93023288,"statfs":{"total":96636764160,"available":95255846912,"internally_reserved":0,"allocated":1303502848,"data_stored":1303053612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":77396420},"hb_peers":[0,2,4,7,8,10,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":12,"up_from":22,"seq":94489280537,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1060804,"kb_used_data":985204,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":93311036,"statfs":{"total":96636764160,"available":95550500864,"internally_reserved":0,"allocated":1008848896,"data_stored":1008396947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,1,2,3,5,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":14,"up_from":22,"seq":94489280537,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1109572,"kb_used_data":1042292,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":93262268,"statfs":{"total":96636764160,"available":95500562432,"internally_reserved":0,"allocated":1067307008,"data_stored":1066853406,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,1,5,6,7,8,9,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1075773440,"data_stored":1075773440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1282670592,"data_stored":1282670592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":404881408,"data_stored":404881408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":958136320,"data_stored":958136320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":479461376,"data_stored":479461376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":250216448,"data_stored":250216448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":393216000,"data_stored":393216000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1162280960,"data_stored":1162280960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":480575488,"data_stored":480575488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":403898368,"data_stored":403898368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":618397696,"data_stored":618397696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":1253244928,"data_stored":1253244928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":1007812608,"data_stored":1007812608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":1027538944,"data_stored":1027538944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":1066336256,"data_stored":1066336256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":861798400,"data_stored":861798400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T11:24:02.458 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-09-13T11:24:02.457+0000 7f97befc9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26315) UID: 0 2024-09-13T11:24:02.518 INFO:teuthology.orchestra.run.smithi092.stdout:ERROR: (22) Invalid argument 2024-09-13T11:24:02.519 INFO:teuthology.orchestra.run.smithi092.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:24:02.536 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-13T11:24:02.582 INFO:tasks.ceph.osd.5.smithi092.stderr:2024-09-13T11:24:02.579+0000 7f6f0b705640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26346) UID: 0 2024-09-13T11:24:02.661 INFO:tasks.ceph.osd.15.smithi177.stderr:2024-09-13T11:24:02.660+0000 7f9d6ce77640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26578) UID: 0 2024-09-13T11:24:02.679 INFO:teuthology.orchestra.run.smithi092.stdout:ERROR: (22) Invalid argument 2024-09-13T11:24:02.679 INFO:teuthology.orchestra.run.smithi092.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:24:02.698 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-13T11:24:02.762 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-13T11:24:02.755+0000 7f6a68775640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26307) UID: 0 2024-09-13T11:24:02.862 INFO:tasks.ceph.osd.1.smithi092.stderr:2024-09-13T11:24:02.859+0000 7fe7789c5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33718) UID: 0 2024-09-13T11:24:02.895 INFO:teuthology.orchestra.run.smithi092.stdout:ERROR: (22) Invalid argument 2024-09-13T11:24:02.896 INFO:teuthology.orchestra.run.smithi092.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:24:02.917 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-09-13T11:24:02.927 INFO:tasks.ceph.osd.3.smithi177.stderr:2024-09-13T11:24:02.924+0000 7f10a45d5640 -1 osd.3 104 handle_osd_map injecting map crc failure 2024-09-13T11:24:02.927 INFO:tasks.ceph.osd.1.smithi092.stderr:2024-09-13T11:24:02.923+0000 7fe76f3f6640 -1 osd.1 104 handle_osd_map injecting map crc failure 2024-09-13T11:24:02.929 INFO:tasks.ceph.osd.1.smithi092.stderr:2024-09-13T11:24:02.923+0000 7fe76d3f2640 -1 osd.1 104 handle_osd_map injecting map crc failure 2024-09-13T11:24:02.963 INFO:tasks.ceph.osd.12.smithi001.stderr:2024-09-13T11:24:02.955+0000 7f892c6e6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26314) UID: 0 2024-09-13T11:24:03.007 INFO:tasks.radosbench.radosbench.0.smithi177.stdout: 96 16 82440 82424 53.656 78.875 0.00300472 0.0186287 2024-09-13T11:24:03.053 INFO:teuthology.orchestra.run.smithi092.stdout:ERROR: (22) Invalid argument 2024-09-13T11:24:03.053 INFO:teuthology.orchestra.run.smithi092.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:24:03.064 INFO:tasks.ceph.osd.5.smithi092.stderr:2024-09-13T11:24:03.059+0000 7f6f0b705640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26346) UID: 0 2024-09-13T11:24:03.069 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-09-13T11:24:03.147 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T11:24:03.166 INFO:tasks.ceph.osd.7.smithi177.stderr:2024-09-13T11:24:03.164+0000 7f897b51d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26575) UID: 0 2024-09-13T11:24:03.237 INFO:teuthology.orchestra.run.smithi092.stdout:ERROR: (22) Invalid argument 2024-09-13T11:24:03.238 INFO:teuthology.orchestra.run.smithi092.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:24:03.253 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-09-13T11:24:03.267 INFO:tasks.ceph.osd.4.smithi001.stderr:2024-09-13T11:24:03.259+0000 7ff8e0218640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26310) UID: 0 2024-09-13T11:24:03.318 INFO:teuthology.orchestra.run.smithi001.stdout:osd.0: {} 2024-09-13T11:24:03.318 INFO:teuthology.orchestra.run.smithi001.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T11:24:03.338 INFO:teuthology.orchestra.run.smithi001.stdout:osd.1: {} 2024-09-13T11:24:03.339 INFO:teuthology.orchestra.run.smithi001.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T11:24:03.364 INFO:teuthology.orchestra.run.smithi001.stdout:osd.2: {} 2024-09-13T11:24:03.364 INFO:teuthology.orchestra.run.smithi001.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T11:24:03.368 INFO:tasks.ceph.osd.2.smithi162.stderr:2024-09-13T11:24:03.365+0000 7fe92b35a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26313) UID: 0 2024-09-13T11:24:03.381 INFO:teuthology.orchestra.run.smithi092.stdout:ERROR: (22) Invalid argument 2024-09-13T11:24:03.381 INFO:teuthology.orchestra.run.smithi092.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:24:03.388 INFO:teuthology.orchestra.run.smithi001.stdout:osd.3: {} 2024-09-13T11:24:03.389 INFO:teuthology.orchestra.run.smithi001.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T11:24:03.395 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_ops_in_flight 2024-09-13T11:24:03.412 INFO:teuthology.orchestra.run.smithi001.stdout:osd.4: {} 2024-09-13T11:24:03.413 INFO:teuthology.orchestra.run.smithi001.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T11:24:03.438 INFO:teuthology.orchestra.run.smithi001.stdout:osd.5: {} 2024-09-13T11:24:03.438 INFO:teuthology.orchestra.run.smithi001.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T11:24:03.464 INFO:teuthology.orchestra.run.smithi001.stdout:osd.6: {} 2024-09-13T11:24:03.464 INFO:teuthology.orchestra.run.smithi001.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T11:24:03.469 INFO:tasks.ceph.osd.5.smithi092.stderr:2024-09-13T11:24:03.463+0000 7f6f0b705640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26346) UID: 0 2024-09-13T11:24:03.489 INFO:teuthology.orchestra.run.smithi001.stdout:osd.7: {} 2024-09-13T11:24:03.489 INFO:teuthology.orchestra.run.smithi001.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T11:24:03.513 INFO:teuthology.orchestra.run.smithi001.stdout:osd.8: {} 2024-09-13T11:24:03.513 INFO:teuthology.orchestra.run.smithi001.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-13T11:24:03.518 INFO:teuthology.orchestra.run.smithi092.stdout:ERROR: (22) Invalid argument 2024-09-13T11:24:03.518 INFO:teuthology.orchestra.run.smithi092.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:24:03.533 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_blocked_ops 2024-09-13T11:24:03.543 INFO:teuthology.orchestra.run.smithi001.stdout:osd.9: {} 2024-09-13T11:24:03.543 INFO:teuthology.orchestra.run.smithi001.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-13T11:24:03.570 INFO:tasks.ceph.osd.12.smithi001.stderr:2024-09-13T11:24:03.563+0000 7f892c6e6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26314) UID: 0 2024-09-13T11:24:03.582 INFO:teuthology.orchestra.run.smithi001.stdout:osd.10: {} 2024-09-13T11:24:03.583 INFO:teuthology.orchestra.run.smithi001.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-13T11:24:03.617 INFO:teuthology.orchestra.run.smithi001.stdout:osd.11: {} 2024-09-13T11:24:03.618 INFO:teuthology.orchestra.run.smithi001.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-13T11:24:03.646 INFO:teuthology.orchestra.run.smithi001.stdout:osd.12: {} 2024-09-13T11:24:03.646 INFO:teuthology.orchestra.run.smithi001.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-13T11:24:03.673 INFO:tasks.ceph.osd.12.smithi001.stderr:2024-09-13T11:24:03.667+0000 7f892c6e6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26314) UID: 0 2024-09-13T11:24:03.676 INFO:teuthology.orchestra.run.smithi001.stdout:osd.13: {} 2024-09-13T11:24:03.677 INFO:teuthology.orchestra.run.smithi001.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-13T11:24:03.706 INFO:teuthology.orchestra.run.smithi001.stdout:osd.14: {} 2024-09-13T11:24:03.706 INFO:teuthology.orchestra.run.smithi001.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-13T11:24:03.707 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-09-13T11:24:03.707 INFO:teuthology.orchestra.run.smithi092.stdout: "ops": [], 2024-09-13T11:24:03.707 INFO:teuthology.orchestra.run.smithi092.stdout: "complaint_time": 30, 2024-09-13T11:24:03.707 INFO:teuthology.orchestra.run.smithi092.stdout: "num_blocked_ops": 0 2024-09-13T11:24:03.707 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-09-13T11:24:03.721 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_historic_ops 2024-09-13T11:24:03.736 INFO:teuthology.orchestra.run.smithi001.stdout:osd.15: {} 2024-09-13T11:24:03.736 INFO:teuthology.orchestra.run.smithi001.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-13T11:24:03.773 INFO:tasks.ceph.osd.1.smithi092.stderr:2024-09-13T11:24:03.771+0000 7fe7789c5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33718) UID: 0 2024-09-13T11:24:03.849 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-09-13T11:24:03.849 INFO:teuthology.orchestra.run.smithi092.stdout: "size": 20, 2024-09-13T11:24:03.849 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 600, 2024-09-13T11:24:03.850 INFO:teuthology.orchestra.run.smithi092.stdout: "ops": [ 2024-09-13T11:24:03.850 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.850 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:79 3.1 3:841df5b5:::benchmark_data_smithi177_31116_object78:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-09-13T11:24:03.850 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:27.056083+0000", 2024-09-13T11:24:03.850 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 96.78902617, 2024-09-13T11:24:03.850 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.89469730199999997, 2024-09-13T11:24:03.850 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:24:03.850 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:24:03.850 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:03.850 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:24:03.850 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:24:03.850 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:03.850 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 79 2024-09-13T11:24:03.851 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:03.851 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:24:03.851 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.851 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:24:03.851 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.056083+0000", 2024-09-13T11:24:03.851 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:03.851 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:03.851 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.851 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:24:03.851 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.056083+0000", 2024-09-13T11:24:03.851 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:03.851 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:03.851 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.851 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:24:03.852 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.056087+0000", 2024-09-13T11:24:03.852 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.0659999999999997e-06 2024-09-13T11:24:03.852 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:03.852 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.852 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:24:03.852 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.056148+0000", 2024-09-13T11:24:03.852 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.1376000000000003e-05 2024-09-13T11:24:03.852 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:03.852 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.852 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:24:03.852 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.056180+0000", 2024-09-13T11:24:03.852 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.2370999999999999e-05 2024-09-13T11:24:03.852 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:03.853 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.853 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:24:03.853 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.056195+0000", 2024-09-13T11:24:03.853 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.5119e-05 2024-09-13T11:24:03.853 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:03.853 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.853 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:24:03.853 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.056304+0000", 2024-09-13T11:24:03.853 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000108365 2024-09-13T11:24:03.853 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:03.853 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.853 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:24:03.853 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.056478+0000", 2024-09-13T11:24:03.854 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000174542 2024-09-13T11:24:03.854 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:03.854 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.854 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 11", 2024-09-13T11:24:03.854 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.056690+0000", 2024-09-13T11:24:03.854 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00021167500000000001 2024-09-13T11:24:03.854 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:03.854 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.854 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:24:03.854 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.058063+0000", 2024-09-13T11:24:03.854 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0013731819999999999 2024-09-13T11:24:03.854 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:03.854 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.854 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:03.855 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.950477+0000", 2024-09-13T11:24:03.855 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.89241369400000004 2024-09-13T11:24:03.855 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:03.855 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.855 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:24:03.855 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.950652+0000", 2024-09-13T11:24:03.855 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00017526400000000001 2024-09-13T11:24:03.855 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:03.855 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.855 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:24:03.855 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.950780+0000", 2024-09-13T11:24:03.855 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00012764800000000001 2024-09-13T11:24:03.855 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:03.855 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:24:03.856 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:03.856 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:03.856 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.856 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:7 3.b 3:de78393b:::benchmark_data_smithi177_31116_object6:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T11:24:03.856 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:48.313952+0000", 2024-09-13T11:24:03.856 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 75.531156421000006, 2024-09-13T11:24:03.856 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.86421288600000001, 2024-09-13T11:24:03.856 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:24:03.856 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:24:03.856 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:03.856 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:24:03.856 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:24:03.857 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:03.857 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 7 2024-09-13T11:24:03.857 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:03.857 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:24:03.857 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.857 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:24:03.857 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.313952+0000", 2024-09-13T11:24:03.857 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:03.857 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:03.857 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.857 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:24:03.857 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.313952+0000", 2024-09-13T11:24:03.857 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:03.858 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:03.858 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.858 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:24:03.858 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.313957+0000", 2024-09-13T11:24:03.858 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.3599999999999998e-06 2024-09-13T11:24:03.858 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:03.858 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.858 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:24:03.858 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314043+0000", 2024-09-13T11:24:03.858 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.6766000000000002e-05 2024-09-13T11:24:03.858 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:03.858 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.858 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:24:03.858 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314065+0000", 2024-09-13T11:24:03.859 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.1416e-05 2024-09-13T11:24:03.859 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:03.859 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.859 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:24:03.859 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314080+0000", 2024-09-13T11:24:03.859 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.5519000000000002e-05 2024-09-13T11:24:03.859 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:03.859 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.859 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:24:03.859 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315478+0000", 2024-09-13T11:24:03.859 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.001397765 2024-09-13T11:24:03.859 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:03.860 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.860 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-09-13T11:24:03.860 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315507+0000", 2024-09-13T11:24:03.860 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.9128999999999998e-05 2024-09-13T11:24:03.860 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:03.860 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.860 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:24:03.860 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.167978+0000", 2024-09-13T11:24:03.860 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.85247068400000003 2024-09-13T11:24:03.860 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:03.860 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.860 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:24:03.860 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.168307+0000", 2024-09-13T11:24:03.861 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00032886699999999999 2024-09-13T11:24:03.861 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:03.861 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.861 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1", 2024-09-13T11:24:03.861 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.168647+0000", 2024-09-13T11:24:03.861 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00033998399999999999 2024-09-13T11:24:03.861 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:03.861 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.861 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:03.861 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.178011+0000", 2024-09-13T11:24:03.861 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0093639169999999994 2024-09-13T11:24:03.861 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:03.861 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.862 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:24:03.862 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.178026+0000", 2024-09-13T11:24:03.862 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.5274999999999999e-05 2024-09-13T11:24:03.862 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:03.862 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.862 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:24:03.862 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.178080+0000", 2024-09-13T11:24:03.862 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.3601999999999999e-05 2024-09-13T11:24:03.862 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:03.862 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.862 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:24:03.862 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.178165+0000", 2024-09-13T11:24:03.862 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.5601999999999998e-05 2024-09-13T11:24:03.862 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:03.863 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:24:03.863 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:03.863 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:03.863 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.863 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:9 3.b 3:dafadff6:::benchmark_data_smithi177_31116_object8:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T11:24:03.863 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:48.314114+0000", 2024-09-13T11:24:03.863 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 75.530994606999997, 2024-09-13T11:24:03.863 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.86613615899999996, 2024-09-13T11:24:03.863 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:24:03.863 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:24:03.863 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:03.863 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:24:03.863 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:24:03.864 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:03.864 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 9 2024-09-13T11:24:03.864 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:03.864 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:24:03.864 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.864 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:24:03.864 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314114+0000", 2024-09-13T11:24:03.864 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:03.864 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:03.864 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.864 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:24:03.864 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314114+0000", 2024-09-13T11:24:03.864 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:03.865 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:03.865 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.865 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:24:03.865 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314117+0000", 2024-09-13T11:24:03.865 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.3860000000000001e-06 2024-09-13T11:24:03.865 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:03.865 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.865 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:24:03.865 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314215+0000", 2024-09-13T11:24:03.865 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.8900000000000005e-05 2024-09-13T11:24:03.865 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:03.865 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.865 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:24:03.866 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314229+0000", 2024-09-13T11:24:03.866 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.3444999999999999e-05 2024-09-13T11:24:03.866 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:03.866 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.866 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:24:03.866 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314239+0000", 2024-09-13T11:24:03.866 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.8470000000000008e-06 2024-09-13T11:24:03.866 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:03.866 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.866 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:24:03.866 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315648+0000", 2024-09-13T11:24:03.866 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.001409291 2024-09-13T11:24:03.866 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:03.866 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.867 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-09-13T11:24:03.867 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315671+0000", 2024-09-13T11:24:03.867 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.3169000000000001e-05 2024-09-13T11:24:03.867 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:03.867 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.867 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:24:03.867 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.176036+0000", 2024-09-13T11:24:03.867 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.86036497499999998 2024-09-13T11:24:03.867 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:03.867 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.867 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:24:03.867 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.176189+0000", 2024-09-13T11:24:03.867 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00015271100000000001 2024-09-13T11:24:03.868 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:03.868 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.868 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1", 2024-09-13T11:24:03.868 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.176339+0000", 2024-09-13T11:24:03.868 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00015057499999999999 2024-09-13T11:24:03.868 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:03.868 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.868 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:03.868 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.180148+0000", 2024-09-13T11:24:03.868 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.003808891 2024-09-13T11:24:03.868 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:03.868 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.868 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:24:03.868 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.180168+0000", 2024-09-13T11:24:03.869 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.9507000000000001e-05 2024-09-13T11:24:03.869 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:03.869 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.869 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:24:03.869 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.180213+0000", 2024-09-13T11:24:03.869 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.5504999999999999e-05 2024-09-13T11:24:03.869 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:03.869 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.869 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:24:03.869 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.180250+0000", 2024-09-13T11:24:03.869 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.6956999999999999e-05 2024-09-13T11:24:03.869 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:03.869 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:24:03.870 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:03.870 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:03.870 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.870 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:10 3.b 3:dbe028a8:::benchmark_data_smithi177_31116_object9:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T11:24:03.870 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:48.314259+0000", 2024-09-13T11:24:03.871 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 75.530849477000004, 2024-09-13T11:24:03.871 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.86637313599999999, 2024-09-13T11:24:03.871 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:24:03.871 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:24:03.871 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:03.872 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:24:03.872 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:24:03.872 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:03.872 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 10 2024-09-13T11:24:03.872 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:03.872 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:24:03.872 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.872 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:24:03.872 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314259+0000", 2024-09-13T11:24:03.872 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:03.872 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:03.872 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.872 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:24:03.872 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314259+0000", 2024-09-13T11:24:03.873 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:03.873 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:03.873 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.873 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:24:03.873 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314261+0000", 2024-09-13T11:24:03.873 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.1500000000000002e-06 2024-09-13T11:24:03.873 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:03.873 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.876 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:24:03.876 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314349+0000", 2024-09-13T11:24:03.876 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.8070000000000002e-05 2024-09-13T11:24:03.876 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:03.876 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.876 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:24:03.876 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314365+0000", 2024-09-13T11:24:03.876 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.5835999999999999e-05 2024-09-13T11:24:03.877 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:03.877 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.877 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:24:03.877 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314377+0000", 2024-09-13T11:24:03.877 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.1589e-05 2024-09-13T11:24:03.877 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:03.877 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.877 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:24:03.877 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315788+0000", 2024-09-13T11:24:03.877 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0014109579999999999 2024-09-13T11:24:03.877 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:03.877 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.877 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-09-13T11:24:03.878 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315809+0000", 2024-09-13T11:24:03.878 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.0985999999999999e-05 2024-09-13T11:24:03.878 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:03.878 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.878 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:24:03.878 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.176822+0000", 2024-09-13T11:24:03.878 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.86101310799999997 2024-09-13T11:24:03.878 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:03.878 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.878 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:24:03.878 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.177021+0000", 2024-09-13T11:24:03.878 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00019911199999999999 2024-09-13T11:24:03.878 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:03.878 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.879 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1", 2024-09-13T11:24:03.879 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.177226+0000", 2024-09-13T11:24:03.879 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00020516600000000001 2024-09-13T11:24:03.879 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:03.879 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.879 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:24:03.879 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.180264+0000", 2024-09-13T11:24:03.879 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0030381739999999998 2024-09-13T11:24:03.879 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:03.879 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.879 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:03.879 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.180387+0000", 2024-09-13T11:24:03.880 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00012257699999999999 2024-09-13T11:24:03.880 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:03.880 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.880 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:24:03.880 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.180518+0000", 2024-09-13T11:24:03.880 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00013135800000000001 2024-09-13T11:24:03.880 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:03.880 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.880 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:24:03.880 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.180632+0000", 2024-09-13T11:24:03.880 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000114052 2024-09-13T11:24:03.880 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:03.880 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:24:03.881 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:03.881 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:03.881 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.881 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:39 3.b 3:d91854b3:::benchmark_data_smithi177_31116_object38:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T11:24:03.881 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:48.314393+0000", 2024-09-13T11:24:03.881 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 75.530715935000003, 2024-09-13T11:24:03.881 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.86850121700000005, 2024-09-13T11:24:03.881 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:24:03.881 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:24:03.881 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:03.881 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:24:03.881 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:24:03.881 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:03.882 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 39 2024-09-13T11:24:03.882 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:03.882 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:24:03.882 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.882 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:24:03.882 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314393+0000", 2024-09-13T11:24:03.882 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:03.882 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:03.882 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.882 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:24:03.882 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314393+0000", 2024-09-13T11:24:03.882 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:03.882 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:03.883 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.883 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:24:03.883 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314394+0000", 2024-09-13T11:24:03.883 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.513e-06 2024-09-13T11:24:03.883 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:03.883 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.883 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:24:03.883 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314463+0000", 2024-09-13T11:24:03.883 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.8261000000000003e-05 2024-09-13T11:24:03.883 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:03.883 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.883 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:24:03.883 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314477+0000", 2024-09-13T11:24:03.883 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.4619e-05 2024-09-13T11:24:03.884 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:03.884 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.884 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:24:03.884 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314485+0000", 2024-09-13T11:24:03.884 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.1699999999999997e-06 2024-09-13T11:24:03.884 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:03.884 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.884 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:24:03.884 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315917+0000", 2024-09-13T11:24:03.884 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0014316089999999999 2024-09-13T11:24:03.884 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:03.884 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.884 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-09-13T11:24:03.885 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315933+0000", 2024-09-13T11:24:03.885 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.5999999999999999e-05 2024-09-13T11:24:03.885 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:03.885 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.885 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:24:03.885 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.178205+0000", 2024-09-13T11:24:03.885 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.86227247100000004 2024-09-13T11:24:03.885 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:03.885 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.885 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:24:03.885 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.178316+0000", 2024-09-13T11:24:03.885 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000110788 2024-09-13T11:24:03.885 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:03.885 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.886 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1", 2024-09-13T11:24:03.886 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.178448+0000", 2024-09-13T11:24:03.886 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000132056 2024-09-13T11:24:03.886 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:03.886 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.886 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:24:03.886 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.182721+0000", 2024-09-13T11:24:03.886 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0042729780000000002 2024-09-13T11:24:03.886 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:03.886 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.886 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:03.886 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.182801+0000", 2024-09-13T11:24:03.886 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.0209000000000004e-05 2024-09-13T11:24:03.886 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:03.887 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.887 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:24:03.887 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.182855+0000", 2024-09-13T11:24:03.887 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.3749000000000003e-05 2024-09-13T11:24:03.887 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:03.887 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.887 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:24:03.887 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.182894+0000", 2024-09-13T11:24:03.887 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.8794000000000002e-05 2024-09-13T11:24:03.887 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:03.887 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:24:03.887 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:03.887 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:03.888 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.888 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:50 3.b 3:d1c144b8:::benchmark_data_smithi177_31116_object49:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T11:24:03.888 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:48.314498+0000", 2024-09-13T11:24:03.888 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 75.530610519999996, 2024-09-13T11:24:03.888 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.86878673299999998, 2024-09-13T11:24:03.888 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:24:03.888 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:24:03.888 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:03.888 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:24:03.888 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:24:03.888 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:03.888 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 50 2024-09-13T11:24:03.888 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:03.888 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:24:03.889 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.889 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:24:03.889 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314498+0000", 2024-09-13T11:24:03.889 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:03.889 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:03.889 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.889 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:24:03.889 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314498+0000", 2024-09-13T11:24:03.889 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:03.889 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:03.889 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.889 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:24:03.889 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314499+0000", 2024-09-13T11:24:03.889 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.1969999999999999e-06 2024-09-13T11:24:03.890 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:03.890 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.890 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:24:03.890 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314565+0000", 2024-09-13T11:24:03.890 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.5628000000000003e-05 2024-09-13T11:24:03.890 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:03.890 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.890 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:24:03.890 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314579+0000", 2024-09-13T11:24:03.890 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.4222e-05 2024-09-13T11:24:03.890 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:03.890 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.890 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:24:03.891 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314590+0000", 2024-09-13T11:24:03.891 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.1240000000000001e-05 2024-09-13T11:24:03.891 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:03.891 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.891 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:24:03.891 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316060+0000", 2024-09-13T11:24:03.891 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.001469515 2024-09-13T11:24:03.891 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:03.891 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.891 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-09-13T11:24:03.891 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316076+0000", 2024-09-13T11:24:03.891 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.5608999999999999e-05 2024-09-13T11:24:03.891 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:03.891 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.892 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:24:03.892 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.178857+0000", 2024-09-13T11:24:03.892 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.86278099900000005 2024-09-13T11:24:03.892 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:03.892 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.892 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:24:03.892 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.178961+0000", 2024-09-13T11:24:03.892 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000104664 2024-09-13T11:24:03.892 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:03.892 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.892 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1", 2024-09-13T11:24:03.892 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.179123+0000", 2024-09-13T11:24:03.892 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000161738 2024-09-13T11:24:03.893 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:03.893 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.893 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:24:03.893 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.182755+0000", 2024-09-13T11:24:03.893 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0036323200000000001 2024-09-13T11:24:03.893 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:03.893 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.893 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:03.893 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.183023+0000", 2024-09-13T11:24:03.893 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00026749100000000002 2024-09-13T11:24:03.893 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:03.893 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.893 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:24:03.893 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.183168+0000", 2024-09-13T11:24:03.894 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00014499099999999999 2024-09-13T11:24:03.894 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:03.894 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.894 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:24:03.894 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.183285+0000", 2024-09-13T11:24:03.894 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000117119 2024-09-13T11:24:03.894 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:03.894 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:24:03.894 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:03.894 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:03.894 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.894 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:60 3.b 3:d67218b0:::benchmark_data_smithi177_31116_object59:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T11:24:03.894 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:48.314603+0000", 2024-09-13T11:24:03.894 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 75.530505957000003, 2024-09-13T11:24:03.895 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.87106201100000002, 2024-09-13T11:24:03.895 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:24:03.895 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:24:03.895 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:03.895 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:24:03.895 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:24:03.895 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:03.895 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 60 2024-09-13T11:24:03.895 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:03.895 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:24:03.895 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.895 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:24:03.895 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314603+0000", 2024-09-13T11:24:03.896 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:03.896 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:03.896 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.896 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:24:03.896 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314603+0000", 2024-09-13T11:24:03.896 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:03.896 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:03.896 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.896 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:24:03.896 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314604+0000", 2024-09-13T11:24:03.896 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.2359999999999999e-06 2024-09-13T11:24:03.896 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:03.896 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.896 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:24:03.897 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314667+0000", 2024-09-13T11:24:03.897 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.3077999999999995e-05 2024-09-13T11:24:03.897 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:03.897 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.897 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:24:03.897 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314699+0000", 2024-09-13T11:24:03.897 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.1965999999999999e-05 2024-09-13T11:24:03.897 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:03.897 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.897 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:24:03.897 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314710+0000", 2024-09-13T11:24:03.897 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.1148999999999999e-05 2024-09-13T11:24:03.897 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:03.897 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.898 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:24:03.898 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316291+0000", 2024-09-13T11:24:03.898 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0015804440000000001 2024-09-13T11:24:03.898 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:03.898 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.898 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-09-13T11:24:03.898 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316310+0000", 2024-09-13T11:24:03.898 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.9799000000000001e-05 2024-09-13T11:24:03.898 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:03.898 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.898 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:24:03.898 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.179555+0000", 2024-09-13T11:24:03.898 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.86324469500000001 2024-09-13T11:24:03.899 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:03.899 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.899 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:24:03.899 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.179666+0000", 2024-09-13T11:24:03.899 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000110443 2024-09-13T11:24:03.899 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:03.899 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.899 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1", 2024-09-13T11:24:03.899 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.179791+0000", 2024-09-13T11:24:03.899 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00012560499999999999 2024-09-13T11:24:03.899 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:03.899 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.899 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:24:03.899 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.182761+0000", 2024-09-13T11:24:03.900 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0029697019999999998 2024-09-13T11:24:03.900 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:03.900 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.900 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:03.900 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.185572+0000", 2024-09-13T11:24:03.900 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.002811512 2024-09-13T11:24:03.900 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:03.900 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.900 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:24:03.900 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.185625+0000", 2024-09-13T11:24:03.900 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.2716000000000001e-05 2024-09-13T11:24:03.900 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:03.900 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.900 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:24:03.901 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.185665+0000", 2024-09-13T11:24:03.901 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.9666000000000003e-05 2024-09-13T11:24:03.901 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:03.901 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:24:03.901 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:03.901 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:03.901 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.901 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:62 3.b 3:d20ae5a8:::benchmark_data_smithi177_31116_object61:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T11:24:03.901 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:48.314723+0000", 2024-09-13T11:24:03.901 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 75.530385414999998, 2024-09-13T11:24:03.901 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.87338985099999999, 2024-09-13T11:24:03.901 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:24:03.901 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:24:03.902 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:03.902 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:24:03.902 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:24:03.902 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:03.902 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 62 2024-09-13T11:24:03.902 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:03.902 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:24:03.902 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.902 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:24:03.902 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314723+0000", 2024-09-13T11:24:03.902 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:03.902 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:03.902 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.902 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:24:03.903 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314723+0000", 2024-09-13T11:24:03.903 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:03.903 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:03.903 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.903 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:24:03.903 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314725+0000", 2024-09-13T11:24:03.903 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.3069999999999999e-06 2024-09-13T11:24:03.903 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:03.903 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.903 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:24:03.903 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314797+0000", 2024-09-13T11:24:03.903 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.2652999999999997e-05 2024-09-13T11:24:03.903 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:03.903 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.904 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:24:03.904 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314809+0000", 2024-09-13T11:24:03.904 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.1496e-05 2024-09-13T11:24:03.904 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:03.904 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.904 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:24:03.904 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314816+0000", 2024-09-13T11:24:03.904 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.6639999999999998e-06 2024-09-13T11:24:03.904 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:03.904 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.904 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:24:03.904 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316424+0000", 2024-09-13T11:24:03.904 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0016077559999999999 2024-09-13T11:24:03.905 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:03.905 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.905 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-09-13T11:24:03.905 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316441+0000", 2024-09-13T11:24:03.905 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.7305000000000001e-05 2024-09-13T11:24:03.905 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:03.905 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.905 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:24:03.905 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.180688+0000", 2024-09-13T11:24:03.905 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.86424692800000003 2024-09-13T11:24:03.905 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:03.905 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.905 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:24:03.905 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.180807+0000", 2024-09-13T11:24:03.906 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000118977 2024-09-13T11:24:03.906 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:03.906 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.906 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1", 2024-09-13T11:24:03.906 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.180938+0000", 2024-09-13T11:24:03.906 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00013049700000000001 2024-09-13T11:24:03.906 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:03.906 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.906 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:24:03.906 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.185680+0000", 2024-09-13T11:24:03.906 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0047417029999999999 2024-09-13T11:24:03.906 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:03.906 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.906 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:03.907 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.187856+0000", 2024-09-13T11:24:03.907 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0021760550000000001 2024-09-13T11:24:03.907 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:03.907 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.907 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:24:03.907 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.188007+0000", 2024-09-13T11:24:03.907 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00015185099999999999 2024-09-13T11:24:03.907 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:03.907 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.907 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:24:03.907 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.188113+0000", 2024-09-13T11:24:03.907 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000105659 2024-09-13T11:24:03.907 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:03.908 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:24:03.908 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:03.908 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:03.908 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.908 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:73 3.b 3:d0060d5f:::benchmark_data_smithi177_31116_object72:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T11:24:03.908 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:48.314829+0000", 2024-09-13T11:24:03.908 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 75.530280066000003, 2024-09-13T11:24:03.908 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.87364757199999998, 2024-09-13T11:24:03.908 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:24:03.908 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:24:03.908 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:03.908 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:24:03.908 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:24:03.908 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:03.909 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 73 2024-09-13T11:24:03.909 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:03.909 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:24:03.909 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.909 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:24:03.909 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314829+0000", 2024-09-13T11:24:03.909 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:03.909 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:03.909 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.909 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:24:03.909 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314829+0000", 2024-09-13T11:24:03.909 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:03.909 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:03.910 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.910 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:24:03.910 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314830+0000", 2024-09-13T11:24:03.910 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.243e-06 2024-09-13T11:24:03.910 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:03.910 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.910 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:24:03.910 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314965+0000", 2024-09-13T11:24:03.910 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00013470800000000001 2024-09-13T11:24:03.910 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:03.910 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.910 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:24:03.910 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314979+0000", 2024-09-13T11:24:03.911 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.4375999999999999e-05 2024-09-13T11:24:03.911 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:03.911 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.911 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:24:03.911 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314988+0000", 2024-09-13T11:24:03.911 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.8570000000000001e-06 2024-09-13T11:24:03.911 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:03.911 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.911 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:24:03.911 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316545+0000", 2024-09-13T11:24:03.911 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.001556878 2024-09-13T11:24:03.911 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:03.911 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.911 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-09-13T11:24:03.912 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316556+0000", 2024-09-13T11:24:03.912 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.1453e-05 2024-09-13T11:24:03.912 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:03.912 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.912 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:24:03.912 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.182124+0000", 2024-09-13T11:24:03.912 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.86556744200000002 2024-09-13T11:24:03.912 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:03.912 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.912 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:24:03.912 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.182246+0000", 2024-09-13T11:24:03.912 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00012210500000000001 2024-09-13T11:24:03.912 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:03.912 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.913 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1", 2024-09-13T11:24:03.913 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.182375+0000", 2024-09-13T11:24:03.913 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00012920999999999999 2024-09-13T11:24:03.913 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:03.913 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.913 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:24:03.913 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.185684+0000", 2024-09-13T11:24:03.913 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0033093969999999999 2024-09-13T11:24:03.913 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:03.913 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.913 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:03.913 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.188244+0000", 2024-09-13T11:24:03.913 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0025601550000000002 2024-09-13T11:24:03.913 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:03.914 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.914 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:24:03.914 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.188379+0000", 2024-09-13T11:24:03.914 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00013463400000000001 2024-09-13T11:24:03.914 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:03.914 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.914 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:24:03.914 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.188476+0000", 2024-09-13T11:24:03.914 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.7114000000000006e-05 2024-09-13T11:24:03.914 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:03.914 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:24:03.914 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:03.914 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:03.915 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.915 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:81 3.b 3:d705fd70:::benchmark_data_smithi177_31116_object80:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T11:24:03.915 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:48.315000+0000", 2024-09-13T11:24:03.915 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 75.530108229000007, 2024-09-13T11:24:03.915 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.87561418300000005, 2024-09-13T11:24:03.915 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:24:03.915 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:24:03.915 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:03.915 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:24:03.915 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:24:03.915 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:03.915 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 81 2024-09-13T11:24:03.915 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:03.916 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:24:03.916 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.916 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:24:03.916 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315000+0000", 2024-09-13T11:24:03.916 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:03.916 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:03.916 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.916 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:24:03.916 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315000+0000", 2024-09-13T11:24:03.916 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:03.916 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:03.916 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.916 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:24:03.916 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315002+0000", 2024-09-13T11:24:03.917 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.3799999999999999e-06 2024-09-13T11:24:03.917 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:03.917 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.917 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:24:03.917 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315119+0000", 2024-09-13T11:24:03.917 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000116925 2024-09-13T11:24:03.917 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:03.917 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.917 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:24:03.917 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315133+0000", 2024-09-13T11:24:03.917 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.3971999999999999e-05 2024-09-13T11:24:03.917 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:03.917 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.918 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:24:03.918 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315142+0000", 2024-09-13T11:24:03.918 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.4399999999999994e-06 2024-09-13T11:24:03.918 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:03.918 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.918 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:24:03.918 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316652+0000", 2024-09-13T11:24:03.918 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.001509907 2024-09-13T11:24:03.918 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:03.918 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.918 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-09-13T11:24:03.918 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316668+0000", 2024-09-13T11:24:03.918 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.5945999999999999e-05 2024-09-13T11:24:03.919 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:03.919 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.919 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:24:03.919 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.183405+0000", 2024-09-13T11:24:03.919 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.866737384 2024-09-13T11:24:03.919 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:03.919 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.919 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:24:03.919 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.183701+0000", 2024-09-13T11:24:03.919 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00029511099999999997 2024-09-13T11:24:03.919 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:03.919 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.919 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1", 2024-09-13T11:24:03.919 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.184078+0000", 2024-09-13T11:24:03.920 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00037741000000000001 2024-09-13T11:24:03.920 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:03.920 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.920 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:24:03.920 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.188504+0000", 2024-09-13T11:24:03.920 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0044260280000000003 2024-09-13T11:24:03.920 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:03.920 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.920 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:03.920 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.190562+0000", 2024-09-13T11:24:03.920 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0020580099999999999 2024-09-13T11:24:03.920 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:03.920 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.921 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:24:03.921 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.190592+0000", 2024-09-13T11:24:03.921 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.0057999999999999e-05 2024-09-13T11:24:03.921 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:03.921 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.921 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:24:03.921 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.190615+0000", 2024-09-13T11:24:03.921 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.2612e-05 2024-09-13T11:24:03.921 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:03.921 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:24:03.921 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:03.921 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:03.921 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.921 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:87 3.b 3:dd7de0ad:::benchmark_data_smithi177_31116_object86:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T11:24:03.922 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:48.315155+0000", 2024-09-13T11:24:03.922 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 75.529953431999999, 2024-09-13T11:24:03.922 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.87555916099999997, 2024-09-13T11:24:03.922 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:24:03.922 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:24:03.922 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:03.922 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:24:03.922 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:24:03.922 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:03.922 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 87 2024-09-13T11:24:03.922 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:03.922 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:24:03.922 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.922 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:24:03.923 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315155+0000", 2024-09-13T11:24:03.923 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:03.923 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:03.923 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.923 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:24:03.923 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315155+0000", 2024-09-13T11:24:03.923 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:03.923 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:03.923 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.923 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:24:03.923 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315157+0000", 2024-09-13T11:24:03.923 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.3829999999999999e-06 2024-09-13T11:24:03.923 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:03.924 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.924 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:24:03.924 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315271+0000", 2024-09-13T11:24:03.924 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000114539 2024-09-13T11:24:03.924 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:03.924 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.924 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:24:03.924 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315285+0000", 2024-09-13T11:24:03.924 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.4042e-05 2024-09-13T11:24:03.924 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:03.924 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.924 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:24:03.924 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315295+0000", 2024-09-13T11:24:03.925 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.2739999999999996e-06 2024-09-13T11:24:03.925 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:03.925 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.925 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:24:03.925 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316739+0000", 2024-09-13T11:24:03.925 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0014446960000000001 2024-09-13T11:24:03.925 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:03.925 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.925 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-09-13T11:24:03.925 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316747+0000", 2024-09-13T11:24:03.925 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.0790000000000001e-06 2024-09-13T11:24:03.925 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:03.925 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.925 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:24:03.926 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.184922+0000", 2024-09-13T11:24:03.926 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.86817446399999998 2024-09-13T11:24:03.926 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:03.926 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.926 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:24:03.926 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.185037+0000", 2024-09-13T11:24:03.926 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00011565099999999999 2024-09-13T11:24:03.926 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:03.926 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.926 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1", 2024-09-13T11:24:03.926 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.185171+0000", 2024-09-13T11:24:03.926 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00013316900000000001 2024-09-13T11:24:03.926 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:03.927 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.927 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:24:03.927 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.188520+0000", 2024-09-13T11:24:03.927 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0033489150000000001 2024-09-13T11:24:03.927 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:03.927 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.927 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:03.927 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.190652+0000", 2024-09-13T11:24:03.927 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0021325649999999999 2024-09-13T11:24:03.927 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:03.927 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.927 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:24:03.927 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.190686+0000", 2024-09-13T11:24:03.927 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.3827e-05 2024-09-13T11:24:03.928 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:03.928 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.928 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:24:03.928 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.190714+0000", 2024-09-13T11:24:03.928 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.8557000000000001e-05 2024-09-13T11:24:03.928 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:03.928 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:24:03.928 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:03.928 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:03.928 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.928 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:99 3.b 3:d011b600:::benchmark_data_smithi177_31116_object98:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T11:24:03.928 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:48.315307+0000", 2024-09-13T11:24:03.928 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 75.529801605000003, 2024-09-13T11:24:03.929 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.87551770799999995, 2024-09-13T11:24:03.929 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:24:03.929 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:24:03.929 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:03.929 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:24:03.929 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:24:03.929 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:03.929 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 99 2024-09-13T11:24:03.929 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:03.929 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:24:03.929 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.929 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:24:03.929 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315307+0000", 2024-09-13T11:24:03.929 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:03.930 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:03.930 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.930 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:24:03.930 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315307+0000", 2024-09-13T11:24:03.930 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:03.930 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:03.930 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.930 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:24:03.930 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315308+0000", 2024-09-13T11:24:03.930 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.316e-06 2024-09-13T11:24:03.930 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:03.930 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.930 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:24:03.931 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315453+0000", 2024-09-13T11:24:03.931 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00014473200000000001 2024-09-13T11:24:03.931 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:03.931 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.931 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:24:03.931 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315471+0000", 2024-09-13T11:24:03.931 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.8019000000000001e-05 2024-09-13T11:24:03.931 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:03.931 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.931 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:24:03.931 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315489+0000", 2024-09-13T11:24:03.931 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.7861999999999999e-05 2024-09-13T11:24:03.931 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:03.931 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.932 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:24:03.932 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316837+0000", 2024-09-13T11:24:03.932 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.001347787 2024-09-13T11:24:03.932 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:03.932 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.932 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-09-13T11:24:03.932 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316858+0000", 2024-09-13T11:24:03.932 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.1103000000000001e-05 2024-09-13T11:24:03.932 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:03.932 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.932 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:24:03.932 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.185713+0000", 2024-09-13T11:24:03.932 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.86885540800000005 2024-09-13T11:24:03.932 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:03.933 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.933 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:24:03.933 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.185820+0000", 2024-09-13T11:24:03.933 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000107081 2024-09-13T11:24:03.933 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:03.933 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.933 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1", 2024-09-13T11:24:03.933 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.185948+0000", 2024-09-13T11:24:03.933 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00012775800000000001 2024-09-13T11:24:03.933 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:03.933 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.933 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:24:03.933 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.190527+0000", 2024-09-13T11:24:03.933 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0045783910000000002 2024-09-13T11:24:03.934 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:03.934 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.934 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:03.934 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.190759+0000", 2024-09-13T11:24:03.934 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00023266199999999999 2024-09-13T11:24:03.934 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:03.934 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.934 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:24:03.934 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.190789+0000", 2024-09-13T11:24:03.934 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.9666999999999999e-05 2024-09-13T11:24:03.934 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:03.934 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.934 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:24:03.934 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.190825+0000", 2024-09-13T11:24:03.935 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.5921999999999999e-05 2024-09-13T11:24:03.935 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:03.935 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:24:03.935 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:03.935 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:03.935 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.935 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:108 3.b 3:d5c90217:::benchmark_data_smithi177_31116_object107:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T11:24:03.935 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:48.315513+0000", 2024-09-13T11:24:03.935 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 75.529595900000004, 2024-09-13T11:24:03.935 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.87544772500000001, 2024-09-13T11:24:03.935 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:24:03.935 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:24:03.935 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:03.935 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:24:03.936 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:24:03.936 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:03.936 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 108 2024-09-13T11:24:03.936 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:03.936 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:24:03.936 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.936 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:24:03.936 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315513+0000", 2024-09-13T11:24:03.936 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:03.936 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:03.936 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.936 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:24:03.936 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315513+0000", 2024-09-13T11:24:03.937 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:03.937 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:03.937 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.937 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:24:03.937 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315515+0000", 2024-09-13T11:24:03.937 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.9369999999999998e-06 2024-09-13T11:24:03.937 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:03.937 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.937 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:24:03.937 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315644+0000", 2024-09-13T11:24:03.937 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000129011 2024-09-13T11:24:03.937 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:03.937 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.937 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:24:03.938 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315657+0000", 2024-09-13T11:24:03.938 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.3689999999999999e-05 2024-09-13T11:24:03.938 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:03.938 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.938 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:24:03.938 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315668+0000", 2024-09-13T11:24:03.938 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0589e-05 2024-09-13T11:24:03.938 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:03.938 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.938 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:24:03.938 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316925+0000", 2024-09-13T11:24:03.938 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0012568340000000001 2024-09-13T11:24:03.938 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:03.939 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.939 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-09-13T11:24:03.939 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316932+0000", 2024-09-13T11:24:03.939 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.2230000000000001e-06 2024-09-13T11:24:03.939 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:03.939 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.939 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:24:03.939 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.186619+0000", 2024-09-13T11:24:03.939 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.86968679599999998 2024-09-13T11:24:03.939 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:03.939 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.939 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:24:03.940 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.186914+0000", 2024-09-13T11:24:03.940 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00029528900000000001 2024-09-13T11:24:03.940 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:03.940 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.940 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1", 2024-09-13T11:24:03.940 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.187217+0000", 2024-09-13T11:24:03.940 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00030232099999999998 2024-09-13T11:24:03.940 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:03.940 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.941 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:24:03.941 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.190532+0000", 2024-09-13T11:24:03.941 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.003315391 2024-09-13T11:24:03.941 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:03.941 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.941 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:03.941 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.190875+0000", 2024-09-13T11:24:03.941 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00034326300000000001 2024-09-13T11:24:03.941 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:03.941 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.941 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:24:03.941 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.190925+0000", 2024-09-13T11:24:03.941 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.9429999999999999e-05 2024-09-13T11:24:03.941 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:03.942 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.942 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:24:03.942 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.190961+0000", 2024-09-13T11:24:03.942 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.5951000000000002e-05 2024-09-13T11:24:03.942 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:03.942 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:24:03.942 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:03.942 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:03.942 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.942 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:172 3.b 3:dbec1eeb:::benchmark_data_smithi177_31116_object171:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T11:24:03.942 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:48.315684+0000", 2024-09-13T11:24:03.942 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 75.529424743000007, 2024-09-13T11:24:03.942 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.87615312999999995, 2024-09-13T11:24:03.942 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:24:03.943 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:24:03.943 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:03.943 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:24:03.943 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:24:03.943 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:03.943 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 172 2024-09-13T11:24:03.943 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:03.943 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:24:03.943 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.943 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:24:03.943 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315684+0000", 2024-09-13T11:24:03.943 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:03.943 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:03.944 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.944 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:24:03.944 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315684+0000", 2024-09-13T11:24:03.944 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:03.944 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:03.944 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.944 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:24:03.944 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315686+0000", 2024-09-13T11:24:03.944 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.886e-06 2024-09-13T11:24:03.944 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:03.944 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.944 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:24:03.944 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315820+0000", 2024-09-13T11:24:03.944 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00013388500000000001 2024-09-13T11:24:03.945 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:03.945 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.945 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:24:03.945 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315837+0000", 2024-09-13T11:24:03.945 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.6756e-05 2024-09-13T11:24:03.945 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:03.945 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.945 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:24:03.945 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315851+0000", 2024-09-13T11:24:03.945 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.4566000000000001e-05 2024-09-13T11:24:03.945 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:03.945 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.945 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:24:03.945 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.317021+0000", 2024-09-13T11:24:03.946 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0011701769999999999 2024-09-13T11:24:03.946 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:03.946 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.946 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-09-13T11:24:03.946 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.317037+0000", 2024-09-13T11:24:03.946 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.609e-05 2024-09-13T11:24:03.946 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:03.946 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.946 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:24:03.946 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.188634+0000", 2024-09-13T11:24:03.946 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.87159711900000003 2024-09-13T11:24:03.946 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:03.946 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.946 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:24:03.947 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.188974+0000", 2024-09-13T11:24:03.947 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00033922299999999999 2024-09-13T11:24:03.947 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:03.947 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.947 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1", 2024-09-13T11:24:03.947 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.189510+0000", 2024-09-13T11:24:03.947 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00053618999999999997 2024-09-13T11:24:03.947 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:03.947 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.947 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:03.947 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.191027+0000", 2024-09-13T11:24:03.947 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0015168390000000001 2024-09-13T11:24:03.947 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:03.947 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.948 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:24:03.948 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.191766+0000", 2024-09-13T11:24:03.948 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000739064 2024-09-13T11:24:03.948 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:03.948 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.948 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:24:03.948 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.191810+0000", 2024-09-13T11:24:03.948 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.3755999999999999e-05 2024-09-13T11:24:03.948 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:03.948 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.948 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:24:03.948 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.191837+0000", 2024-09-13T11:24:03.948 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.7579000000000001e-05 2024-09-13T11:24:03.948 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:03.949 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:24:03.949 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:03.949 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:03.949 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.949 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:177 3.b 3:de89c779:::benchmark_data_smithi177_31116_object176:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T11:24:03.949 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:48.315898+0000", 2024-09-13T11:24:03.949 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 75.529210427999999, 2024-09-13T11:24:03.949 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.87600712599999997, 2024-09-13T11:24:03.949 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:24:03.949 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:24:03.949 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:03.949 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:24:03.949 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:24:03.950 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:03.950 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 177 2024-09-13T11:24:03.950 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:03.950 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:24:03.950 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.950 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:24:03.950 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315898+0000", 2024-09-13T11:24:03.950 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:03.950 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:03.950 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.950 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:24:03.950 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315898+0000", 2024-09-13T11:24:03.950 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:03.950 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:03.951 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.951 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:24:03.951 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315900+0000", 2024-09-13T11:24:03.951 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.4929999999999999e-06 2024-09-13T11:24:03.951 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:03.951 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.951 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:24:03.951 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316027+0000", 2024-09-13T11:24:03.951 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000127568 2024-09-13T11:24:03.951 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:03.951 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.951 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:24:03.951 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316048+0000", 2024-09-13T11:24:03.951 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.0795999999999999e-05 2024-09-13T11:24:03.952 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:03.952 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.952 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:24:03.952 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316061+0000", 2024-09-13T11:24:03.952 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.3283e-05 2024-09-13T11:24:03.952 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:03.952 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.952 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:24:03.952 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.317107+0000", 2024-09-13T11:24:03.952 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0010452949999999999 2024-09-13T11:24:03.952 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:03.952 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.952 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-09-13T11:24:03.953 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.317114+0000", 2024-09-13T11:24:03.953 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.4569999999999999e-06 2024-09-13T11:24:03.953 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:03.953 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.953 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:24:03.953 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.190017+0000", 2024-09-13T11:24:03.953 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.87290258499999995 2024-09-13T11:24:03.953 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:03.953 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.953 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:24:03.953 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.190138+0000", 2024-09-13T11:24:03.953 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000121418 2024-09-13T11:24:03.953 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:03.953 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.954 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1", 2024-09-13T11:24:03.954 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.190216+0000", 2024-09-13T11:24:03.954 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.7410999999999995e-05 2024-09-13T11:24:03.954 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:03.954 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.954 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:03.954 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.191755+0000", 2024-09-13T11:24:03.954 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0015395420000000001 2024-09-13T11:24:03.954 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:03.954 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.954 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:24:03.954 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.191843+0000", 2024-09-13T11:24:03.954 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.8022999999999994e-05 2024-09-13T11:24:03.954 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:03.955 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.955 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:24:03.955 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.191876+0000", 2024-09-13T11:24:03.955 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.2619000000000001e-05 2024-09-13T11:24:03.955 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:03.955 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.955 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:24:03.955 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.191905+0000", 2024-09-13T11:24:03.955 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.9635999999999999e-05 2024-09-13T11:24:03.955 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:03.955 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:24:03.955 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:03.955 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:03.955 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.956 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:187 3.b 3:da03c8dd:::benchmark_data_smithi177_31116_object186:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T11:24:03.956 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:48.316083+0000", 2024-09-13T11:24:03.956 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 75.529025472000001, 2024-09-13T11:24:03.956 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.87835499500000003, 2024-09-13T11:24:03.956 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:24:03.956 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:24:03.956 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:03.956 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:24:03.956 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:24:03.956 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:03.956 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 187 2024-09-13T11:24:03.956 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:03.956 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:24:03.957 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.957 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:24:03.957 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316083+0000", 2024-09-13T11:24:03.957 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:03.957 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:03.957 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.957 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:24:03.957 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316083+0000", 2024-09-13T11:24:03.957 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:03.957 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:03.957 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.957 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:24:03.957 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316086+0000", 2024-09-13T11:24:03.957 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.5560000000000002e-06 2024-09-13T11:24:03.958 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:03.958 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.958 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:24:03.958 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316199+0000", 2024-09-13T11:24:03.958 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000113522 2024-09-13T11:24:03.958 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:03.958 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.958 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:24:03.958 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316215+0000", 2024-09-13T11:24:03.958 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.5265999999999999e-05 2024-09-13T11:24:03.958 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:03.958 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.958 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:24:03.959 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316226+0000", 2024-09-13T11:24:03.959 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0967e-05 2024-09-13T11:24:03.959 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:03.959 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.959 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:24:03.959 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.317198+0000", 2024-09-13T11:24:03.959 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00097194100000000004 2024-09-13T11:24:03.959 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:03.959 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.959 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-09-13T11:24:03.959 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.317213+0000", 2024-09-13T11:24:03.959 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.5472999999999999e-05 2024-09-13T11:24:03.959 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:03.959 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.960 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:24:03.960 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.191077+0000", 2024-09-13T11:24:03.960 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.87386435200000001 2024-09-13T11:24:03.960 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:03.960 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.960 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:24:03.960 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.191181+0000", 2024-09-13T11:24:03.960 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000103618 2024-09-13T11:24:03.960 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:03.960 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.960 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1", 2024-09-13T11:24:03.960 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.191305+0000", 2024-09-13T11:24:03.960 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00012358599999999999 2024-09-13T11:24:03.961 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:03.961 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.961 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:03.961 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.194279+0000", 2024-09-13T11:24:03.961 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0029746970000000001 2024-09-13T11:24:03.961 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:03.961 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.961 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:24:03.961 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.194336+0000", 2024-09-13T11:24:03.961 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.7244000000000003e-05 2024-09-13T11:24:03.961 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:03.961 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.961 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:24:03.961 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.194397+0000", 2024-09-13T11:24:03.962 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.0278000000000001e-05 2024-09-13T11:24:03.962 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:03.962 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.962 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:24:03.962 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.194438+0000", 2024-09-13T11:24:03.962 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.1495000000000002e-05 2024-09-13T11:24:03.962 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:03.962 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:24:03.962 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:03.962 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:03.962 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.962 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:200 3.b 3:d90039f1:::benchmark_data_smithi177_31116_object199:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T11:24:03.962 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:48.316280+0000", 2024-09-13T11:24:03.962 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 75.528829006999999, 2024-09-13T11:24:03.963 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.87879273000000002, 2024-09-13T11:24:03.963 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:24:03.963 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:24:03.963 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:03.963 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:24:03.963 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:24:03.963 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:03.963 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 200 2024-09-13T11:24:03.963 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:03.963 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:24:03.963 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.963 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:24:03.963 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316280+0000", 2024-09-13T11:24:03.963 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:03.964 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:03.964 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.964 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:24:03.964 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316280+0000", 2024-09-13T11:24:03.964 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:03.964 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:03.964 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.964 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:24:03.964 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316282+0000", 2024-09-13T11:24:03.964 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.497e-06 2024-09-13T11:24:03.964 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:03.964 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.964 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:24:03.964 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316411+0000", 2024-09-13T11:24:03.965 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00012924799999999999 2024-09-13T11:24:03.965 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:03.965 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.965 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:24:03.965 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316426+0000", 2024-09-13T11:24:03.965 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.4183e-05 2024-09-13T11:24:03.965 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:03.965 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.965 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:24:03.965 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316435+0000", 2024-09-13T11:24:03.965 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.0990000000000005e-06 2024-09-13T11:24:03.965 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:03.965 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.966 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:24:03.966 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.317250+0000", 2024-09-13T11:24:03.966 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00081501100000000003 2024-09-13T11:24:03.966 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:03.966 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.966 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-09-13T11:24:03.966 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.317257+0000", 2024-09-13T11:24:03.966 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.2169999999999997e-06 2024-09-13T11:24:03.966 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:03.966 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.966 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:24:03.966 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.191936+0000", 2024-09-13T11:24:03.966 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.87467922099999995 2024-09-13T11:24:03.966 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:03.967 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.967 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:24:03.967 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.192056+0000", 2024-09-13T11:24:03.967 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000119638 2024-09-13T11:24:03.967 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:03.967 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.967 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1", 2024-09-13T11:24:03.967 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.192184+0000", 2024-09-13T11:24:03.967 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00012813399999999999 2024-09-13T11:24:03.967 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:03.967 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.967 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:03.967 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.194324+0000", 2024-09-13T11:24:03.967 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0021399420000000001 2024-09-13T11:24:03.968 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:03.968 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.968 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:24:03.968 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.194802+0000", 2024-09-13T11:24:03.968 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00047798799999999999 2024-09-13T11:24:03.968 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:03.968 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.968 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:24:03.968 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.194951+0000", 2024-09-13T11:24:03.968 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0001494 2024-09-13T11:24:03.968 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:03.968 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.968 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:24:03.969 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.195072+0000", 2024-09-13T11:24:03.969 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00012115199999999999 2024-09-13T11:24:03.969 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:03.969 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:24:03.969 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:03.969 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:03.969 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.969 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:25525 3.1 3:8e5634af:::benchmark_data_smithi177_31116_object25524:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e56)", 2024-09-13T11:24:03.969 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:04.981590+0000", 2024-09-13T11:24:03.969 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 58.863518730000003, 2024-09-13T11:24:03.969 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.23642401599999999, 2024-09-13T11:24:03.969 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:24:03.969 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:24:03.969 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:03.970 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:24:03.970 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:24:03.970 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:03.970 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 25525 2024-09-13T11:24:03.970 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:03.970 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:24:03.970 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.970 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:24:03.970 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.981590+0000", 2024-09-13T11:24:03.970 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:03.970 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:03.970 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.970 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:24:03.970 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.981590+0000", 2024-09-13T11:24:03.971 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:03.971 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:03.971 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.971 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:24:03.971 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.981598+0000", 2024-09-13T11:24:03.971 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.4230000000000005e-06 2024-09-13T11:24:03.971 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:03.971 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.971 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:24:03.971 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.981652+0000", 2024-09-13T11:24:03.971 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.3226000000000002e-05 2024-09-13T11:24:03.971 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:03.971 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.971 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:24:03.972 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.981680+0000", 2024-09-13T11:24:03.972 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.8348999999999999e-05 2024-09-13T11:24:03.972 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:03.972 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.972 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:24:03.972 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.981695+0000", 2024-09-13T11:24:03.972 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.5085999999999999e-05 2024-09-13T11:24:03.972 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:03.972 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.972 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:24:03.972 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.981809+0000", 2024-09-13T11:24:03.972 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00011382700000000001 2024-09-13T11:24:03.972 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:03.973 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.973 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:24:03.973 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.982042+0000", 2024-09-13T11:24:03.973 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000232697 2024-09-13T11:24:03.973 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:03.973 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.973 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 11", 2024-09-13T11:24:03.973 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.982261+0000", 2024-09-13T11:24:03.973 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00021918400000000001 2024-09-13T11:24:03.973 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:03.973 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.973 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:24:03.973 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.984553+0000", 2024-09-13T11:24:03.973 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0022918460000000002 2024-09-13T11:24:03.974 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:03.974 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.974 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:03.974 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:05.217842+0000", 2024-09-13T11:24:03.974 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.233289106 2024-09-13T11:24:03.974 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:03.974 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.974 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:24:03.974 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:05.217940+0000", 2024-09-13T11:24:03.974 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.7936999999999994e-05 2024-09-13T11:24:03.974 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:03.974 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.974 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:24:03.974 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:05.218014+0000", 2024-09-13T11:24:03.975 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.4334999999999999e-05 2024-09-13T11:24:03.975 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:03.975 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:24:03.975 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:03.975 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:03.975 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.975 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "MOSDPGPushReply(3.16 76/70 [PushReplyOp(3:6efa689b:::benchmark_data_smithi177_31116_object24598:head)])", 2024-09-13T11:24:03.975 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:32.699090+0000", 2024-09-13T11:24:03.975 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 31.14601832, 2024-09-13T11:24:03.975 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.232999069, 2024-09-13T11:24:03.975 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:24:03.975 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:24:03.975 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "reached pg", 2024-09-13T11:24:03.975 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:24:03.976 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.976 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:24:03.976 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:32.699090+0000", 2024-09-13T11:24:03.976 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:03.976 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:03.976 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.976 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:24:03.976 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:32.699090+0000", 2024-09-13T11:24:03.976 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:03.976 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:03.976 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.976 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:24:03.976 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:32.699092+0000", 2024-09-13T11:24:03.976 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.4559999999999999e-06 2024-09-13T11:24:03.977 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:03.977 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.977 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:24:03.977 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:32.699100+0000", 2024-09-13T11:24:03.977 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.1899999999999995e-06 2024-09-13T11:24:03.977 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:03.977 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.977 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:24:03.977 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:32.931118+0000", 2024-09-13T11:24:03.977 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.23201782900000001 2024-09-13T11:24:03.977 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:03.977 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.977 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:24:03.977 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:32.931132+0000", 2024-09-13T11:24:03.978 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.4212999999999999e-05 2024-09-13T11:24:03.978 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:03.978 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.978 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:24:03.978 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:32.931223+0000", 2024-09-13T11:24:03.978 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.1058999999999997e-05 2024-09-13T11:24:03.978 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:03.978 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.978 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:24:03.978 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:32.932089+0000", 2024-09-13T11:24:03.978 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00086632199999999999 2024-09-13T11:24:03.978 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:03.978 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:24:03.978 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:03.979 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:03.979 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.979 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop(client.4665.0:79801 3.5 e102/67 3:a6efdf84:::benchmark_data_smithi177_31116_object79800:head v 102'3851, mlcod=102'3848)", 2024-09-13T11:24:03.979 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:24:00.403754+0000", 2024-09-13T11:24:03.979 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 3.441354655, 2024-09-13T11:24:03.979 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.31749079899999999, 2024-09-13T11:24:03.979 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:24:03.979 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:24:03.979 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:03.979 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:24:03.979 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.979 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:24:03.979 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:00.403754+0000", 2024-09-13T11:24:03.979 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:03.980 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:03.980 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.980 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:24:03.980 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:00.403754+0000", 2024-09-13T11:24:03.980 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:03.980 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:03.980 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.980 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:24:03.980 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:00.403758+0000", 2024-09-13T11:24:03.980 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.507e-06 2024-09-13T11:24:03.980 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:03.980 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.980 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:24:03.981 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:00.403819+0000", 2024-09-13T11:24:03.981 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.1042000000000003e-05 2024-09-13T11:24:03.981 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:03.981 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.981 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:24:03.981 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:00.717963+0000", 2024-09-13T11:24:03.981 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.314144594 2024-09-13T11:24:03.981 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:03.981 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.981 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:24:03.981 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:00.717976+0000", 2024-09-13T11:24:03.981 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.3029e-05 2024-09-13T11:24:03.981 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:03.981 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.982 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:24:03.982 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:00.718143+0000", 2024-09-13T11:24:03.982 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00016639000000000001 2024-09-13T11:24:03.982 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:03.982 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.982 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:24:03.982 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:00.718184+0000", 2024-09-13T11:24:03.982 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.1801999999999997e-05 2024-09-13T11:24:03.982 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:03.982 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.982 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:24:03.982 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:00.721162+0000", 2024-09-13T11:24:03.982 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0029772800000000001 2024-09-13T11:24:03.983 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:03.983 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:03.983 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:24:03.983 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:00.721245+0000", 2024-09-13T11:24:03.983 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.3154999999999999e-05 2024-09-13T11:24:03.983 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:03.983 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:24:03.983 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:03.983 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:03.983 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:24:03.983 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-09-13T11:24:03.984 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-13T11:24:03.985 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-13T11:24:03.986 INFO:tasks.ceph.osd.13.smithi092.stderr:2024-09-13T11:24:03.943+0000 7f3e4ba0f640 -1 osd.13 105 handle_osd_map injecting map crc failure 2024-09-13T11:24:03.986 INFO:tasks.ceph.osd.9.smithi092.stderr:2024-09-13T11:24:03.979+0000 7ff5ab3d0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26348) UID: 0 2024-09-13T11:24:04.007 INFO:tasks.radosbench.radosbench.0.smithi177.stdout: 97 16 82849 82833 53.3663 25.5625 0.00641094 0.0187303 2024-09-13T11:24:04.086 INFO:tasks.ceph.osd.11.smithi177.stderr:2024-09-13T11:24:04.084+0000 7f24792c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26580) UID: 0 2024-09-13T11:24:04.153 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-13T11:24:04.153 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [ 2024-09-13T11:24:04.153 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:04.153 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:83098 3.1d 3:bd20941e:::benchmark_data_smithi177_31116_object83097:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106)", 2024-09-13T11:24:04.154 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:24:04.148449+0000", 2024-09-13T11:24:04.154 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 0.0032118559999999999, 2024-09-13T11:24:04.154 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.003244996, 2024-09-13T11:24:04.154 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:24:04.154 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:24:04.155 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:04.155 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:24:04.155 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:24:04.155 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:04.155 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 83098 2024-09-13T11:24:04.155 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:04.156 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:24:04.156 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:04.156 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:24:04.156 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.148449+0000", 2024-09-13T11:24:04.157 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:04.157 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:04.157 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:04.157 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:24:04.157 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.148449+0000", 2024-09-13T11:24:04.157 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:04.158 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:04.158 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:04.158 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:24:04.158 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.148452+0000", 2024-09-13T11:24:04.158 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.9189999999999999e-06 2024-09-13T11:24:04.159 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:04.159 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:04.159 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:24:04.159 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.148538+0000", 2024-09-13T11:24:04.159 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.6148000000000003e-05 2024-09-13T11:24:04.159 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:04.159 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:04.159 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:24:04.159 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.148552+0000", 2024-09-13T11:24:04.159 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4100999999999999e-05 2024-09-13T11:24:04.160 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:04.160 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:04.160 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:24:04.160 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.148560+0000", 2024-09-13T11:24:04.160 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.5720000000000003e-06 2024-09-13T11:24:04.160 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:04.160 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:04.160 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:24:04.160 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.148617+0000", 2024-09-13T11:24:04.160 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.6648999999999999e-05 2024-09-13T11:24:04.160 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:04.160 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:04.161 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:24:04.161 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.148778+0000", 2024-09-13T11:24:04.161 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000160841 2024-09-13T11:24:04.161 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:04.161 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:04.161 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 15", 2024-09-13T11:24:04.161 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.148885+0000", 2024-09-13T11:24:04.161 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000107042 2024-09-13T11:24:04.161 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:04.161 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:04.161 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:04.161 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.151161+0000", 2024-09-13T11:24:04.162 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0022757379999999998 2024-09-13T11:24:04.162 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:04.162 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:04.162 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:24:04.162 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.151615+0000", 2024-09-13T11:24:04.162 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00045458599999999999 2024-09-13T11:24:04.162 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:04.162 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:04.162 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:24:04.162 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.151652+0000", 2024-09-13T11:24:04.162 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.6996000000000003e-05 2024-09-13T11:24:04.162 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:04.163 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:04.163 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:24:04.163 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.151694+0000", 2024-09-13T11:24:04.163 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.1403999999999999e-05 2024-09-13T11:24:04.163 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:04.164 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:24:04.164 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:04.164 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:04.164 INFO:teuthology.orchestra.run.smithi162.stdout: ], 2024-09-13T11:24:04.164 INFO:teuthology.orchestra.run.smithi162.stdout: "num_ops": 1 2024-09-13T11:24:04.164 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-13T11:24:04.173 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-13T11:24:04.218 INFO:tasks.ceph.osd.14.smithi162.stderr:2024-09-13T11:24:04.217+0000 7f235c0db640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26319) UID: 0 2024-09-13T11:24:04.289 INFO:tasks.ceph.osd.11.smithi177.stderr:2024-09-13T11:24:04.288+0000 7f24792c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26580) UID: 0 2024-09-13T11:24:04.309 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-13T11:24:04.309 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [], 2024-09-13T11:24:04.309 INFO:teuthology.orchestra.run.smithi162.stdout: "complaint_time": 30, 2024-09-13T11:24:04.309 INFO:teuthology.orchestra.run.smithi162.stdout: "num_blocked_ops": 0 2024-09-13T11:24:04.309 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-13T11:24:04.325 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-13T11:24:04.390 INFO:tasks.ceph.osd.12.smithi001.stderr:2024-09-13T11:24:04.383+0000 7f892c6e6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26314) UID: 0 2024-09-13T11:24:04.487 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-13T11:24:04.487 INFO:teuthology.orchestra.run.smithi162.stdout: "size": 20, 2024-09-13T11:24:04.487 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 600, 2024-09-13T11:24:04.488 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [ 2024-09-13T11:24:04.488 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:04.488 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.4665.0:171 3.9 e41/30 3:9e1c7eeb:::benchmark_data_smithi177_31116_object170:head v 41'13, mlcod=41'9)", 2024-09-13T11:24:04.488 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:22:47.418602+0000", 2024-09-13T11:24:04.488 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 77.064968856999997, 2024-09-13T11:24:04.488 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.67235367300000004, 2024-09-13T11:24:04.488 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:24:04.488 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:24:04.488 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:04.489 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:24:04.489 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:04.489 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:24:04.489 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:47.418602+0000", 2024-09-13T11:24:04.489 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:04.489 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:04.489 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:04.489 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:24:04.489 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:47.418602+0000", 2024-09-13T11:24:04.489 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:04.489 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:04.489 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:04.489 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:24:04.490 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:47.418605+0000", 2024-09-13T11:24:04.490 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.2200000000000001e-06 2024-09-13T11:24:04.490 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:04.490 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:04.490 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:24:04.490 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:47.418668+0000", 2024-09-13T11:24:04.490 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.2921000000000003e-05 2024-09-13T11:24:04.490 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:04.490 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:04.490 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:24:04.493 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:48.089380+0000", 2024-09-13T11:24:04.494 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.67071191100000005 2024-09-13T11:24:04.494 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:04.494 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:04.494 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:24:04.494 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:48.089423+0000", 2024-09-13T11:24:04.494 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.2916e-05 2024-09-13T11:24:04.494 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:04.494 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:04.494 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:24:04.494 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:48.089533+0000", 2024-09-13T11:24:04.494 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000109996 2024-09-13T11:24:04.495 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:04.495 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:04.495 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:24:04.495 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:48.089570+0000", 2024-09-13T11:24:04.495 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.7314999999999998e-05 2024-09-13T11:24:04.495 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:04.495 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:04.495 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:24:04.495 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:48.090852+0000", 2024-09-13T11:24:04.495 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001281641 2024-09-13T11:24:04.495 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:04.495 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:04.495 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:24:04.495 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:48.090956+0000", 2024-09-13T11:24:04.496 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00010375299999999999 2024-09-13T11:24:04.496 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:04.496 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:24:04.496 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:04.496 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:04.496 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:04.496 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPushReply(3.19 71/70 [PushReplyOp(3:980058cd:::benchmark_data_smithi177_31116_object4309:head)])", 2024-09-13T11:24:04.496 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:25.123140+0000", 2024-09-13T11:24:04.496 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 39.360431177999999, 2024-09-13T11:24:04.496 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.16436381899999999, 2024-09-13T11:24:04.496 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:24:04.496 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:24:04.497 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-09-13T11:24:04.497 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:24:04.497 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:04.497 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:24:04.497 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.123140+0000", 2024-09-13T11:24:04.497 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:04.497 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:04.497 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:04.497 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:24:04.497 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.123140+0000", 2024-09-13T11:24:04.497 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:04.497 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:04.497 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:04.498 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:24:04.498 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.123141+0000", 2024-09-13T11:24:04.498 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.562e-06 2024-09-13T11:24:04.498 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:04.498 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:04.498 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:24:04.498 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.123151+0000", 2024-09-13T11:24:04.498 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.3030000000000005e-06 2024-09-13T11:24:04.498 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:04.498 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:04.498 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:24:04.498 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.285828+0000", 2024-09-13T11:24:04.498 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.16267689399999999 2024-09-13T11:24:04.498 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:04.499 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:04.499 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:24:04.499 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.285838+0000", 2024-09-13T11:24:04.499 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0016e-05 2024-09-13T11:24:04.499 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:04.499 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:04.499 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:24:04.499 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.285966+0000", 2024-09-13T11:24:04.499 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00012870500000000001 2024-09-13T11:24:04.499 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:04.499 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:04.499 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:24:04.500 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.287504+0000", 2024-09-13T11:24:04.500 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001537339 2024-09-13T11:24:04.500 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:04.500 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:24:04.500 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:04.500 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:04.500 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:04.501 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:53947 3.19 3:9ff49d75:::benchmark_data_smithi177_31116_object53946:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e74)", 2024-09-13T11:24:04.501 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:30.219526+0000", 2024-09-13T11:24:04.501 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 34.264044749, 2024-09-13T11:24:04.501 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.094435565999999999, 2024-09-13T11:24:04.501 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:24:04.501 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:24:04.501 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:04.501 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:24:04.501 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:24:04.502 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:04.502 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 53947 2024-09-13T11:24:04.502 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:04.502 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:24:04.502 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:04.502 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:24:04.502 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.219526+0000", 2024-09-13T11:24:04.502 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:04.502 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:04.502 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:04.502 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:24:04.502 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.219526+0000", 2024-09-13T11:24:04.502 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:04.502 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:04.503 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:04.503 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:24:04.503 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.219533+0000", 2024-09-13T11:24:04.503 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.8549999999999996e-06 2024-09-13T11:24:04.503 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:04.503 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:04.503 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:24:04.503 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.219594+0000", 2024-09-13T11:24:04.503 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.0979e-05 2024-09-13T11:24:04.503 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:04.503 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:04.503 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:24:04.503 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.219612+0000", 2024-09-13T11:24:04.503 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.7396000000000001e-05 2024-09-13T11:24:04.504 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:04.504 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:04.504 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:24:04.504 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.219620+0000", 2024-09-13T11:24:04.504 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.6009999999999995e-06 2024-09-13T11:24:04.504 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:04.504 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:04.504 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:24:04.504 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.223111+0000", 2024-09-13T11:24:04.504 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0034904319999999999 2024-09-13T11:24:04.504 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:04.504 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:04.505 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:24:04.505 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.223349+0000", 2024-09-13T11:24:04.505 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00023811500000000001 2024-09-13T11:24:04.505 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:04.505 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:04.505 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 1,9", 2024-09-13T11:24:04.505 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.223615+0000", 2024-09-13T11:24:04.505 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00026669600000000002 2024-09-13T11:24:04.505 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:04.505 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:04.505 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:24:04.505 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.225114+0000", 2024-09-13T11:24:04.505 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001499071 2024-09-13T11:24:04.505 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:04.506 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:04.506 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:04.506 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.226428+0000", 2024-09-13T11:24:04.506 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0013132230000000001 2024-09-13T11:24:04.506 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:04.506 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:04.506 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:04.506 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.313440+0000", 2024-09-13T11:24:04.506 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.087011863999999994 2024-09-13T11:24:04.506 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:04.506 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:04.507 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:24:04.507 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.313508+0000", 2024-09-13T11:24:04.507 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.8817000000000002e-05 2024-09-13T11:24:04.507 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:04.507 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:04.507 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:24:04.507 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.313962+0000", 2024-09-13T11:24:04.507 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00045351700000000002 2024-09-13T11:24:04.507 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:04.507 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:24:04.507 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:04.507 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:04.507 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:04.508 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPushReply(3.19 74/70 [PushReplyOp(3:9b6cad3a:::benchmark_data_smithi177_31116_object16698:head)])", 2024-09-13T11:24:04.508 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:30.313297+0000", 2024-09-13T11:24:04.508 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 34.170274501999998, 2024-09-13T11:24:04.508 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.162355306, 2024-09-13T11:24:04.508 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:24:04.508 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:24:04.508 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-09-13T11:24:04.508 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:24:04.508 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:04.508 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:24:04.508 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.313297+0000", 2024-09-13T11:24:04.509 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:04.509 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:04.509 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:04.509 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:24:04.509 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.313297+0000", 2024-09-13T11:24:04.509 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:04.509 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:04.509 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:04.509 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:24:04.509 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.313298+0000", 2024-09-13T11:24:04.509 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.61e-06 2024-09-13T11:24:04.509 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:04.510 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:04.510 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:24:04.510 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.313306+0000", 2024-09-13T11:24:04.510 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.9950000000000005e-06 2024-09-13T11:24:04.510 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:04.510 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:04.510 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:24:04.510 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.473757+0000", 2024-09-13T11:24:04.510 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.16045083099999999 2024-09-13T11:24:04.510 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:04.510 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:04.510 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:24:04.511 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.473764+0000", 2024-09-13T11:24:04.511 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.4769999999999997e-06 2024-09-13T11:24:04.511 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:04.511 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:04.511 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:24:04.511 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.473870+0000", 2024-09-13T11:24:04.511 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00010529100000000001 2024-09-13T11:24:04.511 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:04.511 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:04.511 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:24:04.511 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.475652+0000", 2024-09-13T11:24:04.511 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001782102 2024-09-13T11:24:04.511 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:04.512 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:24:04.512 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:04.512 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:04.512 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:04.512 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:54073 3.19 3:9f968cf6:::benchmark_data_smithi177_31116_object54072:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e74)", 2024-09-13T11:24:04.512 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:30.331382+0000", 2024-09-13T11:24:04.512 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 34.152188807999998, 2024-09-13T11:24:04.512 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.15619861600000001, 2024-09-13T11:24:04.512 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:24:04.512 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:24:04.513 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:04.513 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:24:04.513 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:24:04.513 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:04.513 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 54073 2024-09-13T11:24:04.513 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:04.513 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:24:04.513 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:04.513 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:24:04.513 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.331382+0000", 2024-09-13T11:24:04.513 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:04.513 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:04.513 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:04.514 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:24:04.514 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.331382+0000", 2024-09-13T11:24:04.514 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:04.514 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:04.514 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:04.514 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:24:04.514 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.331386+0000", 2024-09-13T11:24:04.514 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.5769999999999998e-06 2024-09-13T11:24:04.514 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:04.514 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:04.514 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:24:04.514 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.331430+0000", 2024-09-13T11:24:04.515 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.4172999999999999e-05 2024-09-13T11:24:04.515 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:04.515 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:04.515 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:24:04.515 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.331450+0000", 2024-09-13T11:24:04.515 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.9922999999999999e-05 2024-09-13T11:24:04.516 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:04.516 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:04.516 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:24:04.516 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.331460+0000", 2024-09-13T11:24:04.516 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0444e-05 2024-09-13T11:24:04.516 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:04.516 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:04.517 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:24:04.517 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.331590+0000", 2024-09-13T11:24:04.517 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00013000899999999999 2024-09-13T11:24:04.517 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:04.517 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:04.517 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:24:04.517 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.331929+0000", 2024-09-13T11:24:04.517 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00033835400000000002 2024-09-13T11:24:04.517 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:04.517 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:04.517 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 1,9", 2024-09-13T11:24:04.518 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.332198+0000", 2024-09-13T11:24:04.518 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00026941900000000002 2024-09-13T11:24:04.518 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:04.518 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:04.518 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:24:04.518 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.334200+0000", 2024-09-13T11:24:04.518 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0020020179999999999 2024-09-13T11:24:04.518 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:04.518 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:04.518 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:04.518 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.334826+0000", 2024-09-13T11:24:04.518 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00062540600000000001 2024-09-13T11:24:04.518 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:04.518 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:04.519 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:04.519 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.486479+0000", 2024-09-13T11:24:04.519 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.151653395 2024-09-13T11:24:04.519 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:04.519 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:04.519 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:24:04.519 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.486691+0000", 2024-09-13T11:24:04.519 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00021176499999999999 2024-09-13T11:24:04.519 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:04.519 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:04.519 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:24:04.519 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.487581+0000", 2024-09-13T11:24:04.519 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00089013299999999998 2024-09-13T11:24:04.520 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:04.520 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:24:04.520 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:04.520 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:04.520 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:04.520 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop_reply(client.4665.0:54073 3.19 e74/70 ondisk, result = 0)", 2024-09-13T11:24:04.520 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:30.334897+0000", 2024-09-13T11:24:04.520 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 34.148674030999999, 2024-09-13T11:24:04.520 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.152699536, 2024-09-13T11:24:04.520 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:24:04.520 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:24:04.520 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:24:04.520 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:24:04.521 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:04.521 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:24:04.521 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.334897+0000", 2024-09-13T11:24:04.521 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:04.521 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:04.521 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:04.521 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:24:04.521 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.334897+0000", 2024-09-13T11:24:04.521 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:04.521 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:04.521 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:04.521 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:24:04.521 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.334898+0000", 2024-09-13T11:24:04.522 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.499e-06 2024-09-13T11:24:04.522 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:04.522 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:04.522 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:24:04.522 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.334905+0000", 2024-09-13T11:24:04.522 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.6470000000000001e-06 2024-09-13T11:24:04.522 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:04.522 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:04.522 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:24:04.522 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.486251+0000", 2024-09-13T11:24:04.522 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.151345432 2024-09-13T11:24:04.522 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:04.522 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:04.523 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:24:04.523 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.486262+0000", 2024-09-13T11:24:04.523 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1005000000000001e-05 2024-09-13T11:24:04.523 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:04.523 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:04.523 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:24:04.523 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.486411+0000", 2024-09-13T11:24:04.524 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00014960099999999999 2024-09-13T11:24:04.524 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:04.524 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:04.524 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:24:04.524 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.486434+0000", 2024-09-13T11:24:04.524 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.3034e-05 2024-09-13T11:24:04.524 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:04.524 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:04.525 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:24:04.525 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.487597+0000", 2024-09-13T11:24:04.525 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0011623180000000001 2024-09-13T11:24:04.525 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:04.525 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:24:04.525 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:04.526 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:04.526 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:04.526 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPushReply(3.19 74/70 [PushReplyOp(3:9d264015:::benchmark_data_smithi177_31116_object16883:head)])", 2024-09-13T11:24:04.526 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:30.637135+0000", 2024-09-13T11:24:04.526 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 33.846436193000002, 2024-09-13T11:24:04.526 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.098447104999999993, 2024-09-13T11:24:04.526 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:24:04.526 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:24:04.526 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-09-13T11:24:04.526 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:24:04.526 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:04.526 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:24:04.526 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.637135+0000", 2024-09-13T11:24:04.526 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:04.527 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:04.527 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:04.527 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:24:04.527 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.637135+0000", 2024-09-13T11:24:04.527 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:04.527 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:04.527 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:04.527 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:24:04.527 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.637137+0000", 2024-09-13T11:24:04.527 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.885e-06 2024-09-13T11:24:04.528 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:04.528 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:04.528 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:24:04.528 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.637144+0000", 2024-09-13T11:24:04.528 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.3760000000000002e-06 2024-09-13T11:24:04.528 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:04.528 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:04.528 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:24:04.528 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.734363+0000", 2024-09-13T11:24:04.528 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.097218465000000004 2024-09-13T11:24:04.528 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:04.528 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:04.528 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:24:04.528 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.734373+0000", 2024-09-13T11:24:04.529 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0472000000000001e-05 2024-09-13T11:24:04.529 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:04.529 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:04.529 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:24:04.529 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.734437+0000", 2024-09-13T11:24:04.529 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.3547999999999995e-05 2024-09-13T11:24:04.529 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:04.529 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:04.529 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:24:04.529 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.735582+0000", 2024-09-13T11:24:04.529 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0011453590000000001 2024-09-13T11:24:04.529 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:04.529 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:24:04.529 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:04.530 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:04.530 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:04.530 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPushReply(3.19 74/70 [PushReplyOp(3:9edf939c:::benchmark_data_smithi177_31116_object17207:head)])", 2024-09-13T11:24:04.530 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:30.897971+0000", 2024-09-13T11:24:04.530 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 33.585600006, 2024-09-13T11:24:04.530 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.40616071100000001, 2024-09-13T11:24:04.530 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:24:04.530 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:24:04.530 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-09-13T11:24:04.530 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:24:04.530 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:04.531 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:24:04.531 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.897971+0000", 2024-09-13T11:24:04.531 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:04.531 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:04.531 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:04.531 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:24:04.531 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.897971+0000", 2024-09-13T11:24:04.531 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:04.531 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:04.531 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:04.531 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:24:04.531 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.897972+0000", 2024-09-13T11:24:04.531 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1429999999999999e-06 2024-09-13T11:24:04.531 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:04.532 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:04.532 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:24:04.532 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.897977+0000", 2024-09-13T11:24:04.532 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.5199999999999999e-06 2024-09-13T11:24:04.532 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:04.532 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:04.532 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:24:04.532 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:31.302500+0000", 2024-09-13T11:24:04.532 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.40452312400000001 2024-09-13T11:24:04.532 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:04.532 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:04.532 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:24:04.532 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:31.302513+0000", 2024-09-13T11:24:04.533 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2753e-05 2024-09-13T11:24:04.533 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:04.533 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:04.533 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:24:04.533 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:31.302645+0000", 2024-09-13T11:24:04.533 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00013292100000000001 2024-09-13T11:24:04.533 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:04.533 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:04.533 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:24:04.533 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:31.304132+0000", 2024-09-13T11:24:04.533 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0014862499999999999 2024-09-13T11:24:04.533 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:04.533 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:24:04.533 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:04.534 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:04.534 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:04.534 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPushReply(3.19 75/70 [PushReplyOp(3:9d1489b1:::benchmark_data_smithi177_31116_object18698:head)])", 2024-09-13T11:24:04.534 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:31.697212+0000", 2024-09-13T11:24:04.534 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 32.786359005000001, 2024-09-13T11:24:04.534 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.22850633200000001, 2024-09-13T11:24:04.534 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:24:04.534 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:24:04.534 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-09-13T11:24:04.534 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:24:04.534 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:04.534 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:24:04.534 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:31.697212+0000", 2024-09-13T11:24:04.534 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:04.535 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:04.535 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:04.535 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:24:04.535 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:31.697212+0000", 2024-09-13T11:24:04.535 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:04.535 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:04.535 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:04.535 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:24:04.535 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:31.697214+0000", 2024-09-13T11:24:04.535 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.7439999999999999e-06 2024-09-13T11:24:04.535 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:04.535 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:04.536 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:24:04.536 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:31.697224+0000", 2024-09-13T11:24:04.536 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0173000000000001e-05 2024-09-13T11:24:04.536 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:04.536 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:04.536 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:24:04.536 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:31.923931+0000", 2024-09-13T11:24:04.536 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.226706926 2024-09-13T11:24:04.536 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:04.536 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:04.536 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:24:04.536 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:31.923950+0000", 2024-09-13T11:24:04.536 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.8899e-05 2024-09-13T11:24:04.537 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:04.537 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:04.537 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:24:04.537 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:31.924449+0000", 2024-09-13T11:24:04.537 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00049907800000000002 2024-09-13T11:24:04.537 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:04.537 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:04.537 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:24:04.537 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:31.925718+0000", 2024-09-13T11:24:04.537 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001269512 2024-09-13T11:24:04.537 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:04.537 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:24:04.537 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:04.538 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:04.538 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:04.538 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPushReply(3.19 76/70 [PushReplyOp(3:9faf6d5f:::benchmark_data_smithi177_31116_object18954:head)])", 2024-09-13T11:24:04.538 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:32.042874+0000", 2024-09-13T11:24:04.538 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 32.440697135999997, 2024-09-13T11:24:04.538 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.40212404299999999, 2024-09-13T11:24:04.538 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:24:04.538 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:24:04.539 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-09-13T11:24:04.539 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:24:04.539 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:04.539 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:24:04.539 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.042874+0000", 2024-09-13T11:24:04.539 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:04.539 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:04.539 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:04.539 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:24:04.539 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.042874+0000", 2024-09-13T11:24:04.539 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:04.539 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:04.539 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:04.539 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:24:04.540 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.042875+0000", 2024-09-13T11:24:04.540 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.432e-06 2024-09-13T11:24:04.540 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:04.540 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:04.540 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:24:04.540 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.042886+0000", 2024-09-13T11:24:04.540 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0265000000000001e-05 2024-09-13T11:24:04.540 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:04.540 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:04.540 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:24:04.540 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.443455+0000", 2024-09-13T11:24:04.540 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.40056951400000002 2024-09-13T11:24:04.540 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:04.541 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:04.541 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:24:04.541 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.443480+0000", 2024-09-13T11:24:04.541 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.4929000000000001e-05 2024-09-13T11:24:04.541 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:04.541 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:04.541 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:24:04.541 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.443666+0000", 2024-09-13T11:24:04.541 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000186446 2024-09-13T11:24:04.541 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:04.542 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:04.542 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:24:04.542 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.444998+0000", 2024-09-13T11:24:04.542 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0013314570000000001 2024-09-13T11:24:04.542 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:04.542 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:24:04.542 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:04.542 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:04.542 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:04.542 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPushReply(3.19 79/70 [PushReplyOp(3:99ae4688:::benchmark_data_smithi177_31116_object30992:head)])", 2024-09-13T11:24:04.542 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:36.183302+0000", 2024-09-13T11:24:04.542 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 28.300269372999999, 2024-09-13T11:24:04.542 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.099416386999999995, 2024-09-13T11:24:04.542 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:24:04.543 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:24:04.543 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-09-13T11:24:04.543 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:24:04.543 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:04.543 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:24:04.543 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:36.183302+0000", 2024-09-13T11:24:04.543 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:04.543 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:04.543 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:04.543 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:24:04.543 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:36.183302+0000", 2024-09-13T11:24:04.543 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:04.543 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:04.543 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:04.544 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:24:04.544 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:36.183303+0000", 2024-09-13T11:24:04.544 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2640000000000001e-06 2024-09-13T11:24:04.544 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:04.544 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:04.544 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:24:04.544 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:36.183308+0000", 2024-09-13T11:24:04.544 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.9790000000000001e-06 2024-09-13T11:24:04.544 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:04.544 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:04.544 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:24:04.544 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:36.282092+0000", 2024-09-13T11:24:04.544 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.098783695000000005 2024-09-13T11:24:04.544 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:04.545 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:04.545 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:24:04.545 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:36.282100+0000", 2024-09-13T11:24:04.545 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.4600000000000003e-06 2024-09-13T11:24:04.545 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:04.545 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:04.545 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:24:04.545 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:36.282157+0000", 2024-09-13T11:24:04.545 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.6844e-05 2024-09-13T11:24:04.545 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:04.545 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:04.546 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:24:04.546 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:36.282718+0000", 2024-09-13T11:24:04.546 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00056114499999999998 2024-09-13T11:24:04.546 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:04.546 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:24:04.546 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:04.546 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:04.546 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:04.546 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:56140 3.19 3:9c6bfb18:::benchmark_data_smithi177_31116_object56139:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e79)", 2024-09-13T11:24:04.546 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:36.279622+0000", 2024-09-13T11:24:04.546 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 28.203948531000002, 2024-09-13T11:24:04.546 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.12757104799999999, 2024-09-13T11:24:04.546 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:24:04.546 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:24:04.547 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:04.547 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:24:04.547 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:24:04.547 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:04.547 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 56140 2024-09-13T11:24:04.547 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:04.547 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:24:04.547 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:04.547 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:24:04.547 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:36.279622+0000", 2024-09-13T11:24:04.547 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:04.547 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:04.547 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:04.548 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:24:04.548 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:36.279622+0000", 2024-09-13T11:24:04.548 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:04.548 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:04.548 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:04.548 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:24:04.548 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:36.279625+0000", 2024-09-13T11:24:04.548 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.4399999999999999e-06 2024-09-13T11:24:04.548 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:04.548 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:04.548 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:24:04.548 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:36.279662+0000", 2024-09-13T11:24:04.548 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.7094000000000001e-05 2024-09-13T11:24:04.548 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:04.549 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:04.549 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:24:04.549 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:36.279679+0000", 2024-09-13T11:24:04.549 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.7147999999999999e-05 2024-09-13T11:24:04.549 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:04.549 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:04.549 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:24:04.549 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:36.279687+0000", 2024-09-13T11:24:04.550 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.7379999999999994e-06 2024-09-13T11:24:04.550 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:04.550 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:04.550 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:24:04.550 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:36.279744+0000", 2024-09-13T11:24:04.550 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.6711e-05 2024-09-13T11:24:04.550 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:04.550 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:04.550 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:24:04.550 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:36.279881+0000", 2024-09-13T11:24:04.550 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000137377 2024-09-13T11:24:04.550 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:04.550 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:04.550 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 1,9", 2024-09-13T11:24:04.551 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:36.280005+0000", 2024-09-13T11:24:04.551 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00012385799999999999 2024-09-13T11:24:04.551 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:04.551 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:04.551 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:24:04.551 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:36.281925+0000", 2024-09-13T11:24:04.551 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0019203410000000001 2024-09-13T11:24:04.551 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:04.552 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:04.552 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:04.552 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:36.286822+0000", 2024-09-13T11:24:04.552 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0048964250000000003 2024-09-13T11:24:04.552 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:04.552 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:04.552 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:04.552 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:36.406831+0000", 2024-09-13T11:24:04.552 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.120009569 2024-09-13T11:24:04.552 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:04.553 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:04.553 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:24:04.553 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:36.406875+0000", 2024-09-13T11:24:04.553 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.3618999999999997e-05 2024-09-13T11:24:04.553 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:04.553 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:04.553 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:24:04.553 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:36.407194+0000", 2024-09-13T11:24:04.553 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000318728 2024-09-13T11:24:04.553 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:04.553 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:24:04.553 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:04.553 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:04.553 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:04.554 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop_reply(client.4665.0:56140 3.19 e79/70 ondisk, result = 0)", 2024-09-13T11:24:04.554 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:36.282064+0000", 2024-09-13T11:24:04.554 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 28.201507190000001, 2024-09-13T11:24:04.554 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.12513713400000001, 2024-09-13T11:24:04.554 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:24:04.554 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:24:04.554 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:24:04.554 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:24:04.554 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:04.554 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:24:04.554 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:36.282064+0000", 2024-09-13T11:24:04.554 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:04.554 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:04.554 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:04.555 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:24:04.555 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:36.282064+0000", 2024-09-13T11:24:04.555 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:04.555 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:04.555 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:04.555 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:24:04.555 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:36.282066+0000", 2024-09-13T11:24:04.555 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.7519999999999999e-06 2024-09-13T11:24:04.555 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:04.555 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:04.555 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:24:04.555 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:36.282071+0000", 2024-09-13T11:24:04.555 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.6960000000000002e-06 2024-09-13T11:24:04.555 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:04.556 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:04.556 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:24:04.556 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:36.406725+0000", 2024-09-13T11:24:04.556 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.12465353999999999 2024-09-13T11:24:04.556 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:04.556 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:04.556 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:24:04.556 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:36.406738+0000", 2024-09-13T11:24:04.556 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2774e-05 2024-09-13T11:24:04.556 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:04.557 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:04.557 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:24:04.557 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:36.406816+0000", 2024-09-13T11:24:04.557 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.8428000000000002e-05 2024-09-13T11:24:04.557 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:04.557 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:04.557 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:24:04.557 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:36.406826+0000", 2024-09-13T11:24:04.557 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0239e-05 2024-09-13T11:24:04.557 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:04.557 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:04.557 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:24:04.557 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:36.407201+0000", 2024-09-13T11:24:04.558 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00037470500000000003 2024-09-13T11:24:04.558 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:04.558 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:24:04.558 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:04.558 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:04.558 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:04.558 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPushReply(3.19 79/70 [PushReplyOp(3:99a0d211:::benchmark_data_smithi177_31116_object31006:head)])", 2024-09-13T11:24:04.558 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:36.286313+0000", 2024-09-13T11:24:04.558 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 28.197257579999999, 2024-09-13T11:24:04.558 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.14795829499999999, 2024-09-13T11:24:04.559 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:24:04.559 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:24:04.559 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-09-13T11:24:04.559 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:24:04.559 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:04.559 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:24:04.559 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:36.286313+0000", 2024-09-13T11:24:04.559 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:04.559 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:04.559 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:04.559 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:24:04.560 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:36.286313+0000", 2024-09-13T11:24:04.560 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:04.560 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:04.560 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:04.560 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:24:04.560 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:36.286315+0000", 2024-09-13T11:24:04.560 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4169999999999999e-06 2024-09-13T11:24:04.560 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:04.560 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:04.560 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:24:04.561 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:36.286321+0000", 2024-09-13T11:24:04.561 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.3260000000000001e-06 2024-09-13T11:24:04.561 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:04.561 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:04.561 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:24:04.561 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:36.431926+0000", 2024-09-13T11:24:04.561 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.14560464300000001 2024-09-13T11:24:04.561 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:04.561 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:04.561 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:24:04.561 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:36.431934+0000", 2024-09-13T11:24:04.561 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.439e-06 2024-09-13T11:24:04.561 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:04.561 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:04.562 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:24:04.562 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:36.433520+0000", 2024-09-13T11:24:04.562 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0015857860000000001 2024-09-13T11:24:04.562 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:04.562 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:04.562 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:24:04.562 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:36.434272+0000", 2024-09-13T11:24:04.562 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00075168399999999997 2024-09-13T11:24:04.562 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:04.562 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:24:04.562 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:04.562 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:04.562 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:04.562 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPushReply(3.19 79/70 [PushReplyOp(3:98fdce99:::benchmark_data_smithi177_31116_object31094:head)])", 2024-09-13T11:24:04.563 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:36.480752+0000", 2024-09-13T11:24:04.563 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 28.002818528999999, 2024-09-13T11:24:04.563 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.091161453000000003, 2024-09-13T11:24:04.563 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:24:04.563 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:24:04.563 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-09-13T11:24:04.563 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:24:04.563 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:04.563 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:24:04.563 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:36.480752+0000", 2024-09-13T11:24:04.563 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:04.563 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:04.563 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:04.564 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:24:04.564 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:36.480752+0000", 2024-09-13T11:24:04.564 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:04.564 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:04.564 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:04.564 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:24:04.564 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:36.480754+0000", 2024-09-13T11:24:04.564 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.469e-06 2024-09-13T11:24:04.564 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:04.564 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:04.564 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:24:04.565 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:36.480760+0000", 2024-09-13T11:24:04.565 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.2829999999999998e-06 2024-09-13T11:24:04.565 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:04.565 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:04.565 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:24:04.565 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:36.571390+0000", 2024-09-13T11:24:04.565 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.090629804999999994 2024-09-13T11:24:04.565 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:04.565 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:04.565 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:24:04.565 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:36.571397+0000", 2024-09-13T11:24:04.565 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.9949999999999999e-06 2024-09-13T11:24:04.565 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:04.565 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:04.566 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:24:04.566 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:36.571455+0000", 2024-09-13T11:24:04.566 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.7717999999999999e-05 2024-09-13T11:24:04.566 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:04.566 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:04.566 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:24:04.566 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:36.571914+0000", 2024-09-13T11:24:04.566 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000459183 2024-09-13T11:24:04.566 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:04.566 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:24:04.566 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:04.567 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:04.567 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:04.567 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPush(3.1d 85/78 [PushOp(3:bc4b9336:::benchmark_data_smithi177_31116_object11944:head, version: 49'734, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:bc4b9336:::benchmark_data_smithi177_31116_object11944:head@49'734, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:24:04.567 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:42.225966+0000", 2024-09-13T11:24:04.567 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 22.257604922999999, 2024-09-13T11:24:04.567 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.23879947800000001, 2024-09-13T11:24:04.567 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:24:04.567 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:24:04.567 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:24:04.567 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:24:04.567 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:04.567 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:24:04.567 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.225966+0000", 2024-09-13T11:24:04.567 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:04.568 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:04.568 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:04.568 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:24:04.568 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.225966+0000", 2024-09-13T11:24:04.568 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:04.568 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:04.568 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:04.568 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:24:04.568 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.225969+0000", 2024-09-13T11:24:04.568 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.0400000000000001e-06 2024-09-13T11:24:04.568 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:04.568 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:04.568 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:24:04.568 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.226036+0000", 2024-09-13T11:24:04.569 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.6978999999999997e-05 2024-09-13T11:24:04.569 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:04.569 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:04.569 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:24:04.569 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.463110+0000", 2024-09-13T11:24:04.569 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.23707400000000001 2024-09-13T11:24:04.569 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:04.569 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:04.569 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:24:04.569 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.463140+0000", 2024-09-13T11:24:04.569 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.9575999999999999e-05 2024-09-13T11:24:04.569 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:04.569 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:04.569 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:24:04.570 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.463326+0000", 2024-09-13T11:24:04.570 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000186298 2024-09-13T11:24:04.570 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:04.570 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:04.570 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:24:04.570 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.463351+0000", 2024-09-13T11:24:04.570 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.4862999999999999e-05 2024-09-13T11:24:04.570 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:04.570 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:04.571 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:24:04.571 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.464766+0000", 2024-09-13T11:24:04.571 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001414722 2024-09-13T11:24:04.571 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:04.571 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:24:04.571 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:04.571 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:04.571 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:04.572 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPush(3.1d 89/78 [PushOp(3:bc1b7cfb:::benchmark_data_smithi177_31116_object22629:head, version: 55'1387, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:bc1b7cfb:::benchmark_data_smithi177_31116_object22629:head@55'1387, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:24:04.572 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:46.660855+0000", 2024-09-13T11:24:04.572 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 17.822716427, 2024-09-13T11:24:04.572 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.26622785599999998, 2024-09-13T11:24:04.572 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:24:04.572 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:24:04.572 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:24:04.572 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:24:04.572 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:04.572 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:24:04.572 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:46.660855+0000", 2024-09-13T11:24:04.572 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:04.572 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:04.573 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:04.573 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:24:04.573 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:46.660855+0000", 2024-09-13T11:24:04.573 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:04.573 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:04.573 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:04.573 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:24:04.573 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:46.660857+0000", 2024-09-13T11:24:04.573 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.6560000000000001e-06 2024-09-13T11:24:04.573 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:04.573 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:04.573 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:24:04.573 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:46.660931+0000", 2024-09-13T11:24:04.573 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.4041999999999993e-05 2024-09-13T11:24:04.574 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:04.574 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:04.574 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:24:04.574 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:46.924654+0000", 2024-09-13T11:24:04.574 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.26372239800000002 2024-09-13T11:24:04.574 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:04.574 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:04.574 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:24:04.574 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:46.924729+0000", 2024-09-13T11:24:04.574 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.5061999999999994e-05 2024-09-13T11:24:04.574 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:04.574 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:04.574 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:24:04.575 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:46.924986+0000", 2024-09-13T11:24:04.575 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00025762000000000002 2024-09-13T11:24:04.575 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:04.575 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:04.575 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:24:04.575 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:46.925013+0000", 2024-09-13T11:24:04.575 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.6179000000000001e-05 2024-09-13T11:24:04.575 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:04.575 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:04.575 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:24:04.575 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:46.927082+0000", 2024-09-13T11:24:04.575 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.002069899 2024-09-13T11:24:04.575 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:04.575 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:24:04.575 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:04.576 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:04.576 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:04.576 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPush(3.1d 90/78 [PushOp(3:bc0e7228:::benchmark_data_smithi177_31116_object24733:head, version: 55'1512, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:bc0e7228:::benchmark_data_smithi177_31116_object24733:head@55'1512, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:24:04.576 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:47.529126+0000", 2024-09-13T11:24:04.576 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 16.954445453000002, 2024-09-13T11:24:04.576 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.21343989499999999, 2024-09-13T11:24:04.576 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:24:04.576 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:24:04.576 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:24:04.576 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:24:04.576 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:04.576 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:24:04.576 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:47.529126+0000", 2024-09-13T11:24:04.577 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:04.577 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:04.577 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:04.577 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:24:04.577 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:47.529126+0000", 2024-09-13T11:24:04.577 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:04.577 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:04.577 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:04.577 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:24:04.577 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:47.529127+0000", 2024-09-13T11:24:04.577 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.252e-06 2024-09-13T11:24:04.577 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:04.577 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:04.577 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:24:04.578 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:47.529185+0000", 2024-09-13T11:24:04.578 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.8224999999999999e-05 2024-09-13T11:24:04.578 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:04.578 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:04.578 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:24:04.578 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:47.740216+0000", 2024-09-13T11:24:04.578 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.21103060800000001 2024-09-13T11:24:04.578 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:04.578 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:04.578 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:24:04.578 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:47.740248+0000", 2024-09-13T11:24:04.578 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.2821999999999998e-05 2024-09-13T11:24:04.578 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:04.579 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:04.579 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:24:04.579 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:47.740504+0000", 2024-09-13T11:24:04.579 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00025518399999999999 2024-09-13T11:24:04.579 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:04.579 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:04.579 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:24:04.579 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:47.740544+0000", 2024-09-13T11:24:04.579 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.0438999999999997e-05 2024-09-13T11:24:04.580 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:04.580 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:04.580 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:24:04.580 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:47.742565+0000", 2024-09-13T11:24:04.580 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0020213649999999998 2024-09-13T11:24:04.580 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:04.580 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:24:04.580 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:04.580 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:04.580 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:04.580 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPush(3.1d 90/78 [PushOp(3:ba836eb6:::benchmark_data_smithi177_31116_object24744:head, version: 55'1513, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:ba836eb6:::benchmark_data_smithi177_31116_object24744:head@55'1513, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:24:04.580 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:47.748907+0000", 2024-09-13T11:24:04.580 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 16.734663591, 2024-09-13T11:24:04.581 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.15906743400000001, 2024-09-13T11:24:04.581 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:24:04.581 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:24:04.581 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:24:04.581 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:24:04.581 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:04.581 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:24:04.581 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:47.748907+0000", 2024-09-13T11:24:04.581 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:04.581 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:04.581 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:04.582 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:24:04.582 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:47.748907+0000", 2024-09-13T11:24:04.582 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:04.582 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:04.582 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:04.582 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:24:04.583 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:47.748911+0000", 2024-09-13T11:24:04.583 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.433e-06 2024-09-13T11:24:04.583 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:04.583 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:04.583 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:24:04.583 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:47.749007+0000", 2024-09-13T11:24:04.583 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.6510999999999998e-05 2024-09-13T11:24:04.583 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:04.583 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:04.583 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:24:04.583 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:47.906101+0000", 2024-09-13T11:24:04.584 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.15709389100000001 2024-09-13T11:24:04.584 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:04.584 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:04.584 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:24:04.584 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:47.906163+0000", 2024-09-13T11:24:04.584 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.1367999999999999e-05 2024-09-13T11:24:04.584 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:04.584 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:04.584 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:24:04.585 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:47.906375+0000", 2024-09-13T11:24:04.585 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00021198500000000001 2024-09-13T11:24:04.585 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:04.585 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:04.585 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:24:04.585 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:47.906397+0000", 2024-09-13T11:24:04.585 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.2739999999999999e-05 2024-09-13T11:24:04.585 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:04.585 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:04.585 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:24:04.585 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:47.907975+0000", 2024-09-13T11:24:04.586 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001577506 2024-09-13T11:24:04.586 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:04.586 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:24:04.586 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:04.586 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:04.586 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:04.586 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPush(3.1d 91/78 [PushOp(3:ba3be734:::benchmark_data_smithi177_31116_object25051:head, version: 55'1531, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:ba3be734:::benchmark_data_smithi177_31116_object25051:head@55'1531, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:24:04.586 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:47.992146+0000", 2024-09-13T11:24:04.586 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 16.491424669000001, 2024-09-13T11:24:04.586 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.091846742999999995, 2024-09-13T11:24:04.586 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:24:04.586 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:24:04.587 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:24:04.587 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:24:04.587 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:04.587 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:24:04.587 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:47.992146+0000", 2024-09-13T11:24:04.587 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:04.587 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:04.587 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:04.587 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:24:04.587 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:47.992146+0000", 2024-09-13T11:24:04.587 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:04.587 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:04.587 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:04.587 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:24:04.588 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:47.992155+0000", 2024-09-13T11:24:04.588 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.0739999999999999e-06 2024-09-13T11:24:04.588 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:04.588 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:04.588 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:24:04.588 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:47.992222+0000", 2024-09-13T11:24:04.588 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.6685000000000003e-05 2024-09-13T11:24:04.588 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:04.588 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:04.588 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:24:04.588 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:48.081829+0000", 2024-09-13T11:24:04.588 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.089606656000000007 2024-09-13T11:24:04.588 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:04.588 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:04.589 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:24:04.589 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:48.081865+0000", 2024-09-13T11:24:04.589 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.6498999999999997e-05 2024-09-13T11:24:04.589 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:04.589 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:04.589 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:24:04.589 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:48.082041+0000", 2024-09-13T11:24:04.589 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000175859 2024-09-13T11:24:04.589 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:04.589 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:04.589 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:24:04.589 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:48.082061+0000", 2024-09-13T11:24:04.589 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.9718999999999999e-05 2024-09-13T11:24:04.589 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:04.590 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:04.590 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:24:04.590 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:48.083993+0000", 2024-09-13T11:24:04.590 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0019322510000000001 2024-09-13T11:24:04.590 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:04.590 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:24:04.590 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:04.590 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:04.590 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:24:04.590 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-13T11:24:04.592 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-13T11:24:04.593 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-13T11:24:04.583+0000 7f6a68775640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26307) UID: 0 2024-09-13T11:24:04.693 INFO:tasks.ceph.osd.14.smithi162.stderr:2024-09-13T11:24:04.689+0000 7f235c0db640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26319) UID: 0 2024-09-13T11:24:04.721 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-13T11:24:04.722 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [ 2024-09-13T11:24:04.722 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:04.722 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:83193 3.8 3:10066d83:::benchmark_data_smithi177_31116_object83192:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106)", 2024-09-13T11:24:04.722 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:24:04.283923+0000", 2024-09-13T11:24:04.723 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 0.43650245999999998, 2024-09-13T11:24:04.723 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.43655854199999999, 2024-09-13T11:24:04.723 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:24:04.723 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:24:04.723 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "waiting for sub ops", 2024-09-13T11:24:04.723 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:24:04.724 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:24:04.724 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:04.724 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 83193 2024-09-13T11:24:04.724 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:04.724 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:24:04.724 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:04.724 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:24:04.725 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.283923+0000", 2024-09-13T11:24:04.725 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:04.725 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:04.725 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:04.725 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:24:04.725 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.283923+0000", 2024-09-13T11:24:04.725 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:04.725 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:04.725 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:04.725 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:24:04.725 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.283925+0000", 2024-09-13T11:24:04.725 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.9489999999999999e-06 2024-09-13T11:24:04.726 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:04.726 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:04.726 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:24:04.726 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.283960+0000", 2024-09-13T11:24:04.726 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.5063e-05 2024-09-13T11:24:04.726 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:04.726 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:04.726 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:24:04.727 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.283974+0000", 2024-09-13T11:24:04.727 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3546e-05 2024-09-13T11:24:04.727 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:04.727 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:04.727 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:24:04.728 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.283983+0000", 2024-09-13T11:24:04.728 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.2499999999999995e-06 2024-09-13T11:24:04.728 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:04.728 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:04.728 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:24:04.728 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.284061+0000", 2024-09-13T11:24:04.728 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.7946000000000002e-05 2024-09-13T11:24:04.728 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:04.729 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:04.729 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:24:04.729 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.284270+0000", 2024-09-13T11:24:04.729 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00020876799999999999 2024-09-13T11:24:04.729 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:04.729 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:04.729 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 1", 2024-09-13T11:24:04.729 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.284489+0000", 2024-09-13T11:24:04.730 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00021956800000000001 2024-09-13T11:24:04.730 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:04.730 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:04.730 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:24:04.730 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.286456+0000", 2024-09-13T11:24:04.730 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0019665809999999998 2024-09-13T11:24:04.730 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:04.731 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:24:04.731 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:04.731 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:04.731 INFO:teuthology.orchestra.run.smithi162.stdout: ], 2024-09-13T11:24:04.731 INFO:teuthology.orchestra.run.smithi162.stdout: "num_ops": 1 2024-09-13T11:24:04.731 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-13T11:24:04.735 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-13T11:24:04.822 INFO:tasks.ceph.osd.10.smithi162.stderr:2024-09-13T11:24:04.821+0000 7fb59c9f0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26317) UID: 0 2024-09-13T11:24:04.894 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-13T11:24:04.887+0000 7f6a68775640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26307) UID: 0 2024-09-13T11:24:04.904 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-13T11:24:04.904 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [], 2024-09-13T11:24:04.904 INFO:teuthology.orchestra.run.smithi162.stdout: "complaint_time": 30, 2024-09-13T11:24:04.905 INFO:teuthology.orchestra.run.smithi162.stdout: "num_blocked_ops": 0 2024-09-13T11:24:04.905 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-13T11:24:04.920 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-13T11:24:04.936 INFO:teuthology.orchestra.run.smithi001.stderr:nodeep-scrub is set 2024-09-13T11:24:04.995 INFO:tasks.ceph.osd.13.smithi092.stderr:2024-09-13T11:24:04.991+0000 7f3e5afea640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26350) UID: 0 2024-09-13T11:24:05.007 INFO:tasks.radosbench.radosbench.0.smithi177.stdout: 98 16 83301 83285 53.11 28.25 0.0361994 0.0188187 2024-09-13T11:24:05.096 INFO:tasks.ceph.osd.12.smithi001.stderr:2024-09-13T11:24:05.091+0000 7f892c6e6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26314) UID: 0 2024-09-13T11:24:05.108 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-13T11:24:05.108 INFO:teuthology.orchestra.run.smithi162.stdout: "size": 20, 2024-09-13T11:24:05.108 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 600, 2024-09-13T11:24:05.108 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [ 2024-09-13T11:24:05.108 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.108 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4257.0:80 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 262144~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-13T11:24:05.108 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:22:12.516065+0000", 2024-09-13T11:24:05.109 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 112.58939842700001, 2024-09-13T11:24:05.109 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.46461997999999999, 2024-09-13T11:24:05.109 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:24:05.109 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:24:05.109 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:05.109 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:24:05.109 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4257", 2024-09-13T11:24:05.109 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/1488131527", 2024-09-13T11:24:05.109 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 80 2024-09-13T11:24:05.109 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.109 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:24:05.110 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.110 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:24:05.110 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.516065+0000", 2024-09-13T11:24:05.110 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:05.110 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.110 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.110 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:24:05.110 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.516065+0000", 2024-09-13T11:24:05.110 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:05.110 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.110 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.111 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:24:05.111 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.516066+0000", 2024-09-13T11:24:05.111 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0869999999999999e-06 2024-09-13T11:24:05.111 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.111 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.111 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:24:05.111 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.516112+0000", 2024-09-13T11:24:05.111 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.5503000000000001e-05 2024-09-13T11:24:05.111 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.112 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.112 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:24:05.112 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.516128+0000", 2024-09-13T11:24:05.112 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5891000000000001e-05 2024-09-13T11:24:05.112 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.112 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.112 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:24:05.112 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.516138+0000", 2024-09-13T11:24:05.112 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.8120000000000002e-06 2024-09-13T11:24:05.113 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.113 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.113 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:24:05.113 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.520398+0000", 2024-09-13T11:24:05.113 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0042605450000000001 2024-09-13T11:24:05.113 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.113 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.114 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:24:05.114 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.520526+0000", 2024-09-13T11:24:05.114 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000128355 2024-09-13T11:24:05.114 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.114 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.114 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 0", 2024-09-13T11:24:05.114 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.520730+0000", 2024-09-13T11:24:05.114 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00020337799999999999 2024-09-13T11:24:05.114 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.114 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.115 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:24:05.115 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.522417+0000", 2024-09-13T11:24:05.115 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001687184 2024-09-13T11:24:05.115 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.115 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.115 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:05.115 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.980408+0000", 2024-09-13T11:24:05.115 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.45799068399999998 2024-09-13T11:24:05.115 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.115 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.116 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:24:05.116 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.980565+0000", 2024-09-13T11:24:05.116 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00015760899999999999 2024-09-13T11:24:05.116 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.116 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.116 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:24:05.116 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.980685+0000", 2024-09-13T11:24:05.116 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000119932 2024-09-13T11:24:05.116 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:05.116 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:24:05.117 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:05.117 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.117 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.117 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop_reply(client.4257.0:80 1.0 e24/23 ondisk, result = 0)", 2024-09-13T11:24:05.117 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:22:12.522938+0000", 2024-09-13T11:24:05.117 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 112.582525394, 2024-09-13T11:24:05.117 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.45776420600000001, 2024-09-13T11:24:05.117 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:24:05.117 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:24:05.118 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:24:05.118 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:24:05.118 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.118 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:24:05.118 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.522938+0000", 2024-09-13T11:24:05.118 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:05.118 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.118 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.118 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:24:05.118 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.522938+0000", 2024-09-13T11:24:05.118 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:05.118 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.119 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.119 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:24:05.119 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.522941+0000", 2024-09-13T11:24:05.119 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.0290000000000001e-06 2024-09-13T11:24:05.119 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.119 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.119 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:24:05.119 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.522950+0000", 2024-09-13T11:24:05.119 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.7499999999999992e-06 2024-09-13T11:24:05.119 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.119 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.119 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:24:05.119 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.980215+0000", 2024-09-13T11:24:05.119 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.45726523800000002 2024-09-13T11:24:05.120 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.120 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.120 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:24:05.120 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.980229+0000", 2024-09-13T11:24:05.120 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3549e-05 2024-09-13T11:24:05.120 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.120 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.120 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:24:05.120 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.980365+0000", 2024-09-13T11:24:05.120 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00013608700000000001 2024-09-13T11:24:05.121 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.121 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.121 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:24:05.121 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.980388+0000", 2024-09-13T11:24:05.121 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.3112000000000002e-05 2024-09-13T11:24:05.121 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.121 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.121 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:24:05.122 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.980702+0000", 2024-09-13T11:24:05.122 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00031444099999999999 2024-09-13T11:24:05.122 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:05.122 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:24:05.122 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:05.122 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.123 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.123 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:25416 3.8 3:1a1fbdd1:::benchmark_data_smithi177_31116_object25415:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e56)", 2024-09-13T11:24:05.123 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:04.512188+0000", 2024-09-13T11:24:05.123 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 60.593275427000002, 2024-09-13T11:24:05.123 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.210208697, 2024-09-13T11:24:05.124 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:24:05.124 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:24:05.124 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:05.124 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:24:05.124 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:24:05.124 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:05.125 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 25416 2024-09-13T11:24:05.125 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.125 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:24:05.125 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.125 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:24:05.125 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.512188+0000", 2024-09-13T11:24:05.125 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:05.125 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.125 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.125 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:24:05.126 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.512188+0000", 2024-09-13T11:24:05.126 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:05.126 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.126 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.126 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:24:05.126 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.512192+0000", 2024-09-13T11:24:05.126 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.2660000000000003e-06 2024-09-13T11:24:05.127 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.127 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.127 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:24:05.127 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.512247+0000", 2024-09-13T11:24:05.127 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.4889000000000003e-05 2024-09-13T11:24:05.127 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.127 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.127 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:24:05.127 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.512279+0000", 2024-09-13T11:24:05.127 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.1146000000000003e-05 2024-09-13T11:24:05.127 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.128 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.128 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:24:05.128 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.512297+0000", 2024-09-13T11:24:05.128 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.825e-05 2024-09-13T11:24:05.128 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.128 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.128 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:24:05.128 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.512377+0000", 2024-09-13T11:24:05.128 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.9875000000000004e-05 2024-09-13T11:24:05.128 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.128 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.128 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:24:05.128 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.512570+0000", 2024-09-13T11:24:05.129 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00019312100000000001 2024-09-13T11:24:05.129 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.129 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.129 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 1", 2024-09-13T11:24:05.129 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.512798+0000", 2024-09-13T11:24:05.129 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00022847899999999999 2024-09-13T11:24:05.129 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.129 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.129 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:24:05.129 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.514975+0000", 2024-09-13T11:24:05.129 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0021769979999999999 2024-09-13T11:24:05.129 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.130 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.130 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:05.130 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.722207+0000", 2024-09-13T11:24:05.130 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.20723217099999999 2024-09-13T11:24:05.130 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.130 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.130 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:24:05.130 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.722314+0000", 2024-09-13T11:24:05.130 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00010648400000000001 2024-09-13T11:24:05.130 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.130 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.130 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:24:05.130 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.722397+0000", 2024-09-13T11:24:05.131 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.3017999999999997e-05 2024-09-13T11:24:05.131 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:05.131 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:24:05.131 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:05.131 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.131 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.131 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop_reply(client.4665.0:25416 3.8 e56/40 ondisk, result = 0)", 2024-09-13T11:24:05.131 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:04.515635+0000", 2024-09-13T11:24:05.131 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 60.589828775000001, 2024-09-13T11:24:05.131 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.20677600500000001, 2024-09-13T11:24:05.132 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:24:05.132 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:24:05.132 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:24:05.132 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:24:05.132 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.132 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:24:05.133 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.515635+0000", 2024-09-13T11:24:05.133 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:05.133 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.133 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.133 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:24:05.134 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.515635+0000", 2024-09-13T11:24:05.134 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:05.134 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.134 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.134 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:24:05.134 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.515639+0000", 2024-09-13T11:24:05.134 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.619e-06 2024-09-13T11:24:05.134 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.135 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.135 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:24:05.135 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.515650+0000", 2024-09-13T11:24:05.135 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.111e-05 2024-09-13T11:24:05.135 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.135 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.135 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:24:05.135 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.722004+0000", 2024-09-13T11:24:05.135 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.206354805 2024-09-13T11:24:05.135 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.135 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.136 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:24:05.136 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.722017+0000", 2024-09-13T11:24:05.136 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2840000000000001e-05 2024-09-13T11:24:05.136 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.136 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.136 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:24:05.136 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.722186+0000", 2024-09-13T11:24:05.136 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00016898299999999999 2024-09-13T11:24:05.136 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.136 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.136 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:24:05.136 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.722199+0000", 2024-09-13T11:24:05.136 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.289e-05 2024-09-13T11:24:05.137 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.137 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.137 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:24:05.137 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.722411+0000", 2024-09-13T11:24:05.137 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00021175799999999999 2024-09-13T11:24:05.137 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:05.137 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:24:05.137 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:05.137 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.137 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.137 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:25432 3.8 3:19671730:::benchmark_data_smithi177_31116_object25431:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e56)", 2024-09-13T11:24:05.137 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:04.717754+0000", 2024-09-13T11:24:05.137 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 60.38770976, 2024-09-13T11:24:05.138 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.102412884, 2024-09-13T11:24:05.138 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:24:05.138 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:24:05.138 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:05.138 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:24:05.138 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:24:05.138 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:05.138 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 25432 2024-09-13T11:24:05.138 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.138 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:24:05.138 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.139 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:24:05.139 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.717754+0000", 2024-09-13T11:24:05.139 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:05.139 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.139 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.139 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:24:05.139 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.717754+0000", 2024-09-13T11:24:05.139 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:05.139 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.139 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.139 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:24:05.139 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.717759+0000", 2024-09-13T11:24:05.140 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.04e-06 2024-09-13T11:24:05.140 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.140 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.140 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:24:05.140 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.717820+0000", 2024-09-13T11:24:05.140 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.0900999999999998e-05 2024-09-13T11:24:05.140 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.140 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.140 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:24:05.140 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.717854+0000", 2024-09-13T11:24:05.140 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.3797000000000002e-05 2024-09-13T11:24:05.140 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.141 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.141 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:24:05.141 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.717871+0000", 2024-09-13T11:24:05.141 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.6898999999999999e-05 2024-09-13T11:24:05.141 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.141 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.141 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:24:05.141 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.717993+0000", 2024-09-13T11:24:05.141 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00012248399999999999 2024-09-13T11:24:05.141 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.141 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.141 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:24:05.141 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.718222+0000", 2024-09-13T11:24:05.141 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00022848500000000001 2024-09-13T11:24:05.142 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.142 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.142 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 1", 2024-09-13T11:24:05.142 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.718447+0000", 2024-09-13T11:24:05.142 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00022588400000000001 2024-09-13T11:24:05.142 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.142 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.142 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:24:05.142 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.720604+0000", 2024-09-13T11:24:05.142 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0021561430000000001 2024-09-13T11:24:05.142 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.142 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.143 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:05.143 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.820012+0000", 2024-09-13T11:24:05.143 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.099408924999999995 2024-09-13T11:24:05.143 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.143 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.143 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:24:05.143 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.820097+0000", 2024-09-13T11:24:05.143 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.4147999999999995e-05 2024-09-13T11:24:05.143 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.143 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.143 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:24:05.143 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.820167+0000", 2024-09-13T11:24:05.143 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.0178000000000005e-05 2024-09-13T11:24:05.143 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:05.144 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:24:05.144 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:05.144 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.144 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.144 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop_reply(client.4665.0:25432 3.8 e56/40 ondisk, result = 0)", 2024-09-13T11:24:05.144 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:04.721952+0000", 2024-09-13T11:24:05.144 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 60.383511419000001, 2024-09-13T11:24:05.144 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.098227587000000005, 2024-09-13T11:24:05.144 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:24:05.144 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:24:05.144 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:24:05.144 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:24:05.145 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.145 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:24:05.145 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.721952+0000", 2024-09-13T11:24:05.145 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:05.145 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.145 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.145 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:24:05.145 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.721952+0000", 2024-09-13T11:24:05.146 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:05.146 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.146 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.146 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:24:05.146 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.721956+0000", 2024-09-13T11:24:05.146 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.8130000000000002e-06 2024-09-13T11:24:05.146 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.146 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.146 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:24:05.146 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.721967+0000", 2024-09-13T11:24:05.146 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1229e-05 2024-09-13T11:24:05.146 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.146 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.146 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:24:05.147 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.819880+0000", 2024-09-13T11:24:05.147 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.097912274999999993 2024-09-13T11:24:05.147 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.147 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.147 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:24:05.147 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.819894+0000", 2024-09-13T11:24:05.147 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.416e-05 2024-09-13T11:24:05.147 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.147 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.147 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:24:05.147 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.819993+0000", 2024-09-13T11:24:05.147 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.8859999999999999e-05 2024-09-13T11:24:05.147 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.147 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.148 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:24:05.148 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.820005+0000", 2024-09-13T11:24:05.148 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1960000000000001e-05 2024-09-13T11:24:05.148 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.148 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.148 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:24:05.148 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.820180+0000", 2024-09-13T11:24:05.148 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00017529000000000001 2024-09-13T11:24:05.148 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:05.148 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:24:05.148 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:05.148 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.148 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.148 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:30833 3.8 3:1c203f8d:::benchmark_data_smithi177_31116_object30832:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e59)", 2024-09-13T11:24:05.149 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:09.506656+0000", 2024-09-13T11:24:05.149 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 55.598807876999999, 2024-09-13T11:24:05.149 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.224862282, 2024-09-13T11:24:05.149 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:24:05.149 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:24:05.149 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:05.149 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:24:05.149 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:24:05.149 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:05.149 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 30833 2024-09-13T11:24:05.149 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.149 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:24:05.149 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.149 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:24:05.150 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.506656+0000", 2024-09-13T11:24:05.150 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:05.150 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.150 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.150 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:24:05.150 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.506656+0000", 2024-09-13T11:24:05.150 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:05.150 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.150 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.150 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:24:05.150 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.506660+0000", 2024-09-13T11:24:05.150 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.6e-06 2024-09-13T11:24:05.151 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.151 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.151 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:24:05.151 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.506717+0000", 2024-09-13T11:24:05.151 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.6131999999999997e-05 2024-09-13T11:24:05.151 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.151 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.151 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:24:05.151 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.506751+0000", 2024-09-13T11:24:05.151 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.4746000000000002e-05 2024-09-13T11:24:05.151 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.151 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.151 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:24:05.151 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.506774+0000", 2024-09-13T11:24:05.152 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.2636000000000001e-05 2024-09-13T11:24:05.152 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.152 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.152 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:24:05.152 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.506878+0000", 2024-09-13T11:24:05.152 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000104057 2024-09-13T11:24:05.152 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.152 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.152 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:24:05.152 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.507052+0000", 2024-09-13T11:24:05.152 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00017402800000000001 2024-09-13T11:24:05.152 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.153 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.153 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 1", 2024-09-13T11:24:05.153 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.507217+0000", 2024-09-13T11:24:05.153 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00016516 2024-09-13T11:24:05.153 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.153 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.153 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:24:05.153 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.508992+0000", 2024-09-13T11:24:05.153 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0017743990000000001 2024-09-13T11:24:05.153 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.153 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.153 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:05.153 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.731429+0000", 2024-09-13T11:24:05.154 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.22243718000000001 2024-09-13T11:24:05.154 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.154 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.154 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:24:05.154 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.731479+0000", 2024-09-13T11:24:05.154 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.0146999999999999e-05 2024-09-13T11:24:05.154 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.154 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.154 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:24:05.154 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.731518+0000", 2024-09-13T11:24:05.154 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.9196999999999998e-05 2024-09-13T11:24:05.154 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:05.154 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:24:05.155 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:05.155 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.155 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.155 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop_reply(client.4665.0:30833 3.8 e59/40 ondisk, result = 0)", 2024-09-13T11:24:05.155 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:09.508907+0000", 2024-09-13T11:24:05.155 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 55.596556782999997, 2024-09-13T11:24:05.155 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.222618862, 2024-09-13T11:24:05.155 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:24:05.155 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:24:05.155 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:24:05.155 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:24:05.155 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.156 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:24:05.156 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.508907+0000", 2024-09-13T11:24:05.156 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:05.156 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.156 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.156 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:24:05.156 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.508907+0000", 2024-09-13T11:24:05.156 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:05.157 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.157 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.157 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:24:05.157 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.508910+0000", 2024-09-13T11:24:05.157 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.993e-06 2024-09-13T11:24:05.157 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.157 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.157 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:24:05.157 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.508918+0000", 2024-09-13T11:24:05.157 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.7000000000000008e-06 2024-09-13T11:24:05.157 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.157 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.157 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:24:05.157 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.731341+0000", 2024-09-13T11:24:05.158 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.22242308699999999 2024-09-13T11:24:05.158 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.158 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.158 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:24:05.158 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.731350+0000", 2024-09-13T11:24:05.158 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.3179999999999996e-06 2024-09-13T11:24:05.158 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.158 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.158 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:24:05.158 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.731418+0000", 2024-09-13T11:24:05.158 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.7586e-05 2024-09-13T11:24:05.158 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.158 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.159 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:24:05.159 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.731424+0000", 2024-09-13T11:24:05.159 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.844e-06 2024-09-13T11:24:05.159 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.159 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.159 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:24:05.159 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.731526+0000", 2024-09-13T11:24:05.159 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000101334 2024-09-13T11:24:05.159 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:05.159 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:24:05.160 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:05.160 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.160 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.160 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:48977 3.8 3:1030343e:::benchmark_data_smithi177_31116_object48976:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e71)", 2024-09-13T11:24:05.160 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:25.517299+0000", 2024-09-13T11:24:05.160 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 39.588164489, 2024-09-13T11:24:05.160 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.14514738599999999, 2024-09-13T11:24:05.160 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:24:05.160 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:24:05.160 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:05.160 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:24:05.160 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:24:05.160 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:05.161 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 48977 2024-09-13T11:24:05.161 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.161 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:24:05.161 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.161 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:24:05.161 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.517299+0000", 2024-09-13T11:24:05.161 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:05.161 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.161 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.161 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:24:05.161 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.517299+0000", 2024-09-13T11:24:05.161 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:05.161 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.162 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.162 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:24:05.162 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.517301+0000", 2024-09-13T11:24:05.162 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.7060000000000001e-06 2024-09-13T11:24:05.162 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.162 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.162 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:24:05.162 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.517354+0000", 2024-09-13T11:24:05.162 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.3148e-05 2024-09-13T11:24:05.162 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.162 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.162 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:24:05.163 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.517365+0000", 2024-09-13T11:24:05.163 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1389000000000001e-05 2024-09-13T11:24:05.163 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.163 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.163 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:24:05.163 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.517374+0000", 2024-09-13T11:24:05.163 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.6740000000000003e-06 2024-09-13T11:24:05.163 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.163 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.163 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:24:05.163 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.517454+0000", 2024-09-13T11:24:05.163 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.9968000000000004e-05 2024-09-13T11:24:05.164 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.164 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.164 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:24:05.164 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.517561+0000", 2024-09-13T11:24:05.164 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00010655 2024-09-13T11:24:05.164 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.164 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.164 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 1", 2024-09-13T11:24:05.164 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.517664+0000", 2024-09-13T11:24:05.164 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000103409 2024-09-13T11:24:05.164 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.164 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.165 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:24:05.165 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.521381+0000", 2024-09-13T11:24:05.165 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0037174019999999999 2024-09-13T11:24:05.165 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.165 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.165 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:05.165 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.662307+0000", 2024-09-13T11:24:05.165 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.14092524000000001 2024-09-13T11:24:05.165 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.165 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.165 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:24:05.165 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.662382+0000", 2024-09-13T11:24:05.166 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.5331000000000001e-05 2024-09-13T11:24:05.166 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.166 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.166 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:24:05.166 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.662447+0000", 2024-09-13T11:24:05.166 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.4568999999999998e-05 2024-09-13T11:24:05.166 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:05.166 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:24:05.166 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:05.166 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.167 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.167 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.4665.0:54351 3.10 e74/70 3:085f283e:::benchmark_data_smithi177_31116_object54350:head v 74'3087, mlcod=49'780)", 2024-09-13T11:24:05.167 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:30.599667+0000", 2024-09-13T11:24:05.167 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 34.505796277000002, 2024-09-13T11:24:05.167 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.16617266999999999, 2024-09-13T11:24:05.167 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:24:05.167 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:24:05.167 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:05.168 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:24:05.168 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.168 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:24:05.168 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.599667+0000", 2024-09-13T11:24:05.168 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:05.168 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.168 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.168 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:24:05.168 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.599667+0000", 2024-09-13T11:24:05.168 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:05.168 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.168 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.168 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:24:05.169 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.599669+0000", 2024-09-13T11:24:05.169 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3599999999999999e-06 2024-09-13T11:24:05.169 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.169 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.169 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:24:05.169 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.599712+0000", 2024-09-13T11:24:05.169 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.3723000000000001e-05 2024-09-13T11:24:05.169 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.169 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.169 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:24:05.169 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.762241+0000", 2024-09-13T11:24:05.169 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.16252863400000001 2024-09-13T11:24:05.169 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.169 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.170 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:24:05.170 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.762255+0000", 2024-09-13T11:24:05.170 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3414999999999999e-05 2024-09-13T11:24:05.170 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.170 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.170 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:24:05.170 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.762329+0000", 2024-09-13T11:24:05.170 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.4735999999999997e-05 2024-09-13T11:24:05.170 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.170 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.170 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:24:05.170 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.762358+0000", 2024-09-13T11:24:05.170 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.8308000000000001e-05 2024-09-13T11:24:05.171 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.171 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.171 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:24:05.171 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.765770+0000", 2024-09-13T11:24:05.171 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0034126590000000002 2024-09-13T11:24:05.171 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.171 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.171 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:24:05.171 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.765840+0000", 2024-09-13T11:24:05.171 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.9834999999999998e-05 2024-09-13T11:24:05.171 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:05.172 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:24:05.172 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:05.172 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.172 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.172 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPush(3.10 74/70 [PushOp(3:0b3e58bf:::benchmark_data_smithi177_31116_object12372:head, version: 49'782, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:0b3e58bf:::benchmark_data_smithi177_31116_object12372:head@49'782, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:24:05.172 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:30.608892+0000", 2024-09-13T11:24:05.172 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 34.496571922999998, 2024-09-13T11:24:05.172 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.15687285300000001, 2024-09-13T11:24:05.172 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:24:05.172 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:24:05.172 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:24:05.172 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:24:05.172 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.172 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:24:05.173 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.608892+0000", 2024-09-13T11:24:05.173 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:05.173 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.173 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.173 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:24:05.173 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.608892+0000", 2024-09-13T11:24:05.173 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:05.173 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.173 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.173 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:24:05.173 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.608893+0000", 2024-09-13T11:24:05.173 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.666e-06 2024-09-13T11:24:05.173 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.173 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.174 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:24:05.174 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.609008+0000", 2024-09-13T11:24:05.174 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00011411 2024-09-13T11:24:05.174 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.174 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.174 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:24:05.174 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.763870+0000", 2024-09-13T11:24:05.174 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.15486217799999999 2024-09-13T11:24:05.174 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.174 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.174 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:24:05.174 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.763904+0000", 2024-09-13T11:24:05.174 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.4082000000000003e-05 2024-09-13T11:24:05.174 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.175 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.175 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:24:05.175 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.764247+0000", 2024-09-13T11:24:05.175 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00034281900000000001 2024-09-13T11:24:05.175 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.175 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.175 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:24:05.175 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.764270+0000", 2024-09-13T11:24:05.175 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.3244e-05 2024-09-13T11:24:05.175 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.175 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.175 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:24:05.175 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.765765+0000", 2024-09-13T11:24:05.175 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001494754 2024-09-13T11:24:05.176 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:05.176 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:24:05.176 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:05.176 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.176 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.176 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPush(3.10 76/70 [PushOp(3:0e7d2700:::benchmark_data_smithi177_31116_object17406:head, version: 52'1075, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:0e7d2700:::benchmark_data_smithi177_31116_object17406:head@52'1075, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:24:05.176 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:32.206198+0000", 2024-09-13T11:24:05.176 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 32.899266074000003, 2024-09-13T11:24:05.176 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.25193395000000002, 2024-09-13T11:24:05.176 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:24:05.176 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:24:05.176 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:24:05.177 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:24:05.177 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.177 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:24:05.177 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.206198+0000", 2024-09-13T11:24:05.177 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:05.177 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.177 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.177 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:24:05.177 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.206198+0000", 2024-09-13T11:24:05.177 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:05.177 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.178 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.178 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:24:05.178 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.206199+0000", 2024-09-13T11:24:05.178 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.46e-06 2024-09-13T11:24:05.178 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.178 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.178 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:24:05.178 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.206250+0000", 2024-09-13T11:24:05.178 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.113e-05 2024-09-13T11:24:05.179 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.179 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.179 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:24:05.179 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.456042+0000", 2024-09-13T11:24:05.179 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.24979182999999999 2024-09-13T11:24:05.179 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.179 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.179 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:24:05.179 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.456084+0000", 2024-09-13T11:24:05.179 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.1814000000000003e-05 2024-09-13T11:24:05.179 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.179 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.179 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:24:05.179 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.456342+0000", 2024-09-13T11:24:05.180 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000258404 2024-09-13T11:24:05.180 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.180 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.180 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:24:05.180 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.456367+0000", 2024-09-13T11:24:05.180 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.5099000000000001e-05 2024-09-13T11:24:05.180 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.180 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.180 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:24:05.180 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.458132+0000", 2024-09-13T11:24:05.181 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0017642129999999999 2024-09-13T11:24:05.181 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:05.181 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:24:05.181 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:05.181 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.181 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.181 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPush(3.10 76/70 [PushOp(3:0eee8bcf:::benchmark_data_smithi177_31116_object17589:head, version: 53'1086, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:0eee8bcf:::benchmark_data_smithi177_31116_object17589:head@53'1086, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:24:05.181 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:32.585183+0000", 2024-09-13T11:24:05.181 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 32.52028018, 2024-09-13T11:24:05.181 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.16696202199999999, 2024-09-13T11:24:05.181 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:24:05.181 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:24:05.182 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:24:05.182 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:24:05.182 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.182 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:24:05.182 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.585183+0000", 2024-09-13T11:24:05.182 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:05.182 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.182 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.182 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:24:05.182 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.585183+0000", 2024-09-13T11:24:05.182 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:05.182 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.182 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.182 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:24:05.183 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.585187+0000", 2024-09-13T11:24:05.183 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.8480000000000003e-06 2024-09-13T11:24:05.183 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.183 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.183 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:24:05.183 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.585264+0000", 2024-09-13T11:24:05.183 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.6327000000000003e-05 2024-09-13T11:24:05.183 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.183 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.183 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:24:05.183 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.750459+0000", 2024-09-13T11:24:05.183 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.16519551900000001 2024-09-13T11:24:05.183 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.183 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.183 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:24:05.184 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.750480+0000", 2024-09-13T11:24:05.184 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.0984000000000001e-05 2024-09-13T11:24:05.184 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.184 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.184 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:24:05.184 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.750633+0000", 2024-09-13T11:24:05.184 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000153148 2024-09-13T11:24:05.184 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.184 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.184 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:24:05.184 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.750652+0000", 2024-09-13T11:24:05.184 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.8981e-05 2024-09-13T11:24:05.185 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.185 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.185 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:24:05.185 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.752145+0000", 2024-09-13T11:24:05.185 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001493215 2024-09-13T11:24:05.185 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:05.185 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:24:05.185 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:05.185 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.185 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.185 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:60417 3.10 3:092808b0:::benchmark_data_smithi177_31116_object60416:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e85)", 2024-09-13T11:24:05.185 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:42.173238+0000", 2024-09-13T11:24:05.185 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 22.932225165999998, 2024-09-13T11:24:05.185 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.38965994599999998, 2024-09-13T11:24:05.186 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:24:05.186 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:24:05.186 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:05.186 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:24:05.186 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:24:05.186 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:05.186 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 60417 2024-09-13T11:24:05.186 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.186 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:24:05.187 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.187 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:24:05.187 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.173238+0000", 2024-09-13T11:24:05.187 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:05.187 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.187 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.187 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:24:05.187 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.173238+0000", 2024-09-13T11:24:05.187 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:05.187 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.187 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.187 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:24:05.187 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.173241+0000", 2024-09-13T11:24:05.188 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.6819999999999999e-06 2024-09-13T11:24:05.188 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.188 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.188 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:24:05.188 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.173335+0000", 2024-09-13T11:24:05.188 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.3957999999999997e-05 2024-09-13T11:24:05.188 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.188 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.188 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:24:05.188 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.173358+0000", 2024-09-13T11:24:05.189 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.3023999999999998e-05 2024-09-13T11:24:05.189 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.189 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.189 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:24:05.189 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.173373+0000", 2024-09-13T11:24:05.189 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4867e-05 2024-09-13T11:24:05.189 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.189 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.189 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:24:05.189 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.173466+0000", 2024-09-13T11:24:05.189 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.2542999999999998e-05 2024-09-13T11:24:05.189 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.189 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.189 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:24:05.190 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.173754+0000", 2024-09-13T11:24:05.190 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000288657 2024-09-13T11:24:05.190 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.190 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.190 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 0", 2024-09-13T11:24:05.190 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.174066+0000", 2024-09-13T11:24:05.190 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00031170999999999998 2024-09-13T11:24:05.190 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.190 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.190 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:24:05.190 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.178245+0000", 2024-09-13T11:24:05.190 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00417864 2024-09-13T11:24:05.191 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.191 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.191 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:05.191 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.562617+0000", 2024-09-13T11:24:05.191 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.38437253199999999 2024-09-13T11:24:05.191 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.191 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.191 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:24:05.191 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.562782+0000", 2024-09-13T11:24:05.191 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00016499200000000001 2024-09-13T11:24:05.191 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.191 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.191 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:24:05.191 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.562898+0000", 2024-09-13T11:24:05.192 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000116341 2024-09-13T11:24:05.192 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:05.192 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:24:05.192 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:05.192 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.192 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.192 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:60460 3.10 3:0c638d77:::benchmark_data_smithi177_31116_object60459:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e85)", 2024-09-13T11:24:05.192 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:42.574713+0000", 2024-09-13T11:24:05.192 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 22.530750513000001, 2024-09-13T11:24:05.192 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.103860569, 2024-09-13T11:24:05.192 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:24:05.192 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:24:05.192 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:05.193 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:24:05.193 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:24:05.193 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:05.193 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 60460 2024-09-13T11:24:05.193 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.193 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:24:05.194 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.194 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:24:05.194 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.574713+0000", 2024-09-13T11:24:05.194 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:05.194 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.194 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.195 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:24:05.195 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.574713+0000", 2024-09-13T11:24:05.195 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:05.195 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.195 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.195 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:24:05.195 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.574717+0000", 2024-09-13T11:24:05.196 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.0929999999999996e-06 2024-09-13T11:24:05.196 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.196 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.196 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:24:05.196 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.574827+0000", 2024-09-13T11:24:05.196 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00011013100000000001 2024-09-13T11:24:05.196 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.197 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.197 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:24:05.197 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.574858+0000", 2024-09-13T11:24:05.197 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.1112000000000003e-05 2024-09-13T11:24:05.197 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.197 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.197 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:24:05.198 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.574874+0000", 2024-09-13T11:24:05.198 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5163e-05 2024-09-13T11:24:05.198 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.198 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.198 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:24:05.198 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.575017+0000", 2024-09-13T11:24:05.198 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000143626 2024-09-13T11:24:05.198 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.199 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.199 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:24:05.199 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.575405+0000", 2024-09-13T11:24:05.199 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00038761999999999998 2024-09-13T11:24:05.199 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.199 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.200 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 0", 2024-09-13T11:24:05.200 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.575791+0000", 2024-09-13T11:24:05.200 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00038626600000000002 2024-09-13T11:24:05.200 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.200 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.200 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:24:05.201 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.578462+0000", 2024-09-13T11:24:05.201 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0026712789999999999 2024-09-13T11:24:05.201 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.201 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.201 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:05.201 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.678390+0000", 2024-09-13T11:24:05.201 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.099927953999999999 2024-09-13T11:24:05.202 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.202 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.202 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:24:05.202 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.678500+0000", 2024-09-13T11:24:05.202 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000109793 2024-09-13T11:24:05.202 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.202 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.203 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:24:05.203 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.678574+0000", 2024-09-13T11:24:05.203 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.3532e-05 2024-09-13T11:24:05.203 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:05.203 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:24:05.203 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:05.203 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.203 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.203 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:60789 3.8 3:154c4aed:::benchmark_data_smithi177_31116_object60788:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86)", 2024-09-13T11:24:05.203 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:43.183911+0000", 2024-09-13T11:24:05.203 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 21.921552219999999, 2024-09-13T11:24:05.203 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.098779539999999999, 2024-09-13T11:24:05.204 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:24:05.204 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:24:05.204 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:05.204 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:24:05.204 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:24:05.204 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:05.204 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 60789 2024-09-13T11:24:05.204 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.204 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:24:05.204 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.204 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:24:05.205 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:43.183911+0000", 2024-09-13T11:24:05.205 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:05.205 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.205 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.205 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:24:05.205 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:43.183911+0000", 2024-09-13T11:24:05.205 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:05.205 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.205 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.206 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:24:05.206 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:43.183916+0000", 2024-09-13T11:24:05.206 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.2309999999999998e-06 2024-09-13T11:24:05.206 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.206 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.206 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:24:05.207 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:43.183971+0000", 2024-09-13T11:24:05.207 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.5284999999999997e-05 2024-09-13T11:24:05.207 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.207 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.207 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:24:05.207 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:43.184000+0000", 2024-09-13T11:24:05.207 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.8937000000000001e-05 2024-09-13T11:24:05.208 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.208 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.208 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:24:05.208 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:43.184016+0000", 2024-09-13T11:24:05.208 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.6064000000000001e-05 2024-09-13T11:24:05.208 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.209 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.209 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:24:05.209 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:43.184119+0000", 2024-09-13T11:24:05.209 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00010286299999999999 2024-09-13T11:24:05.209 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.209 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.209 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:24:05.209 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:43.184348+0000", 2024-09-13T11:24:05.210 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000229614 2024-09-13T11:24:05.210 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.210 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.210 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 1", 2024-09-13T11:24:05.210 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:43.184596+0000", 2024-09-13T11:24:05.210 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00024776900000000001 2024-09-13T11:24:05.210 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.211 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.211 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:24:05.211 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:43.186539+0000", 2024-09-13T11:24:05.211 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0019432329999999999 2024-09-13T11:24:05.211 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.211 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.212 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:05.212 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:43.282553+0000", 2024-09-13T11:24:05.212 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.096013412000000006 2024-09-13T11:24:05.212 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.212 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.212 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:24:05.212 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:43.282632+0000", 2024-09-13T11:24:05.213 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.8955000000000006e-05 2024-09-13T11:24:05.213 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.213 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.213 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:24:05.213 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:43.282691+0000", 2024-09-13T11:24:05.213 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.9176999999999999e-05 2024-09-13T11:24:05.213 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:05.214 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:24:05.214 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:05.214 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.214 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.214 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:64804 3.8 3:104508a0:::benchmark_data_smithi177_31116_object64803:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90)", 2024-09-13T11:24:05.214 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:47.694588+0000", 2024-09-13T11:24:05.214 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 17.410875622999999, 2024-09-13T11:24:05.214 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.11215968, 2024-09-13T11:24:05.214 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:24:05.214 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:24:05.214 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:05.214 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:24:05.215 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:24:05.215 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:05.215 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 64804 2024-09-13T11:24:05.215 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.215 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:24:05.215 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.215 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:24:05.215 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:47.694588+0000", 2024-09-13T11:24:05.215 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:05.215 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.215 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.215 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:24:05.216 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:47.694588+0000", 2024-09-13T11:24:05.216 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:05.216 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.216 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.216 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:24:05.216 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:47.694591+0000", 2024-09-13T11:24:05.216 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.9629999999999998e-06 2024-09-13T11:24:05.216 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.216 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.216 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:24:05.216 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:47.694643+0000", 2024-09-13T11:24:05.216 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.1792000000000002e-05 2024-09-13T11:24:05.216 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.216 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.217 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:24:05.217 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:47.694660+0000", 2024-09-13T11:24:05.217 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.7153e-05 2024-09-13T11:24:05.217 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.217 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.217 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:24:05.217 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:47.694671+0000", 2024-09-13T11:24:05.217 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0682999999999999e-05 2024-09-13T11:24:05.217 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.217 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.217 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:24:05.218 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:47.694882+0000", 2024-09-13T11:24:05.218 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00021096799999999999 2024-09-13T11:24:05.218 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.218 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.218 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:24:05.218 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:47.695113+0000", 2024-09-13T11:24:05.218 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000231265 2024-09-13T11:24:05.218 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.218 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.218 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 1", 2024-09-13T11:24:05.218 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:47.695331+0000", 2024-09-13T11:24:05.218 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000218135 2024-09-13T11:24:05.218 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.218 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.219 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:24:05.219 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:47.699656+0000", 2024-09-13T11:24:05.219 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0043248440000000004 2024-09-13T11:24:05.219 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.219 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.219 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:05.219 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:47.806445+0000", 2024-09-13T11:24:05.219 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.106789124 2024-09-13T11:24:05.219 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.219 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.219 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:24:05.219 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:47.806617+0000", 2024-09-13T11:24:05.219 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00017241199999999999 2024-09-13T11:24:05.220 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.220 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.220 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:24:05.220 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:47.806748+0000", 2024-09-13T11:24:05.220 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00013034099999999999 2024-09-13T11:24:05.220 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:05.220 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:24:05.220 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:05.220 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.220 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.220 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:64871 3.8 3:11d9896a:::benchmark_data_smithi177_31116_object64870:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90)", 2024-09-13T11:24:05.220 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:47.826113+0000", 2024-09-13T11:24:05.220 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 17.279350841999999, 2024-09-13T11:24:05.221 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.132123195, 2024-09-13T11:24:05.221 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:24:05.221 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:24:05.221 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:05.221 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:24:05.221 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:24:05.221 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:05.221 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 64871 2024-09-13T11:24:05.221 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.221 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:24:05.221 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.221 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:24:05.221 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:47.826113+0000", 2024-09-13T11:24:05.221 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:05.222 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.222 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.222 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:24:05.222 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:47.826113+0000", 2024-09-13T11:24:05.222 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:05.222 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.222 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.222 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:24:05.222 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:47.826128+0000", 2024-09-13T11:24:05.222 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4882000000000001e-05 2024-09-13T11:24:05.222 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.222 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.222 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:24:05.223 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:47.826189+0000", 2024-09-13T11:24:05.223 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.1276000000000001e-05 2024-09-13T11:24:05.223 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.223 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.223 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:24:05.223 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:47.826223+0000", 2024-09-13T11:24:05.223 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.3804999999999999e-05 2024-09-13T11:24:05.223 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.223 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.223 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:24:05.223 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:47.826245+0000", 2024-09-13T11:24:05.223 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.2609999999999999e-05 2024-09-13T11:24:05.223 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.224 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.224 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:24:05.224 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:47.826419+0000", 2024-09-13T11:24:05.224 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00017394200000000001 2024-09-13T11:24:05.224 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.224 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.224 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:24:05.224 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:47.826676+0000", 2024-09-13T11:24:05.224 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000256741 2024-09-13T11:24:05.224 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.224 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.224 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 1", 2024-09-13T11:24:05.224 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:47.826903+0000", 2024-09-13T11:24:05.224 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00022646100000000001 2024-09-13T11:24:05.225 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.225 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.225 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:24:05.225 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:47.829731+0000", 2024-09-13T11:24:05.225 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0028284500000000001 2024-09-13T11:24:05.225 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.225 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.225 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:05.225 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:47.957975+0000", 2024-09-13T11:24:05.225 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.12824428500000001 2024-09-13T11:24:05.225 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.225 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.225 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:24:05.225 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:47.958129+0000", 2024-09-13T11:24:05.226 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00015343900000000001 2024-09-13T11:24:05.226 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.226 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.226 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:24:05.226 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:47.958236+0000", 2024-09-13T11:24:05.226 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000107304 2024-09-13T11:24:05.226 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:05.226 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:24:05.226 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:05.226 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.226 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.226 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:71901 3.10 3:0d4d24b7:::benchmark_data_smithi177_31116_object71900:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96)", 2024-09-13T11:24:05.226 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:54.134535+0000", 2024-09-13T11:24:05.226 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 10.970928663, 2024-09-13T11:24:05.226 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.108205147, 2024-09-13T11:24:05.227 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:24:05.227 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:24:05.227 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:05.227 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:24:05.227 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:24:05.227 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:05.227 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 71901 2024-09-13T11:24:05.227 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.227 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:24:05.227 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.227 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:24:05.227 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:54.134535+0000", 2024-09-13T11:24:05.227 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:05.227 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.228 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.228 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:24:05.228 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:54.134535+0000", 2024-09-13T11:24:05.228 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:05.228 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.228 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.228 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:24:05.228 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:54.134537+0000", 2024-09-13T11:24:05.228 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.2900000000000001e-06 2024-09-13T11:24:05.228 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.228 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.228 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:24:05.228 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:54.134602+0000", 2024-09-13T11:24:05.229 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.4590000000000003e-05 2024-09-13T11:24:05.229 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.229 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.229 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:24:05.229 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:54.134612+0000", 2024-09-13T11:24:05.229 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.9920000000000002e-06 2024-09-13T11:24:05.229 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.229 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.229 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:24:05.229 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:54.134619+0000", 2024-09-13T11:24:05.229 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.866e-06 2024-09-13T11:24:05.229 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.229 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.229 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:24:05.230 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:54.136309+0000", 2024-09-13T11:24:05.230 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0016901349999999999 2024-09-13T11:24:05.230 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.230 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.230 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:24:05.230 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:54.136659+0000", 2024-09-13T11:24:05.230 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00035013600000000002 2024-09-13T11:24:05.230 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.230 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.230 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 0", 2024-09-13T11:24:05.230 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:54.137070+0000", 2024-09-13T11:24:05.230 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00041059199999999999 2024-09-13T11:24:05.230 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.230 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.231 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:24:05.231 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:54.140222+0000", 2024-09-13T11:24:05.231 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0031520150000000002 2024-09-13T11:24:05.231 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.231 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.231 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:05.231 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:54.242534+0000", 2024-09-13T11:24:05.231 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.102312296 2024-09-13T11:24:05.231 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.231 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.231 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:24:05.231 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:54.242629+0000", 2024-09-13T11:24:05.231 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.5014999999999997e-05 2024-09-13T11:24:05.231 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.232 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.232 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:24:05.232 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:54.242740+0000", 2024-09-13T11:24:05.232 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00011122 2024-09-13T11:24:05.232 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:05.232 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:24:05.232 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:05.232 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.232 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.232 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:72182 3.10 3:0caf2947:::benchmark_data_smithi177_31116_object72181:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2024-09-13T11:24:05.232 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:54.457345+0000", 2024-09-13T11:24:05.232 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 10.648118675999999, 2024-09-13T11:24:05.232 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.110750628, 2024-09-13T11:24:05.232 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:24:05.233 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:24:05.233 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:05.233 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:24:05.233 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:24:05.233 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:05.233 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 72182 2024-09-13T11:24:05.233 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.233 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:24:05.233 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.233 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:24:05.233 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:54.457345+0000", 2024-09-13T11:24:05.233 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:05.233 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.233 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.234 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:24:05.234 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:54.457345+0000", 2024-09-13T11:24:05.234 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:05.234 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.234 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.234 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:24:05.234 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:54.457348+0000", 2024-09-13T11:24:05.234 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.993e-06 2024-09-13T11:24:05.234 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.234 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.234 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:24:05.234 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:54.457415+0000", 2024-09-13T11:24:05.234 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.7150000000000006e-05 2024-09-13T11:24:05.234 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.235 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.235 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:24:05.235 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:54.457440+0000", 2024-09-13T11:24:05.235 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.4831999999999999e-05 2024-09-13T11:24:05.235 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.235 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.235 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:24:05.235 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:54.457455+0000", 2024-09-13T11:24:05.235 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4598e-05 2024-09-13T11:24:05.235 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.235 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.235 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:24:05.235 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:54.457601+0000", 2024-09-13T11:24:05.235 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000146751 2024-09-13T11:24:05.235 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.236 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.236 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:24:05.236 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:54.457945+0000", 2024-09-13T11:24:05.236 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00034404699999999999 2024-09-13T11:24:05.236 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.236 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.236 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 0", 2024-09-13T11:24:05.236 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:54.458321+0000", 2024-09-13T11:24:05.236 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000375509 2024-09-13T11:24:05.236 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.236 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.236 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:24:05.236 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:54.461092+0000", 2024-09-13T11:24:05.236 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0027710149999999999 2024-09-13T11:24:05.237 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.237 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.237 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:05.237 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:54.567725+0000", 2024-09-13T11:24:05.237 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.10663341799999999 2024-09-13T11:24:05.237 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.237 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.237 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:24:05.237 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:54.567988+0000", 2024-09-13T11:24:05.237 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00026269200000000003 2024-09-13T11:24:05.237 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.237 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.237 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:24:05.237 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:54.568096+0000", 2024-09-13T11:24:05.238 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000107623 2024-09-13T11:24:05.238 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:05.238 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:24:05.238 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:05.238 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:05.238 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:24:05.238 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-13T11:24:05.240 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-09-13T11:24:05.240 INFO:tasks.ceph.osd.10.smithi162.stderr:2024-09-13T11:24:05.237+0000 7fb59c9f0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26317) UID: 0 2024-09-13T11:24:05.340 INFO:tasks.ceph.osd.1.smithi092.stderr:2024-09-13T11:24:05.335+0000 7fe7789c5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33718) UID: 0 2024-09-13T11:24:05.368 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-13T11:24:05.425 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-13T11:24:05.425 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [ 2024-09-13T11:24:05.425 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.425 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.4665.0:84127 3.1f e107/78 3:f8261ead:::benchmark_data_smithi177_31116_object84126:head v 107'4297, mlcod=107'4294)", 2024-09-13T11:24:05.425 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:24:05.423053+0000", 2024-09-13T11:24:05.425 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 0.00080715500000000005, 2024-09-13T11:24:05.425 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000832671, 2024-09-13T11:24:05.425 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:24:05.425 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:24:05.425 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:24:05.426 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:24:05.426 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.426 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:24:05.426 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.423053+0000", 2024-09-13T11:24:05.426 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:05.426 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.426 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.426 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:24:05.426 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.423053+0000", 2024-09-13T11:24:05.426 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:05.427 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.427 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.427 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:24:05.427 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.423055+0000", 2024-09-13T11:24:05.427 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.6300000000000001e-06 2024-09-13T11:24:05.427 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.427 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.427 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:24:05.427 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.423129+0000", 2024-09-13T11:24:05.427 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.3967000000000005e-05 2024-09-13T11:24:05.428 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.428 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.428 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:24:05.428 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.423144+0000", 2024-09-13T11:24:05.428 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5177999999999999e-05 2024-09-13T11:24:05.428 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.428 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.428 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:24:05.428 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.423151+0000", 2024-09-13T11:24:05.428 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.024e-06 2024-09-13T11:24:05.428 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.428 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.429 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:24:05.429 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.423224+0000", 2024-09-13T11:24:05.429 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.2662000000000003e-05 2024-09-13T11:24:05.429 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.429 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.429 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:24:05.429 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.423254+0000", 2024-09-13T11:24:05.429 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.0139999999999999e-05 2024-09-13T11:24:05.429 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:05.429 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:24:05.429 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:05.429 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.429 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.430 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:84032 3.17 3:eb96f552:::benchmark_data_smithi177_31116_object84031:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-09-13T11:24:05.430 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:24:05.376844+0000", 2024-09-13T11:24:05.430 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 0.047017054000000003, 2024-09-13T11:24:05.430 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.047105437999999999, 2024-09-13T11:24:05.430 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:24:05.430 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:24:05.430 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "waiting for sub ops", 2024-09-13T11:24:05.430 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:24:05.430 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:24:05.431 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:05.431 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 84032 2024-09-13T11:24:05.431 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.431 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:24:05.431 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.431 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:24:05.431 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.376844+0000", 2024-09-13T11:24:05.431 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:05.431 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.431 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.431 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:24:05.431 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.376844+0000", 2024-09-13T11:24:05.431 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:05.431 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.431 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.432 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:24:05.432 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.376846+0000", 2024-09-13T11:24:05.432 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.694e-06 2024-09-13T11:24:05.432 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.432 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.432 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:24:05.432 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.376897+0000", 2024-09-13T11:24:05.432 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.0327000000000001e-05 2024-09-13T11:24:05.432 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.432 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.432 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:24:05.432 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.376919+0000", 2024-09-13T11:24:05.433 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.2538999999999999e-05 2024-09-13T11:24:05.433 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.433 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.433 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:24:05.433 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.376928+0000", 2024-09-13T11:24:05.433 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.0380000000000005e-06 2024-09-13T11:24:05.433 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.433 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.433 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:24:05.433 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.376992+0000", 2024-09-13T11:24:05.433 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.3466000000000005e-05 2024-09-13T11:24:05.433 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.433 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.433 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:24:05.434 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.377094+0000", 2024-09-13T11:24:05.434 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00010244800000000001 2024-09-13T11:24:05.434 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.434 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.434 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 3", 2024-09-13T11:24:05.434 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.377184+0000", 2024-09-13T11:24:05.434 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.9843e-05 2024-09-13T11:24:05.434 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.434 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.434 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:24:05.434 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.378582+0000", 2024-09-13T11:24:05.435 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0013983960000000001 2024-09-13T11:24:05.435 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:05.435 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:24:05.435 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:05.435 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:05.435 INFO:teuthology.orchestra.run.smithi162.stdout: ], 2024-09-13T11:24:05.435 INFO:teuthology.orchestra.run.smithi162.stdout: "num_ops": 2 2024-09-13T11:24:05.435 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-13T11:24:05.440 INFO:tasks.ceph.osd.3.smithi177.stderr:2024-09-13T11:24:05.440+0000 7f10adba4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26574) UID: 0 2024-09-13T11:24:05.443 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-09-13T11:24:05.541 INFO:tasks.ceph.osd.3.smithi177.stderr:2024-09-13T11:24:05.539+0000 7f10adba4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26574) UID: 0 2024-09-13T11:24:05.580 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-13T11:24:05.580 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [], 2024-09-13T11:24:05.581 INFO:teuthology.orchestra.run.smithi162.stdout: "complaint_time": 30, 2024-09-13T11:24:05.581 INFO:teuthology.orchestra.run.smithi162.stdout: "num_blocked_ops": 0 2024-09-13T11:24:05.581 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-13T11:24:05.597 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-09-13T11:24:05.642 INFO:tasks.ceph.osd.15.smithi177.stderr:2024-09-13T11:24:05.639+0000 7f9d6ce77640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26578) UID: 0 2024-09-13T11:24:05.729 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-13T11:24:05.729 INFO:teuthology.orchestra.run.smithi162.stdout: "size": 20, 2024-09-13T11:24:05.729 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 600, 2024-09-13T11:24:05.730 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [ 2024-09-13T11:24:05.730 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.730 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:25395 3.3 3:c1df8ccf:::benchmark_data_smithi177_31116_object25394:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e55)", 2024-09-13T11:24:05.730 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:04.465425+0000", 2024-09-13T11:24:05.730 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 61.261453201000002, 2024-09-13T11:24:05.730 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.18641877200000001, 2024-09-13T11:24:05.730 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:24:05.730 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:24:05.730 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:05.730 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:24:05.730 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:24:05.730 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:05.730 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 25395 2024-09-13T11:24:05.730 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.731 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:24:05.731 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.731 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:24:05.731 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.465425+0000", 2024-09-13T11:24:05.731 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:05.731 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.731 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.731 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:24:05.731 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.465425+0000", 2024-09-13T11:24:05.731 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:05.731 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.731 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.731 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:24:05.732 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.465427+0000", 2024-09-13T11:24:05.732 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.9970000000000001e-06 2024-09-13T11:24:05.732 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.732 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.732 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:24:05.732 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.465516+0000", 2024-09-13T11:24:05.732 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.8181000000000002e-05 2024-09-13T11:24:05.732 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.732 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.732 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:24:05.732 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.465534+0000", 2024-09-13T11:24:05.732 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.8204000000000001e-05 2024-09-13T11:24:05.732 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.732 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.733 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:24:05.733 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.465548+0000", 2024-09-13T11:24:05.733 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4195999999999999e-05 2024-09-13T11:24:05.733 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.733 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.733 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:24:05.733 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.468249+0000", 2024-09-13T11:24:05.733 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0027014729999999998 2024-09-13T11:24:05.733 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.733 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.733 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:24:05.733 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.468626+0000", 2024-09-13T11:24:05.734 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000376721 2024-09-13T11:24:05.734 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.734 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.734 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 7", 2024-09-13T11:24:05.734 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.469016+0000", 2024-09-13T11:24:05.734 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00038975000000000001 2024-09-13T11:24:05.734 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.734 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.734 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:24:05.734 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.471140+0000", 2024-09-13T11:24:05.734 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0021243400000000002 2024-09-13T11:24:05.735 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.735 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.735 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:05.735 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.651698+0000", 2024-09-13T11:24:05.735 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.18055781700000001 2024-09-13T11:24:05.735 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.735 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.735 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:24:05.735 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.651781+0000", 2024-09-13T11:24:05.735 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.3350999999999995e-05 2024-09-13T11:24:05.735 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.735 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.735 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:24:05.735 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.651844+0000", 2024-09-13T11:24:05.736 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.2742000000000003e-05 2024-09-13T11:24:05.736 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:05.736 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:24:05.736 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:05.736 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.736 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.736 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop_reply(client.4665.0:25395 3.3 e55/30 ondisk, result = 0)", 2024-09-13T11:24:05.736 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:04.471832+0000", 2024-09-13T11:24:05.736 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 61.255046899, 2024-09-13T11:24:05.736 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.18002319, 2024-09-13T11:24:05.736 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:24:05.736 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:24:05.737 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:24:05.737 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:24:05.737 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.737 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:24:05.737 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.471832+0000", 2024-09-13T11:24:05.737 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:05.737 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.737 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.737 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:24:05.737 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.471832+0000", 2024-09-13T11:24:05.737 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:05.737 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.737 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.737 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:24:05.738 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.471840+0000", 2024-09-13T11:24:05.738 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.7900000000000005e-06 2024-09-13T11:24:05.738 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.738 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.738 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:24:05.738 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.471850+0000", 2024-09-13T11:24:05.738 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.3899999999999999e-06 2024-09-13T11:24:05.738 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.738 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.738 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:24:05.738 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.651528+0000", 2024-09-13T11:24:05.738 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.17967789100000001 2024-09-13T11:24:05.738 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.738 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.739 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:24:05.739 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.651540+0000", 2024-09-13T11:24:05.739 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2e-05 2024-09-13T11:24:05.739 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.739 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.739 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:24:05.739 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.651676+0000", 2024-09-13T11:24:05.739 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000136033 2024-09-13T11:24:05.739 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.739 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.739 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:24:05.739 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.651689+0000", 2024-09-13T11:24:05.739 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.364e-05 2024-09-13T11:24:05.739 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.740 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.740 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:24:05.740 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.651855+0000", 2024-09-13T11:24:05.740 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000165446 2024-09-13T11:24:05.740 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:05.740 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:24:05.740 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:05.740 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.740 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.740 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:45810 3.17 3.9bd13e57 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e69)", 2024-09-13T11:24:05.740 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:21.347960+0000", 2024-09-13T11:24:05.740 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 44.378918247000001, 2024-09-13T11:24:05.740 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.56814464200000003, 2024-09-13T11:24:05.741 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:24:05.741 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:24:05.741 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-09-13T11:24:05.741 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:24:05.741 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:24:05.741 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:05.741 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 45810 2024-09-13T11:24:05.741 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.741 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:24:05.741 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.741 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:24:05.741 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:21.347960+0000", 2024-09-13T11:24:05.741 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:05.741 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.742 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.742 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:24:05.742 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:21.347960+0000", 2024-09-13T11:24:05.742 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:05.742 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.742 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.742 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:24:05.742 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:21.347965+0000", 2024-09-13T11:24:05.742 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.6720000000000003e-06 2024-09-13T11:24:05.742 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.742 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.742 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:24:05.742 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:21.348019+0000", 2024-09-13T11:24:05.743 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.4391000000000002e-05 2024-09-13T11:24:05.743 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.743 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.743 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:24:05.743 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:21.348052+0000", 2024-09-13T11:24:05.743 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.2246000000000003e-05 2024-09-13T11:24:05.743 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.743 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.743 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:24:05.743 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:21.348072+0000", 2024-09-13T11:24:05.743 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.0622e-05 2024-09-13T11:24:05.743 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.743 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.743 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:24:05.744 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:21.348278+0000", 2024-09-13T11:24:05.744 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000205391 2024-09-13T11:24:05.744 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.744 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.744 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-09-13T11:24:05.744 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:21.348292+0000", 2024-09-13T11:24:05.744 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4759e-05 2024-09-13T11:24:05.744 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.744 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.744 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:24:05.745 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:21.916064+0000", 2024-09-13T11:24:05.745 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.56777128600000004 2024-09-13T11:24:05.745 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.745 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.745 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:24:05.746 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:21.916105+0000", 2024-09-13T11:24:05.746 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.1275e-05 2024-09-13T11:24:05.746 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:05.746 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:24:05.746 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:05.746 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.746 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.747 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPush(3.17 72/70 [PushOp(3:ecb2eb17:::benchmark_data_smithi177_31116_object4630:head, version: 44'317, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:ecb2eb17:::benchmark_data_smithi177_31116_object4630:head@44'317, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:24:05.747 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:26.434926+0000", 2024-09-13T11:24:05.747 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 39.291952744, 2024-09-13T11:24:05.747 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.155542491, 2024-09-13T11:24:05.747 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:24:05.747 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:24:05.748 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:24:05.748 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:24:05.748 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.748 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:24:05.748 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:26.434926+0000", 2024-09-13T11:24:05.748 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:05.748 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.748 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.749 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:24:05.749 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:26.434926+0000", 2024-09-13T11:24:05.749 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:05.749 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.749 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.749 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:24:05.749 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:26.434928+0000", 2024-09-13T11:24:05.750 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.7489999999999999e-06 2024-09-13T11:24:05.750 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.750 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.750 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:24:05.750 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:26.434983+0000", 2024-09-13T11:24:05.750 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.4988000000000003e-05 2024-09-13T11:24:05.751 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.751 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.751 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:24:05.751 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:26.589751+0000", 2024-09-13T11:24:05.751 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.15476837299999999 2024-09-13T11:24:05.751 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.752 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.752 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:24:05.752 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:26.589778+0000", 2024-09-13T11:24:05.752 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.7385e-05 2024-09-13T11:24:05.752 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.752 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.753 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:24:05.753 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:26.589891+0000", 2024-09-13T11:24:05.753 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000112849 2024-09-13T11:24:05.753 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.753 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.753 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:24:05.754 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:26.589900+0000", 2024-09-13T11:24:05.754 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.3440000000000007e-06 2024-09-13T11:24:05.754 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.754 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.754 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:24:05.754 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:26.590468+0000", 2024-09-13T11:24:05.754 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00056780299999999997 2024-09-13T11:24:05.754 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:05.754 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:24:05.754 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:05.754 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.754 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.755 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:54653 3.23 3:c7a54800:::benchmark_data_smithi177_31116_object54652:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e75)", 2024-09-13T11:24:05.755 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:30.979059+0000", 2024-09-13T11:24:05.755 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 34.747819747999998, 2024-09-13T11:24:05.755 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.97284747100000002, 2024-09-13T11:24:05.755 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:24:05.756 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:24:05.756 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-09-13T11:24:05.756 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:24:05.756 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:24:05.756 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:05.756 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 54653 2024-09-13T11:24:05.757 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.757 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:24:05.757 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.757 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:24:05.757 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.979059+0000", 2024-09-13T11:24:05.757 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:05.757 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.758 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.758 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:24:05.758 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.979059+0000", 2024-09-13T11:24:05.758 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:05.758 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.758 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.758 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:24:05.759 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.979061+0000", 2024-09-13T11:24:05.759 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.4279999999999999e-06 2024-09-13T11:24:05.759 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.759 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.759 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:24:05.759 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.979114+0000", 2024-09-13T11:24:05.760 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.3254000000000002e-05 2024-09-13T11:24:05.760 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.760 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.760 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:24:05.760 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.979157+0000", 2024-09-13T11:24:05.760 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.2018000000000003e-05 2024-09-13T11:24:05.760 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.760 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.761 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:24:05.761 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.979172+0000", 2024-09-13T11:24:05.761 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5636000000000001e-05 2024-09-13T11:24:05.761 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.761 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.761 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:24:05.762 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:31.045109+0000", 2024-09-13T11:24:05.762 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.065936982000000005 2024-09-13T11:24:05.762 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.762 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.762 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-09-13T11:24:05.762 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:31.045121+0000", 2024-09-13T11:24:05.762 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1608e-05 2024-09-13T11:24:05.763 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.763 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.763 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:24:05.763 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:31.951636+0000", 2024-09-13T11:24:05.763 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.90651503899999997 2024-09-13T11:24:05.763 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.763 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.763 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:24:05.763 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:31.951906+0000", 2024-09-13T11:24:05.764 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000270506 2024-09-13T11:24:05.764 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:05.764 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:24:05.764 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:05.764 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.764 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.764 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPush(3.18 76/70 [PushOp(3:1a32c9cd:::benchmark_data_smithi177_31116_object12833:head, version: 49'818, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:1a32c9cd:::benchmark_data_smithi177_31116_object12833:head@49'818, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:24:05.764 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:32.522305+0000", 2024-09-13T11:24:05.764 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 33.204573664000002, 2024-09-13T11:24:05.764 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.40201207100000003, 2024-09-13T11:24:05.764 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:24:05.764 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:24:05.764 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:24:05.765 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:24:05.765 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.765 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:24:05.765 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.522305+0000", 2024-09-13T11:24:05.765 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:05.765 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.765 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.765 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:24:05.765 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.522305+0000", 2024-09-13T11:24:05.765 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:05.765 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.765 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.765 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:24:05.766 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.522309+0000", 2024-09-13T11:24:05.766 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.9829999999999998e-06 2024-09-13T11:24:05.766 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.766 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.766 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:24:05.766 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.522377+0000", 2024-09-13T11:24:05.766 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.8069000000000002e-05 2024-09-13T11:24:05.766 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.766 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.766 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:24:05.766 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.921950+0000", 2024-09-13T11:24:05.766 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.399573438 2024-09-13T11:24:05.766 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.766 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.767 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:24:05.767 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.921983+0000", 2024-09-13T11:24:05.767 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.2858000000000003e-05 2024-09-13T11:24:05.767 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.767 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.767 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:24:05.767 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.922389+0000", 2024-09-13T11:24:05.767 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00040612999999999999 2024-09-13T11:24:05.767 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.767 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.767 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:24:05.767 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.922421+0000", 2024-09-13T11:24:05.767 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.1356999999999998e-05 2024-09-13T11:24:05.768 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.768 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.768 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:24:05.768 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.924317+0000", 2024-09-13T11:24:05.768 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0018962359999999999 2024-09-13T11:24:05.768 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:05.768 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:24:05.768 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:05.768 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.768 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.768 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPush(3.17 76/70 [PushOp(3:ebcbce67:::benchmark_data_smithi177_31116_object18268:head, version: 53'1102, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:ebcbce67:::benchmark_data_smithi177_31116_object18268:head@53'1102, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:24:05.768 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:32.638908+0000", 2024-09-13T11:24:05.768 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 33.087970216000002, 2024-09-13T11:24:05.769 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.229064031, 2024-09-13T11:24:05.769 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:24:05.769 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:24:05.769 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:24:05.769 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:24:05.769 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.769 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:24:05.769 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.638908+0000", 2024-09-13T11:24:05.769 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:05.769 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.769 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.770 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:24:05.770 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.638908+0000", 2024-09-13T11:24:05.770 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:05.770 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.770 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.770 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:24:05.770 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.638910+0000", 2024-09-13T11:24:05.770 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.7379999999999999e-06 2024-09-13T11:24:05.770 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.770 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.770 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:24:05.770 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.638975+0000", 2024-09-13T11:24:05.770 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.4937000000000005e-05 2024-09-13T11:24:05.770 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.771 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.771 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:24:05.771 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.865860+0000", 2024-09-13T11:24:05.771 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.22688509800000001 2024-09-13T11:24:05.771 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.771 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.771 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:24:05.771 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.865890+0000", 2024-09-13T11:24:05.771 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.0273e-05 2024-09-13T11:24:05.771 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.771 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.771 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:24:05.771 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.866143+0000", 2024-09-13T11:24:05.772 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00025212800000000001 2024-09-13T11:24:05.772 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.772 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.772 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:24:05.772 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.866181+0000", 2024-09-13T11:24:05.772 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.8945000000000001e-05 2024-09-13T11:24:05.772 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.772 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.772 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:24:05.772 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.867972+0000", 2024-09-13T11:24:05.772 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001790912 2024-09-13T11:24:05.772 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:05.772 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:24:05.773 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:05.773 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.773 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.773 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:57435 3.3 3:c0db4777:::benchmark_data_smithi177_31116_object57434:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e80)", 2024-09-13T11:24:05.773 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:37.436034+0000", 2024-09-13T11:24:05.773 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 28.290844755999998, 2024-09-13T11:24:05.773 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.35949142000000001, 2024-09-13T11:24:05.773 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:24:05.773 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:24:05.773 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:05.773 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:24:05.773 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:24:05.774 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:05.774 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 57435 2024-09-13T11:24:05.774 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.774 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:24:05.774 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.774 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:24:05.774 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:37.436034+0000", 2024-09-13T11:24:05.774 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:05.774 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.774 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.774 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:24:05.775 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:37.436034+0000", 2024-09-13T11:24:05.775 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:05.775 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.775 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.775 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:24:05.775 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:37.436037+0000", 2024-09-13T11:24:05.775 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.7829999999999999e-06 2024-09-13T11:24:05.775 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.775 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.775 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:24:05.775 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:37.436090+0000", 2024-09-13T11:24:05.775 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.3665999999999997e-05 2024-09-13T11:24:05.775 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.775 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.776 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:24:05.776 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:37.436112+0000", 2024-09-13T11:24:05.776 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.1556000000000002e-05 2024-09-13T11:24:05.776 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.776 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.776 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:24:05.776 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:37.436122+0000", 2024-09-13T11:24:05.776 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0098000000000001e-05 2024-09-13T11:24:05.776 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.776 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.776 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:24:05.776 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:37.436187+0000", 2024-09-13T11:24:05.776 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.4857999999999995e-05 2024-09-13T11:24:05.776 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.777 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.777 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:24:05.777 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:37.436285+0000", 2024-09-13T11:24:05.777 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.8643999999999999e-05 2024-09-13T11:24:05.777 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.777 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.777 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 7", 2024-09-13T11:24:05.777 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:37.436423+0000", 2024-09-13T11:24:05.777 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00013809299999999999 2024-09-13T11:24:05.777 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.777 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.777 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:24:05.778 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:37.440125+0000", 2024-09-13T11:24:05.778 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0037015239999999999 2024-09-13T11:24:05.778 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.778 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.778 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:05.778 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:37.795397+0000", 2024-09-13T11:24:05.778 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.35527234400000002 2024-09-13T11:24:05.778 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.778 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.778 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:24:05.778 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:37.795485+0000", 2024-09-13T11:24:05.778 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.7538000000000001e-05 2024-09-13T11:24:05.778 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.779 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.779 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:24:05.779 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:37.795525+0000", 2024-09-13T11:24:05.779 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.0315999999999998e-05 2024-09-13T11:24:05.779 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:05.779 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:24:05.779 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:05.779 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.779 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.779 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPush(3.1f 80/78 [PushOp(3:fc3ed0c7:::benchmark_data_smithi177_31116_object672:head, version: 43'36, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:fc3ed0c7:::benchmark_data_smithi177_31116_object672:head@43'36, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:24:05.779 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:37.801754+0000", 2024-09-13T11:24:05.780 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 27.925124396000001, 2024-09-13T11:24:05.780 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.209613519, 2024-09-13T11:24:05.780 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:24:05.780 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:24:05.780 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:24:05.780 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:24:05.780 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.780 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:24:05.780 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:37.801754+0000", 2024-09-13T11:24:05.780 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:05.780 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.781 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.781 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:24:05.781 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:37.801754+0000", 2024-09-13T11:24:05.781 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:05.781 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.781 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.781 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:24:05.781 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:37.801756+0000", 2024-09-13T11:24:05.781 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.804e-06 2024-09-13T11:24:05.781 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.781 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.782 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:24:05.782 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:37.801947+0000", 2024-09-13T11:24:05.782 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000190636 2024-09-13T11:24:05.782 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.782 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.782 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:24:05.782 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:38.010478+0000", 2024-09-13T11:24:05.782 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.20853091100000001 2024-09-13T11:24:05.782 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.782 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.783 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:24:05.783 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:38.010506+0000", 2024-09-13T11:24:05.783 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.8793999999999999e-05 2024-09-13T11:24:05.783 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.783 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.784 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:24:05.784 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:38.010601+0000", 2024-09-13T11:24:05.784 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.4786999999999999e-05 2024-09-13T11:24:05.784 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.784 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.784 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:24:05.785 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:38.010617+0000", 2024-09-13T11:24:05.785 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5790999999999998e-05 2024-09-13T11:24:05.785 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.785 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.785 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:24:05.785 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:38.011368+0000", 2024-09-13T11:24:05.786 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00075079599999999997 2024-09-13T11:24:05.786 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:05.786 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:24:05.786 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:05.786 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.786 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.787 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:57985 3.3 3:c330bb03:::benchmark_data_smithi177_31116_object57984:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e80)", 2024-09-13T11:24:05.787 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:38.006112+0000", 2024-09-13T11:24:05.787 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 27.720766527999999, 2024-09-13T11:24:05.787 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.18277887900000001, 2024-09-13T11:24:05.787 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:24:05.787 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:24:05.788 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:05.788 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:24:05.788 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:24:05.788 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:05.788 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 57985 2024-09-13T11:24:05.788 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.788 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:24:05.789 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.789 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:24:05.789 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:38.006112+0000", 2024-09-13T11:24:05.789 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:05.789 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.789 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.790 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:24:05.790 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:38.006112+0000", 2024-09-13T11:24:05.790 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:05.790 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.790 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.790 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:24:05.790 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:38.006115+0000", 2024-09-13T11:24:05.790 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.6400000000000001e-06 2024-09-13T11:24:05.791 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.791 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.791 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:24:05.791 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:38.006162+0000", 2024-09-13T11:24:05.791 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.7599000000000003e-05 2024-09-13T11:24:05.791 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.791 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.791 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:24:05.791 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:38.006182+0000", 2024-09-13T11:24:05.791 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.9281999999999999e-05 2024-09-13T11:24:05.791 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.791 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.791 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:24:05.791 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:38.006191+0000", 2024-09-13T11:24:05.792 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.6490000000000003e-06 2024-09-13T11:24:05.792 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.792 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.792 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:24:05.792 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:38.006243+0000", 2024-09-13T11:24:05.792 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.1511999999999997e-05 2024-09-13T11:24:05.792 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.792 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.792 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:24:05.792 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:38.006333+0000", 2024-09-13T11:24:05.792 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.9843e-05 2024-09-13T11:24:05.792 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.792 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.792 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 7", 2024-09-13T11:24:05.793 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:38.006480+0000", 2024-09-13T11:24:05.793 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00014701699999999999 2024-09-13T11:24:05.793 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.793 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.793 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:24:05.793 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:38.008806+0000", 2024-09-13T11:24:05.793 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.002326172 2024-09-13T11:24:05.793 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.793 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.793 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:05.793 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:38.188815+0000", 2024-09-13T11:24:05.793 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.18000961100000001 2024-09-13T11:24:05.793 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.794 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.794 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:24:05.794 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:38.188857+0000", 2024-09-13T11:24:05.794 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.1591999999999998e-05 2024-09-13T11:24:05.794 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.794 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.794 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:24:05.794 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:38.188891+0000", 2024-09-13T11:24:05.794 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.3961999999999998e-05 2024-09-13T11:24:05.794 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:05.794 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:24:05.795 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:05.795 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.795 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.795 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:60433 3.3 3:c01f7ddc:::benchmark_data_smithi177_31116_object60432:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e85)", 2024-09-13T11:24:05.795 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:42.199382+0000", 2024-09-13T11:24:05.795 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 23.527496865, 2024-09-13T11:24:05.795 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.33918965200000001, 2024-09-13T11:24:05.795 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:24:05.795 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:24:05.795 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:05.795 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:24:05.795 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:24:05.795 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:05.795 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 60433 2024-09-13T11:24:05.796 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.796 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:24:05.796 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.796 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:24:05.796 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.199382+0000", 2024-09-13T11:24:05.796 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:05.796 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.796 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.796 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:24:05.796 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.199382+0000", 2024-09-13T11:24:05.796 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:05.796 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.797 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.797 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:24:05.797 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.199386+0000", 2024-09-13T11:24:05.797 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.8759999999999998e-06 2024-09-13T11:24:05.797 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.797 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.797 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:24:05.797 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.199455+0000", 2024-09-13T11:24:05.797 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.9305000000000001e-05 2024-09-13T11:24:05.797 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.797 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.797 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:24:05.797 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.199482+0000", 2024-09-13T11:24:05.798 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.6749000000000001e-05 2024-09-13T11:24:05.798 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.798 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.798 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:24:05.798 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.199503+0000", 2024-09-13T11:24:05.798 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.1137000000000001e-05 2024-09-13T11:24:05.798 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.798 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.798 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:24:05.798 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.199602+0000", 2024-09-13T11:24:05.798 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.8743e-05 2024-09-13T11:24:05.798 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.798 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.798 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:24:05.799 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.199784+0000", 2024-09-13T11:24:05.799 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00018265100000000001 2024-09-13T11:24:05.799 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.799 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.799 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 7", 2024-09-13T11:24:05.799 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.199955+0000", 2024-09-13T11:24:05.799 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00017067100000000001 2024-09-13T11:24:05.799 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.799 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.800 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:24:05.800 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.202075+0000", 2024-09-13T11:24:05.800 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0021206749999999998 2024-09-13T11:24:05.800 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.800 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.800 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:05.800 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.538395+0000", 2024-09-13T11:24:05.800 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.33631963599999998 2024-09-13T11:24:05.800 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.800 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.800 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:24:05.800 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.538498+0000", 2024-09-13T11:24:05.800 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00010267099999999999 2024-09-13T11:24:05.801 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.801 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.801 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:24:05.801 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.538571+0000", 2024-09-13T11:24:05.801 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.3538e-05 2024-09-13T11:24:05.801 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:05.801 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:24:05.801 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:05.801 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.801 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.801 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop_reply(client.4665.0:60433 3.3 e85/75 ondisk, result = 0)", 2024-09-13T11:24:05.801 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:42.203186+0000", 2024-09-13T11:24:05.801 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 23.523692687, 2024-09-13T11:24:05.802 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.33540096699999999, 2024-09-13T11:24:05.802 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:24:05.802 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:24:05.802 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:24:05.802 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:24:05.802 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.802 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:24:05.802 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.203186+0000", 2024-09-13T11:24:05.802 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:05.802 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.802 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.802 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:24:05.802 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.203186+0000", 2024-09-13T11:24:05.802 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:05.803 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.803 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.803 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:24:05.803 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.203194+0000", 2024-09-13T11:24:05.803 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.0020000000000006e-06 2024-09-13T11:24:05.803 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.803 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.803 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:24:05.803 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.203202+0000", 2024-09-13T11:24:05.803 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.9829999999999996e-06 2024-09-13T11:24:05.803 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.803 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.803 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:24:05.803 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.538182+0000", 2024-09-13T11:24:05.804 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.33498029899999998 2024-09-13T11:24:05.804 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.804 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.804 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:24:05.804 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.538196+0000", 2024-09-13T11:24:05.804 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3807e-05 2024-09-13T11:24:05.804 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.804 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.804 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:24:05.804 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.538371+0000", 2024-09-13T11:24:05.804 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00017541200000000001 2024-09-13T11:24:05.804 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.804 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.805 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:24:05.805 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.538386+0000", 2024-09-13T11:24:05.805 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4833e-05 2024-09-13T11:24:05.805 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.805 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.805 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:24:05.805 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.538587+0000", 2024-09-13T11:24:05.805 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000200631 2024-09-13T11:24:05.805 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:05.805 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:24:05.805 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:05.805 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.806 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.806 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:64349 3.17 3:ef87a22f:::benchmark_data_smithi177_31116_object64348:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90)", 2024-09-13T11:24:05.806 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:47.158656+0000", 2024-09-13T11:24:05.806 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 18.568222356, 2024-09-13T11:24:05.806 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.155165889, 2024-09-13T11:24:05.806 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:24:05.806 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:24:05.806 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:05.806 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:24:05.806 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:24:05.806 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:05.806 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 64349 2024-09-13T11:24:05.807 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.807 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:24:05.807 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.807 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:24:05.807 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:47.158656+0000", 2024-09-13T11:24:05.807 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:05.807 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.807 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.808 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:24:05.808 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:47.158656+0000", 2024-09-13T11:24:05.808 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:05.808 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.808 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.808 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:24:05.808 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:47.158659+0000", 2024-09-13T11:24:05.809 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.34e-06 2024-09-13T11:24:05.809 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.809 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.809 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:24:05.809 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:47.158708+0000", 2024-09-13T11:24:05.809 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.9416000000000003e-05 2024-09-13T11:24:05.809 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.809 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.810 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:24:05.810 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:47.158724+0000", 2024-09-13T11:24:05.810 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5556000000000001e-05 2024-09-13T11:24:05.810 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.810 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.810 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:24:05.810 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:47.158731+0000", 2024-09-13T11:24:05.810 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.4560000000000003e-06 2024-09-13T11:24:05.810 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.810 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.811 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:24:05.811 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:47.158781+0000", 2024-09-13T11:24:05.811 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.9558999999999998e-05 2024-09-13T11:24:05.811 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.811 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.811 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:24:05.811 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:47.158932+0000", 2024-09-13T11:24:05.811 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00015106900000000001 2024-09-13T11:24:05.811 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.811 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.811 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 3", 2024-09-13T11:24:05.811 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:47.159092+0000", 2024-09-13T11:24:05.811 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000160166 2024-09-13T11:24:05.811 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.812 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.812 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:24:05.812 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:47.162332+0000", 2024-09-13T11:24:05.812 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0032402640000000001 2024-09-13T11:24:05.812 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.812 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.812 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:05.812 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:47.313747+0000", 2024-09-13T11:24:05.812 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.15141496199999999 2024-09-13T11:24:05.812 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.812 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.812 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:24:05.812 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:47.313781+0000", 2024-09-13T11:24:05.812 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.4267999999999998e-05 2024-09-13T11:24:05.813 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.813 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.813 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:24:05.813 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:47.313822+0000", 2024-09-13T11:24:05.813 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.0833e-05 2024-09-13T11:24:05.813 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:05.813 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:24:05.813 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:05.813 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.813 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.813 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:69547 3.18 3:1b8f2c3d:::benchmark_data_smithi177_31116_object69546:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94)", 2024-09-13T11:24:05.813 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:52.131834+0000", 2024-09-13T11:24:05.813 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 13.595044971, 2024-09-13T11:24:05.814 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.17455321700000001, 2024-09-13T11:24:05.814 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:24:05.814 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:24:05.814 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:05.814 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:24:05.814 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:24:05.814 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:05.814 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 69547 2024-09-13T11:24:05.814 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.815 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:24:05.815 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.815 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:24:05.815 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:52.131834+0000", 2024-09-13T11:24:05.815 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:05.815 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.815 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.815 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:24:05.816 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:52.131834+0000", 2024-09-13T11:24:05.816 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:05.816 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.816 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.816 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:24:05.816 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:52.131841+0000", 2024-09-13T11:24:05.816 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.0820000000000001e-06 2024-09-13T11:24:05.816 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.816 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.817 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:24:05.817 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:52.131887+0000", 2024-09-13T11:24:05.817 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.6817000000000003e-05 2024-09-13T11:24:05.817 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.817 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.817 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:24:05.817 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:52.131905+0000", 2024-09-13T11:24:05.817 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.7380999999999998e-05 2024-09-13T11:24:05.817 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.817 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.817 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:24:05.817 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:52.131915+0000", 2024-09-13T11:24:05.817 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0132999999999999e-05 2024-09-13T11:24:05.818 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.818 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.818 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:24:05.818 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:52.132002+0000", 2024-09-13T11:24:05.818 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.6594999999999993e-05 2024-09-13T11:24:05.818 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.818 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.818 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:24:05.818 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:52.132134+0000", 2024-09-13T11:24:05.818 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000132817 2024-09-13T11:24:05.818 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.818 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.818 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 0", 2024-09-13T11:24:05.819 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:52.132258+0000", 2024-09-13T11:24:05.819 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00012339299999999999 2024-09-13T11:24:05.819 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.819 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.819 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:24:05.819 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:52.134724+0000", 2024-09-13T11:24:05.820 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0024660910000000001 2024-09-13T11:24:05.820 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.820 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.820 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:05.820 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:52.306299+0000", 2024-09-13T11:24:05.820 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.171574749 2024-09-13T11:24:05.820 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.820 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.820 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:24:05.820 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:52.306348+0000", 2024-09-13T11:24:05.820 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.9329000000000002e-05 2024-09-13T11:24:05.821 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.821 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.821 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:24:05.821 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:52.306387+0000", 2024-09-13T11:24:05.821 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.8829999999999999e-05 2024-09-13T11:24:05.821 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:05.821 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:24:05.821 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:05.821 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.821 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.821 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:69556 3.18 3:1d3dbeba:::benchmark_data_smithi177_31116_object69555:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94)", 2024-09-13T11:24:05.821 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:52.142180+0000", 2024-09-13T11:24:05.821 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 13.584698617000001, 2024-09-13T11:24:05.822 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.21280995899999999, 2024-09-13T11:24:05.822 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:24:05.822 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:24:05.822 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:05.822 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:24:05.822 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:24:05.822 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:05.822 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 69556 2024-09-13T11:24:05.822 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.822 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:24:05.822 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.822 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:24:05.822 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:52.142180+0000", 2024-09-13T11:24:05.822 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:05.823 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.823 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.823 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:24:05.823 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:52.142180+0000", 2024-09-13T11:24:05.823 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:05.823 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.823 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.823 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:24:05.823 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:52.142181+0000", 2024-09-13T11:24:05.823 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.3699999999999999e-07 2024-09-13T11:24:05.823 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.823 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.823 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:24:05.823 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:52.142212+0000", 2024-09-13T11:24:05.824 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.1334000000000002e-05 2024-09-13T11:24:05.824 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.824 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.824 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:24:05.824 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:52.142219+0000", 2024-09-13T11:24:05.824 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.9990000000000002e-06 2024-09-13T11:24:05.824 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.824 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.824 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:24:05.824 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:52.142224+0000", 2024-09-13T11:24:05.824 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.0259999999999998e-06 2024-09-13T11:24:05.824 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.824 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.825 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:24:05.825 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:52.142276+0000", 2024-09-13T11:24:05.825 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.1436e-05 2024-09-13T11:24:05.825 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.825 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.825 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:24:05.825 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:52.142396+0000", 2024-09-13T11:24:05.825 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00012011 2024-09-13T11:24:05.825 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.825 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.825 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 0", 2024-09-13T11:24:05.826 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:52.142503+0000", 2024-09-13T11:24:05.826 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000107701 2024-09-13T11:24:05.826 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.826 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.826 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:24:05.826 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:52.144566+0000", 2024-09-13T11:24:05.826 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.002062749 2024-09-13T11:24:05.826 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.826 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.826 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:05.826 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:52.354839+0000", 2024-09-13T11:24:05.826 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.21027254000000001 2024-09-13T11:24:05.826 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.826 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.827 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:24:05.827 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:52.354932+0000", 2024-09-13T11:24:05.827 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.3510000000000004e-05 2024-09-13T11:24:05.827 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.827 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.827 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:24:05.827 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:52.354990+0000", 2024-09-13T11:24:05.827 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.7617000000000001e-05 2024-09-13T11:24:05.827 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:05.827 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:24:05.827 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:05.827 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.828 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.828 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:82784 3.18 3:1e10e57e:::benchmark_data_smithi177_31116_object82783:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105)", 2024-09-13T11:24:05.828 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:24:03.815559+0000", 2024-09-13T11:24:05.828 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 1.9113192190000001, 2024-09-13T11:24:05.828 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.19711926199999999, 2024-09-13T11:24:05.828 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:24:05.828 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:24:05.828 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:05.828 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:24:05.828 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:24:05.828 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:05.828 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 82784 2024-09-13T11:24:05.829 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.829 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:24:05.829 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.829 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:24:05.829 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:03.815559+0000", 2024-09-13T11:24:05.829 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:05.829 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.829 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.829 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:24:05.829 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:03.815559+0000", 2024-09-13T11:24:05.829 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:05.829 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.829 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.830 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:24:05.830 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:03.815568+0000", 2024-09-13T11:24:05.830 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.2029999999999992e-06 2024-09-13T11:24:05.830 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.830 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.830 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:24:05.830 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:03.815618+0000", 2024-09-13T11:24:05.830 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.0285999999999999e-05 2024-09-13T11:24:05.830 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.830 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.830 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:24:05.830 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:03.815646+0000", 2024-09-13T11:24:05.831 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.7747e-05 2024-09-13T11:24:05.831 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.831 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.831 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:24:05.831 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:03.815660+0000", 2024-09-13T11:24:05.831 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4633e-05 2024-09-13T11:24:05.831 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.831 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.831 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:24:05.831 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:03.815781+0000", 2024-09-13T11:24:05.831 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000120831 2024-09-13T11:24:05.831 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.831 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.831 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:24:05.832 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:03.816015+0000", 2024-09-13T11:24:05.832 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00023364600000000001 2024-09-13T11:24:05.832 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.832 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.832 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 0", 2024-09-13T11:24:05.832 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:03.816241+0000", 2024-09-13T11:24:05.832 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00022667600000000001 2024-09-13T11:24:05.832 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.832 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.832 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:24:05.832 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:03.819236+0000", 2024-09-13T11:24:05.832 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0029943130000000002 2024-09-13T11:24:05.832 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.833 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.833 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:05.833 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.012545+0000", 2024-09-13T11:24:05.833 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.19330974100000001 2024-09-13T11:24:05.833 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.833 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.833 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:24:05.833 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.012623+0000", 2024-09-13T11:24:05.833 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.7565999999999996e-05 2024-09-13T11:24:05.833 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.833 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.833 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:24:05.833 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.012679+0000", 2024-09-13T11:24:05.834 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.5619999999999999e-05 2024-09-13T11:24:05.834 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:05.834 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:24:05.834 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:05.834 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.834 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.834 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop_reply(client.4665.0:82784 3.18 e105/91 ondisk, result = 0)", 2024-09-13T11:24:05.834 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:24:03.819384+0000", 2024-09-13T11:24:05.834 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 1.907494939, 2024-09-13T11:24:05.834 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.19330499100000001, 2024-09-13T11:24:05.834 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:24:05.834 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:24:05.834 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:24:05.835 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:24:05.835 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.835 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:24:05.835 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:03.819384+0000", 2024-09-13T11:24:05.835 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:05.835 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.835 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.835 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:24:05.835 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:03.819384+0000", 2024-09-13T11:24:05.835 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:05.835 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.835 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.835 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:24:05.835 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:03.819393+0000", 2024-09-13T11:24:05.836 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.5429999999999999e-06 2024-09-13T11:24:05.836 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.836 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.836 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:24:05.836 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:03.819403+0000", 2024-09-13T11:24:05.836 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0309000000000001e-05 2024-09-13T11:24:05.836 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.836 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.836 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:24:05.836 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.010699+0000", 2024-09-13T11:24:05.836 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.19129547699999999 2024-09-13T11:24:05.836 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.837 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.837 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:24:05.837 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.010708+0000", 2024-09-13T11:24:05.837 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.6770000000000001e-06 2024-09-13T11:24:05.837 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.837 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.837 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:24:05.837 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.012528+0000", 2024-09-13T11:24:05.837 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00182087 2024-09-13T11:24:05.837 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.837 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.837 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:24:05.838 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.012538+0000", 2024-09-13T11:24:05.838 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.6399999999999992e-06 2024-09-13T11:24:05.838 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.838 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.838 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:24:05.838 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.012689+0000", 2024-09-13T11:24:05.838 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00015047500000000001 2024-09-13T11:24:05.838 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:05.838 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:24:05.838 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:05.838 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.838 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.838 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:83144 3.18 3:1c2a5085:::benchmark_data_smithi177_31116_object83143:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106)", 2024-09-13T11:24:05.839 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:24:04.186395+0000", 2024-09-13T11:24:05.839 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 1.540483568, 2024-09-13T11:24:05.839 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.29655527599999998, 2024-09-13T11:24:05.839 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:24:05.839 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:24:05.839 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:05.839 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:24:05.839 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:24:05.839 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:05.839 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 83144 2024-09-13T11:24:05.839 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.839 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:24:05.839 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.839 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:24:05.840 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.186395+0000", 2024-09-13T11:24:05.840 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:05.840 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.840 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.840 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:24:05.840 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.186395+0000", 2024-09-13T11:24:05.840 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:05.840 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.840 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.840 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:24:05.840 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.186398+0000", 2024-09-13T11:24:05.840 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.5299999999999999e-06 2024-09-13T11:24:05.841 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.841 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.841 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:24:05.841 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.186438+0000", 2024-09-13T11:24:05.841 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.0293000000000002e-05 2024-09-13T11:24:05.841 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.841 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.841 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:24:05.841 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.186450+0000", 2024-09-13T11:24:05.841 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2687000000000001e-05 2024-09-13T11:24:05.841 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.841 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.841 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:24:05.841 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.186456+0000", 2024-09-13T11:24:05.842 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.446e-06 2024-09-13T11:24:05.842 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.842 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.842 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:24:05.842 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.186503+0000", 2024-09-13T11:24:05.842 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.7491000000000003e-05 2024-09-13T11:24:05.842 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.842 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.842 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:24:05.842 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.186618+0000", 2024-09-13T11:24:05.842 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000114999 2024-09-13T11:24:05.842 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.842 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.843 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 0", 2024-09-13T11:24:05.843 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.186712+0000", 2024-09-13T11:24:05.843 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.3805000000000001e-05 2024-09-13T11:24:05.843 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.843 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.843 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:24:05.843 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.189753+0000", 2024-09-13T11:24:05.843 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.003040713 2024-09-13T11:24:05.843 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.843 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.843 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:05.843 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.482662+0000", 2024-09-13T11:24:05.843 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.29290945699999998 2024-09-13T11:24:05.844 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.844 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.844 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:24:05.844 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.482831+0000", 2024-09-13T11:24:05.844 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00016900699999999999 2024-09-13T11:24:05.844 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.844 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.844 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:24:05.844 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.482950+0000", 2024-09-13T11:24:05.844 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000118848 2024-09-13T11:24:05.844 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:05.844 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:24:05.845 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:05.845 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.845 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.845 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop_reply(client.4665.0:83144 3.18 e106/91 ondisk, result = 0)", 2024-09-13T11:24:05.845 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:24:04.188413+0000", 2024-09-13T11:24:05.845 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 1.538465558, 2024-09-13T11:24:05.845 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.294557562, 2024-09-13T11:24:05.845 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:24:05.845 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:24:05.845 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:24:05.845 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:24:05.845 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.845 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:24:05.845 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.188413+0000", 2024-09-13T11:24:05.846 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:05.846 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.846 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.846 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:24:05.846 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.188413+0000", 2024-09-13T11:24:05.846 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:05.846 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.846 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.846 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:24:05.846 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.188414+0000", 2024-09-13T11:24:05.846 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4309999999999999e-06 2024-09-13T11:24:05.846 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.847 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.847 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:24:05.847 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.188419+0000", 2024-09-13T11:24:05.847 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.2889999999999999e-06 2024-09-13T11:24:05.847 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.847 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.847 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:24:05.847 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.482442+0000", 2024-09-13T11:24:05.847 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.294023168 2024-09-13T11:24:05.847 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.847 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.847 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:24:05.847 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.482459+0000", 2024-09-13T11:24:05.847 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.7013000000000001e-05 2024-09-13T11:24:05.848 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.848 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.848 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:24:05.848 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.482615+0000", 2024-09-13T11:24:05.848 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00015620400000000001 2024-09-13T11:24:05.848 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.848 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.848 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:24:05.848 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.482644+0000", 2024-09-13T11:24:05.848 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.9093000000000001e-05 2024-09-13T11:24:05.848 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.848 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.849 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:24:05.849 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.482971+0000", 2024-09-13T11:24:05.849 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00032636399999999999 2024-09-13T11:24:05.849 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:05.849 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:24:05.849 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:05.849 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.849 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.849 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPush(3.1f 106/78 [PushOp(3:fd4be819:::benchmark_data_smithi177_31116_object54560:head, version: 74'3410, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:fd4be819:::benchmark_data_smithi177_31116_object54560:head@74'3410, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:24:05.849 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:24:04.397161+0000", 2024-09-13T11:24:05.849 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 1.3297171489999999, 2024-09-13T11:24:05.849 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.51611096199999995, 2024-09-13T11:24:05.850 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:24:05.850 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:24:05.850 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:24:05.850 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:24:05.850 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.850 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:24:05.850 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.397161+0000", 2024-09-13T11:24:05.850 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:05.850 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.850 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.850 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:24:05.850 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.397161+0000", 2024-09-13T11:24:05.851 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:05.851 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.851 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.851 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:24:05.851 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.397163+0000", 2024-09-13T11:24:05.851 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4720000000000001e-06 2024-09-13T11:24:05.851 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.851 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.851 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:24:05.851 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.397228+0000", 2024-09-13T11:24:05.851 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.5141999999999994e-05 2024-09-13T11:24:05.851 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.851 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.852 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:24:05.852 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.911583+0000", 2024-09-13T11:24:05.852 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.51435473700000001 2024-09-13T11:24:05.852 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.852 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.852 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:24:05.852 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.911615+0000", 2024-09-13T11:24:05.852 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.2620000000000003e-05 2024-09-13T11:24:05.852 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.852 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.852 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:24:05.853 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.911759+0000", 2024-09-13T11:24:05.853 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00014376300000000001 2024-09-13T11:24:05.853 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.853 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.853 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:24:05.853 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.911779+0000", 2024-09-13T11:24:05.853 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.0003000000000001e-05 2024-09-13T11:24:05.853 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:05.853 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:05.853 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:24:05.853 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.913272+0000", 2024-09-13T11:24:05.853 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0014932249999999999 2024-09-13T11:24:05.853 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:05.854 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:24:05.854 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:05.854 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:05.854 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:24:05.854 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-13T11:24:05.854 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T11:24:05.857 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-13T11:24:05.858 INFO:teuthology.orchestra.run.smithi001.stderr:dumped all 2024-09-13T11:24:05.858 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_ops_in_flight 2024-09-13T11:24:05.860 INFO:teuthology.orchestra.run.smithi001.stdout:{"pg_ready":true,"pg_map":{"version":150,"stamp":"2024-09-13T11:24:04.937315+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":5264900659,"num_objects":80332,"num_object_clones":0,"num_object_copies":160664,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1468,"num_objects_unfound":0,"num_objects_dirty":80332,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":33946,"num_write_kb":2165746,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10892,"num_bytes_recovered":713719827,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":140889,"ondisk_log_size":140889,"up":90,"acting":90,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":118,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":16,"kb":1509949440,"kb_used":13414956,"kb_used_data":12523692,"kb_used_omap":27,"kb_used_meta":890916,"kb_avail":1496534484,"statfs":{"total":1546188226560,"available":1532451311616,"internally_reserved":0,"allocated":12824260608,"data_stored":12817144159,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28358,"internal_metadata":912298298},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,4,1,3,3,0,1,1,0,1],"upper_bound":1024},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":123895808,"num_objects":1890,"num_object_clones":0,"num_object_copies":3780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-606,"num_objects_unfound":0,"num_objects_dirty":1890,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-654,"num_write_kb":-41824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-1430,"num_bytes_recovered":-93716480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"3.696420"},"pg_stats":[{"pgid":"3.23","version":"102'3545","reported_seq":6265,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-13T11:24:00.318559+0000","last_change":"2024-09-13T11:23:48.133184+0000","last_active":"2024-09-13T11:24:00.318559+0000","last_peered":"2024-09-13T11:24:00.318559+0000","last_clean":"2024-09-13T11:24:00.318559+0000","last_became_active":"2024-09-13T11:23:47.926677+0000","last_became_peered":"2024-09-13T11:23:47.926677+0000","last_unstale":"2024-09-13T11:24:00.318559+0000","last_undegraded":"2024-09-13T11:24:00.318559+0000","last_fullsized":"2024-09-13T11:24:00.318559+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":6,"last_scrub":"91'3278","last_scrub_stamp":"2024-09-13T11:23:48.133084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:48.133084+0000","objects_scrubbed":1029,"log_size":3545,"log_dups_size":0,"ondisk_log_size":3545,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:12.874134+0000","scrub_duration":139,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":85000192,"num_objects":1297,"num_object_clones":0,"num_object_copies":2594,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1297,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":268,"num_write_kb":17152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9],"acting":[5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.22","version":"102'3485","reported_seq":5392,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-13T11:24:00.410951+0000","last_change":"2024-09-13T11:23:47.300446+0000","last_active":"2024-09-13T11:24:00.410951+0000","last_peered":"2024-09-13T11:24:00.410951+0000","last_clean":"2024-09-13T11:24:00.410951+0000","last_became_active":"2024-09-13T11:23:46.925318+0000","last_became_peered":"2024-09-13T11:23:46.925318+0000","last_unstale":"2024-09-13T11:24:00.410951+0000","last_undegraded":"2024-09-13T11:24:00.410951+0000","last_fullsized":"2024-09-13T11:24:00.410951+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":6,"last_scrub":"90'3246","last_scrub_stamp":"2024-09-13T11:23:47.300339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:47.300339+0000","objects_scrubbed":1033,"log_size":3485,"log_dups_size":0,"ondisk_log_size":3485,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:24:49.780485+0000","scrub_duration":203,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":83492864,"num_objects":1274,"num_object_clones":0,"num_object_copies":2548,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1274,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":241,"num_write_kb":15424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,14],"acting":[0,14],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.21","version":"101'3436","reported_seq":6136,"reported_epoch":102,"state":"peering","last_fresh":"2024-09-13T11:24:00.239108+0000","last_change":"2024-09-13T11:24:00.239108+0000","last_active":"2024-09-13T11:24:00.239108+0000","last_peered":"2024-09-13T11:24:00.144417+0000","last_clean":"2024-09-13T11:23:32.050222+0000","last_became_active":"2024-09-13T11:23:35.026171+0000","last_became_peered":"2024-09-13T11:23:35.026171+0000","last_unstale":"2024-09-13T11:24:00.239108+0000","last_undegraded":"2024-09-13T11:24:00.239108+0000","last_fullsized":"2024-09-13T11:24:00.239108+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":6,"last_scrub":"69'2784","last_scrub_stamp":"2024-09-13T11:23:21.595419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:21.595419+0000","objects_scrubbed":1386,"log_size":3436,"log_dups_size":0,"ondisk_log_size":3436,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":311,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":78905344,"num_objects":1204,"num_object_clones":0,"num_object_copies":2408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1204,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":511,"num_write_kb":32704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":836,"num_bytes_recovered":54788096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8],"acting":[0,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.20","version":"102'3514","reported_seq":5536,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-13T11:24:00.246567+0000","last_change":"2024-09-13T11:23:47.969503+0000","last_active":"2024-09-13T11:24:00.246567+0000","last_peered":"2024-09-13T11:24:00.246567+0000","last_clean":"2024-09-13T11:24:00.246567+0000","last_became_active":"2024-09-13T11:23:33.954946+0000","last_became_peered":"2024-09-13T11:23:33.954946+0000","last_unstale":"2024-09-13T11:24:00.246567+0000","last_undegraded":"2024-09-13T11:24:00.246567+0000","last_fullsized":"2024-09-13T11:24:00.246567+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":6,"last_scrub":"91'3270","last_scrub_stamp":"2024-09-13T11:23:47.969372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:47.969372+0000","objects_scrubbed":1061,"log_size":3514,"log_dups_size":0,"ondisk_log_size":3514,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:24:48.255263+0000","scrub_duration":1327,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":85852160,"num_objects":1310,"num_object_clones":0,"num_object_copies":2620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1310,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":249,"num_write_kb":15936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,0],"acting":[12,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[]},{"pgid":"3.1f","version":"104'4210","reported_seq":8237,"reported_epoch":104,"state":"active+recovering+remapped","last_fresh":"2024-09-13T11:24:02.066916+0000","last_change":"2024-09-13T11:23:37.329722+0000","last_active":"2024-09-13T11:24:02.066916+0000","last_peered":"2024-09-13T11:24:02.066916+0000","last_clean":"2024-09-13T11:23:32.316727+0000","last_became_active":"2024-09-13T11:23:35.053803+0000","last_became_peered":"2024-09-13T11:23:35.053803+0000","last_unstale":"2024-09-13T11:24:02.066916+0000","last_undegraded":"2024-09-13T11:24:02.066916+0000","last_fullsized":"2024-09-13T11:24:02.066916+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:22:24.896055+0000","objects_scrubbed":0,"log_size":4210,"log_dups_size":0,"ondisk_log_size":4210,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":164069376,"num_objects":2504,"num_object_clones":0,"num_object_copies":5008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":730,"num_objects_unfound":0,"num_objects_dirty":2504,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2504,"num_write_kb":160224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1286,"num_bytes_recovered":84279296,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,14],"acting":[11,10],"avail_no_missing":["11","12"],"object_location_counts":[{"shards":"10,11","objects":2139},{"shards":"11,12","objects":365}],"blocked_by":[],"up_primary":10,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1e","version":"101'4289","reported_seq":14630,"reported_epoch":102,"state":"creating+peering","last_fresh":"2024-09-13T11:24:00.236853+0000","last_change":"2024-09-13T11:24:01.002570+0000","last_active":"2024-09-13T11:24:01.002570+0000","last_peered":"2024-09-13T11:24:00.236853+0000","last_clean":"2024-09-13T11:23:30.974769+0000","last_became_active":"2024-09-13T11:23:35.075137+0000","last_became_peered":"2024-09-13T11:23:35.075137+0000","last_unstale":"2024-09-13T11:24:00.236853+0000","last_undegraded":"2024-09-13T11:24:00.236853+0000","last_fullsized":"2024-09-13T11:24:00.236853+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:22:24.896055+0000","objects_scrubbed":0,"log_size":4289,"log_dups_size":0,"ondisk_log_size":4289,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":165740544,"num_objects":2529,"num_object_clones":0,"num_object_copies":5058,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2529,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2529,"num_write_kb":161856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2573,"num_bytes_recovered":168591360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"101'4115","reported_seq":9379,"reported_epoch":102,"state":"peering","last_fresh":"2024-09-13T11:24:00.240669+0000","last_change":"2024-09-13T11:24:00.240669+0000","last_active":"2024-09-13T11:24:00.240669+0000","last_peered":"2024-09-13T11:24:00.209888+0000","last_clean":"2024-09-13T11:23:32.393844+0000","last_became_active":"2024-09-13T11:23:35.080149+0000","last_became_peered":"2024-09-13T11:23:35.080149+0000","last_unstale":"2024-09-13T11:24:00.240669+0000","last_undegraded":"2024-09-13T11:24:00.240669+0000","last_fullsized":"2024-09-13T11:24:00.240669+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:22:24.896055+0000","objects_scrubbed":0,"log_size":4115,"log_dups_size":0,"ondisk_log_size":4115,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":158859264,"num_objects":2424,"num_object_clones":0,"num_object_copies":4848,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2424,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2424,"num_write_kb":155136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1700,"num_bytes_recovered":111411200,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,15],"acting":[2,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1c","version":"104'4310","reported_seq":7986,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-13T11:24:01.961650+0000","last_change":"2024-09-13T11:24:00.529312+0000","last_active":"2024-09-13T11:24:01.961650+0000","last_peered":"2024-09-13T11:24:01.961650+0000","last_clean":"2024-09-13T11:24:01.961650+0000","last_became_active":"2024-09-13T11:23:33.989492+0000","last_became_peered":"2024-09-13T11:23:33.989492+0000","last_unstale":"2024-09-13T11:24:01.961650+0000","last_undegraded":"2024-09-13T11:24:01.961650+0000","last_fullsized":"2024-09-13T11:24:01.961650+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":5,"last_scrub":"102'4277","last_scrub_stamp":"2024-09-13T11:24:00.529243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:24:00.529243+0000","objects_scrubbed":2544,"log_size":4310,"log_dups_size":0,"ondisk_log_size":4310,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:15.595052+0000","scrub_duration":259,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":169017344,"num_objects":2579,"num_object_clones":0,"num_object_copies":5158,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2579,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":35,"num_write_kb":2240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7],"acting":[11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1b","version":"102'4216","reported_seq":9052,"reported_epoch":102,"state":"active+recovering+remapped","last_fresh":"2024-09-13T11:24:00.295969+0000","last_change":"2024-09-13T11:23:35.097804+0000","last_active":"2024-09-13T11:24:00.295969+0000","last_peered":"2024-09-13T11:24:00.295969+0000","last_clean":"2024-09-13T11:23:32.309147+0000","last_became_active":"2024-09-13T11:23:35.080340+0000","last_became_peered":"2024-09-13T11:23:35.080340+0000","last_unstale":"2024-09-13T11:24:00.295969+0000","last_undegraded":"2024-09-13T11:24:00.295969+0000","last_fullsized":"2024-09-13T11:24:00.295969+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:22:24.896055+0000","objects_scrubbed":0,"log_size":4216,"log_dups_size":0,"ondisk_log_size":4216,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":162889728,"num_objects":2486,"num_object_clones":0,"num_object_copies":4972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":407,"num_objects_unfound":0,"num_objects_dirty":2486,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2486,"num_write_kb":159072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1556,"num_bytes_recovered":101974016,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,15],"acting":[1,12],"avail_no_missing":["1","13"],"object_location_counts":[{"shards":"1,12","objects":2282},{"shards":"1,12,13","objects":1},{"shards":"1,13","objects":203}],"blocked_by":[],"up_primary":12,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1a","version":"101'4113","reported_seq":8724,"reported_epoch":101,"state":"active+recovering+remapped","last_fresh":"2024-09-13T11:24:00.027277+0000","last_change":"2024-09-13T11:23:35.089105+0000","last_active":"2024-09-13T11:24:00.027277+0000","last_peered":"2024-09-13T11:24:00.027277+0000","last_clean":"2024-09-13T11:23:32.455463+0000","last_became_active":"2024-09-13T11:23:35.072595+0000","last_became_peered":"2024-09-13T11:23:35.072595+0000","last_unstale":"2024-09-13T11:24:00.027277+0000","last_undegraded":"2024-09-13T11:24:00.027277+0000","last_fullsized":"2024-09-13T11:24:00.027277+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:22:24.896055+0000","objects_scrubbed":0,"log_size":4113,"log_dups_size":0,"ondisk_log_size":4113,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":157745152,"num_objects":2407,"num_object_clones":0,"num_object_copies":4814,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":331,"num_objects_unfound":0,"num_objects_dirty":2407,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2407,"num_write_kb":154048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1482,"num_bytes_recovered":97124352,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9],"acting":[3,4],"avail_no_missing":["3","7"],"object_location_counts":[{"shards":"3,4","objects":2241},{"shards":"3,7","objects":165},{"shards":"3,7,9","objects":1}],"blocked_by":[],"up_primary":4,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.19","version":"104'3901","reported_seq":8200,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-13T11:24:01.930513+0000","last_change":"2024-09-13T11:23:49.429522+0000","last_active":"2024-09-13T11:24:01.930513+0000","last_peered":"2024-09-13T11:24:01.930513+0000","last_clean":"2024-09-13T11:24:01.930513+0000","last_became_active":"2024-09-13T11:23:41.924941+0000","last_became_peered":"2024-09-13T11:23:41.924941+0000","last_unstale":"2024-09-13T11:24:01.930513+0000","last_undegraded":"2024-09-13T11:24:01.930513+0000","last_fullsized":"2024-09-13T11:24:01.930513+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":5,"last_scrub":"92'3415","last_scrub_stamp":"2024-09-13T11:23:49.429429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:49.429429+0000","objects_scrubbed":1979,"log_size":3901,"log_dups_size":0,"ondisk_log_size":3901,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:04.629542+0000","scrub_duration":587,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":162398208,"num_objects":2478,"num_object_clones":0,"num_object_copies":4956,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2478,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":499,"num_write_kb":31936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1],"acting":[9,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.18","version":"102'3900","reported_seq":8460,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-13T11:24:00.656686+0000","last_change":"2024-09-13T11:23:56.067740+0000","last_active":"2024-09-13T11:24:00.656686+0000","last_peered":"2024-09-13T11:24:00.656686+0000","last_clean":"2024-09-13T11:24:00.656686+0000","last_became_active":"2024-09-13T11:23:48.926014+0000","last_became_peered":"2024-09-13T11:23:48.926014+0000","last_unstale":"2024-09-13T11:24:00.656686+0000","last_undegraded":"2024-09-13T11:24:00.656686+0000","last_fullsized":"2024-09-13T11:24:00.656686+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:22:24.896055+0000","objects_scrubbed":0,"log_size":3900,"log_dups_size":0,"ondisk_log_size":3900,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-13T11:24:01.063698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":166002688,"num_objects":2533,"num_object_clones":0,"num_object_copies":5066,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2533,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2533,"num_write_kb":162112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1458,"num_bytes_recovered":95551488,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0],"acting":[10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.17","version":"102'3796","reported_seq":7826,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-13T11:24:00.298569+0000","last_change":"2024-09-13T11:23:49.349241+0000","last_active":"2024-09-13T11:24:00.298569+0000","last_peered":"2024-09-13T11:24:00.298569+0000","last_clean":"2024-09-13T11:24:00.298569+0000","last_became_active":"2024-09-13T11:23:42.935601+0000","last_became_peered":"2024-09-13T11:23:42.935601+0000","last_unstale":"2024-09-13T11:24:00.298569+0000","last_undegraded":"2024-09-13T11:24:00.298569+0000","last_fullsized":"2024-09-13T11:24:00.298569+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":5,"last_scrub":"92'3359","last_scrub_stamp":"2024-09-13T11:23:49.349167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:49.349167+0000","objects_scrubbed":2004,"log_size":3796,"log_dups_size":0,"ondisk_log_size":3796,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:24:59.401302+0000","scrub_duration":363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":160497664,"num_objects":2449,"num_object_clones":0,"num_object_copies":4898,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2449,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":445,"num_write_kb":28480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3],"acting":[10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"104'4063","reported_seq":8724,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-13T11:24:01.937217+0000","last_change":"2024-09-13T11:23:46.510544+0000","last_active":"2024-09-13T11:24:01.937217+0000","last_peered":"2024-09-13T11:24:01.937217+0000","last_clean":"2024-09-13T11:24:01.937217+0000","last_became_active":"2024-09-13T11:23:46.105237+0000","last_became_peered":"2024-09-13T11:23:46.105237+0000","last_unstale":"2024-09-13T11:24:01.937217+0000","last_undegraded":"2024-09-13T11:24:01.937217+0000","last_fullsized":"2024-09-13T11:24:01.937217+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":5,"last_scrub":"89'3478","last_scrub_stamp":"2024-09-13T11:23:46.510384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:46.510384+0000","objects_scrubbed":2054,"log_size":4063,"log_dups_size":0,"ondisk_log_size":4063,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:09.149878+0000","scrub_duration":363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":173146112,"num_objects":2642,"num_object_clones":0,"num_object_copies":5284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2642,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":588,"num_write_kb":37632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,15],"acting":[11,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.15","version":"102'3821","reported_seq":8240,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-13T11:24:00.357761+0000","last_change":"2024-09-13T11:23:48.332790+0000","last_active":"2024-09-13T11:24:00.357761+0000","last_peered":"2024-09-13T11:24:00.357761+0000","last_clean":"2024-09-13T11:24:00.357761+0000","last_became_active":"2024-09-13T11:23:42.931149+0000","last_became_peered":"2024-09-13T11:23:42.931149+0000","last_unstale":"2024-09-13T11:24:00.357761+0000","last_undegraded":"2024-09-13T11:24:00.357761+0000","last_fullsized":"2024-09-13T11:24:00.357761+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":5,"last_scrub":"91'3384","last_scrub_stamp":"2024-09-13T11:23:48.332600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:48.332600+0000","objects_scrubbed":2032,"log_size":3821,"log_dups_size":0,"ondisk_log_size":3821,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:24:58.239508+0000","scrub_duration":235,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":161873920,"num_objects":2470,"num_object_clones":0,"num_object_copies":4940,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2470,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":438,"num_write_kb":28032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,0],"acting":[14,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"3.14","version":"101'3838","reported_seq":8075,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-09-13T11:24:00.009923+0000","last_change":"2024-09-13T11:23:48.697100+0000","last_active":"2024-09-13T11:24:00.009923+0000","last_peered":"2024-09-13T11:24:00.009923+0000","last_clean":"2024-09-13T11:24:00.009923+0000","last_became_active":"2024-09-13T11:23:41.920989+0000","last_became_peered":"2024-09-13T11:23:41.920989+0000","last_unstale":"2024-09-13T11:24:00.009923+0000","last_undegraded":"2024-09-13T11:24:00.009923+0000","last_fullsized":"2024-09-13T11:24:00.009923+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":5,"last_scrub":"91'3371","last_scrub_stamp":"2024-09-13T11:23:48.696960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:48.696960+0000","objects_scrubbed":1935,"log_size":3838,"log_dups_size":0,"ondisk_log_size":3838,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:02.861530+0000","scrub_duration":735,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":158072832,"num_objects":2412,"num_object_clones":0,"num_object_copies":4824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2412,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":477,"num_write_kb":30528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,14],"acting":[3,14],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":167,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-13T11:24:00.248208+0000","last_change":"2024-09-13T11:22:16.889130+0000","last_active":"2024-09-13T11:24:00.248208+0000","last_peered":"2024-09-13T11:24:00.248208+0000","last_clean":"2024-09-13T11:24:00.248208+0000","last_became_active":"2024-09-13T11:22:14.192897+0000","last_became_peered":"2024-09-13T11:22:14.192897+0000","last_unstale":"2024-09-13T11:24:00.248208+0000","last_undegraded":"2024-09-13T11:24:00.248208+0000","last_fullsized":"2024-09-13T11:24:00.248208+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:22:13.113563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00084655899999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,10],"acting":[15,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"3.6","version":"104'4010","reported_seq":4199,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-13T11:24:02.034252+0000","last_change":"2024-09-13T11:23:22.656608+0000","last_active":"2024-09-13T11:24:02.034252+0000","last_peered":"2024-09-13T11:24:02.034252+0000","last_clean":"2024-09-13T11:24:02.034252+0000","last_became_active":"2024-09-13T11:23:19.939432+0000","last_became_peered":"2024-09-13T11:23:19.939432+0000","last_unstale":"2024-09-13T11:24:02.034252+0000","last_undegraded":"2024-09-13T11:24:02.034252+0000","last_fullsized":"2024-09-13T11:24:02.034252+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'2909","last_scrub_stamp":"2024-09-13T11:23:22.656323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:22.656323+0000","objects_scrubbed":1487,"log_size":4010,"log_dups_size":0,"ondisk_log_size":4010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:24:31.905185+0000","scrub_duration":323,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":169607168,"num_objects":2588,"num_object_clones":0,"num_object_copies":5176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2588,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1101,"num_write_kb":70464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,14],"acting":[13,14],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":178,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-13T11:24:01.930648+0000","last_change":"2024-09-13T11:23:21.875855+0000","last_active":"2024-09-13T11:24:01.930648+0000","last_peered":"2024-09-13T11:24:01.930648+0000","last_clean":"2024-09-13T11:24:01.930648+0000","last_became_active":"2024-09-13T11:22:14.211923+0000","last_became_peered":"2024-09-13T11:22:14.211923+0000","last_unstale":"2024-09-13T11:24:01.930648+0000","last_undegraded":"2024-09-13T11:24:01.930648+0000","last_fullsized":"2024-09-13T11:24:01.930648+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:23:21.875683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:23:21.875683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:24:40.678136+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0011689529999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0],"acting":[9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"104'3857","reported_seq":4047,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-13T11:24:02.041621+0000","last_change":"2024-09-13T11:23:20.443418+0000","last_active":"2024-09-13T11:24:02.041621+0000","last_peered":"2024-09-13T11:24:02.041621+0000","last_clean":"2024-09-13T11:24:02.041621+0000","last_became_active":"2024-09-13T11:23:19.927680+0000","last_became_peered":"2024-09-13T11:23:19.927680+0000","last_unstale":"2024-09-13T11:24:02.041621+0000","last_undegraded":"2024-09-13T11:24:02.041621+0000","last_fullsized":"2024-09-13T11:24:02.041621+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'2736","last_scrub_stamp":"2024-09-13T11:23:20.443329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:20.443329+0000","objects_scrubbed":1369,"log_size":3857,"log_dups_size":0,"ondisk_log_size":3857,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:24:48.147625+0000","scrub_duration":183,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":163381248,"num_objects":2493,"num_object_clones":0,"num_object_copies":4986,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2493,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1124,"num_write_kb":71936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,8],"acting":[13,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":175,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-13T11:24:00.241368+0000","last_change":"2024-09-13T11:23:59.530794+0000","last_active":"2024-09-13T11:24:00.241368+0000","last_peered":"2024-09-13T11:24:00.241368+0000","last_clean":"2024-09-13T11:24:00.241368+0000","last_became_active":"2024-09-13T11:22:14.581458+0000","last_became_peered":"2024-09-13T11:22:14.581458+0000","last_unstale":"2024-09-13T11:24:00.241368+0000","last_undegraded":"2024-09-13T11:24:00.241368+0000","last_fullsized":"2024-09-13T11:24:00.241368+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:23:59.530670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:23:59.530670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:02.655631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001026047,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5],"acting":[8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"102'3860","reported_seq":4026,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-13T11:24:00.240209+0000","last_change":"2024-09-13T11:23:20.254213+0000","last_active":"2024-09-13T11:24:00.240209+0000","last_peered":"2024-09-13T11:24:00.240209+0000","last_clean":"2024-09-13T11:24:00.240209+0000","last_became_active":"2024-09-13T11:23:19.925399+0000","last_became_peered":"2024-09-13T11:23:19.925399+0000","last_unstale":"2024-09-13T11:24:00.240209+0000","last_undegraded":"2024-09-13T11:24:00.240209+0000","last_fullsized":"2024-09-13T11:24:00.240209+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'2755","last_scrub_stamp":"2024-09-13T11:23:20.254124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:20.254124+0000","objects_scrubbed":1437,"log_size":3860,"log_dups_size":0,"ondisk_log_size":3860,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:24:49.993543+0000","scrub_duration":187,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":166920192,"num_objects":2547,"num_object_clones":0,"num_object_copies":5094,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2547,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1110,"num_write_kb":71040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,12],"acting":[1,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":155,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-13T11:24:00.240262+0000","last_change":"2024-09-13T11:22:47.720313+0000","last_active":"2024-09-13T11:24:00.240262+0000","last_peered":"2024-09-13T11:24:00.240262+0000","last_clean":"2024-09-13T11:24:00.240262+0000","last_became_active":"2024-09-13T11:22:47.719636+0000","last_became_peered":"2024-09-13T11:22:47.719636+0000","last_unstale":"2024-09-13T11:24:00.240262+0000","last_undegraded":"2024-09-13T11:24:00.240262+0000","last_fullsized":"2024-09-13T11:24:00.240262+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:22:13.113563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00087476000000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,9],"acting":[1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"102'3851","reported_seq":4037,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-13T11:24:00.407158+0000","last_change":"2024-09-13T11:23:20.686446+0000","last_active":"2024-09-13T11:24:00.407158+0000","last_peered":"2024-09-13T11:24:00.407158+0000","last_clean":"2024-09-13T11:24:00.407158+0000","last_became_active":"2024-09-13T11:23:19.939695+0000","last_became_peered":"2024-09-13T11:23:19.939695+0000","last_unstale":"2024-09-13T11:24:00.407158+0000","last_undegraded":"2024-09-13T11:24:00.407158+0000","last_fullsized":"2024-09-13T11:24:00.407158+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'2739","last_scrub_stamp":"2024-09-13T11:23:20.686358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:20.686358+0000","objects_scrubbed":1386,"log_size":3851,"log_dups_size":0,"ondisk_log_size":3851,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:24:38.891737+0000","scrub_duration":271,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":163774464,"num_objects":2499,"num_object_clones":0,"num_object_copies":4998,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2499,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1113,"num_write_kb":71232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,13],"acting":[15,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"3.3","version":"102'3470","reported_seq":3679,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-13T11:24:00.365009+0000","last_change":"2024-09-13T11:23:32.147115+0000","last_active":"2024-09-13T11:24:00.365009+0000","last_peered":"2024-09-13T11:24:00.365009+0000","last_clean":"2024-09-13T11:24:00.365009+0000","last_became_active":"2024-09-13T11:23:31.930448+0000","last_became_peered":"2024-09-13T11:23:31.930448+0000","last_unstale":"2024-09-13T11:24:00.365009+0000","last_undegraded":"2024-09-13T11:24:00.365009+0000","last_fullsized":"2024-09-13T11:24:00.365009+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'3122","last_scrub_stamp":"2024-09-13T11:23:32.147021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:32.147021+0000","objects_scrubbed":876,"log_size":3470,"log_dups_size":0,"ondisk_log_size":3470,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:24:48.175541+0000","scrub_duration":99,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":80216064,"num_objects":1224,"num_object_clones":0,"num_object_copies":2448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1224,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":348,"num_write_kb":22272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,7],"acting":[10,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"28'2","reported_seq":105,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-13T11:24:00.996197+0000","last_change":"2024-09-13T11:24:00.996197+0000","last_active":"2024-09-13T11:24:00.996197+0000","last_peered":"2024-09-13T11:24:00.996197+0000","last_clean":"2024-09-13T11:24:00.996197+0000","last_became_active":"2024-09-13T11:23:12.546348+0000","last_became_peered":"2024-09-13T11:23:12.546348+0000","last_unstale":"2024-09-13T11:24:00.996197+0000","last_undegraded":"2024-09-13T11:24:00.996197+0000","last_fullsized":"2024-09-13T11:24:00.996197+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2","last_scrub_stamp":"2024-09-13T11:24:00.996028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:24:00.996028+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:12.111097+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0025826730000000002,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8],"acting":[2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"101'3481","reported_seq":3688,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-13T11:24:00.246508+0000","last_change":"2024-09-13T11:23:32.749651+0000","last_active":"2024-09-13T11:24:00.246508+0000","last_peered":"2024-09-13T11:24:00.246508+0000","last_clean":"2024-09-13T11:24:00.246508+0000","last_became_active":"2024-09-13T11:23:31.921388+0000","last_became_peered":"2024-09-13T11:23:31.921388+0000","last_unstale":"2024-09-13T11:24:00.246508+0000","last_undegraded":"2024-09-13T11:24:00.246508+0000","last_fullsized":"2024-09-13T11:24:00.246508+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'3106","last_scrub_stamp":"2024-09-13T11:23:32.749487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:32.749487+0000","objects_scrubbed":856,"log_size":3481,"log_dups_size":0,"ondisk_log_size":3481,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:24:34.709501+0000","scrub_duration":91,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":80674816,"num_objects":1231,"num_object_clones":0,"num_object_copies":2462,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1231,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":375,"num_write_kb":24000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,15],"acting":[12,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":177,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-13T11:24:00.242697+0000","last_change":"2024-09-13T11:22:47.298206+0000","last_active":"2024-09-13T11:24:00.242697+0000","last_peered":"2024-09-13T11:24:00.242697+0000","last_clean":"2024-09-13T11:24:00.242697+0000","last_became_active":"2024-09-13T11:22:47.297319+0000","last_became_peered":"2024-09-13T11:22:47.297319+0000","last_unstale":"2024-09-13T11:24:00.242697+0000","last_undegraded":"2024-09-13T11:24:00.242697+0000","last_fullsized":"2024-09-13T11:24:00.242697+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:22:13.113563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00081225399999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,1],"acting":[14,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"3.1","version":"104'3482","reported_seq":3693,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-13T11:24:02.051997+0000","last_change":"2024-09-13T11:23:33.604966+0000","last_active":"2024-09-13T11:24:02.051997+0000","last_peered":"2024-09-13T11:24:02.051997+0000","last_clean":"2024-09-13T11:24:02.051997+0000","last_became_active":"2024-09-13T11:23:31.926528+0000","last_became_peered":"2024-09-13T11:23:31.926528+0000","last_unstale":"2024-09-13T11:24:02.051997+0000","last_undegraded":"2024-09-13T11:24:02.051997+0000","last_fullsized":"2024-09-13T11:24:02.051997+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'3070","last_scrub_stamp":"2024-09-13T11:23:33.604792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:33.604792+0000","objects_scrubbed":842,"log_size":3482,"log_dups_size":0,"ondisk_log_size":3482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:24:43.880038+0000","scrub_duration":119,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":82182144,"num_objects":1254,"num_object_clones":0,"num_object_copies":2508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1254,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":412,"num_write_kb":26368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,11],"acting":[13,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":165,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-09-13T11:23:59.241768+0000","last_change":"2024-09-13T11:22:16.267999+0000","last_active":"2024-09-13T11:23:59.241768+0000","last_peered":"2024-09-13T11:23:59.241768+0000","last_clean":"2024-09-13T11:23:59.241768+0000","last_became_active":"2024-09-13T11:22:14.211435+0000","last_became_peered":"2024-09-13T11:22:14.211435+0000","last_unstale":"2024-09-13T11:23:59.241768+0000","last_undegraded":"2024-09-13T11:23:59.241768+0000","last_fullsized":"2024-09-13T11:23:59.241768+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:22:13.113563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010010360000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2","version":"102'3473","reported_seq":3681,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-13T11:24:00.385197+0000","last_change":"2024-09-13T11:23:32.977666+0000","last_active":"2024-09-13T11:24:00.385197+0000","last_peered":"2024-09-13T11:24:00.385197+0000","last_clean":"2024-09-13T11:24:00.385197+0000","last_became_active":"2024-09-13T11:23:31.919828+0000","last_became_peered":"2024-09-13T11:23:31.919828+0000","last_unstale":"2024-09-13T11:24:00.385197+0000","last_undegraded":"2024-09-13T11:24:00.385197+0000","last_fullsized":"2024-09-13T11:24:00.385197+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'3057","last_scrub_stamp":"2024-09-13T11:23:32.977559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:32.977559+0000","objects_scrubbed":841,"log_size":3473,"log_dups_size":0,"ondisk_log_size":3473,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:24:52.169339+0000","scrub_duration":151,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":82378752,"num_objects":1257,"num_object_clones":0,"num_object_copies":2514,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1257,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":416,"num_write_kb":26624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,12],"acting":[14,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"2.3","version":"26'1","reported_seq":169,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-13T11:24:00.243013+0000","last_change":"2024-09-13T11:22:16.547691+0000","last_active":"2024-09-13T11:24:00.243013+0000","last_peered":"2024-09-13T11:24:00.243013+0000","last_clean":"2024-09-13T11:24:00.243013+0000","last_became_active":"2024-09-13T11:22:14.146507+0000","last_became_peered":"2024-09-13T11:22:14.146507+0000","last_unstale":"2024-09-13T11:24:00.243013+0000","last_undegraded":"2024-09-13T11:24:00.243013+0000","last_fullsized":"2024-09-13T11:24:00.243013+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:22:13.113563+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00085610999999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,15],"acting":[14,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"1.0","version":"26'112","reported_seq":309,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-13T11:24:00.239283+0000","last_change":"2024-09-13T11:23:49.325249+0000","last_active":"2024-09-13T11:24:00.239283+0000","last_peered":"2024-09-13T11:24:00.239283+0000","last_clean":"2024-09-13T11:24:00.239283+0000","last_became_active":"2024-09-13T11:22:12.129721+0000","last_became_peered":"2024-09-13T11:22:12.129721+0000","last_unstale":"2024-09-13T11:24:00.239283+0000","last_undegraded":"2024-09-13T11:24:00.239283+0000","last_fullsized":"2024-09-13T11:24:00.239283+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"26'112","last_scrub_stamp":"2024-09-13T11:23:49.325175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:11.098388+0000","last_clean_scrub_stamp":"2024-09-13T11:23:49.325175+0000","objects_scrubbed":2,"log_size":112,"log_dups_size":0,"ondisk_log_size":112,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:14.422287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.8","version":"102'3815","reported_seq":4002,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-13T11:24:00.333927+0000","last_change":"2024-09-13T11:23:20.644952+0000","last_active":"2024-09-13T11:24:00.333927+0000","last_peered":"2024-09-13T11:24:00.333927+0000","last_clean":"2024-09-13T11:24:00.333927+0000","last_became_active":"2024-09-13T11:23:19.925802+0000","last_became_peered":"2024-09-13T11:23:19.925802+0000","last_unstale":"2024-09-13T11:24:00.333927+0000","last_undegraded":"2024-09-13T11:24:00.333927+0000","last_fullsized":"2024-09-13T11:24:00.333927+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'2771","last_scrub_stamp":"2024-09-13T11:23:20.644747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:20.644747+0000","objects_scrubbed":1356,"log_size":3815,"log_dups_size":0,"ondisk_log_size":3815,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:24:38.144523+0000","scrub_duration":411,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":158072832,"num_objects":2412,"num_object_clones":0,"num_object_copies":4824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2412,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1056,"num_write_kb":67584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.9","version":"102'3899","reported_seq":4084,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-13T11:24:00.701258+0000","last_change":"2024-09-13T11:23:22.033678+0000","last_active":"2024-09-13T11:24:00.701258+0000","last_peered":"2024-09-13T11:24:00.701258+0000","last_clean":"2024-09-13T11:24:00.701258+0000","last_became_active":"2024-09-13T11:23:19.935114+0000","last_became_peered":"2024-09-13T11:23:19.935114+0000","last_unstale":"2024-09-13T11:24:00.701258+0000","last_undegraded":"2024-09-13T11:24:00.701258+0000","last_fullsized":"2024-09-13T11:24:00.701258+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'2808","last_scrub_stamp":"2024-09-13T11:23:22.033572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:22.033572+0000","objects_scrubbed":1471,"log_size":3899,"log_dups_size":0,"ondisk_log_size":3899,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:24:46.595859+0000","scrub_duration":287,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":167903232,"num_objects":2562,"num_object_clones":0,"num_object_copies":5124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2562,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1091,"num_write_kb":69824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,2],"acting":[12,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[]},{"pgid":"3.a","version":"101'4184","reported_seq":4372,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-09-13T11:24:00.050180+0000","last_change":"2024-09-13T11:23:32.526150+0000","last_active":"2024-09-13T11:24:00.050180+0000","last_peered":"2024-09-13T11:24:00.050180+0000","last_clean":"2024-09-13T11:24:00.050180+0000","last_became_active":"2024-09-13T11:23:31.934170+0000","last_became_peered":"2024-09-13T11:23:31.934170+0000","last_unstale":"2024-09-13T11:24:00.050180+0000","last_undegraded":"2024-09-13T11:24:00.050180+0000","last_fullsized":"2024-09-13T11:24:00.050180+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'3418","last_scrub_stamp":"2024-09-13T11:23:32.526057+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:32.526057+0000","objects_scrubbed":1774,"log_size":4184,"log_dups_size":0,"ondisk_log_size":4184,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:24:34.770161+0000","scrub_duration":507,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":166461440,"num_objects":2540,"num_object_clones":0,"num_object_copies":5080,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2540,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":766,"num_write_kb":49024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.b","version":"104'4264","reported_seq":4457,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-13T11:24:02.000298+0000","last_change":"2024-09-13T11:23:32.681343+0000","last_active":"2024-09-13T11:24:02.000298+0000","last_peered":"2024-09-13T11:24:02.000298+0000","last_clean":"2024-09-13T11:24:02.000298+0000","last_became_active":"2024-09-13T11:23:31.923415+0000","last_became_peered":"2024-09-13T11:23:31.923415+0000","last_unstale":"2024-09-13T11:24:02.000298+0000","last_undegraded":"2024-09-13T11:24:02.000298+0000","last_fullsized":"2024-09-13T11:24:02.000298+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'3471","last_scrub_stamp":"2024-09-13T11:23:32.681257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:32.681257+0000","objects_scrubbed":1720,"log_size":4264,"log_dups_size":0,"ondisk_log_size":4264,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:24:58.857812+0000","scrub_duration":199,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":164691968,"num_objects":2513,"num_object_clones":0,"num_object_copies":5026,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2513,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":793,"num_write_kb":50752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,1],"acting":[13,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"3.c","version":"104'4251","reported_seq":4443,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-13T11:24:02.035553+0000","last_change":"2024-09-13T11:23:32.524870+0000","last_active":"2024-09-13T11:24:02.035553+0000","last_peered":"2024-09-13T11:24:02.035553+0000","last_clean":"2024-09-13T11:24:02.035553+0000","last_became_active":"2024-09-13T11:23:31.922620+0000","last_became_peered":"2024-09-13T11:23:31.922620+0000","last_unstale":"2024-09-13T11:24:02.035553+0000","last_undegraded":"2024-09-13T11:24:02.035553+0000","last_fullsized":"2024-09-13T11:24:02.035553+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'3463","last_scrub_stamp":"2024-09-13T11:23:32.524776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:32.524776+0000","objects_scrubbed":1741,"log_size":4251,"log_dups_size":0,"ondisk_log_size":4251,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:24:37.745747+0000","scrub_duration":235,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":165740544,"num_objects":2529,"num_object_clones":0,"num_object_copies":5058,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2529,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":788,"num_write_kb":50432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,13],"acting":[11,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.d","version":"102'4172","reported_seq":4360,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-13T11:24:00.790903+0000","last_change":"2024-09-13T11:23:32.776835+0000","last_active":"2024-09-13T11:24:00.790903+0000","last_peered":"2024-09-13T11:24:00.790903+0000","last_clean":"2024-09-13T11:24:00.790903+0000","last_became_active":"2024-09-13T11:23:31.939246+0000","last_became_peered":"2024-09-13T11:23:31.939246+0000","last_unstale":"2024-09-13T11:24:00.790903+0000","last_undegraded":"2024-09-13T11:24:00.790903+0000","last_fullsized":"2024-09-13T11:24:00.790903+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'3395","last_scrub_stamp":"2024-09-13T11:23:32.776747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:32.776747+0000","objects_scrubbed":1704,"log_size":4172,"log_dups_size":0,"ondisk_log_size":4172,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:24:56.402862+0000","scrub_duration":179,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":162594816,"num_objects":2481,"num_object_clones":0,"num_object_copies":4962,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2481,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":777,"num_write_kb":49728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.e","version":"102'4304","reported_seq":9421,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-13T11:24:00.731742+0000","last_change":"2024-09-13T11:23:32.368856+0000","last_active":"2024-09-13T11:24:00.731742+0000","last_peered":"2024-09-13T11:24:00.731742+0000","last_clean":"2024-09-13T11:24:00.731742+0000","last_became_active":"2024-09-13T11:23:31.925869+0000","last_became_peered":"2024-09-13T11:23:31.925869+0000","last_unstale":"2024-09-13T11:24:00.731742+0000","last_undegraded":"2024-09-13T11:24:00.731742+0000","last_fullsized":"2024-09-13T11:24:00.731742+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'3527","last_scrub_stamp":"2024-09-13T11:23:32.368727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:32.368727+0000","objects_scrubbed":1774,"log_size":4304,"log_dups_size":0,"ondisk_log_size":4304,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:24:57.811779+0000","scrub_duration":219,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":167247872,"num_objects":2552,"num_object_clones":0,"num_object_copies":5104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2552,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":778,"num_write_kb":49792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"104'4198","reported_seq":4390,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-13T11:24:02.002768+0000","last_change":"2024-09-13T11:23:33.492630+0000","last_active":"2024-09-13T11:24:02.002768+0000","last_peered":"2024-09-13T11:24:02.002768+0000","last_clean":"2024-09-13T11:24:02.002768+0000","last_became_active":"2024-09-13T11:23:31.935239+0000","last_became_peered":"2024-09-13T11:23:31.935239+0000","last_unstale":"2024-09-13T11:24:02.002768+0000","last_undegraded":"2024-09-13T11:24:02.002768+0000","last_fullsized":"2024-09-13T11:24:02.002768+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'3420","last_scrub_stamp":"2024-09-13T11:23:33.492488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:33.492488+0000","objects_scrubbed":1776,"log_size":4198,"log_dups_size":0,"ondisk_log_size":4198,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:24:44.851261+0000","scrub_duration":215,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":167378944,"num_objects":2554,"num_object_clones":0,"num_object_copies":5108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2554,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":778,"num_write_kb":49792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,12],"acting":[11,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.10","version":"102'3954","reported_seq":8298,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-13T11:24:00.735862+0000","last_change":"2024-09-13T11:23:48.519654+0000","last_active":"2024-09-13T11:24:00.735862+0000","last_peered":"2024-09-13T11:24:00.735862+0000","last_clean":"2024-09-13T11:24:00.735862+0000","last_became_active":"2024-09-13T11:23:41.931183+0000","last_became_peered":"2024-09-13T11:23:41.931183+0000","last_unstale":"2024-09-13T11:24:00.735862+0000","last_undegraded":"2024-09-13T11:24:00.735862+0000","last_fullsized":"2024-09-13T11:24:00.735862+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":5,"last_scrub":"91'3468","last_scrub_stamp":"2024-09-13T11:23:48.519574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:48.519574+0000","objects_scrubbed":2031,"log_size":3954,"log_dups_size":0,"ondisk_log_size":3954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:24:57.342213+0000","scrub_duration":215,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":165085184,"num_objects":2519,"num_object_clones":0,"num_object_copies":5038,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2519,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":488,"num_write_kb":31232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.11","version":"102'3863","reported_seq":6985,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-13T11:24:00.234369+0000","last_change":"2024-09-13T11:23:46.102466+0000","last_active":"2024-09-13T11:24:00.234369+0000","last_peered":"2024-09-13T11:24:00.234369+0000","last_clean":"2024-09-13T11:24:00.234369+0000","last_became_active":"2024-09-13T11:23:39.063172+0000","last_became_peered":"2024-09-13T11:23:39.063172+0000","last_unstale":"2024-09-13T11:24:00.234369+0000","last_undegraded":"2024-09-13T11:24:00.234369+0000","last_fullsized":"2024-09-13T11:24:00.234369+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":5,"last_scrub":"88'3345","last_scrub_stamp":"2024-09-13T11:23:46.101973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:46.101973+0000","objects_scrubbed":2011,"log_size":3863,"log_dups_size":0,"ondisk_log_size":3863,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:08.671350+0000","scrub_duration":955,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":166002688,"num_objects":2533,"num_object_clones":0,"num_object_copies":5066,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2533,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":522,"num_write_kb":33408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11],"acting":[4,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.12","version":"102'3871","reported_seq":6917,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-13T11:24:00.530684+0000","last_change":"2024-09-13T11:23:35.111409+0000","last_active":"2024-09-13T11:24:00.530684+0000","last_peered":"2024-09-13T11:24:00.530684+0000","last_clean":"2024-09-13T11:24:00.530684+0000","last_became_active":"2024-09-13T11:23:21.998877+0000","last_became_peered":"2024-09-13T11:23:21.998877+0000","last_unstale":"2024-09-13T11:24:00.530684+0000","last_undegraded":"2024-09-13T11:24:00.530684+0000","last_fullsized":"2024-09-13T11:24:00.530684+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'3109","last_scrub_stamp":"2024-09-13T11:23:35.111320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:35.111320+0000","objects_scrubbed":1732,"log_size":3871,"log_dups_size":0,"ondisk_log_size":3871,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:03.371433+0000","scrub_duration":239,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":163446784,"num_objects":2494,"num_object_clones":0,"num_object_copies":4988,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2494,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":762,"num_write_kb":48768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,7],"acting":[14,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"3.13","version":"104'3963","reported_seq":7013,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-13T11:24:02.020450+0000","last_change":"2024-09-13T11:23:47.565094+0000","last_active":"2024-09-13T11:24:02.020450+0000","last_peered":"2024-09-13T11:24:02.020450+0000","last_clean":"2024-09-13T11:24:02.020450+0000","last_became_active":"2024-09-13T11:23:41.082433+0000","last_became_peered":"2024-09-13T11:23:41.082433+0000","last_unstale":"2024-09-13T11:24:02.020450+0000","last_undegraded":"2024-09-13T11:24:02.020450+0000","last_fullsized":"2024-09-13T11:24:02.020450+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'3408","last_scrub_stamp":"2024-09-13T11:23:47.564931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:47.564931+0000","objects_scrubbed":1981,"log_size":3963,"log_dups_size":0,"ondisk_log_size":3963,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:04.258231+0000","scrub_duration":471,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":166985728,"num_objects":2548,"num_object_clones":0,"num_object_copies":5096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":567,"num_write_kb":36288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7],"acting":[11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":36,"stat_sum":{"num_bytes":5264310272,"num_objects":80328,"num_object_clones":0,"num_object_copies":160656,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1468,"num_objects_unfound":0,"num_objects_dirty":80328,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":33799,"num_write_kb":2163072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10891,"num_bytes_recovered":713719808,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12726239232,"data_stored":12726239232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":140774,"ondisk_log_size":140774,"up":72,"acting":72,"num_store_stats":16},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":10},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":112,"ondisk_log_size":112,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":13,"up_from":23,"seq":98784247834,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1106512,"kb_used_data":1035072,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":93265328,"statfs":{"total":96636764160,"available":95503695872,"internally_reserved":0,"allocated":1059913728,"data_stored":1059453231,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73136581},"hb_peers":[0,1,3,4,6,8,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":9,"up_from":23,"seq":98784247833,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":427964,"kb_used_data":392620,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93943876,"statfs":{"total":96636764160,"available":96198529024,"internally_reserved":0,"allocated":402042880,"data_stored":401601626,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,8,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":40,"seq":171798691856,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1328072,"kb_used_data":1251576,"kb_used_omap":4,"kb_used_meta":76411,"kb_avail":93043768,"statfs":{"total":96636764160,"available":95276818432,"internally_reserved":0,"allocated":1281613824,"data_stored":1281147899,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4433,"internal_metadata":78245551},"hb_peers":[0,2,3,6,8,9,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":21,"seq":90194313241,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":981260,"kb_used_data":932028,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":93390580,"statfs":{"total":96636764160,"available":95631953920,"internally_reserved":0,"allocated":954396672,"data_stored":953931681,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":50395591},"hb_peers":[0,2,4,7,9,10,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":21,"seq":90194313244,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":284948,"kb_used_data":245444,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94086892,"statfs":{"total":96636764160,"available":96344977408,"internally_reserved":0,"allocated":251334656,"data_stored":250865626,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40434119},"hb_peers":[0,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":22,"seq":94489280536,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1129088,"kb_used_data":1049328,"kb_used_omap":1,"kb_used_meta":79742,"kb_avail":93242752,"statfs":{"total":96636764160,"available":95480578048,"internally_reserved":0,"allocated":1074511872,"data_stored":1074116405,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":81656261},"hb_peers":[1,3,4,5,8,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":22,"seq":94489280537,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":433732,"kb_used_data":398388,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93938108,"statfs":{"total":96636764160,"available":96192622592,"internally_reserved":0,"allocated":407949312,"data_stored":407500452,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,7,8,9,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":15,"up_from":22,"seq":94489280537,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919304,"kb_used_data":847864,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":93452536,"statfs":{"total":96636764160,"available":95695396864,"internally_reserved":0,"allocated":868212736,"data_stored":867816677,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":73136582},"hb_peers":[0,3,4,5,6,7,10,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,1],"upper_bound":1024},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":22,"seq":94489280537,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":496284,"kb_used_data":469260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93875556,"statfs":{"total":96636764160,"available":96128569344,"internally_reserved":0,"allocated":480522240,"data_stored":480050731,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[3,5,7,9,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":22,"seq":94489280537,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":421856,"kb_used_data":386512,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93949984,"statfs":{"total":96636764160,"available":96204783616,"internally_reserved":0,"allocated":395788288,"data_stored":395363799,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,3,5,7,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":22,"seq":94489280537,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1205024,"kb_used_data":1137744,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":93166816,"statfs":{"total":96636764160,"available":95402819584,"internally_reserved":0,"allocated":1165049856,"data_stored":1164630327,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":68876746},"hb_peers":[0,3,5,6,8,9,10,11,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":22,"seq":94489280537,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":505444,"kb_used_data":474260,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93866396,"statfs":{"total":96636764160,"available":96119189504,"internally_reserved":0,"allocated":485642240,"data_stored":485158850,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,5,7,9,10,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":22,"seq":94489280536,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":656540,"kb_used_data":603148,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":93715300,"statfs":{"total":96636764160,"available":95964467200,"internally_reserved":0,"allocated":617623552,"data_stored":617202890,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":54655424},"hb_peers":[0,1,3,6,7,9,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":11,"up_from":22,"seq":94489280538,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1348552,"kb_used_data":1272952,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":93023288,"statfs":{"total":96636764160,"available":95255846912,"internally_reserved":0,"allocated":1303502848,"data_stored":1303053612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":77396420},"hb_peers":[0,2,4,7,8,10,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":12,"up_from":22,"seq":94489280537,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1060804,"kb_used_data":985204,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":93311036,"statfs":{"total":96636764160,"available":95550500864,"internally_reserved":0,"allocated":1008848896,"data_stored":1008396947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,1,2,3,5,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":14,"up_from":22,"seq":94489280537,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1109572,"kb_used_data":1042292,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":93262268,"statfs":{"total":96636764160,"available":95500562432,"internally_reserved":0,"allocated":1067307008,"data_stored":1066853406,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,1,5,6,7,8,9,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1075773440,"data_stored":1075773440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1282670592,"data_stored":1282670592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":404881408,"data_stored":404881408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":958136320,"data_stored":958136320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":479461376,"data_stored":479461376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":250216448,"data_stored":250216448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":393216000,"data_stored":393216000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1162280960,"data_stored":1162280960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":480575488,"data_stored":480575488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":403898368,"data_stored":403898368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":618397696,"data_stored":618397696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":1253244928,"data_stored":1253244928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":1007812608,"data_stored":1007812608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":1027538944,"data_stored":1027538944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":1066336256,"data_stored":1066336256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":861798400,"data_stored":861798400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T11:24:05.863 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-13T11:24:05.863 INFO:tasks.ceph.osd.8.smithi001.stderr:2024-09-13T11:24:05.847+0000 7fb9b48ae640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26312) UID: 0 2024-09-13T11:24:05.948 INFO:tasks.ceph.osd.13.smithi092.stderr:2024-09-13T11:24:05.943+0000 7f3e4ba0f640 -1 osd.13 107 handle_osd_map injecting map crc failure 2024-09-13T11:24:05.956 INFO:tasks.ceph.osd.7.smithi177.stderr:2024-09-13T11:24:05.955+0000 7f897b51d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26575) UID: 0 2024-09-13T11:24:06.005 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-13T11:24:06.005 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [ 2024-09-13T11:24:06.005 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:06.005 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:84878 3.12 3:4f6f43a3:::benchmark_data_smithi177_31116_object84877:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108)", 2024-09-13T11:24:06.005 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:24:05.973541+0000", 2024-09-13T11:24:06.005 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 0.030843367999999999, 2024-09-13T11:24:06.006 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.030865172, 2024-09-13T11:24:06.006 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:24:06.006 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:24:06.006 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "waiting for sub ops", 2024-09-13T11:24:06.006 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:24:06.006 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:24:06.006 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:06.006 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 84878 2024-09-13T11:24:06.006 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:06.006 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:24:06.006 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:06.007 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:24:06.007 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.973541+0000", 2024-09-13T11:24:06.007 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:06.007 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:06.007 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:06.007 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:24:06.007 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.973541+0000", 2024-09-13T11:24:06.007 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:06.007 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:06.007 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:06.007 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:24:06.007 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.973543+0000", 2024-09-13T11:24:06.007 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.697e-06 2024-09-13T11:24:06.008 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:06.008 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:06.008 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:24:06.008 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.973628+0000", 2024-09-13T11:24:06.008 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.5350000000000001e-05 2024-09-13T11:24:06.008 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:06.008 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:06.008 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:24:06.009 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.973642+0000", 2024-09-13T11:24:06.009 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3934000000000001e-05 2024-09-13T11:24:06.009 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:06.009 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:06.009 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:24:06.009 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.973652+0000", 2024-09-13T11:24:06.009 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.7240000000000004e-06 2024-09-13T11:24:06.009 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:06.009 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:06.009 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:24:06.009 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.973713+0000", 2024-09-13T11:24:06.009 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.1068999999999994e-05 2024-09-13T11:24:06.009 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:06.010 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:06.010 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:24:06.010 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.973859+0000", 2024-09-13T11:24:06.010 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000146265 2024-09-13T11:24:06.010 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:06.010 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:06.010 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 7", 2024-09-13T11:24:06.010 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.974003+0000", 2024-09-13T11:24:06.010 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00014405700000000001 2024-09-13T11:24:06.010 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:06.010 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:06.010 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:24:06.010 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.975676+0000", 2024-09-13T11:24:06.011 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0016735809999999999 2024-09-13T11:24:06.011 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:06.011 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:24:06.011 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:06.011 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:06.011 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:06.011 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:84879 3.2 3:40162d9f:::benchmark_data_smithi177_31116_object84878:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108)", 2024-09-13T11:24:06.011 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:24:05.973920+0000", 2024-09-13T11:24:06.011 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 0.030464583999999999, 2024-09-13T11:24:06.011 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.030535545000000001, 2024-09-13T11:24:06.011 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:24:06.011 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:24:06.012 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "waiting for sub ops", 2024-09-13T11:24:06.012 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:24:06.012 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:24:06.012 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:06.012 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 84879 2024-09-13T11:24:06.012 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:06.012 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:24:06.012 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:06.012 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:24:06.012 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.973920+0000", 2024-09-13T11:24:06.012 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:06.012 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:06.013 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:06.013 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:24:06.013 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.973920+0000", 2024-09-13T11:24:06.013 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:06.013 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:06.013 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:06.013 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:24:06.013 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.973921+0000", 2024-09-13T11:24:06.013 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5179999999999999e-06 2024-09-13T11:24:06.013 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:06.014 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:06.014 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:24:06.014 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.974001+0000", 2024-09-13T11:24:06.014 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.9887000000000003e-05 2024-09-13T11:24:06.014 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:06.014 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:06.014 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:24:06.014 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.974010+0000", 2024-09-13T11:24:06.014 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.4329999999999993e-06 2024-09-13T11:24:06.014 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:06.014 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:06.014 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:24:06.014 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.974039+0000", 2024-09-13T11:24:06.015 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.9037e-05 2024-09-13T11:24:06.015 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:06.015 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:06.015 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:24:06.015 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.974104+0000", 2024-09-13T11:24:06.015 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.4349000000000003e-05 2024-09-13T11:24:06.015 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:06.015 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:06.015 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:24:06.015 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.974190+0000", 2024-09-13T11:24:06.015 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.6073000000000001e-05 2024-09-13T11:24:06.015 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:06.015 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:06.015 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 12", 2024-09-13T11:24:06.016 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.974266+0000", 2024-09-13T11:24:06.016 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.6205000000000006e-05 2024-09-13T11:24:06.016 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:06.016 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:06.016 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:24:06.016 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.975441+0000", 2024-09-13T11:24:06.016 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001174418 2024-09-13T11:24:06.017 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:06.017 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:24:06.017 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:06.017 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:06.017 INFO:teuthology.orchestra.run.smithi162.stdout: ], 2024-09-13T11:24:06.017 INFO:teuthology.orchestra.run.smithi162.stdout: "num_ops": 2 2024-09-13T11:24:06.017 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-13T11:24:06.018 INFO:tasks.radosbench.radosbench.0.smithi177.stdout: 99 16 84921 84905 53.5962 101.25 0.00411241 0.0186473 2024-09-13T11:24:06.022 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_blocked_ops 2024-09-13T11:24:06.056 INFO:tasks.ceph.osd.7.smithi177.stderr:2024-09-13T11:24:06.055+0000 7f897b51d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26575) UID: 0 2024-09-13T11:24:06.083 INFO:teuthology.orchestra.run.smithi001.stdout:osd.0: {} 2024-09-13T11:24:06.084 INFO:teuthology.orchestra.run.smithi001.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T11:24:06.111 INFO:teuthology.orchestra.run.smithi001.stdout:osd.1: {} 2024-09-13T11:24:06.112 INFO:teuthology.orchestra.run.smithi001.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T11:24:06.136 INFO:teuthology.orchestra.run.smithi001.stdout:osd.2: {} 2024-09-13T11:24:06.136 INFO:teuthology.orchestra.run.smithi001.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T11:24:06.144 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-13T11:24:06.144 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [], 2024-09-13T11:24:06.144 INFO:teuthology.orchestra.run.smithi162.stdout: "complaint_time": 30, 2024-09-13T11:24:06.144 INFO:teuthology.orchestra.run.smithi162.stdout: "num_blocked_ops": 0 2024-09-13T11:24:06.144 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-13T11:24:06.158 INFO:tasks.ceph.osd.11.smithi177.stderr:2024-09-13T11:24:06.155+0000 7f24792c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26580) UID: 0 2024-09-13T11:24:06.159 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_historic_ops 2024-09-13T11:24:06.160 INFO:teuthology.orchestra.run.smithi001.stdout:osd.3: {} 2024-09-13T11:24:06.161 INFO:teuthology.orchestra.run.smithi001.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T11:24:06.182 INFO:teuthology.orchestra.run.smithi001.stdout:osd.4: {} 2024-09-13T11:24:06.183 INFO:teuthology.orchestra.run.smithi001.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T11:24:06.208 INFO:teuthology.orchestra.run.smithi001.stdout:osd.5: {} 2024-09-13T11:24:06.208 INFO:teuthology.orchestra.run.smithi001.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T11:24:06.230 INFO:teuthology.orchestra.run.smithi001.stdout:osd.6: {} 2024-09-13T11:24:06.231 INFO:teuthology.orchestra.run.smithi001.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T11:24:06.253 INFO:teuthology.orchestra.run.smithi001.stdout:osd.7: {} 2024-09-13T11:24:06.254 INFO:teuthology.orchestra.run.smithi001.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T11:24:06.258 INFO:tasks.ceph.osd.7.smithi177.stderr:2024-09-13T11:24:06.255+0000 7f897b51d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26575) UID: 0 2024-09-13T11:24:06.276 INFO:teuthology.orchestra.run.smithi001.stdout:osd.8: {} 2024-09-13T11:24:06.276 INFO:teuthology.orchestra.run.smithi001.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-13T11:24:06.284 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-13T11:24:06.284 INFO:teuthology.orchestra.run.smithi162.stdout: "size": 20, 2024-09-13T11:24:06.284 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 600, 2024-09-13T11:24:06.284 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [ 2024-09-13T11:24:06.285 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:06.285 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:48911 3.12 3:4a364a6a:::benchmark_data_smithi177_31116_object48910:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e71)", 2024-09-13T11:24:06.285 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:25.457776+0000", 2024-09-13T11:24:06.285 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 40.822817385999997, 2024-09-13T11:24:06.285 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.22561999599999999, 2024-09-13T11:24:06.285 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:24:06.285 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:24:06.285 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:06.285 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:24:06.285 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:24:06.286 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:06.286 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 48911 2024-09-13T11:24:06.286 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:06.286 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:24:06.286 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:06.286 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:24:06.286 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.457776+0000", 2024-09-13T11:24:06.286 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:06.286 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:06.286 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:06.286 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:24:06.286 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.457776+0000", 2024-09-13T11:24:06.286 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:06.286 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:06.287 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:06.287 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:24:06.287 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.457788+0000", 2024-09-13T11:24:06.287 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2084e-05 2024-09-13T11:24:06.287 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:06.287 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:06.287 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:24:06.287 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.457823+0000", 2024-09-13T11:24:06.287 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.5002000000000001e-05 2024-09-13T11:24:06.287 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:06.287 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:06.287 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:24:06.287 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.457836+0000", 2024-09-13T11:24:06.287 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2988e-05 2024-09-13T11:24:06.288 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:06.288 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:06.288 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:24:06.288 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.457843+0000", 2024-09-13T11:24:06.288 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.5320000000000003e-06 2024-09-13T11:24:06.288 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:06.288 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:06.288 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:24:06.288 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.457895+0000", 2024-09-13T11:24:06.288 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.1690000000000001e-05 2024-09-13T11:24:06.288 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:06.288 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:06.289 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:24:06.289 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.458001+0000", 2024-09-13T11:24:06.289 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000106024 2024-09-13T11:24:06.289 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:06.289 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:06.289 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 7", 2024-09-13T11:24:06.289 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.458101+0000", 2024-09-13T11:24:06.289 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00010060699999999999 2024-09-13T11:24:06.289 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:06.289 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:06.290 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:24:06.290 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.459250+0000", 2024-09-13T11:24:06.290 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0011485670000000001 2024-09-13T11:24:06.290 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:06.290 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:06.290 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:06.290 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.682166+0000", 2024-09-13T11:24:06.290 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.22291609500000001 2024-09-13T11:24:06.290 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:06.290 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:06.290 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:24:06.290 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.682271+0000", 2024-09-13T11:24:06.290 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000105179 2024-09-13T11:24:06.291 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:06.291 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:06.291 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:24:06.291 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.683396+0000", 2024-09-13T11:24:06.291 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001125228 2024-09-13T11:24:06.291 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:06.291 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:24:06.291 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:06.291 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:06.291 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:06.291 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:48923 3.12 3:4ae1afd7:::benchmark_data_smithi177_31116_object48922:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e71)", 2024-09-13T11:24:06.291 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:25.465697+0000", 2024-09-13T11:24:06.292 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 40.814896220999998, 2024-09-13T11:24:06.292 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.21931837100000001, 2024-09-13T11:24:06.292 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:24:06.292 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:24:06.292 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:06.292 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:24:06.292 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:24:06.292 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:06.292 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 48923 2024-09-13T11:24:06.292 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:06.292 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:24:06.292 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:06.293 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:24:06.293 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.465697+0000", 2024-09-13T11:24:06.293 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:06.293 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:06.293 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:06.293 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:24:06.293 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.465697+0000", 2024-09-13T11:24:06.293 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:06.293 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:06.293 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:06.293 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:24:06.294 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.465700+0000", 2024-09-13T11:24:06.294 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.041e-06 2024-09-13T11:24:06.294 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:06.294 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:06.294 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:24:06.294 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.465757+0000", 2024-09-13T11:24:06.294 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.7664999999999998e-05 2024-09-13T11:24:06.294 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:06.294 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:06.294 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:24:06.294 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.465769+0000", 2024-09-13T11:24:06.294 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.219e-05 2024-09-13T11:24:06.294 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:06.294 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:06.295 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:24:06.295 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.465775+0000", 2024-09-13T11:24:06.295 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.6779999999999997e-06 2024-09-13T11:24:06.295 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:06.295 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:06.295 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:24:06.295 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.465821+0000", 2024-09-13T11:24:06.295 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.5577000000000001e-05 2024-09-13T11:24:06.295 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:06.295 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:06.295 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:24:06.295 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.465928+0000", 2024-09-13T11:24:06.295 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000107502 2024-09-13T11:24:06.295 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:06.296 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:06.296 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 7", 2024-09-13T11:24:06.296 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.466022+0000", 2024-09-13T11:24:06.296 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.3802999999999996e-05 2024-09-13T11:24:06.296 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:06.296 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:06.296 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:24:06.296 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.466901+0000", 2024-09-13T11:24:06.296 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00087950699999999997 2024-09-13T11:24:06.296 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:06.296 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:06.296 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:06.296 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.683622+0000", 2024-09-13T11:24:06.297 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.216720462 2024-09-13T11:24:06.297 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:06.297 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:06.297 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:24:06.297 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.683771+0000", 2024-09-13T11:24:06.297 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00014900199999999999 2024-09-13T11:24:06.297 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:06.297 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:06.297 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:24:06.297 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.685016+0000", 2024-09-13T11:24:06.297 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001244944 2024-09-13T11:24:06.297 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:06.297 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:24:06.297 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:06.298 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:06.298 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:06.298 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:48955 3.12 3:4ef2991b:::benchmark_data_smithi177_31116_object48954:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e71)", 2024-09-13T11:24:06.298 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:25.494091+0000", 2024-09-13T11:24:06.298 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 40.786502583999997, 2024-09-13T11:24:06.298 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.19235933599999999, 2024-09-13T11:24:06.298 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:24:06.298 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:24:06.298 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:06.298 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:24:06.298 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:24:06.298 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:06.298 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 48955 2024-09-13T11:24:06.298 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:06.299 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:24:06.299 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:06.299 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:24:06.299 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.494091+0000", 2024-09-13T11:24:06.299 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:06.299 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:06.299 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:06.300 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:24:06.300 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.494091+0000", 2024-09-13T11:24:06.300 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:06.300 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:06.300 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:06.300 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:24:06.301 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.494093+0000", 2024-09-13T11:24:06.301 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.0770000000000002e-06 2024-09-13T11:24:06.301 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:06.301 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:06.301 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:24:06.301 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.494160+0000", 2024-09-13T11:24:06.301 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.6798999999999995e-05 2024-09-13T11:24:06.301 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:06.302 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:06.302 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:24:06.302 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.494174+0000", 2024-09-13T11:24:06.302 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4489e-05 2024-09-13T11:24:06.302 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:06.302 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:06.302 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:24:06.302 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.494183+0000", 2024-09-13T11:24:06.302 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.4460000000000001e-06 2024-09-13T11:24:06.302 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:06.302 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:06.302 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:24:06.302 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.494239+0000", 2024-09-13T11:24:06.302 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.6140999999999997e-05 2024-09-13T11:24:06.303 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:06.303 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:06.303 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:24:06.303 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.494385+0000", 2024-09-13T11:24:06.303 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00014546399999999999 2024-09-13T11:24:06.303 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:06.303 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:06.303 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 7", 2024-09-13T11:24:06.303 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.494506+0000", 2024-09-13T11:24:06.303 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000121328 2024-09-13T11:24:06.303 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:06.303 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:06.303 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:24:06.304 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.495528+0000", 2024-09-13T11:24:06.304 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0010226079999999999 2024-09-13T11:24:06.304 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:06.304 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:06.304 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:06.304 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.685177+0000", 2024-09-13T11:24:06.304 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.18964847500000001 2024-09-13T11:24:06.305 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:06.305 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:06.305 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:24:06.305 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.685337+0000", 2024-09-13T11:24:06.305 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00016055400000000001 2024-09-13T11:24:06.305 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:06.305 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:06.305 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:24:06.305 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.686450+0000", 2024-09-13T11:24:06.305 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0011129550000000001 2024-09-13T11:24:06.305 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:06.306 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:24:06.306 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:06.306 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:06.306 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:06.306 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:48967 3.12 3:4abf2346:::benchmark_data_smithi177_31116_object48966:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e71)", 2024-09-13T11:24:06.306 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:25.509687+0000", 2024-09-13T11:24:06.306 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 40.770906459999999, 2024-09-13T11:24:06.306 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.17798804900000001, 2024-09-13T11:24:06.306 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:24:06.306 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:24:06.306 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:06.306 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:24:06.306 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:24:06.306 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:06.307 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 48967 2024-09-13T11:24:06.307 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:06.307 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:24:06.307 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:06.307 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:24:06.307 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.509687+0000", 2024-09-13T11:24:06.307 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:06.307 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:06.307 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:06.307 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:24:06.307 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.509687+0000", 2024-09-13T11:24:06.307 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:06.307 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:06.307 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:06.308 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:24:06.308 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.509689+0000", 2024-09-13T11:24:06.308 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.9719999999999999e-06 2024-09-13T11:24:06.308 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:06.308 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:06.308 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:24:06.308 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.509754+0000", 2024-09-13T11:24:06.308 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.5031999999999997e-05 2024-09-13T11:24:06.308 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:06.308 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:06.308 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:24:06.308 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.509787+0000", 2024-09-13T11:24:06.308 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.2891000000000001e-05 2024-09-13T11:24:06.309 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:06.309 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:06.309 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:24:06.309 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.509793+0000", 2024-09-13T11:24:06.309 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.1500000000000004e-06 2024-09-13T11:24:06.309 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:06.309 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:06.309 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:24:06.309 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.509846+0000", 2024-09-13T11:24:06.309 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.2926000000000001e-05 2024-09-13T11:24:06.309 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:06.309 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:06.309 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:24:06.309 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.509957+0000", 2024-09-13T11:24:06.310 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000110734 2024-09-13T11:24:06.310 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:06.310 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:06.310 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 7", 2024-09-13T11:24:06.310 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.510083+0000", 2024-09-13T11:24:06.310 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00012628300000000001 2024-09-13T11:24:06.310 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:06.310 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:06.310 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:24:06.310 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.511047+0000", 2024-09-13T11:24:06.310 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00096376800000000005 2024-09-13T11:24:06.310 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:06.310 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:06.311 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:06.311 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.686579+0000", 2024-09-13T11:24:06.311 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.17553200999999999 2024-09-13T11:24:06.311 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:06.311 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:06.311 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:24:06.311 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.686846+0000", 2024-09-13T11:24:06.311 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000267039 2024-09-13T11:24:06.311 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:06.311 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:06.311 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:24:06.311 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.687675+0000", 2024-09-13T11:24:06.311 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00082924400000000003 2024-09-13T11:24:06.311 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:06.311 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:24:06.312 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:06.312 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:06.312 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:06.312 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:48987 3.12 3:4e1e6d0c:::benchmark_data_smithi177_31116_object48986:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e71)", 2024-09-13T11:24:06.312 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:25.525640+0000", 2024-09-13T11:24:06.312 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 40.754953815999997, 2024-09-13T11:24:06.312 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.163044945, 2024-09-13T11:24:06.312 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:24:06.312 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:24:06.312 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:06.312 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:24:06.312 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:24:06.312 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:06.313 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 48987 2024-09-13T11:24:06.313 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:06.313 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:24:06.313 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:06.313 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:24:06.313 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.525640+0000", 2024-09-13T11:24:06.313 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:06.313 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:06.313 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:06.313 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:24:06.313 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.525640+0000", 2024-09-13T11:24:06.313 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:06.313 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:06.313 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:06.314 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:24:06.314 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.525651+0000", 2024-09-13T11:24:06.314 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1092e-05 2024-09-13T11:24:06.314 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:06.314 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:06.314 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:24:06.314 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.525795+0000", 2024-09-13T11:24:06.314 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00014417400000000001 2024-09-13T11:24:06.314 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:06.314 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:06.314 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:24:06.315 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.525834+0000", 2024-09-13T11:24:06.315 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.9026000000000002e-05 2024-09-13T11:24:06.315 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:06.315 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:06.315 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:24:06.315 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.525857+0000", 2024-09-13T11:24:06.315 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.2904999999999998e-05 2024-09-13T11:24:06.315 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:06.315 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:06.315 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:24:06.315 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.525967+0000", 2024-09-13T11:24:06.316 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000110188 2024-09-13T11:24:06.316 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:06.316 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:06.316 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:24:06.316 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.526214+0000", 2024-09-13T11:24:06.316 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00024667800000000001 2024-09-13T11:24:06.316 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:06.316 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:06.316 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 7", 2024-09-13T11:24:06.316 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.526499+0000", 2024-09-13T11:24:06.316 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00028463799999999998 2024-09-13T11:24:06.317 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:06.317 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:06.317 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:24:06.317 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.528226+0000", 2024-09-13T11:24:06.317 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0017276679999999999 2024-09-13T11:24:06.317 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:06.317 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:06.317 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:06.317 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.687803+0000", 2024-09-13T11:24:06.317 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.15957658899999999 2024-09-13T11:24:06.317 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:06.317 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:06.317 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:24:06.318 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.687986+0000", 2024-09-13T11:24:06.318 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00018332400000000001 2024-09-13T11:24:06.318 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:06.318 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:06.318 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:24:06.318 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.688685+0000", 2024-09-13T11:24:06.318 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00069866299999999998 2024-09-13T11:24:06.319 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:06.319 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:24:06.319 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:06.319 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:06.319 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:06.319 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:54637 3.22 3:44450106:::benchmark_data_smithi177_31116_object54636:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e75)", 2024-09-13T11:24:06.320 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:30.978960+0000", 2024-09-13T11:24:06.320 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 35.301633383999999, 2024-09-13T11:24:06.320 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.960188928, 2024-09-13T11:24:06.320 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:24:06.320 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:24:06.320 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-09-13T11:24:06.321 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:24:06.321 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:24:06.321 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:06.321 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 54637 2024-09-13T11:24:06.321 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:06.321 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:24:06.322 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:06.322 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:24:06.322 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.978960+0000", 2024-09-13T11:24:06.322 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:06.322 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:06.322 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:06.322 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:24:06.322 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.978960+0000", 2024-09-13T11:24:06.322 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:06.322 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:06.322 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:06.323 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:24:06.323 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.978962+0000", 2024-09-13T11:24:06.323 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.46e-06 2024-09-13T11:24:06.323 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:06.323 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:06.323 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:24:06.324 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.979024+0000", 2024-09-13T11:24:06.324 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.1779999999999995e-05 2024-09-13T11:24:06.324 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:06.324 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:06.324 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:24:06.324 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.979037+0000", 2024-09-13T11:24:06.324 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2948e-05 2024-09-13T11:24:06.325 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:06.325 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:06.325 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:24:06.325 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.979048+0000", 2024-09-13T11:24:06.325 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1241e-05 2024-09-13T11:24:06.325 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:06.325 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:06.326 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:24:06.326 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:31.032951+0000", 2024-09-13T11:24:06.326 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.053903286000000002 2024-09-13T11:24:06.326 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:06.326 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:06.326 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-09-13T11:24:06.326 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:31.032958+0000", 2024-09-13T11:24:06.326 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.8109999999999997e-06 2024-09-13T11:24:06.326 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:06.327 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:06.327 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:24:06.327 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:31.933462+0000", 2024-09-13T11:24:06.327 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.90050463999999997 2024-09-13T11:24:06.327 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:06.327 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:06.327 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for degraded object", 2024-09-13T11:24:06.327 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:31.934668+0000", 2024-09-13T11:24:06.327 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0012053249999999999 2024-09-13T11:24:06.327 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:06.327 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:06.327 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:24:06.327 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:31.938887+0000", 2024-09-13T11:24:06.327 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0042187120000000003 2024-09-13T11:24:06.328 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:06.328 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:06.328 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:24:06.328 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:31.939149+0000", 2024-09-13T11:24:06.328 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000262725 2024-09-13T11:24:06.328 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:06.328 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:24:06.328 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:06.328 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:06.328 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:06.328 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPushReply(3.12 75/69 [PushReplyOp(3:4db63d39:::benchmark_data_smithi177_31116_object32641:head)])", 2024-09-13T11:24:06.328 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:31.646428+0000", 2024-09-13T11:24:06.328 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 34.634165260000003, 2024-09-13T11:24:06.329 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.28409169299999998, 2024-09-13T11:24:06.329 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:24:06.329 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:24:06.329 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-09-13T11:24:06.329 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:24:06.329 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:06.330 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:24:06.330 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:31.646428+0000", 2024-09-13T11:24:06.330 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:06.330 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:06.330 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:06.330 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:24:06.330 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:31.646428+0000", 2024-09-13T11:24:06.330 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:06.330 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:06.330 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:06.330 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:24:06.330 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:31.646431+0000", 2024-09-13T11:24:06.330 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.3149999999999999e-06 2024-09-13T11:24:06.330 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:06.331 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:06.331 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:24:06.331 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:31.646438+0000", 2024-09-13T11:24:06.331 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.7379999999999994e-06 2024-09-13T11:24:06.331 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:06.331 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:06.331 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:24:06.331 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:31.928787+0000", 2024-09-13T11:24:06.331 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.28234828699999998 2024-09-13T11:24:06.331 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:06.331 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:06.331 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:24:06.331 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:31.929066+0000", 2024-09-13T11:24:06.332 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00027956500000000001 2024-09-13T11:24:06.332 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:06.332 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:06.332 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:24:06.332 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:31.929216+0000", 2024-09-13T11:24:06.332 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00014963500000000001 2024-09-13T11:24:06.332 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:06.332 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:06.332 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:24:06.332 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:31.930520+0000", 2024-09-13T11:24:06.332 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001304153 2024-09-13T11:24:06.333 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:06.333 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:24:06.333 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:06.333 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:06.333 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:06.334 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPushReply(3.12 76/69 [PushReplyOp(3:4a5f7a67:::benchmark_data_smithi177_31116_object32993:head)])", 2024-09-13T11:24:06.334 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:32.069056+0000", 2024-09-13T11:24:06.334 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 34.211537978999999, 2024-09-13T11:24:06.334 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.17843624299999999, 2024-09-13T11:24:06.334 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:24:06.334 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:24:06.334 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-09-13T11:24:06.334 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:24:06.334 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:06.334 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:24:06.334 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.069056+0000", 2024-09-13T11:24:06.334 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:06.334 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:06.334 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:06.335 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:24:06.335 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.069056+0000", 2024-09-13T11:24:06.335 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:06.335 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:06.335 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:06.335 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:24:06.335 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.069057+0000", 2024-09-13T11:24:06.335 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4699999999999999e-06 2024-09-13T11:24:06.335 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:06.335 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:06.335 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:24:06.335 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.069065+0000", 2024-09-13T11:24:06.335 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.9740000000000002e-06 2024-09-13T11:24:06.336 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:06.336 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:06.336 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:24:06.336 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.246435+0000", 2024-09-13T11:24:06.336 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.17736979 2024-09-13T11:24:06.336 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:06.336 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:06.336 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:24:06.337 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.246450+0000", 2024-09-13T11:24:06.337 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4582e-05 2024-09-13T11:24:06.337 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:06.337 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:06.337 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:24:06.337 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.246556+0000", 2024-09-13T11:24:06.337 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00010608200000000001 2024-09-13T11:24:06.337 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:06.337 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:06.337 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:24:06.337 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.247492+0000", 2024-09-13T11:24:06.337 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00093634499999999999 2024-09-13T11:24:06.337 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:06.337 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:24:06.338 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:06.338 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:06.338 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:06.338 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPush(3.15 76/70 [PushOp(3:ab4c41a1:::benchmark_data_smithi177_31116_object23392:head, version: 55'1416, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:ab4c41a1:::benchmark_data_smithi177_31116_object23392:head@55'1416, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:24:06.338 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:32.723597+0000", 2024-09-13T11:24:06.338 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 33.556996669, 2024-09-13T11:24:06.338 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.205507309, 2024-09-13T11:24:06.338 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:24:06.338 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:24:06.338 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:24:06.338 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:24:06.338 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:06.338 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:24:06.339 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.723597+0000", 2024-09-13T11:24:06.339 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:06.339 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:06.339 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:06.339 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:24:06.339 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.723597+0000", 2024-09-13T11:24:06.339 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:06.339 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:06.339 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:06.339 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:24:06.339 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.723599+0000", 2024-09-13T11:24:06.339 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.1639999999999999e-06 2024-09-13T11:24:06.339 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:06.339 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:06.340 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:24:06.340 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.723661+0000", 2024-09-13T11:24:06.340 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.1851000000000001e-05 2024-09-13T11:24:06.340 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:06.340 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:06.340 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:24:06.340 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.927067+0000", 2024-09-13T11:24:06.340 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.20340633899999999 2024-09-13T11:24:06.340 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:06.340 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:06.340 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:24:06.340 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.927102+0000", 2024-09-13T11:24:06.340 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.4178999999999999e-05 2024-09-13T11:24:06.341 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:06.341 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:06.341 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:24:06.341 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.927424+0000", 2024-09-13T11:24:06.341 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000322536 2024-09-13T11:24:06.341 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:06.341 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:06.341 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:24:06.341 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.927457+0000", 2024-09-13T11:24:06.341 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.3191000000000001e-05 2024-09-13T11:24:06.341 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:06.341 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:06.341 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:24:06.341 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.929104+0000", 2024-09-13T11:24:06.342 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001647049 2024-09-13T11:24:06.342 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:06.342 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:24:06.342 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:06.342 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:06.342 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:06.342 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop_reply(client.4665.0:55795 3.12 e79/69 ondisk, result = 0)", 2024-09-13T11:24:06.344 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:35.893777+0000", 2024-09-13T11:24:06.344 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 30.386817036, 2024-09-13T11:24:06.344 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.189196325, 2024-09-13T11:24:06.344 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:24:06.344 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:24:06.345 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:24:06.345 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:24:06.345 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:06.345 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:24:06.345 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:35.893777+0000", 2024-09-13T11:24:06.345 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:06.345 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:06.345 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:06.345 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:24:06.345 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:35.893777+0000", 2024-09-13T11:24:06.345 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:06.345 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:06.345 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:06.345 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:24:06.346 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:35.893778+0000", 2024-09-13T11:24:06.346 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.6670000000000001e-06 2024-09-13T11:24:06.346 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:06.346 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:06.346 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:24:06.346 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:35.893784+0000", 2024-09-13T11:24:06.346 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.1989999999999997e-06 2024-09-13T11:24:06.346 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:06.346 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:06.346 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:24:06.346 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:36.082661+0000", 2024-09-13T11:24:06.346 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.18887743600000001 2024-09-13T11:24:06.346 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:06.347 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:06.347 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:24:06.347 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:36.082666+0000", 2024-09-13T11:24:06.347 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.2270000000000001e-06 2024-09-13T11:24:06.347 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:06.347 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:06.347 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:24:06.347 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:36.082784+0000", 2024-09-13T11:24:06.347 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00011776400000000001 2024-09-13T11:24:06.347 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:06.347 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:06.347 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:24:06.347 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:36.082799+0000", 2024-09-13T11:24:06.347 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5058999999999999e-05 2024-09-13T11:24:06.348 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:06.348 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:06.348 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:24:06.348 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:36.082973+0000", 2024-09-13T11:24:06.348 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00017397300000000001 2024-09-13T11:24:06.348 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:06.348 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:24:06.348 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:06.348 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:06.348 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:06.348 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPush(3.1f 84/78 [PushOp(3:fd5621e0:::benchmark_data_smithi177_31116_object7011:head, version: 46'483, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:fd5621e0:::benchmark_data_smithi177_31116_object7011:head@46'483, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:24:06.348 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:41.510296+0000", 2024-09-13T11:24:06.348 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 24.770297552999999, 2024-09-13T11:24:06.349 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.41419089999999997, 2024-09-13T11:24:06.349 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:24:06.349 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:24:06.349 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:24:06.349 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:24:06.349 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:06.349 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:24:06.349 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:41.510296+0000", 2024-09-13T11:24:06.349 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:06.349 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:06.349 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:06.349 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:24:06.349 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:41.510296+0000", 2024-09-13T11:24:06.350 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:06.350 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:06.350 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:06.350 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:24:06.350 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:41.510305+0000", 2024-09-13T11:24:06.350 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.9700000000000005e-06 2024-09-13T11:24:06.350 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:06.350 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:06.350 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:24:06.350 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:41.510458+0000", 2024-09-13T11:24:06.350 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000152418 2024-09-13T11:24:06.350 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:06.350 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:06.351 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:24:06.351 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:41.918225+0000", 2024-09-13T11:24:06.351 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.40776698 2024-09-13T11:24:06.351 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:06.351 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:06.351 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:24:06.351 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:41.918251+0000", 2024-09-13T11:24:06.351 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.6531000000000001e-05 2024-09-13T11:24:06.351 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:06.351 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:06.352 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:24:06.352 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:41.922211+0000", 2024-09-13T11:24:06.352 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0039603040000000004 2024-09-13T11:24:06.352 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:06.352 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:06.353 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:24:06.353 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:41.922253+0000", 2024-09-13T11:24:06.353 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.1155999999999997e-05 2024-09-13T11:24:06.353 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:06.353 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:06.353 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:24:06.353 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:41.924487+0000", 2024-09-13T11:24:06.353 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.002234541 2024-09-13T11:24:06.353 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:06.353 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:24:06.354 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:06.354 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:06.354 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:06.354 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPush(3.1f 97/78 [PushOp(3:fe604f47:::benchmark_data_smithi177_31116_object27978:head, version: 59'1753, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:fe604f47:::benchmark_data_smithi177_31116_object27978:head@59'1753, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:24:06.354 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:54.255485+0000", 2024-09-13T11:24:06.354 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 12.025109069999999, 2024-09-13T11:24:06.354 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.177312991, 2024-09-13T11:24:06.354 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:24:06.354 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:24:06.354 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:24:06.354 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:24:06.355 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:06.355 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:24:06.355 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:54.255485+0000", 2024-09-13T11:24:06.355 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:06.355 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:06.355 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:06.355 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:24:06.355 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:54.255485+0000", 2024-09-13T11:24:06.355 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:06.355 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:06.355 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:06.355 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:24:06.355 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:54.255488+0000", 2024-09-13T11:24:06.355 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.19e-06 2024-09-13T11:24:06.356 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:06.356 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:06.356 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:24:06.356 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:54.255596+0000", 2024-09-13T11:24:06.356 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000107964 2024-09-13T11:24:06.356 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:06.356 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:06.356 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:24:06.356 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:54.430551+0000", 2024-09-13T11:24:06.356 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.17495527999999999 2024-09-13T11:24:06.356 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:06.356 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:06.357 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:24:06.357 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:54.430592+0000", 2024-09-13T11:24:06.357 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.1384999999999998e-05 2024-09-13T11:24:06.357 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:06.357 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:06.357 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:24:06.357 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:54.430789+0000", 2024-09-13T11:24:06.357 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000196796 2024-09-13T11:24:06.357 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:06.357 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:06.357 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:24:06.357 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:54.430818+0000", 2024-09-13T11:24:06.358 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.8979000000000002e-05 2024-09-13T11:24:06.358 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:06.358 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:06.358 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:24:06.358 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:54.432798+0000", 2024-09-13T11:24:06.358 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0019793969999999999 2024-09-13T11:24:06.358 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:06.358 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:24:06.358 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:06.358 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:06.358 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:06.358 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:79480 3.2 3:42e4e644:::benchmark_data_smithi177_31116_object79479:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101)", 2024-09-13T11:24:06.358 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:24:00.046305+0000", 2024-09-13T11:24:06.358 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 6.2342890740000003, 2024-09-13T11:24:06.359 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.15557262599999999, 2024-09-13T11:24:06.359 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:24:06.359 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:24:06.359 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:06.359 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:24:06.359 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:24:06.359 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:06.359 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 79480 2024-09-13T11:24:06.359 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:06.359 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:24:06.359 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:06.359 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:24:06.359 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:00.046305+0000", 2024-09-13T11:24:06.360 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:06.360 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:06.360 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:06.360 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:24:06.360 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:00.046305+0000", 2024-09-13T11:24:06.360 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:06.360 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:06.360 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:06.360 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:24:06.360 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:00.046307+0000", 2024-09-13T11:24:06.360 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.2390000000000001e-06 2024-09-13T11:24:06.360 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:06.360 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:06.360 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:24:06.361 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:00.046344+0000", 2024-09-13T11:24:06.361 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.7477e-05 2024-09-13T11:24:06.361 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:06.361 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:06.361 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:24:06.361 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:00.046362+0000", 2024-09-13T11:24:06.362 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.7167e-05 2024-09-13T11:24:06.362 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:06.362 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:06.362 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:24:06.362 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:00.046368+0000", 2024-09-13T11:24:06.362 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.7390000000000002e-06 2024-09-13T11:24:06.363 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:06.363 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:06.363 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:24:06.363 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:00.046418+0000", 2024-09-13T11:24:06.363 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.9959000000000001e-05 2024-09-13T11:24:06.363 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:06.363 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:06.364 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:24:06.364 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:00.046504+0000", 2024-09-13T11:24:06.364 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.6155000000000004e-05 2024-09-13T11:24:06.364 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:06.364 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:06.365 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 12", 2024-09-13T11:24:06.365 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:00.046588+0000", 2024-09-13T11:24:06.365 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.3258999999999997e-05 2024-09-13T11:24:06.365 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:06.365 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:06.365 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:24:06.366 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:00.047803+0000", 2024-09-13T11:24:06.366 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0012155810000000001 2024-09-13T11:24:06.366 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:06.366 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:06.366 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:06.366 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:00.201685+0000", 2024-09-13T11:24:06.367 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.15388138300000001 2024-09-13T11:24:06.367 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:06.367 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:06.367 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:24:06.367 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:00.201796+0000", 2024-09-13T11:24:06.367 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000110939 2024-09-13T11:24:06.367 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:06.368 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:06.368 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:24:06.368 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:00.201877+0000", 2024-09-13T11:24:06.368 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.1728000000000001e-05 2024-09-13T11:24:06.368 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:06.369 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:24:06.369 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:06.369 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:06.369 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:06.369 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop_reply(client.4665.0:79480 3.2 e101/75 ondisk, result = 0)", 2024-09-13T11:24:06.369 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:24:00.048163+0000", 2024-09-13T11:24:06.370 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 6.2324308000000004, 2024-09-13T11:24:06.370 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.15372741100000001, 2024-09-13T11:24:06.370 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:24:06.370 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:24:06.370 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:24:06.370 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:24:06.371 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:06.371 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:24:06.371 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:00.048163+0000", 2024-09-13T11:24:06.371 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:06.371 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:06.371 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:06.372 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:24:06.372 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:00.048163+0000", 2024-09-13T11:24:06.372 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:06.372 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:06.372 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:06.373 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:24:06.373 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:00.048164+0000", 2024-09-13T11:24:06.373 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5269999999999999e-06 2024-09-13T11:24:06.373 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:06.373 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:06.373 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:24:06.374 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:00.048169+0000", 2024-09-13T11:24:06.374 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.7979999999999996e-06 2024-09-13T11:24:06.374 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:06.374 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:06.374 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:24:06.374 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:00.201498+0000", 2024-09-13T11:24:06.375 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.153328886 2024-09-13T11:24:06.375 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:06.375 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:06.375 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:24:06.375 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:00.201510+0000", 2024-09-13T11:24:06.375 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1768e-05 2024-09-13T11:24:06.376 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:06.376 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:06.376 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:24:06.376 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:00.201645+0000", 2024-09-13T11:24:06.376 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00013516900000000001 2024-09-13T11:24:06.376 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:06.377 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:06.377 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:24:06.377 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:00.201662+0000", 2024-09-13T11:24:06.377 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.647e-05 2024-09-13T11:24:06.377 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:06.377 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:06.378 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:24:06.378 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:00.201890+0000", 2024-09-13T11:24:06.378 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000228793 2024-09-13T11:24:06.378 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:06.378 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:24:06.378 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:06.379 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:06.379 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:06.379 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:84139 3.2 3:41ddbd59:::benchmark_data_smithi177_31116_object84138:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-09-13T11:24:06.379 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:24:05.428921+0000", 2024-09-13T11:24:06.379 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 0.85167266500000005, 2024-09-13T11:24:06.379 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.203750139, 2024-09-13T11:24:06.380 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:24:06.380 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:24:06.380 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:06.380 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:24:06.380 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:24:06.381 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:06.381 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 84139 2024-09-13T11:24:06.381 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:06.381 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:24:06.381 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:06.381 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:24:06.382 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.428921+0000", 2024-09-13T11:24:06.382 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:06.382 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:06.382 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:06.382 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:24:06.382 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.428921+0000", 2024-09-13T11:24:06.383 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:06.383 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:06.383 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:06.383 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:24:06.383 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.428926+0000", 2024-09-13T11:24:06.383 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.7299999999999996e-06 2024-09-13T11:24:06.384 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:06.384 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:06.384 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:24:06.384 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.428966+0000", 2024-09-13T11:24:06.384 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.0629e-05 2024-09-13T11:24:06.384 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:06.385 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:06.385 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:24:06.385 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.428978+0000", 2024-09-13T11:24:06.385 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1749999999999999e-05 2024-09-13T11:24:06.385 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:06.386 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:06.386 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:24:06.386 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.428987+0000", 2024-09-13T11:24:06.386 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.0909999999999999e-06 2024-09-13T11:24:06.386 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:06.386 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:06.387 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:24:06.387 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.429055+0000", 2024-09-13T11:24:06.387 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.7937999999999999e-05 2024-09-13T11:24:06.387 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:06.387 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:06.387 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:24:06.387 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.429240+0000", 2024-09-13T11:24:06.387 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00018448499999999999 2024-09-13T11:24:06.388 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:06.388 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:06.388 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 12", 2024-09-13T11:24:06.388 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.429423+0000", 2024-09-13T11:24:06.388 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000183012 2024-09-13T11:24:06.389 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:06.389 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:06.389 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:24:06.389 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.433639+0000", 2024-09-13T11:24:06.389 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0042158539999999998 2024-09-13T11:24:06.389 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:06.389 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:06.389 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:06.389 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.632566+0000", 2024-09-13T11:24:06.389 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.198927885 2024-09-13T11:24:06.389 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:06.390 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:06.390 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:24:06.390 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.632622+0000", 2024-09-13T11:24:06.390 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.5924000000000002e-05 2024-09-13T11:24:06.390 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:06.390 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:06.390 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:24:06.390 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.632671+0000", 2024-09-13T11:24:06.390 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.8841000000000002e-05 2024-09-13T11:24:06.390 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:06.390 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:24:06.390 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:06.391 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:06.391 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:06.391 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop_reply(client.4665.0:84139 3.2 e107/75 ondisk, result = 0)", 2024-09-13T11:24:06.391 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:24:05.431065+0000", 2024-09-13T11:24:06.391 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 0.84952914899999998, 2024-09-13T11:24:06.391 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.20161405499999999, 2024-09-13T11:24:06.391 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:24:06.391 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:24:06.391 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:24:06.391 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:24:06.391 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:06.391 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:24:06.392 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.431065+0000", 2024-09-13T11:24:06.392 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:06.392 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:06.392 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:06.392 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:24:06.392 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.431065+0000", 2024-09-13T11:24:06.392 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:06.392 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:06.392 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:06.392 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:24:06.392 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.431066+0000", 2024-09-13T11:24:06.392 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.454e-06 2024-09-13T11:24:06.392 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:06.393 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:06.393 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:24:06.393 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.431071+0000", 2024-09-13T11:24:06.393 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.5820000000000003e-06 2024-09-13T11:24:06.393 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:06.393 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:06.393 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:24:06.393 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.632499+0000", 2024-09-13T11:24:06.393 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.20142855700000001 2024-09-13T11:24:06.393 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:06.393 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:06.393 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:24:06.394 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.632507+0000", 2024-09-13T11:24:06.394 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.011e-06 2024-09-13T11:24:06.394 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:06.394 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:06.394 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:24:06.394 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.632553+0000", 2024-09-13T11:24:06.394 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.6326000000000003e-05 2024-09-13T11:24:06.394 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:06.394 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:06.394 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:24:06.394 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.632561+0000", 2024-09-13T11:24:06.394 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.4250000000000001e-06 2024-09-13T11:24:06.394 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:06.394 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:06.395 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:24:06.395 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.632679+0000", 2024-09-13T11:24:06.395 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0001177 2024-09-13T11:24:06.395 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:06.395 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:24:06.395 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:06.395 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:06.395 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:06.395 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:84707 3.2 3:4107e60f:::benchmark_data_smithi177_31116_object84706:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-09-13T11:24:06.395 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:24:05.810565+0000", 2024-09-13T11:24:06.395 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 0.470029162, 2024-09-13T11:24:06.395 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.16585786299999999, 2024-09-13T11:24:06.395 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:24:06.395 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:24:06.396 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:06.396 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:24:06.396 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:24:06.396 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:06.396 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 84707 2024-09-13T11:24:06.396 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:06.396 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:24:06.396 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:06.396 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:24:06.396 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.810565+0000", 2024-09-13T11:24:06.396 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:06.396 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:06.396 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:06.396 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:24:06.397 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.810565+0000", 2024-09-13T11:24:06.397 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:06.397 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:06.397 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:06.397 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:24:06.397 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.810568+0000", 2024-09-13T11:24:06.397 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.1970000000000001e-06 2024-09-13T11:24:06.397 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:06.397 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:06.397 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:24:06.397 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.810620+0000", 2024-09-13T11:24:06.397 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.2193e-05 2024-09-13T11:24:06.397 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:06.398 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:06.398 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:24:06.398 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.810645+0000", 2024-09-13T11:24:06.398 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.5491e-05 2024-09-13T11:24:06.398 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:06.398 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:06.398 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:24:06.398 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.810660+0000", 2024-09-13T11:24:06.398 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4620000000000001e-05 2024-09-13T11:24:06.398 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:06.398 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:06.398 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:24:06.399 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.810733+0000", 2024-09-13T11:24:06.399 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.2935999999999994e-05 2024-09-13T11:24:06.399 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:06.399 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:06.399 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:24:06.399 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.810845+0000", 2024-09-13T11:24:06.399 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000112259 2024-09-13T11:24:06.399 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:06.399 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:06.399 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 12", 2024-09-13T11:24:06.399 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.810960+0000", 2024-09-13T11:24:06.399 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00011454400000000001 2024-09-13T11:24:06.399 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:06.399 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:06.400 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:24:06.400 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.813269+0000", 2024-09-13T11:24:06.400 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0023093530000000001 2024-09-13T11:24:06.400 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:06.400 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:06.400 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:06.400 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.976324+0000", 2024-09-13T11:24:06.400 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.16305488200000001 2024-09-13T11:24:06.400 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:06.400 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:06.400 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:24:06.400 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.976380+0000", 2024-09-13T11:24:06.400 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.6403999999999997e-05 2024-09-13T11:24:06.401 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:06.401 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:06.401 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:24:06.401 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.976422+0000", 2024-09-13T11:24:06.401 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.1984000000000003e-05 2024-09-13T11:24:06.401 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:06.401 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:24:06.401 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:06.401 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:06.401 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:06.401 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop_reply(client.4665.0:84707 3.2 e107/75 ondisk, result = 0)", 2024-09-13T11:24:06.401 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:24:05.812491+0000", 2024-09-13T11:24:06.401 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 0.468102729, 2024-09-13T11:24:06.401 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.16393901399999999, 2024-09-13T11:24:06.402 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:24:06.402 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:24:06.402 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:24:06.402 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:24:06.402 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:06.402 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:24:06.402 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.812491+0000", 2024-09-13T11:24:06.402 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:06.402 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:06.402 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:06.402 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:24:06.402 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.812491+0000", 2024-09-13T11:24:06.403 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:06.403 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:06.403 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:06.403 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:24:06.403 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.812496+0000", 2024-09-13T11:24:06.403 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.2510000000000002e-06 2024-09-13T11:24:06.404 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:06.404 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:06.404 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:24:06.404 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.812500+0000", 2024-09-13T11:24:06.404 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.6770000000000001e-06 2024-09-13T11:24:06.404 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:06.404 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:06.404 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:24:06.404 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.976256+0000", 2024-09-13T11:24:06.404 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.163756549 2024-09-13T11:24:06.404 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:06.405 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:06.405 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:24:06.405 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.976261+0000", 2024-09-13T11:24:06.405 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.5229999999999997e-06 2024-09-13T11:24:06.405 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:06.405 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:06.405 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:24:06.405 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.976310+0000", 2024-09-13T11:24:06.405 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.8677999999999997e-05 2024-09-13T11:24:06.405 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:06.405 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:06.405 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:24:06.405 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.976318+0000", 2024-09-13T11:24:06.405 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.3659999999999992e-06 2024-09-13T11:24:06.406 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:06.406 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:06.406 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:24:06.406 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.976430+0000", 2024-09-13T11:24:06.406 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00011197 2024-09-13T11:24:06.406 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:06.406 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:24:06.406 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:06.406 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:06.406 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:06.406 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:84879 3.2 3:40162d9f:::benchmark_data_smithi177_31116_object84878:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108)", 2024-09-13T11:24:06.406 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:24:05.973920+0000", 2024-09-13T11:24:06.406 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 0.306674104, 2024-09-13T11:24:06.407 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.17611384699999999, 2024-09-13T11:24:06.407 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:24:06.407 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:24:06.407 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:06.407 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:24:06.407 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:24:06.407 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:06.407 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 84879 2024-09-13T11:24:06.407 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:06.407 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:24:06.407 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:06.407 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:24:06.407 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.973920+0000", 2024-09-13T11:24:06.408 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:06.408 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:06.408 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:06.408 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:24:06.408 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.973920+0000", 2024-09-13T11:24:06.408 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:06.408 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:06.408 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:06.408 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:24:06.408 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.973921+0000", 2024-09-13T11:24:06.408 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5179999999999999e-06 2024-09-13T11:24:06.408 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:06.408 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:06.409 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:24:06.409 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.974001+0000", 2024-09-13T11:24:06.409 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.9887000000000003e-05 2024-09-13T11:24:06.409 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:06.409 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:06.409 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:24:06.409 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.974010+0000", 2024-09-13T11:24:06.409 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.4329999999999993e-06 2024-09-13T11:24:06.409 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:06.409 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:06.410 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:24:06.410 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.974039+0000", 2024-09-13T11:24:06.410 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.9037e-05 2024-09-13T11:24:06.410 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:06.410 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:06.410 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:24:06.410 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.974104+0000", 2024-09-13T11:24:06.410 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.4349000000000003e-05 2024-09-13T11:24:06.410 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:06.411 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:06.411 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:24:06.411 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.974190+0000", 2024-09-13T11:24:06.411 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.6073000000000001e-05 2024-09-13T11:24:06.411 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:06.411 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:06.411 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 12", 2024-09-13T11:24:06.411 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.974266+0000", 2024-09-13T11:24:06.411 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.6205000000000006e-05 2024-09-13T11:24:06.411 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:06.411 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:06.411 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:24:06.411 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.975441+0000", 2024-09-13T11:24:06.412 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001174418 2024-09-13T11:24:06.412 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:06.412 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:06.412 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:06.412 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:06.149950+0000", 2024-09-13T11:24:06.412 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.17450983 2024-09-13T11:24:06.412 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:06.412 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:06.412 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:24:06.412 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:06.149988+0000", 2024-09-13T11:24:06.413 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.8136999999999998e-05 2024-09-13T11:24:06.413 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:06.413 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:06.413 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:24:06.413 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:06.150033+0000", 2024-09-13T11:24:06.413 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.4960000000000003e-05 2024-09-13T11:24:06.414 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:06.414 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:24:06.414 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:06.414 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:06.414 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:06.414 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop_reply(client.4665.0:84879 3.2 e108/75 ondisk, result = 0)", 2024-09-13T11:24:06.414 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:24:05.976234+0000", 2024-09-13T11:24:06.414 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 0.30435980200000001, 2024-09-13T11:24:06.414 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.17380630999999999, 2024-09-13T11:24:06.415 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:24:06.415 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:24:06.415 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:24:06.415 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:24:06.415 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:06.416 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:24:06.416 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.976234+0000", 2024-09-13T11:24:06.416 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:06.416 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:06.416 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:06.416 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:24:06.416 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.976234+0000", 2024-09-13T11:24:06.417 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:06.417 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:06.417 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:06.417 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:24:06.417 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.976235+0000", 2024-09-13T11:24:06.417 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.545e-06 2024-09-13T11:24:06.417 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:06.417 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:06.417 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:24:06.417 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.976245+0000", 2024-09-13T11:24:06.417 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.0969999999999995e-06 2024-09-13T11:24:06.417 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:06.417 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:06.417 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:24:06.418 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:06.149889+0000", 2024-09-13T11:24:06.418 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.173644405 2024-09-13T11:24:06.418 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:06.418 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:06.418 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:24:06.418 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:06.149894+0000", 2024-09-13T11:24:06.418 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.6979999999999997e-06 2024-09-13T11:24:06.418 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:06.419 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:06.419 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:24:06.419 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:06.149941+0000", 2024-09-13T11:24:06.419 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.6968000000000002e-05 2024-09-13T11:24:06.419 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:06.419 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:06.420 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:24:06.420 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:06.149946+0000", 2024-09-13T11:24:06.420 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.7780000000000004e-06 2024-09-13T11:24:06.420 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:06.420 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:06.420 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:24:06.421 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:06.150040+0000", 2024-09-13T11:24:06.421 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.3819000000000004e-05 2024-09-13T11:24:06.421 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:06.421 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:24:06.421 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:06.421 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:06.421 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:24:06.421 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-13T11:24:06.423 INFO:teuthology.orchestra.run.smithi001.stdout:osd.9: {} 2024-09-13T11:24:06.423 INFO:teuthology.orchestra.run.smithi001.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-13T11:24:06.424 INFO:teuthology.orchestra.run.smithi001.stdout:osd.10: {} 2024-09-13T11:24:06.424 INFO:teuthology.orchestra.run.smithi001.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-13T11:24:06.425 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-13T11:24:06.426 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-13T11:24:06.426 INFO:teuthology.orchestra.run.smithi001.stdout:{"fsid":"2fd68ac7-5581-4d90-80a4-30bcf5252a7a","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noscrub,nodeep-scrub flag(s) set","count":20},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":129,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":108,"num_osds":16,"num_up_osds":16,"osd_up_since":1726226566,"num_in_osds":16,"osd_in_since":1726226521,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":39},{"state_name":"active+recovering+remapped","count":3},{"state_name":"peering","count":2},{"state_name":"creating+peering","count":1}],"num_pgs":45,"num_pools":3,"num_objects":80332,"data_bytes":5264900659,"bytes_used":13736914944,"bytes_avail":1532451311616,"bytes_total":1546188226560,"inactive_pgs_ratio":0.066666670143604279,"misplaced_objects":1468,"misplaced_total":160664,"misplaced_ratio":0.009137081113379476,"write_bytes_sec":41268700,"read_op_per_sec":0,"write_op_per_sec":629},"fsmap":{"epoch":1,"btime":"2024-09-13T11:21:57:222574+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":5,"modified":"2024-09-13T11:23:46.137611+0000","services":{"osd":{"daemons":{"summary":"","5":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-09-13T11:24:06.427 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-09-13T11:24:06.427 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-13T11:24:06.427 INFO:teuthology.orchestra.run.smithi001.stdout:osd.11: {} 2024-09-13T11:24:06.428 INFO:teuthology.orchestra.run.smithi001.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-13T11:24:06.428 INFO:teuthology.orchestra.run.smithi001.stdout:osd.12: {} 2024-09-13T11:24:06.428 INFO:teuthology.orchestra.run.smithi001.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-13T11:24:06.429 INFO:tasks.ceph.osd.5.smithi092.stderr:2024-09-13T11:24:06.419+0000 7f6f0b705640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26346) UID: 0 2024-09-13T11:24:06.429 INFO:teuthology.orchestra.run.smithi001.stdout:osd.13: {} 2024-09-13T11:24:06.441 INFO:teuthology.orchestra.run.smithi001.stdout:osd.14: {} 2024-09-13T11:24:06.442 INFO:teuthology.orchestra.run.smithi001.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-13T11:24:06.473 INFO:teuthology.orchestra.run.smithi001.stdout:osd.15: {} 2024-09-13T11:24:06.473 INFO:teuthology.orchestra.run.smithi001.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-13T11:24:06.524 INFO:tasks.ceph.osd.14.smithi162.stderr:2024-09-13T11:24:06.521+0000 7f235c0db640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26319) UID: 0 2024-09-13T11:24:06.571 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-09-13T11:24:06.571 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:24:06.585 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-13T11:24:06.628 INFO:tasks.ceph.osd.11.smithi177.stderr:2024-09-13T11:24:06.627+0000 7f24792c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26580) UID: 0 2024-09-13T11:24:06.703 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-09-13T11:24:06.703 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:24:06.717 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-13T11:24:06.726 INFO:tasks.ceph.osd.10.smithi162.stderr:2024-09-13T11:24:06.725+0000 7fb59c9f0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26317) UID: 0 2024-09-13T11:24:06.774 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-13T11:24:06.775 INFO:teuthology.orchestra.run.smithi001.stderr:dumped all 2024-09-13T11:24:06.793 INFO:teuthology.orchestra.run.smithi001.stdout:{"pg_ready":true,"pg_map":{"version":152,"stamp":"2024-09-13T11:24:06.142567+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":5479793203,"num_objects":83611,"num_object_clones":0,"num_object_copies":167222,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":83611,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":37225,"num_write_kb":2375602,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11627,"num_bytes_recovered":761888787,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":144168,"ondisk_log_size":144168,"up":90,"acting":90,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":117,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":16,"kb":1509949440,"kb_used":13748580,"kb_used_data":12855972,"kb_used_omap":27,"kb_used_meta":892324,"kb_avail":1496200860,"statfs":{"total":1546188226560,"available":1532109680640,"internally_reserved":0,"allocated":13164515328,"data_stored":13157103841,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28358,"internal_metadata":913740090},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,1,1,4,3,0,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":214892544,"num_objects":3279,"num_object_clones":0,"num_object_copies":6558,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1468,"num_objects_unfound":0,"num_objects_dirty":3279,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3279,"num_write_kb":209856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":735,"num_bytes_recovered":48168960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000894"},"pg_stats":[{"pgid":"3.23","version":"108'3609","reported_seq":6341,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T11:24:05.966241+0000","last_change":"2024-09-13T11:23:48.133184+0000","last_active":"2024-09-13T11:24:05.966241+0000","last_peered":"2024-09-13T11:24:05.966241+0000","last_clean":"2024-09-13T11:24:05.966241+0000","last_became_active":"2024-09-13T11:23:47.926677+0000","last_became_peered":"2024-09-13T11:23:47.926677+0000","last_unstale":"2024-09-13T11:24:05.966241+0000","last_undegraded":"2024-09-13T11:24:05.966241+0000","last_fullsized":"2024-09-13T11:24:05.966241+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":6,"last_scrub":"91'3278","last_scrub_stamp":"2024-09-13T11:23:48.133084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:48.133084+0000","objects_scrubbed":1029,"log_size":3609,"log_dups_size":0,"ondisk_log_size":3609,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:12.874134+0000","scrub_duration":139,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":89194496,"num_objects":1361,"num_object_clones":0,"num_object_copies":2722,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1361,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":332,"num_write_kb":21248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9],"acting":[5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.22","version":"108'3574","reported_seq":5493,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T11:24:05.958953+0000","last_change":"2024-09-13T11:23:47.300446+0000","last_active":"2024-09-13T11:24:05.958953+0000","last_peered":"2024-09-13T11:24:05.958953+0000","last_clean":"2024-09-13T11:24:05.958953+0000","last_became_active":"2024-09-13T11:23:46.925318+0000","last_became_peered":"2024-09-13T11:23:46.925318+0000","last_unstale":"2024-09-13T11:24:05.958953+0000","last_undegraded":"2024-09-13T11:24:05.958953+0000","last_fullsized":"2024-09-13T11:24:05.958953+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":6,"last_scrub":"90'3246","last_scrub_stamp":"2024-09-13T11:23:47.300339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:47.300339+0000","objects_scrubbed":1033,"log_size":3574,"log_dups_size":0,"ondisk_log_size":3574,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:24:49.780485+0000","scrub_duration":203,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":89325568,"num_objects":1363,"num_object_clones":0,"num_object_copies":2726,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1363,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":330,"num_write_kb":21120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,14],"acting":[0,14],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.21","version":"108'3525","reported_seq":6247,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T11:24:05.984268+0000","last_change":"2024-09-13T11:24:02.142682+0000","last_active":"2024-09-13T11:24:05.984268+0000","last_peered":"2024-09-13T11:24:05.984268+0000","last_clean":"2024-09-13T11:24:05.984268+0000","last_became_active":"2024-09-13T11:24:01.250308+0000","last_became_peered":"2024-09-13T11:24:01.250308+0000","last_unstale":"2024-09-13T11:24:05.984268+0000","last_undegraded":"2024-09-13T11:24:05.984268+0000","last_fullsized":"2024-09-13T11:24:05.984268+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":6,"last_scrub":"69'2784","last_scrub_stamp":"2024-09-13T11:23:21.595419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:21.595419+0000","objects_scrubbed":0,"log_size":3525,"log_dups_size":0,"ondisk_log_size":3525,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"user requested scrub scheduled @ 2024-09-13T11:24:07.138422+0000","scrub_duration":311,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":84738048,"num_objects":1293,"num_object_clones":0,"num_object_copies":2586,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1293,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":600,"num_write_kb":38400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":836,"num_bytes_recovered":54788096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8],"acting":[0,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.20","version":"108'3606","reported_seq":5640,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T11:24:05.969931+0000","last_change":"2024-09-13T11:23:47.969503+0000","last_active":"2024-09-13T11:24:05.969931+0000","last_peered":"2024-09-13T11:24:05.969931+0000","last_clean":"2024-09-13T11:24:05.969931+0000","last_became_active":"2024-09-13T11:23:33.954946+0000","last_became_peered":"2024-09-13T11:23:33.954946+0000","last_unstale":"2024-09-13T11:24:05.969931+0000","last_undegraded":"2024-09-13T11:24:05.969931+0000","last_fullsized":"2024-09-13T11:24:05.969931+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":6,"last_scrub":"91'3270","last_scrub_stamp":"2024-09-13T11:23:47.969372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:47.969372+0000","objects_scrubbed":1061,"log_size":3606,"log_dups_size":0,"ondisk_log_size":3606,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:24:48.255263+0000","scrub_duration":1327,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":91881472,"num_objects":1402,"num_object_clones":0,"num_object_copies":2804,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1402,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":341,"num_write_kb":21824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,0],"acting":[12,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[]},{"pgid":"3.1f","version":"107'4324","reported_seq":9453,"reported_epoch":108,"state":"peering","last_fresh":"2024-09-13T11:24:05.943810+0000","last_change":"2024-09-13T11:24:05.943810+0000","last_active":"2024-09-13T11:24:05.943810+0000","last_peered":"2024-09-13T11:24:05.897907+0000","last_clean":"2024-09-13T11:23:32.316727+0000","last_became_active":"2024-09-13T11:23:35.053803+0000","last_became_peered":"2024-09-13T11:23:35.053803+0000","last_unstale":"2024-09-13T11:24:05.943810+0000","last_undegraded":"2024-09-13T11:24:05.943810+0000","last_fullsized":"2024-09-13T11:24:05.943810+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:22:24.896055+0000","objects_scrubbed":0,"log_size":4324,"log_dups_size":0,"ondisk_log_size":4324,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":171540480,"num_objects":2618,"num_object_clones":0,"num_object_copies":5236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2618,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2618,"num_write_kb":167520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1651,"num_bytes_recovered":108199936,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,14],"acting":[10,14],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1e","version":"107'4444","reported_seq":14807,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T11:24:05.956882+0000","last_change":"2024-09-13T11:24:02.037297+0000","last_active":"2024-09-13T11:24:05.956882+0000","last_peered":"2024-09-13T11:24:05.956882+0000","last_clean":"2024-09-13T11:24:05.956882+0000","last_became_active":"2024-09-13T11:24:01.261448+0000","last_became_peered":"2024-09-13T11:24:01.261448+0000","last_unstale":"2024-09-13T11:24:05.956882+0000","last_undegraded":"2024-09-13T11:24:05.956882+0000","last_fullsized":"2024-09-13T11:24:05.956882+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:22:24.896055+0000","objects_scrubbed":0,"log_size":4444,"log_dups_size":0,"ondisk_log_size":4444,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-13T11:24:06.974692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":175898624,"num_objects":2684,"num_object_clones":0,"num_object_copies":5368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2684,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2684,"num_write_kb":171776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2573,"num_bytes_recovered":168591360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"108'4306","reported_seq":9591,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T11:24:06.011843+0000","last_change":"2024-09-13T11:24:02.078835+0000","last_active":"2024-09-13T11:24:06.011843+0000","last_peered":"2024-09-13T11:24:06.011843+0000","last_clean":"2024-09-13T11:24:06.011843+0000","last_became_active":"2024-09-13T11:24:01.258225+0000","last_became_peered":"2024-09-13T11:24:01.258225+0000","last_unstale":"2024-09-13T11:24:06.011843+0000","last_undegraded":"2024-09-13T11:24:06.011843+0000","last_fullsized":"2024-09-13T11:24:06.011843+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:22:24.896055+0000","objects_scrubbed":0,"log_size":4306,"log_dups_size":0,"ondisk_log_size":4306,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-13T11:24:07.039585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":171376640,"num_objects":2615,"num_object_clones":0,"num_object_copies":5230,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2615,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2615,"num_write_kb":167360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1700,"num_bytes_recovered":111411200,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,15],"acting":[2,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1c","version":"104'4310","reported_seq":7986,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-13T11:24:01.961650+0000","last_change":"2024-09-13T11:24:00.529312+0000","last_active":"2024-09-13T11:24:01.961650+0000","last_peered":"2024-09-13T11:24:01.961650+0000","last_clean":"2024-09-13T11:24:01.961650+0000","last_became_active":"2024-09-13T11:23:33.989492+0000","last_became_peered":"2024-09-13T11:23:33.989492+0000","last_unstale":"2024-09-13T11:24:01.961650+0000","last_undegraded":"2024-09-13T11:24:01.961650+0000","last_fullsized":"2024-09-13T11:24:01.961650+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":5,"last_scrub":"102'4277","last_scrub_stamp":"2024-09-13T11:24:00.529243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:24:00.529243+0000","objects_scrubbed":2544,"log_size":4310,"log_dups_size":0,"ondisk_log_size":4310,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:15.595052+0000","scrub_duration":259,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":169017344,"num_objects":2579,"num_object_clones":0,"num_object_copies":5158,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2579,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":35,"num_write_kb":2240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7],"acting":[11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1b","version":"108'4380","reported_seq":9848,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T11:24:06.092111+0000","last_change":"2024-09-13T11:24:05.835103+0000","last_active":"2024-09-13T11:24:06.092111+0000","last_peered":"2024-09-13T11:24:06.092111+0000","last_clean":"2024-09-13T11:24:06.092111+0000","last_became_active":"2024-09-13T11:24:04.950821+0000","last_became_peered":"2024-09-13T11:24:04.950821+0000","last_unstale":"2024-09-13T11:24:06.092111+0000","last_undegraded":"2024-09-13T11:24:06.092111+0000","last_fullsized":"2024-09-13T11:24:06.092111+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:22:24.896055+0000","objects_scrubbed":0,"log_size":4380,"log_dups_size":0,"ondisk_log_size":4380,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-13T11:24:10.806111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":173637632,"num_objects":2650,"num_object_clones":0,"num_object_copies":5300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2650,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2650,"num_write_kb":169568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1760,"num_bytes_recovered":115343360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,15],"acting":[12,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[]},{"pgid":"3.1a","version":"108'4273","reported_seq":9405,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T11:24:06.022201+0000","last_change":"2024-09-13T11:24:04.144689+0000","last_active":"2024-09-13T11:24:06.022201+0000","last_peered":"2024-09-13T11:24:06.022201+0000","last_clean":"2024-09-13T11:24:06.022201+0000","last_became_active":"2024-09-13T11:24:03.935185+0000","last_became_peered":"2024-09-13T11:24:03.935185+0000","last_unstale":"2024-09-13T11:24:06.022201+0000","last_undegraded":"2024-09-13T11:24:06.022201+0000","last_fullsized":"2024-09-13T11:24:06.022201+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:22:24.896055+0000","objects_scrubbed":0,"log_size":4273,"log_dups_size":0,"ondisk_log_size":4273,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-13T11:24:09.133075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":168230912,"num_objects":2567,"num_object_clones":0,"num_object_copies":5134,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2567,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2567,"num_write_kb":164288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1648,"num_bytes_recovered":108003328,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9],"acting":[4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.19","version":"104'3901","reported_seq":8200,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-13T11:24:01.930513+0000","last_change":"2024-09-13T11:23:49.429522+0000","last_active":"2024-09-13T11:24:01.930513+0000","last_peered":"2024-09-13T11:24:01.930513+0000","last_clean":"2024-09-13T11:24:01.930513+0000","last_became_active":"2024-09-13T11:23:41.924941+0000","last_became_peered":"2024-09-13T11:23:41.924941+0000","last_unstale":"2024-09-13T11:24:01.930513+0000","last_undegraded":"2024-09-13T11:24:01.930513+0000","last_fullsized":"2024-09-13T11:24:01.930513+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":5,"last_scrub":"92'3415","last_scrub_stamp":"2024-09-13T11:23:49.429429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:49.429429+0000","objects_scrubbed":1979,"log_size":3901,"log_dups_size":0,"ondisk_log_size":3901,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:04.629542+0000","scrub_duration":587,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":162398208,"num_objects":2478,"num_object_clones":0,"num_object_copies":4956,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2478,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":499,"num_write_kb":31936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1],"acting":[9,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.18","version":"108'4042","reported_seq":8617,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T11:24:05.971274+0000","last_change":"2024-09-13T11:24:02.096598+0000","last_active":"2024-09-13T11:24:05.971274+0000","last_peered":"2024-09-13T11:24:05.971274+0000","last_clean":"2024-09-13T11:24:05.971274+0000","last_became_active":"2024-09-13T11:23:48.926014+0000","last_became_peered":"2024-09-13T11:23:48.926014+0000","last_unstale":"2024-09-13T11:24:05.971274+0000","last_undegraded":"2024-09-13T11:24:05.971274+0000","last_fullsized":"2024-09-13T11:24:05.971274+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:22:24.896055+0000","objects_scrubbed":0,"log_size":4042,"log_dups_size":0,"ondisk_log_size":4042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-13T11:24:07.029225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":175308800,"num_objects":2675,"num_object_clones":0,"num_object_copies":5350,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2675,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2675,"num_write_kb":171200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1458,"num_bytes_recovered":95551488,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0],"acting":[10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.17","version":"107'3954","reported_seq":7995,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T11:24:05.943810+0000","last_change":"2024-09-13T11:23:49.349241+0000","last_active":"2024-09-13T11:24:05.943810+0000","last_peered":"2024-09-13T11:24:05.943810+0000","last_clean":"2024-09-13T11:24:05.943810+0000","last_became_active":"2024-09-13T11:23:42.935601+0000","last_became_peered":"2024-09-13T11:23:42.935601+0000","last_unstale":"2024-09-13T11:24:05.943810+0000","last_undegraded":"2024-09-13T11:24:05.943810+0000","last_fullsized":"2024-09-13T11:24:05.943810+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":5,"last_scrub":"92'3359","last_scrub_stamp":"2024-09-13T11:23:49.349167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:49.349167+0000","objects_scrubbed":2004,"log_size":3954,"log_dups_size":0,"ondisk_log_size":3954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:24:59.401302+0000","scrub_duration":363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":170852352,"num_objects":2607,"num_object_clones":0,"num_object_copies":5214,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2607,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":603,"num_write_kb":38592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3],"acting":[10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"104'4063","reported_seq":8724,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-13T11:24:01.937217+0000","last_change":"2024-09-13T11:23:46.510544+0000","last_active":"2024-09-13T11:24:01.937217+0000","last_peered":"2024-09-13T11:24:01.937217+0000","last_clean":"2024-09-13T11:24:01.937217+0000","last_became_active":"2024-09-13T11:23:46.105237+0000","last_became_peered":"2024-09-13T11:23:46.105237+0000","last_unstale":"2024-09-13T11:24:01.937217+0000","last_undegraded":"2024-09-13T11:24:01.937217+0000","last_fullsized":"2024-09-13T11:24:01.937217+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":5,"last_scrub":"89'3478","last_scrub_stamp":"2024-09-13T11:23:46.510384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:46.510384+0000","objects_scrubbed":2054,"log_size":4063,"log_dups_size":0,"ondisk_log_size":4063,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:09.149878+0000","scrub_duration":363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":173146112,"num_objects":2642,"num_object_clones":0,"num_object_copies":5284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2642,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":588,"num_write_kb":37632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,15],"acting":[11,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.15","version":"108'3978","reported_seq":8409,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T11:24:06.018660+0000","last_change":"2024-09-13T11:23:48.332790+0000","last_active":"2024-09-13T11:24:06.018660+0000","last_peered":"2024-09-13T11:24:06.018660+0000","last_clean":"2024-09-13T11:24:06.018660+0000","last_became_active":"2024-09-13T11:23:42.931149+0000","last_became_peered":"2024-09-13T11:23:42.931149+0000","last_unstale":"2024-09-13T11:24:06.018660+0000","last_undegraded":"2024-09-13T11:24:06.018660+0000","last_fullsized":"2024-09-13T11:24:06.018660+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":5,"last_scrub":"91'3384","last_scrub_stamp":"2024-09-13T11:23:48.332600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:48.332600+0000","objects_scrubbed":2032,"log_size":3978,"log_dups_size":0,"ondisk_log_size":3978,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:24:58.239508+0000","scrub_duration":235,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":172163072,"num_objects":2627,"num_object_clones":0,"num_object_copies":5254,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2627,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":595,"num_write_kb":38080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,0],"acting":[14,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"3.14","version":"107'3984","reported_seq":8233,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-13T11:24:05.038872+0000","last_change":"2024-09-13T11:23:48.697100+0000","last_active":"2024-09-13T11:24:05.038872+0000","last_peered":"2024-09-13T11:24:05.038872+0000","last_clean":"2024-09-13T11:24:05.038872+0000","last_became_active":"2024-09-13T11:23:41.920989+0000","last_became_peered":"2024-09-13T11:23:41.920989+0000","last_unstale":"2024-09-13T11:24:05.038872+0000","last_undegraded":"2024-09-13T11:24:05.038872+0000","last_fullsized":"2024-09-13T11:24:05.038872+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":5,"last_scrub":"91'3371","last_scrub_stamp":"2024-09-13T11:23:48.696960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:48.696960+0000","objects_scrubbed":1935,"log_size":3984,"log_dups_size":0,"ondisk_log_size":3984,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:02.861530+0000","scrub_duration":735,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":167641088,"num_objects":2558,"num_object_clones":0,"num_object_copies":5116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2558,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":623,"num_write_kb":39872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,14],"acting":[3,14],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":179,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T11:24:05.944977+0000","last_change":"2024-09-13T11:22:16.889130+0000","last_active":"2024-09-13T11:24:05.944977+0000","last_peered":"2024-09-13T11:24:05.944977+0000","last_clean":"2024-09-13T11:24:05.944977+0000","last_became_active":"2024-09-13T11:22:14.192897+0000","last_became_peered":"2024-09-13T11:22:14.192897+0000","last_unstale":"2024-09-13T11:24:05.944977+0000","last_undegraded":"2024-09-13T11:24:05.944977+0000","last_fullsized":"2024-09-13T11:24:05.944977+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:22:13.113563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:24:09.524691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00084655899999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,10],"acting":[15,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"3.6","version":"104'4010","reported_seq":4199,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-13T11:24:02.034252+0000","last_change":"2024-09-13T11:23:22.656608+0000","last_active":"2024-09-13T11:24:02.034252+0000","last_peered":"2024-09-13T11:24:02.034252+0000","last_clean":"2024-09-13T11:24:02.034252+0000","last_became_active":"2024-09-13T11:23:19.939432+0000","last_became_peered":"2024-09-13T11:23:19.939432+0000","last_unstale":"2024-09-13T11:24:02.034252+0000","last_undegraded":"2024-09-13T11:24:02.034252+0000","last_fullsized":"2024-09-13T11:24:02.034252+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'2909","last_scrub_stamp":"2024-09-13T11:23:22.656323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:22.656323+0000","objects_scrubbed":1487,"log_size":4010,"log_dups_size":0,"ondisk_log_size":4010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:24:31.905185+0000","scrub_duration":323,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":169607168,"num_objects":2588,"num_object_clones":0,"num_object_copies":5176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2588,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1101,"num_write_kb":70464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,14],"acting":[13,14],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":178,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-13T11:24:01.930648+0000","last_change":"2024-09-13T11:23:21.875855+0000","last_active":"2024-09-13T11:24:01.930648+0000","last_peered":"2024-09-13T11:24:01.930648+0000","last_clean":"2024-09-13T11:24:01.930648+0000","last_became_active":"2024-09-13T11:22:14.211923+0000","last_became_peered":"2024-09-13T11:22:14.211923+0000","last_unstale":"2024-09-13T11:24:01.930648+0000","last_undegraded":"2024-09-13T11:24:01.930648+0000","last_fullsized":"2024-09-13T11:24:01.930648+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:23:21.875683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:23:21.875683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:24:40.678136+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0011689529999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0],"acting":[9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"104'3857","reported_seq":4047,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-13T11:24:02.041621+0000","last_change":"2024-09-13T11:23:20.443418+0000","last_active":"2024-09-13T11:24:02.041621+0000","last_peered":"2024-09-13T11:24:02.041621+0000","last_clean":"2024-09-13T11:24:02.041621+0000","last_became_active":"2024-09-13T11:23:19.927680+0000","last_became_peered":"2024-09-13T11:23:19.927680+0000","last_unstale":"2024-09-13T11:24:02.041621+0000","last_undegraded":"2024-09-13T11:24:02.041621+0000","last_fullsized":"2024-09-13T11:24:02.041621+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'2736","last_scrub_stamp":"2024-09-13T11:23:20.443329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:20.443329+0000","objects_scrubbed":1369,"log_size":3857,"log_dups_size":0,"ondisk_log_size":3857,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:24:48.147625+0000","scrub_duration":183,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":163381248,"num_objects":2493,"num_object_clones":0,"num_object_copies":4986,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2493,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1124,"num_write_kb":71936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,8],"acting":[13,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":187,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T11:24:05.957598+0000","last_change":"2024-09-13T11:23:59.530794+0000","last_active":"2024-09-13T11:24:05.957598+0000","last_peered":"2024-09-13T11:24:05.957598+0000","last_clean":"2024-09-13T11:24:05.957598+0000","last_became_active":"2024-09-13T11:22:14.581458+0000","last_became_peered":"2024-09-13T11:22:14.581458+0000","last_unstale":"2024-09-13T11:24:05.957598+0000","last_undegraded":"2024-09-13T11:24:05.957598+0000","last_fullsized":"2024-09-13T11:24:05.957598+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:23:59.530670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:23:59.530670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:02.655631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001026047,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5],"acting":[8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"107'3984","reported_seq":4160,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-13T11:24:05.321411+0000","last_change":"2024-09-13T11:23:20.254213+0000","last_active":"2024-09-13T11:24:05.321411+0000","last_peered":"2024-09-13T11:24:05.321411+0000","last_clean":"2024-09-13T11:24:05.321411+0000","last_became_active":"2024-09-13T11:23:19.925399+0000","last_became_peered":"2024-09-13T11:23:19.925399+0000","last_unstale":"2024-09-13T11:24:05.321411+0000","last_undegraded":"2024-09-13T11:24:05.321411+0000","last_fullsized":"2024-09-13T11:24:05.321411+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'2755","last_scrub_stamp":"2024-09-13T11:23:20.254124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:20.254124+0000","objects_scrubbed":1437,"log_size":3984,"log_dups_size":0,"ondisk_log_size":3984,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:24:49.993543+0000","scrub_duration":187,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":175046656,"num_objects":2671,"num_object_clones":0,"num_object_copies":5342,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2671,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1234,"num_write_kb":78976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,12],"acting":[1,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":165,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-13T11:24:04.977227+0000","last_change":"2024-09-13T11:22:47.720313+0000","last_active":"2024-09-13T11:24:04.977227+0000","last_peered":"2024-09-13T11:24:04.977227+0000","last_clean":"2024-09-13T11:24:04.977227+0000","last_became_active":"2024-09-13T11:22:47.719636+0000","last_became_peered":"2024-09-13T11:22:47.719636+0000","last_unstale":"2024-09-13T11:24:04.977227+0000","last_undegraded":"2024-09-13T11:24:04.977227+0000","last_fullsized":"2024-09-13T11:24:04.977227+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:22:13.113563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00087476000000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,9],"acting":[1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"108'4001","reported_seq":4199,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T11:24:05.961301+0000","last_change":"2024-09-13T11:23:20.686446+0000","last_active":"2024-09-13T11:24:05.961301+0000","last_peered":"2024-09-13T11:24:05.961301+0000","last_clean":"2024-09-13T11:24:05.961301+0000","last_became_active":"2024-09-13T11:23:19.939695+0000","last_became_peered":"2024-09-13T11:23:19.939695+0000","last_unstale":"2024-09-13T11:24:05.961301+0000","last_undegraded":"2024-09-13T11:24:05.961301+0000","last_fullsized":"2024-09-13T11:24:05.961301+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'2739","last_scrub_stamp":"2024-09-13T11:23:20.686358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:20.686358+0000","objects_scrubbed":1386,"log_size":4001,"log_dups_size":0,"ondisk_log_size":4001,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:24:38.891737+0000","scrub_duration":271,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":173604864,"num_objects":2649,"num_object_clones":0,"num_object_copies":5298,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2649,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1263,"num_write_kb":80832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,13],"acting":[15,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"3.3","version":"107'3547","reported_seq":3768,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T11:24:05.942396+0000","last_change":"2024-09-13T11:23:32.147115+0000","last_active":"2024-09-13T11:24:05.942396+0000","last_peered":"2024-09-13T11:24:05.942396+0000","last_clean":"2024-09-13T11:24:05.942396+0000","last_became_active":"2024-09-13T11:23:31.930448+0000","last_became_peered":"2024-09-13T11:23:31.930448+0000","last_unstale":"2024-09-13T11:24:05.942396+0000","last_undegraded":"2024-09-13T11:24:05.942396+0000","last_fullsized":"2024-09-13T11:24:05.942396+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'3122","last_scrub_stamp":"2024-09-13T11:23:32.147021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:32.147021+0000","objects_scrubbed":876,"log_size":3547,"log_dups_size":0,"ondisk_log_size":3547,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:24:48.175541+0000","scrub_duration":99,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":85262336,"num_objects":1301,"num_object_clones":0,"num_object_copies":2602,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1301,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":425,"num_write_kb":27200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,7],"acting":[10,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"28'2","reported_seq":117,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T11:24:05.963496+0000","last_change":"2024-09-13T11:24:00.996197+0000","last_active":"2024-09-13T11:24:05.963496+0000","last_peered":"2024-09-13T11:24:05.963496+0000","last_clean":"2024-09-13T11:24:05.963496+0000","last_became_active":"2024-09-13T11:23:12.546348+0000","last_became_peered":"2024-09-13T11:23:12.546348+0000","last_unstale":"2024-09-13T11:24:05.963496+0000","last_undegraded":"2024-09-13T11:24:05.963496+0000","last_fullsized":"2024-09-13T11:24:05.963496+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2","last_scrub_stamp":"2024-09-13T11:24:00.996028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:24:00.996028+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:12.111097+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0025826730000000002,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8],"acting":[2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"108'3566","reported_seq":3785,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T11:24:06.031700+0000","last_change":"2024-09-13T11:23:32.749651+0000","last_active":"2024-09-13T11:24:06.031700+0000","last_peered":"2024-09-13T11:24:06.031700+0000","last_clean":"2024-09-13T11:24:06.031700+0000","last_became_active":"2024-09-13T11:23:31.921388+0000","last_became_peered":"2024-09-13T11:23:31.921388+0000","last_unstale":"2024-09-13T11:24:06.031700+0000","last_undegraded":"2024-09-13T11:24:06.031700+0000","last_fullsized":"2024-09-13T11:24:06.031700+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'3106","last_scrub_stamp":"2024-09-13T11:23:32.749487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:32.749487+0000","objects_scrubbed":856,"log_size":3566,"log_dups_size":0,"ondisk_log_size":3566,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:24:34.709501+0000","scrub_duration":91,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":86245376,"num_objects":1316,"num_object_clones":0,"num_object_copies":2632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1316,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":460,"num_write_kb":29440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,15],"acting":[12,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":189,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T11:24:05.943005+0000","last_change":"2024-09-13T11:22:47.298206+0000","last_active":"2024-09-13T11:24:05.943005+0000","last_peered":"2024-09-13T11:24:05.943005+0000","last_clean":"2024-09-13T11:24:05.943005+0000","last_became_active":"2024-09-13T11:22:47.297319+0000","last_became_peered":"2024-09-13T11:22:47.297319+0000","last_unstale":"2024-09-13T11:24:05.943005+0000","last_undegraded":"2024-09-13T11:24:05.943005+0000","last_fullsized":"2024-09-13T11:24:05.943005+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:22:13.113563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00081225399999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,1],"acting":[14,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"3.1","version":"104'3482","reported_seq":3693,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-13T11:24:02.051997+0000","last_change":"2024-09-13T11:23:33.604966+0000","last_active":"2024-09-13T11:24:02.051997+0000","last_peered":"2024-09-13T11:24:02.051997+0000","last_clean":"2024-09-13T11:24:02.051997+0000","last_became_active":"2024-09-13T11:23:31.926528+0000","last_became_peered":"2024-09-13T11:23:31.926528+0000","last_unstale":"2024-09-13T11:24:02.051997+0000","last_undegraded":"2024-09-13T11:24:02.051997+0000","last_fullsized":"2024-09-13T11:24:02.051997+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'3070","last_scrub_stamp":"2024-09-13T11:23:33.604792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:33.604792+0000","objects_scrubbed":842,"log_size":3482,"log_dups_size":0,"ondisk_log_size":3482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:24:43.880038+0000","scrub_duration":119,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":82182144,"num_objects":1254,"num_object_clones":0,"num_object_copies":2508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1254,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":412,"num_write_kb":26368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,11],"acting":[13,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":177,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-13T11:24:04.954068+0000","last_change":"2024-09-13T11:22:16.267999+0000","last_active":"2024-09-13T11:24:04.954068+0000","last_peered":"2024-09-13T11:24:04.954068+0000","last_clean":"2024-09-13T11:24:04.954068+0000","last_became_active":"2024-09-13T11:22:14.211435+0000","last_became_peered":"2024-09-13T11:22:14.211435+0000","last_unstale":"2024-09-13T11:24:04.954068+0000","last_undegraded":"2024-09-13T11:24:04.954068+0000","last_fullsized":"2024-09-13T11:24:04.954068+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:22:13.113563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010010360000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2","version":"108'3556","reported_seq":3775,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T11:24:05.944539+0000","last_change":"2024-09-13T11:23:32.977666+0000","last_active":"2024-09-13T11:24:05.944539+0000","last_peered":"2024-09-13T11:24:05.944539+0000","last_clean":"2024-09-13T11:24:05.944539+0000","last_became_active":"2024-09-13T11:23:31.919828+0000","last_became_peered":"2024-09-13T11:23:31.919828+0000","last_unstale":"2024-09-13T11:24:05.944539+0000","last_undegraded":"2024-09-13T11:24:05.944539+0000","last_fullsized":"2024-09-13T11:24:05.944539+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'3057","last_scrub_stamp":"2024-09-13T11:23:32.977559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:32.977559+0000","objects_scrubbed":841,"log_size":3556,"log_dups_size":0,"ondisk_log_size":3556,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:24:52.169339+0000","scrub_duration":151,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":87818240,"num_objects":1340,"num_object_clones":0,"num_object_copies":2680,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1340,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":499,"num_write_kb":31936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,12],"acting":[14,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"2.3","version":"26'1","reported_seq":181,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T11:24:05.943338+0000","last_change":"2024-09-13T11:22:16.547691+0000","last_active":"2024-09-13T11:24:05.943338+0000","last_peered":"2024-09-13T11:24:05.943338+0000","last_clean":"2024-09-13T11:24:05.943338+0000","last_became_active":"2024-09-13T11:22:14.146507+0000","last_became_peered":"2024-09-13T11:22:14.146507+0000","last_unstale":"2024-09-13T11:24:05.943338+0000","last_undegraded":"2024-09-13T11:24:05.943338+0000","last_fullsized":"2024-09-13T11:24:05.943338+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:22:13.113563+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00085610999999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,15],"acting":[14,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"1.0","version":"26'112","reported_seq":321,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T11:24:05.945516+0000","last_change":"2024-09-13T11:23:49.325249+0000","last_active":"2024-09-13T11:24:05.945516+0000","last_peered":"2024-09-13T11:24:05.945516+0000","last_clean":"2024-09-13T11:24:05.945516+0000","last_became_active":"2024-09-13T11:22:12.129721+0000","last_became_peered":"2024-09-13T11:22:12.129721+0000","last_unstale":"2024-09-13T11:24:05.945516+0000","last_undegraded":"2024-09-13T11:24:05.945516+0000","last_fullsized":"2024-09-13T11:24:05.945516+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"26'112","last_scrub_stamp":"2024-09-13T11:23:49.325175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:11.098388+0000","last_clean_scrub_stamp":"2024-09-13T11:23:49.325175+0000","objects_scrubbed":2,"log_size":112,"log_dups_size":0,"ondisk_log_size":112,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:14.422287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.8","version":"108'3977","reported_seq":4176,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T11:24:06.016291+0000","last_change":"2024-09-13T11:23:20.644952+0000","last_active":"2024-09-13T11:24:06.016291+0000","last_peered":"2024-09-13T11:24:06.016291+0000","last_clean":"2024-09-13T11:24:06.016291+0000","last_became_active":"2024-09-13T11:23:19.925802+0000","last_became_peered":"2024-09-13T11:23:19.925802+0000","last_unstale":"2024-09-13T11:24:06.016291+0000","last_undegraded":"2024-09-13T11:24:06.016291+0000","last_fullsized":"2024-09-13T11:24:06.016291+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'2771","last_scrub_stamp":"2024-09-13T11:23:20.644747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:20.644747+0000","objects_scrubbed":1356,"log_size":3977,"log_dups_size":0,"ondisk_log_size":3977,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:24:38.144523+0000","scrub_duration":411,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":168689664,"num_objects":2574,"num_object_clones":0,"num_object_copies":5148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2574,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1218,"num_write_kb":77952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.9","version":"108'4078","reported_seq":4275,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T11:24:06.044946+0000","last_change":"2024-09-13T11:23:22.033678+0000","last_active":"2024-09-13T11:24:06.044946+0000","last_peered":"2024-09-13T11:24:06.044946+0000","last_clean":"2024-09-13T11:24:06.044946+0000","last_became_active":"2024-09-13T11:23:19.935114+0000","last_became_peered":"2024-09-13T11:23:19.935114+0000","last_unstale":"2024-09-13T11:24:06.044946+0000","last_undegraded":"2024-09-13T11:24:06.044946+0000","last_fullsized":"2024-09-13T11:24:06.044946+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'2808","last_scrub_stamp":"2024-09-13T11:23:22.033572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:22.033572+0000","objects_scrubbed":1471,"log_size":4078,"log_dups_size":0,"ondisk_log_size":4078,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:24:46.595859+0000","scrub_duration":287,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":179634176,"num_objects":2741,"num_object_clones":0,"num_object_copies":5482,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2741,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1270,"num_write_kb":81280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,2],"acting":[12,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[]},{"pgid":"3.a","version":"107'4299","reported_seq":4499,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-13T11:24:05.018512+0000","last_change":"2024-09-13T11:23:32.526150+0000","last_active":"2024-09-13T11:24:05.018512+0000","last_peered":"2024-09-13T11:24:05.018512+0000","last_clean":"2024-09-13T11:24:05.018512+0000","last_became_active":"2024-09-13T11:23:31.934170+0000","last_became_peered":"2024-09-13T11:23:31.934170+0000","last_unstale":"2024-09-13T11:24:05.018512+0000","last_undegraded":"2024-09-13T11:24:05.018512+0000","last_fullsized":"2024-09-13T11:24:05.018512+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'3418","last_scrub_stamp":"2024-09-13T11:23:32.526057+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:32.526057+0000","objects_scrubbed":1774,"log_size":4299,"log_dups_size":0,"ondisk_log_size":4299,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:24:34.770161+0000","scrub_duration":507,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":173998080,"num_objects":2655,"num_object_clones":0,"num_object_copies":5310,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2655,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":881,"num_write_kb":56384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.b","version":"104'4264","reported_seq":4457,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-13T11:24:02.000298+0000","last_change":"2024-09-13T11:23:32.681343+0000","last_active":"2024-09-13T11:24:02.000298+0000","last_peered":"2024-09-13T11:24:02.000298+0000","last_clean":"2024-09-13T11:24:02.000298+0000","last_became_active":"2024-09-13T11:23:31.923415+0000","last_became_peered":"2024-09-13T11:23:31.923415+0000","last_unstale":"2024-09-13T11:24:02.000298+0000","last_undegraded":"2024-09-13T11:24:02.000298+0000","last_fullsized":"2024-09-13T11:24:02.000298+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'3471","last_scrub_stamp":"2024-09-13T11:23:32.681257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:32.681257+0000","objects_scrubbed":1720,"log_size":4264,"log_dups_size":0,"ondisk_log_size":4264,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:24:58.857812+0000","scrub_duration":199,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":164691968,"num_objects":2513,"num_object_clones":0,"num_object_copies":5026,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2513,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":793,"num_write_kb":50752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,1],"acting":[13,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"3.c","version":"104'4251","reported_seq":4443,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-13T11:24:02.035553+0000","last_change":"2024-09-13T11:23:32.524870+0000","last_active":"2024-09-13T11:24:02.035553+0000","last_peered":"2024-09-13T11:24:02.035553+0000","last_clean":"2024-09-13T11:24:02.035553+0000","last_became_active":"2024-09-13T11:23:31.922620+0000","last_became_peered":"2024-09-13T11:23:31.922620+0000","last_unstale":"2024-09-13T11:24:02.035553+0000","last_undegraded":"2024-09-13T11:24:02.035553+0000","last_fullsized":"2024-09-13T11:24:02.035553+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'3463","last_scrub_stamp":"2024-09-13T11:23:32.524776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:32.524776+0000","objects_scrubbed":1741,"log_size":4251,"log_dups_size":0,"ondisk_log_size":4251,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:24:37.745747+0000","scrub_duration":235,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":165740544,"num_objects":2529,"num_object_clones":0,"num_object_copies":5058,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2529,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":788,"num_write_kb":50432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,13],"acting":[11,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.d","version":"102'4172","reported_seq":4360,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-13T11:24:00.790903+0000","last_change":"2024-09-13T11:23:32.776835+0000","last_active":"2024-09-13T11:24:00.790903+0000","last_peered":"2024-09-13T11:24:00.790903+0000","last_clean":"2024-09-13T11:24:00.790903+0000","last_became_active":"2024-09-13T11:23:31.939246+0000","last_became_peered":"2024-09-13T11:23:31.939246+0000","last_unstale":"2024-09-13T11:24:00.790903+0000","last_undegraded":"2024-09-13T11:24:00.790903+0000","last_fullsized":"2024-09-13T11:24:00.790903+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'3395","last_scrub_stamp":"2024-09-13T11:23:32.776747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:32.776747+0000","objects_scrubbed":1704,"log_size":4172,"log_dups_size":0,"ondisk_log_size":4172,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:24:56.402862+0000","scrub_duration":179,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":162594816,"num_objects":2481,"num_object_clones":0,"num_object_copies":4962,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2481,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":777,"num_write_kb":49728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.e","version":"108'4435","reported_seq":9563,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T11:24:06.003331+0000","last_change":"2024-09-13T11:23:32.368856+0000","last_active":"2024-09-13T11:24:06.003331+0000","last_peered":"2024-09-13T11:24:06.003331+0000","last_clean":"2024-09-13T11:24:06.003331+0000","last_became_active":"2024-09-13T11:23:31.925869+0000","last_became_peered":"2024-09-13T11:23:31.925869+0000","last_unstale":"2024-09-13T11:24:06.003331+0000","last_undegraded":"2024-09-13T11:24:06.003331+0000","last_fullsized":"2024-09-13T11:24:06.003331+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'3527","last_scrub_stamp":"2024-09-13T11:23:32.368727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:32.368727+0000","objects_scrubbed":1774,"log_size":4435,"log_dups_size":0,"ondisk_log_size":4435,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:24:57.811779+0000","scrub_duration":219,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":175833088,"num_objects":2683,"num_object_clones":0,"num_object_copies":5366,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2683,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":909,"num_write_kb":58176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"104'4198","reported_seq":4390,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-13T11:24:02.002768+0000","last_change":"2024-09-13T11:23:33.492630+0000","last_active":"2024-09-13T11:24:02.002768+0000","last_peered":"2024-09-13T11:24:02.002768+0000","last_clean":"2024-09-13T11:24:02.002768+0000","last_became_active":"2024-09-13T11:23:31.935239+0000","last_became_peered":"2024-09-13T11:23:31.935239+0000","last_unstale":"2024-09-13T11:24:02.002768+0000","last_undegraded":"2024-09-13T11:24:02.002768+0000","last_fullsized":"2024-09-13T11:24:02.002768+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'3420","last_scrub_stamp":"2024-09-13T11:23:33.492488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:33.492488+0000","objects_scrubbed":1776,"log_size":4198,"log_dups_size":0,"ondisk_log_size":4198,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:24:44.851261+0000","scrub_duration":215,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":167378944,"num_objects":2554,"num_object_clones":0,"num_object_copies":5108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2554,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":778,"num_write_kb":49792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,12],"acting":[11,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.10","version":"108'4131","reported_seq":8487,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T11:24:06.004696+0000","last_change":"2024-09-13T11:23:48.519654+0000","last_active":"2024-09-13T11:24:06.004696+0000","last_peered":"2024-09-13T11:24:06.004696+0000","last_clean":"2024-09-13T11:24:06.004696+0000","last_became_active":"2024-09-13T11:23:41.931183+0000","last_became_peered":"2024-09-13T11:23:41.931183+0000","last_unstale":"2024-09-13T11:24:06.004696+0000","last_undegraded":"2024-09-13T11:24:06.004696+0000","last_fullsized":"2024-09-13T11:24:06.004696+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":5,"last_scrub":"91'3468","last_scrub_stamp":"2024-09-13T11:23:48.519574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:48.519574+0000","objects_scrubbed":2031,"log_size":4131,"log_dups_size":0,"ondisk_log_size":4131,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:24:57.342213+0000","scrub_duration":215,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":176685056,"num_objects":2696,"num_object_clones":0,"num_object_copies":5392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2696,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":665,"num_write_kb":42560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.11","version":"108'4005","reported_seq":7139,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T11:24:06.005767+0000","last_change":"2024-09-13T11:23:46.102466+0000","last_active":"2024-09-13T11:24:06.005767+0000","last_peered":"2024-09-13T11:24:06.005767+0000","last_clean":"2024-09-13T11:24:06.005767+0000","last_became_active":"2024-09-13T11:23:39.063172+0000","last_became_peered":"2024-09-13T11:23:39.063172+0000","last_unstale":"2024-09-13T11:24:06.005767+0000","last_undegraded":"2024-09-13T11:24:06.005767+0000","last_fullsized":"2024-09-13T11:24:06.005767+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":5,"last_scrub":"88'3345","last_scrub_stamp":"2024-09-13T11:23:46.101973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:46.101973+0000","objects_scrubbed":2011,"log_size":4005,"log_dups_size":0,"ondisk_log_size":4005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:08.671350+0000","scrub_duration":955,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":175308800,"num_objects":2675,"num_object_clones":0,"num_object_copies":5350,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2675,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":664,"num_write_kb":42496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11],"acting":[4,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.12","version":"108'4004","reported_seq":7062,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T11:24:06.047245+0000","last_change":"2024-09-13T11:23:35.111409+0000","last_active":"2024-09-13T11:24:06.047245+0000","last_peered":"2024-09-13T11:24:06.047245+0000","last_clean":"2024-09-13T11:24:06.047245+0000","last_became_active":"2024-09-13T11:23:21.998877+0000","last_became_peered":"2024-09-13T11:23:21.998877+0000","last_unstale":"2024-09-13T11:24:06.047245+0000","last_undegraded":"2024-09-13T11:24:06.047245+0000","last_fullsized":"2024-09-13T11:24:06.047245+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'3109","last_scrub_stamp":"2024-09-13T11:23:35.111320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:35.111320+0000","objects_scrubbed":1732,"log_size":4004,"log_dups_size":0,"ondisk_log_size":4004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:03.371433+0000","scrub_duration":239,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":172163072,"num_objects":2627,"num_object_clones":0,"num_object_copies":5254,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2627,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":895,"num_write_kb":57280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,7],"acting":[14,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"3.13","version":"104'3963","reported_seq":7013,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-13T11:24:02.020450+0000","last_change":"2024-09-13T11:23:47.565094+0000","last_active":"2024-09-13T11:24:02.020450+0000","last_peered":"2024-09-13T11:24:02.020450+0000","last_clean":"2024-09-13T11:24:02.020450+0000","last_became_active":"2024-09-13T11:23:41.082433+0000","last_became_peered":"2024-09-13T11:23:41.082433+0000","last_unstale":"2024-09-13T11:24:02.020450+0000","last_undegraded":"2024-09-13T11:24:02.020450+0000","last_fullsized":"2024-09-13T11:24:02.020450+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'3408","last_scrub_stamp":"2024-09-13T11:23:47.564931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:47.564931+0000","objects_scrubbed":1981,"log_size":3963,"log_dups_size":0,"ondisk_log_size":3963,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:04.258231+0000","scrub_duration":471,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":166985728,"num_objects":2548,"num_object_clones":0,"num_object_copies":5096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":567,"num_write_kb":36288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7],"acting":[11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":36,"stat_sum":{"num_bytes":5479202816,"num_objects":83607,"num_object_clones":0,"num_object_copies":167214,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":83607,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":37078,"num_write_kb":2372928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11626,"num_bytes_recovered":761888768,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":13112246272,"data_stored":13112246272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":144053,"ondisk_log_size":144053,"up":72,"acting":72,"num_store_stats":16},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":10},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":112,"ondisk_log_size":112,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":13,"up_from":23,"seq":98784247834,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1106512,"kb_used_data":1035072,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":93265328,"statfs":{"total":96636764160,"available":95503695872,"internally_reserved":0,"allocated":1059913728,"data_stored":1059453231,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73136581},"hb_peers":[0,1,3,4,6,8,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":9,"up_from":23,"seq":98784247833,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":427964,"kb_used_data":392620,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93943876,"statfs":{"total":96636764160,"available":96198529024,"internally_reserved":0,"allocated":402042880,"data_stored":401601626,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,8,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":40,"seq":171798691857,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1343948,"kb_used_data":1267516,"kb_used_omap":4,"kb_used_meta":76411,"kb_avail":93027892,"statfs":{"total":96636764160,"available":95260561408,"internally_reserved":0,"allocated":1297936384,"data_stored":1297445342,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4433,"internal_metadata":78245551},"hb_peers":[0,2,3,6,8,9,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":21,"seq":90194313242,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1019848,"kb_used_data":966456,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":93351992,"statfs":{"total":96636764160,"available":95592439808,"internally_reserved":0,"allocated":989650944,"data_stored":989161893,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":54655431},"hb_peers":[0,2,4,7,9,10,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":21,"seq":90194313245,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":286288,"kb_used_data":259264,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94085552,"statfs":{"total":96636764160,"available":96343605248,"internally_reserved":0,"allocated":265486336,"data_stored":264993109,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,6,8,9,10,11,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":22,"seq":94489280537,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1149000,"kb_used_data":1085880,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":93222840,"statfs":{"total":96636764160,"available":95460188160,"internally_reserved":0,"allocated":1111941120,"data_stored":1111516713,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[1,3,4,5,8,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":22,"seq":94489280538,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":445056,"kb_used_data":405552,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":93926784,"statfs":{"total":96636764160,"available":96181026816,"internally_reserved":0,"allocated":415285248,"data_stored":414812191,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,3,7,8,9,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":15,"up_from":22,"seq":94489280538,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":949184,"kb_used_data":873584,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":93422656,"statfs":{"total":96636764160,"available":95664799744,"internally_reserved":0,"allocated":894550016,"data_stored":894133583,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":77396422},"hb_peers":[0,3,4,5,6,7,10,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":22,"seq":94489280538,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":546264,"kb_used_data":497032,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":93825576,"statfs":{"total":96636764160,"available":96077389824,"internally_reserved":0,"allocated":508960768,"data_stored":508465062,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[3,5,7,9,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":22,"seq":94489280538,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":429992,"kb_used_data":394648,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93941848,"statfs":{"total":96636764160,"available":96196452352,"internally_reserved":0,"allocated":404119552,"data_stored":403666123,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,3,5,7,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":22,"seq":94489280537,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1205024,"kb_used_data":1137744,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":93166816,"statfs":{"total":96636764160,"available":95402819584,"internally_reserved":0,"allocated":1165049856,"data_stored":1164630327,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":68876746},"hb_peers":[0,3,5,6,8,9,10,11,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":22,"seq":94489280538,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":489692,"kb_used_data":458508,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93882148,"statfs":{"total":96636764160,"available":96135319552,"internally_reserved":0,"allocated":469512192,"data_stored":469008420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,5,7,9,10,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":22,"seq":94489280537,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":699360,"kb_used_data":641808,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":93672480,"statfs":{"total":96636764160,"available":95920619520,"internally_reserved":0,"allocated":657211392,"data_stored":656765555,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":58915264},"hb_peers":[0,1,3,6,7,9,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":11,"up_from":22,"seq":94489280538,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1348552,"kb_used_data":1272952,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":93023288,"statfs":{"total":96636764160,"available":95255846912,"internally_reserved":0,"allocated":1303502848,"data_stored":1303053612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":77396420},"hb_peers":[0,2,4,7,8,10,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":12,"up_from":22,"seq":94489280538,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1116988,"kb_used_data":1053868,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":93254852,"statfs":{"total":96636764160,"available":95492968448,"internally_reserved":0,"allocated":1079160832,"data_stored":1078688509,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[0,1,2,3,5,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":14,"up_from":22,"seq":94489280538,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1184908,"kb_used_data":1113468,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":93186932,"statfs":{"total":96636764160,"available":95423418368,"internally_reserved":0,"allocated":1140191232,"data_stored":1139708545,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73136581},"hb_peers":[0,1,5,6,7,8,9,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1123614720,"data_stored":1123614720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1298071552,"data_stored":1298071552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":419233792,"data_stored":419233792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":987889664,"data_stored":987889664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":514981888,"data_stored":514981888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":264568832,"data_stored":264568832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":405602304,"data_stored":405602304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1162280960,"data_stored":1162280960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":466944000,"data_stored":466944000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":403898368,"data_stored":403898368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":663617536,"data_stored":663617536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":1253244928,"data_stored":1253244928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":1081606144,"data_stored":1081606144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":1027538944,"data_stored":1027538944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":1139933184,"data_stored":1139933184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":899219456,"data_stored":899219456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T11:24:06.826 INFO:tasks.ceph.osd.8.smithi001.stderr:2024-09-13T11:24:06.819+0000 7fb9b48ae640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26312) UID: 0 2024-09-13T11:24:06.886 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-09-13T11:24:06.886 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:24:06.900 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-13T11:24:06.928 INFO:tasks.ceph.osd.5.smithi092.stderr:2024-09-13T11:24:06.923+0000 7f6f0b705640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26346) UID: 0 2024-09-13T11:24:06.949 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-09-13T11:24:06.945+0000 7f97b59fa640 -1 osd.6 108 handle_osd_map injecting map crc failure 2024-09-13T11:24:06.960 INFO:tasks.ceph.osd.14.smithi162.stderr:2024-09-13T11:24:06.957+0000 7f2350b08640 -1 osd.14 108 handle_osd_map injecting map crc failure 2024-09-13T11:24:06.967 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-13T11:24:06.988 INFO:tasks.ceph.osd.2.smithi162.stderr:2024-09-13T11:24:06.985+0000 7fe921d8b640 -1 osd.2 108 handle_osd_map injecting map crc failure 2024-09-13T11:24:07.007 INFO:tasks.radosbench.radosbench.0.smithi177.stdout:2024-09-13T11:24:07.007762+0000 min lat: 0.00139681 max lat: 22.1835 avg lat: 0.0187332 2024-09-13T11:24:07.008 INFO:tasks.radosbench.radosbench.0.smithi177.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-13T11:24:07.008 INFO:tasks.radosbench.radosbench.0.smithi177.stdout: 100 16 85267 85251 53.2764 21.625 0.00550726 0.0187332 2024-09-13T11:24:07.029 INFO:tasks.ceph.osd.13.smithi092.stderr:2024-09-13T11:24:07.027+0000 7f3e5afea640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26350) UID: 0 2024-09-13T11:24:07.052 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-09-13T11:24:07.052 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:24:07.074 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-13T11:24:07.130 INFO:tasks.ceph.osd.14.smithi162.stderr:2024-09-13T11:24:07.129+0000 7f235c0db640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26319) UID: 0 2024-09-13T11:24:07.231 INFO:tasks.ceph.osd.15.smithi177.stderr:2024-09-13T11:24:07.227+0000 7f9d6ce77640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26578) UID: 0 2024-09-13T11:24:07.256 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-09-13T11:24:07.256 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:24:07.280 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-13T11:24:07.331 INFO:tasks.ceph.osd.2.smithi162.stderr:2024-09-13T11:24:07.329+0000 7fe92b35a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26313) UID: 0 2024-09-13T11:24:07.433 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-09-13T11:24:07.429+0000 7f97befc9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26315) UID: 0 2024-09-13T11:24:07.478 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-09-13T11:24:07.478 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:24:07.502 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-09-13T11:24:07.533 INFO:tasks.ceph.osd.1.smithi092.stderr:2024-09-13T11:24:07.531+0000 7fe7789c5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33718) UID: 0 2024-09-13T11:24:07.634 INFO:tasks.ceph.osd.1.smithi092.stderr:2024-09-13T11:24:07.631+0000 7fe7789c5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33718) UID: 0 2024-09-13T11:24:07.646 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-09-13T11:24:07.646 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:24:07.667 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-09-13T11:24:07.736 INFO:tasks.ceph.osd.13.smithi092.stderr:2024-09-13T11:24:07.731+0000 7f3e5afea640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26350) UID: 0 2024-09-13T11:24:07.800 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-09-13T11:24:07.801 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:24:07.820 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-09-13T11:24:07.864 INFO:tasks.ceph.osd.7.smithi177.stderr:2024-09-13T11:24:07.863+0000 7f897b51d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26575) UID: 0 2024-09-13T11:24:07.938 INFO:tasks.ceph.osd.10.smithi162.stderr:2024-09-13T11:24:07.937+0000 7fb59c9f0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26317) UID: 0 2024-09-13T11:24:07.956 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-09-13T11:24:07.957 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:24:07.960 INFO:teuthology.orchestra.run.smithi001.stderr:noscrub is unset 2024-09-13T11:24:07.961 INFO:tasks.ceph.osd.1.smithi092.stderr:2024-09-13T11:24:07.955+0000 7fe76d3f2640 -1 osd.1 109 handle_osd_map injecting map crc failure 2024-09-13T11:24:07.975 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_ops_in_flight 2024-09-13T11:24:08.008 INFO:tasks.radosbench.radosbench.0.smithi177.stdout: 101 16 87070 87054 53.8646 112.688 0.0272284 0.0185587 2024-09-13T11:24:08.052 INFO:tasks.ceph.osd.7.smithi177.stderr:2024-09-13T11:24:08.051+0000 7f897b51d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26575) UID: 0 2024-09-13T11:24:08.140 INFO:tasks.ceph.osd.3.smithi177.stderr:2024-09-13T11:24:08.135+0000 7f10adba4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26574) UID: 0 2024-09-13T11:24:08.144 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-09-13T11:24:08.144 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:24:08.164 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_blocked_ops 2024-09-13T11:24:08.252 INFO:tasks.ceph.osd.15.smithi177.stderr:2024-09-13T11:24:08.251+0000 7f9d6ce77640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26578) UID: 0 2024-09-13T11:24:08.342 INFO:tasks.ceph.osd.5.smithi092.stderr:2024-09-13T11:24:08.339+0000 7f6f0b705640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26346) UID: 0 2024-09-13T11:24:08.343 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-09-13T11:24:08.344 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:24:08.364 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_historic_ops 2024-09-13T11:24:08.442 INFO:tasks.ceph.osd.10.smithi162.stderr:2024-09-13T11:24:08.441+0000 7fb59c9f0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26317) UID: 0 2024-09-13T11:24:08.501 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T11:24:08.532 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-09-13T11:24:08.532 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:24:08.543 INFO:tasks.ceph.osd.1.smithi092.stderr:2024-09-13T11:24:08.539+0000 7fe7789c5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33718) UID: 0 2024-09-13T11:24:08.548 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-13T11:24:08.644 INFO:tasks.ceph.osd.7.smithi177.stderr:2024-09-13T11:24:08.643+0000 7f897b51d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26575) UID: 0 2024-09-13T11:24:08.691 INFO:teuthology.orchestra.run.smithi001.stdout:osd.0: {} 2024-09-13T11:24:08.692 INFO:teuthology.orchestra.run.smithi001.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T11:24:08.699 INFO:teuthology.orchestra.run.smithi092.stdout:ERROR: (22) Invalid argument 2024-09-13T11:24:08.699 INFO:teuthology.orchestra.run.smithi092.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:24:08.717 INFO:teuthology.orchestra.run.smithi001.stdout:osd.1: {} 2024-09-13T11:24:08.717 INFO:teuthology.orchestra.run.smithi001.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T11:24:08.720 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-13T11:24:08.744 INFO:tasks.ceph.osd.14.smithi162.stderr:2024-09-13T11:24:08.741+0000 7f235c0db640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26319) UID: 0 2024-09-13T11:24:08.746 INFO:teuthology.orchestra.run.smithi001.stdout:osd.2: {} 2024-09-13T11:24:08.746 INFO:teuthology.orchestra.run.smithi001.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T11:24:08.774 INFO:teuthology.orchestra.run.smithi001.stdout:osd.3: {} 2024-09-13T11:24:08.774 INFO:teuthology.orchestra.run.smithi001.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T11:24:08.800 INFO:teuthology.orchestra.run.smithi001.stdout:osd.4: {} 2024-09-13T11:24:08.800 INFO:teuthology.orchestra.run.smithi001.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T11:24:08.826 INFO:teuthology.orchestra.run.smithi001.stdout:osd.5: {} 2024-09-13T11:24:08.826 INFO:teuthology.orchestra.run.smithi001.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T11:24:08.845 INFO:tasks.ceph.osd.1.smithi092.stderr:2024-09-13T11:24:08.839+0000 7fe7789c5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33718) UID: 0 2024-09-13T11:24:08.855 INFO:teuthology.orchestra.run.smithi001.stdout:osd.6: {} 2024-09-13T11:24:08.855 INFO:teuthology.orchestra.run.smithi001.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T11:24:08.865 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-09-13T11:24:08.865 INFO:teuthology.orchestra.run.smithi092.stdout: "ops": [], 2024-09-13T11:24:08.865 INFO:teuthology.orchestra.run.smithi092.stdout: "complaint_time": 30, 2024-09-13T11:24:08.865 INFO:teuthology.orchestra.run.smithi092.stdout: "num_blocked_ops": 0 2024-09-13T11:24:08.865 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-09-13T11:24:08.881 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-13T11:24:08.882 INFO:teuthology.orchestra.run.smithi001.stdout:osd.7: {} 2024-09-13T11:24:08.882 INFO:teuthology.orchestra.run.smithi001.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T11:24:08.907 INFO:teuthology.orchestra.run.smithi001.stdout:osd.8: {} 2024-09-13T11:24:08.907 INFO:teuthology.orchestra.run.smithi001.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-13T11:24:08.931 INFO:teuthology.orchestra.run.smithi001.stdout:osd.9: {} 2024-09-13T11:24:08.931 INFO:teuthology.orchestra.run.smithi001.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-13T11:24:08.946 INFO:tasks.ceph.osd.12.smithi001.stderr:2024-09-13T11:24:08.939+0000 7f892c6e6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26314) UID: 0 2024-09-13T11:24:08.956 INFO:teuthology.orchestra.run.smithi001.stdout:osd.10: {} 2024-09-13T11:24:08.956 INFO:teuthology.orchestra.run.smithi001.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-13T11:24:08.982 INFO:teuthology.orchestra.run.smithi001.stdout:osd.11: {} 2024-09-13T11:24:08.982 INFO:teuthology.orchestra.run.smithi001.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-13T11:24:09.008 INFO:tasks.radosbench.radosbench.0.smithi177.stdout: 102 16 88797 88781 54.3946 107.938 0.00258817 0.018376 2024-09-13T11:24:09.010 INFO:teuthology.orchestra.run.smithi001.stdout:osd.12: {} 2024-09-13T11:24:09.010 INFO:teuthology.orchestra.run.smithi001.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-13T11:24:09.037 INFO:teuthology.orchestra.run.smithi001.stdout:osd.13: {} 2024-09-13T11:24:09.037 INFO:teuthology.orchestra.run.smithi001.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-13T11:24:09.046 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-13T11:24:09.039+0000 7f6a68775640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26307) UID: 0 2024-09-13T11:24:09.060 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-09-13T11:24:09.060 INFO:teuthology.orchestra.run.smithi092.stdout: "size": 20, 2024-09-13T11:24:09.060 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 600, 2024-09-13T11:24:09.060 INFO:teuthology.orchestra.run.smithi092.stdout: "ops": [ 2024-09-13T11:24:09.060 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:09.060 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:3 3.4 3.125e09e4 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e40)", 2024-09-13T11:24:09.061 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:46.304641+0000", 2024-09-13T11:24:09.061 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 82.754016250000006, 2024-09-13T11:24:09.061 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.374372119, 2024-09-13T11:24:09.061 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:24:09.061 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:24:09.061 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "reached pg", 2024-09-13T11:24:09.061 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:24:09.061 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:24:09.061 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:09.061 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 3 2024-09-13T11:24:09.061 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:09.061 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:24:09.062 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:09.062 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:24:09.062 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.304641+0000", 2024-09-13T11:24:09.062 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:09.062 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:09.062 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:09.062 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:24:09.062 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.304641+0000", 2024-09-13T11:24:09.062 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:09.062 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:09.062 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:09.062 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:24:09.062 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.304644+0000", 2024-09-13T11:24:09.063 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.5130000000000001e-06 2024-09-13T11:24:09.063 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:09.063 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:09.063 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:24:09.063 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305058+0000", 2024-09-13T11:24:09.063 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000414277 2024-09-13T11:24:09.063 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:09.063 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:09.063 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:24:09.063 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305124+0000", 2024-09-13T11:24:09.063 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.5813000000000002e-05 2024-09-13T11:24:09.063 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:09.063 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:09.063 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:24:09.064 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305156+0000", 2024-09-13T11:24:09.064 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.1489000000000003e-05 2024-09-13T11:24:09.064 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:09.064 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:09.064 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:24:09.064 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.678939+0000", 2024-09-13T11:24:09.064 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.37378339999999999 2024-09-13T11:24:09.064 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:09.064 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:09.064 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:24:09.064 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.679013+0000", 2024-09-13T11:24:09.064 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.3627000000000005e-05 2024-09-13T11:24:09.065 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:09.065 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:24:09.065 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:09.065 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:09.065 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:09.065 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:29 3.4 3.b7bbfbe4 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-09-13T11:24:09.065 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:46.305215+0000", 2024-09-13T11:24:09.065 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 82.753441946999999, 2024-09-13T11:24:09.065 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.37387964600000001, 2024-09-13T11:24:09.065 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:24:09.065 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:24:09.065 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "reached pg", 2024-09-13T11:24:09.065 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:24:09.066 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:24:09.066 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:09.066 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 29 2024-09-13T11:24:09.066 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:09.066 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:24:09.066 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:09.066 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:24:09.066 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305215+0000", 2024-09-13T11:24:09.066 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:09.066 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:09.066 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:09.066 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:24:09.066 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305215+0000", 2024-09-13T11:24:09.066 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:09.067 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:09.067 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:09.067 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:24:09.067 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305218+0000", 2024-09-13T11:24:09.067 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.7030000000000002e-06 2024-09-13T11:24:09.067 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:09.067 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:09.067 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:24:09.067 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305366+0000", 2024-09-13T11:24:09.067 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00014825400000000001 2024-09-13T11:24:09.067 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:09.067 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:09.067 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:24:09.068 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305386+0000", 2024-09-13T11:24:09.068 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.9689000000000001e-05 2024-09-13T11:24:09.068 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:09.068 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:09.068 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:24:09.068 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305403+0000", 2024-09-13T11:24:09.068 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.7600000000000001e-05 2024-09-13T11:24:09.068 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:09.068 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:09.068 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:24:09.068 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.679067+0000", 2024-09-13T11:24:09.068 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.37366339100000001 2024-09-13T11:24:09.068 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:09.069 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:09.069 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:24:09.069 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.679095+0000", 2024-09-13T11:24:09.069 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.8008999999999999e-05 2024-09-13T11:24:09.069 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:09.069 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:24:09.069 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:09.069 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:09.069 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:09.069 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:30 3.4 3.547ec0f4 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-09-13T11:24:09.069 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:46.305453+0000", 2024-09-13T11:24:09.069 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 82.753203823000007, 2024-09-13T11:24:09.070 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.373764914, 2024-09-13T11:24:09.070 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:24:09.070 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:24:09.070 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "reached pg", 2024-09-13T11:24:09.070 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:24:09.070 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:24:09.070 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:09.070 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 30 2024-09-13T11:24:09.070 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:09.070 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:24:09.070 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:09.070 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:24:09.071 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305453+0000", 2024-09-13T11:24:09.071 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:09.071 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:09.071 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:09.071 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:24:09.071 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305453+0000", 2024-09-13T11:24:09.071 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:09.071 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:09.071 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:09.071 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:24:09.071 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305456+0000", 2024-09-13T11:24:09.071 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.9560000000000002e-06 2024-09-13T11:24:09.071 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:09.072 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:09.072 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:24:09.072 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305597+0000", 2024-09-13T11:24:09.072 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00014122700000000001 2024-09-13T11:24:09.072 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:09.072 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:09.072 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:24:09.072 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305615+0000", 2024-09-13T11:24:09.072 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.8139999999999999e-05 2024-09-13T11:24:09.072 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:09.072 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:09.072 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:24:09.072 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305632+0000", 2024-09-13T11:24:09.072 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.6101999999999999e-05 2024-09-13T11:24:09.073 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:09.073 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:09.073 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:24:09.073 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.679195+0000", 2024-09-13T11:24:09.073 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.373563486 2024-09-13T11:24:09.073 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:09.073 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:09.073 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:24:09.073 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.679218+0000", 2024-09-13T11:24:09.073 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.3003e-05 2024-09-13T11:24:09.073 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:09.073 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:24:09.073 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:09.074 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:09.074 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:09.074 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:70 3.4 3.1404a604 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-09-13T11:24:09.074 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:46.305665+0000", 2024-09-13T11:24:09.074 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 82.752991675000004, 2024-09-13T11:24:09.074 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.37365550199999997, 2024-09-13T11:24:09.074 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:24:09.074 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:24:09.074 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "reached pg", 2024-09-13T11:24:09.074 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:24:09.074 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:24:09.074 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:09.074 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 70 2024-09-13T11:24:09.075 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:09.075 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:24:09.075 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:09.075 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:24:09.075 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305665+0000", 2024-09-13T11:24:09.075 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:09.075 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:09.075 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:09.075 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:24:09.075 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305665+0000", 2024-09-13T11:24:09.075 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:09.075 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:09.075 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:09.076 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:24:09.076 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305668+0000", 2024-09-13T11:24:09.076 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.5890000000000001e-06 2024-09-13T11:24:09.076 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:09.076 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:09.076 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:24:09.076 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305791+0000", 2024-09-13T11:24:09.076 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000123545 2024-09-13T11:24:09.076 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:09.076 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:09.076 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:24:09.076 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305808+0000", 2024-09-13T11:24:09.076 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.668e-05 2024-09-13T11:24:09.076 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:09.077 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:09.077 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:24:09.077 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305824+0000", 2024-09-13T11:24:09.077 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.6385e-05 2024-09-13T11:24:09.077 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:09.077 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:09.077 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:24:09.077 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.679305+0000", 2024-09-13T11:24:09.077 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.37348059900000002 2024-09-13T11:24:09.077 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:09.077 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:09.077 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:24:09.077 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.679321+0000", 2024-09-13T11:24:09.078 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.5704000000000001e-05 2024-09-13T11:24:09.078 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:09.078 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:24:09.078 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:09.078 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:09.078 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:09.078 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:25408 3.4 3:294e23bd:::benchmark_data_smithi177_31116_object25407:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e56)", 2024-09-13T11:24:09.078 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:04.495455+0000", 2024-09-13T11:24:09.078 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 64.563202023000002, 2024-09-13T11:24:09.078 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.29897367600000002, 2024-09-13T11:24:09.078 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:24:09.079 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:24:09.079 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:09.079 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:24:09.079 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:24:09.079 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:09.079 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 25408 2024-09-13T11:24:09.079 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:09.079 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:24:09.079 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:09.079 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:24:09.079 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.495455+0000", 2024-09-13T11:24:09.079 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:09.079 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:09.080 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:09.080 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:24:09.080 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.495455+0000", 2024-09-13T11:24:09.080 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:09.080 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:09.080 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:09.080 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:24:09.080 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.495459+0000", 2024-09-13T11:24:09.080 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.5630000000000002e-06 2024-09-13T11:24:09.080 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:09.080 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:09.080 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:24:09.080 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.495545+0000", 2024-09-13T11:24:09.080 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.5704000000000005e-05 2024-09-13T11:24:09.081 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:09.081 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:09.081 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:24:09.081 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.495581+0000", 2024-09-13T11:24:09.081 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.5689000000000003e-05 2024-09-13T11:24:09.081 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:09.081 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:09.081 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:24:09.081 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.495597+0000", 2024-09-13T11:24:09.081 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.6053e-05 2024-09-13T11:24:09.081 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:09.081 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:09.081 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:24:09.082 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.495721+0000", 2024-09-13T11:24:09.082 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00012379099999999999 2024-09-13T11:24:09.082 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:09.082 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:09.082 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:24:09.082 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.495940+0000", 2024-09-13T11:24:09.082 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00021967699999999999 2024-09-13T11:24:09.082 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:09.082 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:09.082 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 12", 2024-09-13T11:24:09.082 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.496162+0000", 2024-09-13T11:24:09.082 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000221917 2024-09-13T11:24:09.082 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:09.083 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:09.083 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:24:09.083 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.498699+0000", 2024-09-13T11:24:09.083 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0025366690000000001 2024-09-13T11:24:09.083 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:09.083 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:09.083 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:09.083 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.794139+0000", 2024-09-13T11:24:09.083 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.295440025 2024-09-13T11:24:09.083 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:09.083 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:09.083 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:24:09.083 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.794306+0000", 2024-09-13T11:24:09.084 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00016689799999999999 2024-09-13T11:24:09.084 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:09.084 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:09.084 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:24:09.084 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.794429+0000", 2024-09-13T11:24:09.084 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00012269000000000001 2024-09-13T11:24:09.084 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:09.084 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:24:09.084 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:09.084 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:09.084 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:09.084 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:49963 3.18 3:19b86b73:::benchmark_data_smithi177_31116_object49962:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e72)", 2024-09-13T11:24:09.084 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:26.307557+0000", 2024-09-13T11:24:09.085 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 42.751100047999998, 2024-09-13T11:24:09.085 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.22140597300000001, 2024-09-13T11:24:09.085 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:24:09.085 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:24:09.085 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:09.085 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:24:09.085 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:24:09.085 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:09.085 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 49963 2024-09-13T11:24:09.085 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:09.085 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:24:09.085 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:09.085 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:24:09.085 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:26.307557+0000", 2024-09-13T11:24:09.086 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:09.086 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:09.086 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:09.086 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:24:09.086 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:26.307557+0000", 2024-09-13T11:24:09.086 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:09.086 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:09.086 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:09.086 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:24:09.086 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:26.307558+0000", 2024-09-13T11:24:09.086 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.3740000000000001e-06 2024-09-13T11:24:09.086 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:09.086 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:09.087 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:24:09.087 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:26.516580+0000", 2024-09-13T11:24:09.087 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.209021596 2024-09-13T11:24:09.087 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:09.087 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:09.087 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:24:09.087 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:26.516614+0000", 2024-09-13T11:24:09.087 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.3858e-05 2024-09-13T11:24:09.087 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:09.087 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:09.087 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:24:09.087 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:26.516628+0000", 2024-09-13T11:24:09.087 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.4715999999999999e-05 2024-09-13T11:24:09.087 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:09.088 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:09.088 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:24:09.088 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:26.516790+0000", 2024-09-13T11:24:09.088 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00016138 2024-09-13T11:24:09.088 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:09.088 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:09.088 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:24:09.088 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:26.517123+0000", 2024-09-13T11:24:09.088 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00033296600000000002 2024-09-13T11:24:09.088 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:09.088 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:09.088 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 0,10", 2024-09-13T11:24:09.088 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:26.517367+0000", 2024-09-13T11:24:09.089 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00024416900000000003 2024-09-13T11:24:09.089 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:09.089 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:09.089 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:24:09.089 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:26.526939+0000", 2024-09-13T11:24:09.089 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0095718650000000006 2024-09-13T11:24:09.089 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:09.089 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:09.089 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:09.089 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:26.527095+0000", 2024-09-13T11:24:09.089 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00015643800000000001 2024-09-13T11:24:09.089 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:09.089 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:09.090 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:09.090 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:26.527254+0000", 2024-09-13T11:24:09.090 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000158383 2024-09-13T11:24:09.090 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:09.090 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:09.090 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:24:09.090 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:26.527405+0000", 2024-09-13T11:24:09.090 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000151731 2024-09-13T11:24:09.090 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:09.090 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:09.090 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:24:09.090 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:26.528963+0000", 2024-09-13T11:24:09.090 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0015574969999999999 2024-09-13T11:24:09.090 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:09.091 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:24:09.091 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:09.091 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:09.091 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:09.091 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "MOSDPGPushReply(3.14 75/70 [PushReplyOp(3:2f14928a:::benchmark_data_smithi177_31116_object15123:head)])", 2024-09-13T11:24:09.091 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:31.569662+0000", 2024-09-13T11:24:09.091 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 37.488995250999999, 2024-09-13T11:24:09.091 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.35759148299999999, 2024-09-13T11:24:09.091 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:24:09.091 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:24:09.091 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "reached pg", 2024-09-13T11:24:09.091 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:24:09.091 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:09.092 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:24:09.092 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:31.569662+0000", 2024-09-13T11:24:09.092 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:09.092 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:09.092 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:09.092 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:24:09.092 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:31.569662+0000", 2024-09-13T11:24:09.092 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:09.092 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:09.092 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:09.092 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:24:09.092 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:31.569663+0000", 2024-09-13T11:24:09.092 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.333e-06 2024-09-13T11:24:09.093 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:09.093 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:09.093 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:24:09.093 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:31.569667+0000", 2024-09-13T11:24:09.093 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.8929999999999998e-06 2024-09-13T11:24:09.093 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:09.093 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:09.093 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:24:09.093 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:31.923221+0000", 2024-09-13T11:24:09.093 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.35355413699999999 2024-09-13T11:24:09.093 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:09.093 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:09.093 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:24:09.093 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:31.923230+0000", 2024-09-13T11:24:09.094 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.6829999999999997e-06 2024-09-13T11:24:09.094 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:09.094 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:09.094 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:24:09.094 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:31.923807+0000", 2024-09-13T11:24:09.094 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00057686800000000002 2024-09-13T11:24:09.094 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:09.094 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:09.094 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:24:09.094 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:31.927253+0000", 2024-09-13T11:24:09.094 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.003446569 2024-09-13T11:24:09.094 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:09.094 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:24:09.095 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:09.095 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:09.095 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:09.095 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "MOSDPGPushReply(3.14 76/70 [PushReplyOp(3:297981b4:::benchmark_data_smithi177_31116_object16264:head)])", 2024-09-13T11:24:09.095 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:32.122153+0000", 2024-09-13T11:24:09.095 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 36.936503561999999, 2024-09-13T11:24:09.095 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.20534369899999999, 2024-09-13T11:24:09.095 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:24:09.095 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:24:09.095 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "reached pg", 2024-09-13T11:24:09.095 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:24:09.095 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:09.095 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:24:09.096 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:32.122153+0000", 2024-09-13T11:24:09.096 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:09.096 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:09.096 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:09.096 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:24:09.096 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:32.122153+0000", 2024-09-13T11:24:09.096 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:09.096 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:09.096 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:09.096 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:24:09.096 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:32.122154+0000", 2024-09-13T11:24:09.096 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.5300000000000003e-07 2024-09-13T11:24:09.096 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:09.096 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:09.097 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:24:09.097 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:32.122158+0000", 2024-09-13T11:24:09.097 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.8770000000000003e-06 2024-09-13T11:24:09.097 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:09.097 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:09.097 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:24:09.097 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:32.326042+0000", 2024-09-13T11:24:09.097 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.20388401 2024-09-13T11:24:09.097 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:09.097 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:09.097 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:24:09.097 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:32.326049+0000", 2024-09-13T11:24:09.098 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.8369999999999999e-06 2024-09-13T11:24:09.098 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:09.098 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:09.098 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:24:09.098 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:32.326135+0000", 2024-09-13T11:24:09.098 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.6215e-05 2024-09-13T11:24:09.098 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:09.098 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:09.098 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:24:09.098 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:32.327497+0000", 2024-09-13T11:24:09.098 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0013619070000000001 2024-09-13T11:24:09.099 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:09.099 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:24:09.099 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:09.099 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:09.099 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:09.099 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:55670 3.14 3:2e43484c:::benchmark_data_smithi177_31116_object55669:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e79)", 2024-09-13T11:24:09.099 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:35.763374+0000", 2024-09-13T11:24:09.099 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 33.295283339000001, 2024-09-13T11:24:09.099 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.18805201299999999, 2024-09-13T11:24:09.099 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:24:09.099 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:24:09.099 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:09.099 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:24:09.100 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:24:09.100 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:09.100 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 55670 2024-09-13T11:24:09.100 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:09.100 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:24:09.100 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:09.100 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:24:09.100 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:35.763374+0000", 2024-09-13T11:24:09.100 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:09.100 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:09.101 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:09.101 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:24:09.101 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:35.763374+0000", 2024-09-13T11:24:09.101 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:09.101 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:09.101 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:09.101 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:24:09.101 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:35.763376+0000", 2024-09-13T11:24:09.101 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.1239999999999999e-06 2024-09-13T11:24:09.101 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:09.101 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:09.101 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:24:09.101 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:35.763428+0000", 2024-09-13T11:24:09.101 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.2553000000000003e-05 2024-09-13T11:24:09.102 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:09.102 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:09.102 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:24:09.102 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:35.763437+0000", 2024-09-13T11:24:09.102 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.4649999999999994e-06 2024-09-13T11:24:09.102 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:09.102 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:09.102 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:24:09.102 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:35.763442+0000", 2024-09-13T11:24:09.102 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.5310000000000001e-06 2024-09-13T11:24:09.102 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:09.102 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:09.102 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:24:09.102 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:35.763498+0000", 2024-09-13T11:24:09.102 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.5591999999999999e-05 2024-09-13T11:24:09.103 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:09.103 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:09.103 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:24:09.103 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:35.763660+0000", 2024-09-13T11:24:09.103 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000161965 2024-09-13T11:24:09.103 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:09.103 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:09.103 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 3,14", 2024-09-13T11:24:09.103 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:35.763850+0000", 2024-09-13T11:24:09.103 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00019013999999999999 2024-09-13T11:24:09.103 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:09.103 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:09.103 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:24:09.103 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:35.768339+0000", 2024-09-13T11:24:09.104 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0044892739999999997 2024-09-13T11:24:09.104 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:09.104 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:09.104 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:09.104 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:35.770403+0000", 2024-09-13T11:24:09.104 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0020638150000000001 2024-09-13T11:24:09.104 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:09.104 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:09.104 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:09.104 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:35.951091+0000", 2024-09-13T11:24:09.104 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.18068817500000001 2024-09-13T11:24:09.104 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:09.104 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:09.104 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:24:09.105 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:35.951146+0000", 2024-09-13T11:24:09.105 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.4537000000000003e-05 2024-09-13T11:24:09.105 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:09.105 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:09.105 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:24:09.105 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:35.951426+0000", 2024-09-13T11:24:09.105 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00027984199999999998 2024-09-13T11:24:09.105 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:09.105 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:24:09.105 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:09.105 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:09.105 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:09.105 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "MOSDPGPush(3.19 83/70 [PushOp(3:9863b4ba:::benchmark_data_smithi177_31116_object45096:head, version: 68'2794, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:9863b4ba:::benchmark_data_smithi177_31116_object45096:head@68'2794, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:24:09.106 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:40.671028+0000", 2024-09-13T11:24:09.106 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 28.387628655, 2024-09-13T11:24:09.106 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.22869726100000001, 2024-09-13T11:24:09.106 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:24:09.106 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:24:09.106 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-09-13T11:24:09.106 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:24:09.106 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:09.106 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:24:09.106 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:40.671028+0000", 2024-09-13T11:24:09.106 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:09.106 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:09.106 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:09.107 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:24:09.107 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:40.671028+0000", 2024-09-13T11:24:09.107 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:09.107 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:09.107 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:09.107 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:24:09.107 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:40.671029+0000", 2024-09-13T11:24:09.107 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.2200000000000003e-07 2024-09-13T11:24:09.107 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:09.107 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:09.107 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:24:09.107 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:40.671098+0000", 2024-09-13T11:24:09.107 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.9241999999999999e-05 2024-09-13T11:24:09.108 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:09.108 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:09.108 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:24:09.108 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:40.898838+0000", 2024-09-13T11:24:09.108 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.22773932699999999 2024-09-13T11:24:09.108 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:09.108 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:09.108 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:24:09.108 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:40.898848+0000", 2024-09-13T11:24:09.108 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0373999999999999e-05 2024-09-13T11:24:09.108 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:09.108 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:09.108 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:24:09.108 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:40.898930+0000", 2024-09-13T11:24:09.109 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.1865000000000003e-05 2024-09-13T11:24:09.109 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:09.109 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:09.109 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:24:09.109 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:40.898938+0000", 2024-09-13T11:24:09.109 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.6109999999999996e-06 2024-09-13T11:24:09.109 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:09.109 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:09.109 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:24:09.109 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:40.899726+0000", 2024-09-13T11:24:09.109 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00078801999999999995 2024-09-13T11:24:09.109 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:09.109 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:24:09.110 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:09.110 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:09.110 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:09.110 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "MOSDPGPushReply(3.18 85/70 [PushReplyOp(3:1e410119:::benchmark_data_smithi177_31116_object35700:head)])", 2024-09-13T11:24:09.110 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:42.216614+0000", 2024-09-13T11:24:09.110 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 26.842042907, 2024-09-13T11:24:09.110 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.54260090699999997, 2024-09-13T11:24:09.110 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:24:09.110 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:24:09.110 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "reached pg", 2024-09-13T11:24:09.110 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:24:09.110 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:09.110 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:24:09.111 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.216614+0000", 2024-09-13T11:24:09.111 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:09.111 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:09.111 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:09.111 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:24:09.111 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.216614+0000", 2024-09-13T11:24:09.111 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:09.111 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:09.111 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:09.111 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:24:09.111 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.216615+0000", 2024-09-13T11:24:09.111 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.0200000000000001e-07 2024-09-13T11:24:09.111 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:09.112 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:09.112 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:24:09.112 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.216619+0000", 2024-09-13T11:24:09.112 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.805e-06 2024-09-13T11:24:09.112 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:09.112 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:09.112 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:24:09.112 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.757476+0000", 2024-09-13T11:24:09.112 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.54085762000000004 2024-09-13T11:24:09.112 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:09.112 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:09.112 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:24:09.112 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.757498+0000", 2024-09-13T11:24:09.112 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.1936000000000001e-05 2024-09-13T11:24:09.113 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:09.113 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:09.113 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:24:09.113 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.757678+0000", 2024-09-13T11:24:09.113 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00018006599999999999 2024-09-13T11:24:09.113 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:09.113 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:09.113 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:24:09.113 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.759215+0000", 2024-09-13T11:24:09.113 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0015366780000000001 2024-09-13T11:24:09.113 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:09.113 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:24:09.113 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:09.113 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:09.114 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:09.114 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "MOSDPGPushReply(3.1b 85/78 [PushReplyOp(3:dabda1cc:::benchmark_data_smithi177_31116_object12988:head)])", 2024-09-13T11:24:09.114 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:42.485171+0000", 2024-09-13T11:24:09.114 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 26.573485998999999, 2024-09-13T11:24:09.114 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.308420427, 2024-09-13T11:24:09.114 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:24:09.114 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:24:09.114 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "reached pg", 2024-09-13T11:24:09.114 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:24:09.114 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:09.114 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:24:09.114 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.485171+0000", 2024-09-13T11:24:09.114 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:09.114 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:09.115 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:09.115 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:24:09.115 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.485171+0000", 2024-09-13T11:24:09.115 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:09.115 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:09.115 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:09.115 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:24:09.115 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.485173+0000", 2024-09-13T11:24:09.115 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.1569999999999998e-06 2024-09-13T11:24:09.115 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:09.115 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:09.115 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:24:09.115 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.485182+0000", 2024-09-13T11:24:09.116 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.6759999999999996e-06 2024-09-13T11:24:09.116 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:09.116 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:09.116 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:24:09.116 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.791336+0000", 2024-09-13T11:24:09.116 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.30615446499999999 2024-09-13T11:24:09.116 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:09.116 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:09.116 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:24:09.116 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.791369+0000", 2024-09-13T11:24:09.116 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.3155999999999999e-05 2024-09-13T11:24:09.116 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:09.116 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:09.117 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:24:09.117 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.791472+0000", 2024-09-13T11:24:09.117 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00010247900000000001 2024-09-13T11:24:09.117 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:09.117 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:09.117 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:24:09.117 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.793591+0000", 2024-09-13T11:24:09.117 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.002119494 2024-09-13T11:24:09.117 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:09.117 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:24:09.117 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:09.117 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:09.117 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:09.118 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:64066 3.1b 3:dd84e0ec:::benchmark_data_smithi177_31116_object64065:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89)", 2024-09-13T11:24:09.118 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:46.633608+0000", 2024-09-13T11:24:09.118 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 22.42504856, 2024-09-13T11:24:09.118 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.32564315399999999, 2024-09-13T11:24:09.118 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:24:09.118 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:24:09.118 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:09.118 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:24:09.118 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:24:09.118 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:09.118 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 64066 2024-09-13T11:24:09.118 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:09.118 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:24:09.119 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:09.119 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:24:09.119 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.633608+0000", 2024-09-13T11:24:09.119 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:09.119 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:09.119 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:09.119 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:24:09.119 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.633608+0000", 2024-09-13T11:24:09.119 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:09.119 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:09.119 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:09.119 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:24:09.119 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.633612+0000", 2024-09-13T11:24:09.120 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.856e-06 2024-09-13T11:24:09.120 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:09.120 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:09.120 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:24:09.120 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.633683+0000", 2024-09-13T11:24:09.120 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.1000999999999993e-05 2024-09-13T11:24:09.120 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:09.120 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:09.120 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:24:09.120 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.633711+0000", 2024-09-13T11:24:09.120 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.8204999999999998e-05 2024-09-13T11:24:09.120 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:09.120 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:09.120 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:24:09.121 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.633732+0000", 2024-09-13T11:24:09.121 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.0662999999999999e-05 2024-09-13T11:24:09.121 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:09.121 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:09.121 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:24:09.121 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.635573+0000", 2024-09-13T11:24:09.121 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.001841365 2024-09-13T11:24:09.121 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:09.121 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:09.121 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:24:09.121 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.635995+0000", 2024-09-13T11:24:09.121 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00042113699999999998 2024-09-13T11:24:09.122 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:09.122 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:09.122 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 12,15", 2024-09-13T11:24:09.122 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.636475+0000", 2024-09-13T11:24:09.122 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00048067699999999999 2024-09-13T11:24:09.122 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:09.122 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:09.122 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:09.122 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.640247+0000", 2024-09-13T11:24:09.122 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0037716450000000001 2024-09-13T11:24:09.122 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:09.122 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:09.122 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:24:09.122 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.649308+0000", 2024-09-13T11:24:09.123 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0090613240000000008 2024-09-13T11:24:09.123 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:09.123 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:09.123 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:09.123 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.956912+0000", 2024-09-13T11:24:09.123 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.30760338500000001 2024-09-13T11:24:09.123 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:09.123 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:09.123 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:24:09.123 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.957156+0000", 2024-09-13T11:24:09.123 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000244031 2024-09-13T11:24:09.123 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:09.123 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:09.123 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:24:09.124 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.959252+0000", 2024-09-13T11:24:09.124 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0020958650000000001 2024-09-13T11:24:09.124 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:09.124 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:24:09.124 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:09.124 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:09.124 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:09.124 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop_reply(client.4665.0:64066 3.1b e89/78 ondisk, result = 0)", 2024-09-13T11:24:09.124 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:46.651596+0000", 2024-09-13T11:24:09.124 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 22.407060729000001, 2024-09-13T11:24:09.124 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.30767612599999999, 2024-09-13T11:24:09.124 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:24:09.124 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:24:09.125 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-09-13T11:24:09.125 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:24:09.125 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:09.125 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:24:09.125 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.651596+0000", 2024-09-13T11:24:09.125 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:09.125 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:09.125 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:09.125 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:24:09.125 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.651596+0000", 2024-09-13T11:24:09.125 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:09.125 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:09.125 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:09.126 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:24:09.126 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.651600+0000", 2024-09-13T11:24:09.126 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.6969999999999999e-06 2024-09-13T11:24:09.126 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:09.126 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:09.126 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:24:09.126 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.651611+0000", 2024-09-13T11:24:09.126 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.1516e-05 2024-09-13T11:24:09.126 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:09.126 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:09.126 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:24:09.126 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.948118+0000", 2024-09-13T11:24:09.126 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.29650663500000002 2024-09-13T11:24:09.126 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:09.127 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:09.127 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:24:09.127 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.948137+0000", 2024-09-13T11:24:09.127 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.8529000000000002e-05 2024-09-13T11:24:09.127 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:09.127 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:09.127 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:24:09.127 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.956870+0000", 2024-09-13T11:24:09.127 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0087336380000000002 2024-09-13T11:24:09.127 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:09.127 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:09.127 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:24:09.127 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.956893+0000", 2024-09-13T11:24:09.128 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.317e-05 2024-09-13T11:24:09.128 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:09.128 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:09.128 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:24:09.128 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.959272+0000", 2024-09-13T11:24:09.128 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0023789409999999999 2024-09-13T11:24:09.128 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:09.128 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:24:09.128 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:09.128 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:09.128 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:09.128 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "MOSDPGPushReply(3.1b 91/78 [PushReplyOp(3:d9ac6889:::benchmark_data_smithi177_31116_object22254:head)])", 2024-09-13T11:24:09.128 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:48.392729+0000", 2024-09-13T11:24:09.129 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 20.665927814, 2024-09-13T11:24:09.129 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.30924993099999998, 2024-09-13T11:24:09.129 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:24:09.129 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:24:09.129 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "reached pg", 2024-09-13T11:24:09.129 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:24:09.129 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:09.129 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:24:09.129 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:48.392729+0000", 2024-09-13T11:24:09.129 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:09.129 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:09.129 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:09.129 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:24:09.130 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:48.392729+0000", 2024-09-13T11:24:09.130 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:09.130 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:09.130 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:09.130 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:24:09.130 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:48.392730+0000", 2024-09-13T11:24:09.130 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.336e-06 2024-09-13T11:24:09.130 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:09.130 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:09.130 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:24:09.130 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:48.392739+0000", 2024-09-13T11:24:09.130 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.8580000000000006e-06 2024-09-13T11:24:09.130 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:09.130 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:09.131 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:24:09.131 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:48.700180+0000", 2024-09-13T11:24:09.131 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.30744035600000003 2024-09-13T11:24:09.131 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:09.131 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:09.131 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:24:09.131 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:48.700197+0000", 2024-09-13T11:24:09.131 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.7649999999999999e-05 2024-09-13T11:24:09.131 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:09.131 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:09.131 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:24:09.131 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:48.700398+0000", 2024-09-13T11:24:09.131 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000201113 2024-09-13T11:24:09.131 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:09.132 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:09.132 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:24:09.132 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:48.701979+0000", 2024-09-13T11:24:09.132 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.001580618 2024-09-13T11:24:09.132 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:09.132 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:24:09.132 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:09.132 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:09.132 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:09.132 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop(client.4665.0:69487 3.e e94/75 3:73f7f990:::benchmark_data_smithi177_31116_object69486:head v 94'3977, mlcod=94'3973)", 2024-09-13T11:24:09.132 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:52.064811+0000", 2024-09-13T11:24:09.132 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 16.993846119000001, 2024-09-13T11:24:09.132 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.20068219600000001, 2024-09-13T11:24:09.133 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:24:09.133 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:24:09.133 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:09.133 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:24:09.133 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:09.133 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:24:09.133 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:52.064811+0000", 2024-09-13T11:24:09.133 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:09.133 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:09.133 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:09.133 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:24:09.133 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:52.064811+0000", 2024-09-13T11:24:09.134 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:09.134 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:09.134 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:09.134 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:24:09.134 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:52.064814+0000", 2024-09-13T11:24:09.134 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.9280000000000002e-06 2024-09-13T11:24:09.134 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:09.134 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:09.134 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:24:09.134 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:52.064877+0000", 2024-09-13T11:24:09.134 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.3214999999999997e-05 2024-09-13T11:24:09.134 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:09.134 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:09.135 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:24:09.135 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:52.262874+0000", 2024-09-13T11:24:09.135 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.19799718499999999 2024-09-13T11:24:09.135 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:09.135 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:09.135 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:24:09.135 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:52.262883+0000", 2024-09-13T11:24:09.135 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.9120000000000004e-06 2024-09-13T11:24:09.135 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:09.135 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:09.135 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:24:09.135 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:52.262994+0000", 2024-09-13T11:24:09.136 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000110952 2024-09-13T11:24:09.136 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:09.136 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:09.136 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:24:09.136 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:52.263021+0000", 2024-09-13T11:24:09.136 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.6755999999999999e-05 2024-09-13T11:24:09.136 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:09.136 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:09.136 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:24:09.136 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:52.265387+0000", 2024-09-13T11:24:09.136 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.002366085 2024-09-13T11:24:09.136 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:09.136 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:09.136 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:24:09.137 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:52.265493+0000", 2024-09-13T11:24:09.137 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00010616300000000001 2024-09-13T11:24:09.137 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:09.137 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:24:09.137 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:09.137 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:09.137 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:09.137 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop(client.4665.0:79756 3.b e102/75 3:d6d170d7:::benchmark_data_smithi177_31116_object79755:head v 102'4217, mlcod=102'4215)", 2024-09-13T11:24:09.137 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:24:00.314794+0000", 2024-09-13T11:24:09.137 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 8.7438630249999996, 2024-09-13T11:24:09.137 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.200748656, 2024-09-13T11:24:09.137 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:24:09.137 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:24:09.138 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:09.138 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:24:09.138 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:09.138 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:24:09.138 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:00.314794+0000", 2024-09-13T11:24:09.138 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:09.138 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:09.138 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:09.138 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:24:09.138 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:00.314794+0000", 2024-09-13T11:24:09.138 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:09.138 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:09.138 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:09.139 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:24:09.139 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:00.314795+0000", 2024-09-13T11:24:09.139 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.325e-06 2024-09-13T11:24:09.139 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:09.139 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:09.139 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:24:09.139 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:00.314818+0000", 2024-09-13T11:24:09.139 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.2736999999999999e-05 2024-09-13T11:24:09.139 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:09.139 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:09.139 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:24:09.139 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:00.512052+0000", 2024-09-13T11:24:09.139 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.197234034 2024-09-13T11:24:09.139 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:09.140 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:09.140 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:24:09.140 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:00.512064+0000", 2024-09-13T11:24:09.140 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.2252999999999999e-05 2024-09-13T11:24:09.140 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:09.140 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:09.140 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:24:09.140 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:00.512187+0000", 2024-09-13T11:24:09.140 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00012311700000000001 2024-09-13T11:24:09.140 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:09.140 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:09.140 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:24:09.140 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:00.512225+0000", 2024-09-13T11:24:09.141 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.7376000000000003e-05 2024-09-13T11:24:09.141 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:09.141 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:09.141 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:24:09.141 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:00.515471+0000", 2024-09-13T11:24:09.141 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0032466819999999999 2024-09-13T11:24:09.141 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:09.141 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:09.141 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:24:09.141 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:00.515543+0000", 2024-09-13T11:24:09.141 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.1131999999999996e-05 2024-09-13T11:24:09.141 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:09.141 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:24:09.142 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:09.142 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:09.142 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:09.142 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop_reply(client.4665.0:82647 3.1b e105/78 ondisk, result = 0)", 2024-09-13T11:24:09.142 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:24:03.320583+0000", 2024-09-13T11:24:09.142 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 5.7380743870000002, 2024-09-13T11:24:09.142 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.20528595499999999, 2024-09-13T11:24:09.142 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:24:09.142 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:24:09.142 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-09-13T11:24:09.142 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:24:09.142 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:09.142 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:24:09.142 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:03.320583+0000", 2024-09-13T11:24:09.143 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:09.143 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:09.143 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:09.143 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:24:09.143 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:03.320583+0000", 2024-09-13T11:24:09.143 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:09.143 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:09.143 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:09.143 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:24:09.143 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:03.320584+0000", 2024-09-13T11:24:09.143 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.618e-06 2024-09-13T11:24:09.143 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:09.143 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:09.143 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:24:09.144 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:03.320589+0000", 2024-09-13T11:24:09.144 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.5369999999999999e-06 2024-09-13T11:24:09.144 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:09.144 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:09.144 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:24:09.144 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:03.525397+0000", 2024-09-13T11:24:09.144 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.204807988 2024-09-13T11:24:09.144 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:09.144 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:09.144 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:24:09.144 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:03.525410+0000", 2024-09-13T11:24:09.144 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.3039e-05 2024-09-13T11:24:09.144 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:09.145 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:09.145 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:24:09.145 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:03.525519+0000", 2024-09-13T11:24:09.145 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00010935800000000001 2024-09-13T11:24:09.145 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:09.145 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:09.145 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:24:09.145 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:03.525536+0000", 2024-09-13T11:24:09.145 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.6562000000000002e-05 2024-09-13T11:24:09.145 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:09.145 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:09.145 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:24:09.145 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:03.525869+0000", 2024-09-13T11:24:09.146 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00033285299999999998 2024-09-13T11:24:09.146 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:09.146 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:24:09.146 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:09.146 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:09.146 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:09.146 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop(client.4665.0:83193 3.8 e106/67 3:10066d83:::benchmark_data_smithi177_31116_object83192:head v 106'3925, mlcod=106'3922)", 2024-09-13T11:24:09.146 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:24:04.285068+0000", 2024-09-13T11:24:09.146 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 4.7735890589999999, 2024-09-13T11:24:09.146 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.64647716600000005, 2024-09-13T11:24:09.146 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:24:09.146 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:24:09.146 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:09.147 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:24:09.147 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:09.147 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:24:09.147 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:04.285068+0000", 2024-09-13T11:24:09.147 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:09.147 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:09.147 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:09.147 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:24:09.147 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:04.285068+0000", 2024-09-13T11:24:09.147 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:09.147 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:09.147 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:09.147 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:24:09.148 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:04.285071+0000", 2024-09-13T11:24:09.148 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.729e-06 2024-09-13T11:24:09.148 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:09.148 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:09.148 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:24:09.148 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:04.285215+0000", 2024-09-13T11:24:09.148 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00014410000000000001 2024-09-13T11:24:09.148 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:09.148 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:09.148 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:24:09.148 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:04.928221+0000", 2024-09-13T11:24:09.148 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.64300653299999999 2024-09-13T11:24:09.148 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:09.149 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:09.149 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:24:09.149 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:04.928240+0000", 2024-09-13T11:24:09.149 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.8678999999999999e-05 2024-09-13T11:24:09.149 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:09.149 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:09.149 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:24:09.149 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:04.928388+0000", 2024-09-13T11:24:09.149 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00014809 2024-09-13T11:24:09.149 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:09.149 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:09.149 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:24:09.149 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:04.928428+0000", 2024-09-13T11:24:09.149 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.9776e-05 2024-09-13T11:24:09.149 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:09.150 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:09.150 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:24:09.150 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:04.931463+0000", 2024-09-13T11:24:09.150 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.003035168 2024-09-13T11:24:09.150 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:09.150 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:09.150 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:24:09.150 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:04.931545+0000", 2024-09-13T11:24:09.150 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.2090999999999997e-05 2024-09-13T11:24:09.150 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:09.150 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:24:09.150 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:09.150 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:09.150 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:09.150 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop(client.4665.0:83239 3.e e106/75 3:74547f17:::benchmark_data_smithi177_31116_object83238:head v 106'4391, mlcod=106'4388)", 2024-09-13T11:24:09.151 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:24:04.581196+0000", 2024-09-13T11:24:09.151 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 4.4774605530000002, 2024-09-13T11:24:09.151 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.33617203600000001, 2024-09-13T11:24:09.151 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:24:09.151 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:24:09.151 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:09.151 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:24:09.151 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:09.151 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:24:09.151 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:04.581196+0000", 2024-09-13T11:24:09.151 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:09.151 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:09.151 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:09.151 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:24:09.151 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:04.581196+0000", 2024-09-13T11:24:09.151 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:09.152 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:09.152 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:09.152 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:24:09.152 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:04.581200+0000", 2024-09-13T11:24:09.152 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.19e-06 2024-09-13T11:24:09.152 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:09.152 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:09.152 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:24:09.152 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:04.581257+0000", 2024-09-13T11:24:09.152 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.7531999999999997e-05 2024-09-13T11:24:09.152 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:09.152 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:09.152 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:24:09.153 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:04.914011+0000", 2024-09-13T11:24:09.153 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.33275413700000001 2024-09-13T11:24:09.153 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:09.153 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:09.153 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:24:09.153 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:04.914028+0000", 2024-09-13T11:24:09.153 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.7082999999999999e-05 2024-09-13T11:24:09.153 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:09.153 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:09.153 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:24:09.153 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:04.914151+0000", 2024-09-13T11:24:09.153 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00012283300000000001 2024-09-13T11:24:09.153 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:09.154 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:09.154 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:24:09.154 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:04.914185+0000", 2024-09-13T11:24:09.154 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.4255999999999999e-05 2024-09-13T11:24:09.154 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:09.154 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:09.154 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:24:09.154 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:04.917300+0000", 2024-09-13T11:24:09.154 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0031141599999999999 2024-09-13T11:24:09.154 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:09.154 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:09.154 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:24:09.154 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:04.917368+0000", 2024-09-13T11:24:09.154 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.8844999999999996e-05 2024-09-13T11:24:09.154 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:09.155 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:24:09.155 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:09.155 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:09.155 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:24:09.155 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-09-13T11:24:09.156 INFO:teuthology.orchestra.run.smithi001.stdout:osd.14: {} 2024-09-13T11:24:09.156 INFO:tasks.ceph.osd.2.smithi162.stderr:2024-09-13T11:24:09.153+0000 7fe92b35a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26313) UID: 0 2024-09-13T11:24:09.157 INFO:teuthology.orchestra.run.smithi001.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-13T11:24:09.157 INFO:teuthology.orchestra.run.smithi001.stdout:osd.15: {} 2024-09-13T11:24:09.158 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-13T11:24:09.159 INFO:teuthology.orchestra.run.smithi001.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-13T11:24:09.257 INFO:tasks.ceph.osd.3.smithi177.stderr:2024-09-13T11:24:09.255+0000 7f10adba4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26574) UID: 0 2024-09-13T11:24:09.358 INFO:tasks.ceph.osd.7.smithi177.stderr:2024-09-13T11:24:09.355+0000 7f897b51d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26575) UID: 0 2024-09-13T11:24:09.361 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-13T11:24:09.361 INFO:teuthology.orchestra.run.smithi001.stdout: "ops": [], 2024-09-13T11:24:09.361 INFO:teuthology.orchestra.run.smithi001.stdout: "num_ops": 0 2024-09-13T11:24:09.361 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-13T11:24:09.386 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-13T11:24:09.459 INFO:tasks.ceph.osd.7.smithi177.stderr:2024-09-13T11:24:09.455+0000 7f897b51d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26575) UID: 0 2024-09-13T11:24:09.528 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-13T11:24:09.528 INFO:teuthology.orchestra.run.smithi001.stdout: "ops": [], 2024-09-13T11:24:09.528 INFO:teuthology.orchestra.run.smithi001.stdout: "complaint_time": 30, 2024-09-13T11:24:09.528 INFO:teuthology.orchestra.run.smithi001.stdout: "num_blocked_ops": 0 2024-09-13T11:24:09.528 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-13T11:24:09.546 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-13T11:24:09.561 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-09-13T11:24:09.557+0000 7f97befc9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26315) UID: 0 2024-09-13T11:24:09.661 INFO:tasks.ceph.osd.3.smithi177.stderr:2024-09-13T11:24:09.659+0000 7f10adba4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26574) UID: 0 2024-09-13T11:24:09.674 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-13T11:24:09.674 INFO:teuthology.orchestra.run.smithi001.stdout: "size": 20, 2024-09-13T11:24:09.674 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 600, 2024-09-13T11:24:09.674 INFO:teuthology.orchestra.run.smithi001.stdout: "ops": [ 2024-09-13T11:24:09.674 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:09.674 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDPGPushReply(3.1e 79/78 [PushReplyOp(3:7a3a85df:::benchmark_data_smithi177_31116_object2607:head)])", 2024-09-13T11:24:09.674 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:36.410722+0000", 2024-09-13T11:24:09.674 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 33.261205584999999, 2024-09-13T11:24:09.674 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.15957022300000001, 2024-09-13T11:24:09.675 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:24:09.675 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:24:09.675 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-13T11:24:09.675 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:24:09.675 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:09.675 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:24:09.675 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:36.410722+0000", 2024-09-13T11:24:09.675 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:09.675 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:09.675 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:09.675 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:24:09.675 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:36.410722+0000", 2024-09-13T11:24:09.675 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:09.676 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:09.676 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:09.676 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:24:09.676 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:36.410724+0000", 2024-09-13T11:24:09.676 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.367e-06 2024-09-13T11:24:09.676 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:09.676 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:09.676 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:24:09.676 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:36.410729+0000", 2024-09-13T11:24:09.676 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.4870000000000002e-06 2024-09-13T11:24:09.676 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:09.676 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:09.676 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:24:09.677 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:36.568151+0000", 2024-09-13T11:24:09.677 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.15742154999999999 2024-09-13T11:24:09.677 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:09.677 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:09.677 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:24:09.677 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:36.568162+0000", 2024-09-13T11:24:09.677 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.117e-05 2024-09-13T11:24:09.677 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:09.677 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:09.677 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:24:09.677 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:36.568214+0000", 2024-09-13T11:24:09.677 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.2187e-05 2024-09-13T11:24:09.677 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:09.678 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:09.678 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:24:09.678 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:36.570292+0000", 2024-09-13T11:24:09.678 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0020784620000000001 2024-09-13T11:24:09.678 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:09.678 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:24:09.678 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:09.678 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:09.678 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:09.678 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDPGPush(3.20 83/77 [PushOp(3:05148b46:::benchmark_data_smithi177_31116_object1401:head, version: 43'74, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:05148b46:::benchmark_data_smithi177_31116_object1401:head@43'74, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:24:09.678 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:40.912122+0000", 2024-09-13T11:24:09.678 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 28.759805757999999, 2024-09-13T11:24:09.678 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.158959773, 2024-09-13T11:24:09.679 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:24:09.679 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:24:09.679 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:24:09.679 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:24:09.679 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:09.679 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:24:09.679 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:40.912122+0000", 2024-09-13T11:24:09.679 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:09.679 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:09.679 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:09.679 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:24:09.679 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:40.912122+0000", 2024-09-13T11:24:09.679 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:09.680 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:09.680 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:09.680 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:24:09.680 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:40.912123+0000", 2024-09-13T11:24:09.680 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0589999999999999e-06 2024-09-13T11:24:09.680 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:09.680 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:09.680 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:24:09.680 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:40.912176+0000", 2024-09-13T11:24:09.680 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.2431e-05 2024-09-13T11:24:09.680 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:09.681 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:09.681 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:24:09.681 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:41.070227+0000", 2024-09-13T11:24:09.681 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.15805137499999999 2024-09-13T11:24:09.681 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:09.681 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:09.681 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:24:09.681 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:41.070243+0000", 2024-09-13T11:24:09.681 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.5948999999999999e-05 2024-09-13T11:24:09.681 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:09.681 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:09.682 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:24:09.682 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:41.070360+0000", 2024-09-13T11:24:09.682 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000117267 2024-09-13T11:24:09.682 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:09.682 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:09.682 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:24:09.682 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:41.070373+0000", 2024-09-13T11:24:09.682 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2665e-05 2024-09-13T11:24:09.682 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:09.682 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:09.682 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:24:09.682 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:41.071082+0000", 2024-09-13T11:24:09.683 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00070902699999999999 2024-09-13T11:24:09.683 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:09.683 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:24:09.683 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:09.683 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:09.683 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:09.683 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDPGPushReply(3.1e 84/78 [PushReplyOp(3:79ee54b9:::benchmark_data_smithi177_31116_object11701:head)])", 2024-09-13T11:24:09.683 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:41.407335+0000", 2024-09-13T11:24:09.683 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 28.264592490999998, 2024-09-13T11:24:09.683 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.51622350400000006, 2024-09-13T11:24:09.683 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:24:09.683 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:24:09.683 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-13T11:24:09.683 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:24:09.684 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:09.684 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:24:09.684 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:41.407335+0000", 2024-09-13T11:24:09.684 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:09.684 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:09.684 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:09.684 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:24:09.684 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:41.407335+0000", 2024-09-13T11:24:09.684 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:09.684 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:09.684 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:09.684 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:24:09.684 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:41.407337+0000", 2024-09-13T11:24:09.684 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.505e-06 2024-09-13T11:24:09.685 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:09.685 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:09.685 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:24:09.685 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:41.407343+0000", 2024-09-13T11:24:09.685 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.9089999999999997e-06 2024-09-13T11:24:09.685 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:09.685 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:09.685 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:24:09.685 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:41.921977+0000", 2024-09-13T11:24:09.685 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.51463463499999995 2024-09-13T11:24:09.685 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:09.685 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:09.685 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:24:09.686 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:41.921988+0000", 2024-09-13T11:24:09.686 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.092e-05 2024-09-13T11:24:09.686 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:09.686 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:09.686 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:24:09.686 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:41.922088+0000", 2024-09-13T11:24:09.686 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.9528e-05 2024-09-13T11:24:09.686 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:09.686 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:09.686 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:24:09.686 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:41.923559+0000", 2024-09-13T11:24:09.686 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0014710070000000001 2024-09-13T11:24:09.687 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:09.687 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:24:09.687 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:09.687 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:09.687 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:09.687 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4665.0:60417 3.10 e85/84 3:092808b0:::benchmark_data_smithi177_31116_object60416:head v 85'3295, mlcod=85'3292)", 2024-09-13T11:24:09.687 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:42.174488+0000", 2024-09-13T11:24:09.687 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 27.497439936999999, 2024-09-13T11:24:09.687 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.387908896, 2024-09-13T11:24:09.687 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:24:09.687 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:24:09.687 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:09.687 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:24:09.688 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:09.688 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:24:09.688 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.174488+0000", 2024-09-13T11:24:09.688 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:09.688 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:09.688 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:09.688 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:24:09.688 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.174488+0000", 2024-09-13T11:24:09.688 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:09.688 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:09.688 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:09.688 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:24:09.688 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.174490+0000", 2024-09-13T11:24:09.688 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.9099999999999999e-06 2024-09-13T11:24:09.689 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:09.689 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:09.689 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:24:09.689 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.174527+0000", 2024-09-13T11:24:09.689 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.7543000000000003e-05 2024-09-13T11:24:09.689 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:09.689 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:09.689 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:24:09.689 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.561093+0000", 2024-09-13T11:24:09.689 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.38656578400000002 2024-09-13T11:24:09.689 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:09.689 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:09.690 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:24:09.690 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.561101+0000", 2024-09-13T11:24:09.690 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.6090000000000003e-06 2024-09-13T11:24:09.690 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:09.690 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:09.690 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:24:09.690 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.561143+0000", 2024-09-13T11:24:09.690 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.2435000000000003e-05 2024-09-13T11:24:09.690 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:09.690 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:09.690 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:24:09.691 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.561164+0000", 2024-09-13T11:24:09.691 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.1276000000000001e-05 2024-09-13T11:24:09.691 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:09.691 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:09.691 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:24:09.691 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.562360+0000", 2024-09-13T11:24:09.691 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00119596 2024-09-13T11:24:09.691 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:09.691 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:09.692 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:24:09.692 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.562397+0000", 2024-09-13T11:24:09.692 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.6378999999999998e-05 2024-09-13T11:24:09.692 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:09.692 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:24:09.692 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:09.692 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:09.692 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:09.692 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDPGPush(3.22 85/78 [PushOp(3:456e8765:::benchmark_data_smithi177_31116_object16212:head, version: 52'1006, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:456e8765:::benchmark_data_smithi177_31116_object16212:head@52'1006, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:24:09.692 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:42.673067+0000", 2024-09-13T11:24:09.692 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 26.998860794999999, 2024-09-13T11:24:09.692 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.27935889899999999, 2024-09-13T11:24:09.692 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:24:09.692 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:24:09.693 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:24:09.693 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:24:09.693 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:09.693 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:24:09.693 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.673067+0000", 2024-09-13T11:24:09.693 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:09.693 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:09.693 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:09.693 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:24:09.693 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.673067+0000", 2024-09-13T11:24:09.693 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:09.693 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:09.693 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:09.694 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:24:09.694 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.673068+0000", 2024-09-13T11:24:09.694 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.9699999999999994e-07 2024-09-13T11:24:09.694 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:09.694 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:09.694 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:24:09.694 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.673090+0000", 2024-09-13T11:24:09.694 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.2416999999999999e-05 2024-09-13T11:24:09.694 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:09.694 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:09.694 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:24:09.694 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.951363+0000", 2024-09-13T11:24:09.694 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.27827291199999998 2024-09-13T11:24:09.695 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:09.695 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:09.695 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:24:09.695 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.951386+0000", 2024-09-13T11:24:09.695 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.2226999999999999e-05 2024-09-13T11:24:09.695 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:09.696 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:09.696 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:24:09.696 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.951581+0000", 2024-09-13T11:24:09.696 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000195713 2024-09-13T11:24:09.696 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:09.696 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:09.696 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:24:09.697 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.951605+0000", 2024-09-13T11:24:09.697 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.3767000000000001e-05 2024-09-13T11:24:09.697 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:09.697 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:09.697 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:24:09.697 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.952426+0000", 2024-09-13T11:24:09.698 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00082086599999999996 2024-09-13T11:24:09.698 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:09.698 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:24:09.698 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:09.698 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:09.698 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:09.699 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "replica_scrub(pg: 3.20,from:0'0,to:89'3254,epoch:89/77,start:3:04000000::::0,end:3:045ba805:::benchmark_data_smithi177_31116_object43251:0,chunky:1,deep:0,version:9,allow_preemption:1,priority=5)", 2024-09-13T11:24:09.699 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:46.641364+0000", 2024-09-13T11:24:09.699 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 23.030564114000001, 2024-09-13T11:24:09.699 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.27141393000000003, 2024-09-13T11:24:09.699 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:24:09.699 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:24:09.700 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:24:09.700 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:24:09.700 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:09.700 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:24:09.700 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.641364+0000", 2024-09-13T11:24:09.701 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:09.701 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:09.701 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:09.701 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:24:09.702 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.641364+0000", 2024-09-13T11:24:09.702 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:09.702 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:09.702 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:09.702 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:24:09.702 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.641365+0000", 2024-09-13T11:24:09.703 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.266e-06 2024-09-13T11:24:09.703 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:09.703 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:09.703 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:24:09.703 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.641370+0000", 2024-09-13T11:24:09.704 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.8539999999999996e-06 2024-09-13T11:24:09.704 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:09.704 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:09.704 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:24:09.704 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.912491+0000", 2024-09-13T11:24:09.704 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.27112103100000001 2024-09-13T11:24:09.705 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:09.705 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:09.705 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:24:09.705 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.912520+0000", 2024-09-13T11:24:09.705 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.9379000000000001e-05 2024-09-13T11:24:09.706 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:09.706 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:09.706 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:24:09.706 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.912679+0000", 2024-09-13T11:24:09.706 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00015915800000000001 2024-09-13T11:24:09.706 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:09.707 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:09.707 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:24:09.707 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.912714+0000", 2024-09-13T11:24:09.707 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.4687999999999997e-05 2024-09-13T11:24:09.707 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:09.708 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:09.708 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:24:09.708 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.912778+0000", 2024-09-13T11:24:09.708 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.3553999999999995e-05 2024-09-13T11:24:09.708 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:09.709 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:24:09.709 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:09.709 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:09.709 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:09.709 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDPGPush(3.18 89/70 [PushOp(3:1ac6c361:::benchmark_data_smithi177_31116_object44972:head, version: 68'2767, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:1ac6c361:::benchmark_data_smithi177_31116_object44972:head@68'2767, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:24:09.709 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:46.671018+0000", 2024-09-13T11:24:09.710 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 23.000909982, 2024-09-13T11:24:09.710 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.271282931, 2024-09-13T11:24:09.710 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:24:09.710 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:24:09.710 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:24:09.710 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:24:09.711 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:09.711 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:24:09.711 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.671018+0000", 2024-09-13T11:24:09.711 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:09.711 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:09.712 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:09.712 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:24:09.712 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.671018+0000", 2024-09-13T11:24:09.712 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:09.712 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:09.712 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:09.713 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:24:09.713 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.671020+0000", 2024-09-13T11:24:09.713 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.018e-06 2024-09-13T11:24:09.713 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:09.713 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:09.714 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:24:09.714 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.671097+0000", 2024-09-13T11:24:09.714 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.7218999999999994e-05 2024-09-13T11:24:09.714 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:09.714 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:09.714 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:24:09.715 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.939773+0000", 2024-09-13T11:24:09.715 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.26867550200000001 2024-09-13T11:24:09.715 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:09.715 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:09.715 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:24:09.716 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.939791+0000", 2024-09-13T11:24:09.716 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.8185999999999998e-05 2024-09-13T11:24:09.716 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:09.716 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:09.716 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:24:09.716 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.939926+0000", 2024-09-13T11:24:09.717 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000135291 2024-09-13T11:24:09.717 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:09.717 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:09.717 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:24:09.717 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.939938+0000", 2024-09-13T11:24:09.718 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2393e-05 2024-09-13T11:24:09.718 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:09.718 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:09.718 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:24:09.718 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.942301+0000", 2024-09-13T11:24:09.718 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.002362322 2024-09-13T11:24:09.719 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:09.719 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:24:09.719 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:09.719 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:09.719 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:09.719 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "replica_scrub(pg: 3.20,from:0'0,to:89'3259,epoch:90/77,start:3:057b84ef:::benchmark_data_smithi177_31116_object49952:0,end:3:05ddb3a2:::benchmark_data_smithi177_31116_object24337:0,chunky:1,deep:0,version:9,allow_preemption:1,priority=5)", 2024-09-13T11:24:09.720 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:47.075638+0000", 2024-09-13T11:24:09.720 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 22.596289407, 2024-09-13T11:24:09.720 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.14094659600000001, 2024-09-13T11:24:09.720 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:24:09.720 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:24:09.721 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:24:09.721 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:24:09.721 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:09.721 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:24:09.721 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:47.075638+0000", 2024-09-13T11:24:09.721 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:09.722 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:09.722 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:09.722 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:24:09.722 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:47.075638+0000", 2024-09-13T11:24:09.722 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:09.722 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:09.722 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:09.723 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:24:09.723 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:47.075640+0000", 2024-09-13T11:24:09.723 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.615e-06 2024-09-13T11:24:09.723 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:09.723 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:09.723 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:24:09.723 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:47.075646+0000", 2024-09-13T11:24:09.723 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.3829999999999997e-06 2024-09-13T11:24:09.724 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:09.724 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:09.724 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:24:09.724 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:47.216480+0000", 2024-09-13T11:24:09.724 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.14083332100000001 2024-09-13T11:24:09.724 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:09.724 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:09.724 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:24:09.724 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:47.216490+0000", 2024-09-13T11:24:09.724 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0292000000000001e-05 2024-09-13T11:24:09.725 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:09.725 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:09.725 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:24:09.725 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:47.216552+0000", 2024-09-13T11:24:09.725 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.1925000000000001e-05 2024-09-13T11:24:09.725 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:09.725 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:09.725 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:24:09.725 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:47.216562+0000", 2024-09-13T11:24:09.725 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.9529999999999994e-06 2024-09-13T11:24:09.725 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:09.725 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:09.725 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:24:09.726 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:47.216585+0000", 2024-09-13T11:24:09.726 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.3107000000000001e-05 2024-09-13T11:24:09.726 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:09.726 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:24:09.726 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:09.726 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:09.726 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:09.726 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "replica_scrub(pg: 3.20,from:0'0,to:90'3263,epoch:90/77,start:3:05ddb3a2:::benchmark_data_smithi177_31116_object24337:0,end:3:0650aac8:::benchmark_data_smithi177_31116_object1734:0,chunky:1,deep:0,version:9,allow_preemption:1,priority=5)", 2024-09-13T11:24:09.726 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:47.231682+0000", 2024-09-13T11:24:09.726 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 22.440246057, 2024-09-13T11:24:09.726 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.13889505799999999, 2024-09-13T11:24:09.726 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:24:09.726 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:24:09.727 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:24:09.727 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:24:09.727 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:09.727 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:24:09.727 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:47.231682+0000", 2024-09-13T11:24:09.727 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:09.727 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:09.727 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:09.727 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:24:09.727 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:47.231682+0000", 2024-09-13T11:24:09.728 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:09.728 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:09.728 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:09.728 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:24:09.728 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:47.231683+0000", 2024-09-13T11:24:09.728 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.282e-06 2024-09-13T11:24:09.728 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:09.728 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:09.728 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:24:09.728 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:47.231688+0000", 2024-09-13T11:24:09.728 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.3469999999999998e-06 2024-09-13T11:24:09.728 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:09.728 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:09.729 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:24:09.729 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:47.370451+0000", 2024-09-13T11:24:09.729 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.13876312499999999 2024-09-13T11:24:09.729 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:09.729 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:09.729 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:24:09.729 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:47.370461+0000", 2024-09-13T11:24:09.729 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.6059999999999992e-06 2024-09-13T11:24:09.729 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:09.729 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:09.729 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:24:09.729 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:47.370527+0000", 2024-09-13T11:24:09.729 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.5668999999999998e-05 2024-09-13T11:24:09.730 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:09.730 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:09.730 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:24:09.730 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:47.370543+0000", 2024-09-13T11:24:09.730 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.6464e-05 2024-09-13T11:24:09.730 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:09.730 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:09.730 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:24:09.730 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:47.370577+0000", 2024-09-13T11:24:09.730 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.3565000000000001e-05 2024-09-13T11:24:09.730 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:09.730 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:24:09.730 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:09.730 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:09.731 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:09.731 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "replica_scrub(pg: 3.20,from:0'0,to:90'3264,epoch:90/77,start:3:0650aac8:::benchmark_data_smithi177_31116_object1734:0,end:3:06af01e3:::benchmark_data_smithi177_31116_object31572:0,chunky:1,deep:0,version:9,allow_preemption:1,priority=5)", 2024-09-13T11:24:09.731 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:47.380615+0000", 2024-09-13T11:24:09.731 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 22.291312559000001, 2024-09-13T11:24:09.731 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.51951398400000004, 2024-09-13T11:24:09.731 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:24:09.731 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:24:09.731 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:24:09.731 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:24:09.731 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:09.731 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:24:09.731 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:47.380615+0000", 2024-09-13T11:24:09.731 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:09.732 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:09.732 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:09.732 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:24:09.732 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:47.380615+0000", 2024-09-13T11:24:09.732 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:09.732 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:09.732 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:09.732 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:24:09.732 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:47.380617+0000", 2024-09-13T11:24:09.732 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.671e-06 2024-09-13T11:24:09.733 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:09.733 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:09.733 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:24:09.733 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:47.380623+0000", 2024-09-13T11:24:09.733 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.1190000000000002e-06 2024-09-13T11:24:09.733 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:09.733 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:09.733 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:24:09.733 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:47.900050+0000", 2024-09-13T11:24:09.733 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.51942737400000005 2024-09-13T11:24:09.733 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:09.734 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:09.734 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:24:09.734 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:47.900057+0000", 2024-09-13T11:24:09.734 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.4359999999999999e-06 2024-09-13T11:24:09.734 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:09.734 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:09.734 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:24:09.734 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:47.900101+0000", 2024-09-13T11:24:09.734 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.4675000000000002e-05 2024-09-13T11:24:09.734 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:09.734 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:09.734 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:24:09.735 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:47.900110+0000", 2024-09-13T11:24:09.735 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.0569999999999993e-06 2024-09-13T11:24:09.735 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:09.735 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:09.735 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:24:09.735 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:47.900129+0000", 2024-09-13T11:24:09.735 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.9652000000000001e-05 2024-09-13T11:24:09.735 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:09.735 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:24:09.735 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:09.735 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:09.735 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:09.735 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDPGPush(3.21 91/78 [PushOp(3:8492321f:::benchmark_data_smithi177_31116_object13767:head, version: 50'840, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:8492321f:::benchmark_data_smithi177_31116_object13767:head@50'840, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:24:09.735 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:48.319251+0000", 2024-09-13T11:24:09.736 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 21.352677029999999, 2024-09-13T11:24:09.736 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.60903802200000001, 2024-09-13T11:24:09.736 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:24:09.736 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:24:09.736 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:24:09.736 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:24:09.736 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:09.736 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:24:09.736 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:48.319251+0000", 2024-09-13T11:24:09.736 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:09.736 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:09.736 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:09.736 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:24:09.736 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:48.319251+0000", 2024-09-13T11:24:09.737 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:09.737 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:09.737 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:09.737 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:24:09.737 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:48.319252+0000", 2024-09-13T11:24:09.737 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2640000000000001e-06 2024-09-13T11:24:09.737 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:09.737 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:09.737 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:24:09.738 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:48.319363+0000", 2024-09-13T11:24:09.738 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000110559 2024-09-13T11:24:09.738 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:09.738 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:09.738 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:24:09.738 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:48.927115+0000", 2024-09-13T11:24:09.738 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.60775277800000005 2024-09-13T11:24:09.738 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:09.738 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:09.738 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:24:09.738 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:48.927144+0000", 2024-09-13T11:24:09.738 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.8861e-05 2024-09-13T11:24:09.738 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:09.738 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:09.739 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:24:09.739 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:48.927318+0000", 2024-09-13T11:24:09.739 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00017399000000000001 2024-09-13T11:24:09.739 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:09.739 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:09.739 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:24:09.739 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:48.927340+0000", 2024-09-13T11:24:09.739 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.1433999999999998e-05 2024-09-13T11:24:09.739 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:09.740 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:09.740 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:24:09.740 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:48.928289+0000", 2024-09-13T11:24:09.740 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00094913599999999995 2024-09-13T11:24:09.740 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:09.740 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:24:09.740 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:09.740 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:09.740 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:09.740 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:69487 3.e 3:73f7f990:::benchmark_data_smithi177_31116_object69486:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94)", 2024-09-13T11:24:09.740 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:52.063559+0000", 2024-09-13T11:24:09.741 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 17.608368996999999, 2024-09-13T11:24:09.741 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.20206200199999999, 2024-09-13T11:24:09.741 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:24:09.741 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:24:09.741 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:09.741 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:24:09.741 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:24:09.741 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:09.741 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 69487 2024-09-13T11:24:09.741 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:09.741 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:24:09.741 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:09.742 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:24:09.742 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:52.063559+0000", 2024-09-13T11:24:09.742 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:09.742 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:09.742 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:09.742 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:24:09.742 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:52.063559+0000", 2024-09-13T11:24:09.742 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:09.742 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:09.742 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:09.742 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:24:09.742 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:52.063560+0000", 2024-09-13T11:24:09.742 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.167e-06 2024-09-13T11:24:09.742 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:09.743 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:09.743 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:24:09.743 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:52.063612+0000", 2024-09-13T11:24:09.743 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.1709999999999998e-05 2024-09-13T11:24:09.743 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:09.743 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:09.743 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:24:09.743 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:52.063623+0000", 2024-09-13T11:24:09.743 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.1251e-05 2024-09-13T11:24:09.743 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:09.743 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:09.743 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:24:09.743 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:52.063631+0000", 2024-09-13T11:24:09.744 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.7319999999999998e-06 2024-09-13T11:24:09.744 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:09.744 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:09.744 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:24:09.744 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:52.064267+0000", 2024-09-13T11:24:09.744 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00063608600000000001 2024-09-13T11:24:09.744 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:09.744 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:09.744 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:24:09.744 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:52.064407+0000", 2024-09-13T11:24:09.744 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000140686 2024-09-13T11:24:09.744 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:09.744 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:09.745 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 1", 2024-09-13T11:24:09.745 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:52.064540+0000", 2024-09-13T11:24:09.745 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000132244 2024-09-13T11:24:09.745 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:09.745 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:09.745 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:24:09.745 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:52.067019+0000", 2024-09-13T11:24:09.745 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00247919 2024-09-13T11:24:09.745 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:09.745 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:09.745 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:09.745 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:52.265543+0000", 2024-09-13T11:24:09.745 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.19852367800000001 2024-09-13T11:24:09.745 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:09.746 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:09.746 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:24:09.746 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:52.265576+0000", 2024-09-13T11:24:09.746 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.3550999999999998e-05 2024-09-13T11:24:09.746 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:09.746 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:09.746 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:24:09.746 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:52.265621+0000", 2024-09-13T11:24:09.746 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.4706999999999997e-05 2024-09-13T11:24:09.746 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:09.746 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:24:09.746 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:09.746 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:09.746 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:09.746 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:69529 3.e 3:759f9e7b:::benchmark_data_smithi177_31116_object69528:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94)", 2024-09-13T11:24:09.747 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:52.113007+0000", 2024-09-13T11:24:09.747 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 17.558920920999999, 2024-09-13T11:24:09.747 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.153112741, 2024-09-13T11:24:09.747 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:24:09.747 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:24:09.747 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:09.747 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:24:09.747 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:24:09.747 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:09.747 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 69529 2024-09-13T11:24:09.747 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:09.747 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:24:09.748 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:09.748 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:24:09.748 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:52.113007+0000", 2024-09-13T11:24:09.748 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:09.748 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:09.748 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:09.748 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:24:09.748 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:52.113007+0000", 2024-09-13T11:24:09.748 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:09.748 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:09.748 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:09.749 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:24:09.749 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:52.113009+0000", 2024-09-13T11:24:09.749 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.0490000000000002e-06 2024-09-13T11:24:09.749 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:09.749 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:09.749 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:24:09.749 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:52.113067+0000", 2024-09-13T11:24:09.749 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.8427000000000002e-05 2024-09-13T11:24:09.749 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:09.749 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:09.749 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:24:09.749 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:52.113082+0000", 2024-09-13T11:24:09.749 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.4545000000000001e-05 2024-09-13T11:24:09.750 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:09.750 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:09.750 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:24:09.750 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:52.113089+0000", 2024-09-13T11:24:09.750 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.9639999999999997e-06 2024-09-13T11:24:09.750 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:09.750 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:09.750 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:24:09.750 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:52.113154+0000", 2024-09-13T11:24:09.750 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.4856000000000004e-05 2024-09-13T11:24:09.750 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:09.750 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:09.750 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:24:09.750 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:52.113322+0000", 2024-09-13T11:24:09.750 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00016787200000000001 2024-09-13T11:24:09.751 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:09.751 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:09.751 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 1", 2024-09-13T11:24:09.751 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:52.113496+0000", 2024-09-13T11:24:09.751 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00017486800000000001 2024-09-13T11:24:09.751 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:09.751 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:09.751 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:24:09.751 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:52.115130+0000", 2024-09-13T11:24:09.751 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.001633589 2024-09-13T11:24:09.751 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:09.751 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:09.751 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:09.751 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:52.266036+0000", 2024-09-13T11:24:09.752 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.15090620299999999 2024-09-13T11:24:09.752 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:09.752 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:09.752 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:24:09.752 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:52.266080+0000", 2024-09-13T11:24:09.752 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.3457000000000001e-05 2024-09-13T11:24:09.752 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:09.752 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:09.752 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:24:09.752 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:52.266120+0000", 2024-09-13T11:24:09.752 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.9910999999999998e-05 2024-09-13T11:24:09.752 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:09.752 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:24:09.752 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:09.753 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:09.753 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:09.753 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4665.0:69547 3.18 e94/91 3:1b8f2c3d:::benchmark_data_smithi177_31116_object69546:head v 94'3569, mlcod=94'3565)", 2024-09-13T11:24:09.753 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:52.132749+0000", 2024-09-13T11:24:09.753 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 17.539179226999998, 2024-09-13T11:24:09.753 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.17361655200000001, 2024-09-13T11:24:09.753 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:24:09.753 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:24:09.753 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:09.753 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:24:09.753 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:09.753 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:24:09.753 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:52.132749+0000", 2024-09-13T11:24:09.754 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:09.754 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:09.754 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:09.754 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:24:09.754 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:52.132749+0000", 2024-09-13T11:24:09.754 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:09.754 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:09.754 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:09.754 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:24:09.754 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:52.132759+0000", 2024-09-13T11:24:09.754 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0910999999999999e-05 2024-09-13T11:24:09.754 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:09.754 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:09.754 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:24:09.754 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:52.132793+0000", 2024-09-13T11:24:09.755 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.3164999999999998e-05 2024-09-13T11:24:09.755 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:09.755 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:09.755 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:24:09.755 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:52.304761+0000", 2024-09-13T11:24:09.755 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.17196884900000001 2024-09-13T11:24:09.755 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:09.755 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:09.755 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:24:09.755 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:52.304768+0000", 2024-09-13T11:24:09.755 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.8000000000000001e-06 2024-09-13T11:24:09.755 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:09.755 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:09.755 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:24:09.756 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:52.304852+0000", 2024-09-13T11:24:09.756 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.3378e-05 2024-09-13T11:24:09.756 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:09.756 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:09.756 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:24:09.756 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:52.304872+0000", 2024-09-13T11:24:09.756 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.0072999999999999e-05 2024-09-13T11:24:09.756 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:09.756 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:09.756 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:24:09.756 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:52.306322+0000", 2024-09-13T11:24:09.756 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0014506370000000001 2024-09-13T11:24:09.756 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:09.756 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:09.757 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:24:09.757 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:52.306365+0000", 2024-09-13T11:24:09.757 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.2738999999999999e-05 2024-09-13T11:24:09.757 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:09.757 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:24:09.757 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:09.757 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:09.757 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:09.757 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4665.0:69556 3.18 e94/91 3:1d3dbeba:::benchmark_data_smithi177_31116_object69555:head v 94'3570, mlcod=94'3566)", 2024-09-13T11:24:09.757 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:52.142942+0000", 2024-09-13T11:24:09.757 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 17.528985495000001, 2024-09-13T11:24:09.757 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.21178485399999999, 2024-09-13T11:24:09.757 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:24:09.757 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:24:09.758 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:09.758 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:24:09.758 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:09.758 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:24:09.758 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:52.142942+0000", 2024-09-13T11:24:09.758 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:09.758 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:09.758 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:09.758 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:24:09.758 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:52.142942+0000", 2024-09-13T11:24:09.758 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:09.758 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:09.758 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:09.758 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:24:09.759 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:52.142944+0000", 2024-09-13T11:24:09.759 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.4500000000000001e-06 2024-09-13T11:24:09.759 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:09.759 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:09.759 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:24:09.759 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:52.143047+0000", 2024-09-13T11:24:09.759 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000103384 2024-09-13T11:24:09.759 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:09.759 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:09.759 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:24:09.759 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:52.353322+0000", 2024-09-13T11:24:09.759 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.21027483299999999 2024-09-13T11:24:09.759 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:09.759 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:09.760 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:24:09.760 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:52.353330+0000", 2024-09-13T11:24:09.760 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.8879999999999997e-06 2024-09-13T11:24:09.760 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:09.760 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:09.761 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:24:09.761 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:52.353402+0000", 2024-09-13T11:24:09.761 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.2155000000000003e-05 2024-09-13T11:24:09.761 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:09.761 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:09.761 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:24:09.761 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:52.353426+0000", 2024-09-13T11:24:09.761 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.3774e-05 2024-09-13T11:24:09.761 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:09.761 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:09.761 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:24:09.761 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:52.354674+0000", 2024-09-13T11:24:09.761 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.001248706 2024-09-13T11:24:09.761 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:09.762 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:09.762 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:24:09.762 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:52.354727+0000", 2024-09-13T11:24:09.762 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.2664000000000003e-05 2024-09-13T11:24:09.762 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:09.762 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:24:09.762 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:09.762 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:09.762 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:09.762 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:70749 3.1e 3:7bd7bfc6:::benchmark_data_smithi177_31116_object70748:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95)", 2024-09-13T11:24:09.762 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:53.156250+0000", 2024-09-13T11:24:09.762 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 16.515677673999999, 2024-09-13T11:24:09.762 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.22394481799999999, 2024-09-13T11:24:09.763 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:24:09.763 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:24:09.763 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:09.763 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:24:09.763 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:24:09.763 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:09.763 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 70749 2024-09-13T11:24:09.763 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:09.763 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:24:09.763 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:09.763 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:24:09.763 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:53.156250+0000", 2024-09-13T11:24:09.763 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:09.763 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:09.763 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:09.764 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:24:09.764 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:53.156250+0000", 2024-09-13T11:24:09.764 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:09.764 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:09.764 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:09.764 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:24:09.764 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:53.156252+0000", 2024-09-13T11:24:09.764 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.3460000000000001e-06 2024-09-13T11:24:09.764 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:09.764 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:09.764 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:24:09.764 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:53.156336+0000", 2024-09-13T11:24:09.764 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.3714999999999994e-05 2024-09-13T11:24:09.764 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:09.765 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:09.765 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:24:09.765 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:53.156347+0000", 2024-09-13T11:24:09.765 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0903999999999999e-05 2024-09-13T11:24:09.765 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:09.765 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:09.765 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:24:09.765 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:53.156352+0000", 2024-09-13T11:24:09.765 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.9139999999999999e-06 2024-09-13T11:24:09.765 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:09.765 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:09.765 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:24:09.765 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:53.158028+0000", 2024-09-13T11:24:09.765 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.001675958 2024-09-13T11:24:09.766 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:09.766 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:09.766 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:24:09.766 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:53.158154+0000", 2024-09-13T11:24:09.766 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00012653 2024-09-13T11:24:09.766 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:09.766 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:09.766 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 4,5", 2024-09-13T11:24:09.766 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:53.158277+0000", 2024-09-13T11:24:09.766 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000122168 2024-09-13T11:24:09.766 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:09.766 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:09.766 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:24:09.766 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:53.159585+0000", 2024-09-13T11:24:09.766 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.001308528 2024-09-13T11:24:09.767 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:09.767 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:09.767 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:09.767 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:53.163172+0000", 2024-09-13T11:24:09.767 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0035869690000000002 2024-09-13T11:24:09.767 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:09.767 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:09.767 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:09.767 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:53.379730+0000", 2024-09-13T11:24:09.767 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.21655822599999999 2024-09-13T11:24:09.767 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:09.767 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:09.767 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:24:09.767 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:53.379810+0000", 2024-09-13T11:24:09.768 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.0041000000000002e-05 2024-09-13T11:24:09.768 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:09.768 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:09.768 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:24:09.768 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:53.380195+0000", 2024-09-13T11:24:09.768 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000384519 2024-09-13T11:24:09.768 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:09.768 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:24:09.768 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:09.768 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:09.768 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:09.768 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:70830 3.1e 3:786f8cff:::benchmark_data_smithi177_31116_object70829:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96)", 2024-09-13T11:24:09.768 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:53.227814+0000", 2024-09-13T11:24:09.768 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 16.444113415, 2024-09-13T11:24:09.769 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.15346868599999999, 2024-09-13T11:24:09.769 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:24:09.769 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:24:09.769 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:09.769 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:24:09.769 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:24:09.769 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:09.769 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 70830 2024-09-13T11:24:09.769 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:09.769 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:24:09.769 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:09.769 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:24:09.769 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:53.227814+0000", 2024-09-13T11:24:09.769 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:09.769 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:09.770 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:09.770 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:24:09.770 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:53.227814+0000", 2024-09-13T11:24:09.770 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:09.770 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:09.770 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:09.770 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:24:09.770 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:53.227816+0000", 2024-09-13T11:24:09.770 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.136e-06 2024-09-13T11:24:09.770 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:09.770 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:09.770 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:24:09.771 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:53.227889+0000", 2024-09-13T11:24:09.771 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.2186000000000003e-05 2024-09-13T11:24:09.771 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:09.771 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:09.771 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:24:09.771 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:53.227903+0000", 2024-09-13T11:24:09.771 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.4337e-05 2024-09-13T11:24:09.771 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:09.771 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:09.771 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:24:09.771 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:53.227910+0000", 2024-09-13T11:24:09.771 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.4259999999999997e-06 2024-09-13T11:24:09.771 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:09.771 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:09.772 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:24:09.772 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:53.227973+0000", 2024-09-13T11:24:09.772 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.2811999999999994e-05 2024-09-13T11:24:09.772 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:09.772 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:09.772 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:24:09.772 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:53.228145+0000", 2024-09-13T11:24:09.772 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00017218100000000001 2024-09-13T11:24:09.772 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:09.772 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:09.772 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 4,5", 2024-09-13T11:24:09.772 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:53.228338+0000", 2024-09-13T11:24:09.772 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00019233999999999999 2024-09-13T11:24:09.772 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:09.773 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:09.773 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:24:09.773 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:53.229614+0000", 2024-09-13T11:24:09.773 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.001276665 2024-09-13T11:24:09.773 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:09.773 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:09.773 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:09.773 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:53.230256+0000", 2024-09-13T11:24:09.773 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00064173300000000002 2024-09-13T11:24:09.773 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:09.773 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:09.773 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:09.773 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:53.380944+0000", 2024-09-13T11:24:09.773 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.15068795500000001 2024-09-13T11:24:09.773 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:09.774 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:09.774 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:24:09.774 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:53.380984+0000", 2024-09-13T11:24:09.774 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.036e-05 2024-09-13T11:24:09.774 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:09.774 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:09.774 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:24:09.774 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:53.381283+0000", 2024-09-13T11:24:09.774 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00029855499999999999 2024-09-13T11:24:09.774 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:09.774 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:24:09.774 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:09.774 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:09.774 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:09.775 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDPGPush(3.21 96/78 [PushOp(3:862d73af:::benchmark_data_smithi177_31116_object32719:head, version: 60'2029, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:862d73af:::benchmark_data_smithi177_31116_object32719:head@60'2029, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:24:09.775 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:53.884628+0000", 2024-09-13T11:24:09.775 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 15.787299832, 2024-09-13T11:24:09.775 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.21020203400000001, 2024-09-13T11:24:09.775 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:24:09.775 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:24:09.775 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:24:09.775 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:24:09.775 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:09.775 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:24:09.775 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:53.884628+0000", 2024-09-13T11:24:09.775 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:09.775 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:09.775 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:09.776 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:24:09.776 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:53.884628+0000", 2024-09-13T11:24:09.776 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:09.776 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:09.776 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:09.776 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:24:09.776 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:53.884630+0000", 2024-09-13T11:24:09.776 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.68e-06 2024-09-13T11:24:09.776 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:09.776 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:09.776 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:24:09.776 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:53.884745+0000", 2024-09-13T11:24:09.776 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000115599 2024-09-13T11:24:09.776 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:09.777 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:09.777 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:24:09.777 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:54.093948+0000", 2024-09-13T11:24:09.777 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.20920306999999999 2024-09-13T11:24:09.777 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:09.777 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:09.777 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:24:09.777 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:54.093961+0000", 2024-09-13T11:24:09.777 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2815e-05 2024-09-13T11:24:09.777 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:09.777 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:09.777 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:24:09.777 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:54.094139+0000", 2024-09-13T11:24:09.777 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00017824 2024-09-13T11:24:09.778 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:09.778 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:09.778 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:24:09.778 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:54.094150+0000", 2024-09-13T11:24:09.778 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0593e-05 2024-09-13T11:24:09.778 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:09.778 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:09.778 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:24:09.778 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:54.094830+0000", 2024-09-13T11:24:09.778 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00068003699999999998 2024-09-13T11:24:09.778 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:09.778 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:24:09.778 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:09.778 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:09.779 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:09.779 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:83239 3.e 3:74547f17:::benchmark_data_smithi177_31116_object83238:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106)", 2024-09-13T11:24:09.779 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:24:04.580085+0000", 2024-09-13T11:24:09.779 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 5.0918429490000001, 2024-09-13T11:24:09.779 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.33785762600000002, 2024-09-13T11:24:09.779 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:24:09.779 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:24:09.779 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:09.779 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:24:09.779 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:24:09.779 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:09.779 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 83239 2024-09-13T11:24:09.779 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:09.779 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:24:09.780 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:09.780 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:24:09.780 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:04.580085+0000", 2024-09-13T11:24:09.780 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:09.780 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:09.780 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:09.780 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:24:09.780 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:04.580085+0000", 2024-09-13T11:24:09.780 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:09.780 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:09.780 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:09.780 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:24:09.780 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:04.580089+0000", 2024-09-13T11:24:09.780 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.053e-06 2024-09-13T11:24:09.781 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:09.781 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:09.781 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:24:09.781 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:04.580166+0000", 2024-09-13T11:24:09.781 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.7102999999999997e-05 2024-09-13T11:24:09.781 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:09.781 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:09.781 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:24:09.781 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:04.580196+0000", 2024-09-13T11:24:09.781 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.0196e-05 2024-09-13T11:24:09.781 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:09.781 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:09.781 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:24:09.781 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:04.580212+0000", 2024-09-13T11:24:09.782 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.535e-05 2024-09-13T11:24:09.782 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:09.782 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:09.782 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:24:09.782 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:04.580336+0000", 2024-09-13T11:24:09.782 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00012431899999999999 2024-09-13T11:24:09.782 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:09.782 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:09.782 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:24:09.782 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:04.580575+0000", 2024-09-13T11:24:09.782 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00023924100000000001 2024-09-13T11:24:09.782 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:09.782 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:09.782 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 1", 2024-09-13T11:24:09.783 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:04.580798+0000", 2024-09-13T11:24:09.783 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00022301499999999999 2024-09-13T11:24:09.783 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:09.783 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:09.783 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:24:09.783 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:04.583404+0000", 2024-09-13T11:24:09.783 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0026056120000000002 2024-09-13T11:24:09.783 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:09.783 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:09.783 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:09.783 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:04.917768+0000", 2024-09-13T11:24:09.783 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.33436387499999998 2024-09-13T11:24:09.783 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:09.783 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:09.784 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:24:09.784 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:04.917859+0000", 2024-09-13T11:24:09.784 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.1732999999999997e-05 2024-09-13T11:24:09.784 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:09.784 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:09.784 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:24:09.784 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:04.917942+0000", 2024-09-13T11:24:09.784 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.3128999999999996e-05 2024-09-13T11:24:09.785 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:09.785 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:24:09.785 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:09.785 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:09.785 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:09.785 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:84571 3.21 3:85e7a789:::benchmark_data_smithi177_31116_object84570:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-09-13T11:24:09.785 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:24:05.695256+0000", 2024-09-13T11:24:09.785 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 3.9766719180000001, 2024-09-13T11:24:09.785 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.18917198299999999, 2024-09-13T11:24:09.785 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:24:09.785 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:24:09.785 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:09.785 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:24:09.785 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:24:09.786 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:09.786 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 84571 2024-09-13T11:24:09.786 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:09.786 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:24:09.786 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:09.786 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:24:09.786 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:05.695256+0000", 2024-09-13T11:24:09.786 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:09.786 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:09.786 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:09.786 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:24:09.786 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:05.695256+0000", 2024-09-13T11:24:09.786 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:09.786 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:09.787 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:09.787 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:24:09.787 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:05.695258+0000", 2024-09-13T11:24:09.787 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.0530000000000001e-06 2024-09-13T11:24:09.787 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:09.787 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:09.787 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:24:09.787 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:05.695304+0000", 2024-09-13T11:24:09.787 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.5769999999999997e-05 2024-09-13T11:24:09.787 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:09.787 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:09.787 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:24:09.787 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:05.695313+0000", 2024-09-13T11:24:09.787 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.7049999999999994e-06 2024-09-13T11:24:09.788 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:09.788 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:09.788 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:24:09.788 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:05.695321+0000", 2024-09-13T11:24:09.788 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.7190000000000001e-06 2024-09-13T11:24:09.788 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:09.788 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:09.788 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:24:09.788 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:05.695404+0000", 2024-09-13T11:24:09.788 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.3188000000000004e-05 2024-09-13T11:24:09.788 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:09.788 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:09.788 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:24:09.788 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:05.695635+0000", 2024-09-13T11:24:09.789 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00023032900000000001 2024-09-13T11:24:09.789 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:09.789 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:09.789 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 8", 2024-09-13T11:24:09.789 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:05.695852+0000", 2024-09-13T11:24:09.789 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00021749599999999999 2024-09-13T11:24:09.789 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:09.789 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:09.789 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:24:09.789 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:05.697208+0000", 2024-09-13T11:24:09.789 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00135633 2024-09-13T11:24:09.789 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:09.789 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:09.789 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:09.790 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:05.884337+0000", 2024-09-13T11:24:09.790 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.187128867 2024-09-13T11:24:09.790 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:09.790 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:09.790 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:24:09.790 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:05.884391+0000", 2024-09-13T11:24:09.790 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.3724999999999998e-05 2024-09-13T11:24:09.790 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:09.790 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:09.790 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:24:09.790 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:05.884428+0000", 2024-09-13T11:24:09.790 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.6801000000000002e-05 2024-09-13T11:24:09.790 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:09.790 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:24:09.791 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:09.791 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:09.791 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:24:09.791 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-13T11:24:09.792 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-13T11:24:09.793 INFO:tasks.ceph.osd.1.smithi092.stderr:2024-09-13T11:24:09.787+0000 7fe7789c5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33718) UID: 0 2024-09-13T11:24:09.796 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-13T11:24:09.894 INFO:tasks.ceph.osd.14.smithi162.stderr:2024-09-13T11:24:09.889+0000 7f235c0db640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26319) UID: 0 2024-09-13T11:24:09.930 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-13T11:24:09.931 INFO:teuthology.orchestra.run.smithi001.stdout: "ops": [ 2024-09-13T11:24:09.931 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:09.931 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:90027 3.11 3:8df0276b:::benchmark_data_smithi177_31116_object90026:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110)", 2024-09-13T11:24:09.931 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:24:09.882180+0000", 2024-09-13T11:24:09.931 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 0.047992275000000001, 2024-09-13T11:24:09.931 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.048012441000000003, 2024-09-13T11:24:09.931 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:24:09.931 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:24:09.931 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "waiting for sub ops", 2024-09-13T11:24:09.931 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:24:09.931 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:24:09.931 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:09.931 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 90027 2024-09-13T11:24:09.932 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:09.932 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:24:09.932 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:09.932 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:24:09.932 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:09.882180+0000", 2024-09-13T11:24:09.932 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:09.932 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:09.932 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:09.932 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:24:09.932 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:09.882180+0000", 2024-09-13T11:24:09.932 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:09.932 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:09.932 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:09.932 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:24:09.933 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:09.882182+0000", 2024-09-13T11:24:09.933 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.667e-06 2024-09-13T11:24:09.933 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:09.933 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:09.933 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:24:09.933 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:09.882234+0000", 2024-09-13T11:24:09.933 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.2008000000000001e-05 2024-09-13T11:24:09.933 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:09.933 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:09.934 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:24:09.934 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:09.882251+0000", 2024-09-13T11:24:09.934 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.7141000000000001e-05 2024-09-13T11:24:09.934 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:09.934 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:09.934 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:24:09.934 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:09.882259+0000", 2024-09-13T11:24:09.934 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.2599999999999999e-06 2024-09-13T11:24:09.934 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:09.934 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:09.934 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:24:09.934 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:09.882332+0000", 2024-09-13T11:24:09.934 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.3869999999999996e-05 2024-09-13T11:24:09.935 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:09.935 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:09.935 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:24:09.935 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:09.882523+0000", 2024-09-13T11:24:09.935 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00019037700000000001 2024-09-13T11:24:09.935 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:09.935 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:09.935 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 11", 2024-09-13T11:24:09.935 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:09.882718+0000", 2024-09-13T11:24:09.935 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00019472200000000001 2024-09-13T11:24:09.935 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:09.935 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:09.935 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:24:09.936 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:09.884537+0000", 2024-09-13T11:24:09.936 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0018189390000000001 2024-09-13T11:24:09.936 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:09.936 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:24:09.936 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:09.936 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:09.936 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-09-13T11:24:09.936 INFO:teuthology.orchestra.run.smithi001.stdout: "num_ops": 1 2024-09-13T11:24:09.936 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-13T11:24:09.948 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-13T11:24:09.989 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-13T11:24:09.994 INFO:tasks.ceph.osd.10.smithi162.stderr:2024-09-13T11:24:09.993+0000 7fb59c9f0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26317) UID: 0 2024-09-13T11:24:10.008 INFO:tasks.radosbench.radosbench.0.smithi177.stdout: 103 16 90158 90142 54.6923 85.0625 0.00526708 0.0182777 2024-09-13T11:24:10.089 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-13T11:24:10.090 INFO:teuthology.orchestra.run.smithi001.stdout: "ops": [], 2024-09-13T11:24:10.090 INFO:teuthology.orchestra.run.smithi001.stdout: "complaint_time": 30, 2024-09-13T11:24:10.090 INFO:teuthology.orchestra.run.smithi001.stdout: "num_blocked_ops": 0 2024-09-13T11:24:10.090 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-13T11:24:10.096 INFO:tasks.ceph.osd.5.smithi092.stderr:2024-09-13T11:24:10.091+0000 7f6f0b705640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26346) UID: 0 2024-09-13T11:24:10.106 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-13T11:24:10.196 INFO:tasks.ceph.osd.9.smithi092.stderr:2024-09-13T11:24:10.191+0000 7ff5ab3d0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26348) UID: 0 2024-09-13T11:24:10.207 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-13T11:24:10.208 INFO:teuthology.orchestra.run.smithi001.stderr:dumped all 2024-09-13T11:24:10.228 INFO:teuthology.orchestra.run.smithi001.stdout:{"pg_ready":true,"pg_map":{"version":156,"stamp":"2024-09-13T11:24:10.143739+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":5573444147,"num_objects":85040,"num_object_clones":0,"num_object_copies":170080,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":85040,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":38654,"num_write_kb":2467058,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11627,"num_bytes_recovered":761888787,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":145597,"ondisk_log_size":145597,"up":90,"acting":90,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":114,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":16,"kb":1509949440,"kb_used":13714124,"kb_used_data":12813196,"kb_used_omap":27,"kb_used_meta":900644,"kb_avail":1496235316,"statfs":{"total":1546188226560,"available":1532144963584,"internally_reserved":0,"allocated":13120712704,"data_stored":13113211821,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28358,"internal_metadata":922259770},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,1,2,3,0,1,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":308543488,"num_objects":4708,"num_object_clones":0,"num_object_copies":9416,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1468,"num_objects_unfound":0,"num_objects_dirty":4708,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4708,"num_write_kb":301312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":735,"num_bytes_recovered":48168960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.206423"},"pg_stats":[{"pgid":"3.23","version":"108'3609","reported_seq":6341,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T11:24:05.966241+0000","last_change":"2024-09-13T11:23:48.133184+0000","last_active":"2024-09-13T11:24:05.966241+0000","last_peered":"2024-09-13T11:24:05.966241+0000","last_clean":"2024-09-13T11:24:05.966241+0000","last_became_active":"2024-09-13T11:23:47.926677+0000","last_became_peered":"2024-09-13T11:23:47.926677+0000","last_unstale":"2024-09-13T11:24:05.966241+0000","last_undegraded":"2024-09-13T11:24:05.966241+0000","last_fullsized":"2024-09-13T11:24:05.966241+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":6,"last_scrub":"91'3278","last_scrub_stamp":"2024-09-13T11:23:48.133084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:48.133084+0000","objects_scrubbed":1029,"log_size":3609,"log_dups_size":0,"ondisk_log_size":3609,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:12.874134+0000","scrub_duration":139,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":89194496,"num_objects":1361,"num_object_clones":0,"num_object_copies":2722,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1361,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":332,"num_write_kb":21248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9],"acting":[5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.22","version":"108'3574","reported_seq":5493,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T11:24:05.958953+0000","last_change":"2024-09-13T11:23:47.300446+0000","last_active":"2024-09-13T11:24:05.958953+0000","last_peered":"2024-09-13T11:24:05.958953+0000","last_clean":"2024-09-13T11:24:05.958953+0000","last_became_active":"2024-09-13T11:23:46.925318+0000","last_became_peered":"2024-09-13T11:23:46.925318+0000","last_unstale":"2024-09-13T11:24:05.958953+0000","last_undegraded":"2024-09-13T11:24:05.958953+0000","last_fullsized":"2024-09-13T11:24:05.958953+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":6,"last_scrub":"90'3246","last_scrub_stamp":"2024-09-13T11:23:47.300339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:47.300339+0000","objects_scrubbed":1033,"log_size":3574,"log_dups_size":0,"ondisk_log_size":3574,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:24:49.780485+0000","scrub_duration":203,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":89325568,"num_objects":1363,"num_object_clones":0,"num_object_copies":2726,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1363,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":330,"num_write_kb":21120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,14],"acting":[0,14],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.21","version":"108'3525","reported_seq":6247,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T11:24:05.984268+0000","last_change":"2024-09-13T11:24:02.142682+0000","last_active":"2024-09-13T11:24:05.984268+0000","last_peered":"2024-09-13T11:24:05.984268+0000","last_clean":"2024-09-13T11:24:05.984268+0000","last_became_active":"2024-09-13T11:24:01.250308+0000","last_became_peered":"2024-09-13T11:24:01.250308+0000","last_unstale":"2024-09-13T11:24:05.984268+0000","last_undegraded":"2024-09-13T11:24:05.984268+0000","last_fullsized":"2024-09-13T11:24:05.984268+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":6,"last_scrub":"69'2784","last_scrub_stamp":"2024-09-13T11:23:21.595419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:21.595419+0000","objects_scrubbed":0,"log_size":3525,"log_dups_size":0,"ondisk_log_size":3525,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"user requested scrub scheduled @ 2024-09-13T11:24:07.138422+0000","scrub_duration":311,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":84738048,"num_objects":1293,"num_object_clones":0,"num_object_copies":2586,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1293,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":600,"num_write_kb":38400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":836,"num_bytes_recovered":54788096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8],"acting":[0,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.20","version":"108'3606","reported_seq":5640,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T11:24:05.969931+0000","last_change":"2024-09-13T11:23:47.969503+0000","last_active":"2024-09-13T11:24:05.969931+0000","last_peered":"2024-09-13T11:24:05.969931+0000","last_clean":"2024-09-13T11:24:05.969931+0000","last_became_active":"2024-09-13T11:23:33.954946+0000","last_became_peered":"2024-09-13T11:23:33.954946+0000","last_unstale":"2024-09-13T11:24:05.969931+0000","last_undegraded":"2024-09-13T11:24:05.969931+0000","last_fullsized":"2024-09-13T11:24:05.969931+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":6,"last_scrub":"91'3270","last_scrub_stamp":"2024-09-13T11:23:47.969372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:47.969372+0000","objects_scrubbed":1061,"log_size":3606,"log_dups_size":0,"ondisk_log_size":3606,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:24:48.255263+0000","scrub_duration":1327,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":91881472,"num_objects":1402,"num_object_clones":0,"num_object_copies":2804,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1402,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":341,"num_write_kb":21824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,0],"acting":[12,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[]},{"pgid":"3.1f","version":"107'4324","reported_seq":9453,"reported_epoch":108,"state":"peering","last_fresh":"2024-09-13T11:24:05.943810+0000","last_change":"2024-09-13T11:24:05.943810+0000","last_active":"2024-09-13T11:24:05.943810+0000","last_peered":"2024-09-13T11:24:05.897907+0000","last_clean":"2024-09-13T11:23:32.316727+0000","last_became_active":"2024-09-13T11:23:35.053803+0000","last_became_peered":"2024-09-13T11:23:35.053803+0000","last_unstale":"2024-09-13T11:24:05.943810+0000","last_undegraded":"2024-09-13T11:24:05.943810+0000","last_fullsized":"2024-09-13T11:24:05.943810+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:22:24.896055+0000","objects_scrubbed":0,"log_size":4324,"log_dups_size":0,"ondisk_log_size":4324,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":171540480,"num_objects":2618,"num_object_clones":0,"num_object_copies":5236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2618,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2618,"num_write_kb":167520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1651,"num_bytes_recovered":108199936,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,14],"acting":[10,14],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1e","version":"107'4444","reported_seq":14807,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T11:24:05.956882+0000","last_change":"2024-09-13T11:24:02.037297+0000","last_active":"2024-09-13T11:24:05.956882+0000","last_peered":"2024-09-13T11:24:05.956882+0000","last_clean":"2024-09-13T11:24:05.956882+0000","last_became_active":"2024-09-13T11:24:01.261448+0000","last_became_peered":"2024-09-13T11:24:01.261448+0000","last_unstale":"2024-09-13T11:24:05.956882+0000","last_undegraded":"2024-09-13T11:24:05.956882+0000","last_fullsized":"2024-09-13T11:24:05.956882+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:22:24.896055+0000","objects_scrubbed":0,"log_size":4444,"log_dups_size":0,"ondisk_log_size":4444,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-13T11:24:06.974692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":175898624,"num_objects":2684,"num_object_clones":0,"num_object_copies":5368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2684,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2684,"num_write_kb":171776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2573,"num_bytes_recovered":168591360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"108'4306","reported_seq":9591,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T11:24:06.011843+0000","last_change":"2024-09-13T11:24:02.078835+0000","last_active":"2024-09-13T11:24:06.011843+0000","last_peered":"2024-09-13T11:24:06.011843+0000","last_clean":"2024-09-13T11:24:06.011843+0000","last_became_active":"2024-09-13T11:24:01.258225+0000","last_became_peered":"2024-09-13T11:24:01.258225+0000","last_unstale":"2024-09-13T11:24:06.011843+0000","last_undegraded":"2024-09-13T11:24:06.011843+0000","last_fullsized":"2024-09-13T11:24:06.011843+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:22:24.896055+0000","objects_scrubbed":0,"log_size":4306,"log_dups_size":0,"ondisk_log_size":4306,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-13T11:24:07.039585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":171376640,"num_objects":2615,"num_object_clones":0,"num_object_copies":5230,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2615,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2615,"num_write_kb":167360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1700,"num_bytes_recovered":111411200,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,15],"acting":[2,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1c","version":"109'4429","reported_seq":8115,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T11:24:07.057488+0000","last_change":"2024-09-13T11:24:00.529312+0000","last_active":"2024-09-13T11:24:07.057488+0000","last_peered":"2024-09-13T11:24:07.057488+0000","last_clean":"2024-09-13T11:24:07.057488+0000","last_became_active":"2024-09-13T11:23:33.989492+0000","last_became_peered":"2024-09-13T11:23:33.989492+0000","last_unstale":"2024-09-13T11:24:07.057488+0000","last_undegraded":"2024-09-13T11:24:07.057488+0000","last_fullsized":"2024-09-13T11:24:07.057488+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":5,"last_scrub":"102'4277","last_scrub_stamp":"2024-09-13T11:24:00.529243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:24:00.529243+0000","objects_scrubbed":2544,"log_size":4429,"log_dups_size":0,"ondisk_log_size":4429,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:15.595052+0000","scrub_duration":259,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":176816128,"num_objects":2698,"num_object_clones":0,"num_object_copies":5396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2698,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":154,"num_write_kb":9856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7],"acting":[11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1b","version":"108'4380","reported_seq":9848,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T11:24:06.092111+0000","last_change":"2024-09-13T11:24:05.835103+0000","last_active":"2024-09-13T11:24:06.092111+0000","last_peered":"2024-09-13T11:24:06.092111+0000","last_clean":"2024-09-13T11:24:06.092111+0000","last_became_active":"2024-09-13T11:24:04.950821+0000","last_became_peered":"2024-09-13T11:24:04.950821+0000","last_unstale":"2024-09-13T11:24:06.092111+0000","last_undegraded":"2024-09-13T11:24:06.092111+0000","last_fullsized":"2024-09-13T11:24:06.092111+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:22:24.896055+0000","objects_scrubbed":0,"log_size":4380,"log_dups_size":0,"ondisk_log_size":4380,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-13T11:24:10.806111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":173637632,"num_objects":2650,"num_object_clones":0,"num_object_copies":5300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2650,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2650,"num_write_kb":169568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1760,"num_bytes_recovered":115343360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,15],"acting":[12,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[]},{"pgid":"3.1a","version":"108'4273","reported_seq":9405,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T11:24:06.022201+0000","last_change":"2024-09-13T11:24:04.144689+0000","last_active":"2024-09-13T11:24:06.022201+0000","last_peered":"2024-09-13T11:24:06.022201+0000","last_clean":"2024-09-13T11:24:06.022201+0000","last_became_active":"2024-09-13T11:24:03.935185+0000","last_became_peered":"2024-09-13T11:24:03.935185+0000","last_unstale":"2024-09-13T11:24:06.022201+0000","last_undegraded":"2024-09-13T11:24:06.022201+0000","last_fullsized":"2024-09-13T11:24:06.022201+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:22:24.896055+0000","objects_scrubbed":0,"log_size":4273,"log_dups_size":0,"ondisk_log_size":4273,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-13T11:24:09.133075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":168230912,"num_objects":2567,"num_object_clones":0,"num_object_copies":5134,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2567,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2567,"num_write_kb":164288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1648,"num_bytes_recovered":108003328,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9],"acting":[4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.19","version":"109'4031","reported_seq":8340,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T11:24:07.006880+0000","last_change":"2024-09-13T11:23:49.429522+0000","last_active":"2024-09-13T11:24:07.006880+0000","last_peered":"2024-09-13T11:24:07.006880+0000","last_clean":"2024-09-13T11:24:07.006880+0000","last_became_active":"2024-09-13T11:23:41.924941+0000","last_became_peered":"2024-09-13T11:23:41.924941+0000","last_unstale":"2024-09-13T11:24:07.006880+0000","last_undegraded":"2024-09-13T11:24:07.006880+0000","last_fullsized":"2024-09-13T11:24:07.006880+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":5,"last_scrub":"92'3415","last_scrub_stamp":"2024-09-13T11:23:49.429429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:49.429429+0000","objects_scrubbed":1979,"log_size":4031,"log_dups_size":0,"ondisk_log_size":4031,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:04.629542+0000","scrub_duration":587,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":170917888,"num_objects":2608,"num_object_clones":0,"num_object_copies":5216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2608,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":629,"num_write_kb":40256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1],"acting":[9,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.18","version":"108'4042","reported_seq":8617,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T11:24:05.971274+0000","last_change":"2024-09-13T11:24:02.096598+0000","last_active":"2024-09-13T11:24:05.971274+0000","last_peered":"2024-09-13T11:24:05.971274+0000","last_clean":"2024-09-13T11:24:05.971274+0000","last_became_active":"2024-09-13T11:23:48.926014+0000","last_became_peered":"2024-09-13T11:23:48.926014+0000","last_unstale":"2024-09-13T11:24:05.971274+0000","last_undegraded":"2024-09-13T11:24:05.971274+0000","last_fullsized":"2024-09-13T11:24:05.971274+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:22:24.896055+0000","objects_scrubbed":0,"log_size":4042,"log_dups_size":0,"ondisk_log_size":4042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-13T11:24:07.029225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":175308800,"num_objects":2675,"num_object_clones":0,"num_object_copies":5350,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2675,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2675,"num_write_kb":171200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1458,"num_bytes_recovered":95551488,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0],"acting":[10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.17","version":"107'3954","reported_seq":7995,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T11:24:05.943810+0000","last_change":"2024-09-13T11:23:49.349241+0000","last_active":"2024-09-13T11:24:05.943810+0000","last_peered":"2024-09-13T11:24:05.943810+0000","last_clean":"2024-09-13T11:24:05.943810+0000","last_became_active":"2024-09-13T11:23:42.935601+0000","last_became_peered":"2024-09-13T11:23:42.935601+0000","last_unstale":"2024-09-13T11:24:05.943810+0000","last_undegraded":"2024-09-13T11:24:05.943810+0000","last_fullsized":"2024-09-13T11:24:05.943810+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":5,"last_scrub":"92'3359","last_scrub_stamp":"2024-09-13T11:23:49.349167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:49.349167+0000","objects_scrubbed":2004,"log_size":3954,"log_dups_size":0,"ondisk_log_size":3954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:24:59.401302+0000","scrub_duration":363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":170852352,"num_objects":2607,"num_object_clones":0,"num_object_copies":5214,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2607,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":603,"num_write_kb":38592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3],"acting":[10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"109'4209","reported_seq":8882,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T11:24:07.047128+0000","last_change":"2024-09-13T11:23:46.510544+0000","last_active":"2024-09-13T11:24:07.047128+0000","last_peered":"2024-09-13T11:24:07.047128+0000","last_clean":"2024-09-13T11:24:07.047128+0000","last_became_active":"2024-09-13T11:23:46.105237+0000","last_became_peered":"2024-09-13T11:23:46.105237+0000","last_unstale":"2024-09-13T11:24:07.047128+0000","last_undegraded":"2024-09-13T11:24:07.047128+0000","last_fullsized":"2024-09-13T11:24:07.047128+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":5,"last_scrub":"89'3478","last_scrub_stamp":"2024-09-13T11:23:46.510384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:46.510384+0000","objects_scrubbed":2054,"log_size":4209,"log_dups_size":0,"ondisk_log_size":4209,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:09.149878+0000","scrub_duration":363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":182714368,"num_objects":2788,"num_object_clones":0,"num_object_copies":5576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2788,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":734,"num_write_kb":46976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,15],"acting":[11,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.15","version":"108'3978","reported_seq":8409,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T11:24:06.018660+0000","last_change":"2024-09-13T11:23:48.332790+0000","last_active":"2024-09-13T11:24:06.018660+0000","last_peered":"2024-09-13T11:24:06.018660+0000","last_clean":"2024-09-13T11:24:06.018660+0000","last_became_active":"2024-09-13T11:23:42.931149+0000","last_became_peered":"2024-09-13T11:23:42.931149+0000","last_unstale":"2024-09-13T11:24:06.018660+0000","last_undegraded":"2024-09-13T11:24:06.018660+0000","last_fullsized":"2024-09-13T11:24:06.018660+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":5,"last_scrub":"91'3384","last_scrub_stamp":"2024-09-13T11:23:48.332600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:48.332600+0000","objects_scrubbed":2032,"log_size":3978,"log_dups_size":0,"ondisk_log_size":3978,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:24:58.239508+0000","scrub_duration":235,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":172163072,"num_objects":2627,"num_object_clones":0,"num_object_copies":5254,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2627,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":595,"num_write_kb":38080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,0],"acting":[14,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"3.14","version":"107'3984","reported_seq":8233,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-13T11:24:05.038872+0000","last_change":"2024-09-13T11:23:48.697100+0000","last_active":"2024-09-13T11:24:05.038872+0000","last_peered":"2024-09-13T11:24:05.038872+0000","last_clean":"2024-09-13T11:24:05.038872+0000","last_became_active":"2024-09-13T11:23:41.920989+0000","last_became_peered":"2024-09-13T11:23:41.920989+0000","last_unstale":"2024-09-13T11:24:05.038872+0000","last_undegraded":"2024-09-13T11:24:05.038872+0000","last_fullsized":"2024-09-13T11:24:05.038872+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":5,"last_scrub":"91'3371","last_scrub_stamp":"2024-09-13T11:23:48.696960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:48.696960+0000","objects_scrubbed":1935,"log_size":3984,"log_dups_size":0,"ondisk_log_size":3984,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:02.861530+0000","scrub_duration":735,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":167641088,"num_objects":2558,"num_object_clones":0,"num_object_copies":5116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2558,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":623,"num_write_kb":39872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,14],"acting":[3,14],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":179,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T11:24:05.944977+0000","last_change":"2024-09-13T11:22:16.889130+0000","last_active":"2024-09-13T11:24:05.944977+0000","last_peered":"2024-09-13T11:24:05.944977+0000","last_clean":"2024-09-13T11:24:05.944977+0000","last_became_active":"2024-09-13T11:22:14.192897+0000","last_became_peered":"2024-09-13T11:22:14.192897+0000","last_unstale":"2024-09-13T11:24:05.944977+0000","last_undegraded":"2024-09-13T11:24:05.944977+0000","last_fullsized":"2024-09-13T11:24:05.944977+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:22:13.113563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:24:09.524691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00084655899999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,10],"acting":[15,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"3.6","version":"109'4151","reported_seq":4350,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T11:24:07.043658+0000","last_change":"2024-09-13T11:23:22.656608+0000","last_active":"2024-09-13T11:24:07.043658+0000","last_peered":"2024-09-13T11:24:07.043658+0000","last_clean":"2024-09-13T11:24:07.043658+0000","last_became_active":"2024-09-13T11:23:19.939432+0000","last_became_peered":"2024-09-13T11:23:19.939432+0000","last_unstale":"2024-09-13T11:24:07.043658+0000","last_undegraded":"2024-09-13T11:24:07.043658+0000","last_fullsized":"2024-09-13T11:24:07.043658+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'2909","last_scrub_stamp":"2024-09-13T11:23:22.656323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:22.656323+0000","objects_scrubbed":1487,"log_size":4151,"log_dups_size":0,"ondisk_log_size":4151,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:24:31.905185+0000","scrub_duration":323,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":178847744,"num_objects":2729,"num_object_clones":0,"num_object_copies":5458,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2729,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1242,"num_write_kb":79488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,14],"acting":[13,14],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":188,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T11:24:06.994925+0000","last_change":"2024-09-13T11:23:21.875855+0000","last_active":"2024-09-13T11:24:06.994925+0000","last_peered":"2024-09-13T11:24:06.994925+0000","last_clean":"2024-09-13T11:24:06.994925+0000","last_became_active":"2024-09-13T11:22:14.211923+0000","last_became_peered":"2024-09-13T11:22:14.211923+0000","last_unstale":"2024-09-13T11:24:06.994925+0000","last_undegraded":"2024-09-13T11:24:06.994925+0000","last_fullsized":"2024-09-13T11:24:06.994925+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:23:21.875683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:23:21.875683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:24:40.678136+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0011689529999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0],"acting":[9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"109'3978","reported_seq":4178,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T11:24:07.040769+0000","last_change":"2024-09-13T11:23:20.443418+0000","last_active":"2024-09-13T11:24:07.040769+0000","last_peered":"2024-09-13T11:24:07.040769+0000","last_clean":"2024-09-13T11:24:07.040769+0000","last_became_active":"2024-09-13T11:23:19.927680+0000","last_became_peered":"2024-09-13T11:23:19.927680+0000","last_unstale":"2024-09-13T11:24:07.040769+0000","last_undegraded":"2024-09-13T11:24:07.040769+0000","last_fullsized":"2024-09-13T11:24:07.040769+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'2736","last_scrub_stamp":"2024-09-13T11:23:20.443329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:20.443329+0000","objects_scrubbed":1369,"log_size":3978,"log_dups_size":0,"ondisk_log_size":3978,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:24:48.147625+0000","scrub_duration":183,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":171311104,"num_objects":2614,"num_object_clones":0,"num_object_copies":5228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2614,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1245,"num_write_kb":79680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,8],"acting":[13,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":187,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T11:24:05.957598+0000","last_change":"2024-09-13T11:23:59.530794+0000","last_active":"2024-09-13T11:24:05.957598+0000","last_peered":"2024-09-13T11:24:05.957598+0000","last_clean":"2024-09-13T11:24:05.957598+0000","last_became_active":"2024-09-13T11:22:14.581458+0000","last_became_peered":"2024-09-13T11:22:14.581458+0000","last_unstale":"2024-09-13T11:24:05.957598+0000","last_undegraded":"2024-09-13T11:24:05.957598+0000","last_fullsized":"2024-09-13T11:24:05.957598+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:23:59.530670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:23:59.530670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:02.655631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001026047,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5],"acting":[8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"107'3984","reported_seq":4160,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-13T11:24:05.321411+0000","last_change":"2024-09-13T11:23:20.254213+0000","last_active":"2024-09-13T11:24:05.321411+0000","last_peered":"2024-09-13T11:24:05.321411+0000","last_clean":"2024-09-13T11:24:05.321411+0000","last_became_active":"2024-09-13T11:23:19.925399+0000","last_became_peered":"2024-09-13T11:23:19.925399+0000","last_unstale":"2024-09-13T11:24:05.321411+0000","last_undegraded":"2024-09-13T11:24:05.321411+0000","last_fullsized":"2024-09-13T11:24:05.321411+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'2755","last_scrub_stamp":"2024-09-13T11:23:20.254124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:20.254124+0000","objects_scrubbed":1437,"log_size":3984,"log_dups_size":0,"ondisk_log_size":3984,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:24:49.993543+0000","scrub_duration":187,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":175046656,"num_objects":2671,"num_object_clones":0,"num_object_copies":5342,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2671,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1234,"num_write_kb":78976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,12],"acting":[1,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":165,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-13T11:24:04.977227+0000","last_change":"2024-09-13T11:22:47.720313+0000","last_active":"2024-09-13T11:24:04.977227+0000","last_peered":"2024-09-13T11:24:04.977227+0000","last_clean":"2024-09-13T11:24:04.977227+0000","last_became_active":"2024-09-13T11:22:47.719636+0000","last_became_peered":"2024-09-13T11:22:47.719636+0000","last_unstale":"2024-09-13T11:24:04.977227+0000","last_undegraded":"2024-09-13T11:24:04.977227+0000","last_fullsized":"2024-09-13T11:24:04.977227+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:22:13.113563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00087476000000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,9],"acting":[1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"108'4001","reported_seq":4199,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T11:24:05.961301+0000","last_change":"2024-09-13T11:23:20.686446+0000","last_active":"2024-09-13T11:24:05.961301+0000","last_peered":"2024-09-13T11:24:05.961301+0000","last_clean":"2024-09-13T11:24:05.961301+0000","last_became_active":"2024-09-13T11:23:19.939695+0000","last_became_peered":"2024-09-13T11:23:19.939695+0000","last_unstale":"2024-09-13T11:24:05.961301+0000","last_undegraded":"2024-09-13T11:24:05.961301+0000","last_fullsized":"2024-09-13T11:24:05.961301+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'2739","last_scrub_stamp":"2024-09-13T11:23:20.686358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:20.686358+0000","objects_scrubbed":1386,"log_size":4001,"log_dups_size":0,"ondisk_log_size":4001,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:24:38.891737+0000","scrub_duration":271,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":173604864,"num_objects":2649,"num_object_clones":0,"num_object_copies":5298,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2649,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1263,"num_write_kb":80832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,13],"acting":[15,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"3.3","version":"107'3547","reported_seq":3768,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T11:24:05.942396+0000","last_change":"2024-09-13T11:23:32.147115+0000","last_active":"2024-09-13T11:24:05.942396+0000","last_peered":"2024-09-13T11:24:05.942396+0000","last_clean":"2024-09-13T11:24:05.942396+0000","last_became_active":"2024-09-13T11:23:31.930448+0000","last_became_peered":"2024-09-13T11:23:31.930448+0000","last_unstale":"2024-09-13T11:24:05.942396+0000","last_undegraded":"2024-09-13T11:24:05.942396+0000","last_fullsized":"2024-09-13T11:24:05.942396+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'3122","last_scrub_stamp":"2024-09-13T11:23:32.147021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:32.147021+0000","objects_scrubbed":876,"log_size":3547,"log_dups_size":0,"ondisk_log_size":3547,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:24:48.175541+0000","scrub_duration":99,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":85262336,"num_objects":1301,"num_object_clones":0,"num_object_copies":2602,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1301,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":425,"num_write_kb":27200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,7],"acting":[10,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"28'2","reported_seq":117,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T11:24:05.963496+0000","last_change":"2024-09-13T11:24:00.996197+0000","last_active":"2024-09-13T11:24:05.963496+0000","last_peered":"2024-09-13T11:24:05.963496+0000","last_clean":"2024-09-13T11:24:05.963496+0000","last_became_active":"2024-09-13T11:23:12.546348+0000","last_became_peered":"2024-09-13T11:23:12.546348+0000","last_unstale":"2024-09-13T11:24:05.963496+0000","last_undegraded":"2024-09-13T11:24:05.963496+0000","last_fullsized":"2024-09-13T11:24:05.963496+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2","last_scrub_stamp":"2024-09-13T11:24:00.996028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:24:00.996028+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:12.111097+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0025826730000000002,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8],"acting":[2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"108'3566","reported_seq":3785,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T11:24:06.031700+0000","last_change":"2024-09-13T11:23:32.749651+0000","last_active":"2024-09-13T11:24:06.031700+0000","last_peered":"2024-09-13T11:24:06.031700+0000","last_clean":"2024-09-13T11:24:06.031700+0000","last_became_active":"2024-09-13T11:23:31.921388+0000","last_became_peered":"2024-09-13T11:23:31.921388+0000","last_unstale":"2024-09-13T11:24:06.031700+0000","last_undegraded":"2024-09-13T11:24:06.031700+0000","last_fullsized":"2024-09-13T11:24:06.031700+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'3106","last_scrub_stamp":"2024-09-13T11:23:32.749487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:32.749487+0000","objects_scrubbed":856,"log_size":3566,"log_dups_size":0,"ondisk_log_size":3566,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:24:34.709501+0000","scrub_duration":91,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":86245376,"num_objects":1316,"num_object_clones":0,"num_object_copies":2632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1316,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":460,"num_write_kb":29440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,15],"acting":[12,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":189,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T11:24:05.943005+0000","last_change":"2024-09-13T11:22:47.298206+0000","last_active":"2024-09-13T11:24:05.943005+0000","last_peered":"2024-09-13T11:24:05.943005+0000","last_clean":"2024-09-13T11:24:05.943005+0000","last_became_active":"2024-09-13T11:22:47.297319+0000","last_became_peered":"2024-09-13T11:22:47.297319+0000","last_unstale":"2024-09-13T11:24:05.943005+0000","last_undegraded":"2024-09-13T11:24:05.943005+0000","last_fullsized":"2024-09-13T11:24:05.943005+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:22:13.113563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00081225399999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,1],"acting":[14,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"3.1","version":"109'3544","reported_seq":3764,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T11:24:07.048279+0000","last_change":"2024-09-13T11:23:33.604966+0000","last_active":"2024-09-13T11:24:07.048279+0000","last_peered":"2024-09-13T11:24:07.048279+0000","last_clean":"2024-09-13T11:24:07.048279+0000","last_became_active":"2024-09-13T11:23:31.926528+0000","last_became_peered":"2024-09-13T11:23:31.926528+0000","last_unstale":"2024-09-13T11:24:07.048279+0000","last_undegraded":"2024-09-13T11:24:07.048279+0000","last_fullsized":"2024-09-13T11:24:07.048279+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'3070","last_scrub_stamp":"2024-09-13T11:23:33.604792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:33.604792+0000","objects_scrubbed":842,"log_size":3544,"log_dups_size":0,"ondisk_log_size":3544,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:24:43.880038+0000","scrub_duration":119,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":86245376,"num_objects":1316,"num_object_clones":0,"num_object_copies":2632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1316,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":474,"num_write_kb":30336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,11],"acting":[13,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":177,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-13T11:24:04.954068+0000","last_change":"2024-09-13T11:22:16.267999+0000","last_active":"2024-09-13T11:24:04.954068+0000","last_peered":"2024-09-13T11:24:04.954068+0000","last_clean":"2024-09-13T11:24:04.954068+0000","last_became_active":"2024-09-13T11:22:14.211435+0000","last_became_peered":"2024-09-13T11:22:14.211435+0000","last_unstale":"2024-09-13T11:24:04.954068+0000","last_undegraded":"2024-09-13T11:24:04.954068+0000","last_fullsized":"2024-09-13T11:24:04.954068+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:22:13.113563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010010360000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2","version":"108'3556","reported_seq":3775,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T11:24:05.944539+0000","last_change":"2024-09-13T11:23:32.977666+0000","last_active":"2024-09-13T11:24:05.944539+0000","last_peered":"2024-09-13T11:24:05.944539+0000","last_clean":"2024-09-13T11:24:05.944539+0000","last_became_active":"2024-09-13T11:23:31.919828+0000","last_became_peered":"2024-09-13T11:23:31.919828+0000","last_unstale":"2024-09-13T11:24:05.944539+0000","last_undegraded":"2024-09-13T11:24:05.944539+0000","last_fullsized":"2024-09-13T11:24:05.944539+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'3057","last_scrub_stamp":"2024-09-13T11:23:32.977559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:32.977559+0000","objects_scrubbed":841,"log_size":3556,"log_dups_size":0,"ondisk_log_size":3556,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:24:52.169339+0000","scrub_duration":151,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":87818240,"num_objects":1340,"num_object_clones":0,"num_object_copies":2680,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1340,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":499,"num_write_kb":31936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,12],"acting":[14,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"2.3","version":"26'1","reported_seq":181,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T11:24:05.943338+0000","last_change":"2024-09-13T11:22:16.547691+0000","last_active":"2024-09-13T11:24:05.943338+0000","last_peered":"2024-09-13T11:24:05.943338+0000","last_clean":"2024-09-13T11:24:05.943338+0000","last_became_active":"2024-09-13T11:22:14.146507+0000","last_became_peered":"2024-09-13T11:22:14.146507+0000","last_unstale":"2024-09-13T11:24:05.943338+0000","last_undegraded":"2024-09-13T11:24:05.943338+0000","last_fullsized":"2024-09-13T11:24:05.943338+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:22:13.113563+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00085610999999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,15],"acting":[14,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"1.0","version":"26'112","reported_seq":321,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T11:24:05.945516+0000","last_change":"2024-09-13T11:23:49.325249+0000","last_active":"2024-09-13T11:24:05.945516+0000","last_peered":"2024-09-13T11:24:05.945516+0000","last_clean":"2024-09-13T11:24:05.945516+0000","last_became_active":"2024-09-13T11:22:12.129721+0000","last_became_peered":"2024-09-13T11:22:12.129721+0000","last_unstale":"2024-09-13T11:24:05.945516+0000","last_undegraded":"2024-09-13T11:24:05.945516+0000","last_fullsized":"2024-09-13T11:24:05.945516+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"26'112","last_scrub_stamp":"2024-09-13T11:23:49.325175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:11.098388+0000","last_clean_scrub_stamp":"2024-09-13T11:23:49.325175+0000","objects_scrubbed":2,"log_size":112,"log_dups_size":0,"ondisk_log_size":112,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:14.422287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.8","version":"108'3977","reported_seq":4176,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T11:24:06.016291+0000","last_change":"2024-09-13T11:23:20.644952+0000","last_active":"2024-09-13T11:24:06.016291+0000","last_peered":"2024-09-13T11:24:06.016291+0000","last_clean":"2024-09-13T11:24:06.016291+0000","last_became_active":"2024-09-13T11:23:19.925802+0000","last_became_peered":"2024-09-13T11:23:19.925802+0000","last_unstale":"2024-09-13T11:24:06.016291+0000","last_undegraded":"2024-09-13T11:24:06.016291+0000","last_fullsized":"2024-09-13T11:24:06.016291+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'2771","last_scrub_stamp":"2024-09-13T11:23:20.644747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:20.644747+0000","objects_scrubbed":1356,"log_size":3977,"log_dups_size":0,"ondisk_log_size":3977,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:24:38.144523+0000","scrub_duration":411,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":168689664,"num_objects":2574,"num_object_clones":0,"num_object_copies":5148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2574,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1218,"num_write_kb":77952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.9","version":"108'4078","reported_seq":4275,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T11:24:06.044946+0000","last_change":"2024-09-13T11:23:22.033678+0000","last_active":"2024-09-13T11:24:06.044946+0000","last_peered":"2024-09-13T11:24:06.044946+0000","last_clean":"2024-09-13T11:24:06.044946+0000","last_became_active":"2024-09-13T11:23:19.935114+0000","last_became_peered":"2024-09-13T11:23:19.935114+0000","last_unstale":"2024-09-13T11:24:06.044946+0000","last_undegraded":"2024-09-13T11:24:06.044946+0000","last_fullsized":"2024-09-13T11:24:06.044946+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'2808","last_scrub_stamp":"2024-09-13T11:23:22.033572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:22.033572+0000","objects_scrubbed":1471,"log_size":4078,"log_dups_size":0,"ondisk_log_size":4078,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:24:46.595859+0000","scrub_duration":287,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":179634176,"num_objects":2741,"num_object_clones":0,"num_object_copies":5482,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2741,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1270,"num_write_kb":81280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,2],"acting":[12,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[]},{"pgid":"3.a","version":"107'4299","reported_seq":4499,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-13T11:24:05.018512+0000","last_change":"2024-09-13T11:23:32.526150+0000","last_active":"2024-09-13T11:24:05.018512+0000","last_peered":"2024-09-13T11:24:05.018512+0000","last_clean":"2024-09-13T11:24:05.018512+0000","last_became_active":"2024-09-13T11:23:31.934170+0000","last_became_peered":"2024-09-13T11:23:31.934170+0000","last_unstale":"2024-09-13T11:24:05.018512+0000","last_undegraded":"2024-09-13T11:24:05.018512+0000","last_fullsized":"2024-09-13T11:24:05.018512+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'3418","last_scrub_stamp":"2024-09-13T11:23:32.526057+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:32.526057+0000","objects_scrubbed":1774,"log_size":4299,"log_dups_size":0,"ondisk_log_size":4299,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:24:34.770161+0000","scrub_duration":507,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":173998080,"num_objects":2655,"num_object_clones":0,"num_object_copies":5310,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2655,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":881,"num_write_kb":56384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.b","version":"108'4387","reported_seq":4590,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T11:24:06.977573+0000","last_change":"2024-09-13T11:23:32.681343+0000","last_active":"2024-09-13T11:24:06.977573+0000","last_peered":"2024-09-13T11:24:06.977573+0000","last_clean":"2024-09-13T11:24:06.977573+0000","last_became_active":"2024-09-13T11:23:31.923415+0000","last_became_peered":"2024-09-13T11:23:31.923415+0000","last_unstale":"2024-09-13T11:24:06.977573+0000","last_undegraded":"2024-09-13T11:24:06.977573+0000","last_fullsized":"2024-09-13T11:24:06.977573+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'3471","last_scrub_stamp":"2024-09-13T11:23:32.681257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:32.681257+0000","objects_scrubbed":1720,"log_size":4387,"log_dups_size":0,"ondisk_log_size":4387,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:24:58.857812+0000","scrub_duration":199,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":172752896,"num_objects":2636,"num_object_clones":0,"num_object_copies":5272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":916,"num_write_kb":58624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,1],"acting":[13,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"3.c","version":"109'4386","reported_seq":4588,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T11:24:07.044343+0000","last_change":"2024-09-13T11:23:32.524870+0000","last_active":"2024-09-13T11:24:07.044343+0000","last_peered":"2024-09-13T11:24:07.044343+0000","last_clean":"2024-09-13T11:24:07.044343+0000","last_became_active":"2024-09-13T11:23:31.922620+0000","last_became_peered":"2024-09-13T11:23:31.922620+0000","last_unstale":"2024-09-13T11:24:07.044343+0000","last_undegraded":"2024-09-13T11:24:07.044343+0000","last_fullsized":"2024-09-13T11:24:07.044343+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'3463","last_scrub_stamp":"2024-09-13T11:23:32.524776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:32.524776+0000","objects_scrubbed":1741,"log_size":4386,"log_dups_size":0,"ondisk_log_size":4386,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:24:37.745747+0000","scrub_duration":235,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":174587904,"num_objects":2664,"num_object_clones":0,"num_object_copies":5328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2664,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":923,"num_write_kb":59072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,13],"acting":[11,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.d","version":"108'4363","reported_seq":4563,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T11:24:06.133586+0000","last_change":"2024-09-13T11:23:32.776835+0000","last_active":"2024-09-13T11:24:06.133586+0000","last_peered":"2024-09-13T11:24:06.133586+0000","last_clean":"2024-09-13T11:24:06.133586+0000","last_became_active":"2024-09-13T11:23:31.939246+0000","last_became_peered":"2024-09-13T11:23:31.939246+0000","last_unstale":"2024-09-13T11:24:06.133586+0000","last_undegraded":"2024-09-13T11:24:06.133586+0000","last_fullsized":"2024-09-13T11:24:06.133586+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'3395","last_scrub_stamp":"2024-09-13T11:23:32.776747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:32.776747+0000","objects_scrubbed":1704,"log_size":4363,"log_dups_size":0,"ondisk_log_size":4363,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:24:56.402862+0000","scrub_duration":179,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":175112192,"num_objects":2672,"num_object_clones":0,"num_object_copies":5344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2672,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":968,"num_write_kb":61952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.e","version":"108'4435","reported_seq":9563,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T11:24:06.003331+0000","last_change":"2024-09-13T11:23:32.368856+0000","last_active":"2024-09-13T11:24:06.003331+0000","last_peered":"2024-09-13T11:24:06.003331+0000","last_clean":"2024-09-13T11:24:06.003331+0000","last_became_active":"2024-09-13T11:23:31.925869+0000","last_became_peered":"2024-09-13T11:23:31.925869+0000","last_unstale":"2024-09-13T11:24:06.003331+0000","last_undegraded":"2024-09-13T11:24:06.003331+0000","last_fullsized":"2024-09-13T11:24:06.003331+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'3527","last_scrub_stamp":"2024-09-13T11:23:32.368727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:32.368727+0000","objects_scrubbed":1774,"log_size":4435,"log_dups_size":0,"ondisk_log_size":4435,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:24:57.811779+0000","scrub_duration":219,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":175833088,"num_objects":2683,"num_object_clones":0,"num_object_copies":5366,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2683,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":909,"num_write_kb":58176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"109'4353","reported_seq":4555,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T11:24:06.962608+0000","last_change":"2024-09-13T11:23:33.492630+0000","last_active":"2024-09-13T11:24:06.962608+0000","last_peered":"2024-09-13T11:24:06.962608+0000","last_clean":"2024-09-13T11:24:06.962608+0000","last_became_active":"2024-09-13T11:23:31.935239+0000","last_became_peered":"2024-09-13T11:23:31.935239+0000","last_unstale":"2024-09-13T11:24:06.962608+0000","last_undegraded":"2024-09-13T11:24:06.962608+0000","last_fullsized":"2024-09-13T11:24:06.962608+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'3420","last_scrub_stamp":"2024-09-13T11:23:33.492488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:33.492488+0000","objects_scrubbed":1776,"log_size":4353,"log_dups_size":0,"ondisk_log_size":4353,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:24:44.851261+0000","scrub_duration":215,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":177537024,"num_objects":2709,"num_object_clones":0,"num_object_copies":5418,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2709,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":933,"num_write_kb":59712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,12],"acting":[11,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.10","version":"108'4131","reported_seq":8487,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T11:24:06.004696+0000","last_change":"2024-09-13T11:23:48.519654+0000","last_active":"2024-09-13T11:24:06.004696+0000","last_peered":"2024-09-13T11:24:06.004696+0000","last_clean":"2024-09-13T11:24:06.004696+0000","last_became_active":"2024-09-13T11:23:41.931183+0000","last_became_peered":"2024-09-13T11:23:41.931183+0000","last_unstale":"2024-09-13T11:24:06.004696+0000","last_undegraded":"2024-09-13T11:24:06.004696+0000","last_fullsized":"2024-09-13T11:24:06.004696+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":5,"last_scrub":"91'3468","last_scrub_stamp":"2024-09-13T11:23:48.519574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:48.519574+0000","objects_scrubbed":2031,"log_size":4131,"log_dups_size":0,"ondisk_log_size":4131,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:24:57.342213+0000","scrub_duration":215,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":176685056,"num_objects":2696,"num_object_clones":0,"num_object_copies":5392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2696,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":665,"num_write_kb":42560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.11","version":"108'4005","reported_seq":7139,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T11:24:06.005767+0000","last_change":"2024-09-13T11:23:46.102466+0000","last_active":"2024-09-13T11:24:06.005767+0000","last_peered":"2024-09-13T11:24:06.005767+0000","last_clean":"2024-09-13T11:24:06.005767+0000","last_became_active":"2024-09-13T11:23:39.063172+0000","last_became_peered":"2024-09-13T11:23:39.063172+0000","last_unstale":"2024-09-13T11:24:06.005767+0000","last_undegraded":"2024-09-13T11:24:06.005767+0000","last_fullsized":"2024-09-13T11:24:06.005767+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":5,"last_scrub":"88'3345","last_scrub_stamp":"2024-09-13T11:23:46.101973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:46.101973+0000","objects_scrubbed":2011,"log_size":4005,"log_dups_size":0,"ondisk_log_size":4005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:08.671350+0000","scrub_duration":955,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":175308800,"num_objects":2675,"num_object_clones":0,"num_object_copies":5350,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2675,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":664,"num_write_kb":42496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11],"acting":[4,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.12","version":"108'4004","reported_seq":7062,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T11:24:06.047245+0000","last_change":"2024-09-13T11:23:35.111409+0000","last_active":"2024-09-13T11:24:06.047245+0000","last_peered":"2024-09-13T11:24:06.047245+0000","last_clean":"2024-09-13T11:24:06.047245+0000","last_became_active":"2024-09-13T11:23:21.998877+0000","last_became_peered":"2024-09-13T11:23:21.998877+0000","last_unstale":"2024-09-13T11:24:06.047245+0000","last_undegraded":"2024-09-13T11:24:06.047245+0000","last_fullsized":"2024-09-13T11:24:06.047245+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'3109","last_scrub_stamp":"2024-09-13T11:23:35.111320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:35.111320+0000","objects_scrubbed":1732,"log_size":4004,"log_dups_size":0,"ondisk_log_size":4004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:03.371433+0000","scrub_duration":239,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":172163072,"num_objects":2627,"num_object_clones":0,"num_object_copies":5254,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2627,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":895,"num_write_kb":57280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,7],"acting":[14,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"3.13","version":"109'4069","reported_seq":7130,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T11:24:07.048487+0000","last_change":"2024-09-13T11:23:47.565094+0000","last_active":"2024-09-13T11:24:07.048487+0000","last_peered":"2024-09-13T11:24:07.048487+0000","last_clean":"2024-09-13T11:24:07.048487+0000","last_became_active":"2024-09-13T11:23:41.082433+0000","last_became_peered":"2024-09-13T11:23:41.082433+0000","last_unstale":"2024-09-13T11:24:07.048487+0000","last_undegraded":"2024-09-13T11:24:07.048487+0000","last_fullsized":"2024-09-13T11:24:07.048487+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'3408","last_scrub_stamp":"2024-09-13T11:23:47.564931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:47.564931+0000","objects_scrubbed":1981,"log_size":4069,"log_dups_size":0,"ondisk_log_size":4069,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:04.258231+0000","scrub_duration":471,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":173932544,"num_objects":2654,"num_object_clones":0,"num_object_copies":5308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2654,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":673,"num_write_kb":43072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7],"acting":[11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":36,"stat_sum":{"num_bytes":5572853760,"num_objects":85036,"num_object_clones":0,"num_object_copies":170072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":85036,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":38507,"num_write_kb":2464384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11626,"num_bytes_recovered":761888768,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":13115850752,"data_stored":13115850752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":145482,"ondisk_log_size":145482,"up":72,"acting":72,"num_store_stats":16},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":10},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":112,"ondisk_log_size":112,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":13,"up_from":23,"seq":98784247835,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1008836,"kb_used_data":933236,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":93363004,"statfs":{"total":96636764160,"available":95603716096,"internally_reserved":0,"allocated":955633664,"data_stored":955156526,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,1,3,4,6,8,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":9,"up_from":23,"seq":98784247834,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":450936,"kb_used_data":415592,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93920904,"statfs":{"total":96636764160,"available":96175005696,"internally_reserved":0,"allocated":425566208,"data_stored":425100675,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,8,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":40,"seq":171798691857,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1343948,"kb_used_data":1267516,"kb_used_omap":4,"kb_used_meta":76411,"kb_avail":93027892,"statfs":{"total":96636764160,"available":95260561408,"internally_reserved":0,"allocated":1297936384,"data_stored":1297445342,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4433,"internal_metadata":78245551},"hb_peers":[0,2,3,6,8,9,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":21,"seq":90194313242,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1019848,"kb_used_data":966456,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":93351992,"statfs":{"total":96636764160,"available":95592439808,"internally_reserved":0,"allocated":989650944,"data_stored":989161893,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":54655431},"hb_peers":[0,2,4,7,9,10,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":21,"seq":90194313245,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":286288,"kb_used_data":259264,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94085552,"statfs":{"total":96636764160,"available":96343605248,"internally_reserved":0,"allocated":265486336,"data_stored":264993109,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,6,8,9,10,11,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":22,"seq":94489280537,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1149000,"kb_used_data":1085880,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":93222840,"statfs":{"total":96636764160,"available":95460188160,"internally_reserved":0,"allocated":1111941120,"data_stored":1111516713,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[1,3,4,5,8,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":22,"seq":94489280538,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":445056,"kb_used_data":405552,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":93926784,"statfs":{"total":96636764160,"available":96181026816,"internally_reserved":0,"allocated":415285248,"data_stored":414812191,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,3,7,8,9,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":15,"up_from":22,"seq":94489280538,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":949184,"kb_used_data":873584,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":93422656,"statfs":{"total":96636764160,"available":95664799744,"internally_reserved":0,"allocated":894550016,"data_stored":894133583,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":77396422},"hb_peers":[0,3,4,5,6,7,10,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":22,"seq":94489280538,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":546264,"kb_used_data":497032,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":93825576,"statfs":{"total":96636764160,"available":96077389824,"internally_reserved":0,"allocated":508960768,"data_stored":508465062,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[3,5,7,9,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":22,"seq":94489280538,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":429992,"kb_used_data":394648,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93941848,"statfs":{"total":96636764160,"available":96196452352,"internally_reserved":0,"allocated":404119552,"data_stored":403666123,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,3,5,7,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":22,"seq":94489280538,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1232220,"kb_used_data":1164940,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":93139620,"statfs":{"total":96636764160,"available":95374970880,"internally_reserved":0,"allocated":1192898560,"data_stored":1192454834,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":68876746},"hb_peers":[0,3,5,6,8,9,10,11,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":22,"seq":94489280538,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":489692,"kb_used_data":458508,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93882148,"statfs":{"total":96636764160,"available":96135319552,"internally_reserved":0,"allocated":469512192,"data_stored":469008420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,5,7,9,10,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":22,"seq":94489280537,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":699360,"kb_used_data":641808,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":93672480,"statfs":{"total":96636764160,"available":95920619520,"internally_reserved":0,"allocated":657211392,"data_stored":656765555,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":58915264},"hb_peers":[0,1,3,6,7,9,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":11,"up_from":22,"seq":94489280539,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1361604,"kb_used_data":1281844,"kb_used_omap":1,"kb_used_meta":79742,"kb_avail":93010236,"statfs":{"total":96636764160,"available":95242481664,"internally_reserved":0,"allocated":1312608256,"data_stored":1312134741,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":81656260},"hb_peers":[0,2,4,7,8,10,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":12,"up_from":22,"seq":94489280538,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1116988,"kb_used_data":1053868,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":93254852,"statfs":{"total":96636764160,"available":95492968448,"internally_reserved":0,"allocated":1079160832,"data_stored":1078688509,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[0,1,2,3,5,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":14,"up_from":22,"seq":94489280538,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1184908,"kb_used_data":1113468,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":93186932,"statfs":{"total":96636764160,"available":95423418368,"internally_reserved":0,"allocated":1140191232,"data_stored":1139708545,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73136581},"hb_peers":[0,1,5,6,7,8,9,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1123614720,"data_stored":1123614720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1298071552,"data_stored":1298071552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":419233792,"data_stored":419233792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":987889664,"data_stored":987889664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":514981888,"data_stored":514981888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":264568832,"data_stored":264568832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":405602304,"data_stored":405602304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1192886272,"data_stored":1192886272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":466944000,"data_stored":466944000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":425328640,"data_stored":425328640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":663617536,"data_stored":663617536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":1274019840,"data_stored":1274019840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":1081606144,"data_stored":1081606144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":958332928,"data_stored":958332928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":1139933184,"data_stored":1139933184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":899219456,"data_stored":899219456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T11:24:10.230 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-13T11:24:10.243 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-13T11:24:10.243 INFO:teuthology.orchestra.run.smithi001.stdout: "size": 20, 2024-09-13T11:24:10.243 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 600, 2024-09-13T11:24:10.243 INFO:teuthology.orchestra.run.smithi001.stdout: "ops": [ 2024-09-13T11:24:10.243 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.243 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDPGPush(3.11 72/70 [PushOp(3:8ae6b6fc:::benchmark_data_smithi177_31116_object7617:head, version: 46'455, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:8ae6b6fc:::benchmark_data_smithi177_31116_object7617:head@46'455, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:24:10.243 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:26.176296+0000", 2024-09-13T11:24:10.243 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 44.064140844999997, 2024-09-13T11:24:10.243 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.13349134900000001, 2024-09-13T11:24:10.243 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:24:10.243 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:24:10.244 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:24:10.244 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:24:10.244 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.244 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:24:10.244 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:26.176296+0000", 2024-09-13T11:24:10.244 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:10.244 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.244 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.244 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:24:10.244 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:26.176296+0000", 2024-09-13T11:24:10.244 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:10.244 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.244 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.245 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:24:10.245 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:26.176297+0000", 2024-09-13T11:24:10.245 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2440000000000001e-06 2024-09-13T11:24:10.245 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.245 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.245 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:24:10.245 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:26.176433+0000", 2024-09-13T11:24:10.245 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00013554999999999999 2024-09-13T11:24:10.245 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.245 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.245 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:24:10.245 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:26.308747+0000", 2024-09-13T11:24:10.245 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.132314392 2024-09-13T11:24:10.246 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.246 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.246 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:24:10.246 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:26.308768+0000", 2024-09-13T11:24:10.246 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.0758999999999999e-05 2024-09-13T11:24:10.246 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.246 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.246 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:24:10.246 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:26.308874+0000", 2024-09-13T11:24:10.246 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000105334 2024-09-13T11:24:10.246 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.246 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.246 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:24:10.246 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:26.308890+0000", 2024-09-13T11:24:10.247 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.6367000000000001e-05 2024-09-13T11:24:10.247 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.247 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.247 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:24:10.247 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:26.309788+0000", 2024-09-13T11:24:10.247 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00089770300000000002 2024-09-13T11:24:10.247 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:10.247 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:24:10.247 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:10.247 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.247 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.247 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDPGPush(3.11 72/70 [PushOp(3:8da9c977:::benchmark_data_smithi177_31116_object7736:head, version: 46'461, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:8da9c977:::benchmark_data_smithi177_31116_object7736:head@46'461, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:24:10.247 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:26.393216+0000", 2024-09-13T11:24:10.247 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 43.847220985999996, 2024-09-13T11:24:10.248 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.15683955499999999, 2024-09-13T11:24:10.248 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:24:10.248 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:24:10.248 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:24:10.248 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:24:10.248 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.248 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:24:10.248 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:26.393216+0000", 2024-09-13T11:24:10.248 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:10.248 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.248 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.248 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:24:10.248 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:26.393216+0000", 2024-09-13T11:24:10.249 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:10.249 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.249 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.249 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:24:10.249 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:26.393217+0000", 2024-09-13T11:24:10.249 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2410000000000001e-06 2024-09-13T11:24:10.249 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.249 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.249 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:24:10.249 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:26.393309+0000", 2024-09-13T11:24:10.249 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.2101000000000005e-05 2024-09-13T11:24:10.249 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.249 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.249 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:24:10.250 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:26.547936+0000", 2024-09-13T11:24:10.250 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.15462682699999999 2024-09-13T11:24:10.250 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.250 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.250 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:24:10.250 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:26.548021+0000", 2024-09-13T11:24:10.250 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.4529000000000003e-05 2024-09-13T11:24:10.250 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.250 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.250 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:24:10.250 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:26.548321+0000", 2024-09-13T11:24:10.250 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00029988399999999999 2024-09-13T11:24:10.250 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.250 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.251 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:24:10.251 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:26.548353+0000", 2024-09-13T11:24:10.251 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.2573000000000001e-05 2024-09-13T11:24:10.251 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.251 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.251 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:24:10.251 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:26.550056+0000", 2024-09-13T11:24:10.251 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0017024 2024-09-13T11:24:10.251 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:10.251 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:24:10.251 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:10.251 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.251 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.252 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4665.0:54052 3.11 e74/70 3:8c6be821:::benchmark_data_smithi177_31116_object54051:head v 74'3049, mlcod=55'1352)", 2024-09-13T11:24:10.252 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:30.314329+0000", 2024-09-13T11:24:10.252 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 39.926107592000001, 2024-09-13T11:24:10.252 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.100070243, 2024-09-13T11:24:10.252 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:24:10.252 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:24:10.252 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:10.252 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:24:10.252 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.252 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:24:10.252 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:30.314329+0000", 2024-09-13T11:24:10.252 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:10.252 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.252 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.253 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:24:10.253 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:30.314329+0000", 2024-09-13T11:24:10.253 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:10.253 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.253 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.253 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:24:10.253 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:30.314332+0000", 2024-09-13T11:24:10.253 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.1900000000000002e-06 2024-09-13T11:24:10.253 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.253 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.253 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:24:10.253 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:30.314372+0000", 2024-09-13T11:24:10.253 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.0788000000000003e-05 2024-09-13T11:24:10.253 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.254 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.254 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:24:10.254 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:30.412014+0000", 2024-09-13T11:24:10.254 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.097641172999999998 2024-09-13T11:24:10.254 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.254 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.254 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:24:10.254 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:30.412026+0000", 2024-09-13T11:24:10.254 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2551999999999999e-05 2024-09-13T11:24:10.254 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.254 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.254 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:24:10.254 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:30.412134+0000", 2024-09-13T11:24:10.255 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00010763400000000001 2024-09-13T11:24:10.255 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.255 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.255 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:24:10.255 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:30.412172+0000", 2024-09-13T11:24:10.255 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.8333e-05 2024-09-13T11:24:10.255 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.255 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.255 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:24:10.255 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:30.414340+0000", 2024-09-13T11:24:10.255 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0021680879999999999 2024-09-13T11:24:10.255 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.255 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.255 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:24:10.256 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:30.414400+0000", 2024-09-13T11:24:10.256 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.9484999999999997e-05 2024-09-13T11:24:10.256 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:10.256 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:24:10.256 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:10.256 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.256 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.256 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDPGPush(3.1e 80/78 [PushOp(3:7c63aba7:::benchmark_data_smithi177_31116_object2869:head, version: 44'199, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:7c63aba7:::benchmark_data_smithi177_31116_object2869:head@44'199, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:24:10.256 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:36.987368+0000", 2024-09-13T11:24:10.256 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 33.253068648000003, 2024-09-13T11:24:10.256 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.078572671999999996, 2024-09-13T11:24:10.256 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:24:10.256 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:24:10.256 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:24:10.257 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:24:10.257 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.257 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:24:10.257 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:36.987368+0000", 2024-09-13T11:24:10.257 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:10.257 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.257 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.257 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:24:10.257 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:36.987368+0000", 2024-09-13T11:24:10.257 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:10.257 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.257 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.257 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:24:10.257 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:36.987370+0000", 2024-09-13T11:24:10.258 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.66e-06 2024-09-13T11:24:10.258 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.258 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.258 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:24:10.258 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:36.987402+0000", 2024-09-13T11:24:10.258 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.2406000000000001e-05 2024-09-13T11:24:10.258 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.258 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.258 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:24:10.258 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:37.065145+0000", 2024-09-13T11:24:10.258 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.077742364999999994 2024-09-13T11:24:10.258 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.258 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.259 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:24:10.259 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:37.065164+0000", 2024-09-13T11:24:10.259 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.8706999999999999e-05 2024-09-13T11:24:10.259 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.259 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.259 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:24:10.259 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:37.065273+0000", 2024-09-13T11:24:10.259 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000109788 2024-09-13T11:24:10.259 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.259 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.260 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:24:10.260 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:37.065288+0000", 2024-09-13T11:24:10.261 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.4649e-05 2024-09-13T11:24:10.261 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.261 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.261 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:24:10.261 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:37.065941+0000", 2024-09-13T11:24:10.261 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00065309700000000005 2024-09-13T11:24:10.261 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:10.261 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:24:10.262 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:10.262 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.262 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.262 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDPGPush(3.1a 86/78 [PushOp(3:5df655d4:::benchmark_data_smithi177_31116_object12486:head, version: 49'767, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:5df655d4:::benchmark_data_smithi177_31116_object12486:head@49'767, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:24:10.262 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:43.456210+0000", 2024-09-13T11:24:10.262 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 26.784226684, 2024-09-13T11:24:10.262 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.131081792, 2024-09-13T11:24:10.262 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:24:10.262 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:24:10.262 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:24:10.262 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:24:10.262 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.262 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:24:10.263 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:43.456210+0000", 2024-09-13T11:24:10.263 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:10.263 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.263 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.263 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:24:10.263 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:43.456210+0000", 2024-09-13T11:24:10.263 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:10.263 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.263 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.263 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:24:10.263 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:43.456212+0000", 2024-09-13T11:24:10.263 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.922e-06 2024-09-13T11:24:10.263 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.263 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.264 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:24:10.264 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:43.456262+0000", 2024-09-13T11:24:10.264 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.9242000000000001e-05 2024-09-13T11:24:10.264 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.264 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.264 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:24:10.264 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:43.586188+0000", 2024-09-13T11:24:10.264 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.12992674400000001 2024-09-13T11:24:10.264 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.264 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.264 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:24:10.264 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:43.586209+0000", 2024-09-13T11:24:10.264 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.031e-05 2024-09-13T11:24:10.264 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.265 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.265 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:24:10.265 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:43.586325+0000", 2024-09-13T11:24:10.265 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000115983 2024-09-13T11:24:10.265 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.265 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.265 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:24:10.265 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:43.586340+0000", 2024-09-13T11:24:10.265 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.4941e-05 2024-09-13T11:24:10.265 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.265 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.265 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:24:10.265 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:43.587292+0000", 2024-09-13T11:24:10.266 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00095264999999999998 2024-09-13T11:24:10.266 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:10.266 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:24:10.266 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:10.266 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.266 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.266 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDPGPush(3.1e 89/78 [PushOp(3:7a07f38c:::benchmark_data_smithi177_31116_object20966:head, version: 54'1402, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:7a07f38c:::benchmark_data_smithi177_31116_object20966:head@54'1402, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:24:10.266 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:46.645251+0000", 2024-09-13T11:24:10.266 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 23.595186270999999, 2024-09-13T11:24:10.266 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.17670883800000001, 2024-09-13T11:24:10.266 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:24:10.266 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:24:10.266 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:24:10.266 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:24:10.267 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.267 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:24:10.267 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.645251+0000", 2024-09-13T11:24:10.267 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:10.267 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.267 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.267 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:24:10.267 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.645251+0000", 2024-09-13T11:24:10.267 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:10.267 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.267 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.267 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:24:10.267 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.645253+0000", 2024-09-13T11:24:10.268 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.1399999999999998e-06 2024-09-13T11:24:10.268 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.268 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.268 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:24:10.268 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.645295+0000", 2024-09-13T11:24:10.268 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.2323000000000001e-05 2024-09-13T11:24:10.268 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.268 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.268 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:24:10.268 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.820731+0000", 2024-09-13T11:24:10.268 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.17543607999999999 2024-09-13T11:24:10.268 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.268 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.269 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:24:10.269 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.820750+0000", 2024-09-13T11:24:10.269 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.8793000000000001e-05 2024-09-13T11:24:10.269 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.269 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.269 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:24:10.269 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.820914+0000", 2024-09-13T11:24:10.269 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000164062 2024-09-13T11:24:10.269 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.269 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.269 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:24:10.269 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.820939+0000", 2024-09-13T11:24:10.269 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.4672e-05 2024-09-13T11:24:10.269 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.270 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.270 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:24:10.270 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.821960+0000", 2024-09-13T11:24:10.270 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.001020768 2024-09-13T11:24:10.270 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:10.270 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:24:10.270 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:10.270 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.270 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.270 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:64320 3.11 3:8de5de4c:::benchmark_data_smithi177_31116_object64319:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90)", 2024-09-13T11:24:10.271 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:47.132785+0000", 2024-09-13T11:24:10.271 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 23.107651969999999, 2024-09-13T11:24:10.271 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.11946314600000001, 2024-09-13T11:24:10.271 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:24:10.271 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:24:10.271 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:10.271 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:24:10.271 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:24:10.271 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:10.271 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 64320 2024-09-13T11:24:10.271 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.271 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:24:10.271 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.272 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:24:10.272 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:47.132785+0000", 2024-09-13T11:24:10.272 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:10.272 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.272 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.272 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:24:10.272 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:47.132785+0000", 2024-09-13T11:24:10.272 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:10.272 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.272 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.272 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:24:10.272 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:47.132788+0000", 2024-09-13T11:24:10.272 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.4619999999999999e-06 2024-09-13T11:24:10.273 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.273 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.273 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:24:10.273 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:47.132838+0000", 2024-09-13T11:24:10.273 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.0739000000000003e-05 2024-09-13T11:24:10.273 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.273 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.273 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:24:10.273 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:47.132853+0000", 2024-09-13T11:24:10.273 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.518e-05 2024-09-13T11:24:10.273 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.273 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.273 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:24:10.274 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:47.132860+0000", 2024-09-13T11:24:10.274 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.8689999999999998e-06 2024-09-13T11:24:10.274 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.274 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.274 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:24:10.274 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:47.132922+0000", 2024-09-13T11:24:10.274 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.1251e-05 2024-09-13T11:24:10.274 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.274 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.274 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:24:10.274 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:47.133034+0000", 2024-09-13T11:24:10.276 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000112742 2024-09-13T11:24:10.276 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.276 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.276 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 11", 2024-09-13T11:24:10.276 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:47.133141+0000", 2024-09-13T11:24:10.276 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000106968 2024-09-13T11:24:10.276 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.276 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.276 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:24:10.277 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:47.135566+0000", 2024-09-13T11:24:10.277 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0024251839999999999 2024-09-13T11:24:10.277 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.277 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.277 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:10.277 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:47.252135+0000", 2024-09-13T11:24:10.277 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.11656870900000001 2024-09-13T11:24:10.277 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.277 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.277 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:24:10.277 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:47.252201+0000", 2024-09-13T11:24:10.277 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.6223999999999995e-05 2024-09-13T11:24:10.277 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.278 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.278 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:24:10.278 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:47.252248+0000", 2024-09-13T11:24:10.278 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.6817999999999998e-05 2024-09-13T11:24:10.278 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:10.278 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:24:10.278 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:10.278 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.278 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.278 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDPGPush(3.1e 90/78 [PushOp(3:7d0d3b0f:::benchmark_data_smithi177_31116_object22844:head, version: 55'1508, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:7d0d3b0f:::benchmark_data_smithi177_31116_object22844:head@55'1508, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:24:10.278 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:47.846941+0000", 2024-09-13T11:24:10.278 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 22.393495615999999, 2024-09-13T11:24:10.278 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.074054810999999998, 2024-09-13T11:24:10.278 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:24:10.279 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:24:10.279 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:24:10.279 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:24:10.279 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.279 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:24:10.279 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:47.846941+0000", 2024-09-13T11:24:10.279 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:10.279 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.279 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.279 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:24:10.279 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:47.846941+0000", 2024-09-13T11:24:10.279 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:10.279 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.279 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.280 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:24:10.280 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:47.846944+0000", 2024-09-13T11:24:10.280 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.8830000000000002e-06 2024-09-13T11:24:10.280 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.280 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.280 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:24:10.280 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:47.846999+0000", 2024-09-13T11:24:10.280 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.4326000000000001e-05 2024-09-13T11:24:10.280 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.280 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.280 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:24:10.280 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:47.919220+0000", 2024-09-13T11:24:10.280 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.072221718000000004 2024-09-13T11:24:10.280 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.281 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.281 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:24:10.281 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:47.919246+0000", 2024-09-13T11:24:10.281 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.5230000000000001e-05 2024-09-13T11:24:10.281 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.281 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.281 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:24:10.281 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:47.919392+0000", 2024-09-13T11:24:10.281 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00014595999999999999 2024-09-13T11:24:10.281 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.281 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.281 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:24:10.281 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:47.919415+0000", 2024-09-13T11:24:10.282 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.3414e-05 2024-09-13T11:24:10.282 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.282 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.282 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:24:10.282 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:47.920996+0000", 2024-09-13T11:24:10.282 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00158128 2024-09-13T11:24:10.282 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:10.282 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:24:10.282 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:10.282 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.282 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.282 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDPGPush(3.1e 91/78 [PushOp(3:7ba8a664:::benchmark_data_smithi177_31116_object23405:head, version: 55'1537, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:7ba8a664:::benchmark_data_smithi177_31116_object23405:head@55'1537, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:24:10.282 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:48.176337+0000", 2024-09-13T11:24:10.283 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 22.064100074999999, 2024-09-13T11:24:10.283 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.073969752999999999, 2024-09-13T11:24:10.283 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:24:10.283 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:24:10.283 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:24:10.283 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:24:10.283 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.283 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:24:10.283 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:48.176337+0000", 2024-09-13T11:24:10.283 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:10.283 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.283 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.283 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:24:10.284 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:48.176337+0000", 2024-09-13T11:24:10.284 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:10.284 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.284 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.284 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:24:10.284 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:48.176339+0000", 2024-09-13T11:24:10.284 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.4080000000000001e-06 2024-09-13T11:24:10.284 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.284 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.285 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:24:10.285 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:48.176371+0000", 2024-09-13T11:24:10.285 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.1930000000000001e-05 2024-09-13T11:24:10.285 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.285 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.285 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:24:10.285 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:48.249309+0000", 2024-09-13T11:24:10.285 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.072937362000000006 2024-09-13T11:24:10.285 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.286 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.286 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:24:10.286 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:48.249325+0000", 2024-09-13T11:24:10.286 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.6538e-05 2024-09-13T11:24:10.286 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.286 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.286 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:24:10.286 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:48.249436+0000", 2024-09-13T11:24:10.286 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000111241 2024-09-13T11:24:10.286 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.286 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.286 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:24:10.286 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:48.249455+0000", 2024-09-13T11:24:10.287 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.8848999999999999e-05 2024-09-13T11:24:10.287 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.287 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.287 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:24:10.287 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:48.250307+0000", 2024-09-13T11:24:10.287 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00085142499999999995 2024-09-13T11:24:10.287 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:10.287 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:24:10.287 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:10.287 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.287 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.288 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4665.0:65304 3.1e e91/78 3:79ea72f2:::benchmark_data_smithi177_31116_object65303:head v 91'3873, mlcod=55'1534)", 2024-09-13T11:24:10.288 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:48.249255+0000", 2024-09-13T11:24:10.288 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 21.991182554000002, 2024-09-13T11:24:10.288 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.089013825000000005, 2024-09-13T11:24:10.288 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:24:10.288 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:24:10.288 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:10.288 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:24:10.288 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.288 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:24:10.288 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:48.249255+0000", 2024-09-13T11:24:10.288 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:10.289 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.289 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.289 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:24:10.289 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:48.249255+0000", 2024-09-13T11:24:10.289 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:10.289 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.289 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.289 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:24:10.289 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:48.249256+0000", 2024-09-13T11:24:10.289 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.1599999999999999e-06 2024-09-13T11:24:10.289 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.289 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.289 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:24:10.290 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:48.249288+0000", 2024-09-13T11:24:10.290 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.1987999999999999e-05 2024-09-13T11:24:10.290 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.290 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.290 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:24:10.290 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:48.335995+0000", 2024-09-13T11:24:10.290 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.086707291000000006 2024-09-13T11:24:10.290 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.290 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.290 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:24:10.290 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:48.336002+0000", 2024-09-13T11:24:10.290 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.2030000000000003e-06 2024-09-13T11:24:10.290 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.291 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.291 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:24:10.291 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:48.336060+0000", 2024-09-13T11:24:10.291 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.8010999999999997e-05 2024-09-13T11:24:10.291 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.291 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.291 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:24:10.291 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:48.336095+0000", 2024-09-13T11:24:10.291 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.4819e-05 2024-09-13T11:24:10.291 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.291 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.291 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:24:10.291 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:48.338197+0000", 2024-09-13T11:24:10.292 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.002102217 2024-09-13T11:24:10.292 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.292 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.292 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:24:10.292 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:48.338268+0000", 2024-09-13T11:24:10.292 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.1136000000000005e-05 2024-09-13T11:24:10.292 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:10.292 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:24:10.292 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:10.292 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.293 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.293 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDPGPush(3.1e 91/78 [PushOp(3:799e2e7a:::benchmark_data_smithi177_31116_object23408:head, version: 55'1538, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:799e2e7a:::benchmark_data_smithi177_31116_object23408:head@55'1538, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:24:10.293 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:48.255362+0000", 2024-09-13T11:24:10.293 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 21.985074609000002, 2024-09-13T11:24:10.293 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.082831283000000006, 2024-09-13T11:24:10.293 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:24:10.293 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:24:10.294 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:24:10.294 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:24:10.294 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.294 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:24:10.294 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:48.255362+0000", 2024-09-13T11:24:10.294 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:10.294 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.294 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.294 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:24:10.294 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:48.255362+0000", 2024-09-13T11:24:10.294 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:10.294 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.294 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.295 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:24:10.295 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:48.255364+0000", 2024-09-13T11:24:10.295 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.7039999999999999e-06 2024-09-13T11:24:10.295 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.295 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.295 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:24:10.295 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:48.255390+0000", 2024-09-13T11:24:10.295 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.5829999999999998e-05 2024-09-13T11:24:10.295 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.295 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.295 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:24:10.295 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:48.336037+0000", 2024-09-13T11:24:10.295 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.080647336999999999 2024-09-13T11:24:10.296 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.296 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.296 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:24:10.296 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:48.336054+0000", 2024-09-13T11:24:10.296 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.7110999999999999e-05 2024-09-13T11:24:10.296 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.296 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.296 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:24:10.296 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:48.337347+0000", 2024-09-13T11:24:10.296 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.001292739 2024-09-13T11:24:10.296 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.296 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.296 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:24:10.297 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:48.337361+0000", 2024-09-13T11:24:10.297 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.3696999999999999e-05 2024-09-13T11:24:10.297 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.297 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.297 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:24:10.297 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:48.338194+0000", 2024-09-13T11:24:10.297 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00083286500000000002 2024-09-13T11:24:10.297 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:10.297 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:24:10.297 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:10.297 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.297 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.298 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDPGPush(3.1e 91/78 [PushOp(3:78b7dd1b:::benchmark_data_smithi177_31116_object23441:head, version: 55'1540, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:78b7dd1b:::benchmark_data_smithi177_31116_object23441:head@55'1540, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:24:10.298 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:48.342756+0000", 2024-09-13T11:24:10.298 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 21.897680898000001, 2024-09-13T11:24:10.298 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.114126449, 2024-09-13T11:24:10.298 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:24:10.298 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:24:10.298 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:24:10.298 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:24:10.298 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.298 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:24:10.298 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:48.342756+0000", 2024-09-13T11:24:10.299 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:10.299 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.299 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.299 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:24:10.299 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:48.342756+0000", 2024-09-13T11:24:10.299 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:10.299 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.299 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.299 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:24:10.299 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:48.342758+0000", 2024-09-13T11:24:10.299 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.9769999999999999e-06 2024-09-13T11:24:10.299 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.299 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.300 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:24:10.300 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:48.342796+0000", 2024-09-13T11:24:10.300 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.8331000000000003e-05 2024-09-13T11:24:10.300 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.300 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.300 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:24:10.300 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:48.455578+0000", 2024-09-13T11:24:10.300 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.11278160800000001 2024-09-13T11:24:10.300 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.300 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.300 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:24:10.300 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:48.455631+0000", 2024-09-13T11:24:10.300 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.3229999999999997e-05 2024-09-13T11:24:10.301 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.301 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.301 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:24:10.301 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:48.455799+0000", 2024-09-13T11:24:10.301 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00016763100000000001 2024-09-13T11:24:10.301 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.301 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.301 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:24:10.301 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:48.455813+0000", 2024-09-13T11:24:10.302 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.4416999999999999e-05 2024-09-13T11:24:10.302 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.302 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.302 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:24:10.302 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:48.456883+0000", 2024-09-13T11:24:10.302 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0010692550000000001 2024-09-13T11:24:10.302 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:10.302 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:24:10.302 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:10.302 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.302 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.302 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDPGPush(3.1e 96/78 [PushOp(3:7b6149da:::benchmark_data_smithi177_31116_object33423:head, version: 60'2163, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:7b6149da:::benchmark_data_smithi177_31116_object33423:head@60'2163, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:24:10.303 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:53.486432+0000", 2024-09-13T11:24:10.303 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 16.754005517, 2024-09-13T11:24:10.303 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.075823480999999998, 2024-09-13T11:24:10.303 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:24:10.303 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:24:10.303 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:24:10.303 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:24:10.303 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.304 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:24:10.304 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:53.486432+0000", 2024-09-13T11:24:10.304 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:10.304 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.304 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.304 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:24:10.304 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:53.486432+0000", 2024-09-13T11:24:10.304 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:10.304 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.304 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.304 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:24:10.305 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:53.486433+0000", 2024-09-13T11:24:10.305 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.3289999999999999e-06 2024-09-13T11:24:10.305 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.305 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.305 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:24:10.305 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:53.486457+0000", 2024-09-13T11:24:10.305 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.4315e-05 2024-09-13T11:24:10.305 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.305 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.305 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:24:10.305 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:53.560990+0000", 2024-09-13T11:24:10.305 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.074532452999999999 2024-09-13T11:24:10.305 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.306 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.306 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:24:10.306 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:53.561001+0000", 2024-09-13T11:24:10.306 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.1449e-05 2024-09-13T11:24:10.306 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.306 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.306 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:24:10.306 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:53.561097+0000", 2024-09-13T11:24:10.306 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.6021999999999996e-05 2024-09-13T11:24:10.306 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.306 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.306 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:24:10.307 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:53.561113+0000", 2024-09-13T11:24:10.307 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.5775999999999999e-05 2024-09-13T11:24:10.307 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.307 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.307 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:24:10.307 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:53.562255+0000", 2024-09-13T11:24:10.307 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0011421370000000001 2024-09-13T11:24:10.307 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:10.307 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:24:10.307 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:10.307 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.307 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.307 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:71903 3.11 3:8da9fc5e:::benchmark_data_smithi177_31116_object71902:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96)", 2024-09-13T11:24:10.308 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:54.140481+0000", 2024-09-13T11:24:10.308 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 16.099955718, 2024-09-13T11:24:10.308 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.094185646999999997, 2024-09-13T11:24:10.308 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:24:10.308 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:24:10.308 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:10.308 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:24:10.308 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:24:10.308 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:10.308 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 71903 2024-09-13T11:24:10.308 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.309 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:24:10.309 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.309 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:24:10.309 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:54.140481+0000", 2024-09-13T11:24:10.309 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:10.309 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.309 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.309 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:24:10.309 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:54.140481+0000", 2024-09-13T11:24:10.310 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:10.310 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.310 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.310 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:24:10.310 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:54.140485+0000", 2024-09-13T11:24:10.310 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.3610000000000001e-06 2024-09-13T11:24:10.310 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.310 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.310 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:24:10.310 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:54.140549+0000", 2024-09-13T11:24:10.310 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.4578000000000004e-05 2024-09-13T11:24:10.310 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.310 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.310 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:24:10.311 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:54.140574+0000", 2024-09-13T11:24:10.311 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.4640000000000001e-05 2024-09-13T11:24:10.311 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.311 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.311 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:24:10.311 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:54.140588+0000", 2024-09-13T11:24:10.311 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.4154e-05 2024-09-13T11:24:10.311 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.311 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.311 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:24:10.311 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:54.140718+0000", 2024-09-13T11:24:10.311 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00013008900000000001 2024-09-13T11:24:10.311 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.312 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.312 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:24:10.312 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:54.141021+0000", 2024-09-13T11:24:10.312 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00030258100000000001 2024-09-13T11:24:10.312 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.312 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.312 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 11", 2024-09-13T11:24:10.312 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:54.141326+0000", 2024-09-13T11:24:10.312 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00030553900000000002 2024-09-13T11:24:10.312 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.312 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.312 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:24:10.312 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:54.143701+0000", 2024-09-13T11:24:10.313 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.002374751 2024-09-13T11:24:10.313 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.313 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.313 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:10.313 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:54.234577+0000", 2024-09-13T11:24:10.313 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.090875517000000003 2024-09-13T11:24:10.313 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.313 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.313 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:24:10.314 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:54.234617+0000", 2024-09-13T11:24:10.314 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.0930999999999998e-05 2024-09-13T11:24:10.314 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.314 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.314 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:24:10.314 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:54.234667+0000", 2024-09-13T11:24:10.314 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.9505999999999997e-05 2024-09-13T11:24:10.314 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:10.314 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:24:10.314 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:10.314 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.314 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.314 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:71961 3.11 3:89bd9466:::benchmark_data_smithi177_31116_object71960:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2024-09-13T11:24:10.315 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:54.267475+0000", 2024-09-13T11:24:10.315 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 15.972962072, 2024-09-13T11:24:10.315 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.15027747899999999, 2024-09-13T11:24:10.315 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:24:10.315 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:24:10.315 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:10.315 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:24:10.315 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:24:10.315 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:10.315 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 71961 2024-09-13T11:24:10.315 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.315 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:24:10.315 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.315 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:24:10.316 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:54.267475+0000", 2024-09-13T11:24:10.316 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:10.316 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.316 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.316 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:24:10.316 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:54.267475+0000", 2024-09-13T11:24:10.316 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:10.316 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.316 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.316 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:24:10.316 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:54.267478+0000", 2024-09-13T11:24:10.316 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.8080000000000001e-06 2024-09-13T11:24:10.316 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.316 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.317 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:24:10.317 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:54.267534+0000", 2024-09-13T11:24:10.317 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.6276999999999997e-05 2024-09-13T11:24:10.317 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.317 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.317 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:24:10.317 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:54.267557+0000", 2024-09-13T11:24:10.317 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.2514000000000002e-05 2024-09-13T11:24:10.317 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.317 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.317 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:24:10.318 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:54.267570+0000", 2024-09-13T11:24:10.318 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.3251999999999999e-05 2024-09-13T11:24:10.318 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.318 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.318 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:24:10.318 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:54.267650+0000", 2024-09-13T11:24:10.318 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.0066000000000002e-05 2024-09-13T11:24:10.318 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.318 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.318 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:24:10.318 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:54.267886+0000", 2024-09-13T11:24:10.318 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00023637800000000001 2024-09-13T11:24:10.319 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.319 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.319 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 11", 2024-09-13T11:24:10.319 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:54.268020+0000", 2024-09-13T11:24:10.319 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00013392100000000001 2024-09-13T11:24:10.319 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.320 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.320 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:24:10.320 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:54.270076+0000", 2024-09-13T11:24:10.320 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.002055393 2024-09-13T11:24:10.320 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.320 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.321 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:10.321 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:54.417584+0000", 2024-09-13T11:24:10.321 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.147508008 2024-09-13T11:24:10.321 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.321 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.321 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:24:10.321 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:54.417667+0000", 2024-09-13T11:24:10.321 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.3666999999999997e-05 2024-09-13T11:24:10.321 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.321 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.321 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:24:10.321 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:54.417752+0000", 2024-09-13T11:24:10.321 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.5195e-05 2024-09-13T11:24:10.322 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:10.322 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:24:10.322 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:10.322 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.322 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.322 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDPGPush(3.1e 100/78 [PushOp(3:7edefe13:::benchmark_data_smithi177_31116_object48805:head, version: 71'3135, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:7edefe13:::benchmark_data_smithi177_31116_object48805:head@71'3135, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:24:10.322 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:58.496731+0000", 2024-09-13T11:24:10.322 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 11.74370571, 2024-09-13T11:24:10.322 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.087731966999999994, 2024-09-13T11:24:10.322 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:24:10.322 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:24:10.323 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:24:10.323 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:24:10.323 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.323 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:24:10.323 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:58.496731+0000", 2024-09-13T11:24:10.323 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:10.323 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.323 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.323 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:24:10.323 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:58.496731+0000", 2024-09-13T11:24:10.323 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:10.323 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.323 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.323 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:24:10.324 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:58.496733+0000", 2024-09-13T11:24:10.324 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.922e-06 2024-09-13T11:24:10.324 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.324 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.324 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:24:10.324 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:58.496762+0000", 2024-09-13T11:24:10.324 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.9184999999999999e-05 2024-09-13T11:24:10.324 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.324 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.324 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:24:10.324 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:58.583093+0000", 2024-09-13T11:24:10.324 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.086330544999999995 2024-09-13T11:24:10.324 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.324 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.325 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:24:10.325 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:58.583117+0000", 2024-09-13T11:24:10.325 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.3862e-05 2024-09-13T11:24:10.325 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.325 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.325 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:24:10.325 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:58.583214+0000", 2024-09-13T11:24:10.325 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.7090999999999996e-05 2024-09-13T11:24:10.325 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.325 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.325 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:24:10.325 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:58.583230+0000", 2024-09-13T11:24:10.325 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.6273000000000002e-05 2024-09-13T11:24:10.326 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.326 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.326 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:24:10.326 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:58.584463+0000", 2024-09-13T11:24:10.326 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0012330889999999999 2024-09-13T11:24:10.326 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:10.326 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:24:10.326 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:10.326 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.326 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.326 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDPGPush(3.1e 101/78 [PushOp(3:7cb23639:::benchmark_data_smithi177_31116_object50609:head, version: 72'3258, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:7cb23639:::benchmark_data_smithi177_31116_object50609:head@72'3258, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:24:10.326 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:59.335464+0000", 2024-09-13T11:24:10.326 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 10.904973374000001, 2024-09-13T11:24:10.327 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.073690760999999994, 2024-09-13T11:24:10.327 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:24:10.327 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:24:10.327 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:24:10.327 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:24:10.327 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.327 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:24:10.327 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:59.335464+0000", 2024-09-13T11:24:10.327 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:10.327 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.327 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.327 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:24:10.328 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:59.335464+0000", 2024-09-13T11:24:10.328 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:10.328 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.328 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.328 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:24:10.328 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:59.335465+0000", 2024-09-13T11:24:10.328 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.4479999999999999e-06 2024-09-13T11:24:10.328 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.328 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.328 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:24:10.328 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:59.335491+0000", 2024-09-13T11:24:10.328 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.599e-05 2024-09-13T11:24:10.328 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.328 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.329 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:24:10.329 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:59.407764+0000", 2024-09-13T11:24:10.329 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.072272946000000005 2024-09-13T11:24:10.329 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.329 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.329 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:24:10.329 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:59.407781+0000", 2024-09-13T11:24:10.329 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.7316000000000002e-05 2024-09-13T11:24:10.329 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.330 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.330 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:24:10.330 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:59.407898+0000", 2024-09-13T11:24:10.330 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000116594 2024-09-13T11:24:10.330 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.330 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.330 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:24:10.330 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:59.407912+0000", 2024-09-13T11:24:10.330 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.4399999999999999e-05 2024-09-13T11:24:10.330 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.330 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.330 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:24:10.330 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:59.409154+0000", 2024-09-13T11:24:10.330 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0012420669999999999 2024-09-13T11:24:10.331 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:10.331 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:24:10.331 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:10.331 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.331 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.331 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4665.0:79776 3.1a e102/78 3:5d637d57:::benchmark_data_smithi177_31116_object79775:head v 102'4120, mlcod=72'3145)", 2024-09-13T11:24:10.331 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:24:00.354950+0000", 2024-09-13T11:24:10.331 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 9.8854867280000001, 2024-09-13T11:24:10.331 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.14753330100000001, 2024-09-13T11:24:10.331 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:24:10.331 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:24:10.331 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:10.331 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:24:10.332 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.332 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:24:10.332 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:00.354950+0000", 2024-09-13T11:24:10.332 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:10.332 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.332 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.332 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:24:10.332 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:00.354950+0000", 2024-09-13T11:24:10.332 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:10.332 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.332 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.332 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:24:10.332 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:00.354960+0000", 2024-09-13T11:24:10.333 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.5219999999999996e-06 2024-09-13T11:24:10.333 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.333 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.333 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:24:10.333 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:00.355024+0000", 2024-09-13T11:24:10.333 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.4163000000000002e-05 2024-09-13T11:24:10.333 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.333 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.333 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:24:10.333 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:00.499446+0000", 2024-09-13T11:24:10.333 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.144422352 2024-09-13T11:24:10.333 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.333 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.334 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:24:10.334 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:00.499466+0000", 2024-09-13T11:24:10.334 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.9694000000000002e-05 2024-09-13T11:24:10.334 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.334 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.334 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:24:10.334 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:00.499605+0000", 2024-09-13T11:24:10.334 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00013935099999999999 2024-09-13T11:24:10.334 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.334 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.334 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:24:10.334 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:00.499655+0000", 2024-09-13T11:24:10.334 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.9787000000000003e-05 2024-09-13T11:24:10.334 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.335 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.335 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:24:10.335 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:00.502418+0000", 2024-09-13T11:24:10.335 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.002763012 2024-09-13T11:24:10.335 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.335 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.335 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:24:10.335 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:00.502484+0000", 2024-09-13T11:24:10.335 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.5419999999999994e-05 2024-09-13T11:24:10.335 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:10.335 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:24:10.335 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:10.335 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.336 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.336 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDPGPush(3.1a 102/78 [PushOp(3:5df50766:::benchmark_data_smithi177_31116_object50543:head, version: 72'3152, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:5df50766:::benchmark_data_smithi177_31116_object50543:head@72'3152, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:24:10.336 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:24:00.499311+0000", 2024-09-13T11:24:10.336 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 9.7411261860000007, 2024-09-13T11:24:10.336 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.26146664800000002, 2024-09-13T11:24:10.336 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:24:10.336 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:24:10.336 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:24:10.336 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:24:10.336 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.336 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:24:10.336 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:00.499311+0000", 2024-09-13T11:24:10.336 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:10.337 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.337 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.337 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:24:10.337 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:00.499311+0000", 2024-09-13T11:24:10.337 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:10.337 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.337 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.337 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:24:10.337 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:00.499320+0000", 2024-09-13T11:24:10.337 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.6400000000000003e-06 2024-09-13T11:24:10.337 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.337 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.337 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:24:10.337 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:00.499384+0000", 2024-09-13T11:24:10.338 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.4554999999999994e-05 2024-09-13T11:24:10.338 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.338 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.338 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:24:10.338 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:00.759713+0000", 2024-09-13T11:24:10.338 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.260328742 2024-09-13T11:24:10.338 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.338 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.338 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:24:10.339 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:00.759742+0000", 2024-09-13T11:24:10.339 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.9298e-05 2024-09-13T11:24:10.339 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.339 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.339 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:24:10.339 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:00.759871+0000", 2024-09-13T11:24:10.339 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000128992 2024-09-13T11:24:10.339 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.339 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.339 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:24:10.339 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:00.759884+0000", 2024-09-13T11:24:10.339 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2687999999999999e-05 2024-09-13T11:24:10.340 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.340 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.340 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:24:10.340 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:00.760778+0000", 2024-09-13T11:24:10.340 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00089373299999999996 2024-09-13T11:24:10.340 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:10.340 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:24:10.340 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:10.340 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.341 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.341 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4665.0:82769 3.1e e105/102 3:7a4e88b5:::benchmark_data_smithi177_31116_object82768:head v 105'4381, mlcod=105'4379)", 2024-09-13T11:24:10.341 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:24:03.745047+0000", 2024-09-13T11:24:10.341 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 6.4953899609999999, 2024-09-13T11:24:10.341 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.213806145, 2024-09-13T11:24:10.341 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:24:10.341 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:24:10.341 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:10.341 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:24:10.342 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.342 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:24:10.342 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:03.745047+0000", 2024-09-13T11:24:10.342 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:10.342 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.342 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.342 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:24:10.342 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:03.745047+0000", 2024-09-13T11:24:10.342 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:10.342 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.342 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.342 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:24:10.342 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:03.745049+0000", 2024-09-13T11:24:10.342 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.5710000000000001e-06 2024-09-13T11:24:10.343 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.343 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.343 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:24:10.343 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:03.745107+0000", 2024-09-13T11:24:10.343 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.8799999999999999e-05 2024-09-13T11:24:10.343 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.343 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.343 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:24:10.343 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:03.955145+0000", 2024-09-13T11:24:10.343 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.210037735 2024-09-13T11:24:10.344 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.344 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.344 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:24:10.344 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:03.955163+0000", 2024-09-13T11:24:10.344 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.7520000000000002e-05 2024-09-13T11:24:10.344 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.344 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.344 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:24:10.344 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:03.955473+0000", 2024-09-13T11:24:10.344 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00031014399999999999 2024-09-13T11:24:10.344 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.344 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.344 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:24:10.345 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:03.955538+0000", 2024-09-13T11:24:10.345 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.5059000000000002e-05 2024-09-13T11:24:10.345 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.345 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.345 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:24:10.345 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:03.958753+0000", 2024-09-13T11:24:10.345 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0032152769999999999 2024-09-13T11:24:10.345 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.345 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.345 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:24:10.345 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:03.958853+0000", 2024-09-13T11:24:10.345 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000100039 2024-09-13T11:24:10.345 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:10.345 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:24:10.346 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:10.346 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:10.346 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:24:10.346 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-13T11:24:10.348 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-09-13T11:24:10.348 INFO:tasks.ceph.osd.8.smithi001.stderr:2024-09-13T11:24:10.339+0000 7fb9b48ae640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26312) UID: 0 2024-09-13T11:24:10.448 INFO:tasks.ceph.osd.13.smithi092.stderr:2024-09-13T11:24:10.443+0000 7f3e5afea640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26350) UID: 0 2024-09-13T11:24:10.519 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-13T11:24:10.519 INFO:teuthology.orchestra.run.smithi001.stdout: "ops": [], 2024-09-13T11:24:10.519 INFO:teuthology.orchestra.run.smithi001.stdout: "num_ops": 0 2024-09-13T11:24:10.519 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-13T11:24:10.535 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-09-13T11:24:10.548 INFO:tasks.ceph.osd.7.smithi177.stderr:2024-09-13T11:24:10.547+0000 7f897b51d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26575) UID: 0 2024-09-13T11:24:10.639 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-13T11:24:10.639 INFO:teuthology.orchestra.run.smithi001.stdout:{"fsid":"2fd68ac7-5581-4d90-80a4-30bcf5252a7a","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"nodeep-scrub flag(s) set","count":12},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":133,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":110,"num_osds":16,"num_up_osds":16,"osd_up_since":1726226566,"num_in_osds":16,"osd_in_since":1726226521,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":44},{"state_name":"peering","count":1}],"num_pgs":45,"num_pools":3,"num_objects":85040,"data_bytes":5573444147,"bytes_used":14043262976,"bytes_avail":1532144963584,"bytes_total":1546188226560,"inactive_pgs_ratio":0.02222222276031971,"recovering_objects_per_sec":183,"recovering_bytes_per_sec":12038548,"recovering_keys_per_sec":0,"num_objects_recovered":735,"num_bytes_recovered":48168960,"num_keys_recovered":0,"write_bytes_sec":77112226,"read_op_per_sec":0,"write_op_per_sec":1176},"fsmap":{"epoch":1,"btime":"2024-09-13T11:21:57:222574+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-13T11:24:06.142821+0000","services":{}},"progress_events":{}} 2024-09-13T11:24:10.649 INFO:tasks.ceph.osd.14.smithi162.stderr:2024-09-13T11:24:10.644+0000 7f235c0db640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26319) UID: 0 2024-09-13T11:24:10.661 INFO:tasks.ceph.ceph_manager.ceph:making progress, resetting timeout 2024-09-13T11:24:10.661 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-13T11:24:10.665 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-13T11:24:10.666 INFO:teuthology.orchestra.run.smithi001.stdout: "ops": [], 2024-09-13T11:24:10.666 INFO:teuthology.orchestra.run.smithi001.stdout: "complaint_time": 30, 2024-09-13T11:24:10.666 INFO:teuthology.orchestra.run.smithi001.stdout: "num_blocked_ops": 0 2024-09-13T11:24:10.666 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-13T11:24:10.684 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-09-13T11:24:10.750 INFO:tasks.ceph.osd.2.smithi162.stderr:2024-09-13T11:24:10.748+0000 7fe92b35a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26313) UID: 0 2024-09-13T11:24:10.853 INFO:tasks.ceph.osd.1.smithi092.stderr:2024-09-13T11:24:10.851+0000 7fe7789c5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33718) UID: 0 2024-09-13T11:24:10.868 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-13T11:24:10.868 INFO:teuthology.orchestra.run.smithi001.stdout: "size": 20, 2024-09-13T11:24:10.868 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 600, 2024-09-13T11:24:10.869 INFO:teuthology.orchestra.run.smithi001.stdout: "ops": [ 2024-09-13T11:24:10.869 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.869 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:30779 3.e 3:77c07d05:::benchmark_data_smithi177_31116_object30778:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e59)", 2024-09-13T11:24:10.869 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:09.473906+0000", 2024-09-13T11:24:10.869 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 61.392430220999998, 2024-09-13T11:24:10.869 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.224949224, 2024-09-13T11:24:10.869 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:24:10.869 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:24:10.869 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:10.869 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:24:10.869 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:24:10.869 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:10.869 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 30779 2024-09-13T11:24:10.869 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.870 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:24:10.870 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.870 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:24:10.870 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.473906+0000", 2024-09-13T11:24:10.870 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:10.870 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.870 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.870 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:24:10.870 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.473906+0000", 2024-09-13T11:24:10.870 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:10.870 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.870 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.870 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:24:10.871 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.473908+0000", 2024-09-13T11:24:10.871 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.6230000000000001e-06 2024-09-13T11:24:10.871 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.871 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.871 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:24:10.871 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.473961+0000", 2024-09-13T11:24:10.871 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.2374999999999999e-05 2024-09-13T11:24:10.871 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.871 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.871 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:24:10.871 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.473976+0000", 2024-09-13T11:24:10.871 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.56e-05 2024-09-13T11:24:10.871 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.871 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.872 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:24:10.872 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.473987+0000", 2024-09-13T11:24:10.872 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0394000000000001e-05 2024-09-13T11:24:10.872 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.872 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.872 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:24:10.872 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.474275+0000", 2024-09-13T11:24:10.872 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00028800800000000002 2024-09-13T11:24:10.872 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.872 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.872 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:24:10.872 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.474466+0000", 2024-09-13T11:24:10.872 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000191499 2024-09-13T11:24:10.872 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.873 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.873 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 0,1", 2024-09-13T11:24:10.873 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.474666+0000", 2024-09-13T11:24:10.873 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00019994899999999999 2024-09-13T11:24:10.873 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.873 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.873 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:10.873 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.478354+0000", 2024-09-13T11:24:10.873 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0036876779999999998 2024-09-13T11:24:10.873 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.873 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.873 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:24:10.873 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.483128+0000", 2024-09-13T11:24:10.874 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0047739410000000003 2024-09-13T11:24:10.874 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.874 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.874 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:10.874 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.697695+0000", 2024-09-13T11:24:10.874 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.214566955 2024-09-13T11:24:10.874 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.874 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.874 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:24:10.874 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.697786+0000", 2024-09-13T11:24:10.874 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.1310000000000005e-05 2024-09-13T11:24:10.874 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.874 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.874 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:24:10.875 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.698855+0000", 2024-09-13T11:24:10.875 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0010688920000000001 2024-09-13T11:24:10.875 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:10.875 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:24:10.875 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:10.875 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.875 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.875 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:30801 3.e 3:73b0f21a:::benchmark_data_smithi177_31116_object30800:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e59)", 2024-09-13T11:24:10.875 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:09.485629+0000", 2024-09-13T11:24:10.875 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 61.380707418999997, 2024-09-13T11:24:10.875 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.22604919900000001, 2024-09-13T11:24:10.875 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:24:10.875 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:24:10.875 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:10.876 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:24:10.876 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:24:10.876 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:10.876 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 30801 2024-09-13T11:24:10.876 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.876 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:24:10.876 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.876 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:24:10.876 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.485629+0000", 2024-09-13T11:24:10.876 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:10.876 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.876 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.876 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:24:10.877 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.485629+0000", 2024-09-13T11:24:10.877 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:10.877 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.877 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.877 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:24:10.877 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.485631+0000", 2024-09-13T11:24:10.877 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.9989999999999998e-06 2024-09-13T11:24:10.877 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.878 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.878 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:24:10.878 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.485664+0000", 2024-09-13T11:24:10.878 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.3896000000000002e-05 2024-09-13T11:24:10.878 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.878 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.878 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:24:10.878 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.485679+0000", 2024-09-13T11:24:10.878 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.4438e-05 2024-09-13T11:24:10.878 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.878 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.878 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:24:10.878 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.485688+0000", 2024-09-13T11:24:10.879 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.0180000000000007e-06 2024-09-13T11:24:10.879 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.879 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.879 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:24:10.879 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.485742+0000", 2024-09-13T11:24:10.879 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.4026000000000001e-05 2024-09-13T11:24:10.879 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.879 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.879 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:24:10.879 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.485841+0000", 2024-09-13T11:24:10.879 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.8875000000000005e-05 2024-09-13T11:24:10.879 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.879 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.879 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 0,1", 2024-09-13T11:24:10.880 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.486005+0000", 2024-09-13T11:24:10.880 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000163747 2024-09-13T11:24:10.880 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.880 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.880 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:24:10.880 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.487867+0000", 2024-09-13T11:24:10.880 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.001862945 2024-09-13T11:24:10.880 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.880 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.880 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:10.880 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.488570+0000", 2024-09-13T11:24:10.880 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00070255700000000003 2024-09-13T11:24:10.880 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.881 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.881 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:10.881 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.711140+0000", 2024-09-13T11:24:10.881 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.222570348 2024-09-13T11:24:10.881 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.881 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.881 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:24:10.881 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.711180+0000", 2024-09-13T11:24:10.881 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.9273999999999997e-05 2024-09-13T11:24:10.881 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.882 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.882 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:24:10.882 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.711678+0000", 2024-09-13T11:24:10.882 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00049807599999999997 2024-09-13T11:24:10.882 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:10.882 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:24:10.882 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:10.882 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.882 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.882 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:54656 3.1e 3.6a49163e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e75)", 2024-09-13T11:24:10.882 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:30.979993+0000", 2024-09-13T11:24:10.882 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 39.886342986999999, 2024-09-13T11:24:10.882 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.9269557150000001, 2024-09-13T11:24:10.882 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:24:10.882 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:24:10.883 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-13T11:24:10.883 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:24:10.883 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:24:10.883 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:10.883 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 54656 2024-09-13T11:24:10.883 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.883 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:24:10.883 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.883 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:24:10.883 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:30.979993+0000", 2024-09-13T11:24:10.883 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:10.883 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.883 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.884 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:24:10.884 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:30.979993+0000", 2024-09-13T11:24:10.884 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:10.884 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.884 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.884 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:24:10.884 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:30.979997+0000", 2024-09-13T11:24:10.884 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.0319999999999997e-06 2024-09-13T11:24:10.884 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.884 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.884 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:24:10.884 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:30.980057+0000", 2024-09-13T11:24:10.884 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.0341999999999999e-05 2024-09-13T11:24:10.885 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.885 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.885 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:24:10.885 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:30.980094+0000", 2024-09-13T11:24:10.885 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.6770000000000002e-05 2024-09-13T11:24:10.885 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.885 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.885 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:24:10.885 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:30.980113+0000", 2024-09-13T11:24:10.885 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.8791999999999999e-05 2024-09-13T11:24:10.885 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.885 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.885 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:24:10.886 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.097773+0000", 2024-09-13T11:24:10.886 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.117660379 2024-09-13T11:24:10.886 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.886 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.886 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for peered", 2024-09-13T11:24:10.886 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.097784+0000", 2024-09-13T11:24:10.886 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.1025e-05 2024-09-13T11:24:10.886 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.886 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.886 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:24:10.886 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.906937+0000", 2024-09-13T11:24:10.886 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.8091523789999999 2024-09-13T11:24:10.886 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.886 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.887 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:24:10.887 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.906949+0000", 2024-09-13T11:24:10.887 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.1996e-05 2024-09-13T11:24:10.887 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:10.887 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:24:10.887 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:10.887 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.887 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.887 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDPGPushReply(3.17 75/70 [PushReplyOp(3:e82800c8:::benchmark_data_smithi177_31116_object13909:head)])", 2024-09-13T11:24:10.887 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:31.632344+0000", 2024-09-13T11:24:10.888 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 39.233992131999997, 2024-09-13T11:24:10.888 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.28592102899999999, 2024-09-13T11:24:10.888 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:24:10.888 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:24:10.888 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-13T11:24:10.888 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:24:10.888 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.888 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:24:10.888 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.632344+0000", 2024-09-13T11:24:10.888 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:10.888 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.888 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.888 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:24:10.889 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.632344+0000", 2024-09-13T11:24:10.889 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:10.889 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.889 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.889 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:24:10.889 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.632346+0000", 2024-09-13T11:24:10.889 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.8819999999999999e-06 2024-09-13T11:24:10.889 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.889 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.889 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:24:10.889 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.632353+0000", 2024-09-13T11:24:10.889 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.5059999999999998e-06 2024-09-13T11:24:10.890 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.890 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.890 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:24:10.890 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.917361+0000", 2024-09-13T11:24:10.890 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.285008231 2024-09-13T11:24:10.890 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.890 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.890 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:24:10.890 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.917372+0000", 2024-09-13T11:24:10.890 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0764999999999999e-05 2024-09-13T11:24:10.891 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.891 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.891 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:24:10.891 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.917458+0000", 2024-09-13T11:24:10.891 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.6136000000000003e-05 2024-09-13T11:24:10.891 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.891 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.891 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:24:10.891 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.918265+0000", 2024-09-13T11:24:10.891 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00080650899999999996 2024-09-13T11:24:10.891 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:10.891 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:24:10.891 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:10.891 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.892 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.892 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:54720 3.1e 3.a944303e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-13T11:24:10.892 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:31.973315+0000", 2024-09-13T11:24:10.892 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 38.893020751999998, 2024-09-13T11:24:10.892 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.93366527200000005, 2024-09-13T11:24:10.892 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:24:10.892 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:24:10.892 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-13T11:24:10.892 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:24:10.892 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:24:10.892 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:10.892 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 54720 2024-09-13T11:24:10.892 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.893 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:24:10.893 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.893 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:24:10.893 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.973315+0000", 2024-09-13T11:24:10.893 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:10.893 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.893 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.893 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:24:10.893 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.973315+0000", 2024-09-13T11:24:10.893 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:10.893 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.893 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.893 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:24:10.894 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.973318+0000", 2024-09-13T11:24:10.894 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.5830000000000001e-06 2024-09-13T11:24:10.894 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.894 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.894 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:24:10.894 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.973465+0000", 2024-09-13T11:24:10.894 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000146816 2024-09-13T11:24:10.894 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.894 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.894 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:24:10.894 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.973478+0000", 2024-09-13T11:24:10.894 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2931e-05 2024-09-13T11:24:10.894 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.894 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.894 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:24:10.895 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.973483+0000", 2024-09-13T11:24:10.895 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.8969999999999996e-06 2024-09-13T11:24:10.895 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.895 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.895 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:24:10.895 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.973548+0000", 2024-09-13T11:24:10.895 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.4771999999999996e-05 2024-09-13T11:24:10.895 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.895 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.895 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for peered", 2024-09-13T11:24:10.895 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.973557+0000", 2024-09-13T11:24:10.895 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.0359999999999995e-06 2024-09-13T11:24:10.895 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.895 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.896 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:24:10.896 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.906974+0000", 2024-09-13T11:24:10.896 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.93341706000000002 2024-09-13T11:24:10.896 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.896 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.896 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:24:10.896 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.906980+0000", 2024-09-13T11:24:10.896 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.1770000000000003e-06 2024-09-13T11:24:10.896 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:10.896 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:24:10.896 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:10.896 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.896 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.896 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:54734 3.1e 3.d7f6f7e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-13T11:24:10.897 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:31.983342+0000", 2024-09-13T11:24:10.897 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 38.882993949000003, 2024-09-13T11:24:10.897 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.92367058599999996, 2024-09-13T11:24:10.897 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:24:10.897 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:24:10.897 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-13T11:24:10.897 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:24:10.897 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:24:10.897 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:10.897 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 54734 2024-09-13T11:24:10.897 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.897 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:24:10.897 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.897 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:24:10.898 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.983342+0000", 2024-09-13T11:24:10.898 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:10.898 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.898 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.898 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:24:10.898 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.983342+0000", 2024-09-13T11:24:10.898 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:10.898 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.898 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.898 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:24:10.898 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.983345+0000", 2024-09-13T11:24:10.898 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.6220000000000001e-06 2024-09-13T11:24:10.898 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.898 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.899 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:24:10.899 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.983438+0000", 2024-09-13T11:24:10.899 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.3517000000000006e-05 2024-09-13T11:24:10.899 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.899 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.899 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:24:10.899 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.983451+0000", 2024-09-13T11:24:10.899 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2527999999999999e-05 2024-09-13T11:24:10.899 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.899 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.899 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:24:10.899 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.983458+0000", 2024-09-13T11:24:10.899 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.6380000000000004e-06 2024-09-13T11:24:10.900 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.900 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.900 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:24:10.900 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.983521+0000", 2024-09-13T11:24:10.900 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.2729000000000002e-05 2024-09-13T11:24:10.900 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.900 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.900 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for peered", 2024-09-13T11:24:10.900 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.983548+0000", 2024-09-13T11:24:10.900 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.6537999999999999e-05 2024-09-13T11:24:10.900 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.900 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.900 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:24:10.900 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.907005+0000", 2024-09-13T11:24:10.901 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.923456991 2024-09-13T11:24:10.901 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.901 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.901 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:24:10.901 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.907013+0000", 2024-09-13T11:24:10.901 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.0229999999999992e-06 2024-09-13T11:24:10.901 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:10.901 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:24:10.901 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:10.901 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.901 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.901 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:54760 3.1e 3.8c32c13e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-13T11:24:10.901 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:31.999685+0000", 2024-09-13T11:24:10.902 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 38.866650716999999, 2024-09-13T11:24:10.902 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.90735639599999995, 2024-09-13T11:24:10.902 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:24:10.902 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:24:10.902 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-13T11:24:10.902 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:24:10.902 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:24:10.902 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:10.902 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 54760 2024-09-13T11:24:10.902 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.902 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:24:10.903 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.903 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:24:10.903 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.999685+0000", 2024-09-13T11:24:10.903 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:10.903 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.903 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.903 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:24:10.903 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.999685+0000", 2024-09-13T11:24:10.903 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:10.903 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.903 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.903 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:24:10.903 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.999690+0000", 2024-09-13T11:24:10.904 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.7879999999999997e-06 2024-09-13T11:24:10.904 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.904 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.904 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:24:10.904 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.999729+0000", 2024-09-13T11:24:10.904 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.9032000000000002e-05 2024-09-13T11:24:10.904 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.904 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.904 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:24:10.904 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.999741+0000", 2024-09-13T11:24:10.904 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2136999999999999e-05 2024-09-13T11:24:10.904 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.904 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.905 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:24:10.905 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.999748+0000", 2024-09-13T11:24:10.905 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.9120000000000001e-06 2024-09-13T11:24:10.905 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.905 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.905 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:24:10.905 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.999803+0000", 2024-09-13T11:24:10.905 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.5064e-05 2024-09-13T11:24:10.905 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.906 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.906 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for peered", 2024-09-13T11:24:10.906 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.999815+0000", 2024-09-13T11:24:10.906 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2041e-05 2024-09-13T11:24:10.906 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.906 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.906 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:24:10.906 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.907034+0000", 2024-09-13T11:24:10.906 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.90721858200000005 2024-09-13T11:24:10.906 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.906 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.906 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:24:10.906 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.907042+0000", 2024-09-13T11:24:10.907 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.8399999999999995e-06 2024-09-13T11:24:10.907 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:10.907 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:24:10.907 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:10.907 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.907 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.907 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:54768 3.1e 3.6dad499e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-13T11:24:10.907 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:32.003482+0000", 2024-09-13T11:24:10.907 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 38.862854067000001, 2024-09-13T11:24:10.907 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.90358804699999995, 2024-09-13T11:24:10.907 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:24:10.907 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:24:10.907 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-13T11:24:10.907 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:24:10.908 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:24:10.908 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:10.908 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 54768 2024-09-13T11:24:10.908 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.908 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:24:10.908 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.908 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:24:10.908 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.003482+0000", 2024-09-13T11:24:10.908 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:10.908 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.908 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.908 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:24:10.908 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.003482+0000", 2024-09-13T11:24:10.909 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:10.909 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.909 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.909 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:24:10.909 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.003486+0000", 2024-09-13T11:24:10.909 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.1030000000000004e-06 2024-09-13T11:24:10.909 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.909 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.909 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:24:10.909 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.003523+0000", 2024-09-13T11:24:10.910 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.6732e-05 2024-09-13T11:24:10.910 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.910 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.910 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:24:10.910 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.003535+0000", 2024-09-13T11:24:10.910 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2520999999999999e-05 2024-09-13T11:24:10.910 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.910 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.910 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:24:10.910 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.003541+0000", 2024-09-13T11:24:10.910 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.8690000000000001e-06 2024-09-13T11:24:10.910 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.910 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.911 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:24:10.911 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.003593+0000", 2024-09-13T11:24:10.911 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.1786000000000002e-05 2024-09-13T11:24:10.911 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.911 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.911 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for peered", 2024-09-13T11:24:10.911 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.003602+0000", 2024-09-13T11:24:10.911 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.9679999999999995e-06 2024-09-13T11:24:10.911 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.911 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.911 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:24:10.911 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.907063+0000", 2024-09-13T11:24:10.911 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.90346116600000004 2024-09-13T11:24:10.912 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.912 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.912 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:24:10.912 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.907070+0000", 2024-09-13T11:24:10.912 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.9020000000000002e-06 2024-09-13T11:24:10.912 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:10.912 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:24:10.912 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:10.912 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.912 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.912 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:54785 3.1e 3.d32f1ede (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-13T11:24:10.912 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:32.016238+0000", 2024-09-13T11:24:10.912 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 38.850097857000002, 2024-09-13T11:24:10.912 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.89086243399999998, 2024-09-13T11:24:10.913 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:24:10.913 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:24:10.913 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-13T11:24:10.913 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:24:10.913 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:24:10.913 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:10.913 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 54785 2024-09-13T11:24:10.913 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.913 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:24:10.913 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.913 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:24:10.913 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.016238+0000", 2024-09-13T11:24:10.913 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:10.914 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.914 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.914 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:24:10.914 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.016238+0000", 2024-09-13T11:24:10.914 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:10.914 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.914 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.914 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:24:10.914 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.016241+0000", 2024-09-13T11:24:10.915 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.6469999999999998e-06 2024-09-13T11:24:10.915 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.915 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.915 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:24:10.915 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.016271+0000", 2024-09-13T11:24:10.915 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.0505999999999999e-05 2024-09-13T11:24:10.915 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.915 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.915 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:24:10.915 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.016287+0000", 2024-09-13T11:24:10.915 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.6073999999999999e-05 2024-09-13T11:24:10.915 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.915 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.916 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:24:10.916 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.016295+0000", 2024-09-13T11:24:10.916 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.0590000000000003e-06 2024-09-13T11:24:10.916 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.916 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.916 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:24:10.916 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.016373+0000", 2024-09-13T11:24:10.916 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.7634999999999998e-05 2024-09-13T11:24:10.916 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.916 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.916 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for peered", 2024-09-13T11:24:10.916 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.016383+0000", 2024-09-13T11:24:10.916 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0246e-05 2024-09-13T11:24:10.917 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.917 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.917 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:24:10.917 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.907092+0000", 2024-09-13T11:24:10.917 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.89070849200000002 2024-09-13T11:24:10.917 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.917 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.917 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:24:10.917 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.907101+0000", 2024-09-13T11:24:10.917 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.7749999999999998e-06 2024-09-13T11:24:10.917 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:10.917 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:24:10.917 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:10.918 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.918 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.918 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:54811 3.1e 3.3754367e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-13T11:24:10.918 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:32.045424+0000", 2024-09-13T11:24:10.918 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 38.820912206000003, 2024-09-13T11:24:10.918 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.86170719399999995, 2024-09-13T11:24:10.918 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:24:10.918 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:24:10.918 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-13T11:24:10.918 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:24:10.918 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:24:10.918 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:10.918 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 54811 2024-09-13T11:24:10.918 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.919 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:24:10.919 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.919 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:24:10.919 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.045424+0000", 2024-09-13T11:24:10.919 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:10.919 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.919 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.919 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:24:10.919 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.045424+0000", 2024-09-13T11:24:10.919 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:10.919 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.919 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.919 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:24:10.920 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.045426+0000", 2024-09-13T11:24:10.920 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.0260000000000002e-06 2024-09-13T11:24:10.920 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.920 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.920 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:24:10.920 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.045556+0000", 2024-09-13T11:24:10.920 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00013003099999999999 2024-09-13T11:24:10.920 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.920 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.920 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:24:10.920 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.045568+0000", 2024-09-13T11:24:10.920 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.1727e-05 2024-09-13T11:24:10.920 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.921 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.921 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:24:10.921 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.045576+0000", 2024-09-13T11:24:10.921 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.6459999999999999e-06 2024-09-13T11:24:10.921 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.921 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.921 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:24:10.921 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.045650+0000", 2024-09-13T11:24:10.921 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.4000999999999998e-05 2024-09-13T11:24:10.921 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.921 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.922 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for peered", 2024-09-13T11:24:10.922 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.045658+0000", 2024-09-13T11:24:10.922 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.3869999999999997e-06 2024-09-13T11:24:10.922 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.922 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.922 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:24:10.922 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.907123+0000", 2024-09-13T11:24:10.922 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.86146508399999999 2024-09-13T11:24:10.922 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.922 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.922 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:24:10.922 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.907131+0000", 2024-09-13T11:24:10.923 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.2919999999999996e-06 2024-09-13T11:24:10.923 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:10.923 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:24:10.923 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:10.923 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.923 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.923 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:54817 3.1e 3.36dd2fde (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-13T11:24:10.923 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:32.050108+0000", 2024-09-13T11:24:10.923 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 38.816228314, 2024-09-13T11:24:10.923 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.85705549199999997, 2024-09-13T11:24:10.923 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:24:10.923 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:24:10.923 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-13T11:24:10.924 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:24:10.924 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:24:10.924 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:10.924 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 54817 2024-09-13T11:24:10.924 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.924 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:24:10.924 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.924 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:24:10.924 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.050108+0000", 2024-09-13T11:24:10.924 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:10.924 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.924 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.925 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:24:10.925 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.050108+0000", 2024-09-13T11:24:10.925 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:10.925 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.925 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.925 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:24:10.925 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.050110+0000", 2024-09-13T11:24:10.925 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.0209999999999998e-06 2024-09-13T11:24:10.925 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.925 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.925 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:24:10.925 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.050166+0000", 2024-09-13T11:24:10.925 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.6026000000000002e-05 2024-09-13T11:24:10.925 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.926 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.926 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:24:10.926 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.050180+0000", 2024-09-13T11:24:10.926 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.4307e-05 2024-09-13T11:24:10.926 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.926 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.926 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:24:10.926 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.050188+0000", 2024-09-13T11:24:10.926 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.5410000000000003e-06 2024-09-13T11:24:10.926 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.926 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.926 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:24:10.926 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.050260+0000", 2024-09-13T11:24:10.926 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.2157999999999996e-05 2024-09-13T11:24:10.926 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.927 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.927 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for peered", 2024-09-13T11:24:10.927 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.050267+0000", 2024-09-13T11:24:10.927 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.0190000000000001e-06 2024-09-13T11:24:10.927 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.927 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.927 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:24:10.927 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.907155+0000", 2024-09-13T11:24:10.927 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.85688824399999997 2024-09-13T11:24:10.927 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.927 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.927 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:24:10.927 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.907163+0000", 2024-09-13T11:24:10.928 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.1759999999999993e-06 2024-09-13T11:24:10.928 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:10.928 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:24:10.928 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:10.928 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.928 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.928 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:54818 3.1e 3.5cfcc69e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-13T11:24:10.928 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:32.051374+0000", 2024-09-13T11:24:10.928 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 38.814961898, 2024-09-13T11:24:10.928 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.85581781000000001, 2024-09-13T11:24:10.928 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:24:10.928 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:24:10.928 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-13T11:24:10.928 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:24:10.929 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:24:10.929 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:10.929 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 54818 2024-09-13T11:24:10.929 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.929 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:24:10.929 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.929 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:24:10.929 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.051374+0000", 2024-09-13T11:24:10.929 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:10.929 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.929 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.929 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:24:10.929 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.051374+0000", 2024-09-13T11:24:10.930 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:10.930 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.930 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.930 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:24:10.930 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.051375+0000", 2024-09-13T11:24:10.930 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.4440000000000001e-06 2024-09-13T11:24:10.930 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.930 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.930 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:24:10.930 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.051426+0000", 2024-09-13T11:24:10.930 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.0809000000000001e-05 2024-09-13T11:24:10.930 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.930 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.930 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:24:10.931 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.051441+0000", 2024-09-13T11:24:10.931 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.4460999999999999e-05 2024-09-13T11:24:10.931 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.931 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.931 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:24:10.931 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.051449+0000", 2024-09-13T11:24:10.931 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.4880000000000007e-06 2024-09-13T11:24:10.931 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.931 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.931 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:24:10.931 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.051499+0000", 2024-09-13T11:24:10.931 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.9691999999999998e-05 2024-09-13T11:24:10.931 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.931 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.932 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for peered", 2024-09-13T11:24:10.932 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.051505+0000", 2024-09-13T11:24:10.932 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.2029999999999997e-06 2024-09-13T11:24:10.932 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.932 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.932 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:24:10.932 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.907184+0000", 2024-09-13T11:24:10.932 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.85567906599999999 2024-09-13T11:24:10.933 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.933 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.933 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:24:10.933 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.907192+0000", 2024-09-13T11:24:10.933 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.6469999999999998e-06 2024-09-13T11:24:10.933 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:10.933 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:24:10.933 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:10.933 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.933 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.933 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:54828 3.1e 3.e920c11e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-13T11:24:10.934 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:32.063344+0000", 2024-09-13T11:24:10.934 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 38.802991470999999, 2024-09-13T11:24:10.934 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.84387899099999997, 2024-09-13T11:24:10.934 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:24:10.934 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:24:10.934 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-13T11:24:10.934 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:24:10.934 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:24:10.934 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:10.934 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 54828 2024-09-13T11:24:10.934 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.934 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:24:10.934 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.935 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:24:10.935 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.063344+0000", 2024-09-13T11:24:10.935 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:10.935 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.935 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.935 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:24:10.935 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.063344+0000", 2024-09-13T11:24:10.935 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:10.935 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.935 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.935 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:24:10.935 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.063347+0000", 2024-09-13T11:24:10.935 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.2299999999999998e-06 2024-09-13T11:24:10.936 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.936 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.936 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:24:10.936 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.063384+0000", 2024-09-13T11:24:10.936 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.7290999999999999e-05 2024-09-13T11:24:10.936 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.936 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.936 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:24:10.936 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.063397+0000", 2024-09-13T11:24:10.936 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2955999999999999e-05 2024-09-13T11:24:10.936 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.936 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.937 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:24:10.937 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.063404+0000", 2024-09-13T11:24:10.937 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.4889999999999998e-06 2024-09-13T11:24:10.937 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.937 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.937 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:24:10.937 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.063470+0000", 2024-09-13T11:24:10.937 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.5516000000000001e-05 2024-09-13T11:24:10.937 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.937 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.937 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for peered", 2024-09-13T11:24:10.937 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.063482+0000", 2024-09-13T11:24:10.937 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2065000000000001e-05 2024-09-13T11:24:10.938 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.938 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.938 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:24:10.938 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.907216+0000", 2024-09-13T11:24:10.938 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.84373388900000001 2024-09-13T11:24:10.938 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.938 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.938 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:24:10.938 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.907223+0000", 2024-09-13T11:24:10.938 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.5549999999999997e-06 2024-09-13T11:24:10.938 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:10.938 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:24:10.939 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:10.939 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.939 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.939 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:54829 3.1e 3.192f62be (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-13T11:24:10.939 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:32.065642+0000", 2024-09-13T11:24:10.939 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 38.800694403000001, 2024-09-13T11:24:10.939 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.84161483100000001, 2024-09-13T11:24:10.939 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:24:10.939 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:24:10.939 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-13T11:24:10.939 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:24:10.939 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:24:10.939 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:10.940 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 54829 2024-09-13T11:24:10.940 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.940 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:24:10.940 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.940 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:24:10.940 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.065642+0000", 2024-09-13T11:24:10.940 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:10.940 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.940 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.940 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:24:10.940 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.065642+0000", 2024-09-13T11:24:10.940 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:10.940 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.941 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.941 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:24:10.941 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.065644+0000", 2024-09-13T11:24:10.941 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.1780000000000002e-06 2024-09-13T11:24:10.941 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.941 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.941 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:24:10.941 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.065695+0000", 2024-09-13T11:24:10.941 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.1632000000000003e-05 2024-09-13T11:24:10.941 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.941 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.941 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:24:10.941 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.065711+0000", 2024-09-13T11:24:10.941 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.5522000000000001e-05 2024-09-13T11:24:10.942 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.942 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.942 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:24:10.942 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.065719+0000", 2024-09-13T11:24:10.942 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.3480000000000004e-06 2024-09-13T11:24:10.942 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.942 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.942 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:24:10.942 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.065785+0000", 2024-09-13T11:24:10.942 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.5627e-05 2024-09-13T11:24:10.942 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.942 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.942 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for peered", 2024-09-13T11:24:10.942 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.065797+0000", 2024-09-13T11:24:10.943 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.1766000000000001e-05 2024-09-13T11:24:10.943 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.943 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.943 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:24:10.943 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.907248+0000", 2024-09-13T11:24:10.943 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.84145150300000005 2024-09-13T11:24:10.943 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.943 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.943 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:24:10.943 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.907256+0000", 2024-09-13T11:24:10.944 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.2549999999999997e-06 2024-09-13T11:24:10.944 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:10.944 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:24:10.944 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:10.944 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.944 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.944 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:54830 3.1e 3.b31554de (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-13T11:24:10.944 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:32.068259+0000", 2024-09-13T11:24:10.944 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 38.798076535, 2024-09-13T11:24:10.944 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.83902489599999996, 2024-09-13T11:24:10.944 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:24:10.944 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:24:10.944 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-13T11:24:10.945 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:24:10.945 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:24:10.945 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:10.945 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 54830 2024-09-13T11:24:10.945 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.945 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:24:10.945 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.945 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:24:10.945 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.068259+0000", 2024-09-13T11:24:10.945 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:10.945 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.945 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.946 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:24:10.946 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.068259+0000", 2024-09-13T11:24:10.946 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:10.946 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.946 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.946 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:24:10.946 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.068262+0000", 2024-09-13T11:24:10.946 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.1799999999999999e-06 2024-09-13T11:24:10.946 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.946 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.946 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:24:10.946 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.068301+0000", 2024-09-13T11:24:10.946 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.9289999999999998e-05 2024-09-13T11:24:10.947 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.947 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.947 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:24:10.947 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.068314+0000", 2024-09-13T11:24:10.947 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.3502999999999999e-05 2024-09-13T11:24:10.947 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.947 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.947 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:24:10.947 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.068324+0000", 2024-09-13T11:24:10.947 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.1630000000000002e-06 2024-09-13T11:24:10.947 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.947 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.947 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:24:10.948 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.068375+0000", 2024-09-13T11:24:10.948 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.1805000000000003e-05 2024-09-13T11:24:10.948 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.948 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.948 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for peered", 2024-09-13T11:24:10.948 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.068382+0000", 2024-09-13T11:24:10.948 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.224e-06 2024-09-13T11:24:10.948 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.948 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.948 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:24:10.948 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.907277+0000", 2024-09-13T11:24:10.948 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.83889509500000004 2024-09-13T11:24:10.948 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.948 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.949 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:24:10.949 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.907284+0000", 2024-09-13T11:24:10.949 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.6359999999999994e-06 2024-09-13T11:24:10.949 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:10.949 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:24:10.949 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:10.949 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.949 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.949 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:54846 3.1e 3.742e049e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-13T11:24:10.949 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:32.153567+0000", 2024-09-13T11:24:10.949 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 38.712769311000002, 2024-09-13T11:24:10.949 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.75378642399999995, 2024-09-13T11:24:10.949 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:24:10.950 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:24:10.950 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-13T11:24:10.950 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:24:10.950 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:24:10.950 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:10.950 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 54846 2024-09-13T11:24:10.950 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.950 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:24:10.950 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.950 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:24:10.950 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.153567+0000", 2024-09-13T11:24:10.950 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:10.950 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.951 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.951 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:24:10.951 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.153567+0000", 2024-09-13T11:24:10.951 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:10.951 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.951 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.951 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:24:10.951 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.153569+0000", 2024-09-13T11:24:10.951 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.4660000000000002e-06 2024-09-13T11:24:10.951 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.951 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.951 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:24:10.951 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.153619+0000", 2024-09-13T11:24:10.952 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.9617000000000003e-05 2024-09-13T11:24:10.952 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.952 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.952 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:24:10.952 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.153634+0000", 2024-09-13T11:24:10.952 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.4904e-05 2024-09-13T11:24:10.952 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.952 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.952 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:24:10.952 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.153642+0000", 2024-09-13T11:24:10.952 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.5599999999999994e-06 2024-09-13T11:24:10.952 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.953 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.953 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:24:10.953 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.153721+0000", 2024-09-13T11:24:10.953 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.8603999999999995e-05 2024-09-13T11:24:10.953 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.953 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.953 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for peered", 2024-09-13T11:24:10.953 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.153730+0000", 2024-09-13T11:24:10.953 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.3840000000000003e-06 2024-09-13T11:24:10.953 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.953 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.953 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:24:10.953 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.907346+0000", 2024-09-13T11:24:10.954 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.75361563099999995 2024-09-13T11:24:10.954 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.954 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.954 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:24:10.954 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.907353+0000", 2024-09-13T11:24:10.954 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.2579999999999998e-06 2024-09-13T11:24:10.954 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:10.954 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:24:10.954 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:10.954 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.954 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.954 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:54888 3.1e 3.43e14ede (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-13T11:24:10.954 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:32.423064+0000", 2024-09-13T11:24:10.954 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 38.443271639999999, 2024-09-13T11:24:10.955 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.48433585299999998, 2024-09-13T11:24:10.955 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:24:10.955 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:24:10.955 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-13T11:24:10.955 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:24:10.955 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:24:10.955 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:10.955 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 54888 2024-09-13T11:24:10.955 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.955 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:24:10.955 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.955 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:24:10.955 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.423064+0000", 2024-09-13T11:24:10.955 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:10.956 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.956 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.956 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:24:10.956 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.423064+0000", 2024-09-13T11:24:10.956 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:10.956 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.956 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.956 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:24:10.956 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.423069+0000", 2024-09-13T11:24:10.956 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.7389999999999999e-06 2024-09-13T11:24:10.956 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.957 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.957 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:24:10.957 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.423115+0000", 2024-09-13T11:24:10.957 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.5847000000000003e-05 2024-09-13T11:24:10.957 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.957 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.957 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:24:10.957 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.423126+0000", 2024-09-13T11:24:10.957 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.1418e-05 2024-09-13T11:24:10.957 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.957 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.958 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:24:10.958 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.423132+0000", 2024-09-13T11:24:10.958 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.57e-06 2024-09-13T11:24:10.958 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.958 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.958 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:24:10.958 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.423191+0000", 2024-09-13T11:24:10.958 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.9250999999999999e-05 2024-09-13T11:24:10.958 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.958 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.958 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for peered", 2024-09-13T11:24:10.958 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.423198+0000", 2024-09-13T11:24:10.958 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.9179999999999997e-06 2024-09-13T11:24:10.958 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.959 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.959 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:24:10.959 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.907392+0000", 2024-09-13T11:24:10.959 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.48419437500000001 2024-09-13T11:24:10.959 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.959 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.959 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:24:10.959 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.907400+0000", 2024-09-13T11:24:10.959 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.7349999999999996e-06 2024-09-13T11:24:10.959 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:10.959 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:24:10.959 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:10.959 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.960 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.960 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDPGPushReply(3.17 76/70 [PushReplyOp(3:eb8d2fae:::benchmark_data_smithi177_31116_object18267:head)])", 2024-09-13T11:24:10.960 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:32.636274+0000", 2024-09-13T11:24:10.960 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 38.230062046999997, 2024-09-13T11:24:10.960 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.269081036, 2024-09-13T11:24:10.960 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:24:10.960 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:24:10.960 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-13T11:24:10.960 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:24:10.960 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.960 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:24:10.961 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.636274+0000", 2024-09-13T11:24:10.961 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:10.961 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.961 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.961 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:24:10.961 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.636274+0000", 2024-09-13T11:24:10.961 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:10.961 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.961 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.961 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:24:10.961 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.636276+0000", 2024-09-13T11:24:10.961 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.1679999999999998e-06 2024-09-13T11:24:10.961 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.961 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.961 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:24:10.962 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.636284+0000", 2024-09-13T11:24:10.962 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.7440000000000007e-06 2024-09-13T11:24:10.962 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.962 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.962 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:24:10.962 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.904155+0000", 2024-09-13T11:24:10.962 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.26787080800000002 2024-09-13T11:24:10.962 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.962 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.962 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:24:10.962 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.904165+0000", 2024-09-13T11:24:10.962 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0750999999999999e-05 2024-09-13T11:24:10.962 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.962 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.963 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:24:10.963 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.904238+0000", 2024-09-13T11:24:10.963 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.2335000000000005e-05 2024-09-13T11:24:10.963 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.963 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.963 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:24:10.963 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.905355+0000", 2024-09-13T11:24:10.963 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0011172300000000001 2024-09-13T11:24:10.963 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:10.963 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:24:10.963 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:10.963 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.963 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.963 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDPGPushReply(3.17 76/70 [PushReplyOp(3:ebcbce67:::benchmark_data_smithi177_31116_object18268:head)])", 2024-09-13T11:24:10.964 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:32.641093+0000", 2024-09-13T11:24:10.964 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 38.225242498999997, 2024-09-13T11:24:10.964 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.27471406799999998, 2024-09-13T11:24:10.964 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:24:10.964 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:24:10.964 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-13T11:24:10.964 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:24:10.964 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.964 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:24:10.964 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.641093+0000", 2024-09-13T11:24:10.964 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:10.964 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.964 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.965 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:24:10.965 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.641093+0000", 2024-09-13T11:24:10.965 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:10.965 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.965 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.965 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:24:10.965 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.641094+0000", 2024-09-13T11:24:10.965 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.0299999999999999e-07 2024-09-13T11:24:10.965 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.965 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.965 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:24:10.965 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.641098+0000", 2024-09-13T11:24:10.965 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.2200000000000003e-06 2024-09-13T11:24:10.966 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.966 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.966 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:24:10.966 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.910434+0000", 2024-09-13T11:24:10.966 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.26933570600000001 2024-09-13T11:24:10.966 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.966 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.966 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:24:10.966 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.910442+0000", 2024-09-13T11:24:10.966 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.2700000000000004e-06 2024-09-13T11:24:10.966 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.966 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.966 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:24:10.966 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.914989+0000", 2024-09-13T11:24:10.967 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0045465059999999996 2024-09-13T11:24:10.967 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.967 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.967 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:24:10.967 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.915808+0000", 2024-09-13T11:24:10.967 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00081876300000000002 2024-09-13T11:24:10.967 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:10.967 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:24:10.967 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:10.967 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.967 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.967 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4665.0:59977 3.21 e83/78 3:86b4d8f6:::benchmark_data_smithi177_31116_object59976:head v 83'3136, mlcod=0'0)", 2024-09-13T11:24:10.967 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:40.571147+0000", 2024-09-13T11:24:10.968 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 30.295189190999999, 2024-09-13T11:24:10.968 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.51907542799999995, 2024-09-13T11:24:10.968 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:24:10.968 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:24:10.968 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:10.968 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:24:10.968 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.968 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:24:10.968 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:40.571147+0000", 2024-09-13T11:24:10.968 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:10.968 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.968 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.968 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:24:10.969 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:40.571147+0000", 2024-09-13T11:24:10.969 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:10.969 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.969 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.969 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:24:10.969 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:40.571151+0000", 2024-09-13T11:24:10.969 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.5650000000000003e-06 2024-09-13T11:24:10.969 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.969 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.969 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:24:10.969 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:40.571204+0000", 2024-09-13T11:24:10.969 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.2720999999999999e-05 2024-09-13T11:24:10.969 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.969 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.970 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:24:10.970 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:41.088332+0000", 2024-09-13T11:24:10.970 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.51712822000000003 2024-09-13T11:24:10.970 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.970 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.970 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:24:10.970 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:41.088341+0000", 2024-09-13T11:24:10.970 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.3100000000000001e-06 2024-09-13T11:24:10.970 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.970 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.970 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:24:10.970 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:41.088444+0000", 2024-09-13T11:24:10.970 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000103708 2024-09-13T11:24:10.970 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.970 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.971 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:24:10.971 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:41.088475+0000", 2024-09-13T11:24:10.971 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.0497e-05 2024-09-13T11:24:10.971 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.971 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.971 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:24:10.971 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:41.090192+0000", 2024-09-13T11:24:10.971 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.001717288 2024-09-13T11:24:10.971 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:10.971 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:10.971 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:24:10.971 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:41.090222+0000", 2024-09-13T11:24:10.971 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.0119000000000001e-05 2024-09-13T11:24:10.971 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:10.972 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:24:10.972 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:10.972 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:10.972 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:24:10.972 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-13T11:24:10.973 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_ops_in_flight 2024-09-13T11:24:10.979 INFO:teuthology.orchestra.run.smithi001.stderr:nodeep-scrub is unset 2024-09-13T11:24:10.983 INFO:tasks.ceph.osd.1.smithi092.stderr:2024-09-13T11:24:10.979+0000 7fe76d3f2640 -1 osd.1 110 handle_osd_map injecting map crc failure 2024-09-13T11:24:10.999 INFO:tasks.ceph.osd.7.smithi177.stderr:2024-09-13T11:24:10.971+0000 7f897b51d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26575) UID: 0 2024-09-13T11:24:11.004 INFO:tasks.ceph.osd.5.smithi092.stderr:2024-09-13T11:24:10.999+0000 7f6f02136640 -1 osd.5 110 handle_osd_map injecting map crc failure 2024-09-13T11:24:11.009 INFO:tasks.radosbench.radosbench.0.smithi177.stdout: 104 16 91571 91555 55.0155 88.3125 0.0422413 0.018168 2024-09-13T11:24:11.015 INFO:tasks.ceph.osd.11.smithi177.stderr:2024-09-13T11:24:10.995+0000 7f2469cec640 -1 osd.11 110 handle_osd_map injecting map crc failure 2024-09-13T11:24:11.035 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-13T11:24:11.036 INFO:teuthology.orchestra.run.smithi001.stderr:dumped all 2024-09-13T11:24:11.055 INFO:teuthology.orchestra.run.smithi001.stdout:{"pg_ready":true,"pg_map":{"version":157,"stamp":"2024-09-13T11:24:10.979943+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":5617091123,"num_objects":85706,"num_object_clones":0,"num_object_copies":171412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":85706,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":39320,"num_write_kb":2509682,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11627,"num_bytes_recovered":761888787,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":146263,"ondisk_log_size":146263,"up":90,"acting":90,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":114,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":16,"kb":1509949440,"kb_used":13791004,"kb_used_data":12880348,"kb_used_omap":27,"kb_used_meta":910372,"kb_avail":1496158436,"statfs":{"total":1546188226560,"available":1532066238464,"internally_reserved":0,"allocated":13189476352,"data_stored":13181945005,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28358,"internal_metadata":932221242},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,2,3,3,1,1,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":326959104,"num_objects":4989,"num_object_clones":0,"num_object_copies":9978,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1468,"num_objects_unfound":0,"num_objects_dirty":4989,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4989,"num_write_kb":319296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":735,"num_bytes_recovered":48168960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.041592"},"pg_stats":[{"pgid":"3.23","version":"108'3609","reported_seq":6341,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T11:24:05.966241+0000","last_change":"2024-09-13T11:23:48.133184+0000","last_active":"2024-09-13T11:24:05.966241+0000","last_peered":"2024-09-13T11:24:05.966241+0000","last_clean":"2024-09-13T11:24:05.966241+0000","last_became_active":"2024-09-13T11:23:47.926677+0000","last_became_peered":"2024-09-13T11:23:47.926677+0000","last_unstale":"2024-09-13T11:24:05.966241+0000","last_undegraded":"2024-09-13T11:24:05.966241+0000","last_fullsized":"2024-09-13T11:24:05.966241+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":6,"last_scrub":"91'3278","last_scrub_stamp":"2024-09-13T11:23:48.133084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:48.133084+0000","objects_scrubbed":1029,"log_size":3609,"log_dups_size":0,"ondisk_log_size":3609,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:12.874134+0000","scrub_duration":139,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":89194496,"num_objects":1361,"num_object_clones":0,"num_object_copies":2722,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1361,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":332,"num_write_kb":21248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9],"acting":[5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.22","version":"108'3574","reported_seq":5493,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T11:24:05.958953+0000","last_change":"2024-09-13T11:23:47.300446+0000","last_active":"2024-09-13T11:24:05.958953+0000","last_peered":"2024-09-13T11:24:05.958953+0000","last_clean":"2024-09-13T11:24:05.958953+0000","last_became_active":"2024-09-13T11:23:46.925318+0000","last_became_peered":"2024-09-13T11:23:46.925318+0000","last_unstale":"2024-09-13T11:24:05.958953+0000","last_undegraded":"2024-09-13T11:24:05.958953+0000","last_fullsized":"2024-09-13T11:24:05.958953+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":6,"last_scrub":"90'3246","last_scrub_stamp":"2024-09-13T11:23:47.300339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:47.300339+0000","objects_scrubbed":1033,"log_size":3574,"log_dups_size":0,"ondisk_log_size":3574,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:24:49.780485+0000","scrub_duration":203,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":89325568,"num_objects":1363,"num_object_clones":0,"num_object_copies":2726,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1363,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":330,"num_write_kb":21120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,14],"acting":[0,14],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.21","version":"108'3525","reported_seq":6247,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T11:24:05.984268+0000","last_change":"2024-09-13T11:24:02.142682+0000","last_active":"2024-09-13T11:24:05.984268+0000","last_peered":"2024-09-13T11:24:05.984268+0000","last_clean":"2024-09-13T11:24:05.984268+0000","last_became_active":"2024-09-13T11:24:01.250308+0000","last_became_peered":"2024-09-13T11:24:01.250308+0000","last_unstale":"2024-09-13T11:24:05.984268+0000","last_undegraded":"2024-09-13T11:24:05.984268+0000","last_fullsized":"2024-09-13T11:24:05.984268+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":6,"last_scrub":"69'2784","last_scrub_stamp":"2024-09-13T11:23:21.595419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:21.595419+0000","objects_scrubbed":0,"log_size":3525,"log_dups_size":0,"ondisk_log_size":3525,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"user requested scrub scheduled @ 2024-09-13T11:24:07.138422+0000","scrub_duration":311,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":84738048,"num_objects":1293,"num_object_clones":0,"num_object_copies":2586,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1293,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":600,"num_write_kb":38400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":836,"num_bytes_recovered":54788096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8],"acting":[0,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.20","version":"108'3606","reported_seq":5640,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T11:24:05.969931+0000","last_change":"2024-09-13T11:23:47.969503+0000","last_active":"2024-09-13T11:24:05.969931+0000","last_peered":"2024-09-13T11:24:05.969931+0000","last_clean":"2024-09-13T11:24:05.969931+0000","last_became_active":"2024-09-13T11:23:33.954946+0000","last_became_peered":"2024-09-13T11:23:33.954946+0000","last_unstale":"2024-09-13T11:24:05.969931+0000","last_undegraded":"2024-09-13T11:24:05.969931+0000","last_fullsized":"2024-09-13T11:24:05.969931+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":6,"last_scrub":"91'3270","last_scrub_stamp":"2024-09-13T11:23:47.969372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:47.969372+0000","objects_scrubbed":1061,"log_size":3606,"log_dups_size":0,"ondisk_log_size":3606,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:24:48.255263+0000","scrub_duration":1327,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":91881472,"num_objects":1402,"num_object_clones":0,"num_object_copies":2804,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1402,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":341,"num_write_kb":21824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,0],"acting":[12,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[]},{"pgid":"3.1f","version":"107'4324","reported_seq":9453,"reported_epoch":108,"state":"peering","last_fresh":"2024-09-13T11:24:05.943810+0000","last_change":"2024-09-13T11:24:05.943810+0000","last_active":"2024-09-13T11:24:05.943810+0000","last_peered":"2024-09-13T11:24:05.897907+0000","last_clean":"2024-09-13T11:23:32.316727+0000","last_became_active":"2024-09-13T11:23:35.053803+0000","last_became_peered":"2024-09-13T11:23:35.053803+0000","last_unstale":"2024-09-13T11:24:05.943810+0000","last_undegraded":"2024-09-13T11:24:05.943810+0000","last_fullsized":"2024-09-13T11:24:05.943810+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:22:24.896055+0000","objects_scrubbed":0,"log_size":4324,"log_dups_size":0,"ondisk_log_size":4324,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":171540480,"num_objects":2618,"num_object_clones":0,"num_object_copies":5236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2618,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2618,"num_write_kb":167520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1651,"num_bytes_recovered":108199936,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,14],"acting":[10,14],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1e","version":"107'4444","reported_seq":14807,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T11:24:05.956882+0000","last_change":"2024-09-13T11:24:02.037297+0000","last_active":"2024-09-13T11:24:05.956882+0000","last_peered":"2024-09-13T11:24:05.956882+0000","last_clean":"2024-09-13T11:24:05.956882+0000","last_became_active":"2024-09-13T11:24:01.261448+0000","last_became_peered":"2024-09-13T11:24:01.261448+0000","last_unstale":"2024-09-13T11:24:05.956882+0000","last_undegraded":"2024-09-13T11:24:05.956882+0000","last_fullsized":"2024-09-13T11:24:05.956882+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:22:24.896055+0000","objects_scrubbed":0,"log_size":4444,"log_dups_size":0,"ondisk_log_size":4444,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-13T11:24:06.974692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":175898624,"num_objects":2684,"num_object_clones":0,"num_object_copies":5368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2684,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2684,"num_write_kb":171776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2573,"num_bytes_recovered":168591360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"108'4306","reported_seq":9591,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T11:24:06.011843+0000","last_change":"2024-09-13T11:24:02.078835+0000","last_active":"2024-09-13T11:24:06.011843+0000","last_peered":"2024-09-13T11:24:06.011843+0000","last_clean":"2024-09-13T11:24:06.011843+0000","last_became_active":"2024-09-13T11:24:01.258225+0000","last_became_peered":"2024-09-13T11:24:01.258225+0000","last_unstale":"2024-09-13T11:24:06.011843+0000","last_undegraded":"2024-09-13T11:24:06.011843+0000","last_fullsized":"2024-09-13T11:24:06.011843+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:22:24.896055+0000","objects_scrubbed":0,"log_size":4306,"log_dups_size":0,"ondisk_log_size":4306,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-13T11:24:07.039585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":171376640,"num_objects":2615,"num_object_clones":0,"num_object_copies":5230,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2615,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2615,"num_write_kb":167360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1700,"num_bytes_recovered":111411200,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,15],"acting":[2,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1c","version":"109'4429","reported_seq":8115,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T11:24:07.057488+0000","last_change":"2024-09-13T11:24:00.529312+0000","last_active":"2024-09-13T11:24:07.057488+0000","last_peered":"2024-09-13T11:24:07.057488+0000","last_clean":"2024-09-13T11:24:07.057488+0000","last_became_active":"2024-09-13T11:23:33.989492+0000","last_became_peered":"2024-09-13T11:23:33.989492+0000","last_unstale":"2024-09-13T11:24:07.057488+0000","last_undegraded":"2024-09-13T11:24:07.057488+0000","last_fullsized":"2024-09-13T11:24:07.057488+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":5,"last_scrub":"102'4277","last_scrub_stamp":"2024-09-13T11:24:00.529243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:24:00.529243+0000","objects_scrubbed":2544,"log_size":4429,"log_dups_size":0,"ondisk_log_size":4429,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:15.595052+0000","scrub_duration":259,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":176816128,"num_objects":2698,"num_object_clones":0,"num_object_copies":5396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2698,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":154,"num_write_kb":9856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7],"acting":[11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1b","version":"108'4380","reported_seq":9848,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T11:24:06.092111+0000","last_change":"2024-09-13T11:24:05.835103+0000","last_active":"2024-09-13T11:24:06.092111+0000","last_peered":"2024-09-13T11:24:06.092111+0000","last_clean":"2024-09-13T11:24:06.092111+0000","last_became_active":"2024-09-13T11:24:04.950821+0000","last_became_peered":"2024-09-13T11:24:04.950821+0000","last_unstale":"2024-09-13T11:24:06.092111+0000","last_undegraded":"2024-09-13T11:24:06.092111+0000","last_fullsized":"2024-09-13T11:24:06.092111+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:22:24.896055+0000","objects_scrubbed":0,"log_size":4380,"log_dups_size":0,"ondisk_log_size":4380,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-13T11:24:10.806111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":173637632,"num_objects":2650,"num_object_clones":0,"num_object_copies":5300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2650,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2650,"num_write_kb":169568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1760,"num_bytes_recovered":115343360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,15],"acting":[12,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[]},{"pgid":"3.1a","version":"108'4273","reported_seq":9405,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T11:24:06.022201+0000","last_change":"2024-09-13T11:24:04.144689+0000","last_active":"2024-09-13T11:24:06.022201+0000","last_peered":"2024-09-13T11:24:06.022201+0000","last_clean":"2024-09-13T11:24:06.022201+0000","last_became_active":"2024-09-13T11:24:03.935185+0000","last_became_peered":"2024-09-13T11:24:03.935185+0000","last_unstale":"2024-09-13T11:24:06.022201+0000","last_undegraded":"2024-09-13T11:24:06.022201+0000","last_fullsized":"2024-09-13T11:24:06.022201+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:22:24.896055+0000","objects_scrubbed":0,"log_size":4273,"log_dups_size":0,"ondisk_log_size":4273,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-13T11:24:09.133075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":168230912,"num_objects":2567,"num_object_clones":0,"num_object_copies":5134,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2567,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2567,"num_write_kb":164288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1648,"num_bytes_recovered":108003328,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9],"acting":[4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.19","version":"109'4031","reported_seq":8340,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T11:24:07.006880+0000","last_change":"2024-09-13T11:23:49.429522+0000","last_active":"2024-09-13T11:24:07.006880+0000","last_peered":"2024-09-13T11:24:07.006880+0000","last_clean":"2024-09-13T11:24:07.006880+0000","last_became_active":"2024-09-13T11:23:41.924941+0000","last_became_peered":"2024-09-13T11:23:41.924941+0000","last_unstale":"2024-09-13T11:24:07.006880+0000","last_undegraded":"2024-09-13T11:24:07.006880+0000","last_fullsized":"2024-09-13T11:24:07.006880+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":5,"last_scrub":"92'3415","last_scrub_stamp":"2024-09-13T11:23:49.429429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:49.429429+0000","objects_scrubbed":1979,"log_size":4031,"log_dups_size":0,"ondisk_log_size":4031,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:04.629542+0000","scrub_duration":587,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":170917888,"num_objects":2608,"num_object_clones":0,"num_object_copies":5216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2608,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":629,"num_write_kb":40256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1],"acting":[9,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.18","version":"108'4042","reported_seq":8617,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T11:24:05.971274+0000","last_change":"2024-09-13T11:24:02.096598+0000","last_active":"2024-09-13T11:24:05.971274+0000","last_peered":"2024-09-13T11:24:05.971274+0000","last_clean":"2024-09-13T11:24:05.971274+0000","last_became_active":"2024-09-13T11:23:48.926014+0000","last_became_peered":"2024-09-13T11:23:48.926014+0000","last_unstale":"2024-09-13T11:24:05.971274+0000","last_undegraded":"2024-09-13T11:24:05.971274+0000","last_fullsized":"2024-09-13T11:24:05.971274+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:22:24.896055+0000","objects_scrubbed":0,"log_size":4042,"log_dups_size":0,"ondisk_log_size":4042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-13T11:24:07.029225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":175308800,"num_objects":2675,"num_object_clones":0,"num_object_copies":5350,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2675,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2675,"num_write_kb":171200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1458,"num_bytes_recovered":95551488,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0],"acting":[10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.17","version":"107'3954","reported_seq":7995,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T11:24:05.943810+0000","last_change":"2024-09-13T11:23:49.349241+0000","last_active":"2024-09-13T11:24:05.943810+0000","last_peered":"2024-09-13T11:24:05.943810+0000","last_clean":"2024-09-13T11:24:05.943810+0000","last_became_active":"2024-09-13T11:23:42.935601+0000","last_became_peered":"2024-09-13T11:23:42.935601+0000","last_unstale":"2024-09-13T11:24:05.943810+0000","last_undegraded":"2024-09-13T11:24:05.943810+0000","last_fullsized":"2024-09-13T11:24:05.943810+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":5,"last_scrub":"92'3359","last_scrub_stamp":"2024-09-13T11:23:49.349167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:49.349167+0000","objects_scrubbed":2004,"log_size":3954,"log_dups_size":0,"ondisk_log_size":3954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:24:59.401302+0000","scrub_duration":363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":170852352,"num_objects":2607,"num_object_clones":0,"num_object_copies":5214,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2607,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":603,"num_write_kb":38592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3],"acting":[10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"109'4209","reported_seq":8882,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T11:24:07.047128+0000","last_change":"2024-09-13T11:23:46.510544+0000","last_active":"2024-09-13T11:24:07.047128+0000","last_peered":"2024-09-13T11:24:07.047128+0000","last_clean":"2024-09-13T11:24:07.047128+0000","last_became_active":"2024-09-13T11:23:46.105237+0000","last_became_peered":"2024-09-13T11:23:46.105237+0000","last_unstale":"2024-09-13T11:24:07.047128+0000","last_undegraded":"2024-09-13T11:24:07.047128+0000","last_fullsized":"2024-09-13T11:24:07.047128+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":5,"last_scrub":"89'3478","last_scrub_stamp":"2024-09-13T11:23:46.510384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:46.510384+0000","objects_scrubbed":2054,"log_size":4209,"log_dups_size":0,"ondisk_log_size":4209,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:09.149878+0000","scrub_duration":363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":182714368,"num_objects":2788,"num_object_clones":0,"num_object_copies":5576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2788,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":734,"num_write_kb":46976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,15],"acting":[11,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.15","version":"108'3978","reported_seq":8409,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T11:24:06.018660+0000","last_change":"2024-09-13T11:23:48.332790+0000","last_active":"2024-09-13T11:24:06.018660+0000","last_peered":"2024-09-13T11:24:06.018660+0000","last_clean":"2024-09-13T11:24:06.018660+0000","last_became_active":"2024-09-13T11:23:42.931149+0000","last_became_peered":"2024-09-13T11:23:42.931149+0000","last_unstale":"2024-09-13T11:24:06.018660+0000","last_undegraded":"2024-09-13T11:24:06.018660+0000","last_fullsized":"2024-09-13T11:24:06.018660+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":5,"last_scrub":"91'3384","last_scrub_stamp":"2024-09-13T11:23:48.332600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:48.332600+0000","objects_scrubbed":2032,"log_size":3978,"log_dups_size":0,"ondisk_log_size":3978,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:24:58.239508+0000","scrub_duration":235,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":172163072,"num_objects":2627,"num_object_clones":0,"num_object_copies":5254,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2627,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":595,"num_write_kb":38080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,0],"acting":[14,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"3.14","version":"110'4212","reported_seq":8467,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T11:24:10.043578+0000","last_change":"2024-09-13T11:23:48.697100+0000","last_active":"2024-09-13T11:24:10.043578+0000","last_peered":"2024-09-13T11:24:10.043578+0000","last_clean":"2024-09-13T11:24:10.043578+0000","last_became_active":"2024-09-13T11:23:41.920989+0000","last_became_peered":"2024-09-13T11:23:41.920989+0000","last_unstale":"2024-09-13T11:24:10.043578+0000","last_undegraded":"2024-09-13T11:24:10.043578+0000","last_fullsized":"2024-09-13T11:24:10.043578+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":5,"last_scrub":"91'3371","last_scrub_stamp":"2024-09-13T11:23:48.696960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:48.696960+0000","objects_scrubbed":1935,"log_size":4212,"log_dups_size":0,"ondisk_log_size":4212,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:02.861530+0000","scrub_duration":735,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":182583296,"num_objects":2786,"num_object_clones":0,"num_object_copies":5572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2786,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":851,"num_write_kb":54464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,14],"acting":[3,14],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":179,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T11:24:05.944977+0000","last_change":"2024-09-13T11:22:16.889130+0000","last_active":"2024-09-13T11:24:05.944977+0000","last_peered":"2024-09-13T11:24:05.944977+0000","last_clean":"2024-09-13T11:24:05.944977+0000","last_became_active":"2024-09-13T11:22:14.192897+0000","last_became_peered":"2024-09-13T11:22:14.192897+0000","last_unstale":"2024-09-13T11:24:05.944977+0000","last_undegraded":"2024-09-13T11:24:05.944977+0000","last_fullsized":"2024-09-13T11:24:05.944977+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:22:13.113563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:24:09.524691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00084655899999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,10],"acting":[15,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"3.6","version":"109'4151","reported_seq":4350,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T11:24:07.043658+0000","last_change":"2024-09-13T11:23:22.656608+0000","last_active":"2024-09-13T11:24:07.043658+0000","last_peered":"2024-09-13T11:24:07.043658+0000","last_clean":"2024-09-13T11:24:07.043658+0000","last_became_active":"2024-09-13T11:23:19.939432+0000","last_became_peered":"2024-09-13T11:23:19.939432+0000","last_unstale":"2024-09-13T11:24:07.043658+0000","last_undegraded":"2024-09-13T11:24:07.043658+0000","last_fullsized":"2024-09-13T11:24:07.043658+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'2909","last_scrub_stamp":"2024-09-13T11:23:22.656323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:22.656323+0000","objects_scrubbed":1487,"log_size":4151,"log_dups_size":0,"ondisk_log_size":4151,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:24:31.905185+0000","scrub_duration":323,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":178847744,"num_objects":2729,"num_object_clones":0,"num_object_copies":5458,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2729,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1242,"num_write_kb":79488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,14],"acting":[13,14],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":188,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T11:24:06.994925+0000","last_change":"2024-09-13T11:23:21.875855+0000","last_active":"2024-09-13T11:24:06.994925+0000","last_peered":"2024-09-13T11:24:06.994925+0000","last_clean":"2024-09-13T11:24:06.994925+0000","last_became_active":"2024-09-13T11:22:14.211923+0000","last_became_peered":"2024-09-13T11:22:14.211923+0000","last_unstale":"2024-09-13T11:24:06.994925+0000","last_undegraded":"2024-09-13T11:24:06.994925+0000","last_fullsized":"2024-09-13T11:24:06.994925+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:23:21.875683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:23:21.875683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:24:40.678136+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0011689529999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0],"acting":[9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"109'3978","reported_seq":4178,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T11:24:07.040769+0000","last_change":"2024-09-13T11:23:20.443418+0000","last_active":"2024-09-13T11:24:07.040769+0000","last_peered":"2024-09-13T11:24:07.040769+0000","last_clean":"2024-09-13T11:24:07.040769+0000","last_became_active":"2024-09-13T11:23:19.927680+0000","last_became_peered":"2024-09-13T11:23:19.927680+0000","last_unstale":"2024-09-13T11:24:07.040769+0000","last_undegraded":"2024-09-13T11:24:07.040769+0000","last_fullsized":"2024-09-13T11:24:07.040769+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'2736","last_scrub_stamp":"2024-09-13T11:23:20.443329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:20.443329+0000","objects_scrubbed":1369,"log_size":3978,"log_dups_size":0,"ondisk_log_size":3978,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:24:48.147625+0000","scrub_duration":183,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":171311104,"num_objects":2614,"num_object_clones":0,"num_object_copies":5228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2614,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1245,"num_write_kb":79680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,8],"acting":[13,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":187,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T11:24:05.957598+0000","last_change":"2024-09-13T11:23:59.530794+0000","last_active":"2024-09-13T11:24:05.957598+0000","last_peered":"2024-09-13T11:24:05.957598+0000","last_clean":"2024-09-13T11:24:05.957598+0000","last_became_active":"2024-09-13T11:22:14.581458+0000","last_became_peered":"2024-09-13T11:22:14.581458+0000","last_unstale":"2024-09-13T11:24:05.957598+0000","last_undegraded":"2024-09-13T11:24:05.957598+0000","last_fullsized":"2024-09-13T11:24:05.957598+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:23:59.530670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:23:59.530670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:02.655631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001026047,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5],"acting":[8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"110'4190","reported_seq":4372,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T11:24:10.320117+0000","last_change":"2024-09-13T11:23:20.254213+0000","last_active":"2024-09-13T11:24:10.320117+0000","last_peered":"2024-09-13T11:24:10.320117+0000","last_clean":"2024-09-13T11:24:10.320117+0000","last_became_active":"2024-09-13T11:23:19.925399+0000","last_became_peered":"2024-09-13T11:23:19.925399+0000","last_unstale":"2024-09-13T11:24:10.320117+0000","last_undegraded":"2024-09-13T11:24:10.320117+0000","last_fullsized":"2024-09-13T11:24:10.320117+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'2755","last_scrub_stamp":"2024-09-13T11:23:20.254124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:20.254124+0000","objects_scrubbed":1437,"log_size":4190,"log_dups_size":0,"ondisk_log_size":4190,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:24:49.993543+0000","scrub_duration":187,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":188547072,"num_objects":2877,"num_object_clones":0,"num_object_copies":5754,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2877,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1440,"num_write_kb":92160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,12],"acting":[1,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":179,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T11:24:08.271026+0000","last_change":"2024-09-13T11:24:08.271026+0000","last_active":"2024-09-13T11:24:08.271026+0000","last_peered":"2024-09-13T11:24:08.271026+0000","last_clean":"2024-09-13T11:24:08.271026+0000","last_became_active":"2024-09-13T11:22:47.719636+0000","last_became_peered":"2024-09-13T11:22:47.719636+0000","last_unstale":"2024-09-13T11:24:08.271026+0000","last_undegraded":"2024-09-13T11:24:08.271026+0000","last_fullsized":"2024-09-13T11:24:08.271026+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:24:08.270972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:24:08.270972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:36.805605+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00087476000000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,9],"acting":[1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"108'4001","reported_seq":4199,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T11:24:05.961301+0000","last_change":"2024-09-13T11:23:20.686446+0000","last_active":"2024-09-13T11:24:05.961301+0000","last_peered":"2024-09-13T11:24:05.961301+0000","last_clean":"2024-09-13T11:24:05.961301+0000","last_became_active":"2024-09-13T11:23:19.939695+0000","last_became_peered":"2024-09-13T11:23:19.939695+0000","last_unstale":"2024-09-13T11:24:05.961301+0000","last_undegraded":"2024-09-13T11:24:05.961301+0000","last_fullsized":"2024-09-13T11:24:05.961301+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'2739","last_scrub_stamp":"2024-09-13T11:23:20.686358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:20.686358+0000","objects_scrubbed":1386,"log_size":4001,"log_dups_size":0,"ondisk_log_size":4001,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:24:38.891737+0000","scrub_duration":271,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":173604864,"num_objects":2649,"num_object_clones":0,"num_object_copies":5298,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2649,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1263,"num_write_kb":80832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,13],"acting":[15,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"3.3","version":"107'3547","reported_seq":3768,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T11:24:05.942396+0000","last_change":"2024-09-13T11:23:32.147115+0000","last_active":"2024-09-13T11:24:05.942396+0000","last_peered":"2024-09-13T11:24:05.942396+0000","last_clean":"2024-09-13T11:24:05.942396+0000","last_became_active":"2024-09-13T11:23:31.930448+0000","last_became_peered":"2024-09-13T11:23:31.930448+0000","last_unstale":"2024-09-13T11:24:05.942396+0000","last_undegraded":"2024-09-13T11:24:05.942396+0000","last_fullsized":"2024-09-13T11:24:05.942396+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'3122","last_scrub_stamp":"2024-09-13T11:23:32.147021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:32.147021+0000","objects_scrubbed":876,"log_size":3547,"log_dups_size":0,"ondisk_log_size":3547,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:24:48.175541+0000","scrub_duration":99,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":85262336,"num_objects":1301,"num_object_clones":0,"num_object_copies":2602,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1301,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":425,"num_write_kb":27200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,7],"acting":[10,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"28'2","reported_seq":117,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T11:24:05.963496+0000","last_change":"2024-09-13T11:24:00.996197+0000","last_active":"2024-09-13T11:24:05.963496+0000","last_peered":"2024-09-13T11:24:05.963496+0000","last_clean":"2024-09-13T11:24:05.963496+0000","last_became_active":"2024-09-13T11:23:12.546348+0000","last_became_peered":"2024-09-13T11:23:12.546348+0000","last_unstale":"2024-09-13T11:24:05.963496+0000","last_undegraded":"2024-09-13T11:24:05.963496+0000","last_fullsized":"2024-09-13T11:24:05.963496+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2","last_scrub_stamp":"2024-09-13T11:24:00.996028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:24:00.996028+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:12.111097+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0025826730000000002,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8],"acting":[2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"108'3566","reported_seq":3785,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T11:24:06.031700+0000","last_change":"2024-09-13T11:23:32.749651+0000","last_active":"2024-09-13T11:24:06.031700+0000","last_peered":"2024-09-13T11:24:06.031700+0000","last_clean":"2024-09-13T11:24:06.031700+0000","last_became_active":"2024-09-13T11:23:31.921388+0000","last_became_peered":"2024-09-13T11:23:31.921388+0000","last_unstale":"2024-09-13T11:24:06.031700+0000","last_undegraded":"2024-09-13T11:24:06.031700+0000","last_fullsized":"2024-09-13T11:24:06.031700+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'3106","last_scrub_stamp":"2024-09-13T11:23:32.749487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:32.749487+0000","objects_scrubbed":856,"log_size":3566,"log_dups_size":0,"ondisk_log_size":3566,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:24:34.709501+0000","scrub_duration":91,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":86245376,"num_objects":1316,"num_object_clones":0,"num_object_copies":2632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1316,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":460,"num_write_kb":29440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,15],"acting":[12,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":189,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T11:24:05.943005+0000","last_change":"2024-09-13T11:22:47.298206+0000","last_active":"2024-09-13T11:24:05.943005+0000","last_peered":"2024-09-13T11:24:05.943005+0000","last_clean":"2024-09-13T11:24:05.943005+0000","last_became_active":"2024-09-13T11:22:47.297319+0000","last_became_peered":"2024-09-13T11:22:47.297319+0000","last_unstale":"2024-09-13T11:24:05.943005+0000","last_undegraded":"2024-09-13T11:24:05.943005+0000","last_fullsized":"2024-09-13T11:24:05.943005+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:22:13.113563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00081225399999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,1],"acting":[14,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"3.1","version":"109'3544","reported_seq":3764,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T11:24:07.048279+0000","last_change":"2024-09-13T11:23:33.604966+0000","last_active":"2024-09-13T11:24:07.048279+0000","last_peered":"2024-09-13T11:24:07.048279+0000","last_clean":"2024-09-13T11:24:07.048279+0000","last_became_active":"2024-09-13T11:23:31.926528+0000","last_became_peered":"2024-09-13T11:23:31.926528+0000","last_unstale":"2024-09-13T11:24:07.048279+0000","last_undegraded":"2024-09-13T11:24:07.048279+0000","last_fullsized":"2024-09-13T11:24:07.048279+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'3070","last_scrub_stamp":"2024-09-13T11:23:33.604792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:33.604792+0000","objects_scrubbed":842,"log_size":3544,"log_dups_size":0,"ondisk_log_size":3544,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:24:43.880038+0000","scrub_duration":119,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":86245376,"num_objects":1316,"num_object_clones":0,"num_object_copies":2632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1316,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":474,"num_write_kb":30336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,11],"acting":[13,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":183,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T11:24:07.959659+0000","last_change":"2024-09-13T11:22:16.267999+0000","last_active":"2024-09-13T11:24:07.959659+0000","last_peered":"2024-09-13T11:24:07.959659+0000","last_clean":"2024-09-13T11:24:07.959659+0000","last_became_active":"2024-09-13T11:22:14.211435+0000","last_became_peered":"2024-09-13T11:22:14.211435+0000","last_unstale":"2024-09-13T11:24:07.959659+0000","last_undegraded":"2024-09-13T11:24:07.959659+0000","last_fullsized":"2024-09-13T11:24:07.959659+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:22:13.113563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:24:10.770081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010010360000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2","version":"108'3556","reported_seq":3775,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T11:24:05.944539+0000","last_change":"2024-09-13T11:23:32.977666+0000","last_active":"2024-09-13T11:24:05.944539+0000","last_peered":"2024-09-13T11:24:05.944539+0000","last_clean":"2024-09-13T11:24:05.944539+0000","last_became_active":"2024-09-13T11:23:31.919828+0000","last_became_peered":"2024-09-13T11:23:31.919828+0000","last_unstale":"2024-09-13T11:24:05.944539+0000","last_undegraded":"2024-09-13T11:24:05.944539+0000","last_fullsized":"2024-09-13T11:24:05.944539+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'3057","last_scrub_stamp":"2024-09-13T11:23:32.977559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:32.977559+0000","objects_scrubbed":841,"log_size":3556,"log_dups_size":0,"ondisk_log_size":3556,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:24:52.169339+0000","scrub_duration":151,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":87818240,"num_objects":1340,"num_object_clones":0,"num_object_copies":2680,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1340,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":499,"num_write_kb":31936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,12],"acting":[14,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"2.3","version":"26'1","reported_seq":181,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T11:24:05.943338+0000","last_change":"2024-09-13T11:22:16.547691+0000","last_active":"2024-09-13T11:24:05.943338+0000","last_peered":"2024-09-13T11:24:05.943338+0000","last_clean":"2024-09-13T11:24:05.943338+0000","last_became_active":"2024-09-13T11:22:14.146507+0000","last_became_peered":"2024-09-13T11:22:14.146507+0000","last_unstale":"2024-09-13T11:24:05.943338+0000","last_undegraded":"2024-09-13T11:24:05.943338+0000","last_fullsized":"2024-09-13T11:24:05.943338+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:22:13.113563+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00085610999999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,15],"acting":[14,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"1.0","version":"26'112","reported_seq":321,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T11:24:05.945516+0000","last_change":"2024-09-13T11:23:49.325249+0000","last_active":"2024-09-13T11:24:05.945516+0000","last_peered":"2024-09-13T11:24:05.945516+0000","last_clean":"2024-09-13T11:24:05.945516+0000","last_became_active":"2024-09-13T11:22:12.129721+0000","last_became_peered":"2024-09-13T11:22:12.129721+0000","last_unstale":"2024-09-13T11:24:05.945516+0000","last_undegraded":"2024-09-13T11:24:05.945516+0000","last_fullsized":"2024-09-13T11:24:05.945516+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"26'112","last_scrub_stamp":"2024-09-13T11:23:49.325175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:11.098388+0000","last_clean_scrub_stamp":"2024-09-13T11:23:49.325175+0000","objects_scrubbed":2,"log_size":112,"log_dups_size":0,"ondisk_log_size":112,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:14.422287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.8","version":"108'3977","reported_seq":4176,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T11:24:06.016291+0000","last_change":"2024-09-13T11:23:20.644952+0000","last_active":"2024-09-13T11:24:06.016291+0000","last_peered":"2024-09-13T11:24:06.016291+0000","last_clean":"2024-09-13T11:24:06.016291+0000","last_became_active":"2024-09-13T11:23:19.925802+0000","last_became_peered":"2024-09-13T11:23:19.925802+0000","last_unstale":"2024-09-13T11:24:06.016291+0000","last_undegraded":"2024-09-13T11:24:06.016291+0000","last_fullsized":"2024-09-13T11:24:06.016291+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'2771","last_scrub_stamp":"2024-09-13T11:23:20.644747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:20.644747+0000","objects_scrubbed":1356,"log_size":3977,"log_dups_size":0,"ondisk_log_size":3977,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:24:38.144523+0000","scrub_duration":411,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":168689664,"num_objects":2574,"num_object_clones":0,"num_object_copies":5148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2574,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1218,"num_write_kb":77952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.9","version":"108'4078","reported_seq":4275,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T11:24:06.044946+0000","last_change":"2024-09-13T11:23:22.033678+0000","last_active":"2024-09-13T11:24:06.044946+0000","last_peered":"2024-09-13T11:24:06.044946+0000","last_clean":"2024-09-13T11:24:06.044946+0000","last_became_active":"2024-09-13T11:23:19.935114+0000","last_became_peered":"2024-09-13T11:23:19.935114+0000","last_unstale":"2024-09-13T11:24:06.044946+0000","last_undegraded":"2024-09-13T11:24:06.044946+0000","last_fullsized":"2024-09-13T11:24:06.044946+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'2808","last_scrub_stamp":"2024-09-13T11:23:22.033572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:22.033572+0000","objects_scrubbed":1471,"log_size":4078,"log_dups_size":0,"ondisk_log_size":4078,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:24:46.595859+0000","scrub_duration":287,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":179634176,"num_objects":2741,"num_object_clones":0,"num_object_copies":5482,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2741,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1270,"num_write_kb":81280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,2],"acting":[12,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[]},{"pgid":"3.a","version":"110'4531","reported_seq":4737,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T11:24:10.020132+0000","last_change":"2024-09-13T11:23:32.526150+0000","last_active":"2024-09-13T11:24:10.020132+0000","last_peered":"2024-09-13T11:24:10.020132+0000","last_clean":"2024-09-13T11:24:10.020132+0000","last_became_active":"2024-09-13T11:23:31.934170+0000","last_became_peered":"2024-09-13T11:23:31.934170+0000","last_unstale":"2024-09-13T11:24:10.020132+0000","last_undegraded":"2024-09-13T11:24:10.020132+0000","last_fullsized":"2024-09-13T11:24:10.020132+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'3418","last_scrub_stamp":"2024-09-13T11:23:32.526057+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:32.526057+0000","objects_scrubbed":1774,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:24:34.770161+0000","scrub_duration":507,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":189202432,"num_objects":2887,"num_object_clones":0,"num_object_copies":5774,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2887,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1113,"num_write_kb":71232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.b","version":"108'4387","reported_seq":4590,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T11:24:06.977573+0000","last_change":"2024-09-13T11:23:32.681343+0000","last_active":"2024-09-13T11:24:06.977573+0000","last_peered":"2024-09-13T11:24:06.977573+0000","last_clean":"2024-09-13T11:24:06.977573+0000","last_became_active":"2024-09-13T11:23:31.923415+0000","last_became_peered":"2024-09-13T11:23:31.923415+0000","last_unstale":"2024-09-13T11:24:06.977573+0000","last_undegraded":"2024-09-13T11:24:06.977573+0000","last_fullsized":"2024-09-13T11:24:06.977573+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'3471","last_scrub_stamp":"2024-09-13T11:23:32.681257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:32.681257+0000","objects_scrubbed":1720,"log_size":4387,"log_dups_size":0,"ondisk_log_size":4387,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:24:58.857812+0000","scrub_duration":199,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":172752896,"num_objects":2636,"num_object_clones":0,"num_object_copies":5272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":916,"num_write_kb":58624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,1],"acting":[13,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"3.c","version":"109'4386","reported_seq":4588,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T11:24:07.044343+0000","last_change":"2024-09-13T11:23:32.524870+0000","last_active":"2024-09-13T11:24:07.044343+0000","last_peered":"2024-09-13T11:24:07.044343+0000","last_clean":"2024-09-13T11:24:07.044343+0000","last_became_active":"2024-09-13T11:23:31.922620+0000","last_became_peered":"2024-09-13T11:23:31.922620+0000","last_unstale":"2024-09-13T11:24:07.044343+0000","last_undegraded":"2024-09-13T11:24:07.044343+0000","last_fullsized":"2024-09-13T11:24:07.044343+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'3463","last_scrub_stamp":"2024-09-13T11:23:32.524776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:32.524776+0000","objects_scrubbed":1741,"log_size":4386,"log_dups_size":0,"ondisk_log_size":4386,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:24:37.745747+0000","scrub_duration":235,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":174587904,"num_objects":2664,"num_object_clones":0,"num_object_copies":5328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2664,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":923,"num_write_kb":59072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,13],"acting":[11,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.d","version":"108'4363","reported_seq":4563,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T11:24:06.133586+0000","last_change":"2024-09-13T11:23:32.776835+0000","last_active":"2024-09-13T11:24:06.133586+0000","last_peered":"2024-09-13T11:24:06.133586+0000","last_clean":"2024-09-13T11:24:06.133586+0000","last_became_active":"2024-09-13T11:23:31.939246+0000","last_became_peered":"2024-09-13T11:23:31.939246+0000","last_unstale":"2024-09-13T11:24:06.133586+0000","last_undegraded":"2024-09-13T11:24:06.133586+0000","last_fullsized":"2024-09-13T11:24:06.133586+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'3395","last_scrub_stamp":"2024-09-13T11:23:32.776747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:32.776747+0000","objects_scrubbed":1704,"log_size":4363,"log_dups_size":0,"ondisk_log_size":4363,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:24:56.402862+0000","scrub_duration":179,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":175112192,"num_objects":2672,"num_object_clones":0,"num_object_copies":5344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2672,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":968,"num_write_kb":61952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.e","version":"108'4435","reported_seq":9563,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T11:24:06.003331+0000","last_change":"2024-09-13T11:23:32.368856+0000","last_active":"2024-09-13T11:24:06.003331+0000","last_peered":"2024-09-13T11:24:06.003331+0000","last_clean":"2024-09-13T11:24:06.003331+0000","last_became_active":"2024-09-13T11:23:31.925869+0000","last_became_peered":"2024-09-13T11:23:31.925869+0000","last_unstale":"2024-09-13T11:24:06.003331+0000","last_undegraded":"2024-09-13T11:24:06.003331+0000","last_fullsized":"2024-09-13T11:24:06.003331+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'3527","last_scrub_stamp":"2024-09-13T11:23:32.368727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:32.368727+0000","objects_scrubbed":1774,"log_size":4435,"log_dups_size":0,"ondisk_log_size":4435,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:24:57.811779+0000","scrub_duration":219,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":175833088,"num_objects":2683,"num_object_clones":0,"num_object_copies":5366,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2683,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":909,"num_write_kb":58176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"109'4353","reported_seq":4555,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T11:24:06.962608+0000","last_change":"2024-09-13T11:23:33.492630+0000","last_active":"2024-09-13T11:24:06.962608+0000","last_peered":"2024-09-13T11:24:06.962608+0000","last_clean":"2024-09-13T11:24:06.962608+0000","last_became_active":"2024-09-13T11:23:31.935239+0000","last_became_peered":"2024-09-13T11:23:31.935239+0000","last_unstale":"2024-09-13T11:24:06.962608+0000","last_undegraded":"2024-09-13T11:24:06.962608+0000","last_fullsized":"2024-09-13T11:24:06.962608+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'3420","last_scrub_stamp":"2024-09-13T11:23:33.492488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:33.492488+0000","objects_scrubbed":1776,"log_size":4353,"log_dups_size":0,"ondisk_log_size":4353,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:24:44.851261+0000","scrub_duration":215,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":177537024,"num_objects":2709,"num_object_clones":0,"num_object_copies":5418,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2709,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":933,"num_write_kb":59712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,12],"acting":[11,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.10","version":"108'4131","reported_seq":8487,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T11:24:06.004696+0000","last_change":"2024-09-13T11:23:48.519654+0000","last_active":"2024-09-13T11:24:06.004696+0000","last_peered":"2024-09-13T11:24:06.004696+0000","last_clean":"2024-09-13T11:24:06.004696+0000","last_became_active":"2024-09-13T11:23:41.931183+0000","last_became_peered":"2024-09-13T11:23:41.931183+0000","last_unstale":"2024-09-13T11:24:06.004696+0000","last_undegraded":"2024-09-13T11:24:06.004696+0000","last_fullsized":"2024-09-13T11:24:06.004696+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":5,"last_scrub":"91'3468","last_scrub_stamp":"2024-09-13T11:23:48.519574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:48.519574+0000","objects_scrubbed":2031,"log_size":4131,"log_dups_size":0,"ondisk_log_size":4131,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:24:57.342213+0000","scrub_duration":215,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":176685056,"num_objects":2696,"num_object_clones":0,"num_object_copies":5392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2696,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":665,"num_write_kb":42560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.11","version":"108'4005","reported_seq":7139,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T11:24:06.005767+0000","last_change":"2024-09-13T11:23:46.102466+0000","last_active":"2024-09-13T11:24:06.005767+0000","last_peered":"2024-09-13T11:24:06.005767+0000","last_clean":"2024-09-13T11:24:06.005767+0000","last_became_active":"2024-09-13T11:23:39.063172+0000","last_became_peered":"2024-09-13T11:23:39.063172+0000","last_unstale":"2024-09-13T11:24:06.005767+0000","last_undegraded":"2024-09-13T11:24:06.005767+0000","last_fullsized":"2024-09-13T11:24:06.005767+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":5,"last_scrub":"88'3345","last_scrub_stamp":"2024-09-13T11:23:46.101973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:46.101973+0000","objects_scrubbed":2011,"log_size":4005,"log_dups_size":0,"ondisk_log_size":4005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:08.671350+0000","scrub_duration":955,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":175308800,"num_objects":2675,"num_object_clones":0,"num_object_copies":5350,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2675,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":664,"num_write_kb":42496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11],"acting":[4,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.12","version":"108'4004","reported_seq":7062,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T11:24:06.047245+0000","last_change":"2024-09-13T11:23:35.111409+0000","last_active":"2024-09-13T11:24:06.047245+0000","last_peered":"2024-09-13T11:24:06.047245+0000","last_clean":"2024-09-13T11:24:06.047245+0000","last_became_active":"2024-09-13T11:23:21.998877+0000","last_became_peered":"2024-09-13T11:23:21.998877+0000","last_unstale":"2024-09-13T11:24:06.047245+0000","last_undegraded":"2024-09-13T11:24:06.047245+0000","last_fullsized":"2024-09-13T11:24:06.047245+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'3109","last_scrub_stamp":"2024-09-13T11:23:35.111320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:35.111320+0000","objects_scrubbed":1732,"log_size":4004,"log_dups_size":0,"ondisk_log_size":4004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:03.371433+0000","scrub_duration":239,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":172163072,"num_objects":2627,"num_object_clones":0,"num_object_copies":5254,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2627,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":895,"num_write_kb":57280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,7],"acting":[14,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"3.13","version":"109'4069","reported_seq":7130,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T11:24:07.048487+0000","last_change":"2024-09-13T11:23:47.565094+0000","last_active":"2024-09-13T11:24:07.048487+0000","last_peered":"2024-09-13T11:24:07.048487+0000","last_clean":"2024-09-13T11:24:07.048487+0000","last_became_active":"2024-09-13T11:23:41.082433+0000","last_became_peered":"2024-09-13T11:23:41.082433+0000","last_unstale":"2024-09-13T11:24:07.048487+0000","last_undegraded":"2024-09-13T11:24:07.048487+0000","last_fullsized":"2024-09-13T11:24:07.048487+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'3408","last_scrub_stamp":"2024-09-13T11:23:47.564931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:47.564931+0000","objects_scrubbed":1981,"log_size":4069,"log_dups_size":0,"ondisk_log_size":4069,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:04.258231+0000","scrub_duration":471,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":173932544,"num_objects":2654,"num_object_clones":0,"num_object_copies":5308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2654,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":673,"num_write_kb":43072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7],"acting":[11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":36,"stat_sum":{"num_bytes":5616500736,"num_objects":85702,"num_object_clones":0,"num_object_copies":171404,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":85702,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":39173,"num_write_kb":2507008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11626,"num_bytes_recovered":761888768,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":13182435328,"data_stored":13182435328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":146148,"ondisk_log_size":146148,"up":72,"acting":72,"num_store_stats":16},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":10},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":112,"ondisk_log_size":112,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":13,"up_from":23,"seq":98784247835,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1008836,"kb_used_data":933236,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":93363004,"statfs":{"total":96636764160,"available":95603716096,"internally_reserved":0,"allocated":955633664,"data_stored":955156526,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,1,3,4,6,8,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":9,"up_from":23,"seq":98784247834,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":450936,"kb_used_data":415592,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93920904,"statfs":{"total":96636764160,"available":96175005696,"internally_reserved":0,"allocated":425566208,"data_stored":425100675,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,8,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":40,"seq":171798691858,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1363564,"kb_used_data":1299612,"kb_used_omap":4,"kb_used_meta":63931,"kb_avail":93008276,"statfs":{"total":96636764160,"available":95240474624,"internally_reserved":0,"allocated":1330802688,"data_stored":1330300776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4433,"internal_metadata":65466031},"hb_peers":[0,2,3,6,8,9,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":21,"seq":90194313243,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1077112,"kb_used_data":1001512,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":93294728,"statfs":{"total":96636764160,"available":95533801472,"internally_reserved":0,"allocated":1025548288,"data_stored":1025039643,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":77396423},"hb_peers":[0,2,4,7,9,10,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,1,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":21,"seq":90194313245,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":286288,"kb_used_data":259264,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94085552,"statfs":{"total":96636764160,"available":96343605248,"internally_reserved":0,"allocated":265486336,"data_stored":264993109,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,6,8,9,10,11,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":22,"seq":94489280537,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1149000,"kb_used_data":1085880,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":93222840,"statfs":{"total":96636764160,"available":95460188160,"internally_reserved":0,"allocated":1111941120,"data_stored":1111516713,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[1,3,4,5,8,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":22,"seq":94489280538,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":445056,"kb_used_data":405552,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":93926784,"statfs":{"total":96636764160,"available":96181026816,"internally_reserved":0,"allocated":415285248,"data_stored":414812191,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,3,7,8,9,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":15,"up_from":22,"seq":94489280538,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":949184,"kb_used_data":873584,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":93422656,"statfs":{"total":96636764160,"available":95664799744,"internally_reserved":0,"allocated":894550016,"data_stored":894133583,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":77396422},"hb_peers":[0,3,4,5,6,7,10,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":22,"seq":94489280538,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":546264,"kb_used_data":497032,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":93825576,"statfs":{"total":96636764160,"available":96077389824,"internally_reserved":0,"allocated":508960768,"data_stored":508465062,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[3,5,7,9,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":22,"seq":94489280538,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":429992,"kb_used_data":394648,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93941848,"statfs":{"total":96636764160,"available":96196452352,"internally_reserved":0,"allocated":404119552,"data_stored":403666123,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,3,5,7,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":22,"seq":94489280538,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1232220,"kb_used_data":1164940,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":93139620,"statfs":{"total":96636764160,"available":95374970880,"internally_reserved":0,"allocated":1192898560,"data_stored":1192454834,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":68876746},"hb_peers":[0,3,5,6,8,9,10,11,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":22,"seq":94489280538,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":489692,"kb_used_data":458508,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93882148,"statfs":{"total":96636764160,"available":96135319552,"internally_reserved":0,"allocated":469512192,"data_stored":469008420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,5,7,9,10,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":22,"seq":94489280537,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":699360,"kb_used_data":641808,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":93672480,"statfs":{"total":96636764160,"available":95920619520,"internally_reserved":0,"allocated":657211392,"data_stored":656765555,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":58915264},"hb_peers":[0,1,3,6,7,9,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":11,"up_from":22,"seq":94489280539,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1361604,"kb_used_data":1281844,"kb_used_omap":1,"kb_used_meta":79742,"kb_avail":93010236,"statfs":{"total":96636764160,"available":95242481664,"internally_reserved":0,"allocated":1312608256,"data_stored":1312134741,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":81656260},"hb_peers":[0,2,4,7,8,10,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":12,"up_from":22,"seq":94489280538,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1116988,"kb_used_data":1053868,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":93254852,"statfs":{"total":96636764160,"available":95492968448,"internally_reserved":0,"allocated":1079160832,"data_stored":1078688509,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[0,1,2,3,5,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":14,"up_from":22,"seq":94489280538,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1184908,"kb_used_data":1113468,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":93186932,"statfs":{"total":96636764160,"available":95423418368,"internally_reserved":0,"allocated":1140191232,"data_stored":1139708545,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73136581},"hb_peers":[0,1,5,6,7,8,9,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1123614720,"data_stored":1123614720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1328545792,"data_stored":1328545792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":419233792,"data_stored":419233792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1024000000,"data_stored":1024000000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":514981888,"data_stored":514981888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":264568832,"data_stored":264568832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":405602304,"data_stored":405602304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1192886272,"data_stored":1192886272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":466944000,"data_stored":466944000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":425328640,"data_stored":425328640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":663617536,"data_stored":663617536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":1274019840,"data_stored":1274019840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":1081606144,"data_stored":1081606144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":958332928,"data_stored":958332928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":1139933184,"data_stored":1139933184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":899219456,"data_stored":899219456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T11:24:11.096 INFO:tasks.ceph.osd.12.smithi001.stderr:2024-09-13T11:24:11.091+0000 7f892c6e6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26314) UID: 0 2024-09-13T11:24:11.107 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-13T11:24:11.107 INFO:teuthology.orchestra.run.smithi001.stdout: "ops": [], 2024-09-13T11:24:11.107 INFO:teuthology.orchestra.run.smithi001.stdout: "num_ops": 0 2024-09-13T11:24:11.107 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-13T11:24:11.122 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_blocked_ops 2024-09-13T11:24:11.160 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T11:24:11.174 INFO:tasks.ceph.osd.15.smithi177.stderr:2024-09-13T11:24:11.171+0000 7f9d6ce77640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26578) UID: 0 2024-09-13T11:24:11.275 INFO:tasks.ceph.osd.8.smithi001.stderr:2024-09-13T11:24:11.267+0000 7fb9b48ae640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26312) UID: 0 2024-09-13T11:24:11.289 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-13T11:24:11.290 INFO:teuthology.orchestra.run.smithi001.stdout: "ops": [], 2024-09-13T11:24:11.290 INFO:teuthology.orchestra.run.smithi001.stdout: "complaint_time": 30, 2024-09-13T11:24:11.290 INFO:teuthology.orchestra.run.smithi001.stdout: "num_blocked_ops": 0 2024-09-13T11:24:11.290 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-13T11:24:11.308 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_historic_ops 2024-09-13T11:24:11.323 INFO:teuthology.orchestra.run.smithi001.stdout:osd.0: {} 2024-09-13T11:24:11.323 INFO:teuthology.orchestra.run.smithi001.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T11:24:11.352 INFO:teuthology.orchestra.run.smithi001.stdout:osd.1: {} 2024-09-13T11:24:11.352 INFO:teuthology.orchestra.run.smithi001.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T11:24:11.376 INFO:tasks.ceph.osd.8.smithi001.stderr:2024-09-13T11:24:11.371+0000 7fb9b48ae640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26312) UID: 0 2024-09-13T11:24:11.380 INFO:teuthology.orchestra.run.smithi001.stdout:osd.2: {} 2024-09-13T11:24:11.380 INFO:teuthology.orchestra.run.smithi001.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T11:24:11.407 INFO:teuthology.orchestra.run.smithi001.stdout:osd.3: {} 2024-09-13T11:24:11.407 INFO:teuthology.orchestra.run.smithi001.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T11:24:11.433 INFO:teuthology.orchestra.run.smithi001.stdout:osd.4: {} 2024-09-13T11:24:11.433 INFO:teuthology.orchestra.run.smithi001.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T11:24:11.458 INFO:teuthology.orchestra.run.smithi001.stdout:osd.5: {} 2024-09-13T11:24:11.459 INFO:teuthology.orchestra.run.smithi001.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T11:24:11.462 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-13T11:24:11.463 INFO:teuthology.orchestra.run.smithi001.stdout: "size": 20, 2024-09-13T11:24:11.463 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 600, 2024-09-13T11:24:11.463 INFO:teuthology.orchestra.run.smithi001.stdout: "ops": [ 2024-09-13T11:24:11.463 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:11.463 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop_reply(client.4665.0:167 3.9 e41/30 ondisk, result = 0)", 2024-09-13T11:24:11.463 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:22:47.419696+0000", 2024-09-13T11:24:11.463 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 84.041284289999993, 2024-09-13T11:24:11.463 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.64615778199999996, 2024-09-13T11:24:11.463 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:24:11.463 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:24:11.463 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:24:11.464 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:24:11.464 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:11.464 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:24:11.464 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:47.419696+0000", 2024-09-13T11:24:11.464 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:11.464 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:11.464 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:11.464 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:24:11.464 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:47.419696+0000", 2024-09-13T11:24:11.464 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:11.464 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:11.464 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:11.465 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:24:11.465 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:47.419697+0000", 2024-09-13T11:24:11.465 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.15e-06 2024-09-13T11:24:11.465 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:11.465 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:11.465 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:24:11.465 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:47.419700+0000", 2024-09-13T11:24:11.468 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.2449999999999998e-06 2024-09-13T11:24:11.468 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:11.468 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:11.468 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:24:11.468 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:48.065637+0000", 2024-09-13T11:24:11.469 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.64593674300000004 2024-09-13T11:24:11.469 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:11.469 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:11.469 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:24:11.469 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:48.065648+0000", 2024-09-13T11:24:11.469 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0865e-05 2024-09-13T11:24:11.469 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:11.469 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:11.469 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:24:11.469 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:48.065720+0000", 2024-09-13T11:24:11.469 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.2020999999999994e-05 2024-09-13T11:24:11.469 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:11.469 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:11.469 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:24:11.470 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:48.065728+0000", 2024-09-13T11:24:11.470 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.7660000000000004e-06 2024-09-13T11:24:11.470 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:11.470 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:11.470 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:24:11.470 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:48.065853+0000", 2024-09-13T11:24:11.470 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00012499200000000001 2024-09-13T11:24:11.470 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:11.470 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:24:11.470 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:11.470 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:11.470 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:11.470 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:25383 3.0 3:0a6bb869:::benchmark_data_smithi177_31116_object25382:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e55)", 2024-09-13T11:24:11.470 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:04.447376+0000", 2024-09-13T11:24:11.471 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 67.013603621000001, 2024-09-13T11:24:11.471 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.21239864, 2024-09-13T11:24:11.471 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:24:11.471 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:24:11.471 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:11.471 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:24:11.471 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:24:11.471 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:11.471 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 25383 2024-09-13T11:24:11.471 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:11.471 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:24:11.471 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:11.471 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:24:11.471 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.447376+0000", 2024-09-13T11:24:11.472 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:11.472 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:11.472 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:11.472 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:24:11.472 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.447376+0000", 2024-09-13T11:24:11.472 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:11.472 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:11.472 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:11.472 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:24:11.472 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.447378+0000", 2024-09-13T11:24:11.472 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.6729999999999999e-06 2024-09-13T11:24:11.472 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:11.472 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:11.473 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:24:11.473 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.447433+0000", 2024-09-13T11:24:11.473 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.5371000000000002e-05 2024-09-13T11:24:11.473 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:11.473 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:11.473 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:24:11.473 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.447446+0000", 2024-09-13T11:24:11.473 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.3179999999999999e-05 2024-09-13T11:24:11.473 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:11.473 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:11.473 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:24:11.473 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.447456+0000", 2024-09-13T11:24:11.473 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0008e-05 2024-09-13T11:24:11.473 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:11.474 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:11.474 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:24:11.474 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.447514+0000", 2024-09-13T11:24:11.474 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.8023999999999999e-05 2024-09-13T11:24:11.474 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:11.474 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:11.474 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:24:11.474 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.447642+0000", 2024-09-13T11:24:11.474 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000127296 2024-09-13T11:24:11.474 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:11.474 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:11.474 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 15", 2024-09-13T11:24:11.475 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.447767+0000", 2024-09-13T11:24:11.475 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000125547 2024-09-13T11:24:11.475 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:11.475 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:11.475 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:24:11.475 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.449188+0000", 2024-09-13T11:24:11.475 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.001420604 2024-09-13T11:24:11.475 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:11.475 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:11.475 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:11.475 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.659668+0000", 2024-09-13T11:24:11.475 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.210479685 2024-09-13T11:24:11.475 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:11.475 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:11.476 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:24:11.476 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.659716+0000", 2024-09-13T11:24:11.476 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.8609999999999997e-05 2024-09-13T11:24:11.476 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:11.476 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:11.476 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:24:11.476 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.659775+0000", 2024-09-13T11:24:11.476 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.8641999999999999e-05 2024-09-13T11:24:11.476 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:11.476 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:24:11.477 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:11.477 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:11.477 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:11.477 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4665.0:25408 3.4 e56/40 3:294e23bd:::benchmark_data_smithi177_31116_object25407:head v 56'1645, mlcod=55'1642)", 2024-09-13T11:24:11.477 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:04.494686+0000", 2024-09-13T11:24:11.477 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 66.966293961999995, 2024-09-13T11:24:11.477 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.29725566399999998, 2024-09-13T11:24:11.477 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:24:11.477 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:24:11.477 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:11.477 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:24:11.477 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:11.477 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:24:11.478 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.494686+0000", 2024-09-13T11:24:11.478 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:11.478 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:11.478 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:11.478 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:24:11.478 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.494686+0000", 2024-09-13T11:24:11.478 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:11.478 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:11.478 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:11.478 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:24:11.478 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.494686+0000", 2024-09-13T11:24:11.478 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.2399999999999999e-07 2024-09-13T11:24:11.478 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:11.479 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:11.479 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:24:11.479 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.494809+0000", 2024-09-13T11:24:11.479 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00012308600000000001 2024-09-13T11:24:11.479 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:11.479 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:11.479 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:24:11.479 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.789410+0000", 2024-09-13T11:24:11.479 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.29460066899999998 2024-09-13T11:24:11.479 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:11.479 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:11.479 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:24:11.479 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.789425+0000", 2024-09-13T11:24:11.479 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.4600000000000001e-05 2024-09-13T11:24:11.480 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:11.480 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:11.480 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:24:11.480 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.789507+0000", 2024-09-13T11:24:11.480 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.2785000000000001e-05 2024-09-13T11:24:11.480 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:11.480 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:11.480 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:24:11.480 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.789525+0000", 2024-09-13T11:24:11.480 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.8117999999999998e-05 2024-09-13T11:24:11.480 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:11.481 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:11.481 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:24:11.481 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.791852+0000", 2024-09-13T11:24:11.481 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0023264459999999998 2024-09-13T11:24:11.481 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:11.481 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:11.481 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:24:11.481 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.791942+0000", 2024-09-13T11:24:11.481 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.9635999999999994e-05 2024-09-13T11:24:11.482 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:11.482 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:24:11.482 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:11.482 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:11.482 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:11.482 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4665.0:30815 3.f e59/30 3:fde17cb1:::benchmark_data_smithi177_31116_object30814:head v 59'1919, mlcod=59'1914)", 2024-09-13T11:24:11.482 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:09.494342+0000", 2024-09-13T11:24:11.482 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 61.966637976000001, 2024-09-13T11:24:11.482 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.22787855700000001, 2024-09-13T11:24:11.482 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:24:11.482 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:24:11.482 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:11.483 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:24:11.483 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:11.483 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:24:11.483 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.494342+0000", 2024-09-13T11:24:11.483 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:11.483 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:11.483 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:11.483 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:24:11.483 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.494342+0000", 2024-09-13T11:24:11.483 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:11.483 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:11.483 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:11.483 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:24:11.483 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.494343+0000", 2024-09-13T11:24:11.484 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.251e-06 2024-09-13T11:24:11.484 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:11.484 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:11.484 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:24:11.484 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.494387+0000", 2024-09-13T11:24:11.484 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.4276000000000001e-05 2024-09-13T11:24:11.484 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:11.484 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:11.484 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:24:11.484 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.720454+0000", 2024-09-13T11:24:11.484 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.226066614 2024-09-13T11:24:11.484 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:11.484 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:11.485 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:24:11.485 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.720462+0000", 2024-09-13T11:24:11.485 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.9240000000000007e-06 2024-09-13T11:24:11.485 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:11.485 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:11.485 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:24:11.485 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.720537+0000", 2024-09-13T11:24:11.485 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.4967000000000002e-05 2024-09-13T11:24:11.485 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:11.485 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:11.485 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:24:11.485 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.720563+0000", 2024-09-13T11:24:11.485 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.5630999999999999e-05 2024-09-13T11:24:11.485 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:11.486 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:11.486 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:24:11.486 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.722183+0000", 2024-09-13T11:24:11.486 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0016209670000000001 2024-09-13T11:24:11.486 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:11.486 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:11.486 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:24:11.486 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.722220+0000", 2024-09-13T11:24:11.486 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.6927000000000001e-05 2024-09-13T11:24:11.486 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:11.486 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:24:11.486 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:11.487 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:11.487 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:11.487 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4665.0:30828 3.f e59/30 3:f3a9c6a9:::benchmark_data_smithi177_31116_object30827:head v 59'1920, mlcod=59'1916)", 2024-09-13T11:24:11.487 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:09.502607+0000", 2024-09-13T11:24:11.487 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 61.958372918999999, 2024-09-13T11:24:11.487 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.26374408300000002, 2024-09-13T11:24:11.487 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:24:11.487 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:24:11.487 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:11.487 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:24:11.487 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:11.487 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:24:11.487 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.502607+0000", 2024-09-13T11:24:11.488 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:11.488 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:11.488 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:11.488 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:24:11.488 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.502607+0000", 2024-09-13T11:24:11.488 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:11.488 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:11.488 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:11.488 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:24:11.488 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.502609+0000", 2024-09-13T11:24:11.488 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.84e-06 2024-09-13T11:24:11.488 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:11.488 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:11.489 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:24:11.489 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.502663+0000", 2024-09-13T11:24:11.489 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.3971000000000002e-05 2024-09-13T11:24:11.489 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:11.489 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:11.489 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:24:11.489 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.763912+0000", 2024-09-13T11:24:11.489 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.26124904300000001 2024-09-13T11:24:11.489 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:11.489 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:11.489 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:24:11.489 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.763920+0000", 2024-09-13T11:24:11.489 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.1149999999999994e-06 2024-09-13T11:24:11.489 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:11.490 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:11.490 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:24:11.490 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.764024+0000", 2024-09-13T11:24:11.490 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00010380899999999999 2024-09-13T11:24:11.490 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:11.490 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:11.490 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:24:11.490 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.764068+0000", 2024-09-13T11:24:11.490 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.4638999999999997e-05 2024-09-13T11:24:11.490 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:11.490 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:11.490 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:24:11.490 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.766308+0000", 2024-09-13T11:24:11.490 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0022392039999999998 2024-09-13T11:24:11.491 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:11.491 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:11.491 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:24:11.491 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.766351+0000", 2024-09-13T11:24:11.491 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.3461999999999998e-05 2024-09-13T11:24:11.491 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:11.491 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:24:11.491 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:11.491 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:11.491 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:11.491 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:39894 3.9 3:9190c04c:::benchmark_data_smithi177_31116_object39893:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-13T11:24:11.491 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:15.746095+0000", 2024-09-13T11:24:11.491 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 55.714885293999998, 2024-09-13T11:24:11.492 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.129842069, 2024-09-13T11:24:11.492 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:24:11.492 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:24:11.492 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:11.492 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:24:11.492 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:24:11.492 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:11.492 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 39894 2024-09-13T11:24:11.492 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:11.492 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:24:11.492 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:11.492 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:24:11.492 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:15.746095+0000", 2024-09-13T11:24:11.492 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:11.493 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:11.493 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:11.493 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:24:11.493 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:15.746095+0000", 2024-09-13T11:24:11.493 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:11.493 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:11.493 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:11.493 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:24:11.493 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:15.746096+0000", 2024-09-13T11:24:11.493 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.9800000000000002e-07 2024-09-13T11:24:11.493 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:11.493 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:11.493 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:24:11.494 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:15.746154+0000", 2024-09-13T11:24:11.494 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.8734999999999999e-05 2024-09-13T11:24:11.494 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:11.494 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:11.494 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:24:11.494 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:15.746163+0000", 2024-09-13T11:24:11.494 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.6780000000000006e-06 2024-09-13T11:24:11.494 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:11.494 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:11.494 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:24:11.494 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:15.746168+0000", 2024-09-13T11:24:11.494 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.622e-06 2024-09-13T11:24:11.494 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:11.495 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:11.495 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:24:11.495 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:15.747109+0000", 2024-09-13T11:24:11.495 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00094147399999999998 2024-09-13T11:24:11.495 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:11.495 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:11.495 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:24:11.495 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:15.747196+0000", 2024-09-13T11:24:11.495 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.6537000000000002e-05 2024-09-13T11:24:11.495 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:11.495 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:11.495 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 2", 2024-09-13T11:24:11.495 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:15.747304+0000", 2024-09-13T11:24:11.496 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000108208 2024-09-13T11:24:11.496 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:11.496 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:11.496 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:24:11.496 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:15.748640+0000", 2024-09-13T11:24:11.496 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0013365339999999999 2024-09-13T11:24:11.496 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:11.496 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:11.496 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:11.496 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:15.875866+0000", 2024-09-13T11:24:11.496 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.12722560399999999 2024-09-13T11:24:11.496 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:11.496 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:11.496 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:24:11.497 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:15.875910+0000", 2024-09-13T11:24:11.497 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.4409000000000001e-05 2024-09-13T11:24:11.497 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:11.497 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:11.497 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:24:11.497 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:15.875937+0000", 2024-09-13T11:24:11.497 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.6270000000000001e-05 2024-09-13T11:24:11.497 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:11.497 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:24:11.497 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:11.497 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:11.497 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:11.497 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:54351 3.10 3:085f283e:::benchmark_data_smithi177_31116_object54350:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e74)", 2024-09-13T11:24:11.498 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:30.599125+0000", 2024-09-13T11:24:11.498 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 40.861855214000002, 2024-09-13T11:24:11.498 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.167573261, 2024-09-13T11:24:11.498 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:24:11.498 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:24:11.498 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:11.498 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:24:11.498 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:24:11.498 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:11.498 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 54351 2024-09-13T11:24:11.498 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:11.498 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:24:11.498 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:11.498 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:24:11.499 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:30.599125+0000", 2024-09-13T11:24:11.499 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:11.499 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:11.499 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:11.499 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:24:11.499 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:30.599125+0000", 2024-09-13T11:24:11.499 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:11.499 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:11.499 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:11.499 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:24:11.499 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:30.599127+0000", 2024-09-13T11:24:11.499 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.847e-06 2024-09-13T11:24:11.499 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:11.499 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:11.499 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:24:11.500 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:30.599164+0000", 2024-09-13T11:24:11.500 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.6758999999999998e-05 2024-09-13T11:24:11.500 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:11.500 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:11.500 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:24:11.500 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:30.599184+0000", 2024-09-13T11:24:11.500 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.9928e-05 2024-09-13T11:24:11.500 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:11.500 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:11.500 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:24:11.500 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:30.599194+0000", 2024-09-13T11:24:11.500 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0035e-05 2024-09-13T11:24:11.500 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:11.501 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:11.501 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:24:11.501 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:30.599259+0000", 2024-09-13T11:24:11.501 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.5147000000000005e-05 2024-09-13T11:24:11.501 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:11.501 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:11.501 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:24:11.501 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:30.599381+0000", 2024-09-13T11:24:11.501 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000121876 2024-09-13T11:24:11.501 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:11.501 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:11.501 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 0,6", 2024-09-13T11:24:11.501 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:30.599496+0000", 2024-09-13T11:24:11.501 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000114307 2024-09-13T11:24:11.502 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:11.502 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:11.502 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:24:11.502 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:30.601094+0000", 2024-09-13T11:24:11.502 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.001598857 2024-09-13T11:24:11.502 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:11.502 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:11.502 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:11.502 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:30.601896+0000", 2024-09-13T11:24:11.502 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00080183800000000003 2024-09-13T11:24:11.502 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:11.502 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:11.503 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:11.503 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:30.766040+0000", 2024-09-13T11:24:11.503 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.16414414099999999 2024-09-13T11:24:11.503 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:11.503 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:11.503 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:24:11.503 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:30.766092+0000", 2024-09-13T11:24:11.503 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.1230000000000002e-05 2024-09-13T11:24:11.503 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:11.504 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:11.504 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:24:11.504 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:30.766698+0000", 2024-09-13T11:24:11.504 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00060629599999999998 2024-09-13T11:24:11.504 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:11.504 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:24:11.504 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:11.504 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:11.504 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:11.504 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4665.0:54570 3.f e74/30 3:ff206a5c:::benchmark_data_smithi177_31116_object54569:head v 74'3411, mlcod=74'3408)", 2024-09-13T11:24:11.504 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:30.829251+0000", 2024-09-13T11:24:11.504 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 40.631728484999996, 2024-09-13T11:24:11.504 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.12680533199999999, 2024-09-13T11:24:11.505 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:24:11.505 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:24:11.505 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:11.505 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:24:11.505 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:11.505 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:24:11.505 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:30.829251+0000", 2024-09-13T11:24:11.505 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:11.505 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:11.505 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:11.505 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:24:11.505 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:30.829251+0000", 2024-09-13T11:24:11.505 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:11.505 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:11.506 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:11.506 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:24:11.506 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:30.829253+0000", 2024-09-13T11:24:11.506 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.5629999999999999e-06 2024-09-13T11:24:11.506 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:11.506 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:11.506 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:24:11.506 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:30.829313+0000", 2024-09-13T11:24:11.506 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.9864000000000001e-05 2024-09-13T11:24:11.506 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:11.506 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:11.506 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:24:11.506 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:30.952404+0000", 2024-09-13T11:24:11.506 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.123091229 2024-09-13T11:24:11.507 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:11.507 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:11.507 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:24:11.507 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:30.952440+0000", 2024-09-13T11:24:11.507 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.6161999999999997e-05 2024-09-13T11:24:11.507 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:11.507 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:11.507 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:24:11.507 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:30.952544+0000", 2024-09-13T11:24:11.507 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000103742 2024-09-13T11:24:11.507 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:11.507 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:11.507 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:24:11.507 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:30.952566+0000", 2024-09-13T11:24:11.508 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.2481e-05 2024-09-13T11:24:11.508 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:11.508 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:11.508 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:24:11.508 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:30.955979+0000", 2024-09-13T11:24:11.508 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0034129999999999998 2024-09-13T11:24:11.508 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:11.508 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:11.508 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:24:11.508 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:30.956057+0000", 2024-09-13T11:24:11.508 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.7291000000000002e-05 2024-09-13T11:24:11.508 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:11.509 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:24:11.509 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:11.509 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:11.509 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:11.509 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDPGPush(3.1b 84/78 [PushOp(3:dcddd139:::benchmark_data_smithi177_31116_object10384:head, version: 48'673, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:dcddd139:::benchmark_data_smithi177_31116_object10384:head@48'673, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:24:11.509 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:41.400602+0000", 2024-09-13T11:24:11.509 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 30.060377433999999, 2024-09-13T11:24:11.509 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.43365821999999998, 2024-09-13T11:24:11.509 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:24:11.509 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:24:11.509 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:24:11.509 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:24:11.509 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:11.510 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:24:11.510 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:41.400602+0000", 2024-09-13T11:24:11.510 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:11.510 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:11.510 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:11.510 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:24:11.510 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:41.400602+0000", 2024-09-13T11:24:11.510 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:11.510 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:11.510 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:11.510 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:24:11.510 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:41.400604+0000", 2024-09-13T11:24:11.510 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.99e-06 2024-09-13T11:24:11.511 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:11.511 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:11.511 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:24:11.511 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:41.400653+0000", 2024-09-13T11:24:11.511 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.8195000000000002e-05 2024-09-13T11:24:11.511 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:11.511 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:11.511 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:24:11.511 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:41.833180+0000", 2024-09-13T11:24:11.511 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.43252769200000002 2024-09-13T11:24:11.511 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:11.511 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:11.512 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:24:11.512 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:41.833213+0000", 2024-09-13T11:24:11.512 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.2737999999999997e-05 2024-09-13T11:24:11.512 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:11.512 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:11.512 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:24:11.512 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:41.833408+0000", 2024-09-13T11:24:11.512 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00019473799999999999 2024-09-13T11:24:11.512 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:11.512 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:11.512 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:24:11.512 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:41.833417+0000", 2024-09-13T11:24:11.512 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.5349999999999993e-06 2024-09-13T11:24:11.512 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:11.513 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:11.513 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:24:11.513 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:41.834261+0000", 2024-09-13T11:24:11.513 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00084333200000000002 2024-09-13T11:24:11.513 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:11.513 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:24:11.513 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:11.513 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:11.513 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:11.513 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDPGPushReply(3.20 85/77 [PushReplyOp(3:066982a6:::benchmark_data_smithi177_31116_object16400:head)])", 2024-09-13T11:24:11.513 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:42.671551+0000", 2024-09-13T11:24:11.513 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 28.789428610999998, 2024-09-13T11:24:11.513 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.28249085499999999, 2024-09-13T11:24:11.514 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:24:11.514 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:24:11.514 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-13T11:24:11.514 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:24:11.514 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:11.514 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:24:11.514 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.671551+0000", 2024-09-13T11:24:11.514 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:11.514 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:11.514 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:11.514 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:24:11.514 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.671551+0000", 2024-09-13T11:24:11.514 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:11.514 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:11.514 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:11.515 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:24:11.515 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.671552+0000", 2024-09-13T11:24:11.515 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.097e-06 2024-09-13T11:24:11.515 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:11.515 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:11.515 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:24:11.515 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.671556+0000", 2024-09-13T11:24:11.515 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.8519999999999997e-06 2024-09-13T11:24:11.515 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:11.515 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:11.515 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:24:11.515 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.951769+0000", 2024-09-13T11:24:11.515 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.28021293000000003 2024-09-13T11:24:11.515 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:11.515 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:11.516 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:24:11.516 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.951780+0000", 2024-09-13T11:24:11.516 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.1059e-05 2024-09-13T11:24:11.516 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:11.516 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:11.516 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:24:11.516 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.951992+0000", 2024-09-13T11:24:11.516 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000211478 2024-09-13T11:24:11.516 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:11.516 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:11.516 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:24:11.516 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.954042+0000", 2024-09-13T11:24:11.516 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0020504389999999998 2024-09-13T11:24:11.517 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:11.517 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:24:11.517 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:11.517 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:11.517 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:11.517 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDPGPush(3.1b 89/78 [PushOp(3:dc4c79e7:::benchmark_data_smithi177_31116_object19857:head, version: 54'1266, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:dc4c79e7:::benchmark_data_smithi177_31116_object19857:head@54'1266, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:24:11.517 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:46.649128+0000", 2024-09-13T11:24:11.517 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 24.811851503, 2024-09-13T11:24:11.517 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.29963603, 2024-09-13T11:24:11.517 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:24:11.517 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:24:11.517 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:24:11.517 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:24:11.518 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:11.518 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:24:11.518 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.649128+0000", 2024-09-13T11:24:11.518 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:11.518 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:11.518 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:11.518 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:24:11.518 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.649128+0000", 2024-09-13T11:24:11.518 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:11.518 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:11.518 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:11.518 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:24:11.518 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.649130+0000", 2024-09-13T11:24:11.518 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2950000000000001e-06 2024-09-13T11:24:11.519 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:11.519 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:11.519 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:24:11.519 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.649197+0000", 2024-09-13T11:24:11.519 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.7033999999999995e-05 2024-09-13T11:24:11.519 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:11.519 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:11.519 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:24:11.519 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.947691+0000", 2024-09-13T11:24:11.519 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.29849387300000002 2024-09-13T11:24:11.519 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:11.519 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:11.519 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:24:11.519 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.947702+0000", 2024-09-13T11:24:11.519 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.1239e-05 2024-09-13T11:24:11.520 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:11.520 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:11.520 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:24:11.520 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.947799+0000", 2024-09-13T11:24:11.520 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.7283999999999999e-05 2024-09-13T11:24:11.520 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:11.520 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:11.520 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:24:11.520 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.947809+0000", 2024-09-13T11:24:11.520 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.008e-05 2024-09-13T11:24:11.520 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:11.520 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:11.520 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:24:11.520 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.948764+0000", 2024-09-13T11:24:11.521 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00095522499999999998 2024-09-13T11:24:11.521 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:11.521 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:24:11.521 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:11.521 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:11.521 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:11.521 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:64407 3.20 3:0477cf3a:::benchmark_data_smithi177_31116_object64406:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90)", 2024-09-13T11:24:11.521 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:47.219673+0000", 2024-09-13T11:24:11.521 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 24.241306674, 2024-09-13T11:24:11.521 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.126665629, 2024-09-13T11:24:11.521 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:24:11.521 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:24:11.521 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:11.522 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:24:11.522 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:24:11.522 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:11.522 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 64407 2024-09-13T11:24:11.522 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:11.522 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:24:11.522 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:11.522 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:24:11.522 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:47.219673+0000", 2024-09-13T11:24:11.522 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:11.522 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:11.522 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:11.522 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:24:11.522 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:47.219673+0000", 2024-09-13T11:24:11.523 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:11.523 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:11.523 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:11.523 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:24:11.523 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:47.219675+0000", 2024-09-13T11:24:11.523 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.148e-06 2024-09-13T11:24:11.523 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:11.523 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:11.523 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:24:11.523 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:47.219721+0000", 2024-09-13T11:24:11.523 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.5555e-05 2024-09-13T11:24:11.523 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:11.523 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:11.523 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:24:11.524 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:47.219733+0000", 2024-09-13T11:24:11.524 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2098e-05 2024-09-13T11:24:11.524 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:11.524 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:11.524 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:24:11.524 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:47.219741+0000", 2024-09-13T11:24:11.524 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.5549999999999997e-06 2024-09-13T11:24:11.524 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:11.524 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:11.524 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:24:11.524 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:47.229374+0000", 2024-09-13T11:24:11.524 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0096331850000000007 2024-09-13T11:24:11.524 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:11.524 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:11.524 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:24:11.525 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:47.229554+0000", 2024-09-13T11:24:11.525 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00018005899999999999 2024-09-13T11:24:11.525 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:11.525 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:11.525 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 0", 2024-09-13T11:24:11.525 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:47.229742+0000", 2024-09-13T11:24:11.525 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000188469 2024-09-13T11:24:11.525 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:11.525 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:11.525 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:24:11.525 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:47.231370+0000", 2024-09-13T11:24:11.525 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.001627451 2024-09-13T11:24:11.525 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:11.526 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:11.526 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:11.526 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:47.346217+0000", 2024-09-13T11:24:11.526 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.114847115 2024-09-13T11:24:11.526 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:11.526 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:11.526 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:24:11.526 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:47.346285+0000", 2024-09-13T11:24:11.526 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.7891000000000004e-05 2024-09-13T11:24:11.526 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:11.526 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:11.526 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:24:11.526 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:47.346339+0000", 2024-09-13T11:24:11.526 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.4103e-05 2024-09-13T11:24:11.527 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:11.527 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:24:11.527 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:11.527 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:11.527 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:11.527 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4665.0:70358 3.f e95/75 3:f676477d:::benchmark_data_smithi177_31116_object70357:head v 95'3868, mlcod=95'3864)", 2024-09-13T11:24:11.527 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:52.802471+0000", 2024-09-13T11:24:11.527 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 18.658508511000001, 2024-09-13T11:24:11.527 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.144774187, 2024-09-13T11:24:11.527 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:24:11.527 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:24:11.527 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:11.527 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:24:11.527 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:11.528 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:24:11.528 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:52.802471+0000", 2024-09-13T11:24:11.528 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:11.528 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:11.528 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:11.528 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:24:11.528 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:52.802471+0000", 2024-09-13T11:24:11.528 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:11.528 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:11.528 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:11.528 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:24:11.528 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:52.802473+0000", 2024-09-13T11:24:11.528 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.8610000000000001e-06 2024-09-13T11:24:11.528 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:11.529 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:11.529 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:24:11.529 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:52.802559+0000", 2024-09-13T11:24:11.529 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.5884000000000006e-05 2024-09-13T11:24:11.529 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:11.529 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:11.529 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:24:11.529 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:52.945854+0000", 2024-09-13T11:24:11.529 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.14329460199999999 2024-09-13T11:24:11.529 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:11.529 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:11.529 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:24:11.529 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:52.945858+0000", 2024-09-13T11:24:11.529 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.42e-06 2024-09-13T11:24:11.530 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:11.530 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:11.530 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:24:11.530 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:52.945904+0000", 2024-09-13T11:24:11.530 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.5719000000000001e-05 2024-09-13T11:24:11.530 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:11.530 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:11.530 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:24:11.530 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:52.945917+0000", 2024-09-13T11:24:11.530 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.3002e-05 2024-09-13T11:24:11.530 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:11.530 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:11.530 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:24:11.530 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:52.947220+0000", 2024-09-13T11:24:11.531 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.001302903 2024-09-13T11:24:11.531 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:11.531 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:11.531 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:24:11.531 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:52.947246+0000", 2024-09-13T11:24:11.531 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.5795999999999998e-05 2024-09-13T11:24:11.531 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:11.531 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:24:11.531 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:11.531 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:11.531 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:11.531 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:70459 3.20 3:055da73a:::benchmark_data_smithi177_31116_object70458:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95)", 2024-09-13T11:24:11.531 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:52.920422+0000", 2024-09-13T11:24:11.531 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 18.540557510999999, 2024-09-13T11:24:11.531 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.132136635, 2024-09-13T11:24:11.532 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:24:11.532 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:24:11.532 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:11.532 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:24:11.532 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:24:11.532 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:11.532 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 70459 2024-09-13T11:24:11.532 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:11.532 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:24:11.532 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:11.532 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:24:11.532 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:52.920422+0000", 2024-09-13T11:24:11.532 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:11.533 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:11.533 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:11.533 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:24:11.533 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:52.920422+0000", 2024-09-13T11:24:11.533 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:11.533 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:11.533 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:11.533 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:24:11.533 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:52.920431+0000", 2024-09-13T11:24:11.533 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.7329999999999992e-06 2024-09-13T11:24:11.533 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:11.534 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:11.534 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:24:11.534 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:52.920532+0000", 2024-09-13T11:24:11.534 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00010066899999999999 2024-09-13T11:24:11.534 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:11.534 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:11.534 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:24:11.534 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:52.920566+0000", 2024-09-13T11:24:11.534 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.4452999999999997e-05 2024-09-13T11:24:11.534 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:11.534 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:11.534 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:24:11.534 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:52.920583+0000", 2024-09-13T11:24:11.535 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.7166000000000001e-05 2024-09-13T11:24:11.535 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:11.535 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:11.535 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:24:11.535 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:52.920687+0000", 2024-09-13T11:24:11.535 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000103463 2024-09-13T11:24:11.535 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:11.535 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:11.535 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:24:11.535 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:52.920816+0000", 2024-09-13T11:24:11.535 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00012956799999999999 2024-09-13T11:24:11.535 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:11.535 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:11.536 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 0", 2024-09-13T11:24:11.536 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:52.920916+0000", 2024-09-13T11:24:11.536 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.9803999999999995e-05 2024-09-13T11:24:11.536 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:11.536 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:11.536 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:24:11.536 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:52.921989+0000", 2024-09-13T11:24:11.536 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0010728560000000001 2024-09-13T11:24:11.536 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:11.536 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:11.536 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:11.536 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:53.052432+0000", 2024-09-13T11:24:11.536 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.13044277400000001 2024-09-13T11:24:11.536 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:11.537 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:11.537 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:24:11.537 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:53.052503+0000", 2024-09-13T11:24:11.537 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.1503999999999998e-05 2024-09-13T11:24:11.537 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:11.537 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:11.537 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:24:11.537 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:53.052559+0000", 2024-09-13T11:24:11.537 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.5645e-05 2024-09-13T11:24:11.537 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:11.537 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:24:11.537 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:11.537 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:11.538 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:11.538 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop_reply(client.4665.0:70459 3.20 e95/77 ondisk, result = 0)", 2024-09-13T11:24:11.538 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:52.922483+0000", 2024-09-13T11:24:11.538 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 18.538497268, 2024-09-13T11:24:11.538 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.13011756199999999, 2024-09-13T11:24:11.538 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:24:11.538 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:24:11.538 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:24:11.538 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:24:11.538 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:11.538 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:24:11.538 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:52.922483+0000", 2024-09-13T11:24:11.538 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:11.538 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:11.539 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:11.539 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:24:11.539 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:52.922483+0000", 2024-09-13T11:24:11.539 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:11.539 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:11.539 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:11.539 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:24:11.539 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:52.922484+0000", 2024-09-13T11:24:11.539 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.4449999999999999e-06 2024-09-13T11:24:11.539 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:11.539 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:11.539 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:24:11.539 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:52.922488+0000", 2024-09-13T11:24:11.539 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.2860000000000001e-06 2024-09-13T11:24:11.539 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:11.540 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:11.540 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:24:11.540 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:53.052355+0000", 2024-09-13T11:24:11.540 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.129866694 2024-09-13T11:24:11.540 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:11.540 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:11.540 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:24:11.540 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:53.052360+0000", 2024-09-13T11:24:11.540 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.6290000000000001e-06 2024-09-13T11:24:11.540 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:11.540 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:11.540 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:24:11.540 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:53.052414+0000", 2024-09-13T11:24:11.540 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.4521999999999997e-05 2024-09-13T11:24:11.541 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:11.541 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:11.541 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:24:11.541 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:53.052424+0000", 2024-09-13T11:24:11.541 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0081000000000001e-05 2024-09-13T11:24:11.541 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:11.541 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:11.541 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:24:11.541 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:53.052600+0000", 2024-09-13T11:24:11.541 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00017590499999999999 2024-09-13T11:24:11.541 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:11.541 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:24:11.541 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:11.541 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:11.542 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:11.542 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:71927 3.0 3:0197244a:::benchmark_data_smithi177_31116_object71926:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2024-09-13T11:24:11.542 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:54.199022+0000", 2024-09-13T11:24:11.542 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 17.261958147000001, 2024-09-13T11:24:11.542 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.26819617800000001, 2024-09-13T11:24:11.542 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:24:11.542 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:24:11.542 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:11.542 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:24:11.542 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:24:11.542 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:11.542 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 71927 2024-09-13T11:24:11.542 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:11.542 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:24:11.542 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:11.543 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:24:11.543 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:54.199022+0000", 2024-09-13T11:24:11.543 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:11.543 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:11.543 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:11.543 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:24:11.543 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:54.199022+0000", 2024-09-13T11:24:11.543 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:11.543 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:11.543 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:11.543 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:24:11.543 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:54.199025+0000", 2024-09-13T11:24:11.543 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.2200000000000001e-06 2024-09-13T11:24:11.543 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:11.544 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:11.544 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:24:11.544 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:54.199076+0000", 2024-09-13T11:24:11.544 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.1184000000000003e-05 2024-09-13T11:24:11.544 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:11.544 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:11.544 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:24:11.544 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:54.199095+0000", 2024-09-13T11:24:11.544 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.9253999999999999e-05 2024-09-13T11:24:11.544 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:11.544 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:11.544 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:24:11.544 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:54.199107+0000", 2024-09-13T11:24:11.545 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.1916000000000001e-05 2024-09-13T11:24:11.545 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:11.545 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:11.545 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:24:11.545 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:54.199196+0000", 2024-09-13T11:24:11.545 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.8645000000000003e-05 2024-09-13T11:24:11.545 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:11.545 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:11.545 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:24:11.545 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:54.199421+0000", 2024-09-13T11:24:11.546 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000224868 2024-09-13T11:24:11.546 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:11.546 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:11.546 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 15", 2024-09-13T11:24:11.546 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:54.199647+0000", 2024-09-13T11:24:11.546 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00022666099999999999 2024-09-13T11:24:11.546 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:11.546 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:11.546 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:24:11.546 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:54.201333+0000", 2024-09-13T11:24:11.546 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.001685761 2024-09-13T11:24:11.546 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:11.546 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:11.547 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:11.547 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:54.467097+0000", 2024-09-13T11:24:11.547 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.26576393399999998 2024-09-13T11:24:11.547 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:11.547 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:11.547 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:24:11.547 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:54.467163+0000", 2024-09-13T11:24:11.547 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.6088999999999997e-05 2024-09-13T11:24:11.547 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:11.547 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:11.547 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:24:11.547 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:54.467218+0000", 2024-09-13T11:24:11.547 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.4645999999999998e-05 2024-09-13T11:24:11.547 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:11.548 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:24:11.548 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:11.548 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:11.548 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:11.548 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:72250 3.0 3:010c7171:::benchmark_data_smithi177_31116_object72249:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2024-09-13T11:24:11.548 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:54.515776+0000", 2024-09-13T11:24:11.548 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 16.945203643999999, 2024-09-13T11:24:11.548 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.137300168, 2024-09-13T11:24:11.548 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:24:11.548 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:24:11.548 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:11.548 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:24:11.548 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:24:11.548 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:11.549 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 72250 2024-09-13T11:24:11.549 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:11.549 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:24:11.549 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:11.549 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:24:11.549 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:54.515776+0000", 2024-09-13T11:24:11.549 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:11.549 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:11.549 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:11.549 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:24:11.549 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:54.515776+0000", 2024-09-13T11:24:11.549 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:11.550 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:11.550 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:11.550 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:24:11.550 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:54.515778+0000", 2024-09-13T11:24:11.550 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.001e-06 2024-09-13T11:24:11.550 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:11.550 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:11.550 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:24:11.550 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:54.515802+0000", 2024-09-13T11:24:11.550 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.3649e-05 2024-09-13T11:24:11.550 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:11.550 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:11.550 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:24:11.550 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:54.515813+0000", 2024-09-13T11:24:11.551 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.1309e-05 2024-09-13T11:24:11.551 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:11.551 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:11.551 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:24:11.551 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:54.515819+0000", 2024-09-13T11:24:11.551 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.0580000000000003e-06 2024-09-13T11:24:11.551 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:11.551 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:11.551 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:24:11.551 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:54.515907+0000", 2024-09-13T11:24:11.551 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.7688000000000005e-05 2024-09-13T11:24:11.551 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:11.551 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:11.551 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:24:11.552 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:54.516003+0000", 2024-09-13T11:24:11.552 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.6050000000000004e-05 2024-09-13T11:24:11.552 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:11.552 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:11.552 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 15", 2024-09-13T11:24:11.552 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:54.516095+0000", 2024-09-13T11:24:11.552 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.2541999999999996e-05 2024-09-13T11:24:11.552 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:11.552 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:11.552 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:24:11.552 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:54.517494+0000", 2024-09-13T11:24:11.552 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.001398954 2024-09-13T11:24:11.552 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:11.553 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:11.553 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:11.553 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:54.653006+0000", 2024-09-13T11:24:11.553 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.135511509 2024-09-13T11:24:11.553 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:11.553 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:11.553 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:24:11.553 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:54.653043+0000", 2024-09-13T11:24:11.553 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.7546000000000003e-05 2024-09-13T11:24:11.553 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:11.553 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:11.553 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:24:11.553 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:54.653076+0000", 2024-09-13T11:24:11.554 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.2861999999999998e-05 2024-09-13T11:24:11.554 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:11.554 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:24:11.554 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:11.554 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:11.554 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:11.554 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:72993 3.0 3:02747b8f:::benchmark_data_smithi177_31116_object72992:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2024-09-13T11:24:11.554 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:55.114916+0000", 2024-09-13T11:24:11.554 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 16.346063745999999, 2024-09-13T11:24:11.554 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.14125180900000001, 2024-09-13T11:24:11.554 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:24:11.554 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:24:11.554 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:11.555 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:24:11.555 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:24:11.555 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:11.555 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 72993 2024-09-13T11:24:11.555 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:11.555 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:24:11.555 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:11.555 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:24:11.555 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:55.114916+0000", 2024-09-13T11:24:11.555 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:11.555 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:11.555 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:11.555 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:24:11.555 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:55.114916+0000", 2024-09-13T11:24:11.556 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:11.556 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:11.556 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:11.556 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:24:11.556 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:55.114922+0000", 2024-09-13T11:24:11.556 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.9889999999999997e-06 2024-09-13T11:24:11.556 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:11.556 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:11.556 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:24:11.556 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:55.115018+0000", 2024-09-13T11:24:11.556 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.6419e-05 2024-09-13T11:24:11.556 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:11.557 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:11.557 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:24:11.557 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:55.115034+0000", 2024-09-13T11:24:11.557 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.5376999999999999e-05 2024-09-13T11:24:11.557 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:11.557 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:11.557 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:24:11.557 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:55.115042+0000", 2024-09-13T11:24:11.557 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.6310000000000003e-06 2024-09-13T11:24:11.557 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:11.557 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:11.557 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:24:11.557 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:55.115126+0000", 2024-09-13T11:24:11.557 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.4856000000000002e-05 2024-09-13T11:24:11.558 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:11.558 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:11.558 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:24:11.558 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:55.115434+0000", 2024-09-13T11:24:11.558 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00030747599999999999 2024-09-13T11:24:11.558 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:11.558 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:11.558 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 15", 2024-09-13T11:24:11.558 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:55.115670+0000", 2024-09-13T11:24:11.558 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00023618200000000001 2024-09-13T11:24:11.558 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:11.558 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:11.558 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:24:11.558 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:55.117162+0000", 2024-09-13T11:24:11.559 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0014920650000000001 2024-09-13T11:24:11.559 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:11.559 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:11.559 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:11.559 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:55.256062+0000", 2024-09-13T11:24:11.559 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.13890022399999999 2024-09-13T11:24:11.559 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:11.559 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:11.559 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:24:11.559 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:55.256122+0000", 2024-09-13T11:24:11.559 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.9935999999999997e-05 2024-09-13T11:24:11.559 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:11.560 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:11.560 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:24:11.560 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:55.256168+0000", 2024-09-13T11:24:11.560 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.5654e-05 2024-09-13T11:24:11.560 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:11.560 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:24:11.560 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:11.560 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:11.560 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:11.560 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:73523 3.0 3:004115ff:::benchmark_data_smithi177_31116_object73522:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98)", 2024-09-13T11:24:11.560 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:55.537486+0000", 2024-09-13T11:24:11.560 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 15.923494041, 2024-09-13T11:24:11.560 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.128377828, 2024-09-13T11:24:11.560 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:24:11.561 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:24:11.561 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:11.561 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:24:11.561 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:24:11.561 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:11.561 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 73523 2024-09-13T11:24:11.561 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:11.561 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:24:11.561 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:11.561 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:24:11.561 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:55.537486+0000", 2024-09-13T11:24:11.561 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:11.561 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:11.561 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:11.562 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:24:11.562 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:55.537486+0000", 2024-09-13T11:24:11.562 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:11.562 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:11.562 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:11.562 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:24:11.562 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:55.537488+0000", 2024-09-13T11:24:11.562 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.951e-06 2024-09-13T11:24:11.562 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:11.562 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:11.562 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:24:11.562 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:55.537527+0000", 2024-09-13T11:24:11.562 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.9594000000000001e-05 2024-09-13T11:24:11.562 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:11.563 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:11.563 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:24:11.563 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:55.537539+0000", 2024-09-13T11:24:11.563 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.1904e-05 2024-09-13T11:24:11.563 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:11.563 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:11.563 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:24:11.563 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:55.537547+0000", 2024-09-13T11:24:11.563 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.9039999999999992e-06 2024-09-13T11:24:11.563 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:11.563 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:11.563 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:24:11.563 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:55.537630+0000", 2024-09-13T11:24:11.564 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.2620000000000005e-05 2024-09-13T11:24:11.564 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:11.564 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:11.564 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:24:11.564 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:55.537752+0000", 2024-09-13T11:24:11.564 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00012202200000000001 2024-09-13T11:24:11.564 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:11.564 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:11.564 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 15", 2024-09-13T11:24:11.564 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:55.537882+0000", 2024-09-13T11:24:11.564 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00013000699999999999 2024-09-13T11:24:11.564 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:11.564 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:11.565 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:24:11.565 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:55.540025+0000", 2024-09-13T11:24:11.565 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0021435769999999998 2024-09-13T11:24:11.565 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:11.565 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:11.565 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:11.565 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:55.665787+0000", 2024-09-13T11:24:11.565 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.12576148200000001 2024-09-13T11:24:11.565 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:11.565 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:11.565 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:24:11.565 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:55.665832+0000", 2024-09-13T11:24:11.565 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.4832e-05 2024-09-13T11:24:11.566 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:11.566 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:11.566 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:24:11.566 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:55.665864+0000", 2024-09-13T11:24:11.566 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.1934999999999998e-05 2024-09-13T11:24:11.566 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:11.566 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:24:11.566 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:11.566 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:11.566 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:11.566 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:89731 3.20 3:04de772d:::benchmark_data_smithi177_31116_object89730:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110)", 2024-09-13T11:24:11.566 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:24:09.624252+0000", 2024-09-13T11:24:11.566 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 1.836727515, 2024-09-13T11:24:11.567 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.165003973, 2024-09-13T11:24:11.567 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:24:11.567 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:24:11.567 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:11.567 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:24:11.567 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:24:11.567 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:11.567 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 89731 2024-09-13T11:24:11.567 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:11.567 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:24:11.567 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:11.567 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:24:11.567 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:09.624252+0000", 2024-09-13T11:24:11.567 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:11.568 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:11.568 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:11.568 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:24:11.568 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:09.624252+0000", 2024-09-13T11:24:11.568 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:11.568 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:11.568 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:11.568 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:24:11.568 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:09.624255+0000", 2024-09-13T11:24:11.568 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.0450000000000001e-06 2024-09-13T11:24:11.568 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:11.568 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:11.568 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:24:11.569 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:09.624306+0000", 2024-09-13T11:24:11.569 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.0769999999999997e-05 2024-09-13T11:24:11.569 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:11.569 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:11.569 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:24:11.569 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:09.624326+0000", 2024-09-13T11:24:11.569 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.9409999999999999e-05 2024-09-13T11:24:11.569 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:11.569 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:11.569 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:24:11.569 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:09.624336+0000", 2024-09-13T11:24:11.569 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0709e-05 2024-09-13T11:24:11.569 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:11.570 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:11.570 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:24:11.570 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:09.624422+0000", 2024-09-13T11:24:11.570 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.619e-05 2024-09-13T11:24:11.570 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:11.570 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:11.570 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:24:11.570 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:09.624528+0000", 2024-09-13T11:24:11.570 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00010567499999999999 2024-09-13T11:24:11.570 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:11.570 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:11.570 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 0", 2024-09-13T11:24:11.570 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:09.624647+0000", 2024-09-13T11:24:11.570 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000118418 2024-09-13T11:24:11.571 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:11.571 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:11.571 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:24:11.571 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:09.625695+0000", 2024-09-13T11:24:11.571 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.001048799 2024-09-13T11:24:11.571 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:11.571 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:11.571 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:11.571 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:09.789054+0000", 2024-09-13T11:24:11.571 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.163358222 2024-09-13T11:24:11.571 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:11.571 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:11.571 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:24:11.572 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:09.789173+0000", 2024-09-13T11:24:11.572 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00011897599999999999 2024-09-13T11:24:11.572 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:11.572 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:11.572 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:24:11.572 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:09.789256+0000", 2024-09-13T11:24:11.572 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.3758999999999995e-05 2024-09-13T11:24:11.572 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:11.572 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:24:11.572 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:11.572 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:11.572 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:24:11.572 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-13T11:24:11.574 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-13T11:24:11.575 INFO:teuthology.orchestra.run.smithi001.stdout:osd.6: {} 2024-09-13T11:24:11.575 INFO:teuthology.orchestra.run.smithi001.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T11:24:11.576 INFO:tasks.ceph.osd.11.smithi177.stderr:2024-09-13T11:24:11.571+0000 7f24792c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26580) UID: 0 2024-09-13T11:24:11.674 INFO:tasks.ceph.osd.14.smithi162.stderr:2024-09-13T11:24:11.672+0000 7f235c0db640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26319) UID: 0 2024-09-13T11:24:11.716 INFO:teuthology.orchestra.run.smithi092.stdout:ERROR: (22) Invalid argument 2024-09-13T11:24:11.716 INFO:teuthology.orchestra.run.smithi092.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:24:11.735 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-13T11:24:11.775 INFO:tasks.ceph.osd.14.smithi162.stderr:2024-09-13T11:24:11.772+0000 7f235c0db640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26319) UID: 0 2024-09-13T11:24:11.875 INFO:teuthology.orchestra.run.smithi092.stdout:ERROR: (22) Invalid argument 2024-09-13T11:24:11.875 INFO:teuthology.orchestra.run.smithi092.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:24:11.876 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-09-13T11:24:11.872+0000 7f97befc9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26315) UID: 0 2024-09-13T11:24:11.892 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-13T11:24:11.927 INFO:tasks.ceph.osd.1.smithi092.stderr:2024-09-13T11:24:11.923+0000 7fe76d3f2640 -1 osd.1 111 handle_osd_map injecting map crc failure 2024-09-13T11:24:11.977 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-09-13T11:24:11.972+0000 7f97befc9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26315) UID: 0 2024-09-13T11:24:12.008 INFO:tasks.radosbench.radosbench.0.smithi177.stdout: 105 16 93056 93040 55.3754 92.8125 0.00352495 0.0180529 2024-09-13T11:24:12.029 INFO:teuthology.orchestra.run.smithi092.stdout:ERROR: (22) Invalid argument 2024-09-13T11:24:12.029 INFO:teuthology.orchestra.run.smithi092.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:24:12.044 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-09-13T11:24:12.081 INFO:tasks.ceph.osd.15.smithi177.stderr:2024-09-13T11:24:12.079+0000 7f9d6ce77640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26578) UID: 0 2024-09-13T11:24:12.182 INFO:tasks.ceph.osd.14.smithi162.stderr:2024-09-13T11:24:12.180+0000 7f235c0db640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26319) UID: 0 2024-09-13T11:24:12.216 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-09-13T11:24:12.216 INFO:teuthology.orchestra.run.smithi092.stdout: "ops": [], 2024-09-13T11:24:12.216 INFO:teuthology.orchestra.run.smithi092.stdout: "num_ops": 0 2024-09-13T11:24:12.217 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-09-13T11:24:12.233 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-09-13T11:24:12.285 INFO:tasks.ceph.osd.12.smithi001.stderr:2024-09-13T11:24:12.279+0000 7f892c6e6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26314) UID: 0 2024-09-13T11:24:12.314 INFO:teuthology.orchestra.run.smithi001.stdout:osd.7: {} 2024-09-13T11:24:12.314 INFO:teuthology.orchestra.run.smithi001.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T11:24:12.338 INFO:teuthology.orchestra.run.smithi001.stdout:osd.8: {} 2024-09-13T11:24:12.338 INFO:teuthology.orchestra.run.smithi001.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-13T11:24:12.365 INFO:teuthology.orchestra.run.smithi001.stdout:osd.9: {} 2024-09-13T11:24:12.365 INFO:teuthology.orchestra.run.smithi001.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-13T11:24:12.375 INFO:teuthology.orchestra.run.smithi092.stdout:ERROR: (22) Invalid argument 2024-09-13T11:24:12.375 INFO:teuthology.orchestra.run.smithi092.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:24:12.386 INFO:tasks.ceph.osd.2.smithi162.stderr:2024-09-13T11:24:12.380+0000 7fe92b35a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26313) UID: 0 2024-09-13T11:24:12.393 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-09-13T11:24:12.394 INFO:teuthology.orchestra.run.smithi001.stdout:osd.10: {} 2024-09-13T11:24:12.394 INFO:teuthology.orchestra.run.smithi001.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-13T11:24:12.425 INFO:teuthology.orchestra.run.smithi001.stdout:osd.11: {} 2024-09-13T11:24:12.425 INFO:teuthology.orchestra.run.smithi001.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-13T11:24:12.453 INFO:teuthology.orchestra.run.smithi001.stdout:osd.12: {} 2024-09-13T11:24:12.453 INFO:teuthology.orchestra.run.smithi001.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-13T11:24:12.480 INFO:teuthology.orchestra.run.smithi001.stdout:osd.13: {} 2024-09-13T11:24:12.480 INFO:teuthology.orchestra.run.smithi001.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-13T11:24:12.487 INFO:tasks.ceph.osd.4.smithi001.stderr:2024-09-13T11:24:12.479+0000 7ff8e0218640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26310) UID: 0 2024-09-13T11:24:12.506 INFO:teuthology.orchestra.run.smithi001.stdout:osd.14: {} 2024-09-13T11:24:12.507 INFO:teuthology.orchestra.run.smithi001.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-13T11:24:12.533 INFO:teuthology.orchestra.run.smithi001.stdout:osd.15: {} 2024-09-13T11:24:12.533 INFO:teuthology.orchestra.run.smithi001.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-13T11:24:12.533 INFO:teuthology.orchestra.run.smithi092.stdout:ERROR: (22) Invalid argument 2024-09-13T11:24:12.533 INFO:teuthology.orchestra.run.smithi092.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:24:12.550 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_ops_in_flight 2024-09-13T11:24:12.587 INFO:tasks.ceph.osd.14.smithi162.stderr:2024-09-13T11:24:12.584+0000 7f235c0db640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26319) UID: 0 2024-09-13T11:24:12.689 INFO:tasks.ceph.osd.10.smithi162.stderr:2024-09-13T11:24:12.684+0000 7fb59c9f0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26317) UID: 0 2024-09-13T11:24:12.695 INFO:teuthology.orchestra.run.smithi092.stdout:ERROR: (22) Invalid argument 2024-09-13T11:24:12.695 INFO:teuthology.orchestra.run.smithi092.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:24:12.714 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_blocked_ops 2024-09-13T11:24:12.790 INFO:tasks.ceph.osd.15.smithi177.stderr:2024-09-13T11:24:12.787+0000 7f9d6ce77640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26578) UID: 0 2024-09-13T11:24:12.863 INFO:teuthology.orchestra.run.smithi092.stdout:ERROR: (22) Invalid argument 2024-09-13T11:24:12.864 INFO:teuthology.orchestra.run.smithi092.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:24:12.883 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_historic_ops 2024-09-13T11:24:12.930 INFO:tasks.ceph.osd.13.smithi092.stderr:2024-09-13T11:24:12.927+0000 7f3e5afea640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26350) UID: 0 2024-09-13T11:24:12.992 INFO:tasks.ceph.osd.5.smithi092.stderr:2024-09-13T11:24:12.987+0000 7f6f0b705640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26346) UID: 0 2024-09-13T11:24:13.008 INFO:tasks.radosbench.radosbench.0.smithi177.stdout: 106 16 94913 94897 55.9478 116.062 0.00887108 0.0178684 2024-09-13T11:24:13.011 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-13T11:24:13.022 INFO:teuthology.orchestra.run.smithi092.stdout:ERROR: (22) Invalid argument 2024-09-13T11:24:13.022 INFO:teuthology.orchestra.run.smithi092.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:24:13.042 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-13T11:24:13.093 INFO:tasks.ceph.osd.8.smithi001.stderr:2024-09-13T11:24:13.087+0000 7fb9b48ae640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26312) UID: 0 2024-09-13T11:24:13.189 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-13T11:24:13.189 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:24:13.193 INFO:tasks.ceph.osd.3.smithi177.stderr:2024-09-13T11:24:13.191+0000 7f10adba4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26574) UID: 0 2024-09-13T11:24:13.207 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-13T11:24:13.294 INFO:tasks.ceph.osd.2.smithi162.stderr:2024-09-13T11:24:13.292+0000 7fe92b35a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26313) UID: 0 2024-09-13T11:24:13.353 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-13T11:24:13.353 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:24:13.372 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-13T11:24:13.395 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-13T11:24:13.387+0000 7f6a68775640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26307) UID: 0 2024-09-13T11:24:13.496 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-09-13T11:24:13.492+0000 7f97befc9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26315) UID: 0 2024-09-13T11:24:13.551 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-13T11:24:13.551 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:24:13.569 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-13T11:24:13.610 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-09-13T11:24:13.608+0000 7f97befc9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26315) UID: 0 2024-09-13T11:24:13.698 INFO:tasks.ceph.osd.14.smithi162.stderr:2024-09-13T11:24:13.696+0000 7f235c0db640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26319) UID: 0 2024-09-13T11:24:13.753 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-13T11:24:13.754 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:24:13.771 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-13T11:24:13.799 INFO:tasks.ceph.osd.1.smithi092.stderr:2024-09-13T11:24:13.795+0000 7fe7789c5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33718) UID: 0 2024-09-13T11:24:13.900 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-13T11:24:13.895+0000 7f6a68775640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26307) UID: 0 2024-09-13T11:24:13.956 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-13T11:24:13.956 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:24:13.975 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-13T11:24:13.998 INFO:tasks.ceph.osd.3.smithi177.stderr:2024-09-13T11:24:13.995+0000 7f10a45d5640 -1 osd.3 112 handle_osd_map injecting map crc failure 2024-09-13T11:24:14.000 INFO:teuthology.orchestra.run.smithi001.stderr:noscrub is set 2024-09-13T11:24:14.000 INFO:tasks.ceph.osd.13.smithi092.stderr:2024-09-13T11:24:13.995+0000 7f3e5afea640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26350) UID: 0 2024-09-13T11:24:14.004 INFO:tasks.ceph.osd.2.smithi162.stderr:2024-09-13T11:24:14.000+0000 7fe91fd87640 -1 osd.2 112 handle_osd_map injecting map crc failure 2024-09-13T11:24:14.008 INFO:tasks.radosbench.radosbench.0.smithi177.stdout: 107 16 96438 96422 56.3156 95.3125 0.00785897 0.0177511 2024-09-13T11:24:14.058 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-13T11:24:14.101 INFO:tasks.ceph.osd.3.smithi177.stderr:2024-09-13T11:24:14.099+0000 7f10adba4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26574) UID: 0 2024-09-13T11:24:14.113 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-13T11:24:14.113 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:24:14.134 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-09-13T11:24:14.202 INFO:tasks.ceph.osd.13.smithi092.stderr:2024-09-13T11:24:14.199+0000 7f3e5afea640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26350) UID: 0 2024-09-13T11:24:14.303 INFO:tasks.ceph.osd.10.smithi162.stderr:2024-09-13T11:24:14.300+0000 7fb59c9f0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26317) UID: 0 2024-09-13T11:24:14.324 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-13T11:24:14.325 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:24:14.341 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-09-13T11:24:14.404 INFO:tasks.ceph.osd.15.smithi177.stderr:2024-09-13T11:24:14.403+0000 7f9d6ce77640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26578) UID: 0 2024-09-13T11:24:14.462 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-13T11:24:14.463 INFO:teuthology.orchestra.run.smithi001.stderr:dumped all 2024-09-13T11:24:14.485 INFO:teuthology.orchestra.run.smithi001.stdout:{"pg_ready":true,"pg_map":{"version":161,"stamp":"2024-09-13T11:24:14.144540+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":6024233523,"num_objects":91918,"num_object_clones":0,"num_object_copies":183836,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":91918,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":37467,"num_write_kb":2391122,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5755,"num_bytes_recovered":377094163,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":152608,"ondisk_log_size":152608,"up":90,"acting":90,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":112,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":16,"kb":1509949440,"kb_used":14101452,"kb_used_data":13203276,"kb_used_omap":27,"kb_used_meta":897892,"kb_avail":1495847988,"statfs":{"total":1546188226560,"available":1531748339712,"internally_reserved":0,"allocated":13520154624,"data_stored":13512431455,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28358,"internal_metadata":919441722},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,2,1,1,3,2,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":450789376,"num_objects":6878,"num_object_clones":0,"num_object_copies":13756,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6878,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-1187,"num_write_kb":-75936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-5872,"num_bytes_recovered":-384794624,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001187"},"pg_stats":[{"pgid":"3.23","version":"110'3710","reported_seq":6446,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T11:24:10.829789+0000","last_change":"2024-09-13T11:23:48.133184+0000","last_active":"2024-09-13T11:24:10.829789+0000","last_peered":"2024-09-13T11:24:10.829789+0000","last_clean":"2024-09-13T11:24:10.829789+0000","last_became_active":"2024-09-13T11:23:47.926677+0000","last_became_peered":"2024-09-13T11:23:47.926677+0000","last_unstale":"2024-09-13T11:24:10.829789+0000","last_undegraded":"2024-09-13T11:24:10.829789+0000","last_fullsized":"2024-09-13T11:24:10.829789+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":6,"last_scrub":"91'3278","last_scrub_stamp":"2024-09-13T11:23:48.133084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:48.133084+0000","objects_scrubbed":1029,"log_size":3710,"log_dups_size":0,"ondisk_log_size":3710,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:12.874134+0000","scrub_duration":139,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":95813632,"num_objects":1462,"num_object_clones":0,"num_object_copies":2924,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1462,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":433,"num_write_kb":27712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9],"acting":[5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.22","version":"111'3684","reported_seq":5609,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T11:24:11.003707+0000","last_change":"2024-09-13T11:23:47.300446+0000","last_active":"2024-09-13T11:24:11.003707+0000","last_peered":"2024-09-13T11:24:11.003707+0000","last_clean":"2024-09-13T11:24:11.003707+0000","last_became_active":"2024-09-13T11:23:46.925318+0000","last_became_peered":"2024-09-13T11:23:46.925318+0000","last_unstale":"2024-09-13T11:24:11.003707+0000","last_undegraded":"2024-09-13T11:24:11.003707+0000","last_fullsized":"2024-09-13T11:24:11.003707+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":6,"last_scrub":"90'3246","last_scrub_stamp":"2024-09-13T11:23:47.300339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:47.300339+0000","objects_scrubbed":1033,"log_size":3684,"log_dups_size":0,"ondisk_log_size":3684,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:24:49.780485+0000","scrub_duration":203,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96534528,"num_objects":1473,"num_object_clones":0,"num_object_copies":2946,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1473,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":440,"num_write_kb":28160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,14],"acting":[0,14],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.21","version":"110'3622","reported_seq":6353,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T11:24:11.003655+0000","last_change":"2024-09-13T11:24:07.306104+0000","last_active":"2024-09-13T11:24:11.003655+0000","last_peered":"2024-09-13T11:24:11.003655+0000","last_clean":"2024-09-13T11:24:11.003655+0000","last_became_active":"2024-09-13T11:24:01.250308+0000","last_became_peered":"2024-09-13T11:24:01.250308+0000","last_unstale":"2024-09-13T11:24:11.003655+0000","last_undegraded":"2024-09-13T11:24:11.003655+0000","last_fullsized":"2024-09-13T11:24:11.003655+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":6,"last_scrub":"69'2784","last_scrub_stamp":"2024-09-13T11:23:21.595419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:21.595419+0000","objects_scrubbed":0,"log_size":3622,"log_dups_size":0,"ondisk_log_size":3622,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"user requested scrub scheduled @ 2024-09-13T11:24:12.261215+0000","scrub_duration":311,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":91095040,"num_objects":1390,"num_object_clones":0,"num_object_copies":2780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1390,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":697,"num_write_kb":44608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":836,"num_bytes_recovered":54788096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8],"acting":[0,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.20","version":"111'3700","reported_seq":5740,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T11:24:11.039984+0000","last_change":"2024-09-13T11:23:47.969503+0000","last_active":"2024-09-13T11:24:11.039984+0000","last_peered":"2024-09-13T11:24:11.039984+0000","last_clean":"2024-09-13T11:24:11.039984+0000","last_became_active":"2024-09-13T11:23:33.954946+0000","last_became_peered":"2024-09-13T11:23:33.954946+0000","last_unstale":"2024-09-13T11:24:11.039984+0000","last_undegraded":"2024-09-13T11:24:11.039984+0000","last_fullsized":"2024-09-13T11:24:11.039984+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":6,"last_scrub":"91'3270","last_scrub_stamp":"2024-09-13T11:23:47.969372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:47.969372+0000","objects_scrubbed":1061,"log_size":3700,"log_dups_size":0,"ondisk_log_size":3700,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:24:48.255263+0000","scrub_duration":1327,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":98041856,"num_objects":1496,"num_object_clones":0,"num_object_copies":2992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1496,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":435,"num_write_kb":27840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,0],"acting":[12,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[]},{"pgid":"3.1f","version":"110'4534","reported_seq":9711,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T11:24:11.028813+0000","last_change":"2024-09-13T11:24:08.388852+0000","last_active":"2024-09-13T11:24:11.028813+0000","last_peered":"2024-09-13T11:24:11.028813+0000","last_clean":"2024-09-13T11:24:11.028813+0000","last_became_active":"2024-09-13T11:24:06.967832+0000","last_became_peered":"2024-09-13T11:24:06.967832+0000","last_unstale":"2024-09-13T11:24:11.028813+0000","last_undegraded":"2024-09-13T11:24:11.028813+0000","last_fullsized":"2024-09-13T11:24:11.028813+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":5,"last_scrub":"110'4427","last_scrub_stamp":"2024-09-13T11:24:08.388782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:24:08.388782+0000","objects_scrubbed":2651,"log_size":4534,"log_dups_size":0,"ondisk_log_size":4534,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:37.901131+0000","scrub_duration":315,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":181207040,"num_objects":2765,"num_object_clones":0,"num_object_copies":5530,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2765,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":114,"num_write_kb":7296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,14],"acting":[10,14],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1e","version":"110'4666","reported_seq":15068,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T11:24:10.910459+0000","last_change":"2024-09-13T11:24:08.338485+0000","last_active":"2024-09-13T11:24:10.910459+0000","last_peered":"2024-09-13T11:24:10.910459+0000","last_clean":"2024-09-13T11:24:10.910459+0000","last_became_active":"2024-09-13T11:24:01.261448+0000","last_became_peered":"2024-09-13T11:24:01.261448+0000","last_unstale":"2024-09-13T11:24:10.910459+0000","last_undegraded":"2024-09-13T11:24:10.910459+0000","last_fullsized":"2024-09-13T11:24:10.910459+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":5,"last_scrub":"110'4537","last_scrub_stamp":"2024-09-13T11:24:08.338354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:24:08.338354+0000","objects_scrubbed":2760,"log_size":4666,"log_dups_size":0,"ondisk_log_size":4666,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:11.010072+0000","scrub_duration":367,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":189923328,"num_objects":2898,"num_object_clones":0,"num_object_copies":5796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2898,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":138,"num_write_kb":8832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"111'4502","reported_seq":9796,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T11:24:11.011338+0000","last_change":"2024-09-13T11:24:07.062288+0000","last_active":"2024-09-13T11:24:11.011338+0000","last_peered":"2024-09-13T11:24:11.011338+0000","last_clean":"2024-09-13T11:24:11.011338+0000","last_became_active":"2024-09-13T11:24:01.258225+0000","last_became_peered":"2024-09-13T11:24:01.258225+0000","last_unstale":"2024-09-13T11:24:11.011338+0000","last_undegraded":"2024-09-13T11:24:11.011338+0000","last_fullsized":"2024-09-13T11:24:11.011338+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:22:24.896055+0000","objects_scrubbed":0,"log_size":4502,"log_dups_size":0,"ondisk_log_size":4502,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-13T11:24:12.048254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":184221696,"num_objects":2811,"num_object_clones":0,"num_object_copies":5622,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2811,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2811,"num_write_kb":179904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1700,"num_bytes_recovered":111411200,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,15],"acting":[2,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1c","version":"112'4660","reported_seq":8352,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-09-13T11:24:12.063510+0000","last_change":"2024-09-13T11:24:00.529312+0000","last_active":"2024-09-13T11:24:12.063510+0000","last_peered":"2024-09-13T11:24:12.063510+0000","last_clean":"2024-09-13T11:24:12.063510+0000","last_became_active":"2024-09-13T11:23:33.989492+0000","last_became_peered":"2024-09-13T11:23:33.989492+0000","last_unstale":"2024-09-13T11:24:12.063510+0000","last_undegraded":"2024-09-13T11:24:12.063510+0000","last_fullsized":"2024-09-13T11:24:12.063510+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":5,"last_scrub":"102'4277","last_scrub_stamp":"2024-09-13T11:24:00.529243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:24:00.529243+0000","objects_scrubbed":2544,"log_size":4660,"log_dups_size":0,"ondisk_log_size":4660,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:15.595052+0000","scrub_duration":259,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":191954944,"num_objects":2929,"num_object_clones":0,"num_object_copies":5858,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2929,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":385,"num_write_kb":24640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7],"acting":[11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1b","version":"111'4594","reported_seq":10068,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T11:24:11.086712+0000","last_change":"2024-09-13T11:24:05.835103+0000","last_active":"2024-09-13T11:24:11.086712+0000","last_peered":"2024-09-13T11:24:11.086712+0000","last_clean":"2024-09-13T11:24:11.086712+0000","last_became_active":"2024-09-13T11:24:04.950821+0000","last_became_peered":"2024-09-13T11:24:04.950821+0000","last_unstale":"2024-09-13T11:24:11.086712+0000","last_undegraded":"2024-09-13T11:24:11.086712+0000","last_fullsized":"2024-09-13T11:24:11.086712+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:22:24.896055+0000","objects_scrubbed":0,"log_size":4594,"log_dups_size":0,"ondisk_log_size":4594,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":187662336,"num_objects":2864,"num_object_clones":0,"num_object_copies":5728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2864,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2864,"num_write_kb":183264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1760,"num_bytes_recovered":115343360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,15],"acting":[12,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[]},{"pgid":"3.1a","version":"111'4503","reported_seq":9676,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T11:24:11.023180+0000","last_change":"2024-09-13T11:24:09.556174+0000","last_active":"2024-09-13T11:24:11.023180+0000","last_peered":"2024-09-13T11:24:11.023180+0000","last_clean":"2024-09-13T11:24:11.023180+0000","last_became_active":"2024-09-13T11:24:03.935185+0000","last_became_peered":"2024-09-13T11:24:03.935185+0000","last_unstale":"2024-09-13T11:24:11.023180+0000","last_undegraded":"2024-09-13T11:24:11.023180+0000","last_fullsized":"2024-09-13T11:24:11.023180+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":5,"last_scrub":"110'4433","last_scrub_stamp":"2024-09-13T11:24:09.556022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:24:09.556022+0000","objects_scrubbed":2654,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:21.773175+0000","scrub_duration":387,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":179240960,"num_objects":2735,"num_object_clones":0,"num_object_copies":5470,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2735,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":81,"num_write_kb":5184,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9],"acting":[4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.19","version":"112'4290","reported_seq":8605,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-09-13T11:24:11.983431+0000","last_change":"2024-09-13T11:23:49.429522+0000","last_active":"2024-09-13T11:24:11.983431+0000","last_peered":"2024-09-13T11:24:11.983431+0000","last_clean":"2024-09-13T11:24:11.983431+0000","last_became_active":"2024-09-13T11:23:41.924941+0000","last_became_peered":"2024-09-13T11:23:41.924941+0000","last_unstale":"2024-09-13T11:24:11.983431+0000","last_undegraded":"2024-09-13T11:24:11.983431+0000","last_fullsized":"2024-09-13T11:24:11.983431+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":5,"last_scrub":"92'3415","last_scrub_stamp":"2024-09-13T11:23:49.429429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:49.429429+0000","objects_scrubbed":1979,"log_size":4290,"log_dups_size":0,"ondisk_log_size":4290,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:04.629542+0000","scrub_duration":587,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":187891712,"num_objects":2867,"num_object_clones":0,"num_object_copies":5734,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2867,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":888,"num_write_kb":56832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1],"acting":[9,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.18","version":"110'4250","reported_seq":8834,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T11:24:11.028727+0000","last_change":"2024-09-13T11:24:07.089953+0000","last_active":"2024-09-13T11:24:11.028727+0000","last_peered":"2024-09-13T11:24:11.028727+0000","last_clean":"2024-09-13T11:24:11.028727+0000","last_became_active":"2024-09-13T11:23:48.926014+0000","last_became_peered":"2024-09-13T11:23:48.926014+0000","last_unstale":"2024-09-13T11:24:11.028727+0000","last_undegraded":"2024-09-13T11:24:11.028727+0000","last_fullsized":"2024-09-13T11:24:11.028727+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:22:24.896055+0000","objects_scrubbed":0,"log_size":4250,"log_dups_size":0,"ondisk_log_size":4250,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-13T11:24:12.084771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":188940288,"num_objects":2883,"num_object_clones":0,"num_object_copies":5766,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2883,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2883,"num_write_kb":184512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1458,"num_bytes_recovered":95551488,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0],"acting":[10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.17","version":"110'4131","reported_seq":8179,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T11:24:11.028948+0000","last_change":"2024-09-13T11:23:49.349241+0000","last_active":"2024-09-13T11:24:11.028948+0000","last_peered":"2024-09-13T11:24:11.028948+0000","last_clean":"2024-09-13T11:24:11.028948+0000","last_became_active":"2024-09-13T11:23:42.935601+0000","last_became_peered":"2024-09-13T11:23:42.935601+0000","last_unstale":"2024-09-13T11:24:11.028948+0000","last_undegraded":"2024-09-13T11:24:11.028948+0000","last_fullsized":"2024-09-13T11:24:11.028948+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":5,"last_scrub":"92'3359","last_scrub_stamp":"2024-09-13T11:23:49.349167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:49.349167+0000","objects_scrubbed":2004,"log_size":4131,"log_dups_size":0,"ondisk_log_size":4131,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:24:59.401302+0000","scrub_duration":363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":182452224,"num_objects":2784,"num_object_clones":0,"num_object_copies":5568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2784,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":780,"num_write_kb":49920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3],"acting":[10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"112'4471","reported_seq":9150,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-09-13T11:24:11.998728+0000","last_change":"2024-09-13T11:23:46.510544+0000","last_active":"2024-09-13T11:24:11.998728+0000","last_peered":"2024-09-13T11:24:11.998728+0000","last_clean":"2024-09-13T11:24:11.998728+0000","last_became_active":"2024-09-13T11:23:46.105237+0000","last_became_peered":"2024-09-13T11:23:46.105237+0000","last_unstale":"2024-09-13T11:24:11.998728+0000","last_undegraded":"2024-09-13T11:24:11.998728+0000","last_fullsized":"2024-09-13T11:24:11.998728+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":5,"last_scrub":"89'3478","last_scrub_stamp":"2024-09-13T11:23:46.510384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:46.510384+0000","objects_scrubbed":2054,"log_size":4471,"log_dups_size":0,"ondisk_log_size":4471,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:09.149878+0000","scrub_duration":363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":199884800,"num_objects":3050,"num_object_clones":0,"num_object_copies":6100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3050,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":996,"num_write_kb":63744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,15],"acting":[11,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.15","version":"111'4187","reported_seq":8623,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T11:24:11.010536+0000","last_change":"2024-09-13T11:23:48.332790+0000","last_active":"2024-09-13T11:24:11.010536+0000","last_peered":"2024-09-13T11:24:11.010536+0000","last_clean":"2024-09-13T11:24:11.010536+0000","last_became_active":"2024-09-13T11:23:42.931149+0000","last_became_peered":"2024-09-13T11:23:42.931149+0000","last_unstale":"2024-09-13T11:24:11.010536+0000","last_undegraded":"2024-09-13T11:24:11.010536+0000","last_fullsized":"2024-09-13T11:24:11.010536+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":5,"last_scrub":"91'3384","last_scrub_stamp":"2024-09-13T11:23:48.332600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:48.332600+0000","objects_scrubbed":2032,"log_size":4187,"log_dups_size":0,"ondisk_log_size":4187,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:24:58.239508+0000","scrub_duration":235,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":185860096,"num_objects":2836,"num_object_clones":0,"num_object_copies":5672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2836,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":804,"num_write_kb":51456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,0],"acting":[14,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"3.14","version":"110'4212","reported_seq":8467,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T11:24:10.043578+0000","last_change":"2024-09-13T11:23:48.697100+0000","last_active":"2024-09-13T11:24:10.043578+0000","last_peered":"2024-09-13T11:24:10.043578+0000","last_clean":"2024-09-13T11:24:10.043578+0000","last_became_active":"2024-09-13T11:23:41.920989+0000","last_became_peered":"2024-09-13T11:23:41.920989+0000","last_unstale":"2024-09-13T11:24:10.043578+0000","last_undegraded":"2024-09-13T11:24:10.043578+0000","last_fullsized":"2024-09-13T11:24:10.043578+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":5,"last_scrub":"91'3371","last_scrub_stamp":"2024-09-13T11:23:48.696960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:48.696960+0000","objects_scrubbed":1935,"log_size":4212,"log_dups_size":0,"ondisk_log_size":4212,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:02.861530+0000","scrub_duration":735,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":182583296,"num_objects":2786,"num_object_clones":0,"num_object_copies":5572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2786,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":851,"num_write_kb":54464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,14],"acting":[3,14],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":185,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T11:24:10.999083+0000","last_change":"2024-09-13T11:22:16.889130+0000","last_active":"2024-09-13T11:24:10.999083+0000","last_peered":"2024-09-13T11:24:10.999083+0000","last_clean":"2024-09-13T11:24:10.999083+0000","last_became_active":"2024-09-13T11:22:14.192897+0000","last_became_peered":"2024-09-13T11:22:14.192897+0000","last_unstale":"2024-09-13T11:24:10.999083+0000","last_undegraded":"2024-09-13T11:24:10.999083+0000","last_fullsized":"2024-09-13T11:24:10.999083+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:22:13.113563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00084655899999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,10],"acting":[15,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"3.6","version":"112'4370","reported_seq":4575,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-09-13T11:24:12.047558+0000","last_change":"2024-09-13T11:23:22.656608+0000","last_active":"2024-09-13T11:24:12.047558+0000","last_peered":"2024-09-13T11:24:12.047558+0000","last_clean":"2024-09-13T11:24:12.047558+0000","last_became_active":"2024-09-13T11:23:19.939432+0000","last_became_peered":"2024-09-13T11:23:19.939432+0000","last_unstale":"2024-09-13T11:24:12.047558+0000","last_undegraded":"2024-09-13T11:24:12.047558+0000","last_fullsized":"2024-09-13T11:24:12.047558+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'2909","last_scrub_stamp":"2024-09-13T11:23:22.656323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:22.656323+0000","objects_scrubbed":1487,"log_size":4370,"log_dups_size":0,"ondisk_log_size":4370,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:24:31.905185+0000","scrub_duration":323,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":193200128,"num_objects":2948,"num_object_clones":0,"num_object_copies":5896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2948,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1461,"num_write_kb":93504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,14],"acting":[13,14],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":194,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-09-13T11:24:11.920722+0000","last_change":"2024-09-13T11:23:21.875855+0000","last_active":"2024-09-13T11:24:11.920722+0000","last_peered":"2024-09-13T11:24:11.920722+0000","last_clean":"2024-09-13T11:24:11.920722+0000","last_became_active":"2024-09-13T11:22:14.211923+0000","last_became_peered":"2024-09-13T11:22:14.211923+0000","last_unstale":"2024-09-13T11:24:11.920722+0000","last_undegraded":"2024-09-13T11:24:11.920722+0000","last_fullsized":"2024-09-13T11:24:11.920722+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:23:21.875683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:23:21.875683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:24:40.678136+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0011689529999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0],"acting":[9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"112'4229","reported_seq":4435,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-09-13T11:24:12.069569+0000","last_change":"2024-09-13T11:23:20.443418+0000","last_active":"2024-09-13T11:24:12.069569+0000","last_peered":"2024-09-13T11:24:12.069569+0000","last_clean":"2024-09-13T11:24:12.069569+0000","last_became_active":"2024-09-13T11:23:19.927680+0000","last_became_peered":"2024-09-13T11:23:19.927680+0000","last_unstale":"2024-09-13T11:24:12.069569+0000","last_undegraded":"2024-09-13T11:24:12.069569+0000","last_fullsized":"2024-09-13T11:24:12.069569+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'2736","last_scrub_stamp":"2024-09-13T11:23:20.443329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:20.443329+0000","objects_scrubbed":1369,"log_size":4229,"log_dups_size":0,"ondisk_log_size":4229,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:24:48.147625+0000","scrub_duration":183,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":187760640,"num_objects":2865,"num_object_clones":0,"num_object_copies":5730,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2865,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1496,"num_write_kb":95744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,8],"acting":[13,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":193,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T11:24:11.036462+0000","last_change":"2024-09-13T11:23:59.530794+0000","last_active":"2024-09-13T11:24:11.036462+0000","last_peered":"2024-09-13T11:24:11.036462+0000","last_clean":"2024-09-13T11:24:11.036462+0000","last_became_active":"2024-09-13T11:22:14.581458+0000","last_became_peered":"2024-09-13T11:22:14.581458+0000","last_unstale":"2024-09-13T11:24:11.036462+0000","last_undegraded":"2024-09-13T11:24:11.036462+0000","last_fullsized":"2024-09-13T11:24:11.036462+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:23:59.530670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:23:59.530670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:02.655631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001026047,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5],"acting":[8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"110'4190","reported_seq":4372,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T11:24:10.320117+0000","last_change":"2024-09-13T11:23:20.254213+0000","last_active":"2024-09-13T11:24:10.320117+0000","last_peered":"2024-09-13T11:24:10.320117+0000","last_clean":"2024-09-13T11:24:10.320117+0000","last_became_active":"2024-09-13T11:23:19.925399+0000","last_became_peered":"2024-09-13T11:23:19.925399+0000","last_unstale":"2024-09-13T11:24:10.320117+0000","last_undegraded":"2024-09-13T11:24:10.320117+0000","last_fullsized":"2024-09-13T11:24:10.320117+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'2755","last_scrub_stamp":"2024-09-13T11:23:20.254124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:20.254124+0000","objects_scrubbed":1437,"log_size":4190,"log_dups_size":0,"ondisk_log_size":4190,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:24:49.993543+0000","scrub_duration":187,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":188547072,"num_objects":2877,"num_object_clones":0,"num_object_copies":5754,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2877,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1440,"num_write_kb":92160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,12],"acting":[1,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":179,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T11:24:08.271026+0000","last_change":"2024-09-13T11:24:08.271026+0000","last_active":"2024-09-13T11:24:08.271026+0000","last_peered":"2024-09-13T11:24:08.271026+0000","last_clean":"2024-09-13T11:24:08.271026+0000","last_became_active":"2024-09-13T11:22:47.719636+0000","last_became_peered":"2024-09-13T11:22:47.719636+0000","last_unstale":"2024-09-13T11:24:08.271026+0000","last_undegraded":"2024-09-13T11:24:08.271026+0000","last_fullsized":"2024-09-13T11:24:08.271026+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:24:08.270972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:24:08.270972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:36.805605+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00087476000000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,9],"acting":[1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"111'4202","reported_seq":4406,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T11:24:11.020631+0000","last_change":"2024-09-13T11:23:20.686446+0000","last_active":"2024-09-13T11:24:11.020631+0000","last_peered":"2024-09-13T11:24:11.020631+0000","last_clean":"2024-09-13T11:24:11.020631+0000","last_became_active":"2024-09-13T11:23:19.939695+0000","last_became_peered":"2024-09-13T11:23:19.939695+0000","last_unstale":"2024-09-13T11:24:11.020631+0000","last_undegraded":"2024-09-13T11:24:11.020631+0000","last_fullsized":"2024-09-13T11:24:11.020631+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'2739","last_scrub_stamp":"2024-09-13T11:23:20.686358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:20.686358+0000","objects_scrubbed":1386,"log_size":4202,"log_dups_size":0,"ondisk_log_size":4202,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:24:38.891737+0000","scrub_duration":271,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":186777600,"num_objects":2850,"num_object_clones":0,"num_object_copies":5700,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2850,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1464,"num_write_kb":93696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,13],"acting":[15,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"3.3","version":"111'3639","reported_seq":3866,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T11:24:11.028897+0000","last_change":"2024-09-13T11:23:32.147115+0000","last_active":"2024-09-13T11:24:11.028897+0000","last_peered":"2024-09-13T11:24:11.028897+0000","last_clean":"2024-09-13T11:24:11.028897+0000","last_became_active":"2024-09-13T11:23:31.930448+0000","last_became_peered":"2024-09-13T11:23:31.930448+0000","last_unstale":"2024-09-13T11:24:11.028897+0000","last_undegraded":"2024-09-13T11:24:11.028897+0000","last_fullsized":"2024-09-13T11:24:11.028897+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'3122","last_scrub_stamp":"2024-09-13T11:23:32.147021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:32.147021+0000","objects_scrubbed":876,"log_size":3639,"log_dups_size":0,"ondisk_log_size":3639,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:24:48.175541+0000","scrub_duration":99,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":91291648,"num_objects":1393,"num_object_clones":0,"num_object_copies":2786,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1393,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":517,"num_write_kb":33088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,7],"acting":[10,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"28'2","reported_seq":123,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T11:24:11.011218+0000","last_change":"2024-09-13T11:24:00.996197+0000","last_active":"2024-09-13T11:24:11.011218+0000","last_peered":"2024-09-13T11:24:11.011218+0000","last_clean":"2024-09-13T11:24:11.011218+0000","last_became_active":"2024-09-13T11:23:12.546348+0000","last_became_peered":"2024-09-13T11:23:12.546348+0000","last_unstale":"2024-09-13T11:24:11.011218+0000","last_undegraded":"2024-09-13T11:24:11.011218+0000","last_fullsized":"2024-09-13T11:24:11.011218+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2","last_scrub_stamp":"2024-09-13T11:24:00.996028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:24:00.996028+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:12.111097+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0025826730000000002,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8],"acting":[2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"111'3662","reported_seq":3887,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T11:24:11.057441+0000","last_change":"2024-09-13T11:23:32.749651+0000","last_active":"2024-09-13T11:24:11.057441+0000","last_peered":"2024-09-13T11:24:11.057441+0000","last_clean":"2024-09-13T11:24:11.057441+0000","last_became_active":"2024-09-13T11:23:31.921388+0000","last_became_peered":"2024-09-13T11:23:31.921388+0000","last_unstale":"2024-09-13T11:24:11.057441+0000","last_undegraded":"2024-09-13T11:24:11.057441+0000","last_fullsized":"2024-09-13T11:24:11.057441+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'3106","last_scrub_stamp":"2024-09-13T11:23:32.749487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:32.749487+0000","objects_scrubbed":856,"log_size":3662,"log_dups_size":0,"ondisk_log_size":3662,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:24:34.709501+0000","scrub_duration":91,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92536832,"num_objects":1412,"num_object_clones":0,"num_object_copies":2824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1412,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":556,"num_write_kb":35584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,15],"acting":[12,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":195,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T11:24:11.003029+0000","last_change":"2024-09-13T11:22:47.298206+0000","last_active":"2024-09-13T11:24:11.003029+0000","last_peered":"2024-09-13T11:24:11.003029+0000","last_clean":"2024-09-13T11:24:11.003029+0000","last_became_active":"2024-09-13T11:22:47.297319+0000","last_became_peered":"2024-09-13T11:22:47.297319+0000","last_unstale":"2024-09-13T11:24:11.003029+0000","last_undegraded":"2024-09-13T11:24:11.003029+0000","last_fullsized":"2024-09-13T11:24:11.003029+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:22:13.113563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00081225399999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,1],"acting":[14,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"3.1","version":"112'3655","reported_seq":3882,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-09-13T11:24:12.029251+0000","last_change":"2024-09-13T11:23:33.604966+0000","last_active":"2024-09-13T11:24:12.029251+0000","last_peered":"2024-09-13T11:24:12.029251+0000","last_clean":"2024-09-13T11:24:12.029251+0000","last_became_active":"2024-09-13T11:23:31.926528+0000","last_became_peered":"2024-09-13T11:23:31.926528+0000","last_unstale":"2024-09-13T11:24:12.029251+0000","last_undegraded":"2024-09-13T11:24:12.029251+0000","last_fullsized":"2024-09-13T11:24:12.029251+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'3070","last_scrub_stamp":"2024-09-13T11:23:33.604792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:33.604792+0000","objects_scrubbed":842,"log_size":3655,"log_dups_size":0,"ondisk_log_size":3655,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:24:43.880038+0000","scrub_duration":119,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":93519872,"num_objects":1427,"num_object_clones":0,"num_object_copies":2854,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1427,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":585,"num_write_kb":37440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,11],"acting":[13,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":183,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T11:24:07.959659+0000","last_change":"2024-09-13T11:22:16.267999+0000","last_active":"2024-09-13T11:24:07.959659+0000","last_peered":"2024-09-13T11:24:07.959659+0000","last_clean":"2024-09-13T11:24:07.959659+0000","last_became_active":"2024-09-13T11:22:14.211435+0000","last_became_peered":"2024-09-13T11:22:14.211435+0000","last_unstale":"2024-09-13T11:24:07.959659+0000","last_undegraded":"2024-09-13T11:24:07.959659+0000","last_fullsized":"2024-09-13T11:24:07.959659+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:22:13.113563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:24:10.770081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010010360000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2","version":"110'3674","reported_seq":3900,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T11:24:11.003116+0000","last_change":"2024-09-13T11:23:32.977666+0000","last_active":"2024-09-13T11:24:11.003116+0000","last_peered":"2024-09-13T11:24:11.003116+0000","last_clean":"2024-09-13T11:24:11.003116+0000","last_became_active":"2024-09-13T11:23:31.919828+0000","last_became_peered":"2024-09-13T11:23:31.919828+0000","last_unstale":"2024-09-13T11:24:11.003116+0000","last_undegraded":"2024-09-13T11:24:11.003116+0000","last_fullsized":"2024-09-13T11:24:11.003116+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'3057","last_scrub_stamp":"2024-09-13T11:23:32.977559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:32.977559+0000","objects_scrubbed":841,"log_size":3674,"log_dups_size":0,"ondisk_log_size":3674,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:24:52.169339+0000","scrub_duration":151,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":95551488,"num_objects":1458,"num_object_clones":0,"num_object_copies":2916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1458,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":617,"num_write_kb":39488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,12],"acting":[14,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"2.3","version":"26'1","reported_seq":187,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T11:24:11.003014+0000","last_change":"2024-09-13T11:22:16.547691+0000","last_active":"2024-09-13T11:24:11.003014+0000","last_peered":"2024-09-13T11:24:11.003014+0000","last_clean":"2024-09-13T11:24:11.003014+0000","last_became_active":"2024-09-13T11:22:14.146507+0000","last_became_peered":"2024-09-13T11:22:14.146507+0000","last_unstale":"2024-09-13T11:24:11.003014+0000","last_undegraded":"2024-09-13T11:24:11.003014+0000","last_fullsized":"2024-09-13T11:24:11.003014+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:22:13.113563+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00085610999999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,15],"acting":[14,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"1.0","version":"26'112","reported_seq":327,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T11:24:10.981430+0000","last_change":"2024-09-13T11:23:49.325249+0000","last_active":"2024-09-13T11:24:10.981430+0000","last_peered":"2024-09-13T11:24:10.981430+0000","last_clean":"2024-09-13T11:24:10.981430+0000","last_became_active":"2024-09-13T11:22:12.129721+0000","last_became_peered":"2024-09-13T11:22:12.129721+0000","last_unstale":"2024-09-13T11:24:10.981430+0000","last_undegraded":"2024-09-13T11:24:10.981430+0000","last_fullsized":"2024-09-13T11:24:10.981430+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"26'112","last_scrub_stamp":"2024-09-13T11:23:49.325175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:11.098388+0000","last_clean_scrub_stamp":"2024-09-13T11:23:49.325175+0000","objects_scrubbed":2,"log_size":112,"log_dups_size":0,"ondisk_log_size":112,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:14.422287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.8","version":"110'4197","reported_seq":4402,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T11:24:10.981305+0000","last_change":"2024-09-13T11:23:20.644952+0000","last_active":"2024-09-13T11:24:10.981305+0000","last_peered":"2024-09-13T11:24:10.981305+0000","last_clean":"2024-09-13T11:24:10.981305+0000","last_became_active":"2024-09-13T11:23:19.925802+0000","last_became_peered":"2024-09-13T11:23:19.925802+0000","last_unstale":"2024-09-13T11:24:10.981305+0000","last_undegraded":"2024-09-13T11:24:10.981305+0000","last_fullsized":"2024-09-13T11:24:10.981305+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'2771","last_scrub_stamp":"2024-09-13T11:23:20.644747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:20.644747+0000","objects_scrubbed":1356,"log_size":4197,"log_dups_size":0,"ondisk_log_size":4197,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:24:38.144523+0000","scrub_duration":411,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":183107584,"num_objects":2794,"num_object_clones":0,"num_object_copies":5588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2794,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1438,"num_write_kb":92032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.9","version":"111'4275","reported_seq":4478,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T11:24:11.080932+0000","last_change":"2024-09-13T11:23:22.033678+0000","last_active":"2024-09-13T11:24:11.080932+0000","last_peered":"2024-09-13T11:24:11.080932+0000","last_clean":"2024-09-13T11:24:11.080932+0000","last_became_active":"2024-09-13T11:23:19.935114+0000","last_became_peered":"2024-09-13T11:23:19.935114+0000","last_unstale":"2024-09-13T11:24:11.080932+0000","last_undegraded":"2024-09-13T11:24:11.080932+0000","last_fullsized":"2024-09-13T11:24:11.080932+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'2808","last_scrub_stamp":"2024-09-13T11:23:22.033572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:22.033572+0000","objects_scrubbed":1471,"log_size":4275,"log_dups_size":0,"ondisk_log_size":4275,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:24:46.595859+0000","scrub_duration":287,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":192544768,"num_objects":2938,"num_object_clones":0,"num_object_copies":5876,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2938,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1467,"num_write_kb":93888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,2],"acting":[12,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[]},{"pgid":"3.a","version":"110'4531","reported_seq":4737,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T11:24:10.020132+0000","last_change":"2024-09-13T11:23:32.526150+0000","last_active":"2024-09-13T11:24:10.020132+0000","last_peered":"2024-09-13T11:24:10.020132+0000","last_clean":"2024-09-13T11:24:10.020132+0000","last_became_active":"2024-09-13T11:23:31.934170+0000","last_became_peered":"2024-09-13T11:23:31.934170+0000","last_unstale":"2024-09-13T11:24:10.020132+0000","last_undegraded":"2024-09-13T11:24:10.020132+0000","last_fullsized":"2024-09-13T11:24:10.020132+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'3418","last_scrub_stamp":"2024-09-13T11:23:32.526057+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:32.526057+0000","objects_scrubbed":1774,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:24:34.770161+0000","scrub_duration":507,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":189202432,"num_objects":2887,"num_object_clones":0,"num_object_copies":5774,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2887,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1113,"num_write_kb":71232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.b","version":"112'4627","reported_seq":4836,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-09-13T11:24:12.015276+0000","last_change":"2024-09-13T11:23:32.681343+0000","last_active":"2024-09-13T11:24:12.015276+0000","last_peered":"2024-09-13T11:24:12.015276+0000","last_clean":"2024-09-13T11:24:12.015276+0000","last_became_active":"2024-09-13T11:23:31.923415+0000","last_became_peered":"2024-09-13T11:23:31.923415+0000","last_unstale":"2024-09-13T11:24:12.015276+0000","last_undegraded":"2024-09-13T11:24:12.015276+0000","last_fullsized":"2024-09-13T11:24:12.015276+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'3471","last_scrub_stamp":"2024-09-13T11:23:32.681257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:32.681257+0000","objects_scrubbed":1720,"log_size":4627,"log_dups_size":0,"ondisk_log_size":4627,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:24:58.857812+0000","scrub_duration":199,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":188481536,"num_objects":2876,"num_object_clones":0,"num_object_copies":5752,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2876,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1156,"num_write_kb":73984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,1],"acting":[13,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"3.c","version":"112'4619","reported_seq":4827,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-09-13T11:24:11.995484+0000","last_change":"2024-09-13T11:23:32.524870+0000","last_active":"2024-09-13T11:24:11.995484+0000","last_peered":"2024-09-13T11:24:11.995484+0000","last_clean":"2024-09-13T11:24:11.995484+0000","last_became_active":"2024-09-13T11:23:31.922620+0000","last_became_peered":"2024-09-13T11:23:31.922620+0000","last_unstale":"2024-09-13T11:24:11.995484+0000","last_undegraded":"2024-09-13T11:24:11.995484+0000","last_fullsized":"2024-09-13T11:24:11.995484+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'3463","last_scrub_stamp":"2024-09-13T11:23:32.524776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:32.524776+0000","objects_scrubbed":1741,"log_size":4619,"log_dups_size":0,"ondisk_log_size":4619,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:24:37.745747+0000","scrub_duration":235,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":189857792,"num_objects":2897,"num_object_clones":0,"num_object_copies":5794,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2897,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1156,"num_write_kb":73984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,13],"acting":[11,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.d","version":"111'4567","reported_seq":4773,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T11:24:11.080180+0000","last_change":"2024-09-13T11:23:32.776835+0000","last_active":"2024-09-13T11:24:11.080180+0000","last_peered":"2024-09-13T11:24:11.080180+0000","last_clean":"2024-09-13T11:24:11.080180+0000","last_became_active":"2024-09-13T11:23:31.939246+0000","last_became_peered":"2024-09-13T11:23:31.939246+0000","last_unstale":"2024-09-13T11:24:11.080180+0000","last_undegraded":"2024-09-13T11:24:11.080180+0000","last_fullsized":"2024-09-13T11:24:11.080180+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'3395","last_scrub_stamp":"2024-09-13T11:23:32.776747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:32.776747+0000","objects_scrubbed":1704,"log_size":4567,"log_dups_size":0,"ondisk_log_size":4567,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:24:56.402862+0000","scrub_duration":179,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":188481536,"num_objects":2876,"num_object_clones":0,"num_object_copies":5752,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2876,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1172,"num_write_kb":75008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.e","version":"111'4643","reported_seq":9778,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T11:24:11.091345+0000","last_change":"2024-09-13T11:23:32.368856+0000","last_active":"2024-09-13T11:24:11.091345+0000","last_peered":"2024-09-13T11:24:11.091345+0000","last_clean":"2024-09-13T11:24:11.091345+0000","last_became_active":"2024-09-13T11:23:31.925869+0000","last_became_peered":"2024-09-13T11:23:31.925869+0000","last_unstale":"2024-09-13T11:24:11.091345+0000","last_undegraded":"2024-09-13T11:24:11.091345+0000","last_fullsized":"2024-09-13T11:24:11.091345+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'3527","last_scrub_stamp":"2024-09-13T11:23:32.368727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:32.368727+0000","objects_scrubbed":1774,"log_size":4643,"log_dups_size":0,"ondisk_log_size":4643,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:24:57.811779+0000","scrub_duration":219,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":189464576,"num_objects":2891,"num_object_clones":0,"num_object_copies":5782,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2891,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1117,"num_write_kb":71488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"112'4589","reported_seq":4797,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-09-13T11:24:12.056478+0000","last_change":"2024-09-13T11:23:33.492630+0000","last_active":"2024-09-13T11:24:12.056478+0000","last_peered":"2024-09-13T11:24:12.056478+0000","last_clean":"2024-09-13T11:24:12.056478+0000","last_became_active":"2024-09-13T11:23:31.935239+0000","last_became_peered":"2024-09-13T11:23:31.935239+0000","last_unstale":"2024-09-13T11:24:12.056478+0000","last_undegraded":"2024-09-13T11:24:12.056478+0000","last_fullsized":"2024-09-13T11:24:12.056478+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'3420","last_scrub_stamp":"2024-09-13T11:23:33.492488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:33.492488+0000","objects_scrubbed":1776,"log_size":4589,"log_dups_size":0,"ondisk_log_size":4589,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:24:44.851261+0000","scrub_duration":215,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":193003520,"num_objects":2945,"num_object_clones":0,"num_object_copies":5890,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2945,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1169,"num_write_kb":74816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,12],"acting":[11,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.10","version":"111'4361","reported_seq":8722,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T11:24:11.004908+0000","last_change":"2024-09-13T11:23:48.519654+0000","last_active":"2024-09-13T11:24:11.004908+0000","last_peered":"2024-09-13T11:24:11.004908+0000","last_clean":"2024-09-13T11:24:11.004908+0000","last_became_active":"2024-09-13T11:23:41.931183+0000","last_became_peered":"2024-09-13T11:23:41.931183+0000","last_unstale":"2024-09-13T11:24:11.004908+0000","last_undegraded":"2024-09-13T11:24:11.004908+0000","last_fullsized":"2024-09-13T11:24:11.004908+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":5,"last_scrub":"91'3468","last_scrub_stamp":"2024-09-13T11:23:48.519574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:48.519574+0000","objects_scrubbed":2031,"log_size":4361,"log_dups_size":0,"ondisk_log_size":4361,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:24:57.342213+0000","scrub_duration":215,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":191758336,"num_objects":2926,"num_object_clones":0,"num_object_copies":5852,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2926,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":895,"num_write_kb":57280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.11","version":"111'4217","reported_seq":7357,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T11:24:11.043147+0000","last_change":"2024-09-13T11:23:46.102466+0000","last_active":"2024-09-13T11:24:11.043147+0000","last_peered":"2024-09-13T11:24:11.043147+0000","last_clean":"2024-09-13T11:24:11.043147+0000","last_became_active":"2024-09-13T11:23:39.063172+0000","last_became_peered":"2024-09-13T11:23:39.063172+0000","last_unstale":"2024-09-13T11:24:11.043147+0000","last_undegraded":"2024-09-13T11:24:11.043147+0000","last_fullsized":"2024-09-13T11:24:11.043147+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":5,"last_scrub":"88'3345","last_scrub_stamp":"2024-09-13T11:23:46.101973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:46.101973+0000","objects_scrubbed":2011,"log_size":4217,"log_dups_size":0,"ondisk_log_size":4217,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:08.671350+0000","scrub_duration":955,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":189202432,"num_objects":2887,"num_object_clones":0,"num_object_copies":5774,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2887,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":876,"num_write_kb":56064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11],"acting":[4,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.12","version":"111'4218","reported_seq":7282,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T11:24:11.003159+0000","last_change":"2024-09-13T11:23:35.111409+0000","last_active":"2024-09-13T11:24:11.003159+0000","last_peered":"2024-09-13T11:24:11.003159+0000","last_clean":"2024-09-13T11:24:11.003159+0000","last_became_active":"2024-09-13T11:23:21.998877+0000","last_became_peered":"2024-09-13T11:23:21.998877+0000","last_unstale":"2024-09-13T11:24:11.003159+0000","last_undegraded":"2024-09-13T11:24:11.003159+0000","last_fullsized":"2024-09-13T11:24:11.003159+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'3109","last_scrub_stamp":"2024-09-13T11:23:35.111320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:35.111320+0000","objects_scrubbed":1732,"log_size":4218,"log_dups_size":0,"ondisk_log_size":4218,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:03.371433+0000","scrub_duration":239,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":186187776,"num_objects":2841,"num_object_clones":0,"num_object_copies":5682,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2841,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1109,"num_write_kb":70976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,7],"acting":[14,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"3.13","version":"112'4312","reported_seq":7379,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-09-13T11:24:12.059551+0000","last_change":"2024-09-13T11:23:47.565094+0000","last_active":"2024-09-13T11:24:12.059551+0000","last_peered":"2024-09-13T11:24:12.059551+0000","last_clean":"2024-09-13T11:24:12.059551+0000","last_became_active":"2024-09-13T11:23:41.082433+0000","last_became_peered":"2024-09-13T11:23:41.082433+0000","last_unstale":"2024-09-13T11:24:12.059551+0000","last_undegraded":"2024-09-13T11:24:12.059551+0000","last_fullsized":"2024-09-13T11:24:12.059551+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'3408","last_scrub_stamp":"2024-09-13T11:23:47.564931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:47.564931+0000","objects_scrubbed":1981,"log_size":4312,"log_dups_size":0,"ondisk_log_size":4312,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:04.258231+0000","scrub_duration":471,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":189857792,"num_objects":2897,"num_object_clones":0,"num_object_copies":5794,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2897,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":916,"num_write_kb":58624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7],"acting":[11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":36,"stat_sum":{"num_bytes":6023643136,"num_objects":91914,"num_object_clones":0,"num_object_copies":183828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":91914,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":37320,"num_write_kb":2388448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5754,"num_bytes_recovered":377094144,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":13574275072,"data_stored":13574275072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":152493,"ondisk_log_size":152493,"up":72,"acting":72,"num_store_stats":16},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":10},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":112,"ondisk_log_size":112,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":13,"up_from":23,"seq":98784247836,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1066984,"kb_used_data":1003864,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":93304856,"statfs":{"total":96636764160,"available":95544172544,"internally_reserved":0,"allocated":1027956736,"data_stored":1027464855,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[0,1,3,4,6,8,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":9,"up_from":23,"seq":98784247835,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":495128,"kb_used_data":455624,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":93876712,"statfs":{"total":96636764160,"available":96129753088,"internally_reserved":0,"allocated":466558976,"data_stored":466082552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":40434112},"hb_peers":[0,1,2,8,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":40,"seq":171798691858,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1363564,"kb_used_data":1299612,"kb_used_omap":4,"kb_used_meta":63931,"kb_avail":93008276,"statfs":{"total":96636764160,"available":95240474624,"internally_reserved":0,"allocated":1330802688,"data_stored":1330300776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4433,"internal_metadata":65466031},"hb_peers":[0,2,3,6,8,9,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":21,"seq":90194313243,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1077112,"kb_used_data":1001512,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":93294728,"statfs":{"total":96636764160,"available":95533801472,"internally_reserved":0,"allocated":1025548288,"data_stored":1025039643,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":77396423},"hb_peers":[0,2,4,7,9,10,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,1,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":21,"seq":90194313246,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303284,"kb_used_data":276260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94068556,"statfs":{"total":96636764160,"available":96326201344,"internally_reserved":0,"allocated":282890240,"data_stored":282382299,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,6,8,9,10,11,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":22,"seq":94489280538,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1220588,"kb_used_data":1153308,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":93151252,"statfs":{"total":96636764160,"available":95386882048,"internally_reserved":0,"allocated":1180987392,"data_stored":1180548271,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[1,3,4,5,8,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":22,"seq":94489280539,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":461856,"kb_used_data":422352,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":93909984,"statfs":{"total":96636764160,"available":96163823616,"internally_reserved":0,"allocated":432488448,"data_stored":432004521,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,3,7,8,9,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":15,"up_from":22,"seq":94489280539,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":979620,"kb_used_data":904020,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":93392220,"statfs":{"total":96636764160,"available":95633633280,"internally_reserved":0,"allocated":925716480,"data_stored":925285333,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":77396422},"hb_peers":[0,3,4,5,6,7,10,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":22,"seq":94489280539,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":587324,"kb_used_data":538092,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":93784516,"statfs":{"total":96636764160,"available":96035344384,"internally_reserved":0,"allocated":551006208,"data_stored":550495788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[3,5,7,9,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":22,"seq":94489280539,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":453512,"kb_used_data":414008,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":93918328,"statfs":{"total":96636764160,"available":96172367872,"internally_reserved":0,"allocated":423944192,"data_stored":423479893,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40434118},"hb_peers":[0,1,3,5,7,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":22,"seq":94489280539,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1279808,"kb_used_data":1208368,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":93092032,"statfs":{"total":96636764160,"available":95326240768,"internally_reserved":0,"allocated":1237368832,"data_stored":1236910392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":73136586},"hb_peers":[0,3,5,6,8,9,10,11,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":22,"seq":94489280539,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":479740,"kb_used_data":448556,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93892100,"statfs":{"total":96636764160,"available":96145510400,"internally_reserved":0,"allocated":459321344,"data_stored":458806681,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,5,7,9,10,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":22,"seq":94489280538,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":719056,"kb_used_data":673984,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":93652784,"statfs":{"total":96636764160,"available":95900450816,"internally_reserved":0,"allocated":690159616,"data_stored":689694148,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":46135744},"hb_peers":[0,1,3,6,7,9,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":11,"up_from":22,"seq":94489280540,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1276712,"kb_used_data":1209432,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":93095128,"statfs":{"total":96636764160,"available":95329411072,"internally_reserved":0,"allocated":1238458368,"data_stored":1237970110,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":68876740},"hb_peers":[0,2,4,7,8,10,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":12,"up_from":22,"seq":94489280539,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1085344,"kb_used_data":1018064,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":93286496,"statfs":{"total":96636764160,"available":95525371904,"internally_reserved":0,"allocated":1042497536,"data_stored":1042010499,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,1,2,3,5,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":14,"up_from":22,"seq":94489280539,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1251820,"kb_used_data":1176220,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":93120020,"statfs":{"total":96636764160,"available":95354900480,"internally_reserved":0,"allocated":1204449280,"data_stored":1203955694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,1,5,6,7,8,9,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1189871616,"data_stored":1189871616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1328545792,"data_stored":1328545792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":435093504,"data_stored":435093504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1024000000,"data_stored":1024000000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":558497792,"data_stored":558497792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":285802496,"data_stored":285802496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":425328640,"data_stored":425328640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1238433792,"data_stored":1238433792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":457113600,"data_stored":457113600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":468123648,"data_stored":468123648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":688979968,"data_stored":688979968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":1249443840,"data_stored":1249443840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":1047789568,"data_stored":1047789568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":1043136512,"data_stored":1043136512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":1203437568,"data_stored":1203437568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":930676736,"data_stored":930676736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T11:24:14.488 INFO:tasks.ceph.ceph_manager.ceph:recovered! 2024-09-13T11:24:14.488 INFO:tasks.thrashosds.thrasher:choose_action: min_in 2 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-13T11:24:14.488 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-13T11:24:14.514 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-13T11:24:14.514 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:24:14.532 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-09-13T11:24:14.532 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-13T11:24:14.527+0000 7f6a68775640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26307) UID: 0 2024-09-13T11:24:14.554 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T11:24:14.605 INFO:tasks.ceph.osd.14.smithi162.stderr:2024-09-13T11:24:14.600+0000 7f235c0db640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26319) UID: 0 2024-09-13T11:24:14.668 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-13T11:24:14.668 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:24:14.685 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_ops_in_flight 2024-09-13T11:24:14.706 INFO:tasks.ceph.osd.5.smithi092.stderr:2024-09-13T11:24:14.703+0000 7f6f0b705640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26346) UID: 0 2024-09-13T11:24:14.737 INFO:teuthology.orchestra.run.smithi001.stdout:osd.0: {} 2024-09-13T11:24:14.737 INFO:teuthology.orchestra.run.smithi001.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T11:24:14.776 INFO:teuthology.orchestra.run.smithi001.stdout:osd.1: {} 2024-09-13T11:24:14.776 INFO:teuthology.orchestra.run.smithi001.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T11:24:14.799 INFO:teuthology.orchestra.run.smithi001.stdout:osd.2: {} 2024-09-13T11:24:14.799 INFO:teuthology.orchestra.run.smithi001.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T11:24:14.806 INFO:tasks.ceph.osd.15.smithi177.stderr:2024-09-13T11:24:14.803+0000 7f9d6ce77640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26578) UID: 0 2024-09-13T11:24:14.824 INFO:teuthology.orchestra.run.smithi001.stdout:osd.3: {} 2024-09-13T11:24:14.824 INFO:teuthology.orchestra.run.smithi001.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T11:24:14.851 INFO:teuthology.orchestra.run.smithi001.stdout:osd.4: {} 2024-09-13T11:24:14.851 INFO:teuthology.orchestra.run.smithi001.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T11:24:14.858 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-13T11:24:14.858 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:24:14.873 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_blocked_ops 2024-09-13T11:24:14.880 INFO:teuthology.orchestra.run.smithi001.stdout:osd.5: {} 2024-09-13T11:24:14.881 INFO:teuthology.orchestra.run.smithi001.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T11:24:14.899 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-13T11:24:14.899 INFO:teuthology.orchestra.run.smithi001.stderr:dumped all 2024-09-13T11:24:14.905 INFO:teuthology.orchestra.run.smithi001.stdout:osd.6: {} 2024-09-13T11:24:14.906 INFO:teuthology.orchestra.run.smithi001.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T11:24:14.907 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-13T11:24:14.899+0000 7f6a68775640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26307) UID: 0 2024-09-13T11:24:14.922 INFO:teuthology.orchestra.run.smithi001.stdout:{"pg_ready":true,"pg_map":{"version":161,"stamp":"2024-09-13T11:24:14.144540+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":6024233523,"num_objects":91918,"num_object_clones":0,"num_object_copies":183836,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":91918,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":37467,"num_write_kb":2391122,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5755,"num_bytes_recovered":377094163,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":152608,"ondisk_log_size":152608,"up":90,"acting":90,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":112,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":16,"kb":1509949440,"kb_used":14101452,"kb_used_data":13203276,"kb_used_omap":27,"kb_used_meta":897892,"kb_avail":1495847988,"statfs":{"total":1546188226560,"available":1531748339712,"internally_reserved":0,"allocated":13520154624,"data_stored":13512431455,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28358,"internal_metadata":919441722},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,2,1,1,3,2,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":450789376,"num_objects":6878,"num_object_clones":0,"num_object_copies":13756,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6878,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-1187,"num_write_kb":-75936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-5872,"num_bytes_recovered":-384794624,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001187"},"pg_stats":[{"pgid":"3.23","version":"110'3710","reported_seq":6446,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T11:24:10.829789+0000","last_change":"2024-09-13T11:23:48.133184+0000","last_active":"2024-09-13T11:24:10.829789+0000","last_peered":"2024-09-13T11:24:10.829789+0000","last_clean":"2024-09-13T11:24:10.829789+0000","last_became_active":"2024-09-13T11:23:47.926677+0000","last_became_peered":"2024-09-13T11:23:47.926677+0000","last_unstale":"2024-09-13T11:24:10.829789+0000","last_undegraded":"2024-09-13T11:24:10.829789+0000","last_fullsized":"2024-09-13T11:24:10.829789+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":6,"last_scrub":"91'3278","last_scrub_stamp":"2024-09-13T11:23:48.133084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:48.133084+0000","objects_scrubbed":1029,"log_size":3710,"log_dups_size":0,"ondisk_log_size":3710,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:12.874134+0000","scrub_duration":139,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":95813632,"num_objects":1462,"num_object_clones":0,"num_object_copies":2924,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1462,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":433,"num_write_kb":27712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9],"acting":[5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.22","version":"111'3684","reported_seq":5609,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T11:24:11.003707+0000","last_change":"2024-09-13T11:23:47.300446+0000","last_active":"2024-09-13T11:24:11.003707+0000","last_peered":"2024-09-13T11:24:11.003707+0000","last_clean":"2024-09-13T11:24:11.003707+0000","last_became_active":"2024-09-13T11:23:46.925318+0000","last_became_peered":"2024-09-13T11:23:46.925318+0000","last_unstale":"2024-09-13T11:24:11.003707+0000","last_undegraded":"2024-09-13T11:24:11.003707+0000","last_fullsized":"2024-09-13T11:24:11.003707+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":6,"last_scrub":"90'3246","last_scrub_stamp":"2024-09-13T11:23:47.300339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:47.300339+0000","objects_scrubbed":1033,"log_size":3684,"log_dups_size":0,"ondisk_log_size":3684,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:24:49.780485+0000","scrub_duration":203,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96534528,"num_objects":1473,"num_object_clones":0,"num_object_copies":2946,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1473,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":440,"num_write_kb":28160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,14],"acting":[0,14],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.21","version":"110'3622","reported_seq":6353,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T11:24:11.003655+0000","last_change":"2024-09-13T11:24:07.306104+0000","last_active":"2024-09-13T11:24:11.003655+0000","last_peered":"2024-09-13T11:24:11.003655+0000","last_clean":"2024-09-13T11:24:11.003655+0000","last_became_active":"2024-09-13T11:24:01.250308+0000","last_became_peered":"2024-09-13T11:24:01.250308+0000","last_unstale":"2024-09-13T11:24:11.003655+0000","last_undegraded":"2024-09-13T11:24:11.003655+0000","last_fullsized":"2024-09-13T11:24:11.003655+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":6,"last_scrub":"69'2784","last_scrub_stamp":"2024-09-13T11:23:21.595419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:21.595419+0000","objects_scrubbed":0,"log_size":3622,"log_dups_size":0,"ondisk_log_size":3622,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"user requested scrub scheduled @ 2024-09-13T11:24:12.261215+0000","scrub_duration":311,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":91095040,"num_objects":1390,"num_object_clones":0,"num_object_copies":2780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1390,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":697,"num_write_kb":44608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":836,"num_bytes_recovered":54788096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8],"acting":[0,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.20","version":"111'3700","reported_seq":5740,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T11:24:11.039984+0000","last_change":"2024-09-13T11:23:47.969503+0000","last_active":"2024-09-13T11:24:11.039984+0000","last_peered":"2024-09-13T11:24:11.039984+0000","last_clean":"2024-09-13T11:24:11.039984+0000","last_became_active":"2024-09-13T11:23:33.954946+0000","last_became_peered":"2024-09-13T11:23:33.954946+0000","last_unstale":"2024-09-13T11:24:11.039984+0000","last_undegraded":"2024-09-13T11:24:11.039984+0000","last_fullsized":"2024-09-13T11:24:11.039984+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":6,"last_scrub":"91'3270","last_scrub_stamp":"2024-09-13T11:23:47.969372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:47.969372+0000","objects_scrubbed":1061,"log_size":3700,"log_dups_size":0,"ondisk_log_size":3700,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:24:48.255263+0000","scrub_duration":1327,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":98041856,"num_objects":1496,"num_object_clones":0,"num_object_copies":2992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1496,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":435,"num_write_kb":27840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,0],"acting":[12,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[]},{"pgid":"3.1f","version":"110'4534","reported_seq":9711,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T11:24:11.028813+0000","last_change":"2024-09-13T11:24:08.388852+0000","last_active":"2024-09-13T11:24:11.028813+0000","last_peered":"2024-09-13T11:24:11.028813+0000","last_clean":"2024-09-13T11:24:11.028813+0000","last_became_active":"2024-09-13T11:24:06.967832+0000","last_became_peered":"2024-09-13T11:24:06.967832+0000","last_unstale":"2024-09-13T11:24:11.028813+0000","last_undegraded":"2024-09-13T11:24:11.028813+0000","last_fullsized":"2024-09-13T11:24:11.028813+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":5,"last_scrub":"110'4427","last_scrub_stamp":"2024-09-13T11:24:08.388782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:24:08.388782+0000","objects_scrubbed":2651,"log_size":4534,"log_dups_size":0,"ondisk_log_size":4534,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:37.901131+0000","scrub_duration":315,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":181207040,"num_objects":2765,"num_object_clones":0,"num_object_copies":5530,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2765,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":114,"num_write_kb":7296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,14],"acting":[10,14],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1e","version":"110'4666","reported_seq":15068,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T11:24:10.910459+0000","last_change":"2024-09-13T11:24:08.338485+0000","last_active":"2024-09-13T11:24:10.910459+0000","last_peered":"2024-09-13T11:24:10.910459+0000","last_clean":"2024-09-13T11:24:10.910459+0000","last_became_active":"2024-09-13T11:24:01.261448+0000","last_became_peered":"2024-09-13T11:24:01.261448+0000","last_unstale":"2024-09-13T11:24:10.910459+0000","last_undegraded":"2024-09-13T11:24:10.910459+0000","last_fullsized":"2024-09-13T11:24:10.910459+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":5,"last_scrub":"110'4537","last_scrub_stamp":"2024-09-13T11:24:08.338354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:24:08.338354+0000","objects_scrubbed":2760,"log_size":4666,"log_dups_size":0,"ondisk_log_size":4666,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:11.010072+0000","scrub_duration":367,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":189923328,"num_objects":2898,"num_object_clones":0,"num_object_copies":5796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2898,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":138,"num_write_kb":8832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"111'4502","reported_seq":9796,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T11:24:11.011338+0000","last_change":"2024-09-13T11:24:07.062288+0000","last_active":"2024-09-13T11:24:11.011338+0000","last_peered":"2024-09-13T11:24:11.011338+0000","last_clean":"2024-09-13T11:24:11.011338+0000","last_became_active":"2024-09-13T11:24:01.258225+0000","last_became_peered":"2024-09-13T11:24:01.258225+0000","last_unstale":"2024-09-13T11:24:11.011338+0000","last_undegraded":"2024-09-13T11:24:11.011338+0000","last_fullsized":"2024-09-13T11:24:11.011338+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:22:24.896055+0000","objects_scrubbed":0,"log_size":4502,"log_dups_size":0,"ondisk_log_size":4502,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-13T11:24:12.048254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":184221696,"num_objects":2811,"num_object_clones":0,"num_object_copies":5622,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2811,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2811,"num_write_kb":179904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1700,"num_bytes_recovered":111411200,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,15],"acting":[2,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1c","version":"112'4660","reported_seq":8352,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-09-13T11:24:12.063510+0000","last_change":"2024-09-13T11:24:00.529312+0000","last_active":"2024-09-13T11:24:12.063510+0000","last_peered":"2024-09-13T11:24:12.063510+0000","last_clean":"2024-09-13T11:24:12.063510+0000","last_became_active":"2024-09-13T11:23:33.989492+0000","last_became_peered":"2024-09-13T11:23:33.989492+0000","last_unstale":"2024-09-13T11:24:12.063510+0000","last_undegraded":"2024-09-13T11:24:12.063510+0000","last_fullsized":"2024-09-13T11:24:12.063510+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":5,"last_scrub":"102'4277","last_scrub_stamp":"2024-09-13T11:24:00.529243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:24:00.529243+0000","objects_scrubbed":2544,"log_size":4660,"log_dups_size":0,"ondisk_log_size":4660,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:15.595052+0000","scrub_duration":259,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":191954944,"num_objects":2929,"num_object_clones":0,"num_object_copies":5858,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2929,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":385,"num_write_kb":24640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7],"acting":[11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1b","version":"111'4594","reported_seq":10068,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T11:24:11.086712+0000","last_change":"2024-09-13T11:24:05.835103+0000","last_active":"2024-09-13T11:24:11.086712+0000","last_peered":"2024-09-13T11:24:11.086712+0000","last_clean":"2024-09-13T11:24:11.086712+0000","last_became_active":"2024-09-13T11:24:04.950821+0000","last_became_peered":"2024-09-13T11:24:04.950821+0000","last_unstale":"2024-09-13T11:24:11.086712+0000","last_undegraded":"2024-09-13T11:24:11.086712+0000","last_fullsized":"2024-09-13T11:24:11.086712+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:22:24.896055+0000","objects_scrubbed":0,"log_size":4594,"log_dups_size":0,"ondisk_log_size":4594,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":187662336,"num_objects":2864,"num_object_clones":0,"num_object_copies":5728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2864,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2864,"num_write_kb":183264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1760,"num_bytes_recovered":115343360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,15],"acting":[12,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[]},{"pgid":"3.1a","version":"111'4503","reported_seq":9676,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T11:24:11.023180+0000","last_change":"2024-09-13T11:24:09.556174+0000","last_active":"2024-09-13T11:24:11.023180+0000","last_peered":"2024-09-13T11:24:11.023180+0000","last_clean":"2024-09-13T11:24:11.023180+0000","last_became_active":"2024-09-13T11:24:03.935185+0000","last_became_peered":"2024-09-13T11:24:03.935185+0000","last_unstale":"2024-09-13T11:24:11.023180+0000","last_undegraded":"2024-09-13T11:24:11.023180+0000","last_fullsized":"2024-09-13T11:24:11.023180+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":5,"last_scrub":"110'4433","last_scrub_stamp":"2024-09-13T11:24:09.556022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:24:09.556022+0000","objects_scrubbed":2654,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:21.773175+0000","scrub_duration":387,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":179240960,"num_objects":2735,"num_object_clones":0,"num_object_copies":5470,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2735,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":81,"num_write_kb":5184,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9],"acting":[4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.19","version":"112'4290","reported_seq":8605,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-09-13T11:24:11.983431+0000","last_change":"2024-09-13T11:23:49.429522+0000","last_active":"2024-09-13T11:24:11.983431+0000","last_peered":"2024-09-13T11:24:11.983431+0000","last_clean":"2024-09-13T11:24:11.983431+0000","last_became_active":"2024-09-13T11:23:41.924941+0000","last_became_peered":"2024-09-13T11:23:41.924941+0000","last_unstale":"2024-09-13T11:24:11.983431+0000","last_undegraded":"2024-09-13T11:24:11.983431+0000","last_fullsized":"2024-09-13T11:24:11.983431+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":5,"last_scrub":"92'3415","last_scrub_stamp":"2024-09-13T11:23:49.429429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:49.429429+0000","objects_scrubbed":1979,"log_size":4290,"log_dups_size":0,"ondisk_log_size":4290,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:04.629542+0000","scrub_duration":587,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":187891712,"num_objects":2867,"num_object_clones":0,"num_object_copies":5734,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2867,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":888,"num_write_kb":56832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1],"acting":[9,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.18","version":"110'4250","reported_seq":8834,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T11:24:11.028727+0000","last_change":"2024-09-13T11:24:07.089953+0000","last_active":"2024-09-13T11:24:11.028727+0000","last_peered":"2024-09-13T11:24:11.028727+0000","last_clean":"2024-09-13T11:24:11.028727+0000","last_became_active":"2024-09-13T11:23:48.926014+0000","last_became_peered":"2024-09-13T11:23:48.926014+0000","last_unstale":"2024-09-13T11:24:11.028727+0000","last_undegraded":"2024-09-13T11:24:11.028727+0000","last_fullsized":"2024-09-13T11:24:11.028727+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:22:24.896055+0000","objects_scrubbed":0,"log_size":4250,"log_dups_size":0,"ondisk_log_size":4250,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-13T11:24:12.084771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":188940288,"num_objects":2883,"num_object_clones":0,"num_object_copies":5766,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2883,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2883,"num_write_kb":184512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1458,"num_bytes_recovered":95551488,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0],"acting":[10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.17","version":"110'4131","reported_seq":8179,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T11:24:11.028948+0000","last_change":"2024-09-13T11:23:49.349241+0000","last_active":"2024-09-13T11:24:11.028948+0000","last_peered":"2024-09-13T11:24:11.028948+0000","last_clean":"2024-09-13T11:24:11.028948+0000","last_became_active":"2024-09-13T11:23:42.935601+0000","last_became_peered":"2024-09-13T11:23:42.935601+0000","last_unstale":"2024-09-13T11:24:11.028948+0000","last_undegraded":"2024-09-13T11:24:11.028948+0000","last_fullsized":"2024-09-13T11:24:11.028948+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":5,"last_scrub":"92'3359","last_scrub_stamp":"2024-09-13T11:23:49.349167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:49.349167+0000","objects_scrubbed":2004,"log_size":4131,"log_dups_size":0,"ondisk_log_size":4131,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:24:59.401302+0000","scrub_duration":363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":182452224,"num_objects":2784,"num_object_clones":0,"num_object_copies":5568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2784,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":780,"num_write_kb":49920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3],"acting":[10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"112'4471","reported_seq":9150,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-09-13T11:24:11.998728+0000","last_change":"2024-09-13T11:23:46.510544+0000","last_active":"2024-09-13T11:24:11.998728+0000","last_peered":"2024-09-13T11:24:11.998728+0000","last_clean":"2024-09-13T11:24:11.998728+0000","last_became_active":"2024-09-13T11:23:46.105237+0000","last_became_peered":"2024-09-13T11:23:46.105237+0000","last_unstale":"2024-09-13T11:24:11.998728+0000","last_undegraded":"2024-09-13T11:24:11.998728+0000","last_fullsized":"2024-09-13T11:24:11.998728+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":5,"last_scrub":"89'3478","last_scrub_stamp":"2024-09-13T11:23:46.510384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:46.510384+0000","objects_scrubbed":2054,"log_size":4471,"log_dups_size":0,"ondisk_log_size":4471,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:09.149878+0000","scrub_duration":363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":199884800,"num_objects":3050,"num_object_clones":0,"num_object_copies":6100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3050,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":996,"num_write_kb":63744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,15],"acting":[11,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.15","version":"111'4187","reported_seq":8623,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T11:24:11.010536+0000","last_change":"2024-09-13T11:23:48.332790+0000","last_active":"2024-09-13T11:24:11.010536+0000","last_peered":"2024-09-13T11:24:11.010536+0000","last_clean":"2024-09-13T11:24:11.010536+0000","last_became_active":"2024-09-13T11:23:42.931149+0000","last_became_peered":"2024-09-13T11:23:42.931149+0000","last_unstale":"2024-09-13T11:24:11.010536+0000","last_undegraded":"2024-09-13T11:24:11.010536+0000","last_fullsized":"2024-09-13T11:24:11.010536+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":5,"last_scrub":"91'3384","last_scrub_stamp":"2024-09-13T11:23:48.332600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:48.332600+0000","objects_scrubbed":2032,"log_size":4187,"log_dups_size":0,"ondisk_log_size":4187,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:24:58.239508+0000","scrub_duration":235,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":185860096,"num_objects":2836,"num_object_clones":0,"num_object_copies":5672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2836,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":804,"num_write_kb":51456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,0],"acting":[14,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"3.14","version":"110'4212","reported_seq":8467,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T11:24:10.043578+0000","last_change":"2024-09-13T11:23:48.697100+0000","last_active":"2024-09-13T11:24:10.043578+0000","last_peered":"2024-09-13T11:24:10.043578+0000","last_clean":"2024-09-13T11:24:10.043578+0000","last_became_active":"2024-09-13T11:23:41.920989+0000","last_became_peered":"2024-09-13T11:23:41.920989+0000","last_unstale":"2024-09-13T11:24:10.043578+0000","last_undegraded":"2024-09-13T11:24:10.043578+0000","last_fullsized":"2024-09-13T11:24:10.043578+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":5,"last_scrub":"91'3371","last_scrub_stamp":"2024-09-13T11:23:48.696960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:48.696960+0000","objects_scrubbed":1935,"log_size":4212,"log_dups_size":0,"ondisk_log_size":4212,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:02.861530+0000","scrub_duration":735,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":182583296,"num_objects":2786,"num_object_clones":0,"num_object_copies":5572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2786,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":851,"num_write_kb":54464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,14],"acting":[3,14],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":185,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T11:24:10.999083+0000","last_change":"2024-09-13T11:22:16.889130+0000","last_active":"2024-09-13T11:24:10.999083+0000","last_peered":"2024-09-13T11:24:10.999083+0000","last_clean":"2024-09-13T11:24:10.999083+0000","last_became_active":"2024-09-13T11:22:14.192897+0000","last_became_peered":"2024-09-13T11:22:14.192897+0000","last_unstale":"2024-09-13T11:24:10.999083+0000","last_undegraded":"2024-09-13T11:24:10.999083+0000","last_fullsized":"2024-09-13T11:24:10.999083+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:22:13.113563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00084655899999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,10],"acting":[15,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"3.6","version":"112'4370","reported_seq":4575,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-09-13T11:24:12.047558+0000","last_change":"2024-09-13T11:23:22.656608+0000","last_active":"2024-09-13T11:24:12.047558+0000","last_peered":"2024-09-13T11:24:12.047558+0000","last_clean":"2024-09-13T11:24:12.047558+0000","last_became_active":"2024-09-13T11:23:19.939432+0000","last_became_peered":"2024-09-13T11:23:19.939432+0000","last_unstale":"2024-09-13T11:24:12.047558+0000","last_undegraded":"2024-09-13T11:24:12.047558+0000","last_fullsized":"2024-09-13T11:24:12.047558+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'2909","last_scrub_stamp":"2024-09-13T11:23:22.656323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:22.656323+0000","objects_scrubbed":1487,"log_size":4370,"log_dups_size":0,"ondisk_log_size":4370,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:24:31.905185+0000","scrub_duration":323,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":193200128,"num_objects":2948,"num_object_clones":0,"num_object_copies":5896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2948,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1461,"num_write_kb":93504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,14],"acting":[13,14],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":194,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-09-13T11:24:11.920722+0000","last_change":"2024-09-13T11:23:21.875855+0000","last_active":"2024-09-13T11:24:11.920722+0000","last_peered":"2024-09-13T11:24:11.920722+0000","last_clean":"2024-09-13T11:24:11.920722+0000","last_became_active":"2024-09-13T11:22:14.211923+0000","last_became_peered":"2024-09-13T11:22:14.211923+0000","last_unstale":"2024-09-13T11:24:11.920722+0000","last_undegraded":"2024-09-13T11:24:11.920722+0000","last_fullsized":"2024-09-13T11:24:11.920722+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:23:21.875683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:23:21.875683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:24:40.678136+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0011689529999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0],"acting":[9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"112'4229","reported_seq":4435,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-09-13T11:24:12.069569+0000","last_change":"2024-09-13T11:23:20.443418+0000","last_active":"2024-09-13T11:24:12.069569+0000","last_peered":"2024-09-13T11:24:12.069569+0000","last_clean":"2024-09-13T11:24:12.069569+0000","last_became_active":"2024-09-13T11:23:19.927680+0000","last_became_peered":"2024-09-13T11:23:19.927680+0000","last_unstale":"2024-09-13T11:24:12.069569+0000","last_undegraded":"2024-09-13T11:24:12.069569+0000","last_fullsized":"2024-09-13T11:24:12.069569+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'2736","last_scrub_stamp":"2024-09-13T11:23:20.443329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:20.443329+0000","objects_scrubbed":1369,"log_size":4229,"log_dups_size":0,"ondisk_log_size":4229,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:24:48.147625+0000","scrub_duration":183,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":187760640,"num_objects":2865,"num_object_clones":0,"num_object_copies":5730,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2865,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1496,"num_write_kb":95744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,8],"acting":[13,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":193,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T11:24:11.036462+0000","last_change":"2024-09-13T11:23:59.530794+0000","last_active":"2024-09-13T11:24:11.036462+0000","last_peered":"2024-09-13T11:24:11.036462+0000","last_clean":"2024-09-13T11:24:11.036462+0000","last_became_active":"2024-09-13T11:22:14.581458+0000","last_became_peered":"2024-09-13T11:22:14.581458+0000","last_unstale":"2024-09-13T11:24:11.036462+0000","last_undegraded":"2024-09-13T11:24:11.036462+0000","last_fullsized":"2024-09-13T11:24:11.036462+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:23:59.530670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:23:59.530670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:02.655631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001026047,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5],"acting":[8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"110'4190","reported_seq":4372,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T11:24:10.320117+0000","last_change":"2024-09-13T11:23:20.254213+0000","last_active":"2024-09-13T11:24:10.320117+0000","last_peered":"2024-09-13T11:24:10.320117+0000","last_clean":"2024-09-13T11:24:10.320117+0000","last_became_active":"2024-09-13T11:23:19.925399+0000","last_became_peered":"2024-09-13T11:23:19.925399+0000","last_unstale":"2024-09-13T11:24:10.320117+0000","last_undegraded":"2024-09-13T11:24:10.320117+0000","last_fullsized":"2024-09-13T11:24:10.320117+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'2755","last_scrub_stamp":"2024-09-13T11:23:20.254124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:20.254124+0000","objects_scrubbed":1437,"log_size":4190,"log_dups_size":0,"ondisk_log_size":4190,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:24:49.993543+0000","scrub_duration":187,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":188547072,"num_objects":2877,"num_object_clones":0,"num_object_copies":5754,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2877,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1440,"num_write_kb":92160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,12],"acting":[1,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":179,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T11:24:08.271026+0000","last_change":"2024-09-13T11:24:08.271026+0000","last_active":"2024-09-13T11:24:08.271026+0000","last_peered":"2024-09-13T11:24:08.271026+0000","last_clean":"2024-09-13T11:24:08.271026+0000","last_became_active":"2024-09-13T11:22:47.719636+0000","last_became_peered":"2024-09-13T11:22:47.719636+0000","last_unstale":"2024-09-13T11:24:08.271026+0000","last_undegraded":"2024-09-13T11:24:08.271026+0000","last_fullsized":"2024-09-13T11:24:08.271026+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:24:08.270972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:24:08.270972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:36.805605+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00087476000000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,9],"acting":[1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"111'4202","reported_seq":4406,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T11:24:11.020631+0000","last_change":"2024-09-13T11:23:20.686446+0000","last_active":"2024-09-13T11:24:11.020631+0000","last_peered":"2024-09-13T11:24:11.020631+0000","last_clean":"2024-09-13T11:24:11.020631+0000","last_became_active":"2024-09-13T11:23:19.939695+0000","last_became_peered":"2024-09-13T11:23:19.939695+0000","last_unstale":"2024-09-13T11:24:11.020631+0000","last_undegraded":"2024-09-13T11:24:11.020631+0000","last_fullsized":"2024-09-13T11:24:11.020631+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'2739","last_scrub_stamp":"2024-09-13T11:23:20.686358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:20.686358+0000","objects_scrubbed":1386,"log_size":4202,"log_dups_size":0,"ondisk_log_size":4202,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:24:38.891737+0000","scrub_duration":271,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":186777600,"num_objects":2850,"num_object_clones":0,"num_object_copies":5700,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2850,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1464,"num_write_kb":93696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,13],"acting":[15,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"3.3","version":"111'3639","reported_seq":3866,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T11:24:11.028897+0000","last_change":"2024-09-13T11:23:32.147115+0000","last_active":"2024-09-13T11:24:11.028897+0000","last_peered":"2024-09-13T11:24:11.028897+0000","last_clean":"2024-09-13T11:24:11.028897+0000","last_became_active":"2024-09-13T11:23:31.930448+0000","last_became_peered":"2024-09-13T11:23:31.930448+0000","last_unstale":"2024-09-13T11:24:11.028897+0000","last_undegraded":"2024-09-13T11:24:11.028897+0000","last_fullsized":"2024-09-13T11:24:11.028897+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'3122","last_scrub_stamp":"2024-09-13T11:23:32.147021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:32.147021+0000","objects_scrubbed":876,"log_size":3639,"log_dups_size":0,"ondisk_log_size":3639,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:24:48.175541+0000","scrub_duration":99,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":91291648,"num_objects":1393,"num_object_clones":0,"num_object_copies":2786,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1393,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":517,"num_write_kb":33088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,7],"acting":[10,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"28'2","reported_seq":123,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T11:24:11.011218+0000","last_change":"2024-09-13T11:24:00.996197+0000","last_active":"2024-09-13T11:24:11.011218+0000","last_peered":"2024-09-13T11:24:11.011218+0000","last_clean":"2024-09-13T11:24:11.011218+0000","last_became_active":"2024-09-13T11:23:12.546348+0000","last_became_peered":"2024-09-13T11:23:12.546348+0000","last_unstale":"2024-09-13T11:24:11.011218+0000","last_undegraded":"2024-09-13T11:24:11.011218+0000","last_fullsized":"2024-09-13T11:24:11.011218+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2","last_scrub_stamp":"2024-09-13T11:24:00.996028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:24:00.996028+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:12.111097+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0025826730000000002,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8],"acting":[2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"111'3662","reported_seq":3887,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T11:24:11.057441+0000","last_change":"2024-09-13T11:23:32.749651+0000","last_active":"2024-09-13T11:24:11.057441+0000","last_peered":"2024-09-13T11:24:11.057441+0000","last_clean":"2024-09-13T11:24:11.057441+0000","last_became_active":"2024-09-13T11:23:31.921388+0000","last_became_peered":"2024-09-13T11:23:31.921388+0000","last_unstale":"2024-09-13T11:24:11.057441+0000","last_undegraded":"2024-09-13T11:24:11.057441+0000","last_fullsized":"2024-09-13T11:24:11.057441+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'3106","last_scrub_stamp":"2024-09-13T11:23:32.749487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:32.749487+0000","objects_scrubbed":856,"log_size":3662,"log_dups_size":0,"ondisk_log_size":3662,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:24:34.709501+0000","scrub_duration":91,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92536832,"num_objects":1412,"num_object_clones":0,"num_object_copies":2824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1412,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":556,"num_write_kb":35584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,15],"acting":[12,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":195,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T11:24:11.003029+0000","last_change":"2024-09-13T11:22:47.298206+0000","last_active":"2024-09-13T11:24:11.003029+0000","last_peered":"2024-09-13T11:24:11.003029+0000","last_clean":"2024-09-13T11:24:11.003029+0000","last_became_active":"2024-09-13T11:22:47.297319+0000","last_became_peered":"2024-09-13T11:22:47.297319+0000","last_unstale":"2024-09-13T11:24:11.003029+0000","last_undegraded":"2024-09-13T11:24:11.003029+0000","last_fullsized":"2024-09-13T11:24:11.003029+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:22:13.113563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00081225399999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,1],"acting":[14,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"3.1","version":"112'3655","reported_seq":3882,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-09-13T11:24:12.029251+0000","last_change":"2024-09-13T11:23:33.604966+0000","last_active":"2024-09-13T11:24:12.029251+0000","last_peered":"2024-09-13T11:24:12.029251+0000","last_clean":"2024-09-13T11:24:12.029251+0000","last_became_active":"2024-09-13T11:23:31.926528+0000","last_became_peered":"2024-09-13T11:23:31.926528+0000","last_unstale":"2024-09-13T11:24:12.029251+0000","last_undegraded":"2024-09-13T11:24:12.029251+0000","last_fullsized":"2024-09-13T11:24:12.029251+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'3070","last_scrub_stamp":"2024-09-13T11:23:33.604792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:33.604792+0000","objects_scrubbed":842,"log_size":3655,"log_dups_size":0,"ondisk_log_size":3655,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:24:43.880038+0000","scrub_duration":119,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":93519872,"num_objects":1427,"num_object_clones":0,"num_object_copies":2854,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1427,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":585,"num_write_kb":37440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,11],"acting":[13,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":183,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T11:24:07.959659+0000","last_change":"2024-09-13T11:22:16.267999+0000","last_active":"2024-09-13T11:24:07.959659+0000","last_peered":"2024-09-13T11:24:07.959659+0000","last_clean":"2024-09-13T11:24:07.959659+0000","last_became_active":"2024-09-13T11:22:14.211435+0000","last_became_peered":"2024-09-13T11:22:14.211435+0000","last_unstale":"2024-09-13T11:24:07.959659+0000","last_undegraded":"2024-09-13T11:24:07.959659+0000","last_fullsized":"2024-09-13T11:24:07.959659+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:22:13.113563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:24:10.770081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010010360000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2","version":"110'3674","reported_seq":3900,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T11:24:11.003116+0000","last_change":"2024-09-13T11:23:32.977666+0000","last_active":"2024-09-13T11:24:11.003116+0000","last_peered":"2024-09-13T11:24:11.003116+0000","last_clean":"2024-09-13T11:24:11.003116+0000","last_became_active":"2024-09-13T11:23:31.919828+0000","last_became_peered":"2024-09-13T11:23:31.919828+0000","last_unstale":"2024-09-13T11:24:11.003116+0000","last_undegraded":"2024-09-13T11:24:11.003116+0000","last_fullsized":"2024-09-13T11:24:11.003116+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'3057","last_scrub_stamp":"2024-09-13T11:23:32.977559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:32.977559+0000","objects_scrubbed":841,"log_size":3674,"log_dups_size":0,"ondisk_log_size":3674,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:24:52.169339+0000","scrub_duration":151,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":95551488,"num_objects":1458,"num_object_clones":0,"num_object_copies":2916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1458,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":617,"num_write_kb":39488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,12],"acting":[14,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"2.3","version":"26'1","reported_seq":187,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T11:24:11.003014+0000","last_change":"2024-09-13T11:22:16.547691+0000","last_active":"2024-09-13T11:24:11.003014+0000","last_peered":"2024-09-13T11:24:11.003014+0000","last_clean":"2024-09-13T11:24:11.003014+0000","last_became_active":"2024-09-13T11:22:14.146507+0000","last_became_peered":"2024-09-13T11:22:14.146507+0000","last_unstale":"2024-09-13T11:24:11.003014+0000","last_undegraded":"2024-09-13T11:24:11.003014+0000","last_fullsized":"2024-09-13T11:24:11.003014+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:22:13.113563+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00085610999999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,15],"acting":[14,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"1.0","version":"26'112","reported_seq":327,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T11:24:10.981430+0000","last_change":"2024-09-13T11:23:49.325249+0000","last_active":"2024-09-13T11:24:10.981430+0000","last_peered":"2024-09-13T11:24:10.981430+0000","last_clean":"2024-09-13T11:24:10.981430+0000","last_became_active":"2024-09-13T11:22:12.129721+0000","last_became_peered":"2024-09-13T11:22:12.129721+0000","last_unstale":"2024-09-13T11:24:10.981430+0000","last_undegraded":"2024-09-13T11:24:10.981430+0000","last_fullsized":"2024-09-13T11:24:10.981430+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"26'112","last_scrub_stamp":"2024-09-13T11:23:49.325175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:11.098388+0000","last_clean_scrub_stamp":"2024-09-13T11:23:49.325175+0000","objects_scrubbed":2,"log_size":112,"log_dups_size":0,"ondisk_log_size":112,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:14.422287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.8","version":"110'4197","reported_seq":4402,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T11:24:10.981305+0000","last_change":"2024-09-13T11:23:20.644952+0000","last_active":"2024-09-13T11:24:10.981305+0000","last_peered":"2024-09-13T11:24:10.981305+0000","last_clean":"2024-09-13T11:24:10.981305+0000","last_became_active":"2024-09-13T11:23:19.925802+0000","last_became_peered":"2024-09-13T11:23:19.925802+0000","last_unstale":"2024-09-13T11:24:10.981305+0000","last_undegraded":"2024-09-13T11:24:10.981305+0000","last_fullsized":"2024-09-13T11:24:10.981305+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'2771","last_scrub_stamp":"2024-09-13T11:23:20.644747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:20.644747+0000","objects_scrubbed":1356,"log_size":4197,"log_dups_size":0,"ondisk_log_size":4197,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:24:38.144523+0000","scrub_duration":411,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":183107584,"num_objects":2794,"num_object_clones":0,"num_object_copies":5588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2794,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1438,"num_write_kb":92032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.9","version":"111'4275","reported_seq":4478,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T11:24:11.080932+0000","last_change":"2024-09-13T11:23:22.033678+0000","last_active":"2024-09-13T11:24:11.080932+0000","last_peered":"2024-09-13T11:24:11.080932+0000","last_clean":"2024-09-13T11:24:11.080932+0000","last_became_active":"2024-09-13T11:23:19.935114+0000","last_became_peered":"2024-09-13T11:23:19.935114+0000","last_unstale":"2024-09-13T11:24:11.080932+0000","last_undegraded":"2024-09-13T11:24:11.080932+0000","last_fullsized":"2024-09-13T11:24:11.080932+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'2808","last_scrub_stamp":"2024-09-13T11:23:22.033572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:22.033572+0000","objects_scrubbed":1471,"log_size":4275,"log_dups_size":0,"ondisk_log_size":4275,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:24:46.595859+0000","scrub_duration":287,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":192544768,"num_objects":2938,"num_object_clones":0,"num_object_copies":5876,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2938,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1467,"num_write_kb":93888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,2],"acting":[12,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[]},{"pgid":"3.a","version":"110'4531","reported_seq":4737,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T11:24:10.020132+0000","last_change":"2024-09-13T11:23:32.526150+0000","last_active":"2024-09-13T11:24:10.020132+0000","last_peered":"2024-09-13T11:24:10.020132+0000","last_clean":"2024-09-13T11:24:10.020132+0000","last_became_active":"2024-09-13T11:23:31.934170+0000","last_became_peered":"2024-09-13T11:23:31.934170+0000","last_unstale":"2024-09-13T11:24:10.020132+0000","last_undegraded":"2024-09-13T11:24:10.020132+0000","last_fullsized":"2024-09-13T11:24:10.020132+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'3418","last_scrub_stamp":"2024-09-13T11:23:32.526057+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:32.526057+0000","objects_scrubbed":1774,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:24:34.770161+0000","scrub_duration":507,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":189202432,"num_objects":2887,"num_object_clones":0,"num_object_copies":5774,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2887,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1113,"num_write_kb":71232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.b","version":"112'4627","reported_seq":4836,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-09-13T11:24:12.015276+0000","last_change":"2024-09-13T11:23:32.681343+0000","last_active":"2024-09-13T11:24:12.015276+0000","last_peered":"2024-09-13T11:24:12.015276+0000","last_clean":"2024-09-13T11:24:12.015276+0000","last_became_active":"2024-09-13T11:23:31.923415+0000","last_became_peered":"2024-09-13T11:23:31.923415+0000","last_unstale":"2024-09-13T11:24:12.015276+0000","last_undegraded":"2024-09-13T11:24:12.015276+0000","last_fullsized":"2024-09-13T11:24:12.015276+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'3471","last_scrub_stamp":"2024-09-13T11:23:32.681257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:32.681257+0000","objects_scrubbed":1720,"log_size":4627,"log_dups_size":0,"ondisk_log_size":4627,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:24:58.857812+0000","scrub_duration":199,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":188481536,"num_objects":2876,"num_object_clones":0,"num_object_copies":5752,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2876,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1156,"num_write_kb":73984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,1],"acting":[13,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"3.c","version":"112'4619","reported_seq":4827,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-09-13T11:24:11.995484+0000","last_change":"2024-09-13T11:23:32.524870+0000","last_active":"2024-09-13T11:24:11.995484+0000","last_peered":"2024-09-13T11:24:11.995484+0000","last_clean":"2024-09-13T11:24:11.995484+0000","last_became_active":"2024-09-13T11:23:31.922620+0000","last_became_peered":"2024-09-13T11:23:31.922620+0000","last_unstale":"2024-09-13T11:24:11.995484+0000","last_undegraded":"2024-09-13T11:24:11.995484+0000","last_fullsized":"2024-09-13T11:24:11.995484+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'3463","last_scrub_stamp":"2024-09-13T11:23:32.524776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:32.524776+0000","objects_scrubbed":1741,"log_size":4619,"log_dups_size":0,"ondisk_log_size":4619,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:24:37.745747+0000","scrub_duration":235,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":189857792,"num_objects":2897,"num_object_clones":0,"num_object_copies":5794,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2897,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1156,"num_write_kb":73984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,13],"acting":[11,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.d","version":"111'4567","reported_seq":4773,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T11:24:11.080180+0000","last_change":"2024-09-13T11:23:32.776835+0000","last_active":"2024-09-13T11:24:11.080180+0000","last_peered":"2024-09-13T11:24:11.080180+0000","last_clean":"2024-09-13T11:24:11.080180+0000","last_became_active":"2024-09-13T11:23:31.939246+0000","last_became_peered":"2024-09-13T11:23:31.939246+0000","last_unstale":"2024-09-13T11:24:11.080180+0000","last_undegraded":"2024-09-13T11:24:11.080180+0000","last_fullsized":"2024-09-13T11:24:11.080180+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'3395","last_scrub_stamp":"2024-09-13T11:23:32.776747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:32.776747+0000","objects_scrubbed":1704,"log_size":4567,"log_dups_size":0,"ondisk_log_size":4567,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:24:56.402862+0000","scrub_duration":179,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":188481536,"num_objects":2876,"num_object_clones":0,"num_object_copies":5752,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2876,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1172,"num_write_kb":75008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.e","version":"111'4643","reported_seq":9778,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T11:24:11.091345+0000","last_change":"2024-09-13T11:23:32.368856+0000","last_active":"2024-09-13T11:24:11.091345+0000","last_peered":"2024-09-13T11:24:11.091345+0000","last_clean":"2024-09-13T11:24:11.091345+0000","last_became_active":"2024-09-13T11:23:31.925869+0000","last_became_peered":"2024-09-13T11:23:31.925869+0000","last_unstale":"2024-09-13T11:24:11.091345+0000","last_undegraded":"2024-09-13T11:24:11.091345+0000","last_fullsized":"2024-09-13T11:24:11.091345+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'3527","last_scrub_stamp":"2024-09-13T11:23:32.368727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:32.368727+0000","objects_scrubbed":1774,"log_size":4643,"log_dups_size":0,"ondisk_log_size":4643,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:24:57.811779+0000","scrub_duration":219,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":189464576,"num_objects":2891,"num_object_clones":0,"num_object_copies":5782,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2891,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1117,"num_write_kb":71488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"112'4589","reported_seq":4797,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-09-13T11:24:12.056478+0000","last_change":"2024-09-13T11:23:33.492630+0000","last_active":"2024-09-13T11:24:12.056478+0000","last_peered":"2024-09-13T11:24:12.056478+0000","last_clean":"2024-09-13T11:24:12.056478+0000","last_became_active":"2024-09-13T11:23:31.935239+0000","last_became_peered":"2024-09-13T11:23:31.935239+0000","last_unstale":"2024-09-13T11:24:12.056478+0000","last_undegraded":"2024-09-13T11:24:12.056478+0000","last_fullsized":"2024-09-13T11:24:12.056478+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'3420","last_scrub_stamp":"2024-09-13T11:23:33.492488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:33.492488+0000","objects_scrubbed":1776,"log_size":4589,"log_dups_size":0,"ondisk_log_size":4589,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:24:44.851261+0000","scrub_duration":215,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":193003520,"num_objects":2945,"num_object_clones":0,"num_object_copies":5890,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2945,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1169,"num_write_kb":74816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,12],"acting":[11,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.10","version":"111'4361","reported_seq":8722,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T11:24:11.004908+0000","last_change":"2024-09-13T11:23:48.519654+0000","last_active":"2024-09-13T11:24:11.004908+0000","last_peered":"2024-09-13T11:24:11.004908+0000","last_clean":"2024-09-13T11:24:11.004908+0000","last_became_active":"2024-09-13T11:23:41.931183+0000","last_became_peered":"2024-09-13T11:23:41.931183+0000","last_unstale":"2024-09-13T11:24:11.004908+0000","last_undegraded":"2024-09-13T11:24:11.004908+0000","last_fullsized":"2024-09-13T11:24:11.004908+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":5,"last_scrub":"91'3468","last_scrub_stamp":"2024-09-13T11:23:48.519574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:48.519574+0000","objects_scrubbed":2031,"log_size":4361,"log_dups_size":0,"ondisk_log_size":4361,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:24:57.342213+0000","scrub_duration":215,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":191758336,"num_objects":2926,"num_object_clones":0,"num_object_copies":5852,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2926,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":895,"num_write_kb":57280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.11","version":"111'4217","reported_seq":7357,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T11:24:11.043147+0000","last_change":"2024-09-13T11:23:46.102466+0000","last_active":"2024-09-13T11:24:11.043147+0000","last_peered":"2024-09-13T11:24:11.043147+0000","last_clean":"2024-09-13T11:24:11.043147+0000","last_became_active":"2024-09-13T11:23:39.063172+0000","last_became_peered":"2024-09-13T11:23:39.063172+0000","last_unstale":"2024-09-13T11:24:11.043147+0000","last_undegraded":"2024-09-13T11:24:11.043147+0000","last_fullsized":"2024-09-13T11:24:11.043147+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":5,"last_scrub":"88'3345","last_scrub_stamp":"2024-09-13T11:23:46.101973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:46.101973+0000","objects_scrubbed":2011,"log_size":4217,"log_dups_size":0,"ondisk_log_size":4217,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:08.671350+0000","scrub_duration":955,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":189202432,"num_objects":2887,"num_object_clones":0,"num_object_copies":5774,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2887,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":876,"num_write_kb":56064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11],"acting":[4,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.12","version":"111'4218","reported_seq":7282,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T11:24:11.003159+0000","last_change":"2024-09-13T11:23:35.111409+0000","last_active":"2024-09-13T11:24:11.003159+0000","last_peered":"2024-09-13T11:24:11.003159+0000","last_clean":"2024-09-13T11:24:11.003159+0000","last_became_active":"2024-09-13T11:23:21.998877+0000","last_became_peered":"2024-09-13T11:23:21.998877+0000","last_unstale":"2024-09-13T11:24:11.003159+0000","last_undegraded":"2024-09-13T11:24:11.003159+0000","last_fullsized":"2024-09-13T11:24:11.003159+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'3109","last_scrub_stamp":"2024-09-13T11:23:35.111320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:35.111320+0000","objects_scrubbed":1732,"log_size":4218,"log_dups_size":0,"ondisk_log_size":4218,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:03.371433+0000","scrub_duration":239,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":186187776,"num_objects":2841,"num_object_clones":0,"num_object_copies":5682,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2841,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1109,"num_write_kb":70976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,7],"acting":[14,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"3.13","version":"112'4312","reported_seq":7379,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-09-13T11:24:12.059551+0000","last_change":"2024-09-13T11:23:47.565094+0000","last_active":"2024-09-13T11:24:12.059551+0000","last_peered":"2024-09-13T11:24:12.059551+0000","last_clean":"2024-09-13T11:24:12.059551+0000","last_became_active":"2024-09-13T11:23:41.082433+0000","last_became_peered":"2024-09-13T11:23:41.082433+0000","last_unstale":"2024-09-13T11:24:12.059551+0000","last_undegraded":"2024-09-13T11:24:12.059551+0000","last_fullsized":"2024-09-13T11:24:12.059551+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'3408","last_scrub_stamp":"2024-09-13T11:23:47.564931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:47.564931+0000","objects_scrubbed":1981,"log_size":4312,"log_dups_size":0,"ondisk_log_size":4312,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:04.258231+0000","scrub_duration":471,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":189857792,"num_objects":2897,"num_object_clones":0,"num_object_copies":5794,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2897,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":916,"num_write_kb":58624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7],"acting":[11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":36,"stat_sum":{"num_bytes":6023643136,"num_objects":91914,"num_object_clones":0,"num_object_copies":183828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":91914,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":37320,"num_write_kb":2388448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5754,"num_bytes_recovered":377094144,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":13574275072,"data_stored":13574275072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":152493,"ondisk_log_size":152493,"up":72,"acting":72,"num_store_stats":16},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":10},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":112,"ondisk_log_size":112,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":13,"up_from":23,"seq":98784247836,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1066984,"kb_used_data":1003864,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":93304856,"statfs":{"total":96636764160,"available":95544172544,"internally_reserved":0,"allocated":1027956736,"data_stored":1027464855,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[0,1,3,4,6,8,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":9,"up_from":23,"seq":98784247835,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":495128,"kb_used_data":455624,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":93876712,"statfs":{"total":96636764160,"available":96129753088,"internally_reserved":0,"allocated":466558976,"data_stored":466082552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":40434112},"hb_peers":[0,1,2,8,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":40,"seq":171798691858,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1363564,"kb_used_data":1299612,"kb_used_omap":4,"kb_used_meta":63931,"kb_avail":93008276,"statfs":{"total":96636764160,"available":95240474624,"internally_reserved":0,"allocated":1330802688,"data_stored":1330300776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4433,"internal_metadata":65466031},"hb_peers":[0,2,3,6,8,9,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":21,"seq":90194313243,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1077112,"kb_used_data":1001512,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":93294728,"statfs":{"total":96636764160,"available":95533801472,"internally_reserved":0,"allocated":1025548288,"data_stored":1025039643,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":77396423},"hb_peers":[0,2,4,7,9,10,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,1,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":21,"seq":90194313246,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303284,"kb_used_data":276260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94068556,"statfs":{"total":96636764160,"available":96326201344,"internally_reserved":0,"allocated":282890240,"data_stored":282382299,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,6,8,9,10,11,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":22,"seq":94489280538,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1220588,"kb_used_data":1153308,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":93151252,"statfs":{"total":96636764160,"available":95386882048,"internally_reserved":0,"allocated":1180987392,"data_stored":1180548271,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[1,3,4,5,8,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":22,"seq":94489280539,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":461856,"kb_used_data":422352,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":93909984,"statfs":{"total":96636764160,"available":96163823616,"internally_reserved":0,"allocated":432488448,"data_stored":432004521,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,3,7,8,9,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":15,"up_from":22,"seq":94489280539,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":979620,"kb_used_data":904020,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":93392220,"statfs":{"total":96636764160,"available":95633633280,"internally_reserved":0,"allocated":925716480,"data_stored":925285333,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":77396422},"hb_peers":[0,3,4,5,6,7,10,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":22,"seq":94489280539,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":587324,"kb_used_data":538092,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":93784516,"statfs":{"total":96636764160,"available":96035344384,"internally_reserved":0,"allocated":551006208,"data_stored":550495788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[3,5,7,9,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":22,"seq":94489280539,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":453512,"kb_used_data":414008,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":93918328,"statfs":{"total":96636764160,"available":96172367872,"internally_reserved":0,"allocated":423944192,"data_stored":423479893,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40434118},"hb_peers":[0,1,3,5,7,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":22,"seq":94489280539,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1279808,"kb_used_data":1208368,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":93092032,"statfs":{"total":96636764160,"available":95326240768,"internally_reserved":0,"allocated":1237368832,"data_stored":1236910392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":73136586},"hb_peers":[0,3,5,6,8,9,10,11,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":22,"seq":94489280539,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":479740,"kb_used_data":448556,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93892100,"statfs":{"total":96636764160,"available":96145510400,"internally_reserved":0,"allocated":459321344,"data_stored":458806681,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,5,7,9,10,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":22,"seq":94489280538,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":719056,"kb_used_data":673984,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":93652784,"statfs":{"total":96636764160,"available":95900450816,"internally_reserved":0,"allocated":690159616,"data_stored":689694148,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":46135744},"hb_peers":[0,1,3,6,7,9,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":11,"up_from":22,"seq":94489280540,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1276712,"kb_used_data":1209432,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":93095128,"statfs":{"total":96636764160,"available":95329411072,"internally_reserved":0,"allocated":1238458368,"data_stored":1237970110,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":68876740},"hb_peers":[0,2,4,7,8,10,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":12,"up_from":22,"seq":94489280539,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1085344,"kb_used_data":1018064,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":93286496,"statfs":{"total":96636764160,"available":95525371904,"internally_reserved":0,"allocated":1042497536,"data_stored":1042010499,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,1,2,3,5,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":14,"up_from":22,"seq":94489280539,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1251820,"kb_used_data":1176220,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":93120020,"statfs":{"total":96636764160,"available":95354900480,"internally_reserved":0,"allocated":1204449280,"data_stored":1203955694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,1,5,6,7,8,9,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1189871616,"data_stored":1189871616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1328545792,"data_stored":1328545792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":435093504,"data_stored":435093504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1024000000,"data_stored":1024000000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":558497792,"data_stored":558497792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":285802496,"data_stored":285802496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":425328640,"data_stored":425328640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1238433792,"data_stored":1238433792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":457113600,"data_stored":457113600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":468123648,"data_stored":468123648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":688979968,"data_stored":688979968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":1249443840,"data_stored":1249443840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":1047789568,"data_stored":1047789568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":1043136512,"data_stored":1043136512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":1203437568,"data_stored":1203437568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":930676736,"data_stored":930676736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T11:24:14.933 INFO:teuthology.orchestra.run.smithi001.stdout:osd.7: {} 2024-09-13T11:24:14.934 INFO:teuthology.orchestra.run.smithi001.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T11:24:14.960 INFO:teuthology.orchestra.run.smithi001.stdout:osd.8: {} 2024-09-13T11:24:14.961 INFO:teuthology.orchestra.run.smithi001.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-13T11:24:14.986 INFO:teuthology.orchestra.run.smithi001.stdout:osd.9: {} 2024-09-13T11:24:14.986 INFO:teuthology.orchestra.run.smithi001.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-13T11:24:15.008 INFO:tasks.radosbench.radosbench.0.smithi177.stdout: 108 16 98058 98042 56.7316 101.25 0.00614258 0.0176213 2024-09-13T11:24:15.009 INFO:tasks.ceph.osd.3.smithi177.stderr:2024-09-13T11:24:15.007+0000 7f10adba4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26574) UID: 0 2024-09-13T11:24:15.012 INFO:teuthology.orchestra.run.smithi001.stdout:osd.10: {} 2024-09-13T11:24:15.012 INFO:teuthology.orchestra.run.smithi001.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-13T11:24:15.016 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-13T11:24:15.016 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:24:15.031 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_historic_ops 2024-09-13T11:24:15.036 INFO:teuthology.orchestra.run.smithi001.stdout:osd.11: {} 2024-09-13T11:24:15.037 INFO:teuthology.orchestra.run.smithi001.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-13T11:24:15.059 INFO:teuthology.orchestra.run.smithi001.stdout:osd.12: {} 2024-09-13T11:24:15.060 INFO:teuthology.orchestra.run.smithi001.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-13T11:24:15.086 INFO:teuthology.orchestra.run.smithi001.stdout:osd.13: {} 2024-09-13T11:24:15.086 INFO:teuthology.orchestra.run.smithi001.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-13T11:24:15.109 INFO:tasks.ceph.osd.4.smithi001.stderr:2024-09-13T11:24:15.103+0000 7ff8e0218640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26310) UID: 0 2024-09-13T11:24:15.110 INFO:teuthology.orchestra.run.smithi001.stdout:osd.14: {} 2024-09-13T11:24:15.110 INFO:teuthology.orchestra.run.smithi001.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-13T11:24:15.133 INFO:teuthology.orchestra.run.smithi001.stdout:osd.15: {} 2024-09-13T11:24:15.133 INFO:teuthology.orchestra.run.smithi001.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-13T11:24:15.161 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-13T11:24:15.161 INFO:teuthology.orchestra.run.smithi162.stdout: "size": 20, 2024-09-13T11:24:15.161 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 600, 2024-09-13T11:24:15.161 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [ 2024-09-13T11:24:15.161 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:15.161 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:48911 3.12 3:4a364a6a:::benchmark_data_smithi177_31116_object48910:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e71)", 2024-09-13T11:24:15.161 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:25.457776+0000", 2024-09-13T11:24:15.161 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 49.700337578999999, 2024-09-13T11:24:15.161 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.22561999599999999, 2024-09-13T11:24:15.162 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:24:15.162 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:24:15.162 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:15.162 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:24:15.162 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:24:15.162 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:15.162 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 48911 2024-09-13T11:24:15.162 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:15.162 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:24:15.162 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:15.162 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:24:15.162 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.457776+0000", 2024-09-13T11:24:15.162 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:15.163 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:15.163 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:15.163 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:24:15.163 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.457776+0000", 2024-09-13T11:24:15.163 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:15.163 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:15.163 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:15.163 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:24:15.163 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.457788+0000", 2024-09-13T11:24:15.163 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2084e-05 2024-09-13T11:24:15.163 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:15.163 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:15.163 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:24:15.164 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.457823+0000", 2024-09-13T11:24:15.164 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.5002000000000001e-05 2024-09-13T11:24:15.164 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:15.164 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:15.164 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:24:15.164 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.457836+0000", 2024-09-13T11:24:15.164 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2988e-05 2024-09-13T11:24:15.164 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:15.164 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:15.164 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:24:15.164 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.457843+0000", 2024-09-13T11:24:15.164 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.5320000000000003e-06 2024-09-13T11:24:15.164 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:15.164 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:15.165 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:24:15.165 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.457895+0000", 2024-09-13T11:24:15.165 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.1690000000000001e-05 2024-09-13T11:24:15.165 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:15.165 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:15.165 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:24:15.165 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.458001+0000", 2024-09-13T11:24:15.165 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000106024 2024-09-13T11:24:15.165 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:15.165 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:15.165 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 7", 2024-09-13T11:24:15.165 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.458101+0000", 2024-09-13T11:24:15.165 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00010060699999999999 2024-09-13T11:24:15.165 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:15.166 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:15.166 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:24:15.166 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.459250+0000", 2024-09-13T11:24:15.166 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0011485670000000001 2024-09-13T11:24:15.166 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:15.166 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:15.166 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:15.166 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.682166+0000", 2024-09-13T11:24:15.166 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.22291609500000001 2024-09-13T11:24:15.166 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:15.166 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:15.166 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:24:15.166 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.682271+0000", 2024-09-13T11:24:15.166 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000105179 2024-09-13T11:24:15.167 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:15.167 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:15.167 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:24:15.167 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.683396+0000", 2024-09-13T11:24:15.167 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001125228 2024-09-13T11:24:15.167 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:15.167 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:24:15.167 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:15.167 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:15.167 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:15.167 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:48923 3.12 3:4ae1afd7:::benchmark_data_smithi177_31116_object48922:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e71)", 2024-09-13T11:24:15.167 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:25.465697+0000", 2024-09-13T11:24:15.167 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 49.692416414, 2024-09-13T11:24:15.168 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.21931837100000001, 2024-09-13T11:24:15.168 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:24:15.168 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:24:15.168 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:15.168 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:24:15.168 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:24:15.168 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:15.168 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 48923 2024-09-13T11:24:15.168 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:15.168 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:24:15.168 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:15.168 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:24:15.168 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.465697+0000", 2024-09-13T11:24:15.168 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:15.169 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:15.169 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:15.169 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:24:15.169 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.465697+0000", 2024-09-13T11:24:15.169 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:15.169 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:15.169 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:15.169 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:24:15.169 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.465700+0000", 2024-09-13T11:24:15.169 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.041e-06 2024-09-13T11:24:15.169 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:15.169 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:15.169 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:24:15.170 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.465757+0000", 2024-09-13T11:24:15.170 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.7664999999999998e-05 2024-09-13T11:24:15.170 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:15.170 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:15.170 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:24:15.170 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.465769+0000", 2024-09-13T11:24:15.170 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.219e-05 2024-09-13T11:24:15.170 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:15.170 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:15.170 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:24:15.170 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.465775+0000", 2024-09-13T11:24:15.170 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.6779999999999997e-06 2024-09-13T11:24:15.170 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:15.170 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:15.171 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:24:15.171 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.465821+0000", 2024-09-13T11:24:15.171 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.5577000000000001e-05 2024-09-13T11:24:15.171 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:15.171 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:15.171 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:24:15.171 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.465928+0000", 2024-09-13T11:24:15.171 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000107502 2024-09-13T11:24:15.171 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:15.171 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:15.171 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 7", 2024-09-13T11:24:15.171 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.466022+0000", 2024-09-13T11:24:15.171 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.3802999999999996e-05 2024-09-13T11:24:15.172 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:15.172 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:15.172 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:24:15.172 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.466901+0000", 2024-09-13T11:24:15.172 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00087950699999999997 2024-09-13T11:24:15.172 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:15.172 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:15.172 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:15.172 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.683622+0000", 2024-09-13T11:24:15.172 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.216720462 2024-09-13T11:24:15.172 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:15.172 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:15.172 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:24:15.173 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.683771+0000", 2024-09-13T11:24:15.173 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00014900199999999999 2024-09-13T11:24:15.173 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:15.173 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:15.173 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:24:15.173 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.685016+0000", 2024-09-13T11:24:15.173 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001244944 2024-09-13T11:24:15.173 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:15.173 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:24:15.173 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:15.173 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:15.173 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:15.173 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:48955 3.12 3:4ef2991b:::benchmark_data_smithi177_31116_object48954:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e71)", 2024-09-13T11:24:15.174 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:25.494091+0000", 2024-09-13T11:24:15.174 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 49.664022777, 2024-09-13T11:24:15.174 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.19235933599999999, 2024-09-13T11:24:15.174 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:24:15.174 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:24:15.174 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:15.174 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:24:15.174 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:24:15.174 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:15.174 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 48955 2024-09-13T11:24:15.174 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:15.174 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:24:15.174 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:15.174 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:24:15.175 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.494091+0000", 2024-09-13T11:24:15.175 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:15.175 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:15.175 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:15.175 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:24:15.175 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.494091+0000", 2024-09-13T11:24:15.176 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:15.176 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:15.176 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:15.176 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:24:15.176 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.494093+0000", 2024-09-13T11:24:15.176 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.0770000000000002e-06 2024-09-13T11:24:15.176 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:15.176 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:15.176 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:24:15.176 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.494160+0000", 2024-09-13T11:24:15.176 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.6798999999999995e-05 2024-09-13T11:24:15.176 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:15.176 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:15.176 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:24:15.177 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.494174+0000", 2024-09-13T11:24:15.177 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4489e-05 2024-09-13T11:24:15.177 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:15.177 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:15.177 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:24:15.177 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.494183+0000", 2024-09-13T11:24:15.177 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.4460000000000001e-06 2024-09-13T11:24:15.177 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:15.177 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:15.177 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:24:15.177 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.494239+0000", 2024-09-13T11:24:15.177 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.6140999999999997e-05 2024-09-13T11:24:15.177 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:15.177 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:15.178 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:24:15.178 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.494385+0000", 2024-09-13T11:24:15.178 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00014546399999999999 2024-09-13T11:24:15.178 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:15.178 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:15.178 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 7", 2024-09-13T11:24:15.178 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.494506+0000", 2024-09-13T11:24:15.178 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000121328 2024-09-13T11:24:15.178 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:15.178 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:15.178 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:24:15.178 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.495528+0000", 2024-09-13T11:24:15.178 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0010226079999999999 2024-09-13T11:24:15.178 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:15.179 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:15.179 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:15.179 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.685177+0000", 2024-09-13T11:24:15.179 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.18964847500000001 2024-09-13T11:24:15.179 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:15.179 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:15.179 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:24:15.179 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.685337+0000", 2024-09-13T11:24:15.179 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00016055400000000001 2024-09-13T11:24:15.179 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:15.179 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:15.179 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:24:15.179 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.686450+0000", 2024-09-13T11:24:15.179 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0011129550000000001 2024-09-13T11:24:15.180 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:15.180 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:24:15.180 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:15.180 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:15.180 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:15.180 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:48967 3.12 3:4abf2346:::benchmark_data_smithi177_31116_object48966:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e71)", 2024-09-13T11:24:15.180 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:25.509687+0000", 2024-09-13T11:24:15.180 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 49.648426653000001, 2024-09-13T11:24:15.180 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.17798804900000001, 2024-09-13T11:24:15.180 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:24:15.180 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:24:15.180 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:15.180 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:24:15.180 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:24:15.181 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:15.181 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 48967 2024-09-13T11:24:15.181 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:15.181 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:24:15.181 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:15.181 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:24:15.181 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.509687+0000", 2024-09-13T11:24:15.181 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:15.181 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:15.181 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:15.181 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:24:15.181 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.509687+0000", 2024-09-13T11:24:15.181 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:15.181 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:15.182 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:15.182 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:24:15.182 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.509689+0000", 2024-09-13T11:24:15.182 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.9719999999999999e-06 2024-09-13T11:24:15.182 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:15.182 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:15.182 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:24:15.182 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.509754+0000", 2024-09-13T11:24:15.182 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.5031999999999997e-05 2024-09-13T11:24:15.182 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:15.182 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:15.182 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:24:15.182 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.509787+0000", 2024-09-13T11:24:15.182 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.2891000000000001e-05 2024-09-13T11:24:15.183 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:15.183 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:15.183 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:24:15.183 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.509793+0000", 2024-09-13T11:24:15.183 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.1500000000000004e-06 2024-09-13T11:24:15.183 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:15.183 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:15.183 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:24:15.183 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.509846+0000", 2024-09-13T11:24:15.183 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.2926000000000001e-05 2024-09-13T11:24:15.183 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:15.183 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:15.183 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:24:15.183 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.509957+0000", 2024-09-13T11:24:15.184 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000110734 2024-09-13T11:24:15.184 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:15.184 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:15.184 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 7", 2024-09-13T11:24:15.184 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.510083+0000", 2024-09-13T11:24:15.184 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00012628300000000001 2024-09-13T11:24:15.184 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:15.184 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:15.184 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:24:15.184 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.511047+0000", 2024-09-13T11:24:15.184 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00096376800000000005 2024-09-13T11:24:15.184 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:15.184 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:15.185 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:15.185 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.686579+0000", 2024-09-13T11:24:15.185 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.17553200999999999 2024-09-13T11:24:15.185 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:15.185 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:15.185 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:24:15.185 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.686846+0000", 2024-09-13T11:24:15.185 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000267039 2024-09-13T11:24:15.185 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:15.185 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:15.185 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:24:15.185 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.687675+0000", 2024-09-13T11:24:15.185 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00082924400000000003 2024-09-13T11:24:15.185 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:15.185 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:24:15.186 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:15.186 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:15.186 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:15.186 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:48987 3.12 3:4e1e6d0c:::benchmark_data_smithi177_31116_object48986:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e71)", 2024-09-13T11:24:15.186 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:25.525640+0000", 2024-09-13T11:24:15.186 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 49.632474008999999, 2024-09-13T11:24:15.186 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.163044945, 2024-09-13T11:24:15.186 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:24:15.186 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:24:15.186 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:15.186 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:24:15.186 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:24:15.186 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:15.186 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 48987 2024-09-13T11:24:15.187 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:15.187 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:24:15.187 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:15.187 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:24:15.187 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.525640+0000", 2024-09-13T11:24:15.187 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:15.187 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:15.187 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:15.187 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:24:15.187 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.525640+0000", 2024-09-13T11:24:15.187 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:15.187 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:15.187 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:15.187 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:24:15.188 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.525651+0000", 2024-09-13T11:24:15.188 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1092e-05 2024-09-13T11:24:15.188 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:15.188 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:15.188 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:24:15.188 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.525795+0000", 2024-09-13T11:24:15.188 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00014417400000000001 2024-09-13T11:24:15.188 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:15.188 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:15.188 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:24:15.188 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.525834+0000", 2024-09-13T11:24:15.188 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.9026000000000002e-05 2024-09-13T11:24:15.188 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:15.188 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:15.189 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:24:15.189 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.525857+0000", 2024-09-13T11:24:15.189 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.2904999999999998e-05 2024-09-13T11:24:15.189 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:15.189 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:15.189 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:24:15.189 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.525967+0000", 2024-09-13T11:24:15.189 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000110188 2024-09-13T11:24:15.189 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:15.189 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:15.189 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:24:15.189 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.526214+0000", 2024-09-13T11:24:15.189 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00024667800000000001 2024-09-13T11:24:15.190 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:15.190 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:15.190 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 7", 2024-09-13T11:24:15.190 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.526499+0000", 2024-09-13T11:24:15.190 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00028463799999999998 2024-09-13T11:24:15.190 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:15.190 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:15.190 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:24:15.190 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.528226+0000", 2024-09-13T11:24:15.190 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0017276679999999999 2024-09-13T11:24:15.190 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:15.190 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:15.190 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:15.190 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.687803+0000", 2024-09-13T11:24:15.191 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.15957658899999999 2024-09-13T11:24:15.191 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:15.191 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:15.191 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:24:15.191 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.687986+0000", 2024-09-13T11:24:15.191 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00018332400000000001 2024-09-13T11:24:15.191 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:15.191 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:15.191 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:24:15.191 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.688685+0000", 2024-09-13T11:24:15.191 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00069866299999999998 2024-09-13T11:24:15.191 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:15.191 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:24:15.191 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:15.191 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:15.192 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:15.192 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:54637 3.22 3:44450106:::benchmark_data_smithi177_31116_object54636:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e75)", 2024-09-13T11:24:15.192 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:30.978960+0000", 2024-09-13T11:24:15.192 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 44.179153577000001, 2024-09-13T11:24:15.192 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.960188928, 2024-09-13T11:24:15.192 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:24:15.192 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:24:15.192 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-09-13T11:24:15.192 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:24:15.192 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:24:15.192 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:15.192 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 54637 2024-09-13T11:24:15.192 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:15.193 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:24:15.193 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:15.193 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:24:15.193 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.978960+0000", 2024-09-13T11:24:15.193 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:15.193 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:15.193 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:15.193 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:24:15.193 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.978960+0000", 2024-09-13T11:24:15.193 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:15.193 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:15.193 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:15.193 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:24:15.193 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.978962+0000", 2024-09-13T11:24:15.194 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.46e-06 2024-09-13T11:24:15.194 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:15.194 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:15.194 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:24:15.194 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.979024+0000", 2024-09-13T11:24:15.194 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.1779999999999995e-05 2024-09-13T11:24:15.194 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:15.194 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:15.194 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:24:15.194 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.979037+0000", 2024-09-13T11:24:15.194 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2948e-05 2024-09-13T11:24:15.194 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:15.194 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:15.194 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:24:15.194 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.979048+0000", 2024-09-13T11:24:15.195 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1241e-05 2024-09-13T11:24:15.195 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:15.195 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:15.195 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:24:15.195 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:31.032951+0000", 2024-09-13T11:24:15.195 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.053903286000000002 2024-09-13T11:24:15.195 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:15.195 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:15.195 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-09-13T11:24:15.195 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:31.032958+0000", 2024-09-13T11:24:15.195 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.8109999999999997e-06 2024-09-13T11:24:15.195 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:15.195 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:15.195 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:24:15.196 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:31.933462+0000", 2024-09-13T11:24:15.196 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.90050463999999997 2024-09-13T11:24:15.196 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:15.196 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:15.196 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for degraded object", 2024-09-13T11:24:15.196 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:31.934668+0000", 2024-09-13T11:24:15.196 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0012053249999999999 2024-09-13T11:24:15.196 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:15.196 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:15.196 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:24:15.196 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:31.938887+0000", 2024-09-13T11:24:15.196 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0042187120000000003 2024-09-13T11:24:15.196 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:15.196 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:15.197 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:24:15.197 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:31.939149+0000", 2024-09-13T11:24:15.197 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000262725 2024-09-13T11:24:15.197 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:15.197 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:24:15.197 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:15.197 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:15.197 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:15.197 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPushReply(3.12 75/69 [PushReplyOp(3:4db63d39:::benchmark_data_smithi177_31116_object32641:head)])", 2024-09-13T11:24:15.197 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:31.646428+0000", 2024-09-13T11:24:15.197 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 43.511685452999998, 2024-09-13T11:24:15.197 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.28409169299999998, 2024-09-13T11:24:15.197 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:24:15.197 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:24:15.197 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-09-13T11:24:15.198 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:24:15.198 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:15.198 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:24:15.198 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:31.646428+0000", 2024-09-13T11:24:15.198 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:15.198 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:15.198 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:15.198 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:24:15.198 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:31.646428+0000", 2024-09-13T11:24:15.198 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:15.198 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:15.198 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:15.198 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:24:15.198 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:31.646431+0000", 2024-09-13T11:24:15.199 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.3149999999999999e-06 2024-09-13T11:24:15.199 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:15.199 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:15.199 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:24:15.199 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:31.646438+0000", 2024-09-13T11:24:15.199 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.7379999999999994e-06 2024-09-13T11:24:15.199 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:15.199 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:15.199 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:24:15.199 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:31.928787+0000", 2024-09-13T11:24:15.199 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.28234828699999998 2024-09-13T11:24:15.199 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:15.199 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:15.199 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:24:15.200 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:31.929066+0000", 2024-09-13T11:24:15.200 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00027956500000000001 2024-09-13T11:24:15.200 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:15.200 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:15.200 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:24:15.200 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:31.929216+0000", 2024-09-13T11:24:15.200 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00014963500000000001 2024-09-13T11:24:15.200 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:15.200 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:15.200 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:24:15.200 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:31.930520+0000", 2024-09-13T11:24:15.200 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001304153 2024-09-13T11:24:15.200 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:15.200 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:24:15.201 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:15.201 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:15.201 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:15.201 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPushReply(3.12 76/69 [PushReplyOp(3:4a5f7a67:::benchmark_data_smithi177_31116_object32993:head)])", 2024-09-13T11:24:15.201 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:32.069056+0000", 2024-09-13T11:24:15.201 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 43.089058172000001, 2024-09-13T11:24:15.201 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.17843624299999999, 2024-09-13T11:24:15.201 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:24:15.201 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:24:15.201 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-09-13T11:24:15.201 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:24:15.201 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:15.201 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:24:15.201 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.069056+0000", 2024-09-13T11:24:15.201 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:15.202 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:15.202 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:15.202 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:24:15.202 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.069056+0000", 2024-09-13T11:24:15.202 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:15.202 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:15.202 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:15.202 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:24:15.202 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.069057+0000", 2024-09-13T11:24:15.202 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4699999999999999e-06 2024-09-13T11:24:15.202 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:15.202 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:15.202 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:24:15.202 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.069065+0000", 2024-09-13T11:24:15.203 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.9740000000000002e-06 2024-09-13T11:24:15.203 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:15.203 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:15.203 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:24:15.203 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.246435+0000", 2024-09-13T11:24:15.203 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.17736979 2024-09-13T11:24:15.203 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:15.203 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:15.203 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:24:15.203 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.246450+0000", 2024-09-13T11:24:15.203 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4582e-05 2024-09-13T11:24:15.203 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:15.203 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:15.203 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:24:15.204 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.246556+0000", 2024-09-13T11:24:15.204 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00010608200000000001 2024-09-13T11:24:15.204 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:15.204 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:15.204 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:24:15.204 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.247492+0000", 2024-09-13T11:24:15.204 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00093634499999999999 2024-09-13T11:24:15.204 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:15.204 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:24:15.204 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:15.204 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:15.204 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:15.204 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPush(3.15 76/70 [PushOp(3:ab4c41a1:::benchmark_data_smithi177_31116_object23392:head, version: 55'1416, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:ab4c41a1:::benchmark_data_smithi177_31116_object23392:head@55'1416, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:24:15.204 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:32.723597+0000", 2024-09-13T11:24:15.205 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 42.434516862000002, 2024-09-13T11:24:15.205 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.205507309, 2024-09-13T11:24:15.205 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:24:15.205 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:24:15.205 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:24:15.205 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:24:15.205 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:15.205 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:24:15.205 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.723597+0000", 2024-09-13T11:24:15.205 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:15.205 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:15.205 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:15.205 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:24:15.205 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.723597+0000", 2024-09-13T11:24:15.206 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:15.206 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:15.206 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:15.206 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:24:15.206 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.723599+0000", 2024-09-13T11:24:15.206 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.1639999999999999e-06 2024-09-13T11:24:15.206 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:15.206 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:15.206 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:24:15.206 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.723661+0000", 2024-09-13T11:24:15.206 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.1851000000000001e-05 2024-09-13T11:24:15.206 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:15.206 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:15.206 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:24:15.206 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.927067+0000", 2024-09-13T11:24:15.207 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.20340633899999999 2024-09-13T11:24:15.207 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:15.207 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:15.207 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:24:15.207 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.927102+0000", 2024-09-13T11:24:15.207 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.4178999999999999e-05 2024-09-13T11:24:15.207 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:15.207 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:15.207 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:24:15.207 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.927424+0000", 2024-09-13T11:24:15.207 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000322536 2024-09-13T11:24:15.207 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:15.207 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:15.208 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:24:15.208 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.927457+0000", 2024-09-13T11:24:15.208 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.3191000000000001e-05 2024-09-13T11:24:15.208 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:15.208 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:15.208 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:24:15.208 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.929104+0000", 2024-09-13T11:24:15.208 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001647049 2024-09-13T11:24:15.208 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:15.208 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:24:15.208 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:15.208 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:15.208 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:15.208 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop_reply(client.4665.0:55795 3.12 e79/69 ondisk, result = 0)", 2024-09-13T11:24:15.209 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:35.893777+0000", 2024-09-13T11:24:15.209 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 39.264337228999999, 2024-09-13T11:24:15.209 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.189196325, 2024-09-13T11:24:15.209 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:24:15.209 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:24:15.209 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:24:15.209 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:24:15.209 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:15.209 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:24:15.209 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:35.893777+0000", 2024-09-13T11:24:15.209 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:15.209 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:15.209 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:15.209 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:24:15.210 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:35.893777+0000", 2024-09-13T11:24:15.210 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:15.210 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:15.210 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:15.210 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:24:15.210 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:35.893778+0000", 2024-09-13T11:24:15.210 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.6670000000000001e-06 2024-09-13T11:24:15.210 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:15.210 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:15.210 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:24:15.210 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:35.893784+0000", 2024-09-13T11:24:15.210 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.1989999999999997e-06 2024-09-13T11:24:15.210 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:15.210 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:15.210 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:24:15.211 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:36.082661+0000", 2024-09-13T11:24:15.211 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.18887743600000001 2024-09-13T11:24:15.211 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:15.211 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:15.211 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:24:15.211 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:36.082666+0000", 2024-09-13T11:24:15.211 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.2270000000000001e-06 2024-09-13T11:24:15.211 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:15.211 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:15.211 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:24:15.211 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:36.082784+0000", 2024-09-13T11:24:15.211 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00011776400000000001 2024-09-13T11:24:15.211 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:15.211 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:15.212 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:24:15.212 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:36.082799+0000", 2024-09-13T11:24:15.212 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5058999999999999e-05 2024-09-13T11:24:15.212 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:15.212 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:15.212 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:24:15.212 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:36.082973+0000", 2024-09-13T11:24:15.212 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00017397300000000001 2024-09-13T11:24:15.212 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:15.212 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:24:15.212 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:15.212 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:15.212 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:15.212 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPush(3.1f 84/78 [PushOp(3:fd5621e0:::benchmark_data_smithi177_31116_object7011:head, version: 46'483, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:fd5621e0:::benchmark_data_smithi177_31116_object7011:head@46'483, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:24:15.213 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:41.510296+0000", 2024-09-13T11:24:15.213 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 33.647817746000001, 2024-09-13T11:24:15.213 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.41419089999999997, 2024-09-13T11:24:15.213 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:24:15.213 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:24:15.213 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:24:15.213 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:24:15.213 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:15.213 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:24:15.213 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:41.510296+0000", 2024-09-13T11:24:15.213 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:15.213 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:15.213 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:15.213 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:24:15.213 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:41.510296+0000", 2024-09-13T11:24:15.214 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:15.214 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:15.214 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:15.214 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:24:15.214 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:41.510305+0000", 2024-09-13T11:24:15.214 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.9700000000000005e-06 2024-09-13T11:24:15.214 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:15.214 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:15.214 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:24:15.214 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:41.510458+0000", 2024-09-13T11:24:15.214 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000152418 2024-09-13T11:24:15.214 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:15.214 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:15.215 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:24:15.215 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:41.918225+0000", 2024-09-13T11:24:15.215 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.40776698 2024-09-13T11:24:15.215 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:15.215 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:15.215 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:24:15.215 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:41.918251+0000", 2024-09-13T11:24:15.215 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.6531000000000001e-05 2024-09-13T11:24:15.215 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:15.215 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:15.215 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:24:15.215 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:41.922211+0000", 2024-09-13T11:24:15.216 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0039603040000000004 2024-09-13T11:24:15.216 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:15.216 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:15.216 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:24:15.216 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:41.922253+0000", 2024-09-13T11:24:15.216 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.1155999999999997e-05 2024-09-13T11:24:15.216 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:15.216 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:15.216 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:24:15.216 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:41.924487+0000", 2024-09-13T11:24:15.216 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.002234541 2024-09-13T11:24:15.216 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:15.216 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:24:15.216 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:15.217 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:15.217 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:15.217 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPush(3.1f 97/78 [PushOp(3:fe604f47:::benchmark_data_smithi177_31116_object27978:head, version: 59'1753, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:fe604f47:::benchmark_data_smithi177_31116_object27978:head@59'1753, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:24:15.217 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:54.255485+0000", 2024-09-13T11:24:15.217 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 20.902629263000001, 2024-09-13T11:24:15.217 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.177312991, 2024-09-13T11:24:15.217 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:24:15.217 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:24:15.217 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:24:15.217 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:24:15.217 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:15.217 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:24:15.217 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:54.255485+0000", 2024-09-13T11:24:15.217 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:15.218 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:15.218 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:15.218 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:24:15.218 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:54.255485+0000", 2024-09-13T11:24:15.218 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:15.218 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:15.218 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:15.218 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:24:15.218 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:54.255488+0000", 2024-09-13T11:24:15.218 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.19e-06 2024-09-13T11:24:15.218 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:15.218 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:15.218 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:24:15.218 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:54.255596+0000", 2024-09-13T11:24:15.219 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000107964 2024-09-13T11:24:15.219 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:15.219 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:15.219 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:24:15.219 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:54.430551+0000", 2024-09-13T11:24:15.219 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.17495527999999999 2024-09-13T11:24:15.219 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:15.219 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:15.219 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:24:15.219 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:54.430592+0000", 2024-09-13T11:24:15.219 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.1384999999999998e-05 2024-09-13T11:24:15.219 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:15.219 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:15.219 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:24:15.220 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:54.430789+0000", 2024-09-13T11:24:15.220 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000196796 2024-09-13T11:24:15.220 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:15.220 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:15.220 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:24:15.220 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:54.430818+0000", 2024-09-13T11:24:15.220 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.8979000000000002e-05 2024-09-13T11:24:15.220 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:15.220 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:15.220 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:24:15.220 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:54.432798+0000", 2024-09-13T11:24:15.220 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0019793969999999999 2024-09-13T11:24:15.220 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:15.220 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:24:15.221 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:15.221 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:15.221 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:15.221 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:79480 3.2 3:42e4e644:::benchmark_data_smithi177_31116_object79479:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101)", 2024-09-13T11:24:15.221 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:24:00.046305+0000", 2024-09-13T11:24:15.221 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 15.111809267, 2024-09-13T11:24:15.221 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.15557262599999999, 2024-09-13T11:24:15.221 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:24:15.221 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:24:15.221 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:15.221 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:24:15.221 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:24:15.221 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:15.221 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 79480 2024-09-13T11:24:15.222 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:15.222 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:24:15.222 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:15.222 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:24:15.222 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:00.046305+0000", 2024-09-13T11:24:15.222 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:15.222 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:15.222 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:15.222 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:24:15.222 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:00.046305+0000", 2024-09-13T11:24:15.222 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:15.222 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:15.222 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:15.222 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:24:15.223 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:00.046307+0000", 2024-09-13T11:24:15.223 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.2390000000000001e-06 2024-09-13T11:24:15.223 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:15.223 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:15.223 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:24:15.223 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:00.046344+0000", 2024-09-13T11:24:15.223 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.7477e-05 2024-09-13T11:24:15.223 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:15.223 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:15.223 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:24:15.223 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:00.046362+0000", 2024-09-13T11:24:15.224 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.7167e-05 2024-09-13T11:24:15.224 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:15.224 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:15.224 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:24:15.224 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:00.046368+0000", 2024-09-13T11:24:15.224 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.7390000000000002e-06 2024-09-13T11:24:15.224 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:15.224 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:15.224 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:24:15.225 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:00.046418+0000", 2024-09-13T11:24:15.225 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.9959000000000001e-05 2024-09-13T11:24:15.225 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:15.225 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:15.225 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:24:15.225 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:00.046504+0000", 2024-09-13T11:24:15.225 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.6155000000000004e-05 2024-09-13T11:24:15.225 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:15.225 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:15.225 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 12", 2024-09-13T11:24:15.225 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:00.046588+0000", 2024-09-13T11:24:15.225 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.3258999999999997e-05 2024-09-13T11:24:15.225 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:15.225 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:15.226 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:24:15.226 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:00.047803+0000", 2024-09-13T11:24:15.226 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0012155810000000001 2024-09-13T11:24:15.226 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:15.226 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:15.226 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:15.226 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:00.201685+0000", 2024-09-13T11:24:15.226 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.15388138300000001 2024-09-13T11:24:15.226 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:15.226 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:15.226 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:24:15.226 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:00.201796+0000", 2024-09-13T11:24:15.226 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000110939 2024-09-13T11:24:15.226 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:15.227 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:15.227 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:24:15.227 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:00.201877+0000", 2024-09-13T11:24:15.227 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.1728000000000001e-05 2024-09-13T11:24:15.227 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:15.227 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:24:15.227 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:15.227 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:15.227 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:15.227 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:84139 3.2 3:41ddbd59:::benchmark_data_smithi177_31116_object84138:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-09-13T11:24:15.227 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:24:05.428921+0000", 2024-09-13T11:24:15.227 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 9.7291928579999993, 2024-09-13T11:24:15.227 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.203750139, 2024-09-13T11:24:15.228 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:24:15.228 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:24:15.228 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:15.228 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:24:15.228 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:24:15.228 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:15.228 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 84139 2024-09-13T11:24:15.228 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:15.228 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:24:15.229 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:15.229 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:24:15.229 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.428921+0000", 2024-09-13T11:24:15.229 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:15.229 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:15.229 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:15.229 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:24:15.229 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.428921+0000", 2024-09-13T11:24:15.229 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:15.229 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:15.229 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:15.229 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:24:15.229 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.428926+0000", 2024-09-13T11:24:15.230 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.7299999999999996e-06 2024-09-13T11:24:15.230 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:15.230 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:15.230 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:24:15.230 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.428966+0000", 2024-09-13T11:24:15.230 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.0629e-05 2024-09-13T11:24:15.230 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:15.230 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:15.230 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:24:15.230 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.428978+0000", 2024-09-13T11:24:15.230 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1749999999999999e-05 2024-09-13T11:24:15.230 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:15.230 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:15.231 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:24:15.231 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.428987+0000", 2024-09-13T11:24:15.231 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.0909999999999999e-06 2024-09-13T11:24:15.231 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:15.231 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:15.231 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:24:15.231 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.429055+0000", 2024-09-13T11:24:15.231 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.7937999999999999e-05 2024-09-13T11:24:15.231 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:15.231 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:15.231 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:24:15.231 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.429240+0000", 2024-09-13T11:24:15.231 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00018448499999999999 2024-09-13T11:24:15.232 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:15.232 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:15.232 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 12", 2024-09-13T11:24:15.232 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.429423+0000", 2024-09-13T11:24:15.232 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000183012 2024-09-13T11:24:15.232 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:15.232 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:15.232 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:24:15.232 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.433639+0000", 2024-09-13T11:24:15.232 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0042158539999999998 2024-09-13T11:24:15.232 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:15.232 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:15.232 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:15.232 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.632566+0000", 2024-09-13T11:24:15.233 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.198927885 2024-09-13T11:24:15.233 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:15.233 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:15.233 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:24:15.233 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.632622+0000", 2024-09-13T11:24:15.233 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.5924000000000002e-05 2024-09-13T11:24:15.233 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:15.233 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:15.233 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:24:15.233 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.632671+0000", 2024-09-13T11:24:15.233 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.8841000000000002e-05 2024-09-13T11:24:15.233 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:15.233 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:24:15.233 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:15.234 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:15.234 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:15.234 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop_reply(client.4665.0:84139 3.2 e107/75 ondisk, result = 0)", 2024-09-13T11:24:15.234 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:24:05.431065+0000", 2024-09-13T11:24:15.234 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 9.7270493420000008, 2024-09-13T11:24:15.234 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.20161405499999999, 2024-09-13T11:24:15.234 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:24:15.234 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:24:15.234 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:24:15.234 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:24:15.234 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:15.234 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:24:15.234 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.431065+0000", 2024-09-13T11:24:15.235 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:15.235 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:15.235 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:15.235 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:24:15.235 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.431065+0000", 2024-09-13T11:24:15.235 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:15.235 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:15.235 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:15.235 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:24:15.235 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.431066+0000", 2024-09-13T11:24:15.235 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.454e-06 2024-09-13T11:24:15.235 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:15.235 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:15.235 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:24:15.236 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.431071+0000", 2024-09-13T11:24:15.236 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.5820000000000003e-06 2024-09-13T11:24:15.236 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:15.236 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:15.236 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:24:15.236 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.632499+0000", 2024-09-13T11:24:15.236 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.20142855700000001 2024-09-13T11:24:15.236 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:15.236 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:15.236 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:24:15.236 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.632507+0000", 2024-09-13T11:24:15.236 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.011e-06 2024-09-13T11:24:15.236 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:15.236 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:15.237 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:24:15.237 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.632553+0000", 2024-09-13T11:24:15.237 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.6326000000000003e-05 2024-09-13T11:24:15.237 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:15.237 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:15.237 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:24:15.237 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.632561+0000", 2024-09-13T11:24:15.237 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.4250000000000001e-06 2024-09-13T11:24:15.237 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:15.237 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:15.237 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:24:15.237 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.632679+0000", 2024-09-13T11:24:15.237 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0001177 2024-09-13T11:24:15.237 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:15.237 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:24:15.238 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:15.238 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:15.238 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:15.238 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:84707 3.2 3:4107e60f:::benchmark_data_smithi177_31116_object84706:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-09-13T11:24:15.238 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:24:05.810565+0000", 2024-09-13T11:24:15.238 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 9.3475493549999999, 2024-09-13T11:24:15.238 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.16585786299999999, 2024-09-13T11:24:15.238 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:24:15.238 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:24:15.238 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:15.238 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:24:15.238 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:24:15.238 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:15.238 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 84707 2024-09-13T11:24:15.239 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:15.239 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:24:15.239 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:15.239 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:24:15.239 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.810565+0000", 2024-09-13T11:24:15.239 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:15.239 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:15.239 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:15.239 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:24:15.239 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.810565+0000", 2024-09-13T11:24:15.239 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:15.239 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:15.239 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:15.239 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:24:15.239 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.810568+0000", 2024-09-13T11:24:15.240 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.1970000000000001e-06 2024-09-13T11:24:15.240 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:15.240 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:15.240 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:24:15.240 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.810620+0000", 2024-09-13T11:24:15.240 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.2193e-05 2024-09-13T11:24:15.240 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:15.240 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:15.240 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:24:15.240 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.810645+0000", 2024-09-13T11:24:15.240 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.5491e-05 2024-09-13T11:24:15.240 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:15.240 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:15.240 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:24:15.241 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.810660+0000", 2024-09-13T11:24:15.241 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4620000000000001e-05 2024-09-13T11:24:15.241 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:15.241 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:15.241 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:24:15.241 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.810733+0000", 2024-09-13T11:24:15.241 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.2935999999999994e-05 2024-09-13T11:24:15.241 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:15.241 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:15.241 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:24:15.241 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.810845+0000", 2024-09-13T11:24:15.241 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000112259 2024-09-13T11:24:15.241 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:15.242 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:15.242 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 12", 2024-09-13T11:24:15.242 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.810960+0000", 2024-09-13T11:24:15.242 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00011454400000000001 2024-09-13T11:24:15.242 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:15.242 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:15.242 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:24:15.242 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.813269+0000", 2024-09-13T11:24:15.242 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0023093530000000001 2024-09-13T11:24:15.242 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:15.242 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:15.242 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:15.242 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.976324+0000", 2024-09-13T11:24:15.242 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.16305488200000001 2024-09-13T11:24:15.243 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:15.243 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:15.243 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:24:15.243 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.976380+0000", 2024-09-13T11:24:15.243 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.6403999999999997e-05 2024-09-13T11:24:15.243 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:15.243 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:15.243 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:24:15.243 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.976422+0000", 2024-09-13T11:24:15.243 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.1984000000000003e-05 2024-09-13T11:24:15.243 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:15.243 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:24:15.244 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:15.244 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:15.244 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:15.244 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop_reply(client.4665.0:84707 3.2 e107/75 ondisk, result = 0)", 2024-09-13T11:24:15.244 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:24:05.812491+0000", 2024-09-13T11:24:15.244 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 9.3456229220000004, 2024-09-13T11:24:15.244 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.16393901399999999, 2024-09-13T11:24:15.244 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:24:15.244 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:24:15.244 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:24:15.244 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:24:15.244 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:15.244 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:24:15.245 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.812491+0000", 2024-09-13T11:24:15.245 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:15.245 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:15.245 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:15.245 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:24:15.245 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.812491+0000", 2024-09-13T11:24:15.245 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:15.245 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:15.245 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:15.245 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:24:15.245 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.812496+0000", 2024-09-13T11:24:15.245 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.2510000000000002e-06 2024-09-13T11:24:15.245 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:15.245 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:15.246 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:24:15.246 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.812500+0000", 2024-09-13T11:24:15.246 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.6770000000000001e-06 2024-09-13T11:24:15.246 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:15.246 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:15.246 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:24:15.246 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.976256+0000", 2024-09-13T11:24:15.246 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.163756549 2024-09-13T11:24:15.246 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:15.246 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:15.246 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:24:15.246 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.976261+0000", 2024-09-13T11:24:15.246 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.5229999999999997e-06 2024-09-13T11:24:15.246 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:15.247 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:15.247 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:24:15.247 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.976310+0000", 2024-09-13T11:24:15.247 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.8677999999999997e-05 2024-09-13T11:24:15.247 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:15.247 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:15.247 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:24:15.247 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.976318+0000", 2024-09-13T11:24:15.247 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.3659999999999992e-06 2024-09-13T11:24:15.247 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:15.247 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:15.247 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:24:15.247 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.976430+0000", 2024-09-13T11:24:15.247 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00011197 2024-09-13T11:24:15.247 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:15.248 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:24:15.248 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:15.248 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:15.248 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:15.248 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:84879 3.2 3:40162d9f:::benchmark_data_smithi177_31116_object84878:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108)", 2024-09-13T11:24:15.248 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:24:05.973920+0000", 2024-09-13T11:24:15.248 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 9.1841942969999995, 2024-09-13T11:24:15.248 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.17611384699999999, 2024-09-13T11:24:15.248 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:24:15.248 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:24:15.248 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:15.248 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:24:15.248 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:24:15.249 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:15.249 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 84879 2024-09-13T11:24:15.249 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:15.249 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:24:15.249 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:15.249 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:24:15.249 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.973920+0000", 2024-09-13T11:24:15.249 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:15.249 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:15.249 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:15.249 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:24:15.249 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.973920+0000", 2024-09-13T11:24:15.249 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:15.249 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:15.250 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:15.250 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:24:15.250 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.973921+0000", 2024-09-13T11:24:15.250 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5179999999999999e-06 2024-09-13T11:24:15.250 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:15.250 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:15.250 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:24:15.250 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.974001+0000", 2024-09-13T11:24:15.250 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.9887000000000003e-05 2024-09-13T11:24:15.250 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:15.250 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:15.250 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:24:15.250 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.974010+0000", 2024-09-13T11:24:15.250 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.4329999999999993e-06 2024-09-13T11:24:15.251 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:15.251 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:15.251 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:24:15.251 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.974039+0000", 2024-09-13T11:24:15.251 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.9037e-05 2024-09-13T11:24:15.251 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:15.251 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:15.251 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:24:15.251 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.974104+0000", 2024-09-13T11:24:15.251 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.4349000000000003e-05 2024-09-13T11:24:15.251 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:15.251 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:15.251 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:24:15.251 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.974190+0000", 2024-09-13T11:24:15.252 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.6073000000000001e-05 2024-09-13T11:24:15.252 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:15.252 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:15.252 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 12", 2024-09-13T11:24:15.252 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.974266+0000", 2024-09-13T11:24:15.252 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.6205000000000006e-05 2024-09-13T11:24:15.252 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:15.252 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:15.252 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:24:15.252 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.975441+0000", 2024-09-13T11:24:15.252 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001174418 2024-09-13T11:24:15.252 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:15.252 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:15.252 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:15.253 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:06.149950+0000", 2024-09-13T11:24:15.253 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.17450983 2024-09-13T11:24:15.253 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:15.253 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:15.253 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:24:15.253 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:06.149988+0000", 2024-09-13T11:24:15.253 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.8136999999999998e-05 2024-09-13T11:24:15.253 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:15.253 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:15.253 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:24:15.253 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:06.150033+0000", 2024-09-13T11:24:15.253 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.4960000000000003e-05 2024-09-13T11:24:15.253 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:15.254 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:24:15.254 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:15.254 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:15.254 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:15.254 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop_reply(client.4665.0:84879 3.2 e108/75 ondisk, result = 0)", 2024-09-13T11:24:15.254 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:24:05.976234+0000", 2024-09-13T11:24:15.254 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 9.1818799949999992, 2024-09-13T11:24:15.254 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.17380630999999999, 2024-09-13T11:24:15.254 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:24:15.254 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:24:15.254 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:24:15.254 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:24:15.254 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:15.254 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:24:15.255 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.976234+0000", 2024-09-13T11:24:15.255 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:15.255 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:15.255 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:15.255 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:24:15.255 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.976234+0000", 2024-09-13T11:24:15.255 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:15.255 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:15.255 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:15.255 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:24:15.255 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.976235+0000", 2024-09-13T11:24:15.255 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.545e-06 2024-09-13T11:24:15.255 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:15.255 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:15.256 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:24:15.256 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.976245+0000", 2024-09-13T11:24:15.256 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.0969999999999995e-06 2024-09-13T11:24:15.256 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:15.256 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:15.256 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:24:15.256 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:06.149889+0000", 2024-09-13T11:24:15.256 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.173644405 2024-09-13T11:24:15.256 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:15.256 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:15.256 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:24:15.256 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:06.149894+0000", 2024-09-13T11:24:15.256 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.6979999999999997e-06 2024-09-13T11:24:15.257 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:15.257 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:15.257 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:24:15.257 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:06.149941+0000", 2024-09-13T11:24:15.257 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.6968000000000002e-05 2024-09-13T11:24:15.257 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:15.257 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:15.257 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:24:15.257 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:06.149946+0000", 2024-09-13T11:24:15.257 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.7780000000000004e-06 2024-09-13T11:24:15.257 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:15.257 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:15.257 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:24:15.258 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:06.150040+0000", 2024-09-13T11:24:15.258 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.3819000000000004e-05 2024-09-13T11:24:15.258 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:15.258 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:24:15.258 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:15.258 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:15.258 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:15.258 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.4665.0:91603 3.14 e111/84 3:2d13add5:::benchmark_data_smithi177_31116_object91602:head v 111'4262, mlcod=110'4259)", 2024-09-13T11:24:15.258 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:24:11.037172+0000", 2024-09-13T11:24:15.258 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 4.120941406, 2024-09-13T11:24:15.258 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.16471824500000001, 2024-09-13T11:24:15.258 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:24:15.258 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:24:15.258 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:15.259 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:24:15.259 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:15.259 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:24:15.259 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:11.037172+0000", 2024-09-13T11:24:15.259 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:15.259 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:15.259 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:15.259 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:24:15.259 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:11.037172+0000", 2024-09-13T11:24:15.260 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:15.260 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:15.260 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:15.260 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:24:15.260 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:11.037175+0000", 2024-09-13T11:24:15.260 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.2469999999999998e-06 2024-09-13T11:24:15.260 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:15.260 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:15.260 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:24:15.260 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:11.037245+0000", 2024-09-13T11:24:15.260 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.0653000000000003e-05 2024-09-13T11:24:15.260 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:15.260 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:15.260 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:24:15.261 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:11.197835+0000", 2024-09-13T11:24:15.261 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.16058927000000001 2024-09-13T11:24:15.261 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:15.261 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:15.261 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:24:15.261 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:11.197850+0000", 2024-09-13T11:24:15.261 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.56e-05 2024-09-13T11:24:15.261 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:15.261 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:15.261 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:24:15.261 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:11.197961+0000", 2024-09-13T11:24:15.261 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000110852 2024-09-13T11:24:15.261 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:15.261 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:15.262 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:24:15.262 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:11.198013+0000", 2024-09-13T11:24:15.262 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.1554000000000001e-05 2024-09-13T11:24:15.262 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:15.262 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:15.262 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:24:15.262 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:11.201796+0000", 2024-09-13T11:24:15.262 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0037835780000000001 2024-09-13T11:24:15.262 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:15.262 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:15.262 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:24:15.262 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:11.201891+0000", 2024-09-13T11:24:15.262 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.4491e-05 2024-09-13T11:24:15.262 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:15.263 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:24:15.263 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:15.263 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:15.263 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:24:15.263 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-13T11:24:15.264 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-13T11:24:15.265 INFO:tasks.ceph.osd.9.smithi092.stderr:2024-09-13T11:24:15.259+0000 7ff5ab3d0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26348) UID: 0 2024-09-13T11:24:15.365 INFO:tasks.ceph.osd.4.smithi001.stderr:2024-09-13T11:24:15.359+0000 7ff8e0218640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26310) UID: 0 2024-09-13T11:24:15.407 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-09-13T11:24:15.408 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [], 2024-09-13T11:24:15.408 INFO:teuthology.orchestra.run.smithi177.stdout: "num_ops": 0 2024-09-13T11:24:15.408 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-09-13T11:24:15.427 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-13T11:24:15.476 INFO:tasks.ceph.osd.11.smithi177.stderr:2024-09-13T11:24:15.475+0000 7f24792c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26580) UID: 0 2024-09-13T11:24:15.558 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-09-13T11:24:15.558 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [], 2024-09-13T11:24:15.558 INFO:teuthology.orchestra.run.smithi177.stdout: "complaint_time": 30, 2024-09-13T11:24:15.558 INFO:teuthology.orchestra.run.smithi177.stdout: "num_blocked_ops": 0 2024-09-13T11:24:15.558 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-09-13T11:24:15.567 INFO:tasks.ceph.osd.2.smithi162.stderr:2024-09-13T11:24:15.564+0000 7fe92b35a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26313) UID: 0 2024-09-13T11:24:15.575 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-13T11:24:15.668 INFO:tasks.ceph.osd.13.smithi092.stderr:2024-09-13T11:24:15.663+0000 7f3e5afea640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26350) UID: 0 2024-09-13T11:24:15.755 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-09-13T11:24:15.755 INFO:teuthology.orchestra.run.smithi177.stdout: "size": 20, 2024-09-13T11:24:15.755 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 600, 2024-09-13T11:24:15.755 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [ 2024-09-13T11:24:15.755 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:15.755 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:30852 3.a 3:522fb0d3:::benchmark_data_smithi177_31116_object30851:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e59)", 2024-09-13T11:24:15.755 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:09.538712+0000", 2024-09-13T11:24:15.755 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 66.213802001000005, 2024-09-13T11:24:15.755 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.19661263300000001, 2024-09-13T11:24:15.755 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:15.755 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:15.756 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:15.756 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:24:15.756 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:24:15.756 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:15.756 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 30852 2024-09-13T11:24:15.756 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:15.756 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:15.756 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:15.756 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:15.756 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.538712+0000", 2024-09-13T11:24:15.756 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:15.756 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:15.757 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:15.757 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:15.757 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.538712+0000", 2024-09-13T11:24:15.757 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:15.757 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:15.757 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:15.757 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:15.757 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.538714+0000", 2024-09-13T11:24:15.757 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.658e-06 2024-09-13T11:24:15.757 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:15.757 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:15.757 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:15.757 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.538735+0000", 2024-09-13T11:24:15.757 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.141e-05 2024-09-13T11:24:15.758 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:15.758 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:15.758 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:15.760 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.538747+0000", 2024-09-13T11:24:15.760 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.2258e-05 2024-09-13T11:24:15.760 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:15.760 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:15.760 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:15.760 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.538754+0000", 2024-09-13T11:24:15.760 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.2859999999999996e-06 2024-09-13T11:24:15.761 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:15.761 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:15.761 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:15.761 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.538798+0000", 2024-09-13T11:24:15.761 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.4258000000000002e-05 2024-09-13T11:24:15.762 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:15.763 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:15.763 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:24:15.763 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.538890+0000", 2024-09-13T11:24:15.763 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.1676999999999997e-05 2024-09-13T11:24:15.763 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:15.764 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:15.764 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 7", 2024-09-13T11:24:15.764 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.538991+0000", 2024-09-13T11:24:15.764 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00010173700000000001 2024-09-13T11:24:15.764 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:15.764 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:15.764 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:24:15.765 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.540171+0000", 2024-09-13T11:24:15.765 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0011800479999999999 2024-09-13T11:24:15.765 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:15.765 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:15.765 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:15.765 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.735031+0000", 2024-09-13T11:24:15.765 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.194859798 2024-09-13T11:24:15.766 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:15.766 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:15.766 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:24:15.766 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.735197+0000", 2024-09-13T11:24:15.766 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00016569499999999999 2024-09-13T11:24:15.766 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:15.766 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:15.766 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:24:15.766 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.735325+0000", 2024-09-13T11:24:15.766 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00012780800000000001 2024-09-13T11:24:15.766 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:15.767 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:15.767 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:15.767 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:15.767 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:15.767 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop_reply(client.4665.0:30852 3.a e59/30 ondisk, result = 0)", 2024-09-13T11:24:15.767 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:09.540458+0000", 2024-09-13T11:24:15.767 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 66.212056512999993, 2024-09-13T11:24:15.767 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.194887, 2024-09-13T11:24:15.767 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:15.767 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:15.767 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:24:15.767 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:15.767 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:15.767 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:15.768 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.540458+0000", 2024-09-13T11:24:15.768 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:15.768 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:15.768 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:15.768 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:15.768 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.540458+0000", 2024-09-13T11:24:15.768 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:15.768 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:15.769 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:15.769 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:15.769 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.540459+0000", 2024-09-13T11:24:15.769 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.2699999999999999e-06 2024-09-13T11:24:15.769 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:15.769 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:15.769 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:15.769 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.540463+0000", 2024-09-13T11:24:15.770 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.8500000000000004e-06 2024-09-13T11:24:15.770 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:15.770 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:15.770 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:15.770 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.734813+0000", 2024-09-13T11:24:15.770 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.194350047 2024-09-13T11:24:15.770 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:15.770 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:15.770 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:15.771 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.734832+0000", 2024-09-13T11:24:15.771 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.9610999999999999e-05 2024-09-13T11:24:15.771 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:15.771 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:15.771 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:15.771 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.734989+0000", 2024-09-13T11:24:15.771 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00015694799999999999 2024-09-13T11:24:15.771 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:15.771 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:15.771 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:24:15.771 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.735015+0000", 2024-09-13T11:24:15.771 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.6165000000000001e-05 2024-09-13T11:24:15.771 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:15.771 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:15.772 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:24:15.772 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.735345+0000", 2024-09-13T11:24:15.772 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00032910900000000001 2024-09-13T11:24:15.772 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:15.772 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:15.772 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:15.772 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:15.772 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:15.773 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPush(3.17 75/70 [PushOp(3:eda94455:::benchmark_data_smithi177_31116_object13017:head, version: 50'788, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:eda94455:::benchmark_data_smithi177_31116_object13017:head@50'788, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:24:15.773 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:31.038029+0000", 2024-09-13T11:24:15.773 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 44.714485447999998, 2024-09-13T11:24:15.773 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.42064623499999998, 2024-09-13T11:24:15.773 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:15.773 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:15.774 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:24:15.774 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:15.774 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:15.774 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:15.774 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.038029+0000", 2024-09-13T11:24:15.774 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:15.775 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:15.775 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:15.775 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:15.775 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.038029+0000", 2024-09-13T11:24:15.775 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:15.775 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:15.776 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:15.776 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:15.776 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.038030+0000", 2024-09-13T11:24:15.776 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.499e-06 2024-09-13T11:24:15.776 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:15.776 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:15.776 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:15.777 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.038088+0000", 2024-09-13T11:24:15.777 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.7771e-05 2024-09-13T11:24:15.777 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:15.777 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:15.778 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:15.778 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.457025+0000", 2024-09-13T11:24:15.778 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.41893754900000002 2024-09-13T11:24:15.778 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:15.778 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:15.778 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:15.778 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.457073+0000", 2024-09-13T11:24:15.778 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.7450999999999997e-05 2024-09-13T11:24:15.779 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:15.779 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:15.779 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:15.779 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.457304+0000", 2024-09-13T11:24:15.779 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000231351 2024-09-13T11:24:15.779 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:15.779 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:15.779 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:24:15.779 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.457331+0000", 2024-09-13T11:24:15.779 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.6372000000000001e-05 2024-09-13T11:24:15.780 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:15.780 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:15.780 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:24:15.780 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.458675+0000", 2024-09-13T11:24:15.780 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001344242 2024-09-13T11:24:15.780 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:15.780 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:15.780 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:15.780 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:15.780 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:15.780 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "rep_scrubmap(3.a e75 from shard 7)", 2024-09-13T11:24:15.780 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:32.151191+0000", 2024-09-13T11:24:15.781 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 43.601323395000001, 2024-09-13T11:24:15.781 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.164608173, 2024-09-13T11:24:15.781 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:15.781 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:15.781 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:24:15.781 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:15.781 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:15.781 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:15.781 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:32.151191+0000", 2024-09-13T11:24:15.781 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:15.781 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:15.782 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:15.782 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:15.782 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:32.151191+0000", 2024-09-13T11:24:15.782 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:15.782 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:15.782 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:15.783 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:15.783 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:32.151192+0000", 2024-09-13T11:24:15.783 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.28e-06 2024-09-13T11:24:15.783 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:15.783 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:15.783 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:15.783 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:32.151220+0000", 2024-09-13T11:24:15.783 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.8045e-05 2024-09-13T11:24:15.783 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:15.783 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:15.783 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:15.784 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:32.315480+0000", 2024-09-13T11:24:15.784 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.16425959000000001 2024-09-13T11:24:15.784 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:15.784 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:15.784 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:15.784 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:32.315488+0000", 2024-09-13T11:24:15.784 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.0819999999999999e-06 2024-09-13T11:24:15.784 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:15.784 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:15.784 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:15.784 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:32.315558+0000", 2024-09-13T11:24:15.784 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.0500999999999995e-05 2024-09-13T11:24:15.785 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:15.785 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:15.785 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:24:15.785 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:32.315570+0000", 2024-09-13T11:24:15.785 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.2238999999999999e-05 2024-09-13T11:24:15.785 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:15.785 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:15.785 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:24:15.785 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:32.315799+0000", 2024-09-13T11:24:15.785 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00022843599999999999 2024-09-13T11:24:15.785 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:15.786 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:15.786 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:15.786 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:15.786 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:15.786 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPush(3.14 79/70 [PushOp(3:2c86f72b:::benchmark_data_smithi177_31116_object32132:head, version: 60'2019, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:2c86f72b:::benchmark_data_smithi177_31116_object32132:head@60'2019, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:24:15.786 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:35.768489+0000", 2024-09-13T11:24:15.786 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 39.984025230999997, 2024-09-13T11:24:15.786 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.179450581, 2024-09-13T11:24:15.786 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:15.786 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:15.786 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:24:15.786 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:15.787 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:15.787 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:15.787 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:35.768489+0000", 2024-09-13T11:24:15.787 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:15.787 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:15.787 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:15.787 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:15.787 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:35.768489+0000", 2024-09-13T11:24:15.787 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:15.787 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:15.787 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:15.787 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:15.788 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:35.768490+0000", 2024-09-13T11:24:15.788 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5999999999999999e-06 2024-09-13T11:24:15.788 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:15.788 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:15.788 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:15.788 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:35.768592+0000", 2024-09-13T11:24:15.788 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00010194799999999999 2024-09-13T11:24:15.788 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:15.788 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:15.789 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:15.789 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:35.946933+0000", 2024-09-13T11:24:15.789 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.17834060800000001 2024-09-13T11:24:15.789 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:15.789 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:15.789 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:15.789 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:35.946953+0000", 2024-09-13T11:24:15.789 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.0347e-05 2024-09-13T11:24:15.789 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:15.789 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:15.789 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:15.789 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:35.947056+0000", 2024-09-13T11:24:15.790 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000103164 2024-09-13T11:24:15.790 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:15.790 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:15.790 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:24:15.790 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:35.947071+0000", 2024-09-13T11:24:15.790 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.4854e-05 2024-09-13T11:24:15.790 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:15.790 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:15.790 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:24:15.790 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:35.947939+0000", 2024-09-13T11:24:15.790 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00086806000000000003 2024-09-13T11:24:15.790 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:15.791 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:15.791 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:15.791 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:15.791 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:15.791 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPush(3.17 83/70 [PushOp(3:eb574268:::benchmark_data_smithi177_31116_object40622:head, version: 64'2475, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:eb574268:::benchmark_data_smithi177_31116_object40622:head@64'2475, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:24:15.791 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:40.760042+0000", 2024-09-13T11:24:15.791 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 34.992471731000002, 2024-09-13T11:24:15.791 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.30544642100000002, 2024-09-13T11:24:15.791 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:15.791 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:15.791 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:24:15.791 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:15.792 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:15.792 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:15.792 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:40.760042+0000", 2024-09-13T11:24:15.792 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:15.792 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:15.793 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:15.793 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:15.793 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:40.760042+0000", 2024-09-13T11:24:15.793 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:15.793 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:15.793 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:15.793 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:15.793 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:40.760044+0000", 2024-09-13T11:24:15.793 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.511e-06 2024-09-13T11:24:15.793 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:15.793 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:15.794 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:15.794 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:40.760149+0000", 2024-09-13T11:24:15.794 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000105473 2024-09-13T11:24:15.794 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:15.794 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:15.794 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:15.794 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.064465+0000", 2024-09-13T11:24:15.794 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.30431613800000001 2024-09-13T11:24:15.794 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:15.794 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:15.794 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:15.794 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.064479+0000", 2024-09-13T11:24:15.795 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3586e-05 2024-09-13T11:24:15.795 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:15.795 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:15.795 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:15.795 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.064553+0000", 2024-09-13T11:24:15.795 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.3646000000000006e-05 2024-09-13T11:24:15.796 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:15.796 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:15.796 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:24:15.796 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.064568+0000", 2024-09-13T11:24:15.796 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.52e-05 2024-09-13T11:24:15.796 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:15.796 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:15.796 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:24:15.796 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.065489+0000", 2024-09-13T11:24:15.796 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00092086700000000002 2024-09-13T11:24:15.797 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:15.797 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:15.797 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:15.797 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:15.797 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:15.797 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPushReply(3.1a 83/78 [PushReplyOp(3:5c3834bd:::benchmark_data_smithi177_31116_object8669:head)])", 2024-09-13T11:24:15.797 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:40.798236+0000", 2024-09-13T11:24:15.797 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 34.954278301000002, 2024-09-13T11:24:15.797 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.29701520999999997, 2024-09-13T11:24:15.798 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:15.798 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:15.798 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-13T11:24:15.798 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:15.798 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:15.798 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:15.798 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:40.798236+0000", 2024-09-13T11:24:15.798 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:15.798 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:15.798 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:15.798 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:15.799 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:40.798236+0000", 2024-09-13T11:24:15.799 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:15.799 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:15.799 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:15.799 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:15.799 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:40.798237+0000", 2024-09-13T11:24:15.799 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.0500000000000002e-07 2024-09-13T11:24:15.799 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:15.799 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:15.799 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:15.799 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:40.798240+0000", 2024-09-13T11:24:15.799 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.4819999999999999e-06 2024-09-13T11:24:15.800 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:15.800 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:15.800 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:15.800 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.093360+0000", 2024-09-13T11:24:15.800 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.29512009 2024-09-13T11:24:15.800 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:15.800 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:15.800 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:15.800 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.093372+0000", 2024-09-13T11:24:15.800 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.2221e-05 2024-09-13T11:24:15.800 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:15.800 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:15.801 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:15.801 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.093482+0000", 2024-09-13T11:24:15.801 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000109571 2024-09-13T11:24:15.801 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:15.801 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:15.801 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:24:15.801 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.095251+0000", 2024-09-13T11:24:15.801 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001768941 2024-09-13T11:24:15.801 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:15.801 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:15.801 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:15.801 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:15.802 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:15.802 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPushReply(3.1a 83/78 [PushReplyOp(3:58c7ffc0:::benchmark_data_smithi177_31116_object8738:head)])", 2024-09-13T11:24:15.802 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:40.805941+0000", 2024-09-13T11:24:15.802 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 34.946573047000001, 2024-09-13T11:24:15.802 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.30315678499999998, 2024-09-13T11:24:15.802 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:15.802 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:15.802 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-13T11:24:15.802 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:15.802 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:15.803 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:15.803 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:40.805941+0000", 2024-09-13T11:24:15.803 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:15.803 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:15.803 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:15.803 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:15.803 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:40.805941+0000", 2024-09-13T11:24:15.803 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:15.803 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:15.804 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:15.804 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:15.804 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:40.805942+0000", 2024-09-13T11:24:15.804 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3769999999999999e-06 2024-09-13T11:24:15.804 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:15.804 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:15.804 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:15.804 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:40.805947+0000", 2024-09-13T11:24:15.804 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.8980000000000004e-06 2024-09-13T11:24:15.804 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:15.805 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:15.805 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:15.805 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.106505+0000", 2024-09-13T11:24:15.805 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.30055777700000003 2024-09-13T11:24:15.805 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:15.805 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:15.805 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:15.805 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.106512+0000", 2024-09-13T11:24:15.805 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.3749999999999997e-06 2024-09-13T11:24:15.806 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:15.806 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:15.806 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:15.806 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.106566+0000", 2024-09-13T11:24:15.806 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.3103999999999998e-05 2024-09-13T11:24:15.806 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:15.807 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:15.807 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:24:15.807 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.109098+0000", 2024-09-13T11:24:15.807 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0025322539999999998 2024-09-13T11:24:15.807 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:15.807 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:15.808 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:15.808 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:15.808 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:15.808 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPushReply(3.1a 83/78 [PushReplyOp(3:58c7ffc0:::benchmark_data_smithi177_31116_object8738:head)])", 2024-09-13T11:24:15.808 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:40.805973+0000", 2024-09-13T11:24:15.808 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 34.946541044, 2024-09-13T11:24:15.809 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.28729672099999998, 2024-09-13T11:24:15.809 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:15.809 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:15.809 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-13T11:24:15.809 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:15.809 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:15.810 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:15.810 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:40.805973+0000", 2024-09-13T11:24:15.810 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:15.810 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:15.810 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:15.810 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:15.811 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:40.805973+0000", 2024-09-13T11:24:15.811 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:15.811 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:15.811 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:15.811 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:15.812 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:40.805974+0000", 2024-09-13T11:24:15.812 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.8599999999999998e-07 2024-09-13T11:24:15.812 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:15.812 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:15.812 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:15.812 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:40.805976+0000", 2024-09-13T11:24:15.813 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.2809999999999998e-06 2024-09-13T11:24:15.813 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:15.813 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:15.813 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:15.813 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.091588+0000", 2024-09-13T11:24:15.814 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.28561173899999998 2024-09-13T11:24:15.814 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:15.814 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:15.814 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:15.814 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.091596+0000", 2024-09-13T11:24:15.815 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.0520000000000002e-06 2024-09-13T11:24:15.815 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:15.815 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:15.815 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:15.815 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.091907+0000", 2024-09-13T11:24:15.815 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00031179599999999998 2024-09-13T11:24:15.816 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:15.816 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:15.816 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:24:15.816 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.093270+0000", 2024-09-13T11:24:15.816 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0013622669999999999 2024-09-13T11:24:15.817 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:15.817 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:15.817 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:15.817 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:15.817 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:15.818 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPushReply(3.1a 84/78 [PushReplyOp(3:581ea97e:::benchmark_data_smithi177_31116_object10255:head)])", 2024-09-13T11:24:15.818 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:41.514228+0000", 2024-09-13T11:24:15.818 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 34.238285556000001, 2024-09-13T11:24:15.818 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.40695564200000001, 2024-09-13T11:24:15.818 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:15.818 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:15.819 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-13T11:24:15.819 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:15.819 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:15.819 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:15.819 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.514228+0000", 2024-09-13T11:24:15.820 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:15.820 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:15.820 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:15.820 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:15.820 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.514228+0000", 2024-09-13T11:24:15.820 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:15.821 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:15.821 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:15.821 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:15.821 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.514230+0000", 2024-09-13T11:24:15.821 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.499e-06 2024-09-13T11:24:15.821 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:15.822 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:15.822 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:15.822 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.514236+0000", 2024-09-13T11:24:15.822 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.5389999999999998e-06 2024-09-13T11:24:15.822 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:15.823 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:15.823 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:15.823 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.919787+0000", 2024-09-13T11:24:15.823 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.40555152999999999 2024-09-13T11:24:15.823 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:15.824 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:15.824 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:15.824 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.919796+0000", 2024-09-13T11:24:15.824 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.4829999999999999e-06 2024-09-13T11:24:15.824 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:15.825 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:15.825 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:15.825 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.920366+0000", 2024-09-13T11:24:15.825 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00057061700000000004 2024-09-13T11:24:15.825 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:15.825 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:15.826 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:24:15.826 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.921184+0000", 2024-09-13T11:24:15.826 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00081797399999999996 2024-09-13T11:24:15.826 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:15.826 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:15.827 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:15.827 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:15.827 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:15.827 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:60312 3.14 3:2e846241:::benchmark_data_smithi177_31116_object60311:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e85)", 2024-09-13T11:24:15.827 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:42.065725+0000", 2024-09-13T11:24:15.827 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 33.686788913000001, 2024-09-13T11:24:15.828 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.211323176, 2024-09-13T11:24:15.828 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:15.828 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:15.828 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:15.828 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:24:15.829 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:24:15.829 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:15.829 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 60312 2024-09-13T11:24:15.829 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:15.829 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:15.829 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:15.830 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:15.830 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.065725+0000", 2024-09-13T11:24:15.830 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:15.830 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:15.830 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:15.830 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:15.831 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.065725+0000", 2024-09-13T11:24:15.831 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:15.831 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:15.832 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:15.832 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:15.832 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.065728+0000", 2024-09-13T11:24:15.832 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.3400000000000002e-06 2024-09-13T11:24:15.832 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:15.833 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:15.833 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:15.833 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.065759+0000", 2024-09-13T11:24:15.833 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.0074e-05 2024-09-13T11:24:15.833 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:15.834 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:15.834 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:15.834 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.065771+0000", 2024-09-13T11:24:15.834 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.2869e-05 2024-09-13T11:24:15.834 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:15.834 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:15.835 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:15.835 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.065780+0000", 2024-09-13T11:24:15.835 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.3489999999999992e-06 2024-09-13T11:24:15.835 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:15.835 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:15.835 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:15.835 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.065837+0000", 2024-09-13T11:24:15.835 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.7444000000000001e-05 2024-09-13T11:24:15.835 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:15.835 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:15.835 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:24:15.836 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.065966+0000", 2024-09-13T11:24:15.836 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000128973 2024-09-13T11:24:15.836 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:15.836 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:15.836 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 14", 2024-09-13T11:24:15.836 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.066114+0000", 2024-09-13T11:24:15.836 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00014788299999999999 2024-09-13T11:24:15.836 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:15.836 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:15.836 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:24:15.836 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.067604+0000", 2024-09-13T11:24:15.836 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001489637 2024-09-13T11:24:15.836 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:15.836 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:15.837 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:15.837 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.276963+0000", 2024-09-13T11:24:15.837 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.20935888499999999 2024-09-13T11:24:15.837 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:15.837 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:15.837 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:24:15.837 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.277003+0000", 2024-09-13T11:24:15.837 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.0241000000000003e-05 2024-09-13T11:24:15.837 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:15.837 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:15.837 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:24:15.837 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.277048+0000", 2024-09-13T11:24:15.837 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.5481e-05 2024-09-13T11:24:15.838 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:15.838 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:15.838 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:15.838 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:15.838 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:15.838 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop_reply(client.4665.0:60312 3.14 e85/84 ondisk, result = 0)", 2024-09-13T11:24:15.838 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:42.069120+0000", 2024-09-13T11:24:15.838 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 33.683393793, 2024-09-13T11:24:15.838 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.20804031000000001, 2024-09-13T11:24:15.838 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:15.838 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:15.838 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:24:15.838 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:15.839 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:15.839 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:15.839 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.069120+0000", 2024-09-13T11:24:15.839 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:15.839 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:15.839 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:15.839 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:15.839 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.069120+0000", 2024-09-13T11:24:15.839 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:15.839 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:15.839 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:15.839 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:15.839 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.069123+0000", 2024-09-13T11:24:15.840 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.289e-06 2024-09-13T11:24:15.840 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:15.840 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:15.840 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:15.840 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.069128+0000", 2024-09-13T11:24:15.840 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.3530000000000002e-06 2024-09-13T11:24:15.840 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:15.840 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:15.840 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:15.840 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.276885+0000", 2024-09-13T11:24:15.840 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.207756989 2024-09-13T11:24:15.840 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:15.840 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:15.841 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:15.841 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.276892+0000", 2024-09-13T11:24:15.841 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.922e-06 2024-09-13T11:24:15.841 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:15.841 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:15.841 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:15.841 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.276953+0000", 2024-09-13T11:24:15.841 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.0890000000000001e-05 2024-09-13T11:24:15.841 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:15.841 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:15.841 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:24:15.841 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.276959+0000", 2024-09-13T11:24:15.841 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.9179999999999999e-06 2024-09-13T11:24:15.841 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:15.842 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:15.842 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:24:15.842 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.277161+0000", 2024-09-13T11:24:15.842 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00020194899999999999 2024-09-13T11:24:15.842 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:15.842 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:15.842 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:15.842 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:15.842 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:15.842 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:60427 3.14 3:2afea229:::benchmark_data_smithi177_31116_object60426:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e85)", 2024-09-13T11:24:15.842 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:42.187231+0000", 2024-09-13T11:24:15.842 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 33.565282687, 2024-09-13T11:24:15.842 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.83900596999999999, 2024-09-13T11:24:15.842 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:15.843 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:15.843 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:15.843 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:24:15.843 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:24:15.843 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:15.843 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 60427 2024-09-13T11:24:15.843 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:15.843 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:15.843 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:15.843 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:15.844 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.187231+0000", 2024-09-13T11:24:15.844 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:15.844 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:15.844 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:15.844 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:15.844 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.187231+0000", 2024-09-13T11:24:15.844 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:15.844 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:15.844 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:15.844 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:15.844 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.187235+0000", 2024-09-13T11:24:15.844 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.3579999999999999e-06 2024-09-13T11:24:15.844 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:15.845 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:15.845 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:15.845 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.187261+0000", 2024-09-13T11:24:15.845 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.6089e-05 2024-09-13T11:24:15.845 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:15.845 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:15.845 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:15.845 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.187275+0000", 2024-09-13T11:24:15.845 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.4653e-05 2024-09-13T11:24:15.845 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:15.845 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:15.845 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:15.845 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.187282+0000", 2024-09-13T11:24:15.846 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.8870000000000003e-06 2024-09-13T11:24:15.846 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:15.846 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:15.846 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:15.846 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.187336+0000", 2024-09-13T11:24:15.846 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.3875999999999997e-05 2024-09-13T11:24:15.846 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:15.846 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:15.846 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:24:15.846 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.187480+0000", 2024-09-13T11:24:15.846 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00014330899999999999 2024-09-13T11:24:15.846 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:15.846 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:15.847 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 14", 2024-09-13T11:24:15.847 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.187658+0000", 2024-09-13T11:24:15.847 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00017861900000000001 2024-09-13T11:24:15.847 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:15.847 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:15.847 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:24:15.847 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.189353+0000", 2024-09-13T11:24:15.847 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001695144 2024-09-13T11:24:15.847 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:15.847 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:15.847 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:15.847 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:43.026081+0000", 2024-09-13T11:24:15.847 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.83672736199999997 2024-09-13T11:24:15.847 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:15.848 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:15.848 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:24:15.848 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:43.026169+0000", 2024-09-13T11:24:15.848 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.8411000000000004e-05 2024-09-13T11:24:15.848 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:15.848 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:15.848 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:24:15.848 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:43.026237+0000", 2024-09-13T11:24:15.848 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.8262000000000005e-05 2024-09-13T11:24:15.848 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:15.848 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:15.848 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:15.848 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:15.849 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:15.849 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop_reply(client.4665.0:60427 3.14 e85/84 ondisk, result = 0)", 2024-09-13T11:24:15.849 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:42.190914+0000", 2024-09-13T11:24:15.849 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 33.561600022999997, 2024-09-13T11:24:15.849 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.83534071499999996, 2024-09-13T11:24:15.849 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:15.849 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:15.849 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:24:15.849 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:15.849 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:15.849 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:15.849 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.190914+0000", 2024-09-13T11:24:15.849 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:15.850 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:15.850 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:15.850 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:15.850 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.190914+0000", 2024-09-13T11:24:15.850 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:15.850 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:15.850 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:15.850 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:15.850 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.190916+0000", 2024-09-13T11:24:15.850 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.84e-06 2024-09-13T11:24:15.850 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:15.850 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:15.850 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:15.850 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.190921+0000", 2024-09-13T11:24:15.851 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.2039999999999996e-06 2024-09-13T11:24:15.851 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:15.851 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:15.851 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:15.851 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:43.025926+0000", 2024-09-13T11:24:15.851 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.83500542 2024-09-13T11:24:15.851 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:15.851 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:15.851 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:15.851 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:43.025937+0000", 2024-09-13T11:24:15.851 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0597e-05 2024-09-13T11:24:15.851 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:15.851 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:15.852 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:15.852 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:43.026052+0000", 2024-09-13T11:24:15.852 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000114838 2024-09-13T11:24:15.852 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:15.852 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:15.852 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:24:15.852 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:43.026070+0000", 2024-09-13T11:24:15.852 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.8131e-05 2024-09-13T11:24:15.852 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:15.852 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:15.852 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:24:15.852 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:43.026255+0000", 2024-09-13T11:24:15.852 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000184685 2024-09-13T11:24:15.852 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:15.853 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:15.853 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:15.853 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:15.853 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:15.853 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPushReply(3.1d 89/78 [PushReplyOp(3:bc1b7cfb:::benchmark_data_smithi177_31116_object22629:head)])", 2024-09-13T11:24:15.853 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:46.663106+0000", 2024-09-13T11:24:15.853 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 29.08940784, 2024-09-13T11:24:15.853 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.26645533199999999, 2024-09-13T11:24:15.853 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:15.853 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:15.853 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-13T11:24:15.853 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:15.853 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:15.854 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:15.854 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.663106+0000", 2024-09-13T11:24:15.854 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:15.854 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:15.854 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:15.854 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:15.854 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.663106+0000", 2024-09-13T11:24:15.854 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:15.854 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:15.854 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:15.854 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:15.854 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.663108+0000", 2024-09-13T11:24:15.854 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.0140000000000001e-06 2024-09-13T11:24:15.855 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:15.855 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:15.855 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:15.855 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.663116+0000", 2024-09-13T11:24:15.855 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.5440000000000001e-06 2024-09-13T11:24:15.855 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:15.855 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:15.855 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:15.855 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.928111+0000", 2024-09-13T11:24:15.855 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.26499566899999999 2024-09-13T11:24:15.855 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:15.855 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:15.855 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:15.856 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.928135+0000", 2024-09-13T11:24:15.856 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.4003e-05 2024-09-13T11:24:15.856 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:15.856 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:15.856 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:15.856 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.928559+0000", 2024-09-13T11:24:15.856 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000423318 2024-09-13T11:24:15.856 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:15.856 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:15.856 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:24:15.856 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.929562+0000", 2024-09-13T11:24:15.856 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0010027840000000001 2024-09-13T11:24:15.856 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:15.856 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:15.857 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:15.857 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:15.857 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:15.857 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:64349 3.17 e90/85 3:ef87a22f:::benchmark_data_smithi177_31116_object64348:head v 90'3298, mlcod=90'3295)", 2024-09-13T11:24:15.857 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:47.159683+0000", 2024-09-13T11:24:15.857 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 28.592831323999999, 2024-09-13T11:24:15.857 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.153974149, 2024-09-13T11:24:15.857 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:15.857 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:15.857 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:15.857 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:15.857 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:15.857 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:15.858 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:47.159683+0000", 2024-09-13T11:24:15.858 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:15.858 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:15.858 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:15.858 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:15.858 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:47.159683+0000", 2024-09-13T11:24:15.858 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:15.858 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:15.858 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:15.858 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:15.858 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:47.159684+0000", 2024-09-13T11:24:15.858 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.6649999999999999e-06 2024-09-13T11:24:15.859 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:15.859 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:15.859 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:15.859 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:47.159742+0000", 2024-09-13T11:24:15.859 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.8004999999999998e-05 2024-09-13T11:24:15.859 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:15.859 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:15.859 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:15.859 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:47.312341+0000", 2024-09-13T11:24:15.859 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.15259900300000001 2024-09-13T11:24:15.859 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:15.859 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:15.859 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:15.859 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:47.312348+0000", 2024-09-13T11:24:15.860 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.7449999999999998e-06 2024-09-13T11:24:15.860 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:15.860 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:15.860 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:15.860 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:47.312395+0000", 2024-09-13T11:24:15.860 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.7123999999999998e-05 2024-09-13T11:24:15.860 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:15.860 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:15.860 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:24:15.860 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:47.312410+0000", 2024-09-13T11:24:15.860 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.504e-05 2024-09-13T11:24:15.860 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:15.860 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:15.861 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:24:15.861 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:47.313619+0000", 2024-09-13T11:24:15.861 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001208558 2024-09-13T11:24:15.861 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:15.861 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:15.861 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:24:15.861 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:47.313657+0000", 2024-09-13T11:24:15.861 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.8009000000000001e-05 2024-09-13T11:24:15.861 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:15.861 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:15.861 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:15.861 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:15.861 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:15.862 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "rep_scrubmap(3.14 e84 from shard 14)", 2024-09-13T11:24:15.862 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:48.313615+0000", 2024-09-13T11:24:15.862 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 27.438899047, 2024-09-13T11:24:15.862 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.35633836099999999, 2024-09-13T11:24:15.862 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:15.862 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:15.862 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:24:15.862 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:15.862 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:15.862 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:15.862 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.313615+0000", 2024-09-13T11:24:15.862 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:15.862 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:15.862 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:15.863 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:15.863 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.313615+0000", 2024-09-13T11:24:15.863 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:15.863 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:15.863 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:15.863 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:15.863 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.313617+0000", 2024-09-13T11:24:15.863 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.7990000000000001e-06 2024-09-13T11:24:15.863 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:15.863 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:15.863 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:15.863 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.313688+0000", 2024-09-13T11:24:15.863 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.1481000000000002e-05 2024-09-13T11:24:15.863 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:15.864 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:15.864 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:15.864 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.669785+0000", 2024-09-13T11:24:15.864 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.356096466 2024-09-13T11:24:15.864 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:15.864 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:15.864 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:15.864 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.669792+0000", 2024-09-13T11:24:15.864 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.1740000000000003e-06 2024-09-13T11:24:15.864 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:15.864 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:15.864 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:15.864 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.669839+0000", 2024-09-13T11:24:15.865 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.7323000000000001e-05 2024-09-13T11:24:15.865 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:15.865 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:15.865 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:24:15.865 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.669847+0000", 2024-09-13T11:24:15.865 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.0469999999999994e-06 2024-09-13T11:24:15.865 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:15.865 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:15.865 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:24:15.865 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.669953+0000", 2024-09-13T11:24:15.865 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00010607099999999999 2024-09-13T11:24:15.865 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:15.865 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:15.866 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:15.866 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:15.866 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:15.866 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:70186 3.14 3:28350fee:::benchmark_data_smithi177_31116_object70185:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95)", 2024-09-13T11:24:15.866 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:52.670635+0000", 2024-09-13T11:24:15.866 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 23.081878884000002, 2024-09-13T11:24:15.866 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.145601377, 2024-09-13T11:24:15.866 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:15.866 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:15.866 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:15.866 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:24:15.866 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:24:15.867 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:15.867 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 70186 2024-09-13T11:24:15.867 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:15.867 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:15.867 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:15.867 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:15.867 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:52.670635+0000", 2024-09-13T11:24:15.867 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:15.867 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:15.867 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:15.867 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:15.867 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:52.670635+0000", 2024-09-13T11:24:15.867 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:15.867 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:15.868 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:15.868 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:15.868 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:52.670637+0000", 2024-09-13T11:24:15.868 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.6610000000000001e-06 2024-09-13T11:24:15.868 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:15.868 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:15.868 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:15.868 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:52.670661+0000", 2024-09-13T11:24:15.868 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.3887e-05 2024-09-13T11:24:15.868 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:15.868 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:15.868 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:15.868 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:52.670672+0000", 2024-09-13T11:24:15.868 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0825000000000001e-05 2024-09-13T11:24:15.869 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:15.869 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:15.869 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:15.869 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:52.670679+0000", 2024-09-13T11:24:15.869 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.0990000000000001e-06 2024-09-13T11:24:15.869 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:15.869 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:15.869 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:15.869 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:52.670735+0000", 2024-09-13T11:24:15.869 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.6609e-05 2024-09-13T11:24:15.869 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:15.869 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:15.869 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:24:15.869 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:52.670829+0000", 2024-09-13T11:24:15.870 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.3723999999999999e-05 2024-09-13T11:24:15.870 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:15.870 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:15.870 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 14", 2024-09-13T11:24:15.870 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:52.670938+0000", 2024-09-13T11:24:15.870 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000109058 2024-09-13T11:24:15.870 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:15.870 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:15.870 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:24:15.870 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:52.672723+0000", 2024-09-13T11:24:15.870 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001785355 2024-09-13T11:24:15.870 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:15.870 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:15.870 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:15.871 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:52.816180+0000", 2024-09-13T11:24:15.871 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.14345624900000001 2024-09-13T11:24:15.871 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:15.871 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:15.871 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:24:15.871 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:52.816212+0000", 2024-09-13T11:24:15.871 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.2486e-05 2024-09-13T11:24:15.871 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:15.871 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:15.871 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:24:15.871 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:52.816237+0000", 2024-09-13T11:24:15.871 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.4423999999999999e-05 2024-09-13T11:24:15.871 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:15.871 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:15.872 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:15.872 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:15.872 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:15.872 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:79776 3.1a 3:5d637d57:::benchmark_data_smithi177_31116_object79775:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102)", 2024-09-13T11:24:15.872 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:00.354543+0000", 2024-09-13T11:24:15.872 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 15.397970576000001, 2024-09-13T11:24:15.872 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.14881462100000001, 2024-09-13T11:24:15.872 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:15.872 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:15.872 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:15.872 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:24:15.872 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:24:15.872 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:15.872 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 79776 2024-09-13T11:24:15.873 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:15.873 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:15.873 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:15.873 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:15.873 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.354543+0000", 2024-09-13T11:24:15.873 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:15.873 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:15.873 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:15.873 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:15.873 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.354543+0000", 2024-09-13T11:24:15.873 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:15.873 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:15.873 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:15.873 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:15.874 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.354545+0000", 2024-09-13T11:24:15.874 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.508e-06 2024-09-13T11:24:15.874 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:15.874 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:15.874 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:15.874 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.354569+0000", 2024-09-13T11:24:15.874 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.4338e-05 2024-09-13T11:24:15.874 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:15.874 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:15.874 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:15.874 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.354581+0000", 2024-09-13T11:24:15.874 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.2125e-05 2024-09-13T11:24:15.874 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:15.874 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:15.875 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:15.875 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.354589+0000", 2024-09-13T11:24:15.875 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.3150000000000003e-06 2024-09-13T11:24:15.875 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:15.875 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:15.875 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:15.875 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.354638+0000", 2024-09-13T11:24:15.875 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.9292000000000002e-05 2024-09-13T11:24:15.875 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:15.875 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:15.875 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:24:15.875 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.354760+0000", 2024-09-13T11:24:15.875 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00012205399999999999 2024-09-13T11:24:15.876 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:15.876 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:15.876 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 4,9", 2024-09-13T11:24:15.876 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.354862+0000", 2024-09-13T11:24:15.876 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000101854 2024-09-13T11:24:15.876 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:15.876 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:15.876 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:24:15.876 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.356324+0000", 2024-09-13T11:24:15.876 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001461929 2024-09-13T11:24:15.876 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:15.876 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:15.876 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:15.876 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.357114+0000", 2024-09-13T11:24:15.877 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00079038800000000005 2024-09-13T11:24:15.877 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:15.877 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:15.877 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:15.877 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.502891+0000", 2024-09-13T11:24:15.877 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.14577642199999999 2024-09-13T11:24:15.877 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:15.877 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:15.877 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:24:15.877 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.503016+0000", 2024-09-13T11:24:15.877 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000124857 2024-09-13T11:24:15.877 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:15.877 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:15.877 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:24:15.878 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.503358+0000", 2024-09-13T11:24:15.878 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00034253899999999999 2024-09-13T11:24:15.878 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:15.878 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:15.878 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:15.878 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:15.878 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:15.878 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:91603 3.14 3:2d13add5:::benchmark_data_smithi177_31116_object91602:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111)", 2024-09-13T11:24:15.878 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:11.037157+0000", 2024-09-13T11:24:15.878 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 4.7153574349999996, 2024-09-13T11:24:15.878 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.165525535, 2024-09-13T11:24:15.878 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:15.878 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:15.879 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:15.879 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:24:15.879 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:24:15.879 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:15.879 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 91603 2024-09-13T11:24:15.879 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:15.879 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:15.879 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:15.879 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:15.879 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:11.037157+0000", 2024-09-13T11:24:15.879 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:15.879 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:15.879 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:15.879 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:15.880 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:11.037157+0000", 2024-09-13T11:24:15.880 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:15.880 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:15.880 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:15.880 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:15.880 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:11.037159+0000", 2024-09-13T11:24:15.880 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.261e-06 2024-09-13T11:24:15.880 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:15.880 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:15.880 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:15.880 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:11.037188+0000", 2024-09-13T11:24:15.880 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.8798000000000001e-05 2024-09-13T11:24:15.880 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:15.881 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:15.881 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:15.881 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:11.037204+0000", 2024-09-13T11:24:15.881 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.6739999999999999e-05 2024-09-13T11:24:15.881 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:15.881 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:15.881 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:15.881 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:11.037213+0000", 2024-09-13T11:24:15.881 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.1110000000000008e-06 2024-09-13T11:24:15.881 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:15.881 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:15.881 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:15.881 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:11.037270+0000", 2024-09-13T11:24:15.882 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.7629e-05 2024-09-13T11:24:15.882 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:15.882 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:15.882 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:24:15.882 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:11.037364+0000", 2024-09-13T11:24:15.882 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.3871999999999998e-05 2024-09-13T11:24:15.882 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:15.882 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:15.882 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 14", 2024-09-13T11:24:15.882 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:11.037450+0000", 2024-09-13T11:24:15.882 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.5568000000000005e-05 2024-09-13T11:24:15.882 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:15.882 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:15.882 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:24:15.883 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:11.039799+0000", 2024-09-13T11:24:15.883 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0023494039999999998 2024-09-13T11:24:15.883 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:15.883 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:15.883 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:15.883 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:11.202576+0000", 2024-09-13T11:24:15.883 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.16277712499999999 2024-09-13T11:24:15.883 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:15.883 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:15.883 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:24:15.883 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:11.202647+0000", 2024-09-13T11:24:15.883 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.0934999999999998e-05 2024-09-13T11:24:15.883 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:15.883 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:15.884 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:24:15.884 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:11.202682+0000", 2024-09-13T11:24:15.884 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.5092000000000002e-05 2024-09-13T11:24:15.884 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:15.884 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:15.884 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:15.884 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:15.884 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:15.884 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-09-13T11:24:15.886 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-13T11:24:15.886 INFO:tasks.ceph.osd.13.smithi092.stderr:2024-09-13T11:24:15.883+0000 7f3e5afea640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26350) UID: 0 2024-09-13T11:24:15.986 INFO:tasks.ceph.osd.9.smithi092.stderr:2024-09-13T11:24:15.983+0000 7ff5ab3d0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26348) UID: 0 2024-09-13T11:24:16.008 INFO:tasks.radosbench.radosbench.0.smithi177.stdout: 109 15 99157 99142 56.8418 68.75 0.0115305 0.0175869 2024-09-13T11:24:16.022 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-09-13T11:24:16.023 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [], 2024-09-13T11:24:16.023 INFO:teuthology.orchestra.run.smithi177.stdout: "num_ops": 0 2024-09-13T11:24:16.023 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-09-13T11:24:16.023 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-13T11:24:16.037 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-13T11:24:16.087 INFO:tasks.ceph.osd.3.smithi177.stderr:2024-09-13T11:24:16.083+0000 7f10adba4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26574) UID: 0 2024-09-13T11:24:16.181 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-09-13T11:24:16.182 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [], 2024-09-13T11:24:16.182 INFO:teuthology.orchestra.run.smithi177.stdout: "complaint_time": 30, 2024-09-13T11:24:16.182 INFO:teuthology.orchestra.run.smithi177.stdout: "num_blocked_ops": 0 2024-09-13T11:24:16.183 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-09-13T11:24:16.188 INFO:tasks.ceph.osd.15.smithi177.stderr:2024-09-13T11:24:16.187+0000 7f9d6ce77640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26578) UID: 0 2024-09-13T11:24:16.200 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-13T11:24:16.288 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-13T11:24:16.283+0000 7f6a68775640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26307) UID: 0 2024-09-13T11:24:16.390 INFO:tasks.ceph.osd.5.smithi092.stderr:2024-09-13T11:24:16.387+0000 7f6f0b705640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26346) UID: 0 2024-09-13T11:24:16.393 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-09-13T11:24:16.393 INFO:teuthology.orchestra.run.smithi177.stdout: "size": 20, 2024-09-13T11:24:16.394 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 600, 2024-09-13T11:24:16.394 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [ 2024-09-13T11:24:16.394 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:16.394 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPush(3.12 74/69 [PushOp(3:4f110bb2:::benchmark_data_smithi177_31116_object30564:head, version: 59'1932, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:4f110bb2:::benchmark_data_smithi177_31116_object30564:head@59'1932, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:24:16.394 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:30.835784+0000", 2024-09-13T11:24:16.394 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 45.555809500999999, 2024-09-13T11:24:16.394 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.12151646200000001, 2024-09-13T11:24:16.394 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:16.394 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:16.394 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:24:16.394 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:16.394 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:16.394 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:16.395 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.835784+0000", 2024-09-13T11:24:16.395 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:16.395 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:16.395 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:16.395 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:16.395 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.835784+0000", 2024-09-13T11:24:16.395 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:16.395 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:16.395 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:16.395 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:16.395 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.835785+0000", 2024-09-13T11:24:16.395 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.734e-06 2024-09-13T11:24:16.395 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:16.395 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:16.396 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:16.396 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.835824+0000", 2024-09-13T11:24:16.396 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.8804999999999999e-05 2024-09-13T11:24:16.396 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:16.396 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:16.396 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:16.396 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.955218+0000", 2024-09-13T11:24:16.396 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.11939419800000001 2024-09-13T11:24:16.396 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:16.396 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:16.396 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:16.396 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.955253+0000", 2024-09-13T11:24:16.397 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.4254000000000001e-05 2024-09-13T11:24:16.397 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:16.397 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:16.397 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:16.397 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.955436+0000", 2024-09-13T11:24:16.397 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000183411 2024-09-13T11:24:16.397 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:16.397 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:16.397 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:24:16.397 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.955458+0000", 2024-09-13T11:24:16.397 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.2062999999999999e-05 2024-09-13T11:24:16.397 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:16.397 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:16.398 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:24:16.398 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.957300+0000", 2024-09-13T11:24:16.398 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0018419969999999999 2024-09-13T11:24:16.398 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:16.398 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:16.398 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:16.398 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:16.398 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:16.398 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:54663 3.12 e75/69 3:49ad4b1b:::benchmark_data_smithi177_31116_object54662:head v 75'3095, mlcod=59'1933)", 2024-09-13T11:24:16.398 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:31.033716+0000", 2024-09-13T11:24:16.398 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 45.357877438999999, 2024-09-13T11:24:16.398 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.121029233, 2024-09-13T11:24:16.398 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:16.399 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:16.399 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:16.399 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:16.399 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:16.399 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:16.399 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.033716+0000", 2024-09-13T11:24:16.399 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:16.399 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:16.399 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:16.399 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:16.399 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.033716+0000", 2024-09-13T11:24:16.399 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:16.399 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:16.399 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:16.400 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:16.400 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.033724+0000", 2024-09-13T11:24:16.400 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.2910000000000007e-06 2024-09-13T11:24:16.400 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:16.400 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:16.400 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:16.400 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.033783+0000", 2024-09-13T11:24:16.400 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.8826999999999998e-05 2024-09-13T11:24:16.400 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:16.400 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:16.400 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:16.400 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.138806+0000", 2024-09-13T11:24:16.400 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.105023701 2024-09-13T11:24:16.401 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:16.401 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:16.401 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:16.401 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.138814+0000", 2024-09-13T11:24:16.401 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.5340000000000002e-06 2024-09-13T11:24:16.401 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:16.401 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:16.401 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:16.401 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.138871+0000", 2024-09-13T11:24:16.401 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.6988999999999999e-05 2024-09-13T11:24:16.401 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:16.401 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:16.401 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:24:16.402 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.138895+0000", 2024-09-13T11:24:16.402 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.3887e-05 2024-09-13T11:24:16.402 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:16.402 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:16.402 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:24:16.402 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.154684+0000", 2024-09-13T11:24:16.402 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.015788710000000001 2024-09-13T11:24:16.402 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:16.402 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:16.402 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:24:16.402 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.154745+0000", 2024-09-13T11:24:16.402 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.1294e-05 2024-09-13T11:24:16.402 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:16.402 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:16.403 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:16.403 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:16.403 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:16.403 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPushReply(3.13 75/70 [PushReplyOp(3:cfab03d6:::benchmark_data_smithi177_31116_object16599:head)])", 2024-09-13T11:24:16.403 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:31.163480+0000", 2024-09-13T11:24:16.403 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 45.228113077000003, 2024-09-13T11:24:16.403 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.63009029299999997, 2024-09-13T11:24:16.403 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:16.403 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:16.403 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-13T11:24:16.403 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:16.403 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:16.403 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:16.404 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.163480+0000", 2024-09-13T11:24:16.404 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:16.404 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:16.404 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:16.404 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:16.404 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.163480+0000", 2024-09-13T11:24:16.404 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:16.404 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:16.404 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:16.404 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:16.404 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.163481+0000", 2024-09-13T11:24:16.404 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0410000000000001e-06 2024-09-13T11:24:16.404 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:16.405 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:16.405 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:16.405 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.163485+0000", 2024-09-13T11:24:16.405 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.1629999999999998e-06 2024-09-13T11:24:16.405 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:16.405 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:16.405 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:16.405 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.792166+0000", 2024-09-13T11:24:16.405 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.62868067400000005 2024-09-13T11:24:16.405 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:16.405 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:16.405 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:16.405 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.792184+0000", 2024-09-13T11:24:16.405 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.7898999999999999e-05 2024-09-13T11:24:16.406 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:16.406 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:16.406 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:16.406 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.792312+0000", 2024-09-13T11:24:16.406 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00012871000000000001 2024-09-13T11:24:16.406 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:16.406 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:16.406 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:24:16.406 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.793570+0000", 2024-09-13T11:24:16.406 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001257806 2024-09-13T11:24:16.406 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:16.406 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:16.406 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:16.406 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:16.407 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:16.407 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPush(3.12 75/69 [PushOp(3:4a36ca6d:::benchmark_data_smithi177_31116_object31707:head, version: 59'1993, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:4a36ca6d:::benchmark_data_smithi177_31116_object31707:head@59'1993, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:24:16.407 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:31.309623+0000", 2024-09-13T11:24:16.407 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 45.081970122000001, 2024-09-13T11:24:16.407 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.165459152, 2024-09-13T11:24:16.407 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:16.407 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:16.407 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:24:16.407 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:16.407 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:16.407 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:16.407 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.309623+0000", 2024-09-13T11:24:16.407 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:16.408 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:16.408 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:16.408 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:16.408 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.309623+0000", 2024-09-13T11:24:16.408 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:16.408 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:16.408 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:16.408 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:16.408 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.309625+0000", 2024-09-13T11:24:16.408 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.3860000000000001e-06 2024-09-13T11:24:16.408 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:16.408 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:16.408 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:16.409 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.309680+0000", 2024-09-13T11:24:16.409 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.4388000000000002e-05 2024-09-13T11:24:16.409 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:16.409 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:16.409 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:16.409 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.473157+0000", 2024-09-13T11:24:16.409 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.163476813 2024-09-13T11:24:16.409 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:16.409 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:16.409 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:16.410 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.473189+0000", 2024-09-13T11:24:16.410 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.2975000000000002e-05 2024-09-13T11:24:16.410 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:16.410 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:16.410 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:16.410 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.473395+0000", 2024-09-13T11:24:16.410 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000206009 2024-09-13T11:24:16.410 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:16.410 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:16.410 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:24:16.411 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.473424+0000", 2024-09-13T11:24:16.411 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.8255e-05 2024-09-13T11:24:16.411 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:16.411 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:16.411 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:24:16.411 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.475082+0000", 2024-09-13T11:24:16.411 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0016583259999999999 2024-09-13T11:24:16.411 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:16.411 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:16.411 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:16.411 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:16.411 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:16.411 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPushReply(3.13 76/70 [PushReplyOp(3:c9d43a82:::benchmark_data_smithi177_31116_object20026:head)])", 2024-09-13T11:24:16.412 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:32.427709+0000", 2024-09-13T11:24:16.412 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 43.963883561999999, 2024-09-13T11:24:16.412 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.46547316300000002, 2024-09-13T11:24:16.412 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:16.412 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:16.412 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-13T11:24:16.412 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:16.412 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:16.412 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:16.412 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:32.427709+0000", 2024-09-13T11:24:16.412 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:16.412 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:16.412 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:16.413 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:16.413 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:32.427709+0000", 2024-09-13T11:24:16.413 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:16.413 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:16.413 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:16.413 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:16.413 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:32.427711+0000", 2024-09-13T11:24:16.413 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5099999999999999e-06 2024-09-13T11:24:16.413 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:16.413 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:16.413 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:16.413 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:32.427717+0000", 2024-09-13T11:24:16.413 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.1569999999999997e-06 2024-09-13T11:24:16.414 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:16.414 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:16.414 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:16.414 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:32.892390+0000", 2024-09-13T11:24:16.414 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.46467251100000001 2024-09-13T11:24:16.414 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:16.414 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:16.414 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:16.414 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:32.892405+0000", 2024-09-13T11:24:16.414 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5296999999999999e-05 2024-09-13T11:24:16.414 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:16.414 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:16.414 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:16.414 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:32.892503+0000", 2024-09-13T11:24:16.415 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.7820999999999997e-05 2024-09-13T11:24:16.415 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:16.415 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:16.415 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:24:16.415 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:32.893183+0000", 2024-09-13T11:24:16.415 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00067986700000000001 2024-09-13T11:24:16.415 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:16.415 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:16.415 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:16.415 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:16.415 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:16.415 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:56066 3.a e79/75 3:561a1dd2:::benchmark_data_smithi177_31116_object56065:head v 79'3452, mlcod=79'3449)", 2024-09-13T11:24:16.415 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:36.194429+0000", 2024-09-13T11:24:16.416 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 40.197163676999999, 2024-09-13T11:24:16.416 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.13436872499999999, 2024-09-13T11:24:16.416 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:16.416 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:16.416 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:16.416 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:16.416 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:16.416 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:16.416 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:36.194429+0000", 2024-09-13T11:24:16.416 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:16.416 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:16.416 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:16.417 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:16.417 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:36.194429+0000", 2024-09-13T11:24:16.417 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:16.417 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:16.417 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:16.417 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:16.417 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:36.194431+0000", 2024-09-13T11:24:16.417 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.4279999999999999e-06 2024-09-13T11:24:16.417 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:16.417 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:16.417 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:16.417 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:36.194467+0000", 2024-09-13T11:24:16.417 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.6402000000000001e-05 2024-09-13T11:24:16.418 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:16.418 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:16.418 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:16.418 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:36.327354+0000", 2024-09-13T11:24:16.418 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.13288720800000001 2024-09-13T11:24:16.418 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:16.418 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:16.418 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:16.418 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:36.327361+0000", 2024-09-13T11:24:16.418 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.302e-06 2024-09-13T11:24:16.418 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:16.418 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:16.418 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:16.418 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:36.327441+0000", 2024-09-13T11:24:16.419 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.0406000000000002e-05 2024-09-13T11:24:16.419 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:16.419 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:16.419 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:24:16.419 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:36.327466+0000", 2024-09-13T11:24:16.419 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.5367e-05 2024-09-13T11:24:16.419 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:16.419 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:16.419 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:24:16.419 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:36.328754+0000", 2024-09-13T11:24:16.419 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001287627 2024-09-13T11:24:16.419 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:16.419 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:16.419 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:24:16.420 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:36.328798+0000", 2024-09-13T11:24:16.420 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.3985e-05 2024-09-13T11:24:16.420 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:16.420 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:16.420 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:16.420 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:16.420 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:16.420 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:56096 3.d 3:b305b87e:::benchmark_data_smithi177_31116_object56095:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e79)", 2024-09-13T11:24:16.420 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:36.226921+0000", 2024-09-13T11:24:16.420 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 40.164672322000001, 2024-09-13T11:24:16.420 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.11682587899999999, 2024-09-13T11:24:16.420 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:16.420 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:16.421 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:16.421 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:24:16.421 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:24:16.421 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:16.421 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 56096 2024-09-13T11:24:16.421 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:16.421 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:16.421 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:16.421 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:16.421 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:36.226921+0000", 2024-09-13T11:24:16.421 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:16.421 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:16.421 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:16.422 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:16.422 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:36.226921+0000", 2024-09-13T11:24:16.422 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:16.422 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:16.422 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:16.422 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:16.422 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:36.226923+0000", 2024-09-13T11:24:16.422 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.843e-06 2024-09-13T11:24:16.422 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:16.422 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:16.422 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:16.422 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:36.226944+0000", 2024-09-13T11:24:16.422 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.1571999999999999e-05 2024-09-13T11:24:16.423 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:16.423 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:16.423 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:16.423 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:36.226954+0000", 2024-09-13T11:24:16.423 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.8970000000000003e-06 2024-09-13T11:24:16.423 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:16.423 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:16.423 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:16.423 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:36.226959+0000", 2024-09-13T11:24:16.423 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.4210000000000003e-06 2024-09-13T11:24:16.423 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:16.423 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:16.423 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:16.424 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:36.227012+0000", 2024-09-13T11:24:16.424 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.2578999999999999e-05 2024-09-13T11:24:16.424 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:16.424 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:16.424 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:24:16.424 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:36.227112+0000", 2024-09-13T11:24:16.424 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000100231 2024-09-13T11:24:16.424 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:16.424 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:16.424 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 3", 2024-09-13T11:24:16.424 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:36.227352+0000", 2024-09-13T11:24:16.424 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00023945999999999999 2024-09-13T11:24:16.424 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:16.425 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:16.425 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:24:16.425 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:36.228973+0000", 2024-09-13T11:24:16.425 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001621003 2024-09-13T11:24:16.425 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:16.425 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:16.425 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:16.425 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:36.343652+0000", 2024-09-13T11:24:16.425 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.114678868 2024-09-13T11:24:16.425 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:16.425 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:16.425 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:24:16.425 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:36.343706+0000", 2024-09-13T11:24:16.426 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.4505000000000001e-05 2024-09-13T11:24:16.426 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:16.426 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:16.426 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:24:16.426 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:36.343747+0000", 2024-09-13T11:24:16.426 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.0500000000000002e-05 2024-09-13T11:24:16.426 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:16.426 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:16.426 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:16.426 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:16.426 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:16.426 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:57435 3.3 e80/75 3:c0db4777:::benchmark_data_smithi177_31116_object57434:head v 80'3155, mlcod=80'3153)", 2024-09-13T11:24:16.426 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:37.437070+0000", 2024-09-13T11:24:16.426 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 38.954522744999998, 2024-09-13T11:24:16.427 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.35861845999999997, 2024-09-13T11:24:16.427 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:16.427 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:16.427 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:16.427 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:16.427 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:16.427 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:16.427 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:37.437070+0000", 2024-09-13T11:24:16.427 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:16.427 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:16.427 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:16.427 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:16.427 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:37.437070+0000", 2024-09-13T11:24:16.428 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:16.428 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:16.428 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:16.428 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:16.428 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:37.437072+0000", 2024-09-13T11:24:16.428 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5549999999999999e-06 2024-09-13T11:24:16.428 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:16.428 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:16.428 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:16.428 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:37.437141+0000", 2024-09-13T11:24:16.428 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.9494999999999998e-05 2024-09-13T11:24:16.428 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:16.428 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:16.429 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:16.429 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:37.793400+0000", 2024-09-13T11:24:16.429 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.35625817999999998 2024-09-13T11:24:16.429 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:16.429 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:16.429 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:16.429 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:37.793406+0000", 2024-09-13T11:24:16.429 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.6830000000000002e-06 2024-09-13T11:24:16.429 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:16.429 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:16.430 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:16.430 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:37.793472+0000", 2024-09-13T11:24:16.430 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.5671000000000002e-05 2024-09-13T11:24:16.430 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:16.430 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:16.430 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:24:16.430 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:37.793492+0000", 2024-09-13T11:24:16.430 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.0051000000000002e-05 2024-09-13T11:24:16.430 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:16.430 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:16.430 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:24:16.430 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:37.795626+0000", 2024-09-13T11:24:16.430 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0021344990000000002 2024-09-13T11:24:16.431 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:16.431 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:16.431 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:24:16.431 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:37.795689+0000", 2024-09-13T11:24:16.431 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.2325999999999999e-05 2024-09-13T11:24:16.431 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:16.431 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:16.431 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:16.431 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:16.431 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:16.431 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:57985 3.3 e80/75 3:c330bb03:::benchmark_data_smithi177_31116_object57984:head v 80'3162, mlcod=80'3159)", 2024-09-13T11:24:16.431 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:38.007076+0000", 2024-09-13T11:24:16.432 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 38.384517393000003, 2024-09-13T11:24:16.432 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.18211438199999999, 2024-09-13T11:24:16.432 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:16.432 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:16.432 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:16.432 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:16.432 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:16.432 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:16.432 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:38.007076+0000", 2024-09-13T11:24:16.432 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:16.432 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:16.432 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:16.432 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:16.433 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:38.007076+0000", 2024-09-13T11:24:16.433 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:16.433 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:16.433 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:16.433 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:16.433 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:38.007077+0000", 2024-09-13T11:24:16.433 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.6759999999999999e-06 2024-09-13T11:24:16.433 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:16.433 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:16.433 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:16.433 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:38.007221+0000", 2024-09-13T11:24:16.433 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00014344299999999999 2024-09-13T11:24:16.433 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:16.433 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:16.434 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:16.434 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:38.186808+0000", 2024-09-13T11:24:16.434 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.17958726699999999 2024-09-13T11:24:16.434 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:16.434 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:16.434 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:16.434 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:38.186815+0000", 2024-09-13T11:24:16.434 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.0890000000000002e-06 2024-09-13T11:24:16.434 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:16.434 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:16.434 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:16.434 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:38.186887+0000", 2024-09-13T11:24:16.434 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.2006000000000001e-05 2024-09-13T11:24:16.434 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:16.435 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:16.435 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:24:16.435 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:38.186913+0000", 2024-09-13T11:24:16.435 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.5962e-05 2024-09-13T11:24:16.435 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:16.435 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:16.435 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:24:16.435 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:38.189123+0000", 2024-09-13T11:24:16.435 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0022099649999999999 2024-09-13T11:24:16.435 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:16.435 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:16.435 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:24:16.435 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:38.189190+0000", 2024-09-13T11:24:16.435 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.6973999999999999e-05 2024-09-13T11:24:16.436 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:16.436 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:16.436 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:16.436 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:16.436 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:16.436 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPushReply(3.23 84/78 [PushReplyOp(3:c7e44616:::benchmark_data_smithi177_31116_object32371:head)])", 2024-09-13T11:24:16.436 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:41.338210+0000", 2024-09-13T11:24:16.436 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 35.053382673999998, 2024-09-13T11:24:16.436 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.41118492299999998, 2024-09-13T11:24:16.436 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:16.436 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:16.436 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-13T11:24:16.436 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:16.437 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:16.437 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:16.437 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.338210+0000", 2024-09-13T11:24:16.437 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:16.437 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:16.437 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:16.437 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:16.437 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.338210+0000", 2024-09-13T11:24:16.437 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:16.437 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:16.437 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:16.438 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:16.438 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.338212+0000", 2024-09-13T11:24:16.438 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.314e-06 2024-09-13T11:24:16.438 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:16.438 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:16.438 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:16.438 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.338219+0000", 2024-09-13T11:24:16.438 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.6469999999999998e-06 2024-09-13T11:24:16.438 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:16.438 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:16.438 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:16.438 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.748675+0000", 2024-09-13T11:24:16.438 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.41045567500000002 2024-09-13T11:24:16.438 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:16.439 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:16.439 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:16.439 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.748684+0000", 2024-09-13T11:24:16.439 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.2150000000000007e-06 2024-09-13T11:24:16.439 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:16.439 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:16.439 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:16.439 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.748739+0000", 2024-09-13T11:24:16.439 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.4543000000000003e-05 2024-09-13T11:24:16.439 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:16.439 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:16.439 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:24:16.439 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.749395+0000", 2024-09-13T11:24:16.439 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00065652899999999999 2024-09-13T11:24:16.440 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:16.440 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:16.440 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:16.440 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:16.440 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:16.440 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPushReply(3.23 84/78 [PushReplyOp(3:c57c3ccc:::benchmark_data_smithi177_31116_object33278:head)])", 2024-09-13T11:24:16.440 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:41.825672+0000", 2024-09-13T11:24:16.440 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 34.565921367999998, 2024-09-13T11:24:16.440 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.11608001499999999, 2024-09-13T11:24:16.440 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:16.440 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:16.440 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-13T11:24:16.440 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:16.441 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:16.441 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:16.441 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.825672+0000", 2024-09-13T11:24:16.441 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:16.441 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:16.441 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:16.441 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:16.441 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.825672+0000", 2024-09-13T11:24:16.441 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:16.441 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:16.441 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:16.441 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:16.441 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.825673+0000", 2024-09-13T11:24:16.441 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0300000000000001e-06 2024-09-13T11:24:16.442 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:16.442 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:16.442 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:16.442 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.825680+0000", 2024-09-13T11:24:16.442 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.1520000000000003e-06 2024-09-13T11:24:16.442 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:16.442 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:16.442 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:16.442 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.940925+0000", 2024-09-13T11:24:16.442 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.115244816 2024-09-13T11:24:16.442 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:16.442 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:16.442 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:16.442 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.940932+0000", 2024-09-13T11:24:16.443 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.7319999999999998e-06 2024-09-13T11:24:16.443 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:16.443 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:16.443 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:16.443 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.941218+0000", 2024-09-13T11:24:16.443 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00028532699999999999 2024-09-13T11:24:16.443 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:16.443 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:16.443 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:24:16.443 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.941752+0000", 2024-09-13T11:24:16.443 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00053395799999999996 2024-09-13T11:24:16.443 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:16.443 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:16.443 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:16.444 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:16.444 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:16.444 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPushReply(3.23 85/78 [PushReplyOp(3:c5c6dce1:::benchmark_data_smithi177_31116_object37293:head)])", 2024-09-13T11:24:16.444 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:42.377862+0000", 2024-09-13T11:24:16.444 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 34.013731215, 2024-09-13T11:24:16.444 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.62538808599999995, 2024-09-13T11:24:16.444 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:16.444 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:16.444 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-13T11:24:16.444 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:16.444 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:16.444 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:16.444 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.377862+0000", 2024-09-13T11:24:16.445 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:16.445 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:16.445 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:16.445 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:16.445 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.377862+0000", 2024-09-13T11:24:16.445 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:16.445 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:16.445 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:16.445 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:16.445 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.377863+0000", 2024-09-13T11:24:16.445 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.6899999999999996e-07 2024-09-13T11:24:16.445 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:16.445 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:16.445 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:16.446 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.377866+0000", 2024-09-13T11:24:16.446 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.219e-06 2024-09-13T11:24:16.446 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:16.446 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:16.446 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:16.446 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:43.001857+0000", 2024-09-13T11:24:16.446 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.62399084299999996 2024-09-13T11:24:16.446 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:16.446 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:16.446 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:16.446 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:43.001869+0000", 2024-09-13T11:24:16.446 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.2415999999999999e-05 2024-09-13T11:24:16.446 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:16.446 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:16.447 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:16.447 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:43.002194+0000", 2024-09-13T11:24:16.447 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000324774 2024-09-13T11:24:16.447 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:16.447 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:16.447 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:24:16.447 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:43.003250+0000", 2024-09-13T11:24:16.447 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001055865 2024-09-13T11:24:16.447 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:16.447 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:16.447 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:16.447 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:16.447 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:16.447 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPushReply(3.23 86/78 [PushReplyOp(3:c4ca4518:::benchmark_data_smithi177_31116_object39787:head)])", 2024-09-13T11:24:16.448 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:43.580816+0000", 2024-09-13T11:24:16.448 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 32.810777524000002, 2024-09-13T11:24:16.448 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.14999331199999999, 2024-09-13T11:24:16.448 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:16.448 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:16.448 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-13T11:24:16.448 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:16.448 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:16.448 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:16.448 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:43.580816+0000", 2024-09-13T11:24:16.448 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:16.448 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:16.448 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:16.448 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:16.449 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:43.580816+0000", 2024-09-13T11:24:16.449 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:16.449 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:16.449 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:16.449 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:16.449 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:43.580818+0000", 2024-09-13T11:24:16.449 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.7250000000000002e-06 2024-09-13T11:24:16.449 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:16.449 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:16.449 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:16.449 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:43.580830+0000", 2024-09-13T11:24:16.449 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.2247e-05 2024-09-13T11:24:16.449 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:16.449 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:16.450 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:16.450 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:43.730204+0000", 2024-09-13T11:24:16.450 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.149373531 2024-09-13T11:24:16.450 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:16.450 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:16.450 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:16.450 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:43.730213+0000", 2024-09-13T11:24:16.450 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.6989999999999992e-06 2024-09-13T11:24:16.450 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:16.450 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:16.450 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:16.450 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:43.730274+0000", 2024-09-13T11:24:16.450 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.1419000000000003e-05 2024-09-13T11:24:16.450 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:16.451 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:16.451 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:24:16.451 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:43.730809+0000", 2024-09-13T11:24:16.451 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00053469100000000001 2024-09-13T11:24:16.451 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:16.451 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:16.451 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:16.451 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:16.451 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:16.451 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPushReply(3.23 86/78 [PushReplyOp(3:c6ae38b7:::benchmark_data_smithi177_31116_object39934:head)])", 2024-09-13T11:24:16.451 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:43.748700+0000", 2024-09-13T11:24:16.451 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 32.642893350999998, 2024-09-13T11:24:16.451 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.11236333, 2024-09-13T11:24:16.452 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:16.452 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:16.452 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-13T11:24:16.452 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:16.452 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:16.452 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:16.452 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:43.748700+0000", 2024-09-13T11:24:16.452 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:16.452 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:16.452 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:16.452 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:16.452 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:43.748700+0000", 2024-09-13T11:24:16.452 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:16.453 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:16.453 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:16.453 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:16.453 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:43.748701+0000", 2024-09-13T11:24:16.453 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.398e-06 2024-09-13T11:24:16.453 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:16.453 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:16.453 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:16.453 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:43.748707+0000", 2024-09-13T11:24:16.453 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.993e-06 2024-09-13T11:24:16.453 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:16.453 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:16.453 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:16.454 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:43.860552+0000", 2024-09-13T11:24:16.454 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.11184472099999999 2024-09-13T11:24:16.454 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:16.454 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:16.454 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:16.454 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:43.860560+0000", 2024-09-13T11:24:16.454 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.5949999999999999e-06 2024-09-13T11:24:16.454 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:16.454 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:16.454 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:16.454 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:43.860619+0000", 2024-09-13T11:24:16.454 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.8455999999999997e-05 2024-09-13T11:24:16.454 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:16.454 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:16.455 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:24:16.455 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:43.861063+0000", 2024-09-13T11:24:16.455 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00044416699999999998 2024-09-13T11:24:16.455 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:16.455 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:16.455 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:16.455 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:16.455 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:16.455 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPush(3.1c 89/77 [PushOp(3:38bcbc71:::benchmark_data_smithi177_31116_object31448:head, version: 59'1946, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:38bcbc71:::benchmark_data_smithi177_31116_object31448:head@59'1946, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:24:16.455 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:46.670723+0000", 2024-09-13T11:24:16.455 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 29.720869991000001, 2024-09-13T11:24:16.455 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.27576065, 2024-09-13T11:24:16.455 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:16.456 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:16.456 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:24:16.456 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:16.456 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:16.456 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:16.456 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.670723+0000", 2024-09-13T11:24:16.456 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:16.456 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:16.456 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:16.456 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:16.456 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.670723+0000", 2024-09-13T11:24:16.456 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:16.456 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:16.456 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:16.457 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:16.457 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.670724+0000", 2024-09-13T11:24:16.457 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.4389999999999999e-06 2024-09-13T11:24:16.457 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:16.457 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:16.457 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:16.457 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.670754+0000", 2024-09-13T11:24:16.457 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.9224999999999999e-05 2024-09-13T11:24:16.457 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:16.457 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:16.457 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:16.457 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.944369+0000", 2024-09-13T11:24:16.457 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.27361529800000001 2024-09-13T11:24:16.458 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:16.458 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:16.458 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:16.458 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.944422+0000", 2024-09-13T11:24:16.458 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.3318e-05 2024-09-13T11:24:16.458 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:16.458 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:16.458 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:16.458 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.944711+0000", 2024-09-13T11:24:16.458 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00028912399999999998 2024-09-13T11:24:16.458 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:16.458 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:16.458 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:24:16.458 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.944735+0000", 2024-09-13T11:24:16.459 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.3366999999999998e-05 2024-09-13T11:24:16.459 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:16.459 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:16.459 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:24:16.459 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.946484+0000", 2024-09-13T11:24:16.459 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0017488790000000001 2024-09-13T11:24:16.459 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:16.459 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:16.459 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:16.459 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:16.459 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:16.459 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPush(3.1c 91/77 [PushOp(3:3e18abc2:::benchmark_data_smithi177_31116_object35874:head, version: 62'2235, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:3e18abc2:::benchmark_data_smithi177_31116_object35874:head@62'2235, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:24:16.459 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:48.350968+0000", 2024-09-13T11:24:16.459 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 28.040624678, 2024-09-13T11:24:16.460 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.19257698000000001, 2024-09-13T11:24:16.460 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:16.460 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:16.460 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:24:16.460 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:16.460 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:16.460 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:16.460 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.350968+0000", 2024-09-13T11:24:16.460 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:16.460 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:16.460 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:16.460 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:16.460 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.350968+0000", 2024-09-13T11:24:16.461 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:16.461 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:16.461 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:16.461 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:16.461 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.350970+0000", 2024-09-13T11:24:16.461 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.796e-06 2024-09-13T11:24:16.461 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:16.461 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:16.461 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:16.461 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.351003+0000", 2024-09-13T11:24:16.461 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.2631e-05 2024-09-13T11:24:16.461 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:16.461 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:16.461 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:16.462 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.542140+0000", 2024-09-13T11:24:16.462 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.191136737 2024-09-13T11:24:16.462 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:16.462 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:16.462 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:16.462 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.542158+0000", 2024-09-13T11:24:16.462 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.8896999999999999e-05 2024-09-13T11:24:16.462 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:16.462 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:16.462 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:16.462 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.542263+0000", 2024-09-13T11:24:16.462 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000104158 2024-09-13T11:24:16.462 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:16.462 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:16.463 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:24:16.463 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.542279+0000", 2024-09-13T11:24:16.463 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5926999999999998e-05 2024-09-13T11:24:16.463 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:16.463 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:16.463 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:24:16.463 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.543545+0000", 2024-09-13T11:24:16.463 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0012668340000000001 2024-09-13T11:24:16.463 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:16.463 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:16.463 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:16.463 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:16.463 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:16.464 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:77760 3.d 3:b2c17b2a:::benchmark_data_smithi177_31116_object77759:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-09-13T11:24:16.464 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:58.900728+0000", 2024-09-13T11:24:16.464 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 17.490865267, 2024-09-13T11:24:16.464 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.12741966499999999, 2024-09-13T11:24:16.464 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:16.464 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:16.464 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:16.464 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:24:16.464 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:24:16.464 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:16.464 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 77760 2024-09-13T11:24:16.464 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:16.464 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:16.465 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:16.465 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:16.465 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:58.900728+0000", 2024-09-13T11:24:16.465 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:16.465 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:16.465 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:16.465 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:16.465 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:58.900728+0000", 2024-09-13T11:24:16.465 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:16.465 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:16.465 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:16.465 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:16.465 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:58.900731+0000", 2024-09-13T11:24:16.466 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.819e-06 2024-09-13T11:24:16.466 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:16.466 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:16.466 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:16.466 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:58.900755+0000", 2024-09-13T11:24:16.466 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.4383e-05 2024-09-13T11:24:16.466 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:16.466 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:16.466 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:16.466 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:58.900766+0000", 2024-09-13T11:24:16.466 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.1105e-05 2024-09-13T11:24:16.466 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:16.466 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:16.467 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:16.467 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:58.900774+0000", 2024-09-13T11:24:16.467 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.5159999999999997e-06 2024-09-13T11:24:16.467 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:16.467 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:16.467 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:16.467 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:58.900838+0000", 2024-09-13T11:24:16.467 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.4425999999999996e-05 2024-09-13T11:24:16.467 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:16.467 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:16.467 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:24:16.467 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:58.900991+0000", 2024-09-13T11:24:16.467 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00015275400000000001 2024-09-13T11:24:16.468 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:16.468 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:16.468 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 3", 2024-09-13T11:24:16.468 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:58.901126+0000", 2024-09-13T11:24:16.468 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00013527 2024-09-13T11:24:16.468 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:16.468 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:16.468 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:24:16.468 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:58.903053+0000", 2024-09-13T11:24:16.468 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0019274509999999999 2024-09-13T11:24:16.468 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:16.468 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:16.468 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:16.469 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:59.028046+0000", 2024-09-13T11:24:16.469 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.124992257 2024-09-13T11:24:16.469 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:16.469 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:16.469 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:24:16.469 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:59.028113+0000", 2024-09-13T11:24:16.469 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.7744999999999996e-05 2024-09-13T11:24:16.469 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:16.470 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:16.470 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:24:16.470 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:59.028147+0000", 2024-09-13T11:24:16.470 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.3939000000000002e-05 2024-09-13T11:24:16.470 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:16.470 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:16.470 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:16.470 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:16.470 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:16.470 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop_reply(client.4665.0:77760 3.d e100/75 ondisk, result = 0)", 2024-09-13T11:24:16.470 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:58.906341+0000", 2024-09-13T11:24:16.470 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 17.485252188, 2024-09-13T11:24:16.470 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.121813597, 2024-09-13T11:24:16.471 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:16.471 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:16.471 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:24:16.471 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:16.471 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:16.471 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:16.471 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:58.906341+0000", 2024-09-13T11:24:16.471 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:16.471 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:16.471 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:16.471 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:16.471 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:58.906341+0000", 2024-09-13T11:24:16.472 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:16.472 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:16.472 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:16.472 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:16.472 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:58.906343+0000", 2024-09-13T11:24:16.472 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.694e-06 2024-09-13T11:24:16.472 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:16.472 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:16.472 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:16.472 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:58.906348+0000", 2024-09-13T11:24:16.472 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.1529999999999996e-06 2024-09-13T11:24:16.472 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:16.472 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:16.473 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:16.473 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:59.027972+0000", 2024-09-13T11:24:16.473 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.121624759 2024-09-13T11:24:16.473 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:16.473 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:16.473 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:16.473 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:59.027979+0000", 2024-09-13T11:24:16.473 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.0169999999999999e-06 2024-09-13T11:24:16.473 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:16.473 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:16.474 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:16.474 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:59.028032+0000", 2024-09-13T11:24:16.474 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.2305999999999997e-05 2024-09-13T11:24:16.474 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:16.474 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:16.474 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:24:16.474 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:59.028040+0000", 2024-09-13T11:24:16.474 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.9610000000000005e-06 2024-09-13T11:24:16.474 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:16.474 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:16.474 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:24:16.474 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:59.028154+0000", 2024-09-13T11:24:16.474 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000114707 2024-09-13T11:24:16.474 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:16.475 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:16.475 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:16.475 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:16.475 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:16.475 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:83211 3.1c e106/77 3:3ae31a0f:::benchmark_data_smithi177_31116_object83210:head v 106'4362, mlcod=106'4360)", 2024-09-13T11:24:16.475 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:04.345264+0000", 2024-09-13T11:24:16.475 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 12.046329026, 2024-09-13T11:24:16.475 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.17036547399999999, 2024-09-13T11:24:16.475 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:16.475 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:16.475 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:16.475 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:16.475 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:16.476 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:16.476 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.345264+0000", 2024-09-13T11:24:16.476 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:16.476 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:16.476 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:16.476 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:16.476 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.345264+0000", 2024-09-13T11:24:16.476 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:16.476 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:16.476 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:16.476 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:16.476 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.345267+0000", 2024-09-13T11:24:16.476 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.7590000000000002e-06 2024-09-13T11:24:16.476 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:16.477 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:16.477 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:16.477 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.345298+0000", 2024-09-13T11:24:16.477 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.1173000000000001e-05 2024-09-13T11:24:16.477 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:16.477 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:16.477 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:16.477 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.512265+0000", 2024-09-13T11:24:16.477 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.16696657100000001 2024-09-13T11:24:16.477 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:16.477 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:16.477 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:16.477 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.512279+0000", 2024-09-13T11:24:16.478 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.4953999999999999e-05 2024-09-13T11:24:16.478 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:16.478 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:16.478 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:16.478 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.512402+0000", 2024-09-13T11:24:16.478 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00012277999999999999 2024-09-13T11:24:16.478 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:16.478 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:16.478 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:24:16.478 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.512440+0000", 2024-09-13T11:24:16.478 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.7861999999999998e-05 2024-09-13T11:24:16.478 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:16.478 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:16.479 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:24:16.479 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.515515+0000", 2024-09-13T11:24:16.479 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00307472 2024-09-13T11:24:16.479 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:16.479 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:16.479 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:24:16.479 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.515629+0000", 2024-09-13T11:24:16.479 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00011465500000000001 2024-09-13T11:24:16.479 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:16.479 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:16.479 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:16.479 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:16.479 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:16.479 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:83223 3.1c e106/77 3:3bc6018d:::benchmark_data_smithi177_31116_object83222:head v 106'4363, mlcod=106'4360)", 2024-09-13T11:24:16.480 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:04.512139+0000", 2024-09-13T11:24:16.480 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 11.87945388, 2024-09-13T11:24:16.480 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.471923235, 2024-09-13T11:24:16.480 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:16.480 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:16.480 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:16.480 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:16.480 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:16.480 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:16.480 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.512139+0000", 2024-09-13T11:24:16.480 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:16.480 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:16.481 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:16.481 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:16.481 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.512139+0000", 2024-09-13T11:24:16.481 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:16.481 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:16.481 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:16.481 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:16.481 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.512143+0000", 2024-09-13T11:24:16.481 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.427e-06 2024-09-13T11:24:16.481 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:16.481 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:16.481 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:16.481 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.512227+0000", 2024-09-13T11:24:16.481 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.3965e-05 2024-09-13T11:24:16.482 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:16.482 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:16.482 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:16.482 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.976330+0000", 2024-09-13T11:24:16.482 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.46410342599999999 2024-09-13T11:24:16.482 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:16.482 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:16.482 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:16.482 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.976350+0000", 2024-09-13T11:24:16.482 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.9533000000000001e-05 2024-09-13T11:24:16.482 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:16.482 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:16.482 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:16.482 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.976488+0000", 2024-09-13T11:24:16.483 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00013799400000000001 2024-09-13T11:24:16.483 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:16.483 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:16.483 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:24:16.483 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.976543+0000", 2024-09-13T11:24:16.483 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.5442000000000002e-05 2024-09-13T11:24:16.483 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:16.483 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:16.483 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:24:16.483 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.983227+0000", 2024-09-13T11:24:16.483 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0066842899999999998 2024-09-13T11:24:16.483 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:16.483 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:16.483 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:24:16.484 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.984062+0000", 2024-09-13T11:24:16.484 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00083515800000000002 2024-09-13T11:24:16.484 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:16.484 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:16.484 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:16.484 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:16.484 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:16.484 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-09-13T11:24:16.485 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-09-13T11:24:16.490 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-09-13T11:24:16.488+0000 7f97befc9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26315) UID: 0 2024-09-13T11:24:16.591 INFO:tasks.ceph.osd.7.smithi177.stderr:2024-09-13T11:24:16.587+0000 7f897b51d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26575) UID: 0 2024-09-13T11:24:16.629 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-09-13T11:24:16.630 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [ 2024-09-13T11:24:16.630 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:16.630 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:99904 3.1c 3:3dec668e:::benchmark_data_smithi177_31116_object99903:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113)", 2024-09-13T11:24:16.630 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:16.613187+0000", 2024-09-13T11:24:16.630 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 0.016121478000000002, 2024-09-13T11:24:16.630 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.016147390000000001, 2024-09-13T11:24:16.630 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:16.630 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:16.630 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "waiting for sub ops", 2024-09-13T11:24:16.630 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:24:16.630 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:24:16.630 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:16.631 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 99904 2024-09-13T11:24:16.631 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:16.631 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:16.631 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:16.631 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:16.631 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:16.613187+0000", 2024-09-13T11:24:16.631 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:16.631 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:16.631 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:16.631 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:16.631 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:16.613187+0000", 2024-09-13T11:24:16.631 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:16.631 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:16.632 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:16.632 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:16.632 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:16.613189+0000", 2024-09-13T11:24:16.632 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.0959999999999999e-06 2024-09-13T11:24:16.632 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:16.632 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:16.632 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:16.632 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:16.613214+0000", 2024-09-13T11:24:16.632 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.4496999999999999e-05 2024-09-13T11:24:16.632 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:16.632 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:16.632 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:16.633 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:16.613226+0000", 2024-09-13T11:24:16.633 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.2548000000000001e-05 2024-09-13T11:24:16.633 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:16.633 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:16.633 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:16.633 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:16.613235+0000", 2024-09-13T11:24:16.633 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.7369999999999995e-06 2024-09-13T11:24:16.633 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:16.633 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:16.633 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:16.633 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:16.613304+0000", 2024-09-13T11:24:16.633 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.8724000000000001e-05 2024-09-13T11:24:16.633 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:16.634 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:16.634 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:24:16.634 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:16.613473+0000", 2024-09-13T11:24:16.634 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00016924 2024-09-13T11:24:16.634 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:16.634 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:16.634 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 7", 2024-09-13T11:24:16.634 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:16.613636+0000", 2024-09-13T11:24:16.634 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00016300700000000001 2024-09-13T11:24:16.634 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:16.634 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:16.634 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:24:16.635 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:16.619889+0000", 2024-09-13T11:24:16.635 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0062533170000000004 2024-09-13T11:24:16.635 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:16.635 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:16.635 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:16.635 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:16.635 INFO:teuthology.orchestra.run.smithi177.stdout: ], 2024-09-13T11:24:16.635 INFO:teuthology.orchestra.run.smithi177.stdout: "num_ops": 1 2024-09-13T11:24:16.635 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-09-13T11:24:16.650 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-09-13T11:24:16.692 INFO:tasks.ceph.osd.14.smithi162.stderr:2024-09-13T11:24:16.688+0000 7f235c0db640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26319) UID: 0 2024-09-13T11:24:16.794 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-09-13T11:24:16.788+0000 7f97befc9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26315) UID: 0 2024-09-13T11:24:16.838 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-09-13T11:24:16.838 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [], 2024-09-13T11:24:16.838 INFO:teuthology.orchestra.run.smithi177.stdout: "complaint_time": 30, 2024-09-13T11:24:16.838 INFO:teuthology.orchestra.run.smithi177.stdout: "num_blocked_ops": 0 2024-09-13T11:24:16.839 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-09-13T11:24:16.857 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-09-13T11:24:16.908 INFO:tasks.ceph.osd.7.smithi177.stderr:2024-09-13T11:24:16.907+0000 7f897b51d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26575) UID: 0 2024-09-13T11:24:16.916 INFO:teuthology.orchestra.run.smithi001.stderr:nodeep-scrub is set 2024-09-13T11:24:16.916 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-09-13T11:24:16.912+0000 7f97b59fa640 -1 osd.6 113 handle_osd_map injecting map crc failure 2024-09-13T11:24:16.923 INFO:tasks.ceph.osd.1.smithi092.stderr:2024-09-13T11:24:16.919+0000 7fe76d3f2640 -1 osd.1 113 handle_osd_map injecting map crc failure 2024-09-13T11:24:16.995 INFO:tasks.ceph.osd.15.smithi177.stderr:2024-09-13T11:24:16.991+0000 7f9d6ce77640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26578) UID: 0 2024-09-13T11:24:16.998 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-09-13T11:24:16.998 INFO:teuthology.orchestra.run.smithi177.stdout: "size": 20, 2024-09-13T11:24:16.999 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 600, 2024-09-13T11:24:16.999 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [ 2024-09-13T11:24:16.999 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:16.999 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:79 3.1 e32/30 3:841df5b5:::benchmark_data_smithi177_31116_object78:head v 32'6, mlcod=32'4)", 2024-09-13T11:24:16.999 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:22:27.057292+0000", 2024-09-13T11:24:16.999 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 109.93887569100001, 2024-09-13T11:24:16.999 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.89296786800000005, 2024-09-13T11:24:16.999 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:16.999 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:16.999 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:16.999 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:16.999 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:16.999 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:17.000 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.057292+0000", 2024-09-13T11:24:17.000 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:17.000 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.000 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.000 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:17.000 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.057292+0000", 2024-09-13T11:24:17.000 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:17.000 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.000 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.000 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:17.000 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.057293+0000", 2024-09-13T11:24:17.000 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.6109999999999999e-06 2024-09-13T11:24:17.001 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.001 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.001 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:17.001 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.057330+0000", 2024-09-13T11:24:17.001 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.6721999999999998e-05 2024-09-13T11:24:17.001 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.001 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.001 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:17.001 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.948197+0000", 2024-09-13T11:24:17.001 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.890867042 2024-09-13T11:24:17.002 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.002 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.002 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:17.002 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.948213+0000", 2024-09-13T11:24:17.002 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5826000000000001e-05 2024-09-13T11:24:17.002 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.002 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.002 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:17.002 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.948377+0000", 2024-09-13T11:24:17.002 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00016348600000000001 2024-09-13T11:24:17.002 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.003 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.003 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:24:17.003 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.948447+0000", 2024-09-13T11:24:17.003 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.0891999999999998e-05 2024-09-13T11:24:17.003 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.003 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.003 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:24:17.003 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.950150+0000", 2024-09-13T11:24:17.003 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001702329 2024-09-13T11:24:17.003 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.003 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.003 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:24:17.003 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.950260+0000", 2024-09-13T11:24:17.004 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00010996 2024-09-13T11:24:17.004 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:17.004 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:17.004 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:17.004 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.004 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.004 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:30815 3.f 3:fde17cb1:::benchmark_data_smithi177_31116_object30814:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e59)", 2024-09-13T11:24:17.004 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:09.495035+0000", 2024-09-13T11:24:17.004 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 67.501132526000006, 2024-09-13T11:24:17.004 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.229307711, 2024-09-13T11:24:17.004 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:17.004 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:17.004 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:17.005 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:24:17.005 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:24:17.005 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:17.005 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 30815 2024-09-13T11:24:17.005 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.005 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:17.005 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.005 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:17.005 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.495035+0000", 2024-09-13T11:24:17.005 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:17.005 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.005 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.005 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:17.005 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.495035+0000", 2024-09-13T11:24:17.006 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:17.006 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.006 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.006 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:17.006 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.495037+0000", 2024-09-13T11:24:17.006 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.871e-06 2024-09-13T11:24:17.006 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.006 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.006 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:17.006 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.495062+0000", 2024-09-13T11:24:17.006 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.5164000000000002e-05 2024-09-13T11:24:17.006 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.006 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.006 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:17.007 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.495075+0000", 2024-09-13T11:24:17.007 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3369e-05 2024-09-13T11:24:17.007 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.007 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.007 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:17.007 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.495084+0000", 2024-09-13T11:24:17.007 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.3510000000000002e-06 2024-09-13T11:24:17.007 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.007 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.007 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:17.007 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.495346+0000", 2024-09-13T11:24:17.007 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00026264000000000002 2024-09-13T11:24:17.007 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.007 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.008 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:24:17.008 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.495523+0000", 2024-09-13T11:24:17.008 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00017640000000000001 2024-09-13T11:24:17.008 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.008 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.008 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 12", 2024-09-13T11:24:17.008 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.495699+0000", 2024-09-13T11:24:17.008 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00017653600000000001 2024-09-13T11:24:17.008 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.008 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.008 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:24:17.008 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.497040+0000", 2024-09-13T11:24:17.009 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001340473 2024-09-13T11:24:17.009 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.009 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.009 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:17.009 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.724049+0000", 2024-09-13T11:24:17.009 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.22700931799999999 2024-09-13T11:24:17.009 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.009 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.009 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:24:17.010 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.724215+0000", 2024-09-13T11:24:17.010 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000165988 2024-09-13T11:24:17.010 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.010 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.010 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:24:17.010 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.724343+0000", 2024-09-13T11:24:17.010 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000127601 2024-09-13T11:24:17.010 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:17.010 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:17.010 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:17.010 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.010 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.010 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:30828 3.f 3:f3a9c6a9:::benchmark_data_smithi177_31116_object30827:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e59)", 2024-09-13T11:24:17.011 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:09.503661+0000", 2024-09-13T11:24:17.011 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 67.492506341999999, 2024-09-13T11:24:17.011 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.26477667100000002, 2024-09-13T11:24:17.011 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:17.011 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:17.011 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:17.011 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:24:17.011 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:24:17.011 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:17.011 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 30828 2024-09-13T11:24:17.011 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.011 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:17.011 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.011 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:17.011 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.503661+0000", 2024-09-13T11:24:17.012 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:17.012 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.012 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.012 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:17.012 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.503661+0000", 2024-09-13T11:24:17.012 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:17.012 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.012 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.013 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:17.013 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.503663+0000", 2024-09-13T11:24:17.013 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.968e-06 2024-09-13T11:24:17.013 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.013 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.013 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:17.013 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.503687+0000", 2024-09-13T11:24:17.013 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.3983999999999999e-05 2024-09-13T11:24:17.013 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.013 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.013 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:17.013 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.503701+0000", 2024-09-13T11:24:17.013 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3416e-05 2024-09-13T11:24:17.013 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.014 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.014 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:17.014 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.503708+0000", 2024-09-13T11:24:17.014 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.9530000000000002e-06 2024-09-13T11:24:17.014 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.014 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.014 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:17.014 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.503759+0000", 2024-09-13T11:24:17.014 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.1075999999999997e-05 2024-09-13T11:24:17.014 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.014 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.014 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:24:17.014 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.503861+0000", 2024-09-13T11:24:17.015 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000102372 2024-09-13T11:24:17.015 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.015 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.015 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 12", 2024-09-13T11:24:17.015 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.503952+0000", 2024-09-13T11:24:17.015 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.1135000000000001e-05 2024-09-13T11:24:17.015 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.015 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.015 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:24:17.015 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.505340+0000", 2024-09-13T11:24:17.015 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0013877169999999999 2024-09-13T11:24:17.015 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.015 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.015 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:17.016 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.768281+0000", 2024-09-13T11:24:17.016 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.26294092400000002 2024-09-13T11:24:17.016 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.016 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.016 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:24:17.016 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.768369+0000", 2024-09-13T11:24:17.016 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.7818999999999994e-05 2024-09-13T11:24:17.016 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.016 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.016 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:24:17.016 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.768438+0000", 2024-09-13T11:24:17.016 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.9307000000000006e-05 2024-09-13T11:24:17.017 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:17.017 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:17.017 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:17.017 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.017 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.017 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:45783 3.16 3.17c10a76 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e69)", 2024-09-13T11:24:17.017 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:20.947774+0000", 2024-09-13T11:24:17.017 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 56.048393439999998, 2024-09-13T11:24:17.017 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.96682922400000004, 2024-09-13T11:24:17.017 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:17.018 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:17.018 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-13T11:24:17.018 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:24:17.018 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:24:17.018 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:17.018 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 45783 2024-09-13T11:24:17.018 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.018 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:17.018 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.018 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:17.018 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:20.947774+0000", 2024-09-13T11:24:17.018 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:17.018 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.018 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.019 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:17.019 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:20.947774+0000", 2024-09-13T11:24:17.019 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:17.019 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.019 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.019 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:17.019 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:20.947777+0000", 2024-09-13T11:24:17.019 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.57e-06 2024-09-13T11:24:17.019 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.019 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.019 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:17.019 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:20.947802+0000", 2024-09-13T11:24:17.019 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.5528999999999999e-05 2024-09-13T11:24:17.019 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.019 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.020 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:17.020 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:20.947813+0000", 2024-09-13T11:24:17.020 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0473e-05 2024-09-13T11:24:17.020 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.020 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.020 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:17.020 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:20.947821+0000", 2024-09-13T11:24:17.020 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.3440000000000001e-06 2024-09-13T11:24:17.020 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.020 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.020 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:17.020 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:20.947933+0000", 2024-09-13T11:24:17.020 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00011210200000000001 2024-09-13T11:24:17.020 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.021 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.021 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for peered", 2024-09-13T11:24:17.021 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:20.947942+0000", 2024-09-13T11:24:17.021 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.6610000000000006e-06 2024-09-13T11:24:17.021 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.021 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.021 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:17.021 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:21.914579+0000", 2024-09-13T11:24:17.021 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.96663672300000003 2024-09-13T11:24:17.021 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.021 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.021 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:24:17.021 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:21.914603+0000", 2024-09-13T11:24:17.022 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.4822000000000001e-05 2024-09-13T11:24:17.022 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:17.022 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:17.022 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:17.022 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.022 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.022 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:54625 3.1f 3:f812cdfb:::benchmark_data_smithi177_31116_object54624:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e75)", 2024-09-13T11:24:17.022 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:30.979327+0000", 2024-09-13T11:24:17.022 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 46.016840651000003, 2024-09-13T11:24:17.022 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.96163670899999998, 2024-09-13T11:24:17.022 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:17.022 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:17.022 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-13T11:24:17.023 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:24:17.023 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:24:17.023 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:17.023 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 54625 2024-09-13T11:24:17.023 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.023 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:17.023 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.023 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:17.023 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.979327+0000", 2024-09-13T11:24:17.023 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:17.023 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.023 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.024 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:17.024 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.979327+0000", 2024-09-13T11:24:17.024 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:17.024 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.024 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.024 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:17.024 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.979329+0000", 2024-09-13T11:24:17.024 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.686e-06 2024-09-13T11:24:17.024 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.024 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.024 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:17.024 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.979364+0000", 2024-09-13T11:24:17.024 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.5750000000000002e-05 2024-09-13T11:24:17.025 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.025 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.025 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:17.025 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.979408+0000", 2024-09-13T11:24:17.025 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.3795999999999999e-05 2024-09-13T11:24:17.025 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.025 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.025 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:17.025 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.979427+0000", 2024-09-13T11:24:17.025 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.9040999999999999e-05 2024-09-13T11:24:17.025 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.025 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.025 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:17.026 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.122291+0000", 2024-09-13T11:24:17.026 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.14286373599999999 2024-09-13T11:24:17.026 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.026 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.026 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for peered", 2024-09-13T11:24:17.026 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.122298+0000", 2024-09-13T11:24:17.026 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.3719999999999999e-06 2024-09-13T11:24:17.026 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.026 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.026 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:17.026 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.940341+0000", 2024-09-13T11:24:17.026 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.81804249699999998 2024-09-13T11:24:17.026 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.027 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.027 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:24:17.027 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.940964+0000", 2024-09-13T11:24:17.027 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00062283100000000001 2024-09-13T11:24:17.027 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:17.027 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:17.027 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:17.027 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.027 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.027 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:54649 3.1c 3:3a457a93:::benchmark_data_smithi177_31116_object54648:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e75)", 2024-09-13T11:24:17.027 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:30.979460+0000", 2024-09-13T11:24:17.027 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 46.016707461000003, 2024-09-13T11:24:17.027 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.95579424000000002, 2024-09-13T11:24:17.027 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:17.028 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:17.028 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-13T11:24:17.028 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:24:17.028 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:24:17.028 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:17.028 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 54649 2024-09-13T11:24:17.028 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.028 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:17.028 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.029 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:17.029 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.979460+0000", 2024-09-13T11:24:17.029 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:17.029 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.029 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.029 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:17.029 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.979460+0000", 2024-09-13T11:24:17.029 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:17.029 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.029 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.029 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:17.029 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.979462+0000", 2024-09-13T11:24:17.029 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5799999999999999e-06 2024-09-13T11:24:17.029 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.030 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.030 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:17.030 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.979524+0000", 2024-09-13T11:24:17.030 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.2521999999999995e-05 2024-09-13T11:24:17.030 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.030 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.030 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:17.030 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.979561+0000", 2024-09-13T11:24:17.030 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.6349e-05 2024-09-13T11:24:17.030 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.030 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.030 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:17.030 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.979571+0000", 2024-09-13T11:24:17.030 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0457e-05 2024-09-13T11:24:17.031 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.031 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.031 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:17.031 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.121029+0000", 2024-09-13T11:24:17.031 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.14145766200000001 2024-09-13T11:24:17.031 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.031 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.031 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for peered", 2024-09-13T11:24:17.031 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.121039+0000", 2024-09-13T11:24:17.031 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0084e-05 2024-09-13T11:24:17.031 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.031 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.031 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:17.032 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.935157+0000", 2024-09-13T11:24:17.032 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.81411807899999999 2024-09-13T11:24:17.032 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.032 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.032 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:24:17.032 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.935254+0000", 2024-09-13T11:24:17.032 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.7507e-05 2024-09-13T11:24:17.032 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:17.032 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:17.032 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:17.032 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.033 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.033 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:54654 3.1c 3:3f0bb31a:::benchmark_data_smithi177_31116_object54653:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e75)", 2024-09-13T11:24:17.033 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:30.979665+0000", 2024-09-13T11:24:17.033 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 46.016502295999999, 2024-09-13T11:24:17.033 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.95623908099999999, 2024-09-13T11:24:17.033 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:17.033 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:17.033 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-13T11:24:17.033 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:24:17.033 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:24:17.033 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:17.033 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 54654 2024-09-13T11:24:17.033 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.034 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:17.034 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.034 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:17.034 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.979665+0000", 2024-09-13T11:24:17.034 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:17.034 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.034 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.034 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:17.034 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.979665+0000", 2024-09-13T11:24:17.034 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:17.034 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.034 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.035 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:17.035 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.979667+0000", 2024-09-13T11:24:17.035 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3540000000000001e-06 2024-09-13T11:24:17.035 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.035 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.035 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:17.035 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.979703+0000", 2024-09-13T11:24:17.035 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.6758000000000003e-05 2024-09-13T11:24:17.035 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.035 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.035 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:17.036 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.979714+0000", 2024-09-13T11:24:17.036 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0804e-05 2024-09-13T11:24:17.036 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.036 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.036 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:17.036 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.979724+0000", 2024-09-13T11:24:17.036 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0040999999999999e-05 2024-09-13T11:24:17.036 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.036 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.036 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:17.036 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.121061+0000", 2024-09-13T11:24:17.037 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.141336713 2024-09-13T11:24:17.037 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.037 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.037 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for peered", 2024-09-13T11:24:17.037 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.121067+0000", 2024-09-13T11:24:17.037 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.4150000000000004e-06 2024-09-13T11:24:17.037 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.037 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.037 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:17.037 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.935776+0000", 2024-09-13T11:24:17.037 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.81470863100000002 2024-09-13T11:24:17.037 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.037 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.038 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:24:17.038 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.935904+0000", 2024-09-13T11:24:17.038 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00012836499999999999 2024-09-13T11:24:17.038 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:17.038 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:17.038 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:17.038 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.038 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.038 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPushReply(3.11 75/70 [PushReplyOp(3:8df10771:::benchmark_data_smithi177_31116_object24734:head)])", 2024-09-13T11:24:17.038 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:31.149798+0000", 2024-09-13T11:24:17.038 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 45.846369285999998, 2024-09-13T11:24:17.038 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.32402702500000002, 2024-09-13T11:24:17.038 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:17.039 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:17.039 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-13T11:24:17.039 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:17.039 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.039 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:17.039 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.149798+0000", 2024-09-13T11:24:17.039 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:17.039 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.039 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.039 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:17.039 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.149798+0000", 2024-09-13T11:24:17.039 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:17.039 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.039 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.040 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:17.040 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.149799+0000", 2024-09-13T11:24:17.040 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.5700000000000001e-07 2024-09-13T11:24:17.040 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.040 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.040 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:17.040 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.149803+0000", 2024-09-13T11:24:17.040 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.0040000000000001e-06 2024-09-13T11:24:17.040 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.040 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.040 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:17.040 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.472172+0000", 2024-09-13T11:24:17.040 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.32236856699999999 2024-09-13T11:24:17.040 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.041 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.041 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:17.041 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.472187+0000", 2024-09-13T11:24:17.041 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5016e-05 2024-09-13T11:24:17.041 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.041 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.041 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:17.041 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.472302+0000", 2024-09-13T11:24:17.041 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000115121 2024-09-13T11:24:17.041 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.041 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.041 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:24:17.041 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.473825+0000", 2024-09-13T11:24:17.041 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0015234599999999999 2024-09-13T11:24:17.042 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:17.042 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:17.042 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:17.042 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.042 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.042 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:54668 3.1c 3:3d2c5e40:::benchmark_data_smithi177_31116_object54667:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e75)", 2024-09-13T11:24:17.042 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:31.245525+0000", 2024-09-13T11:24:17.042 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 45.750643029000003, 2024-09-13T11:24:17.042 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.69828973000000005, 2024-09-13T11:24:17.042 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:17.042 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:17.042 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:17.042 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:24:17.042 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:24:17.043 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:17.043 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 54668 2024-09-13T11:24:17.043 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.043 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:17.043 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.043 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:17.043 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.245525+0000", 2024-09-13T11:24:17.043 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:17.043 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.043 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.043 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:17.043 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.245525+0000", 2024-09-13T11:24:17.043 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:17.043 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.044 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.044 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:17.044 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.245526+0000", 2024-09-13T11:24:17.044 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.719e-06 2024-09-13T11:24:17.044 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.044 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.044 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:17.044 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.245547+0000", 2024-09-13T11:24:17.044 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.1084999999999999e-05 2024-09-13T11:24:17.044 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.044 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.044 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:17.044 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.245558+0000", 2024-09-13T11:24:17.044 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0934000000000001e-05 2024-09-13T11:24:17.045 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.045 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.045 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:17.045 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.245564+0000", 2024-09-13T11:24:17.045 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.6269999999999997e-06 2024-09-13T11:24:17.045 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.045 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.045 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:17.045 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.245620+0000", 2024-09-13T11:24:17.045 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.6289999999999998e-05 2024-09-13T11:24:17.045 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.045 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.045 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for peered", 2024-09-13T11:24:17.045 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.245632+0000", 2024-09-13T11:24:17.046 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.1748e-05 2024-09-13T11:24:17.046 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.046 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.046 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:17.046 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.935976+0000", 2024-09-13T11:24:17.046 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.69034455299999997 2024-09-13T11:24:17.046 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.046 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.046 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:24:17.046 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.936302+0000", 2024-09-13T11:24:17.046 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000325397 2024-09-13T11:24:17.046 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.046 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.046 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 13", 2024-09-13T11:24:17.047 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.936459+0000", 2024-09-13T11:24:17.047 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00015754300000000001 2024-09-13T11:24:17.047 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.047 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.047 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:24:17.047 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.938813+0000", 2024-09-13T11:24:17.047 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0023530999999999999 2024-09-13T11:24:17.047 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.047 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.047 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:17.047 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.943695+0000", 2024-09-13T11:24:17.047 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.004882506 2024-09-13T11:24:17.047 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.047 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.048 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:24:17.048 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.943767+0000", 2024-09-13T11:24:17.048 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.2454999999999997e-05 2024-09-13T11:24:17.048 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.048 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.048 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:24:17.048 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.943814+0000", 2024-09-13T11:24:17.048 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.6773000000000001e-05 2024-09-13T11:24:17.048 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:17.048 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:17.048 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:17.048 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.048 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.048 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPush(3.16 84/70 [PushOp(3:6f7be541:::benchmark_data_smithi177_31116_object43332:head, version: 66'2788, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:6f7be541:::benchmark_data_smithi177_31116_object43332:head@66'2788, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:24:17.049 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:41.207515+0000", 2024-09-13T11:24:17.049 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 35.788652257999999, 2024-09-13T11:24:17.049 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.52643771500000003, 2024-09-13T11:24:17.049 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:17.049 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:17.049 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:24:17.049 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:17.049 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.049 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:17.049 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.207515+0000", 2024-09-13T11:24:17.049 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:17.049 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.050 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.050 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:17.050 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.207515+0000", 2024-09-13T11:24:17.050 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:17.050 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.050 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.050 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:17.050 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.207516+0000", 2024-09-13T11:24:17.050 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.096e-06 2024-09-13T11:24:17.050 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.050 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.050 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:17.050 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.207717+0000", 2024-09-13T11:24:17.050 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000200199 2024-09-13T11:24:17.050 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.051 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.051 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:17.051 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.732785+0000", 2024-09-13T11:24:17.051 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.52506869599999995 2024-09-13T11:24:17.051 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.051 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.051 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:17.051 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.732801+0000", 2024-09-13T11:24:17.051 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5305999999999999e-05 2024-09-13T11:24:17.051 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.051 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.051 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:17.051 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.732889+0000", 2024-09-13T11:24:17.051 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.8157000000000003e-05 2024-09-13T11:24:17.052 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.052 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.052 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:24:17.052 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.732901+0000", 2024-09-13T11:24:17.052 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.1866e-05 2024-09-13T11:24:17.052 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.052 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.052 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:24:17.052 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.733953+0000", 2024-09-13T11:24:17.052 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001052395 2024-09-13T11:24:17.052 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:17.052 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:17.052 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:17.052 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.053 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.053 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPushReply(3.1f 85/78 [PushReplyOp(3:fd221d21:::benchmark_data_smithi177_31116_object7639:head)])", 2024-09-13T11:24:17.053 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:42.198715+0000", 2024-09-13T11:24:17.053 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 34.797452688999996, 2024-09-13T11:24:17.053 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.48930391099999998, 2024-09-13T11:24:17.053 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:17.053 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:17.053 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-13T11:24:17.053 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:17.053 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.053 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:17.053 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.198715+0000", 2024-09-13T11:24:17.053 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:17.053 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.054 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.054 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:17.054 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.198715+0000", 2024-09-13T11:24:17.054 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:17.054 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.054 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.054 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:17.054 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.198716+0000", 2024-09-13T11:24:17.054 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.3500000000000005e-07 2024-09-13T11:24:17.054 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.054 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.054 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:17.054 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.198719+0000", 2024-09-13T11:24:17.054 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.4539999999999999e-06 2024-09-13T11:24:17.054 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.055 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.055 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:17.055 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.684732+0000", 2024-09-13T11:24:17.055 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.48601262099999998 2024-09-13T11:24:17.055 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.055 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.055 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:17.055 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.684746+0000", 2024-09-13T11:24:17.055 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.4518999999999999e-05 2024-09-13T11:24:17.055 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.055 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.055 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:17.055 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.684911+0000", 2024-09-13T11:24:17.055 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000164728 2024-09-13T11:24:17.055 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.056 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.056 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:24:17.056 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.688019+0000", 2024-09-13T11:24:17.056 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0031077539999999999 2024-09-13T11:24:17.056 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:17.056 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:17.056 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:17.056 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.056 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.056 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPushReply(3.1c 85/77 [PushReplyOp(3:3a669437:::benchmark_data_smithi177_31116_object25280:head)])", 2024-09-13T11:24:17.056 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:42.352466+0000", 2024-09-13T11:24:17.056 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 34.643701794999998, 2024-09-13T11:24:17.056 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.36670725300000001, 2024-09-13T11:24:17.057 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:17.057 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:17.057 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-13T11:24:17.057 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:17.057 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.057 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:17.057 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.352466+0000", 2024-09-13T11:24:17.057 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:17.057 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.057 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.057 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:17.057 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.352466+0000", 2024-09-13T11:24:17.057 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:17.057 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.058 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.058 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:17.058 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.352467+0000", 2024-09-13T11:24:17.058 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.1429999999999999e-06 2024-09-13T11:24:17.058 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.058 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.058 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:17.058 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.352474+0000", 2024-09-13T11:24:17.058 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.6100000000000002e-06 2024-09-13T11:24:17.058 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.058 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.058 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:17.058 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.717011+0000", 2024-09-13T11:24:17.058 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.36453790200000002 2024-09-13T11:24:17.059 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.059 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.059 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:17.059 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.717032+0000", 2024-09-13T11:24:17.059 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.0999e-05 2024-09-13T11:24:17.059 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.059 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.059 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:17.059 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.717162+0000", 2024-09-13T11:24:17.059 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000129159 2024-09-13T11:24:17.059 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.059 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.059 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:24:17.059 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.719173+0000", 2024-09-13T11:24:17.059 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0020114400000000002 2024-09-13T11:24:17.060 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:17.060 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:17.060 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:17.060 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.060 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.060 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPush(3.16 85/70 [PushOp(3:6aef2f27:::benchmark_data_smithi177_31116_object44090:head, version: 68'2844, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:6aef2f27:::benchmark_data_smithi177_31116_object44090:head@68'2844, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:24:17.060 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:42.574221+0000", 2024-09-13T11:24:17.060 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 34.421946687000002, 2024-09-13T11:24:17.060 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.25345585700000001, 2024-09-13T11:24:17.060 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:17.060 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:17.060 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:24:17.060 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:17.061 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.061 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:17.061 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.574221+0000", 2024-09-13T11:24:17.061 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:17.061 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.061 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.061 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:17.061 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.574221+0000", 2024-09-13T11:24:17.061 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:17.061 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.061 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.061 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:17.061 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.574252+0000", 2024-09-13T11:24:17.062 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.0973000000000003e-05 2024-09-13T11:24:17.062 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.062 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.062 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:17.062 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.574434+0000", 2024-09-13T11:24:17.062 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000182265 2024-09-13T11:24:17.063 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.063 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.063 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:17.063 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.825391+0000", 2024-09-13T11:24:17.063 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.250957023 2024-09-13T11:24:17.063 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.063 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.064 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:17.064 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.825442+0000", 2024-09-13T11:24:17.064 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.1267999999999998e-05 2024-09-13T11:24:17.064 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.064 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.064 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:17.065 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.825692+0000", 2024-09-13T11:24:17.065 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00024924700000000002 2024-09-13T11:24:17.065 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.065 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.065 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:24:17.065 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.825715+0000", 2024-09-13T11:24:17.065 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.3176e-05 2024-09-13T11:24:17.066 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.066 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.066 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:24:17.066 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.827677+0000", 2024-09-13T11:24:17.066 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0019619049999999999 2024-09-13T11:24:17.066 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:17.066 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:17.066 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:17.066 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.067 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.067 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPushReply(3.1f 85/78 [PushReplyOp(3:fecb0d6a:::benchmark_data_smithi177_31116_object7644:head)])", 2024-09-13T11:24:17.067 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:42.705351+0000", 2024-09-13T11:24:17.067 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 34.290816952999997, 2024-09-13T11:24:17.067 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.290549681, 2024-09-13T11:24:17.067 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:17.067 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:17.068 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-13T11:24:17.068 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:17.068 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.068 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:17.068 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.705351+0000", 2024-09-13T11:24:17.068 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:17.069 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.069 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.069 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:17.069 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.705351+0000", 2024-09-13T11:24:17.069 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:17.069 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.070 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.070 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:17.070 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.705353+0000", 2024-09-13T11:24:17.070 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.4169999999999999e-06 2024-09-13T11:24:17.070 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.070 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.070 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:17.071 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.705365+0000", 2024-09-13T11:24:17.071 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.1894e-05 2024-09-13T11:24:17.071 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.071 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.071 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:17.071 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.992081+0000", 2024-09-13T11:24:17.071 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.28671641199999998 2024-09-13T11:24:17.071 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.071 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.071 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:17.072 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.992111+0000", 2024-09-13T11:24:17.072 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.9427000000000001e-05 2024-09-13T11:24:17.072 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.072 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.072 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:17.072 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.993573+0000", 2024-09-13T11:24:17.072 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0014623590000000001 2024-09-13T11:24:17.072 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.072 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.073 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:24:17.073 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.995900+0000", 2024-09-13T11:24:17.073 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0023271720000000002 2024-09-13T11:24:17.073 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:17.073 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:17.073 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:17.073 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.073 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.073 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPushReply(3.21 89/78 [PushReplyOp(3:8534d087:::benchmark_data_smithi177_31116_object5474:head)])", 2024-09-13T11:24:17.073 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:46.632834+0000", 2024-09-13T11:24:17.073 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 30.363333513000001, 2024-09-13T11:24:17.073 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.324805556, 2024-09-13T11:24:17.073 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:17.073 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:17.074 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-13T11:24:17.074 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:17.074 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.074 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:17.074 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.632834+0000", 2024-09-13T11:24:17.074 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:17.074 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.074 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.074 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:17.074 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.632834+0000", 2024-09-13T11:24:17.074 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:17.074 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.074 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.074 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:17.075 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.632835+0000", 2024-09-13T11:24:17.075 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.4270000000000001e-06 2024-09-13T11:24:17.075 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.075 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.075 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:17.075 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.632840+0000", 2024-09-13T11:24:17.075 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.7809999999999996e-06 2024-09-13T11:24:17.075 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.075 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.075 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:17.075 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.956282+0000", 2024-09-13T11:24:17.075 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.323441639 2024-09-13T11:24:17.075 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.076 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.076 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:17.076 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.956293+0000", 2024-09-13T11:24:17.076 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.083e-05 2024-09-13T11:24:17.076 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.076 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.076 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:17.077 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.956395+0000", 2024-09-13T11:24:17.077 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00010184200000000001 2024-09-13T11:24:17.077 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.077 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.077 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:24:17.077 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.957640+0000", 2024-09-13T11:24:17.077 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0012450370000000001 2024-09-13T11:24:17.077 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:17.077 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:17.077 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:17.077 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.077 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.077 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPushReply(3.1f 89/78 [PushReplyOp(3:ff150976:::benchmark_data_smithi177_31116_object12907:head)])", 2024-09-13T11:24:17.077 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:46.697504+0000", 2024-09-13T11:24:17.078 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 30.298663913999999, 2024-09-13T11:24:17.078 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.26836818099999998, 2024-09-13T11:24:17.078 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:17.078 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:17.078 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-13T11:24:17.078 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:17.078 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.078 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:17.078 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.697504+0000", 2024-09-13T11:24:17.078 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:17.078 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.078 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.078 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:17.078 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.697504+0000", 2024-09-13T11:24:17.079 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:17.079 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.079 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.079 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:17.079 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.697505+0000", 2024-09-13T11:24:17.079 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0529999999999999e-06 2024-09-13T11:24:17.079 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.079 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.079 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:17.079 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.697509+0000", 2024-09-13T11:24:17.079 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.9249999999999997e-06 2024-09-13T11:24:17.079 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.079 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.079 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:17.080 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.964086+0000", 2024-09-13T11:24:17.080 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.26657776100000002 2024-09-13T11:24:17.080 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.080 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.080 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:17.080 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.964110+0000", 2024-09-13T11:24:17.080 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.3189000000000001e-05 2024-09-13T11:24:17.080 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.080 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.080 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:17.080 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.964354+0000", 2024-09-13T11:24:17.080 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00024481399999999999 2024-09-13T11:24:17.080 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.080 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.081 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:24:17.081 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.965872+0000", 2024-09-13T11:24:17.081 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001517439 2024-09-13T11:24:17.081 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:17.081 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:17.081 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:17.081 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.081 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.081 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:71589 3.21 3:85f71b95:::benchmark_data_smithi177_31116_object71588:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96)", 2024-09-13T11:24:17.081 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:53.876584+0000", 2024-09-13T11:24:17.081 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 23.119583461000001, 2024-09-13T11:24:17.081 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.24303730800000001, 2024-09-13T11:24:17.081 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:17.082 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:17.082 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:17.082 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:24:17.082 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:24:17.082 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:17.082 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 71589 2024-09-13T11:24:17.082 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.082 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:17.082 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.082 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:17.082 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:53.876584+0000", 2024-09-13T11:24:17.082 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:17.082 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.082 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.083 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:17.083 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:53.876584+0000", 2024-09-13T11:24:17.083 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:17.083 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.083 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.083 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:17.083 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:53.876586+0000", 2024-09-13T11:24:17.083 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.4100000000000001e-06 2024-09-13T11:24:17.083 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.083 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.083 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:17.083 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:53.876604+0000", 2024-09-13T11:24:17.083 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.8389e-05 2024-09-13T11:24:17.083 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.084 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.084 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:17.084 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:53.876613+0000", 2024-09-13T11:24:17.084 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.8789999999999992e-06 2024-09-13T11:24:17.084 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.084 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.084 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:17.084 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:53.876619+0000", 2024-09-13T11:24:17.084 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.9699999999999996e-06 2024-09-13T11:24:17.084 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.084 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.084 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:17.084 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:53.876681+0000", 2024-09-13T11:24:17.085 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.2699000000000004e-05 2024-09-13T11:24:17.085 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.085 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.085 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:24:17.085 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:53.876849+0000", 2024-09-13T11:24:17.085 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00016714799999999999 2024-09-13T11:24:17.085 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.085 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.085 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 0,8", 2024-09-13T11:24:17.085 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:53.877032+0000", 2024-09-13T11:24:17.085 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00018387199999999999 2024-09-13T11:24:17.085 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.085 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.085 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:24:17.086 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:53.879473+0000", 2024-09-13T11:24:17.086 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0024403789999999999 2024-09-13T11:24:17.086 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.086 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.086 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:17.086 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:53.880046+0000", 2024-09-13T11:24:17.086 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00057281200000000002 2024-09-13T11:24:17.086 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.086 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.086 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:17.086 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.119299+0000", 2024-09-13T11:24:17.086 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.23925310399999999 2024-09-13T11:24:17.086 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.087 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.087 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:24:17.087 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.119337+0000", 2024-09-13T11:24:17.087 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.8109999999999999e-05 2024-09-13T11:24:17.087 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.087 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.087 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:24:17.087 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.119621+0000", 2024-09-13T11:24:17.087 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000284536 2024-09-13T11:24:17.087 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:17.087 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:17.087 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:17.087 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.087 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.087 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop_reply(client.4665.0:71589 3.21 e96/78 ondisk, result = 0)", 2024-09-13T11:24:17.088 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:53.887108+0000", 2024-09-13T11:24:17.088 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 23.109059860999999, 2024-09-13T11:24:17.088 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.232519532, 2024-09-13T11:24:17.088 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:17.088 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:17.088 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:24:17.088 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:17.088 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.088 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:17.088 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:53.887108+0000", 2024-09-13T11:24:17.088 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:17.088 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.088 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.089 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:17.089 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:53.887108+0000", 2024-09-13T11:24:17.089 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:17.089 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.089 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.089 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:17.089 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:53.887109+0000", 2024-09-13T11:24:17.089 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3370000000000001e-06 2024-09-13T11:24:17.089 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.089 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.089 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:17.089 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:53.887113+0000", 2024-09-13T11:24:17.089 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.4630000000000001e-06 2024-09-13T11:24:17.090 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.090 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.090 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:17.090 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.119248+0000", 2024-09-13T11:24:17.090 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.232135434 2024-09-13T11:24:17.090 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.090 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.090 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:17.090 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.119252+0000", 2024-09-13T11:24:17.090 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.1930000000000003e-06 2024-09-13T11:24:17.090 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.090 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.090 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:17.091 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.119289+0000", 2024-09-13T11:24:17.091 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.701e-05 2024-09-13T11:24:17.091 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.091 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.091 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:24:17.091 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.119295+0000", 2024-09-13T11:24:17.091 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.7250000000000002e-06 2024-09-13T11:24:17.091 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.091 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.091 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:24:17.091 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.119627+0000", 2024-09-13T11:24:17.091 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00033237000000000002 2024-09-13T11:24:17.091 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:17.091 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:17.092 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:17.092 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.092 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.092 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:83223 3.1c 3:3bc6018d:::benchmark_data_smithi177_31116_object83222:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106)", 2024-09-13T11:24:17.092 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:04.511163+0000", 2024-09-13T11:24:17.092 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 12.485004136000001, 2024-09-13T11:24:17.092 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.47558093200000001, 2024-09-13T11:24:17.092 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:17.092 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:17.092 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:17.092 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:24:17.092 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:24:17.092 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:17.092 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 83223 2024-09-13T11:24:17.092 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.093 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:17.093 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.093 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:17.093 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.511163+0000", 2024-09-13T11:24:17.093 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:17.093 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.093 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.093 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:17.093 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.511163+0000", 2024-09-13T11:24:17.093 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:17.093 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.093 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.093 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:17.094 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.511167+0000", 2024-09-13T11:24:17.094 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.8330000000000004e-06 2024-09-13T11:24:17.094 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.094 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.094 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:17.094 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.511219+0000", 2024-09-13T11:24:17.094 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.1771999999999998e-05 2024-09-13T11:24:17.094 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.094 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.094 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:17.094 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.511252+0000", 2024-09-13T11:24:17.094 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.3315999999999997e-05 2024-09-13T11:24:17.094 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.094 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.095 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:17.095 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.511268+0000", 2024-09-13T11:24:17.095 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5812999999999999e-05 2024-09-13T11:24:17.095 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.095 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.095 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:17.095 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.511418+0000", 2024-09-13T11:24:17.095 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000150023 2024-09-13T11:24:17.095 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.095 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.095 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:24:17.095 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.511670+0000", 2024-09-13T11:24:17.095 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00025185499999999999 2024-09-13T11:24:17.095 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.096 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.096 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 7", 2024-09-13T11:24:17.096 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.511905+0000", 2024-09-13T11:24:17.096 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00023493799999999999 2024-09-13T11:24:17.096 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.096 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.096 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:24:17.096 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.514960+0000", 2024-09-13T11:24:17.096 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.003054977 2024-09-13T11:24:17.096 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.096 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.096 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:17.096 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.986432+0000", 2024-09-13T11:24:17.097 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.47147209200000001 2024-09-13T11:24:17.097 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.097 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.097 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:24:17.097 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.986635+0000", 2024-09-13T11:24:17.097 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00020309899999999999 2024-09-13T11:24:17.097 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.097 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.097 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:24:17.097 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.986744+0000", 2024-09-13T11:24:17.097 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000109214 2024-09-13T11:24:17.097 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:17.098 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:17.098 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:17.098 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.098 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.098 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop_reply(client.4665.0:84803 3.1f e107/78)", 2024-09-13T11:24:17.098 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:05.912627+0000", 2024-09-13T11:24:17.098 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 11.083540229, 2024-09-13T11:24:17.098 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.33726566600000002, 2024-09-13T11:24:17.098 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:17.098 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:17.098 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-13T11:24:17.098 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:17.098 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.099 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:17.099 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:05.912627+0000", 2024-09-13T11:24:17.099 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:17.099 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.099 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.099 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:17.099 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:05.912627+0000", 2024-09-13T11:24:17.099 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:17.099 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.099 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.099 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:17.099 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:05.912629+0000", 2024-09-13T11:24:17.099 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.4440000000000001e-06 2024-09-13T11:24:17.099 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.100 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.100 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:17.100 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:05.912634+0000", 2024-09-13T11:24:17.100 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.6869999999999999e-06 2024-09-13T11:24:17.100 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.100 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.100 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:17.100 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:06.249722+0000", 2024-09-13T11:24:17.100 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.33708766099999998 2024-09-13T11:24:17.100 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.100 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.100 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:17.100 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:06.249736+0000", 2024-09-13T11:24:17.101 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3623e-05 2024-09-13T11:24:17.101 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.101 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.101 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:17.101 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:06.249878+0000", 2024-09-13T11:24:17.101 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00014264599999999999 2024-09-13T11:24:17.101 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.101 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.101 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:24:17.101 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:06.249893+0000", 2024-09-13T11:24:17.101 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.4605e-05 2024-09-13T11:24:17.101 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:17.101 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:17.102 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:17.102 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:17.102 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:17.102 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-09-13T11:24:17.103 INFO:tasks.radosbench.radosbench.0.smithi177.stdout: 110 16 100466 100450 57.0681 81.75 0.00922296 0.0175169 2024-09-13T11:24:17.104 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_ops_in_flight 2024-09-13T11:24:17.104 INFO:tasks.ceph.osd.3.smithi177.stderr:2024-09-13T11:24:17.099+0000 7f10adba4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26574) UID: 0 2024-09-13T11:24:17.152 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T11:24:17.204 INFO:tasks.ceph.osd.11.smithi177.stderr:2024-09-13T11:24:17.203+0000 7f24792c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26580) UID: 0 2024-09-13T11:24:17.288 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-09-13T11:24:17.289 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [], 2024-09-13T11:24:17.289 INFO:teuthology.orchestra.run.smithi177.stdout: "num_ops": 0 2024-09-13T11:24:17.289 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-09-13T11:24:17.305 INFO:tasks.ceph.osd.7.smithi177.stderr:2024-09-13T11:24:17.303+0000 7f897b51d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26575) UID: 0 2024-09-13T11:24:17.308 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_blocked_ops 2024-09-13T11:24:17.325 INFO:teuthology.orchestra.run.smithi001.stdout:osd.0: {} 2024-09-13T11:24:17.326 INFO:teuthology.orchestra.run.smithi001.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T11:24:17.350 INFO:teuthology.orchestra.run.smithi001.stdout:osd.1: {} 2024-09-13T11:24:17.350 INFO:teuthology.orchestra.run.smithi001.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T11:24:17.373 INFO:teuthology.orchestra.run.smithi001.stdout:osd.2: {} 2024-09-13T11:24:17.373 INFO:teuthology.orchestra.run.smithi001.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T11:24:17.396 INFO:teuthology.orchestra.run.smithi001.stdout:osd.3: {} 2024-09-13T11:24:17.396 INFO:teuthology.orchestra.run.smithi001.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T11:24:17.406 INFO:tasks.ceph.osd.12.smithi001.stderr:2024-09-13T11:24:17.399+0000 7f892c6e6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26314) UID: 0 2024-09-13T11:24:17.419 INFO:teuthology.orchestra.run.smithi001.stdout:osd.4: {} 2024-09-13T11:24:17.419 INFO:teuthology.orchestra.run.smithi001.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T11:24:17.443 INFO:teuthology.orchestra.run.smithi001.stdout:osd.5: {} 2024-09-13T11:24:17.443 INFO:teuthology.orchestra.run.smithi001.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T11:24:17.469 INFO:teuthology.orchestra.run.smithi001.stdout:osd.6: {} 2024-09-13T11:24:17.470 INFO:teuthology.orchestra.run.smithi001.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T11:24:17.472 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-09-13T11:24:17.473 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [], 2024-09-13T11:24:17.473 INFO:teuthology.orchestra.run.smithi177.stdout: "complaint_time": 30, 2024-09-13T11:24:17.473 INFO:teuthology.orchestra.run.smithi177.stdout: "num_blocked_ops": 0 2024-09-13T11:24:17.473 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-09-13T11:24:17.490 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_historic_ops 2024-09-13T11:24:17.493 INFO:teuthology.orchestra.run.smithi001.stdout:osd.7: {} 2024-09-13T11:24:17.494 INFO:teuthology.orchestra.run.smithi001.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T11:24:17.508 INFO:tasks.ceph.osd.13.smithi092.stderr:2024-09-13T11:24:17.503+0000 7f3e5afea640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26350) UID: 0 2024-09-13T11:24:17.516 INFO:teuthology.orchestra.run.smithi001.stdout:osd.8: {} 2024-09-13T11:24:17.517 INFO:teuthology.orchestra.run.smithi001.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-13T11:24:17.545 INFO:teuthology.orchestra.run.smithi001.stdout:osd.9: {} 2024-09-13T11:24:17.545 INFO:teuthology.orchestra.run.smithi001.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-13T11:24:17.572 INFO:teuthology.orchestra.run.smithi001.stdout:osd.10: {} 2024-09-13T11:24:17.572 INFO:teuthology.orchestra.run.smithi001.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-13T11:24:17.597 INFO:teuthology.orchestra.run.smithi001.stdout:osd.11: {} 2024-09-13T11:24:17.597 INFO:teuthology.orchestra.run.smithi001.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-13T11:24:17.608 INFO:tasks.ceph.osd.15.smithi177.stderr:2024-09-13T11:24:17.607+0000 7f9d6ce77640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26578) UID: 0 2024-09-13T11:24:17.617 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-09-13T11:24:17.618 INFO:teuthology.orchestra.run.smithi177.stdout: "size": 20, 2024-09-13T11:24:17.618 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 600, 2024-09-13T11:24:17.618 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [ 2024-09-13T11:24:17.618 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.618 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:25383 3.0 e55/30 3:0a6bb869:::benchmark_data_smithi177_31116_object25382:head v 55'1620, mlcod=55'1617)", 2024-09-13T11:24:17.618 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:04.449956+0000", 2024-09-13T11:24:17.618 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 73.166036462999998, 2024-09-13T11:24:17.618 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.21143479600000001, 2024-09-13T11:24:17.618 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:17.618 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:17.618 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:17.618 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:17.618 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.619 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:17.619 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.449956+0000", 2024-09-13T11:24:17.619 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:17.619 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.619 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.619 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:17.619 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.449956+0000", 2024-09-13T11:24:17.619 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:17.619 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.619 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.619 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:17.619 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.449959+0000", 2024-09-13T11:24:17.620 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.393e-06 2024-09-13T11:24:17.620 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.620 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.620 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:17.620 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.450014+0000", 2024-09-13T11:24:17.620 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.4200000000000003e-05 2024-09-13T11:24:17.620 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.620 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.620 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:17.620 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.658637+0000", 2024-09-13T11:24:17.620 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.20862334799999999 2024-09-13T11:24:17.620 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.620 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.620 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:17.621 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.658650+0000", 2024-09-13T11:24:17.621 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3148e-05 2024-09-13T11:24:17.621 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.621 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.621 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:17.621 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.658867+0000", 2024-09-13T11:24:17.621 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00021673499999999999 2024-09-13T11:24:17.621 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.621 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.621 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:24:17.621 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.658931+0000", 2024-09-13T11:24:17.621 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.4045000000000001e-05 2024-09-13T11:24:17.621 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.621 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.622 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:24:17.622 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.661317+0000", 2024-09-13T11:24:17.622 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.002385823 2024-09-13T11:24:17.622 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.622 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.622 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:24:17.622 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.661391+0000", 2024-09-13T11:24:17.622 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.4103999999999994e-05 2024-09-13T11:24:17.622 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:17.622 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:17.622 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:17.622 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.622 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.623 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPush(3.16 85/70 [PushOp(3:6ac421eb:::benchmark_data_smithi177_31116_object43910:head, version: 66'2836, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:6ac421eb:::benchmark_data_smithi177_31116_object43910:head@66'2836, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:24:17.623 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:42.145291+0000", 2024-09-13T11:24:17.623 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 35.470701257000002, 2024-09-13T11:24:17.623 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.148514919, 2024-09-13T11:24:17.623 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:17.623 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:17.623 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:24:17.623 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:17.623 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.623 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:17.623 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.145291+0000", 2024-09-13T11:24:17.623 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:17.623 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.623 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.624 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:17.624 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.145291+0000", 2024-09-13T11:24:17.624 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:17.624 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.624 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.624 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:17.624 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.145293+0000", 2024-09-13T11:24:17.624 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.699e-06 2024-09-13T11:24:17.624 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.624 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.624 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:17.624 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.145417+0000", 2024-09-13T11:24:17.624 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00012415200000000001 2024-09-13T11:24:17.625 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.625 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.625 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:17.625 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.292964+0000", 2024-09-13T11:24:17.625 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.14754715400000001 2024-09-13T11:24:17.625 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.625 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.625 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:17.625 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.292981+0000", 2024-09-13T11:24:17.625 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.6790999999999999e-05 2024-09-13T11:24:17.625 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.625 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.625 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:17.626 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.293083+0000", 2024-09-13T11:24:17.626 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00010199799999999999 2024-09-13T11:24:17.626 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.626 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.626 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:24:17.626 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.293098+0000", 2024-09-13T11:24:17.626 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.4612e-05 2024-09-13T11:24:17.626 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.626 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.626 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:24:17.626 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.293806+0000", 2024-09-13T11:24:17.626 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00070851300000000002 2024-09-13T11:24:17.626 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:17.627 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:17.627 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:17.627 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.627 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.627 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:60390 3.16 e85/70 3:6a388186:::benchmark_data_smithi177_31116_object60389:head v 85'3357, mlcod=66'2835)", 2024-09-13T11:24:17.627 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:42.149102+0000", 2024-09-13T11:24:17.627 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 35.466890200000002, 2024-09-13T11:24:17.627 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.14821161799999999, 2024-09-13T11:24:17.627 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:17.627 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:17.627 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:17.627 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:17.627 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.627 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:17.628 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.149102+0000", 2024-09-13T11:24:17.628 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:17.628 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.628 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.628 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:17.628 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.149102+0000", 2024-09-13T11:24:17.628 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:17.628 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.628 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.628 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:17.628 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.149104+0000", 2024-09-13T11:24:17.628 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.708e-06 2024-09-13T11:24:17.629 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.629 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.629 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:17.629 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.149160+0000", 2024-09-13T11:24:17.629 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.5618000000000002e-05 2024-09-13T11:24:17.629 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.629 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.629 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:17.629 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.296071+0000", 2024-09-13T11:24:17.629 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.14691148600000001 2024-09-13T11:24:17.629 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.629 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.629 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:17.629 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.296076+0000", 2024-09-13T11:24:17.630 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.5580000000000002e-06 2024-09-13T11:24:17.630 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.630 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.630 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:17.630 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.296127+0000", 2024-09-13T11:24:17.630 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.1405e-05 2024-09-13T11:24:17.630 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.630 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.630 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:24:17.630 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.296150+0000", 2024-09-13T11:24:17.630 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.3113e-05 2024-09-13T11:24:17.630 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.630 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.630 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:24:17.631 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.297274+0000", 2024-09-13T11:24:17.631 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0011239900000000001 2024-09-13T11:24:17.631 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.631 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.631 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:24:17.631 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.297314+0000", 2024-09-13T11:24:17.631 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.9740000000000002e-05 2024-09-13T11:24:17.631 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:17.631 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:17.631 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:17.631 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.631 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.631 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPush(3.16 85/70 [PushOp(3:69aac9fd:::benchmark_data_smithi177_31116_object43914:head, version: 66'2837, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:69aac9fd:::benchmark_data_smithi177_31116_object43914:head@66'2837, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:24:17.632 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:42.295995+0000", 2024-09-13T11:24:17.632 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 35.319997143000002, 2024-09-13T11:24:17.632 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.215089431, 2024-09-13T11:24:17.632 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:17.632 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:17.632 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:24:17.632 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:17.632 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.632 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:17.632 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.295995+0000", 2024-09-13T11:24:17.632 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:17.632 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.633 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.633 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:17.633 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.295995+0000", 2024-09-13T11:24:17.633 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:17.633 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.633 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.633 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:17.633 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.295997+0000", 2024-09-13T11:24:17.633 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3170000000000001e-06 2024-09-13T11:24:17.633 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.633 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.633 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:17.633 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.296058+0000", 2024-09-13T11:24:17.634 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.1690000000000001e-05 2024-09-13T11:24:17.634 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.634 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.634 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:17.634 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.509606+0000", 2024-09-13T11:24:17.634 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.21354748300000001 2024-09-13T11:24:17.634 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.634 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.634 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:17.634 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.509638+0000", 2024-09-13T11:24:17.634 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.1644999999999999e-05 2024-09-13T11:24:17.634 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.634 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.635 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:17.635 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.509814+0000", 2024-09-13T11:24:17.635 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00017679499999999999 2024-09-13T11:24:17.635 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.635 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.635 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:24:17.635 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.509840+0000", 2024-09-13T11:24:17.635 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.5871e-05 2024-09-13T11:24:17.635 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.635 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.635 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:24:17.635 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.511085+0000", 2024-09-13T11:24:17.635 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00124463 2024-09-13T11:24:17.635 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:17.636 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:17.636 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:17.636 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.636 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.636 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPush(3.1d 91/78 [PushOp(3:bb0a08cd:::benchmark_data_smithi177_31116_object25660:head, version: 58'1568, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:bb0a08cd:::benchmark_data_smithi177_31116_object25660:head@58'1568, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:24:17.636 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:48.351121+0000", 2024-09-13T11:24:17.636 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 29.264871848999999, 2024-09-13T11:24:17.636 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.12585117900000001, 2024-09-13T11:24:17.636 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:17.636 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:17.636 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:24:17.636 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:17.636 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.637 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:17.637 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.351121+0000", 2024-09-13T11:24:17.637 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:17.637 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.637 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.637 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:17.637 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.351121+0000", 2024-09-13T11:24:17.637 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:17.637 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.637 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.637 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:17.637 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.351122+0000", 2024-09-13T11:24:17.637 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3009999999999999e-06 2024-09-13T11:24:17.638 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.638 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.638 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:17.638 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.351147+0000", 2024-09-13T11:24:17.638 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.4967999999999999e-05 2024-09-13T11:24:17.638 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.638 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.638 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:17.638 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.475788+0000", 2024-09-13T11:24:17.638 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.12464135799999999 2024-09-13T11:24:17.638 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.638 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.638 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:17.638 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.475808+0000", 2024-09-13T11:24:17.638 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.0143999999999999e-05 2024-09-13T11:24:17.639 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.639 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.639 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:17.639 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.475908+0000", 2024-09-13T11:24:17.639 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.9134000000000003e-05 2024-09-13T11:24:17.639 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.639 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.639 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:24:17.639 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.475922+0000", 2024-09-13T11:24:17.639 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.469e-05 2024-09-13T11:24:17.639 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.639 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.639 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:24:17.640 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.476972+0000", 2024-09-13T11:24:17.640 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0010495840000000001 2024-09-13T11:24:17.640 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:17.640 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:17.640 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:17.640 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.640 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.640 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPush(3.1d 91/78 [PushOp(3:be6a0ae4:::benchmark_data_smithi177_31116_object25671:head, version: 58'1569, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:be6a0ae4:::benchmark_data_smithi177_31116_object25671:head@58'1569, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:24:17.640 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:48.481971+0000", 2024-09-13T11:24:17.640 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 29.134021898, 2024-09-13T11:24:17.640 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.47945454199999998, 2024-09-13T11:24:17.640 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:17.640 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:17.640 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:24:17.641 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:17.641 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.641 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:17.641 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.481971+0000", 2024-09-13T11:24:17.641 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:17.641 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.641 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.641 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:17.641 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.481971+0000", 2024-09-13T11:24:17.641 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:17.641 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.641 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.641 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:17.642 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.481972+0000", 2024-09-13T11:24:17.642 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5179999999999999e-06 2024-09-13T11:24:17.642 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.642 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.642 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:17.642 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.482000+0000", 2024-09-13T11:24:17.642 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.8249e-05 2024-09-13T11:24:17.642 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.642 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.642 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:17.642 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.959926+0000", 2024-09-13T11:24:17.642 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.47792585100000001 2024-09-13T11:24:17.642 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.642 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.643 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:17.643 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.959944+0000", 2024-09-13T11:24:17.643 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.7620000000000001e-05 2024-09-13T11:24:17.643 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.643 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.643 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:17.643 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.960059+0000", 2024-09-13T11:24:17.643 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000115188 2024-09-13T11:24:17.643 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.643 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.643 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:24:17.643 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.960074+0000", 2024-09-13T11:24:17.643 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.4738e-05 2024-09-13T11:24:17.644 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.644 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.644 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:24:17.644 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.961425+0000", 2024-09-13T11:24:17.644 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0013513780000000001 2024-09-13T11:24:17.644 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:17.644 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:17.644 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:17.644 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.644 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.644 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:71884 3.16 e96/88 3:6ca846b2:::benchmark_data_smithi177_31116_object71883:head v 96'3755, mlcod=96'3753)", 2024-09-13T11:24:17.644 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:54.108299+0000", 2024-09-13T11:24:17.644 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 23.507693724999999, 2024-09-13T11:24:17.645 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.20685140099999999, 2024-09-13T11:24:17.645 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:17.645 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:17.645 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:17.645 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:17.645 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.645 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:17.645 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.108299+0000", 2024-09-13T11:24:17.645 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:17.645 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.645 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.645 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:17.645 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.108299+0000", 2024-09-13T11:24:17.645 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:17.646 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.646 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.646 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:17.646 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.108300+0000", 2024-09-13T11:24:17.646 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.173e-06 2024-09-13T11:24:17.646 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.646 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.646 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:17.646 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.108323+0000", 2024-09-13T11:24:17.646 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.2591000000000001e-05 2024-09-13T11:24:17.646 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.646 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.647 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:17.647 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.312377+0000", 2024-09-13T11:24:17.647 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.20405456599999999 2024-09-13T11:24:17.647 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.647 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.647 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:17.647 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.312382+0000", 2024-09-13T11:24:17.647 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.1429999999999997e-06 2024-09-13T11:24:17.647 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.647 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.647 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:17.647 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.312480+0000", 2024-09-13T11:24:17.647 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.7742e-05 2024-09-13T11:24:17.647 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.648 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.648 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:24:17.648 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.312521+0000", 2024-09-13T11:24:17.648 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.1369000000000003e-05 2024-09-13T11:24:17.648 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.648 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.648 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:24:17.648 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.315105+0000", 2024-09-13T11:24:17.648 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0025835060000000002 2024-09-13T11:24:17.648 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.648 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.648 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:24:17.648 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.315150+0000", 2024-09-13T11:24:17.649 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.5311e-05 2024-09-13T11:24:17.649 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:17.649 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:17.649 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:17.649 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.649 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.649 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:71927 3.0 e97/75 3:0197244a:::benchmark_data_smithi177_31116_object71926:head v 97'3351, mlcod=96'3348)", 2024-09-13T11:24:17.649 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:54.200161+0000", 2024-09-13T11:24:17.649 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 23.415831496999999, 2024-09-13T11:24:17.649 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.26691738999999998, 2024-09-13T11:24:17.649 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:17.649 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:17.649 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:17.650 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:17.650 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.650 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:17.650 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.200161+0000", 2024-09-13T11:24:17.650 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:17.650 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.650 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.650 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:17.650 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.200161+0000", 2024-09-13T11:24:17.650 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:17.650 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.650 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.650 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:17.650 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.200162+0000", 2024-09-13T11:24:17.651 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.78e-07 2024-09-13T11:24:17.651 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.651 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.651 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:17.651 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.200200+0000", 2024-09-13T11:24:17.651 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.8121000000000003e-05 2024-09-13T11:24:17.651 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.651 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.651 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:17.651 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.465397+0000", 2024-09-13T11:24:17.651 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.26519681099999998 2024-09-13T11:24:17.651 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.651 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.652 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:17.652 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.465404+0000", 2024-09-13T11:24:17.652 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.7399999999999998e-06 2024-09-13T11:24:17.652 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.652 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.652 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:17.652 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.465456+0000", 2024-09-13T11:24:17.652 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.2626000000000001e-05 2024-09-13T11:24:17.652 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.652 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.652 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:24:17.652 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.465482+0000", 2024-09-13T11:24:17.652 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.6086000000000001e-05 2024-09-13T11:24:17.653 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.653 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.653 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:24:17.653 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.467035+0000", 2024-09-13T11:24:17.653 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0015523729999999999 2024-09-13T11:24:17.653 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.653 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.653 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:24:17.653 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.467078+0000", 2024-09-13T11:24:17.653 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.3754999999999997e-05 2024-09-13T11:24:17.653 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:17.653 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:17.653 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:17.653 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.654 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.654 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:72250 3.0 e97/75 3:010c7171:::benchmark_data_smithi177_31116_object72249:head v 97'3353, mlcod=96'3350)", 2024-09-13T11:24:17.654 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:54.516381+0000", 2024-09-13T11:24:17.654 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 23.099611983999999, 2024-09-13T11:24:17.654 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.13663347300000001, 2024-09-13T11:24:17.654 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:17.654 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:17.654 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:17.654 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:17.654 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.654 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:17.654 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.516381+0000", 2024-09-13T11:24:17.654 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:17.654 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.655 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.655 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:17.655 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.516381+0000", 2024-09-13T11:24:17.655 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:17.655 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.655 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.655 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:17.655 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.516382+0000", 2024-09-13T11:24:17.655 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3239999999999999e-06 2024-09-13T11:24:17.655 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.655 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.655 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:17.656 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.516437+0000", 2024-09-13T11:24:17.656 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.5408999999999998e-05 2024-09-13T11:24:17.656 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.656 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.656 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:17.656 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.651731+0000", 2024-09-13T11:24:17.656 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.135293583 2024-09-13T11:24:17.656 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.656 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.656 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:17.656 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.651736+0000", 2024-09-13T11:24:17.656 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.3410000000000002e-06 2024-09-13T11:24:17.656 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.657 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.657 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:17.657 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.651791+0000", 2024-09-13T11:24:17.657 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.4758e-05 2024-09-13T11:24:17.657 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.657 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.657 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:24:17.657 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.651814+0000", 2024-09-13T11:24:17.657 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.3283999999999999e-05 2024-09-13T11:24:17.657 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.657 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.657 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:24:17.657 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.652975+0000", 2024-09-13T11:24:17.657 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0011604370000000001 2024-09-13T11:24:17.658 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.658 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.658 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:24:17.658 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.653014+0000", 2024-09-13T11:24:17.658 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.9337e-05 2024-09-13T11:24:17.658 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:17.658 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:17.658 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:17.658 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.658 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.658 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:72993 3.0 e97/75 3:02747b8f:::benchmark_data_smithi177_31116_object72992:head v 97'3366, mlcod=97'3363)", 2024-09-13T11:24:17.658 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:55.116009+0000", 2024-09-13T11:24:17.658 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 22.499983378, 2024-09-13T11:24:17.658 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.13962660099999999, 2024-09-13T11:24:17.659 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:17.659 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:17.659 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:17.659 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:17.659 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.659 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:17.659 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:55.116009+0000", 2024-09-13T11:24:17.659 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:17.659 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.659 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.659 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:17.659 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:55.116009+0000", 2024-09-13T11:24:17.659 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:17.660 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.660 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.660 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:17.660 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:55.116010+0000", 2024-09-13T11:24:17.660 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.291e-06 2024-09-13T11:24:17.660 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.660 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.660 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:17.660 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:55.116062+0000", 2024-09-13T11:24:17.660 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.1949e-05 2024-09-13T11:24:17.660 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.660 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.660 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:17.661 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:55.254100+0000", 2024-09-13T11:24:17.661 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.13803723000000001 2024-09-13T11:24:17.661 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.661 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.661 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:17.661 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:55.254106+0000", 2024-09-13T11:24:17.661 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.5939999999999999e-06 2024-09-13T11:24:17.661 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.661 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.661 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:17.661 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:55.254196+0000", 2024-09-13T11:24:17.661 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.9567999999999994e-05 2024-09-13T11:24:17.661 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.662 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.662 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:24:17.662 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:55.254219+0000", 2024-09-13T11:24:17.662 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.3266000000000001e-05 2024-09-13T11:24:17.662 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.662 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.662 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:24:17.662 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:55.255596+0000", 2024-09-13T11:24:17.662 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0013767409999999999 2024-09-13T11:24:17.662 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.662 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.662 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:24:17.662 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:55.255636+0000", 2024-09-13T11:24:17.662 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.9962000000000001e-05 2024-09-13T11:24:17.663 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:17.663 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:17.663 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:17.663 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.663 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.663 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:73523 3.0 e98/75 3:004115ff:::benchmark_data_smithi177_31116_object73522:head v 98'3382, mlcod=98'3378)", 2024-09-13T11:24:17.663 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:55.538309+0000", 2024-09-13T11:24:17.663 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 22.07768338, 2024-09-13T11:24:17.663 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.127489824, 2024-09-13T11:24:17.663 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:17.663 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:17.663 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:17.663 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:17.664 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.664 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:17.664 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:55.538309+0000", 2024-09-13T11:24:17.664 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:17.664 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.664 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.664 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:17.664 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:55.538309+0000", 2024-09-13T11:24:17.664 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:17.664 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.664 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.664 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:17.664 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:55.538310+0000", 2024-09-13T11:24:17.665 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.237e-06 2024-09-13T11:24:17.665 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.665 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.665 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:17.665 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:55.538343+0000", 2024-09-13T11:24:17.665 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.2480000000000001e-05 2024-09-13T11:24:17.665 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.665 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.665 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:17.665 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:55.663154+0000", 2024-09-13T11:24:17.665 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.124810988 2024-09-13T11:24:17.665 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.665 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.665 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:17.666 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:55.663161+0000", 2024-09-13T11:24:17.666 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.2389999999999996e-06 2024-09-13T11:24:17.666 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.666 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.666 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:17.666 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:55.663215+0000", 2024-09-13T11:24:17.666 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.3893000000000001e-05 2024-09-13T11:24:17.666 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.666 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.666 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:24:17.666 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:55.663235+0000", 2024-09-13T11:24:17.666 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.9624e-05 2024-09-13T11:24:17.666 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.666 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.667 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:24:17.667 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:55.665764+0000", 2024-09-13T11:24:17.667 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0025292859999999999 2024-09-13T11:24:17.667 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.667 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.667 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:24:17.667 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:55.665799+0000", 2024-09-13T11:24:17.667 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.5077000000000003e-05 2024-09-13T11:24:17.667 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:17.667 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:17.667 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:17.667 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.667 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.668 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:79634 3.5 3:a6c32e6f:::benchmark_data_smithi177_31116_object79633:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101)", 2024-09-13T11:24:17.668 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:00.168616+0000", 2024-09-13T11:24:17.668 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 17.447376109, 2024-09-13T11:24:17.668 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.169445552, 2024-09-13T11:24:17.668 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:17.668 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:17.668 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:17.668 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:24:17.668 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:24:17.668 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:17.668 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 79634 2024-09-13T11:24:17.668 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.668 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:17.669 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.669 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:17.669 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.168616+0000", 2024-09-13T11:24:17.669 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:17.669 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.669 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.669 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:17.669 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.168616+0000", 2024-09-13T11:24:17.669 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:17.669 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.669 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.669 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:17.669 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.168619+0000", 2024-09-13T11:24:17.669 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.565e-06 2024-09-13T11:24:17.670 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.670 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.670 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:17.670 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.168639+0000", 2024-09-13T11:24:17.670 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.0384999999999999e-05 2024-09-13T11:24:17.670 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.670 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.670 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:17.670 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.168652+0000", 2024-09-13T11:24:17.670 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.2779000000000001e-05 2024-09-13T11:24:17.670 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.670 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.670 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:17.670 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.168658+0000", 2024-09-13T11:24:17.670 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.2269999999999998e-06 2024-09-13T11:24:17.671 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.671 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.671 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:17.671 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.168722+0000", 2024-09-13T11:24:17.671 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.3888999999999998e-05 2024-09-13T11:24:17.671 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.671 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.671 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:24:17.671 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.168863+0000", 2024-09-13T11:24:17.671 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00014050799999999999 2024-09-13T11:24:17.671 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.671 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.671 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 13", 2024-09-13T11:24:17.672 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.169010+0000", 2024-09-13T11:24:17.672 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00014746200000000001 2024-09-13T11:24:17.672 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.672 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.672 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:24:17.672 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.170891+0000", 2024-09-13T11:24:17.672 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001880945 2024-09-13T11:24:17.672 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.672 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.672 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:17.672 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.337961+0000", 2024-09-13T11:24:17.672 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.16706952 2024-09-13T11:24:17.672 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.672 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.673 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:24:17.673 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.338019+0000", 2024-09-13T11:24:17.673 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.7958000000000003e-05 2024-09-13T11:24:17.673 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.673 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.673 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:24:17.673 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.338062+0000", 2024-09-13T11:24:17.673 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.3313999999999999e-05 2024-09-13T11:24:17.673 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:17.673 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:17.673 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:17.673 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.673 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.673 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPush(3.1b 102/78 [PushOp(3:df786fc7:::benchmark_data_smithi177_31116_object48840:head, version: 71'3079, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:df786fc7:::benchmark_data_smithi177_31116_object48840:head@71'3079, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:24:17.674 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:00.290536+0000", 2024-09-13T11:24:17.674 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 17.325456734999999, 2024-09-13T11:24:17.674 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.20017585399999999, 2024-09-13T11:24:17.674 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:17.674 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:17.674 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:24:17.674 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:17.674 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.674 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:17.674 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.290536+0000", 2024-09-13T11:24:17.674 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:17.675 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.675 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.675 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:17.675 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.290536+0000", 2024-09-13T11:24:17.675 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:17.675 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.675 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.675 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:17.675 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.290537+0000", 2024-09-13T11:24:17.675 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.327e-06 2024-09-13T11:24:17.675 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.675 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.675 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:17.676 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.290581+0000", 2024-09-13T11:24:17.676 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.3893999999999997e-05 2024-09-13T11:24:17.676 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.676 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.676 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:17.676 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.488588+0000", 2024-09-13T11:24:17.676 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.198006708 2024-09-13T11:24:17.676 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.676 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.676 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:17.676 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.488607+0000", 2024-09-13T11:24:17.676 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.9162999999999999e-05 2024-09-13T11:24:17.676 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.676 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.677 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:17.677 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.488747+0000", 2024-09-13T11:24:17.677 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00014040799999999999 2024-09-13T11:24:17.677 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.677 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.677 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:24:17.677 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.488770+0000", 2024-09-13T11:24:17.677 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.279e-05 2024-09-13T11:24:17.677 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.677 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.677 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:24:17.677 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.490712+0000", 2024-09-13T11:24:17.677 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001941564 2024-09-13T11:24:17.678 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:17.678 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:17.678 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:17.678 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.678 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.678 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPush(3.1b 102/78 [PushOp(3:d82bf6ec:::benchmark_data_smithi177_31116_object48847:head, version: 71'3080, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:d82bf6ec:::benchmark_data_smithi177_31116_object48847:head@71'3080, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:24:17.678 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:00.293979+0000", 2024-09-13T11:24:17.678 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 17.322013208000001, 2024-09-13T11:24:17.678 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.20314832099999999, 2024-09-13T11:24:17.678 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:17.678 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:17.678 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:24:17.678 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:17.678 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.679 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:17.679 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.293979+0000", 2024-09-13T11:24:17.679 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:17.679 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.679 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.679 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:17.679 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.293979+0000", 2024-09-13T11:24:17.679 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:17.679 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.679 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.679 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:17.679 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.293981+0000", 2024-09-13T11:24:17.679 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.4449999999999999e-06 2024-09-13T11:24:17.679 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.680 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.680 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:17.680 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.294085+0000", 2024-09-13T11:24:17.680 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000103848 2024-09-13T11:24:17.680 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.680 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.680 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:17.680 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.495571+0000", 2024-09-13T11:24:17.680 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.20148640800000001 2024-09-13T11:24:17.680 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.680 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.680 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:17.680 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.495609+0000", 2024-09-13T11:24:17.681 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.7904000000000002e-05 2024-09-13T11:24:17.681 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.681 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.681 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:17.681 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.495729+0000", 2024-09-13T11:24:17.681 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000120556 2024-09-13T11:24:17.681 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.681 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.681 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:24:17.681 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.495751+0000", 2024-09-13T11:24:17.681 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.1511000000000001e-05 2024-09-13T11:24:17.681 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.681 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.681 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:24:17.682 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.497128+0000", 2024-09-13T11:24:17.682 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001376649 2024-09-13T11:24:17.682 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:17.682 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:17.682 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:17.682 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.682 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.682 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:79801 3.5 3:a6efdf84:::benchmark_data_smithi177_31116_object79800:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102)", 2024-09-13T11:24:17.682 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:00.403201+0000", 2024-09-13T11:24:17.682 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 17.212791621000001, 2024-09-13T11:24:17.682 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.31868626300000003, 2024-09-13T11:24:17.682 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:17.682 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:17.682 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:17.683 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:24:17.683 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:24:17.683 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:17.683 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 79801 2024-09-13T11:24:17.683 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.683 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:17.683 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.683 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:17.683 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.403201+0000", 2024-09-13T11:24:17.683 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:17.683 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.683 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.683 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:17.683 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.403201+0000", 2024-09-13T11:24:17.684 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:17.684 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.684 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.684 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:17.684 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.403204+0000", 2024-09-13T11:24:17.684 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.0819999999999999e-06 2024-09-13T11:24:17.684 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.684 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.684 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:17.684 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.403260+0000", 2024-09-13T11:24:17.684 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.5624999999999997e-05 2024-09-13T11:24:17.684 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.684 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.685 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:17.685 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.403280+0000", 2024-09-13T11:24:17.685 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.0060000000000001e-05 2024-09-13T11:24:17.685 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.685 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.685 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:17.685 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.403291+0000", 2024-09-13T11:24:17.685 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.1335000000000001e-05 2024-09-13T11:24:17.685 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.685 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.685 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:17.685 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.403344+0000", 2024-09-13T11:24:17.685 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.3217000000000002e-05 2024-09-13T11:24:17.685 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.686 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.686 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:24:17.686 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.403456+0000", 2024-09-13T11:24:17.686 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000111706 2024-09-13T11:24:17.686 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.686 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.686 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 13", 2024-09-13T11:24:17.686 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.403547+0000", 2024-09-13T11:24:17.686 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.1556000000000002e-05 2024-09-13T11:24:17.686 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.686 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.686 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:24:17.686 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.407436+0000", 2024-09-13T11:24:17.687 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.003888856 2024-09-13T11:24:17.687 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.687 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.687 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:17.687 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.721722+0000", 2024-09-13T11:24:17.687 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.31428562700000001 2024-09-13T11:24:17.687 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.687 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.687 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:24:17.687 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.721814+0000", 2024-09-13T11:24:17.687 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.1564000000000006e-05 2024-09-13T11:24:17.687 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.687 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.687 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:24:17.688 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.721887+0000", 2024-09-13T11:24:17.688 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.3634999999999995e-05 2024-09-13T11:24:17.688 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:17.688 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:17.688 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:17.688 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.688 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.688 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPush(3.1b 102/78 [PushOp(3:dcbefe42:::benchmark_data_smithi177_31116_object49001:head, version: 71'3087, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:dcbefe42:::benchmark_data_smithi177_31116_object49001:head@71'3087, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:24:17.688 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:00.511027+0000", 2024-09-13T11:24:17.688 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 17.104965685, 2024-09-13T11:24:17.688 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.13427292699999999, 2024-09-13T11:24:17.688 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:17.688 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:17.689 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:24:17.689 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:17.689 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.689 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:17.689 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.511027+0000", 2024-09-13T11:24:17.689 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:17.689 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.689 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.689 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:17.689 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.511027+0000", 2024-09-13T11:24:17.689 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:17.689 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.689 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.690 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:17.690 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.511028+0000", 2024-09-13T11:24:17.690 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.4869999999999999e-06 2024-09-13T11:24:17.690 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.690 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.690 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:17.690 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.511095+0000", 2024-09-13T11:24:17.690 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.6382000000000002e-05 2024-09-13T11:24:17.690 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.690 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.690 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:17.690 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.643319+0000", 2024-09-13T11:24:17.690 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.13222455699999999 2024-09-13T11:24:17.691 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.691 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.691 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:17.691 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.643349+0000", 2024-09-13T11:24:17.691 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.9356000000000002e-05 2024-09-13T11:24:17.691 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.691 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.691 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:17.691 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.643518+0000", 2024-09-13T11:24:17.691 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00016963500000000001 2024-09-13T11:24:17.691 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.691 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.691 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:24:17.691 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.643541+0000", 2024-09-13T11:24:17.692 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.2367000000000001e-05 2024-09-13T11:24:17.692 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.692 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.692 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:24:17.692 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.645300+0000", 2024-09-13T11:24:17.692 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0017591429999999999 2024-09-13T11:24:17.692 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:17.692 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:17.692 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:17.692 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.692 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.692 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:79813 3.16 e102/88 3:6c2df1ae:::benchmark_data_smithi177_31116_object79812:head v 102'4016, mlcod=102'4014)", 2024-09-13T11:24:17.692 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:00.516959+0000", 2024-09-13T11:24:17.693 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 17.09903366, 2024-09-13T11:24:17.693 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.15075599100000001, 2024-09-13T11:24:17.693 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:17.693 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:17.693 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:17.693 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:17.693 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.693 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:17.693 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.516959+0000", 2024-09-13T11:24:17.693 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:17.693 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.693 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.693 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:17.693 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.516959+0000", 2024-09-13T11:24:17.694 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:17.694 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.694 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.694 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:17.694 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.516961+0000", 2024-09-13T11:24:17.694 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.0279999999999999e-06 2024-09-13T11:24:17.694 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.694 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.694 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:17.694 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.517002+0000", 2024-09-13T11:24:17.694 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.0754999999999999e-05 2024-09-13T11:24:17.694 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.694 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.694 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:17.695 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.664491+0000", 2024-09-13T11:24:17.695 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.14748925099999999 2024-09-13T11:24:17.695 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.695 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.695 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:17.695 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.664504+0000", 2024-09-13T11:24:17.695 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.2747e-05 2024-09-13T11:24:17.695 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.695 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.695 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:17.695 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.664642+0000", 2024-09-13T11:24:17.695 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00013850900000000001 2024-09-13T11:24:17.695 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.696 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.696 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:24:17.696 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.664678+0000", 2024-09-13T11:24:17.696 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.5403e-05 2024-09-13T11:24:17.696 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.696 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.696 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:24:17.696 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.667646+0000", 2024-09-13T11:24:17.696 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.002968133 2024-09-13T11:24:17.696 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.696 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.696 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:24:17.696 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.667715+0000", 2024-09-13T11:24:17.696 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.9165000000000006e-05 2024-09-13T11:24:17.697 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:17.697 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:17.697 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:17.697 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.697 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.697 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:83199 3.1d e106/102 3:be2c6fae:::benchmark_data_smithi177_31116_object83198:head v 106'4244, mlcod=106'4242)", 2024-09-13T11:24:17.697 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:04.298299+0000", 2024-09-13T11:24:17.697 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 13.317693494, 2024-09-13T11:24:17.697 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.25001107099999997, 2024-09-13T11:24:17.697 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:17.697 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:17.697 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:17.697 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:17.698 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.698 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:17.698 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.298299+0000", 2024-09-13T11:24:17.698 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:17.698 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.698 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.698 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:17.698 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.298299+0000", 2024-09-13T11:24:17.698 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:17.698 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.698 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.698 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:17.698 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.298301+0000", 2024-09-13T11:24:17.698 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5859999999999999e-06 2024-09-13T11:24:17.699 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.699 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.699 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:17.699 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.298368+0000", 2024-09-13T11:24:17.699 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.6928999999999996e-05 2024-09-13T11:24:17.699 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.699 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.699 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:17.699 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.545180+0000", 2024-09-13T11:24:17.699 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.24681209700000001 2024-09-13T11:24:17.699 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.699 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.699 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:17.700 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.545193+0000", 2024-09-13T11:24:17.700 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3427e-05 2024-09-13T11:24:17.700 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.700 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.700 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:17.700 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.545327+0000", 2024-09-13T11:24:17.700 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00013359400000000001 2024-09-13T11:24:17.700 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.700 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.700 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:24:17.700 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.545385+0000", 2024-09-13T11:24:17.700 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.8492000000000002e-05 2024-09-13T11:24:17.700 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.701 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.701 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:24:17.701 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.548225+0000", 2024-09-13T11:24:17.701 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.002839472 2024-09-13T11:24:17.701 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.701 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.701 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:24:17.701 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.548310+0000", 2024-09-13T11:24:17.701 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.5474000000000002e-05 2024-09-13T11:24:17.701 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:17.701 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:17.701 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:17.701 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.701 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.702 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:84621 3.1d e107/102 3:becd72aa:::benchmark_data_smithi177_31116_object84620:head v 107'4296, mlcod=107'4293)", 2024-09-13T11:24:17.702 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:05.742238+0000", 2024-09-13T11:24:17.702 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 11.873754394000001, 2024-09-13T11:24:17.702 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.17888174300000001, 2024-09-13T11:24:17.702 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:17.702 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:17.702 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:17.702 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:17.702 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.702 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:17.702 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:05.742238+0000", 2024-09-13T11:24:17.702 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:17.702 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.703 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.703 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:17.703 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:05.742238+0000", 2024-09-13T11:24:17.703 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:17.703 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.703 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.703 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:17.703 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:05.742240+0000", 2024-09-13T11:24:17.703 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.3159999999999999e-06 2024-09-13T11:24:17.703 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.703 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.703 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:17.703 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:05.742315+0000", 2024-09-13T11:24:17.704 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.4122000000000006e-05 2024-09-13T11:24:17.704 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.704 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.704 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:17.704 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:05.917757+0000", 2024-09-13T11:24:17.704 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.175442031 2024-09-13T11:24:17.704 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.704 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.704 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:17.704 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:05.917764+0000", 2024-09-13T11:24:17.704 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.661e-06 2024-09-13T11:24:17.704 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.704 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.704 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:17.705 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:05.917846+0000", 2024-09-13T11:24:17.705 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.2006000000000001e-05 2024-09-13T11:24:17.705 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.705 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.705 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:24:17.705 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:05.917879+0000", 2024-09-13T11:24:17.705 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.2796000000000002e-05 2024-09-13T11:24:17.705 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.705 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.705 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:24:17.705 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:05.921070+0000", 2024-09-13T11:24:17.705 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0031908489999999999 2024-09-13T11:24:17.705 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.705 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.706 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:24:17.706 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:05.921120+0000", 2024-09-13T11:24:17.706 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.9962e-05 2024-09-13T11:24:17.706 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:17.706 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:17.706 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:17.706 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.706 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.706 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:100995 3.1d e114/102 3:b9d3a02e:::benchmark_data_smithi177_31116_object100994:head v 114'4771, mlcod=114'4769)", 2024-09-13T11:24:17.706 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:17.338503+0000", 2024-09-13T11:24:17.706 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 0.27748972999999999, 2024-09-13T11:24:17.706 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.12755466100000001, 2024-09-13T11:24:17.706 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:17.707 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:17.707 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:17.707 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:17.707 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.707 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:17.707 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:17.338503+0000", 2024-09-13T11:24:17.707 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:17.707 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.707 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.707 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:17.707 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:17.338503+0000", 2024-09-13T11:24:17.707 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:17.707 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.707 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.708 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:17.708 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:17.338504+0000", 2024-09-13T11:24:17.708 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.635e-06 2024-09-13T11:24:17.708 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.708 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.708 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:17.708 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:17.338575+0000", 2024-09-13T11:24:17.708 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.0869000000000002e-05 2024-09-13T11:24:17.708 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.708 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.708 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:17.708 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:17.462756+0000", 2024-09-13T11:24:17.708 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.124180943 2024-09-13T11:24:17.709 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.709 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.709 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:17.709 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:17.462764+0000", 2024-09-13T11:24:17.709 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.3170000000000004e-06 2024-09-13T11:24:17.709 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.709 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.709 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:17.709 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:17.462887+0000", 2024-09-13T11:24:17.709 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00012359200000000001 2024-09-13T11:24:17.709 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.709 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.709 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:24:17.709 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:17.462948+0000", 2024-09-13T11:24:17.710 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.0563999999999998e-05 2024-09-13T11:24:17.710 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.710 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.710 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:24:17.710 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:17.465984+0000", 2024-09-13T11:24:17.710 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0030363669999999999 2024-09-13T11:24:17.710 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:17.710 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:17.710 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:24:17.710 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:17.466057+0000", 2024-09-13T11:24:17.710 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.3374000000000006e-05 2024-09-13T11:24:17.710 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:17.710 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:17.710 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:17.710 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:17.711 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:17.711 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-09-13T11:24:17.712 INFO:teuthology.orchestra.run.smithi001.stdout:osd.12: {} 2024-09-13T11:24:17.712 INFO:teuthology.orchestra.run.smithi001.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-13T11:24:17.713 INFO:tasks.ceph.osd.8.smithi001.stderr:2024-09-13T11:24:17.707+0000 7fb9b48ae640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26312) UID: 0 2024-09-13T11:24:17.713 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-13T11:24:17.773 INFO:teuthology.orchestra.run.smithi001.stdout:osd.13: {} 2024-09-13T11:24:17.773 INFO:teuthology.orchestra.run.smithi001.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-13T11:24:17.813 INFO:teuthology.orchestra.run.smithi001.stdout:osd.14: {} 2024-09-13T11:24:17.813 INFO:teuthology.orchestra.run.smithi001.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-13T11:24:17.813 INFO:tasks.ceph.osd.15.smithi177.stderr:2024-09-13T11:24:17.811+0000 7f9d6ce77640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26578) UID: 0 2024-09-13T11:24:17.852 INFO:teuthology.orchestra.run.smithi001.stdout:osd.15: {} 2024-09-13T11:24:17.852 INFO:teuthology.orchestra.run.smithi001.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-13T11:24:17.862 INFO:teuthology.orchestra.run.smithi092.stdout:ERROR: (22) Invalid argument 2024-09-13T11:24:17.862 INFO:teuthology.orchestra.run.smithi092.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:24:17.879 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-13T11:24:17.913 INFO:tasks.ceph.osd.11.smithi177.stderr:2024-09-13T11:24:17.911+0000 7f24792c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26580) UID: 0 2024-09-13T11:24:18.008 INFO:tasks.radosbench.radosbench.0.smithi177.stdout: 111 16 101710 101694 57.2544 77.75 0.0158128 0.0174598 2024-09-13T11:24:18.013 INFO:teuthology.orchestra.run.smithi092.stdout:ERROR: (22) Invalid argument 2024-09-13T11:24:18.013 INFO:teuthology.orchestra.run.smithi092.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:24:18.014 INFO:tasks.ceph.osd.8.smithi001.stderr:2024-09-13T11:24:18.011+0000 7fb9b48ae640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26312) UID: 0 2024-09-13T11:24:18.030 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-13T11:24:18.115 INFO:tasks.ceph.osd.12.smithi001.stderr:2024-09-13T11:24:18.107+0000 7f892c6e6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26314) UID: 0 2024-09-13T11:24:18.162 INFO:teuthology.orchestra.run.smithi092.stdout:ERROR: (22) Invalid argument 2024-09-13T11:24:18.162 INFO:teuthology.orchestra.run.smithi092.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:24:18.183 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-13T11:24:18.216 INFO:tasks.ceph.osd.9.smithi092.stderr:2024-09-13T11:24:18.211+0000 7ff5ab3d0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26348) UID: 0 2024-09-13T11:24:18.317 INFO:tasks.ceph.osd.9.smithi092.stderr:2024-09-13T11:24:18.311+0000 7ff5ab3d0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26348) UID: 0 2024-09-13T11:24:18.340 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-13T11:24:18.340 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:24:18.355 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-13T11:24:18.417 INFO:tasks.ceph.osd.11.smithi177.stderr:2024-09-13T11:24:18.415+0000 7f24792c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26580) UID: 0 2024-09-13T11:24:18.485 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-13T11:24:18.486 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:24:18.502 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-13T11:24:18.548 INFO:tasks.ceph.osd.12.smithi001.stderr:2024-09-13T11:24:18.543+0000 7f892c6e6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26314) UID: 0 2024-09-13T11:24:18.619 INFO:tasks.ceph.osd.15.smithi177.stderr:2024-09-13T11:24:18.615+0000 7f9d6ce77640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26578) UID: 0 2024-09-13T11:24:18.639 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-13T11:24:18.640 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:24:18.657 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-13T11:24:18.722 INFO:tasks.ceph.osd.9.smithi092.stderr:2024-09-13T11:24:18.715+0000 7ff5ab3d0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26348) UID: 0 2024-09-13T11:24:18.821 INFO:tasks.ceph.osd.3.smithi177.stderr:2024-09-13T11:24:18.819+0000 7f10adba4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26574) UID: 0 2024-09-13T11:24:18.840 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-13T11:24:18.840 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:24:18.857 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-13T11:24:18.922 INFO:tasks.ceph.osd.2.smithi162.stderr:2024-09-13T11:24:18.920+0000 7fe92b35a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26313) UID: 0 2024-09-13T11:24:18.945 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-13T11:24:18.994 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-13T11:24:18.995 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:24:19.008 INFO:tasks.radosbench.radosbench.0.smithi177.stdout: 112 16 102939 102923 57.429 76.8125 0.0229929 0.0174062 2024-09-13T11:24:19.013 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-13T11:24:19.024 INFO:tasks.ceph.osd.1.smithi092.stderr:2024-09-13T11:24:19.019+0000 7fe7789c5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33718) UID: 0 2024-09-13T11:24:19.125 INFO:tasks.ceph.osd.8.smithi001.stderr:2024-09-13T11:24:19.119+0000 7fb9b48ae640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26312) UID: 0 2024-09-13T11:24:19.195 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-13T11:24:19.195 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:24:19.211 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-09-13T11:24:19.225 INFO:tasks.ceph.osd.1.smithi092.stderr:2024-09-13T11:24:19.223+0000 7fe7789c5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33718) UID: 0 2024-09-13T11:24:19.326 INFO:tasks.ceph.osd.12.smithi001.stderr:2024-09-13T11:24:19.319+0000 7f892c6e6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26314) UID: 0 2024-09-13T11:24:19.393 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-13T11:24:19.394 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:24:19.411 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-09-13T11:24:19.426 INFO:tasks.ceph.osd.5.smithi092.stderr:2024-09-13T11:24:19.423+0000 7f6f0b705640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26346) UID: 0 2024-09-13T11:24:19.528 INFO:tasks.ceph.osd.2.smithi162.stderr:2024-09-13T11:24:19.524+0000 7fe92b35a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26313) UID: 0 2024-09-13T11:24:19.586 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-13T11:24:19.586 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:24:19.604 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-09-13T11:24:19.629 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-09-13T11:24:19.624+0000 7f97befc9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26315) UID: 0 2024-09-13T11:24:19.729 INFO:tasks.ceph.osd.10.smithi162.stderr:2024-09-13T11:24:19.724+0000 7fb59c9f0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26317) UID: 0 2024-09-13T11:24:19.747 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-13T11:24:19.747 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:24:19.763 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_ops_in_flight 2024-09-13T11:24:19.830 INFO:tasks.ceph.osd.12.smithi001.stderr:2024-09-13T11:24:19.823+0000 7f892c6e6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26314) UID: 0 2024-09-13T11:24:19.878 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T11:24:19.903 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-13T11:24:19.904 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:24:19.921 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_blocked_ops 2024-09-13T11:24:19.926 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15] out_osds: [] dead_osds: [] live_osds: [0, 4, 8, 12, 5, 9, 13, 2, 6, 10, 14, 3, 7, 11, 15, 1] 2024-09-13T11:24:19.926 INFO:tasks.thrashosds.thrasher:choose_action: min_in 2 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-13T11:24:19.927 INFO:tasks.thrashosds.thrasher:Growing pool unique_pool_0 2024-09-13T11:24:19.927 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-13T11:24:19.964 INFO:tasks.ceph.osd.12.smithi001.stderr:2024-09-13T11:24:19.963+0000 7f892c6e6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26314) UID: 0 2024-09-13T11:24:20.009 INFO:tasks.radosbench.radosbench.0.smithi177.stdout: 113 16 104072 104056 57.5474 70.8125 0.00721787 0.0173713 2024-09-13T11:24:20.031 INFO:tasks.ceph.osd.7.smithi177.stderr:2024-09-13T11:24:20.027+0000 7f897b51d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26575) UID: 0 2024-09-13T11:24:20.040 INFO:teuthology.orchestra.run.smithi001.stderr:noscrub is unset 2024-09-13T11:24:20.050 INFO:teuthology.orchestra.run.smithi001.stdout:osd.0: {} 2024-09-13T11:24:20.051 INFO:teuthology.orchestra.run.smithi001.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T11:24:20.091 INFO:teuthology.orchestra.run.smithi001.stdout:osd.1: {} 2024-09-13T11:24:20.106 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-13T11:24:20.106 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:24:20.128 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_historic_ops 2024-09-13T11:24:20.129 INFO:teuthology.orchestra.run.smithi001.stdout:osd.2: {} 2024-09-13T11:24:20.130 INFO:teuthology.orchestra.run.smithi001.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T11:24:20.132 INFO:tasks.ceph.osd.9.smithi092.stderr:2024-09-13T11:24:20.126+0000 7ff5ab3d0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26348) UID: 0 2024-09-13T11:24:20.158 INFO:teuthology.orchestra.run.smithi001.stdout:osd.3: {} 2024-09-13T11:24:20.158 INFO:teuthology.orchestra.run.smithi001.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T11:24:20.180 INFO:teuthology.orchestra.run.smithi001.stdout:osd.4: {} 2024-09-13T11:24:20.180 INFO:teuthology.orchestra.run.smithi001.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T11:24:20.219 INFO:teuthology.orchestra.run.smithi001.stdout:osd.5: {} 2024-09-13T11:24:20.219 INFO:teuthology.orchestra.run.smithi001.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T11:24:20.233 INFO:tasks.ceph.osd.5.smithi092.stderr:2024-09-13T11:24:20.226+0000 7f6f0b705640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26346) UID: 0 2024-09-13T11:24:20.253 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-13T11:24:20.254 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:24:20.258 INFO:teuthology.orchestra.run.smithi001.stdout:osd.6: {} 2024-09-13T11:24:20.258 INFO:teuthology.orchestra.run.smithi001.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T11:24:20.268 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-13T11:24:20.282 INFO:teuthology.orchestra.run.smithi001.stdout:osd.7: {} 2024-09-13T11:24:20.282 INFO:teuthology.orchestra.run.smithi001.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T11:24:20.307 INFO:teuthology.orchestra.run.smithi001.stdout:osd.8: {} 2024-09-13T11:24:20.308 INFO:teuthology.orchestra.run.smithi001.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-13T11:24:20.330 INFO:teuthology.orchestra.run.smithi001.stdout:osd.9: {} 2024-09-13T11:24:20.330 INFO:teuthology.orchestra.run.smithi001.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-13T11:24:20.334 INFO:tasks.ceph.osd.5.smithi092.stderr:2024-09-13T11:24:20.330+0000 7f6f0b705640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26346) UID: 0 2024-09-13T11:24:20.349 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-13T11:24:20.351 INFO:teuthology.orchestra.run.smithi001.stderr:dumped all 2024-09-13T11:24:20.357 INFO:teuthology.orchestra.run.smithi001.stdout:osd.10: {} 2024-09-13T11:24:20.357 INFO:teuthology.orchestra.run.smithi001.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-13T11:24:20.369 INFO:teuthology.orchestra.run.smithi001.stdout:{"pg_ready":true,"pg_map":{"version":166,"stamp":"2024-09-13T11:24:20.145869+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":6548095539,"num_objects":99911,"num_object_clones":0,"num_object_copies":199822,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":99911,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":35877,"num_write_kb":2289394,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160549,"ondisk_log_size":160549,"up":90,"acting":90,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":112,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":16,"kb":1509949440,"kb_used":14900044,"kb_used_data":14006028,"kb_used_omap":27,"kb_used_meta":893732,"kb_avail":1495049396,"statfs":{"total":1546188226560,"available":1530930581504,"internally_reserved":0,"allocated":14342172672,"data_stored":14334248563,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28358,"internal_metadata":915181882},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,0,0,2,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":523862016,"num_objects":7993,"num_object_clones":0,"num_object_copies":15986,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7993,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-1590,"num_write_kb":-101728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-5754,"num_bytes_recovered":-377094144,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.148503"},"pg_stats":[{"pgid":"3.23","version":"113'3847","reported_seq":6589,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-09-13T11:24:15.919551+0000","last_change":"2024-09-13T11:23:48.133184+0000","last_active":"2024-09-13T11:24:15.919551+0000","last_peered":"2024-09-13T11:24:15.919551+0000","last_clean":"2024-09-13T11:24:15.919551+0000","last_became_active":"2024-09-13T11:23:47.926677+0000","last_became_peered":"2024-09-13T11:23:47.926677+0000","last_unstale":"2024-09-13T11:24:15.919551+0000","last_undegraded":"2024-09-13T11:24:15.919551+0000","last_fullsized":"2024-09-13T11:24:15.919551+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":6,"last_scrub":"91'3278","last_scrub_stamp":"2024-09-13T11:23:48.133084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:48.133084+0000","objects_scrubbed":1029,"log_size":3847,"log_dups_size":0,"ondisk_log_size":3847,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:12.874134+0000","scrub_duration":139,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":104792064,"num_objects":1599,"num_object_clones":0,"num_object_copies":3198,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1599,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":570,"num_write_kb":36480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9],"acting":[5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.22","version":"113'3808","reported_seq":5737,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-09-13T11:24:16.026387+0000","last_change":"2024-09-13T11:23:47.300446+0000","last_active":"2024-09-13T11:24:16.026387+0000","last_peered":"2024-09-13T11:24:16.026387+0000","last_clean":"2024-09-13T11:24:16.026387+0000","last_became_active":"2024-09-13T11:23:46.925318+0000","last_became_peered":"2024-09-13T11:23:46.925318+0000","last_unstale":"2024-09-13T11:24:16.026387+0000","last_undegraded":"2024-09-13T11:24:16.026387+0000","last_fullsized":"2024-09-13T11:24:16.026387+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":6,"last_scrub":"90'3246","last_scrub_stamp":"2024-09-13T11:23:47.300339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:47.300339+0000","objects_scrubbed":1033,"log_size":3808,"log_dups_size":0,"ondisk_log_size":3808,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:24:49.780485+0000","scrub_duration":203,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":104660992,"num_objects":1597,"num_object_clones":0,"num_object_copies":3194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1597,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":564,"num_write_kb":36096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,14],"acting":[0,14],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.21","version":"113'3748","reported_seq":6505,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-09-13T11:24:16.026856+0000","last_change":"2024-09-13T11:24:12.564117+0000","last_active":"2024-09-13T11:24:16.026856+0000","last_peered":"2024-09-13T11:24:16.026856+0000","last_clean":"2024-09-13T11:24:16.026856+0000","last_became_active":"2024-09-13T11:24:01.250308+0000","last_became_peered":"2024-09-13T11:24:01.250308+0000","last_unstale":"2024-09-13T11:24:16.026856+0000","last_undegraded":"2024-09-13T11:24:16.026856+0000","last_fullsized":"2024-09-13T11:24:16.026856+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":6,"last_scrub":"112'3670","last_scrub_stamp":"2024-09-13T11:24:12.564031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:24:12.564031+0000","objects_scrubbed":1431,"log_size":3748,"log_dups_size":0,"ondisk_log_size":3748,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:13.956525+0000","scrub_duration":183,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":99090432,"num_objects":1512,"num_object_clones":0,"num_object_copies":3024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1512,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":81,"num_write_kb":5184,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8],"acting":[0,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.20","version":"113'3825","reported_seq":5869,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-09-13T11:24:15.923715+0000","last_change":"2024-09-13T11:23:47.969503+0000","last_active":"2024-09-13T11:24:15.923715+0000","last_peered":"2024-09-13T11:24:15.923715+0000","last_clean":"2024-09-13T11:24:15.923715+0000","last_became_active":"2024-09-13T11:23:33.954946+0000","last_became_peered":"2024-09-13T11:23:33.954946+0000","last_unstale":"2024-09-13T11:24:15.923715+0000","last_undegraded":"2024-09-13T11:24:15.923715+0000","last_fullsized":"2024-09-13T11:24:15.923715+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":6,"last_scrub":"91'3270","last_scrub_stamp":"2024-09-13T11:23:47.969372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:47.969372+0000","objects_scrubbed":1061,"log_size":3825,"log_dups_size":0,"ondisk_log_size":3825,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:24:48.255263+0000","scrub_duration":1327,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":106233856,"num_objects":1621,"num_object_clones":0,"num_object_copies":3242,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1621,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":560,"num_write_kb":35840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,0],"acting":[12,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[]},{"pgid":"3.1f","version":"113'4766","reported_seq":9946,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-09-13T11:24:15.971350+0000","last_change":"2024-09-13T11:24:08.388852+0000","last_active":"2024-09-13T11:24:15.971350+0000","last_peered":"2024-09-13T11:24:15.971350+0000","last_clean":"2024-09-13T11:24:15.971350+0000","last_became_active":"2024-09-13T11:24:06.967832+0000","last_became_peered":"2024-09-13T11:24:06.967832+0000","last_unstale":"2024-09-13T11:24:15.971350+0000","last_undegraded":"2024-09-13T11:24:15.971350+0000","last_fullsized":"2024-09-13T11:24:15.971350+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":5,"last_scrub":"110'4427","last_scrub_stamp":"2024-09-13T11:24:08.388782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:24:08.388782+0000","objects_scrubbed":2651,"log_size":4766,"log_dups_size":0,"ondisk_log_size":4766,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:37.901131+0000","scrub_duration":315,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":196411392,"num_objects":2997,"num_object_clones":0,"num_object_copies":5994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":346,"num_write_kb":22144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,14],"acting":[10,14],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1e","version":"113'4900","reported_seq":15308,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-09-13T11:24:15.947808+0000","last_change":"2024-09-13T11:24:08.338485+0000","last_active":"2024-09-13T11:24:15.947808+0000","last_peered":"2024-09-13T11:24:15.947808+0000","last_clean":"2024-09-13T11:24:15.947808+0000","last_became_active":"2024-09-13T11:24:01.261448+0000","last_became_peered":"2024-09-13T11:24:01.261448+0000","last_unstale":"2024-09-13T11:24:15.947808+0000","last_undegraded":"2024-09-13T11:24:15.947808+0000","last_fullsized":"2024-09-13T11:24:15.947808+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":5,"last_scrub":"110'4537","last_scrub_stamp":"2024-09-13T11:24:08.338354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:24:08.338354+0000","objects_scrubbed":2760,"log_size":4900,"log_dups_size":0,"ondisk_log_size":4900,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:11.010072+0000","scrub_duration":367,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":205258752,"num_objects":3132,"num_object_clones":0,"num_object_copies":6264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3132,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":372,"num_write_kb":23808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"113'4710","reported_seq":10046,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-09-13T11:24:15.989883+0000","last_change":"2024-09-13T11:24:12.492592+0000","last_active":"2024-09-13T11:24:15.989883+0000","last_peered":"2024-09-13T11:24:15.989883+0000","last_clean":"2024-09-13T11:24:15.989883+0000","last_became_active":"2024-09-13T11:24:01.258225+0000","last_became_peered":"2024-09-13T11:24:01.258225+0000","last_unstale":"2024-09-13T11:24:15.989883+0000","last_undegraded":"2024-09-13T11:24:15.989883+0000","last_fullsized":"2024-09-13T11:24:15.989883+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":5,"last_scrub":"112'4579","last_scrub_stamp":"2024-09-13T11:24:12.492507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:24:12.492507+0000","objects_scrubbed":2875,"log_size":4710,"log_dups_size":0,"ondisk_log_size":4710,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:23.837342+0000","scrub_duration":375,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":197853184,"num_objects":3019,"num_object_clones":0,"num_object_copies":6038,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3019,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":144,"num_write_kb":9216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,15],"acting":[2,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1c","version":"114'4892","reported_seq":8588,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-09-13T11:24:17.054399+0000","last_change":"2024-09-13T11:24:00.529312+0000","last_active":"2024-09-13T11:24:17.054399+0000","last_peered":"2024-09-13T11:24:17.054399+0000","last_clean":"2024-09-13T11:24:17.054399+0000","last_became_active":"2024-09-13T11:23:33.989492+0000","last_became_peered":"2024-09-13T11:23:33.989492+0000","last_unstale":"2024-09-13T11:24:17.054399+0000","last_undegraded":"2024-09-13T11:24:17.054399+0000","last_fullsized":"2024-09-13T11:24:17.054399+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":5,"last_scrub":"102'4277","last_scrub_stamp":"2024-09-13T11:24:00.529243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:24:00.529243+0000","objects_scrubbed":2544,"log_size":4892,"log_dups_size":0,"ondisk_log_size":4892,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:15.595052+0000","scrub_duration":259,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":207159296,"num_objects":3161,"num_object_clones":0,"num_object_copies":6322,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3161,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":617,"num_write_kb":39488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7],"acting":[11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1b","version":"113'4816","reported_seq":10332,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-09-13T11:24:16.081139+0000","last_change":"2024-09-13T11:24:12.392093+0000","last_active":"2024-09-13T11:24:16.081139+0000","last_peered":"2024-09-13T11:24:16.081139+0000","last_clean":"2024-09-13T11:24:16.081139+0000","last_became_active":"2024-09-13T11:24:04.950821+0000","last_became_peered":"2024-09-13T11:24:04.950821+0000","last_unstale":"2024-09-13T11:24:16.081139+0000","last_undegraded":"2024-09-13T11:24:16.081139+0000","last_fullsized":"2024-09-13T11:24:16.081139+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":5,"last_scrub":"112'4665","last_scrub_stamp":"2024-09-13T11:24:12.391890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:24:12.391890+0000","objects_scrubbed":2934,"log_size":4816,"log_dups_size":0,"ondisk_log_size":4816,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:35.248822+0000","scrub_duration":675,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":203554816,"num_objects":3106,"num_object_clones":0,"num_object_copies":6212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3106,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":172,"num_write_kb":11008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,15],"acting":[12,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[]},{"pgid":"3.1a","version":"113'4749","reported_seq":9925,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-09-13T11:24:15.925660+0000","last_change":"2024-09-13T11:24:09.556174+0000","last_active":"2024-09-13T11:24:15.925660+0000","last_peered":"2024-09-13T11:24:15.925660+0000","last_clean":"2024-09-13T11:24:15.925660+0000","last_became_active":"2024-09-13T11:24:03.935185+0000","last_became_peered":"2024-09-13T11:24:03.935185+0000","last_unstale":"2024-09-13T11:24:15.925660+0000","last_undegraded":"2024-09-13T11:24:15.925660+0000","last_fullsized":"2024-09-13T11:24:15.925660+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":5,"last_scrub":"110'4433","last_scrub_stamp":"2024-09-13T11:24:09.556022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:24:09.556022+0000","objects_scrubbed":2654,"log_size":4749,"log_dups_size":0,"ondisk_log_size":4749,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:21.773175+0000","scrub_duration":387,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":195362816,"num_objects":2981,"num_object_clones":0,"num_object_copies":5962,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2981,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":327,"num_write_kb":20928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9],"acting":[4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.19","version":"114'4553","reported_seq":8872,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-09-13T11:24:16.972187+0000","last_change":"2024-09-13T11:23:49.429522+0000","last_active":"2024-09-13T11:24:16.972187+0000","last_peered":"2024-09-13T11:24:16.972187+0000","last_clean":"2024-09-13T11:24:16.972187+0000","last_became_active":"2024-09-13T11:23:41.924941+0000","last_became_peered":"2024-09-13T11:23:41.924941+0000","last_unstale":"2024-09-13T11:24:16.972187+0000","last_undegraded":"2024-09-13T11:24:16.972187+0000","last_fullsized":"2024-09-13T11:24:16.972187+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":5,"last_scrub":"92'3415","last_scrub_stamp":"2024-09-13T11:23:49.429429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:49.429429+0000","objects_scrubbed":1979,"log_size":4553,"log_dups_size":0,"ondisk_log_size":4553,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:04.629542+0000","scrub_duration":587,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":205127680,"num_objects":3130,"num_object_clones":0,"num_object_copies":6260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3130,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1151,"num_write_kb":73664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1],"acting":[9,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.18","version":"113'4484","reported_seq":9110,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-09-13T11:24:16.002203+0000","last_change":"2024-09-13T11:24:13.454659+0000","last_active":"2024-09-13T11:24:16.002203+0000","last_peered":"2024-09-13T11:24:16.002203+0000","last_clean":"2024-09-13T11:24:16.002203+0000","last_became_active":"2024-09-13T11:23:48.926014+0000","last_became_peered":"2024-09-13T11:23:48.926014+0000","last_unstale":"2024-09-13T11:24:16.002203+0000","last_undegraded":"2024-09-13T11:24:16.002203+0000","last_fullsized":"2024-09-13T11:24:16.002203+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":5,"last_scrub":"112'4376","last_scrub_stamp":"2024-09-13T11:24:13.454575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:24:13.454575+0000","objects_scrubbed":3036,"log_size":4484,"log_dups_size":0,"ondisk_log_size":4484,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:15.619410+0000","scrub_duration":375,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":206635008,"num_objects":3153,"num_object_clones":0,"num_object_copies":6306,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3153,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":117,"num_write_kb":7488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0],"acting":[10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.17","version":"113'4348","reported_seq":8399,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-09-13T11:24:15.961982+0000","last_change":"2024-09-13T11:23:49.349241+0000","last_active":"2024-09-13T11:24:15.961982+0000","last_peered":"2024-09-13T11:24:15.961982+0000","last_clean":"2024-09-13T11:24:15.961982+0000","last_became_active":"2024-09-13T11:23:42.935601+0000","last_became_peered":"2024-09-13T11:23:42.935601+0000","last_unstale":"2024-09-13T11:24:15.961982+0000","last_undegraded":"2024-09-13T11:24:15.961982+0000","last_fullsized":"2024-09-13T11:24:15.961982+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":5,"last_scrub":"92'3359","last_scrub_stamp":"2024-09-13T11:23:49.349167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:49.349167+0000","objects_scrubbed":2004,"log_size":4348,"log_dups_size":0,"ondisk_log_size":4348,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:24:59.401302+0000","scrub_duration":363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":196673536,"num_objects":3001,"num_object_clones":0,"num_object_copies":6002,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3001,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":997,"num_write_kb":63808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3],"acting":[10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"114'4693","reported_seq":9376,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-09-13T11:24:17.026325+0000","last_change":"2024-09-13T11:23:46.510544+0000","last_active":"2024-09-13T11:24:17.026325+0000","last_peered":"2024-09-13T11:24:17.026325+0000","last_clean":"2024-09-13T11:24:17.026325+0000","last_became_active":"2024-09-13T11:23:46.105237+0000","last_became_peered":"2024-09-13T11:23:46.105237+0000","last_unstale":"2024-09-13T11:24:17.026325+0000","last_undegraded":"2024-09-13T11:24:17.026325+0000","last_fullsized":"2024-09-13T11:24:17.026325+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":5,"last_scrub":"89'3478","last_scrub_stamp":"2024-09-13T11:23:46.510384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:46.510384+0000","objects_scrubbed":2054,"log_size":4693,"log_dups_size":0,"ondisk_log_size":4693,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:09.149878+0000","scrub_duration":363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":214433792,"num_objects":3272,"num_object_clones":0,"num_object_copies":6544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3272,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1218,"num_write_kb":77952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,15],"acting":[11,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.15","version":"113'4410","reported_seq":8851,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-09-13T11:24:15.957511+0000","last_change":"2024-09-13T11:23:48.332790+0000","last_active":"2024-09-13T11:24:15.957511+0000","last_peered":"2024-09-13T11:24:15.957511+0000","last_clean":"2024-09-13T11:24:15.957511+0000","last_became_active":"2024-09-13T11:23:42.931149+0000","last_became_peered":"2024-09-13T11:23:42.931149+0000","last_unstale":"2024-09-13T11:24:15.957511+0000","last_undegraded":"2024-09-13T11:24:15.957511+0000","last_fullsized":"2024-09-13T11:24:15.957511+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":5,"last_scrub":"91'3384","last_scrub_stamp":"2024-09-13T11:23:48.332600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:48.332600+0000","objects_scrubbed":2032,"log_size":4410,"log_dups_size":0,"ondisk_log_size":4410,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:24:58.239508+0000","scrub_duration":235,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":200474624,"num_objects":3059,"num_object_clones":0,"num_object_copies":6118,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3059,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1027,"num_write_kb":65728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,0],"acting":[14,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"3.14","version":"114'4651","reported_seq":8916,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-13T11:24:20.049621+0000","last_change":"2024-09-13T11:23:48.697100+0000","last_active":"2024-09-13T11:24:20.049621+0000","last_peered":"2024-09-13T11:24:20.049621+0000","last_clean":"2024-09-13T11:24:20.049621+0000","last_became_active":"2024-09-13T11:23:41.920989+0000","last_became_peered":"2024-09-13T11:23:41.920989+0000","last_unstale":"2024-09-13T11:24:20.049621+0000","last_undegraded":"2024-09-13T11:24:20.049621+0000","last_fullsized":"2024-09-13T11:24:20.049621+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":5,"last_scrub":"91'3371","last_scrub_stamp":"2024-09-13T11:23:48.696960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:48.696960+0000","objects_scrubbed":1935,"log_size":4651,"log_dups_size":0,"ondisk_log_size":4651,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:02.861530+0000","scrub_duration":735,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":211353600,"num_objects":3225,"num_object_clones":0,"num_object_copies":6450,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3225,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1290,"num_write_kb":82560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,14],"acting":[3,14],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":189,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-09-13T11:24:14.014542+0000","last_change":"2024-09-13T11:22:16.889130+0000","last_active":"2024-09-13T11:24:14.014542+0000","last_peered":"2024-09-13T11:24:14.014542+0000","last_clean":"2024-09-13T11:24:14.014542+0000","last_became_active":"2024-09-13T11:22:14.192897+0000","last_became_peered":"2024-09-13T11:22:14.192897+0000","last_unstale":"2024-09-13T11:24:14.014542+0000","last_undegraded":"2024-09-13T11:24:14.014542+0000","last_fullsized":"2024-09-13T11:24:14.014542+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:22:13.113563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00084655899999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,10],"acting":[15,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"3.6","version":"114'4604","reported_seq":4812,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-09-13T11:24:17.017135+0000","last_change":"2024-09-13T11:23:22.656608+0000","last_active":"2024-09-13T11:24:17.017135+0000","last_peered":"2024-09-13T11:24:17.017135+0000","last_clean":"2024-09-13T11:24:17.017135+0000","last_became_active":"2024-09-13T11:23:19.939432+0000","last_became_peered":"2024-09-13T11:23:19.939432+0000","last_unstale":"2024-09-13T11:24:17.017135+0000","last_undegraded":"2024-09-13T11:24:17.017135+0000","last_fullsized":"2024-09-13T11:24:17.017135+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'2909","last_scrub_stamp":"2024-09-13T11:23:22.656323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:22.656323+0000","objects_scrubbed":1487,"log_size":4604,"log_dups_size":0,"ondisk_log_size":4604,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:24:31.905185+0000","scrub_duration":323,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":208535552,"num_objects":3182,"num_object_clones":0,"num_object_copies":6364,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3182,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1695,"num_write_kb":108480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,14],"acting":[13,14],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":198,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-09-13T11:24:16.922584+0000","last_change":"2024-09-13T11:23:21.875855+0000","last_active":"2024-09-13T11:24:16.922584+0000","last_peered":"2024-09-13T11:24:16.922584+0000","last_clean":"2024-09-13T11:24:16.922584+0000","last_became_active":"2024-09-13T11:22:14.211923+0000","last_became_peered":"2024-09-13T11:22:14.211923+0000","last_unstale":"2024-09-13T11:24:16.922584+0000","last_undegraded":"2024-09-13T11:24:16.922584+0000","last_fullsized":"2024-09-13T11:24:16.922584+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:23:21.875683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:23:21.875683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:24:40.678136+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0011689529999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0],"acting":[9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"114'4470","reported_seq":4680,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-09-13T11:24:17.056548+0000","last_change":"2024-09-13T11:23:20.443418+0000","last_active":"2024-09-13T11:24:17.056548+0000","last_peered":"2024-09-13T11:24:17.056548+0000","last_clean":"2024-09-13T11:24:17.056548+0000","last_became_active":"2024-09-13T11:23:19.927680+0000","last_became_peered":"2024-09-13T11:23:19.927680+0000","last_unstale":"2024-09-13T11:24:17.056548+0000","last_undegraded":"2024-09-13T11:24:17.056548+0000","last_fullsized":"2024-09-13T11:24:17.056548+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'2736","last_scrub_stamp":"2024-09-13T11:23:20.443329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:20.443329+0000","objects_scrubbed":1369,"log_size":4470,"log_dups_size":0,"ondisk_log_size":4470,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:24:48.147625+0000","scrub_duration":183,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":203554816,"num_objects":3106,"num_object_clones":0,"num_object_copies":6212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3106,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1737,"num_write_kb":111168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,8],"acting":[13,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":197,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-09-13T11:24:13.998508+0000","last_change":"2024-09-13T11:23:59.530794+0000","last_active":"2024-09-13T11:24:13.998508+0000","last_peered":"2024-09-13T11:24:13.998508+0000","last_clean":"2024-09-13T11:24:13.998508+0000","last_became_active":"2024-09-13T11:22:14.581458+0000","last_became_peered":"2024-09-13T11:22:14.581458+0000","last_unstale":"2024-09-13T11:24:13.998508+0000","last_undegraded":"2024-09-13T11:24:13.998508+0000","last_fullsized":"2024-09-13T11:24:13.998508+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:23:59.530670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:23:59.530670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:02.655631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001026047,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5],"acting":[8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"113'4422","reported_seq":4610,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-09-13T11:24:15.271794+0000","last_change":"2024-09-13T11:23:20.254213+0000","last_active":"2024-09-13T11:24:15.271794+0000","last_peered":"2024-09-13T11:24:15.271794+0000","last_clean":"2024-09-13T11:24:15.271794+0000","last_became_active":"2024-09-13T11:23:19.925399+0000","last_became_peered":"2024-09-13T11:23:19.925399+0000","last_unstale":"2024-09-13T11:24:15.271794+0000","last_undegraded":"2024-09-13T11:24:15.271794+0000","last_fullsized":"2024-09-13T11:24:15.271794+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'2755","last_scrub_stamp":"2024-09-13T11:23:20.254124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:20.254124+0000","objects_scrubbed":1437,"log_size":4422,"log_dups_size":0,"ondisk_log_size":4422,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:24:49.993543+0000","scrub_duration":187,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":203751424,"num_objects":3109,"num_object_clones":0,"num_object_copies":6218,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3109,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1672,"num_write_kb":107008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,12],"acting":[1,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":185,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-09-13T11:24:14.014242+0000","last_change":"2024-09-13T11:24:08.271026+0000","last_active":"2024-09-13T11:24:14.014242+0000","last_peered":"2024-09-13T11:24:14.014242+0000","last_clean":"2024-09-13T11:24:14.014242+0000","last_became_active":"2024-09-13T11:22:47.719636+0000","last_became_peered":"2024-09-13T11:22:47.719636+0000","last_unstale":"2024-09-13T11:24:14.014242+0000","last_undegraded":"2024-09-13T11:24:14.014242+0000","last_fullsized":"2024-09-13T11:24:14.014242+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:24:08.270972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:24:08.270972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:36.805605+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00087476000000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,9],"acting":[1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"113'4439","reported_seq":4647,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-09-13T11:24:16.014103+0000","last_change":"2024-09-13T11:23:20.686446+0000","last_active":"2024-09-13T11:24:16.014103+0000","last_peered":"2024-09-13T11:24:16.014103+0000","last_clean":"2024-09-13T11:24:16.014103+0000","last_became_active":"2024-09-13T11:23:19.939695+0000","last_became_peered":"2024-09-13T11:23:19.939695+0000","last_unstale":"2024-09-13T11:24:16.014103+0000","last_undegraded":"2024-09-13T11:24:16.014103+0000","last_fullsized":"2024-09-13T11:24:16.014103+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'2739","last_scrub_stamp":"2024-09-13T11:23:20.686358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:20.686358+0000","objects_scrubbed":1386,"log_size":4439,"log_dups_size":0,"ondisk_log_size":4439,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:24:38.891737+0000","scrub_duration":271,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":202309632,"num_objects":3087,"num_object_clones":0,"num_object_copies":6174,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3087,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1701,"num_write_kb":108864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,13],"acting":[15,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"3.3","version":"113'3770","reported_seq":4001,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-09-13T11:24:15.985567+0000","last_change":"2024-09-13T11:23:32.147115+0000","last_active":"2024-09-13T11:24:15.985567+0000","last_peered":"2024-09-13T11:24:15.985567+0000","last_clean":"2024-09-13T11:24:15.985567+0000","last_became_active":"2024-09-13T11:23:31.930448+0000","last_became_peered":"2024-09-13T11:23:31.930448+0000","last_unstale":"2024-09-13T11:24:15.985567+0000","last_undegraded":"2024-09-13T11:24:15.985567+0000","last_fullsized":"2024-09-13T11:24:15.985567+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'3122","last_scrub_stamp":"2024-09-13T11:23:32.147021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:32.147021+0000","objects_scrubbed":876,"log_size":3770,"log_dups_size":0,"ondisk_log_size":3770,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:24:48.175541+0000","scrub_duration":99,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":99876864,"num_objects":1524,"num_object_clones":0,"num_object_copies":3048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1524,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":648,"num_write_kb":41472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,7],"acting":[10,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"28'2","reported_seq":127,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-09-13T11:24:14.007613+0000","last_change":"2024-09-13T11:24:00.996197+0000","last_active":"2024-09-13T11:24:14.007613+0000","last_peered":"2024-09-13T11:24:14.007613+0000","last_clean":"2024-09-13T11:24:14.007613+0000","last_became_active":"2024-09-13T11:23:12.546348+0000","last_became_peered":"2024-09-13T11:23:12.546348+0000","last_unstale":"2024-09-13T11:24:14.007613+0000","last_undegraded":"2024-09-13T11:24:14.007613+0000","last_fullsized":"2024-09-13T11:24:14.007613+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2","last_scrub_stamp":"2024-09-13T11:24:00.996028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:24:00.996028+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:12.111097+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0025826730000000002,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8],"acting":[2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"113'3792","reported_seq":4021,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-09-13T11:24:15.971739+0000","last_change":"2024-09-13T11:23:32.749651+0000","last_active":"2024-09-13T11:24:15.971739+0000","last_peered":"2024-09-13T11:24:15.971739+0000","last_clean":"2024-09-13T11:24:15.971739+0000","last_became_active":"2024-09-13T11:23:31.921388+0000","last_became_peered":"2024-09-13T11:23:31.921388+0000","last_unstale":"2024-09-13T11:24:15.971739+0000","last_undegraded":"2024-09-13T11:24:15.971739+0000","last_fullsized":"2024-09-13T11:24:15.971739+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'3106","last_scrub_stamp":"2024-09-13T11:23:32.749487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:32.749487+0000","objects_scrubbed":856,"log_size":3792,"log_dups_size":0,"ondisk_log_size":3792,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:24:34.709501+0000","scrub_duration":91,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":101056512,"num_objects":1542,"num_object_clones":0,"num_object_copies":3084,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1542,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":686,"num_write_kb":43904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,15],"acting":[12,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":199,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-09-13T11:24:14.007591+0000","last_change":"2024-09-13T11:22:47.298206+0000","last_active":"2024-09-13T11:24:14.007591+0000","last_peered":"2024-09-13T11:24:14.007591+0000","last_clean":"2024-09-13T11:24:14.007591+0000","last_became_active":"2024-09-13T11:22:47.297319+0000","last_became_peered":"2024-09-13T11:22:47.297319+0000","last_unstale":"2024-09-13T11:24:14.007591+0000","last_undegraded":"2024-09-13T11:24:14.007591+0000","last_fullsized":"2024-09-13T11:24:14.007591+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:22:13.113563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00081225399999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,1],"acting":[14,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"3.1","version":"114'3782","reported_seq":4013,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-09-13T11:24:16.998070+0000","last_change":"2024-09-13T11:23:33.604966+0000","last_active":"2024-09-13T11:24:16.998070+0000","last_peered":"2024-09-13T11:24:16.998070+0000","last_clean":"2024-09-13T11:24:16.998070+0000","last_became_active":"2024-09-13T11:23:31.926528+0000","last_became_peered":"2024-09-13T11:23:31.926528+0000","last_unstale":"2024-09-13T11:24:16.998070+0000","last_undegraded":"2024-09-13T11:24:16.998070+0000","last_fullsized":"2024-09-13T11:24:16.998070+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'3070","last_scrub_stamp":"2024-09-13T11:23:33.604792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:33.604792+0000","objects_scrubbed":842,"log_size":3782,"log_dups_size":0,"ondisk_log_size":3782,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:24:43.880038+0000","scrub_duration":119,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":101842944,"num_objects":1554,"num_object_clones":0,"num_object_copies":3108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1554,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":712,"num_write_kb":45568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,11],"acting":[13,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":202,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-13T11:24:20.049556+0000","last_change":"2024-09-13T11:24:11.895017+0000","last_active":"2024-09-13T11:24:20.049556+0000","last_peered":"2024-09-13T11:24:20.049556+0000","last_clean":"2024-09-13T11:24:20.049556+0000","last_became_active":"2024-09-13T11:22:14.211435+0000","last_became_peered":"2024-09-13T11:22:14.211435+0000","last_unstale":"2024-09-13T11:24:20.049556+0000","last_undegraded":"2024-09-13T11:24:20.049556+0000","last_fullsized":"2024-09-13T11:24:20.049556+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:24:11.894937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:24:11.894937+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:19.959531+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0010010360000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2","version":"113'3791","reported_seq":4020,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-09-13T11:24:15.825397+0000","last_change":"2024-09-13T11:23:32.977666+0000","last_active":"2024-09-13T11:24:15.825397+0000","last_peered":"2024-09-13T11:24:15.825397+0000","last_clean":"2024-09-13T11:24:15.825397+0000","last_became_active":"2024-09-13T11:23:31.919828+0000","last_became_peered":"2024-09-13T11:23:31.919828+0000","last_unstale":"2024-09-13T11:24:15.825397+0000","last_undegraded":"2024-09-13T11:24:15.825397+0000","last_fullsized":"2024-09-13T11:24:15.825397+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'3057","last_scrub_stamp":"2024-09-13T11:23:32.977559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:32.977559+0000","objects_scrubbed":841,"log_size":3791,"log_dups_size":0,"ondisk_log_size":3791,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:24:52.169339+0000","scrub_duration":151,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":103219200,"num_objects":1575,"num_object_clones":0,"num_object_copies":3150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1575,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":734,"num_write_kb":46976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,12],"acting":[14,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"2.3","version":"26'1","reported_seq":199,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-09-13T11:24:14.007826+0000","last_change":"2024-09-13T11:24:12.081967+0000","last_active":"2024-09-13T11:24:14.007826+0000","last_peered":"2024-09-13T11:24:14.007826+0000","last_clean":"2024-09-13T11:24:14.007826+0000","last_became_active":"2024-09-13T11:22:14.146507+0000","last_became_peered":"2024-09-13T11:22:14.146507+0000","last_unstale":"2024-09-13T11:24:14.007826+0000","last_undegraded":"2024-09-13T11:24:14.007826+0000","last_fullsized":"2024-09-13T11:24:14.007826+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"26'1","last_scrub_stamp":"2024-09-13T11:24:12.081872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:24:12.081872+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:19.062650+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00085610999999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,15],"acting":[14,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"1.0","version":"26'112","reported_seq":332,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-09-13T11:24:14.002225+0000","last_change":"2024-09-13T11:23:49.325249+0000","last_active":"2024-09-13T11:24:14.002225+0000","last_peered":"2024-09-13T11:24:14.002225+0000","last_clean":"2024-09-13T11:24:14.002225+0000","last_became_active":"2024-09-13T11:22:12.129721+0000","last_became_peered":"2024-09-13T11:22:12.129721+0000","last_unstale":"2024-09-13T11:24:14.002225+0000","last_undegraded":"2024-09-13T11:24:14.002225+0000","last_fullsized":"2024-09-13T11:24:14.002225+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"26'112","last_scrub_stamp":"2024-09-13T11:23:49.325175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:11.098388+0000","last_clean_scrub_stamp":"2024-09-13T11:23:49.325175+0000","objects_scrubbed":2,"log_size":112,"log_dups_size":0,"ondisk_log_size":112,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:14.422287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.8","version":"113'4439","reported_seq":4648,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-09-13T11:24:16.007584+0000","last_change":"2024-09-13T11:23:20.644952+0000","last_active":"2024-09-13T11:24:16.007584+0000","last_peered":"2024-09-13T11:24:16.007584+0000","last_clean":"2024-09-13T11:24:16.007584+0000","last_became_active":"2024-09-13T11:23:19.925802+0000","last_became_peered":"2024-09-13T11:23:19.925802+0000","last_unstale":"2024-09-13T11:24:16.007584+0000","last_undegraded":"2024-09-13T11:24:16.007584+0000","last_fullsized":"2024-09-13T11:24:16.007584+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'2771","last_scrub_stamp":"2024-09-13T11:23:20.644747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:20.644747+0000","objects_scrubbed":1356,"log_size":4439,"log_dups_size":0,"ondisk_log_size":4439,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:24:38.144523+0000","scrub_duration":411,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":198967296,"num_objects":3036,"num_object_clones":0,"num_object_copies":6072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3036,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1680,"num_write_kb":107520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.9","version":"113'4516","reported_seq":4723,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-09-13T11:24:16.095007+0000","last_change":"2024-09-13T11:23:22.033678+0000","last_active":"2024-09-13T11:24:16.095007+0000","last_peered":"2024-09-13T11:24:16.095007+0000","last_clean":"2024-09-13T11:24:16.095007+0000","last_became_active":"2024-09-13T11:23:19.935114+0000","last_became_peered":"2024-09-13T11:23:19.935114+0000","last_unstale":"2024-09-13T11:24:16.095007+0000","last_undegraded":"2024-09-13T11:24:16.095007+0000","last_fullsized":"2024-09-13T11:24:16.095007+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'2808","last_scrub_stamp":"2024-09-13T11:23:22.033572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:22.033572+0000","objects_scrubbed":1471,"log_size":4516,"log_dups_size":0,"ondisk_log_size":4516,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:24:46.595859+0000","scrub_duration":287,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":208338944,"num_objects":3179,"num_object_clones":0,"num_object_copies":6358,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3179,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1708,"num_write_kb":109312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,2],"acting":[12,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[]},{"pgid":"3.a","version":"114'4985","reported_seq":5201,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-13T11:24:20.049682+0000","last_change":"2024-09-13T11:23:32.526150+0000","last_active":"2024-09-13T11:24:20.049682+0000","last_peered":"2024-09-13T11:24:20.049682+0000","last_clean":"2024-09-13T11:24:20.049682+0000","last_became_active":"2024-09-13T11:23:31.934170+0000","last_became_peered":"2024-09-13T11:23:31.934170+0000","last_unstale":"2024-09-13T11:24:20.049682+0000","last_undegraded":"2024-09-13T11:24:20.049682+0000","last_fullsized":"2024-09-13T11:24:20.049682+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'3418","last_scrub_stamp":"2024-09-13T11:23:32.526057+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:32.526057+0000","objects_scrubbed":1774,"log_size":4985,"log_dups_size":0,"ondisk_log_size":4985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:24:34.770161+0000","scrub_duration":507,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":218955776,"num_objects":3341,"num_object_clones":0,"num_object_copies":6682,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3341,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1567,"num_write_kb":100288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.b","version":"114'4852","reported_seq":5065,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-09-13T11:24:16.987643+0000","last_change":"2024-09-13T11:23:32.681343+0000","last_active":"2024-09-13T11:24:16.987643+0000","last_peered":"2024-09-13T11:24:16.987643+0000","last_clean":"2024-09-13T11:24:16.987643+0000","last_became_active":"2024-09-13T11:23:31.923415+0000","last_became_peered":"2024-09-13T11:23:31.923415+0000","last_unstale":"2024-09-13T11:24:16.987643+0000","last_undegraded":"2024-09-13T11:24:16.987643+0000","last_fullsized":"2024-09-13T11:24:16.987643+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'3471","last_scrub_stamp":"2024-09-13T11:23:32.681257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:32.681257+0000","objects_scrubbed":1720,"log_size":4852,"log_dups_size":0,"ondisk_log_size":4852,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:24:58.857812+0000","scrub_duration":199,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":203227136,"num_objects":3101,"num_object_clones":0,"num_object_copies":6202,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3101,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1381,"num_write_kb":88384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,1],"acting":[13,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"3.c","version":"114'4872","reported_seq":5084,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-09-13T11:24:17.062079+0000","last_change":"2024-09-13T11:23:32.524870+0000","last_active":"2024-09-13T11:24:17.062079+0000","last_peered":"2024-09-13T11:24:17.062079+0000","last_clean":"2024-09-13T11:24:17.062079+0000","last_became_active":"2024-09-13T11:23:31.922620+0000","last_became_peered":"2024-09-13T11:23:31.922620+0000","last_unstale":"2024-09-13T11:24:17.062079+0000","last_undegraded":"2024-09-13T11:24:17.062079+0000","last_fullsized":"2024-09-13T11:24:17.062079+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'3463","last_scrub_stamp":"2024-09-13T11:23:32.524776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:32.524776+0000","objects_scrubbed":1741,"log_size":4872,"log_dups_size":0,"ondisk_log_size":4872,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:24:37.745747+0000","scrub_duration":235,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":206438400,"num_objects":3150,"num_object_clones":0,"num_object_copies":6300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3150,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1409,"num_write_kb":90176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,13],"acting":[11,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.d","version":"113'4782","reported_seq":4991,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-09-13T11:24:16.117284+0000","last_change":"2024-09-13T11:23:32.776835+0000","last_active":"2024-09-13T11:24:16.117284+0000","last_peered":"2024-09-13T11:24:16.117284+0000","last_clean":"2024-09-13T11:24:16.117284+0000","last_became_active":"2024-09-13T11:23:31.939246+0000","last_became_peered":"2024-09-13T11:23:31.939246+0000","last_unstale":"2024-09-13T11:24:16.117284+0000","last_undegraded":"2024-09-13T11:24:16.117284+0000","last_fullsized":"2024-09-13T11:24:16.117284+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'3395","last_scrub_stamp":"2024-09-13T11:23:32.776747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:32.776747+0000","objects_scrubbed":1704,"log_size":4782,"log_dups_size":0,"ondisk_log_size":4782,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:24:56.402862+0000","scrub_duration":179,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":202571776,"num_objects":3091,"num_object_clones":0,"num_object_copies":6182,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3091,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1387,"num_write_kb":88768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.e","version":"113'4887","reported_seq":10025,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-09-13T11:24:16.083156+0000","last_change":"2024-09-13T11:23:32.368856+0000","last_active":"2024-09-13T11:24:16.083156+0000","last_peered":"2024-09-13T11:24:16.083156+0000","last_clean":"2024-09-13T11:24:16.083156+0000","last_became_active":"2024-09-13T11:23:31.925869+0000","last_became_peered":"2024-09-13T11:23:31.925869+0000","last_unstale":"2024-09-13T11:24:16.083156+0000","last_undegraded":"2024-09-13T11:24:16.083156+0000","last_fullsized":"2024-09-13T11:24:16.083156+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'3527","last_scrub_stamp":"2024-09-13T11:23:32.368727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:32.368727+0000","objects_scrubbed":1774,"log_size":4887,"log_dups_size":0,"ondisk_log_size":4887,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:24:57.811779+0000","scrub_duration":219,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":205455360,"num_objects":3135,"num_object_clones":0,"num_object_copies":6270,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3135,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1361,"num_write_kb":87104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"114'4835","reported_seq":5046,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-09-13T11:24:17.041889+0000","last_change":"2024-09-13T11:23:33.492630+0000","last_active":"2024-09-13T11:24:17.041889+0000","last_peered":"2024-09-13T11:24:17.041889+0000","last_clean":"2024-09-13T11:24:17.041889+0000","last_became_active":"2024-09-13T11:23:31.935239+0000","last_became_peered":"2024-09-13T11:23:31.935239+0000","last_unstale":"2024-09-13T11:24:17.041889+0000","last_undegraded":"2024-09-13T11:24:17.041889+0000","last_fullsized":"2024-09-13T11:24:17.041889+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'3420","last_scrub_stamp":"2024-09-13T11:23:33.492488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:33.492488+0000","objects_scrubbed":1776,"log_size":4835,"log_dups_size":0,"ondisk_log_size":4835,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:24:44.851261+0000","scrub_duration":215,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":209125376,"num_objects":3191,"num_object_clones":0,"num_object_copies":6382,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3191,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1415,"num_write_kb":90560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,12],"acting":[11,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.10","version":"113'4581","reported_seq":8947,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-09-13T11:24:15.926860+0000","last_change":"2024-09-13T11:23:48.519654+0000","last_active":"2024-09-13T11:24:15.926860+0000","last_peered":"2024-09-13T11:24:15.926860+0000","last_clean":"2024-09-13T11:24:15.926860+0000","last_became_active":"2024-09-13T11:23:41.931183+0000","last_became_peered":"2024-09-13T11:23:41.931183+0000","last_unstale":"2024-09-13T11:24:15.926860+0000","last_undegraded":"2024-09-13T11:24:15.926860+0000","last_fullsized":"2024-09-13T11:24:15.926860+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":5,"last_scrub":"91'3468","last_scrub_stamp":"2024-09-13T11:23:48.519574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:48.519574+0000","objects_scrubbed":2031,"log_size":4581,"log_dups_size":0,"ondisk_log_size":4581,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:24:57.342213+0000","scrub_duration":215,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":206176256,"num_objects":3146,"num_object_clones":0,"num_object_copies":6292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3146,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1115,"num_write_kb":71360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.11","version":"113'4421","reported_seq":7565,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-09-13T11:24:15.910676+0000","last_change":"2024-09-13T11:23:46.102466+0000","last_active":"2024-09-13T11:24:15.910676+0000","last_peered":"2024-09-13T11:24:15.910676+0000","last_clean":"2024-09-13T11:24:15.910676+0000","last_became_active":"2024-09-13T11:23:39.063172+0000","last_became_peered":"2024-09-13T11:23:39.063172+0000","last_unstale":"2024-09-13T11:24:15.910676+0000","last_undegraded":"2024-09-13T11:24:15.910676+0000","last_fullsized":"2024-09-13T11:24:15.910676+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":5,"last_scrub":"88'3345","last_scrub_stamp":"2024-09-13T11:23:46.101973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:46.101973+0000","objects_scrubbed":2011,"log_size":4421,"log_dups_size":0,"ondisk_log_size":4421,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:08.671350+0000","scrub_duration":955,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":202571776,"num_objects":3091,"num_object_clones":0,"num_object_copies":6182,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3091,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1080,"num_write_kb":69120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11],"acting":[4,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.12","version":"113'4476","reported_seq":7544,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-09-13T11:24:16.008724+0000","last_change":"2024-09-13T11:23:35.111409+0000","last_active":"2024-09-13T11:24:16.008724+0000","last_peered":"2024-09-13T11:24:16.008724+0000","last_clean":"2024-09-13T11:24:16.008724+0000","last_became_active":"2024-09-13T11:23:21.998877+0000","last_became_peered":"2024-09-13T11:23:21.998877+0000","last_unstale":"2024-09-13T11:24:16.008724+0000","last_undegraded":"2024-09-13T11:24:16.008724+0000","last_fullsized":"2024-09-13T11:24:16.008724+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'3109","last_scrub_stamp":"2024-09-13T11:23:35.111320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:35.111320+0000","objects_scrubbed":1732,"log_size":4476,"log_dups_size":0,"ondisk_log_size":4476,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:03.371433+0000","scrub_duration":239,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":203096064,"num_objects":3099,"num_object_clones":0,"num_object_copies":6198,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3099,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1367,"num_write_kb":87488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,7],"acting":[14,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"3.13","version":"114'4518","reported_seq":7589,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-09-13T11:24:17.035577+0000","last_change":"2024-09-13T11:23:47.565094+0000","last_active":"2024-09-13T11:24:17.035577+0000","last_peered":"2024-09-13T11:24:17.035577+0000","last_clean":"2024-09-13T11:24:17.035577+0000","last_became_active":"2024-09-13T11:23:41.082433+0000","last_became_peered":"2024-09-13T11:23:41.082433+0000","last_unstale":"2024-09-13T11:24:17.035577+0000","last_undegraded":"2024-09-13T11:24:17.035577+0000","last_fullsized":"2024-09-13T11:24:17.035577+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'3408","last_scrub_stamp":"2024-09-13T11:23:47.564931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:47.564931+0000","objects_scrubbed":1981,"log_size":4518,"log_dups_size":0,"ondisk_log_size":4518,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:04.258231+0000","scrub_duration":471,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":203358208,"num_objects":3103,"num_object_clones":0,"num_object_copies":6206,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3103,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1122,"num_write_kb":71808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7],"acting":[11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":36,"stat_sum":{"num_bytes":6547505152,"num_objects":99907,"num_object_clones":0,"num_object_copies":199814,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":99907,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":35730,"num_write_kb":2286720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":14383185920,"data_stored":14383185920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160434,"ondisk_log_size":160434,"up":72,"acting":72,"num_store_stats":16},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":10},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":112,"ondisk_log_size":112,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":13,"up_from":23,"seq":98784247837,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1156288,"kb_used_data":1089008,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":93215552,"statfs":{"total":96636764160,"available":95452725248,"internally_reserved":0,"allocated":1115144192,"data_stored":1114642477,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,1,3,4,6,8,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":9,"up_from":23,"seq":98784247836,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":523632,"kb_used_data":496608,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93848208,"statfs":{"total":96636764160,"available":96100564992,"internally_reserved":0,"allocated":508526592,"data_stored":508040334,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,8,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":40,"seq":171798691859,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1412108,"kb_used_data":1339836,"kb_used_omap":4,"kb_used_meta":72251,"kb_avail":92959732,"statfs":{"total":96636764160,"available":95190765568,"internally_reserved":0,"allocated":1371992064,"data_stored":1371479253,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4433,"internal_metadata":73985711},"hb_peers":[0,2,3,6,8,9,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":21,"seq":90194313245,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1139752,"kb_used_data":1072472,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":93232088,"statfs":{"total":96636764160,"available":95469658112,"internally_reserved":0,"allocated":1098211328,"data_stored":1097683015,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":68876743},"hb_peers":[0,2,4,7,9,10,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,1,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":21,"seq":90194313247,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":328212,"kb_used_data":301188,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94043628,"statfs":{"total":96636764160,"available":96300675072,"internally_reserved":0,"allocated":308416512,"data_stored":307897672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,6,8,9,10,11,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":22,"seq":94489280539,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1303504,"kb_used_data":1227904,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":93068336,"statfs":{"total":96636764160,"available":95301976064,"internally_reserved":0,"allocated":1257373696,"data_stored":1256919824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[1,3,4,5,8,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":22,"seq":94489280540,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":487808,"kb_used_data":444144,"kb_used_omap":1,"kb_used_meta":43646,"kb_avail":93884032,"statfs":{"total":96636764160,"available":96137248768,"internally_reserved":0,"allocated":454803456,"data_stored":454308630,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":44693957},"hb_peers":[0,1,3,7,8,9,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":15,"up_from":22,"seq":94489280540,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1011208,"kb_used_data":943928,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":93360632,"statfs":{"total":96636764160,"available":95601287168,"internally_reserved":0,"allocated":966582272,"data_stored":966136374,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":68876742},"hb_peers":[0,3,4,5,6,7,10,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":22,"seq":94489280540,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":639132,"kb_used_data":585740,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":93732708,"statfs":{"total":96636764160,"available":95982292992,"internally_reserved":0,"allocated":599797760,"data_stored":599276441,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[3,5,7,9,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":22,"seq":94489280540,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":475180,"kb_used_data":435676,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":93896660,"statfs":{"total":96636764160,"available":96150179840,"internally_reserved":0,"allocated":446132224,"data_stored":445653174,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40434118},"hb_peers":[0,1,3,5,7,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":22,"seq":94489280540,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1328100,"kb_used_data":1264980,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":93043740,"statfs":{"total":96636764160,"available":95276789760,"internally_reserved":0,"allocated":1295339520,"data_stored":1294866329,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":64616906},"hb_peers":[0,3,5,6,8,9,10,11,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":22,"seq":94489280540,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":479840,"kb_used_data":444496,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93892000,"statfs":{"total":96636764160,"available":96145408000,"internally_reserved":0,"allocated":455163904,"data_stored":454634490,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,5,7,9,10,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":22,"seq":94489280539,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":748776,"kb_used_data":703704,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":93623064,"statfs":{"total":96636764160,"available":95870017536,"internally_reserved":0,"allocated":720592896,"data_stored":720117594,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":46135744},"hb_peers":[0,1,3,6,7,9,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":11,"up_from":22,"seq":94489280541,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1374976,"kb_used_data":1303536,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":92996864,"statfs":{"total":96636764160,"available":95228788736,"internally_reserved":0,"allocated":1334820864,"data_stored":1334322772,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":73136580},"hb_peers":[0,2,4,7,8,10,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":12,"up_from":22,"seq":94489280540,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1175492,"kb_used_data":1104052,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":93196348,"statfs":{"total":96636764160,"available":95433060352,"internally_reserved":0,"allocated":1130549248,"data_stored":1130047460,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73136581},"hb_peers":[0,1,2,3,5,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":14,"up_from":22,"seq":94489280540,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1316036,"kb_used_data":1248756,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":93055804,"statfs":{"total":96636764160,"available":95289143296,"internally_reserved":0,"allocated":1278726144,"data_stored":1278222724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,1,5,6,7,8,9,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,0,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1264844800,"data_stored":1264844800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1370226688,"data_stored":1370226688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":456589312,"data_stored":456589312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1096876032,"data_stored":1096876032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":603324416,"data_stored":603324416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":310050816,"data_stored":310050816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":445710336,"data_stored":445710336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1294532608,"data_stored":1294532608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":452788224,"data_stored":452788224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":508821504,"data_stored":508821504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":722665472,"data_stored":722665472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":1347944448,"data_stored":1347944448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":1133314048,"data_stored":1133314048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":1129185280,"data_stored":1129185280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":1277558784,"data_stored":1277558784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":968753152,"data_stored":968753152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T11:24:20.371 INFO:tasks.ceph.ceph_manager.ceph:increase pool size by 10 2024-09-13T11:24:20.371 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool set unique_pool_0 pg_num 46 2024-09-13T11:24:20.381 INFO:teuthology.orchestra.run.smithi001.stdout:osd.11: {} 2024-09-13T11:24:20.382 INFO:teuthology.orchestra.run.smithi001.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-13T11:24:20.405 INFO:teuthology.orchestra.run.smithi001.stdout:osd.12: {} 2024-09-13T11:24:20.405 INFO:teuthology.orchestra.run.smithi001.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-13T11:24:20.415 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-09-13T11:24:20.415 INFO:teuthology.orchestra.run.smithi092.stdout: "ops": [ 2024-09-13T11:24:20.415 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:20.415 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:104501 3.1e 3:7aaeb18b:::benchmark_data_smithi177_31116_object104500:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-09-13T11:24:20.415 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:24:20.407975+0000", 2024-09-13T11:24:20.415 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 0.0063434709999999998, 2024-09-13T11:24:20.415 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0063655309999999998, 2024-09-13T11:24:20.415 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:24:20.415 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:24:20.415 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "waiting for sub ops", 2024-09-13T11:24:20.416 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:24:20.416 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:24:20.416 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:20.416 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 104501 2024-09-13T11:24:20.416 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:20.416 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:24:20.416 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:20.416 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:24:20.416 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:20.407975+0000", 2024-09-13T11:24:20.416 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:20.416 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:20.416 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:20.416 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:24:20.417 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:20.407975+0000", 2024-09-13T11:24:20.417 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:20.417 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:20.417 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:20.417 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:24:20.417 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:20.407979+0000", 2024-09-13T11:24:20.417 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.3590000000000001e-06 2024-09-13T11:24:20.417 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:20.417 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:20.417 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:24:20.417 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:20.408126+0000", 2024-09-13T11:24:20.417 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00014695600000000001 2024-09-13T11:24:20.417 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:20.418 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:20.418 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:24:20.418 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:20.408155+0000", 2024-09-13T11:24:20.418 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.8829000000000001e-05 2024-09-13T11:24:20.418 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:20.418 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:20.418 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:24:20.418 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:20.408172+0000", 2024-09-13T11:24:20.418 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.6342999999999999e-05 2024-09-13T11:24:20.418 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:20.418 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:20.418 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:24:20.418 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:20.408252+0000", 2024-09-13T11:24:20.418 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.0016000000000001e-05 2024-09-13T11:24:20.419 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:20.419 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:20.419 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:24:20.419 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:20.408373+0000", 2024-09-13T11:24:20.419 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000121042 2024-09-13T11:24:20.419 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:20.419 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:20.419 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 4", 2024-09-13T11:24:20.419 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:20.408491+0000", 2024-09-13T11:24:20.419 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00011836 2024-09-13T11:24:20.419 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:20.419 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:20.419 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:24:20.420 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:20.410175+0000", 2024-09-13T11:24:20.420 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00168373 2024-09-13T11:24:20.420 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:20.420 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:24:20.420 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:20.420 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:20.420 INFO:teuthology.orchestra.run.smithi092.stdout: ], 2024-09-13T11:24:20.420 INFO:teuthology.orchestra.run.smithi092.stdout: "num_ops": 1 2024-09-13T11:24:20.420 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-09-13T11:24:20.431 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-13T11:24:20.432 INFO:teuthology.orchestra.run.smithi001.stdout:osd.13: {} 2024-09-13T11:24:20.432 INFO:teuthology.orchestra.run.smithi001.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-13T11:24:20.435 INFO:tasks.ceph.osd.12.smithi001.stderr:2024-09-13T11:24:20.431+0000 7f892c6e6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26314) UID: 0 2024-09-13T11:24:20.474 INFO:teuthology.orchestra.run.smithi001.stdout:osd.14: {} 2024-09-13T11:24:20.474 INFO:teuthology.orchestra.run.smithi001.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-13T11:24:20.506 INFO:teuthology.orchestra.run.smithi001.stdout:osd.15: {} 2024-09-13T11:24:20.506 INFO:teuthology.orchestra.run.smithi001.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-13T11:24:20.536 INFO:tasks.ceph.osd.3.smithi177.stderr:2024-09-13T11:24:20.535+0000 7f10adba4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26574) UID: 0 2024-09-13T11:24:20.612 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-09-13T11:24:20.612 INFO:teuthology.orchestra.run.smithi092.stdout: "ops": [], 2024-09-13T11:24:20.612 INFO:teuthology.orchestra.run.smithi092.stdout: "complaint_time": 30, 2024-09-13T11:24:20.613 INFO:teuthology.orchestra.run.smithi092.stdout: "num_blocked_ops": 0 2024-09-13T11:24:20.613 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-09-13T11:24:20.628 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-13T11:24:20.678 INFO:tasks.ceph.osd.1.smithi092.stderr:2024-09-13T11:24:20.674+0000 7fe7789c5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33718) UID: 0 2024-09-13T11:24:20.738 INFO:tasks.ceph.osd.12.smithi001.stderr:2024-09-13T11:24:20.735+0000 7f892c6e6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26314) UID: 0 2024-09-13T11:24:20.769 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-09-13T11:24:20.769 INFO:teuthology.orchestra.run.smithi092.stdout: "size": 20, 2024-09-13T11:24:20.770 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 600, 2024-09-13T11:24:20.770 INFO:teuthology.orchestra.run.smithi092.stdout: "ops": [ 2024-09-13T11:24:20.770 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:20.770 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "MOSDPGPush(3.23 83/78 [PushOp(3:c61f6bab:::benchmark_data_smithi177_31116_object29479:head, version: 59'1860, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:c61f6bab:::benchmark_data_smithi177_31116_object29479:head@59'1860, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:24:20.770 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:40.770864+0000", 2024-09-13T11:24:20.770 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 39.996471556000003, 2024-09-13T11:24:20.770 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.31229662800000002, 2024-09-13T11:24:20.770 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:24:20.770 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:24:20.770 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-09-13T11:24:20.770 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:24:20.770 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:20.771 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:24:20.771 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:40.770864+0000", 2024-09-13T11:24:20.771 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:20.771 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:20.771 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:20.771 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:24:20.771 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:40.770864+0000", 2024-09-13T11:24:20.771 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:20.771 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:20.771 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:20.771 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:24:20.771 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:40.770865+0000", 2024-09-13T11:24:20.771 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.288e-06 2024-09-13T11:24:20.772 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:20.772 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:20.772 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:24:20.772 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:40.770955+0000", 2024-09-13T11:24:20.772 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.9871999999999996e-05 2024-09-13T11:24:20.772 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:20.772 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:20.772 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:24:20.772 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:41.082105+0000", 2024-09-13T11:24:20.772 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.31114963400000001 2024-09-13T11:24:20.772 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:20.772 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:20.773 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:24:20.773 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:41.082136+0000", 2024-09-13T11:24:20.773 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.0972000000000001e-05 2024-09-13T11:24:20.773 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:20.773 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:20.773 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:24:20.773 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:41.082404+0000", 2024-09-13T11:24:20.773 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00026844800000000002 2024-09-13T11:24:20.773 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:20.773 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:20.773 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:24:20.773 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:41.082429+0000", 2024-09-13T11:24:20.773 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.5021e-05 2024-09-13T11:24:20.773 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:20.774 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:20.774 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:24:20.774 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:41.083161+0000", 2024-09-13T11:24:20.774 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000731393 2024-09-13T11:24:20.774 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:20.774 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:24:20.774 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:20.774 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:20.774 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:20.774 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "MOSDPGPush(3.1e 85/78 [PushOp(3:7942cf28:::benchmark_data_smithi177_31116_object14056:head, version: 50'925, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:7942cf28:::benchmark_data_smithi177_31116_object14056:head@50'925, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:24:20.774 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:42.656796+0000", 2024-09-13T11:24:20.774 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 38.110539711000001, 2024-09-13T11:24:20.774 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.086013249, 2024-09-13T11:24:20.775 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:24:20.775 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:24:20.775 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-09-13T11:24:20.775 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:24:20.775 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:20.775 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:24:20.775 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.656796+0000", 2024-09-13T11:24:20.775 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:20.775 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:20.775 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:20.775 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:24:20.776 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.656796+0000", 2024-09-13T11:24:20.776 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:20.776 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:20.776 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:20.776 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:24:20.776 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.656799+0000", 2024-09-13T11:24:20.776 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.693e-06 2024-09-13T11:24:20.776 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:20.776 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:20.776 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:24:20.776 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.656899+0000", 2024-09-13T11:24:20.776 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.9568999999999995e-05 2024-09-13T11:24:20.776 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:20.777 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:20.777 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:24:20.777 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.741110+0000", 2024-09-13T11:24:20.777 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.084210544999999998 2024-09-13T11:24:20.777 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:20.777 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:20.777 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:24:20.777 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.741140+0000", 2024-09-13T11:24:20.777 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.0306000000000001e-05 2024-09-13T11:24:20.777 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:20.777 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:20.777 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:24:20.777 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.741362+0000", 2024-09-13T11:24:20.777 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000221948 2024-09-13T11:24:20.778 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:20.778 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:20.778 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:24:20.778 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.741385+0000", 2024-09-13T11:24:20.778 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.3632999999999999e-05 2024-09-13T11:24:20.778 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:20.778 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:20.778 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:24:20.778 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.742809+0000", 2024-09-13T11:24:20.778 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.001423555 2024-09-13T11:24:20.778 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:20.778 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:24:20.778 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:20.779 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:20.779 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:20.779 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "MOSDPGPush(3.1e 85/78 [PushOp(3:7f40e925:::benchmark_data_smithi177_31116_object14084:head, version: 50'927, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:7f40e925:::benchmark_data_smithi177_31116_object14084:head@50'927, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:24:20.779 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:42.751000+0000", 2024-09-13T11:24:20.779 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 38.016335249000001, 2024-09-13T11:24:20.779 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.19208357500000001, 2024-09-13T11:24:20.779 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:24:20.779 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:24:20.779 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-09-13T11:24:20.779 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:24:20.779 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:20.780 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:24:20.780 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.751000+0000", 2024-09-13T11:24:20.780 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:20.780 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:20.780 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:20.780 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:24:20.780 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.751000+0000", 2024-09-13T11:24:20.780 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:20.780 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:20.780 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:20.780 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:24:20.780 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.751003+0000", 2024-09-13T11:24:20.780 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.0800000000000002e-06 2024-09-13T11:24:20.781 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:20.781 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:20.781 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:24:20.781 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.751069+0000", 2024-09-13T11:24:20.781 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.5417000000000001e-05 2024-09-13T11:24:20.781 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:20.781 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:20.781 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:24:20.781 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.941261+0000", 2024-09-13T11:24:20.781 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.19019233899999999 2024-09-13T11:24:20.781 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:20.781 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:20.781 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:24:20.782 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.941300+0000", 2024-09-13T11:24:20.782 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.9091999999999998e-05 2024-09-13T11:24:20.782 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:20.782 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:20.782 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:24:20.782 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.941519+0000", 2024-09-13T11:24:20.782 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000218521 2024-09-13T11:24:20.782 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:20.782 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:20.782 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:24:20.782 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.941544+0000", 2024-09-13T11:24:20.782 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.5412999999999999e-05 2024-09-13T11:24:20.782 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:20.783 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:20.783 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:24:20.783 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.943084+0000", 2024-09-13T11:24:20.783 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0015397130000000001 2024-09-13T11:24:20.783 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:20.783 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:24:20.783 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:20.783 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:20.783 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:20.783 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "MOSDPGPush(3.1e 86/78 [PushOp(3:7aada7c3:::benchmark_data_smithi177_31116_object14640:head, version: 51'967, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:7aada7c3:::benchmark_data_smithi177_31116_object14640:head@51'967, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:24:20.783 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:43.291230+0000", 2024-09-13T11:24:20.783 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 37.476105654000001, 2024-09-13T11:24:20.783 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.058273907999999999, 2024-09-13T11:24:20.784 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:24:20.784 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:24:20.784 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-09-13T11:24:20.784 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:24:20.784 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:20.784 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:24:20.784 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:43.291230+0000", 2024-09-13T11:24:20.784 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:20.784 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:20.784 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:20.784 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:24:20.784 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:43.291230+0000", 2024-09-13T11:24:20.784 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:20.785 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:20.785 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:20.785 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:24:20.785 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:43.291231+0000", 2024-09-13T11:24:20.785 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.5969999999999999e-06 2024-09-13T11:24:20.785 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:20.785 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:20.785 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:24:20.785 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:43.291296+0000", 2024-09-13T11:24:20.785 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.4956999999999995e-05 2024-09-13T11:24:20.785 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:20.785 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:20.785 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:24:20.786 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:43.348751+0000", 2024-09-13T11:24:20.786 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.057454572000000002 2024-09-13T11:24:20.786 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:20.786 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:20.786 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:24:20.786 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:43.348763+0000", 2024-09-13T11:24:20.786 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.2296000000000001e-05 2024-09-13T11:24:20.786 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:20.786 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:20.786 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:24:20.786 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:43.348833+0000", 2024-09-13T11:24:20.786 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.0197000000000006e-05 2024-09-13T11:24:20.786 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:20.787 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:20.787 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:24:20.787 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:43.348843+0000", 2024-09-13T11:24:20.787 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.1640000000000007e-06 2024-09-13T11:24:20.787 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:20.787 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:20.787 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:24:20.787 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:43.349504+0000", 2024-09-13T11:24:20.787 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00066112500000000004 2024-09-13T11:24:20.787 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:20.787 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:24:20.787 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:20.787 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:20.788 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:20.788 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "MOSDPGPush(3.1e 90/78 [PushOp(3:7f342246:::benchmark_data_smithi177_31116_object22523:head, version: 55'1491, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:7f342246:::benchmark_data_smithi177_31116_object22523:head@55'1491, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:24:20.788 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:47.356265+0000", 2024-09-13T11:24:20.788 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 33.411069972999996, 2024-09-13T11:24:20.788 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.058328110000000002, 2024-09-13T11:24:20.788 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:24:20.788 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:24:20.788 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-09-13T11:24:20.788 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:24:20.788 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:20.788 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:24:20.788 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:47.356265+0000", 2024-09-13T11:24:20.789 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:20.789 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:20.789 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:20.789 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:24:20.789 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:47.356265+0000", 2024-09-13T11:24:20.789 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:20.789 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:20.789 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:20.789 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:24:20.789 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:47.356268+0000", 2024-09-13T11:24:20.789 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.4439999999999998e-06 2024-09-13T11:24:20.790 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:20.790 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:20.790 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:24:20.790 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:47.356335+0000", 2024-09-13T11:24:20.790 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.7006000000000002e-05 2024-09-13T11:24:20.790 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:20.790 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:20.790 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:24:20.790 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:47.412193+0000", 2024-09-13T11:24:20.790 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.055858374000000002 2024-09-13T11:24:20.790 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:20.790 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:20.791 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:24:20.791 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:47.412231+0000", 2024-09-13T11:24:20.791 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.7267000000000001e-05 2024-09-13T11:24:20.791 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:20.791 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:20.791 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:24:20.791 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:47.412451+0000", 2024-09-13T11:24:20.791 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000220836 2024-09-13T11:24:20.791 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:20.791 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:20.791 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:24:20.791 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:47.412483+0000", 2024-09-13T11:24:20.791 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.1751000000000001e-05 2024-09-13T11:24:20.791 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:20.792 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:20.792 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:24:20.792 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:47.414594+0000", 2024-09-13T11:24:20.792 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0021104320000000002 2024-09-13T11:24:20.792 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:20.792 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:24:20.792 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:20.792 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:20.792 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:20.792 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "MOSDPGPush(3.1e 90/78 [PushOp(3:782a0814:::benchmark_data_smithi177_31116_object22606:head, version: 55'1494, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:782a0814:::benchmark_data_smithi177_31116_object22606:head@55'1494, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:24:20.792 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:47.480379+0000", 2024-09-13T11:24:20.792 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 33.286956468, 2024-09-13T11:24:20.792 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.061871597, 2024-09-13T11:24:20.793 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:24:20.793 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:24:20.793 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-09-13T11:24:20.793 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:24:20.793 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:20.793 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:24:20.793 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:47.480379+0000", 2024-09-13T11:24:20.793 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:20.793 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:20.793 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:20.793 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:24:20.793 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:47.480379+0000", 2024-09-13T11:24:20.793 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:20.793 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:20.794 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:20.794 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:24:20.794 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:47.480381+0000", 2024-09-13T11:24:20.794 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.1430000000000001e-06 2024-09-13T11:24:20.794 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:20.794 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:20.794 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:24:20.794 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:47.480431+0000", 2024-09-13T11:24:20.794 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.9812999999999999e-05 2024-09-13T11:24:20.794 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:20.794 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:20.794 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:24:20.795 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:47.541318+0000", 2024-09-13T11:24:20.795 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.060886719999999998 2024-09-13T11:24:20.795 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:20.795 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:20.795 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:24:20.795 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:47.541334+0000", 2024-09-13T11:24:20.795 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.6001000000000001e-05 2024-09-13T11:24:20.795 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:20.795 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:20.795 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:24:20.795 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:47.541439+0000", 2024-09-13T11:24:20.795 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000105816 2024-09-13T11:24:20.795 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:20.796 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:20.796 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:24:20.796 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:47.541451+0000", 2024-09-13T11:24:20.796 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.202e-05 2024-09-13T11:24:20.796 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:20.796 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:20.796 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:24:20.796 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:47.542251+0000", 2024-09-13T11:24:20.796 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00079908400000000004 2024-09-13T11:24:20.796 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:20.796 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:24:20.796 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:20.796 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:20.796 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:20.797 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "MOSDPGPush(3.1e 90/78 [PushOp(3:7c65c13b:::benchmark_data_smithi177_31116_object22761:head, version: 55'1503, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:7c65c13b:::benchmark_data_smithi177_31116_object22761:head@55'1503, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:24:20.797 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:47.705735+0000", 2024-09-13T11:24:20.797 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 33.061600144000003, 2024-09-13T11:24:20.797 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.107567331, 2024-09-13T11:24:20.797 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:24:20.797 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:24:20.797 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-09-13T11:24:20.797 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:24:20.797 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:20.797 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:24:20.797 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:47.705735+0000", 2024-09-13T11:24:20.797 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:20.798 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:20.798 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:20.798 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:24:20.798 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:47.705735+0000", 2024-09-13T11:24:20.798 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:20.798 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:20.798 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:20.798 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:24:20.798 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:47.705738+0000", 2024-09-13T11:24:20.798 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.5940000000000001e-06 2024-09-13T11:24:20.798 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:20.798 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:20.798 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:24:20.798 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:47.705798+0000", 2024-09-13T11:24:20.799 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.9848e-05 2024-09-13T11:24:20.799 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:20.799 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:20.799 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:24:20.799 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:47.811473+0000", 2024-09-13T11:24:20.799 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.10567505100000001 2024-09-13T11:24:20.799 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:20.799 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:20.799 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:24:20.799 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:47.811488+0000", 2024-09-13T11:24:20.799 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.5311999999999998e-05 2024-09-13T11:24:20.799 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:20.800 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:20.800 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:24:20.800 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:47.811684+0000", 2024-09-13T11:24:20.800 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00019606100000000001 2024-09-13T11:24:20.800 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:20.800 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:20.800 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:24:20.800 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:47.811709+0000", 2024-09-13T11:24:20.800 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.5066e-05 2024-09-13T11:24:20.800 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:20.800 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:20.800 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:24:20.800 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:47.813303+0000", 2024-09-13T11:24:20.801 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0015933989999999999 2024-09-13T11:24:20.801 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:20.801 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:24:20.801 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:20.801 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:20.801 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:20.801 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "MOSDPGPush(3.1e 90/78 [PushOp(3:7d0d3b0f:::benchmark_data_smithi177_31116_object22844:head, version: 55'1508, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:7d0d3b0f:::benchmark_data_smithi177_31116_object22844:head@55'1508, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:24:20.801 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:47.847160+0000", 2024-09-13T11:24:20.801 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 32.920175499000003, 2024-09-13T11:24:20.801 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.077491803999999997, 2024-09-13T11:24:20.801 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:24:20.802 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:24:20.802 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-09-13T11:24:20.802 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:24:20.802 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:20.802 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:24:20.802 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:47.847160+0000", 2024-09-13T11:24:20.802 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:20.802 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:20.802 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:20.802 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:24:20.802 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:47.847160+0000", 2024-09-13T11:24:20.802 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:20.802 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:20.803 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:20.803 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:24:20.803 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:47.847162+0000", 2024-09-13T11:24:20.803 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.7099999999999999e-06 2024-09-13T11:24:20.803 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:20.803 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:20.803 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:24:20.803 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:47.847233+0000", 2024-09-13T11:24:20.803 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.1824999999999998e-05 2024-09-13T11:24:20.803 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:20.803 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:20.803 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:24:20.803 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:47.919816+0000", 2024-09-13T11:24:20.803 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.072582926000000006 2024-09-13T11:24:20.804 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:20.804 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:20.804 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:24:20.804 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:47.919905+0000", 2024-09-13T11:24:20.804 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.8116999999999997e-05 2024-09-13T11:24:20.804 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:20.804 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:20.804 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:24:20.804 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:47.921939+0000", 2024-09-13T11:24:20.804 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0020349140000000001 2024-09-13T11:24:20.804 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:20.804 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:20.804 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:24:20.804 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:47.922021+0000", 2024-09-13T11:24:20.805 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.1694999999999996e-05 2024-09-13T11:24:20.805 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:20.805 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:20.805 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:24:20.805 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:47.924652+0000", 2024-09-13T11:24:20.805 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.002630617 2024-09-13T11:24:20.805 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:20.805 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:24:20.805 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:20.805 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:20.805 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:20.805 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "MOSDPGPush(3.1e 94/78 [PushOp(3:7e3b05ed:::benchmark_data_smithi177_31116_object30052:head, version: 59'1945, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:7e3b05ed:::benchmark_data_smithi177_31116_object30052:head@59'1945, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:24:20.805 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:52.086403+0000", 2024-09-13T11:24:20.805 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 28.680932633000001, 2024-09-13T11:24:20.806 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.088501402000000007, 2024-09-13T11:24:20.806 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:24:20.806 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:24:20.806 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-09-13T11:24:20.806 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:24:20.806 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:20.806 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:24:20.806 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:52.086403+0000", 2024-09-13T11:24:20.806 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:20.806 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:20.806 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:20.806 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:24:20.806 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:52.086403+0000", 2024-09-13T11:24:20.807 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:20.807 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:20.807 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:20.807 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:24:20.807 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:52.086405+0000", 2024-09-13T11:24:20.807 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.5620000000000002e-06 2024-09-13T11:24:20.807 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:20.807 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:20.807 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:24:20.807 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:52.086472+0000", 2024-09-13T11:24:20.807 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.7053999999999999e-05 2024-09-13T11:24:20.807 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:20.807 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:20.807 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:24:20.808 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:52.172902+0000", 2024-09-13T11:24:20.808 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.086430050999999994 2024-09-13T11:24:20.808 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:20.808 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:20.808 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:24:20.808 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:52.172928+0000", 2024-09-13T11:24:20.808 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.5106999999999999e-05 2024-09-13T11:24:20.808 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:20.808 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:20.808 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:24:20.808 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:52.173094+0000", 2024-09-13T11:24:20.808 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00016682200000000001 2024-09-13T11:24:20.808 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:20.808 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:20.809 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:24:20.809 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:52.173117+0000", 2024-09-13T11:24:20.809 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.2459e-05 2024-09-13T11:24:20.809 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:20.809 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:20.809 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:24:20.809 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:52.174904+0000", 2024-09-13T11:24:20.809 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0017873469999999999 2024-09-13T11:24:20.809 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:20.809 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:24:20.809 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:20.809 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:20.810 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:20.810 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:82769 3.1e 3:7a4e88b5:::benchmark_data_smithi177_31116_object82768:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105)", 2024-09-13T11:24:20.810 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:24:03.744632+0000", 2024-09-13T11:24:20.810 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 17.022703104000001, 2024-09-13T11:24:20.810 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.21475722699999999, 2024-09-13T11:24:20.810 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:24:20.810 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:24:20.810 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:20.810 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:24:20.810 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:24:20.810 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:20.810 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 82769 2024-09-13T11:24:20.811 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:20.811 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:24:20.811 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:20.811 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:24:20.811 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:03.744632+0000", 2024-09-13T11:24:20.811 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:20.811 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:20.811 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:20.811 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:24:20.811 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:03.744632+0000", 2024-09-13T11:24:20.812 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:20.812 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:20.812 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:20.812 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:24:20.812 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:03.744634+0000", 2024-09-13T11:24:20.812 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.672e-06 2024-09-13T11:24:20.812 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:20.812 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:20.812 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:24:20.812 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:03.744724+0000", 2024-09-13T11:24:20.812 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.0237999999999999e-05 2024-09-13T11:24:20.812 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:20.813 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:20.813 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:24:20.813 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:03.744736+0000", 2024-09-13T11:24:20.813 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.2038999999999999e-05 2024-09-13T11:24:20.813 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:20.813 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:20.813 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:24:20.813 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:03.744742+0000", 2024-09-13T11:24:20.813 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.2129999999999996e-06 2024-09-13T11:24:20.813 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:20.813 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:20.813 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:24:20.814 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:03.744785+0000", 2024-09-13T11:24:20.814 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.2898000000000002e-05 2024-09-13T11:24:20.814 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:20.814 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:20.814 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:24:20.814 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:03.744894+0000", 2024-09-13T11:24:20.814 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000109073 2024-09-13T11:24:20.814 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:20.814 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:20.814 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 4", 2024-09-13T11:24:20.814 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:03.745009+0000", 2024-09-13T11:24:20.814 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00011428499999999999 2024-09-13T11:24:20.815 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:20.815 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:20.815 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:24:20.815 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:03.746451+0000", 2024-09-13T11:24:20.815 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0014419179999999999 2024-09-13T11:24:20.815 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:20.815 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:20.815 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:20.815 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:03.959257+0000", 2024-09-13T11:24:20.815 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.212806361 2024-09-13T11:24:20.815 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:20.815 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:20.816 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:24:20.816 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:03.959333+0000", 2024-09-13T11:24:20.816 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.6001000000000006e-05 2024-09-13T11:24:20.816 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:20.816 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:20.816 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:24:20.816 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:03.959390+0000", 2024-09-13T11:24:20.816 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.6529e-05 2024-09-13T11:24:20.816 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:20.816 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:24:20.816 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:20.816 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:20.816 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:20.817 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:82786 3.1e 3:79037e3f:::benchmark_data_smithi177_31116_object82785:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106)", 2024-09-13T11:24:20.817 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:24:03.951546+0000", 2024-09-13T11:24:20.817 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 16.815788940000001, 2024-09-13T11:24:20.817 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.063925151, 2024-09-13T11:24:20.817 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:24:20.817 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:24:20.817 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:20.817 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:24:20.817 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:24:20.817 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:20.817 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 82786 2024-09-13T11:24:20.817 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:20.817 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:24:20.818 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:20.818 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:24:20.818 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:03.951546+0000", 2024-09-13T11:24:20.818 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:20.818 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:20.818 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:20.818 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:24:20.818 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:03.951546+0000", 2024-09-13T11:24:20.818 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:20.818 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:20.818 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:20.818 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:24:20.818 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:03.951552+0000", 2024-09-13T11:24:20.818 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.4199999999999998e-06 2024-09-13T11:24:20.819 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:20.819 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:20.819 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:24:20.819 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:03.951639+0000", 2024-09-13T11:24:20.819 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.6934999999999994e-05 2024-09-13T11:24:20.819 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:20.819 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:20.819 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:24:20.819 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:03.951695+0000", 2024-09-13T11:24:20.819 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.5915999999999998e-05 2024-09-13T11:24:20.819 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:20.819 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:20.819 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:24:20.819 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:03.951719+0000", 2024-09-13T11:24:20.820 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.4295e-05 2024-09-13T11:24:20.820 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:20.820 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:20.820 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:24:20.820 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:03.951832+0000", 2024-09-13T11:24:20.820 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000112495 2024-09-13T11:24:20.820 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:20.820 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:20.820 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op must wait for map", 2024-09-13T11:24:20.820 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:03.951838+0000", 2024-09-13T11:24:20.820 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.9009999999999997e-06 2024-09-13T11:24:20.820 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:20.820 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:20.821 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:24:20.821 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:03.954386+0000", 2024-09-13T11:24:20.821 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0025478110000000001 2024-09-13T11:24:20.821 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:20.821 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:20.821 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:24:20.821 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:03.954512+0000", 2024-09-13T11:24:20.821 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000126217 2024-09-13T11:24:20.821 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:20.821 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:20.821 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 4", 2024-09-13T11:24:20.821 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:03.954630+0000", 2024-09-13T11:24:20.821 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00011701699999999999 2024-09-13T11:24:20.821 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:20.822 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:20.822 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:24:20.822 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:03.956830+0000", 2024-09-13T11:24:20.822 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.002200572 2024-09-13T11:24:20.822 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:20.822 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:20.822 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:20.822 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:04.015039+0000", 2024-09-13T11:24:20.822 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.058209126 2024-09-13T11:24:20.822 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:20.822 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:20.822 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:24:20.823 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:04.015327+0000", 2024-09-13T11:24:20.823 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00028825299999999998 2024-09-13T11:24:20.823 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:20.823 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:20.823 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:24:20.823 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:04.015472+0000", 2024-09-13T11:24:20.823 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00014419300000000001 2024-09-13T11:24:20.823 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:20.823 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:24:20.823 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:20.823 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:20.824 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:20.824 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:82869 3.1e 3:7b71bee4:::benchmark_data_smithi177_31116_object82868:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106)", 2024-09-13T11:24:20.824 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:24:04.019636+0000", 2024-09-13T11:24:20.824 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 16.747699871000002, 2024-09-13T11:24:20.824 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.056222291000000001, 2024-09-13T11:24:20.824 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:24:20.824 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:24:20.824 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:20.824 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:24:20.824 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:24:20.824 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:20.825 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 82869 2024-09-13T11:24:20.825 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:20.825 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:24:20.825 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:20.825 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:24:20.825 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:04.019636+0000", 2024-09-13T11:24:20.825 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:20.825 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:20.825 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:20.825 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:24:20.825 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:04.019636+0000", 2024-09-13T11:24:20.825 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:20.825 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:20.825 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:20.826 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:24:20.826 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:04.019640+0000", 2024-09-13T11:24:20.826 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.656e-06 2024-09-13T11:24:20.826 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:20.826 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:20.826 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:24:20.826 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:04.019707+0000", 2024-09-13T11:24:20.826 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.6322000000000006e-05 2024-09-13T11:24:20.826 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:20.826 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:20.826 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:24:20.826 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:04.019736+0000", 2024-09-13T11:24:20.826 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.9499999999999999e-05 2024-09-13T11:24:20.827 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:20.827 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:20.827 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:24:20.827 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:04.019751+0000", 2024-09-13T11:24:20.827 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.4943000000000001e-05 2024-09-13T11:24:20.827 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:20.827 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:20.827 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:24:20.827 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:04.019870+0000", 2024-09-13T11:24:20.827 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0001186 2024-09-13T11:24:20.827 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:20.827 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:20.827 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:24:20.827 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:04.020132+0000", 2024-09-13T11:24:20.828 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00026221699999999997 2024-09-13T11:24:20.828 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:20.828 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:20.828 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 4", 2024-09-13T11:24:20.828 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:04.020518+0000", 2024-09-13T11:24:20.828 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00038653400000000001 2024-09-13T11:24:20.828 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:20.828 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:20.828 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:24:20.828 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:04.024361+0000", 2024-09-13T11:24:20.828 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0038429739999999999 2024-09-13T11:24:20.828 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:20.829 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:20.829 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:20.829 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:04.075715+0000", 2024-09-13T11:24:20.829 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.051353337999999998 2024-09-13T11:24:20.829 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:20.829 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:20.829 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:24:20.829 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:04.075779+0000", 2024-09-13T11:24:20.829 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.4059000000000005e-05 2024-09-13T11:24:20.829 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:20.829 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:20.829 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:24:20.829 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:04.075858+0000", 2024-09-13T11:24:20.829 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.9147999999999995e-05 2024-09-13T11:24:20.829 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:20.830 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:24:20.830 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:20.830 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:20.830 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:20.830 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:84705 3.23 3:c7ff5ed5:::benchmark_data_smithi177_31116_object84704:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-09-13T11:24:20.830 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:24:05.809034+0000", 2024-09-13T11:24:20.830 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 14.958301348999999, 2024-09-13T11:24:20.830 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.075598808000000003, 2024-09-13T11:24:20.830 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:24:20.830 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:24:20.830 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:20.830 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:24:20.830 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:24:20.831 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:20.831 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 84705 2024-09-13T11:24:20.831 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:20.831 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:24:20.831 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:20.831 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:24:20.831 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:05.809034+0000", 2024-09-13T11:24:20.831 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:20.831 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:20.831 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:20.831 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:24:20.831 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:05.809034+0000", 2024-09-13T11:24:20.831 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:20.831 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:20.832 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:20.832 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:24:20.832 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:05.809042+0000", 2024-09-13T11:24:20.832 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.8690000000000004e-06 2024-09-13T11:24:20.832 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:20.832 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:20.832 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:24:20.832 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:05.809095+0000", 2024-09-13T11:24:20.832 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.3052e-05 2024-09-13T11:24:20.832 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:20.832 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:20.832 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:24:20.832 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:05.809123+0000", 2024-09-13T11:24:20.832 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.8096999999999999e-05 2024-09-13T11:24:20.833 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:20.833 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:20.833 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:24:20.833 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:05.809137+0000", 2024-09-13T11:24:20.833 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.4195999999999999e-05 2024-09-13T11:24:20.833 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:20.833 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:20.833 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:24:20.833 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:05.809234+0000", 2024-09-13T11:24:20.833 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.6309000000000002e-05 2024-09-13T11:24:20.833 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:20.833 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:20.833 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:24:20.833 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:05.809384+0000", 2024-09-13T11:24:20.834 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000150209 2024-09-13T11:24:20.834 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:20.834 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:20.834 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 9", 2024-09-13T11:24:20.834 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:05.809543+0000", 2024-09-13T11:24:20.834 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00015886599999999999 2024-09-13T11:24:20.834 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:20.834 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:20.834 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:24:20.834 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:05.811303+0000", 2024-09-13T11:24:20.834 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0017600210000000001 2024-09-13T11:24:20.834 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:20.834 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:20.834 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:20.835 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:05.884336+0000", 2024-09-13T11:24:20.835 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.073033106 2024-09-13T11:24:20.835 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:20.835 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:20.835 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:24:20.835 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:05.884506+0000", 2024-09-13T11:24:20.835 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00016986599999999999 2024-09-13T11:24:20.835 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:20.835 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:20.835 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:24:20.835 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:05.884633+0000", 2024-09-13T11:24:20.835 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000127217 2024-09-13T11:24:20.835 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:20.836 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:24:20.836 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:20.836 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:20.836 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:20.836 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop_reply(client.4665.0:84705 3.23 e107/90 ondisk, result = 0)", 2024-09-13T11:24:20.836 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:24:05.812095+0000", 2024-09-13T11:24:20.836 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 14.955240289000001, 2024-09-13T11:24:20.836 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.072554935000000001, 2024-09-13T11:24:20.836 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:24:20.836 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:24:20.836 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-09-13T11:24:20.836 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:24:20.836 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:20.836 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:24:20.837 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:05.812095+0000", 2024-09-13T11:24:20.837 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:20.837 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:20.837 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:20.837 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:24:20.837 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:05.812095+0000", 2024-09-13T11:24:20.837 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:20.837 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:20.837 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:20.837 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:24:20.837 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:05.812098+0000", 2024-09-13T11:24:20.837 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.6819999999999999e-06 2024-09-13T11:24:20.837 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:20.838 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:20.838 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:24:20.838 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:05.812106+0000", 2024-09-13T11:24:20.838 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.1839999999999999e-06 2024-09-13T11:24:20.838 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:20.838 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:20.838 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:24:20.838 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:05.884155+0000", 2024-09-13T11:24:20.838 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.072049118999999995 2024-09-13T11:24:20.838 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:20.838 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:20.838 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:24:20.839 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:05.884168+0000", 2024-09-13T11:24:20.839 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.2517000000000001e-05 2024-09-13T11:24:20.839 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:20.839 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:20.839 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:24:20.839 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:05.884294+0000", 2024-09-13T11:24:20.839 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00012600400000000001 2024-09-13T11:24:20.839 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:20.839 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:20.839 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:24:20.839 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:05.884316+0000", 2024-09-13T11:24:20.839 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.2702999999999999e-05 2024-09-13T11:24:20.840 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:20.840 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:20.840 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:24:20.840 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:05.884650+0000", 2024-09-13T11:24:20.840 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00033372600000000001 2024-09-13T11:24:20.840 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:20.840 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:24:20.840 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:20.840 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:20.840 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:20.840 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:90679 3.1e 3:7fd1e7e8:::benchmark_data_smithi177_31116_object90678:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110)", 2024-09-13T11:24:20.840 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:24:10.371726+0000", 2024-09-13T11:24:20.841 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 10.395609306000001, 2024-09-13T11:24:20.841 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.062121543000000001, 2024-09-13T11:24:20.841 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:24:20.841 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:24:20.841 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:20.841 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:24:20.841 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:24:20.841 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:20.841 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 90679 2024-09-13T11:24:20.841 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:20.841 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:24:20.841 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:20.842 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:24:20.842 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:10.371726+0000", 2024-09-13T11:24:20.842 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:20.842 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:20.842 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:20.842 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:24:20.842 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:10.371726+0000", 2024-09-13T11:24:20.842 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:20.842 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:20.842 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:20.842 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:24:20.842 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:10.371730+0000", 2024-09-13T11:24:20.842 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.0879999999999997e-06 2024-09-13T11:24:20.843 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:20.843 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:20.843 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:24:20.843 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:10.371831+0000", 2024-09-13T11:24:20.843 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000101068 2024-09-13T11:24:20.843 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:20.843 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:20.843 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:24:20.843 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:10.371855+0000", 2024-09-13T11:24:20.843 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.4088e-05 2024-09-13T11:24:20.843 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:20.843 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:20.844 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:24:20.844 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:10.371867+0000", 2024-09-13T11:24:20.844 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.1704e-05 2024-09-13T11:24:20.844 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:20.844 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:20.844 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:24:20.844 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:10.371957+0000", 2024-09-13T11:24:20.844 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.9839000000000005e-05 2024-09-13T11:24:20.844 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:20.844 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:20.844 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:24:20.845 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:10.372103+0000", 2024-09-13T11:24:20.845 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00014578799999999999 2024-09-13T11:24:20.845 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:20.845 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:20.845 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 4", 2024-09-13T11:24:20.845 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:10.372239+0000", 2024-09-13T11:24:20.845 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00013661599999999999 2024-09-13T11:24:20.845 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:20.845 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:20.845 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:24:20.845 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:10.374246+0000", 2024-09-13T11:24:20.845 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.002006425 2024-09-13T11:24:20.845 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:20.846 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:20.846 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:20.846 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:10.433659+0000", 2024-09-13T11:24:20.846 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.059413277 2024-09-13T11:24:20.846 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:20.846 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:20.846 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:24:20.846 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:10.433767+0000", 2024-09-13T11:24:20.846 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00010771400000000001 2024-09-13T11:24:20.846 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:20.846 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:20.847 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:24:20.847 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:10.433848+0000", 2024-09-13T11:24:20.847 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.0935999999999999e-05 2024-09-13T11:24:20.847 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:20.847 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:24:20.847 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:20.847 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:20.847 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:20.847 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:90827 3.1e 3:7f623a7e:::benchmark_data_smithi177_31116_object90826:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110)", 2024-09-13T11:24:20.847 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:24:10.496964+0000", 2024-09-13T11:24:20.847 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 10.270371124, 2024-09-13T11:24:20.847 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.060601799999999997, 2024-09-13T11:24:20.848 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:24:20.848 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:24:20.848 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:20.848 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:24:20.848 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:24:20.848 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:20.848 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 90827 2024-09-13T11:24:20.848 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:20.848 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:24:20.848 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:20.849 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:24:20.849 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:10.496964+0000", 2024-09-13T11:24:20.849 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:20.849 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:20.849 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:20.849 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:24:20.849 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:10.496964+0000", 2024-09-13T11:24:20.849 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:20.849 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:20.849 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:20.849 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:24:20.849 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:10.496967+0000", 2024-09-13T11:24:20.849 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.802e-06 2024-09-13T11:24:20.850 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:20.850 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:20.850 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:24:20.850 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:10.497055+0000", 2024-09-13T11:24:20.850 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.8300999999999994e-05 2024-09-13T11:24:20.850 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:20.850 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:20.850 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:24:20.850 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:10.497078+0000", 2024-09-13T11:24:20.850 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.2912999999999999e-05 2024-09-13T11:24:20.850 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:20.850 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:20.851 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:24:20.851 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:10.497090+0000", 2024-09-13T11:24:20.851 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.2166e-05 2024-09-13T11:24:20.851 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:20.851 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:20.851 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:24:20.851 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:10.497635+0000", 2024-09-13T11:24:20.851 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000544682 2024-09-13T11:24:20.851 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:20.851 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:20.851 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:24:20.852 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:10.497775+0000", 2024-09-13T11:24:20.852 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000139431 2024-09-13T11:24:20.852 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:20.852 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:20.852 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 4", 2024-09-13T11:24:20.852 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:10.497914+0000", 2024-09-13T11:24:20.852 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000138975 2024-09-13T11:24:20.852 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:20.852 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:20.852 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:24:20.852 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:10.500177+0000", 2024-09-13T11:24:20.852 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0022637849999999999 2024-09-13T11:24:20.853 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:20.853 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:20.853 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:20.853 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:10.557379+0000", 2024-09-13T11:24:20.853 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.057201308999999999 2024-09-13T11:24:20.853 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:20.853 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:20.853 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:24:20.853 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:10.557496+0000", 2024-09-13T11:24:20.853 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00011735600000000001 2024-09-13T11:24:20.853 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:20.853 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:20.854 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:24:20.854 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:10.557566+0000", 2024-09-13T11:24:20.854 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.0080000000000007e-05 2024-09-13T11:24:20.854 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:20.854 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:24:20.854 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:20.854 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:20.854 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:20.854 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:91530 3.23 3:c6f704a1:::benchmark_data_smithi177_31116_object91529:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110)", 2024-09-13T11:24:20.854 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:24:10.942076+0000", 2024-09-13T11:24:20.854 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 9.825259097, 2024-09-13T11:24:20.854 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.073081012000000001, 2024-09-13T11:24:20.854 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:24:20.855 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:24:20.855 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:20.855 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:24:20.855 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:24:20.855 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:20.855 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 91530 2024-09-13T11:24:20.855 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:20.855 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:24:20.855 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:20.855 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:24:20.855 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:10.942076+0000", 2024-09-13T11:24:20.855 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:20.856 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:20.856 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:20.856 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:24:20.856 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:10.942076+0000", 2024-09-13T11:24:20.856 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:20.856 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:20.856 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:20.856 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:24:20.856 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:10.942080+0000", 2024-09-13T11:24:20.856 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.8920000000000002e-06 2024-09-13T11:24:20.856 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:20.856 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:20.857 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:24:20.857 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:10.942147+0000", 2024-09-13T11:24:20.857 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.6626999999999998e-05 2024-09-13T11:24:20.857 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:20.857 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:20.857 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:24:20.857 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:10.942174+0000", 2024-09-13T11:24:20.857 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.69e-05 2024-09-13T11:24:20.857 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:20.857 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:20.857 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:24:20.857 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:10.942189+0000", 2024-09-13T11:24:20.857 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.5038000000000001e-05 2024-09-13T11:24:20.858 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:20.858 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:20.858 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:24:20.858 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:10.942299+0000", 2024-09-13T11:24:20.858 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000110182 2024-09-13T11:24:20.858 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:20.858 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:20.858 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:24:20.858 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:10.942491+0000", 2024-09-13T11:24:20.858 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00019248299999999999 2024-09-13T11:24:20.859 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:20.859 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:20.859 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 9", 2024-09-13T11:24:20.859 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:10.942696+0000", 2024-09-13T11:24:20.859 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00020456900000000001 2024-09-13T11:24:20.859 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:20.859 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:20.859 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:24:20.859 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:10.944163+0000", 2024-09-13T11:24:20.859 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0014671930000000001 2024-09-13T11:24:20.859 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:20.859 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:20.859 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:20.860 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:11.014961+0000", 2024-09-13T11:24:20.860 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.070797918000000001 2024-09-13T11:24:20.860 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:20.860 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:20.860 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:24:20.860 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:11.015076+0000", 2024-09-13T11:24:20.860 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000114652 2024-09-13T11:24:20.860 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:20.860 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:20.860 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:24:20.860 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:11.015157+0000", 2024-09-13T11:24:20.861 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.1557999999999994e-05 2024-09-13T11:24:20.861 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:20.861 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:24:20.861 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:20.861 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:20.861 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:20.861 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:101001 3.1e 3:7a04ac47:::benchmark_data_smithi177_31116_object101000:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114)", 2024-09-13T11:24:20.861 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:24:17.344187+0000", 2024-09-13T11:24:20.861 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 3.423148657, 2024-09-13T11:24:20.861 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.058417968000000001, 2024-09-13T11:24:20.861 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:24:20.861 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:24:20.862 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:20.862 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:24:20.862 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:24:20.862 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:20.862 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 101001 2024-09-13T11:24:20.862 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:20.862 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:24:20.862 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:20.862 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:24:20.862 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:17.344187+0000", 2024-09-13T11:24:20.862 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:20.862 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:20.862 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:20.863 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:24:20.863 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:17.344187+0000", 2024-09-13T11:24:20.863 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:20.863 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:20.863 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:20.863 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:24:20.863 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:17.344189+0000", 2024-09-13T11:24:20.863 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.1270000000000001e-06 2024-09-13T11:24:20.863 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:20.863 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:20.863 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:24:20.863 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:17.344278+0000", 2024-09-13T11:24:20.863 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.8931000000000006e-05 2024-09-13T11:24:20.864 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:20.864 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:20.864 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:24:20.864 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:17.344296+0000", 2024-09-13T11:24:20.864 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.8065999999999999e-05 2024-09-13T11:24:20.864 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:20.864 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:20.864 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:24:20.864 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:17.344307+0000", 2024-09-13T11:24:20.864 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.1154e-05 2024-09-13T11:24:20.864 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:20.864 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:20.864 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:24:20.864 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:17.344366+0000", 2024-09-13T11:24:20.865 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.9206999999999997e-05 2024-09-13T11:24:20.865 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:20.865 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:20.865 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:24:20.865 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:17.344503+0000", 2024-09-13T11:24:20.865 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00013630099999999999 2024-09-13T11:24:20.865 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:20.865 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:20.865 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 4", 2024-09-13T11:24:20.865 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:17.344624+0000", 2024-09-13T11:24:20.865 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000121109 2024-09-13T11:24:20.865 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:20.865 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:20.866 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:24:20.866 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:17.346688+0000", 2024-09-13T11:24:20.866 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0020643139999999998 2024-09-13T11:24:20.866 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:20.866 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:20.866 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:20.866 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:17.402517+0000", 2024-09-13T11:24:20.866 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.055828728000000001 2024-09-13T11:24:20.866 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:20.866 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:20.866 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:24:20.866 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:17.402567+0000", 2024-09-13T11:24:20.866 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.9999e-05 2024-09-13T11:24:20.867 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:20.867 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:20.867 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:24:20.867 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:17.402605+0000", 2024-09-13T11:24:20.867 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.8031999999999998e-05 2024-09-13T11:24:20.867 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:20.867 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:24:20.867 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:20.867 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:20.867 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:20.867 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:104331 3.1e 3:79890035:::benchmark_data_smithi177_31116_object104330:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-09-13T11:24:20.867 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:24:20.241529+0000", 2024-09-13T11:24:20.867 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 0.52580658599999996, 2024-09-13T11:24:20.868 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.16976961700000001, 2024-09-13T11:24:20.868 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:24:20.868 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:24:20.868 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:20.868 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:24:20.868 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:24:20.868 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:20.868 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 104331 2024-09-13T11:24:20.868 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:20.868 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:24:20.868 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:20.868 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:24:20.869 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:20.241529+0000", 2024-09-13T11:24:20.869 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:20.869 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:20.869 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:20.869 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:24:20.869 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:20.241529+0000", 2024-09-13T11:24:20.869 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:20.869 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:20.869 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:20.869 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:24:20.869 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:20.241534+0000", 2024-09-13T11:24:20.869 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.775e-06 2024-09-13T11:24:20.870 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:20.870 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:20.870 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:24:20.870 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:20.241660+0000", 2024-09-13T11:24:20.870 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000126499 2024-09-13T11:24:20.870 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:20.870 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:20.870 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:24:20.870 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:20.241687+0000", 2024-09-13T11:24:20.870 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.6826999999999999e-05 2024-09-13T11:24:20.870 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:20.870 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:20.871 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:24:20.871 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:20.241703+0000", 2024-09-13T11:24:20.871 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.5858999999999998e-05 2024-09-13T11:24:20.871 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:20.871 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:20.871 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:24:20.871 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:20.241832+0000", 2024-09-13T11:24:20.871 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00012925799999999999 2024-09-13T11:24:20.871 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:20.871 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:20.871 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:24:20.871 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:20.242133+0000", 2024-09-13T11:24:20.872 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00030115900000000001 2024-09-13T11:24:20.872 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:20.872 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:20.872 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 4", 2024-09-13T11:24:20.872 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:20.242408+0000", 2024-09-13T11:24:20.872 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00027487799999999999 2024-09-13T11:24:20.872 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:20.872 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:20.872 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:24:20.872 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:20.246683+0000", 2024-09-13T11:24:20.873 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0042750469999999997 2024-09-13T11:24:20.873 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:20.873 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:20.873 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:20.873 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:20.411076+0000", 2024-09-13T11:24:20.873 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.16439300800000001 2024-09-13T11:24:20.873 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:20.873 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:20.873 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:24:20.873 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:20.411188+0000", 2024-09-13T11:24:20.873 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00011191 2024-09-13T11:24:20.873 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:20.874 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:20.874 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:24:20.874 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:20.411298+0000", 2024-09-13T11:24:20.874 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00011039699999999999 2024-09-13T11:24:20.874 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:20.874 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:24:20.874 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:20.874 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:20.874 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:20.874 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:104671 3.1e 3:794c2a6d:::benchmark_data_smithi177_31116_object104670:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-09-13T11:24:20.874 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:24:20.543331+0000", 2024-09-13T11:24:20.875 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 0.224004863, 2024-09-13T11:24:20.875 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.088033315000000001, 2024-09-13T11:24:20.875 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:24:20.875 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:24:20.875 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:20.875 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:24:20.875 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:24:20.875 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:20.875 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 104671 2024-09-13T11:24:20.875 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:20.876 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:24:20.876 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:20.876 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:24:20.877 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:20.543331+0000", 2024-09-13T11:24:20.877 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:20.877 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:20.877 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:20.877 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:24:20.877 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:20.543331+0000", 2024-09-13T11:24:20.877 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:20.877 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:20.877 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:20.877 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:24:20.877 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:20.543334+0000", 2024-09-13T11:24:20.877 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.1370000000000002e-06 2024-09-13T11:24:20.877 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:20.878 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:20.878 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:24:20.878 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:20.543386+0000", 2024-09-13T11:24:20.878 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.2527e-05 2024-09-13T11:24:20.878 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:20.878 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:20.878 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:24:20.878 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:20.543402+0000", 2024-09-13T11:24:20.878 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.6169000000000001e-05 2024-09-13T11:24:20.878 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:20.878 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:20.878 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:24:20.878 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:20.543412+0000", 2024-09-13T11:24:20.879 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.9960000000000005e-06 2024-09-13T11:24:20.879 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:20.879 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:20.879 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:24:20.879 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:20.543469+0000", 2024-09-13T11:24:20.879 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.6465000000000002e-05 2024-09-13T11:24:20.879 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:20.879 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:20.879 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:24:20.879 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:20.543579+0000", 2024-09-13T11:24:20.879 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000110263 2024-09-13T11:24:20.879 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:20.879 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:20.879 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 4", 2024-09-13T11:24:20.880 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:20.543676+0000", 2024-09-13T11:24:20.880 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.6725000000000007e-05 2024-09-13T11:24:20.880 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:20.880 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:20.880 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:24:20.880 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:20.545201+0000", 2024-09-13T11:24:20.880 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00152556 2024-09-13T11:24:20.880 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:20.880 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:20.880 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:20.881 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:20.631256+0000", 2024-09-13T11:24:20.881 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.08605438 2024-09-13T11:24:20.881 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:20.881 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:20.881 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:24:20.881 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:20.631313+0000", 2024-09-13T11:24:20.881 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.6860000000000001e-05 2024-09-13T11:24:20.881 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:20.881 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:20.881 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:24:20.881 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:20.631364+0000", 2024-09-13T11:24:20.882 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.1233000000000002e-05 2024-09-13T11:24:20.882 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:20.882 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:24:20.882 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:20.882 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:20.882 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:24:20.882 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-09-13T11:24:20.884 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-09-13T11:24:20.884 INFO:tasks.ceph.osd.14.smithi162.stderr:2024-09-13T11:24:20.880+0000 7f235c0db640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26319) UID: 0 2024-09-13T11:24:20.984 INFO:tasks.ceph.osd.7.smithi177.stderr:2024-09-13T11:24:20.983+0000 7f897b51d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26575) UID: 0 2024-09-13T11:24:21.009 INFO:tasks.radosbench.radosbench.0.smithi177.stdout: 114 16 105218 105202 57.6708 71.625 0.00702297 0.0173341 2024-09-13T11:24:21.055 INFO:teuthology.orchestra.run.smithi001.stderr:set pool 3 pg_num to 46 2024-09-13T11:24:21.061 INFO:tasks.ceph.osd.9.smithi092.stderr:2024-09-13T11:24:21.058+0000 7ff59ddf9640 -1 osd.9 115 handle_osd_map injecting map crc failure 2024-09-13T11:24:21.075 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-09-13T11:24:21.075 INFO:teuthology.orchestra.run.smithi092.stdout: "ops": [], 2024-09-13T11:24:21.075 INFO:teuthology.orchestra.run.smithi092.stdout: "num_ops": 0 2024-09-13T11:24:21.075 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-09-13T11:24:21.085 INFO:tasks.ceph.osd.14.smithi162.stderr:2024-09-13T11:24:21.080+0000 7f235c0db640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26319) UID: 0 2024-09-13T11:24:21.096 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-09-13T11:24:21.185 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-13T11:24:21.183+0000 7f6a68775640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26307) UID: 0 2024-09-13T11:24:21.228 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-09-13T11:24:21.228 INFO:teuthology.orchestra.run.smithi092.stdout: "ops": [], 2024-09-13T11:24:21.228 INFO:teuthology.orchestra.run.smithi092.stdout: "complaint_time": 30, 2024-09-13T11:24:21.228 INFO:teuthology.orchestra.run.smithi092.stdout: "num_blocked_ops": 0 2024-09-13T11:24:21.229 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-09-13T11:24:21.247 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-09-13T11:24:21.293 INFO:tasks.ceph.osd.1.smithi092.stderr:2024-09-13T11:24:21.290+0000 7fe7789c5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33718) UID: 0 2024-09-13T11:24:21.387 INFO:tasks.ceph.osd.15.smithi177.stderr:2024-09-13T11:24:21.383+0000 7f9d6ce77640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26578) UID: 0 2024-09-13T11:24:21.388 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-09-13T11:24:21.388 INFO:teuthology.orchestra.run.smithi092.stdout: "size": 20, 2024-09-13T11:24:21.389 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 600, 2024-09-13T11:24:21.389 INFO:teuthology.orchestra.run.smithi092.stdout: "ops": [ 2024-09-13T11:24:21.389 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:21.389 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "MOSDPGPush(3.19 76/70 [PushOp(3:9eacf3d0:::benchmark_data_smithi177_31116_object19747:head, version: 54'1236, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:9eacf3d0:::benchmark_data_smithi177_31116_object19747:head@54'1236, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:24:21.389 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:32.704676+0000", 2024-09-13T11:24:21.389 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 48.681928397999997, 2024-09-13T11:24:21.389 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.23525014899999999, 2024-09-13T11:24:21.389 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:24:21.389 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:24:21.389 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-09-13T11:24:21.389 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:24:21.389 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:21.389 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:24:21.389 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:32.704676+0000", 2024-09-13T11:24:21.390 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:21.390 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:21.390 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:21.390 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:24:21.390 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:32.704676+0000", 2024-09-13T11:24:21.390 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:21.390 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:21.390 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:21.390 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:24:21.390 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:32.704677+0000", 2024-09-13T11:24:21.390 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.531e-06 2024-09-13T11:24:21.390 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:21.390 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:21.391 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:24:21.391 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:32.704749+0000", 2024-09-13T11:24:21.391 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.1451000000000004e-05 2024-09-13T11:24:21.391 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:21.391 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:21.392 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:24:21.392 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:32.938053+0000", 2024-09-13T11:24:21.392 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.233304752 2024-09-13T11:24:21.392 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:21.392 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:21.392 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:24:21.392 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:32.938087+0000", 2024-09-13T11:24:21.392 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.4016000000000001e-05 2024-09-13T11:24:21.392 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:21.392 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:21.392 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:24:21.392 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:32.938351+0000", 2024-09-13T11:24:21.392 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00026397500000000001 2024-09-13T11:24:21.393 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:21.393 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:21.393 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:24:21.393 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:32.938381+0000", 2024-09-13T11:24:21.393 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.0001999999999998e-05 2024-09-13T11:24:21.393 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:21.393 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:21.393 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:24:21.393 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:32.939926+0000", 2024-09-13T11:24:21.393 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0015444219999999999 2024-09-13T11:24:21.393 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:21.393 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:24:21.393 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:21.393 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:21.394 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:21.394 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "MOSDPGPush(3.23 80/78 [PushOp(3:c54a2e13:::benchmark_data_smithi177_31116_object16923:head, version: 52'1050, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:c54a2e13:::benchmark_data_smithi177_31116_object16923:head@52'1050, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:24:21.394 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:37.274061+0000", 2024-09-13T11:24:21.394 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 44.112542677999997, 2024-09-13T11:24:21.394 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.099925987999999993, 2024-09-13T11:24:21.394 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:24:21.394 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:24:21.394 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-09-13T11:24:21.394 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:24:21.394 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:21.394 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:24:21.394 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:37.274061+0000", 2024-09-13T11:24:21.394 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:21.394 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:21.395 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:21.395 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:24:21.395 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:37.274061+0000", 2024-09-13T11:24:21.395 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:21.395 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:21.395 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:21.395 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:24:21.395 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:37.274064+0000", 2024-09-13T11:24:21.395 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.3439999999999999e-06 2024-09-13T11:24:21.395 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:21.395 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:21.395 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:24:21.395 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:37.274135+0000", 2024-09-13T11:24:21.396 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.1716000000000003e-05 2024-09-13T11:24:21.396 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:21.396 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:21.396 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:24:21.396 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:37.372114+0000", 2024-09-13T11:24:21.396 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.097978407000000003 2024-09-13T11:24:21.396 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:21.396 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:21.396 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:24:21.396 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:37.372133+0000", 2024-09-13T11:24:21.396 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.8983000000000001e-05 2024-09-13T11:24:21.396 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:21.396 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:21.397 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:24:21.397 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:37.373424+0000", 2024-09-13T11:24:21.397 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0012912220000000001 2024-09-13T11:24:21.397 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:21.397 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:21.397 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:24:21.397 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:37.373437+0000", 2024-09-13T11:24:21.397 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.2608e-05 2024-09-13T11:24:21.397 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:21.397 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:21.397 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:24:21.397 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:37.373987+0000", 2024-09-13T11:24:21.397 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00055070800000000001 2024-09-13T11:24:21.397 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:21.398 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:24:21.398 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:21.398 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:21.398 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:21.398 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "MOSDPGPush(3.23 80/78 [PushOp(3:c59ecc6d:::benchmark_data_smithi177_31116_object17005:head, version: 52'1054, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:c59ecc6d:::benchmark_data_smithi177_31116_object17005:head@52'1054, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:24:21.398 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:37.433040+0000", 2024-09-13T11:24:21.398 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 43.953564421999999, 2024-09-13T11:24:21.398 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.26658219100000002, 2024-09-13T11:24:21.398 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:24:21.398 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:24:21.398 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-09-13T11:24:21.398 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:24:21.398 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:21.399 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:24:21.399 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:37.433040+0000", 2024-09-13T11:24:21.399 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:21.399 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:21.399 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:21.399 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:24:21.399 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:37.433040+0000", 2024-09-13T11:24:21.399 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:21.399 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:21.399 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:21.399 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:24:21.399 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:37.433042+0000", 2024-09-13T11:24:21.399 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.3999999999999999e-06 2024-09-13T11:24:21.400 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:21.400 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:21.400 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:24:21.400 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:37.433090+0000", 2024-09-13T11:24:21.400 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.8065999999999997e-05 2024-09-13T11:24:21.400 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:21.400 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:21.400 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:24:21.400 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:37.698797+0000", 2024-09-13T11:24:21.400 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.26570674500000002 2024-09-13T11:24:21.400 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:21.400 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:21.400 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:24:21.400 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:37.698827+0000", 2024-09-13T11:24:21.401 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.0505e-05 2024-09-13T11:24:21.401 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:21.401 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:21.401 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:24:21.401 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:37.698941+0000", 2024-09-13T11:24:21.401 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00011403799999999999 2024-09-13T11:24:21.401 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:21.401 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:21.401 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:24:21.401 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:37.698956+0000", 2024-09-13T11:24:21.401 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.4765e-05 2024-09-13T11:24:21.401 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:21.401 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:21.402 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:24:21.402 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:37.699622+0000", 2024-09-13T11:24:21.402 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00066567199999999997 2024-09-13T11:24:21.402 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:21.402 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:24:21.402 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:21.402 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:21.402 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:21.402 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "MOSDPGPush(3.1a 81/78 [PushOp(3:5a9c05b8:::benchmark_data_smithi177_31116_object4600:head, version: 44'280, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:5a9c05b8:::benchmark_data_smithi177_31116_object4600:head@44'280, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:24:21.402 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:38.261429+0000", 2024-09-13T11:24:21.402 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 43.125174985999998, 2024-09-13T11:24:21.402 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.13814789799999999, 2024-09-13T11:24:21.402 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:24:21.402 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:24:21.403 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-09-13T11:24:21.403 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:24:21.403 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:21.403 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:24:21.403 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:38.261429+0000", 2024-09-13T11:24:21.403 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:21.403 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:21.403 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:21.403 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:24:21.403 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:38.261429+0000", 2024-09-13T11:24:21.403 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:21.403 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:21.403 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:21.403 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:24:21.404 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:38.261430+0000", 2024-09-13T11:24:21.404 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.376e-06 2024-09-13T11:24:21.404 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:21.404 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:21.404 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:24:21.404 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:38.261501+0000", 2024-09-13T11:24:21.404 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.0724999999999998e-05 2024-09-13T11:24:21.404 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:21.404 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:21.404 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:24:21.404 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:38.398898+0000", 2024-09-13T11:24:21.404 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.13739662499999999 2024-09-13T11:24:21.404 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:21.405 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:21.405 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:24:21.405 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:38.398909+0000", 2024-09-13T11:24:21.405 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0926e-05 2024-09-13T11:24:21.405 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:21.405 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:21.405 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:24:21.405 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:38.398974+0000", 2024-09-13T11:24:21.405 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.5467000000000002e-05 2024-09-13T11:24:21.405 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:21.405 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:21.405 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:24:21.405 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:38.398982+0000", 2024-09-13T11:24:21.406 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.841e-06 2024-09-13T11:24:21.406 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:21.406 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:21.406 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:24:21.406 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:38.399577+0000", 2024-09-13T11:24:21.406 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00059493800000000004 2024-09-13T11:24:21.406 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:21.406 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:24:21.406 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:21.406 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:21.406 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:21.406 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "MOSDPGPush(3.1a 85/78 [PushOp(3:5dc85525:::benchmark_data_smithi177_31116_object11755:head, version: 49'723, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:5dc85525:::benchmark_data_smithi177_31116_object11755:head@49'723, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:24:21.406 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:42.422061+0000", 2024-09-13T11:24:21.407 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 38.964542969, 2024-09-13T11:24:21.407 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.52875617500000005, 2024-09-13T11:24:21.407 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:24:21.407 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:24:21.407 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-09-13T11:24:21.407 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:24:21.407 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:21.407 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:24:21.407 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.422061+0000", 2024-09-13T11:24:21.408 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:21.408 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:21.408 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:21.408 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:24:21.408 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.422061+0000", 2024-09-13T11:24:21.408 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:21.408 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:21.408 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:21.408 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:24:21.408 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.422064+0000", 2024-09-13T11:24:21.408 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.5409999999999999e-06 2024-09-13T11:24:21.408 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:21.408 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:21.409 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:24:21.409 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.422139+0000", 2024-09-13T11:24:21.409 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.5488999999999995e-05 2024-09-13T11:24:21.409 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:21.409 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:21.409 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:24:21.409 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.948781+0000", 2024-09-13T11:24:21.409 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.52664230999999995 2024-09-13T11:24:21.409 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:21.409 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:21.409 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:24:21.409 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.948841+0000", 2024-09-13T11:24:21.409 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.9450999999999997e-05 2024-09-13T11:24:21.409 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:21.410 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:21.410 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:24:21.410 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.949035+0000", 2024-09-13T11:24:21.410 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00019380600000000001 2024-09-13T11:24:21.410 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:21.410 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:21.410 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:24:21.410 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.949060+0000", 2024-09-13T11:24:21.410 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.5381999999999999e-05 2024-09-13T11:24:21.410 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:21.410 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:21.410 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:24:21.410 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.950817+0000", 2024-09-13T11:24:21.410 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0017571959999999999 2024-09-13T11:24:21.411 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:21.411 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:24:21.411 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:21.411 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:21.411 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:21.411 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "MOSDPGPush(3.1a 89/78 [PushOp(3:59270734:::benchmark_data_smithi177_31116_object20039:head, version: 54'1256, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:59270734:::benchmark_data_smithi177_31116_object20039:head@54'1256, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:24:21.411 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:46.607562+0000", 2024-09-13T11:24:21.411 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 34.779042191999999, 2024-09-13T11:24:21.411 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.32123945300000001, 2024-09-13T11:24:21.411 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:24:21.411 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:24:21.411 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-09-13T11:24:21.411 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:24:21.412 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:21.412 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:24:21.412 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.607562+0000", 2024-09-13T11:24:21.412 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:21.412 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:21.412 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:21.412 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:24:21.412 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.607562+0000", 2024-09-13T11:24:21.412 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:21.412 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:21.412 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:21.412 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:24:21.412 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.607564+0000", 2024-09-13T11:24:21.412 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.8610000000000001e-06 2024-09-13T11:24:21.413 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:21.413 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:21.413 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:24:21.413 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.607663+0000", 2024-09-13T11:24:21.413 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.9210999999999996e-05 2024-09-13T11:24:21.413 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:21.413 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:21.413 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:24:21.413 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.925193+0000", 2024-09-13T11:24:21.413 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.317530586 2024-09-13T11:24:21.413 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:21.413 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:21.413 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:24:21.413 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.925250+0000", 2024-09-13T11:24:21.414 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.6925000000000001e-05 2024-09-13T11:24:21.414 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:21.414 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:21.414 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:24:21.414 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.927013+0000", 2024-09-13T11:24:21.414 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.001762853 2024-09-13T11:24:21.414 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:21.414 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:21.414 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:24:21.414 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.927067+0000", 2024-09-13T11:24:21.414 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.3601999999999999e-05 2024-09-13T11:24:21.414 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:21.414 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:21.414 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:24:21.415 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.928801+0000", 2024-09-13T11:24:21.415 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.001734415 2024-09-13T11:24:21.415 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:21.415 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:24:21.415 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:21.415 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:21.415 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:21.415 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "MOSDPGPush(3.1a 90/78 [PushOp(3:5d2269a4:::benchmark_data_smithi177_31116_object20650:head, version: 54'1284, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:5d2269a4:::benchmark_data_smithi177_31116_object20650:head@54'1284, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:24:21.415 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:47.156435+0000", 2024-09-13T11:24:21.415 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 34.230168562999999, 2024-09-13T11:24:21.415 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.098664977000000001, 2024-09-13T11:24:21.415 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:24:21.415 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:24:21.415 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-09-13T11:24:21.416 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:24:21.416 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:21.416 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:24:21.416 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:47.156435+0000", 2024-09-13T11:24:21.416 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:21.416 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:21.416 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:21.416 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:24:21.416 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:47.156435+0000", 2024-09-13T11:24:21.416 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:21.416 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:21.416 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:21.416 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:24:21.417 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:47.156438+0000", 2024-09-13T11:24:21.417 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.492e-06 2024-09-13T11:24:21.417 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:21.417 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:21.417 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:24:21.417 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:47.156508+0000", 2024-09-13T11:24:21.417 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.0364000000000006e-05 2024-09-13T11:24:21.417 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:21.417 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:21.417 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:24:21.417 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:47.254084+0000", 2024-09-13T11:24:21.417 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.097576044000000001 2024-09-13T11:24:21.417 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:21.418 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:21.418 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:24:21.418 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:47.254113+0000", 2024-09-13T11:24:21.418 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.8384000000000001e-05 2024-09-13T11:24:21.418 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:21.418 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:21.418 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:24:21.418 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:47.254246+0000", 2024-09-13T11:24:21.418 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000133662 2024-09-13T11:24:21.418 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:21.418 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:21.418 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:24:21.418 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:47.254259+0000", 2024-09-13T11:24:21.418 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.2673999999999999e-05 2024-09-13T11:24:21.419 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:21.419 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:21.419 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:24:21.419 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:47.255100+0000", 2024-09-13T11:24:21.419 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00084135700000000004 2024-09-13T11:24:21.419 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:21.419 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:24:21.419 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:21.419 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:21.419 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:21.419 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:64389 3.19 3:9e2bf991:::benchmark_data_smithi177_31116_object64388:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90)", 2024-09-13T11:24:21.419 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:47.195560+0000", 2024-09-13T11:24:21.419 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 34.191044165000001, 2024-09-13T11:24:21.419 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.108379857, 2024-09-13T11:24:21.420 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:24:21.420 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:24:21.420 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:21.420 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:24:21.420 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:24:21.420 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:21.420 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 64389 2024-09-13T11:24:21.420 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:21.420 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:24:21.420 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:21.420 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:24:21.420 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:47.195560+0000", 2024-09-13T11:24:21.420 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:21.421 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:21.421 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:21.421 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:24:21.421 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:47.195560+0000", 2024-09-13T11:24:21.421 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:21.421 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:21.421 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:21.421 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:24:21.421 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:47.195561+0000", 2024-09-13T11:24:21.421 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.4300000000000001e-06 2024-09-13T11:24:21.421 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:21.421 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:21.421 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:24:21.421 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:47.195649+0000", 2024-09-13T11:24:21.422 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.7379000000000005e-05 2024-09-13T11:24:21.422 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:21.422 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:21.422 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:24:21.422 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:47.195663+0000", 2024-09-13T11:24:21.422 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.4012999999999999e-05 2024-09-13T11:24:21.422 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:21.422 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:21.422 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:24:21.422 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:47.195673+0000", 2024-09-13T11:24:21.422 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0045e-05 2024-09-13T11:24:21.422 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:21.422 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:21.422 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:24:21.423 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:47.195739+0000", 2024-09-13T11:24:21.423 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.6651999999999998e-05 2024-09-13T11:24:21.423 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:21.423 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:21.423 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:24:21.423 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:47.195908+0000", 2024-09-13T11:24:21.423 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00016875700000000001 2024-09-13T11:24:21.423 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:21.423 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:21.423 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1", 2024-09-13T11:24:21.423 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:47.196082+0000", 2024-09-13T11:24:21.423 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00017355199999999999 2024-09-13T11:24:21.423 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:21.423 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:21.424 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:24:21.424 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:47.197432+0000", 2024-09-13T11:24:21.424 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0013507700000000001 2024-09-13T11:24:21.424 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:21.424 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:21.424 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:21.424 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:47.303796+0000", 2024-09-13T11:24:21.424 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.106363729 2024-09-13T11:24:21.424 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:21.424 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:21.424 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:24:21.425 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:47.303875+0000", 2024-09-13T11:24:21.425 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.9092999999999996e-05 2024-09-13T11:24:21.425 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:21.425 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:21.425 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:24:21.425 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:47.303940+0000", 2024-09-13T11:24:21.425 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.4437000000000007e-05 2024-09-13T11:24:21.425 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:21.425 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:24:21.425 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:21.425 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:21.425 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:21.425 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "MOSDPGPush(3.1a 91/78 [PushOp(3:5e5010d2:::benchmark_data_smithi177_31116_object25733:head, version: 58'1609, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:5e5010d2:::benchmark_data_smithi177_31116_object25733:head@58'1609, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:24:21.426 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:48.608485+0000", 2024-09-13T11:24:21.426 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 32.778119074000003, 2024-09-13T11:24:21.426 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.35035599699999997, 2024-09-13T11:24:21.426 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:24:21.426 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:24:21.426 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-09-13T11:24:21.426 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:24:21.426 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:21.426 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:24:21.426 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:48.608485+0000", 2024-09-13T11:24:21.426 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:21.426 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:21.426 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:21.426 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:24:21.427 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:48.608485+0000", 2024-09-13T11:24:21.427 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:21.427 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:21.427 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:21.427 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:24:21.427 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:48.608488+0000", 2024-09-13T11:24:21.427 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.9399999999999998e-06 2024-09-13T11:24:21.427 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:21.427 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:21.427 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:24:21.427 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:48.608552+0000", 2024-09-13T11:24:21.427 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.3731000000000004e-05 2024-09-13T11:24:21.427 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:21.427 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:21.428 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:24:21.428 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:48.957330+0000", 2024-09-13T11:24:21.428 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.34877806099999997 2024-09-13T11:24:21.428 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:21.428 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:21.428 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:24:21.428 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:48.957361+0000", 2024-09-13T11:24:21.428 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.0852000000000002e-05 2024-09-13T11:24:21.428 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:21.428 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:21.428 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:24:21.428 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:48.957544+0000", 2024-09-13T11:24:21.429 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000183537 2024-09-13T11:24:21.429 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:21.429 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:21.429 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:24:21.429 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:48.957582+0000", 2024-09-13T11:24:21.429 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.7472999999999998e-05 2024-09-13T11:24:21.429 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:21.429 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:21.429 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:24:21.429 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:48.958841+0000", 2024-09-13T11:24:21.430 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0012594030000000001 2024-09-13T11:24:21.430 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:21.430 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:24:21.430 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:21.430 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:21.430 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:21.430 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "MOSDPGPush(3.1a 95/78 [PushOp(3:5f9270ec:::benchmark_data_smithi177_31116_object33007:head, version: 60'2076, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:5f9270ec:::benchmark_data_smithi177_31116_object33007:head@60'2076, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:24:21.430 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:52.817095+0000", 2024-09-13T11:24:21.430 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 28.569509371999999, 2024-09-13T11:24:21.430 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.097724647999999997, 2024-09-13T11:24:21.430 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:24:21.431 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:24:21.431 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-09-13T11:24:21.431 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:24:21.431 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:21.431 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:24:21.431 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:52.817095+0000", 2024-09-13T11:24:21.431 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:21.431 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:21.431 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:21.431 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:24:21.431 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:52.817095+0000", 2024-09-13T11:24:21.431 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:21.431 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:21.431 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:21.432 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:24:21.432 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:52.817097+0000", 2024-09-13T11:24:21.432 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.4899999999999999e-06 2024-09-13T11:24:21.432 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:21.432 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:21.432 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:24:21.432 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:52.817175+0000", 2024-09-13T11:24:21.432 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.8320999999999998e-05 2024-09-13T11:24:21.432 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:21.432 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:21.432 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:24:21.432 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:52.913313+0000", 2024-09-13T11:24:21.432 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.096137632000000001 2024-09-13T11:24:21.433 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:21.433 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:21.433 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:24:21.433 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:52.913346+0000", 2024-09-13T11:24:21.433 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.2835e-05 2024-09-13T11:24:21.433 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:21.433 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:21.433 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:24:21.433 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:52.913580+0000", 2024-09-13T11:24:21.433 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000234579 2024-09-13T11:24:21.433 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:21.433 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:21.433 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:24:21.434 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:52.913598+0000", 2024-09-13T11:24:21.434 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.7447000000000001e-05 2024-09-13T11:24:21.434 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:21.434 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:21.434 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:24:21.434 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:52.914819+0000", 2024-09-13T11:24:21.434 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0012213440000000001 2024-09-13T11:24:21.434 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:21.434 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:24:21.434 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:21.434 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:21.434 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:21.434 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "MOSDPGPush(3.1a 95/78 [PushOp(3:58a56058:::benchmark_data_smithi177_31116_object33115:head, version: 60'2079, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:58a56058:::benchmark_data_smithi177_31116_object33115:head@60'2079, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:24:21.434 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:53.080252+0000", 2024-09-13T11:24:21.435 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 28.30635225, 2024-09-13T11:24:21.435 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.10589196200000001, 2024-09-13T11:24:21.435 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:24:21.435 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:24:21.435 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-09-13T11:24:21.435 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:24:21.435 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:21.435 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:24:21.435 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:53.080252+0000", 2024-09-13T11:24:21.435 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:21.435 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:21.435 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:21.435 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:24:21.435 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:53.080252+0000", 2024-09-13T11:24:21.436 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:21.436 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:21.436 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:21.436 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:24:21.436 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:53.080254+0000", 2024-09-13T11:24:21.436 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.2639999999999998e-06 2024-09-13T11:24:21.436 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:21.436 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:21.436 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:24:21.436 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:53.080316+0000", 2024-09-13T11:24:21.436 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.2283e-05 2024-09-13T11:24:21.436 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:21.436 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:21.437 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:24:21.437 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:53.183273+0000", 2024-09-13T11:24:21.437 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.102956333 2024-09-13T11:24:21.437 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:21.437 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:21.437 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:24:21.437 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:53.183295+0000", 2024-09-13T11:24:21.437 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.2459e-05 2024-09-13T11:24:21.437 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:21.437 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:21.437 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:24:21.437 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:53.185153+0000", 2024-09-13T11:24:21.437 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.001857893 2024-09-13T11:24:21.437 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:21.438 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:21.438 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:24:21.438 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:53.185177+0000", 2024-09-13T11:24:21.438 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.3923000000000001e-05 2024-09-13T11:24:21.438 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:21.438 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:21.438 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:24:21.438 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:53.186144+0000", 2024-09-13T11:24:21.438 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00096680699999999997 2024-09-13T11:24:21.438 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:21.438 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:24:21.438 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:21.438 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:21.439 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:21.439 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop(client.4665.0:72923 3.23 e97/90 3:c6230123:::benchmark_data_smithi177_31116_object72922:head v 97'3418, mlcod=97'3414)", 2024-09-13T11:24:21.439 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:55.047734+0000", 2024-09-13T11:24:21.439 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 26.338869556999999, 2024-09-13T11:24:21.439 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.20075528000000001, 2024-09-13T11:24:21.439 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:24:21.439 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:24:21.439 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:21.439 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:24:21.439 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:21.439 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:24:21.439 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:55.047734+0000", 2024-09-13T11:24:21.439 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:21.439 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:21.440 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:21.440 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:24:21.440 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:55.047734+0000", 2024-09-13T11:24:21.440 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:21.440 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:21.440 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:21.440 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:24:21.440 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:55.047736+0000", 2024-09-13T11:24:21.440 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.466e-06 2024-09-13T11:24:21.440 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:21.440 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:21.440 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:24:21.440 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:55.047772+0000", 2024-09-13T11:24:21.440 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.6576999999999999e-05 2024-09-13T11:24:21.441 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:21.441 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:21.441 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:24:21.441 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:55.246478+0000", 2024-09-13T11:24:21.441 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.19870581300000001 2024-09-13T11:24:21.441 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:21.441 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:21.441 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:24:21.441 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:55.246488+0000", 2024-09-13T11:24:21.441 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.5419999999999994e-06 2024-09-13T11:24:21.441 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:21.441 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:21.441 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:24:21.442 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:55.246631+0000", 2024-09-13T11:24:21.442 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00014287300000000001 2024-09-13T11:24:21.442 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:21.442 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:21.442 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:24:21.442 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:55.246665+0000", 2024-09-13T11:24:21.442 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.4684999999999997e-05 2024-09-13T11:24:21.442 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:21.442 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:21.442 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:24:21.442 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:55.248453+0000", 2024-09-13T11:24:21.442 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0017879459999999999 2024-09-13T11:24:21.442 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:21.442 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:21.443 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:24:21.443 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:55.248490+0000", 2024-09-13T11:24:21.443 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.6378000000000003e-05 2024-09-13T11:24:21.443 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:21.443 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:24:21.443 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:21.443 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:21.443 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:21.443 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop(client.4665.0:73503 3.23 e98/90 3:c5e498be:::benchmark_data_smithi177_31116_object73502:head v 98'3426, mlcod=98'3423)", 2024-09-13T11:24:21.443 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:55.521226+0000", 2024-09-13T11:24:21.443 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 25.865377761000001, 2024-09-13T11:24:21.443 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.097626377, 2024-09-13T11:24:21.444 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:24:21.444 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:24:21.444 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:21.444 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:24:21.444 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:21.444 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:24:21.444 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:55.521226+0000", 2024-09-13T11:24:21.444 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:21.444 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:21.444 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:21.444 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:24:21.444 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:55.521226+0000", 2024-09-13T11:24:21.444 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:21.445 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:21.445 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:21.445 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:24:21.445 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:55.521228+0000", 2024-09-13T11:24:21.445 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.643e-06 2024-09-13T11:24:21.445 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:21.445 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:21.445 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:24:21.445 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:55.521253+0000", 2024-09-13T11:24:21.445 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.5435e-05 2024-09-13T11:24:21.445 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:21.445 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:21.445 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:24:21.445 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:55.617781+0000", 2024-09-13T11:24:21.446 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.096527811000000005 2024-09-13T11:24:21.446 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:21.446 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:21.446 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:24:21.446 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:55.617787+0000", 2024-09-13T11:24:21.446 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.5969999999999999e-06 2024-09-13T11:24:21.446 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:21.446 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:21.446 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:24:21.446 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:55.617854+0000", 2024-09-13T11:24:21.446 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.6843999999999999e-05 2024-09-13T11:24:21.446 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:21.446 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:21.446 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:24:21.447 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:55.617874+0000", 2024-09-13T11:24:21.447 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.0788000000000002e-05 2024-09-13T11:24:21.447 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:21.447 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:21.447 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:24:21.447 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:55.618810+0000", 2024-09-13T11:24:21.447 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00093603499999999999 2024-09-13T11:24:21.447 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:21.447 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:21.447 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:24:21.447 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:55.618853+0000", 2024-09-13T11:24:21.447 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.2224000000000001e-05 2024-09-13T11:24:21.447 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:21.448 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:24:21.448 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:21.448 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:21.448 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:21.448 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:79823 3.19 3:9e038f3c:::benchmark_data_smithi177_31116_object79822:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102)", 2024-09-13T11:24:21.448 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:24:00.563188+0000", 2024-09-13T11:24:21.448 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 20.823415932, 2024-09-13T11:24:21.448 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.164156422, 2024-09-13T11:24:21.448 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:24:21.448 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:24:21.448 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:21.448 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:24:21.449 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:24:21.449 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:21.449 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 79823 2024-09-13T11:24:21.449 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:21.449 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:24:21.449 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:21.449 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:24:21.449 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:00.563188+0000", 2024-09-13T11:24:21.449 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:21.449 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:21.449 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:21.449 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:24:21.449 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:00.563188+0000", 2024-09-13T11:24:21.450 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:21.450 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:21.450 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:21.450 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:24:21.450 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:00.563201+0000", 2024-09-13T11:24:21.450 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.3183000000000001e-05 2024-09-13T11:24:21.450 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:21.450 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:21.450 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:24:21.450 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:00.563263+0000", 2024-09-13T11:24:21.450 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.1904999999999998e-05 2024-09-13T11:24:21.450 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:21.450 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:21.451 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:24:21.451 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:00.563297+0000", 2024-09-13T11:24:21.451 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.3979000000000001e-05 2024-09-13T11:24:21.451 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:21.451 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:21.451 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:24:21.451 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:00.563318+0000", 2024-09-13T11:24:21.451 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.0703000000000002e-05 2024-09-13T11:24:21.451 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:21.451 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:21.451 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:24:21.451 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:00.563513+0000", 2024-09-13T11:24:21.451 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00019524200000000001 2024-09-13T11:24:21.451 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:21.451 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:21.452 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:24:21.452 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:00.563758+0000", 2024-09-13T11:24:21.452 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000244711 2024-09-13T11:24:21.452 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:21.452 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:21.452 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1", 2024-09-13T11:24:21.452 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:00.563978+0000", 2024-09-13T11:24:21.452 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00022011700000000001 2024-09-13T11:24:21.452 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:21.452 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:21.452 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:24:21.452 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:00.566893+0000", 2024-09-13T11:24:21.452 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0029147650000000002 2024-09-13T11:24:21.453 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:21.453 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:21.453 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:21.453 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:00.727176+0000", 2024-09-13T11:24:21.453 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.16028374100000001 2024-09-13T11:24:21.453 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:21.453 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:21.453 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:24:21.453 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:00.727271+0000", 2024-09-13T11:24:21.453 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.4481000000000006e-05 2024-09-13T11:24:21.453 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:21.453 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:21.454 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:24:21.454 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:00.727344+0000", 2024-09-13T11:24:21.454 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.3595000000000003e-05 2024-09-13T11:24:21.454 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:21.454 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:24:21.454 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:21.454 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:21.454 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:21.454 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop_reply(client.4665.0:79823 3.19 e102/84 ondisk, result = 0)", 2024-09-13T11:24:21.454 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:24:00.567585+0000", 2024-09-13T11:24:21.454 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 20.819018621000001, 2024-09-13T11:24:21.454 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.15977266400000001, 2024-09-13T11:24:21.454 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:24:21.454 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:24:21.455 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-09-13T11:24:21.455 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:24:21.455 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:21.455 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:24:21.455 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:00.567585+0000", 2024-09-13T11:24:21.455 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:21.455 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:21.455 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:21.455 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:24:21.455 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:00.567585+0000", 2024-09-13T11:24:21.455 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:21.455 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:21.455 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:21.456 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:24:21.456 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:00.567589+0000", 2024-09-13T11:24:21.456 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.2969999999999999e-06 2024-09-13T11:24:21.456 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:21.456 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:21.456 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:24:21.456 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:00.567599+0000", 2024-09-13T11:24:21.456 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0142999999999999e-05 2024-09-13T11:24:21.456 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:21.456 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:21.456 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:24:21.456 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:00.726959+0000", 2024-09-13T11:24:21.456 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.15936018399999999 2024-09-13T11:24:21.457 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:21.457 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:21.457 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:24:21.457 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:00.726972+0000", 2024-09-13T11:24:21.457 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.2622999999999999e-05 2024-09-13T11:24:21.457 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:21.457 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:21.457 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:24:21.457 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:00.727151+0000", 2024-09-13T11:24:21.457 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00017909600000000001 2024-09-13T11:24:21.457 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:21.457 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:21.457 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:24:21.458 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:00.727167+0000", 2024-09-13T11:24:21.458 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.6639000000000001e-05 2024-09-13T11:24:21.458 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:21.458 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:21.458 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:24:21.458 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:00.727358+0000", 2024-09-13T11:24:21.458 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00019068199999999999 2024-09-13T11:24:21.458 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:21.458 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:24:21.458 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:21.458 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:21.458 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:21.458 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:98229 3.19 3:98f5bd4f:::benchmark_data_smithi177_31116_object98228:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113)", 2024-09-13T11:24:21.459 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:24:15.139305+0000", 2024-09-13T11:24:21.459 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 6.2472987089999998, 2024-09-13T11:24:21.459 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.116582746, 2024-09-13T11:24:21.459 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:24:21.459 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:24:21.459 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:21.459 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:24:21.459 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:24:21.459 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:21.459 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 98229 2024-09-13T11:24:21.459 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:21.459 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:24:21.459 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:21.459 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:24:21.460 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:15.139305+0000", 2024-09-13T11:24:21.460 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:21.460 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:21.460 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:21.460 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:24:21.460 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:15.139305+0000", 2024-09-13T11:24:21.460 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:21.460 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:21.460 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:21.460 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:24:21.460 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:15.139311+0000", 2024-09-13T11:24:21.460 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.4299999999999997e-06 2024-09-13T11:24:21.460 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:21.461 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:21.461 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:24:21.461 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:15.139427+0000", 2024-09-13T11:24:21.461 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000116258 2024-09-13T11:24:21.461 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:21.461 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:21.461 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:24:21.461 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:15.139458+0000", 2024-09-13T11:24:21.461 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.0892000000000002e-05 2024-09-13T11:24:21.461 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:21.461 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:21.461 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:24:21.462 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:15.139477+0000", 2024-09-13T11:24:21.462 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.9544000000000001e-05 2024-09-13T11:24:21.462 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:21.462 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:21.462 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:24:21.462 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:15.139609+0000", 2024-09-13T11:24:21.462 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00013150799999999999 2024-09-13T11:24:21.462 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:21.462 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:21.462 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:24:21.462 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:15.139873+0000", 2024-09-13T11:24:21.462 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000264288 2024-09-13T11:24:21.462 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:21.463 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:21.463 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1", 2024-09-13T11:24:21.463 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:15.140160+0000", 2024-09-13T11:24:21.463 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00028688999999999999 2024-09-13T11:24:21.463 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:21.463 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:21.463 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:24:21.463 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:15.143356+0000", 2024-09-13T11:24:21.463 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0031956179999999999 2024-09-13T11:24:21.463 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:21.463 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:21.463 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:21.463 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:15.255687+0000", 2024-09-13T11:24:21.463 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.11233159099999999 2024-09-13T11:24:21.463 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:21.464 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:21.464 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:24:21.464 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:15.255799+0000", 2024-09-13T11:24:21.464 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000111492 2024-09-13T11:24:21.464 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:21.464 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:21.464 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:24:21.464 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:15.255888+0000", 2024-09-13T11:24:21.464 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.9234999999999995e-05 2024-09-13T11:24:21.464 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:21.464 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:24:21.464 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:21.465 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:21.465 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:21.465 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop_reply(client.4665.0:98229 3.19 e113/84 ondisk, result = 0)", 2024-09-13T11:24:21.465 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:24:15.144168+0000", 2024-09-13T11:24:21.465 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 6.2424360639999996, 2024-09-13T11:24:21.465 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.111733939, 2024-09-13T11:24:21.465 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:24:21.465 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:24:21.465 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-09-13T11:24:21.465 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:24:21.465 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:21.465 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:24:21.465 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:15.144168+0000", 2024-09-13T11:24:21.466 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:21.466 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:21.466 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:21.466 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:24:21.466 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:15.144168+0000", 2024-09-13T11:24:21.466 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:21.466 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:21.466 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:21.466 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:24:21.466 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:15.144170+0000", 2024-09-13T11:24:21.466 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.272e-06 2024-09-13T11:24:21.466 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:21.467 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:21.467 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:24:21.467 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:15.144177+0000", 2024-09-13T11:24:21.467 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.0879999999999997e-06 2024-09-13T11:24:21.467 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:21.467 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:21.467 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:24:21.467 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:15.255542+0000", 2024-09-13T11:24:21.467 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.111364674 2024-09-13T11:24:21.467 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:21.467 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:21.467 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:24:21.468 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:15.255553+0000", 2024-09-13T11:24:21.468 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.1022999999999999e-05 2024-09-13T11:24:21.468 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:21.468 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:21.468 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:24:21.468 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:15.255656+0000", 2024-09-13T11:24:21.468 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000102503 2024-09-13T11:24:21.468 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:21.468 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:21.468 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:24:21.468 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:15.255674+0000", 2024-09-13T11:24:21.468 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.8431e-05 2024-09-13T11:24:21.468 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:21.469 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:21.469 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:24:21.469 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:15.255902+0000", 2024-09-13T11:24:21.469 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00022794800000000001 2024-09-13T11:24:21.469 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:21.469 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:24:21.469 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:21.469 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:21.469 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:21.469 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop(client.4665.0:98582 3.1a e113/105 3:59f9cd75:::benchmark_data_smithi177_31116_object98581:head v 113'4743, mlcod=113'4741)", 2024-09-13T11:24:21.469 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:24:15.432027+0000", 2024-09-13T11:24:21.469 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 5.9545769420000001, 2024-09-13T11:24:21.469 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.196879784, 2024-09-13T11:24:21.470 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:24:21.470 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:24:21.470 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:21.470 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:24:21.470 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:21.470 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:24:21.470 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:15.432027+0000", 2024-09-13T11:24:21.470 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:21.470 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:21.470 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:21.470 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:24:21.471 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:15.432027+0000", 2024-09-13T11:24:21.471 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:21.471 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:21.471 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:21.471 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:24:21.471 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:15.432029+0000", 2024-09-13T11:24:21.471 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.838e-06 2024-09-13T11:24:21.471 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:21.471 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:21.471 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:24:21.471 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:15.432088+0000", 2024-09-13T11:24:21.471 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.8767999999999997e-05 2024-09-13T11:24:21.471 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:21.472 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:21.472 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:24:21.472 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:15.624731+0000", 2024-09-13T11:24:21.472 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.192643221 2024-09-13T11:24:21.472 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:21.472 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:21.472 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:24:21.472 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:15.624746+0000", 2024-09-13T11:24:21.472 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.4705000000000001e-05 2024-09-13T11:24:21.472 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:21.472 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:21.472 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:24:21.473 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:15.624882+0000", 2024-09-13T11:24:21.473 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00013602300000000001 2024-09-13T11:24:21.473 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:21.473 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:21.473 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:24:21.473 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:15.624941+0000", 2024-09-13T11:24:21.473 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.9351000000000001e-05 2024-09-13T11:24:21.473 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:21.473 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:21.473 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:24:21.473 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:15.628836+0000", 2024-09-13T11:24:21.473 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0038955109999999999 2024-09-13T11:24:21.473 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:21.474 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:21.474 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:24:21.474 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:15.628907+0000", 2024-09-13T11:24:21.474 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.0366999999999999e-05 2024-09-13T11:24:21.474 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:21.474 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:24:21.474 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:21.474 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:21.474 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:21.474 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop(client.4665.0:98872 3.1a e113/105 3:5b894ab8:::benchmark_data_smithi177_31116_object98871:head v 113'4746, mlcod=113'4743)", 2024-09-13T11:24:21.474 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:24:15.736076+0000", 2024-09-13T11:24:21.474 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 5.6505283049999999, 2024-09-13T11:24:21.474 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.111852691, 2024-09-13T11:24:21.475 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:24:21.475 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:24:21.475 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:21.475 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:24:21.475 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:21.475 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:24:21.475 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:15.736076+0000", 2024-09-13T11:24:21.475 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:21.475 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:21.475 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:21.475 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:24:21.475 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:15.736076+0000", 2024-09-13T11:24:21.475 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:21.476 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:21.476 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:21.476 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:24:21.476 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:15.736079+0000", 2024-09-13T11:24:21.476 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.2260000000000001e-06 2024-09-13T11:24:21.476 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:21.476 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:21.476 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:24:21.476 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:15.736171+0000", 2024-09-13T11:24:21.476 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.2260000000000001e-05 2024-09-13T11:24:21.476 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:21.477 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:21.477 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:24:21.477 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:15.843102+0000", 2024-09-13T11:24:21.477 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.10693101100000001 2024-09-13T11:24:21.477 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:21.477 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:21.477 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:24:21.477 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:15.843120+0000", 2024-09-13T11:24:21.477 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.7771999999999999e-05 2024-09-13T11:24:21.477 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:21.477 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:21.477 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:24:21.477 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:15.843247+0000", 2024-09-13T11:24:21.478 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00012656000000000001 2024-09-13T11:24:21.478 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:21.478 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:21.478 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:24:21.478 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:15.843295+0000", 2024-09-13T11:24:21.478 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.8516000000000001e-05 2024-09-13T11:24:21.478 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:21.478 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:21.478 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:24:21.478 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:15.847762+0000", 2024-09-13T11:24:21.478 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0044674149999999998 2024-09-13T11:24:21.478 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:21.478 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:21.479 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:24:21.479 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:15.847928+0000", 2024-09-13T11:24:21.479 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00016593099999999999 2024-09-13T11:24:21.479 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:21.479 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:24:21.479 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:21.479 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:21.479 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:21.479 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop(client.4665.0:99105 3.1a e113/105 3:5ac5090c:::benchmark_data_smithi177_31116_object99104:head v 113'4749, mlcod=113'4746)", 2024-09-13T11:24:21.479 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:24:15.963602+0000", 2024-09-13T11:24:21.479 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 5.42300176, 2024-09-13T11:24:21.479 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.23699906300000001, 2024-09-13T11:24:21.479 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:24:21.480 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:24:21.480 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:21.480 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:24:21.480 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:21.480 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:24:21.480 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:15.963602+0000", 2024-09-13T11:24:21.480 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:21.480 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:21.480 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:21.480 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:24:21.480 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:15.963602+0000", 2024-09-13T11:24:21.480 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:21.480 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:21.481 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:21.481 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:24:21.481 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:15.963605+0000", 2024-09-13T11:24:21.481 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.0400000000000001e-06 2024-09-13T11:24:21.481 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:21.481 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:21.481 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:24:21.481 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:15.963718+0000", 2024-09-13T11:24:21.481 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000112796 2024-09-13T11:24:21.481 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:21.481 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:21.481 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:24:21.482 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:16.197994+0000", 2024-09-13T11:24:21.482 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.234276285 2024-09-13T11:24:21.482 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:21.482 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:21.482 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:24:21.482 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:16.198004+0000", 2024-09-13T11:24:21.482 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.5559999999999996e-06 2024-09-13T11:24:21.482 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:21.482 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:21.482 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:24:21.482 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:16.198110+0000", 2024-09-13T11:24:21.482 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00010614 2024-09-13T11:24:21.482 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:21.483 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:21.483 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:24:21.483 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:16.198152+0000", 2024-09-13T11:24:21.483 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.1899e-05 2024-09-13T11:24:21.483 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:21.483 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:21.483 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:24:21.483 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:16.200528+0000", 2024-09-13T11:24:21.483 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00237587 2024-09-13T11:24:21.483 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:21.483 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:21.483 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:24:21.483 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:16.200601+0000", 2024-09-13T11:24:21.484 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.3477000000000001e-05 2024-09-13T11:24:21.484 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:21.484 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:24:21.484 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:21.484 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:21.484 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:24:21.484 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-09-13T11:24:21.485 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_ops_in_flight 2024-09-13T11:24:21.488 INFO:tasks.ceph.osd.14.smithi162.stderr:2024-09-13T11:24:21.484+0000 7f235c0db640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26319) UID: 0 2024-09-13T11:24:21.589 INFO:tasks.ceph.osd.7.smithi177.stderr:2024-09-13T11:24:21.587+0000 7f897b51d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26575) UID: 0 2024-09-13T11:24:21.673 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-09-13T11:24:21.673 INFO:teuthology.orchestra.run.smithi092.stdout: "ops": [ 2024-09-13T11:24:21.673 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:21.674 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:106369 3.6 3:679aa5f6:::benchmark_data_smithi177_31116_object106368:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116)", 2024-09-13T11:24:21.674 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:24:21.662814+0000", 2024-09-13T11:24:21.674 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 0.0100591, 2024-09-13T11:24:21.674 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.010086227999999999, 2024-09-13T11:24:21.674 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:24:21.674 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:24:21.674 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "waiting for sub ops", 2024-09-13T11:24:21.674 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:24:21.674 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:24:21.674 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:21.674 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 106369 2024-09-13T11:24:21.674 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:21.674 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:24:21.675 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:21.675 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:24:21.675 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:21.662814+0000", 2024-09-13T11:24:21.675 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:21.675 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:21.675 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:21.675 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:24:21.675 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:21.662814+0000", 2024-09-13T11:24:21.675 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:21.675 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:21.675 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:21.675 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:24:21.675 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:21.662816+0000", 2024-09-13T11:24:21.676 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.3599999999999999e-06 2024-09-13T11:24:21.676 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:21.676 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:21.676 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:24:21.676 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:21.662855+0000", 2024-09-13T11:24:21.676 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.8719e-05 2024-09-13T11:24:21.676 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:21.676 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:21.676 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:24:21.676 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:21.662873+0000", 2024-09-13T11:24:21.676 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.8057999999999999e-05 2024-09-13T11:24:21.676 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:21.676 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:21.677 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:24:21.677 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:21.662882+0000", 2024-09-13T11:24:21.677 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.3700000000000001e-06 2024-09-13T11:24:21.677 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:21.677 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:21.677 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:24:21.677 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:21.662953+0000", 2024-09-13T11:24:21.677 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.0524000000000004e-05 2024-09-13T11:24:21.677 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:21.677 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:21.677 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:24:21.677 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:21.663064+0000", 2024-09-13T11:24:21.677 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000111314 2024-09-13T11:24:21.677 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:21.678 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:21.678 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 14", 2024-09-13T11:24:21.678 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:21.663163+0000", 2024-09-13T11:24:21.678 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.8443000000000006e-05 2024-09-13T11:24:21.678 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:21.678 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:21.678 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:24:21.678 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:21.666019+0000", 2024-09-13T11:24:21.678 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0028560510000000001 2024-09-13T11:24:21.678 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:21.678 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:24:21.678 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:21.678 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:21.679 INFO:teuthology.orchestra.run.smithi092.stdout: ], 2024-09-13T11:24:21.679 INFO:teuthology.orchestra.run.smithi092.stdout: "num_ops": 1 2024-09-13T11:24:21.679 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-09-13T11:24:21.690 INFO:tasks.ceph.osd.3.smithi177.stderr:2024-09-13T11:24:21.687+0000 7f10adba4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26574) UID: 0 2024-09-13T11:24:21.693 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_blocked_ops 2024-09-13T11:24:21.791 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-13T11:24:21.787+0000 7f6a68775640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26307) UID: 0 2024-09-13T11:24:21.835 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-09-13T11:24:21.835 INFO:teuthology.orchestra.run.smithi092.stdout: "ops": [], 2024-09-13T11:24:21.835 INFO:teuthology.orchestra.run.smithi092.stdout: "complaint_time": 30, 2024-09-13T11:24:21.835 INFO:teuthology.orchestra.run.smithi092.stdout: "num_blocked_ops": 0 2024-09-13T11:24:21.836 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-09-13T11:24:21.856 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_historic_ops 2024-09-13T11:24:21.901 INFO:tasks.ceph.osd.9.smithi092.stderr:2024-09-13T11:24:21.898+0000 7ff5ab3d0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26348) UID: 0 2024-09-13T11:24:21.993 INFO:tasks.ceph.osd.13.smithi092.stderr:2024-09-13T11:24:21.986+0000 7f3e5afea640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26350) UID: 0 2024-09-13T11:24:21.996 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-09-13T11:24:21.996 INFO:teuthology.orchestra.run.smithi092.stdout: "size": 20, 2024-09-13T11:24:21.996 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 600, 2024-09-13T11:24:21.996 INFO:teuthology.orchestra.run.smithi092.stdout: "ops": [ 2024-09-13T11:24:21.996 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:21.996 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:79 3.1 3:841df5b5:::benchmark_data_smithi177_31116_object78:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-09-13T11:24:21.996 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:27.056083+0000", 2024-09-13T11:24:21.996 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 114.937588435, 2024-09-13T11:24:21.997 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.89469730199999997, 2024-09-13T11:24:21.997 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:24:21.997 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:24:21.997 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:21.997 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:24:21.997 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:24:21.997 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:21.997 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 79 2024-09-13T11:24:21.997 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:21.997 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:24:21.997 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:21.997 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:24:21.998 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.056083+0000", 2024-09-13T11:24:21.998 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:21.998 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:21.998 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:21.998 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:24:21.998 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.056083+0000", 2024-09-13T11:24:21.998 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:21.998 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:21.998 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:21.998 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:24:21.998 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.056087+0000", 2024-09-13T11:24:21.998 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.0659999999999997e-06 2024-09-13T11:24:21.998 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:21.998 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:21.999 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:24:21.999 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.056148+0000", 2024-09-13T11:24:21.999 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.1376000000000003e-05 2024-09-13T11:24:21.999 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:21.999 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:21.999 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:24:21.999 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.056180+0000", 2024-09-13T11:24:21.999 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.2370999999999999e-05 2024-09-13T11:24:21.999 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:21.999 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:21.999 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:24:21.999 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.056195+0000", 2024-09-13T11:24:22.000 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.5119e-05 2024-09-13T11:24:22.000 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:22.000 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:22.000 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:24:22.000 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.056304+0000", 2024-09-13T11:24:22.000 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000108365 2024-09-13T11:24:22.000 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:22.000 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:22.000 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:24:22.000 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.056478+0000", 2024-09-13T11:24:22.000 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000174542 2024-09-13T11:24:22.000 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:22.000 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:22.000 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 11", 2024-09-13T11:24:22.001 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.056690+0000", 2024-09-13T11:24:22.001 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00021167500000000001 2024-09-13T11:24:22.001 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:22.001 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:22.001 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:24:22.001 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.058063+0000", 2024-09-13T11:24:22.001 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0013731819999999999 2024-09-13T11:24:22.001 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:22.001 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:22.001 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:22.001 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.950477+0000", 2024-09-13T11:24:22.001 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.89241369400000004 2024-09-13T11:24:22.001 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:22.002 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:22.002 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:24:22.002 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.950652+0000", 2024-09-13T11:24:22.002 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00017526400000000001 2024-09-13T11:24:22.002 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:22.002 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:22.002 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:24:22.002 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.950780+0000", 2024-09-13T11:24:22.002 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00012764800000000001 2024-09-13T11:24:22.002 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:22.002 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:24:22.002 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:22.002 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:22.003 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:22.003 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:7 3.b 3:de78393b:::benchmark_data_smithi177_31116_object6:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T11:24:22.003 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:48.313952+0000", 2024-09-13T11:24:22.003 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 93.679718686000001, 2024-09-13T11:24:22.003 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.86421288600000001, 2024-09-13T11:24:22.003 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:24:22.003 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:24:22.003 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:22.003 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:24:22.003 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:24:22.003 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:22.003 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 7 2024-09-13T11:24:22.003 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:22.003 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:24:22.004 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:22.004 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:24:22.004 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.313952+0000", 2024-09-13T11:24:22.004 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:22.004 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:22.004 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:22.004 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:24:22.004 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.313952+0000", 2024-09-13T11:24:22.004 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:22.004 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:22.004 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:22.004 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:24:22.004 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.313957+0000", 2024-09-13T11:24:22.005 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.3599999999999998e-06 2024-09-13T11:24:22.005 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:22.005 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:22.005 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:24:22.005 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314043+0000", 2024-09-13T11:24:22.005 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.6766000000000002e-05 2024-09-13T11:24:22.005 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:22.005 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:22.005 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:24:22.005 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314065+0000", 2024-09-13T11:24:22.005 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.1416e-05 2024-09-13T11:24:22.005 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:22.005 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:22.006 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:24:22.006 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314080+0000", 2024-09-13T11:24:22.006 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.5519000000000002e-05 2024-09-13T11:24:22.006 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:22.006 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:22.006 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:24:22.006 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315478+0000", 2024-09-13T11:24:22.006 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.001397765 2024-09-13T11:24:22.006 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:22.006 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:22.006 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-09-13T11:24:22.006 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315507+0000", 2024-09-13T11:24:22.006 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.9128999999999998e-05 2024-09-13T11:24:22.006 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:22.007 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:22.007 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:24:22.007 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.167978+0000", 2024-09-13T11:24:22.007 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.85247068400000003 2024-09-13T11:24:22.007 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:22.007 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:22.007 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:24:22.007 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.168307+0000", 2024-09-13T11:24:22.007 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00032886699999999999 2024-09-13T11:24:22.007 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:22.007 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:22.007 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1", 2024-09-13T11:24:22.007 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.168647+0000", 2024-09-13T11:24:22.007 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00033998399999999999 2024-09-13T11:24:22.008 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:22.008 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:22.008 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:22.008 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.178011+0000", 2024-09-13T11:24:22.008 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0093639169999999994 2024-09-13T11:24:22.008 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:22.008 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:22.008 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:24:22.008 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.178026+0000", 2024-09-13T11:24:22.008 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.5274999999999999e-05 2024-09-13T11:24:22.008 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:22.008 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:22.008 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:24:22.009 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.178080+0000", 2024-09-13T11:24:22.009 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.3601999999999999e-05 2024-09-13T11:24:22.009 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:22.009 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:22.009 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:24:22.009 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.178165+0000", 2024-09-13T11:24:22.009 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.5601999999999998e-05 2024-09-13T11:24:22.009 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:22.009 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:24:22.009 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:22.009 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:22.009 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:22.009 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:9 3.b 3:dafadff6:::benchmark_data_smithi177_31116_object8:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T11:24:22.010 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:48.314114+0000", 2024-09-13T11:24:22.010 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 93.679556872000006, 2024-09-13T11:24:22.010 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.86613615899999996, 2024-09-13T11:24:22.010 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:24:22.010 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:24:22.010 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:22.010 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:24:22.010 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:24:22.010 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:22.010 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 9 2024-09-13T11:24:22.010 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:22.010 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:24:22.010 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:22.011 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:24:22.011 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314114+0000", 2024-09-13T11:24:22.011 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:22.011 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:22.011 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:22.011 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:24:22.011 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314114+0000", 2024-09-13T11:24:22.011 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:22.011 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:22.011 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:22.011 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:24:22.011 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314117+0000", 2024-09-13T11:24:22.011 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.3860000000000001e-06 2024-09-13T11:24:22.011 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:22.011 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:22.012 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:24:22.012 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314215+0000", 2024-09-13T11:24:22.012 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.8900000000000005e-05 2024-09-13T11:24:22.012 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:22.012 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:22.012 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:24:22.012 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314229+0000", 2024-09-13T11:24:22.012 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.3444999999999999e-05 2024-09-13T11:24:22.012 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:22.012 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:22.012 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:24:22.012 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314239+0000", 2024-09-13T11:24:22.012 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.8470000000000008e-06 2024-09-13T11:24:22.013 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:22.013 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:22.013 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:24:22.013 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315648+0000", 2024-09-13T11:24:22.013 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.001409291 2024-09-13T11:24:22.013 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:22.013 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:22.013 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-09-13T11:24:22.013 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315671+0000", 2024-09-13T11:24:22.013 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.3169000000000001e-05 2024-09-13T11:24:22.013 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:22.013 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:22.013 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:24:22.013 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.176036+0000", 2024-09-13T11:24:22.014 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.86036497499999998 2024-09-13T11:24:22.014 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:22.014 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:22.014 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:24:22.014 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.176189+0000", 2024-09-13T11:24:22.014 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00015271100000000001 2024-09-13T11:24:22.014 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:22.014 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:22.014 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1", 2024-09-13T11:24:22.014 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.176339+0000", 2024-09-13T11:24:22.014 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00015057499999999999 2024-09-13T11:24:22.014 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:22.014 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:22.015 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:22.015 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.180148+0000", 2024-09-13T11:24:22.015 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.003808891 2024-09-13T11:24:22.015 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:22.015 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:22.015 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:24:22.015 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.180168+0000", 2024-09-13T11:24:22.015 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.9507000000000001e-05 2024-09-13T11:24:22.015 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:22.015 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:22.015 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:24:22.015 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.180213+0000", 2024-09-13T11:24:22.015 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.5504999999999999e-05 2024-09-13T11:24:22.016 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:22.016 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:22.016 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:24:22.016 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.180250+0000", 2024-09-13T11:24:22.016 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.6956999999999999e-05 2024-09-13T11:24:22.016 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:22.016 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:24:22.016 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:22.016 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:22.016 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:22.016 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:10 3.b 3:dbe028a8:::benchmark_data_smithi177_31116_object9:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T11:24:22.016 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:48.314259+0000", 2024-09-13T11:24:22.016 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 93.679411741999999, 2024-09-13T11:24:22.017 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.86637313599999999, 2024-09-13T11:24:22.017 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:24:22.017 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:24:22.017 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:22.017 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:24:22.017 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:24:22.017 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:22.017 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 10 2024-09-13T11:24:22.017 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:22.017 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:24:22.017 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:22.017 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:24:22.017 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314259+0000", 2024-09-13T11:24:22.018 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:22.018 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:22.018 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:22.018 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:24:22.018 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314259+0000", 2024-09-13T11:24:22.018 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:22.018 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:22.018 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:22.018 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:24:22.018 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314261+0000", 2024-09-13T11:24:22.018 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.1500000000000002e-06 2024-09-13T11:24:22.018 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:22.018 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:22.019 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:24:22.019 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314349+0000", 2024-09-13T11:24:22.019 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.8070000000000002e-05 2024-09-13T11:24:22.019 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:22.019 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:22.019 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:24:22.019 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314365+0000", 2024-09-13T11:24:22.019 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.5835999999999999e-05 2024-09-13T11:24:22.019 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:22.019 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:22.019 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:24:22.019 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314377+0000", 2024-09-13T11:24:22.019 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.1589e-05 2024-09-13T11:24:22.019 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:22.019 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:22.020 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:24:22.020 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315788+0000", 2024-09-13T11:24:22.020 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0014109579999999999 2024-09-13T11:24:22.020 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:22.020 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:22.020 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-09-13T11:24:22.020 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315809+0000", 2024-09-13T11:24:22.020 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.0985999999999999e-05 2024-09-13T11:24:22.020 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:22.020 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:22.020 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:24:22.020 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.176822+0000", 2024-09-13T11:24:22.020 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.86101310799999997 2024-09-13T11:24:22.021 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:22.021 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:22.021 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:24:22.021 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.177021+0000", 2024-09-13T11:24:22.021 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00019911199999999999 2024-09-13T11:24:22.021 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:22.021 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:22.021 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1", 2024-09-13T11:24:22.021 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.177226+0000", 2024-09-13T11:24:22.021 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00020516600000000001 2024-09-13T11:24:22.021 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:22.021 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:22.021 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:24:22.021 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.180264+0000", 2024-09-13T11:24:22.022 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0030381739999999998 2024-09-13T11:24:22.022 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:22.022 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:22.022 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:22.022 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.180387+0000", 2024-09-13T11:24:22.022 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00012257699999999999 2024-09-13T11:24:22.022 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:22.022 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:22.023 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:24:22.023 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.180518+0000", 2024-09-13T11:24:22.023 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00013135800000000001 2024-09-13T11:24:22.023 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:22.023 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:22.023 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:24:22.023 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.180632+0000", 2024-09-13T11:24:22.023 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000114052 2024-09-13T11:24:22.023 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:22.023 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:24:22.023 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:22.023 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:22.023 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:22.024 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:39 3.b 3:d91854b3:::benchmark_data_smithi177_31116_object38:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T11:24:22.024 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:48.314393+0000", 2024-09-13T11:24:22.024 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 93.679278199999999, 2024-09-13T11:24:22.024 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.86850121700000005, 2024-09-13T11:24:22.024 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:24:22.024 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:24:22.024 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:22.024 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:24:22.024 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:24:22.024 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:22.024 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 39 2024-09-13T11:24:22.024 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:22.024 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:24:22.025 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:22.025 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:24:22.025 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314393+0000", 2024-09-13T11:24:22.025 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:22.025 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:22.025 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:22.025 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:24:22.025 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314393+0000", 2024-09-13T11:24:22.025 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:22.025 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:22.025 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:22.025 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:24:22.025 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314394+0000", 2024-09-13T11:24:22.026 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.513e-06 2024-09-13T11:24:22.026 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:22.026 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:22.026 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:24:22.026 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314463+0000", 2024-09-13T11:24:22.026 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.8261000000000003e-05 2024-09-13T11:24:22.026 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:22.026 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:22.026 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:24:22.026 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314477+0000", 2024-09-13T11:24:22.026 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.4619e-05 2024-09-13T11:24:22.026 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:22.026 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:22.027 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:24:22.027 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314485+0000", 2024-09-13T11:24:22.027 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.1699999999999997e-06 2024-09-13T11:24:22.027 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:22.027 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:22.027 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:24:22.027 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315917+0000", 2024-09-13T11:24:22.027 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0014316089999999999 2024-09-13T11:24:22.027 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:22.027 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:22.027 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-09-13T11:24:22.027 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315933+0000", 2024-09-13T11:24:22.027 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.5999999999999999e-05 2024-09-13T11:24:22.027 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:22.028 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:22.028 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:24:22.028 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.178205+0000", 2024-09-13T11:24:22.028 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.86227247100000004 2024-09-13T11:24:22.028 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:22.028 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:22.028 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:24:22.028 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.178316+0000", 2024-09-13T11:24:22.028 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000110788 2024-09-13T11:24:22.028 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:22.028 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:22.028 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1", 2024-09-13T11:24:22.028 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.178448+0000", 2024-09-13T11:24:22.028 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000132056 2024-09-13T11:24:22.029 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:22.029 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:22.029 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:24:22.029 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.182721+0000", 2024-09-13T11:24:22.029 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0042729780000000002 2024-09-13T11:24:22.029 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:22.029 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:22.029 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:22.029 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.182801+0000", 2024-09-13T11:24:22.029 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.0209000000000004e-05 2024-09-13T11:24:22.029 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:22.029 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:22.029 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:24:22.030 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.182855+0000", 2024-09-13T11:24:22.030 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.3749000000000003e-05 2024-09-13T11:24:22.030 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:22.030 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:22.030 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:24:22.030 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.182894+0000", 2024-09-13T11:24:22.030 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.8794000000000002e-05 2024-09-13T11:24:22.030 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:22.030 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:24:22.030 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:22.030 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:22.030 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:22.030 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:50 3.b 3:d1c144b8:::benchmark_data_smithi177_31116_object49:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T11:24:22.030 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:48.314498+0000", 2024-09-13T11:24:22.031 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 93.679172785000006, 2024-09-13T11:24:22.031 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.86878673299999998, 2024-09-13T11:24:22.031 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:24:22.031 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:24:22.031 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:22.031 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:24:22.031 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:24:22.031 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:22.031 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 50 2024-09-13T11:24:22.031 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:22.031 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:24:22.031 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:22.031 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:24:22.031 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314498+0000", 2024-09-13T11:24:22.032 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:22.032 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:22.032 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:22.032 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:24:22.032 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314498+0000", 2024-09-13T11:24:22.032 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:22.032 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:22.032 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:22.032 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:24:22.032 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314499+0000", 2024-09-13T11:24:22.032 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.1969999999999999e-06 2024-09-13T11:24:22.032 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:22.033 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:22.033 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:24:22.033 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314565+0000", 2024-09-13T11:24:22.033 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.5628000000000003e-05 2024-09-13T11:24:22.033 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:22.033 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:22.033 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:24:22.033 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314579+0000", 2024-09-13T11:24:22.033 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.4222e-05 2024-09-13T11:24:22.033 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:22.033 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:22.033 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:24:22.033 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314590+0000", 2024-09-13T11:24:22.033 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.1240000000000001e-05 2024-09-13T11:24:22.033 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:22.034 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:22.034 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:24:22.034 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316060+0000", 2024-09-13T11:24:22.034 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.001469515 2024-09-13T11:24:22.034 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:22.034 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:22.034 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-09-13T11:24:22.034 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316076+0000", 2024-09-13T11:24:22.034 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.5608999999999999e-05 2024-09-13T11:24:22.034 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:22.034 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:22.034 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:24:22.034 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.178857+0000", 2024-09-13T11:24:22.034 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.86278099900000005 2024-09-13T11:24:22.035 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:22.035 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:22.035 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:24:22.035 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.178961+0000", 2024-09-13T11:24:22.035 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000104664 2024-09-13T11:24:22.035 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:22.035 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:22.035 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1", 2024-09-13T11:24:22.035 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.179123+0000", 2024-09-13T11:24:22.035 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000161738 2024-09-13T11:24:22.035 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:22.035 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:22.035 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:24:22.035 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.182755+0000", 2024-09-13T11:24:22.036 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0036323200000000001 2024-09-13T11:24:22.036 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:22.036 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:22.036 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:22.036 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.183023+0000", 2024-09-13T11:24:22.036 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00026749100000000002 2024-09-13T11:24:22.036 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:22.036 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:22.036 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:24:22.036 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.183168+0000", 2024-09-13T11:24:22.036 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00014499099999999999 2024-09-13T11:24:22.036 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:22.036 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:22.036 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:24:22.037 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.183285+0000", 2024-09-13T11:24:22.037 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000117119 2024-09-13T11:24:22.037 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:22.037 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:24:22.037 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:22.037 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:22.037 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:22.037 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:60 3.b 3:d67218b0:::benchmark_data_smithi177_31116_object59:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T11:24:22.037 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:48.314603+0000", 2024-09-13T11:24:22.037 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 93.679068221999998, 2024-09-13T11:24:22.037 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.87106201100000002, 2024-09-13T11:24:22.037 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:24:22.037 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:24:22.037 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:22.038 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:24:22.038 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:24:22.038 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:22.038 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 60 2024-09-13T11:24:22.038 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:22.038 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:24:22.038 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:22.038 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:24:22.038 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314603+0000", 2024-09-13T11:24:22.038 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:22.038 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:22.038 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:22.038 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:24:22.038 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314603+0000", 2024-09-13T11:24:22.038 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:22.039 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:22.039 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:22.039 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:24:22.039 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314604+0000", 2024-09-13T11:24:22.039 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.2359999999999999e-06 2024-09-13T11:24:22.039 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:22.039 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:22.039 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:24:22.039 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314667+0000", 2024-09-13T11:24:22.039 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.3077999999999995e-05 2024-09-13T11:24:22.039 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:22.039 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:22.039 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:24:22.040 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314699+0000", 2024-09-13T11:24:22.040 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.1965999999999999e-05 2024-09-13T11:24:22.040 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:22.040 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:22.040 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:24:22.040 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314710+0000", 2024-09-13T11:24:22.040 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.1148999999999999e-05 2024-09-13T11:24:22.040 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:22.040 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:22.040 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:24:22.040 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316291+0000", 2024-09-13T11:24:22.040 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0015804440000000001 2024-09-13T11:24:22.040 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:22.041 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:22.041 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-09-13T11:24:22.041 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316310+0000", 2024-09-13T11:24:22.041 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.9799000000000001e-05 2024-09-13T11:24:22.041 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:22.041 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:22.041 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:24:22.041 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.179555+0000", 2024-09-13T11:24:22.041 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.86324469500000001 2024-09-13T11:24:22.041 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:22.041 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:22.041 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:24:22.041 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.179666+0000", 2024-09-13T11:24:22.041 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000110443 2024-09-13T11:24:22.041 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:22.042 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:22.042 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1", 2024-09-13T11:24:22.042 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.179791+0000", 2024-09-13T11:24:22.042 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00012560499999999999 2024-09-13T11:24:22.042 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:22.042 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:22.042 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:24:22.042 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.182761+0000", 2024-09-13T11:24:22.042 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0029697019999999998 2024-09-13T11:24:22.042 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:22.042 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:22.042 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:22.042 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.185572+0000", 2024-09-13T11:24:22.042 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.002811512 2024-09-13T11:24:22.043 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:22.043 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:22.043 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:24:22.043 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.185625+0000", 2024-09-13T11:24:22.043 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.2716000000000001e-05 2024-09-13T11:24:22.043 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:22.043 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:22.043 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:24:22.043 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.185665+0000", 2024-09-13T11:24:22.043 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.9666000000000003e-05 2024-09-13T11:24:22.043 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:22.043 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:24:22.043 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:22.043 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:22.044 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:22.044 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:62 3.b 3:d20ae5a8:::benchmark_data_smithi177_31116_object61:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T11:24:22.044 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:48.314723+0000", 2024-09-13T11:24:22.044 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 93.678947679999993, 2024-09-13T11:24:22.044 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.87338985099999999, 2024-09-13T11:24:22.044 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:24:22.044 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:24:22.044 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:22.044 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:24:22.044 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:24:22.044 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:22.044 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 62 2024-09-13T11:24:22.044 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:22.045 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:24:22.045 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:22.045 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:24:22.045 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314723+0000", 2024-09-13T11:24:22.045 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:22.045 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:22.045 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:22.045 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:24:22.045 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314723+0000", 2024-09-13T11:24:22.045 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:22.045 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:22.045 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:22.046 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:24:22.046 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314725+0000", 2024-09-13T11:24:22.046 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.3069999999999999e-06 2024-09-13T11:24:22.046 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:22.046 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:22.046 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:24:22.046 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314797+0000", 2024-09-13T11:24:22.046 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.2652999999999997e-05 2024-09-13T11:24:22.046 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:22.046 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:22.046 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:24:22.046 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314809+0000", 2024-09-13T11:24:22.046 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.1496e-05 2024-09-13T11:24:22.046 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:22.046 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:22.047 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:24:22.047 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314816+0000", 2024-09-13T11:24:22.047 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.6639999999999998e-06 2024-09-13T11:24:22.047 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:22.047 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:22.047 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:24:22.047 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316424+0000", 2024-09-13T11:24:22.047 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0016077559999999999 2024-09-13T11:24:22.047 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:22.047 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:22.047 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-09-13T11:24:22.047 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316441+0000", 2024-09-13T11:24:22.047 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.7305000000000001e-05 2024-09-13T11:24:22.047 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:22.048 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:22.048 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:24:22.048 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.180688+0000", 2024-09-13T11:24:22.048 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.86424692800000003 2024-09-13T11:24:22.048 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:22.048 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:22.048 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:24:22.048 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.180807+0000", 2024-09-13T11:24:22.048 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000118977 2024-09-13T11:24:22.048 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:22.048 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:22.048 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1", 2024-09-13T11:24:22.048 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.180938+0000", 2024-09-13T11:24:22.048 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00013049700000000001 2024-09-13T11:24:22.049 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:22.049 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:22.049 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:24:22.049 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.185680+0000", 2024-09-13T11:24:22.049 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0047417029999999999 2024-09-13T11:24:22.049 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:22.049 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:22.049 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:22.049 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.187856+0000", 2024-09-13T11:24:22.049 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0021760550000000001 2024-09-13T11:24:22.049 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:22.049 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:22.049 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:24:22.049 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.188007+0000", 2024-09-13T11:24:22.049 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00015185099999999999 2024-09-13T11:24:22.050 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:22.050 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:22.050 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:24:22.050 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.188113+0000", 2024-09-13T11:24:22.050 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000105659 2024-09-13T11:24:22.050 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:22.050 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:24:22.050 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:22.050 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:22.050 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:22.050 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:73 3.b 3:d0060d5f:::benchmark_data_smithi177_31116_object72:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T11:24:22.050 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:48.314829+0000", 2024-09-13T11:24:22.050 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 93.678842330999998, 2024-09-13T11:24:22.050 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.87364757199999998, 2024-09-13T11:24:22.051 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:24:22.051 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:24:22.051 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:22.051 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:24:22.051 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:24:22.051 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:22.051 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 73 2024-09-13T11:24:22.051 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:22.051 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:24:22.051 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:22.051 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:24:22.051 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314829+0000", 2024-09-13T11:24:22.051 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:22.051 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:22.051 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:22.052 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:24:22.052 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314829+0000", 2024-09-13T11:24:22.052 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:22.052 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:22.052 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:22.052 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:24:22.052 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314830+0000", 2024-09-13T11:24:22.052 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.243e-06 2024-09-13T11:24:22.052 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:22.052 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:22.053 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:24:22.053 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314965+0000", 2024-09-13T11:24:22.053 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00013470800000000001 2024-09-13T11:24:22.053 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:22.053 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:22.053 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:24:22.053 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314979+0000", 2024-09-13T11:24:22.053 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.4375999999999999e-05 2024-09-13T11:24:22.053 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:22.053 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:22.053 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:24:22.053 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314988+0000", 2024-09-13T11:24:22.054 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.8570000000000001e-06 2024-09-13T11:24:22.054 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:22.054 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:22.054 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:24:22.054 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316545+0000", 2024-09-13T11:24:22.054 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.001556878 2024-09-13T11:24:22.054 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:22.054 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:22.054 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-09-13T11:24:22.054 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316556+0000", 2024-09-13T11:24:22.054 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.1453e-05 2024-09-13T11:24:22.054 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:22.054 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:22.055 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:24:22.055 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.182124+0000", 2024-09-13T11:24:22.055 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.86556744200000002 2024-09-13T11:24:22.055 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:22.055 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:22.055 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:24:22.055 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.182246+0000", 2024-09-13T11:24:22.055 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00012210500000000001 2024-09-13T11:24:22.055 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:22.055 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:22.055 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1", 2024-09-13T11:24:22.055 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.182375+0000", 2024-09-13T11:24:22.055 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00012920999999999999 2024-09-13T11:24:22.055 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:22.056 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:22.056 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:24:22.056 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.185684+0000", 2024-09-13T11:24:22.056 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0033093969999999999 2024-09-13T11:24:22.056 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:22.056 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:22.056 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:22.056 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.188244+0000", 2024-09-13T11:24:22.056 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0025601550000000002 2024-09-13T11:24:22.056 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:22.056 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:22.056 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:24:22.056 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.188379+0000", 2024-09-13T11:24:22.057 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00013463400000000001 2024-09-13T11:24:22.057 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:22.057 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:22.057 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:24:22.057 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.188476+0000", 2024-09-13T11:24:22.057 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.7114000000000006e-05 2024-09-13T11:24:22.057 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:22.057 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:24:22.057 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:22.057 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:22.057 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:22.057 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:81 3.b 3:d705fd70:::benchmark_data_smithi177_31116_object80:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T11:24:22.057 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:48.315000+0000", 2024-09-13T11:24:22.058 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 93.678670494000002, 2024-09-13T11:24:22.058 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.87561418300000005, 2024-09-13T11:24:22.058 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:24:22.058 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:24:22.058 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:22.058 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:24:22.058 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:24:22.058 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:22.058 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 81 2024-09-13T11:24:22.058 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:22.058 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:24:22.058 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:22.058 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:24:22.059 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315000+0000", 2024-09-13T11:24:22.059 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:22.059 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:22.059 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:22.059 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:24:22.059 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315000+0000", 2024-09-13T11:24:22.059 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:22.059 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:22.059 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:22.059 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:24:22.059 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315002+0000", 2024-09-13T11:24:22.059 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.3799999999999999e-06 2024-09-13T11:24:22.059 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:22.059 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:22.060 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:24:22.060 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315119+0000", 2024-09-13T11:24:22.060 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000116925 2024-09-13T11:24:22.060 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:22.060 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:22.060 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:24:22.060 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315133+0000", 2024-09-13T11:24:22.060 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.3971999999999999e-05 2024-09-13T11:24:22.060 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:22.060 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:22.060 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:24:22.060 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315142+0000", 2024-09-13T11:24:22.060 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.4399999999999994e-06 2024-09-13T11:24:22.061 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:22.061 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:22.061 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:24:22.061 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316652+0000", 2024-09-13T11:24:22.061 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.001509907 2024-09-13T11:24:22.061 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:22.061 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:22.061 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-09-13T11:24:22.061 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316668+0000", 2024-09-13T11:24:22.061 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.5945999999999999e-05 2024-09-13T11:24:22.061 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:22.061 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:22.061 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:24:22.061 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.183405+0000", 2024-09-13T11:24:22.062 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.866737384 2024-09-13T11:24:22.062 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:22.062 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:22.062 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:24:22.062 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.183701+0000", 2024-09-13T11:24:22.062 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00029511099999999997 2024-09-13T11:24:22.062 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:22.062 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:22.062 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1", 2024-09-13T11:24:22.062 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.184078+0000", 2024-09-13T11:24:22.062 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00037741000000000001 2024-09-13T11:24:22.062 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:22.062 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:22.062 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:24:22.063 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.188504+0000", 2024-09-13T11:24:22.063 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0044260280000000003 2024-09-13T11:24:22.063 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:22.063 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:22.063 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:22.063 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.190562+0000", 2024-09-13T11:24:22.063 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0020580099999999999 2024-09-13T11:24:22.063 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:22.063 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:22.063 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:24:22.063 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.190592+0000", 2024-09-13T11:24:22.063 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.0057999999999999e-05 2024-09-13T11:24:22.063 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:22.063 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:22.064 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:24:22.064 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.190615+0000", 2024-09-13T11:24:22.064 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.2612e-05 2024-09-13T11:24:22.064 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:22.064 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:24:22.064 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:22.064 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:22.064 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:22.064 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:87 3.b 3:dd7de0ad:::benchmark_data_smithi177_31116_object86:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T11:24:22.064 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:48.315155+0000", 2024-09-13T11:24:22.064 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 93.678515696999995, 2024-09-13T11:24:22.064 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.87555916099999997, 2024-09-13T11:24:22.064 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:24:22.065 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:24:22.065 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:22.065 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:24:22.065 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:24:22.065 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:22.065 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 87 2024-09-13T11:24:22.065 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:22.065 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:24:22.065 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:22.065 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:24:22.065 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315155+0000", 2024-09-13T11:24:22.065 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:22.065 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:22.065 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:22.066 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:24:22.066 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315155+0000", 2024-09-13T11:24:22.066 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:22.066 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:22.066 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:22.066 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:24:22.066 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315157+0000", 2024-09-13T11:24:22.066 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.3829999999999999e-06 2024-09-13T11:24:22.066 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:22.066 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:22.066 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:24:22.066 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315271+0000", 2024-09-13T11:24:22.066 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000114539 2024-09-13T11:24:22.066 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:22.067 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:22.067 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:24:22.067 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315285+0000", 2024-09-13T11:24:22.067 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.4042e-05 2024-09-13T11:24:22.067 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:22.067 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:22.067 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:24:22.067 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315295+0000", 2024-09-13T11:24:22.067 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.2739999999999996e-06 2024-09-13T11:24:22.067 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:22.067 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:22.067 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:24:22.067 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316739+0000", 2024-09-13T11:24:22.067 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0014446960000000001 2024-09-13T11:24:22.068 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:22.068 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:22.068 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-09-13T11:24:22.068 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316747+0000", 2024-09-13T11:24:22.068 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.0790000000000001e-06 2024-09-13T11:24:22.068 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:22.068 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:22.068 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:24:22.068 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.184922+0000", 2024-09-13T11:24:22.068 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.86817446399999998 2024-09-13T11:24:22.068 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:22.068 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:22.068 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:24:22.068 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.185037+0000", 2024-09-13T11:24:22.069 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00011565099999999999 2024-09-13T11:24:22.069 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:22.069 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:22.069 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1", 2024-09-13T11:24:22.069 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.185171+0000", 2024-09-13T11:24:22.069 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00013316900000000001 2024-09-13T11:24:22.069 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:22.069 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:22.069 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:24:22.069 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.188520+0000", 2024-09-13T11:24:22.069 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0033489150000000001 2024-09-13T11:24:22.069 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:22.069 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:22.069 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:22.070 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.190652+0000", 2024-09-13T11:24:22.070 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0021325649999999999 2024-09-13T11:24:22.070 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:22.070 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:22.070 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:24:22.070 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.190686+0000", 2024-09-13T11:24:22.070 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.3827e-05 2024-09-13T11:24:22.070 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:22.070 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:22.070 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:24:22.070 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.190714+0000", 2024-09-13T11:24:22.070 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.8557000000000001e-05 2024-09-13T11:24:22.070 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:22.070 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:24:22.071 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:22.071 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:22.071 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:22.071 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:99 3.b 3:d011b600:::benchmark_data_smithi177_31116_object98:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T11:24:22.071 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:48.315307+0000", 2024-09-13T11:24:22.071 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 93.678363869999998, 2024-09-13T11:24:22.071 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.87551770799999995, 2024-09-13T11:24:22.071 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:24:22.071 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:24:22.071 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:22.071 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:24:22.071 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:24:22.071 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:22.071 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 99 2024-09-13T11:24:22.071 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:22.072 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:24:22.072 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:22.072 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:24:22.072 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315307+0000", 2024-09-13T11:24:22.072 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:22.072 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:22.072 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:22.072 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:24:22.072 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315307+0000", 2024-09-13T11:24:22.072 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:22.072 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:22.072 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:22.072 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:24:22.072 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315308+0000", 2024-09-13T11:24:22.073 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.316e-06 2024-09-13T11:24:22.073 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:22.073 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:22.073 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:24:22.073 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315453+0000", 2024-09-13T11:24:22.073 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00014473200000000001 2024-09-13T11:24:22.073 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:22.073 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:22.073 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:24:22.073 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315471+0000", 2024-09-13T11:24:22.073 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.8019000000000001e-05 2024-09-13T11:24:22.073 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:22.073 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:22.073 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:24:22.074 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315489+0000", 2024-09-13T11:24:22.074 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.7861999999999999e-05 2024-09-13T11:24:22.074 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:22.074 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:22.074 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:24:22.074 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316837+0000", 2024-09-13T11:24:22.074 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.001347787 2024-09-13T11:24:22.074 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:22.074 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:22.074 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-09-13T11:24:22.074 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316858+0000", 2024-09-13T11:24:22.074 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.1103000000000001e-05 2024-09-13T11:24:22.074 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:22.075 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:22.075 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:24:22.075 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.185713+0000", 2024-09-13T11:24:22.075 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.86885540800000005 2024-09-13T11:24:22.075 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:22.075 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:22.075 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:24:22.075 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.185820+0000", 2024-09-13T11:24:22.075 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000107081 2024-09-13T11:24:22.075 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:22.075 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:22.075 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1", 2024-09-13T11:24:22.075 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.185948+0000", 2024-09-13T11:24:22.075 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00012775800000000001 2024-09-13T11:24:22.075 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:22.076 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:22.076 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:24:22.076 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.190527+0000", 2024-09-13T11:24:22.076 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0045783910000000002 2024-09-13T11:24:22.076 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:22.076 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:22.076 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:22.076 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.190759+0000", 2024-09-13T11:24:22.076 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00023266199999999999 2024-09-13T11:24:22.076 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:22.076 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:22.076 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:24:22.076 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.190789+0000", 2024-09-13T11:24:22.077 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.9666999999999999e-05 2024-09-13T11:24:22.077 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:22.077 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:22.077 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:24:22.077 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.190825+0000", 2024-09-13T11:24:22.077 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.5921999999999999e-05 2024-09-13T11:24:22.077 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:22.077 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:24:22.077 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:22.077 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:22.077 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:22.077 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:108 3.b 3:d5c90217:::benchmark_data_smithi177_31116_object107:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T11:24:22.077 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:48.315513+0000", 2024-09-13T11:24:22.077 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 93.678158164999999, 2024-09-13T11:24:22.077 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.87544772500000001, 2024-09-13T11:24:22.078 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:24:22.078 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:24:22.078 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:22.078 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:24:22.078 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:24:22.078 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:22.078 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 108 2024-09-13T11:24:22.078 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:22.078 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:24:22.078 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:22.078 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:24:22.078 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315513+0000", 2024-09-13T11:24:22.078 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:22.078 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:22.078 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:22.079 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:24:22.079 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315513+0000", 2024-09-13T11:24:22.079 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:22.079 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:22.079 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:22.079 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:24:22.079 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315515+0000", 2024-09-13T11:24:22.079 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.9369999999999998e-06 2024-09-13T11:24:22.079 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:22.079 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:22.079 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:24:22.079 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315644+0000", 2024-09-13T11:24:22.080 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000129011 2024-09-13T11:24:22.080 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:22.080 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:22.080 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:24:22.080 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315657+0000", 2024-09-13T11:24:22.080 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.3689999999999999e-05 2024-09-13T11:24:22.080 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:22.080 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:22.080 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:24:22.080 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315668+0000", 2024-09-13T11:24:22.080 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0589e-05 2024-09-13T11:24:22.080 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:22.080 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:22.080 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:24:22.081 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316925+0000", 2024-09-13T11:24:22.081 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0012568340000000001 2024-09-13T11:24:22.081 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:22.081 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:22.081 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-09-13T11:24:22.081 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316932+0000", 2024-09-13T11:24:22.081 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.2230000000000001e-06 2024-09-13T11:24:22.081 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:22.081 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:22.081 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:24:22.081 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.186619+0000", 2024-09-13T11:24:22.081 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.86968679599999998 2024-09-13T11:24:22.082 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:22.082 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:22.082 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:24:22.082 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.186914+0000", 2024-09-13T11:24:22.082 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00029528900000000001 2024-09-13T11:24:22.082 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:22.082 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:22.082 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1", 2024-09-13T11:24:22.082 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.187217+0000", 2024-09-13T11:24:22.082 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00030232099999999998 2024-09-13T11:24:22.082 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:22.082 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:22.082 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:24:22.083 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.190532+0000", 2024-09-13T11:24:22.083 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.003315391 2024-09-13T11:24:22.083 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:22.083 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:22.083 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:22.083 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.190875+0000", 2024-09-13T11:24:22.083 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00034326300000000001 2024-09-13T11:24:22.083 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:22.083 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:22.083 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:24:22.083 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.190925+0000", 2024-09-13T11:24:22.083 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.9429999999999999e-05 2024-09-13T11:24:22.084 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:22.084 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:22.084 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:24:22.084 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.190961+0000", 2024-09-13T11:24:22.084 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.5951000000000002e-05 2024-09-13T11:24:22.084 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:22.084 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:24:22.084 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:22.084 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:22.084 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:22.084 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:172 3.b 3:dbec1eeb:::benchmark_data_smithi177_31116_object171:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T11:24:22.084 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:48.315684+0000", 2024-09-13T11:24:22.084 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 93.677987008000002, 2024-09-13T11:24:22.084 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.87615312999999995, 2024-09-13T11:24:22.085 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:24:22.085 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:24:22.085 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:22.085 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:24:22.085 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:24:22.085 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:22.085 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 172 2024-09-13T11:24:22.085 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:22.085 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:24:22.085 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:22.085 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:24:22.085 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315684+0000", 2024-09-13T11:24:22.086 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:22.086 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:22.086 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:22.086 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:24:22.086 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315684+0000", 2024-09-13T11:24:22.086 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:22.086 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:22.086 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:22.086 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:24:22.086 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315686+0000", 2024-09-13T11:24:22.086 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.886e-06 2024-09-13T11:24:22.086 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:22.086 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:22.086 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:24:22.087 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315820+0000", 2024-09-13T11:24:22.087 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00013388500000000001 2024-09-13T11:24:22.087 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:22.087 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:22.087 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:24:22.087 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315837+0000", 2024-09-13T11:24:22.087 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.6756e-05 2024-09-13T11:24:22.087 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:22.087 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:22.087 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:24:22.087 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315851+0000", 2024-09-13T11:24:22.087 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.4566000000000001e-05 2024-09-13T11:24:22.087 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:22.087 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:22.088 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:24:22.088 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.317021+0000", 2024-09-13T11:24:22.088 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0011701769999999999 2024-09-13T11:24:22.088 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:22.088 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:22.088 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-09-13T11:24:22.088 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.317037+0000", 2024-09-13T11:24:22.088 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.609e-05 2024-09-13T11:24:22.088 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:22.088 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:22.088 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:24:22.088 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.188634+0000", 2024-09-13T11:24:22.088 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.87159711900000003 2024-09-13T11:24:22.088 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:22.089 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:22.089 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:24:22.089 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.188974+0000", 2024-09-13T11:24:22.089 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00033922299999999999 2024-09-13T11:24:22.089 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:22.089 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:22.089 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1", 2024-09-13T11:24:22.089 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.189510+0000", 2024-09-13T11:24:22.089 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00053618999999999997 2024-09-13T11:24:22.089 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:22.089 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:22.090 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:22.090 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.191027+0000", 2024-09-13T11:24:22.090 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0015168390000000001 2024-09-13T11:24:22.090 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:22.090 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:22.090 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:24:22.090 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.191766+0000", 2024-09-13T11:24:22.090 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000739064 2024-09-13T11:24:22.091 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:22.091 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:22.091 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:24:22.091 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.191810+0000", 2024-09-13T11:24:22.091 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.3755999999999999e-05 2024-09-13T11:24:22.091 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:22.091 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:22.091 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:24:22.091 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.191837+0000", 2024-09-13T11:24:22.091 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.7579000000000001e-05 2024-09-13T11:24:22.091 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:22.091 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:24:22.091 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:22.091 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:22.092 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:22.092 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:177 3.b 3:de89c779:::benchmark_data_smithi177_31116_object176:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T11:24:22.092 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:48.315898+0000", 2024-09-13T11:24:22.092 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 93.677772692999994, 2024-09-13T11:24:22.092 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.87600712599999997, 2024-09-13T11:24:22.092 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:24:22.092 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:24:22.092 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:22.092 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:24:22.092 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:24:22.092 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:22.092 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 177 2024-09-13T11:24:22.093 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:22.093 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:24:22.093 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:22.093 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:24:22.093 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315898+0000", 2024-09-13T11:24:22.093 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:22.093 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:22.093 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:22.093 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:24:22.093 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315898+0000", 2024-09-13T11:24:22.093 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:22.093 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:22.093 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:22.094 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:24:22.094 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315900+0000", 2024-09-13T11:24:22.094 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.4929999999999999e-06 2024-09-13T11:24:22.094 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:22.094 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:22.094 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:24:22.094 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316027+0000", 2024-09-13T11:24:22.094 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000127568 2024-09-13T11:24:22.094 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:22.094 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:22.094 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:24:22.094 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316048+0000", 2024-09-13T11:24:22.094 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.0795999999999999e-05 2024-09-13T11:24:22.095 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:22.095 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:22.095 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:24:22.095 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316061+0000", 2024-09-13T11:24:22.095 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.3283e-05 2024-09-13T11:24:22.095 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:22.095 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:22.095 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:24:22.095 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.317107+0000", 2024-09-13T11:24:22.095 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0010452949999999999 2024-09-13T11:24:22.095 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:22.095 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:22.095 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-09-13T11:24:22.096 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.317114+0000", 2024-09-13T11:24:22.096 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.4569999999999999e-06 2024-09-13T11:24:22.096 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:22.096 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:22.096 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:24:22.096 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.190017+0000", 2024-09-13T11:24:22.096 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.87290258499999995 2024-09-13T11:24:22.096 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:22.096 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:22.096 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:24:22.096 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.190138+0000", 2024-09-13T11:24:22.096 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000121418 2024-09-13T11:24:22.096 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:22.097 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:22.097 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1", 2024-09-13T11:24:22.097 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.190216+0000", 2024-09-13T11:24:22.097 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.7410999999999995e-05 2024-09-13T11:24:22.097 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:22.097 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:22.097 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:22.097 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.191755+0000", 2024-09-13T11:24:22.097 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0015395420000000001 2024-09-13T11:24:22.097 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:22.097 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:22.097 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:24:22.097 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.191843+0000", 2024-09-13T11:24:22.097 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.8022999999999994e-05 2024-09-13T11:24:22.098 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:22.098 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:22.098 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:24:22.098 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.191876+0000", 2024-09-13T11:24:22.098 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.2619000000000001e-05 2024-09-13T11:24:22.098 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:22.098 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:22.098 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:24:22.098 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.191905+0000", 2024-09-13T11:24:22.098 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.9635999999999999e-05 2024-09-13T11:24:22.098 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:22.099 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:24:22.099 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:22.099 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:22.099 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:22.099 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:187 3.b 3:da03c8dd:::benchmark_data_smithi177_31116_object186:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T11:24:22.099 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:48.316083+0000", 2024-09-13T11:24:22.099 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 93.677587736999996, 2024-09-13T11:24:22.099 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.87835499500000003, 2024-09-13T11:24:22.099 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:24:22.099 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:24:22.099 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:22.099 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:24:22.099 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:24:22.100 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:22.100 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 187 2024-09-13T11:24:22.100 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:22.100 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:24:22.100 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:22.100 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:24:22.100 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316083+0000", 2024-09-13T11:24:22.100 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:22.100 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:22.100 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:22.100 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:24:22.100 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316083+0000", 2024-09-13T11:24:22.100 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:22.101 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:22.101 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:22.101 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:24:22.101 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316086+0000", 2024-09-13T11:24:22.101 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.5560000000000002e-06 2024-09-13T11:24:22.101 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:22.101 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:22.101 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:24:22.101 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316199+0000", 2024-09-13T11:24:22.101 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000113522 2024-09-13T11:24:22.101 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:22.101 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:22.101 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:24:22.102 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316215+0000", 2024-09-13T11:24:22.102 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.5265999999999999e-05 2024-09-13T11:24:22.102 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:22.102 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:22.102 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:24:22.102 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316226+0000", 2024-09-13T11:24:22.102 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0967e-05 2024-09-13T11:24:22.102 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:22.102 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:22.102 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:24:22.102 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.317198+0000", 2024-09-13T11:24:22.102 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00097194100000000004 2024-09-13T11:24:22.102 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:22.102 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:22.103 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-09-13T11:24:22.103 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.317213+0000", 2024-09-13T11:24:22.103 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.5472999999999999e-05 2024-09-13T11:24:22.103 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:22.103 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:22.103 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:24:22.103 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.191077+0000", 2024-09-13T11:24:22.103 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.87386435200000001 2024-09-13T11:24:22.103 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:22.103 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:22.103 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:24:22.103 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.191181+0000", 2024-09-13T11:24:22.103 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000103618 2024-09-13T11:24:22.103 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:22.104 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:22.104 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1", 2024-09-13T11:24:22.104 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.191305+0000", 2024-09-13T11:24:22.104 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00012358599999999999 2024-09-13T11:24:22.104 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:22.104 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:22.104 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:22.104 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.194279+0000", 2024-09-13T11:24:22.104 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0029746970000000001 2024-09-13T11:24:22.104 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:22.104 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:22.104 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:24:22.104 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.194336+0000", 2024-09-13T11:24:22.104 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.7244000000000003e-05 2024-09-13T11:24:22.105 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:22.105 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:22.105 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:24:22.105 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.194397+0000", 2024-09-13T11:24:22.105 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.0278000000000001e-05 2024-09-13T11:24:22.105 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:22.105 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:22.105 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:24:22.105 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.194438+0000", 2024-09-13T11:24:22.105 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.1495000000000002e-05 2024-09-13T11:24:22.105 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:22.105 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:24:22.105 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:22.105 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:22.106 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:22.106 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:200 3.b 3:d90039f1:::benchmark_data_smithi177_31116_object199:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T11:24:22.106 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:48.316280+0000", 2024-09-13T11:24:22.106 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 93.677391271999994, 2024-09-13T11:24:22.106 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.87879273000000002, 2024-09-13T11:24:22.106 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:24:22.106 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:24:22.106 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:22.107 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:24:22.107 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:24:22.108 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:22.108 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 200 2024-09-13T11:24:22.108 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:22.108 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:24:22.108 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:22.108 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:24:22.108 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316280+0000", 2024-09-13T11:24:22.108 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:22.108 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:22.108 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:22.108 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:24:22.108 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316280+0000", 2024-09-13T11:24:22.108 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:22.108 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:22.109 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:22.109 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:24:22.109 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316282+0000", 2024-09-13T11:24:22.109 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.497e-06 2024-09-13T11:24:22.109 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:22.109 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:22.109 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:24:22.109 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316411+0000", 2024-09-13T11:24:22.109 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00012924799999999999 2024-09-13T11:24:22.109 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:22.109 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:22.109 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:24:22.109 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316426+0000", 2024-09-13T11:24:22.109 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.4183e-05 2024-09-13T11:24:22.110 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:22.110 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:22.110 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:24:22.110 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316435+0000", 2024-09-13T11:24:22.110 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.0990000000000005e-06 2024-09-13T11:24:22.110 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:22.110 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:22.110 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:24:22.110 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.317250+0000", 2024-09-13T11:24:22.110 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00081501100000000003 2024-09-13T11:24:22.110 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:22.110 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:22.110 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-09-13T11:24:22.110 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.317257+0000", 2024-09-13T11:24:22.111 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.2169999999999997e-06 2024-09-13T11:24:22.111 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:22.111 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:22.111 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:24:22.111 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.191936+0000", 2024-09-13T11:24:22.111 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.87467922099999995 2024-09-13T11:24:22.111 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:22.111 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:22.111 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:24:22.111 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.192056+0000", 2024-09-13T11:24:22.112 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000119638 2024-09-13T11:24:22.112 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:22.112 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:22.112 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1", 2024-09-13T11:24:22.112 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.192184+0000", 2024-09-13T11:24:22.112 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00012813399999999999 2024-09-13T11:24:22.112 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:22.112 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:22.112 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:22.112 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.194324+0000", 2024-09-13T11:24:22.112 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0021399420000000001 2024-09-13T11:24:22.112 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:22.112 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:22.112 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:24:22.113 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.194802+0000", 2024-09-13T11:24:22.113 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00047798799999999999 2024-09-13T11:24:22.113 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:22.113 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:22.113 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:24:22.113 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.194951+0000", 2024-09-13T11:24:22.113 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0001494 2024-09-13T11:24:22.113 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:22.113 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:22.113 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:24:22.113 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.195072+0000", 2024-09-13T11:24:22.113 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00012115199999999999 2024-09-13T11:24:22.113 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:22.113 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:24:22.114 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:22.114 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:22.114 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:22.114 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:25525 3.1 3:8e5634af:::benchmark_data_smithi177_31116_object25524:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e56)", 2024-09-13T11:24:22.114 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:04.981590+0000", 2024-09-13T11:24:22.114 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 77.012080995000005, 2024-09-13T11:24:22.114 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.23642401599999999, 2024-09-13T11:24:22.114 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:24:22.114 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:24:22.114 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:22.114 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:24:22.114 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:24:22.114 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:22.114 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 25525 2024-09-13T11:24:22.115 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:22.115 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:24:22.115 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:22.115 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:24:22.115 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.981590+0000", 2024-09-13T11:24:22.115 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:22.115 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:22.115 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:22.115 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:24:22.115 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.981590+0000", 2024-09-13T11:24:22.115 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:22.115 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:22.115 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:22.115 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:24:22.115 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.981598+0000", 2024-09-13T11:24:22.116 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.4230000000000005e-06 2024-09-13T11:24:22.116 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:22.116 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:22.116 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:24:22.116 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.981652+0000", 2024-09-13T11:24:22.116 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.3226000000000002e-05 2024-09-13T11:24:22.116 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:22.116 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:22.116 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:24:22.116 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.981680+0000", 2024-09-13T11:24:22.116 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.8348999999999999e-05 2024-09-13T11:24:22.116 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:22.116 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:22.116 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:24:22.117 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.981695+0000", 2024-09-13T11:24:22.117 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.5085999999999999e-05 2024-09-13T11:24:22.117 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:22.117 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:22.117 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:24:22.117 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.981809+0000", 2024-09-13T11:24:22.117 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00011382700000000001 2024-09-13T11:24:22.117 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:22.117 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:22.117 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:24:22.117 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.982042+0000", 2024-09-13T11:24:22.117 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000232697 2024-09-13T11:24:22.117 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:22.118 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:22.118 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 11", 2024-09-13T11:24:22.118 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.982261+0000", 2024-09-13T11:24:22.118 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00021918400000000001 2024-09-13T11:24:22.118 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:22.118 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:22.118 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:24:22.118 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.984553+0000", 2024-09-13T11:24:22.118 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0022918460000000002 2024-09-13T11:24:22.118 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:22.118 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:22.118 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:22.118 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:05.217842+0000", 2024-09-13T11:24:22.118 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.233289106 2024-09-13T11:24:22.119 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:22.119 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:22.119 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:24:22.119 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:05.217940+0000", 2024-09-13T11:24:22.119 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.7936999999999994e-05 2024-09-13T11:24:22.119 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:22.119 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:22.119 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:24:22.119 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:05.218014+0000", 2024-09-13T11:24:22.119 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.4334999999999999e-05 2024-09-13T11:24:22.119 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:22.119 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:24:22.119 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:22.119 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:22.120 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:22.120 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "MOSDPGPushReply(3.16 76/70 [PushReplyOp(3:6efa689b:::benchmark_data_smithi177_31116_object24598:head)])", 2024-09-13T11:24:22.120 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:32.699090+0000", 2024-09-13T11:24:22.120 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 49.294580584999999, 2024-09-13T11:24:22.120 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.232999069, 2024-09-13T11:24:22.120 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:24:22.120 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:24:22.120 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "reached pg", 2024-09-13T11:24:22.120 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:24:22.120 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:22.120 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:24:22.120 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:32.699090+0000", 2024-09-13T11:24:22.120 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:22.120 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:22.121 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:22.121 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:24:22.121 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:32.699090+0000", 2024-09-13T11:24:22.121 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:22.121 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:22.121 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:22.121 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:24:22.121 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:32.699092+0000", 2024-09-13T11:24:22.121 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.4559999999999999e-06 2024-09-13T11:24:22.121 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:22.121 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:22.121 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:24:22.121 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:32.699100+0000", 2024-09-13T11:24:22.121 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.1899999999999995e-06 2024-09-13T11:24:22.122 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:22.122 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:22.122 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:24:22.122 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:32.931118+0000", 2024-09-13T11:24:22.122 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.23201782900000001 2024-09-13T11:24:22.122 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:22.122 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:22.122 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:24:22.122 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:32.931132+0000", 2024-09-13T11:24:22.122 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.4212999999999999e-05 2024-09-13T11:24:22.122 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:22.122 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:22.122 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:24:22.122 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:32.931223+0000", 2024-09-13T11:24:22.123 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.1058999999999997e-05 2024-09-13T11:24:22.123 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:22.123 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:22.123 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:24:22.123 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:32.932089+0000", 2024-09-13T11:24:22.123 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00086632199999999999 2024-09-13T11:24:22.123 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:22.123 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:24:22.123 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:22.123 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:22.123 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:22.123 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop(client.4665.0:79801 3.5 e102/67 3:a6efdf84:::benchmark_data_smithi177_31116_object79800:head v 102'3851, mlcod=102'3848)", 2024-09-13T11:24:22.123 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:24:00.403754+0000", 2024-09-13T11:24:22.123 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 21.58991692, 2024-09-13T11:24:22.124 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.31749079899999999, 2024-09-13T11:24:22.124 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:24:22.124 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:24:22.124 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:22.124 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:24:22.124 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:22.124 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:24:22.124 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:00.403754+0000", 2024-09-13T11:24:22.124 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:22.124 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:22.124 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:22.124 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:24:22.124 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:00.403754+0000", 2024-09-13T11:24:22.124 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:22.125 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:22.125 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:22.125 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:24:22.125 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:00.403758+0000", 2024-09-13T11:24:22.125 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.507e-06 2024-09-13T11:24:22.125 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:22.125 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:22.125 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:24:22.125 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:00.403819+0000", 2024-09-13T11:24:22.125 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.1042000000000003e-05 2024-09-13T11:24:22.125 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:22.125 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:22.125 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:24:22.125 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:00.717963+0000", 2024-09-13T11:24:22.126 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.314144594 2024-09-13T11:24:22.126 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:22.126 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:22.126 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:24:22.126 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:00.717976+0000", 2024-09-13T11:24:22.126 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.3029e-05 2024-09-13T11:24:22.126 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:22.126 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:22.126 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:24:22.126 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:00.718143+0000", 2024-09-13T11:24:22.126 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00016639000000000001 2024-09-13T11:24:22.126 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:22.126 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:22.126 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:24:22.127 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:00.718184+0000", 2024-09-13T11:24:22.127 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.1801999999999997e-05 2024-09-13T11:24:22.127 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:22.127 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:22.127 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:24:22.127 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:00.721162+0000", 2024-09-13T11:24:22.127 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0029772800000000001 2024-09-13T11:24:22.127 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:22.127 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:22.127 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:24:22.127 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:00.721245+0000", 2024-09-13T11:24:22.127 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.3154999999999999e-05 2024-09-13T11:24:22.127 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:22.127 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:24:22.127 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:22.128 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:22.128 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:24:22.128 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-09-13T11:24:22.128 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-13T11:24:22.129 INFO:tasks.radosbench.radosbench.0.smithi177.stdout: 115 16 107020 107004 58.1486 112.625 0.00372252 0.0171912 2024-09-13T11:24:22.130 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-13T11:24:22.131 INFO:tasks.ceph.osd.11.smithi177.stderr:2024-09-13T11:24:22.127+0000 7f24792c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26580) UID: 0 2024-09-13T11:24:22.230 INFO:tasks.ceph.osd.12.smithi001.stderr:2024-09-13T11:24:22.227+0000 7f892c6e6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26314) UID: 0 2024-09-13T11:24:22.279 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-13T11:24:22.279 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [], 2024-09-13T11:24:22.279 INFO:teuthology.orchestra.run.smithi162.stdout: "num_ops": 0 2024-09-13T11:24:22.279 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-13T11:24:22.296 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-13T11:24:22.332 INFO:tasks.ceph.osd.12.smithi001.stderr:2024-09-13T11:24:22.331+0000 7f892c6e6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26314) UID: 0 2024-09-13T11:24:22.432 INFO:tasks.ceph.osd.9.smithi092.stderr:2024-09-13T11:24:22.426+0000 7ff5ab3d0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26348) UID: 0 2024-09-13T11:24:22.438 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-13T11:24:22.439 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [], 2024-09-13T11:24:22.439 INFO:teuthology.orchestra.run.smithi162.stdout: "complaint_time": 30, 2024-09-13T11:24:22.439 INFO:teuthology.orchestra.run.smithi162.stdout: "num_blocked_ops": 0 2024-09-13T11:24:22.439 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-13T11:24:22.455 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-13T11:24:22.525 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T11:24:22.532 INFO:tasks.ceph.osd.9.smithi092.stderr:2024-09-13T11:24:22.526+0000 7ff5ab3d0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26348) UID: 0 2024-09-13T11:24:22.589 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-13T11:24:22.590 INFO:teuthology.orchestra.run.smithi162.stdout: "size": 20, 2024-09-13T11:24:22.590 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 600, 2024-09-13T11:24:22.590 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [ 2024-09-13T11:24:22.590 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:22.590 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.4665.0:171 3.9 e41/30 3:9e1c7eeb:::benchmark_data_smithi177_31116_object170:head v 41'13, mlcod=41'9)", 2024-09-13T11:24:22.590 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:22:47.418602+0000", 2024-09-13T11:24:22.590 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 95.167977570999994, 2024-09-13T11:24:22.590 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.67235367300000004, 2024-09-13T11:24:22.590 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:24:22.590 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:24:22.591 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:22.591 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:24:22.591 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:22.591 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:24:22.591 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:47.418602+0000", 2024-09-13T11:24:22.591 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:22.591 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:22.591 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:22.591 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:24:22.591 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:47.418602+0000", 2024-09-13T11:24:22.591 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:22.592 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:22.592 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:22.592 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:24:22.592 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:47.418605+0000", 2024-09-13T11:24:22.592 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.2200000000000001e-06 2024-09-13T11:24:22.592 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:22.592 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:22.592 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:24:22.592 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:47.418668+0000", 2024-09-13T11:24:22.592 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.2921000000000003e-05 2024-09-13T11:24:22.592 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:22.592 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:22.593 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:24:22.593 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:48.089380+0000", 2024-09-13T11:24:22.593 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.67071191100000005 2024-09-13T11:24:22.593 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:22.593 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:22.593 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:24:22.593 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:48.089423+0000", 2024-09-13T11:24:22.593 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.2916e-05 2024-09-13T11:24:22.593 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:22.593 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:22.593 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:24:22.593 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:48.089533+0000", 2024-09-13T11:24:22.593 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000109996 2024-09-13T11:24:22.594 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:22.594 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:22.594 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:24:22.594 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:48.089570+0000", 2024-09-13T11:24:22.594 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.7314999999999998e-05 2024-09-13T11:24:22.594 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:22.594 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:22.594 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:24:22.594 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:48.090852+0000", 2024-09-13T11:24:22.594 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001281641 2024-09-13T11:24:22.594 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:22.594 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:22.594 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:24:22.595 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:48.090956+0000", 2024-09-13T11:24:22.595 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00010375299999999999 2024-09-13T11:24:22.595 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:22.595 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:24:22.595 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:22.595 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:22.595 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:22.595 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPushReply(3.19 71/70 [PushReplyOp(3:980058cd:::benchmark_data_smithi177_31116_object4309:head)])", 2024-09-13T11:24:22.595 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:25.123140+0000", 2024-09-13T11:24:22.595 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 57.463439891999997, 2024-09-13T11:24:22.595 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.16436381899999999, 2024-09-13T11:24:22.595 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:24:22.596 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:24:22.596 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-09-13T11:24:22.596 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:24:22.596 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:22.596 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:24:22.596 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.123140+0000", 2024-09-13T11:24:22.596 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:22.596 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:22.596 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:22.596 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:24:22.596 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.123140+0000", 2024-09-13T11:24:22.596 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:22.596 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:22.597 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:22.597 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:24:22.597 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.123141+0000", 2024-09-13T11:24:22.597 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.562e-06 2024-09-13T11:24:22.597 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:22.597 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:22.597 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:24:22.597 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.123151+0000", 2024-09-13T11:24:22.597 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.3030000000000005e-06 2024-09-13T11:24:22.597 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:22.597 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:22.597 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:24:22.598 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.285828+0000", 2024-09-13T11:24:22.598 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.16267689399999999 2024-09-13T11:24:22.598 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:22.598 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:22.598 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:24:22.598 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.285838+0000", 2024-09-13T11:24:22.598 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0016e-05 2024-09-13T11:24:22.598 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:22.598 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:22.598 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:24:22.598 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.285966+0000", 2024-09-13T11:24:22.598 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00012870500000000001 2024-09-13T11:24:22.598 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:22.599 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:22.599 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:24:22.599 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.287504+0000", 2024-09-13T11:24:22.599 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001537339 2024-09-13T11:24:22.599 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:22.599 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:24:22.599 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:22.599 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:22.599 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:22.599 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPushReply(3.19 74/70 [PushReplyOp(3:9b6cad3a:::benchmark_data_smithi177_31116_object16698:head)])", 2024-09-13T11:24:22.599 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:30.313297+0000", 2024-09-13T11:24:22.599 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 52.273283216000003, 2024-09-13T11:24:22.600 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.162355306, 2024-09-13T11:24:22.600 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:24:22.600 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:24:22.600 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-09-13T11:24:22.600 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:24:22.600 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:22.600 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:24:22.600 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.313297+0000", 2024-09-13T11:24:22.600 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:22.600 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:22.600 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:22.600 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:24:22.601 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.313297+0000", 2024-09-13T11:24:22.601 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:22.601 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:22.601 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:22.601 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:24:22.601 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.313298+0000", 2024-09-13T11:24:22.601 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.61e-06 2024-09-13T11:24:22.601 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:22.601 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:22.601 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:24:22.601 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.313306+0000", 2024-09-13T11:24:22.602 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.9950000000000005e-06 2024-09-13T11:24:22.602 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:22.602 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:22.602 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:24:22.602 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.473757+0000", 2024-09-13T11:24:22.602 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.16045083099999999 2024-09-13T11:24:22.602 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:22.602 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:22.602 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:24:22.602 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.473764+0000", 2024-09-13T11:24:22.602 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.4769999999999997e-06 2024-09-13T11:24:22.602 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:22.602 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:22.603 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:24:22.603 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.473870+0000", 2024-09-13T11:24:22.603 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00010529100000000001 2024-09-13T11:24:22.603 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:22.603 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:22.603 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:24:22.603 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.475652+0000", 2024-09-13T11:24:22.603 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001782102 2024-09-13T11:24:22.603 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:22.603 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:24:22.603 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:22.603 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:22.604 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:22.604 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:54073 3.19 3:9f968cf6:::benchmark_data_smithi177_31116_object54072:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e74)", 2024-09-13T11:24:22.604 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:30.331382+0000", 2024-09-13T11:24:22.604 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 52.255197522000003, 2024-09-13T11:24:22.604 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.15619861600000001, 2024-09-13T11:24:22.604 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:24:22.604 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:24:22.604 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:22.604 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:24:22.604 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:24:22.604 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:22.604 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 54073 2024-09-13T11:24:22.604 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:22.605 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:24:22.605 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:22.605 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:24:22.605 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.331382+0000", 2024-09-13T11:24:22.605 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:22.605 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:22.605 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:22.605 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:24:22.605 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.331382+0000", 2024-09-13T11:24:22.605 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:22.605 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:22.605 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:22.606 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:24:22.606 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.331386+0000", 2024-09-13T11:24:22.606 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.5769999999999998e-06 2024-09-13T11:24:22.606 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:22.606 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:22.606 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:24:22.606 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.331430+0000", 2024-09-13T11:24:22.606 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.4172999999999999e-05 2024-09-13T11:24:22.606 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:22.606 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:22.606 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:24:22.606 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.331450+0000", 2024-09-13T11:24:22.607 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.9922999999999999e-05 2024-09-13T11:24:22.607 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:22.607 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:22.607 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:24:22.607 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.331460+0000", 2024-09-13T11:24:22.607 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0444e-05 2024-09-13T11:24:22.607 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:22.607 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:22.607 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:24:22.607 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.331590+0000", 2024-09-13T11:24:22.607 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00013000899999999999 2024-09-13T11:24:22.607 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:22.607 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:22.608 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:24:22.608 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.331929+0000", 2024-09-13T11:24:22.608 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00033835400000000002 2024-09-13T11:24:22.608 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:22.608 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:22.608 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 1,9", 2024-09-13T11:24:22.608 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.332198+0000", 2024-09-13T11:24:22.608 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00026941900000000002 2024-09-13T11:24:22.608 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:22.608 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:22.608 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:24:22.608 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.334200+0000", 2024-09-13T11:24:22.609 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0020020179999999999 2024-09-13T11:24:22.609 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:22.609 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:22.609 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:22.609 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.334826+0000", 2024-09-13T11:24:22.609 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00062540600000000001 2024-09-13T11:24:22.609 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:22.609 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:22.609 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:22.609 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.486479+0000", 2024-09-13T11:24:22.609 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.151653395 2024-09-13T11:24:22.609 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:22.610 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:22.610 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:24:22.610 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.486691+0000", 2024-09-13T11:24:22.610 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00021176499999999999 2024-09-13T11:24:22.610 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:22.610 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:22.610 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:24:22.610 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.487581+0000", 2024-09-13T11:24:22.610 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00089013299999999998 2024-09-13T11:24:22.610 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:22.610 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:24:22.610 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:22.611 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:22.611 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:22.611 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop_reply(client.4665.0:54073 3.19 e74/70 ondisk, result = 0)", 2024-09-13T11:24:22.611 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:30.334897+0000", 2024-09-13T11:24:22.611 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 52.251682744999997, 2024-09-13T11:24:22.611 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.152699536, 2024-09-13T11:24:22.611 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:24:22.611 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:24:22.611 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:24:22.611 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:24:22.611 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:22.611 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:24:22.612 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.334897+0000", 2024-09-13T11:24:22.612 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:22.612 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:22.612 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:22.612 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:24:22.612 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.334897+0000", 2024-09-13T11:24:22.612 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:22.612 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:22.612 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:22.612 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:24:22.612 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.334898+0000", 2024-09-13T11:24:22.612 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.499e-06 2024-09-13T11:24:22.613 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:22.613 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:22.613 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:24:22.613 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.334905+0000", 2024-09-13T11:24:22.613 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.6470000000000001e-06 2024-09-13T11:24:22.613 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:22.613 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:22.613 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:24:22.613 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.486251+0000", 2024-09-13T11:24:22.613 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.151345432 2024-09-13T11:24:22.613 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:22.613 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:22.613 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:24:22.614 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.486262+0000", 2024-09-13T11:24:22.614 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1005000000000001e-05 2024-09-13T11:24:22.614 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:22.614 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:22.614 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:24:22.614 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.486411+0000", 2024-09-13T11:24:22.614 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00014960099999999999 2024-09-13T11:24:22.614 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:22.614 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:22.614 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:24:22.614 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.486434+0000", 2024-09-13T11:24:22.614 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.3034e-05 2024-09-13T11:24:22.614 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:22.615 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:22.615 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:24:22.615 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.487597+0000", 2024-09-13T11:24:22.615 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0011623180000000001 2024-09-13T11:24:22.615 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:22.615 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:24:22.615 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:22.615 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:22.615 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:22.615 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPushReply(3.19 74/70 [PushReplyOp(3:9edf939c:::benchmark_data_smithi177_31116_object17207:head)])", 2024-09-13T11:24:22.615 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:30.897971+0000", 2024-09-13T11:24:22.616 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 51.688608719999998, 2024-09-13T11:24:22.616 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.40616071100000001, 2024-09-13T11:24:22.616 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:24:22.616 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:24:22.616 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-09-13T11:24:22.616 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:24:22.616 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:22.616 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:24:22.616 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.897971+0000", 2024-09-13T11:24:22.616 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:22.616 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:22.616 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:22.616 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:24:22.617 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.897971+0000", 2024-09-13T11:24:22.617 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:22.617 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:22.617 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:22.617 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:24:22.617 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.897972+0000", 2024-09-13T11:24:22.617 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1429999999999999e-06 2024-09-13T11:24:22.617 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:22.617 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:22.617 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:24:22.617 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.897977+0000", 2024-09-13T11:24:22.617 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.5199999999999999e-06 2024-09-13T11:24:22.618 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:22.618 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:22.618 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:24:22.618 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:31.302500+0000", 2024-09-13T11:24:22.618 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.40452312400000001 2024-09-13T11:24:22.618 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:22.618 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:22.618 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:24:22.618 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:31.302513+0000", 2024-09-13T11:24:22.618 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2753e-05 2024-09-13T11:24:22.618 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:22.618 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:22.618 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:24:22.619 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:31.302645+0000", 2024-09-13T11:24:22.619 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00013292100000000001 2024-09-13T11:24:22.619 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:22.619 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:22.619 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:24:22.619 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:31.304132+0000", 2024-09-13T11:24:22.619 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0014862499999999999 2024-09-13T11:24:22.619 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:22.619 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:24:22.619 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:22.620 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:22.620 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:22.620 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPushReply(3.19 75/70 [PushReplyOp(3:9d1489b1:::benchmark_data_smithi177_31116_object18698:head)])", 2024-09-13T11:24:22.620 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:31.697212+0000", 2024-09-13T11:24:22.620 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 50.889367718999999, 2024-09-13T11:24:22.620 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.22850633200000001, 2024-09-13T11:24:22.620 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:24:22.620 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:24:22.620 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-09-13T11:24:22.620 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:24:22.620 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:22.620 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:24:22.621 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:31.697212+0000", 2024-09-13T11:24:22.621 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:22.621 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:22.621 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:22.621 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:24:22.621 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:31.697212+0000", 2024-09-13T11:24:22.621 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:22.621 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:22.621 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:22.621 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:24:22.621 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:31.697214+0000", 2024-09-13T11:24:22.621 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.7439999999999999e-06 2024-09-13T11:24:22.621 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:22.622 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:22.622 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:24:22.622 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:31.697224+0000", 2024-09-13T11:24:22.622 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0173000000000001e-05 2024-09-13T11:24:22.622 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:22.622 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:22.622 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:24:22.622 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:31.923931+0000", 2024-09-13T11:24:22.622 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.226706926 2024-09-13T11:24:22.622 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:22.622 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:22.623 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:24:22.623 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:31.923950+0000", 2024-09-13T11:24:22.623 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.8899e-05 2024-09-13T11:24:22.623 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:22.623 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:22.623 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:24:22.623 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:31.924449+0000", 2024-09-13T11:24:22.623 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00049907800000000002 2024-09-13T11:24:22.623 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:22.623 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:22.623 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:24:22.623 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:31.925718+0000", 2024-09-13T11:24:22.623 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001269512 2024-09-13T11:24:22.623 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:22.624 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:24:22.624 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:22.624 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:22.624 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:22.624 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPushReply(3.19 76/70 [PushReplyOp(3:9faf6d5f:::benchmark_data_smithi177_31116_object18954:head)])", 2024-09-13T11:24:22.624 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:32.042874+0000", 2024-09-13T11:24:22.624 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 50.543705850000002, 2024-09-13T11:24:22.624 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.40212404299999999, 2024-09-13T11:24:22.624 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:24:22.624 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:24:22.624 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-09-13T11:24:22.624 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:24:22.625 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:22.625 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:24:22.625 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.042874+0000", 2024-09-13T11:24:22.625 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:22.625 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:22.625 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:22.625 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:24:22.625 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.042874+0000", 2024-09-13T11:24:22.625 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:22.625 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:22.625 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:22.625 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:24:22.626 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.042875+0000", 2024-09-13T11:24:22.626 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.432e-06 2024-09-13T11:24:22.626 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:22.626 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:22.626 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:24:22.626 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.042886+0000", 2024-09-13T11:24:22.626 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0265000000000001e-05 2024-09-13T11:24:22.626 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:22.626 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:22.626 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:24:22.626 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.443455+0000", 2024-09-13T11:24:22.626 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.40056951400000002 2024-09-13T11:24:22.627 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:22.627 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:22.627 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:24:22.627 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.443480+0000", 2024-09-13T11:24:22.627 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.4929000000000001e-05 2024-09-13T11:24:22.627 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:22.627 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:22.627 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:24:22.627 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.443666+0000", 2024-09-13T11:24:22.627 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000186446 2024-09-13T11:24:22.627 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:22.627 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:22.628 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:24:22.628 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.444998+0000", 2024-09-13T11:24:22.628 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0013314570000000001 2024-09-13T11:24:22.628 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:22.628 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:24:22.628 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:22.628 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:22.628 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:22.628 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:56140 3.19 3:9c6bfb18:::benchmark_data_smithi177_31116_object56139:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e79)", 2024-09-13T11:24:22.628 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:36.279622+0000", 2024-09-13T11:24:22.628 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 46.306957245, 2024-09-13T11:24:22.628 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.12757104799999999, 2024-09-13T11:24:22.628 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:24:22.629 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:24:22.629 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:22.629 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:24:22.629 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:24:22.629 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:22.629 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 56140 2024-09-13T11:24:22.629 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:22.629 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:24:22.629 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:22.629 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:24:22.629 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:36.279622+0000", 2024-09-13T11:24:22.629 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:22.630 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:22.630 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:22.630 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:24:22.630 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:36.279622+0000", 2024-09-13T11:24:22.630 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:22.630 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:22.630 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:22.630 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:24:22.630 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:36.279625+0000", 2024-09-13T11:24:22.630 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.4399999999999999e-06 2024-09-13T11:24:22.630 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:22.630 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:22.631 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:24:22.631 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:36.279662+0000", 2024-09-13T11:24:22.631 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.7094000000000001e-05 2024-09-13T11:24:22.631 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:22.631 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:22.631 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:24:22.631 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:36.279679+0000", 2024-09-13T11:24:22.631 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.7147999999999999e-05 2024-09-13T11:24:22.631 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:22.631 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:22.631 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:24:22.631 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:36.279687+0000", 2024-09-13T11:24:22.632 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.7379999999999994e-06 2024-09-13T11:24:22.632 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:22.632 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:22.632 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:24:22.632 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:36.279744+0000", 2024-09-13T11:24:22.632 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.6711e-05 2024-09-13T11:24:22.632 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:22.632 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:22.632 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:24:22.632 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:36.279881+0000", 2024-09-13T11:24:22.632 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000137377 2024-09-13T11:24:22.632 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:22.633 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:22.633 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 1,9", 2024-09-13T11:24:22.633 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:36.280005+0000", 2024-09-13T11:24:22.633 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00012385799999999999 2024-09-13T11:24:22.633 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:22.633 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:22.633 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:24:22.633 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:36.281925+0000", 2024-09-13T11:24:22.633 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0019203410000000001 2024-09-13T11:24:22.633 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:22.633 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:22.633 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:22.634 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:36.286822+0000", 2024-09-13T11:24:22.634 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0048964250000000003 2024-09-13T11:24:22.634 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:22.634 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:22.634 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:22.634 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:36.406831+0000", 2024-09-13T11:24:22.634 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.120009569 2024-09-13T11:24:22.634 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:22.634 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:22.634 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:24:22.634 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:36.406875+0000", 2024-09-13T11:24:22.634 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.3618999999999997e-05 2024-09-13T11:24:22.634 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:22.635 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:22.635 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:24:22.635 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:36.407194+0000", 2024-09-13T11:24:22.635 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000318728 2024-09-13T11:24:22.635 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:22.635 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:24:22.635 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:22.635 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:22.635 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:22.635 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop_reply(client.4665.0:56140 3.19 e79/70 ondisk, result = 0)", 2024-09-13T11:24:22.635 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:36.282064+0000", 2024-09-13T11:24:22.635 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 46.304515903999999, 2024-09-13T11:24:22.636 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.12513713400000001, 2024-09-13T11:24:22.636 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:24:22.636 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:24:22.636 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:24:22.636 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:24:22.636 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:22.636 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:24:22.636 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:36.282064+0000", 2024-09-13T11:24:22.636 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:22.636 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:22.636 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:22.636 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:24:22.637 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:36.282064+0000", 2024-09-13T11:24:22.637 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:22.637 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:22.637 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:22.637 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:24:22.637 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:36.282066+0000", 2024-09-13T11:24:22.637 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.7519999999999999e-06 2024-09-13T11:24:22.637 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:22.637 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:22.637 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:24:22.637 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:36.282071+0000", 2024-09-13T11:24:22.637 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.6960000000000002e-06 2024-09-13T11:24:22.637 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:22.638 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:22.638 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:24:22.638 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:36.406725+0000", 2024-09-13T11:24:22.638 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.12465353999999999 2024-09-13T11:24:22.638 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:22.638 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:22.638 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:24:22.638 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:36.406738+0000", 2024-09-13T11:24:22.638 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2774e-05 2024-09-13T11:24:22.638 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:22.638 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:22.638 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:24:22.638 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:36.406816+0000", 2024-09-13T11:24:22.639 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.8428000000000002e-05 2024-09-13T11:24:22.639 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:22.639 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:22.639 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:24:22.639 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:36.406826+0000", 2024-09-13T11:24:22.639 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0239e-05 2024-09-13T11:24:22.639 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:22.639 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:22.639 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:24:22.639 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:36.407201+0000", 2024-09-13T11:24:22.639 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00037470500000000003 2024-09-13T11:24:22.639 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:22.639 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:24:22.639 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:22.640 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:22.640 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:22.640 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPushReply(3.19 79/70 [PushReplyOp(3:99a0d211:::benchmark_data_smithi177_31116_object31006:head)])", 2024-09-13T11:24:22.640 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:36.286313+0000", 2024-09-13T11:24:22.640 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 46.300266293999996, 2024-09-13T11:24:22.640 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.14795829499999999, 2024-09-13T11:24:22.640 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:24:22.640 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:24:22.640 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-09-13T11:24:22.640 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:24:22.640 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:22.640 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:24:22.640 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:36.286313+0000", 2024-09-13T11:24:22.641 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:22.641 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:22.641 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:22.641 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:24:22.641 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:36.286313+0000", 2024-09-13T11:24:22.641 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:22.641 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:22.641 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:22.641 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:24:22.641 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:36.286315+0000", 2024-09-13T11:24:22.641 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4169999999999999e-06 2024-09-13T11:24:22.641 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:22.641 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:22.642 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:24:22.642 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:36.286321+0000", 2024-09-13T11:24:22.642 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.3260000000000001e-06 2024-09-13T11:24:22.642 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:22.642 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:22.642 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:24:22.642 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:36.431926+0000", 2024-09-13T11:24:22.642 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.14560464300000001 2024-09-13T11:24:22.642 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:22.642 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:22.642 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:24:22.642 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:36.431934+0000", 2024-09-13T11:24:22.642 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.439e-06 2024-09-13T11:24:22.642 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:22.643 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:22.643 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:24:22.643 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:36.433520+0000", 2024-09-13T11:24:22.643 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0015857860000000001 2024-09-13T11:24:22.643 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:22.643 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:22.643 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:24:22.643 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:36.434272+0000", 2024-09-13T11:24:22.643 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00075168399999999997 2024-09-13T11:24:22.643 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:22.643 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:24:22.643 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:22.644 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:22.644 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:22.644 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPush(3.1d 85/78 [PushOp(3:bc4b9336:::benchmark_data_smithi177_31116_object11944:head, version: 49'734, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:bc4b9336:::benchmark_data_smithi177_31116_object11944:head@49'734, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:24:22.644 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:42.225966+0000", 2024-09-13T11:24:22.644 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 40.360613637, 2024-09-13T11:24:22.644 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.23879947800000001, 2024-09-13T11:24:22.644 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:24:22.644 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:24:22.644 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:24:22.644 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:24:22.644 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:22.644 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:24:22.645 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.225966+0000", 2024-09-13T11:24:22.645 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:22.645 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:22.645 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:22.645 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:24:22.645 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.225966+0000", 2024-09-13T11:24:22.645 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:22.645 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:22.645 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:22.645 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:24:22.645 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.225969+0000", 2024-09-13T11:24:22.645 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.0400000000000001e-06 2024-09-13T11:24:22.645 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:22.646 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:22.646 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:24:22.646 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.226036+0000", 2024-09-13T11:24:22.646 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.6978999999999997e-05 2024-09-13T11:24:22.646 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:22.646 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:22.646 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:24:22.646 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.463110+0000", 2024-09-13T11:24:22.646 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.23707400000000001 2024-09-13T11:24:22.646 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:22.646 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:22.646 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:24:22.646 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.463140+0000", 2024-09-13T11:24:22.647 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.9575999999999999e-05 2024-09-13T11:24:22.647 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:22.647 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:22.647 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:24:22.647 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.463326+0000", 2024-09-13T11:24:22.647 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000186298 2024-09-13T11:24:22.647 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:22.647 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:22.647 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:24:22.647 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.463351+0000", 2024-09-13T11:24:22.647 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.4862999999999999e-05 2024-09-13T11:24:22.647 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:22.647 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:22.647 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:24:22.648 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.464766+0000", 2024-09-13T11:24:22.648 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001414722 2024-09-13T11:24:22.648 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:22.648 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:24:22.648 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:22.648 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:22.648 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:22.648 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPush(3.1d 89/78 [PushOp(3:bc1b7cfb:::benchmark_data_smithi177_31116_object22629:head, version: 55'1387, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:bc1b7cfb:::benchmark_data_smithi177_31116_object22629:head@55'1387, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:24:22.648 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:46.660855+0000", 2024-09-13T11:24:22.648 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 35.925725141000001, 2024-09-13T11:24:22.648 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.26622785599999998, 2024-09-13T11:24:22.648 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:24:22.648 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:24:22.649 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:24:22.649 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:24:22.649 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:22.649 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:24:22.649 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:46.660855+0000", 2024-09-13T11:24:22.649 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:22.649 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:22.649 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:22.649 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:24:22.649 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:46.660855+0000", 2024-09-13T11:24:22.649 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:22.649 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:22.649 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:22.650 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:24:22.650 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:46.660857+0000", 2024-09-13T11:24:22.650 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.6560000000000001e-06 2024-09-13T11:24:22.650 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:22.650 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:22.650 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:24:22.650 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:46.660931+0000", 2024-09-13T11:24:22.650 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.4041999999999993e-05 2024-09-13T11:24:22.650 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:22.650 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:22.650 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:24:22.650 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:46.924654+0000", 2024-09-13T11:24:22.650 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.26372239800000002 2024-09-13T11:24:22.650 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:22.651 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:22.651 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:24:22.651 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:46.924729+0000", 2024-09-13T11:24:22.651 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.5061999999999994e-05 2024-09-13T11:24:22.651 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:22.651 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:22.651 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:24:22.651 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:46.924986+0000", 2024-09-13T11:24:22.651 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00025762000000000002 2024-09-13T11:24:22.651 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:22.651 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:22.651 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:24:22.651 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:46.925013+0000", 2024-09-13T11:24:22.652 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.6179000000000001e-05 2024-09-13T11:24:22.652 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:22.652 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:22.652 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:24:22.652 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:46.927082+0000", 2024-09-13T11:24:22.652 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.002069899 2024-09-13T11:24:22.652 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:22.652 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:24:22.652 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:22.652 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:22.652 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:22.653 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPush(3.1d 90/78 [PushOp(3:bc0e7228:::benchmark_data_smithi177_31116_object24733:head, version: 55'1512, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:bc0e7228:::benchmark_data_smithi177_31116_object24733:head@55'1512, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:24:22.653 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:47.529126+0000", 2024-09-13T11:24:22.653 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 35.057454167000003, 2024-09-13T11:24:22.653 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.21343989499999999, 2024-09-13T11:24:22.653 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:24:22.653 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:24:22.653 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:24:22.653 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:24:22.653 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:22.653 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:24:22.653 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:47.529126+0000", 2024-09-13T11:24:22.653 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:22.653 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:22.654 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:22.654 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:24:22.654 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:47.529126+0000", 2024-09-13T11:24:22.654 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:22.654 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:22.654 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:22.654 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:24:22.654 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:47.529127+0000", 2024-09-13T11:24:22.654 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.252e-06 2024-09-13T11:24:22.654 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:22.654 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:22.654 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:24:22.655 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:47.529185+0000", 2024-09-13T11:24:22.655 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.8224999999999999e-05 2024-09-13T11:24:22.655 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:22.655 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:22.655 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:24:22.655 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:47.740216+0000", 2024-09-13T11:24:22.655 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.21103060800000001 2024-09-13T11:24:22.655 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:22.656 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:22.656 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:24:22.656 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:47.740248+0000", 2024-09-13T11:24:22.656 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.2821999999999998e-05 2024-09-13T11:24:22.656 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:22.656 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:22.656 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:24:22.656 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:47.740504+0000", 2024-09-13T11:24:22.656 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00025518399999999999 2024-09-13T11:24:22.656 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:22.656 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:22.656 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:24:22.657 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:47.740544+0000", 2024-09-13T11:24:22.657 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.0438999999999997e-05 2024-09-13T11:24:22.657 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:22.657 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:22.657 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:24:22.657 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:47.742565+0000", 2024-09-13T11:24:22.657 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0020213649999999998 2024-09-13T11:24:22.657 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:22.657 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:24:22.657 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:22.657 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:22.657 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:22.658 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPush(3.1d 90/78 [PushOp(3:ba836eb6:::benchmark_data_smithi177_31116_object24744:head, version: 55'1513, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:ba836eb6:::benchmark_data_smithi177_31116_object24744:head@55'1513, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:24:22.658 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:47.748907+0000", 2024-09-13T11:24:22.658 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 34.837672304999998, 2024-09-13T11:24:22.658 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.15906743400000001, 2024-09-13T11:24:22.658 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:24:22.658 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:24:22.658 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:24:22.658 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:24:22.658 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:22.658 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:24:22.658 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:47.748907+0000", 2024-09-13T11:24:22.659 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:22.659 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:22.659 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:22.659 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:24:22.659 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:47.748907+0000", 2024-09-13T11:24:22.659 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:22.659 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:22.659 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:22.659 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:24:22.659 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:47.748911+0000", 2024-09-13T11:24:22.659 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.433e-06 2024-09-13T11:24:22.659 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:22.659 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:22.660 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:24:22.660 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:47.749007+0000", 2024-09-13T11:24:22.660 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.6510999999999998e-05 2024-09-13T11:24:22.660 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:22.660 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:22.660 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:24:22.660 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:47.906101+0000", 2024-09-13T11:24:22.660 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.15709389100000001 2024-09-13T11:24:22.660 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:22.660 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:22.660 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:24:22.660 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:47.906163+0000", 2024-09-13T11:24:22.660 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.1367999999999999e-05 2024-09-13T11:24:22.661 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:22.661 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:22.661 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:24:22.661 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:47.906375+0000", 2024-09-13T11:24:22.661 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00021198500000000001 2024-09-13T11:24:22.661 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:22.661 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:22.661 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:24:22.661 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:47.906397+0000", 2024-09-13T11:24:22.661 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.2739999999999999e-05 2024-09-13T11:24:22.661 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:22.661 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:22.661 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:24:22.662 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:47.907975+0000", 2024-09-13T11:24:22.662 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001577506 2024-09-13T11:24:22.662 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:22.662 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:24:22.662 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:22.662 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:22.662 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:22.662 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:83199 3.1d 3:be2c6fae:::benchmark_data_smithi177_31116_object83198:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106)", 2024-09-13T11:24:22.662 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:24:04.297306+0000", 2024-09-13T11:24:22.662 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 18.289274214999999, 2024-09-13T11:24:22.662 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.25114587999999999, 2024-09-13T11:24:22.662 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:24:22.662 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:24:22.663 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:22.663 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:24:22.663 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:24:22.663 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:22.663 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 83199 2024-09-13T11:24:22.663 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:22.663 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:24:22.663 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:22.663 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:24:22.663 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.297306+0000", 2024-09-13T11:24:22.663 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:22.663 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:22.663 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:22.664 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:24:22.664 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.297306+0000", 2024-09-13T11:24:22.664 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:22.664 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:22.664 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:22.664 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:24:22.664 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.297308+0000", 2024-09-13T11:24:22.664 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.4150000000000002e-06 2024-09-13T11:24:22.664 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:22.664 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:22.664 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:24:22.664 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.297352+0000", 2024-09-13T11:24:22.664 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.4431000000000002e-05 2024-09-13T11:24:22.665 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:22.665 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:22.665 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:24:22.665 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.297369+0000", 2024-09-13T11:24:22.665 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.6685e-05 2024-09-13T11:24:22.665 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:22.665 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:22.665 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:24:22.665 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.297376+0000", 2024-09-13T11:24:22.665 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.7299999999999999e-06 2024-09-13T11:24:22.665 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:22.665 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:22.665 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:24:22.666 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.297427+0000", 2024-09-13T11:24:22.666 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.1461000000000001e-05 2024-09-13T11:24:22.666 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:22.666 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:22.666 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:24:22.666 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.297528+0000", 2024-09-13T11:24:22.666 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00010028599999999999 2024-09-13T11:24:22.666 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:22.666 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:22.666 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 15", 2024-09-13T11:24:22.666 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.297685+0000", 2024-09-13T11:24:22.666 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00015760500000000001 2024-09-13T11:24:22.666 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:22.666 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:22.667 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:24:22.667 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.300779+0000", 2024-09-13T11:24:22.667 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0030936700000000002 2024-09-13T11:24:22.667 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:22.667 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:22.667 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:22.667 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.548279+0000", 2024-09-13T11:24:22.667 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.24750059999999999 2024-09-13T11:24:22.667 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:22.667 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:22.667 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:24:22.667 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.548373+0000", 2024-09-13T11:24:22.667 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.3451999999999999e-05 2024-09-13T11:24:22.667 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:22.668 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:22.668 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:24:22.668 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.548451+0000", 2024-09-13T11:24:22.668 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.8545000000000001e-05 2024-09-13T11:24:22.668 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:22.668 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:24:22.668 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:22.668 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:22.668 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:22.668 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:84621 3.1d 3:becd72aa:::benchmark_data_smithi177_31116_object84620:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-09-13T11:24:22.668 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:24:05.741008+0000", 2024-09-13T11:24:22.668 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 16.845572215000001, 2024-09-13T11:24:22.669 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.17994101100000001, 2024-09-13T11:24:22.669 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:24:22.669 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:24:22.669 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:22.669 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:24:22.669 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:24:22.669 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:22.669 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 84621 2024-09-13T11:24:22.669 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:22.669 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:24:22.669 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:22.669 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:24:22.669 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.741008+0000", 2024-09-13T11:24:22.669 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:22.670 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:22.670 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:22.670 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:24:22.670 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.741008+0000", 2024-09-13T11:24:22.670 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:22.670 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:22.670 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:22.670 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:24:22.670 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.741010+0000", 2024-09-13T11:24:22.670 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.898e-06 2024-09-13T11:24:22.670 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:22.670 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:22.670 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:24:22.671 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.741061+0000", 2024-09-13T11:24:22.671 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.0898e-05 2024-09-13T11:24:22.671 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:22.671 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:22.671 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:24:22.671 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.741085+0000", 2024-09-13T11:24:22.671 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.3558e-05 2024-09-13T11:24:22.671 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:22.671 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:22.671 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:24:22.671 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.741093+0000", 2024-09-13T11:24:22.671 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.3890000000000005e-06 2024-09-13T11:24:22.671 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:22.672 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:22.672 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:24:22.672 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.741178+0000", 2024-09-13T11:24:22.672 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.4792999999999999e-05 2024-09-13T11:24:22.672 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:22.672 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:22.672 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:24:22.672 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.741386+0000", 2024-09-13T11:24:22.672 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000207505 2024-09-13T11:24:22.672 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:22.672 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:22.672 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 15", 2024-09-13T11:24:22.672 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.741597+0000", 2024-09-13T11:24:22.673 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00021158200000000001 2024-09-13T11:24:22.673 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:22.673 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:22.673 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:24:22.673 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.743392+0000", 2024-09-13T11:24:22.673 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00179518 2024-09-13T11:24:22.673 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:22.673 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:22.673 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:22.673 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.920866+0000", 2024-09-13T11:24:22.673 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.17747373699999999 2024-09-13T11:24:22.673 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:22.673 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:22.673 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:24:22.674 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.920913+0000", 2024-09-13T11:24:22.674 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.6451e-05 2024-09-13T11:24:22.674 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:22.674 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:22.674 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:24:22.674 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.920949+0000", 2024-09-13T11:24:22.674 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.6019999999999997e-05 2024-09-13T11:24:22.674 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:22.674 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:24:22.674 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:22.674 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:22.674 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:22.674 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:88685 3.1d 3:baceb770:::benchmark_data_smithi177_31116_object88684:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110)", 2024-09-13T11:24:22.674 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:24:08.926194+0000", 2024-09-13T11:24:22.675 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 13.660385570999999, 2024-09-13T11:24:22.675 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.16678378799999999, 2024-09-13T11:24:22.675 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:24:22.675 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:24:22.675 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:22.675 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:24:22.675 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:24:22.675 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:22.675 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 88685 2024-09-13T11:24:22.675 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:22.675 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:24:22.675 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:22.675 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:24:22.675 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:08.926194+0000", 2024-09-13T11:24:22.676 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:22.676 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:22.676 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:22.676 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:24:22.676 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:08.926194+0000", 2024-09-13T11:24:22.676 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:22.676 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:22.676 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:22.676 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:24:22.676 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:08.926196+0000", 2024-09-13T11:24:22.676 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.533e-06 2024-09-13T11:24:22.676 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:22.676 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:22.676 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:24:22.677 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:08.926253+0000", 2024-09-13T11:24:22.677 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.6906e-05 2024-09-13T11:24:22.677 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:22.677 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:22.677 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:24:22.677 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:08.926262+0000", 2024-09-13T11:24:22.677 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.3530000000000001e-06 2024-09-13T11:24:22.677 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:22.677 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:22.677 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:24:22.677 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:08.926268+0000", 2024-09-13T11:24:22.677 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.2890000000000003e-06 2024-09-13T11:24:22.677 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:22.677 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:22.678 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:24:22.678 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:08.928510+0000", 2024-09-13T11:24:22.678 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0022416519999999998 2024-09-13T11:24:22.678 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:22.678 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:22.678 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:24:22.678 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:08.928748+0000", 2024-09-13T11:24:22.678 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00023773800000000001 2024-09-13T11:24:22.678 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:22.678 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:22.678 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 15", 2024-09-13T11:24:22.678 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:08.928953+0000", 2024-09-13T11:24:22.679 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00020532999999999999 2024-09-13T11:24:22.679 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:22.679 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:22.679 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:24:22.679 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:08.930702+0000", 2024-09-13T11:24:22.679 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001749163 2024-09-13T11:24:22.679 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:22.679 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:22.679 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:22.679 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:09.092878+0000", 2024-09-13T11:24:22.679 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.16217616900000001 2024-09-13T11:24:22.679 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:22.679 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:22.679 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:24:22.680 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:09.092932+0000", 2024-09-13T11:24:22.680 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.3523999999999998e-05 2024-09-13T11:24:22.680 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:22.680 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:22.680 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:24:22.680 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:09.092978+0000", 2024-09-13T11:24:22.680 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.6131000000000003e-05 2024-09-13T11:24:22.680 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:22.680 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:24:22.680 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:22.680 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:22.680 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:22.681 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop_reply(client.4665.0:88685 3.1d e110/102 ondisk, result = 0)", 2024-09-13T11:24:22.681 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:24:08.930404+0000", 2024-09-13T11:24:22.681 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 13.656175259999999, 2024-09-13T11:24:22.681 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.162580424, 2024-09-13T11:24:22.681 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:24:22.681 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:24:22.681 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:24:22.681 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:24:22.681 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:22.681 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:24:22.681 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:08.930404+0000", 2024-09-13T11:24:22.681 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:22.681 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:22.681 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:22.682 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:24:22.682 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:08.930404+0000", 2024-09-13T11:24:22.682 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:22.682 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:22.682 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:22.682 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:24:22.682 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:08.930406+0000", 2024-09-13T11:24:22.682 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.8610000000000001e-06 2024-09-13T11:24:22.682 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:22.682 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:22.682 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:24:22.682 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:08.930414+0000", 2024-09-13T11:24:22.683 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.2239999999999998e-06 2024-09-13T11:24:22.683 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:22.683 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:22.683 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:24:22.683 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:09.092807+0000", 2024-09-13T11:24:22.683 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.16239347900000001 2024-09-13T11:24:22.683 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:22.683 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:22.683 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:24:22.683 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:09.092816+0000", 2024-09-13T11:24:22.683 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.5760000000000006e-06 2024-09-13T11:24:22.683 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:22.683 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:22.683 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:24:22.684 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:09.092863+0000", 2024-09-13T11:24:22.684 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.7472000000000002e-05 2024-09-13T11:24:22.684 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:22.684 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:22.684 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:24:22.684 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:09.092872+0000", 2024-09-13T11:24:22.684 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.0250000000000008e-06 2024-09-13T11:24:22.684 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:22.684 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:22.684 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:24:22.684 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:09.092985+0000", 2024-09-13T11:24:22.684 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000112787 2024-09-13T11:24:22.684 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:22.685 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:24:22.685 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:22.685 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:22.685 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:22.685 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:91535 3.1d 3:b9fc6541:::benchmark_data_smithi177_31116_object91534:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110)", 2024-09-13T11:24:22.685 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:24:10.944558+0000", 2024-09-13T11:24:22.685 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 11.642022131999999, 2024-09-13T11:24:22.685 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.10052925, 2024-09-13T11:24:22.685 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:24:22.685 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:24:22.685 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:22.685 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:24:22.685 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:24:22.685 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:22.686 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 91535 2024-09-13T11:24:22.686 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:22.686 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:24:22.686 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:22.686 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:24:22.686 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:10.944558+0000", 2024-09-13T11:24:22.686 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:22.686 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:22.686 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:22.686 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:24:22.686 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:10.944558+0000", 2024-09-13T11:24:22.686 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:22.686 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:22.687 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:22.687 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:24:22.687 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:10.944564+0000", 2024-09-13T11:24:22.687 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.6429999999999998e-06 2024-09-13T11:24:22.687 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:22.687 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:22.687 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:24:22.687 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:10.944689+0000", 2024-09-13T11:24:22.687 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00012511400000000001 2024-09-13T11:24:22.687 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:22.687 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:22.687 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:24:22.687 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:10.944703+0000", 2024-09-13T11:24:22.688 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3405e-05 2024-09-13T11:24:22.688 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:22.688 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:22.688 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:24:22.688 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:10.944709+0000", 2024-09-13T11:24:22.688 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.3659999999999997e-06 2024-09-13T11:24:22.688 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:22.688 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:22.688 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:24:22.688 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:10.944772+0000", 2024-09-13T11:24:22.688 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.2822000000000003e-05 2024-09-13T11:24:22.688 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:22.688 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:22.688 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:24:22.689 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:10.944921+0000", 2024-09-13T11:24:22.689 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000149524 2024-09-13T11:24:22.689 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:22.689 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:22.689 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 15", 2024-09-13T11:24:22.689 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:10.945047+0000", 2024-09-13T11:24:22.689 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000125043 2024-09-13T11:24:22.689 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:22.689 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:22.689 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:24:22.689 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:10.946766+0000", 2024-09-13T11:24:22.689 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0017197359999999999 2024-09-13T11:24:22.689 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:22.689 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:22.690 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:22.690 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:11.044850+0000", 2024-09-13T11:24:22.690 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.098084019999999994 2024-09-13T11:24:22.690 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:22.690 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:22.690 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:24:22.690 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:11.044977+0000", 2024-09-13T11:24:22.690 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000126284 2024-09-13T11:24:22.690 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:22.690 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:22.690 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:24:22.690 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:11.045087+0000", 2024-09-13T11:24:22.690 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000110293 2024-09-13T11:24:22.690 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:22.691 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:24:22.691 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:22.691 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:22.691 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:24:22.691 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-13T11:24:22.692 INFO:teuthology.orchestra.run.smithi001.stdout:osd.0: {} 2024-09-13T11:24:22.692 INFO:teuthology.orchestra.run.smithi001.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T11:24:22.693 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-13T11:24:22.693 INFO:tasks.ceph.osd.2.smithi162.stderr:2024-09-13T11:24:22.688+0000 7fe92b35a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26313) UID: 0 2024-09-13T11:24:22.715 INFO:teuthology.orchestra.run.smithi001.stdout:osd.1: {} 2024-09-13T11:24:22.715 INFO:teuthology.orchestra.run.smithi001.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T11:24:22.739 INFO:teuthology.orchestra.run.smithi001.stdout:osd.2: {} 2024-09-13T11:24:22.739 INFO:teuthology.orchestra.run.smithi001.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T11:24:22.766 INFO:teuthology.orchestra.run.smithi001.stdout:osd.3: {} 2024-09-13T11:24:22.766 INFO:teuthology.orchestra.run.smithi001.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T11:24:22.789 INFO:teuthology.orchestra.run.smithi001.stdout:osd.4: {} 2024-09-13T11:24:22.790 INFO:teuthology.orchestra.run.smithi001.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T11:24:22.793 INFO:tasks.ceph.osd.13.smithi092.stderr:2024-09-13T11:24:22.790+0000 7f3e5afea640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26350) UID: 0 2024-09-13T11:24:22.819 INFO:teuthology.orchestra.run.smithi001.stdout:osd.5: {} 2024-09-13T11:24:22.819 INFO:teuthology.orchestra.run.smithi001.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T11:24:22.845 INFO:teuthology.orchestra.run.smithi001.stdout:osd.6: {} 2024-09-13T11:24:22.845 INFO:teuthology.orchestra.run.smithi001.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T11:24:22.871 INFO:teuthology.orchestra.run.smithi001.stdout:osd.7: {} 2024-09-13T11:24:22.871 INFO:teuthology.orchestra.run.smithi001.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T11:24:22.873 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-13T11:24:22.873 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:24:22.890 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-13T11:24:22.894 INFO:tasks.ceph.osd.13.smithi092.stderr:2024-09-13T11:24:22.890+0000 7f3e5afea640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26350) UID: 0 2024-09-13T11:24:22.900 INFO:teuthology.orchestra.run.smithi001.stdout:osd.8: {} 2024-09-13T11:24:22.900 INFO:teuthology.orchestra.run.smithi001.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-13T11:24:22.931 INFO:teuthology.orchestra.run.smithi001.stdout:osd.9: {} 2024-09-13T11:24:22.931 INFO:teuthology.orchestra.run.smithi001.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-13T11:24:22.956 INFO:teuthology.orchestra.run.smithi001.stdout:osd.10: {} 2024-09-13T11:24:22.956 INFO:teuthology.orchestra.run.smithi001.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-13T11:24:22.981 INFO:teuthology.orchestra.run.smithi001.stdout:osd.11: {} 2024-09-13T11:24:22.981 INFO:teuthology.orchestra.run.smithi001.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-13T11:24:22.994 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-09-13T11:24:22.992+0000 7f97befc9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26315) UID: 0 2024-09-13T11:24:23.004 INFO:teuthology.orchestra.run.smithi001.stdout:osd.12: {} 2024-09-13T11:24:23.004 INFO:teuthology.orchestra.run.smithi001.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-13T11:24:23.009 INFO:tasks.radosbench.radosbench.0.smithi177.stdout: 116 16 108311 108295 58.3428 80.6875 0.0781267 0.0171338 2024-09-13T11:24:23.024 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-13T11:24:23.024 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:24:23.030 INFO:teuthology.orchestra.run.smithi001.stdout:osd.13: {} 2024-09-13T11:24:23.031 INFO:teuthology.orchestra.run.smithi001.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-13T11:24:23.040 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-13T11:24:23.055 INFO:teuthology.orchestra.run.smithi001.stdout:osd.14: {} 2024-09-13T11:24:23.055 INFO:teuthology.orchestra.run.smithi001.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-13T11:24:23.065 INFO:teuthology.orchestra.run.smithi001.stderr:nodeep-scrub is unset 2024-09-13T11:24:23.066 INFO:tasks.ceph.osd.8.smithi001.stderr:2024-09-13T11:24:23.063+0000 7fb9ab2df640 -1 osd.8 116 handle_osd_map injecting map crc failure 2024-09-13T11:24:23.074 INFO:tasks.ceph.osd.15.smithi177.stderr:2024-09-13T11:24:23.071+0000 7f9d638a8640 -1 osd.15 116 handle_osd_map injecting map crc failure 2024-09-13T11:24:23.092 INFO:teuthology.orchestra.run.smithi001.stdout:osd.15: {} 2024-09-13T11:24:23.092 INFO:teuthology.orchestra.run.smithi001.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-13T11:24:23.096 INFO:tasks.ceph.osd.7.smithi177.stderr:2024-09-13T11:24:23.095+0000 7f897b51d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26575) UID: 0 2024-09-13T11:24:23.198 INFO:tasks.ceph.osd.7.smithi177.stderr:2024-09-13T11:24:23.195+0000 7f897b51d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26575) UID: 0 2024-09-13T11:24:23.216 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-13T11:24:23.216 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:24:23.233 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-09-13T11:24:23.297 INFO:tasks.ceph.osd.5.smithi092.stderr:2024-09-13T11:24:23.290+0000 7f6f0b705640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26346) UID: 0 2024-09-13T11:24:23.360 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-13T11:24:23.361 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:24:23.375 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-09-13T11:24:23.398 INFO:tasks.ceph.osd.7.smithi177.stderr:2024-09-13T11:24:23.395+0000 7f897b51d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26575) UID: 0 2024-09-13T11:24:23.497 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-13T11:24:23.497 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:24:23.498 INFO:tasks.ceph.osd.7.smithi177.stderr:2024-09-13T11:24:23.495+0000 7f897b51d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26575) UID: 0 2024-09-13T11:24:23.511 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-09-13T11:24:23.599 INFO:tasks.ceph.osd.3.smithi177.stderr:2024-09-13T11:24:23.595+0000 7f10adba4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26574) UID: 0 2024-09-13T11:24:23.632 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-13T11:24:23.632 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:24:23.646 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_ops_in_flight 2024-09-13T11:24:23.700 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-13T11:24:23.699+0000 7f6a68775640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26307) UID: 0 2024-09-13T11:24:23.768 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-13T11:24:23.768 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:24:23.782 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_blocked_ops 2024-09-13T11:24:23.826 INFO:tasks.ceph.osd.10.smithi162.stderr:2024-09-13T11:24:23.824+0000 7fb59c9f0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26317) UID: 0 2024-09-13T11:24:23.903 INFO:tasks.ceph.osd.3.smithi177.stderr:2024-09-13T11:24:23.899+0000 7f10adba4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26574) UID: 0 2024-09-13T11:24:23.903 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-13T11:24:23.903 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:24:23.917 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_historic_ops 2024-09-13T11:24:24.003 INFO:tasks.ceph.osd.11.smithi177.stderr:2024-09-13T11:24:23.999+0000 7f24792c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26580) UID: 0 2024-09-13T11:24:24.009 INFO:tasks.radosbench.radosbench.0.smithi177.stdout: 117 16 108422 108406 57.9034 6.9375 0.0112277 0.0171586 2024-09-13T11:24:24.091 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-13T11:24:24.092 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:24:24.104 INFO:tasks.ceph.osd.15.smithi177.stderr:2024-09-13T11:24:24.103+0000 7f9d6ce77640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26578) UID: 0 2024-09-13T11:24:24.111 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-13T11:24:24.205 INFO:tasks.ceph.osd.8.smithi001.stderr:2024-09-13T11:24:24.203+0000 7fb9b48ae640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26312) UID: 0 2024-09-13T11:24:24.292 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-09-13T11:24:24.292 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:24:24.305 INFO:tasks.ceph.osd.10.smithi162.stderr:2024-09-13T11:24:24.300+0000 7fb59c9f0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26317) UID: 0 2024-09-13T11:24:24.318 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-13T11:24:24.406 INFO:tasks.ceph.osd.2.smithi162.stderr:2024-09-13T11:24:24.404+0000 7fe92b35a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26313) UID: 0 2024-09-13T11:24:24.494 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-09-13T11:24:24.494 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:24:24.507 INFO:tasks.ceph.osd.5.smithi092.stderr:2024-09-13T11:24:24.502+0000 7f6f0b705640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26346) UID: 0 2024-09-13T11:24:24.521 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-13T11:24:24.609 INFO:tasks.ceph.osd.14.smithi162.stderr:2024-09-13T11:24:24.604+0000 7f235c0db640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26319) UID: 0 2024-09-13T11:24:24.693 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-09-13T11:24:24.693 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:24:24.709 INFO:tasks.ceph.osd.2.smithi162.stderr:2024-09-13T11:24:24.704+0000 7fe92b35a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26313) UID: 0 2024-09-13T11:24:24.718 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-13T11:24:24.810 INFO:tasks.ceph.osd.14.smithi162.stderr:2024-09-13T11:24:24.808+0000 7f235c0db640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26319) UID: 0 2024-09-13T11:24:24.886 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-09-13T11:24:24.886 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:24:24.908 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-13T11:24:24.913 INFO:tasks.ceph.osd.13.smithi092.stderr:2024-09-13T11:24:24.906+0000 7f3e5afea640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26350) UID: 0 2024-09-13T11:24:25.009 INFO:tasks.radosbench.radosbench.0.smithi177.stdout: 118 16 110177 110161 58.3422 109.688 0.00336373 0.0171324 2024-09-13T11:24:25.011 INFO:tasks.ceph.osd.4.smithi001.stderr:2024-09-13T11:24:25.007+0000 7ff8e0218640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26310) UID: 0 2024-09-13T11:24:25.064 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-09-13T11:24:25.065 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:24:25.081 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-13T11:24:25.084 INFO:tasks.ceph.osd.2.smithi162.stderr:2024-09-13T11:24:25.080+0000 7fe91fd87640 -1 osd.2 118 handle_osd_map injecting map crc failure 2024-09-13T11:24:25.090 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-13T11:24:25.112 INFO:tasks.ceph.osd.9.smithi092.stderr:2024-09-13T11:24:25.106+0000 7ff5ab3d0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26348) UID: 0 2024-09-13T11:24:25.116 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T11:24:25.203 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-09-13T11:24:25.203 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:24:25.213 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-13T11:24:25.211+0000 7f6a68775640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26307) UID: 0 2024-09-13T11:24:25.215 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-09-13T11:24:25.284 INFO:teuthology.orchestra.run.smithi001.stdout:osd.0: {} 2024-09-13T11:24:25.285 INFO:teuthology.orchestra.run.smithi001.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T11:24:25.309 INFO:teuthology.orchestra.run.smithi001.stdout:osd.1: {} 2024-09-13T11:24:25.310 INFO:teuthology.orchestra.run.smithi001.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T11:24:25.314 INFO:tasks.ceph.osd.2.smithi162.stderr:2024-09-13T11:24:25.312+0000 7fe92b35a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26313) UID: 0 2024-09-13T11:24:25.323 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-09-13T11:24:25.324 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:24:25.335 INFO:teuthology.orchestra.run.smithi001.stdout:osd.2: {} 2024-09-13T11:24:25.335 INFO:teuthology.orchestra.run.smithi001.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T11:24:25.339 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-09-13T11:24:25.358 INFO:teuthology.orchestra.run.smithi001.stdout:osd.3: {} 2024-09-13T11:24:25.358 INFO:teuthology.orchestra.run.smithi001.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T11:24:25.381 INFO:teuthology.orchestra.run.smithi001.stdout:osd.4: {} 2024-09-13T11:24:25.381 INFO:teuthology.orchestra.run.smithi001.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T11:24:25.405 INFO:teuthology.orchestra.run.smithi001.stdout:osd.5: {} 2024-09-13T11:24:25.405 INFO:teuthology.orchestra.run.smithi001.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T11:24:25.415 INFO:tasks.ceph.osd.9.smithi092.stderr:2024-09-13T11:24:25.410+0000 7ff5ab3d0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26348) UID: 0 2024-09-13T11:24:25.433 INFO:teuthology.orchestra.run.smithi001.stdout:osd.6: {} 2024-09-13T11:24:25.433 INFO:teuthology.orchestra.run.smithi001.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T11:24:25.455 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-09-13T11:24:25.455 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:24:25.461 INFO:teuthology.orchestra.run.smithi001.stdout:osd.7: {} 2024-09-13T11:24:25.461 INFO:teuthology.orchestra.run.smithi001.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T11:24:25.468 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-09-13T11:24:25.490 INFO:teuthology.orchestra.run.smithi001.stdout:osd.8: {} 2024-09-13T11:24:25.490 INFO:teuthology.orchestra.run.smithi001.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-13T11:24:25.516 INFO:tasks.ceph.osd.15.smithi177.stderr:2024-09-13T11:24:25.515+0000 7f9d6ce77640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26578) UID: 0 2024-09-13T11:24:25.523 INFO:teuthology.orchestra.run.smithi001.stdout:osd.9: {} 2024-09-13T11:24:25.523 INFO:teuthology.orchestra.run.smithi001.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-13T11:24:25.556 INFO:teuthology.orchestra.run.smithi001.stdout:osd.10: {} 2024-09-13T11:24:25.556 INFO:teuthology.orchestra.run.smithi001.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-13T11:24:25.577 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-09-13T11:24:25.577 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:24:25.589 INFO:teuthology.orchestra.run.smithi001.stdout:osd.11: {} 2024-09-13T11:24:25.592 INFO:teuthology.orchestra.run.smithi001.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-13T11:24:25.593 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_ops_in_flight 2024-09-13T11:24:25.616 INFO:tasks.ceph.osd.2.smithi162.stderr:2024-09-13T11:24:25.612+0000 7fe92b35a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26313) UID: 0 2024-09-13T11:24:25.622 INFO:teuthology.orchestra.run.smithi001.stdout:osd.12: {} 2024-09-13T11:24:25.622 INFO:teuthology.orchestra.run.smithi001.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-13T11:24:25.663 INFO:teuthology.orchestra.run.smithi001.stdout:osd.13: {} 2024-09-13T11:24:25.663 INFO:teuthology.orchestra.run.smithi001.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-13T11:24:25.698 INFO:teuthology.orchestra.run.smithi001.stdout:osd.14: {} 2024-09-13T11:24:25.698 INFO:teuthology.orchestra.run.smithi001.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-13T11:24:25.717 INFO:tasks.ceph.osd.11.smithi177.stderr:2024-09-13T11:24:25.715+0000 7f24792c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26580) UID: 0 2024-09-13T11:24:25.739 INFO:teuthology.orchestra.run.smithi001.stdout:osd.15: {} 2024-09-13T11:24:25.739 INFO:teuthology.orchestra.run.smithi001.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-13T11:24:25.754 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-09-13T11:24:25.754 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [], 2024-09-13T11:24:25.754 INFO:teuthology.orchestra.run.smithi177.stdout: "num_ops": 0 2024-09-13T11:24:25.754 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-09-13T11:24:25.766 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_blocked_ops 2024-09-13T11:24:25.818 INFO:tasks.ceph.osd.5.smithi092.stderr:2024-09-13T11:24:25.814+0000 7f6f0b705640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26346) UID: 0 2024-09-13T11:24:25.919 INFO:tasks.ceph.osd.2.smithi162.stderr:2024-09-13T11:24:25.916+0000 7fe92b35a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26313) UID: 0 2024-09-13T11:24:25.920 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-09-13T11:24:25.920 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [], 2024-09-13T11:24:25.920 INFO:teuthology.orchestra.run.smithi177.stdout: "complaint_time": 30, 2024-09-13T11:24:25.920 INFO:teuthology.orchestra.run.smithi177.stdout: "num_blocked_ops": 0 2024-09-13T11:24:25.920 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-09-13T11:24:25.933 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_historic_ops 2024-09-13T11:24:26.009 INFO:tasks.radosbench.radosbench.0.smithi177.stdout: 119 16 110303 110287 57.918 7.875 0.00560397 0.017177 2024-09-13T11:24:26.020 INFO:tasks.ceph.osd.12.smithi001.stderr:2024-09-13T11:24:26.019+0000 7f892c6e6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26314) UID: 0 2024-09-13T11:24:26.057 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-09-13T11:24:26.057 INFO:teuthology.orchestra.run.smithi177.stdout: "size": 20, 2024-09-13T11:24:26.057 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 600, 2024-09-13T11:24:26.057 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [ 2024-09-13T11:24:26.057 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:26.057 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:25383 3.0 e55/30 3:0a6bb869:::benchmark_data_smithi177_31116_object25382:head v 55'1620, mlcod=55'1617)", 2024-09-13T11:24:26.057 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:04.449956+0000", 2024-09-13T11:24:26.057 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 81.603075692999994, 2024-09-13T11:24:26.057 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.21143479600000001, 2024-09-13T11:24:26.057 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:26.057 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:26.057 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:26.058 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:26.058 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:26.058 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:26.058 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.449956+0000", 2024-09-13T11:24:26.058 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:26.058 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:26.058 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:26.058 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:26.058 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.449956+0000", 2024-09-13T11:24:26.058 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:26.058 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:26.058 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:26.058 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:26.058 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.449959+0000", 2024-09-13T11:24:26.059 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.393e-06 2024-09-13T11:24:26.059 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:26.059 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:26.059 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:26.059 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.450014+0000", 2024-09-13T11:24:26.059 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.4200000000000003e-05 2024-09-13T11:24:26.059 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:26.059 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:26.059 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:26.059 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.658637+0000", 2024-09-13T11:24:26.059 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.20862334799999999 2024-09-13T11:24:26.059 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:26.060 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:26.060 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:26.060 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.658650+0000", 2024-09-13T11:24:26.060 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3148e-05 2024-09-13T11:24:26.060 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:26.060 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:26.060 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:26.060 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.658867+0000", 2024-09-13T11:24:26.060 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00021673499999999999 2024-09-13T11:24:26.060 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:26.060 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:26.060 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:24:26.060 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.658931+0000", 2024-09-13T11:24:26.061 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.4045000000000001e-05 2024-09-13T11:24:26.061 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:26.061 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:26.061 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:24:26.061 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.661317+0000", 2024-09-13T11:24:26.061 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.002385823 2024-09-13T11:24:26.061 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:26.061 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:26.061 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:24:26.061 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.661391+0000", 2024-09-13T11:24:26.061 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.4103999999999994e-05 2024-09-13T11:24:26.061 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:26.061 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:26.061 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:26.062 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:26.062 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:26.062 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPush(3.16 85/70 [PushOp(3:6ac421eb:::benchmark_data_smithi177_31116_object43910:head, version: 66'2836, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:6ac421eb:::benchmark_data_smithi177_31116_object43910:head@66'2836, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:24:26.062 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:42.145291+0000", 2024-09-13T11:24:26.062 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 43.907740486999998, 2024-09-13T11:24:26.062 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.148514919, 2024-09-13T11:24:26.062 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:26.062 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:26.062 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:24:26.062 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:26.062 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:26.062 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:26.062 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.145291+0000", 2024-09-13T11:24:26.062 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:26.063 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:26.063 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:26.063 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:26.063 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.145291+0000", 2024-09-13T11:24:26.063 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:26.063 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:26.063 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:26.063 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:26.063 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.145293+0000", 2024-09-13T11:24:26.063 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.699e-06 2024-09-13T11:24:26.063 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:26.063 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:26.063 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:26.064 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.145417+0000", 2024-09-13T11:24:26.064 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00012415200000000001 2024-09-13T11:24:26.064 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:26.064 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:26.064 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:26.064 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.292964+0000", 2024-09-13T11:24:26.064 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.14754715400000001 2024-09-13T11:24:26.064 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:26.064 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:26.064 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:26.064 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.292981+0000", 2024-09-13T11:24:26.064 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.6790999999999999e-05 2024-09-13T11:24:26.064 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:26.065 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:26.065 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:26.065 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.293083+0000", 2024-09-13T11:24:26.065 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00010199799999999999 2024-09-13T11:24:26.065 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:26.065 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:26.065 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:24:26.065 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.293098+0000", 2024-09-13T11:24:26.065 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.4612e-05 2024-09-13T11:24:26.065 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:26.065 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:26.065 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:24:26.065 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.293806+0000", 2024-09-13T11:24:26.066 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00070851300000000002 2024-09-13T11:24:26.066 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:26.066 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:26.066 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:26.066 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:26.066 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:26.066 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:60390 3.16 e85/70 3:6a388186:::benchmark_data_smithi177_31116_object60389:head v 85'3357, mlcod=66'2835)", 2024-09-13T11:24:26.066 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:42.149102+0000", 2024-09-13T11:24:26.066 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 43.903929429999998, 2024-09-13T11:24:26.066 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.14821161799999999, 2024-09-13T11:24:26.066 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:26.066 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:26.066 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:26.066 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:26.067 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:26.067 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:26.067 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.149102+0000", 2024-09-13T11:24:26.067 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:26.067 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:26.067 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:26.067 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:26.068 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.149102+0000", 2024-09-13T11:24:26.068 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:26.068 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:26.068 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:26.068 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:26.068 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.149104+0000", 2024-09-13T11:24:26.068 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.708e-06 2024-09-13T11:24:26.068 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:26.068 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:26.068 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:26.069 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.149160+0000", 2024-09-13T11:24:26.069 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.5618000000000002e-05 2024-09-13T11:24:26.069 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:26.069 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:26.069 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:26.069 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.296071+0000", 2024-09-13T11:24:26.069 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.14691148600000001 2024-09-13T11:24:26.069 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:26.069 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:26.069 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:26.069 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.296076+0000", 2024-09-13T11:24:26.069 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.5580000000000002e-06 2024-09-13T11:24:26.070 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:26.070 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:26.070 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:26.070 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.296127+0000", 2024-09-13T11:24:26.070 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.1405e-05 2024-09-13T11:24:26.070 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:26.070 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:26.070 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:24:26.070 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.296150+0000", 2024-09-13T11:24:26.070 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.3113e-05 2024-09-13T11:24:26.070 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:26.070 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:26.071 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:24:26.071 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.297274+0000", 2024-09-13T11:24:26.071 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0011239900000000001 2024-09-13T11:24:26.071 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:26.071 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:26.071 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:24:26.071 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.297314+0000", 2024-09-13T11:24:26.071 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.9740000000000002e-05 2024-09-13T11:24:26.071 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:26.071 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:26.071 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:26.072 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:26.072 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:26.072 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPush(3.16 85/70 [PushOp(3:69aac9fd:::benchmark_data_smithi177_31116_object43914:head, version: 66'2837, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:69aac9fd:::benchmark_data_smithi177_31116_object43914:head@66'2837, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:24:26.072 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:42.295995+0000", 2024-09-13T11:24:26.072 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 43.757036372999998, 2024-09-13T11:24:26.072 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.215089431, 2024-09-13T11:24:26.072 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:26.072 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:26.072 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:24:26.072 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:26.072 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:26.073 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:26.073 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.295995+0000", 2024-09-13T11:24:26.073 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:26.073 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:26.073 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:26.073 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:26.073 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.295995+0000", 2024-09-13T11:24:26.073 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:26.073 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:26.073 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:26.073 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:26.074 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.295997+0000", 2024-09-13T11:24:26.074 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3170000000000001e-06 2024-09-13T11:24:26.074 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:26.074 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:26.074 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:26.074 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.296058+0000", 2024-09-13T11:24:26.074 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.1690000000000001e-05 2024-09-13T11:24:26.074 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:26.074 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:26.074 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:26.074 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.509606+0000", 2024-09-13T11:24:26.075 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.21354748300000001 2024-09-13T11:24:26.075 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:26.075 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:26.075 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:26.075 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.509638+0000", 2024-09-13T11:24:26.075 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.1644999999999999e-05 2024-09-13T11:24:26.075 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:26.075 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:26.075 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:26.075 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.509814+0000", 2024-09-13T11:24:26.075 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00017679499999999999 2024-09-13T11:24:26.076 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:26.076 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:26.076 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:24:26.076 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.509840+0000", 2024-09-13T11:24:26.076 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.5871e-05 2024-09-13T11:24:26.076 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:26.076 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:26.076 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:24:26.076 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.511085+0000", 2024-09-13T11:24:26.076 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00124463 2024-09-13T11:24:26.076 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:26.076 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:26.077 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:26.077 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:26.077 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:26.077 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPush(3.1d 91/78 [PushOp(3:be6a0ae4:::benchmark_data_smithi177_31116_object25671:head, version: 58'1569, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:be6a0ae4:::benchmark_data_smithi177_31116_object25671:head@58'1569, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:24:26.077 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:48.481971+0000", 2024-09-13T11:24:26.077 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 37.571061127999997, 2024-09-13T11:24:26.077 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.47945454199999998, 2024-09-13T11:24:26.077 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:26.077 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:26.077 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:24:26.077 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:26.078 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:26.078 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:26.078 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.481971+0000", 2024-09-13T11:24:26.078 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:26.078 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:26.078 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:26.078 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:26.078 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.481971+0000", 2024-09-13T11:24:26.078 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:26.078 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:26.078 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:26.078 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:26.079 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.481972+0000", 2024-09-13T11:24:26.079 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5179999999999999e-06 2024-09-13T11:24:26.079 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:26.079 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:26.079 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:26.079 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.482000+0000", 2024-09-13T11:24:26.079 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.8249e-05 2024-09-13T11:24:26.079 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:26.079 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:26.079 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:26.079 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.959926+0000", 2024-09-13T11:24:26.080 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.47792585100000001 2024-09-13T11:24:26.080 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:26.080 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:26.080 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:26.080 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.959944+0000", 2024-09-13T11:24:26.080 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.7620000000000001e-05 2024-09-13T11:24:26.080 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:26.080 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:26.080 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:26.080 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.960059+0000", 2024-09-13T11:24:26.080 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000115188 2024-09-13T11:24:26.081 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:26.081 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:26.081 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:24:26.081 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.960074+0000", 2024-09-13T11:24:26.081 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.4738e-05 2024-09-13T11:24:26.081 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:26.081 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:26.081 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:24:26.081 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.961425+0000", 2024-09-13T11:24:26.081 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0013513780000000001 2024-09-13T11:24:26.081 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:26.081 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:26.082 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:26.082 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:26.082 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:26.082 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:71884 3.16 e96/88 3:6ca846b2:::benchmark_data_smithi177_31116_object71883:head v 96'3755, mlcod=96'3753)", 2024-09-13T11:24:26.082 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:54.108299+0000", 2024-09-13T11:24:26.082 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 31.944732954999999, 2024-09-13T11:24:26.082 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.20685140099999999, 2024-09-13T11:24:26.082 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:26.083 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:26.083 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:26.083 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:26.083 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:26.083 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:26.083 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.108299+0000", 2024-09-13T11:24:26.083 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:26.083 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:26.083 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:26.083 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:26.084 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.108299+0000", 2024-09-13T11:24:26.084 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:26.084 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:26.084 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:26.084 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:26.084 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.108300+0000", 2024-09-13T11:24:26.084 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.173e-06 2024-09-13T11:24:26.084 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:26.084 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:26.084 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:26.084 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.108323+0000", 2024-09-13T11:24:26.085 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.2591000000000001e-05 2024-09-13T11:24:26.085 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:26.085 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:26.085 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:26.085 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.312377+0000", 2024-09-13T11:24:26.085 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.20405456599999999 2024-09-13T11:24:26.085 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:26.085 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:26.085 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:26.085 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.312382+0000", 2024-09-13T11:24:26.085 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.1429999999999997e-06 2024-09-13T11:24:26.085 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:26.086 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:26.086 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:26.086 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.312480+0000", 2024-09-13T11:24:26.086 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.7742e-05 2024-09-13T11:24:26.086 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:26.086 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:26.086 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:24:26.086 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.312521+0000", 2024-09-13T11:24:26.086 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.1369000000000003e-05 2024-09-13T11:24:26.086 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:26.086 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:26.086 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:24:26.086 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.315105+0000", 2024-09-13T11:24:26.087 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0025835060000000002 2024-09-13T11:24:26.087 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:26.087 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:26.087 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:24:26.087 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.315150+0000", 2024-09-13T11:24:26.087 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.5311e-05 2024-09-13T11:24:26.087 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:26.087 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:26.087 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:26.087 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:26.087 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:26.088 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:71927 3.0 e97/75 3:0197244a:::benchmark_data_smithi177_31116_object71926:head v 97'3351, mlcod=96'3348)", 2024-09-13T11:24:26.088 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:54.200161+0000", 2024-09-13T11:24:26.088 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 31.852870726999999, 2024-09-13T11:24:26.088 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.26691738999999998, 2024-09-13T11:24:26.088 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:26.088 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:26.088 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:26.088 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:26.088 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:26.088 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:26.088 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.200161+0000", 2024-09-13T11:24:26.089 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:26.089 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:26.089 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:26.089 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:26.089 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.200161+0000", 2024-09-13T11:24:26.089 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:26.089 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:26.089 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:26.089 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:26.089 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.200162+0000", 2024-09-13T11:24:26.089 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.78e-07 2024-09-13T11:24:26.089 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:26.090 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:26.090 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:26.090 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.200200+0000", 2024-09-13T11:24:26.090 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.8121000000000003e-05 2024-09-13T11:24:26.090 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:26.090 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:26.090 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:26.090 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.465397+0000", 2024-09-13T11:24:26.090 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.26519681099999998 2024-09-13T11:24:26.090 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:26.090 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:26.090 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:26.091 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.465404+0000", 2024-09-13T11:24:26.091 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.7399999999999998e-06 2024-09-13T11:24:26.091 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:26.091 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:26.091 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:26.091 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.465456+0000", 2024-09-13T11:24:26.091 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.2626000000000001e-05 2024-09-13T11:24:26.091 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:26.091 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:26.091 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:24:26.091 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.465482+0000", 2024-09-13T11:24:26.091 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.6086000000000001e-05 2024-09-13T11:24:26.092 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:26.092 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:26.092 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:24:26.092 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.467035+0000", 2024-09-13T11:24:26.092 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0015523729999999999 2024-09-13T11:24:26.092 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:26.092 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:26.092 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:24:26.092 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.467078+0000", 2024-09-13T11:24:26.092 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.3754999999999997e-05 2024-09-13T11:24:26.092 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:26.093 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:26.093 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:26.093 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:26.093 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:26.093 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:72250 3.0 e97/75 3:010c7171:::benchmark_data_smithi177_31116_object72249:head v 97'3353, mlcod=96'3350)", 2024-09-13T11:24:26.093 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:54.516381+0000", 2024-09-13T11:24:26.093 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 31.536651213999999, 2024-09-13T11:24:26.093 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.13663347300000001, 2024-09-13T11:24:26.093 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:26.093 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:26.093 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:26.094 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:26.094 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:26.094 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:26.094 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.516381+0000", 2024-09-13T11:24:26.094 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:26.094 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:26.094 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:26.094 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:26.094 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.516381+0000", 2024-09-13T11:24:26.094 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:26.094 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:26.094 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:26.095 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:26.095 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.516382+0000", 2024-09-13T11:24:26.095 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3239999999999999e-06 2024-09-13T11:24:26.095 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:26.095 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:26.095 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:26.095 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.516437+0000", 2024-09-13T11:24:26.095 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.5408999999999998e-05 2024-09-13T11:24:26.095 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:26.095 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:26.095 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:26.096 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.651731+0000", 2024-09-13T11:24:26.096 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.135293583 2024-09-13T11:24:26.096 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:26.096 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:26.096 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:26.096 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.651736+0000", 2024-09-13T11:24:26.096 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.3410000000000002e-06 2024-09-13T11:24:26.096 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:26.096 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:26.096 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:26.096 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.651791+0000", 2024-09-13T11:24:26.096 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.4758e-05 2024-09-13T11:24:26.097 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:26.097 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:26.097 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:24:26.097 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.651814+0000", 2024-09-13T11:24:26.097 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.3283999999999999e-05 2024-09-13T11:24:26.097 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:26.097 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:26.097 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:24:26.097 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.652975+0000", 2024-09-13T11:24:26.097 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0011604370000000001 2024-09-13T11:24:26.097 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:26.097 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:26.098 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:24:26.098 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.653014+0000", 2024-09-13T11:24:26.098 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.9337e-05 2024-09-13T11:24:26.098 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:26.098 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:26.098 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:26.098 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:26.098 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:26.098 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:72993 3.0 e97/75 3:02747b8f:::benchmark_data_smithi177_31116_object72992:head v 97'3366, mlcod=97'3363)", 2024-09-13T11:24:26.098 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:55.116009+0000", 2024-09-13T11:24:26.098 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 30.937022607999999, 2024-09-13T11:24:26.098 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.13962660099999999, 2024-09-13T11:24:26.099 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:26.099 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:26.099 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:26.099 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:26.099 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:26.099 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:26.099 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:55.116009+0000", 2024-09-13T11:24:26.099 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:26.099 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:26.099 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:26.099 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:26.099 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:55.116009+0000", 2024-09-13T11:24:26.100 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:26.100 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:26.100 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:26.100 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:26.100 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:55.116010+0000", 2024-09-13T11:24:26.100 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.291e-06 2024-09-13T11:24:26.100 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:26.100 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:26.100 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:26.100 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:55.116062+0000", 2024-09-13T11:24:26.100 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.1949e-05 2024-09-13T11:24:26.101 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:26.101 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:26.101 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:26.101 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:55.254100+0000", 2024-09-13T11:24:26.101 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.13803723000000001 2024-09-13T11:24:26.101 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:26.101 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:26.101 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:26.101 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:55.254106+0000", 2024-09-13T11:24:26.101 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.5939999999999999e-06 2024-09-13T11:24:26.101 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:26.101 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:26.102 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:26.102 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:55.254196+0000", 2024-09-13T11:24:26.102 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.9567999999999994e-05 2024-09-13T11:24:26.102 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:26.102 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:26.102 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:24:26.102 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:55.254219+0000", 2024-09-13T11:24:26.102 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.3266000000000001e-05 2024-09-13T11:24:26.102 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:26.102 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:26.102 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:24:26.102 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:55.255596+0000", 2024-09-13T11:24:26.103 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0013767409999999999 2024-09-13T11:24:26.103 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:26.103 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:26.103 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:24:26.103 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:55.255636+0000", 2024-09-13T11:24:26.103 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.9962000000000001e-05 2024-09-13T11:24:26.103 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:26.103 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:26.103 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:26.103 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:26.103 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:26.104 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:73523 3.0 e98/75 3:004115ff:::benchmark_data_smithi177_31116_object73522:head v 98'3382, mlcod=98'3378)", 2024-09-13T11:24:26.104 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:55.538309+0000", 2024-09-13T11:24:26.104 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 30.51472261, 2024-09-13T11:24:26.104 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.127489824, 2024-09-13T11:24:26.104 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:26.104 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:26.104 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:26.104 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:26.104 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:26.104 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:26.104 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:55.538309+0000", 2024-09-13T11:24:26.105 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:26.105 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:26.105 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:26.105 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:26.105 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:55.538309+0000", 2024-09-13T11:24:26.105 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:26.105 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:26.105 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:26.105 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:26.105 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:55.538310+0000", 2024-09-13T11:24:26.106 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.237e-06 2024-09-13T11:24:26.106 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:26.106 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:26.106 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:26.106 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:55.538343+0000", 2024-09-13T11:24:26.106 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.2480000000000001e-05 2024-09-13T11:24:26.106 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:26.106 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:26.106 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:26.106 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:55.663154+0000", 2024-09-13T11:24:26.106 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.124810988 2024-09-13T11:24:26.106 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:26.107 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:26.107 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:26.107 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:55.663161+0000", 2024-09-13T11:24:26.107 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.2389999999999996e-06 2024-09-13T11:24:26.107 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:26.107 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:26.107 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:26.107 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:55.663215+0000", 2024-09-13T11:24:26.107 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.3893000000000001e-05 2024-09-13T11:24:26.107 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:26.107 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:26.108 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:24:26.108 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:55.663235+0000", 2024-09-13T11:24:26.108 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.9624e-05 2024-09-13T11:24:26.108 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:26.108 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:26.108 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:24:26.108 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:55.665764+0000", 2024-09-13T11:24:26.108 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0025292859999999999 2024-09-13T11:24:26.108 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:26.108 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:26.109 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:24:26.109 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:55.665799+0000", 2024-09-13T11:24:26.109 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.5077000000000003e-05 2024-09-13T11:24:26.109 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:26.109 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:26.109 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:26.109 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:26.109 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:26.109 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:79634 3.5 3:a6c32e6f:::benchmark_data_smithi177_31116_object79633:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101)", 2024-09-13T11:24:26.109 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:00.168616+0000", 2024-09-13T11:24:26.109 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 25.884415339, 2024-09-13T11:24:26.109 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.169445552, 2024-09-13T11:24:26.110 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:26.110 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:26.110 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:26.110 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:24:26.110 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:24:26.110 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:26.110 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 79634 2024-09-13T11:24:26.110 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:26.110 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:26.110 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:26.110 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:26.110 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.168616+0000", 2024-09-13T11:24:26.111 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:26.111 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:26.111 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:26.111 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:26.111 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.168616+0000", 2024-09-13T11:24:26.111 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:26.111 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:26.111 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:26.111 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:26.111 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.168619+0000", 2024-09-13T11:24:26.111 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.565e-06 2024-09-13T11:24:26.111 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:26.111 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:26.112 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:26.112 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.168639+0000", 2024-09-13T11:24:26.112 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.0384999999999999e-05 2024-09-13T11:24:26.112 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:26.112 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:26.112 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:26.112 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.168652+0000", 2024-09-13T11:24:26.112 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.2779000000000001e-05 2024-09-13T11:24:26.112 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:26.112 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:26.112 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:26.112 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.168658+0000", 2024-09-13T11:24:26.113 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.2269999999999998e-06 2024-09-13T11:24:26.113 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:26.113 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:26.113 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:26.113 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.168722+0000", 2024-09-13T11:24:26.113 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.3888999999999998e-05 2024-09-13T11:24:26.113 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:26.113 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:26.113 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:24:26.113 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.168863+0000", 2024-09-13T11:24:26.113 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00014050799999999999 2024-09-13T11:24:26.113 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:26.114 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:26.114 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 13", 2024-09-13T11:24:26.114 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.169010+0000", 2024-09-13T11:24:26.114 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00014746200000000001 2024-09-13T11:24:26.114 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:26.114 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:26.114 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:24:26.114 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.170891+0000", 2024-09-13T11:24:26.114 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001880945 2024-09-13T11:24:26.114 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:26.114 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:26.114 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:26.114 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.337961+0000", 2024-09-13T11:24:26.114 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.16706952 2024-09-13T11:24:26.115 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:26.115 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:26.115 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:24:26.115 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.338019+0000", 2024-09-13T11:24:26.115 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.7958000000000003e-05 2024-09-13T11:24:26.115 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:26.115 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:26.115 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:24:26.115 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.338062+0000", 2024-09-13T11:24:26.115 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.3313999999999999e-05 2024-09-13T11:24:26.115 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:26.115 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:26.116 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:26.116 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:26.116 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:26.116 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPush(3.1b 102/78 [PushOp(3:df786fc7:::benchmark_data_smithi177_31116_object48840:head, version: 71'3079, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:df786fc7:::benchmark_data_smithi177_31116_object48840:head@71'3079, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:24:26.116 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:00.290536+0000", 2024-09-13T11:24:26.116 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 25.762495964999999, 2024-09-13T11:24:26.116 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.20017585399999999, 2024-09-13T11:24:26.116 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:26.116 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:26.116 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:24:26.116 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:26.116 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:26.116 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:26.117 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.290536+0000", 2024-09-13T11:24:26.117 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:26.117 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:26.117 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:26.117 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:26.117 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.290536+0000", 2024-09-13T11:24:26.117 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:26.117 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:26.117 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:26.117 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:26.117 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.290537+0000", 2024-09-13T11:24:26.117 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.327e-06 2024-09-13T11:24:26.117 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:26.118 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:26.118 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:26.118 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.290581+0000", 2024-09-13T11:24:26.118 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.3893999999999997e-05 2024-09-13T11:24:26.118 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:26.118 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:26.118 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:26.118 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.488588+0000", 2024-09-13T11:24:26.118 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.198006708 2024-09-13T11:24:26.118 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:26.118 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:26.118 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:26.119 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.488607+0000", 2024-09-13T11:24:26.119 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.9162999999999999e-05 2024-09-13T11:24:26.119 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:26.119 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:26.119 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:26.119 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.488747+0000", 2024-09-13T11:24:26.119 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00014040799999999999 2024-09-13T11:24:26.119 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:26.119 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:26.119 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:24:26.119 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.488770+0000", 2024-09-13T11:24:26.119 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.279e-05 2024-09-13T11:24:26.120 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:26.120 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:26.120 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:24:26.120 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.490712+0000", 2024-09-13T11:24:26.120 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001941564 2024-09-13T11:24:26.120 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:26.120 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:26.120 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:26.120 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:26.120 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:26.120 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPush(3.1b 102/78 [PushOp(3:d82bf6ec:::benchmark_data_smithi177_31116_object48847:head, version: 71'3080, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:d82bf6ec:::benchmark_data_smithi177_31116_object48847:head@71'3080, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:24:26.120 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:00.293979+0000", 2024-09-13T11:24:26.120 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 25.759052438000001, 2024-09-13T11:24:26.121 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.20314832099999999, 2024-09-13T11:24:26.121 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:26.121 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:26.121 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:24:26.121 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:26.121 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:26.121 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:26.121 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.293979+0000", 2024-09-13T11:24:26.121 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:26.121 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:26.121 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:26.121 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:26.121 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.293979+0000", 2024-09-13T11:24:26.121 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:26.122 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:26.122 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:26.122 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:26.122 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.293981+0000", 2024-09-13T11:24:26.122 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.4449999999999999e-06 2024-09-13T11:24:26.122 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:26.122 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:26.122 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:26.122 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.294085+0000", 2024-09-13T11:24:26.122 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000103848 2024-09-13T11:24:26.122 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:26.122 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:26.122 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:26.122 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.495571+0000", 2024-09-13T11:24:26.123 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.20148640800000001 2024-09-13T11:24:26.123 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:26.123 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:26.123 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:26.123 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.495609+0000", 2024-09-13T11:24:26.123 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.7904000000000002e-05 2024-09-13T11:24:26.123 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:26.123 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:26.123 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:26.123 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.495729+0000", 2024-09-13T11:24:26.123 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000120556 2024-09-13T11:24:26.123 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:26.123 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:26.124 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:24:26.124 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.495751+0000", 2024-09-13T11:24:26.124 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.1511000000000001e-05 2024-09-13T11:24:26.124 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:26.124 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:26.124 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:24:26.124 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.497128+0000", 2024-09-13T11:24:26.124 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001376649 2024-09-13T11:24:26.124 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:26.124 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:26.124 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:26.124 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:26.125 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:26.125 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:79801 3.5 3:a6efdf84:::benchmark_data_smithi177_31116_object79800:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102)", 2024-09-13T11:24:26.125 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:00.403201+0000", 2024-09-13T11:24:26.125 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 25.649830851000001, 2024-09-13T11:24:26.125 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.31868626300000003, 2024-09-13T11:24:26.125 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:26.125 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:26.125 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:26.125 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:24:26.125 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:24:26.125 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:26.125 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 79801 2024-09-13T11:24:26.126 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:26.126 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:26.126 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:26.126 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:26.126 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.403201+0000", 2024-09-13T11:24:26.126 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:26.126 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:26.126 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:26.126 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:26.126 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.403201+0000", 2024-09-13T11:24:26.126 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:26.126 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:26.127 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:26.127 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:26.127 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.403204+0000", 2024-09-13T11:24:26.127 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.0819999999999999e-06 2024-09-13T11:24:26.127 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:26.127 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:26.127 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:26.127 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.403260+0000", 2024-09-13T11:24:26.127 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.5624999999999997e-05 2024-09-13T11:24:26.127 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:26.127 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:26.127 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:26.127 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.403280+0000", 2024-09-13T11:24:26.128 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.0060000000000001e-05 2024-09-13T11:24:26.128 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:26.128 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:26.128 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:26.128 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.403291+0000", 2024-09-13T11:24:26.128 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.1335000000000001e-05 2024-09-13T11:24:26.128 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:26.128 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:26.128 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:26.128 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.403344+0000", 2024-09-13T11:24:26.128 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.3217000000000002e-05 2024-09-13T11:24:26.128 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:26.128 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:26.129 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:24:26.129 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.403456+0000", 2024-09-13T11:24:26.129 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000111706 2024-09-13T11:24:26.129 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:26.129 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:26.129 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 13", 2024-09-13T11:24:26.129 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.403547+0000", 2024-09-13T11:24:26.129 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.1556000000000002e-05 2024-09-13T11:24:26.129 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:26.129 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:26.129 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:24:26.129 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.407436+0000", 2024-09-13T11:24:26.130 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.003888856 2024-09-13T11:24:26.130 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:26.130 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:26.130 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:26.130 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.721722+0000", 2024-09-13T11:24:26.130 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.31428562700000001 2024-09-13T11:24:26.130 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:26.130 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:26.130 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:24:26.130 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.721814+0000", 2024-09-13T11:24:26.130 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.1564000000000006e-05 2024-09-13T11:24:26.130 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:26.131 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:26.131 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:24:26.131 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.721887+0000", 2024-09-13T11:24:26.131 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.3634999999999995e-05 2024-09-13T11:24:26.131 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:26.131 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:26.131 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:26.131 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:26.131 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:26.131 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPush(3.1b 102/78 [PushOp(3:dcbefe42:::benchmark_data_smithi177_31116_object49001:head, version: 71'3087, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:dcbefe42:::benchmark_data_smithi177_31116_object49001:head@71'3087, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:24:26.131 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:00.511027+0000", 2024-09-13T11:24:26.131 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 25.542004915, 2024-09-13T11:24:26.131 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.13427292699999999, 2024-09-13T11:24:26.132 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:26.132 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:26.132 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:24:26.132 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:26.132 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:26.132 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:26.132 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.511027+0000", 2024-09-13T11:24:26.132 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:26.132 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:26.132 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:26.132 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:26.133 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.511027+0000", 2024-09-13T11:24:26.133 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:26.133 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:26.133 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:26.133 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:26.133 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.511028+0000", 2024-09-13T11:24:26.133 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.4869999999999999e-06 2024-09-13T11:24:26.133 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:26.133 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:26.133 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:26.133 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.511095+0000", 2024-09-13T11:24:26.133 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.6382000000000002e-05 2024-09-13T11:24:26.134 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:26.134 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:26.134 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:26.134 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.643319+0000", 2024-09-13T11:24:26.134 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.13222455699999999 2024-09-13T11:24:26.134 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:26.134 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:26.134 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:26.134 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.643349+0000", 2024-09-13T11:24:26.134 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.9356000000000002e-05 2024-09-13T11:24:26.134 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:26.134 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:26.135 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:26.135 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.643518+0000", 2024-09-13T11:24:26.135 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00016963500000000001 2024-09-13T11:24:26.135 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:26.135 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:26.135 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:24:26.135 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.643541+0000", 2024-09-13T11:24:26.135 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.2367000000000001e-05 2024-09-13T11:24:26.135 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:26.135 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:26.135 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:24:26.135 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.645300+0000", 2024-09-13T11:24:26.136 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0017591429999999999 2024-09-13T11:24:26.136 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:26.136 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:26.136 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:26.136 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:26.136 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:26.136 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:79813 3.16 e102/88 3:6c2df1ae:::benchmark_data_smithi177_31116_object79812:head v 102'4016, mlcod=102'4014)", 2024-09-13T11:24:26.136 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:00.516959+0000", 2024-09-13T11:24:26.136 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 25.53607289, 2024-09-13T11:24:26.136 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.15075599100000001, 2024-09-13T11:24:26.136 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:26.136 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:26.136 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:26.137 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:26.137 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:26.137 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:26.137 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.516959+0000", 2024-09-13T11:24:26.137 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:26.137 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:26.137 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:26.137 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:26.137 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.516959+0000", 2024-09-13T11:24:26.137 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:26.137 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:26.137 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:26.138 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:26.138 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.516961+0000", 2024-09-13T11:24:26.138 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.0279999999999999e-06 2024-09-13T11:24:26.138 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:26.138 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:26.138 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:26.138 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.517002+0000", 2024-09-13T11:24:26.138 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.0754999999999999e-05 2024-09-13T11:24:26.138 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:26.138 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:26.138 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:26.138 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.664491+0000", 2024-09-13T11:24:26.139 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.14748925099999999 2024-09-13T11:24:26.139 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:26.139 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:26.139 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:26.139 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.664504+0000", 2024-09-13T11:24:26.139 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.2747e-05 2024-09-13T11:24:26.139 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:26.139 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:26.139 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:26.139 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.664642+0000", 2024-09-13T11:24:26.139 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00013850900000000001 2024-09-13T11:24:26.139 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:26.140 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:26.140 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:24:26.140 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.664678+0000", 2024-09-13T11:24:26.140 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.5403e-05 2024-09-13T11:24:26.140 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:26.140 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:26.140 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:24:26.140 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.667646+0000", 2024-09-13T11:24:26.140 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.002968133 2024-09-13T11:24:26.140 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:26.140 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:26.140 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:24:26.140 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.667715+0000", 2024-09-13T11:24:26.141 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.9165000000000006e-05 2024-09-13T11:24:26.141 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:26.141 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:26.141 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:26.141 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:26.141 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:26.141 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:83199 3.1d e106/102 3:be2c6fae:::benchmark_data_smithi177_31116_object83198:head v 106'4244, mlcod=106'4242)", 2024-09-13T11:24:26.141 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:04.298299+0000", 2024-09-13T11:24:26.141 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 21.754732724, 2024-09-13T11:24:26.141 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.25001107099999997, 2024-09-13T11:24:26.141 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:26.141 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:26.142 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:26.142 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:26.142 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:26.142 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:26.142 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.298299+0000", 2024-09-13T11:24:26.142 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:26.142 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:26.142 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:26.142 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:26.142 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.298299+0000", 2024-09-13T11:24:26.142 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:26.142 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:26.142 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:26.143 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:26.143 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.298301+0000", 2024-09-13T11:24:26.143 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5859999999999999e-06 2024-09-13T11:24:26.143 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:26.143 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:26.143 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:26.143 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.298368+0000", 2024-09-13T11:24:26.143 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.6928999999999996e-05 2024-09-13T11:24:26.143 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:26.143 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:26.143 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:26.143 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.545180+0000", 2024-09-13T11:24:26.143 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.24681209700000001 2024-09-13T11:24:26.144 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:26.144 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:26.144 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:26.144 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.545193+0000", 2024-09-13T11:24:26.144 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3427e-05 2024-09-13T11:24:26.144 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:26.144 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:26.144 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:26.144 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.545327+0000", 2024-09-13T11:24:26.144 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00013359400000000001 2024-09-13T11:24:26.144 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:26.144 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:26.144 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:24:26.145 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.545385+0000", 2024-09-13T11:24:26.145 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.8492000000000002e-05 2024-09-13T11:24:26.145 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:26.145 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:26.145 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:24:26.145 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.548225+0000", 2024-09-13T11:24:26.145 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.002839472 2024-09-13T11:24:26.145 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:26.145 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:26.145 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:24:26.145 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.548310+0000", 2024-09-13T11:24:26.145 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.5474000000000002e-05 2024-09-13T11:24:26.146 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:26.146 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:26.146 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:26.146 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:26.146 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:26.146 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:84621 3.1d e107/102 3:becd72aa:::benchmark_data_smithi177_31116_object84620:head v 107'4296, mlcod=107'4293)", 2024-09-13T11:24:26.146 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:05.742238+0000", 2024-09-13T11:24:26.146 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 20.310793623999999, 2024-09-13T11:24:26.146 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.17888174300000001, 2024-09-13T11:24:26.146 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:26.146 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:26.146 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:26.146 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:26.147 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:26.147 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:26.147 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:05.742238+0000", 2024-09-13T11:24:26.147 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:26.147 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:26.147 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:26.147 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:26.147 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:05.742238+0000", 2024-09-13T11:24:26.147 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:26.147 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:26.147 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:26.147 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:26.147 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:05.742240+0000", 2024-09-13T11:24:26.148 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.3159999999999999e-06 2024-09-13T11:24:26.148 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:26.148 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:26.148 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:26.148 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:05.742315+0000", 2024-09-13T11:24:26.148 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.4122000000000006e-05 2024-09-13T11:24:26.148 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:26.148 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:26.148 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:26.148 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:05.917757+0000", 2024-09-13T11:24:26.148 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.175442031 2024-09-13T11:24:26.148 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:26.148 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:26.149 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:26.149 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:05.917764+0000", 2024-09-13T11:24:26.149 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.661e-06 2024-09-13T11:24:26.149 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:26.149 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:26.149 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:26.149 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:05.917846+0000", 2024-09-13T11:24:26.149 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.2006000000000001e-05 2024-09-13T11:24:26.149 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:26.149 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:26.149 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:24:26.149 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:05.917879+0000", 2024-09-13T11:24:26.149 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.2796000000000002e-05 2024-09-13T11:24:26.150 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:26.150 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:26.150 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:24:26.150 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:05.921070+0000", 2024-09-13T11:24:26.150 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0031908489999999999 2024-09-13T11:24:26.150 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:26.150 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:26.150 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:24:26.150 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:05.921120+0000", 2024-09-13T11:24:26.150 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.9962e-05 2024-09-13T11:24:26.150 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:26.150 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:26.151 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:26.151 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:26.151 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:26.151 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:100995 3.1d e114/102 3:b9d3a02e:::benchmark_data_smithi177_31116_object100994:head v 114'4771, mlcod=114'4769)", 2024-09-13T11:24:26.151 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:17.338503+0000", 2024-09-13T11:24:26.151 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 8.7145289599999991, 2024-09-13T11:24:26.151 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.12755466100000001, 2024-09-13T11:24:26.151 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:26.151 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:26.151 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:26.151 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:26.151 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:26.151 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:26.152 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:17.338503+0000", 2024-09-13T11:24:26.152 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:26.152 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:26.152 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:26.152 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:26.152 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:17.338503+0000", 2024-09-13T11:24:26.152 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:26.152 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:26.152 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:26.152 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:26.152 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:17.338504+0000", 2024-09-13T11:24:26.152 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.635e-06 2024-09-13T11:24:26.153 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:26.153 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:26.153 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:26.153 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:17.338575+0000", 2024-09-13T11:24:26.153 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.0869000000000002e-05 2024-09-13T11:24:26.153 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:26.153 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:26.153 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:26.153 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:17.462756+0000", 2024-09-13T11:24:26.153 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.124180943 2024-09-13T11:24:26.153 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:26.153 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:26.153 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:26.154 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:17.462764+0000", 2024-09-13T11:24:26.154 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.3170000000000004e-06 2024-09-13T11:24:26.154 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:26.154 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:26.154 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:26.154 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:17.462887+0000", 2024-09-13T11:24:26.154 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00012359200000000001 2024-09-13T11:24:26.154 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:26.154 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:26.154 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:24:26.154 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:17.462948+0000", 2024-09-13T11:24:26.154 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.0563999999999998e-05 2024-09-13T11:24:26.154 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:26.155 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:26.155 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:24:26.155 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:17.465984+0000", 2024-09-13T11:24:26.155 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0030363669999999999 2024-09-13T11:24:26.155 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:26.155 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:26.155 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:24:26.155 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:17.466057+0000", 2024-09-13T11:24:26.155 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.3374000000000006e-05 2024-09-13T11:24:26.155 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:26.155 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:26.155 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:26.156 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:26.156 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:26.156 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:107910 3.16 e116/88 3:6cf57aa3:::benchmark_data_smithi177_31116_object107909:head v 116'4911, mlcod=116'4908)", 2024-09-13T11:24:26.156 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:22.516225+0000", 2024-09-13T11:24:26.156 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 3.5368070810000001, 2024-09-13T11:24:26.156 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.18319666600000001, 2024-09-13T11:24:26.156 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:26.156 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:26.156 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:26.156 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:26.156 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:26.156 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:26.156 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:22.516225+0000", 2024-09-13T11:24:26.157 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:26.157 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:26.157 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:26.157 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:26.157 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:22.516225+0000", 2024-09-13T11:24:26.157 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:26.157 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:26.157 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:26.157 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:26.157 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:22.516226+0000", 2024-09-13T11:24:26.157 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.716e-06 2024-09-13T11:24:26.157 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:26.157 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:26.158 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:26.158 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:22.516250+0000", 2024-09-13T11:24:26.158 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.3975e-05 2024-09-13T11:24:26.158 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:26.158 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:26.158 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:26.158 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:22.697686+0000", 2024-09-13T11:24:26.158 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.18143539 2024-09-13T11:24:26.158 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:26.158 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:26.158 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:26.158 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:22.697692+0000", 2024-09-13T11:24:26.159 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.0769999999999996e-06 2024-09-13T11:24:26.159 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:26.159 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:26.159 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:26.159 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:22.697744+0000", 2024-09-13T11:24:26.159 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.2525000000000003e-05 2024-09-13T11:24:26.159 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:26.159 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:26.159 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:24:26.159 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:22.697757+0000", 2024-09-13T11:24:26.159 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.2819e-05 2024-09-13T11:24:26.159 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:26.159 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:26.160 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:24:26.160 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:22.699388+0000", 2024-09-13T11:24:26.160 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0016303999999999999 2024-09-13T11:24:26.160 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:26.160 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:26.160 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:24:26.160 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:22.699421+0000", 2024-09-13T11:24:26.160 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.3763999999999997e-05 2024-09-13T11:24:26.160 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:26.160 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:26.160 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:26.160 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:26.161 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:26.161 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-09-13T11:24:26.161 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15] out_osds: [] dead_osds: [] live_osds: [0, 4, 8, 12, 5, 9, 13, 2, 6, 10, 14, 3, 7, 11, 15, 1] 2024-09-13T11:24:26.161 INFO:tasks.thrashosds.thrasher:choose_action: min_in 2 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-13T11:24:26.161 INFO:tasks.thrashosds.thrasher:test_bluestore_reshard started 2024-09-13T11:24:26.161 INFO:tasks.thrashosds.thrasher:Killing osd 14, live_osds are [0, 4, 8, 12, 5, 9, 13, 2, 6, 10, 14, 3, 7, 11, 15, 1] 2024-09-13T11:24:26.163 DEBUG:tasks.ceph.osd.14:waiting for process to exit 2024-09-13T11:24:26.163 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T11:24:26.165 INFO:teuthology.orchestra.run.smithi001.stderr:noscrub is set 2024-09-13T11:24:26.165 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-13T11:24:26.209 INFO:tasks.ceph.osd.2.smithi162.stderr:2024-09-13T11:24:26.204+0000 7fe92b35a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26313) UID: 0 2024-09-13T11:24:26.253 INFO:tasks.ceph.osd.14:Stopped 2024-09-13T11:24:26.253 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd down 14 2024-09-13T11:24:26.265 INFO:tasks.ceph.osd.11.smithi177.stderr:2024-09-13T11:24:26.263+0000 7f24792c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26580) UID: 0 2024-09-13T11:24:26.347 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-13T11:24:26.347 INFO:teuthology.orchestra.run.smithi001.stdout: "ops": [], 2024-09-13T11:24:26.347 INFO:teuthology.orchestra.run.smithi001.stdout: "num_ops": 0 2024-09-13T11:24:26.347 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-13T11:24:26.363 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-13T11:24:26.366 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-13T11:24:26.363+0000 7f6a68775640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26307) UID: 0 2024-09-13T11:24:26.467 INFO:tasks.ceph.osd.11.smithi177.stderr:2024-09-13T11:24:26.463+0000 7f24792c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26580) UID: 0 2024-09-13T11:24:26.489 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-13T11:24:26.490 INFO:teuthology.orchestra.run.smithi001.stdout: "ops": [], 2024-09-13T11:24:26.490 INFO:teuthology.orchestra.run.smithi001.stdout: "complaint_time": 30, 2024-09-13T11:24:26.490 INFO:teuthology.orchestra.run.smithi001.stdout: "num_blocked_ops": 0 2024-09-13T11:24:26.490 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-13T11:24:26.504 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-13T11:24:26.568 INFO:tasks.ceph.osd.3.smithi177.stderr:2024-09-13T11:24:26.567+0000 7f10adba4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26574) UID: 0 2024-09-13T11:24:26.590 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-09-13T11:24:26.587+0000 7fb2a5596640 -1 mon.a@0(leader).osd e120 definitely_dead 0 2024-09-13T11:24:26.632 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-13T11:24:26.632 INFO:teuthology.orchestra.run.smithi001.stdout: "size": 20, 2024-09-13T11:24:26.633 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 600, 2024-09-13T11:24:26.633 INFO:teuthology.orchestra.run.smithi001.stdout: "ops": [ 2024-09-13T11:24:26.633 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:26.633 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDPGPushReply(3.1e 79/78 [PushReplyOp(3:7a3a85df:::benchmark_data_smithi177_31116_object2607:head)])", 2024-09-13T11:24:26.633 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:36.410722+0000", 2024-09-13T11:24:26.633 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 50.218229616000002, 2024-09-13T11:24:26.633 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.15957022300000001, 2024-09-13T11:24:26.633 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:24:26.633 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:24:26.633 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-13T11:24:26.633 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:24:26.633 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:26.634 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:24:26.634 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:36.410722+0000", 2024-09-13T11:24:26.634 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:26.634 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:26.634 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:26.634 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:24:26.634 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:36.410722+0000", 2024-09-13T11:24:26.634 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:26.634 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:26.634 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:26.634 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:24:26.634 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:36.410724+0000", 2024-09-13T11:24:26.634 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.367e-06 2024-09-13T11:24:26.634 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:26.635 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:26.635 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:24:26.635 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:36.410729+0000", 2024-09-13T11:24:26.635 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.4870000000000002e-06 2024-09-13T11:24:26.635 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:26.635 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:26.635 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:24:26.635 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:36.568151+0000", 2024-09-13T11:24:26.635 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.15742154999999999 2024-09-13T11:24:26.635 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:26.635 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:26.635 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:24:26.635 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:36.568162+0000", 2024-09-13T11:24:26.636 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.117e-05 2024-09-13T11:24:26.636 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:26.636 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:26.636 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:24:26.636 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:36.568214+0000", 2024-09-13T11:24:26.636 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.2187e-05 2024-09-13T11:24:26.636 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:26.636 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:26.636 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:24:26.636 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:36.570292+0000", 2024-09-13T11:24:26.636 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0020784620000000001 2024-09-13T11:24:26.636 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:26.636 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:24:26.637 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:26.637 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:26.637 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:26.637 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDPGPush(3.20 83/77 [PushOp(3:05148b46:::benchmark_data_smithi177_31116_object1401:head, version: 43'74, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:05148b46:::benchmark_data_smithi177_31116_object1401:head@43'74, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:24:26.637 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:40.912122+0000", 2024-09-13T11:24:26.637 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 45.716829789000002, 2024-09-13T11:24:26.637 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.158959773, 2024-09-13T11:24:26.637 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:24:26.637 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:24:26.637 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:24:26.638 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:24:26.638 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:26.638 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:24:26.638 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:40.912122+0000", 2024-09-13T11:24:26.638 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:26.638 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:26.638 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:26.638 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:24:26.638 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:40.912122+0000", 2024-09-13T11:24:26.638 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:26.638 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:26.638 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:26.638 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:24:26.638 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:40.912123+0000", 2024-09-13T11:24:26.639 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0589999999999999e-06 2024-09-13T11:24:26.639 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:26.639 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:26.639 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:24:26.639 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:40.912176+0000", 2024-09-13T11:24:26.639 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.2431e-05 2024-09-13T11:24:26.639 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:26.639 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:26.639 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:24:26.639 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:41.070227+0000", 2024-09-13T11:24:26.639 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.15805137499999999 2024-09-13T11:24:26.639 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:26.640 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:26.640 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:24:26.640 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:41.070243+0000", 2024-09-13T11:24:26.640 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.5948999999999999e-05 2024-09-13T11:24:26.640 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:26.640 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:26.640 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:24:26.640 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:41.070360+0000", 2024-09-13T11:24:26.640 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000117267 2024-09-13T11:24:26.640 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:26.640 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:26.640 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:24:26.640 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:41.070373+0000", 2024-09-13T11:24:26.641 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2665e-05 2024-09-13T11:24:26.641 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:26.641 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:26.641 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:24:26.641 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:41.071082+0000", 2024-09-13T11:24:26.641 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00070902699999999999 2024-09-13T11:24:26.641 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:26.641 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:24:26.641 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:26.641 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:26.641 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:26.642 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDPGPushReply(3.1e 84/78 [PushReplyOp(3:79ee54b9:::benchmark_data_smithi177_31116_object11701:head)])", 2024-09-13T11:24:26.642 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:41.407335+0000", 2024-09-13T11:24:26.642 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 45.221616521999998, 2024-09-13T11:24:26.642 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.51622350400000006, 2024-09-13T11:24:26.642 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:24:26.642 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:24:26.642 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-13T11:24:26.642 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:24:26.642 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:26.642 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:24:26.642 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:41.407335+0000", 2024-09-13T11:24:26.642 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:26.643 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:26.643 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:26.643 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:24:26.643 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:41.407335+0000", 2024-09-13T11:24:26.643 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:26.643 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:26.643 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:26.643 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:24:26.643 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:41.407337+0000", 2024-09-13T11:24:26.643 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.505e-06 2024-09-13T11:24:26.643 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:26.643 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:26.643 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:24:26.644 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:41.407343+0000", 2024-09-13T11:24:26.644 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.9089999999999997e-06 2024-09-13T11:24:26.644 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:26.644 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:26.644 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:24:26.644 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:41.921977+0000", 2024-09-13T11:24:26.644 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.51463463499999995 2024-09-13T11:24:26.644 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:26.644 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:26.644 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:24:26.644 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:41.921988+0000", 2024-09-13T11:24:26.644 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.092e-05 2024-09-13T11:24:26.644 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:26.645 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:26.645 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:24:26.645 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:41.922088+0000", 2024-09-13T11:24:26.645 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.9528e-05 2024-09-13T11:24:26.645 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:26.645 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:26.645 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:24:26.645 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:41.923559+0000", 2024-09-13T11:24:26.645 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0014710070000000001 2024-09-13T11:24:26.645 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:26.645 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:24:26.645 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:26.645 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:26.646 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:26.646 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4665.0:60417 3.10 e85/84 3:092808b0:::benchmark_data_smithi177_31116_object60416:head v 85'3295, mlcod=85'3292)", 2024-09-13T11:24:26.646 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:42.174488+0000", 2024-09-13T11:24:26.646 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 44.454463967999999, 2024-09-13T11:24:26.646 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.387908896, 2024-09-13T11:24:26.646 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:24:26.646 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:24:26.646 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:26.646 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:24:26.646 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:26.646 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:24:26.646 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.174488+0000", 2024-09-13T11:24:26.646 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:26.646 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:26.647 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:26.647 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:24:26.647 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.174488+0000", 2024-09-13T11:24:26.647 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:26.647 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:26.647 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:26.647 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:24:26.647 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.174490+0000", 2024-09-13T11:24:26.647 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.9099999999999999e-06 2024-09-13T11:24:26.647 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:26.647 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:26.647 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:24:26.647 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.174527+0000", 2024-09-13T11:24:26.648 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.7543000000000003e-05 2024-09-13T11:24:26.648 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:26.648 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:26.648 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:24:26.648 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.561093+0000", 2024-09-13T11:24:26.648 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.38656578400000002 2024-09-13T11:24:26.648 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:26.648 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:26.648 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:24:26.648 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.561101+0000", 2024-09-13T11:24:26.648 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.6090000000000003e-06 2024-09-13T11:24:26.648 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:26.648 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:26.649 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:24:26.649 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.561143+0000", 2024-09-13T11:24:26.649 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.2435000000000003e-05 2024-09-13T11:24:26.649 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:26.649 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:26.649 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:24:26.649 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.561164+0000", 2024-09-13T11:24:26.649 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.1276000000000001e-05 2024-09-13T11:24:26.649 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:26.649 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:26.649 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:24:26.649 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.562360+0000", 2024-09-13T11:24:26.649 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00119596 2024-09-13T11:24:26.650 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:26.650 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:26.650 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:24:26.650 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.562397+0000", 2024-09-13T11:24:26.650 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.6378999999999998e-05 2024-09-13T11:24:26.650 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:26.650 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:24:26.650 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:26.650 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:26.650 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:26.650 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDPGPush(3.22 85/78 [PushOp(3:456e8765:::benchmark_data_smithi177_31116_object16212:head, version: 52'1006, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:456e8765:::benchmark_data_smithi177_31116_object16212:head@52'1006, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:24:26.650 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:42.673067+0000", 2024-09-13T11:24:26.650 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 43.955884826000002, 2024-09-13T11:24:26.651 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.27935889899999999, 2024-09-13T11:24:26.651 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:24:26.651 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:24:26.651 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:24:26.651 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:24:26.651 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:26.651 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:24:26.651 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.673067+0000", 2024-09-13T11:24:26.651 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:26.651 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:26.651 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:26.651 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:24:26.651 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.673067+0000", 2024-09-13T11:24:26.652 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:26.652 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:26.652 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:26.652 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:24:26.652 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.673068+0000", 2024-09-13T11:24:26.652 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.9699999999999994e-07 2024-09-13T11:24:26.652 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:26.652 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:26.652 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:24:26.652 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.673090+0000", 2024-09-13T11:24:26.652 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.2416999999999999e-05 2024-09-13T11:24:26.652 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:26.652 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:26.652 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:24:26.653 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.951363+0000", 2024-09-13T11:24:26.653 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.27827291199999998 2024-09-13T11:24:26.653 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:26.653 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:26.653 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:24:26.653 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.951386+0000", 2024-09-13T11:24:26.653 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.2226999999999999e-05 2024-09-13T11:24:26.653 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:26.653 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:26.653 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:24:26.653 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.951581+0000", 2024-09-13T11:24:26.653 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000195713 2024-09-13T11:24:26.653 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:26.654 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:26.654 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:24:26.654 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.951605+0000", 2024-09-13T11:24:26.654 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.3767000000000001e-05 2024-09-13T11:24:26.654 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:26.654 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:26.654 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:24:26.654 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.952426+0000", 2024-09-13T11:24:26.654 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00082086599999999996 2024-09-13T11:24:26.654 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:26.654 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:24:26.654 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:26.654 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:26.654 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:26.655 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "replica_scrub(pg: 3.20,from:0'0,to:89'3254,epoch:89/77,start:3:04000000::::0,end:3:045ba805:::benchmark_data_smithi177_31116_object43251:0,chunky:1,deep:0,version:9,allow_preemption:1,priority=5)", 2024-09-13T11:24:26.655 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:46.641364+0000", 2024-09-13T11:24:26.655 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 39.987588144999997, 2024-09-13T11:24:26.655 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.27141393000000003, 2024-09-13T11:24:26.655 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:24:26.655 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:24:26.655 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:24:26.655 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:24:26.655 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:26.655 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:24:26.655 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.641364+0000", 2024-09-13T11:24:26.655 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:26.655 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:26.655 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:26.656 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:24:26.656 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.641364+0000", 2024-09-13T11:24:26.656 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:26.656 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:26.656 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:26.656 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:24:26.656 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.641365+0000", 2024-09-13T11:24:26.656 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.266e-06 2024-09-13T11:24:26.656 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:26.656 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:26.656 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:24:26.656 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.641370+0000", 2024-09-13T11:24:26.656 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.8539999999999996e-06 2024-09-13T11:24:26.656 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:26.657 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:26.657 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:24:26.657 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.912491+0000", 2024-09-13T11:24:26.657 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.27112103100000001 2024-09-13T11:24:26.657 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:26.657 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:26.657 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:24:26.657 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.912520+0000", 2024-09-13T11:24:26.657 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.9379000000000001e-05 2024-09-13T11:24:26.657 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:26.657 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:26.657 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:24:26.657 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.912679+0000", 2024-09-13T11:24:26.657 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00015915800000000001 2024-09-13T11:24:26.657 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:26.658 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:26.658 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:24:26.658 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.912714+0000", 2024-09-13T11:24:26.658 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.4687999999999997e-05 2024-09-13T11:24:26.658 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:26.658 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:26.658 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:24:26.658 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.912778+0000", 2024-09-13T11:24:26.658 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.3553999999999995e-05 2024-09-13T11:24:26.658 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:26.658 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:24:26.658 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:26.658 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:26.658 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:26.659 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDPGPush(3.18 89/70 [PushOp(3:1ac6c361:::benchmark_data_smithi177_31116_object44972:head, version: 68'2767, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:1ac6c361:::benchmark_data_smithi177_31116_object44972:head@68'2767, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:24:26.659 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:46.671018+0000", 2024-09-13T11:24:26.659 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 39.957934012999999, 2024-09-13T11:24:26.659 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.271282931, 2024-09-13T11:24:26.659 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:24:26.659 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:24:26.659 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:24:26.659 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:24:26.659 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:26.659 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:24:26.659 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.671018+0000", 2024-09-13T11:24:26.659 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:26.659 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:26.659 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:26.660 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:24:26.660 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.671018+0000", 2024-09-13T11:24:26.660 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:26.660 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:26.660 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:26.660 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:24:26.660 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.671020+0000", 2024-09-13T11:24:26.660 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.018e-06 2024-09-13T11:24:26.660 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:26.660 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:26.660 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:24:26.660 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.671097+0000", 2024-09-13T11:24:26.660 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.7218999999999994e-05 2024-09-13T11:24:26.660 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:26.661 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:26.661 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:24:26.661 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.939773+0000", 2024-09-13T11:24:26.661 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.26867550200000001 2024-09-13T11:24:26.661 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:26.661 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:26.661 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:24:26.661 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.939791+0000", 2024-09-13T11:24:26.661 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.8185999999999998e-05 2024-09-13T11:24:26.661 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:26.661 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:26.661 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:24:26.661 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.939926+0000", 2024-09-13T11:24:26.661 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000135291 2024-09-13T11:24:26.662 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:26.662 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:26.662 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:24:26.662 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.939938+0000", 2024-09-13T11:24:26.662 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2393e-05 2024-09-13T11:24:26.662 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:26.662 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:26.662 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:24:26.662 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.942301+0000", 2024-09-13T11:24:26.662 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.002362322 2024-09-13T11:24:26.662 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:26.662 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:24:26.662 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:26.662 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:26.663 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:26.663 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "replica_scrub(pg: 3.20,from:0'0,to:90'3264,epoch:90/77,start:3:0650aac8:::benchmark_data_smithi177_31116_object1734:0,end:3:06af01e3:::benchmark_data_smithi177_31116_object31572:0,chunky:1,deep:0,version:9,allow_preemption:1,priority=5)", 2024-09-13T11:24:26.663 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:47.380615+0000", 2024-09-13T11:24:26.663 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 39.248336590000001, 2024-09-13T11:24:26.663 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.51951398400000004, 2024-09-13T11:24:26.663 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:24:26.663 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:24:26.663 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:24:26.663 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:24:26.663 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:26.663 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:24:26.663 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:47.380615+0000", 2024-09-13T11:24:26.663 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:26.663 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:26.664 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:26.664 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:24:26.664 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:47.380615+0000", 2024-09-13T11:24:26.664 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:26.664 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:26.664 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:26.664 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:24:26.664 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:47.380617+0000", 2024-09-13T11:24:26.664 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.671e-06 2024-09-13T11:24:26.664 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:26.664 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:26.664 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:24:26.664 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:47.380623+0000", 2024-09-13T11:24:26.665 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.1190000000000002e-06 2024-09-13T11:24:26.665 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:26.665 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:26.665 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:24:26.665 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:47.900050+0000", 2024-09-13T11:24:26.665 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.51942737400000005 2024-09-13T11:24:26.665 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:26.665 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:26.665 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:24:26.665 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:47.900057+0000", 2024-09-13T11:24:26.665 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.4359999999999999e-06 2024-09-13T11:24:26.665 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:26.665 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:26.665 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:24:26.666 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:47.900101+0000", 2024-09-13T11:24:26.666 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.4675000000000002e-05 2024-09-13T11:24:26.666 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:26.666 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:26.666 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:24:26.666 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:47.900110+0000", 2024-09-13T11:24:26.666 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.0569999999999993e-06 2024-09-13T11:24:26.666 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:26.666 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:26.666 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:24:26.666 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:47.900129+0000", 2024-09-13T11:24:26.666 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.9652000000000001e-05 2024-09-13T11:24:26.666 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:26.666 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:24:26.667 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:26.667 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:26.667 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:26.667 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDPGPush(3.21 91/78 [PushOp(3:8492321f:::benchmark_data_smithi177_31116_object13767:head, version: 50'840, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:8492321f:::benchmark_data_smithi177_31116_object13767:head@50'840, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:24:26.667 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:48.319251+0000", 2024-09-13T11:24:26.667 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 38.309701060999998, 2024-09-13T11:24:26.667 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.60903802200000001, 2024-09-13T11:24:26.667 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:24:26.667 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:24:26.667 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:24:26.667 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:24:26.667 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:26.667 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:24:26.667 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:48.319251+0000", 2024-09-13T11:24:26.668 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:26.668 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:26.668 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:26.668 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:24:26.668 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:48.319251+0000", 2024-09-13T11:24:26.668 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:26.668 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:26.668 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:26.668 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:24:26.668 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:48.319252+0000", 2024-09-13T11:24:26.668 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2640000000000001e-06 2024-09-13T11:24:26.668 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:26.669 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:26.669 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:24:26.669 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:48.319363+0000", 2024-09-13T11:24:26.669 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000110559 2024-09-13T11:24:26.669 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:26.669 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:26.669 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:24:26.669 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:48.927115+0000", 2024-09-13T11:24:26.669 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.60775277800000005 2024-09-13T11:24:26.669 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:26.669 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:26.670 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:24:26.670 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:48.927144+0000", 2024-09-13T11:24:26.670 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.8861e-05 2024-09-13T11:24:26.670 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:26.670 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:26.670 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:24:26.670 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:48.927318+0000", 2024-09-13T11:24:26.670 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00017399000000000001 2024-09-13T11:24:26.670 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:26.670 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:26.670 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:24:26.670 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:48.927340+0000", 2024-09-13T11:24:26.670 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.1433999999999998e-05 2024-09-13T11:24:26.670 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:26.671 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:26.671 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:24:26.671 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:48.928289+0000", 2024-09-13T11:24:26.671 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00094913599999999995 2024-09-13T11:24:26.671 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:26.671 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:24:26.671 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:26.671 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:26.671 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:26.671 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:69487 3.e 3:73f7f990:::benchmark_data_smithi177_31116_object69486:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94)", 2024-09-13T11:24:26.671 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:52.063559+0000", 2024-09-13T11:24:26.671 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 34.565393028000003, 2024-09-13T11:24:26.671 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.20206200199999999, 2024-09-13T11:24:26.671 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:24:26.672 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:24:26.672 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:26.672 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:24:26.672 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:24:26.672 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:26.672 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 69487 2024-09-13T11:24:26.672 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:26.672 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:24:26.672 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:26.672 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:24:26.672 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:52.063559+0000", 2024-09-13T11:24:26.672 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:26.672 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:26.673 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:26.673 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:24:26.673 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:52.063559+0000", 2024-09-13T11:24:26.673 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:26.673 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:26.673 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:26.673 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:24:26.673 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:52.063560+0000", 2024-09-13T11:24:26.673 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.167e-06 2024-09-13T11:24:26.673 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:26.673 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:26.673 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:24:26.673 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:52.063612+0000", 2024-09-13T11:24:26.673 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.1709999999999998e-05 2024-09-13T11:24:26.674 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:26.674 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:26.674 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:24:26.674 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:52.063623+0000", 2024-09-13T11:24:26.674 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.1251e-05 2024-09-13T11:24:26.674 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:26.674 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:26.674 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:24:26.674 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:52.063631+0000", 2024-09-13T11:24:26.674 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.7319999999999998e-06 2024-09-13T11:24:26.674 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:26.674 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:26.674 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:24:26.674 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:52.064267+0000", 2024-09-13T11:24:26.675 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00063608600000000001 2024-09-13T11:24:26.675 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:26.675 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:26.675 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:24:26.675 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:52.064407+0000", 2024-09-13T11:24:26.675 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000140686 2024-09-13T11:24:26.675 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:26.675 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:26.675 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 1", 2024-09-13T11:24:26.675 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:52.064540+0000", 2024-09-13T11:24:26.675 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000132244 2024-09-13T11:24:26.675 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:26.675 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:26.675 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:24:26.676 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:52.067019+0000", 2024-09-13T11:24:26.676 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00247919 2024-09-13T11:24:26.676 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:26.676 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:26.676 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:26.676 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:52.265543+0000", 2024-09-13T11:24:26.676 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.19852367800000001 2024-09-13T11:24:26.676 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:26.676 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:26.676 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:24:26.676 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:52.265576+0000", 2024-09-13T11:24:26.676 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.3550999999999998e-05 2024-09-13T11:24:26.676 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:26.677 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:26.677 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:24:26.677 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:52.265621+0000", 2024-09-13T11:24:26.677 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.4706999999999997e-05 2024-09-13T11:24:26.677 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:26.677 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:24:26.677 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:26.677 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:26.677 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:26.677 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4665.0:69547 3.18 e94/91 3:1b8f2c3d:::benchmark_data_smithi177_31116_object69546:head v 94'3569, mlcod=94'3565)", 2024-09-13T11:24:26.677 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:52.132749+0000", 2024-09-13T11:24:26.677 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 34.496203258000001, 2024-09-13T11:24:26.677 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.17361655200000001, 2024-09-13T11:24:26.677 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:24:26.678 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:24:26.678 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:26.678 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:24:26.678 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:26.678 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:24:26.678 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:52.132749+0000", 2024-09-13T11:24:26.678 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:26.678 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:26.678 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:26.678 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:24:26.678 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:52.132749+0000", 2024-09-13T11:24:26.678 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:26.678 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:26.678 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:26.679 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:24:26.679 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:52.132759+0000", 2024-09-13T11:24:26.679 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0910999999999999e-05 2024-09-13T11:24:26.679 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:26.679 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:26.679 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:24:26.679 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:52.132793+0000", 2024-09-13T11:24:26.679 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.3164999999999998e-05 2024-09-13T11:24:26.679 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:26.679 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:26.679 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:24:26.679 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:52.304761+0000", 2024-09-13T11:24:26.679 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.17196884900000001 2024-09-13T11:24:26.679 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:26.680 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:26.680 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:24:26.680 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:52.304768+0000", 2024-09-13T11:24:26.680 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.8000000000000001e-06 2024-09-13T11:24:26.680 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:26.680 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:26.680 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:24:26.680 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:52.304852+0000", 2024-09-13T11:24:26.680 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.3378e-05 2024-09-13T11:24:26.680 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:26.680 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:26.680 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:24:26.680 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:52.304872+0000", 2024-09-13T11:24:26.680 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.0072999999999999e-05 2024-09-13T11:24:26.681 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:26.681 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:26.681 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:24:26.681 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:52.306322+0000", 2024-09-13T11:24:26.681 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0014506370000000001 2024-09-13T11:24:26.681 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:26.681 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:26.681 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:24:26.681 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:52.306365+0000", 2024-09-13T11:24:26.681 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.2738999999999999e-05 2024-09-13T11:24:26.681 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:26.681 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:24:26.681 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:26.682 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:26.682 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:26.682 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4665.0:69556 3.18 e94/91 3:1d3dbeba:::benchmark_data_smithi177_31116_object69555:head v 94'3570, mlcod=94'3566)", 2024-09-13T11:24:26.682 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:52.142942+0000", 2024-09-13T11:24:26.682 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 34.486009525999997, 2024-09-13T11:24:26.682 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.21178485399999999, 2024-09-13T11:24:26.682 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:24:26.682 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:24:26.682 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:26.682 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:24:26.682 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:26.682 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:24:26.682 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:52.142942+0000", 2024-09-13T11:24:26.682 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:26.683 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:26.683 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:26.683 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:24:26.683 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:52.142942+0000", 2024-09-13T11:24:26.683 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:26.683 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:26.683 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:26.683 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:24:26.683 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:52.142944+0000", 2024-09-13T11:24:26.683 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.4500000000000001e-06 2024-09-13T11:24:26.683 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:26.683 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:26.683 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:24:26.683 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:52.143047+0000", 2024-09-13T11:24:26.683 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000103384 2024-09-13T11:24:26.684 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:26.684 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:26.684 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:24:26.684 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:52.353322+0000", 2024-09-13T11:24:26.684 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.21027483299999999 2024-09-13T11:24:26.684 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:26.684 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:26.684 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:24:26.684 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:52.353330+0000", 2024-09-13T11:24:26.684 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.8879999999999997e-06 2024-09-13T11:24:26.684 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:26.684 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:26.684 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:24:26.685 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:52.353402+0000", 2024-09-13T11:24:26.685 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.2155000000000003e-05 2024-09-13T11:24:26.685 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:26.685 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:26.685 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:24:26.685 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:52.353426+0000", 2024-09-13T11:24:26.685 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.3774e-05 2024-09-13T11:24:26.685 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:26.685 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:26.685 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:24:26.685 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:52.354674+0000", 2024-09-13T11:24:26.685 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.001248706 2024-09-13T11:24:26.685 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:26.685 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:26.686 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:24:26.686 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:52.354727+0000", 2024-09-13T11:24:26.686 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.2664000000000003e-05 2024-09-13T11:24:26.686 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:26.686 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:24:26.686 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:26.686 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:26.686 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:26.686 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:70749 3.1e 3:7bd7bfc6:::benchmark_data_smithi177_31116_object70748:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95)", 2024-09-13T11:24:26.686 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:53.156250+0000", 2024-09-13T11:24:26.686 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 33.472701704999999, 2024-09-13T11:24:26.686 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.22394481799999999, 2024-09-13T11:24:26.686 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:24:26.687 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:24:26.687 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:26.687 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:24:26.687 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:24:26.687 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:26.687 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 70749 2024-09-13T11:24:26.687 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:26.687 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:24:26.687 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:26.687 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:24:26.687 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:53.156250+0000", 2024-09-13T11:24:26.688 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:26.688 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:26.688 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:26.688 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:24:26.688 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:53.156250+0000", 2024-09-13T11:24:26.688 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:26.688 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:26.688 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:26.688 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:24:26.688 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:53.156252+0000", 2024-09-13T11:24:26.688 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.3460000000000001e-06 2024-09-13T11:24:26.688 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:26.688 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:26.689 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:24:26.689 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:53.156336+0000", 2024-09-13T11:24:26.689 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.3714999999999994e-05 2024-09-13T11:24:26.689 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:26.689 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:26.689 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:24:26.689 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:53.156347+0000", 2024-09-13T11:24:26.689 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0903999999999999e-05 2024-09-13T11:24:26.689 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:26.689 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:26.689 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:24:26.689 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:53.156352+0000", 2024-09-13T11:24:26.689 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.9139999999999999e-06 2024-09-13T11:24:26.690 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:26.690 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:26.690 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:24:26.690 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:53.158028+0000", 2024-09-13T11:24:26.690 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.001675958 2024-09-13T11:24:26.690 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:26.690 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:26.690 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:24:26.690 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:53.158154+0000", 2024-09-13T11:24:26.690 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00012653 2024-09-13T11:24:26.690 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:26.690 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:26.690 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 4,5", 2024-09-13T11:24:26.690 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:53.158277+0000", 2024-09-13T11:24:26.691 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000122168 2024-09-13T11:24:26.691 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:26.691 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:26.691 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:24:26.691 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:53.159585+0000", 2024-09-13T11:24:26.691 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.001308528 2024-09-13T11:24:26.691 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:26.691 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:26.691 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:26.691 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:53.163172+0000", 2024-09-13T11:24:26.691 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0035869690000000002 2024-09-13T11:24:26.691 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:26.691 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:26.691 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:26.692 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:53.379730+0000", 2024-09-13T11:24:26.692 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.21655822599999999 2024-09-13T11:24:26.692 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:26.692 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:26.692 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:24:26.692 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:53.379810+0000", 2024-09-13T11:24:26.692 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.0041000000000002e-05 2024-09-13T11:24:26.692 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:26.692 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:26.692 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:24:26.692 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:53.380195+0000", 2024-09-13T11:24:26.692 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000384519 2024-09-13T11:24:26.692 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:26.693 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:24:26.693 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:26.693 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:26.693 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:26.693 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:70830 3.1e 3:786f8cff:::benchmark_data_smithi177_31116_object70829:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96)", 2024-09-13T11:24:26.693 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:53.227814+0000", 2024-09-13T11:24:26.693 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 33.401137446, 2024-09-13T11:24:26.693 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.15346868599999999, 2024-09-13T11:24:26.693 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:24:26.693 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:24:26.693 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:26.693 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:24:26.693 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:24:26.693 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:26.694 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 70830 2024-09-13T11:24:26.694 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:26.694 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:24:26.694 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:26.694 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:24:26.694 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:53.227814+0000", 2024-09-13T11:24:26.694 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:26.694 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:26.694 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:26.694 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:24:26.694 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:53.227814+0000", 2024-09-13T11:24:26.694 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:26.694 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:26.694 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:26.695 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:24:26.695 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:53.227816+0000", 2024-09-13T11:24:26.695 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.136e-06 2024-09-13T11:24:26.695 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:26.695 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:26.695 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:24:26.695 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:53.227889+0000", 2024-09-13T11:24:26.695 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.2186000000000003e-05 2024-09-13T11:24:26.695 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:26.695 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:26.695 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:24:26.695 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:53.227903+0000", 2024-09-13T11:24:26.695 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.4337e-05 2024-09-13T11:24:26.695 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:26.696 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:26.696 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:24:26.696 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:53.227910+0000", 2024-09-13T11:24:26.696 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.4259999999999997e-06 2024-09-13T11:24:26.696 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:26.696 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:26.696 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:24:26.696 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:53.227973+0000", 2024-09-13T11:24:26.696 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.2811999999999994e-05 2024-09-13T11:24:26.696 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:26.696 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:26.696 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:24:26.696 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:53.228145+0000", 2024-09-13T11:24:26.696 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00017218100000000001 2024-09-13T11:24:26.697 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:26.697 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:26.697 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 4,5", 2024-09-13T11:24:26.697 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:53.228338+0000", 2024-09-13T11:24:26.697 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00019233999999999999 2024-09-13T11:24:26.697 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:26.697 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:26.697 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:24:26.697 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:53.229614+0000", 2024-09-13T11:24:26.697 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.001276665 2024-09-13T11:24:26.697 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:26.697 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:26.697 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:26.697 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:53.230256+0000", 2024-09-13T11:24:26.698 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00064173300000000002 2024-09-13T11:24:26.698 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:26.698 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:26.698 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:26.698 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:53.380944+0000", 2024-09-13T11:24:26.698 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.15068795500000001 2024-09-13T11:24:26.698 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:26.698 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:26.698 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:24:26.698 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:53.380984+0000", 2024-09-13T11:24:26.698 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.036e-05 2024-09-13T11:24:26.698 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:26.698 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:26.698 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:24:26.698 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:53.381283+0000", 2024-09-13T11:24:26.699 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00029855499999999999 2024-09-13T11:24:26.699 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:26.699 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:24:26.699 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:26.699 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:26.699 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:26.699 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDPGPush(3.21 96/78 [PushOp(3:862d73af:::benchmark_data_smithi177_31116_object32719:head, version: 60'2029, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:862d73af:::benchmark_data_smithi177_31116_object32719:head@60'2029, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:24:26.699 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:53.884628+0000", 2024-09-13T11:24:26.699 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 32.744323862999998, 2024-09-13T11:24:26.699 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.21020203400000001, 2024-09-13T11:24:26.699 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:24:26.699 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:24:26.699 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:24:26.700 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:24:26.700 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:26.700 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:24:26.700 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:53.884628+0000", 2024-09-13T11:24:26.700 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:26.700 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:26.700 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:26.700 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:24:26.700 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:53.884628+0000", 2024-09-13T11:24:26.700 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:26.700 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:26.700 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:26.700 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:24:26.700 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:53.884630+0000", 2024-09-13T11:24:26.701 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.68e-06 2024-09-13T11:24:26.701 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:26.701 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:26.701 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:24:26.701 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:53.884745+0000", 2024-09-13T11:24:26.701 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000115599 2024-09-13T11:24:26.701 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:26.701 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:26.701 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:24:26.701 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:54.093948+0000", 2024-09-13T11:24:26.701 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.20920306999999999 2024-09-13T11:24:26.701 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:26.701 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:26.702 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:24:26.702 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:54.093961+0000", 2024-09-13T11:24:26.702 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2815e-05 2024-09-13T11:24:26.702 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:26.702 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:26.702 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:24:26.702 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:54.094139+0000", 2024-09-13T11:24:26.702 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00017824 2024-09-13T11:24:26.702 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:26.702 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:26.702 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:24:26.702 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:54.094150+0000", 2024-09-13T11:24:26.702 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0593e-05 2024-09-13T11:24:26.702 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:26.703 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:26.703 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:24:26.703 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:54.094830+0000", 2024-09-13T11:24:26.703 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00068003699999999998 2024-09-13T11:24:26.703 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:26.703 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:24:26.703 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:26.703 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:26.703 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:26.703 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:83239 3.e 3:74547f17:::benchmark_data_smithi177_31116_object83238:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106)", 2024-09-13T11:24:26.703 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:24:04.580085+0000", 2024-09-13T11:24:26.703 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 22.04886698, 2024-09-13T11:24:26.703 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.33785762600000002, 2024-09-13T11:24:26.704 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:24:26.704 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:24:26.704 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:26.704 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:24:26.704 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:24:26.704 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:26.704 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 83239 2024-09-13T11:24:26.704 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:26.704 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:24:26.704 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:26.704 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:24:26.704 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:04.580085+0000", 2024-09-13T11:24:26.704 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:26.704 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:26.705 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:26.705 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:24:26.705 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:04.580085+0000", 2024-09-13T11:24:26.705 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:26.705 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:26.705 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:26.705 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:24:26.705 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:04.580089+0000", 2024-09-13T11:24:26.705 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.053e-06 2024-09-13T11:24:26.705 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:26.705 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:26.705 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:24:26.705 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:04.580166+0000", 2024-09-13T11:24:26.706 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.7102999999999997e-05 2024-09-13T11:24:26.706 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:26.706 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:26.706 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:24:26.706 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:04.580196+0000", 2024-09-13T11:24:26.706 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.0196e-05 2024-09-13T11:24:26.706 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:26.706 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:26.706 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:24:26.706 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:04.580212+0000", 2024-09-13T11:24:26.706 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.535e-05 2024-09-13T11:24:26.706 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:26.706 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:26.707 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:24:26.707 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:04.580336+0000", 2024-09-13T11:24:26.707 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00012431899999999999 2024-09-13T11:24:26.707 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:26.707 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:26.707 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:24:26.707 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:04.580575+0000", 2024-09-13T11:24:26.707 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00023924100000000001 2024-09-13T11:24:26.707 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:26.707 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:26.707 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 1", 2024-09-13T11:24:26.707 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:04.580798+0000", 2024-09-13T11:24:26.707 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00022301499999999999 2024-09-13T11:24:26.708 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:26.708 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:26.708 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:24:26.708 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:04.583404+0000", 2024-09-13T11:24:26.708 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0026056120000000002 2024-09-13T11:24:26.708 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:26.708 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:26.708 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:26.708 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:04.917768+0000", 2024-09-13T11:24:26.708 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.33436387499999998 2024-09-13T11:24:26.708 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:26.708 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:26.708 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:24:26.709 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:04.917859+0000", 2024-09-13T11:24:26.709 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.1732999999999997e-05 2024-09-13T11:24:26.709 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:26.709 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:26.709 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:24:26.709 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:04.917942+0000", 2024-09-13T11:24:26.709 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.3128999999999996e-05 2024-09-13T11:24:26.709 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:26.709 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:24:26.709 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:26.709 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:26.709 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:26.709 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:84571 3.21 3:85e7a789:::benchmark_data_smithi177_31116_object84570:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-09-13T11:24:26.710 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:24:05.695256+0000", 2024-09-13T11:24:26.710 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 20.933695949000001, 2024-09-13T11:24:26.710 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.18917198299999999, 2024-09-13T11:24:26.710 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:24:26.710 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:24:26.710 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:26.710 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:24:26.710 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:24:26.710 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:26.710 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 84571 2024-09-13T11:24:26.710 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:26.710 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:24:26.710 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:26.710 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:24:26.710 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:05.695256+0000", 2024-09-13T11:24:26.711 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:26.711 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:26.711 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:26.711 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:24:26.711 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:05.695256+0000", 2024-09-13T11:24:26.711 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:26.711 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:26.711 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:26.711 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:24:26.712 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:05.695258+0000", 2024-09-13T11:24:26.712 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.0530000000000001e-06 2024-09-13T11:24:26.712 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:26.712 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:26.712 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:24:26.712 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:05.695304+0000", 2024-09-13T11:24:26.712 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.5769999999999997e-05 2024-09-13T11:24:26.712 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:26.712 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:26.712 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:24:26.712 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:05.695313+0000", 2024-09-13T11:24:26.712 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.7049999999999994e-06 2024-09-13T11:24:26.712 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:26.713 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:26.713 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:24:26.713 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:05.695321+0000", 2024-09-13T11:24:26.713 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.7190000000000001e-06 2024-09-13T11:24:26.713 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:26.713 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:26.713 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:24:26.713 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:05.695404+0000", 2024-09-13T11:24:26.713 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.3188000000000004e-05 2024-09-13T11:24:26.713 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:26.713 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:26.713 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:24:26.713 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:05.695635+0000", 2024-09-13T11:24:26.713 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00023032900000000001 2024-09-13T11:24:26.714 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:26.714 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:26.714 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 8", 2024-09-13T11:24:26.714 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:05.695852+0000", 2024-09-13T11:24:26.714 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00021749599999999999 2024-09-13T11:24:26.714 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:26.714 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:26.714 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:24:26.714 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:05.697208+0000", 2024-09-13T11:24:26.714 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00135633 2024-09-13T11:24:26.714 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:26.714 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:26.714 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:26.714 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:05.884337+0000", 2024-09-13T11:24:26.715 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.187128867 2024-09-13T11:24:26.715 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:26.715 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:26.715 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:24:26.715 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:05.884391+0000", 2024-09-13T11:24:26.715 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.3724999999999998e-05 2024-09-13T11:24:26.715 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:26.715 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:26.715 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:24:26.715 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:05.884428+0000", 2024-09-13T11:24:26.715 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.6801000000000002e-05 2024-09-13T11:24:26.715 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:26.715 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:24:26.716 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:26.716 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:26.716 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:26.716 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4665.0:89731 3.20 e110/77 3:04de772d:::benchmark_data_smithi177_31116_object89730:head v 110'3675, mlcod=110'3672)", 2024-09-13T11:24:26.716 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:24:09.624756+0000", 2024-09-13T11:24:26.716 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 17.004195714000002, 2024-09-13T11:24:26.716 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.16409006900000001, 2024-09-13T11:24:26.716 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:24:26.716 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:24:26.716 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:26.716 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:24:26.716 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:26.716 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:24:26.717 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:09.624756+0000", 2024-09-13T11:24:26.717 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:26.717 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:26.717 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:26.717 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:24:26.717 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:09.624756+0000", 2024-09-13T11:24:26.717 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:26.717 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:26.717 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:26.717 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:24:26.717 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:09.624758+0000", 2024-09-13T11:24:26.717 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.4559999999999999e-06 2024-09-13T11:24:26.717 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:26.718 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:26.718 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:24:26.718 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:09.624779+0000", 2024-09-13T11:24:26.718 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.1569999999999998e-05 2024-09-13T11:24:26.718 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:26.718 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:26.718 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:24:26.718 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:09.786956+0000", 2024-09-13T11:24:26.718 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.162176447 2024-09-13T11:24:26.718 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:26.718 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:26.718 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:24:26.718 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:09.786960+0000", 2024-09-13T11:24:26.719 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.6040000000000003e-06 2024-09-13T11:24:26.719 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:26.719 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:26.719 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:24:26.719 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:09.787043+0000", 2024-09-13T11:24:26.719 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.2402000000000002e-05 2024-09-13T11:24:26.719 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:26.719 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:26.720 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:24:26.720 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:09.787087+0000", 2024-09-13T11:24:26.720 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.4849999999999999e-05 2024-09-13T11:24:26.720 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:26.720 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:26.720 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:24:26.720 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:09.788792+0000", 2024-09-13T11:24:26.720 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0017043259999999999 2024-09-13T11:24:26.720 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:26.720 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:26.720 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:24:26.720 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:09.788846+0000", 2024-09-13T11:24:26.721 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.4413999999999998e-05 2024-09-13T11:24:26.721 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:26.721 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:24:26.721 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:26.721 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:26.721 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:26.721 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4665.0:104290 3.20 e115/77 3:05d45b57:::benchmark_data_smithi177_31116_object104289:head v 115'3905, mlcod=115'3902)", 2024-09-13T11:24:26.721 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:24:20.200225+0000", 2024-09-13T11:24:26.721 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 6.4287266839999999, 2024-09-13T11:24:26.721 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.21051286899999999, 2024-09-13T11:24:26.721 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:24:26.721 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:24:26.721 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:26.722 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:24:26.722 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:26.722 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:24:26.722 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:20.200225+0000", 2024-09-13T11:24:26.722 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:26.722 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:26.722 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:26.722 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:24:26.722 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:20.200225+0000", 2024-09-13T11:24:26.722 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:26.722 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:26.722 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:26.722 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:24:26.723 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:20.200227+0000", 2024-09-13T11:24:26.723 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.897e-06 2024-09-13T11:24:26.723 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:26.723 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:26.723 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:24:26.723 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:20.200253+0000", 2024-09-13T11:24:26.723 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.5904000000000001e-05 2024-09-13T11:24:26.723 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:26.723 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:26.723 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:24:26.723 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:20.409353+0000", 2024-09-13T11:24:26.723 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.209100339 2024-09-13T11:24:26.723 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:26.724 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:26.724 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:24:26.724 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:20.409359+0000", 2024-09-13T11:24:26.724 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.1689999999999997e-06 2024-09-13T11:24:26.724 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:26.724 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:26.724 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:24:26.724 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:20.409418+0000", 2024-09-13T11:24:26.724 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.8761999999999997e-05 2024-09-13T11:24:26.724 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:26.724 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:26.724 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:24:26.724 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:20.409440+0000", 2024-09-13T11:24:26.724 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.2098000000000001e-05 2024-09-13T11:24:26.725 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:26.725 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:26.725 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:24:26.725 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:20.410705+0000", 2024-09-13T11:24:26.725 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00126483 2024-09-13T11:24:26.725 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:26.725 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:26.725 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:24:26.725 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:20.410738+0000", 2024-09-13T11:24:26.725 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.2870000000000002e-05 2024-09-13T11:24:26.725 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:26.725 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:24:26.725 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:26.725 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:26.726 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:26.726 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:110271 3.2d 3.3c2681ad (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e119)", 2024-09-13T11:24:26.726 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:24:25.403006+0000", 2024-09-13T11:24:26.726 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 1.225945496, 2024-09-13T11:24:26.726 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.68768755999999998, 2024-09-13T11:24:26.726 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:24:26.726 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:24:26.726 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-13T11:24:26.726 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:24:26.726 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:24:26.726 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:26.726 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 110271 2024-09-13T11:24:26.726 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:26.726 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:24:26.727 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:26.727 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:24:26.727 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:25.403006+0000", 2024-09-13T11:24:26.727 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:26.727 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:26.727 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:26.727 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:24:26.727 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:25.403006+0000", 2024-09-13T11:24:26.727 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:26.727 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:26.727 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:26.727 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:24:26.727 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:25.403011+0000", 2024-09-13T11:24:26.728 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.5199999999999999e-06 2024-09-13T11:24:26.728 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:26.728 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:26.728 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:24:26.728 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:25.403065+0000", 2024-09-13T11:24:26.728 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.4478999999999998e-05 2024-09-13T11:24:26.728 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:26.728 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:26.728 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:24:26.728 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:25.403097+0000", 2024-09-13T11:24:26.728 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.1636e-05 2024-09-13T11:24:26.728 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:26.728 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:26.729 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:24:26.729 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:25.403121+0000", 2024-09-13T11:24:26.729 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.3924e-05 2024-09-13T11:24:26.729 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:26.729 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:26.729 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:24:26.729 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:25.403216+0000", 2024-09-13T11:24:26.729 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.5557999999999995e-05 2024-09-13T11:24:26.729 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:26.729 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:26.729 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for peered", 2024-09-13T11:24:26.729 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:25.403234+0000", 2024-09-13T11:24:26.729 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.7104000000000001e-05 2024-09-13T11:24:26.729 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:26.730 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:26.730 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:24:26.730 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:26.090684+0000", 2024-09-13T11:24:26.730 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.68745000700000003 2024-09-13T11:24:26.730 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:26.730 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:26.730 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:24:26.730 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:26.090694+0000", 2024-09-13T11:24:26.730 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0332e-05 2024-09-13T11:24:26.730 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:26.730 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:24:26.730 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:26.730 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:26.730 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:24:26.731 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-13T11:24:26.732 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-13T11:24:26.733 INFO:tasks.ceph.osd.13.smithi092.stderr:2024-09-13T11:24:26.726+0000 7f3e5afea640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26350) UID: 0 2024-09-13T11:24:26.833 INFO:tasks.ceph.osd.4.smithi001.stderr:2024-09-13T11:24:26.831+0000 7ff8e0218640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26310) UID: 0 2024-09-13T11:24:26.865 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-13T11:24:26.865 INFO:teuthology.orchestra.run.smithi001.stdout: "ops": [], 2024-09-13T11:24:26.866 INFO:teuthology.orchestra.run.smithi001.stdout: "num_ops": 0 2024-09-13T11:24:26.866 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-13T11:24:26.880 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-13T11:24:26.934 INFO:tasks.ceph.osd.2.smithi162.stderr:2024-09-13T11:24:26.932+0000 7fe92b35a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26313) UID: 0 2024-09-13T11:24:27.009 INFO:tasks.radosbench.radosbench.0.smithi177.stdout:2024-09-13T11:24:27.009584+0000 min lat: 0.00139681 max lat: 22.1835 avg lat: 0.0171752 2024-09-13T11:24:27.010 INFO:tasks.radosbench.radosbench.0.smithi177.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-13T11:24:27.010 INFO:tasks.radosbench.radosbench.0.smithi177.stdout: 120 16 110321 110305 57.4448 1.125 0.0086047 0.0171752 2024-09-13T11:24:27.035 INFO:tasks.ceph.osd.15.smithi177.stderr:2024-09-13T11:24:27.031+0000 7f9d6ce77640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26578) UID: 0 2024-09-13T11:24:27.054 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-13T11:24:27.054 INFO:teuthology.orchestra.run.smithi001.stdout: "ops": [], 2024-09-13T11:24:27.054 INFO:teuthology.orchestra.run.smithi001.stdout: "complaint_time": 30, 2024-09-13T11:24:27.054 INFO:teuthology.orchestra.run.smithi001.stdout: "num_blocked_ops": 0 2024-09-13T11:24:27.054 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-13T11:24:27.069 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-13T11:24:27.081 INFO:teuthology.orchestra.run.smithi001.stderr:marked down osd.14. 2024-09-13T11:24:27.114 INFO:tasks.thrashosds.thrasher:Removing osd 14, in_osds are: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15] 2024-09-13T11:24:27.114 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd out 14 2024-09-13T11:24:27.156 INFO:tasks.ceph.osd.8.smithi001.stderr:2024-09-13T11:24:27.155+0000 7fb9b48ae640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26312) UID: 0 2024-09-13T11:24:27.237 INFO:tasks.ceph.osd.4.smithi001.stderr:2024-09-13T11:24:27.235+0000 7ff8e0218640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26310) UID: 0 2024-09-13T11:24:27.241 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-13T11:24:27.241 INFO:teuthology.orchestra.run.smithi001.stdout: "size": 20, 2024-09-13T11:24:27.241 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 600, 2024-09-13T11:24:27.241 INFO:teuthology.orchestra.run.smithi001.stdout: "ops": [ 2024-09-13T11:24:27.241 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.241 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDPGPush(3.11 72/70 [PushOp(3:8ae6b6fc:::benchmark_data_smithi177_31116_object7617:head, version: 46'455, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:8ae6b6fc:::benchmark_data_smithi177_31116_object7617:head@46'455, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:24:27.241 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:26.176296+0000", 2024-09-13T11:24:27.242 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 61.062087128000002, 2024-09-13T11:24:27.242 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.13349134900000001, 2024-09-13T11:24:27.242 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:24:27.242 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:24:27.242 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:24:27.242 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:24:27.242 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.242 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:24:27.242 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:26.176296+0000", 2024-09-13T11:24:27.242 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:27.242 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.242 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.242 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:24:27.242 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:26.176296+0000", 2024-09-13T11:24:27.243 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:27.243 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.243 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.243 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:24:27.243 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:26.176297+0000", 2024-09-13T11:24:27.243 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2440000000000001e-06 2024-09-13T11:24:27.243 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.243 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.243 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:24:27.243 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:26.176433+0000", 2024-09-13T11:24:27.243 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00013554999999999999 2024-09-13T11:24:27.243 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.243 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.243 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:24:27.243 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:26.308747+0000", 2024-09-13T11:24:27.244 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.132314392 2024-09-13T11:24:27.244 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.244 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.244 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:24:27.244 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:26.308768+0000", 2024-09-13T11:24:27.244 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.0758999999999999e-05 2024-09-13T11:24:27.244 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.244 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.244 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:24:27.244 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:26.308874+0000", 2024-09-13T11:24:27.244 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000105334 2024-09-13T11:24:27.244 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.244 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.245 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:24:27.245 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:26.308890+0000", 2024-09-13T11:24:27.245 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.6367000000000001e-05 2024-09-13T11:24:27.245 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.245 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.245 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:24:27.245 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:26.309788+0000", 2024-09-13T11:24:27.245 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00089770300000000002 2024-09-13T11:24:27.245 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:27.245 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:24:27.245 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:27.245 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.245 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.245 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDPGPush(3.11 72/70 [PushOp(3:8da9c977:::benchmark_data_smithi177_31116_object7736:head, version: 46'461, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:8da9c977:::benchmark_data_smithi177_31116_object7736:head@46'461, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:24:27.245 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:26.393216+0000", 2024-09-13T11:24:27.246 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 60.845167269000001, 2024-09-13T11:24:27.246 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.15683955499999999, 2024-09-13T11:24:27.246 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:24:27.246 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:24:27.246 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:24:27.246 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:24:27.246 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.246 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:24:27.246 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:26.393216+0000", 2024-09-13T11:24:27.246 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:27.246 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.246 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.246 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:24:27.246 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:26.393216+0000", 2024-09-13T11:24:27.247 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:27.247 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.247 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.247 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:24:27.247 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:26.393217+0000", 2024-09-13T11:24:27.247 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2410000000000001e-06 2024-09-13T11:24:27.247 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.247 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.247 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:24:27.247 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:26.393309+0000", 2024-09-13T11:24:27.247 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.2101000000000005e-05 2024-09-13T11:24:27.247 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.247 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.247 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:24:27.248 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:26.547936+0000", 2024-09-13T11:24:27.248 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.15462682699999999 2024-09-13T11:24:27.248 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.248 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.248 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:24:27.248 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:26.548021+0000", 2024-09-13T11:24:27.248 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.4529000000000003e-05 2024-09-13T11:24:27.248 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.248 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.248 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:24:27.248 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:26.548321+0000", 2024-09-13T11:24:27.248 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00029988399999999999 2024-09-13T11:24:27.248 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.248 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.249 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:24:27.249 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:26.548353+0000", 2024-09-13T11:24:27.249 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.2573000000000001e-05 2024-09-13T11:24:27.249 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.249 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.249 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:24:27.249 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:26.550056+0000", 2024-09-13T11:24:27.249 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0017024 2024-09-13T11:24:27.249 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:27.249 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:24:27.249 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:27.250 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.250 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.250 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4665.0:54052 3.11 e74/70 3:8c6be821:::benchmark_data_smithi177_31116_object54051:head v 74'3049, mlcod=55'1352)", 2024-09-13T11:24:27.250 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:30.314329+0000", 2024-09-13T11:24:27.250 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 56.924053874999998, 2024-09-13T11:24:27.250 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.100070243, 2024-09-13T11:24:27.250 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:24:27.250 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:24:27.250 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:27.250 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:24:27.250 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.251 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:24:27.251 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:30.314329+0000", 2024-09-13T11:24:27.251 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:27.251 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.251 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.251 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:24:27.251 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:30.314329+0000", 2024-09-13T11:24:27.251 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:27.251 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.251 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.251 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:24:27.251 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:30.314332+0000", 2024-09-13T11:24:27.251 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.1900000000000002e-06 2024-09-13T11:24:27.251 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.252 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.252 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:24:27.252 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:30.314372+0000", 2024-09-13T11:24:27.252 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.0788000000000003e-05 2024-09-13T11:24:27.252 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.252 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.252 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:24:27.252 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:30.412014+0000", 2024-09-13T11:24:27.252 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.097641172999999998 2024-09-13T11:24:27.252 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.252 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.252 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:24:27.252 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:30.412026+0000", 2024-09-13T11:24:27.252 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2551999999999999e-05 2024-09-13T11:24:27.253 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.253 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.253 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:24:27.253 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:30.412134+0000", 2024-09-13T11:24:27.253 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00010763400000000001 2024-09-13T11:24:27.253 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.253 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.253 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:24:27.253 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:30.412172+0000", 2024-09-13T11:24:27.253 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.8333e-05 2024-09-13T11:24:27.253 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.253 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.253 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:24:27.253 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:30.414340+0000", 2024-09-13T11:24:27.254 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0021680879999999999 2024-09-13T11:24:27.254 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.254 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.254 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:24:27.254 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:30.414400+0000", 2024-09-13T11:24:27.254 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.9484999999999997e-05 2024-09-13T11:24:27.254 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:27.254 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:24:27.254 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:27.254 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.254 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.254 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDPGPush(3.1a 86/78 [PushOp(3:5df655d4:::benchmark_data_smithi177_31116_object12486:head, version: 49'767, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:5df655d4:::benchmark_data_smithi177_31116_object12486:head@49'767, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:24:27.254 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:43.456210+0000", 2024-09-13T11:24:27.254 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 43.782172967000001, 2024-09-13T11:24:27.255 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.131081792, 2024-09-13T11:24:27.255 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:24:27.255 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:24:27.255 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:24:27.255 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:24:27.255 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.255 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:24:27.255 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:43.456210+0000", 2024-09-13T11:24:27.255 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:27.255 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.255 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.255 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:24:27.255 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:43.456210+0000", 2024-09-13T11:24:27.255 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:27.256 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.256 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.256 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:24:27.256 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:43.456212+0000", 2024-09-13T11:24:27.256 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.922e-06 2024-09-13T11:24:27.256 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.256 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.256 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:24:27.256 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:43.456262+0000", 2024-09-13T11:24:27.256 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.9242000000000001e-05 2024-09-13T11:24:27.256 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.256 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.256 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:24:27.256 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:43.586188+0000", 2024-09-13T11:24:27.256 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.12992674400000001 2024-09-13T11:24:27.257 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.257 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.257 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:24:27.257 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:43.586209+0000", 2024-09-13T11:24:27.257 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.031e-05 2024-09-13T11:24:27.257 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.257 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.257 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:24:27.257 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:43.586325+0000", 2024-09-13T11:24:27.257 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000115983 2024-09-13T11:24:27.257 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.257 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.257 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:24:27.257 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:43.586340+0000", 2024-09-13T11:24:27.258 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.4941e-05 2024-09-13T11:24:27.258 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.258 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.258 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:24:27.258 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:43.587292+0000", 2024-09-13T11:24:27.258 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00095264999999999998 2024-09-13T11:24:27.258 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:27.258 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:24:27.258 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:27.258 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.258 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.258 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDPGPush(3.1e 89/78 [PushOp(3:7a07f38c:::benchmark_data_smithi177_31116_object20966:head, version: 54'1402, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:7a07f38c:::benchmark_data_smithi177_31116_object20966:head@54'1402, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:24:27.258 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:46.645251+0000", 2024-09-13T11:24:27.258 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 40.593132554, 2024-09-13T11:24:27.259 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.17670883800000001, 2024-09-13T11:24:27.259 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:24:27.259 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:24:27.259 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:24:27.259 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:24:27.259 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.259 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:24:27.259 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.645251+0000", 2024-09-13T11:24:27.259 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:27.259 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.259 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.259 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:24:27.259 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.645251+0000", 2024-09-13T11:24:27.259 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:27.260 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.260 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.260 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:24:27.260 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.645253+0000", 2024-09-13T11:24:27.260 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.1399999999999998e-06 2024-09-13T11:24:27.260 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.260 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.260 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:24:27.260 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.645295+0000", 2024-09-13T11:24:27.260 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.2323000000000001e-05 2024-09-13T11:24:27.260 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.260 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.260 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:24:27.260 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.820731+0000", 2024-09-13T11:24:27.261 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.17543607999999999 2024-09-13T11:24:27.261 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.261 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.261 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:24:27.261 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.820750+0000", 2024-09-13T11:24:27.261 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.8793000000000001e-05 2024-09-13T11:24:27.261 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.261 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.261 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:24:27.261 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.820914+0000", 2024-09-13T11:24:27.261 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000164062 2024-09-13T11:24:27.261 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.261 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.261 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:24:27.261 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.820939+0000", 2024-09-13T11:24:27.262 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.4672e-05 2024-09-13T11:24:27.262 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.262 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.262 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:24:27.262 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.821960+0000", 2024-09-13T11:24:27.262 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.001020768 2024-09-13T11:24:27.262 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:27.262 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:24:27.262 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:27.262 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.262 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.262 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:64320 3.11 3:8de5de4c:::benchmark_data_smithi177_31116_object64319:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90)", 2024-09-13T11:24:27.262 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:47.132785+0000", 2024-09-13T11:24:27.262 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 40.105598252999997, 2024-09-13T11:24:27.263 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.11946314600000001, 2024-09-13T11:24:27.263 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:24:27.263 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:24:27.263 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:27.263 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:24:27.263 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:24:27.263 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:27.263 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 64320 2024-09-13T11:24:27.263 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.263 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:24:27.263 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.263 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:24:27.263 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:47.132785+0000", 2024-09-13T11:24:27.263 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:27.264 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.264 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.264 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:24:27.264 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:47.132785+0000", 2024-09-13T11:24:27.264 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:27.264 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.264 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.264 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:24:27.264 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:47.132788+0000", 2024-09-13T11:24:27.264 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.4619999999999999e-06 2024-09-13T11:24:27.264 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.264 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.264 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:24:27.264 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:47.132838+0000", 2024-09-13T11:24:27.265 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.0739000000000003e-05 2024-09-13T11:24:27.265 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.265 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.265 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:24:27.265 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:47.132853+0000", 2024-09-13T11:24:27.265 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.518e-05 2024-09-13T11:24:27.265 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.265 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.265 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:24:27.265 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:47.132860+0000", 2024-09-13T11:24:27.265 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.8689999999999998e-06 2024-09-13T11:24:27.265 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.265 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.265 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:24:27.266 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:47.132922+0000", 2024-09-13T11:24:27.266 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.1251e-05 2024-09-13T11:24:27.266 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.266 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.266 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:24:27.266 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:47.133034+0000", 2024-09-13T11:24:27.266 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000112742 2024-09-13T11:24:27.266 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.266 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.266 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 11", 2024-09-13T11:24:27.266 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:47.133141+0000", 2024-09-13T11:24:27.266 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000106968 2024-09-13T11:24:27.266 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.266 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.266 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:24:27.267 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:47.135566+0000", 2024-09-13T11:24:27.267 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0024251839999999999 2024-09-13T11:24:27.267 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.267 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.267 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:27.267 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:47.252135+0000", 2024-09-13T11:24:27.267 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.11656870900000001 2024-09-13T11:24:27.267 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.267 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.267 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:24:27.267 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:47.252201+0000", 2024-09-13T11:24:27.267 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.6223999999999995e-05 2024-09-13T11:24:27.267 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.267 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.268 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:24:27.268 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:47.252248+0000", 2024-09-13T11:24:27.268 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.6817999999999998e-05 2024-09-13T11:24:27.268 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:27.268 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:24:27.268 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:27.268 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.268 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.268 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4665.0:65304 3.1e e91/78 3:79ea72f2:::benchmark_data_smithi177_31116_object65303:head v 91'3873, mlcod=55'1534)", 2024-09-13T11:24:27.268 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:48.249255+0000", 2024-09-13T11:24:27.268 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 38.989128837000003, 2024-09-13T11:24:27.268 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.089013825000000005, 2024-09-13T11:24:27.268 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:24:27.268 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:24:27.269 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:27.269 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:24:27.269 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.269 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:24:27.269 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:48.249255+0000", 2024-09-13T11:24:27.269 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:27.269 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.269 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.269 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:24:27.269 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:48.249255+0000", 2024-09-13T11:24:27.269 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:27.269 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.269 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.269 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:24:27.269 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:48.249256+0000", 2024-09-13T11:24:27.270 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.1599999999999999e-06 2024-09-13T11:24:27.270 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.270 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.270 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:24:27.270 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:48.249288+0000", 2024-09-13T11:24:27.270 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.1987999999999999e-05 2024-09-13T11:24:27.270 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.270 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.270 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:24:27.270 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:48.335995+0000", 2024-09-13T11:24:27.270 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.086707291000000006 2024-09-13T11:24:27.270 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.270 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.271 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:24:27.271 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:48.336002+0000", 2024-09-13T11:24:27.271 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.2030000000000003e-06 2024-09-13T11:24:27.271 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.271 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.271 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:24:27.271 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:48.336060+0000", 2024-09-13T11:24:27.271 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.8010999999999997e-05 2024-09-13T11:24:27.271 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.271 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.271 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:24:27.271 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:48.336095+0000", 2024-09-13T11:24:27.271 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.4819e-05 2024-09-13T11:24:27.271 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.272 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.272 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:24:27.272 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:48.338197+0000", 2024-09-13T11:24:27.272 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.002102217 2024-09-13T11:24:27.272 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.272 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.272 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:24:27.272 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:48.338268+0000", 2024-09-13T11:24:27.272 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.1136000000000005e-05 2024-09-13T11:24:27.272 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:27.272 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:24:27.272 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:27.272 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.272 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.273 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDPGPush(3.1e 91/78 [PushOp(3:78b7dd1b:::benchmark_data_smithi177_31116_object23441:head, version: 55'1540, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:78b7dd1b:::benchmark_data_smithi177_31116_object23441:head@55'1540, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:24:27.273 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:48.342756+0000", 2024-09-13T11:24:27.273 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 38.895627181000002, 2024-09-13T11:24:27.273 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.114126449, 2024-09-13T11:24:27.273 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:24:27.273 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:24:27.273 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:24:27.273 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:24:27.273 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.273 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:24:27.273 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:48.342756+0000", 2024-09-13T11:24:27.273 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:27.273 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.273 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.273 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:24:27.274 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:48.342756+0000", 2024-09-13T11:24:27.274 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:27.274 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.274 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.274 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:24:27.274 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:48.342758+0000", 2024-09-13T11:24:27.274 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.9769999999999999e-06 2024-09-13T11:24:27.274 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.274 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.274 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:24:27.274 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:48.342796+0000", 2024-09-13T11:24:27.274 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.8331000000000003e-05 2024-09-13T11:24:27.274 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.274 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.275 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:24:27.275 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:48.455578+0000", 2024-09-13T11:24:27.275 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.11278160800000001 2024-09-13T11:24:27.275 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.275 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.275 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:24:27.275 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:48.455631+0000", 2024-09-13T11:24:27.275 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.3229999999999997e-05 2024-09-13T11:24:27.275 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.275 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.275 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:24:27.275 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:48.455799+0000", 2024-09-13T11:24:27.275 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00016763100000000001 2024-09-13T11:24:27.275 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.275 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.276 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:24:27.276 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:48.455813+0000", 2024-09-13T11:24:27.276 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.4416999999999999e-05 2024-09-13T11:24:27.276 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.276 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.276 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:24:27.276 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:48.456883+0000", 2024-09-13T11:24:27.276 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0010692550000000001 2024-09-13T11:24:27.276 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:27.276 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:24:27.276 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:27.276 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.276 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.276 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:71903 3.11 3:8da9fc5e:::benchmark_data_smithi177_31116_object71902:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96)", 2024-09-13T11:24:27.277 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:54.140481+0000", 2024-09-13T11:24:27.277 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 33.097902001000001, 2024-09-13T11:24:27.277 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.094185646999999997, 2024-09-13T11:24:27.277 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:24:27.277 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:24:27.277 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:27.277 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:24:27.277 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:24:27.277 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:27.277 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 71903 2024-09-13T11:24:27.277 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.277 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:24:27.277 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.277 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:24:27.277 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:54.140481+0000", 2024-09-13T11:24:27.278 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:27.278 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.278 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.278 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:24:27.278 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:54.140481+0000", 2024-09-13T11:24:27.278 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:27.278 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.278 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.278 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:24:27.278 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:54.140485+0000", 2024-09-13T11:24:27.278 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.3610000000000001e-06 2024-09-13T11:24:27.278 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.278 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.278 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:24:27.279 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:54.140549+0000", 2024-09-13T11:24:27.279 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.4578000000000004e-05 2024-09-13T11:24:27.279 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.279 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.279 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:24:27.279 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:54.140574+0000", 2024-09-13T11:24:27.279 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.4640000000000001e-05 2024-09-13T11:24:27.279 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.279 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.279 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:24:27.279 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:54.140588+0000", 2024-09-13T11:24:27.279 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.4154e-05 2024-09-13T11:24:27.279 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.279 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.280 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:24:27.280 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:54.140718+0000", 2024-09-13T11:24:27.280 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00013008900000000001 2024-09-13T11:24:27.280 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.280 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.280 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:24:27.280 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:54.141021+0000", 2024-09-13T11:24:27.280 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00030258100000000001 2024-09-13T11:24:27.280 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.280 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.280 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 11", 2024-09-13T11:24:27.280 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:54.141326+0000", 2024-09-13T11:24:27.280 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00030553900000000002 2024-09-13T11:24:27.280 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.281 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.281 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:24:27.281 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:54.143701+0000", 2024-09-13T11:24:27.281 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.002374751 2024-09-13T11:24:27.281 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.281 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.281 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:27.281 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:54.234577+0000", 2024-09-13T11:24:27.281 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.090875517000000003 2024-09-13T11:24:27.281 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.281 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.281 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:24:27.281 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:54.234617+0000", 2024-09-13T11:24:27.281 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.0930999999999998e-05 2024-09-13T11:24:27.281 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.282 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.282 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:24:27.282 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:54.234667+0000", 2024-09-13T11:24:27.282 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.9505999999999997e-05 2024-09-13T11:24:27.282 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:27.282 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:24:27.282 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:27.282 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.282 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.282 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:71961 3.11 3:89bd9466:::benchmark_data_smithi177_31116_object71960:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2024-09-13T11:24:27.282 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:54.267475+0000", 2024-09-13T11:24:27.282 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 32.970908354999999, 2024-09-13T11:24:27.282 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.15027747899999999, 2024-09-13T11:24:27.282 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:24:27.283 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:24:27.283 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:27.283 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:24:27.283 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:24:27.283 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:27.283 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 71961 2024-09-13T11:24:27.283 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.283 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:24:27.283 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.283 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:24:27.283 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:54.267475+0000", 2024-09-13T11:24:27.283 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:27.283 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.283 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.284 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:24:27.284 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:54.267475+0000", 2024-09-13T11:24:27.284 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:27.284 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.284 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.284 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:24:27.284 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:54.267478+0000", 2024-09-13T11:24:27.284 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.8080000000000001e-06 2024-09-13T11:24:27.284 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.284 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.284 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:24:27.284 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:54.267534+0000", 2024-09-13T11:24:27.284 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.6276999999999997e-05 2024-09-13T11:24:27.284 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.285 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.285 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:24:27.285 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:54.267557+0000", 2024-09-13T11:24:27.285 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.2514000000000002e-05 2024-09-13T11:24:27.285 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.285 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.285 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:24:27.285 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:54.267570+0000", 2024-09-13T11:24:27.285 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.3251999999999999e-05 2024-09-13T11:24:27.285 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.285 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.285 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:24:27.286 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:54.267650+0000", 2024-09-13T11:24:27.286 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.0066000000000002e-05 2024-09-13T11:24:27.286 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.286 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.286 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:24:27.286 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:54.267886+0000", 2024-09-13T11:24:27.286 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00023637800000000001 2024-09-13T11:24:27.286 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.286 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.286 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 11", 2024-09-13T11:24:27.286 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:54.268020+0000", 2024-09-13T11:24:27.286 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00013392100000000001 2024-09-13T11:24:27.286 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.286 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.287 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:24:27.287 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:54.270076+0000", 2024-09-13T11:24:27.287 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.002055393 2024-09-13T11:24:27.287 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.287 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.287 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:27.287 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:54.417584+0000", 2024-09-13T11:24:27.287 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.147508008 2024-09-13T11:24:27.287 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.287 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.287 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:24:27.287 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:54.417667+0000", 2024-09-13T11:24:27.287 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.3666999999999997e-05 2024-09-13T11:24:27.287 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.288 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.288 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:24:27.288 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:54.417752+0000", 2024-09-13T11:24:27.288 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.5195e-05 2024-09-13T11:24:27.288 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:27.288 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:24:27.288 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:27.288 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.288 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.288 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4665.0:79776 3.1a e102/78 3:5d637d57:::benchmark_data_smithi177_31116_object79775:head v 102'4120, mlcod=72'3145)", 2024-09-13T11:24:27.288 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:24:00.354950+0000", 2024-09-13T11:24:27.288 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 26.883433011000001, 2024-09-13T11:24:27.288 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.14753330100000001, 2024-09-13T11:24:27.288 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:24:27.289 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:24:27.289 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:27.289 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:24:27.289 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.289 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:24:27.289 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:00.354950+0000", 2024-09-13T11:24:27.289 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:27.289 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.289 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.289 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:24:27.289 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:00.354950+0000", 2024-09-13T11:24:27.289 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:27.289 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.289 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.289 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:24:27.290 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:00.354960+0000", 2024-09-13T11:24:27.290 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.5219999999999996e-06 2024-09-13T11:24:27.290 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.290 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.290 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:24:27.290 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:00.355024+0000", 2024-09-13T11:24:27.290 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.4163000000000002e-05 2024-09-13T11:24:27.290 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.290 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.290 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:24:27.290 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:00.499446+0000", 2024-09-13T11:24:27.290 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.144422352 2024-09-13T11:24:27.290 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.290 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.291 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:24:27.291 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:00.499466+0000", 2024-09-13T11:24:27.291 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.9694000000000002e-05 2024-09-13T11:24:27.291 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.291 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.291 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:24:27.291 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:00.499605+0000", 2024-09-13T11:24:27.291 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00013935099999999999 2024-09-13T11:24:27.291 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.291 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.291 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:24:27.291 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:00.499655+0000", 2024-09-13T11:24:27.291 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.9787000000000003e-05 2024-09-13T11:24:27.291 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.292 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.292 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:24:27.292 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:00.502418+0000", 2024-09-13T11:24:27.292 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.002763012 2024-09-13T11:24:27.292 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.292 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.292 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:24:27.292 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:00.502484+0000", 2024-09-13T11:24:27.292 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.5419999999999994e-05 2024-09-13T11:24:27.292 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:27.292 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:24:27.292 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:27.292 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.292 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.292 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDPGPush(3.1a 102/78 [PushOp(3:5df50766:::benchmark_data_smithi177_31116_object50543:head, version: 72'3152, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:5df50766:::benchmark_data_smithi177_31116_object50543:head@72'3152, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:24:27.293 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:24:00.499311+0000", 2024-09-13T11:24:27.293 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 26.739072469, 2024-09-13T11:24:27.293 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.26146664800000002, 2024-09-13T11:24:27.293 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:24:27.293 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:24:27.293 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:24:27.293 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:24:27.293 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.293 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:24:27.293 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:00.499311+0000", 2024-09-13T11:24:27.293 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:27.293 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.293 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.293 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:24:27.294 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:00.499311+0000", 2024-09-13T11:24:27.294 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:27.294 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.294 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.294 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:24:27.294 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:00.499320+0000", 2024-09-13T11:24:27.294 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.6400000000000003e-06 2024-09-13T11:24:27.294 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.294 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.294 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:24:27.294 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:00.499384+0000", 2024-09-13T11:24:27.294 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.4554999999999994e-05 2024-09-13T11:24:27.294 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.294 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.295 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:24:27.295 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:00.759713+0000", 2024-09-13T11:24:27.295 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.260328742 2024-09-13T11:24:27.295 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.295 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.295 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:24:27.295 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:00.759742+0000", 2024-09-13T11:24:27.295 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.9298e-05 2024-09-13T11:24:27.295 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.295 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.295 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:24:27.295 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:00.759871+0000", 2024-09-13T11:24:27.295 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000128992 2024-09-13T11:24:27.295 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.295 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.296 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:24:27.296 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:00.759884+0000", 2024-09-13T11:24:27.296 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2687999999999999e-05 2024-09-13T11:24:27.296 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.296 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.296 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:24:27.296 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:00.760778+0000", 2024-09-13T11:24:27.296 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00089373299999999996 2024-09-13T11:24:27.296 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:27.296 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:24:27.296 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:27.296 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.296 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.296 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4665.0:82769 3.1e e105/102 3:7a4e88b5:::benchmark_data_smithi177_31116_object82768:head v 105'4381, mlcod=105'4379)", 2024-09-13T11:24:27.297 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:24:03.745047+0000", 2024-09-13T11:24:27.297 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 23.493336243999998, 2024-09-13T11:24:27.297 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.213806145, 2024-09-13T11:24:27.297 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:24:27.297 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:24:27.297 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:27.297 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:24:27.297 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.297 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:24:27.297 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:03.745047+0000", 2024-09-13T11:24:27.297 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:27.297 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.297 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.297 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:24:27.298 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:03.745047+0000", 2024-09-13T11:24:27.298 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:27.298 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.298 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.298 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:24:27.298 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:03.745049+0000", 2024-09-13T11:24:27.298 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.5710000000000001e-06 2024-09-13T11:24:27.298 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.298 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.298 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:24:27.298 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:03.745107+0000", 2024-09-13T11:24:27.298 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.8799999999999999e-05 2024-09-13T11:24:27.298 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.298 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.298 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:24:27.299 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:03.955145+0000", 2024-09-13T11:24:27.299 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.210037735 2024-09-13T11:24:27.299 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.299 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.299 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:24:27.299 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:03.955163+0000", 2024-09-13T11:24:27.299 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.7520000000000002e-05 2024-09-13T11:24:27.299 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.299 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.299 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:24:27.299 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:03.955473+0000", 2024-09-13T11:24:27.299 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00031014399999999999 2024-09-13T11:24:27.299 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.299 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.299 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:24:27.300 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:03.955538+0000", 2024-09-13T11:24:27.300 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.5059000000000002e-05 2024-09-13T11:24:27.300 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.300 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.300 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:24:27.300 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:03.958753+0000", 2024-09-13T11:24:27.300 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0032152769999999999 2024-09-13T11:24:27.300 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.300 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.300 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:24:27.300 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:03.958853+0000", 2024-09-13T11:24:27.300 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000100039 2024-09-13T11:24:27.300 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:27.300 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:24:27.301 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:27.301 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.301 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.301 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:90466 3.11 3:8b153e95:::benchmark_data_smithi177_31116_object90465:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110)", 2024-09-13T11:24:27.301 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:24:10.223535+0000", 2024-09-13T11:24:27.301 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 17.014848022999999, 2024-09-13T11:24:27.301 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.098477190000000006, 2024-09-13T11:24:27.301 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:24:27.301 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:24:27.301 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:27.302 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:24:27.302 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:24:27.302 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:27.302 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 90466 2024-09-13T11:24:27.302 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.302 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:24:27.302 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.302 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:24:27.302 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:10.223535+0000", 2024-09-13T11:24:27.302 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:27.302 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.302 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.302 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:24:27.302 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:10.223535+0000", 2024-09-13T11:24:27.302 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:27.303 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.303 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.303 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:24:27.303 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:10.223537+0000", 2024-09-13T11:24:27.303 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.9069999999999999e-06 2024-09-13T11:24:27.303 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.303 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.303 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:24:27.303 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:10.223607+0000", 2024-09-13T11:24:27.303 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.0219e-05 2024-09-13T11:24:27.303 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.303 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.303 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:24:27.303 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:10.223619+0000", 2024-09-13T11:24:27.304 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.1779e-05 2024-09-13T11:24:27.304 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.304 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.304 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:24:27.304 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:10.223627+0000", 2024-09-13T11:24:27.304 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.4159999999999998e-06 2024-09-13T11:24:27.304 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.304 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.304 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:24:27.304 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:10.223729+0000", 2024-09-13T11:24:27.304 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000102095 2024-09-13T11:24:27.304 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.304 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.304 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:24:27.305 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:10.223828+0000", 2024-09-13T11:24:27.305 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.8851999999999995e-05 2024-09-13T11:24:27.305 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.305 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.305 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 11", 2024-09-13T11:24:27.305 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:10.223928+0000", 2024-09-13T11:24:27.305 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000100048 2024-09-13T11:24:27.305 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.305 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.305 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:24:27.305 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:10.225388+0000", 2024-09-13T11:24:27.305 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.001460781 2024-09-13T11:24:27.305 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.305 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.306 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:27.306 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:10.321944+0000", 2024-09-13T11:24:27.306 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.096555941000000006 2024-09-13T11:24:27.306 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.306 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.306 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:24:27.306 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:10.321981+0000", 2024-09-13T11:24:27.306 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.6287e-05 2024-09-13T11:24:27.306 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.306 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.306 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:24:27.306 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:10.322013+0000", 2024-09-13T11:24:27.306 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.1865000000000001e-05 2024-09-13T11:24:27.306 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:27.307 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:24:27.307 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:27.307 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.307 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.307 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:98582 3.1a 3:59f9cd75:::benchmark_data_smithi177_31116_object98581:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113)", 2024-09-13T11:24:27.307 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:24:15.431226+0000", 2024-09-13T11:24:27.307 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 11.807157775, 2024-09-13T11:24:27.307 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.19820755800000001, 2024-09-13T11:24:27.307 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:24:27.307 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:24:27.307 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:27.307 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:24:27.307 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:24:27.307 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:27.308 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 98582 2024-09-13T11:24:27.308 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.308 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:24:27.308 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.308 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:24:27.308 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.431226+0000", 2024-09-13T11:24:27.308 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:27.308 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.308 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.308 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:24:27.308 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.431226+0000", 2024-09-13T11:24:27.308 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:27.308 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.308 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.309 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:24:27.309 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.431229+0000", 2024-09-13T11:24:27.309 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.7100000000000001e-06 2024-09-13T11:24:27.309 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.309 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.309 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:24:27.309 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.431334+0000", 2024-09-13T11:24:27.309 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00010506900000000001 2024-09-13T11:24:27.309 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.309 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.309 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:24:27.309 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.431364+0000", 2024-09-13T11:24:27.309 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.9607e-05 2024-09-13T11:24:27.309 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.310 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.310 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:24:27.310 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.431388+0000", 2024-09-13T11:24:27.310 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.3808999999999999e-05 2024-09-13T11:24:27.310 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.310 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.310 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:24:27.310 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.431483+0000", 2024-09-13T11:24:27.310 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.5402999999999994e-05 2024-09-13T11:24:27.310 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.310 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.310 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:24:27.310 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.431642+0000", 2024-09-13T11:24:27.310 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000158412 2024-09-13T11:24:27.310 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.311 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.311 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 9", 2024-09-13T11:24:27.311 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.431796+0000", 2024-09-13T11:24:27.311 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00015467899999999999 2024-09-13T11:24:27.311 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.311 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.311 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:24:27.311 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.433731+0000", 2024-09-13T11:24:27.311 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.001934945 2024-09-13T11:24:27.311 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.311 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.311 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:27.311 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.629245+0000", 2024-09-13T11:24:27.311 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.19551390199999999 2024-09-13T11:24:27.312 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.312 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.312 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:24:27.312 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.629346+0000", 2024-09-13T11:24:27.312 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000100869 2024-09-13T11:24:27.312 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.312 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.312 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:24:27.312 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.629433+0000", 2024-09-13T11:24:27.312 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.7152999999999997e-05 2024-09-13T11:24:27.312 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:27.312 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:24:27.312 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:27.312 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.313 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.313 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:98872 3.1a 3:5b894ab8:::benchmark_data_smithi177_31116_object98871:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113)", 2024-09-13T11:24:27.313 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:24:15.735049+0000", 2024-09-13T11:24:27.313 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 11.50333461, 2024-09-13T11:24:27.313 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.113370635, 2024-09-13T11:24:27.313 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:24:27.313 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:24:27.313 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:27.313 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:24:27.313 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:24:27.313 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:27.313 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 98872 2024-09-13T11:24:27.313 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.313 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:24:27.314 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.314 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:24:27.314 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.735049+0000", 2024-09-13T11:24:27.314 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:27.314 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.314 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.314 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:24:27.314 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.735049+0000", 2024-09-13T11:24:27.314 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:27.314 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.314 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.314 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:24:27.315 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.735052+0000", 2024-09-13T11:24:27.315 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.5700000000000001e-06 2024-09-13T11:24:27.315 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.315 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.315 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:24:27.315 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.735097+0000", 2024-09-13T11:24:27.315 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.4824999999999999e-05 2024-09-13T11:24:27.315 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.315 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.315 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:24:27.315 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.735122+0000", 2024-09-13T11:24:27.315 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.4550999999999999e-05 2024-09-13T11:24:27.316 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.316 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.316 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:24:27.316 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.735138+0000", 2024-09-13T11:24:27.316 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.6595e-05 2024-09-13T11:24:27.316 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.316 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.316 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:24:27.316 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.735300+0000", 2024-09-13T11:24:27.316 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00016126700000000001 2024-09-13T11:24:27.316 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.316 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.316 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:24:27.317 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.735558+0000", 2024-09-13T11:24:27.317 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00025827599999999999 2024-09-13T11:24:27.317 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.317 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.317 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 9", 2024-09-13T11:24:27.317 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.735791+0000", 2024-09-13T11:24:27.317 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000232933 2024-09-13T11:24:27.317 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.317 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.317 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:24:27.317 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.738826+0000", 2024-09-13T11:24:27.317 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0030354639999999999 2024-09-13T11:24:27.317 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.317 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.318 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:27.318 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.848242+0000", 2024-09-13T11:24:27.318 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.109415784 2024-09-13T11:24:27.318 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.318 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.318 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:24:27.318 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.848321+0000", 2024-09-13T11:24:27.318 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.9085000000000006e-05 2024-09-13T11:24:27.318 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.318 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.318 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:24:27.318 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.848419+0000", 2024-09-13T11:24:27.318 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.8284999999999998e-05 2024-09-13T11:24:27.318 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:27.319 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:24:27.319 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:27.319 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.319 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.319 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:99105 3.1a 3:5ac5090c:::benchmark_data_smithi177_31116_object99104:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113)", 2024-09-13T11:24:27.319 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:24:15.962466+0000", 2024-09-13T11:24:27.319 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 11.275917486000001, 2024-09-13T11:24:27.319 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.23832849, 2024-09-13T11:24:27.319 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:24:27.319 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:24:27.319 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:27.319 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:24:27.319 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:24:27.319 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:27.320 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 99105 2024-09-13T11:24:27.320 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.320 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:24:27.320 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.320 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:24:27.320 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.962466+0000", 2024-09-13T11:24:27.320 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:27.320 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.320 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.320 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:24:27.320 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.962466+0000", 2024-09-13T11:24:27.320 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:27.320 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.320 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.321 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:24:27.321 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.962471+0000", 2024-09-13T11:24:27.321 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.9570000000000001e-06 2024-09-13T11:24:27.321 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.321 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.321 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:24:27.321 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.962547+0000", 2024-09-13T11:24:27.321 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.5947999999999999e-05 2024-09-13T11:24:27.321 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.321 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.321 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:24:27.321 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.962576+0000", 2024-09-13T11:24:27.321 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.9547999999999999e-05 2024-09-13T11:24:27.321 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.322 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.322 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:24:27.322 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.962594+0000", 2024-09-13T11:24:27.322 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.8023e-05 2024-09-13T11:24:27.322 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.322 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.322 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:24:27.322 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.962770+0000", 2024-09-13T11:24:27.322 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00017551099999999999 2024-09-13T11:24:27.322 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.322 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.322 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:24:27.322 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.963114+0000", 2024-09-13T11:24:27.322 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00034440399999999998 2024-09-13T11:24:27.322 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.323 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.323 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 9", 2024-09-13T11:24:27.323 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.963351+0000", 2024-09-13T11:24:27.323 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000237209 2024-09-13T11:24:27.323 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.323 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.323 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:24:27.323 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.966345+0000", 2024-09-13T11:24:27.323 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0029933519999999999 2024-09-13T11:24:27.323 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.323 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.323 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:27.323 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:16.200724+0000", 2024-09-13T11:24:27.323 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.23437934599999999 2024-09-13T11:24:27.324 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.324 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.324 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:24:27.324 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:16.200761+0000", 2024-09-13T11:24:27.324 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.6406999999999999e-05 2024-09-13T11:24:27.324 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.324 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.324 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:24:27.324 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:16.200794+0000", 2024-09-13T11:24:27.324 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.3785000000000003e-05 2024-09-13T11:24:27.324 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:27.324 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:24:27.324 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:27.324 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.325 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.325 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4665.0:104331 3.1e e115/102 3:79890035:::benchmark_data_smithi177_31116_object104330:head v 115'5083, mlcod=115'5080)", 2024-09-13T11:24:27.325 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:24:20.242656+0000", 2024-09-13T11:24:27.325 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 6.9957273940000002, 2024-09-13T11:24:27.325 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.16813846700000001, 2024-09-13T11:24:27.325 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:24:27.325 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:24:27.325 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:27.325 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:24:27.325 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.325 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:24:27.325 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:20.242656+0000", 2024-09-13T11:24:27.325 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:27.325 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.326 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.326 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:24:27.326 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:20.242656+0000", 2024-09-13T11:24:27.326 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:27.326 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.326 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.326 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:24:27.326 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:20.242657+0000", 2024-09-13T11:24:27.326 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.4750000000000001e-06 2024-09-13T11:24:27.326 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.326 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.326 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:24:27.326 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:20.242699+0000", 2024-09-13T11:24:27.326 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.193e-05 2024-09-13T11:24:27.327 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.327 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.327 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:24:27.327 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:20.408772+0000", 2024-09-13T11:24:27.327 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.16607291199999999 2024-09-13T11:24:27.327 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.327 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.327 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:24:27.327 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:20.408779+0000", 2024-09-13T11:24:27.327 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.3600000000000001e-06 2024-09-13T11:24:27.327 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.327 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.327 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:24:27.327 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:20.408840+0000", 2024-09-13T11:24:27.328 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.1834999999999993e-05 2024-09-13T11:24:27.328 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.328 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.328 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:24:27.328 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:20.408865+0000", 2024-09-13T11:24:27.328 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.4294000000000002e-05 2024-09-13T11:24:27.328 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.328 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.328 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:24:27.328 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:20.410751+0000", 2024-09-13T11:24:27.328 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.001885811 2024-09-13T11:24:27.328 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.328 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.328 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:24:27.329 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:20.410794+0000", 2024-09-13T11:24:27.329 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.3850000000000002e-05 2024-09-13T11:24:27.329 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:27.329 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:24:27.329 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:27.329 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.329 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.329 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4665.0:104823 3.1e e115/102 3:7a73472c:::benchmark_data_smithi177_31116_object104822:head v 115'5094, mlcod=115'5091)", 2024-09-13T11:24:27.329 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:24:20.661863+0000", 2024-09-13T11:24:27.329 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 6.5765200799999999, 2024-09-13T11:24:27.329 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.100084861, 2024-09-13T11:24:27.329 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:24:27.329 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:24:27.329 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:27.329 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:24:27.330 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.330 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:24:27.330 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:20.661863+0000", 2024-09-13T11:24:27.330 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:27.330 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.330 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.330 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:24:27.330 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:20.661863+0000", 2024-09-13T11:24:27.330 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:27.330 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.330 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.330 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:24:27.330 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:20.661866+0000", 2024-09-13T11:24:27.330 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.3479999999999998e-06 2024-09-13T11:24:27.330 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.331 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.331 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:24:27.331 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:20.661906+0000", 2024-09-13T11:24:27.331 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.0076000000000001e-05 2024-09-13T11:24:27.331 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.331 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.331 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:24:27.331 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:20.760131+0000", 2024-09-13T11:24:27.331 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.098225420999999993 2024-09-13T11:24:27.331 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.331 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.331 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:24:27.331 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:20.760141+0000", 2024-09-13T11:24:27.332 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.027e-05 2024-09-13T11:24:27.332 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.332 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.332 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:24:27.332 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:20.760198+0000", 2024-09-13T11:24:27.332 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.6127e-05 2024-09-13T11:24:27.332 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.332 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.332 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:24:27.332 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:20.760214+0000", 2024-09-13T11:24:27.332 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.6932e-05 2024-09-13T11:24:27.332 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.332 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.332 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:24:27.333 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:20.761876+0000", 2024-09-13T11:24:27.333 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0016611690000000001 2024-09-13T11:24:27.333 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.333 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.333 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:24:27.333 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:20.761948+0000", 2024-09-13T11:24:27.333 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.2517999999999999e-05 2024-09-13T11:24:27.333 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:27.333 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:24:27.333 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:27.333 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.333 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.333 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:105026 3.1a 3:5db5c278:::benchmark_data_smithi177_31116_object105025:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-09-13T11:24:27.333 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:24:20.851015+0000", 2024-09-13T11:24:27.334 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 6.3873686640000003, 2024-09-13T11:24:27.334 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.095865153999999994, 2024-09-13T11:24:27.334 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:24:27.334 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:24:27.334 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:27.334 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:24:27.334 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:24:27.334 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:27.334 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 105026 2024-09-13T11:24:27.334 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.334 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:24:27.334 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.334 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:24:27.334 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:20.851015+0000", 2024-09-13T11:24:27.334 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:27.335 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.335 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.335 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:24:27.335 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:20.851015+0000", 2024-09-13T11:24:27.335 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:27.335 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.335 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.335 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:24:27.335 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:20.851018+0000", 2024-09-13T11:24:27.335 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.954e-06 2024-09-13T11:24:27.335 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.335 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.335 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:24:27.335 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:20.851057+0000", 2024-09-13T11:24:27.336 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.9101999999999999e-05 2024-09-13T11:24:27.336 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.336 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.336 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:24:27.336 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:20.851079+0000", 2024-09-13T11:24:27.336 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.2722e-05 2024-09-13T11:24:27.336 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.336 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.336 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:24:27.336 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:20.851097+0000", 2024-09-13T11:24:27.336 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.719e-05 2024-09-13T11:24:27.336 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.336 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.336 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:24:27.337 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:20.851186+0000", 2024-09-13T11:24:27.337 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.8850000000000005e-05 2024-09-13T11:24:27.337 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.337 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.337 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:24:27.337 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:20.851269+0000", 2024-09-13T11:24:27.337 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.3257000000000006e-05 2024-09-13T11:24:27.337 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.337 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.337 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 9", 2024-09-13T11:24:27.337 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:20.851382+0000", 2024-09-13T11:24:27.337 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00011323 2024-09-13T11:24:27.337 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.337 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.337 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:24:27.338 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:20.853102+0000", 2024-09-13T11:24:27.338 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0017195979999999999 2024-09-13T11:24:27.338 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.338 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.338 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:27.338 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:20.946731+0000", 2024-09-13T11:24:27.338 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.093629195999999998 2024-09-13T11:24:27.338 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.338 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.338 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:24:27.338 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:20.946817+0000", 2024-09-13T11:24:27.338 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.6032000000000006e-05 2024-09-13T11:24:27.338 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.338 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.338 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:24:27.339 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:20.946880+0000", 2024-09-13T11:24:27.339 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.3022999999999996e-05 2024-09-13T11:24:27.339 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:27.339 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:24:27.339 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:27.339 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:27.339 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:24:27.339 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-13T11:24:27.340 INFO:tasks.ceph.osd.13.smithi092.stderr:2024-09-13T11:24:27.334+0000 7f3e5afea640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26350) UID: 0 2024-09-13T11:24:27.341 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-09-13T11:24:27.441 INFO:tasks.ceph.osd.15.smithi177.stderr:2024-09-13T11:24:27.439+0000 7f9d6ce77640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26578) UID: 0 2024-09-13T11:24:27.514 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-13T11:24:27.514 INFO:teuthology.orchestra.run.smithi001.stdout: "ops": [ 2024-09-13T11:24:27.514 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.514 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDPGPushReply(3.27 121/120 [PushReplyOp(3:e7681562:::benchmark_data_smithi177_31116_object1501:head)])", 2024-09-13T11:24:27.514 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:24:27.512504+0000", 2024-09-13T11:24:27.515 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 0.001221495, 2024-09-13T11:24:27.515 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0012475590000000001, 2024-09-13T11:24:27.515 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:24:27.515 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:24:27.515 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-13T11:24:27.515 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:24:27.515 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.515 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:24:27.515 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:27.512504+0000", 2024-09-13T11:24:27.515 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:27.515 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.515 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.516 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:24:27.516 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:27.512504+0000", 2024-09-13T11:24:27.516 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:27.516 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.516 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.516 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:24:27.516 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:27.512506+0000", 2024-09-13T11:24:27.516 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.9360000000000002e-06 2024-09-13T11:24:27.516 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.516 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.516 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:24:27.517 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:27.512517+0000", 2024-09-13T11:24:27.517 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0872e-05 2024-09-13T11:24:27.517 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.517 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.517 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:24:27.517 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:27.512559+0000", 2024-09-13T11:24:27.517 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.1767999999999997e-05 2024-09-13T11:24:27.517 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.517 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.517 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:24:27.517 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:27.512568+0000", 2024-09-13T11:24:27.518 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.2520000000000005e-06 2024-09-13T11:24:27.518 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.518 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.518 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:24:27.518 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:27.512643+0000", 2024-09-13T11:24:27.518 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.4474999999999994e-05 2024-09-13T11:24:27.518 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:27.518 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:24:27.518 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:27.518 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:27.518 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-09-13T11:24:27.518 INFO:teuthology.orchestra.run.smithi001.stdout: "num_ops": 1 2024-09-13T11:24:27.518 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-13T11:24:27.531 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-09-13T11:24:27.542 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-09-13T11:24:27.539+0000 7fb2a5596640 -1 mon.a@0(leader).osd e121 definitely_dead 0 2024-09-13T11:24:27.543 INFO:tasks.ceph.osd.2.smithi162.stderr:2024-09-13T11:24:27.540+0000 7fe92b35a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26313) UID: 0 2024-09-13T11:24:27.643 INFO:tasks.ceph.osd.3.smithi177.stderr:2024-09-13T11:24:27.639+0000 7f10adba4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26574) UID: 0 2024-09-13T11:24:27.651 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-13T11:24:27.652 INFO:teuthology.orchestra.run.smithi001.stdout: "ops": [], 2024-09-13T11:24:27.652 INFO:teuthology.orchestra.run.smithi001.stdout: "complaint_time": 30, 2024-09-13T11:24:27.652 INFO:teuthology.orchestra.run.smithi001.stdout: "num_blocked_ops": 0 2024-09-13T11:24:27.652 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-13T11:24:27.667 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-09-13T11:24:27.744 INFO:tasks.ceph.osd.1.smithi092.stderr:2024-09-13T11:24:27.738+0000 7fe7789c5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33718) UID: 0 2024-09-13T11:24:27.764 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T11:24:27.848 INFO:tasks.ceph.osd.8.smithi001.stderr:2024-09-13T11:24:27.843+0000 7fb9b48ae640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26312) UID: 0 2024-09-13T11:24:27.857 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-13T11:24:27.857 INFO:teuthology.orchestra.run.smithi001.stdout: "size": 20, 2024-09-13T11:24:27.858 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 600, 2024-09-13T11:24:27.858 INFO:teuthology.orchestra.run.smithi001.stdout: "ops": [ 2024-09-13T11:24:27.858 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.858 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:30801 3.e 3:73b0f21a:::benchmark_data_smithi177_31116_object30800:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e59)", 2024-09-13T11:24:27.858 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:09.485629+0000", 2024-09-13T11:24:27.858 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 78.370314764, 2024-09-13T11:24:27.858 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.22604919900000001, 2024-09-13T11:24:27.858 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:24:27.858 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:24:27.858 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:27.858 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:24:27.858 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:24:27.858 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:27.859 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 30801 2024-09-13T11:24:27.859 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.859 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:24:27.859 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.859 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:24:27.859 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.485629+0000", 2024-09-13T11:24:27.859 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:27.859 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.859 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.859 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:24:27.859 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.485629+0000", 2024-09-13T11:24:27.859 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:27.859 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.860 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.860 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:24:27.860 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.485631+0000", 2024-09-13T11:24:27.860 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.9989999999999998e-06 2024-09-13T11:24:27.860 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.860 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.860 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:24:27.860 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.485664+0000", 2024-09-13T11:24:27.860 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.3896000000000002e-05 2024-09-13T11:24:27.860 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.860 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.860 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:24:27.860 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.485679+0000", 2024-09-13T11:24:27.860 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.4438e-05 2024-09-13T11:24:27.860 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.861 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.861 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:24:27.861 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.485688+0000", 2024-09-13T11:24:27.861 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.0180000000000007e-06 2024-09-13T11:24:27.861 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.861 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.861 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:24:27.861 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.485742+0000", 2024-09-13T11:24:27.861 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.4026000000000001e-05 2024-09-13T11:24:27.861 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.862 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.862 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:24:27.862 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.485841+0000", 2024-09-13T11:24:27.862 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.8875000000000005e-05 2024-09-13T11:24:27.862 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.862 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.862 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 0,1", 2024-09-13T11:24:27.862 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.486005+0000", 2024-09-13T11:24:27.862 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000163747 2024-09-13T11:24:27.862 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.862 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.862 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:24:27.863 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.487867+0000", 2024-09-13T11:24:27.863 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.001862945 2024-09-13T11:24:27.863 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.863 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.863 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:27.863 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.488570+0000", 2024-09-13T11:24:27.863 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00070255700000000003 2024-09-13T11:24:27.863 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.863 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.863 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:27.863 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.711140+0000", 2024-09-13T11:24:27.863 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.222570348 2024-09-13T11:24:27.863 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.864 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.864 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:24:27.864 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.711180+0000", 2024-09-13T11:24:27.864 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.9273999999999997e-05 2024-09-13T11:24:27.864 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.864 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.864 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:24:27.864 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.711678+0000", 2024-09-13T11:24:27.864 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00049807599999999997 2024-09-13T11:24:27.864 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:27.864 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:24:27.864 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:27.864 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.864 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.865 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:54656 3.1e 3.6a49163e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e75)", 2024-09-13T11:24:27.865 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:30.979993+0000", 2024-09-13T11:24:27.865 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 56.875950332000002, 2024-09-13T11:24:27.865 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.9269557150000001, 2024-09-13T11:24:27.865 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:24:27.865 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:24:27.865 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-13T11:24:27.865 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:24:27.865 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:24:27.865 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:27.865 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 54656 2024-09-13T11:24:27.865 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.866 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:24:27.866 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.866 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:24:27.866 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:30.979993+0000", 2024-09-13T11:24:27.866 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:27.866 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.866 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.866 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:24:27.866 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:30.979993+0000", 2024-09-13T11:24:27.866 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:27.866 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.866 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.866 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:24:27.867 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:30.979997+0000", 2024-09-13T11:24:27.867 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.0319999999999997e-06 2024-09-13T11:24:27.867 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.867 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.867 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:24:27.867 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:30.980057+0000", 2024-09-13T11:24:27.867 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.0341999999999999e-05 2024-09-13T11:24:27.867 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.867 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.867 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:24:27.867 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:30.980094+0000", 2024-09-13T11:24:27.867 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.6770000000000002e-05 2024-09-13T11:24:27.867 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.868 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.868 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:24:27.868 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:30.980113+0000", 2024-09-13T11:24:27.868 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.8791999999999999e-05 2024-09-13T11:24:27.868 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.868 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.868 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:24:27.868 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.097773+0000", 2024-09-13T11:24:27.868 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.117660379 2024-09-13T11:24:27.868 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.868 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.868 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for peered", 2024-09-13T11:24:27.868 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.097784+0000", 2024-09-13T11:24:27.869 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.1025e-05 2024-09-13T11:24:27.869 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.869 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.869 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:24:27.869 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.906937+0000", 2024-09-13T11:24:27.869 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.8091523789999999 2024-09-13T11:24:27.869 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.869 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.869 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:24:27.869 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.906949+0000", 2024-09-13T11:24:27.869 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.1996e-05 2024-09-13T11:24:27.869 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:27.869 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:24:27.870 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:27.870 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.870 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.870 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDPGPushReply(3.17 75/70 [PushReplyOp(3:e82800c8:::benchmark_data_smithi177_31116_object13909:head)])", 2024-09-13T11:24:27.870 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:31.632344+0000", 2024-09-13T11:24:27.870 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 56.223599477, 2024-09-13T11:24:27.870 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.28592102899999999, 2024-09-13T11:24:27.870 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:24:27.870 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:24:27.870 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-13T11:24:27.870 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:24:27.870 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.870 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:24:27.871 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.632344+0000", 2024-09-13T11:24:27.871 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:27.871 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.871 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.871 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:24:27.871 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.632344+0000", 2024-09-13T11:24:27.871 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:27.871 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.871 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.871 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:24:27.871 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.632346+0000", 2024-09-13T11:24:27.871 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.8819999999999999e-06 2024-09-13T11:24:27.871 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.872 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.872 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:24:27.872 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.632353+0000", 2024-09-13T11:24:27.872 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.5059999999999998e-06 2024-09-13T11:24:27.872 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.872 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.872 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:24:27.872 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.917361+0000", 2024-09-13T11:24:27.872 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.285008231 2024-09-13T11:24:27.872 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.872 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.873 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:24:27.873 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.917372+0000", 2024-09-13T11:24:27.873 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0764999999999999e-05 2024-09-13T11:24:27.873 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.873 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.873 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:24:27.873 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.917458+0000", 2024-09-13T11:24:27.873 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.6136000000000003e-05 2024-09-13T11:24:27.873 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.873 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.873 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:24:27.873 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.918265+0000", 2024-09-13T11:24:27.874 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00080650899999999996 2024-09-13T11:24:27.874 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:27.874 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:24:27.874 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:27.874 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.874 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.874 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:54720 3.1e 3.a944303e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-13T11:24:27.874 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:31.973315+0000", 2024-09-13T11:24:27.874 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 55.882628097000001, 2024-09-13T11:24:27.874 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.93366527200000005, 2024-09-13T11:24:27.874 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:24:27.874 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:24:27.874 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-13T11:24:27.875 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:24:27.875 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:24:27.875 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:27.875 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 54720 2024-09-13T11:24:27.875 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.875 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:24:27.875 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.875 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:24:27.875 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.973315+0000", 2024-09-13T11:24:27.875 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:27.875 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.875 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.875 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:24:27.875 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.973315+0000", 2024-09-13T11:24:27.876 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:27.876 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.876 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.876 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:24:27.876 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.973318+0000", 2024-09-13T11:24:27.876 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.5830000000000001e-06 2024-09-13T11:24:27.876 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.876 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.876 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:24:27.876 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.973465+0000", 2024-09-13T11:24:27.876 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000146816 2024-09-13T11:24:27.876 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.876 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.877 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:24:27.877 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.973478+0000", 2024-09-13T11:24:27.877 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2931e-05 2024-09-13T11:24:27.877 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.877 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.877 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:24:27.877 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.973483+0000", 2024-09-13T11:24:27.877 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.8969999999999996e-06 2024-09-13T11:24:27.877 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.877 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.877 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:24:27.877 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.973548+0000", 2024-09-13T11:24:27.877 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.4771999999999996e-05 2024-09-13T11:24:27.878 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.878 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.878 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for peered", 2024-09-13T11:24:27.878 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.973557+0000", 2024-09-13T11:24:27.878 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.0359999999999995e-06 2024-09-13T11:24:27.878 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.878 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.878 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:24:27.878 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.906974+0000", 2024-09-13T11:24:27.878 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.93341706000000002 2024-09-13T11:24:27.878 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.878 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.878 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:24:27.878 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.906980+0000", 2024-09-13T11:24:27.879 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.1770000000000003e-06 2024-09-13T11:24:27.879 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:27.879 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:24:27.879 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:27.879 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.879 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.879 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:54734 3.1e 3.d7f6f7e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-13T11:24:27.879 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:31.983342+0000", 2024-09-13T11:24:27.879 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 55.872601293999999, 2024-09-13T11:24:27.879 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.92367058599999996, 2024-09-13T11:24:27.879 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:24:27.879 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:24:27.879 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-13T11:24:27.880 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:24:27.880 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:24:27.880 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:27.880 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 54734 2024-09-13T11:24:27.880 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.880 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:24:27.880 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.880 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:24:27.880 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.983342+0000", 2024-09-13T11:24:27.880 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:27.880 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.880 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.880 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:24:27.881 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.983342+0000", 2024-09-13T11:24:27.881 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:27.881 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.881 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.881 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:24:27.881 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.983345+0000", 2024-09-13T11:24:27.881 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.6220000000000001e-06 2024-09-13T11:24:27.881 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.881 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.881 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:24:27.881 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.983438+0000", 2024-09-13T11:24:27.881 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.3517000000000006e-05 2024-09-13T11:24:27.881 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.882 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.882 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:24:27.882 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.983451+0000", 2024-09-13T11:24:27.882 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2527999999999999e-05 2024-09-13T11:24:27.882 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.882 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.882 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:24:27.882 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.983458+0000", 2024-09-13T11:24:27.882 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.6380000000000004e-06 2024-09-13T11:24:27.882 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.882 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.882 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:24:27.882 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.983521+0000", 2024-09-13T11:24:27.882 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.2729000000000002e-05 2024-09-13T11:24:27.883 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.883 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.883 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for peered", 2024-09-13T11:24:27.883 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.983548+0000", 2024-09-13T11:24:27.883 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.6537999999999999e-05 2024-09-13T11:24:27.883 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.883 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.883 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:24:27.884 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.907005+0000", 2024-09-13T11:24:27.884 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.923456991 2024-09-13T11:24:27.884 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.884 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.884 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:24:27.884 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.907013+0000", 2024-09-13T11:24:27.884 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.0229999999999992e-06 2024-09-13T11:24:27.884 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:27.885 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:24:27.885 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:27.885 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.885 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.885 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:54760 3.1e 3.8c32c13e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-13T11:24:27.885 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:31.999685+0000", 2024-09-13T11:24:27.885 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 55.856258062000002, 2024-09-13T11:24:27.885 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.90735639599999995, 2024-09-13T11:24:27.885 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:24:27.885 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:24:27.886 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-13T11:24:27.886 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:24:27.886 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:24:27.886 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:27.886 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 54760 2024-09-13T11:24:27.886 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.886 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:24:27.886 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.886 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:24:27.886 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.999685+0000", 2024-09-13T11:24:27.886 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:27.887 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.887 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.887 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:24:27.887 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.999685+0000", 2024-09-13T11:24:27.887 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:27.887 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.887 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.887 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:24:27.887 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.999690+0000", 2024-09-13T11:24:27.887 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.7879999999999997e-06 2024-09-13T11:24:27.887 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.887 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.887 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:24:27.887 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.999729+0000", 2024-09-13T11:24:27.887 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.9032000000000002e-05 2024-09-13T11:24:27.888 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.888 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.888 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:24:27.888 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.999741+0000", 2024-09-13T11:24:27.888 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2136999999999999e-05 2024-09-13T11:24:27.888 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.888 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.888 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:24:27.888 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.999748+0000", 2024-09-13T11:24:27.888 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.9120000000000001e-06 2024-09-13T11:24:27.888 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.888 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.888 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:24:27.888 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.999803+0000", 2024-09-13T11:24:27.889 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.5064e-05 2024-09-13T11:24:27.889 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.889 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.889 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for peered", 2024-09-13T11:24:27.889 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.999815+0000", 2024-09-13T11:24:27.889 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2041e-05 2024-09-13T11:24:27.889 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.889 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.889 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:24:27.889 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.907034+0000", 2024-09-13T11:24:27.889 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.90721858200000005 2024-09-13T11:24:27.889 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.889 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.889 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:24:27.890 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.907042+0000", 2024-09-13T11:24:27.890 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.8399999999999995e-06 2024-09-13T11:24:27.890 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:27.890 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:24:27.890 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:27.890 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.890 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.890 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:54768 3.1e 3.6dad499e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-13T11:24:27.890 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:32.003482+0000", 2024-09-13T11:24:27.890 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 55.852461411999997, 2024-09-13T11:24:27.890 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.90358804699999995, 2024-09-13T11:24:27.890 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:24:27.890 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:24:27.890 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-13T11:24:27.891 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:24:27.891 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:24:27.891 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:27.891 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 54768 2024-09-13T11:24:27.891 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.891 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:24:27.891 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.891 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:24:27.891 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.003482+0000", 2024-09-13T11:24:27.891 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:27.891 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.891 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.891 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:24:27.892 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.003482+0000", 2024-09-13T11:24:27.892 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:27.892 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.892 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.892 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:24:27.892 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.003486+0000", 2024-09-13T11:24:27.892 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.1030000000000004e-06 2024-09-13T11:24:27.892 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.892 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.892 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:24:27.892 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.003523+0000", 2024-09-13T11:24:27.892 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.6732e-05 2024-09-13T11:24:27.892 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.892 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.893 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:24:27.893 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.003535+0000", 2024-09-13T11:24:27.893 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2520999999999999e-05 2024-09-13T11:24:27.893 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.893 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.893 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:24:27.893 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.003541+0000", 2024-09-13T11:24:27.893 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.8690000000000001e-06 2024-09-13T11:24:27.893 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.893 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.893 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:24:27.893 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.003593+0000", 2024-09-13T11:24:27.893 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.1786000000000002e-05 2024-09-13T11:24:27.893 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.894 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.894 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for peered", 2024-09-13T11:24:27.894 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.003602+0000", 2024-09-13T11:24:27.894 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.9679999999999995e-06 2024-09-13T11:24:27.894 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.894 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.894 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:24:27.894 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.907063+0000", 2024-09-13T11:24:27.894 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.90346116600000004 2024-09-13T11:24:27.894 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.894 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.894 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:24:27.894 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.907070+0000", 2024-09-13T11:24:27.894 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.9020000000000002e-06 2024-09-13T11:24:27.895 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:27.895 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:24:27.895 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:27.895 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.895 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.895 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:54785 3.1e 3.d32f1ede (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-13T11:24:27.895 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:32.016238+0000", 2024-09-13T11:24:27.895 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 55.839705201999998, 2024-09-13T11:24:27.895 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.89086243399999998, 2024-09-13T11:24:27.895 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:24:27.895 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:24:27.895 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-13T11:24:27.895 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:24:27.895 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:24:27.896 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:27.896 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 54785 2024-09-13T11:24:27.896 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.896 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:24:27.896 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.896 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:24:27.896 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.016238+0000", 2024-09-13T11:24:27.896 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:27.896 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.896 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.896 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:24:27.896 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.016238+0000", 2024-09-13T11:24:27.896 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:27.896 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.897 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.897 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:24:27.897 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.016241+0000", 2024-09-13T11:24:27.897 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.6469999999999998e-06 2024-09-13T11:24:27.897 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.897 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.897 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:24:27.897 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.016271+0000", 2024-09-13T11:24:27.897 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.0505999999999999e-05 2024-09-13T11:24:27.897 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.897 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.897 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:24:27.897 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.016287+0000", 2024-09-13T11:24:27.897 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.6073999999999999e-05 2024-09-13T11:24:27.898 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.898 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.898 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:24:27.898 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.016295+0000", 2024-09-13T11:24:27.898 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.0590000000000003e-06 2024-09-13T11:24:27.898 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.898 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.898 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:24:27.898 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.016373+0000", 2024-09-13T11:24:27.898 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.7634999999999998e-05 2024-09-13T11:24:27.898 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.898 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.898 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for peered", 2024-09-13T11:24:27.898 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.016383+0000", 2024-09-13T11:24:27.899 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0246e-05 2024-09-13T11:24:27.899 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.899 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.899 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:24:27.899 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.907092+0000", 2024-09-13T11:24:27.899 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.89070849200000002 2024-09-13T11:24:27.899 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.899 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.899 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:24:27.899 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.907101+0000", 2024-09-13T11:24:27.899 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.7749999999999998e-06 2024-09-13T11:24:27.899 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:27.899 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:24:27.899 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:27.900 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.900 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.900 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:54811 3.1e 3.3754367e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-13T11:24:27.900 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:32.045424+0000", 2024-09-13T11:24:27.900 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 55.810519550999999, 2024-09-13T11:24:27.900 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.86170719399999995, 2024-09-13T11:24:27.900 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:24:27.900 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:24:27.900 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-13T11:24:27.900 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:24:27.900 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:24:27.900 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:27.900 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 54811 2024-09-13T11:24:27.901 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.901 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:24:27.901 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.901 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:24:27.901 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.045424+0000", 2024-09-13T11:24:27.901 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:27.901 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.901 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.901 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:24:27.901 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.045424+0000", 2024-09-13T11:24:27.901 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:27.901 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.901 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.902 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:24:27.902 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.045426+0000", 2024-09-13T11:24:27.902 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.0260000000000002e-06 2024-09-13T11:24:27.902 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.902 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.902 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:24:27.902 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.045556+0000", 2024-09-13T11:24:27.902 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00013003099999999999 2024-09-13T11:24:27.902 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.902 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.902 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:24:27.902 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.045568+0000", 2024-09-13T11:24:27.902 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.1727e-05 2024-09-13T11:24:27.902 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.903 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.903 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:24:27.903 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.045576+0000", 2024-09-13T11:24:27.903 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.6459999999999999e-06 2024-09-13T11:24:27.903 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.903 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.903 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:24:27.903 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.045650+0000", 2024-09-13T11:24:27.903 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.4000999999999998e-05 2024-09-13T11:24:27.903 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.903 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.903 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for peered", 2024-09-13T11:24:27.903 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.045658+0000", 2024-09-13T11:24:27.903 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.3869999999999997e-06 2024-09-13T11:24:27.904 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.904 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.904 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:24:27.904 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.907123+0000", 2024-09-13T11:24:27.904 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.86146508399999999 2024-09-13T11:24:27.904 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.904 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.904 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:24:27.904 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.907131+0000", 2024-09-13T11:24:27.904 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.2919999999999996e-06 2024-09-13T11:24:27.904 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:27.904 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:24:27.904 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:27.904 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.905 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.905 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:54817 3.1e 3.36dd2fde (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-13T11:24:27.905 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:32.050108+0000", 2024-09-13T11:24:27.905 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 55.805835659000003, 2024-09-13T11:24:27.905 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.85705549199999997, 2024-09-13T11:24:27.905 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:24:27.905 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:24:27.905 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-13T11:24:27.905 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:24:27.905 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:24:27.905 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:27.905 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 54817 2024-09-13T11:24:27.905 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.905 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:24:27.905 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.906 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:24:27.906 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.050108+0000", 2024-09-13T11:24:27.906 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:27.906 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.906 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.906 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:24:27.906 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.050108+0000", 2024-09-13T11:24:27.906 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:27.906 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.906 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.906 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:24:27.906 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.050110+0000", 2024-09-13T11:24:27.906 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.0209999999999998e-06 2024-09-13T11:24:27.906 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.906 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.907 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:24:27.907 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.050166+0000", 2024-09-13T11:24:27.907 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.6026000000000002e-05 2024-09-13T11:24:27.907 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.907 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.907 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:24:27.907 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.050180+0000", 2024-09-13T11:24:27.907 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.4307e-05 2024-09-13T11:24:27.907 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.907 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.907 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:24:27.907 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.050188+0000", 2024-09-13T11:24:27.908 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.5410000000000003e-06 2024-09-13T11:24:27.908 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.908 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.908 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:24:27.908 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.050260+0000", 2024-09-13T11:24:27.908 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.2157999999999996e-05 2024-09-13T11:24:27.908 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.908 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.908 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for peered", 2024-09-13T11:24:27.908 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.050267+0000", 2024-09-13T11:24:27.908 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.0190000000000001e-06 2024-09-13T11:24:27.908 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.908 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.908 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:24:27.908 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.907155+0000", 2024-09-13T11:24:27.908 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.85688824399999997 2024-09-13T11:24:27.909 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.909 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.909 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:24:27.909 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.907163+0000", 2024-09-13T11:24:27.909 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.1759999999999993e-06 2024-09-13T11:24:27.909 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:27.909 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:24:27.909 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:27.909 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.909 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.909 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:54818 3.1e 3.5cfcc69e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-13T11:24:27.909 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:32.051374+0000", 2024-09-13T11:24:27.909 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 55.804569243000003, 2024-09-13T11:24:27.910 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.85581781000000001, 2024-09-13T11:24:27.910 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:24:27.910 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:24:27.910 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-13T11:24:27.910 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:24:27.910 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:24:27.910 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:27.910 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 54818 2024-09-13T11:24:27.910 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.910 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:24:27.910 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.910 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:24:27.910 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.051374+0000", 2024-09-13T11:24:27.910 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:27.911 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.911 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.911 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:24:27.911 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.051374+0000", 2024-09-13T11:24:27.911 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:27.911 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.911 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.911 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:24:27.911 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.051375+0000", 2024-09-13T11:24:27.911 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.4440000000000001e-06 2024-09-13T11:24:27.911 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.911 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.911 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:24:27.911 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.051426+0000", 2024-09-13T11:24:27.911 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.0809000000000001e-05 2024-09-13T11:24:27.912 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.912 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.912 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:24:27.912 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.051441+0000", 2024-09-13T11:24:27.912 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.4460999999999999e-05 2024-09-13T11:24:27.912 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.912 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.912 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:24:27.912 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.051449+0000", 2024-09-13T11:24:27.912 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.4880000000000007e-06 2024-09-13T11:24:27.912 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.912 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.912 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:24:27.912 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.051499+0000", 2024-09-13T11:24:27.912 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.9691999999999998e-05 2024-09-13T11:24:27.913 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.913 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.913 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for peered", 2024-09-13T11:24:27.913 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.051505+0000", 2024-09-13T11:24:27.913 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.2029999999999997e-06 2024-09-13T11:24:27.913 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.913 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.913 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:24:27.913 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.907184+0000", 2024-09-13T11:24:27.913 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.85567906599999999 2024-09-13T11:24:27.913 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.913 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.913 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:24:27.913 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.907192+0000", 2024-09-13T11:24:27.913 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.6469999999999998e-06 2024-09-13T11:24:27.914 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:27.914 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:24:27.914 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:27.914 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.914 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.914 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:54828 3.1e 3.e920c11e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-13T11:24:27.914 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:32.063344+0000", 2024-09-13T11:24:27.914 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 55.792598816000002, 2024-09-13T11:24:27.914 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.84387899099999997, 2024-09-13T11:24:27.914 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:24:27.914 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:24:27.914 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-13T11:24:27.914 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:24:27.915 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:24:27.915 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:27.915 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 54828 2024-09-13T11:24:27.915 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.915 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:24:27.915 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.915 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:24:27.915 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.063344+0000", 2024-09-13T11:24:27.915 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:27.915 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.915 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.915 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:24:27.915 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.063344+0000", 2024-09-13T11:24:27.915 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:27.916 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.916 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.916 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:24:27.916 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.063347+0000", 2024-09-13T11:24:27.916 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.2299999999999998e-06 2024-09-13T11:24:27.916 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.916 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.916 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:24:27.916 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.063384+0000", 2024-09-13T11:24:27.916 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.7290999999999999e-05 2024-09-13T11:24:27.916 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.916 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.916 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:24:27.917 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.063397+0000", 2024-09-13T11:24:27.917 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2955999999999999e-05 2024-09-13T11:24:27.917 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.917 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.917 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:24:27.917 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.063404+0000", 2024-09-13T11:24:27.917 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.4889999999999998e-06 2024-09-13T11:24:27.917 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.918 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.918 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:24:27.918 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.063470+0000", 2024-09-13T11:24:27.918 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.5516000000000001e-05 2024-09-13T11:24:27.918 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.918 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.918 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for peered", 2024-09-13T11:24:27.918 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.063482+0000", 2024-09-13T11:24:27.918 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2065000000000001e-05 2024-09-13T11:24:27.918 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.918 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.918 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:24:27.918 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.907216+0000", 2024-09-13T11:24:27.918 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.84373388900000001 2024-09-13T11:24:27.919 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.919 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.919 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:24:27.919 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.907223+0000", 2024-09-13T11:24:27.919 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.5549999999999997e-06 2024-09-13T11:24:27.919 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:27.919 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:24:27.919 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:27.919 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.919 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.919 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:54829 3.1e 3.192f62be (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-13T11:24:27.919 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:32.065642+0000", 2024-09-13T11:24:27.919 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 55.790301747999997, 2024-09-13T11:24:27.919 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.84161483100000001, 2024-09-13T11:24:27.920 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:24:27.920 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:24:27.920 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-13T11:24:27.920 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:24:27.920 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:24:27.920 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:27.920 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 54829 2024-09-13T11:24:27.920 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.920 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:24:27.920 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.920 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:24:27.920 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.065642+0000", 2024-09-13T11:24:27.921 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:27.921 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.921 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.921 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:24:27.921 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.065642+0000", 2024-09-13T11:24:27.921 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:27.921 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.921 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.921 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:24:27.921 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.065644+0000", 2024-09-13T11:24:27.921 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.1780000000000002e-06 2024-09-13T11:24:27.921 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.921 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.921 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:24:27.922 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.065695+0000", 2024-09-13T11:24:27.922 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.1632000000000003e-05 2024-09-13T11:24:27.922 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.922 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.922 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:24:27.922 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.065711+0000", 2024-09-13T11:24:27.922 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.5522000000000001e-05 2024-09-13T11:24:27.922 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.922 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.922 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:24:27.922 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.065719+0000", 2024-09-13T11:24:27.922 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.3480000000000004e-06 2024-09-13T11:24:27.922 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.922 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.922 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:24:27.923 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.065785+0000", 2024-09-13T11:24:27.923 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.5627e-05 2024-09-13T11:24:27.923 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.923 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.923 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for peered", 2024-09-13T11:24:27.923 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.065797+0000", 2024-09-13T11:24:27.923 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.1766000000000001e-05 2024-09-13T11:24:27.923 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.923 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.923 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:24:27.923 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.907248+0000", 2024-09-13T11:24:27.923 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.84145150300000005 2024-09-13T11:24:27.923 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.923 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.924 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:24:27.924 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.907256+0000", 2024-09-13T11:24:27.924 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.2549999999999997e-06 2024-09-13T11:24:27.924 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:27.924 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:24:27.924 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:27.924 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.924 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.924 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:54830 3.1e 3.b31554de (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-13T11:24:27.924 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:32.068259+0000", 2024-09-13T11:24:27.924 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 55.787683880000003, 2024-09-13T11:24:27.924 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.83902489599999996, 2024-09-13T11:24:27.924 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:24:27.925 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:24:27.925 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-13T11:24:27.925 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:24:27.925 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:24:27.925 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:27.925 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 54830 2024-09-13T11:24:27.925 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.925 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:24:27.925 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.925 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:24:27.925 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.068259+0000", 2024-09-13T11:24:27.925 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:27.925 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.925 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.926 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:24:27.926 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.068259+0000", 2024-09-13T11:24:27.926 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:27.926 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.926 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.926 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:24:27.926 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.068262+0000", 2024-09-13T11:24:27.926 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.1799999999999999e-06 2024-09-13T11:24:27.926 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.926 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.927 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:24:27.927 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.068301+0000", 2024-09-13T11:24:27.927 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.9289999999999998e-05 2024-09-13T11:24:27.927 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.927 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.927 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:24:27.927 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.068314+0000", 2024-09-13T11:24:27.927 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.3502999999999999e-05 2024-09-13T11:24:27.927 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.927 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.927 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:24:27.927 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.068324+0000", 2024-09-13T11:24:27.927 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.1630000000000002e-06 2024-09-13T11:24:27.928 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.928 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.928 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:24:27.928 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.068375+0000", 2024-09-13T11:24:27.928 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.1805000000000003e-05 2024-09-13T11:24:27.928 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.928 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.928 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for peered", 2024-09-13T11:24:27.928 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.068382+0000", 2024-09-13T11:24:27.928 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.224e-06 2024-09-13T11:24:27.928 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.928 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.928 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:24:27.928 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.907277+0000", 2024-09-13T11:24:27.929 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.83889509500000004 2024-09-13T11:24:27.929 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.929 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.929 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:24:27.929 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.907284+0000", 2024-09-13T11:24:27.929 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.6359999999999994e-06 2024-09-13T11:24:27.929 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:27.929 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:24:27.929 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:27.929 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.929 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.929 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:54846 3.1e 3.742e049e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-13T11:24:27.929 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:32.153567+0000", 2024-09-13T11:24:27.929 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 55.702376655999998, 2024-09-13T11:24:27.930 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.75378642399999995, 2024-09-13T11:24:27.930 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:24:27.930 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:24:27.930 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-13T11:24:27.930 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:24:27.930 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:24:27.930 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:27.930 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 54846 2024-09-13T11:24:27.930 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.931 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:24:27.931 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.931 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:24:27.931 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.153567+0000", 2024-09-13T11:24:27.931 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:27.931 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.931 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.931 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:24:27.931 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.153567+0000", 2024-09-13T11:24:27.931 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:27.931 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.931 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.932 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:24:27.932 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.153569+0000", 2024-09-13T11:24:27.932 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.4660000000000002e-06 2024-09-13T11:24:27.932 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.932 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.932 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:24:27.932 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.153619+0000", 2024-09-13T11:24:27.932 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.9617000000000003e-05 2024-09-13T11:24:27.932 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.932 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.932 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:24:27.932 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.153634+0000", 2024-09-13T11:24:27.932 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.4904e-05 2024-09-13T11:24:27.933 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.933 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.933 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:24:27.933 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.153642+0000", 2024-09-13T11:24:27.933 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.5599999999999994e-06 2024-09-13T11:24:27.933 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.933 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.933 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:24:27.933 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.153721+0000", 2024-09-13T11:24:27.933 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.8603999999999995e-05 2024-09-13T11:24:27.933 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.933 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.933 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for peered", 2024-09-13T11:24:27.934 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.153730+0000", 2024-09-13T11:24:27.934 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.3840000000000003e-06 2024-09-13T11:24:27.934 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.934 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.934 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:24:27.934 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.907346+0000", 2024-09-13T11:24:27.934 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.75361563099999995 2024-09-13T11:24:27.934 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.934 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.934 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:24:27.934 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.907353+0000", 2024-09-13T11:24:27.934 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.2579999999999998e-06 2024-09-13T11:24:27.934 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:27.935 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:24:27.935 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:27.935 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.935 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.935 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:54888 3.1e 3.43e14ede (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-13T11:24:27.935 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:32.423064+0000", 2024-09-13T11:24:27.935 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 55.432878985000002, 2024-09-13T11:24:27.935 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.48433585299999998, 2024-09-13T11:24:27.935 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:24:27.935 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:24:27.935 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-13T11:24:27.935 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:24:27.936 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:24:27.936 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:27.936 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 54888 2024-09-13T11:24:27.936 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.936 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:24:27.936 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.936 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:24:27.936 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.423064+0000", 2024-09-13T11:24:27.936 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:27.936 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.936 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.936 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:24:27.936 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.423064+0000", 2024-09-13T11:24:27.936 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:27.937 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.937 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.937 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:24:27.937 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.423069+0000", 2024-09-13T11:24:27.937 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.7389999999999999e-06 2024-09-13T11:24:27.937 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.937 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.937 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:24:27.937 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.423115+0000", 2024-09-13T11:24:27.937 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.5847000000000003e-05 2024-09-13T11:24:27.937 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.937 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.937 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:24:27.938 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.423126+0000", 2024-09-13T11:24:27.938 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.1418e-05 2024-09-13T11:24:27.938 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.938 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.938 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:24:27.938 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.423132+0000", 2024-09-13T11:24:27.938 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.57e-06 2024-09-13T11:24:27.938 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.938 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.938 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:24:27.938 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.423191+0000", 2024-09-13T11:24:27.938 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.9250999999999999e-05 2024-09-13T11:24:27.938 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.939 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.939 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for peered", 2024-09-13T11:24:27.939 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.423198+0000", 2024-09-13T11:24:27.939 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.9179999999999997e-06 2024-09-13T11:24:27.939 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.939 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.939 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:24:27.939 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.907392+0000", 2024-09-13T11:24:27.939 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.48419437500000001 2024-09-13T11:24:27.939 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.939 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.939 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:24:27.939 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.907400+0000", 2024-09-13T11:24:27.939 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.7349999999999996e-06 2024-09-13T11:24:27.939 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:27.940 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:24:27.940 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:27.940 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.940 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.940 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDPGPushReply(3.17 76/70 [PushReplyOp(3:eb8d2fae:::benchmark_data_smithi177_31116_object18267:head)])", 2024-09-13T11:24:27.940 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:32.636274+0000", 2024-09-13T11:24:27.940 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 55.219669392, 2024-09-13T11:24:27.940 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.269081036, 2024-09-13T11:24:27.940 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:24:27.940 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:24:27.940 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-13T11:24:27.940 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:24:27.940 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.941 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:24:27.941 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.636274+0000", 2024-09-13T11:24:27.941 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:27.941 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.941 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.941 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:24:27.941 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.636274+0000", 2024-09-13T11:24:27.941 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:27.941 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.941 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.941 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:24:27.941 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.636276+0000", 2024-09-13T11:24:27.941 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.1679999999999998e-06 2024-09-13T11:24:27.941 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.942 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.942 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:24:27.942 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.636284+0000", 2024-09-13T11:24:27.942 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.7440000000000007e-06 2024-09-13T11:24:27.942 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.942 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.942 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:24:27.942 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.904155+0000", 2024-09-13T11:24:27.942 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.26787080800000002 2024-09-13T11:24:27.942 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.942 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.942 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:24:27.942 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.904165+0000", 2024-09-13T11:24:27.942 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0750999999999999e-05 2024-09-13T11:24:27.942 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.943 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.943 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:24:27.943 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.904238+0000", 2024-09-13T11:24:27.943 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.2335000000000005e-05 2024-09-13T11:24:27.943 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.943 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.943 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:24:27.943 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.905355+0000", 2024-09-13T11:24:27.943 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0011172300000000001 2024-09-13T11:24:27.943 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:27.943 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:24:27.943 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:27.943 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.943 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.944 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDPGPushReply(3.17 76/70 [PushReplyOp(3:ebcbce67:::benchmark_data_smithi177_31116_object18268:head)])", 2024-09-13T11:24:27.944 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:32.641093+0000", 2024-09-13T11:24:27.944 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 55.214849844, 2024-09-13T11:24:27.944 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.27471406799999998, 2024-09-13T11:24:27.944 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:24:27.944 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:24:27.944 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-13T11:24:27.944 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:24:27.944 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.944 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:24:27.944 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.641093+0000", 2024-09-13T11:24:27.944 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:27.944 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.944 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.945 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:24:27.945 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.641093+0000", 2024-09-13T11:24:27.945 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:27.945 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.945 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.945 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:24:27.945 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.641094+0000", 2024-09-13T11:24:27.945 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.0299999999999999e-07 2024-09-13T11:24:27.945 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.945 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.945 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:24:27.945 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.641098+0000", 2024-09-13T11:24:27.945 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.2200000000000003e-06 2024-09-13T11:24:27.945 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.945 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.946 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:24:27.946 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.910434+0000", 2024-09-13T11:24:27.946 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.26933570600000001 2024-09-13T11:24:27.946 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.946 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.946 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:24:27.946 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.910442+0000", 2024-09-13T11:24:27.946 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.2700000000000004e-06 2024-09-13T11:24:27.946 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.946 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.946 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:24:27.946 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.914989+0000", 2024-09-13T11:24:27.946 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0045465059999999996 2024-09-13T11:24:27.946 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.947 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.947 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:24:27.947 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.915808+0000", 2024-09-13T11:24:27.947 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00081876300000000002 2024-09-13T11:24:27.947 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:27.947 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:24:27.947 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:27.947 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.947 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.947 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4665.0:59977 3.21 e83/78 3:86b4d8f6:::benchmark_data_smithi177_31116_object59976:head v 83'3136, mlcod=0'0)", 2024-09-13T11:24:27.947 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:40.571147+0000", 2024-09-13T11:24:27.947 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 47.284796536000002, 2024-09-13T11:24:27.947 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.51907542799999995, 2024-09-13T11:24:27.947 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:24:27.948 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:24:27.948 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:27.948 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:24:27.948 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.948 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:24:27.948 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:40.571147+0000", 2024-09-13T11:24:27.948 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:27.948 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.948 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.948 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:24:27.948 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:40.571147+0000", 2024-09-13T11:24:27.948 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:27.949 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.949 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.949 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:24:27.949 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:40.571151+0000", 2024-09-13T11:24:27.949 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.5650000000000003e-06 2024-09-13T11:24:27.949 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.949 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.949 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:24:27.949 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:40.571204+0000", 2024-09-13T11:24:27.949 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.2720999999999999e-05 2024-09-13T11:24:27.949 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.949 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.949 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:24:27.950 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:41.088332+0000", 2024-09-13T11:24:27.950 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.51712822000000003 2024-09-13T11:24:27.950 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.950 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.950 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:24:27.950 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:41.088341+0000", 2024-09-13T11:24:27.950 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.3100000000000001e-06 2024-09-13T11:24:27.950 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.950 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.950 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:24:27.950 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:41.088444+0000", 2024-09-13T11:24:27.950 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000103708 2024-09-13T11:24:27.950 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.951 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.951 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:24:27.951 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:41.088475+0000", 2024-09-13T11:24:27.951 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.0497e-05 2024-09-13T11:24:27.951 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.951 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.951 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:24:27.951 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:41.090192+0000", 2024-09-13T11:24:27.951 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.001717288 2024-09-13T11:24:27.951 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.951 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.951 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:24:27.951 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:41.090222+0000", 2024-09-13T11:24:27.951 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.0119000000000001e-05 2024-09-13T11:24:27.951 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:27.952 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:24:27.952 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:27.952 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.952 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.952 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:110286 3.26 3.c41c0fa6 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e119)", 2024-09-13T11:24:27.952 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:24:25.654272+0000", 2024-09-13T11:24:27.952 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 2.2016708440000001, 2024-09-13T11:24:27.952 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.43337235400000002, 2024-09-13T11:24:27.952 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:24:27.952 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:24:27.952 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-13T11:24:27.952 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:24:27.953 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:24:27.953 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:27.953 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 110286 2024-09-13T11:24:27.953 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.953 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:24:27.953 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.953 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:24:27.953 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:25.654272+0000", 2024-09-13T11:24:27.953 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:27.953 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.953 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.953 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:24:27.953 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:25.654272+0000", 2024-09-13T11:24:27.953 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:27.954 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.954 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.954 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:24:27.954 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:25.654275+0000", 2024-09-13T11:24:27.954 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.3199999999999998e-06 2024-09-13T11:24:27.954 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.954 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.954 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:24:27.954 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:25.654517+0000", 2024-09-13T11:24:27.954 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00024270199999999999 2024-09-13T11:24:27.954 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.954 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.954 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:24:27.954 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:25.654550+0000", 2024-09-13T11:24:27.955 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.2549000000000003e-05 2024-09-13T11:24:27.955 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.955 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.955 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:24:27.955 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:25.654566+0000", 2024-09-13T11:24:27.955 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.5517000000000001e-05 2024-09-13T11:24:27.955 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.955 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.955 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:24:27.955 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:25.654655+0000", 2024-09-13T11:24:27.955 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.9961000000000001e-05 2024-09-13T11:24:27.955 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.955 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.956 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for peered", 2024-09-13T11:24:27.956 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:25.654665+0000", 2024-09-13T11:24:27.956 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.0520000000000007e-06 2024-09-13T11:24:27.956 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.956 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.956 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:24:27.956 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:26.087629+0000", 2024-09-13T11:24:27.956 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.43296476900000003 2024-09-13T11:24:27.956 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:27.956 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:27.956 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:24:27.956 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:26.087645+0000", 2024-09-13T11:24:27.956 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.5483999999999999e-05 2024-09-13T11:24:27.957 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:27.957 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:24:27.957 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:27.957 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:27.957 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:24:27.957 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-13T11:24:27.959 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_ops_in_flight 2024-09-13T11:24:27.959 INFO:teuthology.orchestra.run.smithi001.stdout:osd.0: {} 2024-09-13T11:24:27.959 INFO:teuthology.orchestra.run.smithi001.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T11:24:27.960 INFO:tasks.ceph.osd.2.smithi162.stderr:2024-09-13T11:24:27.956+0000 7fe92b35a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26313) UID: 0 2024-09-13T11:24:27.969 INFO:teuthology.orchestra.run.smithi001.stdout:osd.1: {} 2024-09-13T11:24:27.970 INFO:teuthology.orchestra.run.smithi001.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T11:24:27.996 INFO:teuthology.orchestra.run.smithi001.stdout:osd.2: {} 2024-09-13T11:24:27.996 INFO:teuthology.orchestra.run.smithi001.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T11:24:28.009 INFO:tasks.radosbench.radosbench.0.smithi177.stdout: 121 16 110417 110401 57.0196 6 0.00546576 0.017416 2024-09-13T11:24:28.028 INFO:teuthology.orchestra.run.smithi001.stdout:osd.3: {} 2024-09-13T11:24:28.028 INFO:teuthology.orchestra.run.smithi001.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T11:24:28.052 INFO:teuthology.orchestra.run.smithi001.stdout:osd.4: {} 2024-09-13T11:24:28.052 INFO:teuthology.orchestra.run.smithi001.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T11:24:28.059 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-13T11:24:28.055+0000 7f6a68775640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26307) UID: 0 2024-09-13T11:24:28.078 INFO:teuthology.orchestra.run.smithi001.stdout:osd.5: {} 2024-09-13T11:24:28.078 INFO:teuthology.orchestra.run.smithi001.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T11:24:28.082 INFO:teuthology.orchestra.run.smithi001.stderr:marked out osd.14. 2024-09-13T11:24:28.082 INFO:tasks.ceph.osd.8.smithi001.stderr:2024-09-13T11:24:28.079+0000 7fb9ab2df640 -1 osd.8 121 handle_osd_map injecting map crc failure 2024-09-13T11:24:28.107 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-13T11:24:28.107 INFO:teuthology.orchestra.run.smithi001.stdout: "ops": [ 2024-09-13T11:24:28.107 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:28.107 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDPGPush(3.25 121/120 [PushOp(3:a5de2229:::benchmark_data_smithi177_31116_object1005:head, version: 43'62, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:a5de2229:::benchmark_data_smithi177_31116_object1005:head@43'62, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:24:28.107 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:24:27.379880+0000", 2024-09-13T11:24:28.107 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 0.72629573300000005, 2024-09-13T11:24:28.107 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.72634932200000002, 2024-09-13T11:24:28.107 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:24:28.107 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:24:28.107 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:24:28.108 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:24:28.108 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:28.108 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:24:28.108 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:27.379880+0000", 2024-09-13T11:24:28.108 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:28.108 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:28.108 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:28.108 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:24:28.108 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:27.379880+0000", 2024-09-13T11:24:28.108 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:28.108 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:28.108 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:28.108 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:24:28.109 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:27.379882+0000", 2024-09-13T11:24:28.109 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.1050000000000002e-06 2024-09-13T11:24:28.109 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:28.109 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:28.109 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:24:28.109 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:27.379976+0000", 2024-09-13T11:24:28.109 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.4468000000000004e-05 2024-09-13T11:24:28.109 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:28.109 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:28.109 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:24:28.109 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:28.105075+0000", 2024-09-13T11:24:28.109 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.72509911299999996 2024-09-13T11:24:28.110 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:28.110 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:28.110 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:24:28.110 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:28.105111+0000", 2024-09-13T11:24:28.110 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.5403e-05 2024-09-13T11:24:28.110 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:28.110 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:28.110 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:24:28.110 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:28.105267+0000", 2024-09-13T11:24:28.110 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00015663600000000001 2024-09-13T11:24:28.110 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:28.111 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:28.111 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:24:28.111 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:28.105291+0000", 2024-09-13T11:24:28.111 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.4164000000000001e-05 2024-09-13T11:24:28.111 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:28.111 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:24:28.111 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:28.111 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:28.111 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-09-13T11:24:28.111 INFO:teuthology.orchestra.run.smithi001.stdout: "num_ops": 1 2024-09-13T11:24:28.111 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-13T11:24:28.112 INFO:tasks.thrashosds.thrasher:Testing ceph-objectstore-tool on down osd.14 2024-09-13T11:24:28.113 INFO:tasks.thrashosds.thrasher:remote for osd 14 is ubuntu@smithi162.front.sepia.ceph.com 2024-09-13T11:24:28.113 DEBUG:teuthology.orchestra.run.smithi162:> type ceph-objectstore-tool 2024-09-13T11:24:28.118 INFO:teuthology.orchestra.run.smithi162.stdout:ceph-objectstore-tool is /usr/bin/ceph-objectstore-tool 2024-09-13T11:24:28.118 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-14 --journal-path /var/lib/ceph/osd/ceph-14/journal --op list-pgs 2024-09-13T11:24:28.119 INFO:teuthology.orchestra.run.smithi001.stdout:osd.6: {} 2024-09-13T11:24:28.119 INFO:teuthology.orchestra.run.smithi001.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T11:24:28.130 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_blocked_ops 2024-09-13T11:24:28.162 INFO:tasks.ceph.osd.10.smithi162.stderr:2024-09-13T11:24:28.160+0000 7fb59c9f0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26317) UID: 0 2024-09-13T11:24:28.166 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-13T11:24:28.260 INFO:tasks.ceph.osd.9.smithi092.stderr:2024-09-13T11:24:28.254+0000 7ff5ab3d0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26348) UID: 0 2024-09-13T11:24:28.314 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-13T11:24:28.314 INFO:teuthology.orchestra.run.smithi001.stdout: "ops": [], 2024-09-13T11:24:28.314 INFO:teuthology.orchestra.run.smithi001.stdout: "complaint_time": 30, 2024-09-13T11:24:28.315 INFO:teuthology.orchestra.run.smithi001.stdout: "num_blocked_ops": 0 2024-09-13T11:24:28.315 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-13T11:24:28.332 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_historic_ops 2024-09-13T11:24:28.361 INFO:tasks.ceph.osd.5.smithi092.stderr:2024-09-13T11:24:28.354+0000 7f6f0b705640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26346) UID: 0 2024-09-13T11:24:28.461 INFO:tasks.ceph.osd.5.smithi092.stderr:2024-09-13T11:24:28.458+0000 7f6f0b705640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26346) UID: 0 2024-09-13T11:24:28.464 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-13T11:24:28.464 INFO:teuthology.orchestra.run.smithi001.stdout: "size": 20, 2024-09-13T11:24:28.464 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 600, 2024-09-13T11:24:28.464 INFO:teuthology.orchestra.run.smithi001.stdout: "ops": [ 2024-09-13T11:24:28.464 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:28.464 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop_reply(client.4665.0:167 3.9 e41/30 ondisk, result = 0)", 2024-09-13T11:24:28.464 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:22:47.419696+0000", 2024-09-13T11:24:28.464 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 101.04180418599999, 2024-09-13T11:24:28.464 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.64615778199999996, 2024-09-13T11:24:28.464 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:24:28.465 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:24:28.465 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:24:28.465 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:24:28.465 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:28.465 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:24:28.465 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:47.419696+0000", 2024-09-13T11:24:28.465 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:28.465 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:28.465 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:28.465 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:24:28.465 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:47.419696+0000", 2024-09-13T11:24:28.465 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:28.465 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:28.465 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:28.466 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:24:28.466 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:47.419697+0000", 2024-09-13T11:24:28.466 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.15e-06 2024-09-13T11:24:28.466 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:28.466 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:28.466 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:24:28.466 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:47.419700+0000", 2024-09-13T11:24:28.466 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.2449999999999998e-06 2024-09-13T11:24:28.466 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:28.466 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:28.466 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:24:28.466 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:48.065637+0000", 2024-09-13T11:24:28.466 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.64593674300000004 2024-09-13T11:24:28.467 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:28.467 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:28.467 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:24:28.467 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:48.065648+0000", 2024-09-13T11:24:28.467 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0865e-05 2024-09-13T11:24:28.467 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:28.467 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:28.467 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:24:28.467 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:48.065720+0000", 2024-09-13T11:24:28.468 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.2020999999999994e-05 2024-09-13T11:24:28.468 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:28.468 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:28.468 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:24:28.468 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:48.065728+0000", 2024-09-13T11:24:28.468 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.7660000000000004e-06 2024-09-13T11:24:28.468 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:28.468 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:28.468 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:24:28.468 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:48.065853+0000", 2024-09-13T11:24:28.468 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00012499200000000001 2024-09-13T11:24:28.468 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:28.469 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:24:28.469 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:28.469 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:28.469 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:28.469 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:25383 3.0 3:0a6bb869:::benchmark_data_smithi177_31116_object25382:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e55)", 2024-09-13T11:24:28.469 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:04.447376+0000", 2024-09-13T11:24:28.469 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 84.014123517000002, 2024-09-13T11:24:28.469 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.21239864, 2024-09-13T11:24:28.469 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:24:28.469 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:24:28.469 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:28.469 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:24:28.469 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:24:28.469 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:28.470 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 25383 2024-09-13T11:24:28.470 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:28.470 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:24:28.470 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:28.470 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:24:28.470 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.447376+0000", 2024-09-13T11:24:28.470 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:28.470 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:28.470 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:28.470 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:24:28.470 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.447376+0000", 2024-09-13T11:24:28.470 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:28.470 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:28.470 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:28.471 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:24:28.471 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.447378+0000", 2024-09-13T11:24:28.471 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.6729999999999999e-06 2024-09-13T11:24:28.471 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:28.471 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:28.471 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:24:28.471 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.447433+0000", 2024-09-13T11:24:28.471 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.5371000000000002e-05 2024-09-13T11:24:28.471 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:28.471 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:28.471 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:24:28.471 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.447446+0000", 2024-09-13T11:24:28.471 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.3179999999999999e-05 2024-09-13T11:24:28.471 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:28.472 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:28.472 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:24:28.472 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.447456+0000", 2024-09-13T11:24:28.472 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0008e-05 2024-09-13T11:24:28.472 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:28.472 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:28.472 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:24:28.472 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.447514+0000", 2024-09-13T11:24:28.472 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.8023999999999999e-05 2024-09-13T11:24:28.472 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:28.472 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:28.472 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:24:28.473 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.447642+0000", 2024-09-13T11:24:28.473 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000127296 2024-09-13T11:24:28.473 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:28.473 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:28.473 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 15", 2024-09-13T11:24:28.473 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.447767+0000", 2024-09-13T11:24:28.473 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000125547 2024-09-13T11:24:28.473 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:28.473 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:28.473 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:24:28.473 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.449188+0000", 2024-09-13T11:24:28.474 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.001420604 2024-09-13T11:24:28.474 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:28.474 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:28.474 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:28.474 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.659668+0000", 2024-09-13T11:24:28.474 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.210479685 2024-09-13T11:24:28.474 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:28.474 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:28.474 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:24:28.474 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.659716+0000", 2024-09-13T11:24:28.475 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.8609999999999997e-05 2024-09-13T11:24:28.475 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:28.475 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:28.475 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:24:28.475 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.659775+0000", 2024-09-13T11:24:28.475 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.8641999999999999e-05 2024-09-13T11:24:28.475 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:28.475 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:24:28.475 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:28.475 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:28.475 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:28.475 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4665.0:25408 3.4 e56/40 3:294e23bd:::benchmark_data_smithi177_31116_object25407:head v 56'1645, mlcod=55'1642)", 2024-09-13T11:24:28.475 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:04.494686+0000", 2024-09-13T11:24:28.476 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 83.966813857999995, 2024-09-13T11:24:28.476 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.29725566399999998, 2024-09-13T11:24:28.476 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:24:28.476 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:24:28.476 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:28.476 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:24:28.476 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:28.476 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:24:28.476 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.494686+0000", 2024-09-13T11:24:28.476 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:28.476 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:28.476 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:28.476 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:24:28.476 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.494686+0000", 2024-09-13T11:24:28.477 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:28.477 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:28.477 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:28.477 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:24:28.477 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.494686+0000", 2024-09-13T11:24:28.477 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.2399999999999999e-07 2024-09-13T11:24:28.477 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:28.477 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:28.477 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:24:28.477 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.494809+0000", 2024-09-13T11:24:28.477 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00012308600000000001 2024-09-13T11:24:28.477 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:28.477 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:28.477 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:24:28.478 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.789410+0000", 2024-09-13T11:24:28.478 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.29460066899999998 2024-09-13T11:24:28.478 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:28.478 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:28.478 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:24:28.478 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.789425+0000", 2024-09-13T11:24:28.478 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.4600000000000001e-05 2024-09-13T11:24:28.478 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:28.478 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:28.478 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:24:28.478 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.789507+0000", 2024-09-13T11:24:28.478 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.2785000000000001e-05 2024-09-13T11:24:28.478 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:28.478 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:28.479 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:24:28.479 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.789525+0000", 2024-09-13T11:24:28.479 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.8117999999999998e-05 2024-09-13T11:24:28.479 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:28.479 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:28.479 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:24:28.479 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.791852+0000", 2024-09-13T11:24:28.479 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0023264459999999998 2024-09-13T11:24:28.479 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:28.479 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:28.479 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:24:28.479 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.791942+0000", 2024-09-13T11:24:28.479 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.9635999999999994e-05 2024-09-13T11:24:28.480 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:28.480 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:24:28.480 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:28.480 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:28.480 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:28.480 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4665.0:30815 3.f e59/30 3:fde17cb1:::benchmark_data_smithi177_31116_object30814:head v 59'1919, mlcod=59'1914)", 2024-09-13T11:24:28.480 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:09.494342+0000", 2024-09-13T11:24:28.480 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 78.967157872000001, 2024-09-13T11:24:28.480 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.22787855700000001, 2024-09-13T11:24:28.480 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:24:28.480 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:24:28.480 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:28.480 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:24:28.480 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:28.481 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:24:28.481 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.494342+0000", 2024-09-13T11:24:28.481 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:28.481 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:28.481 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:28.481 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:24:28.481 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.494342+0000", 2024-09-13T11:24:28.481 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:28.481 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:28.481 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:28.481 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:24:28.481 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.494343+0000", 2024-09-13T11:24:28.481 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.251e-06 2024-09-13T11:24:28.481 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:28.482 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:28.482 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:24:28.482 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.494387+0000", 2024-09-13T11:24:28.482 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.4276000000000001e-05 2024-09-13T11:24:28.482 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:28.482 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:28.482 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:24:28.482 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.720454+0000", 2024-09-13T11:24:28.482 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.226066614 2024-09-13T11:24:28.482 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:28.482 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:28.482 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:24:28.482 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.720462+0000", 2024-09-13T11:24:28.482 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.9240000000000007e-06 2024-09-13T11:24:28.483 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:28.483 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:28.483 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:24:28.483 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.720537+0000", 2024-09-13T11:24:28.483 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.4967000000000002e-05 2024-09-13T11:24:28.483 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:28.483 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:28.483 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:24:28.483 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.720563+0000", 2024-09-13T11:24:28.483 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.5630999999999999e-05 2024-09-13T11:24:28.483 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:28.483 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:28.483 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:24:28.483 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.722183+0000", 2024-09-13T11:24:28.483 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0016209670000000001 2024-09-13T11:24:28.484 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:28.484 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:28.484 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:24:28.484 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.722220+0000", 2024-09-13T11:24:28.484 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.6927000000000001e-05 2024-09-13T11:24:28.484 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:28.484 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:24:28.484 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:28.484 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:28.484 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:28.484 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4665.0:30828 3.f e59/30 3:f3a9c6a9:::benchmark_data_smithi177_31116_object30827:head v 59'1920, mlcod=59'1916)", 2024-09-13T11:24:28.484 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:09.502607+0000", 2024-09-13T11:24:28.484 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 78.958892814999999, 2024-09-13T11:24:28.485 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.26374408300000002, 2024-09-13T11:24:28.485 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:24:28.485 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:24:28.485 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:28.485 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:24:28.485 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:28.485 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:24:28.485 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.502607+0000", 2024-09-13T11:24:28.485 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:28.485 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:28.485 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:28.485 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:24:28.485 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.502607+0000", 2024-09-13T11:24:28.485 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:28.486 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:28.486 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:28.486 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:24:28.486 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.502609+0000", 2024-09-13T11:24:28.486 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.84e-06 2024-09-13T11:24:28.486 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:28.486 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:28.486 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:24:28.486 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.502663+0000", 2024-09-13T11:24:28.486 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.3971000000000002e-05 2024-09-13T11:24:28.486 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:28.486 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:28.486 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:24:28.486 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.763912+0000", 2024-09-13T11:24:28.487 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.26124904300000001 2024-09-13T11:24:28.487 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:28.487 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:28.487 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:24:28.487 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.763920+0000", 2024-09-13T11:24:28.487 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.1149999999999994e-06 2024-09-13T11:24:28.487 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:28.487 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:28.487 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:24:28.487 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.764024+0000", 2024-09-13T11:24:28.487 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00010380899999999999 2024-09-13T11:24:28.487 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:28.487 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:28.487 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:24:28.488 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.764068+0000", 2024-09-13T11:24:28.488 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.4638999999999997e-05 2024-09-13T11:24:28.488 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:28.488 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:28.488 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:24:28.488 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.766308+0000", 2024-09-13T11:24:28.488 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0022392039999999998 2024-09-13T11:24:28.488 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:28.488 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:28.488 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:24:28.488 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.766351+0000", 2024-09-13T11:24:28.488 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.3461999999999998e-05 2024-09-13T11:24:28.488 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:28.488 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:24:28.489 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:28.489 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:28.489 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:28.489 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:54351 3.10 3:085f283e:::benchmark_data_smithi177_31116_object54350:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e74)", 2024-09-13T11:24:28.489 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:30.599125+0000", 2024-09-13T11:24:28.489 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 57.862375110000002, 2024-09-13T11:24:28.489 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.167573261, 2024-09-13T11:24:28.489 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:24:28.489 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:24:28.489 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:28.489 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:24:28.489 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:24:28.489 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:28.490 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 54351 2024-09-13T11:24:28.490 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:28.490 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:24:28.490 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:28.490 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:24:28.490 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:30.599125+0000", 2024-09-13T11:24:28.490 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:28.490 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:28.490 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:28.490 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:24:28.490 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:30.599125+0000", 2024-09-13T11:24:28.490 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:28.490 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:28.491 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:28.491 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:24:28.491 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:30.599127+0000", 2024-09-13T11:24:28.491 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.847e-06 2024-09-13T11:24:28.491 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:28.491 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:28.491 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:24:28.491 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:30.599164+0000", 2024-09-13T11:24:28.491 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.6758999999999998e-05 2024-09-13T11:24:28.491 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:28.491 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:28.491 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:24:28.491 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:30.599184+0000", 2024-09-13T11:24:28.491 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.9928e-05 2024-09-13T11:24:28.492 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:28.492 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:28.492 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:24:28.492 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:30.599194+0000", 2024-09-13T11:24:28.492 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0035e-05 2024-09-13T11:24:28.492 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:28.492 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:28.492 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:24:28.492 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:30.599259+0000", 2024-09-13T11:24:28.492 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.5147000000000005e-05 2024-09-13T11:24:28.492 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:28.492 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:28.492 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:24:28.493 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:30.599381+0000", 2024-09-13T11:24:28.493 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000121876 2024-09-13T11:24:28.493 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:28.493 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:28.493 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 0,6", 2024-09-13T11:24:28.493 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:30.599496+0000", 2024-09-13T11:24:28.493 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000114307 2024-09-13T11:24:28.493 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:28.493 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:28.493 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:24:28.493 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:30.601094+0000", 2024-09-13T11:24:28.493 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.001598857 2024-09-13T11:24:28.493 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:28.494 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:28.494 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:28.494 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:30.601896+0000", 2024-09-13T11:24:28.494 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00080183800000000003 2024-09-13T11:24:28.494 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:28.494 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:28.494 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:28.494 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:30.766040+0000", 2024-09-13T11:24:28.494 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.16414414099999999 2024-09-13T11:24:28.494 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:28.494 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:28.494 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:24:28.494 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:30.766092+0000", 2024-09-13T11:24:28.495 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.1230000000000002e-05 2024-09-13T11:24:28.495 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:28.495 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:28.495 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:24:28.495 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:30.766698+0000", 2024-09-13T11:24:28.495 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00060629599999999998 2024-09-13T11:24:28.495 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:28.495 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:24:28.495 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:28.495 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:28.496 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:28.496 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDPGPush(3.1b 84/78 [PushOp(3:dcddd139:::benchmark_data_smithi177_31116_object10384:head, version: 48'673, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:dcddd139:::benchmark_data_smithi177_31116_object10384:head@48'673, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:24:28.496 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:41.400602+0000", 2024-09-13T11:24:28.496 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 47.060897330000003, 2024-09-13T11:24:28.496 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.43365821999999998, 2024-09-13T11:24:28.496 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:24:28.496 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:24:28.496 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:24:28.496 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:24:28.496 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:28.496 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:24:28.496 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:41.400602+0000", 2024-09-13T11:24:28.496 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:28.497 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:28.497 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:28.497 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:24:28.497 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:41.400602+0000", 2024-09-13T11:24:28.497 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:28.497 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:28.497 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:28.497 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:24:28.497 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:41.400604+0000", 2024-09-13T11:24:28.497 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.99e-06 2024-09-13T11:24:28.497 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:28.497 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:28.497 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:24:28.498 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:41.400653+0000", 2024-09-13T11:24:28.498 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.8195000000000002e-05 2024-09-13T11:24:28.498 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:28.498 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:28.498 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:24:28.498 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:41.833180+0000", 2024-09-13T11:24:28.498 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.43252769200000002 2024-09-13T11:24:28.498 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:28.498 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:28.498 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:24:28.498 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:41.833213+0000", 2024-09-13T11:24:28.498 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.2737999999999997e-05 2024-09-13T11:24:28.498 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:28.499 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:28.499 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:24:28.499 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:41.833408+0000", 2024-09-13T11:24:28.499 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00019473799999999999 2024-09-13T11:24:28.499 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:28.499 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:28.499 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:24:28.499 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:41.833417+0000", 2024-09-13T11:24:28.499 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.5349999999999993e-06 2024-09-13T11:24:28.499 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:28.499 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:28.499 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:24:28.499 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:41.834261+0000", 2024-09-13T11:24:28.499 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00084333200000000002 2024-09-13T11:24:28.500 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:28.500 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:24:28.500 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:28.500 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:28.500 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:28.500 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDPGPushReply(3.20 85/77 [PushReplyOp(3:066982a6:::benchmark_data_smithi177_31116_object16400:head)])", 2024-09-13T11:24:28.500 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:42.671551+0000", 2024-09-13T11:24:28.500 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 45.789948506999998, 2024-09-13T11:24:28.500 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.28249085499999999, 2024-09-13T11:24:28.500 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:24:28.500 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:24:28.500 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-13T11:24:28.501 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:24:28.501 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:28.501 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:24:28.501 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.671551+0000", 2024-09-13T11:24:28.501 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:28.501 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:28.501 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:28.501 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:24:28.501 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.671551+0000", 2024-09-13T11:24:28.501 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:28.501 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:28.501 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:28.501 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:24:28.501 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.671552+0000", 2024-09-13T11:24:28.502 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.097e-06 2024-09-13T11:24:28.502 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:28.502 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:28.502 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:24:28.502 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.671556+0000", 2024-09-13T11:24:28.502 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.8519999999999997e-06 2024-09-13T11:24:28.502 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:28.502 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:28.502 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:24:28.502 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.951769+0000", 2024-09-13T11:24:28.502 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.28021293000000003 2024-09-13T11:24:28.502 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:28.502 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:28.503 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:24:28.503 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.951780+0000", 2024-09-13T11:24:28.503 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.1059e-05 2024-09-13T11:24:28.503 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:28.503 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:28.503 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:24:28.503 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.951992+0000", 2024-09-13T11:24:28.503 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000211478 2024-09-13T11:24:28.503 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:28.503 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:28.503 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:24:28.503 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.954042+0000", 2024-09-13T11:24:28.503 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0020504389999999998 2024-09-13T11:24:28.503 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:28.504 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:24:28.504 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:28.504 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:28.504 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:28.504 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDPGPush(3.1b 89/78 [PushOp(3:dc4c79e7:::benchmark_data_smithi177_31116_object19857:head, version: 54'1266, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:dc4c79e7:::benchmark_data_smithi177_31116_object19857:head@54'1266, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:24:28.504 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:46.649128+0000", 2024-09-13T11:24:28.504 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 41.812371399, 2024-09-13T11:24:28.504 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.29963603, 2024-09-13T11:24:28.504 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:24:28.504 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:24:28.504 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:24:28.504 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:24:28.505 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:28.505 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:24:28.505 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.649128+0000", 2024-09-13T11:24:28.505 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:28.505 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:28.505 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:28.505 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:24:28.505 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.649128+0000", 2024-09-13T11:24:28.505 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:28.505 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:28.505 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:28.505 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:24:28.505 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.649130+0000", 2024-09-13T11:24:28.505 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2950000000000001e-06 2024-09-13T11:24:28.506 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:28.506 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:28.506 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:24:28.506 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.649197+0000", 2024-09-13T11:24:28.506 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.7033999999999995e-05 2024-09-13T11:24:28.506 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:28.506 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:28.506 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:24:28.506 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.947691+0000", 2024-09-13T11:24:28.506 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.29849387300000002 2024-09-13T11:24:28.506 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:28.506 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:28.506 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:24:28.507 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.947702+0000", 2024-09-13T11:24:28.507 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.1239e-05 2024-09-13T11:24:28.507 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:28.507 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:28.507 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:24:28.507 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.947799+0000", 2024-09-13T11:24:28.507 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.7283999999999999e-05 2024-09-13T11:24:28.507 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:28.507 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:28.507 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:24:28.507 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.947809+0000", 2024-09-13T11:24:28.507 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.008e-05 2024-09-13T11:24:28.507 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:28.507 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:28.507 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:24:28.508 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.948764+0000", 2024-09-13T11:24:28.508 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00095522499999999998 2024-09-13T11:24:28.508 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:28.508 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:24:28.508 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:28.508 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:28.508 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:28.508 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:71927 3.0 3:0197244a:::benchmark_data_smithi177_31116_object71926:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2024-09-13T11:24:28.508 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:54.199022+0000", 2024-09-13T11:24:28.508 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 34.262478043000002, 2024-09-13T11:24:28.508 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.26819617800000001, 2024-09-13T11:24:28.508 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:24:28.508 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:24:28.509 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:28.509 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:24:28.509 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:24:28.509 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:28.509 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 71927 2024-09-13T11:24:28.509 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:28.509 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:24:28.509 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:28.509 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:24:28.509 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:54.199022+0000", 2024-09-13T11:24:28.509 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:28.509 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:28.509 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:28.509 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:24:28.510 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:54.199022+0000", 2024-09-13T11:24:28.510 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:28.510 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:28.510 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:28.510 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:24:28.510 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:54.199025+0000", 2024-09-13T11:24:28.510 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.2200000000000001e-06 2024-09-13T11:24:28.510 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:28.510 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:28.510 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:24:28.510 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:54.199076+0000", 2024-09-13T11:24:28.510 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.1184000000000003e-05 2024-09-13T11:24:28.510 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:28.510 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:28.511 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:24:28.511 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:54.199095+0000", 2024-09-13T11:24:28.511 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.9253999999999999e-05 2024-09-13T11:24:28.511 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:28.511 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:28.511 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:24:28.511 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:54.199107+0000", 2024-09-13T11:24:28.511 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.1916000000000001e-05 2024-09-13T11:24:28.511 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:28.511 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:28.511 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:24:28.511 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:54.199196+0000", 2024-09-13T11:24:28.511 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.8645000000000003e-05 2024-09-13T11:24:28.511 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:28.512 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:28.512 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:24:28.512 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:54.199421+0000", 2024-09-13T11:24:28.512 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000224868 2024-09-13T11:24:28.512 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:28.512 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:28.512 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 15", 2024-09-13T11:24:28.512 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:54.199647+0000", 2024-09-13T11:24:28.512 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00022666099999999999 2024-09-13T11:24:28.512 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:28.512 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:28.512 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:24:28.512 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:54.201333+0000", 2024-09-13T11:24:28.512 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.001685761 2024-09-13T11:24:28.513 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:28.513 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:28.513 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:28.513 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:54.467097+0000", 2024-09-13T11:24:28.513 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.26576393399999998 2024-09-13T11:24:28.513 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:28.513 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:28.513 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:24:28.513 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:54.467163+0000", 2024-09-13T11:24:28.513 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.6088999999999997e-05 2024-09-13T11:24:28.513 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:28.513 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:28.513 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:24:28.513 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:54.467218+0000", 2024-09-13T11:24:28.514 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.4645999999999998e-05 2024-09-13T11:24:28.514 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:28.514 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:24:28.514 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:28.514 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:28.514 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:28.514 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:89731 3.20 3:04de772d:::benchmark_data_smithi177_31116_object89730:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110)", 2024-09-13T11:24:28.514 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:24:09.624252+0000", 2024-09-13T11:24:28.514 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 18.837247411, 2024-09-13T11:24:28.514 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.165003973, 2024-09-13T11:24:28.514 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:24:28.514 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:24:28.514 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:28.514 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:24:28.515 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:24:28.515 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:28.515 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 89731 2024-09-13T11:24:28.515 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:28.515 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:24:28.515 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:28.515 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:24:28.515 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:09.624252+0000", 2024-09-13T11:24:28.515 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:28.515 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:28.515 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:28.515 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:24:28.515 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:09.624252+0000", 2024-09-13T11:24:28.515 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:28.516 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:28.516 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:28.516 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:24:28.516 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:09.624255+0000", 2024-09-13T11:24:28.516 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.0450000000000001e-06 2024-09-13T11:24:28.516 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:28.516 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:28.516 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:24:28.516 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:09.624306+0000", 2024-09-13T11:24:28.516 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.0769999999999997e-05 2024-09-13T11:24:28.516 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:28.516 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:28.516 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:24:28.516 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:09.624326+0000", 2024-09-13T11:24:28.517 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.9409999999999999e-05 2024-09-13T11:24:28.517 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:28.517 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:28.517 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:24:28.517 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:09.624336+0000", 2024-09-13T11:24:28.517 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0709e-05 2024-09-13T11:24:28.517 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:28.517 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:28.517 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:24:28.517 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:09.624422+0000", 2024-09-13T11:24:28.517 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.619e-05 2024-09-13T11:24:28.517 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:28.517 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:28.517 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:24:28.517 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:09.624528+0000", 2024-09-13T11:24:28.518 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00010567499999999999 2024-09-13T11:24:28.518 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:28.518 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:28.518 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 0", 2024-09-13T11:24:28.518 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:09.624647+0000", 2024-09-13T11:24:28.518 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000118418 2024-09-13T11:24:28.518 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:28.518 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:28.518 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:24:28.518 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:09.625695+0000", 2024-09-13T11:24:28.518 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.001048799 2024-09-13T11:24:28.518 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:28.518 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:28.519 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:28.519 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:09.789054+0000", 2024-09-13T11:24:28.519 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.163358222 2024-09-13T11:24:28.519 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:28.519 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:28.519 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:24:28.519 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:09.789173+0000", 2024-09-13T11:24:28.519 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00011897599999999999 2024-09-13T11:24:28.519 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:28.519 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:28.519 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:24:28.519 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:09.789256+0000", 2024-09-13T11:24:28.519 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.3758999999999995e-05 2024-09-13T11:24:28.519 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:28.520 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:24:28.520 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:28.520 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:28.520 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:28.520 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4665.0:98479 3.2 e113/75 3:41084616:::benchmark_data_smithi177_31116_object98478:head v 113'3785, mlcod=113'3782)", 2024-09-13T11:24:28.520 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:24:15.343730+0000", 2024-09-13T11:24:28.520 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 13.117769772999999, 2024-09-13T11:24:28.520 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.196889648, 2024-09-13T11:24:28.520 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:24:28.520 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:24:28.520 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:28.520 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:24:28.520 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:28.521 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:24:28.521 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.343730+0000", 2024-09-13T11:24:28.521 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:28.521 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:28.521 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:28.521 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:24:28.521 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.343730+0000", 2024-09-13T11:24:28.521 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:28.521 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:28.521 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:28.521 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:24:28.521 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.343732+0000", 2024-09-13T11:24:28.521 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.6819999999999999e-06 2024-09-13T11:24:28.521 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:28.522 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:28.522 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:24:28.522 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.343781+0000", 2024-09-13T11:24:28.522 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.9691999999999998e-05 2024-09-13T11:24:28.522 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:28.522 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:28.522 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:24:28.522 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.538204+0000", 2024-09-13T11:24:28.522 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.19442271699999999 2024-09-13T11:24:28.522 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:28.522 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:28.522 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:24:28.522 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.538215+0000", 2024-09-13T11:24:28.522 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0648e-05 2024-09-13T11:24:28.523 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:28.523 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:28.523 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:24:28.523 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.538342+0000", 2024-09-13T11:24:28.523 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000127448 2024-09-13T11:24:28.523 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:28.523 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:28.523 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:24:28.523 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.538395+0000", 2024-09-13T11:24:28.523 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.2895000000000001e-05 2024-09-13T11:24:28.523 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:28.524 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:28.524 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:24:28.524 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.540544+0000", 2024-09-13T11:24:28.524 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0021487379999999999 2024-09-13T11:24:28.524 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:28.524 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:28.524 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:24:28.524 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.540620+0000", 2024-09-13T11:24:28.524 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.5828000000000007e-05 2024-09-13T11:24:28.524 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:28.524 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:24:28.524 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:28.524 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:28.525 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:28.525 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4665.0:98760 3.2 e113/75 3:4041dcea:::benchmark_data_smithi177_31116_object98759:head v 113'3787, mlcod=113'3784)", 2024-09-13T11:24:28.525 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:24:15.612920+0000", 2024-09-13T11:24:28.525 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 12.848579779, 2024-09-13T11:24:28.525 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.17136494999999999, 2024-09-13T11:24:28.525 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:24:28.525 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:24:28.525 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:28.525 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:24:28.525 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:28.526 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:24:28.526 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.612920+0000", 2024-09-13T11:24:28.526 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:28.526 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:28.526 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:28.526 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:24:28.526 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.612920+0000", 2024-09-13T11:24:28.526 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:28.526 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:28.526 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:28.526 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:24:28.526 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.612924+0000", 2024-09-13T11:24:28.526 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.7960000000000002e-06 2024-09-13T11:24:28.526 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:28.526 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:28.527 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:24:28.527 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.613010+0000", 2024-09-13T11:24:28.527 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.6106000000000005e-05 2024-09-13T11:24:28.527 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:28.527 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:28.527 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:24:28.527 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.781465+0000", 2024-09-13T11:24:28.527 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.16845532899999999 2024-09-13T11:24:28.527 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:28.527 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:28.527 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:24:28.527 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.781481+0000", 2024-09-13T11:24:28.527 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.6180000000000001e-05 2024-09-13T11:24:28.527 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:28.528 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:28.528 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:24:28.528 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.781698+0000", 2024-09-13T11:24:28.528 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00021694999999999999 2024-09-13T11:24:28.528 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:28.528 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:28.528 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:24:28.528 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.781750+0000", 2024-09-13T11:24:28.528 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.2129000000000002e-05 2024-09-13T11:24:28.528 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:28.528 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:28.528 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:24:28.528 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.784185+0000", 2024-09-13T11:24:28.528 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0024345180000000001 2024-09-13T11:24:28.529 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:28.529 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:28.529 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:24:28.529 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.784285+0000", 2024-09-13T11:24:28.529 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.9942e-05 2024-09-13T11:24:28.529 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:28.529 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:24:28.529 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:28.529 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:28.529 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:28.529 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4665.0:98954 3.2 e113/75 3:416baeb7:::benchmark_data_smithi177_31116_object98953:head v 113'3790, mlcod=113'3787)", 2024-09-13T11:24:28.529 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:24:15.819065+0000", 2024-09-13T11:24:28.529 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 12.642434891000001, 2024-09-13T11:24:28.529 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.187742985, 2024-09-13T11:24:28.530 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:24:28.530 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:24:28.530 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:28.530 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:24:28.530 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:28.530 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:24:28.530 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.819065+0000", 2024-09-13T11:24:28.530 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:28.530 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:28.530 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:28.530 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:24:28.530 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.819065+0000", 2024-09-13T11:24:28.530 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:28.531 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:28.531 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:28.531 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:24:28.531 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.819068+0000", 2024-09-13T11:24:28.531 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.0469999999999998e-06 2024-09-13T11:24:28.531 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:28.531 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:28.531 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:24:28.531 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.819124+0000", 2024-09-13T11:24:28.531 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.6335999999999997e-05 2024-09-13T11:24:28.531 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:28.531 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:28.531 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:24:28.531 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:16.003736+0000", 2024-09-13T11:24:28.531 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.18461164899999999 2024-09-13T11:24:28.532 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:28.532 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:28.532 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:24:28.532 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:16.003749+0000", 2024-09-13T11:24:28.532 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.3157e-05 2024-09-13T11:24:28.532 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:28.532 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:28.532 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:24:28.532 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:16.003903+0000", 2024-09-13T11:24:28.532 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00015374799999999999 2024-09-13T11:24:28.532 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:28.532 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:28.532 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:24:28.532 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:16.003961+0000", 2024-09-13T11:24:28.533 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.8026000000000003e-05 2024-09-13T11:24:28.533 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:28.533 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:28.533 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:24:28.533 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:16.006699+0000", 2024-09-13T11:24:28.533 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0027384290000000001 2024-09-13T11:24:28.533 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:28.533 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:28.533 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:24:28.533 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:16.006808+0000", 2024-09-13T11:24:28.533 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000108593 2024-09-13T11:24:28.533 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:28.533 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:24:28.533 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:28.534 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:28.534 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:28.534 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4665.0:99149 3.2 e113/75 3:4350ea1a:::benchmark_data_smithi177_31116_object99148:head v 113'3791, mlcod=113'3788)", 2024-09-13T11:24:28.534 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:24:16.003634+0000", 2024-09-13T11:24:28.534 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 12.45786537, 2024-09-13T11:24:28.534 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.19660476800000001, 2024-09-13T11:24:28.534 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:24:28.534 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:24:28.534 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:28.534 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:24:28.534 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:28.534 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:24:28.534 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:16.003634+0000", 2024-09-13T11:24:28.535 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:28.535 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:28.535 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:28.535 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:24:28.535 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:16.003634+0000", 2024-09-13T11:24:28.535 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:28.535 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:28.535 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:28.535 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:24:28.535 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:16.003637+0000", 2024-09-13T11:24:28.535 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.9749999999999999e-06 2024-09-13T11:24:28.535 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:28.535 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:28.535 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:24:28.536 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:16.003701+0000", 2024-09-13T11:24:28.536 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.3764999999999997e-05 2024-09-13T11:24:28.536 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:28.536 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:28.536 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:24:28.536 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:16.198858+0000", 2024-09-13T11:24:28.536 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.19515649900000001 2024-09-13T11:24:28.536 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:28.536 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:28.536 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:24:28.536 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:16.198863+0000", 2024-09-13T11:24:28.536 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.2229999999999998e-06 2024-09-13T11:24:28.536 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:28.536 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:28.537 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:24:28.537 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:16.198918+0000", 2024-09-13T11:24:28.537 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.5427000000000003e-05 2024-09-13T11:24:28.537 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:28.537 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:28.537 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:24:28.537 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:16.198935+0000", 2024-09-13T11:24:28.537 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.6776e-05 2024-09-13T11:24:28.537 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:28.537 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:28.537 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:24:28.537 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:16.200208+0000", 2024-09-13T11:24:28.537 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0012725429999999999 2024-09-13T11:24:28.538 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:28.538 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:28.538 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:24:28.538 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:16.200239+0000", 2024-09-13T11:24:28.538 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.1560000000000003e-05 2024-09-13T11:24:28.538 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:28.538 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:24:28.538 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:28.538 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:28.538 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:28.538 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4665.0:101036 3.4 e114/67 3:207f4fe8:::benchmark_data_smithi177_31116_object101035:head v 114'4508, mlcod=114'4505)", 2024-09-13T11:24:28.538 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:24:17.391594+0000", 2024-09-13T11:24:28.538 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 11.069905994000001, 2024-09-13T11:24:28.538 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.19061766999999999, 2024-09-13T11:24:28.539 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:24:28.539 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:24:28.539 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:28.539 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:24:28.539 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:28.539 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:24:28.539 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:17.391594+0000", 2024-09-13T11:24:28.539 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:28.539 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:28.539 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:28.539 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:24:28.539 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:17.391594+0000", 2024-09-13T11:24:28.539 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:28.539 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:28.540 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:28.540 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:24:28.540 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:17.391595+0000", 2024-09-13T11:24:28.540 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.6729999999999999e-06 2024-09-13T11:24:28.540 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:28.540 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:28.540 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:24:28.540 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:17.391668+0000", 2024-09-13T11:24:28.540 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.2096999999999998e-05 2024-09-13T11:24:28.540 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:28.540 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:28.540 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:24:28.540 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:17.579638+0000", 2024-09-13T11:24:28.540 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.187969993 2024-09-13T11:24:28.541 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:28.541 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:28.541 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:24:28.541 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:17.579644+0000", 2024-09-13T11:24:28.541 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.2049999999999999e-06 2024-09-13T11:24:28.541 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:28.541 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:28.541 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:24:28.541 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:17.579707+0000", 2024-09-13T11:24:28.541 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.2948999999999996e-05 2024-09-13T11:24:28.541 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:28.541 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:28.541 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:24:28.541 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:17.579731+0000", 2024-09-13T11:24:28.541 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.4315999999999999e-05 2024-09-13T11:24:28.542 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:28.542 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:28.542 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:24:28.542 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:17.582172+0000", 2024-09-13T11:24:28.542 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.002440705 2024-09-13T11:24:28.542 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:28.542 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:28.542 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:24:28.542 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:17.582211+0000", 2024-09-13T11:24:28.542 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.9731999999999998e-05 2024-09-13T11:24:28.542 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:28.542 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:24:28.542 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:28.542 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:28.543 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:28.543 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:108129 3.9 3:94038175:::benchmark_data_smithi177_31116_object108128:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116)", 2024-09-13T11:24:28.543 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:24:22.802908+0000", 2024-09-13T11:24:28.543 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 5.6585912739999999, 2024-09-13T11:24:28.543 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.179351336, 2024-09-13T11:24:28.543 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:24:28.543 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:24:28.543 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:28.543 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:24:28.543 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:24:28.543 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:28.543 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 108129 2024-09-13T11:24:28.543 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:28.543 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:24:28.544 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:28.544 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:24:28.544 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:22.802908+0000", 2024-09-13T11:24:28.544 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:28.544 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:28.544 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:28.544 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:24:28.544 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:22.802908+0000", 2024-09-13T11:24:28.544 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:28.544 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:28.544 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:28.544 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:24:28.544 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:22.802910+0000", 2024-09-13T11:24:28.544 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.765e-06 2024-09-13T11:24:28.545 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:28.545 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:28.545 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:24:28.545 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:22.802971+0000", 2024-09-13T11:24:28.545 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.0866000000000003e-05 2024-09-13T11:24:28.545 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:28.545 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:28.545 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:24:28.545 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:22.802985+0000", 2024-09-13T11:24:28.545 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.433e-05 2024-09-13T11:24:28.545 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:28.545 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:28.545 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:24:28.545 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:22.802993+0000", 2024-09-13T11:24:28.545 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.7940000000000003e-06 2024-09-13T11:24:28.546 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:28.546 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:28.546 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:24:28.546 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:22.803050+0000", 2024-09-13T11:24:28.546 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.6567999999999998e-05 2024-09-13T11:24:28.546 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:28.546 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:28.546 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:24:28.546 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:22.803158+0000", 2024-09-13T11:24:28.546 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00010788700000000001 2024-09-13T11:24:28.546 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:28.546 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:28.546 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 2", 2024-09-13T11:24:28.546 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:22.803262+0000", 2024-09-13T11:24:28.547 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00010450099999999999 2024-09-13T11:24:28.547 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:28.547 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:28.547 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:24:28.547 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:22.806107+0000", 2024-09-13T11:24:28.547 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0028447149999999998 2024-09-13T11:24:28.547 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:28.547 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:28.547 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:28.547 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:22.982142+0000", 2024-09-13T11:24:28.547 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.17603523700000001 2024-09-13T11:24:28.547 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:28.547 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:28.547 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:24:28.548 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:22.982196+0000", 2024-09-13T11:24:28.548 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.3832999999999998e-05 2024-09-13T11:24:28.548 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:28.548 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:28.548 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:24:28.548 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:22.982260+0000", 2024-09-13T11:24:28.548 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.3839999999999999e-05 2024-09-13T11:24:28.548 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:28.548 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:24:28.548 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:28.548 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:28.548 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:28.548 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop_reply(client.4665.0:108129 3.9 e116/67 ondisk, result = 0)", 2024-09-13T11:24:28.549 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:24:22.805315+0000", 2024-09-13T11:24:28.549 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 5.6561843879999998, 2024-09-13T11:24:28.549 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.17695082600000001, 2024-09-13T11:24:28.549 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:24:28.549 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:24:28.549 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:24:28.549 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:24:28.549 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:28.549 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:24:28.549 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:22.805315+0000", 2024-09-13T11:24:28.549 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:28.549 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:28.549 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:28.549 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:24:28.550 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:22.805315+0000", 2024-09-13T11:24:28.550 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:28.550 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:28.550 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:28.550 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:24:28.550 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:22.805317+0000", 2024-09-13T11:24:28.550 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.483e-06 2024-09-13T11:24:28.550 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:28.550 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:28.550 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:24:28.550 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:22.805335+0000", 2024-09-13T11:24:28.550 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.8428999999999999e-05 2024-09-13T11:24:28.550 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:28.551 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:28.551 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:24:28.551 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:22.982066+0000", 2024-09-13T11:24:28.551 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.176730521 2024-09-13T11:24:28.551 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:28.551 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:28.551 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:24:28.551 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:22.982072+0000", 2024-09-13T11:24:28.551 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.6359999999999997e-06 2024-09-13T11:24:28.551 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:28.551 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:28.551 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:24:28.551 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:22.982126+0000", 2024-09-13T11:24:28.552 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.3795999999999998e-05 2024-09-13T11:24:28.552 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:28.552 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:28.552 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:24:28.552 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:22.982135+0000", 2024-09-13T11:24:28.552 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.9120000000000004e-06 2024-09-13T11:24:28.552 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:28.552 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:28.552 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:24:28.552 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:22.982266+0000", 2024-09-13T11:24:28.552 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000131049 2024-09-13T11:24:28.552 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:28.552 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:24:28.553 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:28.553 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:28.553 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:28.553 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:110291 3.25 3.7c4a625 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e119)", 2024-09-13T11:24:28.553 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:24:25.933138+0000", 2024-09-13T11:24:28.553 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 2.5283620459999998, 2024-09-13T11:24:28.553 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.15437936399999999, 2024-09-13T11:24:28.553 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:24:28.553 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:24:28.553 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-13T11:24:28.553 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:24:28.553 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:24:28.553 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:28.553 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 110291 2024-09-13T11:24:28.554 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:28.554 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:24:28.554 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:28.554 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:24:28.554 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:25.933138+0000", 2024-09-13T11:24:28.554 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:28.554 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:28.554 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:28.554 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:24:28.554 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:25.933138+0000", 2024-09-13T11:24:28.554 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:28.554 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:28.554 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:28.555 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:24:28.555 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:25.933142+0000", 2024-09-13T11:24:28.555 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.6070000000000001e-06 2024-09-13T11:24:28.555 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:28.555 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:28.555 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:24:28.555 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:25.933199+0000", 2024-09-13T11:24:28.555 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.6913000000000002e-05 2024-09-13T11:24:28.555 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:28.555 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:28.555 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:24:28.555 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:25.933232+0000", 2024-09-13T11:24:28.555 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.3173000000000002e-05 2024-09-13T11:24:28.555 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:28.555 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:28.556 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:24:28.556 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:25.933255+0000", 2024-09-13T11:24:28.556 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.2575999999999999e-05 2024-09-13T11:24:28.556 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:28.556 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:28.556 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:24:28.556 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:25.933420+0000", 2024-09-13T11:24:28.556 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00016529199999999999 2024-09-13T11:24:28.556 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:28.556 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:28.556 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for peered", 2024-09-13T11:24:28.556 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:25.933437+0000", 2024-09-13T11:24:28.556 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.7176999999999998e-05 2024-09-13T11:24:28.557 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:28.557 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:28.557 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:24:28.557 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:26.087493+0000", 2024-09-13T11:24:28.557 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.15405592100000001 2024-09-13T11:24:28.557 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:28.557 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:28.557 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:24:28.557 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:26.087517+0000", 2024-09-13T11:24:28.557 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.3705000000000001e-05 2024-09-13T11:24:28.557 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:28.557 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:24:28.557 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:28.557 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:28.558 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:28.558 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDPGPush(3.25 121/120 [PushOp(3:a5de2229:::benchmark_data_smithi177_31116_object1005:head, version: 43'62, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:a5de2229:::benchmark_data_smithi177_31116_object1005:head@43'62, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:24:28.558 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:24:27.379880+0000", 2024-09-13T11:24:28.558 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 1.0816202050000001, 2024-09-13T11:24:28.558 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.72647719499999996, 2024-09-13T11:24:28.558 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:24:28.558 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:24:28.558 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:24:28.558 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:24:28.558 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:28.558 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:24:28.558 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:27.379880+0000", 2024-09-13T11:24:28.558 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:28.558 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:28.559 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:28.559 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:24:28.559 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:27.379880+0000", 2024-09-13T11:24:28.559 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:28.559 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:28.559 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:28.559 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:24:28.559 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:27.379882+0000", 2024-09-13T11:24:28.559 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.1050000000000002e-06 2024-09-13T11:24:28.559 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:28.559 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:28.559 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:24:28.559 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:27.379976+0000", 2024-09-13T11:24:28.560 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.4468000000000004e-05 2024-09-13T11:24:28.560 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:28.560 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:28.560 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:24:28.560 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:28.105075+0000", 2024-09-13T11:24:28.560 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.72509911299999996 2024-09-13T11:24:28.560 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:28.560 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:28.560 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:24:28.560 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:28.105111+0000", 2024-09-13T11:24:28.560 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.5403e-05 2024-09-13T11:24:28.560 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:28.560 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:28.560 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:24:28.561 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:28.105267+0000", 2024-09-13T11:24:28.561 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00015663600000000001 2024-09-13T11:24:28.561 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:28.561 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:28.561 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:24:28.561 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:28.105291+0000", 2024-09-13T11:24:28.561 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.4164000000000001e-05 2024-09-13T11:24:28.561 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:28.561 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:28.561 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:24:28.561 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:28.106357+0000", 2024-09-13T11:24:28.561 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.001065306 2024-09-13T11:24:28.561 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:28.561 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:24:28.562 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:28.562 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:28.562 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:24:28.562 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-13T11:24:28.563 INFO:tasks.ceph.osd.9.smithi092.stderr:2024-09-13T11:24:28.558+0000 7ff5ab3d0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26348) UID: 0 2024-09-13T11:24:28.564 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-13T11:24:28.664 INFO:tasks.ceph.osd.4.smithi001.stderr:2024-09-13T11:24:28.663+0000 7ff8e0218640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26310) UID: 0 2024-09-13T11:24:28.721 INFO:teuthology.orchestra.run.smithi092.stdout:ERROR: (22) Invalid argument 2024-09-13T11:24:28.721 INFO:teuthology.orchestra.run.smithi092.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:24:28.742 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-13T11:24:28.786 INFO:tasks.ceph.osd.1.smithi092.stderr:2024-09-13T11:24:28.782+0000 7fe7789c5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33718) UID: 0 2024-09-13T11:24:28.867 INFO:tasks.ceph.osd.10.smithi162.stderr:2024-09-13T11:24:28.864+0000 7fb59c9f0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26317) UID: 0 2024-09-13T11:24:28.891 INFO:teuthology.orchestra.run.smithi092.stdout:ERROR: (22) Invalid argument 2024-09-13T11:24:28.891 INFO:teuthology.orchestra.run.smithi092.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:24:28.911 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-13T11:24:28.937 INFO:teuthology.orchestra.run.smithi162.stdout:3.6 2024-09-13T11:24:28.937 INFO:teuthology.orchestra.run.smithi162.stdout:3.26 2024-09-13T11:24:28.937 INFO:teuthology.orchestra.run.smithi162.stdout:3.22 2024-09-13T11:24:28.937 INFO:teuthology.orchestra.run.smithi162.stdout:3.1f 2024-09-13T11:24:28.937 INFO:teuthology.orchestra.run.smithi162.stdout:3.1e 2024-09-13T11:24:28.937 INFO:teuthology.orchestra.run.smithi162.stdout:3.16 2024-09-13T11:24:28.937 INFO:teuthology.orchestra.run.smithi162.stdout:3.15 2024-09-13T11:24:28.937 INFO:teuthology.orchestra.run.smithi162.stdout:3.2 2024-09-13T11:24:28.937 INFO:teuthology.orchestra.run.smithi162.stdout:3.14 2024-09-13T11:24:28.938 INFO:teuthology.orchestra.run.smithi162.stdout:3.12 2024-09-13T11:24:28.938 INFO:teuthology.orchestra.run.smithi162.stdout:2.3 2024-09-13T11:24:28.938 INFO:teuthology.orchestra.run.smithi162.stdout:2.1 2024-09-13T11:24:28.953 INFO:teuthology.orchestra.run.smithi001.stdout:osd.7: {} 2024-09-13T11:24:28.953 INFO:teuthology.orchestra.run.smithi001.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T11:24:28.967 INFO:tasks.ceph.osd.11.smithi177.stderr:2024-09-13T11:24:28.963+0000 7f24792c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26580) UID: 0 2024-09-13T11:24:28.985 INFO:teuthology.orchestra.run.smithi001.stdout:osd.8: {} 2024-09-13T11:24:28.985 INFO:teuthology.orchestra.run.smithi001.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-13T11:24:29.010 INFO:tasks.radosbench.radosbench.0.smithi177.stdout: 122 16 110417 110401 56.5522 0 - 0.017416 2024-09-13T11:24:29.010 INFO:teuthology.orchestra.run.smithi001.stdout:osd.9: {} 2024-09-13T11:24:29.010 INFO:teuthology.orchestra.run.smithi001.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-13T11:24:29.039 INFO:teuthology.orchestra.run.smithi001.stdout:osd.10: {} 2024-09-13T11:24:29.039 INFO:teuthology.orchestra.run.smithi001.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-13T11:24:29.068 INFO:tasks.ceph.osd.5.smithi092.stderr:2024-09-13T11:24:29.062+0000 7f6f0b705640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26346) UID: 0 2024-09-13T11:24:29.069 INFO:teuthology.orchestra.run.smithi001.stdout:osd.11: {} 2024-09-13T11:24:29.069 INFO:teuthology.orchestra.run.smithi001.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-13T11:24:29.084 INFO:teuthology.orchestra.run.smithi001.stderr:nodeep-scrub is set 2024-09-13T11:24:29.098 INFO:teuthology.orchestra.run.smithi001.stdout:osd.12: {} 2024-09-13T11:24:29.098 INFO:teuthology.orchestra.run.smithi001.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-13T11:24:29.102 INFO:teuthology.orchestra.run.smithi092.stdout:ERROR: (22) Invalid argument 2024-09-13T11:24:29.103 INFO:teuthology.orchestra.run.smithi092.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:24:29.129 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-09-13T11:24:29.131 INFO:teuthology.orchestra.run.smithi001.stdout:osd.13: {} 2024-09-13T11:24:29.131 INFO:teuthology.orchestra.run.smithi001.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-13T11:24:29.131 INFO:teuthology.orchestra.run.smithi001.stderr:Error ENXIO: problem getting command descriptions from osd.14 2024-09-13T11:24:29.155 INFO:teuthology.orchestra.run.smithi001.stdout:osd.15: {} 2024-09-13T11:24:29.156 INFO:teuthology.orchestra.run.smithi001.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-13T11:24:29.159 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-13T11:24:29.159 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-13T11:24:29.178 INFO:tasks.ceph.osd.9.smithi092.stderr:2024-09-13T11:24:29.174+0000 7ff5ab3d0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26348) UID: 0 2024-09-13T11:24:29.227 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-14 --journal-path /var/lib/ceph/osd/ceph-14/journal --op export --pgid 3.2 --file /var/log/ceph/exp.3.2.14 2024-09-13T11:24:29.277 INFO:tasks.ceph.osd.13.smithi092.stderr:2024-09-13T11:24:29.266+0000 7f3e5afea640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26350) UID: 0 2024-09-13T11:24:29.365 INFO:teuthology.orchestra.run.smithi092.stdout:ERROR: (22) Invalid argument 2024-09-13T11:24:29.365 INFO:teuthology.orchestra.run.smithi092.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:24:29.370 INFO:tasks.ceph.osd.4.smithi001.stderr:2024-09-13T11:24:29.367+0000 7ff8e0218640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26310) UID: 0 2024-09-13T11:24:29.393 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-09-13T11:24:29.482 INFO:tasks.ceph.osd.9.smithi092.stderr:2024-09-13T11:24:29.478+0000 7ff5ab3d0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26348) UID: 0 2024-09-13T11:24:29.572 INFO:tasks.ceph.osd.12.smithi001.stderr:2024-09-13T11:24:29.571+0000 7f892c6e6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26314) UID: 0 2024-09-13T11:24:29.607 INFO:teuthology.orchestra.run.smithi092.stdout:ERROR: (22) Invalid argument 2024-09-13T11:24:29.607 INFO:teuthology.orchestra.run.smithi092.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:24:29.631 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-09-13T11:24:29.674 INFO:tasks.ceph.osd.2.smithi162.stderr:2024-09-13T11:24:29.672+0000 7fe92b35a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26313) UID: 0 2024-09-13T11:24:29.774 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-13T11:24:29.771+0000 7f6a68775640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26307) UID: 0 2024-09-13T11:24:29.845 INFO:teuthology.orchestra.run.smithi092.stdout:ERROR: (22) Invalid argument 2024-09-13T11:24:29.845 INFO:teuthology.orchestra.run.smithi092.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:24:29.867 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_ops_in_flight 2024-09-13T11:24:29.877 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-13T11:24:29.875+0000 7f6a68775640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26307) UID: 0 2024-09-13T11:24:29.976 INFO:tasks.ceph.osd.13.smithi092.stderr:2024-09-13T11:24:29.970+0000 7f3e5afea640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26350) UID: 0 2024-09-13T11:24:30.010 INFO:tasks.radosbench.radosbench.0.smithi177.stdout: 123 16 111318 111302 56.5502 28.1562 0.0128743 0.0176786 2024-09-13T11:24:30.037 INFO:teuthology.orchestra.run.smithi092.stdout:ERROR: (22) Invalid argument 2024-09-13T11:24:30.037 INFO:teuthology.orchestra.run.smithi092.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:24:30.062 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_blocked_ops 2024-09-13T11:24:30.101 INFO:teuthology.orchestra.run.smithi162.stderr:Exporting 3.2 info 3.2( v 119'3947 (0'0,119'3947] local-lis/les=75/76 n=1731 ec=30/30 lis/c=75/75 les/c/f=76/76/0 sis=75) 2024-09-13T11:24:30.105 INFO:tasks.ceph.osd.9.smithi092.stderr:2024-09-13T11:24:30.102+0000 7ff5ab3d0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26348) UID: 0 2024-09-13T11:24:30.113 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:40002d6b:::benchmark_data_smithi177_31116_object75887:head# 2024-09-13T11:24:30.114 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:400116a8:::benchmark_data_smithi177_31116_object88981:head# 2024-09-13T11:24:30.114 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:40037035:::benchmark_data_smithi177_31116_object89077:head# 2024-09-13T11:24:30.115 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4003dfa7:::benchmark_data_smithi177_31116_object69054:head# 2024-09-13T11:24:30.115 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4003f042:::benchmark_data_smithi177_31116_object73958:head# 2024-09-13T11:24:30.116 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:400448d7:::benchmark_data_smithi177_31116_object59583:head# 2024-09-13T11:24:30.117 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:40047b18:::benchmark_data_smithi177_31116_object58604:head# 2024-09-13T11:24:30.117 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:40052e2a:::benchmark_data_smithi177_31116_object91672:head# 2024-09-13T11:24:30.118 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:40058f96:::benchmark_data_smithi177_31116_object71551:head# 2024-09-13T11:24:30.120 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:40064d0d:::benchmark_data_smithi177_31116_object34318:head# 2024-09-13T11:24:30.120 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:400658dc:::benchmark_data_smithi177_31116_object380:head# 2024-09-13T11:24:30.120 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4006fcc8:::benchmark_data_smithi177_31116_object93228:head# 2024-09-13T11:24:30.124 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:40088aa9:::benchmark_data_smithi177_31116_object52520:head# 2024-09-13T11:24:30.125 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:40097539:::benchmark_data_smithi177_31116_object55848:head# 2024-09-13T11:24:30.125 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:40099444:::benchmark_data_smithi177_31116_object13996:head# 2024-09-13T11:24:30.125 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4009eb1d:::benchmark_data_smithi177_31116_object3494:head# 2024-09-13T11:24:30.125 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4009fd45:::benchmark_data_smithi177_31116_object43169:head# 2024-09-13T11:24:30.125 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:400a8c3a:::benchmark_data_smithi177_31116_object22108:head# 2024-09-13T11:24:30.126 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:400bd5d8:::benchmark_data_smithi177_31116_object15040:head# 2024-09-13T11:24:30.127 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:400d210c:::benchmark_data_smithi177_31116_object75432:head# 2024-09-13T11:24:30.128 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:400d89b0:::benchmark_data_smithi177_31116_object90016:head# 2024-09-13T11:24:30.130 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:400ea071:::benchmark_data_smithi177_31116_object92382:head# 2024-09-13T11:24:30.130 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:400ebb6e:::benchmark_data_smithi177_31116_object72630:head# 2024-09-13T11:24:30.137 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:400f019d:::benchmark_data_smithi177_31116_object96642:head# 2024-09-13T11:24:30.138 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:400f8950:::benchmark_data_smithi177_31116_object15356:head# 2024-09-13T11:24:30.138 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:400f9d51:::benchmark_data_smithi177_31116_object26316:head# 2024-09-13T11:24:30.138 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:400fecec:::benchmark_data_smithi177_31116_object82070:head# 2024-09-13T11:24:30.138 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4010577d:::benchmark_data_smithi177_31116_object30635:head# 2024-09-13T11:24:30.138 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4011168f:::benchmark_data_smithi177_31116_object62373:head# 2024-09-13T11:24:30.138 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:40130995:::benchmark_data_smithi177_31116_object30254:head# 2024-09-13T11:24:30.138 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:40132e23:::benchmark_data_smithi177_31116_object110039:head# 2024-09-13T11:24:30.138 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:40138ea2:::benchmark_data_smithi177_31116_object10011:head# 2024-09-13T11:24:30.138 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4013aea3:::benchmark_data_smithi177_31116_object94789:head# 2024-09-13T11:24:30.138 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:401440a6:::benchmark_data_smithi177_31116_object39885:head# 2024-09-13T11:24:30.139 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4015458f:::benchmark_data_smithi177_31116_object49345:head# 2024-09-13T11:24:30.139 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:40160e25:::benchmark_data_smithi177_31116_object54134:head# 2024-09-13T11:24:30.139 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:40162d9f:::benchmark_data_smithi177_31116_object84878:head# 2024-09-13T11:24:30.140 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:40163fbb:::benchmark_data_smithi177_31116_object95797:head# 2024-09-13T11:24:30.140 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:401716e7:::benchmark_data_smithi177_31116_object62252:head# 2024-09-13T11:24:30.140 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:40189b7d:::benchmark_data_smithi177_31116_object78303:head# 2024-09-13T11:24:30.140 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4019513e:::benchmark_data_smithi177_31116_object16991:head# 2024-09-13T11:24:30.140 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4019a456:::benchmark_data_smithi177_31116_object52363:head# 2024-09-13T11:24:30.141 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:401a1f70:::benchmark_data_smithi177_31116_object81378:head# 2024-09-13T11:24:30.141 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:401c0648:::benchmark_data_smithi177_31116_object23043:head# 2024-09-13T11:24:30.142 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:401c1c3b:::benchmark_data_smithi177_31116_object36717:head# 2024-09-13T11:24:30.143 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:401cd695:::benchmark_data_smithi177_31116_object56793:head# 2024-09-13T11:24:30.143 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:401d43e8:::benchmark_data_smithi177_31116_object91695:head# 2024-09-13T11:24:30.143 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:401d76bc:::benchmark_data_smithi177_31116_object99388:head# 2024-09-13T11:24:30.143 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:401d7835:::benchmark_data_smithi177_31116_object26887:head# 2024-09-13T11:24:30.144 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:401dd4d4:::benchmark_data_smithi177_31116_object56240:head# 2024-09-13T11:24:30.145 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:401e699c:::benchmark_data_smithi177_31116_object31726:head# 2024-09-13T11:24:30.145 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:401edb99:::benchmark_data_smithi177_31116_object100828:head# 2024-09-13T11:24:30.145 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:401ef1a7:::benchmark_data_smithi177_31116_object61250:head# 2024-09-13T11:24:30.145 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:401fa8e4:::benchmark_data_smithi177_31116_object27572:head# 2024-09-13T11:24:30.146 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4020e14d:::benchmark_data_smithi177_31116_object15817:head# 2024-09-13T11:24:30.147 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:402161fc:::benchmark_data_smithi177_31116_object55108:head# 2024-09-13T11:24:30.147 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4021ab69:::benchmark_data_smithi177_31116_object20661:head# 2024-09-13T11:24:30.147 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4022347d:::benchmark_data_smithi177_31116_object65012:head# 2024-09-13T11:24:30.148 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4022ccbd:::benchmark_data_smithi177_31116_object12406:head# 2024-09-13T11:24:30.149 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:40235564:::benchmark_data_smithi177_31116_object59542:head# 2024-09-13T11:24:30.149 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4023797a:::benchmark_data_smithi177_31116_object28457:head# 2024-09-13T11:24:30.150 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:40244812:::benchmark_data_smithi177_31116_object86273:head# 2024-09-13T11:24:30.150 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4024a979:::benchmark_data_smithi177_31116_object51662:head# 2024-09-13T11:24:30.151 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4025a717:::benchmark_data_smithi177_31116_object30274:head# 2024-09-13T11:24:30.151 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:40268a31:::benchmark_data_smithi177_31116_object70755:head# 2024-09-13T11:24:30.152 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4027dacb:::benchmark_data_smithi177_31116_object34591:head# 2024-09-13T11:24:30.153 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:402882e0:::benchmark_data_smithi177_31116_object73024:head# 2024-09-13T11:24:30.153 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4029b1e8:::benchmark_data_smithi177_31116_object61810:head# 2024-09-13T11:24:30.153 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:402a5335:::benchmark_data_smithi177_31116_object75143:head# 2024-09-13T11:24:30.154 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:402a5f44:::benchmark_data_smithi177_31116_object39853:head# 2024-09-13T11:24:30.155 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:402cd08b:::benchmark_data_smithi177_31116_object42476:head# 2024-09-13T11:24:30.155 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:402d046d:::benchmark_data_smithi177_31116_object93094:head# 2024-09-13T11:24:30.156 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:402d43e9:::benchmark_data_smithi177_31116_object34434:head# 2024-09-13T11:24:30.158 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:402dc97b:::benchmark_data_smithi177_31116_object20634:head# 2024-09-13T11:24:30.159 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:402dcb38:::benchmark_data_smithi177_31116_object103916:head# 2024-09-13T11:24:30.159 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:402e5899:::benchmark_data_smithi177_31116_object95399:head# 2024-09-13T11:24:30.160 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:402e627d:::benchmark_data_smithi177_31116_object104154:head# 2024-09-13T11:24:30.161 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:402ebe83:::benchmark_data_smithi177_31116_object80691:head# 2024-09-13T11:24:30.162 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:402f6f46:::benchmark_data_smithi177_31116_object87454:head# 2024-09-13T11:24:30.162 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:403065dd:::benchmark_data_smithi177_31116_object13292:head# 2024-09-13T11:24:30.163 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4030d737:::benchmark_data_smithi177_31116_object16511:head# 2024-09-13T11:24:30.163 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4030df1a:::benchmark_data_smithi177_31116_object70913:head# 2024-09-13T11:24:30.164 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4030f7c7:::benchmark_data_smithi177_31116_object57789:head# 2024-09-13T11:24:30.164 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4031b8f8:::benchmark_data_smithi177_31116_object71384:head# 2024-09-13T11:24:30.165 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4031d7a4:::benchmark_data_smithi177_31116_object86269:head# 2024-09-13T11:24:30.166 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4031fddf:::benchmark_data_smithi177_31116_object64167:head# 2024-09-13T11:24:30.166 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4031fe17:::benchmark_data_smithi177_31116_object60893:head# 2024-09-13T11:24:30.166 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4033dacb:::benchmark_data_smithi177_31116_object94537:head# 2024-09-13T11:24:30.167 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:40348e18:::benchmark_data_smithi177_31116_object64069:head# 2024-09-13T11:24:30.168 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4034ed0e:::benchmark_data_smithi177_31116_object15210:head# 2024-09-13T11:24:30.168 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:40357bd9:::benchmark_data_smithi177_31116_object47811:head# 2024-09-13T11:24:30.169 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:403583e9:::benchmark_data_smithi177_31116_object70931:head# 2024-09-13T11:24:30.169 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4035d24e:::benchmark_data_smithi177_31116_object25709:head# 2024-09-13T11:24:30.170 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:40361b04:::benchmark_data_smithi177_31116_object44813:head# 2024-09-13T11:24:30.170 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:40366f21:::benchmark_data_smithi177_31116_object23308:head# 2024-09-13T11:24:30.171 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4036998b:::benchmark_data_smithi177_31116_object69742:head# 2024-09-13T11:24:30.171 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:40370432:::benchmark_data_smithi177_31116_object21532:head# 2024-09-13T11:24:30.172 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:40371001:::benchmark_data_smithi177_31116_object103605:head# 2024-09-13T11:24:30.172 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:40377689:::benchmark_data_smithi177_31116_object21578:head# 2024-09-13T11:24:30.173 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:403782b2:::benchmark_data_smithi177_31116_object35224:head# 2024-09-13T11:24:30.173 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4037c73c:::benchmark_data_smithi177_31116_object31374:head# 2024-09-13T11:24:30.174 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:40385d5f:::benchmark_data_smithi177_31116_object81729:head# 2024-09-13T11:24:30.174 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4038b956:::benchmark_data_smithi177_31116_object3717:head# 2024-09-13T11:24:30.175 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:40394b85:::benchmark_data_smithi177_31116_object29315:head# 2024-09-13T11:24:30.175 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4039ccf8:::benchmark_data_smithi177_31116_object97045:head# 2024-09-13T11:24:30.176 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4039e8f1:::benchmark_data_smithi177_31116_object1034:head# 2024-09-13T11:24:30.177 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:403a5cf3:::benchmark_data_smithi177_31116_object16059:head# 2024-09-13T11:24:30.177 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:403ae3cd:::benchmark_data_smithi177_31116_object22499:head# 2024-09-13T11:24:30.178 INFO:tasks.ceph.osd.9.smithi092.stderr:2024-09-13T11:24:30.174+0000 7ff5ab3d0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26348) UID: 0 2024-09-13T11:24:30.179 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:403af28b:::benchmark_data_smithi177_31116_object61390:head# 2024-09-13T11:24:30.179 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:403af6ba:::benchmark_data_smithi177_31116_object57620:head# 2024-09-13T11:24:30.179 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:403bdb96:::benchmark_data_smithi177_31116_object5648:head# 2024-09-13T11:24:30.180 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:403c33cb:::benchmark_data_smithi177_31116_object3277:head# 2024-09-13T11:24:30.180 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:403cd19a:::benchmark_data_smithi177_31116_object38672:head# 2024-09-13T11:24:30.180 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:403db0c9:::benchmark_data_smithi177_31116_object39730:head# 2024-09-13T11:24:30.181 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:403de283:::benchmark_data_smithi177_31116_object109827:head# 2024-09-13T11:24:30.181 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:403f3d29:::benchmark_data_smithi177_31116_object39601:head# 2024-09-13T11:24:30.182 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:403f6dec:::benchmark_data_smithi177_31116_object17149:head# 2024-09-13T11:24:30.182 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4040904e:::benchmark_data_smithi177_31116_object59329:head# 2024-09-13T11:24:30.185 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:40409dcf:::benchmark_data_smithi177_31116_object72912:head# 2024-09-13T11:24:30.185 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:40414ec5:::benchmark_data_smithi177_31116_object14666:head# 2024-09-13T11:24:30.185 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4041dcea:::benchmark_data_smithi177_31116_object98759:head# 2024-09-13T11:24:30.185 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4041ff96:::benchmark_data_smithi177_31116_object61493:head# 2024-09-13T11:24:30.185 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4042c786:::benchmark_data_smithi177_31116_object48108:head# 2024-09-13T11:24:30.185 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4042cf94:::benchmark_data_smithi177_31116_object100748:head# 2024-09-13T11:24:30.186 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4042f7e6:::benchmark_data_smithi177_31116_object20525:head# 2024-09-13T11:24:30.187 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4043a50d:::benchmark_data_smithi177_31116_object84470:head# 2024-09-13T11:24:30.187 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:404491f9:::benchmark_data_smithi177_31116_object17623:head# 2024-09-13T11:24:30.188 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:40449c46:::benchmark_data_smithi177_31116_object76566:head# 2024-09-13T11:24:30.188 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4044ad5d:::benchmark_data_smithi177_31116_object20817:head# 2024-09-13T11:24:30.188 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4045117c:::benchmark_data_smithi177_31116_object106876:head# 2024-09-13T11:24:30.189 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:40452e6d:::benchmark_data_smithi177_31116_object74639:head# 2024-09-13T11:24:30.189 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:40457732:::benchmark_data_smithi177_31116_object8057:head# 2024-09-13T11:24:30.190 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4045a38c:::benchmark_data_smithi177_31116_object89567:head# 2024-09-13T11:24:30.190 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:404612b6:::benchmark_data_smithi177_31116_object24160:head# 2024-09-13T11:24:30.191 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:40465ce1:::benchmark_data_smithi177_31116_object5753:head# 2024-09-13T11:24:30.192 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4046fc26:::benchmark_data_smithi177_31116_object10072:head# 2024-09-13T11:24:30.192 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4047dd50:::benchmark_data_smithi177_31116_object46670:head# 2024-09-13T11:24:30.192 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:40481405:::benchmark_data_smithi177_31116_object21794:head# 2024-09-13T11:24:30.193 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4048776d:::benchmark_data_smithi177_31116_object7635:head# 2024-09-13T11:24:30.193 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4048b818:::benchmark_data_smithi177_31116_object37587:head# 2024-09-13T11:24:30.194 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4048f3d1:::benchmark_data_smithi177_31116_object14737:head# 2024-09-13T11:24:30.196 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4049e5ec:::benchmark_data_smithi177_31116_object50499:head# 2024-09-13T11:24:30.196 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:404ad0b9:::benchmark_data_smithi177_31116_object90297:head# 2024-09-13T11:24:30.197 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:404b5fbd:::benchmark_data_smithi177_31116_object55789:head# 2024-09-13T11:24:30.197 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:404b720a:::benchmark_data_smithi177_31116_object6649:head# 2024-09-13T11:24:30.198 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:404c078e:::benchmark_data_smithi177_31116_object62709:head# 2024-09-13T11:24:30.198 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:404e5bfd:::benchmark_data_smithi177_31116_object68453:head# 2024-09-13T11:24:30.199 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:404ec4e0:::benchmark_data_smithi177_31116_object86472:head# 2024-09-13T11:24:30.199 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:404f87df:::benchmark_data_smithi177_31116_object43855:head# 2024-09-13T11:24:30.200 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:404fbe60:::benchmark_data_smithi177_31116_object23568:head# 2024-09-13T11:24:30.200 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4050a7a7:::benchmark_data_smithi177_31116_object47856:head# 2024-09-13T11:24:30.201 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:40511b0b:::benchmark_data_smithi177_31116_object53328:head# 2024-09-13T11:24:30.201 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:40512b53:::benchmark_data_smithi177_31116_object103735:head# 2024-09-13T11:24:30.202 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:40521972:::benchmark_data_smithi177_31116_object87795:head# 2024-09-13T11:24:30.203 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4053e23b:::benchmark_data_smithi177_31116_object5178:head# 2024-09-13T11:24:30.203 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4053fbe5:::benchmark_data_smithi177_31116_object37987:head# 2024-09-13T11:24:30.205 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:40541404:::benchmark_data_smithi177_31116_object11947:head# 2024-09-13T11:24:30.205 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:405467dd:::benchmark_data_smithi177_31116_object30559:head# 2024-09-13T11:24:30.205 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4054a079:::benchmark_data_smithi177_31116_object6529:head# 2024-09-13T11:24:30.206 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4054a638:::benchmark_data_smithi177_31116_object18683:head# 2024-09-13T11:24:30.206 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4054fc5b:::benchmark_data_smithi177_31116_object19605:head# 2024-09-13T11:24:30.206 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:405565e3:::benchmark_data_smithi177_31116_object6965:head# 2024-09-13T11:24:30.207 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:40558416:::benchmark_data_smithi177_31116_object90983:head# 2024-09-13T11:24:30.208 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:40578804:::benchmark_data_smithi177_31116_object67465:head# 2024-09-13T11:24:30.208 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4057c338:::benchmark_data_smithi177_31116_object3442:head# 2024-09-13T11:24:30.209 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:40580b7d:::benchmark_data_smithi177_31116_object79080:head# 2024-09-13T11:24:30.210 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:405830c3:::benchmark_data_smithi177_31116_object96166:head# 2024-09-13T11:24:30.212 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4058e553:::benchmark_data_smithi177_31116_object88690:head# 2024-09-13T11:24:30.213 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4058f3d1:::benchmark_data_smithi177_31116_object99612:head# 2024-09-13T11:24:30.213 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:405915a7:::benchmark_data_smithi177_31116_object64426:head# 2024-09-13T11:24:30.214 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:40595787:::benchmark_data_smithi177_31116_object57149:head# 2024-09-13T11:24:30.214 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:405993e2:::benchmark_data_smithi177_31116_object39203:head# 2024-09-13T11:24:30.215 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4059dada:::benchmark_data_smithi177_31116_object16838:head# 2024-09-13T11:24:30.215 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:405a00c7:::benchmark_data_smithi177_31116_object23300:head# 2024-09-13T11:24:30.217 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:405b6806:::benchmark_data_smithi177_31116_object44618:head# 2024-09-13T11:24:30.217 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:405d6f88:::benchmark_data_smithi177_31116_object16298:head# 2024-09-13T11:24:30.218 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:405fbf73:::benchmark_data_smithi177_31116_object95706:head# 2024-09-13T11:24:30.218 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:405fca5f:::benchmark_data_smithi177_31116_object97509:head# 2024-09-13T11:24:30.219 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4060180d:::benchmark_data_smithi177_31116_object26300:head# 2024-09-13T11:24:30.219 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:40608b41:::benchmark_data_smithi177_31116_object106143:head# 2024-09-13T11:24:30.220 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:40618c9a:::benchmark_data_smithi177_31116_object26181:head# 2024-09-13T11:24:30.220 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4061b84e:::benchmark_data_smithi177_31116_object97221:head# 2024-09-13T11:24:30.221 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4061d8dc:::benchmark_data_smithi177_31116_object105481:head# 2024-09-13T11:24:30.221 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4061e27f:::benchmark_data_smithi177_31116_object10225:head# 2024-09-13T11:24:30.222 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:40629ab5:::benchmark_data_smithi177_31116_object44859:head# 2024-09-13T11:24:30.222 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:406460ac:::benchmark_data_smithi177_31116_object82037:head# 2024-09-13T11:24:30.224 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:40648f3d:::benchmark_data_smithi177_31116_object96143:head# 2024-09-13T11:24:30.224 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:40654a05:::benchmark_data_smithi177_31116_object79975:head# 2024-09-13T11:24:30.224 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:406584a3:::benchmark_data_smithi177_31116_object17522:head# 2024-09-13T11:24:30.224 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:40660b32:::benchmark_data_smithi177_31116_object100364:head# 2024-09-13T11:24:30.225 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:40661b03:::benchmark_data_smithi177_31116_object45300:head# 2024-09-13T11:24:30.225 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:40681e2e:::benchmark_data_smithi177_31116_object77139:head# 2024-09-13T11:24:30.226 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:406856aa:::benchmark_data_smithi177_31116_object96753:head# 2024-09-13T11:24:30.226 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4068f898:::benchmark_data_smithi177_31116_object95892:head# 2024-09-13T11:24:30.228 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:406a1e36:::benchmark_data_smithi177_31116_object7726:head# 2024-09-13T11:24:30.229 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:406af700:::benchmark_data_smithi177_31116_object38150:head# 2024-09-13T11:24:30.229 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:406b3b27:::benchmark_data_smithi177_31116_object95246:head# 2024-09-13T11:24:30.230 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:406b7b6c:::benchmark_data_smithi177_31116_object92039:head# 2024-09-13T11:24:30.232 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:406bc2cd:::benchmark_data_smithi177_31116_object21697:head# 2024-09-13T11:24:30.232 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:406c5c8c:::benchmark_data_smithi177_31116_object1723:head# 2024-09-13T11:24:30.232 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:406cef8a:::benchmark_data_smithi177_31116_object35316:head# 2024-09-13T11:24:30.233 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:406cfcc4:::benchmark_data_smithi177_31116_object12340:head# 2024-09-13T11:24:30.233 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:406e9f2d:::benchmark_data_smithi177_31116_object84610:head# 2024-09-13T11:24:30.233 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:406ed46f:::benchmark_data_smithi177_31116_object109484:head# 2024-09-13T11:24:30.234 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:406f007b:::benchmark_data_smithi177_31116_object24162:head# 2024-09-13T11:24:30.234 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:406f69e5:::benchmark_data_smithi177_31116_object63684:head# 2024-09-13T11:24:30.237 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:406f9ff9:::benchmark_data_smithi177_31116_object81703:head# 2024-09-13T11:24:30.237 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:40707a1f:::benchmark_data_smithi177_31116_object54782:head# 2024-09-13T11:24:30.238 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:40707ad3:::benchmark_data_smithi177_31116_object91144:head# 2024-09-13T11:24:30.238 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4070ad99:::benchmark_data_smithi177_31116_object21741:head# 2024-09-13T11:24:30.239 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4070f3cf:::benchmark_data_smithi177_31116_object93762:head# 2024-09-13T11:24:30.239 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4071052f:::benchmark_data_smithi177_31116_object13435:head# 2024-09-13T11:24:30.240 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4071266a:::benchmark_data_smithi177_31116_object105287:head# 2024-09-13T11:24:30.241 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:40744611:::benchmark_data_smithi177_31116_object90652:head# 2024-09-13T11:24:30.241 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4074ef43:::benchmark_data_smithi177_31116_object64357:head# 2024-09-13T11:24:30.243 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:40784398:::benchmark_data_smithi177_31116_object82598:head# 2024-09-13T11:24:30.243 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:407873b2:::benchmark_data_smithi177_31116_object35539:head# 2024-09-13T11:24:30.246 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4079a4a7:::benchmark_data_smithi177_31116_object9064:head# 2024-09-13T11:24:30.246 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:407b61eb:::benchmark_data_smithi177_31116_object76617:head# 2024-09-13T11:24:30.248 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:407bb35d:::benchmark_data_smithi177_31116_object81599:head# 2024-09-13T11:24:30.248 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:407c9472:::benchmark_data_smithi177_31116_object31988:head# 2024-09-13T11:24:30.249 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:407cac0e:::benchmark_data_smithi177_31116_object102091:head# 2024-09-13T11:24:30.249 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:407d4821:::benchmark_data_smithi177_31116_object98908:head# 2024-09-13T11:24:30.251 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:407d805a:::benchmark_data_smithi177_31116_object44000:head# 2024-09-13T11:24:30.251 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:407d8c80:::benchmark_data_smithi177_31116_object85660:head# 2024-09-13T11:24:30.254 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:407d9c7c:::benchmark_data_smithi177_31116_object48505:head# 2024-09-13T11:24:30.254 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:407ea4da:::benchmark_data_smithi177_31116_object66606:head# 2024-09-13T11:24:30.255 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:407ebd4c:::benchmark_data_smithi177_31116_object26432:head# 2024-09-13T11:24:30.257 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:407f3e8d:::benchmark_data_smithi177_31116_object74793:head# 2024-09-13T11:24:30.260 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:408035a3:::benchmark_data_smithi177_31116_object109727:head# 2024-09-13T11:24:30.261 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:40820a4f:::benchmark_data_smithi177_31116_object87769:head# 2024-09-13T11:24:30.261 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4082f214:::benchmark_data_smithi177_31116_object103403:head# 2024-09-13T11:24:30.262 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:408400ab:::benchmark_data_smithi177_31116_object16748:head# 2024-09-13T11:24:30.263 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:40849903:::benchmark_data_smithi177_31116_object31794:head# 2024-09-13T11:24:30.263 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4084b0be:::benchmark_data_smithi177_31116_object47964:head# 2024-09-13T11:24:30.266 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:408579ea:::benchmark_data_smithi177_31116_object86266:head# 2024-09-13T11:24:30.266 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:40866b6b:::benchmark_data_smithi177_31116_object66543:head# 2024-09-13T11:24:30.267 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:40868ffb:::benchmark_data_smithi177_31116_object12160:head# 2024-09-13T11:24:30.267 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4088abd4:::benchmark_data_smithi177_31116_object16799:head# 2024-09-13T11:24:30.268 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4088b530:::benchmark_data_smithi177_31116_object27051:head# 2024-09-13T11:24:30.268 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4088f40d:::benchmark_data_smithi177_31116_object58187:head# 2024-09-13T11:24:30.269 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4089983f:::benchmark_data_smithi177_31116_object97006:head# 2024-09-13T11:24:30.269 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:408a6711:::benchmark_data_smithi177_31116_object74792:head# 2024-09-13T11:24:30.271 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:408ae0aa:::benchmark_data_smithi177_31116_object69156:head# 2024-09-13T11:24:30.271 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:408aee5d:::benchmark_data_smithi177_31116_object26857:head# 2024-09-13T11:24:30.272 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:408b703e:::benchmark_data_smithi177_31116_object84690:head# 2024-09-13T11:24:30.273 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:408bbdaf:::benchmark_data_smithi177_31116_object18078:head# 2024-09-13T11:24:30.273 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:408bcede:::benchmark_data_smithi177_31116_object71959:head# 2024-09-13T11:24:30.274 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:408c2697:::benchmark_data_smithi177_31116_object36451:head# 2024-09-13T11:24:30.274 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:408c2f58:::benchmark_data_smithi177_31116_object18693:head# 2024-09-13T11:24:30.278 INFO:teuthology.orchestra.run.smithi092.stdout:ERROR: (22) Invalid argument 2024-09-13T11:24:30.278 INFO:teuthology.orchestra.run.smithi092.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:24:30.279 INFO:tasks.ceph.osd.13.smithi092.stderr:2024-09-13T11:24:30.274+0000 7f3e5afea640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26350) UID: 0 2024-09-13T11:24:30.280 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:408cb6df:::benchmark_data_smithi177_31116_object83595:head# 2024-09-13T11:24:30.280 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:408edef3:::benchmark_data_smithi177_31116_object74445:head# 2024-09-13T11:24:30.281 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:408f0315:::benchmark_data_smithi177_31116_object90888:head# 2024-09-13T11:24:30.281 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:408f15a5:::benchmark_data_smithi177_31116_object65625:head# 2024-09-13T11:24:30.281 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:408f4706:::benchmark_data_smithi177_31116_object55363:head# 2024-09-13T11:24:30.281 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:408f7efe:::benchmark_data_smithi177_31116_object36890:head# 2024-09-13T11:24:30.281 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:409105a5:::benchmark_data_smithi177_31116_object64379:head# 2024-09-13T11:24:30.284 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4091a60c:::benchmark_data_smithi177_31116_object2755:head# 2024-09-13T11:24:30.285 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4092dfc2:::benchmark_data_smithi177_31116_object55389:head# 2024-09-13T11:24:30.285 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4093efd1:::benchmark_data_smithi177_31116_object76568:head# 2024-09-13T11:24:30.287 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4095a75c:::benchmark_data_smithi177_31116_object725:head# 2024-09-13T11:24:30.287 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:40971cef:::benchmark_data_smithi177_31116_object931:head# 2024-09-13T11:24:30.288 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:40979a20:::benchmark_data_smithi177_31116_object83710:head# 2024-09-13T11:24:30.288 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:40980402:::benchmark_data_smithi177_31116_object37357:head# 2024-09-13T11:24:30.289 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:40981b34:::benchmark_data_smithi177_31116_object100668:head# 2024-09-13T11:24:30.289 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:40994b44:::benchmark_data_smithi177_31116_object54692:head# 2024-09-13T11:24:30.290 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:40998238:::benchmark_data_smithi177_31116_object30010:head# 2024-09-13T11:24:30.291 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:409a19ba:::benchmark_data_smithi177_31116_object93749:head# 2024-09-13T11:24:30.291 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:409abf85:::benchmark_data_smithi177_31116_object6028:head# 2024-09-13T11:24:30.292 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:409b286b:::benchmark_data_smithi177_31116_object82725:head# 2024-09-13T11:24:30.292 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:409c8657:::benchmark_data_smithi177_31116_object79371:head# 2024-09-13T11:24:30.293 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:409ce5d8:::benchmark_data_smithi177_31116_object36032:head# 2024-09-13T11:24:30.294 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:409d3b63:::benchmark_data_smithi177_31116_object89524:head# 2024-09-13T11:24:30.294 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:409d6ad2:::benchmark_data_smithi177_31116_object97365:head# 2024-09-13T11:24:30.295 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:409e297b:::benchmark_data_smithi177_31116_object195:head# 2024-09-13T11:24:30.296 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:409e6f98:::benchmark_data_smithi177_31116_object80568:head# 2024-09-13T11:24:30.296 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:409ec20a:::benchmark_data_smithi177_31116_object84640:head# 2024-09-13T11:24:30.297 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:409fa531:::benchmark_data_smithi177_31116_object92802:head# 2024-09-13T11:24:30.297 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:409fbcbc:::benchmark_data_smithi177_31116_object52019:head# 2024-09-13T11:24:30.298 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:40a01dbd:::benchmark_data_smithi177_31116_object3315:head# 2024-09-13T11:24:30.298 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:40a04f16:::benchmark_data_smithi177_31116_object80186:head# 2024-09-13T11:24:30.299 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:40a157c5:::benchmark_data_smithi177_31116_object66904:head# 2024-09-13T11:24:30.299 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:40a19f93:::benchmark_data_smithi177_31116_object31776:head# 2024-09-13T11:24:30.300 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_historic_ops 2024-09-13T11:24:30.301 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:40a35ddc:::benchmark_data_smithi177_31116_object69248:head# 2024-09-13T11:24:30.301 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:40a3fc7e:::benchmark_data_smithi177_31116_object106718:head# 2024-09-13T11:24:30.301 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:40a4ddff:::benchmark_data_smithi177_31116_object28485:head# 2024-09-13T11:24:30.302 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:40a510f5:::benchmark_data_smithi177_31116_object24708:head# 2024-09-13T11:24:30.302 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:40a5d6b1:::benchmark_data_smithi177_31116_object42959:head# 2024-09-13T11:24:30.302 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:40a66d2a:::benchmark_data_smithi177_31116_object52316:head# 2024-09-13T11:24:30.303 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:40a670f4:::benchmark_data_smithi177_31116_object50155:head# 2024-09-13T11:24:30.303 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:40a6c2d1:::benchmark_data_smithi177_31116_object23636:head# 2024-09-13T11:24:30.304 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:40a7189f:::benchmark_data_smithi177_31116_object90532:head# 2024-09-13T11:24:30.308 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:40a7ca54:::benchmark_data_smithi177_31116_object45534:head# 2024-09-13T11:24:30.309 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:40a91169:::benchmark_data_smithi177_31116_object95088:head# 2024-09-13T11:24:30.309 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:40a92a5e:::benchmark_data_smithi177_31116_object41648:head# 2024-09-13T11:24:30.309 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:40a9552f:::benchmark_data_smithi177_31116_object4703:head# 2024-09-13T11:24:30.309 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:40aa2bb6:::benchmark_data_smithi177_31116_object68206:head# 2024-09-13T11:24:30.309 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:40abefe4:::benchmark_data_smithi177_31116_object77897:head# 2024-09-13T11:24:30.309 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:40ac4d19:::benchmark_data_smithi177_31116_object65373:head# 2024-09-13T11:24:30.309 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:40ac5e68:::benchmark_data_smithi177_31116_object63862:head# 2024-09-13T11:24:30.310 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:40ac8f47:::benchmark_data_smithi177_31116_object43057:head# 2024-09-13T11:24:30.310 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:40ad487c:::benchmark_data_smithi177_31116_object64366:head# 2024-09-13T11:24:30.311 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:40adf2c0:::benchmark_data_smithi177_31116_object2324:head# 2024-09-13T11:24:30.311 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:40ae4349:::benchmark_data_smithi177_31116_object72880:head# 2024-09-13T11:24:30.312 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:40ae578b:::benchmark_data_smithi177_31116_object77370:head# 2024-09-13T11:24:30.313 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:40ae8d8f:::benchmark_data_smithi177_31116_object82993:head# 2024-09-13T11:24:30.313 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:40b0e1d8:::benchmark_data_smithi177_31116_object83481:head# 2024-09-13T11:24:30.314 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:40b404e9:::benchmark_data_smithi177_31116_object82223:head# 2024-09-13T11:24:30.314 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:40b44312:::benchmark_data_smithi177_31116_object63807:head# 2024-09-13T11:24:30.315 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:40b47f82:::benchmark_data_smithi177_31116_object65629:head# 2024-09-13T11:24:30.315 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:40b53c00:::benchmark_data_smithi177_31116_object31972:head# 2024-09-13T11:24:30.316 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:40b59db7:::benchmark_data_smithi177_31116_object77737:head# 2024-09-13T11:24:30.317 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:40b644b3:::benchmark_data_smithi177_31116_object40111:head# 2024-09-13T11:24:30.318 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:40b6ee61:::benchmark_data_smithi177_31116_object11898:head# 2024-09-13T11:24:30.318 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:40b897c7:::benchmark_data_smithi177_31116_object31108:head# 2024-09-13T11:24:30.319 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:40b98f76:::benchmark_data_smithi177_31116_object52754:head# 2024-09-13T11:24:30.319 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:40b9f246:::benchmark_data_smithi177_31116_object98453:head# 2024-09-13T11:24:30.320 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:40baa63f:::benchmark_data_smithi177_31116_object13190:head# 2024-09-13T11:24:30.320 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:40bacac5:::benchmark_data_smithi177_31116_object80088:head# 2024-09-13T11:24:30.321 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:40bafaff:::benchmark_data_smithi177_31116_object85911:head# 2024-09-13T11:24:30.321 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:40bb6c7e:::benchmark_data_smithi177_31116_object45109:head# 2024-09-13T11:24:30.322 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:40bc66b9:::benchmark_data_smithi177_31116_object54201:head# 2024-09-13T11:24:30.322 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:40bc9cf4:::benchmark_data_smithi177_31116_object32031:head# 2024-09-13T11:24:30.323 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:40bce023:::benchmark_data_smithi177_31116_object11401:head# 2024-09-13T11:24:30.323 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:40bd1e0f:::benchmark_data_smithi177_31116_object70709:head# 2024-09-13T11:24:30.324 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:40bdd695:::benchmark_data_smithi177_31116_object3743:head# 2024-09-13T11:24:30.325 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:40be5f7e:::benchmark_data_smithi177_31116_object76016:head# 2024-09-13T11:24:30.325 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:40be629e:::benchmark_data_smithi177_31116_object14956:head# 2024-09-13T11:24:30.326 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:40bf26c0:::benchmark_data_smithi177_31116_object40744:head# 2024-09-13T11:24:30.327 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:40bfb8fd:::benchmark_data_smithi177_31116_object103063:head# 2024-09-13T11:24:30.327 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:40c072cc:::benchmark_data_smithi177_31116_object62579:head# 2024-09-13T11:24:30.328 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:40c250a3:::benchmark_data_smithi177_31116_object85596:head# 2024-09-13T11:24:30.329 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:40c2e91e:::benchmark_data_smithi177_31116_object20250:head# 2024-09-13T11:24:30.329 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:40c4535c:::benchmark_data_smithi177_31116_object102231:head# 2024-09-13T11:24:30.330 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:40c469c0:::benchmark_data_smithi177_31116_object57322:head# 2024-09-13T11:24:30.330 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:40c69484:::benchmark_data_smithi177_31116_object101907:head# 2024-09-13T11:24:30.331 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:40c7f056:::benchmark_data_smithi177_31116_object64349:head# 2024-09-13T11:24:30.331 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:40c98312:::benchmark_data_smithi177_31116_object24199:head# 2024-09-13T11:24:30.331 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:40c9e423:::benchmark_data_smithi177_31116_object81904:head# 2024-09-13T11:24:30.332 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:40ca46e8:::benchmark_data_smithi177_31116_object71693:head# 2024-09-13T11:24:30.332 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:40caa5fb:::benchmark_data_smithi177_31116_object91362:head# 2024-09-13T11:24:30.333 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:40cb5e0b:::benchmark_data_smithi177_31116_object70603:head# 2024-09-13T11:24:30.333 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:40cbf191:::benchmark_data_smithi177_31116_object2942:head# 2024-09-13T11:24:30.334 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:40cc41fd:::benchmark_data_smithi177_31116_object54498:head# 2024-09-13T11:24:30.334 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:40ccf75f:::benchmark_data_smithi177_31116_object14136:head# 2024-09-13T11:24:30.335 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:40cd46ff:::benchmark_data_smithi177_31116_object45399:head# 2024-09-13T11:24:30.336 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:40cd9401:::benchmark_data_smithi177_31116_object25170:head# 2024-09-13T11:24:30.336 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:40ce5330:::benchmark_data_smithi177_31116_object38763:head# 2024-09-13T11:24:30.337 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:40ce8113:::benchmark_data_smithi177_31116_object89456:head# 2024-09-13T11:24:30.337 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:40cef723:::benchmark_data_smithi177_31116_object72989:head# 2024-09-13T11:24:30.338 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:40cfd2be:::benchmark_data_smithi177_31116_object66962:head# 2024-09-13T11:24:30.339 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:40d157b2:::benchmark_data_smithi177_31116_object23765:head# 2024-09-13T11:24:30.339 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:40d19e35:::benchmark_data_smithi177_31116_object24612:head# 2024-09-13T11:24:30.340 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:40d21263:::benchmark_data_smithi177_31116_object76792:head# 2024-09-13T11:24:30.340 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:40d2355c:::benchmark_data_smithi177_31116_object82745:head# 2024-09-13T11:24:30.341 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:40d269c9:::benchmark_data_smithi177_31116_object74104:head# 2024-09-13T11:24:30.342 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:40d37e45:::benchmark_data_smithi177_31116_object9502:head# 2024-09-13T11:24:30.342 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:40d4196a:::benchmark_data_smithi177_31116_object45744:head# 2024-09-13T11:24:30.343 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:40d48d62:::benchmark_data_smithi177_31116_object856:head# 2024-09-13T11:24:30.344 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:40d49f6a:::benchmark_data_smithi177_31116_object31712:head# 2024-09-13T11:24:30.344 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:40d58aaa:::benchmark_data_smithi177_31116_object72376:head# 2024-09-13T11:24:30.344 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:40d5bf42:::benchmark_data_smithi177_31116_object108381:head# 2024-09-13T11:24:30.345 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:40d61d2c:::benchmark_data_smithi177_31116_object75020:head# 2024-09-13T11:24:30.345 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:40d6eb57:::benchmark_data_smithi177_31116_object108435:head# 2024-09-13T11:24:30.345 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:40d73a9e:::benchmark_data_smithi177_31116_object25011:head# 2024-09-13T11:24:30.346 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:40d77456:::benchmark_data_smithi177_31116_object21059:head# 2024-09-13T11:24:30.346 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:40d979fc:::benchmark_data_smithi177_31116_object94588:head# 2024-09-13T11:24:30.347 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:40da7c14:::benchmark_data_smithi177_31116_object45337:head# 2024-09-13T11:24:30.347 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:40dba6b8:::benchmark_data_smithi177_31116_object5036:head# 2024-09-13T11:24:30.348 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:40dc2b82:::benchmark_data_smithi177_31116_object50636:head# 2024-09-13T11:24:30.349 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:40dd559a:::benchmark_data_smithi177_31116_object55741:head# 2024-09-13T11:24:30.349 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:40dd634d:::benchmark_data_smithi177_31116_object22990:head# 2024-09-13T11:24:30.349 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:40dfaaf1:::benchmark_data_smithi177_31116_object97137:head# 2024-09-13T11:24:30.350 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:40dfb7d2:::benchmark_data_smithi177_31116_object70789:head# 2024-09-13T11:24:30.350 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:40e147d1:::benchmark_data_smithi177_31116_object48619:head# 2024-09-13T11:24:30.351 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:40e26ce1:::benchmark_data_smithi177_31116_object88089:head# 2024-09-13T11:24:30.351 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:40e2990b:::benchmark_data_smithi177_31116_object38783:head# 2024-09-13T11:24:30.352 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:40e54bc3:::benchmark_data_smithi177_31116_object42286:head# 2024-09-13T11:24:30.352 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:40e5a247:::benchmark_data_smithi177_31116_object10549:head# 2024-09-13T11:24:30.355 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:40e653f5:::benchmark_data_smithi177_31116_object14636:head# 2024-09-13T11:24:30.355 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:40e6723b:::benchmark_data_smithi177_31116_object66668:head# 2024-09-13T11:24:30.355 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:40e699f3:::benchmark_data_smithi177_31116_object95980:head# 2024-09-13T11:24:30.358 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:40e6f2bb:::benchmark_data_smithi177_31116_object13093:head# 2024-09-13T11:24:30.358 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:40e704b9:::benchmark_data_smithi177_31116_object44649:head# 2024-09-13T11:24:30.359 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:40e8807a:::benchmark_data_smithi177_31116_object12422:head# 2024-09-13T11:24:30.359 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:40e8a748:::benchmark_data_smithi177_31116_object15914:head# 2024-09-13T11:24:30.360 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:40e90991:::benchmark_data_smithi177_31116_object77215:head# 2024-09-13T11:24:30.360 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:40e9c2e9:::benchmark_data_smithi177_31116_object17667:head# 2024-09-13T11:24:30.361 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:40ea775c:::benchmark_data_smithi177_31116_object91454:head# 2024-09-13T11:24:30.362 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:40eafb7a:::benchmark_data_smithi177_31116_object96536:head# 2024-09-13T11:24:30.362 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:40ebb634:::benchmark_data_smithi177_31116_object33461:head# 2024-09-13T11:24:30.364 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:40ecb240:::benchmark_data_smithi177_31116_object81158:head# 2024-09-13T11:24:30.365 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:40ee042d:::benchmark_data_smithi177_31116_object95483:head# 2024-09-13T11:24:30.365 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:40eee266:::benchmark_data_smithi177_31116_object32683:head# 2024-09-13T11:24:30.366 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:40ef362a:::benchmark_data_smithi177_31116_object102330:head# 2024-09-13T11:24:30.367 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:40efa513:::benchmark_data_smithi177_31116_object96544:head# 2024-09-13T11:24:30.367 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:40f0daf8:::benchmark_data_smithi177_31116_object11622:head# 2024-09-13T11:24:30.368 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:40f16f5d:::benchmark_data_smithi177_31116_object91957:head# 2024-09-13T11:24:30.368 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:40f1c065:::benchmark_data_smithi177_31116_object11686:head# 2024-09-13T11:24:30.369 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:40f22173:::benchmark_data_smithi177_31116_object98167:head# 2024-09-13T11:24:30.369 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:40f285b0:::benchmark_data_smithi177_31116_object3333:head# 2024-09-13T11:24:30.370 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:40f2b542:::benchmark_data_smithi177_31116_object12818:head# 2024-09-13T11:24:30.370 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:40f3d0e3:::benchmark_data_smithi177_31116_object87326:head# 2024-09-13T11:24:30.371 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:40f53f15:::benchmark_data_smithi177_31116_object4244:head# 2024-09-13T11:24:30.371 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:40f5d2f7:::benchmark_data_smithi177_31116_object95634:head# 2024-09-13T11:24:30.372 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:40f5ed8c:::benchmark_data_smithi177_31116_object25946:head# 2024-09-13T11:24:30.372 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:40f62adf:::benchmark_data_smithi177_31116_object49555:head# 2024-09-13T11:24:30.373 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:40f690cd:::benchmark_data_smithi177_31116_object68466:head# 2024-09-13T11:24:30.375 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:40f69ea7:::benchmark_data_smithi177_31116_object100846:head# 2024-09-13T11:24:30.375 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:40f7a9f9:::benchmark_data_smithi177_31116_object64594:head# 2024-09-13T11:24:30.376 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:40f8606f:::benchmark_data_smithi177_31116_object87208:head# 2024-09-13T11:24:30.378 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:40f862e8:::benchmark_data_smithi177_31116_object104754:head# 2024-09-13T11:24:30.378 INFO:tasks.ceph.osd.8.smithi001.stderr:2024-09-13T11:24:30.375+0000 7fb9b48ae640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26312) UID: 0 2024-09-13T11:24:30.379 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:40f8f0ed:::benchmark_data_smithi177_31116_object54799:head# 2024-09-13T11:24:30.381 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:40f9b5eb:::benchmark_data_smithi177_31116_object14007:head# 2024-09-13T11:24:30.383 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:40faef97:::benchmark_data_smithi177_31116_object37970:head# 2024-09-13T11:24:30.385 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:40fb2e6b:::benchmark_data_smithi177_31116_object37727:head# 2024-09-13T11:24:30.385 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:40fb592a:::benchmark_data_smithi177_31116_object10094:head# 2024-09-13T11:24:30.386 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:40fb9ffb:::benchmark_data_smithi177_31116_object63547:head# 2024-09-13T11:24:30.387 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:40fef886:::benchmark_data_smithi177_31116_object40689:head# 2024-09-13T11:24:30.391 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:40ff4348:::benchmark_data_smithi177_31116_object104525:head# 2024-09-13T11:24:30.392 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4100804a:::benchmark_data_smithi177_31116_object8048:head# 2024-09-13T11:24:30.394 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:410222fb:::benchmark_data_smithi177_31116_object107822:head# 2024-09-13T11:24:30.394 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:410358f6:::benchmark_data_smithi177_31116_object18331:head# 2024-09-13T11:24:30.394 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:41036064:::benchmark_data_smithi177_31116_object80099:head# 2024-09-13T11:24:30.394 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4103d6a4:::benchmark_data_smithi177_31116_object88665:head# 2024-09-13T11:24:30.395 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4104510c:::benchmark_data_smithi177_31116_object23248:head# 2024-09-13T11:24:30.395 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:41048580:::benchmark_data_smithi177_31116_object100079:head# 2024-09-13T11:24:30.396 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4104e10a:::benchmark_data_smithi177_31116_object9721:head# 2024-09-13T11:24:30.396 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:410529b4:::benchmark_data_smithi177_31116_object56367:head# 2024-09-13T11:24:30.397 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:41055445:::benchmark_data_smithi177_31116_object3427:head# 2024-09-13T11:24:30.397 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:41059ab4:::benchmark_data_smithi177_31116_object96121:head# 2024-09-13T11:24:30.398 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4106c776:::benchmark_data_smithi177_31116_object15470:head# 2024-09-13T11:24:30.398 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:41073607:::benchmark_data_smithi177_31116_object108085:head# 2024-09-13T11:24:30.399 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:41077b96:::benchmark_data_smithi177_31116_object98414:head# 2024-09-13T11:24:30.399 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:410794c8:::benchmark_data_smithi177_31116_object91057:head# 2024-09-13T11:24:30.402 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:41079a1a:::benchmark_data_smithi177_31116_object6725:head# 2024-09-13T11:24:30.403 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4107a714:::benchmark_data_smithi177_31116_object76356:head# 2024-09-13T11:24:30.403 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4107e60f:::benchmark_data_smithi177_31116_object84706:head# 2024-09-13T11:24:30.404 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:41084616:::benchmark_data_smithi177_31116_object98478:head# 2024-09-13T11:24:30.405 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:41091bfd:::benchmark_data_smithi177_31116_object81942:head# 2024-09-13T11:24:30.405 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4109e623:::benchmark_data_smithi177_31116_object67809:head# 2024-09-13T11:24:30.408 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:410c0404:::benchmark_data_smithi177_31116_object81561:head# 2024-09-13T11:24:30.410 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:410c8219:::benchmark_data_smithi177_31116_object7250:head# 2024-09-13T11:24:30.411 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:410c992d:::benchmark_data_smithi177_31116_object39714:head# 2024-09-13T11:24:30.413 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:410ce493:::benchmark_data_smithi177_31116_object50593:head# 2024-09-13T11:24:30.413 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:410cfa7c:::benchmark_data_smithi177_31116_object56215:head# 2024-09-13T11:24:30.414 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:410eb040:::benchmark_data_smithi177_31116_object6210:head# 2024-09-13T11:24:30.414 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:410fbfc1:::benchmark_data_smithi177_31116_object90082:head# 2024-09-13T11:24:30.415 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:410fd7c4:::benchmark_data_smithi177_31116_object21257:head# 2024-09-13T11:24:30.416 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:410feb1b:::benchmark_data_smithi177_31116_object45780:head# 2024-09-13T11:24:30.416 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4112b23c:::benchmark_data_smithi177_31116_object47976:head# 2024-09-13T11:24:30.417 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4112f69b:::benchmark_data_smithi177_31116_object25097:head# 2024-09-13T11:24:30.418 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:411491ca:::benchmark_data_smithi177_31116_object90775:head# 2024-09-13T11:24:30.418 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4114bb51:::benchmark_data_smithi177_31116_object86788:head# 2024-09-13T11:24:30.419 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4114f495:::benchmark_data_smithi177_31116_object65389:head# 2024-09-13T11:24:30.420 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:41160910:::benchmark_data_smithi177_31116_object47555:head# 2024-09-13T11:24:30.421 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:411787b0:::benchmark_data_smithi177_31116_object49768:head# 2024-09-13T11:24:30.421 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4117fccb:::benchmark_data_smithi177_31116_object56436:head# 2024-09-13T11:24:30.424 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:411868a2:::benchmark_data_smithi177_31116_object85110:head# 2024-09-13T11:24:30.425 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4119065f:::benchmark_data_smithi177_31116_object83837:head# 2024-09-13T11:24:30.426 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:41196f99:::benchmark_data_smithi177_31116_object95219:head# 2024-09-13T11:24:30.426 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4119fc4a:::benchmark_data_smithi177_31116_object40489:head# 2024-09-13T11:24:30.427 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:411afd7e:::benchmark_data_smithi177_31116_object28902:head# 2024-09-13T11:24:30.427 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:411bf70f:::benchmark_data_smithi177_31116_object104893:head# 2024-09-13T11:24:30.428 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:411bffee:::benchmark_data_smithi177_31116_object19199:head# 2024-09-13T11:24:30.429 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:411cfec6:::benchmark_data_smithi177_31116_object68174:head# 2024-09-13T11:24:30.429 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:411d78ef:::benchmark_data_smithi177_31116_object89997:head# 2024-09-13T11:24:30.430 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:411f322b:::benchmark_data_smithi177_31116_object25379:head# 2024-09-13T11:24:30.430 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:411f43d3:::benchmark_data_smithi177_31116_object4704:head# 2024-09-13T11:24:30.431 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:411fffc1:::benchmark_data_smithi177_31116_object44251:head# 2024-09-13T11:24:30.432 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4120dfc1:::benchmark_data_smithi177_31116_object46888:head# 2024-09-13T11:24:30.433 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4122457c:::benchmark_data_smithi177_31116_object12752:head# 2024-09-13T11:24:30.435 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4122de23:::benchmark_data_smithi177_31116_object76580:head# 2024-09-13T11:24:30.435 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:412357db:::benchmark_data_smithi177_31116_object102601:head# 2024-09-13T11:24:30.435 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4124b764:::benchmark_data_smithi177_31116_object109016:head# 2024-09-13T11:24:30.435 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4124d95c:::benchmark_data_smithi177_31116_object82864:head# 2024-09-13T11:24:30.436 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:41258fc9:::benchmark_data_smithi177_31116_object35297:head# 2024-09-13T11:24:30.436 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:41265e08:::benchmark_data_smithi177_31116_object55080:head# 2024-09-13T11:24:30.436 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4126dba6:::benchmark_data_smithi177_31116_object36800:head# 2024-09-13T11:24:30.438 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:412700ec:::benchmark_data_smithi177_31116_object74055:head# 2024-09-13T11:24:30.438 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:41276e8a:::benchmark_data_smithi177_31116_object10441:head# 2024-09-13T11:24:30.438 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4127ce2e:::benchmark_data_smithi177_31116_object65180:head# 2024-09-13T11:24:30.441 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4127e487:::benchmark_data_smithi177_31116_object49637:head# 2024-09-13T11:24:30.441 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4128e83d:::benchmark_data_smithi177_31116_object80479:head# 2024-09-13T11:24:30.441 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4128ed0b:::benchmark_data_smithi177_31116_object19765:head# 2024-09-13T11:24:30.441 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:412a1b4f:::benchmark_data_smithi177_31116_object103536:head# 2024-09-13T11:24:30.441 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:412a5ce2:::benchmark_data_smithi177_31116_object3342:head# 2024-09-13T11:24:30.442 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:412ad804:::benchmark_data_smithi177_31116_object67806:head# 2024-09-13T11:24:30.442 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:412aec0e:::benchmark_data_smithi177_31116_object100734:head# 2024-09-13T11:24:30.442 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:412b8a32:::benchmark_data_smithi177_31116_object40438:head# 2024-09-13T11:24:30.443 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:412c6a17:::benchmark_data_smithi177_31116_object8116:head# 2024-09-13T11:24:30.443 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:412c9765:::benchmark_data_smithi177_31116_object70640:head# 2024-09-13T11:24:30.444 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:412cadb5:::benchmark_data_smithi177_31116_object87878:head# 2024-09-13T11:24:30.444 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:412d2f75:::benchmark_data_smithi177_31116_object10439:head# 2024-09-13T11:24:30.445 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:412e1cc0:::benchmark_data_smithi177_31116_object44853:head# 2024-09-13T11:24:30.445 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:412ecf1a:::benchmark_data_smithi177_31116_object75062:head# 2024-09-13T11:24:30.446 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:412f0cf2:::benchmark_data_smithi177_31116_object74585:head# 2024-09-13T11:24:30.446 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:412f5431:::benchmark_data_smithi177_31116_object83827:head# 2024-09-13T11:24:30.447 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:412f5d11:::benchmark_data_smithi177_31116_object54401:head# 2024-09-13T11:24:30.447 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:413007af:::benchmark_data_smithi177_31116_object55297:head# 2024-09-13T11:24:30.449 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:41303834:::benchmark_data_smithi177_31116_object51005:head# 2024-09-13T11:24:30.449 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4130db71:::benchmark_data_smithi177_31116_object8643:head# 2024-09-13T11:24:30.450 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4131080b:::benchmark_data_smithi177_31116_object39956:head# 2024-09-13T11:24:30.450 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:413132de:::benchmark_data_smithi177_31116_object76519:head# 2024-09-13T11:24:30.451 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:41330121:::benchmark_data_smithi177_31116_object67157:head# 2024-09-13T11:24:30.451 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4134123a:::benchmark_data_smithi177_31116_object58582:head# 2024-09-13T11:24:30.452 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:41348c17:::benchmark_data_smithi177_31116_object58806:head# 2024-09-13T11:24:30.452 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4134da32:::benchmark_data_smithi177_31116_object85299:head# 2024-09-13T11:24:30.453 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4135252d:::benchmark_data_smithi177_31116_object24081:head# 2024-09-13T11:24:30.453 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4135bb23:::benchmark_data_smithi177_31116_object86692:head# 2024-09-13T11:24:30.454 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:41362783:::benchmark_data_smithi177_31116_object78553:head# 2024-09-13T11:24:30.454 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4136a799:::benchmark_data_smithi177_31116_object58789:head# 2024-09-13T11:24:30.455 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4136c1b1:::benchmark_data_smithi177_31116_object2527:head# 2024-09-13T11:24:30.455 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4137b2aa:::benchmark_data_smithi177_31116_object104563:head# 2024-09-13T11:24:30.455 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4138068d:::benchmark_data_smithi177_31116_object11957:head# 2024-09-13T11:24:30.456 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:413a7595:::benchmark_data_smithi177_31116_object73543:head# 2024-09-13T11:24:30.456 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:413ab0ff:::benchmark_data_smithi177_31116_object5770:head# 2024-09-13T11:24:30.457 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:413ad901:::benchmark_data_smithi177_31116_object75574:head# 2024-09-13T11:24:30.457 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:413b2a07:::benchmark_data_smithi177_31116_object106110:head# 2024-09-13T11:24:30.458 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:413b758f:::benchmark_data_smithi177_31116_object32729:head# 2024-09-13T11:24:30.459 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:413c5ef0:::benchmark_data_smithi177_31116_object15967:head# 2024-09-13T11:24:30.459 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:413c7bcf:::benchmark_data_smithi177_31116_object68113:head# 2024-09-13T11:24:30.459 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:413cdb37:::benchmark_data_smithi177_31116_object28086:head# 2024-09-13T11:24:30.461 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:413ecece:::benchmark_data_smithi177_31116_object77374:head# 2024-09-13T11:24:30.461 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:413f3281:::benchmark_data_smithi177_31116_object5056:head# 2024-09-13T11:24:30.461 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4140257b:::benchmark_data_smithi177_31116_object1070:head# 2024-09-13T11:24:30.462 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:41404bdc:::benchmark_data_smithi177_31116_object21119:head# 2024-09-13T11:24:30.462 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4140caa6:::benchmark_data_smithi177_31116_object86276:head# 2024-09-13T11:24:30.463 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4140e20c:::benchmark_data_smithi177_31116_object64482:head# 2024-09-13T11:24:30.463 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:41416e58:::benchmark_data_smithi177_31116_object71613:head# 2024-09-13T11:24:30.464 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:41417e3f:::benchmark_data_smithi177_31116_object89520:head# 2024-09-13T11:24:30.464 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4142ab36:::benchmark_data_smithi177_31116_object53207:head# 2024-09-13T11:24:30.465 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4142c0f1:::benchmark_data_smithi177_31116_object48914:head# 2024-09-13T11:24:30.465 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:41433079:::benchmark_data_smithi177_31116_object47707:head# 2024-09-13T11:24:30.465 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4144618c:::benchmark_data_smithi177_31116_object36957:head# 2024-09-13T11:24:30.466 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:41470a68:::benchmark_data_smithi177_31116_object87964:head# 2024-09-13T11:24:30.466 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:414775a0:::benchmark_data_smithi177_31116_object16708:head# 2024-09-13T11:24:30.466 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:41484df3:::benchmark_data_smithi177_31116_object55062:head# 2024-09-13T11:24:30.467 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:414852e1:::benchmark_data_smithi177_31116_object26160:head# 2024-09-13T11:24:30.467 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:414888a4:::benchmark_data_smithi177_31116_object93685:head# 2024-09-13T11:24:30.468 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:414890b5:::benchmark_data_smithi177_31116_object67033:head# 2024-09-13T11:24:30.468 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:41489392:::benchmark_data_smithi177_31116_object61263:head# 2024-09-13T11:24:30.469 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:41491d90:::benchmark_data_smithi177_31116_object90456:head# 2024-09-13T11:24:30.470 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4149d3cd:::benchmark_data_smithi177_31116_object88510:head# 2024-09-13T11:24:30.471 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4149f436:::benchmark_data_smithi177_31116_object88343:head# 2024-09-13T11:24:30.471 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4149f5e8:::benchmark_data_smithi177_31116_object54078:head# 2024-09-13T11:24:30.472 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:414a58e8:::benchmark_data_smithi177_31116_object71603:head# 2024-09-13T11:24:30.473 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:414af3aa:::benchmark_data_smithi177_31116_object11251:head# 2024-09-13T11:24:30.473 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:414d34ea:::benchmark_data_smithi177_31116_object17470:head# 2024-09-13T11:24:30.473 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:414d5341:::benchmark_data_smithi177_31116_object23261:head# 2024-09-13T11:24:30.473 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:414dc6bc:::benchmark_data_smithi177_31116_object11521:head# 2024-09-13T11:24:30.474 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:414dd8c6:::benchmark_data_smithi177_31116_object108231:head# 2024-09-13T11:24:30.474 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:414f762a:::benchmark_data_smithi177_31116_object29220:head# 2024-09-13T11:24:30.475 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:41508cdb:::benchmark_data_smithi177_31116_object38957:head# 2024-09-13T11:24:30.475 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4150cc84:::benchmark_data_smithi177_31116_object38096:head# 2024-09-13T11:24:30.476 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:41514999:::benchmark_data_smithi177_31116_object68892:head# 2024-09-13T11:24:30.476 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:41515722:::benchmark_data_smithi177_31116_object57027:head# 2024-09-13T11:24:30.477 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:41529633:::benchmark_data_smithi177_31116_object107008:head# 2024-09-13T11:24:30.478 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4152d58b:::benchmark_data_smithi177_31116_object101557:head# 2024-09-13T11:24:30.478 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:41541cf3:::benchmark_data_smithi177_31116_object102371:head# 2024-09-13T11:24:30.480 INFO:tasks.ceph.osd.4.smithi001.stderr:2024-09-13T11:24:30.475+0000 7ff8e0218640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26310) UID: 0 2024-09-13T11:24:30.481 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4154edc7:::benchmark_data_smithi177_31116_object64608:head# 2024-09-13T11:24:30.481 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:41557964:::benchmark_data_smithi177_31116_object62549:head# 2024-09-13T11:24:30.481 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:41557f72:::benchmark_data_smithi177_31116_object68204:head# 2024-09-13T11:24:30.481 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:41560751:::benchmark_data_smithi177_31116_object9396:head# 2024-09-13T11:24:30.482 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4157c860:::benchmark_data_smithi177_31116_object52620:head# 2024-09-13T11:24:30.482 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:415872a1:::benchmark_data_smithi177_31116_object74:head# 2024-09-13T11:24:30.482 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:415960d1:::benchmark_data_smithi177_31116_object96382:head# 2024-09-13T11:24:30.483 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:41597be7:::benchmark_data_smithi177_31116_object77942:head# 2024-09-13T11:24:30.483 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:41598183:::benchmark_data_smithi177_31116_object60921:head# 2024-09-13T11:24:30.484 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:415ab4ca:::benchmark_data_smithi177_31116_object107274:head# 2024-09-13T11:24:30.484 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:415afb36:::benchmark_data_smithi177_31116_object42444:head# 2024-09-13T11:24:30.485 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:415c5d4e:::benchmark_data_smithi177_31116_object5289:head# 2024-09-13T11:24:30.485 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:415c74a2:::benchmark_data_smithi177_31116_object79285:head# 2024-09-13T11:24:30.486 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:415dcc1b:::benchmark_data_smithi177_31116_object24659:head# 2024-09-13T11:24:30.486 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:415e4ddb:::benchmark_data_smithi177_31116_object68305:head# 2024-09-13T11:24:30.487 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:415e9c36:::benchmark_data_smithi177_31116_object29303:head# 2024-09-13T11:24:30.488 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:415ea28d:::benchmark_data_smithi177_31116_object476:head# 2024-09-13T11:24:30.488 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:415f94f0:::benchmark_data_smithi177_31116_object83426:head# 2024-09-13T11:24:30.489 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:41602bcb:::benchmark_data_smithi177_31116_object50520:head# 2024-09-13T11:24:30.489 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:416030ca:::benchmark_data_smithi177_31116_object98686:head# 2024-09-13T11:24:30.490 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4160654c:::benchmark_data_smithi177_31116_object69796:head# 2024-09-13T11:24:30.491 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4160e852:::benchmark_data_smithi177_31116_object2102:head# 2024-09-13T11:24:30.491 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4161b2ed:::benchmark_data_smithi177_31116_object26369:head# 2024-09-13T11:24:30.492 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:416202c7:::benchmark_data_smithi177_31116_object18224:head# 2024-09-13T11:24:30.493 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4162fbaa:::benchmark_data_smithi177_31116_object87667:head# 2024-09-13T11:24:30.493 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:41639be9:::benchmark_data_smithi177_31116_object82774:head# 2024-09-13T11:24:30.494 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:41647159:::benchmark_data_smithi177_31116_object54135:head# 2024-09-13T11:24:30.494 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4164d7cd:::benchmark_data_smithi177_31116_object15882:head# 2024-09-13T11:24:30.495 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:41653c50:::benchmark_data_smithi177_31116_object12655:head# 2024-09-13T11:24:30.495 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:41654608:::benchmark_data_smithi177_31116_object100217:head# 2024-09-13T11:24:30.496 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:416588d9:::benchmark_data_smithi177_31116_object1394:head# 2024-09-13T11:24:30.496 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:41668529:::benchmark_data_smithi177_31116_object71463:head# 2024-09-13T11:24:30.497 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:416714ae:::benchmark_data_smithi177_31116_object85083:head# 2024-09-13T11:24:30.497 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4167294d:::benchmark_data_smithi177_31116_object2787:head# 2024-09-13T11:24:30.498 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:41674bfe:::benchmark_data_smithi177_31116_object1836:head# 2024-09-13T11:24:30.498 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:41678c5a:::benchmark_data_smithi177_31116_object2579:head# 2024-09-13T11:24:30.499 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:416921d8:::benchmark_data_smithi177_31116_object91315:head# 2024-09-13T11:24:30.499 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:416992ed:::benchmark_data_smithi177_31116_object14195:head# 2024-09-13T11:24:30.501 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:416aa42a:::benchmark_data_smithi177_31116_object1796:head# 2024-09-13T11:24:30.501 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:416baeb7:::benchmark_data_smithi177_31116_object98953:head# 2024-09-13T11:24:30.502 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:416bf5f4:::benchmark_data_smithi177_31116_object11088:head# 2024-09-13T11:24:30.502 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:416c07e1:::benchmark_data_smithi177_31116_object19781:head# 2024-09-13T11:24:30.503 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:416c89ff:::benchmark_data_smithi177_31116_object24452:head# 2024-09-13T11:24:30.503 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:416c9f8b:::benchmark_data_smithi177_31116_object67450:head# 2024-09-13T11:24:30.504 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:416caf29:::benchmark_data_smithi177_31116_object100488:head# 2024-09-13T11:24:30.505 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:416cfe5b:::benchmark_data_smithi177_31116_object28355:head# 2024-09-13T11:24:30.505 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:416dee48:::benchmark_data_smithi177_31116_object69146:head# 2024-09-13T11:24:30.506 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:416edb06:::benchmark_data_smithi177_31116_object90618:head# 2024-09-13T11:24:30.506 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:416f2e3c:::benchmark_data_smithi177_31116_object52283:head# 2024-09-13T11:24:30.507 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:416f6884:::benchmark_data_smithi177_31116_object85457:head# 2024-09-13T11:24:30.507 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:416fe4cb:::benchmark_data_smithi177_31116_object33933:head# 2024-09-13T11:24:30.508 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:41719e84:::benchmark_data_smithi177_31116_object72397:head# 2024-09-13T11:24:30.508 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:41728adc:::benchmark_data_smithi177_31116_object69180:head# 2024-09-13T11:24:30.509 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4172b340:::benchmark_data_smithi177_31116_object36568:head# 2024-09-13T11:24:30.512 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4172c7d1:::benchmark_data_smithi177_31116_object2597:head# 2024-09-13T11:24:30.512 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4172d173:::benchmark_data_smithi177_31116_object101733:head# 2024-09-13T11:24:30.513 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:41734f82:::benchmark_data_smithi177_31116_object70601:head# 2024-09-13T11:24:30.513 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:41739173:::benchmark_data_smithi177_31116_object90170:head# 2024-09-13T11:24:30.513 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4173ab0e:::benchmark_data_smithi177_31116_object5870:head# 2024-09-13T11:24:30.513 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4173ac02:::benchmark_data_smithi177_31116_object7994:head# 2024-09-13T11:24:30.514 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4173f81c:::benchmark_data_smithi177_31116_object2902:head# 2024-09-13T11:24:30.514 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:41746743:::benchmark_data_smithi177_31116_object40970:head# 2024-09-13T11:24:30.515 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4174a2ba:::benchmark_data_smithi177_31116_object84623:head# 2024-09-13T11:24:30.515 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:41752f5d:::benchmark_data_smithi177_31116_object25222:head# 2024-09-13T11:24:30.516 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:417575d1:::benchmark_data_smithi177_31116_object62797:head# 2024-09-13T11:24:30.516 INFO:teuthology.orchestra.run.smithi092.stdout:ERROR: (22) Invalid argument 2024-09-13T11:24:30.516 INFO:teuthology.orchestra.run.smithi092.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:24:30.517 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:41758453:::benchmark_data_smithi177_31116_object105707:head# 2024-09-13T11:24:30.517 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4175c8dc:::benchmark_data_smithi177_31116_object31250:head# 2024-09-13T11:24:30.517 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4175d761:::benchmark_data_smithi177_31116_object85973:head# 2024-09-13T11:24:30.518 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:417667a7:::benchmark_data_smithi177_31116_object82679:head# 2024-09-13T11:24:30.519 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4176870c:::benchmark_data_smithi177_31116_object65699:head# 2024-09-13T11:24:30.519 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:41768c3b:::benchmark_data_smithi177_31116_object3776:head# 2024-09-13T11:24:30.519 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:41784e8e:::benchmark_data_smithi177_31116_object26206:head# 2024-09-13T11:24:30.520 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:41787b82:::benchmark_data_smithi177_31116_object33492:head# 2024-09-13T11:24:30.520 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4179b627:::benchmark_data_smithi177_31116_object64126:head# 2024-09-13T11:24:30.521 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:417a9b02:::benchmark_data_smithi177_31116_object88440:head# 2024-09-13T11:24:30.522 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:417b41a0:::benchmark_data_smithi177_31116_object53536:head# 2024-09-13T11:24:30.522 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:417b6571:::benchmark_data_smithi177_31116_object74011:head# 2024-09-13T11:24:30.523 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:417b9e44:::benchmark_data_smithi177_31116_object13201:head# 2024-09-13T11:24:30.523 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:417bbe8c:::benchmark_data_smithi177_31116_object100865:head# 2024-09-13T11:24:30.524 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:417c9ac2:::benchmark_data_smithi177_31116_object53469:head# 2024-09-13T11:24:30.524 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:417cf96c:::benchmark_data_smithi177_31116_object97578:head# 2024-09-13T11:24:30.525 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:417d0664:::benchmark_data_smithi177_31116_object48466:head# 2024-09-13T11:24:30.525 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:417d9626:::benchmark_data_smithi177_31116_object9136:head# 2024-09-13T11:24:30.526 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:417e9147:::benchmark_data_smithi177_31116_object2103:head# 2024-09-13T11:24:30.526 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4180b6c2:::benchmark_data_smithi177_31116_object14871:head# 2024-09-13T11:24:30.527 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:41811b59:::benchmark_data_smithi177_31116_object60573:head# 2024-09-13T11:24:30.528 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4182860e:::benchmark_data_smithi177_31116_object88573:head# 2024-09-13T11:24:30.529 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:418293bc:::benchmark_data_smithi177_31116_object19475:head# 2024-09-13T11:24:30.531 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:41831322:::benchmark_data_smithi177_31116_object102985:head# 2024-09-13T11:24:30.532 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:418335c8:::benchmark_data_smithi177_31116_object120:head# 2024-09-13T11:24:30.533 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:418479f1:::benchmark_data_smithi177_31116_object12972:head# 2024-09-13T11:24:30.534 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:41849bdb:::benchmark_data_smithi177_31116_object82034:head# 2024-09-13T11:24:30.534 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4184aad8:::benchmark_data_smithi177_31116_object99990:head# 2024-09-13T11:24:30.536 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:41851012:::benchmark_data_smithi177_31116_object60548:head# 2024-09-13T11:24:30.537 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:41856b78:::benchmark_data_smithi177_31116_object43013:head# 2024-09-13T11:24:30.537 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4186867e:::benchmark_data_smithi177_31116_object107405:head# 2024-09-13T11:24:30.538 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:41879897:::benchmark_data_smithi177_31116_object37515:head# 2024-09-13T11:24:30.541 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4187e426:::benchmark_data_smithi177_31116_object85434:head# 2024-09-13T11:24:30.541 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4187f4c4:::benchmark_data_smithi177_31116_object25562:head# 2024-09-13T11:24:30.542 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4187fa39:::benchmark_data_smithi177_31116_object31895:head# 2024-09-13T11:24:30.543 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:41889950:::benchmark_data_smithi177_31116_object72936:head# 2024-09-13T11:24:30.543 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:418957fb:::benchmark_data_smithi177_31116_object99590:head# 2024-09-13T11:24:30.544 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4189bcfa:::benchmark_data_smithi177_31116_object22433:head# 2024-09-13T11:24:30.545 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-13T11:24:30.546 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4189d605:::benchmark_data_smithi177_31116_object45084:head# 2024-09-13T11:24:30.546 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4189d656:::benchmark_data_smithi177_31116_object23549:head# 2024-09-13T11:24:30.546 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:418ab56f:::benchmark_data_smithi177_31116_object9569:head# 2024-09-13T11:24:30.548 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:418d0cbd:::benchmark_data_smithi177_31116_object6573:head# 2024-09-13T11:24:30.550 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:418d351d:::benchmark_data_smithi177_31116_object41793:head# 2024-09-13T11:24:30.550 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:418de556:::benchmark_data_smithi177_31116_object2352:head# 2024-09-13T11:24:30.551 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:418e4412:::benchmark_data_smithi177_31116_object888:head# 2024-09-13T11:24:30.551 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:418e4d2e:::benchmark_data_smithi177_31116_object63692:head# 2024-09-13T11:24:30.552 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:418eec3c:::benchmark_data_smithi177_31116_object67521:head# 2024-09-13T11:24:30.553 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:418f2216:::benchmark_data_smithi177_31116_object24865:head# 2024-09-13T11:24:30.553 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:418f7068:::benchmark_data_smithi177_31116_object50491:head# 2024-09-13T11:24:30.554 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:418f80be:::benchmark_data_smithi177_31116_object20192:head# 2024-09-13T11:24:30.554 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:41911e9e:::benchmark_data_smithi177_31116_object64294:head# 2024-09-13T11:24:30.555 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:419176c4:::benchmark_data_smithi177_31116_object61499:head# 2024-09-13T11:24:30.555 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:41917fec:::benchmark_data_smithi177_31116_object75354:head# 2024-09-13T11:24:30.558 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:41919f48:::benchmark_data_smithi177_31116_object94480:head# 2024-09-13T11:24:30.561 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4192bf59:::benchmark_data_smithi177_31116_object23168:head# 2024-09-13T11:24:30.561 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4192c403:::benchmark_data_smithi177_31116_object85666:head# 2024-09-13T11:24:30.566 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4193a4c5:::benchmark_data_smithi177_31116_object23463:head# 2024-09-13T11:24:30.566 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4193d075:::benchmark_data_smithi177_31116_object81276:head# 2024-09-13T11:24:30.567 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4194210a:::benchmark_data_smithi177_31116_object73668:head# 2024-09-13T11:24:30.568 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:41952947:::benchmark_data_smithi177_31116_object27795:head# 2024-09-13T11:24:30.568 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4197da66:::benchmark_data_smithi177_31116_object57043:head# 2024-09-13T11:24:30.569 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4197f066:::benchmark_data_smithi177_31116_object62724:head# 2024-09-13T11:24:30.569 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:419a3d86:::benchmark_data_smithi177_31116_object108355:head# 2024-09-13T11:24:30.570 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:419aa723:::benchmark_data_smithi177_31116_object96343:head# 2024-09-13T11:24:30.570 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:419ab438:::benchmark_data_smithi177_31116_object102157:head# 2024-09-13T11:24:30.571 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:419bb974:::benchmark_data_smithi177_31116_object8612:head# 2024-09-13T11:24:30.572 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:419c4758:::benchmark_data_smithi177_31116_object46233:head# 2024-09-13T11:24:30.574 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:419c8a4e:::benchmark_data_smithi177_31116_object46852:head# 2024-09-13T11:24:30.574 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:419cf409:::benchmark_data_smithi177_31116_object4247:head# 2024-09-13T11:24:30.575 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:419d07ca:::benchmark_data_smithi177_31116_object21317:head# 2024-09-13T11:24:30.576 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:419d7767:::benchmark_data_smithi177_31116_object63999:head# 2024-09-13T11:24:30.577 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:419dd9e6:::benchmark_data_smithi177_31116_object73595:head# 2024-09-13T11:24:30.578 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:419ecb07:::benchmark_data_smithi177_31116_object105667:head# 2024-09-13T11:24:30.579 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:419edbbf:::benchmark_data_smithi177_31116_object59391:head# 2024-09-13T11:24:30.579 INFO:tasks.ceph.osd.9.smithi092.stderr:2024-09-13T11:24:30.574+0000 7ff5ab3d0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26348) UID: 0 2024-09-13T11:24:30.579 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:419f34d0:::benchmark_data_smithi177_31116_object27660:head# 2024-09-13T11:24:30.580 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:41a079bc:::benchmark_data_smithi177_31116_object25521:head# 2024-09-13T11:24:30.581 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:41a14712:::benchmark_data_smithi177_31116_object96132:head# 2024-09-13T11:24:30.581 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:41a172db:::benchmark_data_smithi177_31116_object95135:head# 2024-09-13T11:24:30.582 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:41a18620:::benchmark_data_smithi177_31116_object67045:head# 2024-09-13T11:24:30.583 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:41a1cfeb:::benchmark_data_smithi177_31116_object90669:head# 2024-09-13T11:24:30.584 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:41a2b4e3:::benchmark_data_smithi177_31116_object52017:head# 2024-09-13T11:24:30.584 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:41a2e77e:::benchmark_data_smithi177_31116_object11498:head# 2024-09-13T11:24:30.584 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:41a47e95:::benchmark_data_smithi177_31116_object75419:head# 2024-09-13T11:24:30.585 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:41a4e172:::benchmark_data_smithi177_31116_object91902:head# 2024-09-13T11:24:30.586 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:41a57b9a:::benchmark_data_smithi177_31116_object22810:head# 2024-09-13T11:24:30.587 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:41a6b602:::benchmark_data_smithi177_31116_object55914:head# 2024-09-13T11:24:30.588 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:41a71140:::benchmark_data_smithi177_31116_object11412:head# 2024-09-13T11:24:30.591 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:41a72222:::benchmark_data_smithi177_31116_object54454:head# 2024-09-13T11:24:30.594 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:41a7d592:::benchmark_data_smithi177_31116_object110136:head# 2024-09-13T11:24:30.594 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:41a8c50d:::benchmark_data_smithi177_31116_object67995:head# 2024-09-13T11:24:30.595 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:41aa2705:::benchmark_data_smithi177_31116_object42760:head# 2024-09-13T11:24:30.596 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:41aa614b:::benchmark_data_smithi177_31116_object74606:head# 2024-09-13T11:24:30.596 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:41aa9e64:::benchmark_data_smithi177_31116_object55954:head# 2024-09-13T11:24:30.597 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:41aae21f:::benchmark_data_smithi177_31116_object32713:head# 2024-09-13T11:24:30.597 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:41ab1153:::benchmark_data_smithi177_31116_object94100:head# 2024-09-13T11:24:30.598 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:41ab5856:::benchmark_data_smithi177_31116_object56608:head# 2024-09-13T11:24:30.598 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:41ac8c11:::benchmark_data_smithi177_31116_object105170:head# 2024-09-13T11:24:30.599 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:41acfb7f:::benchmark_data_smithi177_31116_object76139:head# 2024-09-13T11:24:30.599 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:41ad7c50:::benchmark_data_smithi177_31116_object55802:head# 2024-09-13T11:24:30.600 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:41adaf7a:::benchmark_data_smithi177_31116_object56630:head# 2024-09-13T11:24:30.600 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:41adcae9:::benchmark_data_smithi177_31116_object75584:head# 2024-09-13T11:24:30.601 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:41aebb73:::benchmark_data_smithi177_31116_object39961:head# 2024-09-13T11:24:30.602 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:41af844c:::benchmark_data_smithi177_31116_object35687:head# 2024-09-13T11:24:30.603 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:41b07870:::benchmark_data_smithi177_31116_object69595:head# 2024-09-13T11:24:30.603 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:41b10e35:::benchmark_data_smithi177_31116_object74199:head# 2024-09-13T11:24:30.604 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:41b19bed:::benchmark_data_smithi177_31116_object74465:head# 2024-09-13T11:24:30.604 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:41b1c2dd:::benchmark_data_smithi177_31116_object16126:head# 2024-09-13T11:24:30.605 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:41b1c8cc:::benchmark_data_smithi177_31116_object12959:head# 2024-09-13T11:24:30.606 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:41b2d6b1:::benchmark_data_smithi177_31116_object90563:head# 2024-09-13T11:24:30.606 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:41b3651e:::benchmark_data_smithi177_31116_object8112:head# 2024-09-13T11:24:30.607 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:41b3e1b7:::benchmark_data_smithi177_31116_object9731:head# 2024-09-13T11:24:30.607 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:41b44938:::benchmark_data_smithi177_31116_object101894:head# 2024-09-13T11:24:30.608 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:41b45df1:::benchmark_data_smithi177_31116_object7282:head# 2024-09-13T11:24:30.608 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:41b4a870:::benchmark_data_smithi177_31116_object94831:head# 2024-09-13T11:24:30.609 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:41b4ae2d:::benchmark_data_smithi177_31116_object9176:head# 2024-09-13T11:24:30.610 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:41b4bc7a:::benchmark_data_smithi177_31116_object584:head# 2024-09-13T11:24:30.610 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:41b4e7de:::benchmark_data_smithi177_31116_object19431:head# 2024-09-13T11:24:30.611 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:41b53687:::benchmark_data_smithi177_31116_object47763:head# 2024-09-13T11:24:30.612 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:41b5acfe:::benchmark_data_smithi177_31116_object15491:head# 2024-09-13T11:24:30.613 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:41b67a1b:::benchmark_data_smithi177_31116_object25693:head# 2024-09-13T11:24:30.613 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:41b81d75:::benchmark_data_smithi177_31116_object77248:head# 2024-09-13T11:24:30.614 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:41ba392b:::benchmark_data_smithi177_31116_object60509:head# 2024-09-13T11:24:30.615 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:41ba5175:::benchmark_data_smithi177_31116_object73691:head# 2024-09-13T11:24:30.615 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:41ba7f29:::benchmark_data_smithi177_31116_object44762:head# 2024-09-13T11:24:30.616 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:41baaa35:::benchmark_data_smithi177_31116_object84636:head# 2024-09-13T11:24:30.617 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:41baba54:::benchmark_data_smithi177_31116_object91469:head# 2024-09-13T11:24:30.617 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:41bae3f3:::benchmark_data_smithi177_31116_object38471:head# 2024-09-13T11:24:30.618 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:41bb8946:::benchmark_data_smithi177_31116_object55853:head# 2024-09-13T11:24:30.618 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:41bdbcb6:::benchmark_data_smithi177_31116_object70256:head# 2024-09-13T11:24:30.620 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:41be57e2:::benchmark_data_smithi177_31116_object79887:head# 2024-09-13T11:24:30.620 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:41be5b07:::benchmark_data_smithi177_31116_object73153:head# 2024-09-13T11:24:30.620 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:41bfb970:::benchmark_data_smithi177_31116_object43958:head# 2024-09-13T11:24:30.621 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:41bff846:::benchmark_data_smithi177_31116_object11680:head# 2024-09-13T11:24:30.621 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:41c2b5e5:::benchmark_data_smithi177_31116_object82453:head# 2024-09-13T11:24:30.622 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:41c3cec1:::benchmark_data_smithi177_31116_object20659:head# 2024-09-13T11:24:30.623 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:41c5f89c:::benchmark_data_smithi177_31116_object93925:head# 2024-09-13T11:24:30.623 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:41c64cdb:::benchmark_data_smithi177_31116_object19514:head# 2024-09-13T11:24:30.624 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:41c75c67:::benchmark_data_smithi177_31116_object45384:head# 2024-09-13T11:24:30.625 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:41c7b23a:::benchmark_data_smithi177_31116_object83691:head# 2024-09-13T11:24:30.625 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:41c86d98:::benchmark_data_smithi177_31116_object57667:head# 2024-09-13T11:24:30.626 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:41ca3290:::benchmark_data_smithi177_31116_object28436:head# 2024-09-13T11:24:30.626 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:41ca6820:::benchmark_data_smithi177_31116_object109302:head# 2024-09-13T11:24:30.627 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:41cb13de:::benchmark_data_smithi177_31116_object77182:head# 2024-09-13T11:24:30.627 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:41cc07d3:::benchmark_data_smithi177_31116_object86792:head# 2024-09-13T11:24:30.628 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:41cd1957:::benchmark_data_smithi177_31116_object81223:head# 2024-09-13T11:24:30.628 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:41cea0a2:::benchmark_data_smithi177_31116_object100699:head# 2024-09-13T11:24:30.629 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:41cf2e22:::benchmark_data_smithi177_31116_object15842:head# 2024-09-13T11:24:30.629 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:41cf6c5e:::benchmark_data_smithi177_31116_object77485:head# 2024-09-13T11:24:30.630 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:41cf757c:::benchmark_data_smithi177_31116_object29257:head# 2024-09-13T11:24:30.630 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:41cf784e:::benchmark_data_smithi177_31116_object99342:head# 2024-09-13T11:24:30.631 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:41cfb6ea:::benchmark_data_smithi177_31116_object42524:head# 2024-09-13T11:24:30.631 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:41d0710b:::benchmark_data_smithi177_31116_object3255:head# 2024-09-13T11:24:30.632 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:41d1364f:::benchmark_data_smithi177_31116_object77256:head# 2024-09-13T11:24:30.632 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:41d147be:::benchmark_data_smithi177_31116_object13860:head# 2024-09-13T11:24:30.633 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:41d2ea11:::benchmark_data_smithi177_31116_object26983:head# 2024-09-13T11:24:30.633 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:41d31c0a:::benchmark_data_smithi177_31116_object40565:head# 2024-09-13T11:24:30.634 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:41d31f48:::benchmark_data_smithi177_31116_object22366:head# 2024-09-13T11:24:30.634 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:41d3542c:::benchmark_data_smithi177_31116_object78076:head# 2024-09-13T11:24:30.635 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:41d48374:::benchmark_data_smithi177_31116_object24436:head# 2024-09-13T11:24:30.636 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:41d5dffd:::benchmark_data_smithi177_31116_object60120:head# 2024-09-13T11:24:30.636 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:41d62272:::benchmark_data_smithi177_31116_object51012:head# 2024-09-13T11:24:30.637 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:41d65c2c:::benchmark_data_smithi177_31116_object78597:head# 2024-09-13T11:24:30.637 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:41d78a04:::benchmark_data_smithi177_31116_object16973:head# 2024-09-13T11:24:30.638 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:41d8b0a8:::benchmark_data_smithi177_31116_object104572:head# 2024-09-13T11:24:30.638 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:41d8bdc3:::benchmark_data_smithi177_31116_object633:head# 2024-09-13T11:24:30.639 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:41d8dce2:::benchmark_data_smithi177_31116_object73642:head# 2024-09-13T11:24:30.640 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:41d9a918:::benchmark_data_smithi177_31116_object66060:head# 2024-09-13T11:24:30.641 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:41d9b927:::benchmark_data_smithi177_31116_object74127:head# 2024-09-13T11:24:30.641 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:41dac7a5:::benchmark_data_smithi177_31116_object11855:head# 2024-09-13T11:24:30.642 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:41dc061c:::benchmark_data_smithi177_31116_object9685:head# 2024-09-13T11:24:30.642 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:41dc79e6:::benchmark_data_smithi177_31116_object62411:head# 2024-09-13T11:24:30.643 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:41dcac2d:::benchmark_data_smithi177_31116_object85745:head# 2024-09-13T11:24:30.644 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:41dd7da9:::benchmark_data_smithi177_31116_object96854:head# 2024-09-13T11:24:30.645 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:41ddbd59:::benchmark_data_smithi177_31116_object84138:head# 2024-09-13T11:24:30.647 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:41ddd2f2:::benchmark_data_smithi177_31116_object937:head# 2024-09-13T11:24:30.647 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:41de5b5b:::benchmark_data_smithi177_31116_object74748:head# 2024-09-13T11:24:30.648 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:41de8a53:::benchmark_data_smithi177_31116_object101264:head# 2024-09-13T11:24:30.649 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:41dfccab:::benchmark_data_smithi177_31116_object7586:head# 2024-09-13T11:24:30.651 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:41e0c6ab:::benchmark_data_smithi177_31116_object249:head# 2024-09-13T11:24:30.652 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:41e16739:::benchmark_data_smithi177_31116_object99414:head# 2024-09-13T11:24:30.653 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:41e1ffc4:::benchmark_data_smithi177_31116_object82543:head# 2024-09-13T11:24:30.653 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:41e2315d:::benchmark_data_smithi177_31116_object100125:head# 2024-09-13T11:24:30.654 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:41e24c3b:::benchmark_data_smithi177_31116_object8775:head# 2024-09-13T11:24:30.654 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:41e27808:::benchmark_data_smithi177_31116_object21529:head# 2024-09-13T11:24:30.655 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:41e2d3fc:::benchmark_data_smithi177_31116_object106447:head# 2024-09-13T11:24:30.655 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:41e2e6c5:::benchmark_data_smithi177_31116_object92386:head# 2024-09-13T11:24:30.656 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:41e2fac0:::benchmark_data_smithi177_31116_object108495:head# 2024-09-13T11:24:30.656 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:41e43a6c:::benchmark_data_smithi177_31116_object15337:head# 2024-09-13T11:24:30.656 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:41e6b2cd:::benchmark_data_smithi177_31116_object55693:head# 2024-09-13T11:24:30.657 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:41e7987a:::benchmark_data_smithi177_31116_object88585:head# 2024-09-13T11:24:30.657 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:41e7bfb5:::benchmark_data_smithi177_31116_object52908:head# 2024-09-13T11:24:30.658 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:41e7cd26:::benchmark_data_smithi177_31116_object45645:head# 2024-09-13T11:24:30.659 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:41e93e2f:::benchmark_data_smithi177_31116_object76192:head# 2024-09-13T11:24:30.659 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:41ea13f5:::benchmark_data_smithi177_31116_object109496:head# 2024-09-13T11:24:30.660 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:41ea6534:::benchmark_data_smithi177_31116_object68643:head# 2024-09-13T11:24:30.661 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:41eb6ecf:::benchmark_data_smithi177_31116_object31325:head# 2024-09-13T11:24:30.661 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:41eb78e5:::benchmark_data_smithi177_31116_object86966:head# 2024-09-13T11:24:30.662 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:41eb7f51:::benchmark_data_smithi177_31116_object100240:head# 2024-09-13T11:24:30.662 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:41eb86fb:::benchmark_data_smithi177_31116_object79892:head# 2024-09-13T11:24:30.663 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:41ec5e14:::benchmark_data_smithi177_31116_object65842:head# 2024-09-13T11:24:30.663 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:41ecd024:::benchmark_data_smithi177_31116_object61905:head# 2024-09-13T11:24:30.664 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:41ee084a:::benchmark_data_smithi177_31116_object31971:head# 2024-09-13T11:24:30.664 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:41ee1399:::benchmark_data_smithi177_31116_object74396:head# 2024-09-13T11:24:30.665 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:41ef40e6:::benchmark_data_smithi177_31116_object2330:head# 2024-09-13T11:24:30.665 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:41f0e97b:::benchmark_data_smithi177_31116_object55202:head# 2024-09-13T11:24:30.666 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:41f0f326:::benchmark_data_smithi177_31116_object58741:head# 2024-09-13T11:24:30.667 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:41f162ad:::benchmark_data_smithi177_31116_object9156:head# 2024-09-13T11:24:30.667 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:41f25773:::benchmark_data_smithi177_31116_object74134:head# 2024-09-13T11:24:30.668 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:41f25a38:::benchmark_data_smithi177_31116_object42282:head# 2024-09-13T11:24:30.668 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:41f2725e:::benchmark_data_smithi177_31116_object66051:head# 2024-09-13T11:24:30.669 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:41f3010d:::benchmark_data_smithi177_31116_object29514:head# 2024-09-13T11:24:30.669 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:41f30644:::benchmark_data_smithi177_31116_object68673:head# 2024-09-13T11:24:30.672 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:41f34694:::benchmark_data_smithi177_31116_object106609:head# 2024-09-13T11:24:30.672 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:41f348fb:::benchmark_data_smithi177_31116_object47131:head# 2024-09-13T11:24:30.673 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:41f3a2d6:::benchmark_data_smithi177_31116_object109899:head# 2024-09-13T11:24:30.673 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:41f3a5f5:::benchmark_data_smithi177_31116_object56746:head# 2024-09-13T11:24:30.674 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:41f3bd88:::benchmark_data_smithi177_31116_object2887:head# 2024-09-13T11:24:30.674 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:41f43319:::benchmark_data_smithi177_31116_object39645:head# 2024-09-13T11:24:30.675 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:41f44a03:::benchmark_data_smithi177_31116_object55740:head# 2024-09-13T11:24:30.675 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:41f47b4d:::benchmark_data_smithi177_31116_object40505:head# 2024-09-13T11:24:30.676 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:41f48701:::benchmark_data_smithi177_31116_object61012:head# 2024-09-13T11:24:30.677 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:41f550fc:::benchmark_data_smithi177_31116_object56749:head# 2024-09-13T11:24:30.677 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:41f6212f:::benchmark_data_smithi177_31116_object78912:head# 2024-09-13T11:24:30.677 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:41f6935d:::benchmark_data_smithi177_31116_object1407:head# 2024-09-13T11:24:30.678 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:41f70e80:::benchmark_data_smithi177_31116_object22135:head# 2024-09-13T11:24:30.678 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:41f85f7e:::benchmark_data_smithi177_31116_object87654:head# 2024-09-13T11:24:30.679 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:41f9968d:::benchmark_data_smithi177_31116_object61201:head# 2024-09-13T11:24:30.679 INFO:tasks.ceph.osd.7.smithi177.stderr:2024-09-13T11:24:30.675+0000 7f897b51d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26575) UID: 0 2024-09-13T11:24:30.681 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:41f9dc61:::benchmark_data_smithi177_31116_object109828:head# 2024-09-13T11:24:30.682 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:41fa7a17:::benchmark_data_smithi177_31116_object64838:head# 2024-09-13T11:24:30.682 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:41fb1714:::benchmark_data_smithi177_31116_object100619:head# 2024-09-13T11:24:30.683 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:41fb38de:::benchmark_data_smithi177_31116_object86965:head# 2024-09-13T11:24:30.683 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:41fb92ce:::benchmark_data_smithi177_31116_object108759:head# 2024-09-13T11:24:30.684 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:41fbee3c:::benchmark_data_smithi177_31116_object25259:head# 2024-09-13T11:24:30.684 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:41fc70f3:::benchmark_data_smithi177_31116_object74599:head# 2024-09-13T11:24:30.685 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:41fd496c:::benchmark_data_smithi177_31116_object11093:head# 2024-09-13T11:24:30.685 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:41fd5496:::benchmark_data_smithi177_31116_object19886:head# 2024-09-13T11:24:30.686 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:41fdbf6f:::benchmark_data_smithi177_31116_object85474:head# 2024-09-13T11:24:30.687 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:42001945:::benchmark_data_smithi177_31116_object95656:head# 2024-09-13T11:24:30.688 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4200a506:::benchmark_data_smithi177_31116_object99741:head# 2024-09-13T11:24:30.688 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4201eacf:::benchmark_data_smithi177_31116_object32824:head# 2024-09-13T11:24:30.689 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-13T11:24:30.689 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:24:30.690 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:42026393:::benchmark_data_smithi177_31116_object95837:head# 2024-09-13T11:24:30.691 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4202b76f:::benchmark_data_smithi177_31116_object63761:head# 2024-09-13T11:24:30.691 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:420313fa:::benchmark_data_smithi177_31116_object89006:head# 2024-09-13T11:24:30.691 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:420370c5:::benchmark_data_smithi177_31116_object50629:head# 2024-09-13T11:24:30.691 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:420418c0:::benchmark_data_smithi177_31116_object15385:head# 2024-09-13T11:24:30.692 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:42042124:::benchmark_data_smithi177_31116_object61088:head# 2024-09-13T11:24:30.693 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:42042527:::benchmark_data_smithi177_31116_object59674:head# 2024-09-13T11:24:30.694 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4204f44a:::benchmark_data_smithi177_31116_object76485:head# 2024-09-13T11:24:30.695 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:420517fe:::benchmark_data_smithi177_31116_object102366:head# 2024-09-13T11:24:30.696 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:42052033:::benchmark_data_smithi177_31116_object16948:head# 2024-09-13T11:24:30.696 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:42052c1c:::benchmark_data_smithi177_31116_object21764:head# 2024-09-13T11:24:30.697 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:42058250:::benchmark_data_smithi177_31116_object29363:head# 2024-09-13T11:24:30.697 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:42059c06:::benchmark_data_smithi177_31116_object92327:head# 2024-09-13T11:24:30.698 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4205bd2e:::benchmark_data_smithi177_31116_object31274:head# 2024-09-13T11:24:30.698 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4205e255:::benchmark_data_smithi177_31116_object44687:head# 2024-09-13T11:24:30.699 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:42063c69:::benchmark_data_smithi177_31116_object6361:head# 2024-09-13T11:24:30.699 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:42065fb6:::benchmark_data_smithi177_31116_object17414:head# 2024-09-13T11:24:30.700 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4206a99c:::benchmark_data_smithi177_31116_object91302:head# 2024-09-13T11:24:30.701 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4208adad:::benchmark_data_smithi177_31116_object2575:head# 2024-09-13T11:24:30.701 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4208eedb:::benchmark_data_smithi177_31116_object34058:head# 2024-09-13T11:24:30.704 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4209617c:::benchmark_data_smithi177_31116_object87966:head# 2024-09-13T11:24:30.705 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4209e3cf:::benchmark_data_smithi177_31116_object98136:head# 2024-09-13T11:24:30.705 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:420ae81c:::benchmark_data_smithi177_31116_object6493:head# 2024-09-13T11:24:30.706 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:420bacc3:::benchmark_data_smithi177_31116_object63777:head# 2024-09-13T11:24:30.706 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:420bc562:::benchmark_data_smithi177_31116_object104849:head# 2024-09-13T11:24:30.707 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:420c6fee:::benchmark_data_smithi177_31116_object45244:head# 2024-09-13T11:24:30.708 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-13T11:24:30.710 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:420d7a50:::benchmark_data_smithi177_31116_object72072:head# 2024-09-13T11:24:30.710 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:420dac90:::benchmark_data_smithi177_31116_object62462:head# 2024-09-13T11:24:30.710 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:420dea0f:::benchmark_data_smithi177_31116_object7714:head# 2024-09-13T11:24:30.710 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:420e93ab:::benchmark_data_smithi177_31116_object19458:head# 2024-09-13T11:24:30.710 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:420f9ed5:::benchmark_data_smithi177_31116_object60613:head# 2024-09-13T11:24:30.710 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:42108f04:::benchmark_data_smithi177_31116_object44833:head# 2024-09-13T11:24:30.711 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4210a277:::benchmark_data_smithi177_31116_object79003:head# 2024-09-13T11:24:30.711 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:421110dc:::benchmark_data_smithi177_31116_object82497:head# 2024-09-13T11:24:30.712 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:421147a6:::benchmark_data_smithi177_31116_object49433:head# 2024-09-13T11:24:30.712 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:421218ab:::benchmark_data_smithi177_31116_object85808:head# 2024-09-13T11:24:30.713 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:42140b53:::benchmark_data_smithi177_31116_object96525:head# 2024-09-13T11:24:30.714 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:42143735:::benchmark_data_smithi177_31116_object1668:head# 2024-09-13T11:24:30.714 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:42144f94:::benchmark_data_smithi177_31116_object76689:head# 2024-09-13T11:24:30.715 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:421478c8:::benchmark_data_smithi177_31116_object57983:head# 2024-09-13T11:24:30.716 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:42152cb5:::benchmark_data_smithi177_31116_object61134:head# 2024-09-13T11:24:30.718 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:42154ca9:::benchmark_data_smithi177_31116_object51303:head# 2024-09-13T11:24:30.722 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:42167505:::benchmark_data_smithi177_31116_object40874:head# 2024-09-13T11:24:30.722 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:42167a63:::benchmark_data_smithi177_31116_object11940:head# 2024-09-13T11:24:30.723 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4216ca27:::benchmark_data_smithi177_31116_object483:head# 2024-09-13T11:24:30.725 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4216cba0:::benchmark_data_smithi177_31116_object18126:head# 2024-09-13T11:24:30.725 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4216e0e9:::benchmark_data_smithi177_31116_object17316:head# 2024-09-13T11:24:30.726 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:421709d0:::benchmark_data_smithi177_31116_object36452:head# 2024-09-13T11:24:30.727 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:42174e98:::benchmark_data_smithi177_31116_object100010:head# 2024-09-13T11:24:30.727 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:421793d1:::benchmark_data_smithi177_31116_object34774:head# 2024-09-13T11:24:30.728 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4218e965:::benchmark_data_smithi177_31116_object18763:head# 2024-09-13T11:24:30.728 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:42190843:::benchmark_data_smithi177_31116_object42751:head# 2024-09-13T11:24:30.729 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:42195d52:::benchmark_data_smithi177_31116_object82617:head# 2024-09-13T11:24:30.730 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4219c5d8:::benchmark_data_smithi177_31116_object91479:head# 2024-09-13T11:24:30.730 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4219f51e:::benchmark_data_smithi177_31116_object41894:head# 2024-09-13T11:24:30.731 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:421a18c6:::benchmark_data_smithi177_31116_object24633:head# 2024-09-13T11:24:30.731 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:421aeece:::benchmark_data_smithi177_31116_object11631:head# 2024-09-13T11:24:30.732 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:421af8dc:::benchmark_data_smithi177_31116_object88246:head# 2024-09-13T11:24:30.732 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:421b1b64:::benchmark_data_smithi177_31116_object110313:head# 2024-09-13T11:24:30.733 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:421b5864:::benchmark_data_smithi177_31116_object78945:head# 2024-09-13T11:24:30.734 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:421c1c7e:::benchmark_data_smithi177_31116_object95517:head# 2024-09-13T11:24:30.734 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:421c5d9a:::benchmark_data_smithi177_31116_object62822:head# 2024-09-13T11:24:30.735 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:421c6855:::benchmark_data_smithi177_31116_object95097:head# 2024-09-13T11:24:30.735 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:421ce97c:::benchmark_data_smithi177_31116_object43331:head# 2024-09-13T11:24:30.736 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:421d6c4d:::benchmark_data_smithi177_31116_object90102:head# 2024-09-13T11:24:30.737 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:421f01c3:::benchmark_data_smithi177_31116_object58842:head# 2024-09-13T11:24:30.737 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:421f7618:::benchmark_data_smithi177_31116_object4097:head# 2024-09-13T11:24:30.738 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:421fa0f2:::benchmark_data_smithi177_31116_object26952:head# 2024-09-13T11:24:30.741 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:421fac61:::benchmark_data_smithi177_31116_object15859:head# 2024-09-13T11:24:30.742 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:42206407:::benchmark_data_smithi177_31116_object78725:head# 2024-09-13T11:24:30.742 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4220a6ed:::benchmark_data_smithi177_31116_object76098:head# 2024-09-13T11:24:30.743 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4220e71c:::benchmark_data_smithi177_31116_object78408:head# 2024-09-13T11:24:30.743 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:42231a55:::benchmark_data_smithi177_31116_object38310:head# 2024-09-13T11:24:30.744 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4223a567:::benchmark_data_smithi177_31116_object62124:head# 2024-09-13T11:24:30.744 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4223c16a:::benchmark_data_smithi177_31116_object22207:head# 2024-09-13T11:24:30.745 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4225f664:::benchmark_data_smithi177_31116_object44298:head# 2024-09-13T11:24:30.746 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:42271a7e:::benchmark_data_smithi177_31116_object34823:head# 2024-09-13T11:24:30.746 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:42277ca8:::benchmark_data_smithi177_31116_object62935:head# 2024-09-13T11:24:30.747 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4227e189:::benchmark_data_smithi177_31116_object81386:head# 2024-09-13T11:24:30.747 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4228cf54:::benchmark_data_smithi177_31116_object86453:head# 2024-09-13T11:24:30.749 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:42299bdf:::benchmark_data_smithi177_31116_object80785:head# 2024-09-13T11:24:30.750 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4229aef5:::benchmark_data_smithi177_31116_object50559:head# 2024-09-13T11:24:30.750 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4229cc77:::benchmark_data_smithi177_31116_object34641:head# 2024-09-13T11:24:30.751 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:422a272c:::benchmark_data_smithi177_31116_object83945:head# 2024-09-13T11:24:30.752 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:422a3434:::benchmark_data_smithi177_31116_object67292:head# 2024-09-13T11:24:30.752 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:422a4afe:::benchmark_data_smithi177_31116_object63407:head# 2024-09-13T11:24:30.753 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:422a5d0c:::benchmark_data_smithi177_31116_object21787:head# 2024-09-13T11:24:30.753 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:422b97d3:::benchmark_data_smithi177_31116_object81128:head# 2024-09-13T11:24:30.754 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:422befac:::benchmark_data_smithi177_31116_object62327:head# 2024-09-13T11:24:30.755 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:422c1946:::benchmark_data_smithi177_31116_object11620:head# 2024-09-13T11:24:30.755 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:422c420d:::benchmark_data_smithi177_31116_object34746:head# 2024-09-13T11:24:30.757 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:422e2045:::benchmark_data_smithi177_31116_object30093:head# 2024-09-13T11:24:30.758 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:422ed172:::benchmark_data_smithi177_31116_object42210:head# 2024-09-13T11:24:30.758 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:422ee9c1:::benchmark_data_smithi177_31116_object105408:head# 2024-09-13T11:24:30.759 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:422f0586:::benchmark_data_smithi177_31116_object18790:head# 2024-09-13T11:24:30.759 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4230ff6d:::benchmark_data_smithi177_31116_object96321:head# 2024-09-13T11:24:30.761 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4231b0d8:::benchmark_data_smithi177_31116_object10085:head# 2024-09-13T11:24:30.761 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4232e607:::benchmark_data_smithi177_31116_object4236:head# 2024-09-13T11:24:30.761 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4232f1a6:::benchmark_data_smithi177_31116_object30399:head# 2024-09-13T11:24:30.762 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:423303b8:::benchmark_data_smithi177_31116_object79667:head# 2024-09-13T11:24:30.763 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:42337336:::benchmark_data_smithi177_31116_object27193:head# 2024-09-13T11:24:30.764 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:42341fe5:::benchmark_data_smithi177_31116_object61761:head# 2024-09-13T11:24:30.765 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4234341a:::benchmark_data_smithi177_31116_object80541:head# 2024-09-13T11:24:30.765 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4234ee65:::benchmark_data_smithi177_31116_object83087:head# 2024-09-13T11:24:30.766 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4235066c:::benchmark_data_smithi177_31116_object76136:head# 2024-09-13T11:24:30.766 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:42354128:::benchmark_data_smithi177_31116_object38162:head# 2024-09-13T11:24:30.767 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:42358e2b:::benchmark_data_smithi177_31116_object7948:head# 2024-09-13T11:24:30.767 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4235a7d2:::benchmark_data_smithi177_31116_object50855:head# 2024-09-13T11:24:30.768 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:42363918:::benchmark_data_smithi177_31116_object12677:head# 2024-09-13T11:24:30.768 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:42366936:::benchmark_data_smithi177_31116_object28734:head# 2024-09-13T11:24:30.769 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4237955f:::benchmark_data_smithi177_31116_object108817:head# 2024-09-13T11:24:30.771 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4237b303:::benchmark_data_smithi177_31116_object41969:head# 2024-09-13T11:24:30.771 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:423824ae:::benchmark_data_smithi177_31116_object33935:head# 2024-09-13T11:24:30.771 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:42383c70:::benchmark_data_smithi177_31116_object20529:head# 2024-09-13T11:24:30.771 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:42383f3a:::benchmark_data_smithi177_31116_object98431:head# 2024-09-13T11:24:30.771 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:42387358:::benchmark_data_smithi177_31116_object95630:head# 2024-09-13T11:24:30.772 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:42387751:::benchmark_data_smithi177_31116_object20044:head# 2024-09-13T11:24:30.773 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4238a213:::benchmark_data_smithi177_31116_object81610:head# 2024-09-13T11:24:30.773 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4239d69c:::benchmark_data_smithi177_31116_object33064:head# 2024-09-13T11:24:30.774 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4239dafb:::benchmark_data_smithi177_31116_object59922:head# 2024-09-13T11:24:30.774 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:423ae075:::benchmark_data_smithi177_31116_object3746:head# 2024-09-13T11:24:30.774 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:423b1ca3:::benchmark_data_smithi177_31116_object64554:head# 2024-09-13T11:24:30.775 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:423b58aa:::benchmark_data_smithi177_31116_object53971:head# 2024-09-13T11:24:30.776 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:423b92e9:::benchmark_data_smithi177_31116_object40027:head# 2024-09-13T11:24:30.776 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:423e086d:::benchmark_data_smithi177_31116_object39051:head# 2024-09-13T11:24:30.777 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:423f5876:::benchmark_data_smithi177_31116_object55157:head# 2024-09-13T11:24:30.777 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:423f621c:::benchmark_data_smithi177_31116_object59193:head# 2024-09-13T11:24:30.778 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:423f78f3:::benchmark_data_smithi177_31116_object50818:head# 2024-09-13T11:24:30.779 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:424029bd:::benchmark_data_smithi177_31116_object69854:head# 2024-09-13T11:24:30.779 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:424060a7:::benchmark_data_smithi177_31116_object107319:head# 2024-09-13T11:24:30.780 INFO:tasks.ceph.osd.1.smithi092.stderr:2024-09-13T11:24:30.774+0000 7fe7789c5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33718) UID: 0 2024-09-13T11:24:30.780 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4241262c:::benchmark_data_smithi177_31116_object2282:head# 2024-09-13T11:24:30.780 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4241e74a:::benchmark_data_smithi177_31116_object106333:head# 2024-09-13T11:24:30.781 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:42433669:::benchmark_data_smithi177_31116_object63140:head# 2024-09-13T11:24:30.781 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4243bd65:::benchmark_data_smithi177_31116_object90762:head# 2024-09-13T11:24:30.782 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4243cea5:::benchmark_data_smithi177_31116_object33629:head# 2024-09-13T11:24:30.782 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:42447ddd:::benchmark_data_smithi177_31116_object31742:head# 2024-09-13T11:24:30.783 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:42457235:::benchmark_data_smithi177_31116_object19803:head# 2024-09-13T11:24:30.784 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4245d429:::benchmark_data_smithi177_31116_object68084:head# 2024-09-13T11:24:30.784 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:42462273:::benchmark_data_smithi177_31116_object4072:head# 2024-09-13T11:24:30.785 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:42466555:::benchmark_data_smithi177_31116_object62389:head# 2024-09-13T11:24:30.786 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4246a999:::benchmark_data_smithi177_31116_object105436:head# 2024-09-13T11:24:30.786 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:424732c9:::benchmark_data_smithi177_31116_object69766:head# 2024-09-13T11:24:30.789 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:424744e2:::benchmark_data_smithi177_31116_object45571:head# 2024-09-13T11:24:30.791 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4247694c:::benchmark_data_smithi177_31116_object81887:head# 2024-09-13T11:24:30.792 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:42479b2c:::benchmark_data_smithi177_31116_object106879:head# 2024-09-13T11:24:30.792 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:42489a62:::benchmark_data_smithi177_31116_object14761:head# 2024-09-13T11:24:30.793 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:42491bed:::benchmark_data_smithi177_31116_object15969:head# 2024-09-13T11:24:30.793 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:424979db:::benchmark_data_smithi177_31116_object29113:head# 2024-09-13T11:24:30.795 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:424b222d:::benchmark_data_smithi177_31116_object96496:head# 2024-09-13T11:24:30.795 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:424b324c:::benchmark_data_smithi177_31116_object76961:head# 2024-09-13T11:24:30.795 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:424bf8f7:::benchmark_data_smithi177_31116_object75047:head# 2024-09-13T11:24:30.796 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:424c9e53:::benchmark_data_smithi177_31116_object28803:head# 2024-09-13T11:24:30.796 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:424d2517:::benchmark_data_smithi177_31116_object55004:head# 2024-09-13T11:24:30.797 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:424e602f:::benchmark_data_smithi177_31116_object55553:head# 2024-09-13T11:24:30.797 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:424e8e9a:::benchmark_data_smithi177_31116_object39766:head# 2024-09-13T11:24:30.797 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:424f14f7:::benchmark_data_smithi177_31116_object30205:head# 2024-09-13T11:24:30.798 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:425199c2:::benchmark_data_smithi177_31116_object29031:head# 2024-09-13T11:24:30.799 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:42522cb7:::benchmark_data_smithi177_31116_object99361:head# 2024-09-13T11:24:30.800 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4252772a:::benchmark_data_smithi177_31116_object76660:head# 2024-09-13T11:24:30.800 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4254c5bd:::benchmark_data_smithi177_31116_object49682:head# 2024-09-13T11:24:30.801 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:42552b24:::benchmark_data_smithi177_31116_object15445:head# 2024-09-13T11:24:30.801 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:425570b8:::benchmark_data_smithi177_31116_object42103:head# 2024-09-13T11:24:30.802 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4256860f:::benchmark_data_smithi177_31116_object69160:head# 2024-09-13T11:24:30.802 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4256894a:::benchmark_data_smithi177_31116_object36006:head# 2024-09-13T11:24:30.805 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:42568e14:::benchmark_data_smithi177_31116_object105249:head# 2024-09-13T11:24:30.806 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4256f10c:::benchmark_data_smithi177_31116_object17698:head# 2024-09-13T11:24:30.806 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:42572307:::benchmark_data_smithi177_31116_object69090:head# 2024-09-13T11:24:30.807 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:42577009:::benchmark_data_smithi177_31116_object62743:head# 2024-09-13T11:24:30.807 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4258a8aa:::benchmark_data_smithi177_31116_object96058:head# 2024-09-13T11:24:30.808 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:42596ae7:::benchmark_data_smithi177_31116_object87586:head# 2024-09-13T11:24:30.810 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4259cef7:::benchmark_data_smithi177_31116_object58492:head# 2024-09-13T11:24:30.811 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:425a9632:::benchmark_data_smithi177_31116_object40761:head# 2024-09-13T11:24:30.811 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:425b8a6c:::benchmark_data_smithi177_31116_object100332:head# 2024-09-13T11:24:30.812 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:425bb141:::benchmark_data_smithi177_31116_object23241:head# 2024-09-13T11:24:30.812 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:425bbc34:::benchmark_data_smithi177_31116_object85847:head# 2024-09-13T11:24:30.813 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:425cc875:::benchmark_data_smithi177_31116_object38457:head# 2024-09-13T11:24:30.813 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:425cdea9:::benchmark_data_smithi177_31116_object95168:head# 2024-09-13T11:24:30.814 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:425cff8e:::benchmark_data_smithi177_31116_object84089:head# 2024-09-13T11:24:30.814 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:425d7a67:::benchmark_data_smithi177_31116_object92939:head# 2024-09-13T11:24:30.814 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:425daad6:::benchmark_data_smithi177_31116_object102397:head# 2024-09-13T11:24:30.815 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:425e97dc:::benchmark_data_smithi177_31116_object82912:head# 2024-09-13T11:24:30.815 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:425f19ec:::benchmark_data_smithi177_31116_object88578:head# 2024-09-13T11:24:30.815 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:425f4530:::benchmark_data_smithi177_31116_object90006:head# 2024-09-13T11:24:30.816 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:425f6ca0:::benchmark_data_smithi177_31116_object2300:head# 2024-09-13T11:24:30.817 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:425fa8e1:::benchmark_data_smithi177_31116_object62108:head# 2024-09-13T11:24:30.817 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:425fcf41:::benchmark_data_smithi177_31116_object74719:head# 2024-09-13T11:24:30.818 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:426098ce:::benchmark_data_smithi177_31116_object84477:head# 2024-09-13T11:24:30.820 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4260f523:::benchmark_data_smithi177_31116_object51930:head# 2024-09-13T11:24:30.821 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4262019b:::benchmark_data_smithi177_31116_object83713:head# 2024-09-13T11:24:30.824 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:42623a1c:::benchmark_data_smithi177_31116_object66729:head# 2024-09-13T11:24:30.825 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4262be1c:::benchmark_data_smithi177_31116_object85707:head# 2024-09-13T11:24:30.825 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:42632a62:::benchmark_data_smithi177_31116_object63602:head# 2024-09-13T11:24:30.826 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:42637cdf:::benchmark_data_smithi177_31116_object10265:head# 2024-09-13T11:24:30.826 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4264665a:::benchmark_data_smithi177_31116_object3790:head# 2024-09-13T11:24:30.827 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:42648fa0:::benchmark_data_smithi177_31116_object97589:head# 2024-09-13T11:24:30.829 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4264fa81:::benchmark_data_smithi177_31116_object60130:head# 2024-09-13T11:24:30.830 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4265790c:::benchmark_data_smithi177_31116_object14059:head# 2024-09-13T11:24:30.830 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4265cc41:::benchmark_data_smithi177_31116_object72392:head# 2024-09-13T11:24:30.832 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:426690c4:::benchmark_data_smithi177_31116_object80831:head# 2024-09-13T11:24:30.833 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:42676066:::benchmark_data_smithi177_31116_object106003:head# 2024-09-13T11:24:30.833 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4267b9a6:::benchmark_data_smithi177_31116_object28557:head# 2024-09-13T11:24:30.834 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:42688fbf:::benchmark_data_smithi177_31116_object8423:head# 2024-09-13T11:24:30.834 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:42692b96:::benchmark_data_smithi177_31116_object52190:head# 2024-09-13T11:24:30.836 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:42696f4d:::benchmark_data_smithi177_31116_object83711:head# 2024-09-13T11:24:30.836 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4269db30:::benchmark_data_smithi177_31116_object27856:head# 2024-09-13T11:24:30.837 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:426ac6e9:::benchmark_data_smithi177_31116_object19928:head# 2024-09-13T11:24:30.839 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:426af162:::benchmark_data_smithi177_31116_object20231:head# 2024-09-13T11:24:30.839 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:426c5716:::benchmark_data_smithi177_31116_object108753:head# 2024-09-13T11:24:30.839 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:426dc482:::benchmark_data_smithi177_31116_object24165:head# 2024-09-13T11:24:30.840 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:426fa4bc:::benchmark_data_smithi177_31116_object3846:head# 2024-09-13T11:24:30.840 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4270b0a0:::benchmark_data_smithi177_31116_object50539:head# 2024-09-13T11:24:30.841 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4271d86f:::benchmark_data_smithi177_31116_object104835:head# 2024-09-13T11:24:30.841 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:42725d1d:::benchmark_data_smithi177_31116_object54610:head# 2024-09-13T11:24:30.842 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4273d866:::benchmark_data_smithi177_31116_object20433:head# 2024-09-13T11:24:30.843 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4274119b:::benchmark_data_smithi177_31116_object90863:head# 2024-09-13T11:24:30.843 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:42741554:::benchmark_data_smithi177_31116_object37764:head# 2024-09-13T11:24:30.844 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:42747ee7:::benchmark_data_smithi177_31116_object4213:head# 2024-09-13T11:24:30.846 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4274bb20:::benchmark_data_smithi177_31116_object76330:head# 2024-09-13T11:24:30.846 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4275b4f1:::benchmark_data_smithi177_31116_object31829:head# 2024-09-13T11:24:30.846 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:427648b3:::benchmark_data_smithi177_31116_object64842:head# 2024-09-13T11:24:30.847 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4276e010:::benchmark_data_smithi177_31116_object77849:head# 2024-09-13T11:24:30.847 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4277a820:::benchmark_data_smithi177_31116_object46443:head# 2024-09-13T11:24:30.847 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:427801dd:::benchmark_data_smithi177_31116_object97377:head# 2024-09-13T11:24:30.848 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4278adc7:::benchmark_data_smithi177_31116_object30453:head# 2024-09-13T11:24:30.849 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:42791a42:::benchmark_data_smithi177_31116_object86799:head# 2024-09-13T11:24:30.849 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:427a23f9:::benchmark_data_smithi177_31116_object62839:head# 2024-09-13T11:24:30.851 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:427a3ecb:::benchmark_data_smithi177_31116_object37926:head# 2024-09-13T11:24:30.854 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:427a8670:::benchmark_data_smithi177_31116_object88663:head# 2024-09-13T11:24:30.855 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:427b5941:::benchmark_data_smithi177_31116_object13260:head# 2024-09-13T11:24:30.856 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:427be147:::benchmark_data_smithi177_31116_object67786:head# 2024-09-13T11:24:30.857 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:427d02b9:::benchmark_data_smithi177_31116_object68421:head# 2024-09-13T11:24:30.857 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:427d92ae:::benchmark_data_smithi177_31116_object8254:head# 2024-09-13T11:24:30.858 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:427efe55:::benchmark_data_smithi177_31116_object70750:head# 2024-09-13T11:24:30.858 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:427fa6b5:::benchmark_data_smithi177_31116_object22676:head# 2024-09-13T11:24:30.859 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-13T11:24:30.859 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:24:30.861 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:42808b4f:::benchmark_data_smithi177_31116_object47170:head# 2024-09-13T11:24:30.861 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:42817344:::benchmark_data_smithi177_31116_object41357:head# 2024-09-13T11:24:30.863 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:428191a8:::benchmark_data_smithi177_31116_object97945:head# 2024-09-13T11:24:30.864 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:428220e4:::benchmark_data_smithi177_31116_object17664:head# 2024-09-13T11:24:30.864 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:42825729:::benchmark_data_smithi177_31116_object61452:head# 2024-09-13T11:24:30.865 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:428388e2:::benchmark_data_smithi177_31116_object21704:head# 2024-09-13T11:24:30.865 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4283cb55:::benchmark_data_smithi177_31116_object68912:head# 2024-09-13T11:24:30.866 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:428531c0:::benchmark_data_smithi177_31116_object3510:head# 2024-09-13T11:24:30.867 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:42855831:::benchmark_data_smithi177_31116_object24905:head# 2024-09-13T11:24:30.867 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4285edc4:::benchmark_data_smithi177_31116_object70008:head# 2024-09-13T11:24:30.868 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:428725b0:::benchmark_data_smithi177_31116_object72757:head# 2024-09-13T11:24:30.868 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4287b9c4:::benchmark_data_smithi177_31116_object3274:head# 2024-09-13T11:24:30.869 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:428902db:::benchmark_data_smithi177_31116_object29771:head# 2024-09-13T11:24:30.869 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4289785a:::benchmark_data_smithi177_31116_object65989:head# 2024-09-13T11:24:30.872 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:42898c9e:::benchmark_data_smithi177_31116_object82259:head# 2024-09-13T11:24:30.872 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:428a9264:::benchmark_data_smithi177_31116_object94620:head# 2024-09-13T11:24:30.872 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:428ade08:::benchmark_data_smithi177_31116_object61607:head# 2024-09-13T11:24:30.872 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:428b1806:::benchmark_data_smithi177_31116_object89653:head# 2024-09-13T11:24:30.873 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:428c40a9:::benchmark_data_smithi177_31116_object89760:head# 2024-09-13T11:24:30.876 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:428c8086:::benchmark_data_smithi177_31116_object103483:head# 2024-09-13T11:24:30.876 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:428e3b11:::benchmark_data_smithi177_31116_object99470:head# 2024-09-13T11:24:30.878 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:429083e8:::benchmark_data_smithi177_31116_object60250:head# 2024-09-13T11:24:30.879 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:42910a00:::benchmark_data_smithi177_31116_object36989:head# 2024-09-13T11:24:30.881 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-13T11:24:30.882 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4291878e:::benchmark_data_smithi177_31116_object70700:head# 2024-09-13T11:24:30.882 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:42921fa6:::benchmark_data_smithi177_31116_object94838:head# 2024-09-13T11:24:30.882 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:42922964:::benchmark_data_smithi177_31116_object73328:head# 2024-09-13T11:24:30.882 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4292dd80:::benchmark_data_smithi177_31116_object30777:head# 2024-09-13T11:24:30.883 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:429324e8:::benchmark_data_smithi177_31116_object76087:head# 2024-09-13T11:24:30.883 INFO:tasks.ceph.osd.9.smithi092.stderr:2024-09-13T11:24:30.874+0000 7ff5ab3d0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26348) UID: 0 2024-09-13T11:24:30.885 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:42932619:::benchmark_data_smithi177_31116_object20165:head# 2024-09-13T11:24:30.885 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:42934085:::benchmark_data_smithi177_31116_object25276:head# 2024-09-13T11:24:30.885 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:42934db4:::benchmark_data_smithi177_31116_object81539:head# 2024-09-13T11:24:30.885 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:42936a79:::benchmark_data_smithi177_31116_object69313:head# 2024-09-13T11:24:30.885 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:42946cf4:::benchmark_data_smithi177_31116_object88580:head# 2024-09-13T11:24:30.886 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:42946fe1:::benchmark_data_smithi177_31116_object56640:head# 2024-09-13T11:24:30.886 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4294c610:::benchmark_data_smithi177_31116_object85135:head# 2024-09-13T11:24:30.886 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4295acdf:::benchmark_data_smithi177_31116_object1518:head# 2024-09-13T11:24:30.887 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:42966619:::benchmark_data_smithi177_31116_object53539:head# 2024-09-13T11:24:30.887 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4296c09e:::benchmark_data_smithi177_31116_object80196:head# 2024-09-13T11:24:30.888 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4296df08:::benchmark_data_smithi177_31116_object6752:head# 2024-09-13T11:24:30.888 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:42975432:::benchmark_data_smithi177_31116_object720:head# 2024-09-13T11:24:30.889 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4297f852:::benchmark_data_smithi177_31116_object72586:head# 2024-09-13T11:24:30.889 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4297f9a9:::benchmark_data_smithi177_31116_object6222:head# 2024-09-13T11:24:30.890 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:42981ba3:::benchmark_data_smithi177_31116_object32935:head# 2024-09-13T11:24:30.890 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:42988296:::benchmark_data_smithi177_31116_object39602:head# 2024-09-13T11:24:30.891 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:42993bf1:::benchmark_data_smithi177_31116_object31960:head# 2024-09-13T11:24:30.891 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:429949fa:::benchmark_data_smithi177_31116_object70623:head# 2024-09-13T11:24:30.892 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:42999a92:::benchmark_data_smithi177_31116_object82351:head# 2024-09-13T11:24:30.893 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4299f84a:::benchmark_data_smithi177_31116_object99869:head# 2024-09-13T11:24:30.893 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:429a8ca4:::benchmark_data_smithi177_31116_object105839:head# 2024-09-13T11:24:30.894 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:429aac61:::benchmark_data_smithi177_31116_object46229:head# 2024-09-13T11:24:30.894 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:429b97bd:::benchmark_data_smithi177_31116_object70380:head# 2024-09-13T11:24:30.894 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:429be747:::benchmark_data_smithi177_31116_object62110:head# 2024-09-13T11:24:30.895 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:429cb509:::benchmark_data_smithi177_31116_object6492:head# 2024-09-13T11:24:30.895 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:429d2f2a:::benchmark_data_smithi177_31116_object27499:head# 2024-09-13T11:24:30.896 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:42a019df:::benchmark_data_smithi177_31116_object57814:head# 2024-09-13T11:24:30.897 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:42a05d54:::benchmark_data_smithi177_31116_object24359:head# 2024-09-13T11:24:30.897 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:42a099c8:::benchmark_data_smithi177_31116_object72317:head# 2024-09-13T11:24:30.897 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:42a0f9cf:::benchmark_data_smithi177_31116_object94496:head# 2024-09-13T11:24:30.898 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:42a1359e:::benchmark_data_smithi177_31116_object66171:head# 2024-09-13T11:24:30.898 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:42a1cfd5:::benchmark_data_smithi177_31116_object49946:head# 2024-09-13T11:24:30.899 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:42a22955:::benchmark_data_smithi177_31116_object6644:head# 2024-09-13T11:24:30.900 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:42a245d7:::benchmark_data_smithi177_31116_object78640:head# 2024-09-13T11:24:30.900 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:42a3c94f:::benchmark_data_smithi177_31116_object22125:head# 2024-09-13T11:24:30.901 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:42a3f64b:::benchmark_data_smithi177_31116_object97279:head# 2024-09-13T11:24:30.901 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:42a4e4d6:::benchmark_data_smithi177_31116_object65038:head# 2024-09-13T11:24:30.901 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:42a5b133:::benchmark_data_smithi177_31116_object73624:head# 2024-09-13T11:24:30.903 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:42a6430c:::benchmark_data_smithi177_31116_object30111:head# 2024-09-13T11:24:30.903 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:42a7253e:::benchmark_data_smithi177_31116_object56482:head# 2024-09-13T11:24:30.903 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:42a73a42:::benchmark_data_smithi177_31116_object5846:head# 2024-09-13T11:24:30.904 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:42a77d41:::benchmark_data_smithi177_31116_object9096:head# 2024-09-13T11:24:30.904 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:42a7e8f5:::benchmark_data_smithi177_31116_object17904:head# 2024-09-13T11:24:30.905 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:42a8312f:::benchmark_data_smithi177_31116_object50877:head# 2024-09-13T11:24:30.905 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:42a83e85:::benchmark_data_smithi177_31116_object21234:head# 2024-09-13T11:24:30.905 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:42a93940:::benchmark_data_smithi177_31116_object12289:head# 2024-09-13T11:24:30.906 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:42a94ac1:::benchmark_data_smithi177_31116_object62919:head# 2024-09-13T11:24:30.906 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:42a9a7ea:::benchmark_data_smithi177_31116_object33742:head# 2024-09-13T11:24:30.908 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:42abfd46:::benchmark_data_smithi177_31116_object18943:head# 2024-09-13T11:24:30.908 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:42ac9098:::benchmark_data_smithi177_31116_object89488:head# 2024-09-13T11:24:30.908 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:42ad1a6d:::benchmark_data_smithi177_31116_object55337:head# 2024-09-13T11:24:30.909 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:42ad41cd:::benchmark_data_smithi177_31116_object20425:head# 2024-09-13T11:24:30.910 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:42ae31a6:::benchmark_data_smithi177_31116_object9236:head# 2024-09-13T11:24:30.910 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:42ae6291:::benchmark_data_smithi177_31116_object59569:head# 2024-09-13T11:24:30.911 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:42aeb69f:::benchmark_data_smithi177_31116_object25877:head# 2024-09-13T11:24:30.911 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:42af766e:::benchmark_data_smithi177_31116_object65922:head# 2024-09-13T11:24:30.911 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:42b0e450:::benchmark_data_smithi177_31116_object30945:head# 2024-09-13T11:24:30.912 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:42b14589:::benchmark_data_smithi177_31116_object80782:head# 2024-09-13T11:24:30.912 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:42b19fa0:::benchmark_data_smithi177_31116_object46810:head# 2024-09-13T11:24:30.913 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:42b26a21:::benchmark_data_smithi177_31116_object46412:head# 2024-09-13T11:24:30.913 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:42b41476:::benchmark_data_smithi177_31116_object76799:head# 2024-09-13T11:24:30.914 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:42b51343:::benchmark_data_smithi177_31116_object15350:head# 2024-09-13T11:24:30.914 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:42b5d887:::benchmark_data_smithi177_31116_object95962:head# 2024-09-13T11:24:30.915 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:42b601ad:::benchmark_data_smithi177_31116_object77037:head# 2024-09-13T11:24:30.915 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:42b64a0b:::benchmark_data_smithi177_31116_object1078:head# 2024-09-13T11:24:30.916 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:42b6ec5a:::benchmark_data_smithi177_31116_object109649:head# 2024-09-13T11:24:30.917 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:42b7340f:::benchmark_data_smithi177_31116_object6651:head# 2024-09-13T11:24:30.917 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:42b83256:::benchmark_data_smithi177_31116_object78509:head# 2024-09-13T11:24:30.918 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:42b85280:::benchmark_data_smithi177_31116_object75012:head# 2024-09-13T11:24:30.921 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:42b89ed5:::benchmark_data_smithi177_31116_object85972:head# 2024-09-13T11:24:30.921 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:42b8e142:::benchmark_data_smithi177_31116_object2149:head# 2024-09-13T11:24:30.921 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:42b9afc7:::benchmark_data_smithi177_31116_object86525:head# 2024-09-13T11:24:30.921 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:42b9d442:::benchmark_data_smithi177_31116_object70802:head# 2024-09-13T11:24:30.922 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:42b9e997:::benchmark_data_smithi177_31116_object99413:head# 2024-09-13T11:24:30.922 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:42bb216f:::benchmark_data_smithi177_31116_object80139:head# 2024-09-13T11:24:30.922 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:42bb72b0:::benchmark_data_smithi177_31116_object35317:head# 2024-09-13T11:24:30.923 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:42bbe3b7:::benchmark_data_smithi177_31116_object87856:head# 2024-09-13T11:24:30.923 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:42bc22e6:::benchmark_data_smithi177_31116_object18462:head# 2024-09-13T11:24:30.924 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:42bce868:::benchmark_data_smithi177_31116_object60556:head# 2024-09-13T11:24:30.924 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:42bd28ad:::benchmark_data_smithi177_31116_object25024:head# 2024-09-13T11:24:30.924 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:42bf660a:::benchmark_data_smithi177_31116_object2888:head# 2024-09-13T11:24:30.925 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:42c09bb2:::benchmark_data_smithi177_31116_object26201:head# 2024-09-13T11:24:30.925 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:42c0c9f0:::benchmark_data_smithi177_31116_object35282:head# 2024-09-13T11:24:30.926 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:42c2b663:::benchmark_data_smithi177_31116_object6274:head# 2024-09-13T11:24:30.926 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:42c48f6d:::benchmark_data_smithi177_31116_object31997:head# 2024-09-13T11:24:30.927 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:42c4b6b6:::benchmark_data_smithi177_31116_object30407:head# 2024-09-13T11:24:30.927 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:42c4d348:::benchmark_data_smithi177_31116_object51814:head# 2024-09-13T11:24:30.928 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:42c527e0:::benchmark_data_smithi177_31116_object94801:head# 2024-09-13T11:24:30.928 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:42c56ee5:::benchmark_data_smithi177_31116_object28049:head# 2024-09-13T11:24:30.929 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:42c69baf:::benchmark_data_smithi177_31116_object90315:head# 2024-09-13T11:24:30.929 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:42c6e343:::benchmark_data_smithi177_31116_object78336:head# 2024-09-13T11:24:30.930 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:42c85fca:::benchmark_data_smithi177_31116_object348:head# 2024-09-13T11:24:30.930 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:42c87822:::benchmark_data_smithi177_31116_object48696:head# 2024-09-13T11:24:30.931 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:42c8799a:::benchmark_data_smithi177_31116_object11016:head# 2024-09-13T11:24:30.932 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:42c8c78e:::benchmark_data_smithi177_31116_object25444:head# 2024-09-13T11:24:30.932 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:42c8d973:::benchmark_data_smithi177_31116_object101645:head# 2024-09-13T11:24:30.933 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:42c9c846:::benchmark_data_smithi177_31116_object1215:head# 2024-09-13T11:24:30.934 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:42ca1898:::benchmark_data_smithi177_31116_object6308:head# 2024-09-13T11:24:30.934 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:42ca2ede:::benchmark_data_smithi177_31116_object23186:head# 2024-09-13T11:24:30.935 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:42ca4497:::benchmark_data_smithi177_31116_object65014:head# 2024-09-13T11:24:30.935 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:42ca9663:::benchmark_data_smithi177_31116_object79115:head# 2024-09-13T11:24:30.936 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:42ccf6a7:::benchmark_data_smithi177_31116_object18402:head# 2024-09-13T11:24:30.936 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:42cd1e10:::benchmark_data_smithi177_31116_object28189:head# 2024-09-13T11:24:30.937 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:42cd761c:::benchmark_data_smithi177_31116_object67706:head# 2024-09-13T11:24:30.937 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:42ce6f33:::benchmark_data_smithi177_31116_object48264:head# 2024-09-13T11:24:30.938 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:42ce95e8:::benchmark_data_smithi177_31116_object47550:head# 2024-09-13T11:24:30.938 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:42d02c66:::benchmark_data_smithi177_31116_object40526:head# 2024-09-13T11:24:30.939 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:42d102e0:::benchmark_data_smithi177_31116_object38581:head# 2024-09-13T11:24:30.940 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:42d1af5a:::benchmark_data_smithi177_31116_object69152:head# 2024-09-13T11:24:30.941 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:42d24762:::benchmark_data_smithi177_31116_object6679:head# 2024-09-13T11:24:30.941 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:42d3ff15:::benchmark_data_smithi177_31116_object82817:head# 2024-09-13T11:24:30.942 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:42d43192:::benchmark_data_smithi177_31116_object108276:head# 2024-09-13T11:24:30.942 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:42d4c78c:::benchmark_data_smithi177_31116_object68346:head# 2024-09-13T11:24:30.943 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:42d4e66f:::benchmark_data_smithi177_31116_object10433:head# 2024-09-13T11:24:30.943 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:42d521b7:::benchmark_data_smithi177_31116_object66296:head# 2024-09-13T11:24:30.945 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:42d561c6:::benchmark_data_smithi177_31116_object65202:head# 2024-09-13T11:24:30.945 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:42d5cff4:::benchmark_data_smithi177_31116_object15368:head# 2024-09-13T11:24:30.948 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:42d5e481:::benchmark_data_smithi177_31116_object7463:head# 2024-09-13T11:24:30.949 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:42d636d9:::benchmark_data_smithi177_31116_object75281:head# 2024-09-13T11:24:30.949 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:42d64a3c:::benchmark_data_smithi177_31116_object1764:head# 2024-09-13T11:24:30.950 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:42d6e0c7:::benchmark_data_smithi177_31116_object56456:head# 2024-09-13T11:24:30.953 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:42d75da7:::benchmark_data_smithi177_31116_object107664:head# 2024-09-13T11:24:30.956 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:42d761ec:::benchmark_data_smithi177_31116_object93221:head# 2024-09-13T11:24:30.956 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:42d78444:::benchmark_data_smithi177_31116_object36241:head# 2024-09-13T11:24:30.957 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:42d88a63:::benchmark_data_smithi177_31116_object106310:head# 2024-09-13T11:24:30.958 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:42d89b83:::benchmark_data_smithi177_31116_object96726:head# 2024-09-13T11:24:30.959 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:42d90736:::benchmark_data_smithi177_31116_object96403:head# 2024-09-13T11:24:30.959 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:42d941a1:::benchmark_data_smithi177_31116_object14752:head# 2024-09-13T11:24:30.960 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:42d9a4ac:::benchmark_data_smithi177_31116_object79792:head# 2024-09-13T11:24:30.961 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:42da151b:::benchmark_data_smithi177_31116_object36077:head# 2024-09-13T11:24:30.961 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:42dac7ab:::benchmark_data_smithi177_31116_object50720:head# 2024-09-13T11:24:30.964 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:42db4971:::benchmark_data_smithi177_31116_object37310:head# 2024-09-13T11:24:30.965 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:42dc17e4:::benchmark_data_smithi177_31116_object57970:head# 2024-09-13T11:24:30.966 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:42dc1d0a:::benchmark_data_smithi177_31116_object33898:head# 2024-09-13T11:24:30.966 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:42dcb0da:::benchmark_data_smithi177_31116_object42107:head# 2024-09-13T11:24:30.967 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:42dd6e38:::benchmark_data_smithi177_31116_object29102:head# 2024-09-13T11:24:30.967 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:42dde282:::benchmark_data_smithi177_31116_object36135:head# 2024-09-13T11:24:30.968 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:42ddffee:::benchmark_data_smithi177_31116_object9605:head# 2024-09-13T11:24:30.968 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:42de6fa5:::benchmark_data_smithi177_31116_object13286:head# 2024-09-13T11:24:30.969 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:42e141b9:::benchmark_data_smithi177_31116_object63829:head# 2024-09-13T11:24:30.970 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:42e1b56c:::benchmark_data_smithi177_31116_object9754:head# 2024-09-13T11:24:30.973 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:42e369d4:::benchmark_data_smithi177_31116_object104637:head# 2024-09-13T11:24:30.975 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:42e40a83:::benchmark_data_smithi177_31116_object69434:head# 2024-09-13T11:24:30.975 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:42e49e32:::benchmark_data_smithi177_31116_object37172:head# 2024-09-13T11:24:30.976 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:42e4b5ed:::benchmark_data_smithi177_31116_object70729:head# 2024-09-13T11:24:30.977 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:42e4e644:::benchmark_data_smithi177_31116_object79479:head# 2024-09-13T11:24:30.980 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:42e4e98a:::benchmark_data_smithi177_31116_object79377:head# 2024-09-13T11:24:30.981 INFO:tasks.ceph.osd.7.smithi177.stderr:2024-09-13T11:24:30.979+0000 7f897b51d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26575) UID: 0 2024-09-13T11:24:30.983 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:42e53077:::benchmark_data_smithi177_31116_object105970:head# 2024-09-13T11:24:30.984 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:42e556ef:::benchmark_data_smithi177_31116_object19957:head# 2024-09-13T11:24:30.984 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:42e5ce54:::benchmark_data_smithi177_31116_object19365:head# 2024-09-13T11:24:30.986 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:42e5d717:::benchmark_data_smithi177_31116_object18891:head# 2024-09-13T11:24:30.987 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:42e7846e:::benchmark_data_smithi177_31116_object31353:head# 2024-09-13T11:24:30.987 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:42e90fb4:::benchmark_data_smithi177_31116_object92087:head# 2024-09-13T11:24:30.988 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:42e971cc:::benchmark_data_smithi177_31116_object63189:head# 2024-09-13T11:24:30.991 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:42ea1f01:::benchmark_data_smithi177_31116_object71448:head# 2024-09-13T11:24:30.991 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:42ebd60c:::benchmark_data_smithi177_31116_object94560:head# 2024-09-13T11:24:30.992 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:42ed65ee:::benchmark_data_smithi177_31116_object9768:head# 2024-09-13T11:24:30.993 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:42edbe2d:::benchmark_data_smithi177_31116_object76716:head# 2024-09-13T11:24:30.996 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:42edc13b:::benchmark_data_smithi177_31116_object84129:head# 2024-09-13T11:24:30.997 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:42ee3562:::benchmark_data_smithi177_31116_object92748:head# 2024-09-13T11:24:30.997 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:42ee3bb5:::benchmark_data_smithi177_31116_object22752:head# 2024-09-13T11:24:30.998 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:42ef18f9:::benchmark_data_smithi177_31116_object108325:head# 2024-09-13T11:24:30.999 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:42ef24f2:::benchmark_data_smithi177_31116_object73424:head# 2024-09-13T11:24:31.000 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:42ef6b74:::benchmark_data_smithi177_31116_object31714:head# 2024-09-13T11:24:31.002 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:42f094c7:::benchmark_data_smithi177_31116_object92577:head# 2024-09-13T11:24:31.002 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:42f10f66:::benchmark_data_smithi177_31116_object80167:head# 2024-09-13T11:24:31.003 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:42f196c0:::benchmark_data_smithi177_31116_object104006:head# 2024-09-13T11:24:31.003 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:42f2768f:::benchmark_data_smithi177_31116_object4991:head# 2024-09-13T11:24:31.004 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:42f2ff2f:::benchmark_data_smithi177_31116_object40718:head# 2024-09-13T11:24:31.004 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:42f3716f:::benchmark_data_smithi177_31116_object82541:head# 2024-09-13T11:24:31.005 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:42f37745:::benchmark_data_smithi177_31116_object47891:head# 2024-09-13T11:24:31.005 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:42f3dcfd:::benchmark_data_smithi177_31116_object59667:head# 2024-09-13T11:24:31.006 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:42f4fa49:::benchmark_data_smithi177_31116_object80120:head# 2024-09-13T11:24:31.006 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:42f53bb0:::benchmark_data_smithi177_31116_object33191:head# 2024-09-13T11:24:31.007 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:42f5ca38:::benchmark_data_smithi177_31116_object3542:head# 2024-09-13T11:24:31.007 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:42f5d502:::benchmark_data_smithi177_31116_object74103:head# 2024-09-13T11:24:31.008 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:42f623c3:::benchmark_data_smithi177_31116_object36116:head# 2024-09-13T11:24:31.009 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:42f6aac8:::benchmark_data_smithi177_31116_object47499:head# 2024-09-13T11:24:31.009 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:42f6cc9d:::benchmark_data_smithi177_31116_object76598:head# 2024-09-13T11:24:31.010 INFO:tasks.radosbench.radosbench.0.smithi177.stdout: 124 16 112575 112559 56.7277 78.5625 0.00253118 0.0176207 2024-09-13T11:24:31.010 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:42f7655e:::benchmark_data_smithi177_31116_object47915:head# 2024-09-13T11:24:31.011 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:42f76a63:::benchmark_data_smithi177_31116_object53210:head# 2024-09-13T11:24:31.011 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:42f7eec1:::benchmark_data_smithi177_31116_object36841:head# 2024-09-13T11:24:31.012 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:42f83055:::benchmark_data_smithi177_31116_object36604:head# 2024-09-13T11:24:31.013 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:42f85eec:::benchmark_data_smithi177_31116_object58565:head# 2024-09-13T11:24:31.013 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:42f86704:::benchmark_data_smithi177_31116_object43063:head# 2024-09-13T11:24:31.014 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:42f8aa49:::benchmark_data_smithi177_31116_object5367:head# 2024-09-13T11:24:31.014 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:42f8b5b3:::benchmark_data_smithi177_31116_object20867:head# 2024-09-13T11:24:31.015 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:42f994b1:::benchmark_data_smithi177_31116_object15231:head# 2024-09-13T11:24:31.016 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:42f997d0:::benchmark_data_smithi177_31116_object55397:head# 2024-09-13T11:24:31.016 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:42fa8cf3:::benchmark_data_smithi177_31116_object72631:head# 2024-09-13T11:24:31.016 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:42fb3744:::benchmark_data_smithi177_31116_object22232:head# 2024-09-13T11:24:31.017 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:42fbcf8a:::benchmark_data_smithi177_31116_object91884:head# 2024-09-13T11:24:31.018 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:42fc62a3:::benchmark_data_smithi177_31116_object74552:head# 2024-09-13T11:24:31.018 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:42fefe9b:::benchmark_data_smithi177_31116_object28146:head# 2024-09-13T11:24:31.019 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:42ffa92b:::benchmark_data_smithi177_31116_object105737:head# 2024-09-13T11:24:31.019 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:430036e9:::benchmark_data_smithi177_31116_object60201:head# 2024-09-13T11:24:31.020 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:43004fea:::benchmark_data_smithi177_31116_object37575:head# 2024-09-13T11:24:31.020 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4300708b:::benchmark_data_smithi177_31116_object28433:head# 2024-09-13T11:24:31.021 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:43008547:::benchmark_data_smithi177_31116_object63725:head# 2024-09-13T11:24:31.022 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:43011291:::benchmark_data_smithi177_31116_object18849:head# 2024-09-13T11:24:31.023 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:43012179:::benchmark_data_smithi177_31116_object88910:head# 2024-09-13T11:24:31.023 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:43021fb7:::benchmark_data_smithi177_31116_object93951:head# 2024-09-13T11:24:31.024 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4302270e:::benchmark_data_smithi177_31116_object14290:head# 2024-09-13T11:24:31.024 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:43032328:::benchmark_data_smithi177_31116_object2099:head# 2024-09-13T11:24:31.025 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:43034929:::benchmark_data_smithi177_31116_object34551:head# 2024-09-13T11:24:31.025 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4305839c:::benchmark_data_smithi177_31116_object105724:head# 2024-09-13T11:24:31.026 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4305def3:::benchmark_data_smithi177_31116_object100750:head# 2024-09-13T11:24:31.027 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4307b1c9:::benchmark_data_smithi177_31116_object93321:head# 2024-09-13T11:24:31.027 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4307fad4:::benchmark_data_smithi177_31116_object94017:head# 2024-09-13T11:24:31.032 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4309d188:::benchmark_data_smithi177_31116_object27155:head# 2024-09-13T11:24:31.032 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:430a4a53:::benchmark_data_smithi177_31116_object22801:head# 2024-09-13T11:24:31.032 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:430b88d7:::benchmark_data_smithi177_31116_object39048:head# 2024-09-13T11:24:31.032 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:430c7cf2:::benchmark_data_smithi177_31116_object68477:head# 2024-09-13T11:24:31.033 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:430d7a12:::benchmark_data_smithi177_31116_object96572:head# 2024-09-13T11:24:31.033 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:430ece2a:::benchmark_data_smithi177_31116_object75168:head# 2024-09-13T11:24:31.033 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:430f68cd:::benchmark_data_smithi177_31116_object51903:head# 2024-09-13T11:24:31.033 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:43106425:::benchmark_data_smithi177_31116_object66867:head# 2024-09-13T11:24:31.033 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4311244f:::benchmark_data_smithi177_31116_object46847:head# 2024-09-13T11:24:31.034 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:43121685:::benchmark_data_smithi177_31116_object94676:head# 2024-09-13T11:24:31.035 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:43128283:::benchmark_data_smithi177_31116_object39313:head# 2024-09-13T11:24:31.035 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:431314ab:::benchmark_data_smithi177_31116_object87462:head# 2024-09-13T11:24:31.036 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4314ed28:::benchmark_data_smithi177_31116_object54355:head# 2024-09-13T11:24:31.036 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:43158bc7:::benchmark_data_smithi177_31116_object85185:head# 2024-09-13T11:24:31.037 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:43158c8c:::benchmark_data_smithi177_31116_object35392:head# 2024-09-13T11:24:31.037 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:431596eb:::benchmark_data_smithi177_31116_object38714:head# 2024-09-13T11:24:31.038 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4316ef98:::benchmark_data_smithi177_31116_object23071:head# 2024-09-13T11:24:31.038 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:43171e7f:::benchmark_data_smithi177_31116_object108594:head# 2024-09-13T11:24:31.039 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-13T11:24:31.039 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:24:31.039 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:43178492:::benchmark_data_smithi177_31116_object824:head# 2024-09-13T11:24:31.040 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:431846cf:::benchmark_data_smithi177_31116_object17112:head# 2024-09-13T11:24:31.040 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:43198907:::benchmark_data_smithi177_31116_object51459:head# 2024-09-13T11:24:31.041 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:431a62d8:::benchmark_data_smithi177_31116_object26990:head# 2024-09-13T11:24:31.042 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:431c27b1:::benchmark_data_smithi177_31116_object11962:head# 2024-09-13T11:24:31.042 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:431c4cab:::benchmark_data_smithi177_31116_object32586:head# 2024-09-13T11:24:31.043 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:431cbe65:::benchmark_data_smithi177_31116_object86063:head# 2024-09-13T11:24:31.043 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:431d81f7:::benchmark_data_smithi177_31116_object70230:head# 2024-09-13T11:24:31.044 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:431e6c3f:::benchmark_data_smithi177_31116_object62167:head# 2024-09-13T11:24:31.044 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:431e8740:::benchmark_data_smithi177_31116_object32758:head# 2024-09-13T11:24:31.044 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:431ef103:::benchmark_data_smithi177_31116_object72467:head# 2024-09-13T11:24:31.045 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:43203930:::benchmark_data_smithi177_31116_object60419:head# 2024-09-13T11:24:31.046 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4320c89b:::benchmark_data_smithi177_31116_object77792:head# 2024-09-13T11:24:31.046 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4320fa07:::benchmark_data_smithi177_31116_object92727:head# 2024-09-13T11:24:31.047 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:43221ee2:::benchmark_data_smithi177_31116_object91172:head# 2024-09-13T11:24:31.047 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4323045c:::benchmark_data_smithi177_31116_object4582:head# 2024-09-13T11:24:31.048 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4323969f:::benchmark_data_smithi177_31116_object53506:head# 2024-09-13T11:24:31.048 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:43250fc3:::benchmark_data_smithi177_31116_object56570:head# 2024-09-13T11:24:31.049 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:43256736:::benchmark_data_smithi177_31116_object41353:head# 2024-09-13T11:24:31.049 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:432585ea:::benchmark_data_smithi177_31116_object41469:head# 2024-09-13T11:24:31.050 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4325bd7d:::benchmark_data_smithi177_31116_object57379:head# 2024-09-13T11:24:31.050 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4325e03c:::benchmark_data_smithi177_31116_object82276:head# 2024-09-13T11:24:31.051 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:43268ec1:::benchmark_data_smithi177_31116_object936:head# 2024-09-13T11:24:31.052 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:43274b31:::benchmark_data_smithi177_31116_object9381:head# 2024-09-13T11:24:31.052 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4327ec4b:::benchmark_data_smithi177_31116_object38924:head# 2024-09-13T11:24:31.053 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:43293193:::benchmark_data_smithi177_31116_object107558:head# 2024-09-13T11:24:31.053 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:43294bbb:::benchmark_data_smithi177_31116_object5659:head# 2024-09-13T11:24:31.054 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:432b38aa:::benchmark_data_smithi177_31116_object33829:head# 2024-09-13T11:24:31.054 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:432c3c4f:::benchmark_data_smithi177_31116_object8502:head# 2024-09-13T11:24:31.056 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:432cf2a0:::benchmark_data_smithi177_31116_object23399:head# 2024-09-13T11:24:31.056 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:432d05b4:::benchmark_data_smithi177_31116_object83186:head# 2024-09-13T11:24:31.056 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:432d44fa:::benchmark_data_smithi177_31116_object90866:head# 2024-09-13T11:24:31.057 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:432db8c8:::benchmark_data_smithi177_31116_object9570:head# 2024-09-13T11:24:31.057 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:432dc3c8:::benchmark_data_smithi177_31116_object24724:head# 2024-09-13T11:24:31.058 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:432f700b:::benchmark_data_smithi177_31116_object58617:head# 2024-09-13T11:24:31.058 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:43310c67:::benchmark_data_smithi177_31116_object33731:head# 2024-09-13T11:24:31.059 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-13T11:24:31.061 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4331c1e2:::benchmark_data_smithi177_31116_object91328:head# 2024-09-13T11:24:31.061 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4332510c:::benchmark_data_smithi177_31116_object45621:head# 2024-09-13T11:24:31.061 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4332f723:::benchmark_data_smithi177_31116_object72070:head# 2024-09-13T11:24:31.061 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:43342ae7:::benchmark_data_smithi177_31116_object6179:head# 2024-09-13T11:24:31.062 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4334ebb7:::benchmark_data_smithi177_31116_object37851:head# 2024-09-13T11:24:31.062 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:43371ae5:::benchmark_data_smithi177_31116_object33622:head# 2024-09-13T11:24:31.063 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4338d445:::benchmark_data_smithi177_31116_object46285:head# 2024-09-13T11:24:31.063 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:43394c8b:::benchmark_data_smithi177_31116_object49774:head# 2024-09-13T11:24:31.064 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4339b550:::benchmark_data_smithi177_31116_object1244:head# 2024-09-13T11:24:31.064 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4339c71c:::benchmark_data_smithi177_31116_object97809:head# 2024-09-13T11:24:31.065 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:433aa750:::benchmark_data_smithi177_31116_object70992:head# 2024-09-13T11:24:31.065 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:433b5f04:::benchmark_data_smithi177_31116_object26476:head# 2024-09-13T11:24:31.065 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:433be3d6:::benchmark_data_smithi177_31116_object88873:head# 2024-09-13T11:24:31.066 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:433e8c47:::benchmark_data_smithi177_31116_object13544:head# 2024-09-13T11:24:31.066 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:433f0684:::benchmark_data_smithi177_31116_object53295:head# 2024-09-13T11:24:31.067 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:433f5220:::benchmark_data_smithi177_31116_object71679:head# 2024-09-13T11:24:31.067 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:433fde72:::benchmark_data_smithi177_31116_object34601:head# 2024-09-13T11:24:31.071 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:434063ee:::benchmark_data_smithi177_31116_object72323:head# 2024-09-13T11:24:31.071 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:43409335:::benchmark_data_smithi177_31116_object21739:head# 2024-09-13T11:24:31.072 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4340b5a2:::benchmark_data_smithi177_31116_object104660:head# 2024-09-13T11:24:31.073 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4340bf15:::benchmark_data_smithi177_31116_object5283:head# 2024-09-13T11:24:31.073 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4340c8e8:::benchmark_data_smithi177_31116_object77912:head# 2024-09-13T11:24:31.076 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:434166c0:::benchmark_data_smithi177_31116_object71474:head# 2024-09-13T11:24:31.076 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4341f8d7:::benchmark_data_smithi177_31116_object15165:head# 2024-09-13T11:24:31.077 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:43428a82:::benchmark_data_smithi177_31116_object20018:head# 2024-09-13T11:24:31.078 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:434359e2:::benchmark_data_smithi177_31116_object65619:head# 2024-09-13T11:24:31.078 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:43439d13:::benchmark_data_smithi177_31116_object68284:head# 2024-09-13T11:24:31.081 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4343bfcd:::benchmark_data_smithi177_31116_object80330:head# 2024-09-13T11:24:31.081 INFO:tasks.ceph.osd.10.smithi162.stderr:2024-09-13T11:24:31.076+0000 7fb59c9f0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26317) UID: 0 2024-09-13T11:24:31.082 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4343e435:::benchmark_data_smithi177_31116_object101381:head# 2024-09-13T11:24:31.082 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4344468d:::benchmark_data_smithi177_31116_object93736:head# 2024-09-13T11:24:31.083 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4344e251:::benchmark_data_smithi177_31116_object13232:head# 2024-09-13T11:24:31.083 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:434630c3:::benchmark_data_smithi177_31116_object73789:head# 2024-09-13T11:24:31.083 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:43466497:::benchmark_data_smithi177_31116_object72203:head# 2024-09-13T11:24:31.084 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:43469264:::benchmark_data_smithi177_31116_object99865:head# 2024-09-13T11:24:31.085 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:43479640:::benchmark_data_smithi177_31116_object8858:head# 2024-09-13T11:24:31.087 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4347cb43:::benchmark_data_smithi177_31116_object49738:head# 2024-09-13T11:24:31.087 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:43482af6:::benchmark_data_smithi177_31116_object70352:head# 2024-09-13T11:24:31.088 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:43485327:::benchmark_data_smithi177_31116_object85265:head# 2024-09-13T11:24:31.088 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4348b3f7:::benchmark_data_smithi177_31116_object52142:head# 2024-09-13T11:24:31.088 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:43494cfe:::benchmark_data_smithi177_31116_object96610:head# 2024-09-13T11:24:31.089 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:43499e2a:::benchmark_data_smithi177_31116_object78545:head# 2024-09-13T11:24:31.089 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:434a30fc:::benchmark_data_smithi177_31116_object35067:head# 2024-09-13T11:24:31.090 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:434a4458:::benchmark_data_smithi177_31116_object97686:head# 2024-09-13T11:24:31.091 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:434a6889:::benchmark_data_smithi177_31116_object57144:head# 2024-09-13T11:24:31.091 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:434a6de0:::benchmark_data_smithi177_31116_object59116:head# 2024-09-13T11:24:31.092 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:434ac47b:::benchmark_data_smithi177_31116_object8021:head# 2024-09-13T11:24:31.092 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:434b7c7b:::benchmark_data_smithi177_31116_object67539:head# 2024-09-13T11:24:31.093 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:434b84ae:::benchmark_data_smithi177_31116_object86163:head# 2024-09-13T11:24:31.093 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:434bc89f:::benchmark_data_smithi177_31116_object70632:head# 2024-09-13T11:24:31.094 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:434bf54d:::benchmark_data_smithi177_31116_object7102:head# 2024-09-13T11:24:31.095 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:434d158a:::benchmark_data_smithi177_31116_object109091:head# 2024-09-13T11:24:31.098 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:434d9d7c:::benchmark_data_smithi177_31116_object30424:head# 2024-09-13T11:24:31.098 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:434e8463:::benchmark_data_smithi177_31116_object108341:head# 2024-09-13T11:24:31.099 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:434f5eaa:::benchmark_data_smithi177_31116_object16071:head# 2024-09-13T11:24:31.099 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:434fc2e0:::benchmark_data_smithi177_31116_object16738:head# 2024-09-13T11:24:31.099 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4350d029:::benchmark_data_smithi177_31116_object75941:head# 2024-09-13T11:24:31.099 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4350ea1a:::benchmark_data_smithi177_31116_object99148:head# 2024-09-13T11:24:31.100 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:435178db:::benchmark_data_smithi177_31116_object101992:head# 2024-09-13T11:24:31.100 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4351f3cf:::benchmark_data_smithi177_31116_object93593:head# 2024-09-13T11:24:31.101 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:435362d9:::benchmark_data_smithi177_31116_object102665:head# 2024-09-13T11:24:31.102 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4353aa83:::benchmark_data_smithi177_31116_object100003:head# 2024-09-13T11:24:31.102 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4353b781:::benchmark_data_smithi177_31116_object81053:head# 2024-09-13T11:24:31.103 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:43574008:::benchmark_data_smithi177_31116_object12605:head# 2024-09-13T11:24:31.104 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:435982ce:::benchmark_data_smithi177_31116_object65663:head# 2024-09-13T11:24:31.105 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4359f3aa:::benchmark_data_smithi177_31116_object58953:head# 2024-09-13T11:24:31.106 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:435b4d48:::benchmark_data_smithi177_31116_object97715:head# 2024-09-13T11:24:31.107 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:435b90fa:::benchmark_data_smithi177_31116_object86207:head# 2024-09-13T11:24:31.107 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:435c396c:::benchmark_data_smithi177_31116_object48823:head# 2024-09-13T11:24:31.110 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:435cf91e:::benchmark_data_smithi177_31116_object59120:head# 2024-09-13T11:24:31.111 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:435d24b6:::benchmark_data_smithi177_31116_object32154:head# 2024-09-13T11:24:31.111 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:435d89e9:::benchmark_data_smithi177_31116_object7864:head# 2024-09-13T11:24:31.112 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:435dab7e:::benchmark_data_smithi177_31116_object101404:head# 2024-09-13T11:24:31.113 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:435e0763:::benchmark_data_smithi177_31116_object110247:head# 2024-09-13T11:24:31.116 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:435ea0b7:::benchmark_data_smithi177_31116_object89537:head# 2024-09-13T11:24:31.117 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:435f51cc:::benchmark_data_smithi177_31116_object52504:head# 2024-09-13T11:24:31.117 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:435f74f5:::benchmark_data_smithi177_31116_object108828:head# 2024-09-13T11:24:31.118 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:435fbe24:::benchmark_data_smithi177_31116_object2473:head# 2024-09-13T11:24:31.119 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4360072b:::benchmark_data_smithi177_31116_object98345:head# 2024-09-13T11:24:31.119 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:43601390:::benchmark_data_smithi177_31116_object90734:head# 2024-09-13T11:24:31.119 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-13T11:24:31.120 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:43602106:::benchmark_data_smithi177_31116_object13944:head# 2024-09-13T11:24:31.120 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:43607d68:::benchmark_data_smithi177_31116_object3384:head# 2024-09-13T11:24:31.121 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4360df48:::benchmark_data_smithi177_31116_object34842:head# 2024-09-13T11:24:31.127 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:43614758:::benchmark_data_smithi177_31116_object66235:head# 2024-09-13T11:24:31.127 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4361ae8d:::benchmark_data_smithi177_31116_object15946:head# 2024-09-13T11:24:31.127 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4361f7b1:::benchmark_data_smithi177_31116_object13180:head# 2024-09-13T11:24:31.127 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4362ff83:::benchmark_data_smithi177_31116_object31429:head# 2024-09-13T11:24:31.127 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:43642120:::benchmark_data_smithi177_31116_object106481:head# 2024-09-13T11:24:31.127 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:43650941:::benchmark_data_smithi177_31116_object73574:head# 2024-09-13T11:24:31.127 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:43659226:::benchmark_data_smithi177_31116_object70332:head# 2024-09-13T11:24:31.127 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:43667820:::benchmark_data_smithi177_31116_object102134:head# 2024-09-13T11:24:31.128 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4366ea87:::benchmark_data_smithi177_31116_object27557:head# 2024-09-13T11:24:31.129 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:43671663:::benchmark_data_smithi177_31116_object29167:head# 2024-09-13T11:24:31.129 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:43671d99:::benchmark_data_smithi177_31116_object43704:head# 2024-09-13T11:24:31.130 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:43672cca:::benchmark_data_smithi177_31116_object91030:head# 2024-09-13T11:24:31.130 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4367656f:::benchmark_data_smithi177_31116_object20424:head# 2024-09-13T11:24:31.133 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:436a4221:::benchmark_data_smithi177_31116_object9061:head# 2024-09-13T11:24:31.134 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:436b2a54:::benchmark_data_smithi177_31116_object104811:head# 2024-09-13T11:24:31.135 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:436b65f1:::benchmark_data_smithi177_31116_object36227:head# 2024-09-13T11:24:31.136 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:436c5fc1:::benchmark_data_smithi177_31116_object62390:head# 2024-09-13T11:24:31.138 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:436d113e:::benchmark_data_smithi177_31116_object52068:head# 2024-09-13T11:24:31.139 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:436d5227:::benchmark_data_smithi177_31116_object97216:head# 2024-09-13T11:24:31.139 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:436d929b:::benchmark_data_smithi177_31116_object17883:head# 2024-09-13T11:24:31.143 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:436e2500:::benchmark_data_smithi177_31116_object109441:head# 2024-09-13T11:24:31.143 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:436e5978:::benchmark_data_smithi177_31116_object50368:head# 2024-09-13T11:24:31.143 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:436f4bd8:::benchmark_data_smithi177_31116_object30038:head# 2024-09-13T11:24:31.143 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:436f7829:::benchmark_data_smithi177_31116_object91242:head# 2024-09-13T11:24:31.144 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:43703c0a:::benchmark_data_smithi177_31116_object87319:head# 2024-09-13T11:24:31.144 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:43725684:::benchmark_data_smithi177_31116_object7475:head# 2024-09-13T11:24:31.145 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4372e123:::benchmark_data_smithi177_31116_object104842:head# 2024-09-13T11:24:31.146 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:43732f94:::benchmark_data_smithi177_31116_object46237:head# 2024-09-13T11:24:31.146 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:43743406:::benchmark_data_smithi177_31116_object51109:head# 2024-09-13T11:24:31.148 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:43744c2d:::benchmark_data_smithi177_31116_object40566:head# 2024-09-13T11:24:31.148 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:43749f28:::benchmark_data_smithi177_31116_object63322:head# 2024-09-13T11:24:31.149 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:43767c05:::benchmark_data_smithi177_31116_object17995:head# 2024-09-13T11:24:31.149 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4378241d:::benchmark_data_smithi177_31116_object86089:head# 2024-09-13T11:24:31.149 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:437a7a19:::benchmark_data_smithi177_31116_object56673:head# 2024-09-13T11:24:31.150 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:437b48b9:::benchmark_data_smithi177_31116_object97846:head# 2024-09-13T11:24:31.150 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:437b9c91:::benchmark_data_smithi177_31116_object5658:head# 2024-09-13T11:24:31.151 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:437bb958:::benchmark_data_smithi177_31116_object67463:head# 2024-09-13T11:24:31.152 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:437c8b9b:::benchmark_data_smithi177_31116_object45652:head# 2024-09-13T11:24:31.152 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:437d1506:::benchmark_data_smithi177_31116_object62056:head# 2024-09-13T11:24:31.153 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:437d22b9:::benchmark_data_smithi177_31116_object39896:head# 2024-09-13T11:24:31.153 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:437dd06b:::benchmark_data_smithi177_31116_object22750:head# 2024-09-13T11:24:31.154 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:437eb422:::benchmark_data_smithi177_31116_object97189:head# 2024-09-13T11:24:31.156 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:437ee1c6:::benchmark_data_smithi177_31116_object98906:head# 2024-09-13T11:24:31.156 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:437f1c4f:::benchmark_data_smithi177_31116_object64220:head# 2024-09-13T11:24:31.157 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:437fe7fb:::benchmark_data_smithi177_31116_object44470:head# 2024-09-13T11:24:31.157 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:438018b6:::benchmark_data_smithi177_31116_object102549:head# 2024-09-13T11:24:31.159 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4380650b:::benchmark_data_smithi177_31116_object7623:head# 2024-09-13T11:24:31.159 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4381dda7:::benchmark_data_smithi177_31116_object65968:head# 2024-09-13T11:24:31.159 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4383a46b:::benchmark_data_smithi177_31116_object52480:head# 2024-09-13T11:24:31.160 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:43840349:::benchmark_data_smithi177_31116_object35233:head# 2024-09-13T11:24:31.160 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T11:24:31.161 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:43853459:::benchmark_data_smithi177_31116_object13457:head# 2024-09-13T11:24:31.161 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4385c169:::benchmark_data_smithi177_31116_object51650:head# 2024-09-13T11:24:31.161 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4385f551:::benchmark_data_smithi177_31116_object24304:head# 2024-09-13T11:24:31.162 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:43861257:::benchmark_data_smithi177_31116_object20120:head# 2024-09-13T11:24:31.162 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4386930f:::benchmark_data_smithi177_31116_object89332:head# 2024-09-13T11:24:31.163 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:43871f9f:::benchmark_data_smithi177_31116_object6812:head# 2024-09-13T11:24:31.163 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:43881191:::benchmark_data_smithi177_31116_object18747:head# 2024-09-13T11:24:31.164 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:43887f49:::benchmark_data_smithi177_31116_object15739:head# 2024-09-13T11:24:31.164 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4388b472:::benchmark_data_smithi177_31116_object59707:head# 2024-09-13T11:24:31.165 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4388e209:::benchmark_data_smithi177_31116_object96007:head# 2024-09-13T11:24:31.165 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4388fe51:::benchmark_data_smithi177_31116_object39873:head# 2024-09-13T11:24:31.166 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:43896ea2:::benchmark_data_smithi177_31116_object55963:head# 2024-09-13T11:24:31.166 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:438973a8:::benchmark_data_smithi177_31116_object32486:head# 2024-09-13T11:24:31.167 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4389fd20:::benchmark_data_smithi177_31116_object72738:head# 2024-09-13T11:24:31.167 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:438a8697:::benchmark_data_smithi177_31116_object108573:head# 2024-09-13T11:24:31.168 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:438ac691:::benchmark_data_smithi177_31116_object38962:head# 2024-09-13T11:24:31.169 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:438ada23:::benchmark_data_smithi177_31116_object1:head# 2024-09-13T11:24:31.169 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:438b8d19:::benchmark_data_smithi177_31116_object38994:head# 2024-09-13T11:24:31.170 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:438ba540:::benchmark_data_smithi177_31116_object35929:head# 2024-09-13T11:24:31.170 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:438dddcf:::benchmark_data_smithi177_31116_object66560:head# 2024-09-13T11:24:31.171 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:438dfad8:::benchmark_data_smithi177_31116_object4008:head# 2024-09-13T11:24:31.171 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:438ec54b:::benchmark_data_smithi177_31116_object108903:head# 2024-09-13T11:24:31.172 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:438f0723:::benchmark_data_smithi177_31116_object94305:head# 2024-09-13T11:24:31.172 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:438f4594:::benchmark_data_smithi177_31116_object33647:head# 2024-09-13T11:24:31.173 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:438f94fc:::benchmark_data_smithi177_31116_object107070:head# 2024-09-13T11:24:31.174 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:438ffbb1:::benchmark_data_smithi177_31116_object101851:head# 2024-09-13T11:24:31.177 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:439084eb:::benchmark_data_smithi177_31116_object59103:head# 2024-09-13T11:24:31.177 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4390eee7:::benchmark_data_smithi177_31116_object90595:head# 2024-09-13T11:24:31.178 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:43915ddf:::benchmark_data_smithi177_31116_object10899:head# 2024-09-13T11:24:31.178 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4391c046:::benchmark_data_smithi177_31116_object100161:head# 2024-09-13T11:24:31.178 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:43921c4c:::benchmark_data_smithi177_31116_object24036:head# 2024-09-13T11:24:31.178 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:43922688:::benchmark_data_smithi177_31116_object17775:head# 2024-09-13T11:24:31.179 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:43923e9e:::benchmark_data_smithi177_31116_object99885:head# 2024-09-13T11:24:31.180 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4392716c:::benchmark_data_smithi177_31116_object86558:head# 2024-09-13T11:24:31.180 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4392c86b:::benchmark_data_smithi177_31116_object85466:head# 2024-09-13T11:24:31.180 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:43934610:::benchmark_data_smithi177_31116_object52421:head# 2024-09-13T11:24:31.181 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4394b83a:::benchmark_data_smithi177_31116_object72731:head# 2024-09-13T11:24:31.182 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4394f21a:::benchmark_data_smithi177_31116_object50822:head# 2024-09-13T11:24:31.182 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4395698b:::benchmark_data_smithi177_31116_object27397:head# 2024-09-13T11:24:31.183 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4395766c:::benchmark_data_smithi177_31116_object18538:head# 2024-09-13T11:24:31.183 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4396221a:::benchmark_data_smithi177_31116_object106959:head# 2024-09-13T11:24:31.184 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4396aa6d:::benchmark_data_smithi177_31116_object15570:head# 2024-09-13T11:24:31.185 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:43971e15:::benchmark_data_smithi177_31116_object105354:head# 2024-09-13T11:24:31.185 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:43990bce:::benchmark_data_smithi177_31116_object98250:head# 2024-09-13T11:24:31.186 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:439a00a5:::benchmark_data_smithi177_31116_object59038:head# 2024-09-13T11:24:31.188 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:439ab391:::benchmark_data_smithi177_31116_object41147:head# 2024-09-13T11:24:31.189 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:439ce266:::benchmark_data_smithi177_31116_object28555:head# 2024-09-13T11:24:31.189 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:439e520f:::benchmark_data_smithi177_31116_object68942:head# 2024-09-13T11:24:31.190 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:439f8aaf:::benchmark_data_smithi177_31116_object15140:head# 2024-09-13T11:24:31.190 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:439f9c74:::benchmark_data_smithi177_31116_object38702:head# 2024-09-13T11:24:31.191 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:43a07f9e:::benchmark_data_smithi177_31116_object77133:head# 2024-09-13T11:24:31.191 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:43a165d3:::benchmark_data_smithi177_31116_object27150:head# 2024-09-13T11:24:31.193 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:43a22b33:::benchmark_data_smithi177_31116_object91179:head# 2024-09-13T11:24:31.193 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:43a2beee:::benchmark_data_smithi177_31116_object9042:head# 2024-09-13T11:24:31.193 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:43a305d6:::benchmark_data_smithi177_31116_object53782:head# 2024-09-13T11:24:31.194 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:43a374b6:::benchmark_data_smithi177_31116_object105791:head# 2024-09-13T11:24:31.194 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:43a555e9:::benchmark_data_smithi177_31116_object92137:head# 2024-09-13T11:24:31.195 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:43a5872d:::benchmark_data_smithi177_31116_object74997:head# 2024-09-13T11:24:31.197 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:43a60577:::benchmark_data_smithi177_31116_object53989:head# 2024-09-13T11:24:31.197 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:43a60865:::benchmark_data_smithi177_31116_object79671:head# 2024-09-13T11:24:31.197 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:43a662a3:::benchmark_data_smithi177_31116_object30901:head# 2024-09-13T11:24:31.197 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:43a6a01a:::benchmark_data_smithi177_31116_object108829:head# 2024-09-13T11:24:31.198 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:43a6bf6b:::benchmark_data_smithi177_31116_object3872:head# 2024-09-13T11:24:31.198 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:43a71a5e:::benchmark_data_smithi177_31116_object40092:head# 2024-09-13T11:24:31.198 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:43a7317f:::benchmark_data_smithi177_31116_object40079:head# 2024-09-13T11:24:31.199 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:43a7c76c:::benchmark_data_smithi177_31116_object41820:head# 2024-09-13T11:24:31.199 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:43a81c8c:::benchmark_data_smithi177_31116_object108845:head# 2024-09-13T11:24:31.200 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:43a88fdd:::benchmark_data_smithi177_31116_object2169:head# 2024-09-13T11:24:31.202 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:43a8eda1:::benchmark_data_smithi177_31116_object34683:head# 2024-09-13T11:24:31.202 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:43a9a1d9:::benchmark_data_smithi177_31116_object89735:head# 2024-09-13T11:24:31.202 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:43ab7c37:::benchmark_data_smithi177_31116_object99472:head# 2024-09-13T11:24:31.202 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:43ab97a4:::benchmark_data_smithi177_31116_object64118:head# 2024-09-13T11:24:31.203 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:43ac59b9:::benchmark_data_smithi177_31116_object57830:head# 2024-09-13T11:24:31.203 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:43acaf57:::benchmark_data_smithi177_31116_object108277:head# 2024-09-13T11:24:31.204 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:43acf54b:::benchmark_data_smithi177_31116_object79828:head# 2024-09-13T11:24:31.205 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:43ad10d0:::benchmark_data_smithi177_31116_object33546:head# 2024-09-13T11:24:31.206 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:43ad1963:::benchmark_data_smithi177_31116_object45262:head# 2024-09-13T11:24:31.206 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:43ad6af7:::benchmark_data_smithi177_31116_object50858:head# 2024-09-13T11:24:31.207 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:43ae1be0:::benchmark_data_smithi177_31116_object101206:head# 2024-09-13T11:24:31.207 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:43ae24e7:::benchmark_data_smithi177_31116_object35137:head# 2024-09-13T11:24:31.208 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:43af4de5:::benchmark_data_smithi177_31116_object76207:head# 2024-09-13T11:24:31.208 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:43af9fa7:::benchmark_data_smithi177_31116_object80406:head# 2024-09-13T11:24:31.209 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-13T11:24:31.209 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:24:31.209 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:43b05f2d:::benchmark_data_smithi177_31116_object21140:head# 2024-09-13T11:24:31.209 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:43b0ab59:::benchmark_data_smithi177_31116_object3322:head# 2024-09-13T11:24:31.210 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:43b0b2bc:::benchmark_data_smithi177_31116_object103099:head# 2024-09-13T11:24:31.211 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:43b0b38b:::benchmark_data_smithi177_31116_object68578:head# 2024-09-13T11:24:31.212 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:43b18170:::benchmark_data_smithi177_31116_object24730:head# 2024-09-13T11:24:31.212 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:43b1bd47:::benchmark_data_smithi177_31116_object68373:head# 2024-09-13T11:24:31.212 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:43b39ebf:::benchmark_data_smithi177_31116_object9088:head# 2024-09-13T11:24:31.213 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:43b44739:::benchmark_data_smithi177_31116_object60684:head# 2024-09-13T11:24:31.213 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:43b53323:::benchmark_data_smithi177_31116_object26229:head# 2024-09-13T11:24:31.216 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:43b578b8:::benchmark_data_smithi177_31116_object37320:head# 2024-09-13T11:24:31.216 INFO:tasks.ceph.osd.4.smithi001.stderr:2024-09-13T11:24:31.215+0000 7ff8e0218640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26310) UID: 0 2024-09-13T11:24:31.216 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:43b579fe:::benchmark_data_smithi177_31116_object85291:head# 2024-09-13T11:24:31.217 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:43b5a163:::benchmark_data_smithi177_31116_object108408:head# 2024-09-13T11:24:31.218 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:43b5f242:::benchmark_data_smithi177_31116_object44711:head# 2024-09-13T11:24:31.218 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:43b62495:::benchmark_data_smithi177_31116_object53031:head# 2024-09-13T11:24:31.218 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:43b7c8fe:::benchmark_data_smithi177_31116_object66853:head# 2024-09-13T11:24:31.219 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:43b82953:::benchmark_data_smithi177_31116_object26959:head# 2024-09-13T11:24:31.220 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:43b97330:::benchmark_data_smithi177_31116_object80350:head# 2024-09-13T11:24:31.221 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:43ba11b6:::benchmark_data_smithi177_31116_object1475:head# 2024-09-13T11:24:31.221 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:43baab7d:::benchmark_data_smithi177_31116_object74578:head# 2024-09-13T11:24:31.224 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:43bac3c6:::benchmark_data_smithi177_31116_object70033:head# 2024-09-13T11:24:31.225 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:43bbd963:::benchmark_data_smithi177_31116_object96863:head# 2024-09-13T11:24:31.226 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:43bc03c9:::benchmark_data_smithi177_31116_object3718:head# 2024-09-13T11:24:31.226 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:43bc1d5c:::benchmark_data_smithi177_31116_object1072:head# 2024-09-13T11:24:31.226 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:43bc2234:::benchmark_data_smithi177_31116_object81145:head# 2024-09-13T11:24:31.227 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:43bcad49:::benchmark_data_smithi177_31116_object65296:head# 2024-09-13T11:24:31.228 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-13T11:24:31.230 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:43bcd1c3:::benchmark_data_smithi177_31116_object18190:head# 2024-09-13T11:24:31.230 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:43bd42e7:::benchmark_data_smithi177_31116_object103456:head# 2024-09-13T11:24:31.230 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:43bd7231:::benchmark_data_smithi177_31116_object31068:head# 2024-09-13T11:24:31.231 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:43be6e36:::benchmark_data_smithi177_31116_object88034:head# 2024-09-13T11:24:31.231 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:43c07ae3:::benchmark_data_smithi177_31116_object25368:head# 2024-09-13T11:24:31.231 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:43c11835:::benchmark_data_smithi177_31116_object91847:head# 2024-09-13T11:24:31.231 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:43c12527:::benchmark_data_smithi177_31116_object100311:head# 2024-09-13T11:24:31.232 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:43c33125:::benchmark_data_smithi177_31116_object16352:head# 2024-09-13T11:24:31.236 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:43c387b7:::benchmark_data_smithi177_31116_object10334:head# 2024-09-13T11:24:31.238 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:43c3c4df:::benchmark_data_smithi177_31116_object30788:head# 2024-09-13T11:24:31.239 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:43c3e052:::benchmark_data_smithi177_31116_object13238:head# 2024-09-13T11:24:31.239 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:43c3fff5:::benchmark_data_smithi177_31116_object50379:head# 2024-09-13T11:24:31.240 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:43c4f4e3:::benchmark_data_smithi177_31116_object62519:head# 2024-09-13T11:24:31.240 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:43c50776:::benchmark_data_smithi177_31116_object65358:head# 2024-09-13T11:24:31.241 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:43c55ca4:::benchmark_data_smithi177_31116_object104004:head# 2024-09-13T11:24:31.241 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:43c5a517:::benchmark_data_smithi177_31116_object57904:head# 2024-09-13T11:24:31.242 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:43c6b698:::benchmark_data_smithi177_31116_object3371:head# 2024-09-13T11:24:31.242 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:43c71aaa:::benchmark_data_smithi177_31116_object63962:head# 2024-09-13T11:24:31.243 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:43c73c36:::benchmark_data_smithi177_31116_object31734:head# 2024-09-13T11:24:31.243 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:43c7483c:::benchmark_data_smithi177_31116_object29712:head# 2024-09-13T11:24:31.244 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:43c90919:::benchmark_data_smithi177_31116_object60435:head# 2024-09-13T11:24:31.245 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:43cbc345:::benchmark_data_smithi177_31116_object78488:head# 2024-09-13T11:24:31.247 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:43cc7b32:::benchmark_data_smithi177_31116_object41231:head# 2024-09-13T11:24:31.250 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:43ccd630:::benchmark_data_smithi177_31116_object97462:head# 2024-09-13T11:24:31.253 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:43cd1520:::benchmark_data_smithi177_31116_object92364:head# 2024-09-13T11:24:31.256 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:43cd6382:::benchmark_data_smithi177_31116_object72073:head# 2024-09-13T11:24:31.256 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:43cd8505:::benchmark_data_smithi177_31116_object104276:head# 2024-09-13T11:24:31.257 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:43ce36e8:::benchmark_data_smithi177_31116_object66410:head# 2024-09-13T11:24:31.259 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:43ceb2f4:::benchmark_data_smithi177_31116_object67889:head# 2024-09-13T11:24:31.260 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:43cec608:::benchmark_data_smithi177_31116_object96993:head# 2024-09-13T11:24:31.260 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:43d07740:::benchmark_data_smithi177_31116_object25070:head# 2024-09-13T11:24:31.262 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:43d0d1b6:::benchmark_data_smithi177_31116_object21133:head# 2024-09-13T11:24:31.262 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:43d1432c:::benchmark_data_smithi177_31116_object15372:head# 2024-09-13T11:24:31.263 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:43d16408:::benchmark_data_smithi177_31116_object33597:head# 2024-09-13T11:24:31.264 INFO:tasks.ceph.osd.4.smithi001.stderr:2024-09-13T11:24:31.259+0000 7ff8d0c3d640 -1 osd.4 123 handle_osd_map injecting map crc failure 2024-09-13T11:24:31.264 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:43d25993:::benchmark_data_smithi177_31116_object70346:head# 2024-09-13T11:24:31.264 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:43d36171:::benchmark_data_smithi177_31116_object99907:head# 2024-09-13T11:24:31.267 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:43d400d6:::benchmark_data_smithi177_31116_object872:head# 2024-09-13T11:24:31.267 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:43d4456e:::benchmark_data_smithi177_31116_object10450:head# 2024-09-13T11:24:31.270 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:43d452e5:::benchmark_data_smithi177_31116_object21558:head# 2024-09-13T11:24:31.271 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:43d57d46:::benchmark_data_smithi177_31116_object48290:head# 2024-09-13T11:24:31.271 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:43d5ce35:::benchmark_data_smithi177_31116_object106392:head# 2024-09-13T11:24:31.273 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:43d67776:::benchmark_data_smithi177_31116_object5434:head# 2024-09-13T11:24:31.273 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:43d913e2:::benchmark_data_smithi177_31116_object7097:head# 2024-09-13T11:24:31.274 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:43d9e73f:::benchmark_data_smithi177_31116_object16875:head# 2024-09-13T11:24:31.274 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:43da9a88:::benchmark_data_smithi177_31116_object57204:head# 2024-09-13T11:24:31.275 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:43db7b5e:::benchmark_data_smithi177_31116_object43530:head# 2024-09-13T11:24:31.275 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:43dce2cc:::benchmark_data_smithi177_31116_object364:head# 2024-09-13T11:24:31.277 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:43dd8909:::benchmark_data_smithi177_31116_object44174:head# 2024-09-13T11:24:31.278 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:43dda1fc:::benchmark_data_smithi177_31116_object49909:head# 2024-09-13T11:24:31.278 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:43e029b2:::benchmark_data_smithi177_31116_object3317:head# 2024-09-13T11:24:31.279 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:43e07b29:::benchmark_data_smithi177_31116_object61162:head# 2024-09-13T11:24:31.279 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:43e0840f:::benchmark_data_smithi177_31116_object58707:head# 2024-09-13T11:24:31.280 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:43e1d565:::benchmark_data_smithi177_31116_object43324:head# 2024-09-13T11:24:31.283 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:43e29b75:::benchmark_data_smithi177_31116_object9830:head# 2024-09-13T11:24:31.283 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:43e3dadc:::benchmark_data_smithi177_31116_object29732:head# 2024-09-13T11:24:31.284 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:43e43de8:::benchmark_data_smithi177_31116_object21333:head# 2024-09-13T11:24:31.285 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:43e4676b:::benchmark_data_smithi177_31116_object30368:head# 2024-09-13T11:24:31.285 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:43e5a788:::benchmark_data_smithi177_31116_object31539:head# 2024-09-13T11:24:31.286 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:43e8a242:::benchmark_data_smithi177_31116_object37597:head# 2024-09-13T11:24:31.287 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:43e8fda4:::benchmark_data_smithi177_31116_object10082:head# 2024-09-13T11:24:31.287 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:43e917be:::benchmark_data_smithi177_31116_object63897:head# 2024-09-13T11:24:31.287 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:43e95a20:::benchmark_data_smithi177_31116_object55871:head# 2024-09-13T11:24:31.288 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-13T11:24:31.279+0000 7f6a68775640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26307) UID: 0 2024-09-13T11:24:31.288 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:43eaaeaf:::benchmark_data_smithi177_31116_object1160:head# 2024-09-13T11:24:31.288 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:43eb63d0:::benchmark_data_smithi177_31116_object23373:head# 2024-09-13T11:24:31.289 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:43ebb63a:::benchmark_data_smithi177_31116_object46931:head# 2024-09-13T11:24:31.289 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:43ecf23f:::benchmark_data_smithi177_31116_object88383:head# 2024-09-13T11:24:31.290 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:43ed0c1f:::benchmark_data_smithi177_31116_object9407:head# 2024-09-13T11:24:31.290 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:43edea67:::benchmark_data_smithi177_31116_object102331:head# 2024-09-13T11:24:31.291 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:43ee016a:::benchmark_data_smithi177_31116_object23923:head# 2024-09-13T11:24:31.291 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:43ee052d:::benchmark_data_smithi177_31116_object27182:head# 2024-09-13T11:24:31.292 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:43ee1041:::benchmark_data_smithi177_31116_object53717:head# 2024-09-13T11:24:31.293 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:43ee8d95:::benchmark_data_smithi177_31116_object30773:head# 2024-09-13T11:24:31.296 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:43efc1a0:::benchmark_data_smithi177_31116_object60481:head# 2024-09-13T11:24:31.296 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:43f00b2e:::benchmark_data_smithi177_31116_object61257:head# 2024-09-13T11:24:31.297 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:43f24772:::benchmark_data_smithi177_31116_object96513:head# 2024-09-13T11:24:31.298 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:43f34b7a:::benchmark_data_smithi177_31116_object80634:head# 2024-09-13T11:24:31.300 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:43f39f1e:::benchmark_data_smithi177_31116_object26583:head# 2024-09-13T11:24:31.300 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:43f47736:::benchmark_data_smithi177_31116_object74374:head# 2024-09-13T11:24:31.300 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:43f4cf44:::benchmark_data_smithi177_31116_object47394:head# 2024-09-13T11:24:31.300 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:43f4f86c:::benchmark_data_smithi177_31116_object35732:head# 2024-09-13T11:24:31.301 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:43f5203f:::benchmark_data_smithi177_31116_object73878:head# 2024-09-13T11:24:31.301 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:43f56511:::benchmark_data_smithi177_31116_object107592:head# 2024-09-13T11:24:31.302 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:43f614c1:::benchmark_data_smithi177_31116_object59142:head# 2024-09-13T11:24:31.302 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:43f69b50:::benchmark_data_smithi177_31116_object16326:head# 2024-09-13T11:24:31.303 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:43f6cd29:::benchmark_data_smithi177_31116_object100521:head# 2024-09-13T11:24:31.303 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:43f6dd59:::benchmark_data_smithi177_31116_object65428:head# 2024-09-13T11:24:31.304 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:43f93a22:::benchmark_data_smithi177_31116_object107316:head# 2024-09-13T11:24:31.304 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:43fa1ff7:::benchmark_data_smithi177_31116_object86542:head# 2024-09-13T11:24:31.305 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:43fa4029:::benchmark_data_smithi177_31116_object7918:head# 2024-09-13T11:24:31.305 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:43fb4edc:::benchmark_data_smithi177_31116_object10727:head# 2024-09-13T11:24:31.306 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:43fbaffc:::benchmark_data_smithi177_31116_object10325:head# 2024-09-13T11:24:31.306 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:43fc1b43:::benchmark_data_smithi177_31116_object92984:head# 2024-09-13T11:24:31.307 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:43fc7361:::benchmark_data_smithi177_31116_object34971:head# 2024-09-13T11:24:31.308 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:43fd8bdf:::benchmark_data_smithi177_31116_object20642:head# 2024-09-13T11:24:31.308 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:43fd95df:::benchmark_data_smithi177_31116_object52270:head# 2024-09-13T11:24:31.309 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:43fe3402:::benchmark_data_smithi177_31116_object42393:head# 2024-09-13T11:24:31.309 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:43fe38e6:::benchmark_data_smithi177_31116_object17525:head# 2024-09-13T11:24:31.310 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:43fe52c3:::benchmark_data_smithi177_31116_object25715:head# 2024-09-13T11:24:31.311 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:43ff4406:::benchmark_data_smithi177_31116_object8748:head# 2024-09-13T11:24:31.311 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:43ffb701:::benchmark_data_smithi177_31116_object1464:head# 2024-09-13T11:24:31.312 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:43ffce4a:::benchmark_data_smithi177_31116_object63955:head# 2024-09-13T11:24:31.314 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:43fff576:::benchmark_data_smithi177_31116_object18997:head# 2024-09-13T11:24:31.315 INFO:teuthology.orchestra.run.smithi162.stderr:Export successful 2024-09-13T11:24:31.363 INFO:teuthology.orchestra.run.smithi001.stdout:osd.0: {} 2024-09-13T11:24:31.363 INFO:teuthology.orchestra.run.smithi001.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T11:24:31.364 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-13T11:24:31.364 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:24:31.380 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-13T11:24:31.393 INFO:teuthology.orchestra.run.smithi001.stdout:osd.1: {} 2024-09-13T11:24:31.393 INFO:teuthology.orchestra.run.smithi001.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T11:24:31.422 INFO:teuthology.orchestra.run.smithi001.stdout:osd.2: {} 2024-09-13T11:24:31.422 INFO:teuthology.orchestra.run.smithi001.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T11:24:31.426 INFO:tasks.ceph.osd.10.smithi162.stderr:2024-09-13T11:24:31.424+0000 7fb59c9f0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26317) UID: 0 2024-09-13T11:24:31.454 INFO:teuthology.orchestra.run.smithi001.stdout:osd.3: {} 2024-09-13T11:24:31.454 INFO:teuthology.orchestra.run.smithi001.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T11:24:31.481 INFO:teuthology.orchestra.run.smithi001.stdout:osd.4: {} 2024-09-13T11:24:31.481 INFO:teuthology.orchestra.run.smithi001.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T11:24:31.485 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-09-13T11:24:31.480+0000 7f97befc9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26315) UID: 0 2024-09-13T11:24:31.515 INFO:teuthology.orchestra.run.smithi001.stdout:osd.5: {} 2024-09-13T11:24:31.515 INFO:teuthology.orchestra.run.smithi001.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T11:24:31.551 INFO:teuthology.orchestra.run.smithi001.stdout:osd.6: {} 2024-09-13T11:24:31.551 INFO:teuthology.orchestra.run.smithi001.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T11:24:31.561 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-13T11:24:31.561 INFO:teuthology.orchestra.run.smithi162.stdout: "size": 20, 2024-09-13T11:24:31.561 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 600, 2024-09-13T11:24:31.561 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [ 2024-09-13T11:24:31.561 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:31.561 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4257.0:80 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 262144~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-13T11:24:31.561 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:22:12.516065+0000", 2024-09-13T11:24:31.561 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 139.040255395, 2024-09-13T11:24:31.562 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.46461997999999999, 2024-09-13T11:24:31.562 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:24:31.562 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:24:31.562 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:31.562 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:24:31.562 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4257", 2024-09-13T11:24:31.562 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/1488131527", 2024-09-13T11:24:31.562 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 80 2024-09-13T11:24:31.562 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:31.562 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:24:31.562 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:31.562 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:24:31.562 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.516065+0000", 2024-09-13T11:24:31.562 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:31.563 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:31.563 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:31.563 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:24:31.563 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.516065+0000", 2024-09-13T11:24:31.563 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:31.563 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:31.563 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:31.563 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:24:31.563 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.516066+0000", 2024-09-13T11:24:31.563 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0869999999999999e-06 2024-09-13T11:24:31.563 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:31.563 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:31.563 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:24:31.563 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.516112+0000", 2024-09-13T11:24:31.564 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.5503000000000001e-05 2024-09-13T11:24:31.564 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:31.564 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:31.564 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:24:31.564 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.516128+0000", 2024-09-13T11:24:31.564 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5891000000000001e-05 2024-09-13T11:24:31.564 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:31.564 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:31.564 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:24:31.564 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.516138+0000", 2024-09-13T11:24:31.564 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.8120000000000002e-06 2024-09-13T11:24:31.564 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:31.564 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:31.565 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:24:31.565 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.520398+0000", 2024-09-13T11:24:31.565 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0042605450000000001 2024-09-13T11:24:31.565 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:31.565 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:31.566 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:24:31.566 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.520526+0000", 2024-09-13T11:24:31.566 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000128355 2024-09-13T11:24:31.566 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:31.566 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:31.566 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 0", 2024-09-13T11:24:31.566 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.520730+0000", 2024-09-13T11:24:31.567 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00020337799999999999 2024-09-13T11:24:31.567 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:31.567 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:31.567 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:24:31.567 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.522417+0000", 2024-09-13T11:24:31.567 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001687184 2024-09-13T11:24:31.567 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:31.567 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:31.567 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:31.567 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.980408+0000", 2024-09-13T11:24:31.567 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.45799068399999998 2024-09-13T11:24:31.567 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:31.567 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:31.568 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:24:31.568 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.980565+0000", 2024-09-13T11:24:31.568 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00015760899999999999 2024-09-13T11:24:31.568 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:31.568 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:31.568 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:24:31.568 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.980685+0000", 2024-09-13T11:24:31.568 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000119932 2024-09-13T11:24:31.568 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:31.568 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:24:31.568 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:31.568 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:31.568 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:31.569 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop_reply(client.4257.0:80 1.0 e24/23 ondisk, result = 0)", 2024-09-13T11:24:31.569 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:22:12.522938+0000", 2024-09-13T11:24:31.569 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 139.033382362, 2024-09-13T11:24:31.569 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.45776420600000001, 2024-09-13T11:24:31.569 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:24:31.569 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:24:31.569 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:24:31.569 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:24:31.569 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:31.569 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:24:31.569 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.522938+0000", 2024-09-13T11:24:31.569 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:31.570 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:31.570 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:31.570 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:24:31.570 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.522938+0000", 2024-09-13T11:24:31.570 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:31.570 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:31.570 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:31.570 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:24:31.570 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.522941+0000", 2024-09-13T11:24:31.570 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.0290000000000001e-06 2024-09-13T11:24:31.570 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:31.570 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:31.570 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:24:31.570 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.522950+0000", 2024-09-13T11:24:31.571 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.7499999999999992e-06 2024-09-13T11:24:31.571 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:31.571 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:31.571 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:24:31.571 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.980215+0000", 2024-09-13T11:24:31.571 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.45726523800000002 2024-09-13T11:24:31.571 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:31.571 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:31.571 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:24:31.571 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.980229+0000", 2024-09-13T11:24:31.571 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3549e-05 2024-09-13T11:24:31.571 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:31.571 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:31.571 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:24:31.572 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.980365+0000", 2024-09-13T11:24:31.572 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00013608700000000001 2024-09-13T11:24:31.572 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:31.572 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:31.572 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:24:31.572 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.980388+0000", 2024-09-13T11:24:31.572 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.3112000000000002e-05 2024-09-13T11:24:31.572 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:31.572 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:31.572 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:24:31.572 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.980702+0000", 2024-09-13T11:24:31.572 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00031444099999999999 2024-09-13T11:24:31.572 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:31.573 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:24:31.573 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:31.573 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:31.573 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:31.573 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:25416 3.8 3:1a1fbdd1:::benchmark_data_smithi177_31116_object25415:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e56)", 2024-09-13T11:24:31.573 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:04.512188+0000", 2024-09-13T11:24:31.573 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 87.044132395000005, 2024-09-13T11:24:31.573 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.210208697, 2024-09-13T11:24:31.573 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:24:31.573 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:24:31.573 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:31.573 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:24:31.573 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:24:31.574 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:31.574 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 25416 2024-09-13T11:24:31.574 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:31.574 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:24:31.574 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:31.574 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:24:31.574 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.512188+0000", 2024-09-13T11:24:31.574 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:31.574 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:31.574 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:31.574 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:24:31.574 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.512188+0000", 2024-09-13T11:24:31.574 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:31.574 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:31.575 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:31.575 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:24:31.575 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.512192+0000", 2024-09-13T11:24:31.575 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.2660000000000003e-06 2024-09-13T11:24:31.575 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:31.575 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:31.575 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:24:31.575 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.512247+0000", 2024-09-13T11:24:31.575 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.4889000000000003e-05 2024-09-13T11:24:31.575 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:31.575 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:31.575 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:24:31.575 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.512279+0000", 2024-09-13T11:24:31.575 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.1146000000000003e-05 2024-09-13T11:24:31.576 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:31.576 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:31.576 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:24:31.576 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.512297+0000", 2024-09-13T11:24:31.576 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.825e-05 2024-09-13T11:24:31.576 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:31.576 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:31.576 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:24:31.576 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.512377+0000", 2024-09-13T11:24:31.576 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.9875000000000004e-05 2024-09-13T11:24:31.576 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:31.576 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:31.576 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:24:31.577 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.512570+0000", 2024-09-13T11:24:31.577 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00019312100000000001 2024-09-13T11:24:31.577 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:31.577 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:31.577 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 1", 2024-09-13T11:24:31.577 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.512798+0000", 2024-09-13T11:24:31.577 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00022847899999999999 2024-09-13T11:24:31.577 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:31.577 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:31.577 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:24:31.577 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.514975+0000", 2024-09-13T11:24:31.577 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0021769979999999999 2024-09-13T11:24:31.577 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:31.578 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:31.578 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:31.578 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.722207+0000", 2024-09-13T11:24:31.578 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.20723217099999999 2024-09-13T11:24:31.578 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:31.578 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:31.578 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:24:31.578 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.722314+0000", 2024-09-13T11:24:31.578 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00010648400000000001 2024-09-13T11:24:31.578 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:31.578 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:31.578 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:24:31.578 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.722397+0000", 2024-09-13T11:24:31.578 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.3017999999999997e-05 2024-09-13T11:24:31.579 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:31.579 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:24:31.579 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:31.579 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:31.579 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:31.579 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop_reply(client.4665.0:25416 3.8 e56/40 ondisk, result = 0)", 2024-09-13T11:24:31.579 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:04.515635+0000", 2024-09-13T11:24:31.579 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 87.040685742999997, 2024-09-13T11:24:31.579 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.20677600500000001, 2024-09-13T11:24:31.579 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:24:31.579 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:24:31.579 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:24:31.579 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:24:31.579 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:31.580 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:24:31.580 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.515635+0000", 2024-09-13T11:24:31.580 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:31.580 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:31.580 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:31.580 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:24:31.580 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.515635+0000", 2024-09-13T11:24:31.580 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:31.580 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:31.580 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:31.580 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:24:31.580 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.515639+0000", 2024-09-13T11:24:31.580 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.619e-06 2024-09-13T11:24:31.581 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:31.581 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:31.581 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:24:31.581 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.515650+0000", 2024-09-13T11:24:31.581 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.111e-05 2024-09-13T11:24:31.581 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:31.581 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:31.581 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:24:31.581 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.722004+0000", 2024-09-13T11:24:31.581 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.206354805 2024-09-13T11:24:31.581 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:31.582 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:31.582 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:24:31.582 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.722017+0000", 2024-09-13T11:24:31.582 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2840000000000001e-05 2024-09-13T11:24:31.582 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:31.582 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:31.582 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:24:31.582 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.722186+0000", 2024-09-13T11:24:31.582 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00016898299999999999 2024-09-13T11:24:31.582 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:31.582 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:31.583 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:24:31.583 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.722199+0000", 2024-09-13T11:24:31.583 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.289e-05 2024-09-13T11:24:31.583 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:31.583 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:31.583 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:24:31.583 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.722411+0000", 2024-09-13T11:24:31.583 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00021175799999999999 2024-09-13T11:24:31.583 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:31.584 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:24:31.584 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:31.584 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:31.584 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:31.584 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:30833 3.8 3:1c203f8d:::benchmark_data_smithi177_31116_object30832:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e59)", 2024-09-13T11:24:31.584 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:09.506656+0000", 2024-09-13T11:24:31.584 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 82.049664844999995, 2024-09-13T11:24:31.584 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.224862282, 2024-09-13T11:24:31.584 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:24:31.584 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:24:31.584 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:31.584 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:24:31.585 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:24:31.585 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:31.585 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 30833 2024-09-13T11:24:31.585 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:31.585 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:24:31.585 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:31.585 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:24:31.585 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.506656+0000", 2024-09-13T11:24:31.585 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:31.585 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:31.585 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:31.586 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:24:31.586 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.506656+0000", 2024-09-13T11:24:31.586 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:31.586 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:31.586 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:31.586 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:24:31.586 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.506660+0000", 2024-09-13T11:24:31.586 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.6e-06 2024-09-13T11:24:31.586 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:31.586 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:31.586 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:24:31.586 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.506717+0000", 2024-09-13T11:24:31.586 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.6131999999999997e-05 2024-09-13T11:24:31.586 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:31.587 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:31.587 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:24:31.587 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.506751+0000", 2024-09-13T11:24:31.587 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.4746000000000002e-05 2024-09-13T11:24:31.587 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:31.587 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:31.587 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:24:31.587 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.506774+0000", 2024-09-13T11:24:31.587 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.2636000000000001e-05 2024-09-13T11:24:31.587 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:31.587 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:31.587 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:24:31.587 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.506878+0000", 2024-09-13T11:24:31.588 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000104057 2024-09-13T11:24:31.588 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:31.588 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:31.588 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:24:31.588 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.507052+0000", 2024-09-13T11:24:31.588 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00017402800000000001 2024-09-13T11:24:31.588 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:31.588 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:31.588 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 1", 2024-09-13T11:24:31.588 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.507217+0000", 2024-09-13T11:24:31.588 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00016516 2024-09-13T11:24:31.588 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:31.588 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:31.588 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:24:31.589 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.508992+0000", 2024-09-13T11:24:31.589 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0017743990000000001 2024-09-13T11:24:31.589 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:31.589 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:31.589 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:31.589 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.731429+0000", 2024-09-13T11:24:31.589 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.22243718000000001 2024-09-13T11:24:31.589 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:31.589 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:31.589 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:24:31.589 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.731479+0000", 2024-09-13T11:24:31.589 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.0146999999999999e-05 2024-09-13T11:24:31.589 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:31.590 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:31.590 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:24:31.590 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.731518+0000", 2024-09-13T11:24:31.590 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.9196999999999998e-05 2024-09-13T11:24:31.590 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:31.590 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:24:31.590 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:31.590 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:31.590 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:31.590 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop_reply(client.4665.0:30833 3.8 e59/40 ondisk, result = 0)", 2024-09-13T11:24:31.590 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:09.508907+0000", 2024-09-13T11:24:31.590 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 82.047413750999993, 2024-09-13T11:24:31.591 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.222618862, 2024-09-13T11:24:31.591 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:24:31.591 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:24:31.591 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:24:31.591 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:24:31.591 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:31.591 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:24:31.591 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.508907+0000", 2024-09-13T11:24:31.591 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:31.591 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:31.592 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:31.592 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:24:31.592 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.508907+0000", 2024-09-13T11:24:31.592 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:31.592 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:31.592 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:31.592 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:24:31.592 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.508910+0000", 2024-09-13T11:24:31.592 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.993e-06 2024-09-13T11:24:31.592 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:31.592 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:31.592 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:24:31.593 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.508918+0000", 2024-09-13T11:24:31.593 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.7000000000000008e-06 2024-09-13T11:24:31.593 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:31.593 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:31.593 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:24:31.593 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.731341+0000", 2024-09-13T11:24:31.593 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.22242308699999999 2024-09-13T11:24:31.593 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:31.593 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:31.593 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:24:31.593 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.731350+0000", 2024-09-13T11:24:31.593 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.3179999999999996e-06 2024-09-13T11:24:31.593 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:31.594 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:31.594 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:24:31.594 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.731418+0000", 2024-09-13T11:24:31.594 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.7586e-05 2024-09-13T11:24:31.594 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:31.594 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:31.594 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:24:31.594 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.731424+0000", 2024-09-13T11:24:31.594 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.844e-06 2024-09-13T11:24:31.594 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:31.594 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:31.594 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:24:31.594 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.731526+0000", 2024-09-13T11:24:31.594 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000101334 2024-09-13T11:24:31.595 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:31.595 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:24:31.595 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:31.595 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:31.595 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:31.595 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:48977 3.8 3:1030343e:::benchmark_data_smithi177_31116_object48976:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e71)", 2024-09-13T11:24:31.595 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:25.517299+0000", 2024-09-13T11:24:31.595 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 66.039021457000004, 2024-09-13T11:24:31.595 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.14514738599999999, 2024-09-13T11:24:31.595 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:24:31.595 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:24:31.595 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:31.596 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:24:31.596 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:24:31.596 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:31.596 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 48977 2024-09-13T11:24:31.596 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:31.596 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:24:31.596 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:31.596 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:24:31.596 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.517299+0000", 2024-09-13T11:24:31.596 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:31.596 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:31.597 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:31.597 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:24:31.597 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.517299+0000", 2024-09-13T11:24:31.597 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:31.597 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:31.597 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:31.597 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:24:31.597 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.517301+0000", 2024-09-13T11:24:31.597 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.7060000000000001e-06 2024-09-13T11:24:31.597 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:31.597 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:31.597 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:24:31.597 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.517354+0000", 2024-09-13T11:24:31.597 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.3148e-05 2024-09-13T11:24:31.598 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:31.598 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:31.598 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:24:31.598 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.517365+0000", 2024-09-13T11:24:31.598 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1389000000000001e-05 2024-09-13T11:24:31.598 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:31.598 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:31.598 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:24:31.598 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.517374+0000", 2024-09-13T11:24:31.598 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.6740000000000003e-06 2024-09-13T11:24:31.598 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:31.598 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:31.598 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:24:31.599 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.517454+0000", 2024-09-13T11:24:31.599 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.9968000000000004e-05 2024-09-13T11:24:31.599 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:31.599 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:31.599 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:24:31.599 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.517561+0000", 2024-09-13T11:24:31.599 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00010655 2024-09-13T11:24:31.599 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:31.599 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:31.599 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 1", 2024-09-13T11:24:31.599 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.517664+0000", 2024-09-13T11:24:31.600 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000103409 2024-09-13T11:24:31.600 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:31.600 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:31.601 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:24:31.601 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.521381+0000", 2024-09-13T11:24:31.601 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0037174019999999999 2024-09-13T11:24:31.601 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:31.601 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:31.601 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:31.601 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.662307+0000", 2024-09-13T11:24:31.601 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.14092524000000001 2024-09-13T11:24:31.601 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:31.601 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:31.601 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:24:31.601 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.662382+0000", 2024-09-13T11:24:31.601 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.5331000000000001e-05 2024-09-13T11:24:31.602 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:31.602 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:31.602 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:24:31.602 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.662447+0000", 2024-09-13T11:24:31.602 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.4568999999999998e-05 2024-09-13T11:24:31.602 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:31.602 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:24:31.603 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:31.603 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:31.603 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:31.603 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.4665.0:54351 3.10 e74/70 3:085f283e:::benchmark_data_smithi177_31116_object54350:head v 74'3087, mlcod=49'780)", 2024-09-13T11:24:31.603 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:30.599667+0000", 2024-09-13T11:24:31.603 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 60.956653244999998, 2024-09-13T11:24:31.603 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.16617266999999999, 2024-09-13T11:24:31.603 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:24:31.603 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:24:31.604 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:31.604 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:24:31.604 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:31.604 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:24:31.604 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.599667+0000", 2024-09-13T11:24:31.604 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:31.604 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:31.604 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:31.604 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:24:31.604 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.599667+0000", 2024-09-13T11:24:31.604 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:31.604 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:31.604 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:31.604 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:24:31.605 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.599669+0000", 2024-09-13T11:24:31.605 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3599999999999999e-06 2024-09-13T11:24:31.605 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:31.605 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:31.605 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:24:31.605 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.599712+0000", 2024-09-13T11:24:31.605 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.3723000000000001e-05 2024-09-13T11:24:31.605 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:31.605 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:31.605 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:24:31.605 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.762241+0000", 2024-09-13T11:24:31.605 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.16252863400000001 2024-09-13T11:24:31.606 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:31.606 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:31.606 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:24:31.606 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.762255+0000", 2024-09-13T11:24:31.606 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3414999999999999e-05 2024-09-13T11:24:31.606 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:31.606 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:31.606 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:24:31.606 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.762329+0000", 2024-09-13T11:24:31.606 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.4735999999999997e-05 2024-09-13T11:24:31.606 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:31.606 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:31.606 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:24:31.606 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.762358+0000", 2024-09-13T11:24:31.607 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.8308000000000001e-05 2024-09-13T11:24:31.607 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:31.607 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:31.607 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:24:31.607 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.765770+0000", 2024-09-13T11:24:31.607 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0034126590000000002 2024-09-13T11:24:31.607 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:31.607 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:31.607 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:24:31.607 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.765840+0000", 2024-09-13T11:24:31.607 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.9834999999999998e-05 2024-09-13T11:24:31.607 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:31.607 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:24:31.607 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:31.608 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:31.608 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:31.608 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPush(3.10 74/70 [PushOp(3:0b3e58bf:::benchmark_data_smithi177_31116_object12372:head, version: 49'782, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:0b3e58bf:::benchmark_data_smithi177_31116_object12372:head@49'782, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:24:31.608 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:30.608892+0000", 2024-09-13T11:24:31.608 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 60.947428891000001, 2024-09-13T11:24:31.608 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.15687285300000001, 2024-09-13T11:24:31.608 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:24:31.608 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:24:31.608 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:24:31.608 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:24:31.608 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:31.608 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:24:31.608 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.608892+0000", 2024-09-13T11:24:31.609 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:31.609 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:31.609 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:31.609 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:24:31.609 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.608892+0000", 2024-09-13T11:24:31.609 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:31.609 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:31.609 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:31.609 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:24:31.609 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.608893+0000", 2024-09-13T11:24:31.609 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.666e-06 2024-09-13T11:24:31.609 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:31.609 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:31.609 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:24:31.610 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.609008+0000", 2024-09-13T11:24:31.610 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00011411 2024-09-13T11:24:31.610 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:31.610 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:31.610 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:24:31.610 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.763870+0000", 2024-09-13T11:24:31.610 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.15486217799999999 2024-09-13T11:24:31.610 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:31.610 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:31.610 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:24:31.610 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.763904+0000", 2024-09-13T11:24:31.610 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.4082000000000003e-05 2024-09-13T11:24:31.610 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:31.610 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:31.611 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:24:31.611 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.764247+0000", 2024-09-13T11:24:31.611 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00034281900000000001 2024-09-13T11:24:31.611 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:31.611 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:31.611 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:24:31.611 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.764270+0000", 2024-09-13T11:24:31.611 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.3244e-05 2024-09-13T11:24:31.611 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:31.611 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:31.611 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:24:31.611 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.765765+0000", 2024-09-13T11:24:31.611 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001494754 2024-09-13T11:24:31.611 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:31.612 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:24:31.612 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:31.612 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:31.612 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:31.612 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPush(3.10 76/70 [PushOp(3:0e7d2700:::benchmark_data_smithi177_31116_object17406:head, version: 52'1075, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:0e7d2700:::benchmark_data_smithi177_31116_object17406:head@52'1075, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:24:31.612 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:32.206198+0000", 2024-09-13T11:24:31.612 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 59.350123042, 2024-09-13T11:24:31.612 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.25193395000000002, 2024-09-13T11:24:31.612 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:24:31.612 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:24:31.612 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:24:31.612 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:24:31.613 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:31.613 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:24:31.613 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.206198+0000", 2024-09-13T11:24:31.613 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:31.613 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:31.613 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:31.613 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:24:31.613 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.206198+0000", 2024-09-13T11:24:31.613 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:31.613 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:31.614 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:31.614 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:24:31.614 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.206199+0000", 2024-09-13T11:24:31.614 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.46e-06 2024-09-13T11:24:31.614 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:31.614 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:31.614 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:24:31.614 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.206250+0000", 2024-09-13T11:24:31.614 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.113e-05 2024-09-13T11:24:31.614 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:31.614 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:31.614 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:24:31.614 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.456042+0000", 2024-09-13T11:24:31.615 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.24979182999999999 2024-09-13T11:24:31.615 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:31.615 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:31.615 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:24:31.615 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.456084+0000", 2024-09-13T11:24:31.615 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.1814000000000003e-05 2024-09-13T11:24:31.615 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:31.615 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:31.615 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:24:31.615 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.456342+0000", 2024-09-13T11:24:31.615 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000258404 2024-09-13T11:24:31.615 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:31.615 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:31.615 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:24:31.616 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.456367+0000", 2024-09-13T11:24:31.616 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.5099000000000001e-05 2024-09-13T11:24:31.616 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:31.616 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:31.616 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:24:31.616 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.458132+0000", 2024-09-13T11:24:31.616 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0017642129999999999 2024-09-13T11:24:31.616 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:31.616 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:24:31.616 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:31.616 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:31.616 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:31.617 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPush(3.10 76/70 [PushOp(3:0eee8bcf:::benchmark_data_smithi177_31116_object17589:head, version: 53'1086, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:0eee8bcf:::benchmark_data_smithi177_31116_object17589:head@53'1086, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:24:31.617 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:32.585183+0000", 2024-09-13T11:24:31.617 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 58.971137147999997, 2024-09-13T11:24:31.617 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.16696202199999999, 2024-09-13T11:24:31.617 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:24:31.617 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:24:31.617 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:24:31.617 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:24:31.617 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:31.617 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:24:31.617 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.585183+0000", 2024-09-13T11:24:31.617 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:31.617 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:31.617 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:31.618 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:24:31.618 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.585183+0000", 2024-09-13T11:24:31.618 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:31.618 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:31.618 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:31.618 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:24:31.618 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.585187+0000", 2024-09-13T11:24:31.618 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.8480000000000003e-06 2024-09-13T11:24:31.618 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:31.618 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:31.618 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:24:31.618 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.585264+0000", 2024-09-13T11:24:31.618 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.6327000000000003e-05 2024-09-13T11:24:31.618 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:31.619 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:31.619 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:24:31.619 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.750459+0000", 2024-09-13T11:24:31.619 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.16519551900000001 2024-09-13T11:24:31.619 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:31.619 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:31.619 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:24:31.619 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.750480+0000", 2024-09-13T11:24:31.619 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.0984000000000001e-05 2024-09-13T11:24:31.619 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:31.619 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:31.619 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:24:31.619 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.750633+0000", 2024-09-13T11:24:31.619 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000153148 2024-09-13T11:24:31.620 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:31.620 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:31.620 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:24:31.620 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.750652+0000", 2024-09-13T11:24:31.620 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.8981e-05 2024-09-13T11:24:31.620 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:31.620 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:31.620 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:24:31.620 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.752145+0000", 2024-09-13T11:24:31.620 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001493215 2024-09-13T11:24:31.620 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:31.620 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:24:31.620 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:31.620 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:31.621 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:31.621 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:60417 3.10 3:092808b0:::benchmark_data_smithi177_31116_object60416:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e85)", 2024-09-13T11:24:31.621 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:42.173238+0000", 2024-09-13T11:24:31.621 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 49.383082133999999, 2024-09-13T11:24:31.621 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.38965994599999998, 2024-09-13T11:24:31.621 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:24:31.621 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:24:31.621 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:31.621 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:24:31.621 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:24:31.621 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:31.621 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 60417 2024-09-13T11:24:31.621 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:31.621 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:24:31.622 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:31.622 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:24:31.622 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.173238+0000", 2024-09-13T11:24:31.622 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:31.622 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:31.622 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:31.622 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:24:31.622 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.173238+0000", 2024-09-13T11:24:31.622 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:31.622 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:31.622 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:31.622 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:24:31.622 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.173241+0000", 2024-09-13T11:24:31.622 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.6819999999999999e-06 2024-09-13T11:24:31.623 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:31.623 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:31.623 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:24:31.623 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.173335+0000", 2024-09-13T11:24:31.623 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.3957999999999997e-05 2024-09-13T11:24:31.623 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:31.623 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:31.623 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:24:31.623 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.173358+0000", 2024-09-13T11:24:31.623 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.3023999999999998e-05 2024-09-13T11:24:31.623 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:31.623 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:31.623 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:24:31.623 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.173373+0000", 2024-09-13T11:24:31.624 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4867e-05 2024-09-13T11:24:31.624 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:31.624 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:31.624 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:24:31.624 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.173466+0000", 2024-09-13T11:24:31.624 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.2542999999999998e-05 2024-09-13T11:24:31.624 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:31.624 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:31.624 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:24:31.624 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.173754+0000", 2024-09-13T11:24:31.624 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000288657 2024-09-13T11:24:31.624 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:31.624 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:31.625 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 0", 2024-09-13T11:24:31.625 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.174066+0000", 2024-09-13T11:24:31.625 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00031170999999999998 2024-09-13T11:24:31.625 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:31.625 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:31.625 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:24:31.625 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.178245+0000", 2024-09-13T11:24:31.625 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00417864 2024-09-13T11:24:31.625 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:31.625 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:31.625 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:31.625 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.562617+0000", 2024-09-13T11:24:31.626 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.38437253199999999 2024-09-13T11:24:31.626 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:31.626 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:31.626 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:24:31.626 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.562782+0000", 2024-09-13T11:24:31.626 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00016499200000000001 2024-09-13T11:24:31.626 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:31.626 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:31.626 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:24:31.626 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.562898+0000", 2024-09-13T11:24:31.626 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000116341 2024-09-13T11:24:31.626 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:31.626 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:24:31.627 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:31.627 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:31.627 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:31.627 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:64804 3.8 3:104508a0:::benchmark_data_smithi177_31116_object64803:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90)", 2024-09-13T11:24:31.627 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:47.694588+0000", 2024-09-13T11:24:31.627 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 43.861732590999999, 2024-09-13T11:24:31.627 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.11215968, 2024-09-13T11:24:31.627 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:24:31.627 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:24:31.627 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:31.627 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:24:31.627 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:24:31.627 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:31.628 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 64804 2024-09-13T11:24:31.628 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:31.628 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:24:31.628 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:31.628 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:24:31.628 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:47.694588+0000", 2024-09-13T11:24:31.628 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:31.628 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:31.628 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:31.629 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:24:31.629 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:47.694588+0000", 2024-09-13T11:24:31.629 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:31.629 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:31.629 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:31.629 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:24:31.629 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:47.694591+0000", 2024-09-13T11:24:31.629 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.9629999999999998e-06 2024-09-13T11:24:31.629 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:31.629 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:31.629 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:24:31.629 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:47.694643+0000", 2024-09-13T11:24:31.629 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.1792000000000002e-05 2024-09-13T11:24:31.629 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:31.630 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:31.630 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:24:31.630 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:47.694660+0000", 2024-09-13T11:24:31.630 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.7153e-05 2024-09-13T11:24:31.630 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:31.630 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:31.630 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:24:31.630 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:47.694671+0000", 2024-09-13T11:24:31.630 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0682999999999999e-05 2024-09-13T11:24:31.630 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:31.630 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:31.630 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:24:31.630 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:47.694882+0000", 2024-09-13T11:24:31.630 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00021096799999999999 2024-09-13T11:24:31.631 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:31.631 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:31.631 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:24:31.631 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:47.695113+0000", 2024-09-13T11:24:31.631 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000231265 2024-09-13T11:24:31.631 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:31.631 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:31.631 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 1", 2024-09-13T11:24:31.631 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:47.695331+0000", 2024-09-13T11:24:31.631 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000218135 2024-09-13T11:24:31.631 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:31.631 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:31.631 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:24:31.632 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:47.699656+0000", 2024-09-13T11:24:31.632 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0043248440000000004 2024-09-13T11:24:31.632 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:31.632 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:31.632 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:31.632 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:47.806445+0000", 2024-09-13T11:24:31.632 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.106789124 2024-09-13T11:24:31.632 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:31.632 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:31.632 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:24:31.632 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:47.806617+0000", 2024-09-13T11:24:31.632 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00017241199999999999 2024-09-13T11:24:31.632 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:31.632 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:31.633 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:24:31.633 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:47.806748+0000", 2024-09-13T11:24:31.633 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00013034099999999999 2024-09-13T11:24:31.633 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:31.633 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:24:31.633 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:31.633 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:31.633 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:31.633 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:64871 3.8 3:11d9896a:::benchmark_data_smithi177_31116_object64870:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90)", 2024-09-13T11:24:31.633 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:47.826113+0000", 2024-09-13T11:24:31.633 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 43.730207810000003, 2024-09-13T11:24:31.633 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.132123195, 2024-09-13T11:24:31.633 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:24:31.634 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:24:31.634 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:31.634 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:24:31.634 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:24:31.634 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:31.634 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 64871 2024-09-13T11:24:31.634 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:31.634 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:24:31.634 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:31.634 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:24:31.634 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:47.826113+0000", 2024-09-13T11:24:31.634 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:31.634 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:31.635 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:31.635 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:24:31.635 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:47.826113+0000", 2024-09-13T11:24:31.635 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:31.635 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:31.635 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:31.635 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:24:31.635 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:47.826128+0000", 2024-09-13T11:24:31.635 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4882000000000001e-05 2024-09-13T11:24:31.635 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:31.635 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:31.635 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:24:31.636 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:47.826189+0000", 2024-09-13T11:24:31.636 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.1276000000000001e-05 2024-09-13T11:24:31.636 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:31.636 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:31.636 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:24:31.636 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:47.826223+0000", 2024-09-13T11:24:31.636 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.3804999999999999e-05 2024-09-13T11:24:31.636 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:31.636 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:31.636 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:24:31.636 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:47.826245+0000", 2024-09-13T11:24:31.636 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.2609999999999999e-05 2024-09-13T11:24:31.636 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:31.637 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:31.637 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:24:31.637 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:47.826419+0000", 2024-09-13T11:24:31.637 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00017394200000000001 2024-09-13T11:24:31.637 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:31.637 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:31.637 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:24:31.637 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:47.826676+0000", 2024-09-13T11:24:31.637 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000256741 2024-09-13T11:24:31.637 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:31.637 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:31.637 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 1", 2024-09-13T11:24:31.637 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:47.826903+0000", 2024-09-13T11:24:31.637 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00022646100000000001 2024-09-13T11:24:31.638 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:31.638 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:31.638 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:24:31.638 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:47.829731+0000", 2024-09-13T11:24:31.638 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0028284500000000001 2024-09-13T11:24:31.638 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:31.638 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:31.638 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:31.638 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:47.957975+0000", 2024-09-13T11:24:31.638 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.12824428500000001 2024-09-13T11:24:31.638 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:31.638 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:31.638 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:24:31.638 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:47.958129+0000", 2024-09-13T11:24:31.639 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00015343900000000001 2024-09-13T11:24:31.639 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:31.639 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:31.639 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:24:31.639 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:47.958236+0000", 2024-09-13T11:24:31.639 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000107304 2024-09-13T11:24:31.639 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:31.639 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:24:31.639 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:31.639 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:31.639 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:31.639 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:83193 3.8 3:10066d83:::benchmark_data_smithi177_31116_object83192:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106)", 2024-09-13T11:24:31.639 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:24:04.283923+0000", 2024-09-13T11:24:31.640 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 27.272397451, 2024-09-13T11:24:31.640 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.64775463499999997, 2024-09-13T11:24:31.640 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:24:31.640 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:24:31.640 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:31.640 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:24:31.640 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:24:31.640 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:31.640 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 83193 2024-09-13T11:24:31.640 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:31.640 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:24:31.640 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:31.640 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:24:31.641 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.283923+0000", 2024-09-13T11:24:31.641 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:31.641 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:31.641 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:31.641 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:24:31.641 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.283923+0000", 2024-09-13T11:24:31.641 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:31.641 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:31.641 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:31.641 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:24:31.641 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.283925+0000", 2024-09-13T11:24:31.641 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.9489999999999999e-06 2024-09-13T11:24:31.641 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:31.641 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:31.642 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:24:31.642 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.283960+0000", 2024-09-13T11:24:31.642 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.5063e-05 2024-09-13T11:24:31.642 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:31.642 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:31.642 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:24:31.642 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.283974+0000", 2024-09-13T11:24:31.642 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3546e-05 2024-09-13T11:24:31.642 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:31.642 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:31.642 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:24:31.642 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.283983+0000", 2024-09-13T11:24:31.642 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.2499999999999995e-06 2024-09-13T11:24:31.642 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:31.643 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:31.643 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:24:31.643 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.284061+0000", 2024-09-13T11:24:31.643 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.7946000000000002e-05 2024-09-13T11:24:31.643 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:31.643 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:31.643 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:24:31.643 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.284270+0000", 2024-09-13T11:24:31.643 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00020876799999999999 2024-09-13T11:24:31.643 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:31.643 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:31.643 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 1", 2024-09-13T11:24:31.643 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.284489+0000", 2024-09-13T11:24:31.643 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00021956800000000001 2024-09-13T11:24:31.644 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:31.644 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:31.644 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:24:31.644 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.286456+0000", 2024-09-13T11:24:31.644 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0019665809999999998 2024-09-13T11:24:31.644 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:31.644 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:31.644 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:31.644 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.931497+0000", 2024-09-13T11:24:31.644 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.64504147700000003 2024-09-13T11:24:31.644 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:31.644 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:31.644 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:24:31.645 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.931589+0000", 2024-09-13T11:24:31.645 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.2041999999999997e-05 2024-09-13T11:24:31.645 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:31.645 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:31.645 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:24:31.645 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.931678+0000", 2024-09-13T11:24:31.645 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.8444999999999998e-05 2024-09-13T11:24:31.645 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:31.645 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:24:31.645 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:31.645 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:31.645 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:31.645 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:104408 3.8 3:172a042f:::benchmark_data_smithi177_31116_object104407:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-09-13T11:24:31.646 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:24:20.318508+0000", 2024-09-13T11:24:31.646 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 11.237812525000001, 2024-09-13T11:24:31.646 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.115281618, 2024-09-13T11:24:31.646 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:24:31.646 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:24:31.646 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:31.646 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:24:31.646 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:24:31.646 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:31.646 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 104408 2024-09-13T11:24:31.646 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:31.646 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:24:31.646 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:31.646 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:24:31.646 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:20.318508+0000", 2024-09-13T11:24:31.647 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:31.647 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:31.647 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:31.647 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:24:31.647 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:20.318508+0000", 2024-09-13T11:24:31.647 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:31.647 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:31.647 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:31.647 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:24:31.647 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:20.318512+0000", 2024-09-13T11:24:31.647 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.9759999999999997e-06 2024-09-13T11:24:31.647 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:31.647 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:31.647 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:24:31.648 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:20.318573+0000", 2024-09-13T11:24:31.648 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.0748999999999997e-05 2024-09-13T11:24:31.648 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:31.648 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:31.648 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:24:31.648 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:20.318603+0000", 2024-09-13T11:24:31.648 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.9794e-05 2024-09-13T11:24:31.648 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:31.648 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:31.648 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:24:31.648 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:20.318618+0000", 2024-09-13T11:24:31.648 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5255999999999999e-05 2024-09-13T11:24:31.648 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:31.648 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:31.649 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:24:31.649 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:20.318941+0000", 2024-09-13T11:24:31.649 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00032326099999999998 2024-09-13T11:24:31.649 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:31.649 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:31.649 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:24:31.649 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:20.319174+0000", 2024-09-13T11:24:31.649 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000233291 2024-09-13T11:24:31.649 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:31.649 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:31.649 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 1", 2024-09-13T11:24:31.649 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:20.319427+0000", 2024-09-13T11:24:31.649 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000252995 2024-09-13T11:24:31.649 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:31.650 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:31.650 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:24:31.650 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:20.322857+0000", 2024-09-13T11:24:31.650 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0034298330000000002 2024-09-13T11:24:31.650 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:31.650 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:31.650 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:31.650 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:20.433490+0000", 2024-09-13T11:24:31.650 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.110633022 2024-09-13T11:24:31.650 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:31.650 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:31.650 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:24:31.650 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:20.433656+0000", 2024-09-13T11:24:31.650 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00016613899999999999 2024-09-13T11:24:31.651 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:31.651 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:31.651 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:24:31.651 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:20.433790+0000", 2024-09-13T11:24:31.651 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00013330199999999999 2024-09-13T11:24:31.651 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:31.651 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:24:31.651 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:31.651 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:31.651 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:31.651 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:107876 3.10 3:0ee50f6e:::benchmark_data_smithi177_31116_object107875:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116)", 2024-09-13T11:24:31.651 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:24:22.484287+0000", 2024-09-13T11:24:31.651 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 9.0720336689999996, 2024-09-13T11:24:31.651 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.13796866399999999, 2024-09-13T11:24:31.652 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:24:31.652 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:24:31.652 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:31.652 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:24:31.652 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:24:31.652 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:31.652 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 107876 2024-09-13T11:24:31.652 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:31.652 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:24:31.652 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:31.652 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:24:31.652 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:22.484287+0000", 2024-09-13T11:24:31.652 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:31.652 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:31.653 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:31.653 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:24:31.653 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:22.484287+0000", 2024-09-13T11:24:31.653 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:31.653 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:31.653 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:31.653 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:24:31.653 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:22.484290+0000", 2024-09-13T11:24:31.653 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.0489999999999999e-06 2024-09-13T11:24:31.653 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:31.653 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:31.653 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:24:31.653 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:22.484332+0000", 2024-09-13T11:24:31.653 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.2182999999999999e-05 2024-09-13T11:24:31.654 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:31.654 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:31.654 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:24:31.654 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:22.484350+0000", 2024-09-13T11:24:31.654 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.8258e-05 2024-09-13T11:24:31.654 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:31.654 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:31.654 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:24:31.654 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:22.484359+0000", 2024-09-13T11:24:31.654 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.0469999999999999e-06 2024-09-13T11:24:31.654 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:31.654 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:31.654 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:24:31.655 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:22.484432+0000", 2024-09-13T11:24:31.655 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.2683999999999997e-05 2024-09-13T11:24:31.655 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:31.655 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:31.655 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:24:31.655 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:22.484544+0000", 2024-09-13T11:24:31.655 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000111717 2024-09-13T11:24:31.655 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:31.655 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:31.655 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 0", 2024-09-13T11:24:31.655 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:22.484635+0000", 2024-09-13T11:24:31.655 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.0946000000000006e-05 2024-09-13T11:24:31.656 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:31.656 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:31.656 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:24:31.656 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:22.486650+0000", 2024-09-13T11:24:31.656 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0020149959999999998 2024-09-13T11:24:31.656 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:31.656 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:31.656 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:31.656 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:22.622144+0000", 2024-09-13T11:24:31.656 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.13549407699999999 2024-09-13T11:24:31.656 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:31.656 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:31.656 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:24:31.656 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:22.622207+0000", 2024-09-13T11:24:31.657 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.3355999999999994e-05 2024-09-13T11:24:31.657 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:31.657 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:31.657 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:24:31.657 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:22.622256+0000", 2024-09-13T11:24:31.657 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.8350999999999998e-05 2024-09-13T11:24:31.657 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:31.657 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:24:31.657 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:31.657 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:31.657 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:31.657 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:107893 3.10 3:08acae8a:::benchmark_data_smithi177_31116_object107892:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116)", 2024-09-13T11:24:31.657 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:24:22.498074+0000", 2024-09-13T11:24:31.657 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 9.0582470550000007, 2024-09-13T11:24:31.658 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.13439110100000001, 2024-09-13T11:24:31.658 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:24:31.658 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:24:31.658 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:31.658 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:24:31.658 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:24:31.658 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:31.658 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 107893 2024-09-13T11:24:31.658 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:31.658 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:24:31.658 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:31.658 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:24:31.658 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:22.498074+0000", 2024-09-13T11:24:31.658 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:31.659 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:31.659 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:31.659 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:24:31.659 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:22.498074+0000", 2024-09-13T11:24:31.659 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:31.659 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:31.659 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:31.659 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:24:31.659 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:22.498076+0000", 2024-09-13T11:24:31.659 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.481e-06 2024-09-13T11:24:31.659 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:31.659 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:31.659 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:24:31.659 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:22.498103+0000", 2024-09-13T11:24:31.660 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.7225000000000001e-05 2024-09-13T11:24:31.660 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:31.660 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:31.660 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:24:31.660 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:22.498116+0000", 2024-09-13T11:24:31.660 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3162999999999999e-05 2024-09-13T11:24:31.660 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:31.660 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:31.660 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:24:31.660 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:22.498123+0000", 2024-09-13T11:24:31.660 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.6640000000000001e-06 2024-09-13T11:24:31.660 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:31.660 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:31.660 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:24:31.661 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:22.498169+0000", 2024-09-13T11:24:31.661 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.5803999999999997e-05 2024-09-13T11:24:31.661 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:31.661 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:31.661 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:24:31.661 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:22.498268+0000", 2024-09-13T11:24:31.661 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.9376000000000005e-05 2024-09-13T11:24:31.661 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:31.661 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:31.661 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 0", 2024-09-13T11:24:31.661 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:22.498357+0000", 2024-09-13T11:24:31.661 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.8394999999999996e-05 2024-09-13T11:24:31.661 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:31.661 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:31.662 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:24:31.662 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:22.500225+0000", 2024-09-13T11:24:31.662 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0018679460000000001 2024-09-13T11:24:31.662 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:31.662 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:31.662 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:31.662 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:22.632359+0000", 2024-09-13T11:24:31.662 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.132133946 2024-09-13T11:24:31.662 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:31.662 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:31.662 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:24:31.662 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:22.632417+0000", 2024-09-13T11:24:31.662 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.8891999999999998e-05 2024-09-13T11:24:31.663 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:31.663 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:31.663 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:24:31.663 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:22.632465+0000", 2024-09-13T11:24:31.663 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.7209000000000002e-05 2024-09-13T11:24:31.663 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:31.663 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:24:31.663 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:31.663 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:31.663 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:31.663 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop_reply(client.4665.0:107876 3.10 e116/84 ondisk, result = 0)", 2024-09-13T11:24:31.663 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:24:22.501075+0000", 2024-09-13T11:24:31.663 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 9.0552453459999995, 2024-09-13T11:24:31.664 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.121189177, 2024-09-13T11:24:31.664 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:24:31.664 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:24:31.664 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:24:31.664 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:24:31.664 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:31.664 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:24:31.664 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:22.501075+0000", 2024-09-13T11:24:31.664 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:31.664 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:31.664 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:31.664 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:24:31.664 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:22.501075+0000", 2024-09-13T11:24:31.664 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:31.665 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:31.665 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:31.665 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:24:31.665 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:22.501076+0000", 2024-09-13T11:24:31.665 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.127e-06 2024-09-13T11:24:31.665 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:31.665 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:31.665 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:24:31.665 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:22.501080+0000", 2024-09-13T11:24:31.665 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.5889999999999999e-06 2024-09-13T11:24:31.665 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:31.665 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:31.665 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:24:31.665 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:22.622054+0000", 2024-09-13T11:24:31.666 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.12097395900000001 2024-09-13T11:24:31.666 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:31.666 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:31.666 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:24:31.666 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:22.622064+0000", 2024-09-13T11:24:31.666 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0101999999999999e-05 2024-09-13T11:24:31.666 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:31.666 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:31.666 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:24:31.666 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:22.622127+0000", 2024-09-13T11:24:31.666 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.2818999999999996e-05 2024-09-13T11:24:31.666 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:31.666 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:31.666 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:24:31.667 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:22.622137+0000", 2024-09-13T11:24:31.667 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0105999999999999e-05 2024-09-13T11:24:31.667 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:31.667 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:31.667 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:24:31.667 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:22.622264+0000", 2024-09-13T11:24:31.667 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00012747499999999999 2024-09-13T11:24:31.667 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:31.667 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:24:31.667 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:31.667 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:31.667 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:31.667 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:108010 3.10 3:0890c949:::benchmark_data_smithi177_31116_object108009:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116)", 2024-09-13T11:24:31.668 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:24:22.633603+0000", 2024-09-13T11:24:31.668 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 8.9227177179999995, 2024-09-13T11:24:31.668 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.142590525, 2024-09-13T11:24:31.668 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:24:31.668 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:24:31.668 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:31.668 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:24:31.668 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:24:31.668 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:31.668 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 108010 2024-09-13T11:24:31.668 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:31.668 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:24:31.668 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:31.668 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:24:31.669 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:22.633603+0000", 2024-09-13T11:24:31.669 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:31.669 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:31.669 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:31.669 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:24:31.669 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:22.633603+0000", 2024-09-13T11:24:31.669 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:31.669 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:31.669 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:31.669 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:24:31.670 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:22.633604+0000", 2024-09-13T11:24:31.670 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2330000000000001e-06 2024-09-13T11:24:31.670 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:31.670 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:31.670 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:24:31.670 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:22.633666+0000", 2024-09-13T11:24:31.670 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.1976000000000004e-05 2024-09-13T11:24:31.670 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:31.670 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:31.670 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:24:31.670 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:22.633678+0000", 2024-09-13T11:24:31.670 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2265e-05 2024-09-13T11:24:31.670 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:31.670 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:31.671 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:24:31.671 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:22.633684+0000", 2024-09-13T11:24:31.671 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.5559999999999998e-06 2024-09-13T11:24:31.671 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:31.671 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:31.671 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:24:31.671 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:22.633745+0000", 2024-09-13T11:24:31.671 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.1211999999999996e-05 2024-09-13T11:24:31.671 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:31.671 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:31.671 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:24:31.671 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:22.633877+0000", 2024-09-13T11:24:31.671 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000132127 2024-09-13T11:24:31.671 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:31.672 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:31.672 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 0", 2024-09-13T11:24:31.672 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:22.634023+0000", 2024-09-13T11:24:31.672 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00014526 2024-09-13T11:24:31.672 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:31.672 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:31.672 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:24:31.672 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:22.635864+0000", 2024-09-13T11:24:31.672 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0018414429999999999 2024-09-13T11:24:31.672 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:31.672 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:31.672 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:31.672 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:22.776065+0000", 2024-09-13T11:24:31.672 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.14020052399999999 2024-09-13T11:24:31.673 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:31.673 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:31.673 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:24:31.673 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:22.776137+0000", 2024-09-13T11:24:31.673 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.2067e-05 2024-09-13T11:24:31.673 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:31.673 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:31.673 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:24:31.673 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:22.776193+0000", 2024-09-13T11:24:31.673 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.6861999999999999e-05 2024-09-13T11:24:31.673 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:31.673 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:24:31.673 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:31.673 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:31.674 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:24:31.674 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-13T11:24:31.675 INFO:teuthology.orchestra.run.smithi001.stdout:osd.7: {} 2024-09-13T11:24:31.675 INFO:teuthology.orchestra.run.smithi001.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T11:24:31.676 INFO:teuthology.orchestra.run.smithi001.stdout:osd.8: {} 2024-09-13T11:24:31.676 INFO:teuthology.orchestra.run.smithi001.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-13T11:24:31.676 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-09-13T11:24:31.677 INFO:teuthology.orchestra.run.smithi001.stdout:osd.9: {} 2024-09-13T11:24:31.677 INFO:teuthology.orchestra.run.smithi001.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-13T11:24:31.678 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-14 --journal-path /var/lib/ceph/osd/ceph-14/journal --force --op remove --pgid 3.2 2024-09-13T11:24:31.678 INFO:teuthology.orchestra.run.smithi001.stdout:osd.10: {} 2024-09-13T11:24:31.678 INFO:teuthology.orchestra.run.smithi001.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-13T11:24:31.679 INFO:tasks.ceph.osd.3.smithi177.stderr:2024-09-13T11:24:31.671+0000 7f10adba4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26574) UID: 0 2024-09-13T11:24:31.693 INFO:teuthology.orchestra.run.smithi001.stdout:osd.11: {} 2024-09-13T11:24:31.693 INFO:teuthology.orchestra.run.smithi001.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-13T11:24:31.720 INFO:teuthology.orchestra.run.smithi001.stdout:osd.12: {} 2024-09-13T11:24:31.720 INFO:teuthology.orchestra.run.smithi001.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-13T11:24:31.746 INFO:teuthology.orchestra.run.smithi001.stdout:osd.13: {} 2024-09-13T11:24:31.747 INFO:teuthology.orchestra.run.smithi001.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-13T11:24:31.747 INFO:teuthology.orchestra.run.smithi001.stderr:Error ENXIO: problem getting command descriptions from osd.14 2024-09-13T11:24:31.774 INFO:teuthology.orchestra.run.smithi001.stdout:osd.15: {} 2024-09-13T11:24:31.774 INFO:teuthology.orchestra.run.smithi001.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-13T11:24:31.775 INFO:tasks.ceph.osd.7.smithi177.stderr:2024-09-13T11:24:31.771+0000 7f897b51d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26575) UID: 0 2024-09-13T11:24:31.778 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-13T11:24:31.778 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-13T11:24:31.853 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-13T11:24:31.853 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [], 2024-09-13T11:24:31.853 INFO:teuthology.orchestra.run.smithi162.stdout: "num_ops": 0 2024-09-13T11:24:31.853 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-13T11:24:31.869 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-09-13T11:24:31.876 INFO:tasks.ceph.osd.9.smithi092.stderr:2024-09-13T11:24:31.870+0000 7ff5ab3d0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26348) UID: 0 2024-09-13T11:24:31.977 INFO:tasks.ceph.osd.4.smithi001.stderr:2024-09-13T11:24:31.975+0000 7ff8e0218640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26310) UID: 0 2024-09-13T11:24:31.995 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-13T11:24:31.995 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [], 2024-09-13T11:24:31.996 INFO:teuthology.orchestra.run.smithi162.stdout: "complaint_time": 30, 2024-09-13T11:24:31.996 INFO:teuthology.orchestra.run.smithi162.stdout: "num_blocked_ops": 0 2024-09-13T11:24:31.996 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-13T11:24:32.010 INFO:tasks.radosbench.radosbench.0.smithi177.stdout: 125 16 113494 113478 56.7333 57.4375 0.00470553 0.0176198 2024-09-13T11:24:32.012 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-09-13T11:24:32.077 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-09-13T11:24:32.072+0000 7f97befc9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26315) UID: 0 2024-09-13T11:24:32.143 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-13T11:24:32.143 INFO:teuthology.orchestra.run.smithi162.stdout: "size": 20, 2024-09-13T11:24:32.144 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 600, 2024-09-13T11:24:32.144 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [ 2024-09-13T11:24:32.144 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:32.144 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:25395 3.3 3:c1df8ccf:::benchmark_data_smithi177_31116_object25394:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e55)", 2024-09-13T11:24:32.144 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:04.465425+0000", 2024-09-13T11:24:32.144 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 87.675720796999997, 2024-09-13T11:24:32.144 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.18641877200000001, 2024-09-13T11:24:32.144 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:24:32.144 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:24:32.144 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:32.144 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:24:32.144 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:24:32.145 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:32.145 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 25395 2024-09-13T11:24:32.145 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:32.145 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:24:32.145 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:32.145 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:24:32.145 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.465425+0000", 2024-09-13T11:24:32.145 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:32.145 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:32.145 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:32.145 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:24:32.145 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.465425+0000", 2024-09-13T11:24:32.145 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:32.145 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:32.146 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:32.146 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:24:32.146 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.465427+0000", 2024-09-13T11:24:32.146 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.9970000000000001e-06 2024-09-13T11:24:32.146 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:32.146 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:32.146 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:24:32.146 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.465516+0000", 2024-09-13T11:24:32.147 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.8181000000000002e-05 2024-09-13T11:24:32.148 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:32.148 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:32.148 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:24:32.148 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.465534+0000", 2024-09-13T11:24:32.148 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.8204000000000001e-05 2024-09-13T11:24:32.148 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:32.148 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:32.148 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:24:32.148 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.465548+0000", 2024-09-13T11:24:32.148 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4195999999999999e-05 2024-09-13T11:24:32.148 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:32.148 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:32.148 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:24:32.149 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.468249+0000", 2024-09-13T11:24:32.149 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0027014729999999998 2024-09-13T11:24:32.149 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:32.149 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:32.149 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:24:32.149 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.468626+0000", 2024-09-13T11:24:32.149 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000376721 2024-09-13T11:24:32.149 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:32.149 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:32.149 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 7", 2024-09-13T11:24:32.149 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.469016+0000", 2024-09-13T11:24:32.149 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00038975000000000001 2024-09-13T11:24:32.149 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:32.149 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:32.150 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:24:32.150 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.471140+0000", 2024-09-13T11:24:32.150 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0021243400000000002 2024-09-13T11:24:32.150 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:32.150 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:32.150 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:32.150 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.651698+0000", 2024-09-13T11:24:32.150 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.18055781700000001 2024-09-13T11:24:32.150 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:32.150 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:32.150 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:24:32.150 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.651781+0000", 2024-09-13T11:24:32.150 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.3350999999999995e-05 2024-09-13T11:24:32.151 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:32.151 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:32.151 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:24:32.151 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.651844+0000", 2024-09-13T11:24:32.151 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.2742000000000003e-05 2024-09-13T11:24:32.151 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:32.151 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:24:32.151 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:32.151 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:32.151 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:32.151 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:45810 3.17 3.9bd13e57 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e69)", 2024-09-13T11:24:32.151 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:21.347960+0000", 2024-09-13T11:24:32.151 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 70.793185843000003, 2024-09-13T11:24:32.152 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.56814464200000003, 2024-09-13T11:24:32.152 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:24:32.152 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:24:32.152 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-09-13T11:24:32.152 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:24:32.152 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:24:32.152 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:32.152 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 45810 2024-09-13T11:24:32.152 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:32.152 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:24:32.152 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:32.152 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:24:32.152 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:21.347960+0000", 2024-09-13T11:24:32.152 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:32.153 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:32.153 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:32.153 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:24:32.153 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:21.347960+0000", 2024-09-13T11:24:32.153 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:32.153 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:32.153 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:32.153 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:24:32.153 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:21.347965+0000", 2024-09-13T11:24:32.153 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.6720000000000003e-06 2024-09-13T11:24:32.153 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:32.153 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:32.153 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:24:32.154 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:21.348019+0000", 2024-09-13T11:24:32.154 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.4391000000000002e-05 2024-09-13T11:24:32.154 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:32.154 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:32.154 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:24:32.154 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:21.348052+0000", 2024-09-13T11:24:32.154 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.2246000000000003e-05 2024-09-13T11:24:32.154 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:32.154 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:32.154 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:24:32.154 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:21.348072+0000", 2024-09-13T11:24:32.154 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.0622e-05 2024-09-13T11:24:32.154 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:32.155 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:32.155 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:24:32.155 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:21.348278+0000", 2024-09-13T11:24:32.155 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000205391 2024-09-13T11:24:32.155 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:32.155 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:32.155 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-09-13T11:24:32.155 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:21.348292+0000", 2024-09-13T11:24:32.155 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4759e-05 2024-09-13T11:24:32.155 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:32.155 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:32.155 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:24:32.155 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:21.916064+0000", 2024-09-13T11:24:32.155 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.56777128600000004 2024-09-13T11:24:32.156 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:32.156 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:32.156 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:24:32.156 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:21.916105+0000", 2024-09-13T11:24:32.156 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.1275e-05 2024-09-13T11:24:32.156 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:32.156 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:24:32.156 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:32.156 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:32.156 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:32.156 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:54653 3.23 3:c7a54800:::benchmark_data_smithi177_31116_object54652:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e75)", 2024-09-13T11:24:32.156 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:30.979059+0000", 2024-09-13T11:24:32.156 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 61.162087344, 2024-09-13T11:24:32.157 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.97284747100000002, 2024-09-13T11:24:32.157 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:24:32.157 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:24:32.157 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-09-13T11:24:32.157 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:24:32.157 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:24:32.157 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:32.157 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 54653 2024-09-13T11:24:32.157 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:32.157 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:24:32.157 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:32.158 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:24:32.158 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.979059+0000", 2024-09-13T11:24:32.158 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:32.158 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:32.158 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:32.158 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:24:32.158 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.979059+0000", 2024-09-13T11:24:32.158 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:32.158 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:32.158 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:32.158 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:24:32.158 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.979061+0000", 2024-09-13T11:24:32.159 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.4279999999999999e-06 2024-09-13T11:24:32.159 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:32.159 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:32.159 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:24:32.159 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.979114+0000", 2024-09-13T11:24:32.159 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.3254000000000002e-05 2024-09-13T11:24:32.159 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:32.159 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:32.159 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:24:32.159 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.979157+0000", 2024-09-13T11:24:32.159 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.2018000000000003e-05 2024-09-13T11:24:32.160 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:32.160 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:32.160 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:24:32.160 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.979172+0000", 2024-09-13T11:24:32.160 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5636000000000001e-05 2024-09-13T11:24:32.160 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:32.160 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:32.160 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:24:32.160 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:31.045109+0000", 2024-09-13T11:24:32.160 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.065936982000000005 2024-09-13T11:24:32.160 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:32.160 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:32.160 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-09-13T11:24:32.160 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:31.045121+0000", 2024-09-13T11:24:32.161 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1608e-05 2024-09-13T11:24:32.161 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:32.161 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:32.161 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:24:32.161 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:31.951636+0000", 2024-09-13T11:24:32.161 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.90651503899999997 2024-09-13T11:24:32.161 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:32.161 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:32.161 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:24:32.161 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:31.951906+0000", 2024-09-13T11:24:32.161 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000270506 2024-09-13T11:24:32.162 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:32.162 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:24:32.162 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:32.162 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:32.162 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:32.162 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPush(3.18 76/70 [PushOp(3:1a32c9cd:::benchmark_data_smithi177_31116_object12833:head, version: 49'818, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:1a32c9cd:::benchmark_data_smithi177_31116_object12833:head@49'818, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:24:32.162 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:32.522305+0000", 2024-09-13T11:24:32.162 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 59.618841259999996, 2024-09-13T11:24:32.162 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.40201207100000003, 2024-09-13T11:24:32.162 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:24:32.162 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:24:32.162 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:24:32.163 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:24:32.163 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:32.163 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:24:32.163 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.522305+0000", 2024-09-13T11:24:32.163 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:32.163 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:32.163 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:32.163 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:24:32.163 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.522305+0000", 2024-09-13T11:24:32.163 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:32.163 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:32.163 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:32.163 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:24:32.163 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.522309+0000", 2024-09-13T11:24:32.164 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.9829999999999998e-06 2024-09-13T11:24:32.164 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:32.164 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:32.164 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:24:32.164 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.522377+0000", 2024-09-13T11:24:32.164 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.8069000000000002e-05 2024-09-13T11:24:32.164 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:32.164 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:32.164 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:24:32.164 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.921950+0000", 2024-09-13T11:24:32.164 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.399573438 2024-09-13T11:24:32.164 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:32.164 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:32.165 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:24:32.165 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.921983+0000", 2024-09-13T11:24:32.165 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.2858000000000003e-05 2024-09-13T11:24:32.165 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:32.165 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:32.165 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:24:32.165 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.922389+0000", 2024-09-13T11:24:32.165 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00040612999999999999 2024-09-13T11:24:32.165 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:32.165 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:32.165 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:24:32.165 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.922421+0000", 2024-09-13T11:24:32.165 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.1356999999999998e-05 2024-09-13T11:24:32.166 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:32.166 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:32.166 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:24:32.166 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.924317+0000", 2024-09-13T11:24:32.166 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0018962359999999999 2024-09-13T11:24:32.166 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:32.166 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:24:32.166 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:32.166 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:32.166 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:32.166 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPush(3.17 76/70 [PushOp(3:ebcbce67:::benchmark_data_smithi177_31116_object18268:head, version: 53'1102, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:ebcbce67:::benchmark_data_smithi177_31116_object18268:head@53'1102, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:24:32.166 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:32.638908+0000", 2024-09-13T11:24:32.166 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 59.502237811999997, 2024-09-13T11:24:32.166 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.229064031, 2024-09-13T11:24:32.167 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:24:32.167 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:24:32.167 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:24:32.167 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:24:32.167 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:32.167 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:24:32.167 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.638908+0000", 2024-09-13T11:24:32.167 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:32.167 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:32.167 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:32.167 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:24:32.167 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.638908+0000", 2024-09-13T11:24:32.167 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:32.167 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:32.167 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:32.168 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:24:32.168 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.638910+0000", 2024-09-13T11:24:32.168 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.7379999999999999e-06 2024-09-13T11:24:32.168 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:32.168 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:32.168 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:24:32.168 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.638975+0000", 2024-09-13T11:24:32.168 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.4937000000000005e-05 2024-09-13T11:24:32.168 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:32.168 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:32.168 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:24:32.168 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.865860+0000", 2024-09-13T11:24:32.168 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.22688509800000001 2024-09-13T11:24:32.168 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:32.169 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:32.169 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:24:32.169 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.865890+0000", 2024-09-13T11:24:32.169 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.0273e-05 2024-09-13T11:24:32.169 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:32.169 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:32.169 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:24:32.169 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.866143+0000", 2024-09-13T11:24:32.169 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00025212800000000001 2024-09-13T11:24:32.169 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:32.169 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:32.169 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:24:32.169 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.866181+0000", 2024-09-13T11:24:32.169 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.8945000000000001e-05 2024-09-13T11:24:32.170 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:32.170 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:32.170 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:24:32.170 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.867972+0000", 2024-09-13T11:24:32.170 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001790912 2024-09-13T11:24:32.170 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:32.170 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:24:32.170 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:32.170 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:32.170 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:32.170 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:57435 3.3 3:c0db4777:::benchmark_data_smithi177_31116_object57434:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e80)", 2024-09-13T11:24:32.170 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:37.436034+0000", 2024-09-13T11:24:32.170 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 54.705112352, 2024-09-13T11:24:32.170 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.35949142000000001, 2024-09-13T11:24:32.171 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:24:32.171 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:24:32.171 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:32.171 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:24:32.171 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:24:32.171 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:32.171 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 57435 2024-09-13T11:24:32.171 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:32.171 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:24:32.171 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:32.171 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:24:32.171 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:37.436034+0000", 2024-09-13T11:24:32.171 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:32.171 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:32.172 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:32.172 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:24:32.172 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:37.436034+0000", 2024-09-13T11:24:32.172 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:32.172 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:32.172 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:32.172 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:24:32.172 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:37.436037+0000", 2024-09-13T11:24:32.172 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.7829999999999999e-06 2024-09-13T11:24:32.172 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:32.172 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:32.172 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:24:32.172 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:37.436090+0000", 2024-09-13T11:24:32.173 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.3665999999999997e-05 2024-09-13T11:24:32.173 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:32.173 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:32.173 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:24:32.173 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:37.436112+0000", 2024-09-13T11:24:32.173 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.1556000000000002e-05 2024-09-13T11:24:32.173 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:32.173 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:32.173 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:24:32.173 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:37.436122+0000", 2024-09-13T11:24:32.173 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0098000000000001e-05 2024-09-13T11:24:32.173 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:32.174 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:32.174 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:24:32.174 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:37.436187+0000", 2024-09-13T11:24:32.174 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.4857999999999995e-05 2024-09-13T11:24:32.174 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:32.174 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:32.174 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:24:32.174 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:37.436285+0000", 2024-09-13T11:24:32.174 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.8643999999999999e-05 2024-09-13T11:24:32.174 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:32.174 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:32.174 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 7", 2024-09-13T11:24:32.174 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:37.436423+0000", 2024-09-13T11:24:32.174 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00013809299999999999 2024-09-13T11:24:32.175 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:32.175 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:32.175 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:24:32.175 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:37.440125+0000", 2024-09-13T11:24:32.175 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0037015239999999999 2024-09-13T11:24:32.175 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:32.175 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:32.175 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:32.175 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:37.795397+0000", 2024-09-13T11:24:32.175 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.35527234400000002 2024-09-13T11:24:32.175 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:32.175 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:32.175 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:24:32.176 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:37.795485+0000", 2024-09-13T11:24:32.176 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.7538000000000001e-05 2024-09-13T11:24:32.176 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:32.176 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:32.176 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:24:32.176 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:37.795525+0000", 2024-09-13T11:24:32.176 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.0315999999999998e-05 2024-09-13T11:24:32.176 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:32.176 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:24:32.176 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:32.176 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:32.176 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:32.177 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPush(3.1f 80/78 [PushOp(3:fc3ed0c7:::benchmark_data_smithi177_31116_object672:head, version: 43'36, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:fc3ed0c7:::benchmark_data_smithi177_31116_object672:head@43'36, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:24:32.177 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:37.801754+0000", 2024-09-13T11:24:32.177 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 54.339391992000003, 2024-09-13T11:24:32.177 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.209613519, 2024-09-13T11:24:32.177 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:24:32.177 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:24:32.177 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:24:32.177 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:24:32.177 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:32.177 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:24:32.177 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:37.801754+0000", 2024-09-13T11:24:32.177 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:32.177 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:32.178 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:32.178 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:24:32.178 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:37.801754+0000", 2024-09-13T11:24:32.178 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:32.178 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:32.178 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:32.178 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:24:32.178 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:37.801756+0000", 2024-09-13T11:24:32.178 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.804e-06 2024-09-13T11:24:32.178 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:32.178 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:32.178 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:24:32.178 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:37.801947+0000", 2024-09-13T11:24:32.178 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000190636 2024-09-13T11:24:32.179 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:32.179 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:32.179 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:24:32.179 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:38.010478+0000", 2024-09-13T11:24:32.179 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.20853091100000001 2024-09-13T11:24:32.179 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:32.179 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:32.179 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:24:32.179 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:38.010506+0000", 2024-09-13T11:24:32.179 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.8793999999999999e-05 2024-09-13T11:24:32.179 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:32.179 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:32.179 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:24:32.179 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:38.010601+0000", 2024-09-13T11:24:32.180 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.4786999999999999e-05 2024-09-13T11:24:32.180 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:32.180 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:32.180 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:24:32.180 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:38.010617+0000", 2024-09-13T11:24:32.180 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5790999999999998e-05 2024-09-13T11:24:32.180 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:32.180 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:32.180 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:24:32.180 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:38.011368+0000", 2024-09-13T11:24:32.180 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00075079599999999997 2024-09-13T11:24:32.180 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:32.180 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:24:32.181 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:32.181 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:32.181 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:32.181 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:57985 3.3 3:c330bb03:::benchmark_data_smithi177_31116_object57984:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e80)", 2024-09-13T11:24:32.181 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:38.006112+0000", 2024-09-13T11:24:32.181 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 54.135034124000001, 2024-09-13T11:24:32.181 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.18277887900000001, 2024-09-13T11:24:32.181 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:24:32.181 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:24:32.181 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:32.181 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:24:32.181 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:24:32.181 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:32.182 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 57985 2024-09-13T11:24:32.182 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:32.182 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:24:32.182 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:32.182 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:24:32.182 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:38.006112+0000", 2024-09-13T11:24:32.182 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:32.182 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:32.182 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:32.182 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:24:32.182 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:38.006112+0000", 2024-09-13T11:24:32.182 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:32.182 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:32.182 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:32.182 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:24:32.183 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:38.006115+0000", 2024-09-13T11:24:32.183 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.6400000000000001e-06 2024-09-13T11:24:32.183 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:32.183 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:32.183 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:24:32.183 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:38.006162+0000", 2024-09-13T11:24:32.183 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.7599000000000003e-05 2024-09-13T11:24:32.183 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:32.183 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:32.183 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:24:32.183 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:38.006182+0000", 2024-09-13T11:24:32.183 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.9281999999999999e-05 2024-09-13T11:24:32.184 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:32.184 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:32.184 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:24:32.184 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:38.006191+0000", 2024-09-13T11:24:32.184 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.6490000000000003e-06 2024-09-13T11:24:32.184 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:32.184 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:32.184 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:24:32.184 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:38.006243+0000", 2024-09-13T11:24:32.184 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.1511999999999997e-05 2024-09-13T11:24:32.184 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:32.184 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:32.184 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:24:32.185 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:38.006333+0000", 2024-09-13T11:24:32.185 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.9843e-05 2024-09-13T11:24:32.185 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:32.185 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:32.185 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 7", 2024-09-13T11:24:32.185 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:38.006480+0000", 2024-09-13T11:24:32.185 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00014701699999999999 2024-09-13T11:24:32.185 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:32.185 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:32.185 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:24:32.185 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:38.008806+0000", 2024-09-13T11:24:32.185 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.002326172 2024-09-13T11:24:32.185 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:32.186 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:32.186 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:32.186 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:38.188815+0000", 2024-09-13T11:24:32.186 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.18000961100000001 2024-09-13T11:24:32.186 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:32.186 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:32.186 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:24:32.186 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:38.188857+0000", 2024-09-13T11:24:32.186 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.1591999999999998e-05 2024-09-13T11:24:32.186 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:32.186 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:32.186 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:24:32.186 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:38.188891+0000", 2024-09-13T11:24:32.186 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.3961999999999998e-05 2024-09-13T11:24:32.187 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:32.187 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:24:32.187 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:32.187 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:32.187 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:32.187 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:60433 3.3 3:c01f7ddc:::benchmark_data_smithi177_31116_object60432:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e85)", 2024-09-13T11:24:32.187 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:42.199382+0000", 2024-09-13T11:24:32.187 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 49.941764460999998, 2024-09-13T11:24:32.187 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.33918965200000001, 2024-09-13T11:24:32.187 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:24:32.187 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:24:32.187 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:32.188 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:24:32.188 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:24:32.188 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:32.188 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 60433 2024-09-13T11:24:32.188 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:32.188 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:24:32.188 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:32.188 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:24:32.188 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.199382+0000", 2024-09-13T11:24:32.188 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:32.188 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:32.188 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:32.188 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:24:32.189 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.199382+0000", 2024-09-13T11:24:32.189 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:32.189 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:32.189 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:32.189 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:24:32.189 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.199386+0000", 2024-09-13T11:24:32.189 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.8759999999999998e-06 2024-09-13T11:24:32.189 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:32.189 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:32.189 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:24:32.189 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.199455+0000", 2024-09-13T11:24:32.189 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.9305000000000001e-05 2024-09-13T11:24:32.189 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:32.189 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:32.190 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:24:32.190 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.199482+0000", 2024-09-13T11:24:32.190 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.6749000000000001e-05 2024-09-13T11:24:32.190 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:32.190 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:32.190 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:24:32.190 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.199503+0000", 2024-09-13T11:24:32.190 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.1137000000000001e-05 2024-09-13T11:24:32.190 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:32.190 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:32.190 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:24:32.190 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.199602+0000", 2024-09-13T11:24:32.190 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.8743e-05 2024-09-13T11:24:32.190 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:32.191 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:32.191 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:24:32.191 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.199784+0000", 2024-09-13T11:24:32.191 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00018265100000000001 2024-09-13T11:24:32.191 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:32.191 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:32.191 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 7", 2024-09-13T11:24:32.191 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.199955+0000", 2024-09-13T11:24:32.191 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00017067100000000001 2024-09-13T11:24:32.192 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:32.192 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:32.192 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:24:32.192 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.202075+0000", 2024-09-13T11:24:32.192 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0021206749999999998 2024-09-13T11:24:32.192 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:32.192 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:32.192 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:32.192 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.538395+0000", 2024-09-13T11:24:32.192 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.33631963599999998 2024-09-13T11:24:32.192 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:32.192 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:32.193 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:24:32.193 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.538498+0000", 2024-09-13T11:24:32.193 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00010267099999999999 2024-09-13T11:24:32.193 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:32.193 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:32.193 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:24:32.193 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.538571+0000", 2024-09-13T11:24:32.193 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.3538e-05 2024-09-13T11:24:32.193 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:32.193 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:24:32.193 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:32.193 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:32.194 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:32.194 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop_reply(client.4665.0:60433 3.3 e85/75 ondisk, result = 0)", 2024-09-13T11:24:32.194 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:42.203186+0000", 2024-09-13T11:24:32.194 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 49.937960283000002, 2024-09-13T11:24:32.194 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.33540096699999999, 2024-09-13T11:24:32.194 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:24:32.194 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:24:32.194 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:24:32.194 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:24:32.194 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:32.194 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:24:32.194 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.203186+0000", 2024-09-13T11:24:32.194 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:32.194 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:32.195 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:32.195 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:24:32.195 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.203186+0000", 2024-09-13T11:24:32.195 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:32.195 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:32.195 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:32.195 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:24:32.195 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.203194+0000", 2024-09-13T11:24:32.195 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.0020000000000006e-06 2024-09-13T11:24:32.195 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:32.195 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:32.195 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:24:32.195 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.203202+0000", 2024-09-13T11:24:32.195 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.9829999999999996e-06 2024-09-13T11:24:32.196 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:32.196 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:32.196 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:24:32.196 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.538182+0000", 2024-09-13T11:24:32.196 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.33498029899999998 2024-09-13T11:24:32.196 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:32.196 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:32.196 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:24:32.196 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.538196+0000", 2024-09-13T11:24:32.196 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3807e-05 2024-09-13T11:24:32.196 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:32.196 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:32.196 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:24:32.197 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.538371+0000", 2024-09-13T11:24:32.197 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00017541200000000001 2024-09-13T11:24:32.197 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:32.197 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:32.197 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:24:32.197 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.538386+0000", 2024-09-13T11:24:32.197 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4833e-05 2024-09-13T11:24:32.197 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:32.197 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:32.197 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:24:32.197 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.538587+0000", 2024-09-13T11:24:32.197 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000200631 2024-09-13T11:24:32.198 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:32.198 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:24:32.198 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:32.198 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:32.198 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:32.198 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:69556 3.18 3:1d3dbeba:::benchmark_data_smithi177_31116_object69555:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94)", 2024-09-13T11:24:32.198 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:52.142180+0000", 2024-09-13T11:24:32.198 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 39.998966213000003, 2024-09-13T11:24:32.198 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.21280995899999999, 2024-09-13T11:24:32.198 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:24:32.198 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:24:32.198 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:32.198 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:24:32.199 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:24:32.199 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:32.199 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 69556 2024-09-13T11:24:32.199 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:32.199 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:24:32.199 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:32.199 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:24:32.199 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:52.142180+0000", 2024-09-13T11:24:32.199 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:32.200 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:32.200 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:32.200 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:24:32.200 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:52.142180+0000", 2024-09-13T11:24:32.200 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:32.200 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:32.200 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:32.200 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:24:32.200 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:52.142181+0000", 2024-09-13T11:24:32.200 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.3699999999999999e-07 2024-09-13T11:24:32.200 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:32.200 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:32.201 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:24:32.201 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:52.142212+0000", 2024-09-13T11:24:32.201 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.1334000000000002e-05 2024-09-13T11:24:32.201 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:32.201 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:32.201 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:24:32.201 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:52.142219+0000", 2024-09-13T11:24:32.201 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.9990000000000002e-06 2024-09-13T11:24:32.201 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:32.201 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:32.201 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:24:32.201 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:52.142224+0000", 2024-09-13T11:24:32.201 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.0259999999999998e-06 2024-09-13T11:24:32.202 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:32.202 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:32.202 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:24:32.202 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:52.142276+0000", 2024-09-13T11:24:32.202 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.1436e-05 2024-09-13T11:24:32.202 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:32.202 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:32.202 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:24:32.202 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:52.142396+0000", 2024-09-13T11:24:32.202 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00012011 2024-09-13T11:24:32.202 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:32.202 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:32.202 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 0", 2024-09-13T11:24:32.203 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:52.142503+0000", 2024-09-13T11:24:32.203 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000107701 2024-09-13T11:24:32.203 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:32.203 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:32.203 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:24:32.203 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:52.144566+0000", 2024-09-13T11:24:32.203 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.002062749 2024-09-13T11:24:32.203 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:32.203 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:32.203 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:32.203 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:52.354839+0000", 2024-09-13T11:24:32.203 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.21027254000000001 2024-09-13T11:24:32.203 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:32.204 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:32.204 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:24:32.204 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:52.354932+0000", 2024-09-13T11:24:32.204 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.3510000000000004e-05 2024-09-13T11:24:32.204 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:32.204 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:32.204 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:24:32.204 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:52.354990+0000", 2024-09-13T11:24:32.204 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.7617000000000001e-05 2024-09-13T11:24:32.204 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:32.204 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:24:32.204 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:32.204 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:32.205 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:32.205 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:82784 3.18 3:1e10e57e:::benchmark_data_smithi177_31116_object82783:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105)", 2024-09-13T11:24:32.205 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:24:03.815559+0000", 2024-09-13T11:24:32.205 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 28.325586815000001, 2024-09-13T11:24:32.205 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.19711926199999999, 2024-09-13T11:24:32.205 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:24:32.205 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:24:32.205 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:32.205 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:24:32.205 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:24:32.205 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:32.205 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 82784 2024-09-13T11:24:32.205 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:32.205 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:24:32.206 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:32.206 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:24:32.206 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:03.815559+0000", 2024-09-13T11:24:32.206 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:32.206 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:32.206 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:32.206 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:24:32.206 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:03.815559+0000", 2024-09-13T11:24:32.206 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:32.206 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:32.206 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:32.206 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:24:32.206 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:03.815568+0000", 2024-09-13T11:24:32.206 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.2029999999999992e-06 2024-09-13T11:24:32.207 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:32.207 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:32.207 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:24:32.207 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:03.815618+0000", 2024-09-13T11:24:32.207 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.0285999999999999e-05 2024-09-13T11:24:32.207 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:32.207 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:32.207 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:24:32.207 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:03.815646+0000", 2024-09-13T11:24:32.207 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.7747e-05 2024-09-13T11:24:32.207 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:32.207 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:32.207 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:24:32.208 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:03.815660+0000", 2024-09-13T11:24:32.208 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4633e-05 2024-09-13T11:24:32.208 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:32.208 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:32.208 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:24:32.208 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:03.815781+0000", 2024-09-13T11:24:32.208 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000120831 2024-09-13T11:24:32.208 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:32.208 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:32.208 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:24:32.208 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:03.816015+0000", 2024-09-13T11:24:32.208 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00023364600000000001 2024-09-13T11:24:32.208 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:32.208 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:32.209 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 0", 2024-09-13T11:24:32.209 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:03.816241+0000", 2024-09-13T11:24:32.209 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00022667600000000001 2024-09-13T11:24:32.209 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:32.209 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:32.209 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:24:32.209 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:03.819236+0000", 2024-09-13T11:24:32.209 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0029943130000000002 2024-09-13T11:24:32.209 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:32.209 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:32.209 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:32.209 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.012545+0000", 2024-09-13T11:24:32.209 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.19330974100000001 2024-09-13T11:24:32.210 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:32.210 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:32.210 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:24:32.210 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.012623+0000", 2024-09-13T11:24:32.210 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.7565999999999996e-05 2024-09-13T11:24:32.210 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:32.210 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:32.210 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:24:32.210 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.012679+0000", 2024-09-13T11:24:32.210 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.5619999999999999e-05 2024-09-13T11:24:32.210 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:32.210 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:24:32.210 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:32.211 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:32.211 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:32.211 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop_reply(client.4665.0:82784 3.18 e105/91 ondisk, result = 0)", 2024-09-13T11:24:32.211 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:24:03.819384+0000", 2024-09-13T11:24:32.211 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 28.321762535000001, 2024-09-13T11:24:32.211 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.19330499100000001, 2024-09-13T11:24:32.211 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:24:32.211 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:24:32.211 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:24:32.211 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:24:32.211 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:32.211 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:24:32.211 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:03.819384+0000", 2024-09-13T11:24:32.211 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:32.212 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:32.212 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:32.212 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:24:32.212 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:03.819384+0000", 2024-09-13T11:24:32.212 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:32.212 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:32.212 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:32.212 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:24:32.212 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:03.819393+0000", 2024-09-13T11:24:32.212 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.5429999999999999e-06 2024-09-13T11:24:32.212 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:32.212 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:32.212 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:24:32.212 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:03.819403+0000", 2024-09-13T11:24:32.213 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0309000000000001e-05 2024-09-13T11:24:32.213 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:32.213 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:32.213 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:24:32.213 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.010699+0000", 2024-09-13T11:24:32.213 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.19129547699999999 2024-09-13T11:24:32.213 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:32.213 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:32.213 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:24:32.213 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.010708+0000", 2024-09-13T11:24:32.213 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.6770000000000001e-06 2024-09-13T11:24:32.213 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:32.214 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:32.214 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:24:32.214 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.012528+0000", 2024-09-13T11:24:32.214 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00182087 2024-09-13T11:24:32.214 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:32.214 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:32.214 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:24:32.214 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.012538+0000", 2024-09-13T11:24:32.214 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.6399999999999992e-06 2024-09-13T11:24:32.214 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:32.214 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:32.214 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:24:32.214 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.012689+0000", 2024-09-13T11:24:32.215 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00015047500000000001 2024-09-13T11:24:32.215 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:32.215 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:24:32.215 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:32.215 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:32.215 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:32.215 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:83144 3.18 3:1c2a5085:::benchmark_data_smithi177_31116_object83143:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106)", 2024-09-13T11:24:32.215 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:24:04.186395+0000", 2024-09-13T11:24:32.215 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 27.954751164000001, 2024-09-13T11:24:32.215 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.29655527599999998, 2024-09-13T11:24:32.215 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:24:32.215 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:24:32.215 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:32.216 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:24:32.216 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:24:32.216 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:32.216 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 83144 2024-09-13T11:24:32.216 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:32.216 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:24:32.216 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:32.216 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:24:32.216 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.186395+0000", 2024-09-13T11:24:32.216 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:32.216 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:32.216 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:32.216 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:24:32.216 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.186395+0000", 2024-09-13T11:24:32.217 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:32.217 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:32.217 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:32.217 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:24:32.217 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.186398+0000", 2024-09-13T11:24:32.217 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.5299999999999999e-06 2024-09-13T11:24:32.217 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:32.217 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:32.217 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:24:32.217 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.186438+0000", 2024-09-13T11:24:32.217 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.0293000000000002e-05 2024-09-13T11:24:32.217 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:32.217 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:32.218 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:24:32.218 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.186450+0000", 2024-09-13T11:24:32.218 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2687000000000001e-05 2024-09-13T11:24:32.218 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:32.218 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:32.218 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:24:32.218 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.186456+0000", 2024-09-13T11:24:32.218 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.446e-06 2024-09-13T11:24:32.218 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:32.218 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:32.218 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:24:32.218 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.186503+0000", 2024-09-13T11:24:32.219 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.7491000000000003e-05 2024-09-13T11:24:32.219 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:32.219 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:32.219 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:24:32.219 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.186618+0000", 2024-09-13T11:24:32.219 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000114999 2024-09-13T11:24:32.219 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:32.219 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:32.219 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 0", 2024-09-13T11:24:32.219 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.186712+0000", 2024-09-13T11:24:32.219 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.3805000000000001e-05 2024-09-13T11:24:32.220 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:32.220 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:32.220 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:24:32.220 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.189753+0000", 2024-09-13T11:24:32.220 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.003040713 2024-09-13T11:24:32.220 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:32.220 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:32.220 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:32.220 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.482662+0000", 2024-09-13T11:24:32.220 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.29290945699999998 2024-09-13T11:24:32.220 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:32.220 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:32.220 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:24:32.221 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.482831+0000", 2024-09-13T11:24:32.221 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00016900699999999999 2024-09-13T11:24:32.221 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:32.221 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:32.221 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:24:32.221 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.482950+0000", 2024-09-13T11:24:32.221 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000118848 2024-09-13T11:24:32.221 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:32.221 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:24:32.221 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:32.221 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:32.221 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:32.221 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop_reply(client.4665.0:83144 3.18 e106/91 ondisk, result = 0)", 2024-09-13T11:24:32.222 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:24:04.188413+0000", 2024-09-13T11:24:32.222 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 27.952733154000001, 2024-09-13T11:24:32.222 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.294557562, 2024-09-13T11:24:32.222 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:24:32.222 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:24:32.222 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:24:32.222 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:24:32.222 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:32.222 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:24:32.222 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.188413+0000", 2024-09-13T11:24:32.222 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:32.222 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:32.223 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:32.223 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:24:32.223 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.188413+0000", 2024-09-13T11:24:32.223 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:32.223 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:32.223 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:32.223 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:24:32.223 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.188414+0000", 2024-09-13T11:24:32.223 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4309999999999999e-06 2024-09-13T11:24:32.223 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:32.223 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:32.223 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:24:32.223 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.188419+0000", 2024-09-13T11:24:32.223 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.2889999999999999e-06 2024-09-13T11:24:32.224 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:32.224 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:32.224 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:24:32.224 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.482442+0000", 2024-09-13T11:24:32.224 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.294023168 2024-09-13T11:24:32.224 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:32.224 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:32.224 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:24:32.224 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.482459+0000", 2024-09-13T11:24:32.224 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.7013000000000001e-05 2024-09-13T11:24:32.224 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:32.224 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:32.224 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:24:32.224 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.482615+0000", 2024-09-13T11:24:32.225 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00015620400000000001 2024-09-13T11:24:32.225 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:32.225 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:32.225 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:24:32.225 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.482644+0000", 2024-09-13T11:24:32.225 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.9093000000000001e-05 2024-09-13T11:24:32.225 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:32.225 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:32.225 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:24:32.225 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.482971+0000", 2024-09-13T11:24:32.225 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00032636399999999999 2024-09-13T11:24:32.225 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:32.225 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:24:32.225 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:32.225 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:32.226 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:32.226 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPush(3.1f 106/78 [PushOp(3:fd4be819:::benchmark_data_smithi177_31116_object54560:head, version: 74'3410, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:fd4be819:::benchmark_data_smithi177_31116_object54560:head@74'3410, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:24:32.226 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:24:04.397161+0000", 2024-09-13T11:24:32.226 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 27.743984744999999, 2024-09-13T11:24:32.226 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.51611096199999995, 2024-09-13T11:24:32.226 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:24:32.226 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:24:32.226 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:24:32.226 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:24:32.226 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:32.226 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:24:32.226 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.397161+0000", 2024-09-13T11:24:32.226 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:32.226 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:32.227 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:32.227 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:24:32.227 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.397161+0000", 2024-09-13T11:24:32.227 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:32.227 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:32.227 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:32.227 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:24:32.227 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.397163+0000", 2024-09-13T11:24:32.227 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4720000000000001e-06 2024-09-13T11:24:32.227 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:32.227 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:32.227 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:24:32.227 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.397228+0000", 2024-09-13T11:24:32.227 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.5141999999999994e-05 2024-09-13T11:24:32.227 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:32.228 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:32.228 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:24:32.228 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.911583+0000", 2024-09-13T11:24:32.228 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.51435473700000001 2024-09-13T11:24:32.228 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:32.228 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:32.228 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:24:32.228 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.911615+0000", 2024-09-13T11:24:32.228 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.2620000000000003e-05 2024-09-13T11:24:32.228 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:32.228 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:32.228 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:24:32.228 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.911759+0000", 2024-09-13T11:24:32.229 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00014376300000000001 2024-09-13T11:24:32.229 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:32.229 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:32.229 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:24:32.229 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.911779+0000", 2024-09-13T11:24:32.229 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.0003000000000001e-05 2024-09-13T11:24:32.229 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:32.229 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:32.229 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:24:32.229 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.913272+0000", 2024-09-13T11:24:32.229 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0014932249999999999 2024-09-13T11:24:32.229 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:32.230 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:24:32.230 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:32.230 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:32.230 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:32.230 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:84809 3.17 3:e9e52123:::benchmark_data_smithi177_31116_object84808:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-09-13T11:24:32.230 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:24:05.914353+0000", 2024-09-13T11:24:32.230 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 26.226792726999999, 2024-09-13T11:24:32.230 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.27210915899999999, 2024-09-13T11:24:32.230 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:24:32.230 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:24:32.230 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:32.230 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:24:32.230 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:24:32.231 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:32.231 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 84809 2024-09-13T11:24:32.231 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:32.231 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:24:32.231 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:32.231 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:24:32.231 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.914353+0000", 2024-09-13T11:24:32.231 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:32.231 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:32.231 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:32.231 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:24:32.231 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.914353+0000", 2024-09-13T11:24:32.232 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:32.232 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:32.232 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:32.232 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:24:32.232 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.914356+0000", 2024-09-13T11:24:32.232 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.8710000000000001e-06 2024-09-13T11:24:32.232 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:32.232 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:32.232 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:24:32.232 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.914443+0000", 2024-09-13T11:24:32.232 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.7002000000000005e-05 2024-09-13T11:24:32.232 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:32.232 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:32.232 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:24:32.233 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.914461+0000", 2024-09-13T11:24:32.233 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.7691000000000001e-05 2024-09-13T11:24:32.233 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:32.233 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:32.233 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:24:32.233 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.914471+0000", 2024-09-13T11:24:32.233 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0447e-05 2024-09-13T11:24:32.233 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:32.233 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:32.233 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:24:32.233 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.914550+0000", 2024-09-13T11:24:32.233 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.8683000000000005e-05 2024-09-13T11:24:32.233 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:32.233 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:32.234 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:24:32.234 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.914709+0000", 2024-09-13T11:24:32.234 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00015913200000000001 2024-09-13T11:24:32.234 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:32.234 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:32.234 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 3", 2024-09-13T11:24:32.234 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.914880+0000", 2024-09-13T11:24:32.235 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00017071499999999999 2024-09-13T11:24:32.235 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:32.235 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:32.235 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:24:32.235 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.916701+0000", 2024-09-13T11:24:32.235 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0018205369999999999 2024-09-13T11:24:32.235 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:32.235 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:32.235 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:32.235 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:06.186401+0000", 2024-09-13T11:24:32.235 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.26970018699999998 2024-09-13T11:24:32.236 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:32.236 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:32.236 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:24:32.236 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:06.186434+0000", 2024-09-13T11:24:32.236 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.3448999999999998e-05 2024-09-13T11:24:32.236 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:32.236 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:32.236 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:24:32.236 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:06.186463+0000", 2024-09-13T11:24:32.236 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.8445e-05 2024-09-13T11:24:32.236 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:32.236 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:24:32.236 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:32.237 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:32.237 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:32.237 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop_reply(client.4665.0:84809 3.17 e107/85 ondisk, result = 0)", 2024-09-13T11:24:32.237 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:24:05.916779+0000", 2024-09-13T11:24:32.237 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 26.224367256000001, 2024-09-13T11:24:32.237 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.26968993499999999, 2024-09-13T11:24:32.237 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:24:32.237 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:24:32.237 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:24:32.237 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:24:32.237 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:32.237 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:24:32.237 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.916779+0000", 2024-09-13T11:24:32.237 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:32.238 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:32.238 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:32.238 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:24:32.238 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.916779+0000", 2024-09-13T11:24:32.238 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:32.238 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:32.238 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:32.238 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:24:32.238 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.916780+0000", 2024-09-13T11:24:32.238 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.328e-06 2024-09-13T11:24:32.238 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:32.238 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:32.238 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:24:32.238 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.916789+0000", 2024-09-13T11:24:32.239 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.4460000000000001e-06 2024-09-13T11:24:32.239 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:32.239 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:32.239 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:24:32.239 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:06.186296+0000", 2024-09-13T11:24:32.239 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.26950707099999999 2024-09-13T11:24:32.239 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:32.239 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:32.239 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:24:32.239 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:06.186306+0000", 2024-09-13T11:24:32.239 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0137e-05 2024-09-13T11:24:32.239 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:32.239 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:32.239 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:24:32.239 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:06.186388+0000", 2024-09-13T11:24:32.240 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.2149000000000002e-05 2024-09-13T11:24:32.240 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:32.240 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:32.240 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:24:32.240 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:06.186397+0000", 2024-09-13T11:24:32.240 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.5210000000000002e-06 2024-09-13T11:24:32.240 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:32.240 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:32.240 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:24:32.240 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:06.186469+0000", 2024-09-13T11:24:32.240 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.2282999999999999e-05 2024-09-13T11:24:32.240 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:32.240 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:24:32.241 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:32.241 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:32.241 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:32.241 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:99111 3.1f 3:fca4b03f:::benchmark_data_smithi177_31116_object99110:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113)", 2024-09-13T11:24:32.241 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:24:15.967965+0000", 2024-09-13T11:24:32.241 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 16.173181542999998, 2024-09-13T11:24:32.241 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.18645621300000001, 2024-09-13T11:24:32.241 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:24:32.241 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:24:32.241 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:32.241 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:24:32.241 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:24:32.241 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:32.242 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 99111 2024-09-13T11:24:32.242 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:32.242 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:24:32.242 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:32.242 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:24:32.242 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:15.967965+0000", 2024-09-13T11:24:32.242 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:32.242 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:32.242 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:32.242 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:24:32.242 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:15.967965+0000", 2024-09-13T11:24:32.242 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:32.242 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:32.242 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:32.243 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:24:32.243 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:15.967967+0000", 2024-09-13T11:24:32.243 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.79e-06 2024-09-13T11:24:32.243 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:32.243 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:32.243 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:24:32.243 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:15.968023+0000", 2024-09-13T11:24:32.243 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.5600000000000003e-05 2024-09-13T11:24:32.243 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:32.243 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:32.243 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:24:32.243 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:15.968045+0000", 2024-09-13T11:24:32.243 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.1801e-05 2024-09-13T11:24:32.243 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:32.243 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:32.244 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:24:32.244 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:15.968058+0000", 2024-09-13T11:24:32.244 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3522e-05 2024-09-13T11:24:32.244 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:32.244 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:32.244 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:24:32.244 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:15.968175+0000", 2024-09-13T11:24:32.244 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00011650900000000001 2024-09-13T11:24:32.244 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:32.244 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:32.244 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:24:32.244 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:15.968499+0000", 2024-09-13T11:24:32.244 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000324364 2024-09-13T11:24:32.245 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:32.245 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:32.245 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 14", 2024-09-13T11:24:32.245 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:15.968796+0000", 2024-09-13T11:24:32.245 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00029638100000000002 2024-09-13T11:24:32.245 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:32.245 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:32.245 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:24:32.245 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:15.971845+0000", 2024-09-13T11:24:32.245 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.003048931 2024-09-13T11:24:32.245 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:32.245 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:32.245 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:32.245 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:16.154118+0000", 2024-09-13T11:24:32.246 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.18227338700000001 2024-09-13T11:24:32.246 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:32.246 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:32.246 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:24:32.246 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:16.154293+0000", 2024-09-13T11:24:32.246 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00017466500000000001 2024-09-13T11:24:32.246 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:32.246 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:32.246 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:24:32.246 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:16.154421+0000", 2024-09-13T11:24:32.246 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00012826299999999999 2024-09-13T11:24:32.246 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:32.246 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:24:32.246 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:32.246 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:32.247 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:32.247 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPush(3.2a 121/120 [PushOp(3:5595108b:::benchmark_data_smithi177_31116_object899:head, version: 43'62, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:5595108b:::benchmark_data_smithi177_31116_object899:head@43'62, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:24:32.247 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:24:27.352355+0000", 2024-09-13T11:24:32.247 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 4.7887914499999997, 2024-09-13T11:24:32.247 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.61628666099999996, 2024-09-13T11:24:32.247 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:24:32.247 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:24:32.247 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:24:32.247 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:24:32.247 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:32.247 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:24:32.247 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:27.352355+0000", 2024-09-13T11:24:32.247 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:32.247 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:32.248 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:32.248 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:24:32.248 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:27.352355+0000", 2024-09-13T11:24:32.248 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:32.248 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:32.248 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:32.248 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:24:32.248 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:27.352357+0000", 2024-09-13T11:24:32.248 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.328e-06 2024-09-13T11:24:32.248 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:32.248 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:32.248 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:24:32.248 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:27.352428+0000", 2024-09-13T11:24:32.248 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.1118999999999995e-05 2024-09-13T11:24:32.249 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:32.249 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:32.249 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:24:32.249 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:27.966950+0000", 2024-09-13T11:24:32.249 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.61452202499999997 2024-09-13T11:24:32.249 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:32.249 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:32.249 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:24:32.249 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:27.966980+0000", 2024-09-13T11:24:32.249 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.9433000000000001e-05 2024-09-13T11:24:32.249 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:32.249 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:32.250 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:24:32.250 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:27.967220+0000", 2024-09-13T11:24:32.250 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00024065499999999999 2024-09-13T11:24:32.250 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:32.250 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:32.250 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:24:32.250 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:27.967244+0000", 2024-09-13T11:24:32.250 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.4216e-05 2024-09-13T11:24:32.250 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:32.250 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:32.250 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:24:32.250 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:27.968641+0000", 2024-09-13T11:24:32.250 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0013968850000000001 2024-09-13T11:24:32.251 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:32.251 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:24:32.251 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:32.251 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:32.251 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:24:32.251 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-13T11:24:32.252 INFO:tasks.ceph.osd.11.smithi177.stderr:2024-09-13T11:24:32.251+0000 7f24792c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26580) UID: 0 2024-09-13T11:24:32.253 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-13T11:24:32.265 INFO:teuthology.orchestra.run.smithi001.stderr:noscrub is unset 2024-09-13T11:24:32.265 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-13T11:24:32.263+0000 7f6a5f1a6640 -1 osd.0 124 handle_osd_map injecting map crc failure 2024-09-13T11:24:32.353 INFO:tasks.ceph.osd.1.smithi092.stderr:2024-09-13T11:24:32.346+0000 7fe7789c5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33718) UID: 0 2024-09-13T11:24:32.411 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-09-13T11:24:32.411 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [ 2024-09-13T11:24:32.411 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:32.412 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:113813 3.2d 3:b6a854b1:::benchmark_data_smithi177_31116_object113812:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e125)", 2024-09-13T11:24:32.412 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:32.396920+0000", 2024-09-13T11:24:32.412 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 0.014067802000000001, 2024-09-13T11:24:32.412 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.014095168999999999, 2024-09-13T11:24:32.412 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:32.412 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:32.412 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "waiting for sub ops", 2024-09-13T11:24:32.412 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:24:32.412 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:24:32.412 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:32.412 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 113813 2024-09-13T11:24:32.413 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:32.413 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:32.413 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:32.413 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:32.413 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:32.396920+0000", 2024-09-13T11:24:32.413 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:32.413 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:32.413 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:32.413 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:32.413 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:32.396920+0000", 2024-09-13T11:24:32.413 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:32.413 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:32.413 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:32.414 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:32.414 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:32.396922+0000", 2024-09-13T11:24:32.414 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.6220000000000001e-06 2024-09-13T11:24:32.414 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:32.414 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:32.414 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:32.414 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:32.396944+0000", 2024-09-13T11:24:32.414 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.2198e-05 2024-09-13T11:24:32.414 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:32.414 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:32.414 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:32.414 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:32.396957+0000", 2024-09-13T11:24:32.414 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.2874000000000001e-05 2024-09-13T11:24:32.415 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:32.415 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:32.415 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:32.415 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:32.396965+0000", 2024-09-13T11:24:32.415 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.9119999999999998e-06 2024-09-13T11:24:32.415 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:32.415 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:32.415 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:32.415 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:32.397057+0000", 2024-09-13T11:24:32.415 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.1571999999999997e-05 2024-09-13T11:24:32.415 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:32.415 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:32.415 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:24:32.415 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:32.397225+0000", 2024-09-13T11:24:32.416 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00016800300000000001 2024-09-13T11:24:32.416 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:32.416 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:32.416 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 0,5", 2024-09-13T11:24:32.416 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:32.397411+0000", 2024-09-13T11:24:32.416 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00018662299999999999 2024-09-13T11:24:32.416 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:32.416 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:32.416 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:24:32.416 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:32.401043+0000", 2024-09-13T11:24:32.416 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0036321790000000001 2024-09-13T11:24:32.416 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:32.416 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:32.417 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:32.417 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:32.404557+0000", 2024-09-13T11:24:32.417 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0035130449999999998 2024-09-13T11:24:32.417 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:32.417 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:32.417 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:32.417 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:32.417 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:32.417 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:113781 3.2a 3:57e1ce3c:::benchmark_data_smithi177_31116_object113780:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e125)", 2024-09-13T11:24:32.417 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:32.364620+0000", 2024-09-13T11:24:32.417 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 0.046367430000000001, 2024-09-13T11:24:32.417 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.046451372999999997, 2024-09-13T11:24:32.417 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:32.418 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:32.418 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "waiting for sub ops", 2024-09-13T11:24:32.418 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:24:32.418 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:24:32.418 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:32.418 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 113781 2024-09-13T11:24:32.418 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:32.418 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:32.418 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:32.418 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:32.418 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:32.364620+0000", 2024-09-13T11:24:32.418 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:32.418 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:32.418 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:32.419 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:32.419 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:32.364620+0000", 2024-09-13T11:24:32.419 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:32.419 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:32.419 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:32.419 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:32.419 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:32.364622+0000", 2024-09-13T11:24:32.419 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.6110000000000001e-06 2024-09-13T11:24:32.419 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:32.419 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:32.420 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:32.420 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:32.364646+0000", 2024-09-13T11:24:32.420 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.3357999999999999e-05 2024-09-13T11:24:32.420 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:32.420 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:32.420 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:32.420 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:32.364658+0000", 2024-09-13T11:24:32.420 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.2132e-05 2024-09-13T11:24:32.420 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:32.420 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:32.420 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:32.420 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:32.364663+0000", 2024-09-13T11:24:32.420 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.5160000000000002e-06 2024-09-13T11:24:32.421 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:32.421 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:32.421 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:32.421 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:32.366930+0000", 2024-09-13T11:24:32.421 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0022664019999999998 2024-09-13T11:24:32.421 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:32.421 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:32.421 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:24:32.421 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:32.367092+0000", 2024-09-13T11:24:32.421 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00016241199999999999 2024-09-13T11:24:32.421 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:32.421 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:32.421 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 7,10", 2024-09-13T11:24:32.421 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:32.367294+0000", 2024-09-13T11:24:32.421 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00020176699999999999 2024-09-13T11:24:32.422 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:32.422 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:32.422 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:24:32.422 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:32.382911+0000", 2024-09-13T11:24:32.422 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.015616559 2024-09-13T11:24:32.422 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:32.422 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:32.422 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:32.422 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:32.422 INFO:teuthology.orchestra.run.smithi177.stdout: ], 2024-09-13T11:24:32.422 INFO:teuthology.orchestra.run.smithi177.stdout: "num_ops": 2 2024-09-13T11:24:32.422 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-09-13T11:24:32.436 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-13T11:24:32.455 INFO:tasks.ceph.osd.4.smithi001.stderr:2024-09-13T11:24:32.451+0000 7ff8e0218640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26310) UID: 0 2024-09-13T11:24:32.557 INFO:tasks.ceph.osd.1.smithi092.stderr:2024-09-13T11:24:32.550+0000 7fe7789c5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33718) UID: 0 2024-09-13T11:24:32.608 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-09-13T11:24:32.608 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [], 2024-09-13T11:24:32.608 INFO:teuthology.orchestra.run.smithi177.stdout: "complaint_time": 30, 2024-09-13T11:24:32.608 INFO:teuthology.orchestra.run.smithi177.stdout: "num_blocked_ops": 0 2024-09-13T11:24:32.609 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-09-13T11:24:32.640 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-13T11:24:32.658 INFO:tasks.ceph.osd.10.smithi162.stderr:2024-09-13T11:24:32.656+0000 7fb59c9f0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26317) UID: 0 2024-09-13T11:24:32.758 INFO:tasks.ceph.osd.11.smithi177.stderr:2024-09-13T11:24:32.755+0000 7f24792c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26580) UID: 0 2024-09-13T11:24:32.839 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-09-13T11:24:32.839 INFO:teuthology.orchestra.run.smithi177.stdout: "size": 20, 2024-09-13T11:24:32.839 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 600, 2024-09-13T11:24:32.839 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [ 2024-09-13T11:24:32.839 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:32.839 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:30852 3.a 3:522fb0d3:::benchmark_data_smithi177_31116_object30851:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e59)", 2024-09-13T11:24:32.840 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:09.538712+0000", 2024-09-13T11:24:32.840 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 83.297749635000002, 2024-09-13T11:24:32.840 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.19661263300000001, 2024-09-13T11:24:32.840 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:32.840 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:32.840 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:32.840 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:24:32.840 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:24:32.840 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:32.840 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 30852 2024-09-13T11:24:32.841 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:32.841 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:32.841 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:32.841 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:32.841 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.538712+0000", 2024-09-13T11:24:32.841 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:32.841 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:32.842 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:32.842 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:32.842 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.538712+0000", 2024-09-13T11:24:32.842 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:32.842 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:32.842 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:32.842 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:32.842 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.538714+0000", 2024-09-13T11:24:32.842 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.658e-06 2024-09-13T11:24:32.842 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:32.842 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:32.843 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:32.843 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.538735+0000", 2024-09-13T11:24:32.843 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.141e-05 2024-09-13T11:24:32.843 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:32.843 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:32.843 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:32.843 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.538747+0000", 2024-09-13T11:24:32.843 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.2258e-05 2024-09-13T11:24:32.843 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:32.843 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:32.844 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:32.844 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.538754+0000", 2024-09-13T11:24:32.844 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.2859999999999996e-06 2024-09-13T11:24:32.844 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:32.844 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:32.844 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:32.844 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.538798+0000", 2024-09-13T11:24:32.844 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.4258000000000002e-05 2024-09-13T11:24:32.844 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:32.844 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:32.844 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:24:32.844 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.538890+0000", 2024-09-13T11:24:32.845 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.1676999999999997e-05 2024-09-13T11:24:32.845 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:32.845 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:32.845 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 7", 2024-09-13T11:24:32.845 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.538991+0000", 2024-09-13T11:24:32.845 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00010173700000000001 2024-09-13T11:24:32.845 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:32.845 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:32.845 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:24:32.845 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.540171+0000", 2024-09-13T11:24:32.845 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0011800479999999999 2024-09-13T11:24:32.846 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:32.846 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:32.846 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:32.846 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.735031+0000", 2024-09-13T11:24:32.846 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.194859798 2024-09-13T11:24:32.846 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:32.846 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:32.846 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:24:32.846 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.735197+0000", 2024-09-13T11:24:32.846 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00016569499999999999 2024-09-13T11:24:32.846 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:32.847 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:32.847 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:24:32.847 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.735325+0000", 2024-09-13T11:24:32.847 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00012780800000000001 2024-09-13T11:24:32.847 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:32.847 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:32.847 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:32.848 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:32.848 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:32.848 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop_reply(client.4665.0:30852 3.a e59/30 ondisk, result = 0)", 2024-09-13T11:24:32.848 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:09.540458+0000", 2024-09-13T11:24:32.848 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 83.296004147000005, 2024-09-13T11:24:32.848 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.194887, 2024-09-13T11:24:32.848 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:32.848 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:32.848 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:24:32.848 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:32.848 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:32.849 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:32.849 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.540458+0000", 2024-09-13T11:24:32.849 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:32.849 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:32.849 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:32.849 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:32.849 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.540458+0000", 2024-09-13T11:24:32.849 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:32.849 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:32.849 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:32.850 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:32.850 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.540459+0000", 2024-09-13T11:24:32.850 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.2699999999999999e-06 2024-09-13T11:24:32.850 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:32.850 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:32.850 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:32.850 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.540463+0000", 2024-09-13T11:24:32.850 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.8500000000000004e-06 2024-09-13T11:24:32.850 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:32.850 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:32.850 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:32.850 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.734813+0000", 2024-09-13T11:24:32.851 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.194350047 2024-09-13T11:24:32.851 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:32.851 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:32.851 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:32.851 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.734832+0000", 2024-09-13T11:24:32.851 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.9610999999999999e-05 2024-09-13T11:24:32.851 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:32.851 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:32.851 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:32.851 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.734989+0000", 2024-09-13T11:24:32.851 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00015694799999999999 2024-09-13T11:24:32.852 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:32.852 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:32.852 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:24:32.852 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.735015+0000", 2024-09-13T11:24:32.852 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.6165000000000001e-05 2024-09-13T11:24:32.852 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:32.852 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:32.852 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:24:32.852 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.735345+0000", 2024-09-13T11:24:32.852 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00032910900000000001 2024-09-13T11:24:32.852 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:32.853 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:32.853 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:32.853 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:32.853 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:32.853 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPush(3.17 75/70 [PushOp(3:eda94455:::benchmark_data_smithi177_31116_object13017:head, version: 50'788, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:eda94455:::benchmark_data_smithi177_31116_object13017:head@50'788, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:24:32.853 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:31.038029+0000", 2024-09-13T11:24:32.853 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 61.798433082000003, 2024-09-13T11:24:32.853 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.42064623499999998, 2024-09-13T11:24:32.853 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:32.853 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:32.853 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:24:32.853 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:32.854 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:32.854 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:32.854 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.038029+0000", 2024-09-13T11:24:32.854 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:32.854 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:32.854 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:32.854 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:32.854 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.038029+0000", 2024-09-13T11:24:32.854 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:32.854 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:32.854 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:32.855 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:32.855 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.038030+0000", 2024-09-13T11:24:32.855 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.499e-06 2024-09-13T11:24:32.855 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:32.855 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:32.855 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:32.855 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.038088+0000", 2024-09-13T11:24:32.856 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.7771e-05 2024-09-13T11:24:32.856 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:32.856 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:32.856 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:32.856 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.457025+0000", 2024-09-13T11:24:32.856 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.41893754900000002 2024-09-13T11:24:32.856 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:32.856 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:32.857 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:32.857 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.457073+0000", 2024-09-13T11:24:32.857 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.7450999999999997e-05 2024-09-13T11:24:32.857 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:32.858 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:32.858 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:32.858 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.457304+0000", 2024-09-13T11:24:32.858 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000231351 2024-09-13T11:24:32.858 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:32.858 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:32.858 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:24:32.858 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.457331+0000", 2024-09-13T11:24:32.859 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.6372000000000001e-05 2024-09-13T11:24:32.859 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:32.859 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:32.859 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:24:32.859 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.458675+0000", 2024-09-13T11:24:32.859 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001344242 2024-09-13T11:24:32.859 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:32.859 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:32.860 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:32.860 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:32.860 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:32.860 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "rep_scrubmap(3.a e75 from shard 7)", 2024-09-13T11:24:32.860 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:32.151191+0000", 2024-09-13T11:24:32.861 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 60.685271028999999, 2024-09-13T11:24:32.861 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.164608173, 2024-09-13T11:24:32.861 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:32.861 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:32.861 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:24:32.861 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:32.861 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:32.861 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:32.861 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:32.151191+0000", 2024-09-13T11:24:32.861 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:32.862 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:32.862 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:32.862 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:32.862 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:32.151191+0000", 2024-09-13T11:24:32.862 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:32.862 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:32.862 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:32.862 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:32.862 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:32.151192+0000", 2024-09-13T11:24:32.862 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.28e-06 2024-09-13T11:24:32.862 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:32.863 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:32.863 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:32.863 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:32.151220+0000", 2024-09-13T11:24:32.863 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.8045e-05 2024-09-13T11:24:32.863 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:32.863 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:32.863 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:32.863 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:32.315480+0000", 2024-09-13T11:24:32.863 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.16425959000000001 2024-09-13T11:24:32.863 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:32.864 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:32.864 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:32.864 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:32.315488+0000", 2024-09-13T11:24:32.864 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.0819999999999999e-06 2024-09-13T11:24:32.864 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:32.864 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:32.864 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:32.864 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:32.315558+0000", 2024-09-13T11:24:32.864 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.0500999999999995e-05 2024-09-13T11:24:32.864 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:32.864 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:32.865 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:24:32.865 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:32.315570+0000", 2024-09-13T11:24:32.865 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.2238999999999999e-05 2024-09-13T11:24:32.865 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:32.865 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:32.865 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:24:32.865 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:32.315799+0000", 2024-09-13T11:24:32.865 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00022843599999999999 2024-09-13T11:24:32.865 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:32.865 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:32.865 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:32.866 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:32.866 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:32.866 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPush(3.14 79/70 [PushOp(3:2c86f72b:::benchmark_data_smithi177_31116_object32132:head, version: 60'2019, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:2c86f72b:::benchmark_data_smithi177_31116_object32132:head@60'2019, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:24:32.866 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:35.768489+0000", 2024-09-13T11:24:32.866 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 57.067972865000002, 2024-09-13T11:24:32.866 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.179450581, 2024-09-13T11:24:32.866 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:32.866 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:32.866 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:24:32.866 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:32.867 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:32.867 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:32.867 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:35.768489+0000", 2024-09-13T11:24:32.867 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:32.867 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:32.867 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:32.867 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:32.867 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:35.768489+0000", 2024-09-13T11:24:32.867 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:32.867 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:32.867 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:32.868 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:32.868 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:35.768490+0000", 2024-09-13T11:24:32.868 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5999999999999999e-06 2024-09-13T11:24:32.868 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:32.868 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:32.868 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:32.868 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:35.768592+0000", 2024-09-13T11:24:32.868 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00010194799999999999 2024-09-13T11:24:32.868 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:32.869 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:32.869 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:32.869 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:35.946933+0000", 2024-09-13T11:24:32.869 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.17834060800000001 2024-09-13T11:24:32.869 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:32.869 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:32.869 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:32.869 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:35.946953+0000", 2024-09-13T11:24:32.869 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.0347e-05 2024-09-13T11:24:32.869 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:32.869 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:32.869 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:32.870 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:35.947056+0000", 2024-09-13T11:24:32.870 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000103164 2024-09-13T11:24:32.870 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:32.870 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:32.870 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:24:32.870 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:35.947071+0000", 2024-09-13T11:24:32.870 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.4854e-05 2024-09-13T11:24:32.870 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:32.870 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:32.870 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:24:32.870 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:35.947939+0000", 2024-09-13T11:24:32.871 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00086806000000000003 2024-09-13T11:24:32.871 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:32.871 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:32.871 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:32.871 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:32.871 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:32.871 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPush(3.17 83/70 [PushOp(3:eb574268:::benchmark_data_smithi177_31116_object40622:head, version: 64'2475, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:eb574268:::benchmark_data_smithi177_31116_object40622:head@64'2475, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:24:32.871 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:40.760042+0000", 2024-09-13T11:24:32.871 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 52.076419365, 2024-09-13T11:24:32.871 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.30544642100000002, 2024-09-13T11:24:32.871 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:32.872 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:32.872 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:24:32.872 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:32.872 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:32.872 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:32.872 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:40.760042+0000", 2024-09-13T11:24:32.872 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:32.872 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:32.872 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:32.872 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:32.873 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:40.760042+0000", 2024-09-13T11:24:32.873 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:32.873 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:32.873 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:32.873 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:32.873 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:40.760044+0000", 2024-09-13T11:24:32.873 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.511e-06 2024-09-13T11:24:32.873 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:32.873 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:32.873 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:32.873 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:40.760149+0000", 2024-09-13T11:24:32.873 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000105473 2024-09-13T11:24:32.874 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:32.874 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:32.874 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:32.874 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.064465+0000", 2024-09-13T11:24:32.874 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.30431613800000001 2024-09-13T11:24:32.874 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:32.874 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:32.874 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:32.874 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.064479+0000", 2024-09-13T11:24:32.874 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3586e-05 2024-09-13T11:24:32.874 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:32.874 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:32.875 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:32.875 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.064553+0000", 2024-09-13T11:24:32.875 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.3646000000000006e-05 2024-09-13T11:24:32.875 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:32.875 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:32.875 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:24:32.875 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.064568+0000", 2024-09-13T11:24:32.875 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.52e-05 2024-09-13T11:24:32.875 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:32.875 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:32.876 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:24:32.876 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.065489+0000", 2024-09-13T11:24:32.876 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00092086700000000002 2024-09-13T11:24:32.876 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:32.876 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:32.876 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:32.876 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:32.876 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:32.876 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPushReply(3.1a 83/78 [PushReplyOp(3:5c3834bd:::benchmark_data_smithi177_31116_object8669:head)])", 2024-09-13T11:24:32.876 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:40.798236+0000", 2024-09-13T11:24:32.877 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 52.038225935, 2024-09-13T11:24:32.877 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.29701520999999997, 2024-09-13T11:24:32.877 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:32.877 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:32.877 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-13T11:24:32.877 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:32.877 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:32.877 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:32.877 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:40.798236+0000", 2024-09-13T11:24:32.877 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:32.877 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:32.878 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:32.878 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:32.878 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:40.798236+0000", 2024-09-13T11:24:32.878 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:32.878 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:32.878 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:32.878 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:32.878 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:40.798237+0000", 2024-09-13T11:24:32.878 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.0500000000000002e-07 2024-09-13T11:24:32.878 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:32.878 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:32.878 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:32.879 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:40.798240+0000", 2024-09-13T11:24:32.879 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.4819999999999999e-06 2024-09-13T11:24:32.879 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:32.879 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:32.879 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:32.879 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.093360+0000", 2024-09-13T11:24:32.879 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.29512009 2024-09-13T11:24:32.879 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:32.879 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:32.879 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:32.879 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.093372+0000", 2024-09-13T11:24:32.880 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.2221e-05 2024-09-13T11:24:32.880 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:32.880 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:32.880 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:32.880 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.093482+0000", 2024-09-13T11:24:32.880 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000109571 2024-09-13T11:24:32.880 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:32.880 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:32.880 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:24:32.880 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.095251+0000", 2024-09-13T11:24:32.880 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001768941 2024-09-13T11:24:32.881 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:32.881 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:32.881 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:32.881 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:32.881 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:32.881 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPushReply(3.1a 83/78 [PushReplyOp(3:58c7ffc0:::benchmark_data_smithi177_31116_object8738:head)])", 2024-09-13T11:24:32.881 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:40.805941+0000", 2024-09-13T11:24:32.881 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 52.030520680999999, 2024-09-13T11:24:32.881 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.30315678499999998, 2024-09-13T11:24:32.881 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:32.881 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:32.881 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-13T11:24:32.882 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:32.882 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:32.882 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:32.882 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:40.805941+0000", 2024-09-13T11:24:32.882 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:32.882 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:32.882 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:32.882 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:32.882 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:40.805941+0000", 2024-09-13T11:24:32.882 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:32.882 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:32.883 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:32.883 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:32.883 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:40.805942+0000", 2024-09-13T11:24:32.883 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3769999999999999e-06 2024-09-13T11:24:32.883 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:32.883 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:32.883 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:32.883 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:40.805947+0000", 2024-09-13T11:24:32.883 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.8980000000000004e-06 2024-09-13T11:24:32.883 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:32.883 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:32.883 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:32.884 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.106505+0000", 2024-09-13T11:24:32.884 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.30055777700000003 2024-09-13T11:24:32.884 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:32.884 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:32.884 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:32.884 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.106512+0000", 2024-09-13T11:24:32.884 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.3749999999999997e-06 2024-09-13T11:24:32.884 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:32.884 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:32.884 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:32.884 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.106566+0000", 2024-09-13T11:24:32.885 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.3103999999999998e-05 2024-09-13T11:24:32.885 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:32.885 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:32.885 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:24:32.885 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.109098+0000", 2024-09-13T11:24:32.885 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0025322539999999998 2024-09-13T11:24:32.885 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:32.885 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:32.885 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:32.885 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:32.885 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:32.886 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPushReply(3.1a 83/78 [PushReplyOp(3:58c7ffc0:::benchmark_data_smithi177_31116_object8738:head)])", 2024-09-13T11:24:32.886 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:40.805973+0000", 2024-09-13T11:24:32.886 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 52.030488677999998, 2024-09-13T11:24:32.886 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.28729672099999998, 2024-09-13T11:24:32.886 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:32.886 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:32.886 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-13T11:24:32.886 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:32.886 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:32.886 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:32.886 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:40.805973+0000", 2024-09-13T11:24:32.886 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:32.887 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:32.887 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:32.887 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:32.887 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:40.805973+0000", 2024-09-13T11:24:32.887 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:32.887 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:32.887 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:32.887 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:32.887 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:40.805974+0000", 2024-09-13T11:24:32.887 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.8599999999999998e-07 2024-09-13T11:24:32.887 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:32.888 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:32.888 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:32.888 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:40.805976+0000", 2024-09-13T11:24:32.888 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.2809999999999998e-06 2024-09-13T11:24:32.888 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:32.888 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:32.888 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:32.888 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.091588+0000", 2024-09-13T11:24:32.888 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.28561173899999998 2024-09-13T11:24:32.888 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:32.888 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:32.889 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:32.889 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.091596+0000", 2024-09-13T11:24:32.889 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.0520000000000002e-06 2024-09-13T11:24:32.889 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:32.889 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:32.889 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:32.889 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.091907+0000", 2024-09-13T11:24:32.889 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00031179599999999998 2024-09-13T11:24:32.889 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:32.889 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:32.889 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:24:32.890 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.093270+0000", 2024-09-13T11:24:32.890 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0013622669999999999 2024-09-13T11:24:32.890 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:32.890 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:32.890 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:32.890 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:32.890 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:32.890 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPushReply(3.1a 84/78 [PushReplyOp(3:581ea97e:::benchmark_data_smithi177_31116_object10255:head)])", 2024-09-13T11:24:32.890 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:41.514228+0000", 2024-09-13T11:24:32.890 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 51.322233189999999, 2024-09-13T11:24:32.891 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.40695564200000001, 2024-09-13T11:24:32.891 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:32.891 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:32.891 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-13T11:24:32.891 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:32.891 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:32.891 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:32.891 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.514228+0000", 2024-09-13T11:24:32.891 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:32.891 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:32.891 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:32.891 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:32.892 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.514228+0000", 2024-09-13T11:24:32.892 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:32.892 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:32.892 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:32.892 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:32.892 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.514230+0000", 2024-09-13T11:24:32.892 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.499e-06 2024-09-13T11:24:32.892 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:32.892 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:32.892 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:32.892 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.514236+0000", 2024-09-13T11:24:32.893 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.5389999999999998e-06 2024-09-13T11:24:32.893 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:32.893 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:32.893 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:32.893 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.919787+0000", 2024-09-13T11:24:32.893 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.40555152999999999 2024-09-13T11:24:32.893 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:32.893 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:32.894 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:32.894 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.919796+0000", 2024-09-13T11:24:32.894 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.4829999999999999e-06 2024-09-13T11:24:32.894 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:32.894 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:32.894 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:32.894 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.920366+0000", 2024-09-13T11:24:32.894 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00057061700000000004 2024-09-13T11:24:32.894 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:32.894 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:32.894 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:24:32.895 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.921184+0000", 2024-09-13T11:24:32.895 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00081797399999999996 2024-09-13T11:24:32.895 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:32.895 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:32.895 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:32.895 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:32.895 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:32.895 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:60312 3.14 3:2e846241:::benchmark_data_smithi177_31116_object60311:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e85)", 2024-09-13T11:24:32.895 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:42.065725+0000", 2024-09-13T11:24:32.895 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 50.770736546999998, 2024-09-13T11:24:32.895 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.211323176, 2024-09-13T11:24:32.895 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:32.896 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:32.896 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:32.896 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:24:32.896 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:24:32.896 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:32.896 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 60312 2024-09-13T11:24:32.896 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:32.896 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:32.896 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:32.896 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:32.896 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.065725+0000", 2024-09-13T11:24:32.897 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:32.897 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:32.897 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:32.897 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:32.897 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.065725+0000", 2024-09-13T11:24:32.897 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:32.897 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:32.897 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:32.897 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:32.897 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.065728+0000", 2024-09-13T11:24:32.898 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.3400000000000002e-06 2024-09-13T11:24:32.898 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:32.898 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:32.898 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:32.898 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.065759+0000", 2024-09-13T11:24:32.898 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.0074e-05 2024-09-13T11:24:32.898 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:32.898 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:32.898 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:32.898 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.065771+0000", 2024-09-13T11:24:32.898 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.2869e-05 2024-09-13T11:24:32.899 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:32.899 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:32.899 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:32.899 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.065780+0000", 2024-09-13T11:24:32.899 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.3489999999999992e-06 2024-09-13T11:24:32.899 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:32.899 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:32.899 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:32.899 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.065837+0000", 2024-09-13T11:24:32.899 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.7444000000000001e-05 2024-09-13T11:24:32.899 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:32.899 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:32.900 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:24:32.900 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.065966+0000", 2024-09-13T11:24:32.900 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000128973 2024-09-13T11:24:32.900 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:32.900 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:32.900 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 14", 2024-09-13T11:24:32.900 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.066114+0000", 2024-09-13T11:24:32.900 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00014788299999999999 2024-09-13T11:24:32.900 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:32.900 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:32.901 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:24:32.901 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.067604+0000", 2024-09-13T11:24:32.901 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001489637 2024-09-13T11:24:32.901 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:32.901 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:32.901 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:32.901 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.276963+0000", 2024-09-13T11:24:32.901 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.20935888499999999 2024-09-13T11:24:32.901 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:32.901 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:32.901 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:24:32.902 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.277003+0000", 2024-09-13T11:24:32.902 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.0241000000000003e-05 2024-09-13T11:24:32.902 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:32.902 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:32.902 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:24:32.902 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.277048+0000", 2024-09-13T11:24:32.902 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.5481e-05 2024-09-13T11:24:32.902 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:32.902 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:32.902 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:32.902 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:32.902 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:32.902 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop_reply(client.4665.0:60312 3.14 e85/84 ondisk, result = 0)", 2024-09-13T11:24:32.903 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:42.069120+0000", 2024-09-13T11:24:32.903 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 50.767341426999998, 2024-09-13T11:24:32.903 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.20804031000000001, 2024-09-13T11:24:32.903 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:32.903 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:32.903 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:24:32.903 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:32.903 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:32.903 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:32.903 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.069120+0000", 2024-09-13T11:24:32.903 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:32.904 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:32.904 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:32.904 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:32.904 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.069120+0000", 2024-09-13T11:24:32.904 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:32.904 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:32.904 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:32.904 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:32.904 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.069123+0000", 2024-09-13T11:24:32.904 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.289e-06 2024-09-13T11:24:32.905 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:32.905 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:32.905 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:32.905 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.069128+0000", 2024-09-13T11:24:32.905 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.3530000000000002e-06 2024-09-13T11:24:32.905 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:32.905 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:32.905 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:32.905 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.276885+0000", 2024-09-13T11:24:32.905 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.207756989 2024-09-13T11:24:32.905 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:32.905 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:32.906 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:32.906 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.276892+0000", 2024-09-13T11:24:32.906 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.922e-06 2024-09-13T11:24:32.906 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:32.906 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:32.906 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:32.906 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.276953+0000", 2024-09-13T11:24:32.906 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.0890000000000001e-05 2024-09-13T11:24:32.906 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:32.906 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:32.906 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:24:32.906 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.276959+0000", 2024-09-13T11:24:32.907 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.9179999999999999e-06 2024-09-13T11:24:32.907 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:32.907 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:32.907 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:24:32.907 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.277161+0000", 2024-09-13T11:24:32.907 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00020194899999999999 2024-09-13T11:24:32.907 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:32.907 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:32.907 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:32.907 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:32.907 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:32.908 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:60427 3.14 3:2afea229:::benchmark_data_smithi177_31116_object60426:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e85)", 2024-09-13T11:24:32.908 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:42.187231+0000", 2024-09-13T11:24:32.908 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 50.649230320999997, 2024-09-13T11:24:32.908 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.83900596999999999, 2024-09-13T11:24:32.908 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:32.908 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:32.908 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:32.908 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:24:32.908 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:24:32.908 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:32.908 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 60427 2024-09-13T11:24:32.909 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:32.909 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:32.909 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:32.909 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:32.909 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.187231+0000", 2024-09-13T11:24:32.909 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:32.909 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:32.909 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:32.909 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:32.909 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.187231+0000", 2024-09-13T11:24:32.909 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:32.910 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:32.910 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:32.910 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:32.910 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.187235+0000", 2024-09-13T11:24:32.910 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.3579999999999999e-06 2024-09-13T11:24:32.910 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:32.910 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:32.910 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:32.910 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.187261+0000", 2024-09-13T11:24:32.911 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.6089e-05 2024-09-13T11:24:32.911 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:32.911 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:32.911 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:32.911 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.187275+0000", 2024-09-13T11:24:32.911 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.4653e-05 2024-09-13T11:24:32.911 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:32.911 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:32.911 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:32.912 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.187282+0000", 2024-09-13T11:24:32.912 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.8870000000000003e-06 2024-09-13T11:24:32.912 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:32.912 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:32.912 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:32.912 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.187336+0000", 2024-09-13T11:24:32.912 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.3875999999999997e-05 2024-09-13T11:24:32.912 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:32.913 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:32.913 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:24:32.913 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.187480+0000", 2024-09-13T11:24:32.913 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00014330899999999999 2024-09-13T11:24:32.913 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:32.913 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:32.913 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 14", 2024-09-13T11:24:32.913 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.187658+0000", 2024-09-13T11:24:32.913 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00017861900000000001 2024-09-13T11:24:32.913 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:32.913 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:32.914 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:24:32.914 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.189353+0000", 2024-09-13T11:24:32.914 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001695144 2024-09-13T11:24:32.914 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:32.914 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:32.914 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:32.914 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:43.026081+0000", 2024-09-13T11:24:32.914 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.83672736199999997 2024-09-13T11:24:32.914 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:32.914 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:32.915 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:24:32.915 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:43.026169+0000", 2024-09-13T11:24:32.915 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.8411000000000004e-05 2024-09-13T11:24:32.915 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:32.915 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:32.915 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:24:32.915 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:43.026237+0000", 2024-09-13T11:24:32.915 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.8262000000000005e-05 2024-09-13T11:24:32.915 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:32.915 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:32.915 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:32.916 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:32.916 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:32.916 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop_reply(client.4665.0:60427 3.14 e85/84 ondisk, result = 0)", 2024-09-13T11:24:32.916 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:42.190914+0000", 2024-09-13T11:24:32.916 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 50.645547657000002, 2024-09-13T11:24:32.916 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.83534071499999996, 2024-09-13T11:24:32.916 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:32.916 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:32.916 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:24:32.916 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:32.916 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:32.917 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:32.917 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.190914+0000", 2024-09-13T11:24:32.917 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:32.917 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:32.917 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:32.917 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:32.917 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.190914+0000", 2024-09-13T11:24:32.917 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:32.917 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:32.917 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:32.917 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:32.918 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.190916+0000", 2024-09-13T11:24:32.918 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.84e-06 2024-09-13T11:24:32.918 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:32.918 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:32.918 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:32.918 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.190921+0000", 2024-09-13T11:24:32.918 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.2039999999999996e-06 2024-09-13T11:24:32.918 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:32.918 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:32.918 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:32.918 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:43.025926+0000", 2024-09-13T11:24:32.919 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.83500542 2024-09-13T11:24:32.919 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:32.919 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:32.919 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:32.919 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:43.025937+0000", 2024-09-13T11:24:32.919 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0597e-05 2024-09-13T11:24:32.919 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:32.919 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:32.919 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:32.919 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:43.026052+0000", 2024-09-13T11:24:32.919 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000114838 2024-09-13T11:24:32.919 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:32.920 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:32.920 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:24:32.920 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:43.026070+0000", 2024-09-13T11:24:32.920 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.8131e-05 2024-09-13T11:24:32.920 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:32.920 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:32.920 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:24:32.920 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:43.026255+0000", 2024-09-13T11:24:32.920 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000184685 2024-09-13T11:24:32.920 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:32.920 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:32.921 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:32.921 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:32.921 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:32.921 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPushReply(3.1d 89/78 [PushReplyOp(3:bc1b7cfb:::benchmark_data_smithi177_31116_object22629:head)])", 2024-09-13T11:24:32.921 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:46.663106+0000", 2024-09-13T11:24:32.921 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 46.173355473999997, 2024-09-13T11:24:32.921 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.26645533199999999, 2024-09-13T11:24:32.921 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:32.921 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:32.921 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-13T11:24:32.921 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:32.921 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:32.922 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:32.922 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.663106+0000", 2024-09-13T11:24:32.922 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:32.922 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:32.922 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:32.922 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:32.922 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.663106+0000", 2024-09-13T11:24:32.922 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:32.922 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:32.922 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:32.922 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:32.922 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.663108+0000", 2024-09-13T11:24:32.923 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.0140000000000001e-06 2024-09-13T11:24:32.923 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:32.923 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:32.923 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:32.923 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.663116+0000", 2024-09-13T11:24:32.923 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.5440000000000001e-06 2024-09-13T11:24:32.923 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:32.923 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:32.923 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:32.923 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.928111+0000", 2024-09-13T11:24:32.923 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.26499566899999999 2024-09-13T11:24:32.923 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:32.924 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:32.924 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:32.924 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.928135+0000", 2024-09-13T11:24:32.924 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.4003e-05 2024-09-13T11:24:32.924 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:32.924 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:32.924 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:32.924 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.928559+0000", 2024-09-13T11:24:32.924 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000423318 2024-09-13T11:24:32.924 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:32.924 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:32.925 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:24:32.925 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.929562+0000", 2024-09-13T11:24:32.925 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0010027840000000001 2024-09-13T11:24:32.925 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:32.925 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:32.925 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:32.925 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:32.925 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:32.925 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "rep_scrubmap(3.14 e84 from shard 14)", 2024-09-13T11:24:32.925 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:48.313615+0000", 2024-09-13T11:24:32.925 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 44.522846680999997, 2024-09-13T11:24:32.925 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.35633836099999999, 2024-09-13T11:24:32.926 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:32.926 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:32.926 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:24:32.926 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:32.926 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:32.926 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:32.926 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.313615+0000", 2024-09-13T11:24:32.926 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:32.926 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:32.926 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:32.926 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:32.927 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.313615+0000", 2024-09-13T11:24:32.927 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:32.927 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:32.927 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:32.927 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:32.927 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.313617+0000", 2024-09-13T11:24:32.927 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.7990000000000001e-06 2024-09-13T11:24:32.927 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:32.927 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:32.927 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:32.927 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.313688+0000", 2024-09-13T11:24:32.927 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.1481000000000002e-05 2024-09-13T11:24:32.927 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:32.928 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:32.928 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:32.928 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.669785+0000", 2024-09-13T11:24:32.928 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.356096466 2024-09-13T11:24:32.928 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:32.928 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:32.928 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:32.928 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.669792+0000", 2024-09-13T11:24:32.928 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.1740000000000003e-06 2024-09-13T11:24:32.928 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:32.928 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:32.929 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:32.929 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.669839+0000", 2024-09-13T11:24:32.929 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.7323000000000001e-05 2024-09-13T11:24:32.929 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:32.929 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:32.929 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:24:32.929 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.669847+0000", 2024-09-13T11:24:32.929 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.0469999999999994e-06 2024-09-13T11:24:32.929 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:32.929 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:32.929 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:24:32.930 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.669953+0000", 2024-09-13T11:24:32.930 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00010607099999999999 2024-09-13T11:24:32.930 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:32.930 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:32.930 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:32.930 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:32.930 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:32.930 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:91603 3.14 3:2d13add5:::benchmark_data_smithi177_31116_object91602:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111)", 2024-09-13T11:24:32.930 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:11.037157+0000", 2024-09-13T11:24:32.930 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 21.799305068999999, 2024-09-13T11:24:32.930 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.165525535, 2024-09-13T11:24:32.931 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:32.931 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:32.931 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:32.931 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:24:32.931 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:24:32.931 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:32.931 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 91603 2024-09-13T11:24:32.931 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:32.931 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:32.931 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:32.931 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:32.931 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:11.037157+0000", 2024-09-13T11:24:32.931 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:32.932 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:32.932 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:32.932 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:32.932 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:11.037157+0000", 2024-09-13T11:24:32.932 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:32.932 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:32.932 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:32.932 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:32.932 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:11.037159+0000", 2024-09-13T11:24:32.932 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.261e-06 2024-09-13T11:24:32.932 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:32.933 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:32.933 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:32.933 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:11.037188+0000", 2024-09-13T11:24:32.933 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.8798000000000001e-05 2024-09-13T11:24:32.933 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:32.933 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:32.933 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:32.933 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:11.037204+0000", 2024-09-13T11:24:32.933 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.6739999999999999e-05 2024-09-13T11:24:32.933 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:32.933 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:32.933 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:32.934 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:11.037213+0000", 2024-09-13T11:24:32.934 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.1110000000000008e-06 2024-09-13T11:24:32.934 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:32.934 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:32.934 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:32.934 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:11.037270+0000", 2024-09-13T11:24:32.934 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.7629e-05 2024-09-13T11:24:32.934 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:32.934 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:32.934 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:24:32.934 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:11.037364+0000", 2024-09-13T11:24:32.935 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.3871999999999998e-05 2024-09-13T11:24:32.935 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:32.935 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:32.935 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 14", 2024-09-13T11:24:32.935 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:11.037450+0000", 2024-09-13T11:24:32.935 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.5568000000000005e-05 2024-09-13T11:24:32.935 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:32.935 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:32.935 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:24:32.935 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:11.039799+0000", 2024-09-13T11:24:32.935 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0023494039999999998 2024-09-13T11:24:32.936 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:32.936 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:32.936 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:32.936 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:11.202576+0000", 2024-09-13T11:24:32.936 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.16277712499999999 2024-09-13T11:24:32.936 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:32.936 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:32.936 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:24:32.936 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:11.202647+0000", 2024-09-13T11:24:32.936 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.0934999999999998e-05 2024-09-13T11:24:32.937 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:32.937 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:32.937 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:24:32.937 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:11.202682+0000", 2024-09-13T11:24:32.937 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.5092000000000002e-05 2024-09-13T11:24:32.937 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:32.937 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:32.937 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:32.937 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:32.938 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:32.938 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:110268 3.14 3:2997c4c5:::benchmark_data_smithi177_31116_object110267:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119)", 2024-09-13T11:24:32.938 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:25.359386+0000", 2024-09-13T11:24:32.938 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 7.4770753650000001, 2024-09-13T11:24:32.938 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.29072703, 2024-09-13T11:24:32.938 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:32.938 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:32.938 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:32.938 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:24:32.938 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:24:32.938 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:32.938 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 110268 2024-09-13T11:24:32.938 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:32.938 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:32.939 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:32.939 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:32.939 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.359386+0000", 2024-09-13T11:24:32.939 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:32.939 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:32.939 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:32.939 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:32.939 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.359386+0000", 2024-09-13T11:24:32.939 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:32.939 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:32.939 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:32.939 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:32.939 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.359389+0000", 2024-09-13T11:24:32.940 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.8600000000000001e-06 2024-09-13T11:24:32.940 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:32.940 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:32.940 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:32.940 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.359424+0000", 2024-09-13T11:24:32.940 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.4755999999999997e-05 2024-09-13T11:24:32.940 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:32.940 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:32.940 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:32.940 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.359445+0000", 2024-09-13T11:24:32.940 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.1104999999999999e-05 2024-09-13T11:24:32.940 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:32.940 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:32.940 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:32.941 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.359460+0000", 2024-09-13T11:24:32.941 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5396999999999998e-05 2024-09-13T11:24:32.941 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:32.941 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:32.941 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:32.941 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.359581+0000", 2024-09-13T11:24:32.941 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000120455 2024-09-13T11:24:32.941 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:32.941 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:32.941 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:24:32.941 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.359827+0000", 2024-09-13T11:24:32.941 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00024571 2024-09-13T11:24:32.941 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:32.942 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:32.942 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 14", 2024-09-13T11:24:32.942 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.360084+0000", 2024-09-13T11:24:32.942 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00025787999999999999 2024-09-13T11:24:32.942 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:32.942 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:32.942 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:24:32.942 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.362243+0000", 2024-09-13T11:24:32.942 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0021581579999999999 2024-09-13T11:24:32.942 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:32.942 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:32.942 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:32.942 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.650034+0000", 2024-09-13T11:24:32.943 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.287791662 2024-09-13T11:24:32.943 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:32.943 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:32.943 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:24:32.943 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.650077+0000", 2024-09-13T11:24:32.943 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.2367999999999998e-05 2024-09-13T11:24:32.943 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:32.943 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:32.943 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:24:32.943 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.650113+0000", 2024-09-13T11:24:32.943 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.6678999999999999e-05 2024-09-13T11:24:32.944 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:32.944 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:32.944 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:32.944 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:32.944 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:32.944 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:110279 3.14 3:2b17a349:::benchmark_data_smithi177_31116_object110278:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119)", 2024-09-13T11:24:32.944 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:25.645137+0000", 2024-09-13T11:24:32.944 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 7.1913245269999999, 2024-09-13T11:24:32.944 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.28172927800000003, 2024-09-13T11:24:32.944 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:32.944 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:32.945 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:32.945 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:24:32.945 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:24:32.945 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:32.945 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 110279 2024-09-13T11:24:32.945 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:32.945 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:32.945 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:32.945 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:32.945 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.645137+0000", 2024-09-13T11:24:32.945 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:32.945 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:32.946 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:32.946 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:32.946 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.645137+0000", 2024-09-13T11:24:32.946 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:32.946 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:32.946 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:32.946 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:32.946 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.645140+0000", 2024-09-13T11:24:32.946 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.0939999999999999e-06 2024-09-13T11:24:32.946 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:32.947 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:32.947 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:32.947 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.645184+0000", 2024-09-13T11:24:32.947 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.3412999999999999e-05 2024-09-13T11:24:32.947 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:32.947 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:32.947 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:32.947 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.645206+0000", 2024-09-13T11:24:32.947 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.1997e-05 2024-09-13T11:24:32.947 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:32.947 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:32.947 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:32.948 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.645216+0000", 2024-09-13T11:24:32.948 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0740999999999999e-05 2024-09-13T11:24:32.948 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:32.948 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:32.948 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:32.948 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.645397+0000", 2024-09-13T11:24:32.948 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00018077600000000001 2024-09-13T11:24:32.948 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:32.948 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:32.948 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:24:32.948 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.645638+0000", 2024-09-13T11:24:32.948 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000241091 2024-09-13T11:24:32.949 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:32.949 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:32.949 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 14", 2024-09-13T11:24:32.949 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.645922+0000", 2024-09-13T11:24:32.949 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000283565 2024-09-13T11:24:32.949 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:32.949 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:32.949 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:24:32.949 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.648980+0000", 2024-09-13T11:24:32.949 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00305851 2024-09-13T11:24:32.950 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:32.950 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:32.950 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:32.950 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.926702+0000", 2024-09-13T11:24:32.950 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.27772199800000003 2024-09-13T11:24:32.950 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:32.950 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:32.950 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:24:32.950 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.926796+0000", 2024-09-13T11:24:32.950 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.4158000000000002e-05 2024-09-13T11:24:32.950 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:32.950 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:32.951 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:24:32.951 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.926866+0000", 2024-09-13T11:24:32.951 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.9935e-05 2024-09-13T11:24:32.951 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:32.951 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:32.951 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:32.951 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:32.951 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:32.951 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPush(3.2 124/122 [PushOp(3:43fbaffc:::benchmark_data_smithi177_31116_object10325:head, version: 48'645, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:43fbaffc:::benchmark_data_smithi177_31116_object10325:head@48'645, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:24:32.951 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:31.862400+0000", 2024-09-13T11:24:32.951 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 0.97406205400000001, 2024-09-13T11:24:32.952 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.186550994, 2024-09-13T11:24:32.952 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:32.952 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:32.952 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:24:32.952 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:32.952 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:32.952 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:32.952 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:31.862400+0000", 2024-09-13T11:24:32.952 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:32.952 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:32.952 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:32.953 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:32.953 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:31.862400+0000", 2024-09-13T11:24:32.953 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:32.953 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:32.953 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:32.953 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:32.953 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:31.862401+0000", 2024-09-13T11:24:32.953 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.7439999999999999e-06 2024-09-13T11:24:32.953 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:32.953 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:32.953 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:32.953 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:31.862460+0000", 2024-09-13T11:24:32.954 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.897e-05 2024-09-13T11:24:32.954 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:32.954 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:32.954 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:32.954 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:32.047866+0000", 2024-09-13T11:24:32.954 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.18540612400000001 2024-09-13T11:24:32.954 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:32.954 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:32.954 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:32.954 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:32.047878+0000", 2024-09-13T11:24:32.954 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.1618e-05 2024-09-13T11:24:32.955 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:32.955 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:32.955 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:32.955 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:32.047975+0000", 2024-09-13T11:24:32.955 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.7176999999999995e-05 2024-09-13T11:24:32.955 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:32.955 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:32.955 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:24:32.955 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:32.047991+0000", 2024-09-13T11:24:32.955 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5722999999999998e-05 2024-09-13T11:24:32.955 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:32.955 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:32.956 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:24:32.956 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:32.048951+0000", 2024-09-13T11:24:32.956 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00095963799999999996 2024-09-13T11:24:32.956 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:32.956 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:32.956 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:32.956 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:32.956 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:32.956 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-09-13T11:24:32.958 INFO:tasks.ceph.osd.10.smithi162.stderr:2024-09-13T11:24:32.956+0000 7fb59c9f0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26317) UID: 0 2024-09-13T11:24:32.959 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-13T11:24:33.010 INFO:tasks.radosbench.radosbench.0.smithi177.stdout: 126 16 114505 114489 56.7845 63.1875 0.00495532 0.017604 2024-09-13T11:24:33.059 INFO:tasks.ceph.osd.8.smithi001.stderr:2024-09-13T11:24:33.055+0000 7fb9b48ae640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26312) UID: 0 2024-09-13T11:24:33.159 INFO:tasks.ceph.osd.5.smithi092.stderr:2024-09-13T11:24:33.154+0000 7f6f0b705640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26346) UID: 0 2024-09-13T11:24:33.168 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-09-13T11:24:33.168 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [ 2024-09-13T11:24:33.168 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:33.168 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:114646 3.12 3:4c457b82:::benchmark_data_smithi177_31116_object114645:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e125)", 2024-09-13T11:24:33.168 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:33.158078+0000", 2024-09-13T11:24:33.169 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 0.0094930220000000003, 2024-09-13T11:24:33.169 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0095131790000000001, 2024-09-13T11:24:33.169 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:33.169 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:33.169 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "waiting for sub ops", 2024-09-13T11:24:33.169 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:24:33.169 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:24:33.169 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:33.169 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 114646 2024-09-13T11:24:33.169 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:33.169 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:33.169 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:33.170 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:33.170 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:33.158078+0000", 2024-09-13T11:24:33.170 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:33.170 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:33.170 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:33.170 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:33.170 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:33.158078+0000", 2024-09-13T11:24:33.170 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:33.170 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:33.170 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:33.170 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:33.170 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:33.158080+0000", 2024-09-13T11:24:33.170 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.2469999999999998e-06 2024-09-13T11:24:33.171 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:33.171 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:33.171 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:33.171 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:33.158102+0000", 2024-09-13T11:24:33.171 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.1807e-05 2024-09-13T11:24:33.171 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:33.171 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:33.171 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:33.171 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:33.158113+0000", 2024-09-13T11:24:33.171 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.1071e-05 2024-09-13T11:24:33.171 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:33.172 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:33.172 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:33.172 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:33.158118+0000", 2024-09-13T11:24:33.172 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.3149999999999999e-06 2024-09-13T11:24:33.172 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:33.172 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:33.172 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:33.172 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:33.160553+0000", 2024-09-13T11:24:33.172 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0024348389999999998 2024-09-13T11:24:33.172 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:33.172 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:33.172 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:24:33.173 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:33.160706+0000", 2024-09-13T11:24:33.173 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00015243799999999999 2024-09-13T11:24:33.173 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:33.173 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:33.173 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 5", 2024-09-13T11:24:33.173 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:33.160882+0000", 2024-09-13T11:24:33.173 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00017664699999999999 2024-09-13T11:24:33.173 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:33.173 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:33.173 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:24:33.173 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:33.163102+0000", 2024-09-13T11:24:33.174 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0022195969999999998 2024-09-13T11:24:33.174 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:33.174 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:33.174 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:33.174 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:33.174 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:33.174 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPushReply(3.12 125/122 [PushReplyOp(3:4d405308:::benchmark_data_smithi177_31116_object7161:head)])", 2024-09-13T11:24:33.174 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:33.167286+0000", 2024-09-13T11:24:33.174 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 0.00028470500000000001, 2024-09-13T11:24:33.174 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00035067300000000002, 2024-09-13T11:24:33.174 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:33.174 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:33.174 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-13T11:24:33.175 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:33.175 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:33.175 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:33.175 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:33.167286+0000", 2024-09-13T11:24:33.175 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:33.175 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:33.175 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:33.175 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:33.175 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:33.167286+0000", 2024-09-13T11:24:33.175 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:33.175 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:33.175 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:33.175 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:33.175 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:33.167288+0000", 2024-09-13T11:24:33.175 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.362e-06 2024-09-13T11:24:33.176 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:33.176 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:33.176 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:33.176 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:33.167294+0000", 2024-09-13T11:24:33.176 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.2169999999999999e-06 2024-09-13T11:24:33.176 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:33.176 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:33.176 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:33.176 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:33.167330+0000", 2024-09-13T11:24:33.176 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.6554000000000003e-05 2024-09-13T11:24:33.176 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:33.176 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:33.176 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:33.177 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:33.167339+0000", 2024-09-13T11:24:33.177 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.8540000000000003e-06 2024-09-13T11:24:33.177 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:33.177 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:33.177 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:33.177 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:33.167393+0000", 2024-09-13T11:24:33.177 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.3764999999999998e-05 2024-09-13T11:24:33.177 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:33.177 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:33.177 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:33.177 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:33.177 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:33.177 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop_reply(client.4665.0:114646 3.12 e125/122)", 2024-09-13T11:24:33.178 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:33.167314+0000", 2024-09-13T11:24:33.178 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 0.00025668799999999998, 2024-09-13T11:24:33.178 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000345982, 2024-09-13T11:24:33.178 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:33.178 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:33.178 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "queued for pg", 2024-09-13T11:24:33.178 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:33.178 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:33.178 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:33.178 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:33.167314+0000", 2024-09-13T11:24:33.178 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:33.178 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:33.178 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:33.178 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:33.179 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:33.167314+0000", 2024-09-13T11:24:33.179 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:33.179 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:33.179 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:33.179 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:33.179 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:33.167315+0000", 2024-09-13T11:24:33.179 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.0200000000000002e-07 2024-09-13T11:24:33.179 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:33.179 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:33.179 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:33.179 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:33.167318+0000", 2024-09-13T11:24:33.179 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.8739999999999999e-06 2024-09-13T11:24:33.179 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:33.179 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:33.179 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:33.179 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:33.167362+0000", 2024-09-13T11:24:33.180 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.3954e-05 2024-09-13T11:24:33.180 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:33.180 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:33.180 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:33.180 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:33.167366+0000", 2024-09-13T11:24:33.180 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.4340000000000002e-06 2024-09-13T11:24:33.180 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:33.180 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:33.180 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:33.180 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:33.180 INFO:teuthology.orchestra.run.smithi177.stdout: ], 2024-09-13T11:24:33.180 INFO:teuthology.orchestra.run.smithi177.stdout: "num_ops": 3 2024-09-13T11:24:33.180 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-09-13T11:24:33.196 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-13T11:24:33.260 INFO:tasks.ceph.osd.4.smithi001.stderr:2024-09-13T11:24:33.259+0000 7ff8e0218640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26310) UID: 0 2024-09-13T11:24:33.361 INFO:tasks.ceph.osd.10.smithi162.stderr:2024-09-13T11:24:33.356+0000 7fb59c9f0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26317) UID: 0 2024-09-13T11:24:33.421 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-09-13T11:24:33.421 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [], 2024-09-13T11:24:33.421 INFO:teuthology.orchestra.run.smithi177.stdout: "complaint_time": 30, 2024-09-13T11:24:33.421 INFO:teuthology.orchestra.run.smithi177.stdout: "num_blocked_ops": 0 2024-09-13T11:24:33.421 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-09-13T11:24:33.446 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-13T11:24:33.462 INFO:tasks.ceph.osd.12.smithi001.stderr:2024-09-13T11:24:33.459+0000 7f892c6e6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26314) UID: 0 2024-09-13T11:24:33.563 INFO:tasks.ceph.osd.12.smithi001.stderr:2024-09-13T11:24:33.559+0000 7f892c6e6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26314) UID: 0 2024-09-13T11:24:33.610 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-09-13T11:24:33.611 INFO:teuthology.orchestra.run.smithi177.stdout: "size": 20, 2024-09-13T11:24:33.611 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 600, 2024-09-13T11:24:33.611 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [ 2024-09-13T11:24:33.611 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:33.611 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPushReply(3.13 75/70 [PushReplyOp(3:cfab03d6:::benchmark_data_smithi177_31116_object16599:head)])", 2024-09-13T11:24:33.611 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:31.163480+0000", 2024-09-13T11:24:33.611 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 62.445151177, 2024-09-13T11:24:33.611 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.63009029299999997, 2024-09-13T11:24:33.611 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:33.611 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:33.611 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-13T11:24:33.611 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:33.611 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:33.611 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:33.612 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.163480+0000", 2024-09-13T11:24:33.612 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:33.612 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:33.612 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:33.612 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:33.612 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.163480+0000", 2024-09-13T11:24:33.612 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:33.612 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:33.612 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:33.612 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:33.612 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.163481+0000", 2024-09-13T11:24:33.612 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0410000000000001e-06 2024-09-13T11:24:33.612 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:33.612 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:33.613 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:33.613 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.163485+0000", 2024-09-13T11:24:33.613 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.1629999999999998e-06 2024-09-13T11:24:33.613 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:33.613 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:33.613 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:33.613 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.792166+0000", 2024-09-13T11:24:33.613 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.62868067400000005 2024-09-13T11:24:33.613 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:33.613 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:33.613 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:33.613 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.792184+0000", 2024-09-13T11:24:33.613 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.7898999999999999e-05 2024-09-13T11:24:33.613 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:33.614 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:33.614 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:33.614 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.792312+0000", 2024-09-13T11:24:33.614 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00012871000000000001 2024-09-13T11:24:33.614 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:33.614 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:33.614 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:24:33.614 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.793570+0000", 2024-09-13T11:24:33.614 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001257806 2024-09-13T11:24:33.614 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:33.614 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:33.614 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:33.614 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:33.614 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:33.614 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPush(3.12 75/69 [PushOp(3:4a36ca6d:::benchmark_data_smithi177_31116_object31707:head, version: 59'1993, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:4a36ca6d:::benchmark_data_smithi177_31116_object31707:head@59'1993, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:24:33.615 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:31.309623+0000", 2024-09-13T11:24:33.615 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 62.299008221999998, 2024-09-13T11:24:33.615 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.165459152, 2024-09-13T11:24:33.615 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:33.615 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:33.615 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:24:33.615 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:33.615 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:33.615 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:33.615 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.309623+0000", 2024-09-13T11:24:33.615 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:33.615 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:33.615 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:33.615 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:33.616 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.309623+0000", 2024-09-13T11:24:33.616 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:33.616 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:33.616 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:33.616 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:33.616 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.309625+0000", 2024-09-13T11:24:33.616 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.3860000000000001e-06 2024-09-13T11:24:33.616 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:33.616 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:33.616 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:33.616 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.309680+0000", 2024-09-13T11:24:33.616 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.4388000000000002e-05 2024-09-13T11:24:33.616 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:33.616 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:33.616 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:33.617 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.473157+0000", 2024-09-13T11:24:33.617 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.163476813 2024-09-13T11:24:33.617 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:33.617 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:33.617 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:33.617 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.473189+0000", 2024-09-13T11:24:33.617 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.2975000000000002e-05 2024-09-13T11:24:33.617 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:33.617 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:33.617 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:33.617 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.473395+0000", 2024-09-13T11:24:33.617 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000206009 2024-09-13T11:24:33.618 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:33.618 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:33.618 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:24:33.618 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.473424+0000", 2024-09-13T11:24:33.618 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.8255e-05 2024-09-13T11:24:33.618 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:33.618 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:33.618 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:24:33.618 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.475082+0000", 2024-09-13T11:24:33.618 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0016583259999999999 2024-09-13T11:24:33.618 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:33.618 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:33.618 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:33.619 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:33.619 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:33.619 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPushReply(3.13 76/70 [PushReplyOp(3:c9d43a82:::benchmark_data_smithi177_31116_object20026:head)])", 2024-09-13T11:24:33.619 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:32.427709+0000", 2024-09-13T11:24:33.619 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 61.180921662000003, 2024-09-13T11:24:33.619 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.46547316300000002, 2024-09-13T11:24:33.619 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:33.619 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:33.619 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-13T11:24:33.619 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:33.619 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:33.619 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:33.619 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:32.427709+0000", 2024-09-13T11:24:33.619 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:33.620 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:33.620 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:33.620 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:33.620 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:32.427709+0000", 2024-09-13T11:24:33.620 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:33.620 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:33.620 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:33.620 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:33.620 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:32.427711+0000", 2024-09-13T11:24:33.620 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5099999999999999e-06 2024-09-13T11:24:33.620 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:33.620 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:33.620 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:33.621 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:32.427717+0000", 2024-09-13T11:24:33.621 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.1569999999999997e-06 2024-09-13T11:24:33.621 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:33.621 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:33.621 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:33.621 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:32.892390+0000", 2024-09-13T11:24:33.621 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.46467251100000001 2024-09-13T11:24:33.621 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:33.621 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:33.621 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:33.621 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:32.892405+0000", 2024-09-13T11:24:33.621 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5296999999999999e-05 2024-09-13T11:24:33.621 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:33.622 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:33.622 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:33.622 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:32.892503+0000", 2024-09-13T11:24:33.622 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.7820999999999997e-05 2024-09-13T11:24:33.622 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:33.622 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:33.622 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:24:33.622 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:32.893183+0000", 2024-09-13T11:24:33.622 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00067986700000000001 2024-09-13T11:24:33.622 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:33.622 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:33.622 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:33.622 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:33.622 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:33.622 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:56066 3.a e79/75 3:561a1dd2:::benchmark_data_smithi177_31116_object56065:head v 79'3452, mlcod=79'3449)", 2024-09-13T11:24:33.623 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:36.194429+0000", 2024-09-13T11:24:33.623 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 57.414201777000002, 2024-09-13T11:24:33.623 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.13436872499999999, 2024-09-13T11:24:33.623 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:33.623 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:33.623 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:33.623 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:33.623 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:33.623 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:33.623 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:36.194429+0000", 2024-09-13T11:24:33.623 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:33.623 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:33.623 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:33.623 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:33.624 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:36.194429+0000", 2024-09-13T11:24:33.624 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:33.624 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:33.624 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:33.624 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:33.624 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:36.194431+0000", 2024-09-13T11:24:33.624 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.4279999999999999e-06 2024-09-13T11:24:33.624 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:33.624 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:33.624 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:33.624 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:36.194467+0000", 2024-09-13T11:24:33.624 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.6402000000000001e-05 2024-09-13T11:24:33.625 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:33.625 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:33.625 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:33.625 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:36.327354+0000", 2024-09-13T11:24:33.625 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.13288720800000001 2024-09-13T11:24:33.625 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:33.625 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:33.625 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:33.625 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:36.327361+0000", 2024-09-13T11:24:33.625 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.302e-06 2024-09-13T11:24:33.625 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:33.625 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:33.625 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:33.625 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:36.327441+0000", 2024-09-13T11:24:33.626 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.0406000000000002e-05 2024-09-13T11:24:33.626 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:33.626 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:33.626 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:24:33.626 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:36.327466+0000", 2024-09-13T11:24:33.626 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.5367e-05 2024-09-13T11:24:33.626 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:33.626 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:33.626 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:24:33.626 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:36.328754+0000", 2024-09-13T11:24:33.626 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001287627 2024-09-13T11:24:33.626 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:33.626 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:33.626 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:24:33.626 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:36.328798+0000", 2024-09-13T11:24:33.627 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.3985e-05 2024-09-13T11:24:33.627 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:33.627 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:33.627 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:33.627 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:33.627 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:33.627 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:57435 3.3 e80/75 3:c0db4777:::benchmark_data_smithi177_31116_object57434:head v 80'3155, mlcod=80'3153)", 2024-09-13T11:24:33.627 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:37.437070+0000", 2024-09-13T11:24:33.627 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 56.171560845000002, 2024-09-13T11:24:33.627 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.35861845999999997, 2024-09-13T11:24:33.627 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:33.627 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:33.627 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:33.628 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:33.628 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:33.628 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:33.628 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:37.437070+0000", 2024-09-13T11:24:33.628 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:33.628 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:33.628 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:33.628 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:33.628 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:37.437070+0000", 2024-09-13T11:24:33.628 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:33.628 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:33.629 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:33.629 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:33.629 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:37.437072+0000", 2024-09-13T11:24:33.629 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5549999999999999e-06 2024-09-13T11:24:33.629 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:33.629 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:33.629 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:33.629 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:37.437141+0000", 2024-09-13T11:24:33.629 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.9494999999999998e-05 2024-09-13T11:24:33.630 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:33.630 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:33.630 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:33.630 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:37.793400+0000", 2024-09-13T11:24:33.630 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.35625817999999998 2024-09-13T11:24:33.630 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:33.630 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:33.630 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:33.630 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:37.793406+0000", 2024-09-13T11:24:33.630 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.6830000000000002e-06 2024-09-13T11:24:33.630 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:33.630 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:33.630 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:33.631 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:37.793472+0000", 2024-09-13T11:24:33.631 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.5671000000000002e-05 2024-09-13T11:24:33.631 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:33.631 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:33.631 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:24:33.631 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:37.793492+0000", 2024-09-13T11:24:33.631 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.0051000000000002e-05 2024-09-13T11:24:33.631 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:33.631 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:33.631 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:24:33.631 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:37.795626+0000", 2024-09-13T11:24:33.631 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0021344990000000002 2024-09-13T11:24:33.631 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:33.632 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:33.632 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:24:33.632 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:37.795689+0000", 2024-09-13T11:24:33.632 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.2325999999999999e-05 2024-09-13T11:24:33.632 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:33.632 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:33.632 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:33.632 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:33.632 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:33.632 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:57985 3.3 e80/75 3:c330bb03:::benchmark_data_smithi177_31116_object57984:head v 80'3162, mlcod=80'3159)", 2024-09-13T11:24:33.632 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:38.007076+0000", 2024-09-13T11:24:33.632 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 55.601555492999999, 2024-09-13T11:24:33.632 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.18211438199999999, 2024-09-13T11:24:33.633 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:33.633 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:33.633 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:33.633 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:33.633 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:33.633 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:33.633 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:38.007076+0000", 2024-09-13T11:24:33.633 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:33.633 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:33.633 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:33.633 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:33.633 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:38.007076+0000", 2024-09-13T11:24:33.633 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:33.634 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:33.634 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:33.634 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:33.634 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:38.007077+0000", 2024-09-13T11:24:33.634 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.6759999999999999e-06 2024-09-13T11:24:33.634 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:33.634 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:33.634 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:33.634 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:38.007221+0000", 2024-09-13T11:24:33.634 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00014344299999999999 2024-09-13T11:24:33.634 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:33.634 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:33.634 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:33.635 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:38.186808+0000", 2024-09-13T11:24:33.635 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.17958726699999999 2024-09-13T11:24:33.635 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:33.635 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:33.635 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:33.635 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:38.186815+0000", 2024-09-13T11:24:33.635 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.0890000000000002e-06 2024-09-13T11:24:33.635 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:33.635 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:33.635 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:33.635 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:38.186887+0000", 2024-09-13T11:24:33.635 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.2006000000000001e-05 2024-09-13T11:24:33.635 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:33.636 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:33.636 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:24:33.636 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:38.186913+0000", 2024-09-13T11:24:33.636 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.5962e-05 2024-09-13T11:24:33.636 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:33.636 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:33.636 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:24:33.636 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:38.189123+0000", 2024-09-13T11:24:33.636 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0022099649999999999 2024-09-13T11:24:33.636 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:33.636 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:33.636 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:24:33.636 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:38.189190+0000", 2024-09-13T11:24:33.637 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.6973999999999999e-05 2024-09-13T11:24:33.637 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:33.637 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:33.637 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:33.637 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:33.637 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:33.637 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPushReply(3.23 84/78 [PushReplyOp(3:c7e44616:::benchmark_data_smithi177_31116_object32371:head)])", 2024-09-13T11:24:33.637 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:41.338210+0000", 2024-09-13T11:24:33.637 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 52.270420774000002, 2024-09-13T11:24:33.637 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.41118492299999998, 2024-09-13T11:24:33.637 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:33.637 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:33.637 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-13T11:24:33.638 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:33.638 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:33.638 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:33.638 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.338210+0000", 2024-09-13T11:24:33.638 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:33.638 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:33.638 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:33.638 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:33.638 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.338210+0000", 2024-09-13T11:24:33.638 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:33.638 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:33.638 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:33.638 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:33.638 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.338212+0000", 2024-09-13T11:24:33.638 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.314e-06 2024-09-13T11:24:33.639 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:33.639 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:33.639 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:33.639 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.338219+0000", 2024-09-13T11:24:33.639 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.6469999999999998e-06 2024-09-13T11:24:33.639 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:33.639 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:33.639 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:33.639 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.748675+0000", 2024-09-13T11:24:33.639 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.41045567500000002 2024-09-13T11:24:33.639 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:33.639 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:33.639 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:33.639 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.748684+0000", 2024-09-13T11:24:33.639 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.2150000000000007e-06 2024-09-13T11:24:33.640 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:33.640 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:33.640 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:33.640 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.748739+0000", 2024-09-13T11:24:33.640 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.4543000000000003e-05 2024-09-13T11:24:33.640 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:33.640 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:33.640 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:24:33.640 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.749395+0000", 2024-09-13T11:24:33.640 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00065652899999999999 2024-09-13T11:24:33.640 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:33.640 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:33.640 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:33.640 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:33.641 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:33.641 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPushReply(3.23 85/78 [PushReplyOp(3:c5c6dce1:::benchmark_data_smithi177_31116_object37293:head)])", 2024-09-13T11:24:33.641 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:42.377862+0000", 2024-09-13T11:24:33.641 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 51.230769315000003, 2024-09-13T11:24:33.641 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.62538808599999995, 2024-09-13T11:24:33.641 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:33.641 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:33.641 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-13T11:24:33.641 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:33.641 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:33.641 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:33.641 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.377862+0000", 2024-09-13T11:24:33.641 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:33.641 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:33.642 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:33.642 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:33.642 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.377862+0000", 2024-09-13T11:24:33.642 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:33.642 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:33.642 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:33.642 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:33.642 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.377863+0000", 2024-09-13T11:24:33.642 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.6899999999999996e-07 2024-09-13T11:24:33.642 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:33.642 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:33.642 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:33.642 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.377866+0000", 2024-09-13T11:24:33.642 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.219e-06 2024-09-13T11:24:33.643 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:33.643 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:33.643 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:33.643 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:43.001857+0000", 2024-09-13T11:24:33.643 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.62399084299999996 2024-09-13T11:24:33.643 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:33.643 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:33.643 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:33.643 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:43.001869+0000", 2024-09-13T11:24:33.643 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.2415999999999999e-05 2024-09-13T11:24:33.643 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:33.643 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:33.643 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:33.644 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:43.002194+0000", 2024-09-13T11:24:33.644 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000324774 2024-09-13T11:24:33.644 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:33.644 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:33.644 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:24:33.644 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:43.003250+0000", 2024-09-13T11:24:33.644 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001055865 2024-09-13T11:24:33.644 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:33.644 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:33.644 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:33.644 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:33.644 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:33.644 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPushReply(3.23 86/78 [PushReplyOp(3:c4ca4518:::benchmark_data_smithi177_31116_object39787:head)])", 2024-09-13T11:24:33.645 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:43.580816+0000", 2024-09-13T11:24:33.645 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 50.027815623999999, 2024-09-13T11:24:33.645 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.14999331199999999, 2024-09-13T11:24:33.645 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:33.645 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:33.645 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-13T11:24:33.645 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:33.645 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:33.645 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:33.645 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:43.580816+0000", 2024-09-13T11:24:33.645 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:33.645 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:33.645 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:33.646 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:33.646 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:43.580816+0000", 2024-09-13T11:24:33.646 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:33.646 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:33.646 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:33.646 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:33.646 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:43.580818+0000", 2024-09-13T11:24:33.646 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.7250000000000002e-06 2024-09-13T11:24:33.646 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:33.646 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:33.646 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:33.646 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:43.580830+0000", 2024-09-13T11:24:33.646 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.2247e-05 2024-09-13T11:24:33.646 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:33.647 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:33.647 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:33.647 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:43.730204+0000", 2024-09-13T11:24:33.647 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.149373531 2024-09-13T11:24:33.647 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:33.647 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:33.647 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:33.647 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:43.730213+0000", 2024-09-13T11:24:33.647 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.6989999999999992e-06 2024-09-13T11:24:33.647 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:33.647 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:33.647 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:33.647 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:43.730274+0000", 2024-09-13T11:24:33.647 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.1419000000000003e-05 2024-09-13T11:24:33.648 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:33.648 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:33.648 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:24:33.648 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:43.730809+0000", 2024-09-13T11:24:33.648 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00053469100000000001 2024-09-13T11:24:33.648 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:33.648 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:33.648 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:33.648 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:33.648 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:33.648 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPush(3.1c 89/77 [PushOp(3:38bcbc71:::benchmark_data_smithi177_31116_object31448:head, version: 59'1946, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:38bcbc71:::benchmark_data_smithi177_31116_object31448:head@59'1946, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:24:33.648 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:46.670723+0000", 2024-09-13T11:24:33.649 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 46.937908090999997, 2024-09-13T11:24:33.649 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.27576065, 2024-09-13T11:24:33.649 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:33.649 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:33.649 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:24:33.649 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:33.649 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:33.649 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:33.649 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.670723+0000", 2024-09-13T11:24:33.649 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:33.649 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:33.649 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:33.649 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:33.649 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.670723+0000", 2024-09-13T11:24:33.650 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:33.650 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:33.650 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:33.650 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:33.650 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.670724+0000", 2024-09-13T11:24:33.650 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.4389999999999999e-06 2024-09-13T11:24:33.650 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:33.650 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:33.650 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:33.650 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.670754+0000", 2024-09-13T11:24:33.650 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.9224999999999999e-05 2024-09-13T11:24:33.650 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:33.650 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:33.650 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:33.651 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.944369+0000", 2024-09-13T11:24:33.651 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.27361529800000001 2024-09-13T11:24:33.651 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:33.651 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:33.651 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:33.651 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.944422+0000", 2024-09-13T11:24:33.651 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.3318e-05 2024-09-13T11:24:33.651 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:33.651 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:33.651 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:33.651 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.944711+0000", 2024-09-13T11:24:33.651 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00028912399999999998 2024-09-13T11:24:33.651 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:33.652 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:33.652 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:24:33.652 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.944735+0000", 2024-09-13T11:24:33.652 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.3366999999999998e-05 2024-09-13T11:24:33.652 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:33.652 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:33.652 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:24:33.652 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.946484+0000", 2024-09-13T11:24:33.652 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0017488790000000001 2024-09-13T11:24:33.652 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:33.652 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:33.652 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:33.652 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:33.653 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:33.653 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPush(3.1c 91/77 [PushOp(3:3e18abc2:::benchmark_data_smithi177_31116_object35874:head, version: 62'2235, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:3e18abc2:::benchmark_data_smithi177_31116_object35874:head@62'2235, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:24:33.653 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:48.350968+0000", 2024-09-13T11:24:33.653 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 45.257662777999997, 2024-09-13T11:24:33.653 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.19257698000000001, 2024-09-13T11:24:33.653 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:33.653 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:33.653 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:24:33.653 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:33.653 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:33.653 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:33.653 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.350968+0000", 2024-09-13T11:24:33.653 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:33.654 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:33.654 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:33.654 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:33.654 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.350968+0000", 2024-09-13T11:24:33.654 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:33.654 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:33.654 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:33.654 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:33.654 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.350970+0000", 2024-09-13T11:24:33.654 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.796e-06 2024-09-13T11:24:33.654 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:33.654 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:33.654 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:33.654 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.351003+0000", 2024-09-13T11:24:33.655 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.2631e-05 2024-09-13T11:24:33.655 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:33.655 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:33.655 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:33.655 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.542140+0000", 2024-09-13T11:24:33.655 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.191136737 2024-09-13T11:24:33.655 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:33.655 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:33.655 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:33.655 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.542158+0000", 2024-09-13T11:24:33.655 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.8896999999999999e-05 2024-09-13T11:24:33.655 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:33.655 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:33.655 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:33.656 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.542263+0000", 2024-09-13T11:24:33.656 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000104158 2024-09-13T11:24:33.656 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:33.656 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:33.656 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:24:33.656 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.542279+0000", 2024-09-13T11:24:33.656 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5926999999999998e-05 2024-09-13T11:24:33.656 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:33.656 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:33.656 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:24:33.656 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.543545+0000", 2024-09-13T11:24:33.656 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0012668340000000001 2024-09-13T11:24:33.656 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:33.657 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:33.657 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:33.657 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:33.657 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:33.657 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:83211 3.1c e106/77 3:3ae31a0f:::benchmark_data_smithi177_31116_object83210:head v 106'4362, mlcod=106'4360)", 2024-09-13T11:24:33.657 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:04.345264+0000", 2024-09-13T11:24:33.657 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 29.263367125999999, 2024-09-13T11:24:33.657 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.17036547399999999, 2024-09-13T11:24:33.657 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:33.657 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:33.657 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:33.657 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:33.657 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:33.657 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:33.658 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.345264+0000", 2024-09-13T11:24:33.658 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:33.658 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:33.658 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:33.658 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:33.658 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.345264+0000", 2024-09-13T11:24:33.658 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:33.658 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:33.658 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:33.658 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:33.658 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.345267+0000", 2024-09-13T11:24:33.658 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.7590000000000002e-06 2024-09-13T11:24:33.658 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:33.658 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:33.659 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:33.659 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.345298+0000", 2024-09-13T11:24:33.659 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.1173000000000001e-05 2024-09-13T11:24:33.659 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:33.659 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:33.659 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:33.659 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.512265+0000", 2024-09-13T11:24:33.659 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.16696657100000001 2024-09-13T11:24:33.659 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:33.659 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:33.659 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:33.659 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.512279+0000", 2024-09-13T11:24:33.659 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.4953999999999999e-05 2024-09-13T11:24:33.659 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:33.660 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:33.660 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:33.660 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.512402+0000", 2024-09-13T11:24:33.660 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00012277999999999999 2024-09-13T11:24:33.660 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:33.660 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:33.660 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:24:33.660 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.512440+0000", 2024-09-13T11:24:33.660 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.7861999999999998e-05 2024-09-13T11:24:33.660 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:33.660 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:33.660 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:24:33.660 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.515515+0000", 2024-09-13T11:24:33.660 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00307472 2024-09-13T11:24:33.661 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:33.661 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:33.661 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:24:33.661 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.515629+0000", 2024-09-13T11:24:33.661 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00011465500000000001 2024-09-13T11:24:33.661 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:33.661 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:33.661 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:33.661 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:33.661 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:33.661 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:83223 3.1c e106/77 3:3bc6018d:::benchmark_data_smithi177_31116_object83222:head v 106'4363, mlcod=106'4360)", 2024-09-13T11:24:33.661 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:04.512139+0000", 2024-09-13T11:24:33.662 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 29.09649198, 2024-09-13T11:24:33.662 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.471923235, 2024-09-13T11:24:33.662 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:33.662 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:33.662 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:33.662 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:33.662 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:33.662 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:33.662 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.512139+0000", 2024-09-13T11:24:33.662 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:33.662 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:33.662 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:33.662 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:33.663 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.512139+0000", 2024-09-13T11:24:33.663 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:33.663 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:33.663 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:33.663 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:33.663 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.512143+0000", 2024-09-13T11:24:33.663 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.427e-06 2024-09-13T11:24:33.663 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:33.663 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:33.663 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:33.663 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.512227+0000", 2024-09-13T11:24:33.663 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.3965e-05 2024-09-13T11:24:33.663 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:33.664 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:33.664 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:33.664 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.976330+0000", 2024-09-13T11:24:33.664 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.46410342599999999 2024-09-13T11:24:33.664 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:33.664 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:33.664 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:33.664 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.976350+0000", 2024-09-13T11:24:33.664 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.9533000000000001e-05 2024-09-13T11:24:33.664 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:33.664 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:33.664 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:33.665 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.976488+0000", 2024-09-13T11:24:33.665 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00013799400000000001 2024-09-13T11:24:33.665 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:33.665 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:33.665 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:24:33.665 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.976543+0000", 2024-09-13T11:24:33.665 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.5442000000000002e-05 2024-09-13T11:24:33.665 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:33.665 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:33.665 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:24:33.665 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.983227+0000", 2024-09-13T11:24:33.665 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0066842899999999998 2024-09-13T11:24:33.665 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:33.666 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:33.666 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:24:33.666 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.984062+0000", 2024-09-13T11:24:33.666 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00083515800000000002 2024-09-13T11:24:33.666 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:33.666 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:33.666 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:33.666 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:33.666 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:33.666 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:110256 3.13 e119/83 3:cb00d078:::benchmark_data_smithi177_31116_object110255:head v 119'4823, mlcod=119'4821)", 2024-09-13T11:24:33.666 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:25.284530+0000", 2024-09-13T11:24:33.666 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 8.3241015199999993, 2024-09-13T11:24:33.666 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.343491771, 2024-09-13T11:24:33.667 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:33.667 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:33.667 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:33.667 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:33.667 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:33.667 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:33.667 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.284530+0000", 2024-09-13T11:24:33.667 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:33.667 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:33.667 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:33.667 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:33.667 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.284530+0000", 2024-09-13T11:24:33.667 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:33.668 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:33.668 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:33.668 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:33.668 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.284532+0000", 2024-09-13T11:24:33.668 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.8870000000000001e-06 2024-09-13T11:24:33.668 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:33.668 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:33.668 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:33.668 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.284558+0000", 2024-09-13T11:24:33.668 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.6899000000000001e-05 2024-09-13T11:24:33.668 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:33.668 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:33.668 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:33.669 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.626172+0000", 2024-09-13T11:24:33.669 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.34161370800000002 2024-09-13T11:24:33.669 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:33.669 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:33.669 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:33.669 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.626187+0000", 2024-09-13T11:24:33.669 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.485e-05 2024-09-13T11:24:33.669 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:33.669 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:33.669 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:33.669 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.626245+0000", 2024-09-13T11:24:33.669 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.7998000000000003e-05 2024-09-13T11:24:33.669 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:33.670 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:33.670 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:24:33.670 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.626260+0000", 2024-09-13T11:24:33.670 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5429000000000001e-05 2024-09-13T11:24:33.670 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:33.670 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:33.670 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:24:33.670 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.627924+0000", 2024-09-13T11:24:33.670 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0016634060000000001 2024-09-13T11:24:33.670 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:33.670 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:33.670 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:24:33.670 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.628021+0000", 2024-09-13T11:24:33.671 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.7594000000000001e-05 2024-09-13T11:24:33.671 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:33.671 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:33.671 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:33.671 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:33.671 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:33.671 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPush(3.2b 121/119 [PushOp(3:d53a23e8:::benchmark_data_smithi177_31116_object5366:head, version: 45'342, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:d53a23e8:::benchmark_data_smithi177_31116_object5366:head@45'342, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:24:33.671 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:27.268731+0000", 2024-09-13T11:24:33.671 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 6.3399000579999996, 2024-09-13T11:24:33.671 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.72228345000000005, 2024-09-13T11:24:33.671 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:33.671 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:33.671 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:24:33.672 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:33.672 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:33.672 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:33.672 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:27.268731+0000", 2024-09-13T11:24:33.672 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:33.672 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:33.672 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:33.672 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:33.672 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:27.268731+0000", 2024-09-13T11:24:33.672 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:33.672 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:33.672 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:33.673 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:33.673 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:27.268733+0000", 2024-09-13T11:24:33.673 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.9740000000000001e-06 2024-09-13T11:24:33.673 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:33.673 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:33.673 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:33.673 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:27.268778+0000", 2024-09-13T11:24:33.673 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.4679999999999999e-05 2024-09-13T11:24:33.673 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:33.673 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:33.673 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:33.673 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:27.988594+0000", 2024-09-13T11:24:33.673 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.71981667299999996 2024-09-13T11:24:33.674 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:33.674 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:33.674 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:33.674 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:27.988652+0000", 2024-09-13T11:24:33.674 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.7640999999999999e-05 2024-09-13T11:24:33.674 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:33.674 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:33.674 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:33.674 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:27.988889+0000", 2024-09-13T11:24:33.674 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00023706500000000001 2024-09-13T11:24:33.674 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:33.674 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:33.674 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:24:33.674 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:27.988933+0000", 2024-09-13T11:24:33.675 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.3865000000000001e-05 2024-09-13T11:24:33.675 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:33.675 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:33.675 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:24:33.675 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:27.991015+0000", 2024-09-13T11:24:33.675 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.002081552 2024-09-13T11:24:33.675 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:33.675 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:33.675 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:33.675 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:33.675 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:33.675 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPush(3.2b 124/119 [PushOp(3:d64c8aaf:::benchmark_data_smithi177_31116_object31700:head, version: 59'1991, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:d64c8aaf:::benchmark_data_smithi177_31116_object31700:head@59'1991, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:24:33.675 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:31.579304+0000", 2024-09-13T11:24:33.675 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 2.0293273570000001, 2024-09-13T11:24:33.676 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.14279340800000001, 2024-09-13T11:24:33.676 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:33.676 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:33.676 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:24:33.676 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:33.676 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:33.676 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:33.676 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:31.579304+0000", 2024-09-13T11:24:33.676 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:33.676 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:33.676 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:33.676 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:33.676 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:31.579304+0000", 2024-09-13T11:24:33.676 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:33.677 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:33.677 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:33.677 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:33.677 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:31.579305+0000", 2024-09-13T11:24:33.677 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5090000000000001e-06 2024-09-13T11:24:33.677 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:33.677 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:33.677 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:33.677 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:31.579367+0000", 2024-09-13T11:24:33.677 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.1279999999999996e-05 2024-09-13T11:24:33.677 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:33.677 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:33.677 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:33.678 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:31.721234+0000", 2024-09-13T11:24:33.678 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.141867883 2024-09-13T11:24:33.678 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:33.678 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:33.678 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:33.678 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:31.721253+0000", 2024-09-13T11:24:33.678 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.8725000000000001e-05 2024-09-13T11:24:33.678 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:33.678 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:33.678 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:33.678 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:31.721332+0000", 2024-09-13T11:24:33.678 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.8808999999999997e-05 2024-09-13T11:24:33.678 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:33.678 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:33.679 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:24:33.679 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:31.721345+0000", 2024-09-13T11:24:33.679 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3414999999999999e-05 2024-09-13T11:24:33.679 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:33.679 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:33.679 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:24:33.679 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:31.722097+0000", 2024-09-13T11:24:33.679 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00075178699999999996 2024-09-13T11:24:33.679 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:33.679 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:33.679 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:33.679 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:33.679 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:33.680 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:113511 3.2b e124/119 3:d74f5e33:::benchmark_data_smithi177_31116_object113510:head v 124'5156, mlcod=60'2020)", 2024-09-13T11:24:33.680 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:32.032517+0000", 2024-09-13T11:24:33.680 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 1.576114193, 2024-09-13T11:24:33.680 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.13874039699999999, 2024-09-13T11:24:33.680 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:33.680 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:33.680 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:33.680 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:33.680 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:33.680 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:33.680 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:32.032517+0000", 2024-09-13T11:24:33.680 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:33.680 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:33.681 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:33.681 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:33.681 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:32.032517+0000", 2024-09-13T11:24:33.681 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:33.681 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:33.681 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:33.681 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:33.681 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:32.032518+0000", 2024-09-13T11:24:33.681 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3060000000000001e-06 2024-09-13T11:24:33.681 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:33.681 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:33.681 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:33.681 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:32.032545+0000", 2024-09-13T11:24:33.682 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.6800000000000001e-05 2024-09-13T11:24:33.682 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:33.682 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:33.682 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:33.682 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:32.169285+0000", 2024-09-13T11:24:33.682 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.13673950800000001 2024-09-13T11:24:33.682 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:33.682 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:33.682 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:33.682 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:32.169291+0000", 2024-09-13T11:24:33.682 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.934e-06 2024-09-13T11:24:33.682 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:33.682 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:33.683 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:33.683 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:32.169381+0000", 2024-09-13T11:24:33.683 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.9860999999999999e-05 2024-09-13T11:24:33.683 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:33.683 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:33.683 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:24:33.683 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:32.169406+0000", 2024-09-13T11:24:33.683 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.5052e-05 2024-09-13T11:24:33.683 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:33.683 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:33.683 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:24:33.683 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:32.171218+0000", 2024-09-13T11:24:33.683 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0018111659999999999 2024-09-13T11:24:33.683 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:33.684 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:33.684 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:24:33.684 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:32.171257+0000", 2024-09-13T11:24:33.684 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.977e-05 2024-09-13T11:24:33.684 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:33.684 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:33.684 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:33.684 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:33.684 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:33.684 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPush(3.2b 124/119 [PushOp(3:d70bd127:::benchmark_data_smithi177_31116_object32365:head, version: 60'2034, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:d70bd127:::benchmark_data_smithi177_31116_object32365:head@60'2034, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:24:33.684 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:32.037338+0000", 2024-09-13T11:24:33.684 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 1.571292785, 2024-09-13T11:24:33.684 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.24358922699999999, 2024-09-13T11:24:33.685 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:33.685 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:33.685 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:24:33.685 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:33.685 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:33.685 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:33.685 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:32.037338+0000", 2024-09-13T11:24:33.685 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:33.685 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:33.685 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:33.685 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:33.685 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:32.037338+0000", 2024-09-13T11:24:33.685 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:33.685 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:33.686 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:33.686 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:33.686 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:32.037340+0000", 2024-09-13T11:24:33.686 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3790000000000001e-06 2024-09-13T11:24:33.686 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:33.686 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:33.686 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:33.686 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:32.037411+0000", 2024-09-13T11:24:33.686 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.1254000000000006e-05 2024-09-13T11:24:33.686 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:33.686 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:33.686 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:33.686 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:32.277467+0000", 2024-09-13T11:24:33.686 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.24005563199999999 2024-09-13T11:24:33.687 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:33.687 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:33.687 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:33.687 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:32.277494+0000", 2024-09-13T11:24:33.687 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.7120000000000001e-05 2024-09-13T11:24:33.687 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:33.687 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:33.687 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:33.687 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:32.279860+0000", 2024-09-13T11:24:33.687 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0023667559999999998 2024-09-13T11:24:33.687 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:33.687 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:33.687 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:24:33.687 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:32.279883+0000", 2024-09-13T11:24:33.688 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.2138999999999999e-05 2024-09-13T11:24:33.688 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:33.688 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:33.688 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:24:33.688 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:32.280928+0000", 2024-09-13T11:24:33.688 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0010449470000000001 2024-09-13T11:24:33.688 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:33.688 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:33.688 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:33.688 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:33.688 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:33.688 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPush(3.2b 125/119 [PushOp(3:d4c5ede9:::benchmark_data_smithi177_31116_object33277:head, version: 60'2092, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:d4c5ede9:::benchmark_data_smithi177_31116_object33277:head@60'2092, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:24:33.688 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:32.403929+0000", 2024-09-13T11:24:33.689 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 1.2047023349999999, 2024-09-13T11:24:33.689 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.194592864, 2024-09-13T11:24:33.689 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:33.689 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:33.689 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:24:33.689 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:33.689 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:33.689 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:33.689 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:32.403929+0000", 2024-09-13T11:24:33.689 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:33.689 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:33.689 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:33.689 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:33.689 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:32.403929+0000", 2024-09-13T11:24:33.690 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:33.690 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:33.690 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:33.690 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:33.690 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:32.403930+0000", 2024-09-13T11:24:33.690 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.6729999999999999e-06 2024-09-13T11:24:33.690 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:33.690 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:33.690 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:33.690 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:32.403979+0000", 2024-09-13T11:24:33.690 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.8242000000000003e-05 2024-09-13T11:24:33.690 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:33.690 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:33.691 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:33.691 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:32.597445+0000", 2024-09-13T11:24:33.691 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.19346676500000001 2024-09-13T11:24:33.691 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:33.691 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:33.691 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:33.691 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:32.597455+0000", 2024-09-13T11:24:33.691 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.2939999999999994e-06 2024-09-13T11:24:33.691 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:33.691 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:33.691 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:33.691 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:32.597514+0000", 2024-09-13T11:24:33.691 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.9509000000000002e-05 2024-09-13T11:24:33.691 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:33.692 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:33.692 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:24:33.692 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:32.597522+0000", 2024-09-13T11:24:33.692 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.0350000000000001e-06 2024-09-13T11:24:33.692 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:33.692 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:33.692 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:24:33.692 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:32.598522+0000", 2024-09-13T11:24:33.692 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00099934599999999991 2024-09-13T11:24:33.692 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:33.692 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:33.692 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:33.692 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:33.693 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:33.693 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:114438 3.3 e125/75 3:c1ccaedc:::benchmark_data_smithi177_31116_object114437:head v 125'3992, mlcod=125'3989)", 2024-09-13T11:24:33.693 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:32.953502+0000", 2024-09-13T11:24:33.693 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 0.655128975, 2024-09-13T11:24:33.693 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.13405377099999999, 2024-09-13T11:24:33.693 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:33.693 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:33.693 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:33.693 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:33.693 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:33.693 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:33.693 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:32.953502+0000", 2024-09-13T11:24:33.693 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:33.693 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:33.694 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:33.694 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:33.694 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:32.953502+0000", 2024-09-13T11:24:33.694 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:33.694 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:33.694 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:33.694 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:33.694 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:32.953511+0000", 2024-09-13T11:24:33.694 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.0960000000000007e-06 2024-09-13T11:24:33.694 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:33.694 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:33.694 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:33.694 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:32.953547+0000", 2024-09-13T11:24:33.694 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.6100000000000003e-05 2024-09-13T11:24:33.695 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:33.695 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:33.695 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:33.695 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:33.077132+0000", 2024-09-13T11:24:33.695 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.12358451600000001 2024-09-13T11:24:33.695 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:33.695 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:33.695 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:33.695 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:33.077136+0000", 2024-09-13T11:24:33.695 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.1459999999999998e-06 2024-09-13T11:24:33.695 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:33.695 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:33.695 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:33.695 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:33.078483+0000", 2024-09-13T11:24:33.696 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0013465079999999999 2024-09-13T11:24:33.696 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:33.696 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:33.696 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:24:33.696 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:33.078505+0000", 2024-09-13T11:24:33.696 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.2829000000000001e-05 2024-09-13T11:24:33.696 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:33.696 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:33.696 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:24:33.696 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:33.087503+0000", 2024-09-13T11:24:33.696 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0089972049999999994 2024-09-13T11:24:33.696 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:33.696 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:33.697 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:24:33.697 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:33.087556+0000", 2024-09-13T11:24:33.697 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.3371000000000001e-05 2024-09-13T11:24:33.697 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:33.697 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:33.697 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:33.697 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:33.697 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:33.697 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-09-13T11:24:33.699 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-13T11:24:33.695+0000 7f6a68775640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26307) UID: 0 2024-09-13T11:24:33.699 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-09-13T11:24:33.780 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T11:24:33.800 INFO:tasks.ceph.osd.9.smithi092.stderr:2024-09-13T11:24:33.794+0000 7ff5ab3d0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26348) UID: 0 2024-09-13T11:24:33.859 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-09-13T11:24:33.859 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [ 2024-09-13T11:24:33.859 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:33.859 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:115288 3.13 3:cc9f5000:::benchmark_data_smithi177_31116_object115287:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e125)", 2024-09-13T11:24:33.859 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:33.857670+0000", 2024-09-13T11:24:33.859 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 0.00092448599999999997, 2024-09-13T11:24:33.859 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00095461500000000002, 2024-09-13T11:24:33.860 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:33.860 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:33.860 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "waiting for sub ops", 2024-09-13T11:24:33.860 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:24:33.860 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:24:33.860 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:33.860 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 115288 2024-09-13T11:24:33.860 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:33.860 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:33.860 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:33.860 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:33.860 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:33.857670+0000", 2024-09-13T11:24:33.860 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:33.860 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:33.861 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:33.861 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:33.861 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:33.857670+0000", 2024-09-13T11:24:33.861 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:33.861 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:33.861 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:33.861 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:33.861 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:33.857673+0000", 2024-09-13T11:24:33.861 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.4210000000000002e-06 2024-09-13T11:24:33.861 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:33.861 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:33.861 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:33.861 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:33.857698+0000", 2024-09-13T11:24:33.861 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.5052999999999999e-05 2024-09-13T11:24:33.862 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:33.862 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:33.862 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:33.862 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:33.857714+0000", 2024-09-13T11:24:33.862 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.6002e-05 2024-09-13T11:24:33.862 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:33.862 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:33.862 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:33.862 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:33.857723+0000", 2024-09-13T11:24:33.862 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.8729999999999996e-06 2024-09-13T11:24:33.862 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:33.862 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:33.862 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:33.863 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:33.857787+0000", 2024-09-13T11:24:33.863 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.4209999999999997e-05 2024-09-13T11:24:33.863 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:33.863 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:33.863 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:24:33.863 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:33.857931+0000", 2024-09-13T11:24:33.863 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000144542 2024-09-13T11:24:33.863 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:33.863 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:33.864 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 7", 2024-09-13T11:24:33.864 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:33.858080+0000", 2024-09-13T11:24:33.864 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000148271 2024-09-13T11:24:33.864 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:33.864 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:33.864 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:33.864 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:33.864 INFO:teuthology.orchestra.run.smithi177.stdout: ], 2024-09-13T11:24:33.864 INFO:teuthology.orchestra.run.smithi177.stdout: "num_ops": 1 2024-09-13T11:24:33.864 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-09-13T11:24:33.882 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-09-13T11:24:33.901 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-09-13T11:24:33.896+0000 7f97befc9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26315) UID: 0 2024-09-13T11:24:33.956 INFO:teuthology.orchestra.run.smithi001.stdout:osd.0: {} 2024-09-13T11:24:33.957 INFO:teuthology.orchestra.run.smithi001.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T11:24:33.985 INFO:teuthology.orchestra.run.smithi001.stdout:osd.1: {} 2024-09-13T11:24:33.988 INFO:teuthology.orchestra.run.smithi001.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T11:24:34.003 INFO:tasks.ceph.osd.5.smithi092.stderr:2024-09-13T11:24:33.998+0000 7f6f0b705640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26346) UID: 0 2024-09-13T11:24:34.010 INFO:tasks.radosbench.radosbench.0.smithi177.stdout: 127 16 115463 115447 56.8088 59.875 0.00332073 0.0175959 2024-09-13T11:24:34.011 INFO:teuthology.orchestra.run.smithi001.stdout:osd.2: {} 2024-09-13T11:24:34.011 INFO:teuthology.orchestra.run.smithi001.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T11:24:34.037 INFO:teuthology.orchestra.run.smithi001.stdout:osd.3: {} 2024-09-13T11:24:34.038 INFO:teuthology.orchestra.run.smithi001.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T11:24:34.046 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-09-13T11:24:34.046 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [], 2024-09-13T11:24:34.046 INFO:teuthology.orchestra.run.smithi177.stdout: "complaint_time": 30, 2024-09-13T11:24:34.046 INFO:teuthology.orchestra.run.smithi177.stdout: "num_blocked_ops": 0 2024-09-13T11:24:34.046 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-09-13T11:24:34.067 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-09-13T11:24:34.068 INFO:teuthology.orchestra.run.smithi001.stdout:osd.4: {} 2024-09-13T11:24:34.068 INFO:teuthology.orchestra.run.smithi001.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T11:24:34.097 INFO:teuthology.orchestra.run.smithi001.stdout:osd.5: {} 2024-09-13T11:24:34.097 INFO:teuthology.orchestra.run.smithi001.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T11:24:34.103 INFO:tasks.ceph.osd.10.smithi162.stderr:2024-09-13T11:24:34.100+0000 7fb59c9f0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26317) UID: 0 2024-09-13T11:24:34.127 INFO:teuthology.orchestra.run.smithi001.stdout:osd.6: {} 2024-09-13T11:24:34.127 INFO:teuthology.orchestra.run.smithi001.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T11:24:34.157 INFO:teuthology.orchestra.run.smithi001.stdout:osd.7: {} 2024-09-13T11:24:34.158 INFO:teuthology.orchestra.run.smithi001.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T11:24:34.183 INFO:teuthology.orchestra.run.smithi001.stdout:osd.8: {} 2024-09-13T11:24:34.183 INFO:teuthology.orchestra.run.smithi001.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-13T11:24:34.204 INFO:tasks.ceph.osd.10.smithi162.stderr:2024-09-13T11:24:34.200+0000 7fb59c9f0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26317) UID: 0 2024-09-13T11:24:34.209 INFO:teuthology.orchestra.run.smithi001.stdout:osd.9: {} 2024-09-13T11:24:34.209 INFO:teuthology.orchestra.run.smithi001.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-13T11:24:34.232 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-09-13T11:24:34.232 INFO:teuthology.orchestra.run.smithi177.stdout: "size": 20, 2024-09-13T11:24:34.232 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 600, 2024-09-13T11:24:34.232 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [ 2024-09-13T11:24:34.232 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:34.232 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:79 3.1 e32/30 3:841df5b5:::benchmark_data_smithi177_31116_object78:head v 32'6, mlcod=32'4)", 2024-09-13T11:24:34.233 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:22:27.057292+0000", 2024-09-13T11:24:34.233 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 127.172475721, 2024-09-13T11:24:34.233 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.89296786800000005, 2024-09-13T11:24:34.233 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:34.233 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:34.233 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:34.233 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:34.233 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:34.233 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:34.233 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.057292+0000", 2024-09-13T11:24:34.233 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:34.233 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:34.233 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:34.233 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:34.234 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.057292+0000", 2024-09-13T11:24:34.234 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:34.234 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:34.234 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:34.234 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:34.234 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.057293+0000", 2024-09-13T11:24:34.234 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.6109999999999999e-06 2024-09-13T11:24:34.234 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:34.234 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:34.234 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:34.234 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.057330+0000", 2024-09-13T11:24:34.234 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.6721999999999998e-05 2024-09-13T11:24:34.234 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:34.234 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:34.235 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:34.235 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.948197+0000", 2024-09-13T11:24:34.235 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.890867042 2024-09-13T11:24:34.235 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:34.235 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:34.235 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:34.235 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.948213+0000", 2024-09-13T11:24:34.235 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5826000000000001e-05 2024-09-13T11:24:34.235 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:34.235 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:34.235 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:34.235 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.948377+0000", 2024-09-13T11:24:34.235 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00016348600000000001 2024-09-13T11:24:34.235 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:34.236 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:34.236 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:24:34.236 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.948447+0000", 2024-09-13T11:24:34.236 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.0891999999999998e-05 2024-09-13T11:24:34.236 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:34.236 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:34.236 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:24:34.236 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.950150+0000", 2024-09-13T11:24:34.236 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001702329 2024-09-13T11:24:34.236 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:34.236 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:34.236 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:24:34.236 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.950260+0000", 2024-09-13T11:24:34.237 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00010996 2024-09-13T11:24:34.237 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:34.237 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:34.237 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:34.237 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:34.237 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:34.237 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:30828 3.f 3:f3a9c6a9:::benchmark_data_smithi177_31116_object30827:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e59)", 2024-09-13T11:24:34.237 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:09.503661+0000", 2024-09-13T11:24:34.237 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 84.726106372000004, 2024-09-13T11:24:34.237 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.26477667100000002, 2024-09-13T11:24:34.238 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:34.238 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:34.238 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:34.238 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:24:34.238 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:24:34.238 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:34.238 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 30828 2024-09-13T11:24:34.238 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:34.238 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:34.238 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:34.238 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:34.239 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.503661+0000", 2024-09-13T11:24:34.239 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:34.239 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:34.239 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:34.239 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:34.239 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.503661+0000", 2024-09-13T11:24:34.239 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:34.239 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:34.239 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:34.239 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:34.239 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.503663+0000", 2024-09-13T11:24:34.239 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.968e-06 2024-09-13T11:24:34.240 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:34.240 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:34.240 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:34.240 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.503687+0000", 2024-09-13T11:24:34.240 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.3983999999999999e-05 2024-09-13T11:24:34.240 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:34.240 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:34.240 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:34.240 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.503701+0000", 2024-09-13T11:24:34.240 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3416e-05 2024-09-13T11:24:34.240 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:34.240 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:34.240 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:34.241 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.503708+0000", 2024-09-13T11:24:34.241 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.9530000000000002e-06 2024-09-13T11:24:34.241 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:34.241 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:34.241 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:34.241 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.503759+0000", 2024-09-13T11:24:34.241 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.1075999999999997e-05 2024-09-13T11:24:34.241 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:34.241 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:34.241 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:24:34.242 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.503861+0000", 2024-09-13T11:24:34.242 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000102372 2024-09-13T11:24:34.242 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:34.242 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:34.242 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 12", 2024-09-13T11:24:34.242 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.503952+0000", 2024-09-13T11:24:34.242 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.1135000000000001e-05 2024-09-13T11:24:34.242 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:34.242 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:34.242 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:24:34.242 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.505340+0000", 2024-09-13T11:24:34.242 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0013877169999999999 2024-09-13T11:24:34.242 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:34.243 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:34.243 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:34.243 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.768281+0000", 2024-09-13T11:24:34.243 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.26294092400000002 2024-09-13T11:24:34.243 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:34.243 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:34.243 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:24:34.243 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.768369+0000", 2024-09-13T11:24:34.243 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.7818999999999994e-05 2024-09-13T11:24:34.243 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:34.243 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:34.243 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:24:34.243 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.768438+0000", 2024-09-13T11:24:34.243 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.9307000000000006e-05 2024-09-13T11:24:34.244 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:34.244 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:34.244 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:34.244 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:34.244 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:34.244 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:45783 3.16 3.17c10a76 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e69)", 2024-09-13T11:24:34.244 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:20.947774+0000", 2024-09-13T11:24:34.244 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 73.281993470000003, 2024-09-13T11:24:34.244 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.96682922400000004, 2024-09-13T11:24:34.244 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:34.244 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:34.244 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-13T11:24:34.244 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:24:34.245 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:24:34.245 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:34.245 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 45783 2024-09-13T11:24:34.245 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:34.245 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:34.245 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:34.245 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:34.245 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:20.947774+0000", 2024-09-13T11:24:34.245 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:34.245 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:34.245 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:34.245 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:34.245 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:20.947774+0000", 2024-09-13T11:24:34.245 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:34.245 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:34.246 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:34.246 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:34.246 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:20.947777+0000", 2024-09-13T11:24:34.246 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.57e-06 2024-09-13T11:24:34.246 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:34.246 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:34.246 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:34.246 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:20.947802+0000", 2024-09-13T11:24:34.246 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.5528999999999999e-05 2024-09-13T11:24:34.246 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:34.246 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:34.246 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:34.246 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:20.947813+0000", 2024-09-13T11:24:34.246 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0473e-05 2024-09-13T11:24:34.247 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:34.247 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:34.247 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:34.247 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:20.947821+0000", 2024-09-13T11:24:34.247 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.3440000000000001e-06 2024-09-13T11:24:34.247 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:34.247 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:34.247 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:34.247 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:20.947933+0000", 2024-09-13T11:24:34.247 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00011210200000000001 2024-09-13T11:24:34.247 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:34.247 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:34.247 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for peered", 2024-09-13T11:24:34.248 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:20.947942+0000", 2024-09-13T11:24:34.248 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.6610000000000006e-06 2024-09-13T11:24:34.248 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:34.248 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:34.248 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:34.248 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:21.914579+0000", 2024-09-13T11:24:34.248 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.96663672300000003 2024-09-13T11:24:34.248 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:34.248 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:34.248 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:24:34.248 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:21.914603+0000", 2024-09-13T11:24:34.248 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.4822000000000001e-05 2024-09-13T11:24:34.248 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:34.248 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:34.249 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:34.249 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:34.249 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:34.249 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:54625 3.1f 3:f812cdfb:::benchmark_data_smithi177_31116_object54624:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e75)", 2024-09-13T11:24:34.249 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:30.979327+0000", 2024-09-13T11:24:34.249 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 63.250440681000001, 2024-09-13T11:24:34.249 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.96163670899999998, 2024-09-13T11:24:34.249 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:34.249 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:34.249 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-13T11:24:34.249 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:24:34.249 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:24:34.249 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:34.249 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 54625 2024-09-13T11:24:34.250 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:34.250 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:34.250 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:34.250 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:34.250 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.979327+0000", 2024-09-13T11:24:34.250 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:34.250 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:34.250 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:34.250 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:34.250 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.979327+0000", 2024-09-13T11:24:34.250 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:34.250 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:34.250 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:34.250 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:34.251 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.979329+0000", 2024-09-13T11:24:34.251 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.686e-06 2024-09-13T11:24:34.251 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:34.251 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:34.251 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:34.251 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.979364+0000", 2024-09-13T11:24:34.251 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.5750000000000002e-05 2024-09-13T11:24:34.251 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:34.251 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:34.251 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:34.251 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.979408+0000", 2024-09-13T11:24:34.251 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.3795999999999999e-05 2024-09-13T11:24:34.251 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:34.251 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:34.251 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:34.252 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.979427+0000", 2024-09-13T11:24:34.252 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.9040999999999999e-05 2024-09-13T11:24:34.252 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:34.252 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:34.252 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:34.252 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.122291+0000", 2024-09-13T11:24:34.252 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.14286373599999999 2024-09-13T11:24:34.252 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:34.252 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:34.252 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for peered", 2024-09-13T11:24:34.252 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.122298+0000", 2024-09-13T11:24:34.252 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.3719999999999999e-06 2024-09-13T11:24:34.252 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:34.252 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:34.253 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:34.253 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.940341+0000", 2024-09-13T11:24:34.253 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.81804249699999998 2024-09-13T11:24:34.253 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:34.253 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:34.253 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:24:34.253 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.940964+0000", 2024-09-13T11:24:34.253 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00062283100000000001 2024-09-13T11:24:34.253 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:34.253 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:34.253 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:34.253 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:34.253 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:34.254 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:54649 3.1c 3:3a457a93:::benchmark_data_smithi177_31116_object54648:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e75)", 2024-09-13T11:24:34.254 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:30.979460+0000", 2024-09-13T11:24:34.254 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 63.250307491000001, 2024-09-13T11:24:34.254 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.95579424000000002, 2024-09-13T11:24:34.254 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:34.254 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:34.254 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-13T11:24:34.254 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:24:34.254 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:24:34.254 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:34.254 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 54649 2024-09-13T11:24:34.254 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:34.254 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:34.254 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:34.255 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:34.255 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.979460+0000", 2024-09-13T11:24:34.255 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:34.255 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:34.255 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:34.255 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:34.255 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.979460+0000", 2024-09-13T11:24:34.255 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:34.255 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:34.255 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:34.255 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:34.255 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.979462+0000", 2024-09-13T11:24:34.255 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5799999999999999e-06 2024-09-13T11:24:34.255 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:34.256 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:34.256 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:34.256 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.979524+0000", 2024-09-13T11:24:34.256 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.2521999999999995e-05 2024-09-13T11:24:34.256 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:34.256 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:34.256 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:34.256 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.979561+0000", 2024-09-13T11:24:34.256 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.6349e-05 2024-09-13T11:24:34.256 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:34.256 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:34.256 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:34.256 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.979571+0000", 2024-09-13T11:24:34.257 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0457e-05 2024-09-13T11:24:34.257 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:34.257 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:34.257 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:34.257 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.121029+0000", 2024-09-13T11:24:34.257 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.14145766200000001 2024-09-13T11:24:34.257 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:34.257 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:34.257 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for peered", 2024-09-13T11:24:34.257 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.121039+0000", 2024-09-13T11:24:34.257 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0084e-05 2024-09-13T11:24:34.257 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:34.257 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:34.257 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:34.258 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.935157+0000", 2024-09-13T11:24:34.258 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.81411807899999999 2024-09-13T11:24:34.258 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:34.258 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:34.258 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:24:34.258 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.935254+0000", 2024-09-13T11:24:34.258 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.7507e-05 2024-09-13T11:24:34.258 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:34.258 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:34.258 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:34.258 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:34.258 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:34.258 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:54654 3.1c 3:3f0bb31a:::benchmark_data_smithi177_31116_object54653:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e75)", 2024-09-13T11:24:34.258 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:30.979665+0000", 2024-09-13T11:24:34.259 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 63.250102325999997, 2024-09-13T11:24:34.259 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.95623908099999999, 2024-09-13T11:24:34.259 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:34.259 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:34.259 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-13T11:24:34.259 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:24:34.259 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:24:34.259 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:34.259 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 54654 2024-09-13T11:24:34.259 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:34.259 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:34.259 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:34.259 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:34.259 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.979665+0000", 2024-09-13T11:24:34.260 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:34.260 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:34.260 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:34.260 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:34.260 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.979665+0000", 2024-09-13T11:24:34.260 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:34.260 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:34.260 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:34.260 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:34.260 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.979667+0000", 2024-09-13T11:24:34.260 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3540000000000001e-06 2024-09-13T11:24:34.260 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:34.260 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:34.260 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:34.261 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.979703+0000", 2024-09-13T11:24:34.261 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.6758000000000003e-05 2024-09-13T11:24:34.261 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:34.261 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:34.261 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:34.261 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.979714+0000", 2024-09-13T11:24:34.261 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0804e-05 2024-09-13T11:24:34.261 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:34.261 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:34.261 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:34.261 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.979724+0000", 2024-09-13T11:24:34.261 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0040999999999999e-05 2024-09-13T11:24:34.261 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:34.262 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:34.262 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:34.262 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.121061+0000", 2024-09-13T11:24:34.262 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.141336713 2024-09-13T11:24:34.262 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:34.262 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:34.262 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for peered", 2024-09-13T11:24:34.262 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.121067+0000", 2024-09-13T11:24:34.262 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.4150000000000004e-06 2024-09-13T11:24:34.262 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:34.262 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:34.262 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:34.262 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.935776+0000", 2024-09-13T11:24:34.262 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.81470863100000002 2024-09-13T11:24:34.263 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:34.263 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:34.263 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:24:34.263 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.935904+0000", 2024-09-13T11:24:34.263 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00012836499999999999 2024-09-13T11:24:34.263 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:34.263 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:34.263 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:34.263 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:34.263 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:34.263 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPushReply(3.11 75/70 [PushReplyOp(3:8df10771:::benchmark_data_smithi177_31116_object24734:head)])", 2024-09-13T11:24:34.263 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:31.149798+0000", 2024-09-13T11:24:34.263 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 63.079969316000003, 2024-09-13T11:24:34.263 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.32402702500000002, 2024-09-13T11:24:34.264 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:34.264 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:34.264 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-13T11:24:34.264 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:34.264 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:34.264 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:34.264 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.149798+0000", 2024-09-13T11:24:34.264 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:34.264 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:34.264 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:34.264 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:34.264 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.149798+0000", 2024-09-13T11:24:34.265 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:34.265 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:34.265 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:34.265 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:34.265 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.149799+0000", 2024-09-13T11:24:34.265 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.5700000000000001e-07 2024-09-13T11:24:34.265 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:34.265 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:34.265 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:34.266 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.149803+0000", 2024-09-13T11:24:34.266 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.0040000000000001e-06 2024-09-13T11:24:34.266 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:34.266 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:34.266 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:34.266 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.472172+0000", 2024-09-13T11:24:34.266 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.32236856699999999 2024-09-13T11:24:34.266 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:34.266 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:34.266 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:34.266 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.472187+0000", 2024-09-13T11:24:34.266 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5016e-05 2024-09-13T11:24:34.266 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:34.267 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:34.267 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:34.267 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.472302+0000", 2024-09-13T11:24:34.267 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000115121 2024-09-13T11:24:34.267 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:34.267 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:34.267 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:24:34.267 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.473825+0000", 2024-09-13T11:24:34.267 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0015234599999999999 2024-09-13T11:24:34.267 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:34.267 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:34.267 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:34.267 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:34.268 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:34.268 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:54668 3.1c 3:3d2c5e40:::benchmark_data_smithi177_31116_object54667:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e75)", 2024-09-13T11:24:34.268 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:31.245525+0000", 2024-09-13T11:24:34.268 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 62.984243059000001, 2024-09-13T11:24:34.268 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.69828973000000005, 2024-09-13T11:24:34.268 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:34.268 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:34.268 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:34.268 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:24:34.268 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:24:34.268 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:34.268 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 54668 2024-09-13T11:24:34.268 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:34.268 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:34.269 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:34.269 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:34.269 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.245525+0000", 2024-09-13T11:24:34.269 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:34.269 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:34.269 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:34.269 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:34.269 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.245525+0000", 2024-09-13T11:24:34.269 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:34.269 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:34.269 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:34.269 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:34.269 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.245526+0000", 2024-09-13T11:24:34.269 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.719e-06 2024-09-13T11:24:34.270 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:34.270 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:34.270 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:34.270 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.245547+0000", 2024-09-13T11:24:34.270 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.1084999999999999e-05 2024-09-13T11:24:34.270 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:34.270 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:34.270 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:34.270 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.245558+0000", 2024-09-13T11:24:34.270 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0934000000000001e-05 2024-09-13T11:24:34.270 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:34.270 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:34.270 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:34.271 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.245564+0000", 2024-09-13T11:24:34.271 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.6269999999999997e-06 2024-09-13T11:24:34.271 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:34.271 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:34.271 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:34.271 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.245620+0000", 2024-09-13T11:24:34.271 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.6289999999999998e-05 2024-09-13T11:24:34.271 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:34.271 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:34.271 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for peered", 2024-09-13T11:24:34.271 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.245632+0000", 2024-09-13T11:24:34.271 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.1748e-05 2024-09-13T11:24:34.271 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:34.271 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:34.272 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:34.272 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.935976+0000", 2024-09-13T11:24:34.272 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.69034455299999997 2024-09-13T11:24:34.272 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:34.272 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:34.272 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:24:34.272 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.936302+0000", 2024-09-13T11:24:34.272 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000325397 2024-09-13T11:24:34.272 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:34.272 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:34.272 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 13", 2024-09-13T11:24:34.272 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.936459+0000", 2024-09-13T11:24:34.272 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00015754300000000001 2024-09-13T11:24:34.272 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:34.273 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:34.273 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:24:34.273 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.938813+0000", 2024-09-13T11:24:34.273 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0023530999999999999 2024-09-13T11:24:34.273 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:34.273 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:34.273 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:34.273 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.943695+0000", 2024-09-13T11:24:34.273 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.004882506 2024-09-13T11:24:34.273 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:34.273 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:34.273 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:24:34.273 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.943767+0000", 2024-09-13T11:24:34.274 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.2454999999999997e-05 2024-09-13T11:24:34.274 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:34.274 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:34.274 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:24:34.274 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.943814+0000", 2024-09-13T11:24:34.274 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.6773000000000001e-05 2024-09-13T11:24:34.274 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:34.274 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:34.274 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:34.274 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:34.274 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:34.274 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPush(3.16 84/70 [PushOp(3:6f7be541:::benchmark_data_smithi177_31116_object43332:head, version: 66'2788, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:6f7be541:::benchmark_data_smithi177_31116_object43332:head@66'2788, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:24:34.274 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:41.207515+0000", 2024-09-13T11:24:34.274 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 53.022252287999997, 2024-09-13T11:24:34.275 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.52643771500000003, 2024-09-13T11:24:34.275 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:34.275 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:34.275 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:24:34.275 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:34.275 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:34.275 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:34.275 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.207515+0000", 2024-09-13T11:24:34.275 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:34.275 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:34.275 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:34.275 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:34.275 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.207515+0000", 2024-09-13T11:24:34.275 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:34.276 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:34.276 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:34.276 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:34.276 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.207516+0000", 2024-09-13T11:24:34.276 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.096e-06 2024-09-13T11:24:34.276 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:34.276 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:34.276 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:34.276 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.207717+0000", 2024-09-13T11:24:34.276 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000200199 2024-09-13T11:24:34.276 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:34.276 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:34.276 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:34.276 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.732785+0000", 2024-09-13T11:24:34.277 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.52506869599999995 2024-09-13T11:24:34.277 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:34.277 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:34.277 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:34.277 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.732801+0000", 2024-09-13T11:24:34.277 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5305999999999999e-05 2024-09-13T11:24:34.277 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:34.277 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:34.277 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:34.277 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.732889+0000", 2024-09-13T11:24:34.277 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.8157000000000003e-05 2024-09-13T11:24:34.277 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:34.277 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:34.278 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:24:34.278 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.732901+0000", 2024-09-13T11:24:34.278 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.1866e-05 2024-09-13T11:24:34.278 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:34.278 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:34.278 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:24:34.278 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.733953+0000", 2024-09-13T11:24:34.278 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001052395 2024-09-13T11:24:34.278 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:34.278 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:34.278 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:34.278 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:34.278 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:34.278 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPushReply(3.1f 85/78 [PushReplyOp(3:fd221d21:::benchmark_data_smithi177_31116_object7639:head)])", 2024-09-13T11:24:34.279 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:42.198715+0000", 2024-09-13T11:24:34.279 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 52.031052719000002, 2024-09-13T11:24:34.279 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.48930391099999998, 2024-09-13T11:24:34.279 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:34.279 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:34.279 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-13T11:24:34.279 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:34.279 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:34.279 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:34.279 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.198715+0000", 2024-09-13T11:24:34.279 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:34.279 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:34.279 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:34.280 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:34.280 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.198715+0000", 2024-09-13T11:24:34.280 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:34.280 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:34.280 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:34.280 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:34.280 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.198716+0000", 2024-09-13T11:24:34.280 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.3500000000000005e-07 2024-09-13T11:24:34.280 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:34.280 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:34.280 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:34.280 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.198719+0000", 2024-09-13T11:24:34.280 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.4539999999999999e-06 2024-09-13T11:24:34.280 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:34.281 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:34.281 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:34.281 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.684732+0000", 2024-09-13T11:24:34.281 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.48601262099999998 2024-09-13T11:24:34.281 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:34.281 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:34.281 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:34.281 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.684746+0000", 2024-09-13T11:24:34.281 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.4518999999999999e-05 2024-09-13T11:24:34.281 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:34.281 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:34.281 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:34.281 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.684911+0000", 2024-09-13T11:24:34.281 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000164728 2024-09-13T11:24:34.282 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:34.282 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:34.282 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:24:34.282 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.688019+0000", 2024-09-13T11:24:34.282 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0031077539999999999 2024-09-13T11:24:34.282 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:34.282 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:34.282 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:34.282 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:34.282 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:34.282 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPushReply(3.1c 85/77 [PushReplyOp(3:3a669437:::benchmark_data_smithi177_31116_object25280:head)])", 2024-09-13T11:24:34.282 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:42.352466+0000", 2024-09-13T11:24:34.282 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 51.877301825000004, 2024-09-13T11:24:34.282 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.36670725300000001, 2024-09-13T11:24:34.283 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:34.283 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:34.283 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-13T11:24:34.283 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:34.283 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:34.283 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:34.283 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.352466+0000", 2024-09-13T11:24:34.283 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:34.283 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:34.283 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:34.283 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:34.283 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.352466+0000", 2024-09-13T11:24:34.283 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:34.283 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:34.284 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:34.284 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:34.284 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.352467+0000", 2024-09-13T11:24:34.284 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.1429999999999999e-06 2024-09-13T11:24:34.284 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:34.284 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:34.284 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:34.284 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.352474+0000", 2024-09-13T11:24:34.284 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.6100000000000002e-06 2024-09-13T11:24:34.284 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:34.284 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:34.284 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:34.284 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.717011+0000", 2024-09-13T11:24:34.285 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.36453790200000002 2024-09-13T11:24:34.285 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:34.285 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:34.285 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:34.285 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.717032+0000", 2024-09-13T11:24:34.285 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.0999e-05 2024-09-13T11:24:34.285 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:34.285 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:34.285 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:34.285 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.717162+0000", 2024-09-13T11:24:34.285 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000129159 2024-09-13T11:24:34.285 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:34.285 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:34.285 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:24:34.286 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.719173+0000", 2024-09-13T11:24:34.286 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0020114400000000002 2024-09-13T11:24:34.286 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:34.286 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:34.286 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:34.286 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:34.286 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:34.286 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPush(3.16 85/70 [PushOp(3:6aef2f27:::benchmark_data_smithi177_31116_object44090:head, version: 68'2844, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:6aef2f27:::benchmark_data_smithi177_31116_object44090:head@68'2844, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:24:34.286 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:42.574221+0000", 2024-09-13T11:24:34.286 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 51.655546717, 2024-09-13T11:24:34.286 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.25345585700000001, 2024-09-13T11:24:34.286 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:34.286 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:34.286 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:24:34.287 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:34.287 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:34.287 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:34.287 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.574221+0000", 2024-09-13T11:24:34.287 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:34.287 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:34.287 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:34.287 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:34.287 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.574221+0000", 2024-09-13T11:24:34.287 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:34.287 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:34.287 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:34.287 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:34.287 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.574252+0000", 2024-09-13T11:24:34.288 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.0973000000000003e-05 2024-09-13T11:24:34.288 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:34.288 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:34.288 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:34.288 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.574434+0000", 2024-09-13T11:24:34.288 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000182265 2024-09-13T11:24:34.288 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:34.288 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:34.288 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:34.288 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.825391+0000", 2024-09-13T11:24:34.288 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.250957023 2024-09-13T11:24:34.288 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:34.288 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:34.289 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:34.289 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.825442+0000", 2024-09-13T11:24:34.289 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.1267999999999998e-05 2024-09-13T11:24:34.289 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:34.289 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:34.289 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:34.289 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.825692+0000", 2024-09-13T11:24:34.289 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00024924700000000002 2024-09-13T11:24:34.289 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:34.289 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:34.289 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:24:34.289 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.825715+0000", 2024-09-13T11:24:34.289 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.3176e-05 2024-09-13T11:24:34.290 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:34.290 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:34.290 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:24:34.290 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.827677+0000", 2024-09-13T11:24:34.290 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0019619049999999999 2024-09-13T11:24:34.290 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:34.290 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:34.290 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:34.290 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:34.290 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:34.290 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPushReply(3.1f 85/78 [PushReplyOp(3:fecb0d6a:::benchmark_data_smithi177_31116_object7644:head)])", 2024-09-13T11:24:34.290 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:42.705351+0000", 2024-09-13T11:24:34.290 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 51.524416983000002, 2024-09-13T11:24:34.290 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.290549681, 2024-09-13T11:24:34.291 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:34.291 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:34.291 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-13T11:24:34.291 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:34.291 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:34.291 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:34.291 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.705351+0000", 2024-09-13T11:24:34.291 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:34.291 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:34.291 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:34.291 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:34.291 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.705351+0000", 2024-09-13T11:24:34.291 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:34.291 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:34.292 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:34.292 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:34.292 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.705353+0000", 2024-09-13T11:24:34.292 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.4169999999999999e-06 2024-09-13T11:24:34.292 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:34.292 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:34.292 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:34.292 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.705365+0000", 2024-09-13T11:24:34.292 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.1894e-05 2024-09-13T11:24:34.292 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:34.292 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:34.292 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:34.292 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.992081+0000", 2024-09-13T11:24:34.292 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.28671641199999998 2024-09-13T11:24:34.293 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:34.293 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:34.293 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:34.293 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.992111+0000", 2024-09-13T11:24:34.293 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.9427000000000001e-05 2024-09-13T11:24:34.293 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:34.293 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:34.293 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:34.293 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.993573+0000", 2024-09-13T11:24:34.293 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0014623590000000001 2024-09-13T11:24:34.293 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:34.293 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:34.293 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:24:34.294 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.995900+0000", 2024-09-13T11:24:34.294 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0023271720000000002 2024-09-13T11:24:34.294 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:34.294 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:34.294 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:34.294 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:34.294 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:34.294 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPushReply(3.21 89/78 [PushReplyOp(3:8534d087:::benchmark_data_smithi177_31116_object5474:head)])", 2024-09-13T11:24:34.294 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:46.632834+0000", 2024-09-13T11:24:34.294 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 47.596933542999999, 2024-09-13T11:24:34.294 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.324805556, 2024-09-13T11:24:34.294 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:34.294 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:34.294 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-13T11:24:34.295 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:34.295 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:34.295 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:34.295 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.632834+0000", 2024-09-13T11:24:34.295 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:34.295 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:34.295 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:34.295 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:34.295 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.632834+0000", 2024-09-13T11:24:34.295 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:34.295 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:34.295 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:34.295 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:34.296 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.632835+0000", 2024-09-13T11:24:34.296 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.4270000000000001e-06 2024-09-13T11:24:34.296 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:34.296 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:34.296 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:34.296 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.632840+0000", 2024-09-13T11:24:34.296 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.7809999999999996e-06 2024-09-13T11:24:34.296 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:34.296 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:34.296 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:34.296 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.956282+0000", 2024-09-13T11:24:34.296 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.323441639 2024-09-13T11:24:34.296 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:34.296 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:34.297 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:34.297 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.956293+0000", 2024-09-13T11:24:34.297 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.083e-05 2024-09-13T11:24:34.297 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:34.297 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:34.297 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:34.297 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.956395+0000", 2024-09-13T11:24:34.297 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00010184200000000001 2024-09-13T11:24:34.297 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:34.297 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:34.297 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:24:34.297 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.957640+0000", 2024-09-13T11:24:34.297 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0012450370000000001 2024-09-13T11:24:34.298 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:34.298 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:34.298 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:34.298 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:34.298 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:34.298 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPushReply(3.1f 89/78 [PushReplyOp(3:ff150976:::benchmark_data_smithi177_31116_object12907:head)])", 2024-09-13T11:24:34.298 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:46.697504+0000", 2024-09-13T11:24:34.298 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 47.532263944, 2024-09-13T11:24:34.298 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.26836818099999998, 2024-09-13T11:24:34.298 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:34.298 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:34.298 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-13T11:24:34.298 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:34.299 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:34.299 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:34.299 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.697504+0000", 2024-09-13T11:24:34.299 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:34.299 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:34.299 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:34.299 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:34.299 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.697504+0000", 2024-09-13T11:24:34.299 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:34.299 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:34.299 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:34.299 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:34.299 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.697505+0000", 2024-09-13T11:24:34.299 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0529999999999999e-06 2024-09-13T11:24:34.300 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:34.300 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:34.300 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:34.300 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.697509+0000", 2024-09-13T11:24:34.300 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.9249999999999997e-06 2024-09-13T11:24:34.300 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:34.300 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:34.300 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:34.300 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.964086+0000", 2024-09-13T11:24:34.300 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.26657776100000002 2024-09-13T11:24:34.300 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:34.300 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:34.301 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:34.301 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.964110+0000", 2024-09-13T11:24:34.301 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.3189000000000001e-05 2024-09-13T11:24:34.301 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:34.301 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:34.301 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:34.301 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.964354+0000", 2024-09-13T11:24:34.301 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00024481399999999999 2024-09-13T11:24:34.301 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:34.301 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:34.301 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:24:34.301 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.965872+0000", 2024-09-13T11:24:34.301 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001517439 2024-09-13T11:24:34.301 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:34.302 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:34.302 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:34.302 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:34.302 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:34.302 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:71589 3.21 3:85f71b95:::benchmark_data_smithi177_31116_object71588:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96)", 2024-09-13T11:24:34.302 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:53.876584+0000", 2024-09-13T11:24:34.302 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 40.353183491000003, 2024-09-13T11:24:34.302 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.24303730800000001, 2024-09-13T11:24:34.302 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:34.302 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:34.302 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:34.302 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:24:34.302 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:24:34.303 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:34.303 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 71589 2024-09-13T11:24:34.303 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:34.303 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:34.303 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:34.303 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:34.303 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:53.876584+0000", 2024-09-13T11:24:34.303 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:34.303 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:34.303 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:34.303 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:34.303 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:53.876584+0000", 2024-09-13T11:24:34.303 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:34.303 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:34.304 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:34.304 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:34.304 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:53.876586+0000", 2024-09-13T11:24:34.304 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.4100000000000001e-06 2024-09-13T11:24:34.304 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:34.304 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:34.304 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:34.304 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:53.876604+0000", 2024-09-13T11:24:34.304 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.8389e-05 2024-09-13T11:24:34.304 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:34.304 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:34.304 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:34.304 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:53.876613+0000", 2024-09-13T11:24:34.304 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.8789999999999992e-06 2024-09-13T11:24:34.305 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:34.305 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:34.305 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:34.305 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:53.876619+0000", 2024-09-13T11:24:34.305 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.9699999999999996e-06 2024-09-13T11:24:34.305 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:34.305 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:34.305 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:34.305 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:53.876681+0000", 2024-09-13T11:24:34.305 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.2699000000000004e-05 2024-09-13T11:24:34.305 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:34.305 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:34.305 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:24:34.305 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:53.876849+0000", 2024-09-13T11:24:34.306 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00016714799999999999 2024-09-13T11:24:34.306 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:34.306 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:34.306 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 0,8", 2024-09-13T11:24:34.306 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:53.877032+0000", 2024-09-13T11:24:34.306 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00018387199999999999 2024-09-13T11:24:34.306 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:34.306 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:34.306 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:24:34.306 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:53.879473+0000", 2024-09-13T11:24:34.306 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0024403789999999999 2024-09-13T11:24:34.306 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:34.306 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:34.306 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:34.307 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:53.880046+0000", 2024-09-13T11:24:34.307 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00057281200000000002 2024-09-13T11:24:34.307 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:34.307 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:34.307 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:34.307 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.119299+0000", 2024-09-13T11:24:34.307 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.23925310399999999 2024-09-13T11:24:34.307 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:34.307 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:34.307 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:24:34.307 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.119337+0000", 2024-09-13T11:24:34.307 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.8109999999999999e-05 2024-09-13T11:24:34.307 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:34.307 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:34.308 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:24:34.308 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.119621+0000", 2024-09-13T11:24:34.308 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000284536 2024-09-13T11:24:34.308 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:34.308 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:34.308 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:34.308 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:34.308 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:34.308 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:83223 3.1c 3:3bc6018d:::benchmark_data_smithi177_31116_object83222:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106)", 2024-09-13T11:24:34.308 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:04.511163+0000", 2024-09-13T11:24:34.308 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 29.718604165999999, 2024-09-13T11:24:34.308 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.47558093200000001, 2024-09-13T11:24:34.308 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:34.309 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:34.309 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:34.309 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:24:34.309 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:24:34.309 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:34.309 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 83223 2024-09-13T11:24:34.309 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:34.309 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:34.309 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:34.309 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:34.309 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.511163+0000", 2024-09-13T11:24:34.309 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:34.309 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:34.309 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:34.309 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:34.310 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.511163+0000", 2024-09-13T11:24:34.310 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:34.310 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:34.310 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:34.310 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:34.310 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.511167+0000", 2024-09-13T11:24:34.310 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.8330000000000004e-06 2024-09-13T11:24:34.310 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:34.310 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:34.310 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:34.310 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.511219+0000", 2024-09-13T11:24:34.310 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.1771999999999998e-05 2024-09-13T11:24:34.310 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:34.310 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:34.311 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:34.311 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.511252+0000", 2024-09-13T11:24:34.311 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.3315999999999997e-05 2024-09-13T11:24:34.311 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:34.311 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:34.311 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:34.311 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.511268+0000", 2024-09-13T11:24:34.311 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5812999999999999e-05 2024-09-13T11:24:34.311 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:34.311 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:34.311 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:34.311 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.511418+0000", 2024-09-13T11:24:34.311 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000150023 2024-09-13T11:24:34.311 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:34.312 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:34.312 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:24:34.312 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.511670+0000", 2024-09-13T11:24:34.312 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00025185499999999999 2024-09-13T11:24:34.312 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:34.312 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:34.312 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 7", 2024-09-13T11:24:34.312 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.511905+0000", 2024-09-13T11:24:34.312 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00023493799999999999 2024-09-13T11:24:34.312 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:34.312 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:34.312 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:24:34.312 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.514960+0000", 2024-09-13T11:24:34.312 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.003054977 2024-09-13T11:24:34.313 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:34.313 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:34.313 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:34.313 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.986432+0000", 2024-09-13T11:24:34.313 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.47147209200000001 2024-09-13T11:24:34.313 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:34.313 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:34.313 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:24:34.313 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.986635+0000", 2024-09-13T11:24:34.313 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00020309899999999999 2024-09-13T11:24:34.313 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:34.313 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:34.313 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:24:34.313 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.986744+0000", 2024-09-13T11:24:34.314 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000109214 2024-09-13T11:24:34.314 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:34.314 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:34.314 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:34.314 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:34.314 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:34.314 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop_reply(client.4665.0:84803 3.1f e107/78)", 2024-09-13T11:24:34.314 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:05.912627+0000", 2024-09-13T11:24:34.314 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 28.317140258999999, 2024-09-13T11:24:34.314 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.33726566600000002, 2024-09-13T11:24:34.314 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:34.314 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:34.314 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-13T11:24:34.314 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:34.315 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:34.315 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:34.315 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:05.912627+0000", 2024-09-13T11:24:34.315 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:34.315 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:34.315 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:34.315 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:34.315 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:05.912627+0000", 2024-09-13T11:24:34.315 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:34.315 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:34.315 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:34.315 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:34.315 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:05.912629+0000", 2024-09-13T11:24:34.316 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.4440000000000001e-06 2024-09-13T11:24:34.316 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:34.316 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:34.316 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:34.316 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:05.912634+0000", 2024-09-13T11:24:34.316 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.6869999999999999e-06 2024-09-13T11:24:34.316 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:34.316 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:34.316 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:34.316 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:06.249722+0000", 2024-09-13T11:24:34.316 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.33708766099999998 2024-09-13T11:24:34.316 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:34.316 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:34.316 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:34.317 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:06.249736+0000", 2024-09-13T11:24:34.317 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3623e-05 2024-09-13T11:24:34.317 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:34.317 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:34.317 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:34.317 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:06.249878+0000", 2024-09-13T11:24:34.317 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00014264599999999999 2024-09-13T11:24:34.317 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:34.317 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:34.317 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:24:34.317 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:06.249893+0000", 2024-09-13T11:24:34.317 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.4605e-05 2024-09-13T11:24:34.317 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:34.317 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:34.318 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:34.318 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:34.318 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:34.318 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop_reply(client.4665.0:110260 3.f e119/75 ondisk, result = 0)", 2024-09-13T11:24:34.318 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:25.333188+0000", 2024-09-13T11:24:34.318 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 8.8965796319999999, 2024-09-13T11:24:34.318 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.29195486199999998, 2024-09-13T11:24:34.318 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:34.318 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:34.318 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:24:34.318 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:34.318 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:34.318 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:34.318 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.333188+0000", 2024-09-13T11:24:34.319 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:34.319 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:34.319 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:34.319 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:34.319 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.333188+0000", 2024-09-13T11:24:34.319 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:34.319 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:34.319 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:34.319 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:34.319 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.333190+0000", 2024-09-13T11:24:34.319 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.2120000000000002e-06 2024-09-13T11:24:34.319 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:34.319 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:34.319 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:34.320 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.333197+0000", 2024-09-13T11:24:34.320 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.7469999999999999e-06 2024-09-13T11:24:34.320 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:34.320 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:34.320 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:34.320 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.624892+0000", 2024-09-13T11:24:34.320 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.291694913 2024-09-13T11:24:34.320 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:34.320 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:34.320 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:34.320 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.624907+0000", 2024-09-13T11:24:34.320 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5053e-05 2024-09-13T11:24:34.320 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:34.320 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:34.321 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:34.321 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.624999+0000", 2024-09-13T11:24:34.321 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.2603999999999996e-05 2024-09-13T11:24:34.321 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:34.321 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:34.321 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:24:34.321 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.625011+0000", 2024-09-13T11:24:34.321 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.1253999999999999e-05 2024-09-13T11:24:34.321 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:34.321 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:34.321 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:24:34.321 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.625143+0000", 2024-09-13T11:24:34.321 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00013207900000000001 2024-09-13T11:24:34.321 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:34.322 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:34.322 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:34.322 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:34.322 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:34.322 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:110265 3.1 e119/75 3:8236484d:::benchmark_data_smithi177_31116_object110264:head v 119'3935, mlcod=118'3933)", 2024-09-13T11:24:34.322 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:25.343348+0000", 2024-09-13T11:24:34.322 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 8.8864200170000007, 2024-09-13T11:24:34.322 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.60013013500000001, 2024-09-13T11:24:34.322 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:34.322 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:34.322 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:34.322 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:34.322 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:34.322 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:34.323 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.343348+0000", 2024-09-13T11:24:34.323 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:34.323 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:34.323 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:34.323 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:34.323 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.343348+0000", 2024-09-13T11:24:34.323 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:34.323 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:34.323 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:34.323 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:34.323 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.343350+0000", 2024-09-13T11:24:34.323 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.576e-06 2024-09-13T11:24:34.323 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:34.324 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:34.324 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:34.324 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.343443+0000", 2024-09-13T11:24:34.324 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.2989e-05 2024-09-13T11:24:34.324 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:34.324 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:34.324 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:34.324 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.940863+0000", 2024-09-13T11:24:34.324 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.59742003300000002 2024-09-13T11:24:34.324 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:34.324 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:34.324 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:34.324 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.940875+0000", 2024-09-13T11:24:34.324 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.2147000000000001e-05 2024-09-13T11:24:34.325 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:34.325 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:34.325 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:34.325 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.940993+0000", 2024-09-13T11:24:34.325 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000117903 2024-09-13T11:24:34.325 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:34.325 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:34.325 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:24:34.325 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.941017+0000", 2024-09-13T11:24:34.325 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.3442999999999999e-05 2024-09-13T11:24:34.325 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:34.325 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:34.325 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:24:34.325 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.943347+0000", 2024-09-13T11:24:34.326 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0023299869999999999 2024-09-13T11:24:34.326 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:34.326 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:34.326 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:24:34.326 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.943478+0000", 2024-09-13T11:24:34.326 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00013105699999999999 2024-09-13T11:24:34.326 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:34.326 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:34.326 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:34.326 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:34.326 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:34.326 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-09-13T11:24:34.327 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-13T11:24:34.328 INFO:teuthology.orchestra.run.smithi001.stdout:osd.10: {} 2024-09-13T11:24:34.328 INFO:teuthology.orchestra.run.smithi001.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-13T11:24:34.329 INFO:teuthology.orchestra.run.smithi001.stdout:osd.11: {} 2024-09-13T11:24:34.329 INFO:teuthology.orchestra.run.smithi001.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-13T11:24:34.330 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_ops_in_flight 2024-09-13T11:24:34.330 INFO:tasks.ceph.osd.4.smithi001.stderr:2024-09-13T11:24:34.283+0000 7ff8d0c3d640 -1 osd.4 125 handle_osd_map injecting map crc failure 2024-09-13T11:24:34.331 INFO:tasks.ceph.osd.15.smithi177.stderr:2024-09-13T11:24:34.275+0000 7f9d618a4640 -1 osd.15 125 handle_osd_map injecting map crc failure 2024-09-13T11:24:34.331 INFO:tasks.ceph.osd.15.smithi177.stderr:2024-09-13T11:24:34.327+0000 7f9d6ce77640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26578) UID: 0 2024-09-13T11:24:34.331 INFO:teuthology.orchestra.run.smithi001.stdout:osd.12: {} 2024-09-13T11:24:34.331 INFO:teuthology.orchestra.run.smithi001.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-13T11:24:34.332 INFO:teuthology.orchestra.run.smithi001.stdout:osd.13: {} 2024-09-13T11:24:34.332 INFO:teuthology.orchestra.run.smithi001.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-13T11:24:34.332 INFO:teuthology.orchestra.run.smithi001.stderr:Error ENXIO: problem getting command descriptions from osd.14 2024-09-13T11:24:34.363 INFO:teuthology.orchestra.run.smithi001.stdout:osd.15: {} 2024-09-13T11:24:34.363 INFO:teuthology.orchestra.run.smithi001.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-13T11:24:34.367 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-13T11:24:34.368 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-13T11:24:34.430 INFO:tasks.ceph.osd.9.smithi092.stderr:2024-09-13T11:24:34.426+0000 7ff5ab3d0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26348) UID: 0 2024-09-13T11:24:34.530 INFO:tasks.ceph.osd.7.smithi177.stderr:2024-09-13T11:24:34.527+0000 7f897b51d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26575) UID: 0 2024-09-13T11:24:34.542 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-09-13T11:24:34.542 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:24:34.568 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_blocked_ops 2024-09-13T11:24:34.656 INFO:tasks.ceph.osd.15.smithi177.stderr:2024-09-13T11:24:34.655+0000 7f9d6ce77640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26578) UID: 0 2024-09-13T11:24:34.732 INFO:tasks.ceph.osd.3.smithi177.stderr:2024-09-13T11:24:34.731+0000 7f10adba4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26574) UID: 0 2024-09-13T11:24:34.803 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-09-13T11:24:34.803 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:24:34.829 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_historic_ops 2024-09-13T11:24:34.872 INFO:tasks.ceph.osd.11.smithi177.stderr:2024-09-13T11:24:34.871+0000 7f24792c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26580) UID: 0 2024-09-13T11:24:34.934 INFO:tasks.ceph.osd.11.smithi177.stderr:2024-09-13T11:24:34.931+0000 7f24792c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26580) UID: 0 2024-09-13T11:24:35.011 INFO:tasks.radosbench.radosbench.0.smithi177.stdout: 128 16 116750 116734 56.9927 80.4375 0.00408388 0.0175406 2024-09-13T11:24:35.035 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-13T11:24:35.031+0000 7f6a68775640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26307) UID: 0 2024-09-13T11:24:35.070 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-09-13T11:24:35.070 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:24:35.096 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-13T11:24:35.145 INFO:tasks.ceph.osd.13.smithi092.stderr:2024-09-13T11:24:35.142+0000 7f3e5afea640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26350) UID: 0 2024-09-13T11:24:35.237 INFO:tasks.ceph.osd.9.smithi092.stderr:2024-09-13T11:24:35.230+0000 7ff5ab3d0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26348) UID: 0 2024-09-13T11:24:35.245 INFO:teuthology.orchestra.run.smithi092.stdout:ERROR: (22) Invalid argument 2024-09-13T11:24:35.245 INFO:teuthology.orchestra.run.smithi092.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:24:35.269 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-13T11:24:35.272 INFO:teuthology.orchestra.run.smithi001.stderr:nodeep-scrub is unset 2024-09-13T11:24:35.284 INFO:tasks.ceph.osd.7.smithi177.stderr:2024-09-13T11:24:35.279+0000 7f8971f4e640 -1 osd.7 126 handle_osd_map injecting map crc failure 2024-09-13T11:24:35.284 INFO:tasks.ceph.osd.7.smithi177.stderr:2024-09-13T11:24:35.283+0000 7f896ff4a640 -1 osd.7 126 handle_osd_map injecting map crc failure 2024-09-13T11:24:35.291 INFO:tasks.ceph.osd.4.smithi001.stderr:2024-09-13T11:24:35.287+0000 7ff8d2c41640 -1 osd.4 126 handle_osd_map injecting map crc failure 2024-09-13T11:24:35.353 INFO:tasks.ceph.osd.5.smithi092.stderr:2024-09-13T11:24:35.350+0000 7f6f0b705640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26346) UID: 0 2024-09-13T11:24:35.441 INFO:tasks.ceph.osd.1.smithi092.stderr:2024-09-13T11:24:35.434+0000 7fe7789c5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33718) UID: 0 2024-09-13T11:24:35.478 INFO:teuthology.orchestra.run.smithi092.stdout:ERROR: (22) Invalid argument 2024-09-13T11:24:35.478 INFO:teuthology.orchestra.run.smithi092.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:24:35.505 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-13T11:24:35.541 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-13T11:24:35.539+0000 7f6a68775640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26307) UID: 0 2024-09-13T11:24:35.642 INFO:tasks.ceph.osd.4.smithi001.stderr:2024-09-13T11:24:35.639+0000 7ff8e0218640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26310) UID: 0 2024-09-13T11:24:35.708 INFO:teuthology.orchestra.run.smithi092.stdout:ERROR: (22) Invalid argument 2024-09-13T11:24:35.708 INFO:teuthology.orchestra.run.smithi092.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:24:35.734 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-13T11:24:35.743 INFO:tasks.ceph.osd.13.smithi092.stderr:2024-09-13T11:24:35.738+0000 7f3e5afea640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26350) UID: 0 2024-09-13T11:24:35.844 INFO:tasks.ceph.osd.2.smithi162.stderr:2024-09-13T11:24:35.840+0000 7fe92b35a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26313) UID: 0 2024-09-13T11:24:35.884 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-13T11:24:35.884 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:24:35.904 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-13T11:24:35.946 INFO:tasks.ceph.osd.13.smithi092.stderr:2024-09-13T11:24:35.942+0000 7f3e5afea640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26350) UID: 0 2024-09-13T11:24:36.011 INFO:tasks.radosbench.radosbench.0.smithi177.stdout: 129 16 117892 117876 57.1042 71.375 0.00692189 0.0175068 2024-09-13T11:24:36.032 INFO:teuthology.orchestra.run.smithi162.stdout: marking collection for removal 2024-09-13T11:24:36.032 INFO:teuthology.orchestra.run.smithi162.stdout:setting '_remove' omap key 2024-09-13T11:24:36.032 INFO:teuthology.orchestra.run.smithi162.stdout:finish_remove_pgs 3.2_head removing 3.2 2024-09-13T11:24:36.047 INFO:tasks.ceph.osd.13.smithi092.stderr:2024-09-13T11:24:36.042+0000 7f3e5afea640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26350) UID: 0 2024-09-13T11:24:36.052 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-13T11:24:36.053 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:24:36.070 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-13T11:24:36.144 INFO:teuthology.orchestra.run.smithi162.stdout:Remove successful 2024-09-13T11:24:36.148 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-13T11:24:36.147+0000 7f6a68775640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26307) UID: 0 2024-09-13T11:24:36.217 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-13T11:24:36.217 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:24:36.241 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-13T11:24:36.280 INFO:tasks.ceph.osd.12.smithi001.stderr:2024-09-13T11:24:36.279+0000 7f892c6e6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26314) UID: 0 2024-09-13T11:24:36.288 INFO:tasks.ceph.osd.12.smithi001.stderr:2024-09-13T11:24:36.283+0000 7f891d10b640 -1 osd.12 127 handle_osd_map injecting map crc failure 2024-09-13T11:24:36.350 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-13T11:24:36.347+0000 7f6a68775640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26307) UID: 0 2024-09-13T11:24:36.369 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T11:24:36.444 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-13T11:24:36.444 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:24:36.451 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-09-13T11:24:36.448+0000 7f97befc9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26315) UID: 0 2024-09-13T11:24:36.467 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-13T11:24:36.552 INFO:tasks.ceph.osd.10.smithi162.stderr:2024-09-13T11:24:36.548+0000 7fb59c9f0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26317) UID: 0 2024-09-13T11:24:36.584 INFO:teuthology.orchestra.run.smithi001.stdout:osd.0: {} 2024-09-13T11:24:36.584 INFO:teuthology.orchestra.run.smithi001.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T11:24:36.588 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-objectstore-tool --err-to-stderr --data-path /var/lib/ceph/osd/ceph-14 --journal-path /var/lib/ceph/osd/ceph-14/journal '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --op import --file /var/log/ceph/exp.3.2.14 2024-09-13T11:24:36.615 INFO:teuthology.orchestra.run.smithi001.stdout:osd.1: {} 2024-09-13T11:24:36.615 INFO:teuthology.orchestra.run.smithi001.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T11:24:36.629 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-13T11:24:36.630 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:24:36.653 INFO:teuthology.orchestra.run.smithi001.stdout:osd.2: {} 2024-09-13T11:24:36.653 INFO:teuthology.orchestra.run.smithi001.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T11:24:36.655 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-13T11:24:36.678 INFO:tasks.ceph.osd.10.smithi162.stderr:2024-09-13T11:24:36.676+0000 7fb59c9f0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26317) UID: 0 2024-09-13T11:24:36.681 INFO:teuthology.orchestra.run.smithi001.stdout:osd.3: {} 2024-09-13T11:24:36.681 INFO:teuthology.orchestra.run.smithi001.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T11:24:36.714 INFO:teuthology.orchestra.run.smithi001.stdout:osd.4: {} 2024-09-13T11:24:36.715 INFO:teuthology.orchestra.run.smithi001.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T11:24:36.743 INFO:teuthology.orchestra.run.smithi001.stdout:osd.5: {} 2024-09-13T11:24:36.743 INFO:teuthology.orchestra.run.smithi001.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T11:24:36.754 INFO:tasks.ceph.osd.3.smithi177.stderr:2024-09-13T11:24:36.751+0000 7f10adba4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26574) UID: 0 2024-09-13T11:24:36.774 INFO:teuthology.orchestra.run.smithi001.stdout:osd.6: {} 2024-09-13T11:24:36.774 INFO:teuthology.orchestra.run.smithi001.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T11:24:36.804 INFO:teuthology.orchestra.run.smithi001.stdout:osd.7: {} 2024-09-13T11:24:36.804 INFO:teuthology.orchestra.run.smithi001.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T11:24:36.818 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-13T11:24:36.818 INFO:teuthology.orchestra.run.smithi001.stdout: "size": 20, 2024-09-13T11:24:36.818 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 600, 2024-09-13T11:24:36.818 INFO:teuthology.orchestra.run.smithi001.stdout: "ops": [ 2024-09-13T11:24:36.818 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:36.819 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDPGPush(3.11 72/70 [PushOp(3:8ae6b6fc:::benchmark_data_smithi177_31116_object7617:head, version: 46'455, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:8ae6b6fc:::benchmark_data_smithi177_31116_object7617:head@46'455, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:24:36.819 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:26.176296+0000", 2024-09-13T11:24:36.819 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 70.639967874999996, 2024-09-13T11:24:36.819 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.13349134900000001, 2024-09-13T11:24:36.819 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:24:36.819 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:24:36.819 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:24:36.819 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:24:36.819 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:36.819 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:24:36.819 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:26.176296+0000", 2024-09-13T11:24:36.819 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:36.819 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:36.820 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:36.820 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:24:36.820 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:26.176296+0000", 2024-09-13T11:24:36.820 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:36.820 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:36.820 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:36.820 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:24:36.820 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:26.176297+0000", 2024-09-13T11:24:36.820 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2440000000000001e-06 2024-09-13T11:24:36.820 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:36.820 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:36.821 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:24:36.821 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:26.176433+0000", 2024-09-13T11:24:36.821 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00013554999999999999 2024-09-13T11:24:36.821 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:36.821 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:36.821 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:24:36.821 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:26.308747+0000", 2024-09-13T11:24:36.821 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.132314392 2024-09-13T11:24:36.821 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:36.821 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:36.821 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:24:36.821 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:26.308768+0000", 2024-09-13T11:24:36.821 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.0758999999999999e-05 2024-09-13T11:24:36.821 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:36.822 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:36.822 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:24:36.822 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:26.308874+0000", 2024-09-13T11:24:36.822 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000105334 2024-09-13T11:24:36.822 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:36.822 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:36.822 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:24:36.822 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:26.308890+0000", 2024-09-13T11:24:36.822 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.6367000000000001e-05 2024-09-13T11:24:36.822 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:36.822 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:36.822 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:24:36.822 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:26.309788+0000", 2024-09-13T11:24:36.822 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00089770300000000002 2024-09-13T11:24:36.823 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:36.823 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:24:36.823 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:36.823 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:36.823 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:36.823 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDPGPush(3.11 72/70 [PushOp(3:8da9c977:::benchmark_data_smithi177_31116_object7736:head, version: 46'461, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:8da9c977:::benchmark_data_smithi177_31116_object7736:head@46'461, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:24:36.823 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:26.393216+0000", 2024-09-13T11:24:36.823 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 70.423048015999996, 2024-09-13T11:24:36.823 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.15683955499999999, 2024-09-13T11:24:36.823 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:24:36.824 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:24:36.824 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:24:36.824 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:24:36.824 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:36.824 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:24:36.824 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:26.393216+0000", 2024-09-13T11:24:36.824 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:36.824 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:36.824 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:36.824 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:24:36.824 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:26.393216+0000", 2024-09-13T11:24:36.824 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:36.825 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:36.825 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:36.825 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:24:36.825 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:26.393217+0000", 2024-09-13T11:24:36.825 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2410000000000001e-06 2024-09-13T11:24:36.825 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:36.825 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:36.825 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:24:36.825 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:26.393309+0000", 2024-09-13T11:24:36.825 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.2101000000000005e-05 2024-09-13T11:24:36.825 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:36.825 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:36.825 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:24:36.826 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:26.547936+0000", 2024-09-13T11:24:36.826 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.15462682699999999 2024-09-13T11:24:36.826 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:36.826 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:36.826 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:24:36.826 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:26.548021+0000", 2024-09-13T11:24:36.826 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.4529000000000003e-05 2024-09-13T11:24:36.826 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:36.826 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:36.826 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:24:36.826 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:26.548321+0000", 2024-09-13T11:24:36.826 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00029988399999999999 2024-09-13T11:24:36.826 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:36.827 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:36.827 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:24:36.827 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:26.548353+0000", 2024-09-13T11:24:36.827 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.2573000000000001e-05 2024-09-13T11:24:36.827 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:36.827 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:36.827 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:24:36.827 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:26.550056+0000", 2024-09-13T11:24:36.827 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0017024 2024-09-13T11:24:36.827 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:36.827 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:24:36.827 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:36.827 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:36.828 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:36.828 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4665.0:54052 3.11 e74/70 3:8c6be821:::benchmark_data_smithi177_31116_object54051:head v 74'3049, mlcod=55'1352)", 2024-09-13T11:24:36.828 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:30.314329+0000", 2024-09-13T11:24:36.828 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 66.501934621999993, 2024-09-13T11:24:36.828 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.100070243, 2024-09-13T11:24:36.828 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:24:36.828 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:24:36.828 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:36.828 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:24:36.828 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:36.828 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:24:36.828 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:30.314329+0000", 2024-09-13T11:24:36.828 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:36.829 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:36.829 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:36.829 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:24:36.829 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:30.314329+0000", 2024-09-13T11:24:36.829 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:36.829 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:36.829 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:36.829 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:24:36.829 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:30.314332+0000", 2024-09-13T11:24:36.829 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.1900000000000002e-06 2024-09-13T11:24:36.829 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:36.829 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:36.829 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:24:36.830 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:30.314372+0000", 2024-09-13T11:24:36.830 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.0788000000000003e-05 2024-09-13T11:24:36.830 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:36.830 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:36.830 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:24:36.830 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:30.412014+0000", 2024-09-13T11:24:36.830 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.097641172999999998 2024-09-13T11:24:36.830 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:36.830 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:36.830 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:24:36.830 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:30.412026+0000", 2024-09-13T11:24:36.830 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2551999999999999e-05 2024-09-13T11:24:36.831 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:36.831 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:36.831 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:24:36.831 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:30.412134+0000", 2024-09-13T11:24:36.831 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00010763400000000001 2024-09-13T11:24:36.831 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:36.831 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:36.831 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:24:36.831 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:30.412172+0000", 2024-09-13T11:24:36.831 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.8333e-05 2024-09-13T11:24:36.831 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:36.831 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:36.831 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:24:36.832 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:30.414340+0000", 2024-09-13T11:24:36.832 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0021680879999999999 2024-09-13T11:24:36.832 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:36.832 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:36.832 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:24:36.832 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:30.414400+0000", 2024-09-13T11:24:36.832 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.9484999999999997e-05 2024-09-13T11:24:36.832 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:36.832 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:24:36.832 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:36.832 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:36.832 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:36.833 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDPGPush(3.1a 86/78 [PushOp(3:5df655d4:::benchmark_data_smithi177_31116_object12486:head, version: 49'767, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:5df655d4:::benchmark_data_smithi177_31116_object12486:head@49'767, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:24:36.833 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:43.456210+0000", 2024-09-13T11:24:36.833 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 53.360053714000003, 2024-09-13T11:24:36.833 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.131081792, 2024-09-13T11:24:36.833 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:24:36.833 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:24:36.833 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:24:36.833 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:24:36.833 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:36.833 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:24:36.833 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:43.456210+0000", 2024-09-13T11:24:36.833 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:36.833 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:36.834 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:36.834 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:24:36.834 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:43.456210+0000", 2024-09-13T11:24:36.834 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:36.834 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:36.834 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:36.834 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:24:36.834 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:43.456212+0000", 2024-09-13T11:24:36.834 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.922e-06 2024-09-13T11:24:36.834 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:36.834 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:36.834 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:24:36.834 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:43.456262+0000", 2024-09-13T11:24:36.835 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.9242000000000001e-05 2024-09-13T11:24:36.835 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:36.835 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:36.835 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:24:36.835 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:43.586188+0000", 2024-09-13T11:24:36.835 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.12992674400000001 2024-09-13T11:24:36.835 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:36.835 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:36.835 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:24:36.835 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:43.586209+0000", 2024-09-13T11:24:36.835 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.031e-05 2024-09-13T11:24:36.835 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:36.835 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:36.836 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:24:36.836 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:43.586325+0000", 2024-09-13T11:24:36.836 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000115983 2024-09-13T11:24:36.836 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:36.836 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:36.836 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:24:36.836 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:43.586340+0000", 2024-09-13T11:24:36.836 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.4941e-05 2024-09-13T11:24:36.836 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:36.836 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:36.836 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:24:36.836 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:43.587292+0000", 2024-09-13T11:24:36.836 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00095264999999999998 2024-09-13T11:24:36.837 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:36.837 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:24:36.837 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:36.837 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:36.837 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:36.837 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDPGPush(3.1e 89/78 [PushOp(3:7a07f38c:::benchmark_data_smithi177_31116_object20966:head, version: 54'1402, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:7a07f38c:::benchmark_data_smithi177_31116_object20966:head@54'1402, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:24:36.837 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:46.645251+0000", 2024-09-13T11:24:36.837 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 50.171013301000002, 2024-09-13T11:24:36.837 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.17670883800000001, 2024-09-13T11:24:36.837 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:24:36.837 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:24:36.837 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:24:36.837 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:24:36.838 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:36.838 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:24:36.838 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.645251+0000", 2024-09-13T11:24:36.838 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:36.838 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:36.838 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:36.838 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:24:36.838 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.645251+0000", 2024-09-13T11:24:36.838 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:36.838 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:36.838 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:36.838 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:24:36.838 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.645253+0000", 2024-09-13T11:24:36.839 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.1399999999999998e-06 2024-09-13T11:24:36.839 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:36.839 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:36.839 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:24:36.839 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.645295+0000", 2024-09-13T11:24:36.839 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.2323000000000001e-05 2024-09-13T11:24:36.839 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:36.839 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:36.839 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:24:36.839 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.820731+0000", 2024-09-13T11:24:36.839 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.17543607999999999 2024-09-13T11:24:36.839 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:36.839 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:36.840 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:24:36.840 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.820750+0000", 2024-09-13T11:24:36.840 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.8793000000000001e-05 2024-09-13T11:24:36.840 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:36.840 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:36.840 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:24:36.840 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.820914+0000", 2024-09-13T11:24:36.840 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000164062 2024-09-13T11:24:36.840 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:36.840 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:36.840 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:24:36.840 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.820939+0000", 2024-09-13T11:24:36.840 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.4672e-05 2024-09-13T11:24:36.841 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:36.841 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:36.841 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:24:36.841 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.821960+0000", 2024-09-13T11:24:36.841 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.001020768 2024-09-13T11:24:36.841 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:36.841 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:24:36.841 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:36.841 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:36.841 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:36.841 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:64320 3.11 3:8de5de4c:::benchmark_data_smithi177_31116_object64319:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90)", 2024-09-13T11:24:36.841 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:47.132785+0000", 2024-09-13T11:24:36.841 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 49.683478999999998, 2024-09-13T11:24:36.842 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.11946314600000001, 2024-09-13T11:24:36.842 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:24:36.842 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:24:36.842 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:36.842 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:24:36.842 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:24:36.842 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:36.842 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 64320 2024-09-13T11:24:36.842 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:36.842 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:24:36.842 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:36.842 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:24:36.843 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:47.132785+0000", 2024-09-13T11:24:36.843 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:36.843 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:36.843 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:36.843 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:24:36.843 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:47.132785+0000", 2024-09-13T11:24:36.843 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:36.843 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:36.843 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:36.843 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:24:36.843 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:47.132788+0000", 2024-09-13T11:24:36.843 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.4619999999999999e-06 2024-09-13T11:24:36.844 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:36.844 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:36.844 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:24:36.844 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:47.132838+0000", 2024-09-13T11:24:36.844 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.0739000000000003e-05 2024-09-13T11:24:36.844 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:36.844 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:36.844 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:24:36.844 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:47.132853+0000", 2024-09-13T11:24:36.844 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.518e-05 2024-09-13T11:24:36.844 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:36.844 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:36.844 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:24:36.845 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:47.132860+0000", 2024-09-13T11:24:36.845 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.8689999999999998e-06 2024-09-13T11:24:36.845 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:36.845 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:36.845 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:24:36.845 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:47.132922+0000", 2024-09-13T11:24:36.845 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.1251e-05 2024-09-13T11:24:36.845 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:36.845 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:36.845 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:24:36.845 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:47.133034+0000", 2024-09-13T11:24:36.845 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000112742 2024-09-13T11:24:36.846 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:36.846 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:36.846 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 11", 2024-09-13T11:24:36.846 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:47.133141+0000", 2024-09-13T11:24:36.846 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000106968 2024-09-13T11:24:36.846 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:36.846 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:36.846 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:24:36.846 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:47.135566+0000", 2024-09-13T11:24:36.846 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0024251839999999999 2024-09-13T11:24:36.846 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:36.846 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:36.846 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:36.847 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:47.252135+0000", 2024-09-13T11:24:36.847 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.11656870900000001 2024-09-13T11:24:36.847 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:36.847 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:36.847 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:24:36.847 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:47.252201+0000", 2024-09-13T11:24:36.847 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.6223999999999995e-05 2024-09-13T11:24:36.847 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:36.847 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:36.847 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:24:36.847 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:47.252248+0000", 2024-09-13T11:24:36.847 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.6817999999999998e-05 2024-09-13T11:24:36.847 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:36.847 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:24:36.848 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:36.848 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:36.848 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:36.848 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDPGPush(3.1e 91/78 [PushOp(3:78b7dd1b:::benchmark_data_smithi177_31116_object23441:head, version: 55'1540, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:78b7dd1b:::benchmark_data_smithi177_31116_object23441:head@55'1540, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:24:36.848 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:48.342756+0000", 2024-09-13T11:24:36.848 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 48.473507927999997, 2024-09-13T11:24:36.848 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.114126449, 2024-09-13T11:24:36.848 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:24:36.848 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:24:36.848 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:24:36.848 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:24:36.848 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:36.848 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:24:36.848 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:48.342756+0000", 2024-09-13T11:24:36.849 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:36.849 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:36.849 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:36.849 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:24:36.849 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:48.342756+0000", 2024-09-13T11:24:36.849 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:36.849 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:36.849 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:36.849 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:24:36.849 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:48.342758+0000", 2024-09-13T11:24:36.849 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.9769999999999999e-06 2024-09-13T11:24:36.849 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:36.849 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:36.850 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:24:36.850 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:48.342796+0000", 2024-09-13T11:24:36.850 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.8331000000000003e-05 2024-09-13T11:24:36.850 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:36.850 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:36.850 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:24:36.850 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:48.455578+0000", 2024-09-13T11:24:36.850 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.11278160800000001 2024-09-13T11:24:36.850 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:36.850 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:36.850 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:24:36.850 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:48.455631+0000", 2024-09-13T11:24:36.850 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.3229999999999997e-05 2024-09-13T11:24:36.851 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:36.851 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:36.851 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:24:36.851 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:48.455799+0000", 2024-09-13T11:24:36.851 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00016763100000000001 2024-09-13T11:24:36.851 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:36.851 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:36.851 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:24:36.851 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:48.455813+0000", 2024-09-13T11:24:36.851 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.4416999999999999e-05 2024-09-13T11:24:36.851 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:36.851 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:36.851 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:24:36.852 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:48.456883+0000", 2024-09-13T11:24:36.852 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0010692550000000001 2024-09-13T11:24:36.852 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:36.852 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:24:36.852 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:36.852 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:36.852 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:36.852 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:71903 3.11 3:8da9fc5e:::benchmark_data_smithi177_31116_object71902:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96)", 2024-09-13T11:24:36.852 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:54.140481+0000", 2024-09-13T11:24:36.852 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 42.675782748000003, 2024-09-13T11:24:36.852 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.094185646999999997, 2024-09-13T11:24:36.852 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:24:36.852 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:24:36.853 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:36.853 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:24:36.853 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:24:36.853 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:36.853 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 71903 2024-09-13T11:24:36.853 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:36.853 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:24:36.853 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:36.853 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:24:36.853 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:54.140481+0000", 2024-09-13T11:24:36.853 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:36.853 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:36.853 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:36.854 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:24:36.854 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:54.140481+0000", 2024-09-13T11:24:36.854 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:36.854 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:36.854 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:36.854 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:24:36.854 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:54.140485+0000", 2024-09-13T11:24:36.854 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.3610000000000001e-06 2024-09-13T11:24:36.854 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:36.854 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:36.854 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:24:36.854 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:54.140549+0000", 2024-09-13T11:24:36.854 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.4578000000000004e-05 2024-09-13T11:24:36.855 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:36.855 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:36.855 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:24:36.855 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:54.140574+0000", 2024-09-13T11:24:36.855 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.4640000000000001e-05 2024-09-13T11:24:36.855 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:36.855 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:36.855 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:24:36.855 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:54.140588+0000", 2024-09-13T11:24:36.855 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.4154e-05 2024-09-13T11:24:36.855 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:36.855 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:36.855 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:24:36.855 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:54.140718+0000", 2024-09-13T11:24:36.856 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00013008900000000001 2024-09-13T11:24:36.856 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:36.856 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:36.856 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:24:36.856 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:54.141021+0000", 2024-09-13T11:24:36.856 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00030258100000000001 2024-09-13T11:24:36.856 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:36.856 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:36.856 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 11", 2024-09-13T11:24:36.856 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:54.141326+0000", 2024-09-13T11:24:36.856 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00030553900000000002 2024-09-13T11:24:36.856 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:36.856 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:36.857 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:24:36.857 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:54.143701+0000", 2024-09-13T11:24:36.857 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.002374751 2024-09-13T11:24:36.857 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:36.857 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:36.857 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:36.857 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:54.234577+0000", 2024-09-13T11:24:36.857 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.090875517000000003 2024-09-13T11:24:36.857 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:36.857 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:36.857 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:24:36.857 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:54.234617+0000", 2024-09-13T11:24:36.857 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.0930999999999998e-05 2024-09-13T11:24:36.857 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:36.858 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:36.858 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:24:36.860 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:54.234667+0000", 2024-09-13T11:24:36.860 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.9505999999999997e-05 2024-09-13T11:24:36.860 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:36.860 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:24:36.860 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:36.860 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:36.860 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:36.860 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:71961 3.11 3:89bd9466:::benchmark_data_smithi177_31116_object71960:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2024-09-13T11:24:36.860 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:54.267475+0000", 2024-09-13T11:24:36.860 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 42.548789102000001, 2024-09-13T11:24:36.861 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.15027747899999999, 2024-09-13T11:24:36.861 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:24:36.861 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:24:36.861 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:36.861 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:24:36.861 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:24:36.861 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:36.861 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 71961 2024-09-13T11:24:36.861 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:36.861 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:24:36.861 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:36.861 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:24:36.862 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:54.267475+0000", 2024-09-13T11:24:36.862 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:36.862 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:36.862 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:36.862 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:24:36.862 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:54.267475+0000", 2024-09-13T11:24:36.862 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:36.862 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:36.862 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:36.862 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:24:36.862 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:54.267478+0000", 2024-09-13T11:24:36.862 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.8080000000000001e-06 2024-09-13T11:24:36.862 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:36.863 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:36.863 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:24:36.863 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:54.267534+0000", 2024-09-13T11:24:36.863 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.6276999999999997e-05 2024-09-13T11:24:36.863 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:36.863 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:36.863 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:24:36.863 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:54.267557+0000", 2024-09-13T11:24:36.863 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.2514000000000002e-05 2024-09-13T11:24:36.863 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:36.863 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:36.863 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:24:36.863 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:54.267570+0000", 2024-09-13T11:24:36.863 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.3251999999999999e-05 2024-09-13T11:24:36.864 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:36.864 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:36.864 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:24:36.864 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:54.267650+0000", 2024-09-13T11:24:36.864 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.0066000000000002e-05 2024-09-13T11:24:36.864 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:36.864 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:36.864 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:24:36.864 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:54.267886+0000", 2024-09-13T11:24:36.864 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00023637800000000001 2024-09-13T11:24:36.864 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:36.864 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:36.865 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 11", 2024-09-13T11:24:36.865 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:54.268020+0000", 2024-09-13T11:24:36.865 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00013392100000000001 2024-09-13T11:24:36.865 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:36.865 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:36.865 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:24:36.865 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:54.270076+0000", 2024-09-13T11:24:36.865 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.002055393 2024-09-13T11:24:36.865 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:36.865 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:36.865 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:36.865 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:54.417584+0000", 2024-09-13T11:24:36.866 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.147508008 2024-09-13T11:24:36.866 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:36.866 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:36.866 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:24:36.866 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:54.417667+0000", 2024-09-13T11:24:36.866 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.3666999999999997e-05 2024-09-13T11:24:36.866 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:36.866 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:36.866 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:24:36.866 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:54.417752+0000", 2024-09-13T11:24:36.866 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.5195e-05 2024-09-13T11:24:36.866 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:36.866 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:24:36.867 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:36.867 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:36.867 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:36.867 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4665.0:79776 3.1a e102/78 3:5d637d57:::benchmark_data_smithi177_31116_object79775:head v 102'4120, mlcod=72'3145)", 2024-09-13T11:24:36.867 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:24:00.354950+0000", 2024-09-13T11:24:36.867 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 36.461313758000003, 2024-09-13T11:24:36.867 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.14753330100000001, 2024-09-13T11:24:36.867 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:24:36.867 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:24:36.867 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:36.867 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:24:36.867 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:36.868 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:24:36.868 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:00.354950+0000", 2024-09-13T11:24:36.868 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:36.868 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:36.868 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:36.868 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:24:36.868 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:00.354950+0000", 2024-09-13T11:24:36.868 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:36.868 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:36.868 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:36.868 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:24:36.868 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:00.354960+0000", 2024-09-13T11:24:36.869 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.5219999999999996e-06 2024-09-13T11:24:36.869 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:36.869 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:36.869 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:24:36.869 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:00.355024+0000", 2024-09-13T11:24:36.869 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.4163000000000002e-05 2024-09-13T11:24:36.869 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:36.869 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:36.869 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:24:36.869 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:00.499446+0000", 2024-09-13T11:24:36.869 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.144422352 2024-09-13T11:24:36.869 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:36.870 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:36.870 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:24:36.870 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:00.499466+0000", 2024-09-13T11:24:36.870 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.9694000000000002e-05 2024-09-13T11:24:36.870 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:36.870 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:36.870 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:24:36.870 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:00.499605+0000", 2024-09-13T11:24:36.870 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00013935099999999999 2024-09-13T11:24:36.870 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:36.870 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:36.870 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:24:36.870 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:00.499655+0000", 2024-09-13T11:24:36.870 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.9787000000000003e-05 2024-09-13T11:24:36.871 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:36.871 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:36.871 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:24:36.871 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:00.502418+0000", 2024-09-13T11:24:36.871 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.002763012 2024-09-13T11:24:36.871 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:36.871 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:36.871 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:24:36.871 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:00.502484+0000", 2024-09-13T11:24:36.871 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.5419999999999994e-05 2024-09-13T11:24:36.871 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:36.871 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:24:36.872 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:36.872 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:36.872 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:36.872 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDPGPush(3.1a 102/78 [PushOp(3:5df50766:::benchmark_data_smithi177_31116_object50543:head, version: 72'3152, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:5df50766:::benchmark_data_smithi177_31116_object50543:head@72'3152, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:24:36.872 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:24:00.499311+0000", 2024-09-13T11:24:36.872 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 36.316953216000002, 2024-09-13T11:24:36.872 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.26146664800000002, 2024-09-13T11:24:36.872 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:24:36.872 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:24:36.872 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:24:36.872 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:24:36.872 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:36.872 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:24:36.873 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:00.499311+0000", 2024-09-13T11:24:36.873 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:36.873 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:36.873 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:36.873 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:24:36.873 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:00.499311+0000", 2024-09-13T11:24:36.873 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:36.873 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:36.873 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:36.873 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:24:36.873 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:00.499320+0000", 2024-09-13T11:24:36.874 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.6400000000000003e-06 2024-09-13T11:24:36.874 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:36.874 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:36.874 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:24:36.874 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:00.499384+0000", 2024-09-13T11:24:36.874 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.4554999999999994e-05 2024-09-13T11:24:36.874 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:36.874 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:36.874 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:24:36.874 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:00.759713+0000", 2024-09-13T11:24:36.874 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.260328742 2024-09-13T11:24:36.874 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:36.874 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:36.875 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:24:36.875 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:00.759742+0000", 2024-09-13T11:24:36.875 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.9298e-05 2024-09-13T11:24:36.875 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:36.875 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:36.875 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:24:36.875 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:00.759871+0000", 2024-09-13T11:24:36.875 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000128992 2024-09-13T11:24:36.875 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:36.875 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:36.875 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:24:36.875 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:00.759884+0000", 2024-09-13T11:24:36.875 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2687999999999999e-05 2024-09-13T11:24:36.876 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:36.876 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:36.876 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:24:36.876 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:00.760778+0000", 2024-09-13T11:24:36.876 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00089373299999999996 2024-09-13T11:24:36.876 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:36.876 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:24:36.876 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:36.876 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:36.876 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:36.876 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4665.0:82769 3.1e e105/102 3:7a4e88b5:::benchmark_data_smithi177_31116_object82768:head v 105'4381, mlcod=105'4379)", 2024-09-13T11:24:36.876 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:24:03.745047+0000", 2024-09-13T11:24:36.876 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 33.071216991, 2024-09-13T11:24:36.877 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.213806145, 2024-09-13T11:24:36.877 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:24:36.877 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:24:36.877 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:36.877 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:24:36.877 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:36.877 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:24:36.877 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:03.745047+0000", 2024-09-13T11:24:36.877 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:36.877 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:36.877 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:36.877 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:24:36.877 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:03.745047+0000", 2024-09-13T11:24:36.878 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:36.878 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:36.878 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:36.878 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:24:36.878 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:03.745049+0000", 2024-09-13T11:24:36.878 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.5710000000000001e-06 2024-09-13T11:24:36.878 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:36.878 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:36.878 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:24:36.878 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:03.745107+0000", 2024-09-13T11:24:36.878 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.8799999999999999e-05 2024-09-13T11:24:36.878 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:36.878 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:36.878 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:24:36.879 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:03.955145+0000", 2024-09-13T11:24:36.879 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.210037735 2024-09-13T11:24:36.879 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:36.879 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:36.879 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:24:36.879 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:03.955163+0000", 2024-09-13T11:24:36.879 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.7520000000000002e-05 2024-09-13T11:24:36.879 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:36.879 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:36.879 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:24:36.879 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:03.955473+0000", 2024-09-13T11:24:36.879 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00031014399999999999 2024-09-13T11:24:36.879 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:36.880 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:36.880 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:24:36.880 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:03.955538+0000", 2024-09-13T11:24:36.880 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.5059000000000002e-05 2024-09-13T11:24:36.880 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:36.880 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:36.880 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:24:36.880 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:03.958753+0000", 2024-09-13T11:24:36.880 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0032152769999999999 2024-09-13T11:24:36.880 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:36.880 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:36.880 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:24:36.880 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:03.958853+0000", 2024-09-13T11:24:36.881 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000100039 2024-09-13T11:24:36.881 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:36.881 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:24:36.881 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:36.881 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:36.881 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:36.881 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:90466 3.11 3:8b153e95:::benchmark_data_smithi177_31116_object90465:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110)", 2024-09-13T11:24:36.881 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:24:10.223535+0000", 2024-09-13T11:24:36.881 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 26.592728770000001, 2024-09-13T11:24:36.881 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.098477190000000006, 2024-09-13T11:24:36.881 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:24:36.881 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:24:36.881 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:36.881 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:24:36.882 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:24:36.882 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:36.882 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 90466 2024-09-13T11:24:36.882 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:36.882 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:24:36.882 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:36.882 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:24:36.882 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:10.223535+0000", 2024-09-13T11:24:36.882 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:36.882 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:36.882 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:36.882 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:24:36.882 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:10.223535+0000", 2024-09-13T11:24:36.882 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:36.883 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:36.883 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:36.883 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:24:36.883 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:10.223537+0000", 2024-09-13T11:24:36.883 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.9069999999999999e-06 2024-09-13T11:24:36.883 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:36.883 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:36.883 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:24:36.883 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:10.223607+0000", 2024-09-13T11:24:36.883 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.0219e-05 2024-09-13T11:24:36.883 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:36.883 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:36.883 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:24:36.884 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:10.223619+0000", 2024-09-13T11:24:36.884 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.1779e-05 2024-09-13T11:24:36.884 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:36.884 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:36.884 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:24:36.884 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:10.223627+0000", 2024-09-13T11:24:36.884 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.4159999999999998e-06 2024-09-13T11:24:36.884 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:36.884 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:36.884 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:24:36.884 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:10.223729+0000", 2024-09-13T11:24:36.884 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000102095 2024-09-13T11:24:36.884 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:36.885 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:36.885 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:24:36.885 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:10.223828+0000", 2024-09-13T11:24:36.885 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.8851999999999995e-05 2024-09-13T11:24:36.885 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:36.885 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:36.885 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 11", 2024-09-13T11:24:36.885 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:10.223928+0000", 2024-09-13T11:24:36.885 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000100048 2024-09-13T11:24:36.885 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:36.885 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:36.885 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:24:36.885 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:10.225388+0000", 2024-09-13T11:24:36.886 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.001460781 2024-09-13T11:24:36.886 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:36.886 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:36.886 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:36.886 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:10.321944+0000", 2024-09-13T11:24:36.886 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.096555941000000006 2024-09-13T11:24:36.886 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:36.886 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:36.886 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:24:36.886 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:10.321981+0000", 2024-09-13T11:24:36.886 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.6287e-05 2024-09-13T11:24:36.886 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:36.886 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:36.887 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:24:36.887 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:10.322013+0000", 2024-09-13T11:24:36.887 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.1865000000000001e-05 2024-09-13T11:24:36.887 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:36.887 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:24:36.887 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:36.887 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:36.887 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:36.887 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:98582 3.1a 3:59f9cd75:::benchmark_data_smithi177_31116_object98581:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113)", 2024-09-13T11:24:36.887 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:24:15.431226+0000", 2024-09-13T11:24:36.887 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 21.385038521999999, 2024-09-13T11:24:36.887 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.19820755800000001, 2024-09-13T11:24:36.887 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:24:36.888 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:24:36.888 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:36.888 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:24:36.888 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:24:36.888 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:36.888 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 98582 2024-09-13T11:24:36.888 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:36.888 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:24:36.888 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:36.888 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:24:36.888 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.431226+0000", 2024-09-13T11:24:36.888 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:36.888 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:36.888 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:36.889 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:24:36.889 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.431226+0000", 2024-09-13T11:24:36.889 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:36.889 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:36.889 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:36.889 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:24:36.889 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.431229+0000", 2024-09-13T11:24:36.889 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.7100000000000001e-06 2024-09-13T11:24:36.889 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:36.889 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:36.889 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:24:36.889 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.431334+0000", 2024-09-13T11:24:36.889 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00010506900000000001 2024-09-13T11:24:36.889 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:36.890 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:36.890 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:24:36.890 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.431364+0000", 2024-09-13T11:24:36.890 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.9607e-05 2024-09-13T11:24:36.890 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:36.890 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:36.890 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:24:36.890 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.431388+0000", 2024-09-13T11:24:36.890 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.3808999999999999e-05 2024-09-13T11:24:36.891 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:36.891 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:36.891 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:24:36.891 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.431483+0000", 2024-09-13T11:24:36.891 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.5402999999999994e-05 2024-09-13T11:24:36.891 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:36.891 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:36.891 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:24:36.891 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.431642+0000", 2024-09-13T11:24:36.891 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000158412 2024-09-13T11:24:36.891 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:36.891 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:36.891 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 9", 2024-09-13T11:24:36.891 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.431796+0000", 2024-09-13T11:24:36.892 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00015467899999999999 2024-09-13T11:24:36.892 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:36.892 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:36.892 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:24:36.892 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.433731+0000", 2024-09-13T11:24:36.892 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.001934945 2024-09-13T11:24:36.892 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:36.892 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:36.892 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:36.892 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.629245+0000", 2024-09-13T11:24:36.892 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.19551390199999999 2024-09-13T11:24:36.892 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:36.892 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:36.893 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:24:36.893 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.629346+0000", 2024-09-13T11:24:36.893 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000100869 2024-09-13T11:24:36.893 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:36.893 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:36.893 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:24:36.893 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.629433+0000", 2024-09-13T11:24:36.893 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.7152999999999997e-05 2024-09-13T11:24:36.893 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:36.893 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:24:36.893 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:36.893 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:36.893 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:36.893 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:98872 3.1a 3:5b894ab8:::benchmark_data_smithi177_31116_object98871:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113)", 2024-09-13T11:24:36.894 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:24:15.735049+0000", 2024-09-13T11:24:36.894 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 21.081215357000001, 2024-09-13T11:24:36.894 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.113370635, 2024-09-13T11:24:36.894 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:24:36.894 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:24:36.894 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:36.894 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:24:36.894 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:24:36.894 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:36.894 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 98872 2024-09-13T11:24:36.894 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:36.894 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:24:36.894 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:36.894 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:24:36.895 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.735049+0000", 2024-09-13T11:24:36.895 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:36.895 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:36.895 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:36.895 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:24:36.895 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.735049+0000", 2024-09-13T11:24:36.895 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:36.895 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:36.895 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:36.895 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:24:36.895 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.735052+0000", 2024-09-13T11:24:36.895 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.5700000000000001e-06 2024-09-13T11:24:36.895 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:36.896 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:36.896 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:24:36.896 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.735097+0000", 2024-09-13T11:24:36.896 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.4824999999999999e-05 2024-09-13T11:24:36.896 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:36.896 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:36.896 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:24:36.896 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.735122+0000", 2024-09-13T11:24:36.896 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.4550999999999999e-05 2024-09-13T11:24:36.896 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:36.896 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:36.896 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:24:36.896 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.735138+0000", 2024-09-13T11:24:36.896 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.6595e-05 2024-09-13T11:24:36.897 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:36.897 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:36.897 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:24:36.897 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.735300+0000", 2024-09-13T11:24:36.897 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00016126700000000001 2024-09-13T11:24:36.897 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:36.897 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:36.897 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:24:36.897 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.735558+0000", 2024-09-13T11:24:36.897 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00025827599999999999 2024-09-13T11:24:36.897 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:36.897 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:36.897 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 9", 2024-09-13T11:24:36.898 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.735791+0000", 2024-09-13T11:24:36.898 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000232933 2024-09-13T11:24:36.898 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:36.898 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:36.898 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:24:36.898 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.738826+0000", 2024-09-13T11:24:36.898 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0030354639999999999 2024-09-13T11:24:36.898 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:36.898 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:36.898 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:36.898 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.848242+0000", 2024-09-13T11:24:36.898 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.109415784 2024-09-13T11:24:36.898 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:36.899 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:36.899 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:24:36.899 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.848321+0000", 2024-09-13T11:24:36.899 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.9085000000000006e-05 2024-09-13T11:24:36.899 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:36.899 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:36.899 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:24:36.899 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.848419+0000", 2024-09-13T11:24:36.899 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.8284999999999998e-05 2024-09-13T11:24:36.899 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:36.899 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:24:36.899 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:36.899 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:36.900 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:36.900 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:99105 3.1a 3:5ac5090c:::benchmark_data_smithi177_31116_object99104:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113)", 2024-09-13T11:24:36.900 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:24:15.962466+0000", 2024-09-13T11:24:36.900 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 20.853798232999999, 2024-09-13T11:24:36.900 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.23832849, 2024-09-13T11:24:36.900 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:24:36.900 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:24:36.900 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:36.900 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:24:36.900 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:24:36.900 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:36.900 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 99105 2024-09-13T11:24:36.900 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:36.901 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:24:36.901 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:36.901 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:24:36.901 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.962466+0000", 2024-09-13T11:24:36.901 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:36.901 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:36.901 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:36.901 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:24:36.901 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.962466+0000", 2024-09-13T11:24:36.901 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:36.901 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:36.901 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:36.901 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:24:36.902 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.962471+0000", 2024-09-13T11:24:36.902 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.9570000000000001e-06 2024-09-13T11:24:36.902 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:36.902 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:36.902 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:24:36.902 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.962547+0000", 2024-09-13T11:24:36.902 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.5947999999999999e-05 2024-09-13T11:24:36.902 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:36.902 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:36.902 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:24:36.902 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.962576+0000", 2024-09-13T11:24:36.902 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.9547999999999999e-05 2024-09-13T11:24:36.902 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:36.903 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:36.903 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:24:36.903 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.962594+0000", 2024-09-13T11:24:36.903 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.8023e-05 2024-09-13T11:24:36.903 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:36.903 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:36.903 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:24:36.903 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.962770+0000", 2024-09-13T11:24:36.903 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00017551099999999999 2024-09-13T11:24:36.903 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:36.903 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:36.903 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:24:36.903 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.963114+0000", 2024-09-13T11:24:36.904 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00034440399999999998 2024-09-13T11:24:36.904 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:36.904 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:36.904 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 9", 2024-09-13T11:24:36.904 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.963351+0000", 2024-09-13T11:24:36.904 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000237209 2024-09-13T11:24:36.904 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:36.904 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:36.904 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:24:36.904 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.966345+0000", 2024-09-13T11:24:36.904 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0029933519999999999 2024-09-13T11:24:36.904 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:36.904 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:36.905 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:36.905 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:16.200724+0000", 2024-09-13T11:24:36.905 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.23437934599999999 2024-09-13T11:24:36.905 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:36.905 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:36.905 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:24:36.905 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:16.200761+0000", 2024-09-13T11:24:36.905 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.6406999999999999e-05 2024-09-13T11:24:36.905 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:36.905 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:36.905 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:24:36.905 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:16.200794+0000", 2024-09-13T11:24:36.905 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.3785000000000003e-05 2024-09-13T11:24:36.905 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:36.906 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:24:36.906 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:36.906 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:36.906 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:36.906 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4665.0:104331 3.1e e115/102 3:79890035:::benchmark_data_smithi177_31116_object104330:head v 115'5083, mlcod=115'5080)", 2024-09-13T11:24:36.906 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:24:20.242656+0000", 2024-09-13T11:24:36.906 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 16.573608141000001, 2024-09-13T11:24:36.906 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.16813846700000001, 2024-09-13T11:24:36.906 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:24:36.906 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:24:36.906 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:36.906 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:24:36.906 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:36.906 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:24:36.907 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:20.242656+0000", 2024-09-13T11:24:36.907 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:36.907 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:36.907 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:36.907 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:24:36.907 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:20.242656+0000", 2024-09-13T11:24:36.907 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:36.907 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:36.907 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:36.907 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:24:36.907 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:20.242657+0000", 2024-09-13T11:24:36.907 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.4750000000000001e-06 2024-09-13T11:24:36.907 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:36.908 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:36.908 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:24:36.908 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:20.242699+0000", 2024-09-13T11:24:36.908 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.193e-05 2024-09-13T11:24:36.908 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:36.908 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:36.908 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:24:36.908 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:20.408772+0000", 2024-09-13T11:24:36.908 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.16607291199999999 2024-09-13T11:24:36.908 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:36.908 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:36.908 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:24:36.908 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:20.408779+0000", 2024-09-13T11:24:36.909 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.3600000000000001e-06 2024-09-13T11:24:36.909 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:36.909 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:36.909 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:24:36.909 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:20.408840+0000", 2024-09-13T11:24:36.909 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.1834999999999993e-05 2024-09-13T11:24:36.909 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:36.909 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:36.909 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:24:36.909 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:20.408865+0000", 2024-09-13T11:24:36.909 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.4294000000000002e-05 2024-09-13T11:24:36.909 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:36.909 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:36.910 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:24:36.910 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:20.410751+0000", 2024-09-13T11:24:36.910 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.001885811 2024-09-13T11:24:36.910 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:36.910 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:36.910 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:24:36.910 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:20.410794+0000", 2024-09-13T11:24:36.910 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.3850000000000002e-05 2024-09-13T11:24:36.910 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:36.910 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:24:36.910 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:36.910 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:36.910 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:36.910 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4665.0:104823 3.1e e115/102 3:7a73472c:::benchmark_data_smithi177_31116_object104822:head v 115'5094, mlcod=115'5091)", 2024-09-13T11:24:36.911 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:24:20.661863+0000", 2024-09-13T11:24:36.911 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 16.154400827, 2024-09-13T11:24:36.911 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.100084861, 2024-09-13T11:24:36.911 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:24:36.911 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:24:36.911 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:36.911 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:24:36.911 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:36.911 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:24:36.911 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:20.661863+0000", 2024-09-13T11:24:36.911 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:36.911 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:36.911 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:36.912 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:24:36.912 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:20.661863+0000", 2024-09-13T11:24:36.912 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:36.912 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:36.912 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:36.912 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:24:36.912 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:20.661866+0000", 2024-09-13T11:24:36.912 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.3479999999999998e-06 2024-09-13T11:24:36.912 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:36.912 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:36.912 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:24:36.912 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:20.661906+0000", 2024-09-13T11:24:36.912 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.0076000000000001e-05 2024-09-13T11:24:36.912 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:36.913 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:36.913 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:24:36.913 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:20.760131+0000", 2024-09-13T11:24:36.913 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.098225420999999993 2024-09-13T11:24:36.913 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:36.913 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:36.913 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:24:36.913 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:20.760141+0000", 2024-09-13T11:24:36.913 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.027e-05 2024-09-13T11:24:36.913 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:36.913 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:36.913 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:24:36.913 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:20.760198+0000", 2024-09-13T11:24:36.914 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.6127e-05 2024-09-13T11:24:36.914 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:36.914 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:36.914 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:24:36.914 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:20.760214+0000", 2024-09-13T11:24:36.914 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.6932e-05 2024-09-13T11:24:36.914 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:36.914 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:36.914 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:24:36.914 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:20.761876+0000", 2024-09-13T11:24:36.914 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0016611690000000001 2024-09-13T11:24:36.914 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:36.914 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:36.914 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:24:36.915 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:20.761948+0000", 2024-09-13T11:24:36.915 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.2517999999999999e-05 2024-09-13T11:24:36.915 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:36.915 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:24:36.915 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:36.915 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:36.915 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:36.915 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:105026 3.1a 3:5db5c278:::benchmark_data_smithi177_31116_object105025:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-09-13T11:24:36.915 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:24:20.851015+0000", 2024-09-13T11:24:36.915 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 15.965249411, 2024-09-13T11:24:36.915 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.095865153999999994, 2024-09-13T11:24:36.915 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:24:36.915 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:24:36.915 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:36.916 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:24:36.916 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:24:36.916 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:36.916 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 105026 2024-09-13T11:24:36.916 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:36.916 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:24:36.916 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:36.916 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:24:36.916 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:20.851015+0000", 2024-09-13T11:24:36.916 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:36.916 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:36.916 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:36.916 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:24:36.916 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:20.851015+0000", 2024-09-13T11:24:36.917 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:36.917 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:36.917 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:36.917 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:24:36.917 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:20.851018+0000", 2024-09-13T11:24:36.917 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.954e-06 2024-09-13T11:24:36.917 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:36.917 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:36.917 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:24:36.917 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:20.851057+0000", 2024-09-13T11:24:36.917 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.9101999999999999e-05 2024-09-13T11:24:36.917 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:36.917 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:36.917 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:24:36.918 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:20.851079+0000", 2024-09-13T11:24:36.918 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.2722e-05 2024-09-13T11:24:36.918 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:36.918 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:36.918 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:24:36.918 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:20.851097+0000", 2024-09-13T11:24:36.918 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.719e-05 2024-09-13T11:24:36.918 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:36.918 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:36.918 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:24:36.918 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:20.851186+0000", 2024-09-13T11:24:36.918 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.8850000000000005e-05 2024-09-13T11:24:36.918 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:36.919 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:36.919 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:24:36.919 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:20.851269+0000", 2024-09-13T11:24:36.919 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.3257000000000006e-05 2024-09-13T11:24:36.919 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:36.919 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:36.919 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 9", 2024-09-13T11:24:36.919 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:20.851382+0000", 2024-09-13T11:24:36.919 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00011323 2024-09-13T11:24:36.919 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:36.919 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:36.919 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:24:36.919 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:20.853102+0000", 2024-09-13T11:24:36.920 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0017195979999999999 2024-09-13T11:24:36.920 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:36.920 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:36.920 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:36.920 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:20.946731+0000", 2024-09-13T11:24:36.920 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.093629195999999998 2024-09-13T11:24:36.920 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:36.920 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:36.920 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:24:36.920 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:20.946817+0000", 2024-09-13T11:24:36.920 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.6032000000000006e-05 2024-09-13T11:24:36.920 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:36.920 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:36.921 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:24:36.921 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:20.946880+0000", 2024-09-13T11:24:36.921 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.3022999999999996e-05 2024-09-13T11:24:36.921 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:36.921 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:24:36.921 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:36.921 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:36.921 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:36.921 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:113165 3.11 3:8dbd8762:::benchmark_data_smithi177_31116_object113164:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124)", 2024-09-13T11:24:36.921 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:24:31.641595+0000", 2024-09-13T11:24:36.921 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 5.1746695909999998, 2024-09-13T11:24:36.921 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.117623435, 2024-09-13T11:24:36.921 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:24:36.921 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:24:36.922 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:36.922 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:24:36.922 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:24:36.922 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:36.922 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 113165 2024-09-13T11:24:36.922 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:36.922 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:24:36.922 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:36.922 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:24:36.922 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:31.641595+0000", 2024-09-13T11:24:36.922 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:36.922 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:36.922 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:36.923 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:24:36.923 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:31.641595+0000", 2024-09-13T11:24:36.923 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:36.923 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:36.923 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:36.923 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:24:36.923 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:31.641596+0000", 2024-09-13T11:24:36.923 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.553e-06 2024-09-13T11:24:36.923 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:36.923 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:36.923 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:24:36.923 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:31.641655+0000", 2024-09-13T11:24:36.923 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.8737999999999999e-05 2024-09-13T11:24:36.924 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:36.924 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:36.924 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:24:36.924 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:31.641664+0000", 2024-09-13T11:24:36.924 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.2429999999999994e-06 2024-09-13T11:24:36.924 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:36.924 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:36.924 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:24:36.924 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:31.641669+0000", 2024-09-13T11:24:36.924 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.7790000000000003e-06 2024-09-13T11:24:36.924 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:36.924 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:36.924 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:24:36.925 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:31.641723+0000", 2024-09-13T11:24:36.925 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.3718999999999998e-05 2024-09-13T11:24:36.925 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:36.925 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:36.925 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:24:36.925 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:31.641846+0000", 2024-09-13T11:24:36.925 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000123194 2024-09-13T11:24:36.925 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:36.925 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:36.925 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 11", 2024-09-13T11:24:36.925 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:31.641967+0000", 2024-09-13T11:24:36.925 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00012108900000000001 2024-09-13T11:24:36.925 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:36.925 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:36.926 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:24:36.926 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:31.643582+0000", 2024-09-13T11:24:36.926 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.001615214 2024-09-13T11:24:36.926 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:36.926 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:36.926 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:36.926 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:31.759116+0000", 2024-09-13T11:24:36.926 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.115533633 2024-09-13T11:24:36.926 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:36.926 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:36.926 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:24:36.926 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:31.759174+0000", 2024-09-13T11:24:36.926 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.8140999999999998e-05 2024-09-13T11:24:36.926 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:36.927 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:36.927 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:24:36.927 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:31.759218+0000", 2024-09-13T11:24:36.927 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.4131999999999997e-05 2024-09-13T11:24:36.927 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:36.927 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:24:36.927 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:36.927 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:36.927 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:24:36.927 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-13T11:24:36.928 INFO:teuthology.orchestra.run.smithi001.stdout:osd.8: {} 2024-09-13T11:24:36.929 INFO:teuthology.orchestra.run.smithi001.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-13T11:24:36.930 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-09-13T11:24:36.930 INFO:teuthology.orchestra.run.smithi001.stdout:osd.9: {} 2024-09-13T11:24:36.930 INFO:teuthology.orchestra.run.smithi001.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-13T11:24:36.931 INFO:teuthology.orchestra.run.smithi001.stdout:osd.10: {} 2024-09-13T11:24:36.931 INFO:teuthology.orchestra.run.smithi001.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-13T11:24:36.931 INFO:teuthology.orchestra.run.smithi001.stdout:osd.11: {} 2024-09-13T11:24:36.931 INFO:teuthology.orchestra.run.smithi001.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-13T11:24:36.932 INFO:tasks.ceph.osd.8.smithi001.stderr:2024-09-13T11:24:36.927+0000 7fb9b48ae640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26312) UID: 0 2024-09-13T11:24:36.945 INFO:teuthology.orchestra.run.smithi001.stdout:osd.12: {} 2024-09-13T11:24:36.945 INFO:teuthology.orchestra.run.smithi001.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-13T11:24:36.976 INFO:teuthology.orchestra.run.smithi001.stdout:osd.13: {} 2024-09-13T11:24:36.976 INFO:teuthology.orchestra.run.smithi001.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-13T11:24:36.976 INFO:teuthology.orchestra.run.smithi001.stderr:Error ENXIO: problem getting command descriptions from osd.14 2024-09-13T11:24:37.008 INFO:teuthology.orchestra.run.smithi001.stdout:osd.15: {} 2024-09-13T11:24:37.009 INFO:teuthology.orchestra.run.smithi001.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-13T11:24:37.011 INFO:tasks.radosbench.radosbench.0.smithi177.stdout: 130 16 119086 119070 57.2389 74.625 0.0066073 0.0174644 2024-09-13T11:24:37.012 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-13T11:24:37.013 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-13T11:24:37.029 INFO:tasks.ceph.osd.9.smithi092.stderr:2024-09-13T11:24:37.022+0000 7ff5ab3d0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26348) UID: 0 2024-09-13T11:24:37.088 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-13T11:24:37.089 INFO:teuthology.orchestra.run.smithi001.stdout: "ops": [ 2024-09-13T11:24:37.089 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:37.089 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:119160 3.27 3:e5f442cf:::benchmark_data_smithi177_31116_object119159:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e128)", 2024-09-13T11:24:37.089 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:24:37.085234+0000", 2024-09-13T11:24:37.089 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 0.0026432729999999998, 2024-09-13T11:24:37.089 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0026731989999999998, 2024-09-13T11:24:37.089 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:24:37.089 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:24:37.089 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "waiting for sub ops", 2024-09-13T11:24:37.089 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:24:37.089 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:24:37.090 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:37.090 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 119160 2024-09-13T11:24:37.090 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:37.090 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:24:37.090 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:37.090 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:24:37.090 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:37.085234+0000", 2024-09-13T11:24:37.090 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:37.090 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:37.090 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:37.090 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:24:37.090 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:37.085234+0000", 2024-09-13T11:24:37.090 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:37.090 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:37.091 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:37.091 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:24:37.091 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:37.085237+0000", 2024-09-13T11:24:37.091 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.5229999999999998e-06 2024-09-13T11:24:37.091 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:37.091 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:37.091 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:24:37.091 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:37.085274+0000", 2024-09-13T11:24:37.091 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.6820000000000003e-05 2024-09-13T11:24:37.091 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:37.091 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:37.091 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:24:37.091 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:37.085291+0000", 2024-09-13T11:24:37.092 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.7082e-05 2024-09-13T11:24:37.092 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:37.092 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:37.092 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:24:37.092 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:37.085298+0000", 2024-09-13T11:24:37.092 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.4379999999999998e-06 2024-09-13T11:24:37.092 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:37.092 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:37.092 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:24:37.092 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:37.085357+0000", 2024-09-13T11:24:37.092 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.8882000000000003e-05 2024-09-13T11:24:37.092 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:37.092 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:37.093 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:24:37.093 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:37.085506+0000", 2024-09-13T11:24:37.093 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00014866400000000001 2024-09-13T11:24:37.093 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:37.093 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:37.093 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 5,11", 2024-09-13T11:24:37.093 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:37.085609+0000", 2024-09-13T11:24:37.093 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000103263 2024-09-13T11:24:37.093 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:37.093 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:24:37.093 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:37.093 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:37.093 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-09-13T11:24:37.094 INFO:teuthology.orchestra.run.smithi001.stdout: "num_ops": 1 2024-09-13T11:24:37.094 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-13T11:24:37.108 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-09-13T11:24:37.130 INFO:tasks.ceph.osd.15.smithi177.stderr:2024-09-13T11:24:37.127+0000 7f9d6ce77640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26578) UID: 0 2024-09-13T11:24:37.230 INFO:tasks.ceph.osd.3.smithi177.stderr:2024-09-13T11:24:37.227+0000 7f10adba4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26574) UID: 0 2024-09-13T11:24:37.253 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-13T11:24:37.253 INFO:teuthology.orchestra.run.smithi001.stdout: "ops": [], 2024-09-13T11:24:37.253 INFO:teuthology.orchestra.run.smithi001.stdout: "complaint_time": 30, 2024-09-13T11:24:37.253 INFO:teuthology.orchestra.run.smithi001.stdout: "num_blocked_ops": 0 2024-09-13T11:24:37.254 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-13T11:24:37.278 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-09-13T11:24:37.308 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-13T11:24:37.368 INFO:tasks.ceph.osd.12.smithi001.stderr:2024-09-13T11:24:37.367+0000 7f892c6e6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26314) UID: 0 2024-09-13T11:24:37.431 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-13T11:24:37.432 INFO:teuthology.orchestra.run.smithi001.stdout: "size": 20, 2024-09-13T11:24:37.432 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 600, 2024-09-13T11:24:37.432 INFO:teuthology.orchestra.run.smithi001.stdout: "ops": [ 2024-09-13T11:24:37.432 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:37.432 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:30801 3.e 3:73b0f21a:::benchmark_data_smithi177_31116_object30800:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e59)", 2024-09-13T11:24:37.432 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:09.485629+0000", 2024-09-13T11:24:37.432 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 87.943729185999999, 2024-09-13T11:24:37.432 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.22604919900000001, 2024-09-13T11:24:37.432 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:24:37.432 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:24:37.432 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:37.432 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:24:37.432 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:24:37.433 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:37.433 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 30801 2024-09-13T11:24:37.433 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:37.433 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:24:37.433 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:37.433 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:24:37.433 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.485629+0000", 2024-09-13T11:24:37.433 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:37.433 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:37.433 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:37.433 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:24:37.433 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.485629+0000", 2024-09-13T11:24:37.433 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:37.434 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:37.434 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:37.434 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:24:37.434 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.485631+0000", 2024-09-13T11:24:37.434 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.9989999999999998e-06 2024-09-13T11:24:37.434 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:37.434 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:37.434 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:24:37.434 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.485664+0000", 2024-09-13T11:24:37.434 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.3896000000000002e-05 2024-09-13T11:24:37.434 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:37.434 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:37.435 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:24:37.435 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.485679+0000", 2024-09-13T11:24:37.435 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.4438e-05 2024-09-13T11:24:37.435 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:37.435 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:37.435 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:24:37.435 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.485688+0000", 2024-09-13T11:24:37.435 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.0180000000000007e-06 2024-09-13T11:24:37.435 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:37.436 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:37.436 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:24:37.436 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.485742+0000", 2024-09-13T11:24:37.436 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.4026000000000001e-05 2024-09-13T11:24:37.436 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:37.436 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:37.436 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:24:37.436 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.485841+0000", 2024-09-13T11:24:37.436 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.8875000000000005e-05 2024-09-13T11:24:37.436 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:37.436 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:37.436 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 0,1", 2024-09-13T11:24:37.436 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.486005+0000", 2024-09-13T11:24:37.436 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000163747 2024-09-13T11:24:37.437 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:37.437 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:37.437 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:24:37.437 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.487867+0000", 2024-09-13T11:24:37.437 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.001862945 2024-09-13T11:24:37.437 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:37.437 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:37.437 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:37.437 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.488570+0000", 2024-09-13T11:24:37.437 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00070255700000000003 2024-09-13T11:24:37.438 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:37.438 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:37.438 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:37.438 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.711140+0000", 2024-09-13T11:24:37.438 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.222570348 2024-09-13T11:24:37.438 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:37.438 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:37.438 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:24:37.438 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.711180+0000", 2024-09-13T11:24:37.438 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.9273999999999997e-05 2024-09-13T11:24:37.438 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:37.438 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:37.438 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:24:37.439 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.711678+0000", 2024-09-13T11:24:37.439 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00049807599999999997 2024-09-13T11:24:37.439 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:37.439 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:24:37.439 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:37.439 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:37.439 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:37.439 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:54656 3.1e 3.6a49163e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e75)", 2024-09-13T11:24:37.439 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:30.979993+0000", 2024-09-13T11:24:37.439 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 66.449364754000001, 2024-09-13T11:24:37.439 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.9269557150000001, 2024-09-13T11:24:37.439 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:24:37.439 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:24:37.439 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-13T11:24:37.440 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:24:37.440 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:24:37.440 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:37.440 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 54656 2024-09-13T11:24:37.440 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:37.440 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:24:37.440 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:37.440 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:24:37.441 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:30.979993+0000", 2024-09-13T11:24:37.441 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:37.441 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:37.441 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:37.441 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:24:37.441 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:30.979993+0000", 2024-09-13T11:24:37.441 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:37.441 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:37.441 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:37.441 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:24:37.441 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:30.979997+0000", 2024-09-13T11:24:37.441 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.0319999999999997e-06 2024-09-13T11:24:37.441 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:37.441 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:37.442 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:24:37.442 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:30.980057+0000", 2024-09-13T11:24:37.442 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.0341999999999999e-05 2024-09-13T11:24:37.442 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:37.442 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:37.442 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:24:37.442 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:30.980094+0000", 2024-09-13T11:24:37.442 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.6770000000000002e-05 2024-09-13T11:24:37.442 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:37.442 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:37.443 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:24:37.443 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:30.980113+0000", 2024-09-13T11:24:37.443 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.8791999999999999e-05 2024-09-13T11:24:37.443 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:37.443 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:37.443 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:24:37.443 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.097773+0000", 2024-09-13T11:24:37.443 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.117660379 2024-09-13T11:24:37.443 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:37.443 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:37.443 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for peered", 2024-09-13T11:24:37.443 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.097784+0000", 2024-09-13T11:24:37.443 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.1025e-05 2024-09-13T11:24:37.444 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:37.444 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:37.444 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:24:37.444 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.906937+0000", 2024-09-13T11:24:37.444 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.8091523789999999 2024-09-13T11:24:37.444 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:37.444 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:37.444 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:24:37.444 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.906949+0000", 2024-09-13T11:24:37.444 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.1996e-05 2024-09-13T11:24:37.444 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:37.444 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:24:37.445 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:37.445 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:37.445 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:37.445 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDPGPushReply(3.17 75/70 [PushReplyOp(3:e82800c8:::benchmark_data_smithi177_31116_object13909:head)])", 2024-09-13T11:24:37.445 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:31.632344+0000", 2024-09-13T11:24:37.445 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 65.797013899000007, 2024-09-13T11:24:37.445 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.28592102899999999, 2024-09-13T11:24:37.445 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:24:37.445 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:24:37.445 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-13T11:24:37.445 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:24:37.445 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:37.445 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:24:37.445 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.632344+0000", 2024-09-13T11:24:37.446 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:37.446 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:37.446 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:37.446 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:24:37.446 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.632344+0000", 2024-09-13T11:24:37.446 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:37.446 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:37.446 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:37.446 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:24:37.446 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.632346+0000", 2024-09-13T11:24:37.446 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.8819999999999999e-06 2024-09-13T11:24:37.446 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:37.446 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:37.446 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:24:37.447 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.632353+0000", 2024-09-13T11:24:37.447 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.5059999999999998e-06 2024-09-13T11:24:37.447 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:37.447 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:37.447 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:24:37.447 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.917361+0000", 2024-09-13T11:24:37.447 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.285008231 2024-09-13T11:24:37.447 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:37.447 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:37.447 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:24:37.447 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.917372+0000", 2024-09-13T11:24:37.447 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0764999999999999e-05 2024-09-13T11:24:37.448 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:37.448 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:37.448 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:24:37.448 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.917458+0000", 2024-09-13T11:24:37.448 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.6136000000000003e-05 2024-09-13T11:24:37.448 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:37.448 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:37.448 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:24:37.448 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.918265+0000", 2024-09-13T11:24:37.448 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00080650899999999996 2024-09-13T11:24:37.449 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:37.449 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:24:37.449 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:37.449 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:37.449 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:37.449 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:54720 3.1e 3.a944303e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-13T11:24:37.449 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:31.973315+0000", 2024-09-13T11:24:37.449 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 65.456042518999993, 2024-09-13T11:24:37.449 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.93366527200000005, 2024-09-13T11:24:37.449 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:24:37.449 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:24:37.449 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-13T11:24:37.449 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:24:37.450 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:24:37.450 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:37.450 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 54720 2024-09-13T11:24:37.450 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:37.450 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:24:37.450 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:37.450 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:24:37.450 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.973315+0000", 2024-09-13T11:24:37.450 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:37.450 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:37.450 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:37.450 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:24:37.450 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.973315+0000", 2024-09-13T11:24:37.451 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:37.451 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:37.451 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:37.451 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:24:37.451 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.973318+0000", 2024-09-13T11:24:37.451 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.5830000000000001e-06 2024-09-13T11:24:37.451 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:37.451 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:37.451 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:24:37.451 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.973465+0000", 2024-09-13T11:24:37.451 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000146816 2024-09-13T11:24:37.451 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:37.451 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:37.452 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:24:37.452 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.973478+0000", 2024-09-13T11:24:37.452 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2931e-05 2024-09-13T11:24:37.452 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:37.452 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:37.452 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:24:37.452 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.973483+0000", 2024-09-13T11:24:37.452 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.8969999999999996e-06 2024-09-13T11:24:37.452 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:37.453 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:37.453 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:24:37.453 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.973548+0000", 2024-09-13T11:24:37.453 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.4771999999999996e-05 2024-09-13T11:24:37.453 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:37.453 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:37.453 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for peered", 2024-09-13T11:24:37.453 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.973557+0000", 2024-09-13T11:24:37.453 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.0359999999999995e-06 2024-09-13T11:24:37.453 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:37.453 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:37.453 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:24:37.453 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.906974+0000", 2024-09-13T11:24:37.453 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.93341706000000002 2024-09-13T11:24:37.454 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:37.454 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:37.454 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:24:37.454 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.906980+0000", 2024-09-13T11:24:37.454 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.1770000000000003e-06 2024-09-13T11:24:37.454 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:37.454 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:24:37.454 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:37.454 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:37.454 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:37.454 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:54734 3.1e 3.d7f6f7e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-13T11:24:37.454 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:31.983342+0000", 2024-09-13T11:24:37.454 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 65.446015716000005, 2024-09-13T11:24:37.455 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.92367058599999996, 2024-09-13T11:24:37.455 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:24:37.455 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:24:37.455 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-13T11:24:37.455 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:24:37.455 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:24:37.455 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:37.455 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 54734 2024-09-13T11:24:37.455 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:37.455 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:24:37.455 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:37.455 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:24:37.455 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.983342+0000", 2024-09-13T11:24:37.455 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:37.456 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:37.456 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:37.456 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:24:37.456 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.983342+0000", 2024-09-13T11:24:37.456 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:37.456 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:37.456 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:37.456 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:24:37.456 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.983345+0000", 2024-09-13T11:24:37.456 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.6220000000000001e-06 2024-09-13T11:24:37.456 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:37.456 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:37.456 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:24:37.456 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.983438+0000", 2024-09-13T11:24:37.457 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.3517000000000006e-05 2024-09-13T11:24:37.457 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:37.457 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:37.457 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:24:37.457 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.983451+0000", 2024-09-13T11:24:37.457 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2527999999999999e-05 2024-09-13T11:24:37.457 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:37.457 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:37.457 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:24:37.457 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.983458+0000", 2024-09-13T11:24:37.457 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.6380000000000004e-06 2024-09-13T11:24:37.457 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:37.457 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:37.458 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:24:37.458 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.983521+0000", 2024-09-13T11:24:37.458 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.2729000000000002e-05 2024-09-13T11:24:37.458 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:37.458 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:37.458 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for peered", 2024-09-13T11:24:37.458 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.983548+0000", 2024-09-13T11:24:37.458 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.6537999999999999e-05 2024-09-13T11:24:37.458 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:37.458 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:37.458 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:24:37.458 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.907005+0000", 2024-09-13T11:24:37.458 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.923456991 2024-09-13T11:24:37.458 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:37.459 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:37.459 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:24:37.459 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.907013+0000", 2024-09-13T11:24:37.459 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.0229999999999992e-06 2024-09-13T11:24:37.459 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:37.459 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:24:37.459 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:37.459 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:37.459 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:37.459 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:54760 3.1e 3.8c32c13e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-13T11:24:37.459 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:31.999685+0000", 2024-09-13T11:24:37.459 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 65.429672483999994, 2024-09-13T11:24:37.459 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.90735639599999995, 2024-09-13T11:24:37.459 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:24:37.460 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:24:37.460 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-13T11:24:37.460 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:24:37.460 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:24:37.460 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:37.460 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 54760 2024-09-13T11:24:37.460 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:37.460 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:24:37.460 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:37.461 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:24:37.461 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.999685+0000", 2024-09-13T11:24:37.461 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:37.461 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:37.461 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:37.461 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:24:37.461 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.999685+0000", 2024-09-13T11:24:37.461 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:37.461 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:37.461 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:37.461 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:24:37.461 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.999690+0000", 2024-09-13T11:24:37.462 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.7879999999999997e-06 2024-09-13T11:24:37.462 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:37.462 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:37.462 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:24:37.462 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.999729+0000", 2024-09-13T11:24:37.462 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.9032000000000002e-05 2024-09-13T11:24:37.462 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:37.462 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:37.462 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:24:37.462 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.999741+0000", 2024-09-13T11:24:37.462 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2136999999999999e-05 2024-09-13T11:24:37.462 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:37.462 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:37.463 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:24:37.463 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.999748+0000", 2024-09-13T11:24:37.463 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.9120000000000001e-06 2024-09-13T11:24:37.463 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:37.463 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:37.463 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:24:37.463 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.999803+0000", 2024-09-13T11:24:37.463 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.5064e-05 2024-09-13T11:24:37.463 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:37.463 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:37.463 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for peered", 2024-09-13T11:24:37.463 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.999815+0000", 2024-09-13T11:24:37.463 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2041e-05 2024-09-13T11:24:37.464 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:37.464 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:37.464 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:24:37.464 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.907034+0000", 2024-09-13T11:24:37.464 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.90721858200000005 2024-09-13T11:24:37.464 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:37.464 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:37.464 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:24:37.464 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.907042+0000", 2024-09-13T11:24:37.464 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.8399999999999995e-06 2024-09-13T11:24:37.464 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:37.464 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:24:37.465 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:37.465 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:37.465 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:37.465 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:54768 3.1e 3.6dad499e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-13T11:24:37.465 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:32.003482+0000", 2024-09-13T11:24:37.465 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 65.425875833999996, 2024-09-13T11:24:37.465 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.90358804699999995, 2024-09-13T11:24:37.465 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:24:37.465 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:24:37.465 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-13T11:24:37.465 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:24:37.465 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:24:37.466 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:37.466 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 54768 2024-09-13T11:24:37.466 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:37.466 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:24:37.466 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:37.466 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:24:37.466 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.003482+0000", 2024-09-13T11:24:37.466 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:37.466 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:37.466 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:37.466 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:24:37.466 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.003482+0000", 2024-09-13T11:24:37.466 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:37.467 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:37.467 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:37.467 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:24:37.467 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.003486+0000", 2024-09-13T11:24:37.467 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.1030000000000004e-06 2024-09-13T11:24:37.467 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:37.467 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:37.467 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:24:37.467 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.003523+0000", 2024-09-13T11:24:37.467 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.6732e-05 2024-09-13T11:24:37.467 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:37.467 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:37.467 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:24:37.468 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.003535+0000", 2024-09-13T11:24:37.468 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2520999999999999e-05 2024-09-13T11:24:37.468 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:37.468 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:37.468 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:24:37.468 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.003541+0000", 2024-09-13T11:24:37.468 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.8690000000000001e-06 2024-09-13T11:24:37.468 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:37.468 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:37.468 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:24:37.468 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.003593+0000", 2024-09-13T11:24:37.468 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.1786000000000002e-05 2024-09-13T11:24:37.468 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:37.469 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:37.469 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for peered", 2024-09-13T11:24:37.469 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.003602+0000", 2024-09-13T11:24:37.469 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.9679999999999995e-06 2024-09-13T11:24:37.469 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:37.469 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:37.469 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:24:37.469 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.907063+0000", 2024-09-13T11:24:37.469 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.90346116600000004 2024-09-13T11:24:37.469 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:37.469 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:37.469 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:24:37.469 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.907070+0000", 2024-09-13T11:24:37.470 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.9020000000000002e-06 2024-09-13T11:24:37.470 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:37.470 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:24:37.470 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:37.472 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:37.472 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:37.472 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:54785 3.1e 3.d32f1ede (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-13T11:24:37.472 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:32.016238+0000", 2024-09-13T11:24:37.472 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 65.413119624000004, 2024-09-13T11:24:37.472 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.89086243399999998, 2024-09-13T11:24:37.472 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:24:37.472 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:24:37.472 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-13T11:24:37.472 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:24:37.472 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:24:37.472 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:37.473 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 54785 2024-09-13T11:24:37.473 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:37.473 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:24:37.473 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:37.473 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:24:37.473 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.016238+0000", 2024-09-13T11:24:37.473 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:37.473 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:37.473 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:37.473 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:24:37.473 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.016238+0000", 2024-09-13T11:24:37.473 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:37.473 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:37.474 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:37.474 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:24:37.474 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.016241+0000", 2024-09-13T11:24:37.474 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.6469999999999998e-06 2024-09-13T11:24:37.474 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:37.474 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:37.474 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:24:37.474 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.016271+0000", 2024-09-13T11:24:37.474 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.0505999999999999e-05 2024-09-13T11:24:37.474 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:37.474 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:37.474 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:24:37.474 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.016287+0000", 2024-09-13T11:24:37.475 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.6073999999999999e-05 2024-09-13T11:24:37.475 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:37.475 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:37.475 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:24:37.475 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.016295+0000", 2024-09-13T11:24:37.475 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.0590000000000003e-06 2024-09-13T11:24:37.475 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:37.475 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:37.475 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:24:37.475 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.016373+0000", 2024-09-13T11:24:37.475 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.7634999999999998e-05 2024-09-13T11:24:37.475 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:37.475 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:37.475 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for peered", 2024-09-13T11:24:37.475 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.016383+0000", 2024-09-13T11:24:37.476 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0246e-05 2024-09-13T11:24:37.476 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:37.476 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:37.476 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:24:37.476 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.907092+0000", 2024-09-13T11:24:37.476 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.89070849200000002 2024-09-13T11:24:37.476 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:37.476 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:37.476 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:24:37.476 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.907101+0000", 2024-09-13T11:24:37.476 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.7749999999999998e-06 2024-09-13T11:24:37.476 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:37.476 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:24:37.476 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:37.477 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:37.477 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:37.477 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:54811 3.1e 3.3754367e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-13T11:24:37.477 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:32.045424+0000", 2024-09-13T11:24:37.477 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 65.383933972999998, 2024-09-13T11:24:37.477 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.86170719399999995, 2024-09-13T11:24:37.477 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:24:37.477 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:24:37.477 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-13T11:24:37.477 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:24:37.477 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:24:37.477 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:37.478 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 54811 2024-09-13T11:24:37.478 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:37.478 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:24:37.478 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:37.478 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:24:37.478 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.045424+0000", 2024-09-13T11:24:37.478 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:37.478 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:37.478 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:37.478 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:24:37.478 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.045424+0000", 2024-09-13T11:24:37.478 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:37.478 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:37.478 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:37.479 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:24:37.479 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.045426+0000", 2024-09-13T11:24:37.479 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.0260000000000002e-06 2024-09-13T11:24:37.479 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:37.479 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:37.479 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:24:37.479 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.045556+0000", 2024-09-13T11:24:37.479 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00013003099999999999 2024-09-13T11:24:37.479 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:37.479 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:37.479 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:24:37.479 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.045568+0000", 2024-09-13T11:24:37.479 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.1727e-05 2024-09-13T11:24:37.479 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:37.480 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:37.480 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:24:37.480 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.045576+0000", 2024-09-13T11:24:37.480 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.6459999999999999e-06 2024-09-13T11:24:37.480 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:37.480 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:37.480 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:24:37.480 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.045650+0000", 2024-09-13T11:24:37.480 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.4000999999999998e-05 2024-09-13T11:24:37.480 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:37.480 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:37.480 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for peered", 2024-09-13T11:24:37.480 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.045658+0000", 2024-09-13T11:24:37.481 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.3869999999999997e-06 2024-09-13T11:24:37.481 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:37.481 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:37.481 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:24:37.481 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.907123+0000", 2024-09-13T11:24:37.481 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.86146508399999999 2024-09-13T11:24:37.481 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:37.481 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:37.481 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:24:37.481 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.907131+0000", 2024-09-13T11:24:37.481 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.2919999999999996e-06 2024-09-13T11:24:37.481 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:37.481 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:24:37.482 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:37.482 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:37.482 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:37.482 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:54817 3.1e 3.36dd2fde (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-13T11:24:37.482 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:32.050108+0000", 2024-09-13T11:24:37.482 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 65.379250080999995, 2024-09-13T11:24:37.482 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.85705549199999997, 2024-09-13T11:24:37.482 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:24:37.482 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:24:37.482 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-13T11:24:37.482 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:24:37.482 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:24:37.482 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:37.483 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 54817 2024-09-13T11:24:37.483 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:37.483 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:24:37.483 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:37.483 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:24:37.483 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.050108+0000", 2024-09-13T11:24:37.483 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:37.483 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:37.483 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:37.483 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:24:37.483 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.050108+0000", 2024-09-13T11:24:37.483 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:37.484 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:37.484 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:37.484 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:24:37.484 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.050110+0000", 2024-09-13T11:24:37.484 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.0209999999999998e-06 2024-09-13T11:24:37.484 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:37.484 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:37.484 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:24:37.484 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.050166+0000", 2024-09-13T11:24:37.484 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.6026000000000002e-05 2024-09-13T11:24:37.484 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:37.484 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:37.484 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:24:37.484 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.050180+0000", 2024-09-13T11:24:37.485 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.4307e-05 2024-09-13T11:24:37.485 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:37.485 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:37.485 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:24:37.485 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.050188+0000", 2024-09-13T11:24:37.485 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.5410000000000003e-06 2024-09-13T11:24:37.485 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:37.485 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:37.485 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:24:37.485 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.050260+0000", 2024-09-13T11:24:37.485 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.2157999999999996e-05 2024-09-13T11:24:37.485 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:37.486 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:37.486 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for peered", 2024-09-13T11:24:37.486 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.050267+0000", 2024-09-13T11:24:37.486 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.0190000000000001e-06 2024-09-13T11:24:37.486 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:37.486 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:37.486 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:24:37.486 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.907155+0000", 2024-09-13T11:24:37.486 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.85688824399999997 2024-09-13T11:24:37.486 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:37.486 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:37.486 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:24:37.487 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.907163+0000", 2024-09-13T11:24:37.487 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.1759999999999993e-06 2024-09-13T11:24:37.487 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:37.487 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:24:37.487 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:37.487 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:37.487 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:37.487 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:54818 3.1e 3.5cfcc69e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-13T11:24:37.487 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:32.051374+0000", 2024-09-13T11:24:37.487 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 65.377983665000002, 2024-09-13T11:24:37.487 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.85581781000000001, 2024-09-13T11:24:37.487 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:24:37.487 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:24:37.488 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-13T11:24:37.488 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:24:37.488 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:24:37.488 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:37.488 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 54818 2024-09-13T11:24:37.488 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:37.488 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:24:37.488 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:37.488 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:24:37.488 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.051374+0000", 2024-09-13T11:24:37.488 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:37.488 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:37.488 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:37.489 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:24:37.489 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.051374+0000", 2024-09-13T11:24:37.489 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:37.489 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:37.489 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:37.489 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:24:37.489 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.051375+0000", 2024-09-13T11:24:37.489 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.4440000000000001e-06 2024-09-13T11:24:37.489 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:37.489 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:37.489 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:24:37.489 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.051426+0000", 2024-09-13T11:24:37.490 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.0809000000000001e-05 2024-09-13T11:24:37.490 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:37.490 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:37.490 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:24:37.490 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.051441+0000", 2024-09-13T11:24:37.490 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.4460999999999999e-05 2024-09-13T11:24:37.490 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:37.490 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:37.490 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:24:37.490 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.051449+0000", 2024-09-13T11:24:37.490 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.4880000000000007e-06 2024-09-13T11:24:37.490 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:37.490 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:37.490 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:24:37.491 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.051499+0000", 2024-09-13T11:24:37.491 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.9691999999999998e-05 2024-09-13T11:24:37.491 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:37.491 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:37.491 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for peered", 2024-09-13T11:24:37.491 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.051505+0000", 2024-09-13T11:24:37.491 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.2029999999999997e-06 2024-09-13T11:24:37.491 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:37.491 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:37.491 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:24:37.491 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.907184+0000", 2024-09-13T11:24:37.491 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.85567906599999999 2024-09-13T11:24:37.491 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:37.492 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:37.492 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:24:37.492 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.907192+0000", 2024-09-13T11:24:37.492 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.6469999999999998e-06 2024-09-13T11:24:37.492 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:37.492 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:24:37.492 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:37.492 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:37.492 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:37.492 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:54828 3.1e 3.e920c11e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-13T11:24:37.492 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:32.063344+0000", 2024-09-13T11:24:37.493 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 65.366013237999994, 2024-09-13T11:24:37.493 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.84387899099999997, 2024-09-13T11:24:37.493 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:24:37.493 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:24:37.493 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-13T11:24:37.493 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:24:37.493 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:24:37.493 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:37.493 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 54828 2024-09-13T11:24:37.493 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:37.493 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:24:37.493 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:37.493 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:24:37.494 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.063344+0000", 2024-09-13T11:24:37.494 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:37.494 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:37.494 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:37.494 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:24:37.494 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.063344+0000", 2024-09-13T11:24:37.494 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:37.494 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:37.494 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:37.494 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:24:37.494 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.063347+0000", 2024-09-13T11:24:37.494 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.2299999999999998e-06 2024-09-13T11:24:37.494 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:37.495 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:37.495 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:24:37.495 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.063384+0000", 2024-09-13T11:24:37.495 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.7290999999999999e-05 2024-09-13T11:24:37.495 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:37.495 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:37.495 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:24:37.495 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.063397+0000", 2024-09-13T11:24:37.495 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2955999999999999e-05 2024-09-13T11:24:37.495 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:37.495 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:37.495 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:24:37.496 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.063404+0000", 2024-09-13T11:24:37.496 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.4889999999999998e-06 2024-09-13T11:24:37.496 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:37.496 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:37.496 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:24:37.496 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.063470+0000", 2024-09-13T11:24:37.496 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.5516000000000001e-05 2024-09-13T11:24:37.496 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:37.496 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:37.496 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for peered", 2024-09-13T11:24:37.496 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.063482+0000", 2024-09-13T11:24:37.496 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2065000000000001e-05 2024-09-13T11:24:37.496 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:37.497 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:37.497 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:24:37.497 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.907216+0000", 2024-09-13T11:24:37.497 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.84373388900000001 2024-09-13T11:24:37.497 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:37.497 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:37.497 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:24:37.497 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.907223+0000", 2024-09-13T11:24:37.497 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.5549999999999997e-06 2024-09-13T11:24:37.497 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:37.497 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:24:37.497 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:37.497 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:37.498 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:37.498 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:54829 3.1e 3.192f62be (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-13T11:24:37.498 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:32.065642+0000", 2024-09-13T11:24:37.498 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 65.363716170000004, 2024-09-13T11:24:37.498 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.84161483100000001, 2024-09-13T11:24:37.498 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:24:37.498 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:24:37.498 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-13T11:24:37.498 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:24:37.498 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:24:37.498 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:37.498 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 54829 2024-09-13T11:24:37.498 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:37.499 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:24:37.499 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:37.499 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:24:37.499 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.065642+0000", 2024-09-13T11:24:37.499 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:37.499 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:37.499 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:37.499 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:24:37.499 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.065642+0000", 2024-09-13T11:24:37.499 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:37.499 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:37.499 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:37.499 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:24:37.500 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.065644+0000", 2024-09-13T11:24:37.500 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.1780000000000002e-06 2024-09-13T11:24:37.500 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:37.500 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:37.500 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:24:37.500 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.065695+0000", 2024-09-13T11:24:37.500 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.1632000000000003e-05 2024-09-13T11:24:37.500 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:37.500 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:37.500 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:24:37.500 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.065711+0000", 2024-09-13T11:24:37.500 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.5522000000000001e-05 2024-09-13T11:24:37.501 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:37.501 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:37.501 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:24:37.501 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.065719+0000", 2024-09-13T11:24:37.501 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.3480000000000004e-06 2024-09-13T11:24:37.501 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:37.501 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:37.501 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:24:37.501 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.065785+0000", 2024-09-13T11:24:37.501 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.5627e-05 2024-09-13T11:24:37.501 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:37.501 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:37.501 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for peered", 2024-09-13T11:24:37.502 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.065797+0000", 2024-09-13T11:24:37.502 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.1766000000000001e-05 2024-09-13T11:24:37.502 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:37.502 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:37.502 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:24:37.502 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.907248+0000", 2024-09-13T11:24:37.502 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.84145150300000005 2024-09-13T11:24:37.502 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:37.502 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:37.502 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:24:37.502 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.907256+0000", 2024-09-13T11:24:37.502 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.2549999999999997e-06 2024-09-13T11:24:37.502 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:37.503 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:24:37.503 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:37.503 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:37.503 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:37.503 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:54830 3.1e 3.b31554de (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-13T11:24:37.503 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:32.068259+0000", 2024-09-13T11:24:37.503 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 65.361098302000002, 2024-09-13T11:24:37.503 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.83902489599999996, 2024-09-13T11:24:37.503 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:24:37.503 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:24:37.503 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-13T11:24:37.503 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:24:37.503 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:24:37.504 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:37.504 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 54830 2024-09-13T11:24:37.504 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:37.504 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:24:37.504 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:37.504 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:24:37.504 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.068259+0000", 2024-09-13T11:24:37.504 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:37.504 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:37.504 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:37.504 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:24:37.504 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.068259+0000", 2024-09-13T11:24:37.505 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:37.505 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:37.505 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:37.505 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:24:37.505 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.068262+0000", 2024-09-13T11:24:37.505 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.1799999999999999e-06 2024-09-13T11:24:37.505 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:37.505 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:37.505 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:24:37.505 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.068301+0000", 2024-09-13T11:24:37.505 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.9289999999999998e-05 2024-09-13T11:24:37.505 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:37.506 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:37.506 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:24:37.506 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.068314+0000", 2024-09-13T11:24:37.506 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.3502999999999999e-05 2024-09-13T11:24:37.506 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:37.506 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:37.506 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:24:37.506 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.068324+0000", 2024-09-13T11:24:37.506 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.1630000000000002e-06 2024-09-13T11:24:37.506 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:37.506 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:37.506 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:24:37.507 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.068375+0000", 2024-09-13T11:24:37.507 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.1805000000000003e-05 2024-09-13T11:24:37.507 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:37.507 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:37.507 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for peered", 2024-09-13T11:24:37.507 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.068382+0000", 2024-09-13T11:24:37.507 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.224e-06 2024-09-13T11:24:37.507 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:37.507 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:37.507 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:24:37.507 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.907277+0000", 2024-09-13T11:24:37.507 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.83889509500000004 2024-09-13T11:24:37.508 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:37.508 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:37.508 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:24:37.508 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.907284+0000", 2024-09-13T11:24:37.508 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.6359999999999994e-06 2024-09-13T11:24:37.508 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:37.508 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:24:37.508 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:37.508 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:37.508 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:37.508 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:54846 3.1e 3.742e049e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-13T11:24:37.508 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:32.153567+0000", 2024-09-13T11:24:37.508 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 65.275791077999997, 2024-09-13T11:24:37.509 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.75378642399999995, 2024-09-13T11:24:37.509 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:24:37.509 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:24:37.509 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-13T11:24:37.509 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:24:37.509 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:24:37.509 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:37.509 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 54846 2024-09-13T11:24:37.509 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:37.509 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:24:37.509 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:37.510 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:24:37.510 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.153567+0000", 2024-09-13T11:24:37.510 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:37.510 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:37.510 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:37.510 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:24:37.510 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.153567+0000", 2024-09-13T11:24:37.510 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:37.510 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:37.510 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:37.510 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:24:37.510 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.153569+0000", 2024-09-13T11:24:37.510 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.4660000000000002e-06 2024-09-13T11:24:37.511 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:37.511 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:37.511 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:24:37.511 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.153619+0000", 2024-09-13T11:24:37.511 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.9617000000000003e-05 2024-09-13T11:24:37.511 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:37.511 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:37.511 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:24:37.511 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.153634+0000", 2024-09-13T11:24:37.511 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.4904e-05 2024-09-13T11:24:37.511 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:37.512 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:37.512 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:24:37.512 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.153642+0000", 2024-09-13T11:24:37.512 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.5599999999999994e-06 2024-09-13T11:24:37.512 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:37.512 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:37.512 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:24:37.512 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.153721+0000", 2024-09-13T11:24:37.512 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.8603999999999995e-05 2024-09-13T11:24:37.512 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:37.512 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:37.512 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for peered", 2024-09-13T11:24:37.512 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.153730+0000", 2024-09-13T11:24:37.513 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.3840000000000003e-06 2024-09-13T11:24:37.513 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:37.513 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:37.513 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:24:37.513 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.907346+0000", 2024-09-13T11:24:37.513 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.75361563099999995 2024-09-13T11:24:37.513 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:37.513 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:37.513 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:24:37.513 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.907353+0000", 2024-09-13T11:24:37.513 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.2579999999999998e-06 2024-09-13T11:24:37.513 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:37.513 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:24:37.513 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:37.514 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:37.514 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:37.514 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:54888 3.1e 3.43e14ede (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-13T11:24:37.514 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:32.423064+0000", 2024-09-13T11:24:37.514 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 65.006293407000001, 2024-09-13T11:24:37.514 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.48433585299999998, 2024-09-13T11:24:37.514 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:24:37.514 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:24:37.514 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-13T11:24:37.514 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:24:37.514 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:24:37.514 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:37.515 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 54888 2024-09-13T11:24:37.515 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:37.515 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:24:37.515 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:37.515 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:24:37.515 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.423064+0000", 2024-09-13T11:24:37.515 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:37.515 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:37.515 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:37.515 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:24:37.515 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.423064+0000", 2024-09-13T11:24:37.515 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:37.515 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:37.516 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:37.516 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:24:37.516 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.423069+0000", 2024-09-13T11:24:37.516 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.7389999999999999e-06 2024-09-13T11:24:37.516 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:37.516 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:37.516 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:24:37.516 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.423115+0000", 2024-09-13T11:24:37.516 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.5847000000000003e-05 2024-09-13T11:24:37.516 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:37.516 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:37.516 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:24:37.516 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.423126+0000", 2024-09-13T11:24:37.517 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.1418e-05 2024-09-13T11:24:37.517 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:37.517 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:37.517 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:24:37.517 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.423132+0000", 2024-09-13T11:24:37.517 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.57e-06 2024-09-13T11:24:37.517 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:37.517 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:37.517 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:24:37.517 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.423191+0000", 2024-09-13T11:24:37.517 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.9250999999999999e-05 2024-09-13T11:24:37.517 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:37.518 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:37.518 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for peered", 2024-09-13T11:24:37.518 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.423198+0000", 2024-09-13T11:24:37.518 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.9179999999999997e-06 2024-09-13T11:24:37.518 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:37.518 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:37.518 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:24:37.518 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.907392+0000", 2024-09-13T11:24:37.518 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.48419437500000001 2024-09-13T11:24:37.518 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:37.518 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:37.518 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:24:37.518 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.907400+0000", 2024-09-13T11:24:37.518 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.7349999999999996e-06 2024-09-13T11:24:37.519 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:37.519 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:24:37.519 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:37.519 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:37.519 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:37.519 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDPGPushReply(3.17 76/70 [PushReplyOp(3:eb8d2fae:::benchmark_data_smithi177_31116_object18267:head)])", 2024-09-13T11:24:37.519 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:32.636274+0000", 2024-09-13T11:24:37.519 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 64.793083813999999, 2024-09-13T11:24:37.519 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.269081036, 2024-09-13T11:24:37.519 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:24:37.519 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:24:37.519 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-13T11:24:37.519 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:24:37.520 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:37.520 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:24:37.520 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.636274+0000", 2024-09-13T11:24:37.520 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:37.520 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:37.520 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:37.520 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:24:37.520 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.636274+0000", 2024-09-13T11:24:37.520 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:37.520 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:37.520 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:37.520 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:24:37.520 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.636276+0000", 2024-09-13T11:24:37.521 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.1679999999999998e-06 2024-09-13T11:24:37.521 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:37.521 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:37.521 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:24:37.521 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.636284+0000", 2024-09-13T11:24:37.521 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.7440000000000007e-06 2024-09-13T11:24:37.521 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:37.521 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:37.521 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:24:37.521 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.904155+0000", 2024-09-13T11:24:37.521 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.26787080800000002 2024-09-13T11:24:37.521 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:37.522 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:37.522 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:24:37.522 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.904165+0000", 2024-09-13T11:24:37.522 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0750999999999999e-05 2024-09-13T11:24:37.522 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:37.522 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:37.522 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:24:37.522 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.904238+0000", 2024-09-13T11:24:37.522 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.2335000000000005e-05 2024-09-13T11:24:37.522 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:37.522 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:37.522 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:24:37.522 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.905355+0000", 2024-09-13T11:24:37.523 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0011172300000000001 2024-09-13T11:24:37.523 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:37.523 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:24:37.523 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:37.523 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:37.523 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:37.523 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDPGPushReply(3.17 76/70 [PushReplyOp(3:ebcbce67:::benchmark_data_smithi177_31116_object18268:head)])", 2024-09-13T11:24:37.523 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:32.641093+0000", 2024-09-13T11:24:37.523 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 64.788264265999999, 2024-09-13T11:24:37.523 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.27471406799999998, 2024-09-13T11:24:37.523 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:24:37.523 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:24:37.523 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-13T11:24:37.523 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:24:37.524 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:37.524 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:24:37.524 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.641093+0000", 2024-09-13T11:24:37.524 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:37.524 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:37.524 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:37.524 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:24:37.524 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.641093+0000", 2024-09-13T11:24:37.524 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:37.524 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:37.524 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:37.524 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:24:37.524 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.641094+0000", 2024-09-13T11:24:37.525 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.0299999999999999e-07 2024-09-13T11:24:37.525 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:37.525 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:37.525 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:24:37.525 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.641098+0000", 2024-09-13T11:24:37.525 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.2200000000000003e-06 2024-09-13T11:24:37.525 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:37.525 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:37.525 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:24:37.525 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.910434+0000", 2024-09-13T11:24:37.525 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.26933570600000001 2024-09-13T11:24:37.526 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:37.526 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:37.526 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:24:37.526 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.910442+0000", 2024-09-13T11:24:37.526 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.2700000000000004e-06 2024-09-13T11:24:37.526 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:37.526 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:37.526 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:24:37.526 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.914989+0000", 2024-09-13T11:24:37.526 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0045465059999999996 2024-09-13T11:24:37.526 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:37.526 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:37.526 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:24:37.526 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.915808+0000", 2024-09-13T11:24:37.527 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00081876300000000002 2024-09-13T11:24:37.527 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:37.527 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:24:37.527 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:37.527 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:37.527 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:37.527 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4665.0:59977 3.21 e83/78 3:86b4d8f6:::benchmark_data_smithi177_31116_object59976:head v 83'3136, mlcod=0'0)", 2024-09-13T11:24:37.527 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:40.571147+0000", 2024-09-13T11:24:37.527 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 56.858210958000001, 2024-09-13T11:24:37.527 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.51907542799999995, 2024-09-13T11:24:37.527 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:24:37.527 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:24:37.527 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:37.527 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:24:37.528 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:37.528 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:24:37.528 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:40.571147+0000", 2024-09-13T11:24:37.528 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:37.528 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:37.528 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:37.528 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:24:37.528 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:40.571147+0000", 2024-09-13T11:24:37.528 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:37.528 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:37.528 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:37.528 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:24:37.528 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:40.571151+0000", 2024-09-13T11:24:37.529 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.5650000000000003e-06 2024-09-13T11:24:37.529 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:37.529 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:37.529 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:24:37.529 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:40.571204+0000", 2024-09-13T11:24:37.529 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.2720999999999999e-05 2024-09-13T11:24:37.529 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:37.529 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:37.529 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:24:37.529 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:41.088332+0000", 2024-09-13T11:24:37.529 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.51712822000000003 2024-09-13T11:24:37.529 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:37.529 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:37.530 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:24:37.530 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:41.088341+0000", 2024-09-13T11:24:37.530 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.3100000000000001e-06 2024-09-13T11:24:37.530 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:37.530 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:37.530 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:24:37.530 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:41.088444+0000", 2024-09-13T11:24:37.530 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000103708 2024-09-13T11:24:37.530 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:37.530 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:37.530 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:24:37.530 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:41.088475+0000", 2024-09-13T11:24:37.530 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.0497e-05 2024-09-13T11:24:37.531 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:37.531 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:37.531 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:24:37.531 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:41.090192+0000", 2024-09-13T11:24:37.531 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.001717288 2024-09-13T11:24:37.531 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:37.531 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:37.531 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:24:37.531 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:41.090222+0000", 2024-09-13T11:24:37.531 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.0119000000000001e-05 2024-09-13T11:24:37.531 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:37.531 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:24:37.531 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:37.531 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:37.532 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:37.532 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:110286 3.26 3.c41c0fa6 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e119)", 2024-09-13T11:24:37.532 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:24:25.654272+0000", 2024-09-13T11:24:37.532 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 11.775085266, 2024-09-13T11:24:37.532 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.43337235400000002, 2024-09-13T11:24:37.532 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:24:37.532 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:24:37.532 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-13T11:24:37.532 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:24:37.532 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:24:37.532 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:37.532 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 110286 2024-09-13T11:24:37.532 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:37.532 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:24:37.533 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:37.533 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:24:37.533 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:25.654272+0000", 2024-09-13T11:24:37.533 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:37.533 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:37.533 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:37.533 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:24:37.533 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:25.654272+0000", 2024-09-13T11:24:37.533 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:37.533 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:37.533 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:37.533 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:24:37.533 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:25.654275+0000", 2024-09-13T11:24:37.533 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.3199999999999998e-06 2024-09-13T11:24:37.534 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:37.534 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:37.534 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:24:37.534 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:25.654517+0000", 2024-09-13T11:24:37.534 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00024270199999999999 2024-09-13T11:24:37.534 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:37.534 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:37.534 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:24:37.534 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:25.654550+0000", 2024-09-13T11:24:37.534 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.2549000000000003e-05 2024-09-13T11:24:37.534 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:37.534 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:37.534 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:24:37.535 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:25.654566+0000", 2024-09-13T11:24:37.535 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.5517000000000001e-05 2024-09-13T11:24:37.535 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:37.535 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:37.535 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:24:37.535 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:25.654655+0000", 2024-09-13T11:24:37.535 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.9961000000000001e-05 2024-09-13T11:24:37.535 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:37.535 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:37.535 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for peered", 2024-09-13T11:24:37.535 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:25.654665+0000", 2024-09-13T11:24:37.535 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.0520000000000007e-06 2024-09-13T11:24:37.535 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:37.535 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:37.536 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:24:37.536 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:26.087629+0000", 2024-09-13T11:24:37.536 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.43296476900000003 2024-09-13T11:24:37.536 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:37.536 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:37.536 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:24:37.536 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:26.087645+0000", 2024-09-13T11:24:37.536 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.5483999999999999e-05 2024-09-13T11:24:37.536 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:37.536 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:24:37.536 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:37.536 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:37.537 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:24:37.537 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-13T11:24:37.537 INFO:tasks.ceph.osd.4.smithi001.stderr:2024-09-13T11:24:37.427+0000 7ff8e0218640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26310) UID: 0 2024-09-13T11:24:37.538 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_ops_in_flight 2024-09-13T11:24:37.539 INFO:tasks.ceph.osd.5.smithi092.stderr:2024-09-13T11:24:37.534+0000 7f6f0b705640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26346) UID: 0 2024-09-13T11:24:37.640 INFO:tasks.ceph.osd.9.smithi092.stderr:2024-09-13T11:24:37.634+0000 7ff5ab3d0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26348) UID: 0 2024-09-13T11:24:37.738 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-13T11:24:37.739 INFO:teuthology.orchestra.run.smithi001.stdout: "ops": [ 2024-09-13T11:24:37.739 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:37.739 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:119872 3.1b 3:d94cc0af:::benchmark_data_smithi177_31116_object119871:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e128)", 2024-09-13T11:24:37.739 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:24:37.713672+0000", 2024-09-13T11:24:37.739 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 0.024491883999999998, 2024-09-13T11:24:37.739 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.024521686000000001, 2024-09-13T11:24:37.739 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:24:37.739 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:24:37.739 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "waiting for sub ops", 2024-09-13T11:24:37.739 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:24:37.739 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:24:37.740 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:37.740 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 119872 2024-09-13T11:24:37.740 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:37.740 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:24:37.740 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:37.740 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:24:37.740 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:37.713672+0000", 2024-09-13T11:24:37.740 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:37.740 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:37.740 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:37.740 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:24:37.740 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:37.713672+0000", 2024-09-13T11:24:37.740 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:37.741 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:37.741 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:37.741 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:24:37.741 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:37.713674+0000", 2024-09-13T11:24:37.741 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.4569999999999999e-06 2024-09-13T11:24:37.741 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:37.741 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:37.741 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:24:37.741 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:37.713740+0000", 2024-09-13T11:24:37.741 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.5897999999999998e-05 2024-09-13T11:24:37.741 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:37.741 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:37.741 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:24:37.741 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:37.713757+0000", 2024-09-13T11:24:37.742 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.6449000000000001e-05 2024-09-13T11:24:37.742 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:37.742 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:37.742 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:24:37.742 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:37.713766+0000", 2024-09-13T11:24:37.742 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.5440000000000004e-06 2024-09-13T11:24:37.742 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:37.742 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:37.742 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:24:37.742 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:37.713840+0000", 2024-09-13T11:24:37.742 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.3999999999999996e-05 2024-09-13T11:24:37.742 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:37.742 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:37.743 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:24:37.743 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:37.714008+0000", 2024-09-13T11:24:37.743 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00016786899999999999 2024-09-13T11:24:37.743 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:37.743 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:37.743 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 15", 2024-09-13T11:24:37.743 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:37.714173+0000", 2024-09-13T11:24:37.743 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000164542 2024-09-13T11:24:37.743 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:37.743 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:37.743 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:24:37.743 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:37.716367+0000", 2024-09-13T11:24:37.743 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0021947329999999999 2024-09-13T11:24:37.744 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:37.744 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:24:37.744 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:37.744 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:37.744 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:37.744 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:119903 3.0 3:02ef0d56:::benchmark_data_smithi177_31116_object119902:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e128)", 2024-09-13T11:24:37.744 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:24:37.737054+0000", 2024-09-13T11:24:37.744 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 0.0011097749999999999, 2024-09-13T11:24:37.744 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0012309249999999999, 2024-09-13T11:24:37.744 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:24:37.744 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:24:37.744 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "waiting for sub ops", 2024-09-13T11:24:37.744 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:24:37.745 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:24:37.745 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:37.745 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 119903 2024-09-13T11:24:37.745 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:37.745 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:24:37.745 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:37.745 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:24:37.745 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:37.737054+0000", 2024-09-13T11:24:37.745 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:37.745 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:37.745 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:37.745 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:24:37.745 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:37.737054+0000", 2024-09-13T11:24:37.745 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:37.746 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:37.746 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:37.746 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:24:37.746 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:37.737056+0000", 2024-09-13T11:24:37.746 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.0109999999999999e-06 2024-09-13T11:24:37.746 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:37.746 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:37.746 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:24:37.746 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:37.737145+0000", 2024-09-13T11:24:37.746 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.9420999999999996e-05 2024-09-13T11:24:37.746 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:37.746 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:37.746 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:24:37.747 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:37.737163+0000", 2024-09-13T11:24:37.747 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.7209000000000001e-05 2024-09-13T11:24:37.747 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:37.747 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:37.747 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:24:37.747 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:37.737172+0000", 2024-09-13T11:24:37.747 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.3780000000000007e-06 2024-09-13T11:24:37.747 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:37.747 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:37.747 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:24:37.747 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:37.737224+0000", 2024-09-13T11:24:37.747 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.2222000000000002e-05 2024-09-13T11:24:37.747 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:37.747 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:37.748 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:24:37.748 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:37.737324+0000", 2024-09-13T11:24:37.748 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000100159 2024-09-13T11:24:37.748 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:37.748 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:37.748 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 15", 2024-09-13T11:24:37.748 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:37.737411+0000", 2024-09-13T11:24:37.748 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.6431e-05 2024-09-13T11:24:37.748 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:37.748 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:24:37.748 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:37.748 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:37.748 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:37.749 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDPGPushReply(3.2 128/122 [PushReplyOp(3:4219f51e:::benchmark_data_smithi177_31116_object41894:head)])", 2024-09-13T11:24:37.749 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:24:37.737626+0000", 2024-09-13T11:24:37.749 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 0.00053824999999999999, 2024-09-13T11:24:37.749 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00072979800000000004, 2024-09-13T11:24:37.749 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:24:37.749 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:24:37.749 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-13T11:24:37.749 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:24:37.749 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:37.749 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:24:37.749 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:37.737626+0000", 2024-09-13T11:24:37.749 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:37.749 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:37.749 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:37.749 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:24:37.750 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:37.737626+0000", 2024-09-13T11:24:37.750 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:37.750 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:37.750 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:37.750 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:24:37.750 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:37.737626+0000", 2024-09-13T11:24:37.750 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.9999999999999996e-07 2024-09-13T11:24:37.750 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:37.750 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:37.750 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:24:37.750 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:37.737631+0000", 2024-09-13T11:24:37.750 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.4150000000000001e-06 2024-09-13T11:24:37.750 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:37.751 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:37.751 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:24:37.751 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:37.737651+0000", 2024-09-13T11:24:37.751 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.0398e-05 2024-09-13T11:24:37.751 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:37.751 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:37.751 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:24:37.751 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:37.737657+0000", 2024-09-13T11:24:37.751 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.3419999999999998e-06 2024-09-13T11:24:37.751 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:37.751 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:37.751 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:24:37.751 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:37.737702+0000", 2024-09-13T11:24:37.751 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.5824e-05 2024-09-13T11:24:37.752 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:37.752 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:24:37.752 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:37.752 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:37.752 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-09-13T11:24:37.752 INFO:teuthology.orchestra.run.smithi001.stdout: "num_ops": 3 2024-09-13T11:24:37.752 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-13T11:24:37.753 INFO:tasks.ceph.osd.10.smithi162.stderr:2024-09-13T11:24:37.748+0000 7fb59c9f0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26317) UID: 0 2024-09-13T11:24:37.759 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_blocked_ops 2024-09-13T11:24:37.854 INFO:tasks.ceph.osd.4.smithi001.stderr:2024-09-13T11:24:37.851+0000 7ff8e0218640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26310) UID: 0 2024-09-13T11:24:37.911 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-13T11:24:37.911 INFO:teuthology.orchestra.run.smithi001.stdout: "ops": [], 2024-09-13T11:24:37.911 INFO:teuthology.orchestra.run.smithi001.stdout: "complaint_time": 30, 2024-09-13T11:24:37.911 INFO:teuthology.orchestra.run.smithi001.stdout: "num_blocked_ops": 0 2024-09-13T11:24:37.911 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-13T11:24:37.931 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_historic_ops 2024-09-13T11:24:37.972 INFO:tasks.ceph.osd.12.smithi001.stderr:2024-09-13T11:24:37.971+0000 7f892c6e6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26314) UID: 0 2024-09-13T11:24:38.012 INFO:tasks.radosbench.radosbench.0.smithi177.stdout: 131 16 120243 120227 57.3539 72.3125 0.00897572 0.0174303 2024-09-13T11:24:38.056 INFO:tasks.ceph.osd.11.smithi177.stderr:2024-09-13T11:24:38.055+0000 7f24792c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26580) UID: 0 2024-09-13T11:24:38.125 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-13T11:24:38.125 INFO:teuthology.orchestra.run.smithi001.stdout: "size": 20, 2024-09-13T11:24:38.125 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 600, 2024-09-13T11:24:38.125 INFO:teuthology.orchestra.run.smithi001.stdout: "ops": [ 2024-09-13T11:24:38.126 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:38.126 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop_reply(client.4665.0:167 3.9 e41/30 ondisk, result = 0)", 2024-09-13T11:24:38.126 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:22:47.419696+0000", 2024-09-13T11:24:38.126 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 110.70277966499999, 2024-09-13T11:24:38.126 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.64615778199999996, 2024-09-13T11:24:38.126 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:24:38.126 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:24:38.126 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:24:38.126 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:24:38.126 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:38.126 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:24:38.126 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:47.419696+0000", 2024-09-13T11:24:38.126 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:38.127 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:38.127 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:38.127 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:24:38.127 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:47.419696+0000", 2024-09-13T11:24:38.127 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:38.127 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:38.127 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:38.127 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:24:38.127 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:47.419697+0000", 2024-09-13T11:24:38.127 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.15e-06 2024-09-13T11:24:38.127 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:38.127 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:38.127 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:24:38.128 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:47.419700+0000", 2024-09-13T11:24:38.128 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.2449999999999998e-06 2024-09-13T11:24:38.128 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:38.128 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:38.128 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:24:38.128 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:48.065637+0000", 2024-09-13T11:24:38.128 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.64593674300000004 2024-09-13T11:24:38.128 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:38.128 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:38.128 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:24:38.128 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:48.065648+0000", 2024-09-13T11:24:38.128 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0865e-05 2024-09-13T11:24:38.128 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:38.129 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:38.129 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:24:38.129 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:48.065720+0000", 2024-09-13T11:24:38.129 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.2020999999999994e-05 2024-09-13T11:24:38.129 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:38.129 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:38.129 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:24:38.129 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:48.065728+0000", 2024-09-13T11:24:38.129 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.7660000000000004e-06 2024-09-13T11:24:38.129 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:38.129 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:38.129 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:24:38.130 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:48.065853+0000", 2024-09-13T11:24:38.130 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00012499200000000001 2024-09-13T11:24:38.130 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:38.130 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:24:38.130 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:38.130 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:38.130 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:38.130 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:25383 3.0 3:0a6bb869:::benchmark_data_smithi177_31116_object25382:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e55)", 2024-09-13T11:24:38.130 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:04.447376+0000", 2024-09-13T11:24:38.130 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 93.675098996000003, 2024-09-13T11:24:38.130 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.21239864, 2024-09-13T11:24:38.130 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:24:38.130 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:24:38.131 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:38.131 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:24:38.131 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:24:38.131 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:38.131 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 25383 2024-09-13T11:24:38.131 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:38.131 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:24:38.131 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:38.131 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:24:38.131 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.447376+0000", 2024-09-13T11:24:38.131 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:38.131 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:38.131 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:38.132 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:24:38.132 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.447376+0000", 2024-09-13T11:24:38.132 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:38.132 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:38.132 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:38.132 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:24:38.133 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.447378+0000", 2024-09-13T11:24:38.133 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.6729999999999999e-06 2024-09-13T11:24:38.133 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:38.133 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:38.133 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:24:38.133 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.447433+0000", 2024-09-13T11:24:38.133 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.5371000000000002e-05 2024-09-13T11:24:38.133 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:38.133 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:38.133 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:24:38.133 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.447446+0000", 2024-09-13T11:24:38.133 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.3179999999999999e-05 2024-09-13T11:24:38.134 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:38.134 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:38.134 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:24:38.134 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.447456+0000", 2024-09-13T11:24:38.134 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0008e-05 2024-09-13T11:24:38.134 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:38.134 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:38.134 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:24:38.134 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.447514+0000", 2024-09-13T11:24:38.134 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.8023999999999999e-05 2024-09-13T11:24:38.134 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:38.134 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:38.134 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:24:38.135 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.447642+0000", 2024-09-13T11:24:38.135 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000127296 2024-09-13T11:24:38.135 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:38.135 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:38.135 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 15", 2024-09-13T11:24:38.135 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.447767+0000", 2024-09-13T11:24:38.135 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000125547 2024-09-13T11:24:38.136 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:38.136 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:38.136 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:24:38.136 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.449188+0000", 2024-09-13T11:24:38.136 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.001420604 2024-09-13T11:24:38.136 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:38.136 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:38.136 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:38.137 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.659668+0000", 2024-09-13T11:24:38.137 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.210479685 2024-09-13T11:24:38.137 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:38.137 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:38.137 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:24:38.137 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.659716+0000", 2024-09-13T11:24:38.137 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.8609999999999997e-05 2024-09-13T11:24:38.137 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:38.137 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:38.137 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:24:38.137 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.659775+0000", 2024-09-13T11:24:38.137 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.8641999999999999e-05 2024-09-13T11:24:38.137 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:38.138 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:24:38.138 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:38.138 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:38.138 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:38.138 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4665.0:25408 3.4 e56/40 3:294e23bd:::benchmark_data_smithi177_31116_object25407:head v 56'1645, mlcod=55'1642)", 2024-09-13T11:24:38.138 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:04.494686+0000", 2024-09-13T11:24:38.138 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 93.627789336999996, 2024-09-13T11:24:38.138 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.29725566399999998, 2024-09-13T11:24:38.138 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:24:38.138 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:24:38.138 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:38.138 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:24:38.138 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:38.139 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:24:38.139 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.494686+0000", 2024-09-13T11:24:38.139 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:38.139 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:38.139 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:38.139 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:24:38.139 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.494686+0000", 2024-09-13T11:24:38.139 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:38.139 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:38.139 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:38.139 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:24:38.139 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.494686+0000", 2024-09-13T11:24:38.139 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.2399999999999999e-07 2024-09-13T11:24:38.139 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:38.140 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:38.140 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:24:38.140 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.494809+0000", 2024-09-13T11:24:38.140 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00012308600000000001 2024-09-13T11:24:38.140 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:38.140 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:38.140 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:24:38.140 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.789410+0000", 2024-09-13T11:24:38.140 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.29460066899999998 2024-09-13T11:24:38.140 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:38.140 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:38.140 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:24:38.140 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.789425+0000", 2024-09-13T11:24:38.141 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.4600000000000001e-05 2024-09-13T11:24:38.141 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:38.141 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:38.141 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:24:38.141 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.789507+0000", 2024-09-13T11:24:38.141 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.2785000000000001e-05 2024-09-13T11:24:38.141 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:38.141 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:38.141 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:24:38.141 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.789525+0000", 2024-09-13T11:24:38.141 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.8117999999999998e-05 2024-09-13T11:24:38.141 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:38.141 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:38.142 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:24:38.142 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.791852+0000", 2024-09-13T11:24:38.142 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0023264459999999998 2024-09-13T11:24:38.142 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:38.142 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:38.142 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:24:38.142 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.791942+0000", 2024-09-13T11:24:38.142 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.9635999999999994e-05 2024-09-13T11:24:38.142 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:38.142 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:24:38.142 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:38.142 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:38.143 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:38.143 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4665.0:30815 3.f e59/30 3:fde17cb1:::benchmark_data_smithi177_31116_object30814:head v 59'1919, mlcod=59'1914)", 2024-09-13T11:24:38.143 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:09.494342+0000", 2024-09-13T11:24:38.143 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 88.628133351000002, 2024-09-13T11:24:38.143 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.22787855700000001, 2024-09-13T11:24:38.143 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:24:38.143 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:24:38.143 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:38.143 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:24:38.143 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:38.143 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:24:38.143 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.494342+0000", 2024-09-13T11:24:38.143 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:38.144 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:38.144 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:38.144 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:24:38.144 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.494342+0000", 2024-09-13T11:24:38.144 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:38.144 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:38.144 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:38.144 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:24:38.144 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.494343+0000", 2024-09-13T11:24:38.144 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.251e-06 2024-09-13T11:24:38.144 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:38.144 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:38.144 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:24:38.145 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.494387+0000", 2024-09-13T11:24:38.145 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.4276000000000001e-05 2024-09-13T11:24:38.145 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:38.145 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:38.145 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:24:38.145 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.720454+0000", 2024-09-13T11:24:38.145 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.226066614 2024-09-13T11:24:38.145 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:38.145 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:38.145 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:24:38.145 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.720462+0000", 2024-09-13T11:24:38.145 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.9240000000000007e-06 2024-09-13T11:24:38.146 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:38.146 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:38.146 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:24:38.146 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.720537+0000", 2024-09-13T11:24:38.146 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.4967000000000002e-05 2024-09-13T11:24:38.146 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:38.146 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:38.146 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:24:38.146 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.720563+0000", 2024-09-13T11:24:38.146 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.5630999999999999e-05 2024-09-13T11:24:38.146 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:38.146 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:38.146 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:24:38.147 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.722183+0000", 2024-09-13T11:24:38.147 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0016209670000000001 2024-09-13T11:24:38.147 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:38.147 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:38.147 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:24:38.147 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.722220+0000", 2024-09-13T11:24:38.147 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.6927000000000001e-05 2024-09-13T11:24:38.147 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:38.147 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:24:38.147 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:38.147 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:38.147 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:38.147 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4665.0:30828 3.f e59/30 3:f3a9c6a9:::benchmark_data_smithi177_31116_object30827:head v 59'1920, mlcod=59'1916)", 2024-09-13T11:24:38.148 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:09.502607+0000", 2024-09-13T11:24:38.148 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 88.619868294, 2024-09-13T11:24:38.148 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.26374408300000002, 2024-09-13T11:24:38.148 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:24:38.148 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:24:38.148 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:38.148 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:24:38.148 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:38.148 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:24:38.148 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.502607+0000", 2024-09-13T11:24:38.148 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:38.148 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:38.149 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:38.149 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:24:38.149 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.502607+0000", 2024-09-13T11:24:38.149 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:38.149 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:38.149 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:38.149 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:24:38.149 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.502609+0000", 2024-09-13T11:24:38.149 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.84e-06 2024-09-13T11:24:38.149 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:38.149 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:38.149 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:24:38.149 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.502663+0000", 2024-09-13T11:24:38.150 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.3971000000000002e-05 2024-09-13T11:24:38.150 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:38.150 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:38.150 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:24:38.150 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.763912+0000", 2024-09-13T11:24:38.150 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.26124904300000001 2024-09-13T11:24:38.150 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:38.150 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:38.150 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:24:38.150 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.763920+0000", 2024-09-13T11:24:38.150 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.1149999999999994e-06 2024-09-13T11:24:38.150 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:38.151 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:38.151 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:24:38.151 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.764024+0000", 2024-09-13T11:24:38.151 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00010380899999999999 2024-09-13T11:24:38.151 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:38.151 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:38.151 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:24:38.151 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.764068+0000", 2024-09-13T11:24:38.151 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.4638999999999997e-05 2024-09-13T11:24:38.151 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:38.151 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:38.151 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:24:38.151 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.766308+0000", 2024-09-13T11:24:38.152 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0022392039999999998 2024-09-13T11:24:38.152 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:38.152 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:38.152 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:24:38.152 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.766351+0000", 2024-09-13T11:24:38.152 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.3461999999999998e-05 2024-09-13T11:24:38.152 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:38.152 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:24:38.152 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:38.152 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:38.152 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:38.152 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:54351 3.10 3:085f283e:::benchmark_data_smithi177_31116_object54350:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e74)", 2024-09-13T11:24:38.153 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:30.599125+0000", 2024-09-13T11:24:38.153 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 67.523350589000003, 2024-09-13T11:24:38.153 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.167573261, 2024-09-13T11:24:38.153 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:24:38.153 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:24:38.153 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:38.153 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:24:38.153 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:24:38.153 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:38.153 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 54351 2024-09-13T11:24:38.153 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:38.153 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:24:38.153 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:38.154 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:24:38.154 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:30.599125+0000", 2024-09-13T11:24:38.154 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:38.154 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:38.154 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:38.154 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:24:38.154 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:30.599125+0000", 2024-09-13T11:24:38.154 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:38.154 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:38.154 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:38.154 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:24:38.154 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:30.599127+0000", 2024-09-13T11:24:38.154 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.847e-06 2024-09-13T11:24:38.155 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:38.155 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:38.155 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:24:38.155 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:30.599164+0000", 2024-09-13T11:24:38.155 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.6758999999999998e-05 2024-09-13T11:24:38.155 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:38.155 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:38.155 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:24:38.155 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:30.599184+0000", 2024-09-13T11:24:38.155 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.9928e-05 2024-09-13T11:24:38.155 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:38.155 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:38.155 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:24:38.155 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:30.599194+0000", 2024-09-13T11:24:38.156 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0035e-05 2024-09-13T11:24:38.156 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:38.156 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:38.156 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:24:38.156 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:30.599259+0000", 2024-09-13T11:24:38.156 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.5147000000000005e-05 2024-09-13T11:24:38.156 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:38.156 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:38.156 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:24:38.156 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:30.599381+0000", 2024-09-13T11:24:38.156 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000121876 2024-09-13T11:24:38.156 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:38.156 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:38.157 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 0,6", 2024-09-13T11:24:38.157 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:30.599496+0000", 2024-09-13T11:24:38.157 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000114307 2024-09-13T11:24:38.157 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:38.157 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:38.157 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:24:38.157 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:30.601094+0000", 2024-09-13T11:24:38.157 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.001598857 2024-09-13T11:24:38.157 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:38.157 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:38.157 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:38.157 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:30.601896+0000", 2024-09-13T11:24:38.157 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00080183800000000003 2024-09-13T11:24:38.158 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:38.158 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:38.158 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:38.158 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:30.766040+0000", 2024-09-13T11:24:38.158 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.16414414099999999 2024-09-13T11:24:38.158 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:38.158 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:38.158 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:24:38.158 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:30.766092+0000", 2024-09-13T11:24:38.158 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.1230000000000002e-05 2024-09-13T11:24:38.158 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:38.158 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:38.158 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:24:38.159 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:30.766698+0000", 2024-09-13T11:24:38.159 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00060629599999999998 2024-09-13T11:24:38.159 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:38.159 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:24:38.159 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:38.159 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:38.159 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:38.159 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDPGPush(3.1b 84/78 [PushOp(3:dcddd139:::benchmark_data_smithi177_31116_object10384:head, version: 48'673, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:dcddd139:::benchmark_data_smithi177_31116_object10384:head@48'673, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:24:38.159 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:41.400602+0000", 2024-09-13T11:24:38.159 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 56.721872808999997, 2024-09-13T11:24:38.159 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.43365821999999998, 2024-09-13T11:24:38.159 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:24:38.159 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:24:38.160 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:24:38.160 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:24:38.160 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:38.160 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:24:38.160 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:41.400602+0000", 2024-09-13T11:24:38.160 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:38.160 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:38.160 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:38.160 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:24:38.160 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:41.400602+0000", 2024-09-13T11:24:38.160 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:38.160 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:38.161 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:38.161 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:24:38.161 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:41.400604+0000", 2024-09-13T11:24:38.161 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.99e-06 2024-09-13T11:24:38.161 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:38.161 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:38.161 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:24:38.161 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:41.400653+0000", 2024-09-13T11:24:38.161 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.8195000000000002e-05 2024-09-13T11:24:38.161 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:38.161 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:38.161 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:24:38.161 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:41.833180+0000", 2024-09-13T11:24:38.162 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.43252769200000002 2024-09-13T11:24:38.162 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:38.162 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:38.162 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:24:38.162 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:41.833213+0000", 2024-09-13T11:24:38.162 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.2737999999999997e-05 2024-09-13T11:24:38.162 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:38.162 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:38.162 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:24:38.162 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:41.833408+0000", 2024-09-13T11:24:38.162 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00019473799999999999 2024-09-13T11:24:38.162 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:38.163 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:38.163 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:24:38.163 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:41.833417+0000", 2024-09-13T11:24:38.163 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.5349999999999993e-06 2024-09-13T11:24:38.163 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:38.163 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:38.163 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:24:38.163 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:41.834261+0000", 2024-09-13T11:24:38.163 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00084333200000000002 2024-09-13T11:24:38.163 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:38.163 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:24:38.163 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:38.163 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:38.164 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:38.164 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDPGPushReply(3.20 85/77 [PushReplyOp(3:066982a6:::benchmark_data_smithi177_31116_object16400:head)])", 2024-09-13T11:24:38.164 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:42.671551+0000", 2024-09-13T11:24:38.164 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 55.450923985999999, 2024-09-13T11:24:38.164 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.28249085499999999, 2024-09-13T11:24:38.164 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:24:38.164 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:24:38.164 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-13T11:24:38.164 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:24:38.164 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:38.165 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:24:38.165 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.671551+0000", 2024-09-13T11:24:38.165 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:38.165 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:38.165 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:38.165 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:24:38.165 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.671551+0000", 2024-09-13T11:24:38.165 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:38.165 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:38.165 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:38.165 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:24:38.165 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.671552+0000", 2024-09-13T11:24:38.165 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.097e-06 2024-09-13T11:24:38.166 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:38.166 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:38.166 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:24:38.166 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.671556+0000", 2024-09-13T11:24:38.166 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.8519999999999997e-06 2024-09-13T11:24:38.166 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:38.166 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:38.166 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:24:38.166 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.951769+0000", 2024-09-13T11:24:38.166 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.28021293000000003 2024-09-13T11:24:38.166 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:38.166 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:38.167 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:24:38.167 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.951780+0000", 2024-09-13T11:24:38.167 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.1059e-05 2024-09-13T11:24:38.167 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:38.167 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:38.167 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:24:38.167 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.951992+0000", 2024-09-13T11:24:38.167 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000211478 2024-09-13T11:24:38.167 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:38.167 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:38.167 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:24:38.167 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.954042+0000", 2024-09-13T11:24:38.167 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0020504389999999998 2024-09-13T11:24:38.167 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:38.168 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:24:38.168 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:38.168 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:38.168 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:38.168 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDPGPush(3.1b 89/78 [PushOp(3:dc4c79e7:::benchmark_data_smithi177_31116_object19857:head, version: 54'1266, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:dc4c79e7:::benchmark_data_smithi177_31116_object19857:head@54'1266, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:24:38.168 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:46.649128+0000", 2024-09-13T11:24:38.168 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 51.473346878000001, 2024-09-13T11:24:38.168 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.29963603, 2024-09-13T11:24:38.168 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:24:38.168 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:24:38.168 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:24:38.168 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:24:38.168 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:38.169 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:24:38.169 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.649128+0000", 2024-09-13T11:24:38.169 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:38.169 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:38.169 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:38.169 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:24:38.169 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.649128+0000", 2024-09-13T11:24:38.169 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:38.169 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:38.169 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:38.169 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:24:38.169 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.649130+0000", 2024-09-13T11:24:38.169 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2950000000000001e-06 2024-09-13T11:24:38.170 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:38.170 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:38.170 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:24:38.170 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.649197+0000", 2024-09-13T11:24:38.170 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.7033999999999995e-05 2024-09-13T11:24:38.170 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:38.170 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:38.170 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:24:38.170 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.947691+0000", 2024-09-13T11:24:38.170 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.29849387300000002 2024-09-13T11:24:38.170 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:38.170 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:38.170 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:24:38.170 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.947702+0000", 2024-09-13T11:24:38.171 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.1239e-05 2024-09-13T11:24:38.171 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:38.171 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:38.171 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:24:38.171 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.947799+0000", 2024-09-13T11:24:38.171 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.7283999999999999e-05 2024-09-13T11:24:38.171 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:38.171 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:38.171 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:24:38.171 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.947809+0000", 2024-09-13T11:24:38.171 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.008e-05 2024-09-13T11:24:38.171 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:38.171 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:38.171 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:24:38.172 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.948764+0000", 2024-09-13T11:24:38.172 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00095522499999999998 2024-09-13T11:24:38.172 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:38.172 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:24:38.172 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:38.172 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:38.172 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:38.172 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:71927 3.0 3:0197244a:::benchmark_data_smithi177_31116_object71926:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2024-09-13T11:24:38.172 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:54.199022+0000", 2024-09-13T11:24:38.172 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 43.923453522000003, 2024-09-13T11:24:38.172 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.26819617800000001, 2024-09-13T11:24:38.172 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:24:38.172 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:24:38.173 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:38.173 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:24:38.173 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:24:38.173 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:38.173 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 71927 2024-09-13T11:24:38.173 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:38.173 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:24:38.173 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:38.173 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:24:38.173 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:54.199022+0000", 2024-09-13T11:24:38.173 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:38.173 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:38.173 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:38.173 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:24:38.174 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:54.199022+0000", 2024-09-13T11:24:38.174 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:38.174 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:38.174 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:38.174 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:24:38.174 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:54.199025+0000", 2024-09-13T11:24:38.174 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.2200000000000001e-06 2024-09-13T11:24:38.174 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:38.174 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:38.174 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:24:38.174 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:54.199076+0000", 2024-09-13T11:24:38.174 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.1184000000000003e-05 2024-09-13T11:24:38.174 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:38.174 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:38.175 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:24:38.175 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:54.199095+0000", 2024-09-13T11:24:38.175 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.9253999999999999e-05 2024-09-13T11:24:38.175 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:38.175 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:38.175 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:24:38.175 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:54.199107+0000", 2024-09-13T11:24:38.175 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.1916000000000001e-05 2024-09-13T11:24:38.175 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:38.175 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:38.175 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:24:38.175 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:54.199196+0000", 2024-09-13T11:24:38.175 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.8645000000000003e-05 2024-09-13T11:24:38.175 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:38.176 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:38.176 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:24:38.176 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:54.199421+0000", 2024-09-13T11:24:38.176 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000224868 2024-09-13T11:24:38.176 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:38.176 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:38.176 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 15", 2024-09-13T11:24:38.176 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:54.199647+0000", 2024-09-13T11:24:38.176 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00022666099999999999 2024-09-13T11:24:38.177 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:38.177 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:38.177 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:24:38.177 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:54.201333+0000", 2024-09-13T11:24:38.177 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.001685761 2024-09-13T11:24:38.177 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:38.177 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:38.177 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:38.177 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:54.467097+0000", 2024-09-13T11:24:38.177 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.26576393399999998 2024-09-13T11:24:38.177 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:38.177 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:38.177 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:24:38.177 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:54.467163+0000", 2024-09-13T11:24:38.178 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.6088999999999997e-05 2024-09-13T11:24:38.178 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:38.178 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:38.178 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:24:38.178 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:54.467218+0000", 2024-09-13T11:24:38.178 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.4645999999999998e-05 2024-09-13T11:24:38.178 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:38.178 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:24:38.178 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:38.178 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:38.178 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:38.178 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4665.0:98479 3.2 e113/75 3:41084616:::benchmark_data_smithi177_31116_object98478:head v 113'3785, mlcod=113'3782)", 2024-09-13T11:24:38.178 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:24:15.343730+0000", 2024-09-13T11:24:38.178 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 22.778745252, 2024-09-13T11:24:38.179 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.196889648, 2024-09-13T11:24:38.179 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:24:38.179 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:24:38.179 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:38.179 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:24:38.179 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:38.179 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:24:38.179 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.343730+0000", 2024-09-13T11:24:38.179 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:38.179 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:38.179 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:38.179 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:24:38.179 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.343730+0000", 2024-09-13T11:24:38.179 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:38.180 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:38.180 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:38.180 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:24:38.180 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.343732+0000", 2024-09-13T11:24:38.180 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.6819999999999999e-06 2024-09-13T11:24:38.180 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:38.180 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:38.180 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:24:38.180 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.343781+0000", 2024-09-13T11:24:38.180 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.9691999999999998e-05 2024-09-13T11:24:38.180 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:38.180 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:38.180 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:24:38.180 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.538204+0000", 2024-09-13T11:24:38.181 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.19442271699999999 2024-09-13T11:24:38.181 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:38.181 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:38.181 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:24:38.181 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.538215+0000", 2024-09-13T11:24:38.181 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0648e-05 2024-09-13T11:24:38.181 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:38.181 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:38.181 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:24:38.181 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.538342+0000", 2024-09-13T11:24:38.181 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000127448 2024-09-13T11:24:38.181 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:38.181 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:38.181 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:24:38.182 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.538395+0000", 2024-09-13T11:24:38.182 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.2895000000000001e-05 2024-09-13T11:24:38.182 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:38.182 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:38.182 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:24:38.182 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.540544+0000", 2024-09-13T11:24:38.182 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0021487379999999999 2024-09-13T11:24:38.182 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:38.182 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:38.182 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:24:38.182 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.540620+0000", 2024-09-13T11:24:38.183 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.5828000000000007e-05 2024-09-13T11:24:38.183 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:38.183 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:24:38.183 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:38.183 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:38.183 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:38.183 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4665.0:98760 3.2 e113/75 3:4041dcea:::benchmark_data_smithi177_31116_object98759:head v 113'3787, mlcod=113'3784)", 2024-09-13T11:24:38.183 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:24:15.612920+0000", 2024-09-13T11:24:38.183 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 22.509555257999999, 2024-09-13T11:24:38.183 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.17136494999999999, 2024-09-13T11:24:38.183 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:24:38.183 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:24:38.183 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:38.184 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:24:38.184 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:38.184 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:24:38.184 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.612920+0000", 2024-09-13T11:24:38.184 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:38.184 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:38.184 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:38.184 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:24:38.184 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.612920+0000", 2024-09-13T11:24:38.184 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:38.184 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:38.184 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:38.184 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:24:38.185 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.612924+0000", 2024-09-13T11:24:38.185 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.7960000000000002e-06 2024-09-13T11:24:38.185 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:38.185 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:38.185 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:24:38.185 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.613010+0000", 2024-09-13T11:24:38.185 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.6106000000000005e-05 2024-09-13T11:24:38.185 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:38.185 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:38.185 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:24:38.185 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.781465+0000", 2024-09-13T11:24:38.185 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.16845532899999999 2024-09-13T11:24:38.185 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:38.186 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:38.186 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:24:38.186 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.781481+0000", 2024-09-13T11:24:38.186 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.6180000000000001e-05 2024-09-13T11:24:38.186 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:38.186 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:38.186 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:24:38.186 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.781698+0000", 2024-09-13T11:24:38.186 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00021694999999999999 2024-09-13T11:24:38.186 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:38.186 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:38.186 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:24:38.186 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.781750+0000", 2024-09-13T11:24:38.186 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.2129000000000002e-05 2024-09-13T11:24:38.187 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:38.187 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:38.187 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:24:38.187 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.784185+0000", 2024-09-13T11:24:38.187 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0024345180000000001 2024-09-13T11:24:38.187 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:38.187 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:38.187 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:24:38.187 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.784285+0000", 2024-09-13T11:24:38.187 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.9942e-05 2024-09-13T11:24:38.187 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:38.187 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:24:38.187 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:38.188 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:38.188 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:38.188 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4665.0:98954 3.2 e113/75 3:416baeb7:::benchmark_data_smithi177_31116_object98953:head v 113'3790, mlcod=113'3787)", 2024-09-13T11:24:38.188 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:24:15.819065+0000", 2024-09-13T11:24:38.188 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 22.303410370000002, 2024-09-13T11:24:38.188 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.187742985, 2024-09-13T11:24:38.188 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:24:38.188 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:24:38.188 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:38.188 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:24:38.188 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:38.188 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:24:38.189 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.819065+0000", 2024-09-13T11:24:38.189 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:38.189 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:38.189 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:38.189 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:24:38.189 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.819065+0000", 2024-09-13T11:24:38.189 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:38.189 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:38.189 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:38.189 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:24:38.189 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.819068+0000", 2024-09-13T11:24:38.189 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.0469999999999998e-06 2024-09-13T11:24:38.189 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:38.190 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:38.190 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:24:38.190 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.819124+0000", 2024-09-13T11:24:38.190 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.6335999999999997e-05 2024-09-13T11:24:38.190 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:38.190 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:38.190 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:24:38.190 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:16.003736+0000", 2024-09-13T11:24:38.190 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.18461164899999999 2024-09-13T11:24:38.190 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:38.190 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:38.190 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:24:38.190 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:16.003749+0000", 2024-09-13T11:24:38.190 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.3157e-05 2024-09-13T11:24:38.191 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:38.191 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:38.191 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:24:38.191 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:16.003903+0000", 2024-09-13T11:24:38.191 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00015374799999999999 2024-09-13T11:24:38.191 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:38.191 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:38.191 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:24:38.191 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:16.003961+0000", 2024-09-13T11:24:38.191 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.8026000000000003e-05 2024-09-13T11:24:38.191 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:38.192 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:38.192 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:24:38.192 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:16.006699+0000", 2024-09-13T11:24:38.192 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0027384290000000001 2024-09-13T11:24:38.192 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:38.192 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:38.192 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:24:38.192 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:16.006808+0000", 2024-09-13T11:24:38.192 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000108593 2024-09-13T11:24:38.192 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:38.192 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:24:38.192 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:38.193 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:38.193 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:38.193 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4665.0:99149 3.2 e113/75 3:4350ea1a:::benchmark_data_smithi177_31116_object99148:head v 113'3791, mlcod=113'3788)", 2024-09-13T11:24:38.193 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:24:16.003634+0000", 2024-09-13T11:24:38.193 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 22.118840849000001, 2024-09-13T11:24:38.193 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.19660476800000001, 2024-09-13T11:24:38.193 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:24:38.193 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:24:38.193 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:38.193 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:24:38.193 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:38.193 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:24:38.193 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:16.003634+0000", 2024-09-13T11:24:38.194 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:38.194 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:38.194 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:38.194 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:24:38.194 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:16.003634+0000", 2024-09-13T11:24:38.194 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:38.194 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:38.194 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:38.194 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:24:38.194 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:16.003637+0000", 2024-09-13T11:24:38.194 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.9749999999999999e-06 2024-09-13T11:24:38.194 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:38.195 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:38.195 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:24:38.195 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:16.003701+0000", 2024-09-13T11:24:38.195 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.3764999999999997e-05 2024-09-13T11:24:38.195 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:38.195 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:38.195 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:24:38.195 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:16.198858+0000", 2024-09-13T11:24:38.195 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.19515649900000001 2024-09-13T11:24:38.195 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:38.195 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:38.195 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:24:38.196 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:16.198863+0000", 2024-09-13T11:24:38.196 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.2229999999999998e-06 2024-09-13T11:24:38.196 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:38.196 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:38.196 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:24:38.196 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:16.198918+0000", 2024-09-13T11:24:38.196 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.5427000000000003e-05 2024-09-13T11:24:38.196 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:38.196 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:38.197 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:24:38.197 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:16.198935+0000", 2024-09-13T11:24:38.197 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.6776e-05 2024-09-13T11:24:38.197 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:38.197 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:38.197 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:24:38.197 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:16.200208+0000", 2024-09-13T11:24:38.197 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0012725429999999999 2024-09-13T11:24:38.197 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:38.197 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:38.198 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:24:38.198 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:16.200239+0000", 2024-09-13T11:24:38.198 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.1560000000000003e-05 2024-09-13T11:24:38.198 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:38.198 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:24:38.198 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:38.198 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:38.198 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:38.198 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4665.0:101036 3.4 e114/67 3:207f4fe8:::benchmark_data_smithi177_31116_object101035:head v 114'4508, mlcod=114'4505)", 2024-09-13T11:24:38.198 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:24:17.391594+0000", 2024-09-13T11:24:38.198 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 20.730881473, 2024-09-13T11:24:38.198 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.19061766999999999, 2024-09-13T11:24:38.199 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:24:38.199 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:24:38.199 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:38.199 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:24:38.199 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:38.199 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:24:38.199 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:17.391594+0000", 2024-09-13T11:24:38.199 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:38.199 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:38.199 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:38.199 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:24:38.199 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:17.391594+0000", 2024-09-13T11:24:38.200 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:38.200 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:38.200 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:38.200 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:24:38.200 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:17.391595+0000", 2024-09-13T11:24:38.200 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.6729999999999999e-06 2024-09-13T11:24:38.200 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:38.200 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:38.200 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:24:38.200 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:17.391668+0000", 2024-09-13T11:24:38.200 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.2096999999999998e-05 2024-09-13T11:24:38.200 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:38.201 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:38.201 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:24:38.201 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:17.579638+0000", 2024-09-13T11:24:38.201 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.187969993 2024-09-13T11:24:38.201 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:38.201 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:38.201 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:24:38.201 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:17.579644+0000", 2024-09-13T11:24:38.201 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.2049999999999999e-06 2024-09-13T11:24:38.201 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:38.201 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:38.201 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:24:38.202 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:17.579707+0000", 2024-09-13T11:24:38.202 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.2948999999999996e-05 2024-09-13T11:24:38.202 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:38.202 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:38.202 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:24:38.202 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:17.579731+0000", 2024-09-13T11:24:38.202 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.4315999999999999e-05 2024-09-13T11:24:38.202 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:38.202 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:38.202 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:24:38.202 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:17.582172+0000", 2024-09-13T11:24:38.202 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.002440705 2024-09-13T11:24:38.203 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:38.203 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:38.203 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:24:38.203 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:17.582211+0000", 2024-09-13T11:24:38.203 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.9731999999999998e-05 2024-09-13T11:24:38.203 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:38.203 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:24:38.203 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:38.203 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:38.203 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:38.203 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:108129 3.9 3:94038175:::benchmark_data_smithi177_31116_object108128:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116)", 2024-09-13T11:24:38.203 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:24:22.802908+0000", 2024-09-13T11:24:38.203 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 15.319566753, 2024-09-13T11:24:38.204 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.179351336, 2024-09-13T11:24:38.204 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:24:38.204 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:24:38.204 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:38.204 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:24:38.204 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:24:38.204 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:38.204 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 108129 2024-09-13T11:24:38.204 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:38.204 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:24:38.204 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:38.204 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:24:38.204 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:22.802908+0000", 2024-09-13T11:24:38.205 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:38.205 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:38.205 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:38.205 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:24:38.205 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:22.802908+0000", 2024-09-13T11:24:38.205 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:38.205 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:38.205 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:38.205 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:24:38.205 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:22.802910+0000", 2024-09-13T11:24:38.205 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.765e-06 2024-09-13T11:24:38.205 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:38.206 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:38.206 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:24:38.206 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:22.802971+0000", 2024-09-13T11:24:38.206 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.0866000000000003e-05 2024-09-13T11:24:38.206 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:38.206 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:38.206 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:24:38.206 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:22.802985+0000", 2024-09-13T11:24:38.206 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.433e-05 2024-09-13T11:24:38.206 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:38.206 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:38.207 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:24:38.207 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:22.802993+0000", 2024-09-13T11:24:38.207 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.7940000000000003e-06 2024-09-13T11:24:38.207 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:38.207 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:38.207 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:24:38.207 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:22.803050+0000", 2024-09-13T11:24:38.207 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.6567999999999998e-05 2024-09-13T11:24:38.207 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:38.207 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:38.207 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:24:38.208 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:22.803158+0000", 2024-09-13T11:24:38.208 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00010788700000000001 2024-09-13T11:24:38.208 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:38.208 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:38.208 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 2", 2024-09-13T11:24:38.208 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:22.803262+0000", 2024-09-13T11:24:38.208 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00010450099999999999 2024-09-13T11:24:38.208 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:38.208 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:38.208 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:24:38.208 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:22.806107+0000", 2024-09-13T11:24:38.209 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0028447149999999998 2024-09-13T11:24:38.209 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:38.209 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:38.209 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:38.209 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:22.982142+0000", 2024-09-13T11:24:38.209 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.17603523700000001 2024-09-13T11:24:38.209 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:38.209 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:38.210 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:24:38.210 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:22.982196+0000", 2024-09-13T11:24:38.210 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.3832999999999998e-05 2024-09-13T11:24:38.210 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:38.210 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:38.210 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:24:38.210 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:22.982260+0000", 2024-09-13T11:24:38.210 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.3839999999999999e-05 2024-09-13T11:24:38.210 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:38.210 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:24:38.210 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:38.210 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:38.211 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:38.211 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop_reply(client.4665.0:108129 3.9 e116/67 ondisk, result = 0)", 2024-09-13T11:24:38.211 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:24:22.805315+0000", 2024-09-13T11:24:38.211 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 15.317159867000001, 2024-09-13T11:24:38.211 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.17695082600000001, 2024-09-13T11:24:38.211 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:24:38.211 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:24:38.211 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:24:38.211 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:24:38.211 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:38.211 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:24:38.211 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:22.805315+0000", 2024-09-13T11:24:38.212 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:38.212 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:38.212 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:38.212 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:24:38.212 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:22.805315+0000", 2024-09-13T11:24:38.212 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:38.212 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:38.212 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:38.212 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:24:38.212 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:22.805317+0000", 2024-09-13T11:24:38.212 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.483e-06 2024-09-13T11:24:38.212 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:38.213 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:38.213 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:24:38.213 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:22.805335+0000", 2024-09-13T11:24:38.213 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.8428999999999999e-05 2024-09-13T11:24:38.213 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:38.213 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:38.213 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:24:38.213 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:22.982066+0000", 2024-09-13T11:24:38.213 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.176730521 2024-09-13T11:24:38.213 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:38.213 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:38.213 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:24:38.214 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:22.982072+0000", 2024-09-13T11:24:38.214 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.6359999999999997e-06 2024-09-13T11:24:38.214 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:38.214 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:38.214 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:24:38.214 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:22.982126+0000", 2024-09-13T11:24:38.214 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.3795999999999998e-05 2024-09-13T11:24:38.214 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:38.214 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:38.214 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:24:38.214 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:22.982135+0000", 2024-09-13T11:24:38.214 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.9120000000000004e-06 2024-09-13T11:24:38.214 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:38.215 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:38.215 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:24:38.215 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:22.982266+0000", 2024-09-13T11:24:38.215 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000131049 2024-09-13T11:24:38.215 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:38.215 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:24:38.215 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:38.215 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:38.215 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:38.215 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDPGPush(3.25 121/120 [PushOp(3:a5de2229:::benchmark_data_smithi177_31116_object1005:head, version: 43'62, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:a5de2229:::benchmark_data_smithi177_31116_object1005:head@43'62, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:24:38.216 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:24:27.379880+0000", 2024-09-13T11:24:38.216 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 10.742595683999999, 2024-09-13T11:24:38.216 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.72647719499999996, 2024-09-13T11:24:38.216 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:24:38.216 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:24:38.216 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:24:38.216 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:24:38.216 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:38.216 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:24:38.216 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:27.379880+0000", 2024-09-13T11:24:38.216 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:38.217 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:38.217 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:38.217 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:24:38.217 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:27.379880+0000", 2024-09-13T11:24:38.217 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:38.217 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:38.217 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:38.217 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:24:38.217 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:27.379882+0000", 2024-09-13T11:24:38.217 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.1050000000000002e-06 2024-09-13T11:24:38.217 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:38.217 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:38.218 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:24:38.218 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:27.379976+0000", 2024-09-13T11:24:38.218 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.4468000000000004e-05 2024-09-13T11:24:38.218 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:38.218 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:38.218 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:24:38.218 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:28.105075+0000", 2024-09-13T11:24:38.218 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.72509911299999996 2024-09-13T11:24:38.218 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:38.218 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:38.218 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:24:38.218 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:28.105111+0000", 2024-09-13T11:24:38.219 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.5403e-05 2024-09-13T11:24:38.219 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:38.219 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:38.219 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:24:38.219 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:28.105267+0000", 2024-09-13T11:24:38.219 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00015663600000000001 2024-09-13T11:24:38.219 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:38.219 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:38.219 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:24:38.219 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:28.105291+0000", 2024-09-13T11:24:38.219 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.4164000000000001e-05 2024-09-13T11:24:38.219 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:38.219 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:38.220 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:24:38.220 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:28.106357+0000", 2024-09-13T11:24:38.220 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.001065306 2024-09-13T11:24:38.220 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:38.220 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:24:38.220 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:38.220 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:38.220 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:38.220 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:113646 3.20 3:059047a0:::benchmark_data_smithi177_31116_object113645:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124)", 2024-09-13T11:24:38.220 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:24:32.178229+0000", 2024-09-13T11:24:38.220 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 5.944245724, 2024-09-13T11:24:38.220 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.23346934699999999, 2024-09-13T11:24:38.221 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:24:38.221 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:24:38.221 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:38.221 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:24:38.221 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:24:38.221 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:38.221 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 113646 2024-09-13T11:24:38.221 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:38.221 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:24:38.221 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:38.221 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:24:38.221 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:32.178229+0000", 2024-09-13T11:24:38.221 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:38.221 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:38.222 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:38.222 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:24:38.222 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:32.178229+0000", 2024-09-13T11:24:38.222 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:38.222 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:38.222 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:38.222 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:24:38.222 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:32.178232+0000", 2024-09-13T11:24:38.222 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.069e-06 2024-09-13T11:24:38.222 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:38.222 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:38.222 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:24:38.222 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:32.178266+0000", 2024-09-13T11:24:38.223 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.4242999999999997e-05 2024-09-13T11:24:38.223 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:38.223 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:38.223 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:24:38.223 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:32.178278+0000", 2024-09-13T11:24:38.223 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2558000000000001e-05 2024-09-13T11:24:38.223 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:38.223 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:38.223 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:24:38.223 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:32.178288+0000", 2024-09-13T11:24:38.223 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.6469999999999993e-06 2024-09-13T11:24:38.223 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:38.223 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:38.223 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:24:38.223 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:32.178349+0000", 2024-09-13T11:24:38.224 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.0643999999999997e-05 2024-09-13T11:24:38.224 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:38.224 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:38.224 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:24:38.224 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:32.178477+0000", 2024-09-13T11:24:38.224 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000128038 2024-09-13T11:24:38.224 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:38.224 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:38.224 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 0", 2024-09-13T11:24:38.224 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:32.178602+0000", 2024-09-13T11:24:38.224 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00012553600000000001 2024-09-13T11:24:38.224 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:38.224 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:38.225 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:24:38.225 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:32.180157+0000", 2024-09-13T11:24:38.225 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.001554386 2024-09-13T11:24:38.225 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:38.225 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:38.225 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:38.225 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:32.411634+0000", 2024-09-13T11:24:38.225 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.231477561 2024-09-13T11:24:38.225 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:38.225 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:38.225 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:24:38.225 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:32.411672+0000", 2024-09-13T11:24:38.225 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.7357999999999997e-05 2024-09-13T11:24:38.225 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:38.226 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:38.226 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:24:38.226 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:32.411699+0000", 2024-09-13T11:24:38.226 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.7307000000000001e-05 2024-09-13T11:24:38.226 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:38.226 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:24:38.226 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:38.226 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:38.226 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:38.226 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop_reply(client.4665.0:113646 3.20 e124/77 ondisk, result = 0)", 2024-09-13T11:24:38.226 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:24:32.180524+0000", 2024-09-13T11:24:38.226 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 5.9419509430000002, 2024-09-13T11:24:38.226 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.231180894, 2024-09-13T11:24:38.226 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:24:38.227 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:24:38.227 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:24:38.227 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:24:38.227 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:38.227 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:24:38.227 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:32.180524+0000", 2024-09-13T11:24:38.227 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:38.227 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:38.227 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:38.227 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:24:38.227 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:32.180524+0000", 2024-09-13T11:24:38.227 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:38.227 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:38.228 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:38.228 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:24:38.228 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:32.180526+0000", 2024-09-13T11:24:38.228 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.4419999999999999e-06 2024-09-13T11:24:38.228 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:38.228 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:38.228 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:24:38.228 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:32.180530+0000", 2024-09-13T11:24:38.228 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.7330000000000003e-06 2024-09-13T11:24:38.228 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:38.228 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:38.228 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:24:38.229 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:32.411549+0000", 2024-09-13T11:24:38.229 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.23101862300000001 2024-09-13T11:24:38.229 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:38.229 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:38.229 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:24:38.229 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:32.411558+0000", 2024-09-13T11:24:38.229 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.5949999999999999e-06 2024-09-13T11:24:38.229 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:38.229 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:38.229 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:24:38.229 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:32.411622+0000", 2024-09-13T11:24:38.229 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.4251000000000005e-05 2024-09-13T11:24:38.229 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:38.229 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:38.230 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:24:38.230 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:32.411629+0000", 2024-09-13T11:24:38.230 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.328e-06 2024-09-13T11:24:38.230 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:38.230 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:38.230 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:24:38.230 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:32.411705+0000", 2024-09-13T11:24:38.230 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.5921999999999996e-05 2024-09-13T11:24:38.230 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:38.230 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:24:38.230 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:38.230 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:38.230 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:24:38.230 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-13T11:24:38.232 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-13T11:24:38.233 INFO:tasks.ceph.osd.2.smithi162.stderr:2024-09-13T11:24:38.228+0000 7fe92b35a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26313) UID: 0 2024-09-13T11:24:38.292 INFO:teuthology.orchestra.run.smithi001.stderr:noscrub is set 2024-09-13T11:24:38.302 INFO:tasks.ceph.osd.5.smithi092.stderr:2024-09-13T11:24:38.298+0000 7f6f00132640 -1 osd.5 128 handle_osd_map injecting map crc failure 2024-09-13T11:24:38.303 INFO:tasks.ceph.osd.3.smithi177.stderr:2024-09-13T11:24:38.299+0000 7f10a25d1640 -1 osd.3 128 handle_osd_map injecting map crc failure 2024-09-13T11:24:38.305 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-09-13T11:24:38.300+0000 7f97b39f6640 -1 osd.6 128 handle_osd_map injecting map crc failure 2024-09-13T11:24:38.332 INFO:tasks.ceph.osd.7.smithi177.stderr:2024-09-13T11:24:38.331+0000 7f897b51d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26575) UID: 0 2024-09-13T11:24:38.397 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-09-13T11:24:38.398 INFO:teuthology.orchestra.run.smithi092.stdout: "ops": [ 2024-09-13T11:24:38.398 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:38.398 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "MOSDPGPush(3.12 129/122 [PushOp(3:4bc8ec32:::benchmark_data_smithi177_31116_object15834:head, version: 51'975, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:4bc8ec32:::benchmark_data_smithi177_31116_object15834:head@51'975, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:24:38.398 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:24:38.393981+0000", 2024-09-13T11:24:38.398 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 0.0029457670000000002, 2024-09-13T11:24:38.398 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0029968149999999999, 2024-09-13T11:24:38.398 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:24:38.398 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:24:38.398 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-09-13T11:24:38.398 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:24:38.399 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:38.399 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:24:38.399 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:38.393981+0000", 2024-09-13T11:24:38.399 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:38.399 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:38.399 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:38.399 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:24:38.399 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:38.393981+0000", 2024-09-13T11:24:38.399 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:38.399 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:38.399 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:38.399 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:24:38.400 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:38.393982+0000", 2024-09-13T11:24:38.400 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.3629999999999999e-06 2024-09-13T11:24:38.400 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:38.400 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:38.400 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:24:38.400 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:38.394031+0000", 2024-09-13T11:24:38.400 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.8458999999999998e-05 2024-09-13T11:24:38.400 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:38.400 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:38.400 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:24:38.400 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:38.394067+0000", 2024-09-13T11:24:38.400 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.6069999999999999e-05 2024-09-13T11:24:38.401 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:38.401 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:38.401 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:24:38.401 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:38.394082+0000", 2024-09-13T11:24:38.401 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.5129e-05 2024-09-13T11:24:38.401 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:38.401 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:38.401 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:24:38.401 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:38.394170+0000", 2024-09-13T11:24:38.401 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.7880000000000006e-05 2024-09-13T11:24:38.401 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:38.401 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:38.402 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:24:38.402 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:38.394182+0000", 2024-09-13T11:24:38.402 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.2181000000000001e-05 2024-09-13T11:24:38.402 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:38.402 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:24:38.402 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:38.402 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:38.402 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:38.402 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "MOSDPGPush(3.22 129/122 [PushOp(3:478b1ff9:::benchmark_data_smithi177_31116_object59757:head, version: 83'3154, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:478b1ff9:::benchmark_data_smithi177_31116_object59757:head@83'3154, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:24:38.402 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:24:38.396100+0000", 2024-09-13T11:24:38.403 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 0.00082665199999999999, 2024-09-13T11:24:38.403 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00092393800000000002, 2024-09-13T11:24:38.403 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:24:38.403 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:24:38.403 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-09-13T11:24:38.403 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:24:38.403 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:38.403 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:24:38.403 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:38.396100+0000", 2024-09-13T11:24:38.403 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:38.403 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:38.404 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:38.404 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:24:38.404 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:38.396100+0000", 2024-09-13T11:24:38.404 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:38.404 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:38.404 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:38.404 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:24:38.404 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:38.396101+0000", 2024-09-13T11:24:38.404 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.573e-06 2024-09-13T11:24:38.404 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:38.404 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:38.404 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:24:38.405 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:38.396146+0000", 2024-09-13T11:24:38.405 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.4450000000000003e-05 2024-09-13T11:24:38.405 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:38.405 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:38.405 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:24:38.405 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:38.396177+0000", 2024-09-13T11:24:38.405 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.1325000000000002e-05 2024-09-13T11:24:38.405 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:38.405 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:38.405 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:24:38.405 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:38.396190+0000", 2024-09-13T11:24:38.405 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.2700999999999999e-05 2024-09-13T11:24:38.405 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:38.406 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:38.406 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:24:38.406 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:38.396280+0000", 2024-09-13T11:24:38.406 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.0140000000000001e-05 2024-09-13T11:24:38.406 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:38.406 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:38.406 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:24:38.406 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:38.396294+0000", 2024-09-13T11:24:38.406 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.3799e-05 2024-09-13T11:24:38.406 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:38.406 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:24:38.406 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:38.406 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:38.407 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:38.407 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop(client.4665.0:120608 3.29 e129/120 3:97f22d0e:::benchmark_data_smithi177_31116_object120607:head v 129'4989, mlcod=0'0)", 2024-09-13T11:24:38.407 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:24:38.396860+0000", 2024-09-13T11:24:38.407 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 6.6328999999999995e-05, 2024-09-13T11:24:38.407 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00019287899999999999, 2024-09-13T11:24:38.407 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:24:38.407 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:24:38.407 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-09-13T11:24:38.407 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:24:38.407 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:38.407 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:24:38.407 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:38.396860+0000", 2024-09-13T11:24:38.407 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:38.408 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:38.408 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:38.408 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:24:38.408 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:38.396860+0000", 2024-09-13T11:24:38.408 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:38.408 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:38.408 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:38.408 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:24:38.408 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:38.396862+0000", 2024-09-13T11:24:38.408 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.4449999999999999e-06 2024-09-13T11:24:38.408 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:38.408 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:38.408 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:24:38.409 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:38.396891+0000", 2024-09-13T11:24:38.409 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.9876999999999999e-05 2024-09-13T11:24:38.409 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:38.409 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:38.409 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:24:38.409 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:38.396917+0000", 2024-09-13T11:24:38.409 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.5610999999999999e-05 2024-09-13T11:24:38.409 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:38.409 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:38.409 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:24:38.409 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:38.396924+0000", 2024-09-13T11:24:38.409 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.0600000000000002e-06 2024-09-13T11:24:38.410 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:38.410 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:38.410 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:24:38.410 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:38.396971+0000", 2024-09-13T11:24:38.410 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.7169000000000002e-05 2024-09-13T11:24:38.410 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:38.410 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:38.410 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:24:38.410 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:38.396988+0000", 2024-09-13T11:24:38.410 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.6626999999999999e-05 2024-09-13T11:24:38.410 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:38.411 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:24:38.411 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:38.411 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:38.411 INFO:teuthology.orchestra.run.smithi092.stdout: ], 2024-09-13T11:24:38.411 INFO:teuthology.orchestra.run.smithi092.stdout: "num_ops": 3 2024-09-13T11:24:38.411 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-09-13T11:24:38.418 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-13T11:24:38.434 INFO:tasks.ceph.osd.7.smithi177.stderr:2024-09-13T11:24:38.431+0000 7f897b51d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26575) UID: 0 2024-09-13T11:24:38.534 INFO:tasks.ceph.osd.9.smithi092.stderr:2024-09-13T11:24:38.530+0000 7ff5ab3d0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26348) UID: 0 2024-09-13T11:24:38.574 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-09-13T11:24:38.575 INFO:teuthology.orchestra.run.smithi092.stdout: "ops": [], 2024-09-13T11:24:38.575 INFO:teuthology.orchestra.run.smithi092.stdout: "complaint_time": 30, 2024-09-13T11:24:38.575 INFO:teuthology.orchestra.run.smithi092.stdout: "num_blocked_ops": 0 2024-09-13T11:24:38.575 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-09-13T11:24:38.597 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-13T11:24:38.635 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-13T11:24:38.631+0000 7f6a68775640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26307) UID: 0 2024-09-13T11:24:38.696 INFO:teuthology.orchestra.run.smithi162.stdout:get_pg_num_history pg_num_history pg_num_history(e120 pg_nums {1={23=1},2={25=8},3={30=16,67=26,75=36,117=46}} deleted_pools ) 2024-09-13T11:24:38.696 INFO:teuthology.orchestra.run.smithi162.stdout:Importing pgid 3.2 2024-09-13T11:24:38.704 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:40002d6b:::benchmark_data_smithi177_31116_object75887:head# 2024-09-13T11:24:38.705 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.705 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:400116a8:::benchmark_data_smithi177_31116_object88981:head# 2024-09-13T11:24:38.705 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.705 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:40037035:::benchmark_data_smithi177_31116_object89077:head# 2024-09-13T11:24:38.706 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.706 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4003dfa7:::benchmark_data_smithi177_31116_object69054:head# 2024-09-13T11:24:38.706 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.707 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4003f042:::benchmark_data_smithi177_31116_object73958:head# 2024-09-13T11:24:38.707 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.707 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:400448d7:::benchmark_data_smithi177_31116_object59583:head# 2024-09-13T11:24:38.707 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.707 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:40047b18:::benchmark_data_smithi177_31116_object58604:head# 2024-09-13T11:24:38.708 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.708 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:40052e2a:::benchmark_data_smithi177_31116_object91672:head# 2024-09-13T11:24:38.709 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.709 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:40058f96:::benchmark_data_smithi177_31116_object71551:head# 2024-09-13T11:24:38.709 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.709 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:40064d0d:::benchmark_data_smithi177_31116_object34318:head# 2024-09-13T11:24:38.709 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.709 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:400658dc:::benchmark_data_smithi177_31116_object380:head# 2024-09-13T11:24:38.709 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.712 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4006fcc8:::benchmark_data_smithi177_31116_object93228:head# 2024-09-13T11:24:38.712 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.714 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:40088aa9:::benchmark_data_smithi177_31116_object52520:head# 2024-09-13T11:24:38.715 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.715 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:40097539:::benchmark_data_smithi177_31116_object55848:head# 2024-09-13T11:24:38.715 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.717 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:40099444:::benchmark_data_smithi177_31116_object13996:head# 2024-09-13T11:24:38.718 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.719 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4009eb1d:::benchmark_data_smithi177_31116_object3494:head# 2024-09-13T11:24:38.719 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.720 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4009fd45:::benchmark_data_smithi177_31116_object43169:head# 2024-09-13T11:24:38.720 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.721 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:400a8c3a:::benchmark_data_smithi177_31116_object22108:head# 2024-09-13T11:24:38.721 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.721 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:400bd5d8:::benchmark_data_smithi177_31116_object15040:head# 2024-09-13T11:24:38.721 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.721 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:400d210c:::benchmark_data_smithi177_31116_object75432:head# 2024-09-13T11:24:38.721 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.722 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:400d89b0:::benchmark_data_smithi177_31116_object90016:head# 2024-09-13T11:24:38.722 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.722 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:400ea071:::benchmark_data_smithi177_31116_object92382:head# 2024-09-13T11:24:38.723 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.723 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:400ebb6e:::benchmark_data_smithi177_31116_object72630:head# 2024-09-13T11:24:38.723 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.723 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:400f019d:::benchmark_data_smithi177_31116_object96642:head# 2024-09-13T11:24:38.723 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.724 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:400f8950:::benchmark_data_smithi177_31116_object15356:head# 2024-09-13T11:24:38.724 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.724 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:400f9d51:::benchmark_data_smithi177_31116_object26316:head# 2024-09-13T11:24:38.724 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.726 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:400fecec:::benchmark_data_smithi177_31116_object82070:head# 2024-09-13T11:24:38.726 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.729 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4010577d:::benchmark_data_smithi177_31116_object30635:head# 2024-09-13T11:24:38.729 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.729 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4011168f:::benchmark_data_smithi177_31116_object62373:head# 2024-09-13T11:24:38.729 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.730 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:40130995:::benchmark_data_smithi177_31116_object30254:head# 2024-09-13T11:24:38.730 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.730 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:40132e23:::benchmark_data_smithi177_31116_object110039:head# 2024-09-13T11:24:38.730 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.731 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:40138ea2:::benchmark_data_smithi177_31116_object10011:head# 2024-09-13T11:24:38.731 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.731 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4013aea3:::benchmark_data_smithi177_31116_object94789:head# 2024-09-13T11:24:38.732 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.732 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:401440a6:::benchmark_data_smithi177_31116_object39885:head# 2024-09-13T11:24:38.732 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.732 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4015458f:::benchmark_data_smithi177_31116_object49345:head# 2024-09-13T11:24:38.732 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.733 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:40160e25:::benchmark_data_smithi177_31116_object54134:head# 2024-09-13T11:24:38.733 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.734 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:40162d9f:::benchmark_data_smithi177_31116_object84878:head# 2024-09-13T11:24:38.734 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.734 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:40163fbb:::benchmark_data_smithi177_31116_object95797:head# 2024-09-13T11:24:38.734 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.736 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:401716e7:::benchmark_data_smithi177_31116_object62252:head# 2024-09-13T11:24:38.736 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.736 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:40189b7d:::benchmark_data_smithi177_31116_object78303:head# 2024-09-13T11:24:38.736 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.736 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4019513e:::benchmark_data_smithi177_31116_object16991:head# 2024-09-13T11:24:38.736 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.736 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4019a456:::benchmark_data_smithi177_31116_object52363:head# 2024-09-13T11:24:38.736 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.737 INFO:tasks.ceph.osd.11.smithi177.stderr:2024-09-13T11:24:38.731+0000 7f24792c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26580) UID: 0 2024-09-13T11:24:38.738 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:401a1f70:::benchmark_data_smithi177_31116_object81378:head# 2024-09-13T11:24:38.738 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.739 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:401c0648:::benchmark_data_smithi177_31116_object23043:head# 2024-09-13T11:24:38.739 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.739 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:401c1c3b:::benchmark_data_smithi177_31116_object36717:head# 2024-09-13T11:24:38.739 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.739 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:401cd695:::benchmark_data_smithi177_31116_object56793:head# 2024-09-13T11:24:38.739 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.739 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:401d43e8:::benchmark_data_smithi177_31116_object91695:head# 2024-09-13T11:24:38.739 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.739 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:401d76bc:::benchmark_data_smithi177_31116_object99388:head# 2024-09-13T11:24:38.739 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.739 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:401d7835:::benchmark_data_smithi177_31116_object26887:head# 2024-09-13T11:24:38.739 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.741 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:401dd4d4:::benchmark_data_smithi177_31116_object56240:head# 2024-09-13T11:24:38.741 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.741 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:401e699c:::benchmark_data_smithi177_31116_object31726:head# 2024-09-13T11:24:38.741 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.741 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:401edb99:::benchmark_data_smithi177_31116_object100828:head# 2024-09-13T11:24:38.741 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.741 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:401ef1a7:::benchmark_data_smithi177_31116_object61250:head# 2024-09-13T11:24:38.741 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.741 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:401fa8e4:::benchmark_data_smithi177_31116_object27572:head# 2024-09-13T11:24:38.742 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.742 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4020e14d:::benchmark_data_smithi177_31116_object15817:head# 2024-09-13T11:24:38.742 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.742 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:402161fc:::benchmark_data_smithi177_31116_object55108:head# 2024-09-13T11:24:38.742 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.743 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4021ab69:::benchmark_data_smithi177_31116_object20661:head# 2024-09-13T11:24:38.743 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.743 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4022347d:::benchmark_data_smithi177_31116_object65012:head# 2024-09-13T11:24:38.744 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.744 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4022ccbd:::benchmark_data_smithi177_31116_object12406:head# 2024-09-13T11:24:38.744 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.744 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:40235564:::benchmark_data_smithi177_31116_object59542:head# 2024-09-13T11:24:38.744 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.745 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4023797a:::benchmark_data_smithi177_31116_object28457:head# 2024-09-13T11:24:38.745 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.745 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:40244812:::benchmark_data_smithi177_31116_object86273:head# 2024-09-13T11:24:38.745 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.745 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4024a979:::benchmark_data_smithi177_31116_object51662:head# 2024-09-13T11:24:38.745 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.745 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4025a717:::benchmark_data_smithi177_31116_object30274:head# 2024-09-13T11:24:38.746 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.746 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:40268a31:::benchmark_data_smithi177_31116_object70755:head# 2024-09-13T11:24:38.746 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.747 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4027dacb:::benchmark_data_smithi177_31116_object34591:head# 2024-09-13T11:24:38.747 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.747 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:402882e0:::benchmark_data_smithi177_31116_object73024:head# 2024-09-13T11:24:38.747 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.748 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4029b1e8:::benchmark_data_smithi177_31116_object61810:head# 2024-09-13T11:24:38.748 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.748 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:402a5335:::benchmark_data_smithi177_31116_object75143:head# 2024-09-13T11:24:38.748 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.748 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:402a5f44:::benchmark_data_smithi177_31116_object39853:head# 2024-09-13T11:24:38.748 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.748 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:402cd08b:::benchmark_data_smithi177_31116_object42476:head# 2024-09-13T11:24:38.748 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.749 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:402d046d:::benchmark_data_smithi177_31116_object93094:head# 2024-09-13T11:24:38.750 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.750 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:402d43e9:::benchmark_data_smithi177_31116_object34434:head# 2024-09-13T11:24:38.750 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.750 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:402dc97b:::benchmark_data_smithi177_31116_object20634:head# 2024-09-13T11:24:38.750 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.750 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:402dcb38:::benchmark_data_smithi177_31116_object103916:head# 2024-09-13T11:24:38.750 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.751 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:402e5899:::benchmark_data_smithi177_31116_object95399:head# 2024-09-13T11:24:38.751 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.751 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:402e627d:::benchmark_data_smithi177_31116_object104154:head# 2024-09-13T11:24:38.751 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.751 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:402ebe83:::benchmark_data_smithi177_31116_object80691:head# 2024-09-13T11:24:38.751 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.751 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:402f6f46:::benchmark_data_smithi177_31116_object87454:head# 2024-09-13T11:24:38.752 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.752 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:403065dd:::benchmark_data_smithi177_31116_object13292:head# 2024-09-13T11:24:38.752 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.753 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4030d737:::benchmark_data_smithi177_31116_object16511:head# 2024-09-13T11:24:38.753 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.753 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4030df1a:::benchmark_data_smithi177_31116_object70913:head# 2024-09-13T11:24:38.753 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.753 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4030f7c7:::benchmark_data_smithi177_31116_object57789:head# 2024-09-13T11:24:38.754 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.754 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4031b8f8:::benchmark_data_smithi177_31116_object71384:head# 2024-09-13T11:24:38.754 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.754 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4031d7a4:::benchmark_data_smithi177_31116_object86269:head# 2024-09-13T11:24:38.754 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.755 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4031fddf:::benchmark_data_smithi177_31116_object64167:head# 2024-09-13T11:24:38.755 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.755 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4031fe17:::benchmark_data_smithi177_31116_object60893:head# 2024-09-13T11:24:38.755 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.755 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4033dacb:::benchmark_data_smithi177_31116_object94537:head# 2024-09-13T11:24:38.756 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.756 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:40348e18:::benchmark_data_smithi177_31116_object64069:head# 2024-09-13T11:24:38.757 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.757 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4034ed0e:::benchmark_data_smithi177_31116_object15210:head# 2024-09-13T11:24:38.757 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.757 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:40357bd9:::benchmark_data_smithi177_31116_object47811:head# 2024-09-13T11:24:38.757 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.758 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:403583e9:::benchmark_data_smithi177_31116_object70931:head# 2024-09-13T11:24:38.758 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.758 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4035d24e:::benchmark_data_smithi177_31116_object25709:head# 2024-09-13T11:24:38.758 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.758 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:40361b04:::benchmark_data_smithi177_31116_object44813:head# 2024-09-13T11:24:38.758 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.759 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:40366f21:::benchmark_data_smithi177_31116_object23308:head# 2024-09-13T11:24:38.759 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.759 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4036998b:::benchmark_data_smithi177_31116_object69742:head# 2024-09-13T11:24:38.759 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.759 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:40370432:::benchmark_data_smithi177_31116_object21532:head# 2024-09-13T11:24:38.760 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.762 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:40371001:::benchmark_data_smithi177_31116_object103605:head# 2024-09-13T11:24:38.762 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.763 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:40377689:::benchmark_data_smithi177_31116_object21578:head# 2024-09-13T11:24:38.763 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.763 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:403782b2:::benchmark_data_smithi177_31116_object35224:head# 2024-09-13T11:24:38.763 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.764 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4037c73c:::benchmark_data_smithi177_31116_object31374:head# 2024-09-13T11:24:38.764 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.764 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:40385d5f:::benchmark_data_smithi177_31116_object81729:head# 2024-09-13T11:24:38.764 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.765 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4038b956:::benchmark_data_smithi177_31116_object3717:head# 2024-09-13T11:24:38.765 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.765 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:40394b85:::benchmark_data_smithi177_31116_object29315:head# 2024-09-13T11:24:38.765 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.765 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4039ccf8:::benchmark_data_smithi177_31116_object97045:head# 2024-09-13T11:24:38.765 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.766 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4039e8f1:::benchmark_data_smithi177_31116_object1034:head# 2024-09-13T11:24:38.766 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.766 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:403a5cf3:::benchmark_data_smithi177_31116_object16059:head# 2024-09-13T11:24:38.766 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.766 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:403ae3cd:::benchmark_data_smithi177_31116_object22499:head# 2024-09-13T11:24:38.766 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.767 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:403af28b:::benchmark_data_smithi177_31116_object61390:head# 2024-09-13T11:24:38.767 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.767 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:403af6ba:::benchmark_data_smithi177_31116_object57620:head# 2024-09-13T11:24:38.767 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.768 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:403bdb96:::benchmark_data_smithi177_31116_object5648:head# 2024-09-13T11:24:38.768 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.768 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:403c33cb:::benchmark_data_smithi177_31116_object3277:head# 2024-09-13T11:24:38.768 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.769 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:403cd19a:::benchmark_data_smithi177_31116_object38672:head# 2024-09-13T11:24:38.769 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.770 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:403db0c9:::benchmark_data_smithi177_31116_object39730:head# 2024-09-13T11:24:38.770 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.771 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:403de283:::benchmark_data_smithi177_31116_object109827:head# 2024-09-13T11:24:38.771 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.772 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:403f3d29:::benchmark_data_smithi177_31116_object39601:head# 2024-09-13T11:24:38.772 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.772 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:403f6dec:::benchmark_data_smithi177_31116_object17149:head# 2024-09-13T11:24:38.773 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.773 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4040904e:::benchmark_data_smithi177_31116_object59329:head# 2024-09-13T11:24:38.773 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.774 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:40409dcf:::benchmark_data_smithi177_31116_object72912:head# 2024-09-13T11:24:38.775 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.775 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:40414ec5:::benchmark_data_smithi177_31116_object14666:head# 2024-09-13T11:24:38.775 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.775 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4041dcea:::benchmark_data_smithi177_31116_object98759:head# 2024-09-13T11:24:38.776 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.776 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4041ff96:::benchmark_data_smithi177_31116_object61493:head# 2024-09-13T11:24:38.776 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.776 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4042c786:::benchmark_data_smithi177_31116_object48108:head# 2024-09-13T11:24:38.776 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.777 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4042cf94:::benchmark_data_smithi177_31116_object100748:head# 2024-09-13T11:24:38.777 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.777 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4042f7e6:::benchmark_data_smithi177_31116_object20525:head# 2024-09-13T11:24:38.777 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.778 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4043a50d:::benchmark_data_smithi177_31116_object84470:head# 2024-09-13T11:24:38.778 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.778 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:404491f9:::benchmark_data_smithi177_31116_object17623:head# 2024-09-13T11:24:38.778 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.779 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:40449c46:::benchmark_data_smithi177_31116_object76566:head# 2024-09-13T11:24:38.779 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.779 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4044ad5d:::benchmark_data_smithi177_31116_object20817:head# 2024-09-13T11:24:38.779 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.780 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4045117c:::benchmark_data_smithi177_31116_object106876:head# 2024-09-13T11:24:38.780 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.780 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:40452e6d:::benchmark_data_smithi177_31116_object74639:head# 2024-09-13T11:24:38.780 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.781 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:40457732:::benchmark_data_smithi177_31116_object8057:head# 2024-09-13T11:24:38.781 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.781 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4045a38c:::benchmark_data_smithi177_31116_object89567:head# 2024-09-13T11:24:38.781 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.781 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:404612b6:::benchmark_data_smithi177_31116_object24160:head# 2024-09-13T11:24:38.782 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.782 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:40465ce1:::benchmark_data_smithi177_31116_object5753:head# 2024-09-13T11:24:38.782 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.782 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4046fc26:::benchmark_data_smithi177_31116_object10072:head# 2024-09-13T11:24:38.783 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.783 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4047dd50:::benchmark_data_smithi177_31116_object46670:head# 2024-09-13T11:24:38.783 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.784 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:40481405:::benchmark_data_smithi177_31116_object21794:head# 2024-09-13T11:24:38.784 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.784 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4048776d:::benchmark_data_smithi177_31116_object7635:head# 2024-09-13T11:24:38.784 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.785 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4048b818:::benchmark_data_smithi177_31116_object37587:head# 2024-09-13T11:24:38.785 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.785 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4048f3d1:::benchmark_data_smithi177_31116_object14737:head# 2024-09-13T11:24:38.786 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.786 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4049e5ec:::benchmark_data_smithi177_31116_object50499:head# 2024-09-13T11:24:38.786 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.786 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:404ad0b9:::benchmark_data_smithi177_31116_object90297:head# 2024-09-13T11:24:38.786 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.787 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:404b5fbd:::benchmark_data_smithi177_31116_object55789:head# 2024-09-13T11:24:38.787 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.787 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:404b720a:::benchmark_data_smithi177_31116_object6649:head# 2024-09-13T11:24:38.787 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.788 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:404c078e:::benchmark_data_smithi177_31116_object62709:head# 2024-09-13T11:24:38.788 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.788 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:404e5bfd:::benchmark_data_smithi177_31116_object68453:head# 2024-09-13T11:24:38.788 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.788 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:404ec4e0:::benchmark_data_smithi177_31116_object86472:head# 2024-09-13T11:24:38.788 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.788 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:404f87df:::benchmark_data_smithi177_31116_object43855:head# 2024-09-13T11:24:38.789 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.790 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-09-13T11:24:38.790 INFO:teuthology.orchestra.run.smithi092.stdout: "size": 20, 2024-09-13T11:24:38.790 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 600, 2024-09-13T11:24:38.790 INFO:teuthology.orchestra.run.smithi092.stdout: "ops": [ 2024-09-13T11:24:38.790 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:38.790 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "MOSDPGPush(3.23 83/78 [PushOp(3:c61f6bab:::benchmark_data_smithi177_31116_object29479:head, version: 59'1860, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:c61f6bab:::benchmark_data_smithi177_31116_object29479:head@59'1860, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:24:38.790 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:40.770864+0000", 2024-09-13T11:24:38.790 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 58.014435570000003, 2024-09-13T11:24:38.790 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.31229662800000002, 2024-09-13T11:24:38.790 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:24:38.791 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:24:38.791 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-09-13T11:24:38.791 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:24:38.791 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:38.791 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:24:38.791 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:40.770864+0000", 2024-09-13T11:24:38.791 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:38.791 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:38.791 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:38.791 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:24:38.791 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:40.770864+0000", 2024-09-13T11:24:38.791 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:38.791 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:38.791 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:38.792 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:24:38.792 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:40.770865+0000", 2024-09-13T11:24:38.792 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.288e-06 2024-09-13T11:24:38.792 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:38.792 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:38.792 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:24:38.792 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:40.770955+0000", 2024-09-13T11:24:38.792 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.9871999999999996e-05 2024-09-13T11:24:38.792 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:38.792 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:38.792 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:24:38.792 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:41.082105+0000", 2024-09-13T11:24:38.792 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.31114963400000001 2024-09-13T11:24:38.793 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:38.793 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:38.793 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:24:38.793 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:41.082136+0000", 2024-09-13T11:24:38.793 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.0972000000000001e-05 2024-09-13T11:24:38.793 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:38.793 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:38.793 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:24:38.793 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:41.082404+0000", 2024-09-13T11:24:38.793 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00026844800000000002 2024-09-13T11:24:38.793 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:38.793 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:38.793 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:24:38.794 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:41.082429+0000", 2024-09-13T11:24:38.794 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.5021e-05 2024-09-13T11:24:38.794 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:38.794 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:38.794 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:24:38.794 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:41.083161+0000", 2024-09-13T11:24:38.794 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000731393 2024-09-13T11:24:38.794 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:38.794 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:24:38.794 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:38.794 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:38.794 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:38.794 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "MOSDPGPush(3.1e 85/78 [PushOp(3:7f40e925:::benchmark_data_smithi177_31116_object14084:head, version: 50'927, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:7f40e925:::benchmark_data_smithi177_31116_object14084:head@50'927, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:24:38.795 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:42.751000+0000", 2024-09-13T11:24:38.795 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 56.034299263000001, 2024-09-13T11:24:38.795 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.19208357500000001, 2024-09-13T11:24:38.795 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:24:38.795 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:24:38.795 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-09-13T11:24:38.795 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:24:38.795 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:38.795 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:24:38.795 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.751000+0000", 2024-09-13T11:24:38.795 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:38.795 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:38.795 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:38.795 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:24:38.796 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.751000+0000", 2024-09-13T11:24:38.796 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:38.796 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:38.796 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:38.796 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:24:38.796 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.751003+0000", 2024-09-13T11:24:38.796 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.0800000000000002e-06 2024-09-13T11:24:38.796 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:38.796 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:38.796 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:24:38.796 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.751069+0000", 2024-09-13T11:24:38.796 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.5417000000000001e-05 2024-09-13T11:24:38.796 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:38.796 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:38.797 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:24:38.797 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.941261+0000", 2024-09-13T11:24:38.797 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.19019233899999999 2024-09-13T11:24:38.797 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:38.797 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:38.797 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:24:38.797 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.941300+0000", 2024-09-13T11:24:38.797 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.9091999999999998e-05 2024-09-13T11:24:38.797 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:38.797 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:38.797 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:24:38.798 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.941519+0000", 2024-09-13T11:24:38.798 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000218521 2024-09-13T11:24:38.798 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:38.798 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:38.798 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:24:38.798 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.941544+0000", 2024-09-13T11:24:38.798 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.5412999999999999e-05 2024-09-13T11:24:38.798 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:38.798 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:38.799 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:24:38.799 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.943084+0000", 2024-09-13T11:24:38.799 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0015397130000000001 2024-09-13T11:24:38.799 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:38.799 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:24:38.799 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:38.799 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:38.799 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:38.799 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "MOSDPGPush(3.1e 90/78 [PushOp(3:7c65c13b:::benchmark_data_smithi177_31116_object22761:head, version: 55'1503, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:7c65c13b:::benchmark_data_smithi177_31116_object22761:head@55'1503, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:24:38.799 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:47.705735+0000", 2024-09-13T11:24:38.799 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 51.079564157999997, 2024-09-13T11:24:38.799 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.107567331, 2024-09-13T11:24:38.799 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:24:38.799 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:24:38.800 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-09-13T11:24:38.800 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:24:38.800 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:38.800 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:24:38.800 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:47.705735+0000", 2024-09-13T11:24:38.800 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:38.800 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:38.800 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:38.800 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:24:38.800 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:47.705735+0000", 2024-09-13T11:24:38.800 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:38.800 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:38.800 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:38.800 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:24:38.801 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:47.705738+0000", 2024-09-13T11:24:38.801 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.5940000000000001e-06 2024-09-13T11:24:38.801 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:38.801 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:38.801 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:24:38.801 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:47.705798+0000", 2024-09-13T11:24:38.801 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.9848e-05 2024-09-13T11:24:38.801 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:38.801 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:38.801 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:24:38.801 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:47.811473+0000", 2024-09-13T11:24:38.801 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.10567505100000001 2024-09-13T11:24:38.801 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:38.802 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:38.802 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:24:38.802 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:47.811488+0000", 2024-09-13T11:24:38.802 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.5311999999999998e-05 2024-09-13T11:24:38.802 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:38.802 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:38.802 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:24:38.802 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:47.811684+0000", 2024-09-13T11:24:38.802 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00019606100000000001 2024-09-13T11:24:38.802 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:38.802 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:38.802 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:24:38.802 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:47.811709+0000", 2024-09-13T11:24:38.803 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.5066e-05 2024-09-13T11:24:38.803 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:38.803 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:38.803 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:24:38.803 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:47.813303+0000", 2024-09-13T11:24:38.803 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0015933989999999999 2024-09-13T11:24:38.803 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:38.803 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:24:38.803 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:38.803 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:38.803 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:38.803 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "MOSDPGPush(3.1e 94/78 [PushOp(3:7e3b05ed:::benchmark_data_smithi177_31116_object30052:head, version: 59'1945, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:7e3b05ed:::benchmark_data_smithi177_31116_object30052:head@59'1945, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:24:38.803 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:52.086403+0000", 2024-09-13T11:24:38.803 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 46.698896646999998, 2024-09-13T11:24:38.804 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.088501402000000007, 2024-09-13T11:24:38.804 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:24:38.804 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:24:38.804 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-09-13T11:24:38.804 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:24:38.804 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:38.804 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:24:38.804 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:52.086403+0000", 2024-09-13T11:24:38.804 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:38.804 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:38.804 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:38.804 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:24:38.804 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:52.086403+0000", 2024-09-13T11:24:38.805 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:38.805 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:38.805 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:38.805 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:24:38.805 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:52.086405+0000", 2024-09-13T11:24:38.805 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.5620000000000002e-06 2024-09-13T11:24:38.805 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:38.805 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:38.805 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:24:38.805 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:52.086472+0000", 2024-09-13T11:24:38.805 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.7053999999999999e-05 2024-09-13T11:24:38.805 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:38.805 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:38.806 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:24:38.806 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:52.172902+0000", 2024-09-13T11:24:38.806 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.086430050999999994 2024-09-13T11:24:38.806 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:38.806 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:38.806 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:24:38.806 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:52.172928+0000", 2024-09-13T11:24:38.806 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.5106999999999999e-05 2024-09-13T11:24:38.806 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:38.806 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:38.806 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:24:38.806 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:52.173094+0000", 2024-09-13T11:24:38.806 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00016682200000000001 2024-09-13T11:24:38.806 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:38.807 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:38.807 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:24:38.807 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:52.173117+0000", 2024-09-13T11:24:38.807 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.2459e-05 2024-09-13T11:24:38.807 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:38.807 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:38.807 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:24:38.807 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:52.174904+0000", 2024-09-13T11:24:38.807 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0017873469999999999 2024-09-13T11:24:38.807 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:38.807 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:24:38.807 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:38.807 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:38.807 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:38.808 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:82769 3.1e 3:7a4e88b5:::benchmark_data_smithi177_31116_object82768:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105)", 2024-09-13T11:24:38.808 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:24:03.744632+0000", 2024-09-13T11:24:38.808 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 35.040667118000002, 2024-09-13T11:24:38.808 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.21475722699999999, 2024-09-13T11:24:38.808 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:24:38.808 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:24:38.808 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:38.808 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:24:38.808 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:24:38.808 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:38.808 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 82769 2024-09-13T11:24:38.808 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:38.808 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:24:38.809 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:38.809 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:24:38.809 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:03.744632+0000", 2024-09-13T11:24:38.809 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:38.809 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:38.809 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:38.809 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:24:38.809 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:03.744632+0000", 2024-09-13T11:24:38.809 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:38.809 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:38.809 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:38.809 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:24:38.809 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:03.744634+0000", 2024-09-13T11:24:38.810 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.672e-06 2024-09-13T11:24:38.810 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:38.810 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:38.810 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:24:38.810 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:03.744724+0000", 2024-09-13T11:24:38.810 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.0237999999999999e-05 2024-09-13T11:24:38.810 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:38.810 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:38.810 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:24:38.810 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:03.744736+0000", 2024-09-13T11:24:38.810 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.2038999999999999e-05 2024-09-13T11:24:38.810 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:38.810 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:38.811 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:24:38.811 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:03.744742+0000", 2024-09-13T11:24:38.811 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.2129999999999996e-06 2024-09-13T11:24:38.811 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:38.811 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:38.811 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:24:38.811 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:03.744785+0000", 2024-09-13T11:24:38.811 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.2898000000000002e-05 2024-09-13T11:24:38.811 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:38.811 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:38.811 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:24:38.811 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:03.744894+0000", 2024-09-13T11:24:38.811 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000109073 2024-09-13T11:24:38.811 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:38.812 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:38.812 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 4", 2024-09-13T11:24:38.812 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:03.745009+0000", 2024-09-13T11:24:38.812 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00011428499999999999 2024-09-13T11:24:38.812 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:38.812 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:38.812 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:24:38.812 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:03.746451+0000", 2024-09-13T11:24:38.812 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0014419179999999999 2024-09-13T11:24:38.812 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:38.812 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:38.812 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:38.812 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:03.959257+0000", 2024-09-13T11:24:38.813 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.212806361 2024-09-13T11:24:38.813 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:38.813 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:38.813 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:24:38.813 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:03.959333+0000", 2024-09-13T11:24:38.813 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.6001000000000006e-05 2024-09-13T11:24:38.813 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:38.813 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:38.813 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:24:38.813 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:03.959390+0000", 2024-09-13T11:24:38.813 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.6529e-05 2024-09-13T11:24:38.813 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:38.813 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:24:38.813 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:38.814 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:38.814 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:38.814 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:104331 3.1e 3:79890035:::benchmark_data_smithi177_31116_object104330:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-09-13T11:24:38.814 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:24:20.241529+0000", 2024-09-13T11:24:38.814 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 18.543770599999998, 2024-09-13T11:24:38.814 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.16976961700000001, 2024-09-13T11:24:38.814 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:24:38.814 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:24:38.814 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:38.814 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:24:38.814 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:24:38.814 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:38.814 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 104331 2024-09-13T11:24:38.814 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:38.815 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:24:38.815 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:38.815 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:24:38.815 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:20.241529+0000", 2024-09-13T11:24:38.815 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:38.815 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:38.815 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:38.815 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:24:38.815 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:20.241529+0000", 2024-09-13T11:24:38.815 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:38.815 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:38.815 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:38.815 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:24:38.816 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:20.241534+0000", 2024-09-13T11:24:38.816 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.775e-06 2024-09-13T11:24:38.816 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:38.816 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:38.816 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:24:38.816 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:20.241660+0000", 2024-09-13T11:24:38.816 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000126499 2024-09-13T11:24:38.816 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:38.816 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:38.816 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:24:38.816 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:20.241687+0000", 2024-09-13T11:24:38.816 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.6826999999999999e-05 2024-09-13T11:24:38.816 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:38.817 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:38.817 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:24:38.817 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:20.241703+0000", 2024-09-13T11:24:38.817 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.5858999999999998e-05 2024-09-13T11:24:38.817 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:38.817 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:38.817 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:24:38.817 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:20.241832+0000", 2024-09-13T11:24:38.817 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00012925799999999999 2024-09-13T11:24:38.817 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:38.817 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:38.817 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:24:38.817 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:20.242133+0000", 2024-09-13T11:24:38.817 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00030115900000000001 2024-09-13T11:24:38.818 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:38.818 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:38.818 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 4", 2024-09-13T11:24:38.818 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:20.242408+0000", 2024-09-13T11:24:38.818 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00027487799999999999 2024-09-13T11:24:38.818 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:38.818 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:38.818 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:24:38.818 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:20.246683+0000", 2024-09-13T11:24:38.818 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0042750469999999997 2024-09-13T11:24:38.818 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:38.818 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:38.818 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:38.819 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:20.411076+0000", 2024-09-13T11:24:38.819 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.16439300800000001 2024-09-13T11:24:38.819 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:38.819 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:38.819 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:24:38.819 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:20.411188+0000", 2024-09-13T11:24:38.819 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00011191 2024-09-13T11:24:38.819 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:38.819 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:38.819 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:24:38.819 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:20.411298+0000", 2024-09-13T11:24:38.819 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00011039699999999999 2024-09-13T11:24:38.819 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:38.820 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:24:38.820 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:38.820 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:38.820 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:38.820 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:104671 3.1e 3:794c2a6d:::benchmark_data_smithi177_31116_object104670:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-09-13T11:24:38.820 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:24:20.543331+0000", 2024-09-13T11:24:38.820 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 18.241968877000001, 2024-09-13T11:24:38.820 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.088033315000000001, 2024-09-13T11:24:38.820 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:24:38.820 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:24:38.820 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:38.820 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:24:38.820 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:24:38.821 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:38.821 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 104671 2024-09-13T11:24:38.821 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:38.821 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:24:38.821 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:38.821 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:24:38.821 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:20.543331+0000", 2024-09-13T11:24:38.821 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:38.821 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:38.821 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:38.821 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:24:38.821 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:20.543331+0000", 2024-09-13T11:24:38.821 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:38.822 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:38.822 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:38.822 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:24:38.822 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:20.543334+0000", 2024-09-13T11:24:38.822 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.1370000000000002e-06 2024-09-13T11:24:38.822 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:38.822 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:38.822 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:24:38.822 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:20.543386+0000", 2024-09-13T11:24:38.822 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.2527e-05 2024-09-13T11:24:38.822 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:38.822 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:38.822 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:24:38.822 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:20.543402+0000", 2024-09-13T11:24:38.823 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.6169000000000001e-05 2024-09-13T11:24:38.823 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:38.823 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:38.823 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:24:38.823 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:20.543412+0000", 2024-09-13T11:24:38.823 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.9960000000000005e-06 2024-09-13T11:24:38.823 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:38.823 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:38.823 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:24:38.823 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:20.543469+0000", 2024-09-13T11:24:38.823 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.6465000000000002e-05 2024-09-13T11:24:38.823 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:38.823 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:38.824 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:24:38.824 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:20.543579+0000", 2024-09-13T11:24:38.824 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000110263 2024-09-13T11:24:38.824 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:38.824 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:38.824 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 4", 2024-09-13T11:24:38.824 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:20.543676+0000", 2024-09-13T11:24:38.824 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.6725000000000007e-05 2024-09-13T11:24:38.824 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:38.824 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:38.824 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:24:38.824 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:20.545201+0000", 2024-09-13T11:24:38.824 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00152556 2024-09-13T11:24:38.825 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:38.825 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:38.825 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:38.825 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:20.631256+0000", 2024-09-13T11:24:38.825 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.08605438 2024-09-13T11:24:38.825 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:38.825 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:38.825 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:24:38.825 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:20.631313+0000", 2024-09-13T11:24:38.825 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.6860000000000001e-05 2024-09-13T11:24:38.825 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:38.825 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:38.825 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:24:38.825 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:20.631364+0000", 2024-09-13T11:24:38.826 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.1233000000000002e-05 2024-09-13T11:24:38.826 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:38.826 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:24:38.826 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:38.826 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:38.826 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:38.826 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:104823 3.1e 3:7a73472c:::benchmark_data_smithi177_31116_object104822:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-09-13T11:24:38.826 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:24:20.661312+0000", 2024-09-13T11:24:38.826 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 18.123987804999999, 2024-09-13T11:24:38.826 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.10093226499999999, 2024-09-13T11:24:38.826 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:24:38.826 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:24:38.826 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:38.826 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:24:38.827 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:24:38.827 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:38.827 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 104823 2024-09-13T11:24:38.827 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:38.827 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:24:38.827 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:38.827 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:24:38.827 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:20.661312+0000", 2024-09-13T11:24:38.827 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:38.827 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:38.827 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:38.827 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:24:38.827 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:20.661312+0000", 2024-09-13T11:24:38.827 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:38.828 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:38.828 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:38.828 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:24:38.828 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:20.661314+0000", 2024-09-13T11:24:38.828 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.5519999999999999e-06 2024-09-13T11:24:38.828 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:38.828 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:38.828 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:24:38.828 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:20.661381+0000", 2024-09-13T11:24:38.828 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.7246e-05 2024-09-13T11:24:38.828 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:38.828 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:38.829 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:24:38.829 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:20.661399+0000", 2024-09-13T11:24:38.829 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.8029e-05 2024-09-13T11:24:38.829 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:38.829 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:38.829 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:24:38.829 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:20.661407+0000", 2024-09-13T11:24:38.829 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.7239999999999992e-06 2024-09-13T11:24:38.829 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:38.829 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:38.829 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:24:38.829 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:20.661486+0000", 2024-09-13T11:24:38.829 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.8461999999999995e-05 2024-09-13T11:24:38.830 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:38.830 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:38.830 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:24:38.830 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:20.661579+0000", 2024-09-13T11:24:38.830 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.2941000000000003e-05 2024-09-13T11:24:38.830 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:38.830 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:38.830 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 4", 2024-09-13T11:24:38.830 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:20.661664+0000", 2024-09-13T11:24:38.830 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.5772000000000005e-05 2024-09-13T11:24:38.830 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:38.830 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:38.830 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:24:38.830 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:20.663376+0000", 2024-09-13T11:24:38.831 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.001711909 2024-09-13T11:24:38.831 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:38.831 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:38.831 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:38.831 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:20.762129+0000", 2024-09-13T11:24:38.831 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.098753143000000002 2024-09-13T11:24:38.831 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:38.831 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:38.831 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:24:38.831 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:20.762198+0000", 2024-09-13T11:24:38.831 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.8491000000000005e-05 2024-09-13T11:24:38.831 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:38.831 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:38.832 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:24:38.832 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:20.762244+0000", 2024-09-13T11:24:38.832 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.5995999999999998e-05 2024-09-13T11:24:38.832 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:38.832 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:24:38.832 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:38.832 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:38.832 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:38.832 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:110289 3.29 3.27287969 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e119)", 2024-09-13T11:24:38.832 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:24:25.774634+0000", 2024-09-13T11:24:38.832 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 13.010665495, 2024-09-13T11:24:38.832 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.31721633999999999, 2024-09-13T11:24:38.832 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:24:38.832 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:24:38.833 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "reached pg", 2024-09-13T11:24:38.833 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:24:38.833 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:24:38.833 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:38.833 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 110289 2024-09-13T11:24:38.833 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:38.833 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:24:38.833 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:38.833 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:24:38.833 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:25.774634+0000", 2024-09-13T11:24:38.833 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:38.833 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:38.833 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:38.833 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:24:38.834 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:25.774634+0000", 2024-09-13T11:24:38.834 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:38.834 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:38.834 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:38.834 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:24:38.834 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:25.774639+0000", 2024-09-13T11:24:38.834 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.6029999999999998e-06 2024-09-13T11:24:38.834 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:38.834 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:38.834 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:24:38.834 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:25.774695+0000", 2024-09-13T11:24:38.834 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.6694999999999998e-05 2024-09-13T11:24:38.834 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:38.835 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:38.835 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:24:38.835 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:25.774728+0000", 2024-09-13T11:24:38.835 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.2502000000000002e-05 2024-09-13T11:24:38.835 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:38.835 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:38.835 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:24:38.835 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:25.774750+0000", 2024-09-13T11:24:38.835 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.2002999999999999e-05 2024-09-13T11:24:38.835 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:38.835 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:38.835 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:24:38.835 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:25.774942+0000", 2024-09-13T11:24:38.836 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000192265 2024-09-13T11:24:38.836 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:38.836 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:38.836 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-09-13T11:24:38.836 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:25.774960+0000", 2024-09-13T11:24:38.836 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.8275999999999999e-05 2024-09-13T11:24:38.836 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:38.836 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:38.836 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:24:38.836 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:26.091821+0000", 2024-09-13T11:24:38.836 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.31686065000000002 2024-09-13T11:24:38.836 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:38.836 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:38.837 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:24:38.837 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:26.091850+0000", 2024-09-13T11:24:38.837 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.9346e-05 2024-09-13T11:24:38.837 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:38.837 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:24:38.837 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:38.837 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:38.837 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:38.837 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "MOSDPGPush(3.22 124/122 [PushOp(3:4700527b:::benchmark_data_smithi177_31116_object15634:head, version: 51'956, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:4700527b:::benchmark_data_smithi177_31116_object15634:head@51'956, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:24:38.837 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:24:31.967308+0000", 2024-09-13T11:24:38.837 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 6.8179913020000003, 2024-09-13T11:24:38.837 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.098974230999999996, 2024-09-13T11:24:38.837 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:24:38.838 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:24:38.838 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-09-13T11:24:38.838 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:24:38.838 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:38.838 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:24:38.838 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:31.967308+0000", 2024-09-13T11:24:38.838 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:38.838 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:38.838 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:38.838 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:24:38.838 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:31.967308+0000", 2024-09-13T11:24:38.838 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:38.838 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:38.838 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:38.839 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:24:38.839 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:31.967309+0000", 2024-09-13T11:24:38.839 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.198e-06 2024-09-13T11:24:38.839 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:38.839 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:38.839 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:24:38.839 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:31.967391+0000", 2024-09-13T11:24:38.839 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.1748000000000004e-05 2024-09-13T11:24:38.839 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:38.839 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:38.839 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:24:38.839 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:32.065659+0000", 2024-09-13T11:24:38.839 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.098268139000000004 2024-09-13T11:24:38.839 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:38.840 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:38.840 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:24:38.840 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:32.065671+0000", 2024-09-13T11:24:38.840 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.1391e-05 2024-09-13T11:24:38.840 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:38.840 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:38.840 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:24:38.840 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:32.065736+0000", 2024-09-13T11:24:38.840 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.5846999999999995e-05 2024-09-13T11:24:38.840 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:38.840 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:38.840 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:24:38.840 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:32.065745+0000", 2024-09-13T11:24:38.841 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.7129999999999994e-06 2024-09-13T11:24:38.841 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:38.841 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:38.841 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:24:38.841 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:32.066282+0000", 2024-09-13T11:24:38.841 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00053719499999999997 2024-09-13T11:24:38.841 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:38.841 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:24:38.841 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:38.841 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:38.841 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:38.841 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "MOSDPGPush(3.22 125/122 [PushOp(3:44e4f9f2:::benchmark_data_smithi177_31116_object16663:head, version: 52'1032, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:44e4f9f2:::benchmark_data_smithi177_31116_object16663:head@52'1032, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:24:38.841 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:24:32.743254+0000", 2024-09-13T11:24:38.841 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 6.0420454059999997, 2024-09-13T11:24:38.842 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.215485287, 2024-09-13T11:24:38.842 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:24:38.842 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:24:38.842 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-09-13T11:24:38.842 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:24:38.842 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:38.842 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:24:38.842 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:32.743254+0000", 2024-09-13T11:24:38.842 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:38.842 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:38.842 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:38.842 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:24:38.842 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:32.743254+0000", 2024-09-13T11:24:38.843 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:38.843 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:38.843 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:38.843 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:24:38.843 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:32.743255+0000", 2024-09-13T11:24:38.843 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.2500000000000004e-07 2024-09-13T11:24:38.843 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:38.843 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:38.843 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:24:38.843 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:32.743310+0000", 2024-09-13T11:24:38.843 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.5442000000000002e-05 2024-09-13T11:24:38.843 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:38.843 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:38.843 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:24:38.844 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:32.957754+0000", 2024-09-13T11:24:38.844 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.21444387000000001 2024-09-13T11:24:38.844 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:38.844 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:38.844 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:24:38.844 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:32.957765+0000", 2024-09-13T11:24:38.844 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0458999999999999e-05 2024-09-13T11:24:38.844 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:38.844 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:38.844 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:24:38.844 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:32.957841+0000", 2024-09-13T11:24:38.844 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.5993000000000002e-05 2024-09-13T11:24:38.844 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:38.845 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:38.845 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:24:38.845 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:32.957852+0000", 2024-09-13T11:24:38.845 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.1453e-05 2024-09-13T11:24:38.845 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:38.845 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:38.845 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:24:38.845 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:32.958739+0000", 2024-09-13T11:24:38.845 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00088724500000000005 2024-09-13T11:24:38.845 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:38.845 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:24:38.845 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:38.845 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:38.845 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:38.846 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop(client.4665.0:114803 3.28 e125/120 3:14b36e87:::benchmark_data_smithi177_31116_object114802:head v 125'4838, mlcod=47'620)", 2024-09-13T11:24:38.846 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:24:33.303434+0000", 2024-09-13T11:24:38.846 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 5.481865891, 2024-09-13T11:24:38.846 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.088532242999999997, 2024-09-13T11:24:38.846 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:24:38.846 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:24:38.846 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:38.846 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:24:38.846 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:38.846 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:24:38.846 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:33.303434+0000", 2024-09-13T11:24:38.846 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:38.846 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:38.846 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:38.847 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:24:38.847 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:33.303434+0000", 2024-09-13T11:24:38.847 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:38.847 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:38.847 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:38.847 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:24:38.847 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:33.303435+0000", 2024-09-13T11:24:38.847 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.891e-06 2024-09-13T11:24:38.847 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:38.847 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:38.847 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:24:38.847 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:33.303470+0000", 2024-09-13T11:24:38.847 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.4146999999999997e-05 2024-09-13T11:24:38.847 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:38.848 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:38.848 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:24:38.848 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:33.389874+0000", 2024-09-13T11:24:38.848 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.086404694000000004 2024-09-13T11:24:38.848 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:38.848 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:38.848 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:24:38.848 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:33.389881+0000", 2024-09-13T11:24:38.848 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.0260000000000002e-06 2024-09-13T11:24:38.848 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:38.848 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:38.848 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:24:38.848 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:33.389930+0000", 2024-09-13T11:24:38.849 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.8782000000000002e-05 2024-09-13T11:24:38.849 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:38.849 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:38.849 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:24:38.849 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:33.389953+0000", 2024-09-13T11:24:38.849 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.2466000000000001e-05 2024-09-13T11:24:38.849 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:38.849 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:38.849 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:24:38.849 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:33.391926+0000", 2024-09-13T11:24:38.849 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.001973514 2024-09-13T11:24:38.849 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:38.849 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:38.849 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:24:38.850 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:33.391966+0000", 2024-09-13T11:24:38.850 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.9722999999999999e-05 2024-09-13T11:24:38.850 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:38.850 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:24:38.850 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:38.850 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:38.850 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:38.850 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:114846 3.1e 3:794fc9e2:::benchmark_data_smithi177_31116_object114845:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e125)", 2024-09-13T11:24:38.850 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:24:33.345576+0000", 2024-09-13T11:24:38.850 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 5.4397236979999999, 2024-09-13T11:24:38.850 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.088622023999999994, 2024-09-13T11:24:38.850 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:24:38.850 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:24:38.850 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:38.851 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:24:38.851 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:24:38.851 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:38.851 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 114846 2024-09-13T11:24:38.851 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:38.851 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:24:38.851 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:38.851 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:24:38.851 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:33.345576+0000", 2024-09-13T11:24:38.851 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:38.851 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:38.851 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:38.851 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:24:38.851 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:33.345576+0000", 2024-09-13T11:24:38.852 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:38.852 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:38.852 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:38.852 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:24:38.852 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:33.345577+0000", 2024-09-13T11:24:38.852 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.6559999999999999e-06 2024-09-13T11:24:38.852 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:38.852 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:38.852 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:24:38.852 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:33.345668+0000", 2024-09-13T11:24:38.852 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.0162999999999997e-05 2024-09-13T11:24:38.852 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:38.852 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:38.852 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:24:38.853 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:33.345681+0000", 2024-09-13T11:24:38.853 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.2958e-05 2024-09-13T11:24:38.853 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:38.853 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:38.853 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:24:38.853 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:33.345688+0000", 2024-09-13T11:24:38.853 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.8169999999999999e-06 2024-09-13T11:24:38.853 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:38.853 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:38.853 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:24:38.853 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:33.345757+0000", 2024-09-13T11:24:38.853 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.8518999999999999e-05 2024-09-13T11:24:38.853 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:38.853 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:38.854 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:24:38.854 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:33.345896+0000", 2024-09-13T11:24:38.854 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00013874499999999999 2024-09-13T11:24:38.854 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:38.854 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:38.854 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 4", 2024-09-13T11:24:38.854 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:33.346042+0000", 2024-09-13T11:24:38.854 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00014622800000000001 2024-09-13T11:24:38.854 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:38.854 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:38.854 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:24:38.854 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:33.351479+0000", 2024-09-13T11:24:38.854 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0054375459999999997 2024-09-13T11:24:38.854 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:38.855 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:38.855 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:38.855 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:33.434096+0000", 2024-09-13T11:24:38.855 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.082616292999999993 2024-09-13T11:24:38.855 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:38.855 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:38.855 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:24:38.855 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:33.434143+0000", 2024-09-13T11:24:38.855 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.7700000000000001e-05 2024-09-13T11:24:38.855 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:38.855 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:38.855 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:24:38.855 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:33.434198+0000", 2024-09-13T11:24:38.855 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.4398999999999999e-05 2024-09-13T11:24:38.856 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:38.856 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:24:38.856 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:38.856 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:38.856 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:38.856 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:114929 3.1e 3:7a39ebc4:::benchmark_data_smithi177_31116_object114928:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e125)", 2024-09-13T11:24:38.856 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:24:33.446684+0000", 2024-09-13T11:24:38.856 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 5.3386155410000002, 2024-09-13T11:24:38.856 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.10317303, 2024-09-13T11:24:38.856 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:24:38.856 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:24:38.856 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:38.856 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:24:38.857 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:24:38.857 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:38.857 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 114929 2024-09-13T11:24:38.857 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:38.857 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:24:38.857 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:38.857 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:24:38.857 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:33.446684+0000", 2024-09-13T11:24:38.857 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:38.857 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:38.857 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:38.857 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:24:38.857 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:33.446684+0000", 2024-09-13T11:24:38.857 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:38.858 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:38.858 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:38.858 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:24:38.858 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:33.446686+0000", 2024-09-13T11:24:38.858 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.0040000000000002e-06 2024-09-13T11:24:38.858 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:38.858 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:38.858 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:24:38.858 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:33.446731+0000", 2024-09-13T11:24:38.858 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.4916000000000002e-05 2024-09-13T11:24:38.858 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:38.858 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:38.858 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:24:38.858 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:33.446745+0000", 2024-09-13T11:24:38.859 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.418e-05 2024-09-13T11:24:38.859 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:38.859 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:38.859 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:24:38.859 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:33.446752+0000", 2024-09-13T11:24:38.859 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.3150000000000003e-06 2024-09-13T11:24:38.859 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:38.859 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:38.859 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:24:38.859 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:33.446819+0000", 2024-09-13T11:24:38.859 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.6742000000000006e-05 2024-09-13T11:24:38.859 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:38.859 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:38.859 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:24:38.860 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:33.446964+0000", 2024-09-13T11:24:38.860 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00014518199999999999 2024-09-13T11:24:38.860 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:38.860 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:38.860 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 4", 2024-09-13T11:24:38.860 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:33.447106+0000", 2024-09-13T11:24:38.860 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000141791 2024-09-13T11:24:38.860 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:38.860 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:38.860 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:24:38.860 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:33.449960+0000", 2024-09-13T11:24:38.860 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.002853782 2024-09-13T11:24:38.860 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:38.861 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:38.861 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:38.861 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:33.549557+0000", 2024-09-13T11:24:38.861 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.099596911999999996 2024-09-13T11:24:38.861 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:38.861 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:38.861 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:24:38.861 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:33.549610+0000", 2024-09-13T11:24:38.861 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.312e-05 2024-09-13T11:24:38.861 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:38.861 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:38.861 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:24:38.861 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:33.549857+0000", 2024-09-13T11:24:38.861 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00024708599999999999 2024-09-13T11:24:38.862 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:38.862 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:24:38.862 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:38.862 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:38.862 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:38.862 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop(client.4665.0:114934 3.28 e125/120 3:156be749:::benchmark_data_smithi177_31116_object114933:head v 125'4840, mlcod=47'620)", 2024-09-13T11:24:38.862 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:24:33.454671+0000", 2024-09-13T11:24:38.862 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 5.3306281499999999, 2024-09-13T11:24:38.862 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.10349704999999999, 2024-09-13T11:24:38.862 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:24:38.862 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:24:38.862 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:38.862 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:24:38.862 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:38.863 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:24:38.863 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:33.454671+0000", 2024-09-13T11:24:38.863 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:38.863 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:38.863 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:38.863 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:24:38.863 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:33.454671+0000", 2024-09-13T11:24:38.863 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:38.863 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:38.863 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:38.863 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:24:38.863 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:33.454673+0000", 2024-09-13T11:24:38.863 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.3820000000000001e-06 2024-09-13T11:24:38.863 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:38.864 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:38.864 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:24:38.864 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:33.454707+0000", 2024-09-13T11:24:38.864 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.4808000000000003e-05 2024-09-13T11:24:38.864 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:38.864 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:38.864 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:24:38.864 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:33.555701+0000", 2024-09-13T11:24:38.864 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.100993755 2024-09-13T11:24:38.864 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:38.864 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:38.864 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:24:38.864 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:33.555708+0000", 2024-09-13T11:24:38.864 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.9530000000000002e-06 2024-09-13T11:24:38.865 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:38.865 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:38.865 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:24:38.865 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:33.555782+0000", 2024-09-13T11:24:38.865 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.3763000000000005e-05 2024-09-13T11:24:38.865 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:38.865 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:38.865 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:24:38.865 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:33.555808+0000", 2024-09-13T11:24:38.865 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.62e-05 2024-09-13T11:24:38.865 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:38.865 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:38.865 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:24:38.865 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:33.558122+0000", 2024-09-13T11:24:38.866 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0023143230000000001 2024-09-13T11:24:38.866 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:38.866 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:38.866 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:24:38.866 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:33.558168+0000", 2024-09-13T11:24:38.866 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.5865999999999998e-05 2024-09-13T11:24:38.866 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:38.866 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:24:38.866 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:38.866 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:38.866 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:38.866 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:115008 3.1e 3:7a0cff82:::benchmark_data_smithi177_31116_object115007:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e125)", 2024-09-13T11:24:38.866 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:24:33.547002+0000", 2024-09-13T11:24:38.866 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 5.23829777, 2024-09-13T11:24:38.867 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.099273717999999997, 2024-09-13T11:24:38.867 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:24:38.867 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:24:38.867 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:38.867 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:24:38.867 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:24:38.867 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:38.867 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 115008 2024-09-13T11:24:38.867 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:38.867 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:24:38.867 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:38.867 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:24:38.867 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:33.547002+0000", 2024-09-13T11:24:38.867 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:38.868 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:38.868 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:38.868 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:24:38.868 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:33.547002+0000", 2024-09-13T11:24:38.868 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:38.868 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:38.868 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:38.868 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:24:38.868 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:33.547016+0000", 2024-09-13T11:24:38.868 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.4429e-05 2024-09-13T11:24:38.868 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:38.868 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:38.868 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:24:38.868 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:33.547073+0000", 2024-09-13T11:24:38.869 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.7370000000000001e-05 2024-09-13T11:24:38.869 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:38.869 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:38.869 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:24:38.869 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:33.547085+0000", 2024-09-13T11:24:38.869 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.1891e-05 2024-09-13T11:24:38.869 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:38.869 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:38.869 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:24:38.869 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:33.547092+0000", 2024-09-13T11:24:38.869 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.0090000000000001e-06 2024-09-13T11:24:38.869 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:38.869 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:38.869 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:24:38.870 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:33.547149+0000", 2024-09-13T11:24:38.870 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.6508999999999997e-05 2024-09-13T11:24:38.870 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:38.870 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:38.870 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:24:38.870 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:33.547277+0000", 2024-09-13T11:24:38.870 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00012800800000000001 2024-09-13T11:24:38.870 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:38.870 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:38.870 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 4", 2024-09-13T11:24:38.870 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:33.547420+0000", 2024-09-13T11:24:38.870 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00014338900000000001 2024-09-13T11:24:38.870 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:38.871 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:38.871 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:24:38.871 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:33.550093+0000", 2024-09-13T11:24:38.871 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0026726570000000002 2024-09-13T11:24:38.871 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:38.871 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:38.871 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:38.871 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:33.646180+0000", 2024-09-13T11:24:38.871 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.096086798000000001 2024-09-13T11:24:38.871 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:38.871 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:38.871 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:24:38.871 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:33.646233+0000", 2024-09-13T11:24:38.871 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.3291000000000002e-05 2024-09-13T11:24:38.871 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:38.872 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:38.872 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:24:38.872 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:33.646275+0000", 2024-09-13T11:24:38.872 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.2367000000000003e-05 2024-09-13T11:24:38.872 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:38.872 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:24:38.872 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:38.872 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:38.872 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:38.872 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:115047 3.1e 3:791423c5:::benchmark_data_smithi177_31116_object115046:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e125)", 2024-09-13T11:24:38.872 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:24:33.584835+0000", 2024-09-13T11:24:38.872 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 5.2004644280000001, 2024-09-13T11:24:38.872 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.096461196999999999, 2024-09-13T11:24:38.873 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:24:38.873 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:24:38.873 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:38.873 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:24:38.873 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:24:38.873 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:38.873 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 115047 2024-09-13T11:24:38.873 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:38.873 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:24:38.873 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:38.873 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:24:38.873 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:33.584835+0000", 2024-09-13T11:24:38.873 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:38.873 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:38.874 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:38.874 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:24:38.874 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:33.584835+0000", 2024-09-13T11:24:38.874 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:38.874 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:38.874 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:38.874 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:24:38.874 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:33.584837+0000", 2024-09-13T11:24:38.874 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.843e-06 2024-09-13T11:24:38.874 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:38.874 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:38.874 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:24:38.874 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:33.584881+0000", 2024-09-13T11:24:38.874 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.3924999999999997e-05 2024-09-13T11:24:38.875 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:38.875 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:38.875 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:24:38.875 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:33.584892+0000", 2024-09-13T11:24:38.875 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.1450999999999999e-05 2024-09-13T11:24:38.875 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:38.875 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:38.875 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:24:38.875 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:33.584900+0000", 2024-09-13T11:24:38.875 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.5100000000000001e-06 2024-09-13T11:24:38.875 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:38.875 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:38.875 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:24:38.876 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:33.584955+0000", 2024-09-13T11:24:38.876 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.5377999999999997e-05 2024-09-13T11:24:38.876 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:38.876 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:38.876 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:24:38.876 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:33.585103+0000", 2024-09-13T11:24:38.876 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000147878 2024-09-13T11:24:38.876 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:38.876 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:38.876 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 4", 2024-09-13T11:24:38.876 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:33.585267+0000", 2024-09-13T11:24:38.876 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00016389900000000001 2024-09-13T11:24:38.876 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:38.876 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:38.877 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:24:38.877 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:33.587429+0000", 2024-09-13T11:24:38.877 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00216241 2024-09-13T11:24:38.877 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:38.877 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:38.877 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:38.877 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:33.681202+0000", 2024-09-13T11:24:38.877 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.093772890999999997 2024-09-13T11:24:38.877 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:38.877 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:38.877 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:24:38.877 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:33.681263+0000", 2024-09-13T11:24:38.877 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.0606999999999998e-05 2024-09-13T11:24:38.877 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:38.878 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:38.878 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:24:38.878 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:33.681296+0000", 2024-09-13T11:24:38.878 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.3405000000000003e-05 2024-09-13T11:24:38.878 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:38.878 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:24:38.878 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:38.878 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:38.878 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:38.878 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop(client.4665.0:115089 3.28 e125/120 3:15b13357:::benchmark_data_smithi177_31116_object115088:head v 125'4843, mlcod=47'620)", 2024-09-13T11:24:38.878 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:24:33.642654+0000", 2024-09-13T11:24:38.878 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 5.1426449959999996, 2024-09-13T11:24:38.878 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.19312220099999999, 2024-09-13T11:24:38.878 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:24:38.879 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:24:38.879 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:38.879 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:24:38.879 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:38.879 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:24:38.879 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:33.642654+0000", 2024-09-13T11:24:38.879 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:38.879 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:38.879 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:38.879 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:24:38.879 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:33.642654+0000", 2024-09-13T11:24:38.879 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:38.879 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:38.879 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:38.880 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:24:38.880 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:33.642656+0000", 2024-09-13T11:24:38.880 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.869e-06 2024-09-13T11:24:38.880 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:38.880 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:38.880 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:24:38.880 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:33.642688+0000", 2024-09-13T11:24:38.880 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.2169999999999999e-05 2024-09-13T11:24:38.880 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:38.880 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:38.880 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:24:38.880 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:33.832657+0000", 2024-09-13T11:24:38.880 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.18996885099999999 2024-09-13T11:24:38.881 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:38.881 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:38.881 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:24:38.881 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:33.832678+0000", 2024-09-13T11:24:38.881 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.0222000000000001e-05 2024-09-13T11:24:38.881 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:38.881 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:38.881 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:24:38.881 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:33.832760+0000", 2024-09-13T11:24:38.881 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.2113000000000005e-05 2024-09-13T11:24:38.881 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:38.881 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:38.881 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:24:38.881 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:33.832788+0000", 2024-09-13T11:24:38.882 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.8095000000000001e-05 2024-09-13T11:24:38.882 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:38.882 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:38.882 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:24:38.882 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:33.835729+0000", 2024-09-13T11:24:38.882 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0029408920000000001 2024-09-13T11:24:38.882 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:38.882 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:38.882 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:24:38.882 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:33.835777+0000", 2024-09-13T11:24:38.882 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.7988999999999997e-05 2024-09-13T11:24:38.882 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:38.882 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:24:38.883 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:38.883 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:38.883 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:38.883 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:115091 3.1e 3:7d850286:::benchmark_data_smithi177_31116_object115090:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e125)", 2024-09-13T11:24:38.883 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:24:33.643826+0000", 2024-09-13T11:24:38.883 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 5.1414739489999999, 2024-09-13T11:24:38.883 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.10297139199999999, 2024-09-13T11:24:38.883 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:24:38.883 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:24:38.883 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:38.883 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:24:38.883 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:24:38.883 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:38.884 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 115091 2024-09-13T11:24:38.884 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:38.884 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:24:38.884 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:38.884 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:24:38.884 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:33.643826+0000", 2024-09-13T11:24:38.884 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:38.884 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:38.884 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:38.884 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:24:38.884 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:33.643826+0000", 2024-09-13T11:24:38.884 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:38.884 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:38.884 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:38.885 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:24:38.885 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:33.643827+0000", 2024-09-13T11:24:38.885 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.8339999999999999e-06 2024-09-13T11:24:38.885 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:38.885 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:38.885 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:24:38.885 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:33.643874+0000", 2024-09-13T11:24:38.885 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.6579999999999998e-05 2024-09-13T11:24:38.885 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:38.885 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:38.885 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:24:38.885 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:33.643889+0000", 2024-09-13T11:24:38.885 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.4601e-05 2024-09-13T11:24:38.885 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:38.886 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:38.886 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:24:38.886 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:33.643895+0000", 2024-09-13T11:24:38.886 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.4439999999999996e-06 2024-09-13T11:24:38.886 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:38.886 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:38.886 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:24:38.886 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:33.643949+0000", 2024-09-13T11:24:38.886 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.4150999999999997e-05 2024-09-13T11:24:38.886 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:38.886 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:38.886 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:24:38.886 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:33.644110+0000", 2024-09-13T11:24:38.886 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00016054799999999999 2024-09-13T11:24:38.887 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:38.887 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:38.887 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 4", 2024-09-13T11:24:38.887 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:33.644254+0000", 2024-09-13T11:24:38.887 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00014396299999999999 2024-09-13T11:24:38.887 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:38.887 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:38.887 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:24:38.887 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:33.646290+0000", 2024-09-13T11:24:38.887 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0020365890000000001 2024-09-13T11:24:38.887 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:38.887 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:38.887 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:38.887 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:33.746691+0000", 2024-09-13T11:24:38.888 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.100400906 2024-09-13T11:24:38.888 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:38.888 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:38.888 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:24:38.888 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:33.746750+0000", 2024-09-13T11:24:38.888 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.8622000000000002e-05 2024-09-13T11:24:38.888 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:38.888 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:38.888 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:24:38.888 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:33.746797+0000", 2024-09-13T11:24:38.888 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.7154000000000003e-05 2024-09-13T11:24:38.888 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:38.888 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:24:38.889 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:38.889 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:38.889 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:38.889 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop(client.4665.0:115281 3.28 e125/120 3:17aeecb4:::benchmark_data_smithi177_31116_object115280:head v 125'4845, mlcod=47'620)", 2024-09-13T11:24:38.889 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:24:33.852156+0000", 2024-09-13T11:24:38.889 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 4.9331435959999999, 2024-09-13T11:24:38.889 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.16574401999999999, 2024-09-13T11:24:38.889 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:24:38.889 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:24:38.889 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:38.889 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:24:38.889 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:38.889 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:24:38.889 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:33.852156+0000", 2024-09-13T11:24:38.890 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:38.890 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:38.890 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:38.890 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:24:38.890 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:33.852156+0000", 2024-09-13T11:24:38.890 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:38.890 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:38.890 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:38.890 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:24:38.890 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:33.852158+0000", 2024-09-13T11:24:38.890 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.683e-06 2024-09-13T11:24:38.890 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:38.890 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:38.890 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:24:38.891 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:33.852184+0000", 2024-09-13T11:24:38.891 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.6241000000000002e-05 2024-09-13T11:24:38.891 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:38.891 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:38.891 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:24:38.891 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:34.015751+0000", 2024-09-13T11:24:38.891 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.16356685800000001 2024-09-13T11:24:38.891 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:38.891 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:38.891 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:24:38.891 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:34.015757+0000", 2024-09-13T11:24:38.891 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.8980000000000001e-06 2024-09-13T11:24:38.891 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:38.891 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:38.892 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:24:38.892 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:34.015800+0000", 2024-09-13T11:24:38.892 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.3371999999999997e-05 2024-09-13T11:24:38.892 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:38.892 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:38.892 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:24:38.892 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:34.015822+0000", 2024-09-13T11:24:38.892 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.2565000000000002e-05 2024-09-13T11:24:38.892 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:38.892 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:38.892 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:24:38.892 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:34.017861+0000", 2024-09-13T11:24:38.892 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0020388469999999999 2024-09-13T11:24:38.892 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:38.893 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:38.893 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:24:38.893 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:34.017900+0000", 2024-09-13T11:24:38.893 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.8556000000000002e-05 2024-09-13T11:24:38.893 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:38.893 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:24:38.893 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:38.893 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:38.893 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:24:38.893 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-09-13T11:24:38.900 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-09-13T11:24:38.902 INFO:tasks.ceph.osd.12.smithi001.stderr:2024-09-13T11:24:38.895+0000 7f892c6e6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26314) UID: 0 2024-09-13T11:24:38.939 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:404fbe60:::benchmark_data_smithi177_31116_object23568:head# 2024-09-13T11:24:38.939 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.939 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4050a7a7:::benchmark_data_smithi177_31116_object47856:head# 2024-09-13T11:24:38.939 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.940 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:40511b0b:::benchmark_data_smithi177_31116_object53328:head# 2024-09-13T11:24:38.940 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.940 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:40512b53:::benchmark_data_smithi177_31116_object103735:head# 2024-09-13T11:24:38.940 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.940 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:40521972:::benchmark_data_smithi177_31116_object87795:head# 2024-09-13T11:24:38.940 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.940 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4053e23b:::benchmark_data_smithi177_31116_object5178:head# 2024-09-13T11:24:38.940 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.940 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4053fbe5:::benchmark_data_smithi177_31116_object37987:head# 2024-09-13T11:24:38.940 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.940 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:40541404:::benchmark_data_smithi177_31116_object11947:head# 2024-09-13T11:24:38.940 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.940 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:405467dd:::benchmark_data_smithi177_31116_object30559:head# 2024-09-13T11:24:38.941 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.941 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4054a079:::benchmark_data_smithi177_31116_object6529:head# 2024-09-13T11:24:38.941 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.941 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4054a638:::benchmark_data_smithi177_31116_object18683:head# 2024-09-13T11:24:38.941 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.941 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4054fc5b:::benchmark_data_smithi177_31116_object19605:head# 2024-09-13T11:24:38.941 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.941 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:405565e3:::benchmark_data_smithi177_31116_object6965:head# 2024-09-13T11:24:38.941 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.941 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:40558416:::benchmark_data_smithi177_31116_object90983:head# 2024-09-13T11:24:38.941 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.941 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:40578804:::benchmark_data_smithi177_31116_object67465:head# 2024-09-13T11:24:38.941 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.942 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4057c338:::benchmark_data_smithi177_31116_object3442:head# 2024-09-13T11:24:38.942 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.942 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:40580b7d:::benchmark_data_smithi177_31116_object79080:head# 2024-09-13T11:24:38.942 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.942 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:405830c3:::benchmark_data_smithi177_31116_object96166:head# 2024-09-13T11:24:38.942 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.942 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4058e553:::benchmark_data_smithi177_31116_object88690:head# 2024-09-13T11:24:38.942 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.942 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4058f3d1:::benchmark_data_smithi177_31116_object99612:head# 2024-09-13T11:24:38.942 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.942 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:405915a7:::benchmark_data_smithi177_31116_object64426:head# 2024-09-13T11:24:38.942 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.942 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:40595787:::benchmark_data_smithi177_31116_object57149:head# 2024-09-13T11:24:38.943 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.943 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:405993e2:::benchmark_data_smithi177_31116_object39203:head# 2024-09-13T11:24:38.943 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.943 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4059dada:::benchmark_data_smithi177_31116_object16838:head# 2024-09-13T11:24:38.943 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.943 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:405a00c7:::benchmark_data_smithi177_31116_object23300:head# 2024-09-13T11:24:38.943 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.943 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:405b6806:::benchmark_data_smithi177_31116_object44618:head# 2024-09-13T11:24:38.943 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.943 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:405d6f88:::benchmark_data_smithi177_31116_object16298:head# 2024-09-13T11:24:38.943 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.943 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:405fbf73:::benchmark_data_smithi177_31116_object95706:head# 2024-09-13T11:24:38.943 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.943 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:405fca5f:::benchmark_data_smithi177_31116_object97509:head# 2024-09-13T11:24:38.944 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.944 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4060180d:::benchmark_data_smithi177_31116_object26300:head# 2024-09-13T11:24:38.944 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.944 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:40608b41:::benchmark_data_smithi177_31116_object106143:head# 2024-09-13T11:24:38.944 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.944 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:40618c9a:::benchmark_data_smithi177_31116_object26181:head# 2024-09-13T11:24:38.944 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.944 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4061b84e:::benchmark_data_smithi177_31116_object97221:head# 2024-09-13T11:24:38.944 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.944 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4061d8dc:::benchmark_data_smithi177_31116_object105481:head# 2024-09-13T11:24:38.944 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.944 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4061e27f:::benchmark_data_smithi177_31116_object10225:head# 2024-09-13T11:24:38.944 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.945 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:40629ab5:::benchmark_data_smithi177_31116_object44859:head# 2024-09-13T11:24:38.945 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.945 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:406460ac:::benchmark_data_smithi177_31116_object82037:head# 2024-09-13T11:24:38.945 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.945 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:40648f3d:::benchmark_data_smithi177_31116_object96143:head# 2024-09-13T11:24:38.945 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.945 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:40654a05:::benchmark_data_smithi177_31116_object79975:head# 2024-09-13T11:24:38.945 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.945 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:406584a3:::benchmark_data_smithi177_31116_object17522:head# 2024-09-13T11:24:38.945 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.945 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:40660b32:::benchmark_data_smithi177_31116_object100364:head# 2024-09-13T11:24:38.945 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.945 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:40661b03:::benchmark_data_smithi177_31116_object45300:head# 2024-09-13T11:24:38.946 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.946 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:40681e2e:::benchmark_data_smithi177_31116_object77139:head# 2024-09-13T11:24:38.946 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.946 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:406856aa:::benchmark_data_smithi177_31116_object96753:head# 2024-09-13T11:24:38.946 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.946 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4068f898:::benchmark_data_smithi177_31116_object95892:head# 2024-09-13T11:24:38.946 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.946 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:406a1e36:::benchmark_data_smithi177_31116_object7726:head# 2024-09-13T11:24:38.946 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.946 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:406af700:::benchmark_data_smithi177_31116_object38150:head# 2024-09-13T11:24:38.946 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.946 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:406b3b27:::benchmark_data_smithi177_31116_object95246:head# 2024-09-13T11:24:38.946 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.947 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:406b7b6c:::benchmark_data_smithi177_31116_object92039:head# 2024-09-13T11:24:38.947 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.947 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:406bc2cd:::benchmark_data_smithi177_31116_object21697:head# 2024-09-13T11:24:38.947 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.947 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:406c5c8c:::benchmark_data_smithi177_31116_object1723:head# 2024-09-13T11:24:38.947 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.947 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:406cef8a:::benchmark_data_smithi177_31116_object35316:head# 2024-09-13T11:24:38.947 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.947 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:406cfcc4:::benchmark_data_smithi177_31116_object12340:head# 2024-09-13T11:24:38.947 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.947 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:406e9f2d:::benchmark_data_smithi177_31116_object84610:head# 2024-09-13T11:24:38.947 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.947 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:406ed46f:::benchmark_data_smithi177_31116_object109484:head# 2024-09-13T11:24:38.948 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.948 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:406f007b:::benchmark_data_smithi177_31116_object24162:head# 2024-09-13T11:24:38.948 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.948 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:406f69e5:::benchmark_data_smithi177_31116_object63684:head# 2024-09-13T11:24:38.948 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.948 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:406f9ff9:::benchmark_data_smithi177_31116_object81703:head# 2024-09-13T11:24:38.948 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.948 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:40707a1f:::benchmark_data_smithi177_31116_object54782:head# 2024-09-13T11:24:38.948 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.948 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:40707ad3:::benchmark_data_smithi177_31116_object91144:head# 2024-09-13T11:24:38.948 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.948 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4070ad99:::benchmark_data_smithi177_31116_object21741:head# 2024-09-13T11:24:38.948 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.949 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4070f3cf:::benchmark_data_smithi177_31116_object93762:head# 2024-09-13T11:24:38.949 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.949 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4071052f:::benchmark_data_smithi177_31116_object13435:head# 2024-09-13T11:24:38.949 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.949 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4071266a:::benchmark_data_smithi177_31116_object105287:head# 2024-09-13T11:24:38.949 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.949 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:40744611:::benchmark_data_smithi177_31116_object90652:head# 2024-09-13T11:24:38.949 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.949 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4074ef43:::benchmark_data_smithi177_31116_object64357:head# 2024-09-13T11:24:38.949 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.949 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:40784398:::benchmark_data_smithi177_31116_object82598:head# 2024-09-13T11:24:38.949 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.949 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:407873b2:::benchmark_data_smithi177_31116_object35539:head# 2024-09-13T11:24:38.949 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.950 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4079a4a7:::benchmark_data_smithi177_31116_object9064:head# 2024-09-13T11:24:38.950 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.950 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:407b61eb:::benchmark_data_smithi177_31116_object76617:head# 2024-09-13T11:24:38.950 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.950 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:407bb35d:::benchmark_data_smithi177_31116_object81599:head# 2024-09-13T11:24:38.950 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.950 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:407c9472:::benchmark_data_smithi177_31116_object31988:head# 2024-09-13T11:24:38.950 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.950 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:407cac0e:::benchmark_data_smithi177_31116_object102091:head# 2024-09-13T11:24:38.950 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.950 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:407d4821:::benchmark_data_smithi177_31116_object98908:head# 2024-09-13T11:24:38.950 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.950 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:407d805a:::benchmark_data_smithi177_31116_object44000:head# 2024-09-13T11:24:38.951 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.951 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:407d8c80:::benchmark_data_smithi177_31116_object85660:head# 2024-09-13T11:24:38.951 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.951 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:407d9c7c:::benchmark_data_smithi177_31116_object48505:head# 2024-09-13T11:24:38.951 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.951 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:407ea4da:::benchmark_data_smithi177_31116_object66606:head# 2024-09-13T11:24:38.951 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.951 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:407ebd4c:::benchmark_data_smithi177_31116_object26432:head# 2024-09-13T11:24:38.951 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.951 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:407f3e8d:::benchmark_data_smithi177_31116_object74793:head# 2024-09-13T11:24:38.951 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.951 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:408035a3:::benchmark_data_smithi177_31116_object109727:head# 2024-09-13T11:24:38.951 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.951 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:40820a4f:::benchmark_data_smithi177_31116_object87769:head# 2024-09-13T11:24:38.952 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.952 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4082f214:::benchmark_data_smithi177_31116_object103403:head# 2024-09-13T11:24:38.952 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.952 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:408400ab:::benchmark_data_smithi177_31116_object16748:head# 2024-09-13T11:24:38.952 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.952 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:40849903:::benchmark_data_smithi177_31116_object31794:head# 2024-09-13T11:24:38.952 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.952 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4084b0be:::benchmark_data_smithi177_31116_object47964:head# 2024-09-13T11:24:38.952 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.952 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:408579ea:::benchmark_data_smithi177_31116_object86266:head# 2024-09-13T11:24:38.952 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.952 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:40866b6b:::benchmark_data_smithi177_31116_object66543:head# 2024-09-13T11:24:38.952 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.953 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:40868ffb:::benchmark_data_smithi177_31116_object12160:head# 2024-09-13T11:24:38.953 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.953 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4088abd4:::benchmark_data_smithi177_31116_object16799:head# 2024-09-13T11:24:38.953 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.953 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4088b530:::benchmark_data_smithi177_31116_object27051:head# 2024-09-13T11:24:38.953 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.953 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4088f40d:::benchmark_data_smithi177_31116_object58187:head# 2024-09-13T11:24:38.953 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.953 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4089983f:::benchmark_data_smithi177_31116_object97006:head# 2024-09-13T11:24:38.953 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.953 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:408a6711:::benchmark_data_smithi177_31116_object74792:head# 2024-09-13T11:24:38.953 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.953 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:408ae0aa:::benchmark_data_smithi177_31116_object69156:head# 2024-09-13T11:24:38.953 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.954 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:408aee5d:::benchmark_data_smithi177_31116_object26857:head# 2024-09-13T11:24:38.954 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.954 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:408b703e:::benchmark_data_smithi177_31116_object84690:head# 2024-09-13T11:24:38.954 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.954 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:408bbdaf:::benchmark_data_smithi177_31116_object18078:head# 2024-09-13T11:24:38.954 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.954 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:408bcede:::benchmark_data_smithi177_31116_object71959:head# 2024-09-13T11:24:38.954 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.954 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:408c2697:::benchmark_data_smithi177_31116_object36451:head# 2024-09-13T11:24:38.954 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.954 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:408c2f58:::benchmark_data_smithi177_31116_object18693:head# 2024-09-13T11:24:38.954 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.954 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:408cb6df:::benchmark_data_smithi177_31116_object83595:head# 2024-09-13T11:24:38.955 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.955 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:408edef3:::benchmark_data_smithi177_31116_object74445:head# 2024-09-13T11:24:38.955 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.955 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:408f0315:::benchmark_data_smithi177_31116_object90888:head# 2024-09-13T11:24:38.955 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.955 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:408f15a5:::benchmark_data_smithi177_31116_object65625:head# 2024-09-13T11:24:38.955 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.955 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:408f4706:::benchmark_data_smithi177_31116_object55363:head# 2024-09-13T11:24:38.955 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.955 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:408f7efe:::benchmark_data_smithi177_31116_object36890:head# 2024-09-13T11:24:38.955 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.955 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:409105a5:::benchmark_data_smithi177_31116_object64379:head# 2024-09-13T11:24:38.955 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.955 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4091a60c:::benchmark_data_smithi177_31116_object2755:head# 2024-09-13T11:24:38.956 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.956 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4092dfc2:::benchmark_data_smithi177_31116_object55389:head# 2024-09-13T11:24:38.956 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.956 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4093efd1:::benchmark_data_smithi177_31116_object76568:head# 2024-09-13T11:24:38.956 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.956 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4095a75c:::benchmark_data_smithi177_31116_object725:head# 2024-09-13T11:24:38.956 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.956 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:40971cef:::benchmark_data_smithi177_31116_object931:head# 2024-09-13T11:24:38.956 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.956 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:40979a20:::benchmark_data_smithi177_31116_object83710:head# 2024-09-13T11:24:38.956 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.956 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:40980402:::benchmark_data_smithi177_31116_object37357:head# 2024-09-13T11:24:38.956 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.957 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:40981b34:::benchmark_data_smithi177_31116_object100668:head# 2024-09-13T11:24:38.957 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.957 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:40994b44:::benchmark_data_smithi177_31116_object54692:head# 2024-09-13T11:24:38.957 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.957 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:40998238:::benchmark_data_smithi177_31116_object30010:head# 2024-09-13T11:24:38.957 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.957 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:409a19ba:::benchmark_data_smithi177_31116_object93749:head# 2024-09-13T11:24:38.957 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.957 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:409abf85:::benchmark_data_smithi177_31116_object6028:head# 2024-09-13T11:24:38.957 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.957 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:409b286b:::benchmark_data_smithi177_31116_object82725:head# 2024-09-13T11:24:38.957 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.957 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:409c8657:::benchmark_data_smithi177_31116_object79371:head# 2024-09-13T11:24:38.958 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.958 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:409ce5d8:::benchmark_data_smithi177_31116_object36032:head# 2024-09-13T11:24:38.958 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.958 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:409d3b63:::benchmark_data_smithi177_31116_object89524:head# 2024-09-13T11:24:38.958 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.958 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:409d6ad2:::benchmark_data_smithi177_31116_object97365:head# 2024-09-13T11:24:38.958 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.958 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:409e297b:::benchmark_data_smithi177_31116_object195:head# 2024-09-13T11:24:38.958 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.958 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:409e6f98:::benchmark_data_smithi177_31116_object80568:head# 2024-09-13T11:24:38.958 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.958 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:409ec20a:::benchmark_data_smithi177_31116_object84640:head# 2024-09-13T11:24:38.958 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.958 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:409fa531:::benchmark_data_smithi177_31116_object92802:head# 2024-09-13T11:24:38.959 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.959 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:409fbcbc:::benchmark_data_smithi177_31116_object52019:head# 2024-09-13T11:24:38.959 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.959 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:40a01dbd:::benchmark_data_smithi177_31116_object3315:head# 2024-09-13T11:24:38.959 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.959 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:40a04f16:::benchmark_data_smithi177_31116_object80186:head# 2024-09-13T11:24:38.959 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.959 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:40a157c5:::benchmark_data_smithi177_31116_object66904:head# 2024-09-13T11:24:38.959 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.959 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:40a19f93:::benchmark_data_smithi177_31116_object31776:head# 2024-09-13T11:24:38.959 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.959 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:40a35ddc:::benchmark_data_smithi177_31116_object69248:head# 2024-09-13T11:24:38.959 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.960 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:40a3fc7e:::benchmark_data_smithi177_31116_object106718:head# 2024-09-13T11:24:38.960 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.960 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:40a4ddff:::benchmark_data_smithi177_31116_object28485:head# 2024-09-13T11:24:38.960 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.960 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:40a510f5:::benchmark_data_smithi177_31116_object24708:head# 2024-09-13T11:24:38.960 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.960 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:40a5d6b1:::benchmark_data_smithi177_31116_object42959:head# 2024-09-13T11:24:38.960 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.960 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:40a66d2a:::benchmark_data_smithi177_31116_object52316:head# 2024-09-13T11:24:38.960 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.960 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:40a670f4:::benchmark_data_smithi177_31116_object50155:head# 2024-09-13T11:24:38.960 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.960 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:40a6c2d1:::benchmark_data_smithi177_31116_object23636:head# 2024-09-13T11:24:38.961 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.961 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:40a7189f:::benchmark_data_smithi177_31116_object90532:head# 2024-09-13T11:24:38.961 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.961 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:40a7ca54:::benchmark_data_smithi177_31116_object45534:head# 2024-09-13T11:24:38.961 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.961 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:40a91169:::benchmark_data_smithi177_31116_object95088:head# 2024-09-13T11:24:38.961 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.961 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:40a92a5e:::benchmark_data_smithi177_31116_object41648:head# 2024-09-13T11:24:38.961 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.961 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:40a9552f:::benchmark_data_smithi177_31116_object4703:head# 2024-09-13T11:24:38.961 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.961 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:40aa2bb6:::benchmark_data_smithi177_31116_object68206:head# 2024-09-13T11:24:38.961 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.961 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:40abefe4:::benchmark_data_smithi177_31116_object77897:head# 2024-09-13T11:24:38.962 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.962 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:40ac4d19:::benchmark_data_smithi177_31116_object65373:head# 2024-09-13T11:24:38.962 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.962 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:40ac5e68:::benchmark_data_smithi177_31116_object63862:head# 2024-09-13T11:24:38.962 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.962 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:40ac8f47:::benchmark_data_smithi177_31116_object43057:head# 2024-09-13T11:24:38.962 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.962 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:40ad487c:::benchmark_data_smithi177_31116_object64366:head# 2024-09-13T11:24:38.962 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.962 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:40adf2c0:::benchmark_data_smithi177_31116_object2324:head# 2024-09-13T11:24:38.962 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.962 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:40ae4349:::benchmark_data_smithi177_31116_object72880:head# 2024-09-13T11:24:38.962 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.963 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:40ae578b:::benchmark_data_smithi177_31116_object77370:head# 2024-09-13T11:24:38.963 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.963 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:40ae8d8f:::benchmark_data_smithi177_31116_object82993:head# 2024-09-13T11:24:38.963 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.963 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:40b0e1d8:::benchmark_data_smithi177_31116_object83481:head# 2024-09-13T11:24:38.963 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.963 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:40b404e9:::benchmark_data_smithi177_31116_object82223:head# 2024-09-13T11:24:38.963 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.963 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:40b44312:::benchmark_data_smithi177_31116_object63807:head# 2024-09-13T11:24:38.963 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.963 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:40b47f82:::benchmark_data_smithi177_31116_object65629:head# 2024-09-13T11:24:38.963 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.963 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:40b53c00:::benchmark_data_smithi177_31116_object31972:head# 2024-09-13T11:24:38.963 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.964 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:40b59db7:::benchmark_data_smithi177_31116_object77737:head# 2024-09-13T11:24:38.964 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.964 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:40b644b3:::benchmark_data_smithi177_31116_object40111:head# 2024-09-13T11:24:38.964 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.964 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:40b6ee61:::benchmark_data_smithi177_31116_object11898:head# 2024-09-13T11:24:38.964 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.964 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:40b897c7:::benchmark_data_smithi177_31116_object31108:head# 2024-09-13T11:24:38.964 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.964 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:40b98f76:::benchmark_data_smithi177_31116_object52754:head# 2024-09-13T11:24:38.964 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.964 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:40b9f246:::benchmark_data_smithi177_31116_object98453:head# 2024-09-13T11:24:38.964 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.964 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:40baa63f:::benchmark_data_smithi177_31116_object13190:head# 2024-09-13T11:24:38.965 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.965 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:40bacac5:::benchmark_data_smithi177_31116_object80088:head# 2024-09-13T11:24:38.965 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.965 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:40bafaff:::benchmark_data_smithi177_31116_object85911:head# 2024-09-13T11:24:38.965 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.965 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:40bb6c7e:::benchmark_data_smithi177_31116_object45109:head# 2024-09-13T11:24:38.965 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.965 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:40bc66b9:::benchmark_data_smithi177_31116_object54201:head# 2024-09-13T11:24:38.965 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.965 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:40bc9cf4:::benchmark_data_smithi177_31116_object32031:head# 2024-09-13T11:24:38.965 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.965 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:40bce023:::benchmark_data_smithi177_31116_object11401:head# 2024-09-13T11:24:38.965 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.965 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:40bd1e0f:::benchmark_data_smithi177_31116_object70709:head# 2024-09-13T11:24:38.966 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.966 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:40bdd695:::benchmark_data_smithi177_31116_object3743:head# 2024-09-13T11:24:38.966 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.966 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:40be5f7e:::benchmark_data_smithi177_31116_object76016:head# 2024-09-13T11:24:38.966 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.966 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:40be629e:::benchmark_data_smithi177_31116_object14956:head# 2024-09-13T11:24:38.966 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.966 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:40bf26c0:::benchmark_data_smithi177_31116_object40744:head# 2024-09-13T11:24:38.966 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.966 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:40bfb8fd:::benchmark_data_smithi177_31116_object103063:head# 2024-09-13T11:24:38.966 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.966 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:40c072cc:::benchmark_data_smithi177_31116_object62579:head# 2024-09-13T11:24:38.966 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.967 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:40c250a3:::benchmark_data_smithi177_31116_object85596:head# 2024-09-13T11:24:38.967 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.967 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:40c2e91e:::benchmark_data_smithi177_31116_object20250:head# 2024-09-13T11:24:38.967 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.967 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:40c4535c:::benchmark_data_smithi177_31116_object102231:head# 2024-09-13T11:24:38.967 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.967 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:40c469c0:::benchmark_data_smithi177_31116_object57322:head# 2024-09-13T11:24:38.967 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.967 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:40c69484:::benchmark_data_smithi177_31116_object101907:head# 2024-09-13T11:24:38.967 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.967 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:40c7f056:::benchmark_data_smithi177_31116_object64349:head# 2024-09-13T11:24:38.967 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.967 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:40c98312:::benchmark_data_smithi177_31116_object24199:head# 2024-09-13T11:24:38.967 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.968 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:40c9e423:::benchmark_data_smithi177_31116_object81904:head# 2024-09-13T11:24:38.968 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.968 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:40ca46e8:::benchmark_data_smithi177_31116_object71693:head# 2024-09-13T11:24:38.968 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.968 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:40caa5fb:::benchmark_data_smithi177_31116_object91362:head# 2024-09-13T11:24:38.968 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.968 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:40cb5e0b:::benchmark_data_smithi177_31116_object70603:head# 2024-09-13T11:24:38.968 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.968 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:40cbf191:::benchmark_data_smithi177_31116_object2942:head# 2024-09-13T11:24:38.968 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.968 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:40cc41fd:::benchmark_data_smithi177_31116_object54498:head# 2024-09-13T11:24:38.968 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.968 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:40ccf75f:::benchmark_data_smithi177_31116_object14136:head# 2024-09-13T11:24:38.969 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.969 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:40cd46ff:::benchmark_data_smithi177_31116_object45399:head# 2024-09-13T11:24:38.969 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.969 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:40cd9401:::benchmark_data_smithi177_31116_object25170:head# 2024-09-13T11:24:38.969 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.969 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:40ce5330:::benchmark_data_smithi177_31116_object38763:head# 2024-09-13T11:24:38.969 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.969 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:40ce8113:::benchmark_data_smithi177_31116_object89456:head# 2024-09-13T11:24:38.969 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.969 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:40cef723:::benchmark_data_smithi177_31116_object72989:head# 2024-09-13T11:24:38.969 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.969 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:40cfd2be:::benchmark_data_smithi177_31116_object66962:head# 2024-09-13T11:24:38.969 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.969 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:40d157b2:::benchmark_data_smithi177_31116_object23765:head# 2024-09-13T11:24:38.970 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.970 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:40d19e35:::benchmark_data_smithi177_31116_object24612:head# 2024-09-13T11:24:38.970 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.970 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:40d21263:::benchmark_data_smithi177_31116_object76792:head# 2024-09-13T11:24:38.970 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.970 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:40d2355c:::benchmark_data_smithi177_31116_object82745:head# 2024-09-13T11:24:38.970 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.970 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:40d269c9:::benchmark_data_smithi177_31116_object74104:head# 2024-09-13T11:24:38.970 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.970 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:40d37e45:::benchmark_data_smithi177_31116_object9502:head# 2024-09-13T11:24:38.970 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.970 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:40d4196a:::benchmark_data_smithi177_31116_object45744:head# 2024-09-13T11:24:38.970 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.971 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:40d48d62:::benchmark_data_smithi177_31116_object856:head# 2024-09-13T11:24:38.971 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.971 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:40d49f6a:::benchmark_data_smithi177_31116_object31712:head# 2024-09-13T11:24:38.971 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.971 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:40d58aaa:::benchmark_data_smithi177_31116_object72376:head# 2024-09-13T11:24:38.971 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.971 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:40d5bf42:::benchmark_data_smithi177_31116_object108381:head# 2024-09-13T11:24:38.971 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.971 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:40d61d2c:::benchmark_data_smithi177_31116_object75020:head# 2024-09-13T11:24:38.971 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.971 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:40d6eb57:::benchmark_data_smithi177_31116_object108435:head# 2024-09-13T11:24:38.971 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.971 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:40d73a9e:::benchmark_data_smithi177_31116_object25011:head# 2024-09-13T11:24:38.971 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.972 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:40d77456:::benchmark_data_smithi177_31116_object21059:head# 2024-09-13T11:24:38.972 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.972 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:40d979fc:::benchmark_data_smithi177_31116_object94588:head# 2024-09-13T11:24:38.972 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.972 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:40da7c14:::benchmark_data_smithi177_31116_object45337:head# 2024-09-13T11:24:38.972 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.972 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:40dba6b8:::benchmark_data_smithi177_31116_object5036:head# 2024-09-13T11:24:38.972 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.972 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:40dc2b82:::benchmark_data_smithi177_31116_object50636:head# 2024-09-13T11:24:38.972 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.972 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:40dd559a:::benchmark_data_smithi177_31116_object55741:head# 2024-09-13T11:24:38.972 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.972 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:40dd634d:::benchmark_data_smithi177_31116_object22990:head# 2024-09-13T11:24:38.973 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.973 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:40dfaaf1:::benchmark_data_smithi177_31116_object97137:head# 2024-09-13T11:24:38.973 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.973 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:40dfb7d2:::benchmark_data_smithi177_31116_object70789:head# 2024-09-13T11:24:38.973 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.973 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:40e147d1:::benchmark_data_smithi177_31116_object48619:head# 2024-09-13T11:24:38.973 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.973 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:40e26ce1:::benchmark_data_smithi177_31116_object88089:head# 2024-09-13T11:24:38.973 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.973 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:40e2990b:::benchmark_data_smithi177_31116_object38783:head# 2024-09-13T11:24:38.973 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.973 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:40e54bc3:::benchmark_data_smithi177_31116_object42286:head# 2024-09-13T11:24:38.973 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.973 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:40e5a247:::benchmark_data_smithi177_31116_object10549:head# 2024-09-13T11:24:38.974 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.974 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:40e653f5:::benchmark_data_smithi177_31116_object14636:head# 2024-09-13T11:24:38.974 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.974 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:40e6723b:::benchmark_data_smithi177_31116_object66668:head# 2024-09-13T11:24:38.974 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.974 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:40e699f3:::benchmark_data_smithi177_31116_object95980:head# 2024-09-13T11:24:38.974 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.974 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:40e6f2bb:::benchmark_data_smithi177_31116_object13093:head# 2024-09-13T11:24:38.974 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.974 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:40e704b9:::benchmark_data_smithi177_31116_object44649:head# 2024-09-13T11:24:38.974 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.974 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:40e8807a:::benchmark_data_smithi177_31116_object12422:head# 2024-09-13T11:24:38.974 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.974 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:40e8a748:::benchmark_data_smithi177_31116_object15914:head# 2024-09-13T11:24:38.975 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.975 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:40e90991:::benchmark_data_smithi177_31116_object77215:head# 2024-09-13T11:24:38.975 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.975 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:40e9c2e9:::benchmark_data_smithi177_31116_object17667:head# 2024-09-13T11:24:38.975 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.975 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:40ea775c:::benchmark_data_smithi177_31116_object91454:head# 2024-09-13T11:24:38.975 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.975 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:40eafb7a:::benchmark_data_smithi177_31116_object96536:head# 2024-09-13T11:24:38.975 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.981 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:40ebb634:::benchmark_data_smithi177_31116_object33461:head# 2024-09-13T11:24:38.981 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.981 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:40ecb240:::benchmark_data_smithi177_31116_object81158:head# 2024-09-13T11:24:38.981 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.981 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:40ee042d:::benchmark_data_smithi177_31116_object95483:head# 2024-09-13T11:24:38.982 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.982 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:40eee266:::benchmark_data_smithi177_31116_object32683:head# 2024-09-13T11:24:38.982 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.982 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:40ef362a:::benchmark_data_smithi177_31116_object102330:head# 2024-09-13T11:24:38.982 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.982 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:40efa513:::benchmark_data_smithi177_31116_object96544:head# 2024-09-13T11:24:38.982 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.982 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:40f0daf8:::benchmark_data_smithi177_31116_object11622:head# 2024-09-13T11:24:38.982 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.982 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:40f16f5d:::benchmark_data_smithi177_31116_object91957:head# 2024-09-13T11:24:38.982 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.982 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:40f1c065:::benchmark_data_smithi177_31116_object11686:head# 2024-09-13T11:24:38.982 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.983 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:40f22173:::benchmark_data_smithi177_31116_object98167:head# 2024-09-13T11:24:38.983 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.983 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:40f285b0:::benchmark_data_smithi177_31116_object3333:head# 2024-09-13T11:24:38.983 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.983 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:40f2b542:::benchmark_data_smithi177_31116_object12818:head# 2024-09-13T11:24:38.983 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.983 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:40f3d0e3:::benchmark_data_smithi177_31116_object87326:head# 2024-09-13T11:24:38.983 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.983 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:40f53f15:::benchmark_data_smithi177_31116_object4244:head# 2024-09-13T11:24:38.983 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.983 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:40f5d2f7:::benchmark_data_smithi177_31116_object95634:head# 2024-09-13T11:24:38.983 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.983 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:40f5ed8c:::benchmark_data_smithi177_31116_object25946:head# 2024-09-13T11:24:38.984 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.984 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:40f62adf:::benchmark_data_smithi177_31116_object49555:head# 2024-09-13T11:24:38.984 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.984 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:40f690cd:::benchmark_data_smithi177_31116_object68466:head# 2024-09-13T11:24:38.984 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.984 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:40f69ea7:::benchmark_data_smithi177_31116_object100846:head# 2024-09-13T11:24:38.984 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.984 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:40f7a9f9:::benchmark_data_smithi177_31116_object64594:head# 2024-09-13T11:24:38.984 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.984 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:40f8606f:::benchmark_data_smithi177_31116_object87208:head# 2024-09-13T11:24:38.984 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.984 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:40f862e8:::benchmark_data_smithi177_31116_object104754:head# 2024-09-13T11:24:38.984 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.985 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:40f8f0ed:::benchmark_data_smithi177_31116_object54799:head# 2024-09-13T11:24:38.985 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.985 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:40f9b5eb:::benchmark_data_smithi177_31116_object14007:head# 2024-09-13T11:24:38.985 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.985 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:40faef97:::benchmark_data_smithi177_31116_object37970:head# 2024-09-13T11:24:38.985 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.985 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:40fb2e6b:::benchmark_data_smithi177_31116_object37727:head# 2024-09-13T11:24:38.985 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.985 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:40fb592a:::benchmark_data_smithi177_31116_object10094:head# 2024-09-13T11:24:38.985 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.985 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:40fb9ffb:::benchmark_data_smithi177_31116_object63547:head# 2024-09-13T11:24:38.985 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.985 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:40fef886:::benchmark_data_smithi177_31116_object40689:head# 2024-09-13T11:24:38.985 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.986 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:40ff4348:::benchmark_data_smithi177_31116_object104525:head# 2024-09-13T11:24:38.986 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.986 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4100804a:::benchmark_data_smithi177_31116_object8048:head# 2024-09-13T11:24:38.986 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.986 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:410222fb:::benchmark_data_smithi177_31116_object107822:head# 2024-09-13T11:24:38.986 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.986 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:410358f6:::benchmark_data_smithi177_31116_object18331:head# 2024-09-13T11:24:38.986 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.986 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:41036064:::benchmark_data_smithi177_31116_object80099:head# 2024-09-13T11:24:38.986 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.986 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4103d6a4:::benchmark_data_smithi177_31116_object88665:head# 2024-09-13T11:24:38.986 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.986 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4104510c:::benchmark_data_smithi177_31116_object23248:head# 2024-09-13T11:24:38.986 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.987 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:41048580:::benchmark_data_smithi177_31116_object100079:head# 2024-09-13T11:24:38.987 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.987 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4104e10a:::benchmark_data_smithi177_31116_object9721:head# 2024-09-13T11:24:38.987 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.987 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:410529b4:::benchmark_data_smithi177_31116_object56367:head# 2024-09-13T11:24:38.987 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.987 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:41055445:::benchmark_data_smithi177_31116_object3427:head# 2024-09-13T11:24:38.987 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.987 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:41059ab4:::benchmark_data_smithi177_31116_object96121:head# 2024-09-13T11:24:38.987 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.987 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4106c776:::benchmark_data_smithi177_31116_object15470:head# 2024-09-13T11:24:38.987 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.987 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:41073607:::benchmark_data_smithi177_31116_object108085:head# 2024-09-13T11:24:38.988 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.988 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:41077b96:::benchmark_data_smithi177_31116_object98414:head# 2024-09-13T11:24:38.988 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.988 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:410794c8:::benchmark_data_smithi177_31116_object91057:head# 2024-09-13T11:24:38.988 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.988 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:41079a1a:::benchmark_data_smithi177_31116_object6725:head# 2024-09-13T11:24:38.988 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.990 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4107a714:::benchmark_data_smithi177_31116_object76356:head# 2024-09-13T11:24:38.990 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.991 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4107e60f:::benchmark_data_smithi177_31116_object84706:head# 2024-09-13T11:24:38.991 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.991 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:41084616:::benchmark_data_smithi177_31116_object98478:head# 2024-09-13T11:24:38.991 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.991 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:41091bfd:::benchmark_data_smithi177_31116_object81942:head# 2024-09-13T11:24:38.991 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.991 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4109e623:::benchmark_data_smithi177_31116_object67809:head# 2024-09-13T11:24:38.991 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.991 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:410c0404:::benchmark_data_smithi177_31116_object81561:head# 2024-09-13T11:24:38.991 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.991 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:410c8219:::benchmark_data_smithi177_31116_object7250:head# 2024-09-13T11:24:38.991 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.991 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:410c992d:::benchmark_data_smithi177_31116_object39714:head# 2024-09-13T11:24:38.991 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.992 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:410ce493:::benchmark_data_smithi177_31116_object50593:head# 2024-09-13T11:24:38.992 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.992 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:410cfa7c:::benchmark_data_smithi177_31116_object56215:head# 2024-09-13T11:24:38.992 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.992 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:410eb040:::benchmark_data_smithi177_31116_object6210:head# 2024-09-13T11:24:38.992 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.992 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:410fbfc1:::benchmark_data_smithi177_31116_object90082:head# 2024-09-13T11:24:38.992 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.993 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:410fd7c4:::benchmark_data_smithi177_31116_object21257:head# 2024-09-13T11:24:38.993 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.995 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:410feb1b:::benchmark_data_smithi177_31116_object45780:head# 2024-09-13T11:24:38.995 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.997 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4112b23c:::benchmark_data_smithi177_31116_object47976:head# 2024-09-13T11:24:38.998 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.998 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4112f69b:::benchmark_data_smithi177_31116_object25097:head# 2024-09-13T11:24:38.998 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:38.999 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:411491ca:::benchmark_data_smithi177_31116_object90775:head# 2024-09-13T11:24:38.999 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.000 INFO:tasks.ceph.osd.11.smithi177.stderr:2024-09-13T11:24:38.995+0000 7f24792c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26580) UID: 0 2024-09-13T11:24:39.001 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4114bb51:::benchmark_data_smithi177_31116_object86788:head# 2024-09-13T11:24:39.001 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.004 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4114f495:::benchmark_data_smithi177_31116_object65389:head# 2024-09-13T11:24:39.004 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.007 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:41160910:::benchmark_data_smithi177_31116_object47555:head# 2024-09-13T11:24:39.007 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.007 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:411787b0:::benchmark_data_smithi177_31116_object49768:head# 2024-09-13T11:24:39.008 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.008 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4117fccb:::benchmark_data_smithi177_31116_object56436:head# 2024-09-13T11:24:39.008 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.009 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:411868a2:::benchmark_data_smithi177_31116_object85110:head# 2024-09-13T11:24:39.009 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.011 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4119065f:::benchmark_data_smithi177_31116_object83837:head# 2024-09-13T11:24:39.011 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.012 INFO:tasks.radosbench.radosbench.0.smithi177.stdout: 132 16 121298 121282 57.4189 65.9375 0.00349009 0.0174106 2024-09-13T11:24:39.012 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:41196f99:::benchmark_data_smithi177_31116_object95219:head# 2024-09-13T11:24:39.012 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.013 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T11:24:39.014 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4119fc4a:::benchmark_data_smithi177_31116_object40489:head# 2024-09-13T11:24:39.014 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.014 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:411afd7e:::benchmark_data_smithi177_31116_object28902:head# 2024-09-13T11:24:39.014 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.015 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:411bf70f:::benchmark_data_smithi177_31116_object104893:head# 2024-09-13T11:24:39.015 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.016 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:411bffee:::benchmark_data_smithi177_31116_object19199:head# 2024-09-13T11:24:39.016 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.018 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:411cfec6:::benchmark_data_smithi177_31116_object68174:head# 2024-09-13T11:24:39.018 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.018 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:411d78ef:::benchmark_data_smithi177_31116_object89997:head# 2024-09-13T11:24:39.018 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.019 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:411f322b:::benchmark_data_smithi177_31116_object25379:head# 2024-09-13T11:24:39.019 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.019 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:411f43d3:::benchmark_data_smithi177_31116_object4704:head# 2024-09-13T11:24:39.019 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.020 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:411fffc1:::benchmark_data_smithi177_31116_object44251:head# 2024-09-13T11:24:39.020 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.023 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4120dfc1:::benchmark_data_smithi177_31116_object46888:head# 2024-09-13T11:24:39.023 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.023 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4122457c:::benchmark_data_smithi177_31116_object12752:head# 2024-09-13T11:24:39.023 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.023 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4122de23:::benchmark_data_smithi177_31116_object76580:head# 2024-09-13T11:24:39.023 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.023 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:412357db:::benchmark_data_smithi177_31116_object102601:head# 2024-09-13T11:24:39.023 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.024 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4124b764:::benchmark_data_smithi177_31116_object109016:head# 2024-09-13T11:24:39.024 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.024 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4124d95c:::benchmark_data_smithi177_31116_object82864:head# 2024-09-13T11:24:39.024 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.025 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:41258fc9:::benchmark_data_smithi177_31116_object35297:head# 2024-09-13T11:24:39.025 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.025 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:41265e08:::benchmark_data_smithi177_31116_object55080:head# 2024-09-13T11:24:39.025 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.025 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4126dba6:::benchmark_data_smithi177_31116_object36800:head# 2024-09-13T11:24:39.026 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.026 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:412700ec:::benchmark_data_smithi177_31116_object74055:head# 2024-09-13T11:24:39.026 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.027 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:41276e8a:::benchmark_data_smithi177_31116_object10441:head# 2024-09-13T11:24:39.027 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.027 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4127ce2e:::benchmark_data_smithi177_31116_object65180:head# 2024-09-13T11:24:39.027 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.027 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4127e487:::benchmark_data_smithi177_31116_object49637:head# 2024-09-13T11:24:39.027 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.028 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4128e83d:::benchmark_data_smithi177_31116_object80479:head# 2024-09-13T11:24:39.028 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.028 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4128ed0b:::benchmark_data_smithi177_31116_object19765:head# 2024-09-13T11:24:39.028 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.028 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:412a1b4f:::benchmark_data_smithi177_31116_object103536:head# 2024-09-13T11:24:39.028 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.029 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:412a5ce2:::benchmark_data_smithi177_31116_object3342:head# 2024-09-13T11:24:39.029 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.029 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:412ad804:::benchmark_data_smithi177_31116_object67806:head# 2024-09-13T11:24:39.029 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.029 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:412aec0e:::benchmark_data_smithi177_31116_object100734:head# 2024-09-13T11:24:39.029 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.032 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:412b8a32:::benchmark_data_smithi177_31116_object40438:head# 2024-09-13T11:24:39.032 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.035 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:412c6a17:::benchmark_data_smithi177_31116_object8116:head# 2024-09-13T11:24:39.035 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.037 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:412c9765:::benchmark_data_smithi177_31116_object70640:head# 2024-09-13T11:24:39.038 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.040 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:412cadb5:::benchmark_data_smithi177_31116_object87878:head# 2024-09-13T11:24:39.040 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.041 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:412d2f75:::benchmark_data_smithi177_31116_object10439:head# 2024-09-13T11:24:39.041 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.041 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:412e1cc0:::benchmark_data_smithi177_31116_object44853:head# 2024-09-13T11:24:39.041 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.042 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:412ecf1a:::benchmark_data_smithi177_31116_object75062:head# 2024-09-13T11:24:39.042 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.042 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:412f0cf2:::benchmark_data_smithi177_31116_object74585:head# 2024-09-13T11:24:39.042 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.042 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:412f5431:::benchmark_data_smithi177_31116_object83827:head# 2024-09-13T11:24:39.042 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.043 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:412f5d11:::benchmark_data_smithi177_31116_object54401:head# 2024-09-13T11:24:39.043 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.045 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:413007af:::benchmark_data_smithi177_31116_object55297:head# 2024-09-13T11:24:39.045 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.048 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:41303834:::benchmark_data_smithi177_31116_object51005:head# 2024-09-13T11:24:39.048 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.049 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4130db71:::benchmark_data_smithi177_31116_object8643:head# 2024-09-13T11:24:39.049 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.049 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4131080b:::benchmark_data_smithi177_31116_object39956:head# 2024-09-13T11:24:39.049 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.050 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:413132de:::benchmark_data_smithi177_31116_object76519:head# 2024-09-13T11:24:39.050 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.050 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:41330121:::benchmark_data_smithi177_31116_object67157:head# 2024-09-13T11:24:39.050 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.050 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4134123a:::benchmark_data_smithi177_31116_object58582:head# 2024-09-13T11:24:39.051 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.051 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:41348c17:::benchmark_data_smithi177_31116_object58806:head# 2024-09-13T11:24:39.051 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.051 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4134da32:::benchmark_data_smithi177_31116_object85299:head# 2024-09-13T11:24:39.051 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.052 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4135252d:::benchmark_data_smithi177_31116_object24081:head# 2024-09-13T11:24:39.052 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.052 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4135bb23:::benchmark_data_smithi177_31116_object86692:head# 2024-09-13T11:24:39.052 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.053 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:41362783:::benchmark_data_smithi177_31116_object78553:head# 2024-09-13T11:24:39.053 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.053 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4136a799:::benchmark_data_smithi177_31116_object58789:head# 2024-09-13T11:24:39.053 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.054 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4136c1b1:::benchmark_data_smithi177_31116_object2527:head# 2024-09-13T11:24:39.054 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.054 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4137b2aa:::benchmark_data_smithi177_31116_object104563:head# 2024-09-13T11:24:39.054 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.055 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-09-13T11:24:39.055 INFO:teuthology.orchestra.run.smithi092.stdout: "ops": [], 2024-09-13T11:24:39.055 INFO:teuthology.orchestra.run.smithi092.stdout: "num_ops": 0 2024-09-13T11:24:39.055 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-09-13T11:24:39.056 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4138068d:::benchmark_data_smithi177_31116_object11957:head# 2024-09-13T11:24:39.056 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.056 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:413a7595:::benchmark_data_smithi177_31116_object73543:head# 2024-09-13T11:24:39.056 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.056 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:413ab0ff:::benchmark_data_smithi177_31116_object5770:head# 2024-09-13T11:24:39.056 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.057 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:413ad901:::benchmark_data_smithi177_31116_object75574:head# 2024-09-13T11:24:39.057 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.057 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:413b2a07:::benchmark_data_smithi177_31116_object106110:head# 2024-09-13T11:24:39.057 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.060 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:413b758f:::benchmark_data_smithi177_31116_object32729:head# 2024-09-13T11:24:39.060 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.060 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:413c5ef0:::benchmark_data_smithi177_31116_object15967:head# 2024-09-13T11:24:39.060 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.061 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:413c7bcf:::benchmark_data_smithi177_31116_object68113:head# 2024-09-13T11:24:39.061 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.061 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:413cdb37:::benchmark_data_smithi177_31116_object28086:head# 2024-09-13T11:24:39.061 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.061 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:413ecece:::benchmark_data_smithi177_31116_object77374:head# 2024-09-13T11:24:39.061 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.061 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:413f3281:::benchmark_data_smithi177_31116_object5056:head# 2024-09-13T11:24:39.061 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.061 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4140257b:::benchmark_data_smithi177_31116_object1070:head# 2024-09-13T11:24:39.061 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.062 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:41404bdc:::benchmark_data_smithi177_31116_object21119:head# 2024-09-13T11:24:39.062 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.062 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4140caa6:::benchmark_data_smithi177_31116_object86276:head# 2024-09-13T11:24:39.062 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.062 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4140e20c:::benchmark_data_smithi177_31116_object64482:head# 2024-09-13T11:24:39.063 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.063 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:41416e58:::benchmark_data_smithi177_31116_object71613:head# 2024-09-13T11:24:39.063 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.063 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:41417e3f:::benchmark_data_smithi177_31116_object89520:head# 2024-09-13T11:24:39.063 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.063 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4142ab36:::benchmark_data_smithi177_31116_object53207:head# 2024-09-13T11:24:39.063 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.064 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4142c0f1:::benchmark_data_smithi177_31116_object48914:head# 2024-09-13T11:24:39.064 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.064 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:41433079:::benchmark_data_smithi177_31116_object47707:head# 2024-09-13T11:24:39.064 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.065 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4144618c:::benchmark_data_smithi177_31116_object36957:head# 2024-09-13T11:24:39.065 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.065 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:41470a68:::benchmark_data_smithi177_31116_object87964:head# 2024-09-13T11:24:39.065 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.066 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:414775a0:::benchmark_data_smithi177_31116_object16708:head# 2024-09-13T11:24:39.066 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.066 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:41484df3:::benchmark_data_smithi177_31116_object55062:head# 2024-09-13T11:24:39.066 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.067 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:414852e1:::benchmark_data_smithi177_31116_object26160:head# 2024-09-13T11:24:39.067 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.067 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:414888a4:::benchmark_data_smithi177_31116_object93685:head# 2024-09-13T11:24:39.067 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.068 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:414890b5:::benchmark_data_smithi177_31116_object67033:head# 2024-09-13T11:24:39.068 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.068 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:41489392:::benchmark_data_smithi177_31116_object61263:head# 2024-09-13T11:24:39.068 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.071 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:41491d90:::benchmark_data_smithi177_31116_object90456:head# 2024-09-13T11:24:39.071 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.073 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4149d3cd:::benchmark_data_smithi177_31116_object88510:head# 2024-09-13T11:24:39.073 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.076 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4149f436:::benchmark_data_smithi177_31116_object88343:head# 2024-09-13T11:24:39.076 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.078 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-09-13T11:24:39.079 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4149f5e8:::benchmark_data_smithi177_31116_object54078:head# 2024-09-13T11:24:39.079 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.080 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:414a58e8:::benchmark_data_smithi177_31116_object71603:head# 2024-09-13T11:24:39.080 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.081 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:414af3aa:::benchmark_data_smithi177_31116_object11251:head# 2024-09-13T11:24:39.081 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.081 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:414d34ea:::benchmark_data_smithi177_31116_object17470:head# 2024-09-13T11:24:39.081 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.081 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:414d5341:::benchmark_data_smithi177_31116_object23261:head# 2024-09-13T11:24:39.081 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.082 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:414dc6bc:::benchmark_data_smithi177_31116_object11521:head# 2024-09-13T11:24:39.082 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.086 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:414dd8c6:::benchmark_data_smithi177_31116_object108231:head# 2024-09-13T11:24:39.086 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.086 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:414f762a:::benchmark_data_smithi177_31116_object29220:head# 2024-09-13T11:24:39.086 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.089 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:41508cdb:::benchmark_data_smithi177_31116_object38957:head# 2024-09-13T11:24:39.089 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.089 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4150cc84:::benchmark_data_smithi177_31116_object38096:head# 2024-09-13T11:24:39.090 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.090 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:41514999:::benchmark_data_smithi177_31116_object68892:head# 2024-09-13T11:24:39.090 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.091 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:41515722:::benchmark_data_smithi177_31116_object57027:head# 2024-09-13T11:24:39.091 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.091 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:41529633:::benchmark_data_smithi177_31116_object107008:head# 2024-09-13T11:24:39.091 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.092 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4152d58b:::benchmark_data_smithi177_31116_object101557:head# 2024-09-13T11:24:39.092 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.093 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:41541cf3:::benchmark_data_smithi177_31116_object102371:head# 2024-09-13T11:24:39.093 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.093 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4154edc7:::benchmark_data_smithi177_31116_object64608:head# 2024-09-13T11:24:39.093 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.096 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:41557964:::benchmark_data_smithi177_31116_object62549:head# 2024-09-13T11:24:39.096 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.097 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:41557f72:::benchmark_data_smithi177_31116_object68204:head# 2024-09-13T11:24:39.097 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.097 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:41560751:::benchmark_data_smithi177_31116_object9396:head# 2024-09-13T11:24:39.098 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.098 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4157c860:::benchmark_data_smithi177_31116_object52620:head# 2024-09-13T11:24:39.098 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.098 INFO:tasks.ceph.osd.3.smithi177.stderr:2024-09-13T11:24:39.095+0000 7f10adba4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26574) UID: 0 2024-09-13T11:24:39.099 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:415872a1:::benchmark_data_smithi177_31116_object74:head# 2024-09-13T11:24:39.099 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.100 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:415960d1:::benchmark_data_smithi177_31116_object96382:head# 2024-09-13T11:24:39.100 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.100 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:41597be7:::benchmark_data_smithi177_31116_object77942:head# 2024-09-13T11:24:39.100 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.100 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:41598183:::benchmark_data_smithi177_31116_object60921:head# 2024-09-13T11:24:39.100 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.101 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:415ab4ca:::benchmark_data_smithi177_31116_object107274:head# 2024-09-13T11:24:39.101 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.101 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:415afb36:::benchmark_data_smithi177_31116_object42444:head# 2024-09-13T11:24:39.101 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.102 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:415c5d4e:::benchmark_data_smithi177_31116_object5289:head# 2024-09-13T11:24:39.102 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.102 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:415c74a2:::benchmark_data_smithi177_31116_object79285:head# 2024-09-13T11:24:39.102 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.104 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:415dcc1b:::benchmark_data_smithi177_31116_object24659:head# 2024-09-13T11:24:39.104 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.104 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:415e4ddb:::benchmark_data_smithi177_31116_object68305:head# 2024-09-13T11:24:39.104 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.107 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:415e9c36:::benchmark_data_smithi177_31116_object29303:head# 2024-09-13T11:24:39.107 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.107 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:415ea28d:::benchmark_data_smithi177_31116_object476:head# 2024-09-13T11:24:39.108 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.108 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:415f94f0:::benchmark_data_smithi177_31116_object83426:head# 2024-09-13T11:24:39.108 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.108 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:41602bcb:::benchmark_data_smithi177_31116_object50520:head# 2024-09-13T11:24:39.108 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.109 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:416030ca:::benchmark_data_smithi177_31116_object98686:head# 2024-09-13T11:24:39.109 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.109 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4160654c:::benchmark_data_smithi177_31116_object69796:head# 2024-09-13T11:24:39.109 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.109 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4160e852:::benchmark_data_smithi177_31116_object2102:head# 2024-09-13T11:24:39.109 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.110 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4161b2ed:::benchmark_data_smithi177_31116_object26369:head# 2024-09-13T11:24:39.110 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.110 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:416202c7:::benchmark_data_smithi177_31116_object18224:head# 2024-09-13T11:24:39.110 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.111 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4162fbaa:::benchmark_data_smithi177_31116_object87667:head# 2024-09-13T11:24:39.111 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.112 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:41639be9:::benchmark_data_smithi177_31116_object82774:head# 2024-09-13T11:24:39.112 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.112 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:41647159:::benchmark_data_smithi177_31116_object54135:head# 2024-09-13T11:24:39.112 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.113 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4164d7cd:::benchmark_data_smithi177_31116_object15882:head# 2024-09-13T11:24:39.113 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.114 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:41653c50:::benchmark_data_smithi177_31116_object12655:head# 2024-09-13T11:24:39.114 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.114 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:41654608:::benchmark_data_smithi177_31116_object100217:head# 2024-09-13T11:24:39.114 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.114 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:416588d9:::benchmark_data_smithi177_31116_object1394:head# 2024-09-13T11:24:39.114 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.114 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:41668529:::benchmark_data_smithi177_31116_object71463:head# 2024-09-13T11:24:39.115 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.115 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:416714ae:::benchmark_data_smithi177_31116_object85083:head# 2024-09-13T11:24:39.116 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.116 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4167294d:::benchmark_data_smithi177_31116_object2787:head# 2024-09-13T11:24:39.116 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.116 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:41674bfe:::benchmark_data_smithi177_31116_object1836:head# 2024-09-13T11:24:39.116 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.117 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:41678c5a:::benchmark_data_smithi177_31116_object2579:head# 2024-09-13T11:24:39.117 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.117 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:416921d8:::benchmark_data_smithi177_31116_object91315:head# 2024-09-13T11:24:39.117 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.118 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:416992ed:::benchmark_data_smithi177_31116_object14195:head# 2024-09-13T11:24:39.118 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.118 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:416aa42a:::benchmark_data_smithi177_31116_object1796:head# 2024-09-13T11:24:39.118 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.119 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:416baeb7:::benchmark_data_smithi177_31116_object98953:head# 2024-09-13T11:24:39.119 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.119 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:416bf5f4:::benchmark_data_smithi177_31116_object11088:head# 2024-09-13T11:24:39.119 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.119 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:416c07e1:::benchmark_data_smithi177_31116_object19781:head# 2024-09-13T11:24:39.120 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.120 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:416c89ff:::benchmark_data_smithi177_31116_object24452:head# 2024-09-13T11:24:39.120 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.120 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:416c9f8b:::benchmark_data_smithi177_31116_object67450:head# 2024-09-13T11:24:39.120 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.121 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:416caf29:::benchmark_data_smithi177_31116_object100488:head# 2024-09-13T11:24:39.121 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.121 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:416cfe5b:::benchmark_data_smithi177_31116_object28355:head# 2024-09-13T11:24:39.121 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.122 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:416dee48:::benchmark_data_smithi177_31116_object69146:head# 2024-09-13T11:24:39.123 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.125 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:416edb06:::benchmark_data_smithi177_31116_object90618:head# 2024-09-13T11:24:39.125 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.126 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:416f2e3c:::benchmark_data_smithi177_31116_object52283:head# 2024-09-13T11:24:39.126 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.127 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:416f6884:::benchmark_data_smithi177_31116_object85457:head# 2024-09-13T11:24:39.127 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.127 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:416fe4cb:::benchmark_data_smithi177_31116_object33933:head# 2024-09-13T11:24:39.127 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.127 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:41719e84:::benchmark_data_smithi177_31116_object72397:head# 2024-09-13T11:24:39.127 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.128 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:41728adc:::benchmark_data_smithi177_31116_object69180:head# 2024-09-13T11:24:39.128 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.128 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4172b340:::benchmark_data_smithi177_31116_object36568:head# 2024-09-13T11:24:39.128 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.129 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4172c7d1:::benchmark_data_smithi177_31116_object2597:head# 2024-09-13T11:24:39.129 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.129 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4172d173:::benchmark_data_smithi177_31116_object101733:head# 2024-09-13T11:24:39.129 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.130 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:41734f82:::benchmark_data_smithi177_31116_object70601:head# 2024-09-13T11:24:39.130 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.131 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:41739173:::benchmark_data_smithi177_31116_object90170:head# 2024-09-13T11:24:39.131 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.131 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4173ab0e:::benchmark_data_smithi177_31116_object5870:head# 2024-09-13T11:24:39.131 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.132 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4173ac02:::benchmark_data_smithi177_31116_object7994:head# 2024-09-13T11:24:39.132 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.132 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4173f81c:::benchmark_data_smithi177_31116_object2902:head# 2024-09-13T11:24:39.132 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.133 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:41746743:::benchmark_data_smithi177_31116_object40970:head# 2024-09-13T11:24:39.133 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.133 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4174a2ba:::benchmark_data_smithi177_31116_object84623:head# 2024-09-13T11:24:39.133 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.133 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:41752f5d:::benchmark_data_smithi177_31116_object25222:head# 2024-09-13T11:24:39.133 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.134 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:417575d1:::benchmark_data_smithi177_31116_object62797:head# 2024-09-13T11:24:39.135 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.136 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:41758453:::benchmark_data_smithi177_31116_object105707:head# 2024-09-13T11:24:39.136 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.138 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4175c8dc:::benchmark_data_smithi177_31116_object31250:head# 2024-09-13T11:24:39.138 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.140 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4175d761:::benchmark_data_smithi177_31116_object85973:head# 2024-09-13T11:24:39.140 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.141 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:417667a7:::benchmark_data_smithi177_31116_object82679:head# 2024-09-13T11:24:39.141 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.142 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4176870c:::benchmark_data_smithi177_31116_object65699:head# 2024-09-13T11:24:39.143 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.143 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:41768c3b:::benchmark_data_smithi177_31116_object3776:head# 2024-09-13T11:24:39.143 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.143 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:41784e8e:::benchmark_data_smithi177_31116_object26206:head# 2024-09-13T11:24:39.143 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.143 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:41787b82:::benchmark_data_smithi177_31116_object33492:head# 2024-09-13T11:24:39.143 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.144 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4179b627:::benchmark_data_smithi177_31116_object64126:head# 2024-09-13T11:24:39.145 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.145 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:417a9b02:::benchmark_data_smithi177_31116_object88440:head# 2024-09-13T11:24:39.145 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.146 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:417b41a0:::benchmark_data_smithi177_31116_object53536:head# 2024-09-13T11:24:39.146 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.146 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:417b6571:::benchmark_data_smithi177_31116_object74011:head# 2024-09-13T11:24:39.146 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.147 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:417b9e44:::benchmark_data_smithi177_31116_object13201:head# 2024-09-13T11:24:39.147 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.149 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:417bbe8c:::benchmark_data_smithi177_31116_object100865:head# 2024-09-13T11:24:39.149 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.150 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:417c9ac2:::benchmark_data_smithi177_31116_object53469:head# 2024-09-13T11:24:39.150 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.150 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:417cf96c:::benchmark_data_smithi177_31116_object97578:head# 2024-09-13T11:24:39.150 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.150 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:417d0664:::benchmark_data_smithi177_31116_object48466:head# 2024-09-13T11:24:39.150 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.150 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:417d9626:::benchmark_data_smithi177_31116_object9136:head# 2024-09-13T11:24:39.150 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.152 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:417e9147:::benchmark_data_smithi177_31116_object2103:head# 2024-09-13T11:24:39.153 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.153 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4180b6c2:::benchmark_data_smithi177_31116_object14871:head# 2024-09-13T11:24:39.153 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.153 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:41811b59:::benchmark_data_smithi177_31116_object60573:head# 2024-09-13T11:24:39.153 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.153 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4182860e:::benchmark_data_smithi177_31116_object88573:head# 2024-09-13T11:24:39.153 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.153 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:418293bc:::benchmark_data_smithi177_31116_object19475:head# 2024-09-13T11:24:39.153 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.153 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:41831322:::benchmark_data_smithi177_31116_object102985:head# 2024-09-13T11:24:39.153 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.154 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:418335c8:::benchmark_data_smithi177_31116_object120:head# 2024-09-13T11:24:39.154 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.154 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:418479f1:::benchmark_data_smithi177_31116_object12972:head# 2024-09-13T11:24:39.154 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.155 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:41849bdb:::benchmark_data_smithi177_31116_object82034:head# 2024-09-13T11:24:39.155 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.155 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4184aad8:::benchmark_data_smithi177_31116_object99990:head# 2024-09-13T11:24:39.156 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.156 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:41851012:::benchmark_data_smithi177_31116_object60548:head# 2024-09-13T11:24:39.156 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.156 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:41856b78:::benchmark_data_smithi177_31116_object43013:head# 2024-09-13T11:24:39.156 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.156 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4186867e:::benchmark_data_smithi177_31116_object107405:head# 2024-09-13T11:24:39.156 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.156 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:41879897:::benchmark_data_smithi177_31116_object37515:head# 2024-09-13T11:24:39.156 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.161 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4187e426:::benchmark_data_smithi177_31116_object85434:head# 2024-09-13T11:24:39.161 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.161 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4187f4c4:::benchmark_data_smithi177_31116_object25562:head# 2024-09-13T11:24:39.161 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.161 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4187fa39:::benchmark_data_smithi177_31116_object31895:head# 2024-09-13T11:24:39.162 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.162 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:41889950:::benchmark_data_smithi177_31116_object72936:head# 2024-09-13T11:24:39.162 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.162 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:418957fb:::benchmark_data_smithi177_31116_object99590:head# 2024-09-13T11:24:39.162 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.162 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4189bcfa:::benchmark_data_smithi177_31116_object22433:head# 2024-09-13T11:24:39.162 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.162 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4189d605:::benchmark_data_smithi177_31116_object45084:head# 2024-09-13T11:24:39.162 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.162 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4189d656:::benchmark_data_smithi177_31116_object23549:head# 2024-09-13T11:24:39.162 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.162 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:418ab56f:::benchmark_data_smithi177_31116_object9569:head# 2024-09-13T11:24:39.162 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.162 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:418d0cbd:::benchmark_data_smithi177_31116_object6573:head# 2024-09-13T11:24:39.163 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.163 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:418d351d:::benchmark_data_smithi177_31116_object41793:head# 2024-09-13T11:24:39.163 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.163 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:418de556:::benchmark_data_smithi177_31116_object2352:head# 2024-09-13T11:24:39.163 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.164 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:418e4412:::benchmark_data_smithi177_31116_object888:head# 2024-09-13T11:24:39.164 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.165 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:418e4d2e:::benchmark_data_smithi177_31116_object63692:head# 2024-09-13T11:24:39.165 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.165 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:418eec3c:::benchmark_data_smithi177_31116_object67521:head# 2024-09-13T11:24:39.165 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.165 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:418f2216:::benchmark_data_smithi177_31116_object24865:head# 2024-09-13T11:24:39.165 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.165 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:418f7068:::benchmark_data_smithi177_31116_object50491:head# 2024-09-13T11:24:39.165 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.165 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:418f80be:::benchmark_data_smithi177_31116_object20192:head# 2024-09-13T11:24:39.165 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.166 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:41911e9e:::benchmark_data_smithi177_31116_object64294:head# 2024-09-13T11:24:39.166 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.167 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:419176c4:::benchmark_data_smithi177_31116_object61499:head# 2024-09-13T11:24:39.167 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.167 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:41917fec:::benchmark_data_smithi177_31116_object75354:head# 2024-09-13T11:24:39.167 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.169 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:41919f48:::benchmark_data_smithi177_31116_object94480:head# 2024-09-13T11:24:39.169 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.169 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4192bf59:::benchmark_data_smithi177_31116_object23168:head# 2024-09-13T11:24:39.169 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.170 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4192c403:::benchmark_data_smithi177_31116_object85666:head# 2024-09-13T11:24:39.170 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.171 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4193a4c5:::benchmark_data_smithi177_31116_object23463:head# 2024-09-13T11:24:39.171 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.171 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4193d075:::benchmark_data_smithi177_31116_object81276:head# 2024-09-13T11:24:39.171 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.173 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4194210a:::benchmark_data_smithi177_31116_object73668:head# 2024-09-13T11:24:39.173 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.174 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:41952947:::benchmark_data_smithi177_31116_object27795:head# 2024-09-13T11:24:39.174 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.175 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4197da66:::benchmark_data_smithi177_31116_object57043:head# 2024-09-13T11:24:39.175 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.176 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4197f066:::benchmark_data_smithi177_31116_object62724:head# 2024-09-13T11:24:39.176 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.176 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:419a3d86:::benchmark_data_smithi177_31116_object108355:head# 2024-09-13T11:24:39.176 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.177 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:419aa723:::benchmark_data_smithi177_31116_object96343:head# 2024-09-13T11:24:39.177 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.177 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:419ab438:::benchmark_data_smithi177_31116_object102157:head# 2024-09-13T11:24:39.177 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.178 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:419bb974:::benchmark_data_smithi177_31116_object8612:head# 2024-09-13T11:24:39.178 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.178 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:419c4758:::benchmark_data_smithi177_31116_object46233:head# 2024-09-13T11:24:39.179 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.179 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:419c8a4e:::benchmark_data_smithi177_31116_object46852:head# 2024-09-13T11:24:39.179 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.179 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:419cf409:::benchmark_data_smithi177_31116_object4247:head# 2024-09-13T11:24:39.179 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.180 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:419d07ca:::benchmark_data_smithi177_31116_object21317:head# 2024-09-13T11:24:39.180 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.180 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:419d7767:::benchmark_data_smithi177_31116_object63999:head# 2024-09-13T11:24:39.180 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.180 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:419dd9e6:::benchmark_data_smithi177_31116_object73595:head# 2024-09-13T11:24:39.180 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.181 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:419ecb07:::benchmark_data_smithi177_31116_object105667:head# 2024-09-13T11:24:39.181 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.181 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:419edbbf:::benchmark_data_smithi177_31116_object59391:head# 2024-09-13T11:24:39.181 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.181 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:419f34d0:::benchmark_data_smithi177_31116_object27660:head# 2024-09-13T11:24:39.181 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.183 INFO:teuthology.orchestra.run.smithi001.stdout:osd.0: {} 2024-09-13T11:24:39.183 INFO:teuthology.orchestra.run.smithi001.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T11:24:39.183 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:41a079bc:::benchmark_data_smithi177_31116_object25521:head# 2024-09-13T11:24:39.184 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.187 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:41a14712:::benchmark_data_smithi177_31116_object96132:head# 2024-09-13T11:24:39.187 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.192 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:41a172db:::benchmark_data_smithi177_31116_object95135:head# 2024-09-13T11:24:39.192 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.195 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:41a18620:::benchmark_data_smithi177_31116_object67045:head# 2024-09-13T11:24:39.195 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.195 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:41a1cfeb:::benchmark_data_smithi177_31116_object90669:head# 2024-09-13T11:24:39.195 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.198 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:41a2b4e3:::benchmark_data_smithi177_31116_object52017:head# 2024-09-13T11:24:39.198 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.198 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:41a2e77e:::benchmark_data_smithi177_31116_object11498:head# 2024-09-13T11:24:39.198 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.199 INFO:tasks.ceph.osd.5.smithi092.stderr:2024-09-13T11:24:39.194+0000 7f6f0b705640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26346) UID: 0 2024-09-13T11:24:39.200 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:41a47e95:::benchmark_data_smithi177_31116_object75419:head# 2024-09-13T11:24:39.200 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.201 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:41a4e172:::benchmark_data_smithi177_31116_object91902:head# 2024-09-13T11:24:39.201 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.201 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:41a57b9a:::benchmark_data_smithi177_31116_object22810:head# 2024-09-13T11:24:39.201 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.201 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:41a6b602:::benchmark_data_smithi177_31116_object55914:head# 2024-09-13T11:24:39.201 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.201 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:41a71140:::benchmark_data_smithi177_31116_object11412:head# 2024-09-13T11:24:39.201 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.202 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:41a72222:::benchmark_data_smithi177_31116_object54454:head# 2024-09-13T11:24:39.202 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.202 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:41a7d592:::benchmark_data_smithi177_31116_object110136:head# 2024-09-13T11:24:39.202 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.203 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:41a8c50d:::benchmark_data_smithi177_31116_object67995:head# 2024-09-13T11:24:39.203 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.204 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:41aa2705:::benchmark_data_smithi177_31116_object42760:head# 2024-09-13T11:24:39.204 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.204 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:41aa614b:::benchmark_data_smithi177_31116_object74606:head# 2024-09-13T11:24:39.204 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.204 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:41aa9e64:::benchmark_data_smithi177_31116_object55954:head# 2024-09-13T11:24:39.204 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.205 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:41aae21f:::benchmark_data_smithi177_31116_object32713:head# 2024-09-13T11:24:39.205 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.205 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:41ab1153:::benchmark_data_smithi177_31116_object94100:head# 2024-09-13T11:24:39.205 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.205 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:41ab5856:::benchmark_data_smithi177_31116_object56608:head# 2024-09-13T11:24:39.206 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.206 INFO:teuthology.orchestra.run.smithi001.stdout:osd.1: {} 2024-09-13T11:24:39.207 INFO:teuthology.orchestra.run.smithi001.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T11:24:39.208 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:41ac8c11:::benchmark_data_smithi177_31116_object105170:head# 2024-09-13T11:24:39.208 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.208 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:41acfb7f:::benchmark_data_smithi177_31116_object76139:head# 2024-09-13T11:24:39.208 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.208 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:41ad7c50:::benchmark_data_smithi177_31116_object55802:head# 2024-09-13T11:24:39.208 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.208 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:41adaf7a:::benchmark_data_smithi177_31116_object56630:head# 2024-09-13T11:24:39.208 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.208 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:41adcae9:::benchmark_data_smithi177_31116_object75584:head# 2024-09-13T11:24:39.209 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.209 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:41aebb73:::benchmark_data_smithi177_31116_object39961:head# 2024-09-13T11:24:39.210 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.210 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:41af844c:::benchmark_data_smithi177_31116_object35687:head# 2024-09-13T11:24:39.210 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.210 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:41b07870:::benchmark_data_smithi177_31116_object69595:head# 2024-09-13T11:24:39.210 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.210 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:41b10e35:::benchmark_data_smithi177_31116_object74199:head# 2024-09-13T11:24:39.211 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.211 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:41b19bed:::benchmark_data_smithi177_31116_object74465:head# 2024-09-13T11:24:39.211 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.211 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:41b1c2dd:::benchmark_data_smithi177_31116_object16126:head# 2024-09-13T11:24:39.211 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.214 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:41b1c8cc:::benchmark_data_smithi177_31116_object12959:head# 2024-09-13T11:24:39.214 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.218 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:41b2d6b1:::benchmark_data_smithi177_31116_object90563:head# 2024-09-13T11:24:39.218 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.218 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:41b3651e:::benchmark_data_smithi177_31116_object8112:head# 2024-09-13T11:24:39.218 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.218 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:41b3e1b7:::benchmark_data_smithi177_31116_object9731:head# 2024-09-13T11:24:39.218 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.218 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:41b44938:::benchmark_data_smithi177_31116_object101894:head# 2024-09-13T11:24:39.219 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.220 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:41b45df1:::benchmark_data_smithi177_31116_object7282:head# 2024-09-13T11:24:39.220 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.220 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:41b4a870:::benchmark_data_smithi177_31116_object94831:head# 2024-09-13T11:24:39.221 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.221 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:41b4ae2d:::benchmark_data_smithi177_31116_object9176:head# 2024-09-13T11:24:39.221 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.221 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:41b4bc7a:::benchmark_data_smithi177_31116_object584:head# 2024-09-13T11:24:39.221 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.223 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:41b4e7de:::benchmark_data_smithi177_31116_object19431:head# 2024-09-13T11:24:39.223 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.223 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:41b53687:::benchmark_data_smithi177_31116_object47763:head# 2024-09-13T11:24:39.223 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.226 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:41b5acfe:::benchmark_data_smithi177_31116_object15491:head# 2024-09-13T11:24:39.226 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.226 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:41b67a1b:::benchmark_data_smithi177_31116_object25693:head# 2024-09-13T11:24:39.227 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.227 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:41b81d75:::benchmark_data_smithi177_31116_object77248:head# 2024-09-13T11:24:39.227 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.229 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:41ba392b:::benchmark_data_smithi177_31116_object60509:head# 2024-09-13T11:24:39.229 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.232 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:41ba5175:::benchmark_data_smithi177_31116_object73691:head# 2024-09-13T11:24:39.232 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.233 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:41ba7f29:::benchmark_data_smithi177_31116_object44762:head# 2024-09-13T11:24:39.233 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.233 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:41baaa35:::benchmark_data_smithi177_31116_object84636:head# 2024-09-13T11:24:39.233 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.238 INFO:teuthology.orchestra.run.smithi001.stdout:osd.2: {} 2024-09-13T11:24:39.238 INFO:teuthology.orchestra.run.smithi001.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T11:24:39.239 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:41baba54:::benchmark_data_smithi177_31116_object91469:head# 2024-09-13T11:24:39.239 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.239 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:41bae3f3:::benchmark_data_smithi177_31116_object38471:head# 2024-09-13T11:24:39.239 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.239 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:41bb8946:::benchmark_data_smithi177_31116_object55853:head# 2024-09-13T11:24:39.239 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.239 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:41bdbcb6:::benchmark_data_smithi177_31116_object70256:head# 2024-09-13T11:24:39.239 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.239 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:41be57e2:::benchmark_data_smithi177_31116_object79887:head# 2024-09-13T11:24:39.240 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.240 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-09-13T11:24:39.240 INFO:teuthology.orchestra.run.smithi092.stdout: "ops": [], 2024-09-13T11:24:39.240 INFO:teuthology.orchestra.run.smithi092.stdout: "complaint_time": 30, 2024-09-13T11:24:39.240 INFO:teuthology.orchestra.run.smithi092.stdout: "num_blocked_ops": 0 2024-09-13T11:24:39.241 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-09-13T11:24:39.242 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:41be5b07:::benchmark_data_smithi177_31116_object73153:head# 2024-09-13T11:24:39.243 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.243 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:41bfb970:::benchmark_data_smithi177_31116_object43958:head# 2024-09-13T11:24:39.243 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.243 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:41bff846:::benchmark_data_smithi177_31116_object11680:head# 2024-09-13T11:24:39.243 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.243 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:41c2b5e5:::benchmark_data_smithi177_31116_object82453:head# 2024-09-13T11:24:39.243 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.243 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:41c3cec1:::benchmark_data_smithi177_31116_object20659:head# 2024-09-13T11:24:39.243 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.243 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:41c5f89c:::benchmark_data_smithi177_31116_object93925:head# 2024-09-13T11:24:39.243 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.243 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:41c64cdb:::benchmark_data_smithi177_31116_object19514:head# 2024-09-13T11:24:39.243 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.245 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:41c75c67:::benchmark_data_smithi177_31116_object45384:head# 2024-09-13T11:24:39.245 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.246 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:41c7b23a:::benchmark_data_smithi177_31116_object83691:head# 2024-09-13T11:24:39.246 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.246 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:41c86d98:::benchmark_data_smithi177_31116_object57667:head# 2024-09-13T11:24:39.246 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.247 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:41ca3290:::benchmark_data_smithi177_31116_object28436:head# 2024-09-13T11:24:39.247 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.247 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:41ca6820:::benchmark_data_smithi177_31116_object109302:head# 2024-09-13T11:24:39.247 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.249 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:41cb13de:::benchmark_data_smithi177_31116_object77182:head# 2024-09-13T11:24:39.250 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.250 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:41cc07d3:::benchmark_data_smithi177_31116_object86792:head# 2024-09-13T11:24:39.250 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.251 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:41cd1957:::benchmark_data_smithi177_31116_object81223:head# 2024-09-13T11:24:39.251 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.253 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:41cea0a2:::benchmark_data_smithi177_31116_object100699:head# 2024-09-13T11:24:39.253 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.254 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:41cf2e22:::benchmark_data_smithi177_31116_object15842:head# 2024-09-13T11:24:39.254 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.256 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:41cf6c5e:::benchmark_data_smithi177_31116_object77485:head# 2024-09-13T11:24:39.256 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.256 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:41cf757c:::benchmark_data_smithi177_31116_object29257:head# 2024-09-13T11:24:39.257 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.257 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:41cf784e:::benchmark_data_smithi177_31116_object99342:head# 2024-09-13T11:24:39.257 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.258 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:41cfb6ea:::benchmark_data_smithi177_31116_object42524:head# 2024-09-13T11:24:39.258 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.259 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:41d0710b:::benchmark_data_smithi177_31116_object3255:head# 2024-09-13T11:24:39.259 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.259 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-09-13T11:24:39.260 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:41d1364f:::benchmark_data_smithi177_31116_object77256:head# 2024-09-13T11:24:39.260 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.260 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:41d147be:::benchmark_data_smithi177_31116_object13860:head# 2024-09-13T11:24:39.260 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.260 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:41d2ea11:::benchmark_data_smithi177_31116_object26983:head# 2024-09-13T11:24:39.260 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.261 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:41d31c0a:::benchmark_data_smithi177_31116_object40565:head# 2024-09-13T11:24:39.261 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.261 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:41d31f48:::benchmark_data_smithi177_31116_object22366:head# 2024-09-13T11:24:39.261 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.262 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:41d3542c:::benchmark_data_smithi177_31116_object78076:head# 2024-09-13T11:24:39.262 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.262 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:41d48374:::benchmark_data_smithi177_31116_object24436:head# 2024-09-13T11:24:39.263 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.263 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:41d5dffd:::benchmark_data_smithi177_31116_object60120:head# 2024-09-13T11:24:39.263 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.263 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:41d62272:::benchmark_data_smithi177_31116_object51012:head# 2024-09-13T11:24:39.263 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.264 INFO:teuthology.orchestra.run.smithi001.stdout:osd.3: {} 2024-09-13T11:24:39.264 INFO:teuthology.orchestra.run.smithi001.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T11:24:39.265 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:41d65c2c:::benchmark_data_smithi177_31116_object78597:head# 2024-09-13T11:24:39.265 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.265 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:41d78a04:::benchmark_data_smithi177_31116_object16973:head# 2024-09-13T11:24:39.265 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.265 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:41d8b0a8:::benchmark_data_smithi177_31116_object104572:head# 2024-09-13T11:24:39.265 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.267 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:41d8bdc3:::benchmark_data_smithi177_31116_object633:head# 2024-09-13T11:24:39.267 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.267 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:41d8dce2:::benchmark_data_smithi177_31116_object73642:head# 2024-09-13T11:24:39.267 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.268 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:41d9a918:::benchmark_data_smithi177_31116_object66060:head# 2024-09-13T11:24:39.268 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.268 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:41d9b927:::benchmark_data_smithi177_31116_object74127:head# 2024-09-13T11:24:39.268 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.271 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:41dac7a5:::benchmark_data_smithi177_31116_object11855:head# 2024-09-13T11:24:39.271 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.276 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:41dc061c:::benchmark_data_smithi177_31116_object9685:head# 2024-09-13T11:24:39.276 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.277 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:41dc79e6:::benchmark_data_smithi177_31116_object62411:head# 2024-09-13T11:24:39.278 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.279 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:41dcac2d:::benchmark_data_smithi177_31116_object85745:head# 2024-09-13T11:24:39.279 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.281 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:41dd7da9:::benchmark_data_smithi177_31116_object96854:head# 2024-09-13T11:24:39.282 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.282 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:41ddbd59:::benchmark_data_smithi177_31116_object84138:head# 2024-09-13T11:24:39.282 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.284 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:41ddd2f2:::benchmark_data_smithi177_31116_object937:head# 2024-09-13T11:24:39.284 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.287 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:41de5b5b:::benchmark_data_smithi177_31116_object74748:head# 2024-09-13T11:24:39.287 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.287 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:41de8a53:::benchmark_data_smithi177_31116_object101264:head# 2024-09-13T11:24:39.288 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.292 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:41dfccab:::benchmark_data_smithi177_31116_object7586:head# 2024-09-13T11:24:39.292 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.293 INFO:teuthology.orchestra.run.smithi001.stdout:osd.4: {} 2024-09-13T11:24:39.294 INFO:teuthology.orchestra.run.smithi001.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T11:24:39.295 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:41e0c6ab:::benchmark_data_smithi177_31116_object249:head# 2024-09-13T11:24:39.295 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.295 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:41e16739:::benchmark_data_smithi177_31116_object99414:head# 2024-09-13T11:24:39.295 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.296 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:41e1ffc4:::benchmark_data_smithi177_31116_object82543:head# 2024-09-13T11:24:39.296 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.298 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:41e2315d:::benchmark_data_smithi177_31116_object100125:head# 2024-09-13T11:24:39.299 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.299 INFO:tasks.ceph.osd.12.smithi001.stderr:2024-09-13T11:24:39.295+0000 7f892c6e6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26314) UID: 0 2024-09-13T11:24:39.300 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:41e24c3b:::benchmark_data_smithi177_31116_object8775:head# 2024-09-13T11:24:39.300 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.300 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:41e27808:::benchmark_data_smithi177_31116_object21529:head# 2024-09-13T11:24:39.300 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.301 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:41e2d3fc:::benchmark_data_smithi177_31116_object106447:head# 2024-09-13T11:24:39.301 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.302 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:41e2e6c5:::benchmark_data_smithi177_31116_object92386:head# 2024-09-13T11:24:39.302 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.302 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:41e2fac0:::benchmark_data_smithi177_31116_object108495:head# 2024-09-13T11:24:39.302 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.303 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:41e43a6c:::benchmark_data_smithi177_31116_object15337:head# 2024-09-13T11:24:39.303 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.305 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:41e6b2cd:::benchmark_data_smithi177_31116_object55693:head# 2024-09-13T11:24:39.305 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.308 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:41e7987a:::benchmark_data_smithi177_31116_object88585:head# 2024-09-13T11:24:39.308 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.312 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:41e7bfb5:::benchmark_data_smithi177_31116_object52908:head# 2024-09-13T11:24:39.312 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.314 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:41e7cd26:::benchmark_data_smithi177_31116_object45645:head# 2024-09-13T11:24:39.314 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.317 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:41e93e2f:::benchmark_data_smithi177_31116_object76192:head# 2024-09-13T11:24:39.317 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.318 INFO:teuthology.orchestra.run.smithi001.stdout:osd.5: {} 2024-09-13T11:24:39.318 INFO:teuthology.orchestra.run.smithi001.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T11:24:39.319 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:41ea13f5:::benchmark_data_smithi177_31116_object109496:head# 2024-09-13T11:24:39.319 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.320 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:41ea6534:::benchmark_data_smithi177_31116_object68643:head# 2024-09-13T11:24:39.320 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.320 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:41eb6ecf:::benchmark_data_smithi177_31116_object31325:head# 2024-09-13T11:24:39.320 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.322 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:41eb78e5:::benchmark_data_smithi177_31116_object86966:head# 2024-09-13T11:24:39.322 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.322 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:41eb7f51:::benchmark_data_smithi177_31116_object100240:head# 2024-09-13T11:24:39.322 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.323 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:41eb86fb:::benchmark_data_smithi177_31116_object79892:head# 2024-09-13T11:24:39.323 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.324 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:41ec5e14:::benchmark_data_smithi177_31116_object65842:head# 2024-09-13T11:24:39.324 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.324 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:41ecd024:::benchmark_data_smithi177_31116_object61905:head# 2024-09-13T11:24:39.324 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.325 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:41ee084a:::benchmark_data_smithi177_31116_object31971:head# 2024-09-13T11:24:39.325 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.326 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:41ee1399:::benchmark_data_smithi177_31116_object74396:head# 2024-09-13T11:24:39.326 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.327 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:41ef40e6:::benchmark_data_smithi177_31116_object2330:head# 2024-09-13T11:24:39.327 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.327 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:41f0e97b:::benchmark_data_smithi177_31116_object55202:head# 2024-09-13T11:24:39.327 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.328 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:41f0f326:::benchmark_data_smithi177_31116_object58741:head# 2024-09-13T11:24:39.328 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.328 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:41f162ad:::benchmark_data_smithi177_31116_object9156:head# 2024-09-13T11:24:39.328 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.331 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:41f25773:::benchmark_data_smithi177_31116_object74134:head# 2024-09-13T11:24:39.331 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.335 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:41f25a38:::benchmark_data_smithi177_31116_object42282:head# 2024-09-13T11:24:39.335 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.336 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:41f2725e:::benchmark_data_smithi177_31116_object66051:head# 2024-09-13T11:24:39.336 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.336 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:41f3010d:::benchmark_data_smithi177_31116_object29514:head# 2024-09-13T11:24:39.336 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.340 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:41f30644:::benchmark_data_smithi177_31116_object68673:head# 2024-09-13T11:24:39.340 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.341 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:41f34694:::benchmark_data_smithi177_31116_object106609:head# 2024-09-13T11:24:39.341 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.344 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:41f348fb:::benchmark_data_smithi177_31116_object47131:head# 2024-09-13T11:24:39.344 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.347 INFO:teuthology.orchestra.run.smithi001.stdout:osd.6: {} 2024-09-13T11:24:39.347 INFO:teuthology.orchestra.run.smithi001.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T11:24:39.347 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:41f3a2d6:::benchmark_data_smithi177_31116_object109899:head# 2024-09-13T11:24:39.348 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.349 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:41f3a5f5:::benchmark_data_smithi177_31116_object56746:head# 2024-09-13T11:24:39.350 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.350 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:41f3bd88:::benchmark_data_smithi177_31116_object2887:head# 2024-09-13T11:24:39.350 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.350 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:41f43319:::benchmark_data_smithi177_31116_object39645:head# 2024-09-13T11:24:39.351 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.351 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:41f44a03:::benchmark_data_smithi177_31116_object55740:head# 2024-09-13T11:24:39.351 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.351 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:41f47b4d:::benchmark_data_smithi177_31116_object40505:head# 2024-09-13T11:24:39.351 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.353 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:41f48701:::benchmark_data_smithi177_31116_object61012:head# 2024-09-13T11:24:39.353 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.353 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:41f550fc:::benchmark_data_smithi177_31116_object56749:head# 2024-09-13T11:24:39.353 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.353 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:41f6212f:::benchmark_data_smithi177_31116_object78912:head# 2024-09-13T11:24:39.353 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.354 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:41f6935d:::benchmark_data_smithi177_31116_object1407:head# 2024-09-13T11:24:39.354 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.354 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:41f70e80:::benchmark_data_smithi177_31116_object22135:head# 2024-09-13T11:24:39.354 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.354 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:41f85f7e:::benchmark_data_smithi177_31116_object87654:head# 2024-09-13T11:24:39.355 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.355 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:41f9968d:::benchmark_data_smithi177_31116_object61201:head# 2024-09-13T11:24:39.355 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.357 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:41f9dc61:::benchmark_data_smithi177_31116_object109828:head# 2024-09-13T11:24:39.357 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.358 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:41fa7a17:::benchmark_data_smithi177_31116_object64838:head# 2024-09-13T11:24:39.358 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.359 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:41fb1714:::benchmark_data_smithi177_31116_object100619:head# 2024-09-13T11:24:39.359 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.359 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:41fb38de:::benchmark_data_smithi177_31116_object86965:head# 2024-09-13T11:24:39.359 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.361 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:41fb92ce:::benchmark_data_smithi177_31116_object108759:head# 2024-09-13T11:24:39.361 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.363 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:41fbee3c:::benchmark_data_smithi177_31116_object25259:head# 2024-09-13T11:24:39.364 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.364 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:41fc70f3:::benchmark_data_smithi177_31116_object74599:head# 2024-09-13T11:24:39.364 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.366 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:41fd496c:::benchmark_data_smithi177_31116_object11093:head# 2024-09-13T11:24:39.366 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.367 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:41fd5496:::benchmark_data_smithi177_31116_object19886:head# 2024-09-13T11:24:39.367 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.367 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:41fdbf6f:::benchmark_data_smithi177_31116_object85474:head# 2024-09-13T11:24:39.367 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.370 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:42001945:::benchmark_data_smithi177_31116_object95656:head# 2024-09-13T11:24:39.370 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.372 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4200a506:::benchmark_data_smithi177_31116_object99741:head# 2024-09-13T11:24:39.372 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.373 INFO:teuthology.orchestra.run.smithi001.stdout:osd.7: {} 2024-09-13T11:24:39.373 INFO:teuthology.orchestra.run.smithi001.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T11:24:39.373 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4201eacf:::benchmark_data_smithi177_31116_object32824:head# 2024-09-13T11:24:39.374 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.377 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:42026393:::benchmark_data_smithi177_31116_object95837:head# 2024-09-13T11:24:39.377 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.377 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4202b76f:::benchmark_data_smithi177_31116_object63761:head# 2024-09-13T11:24:39.377 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.378 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:420313fa:::benchmark_data_smithi177_31116_object89006:head# 2024-09-13T11:24:39.378 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.378 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:420370c5:::benchmark_data_smithi177_31116_object50629:head# 2024-09-13T11:24:39.378 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.379 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:420418c0:::benchmark_data_smithi177_31116_object15385:head# 2024-09-13T11:24:39.379 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.379 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:42042124:::benchmark_data_smithi177_31116_object61088:head# 2024-09-13T11:24:39.379 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.380 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:42042527:::benchmark_data_smithi177_31116_object59674:head# 2024-09-13T11:24:39.380 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.381 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4204f44a:::benchmark_data_smithi177_31116_object76485:head# 2024-09-13T11:24:39.381 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.381 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:420517fe:::benchmark_data_smithi177_31116_object102366:head# 2024-09-13T11:24:39.381 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.382 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:42052033:::benchmark_data_smithi177_31116_object16948:head# 2024-09-13T11:24:39.382 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.382 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:42052c1c:::benchmark_data_smithi177_31116_object21764:head# 2024-09-13T11:24:39.382 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.383 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:42058250:::benchmark_data_smithi177_31116_object29363:head# 2024-09-13T11:24:39.383 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.384 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:42059c06:::benchmark_data_smithi177_31116_object92327:head# 2024-09-13T11:24:39.384 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.384 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4205bd2e:::benchmark_data_smithi177_31116_object31274:head# 2024-09-13T11:24:39.384 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.387 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4205e255:::benchmark_data_smithi177_31116_object44687:head# 2024-09-13T11:24:39.388 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.390 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:42063c69:::benchmark_data_smithi177_31116_object6361:head# 2024-09-13T11:24:39.390 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.394 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:42065fb6:::benchmark_data_smithi177_31116_object17414:head# 2024-09-13T11:24:39.394 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.394 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4206a99c:::benchmark_data_smithi177_31116_object91302:head# 2024-09-13T11:24:39.394 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.397 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4208adad:::benchmark_data_smithi177_31116_object2575:head# 2024-09-13T11:24:39.397 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.397 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4208eedb:::benchmark_data_smithi177_31116_object34058:head# 2024-09-13T11:24:39.397 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.398 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4209617c:::benchmark_data_smithi177_31116_object87966:head# 2024-09-13T11:24:39.398 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.398 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4209e3cf:::benchmark_data_smithi177_31116_object98136:head# 2024-09-13T11:24:39.398 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.398 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:420ae81c:::benchmark_data_smithi177_31116_object6493:head# 2024-09-13T11:24:39.399 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.399 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:420bacc3:::benchmark_data_smithi177_31116_object63777:head# 2024-09-13T11:24:39.399 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.400 INFO:teuthology.orchestra.run.smithi001.stdout:osd.8: {} 2024-09-13T11:24:39.400 INFO:teuthology.orchestra.run.smithi001.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-13T11:24:39.400 INFO:tasks.ceph.osd.12.smithi001.stderr:2024-09-13T11:24:39.399+0000 7f892c6e6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26314) UID: 0 2024-09-13T11:24:39.402 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:420bc562:::benchmark_data_smithi177_31116_object104849:head# 2024-09-13T11:24:39.402 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.402 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:420c6fee:::benchmark_data_smithi177_31116_object45244:head# 2024-09-13T11:24:39.402 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.403 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:420d7a50:::benchmark_data_smithi177_31116_object72072:head# 2024-09-13T11:24:39.403 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.403 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:420dac90:::benchmark_data_smithi177_31116_object62462:head# 2024-09-13T11:24:39.403 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.403 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:420dea0f:::benchmark_data_smithi177_31116_object7714:head# 2024-09-13T11:24:39.403 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.404 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:420e93ab:::benchmark_data_smithi177_31116_object19458:head# 2024-09-13T11:24:39.404 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.405 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:420f9ed5:::benchmark_data_smithi177_31116_object60613:head# 2024-09-13T11:24:39.405 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.407 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:42108f04:::benchmark_data_smithi177_31116_object44833:head# 2024-09-13T11:24:39.407 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.410 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4210a277:::benchmark_data_smithi177_31116_object79003:head# 2024-09-13T11:24:39.410 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.413 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:421110dc:::benchmark_data_smithi177_31116_object82497:head# 2024-09-13T11:24:39.413 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.413 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:421147a6:::benchmark_data_smithi177_31116_object49433:head# 2024-09-13T11:24:39.413 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.414 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:421218ab:::benchmark_data_smithi177_31116_object85808:head# 2024-09-13T11:24:39.414 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.416 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:42140b53:::benchmark_data_smithi177_31116_object96525:head# 2024-09-13T11:24:39.416 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.417 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:42143735:::benchmark_data_smithi177_31116_object1668:head# 2024-09-13T11:24:39.417 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.417 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:42144f94:::benchmark_data_smithi177_31116_object76689:head# 2024-09-13T11:24:39.417 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.418 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:421478c8:::benchmark_data_smithi177_31116_object57983:head# 2024-09-13T11:24:39.418 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.418 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:42152cb5:::benchmark_data_smithi177_31116_object61134:head# 2024-09-13T11:24:39.419 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.419 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:42154ca9:::benchmark_data_smithi177_31116_object51303:head# 2024-09-13T11:24:39.420 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.420 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:42167505:::benchmark_data_smithi177_31116_object40874:head# 2024-09-13T11:24:39.420 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.420 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:42167a63:::benchmark_data_smithi177_31116_object11940:head# 2024-09-13T11:24:39.420 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.424 INFO:teuthology.orchestra.run.smithi001.stdout:osd.9: {} 2024-09-13T11:24:39.424 INFO:teuthology.orchestra.run.smithi001.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-13T11:24:39.425 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4216ca27:::benchmark_data_smithi177_31116_object483:head# 2024-09-13T11:24:39.425 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.427 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4216cba0:::benchmark_data_smithi177_31116_object18126:head# 2024-09-13T11:24:39.427 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.428 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4216e0e9:::benchmark_data_smithi177_31116_object17316:head# 2024-09-13T11:24:39.428 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.431 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:421709d0:::benchmark_data_smithi177_31116_object36452:head# 2024-09-13T11:24:39.431 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.431 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:42174e98:::benchmark_data_smithi177_31116_object100010:head# 2024-09-13T11:24:39.431 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.433 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:421793d1:::benchmark_data_smithi177_31116_object34774:head# 2024-09-13T11:24:39.433 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.433 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4218e965:::benchmark_data_smithi177_31116_object18763:head# 2024-09-13T11:24:39.434 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.436 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:42190843:::benchmark_data_smithi177_31116_object42751:head# 2024-09-13T11:24:39.436 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.436 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:42195d52:::benchmark_data_smithi177_31116_object82617:head# 2024-09-13T11:24:39.436 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.436 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4219c5d8:::benchmark_data_smithi177_31116_object91479:head# 2024-09-13T11:24:39.436 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.437 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4219f51e:::benchmark_data_smithi177_31116_object41894:head# 2024-09-13T11:24:39.437 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.438 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:421a18c6:::benchmark_data_smithi177_31116_object24633:head# 2024-09-13T11:24:39.438 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.439 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:421aeece:::benchmark_data_smithi177_31116_object11631:head# 2024-09-13T11:24:39.439 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.440 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:421af8dc:::benchmark_data_smithi177_31116_object88246:head# 2024-09-13T11:24:39.440 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.440 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:421b1b64:::benchmark_data_smithi177_31116_object110313:head# 2024-09-13T11:24:39.440 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.441 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:421b5864:::benchmark_data_smithi177_31116_object78945:head# 2024-09-13T11:24:39.441 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.442 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:421c1c7e:::benchmark_data_smithi177_31116_object95517:head# 2024-09-13T11:24:39.442 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.442 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:421c5d9a:::benchmark_data_smithi177_31116_object62822:head# 2024-09-13T11:24:39.442 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.442 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:421c6855:::benchmark_data_smithi177_31116_object95097:head# 2024-09-13T11:24:39.443 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.443 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:421ce97c:::benchmark_data_smithi177_31116_object43331:head# 2024-09-13T11:24:39.443 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.444 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:421d6c4d:::benchmark_data_smithi177_31116_object90102:head# 2024-09-13T11:24:39.444 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.444 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:421f01c3:::benchmark_data_smithi177_31116_object58842:head# 2024-09-13T11:24:39.444 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.445 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:421f7618:::benchmark_data_smithi177_31116_object4097:head# 2024-09-13T11:24:39.445 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.445 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:421fa0f2:::benchmark_data_smithi177_31116_object26952:head# 2024-09-13T11:24:39.445 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.446 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:421fac61:::benchmark_data_smithi177_31116_object15859:head# 2024-09-13T11:24:39.447 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.447 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:42206407:::benchmark_data_smithi177_31116_object78725:head# 2024-09-13T11:24:39.447 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.447 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4220a6ed:::benchmark_data_smithi177_31116_object76098:head# 2024-09-13T11:24:39.447 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.448 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4220e71c:::benchmark_data_smithi177_31116_object78408:head# 2024-09-13T11:24:39.448 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.448 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:42231a55:::benchmark_data_smithi177_31116_object38310:head# 2024-09-13T11:24:39.448 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.449 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4223a567:::benchmark_data_smithi177_31116_object62124:head# 2024-09-13T11:24:39.449 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.449 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4223c16a:::benchmark_data_smithi177_31116_object22207:head# 2024-09-13T11:24:39.449 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.450 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4225f664:::benchmark_data_smithi177_31116_object44298:head# 2024-09-13T11:24:39.450 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.450 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:42271a7e:::benchmark_data_smithi177_31116_object34823:head# 2024-09-13T11:24:39.450 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.451 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:42277ca8:::benchmark_data_smithi177_31116_object62935:head# 2024-09-13T11:24:39.451 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.451 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4227e189:::benchmark_data_smithi177_31116_object81386:head# 2024-09-13T11:24:39.451 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.452 INFO:teuthology.orchestra.run.smithi001.stdout:osd.10: {} 2024-09-13T11:24:39.452 INFO:teuthology.orchestra.run.smithi001.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-13T11:24:39.453 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4228cf54:::benchmark_data_smithi177_31116_object86453:head# 2024-09-13T11:24:39.453 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.453 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:42299bdf:::benchmark_data_smithi177_31116_object80785:head# 2024-09-13T11:24:39.453 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.453 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4229aef5:::benchmark_data_smithi177_31116_object50559:head# 2024-09-13T11:24:39.453 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.454 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4229cc77:::benchmark_data_smithi177_31116_object34641:head# 2024-09-13T11:24:39.454 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.454 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:422a272c:::benchmark_data_smithi177_31116_object83945:head# 2024-09-13T11:24:39.454 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.455 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:422a3434:::benchmark_data_smithi177_31116_object67292:head# 2024-09-13T11:24:39.455 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.455 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:422a4afe:::benchmark_data_smithi177_31116_object63407:head# 2024-09-13T11:24:39.455 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.456 INFO:teuthology.orchestra.run.smithi092.stdout:ERROR: (22) Invalid argument 2024-09-13T11:24:39.456 INFO:teuthology.orchestra.run.smithi092.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:24:39.456 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:422a5d0c:::benchmark_data_smithi177_31116_object21787:head# 2024-09-13T11:24:39.456 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.458 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:422b97d3:::benchmark_data_smithi177_31116_object81128:head# 2024-09-13T11:24:39.459 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.459 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:422befac:::benchmark_data_smithi177_31116_object62327:head# 2024-09-13T11:24:39.459 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.460 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:422c1946:::benchmark_data_smithi177_31116_object11620:head# 2024-09-13T11:24:39.460 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.463 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:422c420d:::benchmark_data_smithi177_31116_object34746:head# 2024-09-13T11:24:39.463 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.466 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:422e2045:::benchmark_data_smithi177_31116_object30093:head# 2024-09-13T11:24:39.466 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.468 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:422ed172:::benchmark_data_smithi177_31116_object42210:head# 2024-09-13T11:24:39.468 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.469 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:422ee9c1:::benchmark_data_smithi177_31116_object105408:head# 2024-09-13T11:24:39.469 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.472 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:422f0586:::benchmark_data_smithi177_31116_object18790:head# 2024-09-13T11:24:39.472 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.475 INFO:teuthology.orchestra.run.smithi001.stdout:osd.11: {} 2024-09-13T11:24:39.475 INFO:teuthology.orchestra.run.smithi001.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-13T11:24:39.476 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4230ff6d:::benchmark_data_smithi177_31116_object96321:head# 2024-09-13T11:24:39.476 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.478 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4231b0d8:::benchmark_data_smithi177_31116_object10085:head# 2024-09-13T11:24:39.478 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.479 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_ops_in_flight 2024-09-13T11:24:39.480 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4232e607:::benchmark_data_smithi177_31116_object4236:head# 2024-09-13T11:24:39.481 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.483 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4232f1a6:::benchmark_data_smithi177_31116_object30399:head# 2024-09-13T11:24:39.483 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.485 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:423303b8:::benchmark_data_smithi177_31116_object79667:head# 2024-09-13T11:24:39.485 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.486 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:42337336:::benchmark_data_smithi177_31116_object27193:head# 2024-09-13T11:24:39.486 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.488 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:42341fe5:::benchmark_data_smithi177_31116_object61761:head# 2024-09-13T11:24:39.488 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.489 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4234341a:::benchmark_data_smithi177_31116_object80541:head# 2024-09-13T11:24:39.489 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.490 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4234ee65:::benchmark_data_smithi177_31116_object83087:head# 2024-09-13T11:24:39.490 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.490 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4235066c:::benchmark_data_smithi177_31116_object76136:head# 2024-09-13T11:24:39.491 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.491 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:42354128:::benchmark_data_smithi177_31116_object38162:head# 2024-09-13T11:24:39.491 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.492 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:42358e2b:::benchmark_data_smithi177_31116_object7948:head# 2024-09-13T11:24:39.492 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.492 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4235a7d2:::benchmark_data_smithi177_31116_object50855:head# 2024-09-13T11:24:39.492 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.493 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:42363918:::benchmark_data_smithi177_31116_object12677:head# 2024-09-13T11:24:39.493 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.494 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:42366936:::benchmark_data_smithi177_31116_object28734:head# 2024-09-13T11:24:39.494 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.495 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4237955f:::benchmark_data_smithi177_31116_object108817:head# 2024-09-13T11:24:39.495 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.496 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4237b303:::benchmark_data_smithi177_31116_object41969:head# 2024-09-13T11:24:39.496 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.497 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:423824ae:::benchmark_data_smithi177_31116_object33935:head# 2024-09-13T11:24:39.497 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.497 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:42383c70:::benchmark_data_smithi177_31116_object20529:head# 2024-09-13T11:24:39.497 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.498 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:42383f3a:::benchmark_data_smithi177_31116_object98431:head# 2024-09-13T11:24:39.498 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.499 INFO:teuthology.orchestra.run.smithi001.stdout:osd.12: {} 2024-09-13T11:24:39.499 INFO:teuthology.orchestra.run.smithi001.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-13T11:24:39.499 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:42387358:::benchmark_data_smithi177_31116_object95630:head# 2024-09-13T11:24:39.499 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.499 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:42387751:::benchmark_data_smithi177_31116_object20044:head# 2024-09-13T11:24:39.500 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.500 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4238a213:::benchmark_data_smithi177_31116_object81610:head# 2024-09-13T11:24:39.501 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.501 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4239d69c:::benchmark_data_smithi177_31116_object33064:head# 2024-09-13T11:24:39.501 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.501 INFO:tasks.ceph.osd.8.smithi001.stderr:2024-09-13T11:24:39.499+0000 7fb9b48ae640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26312) UID: 0 2024-09-13T11:24:39.502 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4239dafb:::benchmark_data_smithi177_31116_object59922:head# 2024-09-13T11:24:39.502 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.502 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:423ae075:::benchmark_data_smithi177_31116_object3746:head# 2024-09-13T11:24:39.502 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.504 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:423b1ca3:::benchmark_data_smithi177_31116_object64554:head# 2024-09-13T11:24:39.505 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.505 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:423b58aa:::benchmark_data_smithi177_31116_object53971:head# 2024-09-13T11:24:39.506 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.508 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:423b92e9:::benchmark_data_smithi177_31116_object40027:head# 2024-09-13T11:24:39.508 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.509 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:423e086d:::benchmark_data_smithi177_31116_object39051:head# 2024-09-13T11:24:39.509 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.510 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:423f5876:::benchmark_data_smithi177_31116_object55157:head# 2024-09-13T11:24:39.510 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.511 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:423f621c:::benchmark_data_smithi177_31116_object59193:head# 2024-09-13T11:24:39.511 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.511 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:423f78f3:::benchmark_data_smithi177_31116_object50818:head# 2024-09-13T11:24:39.511 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.512 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:424029bd:::benchmark_data_smithi177_31116_object69854:head# 2024-09-13T11:24:39.513 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.513 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:424060a7:::benchmark_data_smithi177_31116_object107319:head# 2024-09-13T11:24:39.516 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.517 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4241262c:::benchmark_data_smithi177_31116_object2282:head# 2024-09-13T11:24:39.517 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.517 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4241e74a:::benchmark_data_smithi177_31116_object106333:head# 2024-09-13T11:24:39.517 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.518 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:42433669:::benchmark_data_smithi177_31116_object63140:head# 2024-09-13T11:24:39.518 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.518 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4243bd65:::benchmark_data_smithi177_31116_object90762:head# 2024-09-13T11:24:39.518 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.518 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4243cea5:::benchmark_data_smithi177_31116_object33629:head# 2024-09-13T11:24:39.518 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.520 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:42447ddd:::benchmark_data_smithi177_31116_object31742:head# 2024-09-13T11:24:39.520 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.520 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:42457235:::benchmark_data_smithi177_31116_object19803:head# 2024-09-13T11:24:39.520 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.523 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4245d429:::benchmark_data_smithi177_31116_object68084:head# 2024-09-13T11:24:39.523 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.523 INFO:teuthology.orchestra.run.smithi001.stdout:osd.13: {} 2024-09-13T11:24:39.524 INFO:teuthology.orchestra.run.smithi001.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-13T11:24:39.525 INFO:teuthology.orchestra.run.smithi001.stderr:Error ENXIO: problem getting command descriptions from osd.14 2024-09-13T11:24:39.525 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:42462273:::benchmark_data_smithi177_31116_object4072:head# 2024-09-13T11:24:39.525 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.526 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:42466555:::benchmark_data_smithi177_31116_object62389:head# 2024-09-13T11:24:39.527 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.531 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4246a999:::benchmark_data_smithi177_31116_object105436:head# 2024-09-13T11:24:39.531 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.531 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:424732c9:::benchmark_data_smithi177_31116_object69766:head# 2024-09-13T11:24:39.531 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.533 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:424744e2:::benchmark_data_smithi177_31116_object45571:head# 2024-09-13T11:24:39.533 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.535 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4247694c:::benchmark_data_smithi177_31116_object81887:head# 2024-09-13T11:24:39.535 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.536 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:42479b2c:::benchmark_data_smithi177_31116_object106879:head# 2024-09-13T11:24:39.536 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.538 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:42489a62:::benchmark_data_smithi177_31116_object14761:head# 2024-09-13T11:24:39.538 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.545 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:42491bed:::benchmark_data_smithi177_31116_object15969:head# 2024-09-13T11:24:39.545 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.547 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:424979db:::benchmark_data_smithi177_31116_object29113:head# 2024-09-13T11:24:39.547 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.547 INFO:teuthology.orchestra.run.smithi001.stdout:osd.15: {} 2024-09-13T11:24:39.548 INFO:teuthology.orchestra.run.smithi001.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-13T11:24:39.549 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:424b222d:::benchmark_data_smithi177_31116_object96496:head# 2024-09-13T11:24:39.549 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.551 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-13T11:24:39.551 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-13T11:24:39.552 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:424b324c:::benchmark_data_smithi177_31116_object76961:head# 2024-09-13T11:24:39.552 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.555 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:424bf8f7:::benchmark_data_smithi177_31116_object75047:head# 2024-09-13T11:24:39.555 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.555 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:424c9e53:::benchmark_data_smithi177_31116_object28803:head# 2024-09-13T11:24:39.555 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.556 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:424d2517:::benchmark_data_smithi177_31116_object55004:head# 2024-09-13T11:24:39.556 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.556 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:424e602f:::benchmark_data_smithi177_31116_object55553:head# 2024-09-13T11:24:39.557 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.557 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:424e8e9a:::benchmark_data_smithi177_31116_object39766:head# 2024-09-13T11:24:39.557 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.558 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:424f14f7:::benchmark_data_smithi177_31116_object30205:head# 2024-09-13T11:24:39.558 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.559 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:425199c2:::benchmark_data_smithi177_31116_object29031:head# 2024-09-13T11:24:39.559 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.559 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:42522cb7:::benchmark_data_smithi177_31116_object99361:head# 2024-09-13T11:24:39.559 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.560 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4252772a:::benchmark_data_smithi177_31116_object76660:head# 2024-09-13T11:24:39.560 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.560 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4254c5bd:::benchmark_data_smithi177_31116_object49682:head# 2024-09-13T11:24:39.560 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.561 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:42552b24:::benchmark_data_smithi177_31116_object15445:head# 2024-09-13T11:24:39.561 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.561 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:425570b8:::benchmark_data_smithi177_31116_object42103:head# 2024-09-13T11:24:39.561 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.562 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4256860f:::benchmark_data_smithi177_31116_object69160:head# 2024-09-13T11:24:39.562 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.562 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4256894a:::benchmark_data_smithi177_31116_object36006:head# 2024-09-13T11:24:39.563 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.563 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:42568e14:::benchmark_data_smithi177_31116_object105249:head# 2024-09-13T11:24:39.563 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.564 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4256f10c:::benchmark_data_smithi177_31116_object17698:head# 2024-09-13T11:24:39.564 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.564 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:42572307:::benchmark_data_smithi177_31116_object69090:head# 2024-09-13T11:24:39.565 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.565 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:42577009:::benchmark_data_smithi177_31116_object62743:head# 2024-09-13T11:24:39.565 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.566 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4258a8aa:::benchmark_data_smithi177_31116_object96058:head# 2024-09-13T11:24:39.566 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.569 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:42596ae7:::benchmark_data_smithi177_31116_object87586:head# 2024-09-13T11:24:39.569 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.572 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4259cef7:::benchmark_data_smithi177_31116_object58492:head# 2024-09-13T11:24:39.572 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.573 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:425a9632:::benchmark_data_smithi177_31116_object40761:head# 2024-09-13T11:24:39.573 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.574 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:425b8a6c:::benchmark_data_smithi177_31116_object100332:head# 2024-09-13T11:24:39.575 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.577 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:425bb141:::benchmark_data_smithi177_31116_object23241:head# 2024-09-13T11:24:39.578 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.580 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:425bbc34:::benchmark_data_smithi177_31116_object85847:head# 2024-09-13T11:24:39.581 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.581 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:425cc875:::benchmark_data_smithi177_31116_object38457:head# 2024-09-13T11:24:39.581 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.584 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:425cdea9:::benchmark_data_smithi177_31116_object95168:head# 2024-09-13T11:24:39.585 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.585 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:425cff8e:::benchmark_data_smithi177_31116_object84089:head# 2024-09-13T11:24:39.585 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.588 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:425d7a67:::benchmark_data_smithi177_31116_object92939:head# 2024-09-13T11:24:39.588 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.589 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:425daad6:::benchmark_data_smithi177_31116_object102397:head# 2024-09-13T11:24:39.589 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.591 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:425e97dc:::benchmark_data_smithi177_31116_object82912:head# 2024-09-13T11:24:39.591 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.592 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:425f19ec:::benchmark_data_smithi177_31116_object88578:head# 2024-09-13T11:24:39.592 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.594 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:425f4530:::benchmark_data_smithi177_31116_object90006:head# 2024-09-13T11:24:39.595 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.597 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:425f6ca0:::benchmark_data_smithi177_31116_object2300:head# 2024-09-13T11:24:39.597 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.598 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:425fa8e1:::benchmark_data_smithi177_31116_object62108:head# 2024-09-13T11:24:39.598 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.601 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:425fcf41:::benchmark_data_smithi177_31116_object74719:head# 2024-09-13T11:24:39.601 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.602 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-13T11:24:39.599+0000 7f6a68775640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26307) UID: 0 2024-09-13T11:24:39.602 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:426098ce:::benchmark_data_smithi177_31116_object84477:head# 2024-09-13T11:24:39.602 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.604 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4260f523:::benchmark_data_smithi177_31116_object51930:head# 2024-09-13T11:24:39.604 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.605 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4262019b:::benchmark_data_smithi177_31116_object83713:head# 2024-09-13T11:24:39.605 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.608 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:42623a1c:::benchmark_data_smithi177_31116_object66729:head# 2024-09-13T11:24:39.608 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.608 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4262be1c:::benchmark_data_smithi177_31116_object85707:head# 2024-09-13T11:24:39.609 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.610 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:42632a62:::benchmark_data_smithi177_31116_object63602:head# 2024-09-13T11:24:39.610 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.610 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:42637cdf:::benchmark_data_smithi177_31116_object10265:head# 2024-09-13T11:24:39.610 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.611 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4264665a:::benchmark_data_smithi177_31116_object3790:head# 2024-09-13T11:24:39.611 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.612 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:42648fa0:::benchmark_data_smithi177_31116_object97589:head# 2024-09-13T11:24:39.612 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.612 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4264fa81:::benchmark_data_smithi177_31116_object60130:head# 2024-09-13T11:24:39.613 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.613 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4265790c:::benchmark_data_smithi177_31116_object14059:head# 2024-09-13T11:24:39.613 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.614 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4265cc41:::benchmark_data_smithi177_31116_object72392:head# 2024-09-13T11:24:39.614 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.615 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:426690c4:::benchmark_data_smithi177_31116_object80831:head# 2024-09-13T11:24:39.615 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.615 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:42676066:::benchmark_data_smithi177_31116_object106003:head# 2024-09-13T11:24:39.615 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.616 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4267b9a6:::benchmark_data_smithi177_31116_object28557:head# 2024-09-13T11:24:39.616 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.617 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:42688fbf:::benchmark_data_smithi177_31116_object8423:head# 2024-09-13T11:24:39.617 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.618 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:42692b96:::benchmark_data_smithi177_31116_object52190:head# 2024-09-13T11:24:39.618 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.619 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:42696f4d:::benchmark_data_smithi177_31116_object83711:head# 2024-09-13T11:24:39.619 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.619 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4269db30:::benchmark_data_smithi177_31116_object27856:head# 2024-09-13T11:24:39.619 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.620 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:426ac6e9:::benchmark_data_smithi177_31116_object19928:head# 2024-09-13T11:24:39.620 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.621 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:426af162:::benchmark_data_smithi177_31116_object20231:head# 2024-09-13T11:24:39.621 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.621 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:426c5716:::benchmark_data_smithi177_31116_object108753:head# 2024-09-13T11:24:39.622 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.622 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:426dc482:::benchmark_data_smithi177_31116_object24165:head# 2024-09-13T11:24:39.622 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.623 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:426fa4bc:::benchmark_data_smithi177_31116_object3846:head# 2024-09-13T11:24:39.623 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.623 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4270b0a0:::benchmark_data_smithi177_31116_object50539:head# 2024-09-13T11:24:39.624 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.624 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4271d86f:::benchmark_data_smithi177_31116_object104835:head# 2024-09-13T11:24:39.624 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.625 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:42725d1d:::benchmark_data_smithi177_31116_object54610:head# 2024-09-13T11:24:39.626 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.626 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4273d866:::benchmark_data_smithi177_31116_object20433:head# 2024-09-13T11:24:39.626 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.627 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4274119b:::benchmark_data_smithi177_31116_object90863:head# 2024-09-13T11:24:39.627 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.628 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:42741554:::benchmark_data_smithi177_31116_object37764:head# 2024-09-13T11:24:39.628 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.628 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:42747ee7:::benchmark_data_smithi177_31116_object4213:head# 2024-09-13T11:24:39.629 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.629 INFO:teuthology.orchestra.run.smithi092.stdout:ERROR: (22) Invalid argument 2024-09-13T11:24:39.629 INFO:teuthology.orchestra.run.smithi092.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:24:39.630 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4274bb20:::benchmark_data_smithi177_31116_object76330:head# 2024-09-13T11:24:39.630 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.630 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4275b4f1:::benchmark_data_smithi177_31116_object31829:head# 2024-09-13T11:24:39.630 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.633 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:427648b3:::benchmark_data_smithi177_31116_object64842:head# 2024-09-13T11:24:39.633 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.634 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4276e010:::benchmark_data_smithi177_31116_object77849:head# 2024-09-13T11:24:39.634 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.635 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4277a820:::benchmark_data_smithi177_31116_object46443:head# 2024-09-13T11:24:39.635 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.635 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:427801dd:::benchmark_data_smithi177_31116_object97377:head# 2024-09-13T11:24:39.635 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.636 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4278adc7:::benchmark_data_smithi177_31116_object30453:head# 2024-09-13T11:24:39.636 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.637 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:42791a42:::benchmark_data_smithi177_31116_object86799:head# 2024-09-13T11:24:39.637 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.640 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:427a23f9:::benchmark_data_smithi177_31116_object62839:head# 2024-09-13T11:24:39.640 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.640 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:427a3ecb:::benchmark_data_smithi177_31116_object37926:head# 2024-09-13T11:24:39.640 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.642 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:427a8670:::benchmark_data_smithi177_31116_object88663:head# 2024-09-13T11:24:39.642 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.645 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:427b5941:::benchmark_data_smithi177_31116_object13260:head# 2024-09-13T11:24:39.645 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.646 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:427be147:::benchmark_data_smithi177_31116_object67786:head# 2024-09-13T11:24:39.646 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.646 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:427d02b9:::benchmark_data_smithi177_31116_object68421:head# 2024-09-13T11:24:39.647 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.647 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:427d92ae:::benchmark_data_smithi177_31116_object8254:head# 2024-09-13T11:24:39.648 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.648 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:427efe55:::benchmark_data_smithi177_31116_object70750:head# 2024-09-13T11:24:39.648 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.649 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_blocked_ops 2024-09-13T11:24:39.650 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:427fa6b5:::benchmark_data_smithi177_31116_object22676:head# 2024-09-13T11:24:39.650 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.654 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:42808b4f:::benchmark_data_smithi177_31116_object47170:head# 2024-09-13T11:24:39.654 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.654 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:42817344:::benchmark_data_smithi177_31116_object41357:head# 2024-09-13T11:24:39.654 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.657 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:428191a8:::benchmark_data_smithi177_31116_object97945:head# 2024-09-13T11:24:39.657 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.657 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:428220e4:::benchmark_data_smithi177_31116_object17664:head# 2024-09-13T11:24:39.658 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.659 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:42825729:::benchmark_data_smithi177_31116_object61452:head# 2024-09-13T11:24:39.660 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.662 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:428388e2:::benchmark_data_smithi177_31116_object21704:head# 2024-09-13T11:24:39.662 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.662 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4283cb55:::benchmark_data_smithi177_31116_object68912:head# 2024-09-13T11:24:39.663 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.663 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:428531c0:::benchmark_data_smithi177_31116_object3510:head# 2024-09-13T11:24:39.663 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.665 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:42855831:::benchmark_data_smithi177_31116_object24905:head# 2024-09-13T11:24:39.665 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.666 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4285edc4:::benchmark_data_smithi177_31116_object70008:head# 2024-09-13T11:24:39.666 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.667 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:428725b0:::benchmark_data_smithi177_31116_object72757:head# 2024-09-13T11:24:39.667 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.668 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4287b9c4:::benchmark_data_smithi177_31116_object3274:head# 2024-09-13T11:24:39.668 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.669 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:428902db:::benchmark_data_smithi177_31116_object29771:head# 2024-09-13T11:24:39.669 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.669 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4289785a:::benchmark_data_smithi177_31116_object65989:head# 2024-09-13T11:24:39.670 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.670 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:42898c9e:::benchmark_data_smithi177_31116_object82259:head# 2024-09-13T11:24:39.670 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.671 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:428a9264:::benchmark_data_smithi177_31116_object94620:head# 2024-09-13T11:24:39.671 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.672 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:428ade08:::benchmark_data_smithi177_31116_object61607:head# 2024-09-13T11:24:39.672 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.673 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:428b1806:::benchmark_data_smithi177_31116_object89653:head# 2024-09-13T11:24:39.673 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.673 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:428c40a9:::benchmark_data_smithi177_31116_object89760:head# 2024-09-13T11:24:39.674 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.674 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:428c8086:::benchmark_data_smithi177_31116_object103483:head# 2024-09-13T11:24:39.674 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.675 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:428e3b11:::benchmark_data_smithi177_31116_object99470:head# 2024-09-13T11:24:39.675 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.676 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:429083e8:::benchmark_data_smithi177_31116_object60250:head# 2024-09-13T11:24:39.676 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.677 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:42910a00:::benchmark_data_smithi177_31116_object36989:head# 2024-09-13T11:24:39.677 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.677 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4291878e:::benchmark_data_smithi177_31116_object70700:head# 2024-09-13T11:24:39.677 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.678 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:42921fa6:::benchmark_data_smithi177_31116_object94838:head# 2024-09-13T11:24:39.678 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.679 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:42922964:::benchmark_data_smithi177_31116_object73328:head# 2024-09-13T11:24:39.679 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.680 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4292dd80:::benchmark_data_smithi177_31116_object30777:head# 2024-09-13T11:24:39.680 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.681 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:429324e8:::benchmark_data_smithi177_31116_object76087:head# 2024-09-13T11:24:39.681 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.681 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:42932619:::benchmark_data_smithi177_31116_object20165:head# 2024-09-13T11:24:39.681 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.682 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:42934085:::benchmark_data_smithi177_31116_object25276:head# 2024-09-13T11:24:39.682 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.683 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:42934db4:::benchmark_data_smithi177_31116_object81539:head# 2024-09-13T11:24:39.683 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.686 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:42936a79:::benchmark_data_smithi177_31116_object69313:head# 2024-09-13T11:24:39.686 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.687 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:42946cf4:::benchmark_data_smithi177_31116_object88580:head# 2024-09-13T11:24:39.687 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.687 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:42946fe1:::benchmark_data_smithi177_31116_object56640:head# 2024-09-13T11:24:39.688 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.688 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4294c610:::benchmark_data_smithi177_31116_object85135:head# 2024-09-13T11:24:39.688 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.689 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4295acdf:::benchmark_data_smithi177_31116_object1518:head# 2024-09-13T11:24:39.689 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.690 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:42966619:::benchmark_data_smithi177_31116_object53539:head# 2024-09-13T11:24:39.690 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.691 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4296c09e:::benchmark_data_smithi177_31116_object80196:head# 2024-09-13T11:24:39.691 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.692 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4296df08:::benchmark_data_smithi177_31116_object6752:head# 2024-09-13T11:24:39.692 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.693 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:42975432:::benchmark_data_smithi177_31116_object720:head# 2024-09-13T11:24:39.693 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.694 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4297f852:::benchmark_data_smithi177_31116_object72586:head# 2024-09-13T11:24:39.694 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.695 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4297f9a9:::benchmark_data_smithi177_31116_object6222:head# 2024-09-13T11:24:39.695 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.697 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:42981ba3:::benchmark_data_smithi177_31116_object32935:head# 2024-09-13T11:24:39.697 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.700 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:42988296:::benchmark_data_smithi177_31116_object39602:head# 2024-09-13T11:24:39.700 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.701 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:42993bf1:::benchmark_data_smithi177_31116_object31960:head# 2024-09-13T11:24:39.701 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.702 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:429949fa:::benchmark_data_smithi177_31116_object70623:head# 2024-09-13T11:24:39.702 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.702 INFO:tasks.ceph.osd.5.smithi092.stderr:2024-09-13T11:24:39.698+0000 7f6f0b705640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26346) UID: 0 2024-09-13T11:24:39.703 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:42999a92:::benchmark_data_smithi177_31116_object82351:head# 2024-09-13T11:24:39.703 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.704 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4299f84a:::benchmark_data_smithi177_31116_object99869:head# 2024-09-13T11:24:39.704 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.707 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:429a8ca4:::benchmark_data_smithi177_31116_object105839:head# 2024-09-13T11:24:39.707 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.709 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:429aac61:::benchmark_data_smithi177_31116_object46229:head# 2024-09-13T11:24:39.709 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.712 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:429b97bd:::benchmark_data_smithi177_31116_object70380:head# 2024-09-13T11:24:39.713 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.713 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:429be747:::benchmark_data_smithi177_31116_object62110:head# 2024-09-13T11:24:39.713 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.713 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:429cb509:::benchmark_data_smithi177_31116_object6492:head# 2024-09-13T11:24:39.714 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.716 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:429d2f2a:::benchmark_data_smithi177_31116_object27499:head# 2024-09-13T11:24:39.716 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.719 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:42a019df:::benchmark_data_smithi177_31116_object57814:head# 2024-09-13T11:24:39.719 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.722 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:42a05d54:::benchmark_data_smithi177_31116_object24359:head# 2024-09-13T11:24:39.722 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.725 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:42a099c8:::benchmark_data_smithi177_31116_object72317:head# 2024-09-13T11:24:39.725 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.727 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:42a0f9cf:::benchmark_data_smithi177_31116_object94496:head# 2024-09-13T11:24:39.727 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.730 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:42a1359e:::benchmark_data_smithi177_31116_object66171:head# 2024-09-13T11:24:39.730 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.733 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:42a1cfd5:::benchmark_data_smithi177_31116_object49946:head# 2024-09-13T11:24:39.733 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.733 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:42a22955:::benchmark_data_smithi177_31116_object6644:head# 2024-09-13T11:24:39.734 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.735 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:42a245d7:::benchmark_data_smithi177_31116_object78640:head# 2024-09-13T11:24:39.735 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.739 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:42a3c94f:::benchmark_data_smithi177_31116_object22125:head# 2024-09-13T11:24:39.739 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.740 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:42a3f64b:::benchmark_data_smithi177_31116_object97279:head# 2024-09-13T11:24:39.740 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.741 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:42a4e4d6:::benchmark_data_smithi177_31116_object65038:head# 2024-09-13T11:24:39.741 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.744 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:42a5b133:::benchmark_data_smithi177_31116_object73624:head# 2024-09-13T11:24:39.744 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.747 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:42a6430c:::benchmark_data_smithi177_31116_object30111:head# 2024-09-13T11:24:39.747 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.749 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:42a7253e:::benchmark_data_smithi177_31116_object56482:head# 2024-09-13T11:24:39.749 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.751 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:42a73a42:::benchmark_data_smithi177_31116_object5846:head# 2024-09-13T11:24:39.751 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.752 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:42a77d41:::benchmark_data_smithi177_31116_object9096:head# 2024-09-13T11:24:39.752 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.752 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:42a7e8f5:::benchmark_data_smithi177_31116_object17904:head# 2024-09-13T11:24:39.753 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.753 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:42a8312f:::benchmark_data_smithi177_31116_object50877:head# 2024-09-13T11:24:39.753 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.753 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:42a83e85:::benchmark_data_smithi177_31116_object21234:head# 2024-09-13T11:24:39.753 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.754 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:42a93940:::benchmark_data_smithi177_31116_object12289:head# 2024-09-13T11:24:39.754 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.754 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:42a94ac1:::benchmark_data_smithi177_31116_object62919:head# 2024-09-13T11:24:39.755 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.755 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:42a9a7ea:::benchmark_data_smithi177_31116_object33742:head# 2024-09-13T11:24:39.755 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.756 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:42abfd46:::benchmark_data_smithi177_31116_object18943:head# 2024-09-13T11:24:39.756 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.757 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:42ac9098:::benchmark_data_smithi177_31116_object89488:head# 2024-09-13T11:24:39.757 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.757 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:42ad1a6d:::benchmark_data_smithi177_31116_object55337:head# 2024-09-13T11:24:39.757 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.758 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:42ad41cd:::benchmark_data_smithi177_31116_object20425:head# 2024-09-13T11:24:39.758 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.758 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:42ae31a6:::benchmark_data_smithi177_31116_object9236:head# 2024-09-13T11:24:39.759 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.759 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:42ae6291:::benchmark_data_smithi177_31116_object59569:head# 2024-09-13T11:24:39.759 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.760 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:42aeb69f:::benchmark_data_smithi177_31116_object25877:head# 2024-09-13T11:24:39.760 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.761 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:42af766e:::benchmark_data_smithi177_31116_object65922:head# 2024-09-13T11:24:39.761 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.762 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:42b0e450:::benchmark_data_smithi177_31116_object30945:head# 2024-09-13T11:24:39.762 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.762 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:42b14589:::benchmark_data_smithi177_31116_object80782:head# 2024-09-13T11:24:39.762 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.763 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:42b19fa0:::benchmark_data_smithi177_31116_object46810:head# 2024-09-13T11:24:39.763 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.764 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:42b26a21:::benchmark_data_smithi177_31116_object46412:head# 2024-09-13T11:24:39.764 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.767 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:42b41476:::benchmark_data_smithi177_31116_object76799:head# 2024-09-13T11:24:39.767 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.770 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:42b51343:::benchmark_data_smithi177_31116_object15350:head# 2024-09-13T11:24:39.770 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.771 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:42b5d887:::benchmark_data_smithi177_31116_object95962:head# 2024-09-13T11:24:39.771 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.773 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:42b601ad:::benchmark_data_smithi177_31116_object77037:head# 2024-09-13T11:24:39.773 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.774 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:42b64a0b:::benchmark_data_smithi177_31116_object1078:head# 2024-09-13T11:24:39.774 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.775 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:42b6ec5a:::benchmark_data_smithi177_31116_object109649:head# 2024-09-13T11:24:39.775 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.775 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:42b7340f:::benchmark_data_smithi177_31116_object6651:head# 2024-09-13T11:24:39.775 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.776 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:42b83256:::benchmark_data_smithi177_31116_object78509:head# 2024-09-13T11:24:39.776 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.777 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:42b85280:::benchmark_data_smithi177_31116_object75012:head# 2024-09-13T11:24:39.777 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.777 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:42b89ed5:::benchmark_data_smithi177_31116_object85972:head# 2024-09-13T11:24:39.777 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.780 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:42b8e142:::benchmark_data_smithi177_31116_object2149:head# 2024-09-13T11:24:39.780 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.781 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:42b9afc7:::benchmark_data_smithi177_31116_object86525:head# 2024-09-13T11:24:39.781 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.781 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:42b9d442:::benchmark_data_smithi177_31116_object70802:head# 2024-09-13T11:24:39.782 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.783 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:42b9e997:::benchmark_data_smithi177_31116_object99413:head# 2024-09-13T11:24:39.783 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.785 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:42bb216f:::benchmark_data_smithi177_31116_object80139:head# 2024-09-13T11:24:39.785 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.786 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:42bb72b0:::benchmark_data_smithi177_31116_object35317:head# 2024-09-13T11:24:39.786 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.789 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:42bbe3b7:::benchmark_data_smithi177_31116_object87856:head# 2024-09-13T11:24:39.789 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.793 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:42bc22e6:::benchmark_data_smithi177_31116_object18462:head# 2024-09-13T11:24:39.793 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.793 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:42bce868:::benchmark_data_smithi177_31116_object60556:head# 2024-09-13T11:24:39.793 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.793 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:42bd28ad:::benchmark_data_smithi177_31116_object25024:head# 2024-09-13T11:24:39.793 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.795 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:42bf660a:::benchmark_data_smithi177_31116_object2888:head# 2024-09-13T11:24:39.795 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.795 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:42c09bb2:::benchmark_data_smithi177_31116_object26201:head# 2024-09-13T11:24:39.796 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.796 INFO:teuthology.orchestra.run.smithi092.stdout:ERROR: (22) Invalid argument 2024-09-13T11:24:39.796 INFO:teuthology.orchestra.run.smithi092.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:24:39.797 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:42c0c9f0:::benchmark_data_smithi177_31116_object35282:head# 2024-09-13T11:24:39.797 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.797 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:42c2b663:::benchmark_data_smithi177_31116_object6274:head# 2024-09-13T11:24:39.797 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.800 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:42c48f6d:::benchmark_data_smithi177_31116_object31997:head# 2024-09-13T11:24:39.800 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.802 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:42c4b6b6:::benchmark_data_smithi177_31116_object30407:head# 2024-09-13T11:24:39.803 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.803 INFO:tasks.ceph.osd.13.smithi092.stderr:2024-09-13T11:24:39.798+0000 7f3e5afea640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26350) UID: 0 2024-09-13T11:24:39.805 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:42c4d348:::benchmark_data_smithi177_31116_object51814:head# 2024-09-13T11:24:39.806 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.807 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:42c527e0:::benchmark_data_smithi177_31116_object94801:head# 2024-09-13T11:24:39.808 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.808 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:42c56ee5:::benchmark_data_smithi177_31116_object28049:head# 2024-09-13T11:24:39.808 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.811 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:42c69baf:::benchmark_data_smithi177_31116_object90315:head# 2024-09-13T11:24:39.811 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.811 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:42c6e343:::benchmark_data_smithi177_31116_object78336:head# 2024-09-13T11:24:39.812 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.814 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:42c85fca:::benchmark_data_smithi177_31116_object348:head# 2024-09-13T11:24:39.814 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.815 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:42c87822:::benchmark_data_smithi177_31116_object48696:head# 2024-09-13T11:24:39.815 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.816 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:42c8799a:::benchmark_data_smithi177_31116_object11016:head# 2024-09-13T11:24:39.816 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.816 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:42c8c78e:::benchmark_data_smithi177_31116_object25444:head# 2024-09-13T11:24:39.817 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.818 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_historic_ops 2024-09-13T11:24:39.819 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:42c8d973:::benchmark_data_smithi177_31116_object101645:head# 2024-09-13T11:24:39.819 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.819 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:42c9c846:::benchmark_data_smithi177_31116_object1215:head# 2024-09-13T11:24:39.819 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.819 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:42ca1898:::benchmark_data_smithi177_31116_object6308:head# 2024-09-13T11:24:39.820 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.820 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:42ca2ede:::benchmark_data_smithi177_31116_object23186:head# 2024-09-13T11:24:39.820 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.821 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:42ca4497:::benchmark_data_smithi177_31116_object65014:head# 2024-09-13T11:24:39.821 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.821 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:42ca9663:::benchmark_data_smithi177_31116_object79115:head# 2024-09-13T11:24:39.822 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.822 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:42ccf6a7:::benchmark_data_smithi177_31116_object18402:head# 2024-09-13T11:24:39.822 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.823 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:42cd1e10:::benchmark_data_smithi177_31116_object28189:head# 2024-09-13T11:24:39.823 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.825 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:42cd761c:::benchmark_data_smithi177_31116_object67706:head# 2024-09-13T11:24:39.825 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.825 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:42ce6f33:::benchmark_data_smithi177_31116_object48264:head# 2024-09-13T11:24:39.825 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.826 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:42ce95e8:::benchmark_data_smithi177_31116_object47550:head# 2024-09-13T11:24:39.826 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.829 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:42d02c66:::benchmark_data_smithi177_31116_object40526:head# 2024-09-13T11:24:39.829 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.832 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:42d102e0:::benchmark_data_smithi177_31116_object38581:head# 2024-09-13T11:24:39.832 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.833 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:42d1af5a:::benchmark_data_smithi177_31116_object69152:head# 2024-09-13T11:24:39.833 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.835 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:42d24762:::benchmark_data_smithi177_31116_object6679:head# 2024-09-13T11:24:39.835 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.838 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:42d3ff15:::benchmark_data_smithi177_31116_object82817:head# 2024-09-13T11:24:39.838 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.841 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:42d43192:::benchmark_data_smithi177_31116_object108276:head# 2024-09-13T11:24:39.841 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.842 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:42d4c78c:::benchmark_data_smithi177_31116_object68346:head# 2024-09-13T11:24:39.842 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.843 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:42d4e66f:::benchmark_data_smithi177_31116_object10433:head# 2024-09-13T11:24:39.843 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.844 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:42d521b7:::benchmark_data_smithi177_31116_object66296:head# 2024-09-13T11:24:39.844 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.845 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:42d561c6:::benchmark_data_smithi177_31116_object65202:head# 2024-09-13T11:24:39.845 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.845 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:42d5cff4:::benchmark_data_smithi177_31116_object15368:head# 2024-09-13T11:24:39.846 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.846 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:42d5e481:::benchmark_data_smithi177_31116_object7463:head# 2024-09-13T11:24:39.846 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.848 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:42d636d9:::benchmark_data_smithi177_31116_object75281:head# 2024-09-13T11:24:39.848 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.849 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:42d64a3c:::benchmark_data_smithi177_31116_object1764:head# 2024-09-13T11:24:39.849 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.850 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:42d6e0c7:::benchmark_data_smithi177_31116_object56456:head# 2024-09-13T11:24:39.850 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.851 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:42d75da7:::benchmark_data_smithi177_31116_object107664:head# 2024-09-13T11:24:39.851 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.852 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:42d761ec:::benchmark_data_smithi177_31116_object93221:head# 2024-09-13T11:24:39.853 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.853 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:42d78444:::benchmark_data_smithi177_31116_object36241:head# 2024-09-13T11:24:39.853 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.854 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:42d88a63:::benchmark_data_smithi177_31116_object106310:head# 2024-09-13T11:24:39.854 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.856 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:42d89b83:::benchmark_data_smithi177_31116_object96726:head# 2024-09-13T11:24:39.856 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.857 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:42d90736:::benchmark_data_smithi177_31116_object96403:head# 2024-09-13T11:24:39.857 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.859 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:42d941a1:::benchmark_data_smithi177_31116_object14752:head# 2024-09-13T11:24:39.859 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.861 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:42d9a4ac:::benchmark_data_smithi177_31116_object79792:head# 2024-09-13T11:24:39.861 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.864 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:42da151b:::benchmark_data_smithi177_31116_object36077:head# 2024-09-13T11:24:39.864 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.865 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:42dac7ab:::benchmark_data_smithi177_31116_object50720:head# 2024-09-13T11:24:39.865 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.867 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:42db4971:::benchmark_data_smithi177_31116_object37310:head# 2024-09-13T11:24:39.867 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.868 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:42dc17e4:::benchmark_data_smithi177_31116_object57970:head# 2024-09-13T11:24:39.868 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.869 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:42dc1d0a:::benchmark_data_smithi177_31116_object33898:head# 2024-09-13T11:24:39.869 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.869 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:42dcb0da:::benchmark_data_smithi177_31116_object42107:head# 2024-09-13T11:24:39.869 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.872 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:42dd6e38:::benchmark_data_smithi177_31116_object29102:head# 2024-09-13T11:24:39.872 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.873 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:42dde282:::benchmark_data_smithi177_31116_object36135:head# 2024-09-13T11:24:39.873 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.874 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:42ddffee:::benchmark_data_smithi177_31116_object9605:head# 2024-09-13T11:24:39.874 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.874 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:42de6fa5:::benchmark_data_smithi177_31116_object13286:head# 2024-09-13T11:24:39.874 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.875 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:42e141b9:::benchmark_data_smithi177_31116_object63829:head# 2024-09-13T11:24:39.875 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.876 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:42e1b56c:::benchmark_data_smithi177_31116_object9754:head# 2024-09-13T11:24:39.876 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.878 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:42e369d4:::benchmark_data_smithi177_31116_object104637:head# 2024-09-13T11:24:39.878 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.880 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:42e40a83:::benchmark_data_smithi177_31116_object69434:head# 2024-09-13T11:24:39.880 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.881 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:42e49e32:::benchmark_data_smithi177_31116_object37172:head# 2024-09-13T11:24:39.881 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.881 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:42e4b5ed:::benchmark_data_smithi177_31116_object70729:head# 2024-09-13T11:24:39.881 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.882 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:42e4e644:::benchmark_data_smithi177_31116_object79479:head# 2024-09-13T11:24:39.882 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.883 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:42e4e98a:::benchmark_data_smithi177_31116_object79377:head# 2024-09-13T11:24:39.883 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.884 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:42e53077:::benchmark_data_smithi177_31116_object105970:head# 2024-09-13T11:24:39.884 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.885 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:42e556ef:::benchmark_data_smithi177_31116_object19957:head# 2024-09-13T11:24:39.885 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.885 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:42e5ce54:::benchmark_data_smithi177_31116_object19365:head# 2024-09-13T11:24:39.885 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.886 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:42e5d717:::benchmark_data_smithi177_31116_object18891:head# 2024-09-13T11:24:39.886 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.887 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:42e7846e:::benchmark_data_smithi177_31116_object31353:head# 2024-09-13T11:24:39.887 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.887 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:42e90fb4:::benchmark_data_smithi177_31116_object92087:head# 2024-09-13T11:24:39.888 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.888 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:42e971cc:::benchmark_data_smithi177_31116_object63189:head# 2024-09-13T11:24:39.888 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.889 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:42ea1f01:::benchmark_data_smithi177_31116_object71448:head# 2024-09-13T11:24:39.889 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.889 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:42ebd60c:::benchmark_data_smithi177_31116_object94560:head# 2024-09-13T11:24:39.890 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.892 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:42ed65ee:::benchmark_data_smithi177_31116_object9768:head# 2024-09-13T11:24:39.892 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.893 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:42edbe2d:::benchmark_data_smithi177_31116_object76716:head# 2024-09-13T11:24:39.893 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.895 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:42edc13b:::benchmark_data_smithi177_31116_object84129:head# 2024-09-13T11:24:39.895 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.898 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:42ee3562:::benchmark_data_smithi177_31116_object92748:head# 2024-09-13T11:24:39.898 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.901 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:42ee3bb5:::benchmark_data_smithi177_31116_object22752:head# 2024-09-13T11:24:39.901 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.902 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:42ef18f9:::benchmark_data_smithi177_31116_object108325:head# 2024-09-13T11:24:39.902 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.905 INFO:tasks.ceph.osd.13.smithi092.stderr:2024-09-13T11:24:39.902+0000 7f3e5afea640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26350) UID: 0 2024-09-13T11:24:39.907 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:42ef24f2:::benchmark_data_smithi177_31116_object73424:head# 2024-09-13T11:24:39.907 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.910 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:42ef6b74:::benchmark_data_smithi177_31116_object31714:head# 2024-09-13T11:24:39.910 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.911 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:42f094c7:::benchmark_data_smithi177_31116_object92577:head# 2024-09-13T11:24:39.911 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.913 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:42f10f66:::benchmark_data_smithi177_31116_object80167:head# 2024-09-13T11:24:39.913 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.913 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:42f196c0:::benchmark_data_smithi177_31116_object104006:head# 2024-09-13T11:24:39.913 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.914 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:42f2768f:::benchmark_data_smithi177_31116_object4991:head# 2024-09-13T11:24:39.914 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.915 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:42f2ff2f:::benchmark_data_smithi177_31116_object40718:head# 2024-09-13T11:24:39.915 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.916 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:42f3716f:::benchmark_data_smithi177_31116_object82541:head# 2024-09-13T11:24:39.916 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.918 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:42f37745:::benchmark_data_smithi177_31116_object47891:head# 2024-09-13T11:24:39.919 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.924 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:42f3dcfd:::benchmark_data_smithi177_31116_object59667:head# 2024-09-13T11:24:39.924 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.925 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:42f4fa49:::benchmark_data_smithi177_31116_object80120:head# 2024-09-13T11:24:39.925 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.925 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:42f53bb0:::benchmark_data_smithi177_31116_object33191:head# 2024-09-13T11:24:39.925 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.925 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:42f5ca38:::benchmark_data_smithi177_31116_object3542:head# 2024-09-13T11:24:39.926 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.927 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:42f5d502:::benchmark_data_smithi177_31116_object74103:head# 2024-09-13T11:24:39.928 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.928 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:42f623c3:::benchmark_data_smithi177_31116_object36116:head# 2024-09-13T11:24:39.928 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.933 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:42f6aac8:::benchmark_data_smithi177_31116_object47499:head# 2024-09-13T11:24:39.933 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.936 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:42f6cc9d:::benchmark_data_smithi177_31116_object76598:head# 2024-09-13T11:24:39.936 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.938 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:42f7655e:::benchmark_data_smithi177_31116_object47915:head# 2024-09-13T11:24:39.939 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.942 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:42f76a63:::benchmark_data_smithi177_31116_object53210:head# 2024-09-13T11:24:39.942 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.943 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:42f7eec1:::benchmark_data_smithi177_31116_object36841:head# 2024-09-13T11:24:39.943 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.945 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:42f83055:::benchmark_data_smithi177_31116_object36604:head# 2024-09-13T11:24:39.946 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.946 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:42f85eec:::benchmark_data_smithi177_31116_object58565:head# 2024-09-13T11:24:39.946 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.948 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:42f86704:::benchmark_data_smithi177_31116_object43063:head# 2024-09-13T11:24:39.948 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.948 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:42f8aa49:::benchmark_data_smithi177_31116_object5367:head# 2024-09-13T11:24:39.948 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.948 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:42f8b5b3:::benchmark_data_smithi177_31116_object20867:head# 2024-09-13T11:24:39.949 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.951 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:42f994b1:::benchmark_data_smithi177_31116_object15231:head# 2024-09-13T11:24:39.951 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.951 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:42f997d0:::benchmark_data_smithi177_31116_object55397:head# 2024-09-13T11:24:39.951 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.951 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:42fa8cf3:::benchmark_data_smithi177_31116_object72631:head# 2024-09-13T11:24:39.951 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.952 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:42fb3744:::benchmark_data_smithi177_31116_object22232:head# 2024-09-13T11:24:39.952 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.952 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:42fbcf8a:::benchmark_data_smithi177_31116_object91884:head# 2024-09-13T11:24:39.952 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.953 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:42fc62a3:::benchmark_data_smithi177_31116_object74552:head# 2024-09-13T11:24:39.953 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.955 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:42fefe9b:::benchmark_data_smithi177_31116_object28146:head# 2024-09-13T11:24:39.955 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.955 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:42ffa92b:::benchmark_data_smithi177_31116_object105737:head# 2024-09-13T11:24:39.955 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.956 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:430036e9:::benchmark_data_smithi177_31116_object60201:head# 2024-09-13T11:24:39.957 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.957 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:43004fea:::benchmark_data_smithi177_31116_object37575:head# 2024-09-13T11:24:39.957 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.958 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4300708b:::benchmark_data_smithi177_31116_object28433:head# 2024-09-13T11:24:39.958 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.959 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:43008547:::benchmark_data_smithi177_31116_object63725:head# 2024-09-13T11:24:39.959 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.959 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:43011291:::benchmark_data_smithi177_31116_object18849:head# 2024-09-13T11:24:39.959 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.960 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:43012179:::benchmark_data_smithi177_31116_object88910:head# 2024-09-13T11:24:39.960 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.961 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:43021fb7:::benchmark_data_smithi177_31116_object93951:head# 2024-09-13T11:24:39.961 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.962 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4302270e:::benchmark_data_smithi177_31116_object14290:head# 2024-09-13T11:24:39.962 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.962 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:43032328:::benchmark_data_smithi177_31116_object2099:head# 2024-09-13T11:24:39.962 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.962 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:43034929:::benchmark_data_smithi177_31116_object34551:head# 2024-09-13T11:24:39.962 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.963 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4305839c:::benchmark_data_smithi177_31116_object105724:head# 2024-09-13T11:24:39.963 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.964 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4305def3:::benchmark_data_smithi177_31116_object100750:head# 2024-09-13T11:24:39.964 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.965 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4307b1c9:::benchmark_data_smithi177_31116_object93321:head# 2024-09-13T11:24:39.965 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.965 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4307fad4:::benchmark_data_smithi177_31116_object94017:head# 2024-09-13T11:24:39.965 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.966 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4309d188:::benchmark_data_smithi177_31116_object27155:head# 2024-09-13T11:24:39.966 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.968 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:430a4a53:::benchmark_data_smithi177_31116_object22801:head# 2024-09-13T11:24:39.969 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.969 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:430b88d7:::benchmark_data_smithi177_31116_object39048:head# 2024-09-13T11:24:39.970 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.970 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:430c7cf2:::benchmark_data_smithi177_31116_object68477:head# 2024-09-13T11:24:39.971 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.973 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:430d7a12:::benchmark_data_smithi177_31116_object96572:head# 2024-09-13T11:24:39.973 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.974 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:430ece2a:::benchmark_data_smithi177_31116_object75168:head# 2024-09-13T11:24:39.974 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.974 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:430f68cd:::benchmark_data_smithi177_31116_object51903:head# 2024-09-13T11:24:39.974 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.977 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:43106425:::benchmark_data_smithi177_31116_object66867:head# 2024-09-13T11:24:39.977 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.978 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4311244f:::benchmark_data_smithi177_31116_object46847:head# 2024-09-13T11:24:39.978 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.979 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:43121685:::benchmark_data_smithi177_31116_object94676:head# 2024-09-13T11:24:39.980 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.980 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:43128283:::benchmark_data_smithi177_31116_object39313:head# 2024-09-13T11:24:39.980 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.983 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:431314ab:::benchmark_data_smithi177_31116_object87462:head# 2024-09-13T11:24:39.983 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.984 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4314ed28:::benchmark_data_smithi177_31116_object54355:head# 2024-09-13T11:24:39.985 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.985 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:43158bc7:::benchmark_data_smithi177_31116_object85185:head# 2024-09-13T11:24:39.985 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.987 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:43158c8c:::benchmark_data_smithi177_31116_object35392:head# 2024-09-13T11:24:39.988 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.988 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:431596eb:::benchmark_data_smithi177_31116_object38714:head# 2024-09-13T11:24:39.988 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.988 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4316ef98:::benchmark_data_smithi177_31116_object23071:head# 2024-09-13T11:24:39.988 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.988 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:43171e7f:::benchmark_data_smithi177_31116_object108594:head# 2024-09-13T11:24:39.989 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.989 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:43178492:::benchmark_data_smithi177_31116_object824:head# 2024-09-13T11:24:39.989 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.990 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:431846cf:::benchmark_data_smithi177_31116_object17112:head# 2024-09-13T11:24:39.990 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.991 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:43198907:::benchmark_data_smithi177_31116_object51459:head# 2024-09-13T11:24:39.991 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.993 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:431a62d8:::benchmark_data_smithi177_31116_object26990:head# 2024-09-13T11:24:39.993 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.993 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:431c27b1:::benchmark_data_smithi177_31116_object11962:head# 2024-09-13T11:24:39.993 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.993 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:431c4cab:::benchmark_data_smithi177_31116_object32586:head# 2024-09-13T11:24:39.993 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.994 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:431cbe65:::benchmark_data_smithi177_31116_object86063:head# 2024-09-13T11:24:39.994 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.996 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:431d81f7:::benchmark_data_smithi177_31116_object70230:head# 2024-09-13T11:24:39.996 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.997 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:431e6c3f:::benchmark_data_smithi177_31116_object62167:head# 2024-09-13T11:24:39.997 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:39.998 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:431e8740:::benchmark_data_smithi177_31116_object32758:head# 2024-09-13T11:24:39.998 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.001 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:431ef103:::benchmark_data_smithi177_31116_object72467:head# 2024-09-13T11:24:40.001 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.002 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:43203930:::benchmark_data_smithi177_31116_object60419:head# 2024-09-13T11:24:40.002 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.002 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4320c89b:::benchmark_data_smithi177_31116_object77792:head# 2024-09-13T11:24:40.002 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.003 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4320fa07:::benchmark_data_smithi177_31116_object92727:head# 2024-09-13T11:24:40.003 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.004 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:43221ee2:::benchmark_data_smithi177_31116_object91172:head# 2024-09-13T11:24:40.004 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.005 INFO:tasks.ceph.osd.13.smithi092.stderr:2024-09-13T11:24:39.998+0000 7f3e5afea640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26350) UID: 0 2024-09-13T11:24:40.005 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4323045c:::benchmark_data_smithi177_31116_object4582:head# 2024-09-13T11:24:40.005 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.006 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4323969f:::benchmark_data_smithi177_31116_object53506:head# 2024-09-13T11:24:40.006 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.007 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:43250fc3:::benchmark_data_smithi177_31116_object56570:head# 2024-09-13T11:24:40.007 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.007 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:43256736:::benchmark_data_smithi177_31116_object41353:head# 2024-09-13T11:24:40.007 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.008 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:432585ea:::benchmark_data_smithi177_31116_object41469:head# 2024-09-13T11:24:40.008 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.008 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4325bd7d:::benchmark_data_smithi177_31116_object57379:head# 2024-09-13T11:24:40.009 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.012 INFO:tasks.radosbench.radosbench.0.smithi177.stdout: 133 16 122286 122270 57.4514 61.75 0.00633143 0.0174 2024-09-13T11:24:40.012 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4325e03c:::benchmark_data_smithi177_31116_object82276:head# 2024-09-13T11:24:40.012 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.013 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:43268ec1:::benchmark_data_smithi177_31116_object936:head# 2024-09-13T11:24:40.013 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.014 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:43274b31:::benchmark_data_smithi177_31116_object9381:head# 2024-09-13T11:24:40.014 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.015 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4327ec4b:::benchmark_data_smithi177_31116_object38924:head# 2024-09-13T11:24:40.015 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.018 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:43293193:::benchmark_data_smithi177_31116_object107558:head# 2024-09-13T11:24:40.018 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.019 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:43294bbb:::benchmark_data_smithi177_31116_object5659:head# 2024-09-13T11:24:40.019 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.021 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:432b38aa:::benchmark_data_smithi177_31116_object33829:head# 2024-09-13T11:24:40.021 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.023 INFO:teuthology.orchestra.run.smithi092.stdout:ERROR: (22) Invalid argument 2024-09-13T11:24:40.023 INFO:teuthology.orchestra.run.smithi092.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:24:40.026 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:432c3c4f:::benchmark_data_smithi177_31116_object8502:head# 2024-09-13T11:24:40.026 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.026 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:432cf2a0:::benchmark_data_smithi177_31116_object23399:head# 2024-09-13T11:24:40.026 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.027 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:432d05b4:::benchmark_data_smithi177_31116_object83186:head# 2024-09-13T11:24:40.027 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.027 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:432d44fa:::benchmark_data_smithi177_31116_object90866:head# 2024-09-13T11:24:40.027 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.028 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:432db8c8:::benchmark_data_smithi177_31116_object9570:head# 2024-09-13T11:24:40.028 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.030 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:432dc3c8:::benchmark_data_smithi177_31116_object24724:head# 2024-09-13T11:24:40.030 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.030 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:432f700b:::benchmark_data_smithi177_31116_object58617:head# 2024-09-13T11:24:40.030 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.031 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:43310c67:::benchmark_data_smithi177_31116_object33731:head# 2024-09-13T11:24:40.031 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.031 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4331c1e2:::benchmark_data_smithi177_31116_object91328:head# 2024-09-13T11:24:40.032 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.032 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4332510c:::benchmark_data_smithi177_31116_object45621:head# 2024-09-13T11:24:40.032 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.036 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4332f723:::benchmark_data_smithi177_31116_object72070:head# 2024-09-13T11:24:40.036 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.041 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:43342ae7:::benchmark_data_smithi177_31116_object6179:head# 2024-09-13T11:24:40.041 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.045 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4334ebb7:::benchmark_data_smithi177_31116_object37851:head# 2024-09-13T11:24:40.045 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.047 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-13T11:24:40.048 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:43371ae5:::benchmark_data_smithi177_31116_object33622:head# 2024-09-13T11:24:40.048 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.050 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4338d445:::benchmark_data_smithi177_31116_object46285:head# 2024-09-13T11:24:40.050 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.053 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:43394c8b:::benchmark_data_smithi177_31116_object49774:head# 2024-09-13T11:24:40.053 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.056 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4339b550:::benchmark_data_smithi177_31116_object1244:head# 2024-09-13T11:24:40.056 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.056 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4339c71c:::benchmark_data_smithi177_31116_object97809:head# 2024-09-13T11:24:40.057 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.057 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:433aa750:::benchmark_data_smithi177_31116_object70992:head# 2024-09-13T11:24:40.057 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.058 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:433b5f04:::benchmark_data_smithi177_31116_object26476:head# 2024-09-13T11:24:40.058 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.058 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:433be3d6:::benchmark_data_smithi177_31116_object88873:head# 2024-09-13T11:24:40.058 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.059 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:433e8c47:::benchmark_data_smithi177_31116_object13544:head# 2024-09-13T11:24:40.059 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.059 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:433f0684:::benchmark_data_smithi177_31116_object53295:head# 2024-09-13T11:24:40.060 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.060 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:433f5220:::benchmark_data_smithi177_31116_object71679:head# 2024-09-13T11:24:40.060 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.061 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:433fde72:::benchmark_data_smithi177_31116_object34601:head# 2024-09-13T11:24:40.061 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.061 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:434063ee:::benchmark_data_smithi177_31116_object72323:head# 2024-09-13T11:24:40.061 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.062 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:43409335:::benchmark_data_smithi177_31116_object21739:head# 2024-09-13T11:24:40.062 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.063 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4340b5a2:::benchmark_data_smithi177_31116_object104660:head# 2024-09-13T11:24:40.063 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.064 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4340bf15:::benchmark_data_smithi177_31116_object5283:head# 2024-09-13T11:24:40.064 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.065 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4340c8e8:::benchmark_data_smithi177_31116_object77912:head# 2024-09-13T11:24:40.065 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.066 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:434166c0:::benchmark_data_smithi177_31116_object71474:head# 2024-09-13T11:24:40.066 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.066 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4341f8d7:::benchmark_data_smithi177_31116_object15165:head# 2024-09-13T11:24:40.066 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.067 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:43428a82:::benchmark_data_smithi177_31116_object20018:head# 2024-09-13T11:24:40.067 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.067 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:434359e2:::benchmark_data_smithi177_31116_object65619:head# 2024-09-13T11:24:40.067 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.068 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:43439d13:::benchmark_data_smithi177_31116_object68284:head# 2024-09-13T11:24:40.068 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.069 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4343bfcd:::benchmark_data_smithi177_31116_object80330:head# 2024-09-13T11:24:40.069 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.070 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4343e435:::benchmark_data_smithi177_31116_object101381:head# 2024-09-13T11:24:40.070 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.071 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4344468d:::benchmark_data_smithi177_31116_object93736:head# 2024-09-13T11:24:40.071 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.072 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4344e251:::benchmark_data_smithi177_31116_object13232:head# 2024-09-13T11:24:40.072 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.072 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:434630c3:::benchmark_data_smithi177_31116_object73789:head# 2024-09-13T11:24:40.072 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.075 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:43466497:::benchmark_data_smithi177_31116_object72203:head# 2024-09-13T11:24:40.075 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.076 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:43469264:::benchmark_data_smithi177_31116_object99865:head# 2024-09-13T11:24:40.076 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.076 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:43479640:::benchmark_data_smithi177_31116_object8858:head# 2024-09-13T11:24:40.076 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.078 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4347cb43:::benchmark_data_smithi177_31116_object49738:head# 2024-09-13T11:24:40.078 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.081 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:43482af6:::benchmark_data_smithi177_31116_object70352:head# 2024-09-13T11:24:40.081 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.085 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:43485327:::benchmark_data_smithi177_31116_object85265:head# 2024-09-13T11:24:40.086 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.087 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4348b3f7:::benchmark_data_smithi177_31116_object52142:head# 2024-09-13T11:24:40.087 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.087 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:43494cfe:::benchmark_data_smithi177_31116_object96610:head# 2024-09-13T11:24:40.087 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.088 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:43499e2a:::benchmark_data_smithi177_31116_object78545:head# 2024-09-13T11:24:40.088 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.090 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:434a30fc:::benchmark_data_smithi177_31116_object35067:head# 2024-09-13T11:24:40.090 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.091 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:434a4458:::benchmark_data_smithi177_31116_object97686:head# 2024-09-13T11:24:40.091 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.091 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:434a6889:::benchmark_data_smithi177_31116_object57144:head# 2024-09-13T11:24:40.091 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.094 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:434a6de0:::benchmark_data_smithi177_31116_object59116:head# 2024-09-13T11:24:40.094 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.099 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:434ac47b:::benchmark_data_smithi177_31116_object8021:head# 2024-09-13T11:24:40.099 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.103 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:434b7c7b:::benchmark_data_smithi177_31116_object67539:head# 2024-09-13T11:24:40.103 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.105 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:434b84ae:::benchmark_data_smithi177_31116_object86163:head# 2024-09-13T11:24:40.105 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.106 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-09-13T11:24:40.100+0000 7f97befc9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26315) UID: 0 2024-09-13T11:24:40.108 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:434bc89f:::benchmark_data_smithi177_31116_object70632:head# 2024-09-13T11:24:40.108 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.111 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:434bf54d:::benchmark_data_smithi177_31116_object7102:head# 2024-09-13T11:24:40.111 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.112 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:434d158a:::benchmark_data_smithi177_31116_object109091:head# 2024-09-13T11:24:40.112 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.114 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:434d9d7c:::benchmark_data_smithi177_31116_object30424:head# 2024-09-13T11:24:40.114 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.114 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:434e8463:::benchmark_data_smithi177_31116_object108341:head# 2024-09-13T11:24:40.114 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.116 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:434f5eaa:::benchmark_data_smithi177_31116_object16071:head# 2024-09-13T11:24:40.116 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.117 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:434fc2e0:::benchmark_data_smithi177_31116_object16738:head# 2024-09-13T11:24:40.117 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.121 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4350d029:::benchmark_data_smithi177_31116_object75941:head# 2024-09-13T11:24:40.122 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.122 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4350ea1a:::benchmark_data_smithi177_31116_object99148:head# 2024-09-13T11:24:40.123 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.123 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:435178db:::benchmark_data_smithi177_31116_object101992:head# 2024-09-13T11:24:40.123 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.123 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4351f3cf:::benchmark_data_smithi177_31116_object93593:head# 2024-09-13T11:24:40.124 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.124 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:435362d9:::benchmark_data_smithi177_31116_object102665:head# 2024-09-13T11:24:40.124 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.124 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4353aa83:::benchmark_data_smithi177_31116_object100003:head# 2024-09-13T11:24:40.125 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.125 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4353b781:::benchmark_data_smithi177_31116_object81053:head# 2024-09-13T11:24:40.125 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.126 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:43574008:::benchmark_data_smithi177_31116_object12605:head# 2024-09-13T11:24:40.126 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.130 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:435982ce:::benchmark_data_smithi177_31116_object65663:head# 2024-09-13T11:24:40.130 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.130 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4359f3aa:::benchmark_data_smithi177_31116_object58953:head# 2024-09-13T11:24:40.130 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.131 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:435b4d48:::benchmark_data_smithi177_31116_object97715:head# 2024-09-13T11:24:40.131 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.131 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:435b90fa:::benchmark_data_smithi177_31116_object86207:head# 2024-09-13T11:24:40.131 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.131 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:435c396c:::benchmark_data_smithi177_31116_object48823:head# 2024-09-13T11:24:40.131 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.131 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:435cf91e:::benchmark_data_smithi177_31116_object59120:head# 2024-09-13T11:24:40.131 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.131 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:435d24b6:::benchmark_data_smithi177_31116_object32154:head# 2024-09-13T11:24:40.131 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.132 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:435d89e9:::benchmark_data_smithi177_31116_object7864:head# 2024-09-13T11:24:40.133 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.133 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:435dab7e:::benchmark_data_smithi177_31116_object101404:head# 2024-09-13T11:24:40.133 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.133 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:435e0763:::benchmark_data_smithi177_31116_object110247:head# 2024-09-13T11:24:40.133 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.133 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:435ea0b7:::benchmark_data_smithi177_31116_object89537:head# 2024-09-13T11:24:40.134 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.135 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:435f51cc:::benchmark_data_smithi177_31116_object52504:head# 2024-09-13T11:24:40.135 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.136 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:435f74f5:::benchmark_data_smithi177_31116_object108828:head# 2024-09-13T11:24:40.136 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.140 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:435fbe24:::benchmark_data_smithi177_31116_object2473:head# 2024-09-13T11:24:40.140 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.140 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4360072b:::benchmark_data_smithi177_31116_object98345:head# 2024-09-13T11:24:40.140 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.142 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:43601390:::benchmark_data_smithi177_31116_object90734:head# 2024-09-13T11:24:40.143 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.143 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:43602106:::benchmark_data_smithi177_31116_object13944:head# 2024-09-13T11:24:40.143 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.143 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:43607d68:::benchmark_data_smithi177_31116_object3384:head# 2024-09-13T11:24:40.144 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.144 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4360df48:::benchmark_data_smithi177_31116_object34842:head# 2024-09-13T11:24:40.144 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.145 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:43614758:::benchmark_data_smithi177_31116_object66235:head# 2024-09-13T11:24:40.146 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.146 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4361ae8d:::benchmark_data_smithi177_31116_object15946:head# 2024-09-13T11:24:40.146 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.148 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4361f7b1:::benchmark_data_smithi177_31116_object13180:head# 2024-09-13T11:24:40.148 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.149 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4362ff83:::benchmark_data_smithi177_31116_object31429:head# 2024-09-13T11:24:40.149 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.149 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:43642120:::benchmark_data_smithi177_31116_object106481:head# 2024-09-13T11:24:40.150 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.153 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:43650941:::benchmark_data_smithi177_31116_object73574:head# 2024-09-13T11:24:40.153 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.156 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:43659226:::benchmark_data_smithi177_31116_object70332:head# 2024-09-13T11:24:40.156 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.156 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:43667820:::benchmark_data_smithi177_31116_object102134:head# 2024-09-13T11:24:40.156 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.159 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4366ea87:::benchmark_data_smithi177_31116_object27557:head# 2024-09-13T11:24:40.159 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.162 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:43671663:::benchmark_data_smithi177_31116_object29167:head# 2024-09-13T11:24:40.162 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.163 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:43671d99:::benchmark_data_smithi177_31116_object43704:head# 2024-09-13T11:24:40.163 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.163 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:43672cca:::benchmark_data_smithi177_31116_object91030:head# 2024-09-13T11:24:40.163 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.164 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4367656f:::benchmark_data_smithi177_31116_object20424:head# 2024-09-13T11:24:40.164 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.167 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:436a4221:::benchmark_data_smithi177_31116_object9061:head# 2024-09-13T11:24:40.167 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.169 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:436b2a54:::benchmark_data_smithi177_31116_object104811:head# 2024-09-13T11:24:40.169 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.170 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:436b65f1:::benchmark_data_smithi177_31116_object36227:head# 2024-09-13T11:24:40.170 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.171 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:436c5fc1:::benchmark_data_smithi177_31116_object62390:head# 2024-09-13T11:24:40.171 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.171 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:436d113e:::benchmark_data_smithi177_31116_object52068:head# 2024-09-13T11:24:40.171 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.171 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:436d5227:::benchmark_data_smithi177_31116_object97216:head# 2024-09-13T11:24:40.172 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.172 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:436d929b:::benchmark_data_smithi177_31116_object17883:head# 2024-09-13T11:24:40.172 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.173 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:436e2500:::benchmark_data_smithi177_31116_object109441:head# 2024-09-13T11:24:40.173 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.173 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:436e5978:::benchmark_data_smithi177_31116_object50368:head# 2024-09-13T11:24:40.174 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.174 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:436f4bd8:::benchmark_data_smithi177_31116_object30038:head# 2024-09-13T11:24:40.175 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.176 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:436f7829:::benchmark_data_smithi177_31116_object91242:head# 2024-09-13T11:24:40.176 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.179 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:43703c0a:::benchmark_data_smithi177_31116_object87319:head# 2024-09-13T11:24:40.179 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.182 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:43725684:::benchmark_data_smithi177_31116_object7475:head# 2024-09-13T11:24:40.182 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.182 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4372e123:::benchmark_data_smithi177_31116_object104842:head# 2024-09-13T11:24:40.182 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.183 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:43732f94:::benchmark_data_smithi177_31116_object46237:head# 2024-09-13T11:24:40.183 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.185 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:43743406:::benchmark_data_smithi177_31116_object51109:head# 2024-09-13T11:24:40.185 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.186 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:43744c2d:::benchmark_data_smithi177_31116_object40566:head# 2024-09-13T11:24:40.186 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.187 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:43749f28:::benchmark_data_smithi177_31116_object63322:head# 2024-09-13T11:24:40.187 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.188 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:43767c05:::benchmark_data_smithi177_31116_object17995:head# 2024-09-13T11:24:40.188 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.188 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4378241d:::benchmark_data_smithi177_31116_object86089:head# 2024-09-13T11:24:40.189 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.192 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:437a7a19:::benchmark_data_smithi177_31116_object56673:head# 2024-09-13T11:24:40.193 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.195 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:437b48b9:::benchmark_data_smithi177_31116_object97846:head# 2024-09-13T11:24:40.195 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.197 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:437b9c91:::benchmark_data_smithi177_31116_object5658:head# 2024-09-13T11:24:40.197 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.203 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:437bb958:::benchmark_data_smithi177_31116_object67463:head# 2024-09-13T11:24:40.203 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.205 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-13T11:24:40.205 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:24:40.206 INFO:tasks.ceph.osd.3.smithi177.stderr:2024-09-13T11:24:40.203+0000 7f10adba4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26574) UID: 0 2024-09-13T11:24:40.208 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:437c8b9b:::benchmark_data_smithi177_31116_object45652:head# 2024-09-13T11:24:40.208 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.209 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:437d1506:::benchmark_data_smithi177_31116_object62056:head# 2024-09-13T11:24:40.209 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.212 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:437d22b9:::benchmark_data_smithi177_31116_object39896:head# 2024-09-13T11:24:40.212 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.213 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:437dd06b:::benchmark_data_smithi177_31116_object22750:head# 2024-09-13T11:24:40.213 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.217 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:437eb422:::benchmark_data_smithi177_31116_object97189:head# 2024-09-13T11:24:40.217 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.218 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:437ee1c6:::benchmark_data_smithi177_31116_object98906:head# 2024-09-13T11:24:40.218 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.218 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:437f1c4f:::benchmark_data_smithi177_31116_object64220:head# 2024-09-13T11:24:40.219 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.219 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:437fe7fb:::benchmark_data_smithi177_31116_object44470:head# 2024-09-13T11:24:40.219 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.220 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:438018b6:::benchmark_data_smithi177_31116_object102549:head# 2024-09-13T11:24:40.220 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.222 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4380650b:::benchmark_data_smithi177_31116_object7623:head# 2024-09-13T11:24:40.222 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.223 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4381dda7:::benchmark_data_smithi177_31116_object65968:head# 2024-09-13T11:24:40.224 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.225 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-13T11:24:40.226 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4383a46b:::benchmark_data_smithi177_31116_object52480:head# 2024-09-13T11:24:40.226 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.227 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:43840349:::benchmark_data_smithi177_31116_object35233:head# 2024-09-13T11:24:40.227 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.228 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:43853459:::benchmark_data_smithi177_31116_object13457:head# 2024-09-13T11:24:40.228 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.229 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4385c169:::benchmark_data_smithi177_31116_object51650:head# 2024-09-13T11:24:40.229 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.230 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4385f551:::benchmark_data_smithi177_31116_object24304:head# 2024-09-13T11:24:40.230 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.231 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:43861257:::benchmark_data_smithi177_31116_object20120:head# 2024-09-13T11:24:40.231 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.233 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4386930f:::benchmark_data_smithi177_31116_object89332:head# 2024-09-13T11:24:40.233 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.239 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:43871f9f:::benchmark_data_smithi177_31116_object6812:head# 2024-09-13T11:24:40.239 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.240 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:43881191:::benchmark_data_smithi177_31116_object18747:head# 2024-09-13T11:24:40.240 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.240 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:43887f49:::benchmark_data_smithi177_31116_object15739:head# 2024-09-13T11:24:40.240 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.243 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4388b472:::benchmark_data_smithi177_31116_object59707:head# 2024-09-13T11:24:40.243 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.243 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4388e209:::benchmark_data_smithi177_31116_object96007:head# 2024-09-13T11:24:40.244 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.244 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4388fe51:::benchmark_data_smithi177_31116_object39873:head# 2024-09-13T11:24:40.244 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.246 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:43896ea2:::benchmark_data_smithi177_31116_object55963:head# 2024-09-13T11:24:40.246 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.246 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:438973a8:::benchmark_data_smithi177_31116_object32486:head# 2024-09-13T11:24:40.246 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.247 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4389fd20:::benchmark_data_smithi177_31116_object72738:head# 2024-09-13T11:24:40.247 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.248 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:438a8697:::benchmark_data_smithi177_31116_object108573:head# 2024-09-13T11:24:40.248 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.249 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:438ac691:::benchmark_data_smithi177_31116_object38962:head# 2024-09-13T11:24:40.249 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.249 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:438ada23:::benchmark_data_smithi177_31116_object1:head# 2024-09-13T11:24:40.249 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.250 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:438b8d19:::benchmark_data_smithi177_31116_object38994:head# 2024-09-13T11:24:40.250 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.250 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:438ba540:::benchmark_data_smithi177_31116_object35929:head# 2024-09-13T11:24:40.251 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.251 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:438dddcf:::benchmark_data_smithi177_31116_object66560:head# 2024-09-13T11:24:40.251 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.252 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:438dfad8:::benchmark_data_smithi177_31116_object4008:head# 2024-09-13T11:24:40.252 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.253 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:438ec54b:::benchmark_data_smithi177_31116_object108903:head# 2024-09-13T11:24:40.253 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.253 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:438f0723:::benchmark_data_smithi177_31116_object94305:head# 2024-09-13T11:24:40.253 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.254 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:438f4594:::benchmark_data_smithi177_31116_object33647:head# 2024-09-13T11:24:40.254 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.255 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:438f94fc:::benchmark_data_smithi177_31116_object107070:head# 2024-09-13T11:24:40.255 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.255 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:438ffbb1:::benchmark_data_smithi177_31116_object101851:head# 2024-09-13T11:24:40.255 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.256 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:439084eb:::benchmark_data_smithi177_31116_object59103:head# 2024-09-13T11:24:40.256 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.257 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4390eee7:::benchmark_data_smithi177_31116_object90595:head# 2024-09-13T11:24:40.257 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.261 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:43915ddf:::benchmark_data_smithi177_31116_object10899:head# 2024-09-13T11:24:40.261 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.266 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4391c046:::benchmark_data_smithi177_31116_object100161:head# 2024-09-13T11:24:40.266 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.267 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:43921c4c:::benchmark_data_smithi177_31116_object24036:head# 2024-09-13T11:24:40.267 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.267 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:43922688:::benchmark_data_smithi177_31116_object17775:head# 2024-09-13T11:24:40.267 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.269 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:43923e9e:::benchmark_data_smithi177_31116_object99885:head# 2024-09-13T11:24:40.269 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.273 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4392716c:::benchmark_data_smithi177_31116_object86558:head# 2024-09-13T11:24:40.273 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.273 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4392c86b:::benchmark_data_smithi177_31116_object85466:head# 2024-09-13T11:24:40.273 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.276 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:43934610:::benchmark_data_smithi177_31116_object52421:head# 2024-09-13T11:24:40.277 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.278 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4394b83a:::benchmark_data_smithi177_31116_object72731:head# 2024-09-13T11:24:40.278 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.280 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4394f21a:::benchmark_data_smithi177_31116_object50822:head# 2024-09-13T11:24:40.280 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.282 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4395698b:::benchmark_data_smithi177_31116_object27397:head# 2024-09-13T11:24:40.282 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.283 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4395766c:::benchmark_data_smithi177_31116_object18538:head# 2024-09-13T11:24:40.283 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.283 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4396221a:::benchmark_data_smithi177_31116_object106959:head# 2024-09-13T11:24:40.283 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.284 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4396aa6d:::benchmark_data_smithi177_31116_object15570:head# 2024-09-13T11:24:40.284 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.285 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:43971e15:::benchmark_data_smithi177_31116_object105354:head# 2024-09-13T11:24:40.285 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.285 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:43990bce:::benchmark_data_smithi177_31116_object98250:head# 2024-09-13T11:24:40.285 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.286 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:439a00a5:::benchmark_data_smithi177_31116_object59038:head# 2024-09-13T11:24:40.286 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.286 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:439ab391:::benchmark_data_smithi177_31116_object41147:head# 2024-09-13T11:24:40.286 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.288 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:439ce266:::benchmark_data_smithi177_31116_object28555:head# 2024-09-13T11:24:40.288 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.290 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:439e520f:::benchmark_data_smithi177_31116_object68942:head# 2024-09-13T11:24:40.290 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.291 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:439f8aaf:::benchmark_data_smithi177_31116_object15140:head# 2024-09-13T11:24:40.291 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.291 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:439f9c74:::benchmark_data_smithi177_31116_object38702:head# 2024-09-13T11:24:40.291 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.293 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:43a07f9e:::benchmark_data_smithi177_31116_object77133:head# 2024-09-13T11:24:40.293 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.295 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:43a165d3:::benchmark_data_smithi177_31116_object27150:head# 2024-09-13T11:24:40.295 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.295 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:43a22b33:::benchmark_data_smithi177_31116_object91179:head# 2024-09-13T11:24:40.295 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.296 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:43a2beee:::benchmark_data_smithi177_31116_object9042:head# 2024-09-13T11:24:40.296 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.296 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:43a305d6:::benchmark_data_smithi177_31116_object53782:head# 2024-09-13T11:24:40.296 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.299 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:43a374b6:::benchmark_data_smithi177_31116_object105791:head# 2024-09-13T11:24:40.299 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.300 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:43a555e9:::benchmark_data_smithi177_31116_object92137:head# 2024-09-13T11:24:40.300 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.300 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:43a5872d:::benchmark_data_smithi177_31116_object74997:head# 2024-09-13T11:24:40.300 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.301 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:43a60577:::benchmark_data_smithi177_31116_object53989:head# 2024-09-13T11:24:40.301 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.302 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:43a60865:::benchmark_data_smithi177_31116_object79671:head# 2024-09-13T11:24:40.302 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.302 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:43a662a3:::benchmark_data_smithi177_31116_object30901:head# 2024-09-13T11:24:40.302 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.303 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:43a6a01a:::benchmark_data_smithi177_31116_object108829:head# 2024-09-13T11:24:40.303 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.304 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:43a6bf6b:::benchmark_data_smithi177_31116_object3872:head# 2024-09-13T11:24:40.304 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.305 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:43a71a5e:::benchmark_data_smithi177_31116_object40092:head# 2024-09-13T11:24:40.305 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.305 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:43a7317f:::benchmark_data_smithi177_31116_object40079:head# 2024-09-13T11:24:40.306 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.306 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:43a7c76c:::benchmark_data_smithi177_31116_object41820:head# 2024-09-13T11:24:40.306 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.307 INFO:tasks.ceph.osd.1.smithi092.stderr:2024-09-13T11:24:40.302+0000 7fe7789c5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33718) UID: 0 2024-09-13T11:24:40.307 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:43a81c8c:::benchmark_data_smithi177_31116_object108845:head# 2024-09-13T11:24:40.307 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.308 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:43a88fdd:::benchmark_data_smithi177_31116_object2169:head# 2024-09-13T11:24:40.308 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.309 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:43a8eda1:::benchmark_data_smithi177_31116_object34683:head# 2024-09-13T11:24:40.309 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.312 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:43a9a1d9:::benchmark_data_smithi177_31116_object89735:head# 2024-09-13T11:24:40.312 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.313 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:43ab7c37:::benchmark_data_smithi177_31116_object99472:head# 2024-09-13T11:24:40.313 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.313 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:43ab97a4:::benchmark_data_smithi177_31116_object64118:head# 2024-09-13T11:24:40.313 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.314 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:43ac59b9:::benchmark_data_smithi177_31116_object57830:head# 2024-09-13T11:24:40.314 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.314 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:43acaf57:::benchmark_data_smithi177_31116_object108277:head# 2024-09-13T11:24:40.314 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.316 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:43acf54b:::benchmark_data_smithi177_31116_object79828:head# 2024-09-13T11:24:40.316 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.317 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:43ad10d0:::benchmark_data_smithi177_31116_object33546:head# 2024-09-13T11:24:40.317 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.317 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:43ad1963:::benchmark_data_smithi177_31116_object45262:head# 2024-09-13T11:24:40.317 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.318 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:43ad6af7:::benchmark_data_smithi177_31116_object50858:head# 2024-09-13T11:24:40.318 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.322 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:43ae1be0:::benchmark_data_smithi177_31116_object101206:head# 2024-09-13T11:24:40.322 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.323 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:43ae24e7:::benchmark_data_smithi177_31116_object35137:head# 2024-09-13T11:24:40.323 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.323 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:43af4de5:::benchmark_data_smithi177_31116_object76207:head# 2024-09-13T11:24:40.323 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.324 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:43af9fa7:::benchmark_data_smithi177_31116_object80406:head# 2024-09-13T11:24:40.324 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.325 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-13T11:24:40.326 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:43b05f2d:::benchmark_data_smithi177_31116_object21140:head# 2024-09-13T11:24:40.326 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.327 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:43b0ab59:::benchmark_data_smithi177_31116_object3322:head# 2024-09-13T11:24:40.327 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.333 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:43b0b2bc:::benchmark_data_smithi177_31116_object103099:head# 2024-09-13T11:24:40.333 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.333 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:43b0b38b:::benchmark_data_smithi177_31116_object68578:head# 2024-09-13T11:24:40.333 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.335 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:43b18170:::benchmark_data_smithi177_31116_object24730:head# 2024-09-13T11:24:40.335 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.339 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:43b1bd47:::benchmark_data_smithi177_31116_object68373:head# 2024-09-13T11:24:40.339 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.341 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:43b39ebf:::benchmark_data_smithi177_31116_object9088:head# 2024-09-13T11:24:40.341 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.345 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:43b44739:::benchmark_data_smithi177_31116_object60684:head# 2024-09-13T11:24:40.345 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.347 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:43b53323:::benchmark_data_smithi177_31116_object26229:head# 2024-09-13T11:24:40.347 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.348 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:43b578b8:::benchmark_data_smithi177_31116_object37320:head# 2024-09-13T11:24:40.348 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.348 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:43b579fe:::benchmark_data_smithi177_31116_object85291:head# 2024-09-13T11:24:40.349 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.349 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:43b5a163:::benchmark_data_smithi177_31116_object108408:head# 2024-09-13T11:24:40.349 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.352 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:43b5f242:::benchmark_data_smithi177_31116_object44711:head# 2024-09-13T11:24:40.352 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.354 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:43b62495:::benchmark_data_smithi177_31116_object53031:head# 2024-09-13T11:24:40.354 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.358 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:43b7c8fe:::benchmark_data_smithi177_31116_object66853:head# 2024-09-13T11:24:40.358 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.359 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:43b82953:::benchmark_data_smithi177_31116_object26959:head# 2024-09-13T11:24:40.359 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.359 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:43b97330:::benchmark_data_smithi177_31116_object80350:head# 2024-09-13T11:24:40.360 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.360 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:43ba11b6:::benchmark_data_smithi177_31116_object1475:head# 2024-09-13T11:24:40.360 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.362 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:43baab7d:::benchmark_data_smithi177_31116_object74578:head# 2024-09-13T11:24:40.363 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.363 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:43bac3c6:::benchmark_data_smithi177_31116_object70033:head# 2024-09-13T11:24:40.363 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.363 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:43bbd963:::benchmark_data_smithi177_31116_object96863:head# 2024-09-13T11:24:40.363 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.365 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:43bc03c9:::benchmark_data_smithi177_31116_object3718:head# 2024-09-13T11:24:40.365 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.366 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:43bc1d5c:::benchmark_data_smithi177_31116_object1072:head# 2024-09-13T11:24:40.366 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.367 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:43bc2234:::benchmark_data_smithi177_31116_object81145:head# 2024-09-13T11:24:40.367 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.367 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:43bcad49:::benchmark_data_smithi177_31116_object65296:head# 2024-09-13T11:24:40.367 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.368 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:43bcd1c3:::benchmark_data_smithi177_31116_object18190:head# 2024-09-13T11:24:40.368 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.368 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:43bd42e7:::benchmark_data_smithi177_31116_object103456:head# 2024-09-13T11:24:40.368 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.370 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:43bd7231:::benchmark_data_smithi177_31116_object31068:head# 2024-09-13T11:24:40.370 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.372 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:43be6e36:::benchmark_data_smithi177_31116_object88034:head# 2024-09-13T11:24:40.372 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.375 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:43c07ae3:::benchmark_data_smithi177_31116_object25368:head# 2024-09-13T11:24:40.375 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.378 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:43c11835:::benchmark_data_smithi177_31116_object91847:head# 2024-09-13T11:24:40.378 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.381 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:43c12527:::benchmark_data_smithi177_31116_object100311:head# 2024-09-13T11:24:40.381 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.382 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:43c33125:::benchmark_data_smithi177_31116_object16352:head# 2024-09-13T11:24:40.382 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.383 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-13T11:24:40.384 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:24:40.384 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:43c387b7:::benchmark_data_smithi177_31116_object10334:head# 2024-09-13T11:24:40.384 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.385 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:43c3c4df:::benchmark_data_smithi177_31116_object30788:head# 2024-09-13T11:24:40.385 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.386 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:43c3e052:::benchmark_data_smithi177_31116_object13238:head# 2024-09-13T11:24:40.386 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.388 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:43c3fff5:::benchmark_data_smithi177_31116_object50379:head# 2024-09-13T11:24:40.388 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.393 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:43c4f4e3:::benchmark_data_smithi177_31116_object62519:head# 2024-09-13T11:24:40.393 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.394 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:43c50776:::benchmark_data_smithi177_31116_object65358:head# 2024-09-13T11:24:40.394 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.397 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:43c55ca4:::benchmark_data_smithi177_31116_object104004:head# 2024-09-13T11:24:40.397 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.398 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:43c5a517:::benchmark_data_smithi177_31116_object57904:head# 2024-09-13T11:24:40.398 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.399 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:43c6b698:::benchmark_data_smithi177_31116_object3371:head# 2024-09-13T11:24:40.399 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.399 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:43c71aaa:::benchmark_data_smithi177_31116_object63962:head# 2024-09-13T11:24:40.399 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.400 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:43c73c36:::benchmark_data_smithi177_31116_object31734:head# 2024-09-13T11:24:40.400 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.402 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:43c7483c:::benchmark_data_smithi177_31116_object29712:head# 2024-09-13T11:24:40.402 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.403 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:43c90919:::benchmark_data_smithi177_31116_object60435:head# 2024-09-13T11:24:40.403 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.407 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-09-13T11:24:40.404+0000 7f97befc9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26315) UID: 0 2024-09-13T11:24:40.407 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-13T11:24:40.409 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:43cbc345:::benchmark_data_smithi177_31116_object78488:head# 2024-09-13T11:24:40.409 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.410 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:43cc7b32:::benchmark_data_smithi177_31116_object41231:head# 2024-09-13T11:24:40.410 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.412 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:43ccd630:::benchmark_data_smithi177_31116_object97462:head# 2024-09-13T11:24:40.412 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.412 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:43cd1520:::benchmark_data_smithi177_31116_object92364:head# 2024-09-13T11:24:40.412 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.413 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:43cd6382:::benchmark_data_smithi177_31116_object72073:head# 2024-09-13T11:24:40.413 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.415 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:43cd8505:::benchmark_data_smithi177_31116_object104276:head# 2024-09-13T11:24:40.415 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.416 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:43ce36e8:::benchmark_data_smithi177_31116_object66410:head# 2024-09-13T11:24:40.416 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.417 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:43ceb2f4:::benchmark_data_smithi177_31116_object67889:head# 2024-09-13T11:24:40.417 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.417 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:43cec608:::benchmark_data_smithi177_31116_object96993:head# 2024-09-13T11:24:40.417 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.418 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:43d07740:::benchmark_data_smithi177_31116_object25070:head# 2024-09-13T11:24:40.418 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.418 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:43d0d1b6:::benchmark_data_smithi177_31116_object21133:head# 2024-09-13T11:24:40.418 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.420 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:43d1432c:::benchmark_data_smithi177_31116_object15372:head# 2024-09-13T11:24:40.420 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.421 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:43d16408:::benchmark_data_smithi177_31116_object33597:head# 2024-09-13T11:24:40.421 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.421 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:43d25993:::benchmark_data_smithi177_31116_object70346:head# 2024-09-13T11:24:40.422 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.422 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:43d36171:::benchmark_data_smithi177_31116_object99907:head# 2024-09-13T11:24:40.422 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.424 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:43d400d6:::benchmark_data_smithi177_31116_object872:head# 2024-09-13T11:24:40.424 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.426 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:43d4456e:::benchmark_data_smithi177_31116_object10450:head# 2024-09-13T11:24:40.426 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.427 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:43d452e5:::benchmark_data_smithi177_31116_object21558:head# 2024-09-13T11:24:40.427 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.427 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:43d57d46:::benchmark_data_smithi177_31116_object48290:head# 2024-09-13T11:24:40.428 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.428 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:43d5ce35:::benchmark_data_smithi177_31116_object106392:head# 2024-09-13T11:24:40.428 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.429 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:43d67776:::benchmark_data_smithi177_31116_object5434:head# 2024-09-13T11:24:40.429 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.429 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:43d913e2:::benchmark_data_smithi177_31116_object7097:head# 2024-09-13T11:24:40.430 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.431 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:43d9e73f:::benchmark_data_smithi177_31116_object16875:head# 2024-09-13T11:24:40.431 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.431 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:43da9a88:::benchmark_data_smithi177_31116_object57204:head# 2024-09-13T11:24:40.431 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.432 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:43db7b5e:::benchmark_data_smithi177_31116_object43530:head# 2024-09-13T11:24:40.432 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.433 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:43dce2cc:::benchmark_data_smithi177_31116_object364:head# 2024-09-13T11:24:40.433 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.433 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:43dd8909:::benchmark_data_smithi177_31116_object44174:head# 2024-09-13T11:24:40.433 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.434 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:43dda1fc:::benchmark_data_smithi177_31116_object49909:head# 2024-09-13T11:24:40.434 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.434 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:43e029b2:::benchmark_data_smithi177_31116_object3317:head# 2024-09-13T11:24:40.435 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.436 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:43e07b29:::benchmark_data_smithi177_31116_object61162:head# 2024-09-13T11:24:40.436 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.437 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:43e0840f:::benchmark_data_smithi177_31116_object58707:head# 2024-09-13T11:24:40.437 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.441 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:43e1d565:::benchmark_data_smithi177_31116_object43324:head# 2024-09-13T11:24:40.441 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.442 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:43e29b75:::benchmark_data_smithi177_31116_object9830:head# 2024-09-13T11:24:40.442 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.442 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:43e3dadc:::benchmark_data_smithi177_31116_object29732:head# 2024-09-13T11:24:40.442 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.442 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:43e43de8:::benchmark_data_smithi177_31116_object21333:head# 2024-09-13T11:24:40.442 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.444 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:43e4676b:::benchmark_data_smithi177_31116_object30368:head# 2024-09-13T11:24:40.444 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.445 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:43e5a788:::benchmark_data_smithi177_31116_object31539:head# 2024-09-13T11:24:40.445 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.448 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:43e8a242:::benchmark_data_smithi177_31116_object37597:head# 2024-09-13T11:24:40.448 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.450 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:43e8fda4:::benchmark_data_smithi177_31116_object10082:head# 2024-09-13T11:24:40.450 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.454 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:43e917be:::benchmark_data_smithi177_31116_object63897:head# 2024-09-13T11:24:40.454 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.454 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:43e95a20:::benchmark_data_smithi177_31116_object55871:head# 2024-09-13T11:24:40.454 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.455 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:43eaaeaf:::benchmark_data_smithi177_31116_object1160:head# 2024-09-13T11:24:40.455 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.459 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:43eb63d0:::benchmark_data_smithi177_31116_object23373:head# 2024-09-13T11:24:40.459 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.459 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:43ebb63a:::benchmark_data_smithi177_31116_object46931:head# 2024-09-13T11:24:40.460 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.460 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:43ecf23f:::benchmark_data_smithi177_31116_object88383:head# 2024-09-13T11:24:40.460 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.461 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:43ed0c1f:::benchmark_data_smithi177_31116_object9407:head# 2024-09-13T11:24:40.461 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.462 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:43edea67:::benchmark_data_smithi177_31116_object102331:head# 2024-09-13T11:24:40.462 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.462 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:43ee016a:::benchmark_data_smithi177_31116_object23923:head# 2024-09-13T11:24:40.462 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.463 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:43ee052d:::benchmark_data_smithi177_31116_object27182:head# 2024-09-13T11:24:40.463 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.463 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:43ee1041:::benchmark_data_smithi177_31116_object53717:head# 2024-09-13T11:24:40.463 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.464 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:43ee8d95:::benchmark_data_smithi177_31116_object30773:head# 2024-09-13T11:24:40.464 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.464 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:43efc1a0:::benchmark_data_smithi177_31116_object60481:head# 2024-09-13T11:24:40.464 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.465 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:43f00b2e:::benchmark_data_smithi177_31116_object61257:head# 2024-09-13T11:24:40.465 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.465 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:43f24772:::benchmark_data_smithi177_31116_object96513:head# 2024-09-13T11:24:40.465 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.466 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:43f34b7a:::benchmark_data_smithi177_31116_object80634:head# 2024-09-13T11:24:40.466 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.467 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:43f39f1e:::benchmark_data_smithi177_31116_object26583:head# 2024-09-13T11:24:40.467 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.470 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:43f47736:::benchmark_data_smithi177_31116_object74374:head# 2024-09-13T11:24:40.470 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.472 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:43f4cf44:::benchmark_data_smithi177_31116_object47394:head# 2024-09-13T11:24:40.472 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.475 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:43f4f86c:::benchmark_data_smithi177_31116_object35732:head# 2024-09-13T11:24:40.475 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.476 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:43f5203f:::benchmark_data_smithi177_31116_object73878:head# 2024-09-13T11:24:40.476 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.481 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:43f56511:::benchmark_data_smithi177_31116_object107592:head# 2024-09-13T11:24:40.481 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.481 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:43f614c1:::benchmark_data_smithi177_31116_object59142:head# 2024-09-13T11:24:40.481 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.482 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:43f69b50:::benchmark_data_smithi177_31116_object16326:head# 2024-09-13T11:24:40.483 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.486 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:43f6cd29:::benchmark_data_smithi177_31116_object100521:head# 2024-09-13T11:24:40.486 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.486 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:43f6dd59:::benchmark_data_smithi177_31116_object65428:head# 2024-09-13T11:24:40.486 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.487 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:43f93a22:::benchmark_data_smithi177_31116_object107316:head# 2024-09-13T11:24:40.487 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.490 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:43fa1ff7:::benchmark_data_smithi177_31116_object86542:head# 2024-09-13T11:24:40.490 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.491 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:43fa4029:::benchmark_data_smithi177_31116_object7918:head# 2024-09-13T11:24:40.491 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.494 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:43fb4edc:::benchmark_data_smithi177_31116_object10727:head# 2024-09-13T11:24:40.494 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.495 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:43fbaffc:::benchmark_data_smithi177_31116_object10325:head# 2024-09-13T11:24:40.495 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.498 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:43fc1b43:::benchmark_data_smithi177_31116_object92984:head# 2024-09-13T11:24:40.498 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.500 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:43fc7361:::benchmark_data_smithi177_31116_object34971:head# 2024-09-13T11:24:40.501 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.503 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:43fd8bdf:::benchmark_data_smithi177_31116_object20642:head# 2024-09-13T11:24:40.503 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.507 INFO:tasks.ceph.osd.12.smithi001.stderr:2024-09-13T11:24:40.503+0000 7f892c6e6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26314) UID: 0 2024-09-13T11:24:40.509 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:43fd95df:::benchmark_data_smithi177_31116_object52270:head# 2024-09-13T11:24:40.509 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.511 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:43fe3402:::benchmark_data_smithi177_31116_object42393:head# 2024-09-13T11:24:40.511 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.512 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:43fe38e6:::benchmark_data_smithi177_31116_object17525:head# 2024-09-13T11:24:40.512 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.512 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:43fe52c3:::benchmark_data_smithi177_31116_object25715:head# 2024-09-13T11:24:40.513 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.513 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:43ff4406:::benchmark_data_smithi177_31116_object8748:head# 2024-09-13T11:24:40.513 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.514 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:43ffb701:::benchmark_data_smithi177_31116_object1464:head# 2024-09-13T11:24:40.514 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.514 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:43ffce4a:::benchmark_data_smithi177_31116_object63955:head# 2024-09-13T11:24:40.514 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.515 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:43fff576:::benchmark_data_smithi177_31116_object18997:head# 2024-09-13T11:24:40.515 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:24:40.516 INFO:teuthology.orchestra.run.smithi162.stdout:write_pg epoch 119 info 3.2( v 119'3947 (0'0,119'3947] local-lis/les=75/76 n=1731 ec=30/30 lis/c=75/75 les/c/f=76/76/0 sis=75) 2024-09-13T11:24:40.531 INFO:teuthology.orchestra.run.smithi162.stdout:Import successful 2024-09-13T11:24:40.562 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-13T11:24:40.562 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:24:40.585 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-13T11:24:40.609 INFO:tasks.ceph.osd.3.smithi177.stderr:2024-09-13T11:24:40.607+0000 7f10adba4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26574) UID: 0 2024-09-13T11:24:40.709 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-09-13T11:24:40.708+0000 7f97befc9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26315) UID: 0 2024-09-13T11:24:40.764 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-13T11:24:40.764 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:24:40.783 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-13T11:24:40.810 INFO:tasks.ceph.osd.11.smithi177.stderr:2024-09-13T11:24:40.807+0000 7f24792c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26580) UID: 0 2024-09-13T11:24:40.911 INFO:tasks.ceph.osd.4.smithi001.stderr:2024-09-13T11:24:40.911+0000 7ff8e0218640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26310) UID: 0 2024-09-13T11:24:40.974 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-13T11:24:40.974 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:24:40.993 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-13T11:24:41.012 INFO:tasks.radosbench.radosbench.0.smithi177.stdout: 134 16 123646 123630 57.657 85 0.0108743 0.0173367 2024-09-13T11:24:41.012 INFO:tasks.ceph.osd.7.smithi177.stderr:2024-09-13T11:24:41.011+0000 7f897b51d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26575) UID: 0 2024-09-13T11:24:41.080 DEBUG:teuthology.orchestra.run.smithi162:> sudo rm -f /var/log/ceph/exp.3.2.14 2024-09-13T11:24:41.112 INFO:tasks.thrashosds.thrasher:checking if target objectstore is bluestore on osd.14 2024-09-13T11:24:41.112 DEBUG:teuthology.orchestra.run.smithi162:> sudo ceph-bluestore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/bluestore_tool.$pid.log' --log-level=10 --path /var/lib/ceph/osd/ceph-14 show-label 2024-09-13T11:24:41.113 INFO:tasks.ceph.osd.12.smithi001.stderr:2024-09-13T11:24:41.111+0000 7f892c6e6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26314) UID: 0 2024-09-13T11:24:41.137 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-13T11:24:41.137 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:24:41.158 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-09-13T11:24:41.169 INFO:teuthology.orchestra.run.smithi162.stdout:inferring bluefs devices from bluestore path 2024-09-13T11:24:41.215 INFO:tasks.ceph.osd.13.smithi092.stderr:2024-09-13T11:24:41.210+0000 7f3e5afea640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26350) UID: 0 2024-09-13T11:24:41.296 INFO:teuthology.orchestra.run.smithi001.stderr:nodeep-scrub is set 2024-09-13T11:24:41.302 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-13T11:24:41.302 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:24:41.317 INFO:tasks.ceph.osd.5.smithi092.stderr:2024-09-13T11:24:41.310+0000 7f6f0b705640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26346) UID: 0 2024-09-13T11:24:41.321 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-09-13T11:24:41.417 INFO:tasks.ceph.osd.13.smithi092.stderr:2024-09-13T11:24:41.410+0000 7f3e5afea640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26350) UID: 0 2024-09-13T11:24:41.464 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-13T11:24:41.464 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:24:41.485 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-09-13T11:24:41.486 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-13T11:24:41.486 INFO:teuthology.orchestra.run.smithi162.stdout: "/var/lib/ceph/osd/ceph-14/block": { 2024-09-13T11:24:41.486 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_uuid": "fc3e2499-0c2a-414d-a25e-2628876ae630", 2024-09-13T11:24:41.486 INFO:teuthology.orchestra.run.smithi162.stdout: "size": 96636764160, 2024-09-13T11:24:41.487 INFO:teuthology.orchestra.run.smithi162.stdout: "btime": "2024-09-13T11:21:29.617770+0000", 2024-09-13T11:24:41.487 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "main", 2024-09-13T11:24:41.487 INFO:teuthology.orchestra.run.smithi162.stdout: "bfm_blocks": "23592960", 2024-09-13T11:24:41.487 INFO:teuthology.orchestra.run.smithi162.stdout: "bfm_blocks_per_key": "128", 2024-09-13T11:24:41.487 INFO:teuthology.orchestra.run.smithi162.stdout: "bfm_bytes_per_block": "4096", 2024-09-13T11:24:41.487 INFO:teuthology.orchestra.run.smithi162.stdout: "bfm_size": "96636764160", 2024-09-13T11:24:41.487 INFO:teuthology.orchestra.run.smithi162.stdout: "bluefs": "1", 2024-09-13T11:24:41.487 INFO:teuthology.orchestra.run.smithi162.stdout: "ceph_fsid": "2fd68ac7-5581-4d90-80a4-30bcf5252a7a", 2024-09-13T11:24:41.487 INFO:teuthology.orchestra.run.smithi162.stdout: "ceph_version_when_created": "ceph version 19.3.0-4697-gd58f7306 (d58f73069a8157a9ace12a0247ee3e8497707ea9) squid (dev)", 2024-09-13T11:24:41.487 INFO:teuthology.orchestra.run.smithi162.stdout: "created_at": "2024-09-13T11:21:32.219631Z", 2024-09-13T11:24:41.487 INFO:teuthology.orchestra.run.smithi162.stdout: "elastic_shared_blobs": "1", 2024-09-13T11:24:41.487 INFO:teuthology.orchestra.run.smithi162.stdout: "epoch": "16", 2024-09-13T11:24:41.487 INFO:teuthology.orchestra.run.smithi162.stdout: "kv_backend": "rocksdb", 2024-09-13T11:24:41.488 INFO:teuthology.orchestra.run.smithi162.stdout: "magic": "ceph osd volume v026", 2024-09-13T11:24:41.488 INFO:teuthology.orchestra.run.smithi162.stdout: "multi": "yes", 2024-09-13T11:24:41.488 INFO:teuthology.orchestra.run.smithi162.stdout: "ready": "ready", 2024-09-13T11:24:41.488 INFO:teuthology.orchestra.run.smithi162.stdout: "require_osd_release": "19", 2024-09-13T11:24:41.488 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "bluestore", 2024-09-13T11:24:41.488 INFO:teuthology.orchestra.run.smithi162.stdout: "whoami": "14" 2024-09-13T11:24:41.488 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:41.488 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-13T11:24:41.490 INFO:tasks.thrashosds.thrasher:checking if target bluestore supports sharding on osd.14 2024-09-13T11:24:41.490 DEBUG:teuthology.orchestra.run.smithi162:> sudo ceph-bluestore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/bluestore_tool.$pid.log' --log-level=10 --path /var/lib/ceph/osd/ceph-14 show-sharding 2024-09-13T11:24:41.518 INFO:tasks.ceph.osd.15.smithi177.stderr:2024-09-13T11:24:41.515+0000 7f9d6ce77640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26578) UID: 0 2024-09-13T11:24:41.552 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T11:24:41.619 INFO:tasks.ceph.osd.2.smithi162.stderr:2024-09-13T11:24:41.616+0000 7fe92b35a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26313) UID: 0 2024-09-13T11:24:41.630 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-13T11:24:41.630 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:24:41.650 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-13T11:24:41.720 INFO:tasks.ceph.osd.4.smithi001.stderr:2024-09-13T11:24:41.719+0000 7ff8e0218640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26310) UID: 0 2024-09-13T11:24:41.729 INFO:teuthology.orchestra.run.smithi001.stdout:osd.0: {} 2024-09-13T11:24:41.729 INFO:teuthology.orchestra.run.smithi001.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T11:24:41.757 INFO:teuthology.orchestra.run.smithi001.stdout:osd.1: {} 2024-09-13T11:24:41.757 INFO:teuthology.orchestra.run.smithi001.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T11:24:41.785 INFO:teuthology.orchestra.run.smithi001.stdout:osd.2: {} 2024-09-13T11:24:41.785 INFO:teuthology.orchestra.run.smithi001.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T11:24:41.816 INFO:teuthology.orchestra.run.smithi001.stdout:osd.3: {} 2024-09-13T11:24:41.817 INFO:teuthology.orchestra.run.smithi001.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T11:24:41.821 INFO:tasks.ceph.osd.12.smithi001.stderr:2024-09-13T11:24:41.819+0000 7f892c6e6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26314) UID: 0 2024-09-13T11:24:41.834 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-09-13T11:24:41.834 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [ 2024-09-13T11:24:41.834 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:41.834 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:124574 3.2d 3:b53c1b35:::benchmark_data_smithi177_31116_object124573:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e130)", 2024-09-13T11:24:41.834 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:41.823068+0000", 2024-09-13T11:24:41.834 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 0.010404253, 2024-09-13T11:24:41.834 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.010436104999999999, 2024-09-13T11:24:41.834 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:41.835 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:41.835 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "waiting for sub ops", 2024-09-13T11:24:41.835 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:24:41.835 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:24:41.835 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:41.835 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 124574 2024-09-13T11:24:41.835 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:41.835 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:41.835 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:41.835 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:41.835 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:41.823068+0000", 2024-09-13T11:24:41.835 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:41.835 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:41.835 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:41.836 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:41.836 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:41.823068+0000", 2024-09-13T11:24:41.836 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:41.836 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:41.836 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:41.836 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:41.836 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:41.823070+0000", 2024-09-13T11:24:41.836 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.3800000000000001e-06 2024-09-13T11:24:41.836 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:41.836 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:41.836 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:41.836 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:41.823097+0000", 2024-09-13T11:24:41.836 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.622e-05 2024-09-13T11:24:41.837 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:41.837 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:41.837 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:41.837 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:41.823108+0000", 2024-09-13T11:24:41.837 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.1967000000000001e-05 2024-09-13T11:24:41.837 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:41.837 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:41.837 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:41.837 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:41.823116+0000", 2024-09-13T11:24:41.837 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.3100000000000003e-06 2024-09-13T11:24:41.837 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:41.837 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:41.837 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:41.837 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:41.825875+0000", 2024-09-13T11:24:41.838 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0027587750000000002 2024-09-13T11:24:41.838 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:41.838 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:41.838 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:24:41.838 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:41.826060+0000", 2024-09-13T11:24:41.838 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00018552 2024-09-13T11:24:41.838 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:41.838 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:41.838 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 0,5", 2024-09-13T11:24:41.838 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:41.826263+0000", 2024-09-13T11:24:41.838 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00020312300000000001 2024-09-13T11:24:41.838 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:41.838 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:41.838 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:24:41.838 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:41.827898+0000", 2024-09-13T11:24:41.839 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0016345789999999999 2024-09-13T11:24:41.839 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:41.839 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:41.839 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:41.839 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:41.828803+0000", 2024-09-13T11:24:41.839 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00090513799999999999 2024-09-13T11:24:41.839 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:41.839 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:41.839 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:41.839 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:41.840 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:41.840 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPush(3.15 130/122 [PushOp(3:a972c466:::benchmark_data_smithi177_31116_object28037:head, version: 59'1683, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:a972c466:::benchmark_data_smithi177_31116_object28037:head@59'1683, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:24:41.840 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:41.831314+0000", 2024-09-13T11:24:41.840 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 0.0021585889999999998, 2024-09-13T11:24:41.840 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0022979760000000002, 2024-09-13T11:24:41.840 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:41.840 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:41.840 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "queued for pg", 2024-09-13T11:24:41.841 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:41.841 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:41.841 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:41.841 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:41.831314+0000", 2024-09-13T11:24:41.841 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:41.841 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:41.841 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:41.841 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:41.841 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:41.831314+0000", 2024-09-13T11:24:41.841 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:41.841 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:41.841 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:41.841 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:41.841 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:41.831315+0000", 2024-09-13T11:24:41.842 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.598e-06 2024-09-13T11:24:41.842 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:41.842 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:41.842 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:41.842 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:41.831397+0000", 2024-09-13T11:24:41.842 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.195e-05 2024-09-13T11:24:41.842 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:41.842 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:41.842 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:41.842 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:41.831443+0000", 2024-09-13T11:24:41.842 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.5856999999999998e-05 2024-09-13T11:24:41.842 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:41.842 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:41.842 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:41.843 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:41.831459+0000", 2024-09-13T11:24:41.843 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.6334e-05 2024-09-13T11:24:41.843 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:41.843 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:41.843 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:41.843 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:41.843 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:41.843 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop_reply(client.4665.0:124574 3.2d e130/120)", 2024-09-13T11:24:41.843 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:41.831399+0000", 2024-09-13T11:24:41.843 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 0.0020727229999999998, 2024-09-13T11:24:41.843 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0022518859999999998, 2024-09-13T11:24:41.843 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:41.843 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:41.844 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "queued for pg", 2024-09-13T11:24:41.844 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:41.844 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:41.844 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:41.844 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:41.831399+0000", 2024-09-13T11:24:41.844 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:41.844 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:41.844 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:41.844 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:41.844 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:41.831399+0000", 2024-09-13T11:24:41.844 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:41.845 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:41.845 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:41.845 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:41.845 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:41.831400+0000", 2024-09-13T11:24:41.845 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.13e-07 2024-09-13T11:24:41.845 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:41.845 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:41.845 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:41.845 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:41.831403+0000", 2024-09-13T11:24:41.845 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.8420000000000001e-06 2024-09-13T11:24:41.845 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:41.845 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:41.845 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:41.845 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:41.831490+0000", 2024-09-13T11:24:41.846 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.6991999999999997e-05 2024-09-13T11:24:41.846 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:41.846 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:41.846 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:41.846 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:41.831495+0000", 2024-09-13T11:24:41.846 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.2909999999999998e-06 2024-09-13T11:24:41.846 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:41.846 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:41.846 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:41.846 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:41.846 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:41.846 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPushReply(3.14 130/122 [PushReplyOp(3:28770ffe:::benchmark_data_smithi177_31116_object22387:head)])", 2024-09-13T11:24:41.846 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:41.832047+0000", 2024-09-13T11:24:41.846 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 0.0014248760000000001, 2024-09-13T11:24:41.847 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001648109, 2024-09-13T11:24:41.847 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:41.847 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:41.847 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-13T11:24:41.847 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:41.847 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:41.847 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:41.847 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:41.832047+0000", 2024-09-13T11:24:41.847 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:41.847 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:41.847 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:41.847 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:41.847 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:41.832047+0000", 2024-09-13T11:24:41.847 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:41.848 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:41.848 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:41.848 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:41.848 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:41.832049+0000", 2024-09-13T11:24:41.848 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.486e-06 2024-09-13T11:24:41.848 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:41.848 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:41.848 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:41.848 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:41.832055+0000", 2024-09-13T11:24:41.848 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.5549999999999999e-06 2024-09-13T11:24:41.848 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:41.848 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:41.848 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:41.849 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:41.832115+0000", 2024-09-13T11:24:41.849 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.9441000000000002e-05 2024-09-13T11:24:41.849 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:41.849 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:41.849 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:41.849 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:41.832123+0000", 2024-09-13T11:24:41.849 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.6410000000000008e-06 2024-09-13T11:24:41.849 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:41.849 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:41.849 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:41.849 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:41.832181+0000", 2024-09-13T11:24:41.849 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.8017999999999999e-05 2024-09-13T11:24:41.849 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:41.849 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:41.849 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:41.850 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:41.850 INFO:teuthology.orchestra.run.smithi177.stdout: ], 2024-09-13T11:24:41.850 INFO:teuthology.orchestra.run.smithi177.stdout: "num_ops": 4 2024-09-13T11:24:41.850 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-09-13T11:24:41.850 INFO:teuthology.orchestra.run.smithi001.stdout:osd.4: {} 2024-09-13T11:24:41.851 INFO:teuthology.orchestra.run.smithi001.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T11:24:41.862 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-13T11:24:41.873 INFO:teuthology.orchestra.run.smithi001.stdout:osd.5: {} 2024-09-13T11:24:41.873 INFO:teuthology.orchestra.run.smithi001.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T11:24:41.900 INFO:teuthology.orchestra.run.smithi001.stdout:osd.6: {} 2024-09-13T11:24:41.901 INFO:teuthology.orchestra.run.smithi001.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T11:24:41.922 INFO:tasks.ceph.osd.11.smithi177.stderr:2024-09-13T11:24:41.919+0000 7f24792c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26580) UID: 0 2024-09-13T11:24:41.930 INFO:teuthology.orchestra.run.smithi001.stdout:osd.7: {} 2024-09-13T11:24:41.930 INFO:teuthology.orchestra.run.smithi001.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T11:24:41.953 INFO:teuthology.orchestra.run.smithi001.stdout:osd.8: {} 2024-09-13T11:24:41.954 INFO:teuthology.orchestra.run.smithi001.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-13T11:24:41.981 INFO:teuthology.orchestra.run.smithi001.stdout:osd.9: {} 2024-09-13T11:24:41.981 INFO:teuthology.orchestra.run.smithi001.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-13T11:24:42.007 INFO:teuthology.orchestra.run.smithi001.stdout:osd.10: {} 2024-09-13T11:24:42.007 INFO:teuthology.orchestra.run.smithi001.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-13T11:24:42.012 INFO:tasks.radosbench.radosbench.0.smithi177.stdout: 135 16 124788 124772 57.7585 71.375 0.0159538 0.0173078 2024-09-13T11:24:42.022 INFO:tasks.ceph.osd.10.smithi162.stderr:2024-09-13T11:24:42.020+0000 7fb59c9f0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26317) UID: 0 2024-09-13T11:24:42.033 INFO:teuthology.orchestra.run.smithi001.stdout:osd.11: {} 2024-09-13T11:24:42.034 INFO:teuthology.orchestra.run.smithi001.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-13T11:24:42.048 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-09-13T11:24:42.048 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [], 2024-09-13T11:24:42.048 INFO:teuthology.orchestra.run.smithi177.stdout: "complaint_time": 30, 2024-09-13T11:24:42.048 INFO:teuthology.orchestra.run.smithi177.stdout: "num_blocked_ops": 0 2024-09-13T11:24:42.048 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-09-13T11:24:42.056 INFO:teuthology.orchestra.run.smithi001.stdout:osd.12: {} 2024-09-13T11:24:42.056 INFO:teuthology.orchestra.run.smithi001.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-13T11:24:42.070 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-13T11:24:42.083 INFO:teuthology.orchestra.run.smithi001.stdout:osd.13: {} 2024-09-13T11:24:42.083 INFO:teuthology.orchestra.run.smithi001.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-13T11:24:42.083 INFO:teuthology.orchestra.run.smithi001.stderr:Error ENXIO: problem getting command descriptions from osd.14 2024-09-13T11:24:42.112 INFO:teuthology.orchestra.run.smithi001.stdout:osd.15: {} 2024-09-13T11:24:42.112 INFO:teuthology.orchestra.run.smithi001.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-13T11:24:42.115 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-13T11:24:42.116 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-13T11:24:42.123 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-09-13T11:24:42.120+0000 7f97befc9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26315) UID: 0 2024-09-13T11:24:42.224 INFO:tasks.ceph.osd.4.smithi001.stderr:2024-09-13T11:24:42.223+0000 7ff8e0218640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26310) UID: 0 2024-09-13T11:24:42.286 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-09-13T11:24:42.286 INFO:teuthology.orchestra.run.smithi177.stdout: "size": 20, 2024-09-13T11:24:42.286 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 600, 2024-09-13T11:24:42.286 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [ 2024-09-13T11:24:42.286 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:42.286 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:30852 3.a 3:522fb0d3:::benchmark_data_smithi177_31116_object30851:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e59)", 2024-09-13T11:24:42.286 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:09.538712+0000", 2024-09-13T11:24:42.286 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 92.744682294, 2024-09-13T11:24:42.286 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.19661263300000001, 2024-09-13T11:24:42.286 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:42.287 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:42.287 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:42.287 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:24:42.287 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:24:42.287 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:42.287 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 30852 2024-09-13T11:24:42.287 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:42.287 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:42.287 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:42.287 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:42.287 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.538712+0000", 2024-09-13T11:24:42.287 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:42.287 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:42.287 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:42.288 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:42.288 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.538712+0000", 2024-09-13T11:24:42.288 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:42.288 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:42.288 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:42.288 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:42.288 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.538714+0000", 2024-09-13T11:24:42.288 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.658e-06 2024-09-13T11:24:42.288 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:42.288 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:42.288 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:42.288 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.538735+0000", 2024-09-13T11:24:42.288 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.141e-05 2024-09-13T11:24:42.289 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:42.289 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:42.289 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:42.289 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.538747+0000", 2024-09-13T11:24:42.289 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.2258e-05 2024-09-13T11:24:42.289 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:42.289 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:42.289 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:42.289 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.538754+0000", 2024-09-13T11:24:42.289 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.2859999999999996e-06 2024-09-13T11:24:42.289 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:42.289 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:42.289 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:42.289 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.538798+0000", 2024-09-13T11:24:42.290 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.4258000000000002e-05 2024-09-13T11:24:42.290 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:42.290 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:42.290 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:24:42.290 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.538890+0000", 2024-09-13T11:24:42.290 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.1676999999999997e-05 2024-09-13T11:24:42.290 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:42.290 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:42.290 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 7", 2024-09-13T11:24:42.290 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.538991+0000", 2024-09-13T11:24:42.290 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00010173700000000001 2024-09-13T11:24:42.290 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:42.290 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:42.290 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:24:42.291 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.540171+0000", 2024-09-13T11:24:42.291 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0011800479999999999 2024-09-13T11:24:42.291 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:42.291 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:42.291 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:42.291 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.735031+0000", 2024-09-13T11:24:42.291 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.194859798 2024-09-13T11:24:42.291 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:42.291 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:42.291 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:24:42.291 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.735197+0000", 2024-09-13T11:24:42.291 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00016569499999999999 2024-09-13T11:24:42.291 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:42.291 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:42.292 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:24:42.292 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.735325+0000", 2024-09-13T11:24:42.292 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00012780800000000001 2024-09-13T11:24:42.292 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:42.292 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:42.292 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:42.292 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:42.292 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:42.292 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop_reply(client.4665.0:30852 3.a e59/30 ondisk, result = 0)", 2024-09-13T11:24:42.292 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:09.540458+0000", 2024-09-13T11:24:42.292 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 92.742936806000003, 2024-09-13T11:24:42.292 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.194887, 2024-09-13T11:24:42.293 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:42.293 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:42.293 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:24:42.293 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:42.293 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:42.293 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:42.293 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.540458+0000", 2024-09-13T11:24:42.293 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:42.293 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:42.293 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:42.293 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:42.293 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.540458+0000", 2024-09-13T11:24:42.293 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:42.294 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:42.294 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:42.294 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:42.294 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.540459+0000", 2024-09-13T11:24:42.294 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.2699999999999999e-06 2024-09-13T11:24:42.294 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:42.294 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:42.294 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:42.294 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.540463+0000", 2024-09-13T11:24:42.294 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.8500000000000004e-06 2024-09-13T11:24:42.294 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:42.294 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:42.294 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:42.294 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.734813+0000", 2024-09-13T11:24:42.295 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.194350047 2024-09-13T11:24:42.295 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:42.295 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:42.295 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:42.295 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.734832+0000", 2024-09-13T11:24:42.295 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.9610999999999999e-05 2024-09-13T11:24:42.295 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:42.295 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:42.295 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:42.295 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.734989+0000", 2024-09-13T11:24:42.295 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00015694799999999999 2024-09-13T11:24:42.295 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:42.295 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:42.295 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:24:42.296 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.735015+0000", 2024-09-13T11:24:42.296 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.6165000000000001e-05 2024-09-13T11:24:42.296 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:42.296 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:42.296 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:24:42.296 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.735345+0000", 2024-09-13T11:24:42.296 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00032910900000000001 2024-09-13T11:24:42.296 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:42.296 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:42.297 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:42.297 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:42.297 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:42.297 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPush(3.17 75/70 [PushOp(3:eda94455:::benchmark_data_smithi177_31116_object13017:head, version: 50'788, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:eda94455:::benchmark_data_smithi177_31116_object13017:head@50'788, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:24:42.297 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:31.038029+0000", 2024-09-13T11:24:42.297 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 71.245365741000001, 2024-09-13T11:24:42.297 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.42064623499999998, 2024-09-13T11:24:42.297 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:42.297 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:42.297 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:24:42.297 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:42.297 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:42.297 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:42.298 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.038029+0000", 2024-09-13T11:24:42.298 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:42.298 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:42.298 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:42.298 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:42.298 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.038029+0000", 2024-09-13T11:24:42.298 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:42.298 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:42.298 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:42.298 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:42.298 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.038030+0000", 2024-09-13T11:24:42.298 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.499e-06 2024-09-13T11:24:42.298 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:42.298 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:42.299 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:42.299 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.038088+0000", 2024-09-13T11:24:42.299 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.7771e-05 2024-09-13T11:24:42.299 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:42.299 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:42.299 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:42.299 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.457025+0000", 2024-09-13T11:24:42.299 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.41893754900000002 2024-09-13T11:24:42.299 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:42.299 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:42.299 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:42.299 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.457073+0000", 2024-09-13T11:24:42.299 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.7450999999999997e-05 2024-09-13T11:24:42.299 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:42.300 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:42.300 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:42.300 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.457304+0000", 2024-09-13T11:24:42.300 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000231351 2024-09-13T11:24:42.300 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:42.300 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:42.300 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:24:42.300 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.457331+0000", 2024-09-13T11:24:42.300 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.6372000000000001e-05 2024-09-13T11:24:42.300 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:42.300 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:42.300 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:24:42.300 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.458675+0000", 2024-09-13T11:24:42.301 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001344242 2024-09-13T11:24:42.301 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:42.301 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:42.301 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:42.301 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:42.301 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:42.301 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "rep_scrubmap(3.a e75 from shard 7)", 2024-09-13T11:24:42.301 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:32.151191+0000", 2024-09-13T11:24:42.301 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 70.132203688000004, 2024-09-13T11:24:42.301 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.164608173, 2024-09-13T11:24:42.301 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:42.301 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:42.301 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:24:42.301 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:42.302 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:42.302 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:42.302 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:32.151191+0000", 2024-09-13T11:24:42.302 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:42.302 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:42.302 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:42.302 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:42.302 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:32.151191+0000", 2024-09-13T11:24:42.302 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:42.302 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:42.302 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:42.302 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:42.302 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:32.151192+0000", 2024-09-13T11:24:42.303 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.28e-06 2024-09-13T11:24:42.303 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:42.303 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:42.303 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:42.303 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:32.151220+0000", 2024-09-13T11:24:42.303 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.8045e-05 2024-09-13T11:24:42.303 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:42.303 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:42.303 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:42.303 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:32.315480+0000", 2024-09-13T11:24:42.303 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.16425959000000001 2024-09-13T11:24:42.303 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:42.303 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:42.303 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:42.304 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:32.315488+0000", 2024-09-13T11:24:42.304 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.0819999999999999e-06 2024-09-13T11:24:42.304 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:42.304 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:42.304 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:42.304 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:32.315558+0000", 2024-09-13T11:24:42.304 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.0500999999999995e-05 2024-09-13T11:24:42.304 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:42.304 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:42.304 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:24:42.304 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:32.315570+0000", 2024-09-13T11:24:42.304 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.2238999999999999e-05 2024-09-13T11:24:42.304 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:42.304 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:42.305 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:24:42.305 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:32.315799+0000", 2024-09-13T11:24:42.305 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00022843599999999999 2024-09-13T11:24:42.305 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:42.305 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:42.305 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:42.305 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:42.305 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:42.305 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPush(3.14 79/70 [PushOp(3:2c86f72b:::benchmark_data_smithi177_31116_object32132:head, version: 60'2019, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:2c86f72b:::benchmark_data_smithi177_31116_object32132:head@60'2019, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:24:42.305 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:35.768489+0000", 2024-09-13T11:24:42.305 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 66.514905524, 2024-09-13T11:24:42.305 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.179450581, 2024-09-13T11:24:42.305 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:42.306 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:42.306 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:24:42.306 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:42.306 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:42.306 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:42.306 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:35.768489+0000", 2024-09-13T11:24:42.306 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:42.306 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:42.306 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:42.306 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:42.306 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:35.768489+0000", 2024-09-13T11:24:42.306 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:42.306 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:42.307 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:42.307 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:42.307 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:35.768490+0000", 2024-09-13T11:24:42.307 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5999999999999999e-06 2024-09-13T11:24:42.307 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:42.307 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:42.307 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:42.307 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:35.768592+0000", 2024-09-13T11:24:42.307 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00010194799999999999 2024-09-13T11:24:42.307 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:42.307 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:42.307 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:42.307 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:35.946933+0000", 2024-09-13T11:24:42.308 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.17834060800000001 2024-09-13T11:24:42.308 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:42.308 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:42.308 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:42.308 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:35.946953+0000", 2024-09-13T11:24:42.308 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.0347e-05 2024-09-13T11:24:42.308 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:42.308 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:42.308 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:42.308 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:35.947056+0000", 2024-09-13T11:24:42.308 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000103164 2024-09-13T11:24:42.308 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:42.308 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:42.309 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:24:42.309 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:35.947071+0000", 2024-09-13T11:24:42.309 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.4854e-05 2024-09-13T11:24:42.309 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:42.309 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:42.309 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:24:42.309 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:35.947939+0000", 2024-09-13T11:24:42.309 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00086806000000000003 2024-09-13T11:24:42.309 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:42.309 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:42.309 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:42.309 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:42.309 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:42.309 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPush(3.17 83/70 [PushOp(3:eb574268:::benchmark_data_smithi177_31116_object40622:head, version: 64'2475, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:eb574268:::benchmark_data_smithi177_31116_object40622:head@64'2475, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:24:42.310 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:40.760042+0000", 2024-09-13T11:24:42.310 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 61.523352023999998, 2024-09-13T11:24:42.310 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.30544642100000002, 2024-09-13T11:24:42.310 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:42.310 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:42.310 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:24:42.310 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:42.310 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:42.310 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:42.310 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:40.760042+0000", 2024-09-13T11:24:42.310 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:42.310 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:42.310 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:42.311 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:42.311 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:40.760042+0000", 2024-09-13T11:24:42.311 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:42.311 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:42.311 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:42.311 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:42.311 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:40.760044+0000", 2024-09-13T11:24:42.311 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.511e-06 2024-09-13T11:24:42.311 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:42.311 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:42.311 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:42.311 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:40.760149+0000", 2024-09-13T11:24:42.311 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000105473 2024-09-13T11:24:42.311 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:42.312 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:42.312 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:42.312 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.064465+0000", 2024-09-13T11:24:42.312 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.30431613800000001 2024-09-13T11:24:42.312 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:42.312 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:42.312 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:42.312 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.064479+0000", 2024-09-13T11:24:42.312 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3586e-05 2024-09-13T11:24:42.312 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:42.313 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:42.313 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:42.313 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.064553+0000", 2024-09-13T11:24:42.313 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.3646000000000006e-05 2024-09-13T11:24:42.313 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:42.313 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:42.313 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:24:42.313 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.064568+0000", 2024-09-13T11:24:42.313 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.52e-05 2024-09-13T11:24:42.313 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:42.314 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:42.314 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:24:42.314 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.065489+0000", 2024-09-13T11:24:42.314 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00092086700000000002 2024-09-13T11:24:42.314 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:42.314 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:42.314 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:42.314 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:42.314 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:42.314 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPushReply(3.1a 83/78 [PushReplyOp(3:5c3834bd:::benchmark_data_smithi177_31116_object8669:head)])", 2024-09-13T11:24:42.314 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:40.798236+0000", 2024-09-13T11:24:42.314 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 61.485158593999998, 2024-09-13T11:24:42.314 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.29701520999999997, 2024-09-13T11:24:42.314 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:42.315 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:42.315 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-13T11:24:42.315 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:42.315 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:42.315 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:42.315 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:40.798236+0000", 2024-09-13T11:24:42.315 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:42.315 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:42.315 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:42.315 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:42.315 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:40.798236+0000", 2024-09-13T11:24:42.315 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:42.315 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:42.315 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:42.316 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:42.316 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:40.798237+0000", 2024-09-13T11:24:42.316 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.0500000000000002e-07 2024-09-13T11:24:42.316 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:42.316 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:42.316 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:42.316 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:40.798240+0000", 2024-09-13T11:24:42.316 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.4819999999999999e-06 2024-09-13T11:24:42.316 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:42.316 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:42.316 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:42.316 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.093360+0000", 2024-09-13T11:24:42.316 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.29512009 2024-09-13T11:24:42.317 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:42.317 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:42.317 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:42.317 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.093372+0000", 2024-09-13T11:24:42.317 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.2221e-05 2024-09-13T11:24:42.317 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:42.317 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:42.317 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:42.317 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.093482+0000", 2024-09-13T11:24:42.317 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000109571 2024-09-13T11:24:42.317 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:42.317 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:42.317 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:24:42.317 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.095251+0000", 2024-09-13T11:24:42.318 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001768941 2024-09-13T11:24:42.318 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:42.318 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:42.318 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:42.318 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:42.318 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:42.318 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPushReply(3.1a 83/78 [PushReplyOp(3:58c7ffc0:::benchmark_data_smithi177_31116_object8738:head)])", 2024-09-13T11:24:42.318 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:40.805941+0000", 2024-09-13T11:24:42.318 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 61.477453339999997, 2024-09-13T11:24:42.318 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.30315678499999998, 2024-09-13T11:24:42.318 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:42.318 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:42.318 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-13T11:24:42.319 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:42.319 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:42.319 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:42.319 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:40.805941+0000", 2024-09-13T11:24:42.319 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:42.319 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:42.319 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:42.319 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:42.319 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:40.805941+0000", 2024-09-13T11:24:42.319 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:42.319 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:42.319 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:42.319 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:42.319 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:40.805942+0000", 2024-09-13T11:24:42.320 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3769999999999999e-06 2024-09-13T11:24:42.320 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:42.320 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:42.320 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:42.320 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:40.805947+0000", 2024-09-13T11:24:42.320 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.8980000000000004e-06 2024-09-13T11:24:42.320 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:42.320 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:42.320 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:42.320 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.106505+0000", 2024-09-13T11:24:42.320 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.30055777700000003 2024-09-13T11:24:42.320 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:42.320 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:42.321 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:42.321 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.106512+0000", 2024-09-13T11:24:42.321 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.3749999999999997e-06 2024-09-13T11:24:42.321 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:42.321 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:42.321 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:42.321 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.106566+0000", 2024-09-13T11:24:42.321 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.3103999999999998e-05 2024-09-13T11:24:42.321 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:42.321 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:42.321 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:24:42.321 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.109098+0000", 2024-09-13T11:24:42.321 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0025322539999999998 2024-09-13T11:24:42.321 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:42.322 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:42.322 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:42.322 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:42.322 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:42.322 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPushReply(3.1a 83/78 [PushReplyOp(3:58c7ffc0:::benchmark_data_smithi177_31116_object8738:head)])", 2024-09-13T11:24:42.322 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:40.805973+0000", 2024-09-13T11:24:42.322 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 61.477421337000003, 2024-09-13T11:24:42.322 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.28729672099999998, 2024-09-13T11:24:42.322 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:42.322 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:42.322 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-13T11:24:42.322 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:42.322 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:42.323 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:42.323 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:40.805973+0000", 2024-09-13T11:24:42.323 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:42.323 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:42.323 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:42.323 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:42.323 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:40.805973+0000", 2024-09-13T11:24:42.323 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:42.323 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:42.323 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:42.323 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:42.323 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:40.805974+0000", 2024-09-13T11:24:42.323 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.8599999999999998e-07 2024-09-13T11:24:42.323 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:42.324 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:42.324 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:42.324 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:40.805976+0000", 2024-09-13T11:24:42.324 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.2809999999999998e-06 2024-09-13T11:24:42.324 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:42.324 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:42.324 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:42.324 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.091588+0000", 2024-09-13T11:24:42.324 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.28561173899999998 2024-09-13T11:24:42.324 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:42.324 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:42.324 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:42.324 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.091596+0000", 2024-09-13T11:24:42.325 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.0520000000000002e-06 2024-09-13T11:24:42.325 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:42.325 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:42.325 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:42.325 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.091907+0000", 2024-09-13T11:24:42.325 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00031179599999999998 2024-09-13T11:24:42.325 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:42.325 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:42.325 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:24:42.325 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.093270+0000", 2024-09-13T11:24:42.325 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0013622669999999999 2024-09-13T11:24:42.325 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:42.325 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:42.325 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:42.326 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:42.326 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:42.326 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPushReply(3.1a 84/78 [PushReplyOp(3:581ea97e:::benchmark_data_smithi177_31116_object10255:head)])", 2024-09-13T11:24:42.326 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:41.514228+0000", 2024-09-13T11:24:42.326 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 60.769165848999997, 2024-09-13T11:24:42.326 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.40695564200000001, 2024-09-13T11:24:42.326 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:42.326 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:42.326 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-13T11:24:42.326 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:42.326 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:42.326 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:42.326 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.514228+0000", 2024-09-13T11:24:42.326 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:42.327 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:42.327 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:42.327 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:42.327 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.514228+0000", 2024-09-13T11:24:42.327 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:42.327 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:42.327 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:42.327 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:42.327 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.514230+0000", 2024-09-13T11:24:42.327 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.499e-06 2024-09-13T11:24:42.327 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:42.327 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:42.327 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:42.328 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.514236+0000", 2024-09-13T11:24:42.328 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.5389999999999998e-06 2024-09-13T11:24:42.328 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:42.328 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:42.328 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:42.328 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.919787+0000", 2024-09-13T11:24:42.328 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.40555152999999999 2024-09-13T11:24:42.328 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:42.328 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:42.328 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:42.328 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.919796+0000", 2024-09-13T11:24:42.328 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.4829999999999999e-06 2024-09-13T11:24:42.328 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:42.329 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:42.329 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:42.329 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.920366+0000", 2024-09-13T11:24:42.329 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00057061700000000004 2024-09-13T11:24:42.329 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:42.329 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:42.329 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:24:42.329 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.921184+0000", 2024-09-13T11:24:42.329 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00081797399999999996 2024-09-13T11:24:42.329 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:42.329 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:42.329 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:42.329 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:42.330 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:42.330 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:60312 3.14 3:2e846241:::benchmark_data_smithi177_31116_object60311:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e85)", 2024-09-13T11:24:42.330 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:42.065725+0000", 2024-09-13T11:24:42.330 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 60.217669205999997, 2024-09-13T11:24:42.330 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.211323176, 2024-09-13T11:24:42.330 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:42.330 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:42.330 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:42.330 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:24:42.330 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:24:42.330 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:42.330 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 60312 2024-09-13T11:24:42.330 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:42.330 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:42.331 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:42.331 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:42.331 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.065725+0000", 2024-09-13T11:24:42.331 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:42.331 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:42.331 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:42.331 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:42.331 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.065725+0000", 2024-09-13T11:24:42.331 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:42.331 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:42.331 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:42.331 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:42.331 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.065728+0000", 2024-09-13T11:24:42.332 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.3400000000000002e-06 2024-09-13T11:24:42.332 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:42.332 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:42.332 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:42.332 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.065759+0000", 2024-09-13T11:24:42.332 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.0074e-05 2024-09-13T11:24:42.332 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:42.332 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:42.332 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:42.332 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.065771+0000", 2024-09-13T11:24:42.332 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.2869e-05 2024-09-13T11:24:42.332 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:42.332 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:42.332 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:42.333 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.065780+0000", 2024-09-13T11:24:42.333 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.3489999999999992e-06 2024-09-13T11:24:42.333 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:42.333 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:42.333 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:42.333 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.065837+0000", 2024-09-13T11:24:42.333 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.7444000000000001e-05 2024-09-13T11:24:42.333 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:42.333 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:42.333 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:24:42.333 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.065966+0000", 2024-09-13T11:24:42.333 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000128973 2024-09-13T11:24:42.333 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:42.334 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:42.334 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 14", 2024-09-13T11:24:42.334 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.066114+0000", 2024-09-13T11:24:42.334 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00014788299999999999 2024-09-13T11:24:42.334 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:42.334 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:42.334 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:24:42.334 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.067604+0000", 2024-09-13T11:24:42.334 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001489637 2024-09-13T11:24:42.334 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:42.334 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:42.334 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:42.334 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.276963+0000", 2024-09-13T11:24:42.335 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.20935888499999999 2024-09-13T11:24:42.335 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:42.335 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:42.335 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:24:42.335 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.277003+0000", 2024-09-13T11:24:42.335 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.0241000000000003e-05 2024-09-13T11:24:42.335 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:42.335 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:42.335 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:24:42.335 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.277048+0000", 2024-09-13T11:24:42.335 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.5481e-05 2024-09-13T11:24:42.335 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:42.335 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:42.336 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:42.336 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:42.336 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:42.336 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop_reply(client.4665.0:60312 3.14 e85/84 ondisk, result = 0)", 2024-09-13T11:24:42.336 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:42.069120+0000", 2024-09-13T11:24:42.336 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 60.214274086000003, 2024-09-13T11:24:42.336 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.20804031000000001, 2024-09-13T11:24:42.336 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:42.336 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:42.336 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:24:42.336 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:42.336 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:42.336 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:42.336 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.069120+0000", 2024-09-13T11:24:42.337 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:42.337 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:42.337 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:42.337 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:42.337 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.069120+0000", 2024-09-13T11:24:42.337 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:42.337 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:42.337 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:42.337 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:42.337 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.069123+0000", 2024-09-13T11:24:42.337 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.289e-06 2024-09-13T11:24:42.337 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:42.337 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:42.338 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:42.338 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.069128+0000", 2024-09-13T11:24:42.338 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.3530000000000002e-06 2024-09-13T11:24:42.338 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:42.338 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:42.338 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:42.338 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.276885+0000", 2024-09-13T11:24:42.338 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.207756989 2024-09-13T11:24:42.338 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:42.338 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:42.338 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:42.338 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.276892+0000", 2024-09-13T11:24:42.338 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.922e-06 2024-09-13T11:24:42.338 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:42.339 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:42.339 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:42.339 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.276953+0000", 2024-09-13T11:24:42.339 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.0890000000000001e-05 2024-09-13T11:24:42.339 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:42.339 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:42.339 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:24:42.339 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.276959+0000", 2024-09-13T11:24:42.339 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.9179999999999999e-06 2024-09-13T11:24:42.339 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:42.339 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:42.339 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:24:42.339 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.277161+0000", 2024-09-13T11:24:42.340 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00020194899999999999 2024-09-13T11:24:42.340 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:42.340 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:42.340 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:42.340 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:42.340 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:42.340 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:60427 3.14 3:2afea229:::benchmark_data_smithi177_31116_object60426:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e85)", 2024-09-13T11:24:42.340 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:42.187231+0000", 2024-09-13T11:24:42.340 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 60.096162980000003, 2024-09-13T11:24:42.340 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.83900596999999999, 2024-09-13T11:24:42.340 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:42.340 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:42.340 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:42.341 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:24:42.341 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:24:42.341 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:42.341 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 60427 2024-09-13T11:24:42.341 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:42.341 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:42.341 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:42.341 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:42.341 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.187231+0000", 2024-09-13T11:24:42.341 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:42.341 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:42.341 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:42.341 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:42.342 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.187231+0000", 2024-09-13T11:24:42.342 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:42.342 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:42.342 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:42.342 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:42.342 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.187235+0000", 2024-09-13T11:24:42.342 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.3579999999999999e-06 2024-09-13T11:24:42.342 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:42.342 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:42.342 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:42.342 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.187261+0000", 2024-09-13T11:24:42.342 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.6089e-05 2024-09-13T11:24:42.343 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:42.343 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:42.343 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:42.343 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.187275+0000", 2024-09-13T11:24:42.343 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.4653e-05 2024-09-13T11:24:42.343 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:42.343 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:42.343 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:42.343 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.187282+0000", 2024-09-13T11:24:42.343 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.8870000000000003e-06 2024-09-13T11:24:42.343 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:42.343 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:42.343 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:42.344 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.187336+0000", 2024-09-13T11:24:42.344 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.3875999999999997e-05 2024-09-13T11:24:42.344 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:42.344 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:42.344 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:24:42.344 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.187480+0000", 2024-09-13T11:24:42.344 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00014330899999999999 2024-09-13T11:24:42.344 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:42.344 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:42.344 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 14", 2024-09-13T11:24:42.344 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.187658+0000", 2024-09-13T11:24:42.344 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00017861900000000001 2024-09-13T11:24:42.344 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:42.345 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:42.345 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:24:42.345 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.189353+0000", 2024-09-13T11:24:42.345 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001695144 2024-09-13T11:24:42.345 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:42.345 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:42.345 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:42.345 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:43.026081+0000", 2024-09-13T11:24:42.345 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.83672736199999997 2024-09-13T11:24:42.345 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:42.345 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:42.345 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:24:42.345 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:43.026169+0000", 2024-09-13T11:24:42.345 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.8411000000000004e-05 2024-09-13T11:24:42.346 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:42.346 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:42.346 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:24:42.346 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:43.026237+0000", 2024-09-13T11:24:42.346 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.8262000000000005e-05 2024-09-13T11:24:42.346 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:42.346 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:42.346 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:42.346 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:42.346 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:42.346 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop_reply(client.4665.0:60427 3.14 e85/84 ondisk, result = 0)", 2024-09-13T11:24:42.346 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:42.190914+0000", 2024-09-13T11:24:42.346 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 60.092480316, 2024-09-13T11:24:42.346 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.83534071499999996, 2024-09-13T11:24:42.347 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:42.347 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:42.347 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:24:42.347 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:42.347 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:42.347 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:42.347 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.190914+0000", 2024-09-13T11:24:42.347 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:42.347 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:42.347 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:42.347 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:42.347 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.190914+0000", 2024-09-13T11:24:42.347 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:42.348 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:42.348 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:42.348 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:42.348 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.190916+0000", 2024-09-13T11:24:42.348 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.84e-06 2024-09-13T11:24:42.348 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:42.348 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:42.348 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:42.348 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.190921+0000", 2024-09-13T11:24:42.348 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.2039999999999996e-06 2024-09-13T11:24:42.348 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:42.348 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:42.348 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:42.349 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:43.025926+0000", 2024-09-13T11:24:42.349 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.83500542 2024-09-13T11:24:42.349 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:42.349 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:42.349 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:42.349 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:43.025937+0000", 2024-09-13T11:24:42.349 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0597e-05 2024-09-13T11:24:42.349 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:42.349 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:42.349 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:42.349 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:43.026052+0000", 2024-09-13T11:24:42.349 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000114838 2024-09-13T11:24:42.349 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:42.349 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:42.350 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:24:42.350 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:43.026070+0000", 2024-09-13T11:24:42.350 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.8131e-05 2024-09-13T11:24:42.350 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:42.350 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:42.350 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:24:42.350 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:43.026255+0000", 2024-09-13T11:24:42.350 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000184685 2024-09-13T11:24:42.350 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:42.350 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:42.350 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:42.350 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:42.350 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:42.351 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPushReply(3.1d 89/78 [PushReplyOp(3:bc1b7cfb:::benchmark_data_smithi177_31116_object22629:head)])", 2024-09-13T11:24:42.351 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:46.663106+0000", 2024-09-13T11:24:42.351 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 55.620288133000003, 2024-09-13T11:24:42.351 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.26645533199999999, 2024-09-13T11:24:42.351 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:42.351 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:42.351 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-13T11:24:42.351 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:42.351 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:42.351 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:42.351 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.663106+0000", 2024-09-13T11:24:42.351 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:42.351 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:42.352 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:42.352 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:42.352 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.663106+0000", 2024-09-13T11:24:42.352 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:42.352 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:42.352 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:42.352 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:42.352 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.663108+0000", 2024-09-13T11:24:42.352 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.0140000000000001e-06 2024-09-13T11:24:42.352 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:42.352 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:42.352 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:42.352 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.663116+0000", 2024-09-13T11:24:42.352 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.5440000000000001e-06 2024-09-13T11:24:42.353 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:42.353 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:42.353 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:42.353 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.928111+0000", 2024-09-13T11:24:42.353 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.26499566899999999 2024-09-13T11:24:42.353 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:42.353 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:42.353 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:42.353 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.928135+0000", 2024-09-13T11:24:42.353 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.4003e-05 2024-09-13T11:24:42.353 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:42.353 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:42.353 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:42.354 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.928559+0000", 2024-09-13T11:24:42.354 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000423318 2024-09-13T11:24:42.354 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:42.354 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:42.354 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:24:42.354 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.929562+0000", 2024-09-13T11:24:42.354 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0010027840000000001 2024-09-13T11:24:42.354 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:42.354 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:42.354 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:42.354 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:42.354 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:42.355 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "rep_scrubmap(3.14 e84 from shard 14)", 2024-09-13T11:24:42.355 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:48.313615+0000", 2024-09-13T11:24:42.355 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 53.969779340000002, 2024-09-13T11:24:42.355 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.35633836099999999, 2024-09-13T11:24:42.355 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:42.355 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:42.355 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:24:42.355 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:42.355 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:42.355 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:42.355 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.313615+0000", 2024-09-13T11:24:42.355 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:42.355 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:42.356 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:42.356 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:42.356 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.313615+0000", 2024-09-13T11:24:42.356 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:42.356 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:42.356 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:42.356 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:42.356 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.313617+0000", 2024-09-13T11:24:42.356 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.7990000000000001e-06 2024-09-13T11:24:42.356 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:42.356 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:42.356 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:42.357 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.313688+0000", 2024-09-13T11:24:42.357 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.1481000000000002e-05 2024-09-13T11:24:42.357 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:42.357 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:42.357 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:42.357 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.669785+0000", 2024-09-13T11:24:42.357 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.356096466 2024-09-13T11:24:42.357 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:42.357 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:42.357 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:42.357 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.669792+0000", 2024-09-13T11:24:42.357 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.1740000000000003e-06 2024-09-13T11:24:42.357 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:42.358 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:42.358 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:42.358 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.669839+0000", 2024-09-13T11:24:42.358 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.7323000000000001e-05 2024-09-13T11:24:42.358 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:42.358 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:42.358 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:24:42.358 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.669847+0000", 2024-09-13T11:24:42.358 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.0469999999999994e-06 2024-09-13T11:24:42.358 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:42.358 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:42.358 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:24:42.358 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.669953+0000", 2024-09-13T11:24:42.359 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00010607099999999999 2024-09-13T11:24:42.359 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:42.359 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:42.359 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:42.359 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:42.359 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:42.359 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:91603 3.14 3:2d13add5:::benchmark_data_smithi177_31116_object91602:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111)", 2024-09-13T11:24:42.359 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:11.037157+0000", 2024-09-13T11:24:42.359 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 31.246237728000001, 2024-09-13T11:24:42.359 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.165525535, 2024-09-13T11:24:42.359 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:42.359 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:42.359 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:42.359 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:24:42.360 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:24:42.360 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:42.360 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 91603 2024-09-13T11:24:42.360 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:42.360 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:42.360 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:42.360 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:42.360 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:11.037157+0000", 2024-09-13T11:24:42.360 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:42.360 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:42.360 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:42.360 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:42.360 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:11.037157+0000", 2024-09-13T11:24:42.361 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:42.361 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:42.361 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:42.361 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:42.361 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:11.037159+0000", 2024-09-13T11:24:42.361 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.261e-06 2024-09-13T11:24:42.361 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:42.361 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:42.361 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:42.361 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:11.037188+0000", 2024-09-13T11:24:42.361 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.8798000000000001e-05 2024-09-13T11:24:42.361 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:42.361 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:42.361 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:42.362 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:11.037204+0000", 2024-09-13T11:24:42.362 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.6739999999999999e-05 2024-09-13T11:24:42.362 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:42.362 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:42.362 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:42.362 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:11.037213+0000", 2024-09-13T11:24:42.362 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.1110000000000008e-06 2024-09-13T11:24:42.362 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:42.362 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:42.362 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:42.362 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:11.037270+0000", 2024-09-13T11:24:42.362 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.7629e-05 2024-09-13T11:24:42.362 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:42.363 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:42.363 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:24:42.363 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:11.037364+0000", 2024-09-13T11:24:42.363 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.3871999999999998e-05 2024-09-13T11:24:42.363 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:42.363 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:42.363 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 14", 2024-09-13T11:24:42.363 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:11.037450+0000", 2024-09-13T11:24:42.363 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.5568000000000005e-05 2024-09-13T11:24:42.363 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:42.363 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:42.363 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:24:42.363 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:11.039799+0000", 2024-09-13T11:24:42.363 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0023494039999999998 2024-09-13T11:24:42.364 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:42.364 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:42.364 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:42.364 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:11.202576+0000", 2024-09-13T11:24:42.364 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.16277712499999999 2024-09-13T11:24:42.364 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:42.364 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:42.364 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:24:42.364 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:11.202647+0000", 2024-09-13T11:24:42.364 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.0934999999999998e-05 2024-09-13T11:24:42.364 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:42.364 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:42.364 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:24:42.364 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:11.202682+0000", 2024-09-13T11:24:42.365 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.5092000000000002e-05 2024-09-13T11:24:42.365 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:42.365 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:42.365 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:42.365 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:42.365 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:42.365 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:110268 3.14 3:2997c4c5:::benchmark_data_smithi177_31116_object110267:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119)", 2024-09-13T11:24:42.365 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:25.359386+0000", 2024-09-13T11:24:42.365 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 16.924008023999999, 2024-09-13T11:24:42.365 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.29072703, 2024-09-13T11:24:42.365 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:42.365 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:42.365 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:42.366 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:24:42.366 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:24:42.366 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:42.366 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 110268 2024-09-13T11:24:42.366 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:42.366 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:42.366 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:42.366 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:42.366 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.359386+0000", 2024-09-13T11:24:42.366 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:42.366 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:42.366 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:42.366 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:42.366 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.359386+0000", 2024-09-13T11:24:42.367 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:42.367 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:42.367 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:42.367 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:42.367 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.359389+0000", 2024-09-13T11:24:42.367 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.8600000000000001e-06 2024-09-13T11:24:42.367 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:42.367 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:42.367 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:42.367 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.359424+0000", 2024-09-13T11:24:42.367 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.4755999999999997e-05 2024-09-13T11:24:42.367 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:42.367 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:42.368 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:42.368 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.359445+0000", 2024-09-13T11:24:42.368 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.1104999999999999e-05 2024-09-13T11:24:42.368 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:42.368 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:42.368 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:42.368 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.359460+0000", 2024-09-13T11:24:42.368 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5396999999999998e-05 2024-09-13T11:24:42.368 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:42.368 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:42.368 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:42.368 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.359581+0000", 2024-09-13T11:24:42.368 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000120455 2024-09-13T11:24:42.369 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:42.369 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:42.369 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:24:42.369 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.359827+0000", 2024-09-13T11:24:42.369 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00024571 2024-09-13T11:24:42.369 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:42.369 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:42.369 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 14", 2024-09-13T11:24:42.369 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.360084+0000", 2024-09-13T11:24:42.369 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00025787999999999999 2024-09-13T11:24:42.369 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:42.369 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:42.369 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:24:42.369 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.362243+0000", 2024-09-13T11:24:42.370 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0021581579999999999 2024-09-13T11:24:42.370 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:42.370 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:42.370 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:42.370 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.650034+0000", 2024-09-13T11:24:42.370 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.287791662 2024-09-13T11:24:42.370 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:42.370 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:42.370 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:24:42.370 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.650077+0000", 2024-09-13T11:24:42.370 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.2367999999999998e-05 2024-09-13T11:24:42.370 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:42.370 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:42.371 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:24:42.371 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.650113+0000", 2024-09-13T11:24:42.371 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.6678999999999999e-05 2024-09-13T11:24:42.371 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:42.371 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:42.371 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:42.371 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:42.371 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:42.371 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:110279 3.14 3:2b17a349:::benchmark_data_smithi177_31116_object110278:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119)", 2024-09-13T11:24:42.371 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:25.645137+0000", 2024-09-13T11:24:42.371 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 16.638257186000001, 2024-09-13T11:24:42.371 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.28172927800000003, 2024-09-13T11:24:42.371 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:42.372 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:42.372 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:42.372 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:24:42.372 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:24:42.372 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:42.372 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 110279 2024-09-13T11:24:42.372 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:42.372 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:42.372 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:42.372 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:42.372 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.645137+0000", 2024-09-13T11:24:42.372 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:42.372 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:42.373 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:42.373 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:42.373 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.645137+0000", 2024-09-13T11:24:42.373 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:42.373 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:42.373 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:42.373 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:42.373 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.645140+0000", 2024-09-13T11:24:42.373 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.0939999999999999e-06 2024-09-13T11:24:42.373 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:42.373 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:42.373 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:42.373 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.645184+0000", 2024-09-13T11:24:42.373 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.3412999999999999e-05 2024-09-13T11:24:42.374 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:42.374 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:42.374 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:42.374 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.645206+0000", 2024-09-13T11:24:42.374 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.1997e-05 2024-09-13T11:24:42.374 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:42.374 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:42.374 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:42.374 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.645216+0000", 2024-09-13T11:24:42.374 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0740999999999999e-05 2024-09-13T11:24:42.374 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:42.374 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:42.374 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:42.375 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.645397+0000", 2024-09-13T11:24:42.375 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00018077600000000001 2024-09-13T11:24:42.375 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:42.375 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:42.375 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:24:42.375 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.645638+0000", 2024-09-13T11:24:42.375 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000241091 2024-09-13T11:24:42.375 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:42.375 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:42.375 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 14", 2024-09-13T11:24:42.375 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.645922+0000", 2024-09-13T11:24:42.375 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000283565 2024-09-13T11:24:42.375 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:42.375 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:42.376 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:24:42.376 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.648980+0000", 2024-09-13T11:24:42.376 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00305851 2024-09-13T11:24:42.376 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:42.376 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:42.376 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:42.376 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.926702+0000", 2024-09-13T11:24:42.376 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.27772199800000003 2024-09-13T11:24:42.376 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:42.376 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:42.376 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:24:42.376 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.926796+0000", 2024-09-13T11:24:42.376 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.4158000000000002e-05 2024-09-13T11:24:42.377 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:42.377 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:42.377 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:24:42.377 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.926866+0000", 2024-09-13T11:24:42.377 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.9935e-05 2024-09-13T11:24:42.377 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:42.377 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:42.377 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:42.377 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:42.377 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:42.377 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPush(3.2 124/122 [PushOp(3:43fbaffc:::benchmark_data_smithi177_31116_object10325:head, version: 48'645, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:43fbaffc:::benchmark_data_smithi177_31116_object10325:head@48'645, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:24:42.377 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:31.862400+0000", 2024-09-13T11:24:42.377 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 10.420994713000001, 2024-09-13T11:24:42.377 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.186550994, 2024-09-13T11:24:42.378 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:42.378 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:42.378 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:24:42.378 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:42.378 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:42.378 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:42.378 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:31.862400+0000", 2024-09-13T11:24:42.378 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:42.378 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:42.378 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:42.378 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:42.378 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:31.862400+0000", 2024-09-13T11:24:42.378 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:42.379 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:42.379 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:42.379 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:42.379 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:31.862401+0000", 2024-09-13T11:24:42.379 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.7439999999999999e-06 2024-09-13T11:24:42.379 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:42.379 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:42.379 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:42.379 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:31.862460+0000", 2024-09-13T11:24:42.379 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.897e-05 2024-09-13T11:24:42.379 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:42.379 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:42.379 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:42.379 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:32.047866+0000", 2024-09-13T11:24:42.380 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.18540612400000001 2024-09-13T11:24:42.380 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:42.380 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:42.380 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:42.380 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:32.047878+0000", 2024-09-13T11:24:42.380 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.1618e-05 2024-09-13T11:24:42.380 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:42.380 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:42.380 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:42.380 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:32.047975+0000", 2024-09-13T11:24:42.380 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.7176999999999995e-05 2024-09-13T11:24:42.380 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:42.380 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:42.381 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:24:42.381 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:32.047991+0000", 2024-09-13T11:24:42.381 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5722999999999998e-05 2024-09-13T11:24:42.381 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:42.381 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:42.381 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:24:42.381 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:32.048951+0000", 2024-09-13T11:24:42.381 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00095963799999999996 2024-09-13T11:24:42.381 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:42.381 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:42.381 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:42.381 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:42.384 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:42.384 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-09-13T11:24:42.386 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-13T11:24:42.387 INFO:tasks.ceph.osd.12.smithi001.stderr:2024-09-13T11:24:42.383+0000 7f892c6e6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26314) UID: 0 2024-09-13T11:24:42.487 INFO:tasks.ceph.osd.8.smithi001.stderr:2024-09-13T11:24:42.483+0000 7fb9b48ae640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26312) UID: 0 2024-09-13T11:24:42.557 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-09-13T11:24:42.558 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [ 2024-09-13T11:24:42.558 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:42.558 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:125404 3.12 3:497094bc:::benchmark_data_smithi177_31116_object125403:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e130)", 2024-09-13T11:24:42.558 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:42.544226+0000", 2024-09-13T11:24:42.558 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 0.012656248, 2024-09-13T11:24:42.558 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.012684314, 2024-09-13T11:24:42.558 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:42.558 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:42.558 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "waiting for sub ops", 2024-09-13T11:24:42.558 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:24:42.558 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:24:42.558 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:42.558 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 125404 2024-09-13T11:24:42.558 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:42.559 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:42.559 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:42.559 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:42.559 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:42.544226+0000", 2024-09-13T11:24:42.559 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:42.559 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:42.559 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:42.559 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:42.559 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:42.544226+0000", 2024-09-13T11:24:42.559 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:42.559 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:42.559 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:42.559 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:42.560 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:42.544229+0000", 2024-09-13T11:24:42.560 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.943e-06 2024-09-13T11:24:42.560 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:42.560 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:42.560 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:42.560 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:42.544253+0000", 2024-09-13T11:24:42.560 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.4263000000000001e-05 2024-09-13T11:24:42.560 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:42.560 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:42.560 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:42.560 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:42.544264+0000", 2024-09-13T11:24:42.560 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.1017e-05 2024-09-13T11:24:42.560 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:42.561 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:42.561 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:42.561 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:42.544271+0000", 2024-09-13T11:24:42.561 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.4710000000000001e-06 2024-09-13T11:24:42.561 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:42.561 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:42.561 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:42.561 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:42.544324+0000", 2024-09-13T11:24:42.561 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.2191000000000002e-05 2024-09-13T11:24:42.561 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:42.561 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:42.561 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:24:42.561 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:42.544464+0000", 2024-09-13T11:24:42.561 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00014009100000000001 2024-09-13T11:24:42.562 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:42.562 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:42.562 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 5", 2024-09-13T11:24:42.562 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:42.544602+0000", 2024-09-13T11:24:42.562 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00013798900000000001 2024-09-13T11:24:42.562 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:42.562 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:42.562 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:24:42.562 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:42.547358+0000", 2024-09-13T11:24:42.562 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0027566299999999999 2024-09-13T11:24:42.562 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:42.562 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:42.562 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:42.562 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:42.563 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:42.563 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:125411 3.12 3:4c65a2fc:::benchmark_data_smithi177_31116_object125410:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e130)", 2024-09-13T11:24:42.563 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:42.550635+0000", 2024-09-13T11:24:42.563 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 0.0062467360000000001, 2024-09-13T11:24:42.563 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0063553960000000001, 2024-09-13T11:24:42.563 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:42.563 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:42.563 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "waiting for sub ops", 2024-09-13T11:24:42.563 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:24:42.563 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:24:42.563 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:42.563 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 125411 2024-09-13T11:24:42.563 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:42.563 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:42.564 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:42.564 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:42.564 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:42.550635+0000", 2024-09-13T11:24:42.564 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:42.564 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:42.564 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:42.564 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:42.564 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:42.550635+0000", 2024-09-13T11:24:42.564 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:42.564 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:42.564 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:42.564 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:42.564 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:42.550639+0000", 2024-09-13T11:24:42.564 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.104e-06 2024-09-13T11:24:42.565 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:42.565 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:42.565 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:42.565 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:42.550669+0000", 2024-09-13T11:24:42.565 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.9640000000000001e-05 2024-09-13T11:24:42.565 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:42.565 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:42.565 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:42.565 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:42.550684+0000", 2024-09-13T11:24:42.565 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.484e-05 2024-09-13T11:24:42.565 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:42.565 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:42.565 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:42.565 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:42.550694+0000", 2024-09-13T11:24:42.566 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.8970000000000003e-06 2024-09-13T11:24:42.566 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:42.566 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:42.566 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:42.566 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:42.550968+0000", 2024-09-13T11:24:42.566 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00027453599999999997 2024-09-13T11:24:42.566 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:42.566 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:42.566 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:24:42.566 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:42.551125+0000", 2024-09-13T11:24:42.566 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00015650500000000001 2024-09-13T11:24:42.566 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:42.566 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:42.566 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 5", 2024-09-13T11:24:42.567 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:42.551293+0000", 2024-09-13T11:24:42.567 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00016793799999999999 2024-09-13T11:24:42.567 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:42.567 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:42.567 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:42.567 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:42.567 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:42.567 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop_reply(client.4665.0:125282 3.12 e130/122)", 2024-09-13T11:24:42.567 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:42.554838+0000", 2024-09-13T11:24:42.567 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 0.0020440160000000001, 2024-09-13T11:24:42.567 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.002216614, 2024-09-13T11:24:42.567 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:42.567 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:42.567 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "queued for pg", 2024-09-13T11:24:42.567 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:42.568 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:42.568 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:42.568 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:42.554838+0000", 2024-09-13T11:24:42.568 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:42.568 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:42.568 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:42.568 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:42.568 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:42.554838+0000", 2024-09-13T11:24:42.568 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:42.568 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:42.568 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:42.568 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:42.568 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:42.554839+0000", 2024-09-13T11:24:42.568 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.1909999999999999e-06 2024-09-13T11:24:42.569 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:42.569 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:42.569 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:42.569 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:42.554844+0000", 2024-09-13T11:24:42.569 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.5079999999999999e-06 2024-09-13T11:24:42.569 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:42.569 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:42.569 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:42.569 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:42.554855+0000", 2024-09-13T11:24:42.569 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.1674999999999999e-05 2024-09-13T11:24:42.569 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:42.569 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:42.569 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:42.570 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:42.554861+0000", 2024-09-13T11:24:42.570 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.3510000000000001e-06 2024-09-13T11:24:42.570 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:42.570 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:42.570 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:42.570 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:42.570 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:42.570 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:125414 3.a e130/117)", 2024-09-13T11:24:42.570 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:42.556867+0000", 2024-09-13T11:24:42.570 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 1.5048999999999999e-05, 2024-09-13T11:24:42.570 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000231084, 2024-09-13T11:24:42.570 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:42.570 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:42.570 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "queued for pg", 2024-09-13T11:24:42.571 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:42.571 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:42.571 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:42.571 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:42.556867+0000", 2024-09-13T11:24:42.571 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:42.571 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:42.571 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:42.571 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:42.571 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:42.556867+0000", 2024-09-13T11:24:42.571 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:42.571 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:42.571 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:42.571 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:42.571 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:42.556868+0000", 2024-09-13T11:24:42.572 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.511e-06 2024-09-13T11:24:42.572 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:42.572 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:42.572 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:42.572 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:42.556893+0000", 2024-09-13T11:24:42.572 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.4998e-05 2024-09-13T11:24:42.572 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:42.572 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:42.572 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:42.572 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:42.556912+0000", 2024-09-13T11:24:42.572 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.8762000000000001e-05 2024-09-13T11:24:42.572 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:42.573 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:42.573 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:42.573 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:42.556920+0000", 2024-09-13T11:24:42.573 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.4630000000000004e-06 2024-09-13T11:24:42.573 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:42.573 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:42.573 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:42.573 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:42.573 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:42.573 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:125282 3.12 3:496a42f2:::benchmark_data_smithi177_31116_object125281:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e130)", 2024-09-13T11:24:42.573 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:42.448134+0000", 2024-09-13T11:24:42.573 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 0.108747576, 2024-09-13T11:24:42.573 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.109023579, 2024-09-13T11:24:42.573 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:42.574 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:42.574 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "waiting for sub ops", 2024-09-13T11:24:42.574 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:24:42.574 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:24:42.574 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:42.574 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 125282 2024-09-13T11:24:42.574 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:42.574 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:42.574 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:42.574 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:42.574 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:42.448134+0000", 2024-09-13T11:24:42.574 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:42.574 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:42.575 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:42.575 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:42.575 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:42.448134+0000", 2024-09-13T11:24:42.575 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:42.575 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:42.575 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:42.575 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:42.575 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:42.448137+0000", 2024-09-13T11:24:42.575 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.3499999999999999e-06 2024-09-13T11:24:42.575 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:42.575 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:42.575 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:42.575 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:42.448164+0000", 2024-09-13T11:24:42.575 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.7042999999999998e-05 2024-09-13T11:24:42.576 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:42.576 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:42.576 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:42.576 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:42.448176+0000", 2024-09-13T11:24:42.576 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.2248e-05 2024-09-13T11:24:42.576 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:42.576 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:42.576 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:42.576 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:42.448185+0000", 2024-09-13T11:24:42.576 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.4349999999999997e-06 2024-09-13T11:24:42.576 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:42.576 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:42.576 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:42.576 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:42.451261+0000", 2024-09-13T11:24:42.576 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0030761769999999998 2024-09-13T11:24:42.577 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:42.577 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:42.577 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:24:42.577 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:42.451430+0000", 2024-09-13T11:24:42.577 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000169586 2024-09-13T11:24:42.577 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:42.577 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:42.577 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 5", 2024-09-13T11:24:42.577 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:42.451920+0000", 2024-09-13T11:24:42.577 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00048972999999999996 2024-09-13T11:24:42.577 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:42.577 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:42.577 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:24:42.577 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:42.455575+0000", 2024-09-13T11:24:42.578 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0036550089999999999 2024-09-13T11:24:42.578 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:42.578 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:42.578 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:42.578 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:42.578 INFO:teuthology.orchestra.run.smithi177.stdout: ], 2024-09-13T11:24:42.578 INFO:teuthology.orchestra.run.smithi177.stdout: "num_ops": 5 2024-09-13T11:24:42.578 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-09-13T11:24:42.581 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-13T11:24:42.589 INFO:tasks.ceph.osd.13.smithi092.stderr:2024-09-13T11:24:42.582+0000 7f3e5afea640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26350) UID: 0 2024-09-13T11:24:42.662 INFO:teuthology.orchestra.run.smithi162.stdout:m(3) p(3,0-12) O(3,0-13)=block_cache={type=binned_lru} L=min_write_buffer_number_to_merge=32 P=min_write_buffer_number_to_merge=32 2024-09-13T11:24:42.667 INFO:tasks.thrashosds.thrasher:applying new sharding to bluestore on osd.14 2024-09-13T11:24:42.667 INFO:tasks.thrashosds.thrasher:generate random sharding 2024-09-13T11:24:42.667 INFO:tasks.thrashosds.thrasher:applying new sharding: P(3,8-) 2024-09-13T11:24:42.667 DEBUG:teuthology.orchestra.run.smithi162:> sudo ceph-bluestore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/bluestore_tool.$pid.log' --log-level=10 --path /var/lib/ceph/osd/ceph-14 --sharding 'P(3,8-)' reshard 2024-09-13T11:24:42.710 INFO:tasks.ceph.osd.10.smithi162.stderr:2024-09-13T11:24:42.708+0000 7fb59c9f0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26317) UID: 0 2024-09-13T11:24:42.775 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-09-13T11:24:42.775 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [], 2024-09-13T11:24:42.775 INFO:teuthology.orchestra.run.smithi177.stdout: "complaint_time": 30, 2024-09-13T11:24:42.775 INFO:teuthology.orchestra.run.smithi177.stdout: "num_blocked_ops": 0 2024-09-13T11:24:42.776 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-09-13T11:24:42.791 INFO:tasks.ceph.osd.8.smithi001.stderr:2024-09-13T11:24:42.787+0000 7fb9b48ae640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26312) UID: 0 2024-09-13T11:24:42.813 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-13T11:24:42.893 INFO:tasks.ceph.osd.5.smithi092.stderr:2024-09-13T11:24:42.886+0000 7f6f0b705640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26346) UID: 0 2024-09-13T11:24:42.993 INFO:tasks.ceph.osd.5.smithi092.stderr:2024-09-13T11:24:42.986+0000 7f6f0b705640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26346) UID: 0 2024-09-13T11:24:43.012 INFO:tasks.radosbench.radosbench.0.smithi177.stdout: 136 16 126037 126021 57.9078 78.0625 0.00517739 0.017264 2024-09-13T11:24:43.089 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-09-13T11:24:43.090 INFO:teuthology.orchestra.run.smithi177.stdout: "size": 20, 2024-09-13T11:24:43.090 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 600, 2024-09-13T11:24:43.090 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [ 2024-09-13T11:24:43.090 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.090 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPushReply(3.13 75/70 [PushReplyOp(3:cfab03d6:::benchmark_data_smithi177_31116_object16599:head)])", 2024-09-13T11:24:43.090 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:31.163480+0000", 2024-09-13T11:24:43.090 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 71.923643725000005, 2024-09-13T11:24:43.090 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.63009029299999997, 2024-09-13T11:24:43.090 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:43.090 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:43.090 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-13T11:24:43.090 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:43.091 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.091 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:43.091 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.163480+0000", 2024-09-13T11:24:43.091 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:43.091 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.091 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.091 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:43.091 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.163480+0000", 2024-09-13T11:24:43.091 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:43.091 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.091 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.091 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:43.091 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.163481+0000", 2024-09-13T11:24:43.091 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0410000000000001e-06 2024-09-13T11:24:43.092 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.092 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.092 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:43.092 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.163485+0000", 2024-09-13T11:24:43.092 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.1629999999999998e-06 2024-09-13T11:24:43.092 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.092 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.092 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:43.093 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.792166+0000", 2024-09-13T11:24:43.093 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.62868067400000005 2024-09-13T11:24:43.093 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.093 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.093 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:43.093 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.792184+0000", 2024-09-13T11:24:43.093 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.7898999999999999e-05 2024-09-13T11:24:43.093 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.093 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.093 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:43.093 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.792312+0000", 2024-09-13T11:24:43.093 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00012871000000000001 2024-09-13T11:24:43.094 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.094 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.094 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:24:43.094 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.793570+0000", 2024-09-13T11:24:43.094 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001257806 2024-09-13T11:24:43.094 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:43.094 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:43.094 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:43.094 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.094 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.094 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPush(3.12 75/69 [PushOp(3:4a36ca6d:::benchmark_data_smithi177_31116_object31707:head, version: 59'1993, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:4a36ca6d:::benchmark_data_smithi177_31116_object31707:head@59'1993, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:24:43.095 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:31.309623+0000", 2024-09-13T11:24:43.095 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 71.777500770000003, 2024-09-13T11:24:43.095 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.165459152, 2024-09-13T11:24:43.095 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:43.095 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:43.095 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:24:43.095 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:43.095 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.095 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:43.095 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.309623+0000", 2024-09-13T11:24:43.095 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:43.095 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.095 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.096 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:43.096 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.309623+0000", 2024-09-13T11:24:43.096 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:43.096 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.096 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.096 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:43.096 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.309625+0000", 2024-09-13T11:24:43.096 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.3860000000000001e-06 2024-09-13T11:24:43.096 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.096 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.096 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:43.096 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.309680+0000", 2024-09-13T11:24:43.096 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.4388000000000002e-05 2024-09-13T11:24:43.097 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.097 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.097 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:43.097 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.473157+0000", 2024-09-13T11:24:43.097 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.163476813 2024-09-13T11:24:43.097 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.097 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.097 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:43.097 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.473189+0000", 2024-09-13T11:24:43.097 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.2975000000000002e-05 2024-09-13T11:24:43.097 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.097 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.097 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:43.098 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.473395+0000", 2024-09-13T11:24:43.098 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000206009 2024-09-13T11:24:43.098 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.098 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.098 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:24:43.098 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.473424+0000", 2024-09-13T11:24:43.098 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.8255e-05 2024-09-13T11:24:43.098 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.098 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.098 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:24:43.098 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.475082+0000", 2024-09-13T11:24:43.098 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0016583259999999999 2024-09-13T11:24:43.098 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:43.098 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:43.099 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:43.099 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.099 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.099 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPushReply(3.13 76/70 [PushReplyOp(3:c9d43a82:::benchmark_data_smithi177_31116_object20026:head)])", 2024-09-13T11:24:43.099 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:32.427709+0000", 2024-09-13T11:24:43.099 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 70.659414209999994, 2024-09-13T11:24:43.099 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.46547316300000002, 2024-09-13T11:24:43.099 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:43.099 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:43.099 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-13T11:24:43.099 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:43.099 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.099 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:43.099 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:32.427709+0000", 2024-09-13T11:24:43.100 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:43.100 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.100 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.100 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:43.100 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:32.427709+0000", 2024-09-13T11:24:43.100 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:43.100 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.100 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.100 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:43.100 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:32.427711+0000", 2024-09-13T11:24:43.100 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5099999999999999e-06 2024-09-13T11:24:43.100 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.100 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.101 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:43.101 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:32.427717+0000", 2024-09-13T11:24:43.101 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.1569999999999997e-06 2024-09-13T11:24:43.101 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.101 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.101 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:43.101 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:32.892390+0000", 2024-09-13T11:24:43.101 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.46467251100000001 2024-09-13T11:24:43.101 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.101 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.101 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:43.101 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:32.892405+0000", 2024-09-13T11:24:43.101 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5296999999999999e-05 2024-09-13T11:24:43.102 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.102 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.102 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:43.102 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:32.892503+0000", 2024-09-13T11:24:43.102 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.7820999999999997e-05 2024-09-13T11:24:43.102 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.102 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.102 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:24:43.102 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:32.893183+0000", 2024-09-13T11:24:43.102 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00067986700000000001 2024-09-13T11:24:43.102 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:43.102 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:43.102 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:43.102 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.103 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.103 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:56066 3.a e79/75 3:561a1dd2:::benchmark_data_smithi177_31116_object56065:head v 79'3452, mlcod=79'3449)", 2024-09-13T11:24:43.103 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:36.194429+0000", 2024-09-13T11:24:43.103 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 66.892694324999994, 2024-09-13T11:24:43.103 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.13436872499999999, 2024-09-13T11:24:43.103 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:43.103 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:43.103 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:43.103 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:43.103 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.103 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:43.104 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:36.194429+0000", 2024-09-13T11:24:43.104 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:43.104 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.104 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.104 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:43.104 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:36.194429+0000", 2024-09-13T11:24:43.104 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:43.104 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.104 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.105 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:43.105 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:36.194431+0000", 2024-09-13T11:24:43.105 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.4279999999999999e-06 2024-09-13T11:24:43.105 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.106 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.106 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:43.106 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:36.194467+0000", 2024-09-13T11:24:43.106 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.6402000000000001e-05 2024-09-13T11:24:43.107 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.107 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.107 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:43.107 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:36.327354+0000", 2024-09-13T11:24:43.107 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.13288720800000001 2024-09-13T11:24:43.107 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.107 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.107 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:43.107 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:36.327361+0000", 2024-09-13T11:24:43.107 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.302e-06 2024-09-13T11:24:43.107 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.108 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.108 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:43.108 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:36.327441+0000", 2024-09-13T11:24:43.108 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.0406000000000002e-05 2024-09-13T11:24:43.108 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.108 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.108 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:24:43.108 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:36.327466+0000", 2024-09-13T11:24:43.108 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.5367e-05 2024-09-13T11:24:43.108 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.108 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.109 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:24:43.109 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:36.328754+0000", 2024-09-13T11:24:43.109 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001287627 2024-09-13T11:24:43.109 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.109 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.109 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:24:43.109 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:36.328798+0000", 2024-09-13T11:24:43.109 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.3985e-05 2024-09-13T11:24:43.109 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:43.109 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:43.109 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:43.110 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.110 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.110 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:57435 3.3 e80/75 3:c0db4777:::benchmark_data_smithi177_31116_object57434:head v 80'3155, mlcod=80'3153)", 2024-09-13T11:24:43.110 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:37.437070+0000", 2024-09-13T11:24:43.110 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 65.650053392999993, 2024-09-13T11:24:43.110 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.35861845999999997, 2024-09-13T11:24:43.110 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:43.110 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:43.110 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:43.111 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:43.111 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.111 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:43.111 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:37.437070+0000", 2024-09-13T11:24:43.111 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:43.111 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.111 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.111 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:43.111 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:37.437070+0000", 2024-09-13T11:24:43.111 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:43.112 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.112 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.112 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:43.112 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:37.437072+0000", 2024-09-13T11:24:43.112 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5549999999999999e-06 2024-09-13T11:24:43.112 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.112 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.112 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:43.112 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:37.437141+0000", 2024-09-13T11:24:43.112 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.9494999999999998e-05 2024-09-13T11:24:43.113 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.113 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.113 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:43.113 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:37.793400+0000", 2024-09-13T11:24:43.113 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.35625817999999998 2024-09-13T11:24:43.113 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.113 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.113 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:43.113 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:37.793406+0000", 2024-09-13T11:24:43.113 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.6830000000000002e-06 2024-09-13T11:24:43.113 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.114 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.114 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:43.114 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:37.793472+0000", 2024-09-13T11:24:43.114 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.5671000000000002e-05 2024-09-13T11:24:43.114 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.114 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.114 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:24:43.114 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:37.793492+0000", 2024-09-13T11:24:43.114 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.0051000000000002e-05 2024-09-13T11:24:43.114 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.114 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.114 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:24:43.115 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:37.795626+0000", 2024-09-13T11:24:43.115 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0021344990000000002 2024-09-13T11:24:43.115 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.115 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.115 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:24:43.115 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:37.795689+0000", 2024-09-13T11:24:43.115 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.2325999999999999e-05 2024-09-13T11:24:43.115 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:43.115 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:43.115 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:43.115 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.116 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.116 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:57985 3.3 e80/75 3:c330bb03:::benchmark_data_smithi177_31116_object57984:head v 80'3162, mlcod=80'3159)", 2024-09-13T11:24:43.116 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:38.007076+0000", 2024-09-13T11:24:43.116 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 65.080048040999998, 2024-09-13T11:24:43.116 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.18211438199999999, 2024-09-13T11:24:43.116 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:43.116 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:43.116 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:43.116 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:43.116 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.117 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:43.117 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:38.007076+0000", 2024-09-13T11:24:43.117 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:43.117 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.117 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.117 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:43.117 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:38.007076+0000", 2024-09-13T11:24:43.117 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:43.117 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.117 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.117 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:43.118 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:38.007077+0000", 2024-09-13T11:24:43.118 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.6759999999999999e-06 2024-09-13T11:24:43.118 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.118 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.118 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:43.118 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:38.007221+0000", 2024-09-13T11:24:43.118 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00014344299999999999 2024-09-13T11:24:43.118 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.118 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.118 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:43.118 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:38.186808+0000", 2024-09-13T11:24:43.118 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.17958726699999999 2024-09-13T11:24:43.119 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.119 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.119 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:43.119 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:38.186815+0000", 2024-09-13T11:24:43.119 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.0890000000000002e-06 2024-09-13T11:24:43.119 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.119 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.119 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:43.119 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:38.186887+0000", 2024-09-13T11:24:43.120 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.2006000000000001e-05 2024-09-13T11:24:43.120 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.120 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.120 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:24:43.120 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:38.186913+0000", 2024-09-13T11:24:43.120 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.5962e-05 2024-09-13T11:24:43.120 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.120 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.120 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:24:43.120 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:38.189123+0000", 2024-09-13T11:24:43.120 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0022099649999999999 2024-09-13T11:24:43.121 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.121 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.121 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:24:43.121 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:38.189190+0000", 2024-09-13T11:24:43.121 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.6973999999999999e-05 2024-09-13T11:24:43.121 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:43.121 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:43.121 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:43.121 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.121 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.121 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPushReply(3.23 84/78 [PushReplyOp(3:c7e44616:::benchmark_data_smithi177_31116_object32371:head)])", 2024-09-13T11:24:43.122 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:41.338210+0000", 2024-09-13T11:24:43.122 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 61.748913322, 2024-09-13T11:24:43.122 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.41118492299999998, 2024-09-13T11:24:43.122 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:43.122 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:43.122 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-13T11:24:43.122 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:43.122 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.122 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:43.122 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.338210+0000", 2024-09-13T11:24:43.122 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:43.123 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.123 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.123 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:43.123 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.338210+0000", 2024-09-13T11:24:43.123 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:43.123 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.123 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.123 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:43.123 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.338212+0000", 2024-09-13T11:24:43.123 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.314e-06 2024-09-13T11:24:43.123 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.123 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.123 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:43.124 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.338219+0000", 2024-09-13T11:24:43.124 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.6469999999999998e-06 2024-09-13T11:24:43.124 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.124 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.124 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:43.124 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.748675+0000", 2024-09-13T11:24:43.124 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.41045567500000002 2024-09-13T11:24:43.124 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.124 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.125 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:43.125 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.748684+0000", 2024-09-13T11:24:43.125 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.2150000000000007e-06 2024-09-13T11:24:43.125 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.125 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.125 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:43.125 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.748739+0000", 2024-09-13T11:24:43.125 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.4543000000000003e-05 2024-09-13T11:24:43.125 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.125 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.125 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:24:43.125 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.749395+0000", 2024-09-13T11:24:43.126 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00065652899999999999 2024-09-13T11:24:43.126 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:43.126 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:43.126 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:43.126 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.126 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.126 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPushReply(3.23 85/78 [PushReplyOp(3:c5c6dce1:::benchmark_data_smithi177_31116_object37293:head)])", 2024-09-13T11:24:43.126 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:42.377862+0000", 2024-09-13T11:24:43.126 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 60.709261863000002, 2024-09-13T11:24:43.126 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.62538808599999995, 2024-09-13T11:24:43.126 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:43.126 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:43.126 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-13T11:24:43.127 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:43.127 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.127 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:43.127 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.377862+0000", 2024-09-13T11:24:43.127 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:43.127 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.127 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.127 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:43.127 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.377862+0000", 2024-09-13T11:24:43.127 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:43.127 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.127 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.128 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:43.128 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.377863+0000", 2024-09-13T11:24:43.128 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.6899999999999996e-07 2024-09-13T11:24:43.128 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.128 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.128 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:43.128 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.377866+0000", 2024-09-13T11:24:43.128 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.219e-06 2024-09-13T11:24:43.128 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.128 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.128 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:43.128 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:43.001857+0000", 2024-09-13T11:24:43.129 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.62399084299999996 2024-09-13T11:24:43.129 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.129 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.129 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:43.129 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:43.001869+0000", 2024-09-13T11:24:43.129 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.2415999999999999e-05 2024-09-13T11:24:43.129 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.129 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.129 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:43.129 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:43.002194+0000", 2024-09-13T11:24:43.129 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000324774 2024-09-13T11:24:43.130 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.130 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.130 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:24:43.130 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:43.003250+0000", 2024-09-13T11:24:43.130 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001055865 2024-09-13T11:24:43.130 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:43.130 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:43.130 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:43.130 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.130 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.130 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPushReply(3.23 86/78 [PushReplyOp(3:c4ca4518:::benchmark_data_smithi177_31116_object39787:head)])", 2024-09-13T11:24:43.130 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:43.580816+0000", 2024-09-13T11:24:43.131 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 59.506308171999997, 2024-09-13T11:24:43.131 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.14999331199999999, 2024-09-13T11:24:43.131 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:43.131 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:43.131 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-13T11:24:43.131 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:43.131 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.131 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:43.131 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:43.580816+0000", 2024-09-13T11:24:43.131 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:43.131 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.132 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.132 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:43.132 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:43.580816+0000", 2024-09-13T11:24:43.132 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:43.132 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.132 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.132 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:43.132 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:43.580818+0000", 2024-09-13T11:24:43.132 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.7250000000000002e-06 2024-09-13T11:24:43.132 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.133 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.133 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:43.133 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:43.580830+0000", 2024-09-13T11:24:43.133 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.2247e-05 2024-09-13T11:24:43.133 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.133 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.133 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:43.133 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:43.730204+0000", 2024-09-13T11:24:43.133 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.149373531 2024-09-13T11:24:43.133 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.133 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.133 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:43.134 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:43.730213+0000", 2024-09-13T11:24:43.134 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.6989999999999992e-06 2024-09-13T11:24:43.134 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.134 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.134 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:43.134 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:43.730274+0000", 2024-09-13T11:24:43.134 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.1419000000000003e-05 2024-09-13T11:24:43.134 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.134 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.134 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:24:43.134 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:43.730809+0000", 2024-09-13T11:24:43.134 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00053469100000000001 2024-09-13T11:24:43.135 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:43.135 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:43.135 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:43.135 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.135 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.135 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPush(3.1c 89/77 [PushOp(3:38bcbc71:::benchmark_data_smithi177_31116_object31448:head, version: 59'1946, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:38bcbc71:::benchmark_data_smithi177_31116_object31448:head@59'1946, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:24:43.135 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:46.670723+0000", 2024-09-13T11:24:43.135 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 56.416400639000003, 2024-09-13T11:24:43.135 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.27576065, 2024-09-13T11:24:43.135 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:43.136 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:43.136 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:24:43.136 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:43.136 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.136 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:43.136 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.670723+0000", 2024-09-13T11:24:43.136 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:43.136 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.136 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.136 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:43.136 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.670723+0000", 2024-09-13T11:24:43.136 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:43.137 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.137 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.137 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:43.137 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.670724+0000", 2024-09-13T11:24:43.137 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.4389999999999999e-06 2024-09-13T11:24:43.137 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.137 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.137 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:43.137 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.670754+0000", 2024-09-13T11:24:43.137 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.9224999999999999e-05 2024-09-13T11:24:43.137 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.138 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.138 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:43.138 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.944369+0000", 2024-09-13T11:24:43.138 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.27361529800000001 2024-09-13T11:24:43.138 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.138 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.138 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:43.138 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.944422+0000", 2024-09-13T11:24:43.138 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.3318e-05 2024-09-13T11:24:43.138 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.138 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.138 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:43.139 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.944711+0000", 2024-09-13T11:24:43.139 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00028912399999999998 2024-09-13T11:24:43.139 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.139 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.139 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:24:43.139 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.944735+0000", 2024-09-13T11:24:43.139 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.3366999999999998e-05 2024-09-13T11:24:43.139 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.139 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.139 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:24:43.139 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.946484+0000", 2024-09-13T11:24:43.139 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0017488790000000001 2024-09-13T11:24:43.140 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:43.140 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:43.140 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:43.140 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.140 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.140 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPush(3.1c 91/77 [PushOp(3:3e18abc2:::benchmark_data_smithi177_31116_object35874:head, version: 62'2235, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:3e18abc2:::benchmark_data_smithi177_31116_object35874:head@62'2235, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:24:43.140 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:48.350968+0000", 2024-09-13T11:24:43.140 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 54.736155326000002, 2024-09-13T11:24:43.140 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.19257698000000001, 2024-09-13T11:24:43.140 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:43.141 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:43.141 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:24:43.141 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:43.141 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.141 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:43.141 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.350968+0000", 2024-09-13T11:24:43.141 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:43.141 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.141 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.141 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:43.141 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.350968+0000", 2024-09-13T11:24:43.141 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:43.142 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.142 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.142 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:43.142 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.350970+0000", 2024-09-13T11:24:43.142 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.796e-06 2024-09-13T11:24:43.142 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.142 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.142 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:43.142 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.351003+0000", 2024-09-13T11:24:43.142 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.2631e-05 2024-09-13T11:24:43.142 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.143 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.143 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:43.143 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.542140+0000", 2024-09-13T11:24:43.143 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.191136737 2024-09-13T11:24:43.143 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.143 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.143 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:43.143 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.542158+0000", 2024-09-13T11:24:43.143 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.8896999999999999e-05 2024-09-13T11:24:43.143 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.143 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.143 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:43.144 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.542263+0000", 2024-09-13T11:24:43.144 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000104158 2024-09-13T11:24:43.144 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.144 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.144 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:24:43.144 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.542279+0000", 2024-09-13T11:24:43.144 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5926999999999998e-05 2024-09-13T11:24:43.144 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.144 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.144 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:24:43.144 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.543545+0000", 2024-09-13T11:24:43.145 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0012668340000000001 2024-09-13T11:24:43.145 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:43.145 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:43.145 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:43.145 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.145 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.145 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:83211 3.1c e106/77 3:3ae31a0f:::benchmark_data_smithi177_31116_object83210:head v 106'4362, mlcod=106'4360)", 2024-09-13T11:24:43.145 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:04.345264+0000", 2024-09-13T11:24:43.145 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 38.741859673999997, 2024-09-13T11:24:43.145 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.17036547399999999, 2024-09-13T11:24:43.145 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:43.145 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:43.146 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:43.146 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:43.146 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.146 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:43.146 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.345264+0000", 2024-09-13T11:24:43.146 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:43.146 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.146 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.146 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:43.146 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.345264+0000", 2024-09-13T11:24:43.146 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:43.146 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.147 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.147 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:43.147 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.345267+0000", 2024-09-13T11:24:43.147 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.7590000000000002e-06 2024-09-13T11:24:43.147 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.147 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.147 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:43.147 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.345298+0000", 2024-09-13T11:24:43.147 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.1173000000000001e-05 2024-09-13T11:24:43.147 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.147 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.147 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:43.148 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.512265+0000", 2024-09-13T11:24:43.148 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.16696657100000001 2024-09-13T11:24:43.148 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.148 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.148 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:43.148 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.512279+0000", 2024-09-13T11:24:43.148 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.4953999999999999e-05 2024-09-13T11:24:43.148 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.148 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.148 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:43.148 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.512402+0000", 2024-09-13T11:24:43.149 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00012277999999999999 2024-09-13T11:24:43.149 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.149 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.149 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:24:43.149 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.512440+0000", 2024-09-13T11:24:43.149 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.7861999999999998e-05 2024-09-13T11:24:43.149 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.149 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.149 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:24:43.149 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.515515+0000", 2024-09-13T11:24:43.149 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00307472 2024-09-13T11:24:43.149 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.150 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.150 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:24:43.150 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.515629+0000", 2024-09-13T11:24:43.150 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00011465500000000001 2024-09-13T11:24:43.150 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:43.150 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:43.150 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:43.150 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.150 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.150 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:83223 3.1c e106/77 3:3bc6018d:::benchmark_data_smithi177_31116_object83222:head v 106'4363, mlcod=106'4360)", 2024-09-13T11:24:43.150 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:04.512139+0000", 2024-09-13T11:24:43.151 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 38.574984528000002, 2024-09-13T11:24:43.151 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.471923235, 2024-09-13T11:24:43.151 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:43.151 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:43.151 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:43.151 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:43.151 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.151 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:43.151 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.512139+0000", 2024-09-13T11:24:43.151 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:43.151 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.151 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.152 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:43.152 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.512139+0000", 2024-09-13T11:24:43.152 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:43.152 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.152 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.152 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:43.152 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.512143+0000", 2024-09-13T11:24:43.152 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.427e-06 2024-09-13T11:24:43.152 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.152 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.152 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:43.153 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.512227+0000", 2024-09-13T11:24:43.153 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.3965e-05 2024-09-13T11:24:43.153 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.153 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.153 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:43.153 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.976330+0000", 2024-09-13T11:24:43.153 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.46410342599999999 2024-09-13T11:24:43.153 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.153 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.153 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:43.153 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.976350+0000", 2024-09-13T11:24:43.154 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.9533000000000001e-05 2024-09-13T11:24:43.154 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.154 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.154 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:43.154 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.976488+0000", 2024-09-13T11:24:43.154 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00013799400000000001 2024-09-13T11:24:43.154 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.154 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.154 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:24:43.154 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.976543+0000", 2024-09-13T11:24:43.154 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.5442000000000002e-05 2024-09-13T11:24:43.155 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.155 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.155 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:24:43.155 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.983227+0000", 2024-09-13T11:24:43.155 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0066842899999999998 2024-09-13T11:24:43.155 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.155 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.155 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:24:43.155 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.984062+0000", 2024-09-13T11:24:43.155 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00083515800000000002 2024-09-13T11:24:43.155 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:43.156 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:43.156 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:43.156 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.156 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.156 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:110256 3.13 e119/83 3:cb00d078:::benchmark_data_smithi177_31116_object110255:head v 119'4823, mlcod=119'4821)", 2024-09-13T11:24:43.156 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:25.284530+0000", 2024-09-13T11:24:43.156 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 17.802594068000001, 2024-09-13T11:24:43.156 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.343491771, 2024-09-13T11:24:43.156 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:43.156 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:43.156 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:43.157 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:43.157 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.157 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:43.157 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.284530+0000", 2024-09-13T11:24:43.157 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:43.157 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.157 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.157 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:43.157 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.284530+0000", 2024-09-13T11:24:43.157 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:43.157 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.157 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.158 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:43.158 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.284532+0000", 2024-09-13T11:24:43.158 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.8870000000000001e-06 2024-09-13T11:24:43.158 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.158 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.158 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:43.158 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.284558+0000", 2024-09-13T11:24:43.158 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.6899000000000001e-05 2024-09-13T11:24:43.158 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.158 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.158 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:43.159 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.626172+0000", 2024-09-13T11:24:43.159 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.34161370800000002 2024-09-13T11:24:43.159 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.159 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.159 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:43.159 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.626187+0000", 2024-09-13T11:24:43.159 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.485e-05 2024-09-13T11:24:43.159 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.159 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.159 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:43.159 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.626245+0000", 2024-09-13T11:24:43.159 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.7998000000000003e-05 2024-09-13T11:24:43.160 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.160 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.160 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:24:43.160 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.626260+0000", 2024-09-13T11:24:43.160 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5429000000000001e-05 2024-09-13T11:24:43.160 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.160 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.160 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:24:43.160 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.627924+0000", 2024-09-13T11:24:43.160 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0016634060000000001 2024-09-13T11:24:43.160 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.161 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.161 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:24:43.161 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.628021+0000", 2024-09-13T11:24:43.161 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.7594000000000001e-05 2024-09-13T11:24:43.161 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:43.161 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:43.161 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:43.161 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.162 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.162 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPush(3.2b 121/119 [PushOp(3:d53a23e8:::benchmark_data_smithi177_31116_object5366:head, version: 45'342, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:d53a23e8:::benchmark_data_smithi177_31116_object5366:head@45'342, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:24:43.162 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:27.268731+0000", 2024-09-13T11:24:43.162 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 15.818392606, 2024-09-13T11:24:43.162 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.72228345000000005, 2024-09-13T11:24:43.162 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:43.162 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:43.162 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:24:43.162 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:43.162 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.162 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:43.163 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:27.268731+0000", 2024-09-13T11:24:43.163 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:43.163 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.163 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.163 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:43.163 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:27.268731+0000", 2024-09-13T11:24:43.163 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:43.163 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.163 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.163 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:43.163 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:27.268733+0000", 2024-09-13T11:24:43.164 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.9740000000000001e-06 2024-09-13T11:24:43.164 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.164 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.164 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:43.164 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:27.268778+0000", 2024-09-13T11:24:43.164 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.4679999999999999e-05 2024-09-13T11:24:43.164 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.164 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.164 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:43.164 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:27.988594+0000", 2024-09-13T11:24:43.164 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.71981667299999996 2024-09-13T11:24:43.164 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.165 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.165 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:43.165 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:27.988652+0000", 2024-09-13T11:24:43.165 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.7640999999999999e-05 2024-09-13T11:24:43.165 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.165 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.165 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:43.165 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:27.988889+0000", 2024-09-13T11:24:43.165 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00023706500000000001 2024-09-13T11:24:43.165 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.165 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.166 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:24:43.166 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:27.988933+0000", 2024-09-13T11:24:43.166 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.3865000000000001e-05 2024-09-13T11:24:43.166 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.166 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.166 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:24:43.166 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:27.991015+0000", 2024-09-13T11:24:43.166 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.002081552 2024-09-13T11:24:43.166 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:43.166 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:43.166 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:43.166 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.167 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.167 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPush(3.2b 124/119 [PushOp(3:d64c8aaf:::benchmark_data_smithi177_31116_object31700:head, version: 59'1991, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:d64c8aaf:::benchmark_data_smithi177_31116_object31700:head@59'1991, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:24:43.167 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:31.579304+0000", 2024-09-13T11:24:43.167 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 11.507819905, 2024-09-13T11:24:43.167 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.14279340800000001, 2024-09-13T11:24:43.167 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:43.167 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:43.167 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:24:43.167 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:43.168 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.168 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:43.168 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:31.579304+0000", 2024-09-13T11:24:43.168 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:43.168 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.168 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.168 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:43.168 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:31.579304+0000", 2024-09-13T11:24:43.168 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:43.168 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.169 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.169 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:43.169 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:31.579305+0000", 2024-09-13T11:24:43.169 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5090000000000001e-06 2024-09-13T11:24:43.169 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.169 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.169 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:43.169 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:31.579367+0000", 2024-09-13T11:24:43.169 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.1279999999999996e-05 2024-09-13T11:24:43.170 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.170 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.170 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:43.170 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:31.721234+0000", 2024-09-13T11:24:43.170 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.141867883 2024-09-13T11:24:43.170 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.170 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.170 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:43.170 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:31.721253+0000", 2024-09-13T11:24:43.170 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.8725000000000001e-05 2024-09-13T11:24:43.171 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.171 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.171 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:43.171 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:31.721332+0000", 2024-09-13T11:24:43.171 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.8808999999999997e-05 2024-09-13T11:24:43.171 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.171 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.172 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:24:43.172 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:31.721345+0000", 2024-09-13T11:24:43.172 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3414999999999999e-05 2024-09-13T11:24:43.172 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.172 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.172 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:24:43.172 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:31.722097+0000", 2024-09-13T11:24:43.172 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00075178699999999996 2024-09-13T11:24:43.172 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:43.172 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:43.173 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:43.173 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.173 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.173 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:113511 3.2b e124/119 3:d74f5e33:::benchmark_data_smithi177_31116_object113510:head v 124'5156, mlcod=60'2020)", 2024-09-13T11:24:43.173 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:32.032517+0000", 2024-09-13T11:24:43.173 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 11.054606741000001, 2024-09-13T11:24:43.173 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.13874039699999999, 2024-09-13T11:24:43.173 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:43.173 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:43.173 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:43.174 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:43.174 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.174 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:43.174 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:32.032517+0000", 2024-09-13T11:24:43.174 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:43.174 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.174 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.174 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:43.175 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:32.032517+0000", 2024-09-13T11:24:43.175 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:43.175 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.175 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.175 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:43.175 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:32.032518+0000", 2024-09-13T11:24:43.175 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3060000000000001e-06 2024-09-13T11:24:43.175 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.175 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.175 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:43.175 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:32.032545+0000", 2024-09-13T11:24:43.175 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.6800000000000001e-05 2024-09-13T11:24:43.176 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.176 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.176 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:43.176 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:32.169285+0000", 2024-09-13T11:24:43.176 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.13673950800000001 2024-09-13T11:24:43.176 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.176 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.176 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:43.176 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:32.169291+0000", 2024-09-13T11:24:43.176 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.934e-06 2024-09-13T11:24:43.177 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.177 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.177 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:43.177 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:32.169381+0000", 2024-09-13T11:24:43.177 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.9860999999999999e-05 2024-09-13T11:24:43.177 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.177 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.177 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:24:43.177 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:32.169406+0000", 2024-09-13T11:24:43.177 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.5052e-05 2024-09-13T11:24:43.177 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.178 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.178 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:24:43.178 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:32.171218+0000", 2024-09-13T11:24:43.178 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0018111659999999999 2024-09-13T11:24:43.178 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.178 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.178 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:24:43.178 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:32.171257+0000", 2024-09-13T11:24:43.178 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.977e-05 2024-09-13T11:24:43.178 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:43.178 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:43.179 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:43.179 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.179 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.179 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPush(3.2b 124/119 [PushOp(3:d70bd127:::benchmark_data_smithi177_31116_object32365:head, version: 60'2034, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:d70bd127:::benchmark_data_smithi177_31116_object32365:head@60'2034, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:24:43.179 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:32.037338+0000", 2024-09-13T11:24:43.179 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 11.049785333000001, 2024-09-13T11:24:43.179 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.24358922699999999, 2024-09-13T11:24:43.179 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:43.179 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:43.179 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:24:43.179 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:43.179 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.180 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:43.180 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:32.037338+0000", 2024-09-13T11:24:43.180 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:43.180 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.180 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.180 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:43.180 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:32.037338+0000", 2024-09-13T11:24:43.180 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:43.180 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.180 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.180 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:43.180 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:32.037340+0000", 2024-09-13T11:24:43.181 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3790000000000001e-06 2024-09-13T11:24:43.181 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.181 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.181 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:43.181 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:32.037411+0000", 2024-09-13T11:24:43.181 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.1254000000000006e-05 2024-09-13T11:24:43.181 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.181 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.181 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:43.181 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:32.277467+0000", 2024-09-13T11:24:43.181 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.24005563199999999 2024-09-13T11:24:43.181 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.182 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.182 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:43.182 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:32.277494+0000", 2024-09-13T11:24:43.182 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.7120000000000001e-05 2024-09-13T11:24:43.182 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.182 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.182 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:43.182 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:32.279860+0000", 2024-09-13T11:24:43.182 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0023667559999999998 2024-09-13T11:24:43.182 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.183 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.183 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:24:43.183 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:32.279883+0000", 2024-09-13T11:24:43.183 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.2138999999999999e-05 2024-09-13T11:24:43.183 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.183 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.183 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:24:43.183 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:32.280928+0000", 2024-09-13T11:24:43.183 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0010449470000000001 2024-09-13T11:24:43.183 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:43.183 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:43.183 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:43.184 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.184 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.184 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPush(3.2b 125/119 [PushOp(3:d4c5ede9:::benchmark_data_smithi177_31116_object33277:head, version: 60'2092, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:d4c5ede9:::benchmark_data_smithi177_31116_object33277:head@60'2092, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:24:43.184 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:32.403929+0000", 2024-09-13T11:24:43.184 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 10.683194883000001, 2024-09-13T11:24:43.184 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.194592864, 2024-09-13T11:24:43.184 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:43.184 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:43.184 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:24:43.184 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:43.184 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.184 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:43.185 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:32.403929+0000", 2024-09-13T11:24:43.185 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:43.185 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.185 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.185 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:43.185 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:32.403929+0000", 2024-09-13T11:24:43.185 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:43.185 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.185 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.185 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:43.185 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:32.403930+0000", 2024-09-13T11:24:43.185 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.6729999999999999e-06 2024-09-13T11:24:43.186 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.186 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.186 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:43.186 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:32.403979+0000", 2024-09-13T11:24:43.186 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.8242000000000003e-05 2024-09-13T11:24:43.186 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.186 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.186 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:43.186 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:32.597445+0000", 2024-09-13T11:24:43.186 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.19346676500000001 2024-09-13T11:24:43.186 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.186 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.186 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:43.187 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:32.597455+0000", 2024-09-13T11:24:43.187 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.2939999999999994e-06 2024-09-13T11:24:43.187 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.187 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.187 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:43.187 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:32.597514+0000", 2024-09-13T11:24:43.187 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.9509000000000002e-05 2024-09-13T11:24:43.187 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.187 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.187 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:24:43.187 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:32.597522+0000", 2024-09-13T11:24:43.188 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.0350000000000001e-06 2024-09-13T11:24:43.188 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.188 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.188 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:24:43.188 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:32.598522+0000", 2024-09-13T11:24:43.188 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00099934599999999991 2024-09-13T11:24:43.188 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:43.188 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:43.188 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:43.188 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.188 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.189 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPush(3.2b 128/119 [PushOp(3:d6ff590f:::benchmark_data_smithi177_31116_object64581:head, version: 90'3750, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:d6ff590f:::benchmark_data_smithi177_31116_object64581:head@90'3750, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:24:43.189 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:37.982062+0000", 2024-09-13T11:24:43.189 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 5.1050619839999998, 2024-09-13T11:24:43.189 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.28899675200000002, 2024-09-13T11:24:43.189 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:43.189 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:43.189 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:24:43.189 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:43.189 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.189 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:43.189 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:37.982062+0000", 2024-09-13T11:24:43.190 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:43.190 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.190 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.190 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:43.190 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:37.982062+0000", 2024-09-13T11:24:43.190 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:43.190 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.190 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.190 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:43.190 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:37.982064+0000", 2024-09-13T11:24:43.190 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.2259999999999999e-06 2024-09-13T11:24:43.190 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.191 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.191 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:43.191 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:37.982114+0000", 2024-09-13T11:24:43.191 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.0330000000000001e-05 2024-09-13T11:24:43.191 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.191 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.191 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:43.191 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:38.269983+0000", 2024-09-13T11:24:43.191 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.28786869999999998 2024-09-13T11:24:43.191 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.192 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.192 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:43.192 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:38.270000+0000", 2024-09-13T11:24:43.192 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.6977999999999999e-05 2024-09-13T11:24:43.192 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.192 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.192 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:43.192 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:38.270092+0000", 2024-09-13T11:24:43.192 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.221e-05 2024-09-13T11:24:43.192 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.193 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.193 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:24:43.193 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:38.270103+0000", 2024-09-13T11:24:43.193 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.1253999999999999e-05 2024-09-13T11:24:43.193 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.193 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.193 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:24:43.193 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:38.271058+0000", 2024-09-13T11:24:43.193 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00095505399999999999 2024-09-13T11:24:43.193 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:43.193 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:43.193 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:43.193 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:43.194 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:43.194 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-09-13T11:24:43.195 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-09-13T11:24:43.196 INFO:tasks.ceph.osd.5.smithi092.stderr:2024-09-13T11:24:43.190+0000 7f6f0b705640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26346) UID: 0 2024-09-13T11:24:43.296 INFO:tasks.ceph.osd.2.smithi162.stderr:2024-09-13T11:24:43.292+0000 7fe92b35a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26313) UID: 0 2024-09-13T11:24:43.324 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-13T11:24:43.367 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-09-13T11:24:43.367 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [ 2024-09-13T11:24:43.367 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.368 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:126441 3.16 3:6c442f66:::benchmark_data_smithi177_31116_object126440:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e130)", 2024-09-13T11:24:43.368 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:43.343908+0000", 2024-09-13T11:24:43.368 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 0.022730796000000001, 2024-09-13T11:24:43.368 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.022757936999999999, 2024-09-13T11:24:43.368 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:43.368 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:43.368 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "waiting for sub ops", 2024-09-13T11:24:43.368 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:24:43.369 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:24:43.369 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:43.369 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 126441 2024-09-13T11:24:43.369 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.369 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:43.369 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.369 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:43.369 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:43.343908+0000", 2024-09-13T11:24:43.370 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:43.370 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.370 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.370 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:43.370 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:43.343908+0000", 2024-09-13T11:24:43.370 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:43.370 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.370 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.370 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:43.370 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:43.343910+0000", 2024-09-13T11:24:43.370 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.1119999999999999e-06 2024-09-13T11:24:43.370 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.371 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.371 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:43.371 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:43.343936+0000", 2024-09-13T11:24:43.371 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.5622999999999998e-05 2024-09-13T11:24:43.371 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.371 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.371 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:43.371 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:43.343951+0000", 2024-09-13T11:24:43.371 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.4596000000000001e-05 2024-09-13T11:24:43.371 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.372 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.372 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:43.372 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:43.343960+0000", 2024-09-13T11:24:43.372 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.5130000000000002e-06 2024-09-13T11:24:43.372 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.372 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.372 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:43.372 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:43.344035+0000", 2024-09-13T11:24:43.372 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.5377999999999996e-05 2024-09-13T11:24:43.372 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.372 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.372 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:24:43.372 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:43.344204+0000", 2024-09-13T11:24:43.373 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00016852600000000001 2024-09-13T11:24:43.373 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.373 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.373 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 15", 2024-09-13T11:24:43.373 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:43.344357+0000", 2024-09-13T11:24:43.373 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000153015 2024-09-13T11:24:43.373 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.374 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.374 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:24:43.374 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:43.348741+0000", 2024-09-13T11:24:43.374 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.004383634 2024-09-13T11:24:43.374 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:43.374 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:43.374 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:43.374 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.374 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.374 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:126459 3.13 3:cf74a2a2:::benchmark_data_smithi177_31116_object126458:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e130)", 2024-09-13T11:24:43.374 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:43.362337+0000", 2024-09-13T11:24:43.374 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 0.0043024630000000003, 2024-09-13T11:24:43.375 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0044127410000000004, 2024-09-13T11:24:43.375 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:43.375 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:43.375 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "waiting for sub ops", 2024-09-13T11:24:43.375 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:24:43.375 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:24:43.375 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:43.375 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 126459 2024-09-13T11:24:43.375 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.375 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:43.375 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.375 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:43.375 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:43.362337+0000", 2024-09-13T11:24:43.376 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:43.376 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.376 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.376 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:43.376 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:43.362337+0000", 2024-09-13T11:24:43.376 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:43.376 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.376 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.376 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:43.376 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:43.362339+0000", 2024-09-13T11:24:43.376 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.0329999999999998e-06 2024-09-13T11:24:43.376 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.376 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.377 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:43.377 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:43.362359+0000", 2024-09-13T11:24:43.377 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.0733e-05 2024-09-13T11:24:43.377 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.377 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.377 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:43.377 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:43.362374+0000", 2024-09-13T11:24:43.377 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5073e-05 2024-09-13T11:24:43.377 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.377 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.377 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:43.377 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:43.362384+0000", 2024-09-13T11:24:43.377 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.3430000000000002e-06 2024-09-13T11:24:43.378 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.378 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.378 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:43.378 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:43.362456+0000", 2024-09-13T11:24:43.378 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.2237000000000007e-05 2024-09-13T11:24:43.378 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.378 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.378 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:24:43.378 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:43.362611+0000", 2024-09-13T11:24:43.378 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00015542900000000001 2024-09-13T11:24:43.378 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.378 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.378 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 7", 2024-09-13T11:24:43.379 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:43.362752+0000", 2024-09-13T11:24:43.379 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000140858 2024-09-13T11:24:43.379 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.379 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.379 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:43.379 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:43.366245+0000", 2024-09-13T11:24:43.379 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0034931440000000001 2024-09-13T11:24:43.379 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:43.379 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:43.379 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:43.379 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.379 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.379 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:126464 3.16 3:6eae51a9:::benchmark_data_smithi177_31116_object126463:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e130)", 2024-09-13T11:24:43.380 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:43.365743+0000", 2024-09-13T11:24:43.380 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 0.00089637899999999999, 2024-09-13T11:24:43.380 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001080821, 2024-09-13T11:24:43.380 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:43.380 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:43.380 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "waiting for sub ops", 2024-09-13T11:24:43.380 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:24:43.380 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:24:43.380 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:43.380 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 126464 2024-09-13T11:24:43.380 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.380 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:43.380 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.380 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:43.381 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:43.365743+0000", 2024-09-13T11:24:43.381 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:43.381 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.381 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.381 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:43.381 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:43.365743+0000", 2024-09-13T11:24:43.381 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:43.381 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.381 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.381 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:43.381 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:43.365744+0000", 2024-09-13T11:24:43.381 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.6670000000000001e-06 2024-09-13T11:24:43.381 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.381 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.382 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:43.382 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:43.365765+0000", 2024-09-13T11:24:43.382 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.1072000000000001e-05 2024-09-13T11:24:43.382 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.382 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.382 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:43.382 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:43.365778+0000", 2024-09-13T11:24:43.382 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.2628e-05 2024-09-13T11:24:43.382 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.382 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.382 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:43.382 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:43.365786+0000", 2024-09-13T11:24:43.382 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.0390000000000005e-06 2024-09-13T11:24:43.382 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.383 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.383 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:43.383 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:43.365850+0000", 2024-09-13T11:24:43.383 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.3867000000000004e-05 2024-09-13T11:24:43.383 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.383 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.383 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:24:43.383 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:43.366004+0000", 2024-09-13T11:24:43.383 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00015369200000000001 2024-09-13T11:24:43.383 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.383 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.383 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 15", 2024-09-13T11:24:43.383 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:43.366159+0000", 2024-09-13T11:24:43.384 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00015499599999999999 2024-09-13T11:24:43.384 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:43.384 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:43.384 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:43.384 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:43.384 INFO:teuthology.orchestra.run.smithi177.stdout: ], 2024-09-13T11:24:43.384 INFO:teuthology.orchestra.run.smithi177.stdout: "num_ops": 3 2024-09-13T11:24:43.384 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-09-13T11:24:43.394 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-09-13T11:24:43.397 INFO:tasks.ceph.osd.10.smithi162.stderr:2024-09-13T11:24:43.392+0000 7fb59c9f0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26317) UID: 0 2024-09-13T11:24:43.497 INFO:tasks.ceph.osd.8.smithi001.stderr:2024-09-13T11:24:43.495+0000 7fb9b48ae640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26312) UID: 0 2024-09-13T11:24:43.560 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-09-13T11:24:43.560 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [], 2024-09-13T11:24:43.560 INFO:teuthology.orchestra.run.smithi177.stdout: "complaint_time": 30, 2024-09-13T11:24:43.560 INFO:teuthology.orchestra.run.smithi177.stdout: "num_blocked_ops": 0 2024-09-13T11:24:43.560 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-09-13T11:24:43.585 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-09-13T11:24:43.597 INFO:tasks.ceph.osd.5.smithi092.stderr:2024-09-13T11:24:43.594+0000 7f6f0b705640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26346) UID: 0 2024-09-13T11:24:43.693 INFO:teuthology.orchestra.run.smithi162.stdout:reshard success 2024-09-13T11:24:43.698 INFO:tasks.ceph.osd.2.smithi162.stderr:2024-09-13T11:24:43.696+0000 7fe92b35a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26313) UID: 0 2024-09-13T11:24:43.746 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-09-13T11:24:43.747 INFO:teuthology.orchestra.run.smithi177.stdout: "size": 20, 2024-09-13T11:24:43.747 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 600, 2024-09-13T11:24:43.747 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [ 2024-09-13T11:24:43.747 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.747 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:79 3.1 e32/30 3:841df5b5:::benchmark_data_smithi177_31116_object78:head v 32'6, mlcod=32'4)", 2024-09-13T11:24:43.747 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:22:27.057292+0000", 2024-09-13T11:24:43.747 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 136.687151083, 2024-09-13T11:24:43.747 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.89296786800000005, 2024-09-13T11:24:43.747 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:43.747 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:43.747 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:43.748 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:43.748 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.748 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:43.748 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.057292+0000", 2024-09-13T11:24:43.748 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:43.748 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.748 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.748 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:43.748 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.057292+0000", 2024-09-13T11:24:43.748 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:43.748 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.748 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.748 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:43.749 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.057293+0000", 2024-09-13T11:24:43.749 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.6109999999999999e-06 2024-09-13T11:24:43.749 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.749 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.749 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:43.749 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.057330+0000", 2024-09-13T11:24:43.749 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.6721999999999998e-05 2024-09-13T11:24:43.749 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.749 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.749 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:43.749 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.948197+0000", 2024-09-13T11:24:43.749 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.890867042 2024-09-13T11:24:43.749 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.749 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.750 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:43.750 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.948213+0000", 2024-09-13T11:24:43.750 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5826000000000001e-05 2024-09-13T11:24:43.750 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.750 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.750 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:43.750 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.948377+0000", 2024-09-13T11:24:43.750 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00016348600000000001 2024-09-13T11:24:43.750 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.750 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.750 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:24:43.750 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.948447+0000", 2024-09-13T11:24:43.750 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.0891999999999998e-05 2024-09-13T11:24:43.751 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.751 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.751 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:24:43.751 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.950150+0000", 2024-09-13T11:24:43.751 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001702329 2024-09-13T11:24:43.751 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.751 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.751 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:24:43.751 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.950260+0000", 2024-09-13T11:24:43.752 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00010996 2024-09-13T11:24:43.752 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:43.752 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:43.752 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:43.752 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.752 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.752 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:30828 3.f 3:f3a9c6a9:::benchmark_data_smithi177_31116_object30827:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e59)", 2024-09-13T11:24:43.752 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:09.503661+0000", 2024-09-13T11:24:43.752 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 94.240781733999995, 2024-09-13T11:24:43.752 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.26477667100000002, 2024-09-13T11:24:43.752 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:43.753 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:43.753 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:43.753 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:24:43.753 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:24:43.753 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:43.753 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 30828 2024-09-13T11:24:43.753 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.753 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:43.753 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.753 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:43.753 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.503661+0000", 2024-09-13T11:24:43.753 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:43.754 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.754 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.754 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:43.754 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.503661+0000", 2024-09-13T11:24:43.754 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:43.754 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.754 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.754 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:43.754 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.503663+0000", 2024-09-13T11:24:43.754 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.968e-06 2024-09-13T11:24:43.754 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.754 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.754 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:43.754 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.503687+0000", 2024-09-13T11:24:43.755 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.3983999999999999e-05 2024-09-13T11:24:43.755 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.755 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.755 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:43.755 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.503701+0000", 2024-09-13T11:24:43.755 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3416e-05 2024-09-13T11:24:43.755 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.755 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.755 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:43.755 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.503708+0000", 2024-09-13T11:24:43.755 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.9530000000000002e-06 2024-09-13T11:24:43.755 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.756 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.756 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:43.756 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.503759+0000", 2024-09-13T11:24:43.756 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.1075999999999997e-05 2024-09-13T11:24:43.756 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.756 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.756 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:24:43.756 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.503861+0000", 2024-09-13T11:24:43.756 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000102372 2024-09-13T11:24:43.756 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.756 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.756 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 12", 2024-09-13T11:24:43.757 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.503952+0000", 2024-09-13T11:24:43.757 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.1135000000000001e-05 2024-09-13T11:24:43.757 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.757 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.757 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:24:43.757 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.505340+0000", 2024-09-13T11:24:43.757 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0013877169999999999 2024-09-13T11:24:43.757 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.757 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.757 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:43.757 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.768281+0000", 2024-09-13T11:24:43.757 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.26294092400000002 2024-09-13T11:24:43.758 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.758 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.758 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:24:43.758 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.768369+0000", 2024-09-13T11:24:43.758 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.7818999999999994e-05 2024-09-13T11:24:43.758 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.758 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.758 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:24:43.758 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.768438+0000", 2024-09-13T11:24:43.758 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.9307000000000006e-05 2024-09-13T11:24:43.759 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:43.759 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:43.759 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:43.759 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.759 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.759 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:45783 3.16 3.17c10a76 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e69)", 2024-09-13T11:24:43.759 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:20.947774+0000", 2024-09-13T11:24:43.759 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 82.796668831999995, 2024-09-13T11:24:43.759 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.96682922400000004, 2024-09-13T11:24:43.759 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:43.759 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:43.759 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-13T11:24:43.759 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:24:43.760 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:24:43.760 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:43.760 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 45783 2024-09-13T11:24:43.760 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.760 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:43.760 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.760 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:43.760 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:20.947774+0000", 2024-09-13T11:24:43.760 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:43.760 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.761 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.761 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:43.761 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:20.947774+0000", 2024-09-13T11:24:43.761 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:43.761 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.761 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.761 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:43.761 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:20.947777+0000", 2024-09-13T11:24:43.761 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.57e-06 2024-09-13T11:24:43.761 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.761 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.761 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:43.761 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:20.947802+0000", 2024-09-13T11:24:43.762 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.5528999999999999e-05 2024-09-13T11:24:43.762 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.762 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.762 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:43.762 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:20.947813+0000", 2024-09-13T11:24:43.762 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0473e-05 2024-09-13T11:24:43.762 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.762 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.762 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:43.763 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:20.947821+0000", 2024-09-13T11:24:43.763 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.3440000000000001e-06 2024-09-13T11:24:43.763 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.763 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.763 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:43.763 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:20.947933+0000", 2024-09-13T11:24:43.763 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00011210200000000001 2024-09-13T11:24:43.763 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.763 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.763 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for peered", 2024-09-13T11:24:43.763 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:20.947942+0000", 2024-09-13T11:24:43.763 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.6610000000000006e-06 2024-09-13T11:24:43.763 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.764 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.764 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:43.764 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:21.914579+0000", 2024-09-13T11:24:43.764 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.96663672300000003 2024-09-13T11:24:43.764 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.764 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.764 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:24:43.764 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:21.914603+0000", 2024-09-13T11:24:43.764 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.4822000000000001e-05 2024-09-13T11:24:43.764 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:43.765 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:43.765 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:43.765 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.765 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.765 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:54625 3.1f 3:f812cdfb:::benchmark_data_smithi177_31116_object54624:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e75)", 2024-09-13T11:24:43.765 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:30.979327+0000", 2024-09-13T11:24:43.765 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 72.765116043000006, 2024-09-13T11:24:43.765 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.96163670899999998, 2024-09-13T11:24:43.765 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:43.765 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:43.765 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-13T11:24:43.766 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:24:43.766 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:24:43.766 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:43.766 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 54625 2024-09-13T11:24:43.766 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.766 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:43.766 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.766 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:43.766 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.979327+0000", 2024-09-13T11:24:43.766 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:43.766 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.767 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.767 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:43.767 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.979327+0000", 2024-09-13T11:24:43.767 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:43.767 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.767 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.767 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:43.767 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.979329+0000", 2024-09-13T11:24:43.767 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.686e-06 2024-09-13T11:24:43.767 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.768 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.768 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:43.768 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.979364+0000", 2024-09-13T11:24:43.768 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.5750000000000002e-05 2024-09-13T11:24:43.768 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.768 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.768 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:43.768 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.979408+0000", 2024-09-13T11:24:43.768 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.3795999999999999e-05 2024-09-13T11:24:43.768 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.768 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.768 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:43.769 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.979427+0000", 2024-09-13T11:24:43.769 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.9040999999999999e-05 2024-09-13T11:24:43.769 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.769 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.769 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:43.769 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.122291+0000", 2024-09-13T11:24:43.769 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.14286373599999999 2024-09-13T11:24:43.769 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.769 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.769 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for peered", 2024-09-13T11:24:43.769 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.122298+0000", 2024-09-13T11:24:43.769 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.3719999999999999e-06 2024-09-13T11:24:43.769 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.769 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.770 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:43.770 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.940341+0000", 2024-09-13T11:24:43.770 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.81804249699999998 2024-09-13T11:24:43.770 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.770 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.770 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:24:43.770 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.940964+0000", 2024-09-13T11:24:43.770 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00062283100000000001 2024-09-13T11:24:43.770 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:43.770 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:43.770 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:43.770 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.771 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.771 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:54649 3.1c 3:3a457a93:::benchmark_data_smithi177_31116_object54648:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e75)", 2024-09-13T11:24:43.771 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:30.979460+0000", 2024-09-13T11:24:43.771 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 72.764982853000006, 2024-09-13T11:24:43.771 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.95579424000000002, 2024-09-13T11:24:43.771 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:43.771 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:43.771 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-13T11:24:43.771 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:24:43.771 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:24:43.771 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:43.771 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 54649 2024-09-13T11:24:43.771 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.772 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:43.772 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.772 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:43.772 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.979460+0000", 2024-09-13T11:24:43.772 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:43.772 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.772 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.772 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:43.772 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.979460+0000", 2024-09-13T11:24:43.772 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:43.772 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.772 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.772 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:43.773 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.979462+0000", 2024-09-13T11:24:43.773 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5799999999999999e-06 2024-09-13T11:24:43.773 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.773 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.773 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:43.773 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.979524+0000", 2024-09-13T11:24:43.773 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.2521999999999995e-05 2024-09-13T11:24:43.773 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.773 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.773 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:43.773 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.979561+0000", 2024-09-13T11:24:43.773 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.6349e-05 2024-09-13T11:24:43.773 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.774 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.774 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:43.774 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.979571+0000", 2024-09-13T11:24:43.774 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0457e-05 2024-09-13T11:24:43.774 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.774 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.774 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:43.774 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.121029+0000", 2024-09-13T11:24:43.774 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.14145766200000001 2024-09-13T11:24:43.774 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.775 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.775 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for peered", 2024-09-13T11:24:43.775 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.121039+0000", 2024-09-13T11:24:43.775 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0084e-05 2024-09-13T11:24:43.775 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.775 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.776 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:43.776 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.935157+0000", 2024-09-13T11:24:43.776 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.81411807899999999 2024-09-13T11:24:43.776 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.776 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.776 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:24:43.776 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.935254+0000", 2024-09-13T11:24:43.777 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.7507e-05 2024-09-13T11:24:43.777 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:43.777 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:43.777 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:43.777 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.777 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.777 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:54654 3.1c 3:3f0bb31a:::benchmark_data_smithi177_31116_object54653:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e75)", 2024-09-13T11:24:43.777 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:30.979665+0000", 2024-09-13T11:24:43.777 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 72.764777687999995, 2024-09-13T11:24:43.777 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.95623908099999999, 2024-09-13T11:24:43.778 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:43.778 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:43.778 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-13T11:24:43.778 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:24:43.778 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:24:43.778 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:43.778 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 54654 2024-09-13T11:24:43.778 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.778 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:43.778 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.779 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:43.779 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.979665+0000", 2024-09-13T11:24:43.779 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:43.779 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.779 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.779 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:43.779 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.979665+0000", 2024-09-13T11:24:43.779 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:43.779 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.780 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.780 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:43.780 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.979667+0000", 2024-09-13T11:24:43.780 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3540000000000001e-06 2024-09-13T11:24:43.780 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.780 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.780 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:43.780 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.979703+0000", 2024-09-13T11:24:43.780 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.6758000000000003e-05 2024-09-13T11:24:43.780 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.780 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.780 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:43.781 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.979714+0000", 2024-09-13T11:24:43.781 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0804e-05 2024-09-13T11:24:43.781 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.781 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.781 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:43.781 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.979724+0000", 2024-09-13T11:24:43.781 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0040999999999999e-05 2024-09-13T11:24:43.781 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.781 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.781 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:43.782 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.121061+0000", 2024-09-13T11:24:43.782 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.141336713 2024-09-13T11:24:43.782 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.782 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.782 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for peered", 2024-09-13T11:24:43.782 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.121067+0000", 2024-09-13T11:24:43.782 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.4150000000000004e-06 2024-09-13T11:24:43.782 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.782 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.782 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:43.782 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.935776+0000", 2024-09-13T11:24:43.782 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.81470863100000002 2024-09-13T11:24:43.782 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.782 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.783 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:24:43.783 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.935904+0000", 2024-09-13T11:24:43.783 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00012836499999999999 2024-09-13T11:24:43.783 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:43.783 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:43.783 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:43.783 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.783 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.783 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPushReply(3.11 75/70 [PushReplyOp(3:8df10771:::benchmark_data_smithi177_31116_object24734:head)])", 2024-09-13T11:24:43.783 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:31.149798+0000", 2024-09-13T11:24:43.783 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 72.594644677999995, 2024-09-13T11:24:43.783 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.32402702500000002, 2024-09-13T11:24:43.783 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:43.784 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:43.784 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-13T11:24:43.784 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:43.784 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.784 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:43.784 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.149798+0000", 2024-09-13T11:24:43.784 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:43.784 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.784 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.784 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:43.784 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.149798+0000", 2024-09-13T11:24:43.785 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:43.785 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.785 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.785 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:43.785 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.149799+0000", 2024-09-13T11:24:43.785 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.5700000000000001e-07 2024-09-13T11:24:43.785 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.785 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.785 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:43.785 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.149803+0000", 2024-09-13T11:24:43.785 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.0040000000000001e-06 2024-09-13T11:24:43.785 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.785 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.786 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:43.786 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.472172+0000", 2024-09-13T11:24:43.786 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.32236856699999999 2024-09-13T11:24:43.786 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.786 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.786 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:43.786 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.472187+0000", 2024-09-13T11:24:43.786 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5016e-05 2024-09-13T11:24:43.786 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.786 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.786 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:43.786 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.472302+0000", 2024-09-13T11:24:43.786 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000115121 2024-09-13T11:24:43.786 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.787 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.787 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:24:43.787 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.473825+0000", 2024-09-13T11:24:43.787 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0015234599999999999 2024-09-13T11:24:43.787 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:43.787 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:43.787 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:43.787 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.787 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.787 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:54668 3.1c 3:3d2c5e40:::benchmark_data_smithi177_31116_object54667:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e75)", 2024-09-13T11:24:43.787 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:31.245525+0000", 2024-09-13T11:24:43.787 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 72.498918420999999, 2024-09-13T11:24:43.787 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.69828973000000005, 2024-09-13T11:24:43.788 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:43.788 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:43.788 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:43.788 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:24:43.788 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:24:43.788 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:43.788 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 54668 2024-09-13T11:24:43.788 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.788 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:43.788 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.788 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:43.788 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.245525+0000", 2024-09-13T11:24:43.789 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:43.789 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.789 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.789 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:43.789 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.245525+0000", 2024-09-13T11:24:43.789 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:43.789 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.789 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.789 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:43.789 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.245526+0000", 2024-09-13T11:24:43.789 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.719e-06 2024-09-13T11:24:43.789 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.789 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.789 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:43.790 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.245547+0000", 2024-09-13T11:24:43.790 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.1084999999999999e-05 2024-09-13T11:24:43.790 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.790 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.790 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:43.790 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.245558+0000", 2024-09-13T11:24:43.790 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0934000000000001e-05 2024-09-13T11:24:43.790 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.790 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.790 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:43.790 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.245564+0000", 2024-09-13T11:24:43.790 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.6269999999999997e-06 2024-09-13T11:24:43.790 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.791 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.791 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:43.791 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.245620+0000", 2024-09-13T11:24:43.791 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.6289999999999998e-05 2024-09-13T11:24:43.791 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.791 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.791 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for peered", 2024-09-13T11:24:43.791 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.245632+0000", 2024-09-13T11:24:43.791 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.1748e-05 2024-09-13T11:24:43.791 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.791 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.791 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:43.791 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.935976+0000", 2024-09-13T11:24:43.791 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.69034455299999997 2024-09-13T11:24:43.791 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.792 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.792 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:24:43.792 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.936302+0000", 2024-09-13T11:24:43.792 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000325397 2024-09-13T11:24:43.792 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.792 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.792 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 13", 2024-09-13T11:24:43.792 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.936459+0000", 2024-09-13T11:24:43.792 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00015754300000000001 2024-09-13T11:24:43.792 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.792 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.792 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:24:43.792 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.938813+0000", 2024-09-13T11:24:43.793 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0023530999999999999 2024-09-13T11:24:43.793 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.793 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.793 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:43.793 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.943695+0000", 2024-09-13T11:24:43.793 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.004882506 2024-09-13T11:24:43.793 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.793 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.793 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:24:43.793 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.943767+0000", 2024-09-13T11:24:43.793 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.2454999999999997e-05 2024-09-13T11:24:43.793 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.793 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.794 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:24:43.794 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.943814+0000", 2024-09-13T11:24:43.794 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.6773000000000001e-05 2024-09-13T11:24:43.794 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:43.794 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:43.794 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:43.794 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.794 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.794 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPush(3.16 84/70 [PushOp(3:6f7be541:::benchmark_data_smithi177_31116_object43332:head, version: 66'2788, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:6f7be541:::benchmark_data_smithi177_31116_object43332:head@66'2788, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:24:43.794 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:41.207515+0000", 2024-09-13T11:24:43.794 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 62.536927650000003, 2024-09-13T11:24:43.794 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.52643771500000003, 2024-09-13T11:24:43.795 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:43.795 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:43.795 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:24:43.795 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:43.795 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.795 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:43.795 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.207515+0000", 2024-09-13T11:24:43.795 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:43.795 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.795 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.795 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:43.795 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.207515+0000", 2024-09-13T11:24:43.795 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:43.795 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.796 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.796 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:43.796 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.207516+0000", 2024-09-13T11:24:43.796 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.096e-06 2024-09-13T11:24:43.796 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.796 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.796 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:43.796 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.207717+0000", 2024-09-13T11:24:43.796 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000200199 2024-09-13T11:24:43.796 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.796 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.796 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:43.796 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.732785+0000", 2024-09-13T11:24:43.796 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.52506869599999995 2024-09-13T11:24:43.797 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.797 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.797 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:43.797 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.732801+0000", 2024-09-13T11:24:43.797 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5305999999999999e-05 2024-09-13T11:24:43.797 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.797 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.797 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:43.797 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.732889+0000", 2024-09-13T11:24:43.797 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.8157000000000003e-05 2024-09-13T11:24:43.797 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.798 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.798 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:24:43.798 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.732901+0000", 2024-09-13T11:24:43.798 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.1866e-05 2024-09-13T11:24:43.798 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.798 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.798 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:24:43.798 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.733953+0000", 2024-09-13T11:24:43.798 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001052395 2024-09-13T11:24:43.798 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:43.798 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:43.798 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:43.799 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.799 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.799 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPushReply(3.1f 85/78 [PushReplyOp(3:fd221d21:::benchmark_data_smithi177_31116_object7639:head)])", 2024-09-13T11:24:43.799 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:42.198715+0000", 2024-09-13T11:24:43.799 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 61.545728081, 2024-09-13T11:24:43.799 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.48930391099999998, 2024-09-13T11:24:43.799 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:43.799 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:43.799 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-13T11:24:43.799 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:43.799 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.799 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:43.799 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.198715+0000", 2024-09-13T11:24:43.799 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:43.800 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.800 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.800 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:43.800 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.198715+0000", 2024-09-13T11:24:43.800 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:43.800 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.800 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.800 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:43.800 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.198716+0000", 2024-09-13T11:24:43.800 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.3500000000000005e-07 2024-09-13T11:24:43.800 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.800 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.801 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:43.801 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.198719+0000", 2024-09-13T11:24:43.801 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.4539999999999999e-06 2024-09-13T11:24:43.801 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.801 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.801 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:43.801 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.684732+0000", 2024-09-13T11:24:43.801 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.48601262099999998 2024-09-13T11:24:43.801 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.801 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.801 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:43.801 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.684746+0000", 2024-09-13T11:24:43.802 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.4518999999999999e-05 2024-09-13T11:24:43.802 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.802 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.802 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:43.802 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.684911+0000", 2024-09-13T11:24:43.802 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000164728 2024-09-13T11:24:43.802 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.802 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.802 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:24:43.802 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.688019+0000", 2024-09-13T11:24:43.802 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0031077539999999999 2024-09-13T11:24:43.802 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:43.802 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:43.802 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:43.803 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.803 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.803 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPushReply(3.1c 85/77 [PushReplyOp(3:3a669437:::benchmark_data_smithi177_31116_object25280:head)])", 2024-09-13T11:24:43.803 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:42.352466+0000", 2024-09-13T11:24:43.803 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 61.391977187000002, 2024-09-13T11:24:43.803 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.36670725300000001, 2024-09-13T11:24:43.803 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:43.803 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:43.803 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-13T11:24:43.803 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:43.803 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.803 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:43.803 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.352466+0000", 2024-09-13T11:24:43.804 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:43.804 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.804 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.804 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:43.804 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.352466+0000", 2024-09-13T11:24:43.804 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:43.804 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.804 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.804 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:43.804 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.352467+0000", 2024-09-13T11:24:43.804 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.1429999999999999e-06 2024-09-13T11:24:43.804 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.805 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.805 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:43.805 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.352474+0000", 2024-09-13T11:24:43.805 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.6100000000000002e-06 2024-09-13T11:24:43.805 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.805 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.805 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:43.805 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.717011+0000", 2024-09-13T11:24:43.805 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.36453790200000002 2024-09-13T11:24:43.805 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.805 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.805 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:43.805 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.717032+0000", 2024-09-13T11:24:43.805 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.0999e-05 2024-09-13T11:24:43.806 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.806 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.806 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:43.806 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.717162+0000", 2024-09-13T11:24:43.806 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000129159 2024-09-13T11:24:43.806 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.806 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.806 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:24:43.806 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.719173+0000", 2024-09-13T11:24:43.806 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0020114400000000002 2024-09-13T11:24:43.806 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:43.806 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:43.806 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:43.806 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.807 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.807 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPush(3.16 85/70 [PushOp(3:6aef2f27:::benchmark_data_smithi177_31116_object44090:head, version: 68'2844, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:6aef2f27:::benchmark_data_smithi177_31116_object44090:head@68'2844, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:24:43.807 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:42.574221+0000", 2024-09-13T11:24:43.807 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 61.170222078999998, 2024-09-13T11:24:43.807 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.25345585700000001, 2024-09-13T11:24:43.807 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:43.807 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:43.807 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:24:43.807 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:43.807 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.807 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:43.807 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.574221+0000", 2024-09-13T11:24:43.807 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:43.807 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.808 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.808 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:43.808 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.574221+0000", 2024-09-13T11:24:43.808 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:43.808 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.808 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.808 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:43.808 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.574252+0000", 2024-09-13T11:24:43.808 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.0973000000000003e-05 2024-09-13T11:24:43.808 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.808 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.808 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:43.809 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.574434+0000", 2024-09-13T11:24:43.809 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000182265 2024-09-13T11:24:43.809 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.809 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.809 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:43.809 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.825391+0000", 2024-09-13T11:24:43.809 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.250957023 2024-09-13T11:24:43.809 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.809 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.809 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:43.809 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.825442+0000", 2024-09-13T11:24:43.809 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.1267999999999998e-05 2024-09-13T11:24:43.809 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.809 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.810 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:43.810 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.825692+0000", 2024-09-13T11:24:43.810 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00024924700000000002 2024-09-13T11:24:43.810 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.810 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.810 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:24:43.810 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.825715+0000", 2024-09-13T11:24:43.810 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.3176e-05 2024-09-13T11:24:43.810 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.810 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.810 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:24:43.810 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.827677+0000", 2024-09-13T11:24:43.810 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0019619049999999999 2024-09-13T11:24:43.811 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:43.811 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:43.811 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:43.811 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.811 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.811 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPushReply(3.1f 85/78 [PushReplyOp(3:fecb0d6a:::benchmark_data_smithi177_31116_object7644:head)])", 2024-09-13T11:24:43.811 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:42.705351+0000", 2024-09-13T11:24:43.811 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 61.039092345, 2024-09-13T11:24:43.811 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.290549681, 2024-09-13T11:24:43.811 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:43.811 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:43.811 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-13T11:24:43.811 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:43.812 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.812 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:43.812 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.705351+0000", 2024-09-13T11:24:43.812 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:43.812 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.812 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.812 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:43.812 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.705351+0000", 2024-09-13T11:24:43.812 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:43.813 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.813 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.813 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:43.813 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.705353+0000", 2024-09-13T11:24:43.813 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.4169999999999999e-06 2024-09-13T11:24:43.813 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.813 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.813 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:43.813 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.705365+0000", 2024-09-13T11:24:43.813 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.1894e-05 2024-09-13T11:24:43.814 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.814 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.814 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:43.814 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.992081+0000", 2024-09-13T11:24:43.814 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.28671641199999998 2024-09-13T11:24:43.814 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.814 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.814 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:43.814 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.992111+0000", 2024-09-13T11:24:43.814 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.9427000000000001e-05 2024-09-13T11:24:43.814 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.814 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.815 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:43.815 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.993573+0000", 2024-09-13T11:24:43.815 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0014623590000000001 2024-09-13T11:24:43.815 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.815 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.815 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:24:43.815 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.995900+0000", 2024-09-13T11:24:43.815 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0023271720000000002 2024-09-13T11:24:43.815 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:43.815 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:43.815 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:43.815 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.816 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.816 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPushReply(3.21 89/78 [PushReplyOp(3:8534d087:::benchmark_data_smithi177_31116_object5474:head)])", 2024-09-13T11:24:43.816 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:46.632834+0000", 2024-09-13T11:24:43.816 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 57.111608904999997, 2024-09-13T11:24:43.816 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.324805556, 2024-09-13T11:24:43.816 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:43.816 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:43.816 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-13T11:24:43.816 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:43.816 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.816 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:43.816 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.632834+0000", 2024-09-13T11:24:43.816 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:43.817 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.817 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.817 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:43.817 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.632834+0000", 2024-09-13T11:24:43.817 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:43.817 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.817 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.817 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:43.817 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.632835+0000", 2024-09-13T11:24:43.819 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.4270000000000001e-06 2024-09-13T11:24:43.820 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.820 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.820 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:43.820 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.632840+0000", 2024-09-13T11:24:43.820 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.7809999999999996e-06 2024-09-13T11:24:43.820 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.820 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.820 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:43.820 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.956282+0000", 2024-09-13T11:24:43.820 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.323441639 2024-09-13T11:24:43.820 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.820 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.820 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:43.821 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.956293+0000", 2024-09-13T11:24:43.821 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.083e-05 2024-09-13T11:24:43.821 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.821 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.821 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:43.821 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.956395+0000", 2024-09-13T11:24:43.821 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00010184200000000001 2024-09-13T11:24:43.821 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.821 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.821 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:24:43.821 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.957640+0000", 2024-09-13T11:24:43.821 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0012450370000000001 2024-09-13T11:24:43.821 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:43.821 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:43.822 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:43.822 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.822 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.822 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPushReply(3.1f 89/78 [PushReplyOp(3:ff150976:::benchmark_data_smithi177_31116_object12907:head)])", 2024-09-13T11:24:43.822 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:46.697504+0000", 2024-09-13T11:24:43.822 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 57.046939305999999, 2024-09-13T11:24:43.822 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.26836818099999998, 2024-09-13T11:24:43.822 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:43.822 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:43.822 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-13T11:24:43.822 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:43.822 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.822 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:43.822 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.697504+0000", 2024-09-13T11:24:43.823 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:43.823 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.823 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.823 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:43.823 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.697504+0000", 2024-09-13T11:24:43.823 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:43.823 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.823 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.823 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:43.823 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.697505+0000", 2024-09-13T11:24:43.823 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0529999999999999e-06 2024-09-13T11:24:43.823 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.823 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.823 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:43.824 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.697509+0000", 2024-09-13T11:24:43.824 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.9249999999999997e-06 2024-09-13T11:24:43.824 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.824 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.824 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:43.824 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.964086+0000", 2024-09-13T11:24:43.824 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.26657776100000002 2024-09-13T11:24:43.824 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.824 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.824 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:43.824 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.964110+0000", 2024-09-13T11:24:43.824 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.3189000000000001e-05 2024-09-13T11:24:43.824 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.824 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.825 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:43.825 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.964354+0000", 2024-09-13T11:24:43.825 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00024481399999999999 2024-09-13T11:24:43.825 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.825 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.825 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:24:43.825 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.965872+0000", 2024-09-13T11:24:43.825 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001517439 2024-09-13T11:24:43.825 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:43.825 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:43.825 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:43.825 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.825 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.826 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:71589 3.21 3:85f71b95:::benchmark_data_smithi177_31116_object71588:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96)", 2024-09-13T11:24:43.826 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:53.876584+0000", 2024-09-13T11:24:43.826 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 49.867858853000001, 2024-09-13T11:24:43.826 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.24303730800000001, 2024-09-13T11:24:43.826 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:43.826 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:43.826 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:43.826 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:24:43.826 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:24:43.826 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:43.826 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 71589 2024-09-13T11:24:43.826 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.826 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:43.827 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.827 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:43.827 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:53.876584+0000", 2024-09-13T11:24:43.827 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:43.827 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.827 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.827 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:43.827 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:53.876584+0000", 2024-09-13T11:24:43.827 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:43.827 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.827 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.827 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:43.827 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:53.876586+0000", 2024-09-13T11:24:43.827 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.4100000000000001e-06 2024-09-13T11:24:43.828 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.828 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.828 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:43.828 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:53.876604+0000", 2024-09-13T11:24:43.828 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.8389e-05 2024-09-13T11:24:43.828 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.828 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.828 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:43.828 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:53.876613+0000", 2024-09-13T11:24:43.828 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.8789999999999992e-06 2024-09-13T11:24:43.828 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.828 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.828 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:43.828 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:53.876619+0000", 2024-09-13T11:24:43.829 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.9699999999999996e-06 2024-09-13T11:24:43.829 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.829 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.829 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:43.829 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:53.876681+0000", 2024-09-13T11:24:43.829 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.2699000000000004e-05 2024-09-13T11:24:43.829 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.829 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.829 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:24:43.829 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:53.876849+0000", 2024-09-13T11:24:43.829 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00016714799999999999 2024-09-13T11:24:43.829 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.829 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.829 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 0,8", 2024-09-13T11:24:43.830 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:53.877032+0000", 2024-09-13T11:24:43.830 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00018387199999999999 2024-09-13T11:24:43.830 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.830 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.830 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:24:43.830 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:53.879473+0000", 2024-09-13T11:24:43.830 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0024403789999999999 2024-09-13T11:24:43.830 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.830 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.830 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:43.830 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:53.880046+0000", 2024-09-13T11:24:43.830 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00057281200000000002 2024-09-13T11:24:43.830 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.830 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.831 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:43.831 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.119299+0000", 2024-09-13T11:24:43.831 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.23925310399999999 2024-09-13T11:24:43.831 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.831 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.831 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:24:43.831 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.119337+0000", 2024-09-13T11:24:43.831 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.8109999999999999e-05 2024-09-13T11:24:43.831 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.831 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.831 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:24:43.831 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.119621+0000", 2024-09-13T11:24:43.831 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000284536 2024-09-13T11:24:43.831 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:43.832 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:43.832 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:43.832 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.832 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.832 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:83223 3.1c 3:3bc6018d:::benchmark_data_smithi177_31116_object83222:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106)", 2024-09-13T11:24:43.832 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:04.511163+0000", 2024-09-13T11:24:43.832 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 39.233279527999997, 2024-09-13T11:24:43.832 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.47558093200000001, 2024-09-13T11:24:43.832 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:43.832 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:43.832 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:43.832 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:24:43.832 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:24:43.833 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:43.833 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 83223 2024-09-13T11:24:43.833 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.833 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:43.833 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.833 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:43.833 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.511163+0000", 2024-09-13T11:24:43.833 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:43.833 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.833 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.833 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:43.833 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.511163+0000", 2024-09-13T11:24:43.833 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:43.833 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.834 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.834 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:43.834 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.511167+0000", 2024-09-13T11:24:43.834 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.8330000000000004e-06 2024-09-13T11:24:43.834 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.834 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.834 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:43.834 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.511219+0000", 2024-09-13T11:24:43.834 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.1771999999999998e-05 2024-09-13T11:24:43.834 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.834 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.834 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:43.834 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.511252+0000", 2024-09-13T11:24:43.835 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.3315999999999997e-05 2024-09-13T11:24:43.835 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.835 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.835 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:43.835 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.511268+0000", 2024-09-13T11:24:43.835 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5812999999999999e-05 2024-09-13T11:24:43.835 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.835 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.835 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:43.835 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.511418+0000", 2024-09-13T11:24:43.835 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000150023 2024-09-13T11:24:43.835 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.835 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.835 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:24:43.836 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.511670+0000", 2024-09-13T11:24:43.836 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00025185499999999999 2024-09-13T11:24:43.836 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.836 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.836 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 7", 2024-09-13T11:24:43.836 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.511905+0000", 2024-09-13T11:24:43.836 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00023493799999999999 2024-09-13T11:24:43.836 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.836 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.836 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:24:43.836 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.514960+0000", 2024-09-13T11:24:43.836 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.003054977 2024-09-13T11:24:43.836 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.836 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.837 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:43.837 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.986432+0000", 2024-09-13T11:24:43.837 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.47147209200000001 2024-09-13T11:24:43.837 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.837 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.837 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:24:43.837 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.986635+0000", 2024-09-13T11:24:43.837 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00020309899999999999 2024-09-13T11:24:43.837 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.837 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.837 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:24:43.837 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.986744+0000", 2024-09-13T11:24:43.837 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000109214 2024-09-13T11:24:43.837 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:43.838 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:43.838 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:43.838 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.838 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.838 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop_reply(client.4665.0:84803 3.1f e107/78)", 2024-09-13T11:24:43.838 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:05.912627+0000", 2024-09-13T11:24:43.838 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 37.831815620999997, 2024-09-13T11:24:43.838 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.33726566600000002, 2024-09-13T11:24:43.838 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:43.838 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:43.838 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-13T11:24:43.838 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:43.838 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.839 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:43.839 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:05.912627+0000", 2024-09-13T11:24:43.839 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:43.839 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.839 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.839 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:43.839 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:05.912627+0000", 2024-09-13T11:24:43.839 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:43.839 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.839 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.839 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:43.839 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:05.912629+0000", 2024-09-13T11:24:43.839 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.4440000000000001e-06 2024-09-13T11:24:43.839 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.839 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.840 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:43.840 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:05.912634+0000", 2024-09-13T11:24:43.840 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.6869999999999999e-06 2024-09-13T11:24:43.840 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.840 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.840 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:43.840 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:06.249722+0000", 2024-09-13T11:24:43.840 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.33708766099999998 2024-09-13T11:24:43.840 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.840 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.840 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:43.840 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:06.249736+0000", 2024-09-13T11:24:43.840 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3623e-05 2024-09-13T11:24:43.841 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.841 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.841 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:43.841 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:06.249878+0000", 2024-09-13T11:24:43.841 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00014264599999999999 2024-09-13T11:24:43.841 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.841 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.841 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:24:43.841 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:06.249893+0000", 2024-09-13T11:24:43.841 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.4605e-05 2024-09-13T11:24:43.841 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:43.841 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:43.841 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:43.842 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.842 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.842 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop_reply(client.4665.0:110260 3.f e119/75 ondisk, result = 0)", 2024-09-13T11:24:43.842 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:25.333188+0000", 2024-09-13T11:24:43.842 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 18.411254994, 2024-09-13T11:24:43.842 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.29195486199999998, 2024-09-13T11:24:43.842 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:43.842 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:43.842 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:24:43.842 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:43.842 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.842 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:43.842 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.333188+0000", 2024-09-13T11:24:43.842 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:43.843 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.843 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.843 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:43.843 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.333188+0000", 2024-09-13T11:24:43.843 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:43.843 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.843 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.843 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:43.843 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.333190+0000", 2024-09-13T11:24:43.843 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.2120000000000002e-06 2024-09-13T11:24:43.843 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.843 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.843 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:43.843 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.333197+0000", 2024-09-13T11:24:43.844 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.7469999999999999e-06 2024-09-13T11:24:43.844 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.844 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.844 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:43.844 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.624892+0000", 2024-09-13T11:24:43.844 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.291694913 2024-09-13T11:24:43.844 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.844 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.844 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:43.844 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.624907+0000", 2024-09-13T11:24:43.844 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5053e-05 2024-09-13T11:24:43.844 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.844 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.845 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:43.845 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.624999+0000", 2024-09-13T11:24:43.845 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.2603999999999996e-05 2024-09-13T11:24:43.845 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.845 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.845 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:24:43.845 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.625011+0000", 2024-09-13T11:24:43.845 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.1253999999999999e-05 2024-09-13T11:24:43.845 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.845 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.846 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:24:43.846 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.625143+0000", 2024-09-13T11:24:43.846 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00013207900000000001 2024-09-13T11:24:43.846 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:43.846 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:43.846 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:43.846 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.846 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.846 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:110265 3.1 e119/75 3:8236484d:::benchmark_data_smithi177_31116_object110264:head v 119'3935, mlcod=118'3933)", 2024-09-13T11:24:43.846 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:25.343348+0000", 2024-09-13T11:24:43.846 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 18.401095379000001, 2024-09-13T11:24:43.846 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.60013013500000001, 2024-09-13T11:24:43.846 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:43.846 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:43.847 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:43.847 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:43.847 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.847 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:43.847 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.343348+0000", 2024-09-13T11:24:43.847 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:43.847 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.847 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.847 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:43.847 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.343348+0000", 2024-09-13T11:24:43.847 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:43.847 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.847 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.848 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:43.848 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.343350+0000", 2024-09-13T11:24:43.848 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.576e-06 2024-09-13T11:24:43.848 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.848 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.848 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:43.848 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.343443+0000", 2024-09-13T11:24:43.848 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.2989e-05 2024-09-13T11:24:43.848 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.848 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.848 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:43.848 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.940863+0000", 2024-09-13T11:24:43.848 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.59742003300000002 2024-09-13T11:24:43.848 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.849 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.849 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:43.849 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.940875+0000", 2024-09-13T11:24:43.849 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.2147000000000001e-05 2024-09-13T11:24:43.849 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.849 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.849 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:43.849 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.940993+0000", 2024-09-13T11:24:43.849 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000117903 2024-09-13T11:24:43.849 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.849 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.849 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:24:43.849 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.941017+0000", 2024-09-13T11:24:43.849 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.3442999999999999e-05 2024-09-13T11:24:43.850 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.850 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.850 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:24:43.850 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.943347+0000", 2024-09-13T11:24:43.850 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0023299869999999999 2024-09-13T11:24:43.850 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:43.850 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:43.850 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:24:43.850 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.943478+0000", 2024-09-13T11:24:43.850 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00013105699999999999 2024-09-13T11:24:43.850 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:43.850 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:43.850 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:43.850 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:43.851 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:43.851 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-09-13T11:24:43.852 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_ops_in_flight 2024-09-13T11:24:43.853 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-13T11:24:43.851+0000 7f6a68775640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26307) UID: 0 2024-09-13T11:24:43.953 INFO:tasks.ceph.osd.10.smithi162.stderr:2024-09-13T11:24:43.948+0000 7fb59c9f0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26317) UID: 0 2024-09-13T11:24:44.004 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-09-13T11:24:44.004 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [], 2024-09-13T11:24:44.004 INFO:teuthology.orchestra.run.smithi177.stdout: "num_ops": 0 2024-09-13T11:24:44.004 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-09-13T11:24:44.012 INFO:tasks.radosbench.radosbench.0.smithi177.stdout: 137 16 127140 127124 57.9882 68.9375 0.00459033 0.0172382 2024-09-13T11:24:44.028 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_blocked_ops 2024-09-13T11:24:44.053 INFO:tasks.ceph.osd.10.smithi162.stderr:2024-09-13T11:24:44.052+0000 7fb59c9f0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26317) UID: 0 2024-09-13T11:24:44.088 INFO:tasks.thrashosds.thrasher:running fsck to verify new sharding on osd.14 2024-09-13T11:24:44.088 DEBUG:teuthology.orchestra.run.smithi162:> sudo ceph-bluestore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/bluestore_tool.$pid.log' --log-level=10 --path /var/lib/ceph/osd/ceph-14 fsck 2024-09-13T11:24:44.117 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T11:24:44.164 INFO:tasks.ceph.osd.12.smithi001.stderr:2024-09-13T11:24:44.163+0000 7f892c6e6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26314) UID: 0 2024-09-13T11:24:44.181 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-09-13T11:24:44.181 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [], 2024-09-13T11:24:44.181 INFO:teuthology.orchestra.run.smithi177.stdout: "complaint_time": 30, 2024-09-13T11:24:44.181 INFO:teuthology.orchestra.run.smithi177.stdout: "num_blocked_ops": 0 2024-09-13T11:24:44.181 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-09-13T11:24:44.207 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_historic_ops 2024-09-13T11:24:44.255 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-13T11:24:44.251+0000 7f6a68775640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26307) UID: 0 2024-09-13T11:24:44.296 INFO:teuthology.orchestra.run.smithi001.stdout:osd.0: {} 2024-09-13T11:24:44.296 INFO:teuthology.orchestra.run.smithi001.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T11:24:44.308 INFO:teuthology.orchestra.run.smithi001.stderr:noscrub is unset 2024-09-13T11:24:44.311 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-13T11:24:44.307+0000 7f6a5d1a2640 -1 osd.0 130 handle_osd_map injecting map crc failure 2024-09-13T11:24:44.355 INFO:tasks.ceph.osd.5.smithi092.stderr:2024-09-13T11:24:44.350+0000 7f6f0b705640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26346) UID: 0 2024-09-13T11:24:44.381 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-09-13T11:24:44.381 INFO:teuthology.orchestra.run.smithi177.stdout: "size": 20, 2024-09-13T11:24:44.381 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 600, 2024-09-13T11:24:44.381 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [ 2024-09-13T11:24:44.381 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:44.381 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:25383 3.0 e55/30 3:0a6bb869:::benchmark_data_smithi177_31116_object25382:head v 55'1620, mlcod=55'1617)", 2024-09-13T11:24:44.381 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:04.449956+0000", 2024-09-13T11:24:44.381 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 99.928106352, 2024-09-13T11:24:44.381 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.21143479600000001, 2024-09-13T11:24:44.381 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:44.382 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:44.382 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:44.382 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:44.382 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:44.382 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:44.382 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.449956+0000", 2024-09-13T11:24:44.382 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:44.382 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:44.382 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:44.382 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:44.382 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.449956+0000", 2024-09-13T11:24:44.382 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:44.382 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:44.382 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:44.383 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:44.383 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.449959+0000", 2024-09-13T11:24:44.383 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.393e-06 2024-09-13T11:24:44.383 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:44.383 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:44.383 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:44.383 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.450014+0000", 2024-09-13T11:24:44.383 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.4200000000000003e-05 2024-09-13T11:24:44.383 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:44.383 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:44.383 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:44.383 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.658637+0000", 2024-09-13T11:24:44.383 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.20862334799999999 2024-09-13T11:24:44.383 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:44.384 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:44.384 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:44.384 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.658650+0000", 2024-09-13T11:24:44.384 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3148e-05 2024-09-13T11:24:44.384 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:44.384 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:44.384 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:44.384 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.658867+0000", 2024-09-13T11:24:44.384 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00021673499999999999 2024-09-13T11:24:44.384 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:44.384 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:44.384 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:24:44.385 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.658931+0000", 2024-09-13T11:24:44.385 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.4045000000000001e-05 2024-09-13T11:24:44.385 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:44.385 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:44.385 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:24:44.385 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.661317+0000", 2024-09-13T11:24:44.385 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.002385823 2024-09-13T11:24:44.385 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:44.385 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:44.385 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:24:44.385 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.661391+0000", 2024-09-13T11:24:44.385 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.4103999999999994e-05 2024-09-13T11:24:44.385 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:44.386 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:44.386 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:44.386 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:44.386 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:44.386 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPush(3.16 85/70 [PushOp(3:6ac421eb:::benchmark_data_smithi177_31116_object43910:head, version: 66'2836, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:6ac421eb:::benchmark_data_smithi177_31116_object43910:head@66'2836, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:24:44.386 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:42.145291+0000", 2024-09-13T11:24:44.386 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 62.232771145999997, 2024-09-13T11:24:44.386 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.148514919, 2024-09-13T11:24:44.386 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:44.386 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:44.386 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:24:44.386 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:44.386 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:44.386 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:44.387 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.145291+0000", 2024-09-13T11:24:44.387 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:44.387 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:44.387 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:44.387 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:44.387 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.145291+0000", 2024-09-13T11:24:44.387 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:44.387 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:44.387 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:44.387 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:44.387 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.145293+0000", 2024-09-13T11:24:44.387 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.699e-06 2024-09-13T11:24:44.387 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:44.388 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:44.388 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:44.388 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.145417+0000", 2024-09-13T11:24:44.388 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00012415200000000001 2024-09-13T11:24:44.388 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:44.388 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:44.388 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:44.388 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.292964+0000", 2024-09-13T11:24:44.388 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.14754715400000001 2024-09-13T11:24:44.388 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:44.388 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:44.388 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:44.388 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.292981+0000", 2024-09-13T11:24:44.389 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.6790999999999999e-05 2024-09-13T11:24:44.389 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:44.389 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:44.389 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:44.389 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.293083+0000", 2024-09-13T11:24:44.389 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00010199799999999999 2024-09-13T11:24:44.389 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:44.389 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:44.389 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:24:44.389 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.293098+0000", 2024-09-13T11:24:44.389 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.4612e-05 2024-09-13T11:24:44.389 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:44.389 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:44.389 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:24:44.390 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.293806+0000", 2024-09-13T11:24:44.390 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00070851300000000002 2024-09-13T11:24:44.390 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:44.390 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:44.390 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:44.390 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:44.390 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:44.390 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:60390 3.16 e85/70 3:6a388186:::benchmark_data_smithi177_31116_object60389:head v 85'3357, mlcod=66'2835)", 2024-09-13T11:24:44.390 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:42.149102+0000", 2024-09-13T11:24:44.390 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 62.228960088999997, 2024-09-13T11:24:44.390 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.14821161799999999, 2024-09-13T11:24:44.390 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:44.390 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:44.391 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:44.391 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:44.391 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:44.391 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:44.391 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.149102+0000", 2024-09-13T11:24:44.391 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:44.391 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:44.391 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:44.391 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:44.391 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.149102+0000", 2024-09-13T11:24:44.391 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:44.391 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:44.391 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:44.391 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:44.392 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.149104+0000", 2024-09-13T11:24:44.392 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.708e-06 2024-09-13T11:24:44.392 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:44.392 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:44.392 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:44.392 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.149160+0000", 2024-09-13T11:24:44.392 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.5618000000000002e-05 2024-09-13T11:24:44.392 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:44.392 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:44.392 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:44.392 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.296071+0000", 2024-09-13T11:24:44.392 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.14691148600000001 2024-09-13T11:24:44.392 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:44.393 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:44.393 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:44.393 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.296076+0000", 2024-09-13T11:24:44.393 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.5580000000000002e-06 2024-09-13T11:24:44.393 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:44.393 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:44.393 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:44.393 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.296127+0000", 2024-09-13T11:24:44.393 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.1405e-05 2024-09-13T11:24:44.393 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:44.393 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:44.393 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:24:44.393 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.296150+0000", 2024-09-13T11:24:44.393 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.3113e-05 2024-09-13T11:24:44.394 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:44.394 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:44.394 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:24:44.394 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.297274+0000", 2024-09-13T11:24:44.394 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0011239900000000001 2024-09-13T11:24:44.394 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:44.394 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:44.394 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:24:44.394 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.297314+0000", 2024-09-13T11:24:44.394 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.9740000000000002e-05 2024-09-13T11:24:44.394 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:44.394 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:44.394 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:44.395 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:44.395 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:44.395 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPush(3.16 85/70 [PushOp(3:69aac9fd:::benchmark_data_smithi177_31116_object43914:head, version: 66'2837, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:69aac9fd:::benchmark_data_smithi177_31116_object43914:head@66'2837, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:24:44.395 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:42.295995+0000", 2024-09-13T11:24:44.395 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 62.082067031999998, 2024-09-13T11:24:44.395 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.215089431, 2024-09-13T11:24:44.395 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:44.395 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:44.395 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:24:44.395 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:44.395 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:44.395 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:44.395 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.295995+0000", 2024-09-13T11:24:44.395 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:44.396 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:44.396 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:44.396 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:44.396 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.295995+0000", 2024-09-13T11:24:44.396 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:44.396 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:44.396 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:44.396 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:44.396 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.295997+0000", 2024-09-13T11:24:44.396 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3170000000000001e-06 2024-09-13T11:24:44.396 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:44.396 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:44.396 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:44.397 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.296058+0000", 2024-09-13T11:24:44.397 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.1690000000000001e-05 2024-09-13T11:24:44.397 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:44.397 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:44.397 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:44.397 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.509606+0000", 2024-09-13T11:24:44.397 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.21354748300000001 2024-09-13T11:24:44.397 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:44.397 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:44.397 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:44.397 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.509638+0000", 2024-09-13T11:24:44.397 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.1644999999999999e-05 2024-09-13T11:24:44.397 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:44.397 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:44.398 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:44.398 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.509814+0000", 2024-09-13T11:24:44.398 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00017679499999999999 2024-09-13T11:24:44.398 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:44.398 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:44.398 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:24:44.398 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.509840+0000", 2024-09-13T11:24:44.398 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.5871e-05 2024-09-13T11:24:44.398 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:44.398 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:44.398 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:24:44.398 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.511085+0000", 2024-09-13T11:24:44.398 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00124463 2024-09-13T11:24:44.398 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:44.398 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:44.399 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:44.399 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:44.399 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:44.399 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPush(3.1d 91/78 [PushOp(3:be6a0ae4:::benchmark_data_smithi177_31116_object25671:head, version: 58'1569, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:be6a0ae4:::benchmark_data_smithi177_31116_object25671:head@58'1569, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:24:44.399 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:48.481971+0000", 2024-09-13T11:24:44.399 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 55.896091787000003, 2024-09-13T11:24:44.399 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.47945454199999998, 2024-09-13T11:24:44.399 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:44.399 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:44.399 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:24:44.399 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:44.399 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:44.399 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:44.400 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.481971+0000", 2024-09-13T11:24:44.400 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:44.400 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:44.400 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:44.400 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:44.400 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.481971+0000", 2024-09-13T11:24:44.400 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:44.400 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:44.400 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:44.400 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:44.400 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.481972+0000", 2024-09-13T11:24:44.400 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5179999999999999e-06 2024-09-13T11:24:44.400 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:44.400 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:44.401 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:44.401 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.482000+0000", 2024-09-13T11:24:44.401 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.8249e-05 2024-09-13T11:24:44.401 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:44.401 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:44.401 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:44.401 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.959926+0000", 2024-09-13T11:24:44.401 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.47792585100000001 2024-09-13T11:24:44.401 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:44.401 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:44.401 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:44.401 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.959944+0000", 2024-09-13T11:24:44.401 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.7620000000000001e-05 2024-09-13T11:24:44.401 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:44.402 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:44.402 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:44.402 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.960059+0000", 2024-09-13T11:24:44.402 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000115188 2024-09-13T11:24:44.402 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:44.402 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:44.402 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:24:44.402 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.960074+0000", 2024-09-13T11:24:44.402 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.4738e-05 2024-09-13T11:24:44.402 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:44.402 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:44.402 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:24:44.402 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.961425+0000", 2024-09-13T11:24:44.402 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0013513780000000001 2024-09-13T11:24:44.403 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:44.403 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:44.403 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:44.403 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:44.403 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:44.403 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:71884 3.16 e96/88 3:6ca846b2:::benchmark_data_smithi177_31116_object71883:head v 96'3755, mlcod=96'3753)", 2024-09-13T11:24:44.403 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:54.108299+0000", 2024-09-13T11:24:44.403 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 50.269763613999999, 2024-09-13T11:24:44.403 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.20685140099999999, 2024-09-13T11:24:44.403 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:44.403 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:44.403 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:44.403 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:44.403 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:44.404 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:44.404 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.108299+0000", 2024-09-13T11:24:44.404 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:44.404 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:44.404 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:44.404 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:44.404 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.108299+0000", 2024-09-13T11:24:44.404 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:44.404 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:44.404 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:44.404 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:44.404 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.108300+0000", 2024-09-13T11:24:44.404 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.173e-06 2024-09-13T11:24:44.404 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:44.405 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:44.405 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:44.405 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.108323+0000", 2024-09-13T11:24:44.405 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.2591000000000001e-05 2024-09-13T11:24:44.405 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:44.405 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:44.405 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:44.405 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.312377+0000", 2024-09-13T11:24:44.405 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.20405456599999999 2024-09-13T11:24:44.405 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:44.405 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:44.405 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:44.405 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.312382+0000", 2024-09-13T11:24:44.406 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.1429999999999997e-06 2024-09-13T11:24:44.406 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:44.406 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:44.406 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:44.406 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.312480+0000", 2024-09-13T11:24:44.406 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.7742e-05 2024-09-13T11:24:44.406 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:44.406 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:44.406 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:24:44.406 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.312521+0000", 2024-09-13T11:24:44.406 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.1369000000000003e-05 2024-09-13T11:24:44.406 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:44.406 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:44.406 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:24:44.407 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.315105+0000", 2024-09-13T11:24:44.407 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0025835060000000002 2024-09-13T11:24:44.407 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:44.407 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:44.407 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:24:44.407 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.315150+0000", 2024-09-13T11:24:44.407 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.5311e-05 2024-09-13T11:24:44.407 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:44.408 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:44.408 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:44.408 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:44.408 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:44.408 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:71927 3.0 e97/75 3:0197244a:::benchmark_data_smithi177_31116_object71926:head v 97'3351, mlcod=96'3348)", 2024-09-13T11:24:44.408 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:54.200161+0000", 2024-09-13T11:24:44.408 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 50.177901386000002, 2024-09-13T11:24:44.408 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.26691738999999998, 2024-09-13T11:24:44.408 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:44.408 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:44.409 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:44.409 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:44.409 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:44.409 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:44.409 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.200161+0000", 2024-09-13T11:24:44.409 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:44.409 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:44.409 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:44.409 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:44.409 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.200161+0000", 2024-09-13T11:24:44.409 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:44.409 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:44.409 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:44.410 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:44.410 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.200162+0000", 2024-09-13T11:24:44.410 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.78e-07 2024-09-13T11:24:44.410 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:44.410 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:44.410 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:44.410 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.200200+0000", 2024-09-13T11:24:44.410 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.8121000000000003e-05 2024-09-13T11:24:44.410 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:44.410 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:44.410 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:44.410 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.465397+0000", 2024-09-13T11:24:44.410 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.26519681099999998 2024-09-13T11:24:44.411 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:44.411 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:44.411 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:44.411 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.465404+0000", 2024-09-13T11:24:44.411 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.7399999999999998e-06 2024-09-13T11:24:44.411 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:44.411 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:44.411 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:44.411 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.465456+0000", 2024-09-13T11:24:44.411 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.2626000000000001e-05 2024-09-13T11:24:44.412 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:44.412 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:44.412 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:24:44.412 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.465482+0000", 2024-09-13T11:24:44.412 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.6086000000000001e-05 2024-09-13T11:24:44.412 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:44.412 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:44.412 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:24:44.412 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.467035+0000", 2024-09-13T11:24:44.412 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0015523729999999999 2024-09-13T11:24:44.412 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:44.412 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:44.412 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:24:44.413 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.467078+0000", 2024-09-13T11:24:44.413 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.3754999999999997e-05 2024-09-13T11:24:44.413 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:44.413 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:44.413 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:44.413 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:44.413 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:44.413 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:72250 3.0 e97/75 3:010c7171:::benchmark_data_smithi177_31116_object72249:head v 97'3353, mlcod=96'3350)", 2024-09-13T11:24:44.413 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:54.516381+0000", 2024-09-13T11:24:44.413 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 49.861681873000002, 2024-09-13T11:24:44.413 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.13663347300000001, 2024-09-13T11:24:44.413 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:44.413 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:44.414 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:44.414 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:44.414 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:44.414 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:44.414 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.516381+0000", 2024-09-13T11:24:44.414 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:44.414 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:44.414 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:44.414 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:44.414 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.516381+0000", 2024-09-13T11:24:44.414 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:44.414 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:44.414 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:44.414 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:44.415 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.516382+0000", 2024-09-13T11:24:44.415 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3239999999999999e-06 2024-09-13T11:24:44.415 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:44.415 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:44.415 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:44.415 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.516437+0000", 2024-09-13T11:24:44.415 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.5408999999999998e-05 2024-09-13T11:24:44.415 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:44.415 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:44.415 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:44.415 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.651731+0000", 2024-09-13T11:24:44.415 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.135293583 2024-09-13T11:24:44.415 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:44.416 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:44.416 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:44.416 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.651736+0000", 2024-09-13T11:24:44.416 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.3410000000000002e-06 2024-09-13T11:24:44.416 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:44.416 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:44.416 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:44.416 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.651791+0000", 2024-09-13T11:24:44.416 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.4758e-05 2024-09-13T11:24:44.416 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:44.416 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:44.416 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:24:44.416 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.651814+0000", 2024-09-13T11:24:44.416 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.3283999999999999e-05 2024-09-13T11:24:44.417 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:44.417 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:44.417 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:24:44.417 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.652975+0000", 2024-09-13T11:24:44.417 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0011604370000000001 2024-09-13T11:24:44.417 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:44.417 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:44.417 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:24:44.417 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.653014+0000", 2024-09-13T11:24:44.417 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.9337e-05 2024-09-13T11:24:44.417 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:44.417 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:44.417 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:44.417 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:44.418 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:44.418 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:72993 3.0 e97/75 3:02747b8f:::benchmark_data_smithi177_31116_object72992:head v 97'3366, mlcod=97'3363)", 2024-09-13T11:24:44.418 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:55.116009+0000", 2024-09-13T11:24:44.418 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 49.262053266999999, 2024-09-13T11:24:44.418 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.13962660099999999, 2024-09-13T11:24:44.418 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:44.418 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:44.418 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:44.418 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:44.418 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:44.418 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:44.418 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:55.116009+0000", 2024-09-13T11:24:44.418 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:44.418 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:44.419 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:44.419 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:44.419 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:55.116009+0000", 2024-09-13T11:24:44.419 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:44.419 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:44.419 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:44.419 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:44.419 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:55.116010+0000", 2024-09-13T11:24:44.419 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.291e-06 2024-09-13T11:24:44.419 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:44.419 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:44.419 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:44.419 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:55.116062+0000", 2024-09-13T11:24:44.419 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.1949e-05 2024-09-13T11:24:44.420 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:44.420 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:44.420 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:44.420 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:55.254100+0000", 2024-09-13T11:24:44.420 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.13803723000000001 2024-09-13T11:24:44.420 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:44.420 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:44.420 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:44.420 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:55.254106+0000", 2024-09-13T11:24:44.420 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.5939999999999999e-06 2024-09-13T11:24:44.420 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:44.420 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:44.421 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:44.421 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:55.254196+0000", 2024-09-13T11:24:44.421 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.9567999999999994e-05 2024-09-13T11:24:44.421 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:44.421 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:44.421 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:24:44.421 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:55.254219+0000", 2024-09-13T11:24:44.421 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.3266000000000001e-05 2024-09-13T11:24:44.421 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:44.421 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:44.421 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:24:44.421 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:55.255596+0000", 2024-09-13T11:24:44.421 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0013767409999999999 2024-09-13T11:24:44.421 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:44.422 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:44.422 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:24:44.422 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:55.255636+0000", 2024-09-13T11:24:44.422 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.9962000000000001e-05 2024-09-13T11:24:44.422 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:44.422 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:44.422 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:44.422 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:44.422 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:44.422 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:73523 3.0 e98/75 3:004115ff:::benchmark_data_smithi177_31116_object73522:head v 98'3382, mlcod=98'3378)", 2024-09-13T11:24:44.422 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:55.538309+0000", 2024-09-13T11:24:44.422 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 48.839753268999999, 2024-09-13T11:24:44.422 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.127489824, 2024-09-13T11:24:44.423 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:44.423 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:44.423 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:44.423 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:44.423 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:44.423 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:44.423 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:55.538309+0000", 2024-09-13T11:24:44.423 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:44.423 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:44.423 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:44.423 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:44.423 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:55.538309+0000", 2024-09-13T11:24:44.423 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:44.423 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:44.424 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:44.424 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:44.424 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:55.538310+0000", 2024-09-13T11:24:44.424 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.237e-06 2024-09-13T11:24:44.424 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:44.424 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:44.424 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:44.424 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:55.538343+0000", 2024-09-13T11:24:44.424 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.2480000000000001e-05 2024-09-13T11:24:44.424 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:44.424 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:44.424 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:44.424 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:55.663154+0000", 2024-09-13T11:24:44.425 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.124810988 2024-09-13T11:24:44.425 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:44.425 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:44.425 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:44.425 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:55.663161+0000", 2024-09-13T11:24:44.425 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.2389999999999996e-06 2024-09-13T11:24:44.425 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:44.425 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:44.425 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:44.425 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:55.663215+0000", 2024-09-13T11:24:44.425 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.3893000000000001e-05 2024-09-13T11:24:44.425 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:44.425 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:44.425 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:24:44.426 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:55.663235+0000", 2024-09-13T11:24:44.426 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.9624e-05 2024-09-13T11:24:44.426 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:44.426 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:44.426 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:24:44.426 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:55.665764+0000", 2024-09-13T11:24:44.426 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0025292859999999999 2024-09-13T11:24:44.426 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:44.426 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:44.426 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:24:44.426 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:55.665799+0000", 2024-09-13T11:24:44.426 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.5077000000000003e-05 2024-09-13T11:24:44.426 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:44.426 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:44.427 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:44.427 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:44.427 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:44.427 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:79634 3.5 3:a6c32e6f:::benchmark_data_smithi177_31116_object79633:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101)", 2024-09-13T11:24:44.427 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:00.168616+0000", 2024-09-13T11:24:44.427 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 44.209445998, 2024-09-13T11:24:44.427 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.169445552, 2024-09-13T11:24:44.427 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:44.427 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:44.427 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:44.427 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:24:44.427 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:24:44.427 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:44.428 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 79634 2024-09-13T11:24:44.428 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:44.428 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:44.428 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:44.428 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:44.428 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.168616+0000", 2024-09-13T11:24:44.428 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:44.428 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:44.428 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:44.428 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:44.428 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.168616+0000", 2024-09-13T11:24:44.428 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:44.428 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:44.429 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:44.429 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:44.429 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.168619+0000", 2024-09-13T11:24:44.429 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.565e-06 2024-09-13T11:24:44.429 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:44.429 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:44.429 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:44.429 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.168639+0000", 2024-09-13T11:24:44.429 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.0384999999999999e-05 2024-09-13T11:24:44.429 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:44.429 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:44.429 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:44.429 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.168652+0000", 2024-09-13T11:24:44.429 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.2779000000000001e-05 2024-09-13T11:24:44.430 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:44.430 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:44.430 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:44.430 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.168658+0000", 2024-09-13T11:24:44.430 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.2269999999999998e-06 2024-09-13T11:24:44.430 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:44.430 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:44.430 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:44.430 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.168722+0000", 2024-09-13T11:24:44.430 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.3888999999999998e-05 2024-09-13T11:24:44.430 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:44.430 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:44.431 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:24:44.431 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.168863+0000", 2024-09-13T11:24:44.431 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00014050799999999999 2024-09-13T11:24:44.431 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:44.431 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:44.431 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 13", 2024-09-13T11:24:44.431 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.169010+0000", 2024-09-13T11:24:44.431 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00014746200000000001 2024-09-13T11:24:44.431 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:44.431 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:44.431 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:24:44.431 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.170891+0000", 2024-09-13T11:24:44.431 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001880945 2024-09-13T11:24:44.432 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:44.432 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:44.432 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:44.432 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.337961+0000", 2024-09-13T11:24:44.432 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.16706952 2024-09-13T11:24:44.432 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:44.432 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:44.432 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:24:44.432 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.338019+0000", 2024-09-13T11:24:44.432 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.7958000000000003e-05 2024-09-13T11:24:44.432 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:44.432 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:44.432 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:24:44.433 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.338062+0000", 2024-09-13T11:24:44.433 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.3313999999999999e-05 2024-09-13T11:24:44.433 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:44.433 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:44.433 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:44.433 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:44.433 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:44.433 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPush(3.1b 102/78 [PushOp(3:df786fc7:::benchmark_data_smithi177_31116_object48840:head, version: 71'3079, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:df786fc7:::benchmark_data_smithi177_31116_object48840:head@71'3079, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:24:44.433 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:00.290536+0000", 2024-09-13T11:24:44.433 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 44.087526623999999, 2024-09-13T11:24:44.433 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.20017585399999999, 2024-09-13T11:24:44.433 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:44.436 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:44.436 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:24:44.436 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:44.436 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:44.436 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:44.436 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.290536+0000", 2024-09-13T11:24:44.436 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:44.436 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:44.436 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:44.436 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:44.436 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.290536+0000", 2024-09-13T11:24:44.437 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:44.437 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:44.437 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:44.437 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:44.437 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.290537+0000", 2024-09-13T11:24:44.437 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.327e-06 2024-09-13T11:24:44.437 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:44.437 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:44.437 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:44.437 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.290581+0000", 2024-09-13T11:24:44.437 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.3893999999999997e-05 2024-09-13T11:24:44.437 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:44.437 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:44.437 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:44.438 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.488588+0000", 2024-09-13T11:24:44.438 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.198006708 2024-09-13T11:24:44.438 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:44.438 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:44.438 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:44.438 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.488607+0000", 2024-09-13T11:24:44.438 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.9162999999999999e-05 2024-09-13T11:24:44.438 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:44.438 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:44.438 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:44.438 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.488747+0000", 2024-09-13T11:24:44.438 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00014040799999999999 2024-09-13T11:24:44.438 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:44.439 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:44.439 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:24:44.439 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.488770+0000", 2024-09-13T11:24:44.439 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.279e-05 2024-09-13T11:24:44.439 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:44.439 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:44.439 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:24:44.439 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.490712+0000", 2024-09-13T11:24:44.439 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001941564 2024-09-13T11:24:44.439 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:44.439 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:44.439 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:44.439 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:44.439 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:44.440 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPush(3.1b 102/78 [PushOp(3:d82bf6ec:::benchmark_data_smithi177_31116_object48847:head, version: 71'3080, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:d82bf6ec:::benchmark_data_smithi177_31116_object48847:head@71'3080, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:24:44.440 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:00.293979+0000", 2024-09-13T11:24:44.440 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 44.084083096999997, 2024-09-13T11:24:44.440 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.20314832099999999, 2024-09-13T11:24:44.440 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:44.440 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:44.440 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:24:44.440 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:44.440 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:44.440 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:44.440 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.293979+0000", 2024-09-13T11:24:44.440 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:44.440 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:44.441 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:44.441 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:44.441 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.293979+0000", 2024-09-13T11:24:44.441 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:44.441 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:44.441 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:44.441 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:44.441 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.293981+0000", 2024-09-13T11:24:44.441 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.4449999999999999e-06 2024-09-13T11:24:44.441 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:44.441 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:44.441 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:44.441 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.294085+0000", 2024-09-13T11:24:44.441 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000103848 2024-09-13T11:24:44.442 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:44.442 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:44.442 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:44.442 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.495571+0000", 2024-09-13T11:24:44.442 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.20148640800000001 2024-09-13T11:24:44.442 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:44.442 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:44.442 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:44.442 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.495609+0000", 2024-09-13T11:24:44.442 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.7904000000000002e-05 2024-09-13T11:24:44.443 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:44.443 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:44.443 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:44.443 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.495729+0000", 2024-09-13T11:24:44.443 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000120556 2024-09-13T11:24:44.443 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:44.443 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:44.443 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:24:44.443 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.495751+0000", 2024-09-13T11:24:44.443 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.1511000000000001e-05 2024-09-13T11:24:44.443 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:44.443 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:44.443 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:24:44.443 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.497128+0000", 2024-09-13T11:24:44.444 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001376649 2024-09-13T11:24:44.444 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:44.444 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:44.444 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:44.444 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:44.444 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:44.444 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:79801 3.5 3:a6efdf84:::benchmark_data_smithi177_31116_object79800:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102)", 2024-09-13T11:24:44.444 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:00.403201+0000", 2024-09-13T11:24:44.444 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 43.974861509999997, 2024-09-13T11:24:44.444 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.31868626300000003, 2024-09-13T11:24:44.444 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:44.444 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:44.444 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:44.444 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:24:44.445 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:24:44.445 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:44.445 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 79801 2024-09-13T11:24:44.445 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:44.445 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:44.445 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:44.445 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:44.445 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.403201+0000", 2024-09-13T11:24:44.445 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:44.445 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:44.445 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:44.445 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:44.445 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.403201+0000", 2024-09-13T11:24:44.445 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:44.446 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:44.446 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:44.446 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:44.446 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.403204+0000", 2024-09-13T11:24:44.446 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.0819999999999999e-06 2024-09-13T11:24:44.446 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:44.446 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:44.446 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:44.446 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.403260+0000", 2024-09-13T11:24:44.446 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.5624999999999997e-05 2024-09-13T11:24:44.446 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:44.446 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:44.446 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:44.446 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.403280+0000", 2024-09-13T11:24:44.447 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.0060000000000001e-05 2024-09-13T11:24:44.447 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:44.447 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:44.447 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:44.447 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.403291+0000", 2024-09-13T11:24:44.447 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.1335000000000001e-05 2024-09-13T11:24:44.447 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:44.447 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:44.447 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:44.447 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.403344+0000", 2024-09-13T11:24:44.447 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.3217000000000002e-05 2024-09-13T11:24:44.447 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:44.447 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:44.447 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:24:44.448 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.403456+0000", 2024-09-13T11:24:44.448 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000111706 2024-09-13T11:24:44.448 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:44.448 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:44.448 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 13", 2024-09-13T11:24:44.448 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.403547+0000", 2024-09-13T11:24:44.448 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.1556000000000002e-05 2024-09-13T11:24:44.448 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:44.448 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:44.448 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:24:44.448 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.407436+0000", 2024-09-13T11:24:44.448 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.003888856 2024-09-13T11:24:44.448 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:44.448 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:44.449 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:44.449 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.721722+0000", 2024-09-13T11:24:44.449 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.31428562700000001 2024-09-13T11:24:44.449 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:44.449 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:44.449 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:24:44.449 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.721814+0000", 2024-09-13T11:24:44.449 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.1564000000000006e-05 2024-09-13T11:24:44.449 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:44.449 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:44.449 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:24:44.449 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.721887+0000", 2024-09-13T11:24:44.449 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.3634999999999995e-05 2024-09-13T11:24:44.449 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:44.449 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:44.450 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:44.450 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:44.450 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:44.450 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPush(3.1b 102/78 [PushOp(3:dcbefe42:::benchmark_data_smithi177_31116_object49001:head, version: 71'3087, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:dcbefe42:::benchmark_data_smithi177_31116_object49001:head@71'3087, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:24:44.450 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:00.511027+0000", 2024-09-13T11:24:44.450 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 43.867035573999999, 2024-09-13T11:24:44.450 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.13427292699999999, 2024-09-13T11:24:44.450 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:44.450 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:44.450 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:24:44.450 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:44.450 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:44.450 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:44.451 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.511027+0000", 2024-09-13T11:24:44.451 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:44.451 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:44.451 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:44.451 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:44.451 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.511027+0000", 2024-09-13T11:24:44.451 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:44.451 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:44.451 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:44.451 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:44.451 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.511028+0000", 2024-09-13T11:24:44.451 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.4869999999999999e-06 2024-09-13T11:24:44.451 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:44.451 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:44.451 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:44.452 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.511095+0000", 2024-09-13T11:24:44.452 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.6382000000000002e-05 2024-09-13T11:24:44.452 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:44.452 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:44.452 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:44.452 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.643319+0000", 2024-09-13T11:24:44.452 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.13222455699999999 2024-09-13T11:24:44.452 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:44.452 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:44.452 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:44.452 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.643349+0000", 2024-09-13T11:24:44.452 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.9356000000000002e-05 2024-09-13T11:24:44.452 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:44.452 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:44.453 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:44.453 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.643518+0000", 2024-09-13T11:24:44.453 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00016963500000000001 2024-09-13T11:24:44.453 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:44.453 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:44.453 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:24:44.453 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.643541+0000", 2024-09-13T11:24:44.453 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.2367000000000001e-05 2024-09-13T11:24:44.453 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:44.453 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:44.453 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:24:44.453 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.645300+0000", 2024-09-13T11:24:44.453 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0017591429999999999 2024-09-13T11:24:44.454 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:44.454 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:44.454 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:44.454 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:44.454 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:44.454 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:79813 3.16 e102/88 3:6c2df1ae:::benchmark_data_smithi177_31116_object79812:head v 102'4016, mlcod=102'4014)", 2024-09-13T11:24:44.454 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:00.516959+0000", 2024-09-13T11:24:44.454 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 43.861103548999999, 2024-09-13T11:24:44.454 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.15075599100000001, 2024-09-13T11:24:44.454 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:44.454 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:44.454 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:44.454 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:44.454 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:44.454 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:44.455 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.516959+0000", 2024-09-13T11:24:44.455 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:44.455 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:44.455 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:44.455 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:44.455 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.516959+0000", 2024-09-13T11:24:44.455 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:44.455 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:44.455 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:44.455 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:44.455 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.516961+0000", 2024-09-13T11:24:44.455 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.0279999999999999e-06 2024-09-13T11:24:44.455 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:44.455 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:44.456 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:44.456 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.517002+0000", 2024-09-13T11:24:44.456 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.0754999999999999e-05 2024-09-13T11:24:44.456 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:44.456 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:44.456 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:44.456 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.664491+0000", 2024-09-13T11:24:44.456 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.14748925099999999 2024-09-13T11:24:44.456 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:44.456 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:44.456 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:44.456 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.664504+0000", 2024-09-13T11:24:44.456 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.2747e-05 2024-09-13T11:24:44.457 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:44.457 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:44.457 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:44.457 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.664642+0000", 2024-09-13T11:24:44.457 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00013850900000000001 2024-09-13T11:24:44.457 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:44.457 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:44.457 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:24:44.457 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.664678+0000", 2024-09-13T11:24:44.457 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.5403e-05 2024-09-13T11:24:44.457 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:44.457 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:44.457 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:24:44.458 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.667646+0000", 2024-09-13T11:24:44.458 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.002968133 2024-09-13T11:24:44.458 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:44.458 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:44.458 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:24:44.458 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.667715+0000", 2024-09-13T11:24:44.458 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.9165000000000006e-05 2024-09-13T11:24:44.458 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:44.458 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:44.458 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:44.458 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:44.458 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:44.458 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:83199 3.1d e106/102 3:be2c6fae:::benchmark_data_smithi177_31116_object83198:head v 106'4244, mlcod=106'4242)", 2024-09-13T11:24:44.458 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:04.298299+0000", 2024-09-13T11:24:44.459 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 40.079763383, 2024-09-13T11:24:44.459 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.25001107099999997, 2024-09-13T11:24:44.459 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:44.459 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:44.459 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:44.459 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:44.459 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:44.459 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:44.459 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.298299+0000", 2024-09-13T11:24:44.459 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:44.459 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:44.459 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:44.459 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:44.460 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.298299+0000", 2024-09-13T11:24:44.460 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:44.460 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:44.460 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:44.460 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:44.460 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.298301+0000", 2024-09-13T11:24:44.460 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5859999999999999e-06 2024-09-13T11:24:44.460 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:44.460 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:44.460 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:44.460 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.298368+0000", 2024-09-13T11:24:44.460 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.6928999999999996e-05 2024-09-13T11:24:44.460 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:44.460 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:44.461 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:44.461 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.545180+0000", 2024-09-13T11:24:44.461 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.24681209700000001 2024-09-13T11:24:44.461 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:44.461 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:44.461 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:44.461 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.545193+0000", 2024-09-13T11:24:44.461 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3427e-05 2024-09-13T11:24:44.461 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:44.461 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:44.461 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:44.461 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.545327+0000", 2024-09-13T11:24:44.461 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00013359400000000001 2024-09-13T11:24:44.461 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:44.462 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:44.462 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:24:44.462 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.545385+0000", 2024-09-13T11:24:44.462 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.8492000000000002e-05 2024-09-13T11:24:44.462 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:44.462 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:44.462 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:24:44.462 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.548225+0000", 2024-09-13T11:24:44.462 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.002839472 2024-09-13T11:24:44.462 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:44.462 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:44.462 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:24:44.462 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.548310+0000", 2024-09-13T11:24:44.462 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.5474000000000002e-05 2024-09-13T11:24:44.462 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:44.463 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:44.463 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:44.463 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:44.463 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:44.463 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:84621 3.1d e107/102 3:becd72aa:::benchmark_data_smithi177_31116_object84620:head v 107'4296, mlcod=107'4293)", 2024-09-13T11:24:44.463 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:05.742238+0000", 2024-09-13T11:24:44.463 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 38.635824282999998, 2024-09-13T11:24:44.463 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.17888174300000001, 2024-09-13T11:24:44.463 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:44.463 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:44.463 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:44.463 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:44.463 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:44.463 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:44.464 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:05.742238+0000", 2024-09-13T11:24:44.464 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:44.464 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:44.464 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:44.464 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:44.464 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:05.742238+0000", 2024-09-13T11:24:44.464 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:44.464 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:44.464 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:44.464 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:44.464 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:05.742240+0000", 2024-09-13T11:24:44.464 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.3159999999999999e-06 2024-09-13T11:24:44.464 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:44.465 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:44.465 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:44.465 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:05.742315+0000", 2024-09-13T11:24:44.465 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.4122000000000006e-05 2024-09-13T11:24:44.465 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:44.465 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:44.465 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:44.465 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:05.917757+0000", 2024-09-13T11:24:44.465 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.175442031 2024-09-13T11:24:44.465 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:44.465 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:44.465 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:44.465 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:05.917764+0000", 2024-09-13T11:24:44.465 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.661e-06 2024-09-13T11:24:44.466 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:44.466 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:44.466 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:44.466 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:05.917846+0000", 2024-09-13T11:24:44.466 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.2006000000000001e-05 2024-09-13T11:24:44.466 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:44.466 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:44.466 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:24:44.466 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:05.917879+0000", 2024-09-13T11:24:44.466 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.2796000000000002e-05 2024-09-13T11:24:44.466 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:44.466 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:44.466 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:24:44.466 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:05.921070+0000", 2024-09-13T11:24:44.467 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0031908489999999999 2024-09-13T11:24:44.467 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:44.467 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:44.467 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:24:44.467 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:05.921120+0000", 2024-09-13T11:24:44.467 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.9962e-05 2024-09-13T11:24:44.467 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:44.467 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:44.467 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:44.467 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:44.467 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:44.467 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:100995 3.1d e114/102 3:b9d3a02e:::benchmark_data_smithi177_31116_object100994:head v 114'4771, mlcod=114'4769)", 2024-09-13T11:24:44.467 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:17.338503+0000", 2024-09-13T11:24:44.467 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 27.039559618999998, 2024-09-13T11:24:44.467 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.12755466100000001, 2024-09-13T11:24:44.468 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:44.468 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:44.468 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:44.468 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:44.468 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:44.468 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:44.468 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:17.338503+0000", 2024-09-13T11:24:44.468 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:44.468 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:44.468 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:44.468 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:44.468 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:17.338503+0000", 2024-09-13T11:24:44.468 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:44.469 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:44.469 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:44.469 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:44.469 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:17.338504+0000", 2024-09-13T11:24:44.469 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.635e-06 2024-09-13T11:24:44.469 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:44.469 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:44.469 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:44.469 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:17.338575+0000", 2024-09-13T11:24:44.469 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.0869000000000002e-05 2024-09-13T11:24:44.469 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:44.469 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:44.469 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:44.469 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:17.462756+0000", 2024-09-13T11:24:44.469 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.124180943 2024-09-13T11:24:44.470 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:44.470 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:44.470 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:44.470 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:17.462764+0000", 2024-09-13T11:24:44.470 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.3170000000000004e-06 2024-09-13T11:24:44.470 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:44.470 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:44.470 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:44.470 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:17.462887+0000", 2024-09-13T11:24:44.470 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00012359200000000001 2024-09-13T11:24:44.470 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:44.470 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:44.470 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:24:44.470 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:17.462948+0000", 2024-09-13T11:24:44.471 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.0563999999999998e-05 2024-09-13T11:24:44.471 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:44.471 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:44.471 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:24:44.471 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:17.465984+0000", 2024-09-13T11:24:44.471 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0030363669999999999 2024-09-13T11:24:44.471 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:44.471 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:44.471 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:24:44.471 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:17.466057+0000", 2024-09-13T11:24:44.471 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.3374000000000006e-05 2024-09-13T11:24:44.471 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:44.471 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:44.471 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:44.471 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:44.472 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:44.472 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:107910 3.16 e116/88 3:6cf57aa3:::benchmark_data_smithi177_31116_object107909:head v 116'4911, mlcod=116'4908)", 2024-09-13T11:24:44.472 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:22.516225+0000", 2024-09-13T11:24:44.472 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 21.861837739999999, 2024-09-13T11:24:44.472 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.18319666600000001, 2024-09-13T11:24:44.472 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:44.472 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:44.472 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:44.472 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:44.473 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:44.473 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:44.473 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:22.516225+0000", 2024-09-13T11:24:44.473 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:44.473 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:44.473 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:44.473 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:44.473 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:22.516225+0000", 2024-09-13T11:24:44.473 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:44.473 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:44.473 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:44.473 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:44.473 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:22.516226+0000", 2024-09-13T11:24:44.473 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.716e-06 2024-09-13T11:24:44.474 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:44.474 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:44.474 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:44.474 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:22.516250+0000", 2024-09-13T11:24:44.474 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.3975e-05 2024-09-13T11:24:44.474 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:44.474 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:44.474 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:44.474 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:22.697686+0000", 2024-09-13T11:24:44.474 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.18143539 2024-09-13T11:24:44.474 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:44.474 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:44.475 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:44.475 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:22.697692+0000", 2024-09-13T11:24:44.475 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.0769999999999996e-06 2024-09-13T11:24:44.475 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:44.475 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:44.475 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:44.475 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:22.697744+0000", 2024-09-13T11:24:44.475 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.2525000000000003e-05 2024-09-13T11:24:44.475 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:44.475 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:44.475 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:24:44.475 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:22.697757+0000", 2024-09-13T11:24:44.475 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.2819e-05 2024-09-13T11:24:44.476 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:44.476 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:44.476 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:24:44.476 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:22.699388+0000", 2024-09-13T11:24:44.476 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0016303999999999999 2024-09-13T11:24:44.476 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:44.476 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:44.476 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:24:44.476 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:22.699421+0000", 2024-09-13T11:24:44.476 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.3763999999999997e-05 2024-09-13T11:24:44.476 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:44.476 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:44.476 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:44.476 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:44.477 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:44.477 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-09-13T11:24:44.478 INFO:teuthology.orchestra.run.smithi001.stdout:osd.1: {} 2024-09-13T11:24:44.478 INFO:teuthology.orchestra.run.smithi001.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T11:24:44.479 INFO:teuthology.orchestra.run.smithi001.stdout:osd.2: {} 2024-09-13T11:24:44.479 INFO:teuthology.orchestra.run.smithi001.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T11:24:44.479 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-13T11:24:44.480 INFO:tasks.ceph.osd.12.smithi001.stderr:2024-09-13T11:24:44.475+0000 7f892c6e6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26314) UID: 0 2024-09-13T11:24:44.497 INFO:teuthology.orchestra.run.smithi001.stdout:osd.3: {} 2024-09-13T11:24:44.498 INFO:teuthology.orchestra.run.smithi001.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T11:24:44.522 INFO:teuthology.orchestra.run.smithi001.stdout:osd.4: {} 2024-09-13T11:24:44.522 INFO:teuthology.orchestra.run.smithi001.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T11:24:44.546 INFO:teuthology.orchestra.run.smithi001.stdout:osd.5: {} 2024-09-13T11:24:44.546 INFO:teuthology.orchestra.run.smithi001.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T11:24:44.570 INFO:teuthology.orchestra.run.smithi001.stdout:osd.6: {} 2024-09-13T11:24:44.570 INFO:teuthology.orchestra.run.smithi001.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T11:24:44.579 INFO:tasks.ceph.osd.3.smithi177.stderr:2024-09-13T11:24:44.575+0000 7f10adba4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26574) UID: 0 2024-09-13T11:24:44.597 INFO:teuthology.orchestra.run.smithi001.stdout:osd.7: {} 2024-09-13T11:24:44.597 INFO:teuthology.orchestra.run.smithi001.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T11:24:44.621 INFO:teuthology.orchestra.run.smithi001.stdout:osd.8: {} 2024-09-13T11:24:44.621 INFO:teuthology.orchestra.run.smithi001.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-13T11:24:44.627 INFO:teuthology.orchestra.run.smithi092.stdout:ERROR: (22) Invalid argument 2024-09-13T11:24:44.627 INFO:teuthology.orchestra.run.smithi092.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:24:44.646 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-13T11:24:44.649 INFO:teuthology.orchestra.run.smithi001.stdout:osd.9: {} 2024-09-13T11:24:44.649 INFO:teuthology.orchestra.run.smithi001.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-13T11:24:44.676 INFO:teuthology.orchestra.run.smithi001.stdout:osd.10: {} 2024-09-13T11:24:44.677 INFO:teuthology.orchestra.run.smithi001.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-13T11:24:44.680 INFO:tasks.ceph.osd.2.smithi162.stderr:2024-09-13T11:24:44.676+0000 7fe92b35a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26313) UID: 0 2024-09-13T11:24:44.702 INFO:teuthology.orchestra.run.smithi001.stdout:osd.11: {} 2024-09-13T11:24:44.703 INFO:teuthology.orchestra.run.smithi001.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-13T11:24:44.725 INFO:teuthology.orchestra.run.smithi001.stdout:osd.12: {} 2024-09-13T11:24:44.726 INFO:teuthology.orchestra.run.smithi001.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-13T11:24:44.753 INFO:teuthology.orchestra.run.smithi001.stdout:osd.13: {} 2024-09-13T11:24:44.753 INFO:teuthology.orchestra.run.smithi001.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-13T11:24:44.753 INFO:teuthology.orchestra.run.smithi001.stderr:Error ENXIO: problem getting command descriptions from osd.14 2024-09-13T11:24:44.776 INFO:teuthology.orchestra.run.smithi001.stdout:osd.15: {} 2024-09-13T11:24:44.776 INFO:teuthology.orchestra.run.smithi001.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-13T11:24:44.779 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-13T11:24:44.780 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-13T11:24:44.781 INFO:tasks.ceph.osd.15.smithi177.stderr:2024-09-13T11:24:44.779+0000 7f9d6ce77640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26578) UID: 0 2024-09-13T11:24:44.788 INFO:teuthology.orchestra.run.smithi092.stdout:ERROR: (22) Invalid argument 2024-09-13T11:24:44.788 INFO:teuthology.orchestra.run.smithi092.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:24:44.807 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-13T11:24:44.882 INFO:tasks.ceph.osd.12.smithi001.stderr:2024-09-13T11:24:44.879+0000 7f892c6e6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26314) UID: 0 2024-09-13T11:24:44.941 INFO:teuthology.orchestra.run.smithi092.stdout:ERROR: (22) Invalid argument 2024-09-13T11:24:44.941 INFO:teuthology.orchestra.run.smithi092.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:24:44.957 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-13T11:24:45.004 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-13T11:24:45.003+0000 7f6a68775640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26307) UID: 0 2024-09-13T11:24:45.012 INFO:tasks.radosbench.radosbench.0.smithi177.stdout: 138 16 127792 127776 57.8633 40.75 0.00520001 0.0172309 2024-09-13T11:24:45.083 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-09-13T11:24:45.080+0000 7f97befc9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26315) UID: 0 2024-09-13T11:24:45.112 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-13T11:24:45.112 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:24:45.130 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-13T11:24:45.184 INFO:tasks.ceph.osd.5.smithi092.stderr:2024-09-13T11:24:45.178+0000 7f6f0b705640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26346) UID: 0 2024-09-13T11:24:45.285 INFO:tasks.ceph.osd.9.smithi092.stderr:2024-09-13T11:24:45.278+0000 7ff5ab3d0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26348) UID: 0 2024-09-13T11:24:45.311 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-13T11:24:45.311 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:24:45.327 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-13T11:24:45.345 INFO:tasks.ceph.osd.4.smithi001.stderr:2024-09-13T11:24:45.343+0000 7ff8d2c41640 -1 osd.4 131 handle_osd_map injecting map crc failure 2024-09-13T11:24:45.386 INFO:tasks.ceph.osd.7.smithi177.stderr:2024-09-13T11:24:45.383+0000 7f897b51d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26575) UID: 0 2024-09-13T11:24:45.486 INFO:tasks.ceph.osd.5.smithi092.stderr:2024-09-13T11:24:45.482+0000 7f6f0b705640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26346) UID: 0 2024-09-13T11:24:45.507 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-13T11:24:45.507 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:24:45.531 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-13T11:24:45.587 INFO:tasks.ceph.osd.13.smithi092.stderr:2024-09-13T11:24:45.582+0000 7f3e5afea640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26350) UID: 0 2024-09-13T11:24:45.688 INFO:tasks.ceph.osd.13.smithi092.stderr:2024-09-13T11:24:45.682+0000 7f3e5afea640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26350) UID: 0 2024-09-13T11:24:45.732 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-13T11:24:45.732 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:24:45.751 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-13T11:24:45.796 INFO:tasks.ceph.osd.8.smithi001.stderr:2024-09-13T11:24:45.795+0000 7fb9b48ae640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26312) UID: 0 2024-09-13T11:24:45.797 INFO:teuthology.orchestra.run.smithi162.stdout:fsck success 2024-09-13T11:24:45.807 INFO:tasks.thrashosds.thrasher:resharding successfully completed 2024-09-13T11:24:45.807 INFO:tasks.thrashosds.thrasher:Reviving osd 14 2024-09-13T11:24:45.808 INFO:tasks.ceph.osd.14:Restarting daemon 2024-09-13T11:24:45.808 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 14 2024-09-13T11:24:45.851 INFO:tasks.ceph.osd.14:Started 2024-09-13T11:24:45.851 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_ops_in_flight 2024-09-13T11:24:45.892 INFO:tasks.ceph.osd.15.smithi177.stderr:2024-09-13T11:24:45.886+0000 7f9d6ce77640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26578) UID: 0 2024-09-13T11:24:45.909 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-13T11:24:45.910 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:24:45.929 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-13T11:24:45.986 INFO:teuthology.orchestra.run.smithi162.stderr:no valid command found; 10 closest matches: 2024-09-13T11:24:45.986 INFO:teuthology.orchestra.run.smithi162.stderr:0 2024-09-13T11:24:45.986 INFO:teuthology.orchestra.run.smithi162.stderr:1 2024-09-13T11:24:45.986 INFO:teuthology.orchestra.run.smithi162.stderr:2 2024-09-13T11:24:45.986 INFO:teuthology.orchestra.run.smithi162.stderr:abort 2024-09-13T11:24:45.986 INFO:teuthology.orchestra.run.smithi162.stderr:assert 2024-09-13T11:24:45.986 INFO:teuthology.orchestra.run.smithi162.stderr:config diff 2024-09-13T11:24:45.986 INFO:teuthology.orchestra.run.smithi162.stderr:config diff get 2024-09-13T11:24:45.986 INFO:teuthology.orchestra.run.smithi162.stderr:config get 2024-09-13T11:24:45.986 INFO:teuthology.orchestra.run.smithi162.stderr:config help [] 2024-09-13T11:24:45.986 INFO:teuthology.orchestra.run.smithi162.stderr:config set ... 2024-09-13T11:24:45.987 INFO:teuthology.orchestra.run.smithi162.stderr:admin_socket: invalid command 2024-09-13T11:24:45.988 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-13T11:24:45.988 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-14, ['dump_ops_in_flight'] 2024-09-13T11:24:46.012 INFO:tasks.radosbench.radosbench.0.smithi177.stdout: 139 16 128688 128672 57.8499 56 0.00521304 0.0172807 2024-09-13T11:24:46.016 INFO:tasks.ceph.osd.12.smithi001.stderr:2024-09-13T11:24:46.015+0000 7f892c6e6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26314) UID: 0 2024-09-13T11:24:46.093 INFO:tasks.ceph.osd.7.smithi177.stderr:2024-09-13T11:24:46.090+0000 7f897b51d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26575) UID: 0 2024-09-13T11:24:46.123 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-13T11:24:46.123 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:24:46.143 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-09-13T11:24:46.232 INFO:tasks.ceph.osd.8.smithi001.stderr:2024-09-13T11:24:46.231+0000 7fb9b48ae640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26312) UID: 0 2024-09-13T11:24:46.294 INFO:tasks.ceph.osd.2.smithi162.stderr:2024-09-13T11:24:46.292+0000 7fe92b35a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26313) UID: 0 2024-09-13T11:24:46.314 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-13T11:24:46.314 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:24:46.318 INFO:tasks.ceph.osd.7.smithi177.stderr:2024-09-13T11:24:46.314+0000 7f896ff4a640 -1 osd.7 132 handle_osd_map injecting map crc failure 2024-09-13T11:24:46.330 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-09-13T11:24:46.343 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-13T11:24:46.416 INFO:tasks.ceph.osd.8.smithi001.stderr:2024-09-13T11:24:46.415+0000 7fb9b48ae640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26312) UID: 0 2024-09-13T11:24:46.496 INFO:tasks.ceph.osd.10.smithi162.stderr:2024-09-13T11:24:46.492+0000 7fb59c9f0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26317) UID: 0 2024-09-13T11:24:46.530 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-13T11:24:46.530 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:24:46.552 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-09-13T11:24:46.597 INFO:tasks.ceph.osd.7.smithi177.stderr:2024-09-13T11:24:46.594+0000 7f897b51d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26575) UID: 0 2024-09-13T11:24:46.698 INFO:tasks.ceph.osd.2.smithi162.stderr:2024-09-13T11:24:46.696+0000 7fe92b35a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26313) UID: 0 2024-09-13T11:24:46.728 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-13T11:24:46.728 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:24:46.746 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_ops_in_flight 2024-09-13T11:24:46.781 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T11:24:46.799 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-09-13T11:24:46.796+0000 7f97befc9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26315) UID: 0 2024-09-13T11:24:46.889 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-13T11:24:46.890 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:24:46.900 INFO:tasks.ceph.osd.13.smithi092.stderr:2024-09-13T11:24:46.894+0000 7f3e5afea640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26350) UID: 0 2024-09-13T11:24:46.916 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_blocked_ops 2024-09-13T11:24:46.928 INFO:teuthology.orchestra.run.smithi001.stderr:nodeep-scrub is unset 2024-09-13T11:24:46.931 INFO:tasks.ceph.osd.15.smithi177.stderr:2024-09-13T11:24:46.926+0000 7f9d638a8640 -1 osd.15 133 handle_osd_map injecting map crc failure 2024-09-13T11:24:46.935 INFO:tasks.ceph.osd.3.smithi177.stderr:2024-09-13T11:24:46.930+0000 7f10a25d1640 -1 osd.3 133 handle_osd_map injecting map crc failure 2024-09-13T11:24:46.981 INFO:teuthology.orchestra.run.smithi001.stdout:osd.0: {} 2024-09-13T11:24:46.982 INFO:teuthology.orchestra.run.smithi001.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T11:24:47.001 INFO:tasks.ceph.osd.8.smithi001.stderr:2024-09-13T11:24:46.999+0000 7fb9b48ae640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26312) UID: 0 2024-09-13T11:24:47.006 INFO:teuthology.orchestra.run.smithi001.stdout:osd.1: {} 2024-09-13T11:24:47.006 INFO:teuthology.orchestra.run.smithi001.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T11:24:47.012 INFO:tasks.radosbench.radosbench.0.smithi177.stdout:2024-09-13T11:24:47.012702+0000 min lat: 0.00139681 max lat: 22.1835 avg lat: 0.0172628 2024-09-13T11:24:47.012 INFO:tasks.radosbench.radosbench.0.smithi177.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-13T11:24:47.012 INFO:tasks.radosbench.radosbench.0.smithi177.stdout: 140 16 129750 129734 57.9107 66.375 0.00369106 0.0172628 2024-09-13T11:24:47.044 INFO:teuthology.orchestra.run.smithi001.stdout:osd.2: {} 2024-09-13T11:24:47.045 INFO:teuthology.orchestra.run.smithi001.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T11:24:47.080 INFO:teuthology.orchestra.run.smithi001.stdout:osd.3: {} 2024-09-13T11:24:47.081 INFO:teuthology.orchestra.run.smithi001.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T11:24:47.089 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-13T11:24:47.089 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:24:47.101 INFO:tasks.ceph.osd.10.smithi162.stderr:2024-09-13T11:24:47.096+0000 7fb59c9f0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26317) UID: 0 2024-09-13T11:24:47.110 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_historic_ops 2024-09-13T11:24:47.112 INFO:teuthology.orchestra.run.smithi001.stdout:osd.4: {} 2024-09-13T11:24:47.112 INFO:teuthology.orchestra.run.smithi001.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T11:24:47.138 INFO:teuthology.orchestra.run.smithi001.stdout:osd.5: {} 2024-09-13T11:24:47.138 INFO:teuthology.orchestra.run.smithi001.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T11:24:47.164 INFO:teuthology.orchestra.run.smithi001.stdout:osd.6: {} 2024-09-13T11:24:47.164 INFO:teuthology.orchestra.run.smithi001.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T11:24:47.187 INFO:teuthology.orchestra.run.smithi001.stdout:osd.7: {} 2024-09-13T11:24:47.187 INFO:teuthology.orchestra.run.smithi001.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T11:24:47.202 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-09-13T11:24:47.200+0000 7f97befc9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26315) UID: 0 2024-09-13T11:24:47.208 INFO:teuthology.orchestra.run.smithi001.stdout:osd.8: {} 2024-09-13T11:24:47.208 INFO:teuthology.orchestra.run.smithi001.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-13T11:24:47.228 INFO:teuthology.orchestra.run.smithi001.stdout:osd.9: {} 2024-09-13T11:24:47.228 INFO:teuthology.orchestra.run.smithi001.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-13T11:24:47.236 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-13T11:24:47.236 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:24:47.249 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-13T11:24:47.253 INFO:teuthology.orchestra.run.smithi001.stdout:osd.10: {} 2024-09-13T11:24:47.253 INFO:teuthology.orchestra.run.smithi001.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-13T11:24:47.274 INFO:teuthology.orchestra.run.smithi001.stdout:osd.11: {} 2024-09-13T11:24:47.274 INFO:teuthology.orchestra.run.smithi001.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-13T11:24:47.295 INFO:teuthology.orchestra.run.smithi001.stdout:osd.12: {} 2024-09-13T11:24:47.295 INFO:teuthology.orchestra.run.smithi001.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-13T11:24:47.302 INFO:tasks.ceph.osd.15.smithi177.stderr:2024-09-13T11:24:47.298+0000 7f9d6ce77640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26578) UID: 0 2024-09-13T11:24:47.318 INFO:teuthology.orchestra.run.smithi001.stdout:osd.13: {} 2024-09-13T11:24:47.318 INFO:teuthology.orchestra.run.smithi001.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-13T11:24:47.321 INFO:teuthology.orchestra.run.smithi001.stderr:Error ENXIO: problem getting command descriptions from osd.14 2024-09-13T11:24:47.347 INFO:teuthology.orchestra.run.smithi001.stdout:osd.15: {} 2024-09-13T11:24:47.348 INFO:teuthology.orchestra.run.smithi001.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-13T11:24:47.351 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-13T11:24:47.351 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-13T11:24:47.386 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-09-13T11:24:47.387 INFO:teuthology.orchestra.run.smithi092.stdout: "ops": [], 2024-09-13T11:24:47.387 INFO:teuthology.orchestra.run.smithi092.stdout: "num_ops": 0 2024-09-13T11:24:47.387 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-09-13T11:24:47.403 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-09-13T11:24:47.400+0000 7f97befc9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26315) UID: 0 2024-09-13T11:24:47.408 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-13T11:24:47.505 INFO:tasks.ceph.osd.5.smithi092.stderr:2024-09-13T11:24:47.498+0000 7f6f0b705640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26346) UID: 0 2024-09-13T11:24:47.552 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-09-13T11:24:47.552 INFO:teuthology.orchestra.run.smithi092.stdout: "ops": [], 2024-09-13T11:24:47.552 INFO:teuthology.orchestra.run.smithi092.stdout: "complaint_time": 30, 2024-09-13T11:24:47.552 INFO:teuthology.orchestra.run.smithi092.stdout: "num_blocked_ops": 0 2024-09-13T11:24:47.552 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-09-13T11:24:47.570 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-13T11:24:47.605 INFO:tasks.ceph.osd.8.smithi001.stderr:2024-09-13T11:24:47.603+0000 7fb9b48ae640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26312) UID: 0 2024-09-13T11:24:47.706 INFO:tasks.ceph.osd.15.smithi177.stderr:2024-09-13T11:24:47.702+0000 7f9d6ce77640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26578) UID: 0 2024-09-13T11:24:47.747 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-09-13T11:24:47.748 INFO:teuthology.orchestra.run.smithi092.stdout: "size": 20, 2024-09-13T11:24:47.748 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 600, 2024-09-13T11:24:47.748 INFO:teuthology.orchestra.run.smithi092.stdout: "ops": [ 2024-09-13T11:24:47.748 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:47.748 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "MOSDPGPush(3.23 83/78 [PushOp(3:c61f6bab:::benchmark_data_smithi177_31116_object29479:head, version: 59'1860, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:c61f6bab:::benchmark_data_smithi177_31116_object29479:head@59'1860, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:24:47.748 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:40.770864+0000", 2024-09-13T11:24:47.748 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 66.974583955, 2024-09-13T11:24:47.748 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.31229662800000002, 2024-09-13T11:24:47.748 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:24:47.748 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:24:47.748 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-09-13T11:24:47.748 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:24:47.748 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:47.749 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:24:47.749 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:40.770864+0000", 2024-09-13T11:24:47.749 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:47.749 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:47.749 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:47.749 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:24:47.749 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:40.770864+0000", 2024-09-13T11:24:47.749 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:47.749 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:47.749 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:47.749 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:24:47.749 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:40.770865+0000", 2024-09-13T11:24:47.749 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.288e-06 2024-09-13T11:24:47.749 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:47.750 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:47.750 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:24:47.750 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:40.770955+0000", 2024-09-13T11:24:47.750 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.9871999999999996e-05 2024-09-13T11:24:47.750 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:47.750 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:47.750 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:24:47.750 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:41.082105+0000", 2024-09-13T11:24:47.750 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.31114963400000001 2024-09-13T11:24:47.750 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:47.750 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:47.750 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:24:47.750 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:41.082136+0000", 2024-09-13T11:24:47.750 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.0972000000000001e-05 2024-09-13T11:24:47.751 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:47.751 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:47.751 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:24:47.751 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:41.082404+0000", 2024-09-13T11:24:47.751 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00026844800000000002 2024-09-13T11:24:47.751 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:47.751 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:47.751 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:24:47.751 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:41.082429+0000", 2024-09-13T11:24:47.751 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.5021e-05 2024-09-13T11:24:47.751 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:47.751 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:47.751 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:24:47.751 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:41.083161+0000", 2024-09-13T11:24:47.751 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000731393 2024-09-13T11:24:47.752 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:47.752 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:24:47.752 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:47.752 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:47.752 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:47.752 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "MOSDPGPush(3.1e 85/78 [PushOp(3:7f40e925:::benchmark_data_smithi177_31116_object14084:head, version: 50'927, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:7f40e925:::benchmark_data_smithi177_31116_object14084:head@50'927, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:24:47.752 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:42.751000+0000", 2024-09-13T11:24:47.752 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 64.994447648000005, 2024-09-13T11:24:47.752 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.19208357500000001, 2024-09-13T11:24:47.752 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:24:47.752 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:24:47.752 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-09-13T11:24:47.752 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:24:47.753 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:47.753 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:24:47.753 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.751000+0000", 2024-09-13T11:24:47.753 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:47.753 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:47.753 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:47.753 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:24:47.753 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.751000+0000", 2024-09-13T11:24:47.753 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:47.753 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:47.753 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:47.753 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:24:47.753 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.751003+0000", 2024-09-13T11:24:47.753 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.0800000000000002e-06 2024-09-13T11:24:47.753 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:47.754 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:47.754 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:24:47.754 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.751069+0000", 2024-09-13T11:24:47.754 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.5417000000000001e-05 2024-09-13T11:24:47.754 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:47.754 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:47.754 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:24:47.754 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.941261+0000", 2024-09-13T11:24:47.754 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.19019233899999999 2024-09-13T11:24:47.754 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:47.754 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:47.754 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:24:47.754 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.941300+0000", 2024-09-13T11:24:47.754 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.9091999999999998e-05 2024-09-13T11:24:47.755 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:47.755 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:47.755 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:24:47.755 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.941519+0000", 2024-09-13T11:24:47.755 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000218521 2024-09-13T11:24:47.755 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:47.755 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:47.755 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:24:47.755 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.941544+0000", 2024-09-13T11:24:47.755 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.5412999999999999e-05 2024-09-13T11:24:47.755 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:47.755 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:47.755 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:24:47.755 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.943084+0000", 2024-09-13T11:24:47.756 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0015397130000000001 2024-09-13T11:24:47.756 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:47.756 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:24:47.756 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:47.756 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:47.756 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:47.756 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "MOSDPGPush(3.1e 90/78 [PushOp(3:7c65c13b:::benchmark_data_smithi177_31116_object22761:head, version: 55'1503, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:7c65c13b:::benchmark_data_smithi177_31116_object22761:head@55'1503, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:24:47.756 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:47.705735+0000", 2024-09-13T11:24:47.756 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 60.039712543, 2024-09-13T11:24:47.756 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.107567331, 2024-09-13T11:24:47.756 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:24:47.757 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:24:47.757 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-09-13T11:24:47.757 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:24:47.757 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:47.757 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:24:47.757 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:47.705735+0000", 2024-09-13T11:24:47.757 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:47.757 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:47.757 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:47.757 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:24:47.757 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:47.705735+0000", 2024-09-13T11:24:47.757 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:47.757 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:47.757 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:47.758 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:24:47.758 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:47.705738+0000", 2024-09-13T11:24:47.758 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.5940000000000001e-06 2024-09-13T11:24:47.758 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:47.758 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:47.758 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:24:47.758 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:47.705798+0000", 2024-09-13T11:24:47.758 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.9848e-05 2024-09-13T11:24:47.758 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:47.758 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:47.758 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:24:47.758 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:47.811473+0000", 2024-09-13T11:24:47.758 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.10567505100000001 2024-09-13T11:24:47.758 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:47.758 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:47.759 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:24:47.759 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:47.811488+0000", 2024-09-13T11:24:47.759 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.5311999999999998e-05 2024-09-13T11:24:47.759 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:47.759 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:47.759 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:24:47.759 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:47.811684+0000", 2024-09-13T11:24:47.759 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00019606100000000001 2024-09-13T11:24:47.759 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:47.759 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:47.759 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:24:47.759 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:47.811709+0000", 2024-09-13T11:24:47.759 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.5066e-05 2024-09-13T11:24:47.760 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:47.760 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:47.760 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:24:47.760 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:47.813303+0000", 2024-09-13T11:24:47.760 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0015933989999999999 2024-09-13T11:24:47.760 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:47.760 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:24:47.760 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:47.760 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:47.760 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:47.760 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:82769 3.1e 3:7a4e88b5:::benchmark_data_smithi177_31116_object82768:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105)", 2024-09-13T11:24:47.760 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:24:03.744632+0000", 2024-09-13T11:24:47.760 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 44.000815502999998, 2024-09-13T11:24:47.760 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.21475722699999999, 2024-09-13T11:24:47.761 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:24:47.761 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:24:47.761 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:47.761 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:24:47.761 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:24:47.761 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:47.761 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 82769 2024-09-13T11:24:47.761 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:47.761 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:24:47.761 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:47.761 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:24:47.761 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:03.744632+0000", 2024-09-13T11:24:47.761 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:47.761 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:47.762 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:47.762 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:24:47.762 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:03.744632+0000", 2024-09-13T11:24:47.762 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:47.762 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:47.762 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:47.762 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:24:47.762 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:03.744634+0000", 2024-09-13T11:24:47.762 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.672e-06 2024-09-13T11:24:47.762 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:47.762 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:47.762 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:24:47.762 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:03.744724+0000", 2024-09-13T11:24:47.762 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.0237999999999999e-05 2024-09-13T11:24:47.763 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:47.763 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:47.763 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:24:47.763 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:03.744736+0000", 2024-09-13T11:24:47.763 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.2038999999999999e-05 2024-09-13T11:24:47.763 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:47.763 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:47.763 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:24:47.763 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:03.744742+0000", 2024-09-13T11:24:47.763 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.2129999999999996e-06 2024-09-13T11:24:47.763 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:47.763 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:47.763 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:24:47.763 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:03.744785+0000", 2024-09-13T11:24:47.763 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.2898000000000002e-05 2024-09-13T11:24:47.764 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:47.764 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:47.764 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:24:47.764 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:03.744894+0000", 2024-09-13T11:24:47.764 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000109073 2024-09-13T11:24:47.764 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:47.764 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:47.764 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 4", 2024-09-13T11:24:47.764 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:03.745009+0000", 2024-09-13T11:24:47.764 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00011428499999999999 2024-09-13T11:24:47.764 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:47.764 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:47.764 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:24:47.765 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:03.746451+0000", 2024-09-13T11:24:47.765 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0014419179999999999 2024-09-13T11:24:47.765 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:47.765 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:47.765 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:47.765 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:03.959257+0000", 2024-09-13T11:24:47.765 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.212806361 2024-09-13T11:24:47.765 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:47.765 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:47.765 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:24:47.765 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:03.959333+0000", 2024-09-13T11:24:47.765 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.6001000000000006e-05 2024-09-13T11:24:47.765 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:47.765 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:47.766 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:24:47.766 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:03.959390+0000", 2024-09-13T11:24:47.766 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.6529e-05 2024-09-13T11:24:47.766 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:47.766 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:24:47.766 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:47.766 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:47.766 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:47.766 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:104331 3.1e 3:79890035:::benchmark_data_smithi177_31116_object104330:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-09-13T11:24:47.766 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:24:20.241529+0000", 2024-09-13T11:24:47.766 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 27.503918984999999, 2024-09-13T11:24:47.766 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.16976961700000001, 2024-09-13T11:24:47.766 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:24:47.766 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:24:47.766 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:47.767 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:24:47.767 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:24:47.767 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:47.767 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 104331 2024-09-13T11:24:47.767 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:47.767 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:24:47.767 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:47.767 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:24:47.767 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:20.241529+0000", 2024-09-13T11:24:47.767 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:47.767 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:47.767 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:47.767 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:24:47.767 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:20.241529+0000", 2024-09-13T11:24:47.768 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:47.768 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:47.768 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:47.768 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:24:47.768 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:20.241534+0000", 2024-09-13T11:24:47.768 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.775e-06 2024-09-13T11:24:47.768 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:47.768 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:47.768 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:24:47.768 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:20.241660+0000", 2024-09-13T11:24:47.768 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000126499 2024-09-13T11:24:47.768 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:47.768 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:47.768 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:24:47.769 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:20.241687+0000", 2024-09-13T11:24:47.769 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.6826999999999999e-05 2024-09-13T11:24:47.769 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:47.769 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:47.769 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:24:47.769 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:20.241703+0000", 2024-09-13T11:24:47.769 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.5858999999999998e-05 2024-09-13T11:24:47.769 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:47.769 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:47.769 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:24:47.769 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:20.241832+0000", 2024-09-13T11:24:47.769 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00012925799999999999 2024-09-13T11:24:47.769 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:47.770 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:47.770 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:24:47.770 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:20.242133+0000", 2024-09-13T11:24:47.770 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00030115900000000001 2024-09-13T11:24:47.770 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:47.770 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:47.770 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 4", 2024-09-13T11:24:47.770 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:20.242408+0000", 2024-09-13T11:24:47.770 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00027487799999999999 2024-09-13T11:24:47.770 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:47.770 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:47.770 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:24:47.770 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:20.246683+0000", 2024-09-13T11:24:47.770 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0042750469999999997 2024-09-13T11:24:47.770 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:47.771 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:47.771 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:47.771 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:20.411076+0000", 2024-09-13T11:24:47.771 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.16439300800000001 2024-09-13T11:24:47.771 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:47.771 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:47.771 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:24:47.771 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:20.411188+0000", 2024-09-13T11:24:47.771 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00011191 2024-09-13T11:24:47.771 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:47.771 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:47.771 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:24:47.771 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:20.411298+0000", 2024-09-13T11:24:47.771 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00011039699999999999 2024-09-13T11:24:47.771 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:47.772 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:24:47.772 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:47.772 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:47.772 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:47.772 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:104823 3.1e 3:7a73472c:::benchmark_data_smithi177_31116_object104822:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-09-13T11:24:47.772 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:24:20.661312+0000", 2024-09-13T11:24:47.772 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 27.084136189999999, 2024-09-13T11:24:47.772 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.10093226499999999, 2024-09-13T11:24:47.772 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:24:47.772 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:24:47.772 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:47.772 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:24:47.772 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:24:47.773 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:47.773 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 104823 2024-09-13T11:24:47.773 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:47.773 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:24:47.773 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:47.773 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:24:47.773 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:20.661312+0000", 2024-09-13T11:24:47.773 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:47.773 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:47.773 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:47.773 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:24:47.773 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:20.661312+0000", 2024-09-13T11:24:47.773 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:47.773 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:47.773 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:47.774 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:24:47.774 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:20.661314+0000", 2024-09-13T11:24:47.774 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.5519999999999999e-06 2024-09-13T11:24:47.774 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:47.774 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:47.774 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:24:47.774 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:20.661381+0000", 2024-09-13T11:24:47.774 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.7246e-05 2024-09-13T11:24:47.774 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:47.774 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:47.774 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:24:47.774 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:20.661399+0000", 2024-09-13T11:24:47.774 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.8029e-05 2024-09-13T11:24:47.774 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:47.775 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:47.775 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:24:47.775 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:20.661407+0000", 2024-09-13T11:24:47.775 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.7239999999999992e-06 2024-09-13T11:24:47.775 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:47.775 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:47.775 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:24:47.775 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:20.661486+0000", 2024-09-13T11:24:47.775 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.8461999999999995e-05 2024-09-13T11:24:47.775 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:47.775 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:47.775 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:24:47.775 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:20.661579+0000", 2024-09-13T11:24:47.775 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.2941000000000003e-05 2024-09-13T11:24:47.776 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:47.776 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:47.776 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 4", 2024-09-13T11:24:47.776 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:20.661664+0000", 2024-09-13T11:24:47.776 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.5772000000000005e-05 2024-09-13T11:24:47.776 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:47.776 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:47.776 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:24:47.776 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:20.663376+0000", 2024-09-13T11:24:47.776 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.001711909 2024-09-13T11:24:47.776 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:47.776 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:47.776 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:47.777 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:20.762129+0000", 2024-09-13T11:24:47.777 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.098753143000000002 2024-09-13T11:24:47.777 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:47.777 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:47.777 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:24:47.777 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:20.762198+0000", 2024-09-13T11:24:47.777 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.8491000000000005e-05 2024-09-13T11:24:47.777 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:47.777 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:47.777 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:24:47.777 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:20.762244+0000", 2024-09-13T11:24:47.777 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.5995999999999998e-05 2024-09-13T11:24:47.777 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:47.777 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:24:47.778 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:47.778 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:47.778 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:47.778 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:110289 3.29 3.27287969 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e119)", 2024-09-13T11:24:47.778 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:24:25.774634+0000", 2024-09-13T11:24:47.778 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 21.970813880000001, 2024-09-13T11:24:47.778 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.31721633999999999, 2024-09-13T11:24:47.778 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:24:47.778 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:24:47.778 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "reached pg", 2024-09-13T11:24:47.778 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:24:47.778 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:24:47.778 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:47.778 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 110289 2024-09-13T11:24:47.779 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:47.779 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:24:47.779 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:47.779 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:24:47.779 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:25.774634+0000", 2024-09-13T11:24:47.779 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:47.779 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:47.779 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:47.779 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:24:47.779 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:25.774634+0000", 2024-09-13T11:24:47.779 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:47.779 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:47.779 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:47.779 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:24:47.779 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:25.774639+0000", 2024-09-13T11:24:47.780 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.6029999999999998e-06 2024-09-13T11:24:47.780 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:47.780 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:47.780 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:24:47.780 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:25.774695+0000", 2024-09-13T11:24:47.780 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.6694999999999998e-05 2024-09-13T11:24:47.780 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:47.780 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:47.780 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:24:47.780 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:25.774728+0000", 2024-09-13T11:24:47.780 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.2502000000000002e-05 2024-09-13T11:24:47.780 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:47.780 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:47.780 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:24:47.781 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:25.774750+0000", 2024-09-13T11:24:47.781 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.2002999999999999e-05 2024-09-13T11:24:47.781 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:47.781 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:47.781 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:24:47.781 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:25.774942+0000", 2024-09-13T11:24:47.781 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000192265 2024-09-13T11:24:47.781 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:47.781 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:47.781 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-09-13T11:24:47.781 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:25.774960+0000", 2024-09-13T11:24:47.781 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.8275999999999999e-05 2024-09-13T11:24:47.781 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:47.781 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:47.782 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:24:47.782 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:26.091821+0000", 2024-09-13T11:24:47.782 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.31686065000000002 2024-09-13T11:24:47.782 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:47.782 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:47.782 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:24:47.782 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:26.091850+0000", 2024-09-13T11:24:47.782 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.9346e-05 2024-09-13T11:24:47.782 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:47.782 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:24:47.782 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:47.782 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:47.782 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:47.782 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "MOSDPGPush(3.22 125/122 [PushOp(3:44e4f9f2:::benchmark_data_smithi177_31116_object16663:head, version: 52'1032, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:44e4f9f2:::benchmark_data_smithi177_31116_object16663:head@52'1032, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:24:47.783 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:24:32.743254+0000", 2024-09-13T11:24:47.783 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 15.002193791, 2024-09-13T11:24:47.783 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.215485287, 2024-09-13T11:24:47.783 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:24:47.783 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:24:47.783 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-09-13T11:24:47.783 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:24:47.783 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:47.783 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:24:47.783 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:32.743254+0000", 2024-09-13T11:24:47.783 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:47.783 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:47.783 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:47.783 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:24:47.783 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:32.743254+0000", 2024-09-13T11:24:47.784 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:47.784 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:47.784 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:47.784 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:24:47.784 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:32.743255+0000", 2024-09-13T11:24:47.784 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.2500000000000004e-07 2024-09-13T11:24:47.784 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:47.784 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:47.784 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:24:47.784 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:32.743310+0000", 2024-09-13T11:24:47.784 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.5442000000000002e-05 2024-09-13T11:24:47.784 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:47.784 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:47.784 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:24:47.784 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:32.957754+0000", 2024-09-13T11:24:47.785 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.21444387000000001 2024-09-13T11:24:47.785 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:47.785 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:47.785 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:24:47.785 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:32.957765+0000", 2024-09-13T11:24:47.785 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0458999999999999e-05 2024-09-13T11:24:47.785 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:47.785 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:47.785 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:24:47.785 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:32.957841+0000", 2024-09-13T11:24:47.785 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.5993000000000002e-05 2024-09-13T11:24:47.785 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:47.785 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:47.785 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:24:47.786 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:32.957852+0000", 2024-09-13T11:24:47.786 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.1453e-05 2024-09-13T11:24:47.786 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:47.786 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:47.786 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:24:47.786 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:32.958739+0000", 2024-09-13T11:24:47.786 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00088724500000000005 2024-09-13T11:24:47.786 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:47.786 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:24:47.786 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:47.786 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:47.786 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:47.786 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:114929 3.1e 3:7a39ebc4:::benchmark_data_smithi177_31116_object114928:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e125)", 2024-09-13T11:24:47.786 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:24:33.446684+0000", 2024-09-13T11:24:47.787 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 14.298763925999999, 2024-09-13T11:24:47.787 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.10317303, 2024-09-13T11:24:47.787 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:24:47.787 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:24:47.787 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:47.787 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:24:47.787 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:24:47.787 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:47.787 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 114929 2024-09-13T11:24:47.787 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:47.788 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:24:47.788 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:47.788 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:24:47.788 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:33.446684+0000", 2024-09-13T11:24:47.788 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:47.788 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:47.788 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:47.788 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:24:47.788 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:33.446684+0000", 2024-09-13T11:24:47.788 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:47.788 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:47.788 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:47.788 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:24:47.788 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:33.446686+0000", 2024-09-13T11:24:47.789 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.0040000000000002e-06 2024-09-13T11:24:47.789 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:47.789 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:47.789 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:24:47.789 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:33.446731+0000", 2024-09-13T11:24:47.789 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.4916000000000002e-05 2024-09-13T11:24:47.789 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:47.789 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:47.789 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:24:47.789 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:33.446745+0000", 2024-09-13T11:24:47.789 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.418e-05 2024-09-13T11:24:47.789 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:47.789 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:47.789 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:24:47.789 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:33.446752+0000", 2024-09-13T11:24:47.790 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.3150000000000003e-06 2024-09-13T11:24:47.790 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:47.790 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:47.790 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:24:47.790 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:33.446819+0000", 2024-09-13T11:24:47.790 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.6742000000000006e-05 2024-09-13T11:24:47.790 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:47.790 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:47.790 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:24:47.790 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:33.446964+0000", 2024-09-13T11:24:47.790 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00014518199999999999 2024-09-13T11:24:47.790 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:47.790 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:47.790 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 4", 2024-09-13T11:24:47.791 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:33.447106+0000", 2024-09-13T11:24:47.791 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000141791 2024-09-13T11:24:47.791 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:47.791 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:47.791 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:24:47.791 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:33.449960+0000", 2024-09-13T11:24:47.791 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.002853782 2024-09-13T11:24:47.791 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:47.791 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:47.791 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:47.791 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:33.549557+0000", 2024-09-13T11:24:47.791 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.099596911999999996 2024-09-13T11:24:47.791 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:47.791 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:47.792 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:24:47.792 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:33.549610+0000", 2024-09-13T11:24:47.792 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.312e-05 2024-09-13T11:24:47.792 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:47.792 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:47.792 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:24:47.792 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:33.549857+0000", 2024-09-13T11:24:47.792 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00024708599999999999 2024-09-13T11:24:47.792 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:47.792 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:24:47.792 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:47.792 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:47.792 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:47.792 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop(client.4665.0:114934 3.28 e125/120 3:156be749:::benchmark_data_smithi177_31116_object114933:head v 125'4840, mlcod=47'620)", 2024-09-13T11:24:47.793 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:24:33.454671+0000", 2024-09-13T11:24:47.793 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 14.290776534999999, 2024-09-13T11:24:47.793 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.10349704999999999, 2024-09-13T11:24:47.793 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:24:47.793 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:24:47.793 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:47.793 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:24:47.793 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:47.793 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:24:47.793 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:33.454671+0000", 2024-09-13T11:24:47.793 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:47.793 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:47.793 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:47.793 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:24:47.794 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:33.454671+0000", 2024-09-13T11:24:47.794 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:47.794 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:47.794 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:47.794 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:24:47.794 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:33.454673+0000", 2024-09-13T11:24:47.794 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.3820000000000001e-06 2024-09-13T11:24:47.794 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:47.794 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:47.794 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:24:47.794 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:33.454707+0000", 2024-09-13T11:24:47.794 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.4808000000000003e-05 2024-09-13T11:24:47.794 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:47.794 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:47.794 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:24:47.795 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:33.555701+0000", 2024-09-13T11:24:47.795 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.100993755 2024-09-13T11:24:47.795 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:47.795 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:47.795 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:24:47.795 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:33.555708+0000", 2024-09-13T11:24:47.795 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.9530000000000002e-06 2024-09-13T11:24:47.795 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:47.795 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:47.795 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:24:47.795 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:33.555782+0000", 2024-09-13T11:24:47.795 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.3763000000000005e-05 2024-09-13T11:24:47.795 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:47.795 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:47.796 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:24:47.796 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:33.555808+0000", 2024-09-13T11:24:47.796 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.62e-05 2024-09-13T11:24:47.796 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:47.796 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:47.796 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:24:47.796 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:33.558122+0000", 2024-09-13T11:24:47.796 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0023143230000000001 2024-09-13T11:24:47.796 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:47.796 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:47.796 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:24:47.796 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:33.558168+0000", 2024-09-13T11:24:47.796 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.5865999999999998e-05 2024-09-13T11:24:47.796 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:47.797 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:24:47.797 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:47.797 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:47.797 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:47.797 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:115008 3.1e 3:7a0cff82:::benchmark_data_smithi177_31116_object115007:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e125)", 2024-09-13T11:24:47.797 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:24:33.547002+0000", 2024-09-13T11:24:47.797 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 14.198446154999999, 2024-09-13T11:24:47.797 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.099273717999999997, 2024-09-13T11:24:47.797 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:24:47.797 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:24:47.797 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:47.797 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:24:47.797 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:24:47.797 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:47.798 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 115008 2024-09-13T11:24:47.798 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:47.798 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:24:47.798 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:47.798 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:24:47.798 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:33.547002+0000", 2024-09-13T11:24:47.798 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:47.798 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:47.798 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:47.798 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:24:47.798 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:33.547002+0000", 2024-09-13T11:24:47.798 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:47.798 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:47.798 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:47.798 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:24:47.799 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:33.547016+0000", 2024-09-13T11:24:47.799 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.4429e-05 2024-09-13T11:24:47.799 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:47.799 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:47.799 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:24:47.799 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:33.547073+0000", 2024-09-13T11:24:47.799 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.7370000000000001e-05 2024-09-13T11:24:47.799 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:47.799 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:47.799 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:24:47.799 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:33.547085+0000", 2024-09-13T11:24:47.799 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.1891e-05 2024-09-13T11:24:47.799 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:47.799 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:47.800 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:24:47.800 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:33.547092+0000", 2024-09-13T11:24:47.800 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.0090000000000001e-06 2024-09-13T11:24:47.800 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:47.800 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:47.800 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:24:47.800 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:33.547149+0000", 2024-09-13T11:24:47.800 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.6508999999999997e-05 2024-09-13T11:24:47.800 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:47.800 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:47.800 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:24:47.800 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:33.547277+0000", 2024-09-13T11:24:47.800 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00012800800000000001 2024-09-13T11:24:47.801 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:47.801 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:47.801 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 4", 2024-09-13T11:24:47.801 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:33.547420+0000", 2024-09-13T11:24:47.801 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00014338900000000001 2024-09-13T11:24:47.801 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:47.801 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:47.801 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:24:47.801 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:33.550093+0000", 2024-09-13T11:24:47.801 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0026726570000000002 2024-09-13T11:24:47.801 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:47.801 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:47.801 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:47.801 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:33.646180+0000", 2024-09-13T11:24:47.802 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.096086798000000001 2024-09-13T11:24:47.802 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:47.802 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:47.802 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:24:47.802 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:33.646233+0000", 2024-09-13T11:24:47.802 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.3291000000000002e-05 2024-09-13T11:24:47.802 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:47.802 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:47.802 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:24:47.802 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:33.646275+0000", 2024-09-13T11:24:47.802 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.2367000000000003e-05 2024-09-13T11:24:47.802 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:47.802 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:24:47.802 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:47.803 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:47.803 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:47.803 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop(client.4665.0:115089 3.28 e125/120 3:15b13357:::benchmark_data_smithi177_31116_object115088:head v 125'4843, mlcod=47'620)", 2024-09-13T11:24:47.803 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:24:33.642654+0000", 2024-09-13T11:24:47.803 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 14.102793381, 2024-09-13T11:24:47.803 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.19312220099999999, 2024-09-13T11:24:47.803 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:24:47.803 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:24:47.803 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:47.803 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:24:47.803 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:47.803 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:24:47.803 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:33.642654+0000", 2024-09-13T11:24:47.803 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:47.803 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:47.804 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:47.804 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:24:47.804 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:33.642654+0000", 2024-09-13T11:24:47.804 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:47.804 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:47.804 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:47.804 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:24:47.804 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:33.642656+0000", 2024-09-13T11:24:47.804 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.869e-06 2024-09-13T11:24:47.804 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:47.804 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:47.804 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:24:47.804 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:33.642688+0000", 2024-09-13T11:24:47.805 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.2169999999999999e-05 2024-09-13T11:24:47.805 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:47.805 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:47.805 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:24:47.805 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:33.832657+0000", 2024-09-13T11:24:47.805 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.18996885099999999 2024-09-13T11:24:47.805 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:47.805 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:47.805 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:24:47.805 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:33.832678+0000", 2024-09-13T11:24:47.805 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.0222000000000001e-05 2024-09-13T11:24:47.805 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:47.805 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:47.805 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:24:47.806 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:33.832760+0000", 2024-09-13T11:24:47.806 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.2113000000000005e-05 2024-09-13T11:24:47.806 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:47.806 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:47.806 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:24:47.806 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:33.832788+0000", 2024-09-13T11:24:47.806 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.8095000000000001e-05 2024-09-13T11:24:47.806 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:47.806 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:47.806 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:24:47.806 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:33.835729+0000", 2024-09-13T11:24:47.806 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0029408920000000001 2024-09-13T11:24:47.806 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:47.806 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:47.806 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:24:47.807 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:33.835777+0000", 2024-09-13T11:24:47.807 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.7988999999999997e-05 2024-09-13T11:24:47.807 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:47.807 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:24:47.807 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:47.807 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:47.807 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:47.807 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:115091 3.1e 3:7d850286:::benchmark_data_smithi177_31116_object115090:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e125)", 2024-09-13T11:24:47.807 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:24:33.643826+0000", 2024-09-13T11:24:47.807 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 14.101622334, 2024-09-13T11:24:47.807 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.10297139199999999, 2024-09-13T11:24:47.807 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:24:47.807 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:24:47.807 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:47.808 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:24:47.808 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:24:47.808 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:47.808 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 115091 2024-09-13T11:24:47.808 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:47.808 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:24:47.808 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:47.808 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:24:47.808 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:33.643826+0000", 2024-09-13T11:24:47.808 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:47.808 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:47.808 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:47.808 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:24:47.808 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:33.643826+0000", 2024-09-13T11:24:47.809 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:47.809 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:47.809 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:47.809 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:24:47.809 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:33.643827+0000", 2024-09-13T11:24:47.809 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.8339999999999999e-06 2024-09-13T11:24:47.809 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:47.809 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:47.809 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:24:47.809 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:33.643874+0000", 2024-09-13T11:24:47.809 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.6579999999999998e-05 2024-09-13T11:24:47.809 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:47.809 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:47.809 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:24:47.810 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:33.643889+0000", 2024-09-13T11:24:47.810 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.4601e-05 2024-09-13T11:24:47.810 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:47.810 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:47.810 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:24:47.810 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:33.643895+0000", 2024-09-13T11:24:47.810 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.4439999999999996e-06 2024-09-13T11:24:47.810 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:47.810 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:47.810 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:24:47.810 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:33.643949+0000", 2024-09-13T11:24:47.810 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.4150999999999997e-05 2024-09-13T11:24:47.810 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:47.811 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:47.811 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:24:47.811 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:33.644110+0000", 2024-09-13T11:24:47.811 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00016054799999999999 2024-09-13T11:24:47.811 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:47.811 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:47.811 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 4", 2024-09-13T11:24:47.811 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:33.644254+0000", 2024-09-13T11:24:47.811 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00014396299999999999 2024-09-13T11:24:47.811 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:47.811 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:47.811 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:24:47.811 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:33.646290+0000", 2024-09-13T11:24:47.811 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0020365890000000001 2024-09-13T11:24:47.811 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:47.812 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:47.812 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:47.812 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:33.746691+0000", 2024-09-13T11:24:47.812 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.100400906 2024-09-13T11:24:47.812 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:47.812 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:47.812 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:24:47.812 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:33.746750+0000", 2024-09-13T11:24:47.812 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.8622000000000002e-05 2024-09-13T11:24:47.812 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:47.812 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:47.812 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:24:47.812 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:33.746797+0000", 2024-09-13T11:24:47.812 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.7154000000000003e-05 2024-09-13T11:24:47.813 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:47.813 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:24:47.813 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:47.813 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:47.813 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:47.813 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop(client.4665.0:115281 3.28 e125/120 3:17aeecb4:::benchmark_data_smithi177_31116_object115280:head v 125'4845, mlcod=47'620)", 2024-09-13T11:24:47.813 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:24:33.852156+0000", 2024-09-13T11:24:47.813 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 13.893291981000001, 2024-09-13T11:24:47.813 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.16574401999999999, 2024-09-13T11:24:47.813 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:24:47.813 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:24:47.813 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:47.813 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:24:47.813 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:47.814 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:24:47.814 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:33.852156+0000", 2024-09-13T11:24:47.814 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:47.814 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:47.814 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:47.814 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:24:47.814 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:33.852156+0000", 2024-09-13T11:24:47.814 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:47.814 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:47.814 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:47.814 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:24:47.814 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:33.852158+0000", 2024-09-13T11:24:47.814 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.683e-06 2024-09-13T11:24:47.814 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:47.814 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:47.815 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:24:47.815 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:33.852184+0000", 2024-09-13T11:24:47.815 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.6241000000000002e-05 2024-09-13T11:24:47.815 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:47.815 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:47.815 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:24:47.815 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:34.015751+0000", 2024-09-13T11:24:47.815 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.16356685800000001 2024-09-13T11:24:47.815 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:47.815 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:47.815 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:24:47.815 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:34.015757+0000", 2024-09-13T11:24:47.815 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.8980000000000001e-06 2024-09-13T11:24:47.815 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:47.816 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:47.816 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:24:47.816 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:34.015800+0000", 2024-09-13T11:24:47.816 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.3371999999999997e-05 2024-09-13T11:24:47.816 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:47.816 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:47.816 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:24:47.816 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:34.015822+0000", 2024-09-13T11:24:47.816 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.2565000000000002e-05 2024-09-13T11:24:47.816 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:47.816 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:47.816 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:24:47.816 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:34.017861+0000", 2024-09-13T11:24:47.816 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0020388469999999999 2024-09-13T11:24:47.817 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:47.817 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:47.817 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:24:47.817 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:34.017900+0000", 2024-09-13T11:24:47.817 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.8556000000000002e-05 2024-09-13T11:24:47.817 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:47.817 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:24:47.817 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:47.817 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:47.817 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:47.817 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "MOSDPGPush(3.12 130/122 [PushOp(3:4f16581d:::benchmark_data_smithi177_31116_object22739:head, version: 55'1420, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:4f16581d:::benchmark_data_smithi177_31116_object22739:head@55'1420, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:24:47.817 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:24:42.447824+0000", 2024-09-13T11:24:47.817 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 5.297624162, 2024-09-13T11:24:47.817 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.10016381100000001, 2024-09-13T11:24:47.817 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:24:47.818 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:24:47.818 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-09-13T11:24:47.818 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:24:47.818 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:47.818 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:24:47.818 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:42.447824+0000", 2024-09-13T11:24:47.818 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:47.818 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:47.818 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:47.818 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:24:47.818 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:42.447824+0000", 2024-09-13T11:24:47.818 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:47.818 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:47.818 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:47.819 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:24:47.819 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:42.447825+0000", 2024-09-13T11:24:47.819 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.006e-06 2024-09-13T11:24:47.819 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:47.819 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:47.819 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:24:47.819 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:42.447894+0000", 2024-09-13T11:24:47.819 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.9251000000000005e-05 2024-09-13T11:24:47.819 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:47.819 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:47.819 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:24:47.819 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:42.546740+0000", 2024-09-13T11:24:47.819 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.098846209000000004 2024-09-13T11:24:47.819 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:47.819 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:47.820 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:24:47.820 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:42.546750+0000", 2024-09-13T11:24:47.820 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.8190000000000003e-06 2024-09-13T11:24:47.820 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:47.820 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:47.820 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:24:47.820 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:42.546819+0000", 2024-09-13T11:24:47.820 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.8721999999999997e-05 2024-09-13T11:24:47.820 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:47.820 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:47.820 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:24:47.820 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:42.546826+0000", 2024-09-13T11:24:47.820 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.0199999999999997e-06 2024-09-13T11:24:47.821 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:47.821 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:47.821 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:24:47.821 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:42.547987+0000", 2024-09-13T11:24:47.821 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0011617839999999999 2024-09-13T11:24:47.821 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:47.821 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:24:47.821 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:47.821 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:47.821 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:47.821 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop(client.4665.0:125282 3.12 e130/122 3:496a42f2:::benchmark_data_smithi177_31116_object125281:head v 130'5273, mlcod=55'1419)", 2024-09-13T11:24:47.821 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:24:42.451948+0000", 2024-09-13T11:24:47.821 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 5.2934999740000004, 2024-09-13T11:24:47.821 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.102689248, 2024-09-13T11:24:47.821 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:24:47.822 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:24:47.822 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:47.822 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:24:47.822 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:47.822 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:24:47.822 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:42.451948+0000", 2024-09-13T11:24:47.822 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:47.822 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:47.822 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:47.822 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:24:47.822 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:42.451948+0000", 2024-09-13T11:24:47.822 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:47.822 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:47.822 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:47.823 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:24:47.823 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:42.451949+0000", 2024-09-13T11:24:47.823 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.314e-06 2024-09-13T11:24:47.823 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:47.823 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:47.823 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:24:47.823 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:42.451998+0000", 2024-09-13T11:24:47.823 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.8741e-05 2024-09-13T11:24:47.823 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:47.823 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:47.823 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:24:47.823 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:42.551348+0000", 2024-09-13T11:24:47.823 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.099350529000000007 2024-09-13T11:24:47.823 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:47.823 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:47.824 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:24:47.824 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:42.551355+0000", 2024-09-13T11:24:47.824 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.3679999999999998e-06 2024-09-13T11:24:47.824 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:47.824 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:47.824 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:24:47.824 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:42.551402+0000", 2024-09-13T11:24:47.824 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.7389000000000003e-05 2024-09-13T11:24:47.824 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:47.824 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:47.824 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:24:47.824 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:42.551419+0000", 2024-09-13T11:24:47.824 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.7062e-05 2024-09-13T11:24:47.824 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:47.825 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:47.825 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:24:47.825 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:42.554604+0000", 2024-09-13T11:24:47.825 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0031842860000000001 2024-09-13T11:24:47.825 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:47.825 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:47.825 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:24:47.825 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:42.554637+0000", 2024-09-13T11:24:47.825 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.3559000000000002e-05 2024-09-13T11:24:47.825 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:47.825 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:24:47.825 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:47.825 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:47.825 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:47.826 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop(client.4665.0:126588 3.22 e130/122 3:44abe2fb:::benchmark_data_smithi177_31116_object126587:head v 130'4217, mlcod=114'3848)", 2024-09-13T11:24:47.826 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:24:43.481313+0000", 2024-09-13T11:24:47.826 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 4.2641348130000001, 2024-09-13T11:24:47.826 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.128388699, 2024-09-13T11:24:47.826 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:24:47.826 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:24:47.826 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:47.826 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:24:47.826 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:47.826 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:24:47.826 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:43.481313+0000", 2024-09-13T11:24:47.826 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:47.826 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:47.826 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:47.827 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:24:47.827 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:43.481313+0000", 2024-09-13T11:24:47.827 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:47.827 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:47.827 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:47.827 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:24:47.827 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:43.481315+0000", 2024-09-13T11:24:47.827 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.517e-06 2024-09-13T11:24:47.827 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:47.827 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:47.827 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:24:47.827 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:43.481373+0000", 2024-09-13T11:24:47.827 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.8628000000000002e-05 2024-09-13T11:24:47.827 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:47.827 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:47.828 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:24:47.828 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:43.608269+0000", 2024-09-13T11:24:47.828 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.126896276 2024-09-13T11:24:47.828 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:47.828 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:47.828 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:24:47.828 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:43.608273+0000", 2024-09-13T11:24:47.828 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.4769999999999999e-06 2024-09-13T11:24:47.828 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:47.828 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:47.828 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:24:47.828 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:43.608311+0000", 2024-09-13T11:24:47.828 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.7818999999999998e-05 2024-09-13T11:24:47.828 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:47.829 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:47.829 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:24:47.829 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:43.608329+0000", 2024-09-13T11:24:47.829 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.8369999999999999e-05 2024-09-13T11:24:47.829 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:47.829 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:47.829 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:24:47.829 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:43.609662+0000", 2024-09-13T11:24:47.829 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.001333337 2024-09-13T11:24:47.829 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:47.829 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:47.829 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:24:47.829 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:43.609702+0000", 2024-09-13T11:24:47.829 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.9274999999999999e-05 2024-09-13T11:24:47.830 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:47.830 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:24:47.830 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:47.830 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:47.830 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:47.830 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:126682 3.1e 3:793d3c71:::benchmark_data_smithi177_31116_object126681:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e130)", 2024-09-13T11:24:47.830 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:24:43.563277+0000", 2024-09-13T11:24:47.830 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 4.1821708800000001, 2024-09-13T11:24:47.830 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.108759042, 2024-09-13T11:24:47.830 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:24:47.830 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:24:47.830 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:47.830 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:24:47.830 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:24:47.830 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:47.831 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 126682 2024-09-13T11:24:47.831 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:47.831 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:24:47.831 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:47.831 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:24:47.831 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:43.563277+0000", 2024-09-13T11:24:47.831 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:47.831 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:47.831 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:47.831 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:24:47.831 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:43.563277+0000", 2024-09-13T11:24:47.831 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:47.831 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:47.831 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:47.832 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:24:47.832 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:43.563279+0000", 2024-09-13T11:24:47.832 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.1689999999999999e-06 2024-09-13T11:24:47.832 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:47.832 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:47.832 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:24:47.832 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:43.563323+0000", 2024-09-13T11:24:47.832 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.4011000000000003e-05 2024-09-13T11:24:47.832 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:47.832 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:47.832 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:24:47.832 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:43.563336+0000", 2024-09-13T11:24:47.832 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.2797e-05 2024-09-13T11:24:47.832 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:47.833 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:47.833 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:24:47.833 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:43.563344+0000", 2024-09-13T11:24:47.833 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.9570000000000002e-06 2024-09-13T11:24:47.833 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:47.833 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:47.833 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:24:47.833 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:43.563403+0000", 2024-09-13T11:24:47.833 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.8802999999999999e-05 2024-09-13T11:24:47.833 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:47.833 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:47.833 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:24:47.833 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:43.563537+0000", 2024-09-13T11:24:47.834 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00013432400000000001 2024-09-13T11:24:47.834 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:47.834 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:47.834 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 4", 2024-09-13T11:24:47.834 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:43.563677+0000", 2024-09-13T11:24:47.834 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00013981200000000001 2024-09-13T11:24:47.834 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:47.834 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:47.834 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:24:47.834 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:43.565705+0000", 2024-09-13T11:24:47.834 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0020279069999999998 2024-09-13T11:24:47.834 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:47.834 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:47.834 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:47.834 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:43.671954+0000", 2024-09-13T11:24:47.835 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.106249604 2024-09-13T11:24:47.835 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:47.835 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:47.835 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:24:47.835 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:43.671990+0000", 2024-09-13T11:24:47.835 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.5868000000000003e-05 2024-09-13T11:24:47.835 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:47.835 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:47.835 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:24:47.835 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:43.672036+0000", 2024-09-13T11:24:47.835 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.579e-05 2024-09-13T11:24:47.835 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:47.835 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:24:47.835 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:47.836 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:47.836 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:47.836 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop_reply(client.4665.0:126682 3.1e e130/102 ondisk, result = 0)", 2024-09-13T11:24:47.836 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:24:43.566042+0000", 2024-09-13T11:24:47.836 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 4.1794057699999998, 2024-09-13T11:24:47.836 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.10600035100000001, 2024-09-13T11:24:47.836 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:24:47.836 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:24:47.836 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-09-13T11:24:47.836 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:24:47.836 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:47.836 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:24:47.836 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:43.566042+0000", 2024-09-13T11:24:47.836 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:47.837 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:47.837 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:47.837 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:24:47.837 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:43.566042+0000", 2024-09-13T11:24:47.837 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:47.837 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:47.837 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:47.837 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:24:47.837 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:43.566043+0000", 2024-09-13T11:24:47.837 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.9999999999999995e-07 2024-09-13T11:24:47.837 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:47.837 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:47.837 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:24:47.837 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:43.566047+0000", 2024-09-13T11:24:47.838 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.5329999999999999e-06 2024-09-13T11:24:47.838 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:47.838 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:47.838 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:24:47.838 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:43.671901+0000", 2024-09-13T11:24:47.838 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.105854502 2024-09-13T11:24:47.838 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:47.838 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:47.838 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:24:47.838 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:43.671906+0000", 2024-09-13T11:24:47.838 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.6229999999999996e-06 2024-09-13T11:24:47.838 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:47.838 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:47.838 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:24:47.839 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:43.671945+0000", 2024-09-13T11:24:47.839 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.9279999999999997e-05 2024-09-13T11:24:47.839 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:47.839 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:47.839 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:24:47.839 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:43.671951+0000", 2024-09-13T11:24:47.839 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.728e-06 2024-09-13T11:24:47.839 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:47.839 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:47.839 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:24:47.839 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:43.672042+0000", 2024-09-13T11:24:47.839 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.1685000000000001e-05 2024-09-13T11:24:47.839 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:47.839 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:24:47.840 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:47.840 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:47.840 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:47.840 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "MOSDPGPush(3.29 134/120 [PushOp(3:95648f22:::benchmark_data_smithi177_31116_object7446:head, version: 46'480, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:95648f22:::benchmark_data_smithi177_31116_object7446:head@46'480, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:24:47.840 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:24:47.108593+0000", 2024-09-13T11:24:47.840 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 0.63685515100000001, 2024-09-13T11:24:47.840 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.155295241, 2024-09-13T11:24:47.840 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:24:47.840 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:24:47.840 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-09-13T11:24:47.840 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:24:47.840 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:47.840 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:24:47.840 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:47.108593+0000", 2024-09-13T11:24:47.841 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:47.841 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:47.841 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:47.841 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:24:47.841 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:47.108593+0000", 2024-09-13T11:24:47.841 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:47.841 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:47.841 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:47.841 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:24:47.841 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:47.108594+0000", 2024-09-13T11:24:47.841 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.4190000000000001e-06 2024-09-13T11:24:47.841 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:47.841 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:47.841 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:24:47.842 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:47.108656+0000", 2024-09-13T11:24:47.842 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.1730999999999996e-05 2024-09-13T11:24:47.842 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:47.842 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:47.842 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:24:47.842 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:47.262441+0000", 2024-09-13T11:24:47.842 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.15378476499999999 2024-09-13T11:24:47.842 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:47.842 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:47.842 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:24:47.842 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:47.262449+0000", 2024-09-13T11:24:47.842 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.1049999999999995e-06 2024-09-13T11:24:47.842 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:47.842 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:47.843 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:24:47.843 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:47.262557+0000", 2024-09-13T11:24:47.843 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000108076 2024-09-13T11:24:47.843 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:47.843 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:47.843 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:24:47.843 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:47.262574+0000", 2024-09-13T11:24:47.843 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.6878999999999999e-05 2024-09-13T11:24:47.843 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:47.843 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:47.843 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:24:47.843 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:47.263888+0000", 2024-09-13T11:24:47.843 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0013142659999999999 2024-09-13T11:24:47.843 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:47.844 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:24:47.844 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:47.844 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:47.844 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:24:47.844 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-09-13T11:24:47.845 INFO:tasks.ceph.osd.12.smithi001.stderr:2024-09-13T11:24:47.843+0000 7f892c6e6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26314) UID: 0 2024-09-13T11:24:47.846 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-09-13T11:24:47.924 INFO:tasks.ceph.osd.7.smithi177.stderr:2024-09-13T11:24:47.918+0000 7f8971f4e640 -1 osd.7 134 handle_osd_map injecting map crc failure 2024-09-13T11:24:47.924 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-09-13T11:24:47.920+0000 7f97b59fa640 -1 osd.6 134 handle_osd_map injecting map crc failure 2024-09-13T11:24:47.925 INFO:tasks.ceph.osd.4.smithi001.stderr:2024-09-13T11:24:47.923+0000 7ff8d2c41640 -1 osd.4 134 handle_osd_map injecting map crc failure 2024-09-13T11:24:47.934 INFO:tasks.ceph.osd.14.smithi162.stderr:2024-09-13T11:24:47.932+0000 7f3df89477c0 -1 Falling back to public interface 2024-09-13T11:24:47.945 INFO:tasks.ceph.osd.13.smithi092.stderr:2024-09-13T11:24:47.942+0000 7f3e5afea640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26350) UID: 0 2024-09-13T11:24:48.012 INFO:tasks.radosbench.radosbench.0.smithi177.stdout: 141 16 130567 130551 57.8621 51.0625 0.00624056 0.0172771 2024-09-13T11:24:48.013 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-09-13T11:24:48.013 INFO:teuthology.orchestra.run.smithi092.stdout: "ops": [], 2024-09-13T11:24:48.014 INFO:teuthology.orchestra.run.smithi092.stdout: "num_ops": 0 2024-09-13T11:24:48.014 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-09-13T11:24:48.040 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-09-13T11:24:48.047 INFO:tasks.ceph.osd.10.smithi162.stderr:2024-09-13T11:24:48.044+0000 7fb59c9f0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26317) UID: 0 2024-09-13T11:24:48.150 INFO:tasks.ceph.osd.7.smithi177.stderr:2024-09-13T11:24:48.146+0000 7f897b51d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26575) UID: 0 2024-09-13T11:24:48.238 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-09-13T11:24:48.238 INFO:teuthology.orchestra.run.smithi092.stdout: "ops": [], 2024-09-13T11:24:48.239 INFO:teuthology.orchestra.run.smithi092.stdout: "complaint_time": 30, 2024-09-13T11:24:48.239 INFO:teuthology.orchestra.run.smithi092.stdout: "num_blocked_ops": 0 2024-09-13T11:24:48.239 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-09-13T11:24:48.249 INFO:tasks.ceph.osd.2.smithi162.stderr:2024-09-13T11:24:48.244+0000 7fe92b35a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26313) UID: 0 2024-09-13T11:24:48.259 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-09-13T11:24:48.350 INFO:tasks.ceph.osd.13.smithi092.stderr:2024-09-13T11:24:48.346+0000 7f3e5afea640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26350) UID: 0 2024-09-13T11:24:48.402 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-09-13T11:24:48.402 INFO:teuthology.orchestra.run.smithi092.stdout: "size": 20, 2024-09-13T11:24:48.402 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 600, 2024-09-13T11:24:48.403 INFO:teuthology.orchestra.run.smithi092.stdout: "ops": [ 2024-09-13T11:24:48.403 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:48.403 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "MOSDPGPush(3.19 76/70 [PushOp(3:9eacf3d0:::benchmark_data_smithi177_31116_object19747:head, version: 54'1236, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:9eacf3d0:::benchmark_data_smithi177_31116_object19747:head@54'1236, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:24:48.403 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:32.704676+0000", 2024-09-13T11:24:48.403 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 75.694964705000004, 2024-09-13T11:24:48.404 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.23525014899999999, 2024-09-13T11:24:48.404 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:24:48.404 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:24:48.404 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-09-13T11:24:48.404 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:24:48.404 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:48.405 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:24:48.405 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:32.704676+0000", 2024-09-13T11:24:48.405 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:48.405 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:48.405 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:48.406 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:24:48.406 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:32.704676+0000", 2024-09-13T11:24:48.406 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:48.406 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:48.406 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:48.406 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:24:48.407 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:32.704677+0000", 2024-09-13T11:24:48.407 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.531e-06 2024-09-13T11:24:48.407 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:48.407 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:48.407 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:24:48.408 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:32.704749+0000", 2024-09-13T11:24:48.408 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.1451000000000004e-05 2024-09-13T11:24:48.408 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:48.408 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:48.408 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:24:48.409 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:32.938053+0000", 2024-09-13T11:24:48.409 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.233304752 2024-09-13T11:24:48.409 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:48.409 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:48.410 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:24:48.410 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:32.938087+0000", 2024-09-13T11:24:48.410 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.4016000000000001e-05 2024-09-13T11:24:48.410 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:48.410 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:48.410 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:24:48.410 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:32.938351+0000", 2024-09-13T11:24:48.410 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00026397500000000001 2024-09-13T11:24:48.410 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:48.410 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:48.410 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:24:48.410 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:32.938381+0000", 2024-09-13T11:24:48.411 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.0001999999999998e-05 2024-09-13T11:24:48.411 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:48.411 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:48.411 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:24:48.411 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:32.939926+0000", 2024-09-13T11:24:48.411 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0015444219999999999 2024-09-13T11:24:48.411 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:48.411 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:24:48.411 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:48.411 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:48.411 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:48.411 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "MOSDPGPush(3.23 80/78 [PushOp(3:c59ecc6d:::benchmark_data_smithi177_31116_object17005:head, version: 52'1054, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:c59ecc6d:::benchmark_data_smithi177_31116_object17005:head@52'1054, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:24:48.411 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:37.433040+0000", 2024-09-13T11:24:48.411 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 70.966600729000007, 2024-09-13T11:24:48.412 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.26658219100000002, 2024-09-13T11:24:48.412 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:24:48.412 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:24:48.412 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-09-13T11:24:48.412 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:24:48.412 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:48.412 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:24:48.412 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:37.433040+0000", 2024-09-13T11:24:48.412 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:48.412 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:48.412 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:48.412 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:24:48.412 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:37.433040+0000", 2024-09-13T11:24:48.412 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:48.413 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:48.413 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:48.413 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:24:48.413 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:37.433042+0000", 2024-09-13T11:24:48.413 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.3999999999999999e-06 2024-09-13T11:24:48.413 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:48.413 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:48.413 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:24:48.413 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:37.433090+0000", 2024-09-13T11:24:48.413 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.8065999999999997e-05 2024-09-13T11:24:48.413 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:48.413 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:48.413 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:24:48.413 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:37.698797+0000", 2024-09-13T11:24:48.414 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.26570674500000002 2024-09-13T11:24:48.414 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:48.414 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:48.414 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:24:48.415 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:37.698827+0000", 2024-09-13T11:24:48.415 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.0505e-05 2024-09-13T11:24:48.415 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:48.415 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:48.415 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:24:48.415 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:37.698941+0000", 2024-09-13T11:24:48.415 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00011403799999999999 2024-09-13T11:24:48.415 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:48.415 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:48.415 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:24:48.415 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:37.698956+0000", 2024-09-13T11:24:48.415 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.4765e-05 2024-09-13T11:24:48.415 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:48.415 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:48.416 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:24:48.416 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:37.699622+0000", 2024-09-13T11:24:48.416 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00066567199999999997 2024-09-13T11:24:48.416 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:48.416 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:24:48.416 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:48.416 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:48.416 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:48.416 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "MOSDPGPush(3.1a 81/78 [PushOp(3:5a9c05b8:::benchmark_data_smithi177_31116_object4600:head, version: 44'280, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:5a9c05b8:::benchmark_data_smithi177_31116_object4600:head@44'280, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:24:48.416 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:38.261429+0000", 2024-09-13T11:24:48.416 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 70.138211292999998, 2024-09-13T11:24:48.416 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.13814789799999999, 2024-09-13T11:24:48.416 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:24:48.416 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:24:48.417 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-09-13T11:24:48.417 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:24:48.417 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:48.417 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:24:48.417 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:38.261429+0000", 2024-09-13T11:24:48.417 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:48.417 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:48.417 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:48.417 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:24:48.417 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:38.261429+0000", 2024-09-13T11:24:48.417 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:48.417 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:48.417 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:48.417 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:24:48.417 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:38.261430+0000", 2024-09-13T11:24:48.418 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.376e-06 2024-09-13T11:24:48.418 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:48.418 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:48.418 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:24:48.418 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:38.261501+0000", 2024-09-13T11:24:48.418 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.0724999999999998e-05 2024-09-13T11:24:48.418 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:48.418 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:48.418 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:24:48.418 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:38.398898+0000", 2024-09-13T11:24:48.418 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.13739662499999999 2024-09-13T11:24:48.418 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:48.418 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:48.418 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:24:48.419 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:38.398909+0000", 2024-09-13T11:24:48.419 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0926e-05 2024-09-13T11:24:48.419 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:48.419 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:48.419 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:24:48.419 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:38.398974+0000", 2024-09-13T11:24:48.419 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.5467000000000002e-05 2024-09-13T11:24:48.419 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:48.419 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:48.419 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:24:48.419 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:38.398982+0000", 2024-09-13T11:24:48.419 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.841e-06 2024-09-13T11:24:48.419 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:48.419 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:48.420 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:24:48.420 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:38.399577+0000", 2024-09-13T11:24:48.420 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00059493800000000004 2024-09-13T11:24:48.420 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:48.420 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:24:48.420 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:48.420 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:48.420 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:48.420 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "MOSDPGPush(3.1a 85/78 [PushOp(3:5dc85525:::benchmark_data_smithi177_31116_object11755:head, version: 49'723, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:5dc85525:::benchmark_data_smithi177_31116_object11755:head@49'723, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:24:48.420 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:42.422061+0000", 2024-09-13T11:24:48.420 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 65.977579276, 2024-09-13T11:24:48.420 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.52875617500000005, 2024-09-13T11:24:48.420 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:24:48.420 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:24:48.421 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-09-13T11:24:48.421 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:24:48.421 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:48.421 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:24:48.421 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.422061+0000", 2024-09-13T11:24:48.421 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:48.421 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:48.421 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:48.421 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:24:48.421 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.422061+0000", 2024-09-13T11:24:48.421 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:48.421 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:48.421 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:48.421 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:24:48.422 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.422064+0000", 2024-09-13T11:24:48.422 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.5409999999999999e-06 2024-09-13T11:24:48.422 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:48.422 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:48.422 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:24:48.422 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.422139+0000", 2024-09-13T11:24:48.422 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.5488999999999995e-05 2024-09-13T11:24:48.422 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:48.422 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:48.422 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:24:48.422 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.948781+0000", 2024-09-13T11:24:48.422 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.52664230999999995 2024-09-13T11:24:48.422 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:48.422 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:48.423 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:24:48.423 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.948841+0000", 2024-09-13T11:24:48.423 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.9450999999999997e-05 2024-09-13T11:24:48.423 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:48.423 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:48.423 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:24:48.423 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.949035+0000", 2024-09-13T11:24:48.423 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00019380600000000001 2024-09-13T11:24:48.423 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:48.423 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:48.423 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:24:48.423 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.949060+0000", 2024-09-13T11:24:48.423 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.5381999999999999e-05 2024-09-13T11:24:48.423 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:48.423 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:48.424 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:24:48.424 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.950817+0000", 2024-09-13T11:24:48.424 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0017571959999999999 2024-09-13T11:24:48.424 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:48.424 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:24:48.424 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:48.424 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:48.424 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:48.425 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "MOSDPGPush(3.1a 89/78 [PushOp(3:59270734:::benchmark_data_smithi177_31116_object20039:head, version: 54'1256, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:59270734:::benchmark_data_smithi177_31116_object20039:head@54'1256, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:24:48.425 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:46.607562+0000", 2024-09-13T11:24:48.425 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 61.792078498999999, 2024-09-13T11:24:48.425 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.32123945300000001, 2024-09-13T11:24:48.425 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:24:48.426 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:24:48.426 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-09-13T11:24:48.426 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:24:48.426 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:48.426 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:24:48.426 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.607562+0000", 2024-09-13T11:24:48.426 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:48.427 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:48.427 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:48.427 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:24:48.427 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.607562+0000", 2024-09-13T11:24:48.427 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:48.427 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:48.427 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:48.428 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:24:48.428 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.607564+0000", 2024-09-13T11:24:48.428 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.8610000000000001e-06 2024-09-13T11:24:48.428 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:48.428 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:48.428 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:24:48.428 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.607663+0000", 2024-09-13T11:24:48.428 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.9210999999999996e-05 2024-09-13T11:24:48.429 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:48.429 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:48.429 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:24:48.429 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.925193+0000", 2024-09-13T11:24:48.429 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.317530586 2024-09-13T11:24:48.429 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:48.429 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:48.430 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:24:48.430 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.925250+0000", 2024-09-13T11:24:48.430 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.6925000000000001e-05 2024-09-13T11:24:48.430 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:48.430 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:48.430 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:24:48.430 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.927013+0000", 2024-09-13T11:24:48.431 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.001762853 2024-09-13T11:24:48.431 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:48.431 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:48.431 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:24:48.431 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.927067+0000", 2024-09-13T11:24:48.431 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.3601999999999999e-05 2024-09-13T11:24:48.432 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:48.432 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:48.432 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:24:48.432 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.928801+0000", 2024-09-13T11:24:48.432 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.001734415 2024-09-13T11:24:48.432 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:48.432 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:24:48.432 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:48.433 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:48.433 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:48.433 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:64389 3.19 3:9e2bf991:::benchmark_data_smithi177_31116_object64388:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90)", 2024-09-13T11:24:48.433 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:47.195560+0000", 2024-09-13T11:24:48.433 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 61.204080472000001, 2024-09-13T11:24:48.433 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.108379857, 2024-09-13T11:24:48.433 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:24:48.433 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:24:48.433 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:48.433 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:24:48.433 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:24:48.433 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:48.433 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 64389 2024-09-13T11:24:48.434 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:48.434 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:24:48.434 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:48.434 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:24:48.434 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:47.195560+0000", 2024-09-13T11:24:48.434 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:48.434 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:48.434 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:48.434 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:24:48.434 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:47.195560+0000", 2024-09-13T11:24:48.434 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:48.434 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:48.434 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:48.435 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:24:48.435 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:47.195561+0000", 2024-09-13T11:24:48.435 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.4300000000000001e-06 2024-09-13T11:24:48.435 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:48.435 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:48.435 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:24:48.435 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:47.195649+0000", 2024-09-13T11:24:48.435 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.7379000000000005e-05 2024-09-13T11:24:48.435 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:48.435 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:48.435 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:24:48.435 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:47.195663+0000", 2024-09-13T11:24:48.435 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.4012999999999999e-05 2024-09-13T11:24:48.435 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:48.436 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:48.436 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:24:48.436 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:47.195673+0000", 2024-09-13T11:24:48.436 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0045e-05 2024-09-13T11:24:48.436 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:48.436 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:48.436 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:24:48.436 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:47.195739+0000", 2024-09-13T11:24:48.436 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.6651999999999998e-05 2024-09-13T11:24:48.436 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:48.436 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:48.436 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:24:48.436 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:47.195908+0000", 2024-09-13T11:24:48.436 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00016875700000000001 2024-09-13T11:24:48.437 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:48.437 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:48.437 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1", 2024-09-13T11:24:48.437 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:47.196082+0000", 2024-09-13T11:24:48.437 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00017355199999999999 2024-09-13T11:24:48.437 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:48.437 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:48.437 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:24:48.437 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:47.197432+0000", 2024-09-13T11:24:48.437 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0013507700000000001 2024-09-13T11:24:48.438 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:48.438 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:48.438 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:48.438 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:47.303796+0000", 2024-09-13T11:24:48.438 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.106363729 2024-09-13T11:24:48.438 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:48.438 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:48.438 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:24:48.438 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:47.303875+0000", 2024-09-13T11:24:48.438 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.9092999999999996e-05 2024-09-13T11:24:48.438 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:48.438 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:48.438 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:24:48.438 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:47.303940+0000", 2024-09-13T11:24:48.439 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.4437000000000007e-05 2024-09-13T11:24:48.439 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:48.439 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:24:48.439 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:48.439 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:48.439 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:48.439 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "MOSDPGPush(3.1a 91/78 [PushOp(3:5e5010d2:::benchmark_data_smithi177_31116_object25733:head, version: 58'1609, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:5e5010d2:::benchmark_data_smithi177_31116_object25733:head@58'1609, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:24:48.439 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:48.608485+0000", 2024-09-13T11:24:48.439 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 59.791155381000003, 2024-09-13T11:24:48.439 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.35035599699999997, 2024-09-13T11:24:48.439 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:24:48.439 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:24:48.439 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-09-13T11:24:48.439 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:24:48.440 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:48.440 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:24:48.440 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:48.608485+0000", 2024-09-13T11:24:48.440 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:48.440 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:48.440 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:48.440 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:24:48.440 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:48.608485+0000", 2024-09-13T11:24:48.440 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:48.440 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:48.440 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:48.440 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:24:48.440 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:48.608488+0000", 2024-09-13T11:24:48.441 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.9399999999999998e-06 2024-09-13T11:24:48.441 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:48.441 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:48.441 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:24:48.441 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:48.608552+0000", 2024-09-13T11:24:48.441 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.3731000000000004e-05 2024-09-13T11:24:48.442 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:48.442 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:48.442 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:24:48.442 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:48.957330+0000", 2024-09-13T11:24:48.442 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.34877806099999997 2024-09-13T11:24:48.442 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:48.442 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:48.443 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:24:48.443 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:48.957361+0000", 2024-09-13T11:24:48.443 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.0852000000000002e-05 2024-09-13T11:24:48.443 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:48.443 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:48.443 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:24:48.443 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:48.957544+0000", 2024-09-13T11:24:48.444 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000183537 2024-09-13T11:24:48.444 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:48.444 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:48.444 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:24:48.444 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:48.957582+0000", 2024-09-13T11:24:48.444 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.7472999999999998e-05 2024-09-13T11:24:48.444 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:48.445 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:48.445 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:24:48.445 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:48.958841+0000", 2024-09-13T11:24:48.445 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0012594030000000001 2024-09-13T11:24:48.445 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:48.445 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:24:48.445 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:48.445 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:48.446 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:48.446 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop(client.4665.0:72923 3.23 e97/90 3:c6230123:::benchmark_data_smithi177_31116_object72922:head v 97'3418, mlcod=97'3414)", 2024-09-13T11:24:48.446 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:55.047734+0000", 2024-09-13T11:24:48.446 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 53.351905864000003, 2024-09-13T11:24:48.446 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.20075528000000001, 2024-09-13T11:24:48.446 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:24:48.446 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:24:48.447 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:48.447 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:24:48.447 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:48.447 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:24:48.447 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:55.047734+0000", 2024-09-13T11:24:48.447 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:48.448 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:48.448 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:48.448 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:24:48.448 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:55.047734+0000", 2024-09-13T11:24:48.448 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:48.448 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:48.449 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:48.449 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:24:48.449 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:55.047736+0000", 2024-09-13T11:24:48.449 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.466e-06 2024-09-13T11:24:48.449 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:48.449 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:48.449 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:24:48.449 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:55.047772+0000", 2024-09-13T11:24:48.449 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.6576999999999999e-05 2024-09-13T11:24:48.449 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:48.449 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:48.449 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:24:48.450 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:55.246478+0000", 2024-09-13T11:24:48.450 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.19870581300000001 2024-09-13T11:24:48.450 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:48.450 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:48.450 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:24:48.450 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:55.246488+0000", 2024-09-13T11:24:48.450 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.5419999999999994e-06 2024-09-13T11:24:48.450 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:48.450 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:48.450 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:24:48.450 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:55.246631+0000", 2024-09-13T11:24:48.450 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00014287300000000001 2024-09-13T11:24:48.450 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:48.450 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:48.450 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:24:48.451 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:55.246665+0000", 2024-09-13T11:24:48.451 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.4684999999999997e-05 2024-09-13T11:24:48.451 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:48.451 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:48.451 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:24:48.451 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:55.248453+0000", 2024-09-13T11:24:48.451 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0017879459999999999 2024-09-13T11:24:48.451 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:48.451 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:48.451 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:24:48.451 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:55.248490+0000", 2024-09-13T11:24:48.451 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.6378000000000003e-05 2024-09-13T11:24:48.451 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:48.451 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:24:48.452 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:48.452 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:48.452 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:48.452 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:79823 3.19 3:9e038f3c:::benchmark_data_smithi177_31116_object79822:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102)", 2024-09-13T11:24:48.452 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:24:00.563188+0000", 2024-09-13T11:24:48.452 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 47.836452239000003, 2024-09-13T11:24:48.452 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.164156422, 2024-09-13T11:24:48.452 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:24:48.452 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:24:48.452 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:48.452 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:24:48.452 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:24:48.452 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:48.453 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 79823 2024-09-13T11:24:48.453 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:48.453 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:24:48.453 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:48.453 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:24:48.453 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:00.563188+0000", 2024-09-13T11:24:48.453 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:48.453 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:48.453 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:48.453 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:24:48.453 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:00.563188+0000", 2024-09-13T11:24:48.453 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:48.453 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:48.453 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:48.454 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:24:48.454 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:00.563201+0000", 2024-09-13T11:24:48.454 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.3183000000000001e-05 2024-09-13T11:24:48.454 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:48.454 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:48.454 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:24:48.454 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:00.563263+0000", 2024-09-13T11:24:48.454 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.1904999999999998e-05 2024-09-13T11:24:48.454 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:48.454 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:48.454 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:24:48.454 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:00.563297+0000", 2024-09-13T11:24:48.454 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.3979000000000001e-05 2024-09-13T11:24:48.454 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:48.454 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:48.455 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:24:48.455 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:00.563318+0000", 2024-09-13T11:24:48.455 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.0703000000000002e-05 2024-09-13T11:24:48.455 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:48.455 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:48.455 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:24:48.455 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:00.563513+0000", 2024-09-13T11:24:48.456 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00019524200000000001 2024-09-13T11:24:48.456 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:48.456 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:48.456 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:24:48.456 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:00.563758+0000", 2024-09-13T11:24:48.456 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000244711 2024-09-13T11:24:48.456 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:48.456 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:48.456 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1", 2024-09-13T11:24:48.456 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:00.563978+0000", 2024-09-13T11:24:48.456 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00022011700000000001 2024-09-13T11:24:48.456 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:48.457 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:48.457 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:24:48.457 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:00.566893+0000", 2024-09-13T11:24:48.457 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0029147650000000002 2024-09-13T11:24:48.457 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:48.457 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:48.457 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:48.457 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:00.727176+0000", 2024-09-13T11:24:48.457 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.16028374100000001 2024-09-13T11:24:48.457 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:48.457 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:48.458 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:24:48.458 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:00.727271+0000", 2024-09-13T11:24:48.458 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.4481000000000006e-05 2024-09-13T11:24:48.458 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:48.458 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:48.458 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:24:48.458 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:00.727344+0000", 2024-09-13T11:24:48.458 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.3595000000000003e-05 2024-09-13T11:24:48.458 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:48.458 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:24:48.458 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:48.458 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:48.459 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:48.459 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop_reply(client.4665.0:79823 3.19 e102/84 ondisk, result = 0)", 2024-09-13T11:24:48.459 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:24:00.567585+0000", 2024-09-13T11:24:48.459 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 47.832054927999998, 2024-09-13T11:24:48.459 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.15977266400000001, 2024-09-13T11:24:48.459 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:24:48.459 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:24:48.459 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-09-13T11:24:48.459 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:24:48.459 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:48.459 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:24:48.459 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:00.567585+0000", 2024-09-13T11:24:48.459 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:48.459 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:48.460 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:48.460 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:24:48.460 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:00.567585+0000", 2024-09-13T11:24:48.460 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:48.460 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:48.460 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:48.460 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:24:48.460 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:00.567589+0000", 2024-09-13T11:24:48.460 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.2969999999999999e-06 2024-09-13T11:24:48.460 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:48.460 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:48.460 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:24:48.460 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:00.567599+0000", 2024-09-13T11:24:48.460 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0142999999999999e-05 2024-09-13T11:24:48.461 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:48.461 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:48.461 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:24:48.461 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:00.726959+0000", 2024-09-13T11:24:48.461 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.15936018399999999 2024-09-13T11:24:48.461 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:48.461 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:48.461 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:24:48.461 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:00.726972+0000", 2024-09-13T11:24:48.461 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.2622999999999999e-05 2024-09-13T11:24:48.461 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:48.461 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:48.461 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:24:48.462 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:00.727151+0000", 2024-09-13T11:24:48.462 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00017909600000000001 2024-09-13T11:24:48.462 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:48.462 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:48.462 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:24:48.462 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:00.727167+0000", 2024-09-13T11:24:48.462 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.6639000000000001e-05 2024-09-13T11:24:48.462 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:48.462 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:48.462 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:24:48.462 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:00.727358+0000", 2024-09-13T11:24:48.462 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00019068199999999999 2024-09-13T11:24:48.462 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:48.462 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:24:48.462 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:48.463 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:48.463 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:48.463 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:98229 3.19 3:98f5bd4f:::benchmark_data_smithi177_31116_object98228:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113)", 2024-09-13T11:24:48.463 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:24:15.139305+0000", 2024-09-13T11:24:48.463 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 33.260335015999999, 2024-09-13T11:24:48.463 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.116582746, 2024-09-13T11:24:48.463 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:24:48.463 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:24:48.463 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:48.463 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:24:48.463 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:24:48.463 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:48.463 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 98229 2024-09-13T11:24:48.463 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:48.464 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:24:48.464 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:48.464 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:24:48.464 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:15.139305+0000", 2024-09-13T11:24:48.464 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:48.464 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:48.464 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:48.464 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:24:48.464 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:15.139305+0000", 2024-09-13T11:24:48.464 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:48.464 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:48.464 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:48.464 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:24:48.464 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:15.139311+0000", 2024-09-13T11:24:48.465 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.4299999999999997e-06 2024-09-13T11:24:48.465 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:48.465 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:48.465 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:24:48.465 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:15.139427+0000", 2024-09-13T11:24:48.465 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000116258 2024-09-13T11:24:48.465 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:48.465 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:48.465 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:24:48.465 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:15.139458+0000", 2024-09-13T11:24:48.465 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.0892000000000002e-05 2024-09-13T11:24:48.465 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:48.465 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:48.465 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:24:48.466 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:15.139477+0000", 2024-09-13T11:24:48.466 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.9544000000000001e-05 2024-09-13T11:24:48.466 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:48.466 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:48.466 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:24:48.466 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:15.139609+0000", 2024-09-13T11:24:48.466 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00013150799999999999 2024-09-13T11:24:48.466 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:48.466 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:48.466 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:24:48.466 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:15.139873+0000", 2024-09-13T11:24:48.466 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000264288 2024-09-13T11:24:48.466 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:48.466 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:48.467 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1", 2024-09-13T11:24:48.467 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:15.140160+0000", 2024-09-13T11:24:48.467 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00028688999999999999 2024-09-13T11:24:48.467 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:48.467 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:48.467 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:24:48.467 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:15.143356+0000", 2024-09-13T11:24:48.467 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0031956179999999999 2024-09-13T11:24:48.467 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:48.467 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:48.467 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:48.468 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:15.255687+0000", 2024-09-13T11:24:48.468 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.11233159099999999 2024-09-13T11:24:48.468 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:48.468 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:48.468 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:24:48.468 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:15.255799+0000", 2024-09-13T11:24:48.468 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000111492 2024-09-13T11:24:48.468 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:48.468 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:48.468 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:24:48.468 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:15.255888+0000", 2024-09-13T11:24:48.468 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.9234999999999995e-05 2024-09-13T11:24:48.468 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:48.469 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:24:48.469 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:48.469 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:48.469 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:48.469 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop_reply(client.4665.0:98229 3.19 e113/84 ondisk, result = 0)", 2024-09-13T11:24:48.469 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:24:15.144168+0000", 2024-09-13T11:24:48.469 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 33.255472371000003, 2024-09-13T11:24:48.469 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.111733939, 2024-09-13T11:24:48.470 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:24:48.470 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:24:48.470 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-09-13T11:24:48.470 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:24:48.470 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:48.470 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:24:48.470 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:15.144168+0000", 2024-09-13T11:24:48.470 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:48.470 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:48.470 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:48.470 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:24:48.470 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:15.144168+0000", 2024-09-13T11:24:48.470 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:48.471 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:48.471 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:48.471 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:24:48.471 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:15.144170+0000", 2024-09-13T11:24:48.471 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.272e-06 2024-09-13T11:24:48.471 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:48.471 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:48.472 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:24:48.472 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:15.144177+0000", 2024-09-13T11:24:48.472 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.0879999999999997e-06 2024-09-13T11:24:48.472 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:48.472 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:48.472 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:24:48.473 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:15.255542+0000", 2024-09-13T11:24:48.473 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.111364674 2024-09-13T11:24:48.473 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:48.473 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:48.473 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:24:48.473 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:15.255553+0000", 2024-09-13T11:24:48.473 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.1022999999999999e-05 2024-09-13T11:24:48.473 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:48.473 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:48.473 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:24:48.473 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:15.255656+0000", 2024-09-13T11:24:48.473 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000102503 2024-09-13T11:24:48.473 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:48.474 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:48.474 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:24:48.474 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:15.255674+0000", 2024-09-13T11:24:48.474 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.8431e-05 2024-09-13T11:24:48.474 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:48.474 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:48.475 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:24:48.475 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:15.255902+0000", 2024-09-13T11:24:48.475 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00022794800000000001 2024-09-13T11:24:48.475 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:48.475 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:24:48.475 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:48.476 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:48.476 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:48.476 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop(client.4665.0:98582 3.1a e113/105 3:59f9cd75:::benchmark_data_smithi177_31116_object98581:head v 113'4743, mlcod=113'4741)", 2024-09-13T11:24:48.476 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:24:15.432027+0000", 2024-09-13T11:24:48.476 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 32.967613249000003, 2024-09-13T11:24:48.476 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.196879784, 2024-09-13T11:24:48.477 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:24:48.477 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:24:48.477 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:48.477 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:24:48.477 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:48.477 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:24:48.478 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:15.432027+0000", 2024-09-13T11:24:48.478 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:48.478 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:48.478 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:48.478 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:24:48.478 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:15.432027+0000", 2024-09-13T11:24:48.479 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:48.479 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:48.479 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:48.479 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:24:48.479 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:15.432029+0000", 2024-09-13T11:24:48.479 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.838e-06 2024-09-13T11:24:48.479 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:48.480 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:48.480 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:24:48.480 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:15.432088+0000", 2024-09-13T11:24:48.480 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.8767999999999997e-05 2024-09-13T11:24:48.480 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:48.480 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:48.480 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:24:48.480 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:15.624731+0000", 2024-09-13T11:24:48.481 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.192643221 2024-09-13T11:24:48.481 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:48.481 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:48.481 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:24:48.481 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:15.624746+0000", 2024-09-13T11:24:48.481 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.4705000000000001e-05 2024-09-13T11:24:48.481 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:48.481 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:48.481 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:24:48.481 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:15.624882+0000", 2024-09-13T11:24:48.481 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00013602300000000001 2024-09-13T11:24:48.481 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:48.481 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:48.481 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:24:48.482 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:15.624941+0000", 2024-09-13T11:24:48.482 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.9351000000000001e-05 2024-09-13T11:24:48.482 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:48.482 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:48.482 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:24:48.482 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:15.628836+0000", 2024-09-13T11:24:48.482 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0038955109999999999 2024-09-13T11:24:48.482 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:48.482 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:48.482 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:24:48.482 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:15.628907+0000", 2024-09-13T11:24:48.482 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.0366999999999999e-05 2024-09-13T11:24:48.482 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:48.482 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:24:48.483 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:48.483 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:48.483 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:48.483 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop(client.4665.0:98872 3.1a e113/105 3:5b894ab8:::benchmark_data_smithi177_31116_object98871:head v 113'4746, mlcod=113'4743)", 2024-09-13T11:24:48.483 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:24:15.736076+0000", 2024-09-13T11:24:48.483 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 32.663564612000002, 2024-09-13T11:24:48.483 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.111852691, 2024-09-13T11:24:48.483 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:24:48.483 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:24:48.483 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:48.483 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:24:48.483 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:48.483 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:24:48.483 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:15.736076+0000", 2024-09-13T11:24:48.483 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:48.484 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:48.484 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:48.484 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:24:48.484 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:15.736076+0000", 2024-09-13T11:24:48.484 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:48.484 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:48.484 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:48.484 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:24:48.485 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:15.736079+0000", 2024-09-13T11:24:48.485 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.2260000000000001e-06 2024-09-13T11:24:48.485 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:48.485 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:48.485 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:24:48.485 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:15.736171+0000", 2024-09-13T11:24:48.486 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.2260000000000001e-05 2024-09-13T11:24:48.486 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:48.486 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:48.486 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:24:48.486 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:15.843102+0000", 2024-09-13T11:24:48.486 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.10693101100000001 2024-09-13T11:24:48.486 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:48.487 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:48.487 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:24:48.487 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:15.843120+0000", 2024-09-13T11:24:48.487 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.7771999999999999e-05 2024-09-13T11:24:48.487 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:48.487 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:48.487 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:24:48.487 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:15.843247+0000", 2024-09-13T11:24:48.487 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00012656000000000001 2024-09-13T11:24:48.487 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:48.487 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:48.487 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:24:48.487 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:15.843295+0000", 2024-09-13T11:24:48.488 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.8516000000000001e-05 2024-09-13T11:24:48.488 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:48.488 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:48.488 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:24:48.488 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:15.847762+0000", 2024-09-13T11:24:48.488 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0044674149999999998 2024-09-13T11:24:48.488 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:48.488 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:48.488 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:24:48.488 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:15.847928+0000", 2024-09-13T11:24:48.488 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00016593099999999999 2024-09-13T11:24:48.489 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:48.489 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:24:48.489 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:48.489 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:48.489 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:48.489 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop(client.4665.0:99105 3.1a e113/105 3:5ac5090c:::benchmark_data_smithi177_31116_object99104:head v 113'4749, mlcod=113'4746)", 2024-09-13T11:24:48.489 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:24:15.963602+0000", 2024-09-13T11:24:48.489 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 32.436038066999998, 2024-09-13T11:24:48.489 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.23699906300000001, 2024-09-13T11:24:48.489 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:24:48.489 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:24:48.489 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:48.489 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:24:48.489 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:48.490 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:24:48.490 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:15.963602+0000", 2024-09-13T11:24:48.490 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:48.490 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:48.490 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:48.490 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:24:48.490 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:15.963602+0000", 2024-09-13T11:24:48.490 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:48.491 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:48.491 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:48.491 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:24:48.491 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:15.963605+0000", 2024-09-13T11:24:48.491 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.0400000000000001e-06 2024-09-13T11:24:48.491 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:48.492 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:48.492 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:24:48.492 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:15.963718+0000", 2024-09-13T11:24:48.492 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000112796 2024-09-13T11:24:48.492 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:48.492 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:48.493 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:24:48.493 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:16.197994+0000", 2024-09-13T11:24:48.493 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.234276285 2024-09-13T11:24:48.493 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:48.493 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:48.493 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:24:48.493 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:16.198004+0000", 2024-09-13T11:24:48.493 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.5559999999999996e-06 2024-09-13T11:24:48.493 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:48.493 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:48.493 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:24:48.493 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:16.198110+0000", 2024-09-13T11:24:48.493 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00010614 2024-09-13T11:24:48.493 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:48.494 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:48.494 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:24:48.494 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:16.198152+0000", 2024-09-13T11:24:48.494 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.1899e-05 2024-09-13T11:24:48.494 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:48.494 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:48.494 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:24:48.494 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:16.200528+0000", 2024-09-13T11:24:48.494 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00237587 2024-09-13T11:24:48.494 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:48.494 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:48.494 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:24:48.494 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:16.200601+0000", 2024-09-13T11:24:48.494 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.3477000000000001e-05 2024-09-13T11:24:48.495 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:48.495 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:24:48.495 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:48.495 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:48.495 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:48.495 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop(client.4665.0:119446 3.1a e128/105 3:5b664280:::benchmark_data_smithi177_31116_object119445:head v 128'5409, mlcod=128'5405)", 2024-09-13T11:24:48.495 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:24:37.343796+0000", 2024-09-13T11:24:48.495 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 11.055844571, 2024-09-13T11:24:48.495 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.118847789, 2024-09-13T11:24:48.495 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:24:48.495 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:24:48.495 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:48.496 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:24:48.496 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:48.496 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:24:48.496 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:37.343796+0000", 2024-09-13T11:24:48.496 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:48.496 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:48.496 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:48.496 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:24:48.496 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:37.343796+0000", 2024-09-13T11:24:48.496 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:48.496 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:48.496 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:48.496 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:24:48.497 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:37.343799+0000", 2024-09-13T11:24:48.497 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.8590000000000001e-06 2024-09-13T11:24:48.497 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:48.497 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:48.497 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:24:48.497 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:37.343848+0000", 2024-09-13T11:24:48.497 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.8930000000000001e-05 2024-09-13T11:24:48.497 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:48.497 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:48.497 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:24:48.498 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:37.460590+0000", 2024-09-13T11:24:48.498 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.116742045 2024-09-13T11:24:48.498 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:48.498 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:48.498 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:24:48.498 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:37.460596+0000", 2024-09-13T11:24:48.498 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.4559999999999997e-06 2024-09-13T11:24:48.498 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:48.498 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:48.499 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:24:48.499 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:37.460645+0000", 2024-09-13T11:24:48.499 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.8550000000000001e-05 2024-09-13T11:24:48.499 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:48.499 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:48.499 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:24:48.499 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:37.460660+0000", 2024-09-13T11:24:48.499 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.5662e-05 2024-09-13T11:24:48.499 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:48.499 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:48.499 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:24:48.500 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:37.462612+0000", 2024-09-13T11:24:48.500 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0019521499999999999 2024-09-13T11:24:48.500 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:48.500 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:48.500 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:24:48.500 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:37.462644+0000", 2024-09-13T11:24:48.500 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.1136999999999997e-05 2024-09-13T11:24:48.500 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:48.500 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:24:48.500 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:48.500 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:48.500 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:48.501 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop(client.4665.0:119844 3.1a e128/105 3:5858dd7d:::benchmark_data_smithi177_31116_object119843:head v 128'5419, mlcod=128'5416)", 2024-09-13T11:24:48.501 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:24:37.692299+0000", 2024-09-13T11:24:48.501 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 10.707341142000001, 2024-09-13T11:24:48.501 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.12845056299999999, 2024-09-13T11:24:48.501 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:24:48.501 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:24:48.501 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:48.501 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:24:48.501 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:48.501 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:24:48.501 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:37.692299+0000", 2024-09-13T11:24:48.501 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:48.501 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:48.501 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:48.501 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:24:48.502 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:37.692299+0000", 2024-09-13T11:24:48.502 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:48.502 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:48.502 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:48.502 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:24:48.502 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:37.692301+0000", 2024-09-13T11:24:48.502 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.6300000000000001e-06 2024-09-13T11:24:48.502 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:48.502 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:48.502 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:24:48.502 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:37.692352+0000", 2024-09-13T11:24:48.502 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.1388000000000003e-05 2024-09-13T11:24:48.502 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:48.502 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:48.503 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:24:48.503 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:37.817572+0000", 2024-09-13T11:24:48.503 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.12522014300000001 2024-09-13T11:24:48.503 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:48.503 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:48.503 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:24:48.503 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:37.817579+0000", 2024-09-13T11:24:48.503 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.8549999999999996e-06 2024-09-13T11:24:48.503 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:48.503 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:48.503 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:24:48.503 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:37.817641+0000", 2024-09-13T11:24:48.503 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.2092000000000001e-05 2024-09-13T11:24:48.503 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:48.504 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:48.504 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:24:48.504 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:37.817659+0000", 2024-09-13T11:24:48.504 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.7849000000000001e-05 2024-09-13T11:24:48.504 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:48.504 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:48.504 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:24:48.504 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:37.820701+0000", 2024-09-13T11:24:48.504 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0030416459999999998 2024-09-13T11:24:48.504 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:48.504 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:48.504 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:24:48.504 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:37.820750+0000", 2024-09-13T11:24:48.504 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.8959999999999999e-05 2024-09-13T11:24:48.505 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:48.505 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:24:48.505 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:48.505 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:48.505 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:48.505 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:127499 3.19 3:9b8540f0:::benchmark_data_smithi177_31116_object127498:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e131)", 2024-09-13T11:24:48.505 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:24:44.319661+0000", 2024-09-13T11:24:48.505 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 4.0799788550000002, 2024-09-13T11:24:48.505 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.116384459, 2024-09-13T11:24:48.505 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:24:48.505 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:24:48.505 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:48.505 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:24:48.506 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:24:48.506 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:48.506 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 127499 2024-09-13T11:24:48.506 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:48.506 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:24:48.506 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:48.506 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:24:48.506 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:44.319661+0000", 2024-09-13T11:24:48.506 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:48.506 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:48.506 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:48.506 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:24:48.506 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:44.319661+0000", 2024-09-13T11:24:48.506 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:48.507 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:48.507 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:48.507 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:24:48.507 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:44.319664+0000", 2024-09-13T11:24:48.507 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.835e-06 2024-09-13T11:24:48.507 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:48.507 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:48.507 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:24:48.507 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:44.319706+0000", 2024-09-13T11:24:48.507 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.2035e-05 2024-09-13T11:24:48.507 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:48.507 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:48.507 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:24:48.507 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:44.319725+0000", 2024-09-13T11:24:48.507 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.9004999999999998e-05 2024-09-13T11:24:48.508 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:48.508 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:48.508 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:24:48.508 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:44.319735+0000", 2024-09-13T11:24:48.508 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.842e-06 2024-09-13T11:24:48.508 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:48.508 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:48.508 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:24:48.508 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:44.319813+0000", 2024-09-13T11:24:48.508 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.7713000000000006e-05 2024-09-13T11:24:48.508 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:48.508 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:48.508 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:24:48.509 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:44.319967+0000", 2024-09-13T11:24:48.509 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00015394399999999999 2024-09-13T11:24:48.509 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:48.509 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:48.509 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1", 2024-09-13T11:24:48.509 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:44.320103+0000", 2024-09-13T11:24:48.509 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00013657900000000001 2024-09-13T11:24:48.509 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:48.509 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:48.509 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:24:48.509 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:44.324061+0000", 2024-09-13T11:24:48.509 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0039573359999999997 2024-09-13T11:24:48.509 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:48.510 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:48.510 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:48.510 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:44.435934+0000", 2024-09-13T11:24:48.510 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.111873032 2024-09-13T11:24:48.510 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:48.510 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:48.510 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:24:48.510 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:44.435999+0000", 2024-09-13T11:24:48.510 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.4820999999999995e-05 2024-09-13T11:24:48.510 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:48.510 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:48.510 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:24:48.510 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:44.436046+0000", 2024-09-13T11:24:48.510 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.7317000000000001e-05 2024-09-13T11:24:48.511 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:48.511 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:24:48.511 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:48.511 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:48.511 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:48.511 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:127516 3.19 3:99ce8625:::benchmark_data_smithi177_31116_object127515:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e131)", 2024-09-13T11:24:48.511 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:24:44.334707+0000", 2024-09-13T11:24:48.511 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 4.0649328159999998, 2024-09-13T11:24:48.511 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.11079009099999999, 2024-09-13T11:24:48.511 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:24:48.511 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:24:48.511 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:48.511 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:24:48.511 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:24:48.512 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:48.512 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 127516 2024-09-13T11:24:48.512 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:48.512 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:24:48.512 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:48.512 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:24:48.512 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:44.334707+0000", 2024-09-13T11:24:48.512 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:48.512 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:48.512 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:48.512 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:24:48.512 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:44.334707+0000", 2024-09-13T11:24:48.512 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:48.513 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:48.513 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:48.513 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:24:48.513 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:44.334709+0000", 2024-09-13T11:24:48.513 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.672e-06 2024-09-13T11:24:48.513 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:48.513 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:48.513 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:24:48.513 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:44.334742+0000", 2024-09-13T11:24:48.513 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.2370999999999999e-05 2024-09-13T11:24:48.513 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:48.513 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:48.513 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:24:48.513 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:44.334754+0000", 2024-09-13T11:24:48.513 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.2665e-05 2024-09-13T11:24:48.514 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:48.514 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:48.514 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:24:48.516 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:44.334763+0000", 2024-09-13T11:24:48.516 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.4859999999999997e-06 2024-09-13T11:24:48.516 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:48.516 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:48.516 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:24:48.517 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:44.334843+0000", 2024-09-13T11:24:48.517 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.0755999999999997e-05 2024-09-13T11:24:48.517 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:48.517 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:48.517 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:24:48.517 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:44.334947+0000", 2024-09-13T11:24:48.517 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00010346200000000001 2024-09-13T11:24:48.517 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:48.517 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:48.518 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1", 2024-09-13T11:24:48.518 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:44.335039+0000", 2024-09-13T11:24:48.518 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.1805999999999995e-05 2024-09-13T11:24:48.518 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:48.518 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:48.518 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:24:48.518 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:44.337010+0000", 2024-09-13T11:24:48.518 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0019710470000000001 2024-09-13T11:24:48.518 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:48.518 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:48.518 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:48.518 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:44.445385+0000", 2024-09-13T11:24:48.518 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.108374997 2024-09-13T11:24:48.519 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:48.519 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:48.519 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:24:48.519 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:44.445447+0000", 2024-09-13T11:24:48.519 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.2230999999999994e-05 2024-09-13T11:24:48.519 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:48.519 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:48.519 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:24:48.519 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:44.445498+0000", 2024-09-13T11:24:48.519 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.0597999999999999e-05 2024-09-13T11:24:48.519 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:48.519 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:24:48.519 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:48.520 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:48.520 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:48.520 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop(client.4665.0:129921 3.1a e134/105 3:5fb9088b:::benchmark_data_smithi177_31116_object129920:head v 134'5758, mlcod=134'5755)", 2024-09-13T11:24:48.520 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:24:47.178021+0000", 2024-09-13T11:24:48.520 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 1.221619061, 2024-09-13T11:24:48.520 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.33723278600000001, 2024-09-13T11:24:48.520 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:24:48.520 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:24:48.520 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:48.520 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:24:48.520 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:48.520 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:24:48.520 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:47.178021+0000", 2024-09-13T11:24:48.520 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:48.521 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:48.521 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:48.521 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:24:48.521 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:47.178021+0000", 2024-09-13T11:24:48.521 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:48.521 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:48.521 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:48.521 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:24:48.521 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:47.178024+0000", 2024-09-13T11:24:48.521 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.3609999999999999e-06 2024-09-13T11:24:48.521 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:48.521 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:48.521 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:24:48.521 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:47.178102+0000", 2024-09-13T11:24:48.522 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.8325000000000006e-05 2024-09-13T11:24:48.522 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:48.522 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:48.522 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:24:48.522 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:47.512707+0000", 2024-09-13T11:24:48.522 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.33460516000000001 2024-09-13T11:24:48.522 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:48.522 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:48.522 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:24:48.522 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:47.512716+0000", 2024-09-13T11:24:48.522 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.0389999999999993e-06 2024-09-13T11:24:48.522 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:48.522 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:48.522 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:24:48.523 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:47.512788+0000", 2024-09-13T11:24:48.523 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.1840000000000003e-05 2024-09-13T11:24:48.523 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:48.523 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:48.523 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:24:48.523 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:47.512809+0000", 2024-09-13T11:24:48.523 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.1005e-05 2024-09-13T11:24:48.523 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:48.523 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:48.523 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:24:48.523 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:47.515182+0000", 2024-09-13T11:24:48.523 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0023731939999999999 2024-09-13T11:24:48.523 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:48.523 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:48.524 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:24:48.524 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:47.515254+0000", 2024-09-13T11:24:48.524 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.1861999999999997e-05 2024-09-13T11:24:48.524 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:48.524 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:24:48.524 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:48.524 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:48.524 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:24:48.524 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-09-13T11:24:48.525 INFO:tasks.ceph.osd.2.smithi162.stderr:2024-09-13T11:24:48.520+0000 7fe92b35a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26313) UID: 0 2024-09-13T11:24:48.526 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_ops_in_flight 2024-09-13T11:24:48.626 INFO:tasks.ceph.osd.10.smithi162.stderr:2024-09-13T11:24:48.624+0000 7fb59c9f0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26317) UID: 0 2024-09-13T11:24:48.714 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-09-13T11:24:48.714 INFO:teuthology.orchestra.run.smithi092.stdout: "ops": [ 2024-09-13T11:24:48.714 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:48.715 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:131430 3.26 3:66a89725:::benchmark_data_smithi177_31116_object131429:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e135)", 2024-09-13T11:24:48.715 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:24:48.710309+0000", 2024-09-13T11:24:48.715 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 0.003624361, 2024-09-13T11:24:48.715 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0036445449999999999, 2024-09-13T11:24:48.715 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:24:48.715 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:24:48.715 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "waiting for sub ops", 2024-09-13T11:24:48.715 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:24:48.715 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:24:48.715 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:48.715 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 131430 2024-09-13T11:24:48.715 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:48.715 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:24:48.716 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:48.716 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:24:48.716 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:48.710309+0000", 2024-09-13T11:24:48.716 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:48.716 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:48.716 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:48.716 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:24:48.716 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:48.710309+0000", 2024-09-13T11:24:48.716 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:48.716 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:48.716 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:48.716 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:24:48.716 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:48.710311+0000", 2024-09-13T11:24:48.717 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.1229999999999998e-06 2024-09-13T11:24:48.717 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:48.717 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:48.717 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:24:48.717 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:48.710355+0000", 2024-09-13T11:24:48.717 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.4005999999999998e-05 2024-09-13T11:24:48.717 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:48.717 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:48.717 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:24:48.717 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:48.710366+0000", 2024-09-13T11:24:48.717 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.1389999999999999e-05 2024-09-13T11:24:48.717 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:48.717 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:48.717 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:24:48.718 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:48.710373+0000", 2024-09-13T11:24:48.718 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.5209999999999999e-06 2024-09-13T11:24:48.718 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:48.718 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:48.718 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:24:48.718 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:48.710441+0000", 2024-09-13T11:24:48.718 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.8385000000000003e-05 2024-09-13T11:24:48.718 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:48.718 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:48.718 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:24:48.718 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:48.710576+0000", 2024-09-13T11:24:48.718 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00013501299999999999 2024-09-13T11:24:48.720 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:48.720 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:48.720 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 8,12", 2024-09-13T11:24:48.720 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:48.710701+0000", 2024-09-13T11:24:48.720 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00012468599999999999 2024-09-13T11:24:48.720 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:48.720 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:48.720 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:24:48.720 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:48.712409+0000", 2024-09-13T11:24:48.720 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0017078309999999999 2024-09-13T11:24:48.720 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:48.721 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:24:48.721 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:48.721 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:48.721 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:48.721 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "MOSDPGPushReply(3.26 135/121 [PushReplyOp(3:676a2515:::benchmark_data_smithi177_31116_object83462:head)])", 2024-09-13T11:24:48.721 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:24:48.712973+0000", 2024-09-13T11:24:48.721 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 0.00096080299999999998, 2024-09-13T11:24:48.721 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.001022902, 2024-09-13T11:24:48.721 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:24:48.721 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:24:48.721 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "reached pg", 2024-09-13T11:24:48.721 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:24:48.721 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:48.721 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:24:48.722 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:48.712973+0000", 2024-09-13T11:24:48.722 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:48.722 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:48.722 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:48.722 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:24:48.722 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:48.712973+0000", 2024-09-13T11:24:48.722 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:48.722 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:48.722 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:48.722 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:24:48.722 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:48.712974+0000", 2024-09-13T11:24:48.722 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0100000000000001e-06 2024-09-13T11:24:48.722 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:48.722 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:48.723 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:24:48.723 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:48.712981+0000", 2024-09-13T11:24:48.723 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.0010000000000004e-06 2024-09-13T11:24:48.723 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:48.723 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:48.723 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:24:48.723 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:48.713004+0000", 2024-09-13T11:24:48.723 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.3839e-05 2024-09-13T11:24:48.723 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:48.723 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:48.723 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:24:48.723 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:48.713012+0000", 2024-09-13T11:24:48.723 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.6009999999999997e-06 2024-09-13T11:24:48.723 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:48.724 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:48.724 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:24:48.724 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:48.713327+0000", 2024-09-13T11:24:48.724 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00031460199999999998 2024-09-13T11:24:48.724 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:48.724 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:24:48.724 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:48.724 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:48.724 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:48.724 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop_reply(client.4665.0:131430 3.26 e135/121)", 2024-09-13T11:24:48.724 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:24:48.713204+0000", 2024-09-13T11:24:48.724 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 0.00072894300000000002, 2024-09-13T11:24:48.724 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00081754400000000002, 2024-09-13T11:24:48.724 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:24:48.725 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:24:48.725 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "queued for pg", 2024-09-13T11:24:48.725 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:24:48.725 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:48.725 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:24:48.725 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:48.713204+0000", 2024-09-13T11:24:48.725 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:48.725 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:48.725 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:48.725 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:24:48.725 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:48.713204+0000", 2024-09-13T11:24:48.725 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:48.725 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:48.725 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:48.726 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:24:48.726 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:48.713206+0000", 2024-09-13T11:24:48.726 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.5570000000000001e-06 2024-09-13T11:24:48.726 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:48.726 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:48.726 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:24:48.726 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:48.713211+0000", 2024-09-13T11:24:48.726 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.2610000000000001e-06 2024-09-13T11:24:48.726 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:48.726 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:48.726 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:24:48.726 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:48.713251+0000", 2024-09-13T11:24:48.726 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.0039000000000001e-05 2024-09-13T11:24:48.726 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:48.727 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:48.727 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:24:48.727 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:48.713259+0000", 2024-09-13T11:24:48.727 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.6930000000000007e-06 2024-09-13T11:24:48.727 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:48.727 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:24:48.727 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:48.727 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:48.727 INFO:teuthology.orchestra.run.smithi092.stdout: ], 2024-09-13T11:24:48.727 INFO:teuthology.orchestra.run.smithi092.stdout: "num_ops": 3 2024-09-13T11:24:48.727 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-09-13T11:24:48.728 INFO:tasks.ceph.osd.3.smithi177.stderr:2024-09-13T11:24:48.726+0000 7f10adba4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26574) UID: 0 2024-09-13T11:24:48.734 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_blocked_ops 2024-09-13T11:24:48.831 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-13T11:24:48.827+0000 7f6a68775640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26307) UID: 0 2024-09-13T11:24:48.869 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-09-13T11:24:48.869 INFO:teuthology.orchestra.run.smithi092.stdout: "ops": [], 2024-09-13T11:24:48.869 INFO:teuthology.orchestra.run.smithi092.stdout: "complaint_time": 30, 2024-09-13T11:24:48.869 INFO:teuthology.orchestra.run.smithi092.stdout: "num_blocked_ops": 0 2024-09-13T11:24:48.869 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-09-13T11:24:48.885 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_historic_ops 2024-09-13T11:24:48.931 INFO:tasks.ceph.osd.2.smithi162.stderr:2024-09-13T11:24:48.928+0000 7fe92b35a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26313) UID: 0 2024-09-13T11:24:48.938 INFO:tasks.ceph.osd.13.smithi092.stderr:2024-09-13T11:24:48.934+0000 7f3e4da13640 -1 osd.13 135 handle_osd_map injecting map crc failure 2024-09-13T11:24:48.963 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-13T11:24:49.013 INFO:tasks.radosbench.radosbench.0.smithi177.stdout: 142 16 131756 131740 57.9779 74.3125 0.00674985 0.0172405 2024-09-13T11:24:49.026 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-09-13T11:24:49.026 INFO:teuthology.orchestra.run.smithi092.stdout: "size": 20, 2024-09-13T11:24:49.026 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 600, 2024-09-13T11:24:49.026 INFO:teuthology.orchestra.run.smithi092.stdout: "ops": [ 2024-09-13T11:24:49.026 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:49.027 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:79 3.1 3:841df5b5:::benchmark_data_smithi177_31116_object78:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-09-13T11:24:49.027 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:27.056083+0000", 2024-09-13T11:24:49.027 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 141.96710655300001, 2024-09-13T11:24:49.027 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.89469730199999997, 2024-09-13T11:24:49.027 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:24:49.028 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:24:49.028 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:49.028 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:24:49.028 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:24:49.028 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:49.029 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 79 2024-09-13T11:24:49.029 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:49.029 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:24:49.029 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:49.029 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:24:49.030 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.056083+0000", 2024-09-13T11:24:49.030 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:49.030 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:49.030 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:49.030 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:24:49.030 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.056083+0000", 2024-09-13T11:24:49.031 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:49.031 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:49.031 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:49.031 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:24:49.031 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.056087+0000", 2024-09-13T11:24:49.032 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.0659999999999997e-06 2024-09-13T11:24:49.032 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:49.032 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:49.032 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:24:49.032 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.056148+0000", 2024-09-13T11:24:49.032 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.1376000000000003e-05 2024-09-13T11:24:49.033 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:49.033 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:49.033 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:24:49.033 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.056180+0000", 2024-09-13T11:24:49.033 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.2370999999999999e-05 2024-09-13T11:24:49.034 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:49.034 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:49.034 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:24:49.034 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.056195+0000", 2024-09-13T11:24:49.034 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.5119e-05 2024-09-13T11:24:49.034 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:49.035 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:49.035 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:24:49.035 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.056304+0000", 2024-09-13T11:24:49.035 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000108365 2024-09-13T11:24:49.035 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:49.036 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:49.036 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:24:49.036 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.056478+0000", 2024-09-13T11:24:49.036 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000174542 2024-09-13T11:24:49.036 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:49.037 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:49.037 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 11", 2024-09-13T11:24:49.037 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.056690+0000", 2024-09-13T11:24:49.037 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00021167500000000001 2024-09-13T11:24:49.037 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:49.037 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:49.038 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:24:49.038 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.058063+0000", 2024-09-13T11:24:49.038 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0013731819999999999 2024-09-13T11:24:49.038 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:49.038 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:49.039 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:49.039 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.950477+0000", 2024-09-13T11:24:49.039 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.89241369400000004 2024-09-13T11:24:49.039 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:49.039 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:49.039 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:24:49.040 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.950652+0000", 2024-09-13T11:24:49.040 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00017526400000000001 2024-09-13T11:24:49.040 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:49.040 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:49.040 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:24:49.041 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.950780+0000", 2024-09-13T11:24:49.041 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00012764800000000001 2024-09-13T11:24:49.041 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:49.041 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:24:49.041 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:49.041 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:49.042 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:49.042 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:7 3.b 3:de78393b:::benchmark_data_smithi177_31116_object6:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T11:24:49.042 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:48.313952+0000", 2024-09-13T11:24:49.042 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 120.709236804, 2024-09-13T11:24:49.042 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.86421288600000001, 2024-09-13T11:24:49.043 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:24:49.043 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:24:49.043 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:49.043 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:24:49.043 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:24:49.043 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:49.044 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 7 2024-09-13T11:24:49.044 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:49.044 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:24:49.044 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:49.044 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:24:49.044 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.313952+0000", 2024-09-13T11:24:49.045 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:49.045 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:49.045 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:49.045 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:24:49.045 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.313952+0000", 2024-09-13T11:24:49.046 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:49.046 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:49.046 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:49.046 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:24:49.046 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.313957+0000", 2024-09-13T11:24:49.046 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.3599999999999998e-06 2024-09-13T11:24:49.047 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:49.047 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:49.047 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:24:49.047 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314043+0000", 2024-09-13T11:24:49.047 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.6766000000000002e-05 2024-09-13T11:24:49.048 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:49.048 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:49.048 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:24:49.048 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314065+0000", 2024-09-13T11:24:49.048 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.1416e-05 2024-09-13T11:24:49.048 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:49.049 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:49.049 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:24:49.049 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314080+0000", 2024-09-13T11:24:49.049 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.5519000000000002e-05 2024-09-13T11:24:49.049 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:49.050 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:49.050 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:24:49.050 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315478+0000", 2024-09-13T11:24:49.050 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.001397765 2024-09-13T11:24:49.050 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:49.051 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:49.051 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-09-13T11:24:49.051 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315507+0000", 2024-09-13T11:24:49.051 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.9128999999999998e-05 2024-09-13T11:24:49.051 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:49.051 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:49.052 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:24:49.052 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.167978+0000", 2024-09-13T11:24:49.052 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.85247068400000003 2024-09-13T11:24:49.052 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:49.052 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:49.052 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:24:49.052 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.168307+0000", 2024-09-13T11:24:49.052 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00032886699999999999 2024-09-13T11:24:49.052 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:49.053 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:49.053 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1", 2024-09-13T11:24:49.053 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.168647+0000", 2024-09-13T11:24:49.053 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00033998399999999999 2024-09-13T11:24:49.053 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:49.053 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:49.053 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:49.053 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.178011+0000", 2024-09-13T11:24:49.053 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0093639169999999994 2024-09-13T11:24:49.053 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:49.053 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:49.053 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:24:49.053 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.178026+0000", 2024-09-13T11:24:49.053 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.5274999999999999e-05 2024-09-13T11:24:49.054 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:49.054 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:49.054 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:24:49.054 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.178080+0000", 2024-09-13T11:24:49.054 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.3601999999999999e-05 2024-09-13T11:24:49.054 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:49.054 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:49.054 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:24:49.054 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.178165+0000", 2024-09-13T11:24:49.054 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.5601999999999998e-05 2024-09-13T11:24:49.054 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:49.054 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:24:49.054 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:49.054 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:49.055 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:49.055 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:9 3.b 3:dafadff6:::benchmark_data_smithi177_31116_object8:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T11:24:49.055 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:48.314114+0000", 2024-09-13T11:24:49.055 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 120.70907499, 2024-09-13T11:24:49.055 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.86613615899999996, 2024-09-13T11:24:49.055 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:24:49.055 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:24:49.055 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:49.055 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:24:49.055 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:24:49.055 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:49.055 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 9 2024-09-13T11:24:49.055 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:49.056 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:24:49.056 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:49.056 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:24:49.056 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314114+0000", 2024-09-13T11:24:49.056 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:49.056 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:49.056 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:49.056 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:24:49.056 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314114+0000", 2024-09-13T11:24:49.056 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:49.056 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:49.056 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:49.056 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:24:49.056 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314117+0000", 2024-09-13T11:24:49.057 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.3860000000000001e-06 2024-09-13T11:24:49.057 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:49.057 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:49.057 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:24:49.057 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314215+0000", 2024-09-13T11:24:49.057 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.8900000000000005e-05 2024-09-13T11:24:49.057 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:49.057 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:49.057 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:24:49.057 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314229+0000", 2024-09-13T11:24:49.057 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.3444999999999999e-05 2024-09-13T11:24:49.057 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:49.057 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:49.057 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:24:49.058 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314239+0000", 2024-09-13T11:24:49.058 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.8470000000000008e-06 2024-09-13T11:24:49.058 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:49.058 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:49.058 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:24:49.058 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315648+0000", 2024-09-13T11:24:49.058 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.001409291 2024-09-13T11:24:49.058 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:49.058 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:49.058 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-09-13T11:24:49.058 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315671+0000", 2024-09-13T11:24:49.058 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.3169000000000001e-05 2024-09-13T11:24:49.058 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:49.058 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:49.059 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:24:49.059 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.176036+0000", 2024-09-13T11:24:49.059 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.86036497499999998 2024-09-13T11:24:49.059 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:49.059 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:49.059 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:24:49.059 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.176189+0000", 2024-09-13T11:24:49.059 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00015271100000000001 2024-09-13T11:24:49.059 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:49.059 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:49.059 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1", 2024-09-13T11:24:49.059 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.176339+0000", 2024-09-13T11:24:49.059 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00015057499999999999 2024-09-13T11:24:49.059 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:49.060 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:49.060 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:49.060 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.180148+0000", 2024-09-13T11:24:49.060 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.003808891 2024-09-13T11:24:49.060 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:49.060 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:49.060 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:24:49.060 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.180168+0000", 2024-09-13T11:24:49.060 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.9507000000000001e-05 2024-09-13T11:24:49.060 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:49.060 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:49.060 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:24:49.060 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.180213+0000", 2024-09-13T11:24:49.060 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.5504999999999999e-05 2024-09-13T11:24:49.061 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:49.061 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:49.061 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:24:49.061 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.180250+0000", 2024-09-13T11:24:49.061 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.6956999999999999e-05 2024-09-13T11:24:49.061 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:49.061 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:24:49.061 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:49.061 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:49.061 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:49.061 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:10 3.b 3:dbe028a8:::benchmark_data_smithi177_31116_object9:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T11:24:49.061 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:48.314259+0000", 2024-09-13T11:24:49.061 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 120.70892986, 2024-09-13T11:24:49.062 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.86637313599999999, 2024-09-13T11:24:49.062 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:24:49.062 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:24:49.062 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:49.062 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:24:49.062 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:24:49.062 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:49.062 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 10 2024-09-13T11:24:49.062 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:49.062 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:24:49.062 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:49.062 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:24:49.062 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314259+0000", 2024-09-13T11:24:49.062 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:49.062 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:49.063 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:49.063 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:24:49.063 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314259+0000", 2024-09-13T11:24:49.063 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:49.063 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:49.063 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:49.063 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:24:49.063 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314261+0000", 2024-09-13T11:24:49.063 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.1500000000000002e-06 2024-09-13T11:24:49.063 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:49.063 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:49.063 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:24:49.063 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314349+0000", 2024-09-13T11:24:49.063 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.8070000000000002e-05 2024-09-13T11:24:49.064 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:49.064 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:49.064 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:24:49.064 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314365+0000", 2024-09-13T11:24:49.064 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.5835999999999999e-05 2024-09-13T11:24:49.064 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:49.064 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:49.064 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:24:49.064 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314377+0000", 2024-09-13T11:24:49.064 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.1589e-05 2024-09-13T11:24:49.064 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:49.064 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:49.064 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:24:49.064 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315788+0000", 2024-09-13T11:24:49.065 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0014109579999999999 2024-09-13T11:24:49.065 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:49.065 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:49.065 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-09-13T11:24:49.065 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315809+0000", 2024-09-13T11:24:49.065 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.0985999999999999e-05 2024-09-13T11:24:49.065 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:49.065 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:49.065 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:24:49.065 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.176822+0000", 2024-09-13T11:24:49.065 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.86101310799999997 2024-09-13T11:24:49.065 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:49.065 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:49.065 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:24:49.066 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.177021+0000", 2024-09-13T11:24:49.066 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00019911199999999999 2024-09-13T11:24:49.066 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:49.066 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:49.066 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1", 2024-09-13T11:24:49.066 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.177226+0000", 2024-09-13T11:24:49.066 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00020516600000000001 2024-09-13T11:24:49.066 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:49.066 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:49.066 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:24:49.066 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.180264+0000", 2024-09-13T11:24:49.066 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0030381739999999998 2024-09-13T11:24:49.066 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:49.066 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:49.067 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:49.067 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.180387+0000", 2024-09-13T11:24:49.067 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00012257699999999999 2024-09-13T11:24:49.067 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:49.067 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:49.067 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:24:49.067 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.180518+0000", 2024-09-13T11:24:49.067 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00013135800000000001 2024-09-13T11:24:49.067 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:49.067 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:49.067 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:24:49.067 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.180632+0000", 2024-09-13T11:24:49.067 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000114052 2024-09-13T11:24:49.067 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:49.067 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:24:49.068 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:49.068 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:49.068 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:49.068 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:39 3.b 3:d91854b3:::benchmark_data_smithi177_31116_object38:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T11:24:49.068 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:48.314393+0000", 2024-09-13T11:24:49.068 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 120.708796318, 2024-09-13T11:24:49.068 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.86850121700000005, 2024-09-13T11:24:49.068 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:24:49.068 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:24:49.068 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:49.068 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:24:49.068 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:24:49.068 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:49.068 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 39 2024-09-13T11:24:49.069 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:49.069 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:24:49.069 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:49.069 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:24:49.069 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314393+0000", 2024-09-13T11:24:49.069 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:49.069 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:49.069 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:49.069 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:24:49.069 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314393+0000", 2024-09-13T11:24:49.069 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:49.069 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:49.069 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:49.070 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:24:49.070 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314394+0000", 2024-09-13T11:24:49.070 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.513e-06 2024-09-13T11:24:49.070 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:49.070 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:49.070 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:24:49.070 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314463+0000", 2024-09-13T11:24:49.070 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.8261000000000003e-05 2024-09-13T11:24:49.070 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:49.070 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:49.070 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:24:49.070 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314477+0000", 2024-09-13T11:24:49.070 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.4619e-05 2024-09-13T11:24:49.070 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:49.070 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:49.071 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:24:49.071 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314485+0000", 2024-09-13T11:24:49.071 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.1699999999999997e-06 2024-09-13T11:24:49.071 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:49.071 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:49.071 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:24:49.071 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315917+0000", 2024-09-13T11:24:49.071 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0014316089999999999 2024-09-13T11:24:49.071 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:49.071 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:49.071 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-09-13T11:24:49.071 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315933+0000", 2024-09-13T11:24:49.071 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.5999999999999999e-05 2024-09-13T11:24:49.071 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:49.072 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:49.072 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:24:49.072 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.178205+0000", 2024-09-13T11:24:49.072 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.86227247100000004 2024-09-13T11:24:49.072 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:49.072 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:49.072 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:24:49.072 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.178316+0000", 2024-09-13T11:24:49.072 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000110788 2024-09-13T11:24:49.072 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:49.072 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:49.072 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1", 2024-09-13T11:24:49.072 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.178448+0000", 2024-09-13T11:24:49.072 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000132056 2024-09-13T11:24:49.073 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:49.073 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:49.073 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:24:49.073 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.182721+0000", 2024-09-13T11:24:49.073 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0042729780000000002 2024-09-13T11:24:49.073 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:49.073 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:49.073 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:49.073 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.182801+0000", 2024-09-13T11:24:49.073 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.0209000000000004e-05 2024-09-13T11:24:49.073 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:49.073 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:49.073 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:24:49.073 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.182855+0000", 2024-09-13T11:24:49.074 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.3749000000000003e-05 2024-09-13T11:24:49.074 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:49.074 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:49.074 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:24:49.074 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.182894+0000", 2024-09-13T11:24:49.074 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.8794000000000002e-05 2024-09-13T11:24:49.074 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:49.074 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:24:49.074 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:49.074 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:49.074 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:49.074 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:50 3.b 3:d1c144b8:::benchmark_data_smithi177_31116_object49:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T11:24:49.074 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:48.314498+0000", 2024-09-13T11:24:49.074 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 120.708690903, 2024-09-13T11:24:49.075 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.86878673299999998, 2024-09-13T11:24:49.075 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:24:49.075 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:24:49.075 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:49.075 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:24:49.075 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:24:49.075 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:49.075 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 50 2024-09-13T11:24:49.075 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:49.075 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:24:49.075 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:49.075 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:24:49.075 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314498+0000", 2024-09-13T11:24:49.075 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:49.076 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:49.076 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:49.076 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:24:49.076 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314498+0000", 2024-09-13T11:24:49.076 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:49.076 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:49.076 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:49.076 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:24:49.076 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314499+0000", 2024-09-13T11:24:49.076 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.1969999999999999e-06 2024-09-13T11:24:49.076 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:49.076 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:49.076 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:24:49.077 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314565+0000", 2024-09-13T11:24:49.077 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.5628000000000003e-05 2024-09-13T11:24:49.077 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:49.077 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:49.077 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:24:49.077 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314579+0000", 2024-09-13T11:24:49.077 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.4222e-05 2024-09-13T11:24:49.077 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:49.077 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:49.077 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:24:49.077 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314590+0000", 2024-09-13T11:24:49.077 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.1240000000000001e-05 2024-09-13T11:24:49.077 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:49.077 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:49.077 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:24:49.078 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316060+0000", 2024-09-13T11:24:49.078 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.001469515 2024-09-13T11:24:49.078 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:49.078 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:49.078 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-09-13T11:24:49.078 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316076+0000", 2024-09-13T11:24:49.078 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.5608999999999999e-05 2024-09-13T11:24:49.078 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:49.078 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:49.078 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:24:49.078 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.178857+0000", 2024-09-13T11:24:49.078 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.86278099900000005 2024-09-13T11:24:49.078 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:49.078 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:49.079 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:24:49.079 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.178961+0000", 2024-09-13T11:24:49.079 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000104664 2024-09-13T11:24:49.079 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:49.079 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:49.079 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1", 2024-09-13T11:24:49.079 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.179123+0000", 2024-09-13T11:24:49.079 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000161738 2024-09-13T11:24:49.079 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:49.079 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:49.079 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:24:49.079 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.182755+0000", 2024-09-13T11:24:49.079 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0036323200000000001 2024-09-13T11:24:49.079 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:49.079 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:49.080 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:49.080 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.183023+0000", 2024-09-13T11:24:49.080 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00026749100000000002 2024-09-13T11:24:49.080 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:49.080 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:49.080 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:24:49.080 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.183168+0000", 2024-09-13T11:24:49.080 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00014499099999999999 2024-09-13T11:24:49.080 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:49.080 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:49.080 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:24:49.080 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.183285+0000", 2024-09-13T11:24:49.080 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000117119 2024-09-13T11:24:49.080 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:49.081 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:24:49.081 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:49.081 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:49.081 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:49.081 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:60 3.b 3:d67218b0:::benchmark_data_smithi177_31116_object59:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T11:24:49.081 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:48.314603+0000", 2024-09-13T11:24:49.081 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 120.70858634, 2024-09-13T11:24:49.081 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.87106201100000002, 2024-09-13T11:24:49.081 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:24:49.081 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:24:49.081 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:49.081 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:24:49.081 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:24:49.081 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:49.082 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 60 2024-09-13T11:24:49.082 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:49.082 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:24:49.082 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:49.082 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:24:49.082 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314603+0000", 2024-09-13T11:24:49.082 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:49.082 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:49.082 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:49.082 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:24:49.082 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314603+0000", 2024-09-13T11:24:49.082 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:49.082 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:49.083 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:49.083 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:24:49.083 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314604+0000", 2024-09-13T11:24:49.083 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.2359999999999999e-06 2024-09-13T11:24:49.083 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:49.083 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:49.083 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:24:49.083 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314667+0000", 2024-09-13T11:24:49.083 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.3077999999999995e-05 2024-09-13T11:24:49.083 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:49.083 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:49.083 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:24:49.083 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314699+0000", 2024-09-13T11:24:49.083 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.1965999999999999e-05 2024-09-13T11:24:49.083 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:49.084 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:49.084 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:24:49.084 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314710+0000", 2024-09-13T11:24:49.084 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.1148999999999999e-05 2024-09-13T11:24:49.084 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:49.084 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:49.084 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:24:49.084 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316291+0000", 2024-09-13T11:24:49.084 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0015804440000000001 2024-09-13T11:24:49.084 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:49.084 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:49.084 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-09-13T11:24:49.084 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316310+0000", 2024-09-13T11:24:49.085 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.9799000000000001e-05 2024-09-13T11:24:49.085 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:49.085 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:49.085 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:24:49.085 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.179555+0000", 2024-09-13T11:24:49.085 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.86324469500000001 2024-09-13T11:24:49.085 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:49.085 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:49.085 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:24:49.085 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.179666+0000", 2024-09-13T11:24:49.085 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000110443 2024-09-13T11:24:49.085 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:49.085 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:49.085 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1", 2024-09-13T11:24:49.085 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.179791+0000", 2024-09-13T11:24:49.086 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00012560499999999999 2024-09-13T11:24:49.086 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:49.086 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:49.086 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:24:49.086 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.182761+0000", 2024-09-13T11:24:49.086 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0029697019999999998 2024-09-13T11:24:49.086 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:49.086 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:49.086 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:49.086 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.185572+0000", 2024-09-13T11:24:49.086 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.002811512 2024-09-13T11:24:49.086 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:49.086 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:49.086 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:24:49.087 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.185625+0000", 2024-09-13T11:24:49.087 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.2716000000000001e-05 2024-09-13T11:24:49.087 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:49.087 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:49.087 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:24:49.087 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.185665+0000", 2024-09-13T11:24:49.087 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.9666000000000003e-05 2024-09-13T11:24:49.087 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:49.087 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:24:49.087 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:49.087 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:49.087 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:49.087 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:62 3.b 3:d20ae5a8:::benchmark_data_smithi177_31116_object61:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T11:24:49.087 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:48.314723+0000", 2024-09-13T11:24:49.088 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 120.70846579800001, 2024-09-13T11:24:49.088 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.87338985099999999, 2024-09-13T11:24:49.088 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:24:49.088 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:24:49.088 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:49.088 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:24:49.088 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:24:49.088 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:49.088 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 62 2024-09-13T11:24:49.088 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:49.088 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:24:49.088 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:49.088 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:24:49.088 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314723+0000", 2024-09-13T11:24:49.089 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:49.089 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:49.089 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:49.089 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:24:49.089 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314723+0000", 2024-09-13T11:24:49.089 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:49.089 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:49.089 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:49.089 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:24:49.089 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314725+0000", 2024-09-13T11:24:49.089 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.3069999999999999e-06 2024-09-13T11:24:49.089 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:49.089 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:49.089 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:24:49.090 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314797+0000", 2024-09-13T11:24:49.090 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.2652999999999997e-05 2024-09-13T11:24:49.090 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:49.090 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:49.090 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:24:49.090 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314809+0000", 2024-09-13T11:24:49.090 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.1496e-05 2024-09-13T11:24:49.090 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:49.090 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:49.090 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:24:49.090 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314816+0000", 2024-09-13T11:24:49.090 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.6639999999999998e-06 2024-09-13T11:24:49.090 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:49.090 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:49.091 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:24:49.091 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316424+0000", 2024-09-13T11:24:49.091 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0016077559999999999 2024-09-13T11:24:49.091 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:49.091 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:49.091 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-09-13T11:24:49.091 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316441+0000", 2024-09-13T11:24:49.091 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.7305000000000001e-05 2024-09-13T11:24:49.091 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:49.091 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:49.091 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:24:49.091 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.180688+0000", 2024-09-13T11:24:49.091 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.86424692800000003 2024-09-13T11:24:49.091 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:49.091 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:49.092 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:24:49.092 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.180807+0000", 2024-09-13T11:24:49.092 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000118977 2024-09-13T11:24:49.092 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:49.092 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:49.092 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1", 2024-09-13T11:24:49.092 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.180938+0000", 2024-09-13T11:24:49.092 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00013049700000000001 2024-09-13T11:24:49.092 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:49.092 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:49.092 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:24:49.092 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.185680+0000", 2024-09-13T11:24:49.092 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0047417029999999999 2024-09-13T11:24:49.093 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:49.093 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:49.093 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:49.093 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.187856+0000", 2024-09-13T11:24:49.093 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0021760550000000001 2024-09-13T11:24:49.093 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:49.093 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:49.093 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:24:49.093 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.188007+0000", 2024-09-13T11:24:49.093 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00015185099999999999 2024-09-13T11:24:49.093 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:49.093 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:49.093 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:24:49.094 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.188113+0000", 2024-09-13T11:24:49.094 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000105659 2024-09-13T11:24:49.094 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:49.094 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:24:49.094 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:49.094 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:49.094 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:49.094 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:73 3.b 3:d0060d5f:::benchmark_data_smithi177_31116_object72:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T11:24:49.094 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:48.314829+0000", 2024-09-13T11:24:49.094 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 120.708360449, 2024-09-13T11:24:49.094 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.87364757199999998, 2024-09-13T11:24:49.094 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:24:49.095 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:24:49.095 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:49.095 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:24:49.095 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:24:49.095 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:49.095 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 73 2024-09-13T11:24:49.095 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:49.095 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:24:49.095 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:49.095 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:24:49.095 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314829+0000", 2024-09-13T11:24:49.095 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:49.095 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:49.095 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:49.095 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:24:49.096 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314829+0000", 2024-09-13T11:24:49.096 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:49.096 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:49.096 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:49.096 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:24:49.096 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314830+0000", 2024-09-13T11:24:49.096 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.243e-06 2024-09-13T11:24:49.096 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:49.096 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:49.096 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:24:49.096 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314965+0000", 2024-09-13T11:24:49.096 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00013470800000000001 2024-09-13T11:24:49.096 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:49.097 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:49.097 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:24:49.097 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314979+0000", 2024-09-13T11:24:49.097 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.4375999999999999e-05 2024-09-13T11:24:49.097 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:49.097 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:49.097 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:24:49.097 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314988+0000", 2024-09-13T11:24:49.097 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.8570000000000001e-06 2024-09-13T11:24:49.097 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:49.097 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:49.097 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:24:49.097 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316545+0000", 2024-09-13T11:24:49.097 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.001556878 2024-09-13T11:24:49.098 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:49.098 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:49.098 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-09-13T11:24:49.098 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316556+0000", 2024-09-13T11:24:49.098 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.1453e-05 2024-09-13T11:24:49.098 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:49.098 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:49.098 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:24:49.098 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.182124+0000", 2024-09-13T11:24:49.098 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.86556744200000002 2024-09-13T11:24:49.098 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:49.098 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:49.098 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:24:49.098 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.182246+0000", 2024-09-13T11:24:49.099 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00012210500000000001 2024-09-13T11:24:49.099 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:49.099 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:49.099 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1", 2024-09-13T11:24:49.099 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.182375+0000", 2024-09-13T11:24:49.099 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00012920999999999999 2024-09-13T11:24:49.099 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:49.099 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:49.099 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:24:49.099 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.185684+0000", 2024-09-13T11:24:49.099 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0033093969999999999 2024-09-13T11:24:49.099 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:49.099 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:49.099 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:49.099 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.188244+0000", 2024-09-13T11:24:49.100 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0025601550000000002 2024-09-13T11:24:49.100 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:49.100 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:49.100 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:24:49.100 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.188379+0000", 2024-09-13T11:24:49.100 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00013463400000000001 2024-09-13T11:24:49.100 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:49.100 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:49.100 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:24:49.100 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.188476+0000", 2024-09-13T11:24:49.100 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.7114000000000006e-05 2024-09-13T11:24:49.100 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:49.100 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:24:49.100 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:49.101 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:49.101 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:49.101 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:81 3.b 3:d705fd70:::benchmark_data_smithi177_31116_object80:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T11:24:49.101 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:48.315000+0000", 2024-09-13T11:24:49.101 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 120.708188612, 2024-09-13T11:24:49.101 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.87561418300000005, 2024-09-13T11:24:49.101 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:24:49.101 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:24:49.101 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:49.101 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:24:49.101 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:24:49.101 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:49.101 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 81 2024-09-13T11:24:49.101 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:49.102 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:24:49.102 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:49.102 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:24:49.102 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315000+0000", 2024-09-13T11:24:49.102 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:49.102 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:49.102 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:49.102 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:24:49.102 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315000+0000", 2024-09-13T11:24:49.102 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:49.102 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:49.102 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:49.102 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:24:49.102 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315002+0000", 2024-09-13T11:24:49.103 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.3799999999999999e-06 2024-09-13T11:24:49.103 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:49.103 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:49.103 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:24:49.103 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315119+0000", 2024-09-13T11:24:49.103 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000116925 2024-09-13T11:24:49.103 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:49.103 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:49.103 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:24:49.103 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315133+0000", 2024-09-13T11:24:49.103 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.3971999999999999e-05 2024-09-13T11:24:49.103 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:49.103 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:49.103 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:24:49.103 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315142+0000", 2024-09-13T11:24:49.104 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.4399999999999994e-06 2024-09-13T11:24:49.104 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:49.104 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:49.104 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:24:49.104 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316652+0000", 2024-09-13T11:24:49.104 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.001509907 2024-09-13T11:24:49.104 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:49.104 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:49.104 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-09-13T11:24:49.104 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316668+0000", 2024-09-13T11:24:49.104 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.5945999999999999e-05 2024-09-13T11:24:49.104 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:49.104 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:49.104 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:24:49.105 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.183405+0000", 2024-09-13T11:24:49.105 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.866737384 2024-09-13T11:24:49.105 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:49.105 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:49.105 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:24:49.105 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.183701+0000", 2024-09-13T11:24:49.105 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00029511099999999997 2024-09-13T11:24:49.105 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:49.105 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:49.105 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1", 2024-09-13T11:24:49.105 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.184078+0000", 2024-09-13T11:24:49.105 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00037741000000000001 2024-09-13T11:24:49.105 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:49.105 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:49.106 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:24:49.106 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.188504+0000", 2024-09-13T11:24:49.106 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0044260280000000003 2024-09-13T11:24:49.106 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:49.106 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:49.106 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:49.106 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.190562+0000", 2024-09-13T11:24:49.106 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0020580099999999999 2024-09-13T11:24:49.106 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:49.106 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:49.106 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:24:49.106 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.190592+0000", 2024-09-13T11:24:49.106 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.0057999999999999e-05 2024-09-13T11:24:49.106 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:49.107 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:49.107 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:24:49.107 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.190615+0000", 2024-09-13T11:24:49.107 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.2612e-05 2024-09-13T11:24:49.107 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:49.107 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:24:49.107 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:49.107 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:49.107 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:49.107 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:87 3.b 3:dd7de0ad:::benchmark_data_smithi177_31116_object86:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T11:24:49.107 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:48.315155+0000", 2024-09-13T11:24:49.107 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 120.70803381499999, 2024-09-13T11:24:49.107 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.87555916099999997, 2024-09-13T11:24:49.107 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:24:49.107 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:24:49.108 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:49.108 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:24:49.108 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:24:49.108 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:49.108 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 87 2024-09-13T11:24:49.108 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:49.108 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:24:49.108 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:49.108 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:24:49.108 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315155+0000", 2024-09-13T11:24:49.108 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:49.108 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:49.108 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:49.108 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:24:49.109 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315155+0000", 2024-09-13T11:24:49.109 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:49.109 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:49.109 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:49.109 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:24:49.109 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315157+0000", 2024-09-13T11:24:49.109 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.3829999999999999e-06 2024-09-13T11:24:49.109 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:49.109 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:49.109 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:24:49.109 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315271+0000", 2024-09-13T11:24:49.109 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000114539 2024-09-13T11:24:49.109 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:49.109 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:49.109 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:24:49.110 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315285+0000", 2024-09-13T11:24:49.110 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.4042e-05 2024-09-13T11:24:49.110 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:49.110 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:49.110 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:24:49.110 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315295+0000", 2024-09-13T11:24:49.110 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.2739999999999996e-06 2024-09-13T11:24:49.110 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:49.110 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:49.110 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:24:49.110 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316739+0000", 2024-09-13T11:24:49.110 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0014446960000000001 2024-09-13T11:24:49.110 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:49.110 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:49.111 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-09-13T11:24:49.111 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316747+0000", 2024-09-13T11:24:49.111 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.0790000000000001e-06 2024-09-13T11:24:49.111 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:49.111 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:49.111 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:24:49.111 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.184922+0000", 2024-09-13T11:24:49.111 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.86817446399999998 2024-09-13T11:24:49.111 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:49.111 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:49.111 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:24:49.111 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.185037+0000", 2024-09-13T11:24:49.111 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00011565099999999999 2024-09-13T11:24:49.111 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:49.112 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:49.112 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1", 2024-09-13T11:24:49.112 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.185171+0000", 2024-09-13T11:24:49.112 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00013316900000000001 2024-09-13T11:24:49.112 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:49.112 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:49.112 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:24:49.112 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.188520+0000", 2024-09-13T11:24:49.112 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0033489150000000001 2024-09-13T11:24:49.112 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:49.112 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:49.112 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:49.112 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.190652+0000", 2024-09-13T11:24:49.112 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0021325649999999999 2024-09-13T11:24:49.113 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:49.113 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:49.113 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:24:49.113 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.190686+0000", 2024-09-13T11:24:49.113 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.3827e-05 2024-09-13T11:24:49.113 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:49.113 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:49.113 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:24:49.113 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.190714+0000", 2024-09-13T11:24:49.113 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.8557000000000001e-05 2024-09-13T11:24:49.113 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:49.113 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:24:49.113 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:49.113 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:49.114 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:49.114 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:99 3.b 3:d011b600:::benchmark_data_smithi177_31116_object98:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T11:24:49.114 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:48.315307+0000", 2024-09-13T11:24:49.114 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 120.707881988, 2024-09-13T11:24:49.114 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.87551770799999995, 2024-09-13T11:24:49.114 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:24:49.114 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:24:49.114 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:49.114 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:24:49.114 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:24:49.114 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:49.114 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 99 2024-09-13T11:24:49.114 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:49.114 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:24:49.114 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:49.115 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:24:49.115 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315307+0000", 2024-09-13T11:24:49.115 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:49.115 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:49.115 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:49.115 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:24:49.115 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315307+0000", 2024-09-13T11:24:49.115 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:49.115 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:49.115 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:49.115 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:24:49.115 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315308+0000", 2024-09-13T11:24:49.115 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.316e-06 2024-09-13T11:24:49.115 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:49.116 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:49.116 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:24:49.116 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315453+0000", 2024-09-13T11:24:49.116 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00014473200000000001 2024-09-13T11:24:49.116 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:49.116 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:49.116 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:24:49.116 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315471+0000", 2024-09-13T11:24:49.116 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.8019000000000001e-05 2024-09-13T11:24:49.116 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:49.116 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:49.116 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:24:49.116 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315489+0000", 2024-09-13T11:24:49.116 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.7861999999999999e-05 2024-09-13T11:24:49.117 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:49.117 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:49.117 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:24:49.117 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316837+0000", 2024-09-13T11:24:49.117 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.001347787 2024-09-13T11:24:49.117 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:49.117 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:49.117 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-09-13T11:24:49.117 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316858+0000", 2024-09-13T11:24:49.117 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.1103000000000001e-05 2024-09-13T11:24:49.117 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:49.117 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:49.117 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:24:49.117 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.185713+0000", 2024-09-13T11:24:49.118 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.86885540800000005 2024-09-13T11:24:49.118 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:49.118 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:49.118 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:24:49.118 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.185820+0000", 2024-09-13T11:24:49.118 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000107081 2024-09-13T11:24:49.118 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:49.118 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:49.118 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1", 2024-09-13T11:24:49.118 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.185948+0000", 2024-09-13T11:24:49.118 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00012775800000000001 2024-09-13T11:24:49.118 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:49.118 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:49.118 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:24:49.119 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.190527+0000", 2024-09-13T11:24:49.119 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0045783910000000002 2024-09-13T11:24:49.119 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:49.119 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:49.119 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:49.119 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.190759+0000", 2024-09-13T11:24:49.119 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00023266199999999999 2024-09-13T11:24:49.119 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:49.119 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:49.119 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:24:49.119 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.190789+0000", 2024-09-13T11:24:49.119 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.9666999999999999e-05 2024-09-13T11:24:49.119 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:49.119 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:49.119 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:24:49.120 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.190825+0000", 2024-09-13T11:24:49.120 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.5921999999999999e-05 2024-09-13T11:24:49.120 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:49.120 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:24:49.120 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:49.120 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:49.120 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:49.120 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:108 3.b 3:d5c90217:::benchmark_data_smithi177_31116_object107:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T11:24:49.120 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:48.315513+0000", 2024-09-13T11:24:49.120 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 120.707676283, 2024-09-13T11:24:49.120 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.87544772500000001, 2024-09-13T11:24:49.120 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:24:49.120 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:24:49.120 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:49.120 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:24:49.121 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:24:49.121 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:49.121 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 108 2024-09-13T11:24:49.121 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:49.121 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:24:49.121 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:49.121 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:24:49.121 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315513+0000", 2024-09-13T11:24:49.121 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:49.121 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:49.121 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:49.121 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:24:49.121 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315513+0000", 2024-09-13T11:24:49.122 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:49.122 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:49.122 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:49.122 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:24:49.122 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315515+0000", 2024-09-13T11:24:49.122 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.9369999999999998e-06 2024-09-13T11:24:49.122 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:49.122 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:49.122 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:24:49.122 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315644+0000", 2024-09-13T11:24:49.122 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000129011 2024-09-13T11:24:49.122 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:49.122 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:49.122 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:24:49.123 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315657+0000", 2024-09-13T11:24:49.123 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.3689999999999999e-05 2024-09-13T11:24:49.123 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:49.123 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:49.123 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:24:49.123 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315668+0000", 2024-09-13T11:24:49.123 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0589e-05 2024-09-13T11:24:49.123 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:49.123 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:49.123 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:24:49.123 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316925+0000", 2024-09-13T11:24:49.123 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0012568340000000001 2024-09-13T11:24:49.123 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:49.123 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:49.124 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-09-13T11:24:49.124 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316932+0000", 2024-09-13T11:24:49.124 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.2230000000000001e-06 2024-09-13T11:24:49.124 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:49.124 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:49.124 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:24:49.124 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.186619+0000", 2024-09-13T11:24:49.124 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.86968679599999998 2024-09-13T11:24:49.124 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:49.124 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:49.124 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:24:49.124 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.186914+0000", 2024-09-13T11:24:49.124 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00029528900000000001 2024-09-13T11:24:49.125 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:49.125 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:49.125 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1", 2024-09-13T11:24:49.125 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.187217+0000", 2024-09-13T11:24:49.125 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00030232099999999998 2024-09-13T11:24:49.125 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:49.125 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:49.125 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:24:49.125 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.190532+0000", 2024-09-13T11:24:49.125 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.003315391 2024-09-13T11:24:49.125 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:49.125 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:49.125 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:49.125 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.190875+0000", 2024-09-13T11:24:49.126 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00034326300000000001 2024-09-13T11:24:49.126 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:49.126 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:49.126 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:24:49.126 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.190925+0000", 2024-09-13T11:24:49.126 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.9429999999999999e-05 2024-09-13T11:24:49.126 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:49.126 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:49.126 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:24:49.126 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.190961+0000", 2024-09-13T11:24:49.126 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.5951000000000002e-05 2024-09-13T11:24:49.126 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:49.126 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:24:49.126 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:49.127 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:49.127 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:49.127 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:172 3.b 3:dbec1eeb:::benchmark_data_smithi177_31116_object171:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T11:24:49.127 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:48.315684+0000", 2024-09-13T11:24:49.127 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 120.707505126, 2024-09-13T11:24:49.127 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.87615312999999995, 2024-09-13T11:24:49.127 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:24:49.127 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:24:49.127 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:49.127 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:24:49.127 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:24:49.127 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:49.127 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 172 2024-09-13T11:24:49.127 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:49.128 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:24:49.128 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:49.128 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:24:49.128 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315684+0000", 2024-09-13T11:24:49.128 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:49.128 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:49.128 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:49.128 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:24:49.128 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315684+0000", 2024-09-13T11:24:49.128 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:49.128 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:49.128 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:49.128 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:24:49.128 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315686+0000", 2024-09-13T11:24:49.129 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.886e-06 2024-09-13T11:24:49.129 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:49.129 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:49.129 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:24:49.129 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315820+0000", 2024-09-13T11:24:49.129 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00013388500000000001 2024-09-13T11:24:49.129 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:49.129 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:49.129 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:24:49.129 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315837+0000", 2024-09-13T11:24:49.129 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.6756e-05 2024-09-13T11:24:49.129 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:49.129 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:49.129 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:24:49.130 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315851+0000", 2024-09-13T11:24:49.130 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.4566000000000001e-05 2024-09-13T11:24:49.130 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:49.130 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:49.130 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:24:49.130 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.317021+0000", 2024-09-13T11:24:49.130 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0011701769999999999 2024-09-13T11:24:49.130 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:49.130 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:49.130 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-09-13T11:24:49.130 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.317037+0000", 2024-09-13T11:24:49.130 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.609e-05 2024-09-13T11:24:49.130 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:49.130 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:49.130 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:24:49.131 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.188634+0000", 2024-09-13T11:24:49.131 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.87159711900000003 2024-09-13T11:24:49.131 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:49.131 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:49.131 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:24:49.131 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.188974+0000", 2024-09-13T11:24:49.131 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00033922299999999999 2024-09-13T11:24:49.131 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:49.131 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:49.131 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1", 2024-09-13T11:24:49.131 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.189510+0000", 2024-09-13T11:24:49.131 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00053618999999999997 2024-09-13T11:24:49.131 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:49.131 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:49.132 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:49.132 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.191027+0000", 2024-09-13T11:24:49.132 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0015168390000000001 2024-09-13T11:24:49.132 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:49.132 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:49.132 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:24:49.132 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.191766+0000", 2024-09-13T11:24:49.132 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000739064 2024-09-13T11:24:49.132 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:49.132 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:49.132 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:24:49.132 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.191810+0000", 2024-09-13T11:24:49.132 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.3755999999999999e-05 2024-09-13T11:24:49.132 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:49.133 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:49.133 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:24:49.133 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.191837+0000", 2024-09-13T11:24:49.133 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.7579000000000001e-05 2024-09-13T11:24:49.133 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:49.133 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:24:49.133 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:49.133 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:49.133 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:49.133 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:177 3.b 3:de89c779:::benchmark_data_smithi177_31116_object176:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T11:24:49.133 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:48.315898+0000", 2024-09-13T11:24:49.133 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 120.70729081100001, 2024-09-13T11:24:49.133 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.87600712599999997, 2024-09-13T11:24:49.133 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:24:49.134 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:24:49.134 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:49.134 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:24:49.134 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:24:49.134 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:49.134 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 177 2024-09-13T11:24:49.134 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:49.134 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:24:49.134 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:49.134 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:24:49.134 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315898+0000", 2024-09-13T11:24:49.134 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:49.134 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:49.134 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:49.135 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:24:49.135 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315898+0000", 2024-09-13T11:24:49.135 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:49.135 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:49.135 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:49.135 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:24:49.135 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315900+0000", 2024-09-13T11:24:49.135 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.4929999999999999e-06 2024-09-13T11:24:49.135 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:49.135 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:49.135 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:24:49.135 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316027+0000", 2024-09-13T11:24:49.135 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000127568 2024-09-13T11:24:49.135 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:49.135 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:49.136 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:24:49.136 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316048+0000", 2024-09-13T11:24:49.136 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.0795999999999999e-05 2024-09-13T11:24:49.136 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:49.136 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:49.136 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:24:49.136 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316061+0000", 2024-09-13T11:24:49.136 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.3283e-05 2024-09-13T11:24:49.136 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:49.136 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:49.136 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:24:49.136 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.317107+0000", 2024-09-13T11:24:49.136 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0010452949999999999 2024-09-13T11:24:49.137 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:49.137 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:49.137 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-09-13T11:24:49.137 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.317114+0000", 2024-09-13T11:24:49.137 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.4569999999999999e-06 2024-09-13T11:24:49.137 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:49.137 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:49.137 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:24:49.137 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.190017+0000", 2024-09-13T11:24:49.137 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.87290258499999995 2024-09-13T11:24:49.137 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:49.137 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:49.137 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:24:49.137 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.190138+0000", 2024-09-13T11:24:49.138 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000121418 2024-09-13T11:24:49.138 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:49.138 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:49.138 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1", 2024-09-13T11:24:49.138 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.190216+0000", 2024-09-13T11:24:49.138 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.7410999999999995e-05 2024-09-13T11:24:49.138 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:49.138 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:49.138 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:49.138 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.191755+0000", 2024-09-13T11:24:49.138 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0015395420000000001 2024-09-13T11:24:49.138 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:49.138 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:49.138 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:24:49.138 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.191843+0000", 2024-09-13T11:24:49.139 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.8022999999999994e-05 2024-09-13T11:24:49.139 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:49.139 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:49.139 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:24:49.139 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.191876+0000", 2024-09-13T11:24:49.139 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.2619000000000001e-05 2024-09-13T11:24:49.139 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:49.139 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:49.139 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:24:49.139 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.191905+0000", 2024-09-13T11:24:49.139 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.9635999999999999e-05 2024-09-13T11:24:49.139 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:49.139 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:24:49.139 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:49.140 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:49.140 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:49.140 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:187 3.b 3:da03c8dd:::benchmark_data_smithi177_31116_object186:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T11:24:49.140 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:48.316083+0000", 2024-09-13T11:24:49.140 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 120.70710585499999, 2024-09-13T11:24:49.140 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.87835499500000003, 2024-09-13T11:24:49.140 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:24:49.140 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:24:49.140 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:49.140 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:24:49.140 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:24:49.140 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:49.140 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 187 2024-09-13T11:24:49.140 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:49.141 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:24:49.141 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:49.141 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:24:49.141 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316083+0000", 2024-09-13T11:24:49.141 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:49.141 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:49.141 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:49.141 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:24:49.141 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316083+0000", 2024-09-13T11:24:49.141 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:49.141 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:49.141 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:49.141 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:24:49.141 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316086+0000", 2024-09-13T11:24:49.142 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.5560000000000002e-06 2024-09-13T11:24:49.142 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:49.142 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:49.142 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:24:49.142 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316199+0000", 2024-09-13T11:24:49.142 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000113522 2024-09-13T11:24:49.142 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:49.142 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:49.142 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:24:49.142 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316215+0000", 2024-09-13T11:24:49.142 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.5265999999999999e-05 2024-09-13T11:24:49.142 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:49.142 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:49.142 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:24:49.142 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316226+0000", 2024-09-13T11:24:49.143 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0967e-05 2024-09-13T11:24:49.143 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:49.143 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:49.143 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:24:49.143 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.317198+0000", 2024-09-13T11:24:49.143 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00097194100000000004 2024-09-13T11:24:49.143 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:49.143 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:49.143 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-09-13T11:24:49.143 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.317213+0000", 2024-09-13T11:24:49.143 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.5472999999999999e-05 2024-09-13T11:24:49.143 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:49.143 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:49.143 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:24:49.144 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.191077+0000", 2024-09-13T11:24:49.144 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.87386435200000001 2024-09-13T11:24:49.144 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:49.144 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:49.144 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:24:49.144 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.191181+0000", 2024-09-13T11:24:49.144 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000103618 2024-09-13T11:24:49.144 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:49.144 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:49.144 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1", 2024-09-13T11:24:49.144 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.191305+0000", 2024-09-13T11:24:49.144 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00012358599999999999 2024-09-13T11:24:49.144 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:49.144 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:49.145 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:49.145 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.194279+0000", 2024-09-13T11:24:49.145 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0029746970000000001 2024-09-13T11:24:49.145 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:49.145 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:49.145 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:24:49.145 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.194336+0000", 2024-09-13T11:24:49.145 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.7244000000000003e-05 2024-09-13T11:24:49.145 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:49.145 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:49.145 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:24:49.145 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.194397+0000", 2024-09-13T11:24:49.145 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.0278000000000001e-05 2024-09-13T11:24:49.145 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:49.146 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:49.146 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:24:49.146 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.194438+0000", 2024-09-13T11:24:49.146 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.1495000000000002e-05 2024-09-13T11:24:49.146 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:49.146 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:24:49.146 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:49.146 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:49.146 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:49.146 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:200 3.b 3:d90039f1:::benchmark_data_smithi177_31116_object199:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T11:24:49.146 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:48.316280+0000", 2024-09-13T11:24:49.146 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 120.70690939000001, 2024-09-13T11:24:49.146 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.87879273000000002, 2024-09-13T11:24:49.146 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:24:49.146 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:24:49.147 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:49.147 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:24:49.147 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:24:49.147 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:49.147 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 200 2024-09-13T11:24:49.147 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:49.147 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:24:49.147 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:49.147 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:24:49.147 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316280+0000", 2024-09-13T11:24:49.147 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:49.147 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:49.147 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:49.147 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:24:49.147 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316280+0000", 2024-09-13T11:24:49.148 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:49.148 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:49.148 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:49.148 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:24:49.148 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316282+0000", 2024-09-13T11:24:49.148 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.497e-06 2024-09-13T11:24:49.148 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:49.148 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:49.148 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:24:49.148 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316411+0000", 2024-09-13T11:24:49.148 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00012924799999999999 2024-09-13T11:24:49.148 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:49.148 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:49.148 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:24:49.149 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316426+0000", 2024-09-13T11:24:49.149 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.4183e-05 2024-09-13T11:24:49.149 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:49.149 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:49.149 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:24:49.149 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316435+0000", 2024-09-13T11:24:49.149 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.0990000000000005e-06 2024-09-13T11:24:49.149 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:49.149 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:49.149 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:24:49.149 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.317250+0000", 2024-09-13T11:24:49.149 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00081501100000000003 2024-09-13T11:24:49.149 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:49.150 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:49.150 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-09-13T11:24:49.150 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.317257+0000", 2024-09-13T11:24:49.150 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.2169999999999997e-06 2024-09-13T11:24:49.150 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:49.150 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:49.150 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:24:49.150 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.191936+0000", 2024-09-13T11:24:49.150 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.87467922099999995 2024-09-13T11:24:49.150 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:49.150 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:49.150 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:24:49.150 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.192056+0000", 2024-09-13T11:24:49.150 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000119638 2024-09-13T11:24:49.151 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:49.151 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:49.151 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1", 2024-09-13T11:24:49.151 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.192184+0000", 2024-09-13T11:24:49.151 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00012813399999999999 2024-09-13T11:24:49.151 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:49.151 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:49.151 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:49.151 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.194324+0000", 2024-09-13T11:24:49.151 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0021399420000000001 2024-09-13T11:24:49.151 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:49.151 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:49.151 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:24:49.151 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.194802+0000", 2024-09-13T11:24:49.151 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00047798799999999999 2024-09-13T11:24:49.152 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:49.152 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:49.152 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:24:49.152 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.194951+0000", 2024-09-13T11:24:49.152 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0001494 2024-09-13T11:24:49.152 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:49.152 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:49.152 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:24:49.152 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.195072+0000", 2024-09-13T11:24:49.152 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00012115199999999999 2024-09-13T11:24:49.152 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:49.152 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:24:49.152 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:49.152 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:49.153 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:49.153 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop(client.4665.0:79801 3.5 e102/67 3:a6efdf84:::benchmark_data_smithi177_31116_object79800:head v 102'3851, mlcod=102'3848)", 2024-09-13T11:24:49.153 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:24:00.403754+0000", 2024-09-13T11:24:49.153 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 48.619435037999999, 2024-09-13T11:24:49.153 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.31749079899999999, 2024-09-13T11:24:49.153 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:24:49.153 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:24:49.153 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:49.153 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:24:49.153 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:49.153 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:24:49.153 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:00.403754+0000", 2024-09-13T11:24:49.153 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:49.153 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:49.154 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:49.154 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:24:49.154 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:00.403754+0000", 2024-09-13T11:24:49.154 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:49.154 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:49.154 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:49.154 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:24:49.154 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:00.403758+0000", 2024-09-13T11:24:49.154 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.507e-06 2024-09-13T11:24:49.154 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:49.154 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:49.154 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:24:49.154 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:00.403819+0000", 2024-09-13T11:24:49.154 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.1042000000000003e-05 2024-09-13T11:24:49.155 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:49.155 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:49.155 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:24:49.155 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:00.717963+0000", 2024-09-13T11:24:49.155 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.314144594 2024-09-13T11:24:49.155 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:49.155 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:49.155 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:24:49.155 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:00.717976+0000", 2024-09-13T11:24:49.155 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.3029e-05 2024-09-13T11:24:49.155 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:49.155 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:49.155 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:24:49.155 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:00.718143+0000", 2024-09-13T11:24:49.156 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00016639000000000001 2024-09-13T11:24:49.156 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:49.156 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:49.156 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:24:49.156 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:00.718184+0000", 2024-09-13T11:24:49.156 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.1801999999999997e-05 2024-09-13T11:24:49.156 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:49.156 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:49.156 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:24:49.156 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:00.721162+0000", 2024-09-13T11:24:49.156 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0029772800000000001 2024-09-13T11:24:49.156 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:49.156 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:49.156 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:24:49.157 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:00.721245+0000", 2024-09-13T11:24:49.157 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.3154999999999999e-05 2024-09-13T11:24:49.157 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:49.157 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:24:49.157 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:49.157 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:49.157 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:49.157 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "MOSDPGPushReply(3.26 121/121 [PushReplyOp(3:670b9aab:::benchmark_data_smithi177_31116_object93:head)])", 2024-09-13T11:24:49.157 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:24:27.300318+0000", 2024-09-13T11:24:49.157 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 21.722871170000001, 2024-09-13T11:24:49.157 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.811436728, 2024-09-13T11:24:49.157 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:24:49.157 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:24:49.157 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "reached pg", 2024-09-13T11:24:49.158 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:24:49.158 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:49.158 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:24:49.158 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:27.300318+0000", 2024-09-13T11:24:49.158 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:49.158 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:49.158 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:49.158 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:24:49.158 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:27.300318+0000", 2024-09-13T11:24:49.158 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:49.158 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:49.158 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:49.158 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:24:49.158 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:27.300320+0000", 2024-09-13T11:24:49.158 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.778e-06 2024-09-13T11:24:49.159 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:49.159 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:49.159 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:24:49.159 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:27.300326+0000", 2024-09-13T11:24:49.159 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.7700000000000004e-06 2024-09-13T11:24:49.159 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:49.159 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:49.159 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:24:49.159 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:28.109665+0000", 2024-09-13T11:24:49.159 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.80933853499999997 2024-09-13T11:24:49.159 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:49.159 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:49.159 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:24:49.159 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:28.109678+0000", 2024-09-13T11:24:49.160 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.2870999999999999e-05 2024-09-13T11:24:49.160 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:49.160 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:49.160 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:24:49.160 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:28.109800+0000", 2024-09-13T11:24:49.160 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000122328 2024-09-13T11:24:49.160 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:49.160 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:49.160 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:24:49.160 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:28.111755+0000", 2024-09-13T11:24:49.160 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0019544459999999999 2024-09-13T11:24:49.160 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:49.160 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:24:49.160 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:49.161 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:49.161 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:49.161 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "MOSDPGPushReply(3.6 125/122 [PushReplyOp(3:61422348:::benchmark_data_smithi177_31116_object23374:head)])", 2024-09-13T11:24:49.161 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:24:32.775743+0000", 2024-09-13T11:24:49.161 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 16.247446313000001, 2024-09-13T11:24:49.161 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.304090953, 2024-09-13T11:24:49.161 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:24:49.161 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:24:49.161 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "reached pg", 2024-09-13T11:24:49.161 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:24:49.161 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:49.161 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:24:49.161 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:32.775743+0000", 2024-09-13T11:24:49.161 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:49.161 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:49.162 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:49.162 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:24:49.162 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:32.775743+0000", 2024-09-13T11:24:49.162 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:49.162 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:49.162 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:49.162 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:24:49.162 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:32.775744+0000", 2024-09-13T11:24:49.162 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.522e-06 2024-09-13T11:24:49.162 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:49.162 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:49.162 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:24:49.162 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:32.775753+0000", 2024-09-13T11:24:49.162 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.2889999999999998e-06 2024-09-13T11:24:49.163 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:49.163 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:49.163 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:24:49.163 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:33.078899+0000", 2024-09-13T11:24:49.163 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.30314683399999998 2024-09-13T11:24:49.163 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:49.163 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:49.163 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:24:49.163 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:33.078906+0000", 2024-09-13T11:24:49.163 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.9029999999999998e-06 2024-09-13T11:24:49.163 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:49.163 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:49.163 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:24:49.163 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:33.078954+0000", 2024-09-13T11:24:49.164 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.7754999999999999e-05 2024-09-13T11:24:49.164 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:49.164 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:49.164 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:24:49.164 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:33.079834+0000", 2024-09-13T11:24:49.164 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00087965000000000005 2024-09-13T11:24:49.164 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:49.164 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:24:49.164 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:49.164 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:49.164 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:24:49.164 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-09-13T11:24:49.166 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-13T11:24:49.166 INFO:tasks.ceph.osd.10.smithi162.stderr:2024-09-13T11:24:49.164+0000 7fb59c9f0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26317) UID: 0 2024-09-13T11:24:49.266 INFO:tasks.ceph.osd.10.smithi162.stderr:2024-09-13T11:24:49.264+0000 7fb59c9f0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26317) UID: 0 2024-09-13T11:24:49.312 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-13T11:24:49.313 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [ 2024-09-13T11:24:49.313 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:49.313 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:132010 3.29 3:964e96dd:::benchmark_data_smithi177_31116_object132009:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e136)", 2024-09-13T11:24:49.313 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:24:49.310055+0000", 2024-09-13T11:24:49.313 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 0.00060775200000000003, 2024-09-13T11:24:49.313 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00064623899999999995, 2024-09-13T11:24:49.313 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:24:49.313 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:24:49.313 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:24:49.313 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:24:49.313 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:24:49.313 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:49.314 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 132010 2024-09-13T11:24:49.314 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:49.314 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:24:49.314 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:49.314 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:24:49.314 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:49.310055+0000", 2024-09-13T11:24:49.314 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:49.314 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:49.314 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:49.314 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:24:49.314 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:49.310055+0000", 2024-09-13T11:24:49.315 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:49.315 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:49.315 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:49.315 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:24:49.315 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:49.310058+0000", 2024-09-13T11:24:49.315 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.9639999999999999e-06 2024-09-13T11:24:49.315 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:49.315 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:49.315 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:24:49.315 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:49.310152+0000", 2024-09-13T11:24:49.315 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.3881999999999993e-05 2024-09-13T11:24:49.315 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:49.315 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:49.315 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:24:49.316 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:49.310167+0000", 2024-09-13T11:24:49.316 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4951e-05 2024-09-13T11:24:49.316 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:49.316 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:49.316 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:24:49.316 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:49.310174+0000", 2024-09-13T11:24:49.316 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.182e-06 2024-09-13T11:24:49.316 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:49.316 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:49.316 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:24:49.316 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:49.310254+0000", 2024-09-13T11:24:49.316 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.9208999999999993e-05 2024-09-13T11:24:49.316 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:49.317 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:49.317 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:24:49.317 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:49.310482+0000", 2024-09-13T11:24:49.317 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000227891 2024-09-13T11:24:49.317 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:49.317 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:24:49.317 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:49.317 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:49.317 INFO:teuthology.orchestra.run.smithi162.stdout: ], 2024-09-13T11:24:49.317 INFO:teuthology.orchestra.run.smithi162.stdout: "num_ops": 1 2024-09-13T11:24:49.317 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-13T11:24:49.331 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-13T11:24:49.352 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T11:24:49.396 INFO:tasks.ceph.osd.8.smithi001.stderr:2024-09-13T11:24:49.395+0000 7fb9b48ae640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26312) UID: 0 2024-09-13T11:24:49.468 INFO:tasks.ceph.osd.7.smithi177.stderr:2024-09-13T11:24:49.466+0000 7f897b51d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26575) UID: 0 2024-09-13T11:24:49.516 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-13T11:24:49.516 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [], 2024-09-13T11:24:49.516 INFO:teuthology.orchestra.run.smithi162.stdout: "complaint_time": 30, 2024-09-13T11:24:49.516 INFO:teuthology.orchestra.run.smithi162.stdout: "num_blocked_ops": 0 2024-09-13T11:24:49.516 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-13T11:24:49.539 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-13T11:24:49.551 INFO:teuthology.orchestra.run.smithi001.stdout:osd.0: {} 2024-09-13T11:24:49.551 INFO:teuthology.orchestra.run.smithi001.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T11:24:49.570 INFO:tasks.ceph.osd.12.smithi001.stderr:2024-09-13T11:24:49.567+0000 7f892c6e6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26314) UID: 0 2024-09-13T11:24:49.580 INFO:teuthology.orchestra.run.smithi001.stdout:osd.1: {} 2024-09-13T11:24:49.581 INFO:teuthology.orchestra.run.smithi001.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T11:24:49.614 INFO:teuthology.orchestra.run.smithi001.stdout:osd.2: {} 2024-09-13T11:24:49.614 INFO:teuthology.orchestra.run.smithi001.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T11:24:49.646 INFO:teuthology.orchestra.run.smithi001.stdout:osd.3: {} 2024-09-13T11:24:49.646 INFO:teuthology.orchestra.run.smithi001.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T11:24:49.669 INFO:tasks.ceph.osd.12.smithi001.stderr:2024-09-13T11:24:49.667+0000 7f892c6e6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26314) UID: 0 2024-09-13T11:24:49.677 INFO:teuthology.orchestra.run.smithi001.stdout:osd.4: {} 2024-09-13T11:24:49.678 INFO:teuthology.orchestra.run.smithi001.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T11:24:49.681 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-13T11:24:49.681 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:24:49.700 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-13T11:24:49.712 INFO:teuthology.orchestra.run.smithi001.stdout:osd.5: {} 2024-09-13T11:24:49.712 INFO:teuthology.orchestra.run.smithi001.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T11:24:49.744 INFO:teuthology.orchestra.run.smithi001.stdout:osd.6: {} 2024-09-13T11:24:49.744 INFO:teuthology.orchestra.run.smithi001.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T11:24:49.770 INFO:tasks.ceph.osd.3.smithi177.stderr:2024-09-13T11:24:49.766+0000 7f10adba4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26574) UID: 0 2024-09-13T11:24:49.772 INFO:teuthology.orchestra.run.smithi001.stdout:osd.7: {} 2024-09-13T11:24:49.773 INFO:teuthology.orchestra.run.smithi001.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T11:24:49.803 INFO:teuthology.orchestra.run.smithi001.stdout:osd.8: {} 2024-09-13T11:24:49.803 INFO:teuthology.orchestra.run.smithi001.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-13T11:24:49.831 INFO:teuthology.orchestra.run.smithi001.stdout:osd.9: {} 2024-09-13T11:24:49.832 INFO:teuthology.orchestra.run.smithi001.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-13T11:24:49.842 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-13T11:24:49.842 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:24:49.858 INFO:teuthology.orchestra.run.smithi001.stdout:osd.10: {} 2024-09-13T11:24:49.858 INFO:teuthology.orchestra.run.smithi001.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-13T11:24:49.865 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-13T11:24:49.887 INFO:teuthology.orchestra.run.smithi001.stdout:osd.11: {} 2024-09-13T11:24:49.888 INFO:teuthology.orchestra.run.smithi001.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-13T11:24:49.910 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-09-13T11:24:49.908+0000 7f97befc9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26315) UID: 0 2024-09-13T11:24:49.915 INFO:teuthology.orchestra.run.smithi001.stdout:osd.12: {} 2024-09-13T11:24:49.915 INFO:teuthology.orchestra.run.smithi001.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-13T11:24:49.944 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-09-13T11:24:49.940+0000 7f97b59fa640 -1 osd.6 136 handle_osd_map injecting map crc failure 2024-09-13T11:24:49.946 INFO:teuthology.orchestra.run.smithi001.stderr:noscrub is set 2024-09-13T11:24:49.947 INFO:tasks.ceph.osd.1.smithi092.stderr:2024-09-13T11:24:49.942+0000 7fe76f3f6640 -1 osd.1 136 handle_osd_map injecting map crc failure 2024-09-13T11:24:49.951 INFO:teuthology.orchestra.run.smithi001.stdout:osd.13: {} 2024-09-13T11:24:49.951 INFO:teuthology.orchestra.run.smithi001.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-13T11:24:49.951 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-13T11:24:49.947+0000 7f6a5f1a6640 -1 osd.0 136 handle_osd_map injecting map crc failure 2024-09-13T11:24:49.956 INFO:teuthology.orchestra.run.smithi001.stderr:Error ENXIO: problem getting command descriptions from osd.14 2024-09-13T11:24:49.972 INFO:tasks.ceph.osd.1.smithi092.stderr:2024-09-13T11:24:49.966+0000 7fe7789c5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33718) UID: 0 2024-09-13T11:24:49.987 INFO:teuthology.orchestra.run.smithi001.stdout:osd.15: {} 2024-09-13T11:24:49.988 INFO:teuthology.orchestra.run.smithi001.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-13T11:24:49.991 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-13T11:24:49.992 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-13T11:24:50.004 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-13T11:24:50.004 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:24:50.013 INFO:tasks.radosbench.radosbench.0.smithi177.stdout: 143 16 132994 132978 58.1135 77.375 0.00491869 0.0172031 2024-09-13T11:24:50.027 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-13T11:24:50.114 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-09-13T11:24:50.112+0000 7f97befc9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26315) UID: 0 2024-09-13T11:24:50.175 INFO:tasks.ceph.osd.7.smithi177.stderr:2024-09-13T11:24:50.170+0000 7f897b51d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26575) UID: 0 2024-09-13T11:24:50.210 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-13T11:24:50.210 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:24:50.228 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-09-13T11:24:50.314 INFO:tasks.ceph.osd.10.smithi162.stderr:2024-09-13T11:24:50.312+0000 7fb59c9f0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26317) UID: 0 2024-09-13T11:24:50.379 INFO:tasks.ceph.osd.13.smithi092.stderr:2024-09-13T11:24:50.370+0000 7f3e5afea640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26350) UID: 0 2024-09-13T11:24:50.439 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-13T11:24:50.439 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:24:50.439 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-09-13T11:24:50.477 INFO:tasks.ceph.osd.15.smithi177.stderr:2024-09-13T11:24:50.474+0000 7f9d6ce77640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26578) UID: 0 2024-09-13T11:24:50.579 INFO:tasks.ceph.osd.11.smithi177.stderr:2024-09-13T11:24:50.574+0000 7f24792c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26580) UID: 0 2024-09-13T11:24:50.615 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-13T11:24:50.616 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:24:50.631 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-09-13T11:24:50.679 INFO:tasks.ceph.osd.11.smithi177.stderr:2024-09-13T11:24:50.674+0000 7f24792c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26580) UID: 0 2024-09-13T11:24:50.769 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-13T11:24:50.769 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:24:50.780 INFO:tasks.ceph.osd.12.smithi001.stderr:2024-09-13T11:24:50.779+0000 7f892c6e6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26314) UID: 0 2024-09-13T11:24:50.789 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-13T11:24:50.882 INFO:tasks.ceph.osd.10.smithi162.stderr:2024-09-13T11:24:50.880+0000 7fb59c9f0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26317) UID: 0 2024-09-13T11:24:50.951 INFO:tasks.ceph.osd.4.smithi001.stderr:2024-09-13T11:24:50.947+0000 7ff8d0c3d640 -1 osd.4 137 handle_osd_map injecting map crc failure 2024-09-13T11:24:50.956 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-09-13T11:24:50.956 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:24:50.983 INFO:tasks.ceph.osd.10.smithi162.stderr:2024-09-13T11:24:50.980+0000 7fb59c9f0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26317) UID: 0 2024-09-13T11:24:50.986 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-13T11:24:50.989 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_ops_in_flight 2024-09-13T11:24:51.014 INFO:tasks.radosbench.radosbench.0.smithi177.stdout: 144 16 134398 134382 58.3189 87.75 0.00342899 0.0171425 2024-09-13T11:24:51.120 INFO:teuthology.orchestra.run.smithi162.stderr:no valid command found; 10 closest matches: 2024-09-13T11:24:51.120 INFO:teuthology.orchestra.run.smithi162.stderr:0 2024-09-13T11:24:51.120 INFO:teuthology.orchestra.run.smithi162.stderr:1 2024-09-13T11:24:51.120 INFO:teuthology.orchestra.run.smithi162.stderr:2 2024-09-13T11:24:51.120 INFO:teuthology.orchestra.run.smithi162.stderr:abort 2024-09-13T11:24:51.120 INFO:teuthology.orchestra.run.smithi162.stderr:assert 2024-09-13T11:24:51.120 INFO:teuthology.orchestra.run.smithi162.stderr:config diff 2024-09-13T11:24:51.120 INFO:teuthology.orchestra.run.smithi162.stderr:config diff get 2024-09-13T11:24:51.120 INFO:teuthology.orchestra.run.smithi162.stderr:config get 2024-09-13T11:24:51.120 INFO:teuthology.orchestra.run.smithi162.stderr:config help [] 2024-09-13T11:24:51.121 INFO:teuthology.orchestra.run.smithi162.stderr:config set ... 2024-09-13T11:24:51.121 INFO:teuthology.orchestra.run.smithi162.stderr:admin_socket: invalid command 2024-09-13T11:24:51.122 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-13T11:24:51.122 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-14, ['dump_ops_in_flight'] 2024-09-13T11:24:51.127 INFO:tasks.ceph.osd.11.smithi177.stderr:2024-09-13T11:24:51.078+0000 7f24792c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26580) UID: 0 2024-09-13T11:24:51.136 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-09-13T11:24:51.136 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:24:51.164 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-13T11:24:51.185 INFO:tasks.ceph.osd.5.smithi092.stderr:2024-09-13T11:24:51.178+0000 7f6f0b705640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26346) UID: 0 2024-09-13T11:24:51.285 INFO:tasks.ceph.osd.3.smithi177.stderr:2024-09-13T11:24:51.282+0000 7f10adba4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26574) UID: 0 2024-09-13T11:24:51.320 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-09-13T11:24:51.321 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:24:51.351 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-13T11:24:51.386 INFO:tasks.ceph.osd.10.smithi162.stderr:2024-09-13T11:24:51.384+0000 7fb59c9f0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26317) UID: 0 2024-09-13T11:24:51.489 INFO:tasks.ceph.osd.15.smithi177.stderr:2024-09-13T11:24:51.486+0000 7f9d6ce77640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26578) UID: 0 2024-09-13T11:24:51.572 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-09-13T11:24:51.572 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:24:51.589 INFO:tasks.ceph.osd.12.smithi001.stderr:2024-09-13T11:24:51.587+0000 7f892c6e6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26314) UID: 0 2024-09-13T11:24:51.594 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-13T11:24:51.689 INFO:tasks.ceph.osd.9.smithi092.stderr:2024-09-13T11:24:51.686+0000 7ff5ab3d0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26348) UID: 0 2024-09-13T11:24:51.787 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-09-13T11:24:51.787 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:24:51.790 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-13T11:24:51.787+0000 7f6a68775640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26307) UID: 0 2024-09-13T11:24:51.811 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-13T11:24:51.890 INFO:tasks.ceph.osd.3.smithi177.stderr:2024-09-13T11:24:51.886+0000 7f10adba4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26574) UID: 0 2024-09-13T11:24:51.974 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-13T11:24:51.985 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-09-13T11:24:51.985 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:24:51.993 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T11:24:52.010 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-09-13T11:24:52.014 INFO:tasks.radosbench.radosbench.0.smithi177.stdout: 145 16 135633 135617 58.449 77.1875 0.00288245 0.0171027 2024-09-13T11:24:52.020 INFO:tasks.ceph.osd.12.smithi001.stderr:2024-09-13T11:24:52.019+0000 7f892c6e6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26314) UID: 0 2024-09-13T11:24:52.094 INFO:tasks.ceph.osd.12.smithi001.stderr:2024-09-13T11:24:52.091+0000 7f892c6e6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26314) UID: 0 2024-09-13T11:24:52.171 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-09-13T11:24:52.171 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:24:52.196 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-09-13T11:24:52.196 INFO:tasks.ceph.osd.4.smithi001.stderr:2024-09-13T11:24:52.191+0000 7ff8e0218640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26310) UID: 0 2024-09-13T11:24:52.235 INFO:teuthology.orchestra.run.smithi001.stdout:osd.0: {} 2024-09-13T11:24:52.235 INFO:teuthology.orchestra.run.smithi001.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T11:24:52.267 INFO:teuthology.orchestra.run.smithi001.stdout:osd.1: {} 2024-09-13T11:24:52.267 INFO:teuthology.orchestra.run.smithi001.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T11:24:52.296 INFO:tasks.ceph.osd.7.smithi177.stderr:2024-09-13T11:24:52.294+0000 7f897b51d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26575) UID: 0 2024-09-13T11:24:52.305 INFO:teuthology.orchestra.run.smithi001.stdout:osd.2: {} 2024-09-13T11:24:52.306 INFO:teuthology.orchestra.run.smithi001.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T11:24:52.345 INFO:teuthology.orchestra.run.smithi001.stdout:osd.3: {} 2024-09-13T11:24:52.345 INFO:teuthology.orchestra.run.smithi001.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T11:24:52.365 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-09-13T11:24:52.365 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:24:52.371 INFO:teuthology.orchestra.run.smithi001.stdout:osd.4: {} 2024-09-13T11:24:52.371 INFO:teuthology.orchestra.run.smithi001.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T11:24:52.388 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-09-13T11:24:52.400 INFO:teuthology.orchestra.run.smithi001.stdout:osd.5: {} 2024-09-13T11:24:52.400 INFO:teuthology.orchestra.run.smithi001.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T11:24:52.426 INFO:teuthology.orchestra.run.smithi001.stdout:osd.6: {} 2024-09-13T11:24:52.426 INFO:teuthology.orchestra.run.smithi001.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T11:24:52.437 INFO:tasks.ceph.osd.7.smithi177.stderr:2024-09-13T11:24:52.434+0000 7f897b51d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26575) UID: 0 2024-09-13T11:24:52.455 INFO:teuthology.orchestra.run.smithi001.stdout:osd.7: {} 2024-09-13T11:24:52.455 INFO:teuthology.orchestra.run.smithi001.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T11:24:52.484 INFO:teuthology.orchestra.run.smithi001.stdout:osd.8: {} 2024-09-13T11:24:52.484 INFO:teuthology.orchestra.run.smithi001.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-13T11:24:52.498 INFO:tasks.ceph.osd.13.smithi092.stderr:2024-09-13T11:24:52.494+0000 7f3e5afea640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26350) UID: 0 2024-09-13T11:24:52.514 INFO:teuthology.orchestra.run.smithi001.stdout:osd.9: {} 2024-09-13T11:24:52.514 INFO:teuthology.orchestra.run.smithi001.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-13T11:24:52.546 INFO:teuthology.orchestra.run.smithi001.stdout:osd.10: {} 2024-09-13T11:24:52.546 INFO:teuthology.orchestra.run.smithi001.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-13T11:24:52.571 INFO:teuthology.orchestra.run.smithi001.stdout:osd.11: {} 2024-09-13T11:24:52.572 INFO:teuthology.orchestra.run.smithi001.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-13T11:24:52.598 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-09-13T11:24:52.598 INFO:teuthology.orchestra.run.smithi177.stdout: "size": 20, 2024-09-13T11:24:52.598 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 600, 2024-09-13T11:24:52.598 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [ 2024-09-13T11:24:52.598 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:52.599 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:79 3.1 e32/30 3:841df5b5:::benchmark_data_smithi177_31116_object78:head v 32'6, mlcod=32'4)", 2024-09-13T11:24:52.599 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:22:27.057292+0000", 2024-09-13T11:24:52.599 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 145.537422959, 2024-09-13T11:24:52.599 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.89296786800000005, 2024-09-13T11:24:52.599 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:52.599 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:52.599 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:52.599 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:52.599 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:52.599 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:52.599 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.057292+0000", 2024-09-13T11:24:52.600 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:52.600 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:52.600 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:52.600 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:52.600 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.057292+0000", 2024-09-13T11:24:52.600 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:52.600 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:52.600 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:52.600 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:52.600 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.057293+0000", 2024-09-13T11:24:52.601 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.6109999999999999e-06 2024-09-13T11:24:52.601 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:52.601 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:52.601 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:52.601 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.057330+0000", 2024-09-13T11:24:52.601 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.6721999999999998e-05 2024-09-13T11:24:52.601 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:52.601 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:52.601 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:52.601 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.948197+0000", 2024-09-13T11:24:52.601 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.890867042 2024-09-13T11:24:52.602 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:52.602 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:52.602 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:52.602 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.948213+0000", 2024-09-13T11:24:52.602 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5826000000000001e-05 2024-09-13T11:24:52.602 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:52.602 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:52.602 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:52.602 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.948377+0000", 2024-09-13T11:24:52.602 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00016348600000000001 2024-09-13T11:24:52.602 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:52.602 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:52.603 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:24:52.603 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.948447+0000", 2024-09-13T11:24:52.603 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.0891999999999998e-05 2024-09-13T11:24:52.603 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:52.603 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:52.603 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:24:52.603 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.950150+0000", 2024-09-13T11:24:52.603 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001702329 2024-09-13T11:24:52.603 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:52.603 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:52.603 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:24:52.603 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.950260+0000", 2024-09-13T11:24:52.603 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00010996 2024-09-13T11:24:52.603 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:52.604 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:52.604 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:52.604 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:52.604 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:52.604 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:30828 3.f 3:f3a9c6a9:::benchmark_data_smithi177_31116_object30827:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e59)", 2024-09-13T11:24:52.604 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:09.503661+0000", 2024-09-13T11:24:52.604 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 103.09105361, 2024-09-13T11:24:52.604 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.26477667100000002, 2024-09-13T11:24:52.604 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:52.604 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:52.604 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:52.604 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:24:52.605 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:24:52.605 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:52.605 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 30828 2024-09-13T11:24:52.605 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:52.605 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:52.605 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:52.605 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:52.605 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.503661+0000", 2024-09-13T11:24:52.605 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:52.605 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:52.606 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:52.606 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:52.606 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.503661+0000", 2024-09-13T11:24:52.606 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:52.606 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:52.606 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:52.606 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:52.607 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.503663+0000", 2024-09-13T11:24:52.607 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.968e-06 2024-09-13T11:24:52.607 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:52.607 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:52.607 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:52.607 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.503687+0000", 2024-09-13T11:24:52.607 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.3983999999999999e-05 2024-09-13T11:24:52.607 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:52.607 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:52.607 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:52.607 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.503701+0000", 2024-09-13T11:24:52.607 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3416e-05 2024-09-13T11:24:52.607 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:52.608 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:52.608 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:52.608 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.503708+0000", 2024-09-13T11:24:52.608 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.9530000000000002e-06 2024-09-13T11:24:52.608 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:52.608 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:52.608 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:52.608 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.503759+0000", 2024-09-13T11:24:52.608 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.1075999999999997e-05 2024-09-13T11:24:52.608 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:52.608 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:52.608 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:24:52.608 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.503861+0000", 2024-09-13T11:24:52.609 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000102372 2024-09-13T11:24:52.609 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:52.609 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:52.609 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 12", 2024-09-13T11:24:52.609 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.503952+0000", 2024-09-13T11:24:52.609 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.1135000000000001e-05 2024-09-13T11:24:52.609 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:52.609 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:52.609 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:24:52.609 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.505340+0000", 2024-09-13T11:24:52.609 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0013877169999999999 2024-09-13T11:24:52.609 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:52.609 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:52.610 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:52.610 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.768281+0000", 2024-09-13T11:24:52.610 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.26294092400000002 2024-09-13T11:24:52.610 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:52.610 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:52.610 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:24:52.610 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.768369+0000", 2024-09-13T11:24:52.610 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.7818999999999994e-05 2024-09-13T11:24:52.610 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:52.610 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:52.610 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:24:52.610 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.768438+0000", 2024-09-13T11:24:52.611 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.9307000000000006e-05 2024-09-13T11:24:52.611 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:52.611 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:52.611 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:52.611 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:52.611 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:52.611 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:45783 3.16 3.17c10a76 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e69)", 2024-09-13T11:24:52.611 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:20.947774+0000", 2024-09-13T11:24:52.611 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 91.646940708000002, 2024-09-13T11:24:52.611 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.96682922400000004, 2024-09-13T11:24:52.611 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:52.611 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:52.611 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-13T11:24:52.611 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:24:52.612 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:24:52.612 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:52.612 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 45783 2024-09-13T11:24:52.612 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:52.612 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:52.612 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:52.612 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:52.612 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:20.947774+0000", 2024-09-13T11:24:52.612 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:52.612 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:52.612 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:52.613 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:52.613 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:20.947774+0000", 2024-09-13T11:24:52.613 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:52.613 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:52.613 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:52.613 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:52.613 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:20.947777+0000", 2024-09-13T11:24:52.613 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.57e-06 2024-09-13T11:24:52.613 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:52.613 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:52.613 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:52.613 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:20.947802+0000", 2024-09-13T11:24:52.614 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.5528999999999999e-05 2024-09-13T11:24:52.614 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:52.614 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:52.614 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:52.614 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:20.947813+0000", 2024-09-13T11:24:52.614 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0473e-05 2024-09-13T11:24:52.614 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:52.614 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:52.614 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:52.614 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:20.947821+0000", 2024-09-13T11:24:52.614 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.3440000000000001e-06 2024-09-13T11:24:52.614 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:52.614 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:52.615 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:52.615 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:20.947933+0000", 2024-09-13T11:24:52.615 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00011210200000000001 2024-09-13T11:24:52.615 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:52.615 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:52.615 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for peered", 2024-09-13T11:24:52.615 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:20.947942+0000", 2024-09-13T11:24:52.615 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.6610000000000006e-06 2024-09-13T11:24:52.615 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:52.615 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:52.615 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:52.615 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:21.914579+0000", 2024-09-13T11:24:52.615 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.96663672300000003 2024-09-13T11:24:52.615 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:52.616 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:52.616 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:24:52.616 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:21.914603+0000", 2024-09-13T11:24:52.616 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.4822000000000001e-05 2024-09-13T11:24:52.616 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:52.616 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:52.616 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:52.616 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:52.616 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:52.616 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:54625 3.1f 3:f812cdfb:::benchmark_data_smithi177_31116_object54624:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e75)", 2024-09-13T11:24:52.616 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:30.979327+0000", 2024-09-13T11:24:52.616 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 81.615387919, 2024-09-13T11:24:52.616 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.96163670899999998, 2024-09-13T11:24:52.617 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:52.617 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:52.617 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-13T11:24:52.617 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:24:52.617 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:24:52.617 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:52.617 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 54625 2024-09-13T11:24:52.617 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:52.617 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:52.617 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:52.617 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:52.617 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.979327+0000", 2024-09-13T11:24:52.617 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:52.617 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:52.618 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:52.618 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:52.618 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.979327+0000", 2024-09-13T11:24:52.618 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:52.618 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:52.618 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:52.618 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:52.618 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.979329+0000", 2024-09-13T11:24:52.618 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.686e-06 2024-09-13T11:24:52.618 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:52.618 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:52.618 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:52.618 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.979364+0000", 2024-09-13T11:24:52.619 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.5750000000000002e-05 2024-09-13T11:24:52.619 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:52.619 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:52.619 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:52.619 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.979408+0000", 2024-09-13T11:24:52.619 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.3795999999999999e-05 2024-09-13T11:24:52.619 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:52.619 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:52.619 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:52.619 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.979427+0000", 2024-09-13T11:24:52.619 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.9040999999999999e-05 2024-09-13T11:24:52.619 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:52.619 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:52.619 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:52.620 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.122291+0000", 2024-09-13T11:24:52.620 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.14286373599999999 2024-09-13T11:24:52.620 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:52.620 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:52.620 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for peered", 2024-09-13T11:24:52.620 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.122298+0000", 2024-09-13T11:24:52.620 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.3719999999999999e-06 2024-09-13T11:24:52.620 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:52.620 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:52.620 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:52.620 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.940341+0000", 2024-09-13T11:24:52.620 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.81804249699999998 2024-09-13T11:24:52.620 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:52.620 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:52.621 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:24:52.621 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.940964+0000", 2024-09-13T11:24:52.621 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00062283100000000001 2024-09-13T11:24:52.621 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:52.621 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:52.621 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:52.621 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:52.621 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:52.621 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:54649 3.1c 3:3a457a93:::benchmark_data_smithi177_31116_object54648:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e75)", 2024-09-13T11:24:52.621 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:30.979460+0000", 2024-09-13T11:24:52.622 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 81.615254729, 2024-09-13T11:24:52.622 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.95579424000000002, 2024-09-13T11:24:52.622 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:52.622 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:52.622 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-13T11:24:52.622 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:24:52.622 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:24:52.622 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:52.622 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 54649 2024-09-13T11:24:52.622 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:52.622 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:52.622 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:52.622 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:52.622 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.979460+0000", 2024-09-13T11:24:52.623 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:52.623 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:52.623 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:52.623 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:52.623 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.979460+0000", 2024-09-13T11:24:52.623 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:52.623 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:52.623 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:52.623 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:52.623 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.979462+0000", 2024-09-13T11:24:52.623 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5799999999999999e-06 2024-09-13T11:24:52.623 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:52.624 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:52.624 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:52.624 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.979524+0000", 2024-09-13T11:24:52.624 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.2521999999999995e-05 2024-09-13T11:24:52.624 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:52.624 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:52.624 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:52.624 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.979561+0000", 2024-09-13T11:24:52.624 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.6349e-05 2024-09-13T11:24:52.625 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:52.625 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:52.625 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:52.625 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.979571+0000", 2024-09-13T11:24:52.625 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0457e-05 2024-09-13T11:24:52.625 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:52.625 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:52.625 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:52.626 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.121029+0000", 2024-09-13T11:24:52.626 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.14145766200000001 2024-09-13T11:24:52.626 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:52.626 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:52.626 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for peered", 2024-09-13T11:24:52.626 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.121039+0000", 2024-09-13T11:24:52.626 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0084e-05 2024-09-13T11:24:52.626 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:52.626 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:52.626 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:52.627 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.935157+0000", 2024-09-13T11:24:52.627 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.81411807899999999 2024-09-13T11:24:52.627 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:52.627 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:52.627 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:24:52.627 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.935254+0000", 2024-09-13T11:24:52.627 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.7507e-05 2024-09-13T11:24:52.627 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:52.627 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:52.627 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:52.627 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:52.627 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:52.627 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:54654 3.1c 3:3f0bb31a:::benchmark_data_smithi177_31116_object54653:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e75)", 2024-09-13T11:24:52.628 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:30.979665+0000", 2024-09-13T11:24:52.628 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 81.615049564000003, 2024-09-13T11:24:52.628 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.95623908099999999, 2024-09-13T11:24:52.628 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:52.628 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:52.628 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-13T11:24:52.629 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:24:52.629 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:24:52.629 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:52.629 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 54654 2024-09-13T11:24:52.629 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:52.629 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:52.630 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:52.630 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:52.630 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.979665+0000", 2024-09-13T11:24:52.630 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:52.630 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:52.630 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:52.630 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:52.631 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.979665+0000", 2024-09-13T11:24:52.631 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:52.631 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:52.631 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:52.631 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:52.631 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.979667+0000", 2024-09-13T11:24:52.632 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3540000000000001e-06 2024-09-13T11:24:52.632 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:52.632 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:52.632 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:52.632 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.979703+0000", 2024-09-13T11:24:52.632 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.6758000000000003e-05 2024-09-13T11:24:52.632 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:52.632 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:52.633 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:52.633 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.979714+0000", 2024-09-13T11:24:52.633 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0804e-05 2024-09-13T11:24:52.633 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:52.633 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:52.633 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:52.633 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.979724+0000", 2024-09-13T11:24:52.634 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0040999999999999e-05 2024-09-13T11:24:52.634 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:52.634 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:52.634 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:52.634 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.121061+0000", 2024-09-13T11:24:52.634 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.141336713 2024-09-13T11:24:52.635 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:52.635 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:52.635 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for peered", 2024-09-13T11:24:52.635 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.121067+0000", 2024-09-13T11:24:52.635 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.4150000000000004e-06 2024-09-13T11:24:52.635 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:52.636 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:52.636 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:52.636 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.935776+0000", 2024-09-13T11:24:52.636 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.81470863100000002 2024-09-13T11:24:52.636 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:52.636 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:52.637 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:24:52.637 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.935904+0000", 2024-09-13T11:24:52.637 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00012836499999999999 2024-09-13T11:24:52.637 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:52.637 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:52.637 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:52.637 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:52.637 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:52.637 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPushReply(3.11 75/70 [PushReplyOp(3:8df10771:::benchmark_data_smithi177_31116_object24734:head)])", 2024-09-13T11:24:52.637 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:31.149798+0000", 2024-09-13T11:24:52.637 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 81.444916554000002, 2024-09-13T11:24:52.638 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.32402702500000002, 2024-09-13T11:24:52.638 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:52.638 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:52.638 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-13T11:24:52.638 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:52.638 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:52.638 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:52.638 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.149798+0000", 2024-09-13T11:24:52.638 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:52.638 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:52.638 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:52.638 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:52.638 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.149798+0000", 2024-09-13T11:24:52.638 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:52.639 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:52.639 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:52.639 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:52.639 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.149799+0000", 2024-09-13T11:24:52.639 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.5700000000000001e-07 2024-09-13T11:24:52.639 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:52.639 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:52.639 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:52.639 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.149803+0000", 2024-09-13T11:24:52.639 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.0040000000000001e-06 2024-09-13T11:24:52.639 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:52.640 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:52.640 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:52.640 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.472172+0000", 2024-09-13T11:24:52.640 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.32236856699999999 2024-09-13T11:24:52.640 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:52.640 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:52.640 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:52.640 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.472187+0000", 2024-09-13T11:24:52.640 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5016e-05 2024-09-13T11:24:52.640 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:52.640 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:52.640 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:52.641 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.472302+0000", 2024-09-13T11:24:52.641 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000115121 2024-09-13T11:24:52.641 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:52.641 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:52.641 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:24:52.641 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.473825+0000", 2024-09-13T11:24:52.641 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0015234599999999999 2024-09-13T11:24:52.641 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:52.641 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:52.641 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:52.641 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:52.641 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:52.641 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:54668 3.1c 3:3d2c5e40:::benchmark_data_smithi177_31116_object54667:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e75)", 2024-09-13T11:24:52.642 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:31.245525+0000", 2024-09-13T11:24:52.642 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 81.349190297000007, 2024-09-13T11:24:52.642 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.69828973000000005, 2024-09-13T11:24:52.642 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:52.642 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:52.642 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:52.642 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:24:52.642 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:24:52.642 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:52.642 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 54668 2024-09-13T11:24:52.642 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:52.642 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:52.642 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:52.642 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:52.643 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.245525+0000", 2024-09-13T11:24:52.643 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:52.643 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:52.643 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:52.643 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:52.643 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.245525+0000", 2024-09-13T11:24:52.643 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:52.643 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:52.643 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:52.643 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:52.643 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.245526+0000", 2024-09-13T11:24:52.643 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.719e-06 2024-09-13T11:24:52.643 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:52.644 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:52.644 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:52.644 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.245547+0000", 2024-09-13T11:24:52.644 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.1084999999999999e-05 2024-09-13T11:24:52.644 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:52.644 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:52.644 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:52.644 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.245558+0000", 2024-09-13T11:24:52.644 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0934000000000001e-05 2024-09-13T11:24:52.644 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:52.644 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:52.644 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:52.645 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.245564+0000", 2024-09-13T11:24:52.645 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.6269999999999997e-06 2024-09-13T11:24:52.645 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:52.645 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:52.645 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:52.645 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.245620+0000", 2024-09-13T11:24:52.645 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.6289999999999998e-05 2024-09-13T11:24:52.645 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:52.645 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:52.645 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for peered", 2024-09-13T11:24:52.645 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.245632+0000", 2024-09-13T11:24:52.645 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.1748e-05 2024-09-13T11:24:52.645 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:52.645 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:52.646 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:52.646 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.935976+0000", 2024-09-13T11:24:52.646 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.69034455299999997 2024-09-13T11:24:52.646 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:52.646 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:52.646 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:24:52.646 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.936302+0000", 2024-09-13T11:24:52.646 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000325397 2024-09-13T11:24:52.646 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:52.646 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:52.646 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 13", 2024-09-13T11:24:52.646 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.936459+0000", 2024-09-13T11:24:52.646 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00015754300000000001 2024-09-13T11:24:52.646 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:52.646 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:52.647 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:24:52.647 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.938813+0000", 2024-09-13T11:24:52.647 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0023530999999999999 2024-09-13T11:24:52.647 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:52.647 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:52.647 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:52.647 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.943695+0000", 2024-09-13T11:24:52.647 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.004882506 2024-09-13T11:24:52.647 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:52.647 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:52.647 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:24:52.647 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.943767+0000", 2024-09-13T11:24:52.647 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.2454999999999997e-05 2024-09-13T11:24:52.647 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:52.648 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:52.648 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:24:52.648 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.943814+0000", 2024-09-13T11:24:52.648 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.6773000000000001e-05 2024-09-13T11:24:52.648 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:52.648 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:52.648 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:52.648 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:52.648 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:52.648 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPush(3.16 84/70 [PushOp(3:6f7be541:::benchmark_data_smithi177_31116_object43332:head, version: 66'2788, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:6f7be541:::benchmark_data_smithi177_31116_object43332:head@66'2788, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:24:52.648 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:41.207515+0000", 2024-09-13T11:24:52.648 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 71.387199526000003, 2024-09-13T11:24:52.648 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.52643771500000003, 2024-09-13T11:24:52.649 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:52.649 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:52.649 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:24:52.649 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:52.649 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:52.649 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:52.649 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.207515+0000", 2024-09-13T11:24:52.649 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:52.649 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:52.649 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:52.649 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:52.649 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.207515+0000", 2024-09-13T11:24:52.649 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:52.650 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:52.650 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:52.650 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:52.650 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.207516+0000", 2024-09-13T11:24:52.650 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.096e-06 2024-09-13T11:24:52.650 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:52.650 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:52.650 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:52.650 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.207717+0000", 2024-09-13T11:24:52.650 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000200199 2024-09-13T11:24:52.650 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:52.650 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:52.651 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:52.651 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.732785+0000", 2024-09-13T11:24:52.651 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.52506869599999995 2024-09-13T11:24:52.651 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:52.651 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:52.651 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:52.651 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.732801+0000", 2024-09-13T11:24:52.651 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5305999999999999e-05 2024-09-13T11:24:52.651 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:52.651 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:52.651 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:52.652 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.732889+0000", 2024-09-13T11:24:52.652 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.8157000000000003e-05 2024-09-13T11:24:52.652 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:52.652 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:52.652 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:24:52.652 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.732901+0000", 2024-09-13T11:24:52.652 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.1866e-05 2024-09-13T11:24:52.652 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:52.652 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:52.652 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:24:52.652 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.733953+0000", 2024-09-13T11:24:52.652 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001052395 2024-09-13T11:24:52.652 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:52.653 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:52.653 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:52.653 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:52.653 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:52.653 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPushReply(3.1f 85/78 [PushReplyOp(3:fd221d21:::benchmark_data_smithi177_31116_object7639:head)])", 2024-09-13T11:24:52.653 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:42.198715+0000", 2024-09-13T11:24:52.653 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 70.395999957000001, 2024-09-13T11:24:52.653 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.48930391099999998, 2024-09-13T11:24:52.653 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:52.653 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:52.653 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-13T11:24:52.653 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:52.654 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:52.654 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:52.654 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.198715+0000", 2024-09-13T11:24:52.654 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:52.654 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:52.654 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:52.654 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:52.654 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.198715+0000", 2024-09-13T11:24:52.654 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:52.654 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:52.654 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:52.655 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:52.655 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.198716+0000", 2024-09-13T11:24:52.655 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.3500000000000005e-07 2024-09-13T11:24:52.655 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:52.655 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:52.655 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:52.655 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.198719+0000", 2024-09-13T11:24:52.655 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.4539999999999999e-06 2024-09-13T11:24:52.655 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:52.655 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:52.655 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:52.655 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.684732+0000", 2024-09-13T11:24:52.656 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.48601262099999998 2024-09-13T11:24:52.656 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:52.656 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:52.656 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:52.656 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.684746+0000", 2024-09-13T11:24:52.656 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.4518999999999999e-05 2024-09-13T11:24:52.656 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:52.656 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:52.656 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:52.656 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.684911+0000", 2024-09-13T11:24:52.656 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000164728 2024-09-13T11:24:52.657 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:52.657 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:52.657 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:24:52.657 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.688019+0000", 2024-09-13T11:24:52.657 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0031077539999999999 2024-09-13T11:24:52.657 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:52.657 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:52.657 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:52.657 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:52.657 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:52.657 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPushReply(3.1c 85/77 [PushReplyOp(3:3a669437:::benchmark_data_smithi177_31116_object25280:head)])", 2024-09-13T11:24:52.657 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:42.352466+0000", 2024-09-13T11:24:52.657 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 70.242249063000003, 2024-09-13T11:24:52.658 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.36670725300000001, 2024-09-13T11:24:52.658 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:52.658 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:52.658 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-13T11:24:52.658 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:52.658 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:52.658 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:52.658 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.352466+0000", 2024-09-13T11:24:52.658 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:52.658 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:52.658 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:52.658 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:52.658 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.352466+0000", 2024-09-13T11:24:52.659 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:52.659 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:52.659 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:52.659 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:52.659 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.352467+0000", 2024-09-13T11:24:52.659 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.1429999999999999e-06 2024-09-13T11:24:52.659 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:52.659 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:52.659 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:52.659 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.352474+0000", 2024-09-13T11:24:52.659 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.6100000000000002e-06 2024-09-13T11:24:52.659 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:52.659 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:52.660 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:52.660 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.717011+0000", 2024-09-13T11:24:52.660 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.36453790200000002 2024-09-13T11:24:52.660 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:52.660 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:52.660 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:52.660 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.717032+0000", 2024-09-13T11:24:52.660 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.0999e-05 2024-09-13T11:24:52.660 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:52.660 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:52.660 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:52.660 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.717162+0000", 2024-09-13T11:24:52.660 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000129159 2024-09-13T11:24:52.661 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:52.661 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:52.661 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:24:52.661 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.719173+0000", 2024-09-13T11:24:52.661 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0020114400000000002 2024-09-13T11:24:52.661 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:52.661 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:52.661 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:52.661 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:52.661 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:52.661 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPush(3.16 85/70 [PushOp(3:6aef2f27:::benchmark_data_smithi177_31116_object44090:head, version: 68'2844, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:6aef2f27:::benchmark_data_smithi177_31116_object44090:head@68'2844, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:24:52.661 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:42.574221+0000", 2024-09-13T11:24:52.661 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 70.020493955000006, 2024-09-13T11:24:52.661 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.25345585700000001, 2024-09-13T11:24:52.662 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:52.662 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:52.662 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:24:52.662 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:52.662 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:52.662 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:52.662 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.574221+0000", 2024-09-13T11:24:52.662 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:52.662 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:52.662 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:52.662 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:52.662 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.574221+0000", 2024-09-13T11:24:52.662 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:52.662 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:52.662 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:52.663 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:52.663 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.574252+0000", 2024-09-13T11:24:52.663 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.0973000000000003e-05 2024-09-13T11:24:52.663 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:52.663 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:52.663 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:52.663 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.574434+0000", 2024-09-13T11:24:52.663 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000182265 2024-09-13T11:24:52.663 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:52.663 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:52.663 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:52.664 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.825391+0000", 2024-09-13T11:24:52.664 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.250957023 2024-09-13T11:24:52.664 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:52.664 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:52.664 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:52.664 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.825442+0000", 2024-09-13T11:24:52.665 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.1267999999999998e-05 2024-09-13T11:24:52.665 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:52.665 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:52.665 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:52.665 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.825692+0000", 2024-09-13T11:24:52.665 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00024924700000000002 2024-09-13T11:24:52.665 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:52.665 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:52.665 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:24:52.665 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.825715+0000", 2024-09-13T11:24:52.665 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.3176e-05 2024-09-13T11:24:52.666 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:52.666 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:52.666 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:24:52.666 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.827677+0000", 2024-09-13T11:24:52.666 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0019619049999999999 2024-09-13T11:24:52.666 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:52.666 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:52.666 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:52.666 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:52.666 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:52.666 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPushReply(3.1f 85/78 [PushReplyOp(3:fecb0d6a:::benchmark_data_smithi177_31116_object7644:head)])", 2024-09-13T11:24:52.667 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:42.705351+0000", 2024-09-13T11:24:52.667 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 69.889364220999994, 2024-09-13T11:24:52.667 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.290549681, 2024-09-13T11:24:52.667 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:52.667 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:52.667 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-13T11:24:52.667 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:52.667 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:52.667 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:52.667 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.705351+0000", 2024-09-13T11:24:52.667 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:52.667 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:52.667 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:52.667 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:52.668 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.705351+0000", 2024-09-13T11:24:52.668 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:52.668 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:52.668 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:52.668 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:52.668 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.705353+0000", 2024-09-13T11:24:52.668 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.4169999999999999e-06 2024-09-13T11:24:52.668 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:52.668 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:52.668 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:52.668 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.705365+0000", 2024-09-13T11:24:52.668 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.1894e-05 2024-09-13T11:24:52.668 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:52.669 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:52.669 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:52.669 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.992081+0000", 2024-09-13T11:24:52.669 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.28671641199999998 2024-09-13T11:24:52.669 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:52.669 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:52.669 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:52.669 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.992111+0000", 2024-09-13T11:24:52.669 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.9427000000000001e-05 2024-09-13T11:24:52.669 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:52.669 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:52.670 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:52.670 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.993573+0000", 2024-09-13T11:24:52.670 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0014623590000000001 2024-09-13T11:24:52.670 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:52.670 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:52.670 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:24:52.670 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.995900+0000", 2024-09-13T11:24:52.670 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0023271720000000002 2024-09-13T11:24:52.670 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:52.670 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:52.670 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:52.670 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:52.670 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:52.670 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPushReply(3.21 89/78 [PushReplyOp(3:8534d087:::benchmark_data_smithi177_31116_object5474:head)])", 2024-09-13T11:24:52.671 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:46.632834+0000", 2024-09-13T11:24:52.671 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 65.961880781000005, 2024-09-13T11:24:52.671 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.324805556, 2024-09-13T11:24:52.671 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:52.671 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:52.671 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-13T11:24:52.671 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:52.671 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:52.671 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:52.671 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.632834+0000", 2024-09-13T11:24:52.671 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:52.671 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:52.672 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:52.672 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:52.672 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.632834+0000", 2024-09-13T11:24:52.672 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:52.672 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:52.672 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:52.672 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:52.672 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.632835+0000", 2024-09-13T11:24:52.673 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.4270000000000001e-06 2024-09-13T11:24:52.673 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:52.673 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:52.673 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:52.673 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.632840+0000", 2024-09-13T11:24:52.673 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.7809999999999996e-06 2024-09-13T11:24:52.673 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:52.673 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:52.673 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:52.673 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.956282+0000", 2024-09-13T11:24:52.673 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.323441639 2024-09-13T11:24:52.674 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:52.674 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:52.674 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:52.674 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.956293+0000", 2024-09-13T11:24:52.674 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.083e-05 2024-09-13T11:24:52.674 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:52.674 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:52.674 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:52.674 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.956395+0000", 2024-09-13T11:24:52.674 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00010184200000000001 2024-09-13T11:24:52.674 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:52.674 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:52.675 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:24:52.675 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.957640+0000", 2024-09-13T11:24:52.675 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0012450370000000001 2024-09-13T11:24:52.675 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:52.675 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:52.675 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:52.675 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:52.675 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:52.675 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPushReply(3.1f 89/78 [PushReplyOp(3:ff150976:::benchmark_data_smithi177_31116_object12907:head)])", 2024-09-13T11:24:52.675 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:46.697504+0000", 2024-09-13T11:24:52.675 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 65.897211182000007, 2024-09-13T11:24:52.675 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.26836818099999998, 2024-09-13T11:24:52.675 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:52.675 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:52.676 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-13T11:24:52.676 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:52.676 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:52.676 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:52.676 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.697504+0000", 2024-09-13T11:24:52.676 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:52.676 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:52.676 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:52.677 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:52.677 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.697504+0000", 2024-09-13T11:24:52.677 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:52.677 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:52.677 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:52.677 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:52.677 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.697505+0000", 2024-09-13T11:24:52.678 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0529999999999999e-06 2024-09-13T11:24:52.678 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:52.678 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:52.678 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:52.678 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.697509+0000", 2024-09-13T11:24:52.678 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.9249999999999997e-06 2024-09-13T11:24:52.678 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:52.678 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:52.678 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:52.678 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.964086+0000", 2024-09-13T11:24:52.678 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.26657776100000002 2024-09-13T11:24:52.678 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:52.679 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:52.679 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:52.679 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.964110+0000", 2024-09-13T11:24:52.679 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.3189000000000001e-05 2024-09-13T11:24:52.679 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:52.679 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:52.679 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:52.679 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.964354+0000", 2024-09-13T11:24:52.679 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00024481399999999999 2024-09-13T11:24:52.679 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:52.679 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:52.680 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:24:52.680 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.965872+0000", 2024-09-13T11:24:52.680 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001517439 2024-09-13T11:24:52.680 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:52.680 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:52.680 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:52.680 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:52.680 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:52.680 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:83223 3.1c 3:3bc6018d:::benchmark_data_smithi177_31116_object83222:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106)", 2024-09-13T11:24:52.680 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:04.511163+0000", 2024-09-13T11:24:52.680 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 48.083551403999998, 2024-09-13T11:24:52.680 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.47558093200000001, 2024-09-13T11:24:52.681 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:52.681 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:52.681 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:52.681 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:24:52.681 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:24:52.681 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:52.681 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 83223 2024-09-13T11:24:52.681 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:52.681 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:52.681 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:52.681 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:52.682 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.511163+0000", 2024-09-13T11:24:52.682 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:52.682 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:52.682 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:52.682 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:52.682 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.511163+0000", 2024-09-13T11:24:52.682 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:52.682 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:52.682 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:52.682 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:52.682 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.511167+0000", 2024-09-13T11:24:52.682 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.8330000000000004e-06 2024-09-13T11:24:52.682 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:52.682 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:52.683 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:52.683 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.511219+0000", 2024-09-13T11:24:52.683 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.1771999999999998e-05 2024-09-13T11:24:52.683 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:52.683 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:52.683 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:52.683 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.511252+0000", 2024-09-13T11:24:52.683 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.3315999999999997e-05 2024-09-13T11:24:52.683 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:52.683 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:52.683 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:52.683 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.511268+0000", 2024-09-13T11:24:52.683 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5812999999999999e-05 2024-09-13T11:24:52.684 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:52.684 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:52.684 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:52.684 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.511418+0000", 2024-09-13T11:24:52.684 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000150023 2024-09-13T11:24:52.684 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:52.684 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:52.684 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:24:52.684 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.511670+0000", 2024-09-13T11:24:52.685 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00025185499999999999 2024-09-13T11:24:52.685 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:52.685 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:52.685 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 7", 2024-09-13T11:24:52.685 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.511905+0000", 2024-09-13T11:24:52.685 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00023493799999999999 2024-09-13T11:24:52.685 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:52.685 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:52.685 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:24:52.685 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.514960+0000", 2024-09-13T11:24:52.685 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.003054977 2024-09-13T11:24:52.685 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:52.686 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:52.686 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:52.686 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.986432+0000", 2024-09-13T11:24:52.686 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.47147209200000001 2024-09-13T11:24:52.686 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:52.686 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:52.686 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:24:52.686 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.986635+0000", 2024-09-13T11:24:52.686 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00020309899999999999 2024-09-13T11:24:52.686 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:52.686 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:52.687 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:24:52.687 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.986744+0000", 2024-09-13T11:24:52.687 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000109214 2024-09-13T11:24:52.687 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:52.687 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:52.687 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:52.687 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:52.688 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:52.688 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop_reply(client.4665.0:84803 3.1f e107/78)", 2024-09-13T11:24:52.688 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:05.912627+0000", 2024-09-13T11:24:52.688 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 46.682087496999998, 2024-09-13T11:24:52.688 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.33726566600000002, 2024-09-13T11:24:52.688 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:52.689 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:52.689 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-13T11:24:52.689 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:52.689 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:52.689 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:52.689 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:05.912627+0000", 2024-09-13T11:24:52.689 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:52.689 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:52.689 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:52.690 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:52.690 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:05.912627+0000", 2024-09-13T11:24:52.690 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:52.690 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:52.690 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:52.690 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:52.691 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:05.912629+0000", 2024-09-13T11:24:52.691 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.4440000000000001e-06 2024-09-13T11:24:52.691 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:52.691 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:52.691 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:52.691 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:05.912634+0000", 2024-09-13T11:24:52.691 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.6869999999999999e-06 2024-09-13T11:24:52.691 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:52.692 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:52.692 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:52.692 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:06.249722+0000", 2024-09-13T11:24:52.692 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.33708766099999998 2024-09-13T11:24:52.692 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:52.693 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:52.693 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:52.693 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:06.249736+0000", 2024-09-13T11:24:52.693 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3623e-05 2024-09-13T11:24:52.693 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:52.694 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:52.694 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:52.694 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:06.249878+0000", 2024-09-13T11:24:52.694 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00014264599999999999 2024-09-13T11:24:52.694 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:52.694 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:52.694 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:24:52.694 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:06.249893+0000", 2024-09-13T11:24:52.694 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.4605e-05 2024-09-13T11:24:52.695 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:52.695 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:52.695 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:52.695 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:52.695 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:52.695 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop_reply(client.4665.0:110260 3.f e119/75 ondisk, result = 0)", 2024-09-13T11:24:52.695 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:25.333188+0000", 2024-09-13T11:24:52.696 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 27.261526870000001, 2024-09-13T11:24:52.696 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.29195486199999998, 2024-09-13T11:24:52.696 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:52.696 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:52.696 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:24:52.696 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:52.696 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:52.697 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:52.697 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.333188+0000", 2024-09-13T11:24:52.697 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:52.697 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:52.697 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:52.697 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:52.697 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.333188+0000", 2024-09-13T11:24:52.697 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:52.698 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:52.698 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:52.698 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:52.698 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.333190+0000", 2024-09-13T11:24:52.698 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.2120000000000002e-06 2024-09-13T11:24:52.698 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:52.699 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:52.699 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:52.699 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.333197+0000", 2024-09-13T11:24:52.699 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.7469999999999999e-06 2024-09-13T11:24:52.699 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:52.699 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:52.700 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:52.700 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.624892+0000", 2024-09-13T11:24:52.700 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.291694913 2024-09-13T11:24:52.700 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:52.700 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:52.700 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:52.700 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.624907+0000", 2024-09-13T11:24:52.700 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5053e-05 2024-09-13T11:24:52.700 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:52.700 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:52.700 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:52.700 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.624999+0000", 2024-09-13T11:24:52.701 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.2603999999999996e-05 2024-09-13T11:24:52.701 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:52.701 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:52.701 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:24:52.701 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.625011+0000", 2024-09-13T11:24:52.701 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.1253999999999999e-05 2024-09-13T11:24:52.701 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:52.701 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:52.701 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:24:52.701 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.625143+0000", 2024-09-13T11:24:52.701 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00013207900000000001 2024-09-13T11:24:52.702 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:52.702 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:52.702 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:52.702 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:52.702 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:52.702 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:110265 3.1 e119/75 3:8236484d:::benchmark_data_smithi177_31116_object110264:head v 119'3935, mlcod=118'3933)", 2024-09-13T11:24:52.702 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:25.343348+0000", 2024-09-13T11:24:52.702 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 27.251367255000002, 2024-09-13T11:24:52.702 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.60013013500000001, 2024-09-13T11:24:52.702 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:52.702 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:52.702 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:52.702 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:52.703 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:52.703 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:52.703 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.343348+0000", 2024-09-13T11:24:52.703 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:52.703 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:52.703 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:52.703 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:52.703 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.343348+0000", 2024-09-13T11:24:52.703 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:52.703 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:52.703 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:52.703 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:52.703 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.343350+0000", 2024-09-13T11:24:52.703 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.576e-06 2024-09-13T11:24:52.704 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:52.704 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:52.704 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:52.704 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.343443+0000", 2024-09-13T11:24:52.704 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.2989e-05 2024-09-13T11:24:52.704 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:52.704 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:52.704 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:52.704 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.940863+0000", 2024-09-13T11:24:52.704 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.59742003300000002 2024-09-13T11:24:52.704 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:52.704 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:52.704 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:52.705 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.940875+0000", 2024-09-13T11:24:52.705 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.2147000000000001e-05 2024-09-13T11:24:52.705 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:52.705 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:52.705 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:52.705 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.940993+0000", 2024-09-13T11:24:52.705 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000117903 2024-09-13T11:24:52.705 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:52.705 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:52.705 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:24:52.705 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.941017+0000", 2024-09-13T11:24:52.706 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.3442999999999999e-05 2024-09-13T11:24:52.706 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:52.706 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:52.706 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:24:52.706 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.943347+0000", 2024-09-13T11:24:52.706 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0023299869999999999 2024-09-13T11:24:52.706 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:52.706 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:52.706 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:24:52.706 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.943478+0000", 2024-09-13T11:24:52.706 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00013105699999999999 2024-09-13T11:24:52.706 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:52.707 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:52.707 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:52.707 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:52.707 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:52.707 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop_reply(client.4665.0:129919 3.16 e134/88 ondisk, result = 0)", 2024-09-13T11:24:52.707 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:47.175742+0000", 2024-09-13T11:24:52.707 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 5.4189731850000005, 2024-09-13T11:24:52.707 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.27593726299999999, 2024-09-13T11:24:52.707 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:52.707 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:52.708 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:24:52.708 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:52.708 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:52.708 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:52.708 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:47.175742+0000", 2024-09-13T11:24:52.708 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:52.708 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:52.708 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:52.708 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:52.708 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:47.175742+0000", 2024-09-13T11:24:52.708 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:52.708 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:52.708 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:52.709 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:52.709 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:47.175744+0000", 2024-09-13T11:24:52.709 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.0030000000000001e-06 2024-09-13T11:24:52.709 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:52.709 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:52.709 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:52.709 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:47.175751+0000", 2024-09-13T11:24:52.709 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.9779999999999999e-06 2024-09-13T11:24:52.709 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:52.709 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:52.709 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:52.709 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:47.451490+0000", 2024-09-13T11:24:52.709 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.27573892500000002 2024-09-13T11:24:52.709 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:52.710 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:52.710 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:52.710 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:47.451496+0000", 2024-09-13T11:24:52.710 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.0079999999999999e-06 2024-09-13T11:24:52.710 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:52.710 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:52.710 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:52.711 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:47.451560+0000", 2024-09-13T11:24:52.711 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.3955999999999995e-05 2024-09-13T11:24:52.711 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:52.711 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:52.711 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:24:52.712 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:47.451567+0000", 2024-09-13T11:24:52.712 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.1980000000000004e-06 2024-09-13T11:24:52.712 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:52.712 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:52.712 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:24:52.712 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:47.451679+0000", 2024-09-13T11:24:52.713 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00011219500000000001 2024-09-13T11:24:52.713 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:52.713 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:52.713 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:52.713 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:52.713 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:52.714 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-09-13T11:24:52.715 INFO:teuthology.orchestra.run.smithi001.stdout:osd.12: {} 2024-09-13T11:24:52.716 INFO:teuthology.orchestra.run.smithi001.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-13T11:24:52.717 INFO:teuthology.orchestra.run.smithi001.stdout:osd.13: {} 2024-09-13T11:24:52.718 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_ops_in_flight 2024-09-13T11:24:52.719 INFO:tasks.ceph.osd.15.smithi177.stderr:2024-09-13T11:24:52.710+0000 7f9d6ce77640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26578) UID: 0 2024-09-13T11:24:52.719 INFO:teuthology.orchestra.run.smithi001.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-13T11:24:52.719 INFO:teuthology.orchestra.run.smithi001.stderr:Error ENXIO: problem getting command descriptions from osd.14 2024-09-13T11:24:52.720 INFO:teuthology.orchestra.run.smithi001.stdout:osd.15: {} 2024-09-13T11:24:52.720 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-13T11:24:52.721 INFO:teuthology.orchestra.run.smithi001.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-13T11:24:52.721 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-13T11:24:52.816 INFO:tasks.ceph.osd.13.smithi092.stderr:2024-09-13T11:24:52.810+0000 7f3e5afea640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26350) UID: 0 2024-09-13T11:24:52.916 INFO:tasks.ceph.osd.11.smithi177.stderr:2024-09-13T11:24:52.914+0000 7f24792c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26580) UID: 0 2024-09-13T11:24:52.922 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-09-13T11:24:52.922 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [ 2024-09-13T11:24:52.922 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:52.922 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:136770 3.14 e138/122 3:2f433f31:::benchmark_data_smithi177_31116_object136769:head v 138'5729, mlcod=72'2898)", 2024-09-13T11:24:52.922 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:52.917707+0000", 2024-09-13T11:24:52.922 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 0.0042498429999999997, 2024-09-13T11:24:52.922 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0042734069999999999, 2024-09-13T11:24:52.922 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:52.923 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:52.923 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:24:52.923 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:52.923 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:52.923 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:52.923 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:52.917707+0000", 2024-09-13T11:24:52.923 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:52.923 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:52.923 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:52.923 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:52.923 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:52.917707+0000", 2024-09-13T11:24:52.923 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:52.923 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:52.923 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:52.924 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:52.924 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:52.917709+0000", 2024-09-13T11:24:52.924 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.279e-06 2024-09-13T11:24:52.924 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:52.924 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:52.924 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:52.924 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:52.917739+0000", 2024-09-13T11:24:52.924 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.0715e-05 2024-09-13T11:24:52.924 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:52.924 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:52.924 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:52.924 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:52.917767+0000", 2024-09-13T11:24:52.924 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.8109000000000001e-05 2024-09-13T11:24:52.925 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:52.925 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:52.925 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:52.925 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:52.917776+0000", 2024-09-13T11:24:52.925 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.2630000000000003e-06 2024-09-13T11:24:52.925 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:52.925 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:52.925 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:52.925 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:52.917834+0000", 2024-09-13T11:24:52.925 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.8746000000000003e-05 2024-09-13T11:24:52.925 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:52.925 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:52.925 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:24:52.925 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:52.917857+0000", 2024-09-13T11:24:52.926 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.2554000000000001e-05 2024-09-13T11:24:52.926 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:52.926 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:52.926 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:52.926 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:52.926 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:52.926 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:136780 3.1d e138/102 3:b9d22177:::benchmark_data_smithi177_31116_object136779:head v 138'5922, mlcod=138'5920)", 2024-09-13T11:24:52.926 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:52.918596+0000", 2024-09-13T11:24:52.926 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 0.003361012, 2024-09-13T11:24:52.926 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0034411189999999999, 2024-09-13T11:24:52.926 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:52.926 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:52.926 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:24:52.927 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:52.927 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:52.927 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:52.927 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:52.918596+0000", 2024-09-13T11:24:52.927 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:52.927 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:52.927 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:52.927 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:52.927 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:52.918596+0000", 2024-09-13T11:24:52.927 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:52.927 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:52.927 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:52.927 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:52.927 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:52.918597+0000", 2024-09-13T11:24:52.928 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.2899999999999999e-06 2024-09-13T11:24:52.928 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:52.928 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:52.928 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:52.928 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:52.918684+0000", 2024-09-13T11:24:52.928 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.6423999999999998e-05 2024-09-13T11:24:52.928 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:52.928 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:52.928 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:52.928 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:52.918708+0000", 2024-09-13T11:24:52.928 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.4414e-05 2024-09-13T11:24:52.928 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:52.928 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:52.928 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:52.929 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:52.918715+0000", 2024-09-13T11:24:52.929 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.3999999999999997e-06 2024-09-13T11:24:52.929 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:52.929 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:52.929 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:52.929 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:52.918761+0000", 2024-09-13T11:24:52.929 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.5911999999999997e-05 2024-09-13T11:24:52.929 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:52.929 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:52.929 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:24:52.929 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:52.918779+0000", 2024-09-13T11:24:52.929 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.8413000000000001e-05 2024-09-13T11:24:52.929 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:52.929 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:52.930 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:52.930 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:52.930 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:52.930 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:136772 3.14 e138/122)", 2024-09-13T11:24:52.930 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:52.920161+0000", 2024-09-13T11:24:52.930 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 0.0017961229999999999, 2024-09-13T11:24:52.930 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0019233989999999999, 2024-09-13T11:24:52.930 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:52.930 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:52.930 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "queued for pg", 2024-09-13T11:24:52.930 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:52.930 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:52.930 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:52.930 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:52.920161+0000", 2024-09-13T11:24:52.931 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:52.931 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:52.931 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:52.931 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:52.931 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:52.920161+0000", 2024-09-13T11:24:52.931 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:52.931 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:52.931 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:52.931 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:52.931 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:52.920163+0000", 2024-09-13T11:24:52.931 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.9719999999999999e-06 2024-09-13T11:24:52.931 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:52.931 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:52.931 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:52.931 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:52.920199+0000", 2024-09-13T11:24:52.932 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.5808e-05 2024-09-13T11:24:52.932 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:52.932 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:52.932 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:52.932 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:52.920236+0000", 2024-09-13T11:24:52.932 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.7234999999999998e-05 2024-09-13T11:24:52.932 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:52.932 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:52.932 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:52.932 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:52.920244+0000", 2024-09-13T11:24:52.932 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.8299999999999996e-06 2024-09-13T11:24:52.932 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:52.932 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:52.932 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:52.933 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:52.933 INFO:teuthology.orchestra.run.smithi177.stdout: ], 2024-09-13T11:24:52.933 INFO:teuthology.orchestra.run.smithi177.stdout: "num_ops": 3 2024-09-13T11:24:52.933 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-09-13T11:24:52.947 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_blocked_ops 2024-09-13T11:24:52.953 INFO:tasks.ceph.osd.3.smithi177.stderr:2024-09-13T11:24:52.950+0000 7f10a45d5640 -1 osd.3 138 handle_osd_map injecting map crc failure 2024-09-13T11:24:52.957 INFO:teuthology.orchestra.run.smithi001.stderr:nodeep-scrub is set 2024-09-13T11:24:52.960 INFO:tasks.ceph.osd.8.smithi001.stderr:2024-09-13T11:24:52.959+0000 7fb9a92db640 -1 osd.8 138 handle_osd_map injecting map crc failure 2024-09-13T11:24:52.962 INFO:tasks.ceph.osd.8.smithi001.stderr:2024-09-13T11:24:52.959+0000 7fb9a92db640 -1 osd.8 138 handle_osd_map injecting map crc failure 2024-09-13T11:24:52.976 INFO:tasks.ceph.osd.9.smithi092.stderr:2024-09-13T11:24:52.970+0000 7ff59ddf9640 -1 osd.9 138 handle_osd_map injecting map crc failure 2024-09-13T11:24:53.015 INFO:tasks.radosbench.radosbench.0.smithi177.stdout: 146 16 136896 136880 58.5892 78.9375 0.00339471 0.0170625 2024-09-13T11:24:53.017 INFO:tasks.ceph.osd.12.smithi001.stderr:2024-09-13T11:24:53.015+0000 7f892c6e6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26314) UID: 0 2024-09-13T11:24:53.118 INFO:tasks.ceph.osd.13.smithi092.stderr:2024-09-13T11:24:53.114+0000 7f3e5afea640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26350) UID: 0 2024-09-13T11:24:53.124 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-09-13T11:24:53.125 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [], 2024-09-13T11:24:53.125 INFO:teuthology.orchestra.run.smithi177.stdout: "complaint_time": 30, 2024-09-13T11:24:53.125 INFO:teuthology.orchestra.run.smithi177.stdout: "num_blocked_ops": 0 2024-09-13T11:24:53.125 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-09-13T11:24:53.151 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_historic_ops 2024-09-13T11:24:53.219 INFO:tasks.ceph.osd.10.smithi162.stderr:2024-09-13T11:24:53.216+0000 7fb59c9f0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26317) UID: 0 2024-09-13T11:24:53.321 INFO:tasks.ceph.osd.3.smithi177.stderr:2024-09-13T11:24:53.318+0000 7f10adba4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26574) UID: 0 2024-09-13T11:24:53.329 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-09-13T11:24:53.329 INFO:teuthology.orchestra.run.smithi177.stdout: "size": 20, 2024-09-13T11:24:53.329 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 600, 2024-09-13T11:24:53.329 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [ 2024-09-13T11:24:53.329 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:53.329 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:25383 3.0 e55/30 3:0a6bb869:::benchmark_data_smithi177_31116_object25382:head v 55'1620, mlcod=55'1617)", 2024-09-13T11:24:53.330 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:04.449956+0000", 2024-09-13T11:24:53.330 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 108.876535261, 2024-09-13T11:24:53.330 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.21143479600000001, 2024-09-13T11:24:53.330 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:53.330 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:53.330 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:53.330 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:53.330 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:53.330 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:53.330 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.449956+0000", 2024-09-13T11:24:53.330 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:53.330 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:53.330 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:53.331 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:53.331 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.449956+0000", 2024-09-13T11:24:53.331 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:53.331 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:53.331 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:53.331 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:53.331 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.449959+0000", 2024-09-13T11:24:53.331 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.393e-06 2024-09-13T11:24:53.331 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:53.331 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:53.331 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:53.331 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.450014+0000", 2024-09-13T11:24:53.331 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.4200000000000003e-05 2024-09-13T11:24:53.331 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:53.332 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:53.332 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:53.332 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.658637+0000", 2024-09-13T11:24:53.332 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.20862334799999999 2024-09-13T11:24:53.332 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:53.332 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:53.332 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:53.332 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.658650+0000", 2024-09-13T11:24:53.332 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3148e-05 2024-09-13T11:24:53.332 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:53.332 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:53.332 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:53.332 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.658867+0000", 2024-09-13T11:24:53.332 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00021673499999999999 2024-09-13T11:24:53.333 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:53.333 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:53.333 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:24:53.333 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.658931+0000", 2024-09-13T11:24:53.333 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.4045000000000001e-05 2024-09-13T11:24:53.333 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:53.333 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:53.333 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:24:53.333 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.661317+0000", 2024-09-13T11:24:53.333 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.002385823 2024-09-13T11:24:53.333 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:53.333 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:53.333 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:24:53.334 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.661391+0000", 2024-09-13T11:24:53.334 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.4103999999999994e-05 2024-09-13T11:24:53.334 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:53.334 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:53.334 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:53.334 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:53.334 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:53.334 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPush(3.16 85/70 [PushOp(3:6ac421eb:::benchmark_data_smithi177_31116_object43910:head, version: 66'2836, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:6ac421eb:::benchmark_data_smithi177_31116_object43910:head@66'2836, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:24:53.334 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:42.145291+0000", 2024-09-13T11:24:53.334 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 71.181200055000005, 2024-09-13T11:24:53.334 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.148514919, 2024-09-13T11:24:53.334 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:53.334 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:53.334 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:24:53.335 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:53.335 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:53.335 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:53.335 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.145291+0000", 2024-09-13T11:24:53.335 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:53.335 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:53.335 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:53.335 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:53.335 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.145291+0000", 2024-09-13T11:24:53.335 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:53.335 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:53.335 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:53.335 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:53.336 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.145293+0000", 2024-09-13T11:24:53.336 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.699e-06 2024-09-13T11:24:53.336 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:53.336 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:53.336 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:53.336 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.145417+0000", 2024-09-13T11:24:53.336 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00012415200000000001 2024-09-13T11:24:53.336 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:53.336 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:53.336 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:53.336 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.292964+0000", 2024-09-13T11:24:53.336 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.14754715400000001 2024-09-13T11:24:53.336 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:53.337 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:53.337 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:53.337 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.292981+0000", 2024-09-13T11:24:53.337 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.6790999999999999e-05 2024-09-13T11:24:53.337 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:53.337 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:53.337 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:53.337 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.293083+0000", 2024-09-13T11:24:53.337 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00010199799999999999 2024-09-13T11:24:53.337 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:53.337 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:53.337 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:24:53.337 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.293098+0000", 2024-09-13T11:24:53.337 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.4612e-05 2024-09-13T11:24:53.338 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:53.338 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:53.338 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:24:53.338 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.293806+0000", 2024-09-13T11:24:53.338 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00070851300000000002 2024-09-13T11:24:53.338 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:53.338 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:53.338 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:53.338 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:53.338 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:53.338 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:60390 3.16 e85/70 3:6a388186:::benchmark_data_smithi177_31116_object60389:head v 85'3357, mlcod=66'2835)", 2024-09-13T11:24:53.338 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:42.149102+0000", 2024-09-13T11:24:53.338 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 71.177388997999998, 2024-09-13T11:24:53.338 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.14821161799999999, 2024-09-13T11:24:53.339 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:53.339 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:53.339 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:53.339 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:53.339 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:53.339 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:53.339 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.149102+0000", 2024-09-13T11:24:53.339 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:53.339 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:53.339 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:53.339 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:53.339 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.149102+0000", 2024-09-13T11:24:53.339 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:53.340 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:53.340 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:53.340 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:53.340 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.149104+0000", 2024-09-13T11:24:53.340 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.708e-06 2024-09-13T11:24:53.340 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:53.340 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:53.340 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:53.340 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.149160+0000", 2024-09-13T11:24:53.340 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.5618000000000002e-05 2024-09-13T11:24:53.340 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:53.340 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:53.341 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:53.341 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.296071+0000", 2024-09-13T11:24:53.341 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.14691148600000001 2024-09-13T11:24:53.341 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:53.341 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:53.341 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:53.341 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.296076+0000", 2024-09-13T11:24:53.341 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.5580000000000002e-06 2024-09-13T11:24:53.341 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:53.341 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:53.341 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:53.341 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.296127+0000", 2024-09-13T11:24:53.341 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.1405e-05 2024-09-13T11:24:53.341 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:53.342 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:53.342 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:24:53.342 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.296150+0000", 2024-09-13T11:24:53.342 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.3113e-05 2024-09-13T11:24:53.342 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:53.342 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:53.342 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:24:53.342 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.297274+0000", 2024-09-13T11:24:53.342 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0011239900000000001 2024-09-13T11:24:53.342 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:53.342 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:53.342 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:24:53.342 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.297314+0000", 2024-09-13T11:24:53.342 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.9740000000000002e-05 2024-09-13T11:24:53.343 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:53.343 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:53.343 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:53.343 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:53.343 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:53.343 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPush(3.16 85/70 [PushOp(3:69aac9fd:::benchmark_data_smithi177_31116_object43914:head, version: 66'2837, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:69aac9fd:::benchmark_data_smithi177_31116_object43914:head@66'2837, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:24:53.343 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:42.295995+0000", 2024-09-13T11:24:53.343 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 71.030495940999998, 2024-09-13T11:24:53.343 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.215089431, 2024-09-13T11:24:53.343 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:53.343 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:53.343 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:24:53.344 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:53.344 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:53.344 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:53.344 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.295995+0000", 2024-09-13T11:24:53.344 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:53.344 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:53.344 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:53.344 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:53.344 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.295995+0000", 2024-09-13T11:24:53.344 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:53.344 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:53.344 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:53.345 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:53.345 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.295997+0000", 2024-09-13T11:24:53.345 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3170000000000001e-06 2024-09-13T11:24:53.345 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:53.345 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:53.345 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:53.345 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.296058+0000", 2024-09-13T11:24:53.345 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.1690000000000001e-05 2024-09-13T11:24:53.345 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:53.345 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:53.345 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:53.345 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.509606+0000", 2024-09-13T11:24:53.345 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.21354748300000001 2024-09-13T11:24:53.345 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:53.346 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:53.346 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:53.346 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.509638+0000", 2024-09-13T11:24:53.346 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.1644999999999999e-05 2024-09-13T11:24:53.346 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:53.346 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:53.346 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:53.346 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.509814+0000", 2024-09-13T11:24:53.346 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00017679499999999999 2024-09-13T11:24:53.346 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:53.346 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:53.346 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:24:53.346 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.509840+0000", 2024-09-13T11:24:53.347 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.5871e-05 2024-09-13T11:24:53.347 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:53.347 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:53.347 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:24:53.347 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.511085+0000", 2024-09-13T11:24:53.347 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00124463 2024-09-13T11:24:53.347 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:53.347 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:53.349 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:53.349 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:53.349 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:53.349 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPush(3.1d 91/78 [PushOp(3:be6a0ae4:::benchmark_data_smithi177_31116_object25671:head, version: 58'1569, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:be6a0ae4:::benchmark_data_smithi177_31116_object25671:head@58'1569, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:24:53.349 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:48.481971+0000", 2024-09-13T11:24:53.349 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 64.844520696000004, 2024-09-13T11:24:53.349 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.47945454199999998, 2024-09-13T11:24:53.349 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:53.350 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:53.350 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:24:53.350 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:53.350 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:53.350 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:53.350 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.481971+0000", 2024-09-13T11:24:53.350 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:53.350 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:53.350 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:53.350 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:53.350 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.481971+0000", 2024-09-13T11:24:53.350 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:53.350 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:53.351 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:53.351 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:53.351 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.481972+0000", 2024-09-13T11:24:53.351 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5179999999999999e-06 2024-09-13T11:24:53.351 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:53.351 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:53.351 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:53.351 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.482000+0000", 2024-09-13T11:24:53.351 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.8249e-05 2024-09-13T11:24:53.351 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:53.351 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:53.351 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:53.351 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.959926+0000", 2024-09-13T11:24:53.351 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.47792585100000001 2024-09-13T11:24:53.352 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:53.352 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:53.352 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:53.352 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.959944+0000", 2024-09-13T11:24:53.352 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.7620000000000001e-05 2024-09-13T11:24:53.352 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:53.352 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:53.352 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:53.352 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.960059+0000", 2024-09-13T11:24:53.352 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000115188 2024-09-13T11:24:53.352 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:53.352 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:53.353 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:24:53.353 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.960074+0000", 2024-09-13T11:24:53.353 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.4738e-05 2024-09-13T11:24:53.353 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:53.353 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:53.353 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:24:53.353 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.961425+0000", 2024-09-13T11:24:53.353 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0013513780000000001 2024-09-13T11:24:53.353 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:53.353 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:53.353 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:53.353 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:53.353 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:53.354 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:71884 3.16 e96/88 3:6ca846b2:::benchmark_data_smithi177_31116_object71883:head v 96'3755, mlcod=96'3753)", 2024-09-13T11:24:53.354 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:54.108299+0000", 2024-09-13T11:24:53.354 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 59.218192522999999, 2024-09-13T11:24:53.354 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.20685140099999999, 2024-09-13T11:24:53.354 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:53.354 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:53.354 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:53.354 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:53.354 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:53.354 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:53.354 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.108299+0000", 2024-09-13T11:24:53.354 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:53.355 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:53.355 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:53.355 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:53.355 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.108299+0000", 2024-09-13T11:24:53.355 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:53.355 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:53.355 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:53.355 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:53.355 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.108300+0000", 2024-09-13T11:24:53.355 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.173e-06 2024-09-13T11:24:53.355 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:53.355 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:53.355 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:53.355 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.108323+0000", 2024-09-13T11:24:53.356 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.2591000000000001e-05 2024-09-13T11:24:53.356 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:53.356 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:53.356 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:53.356 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.312377+0000", 2024-09-13T11:24:53.356 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.20405456599999999 2024-09-13T11:24:53.356 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:53.356 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:53.356 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:53.356 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.312382+0000", 2024-09-13T11:24:53.356 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.1429999999999997e-06 2024-09-13T11:24:53.356 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:53.357 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:53.357 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:53.357 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.312480+0000", 2024-09-13T11:24:53.357 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.7742e-05 2024-09-13T11:24:53.357 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:53.357 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:53.357 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:24:53.357 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.312521+0000", 2024-09-13T11:24:53.357 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.1369000000000003e-05 2024-09-13T11:24:53.357 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:53.357 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:53.357 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:24:53.357 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.315105+0000", 2024-09-13T11:24:53.357 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0025835060000000002 2024-09-13T11:24:53.358 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:53.358 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:53.358 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:24:53.358 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.315150+0000", 2024-09-13T11:24:53.358 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.5311e-05 2024-09-13T11:24:53.358 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:53.358 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:53.358 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:53.358 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:53.358 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:53.358 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:71927 3.0 e97/75 3:0197244a:::benchmark_data_smithi177_31116_object71926:head v 97'3351, mlcod=96'3348)", 2024-09-13T11:24:53.358 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:54.200161+0000", 2024-09-13T11:24:53.359 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 59.126330295000002, 2024-09-13T11:24:53.359 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.26691738999999998, 2024-09-13T11:24:53.359 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:53.359 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:53.359 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:53.359 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:53.359 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:53.359 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:53.359 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.200161+0000", 2024-09-13T11:24:53.359 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:53.359 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:53.359 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:53.359 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:53.360 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.200161+0000", 2024-09-13T11:24:53.360 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:53.360 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:53.360 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:53.360 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:53.360 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.200162+0000", 2024-09-13T11:24:53.360 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.78e-07 2024-09-13T11:24:53.360 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:53.360 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:53.360 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:53.360 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.200200+0000", 2024-09-13T11:24:53.360 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.8121000000000003e-05 2024-09-13T11:24:53.361 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:53.361 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:53.361 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:53.361 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.465397+0000", 2024-09-13T11:24:53.361 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.26519681099999998 2024-09-13T11:24:53.361 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:53.361 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:53.361 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:53.361 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.465404+0000", 2024-09-13T11:24:53.361 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.7399999999999998e-06 2024-09-13T11:24:53.361 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:53.361 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:53.361 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:53.362 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.465456+0000", 2024-09-13T11:24:53.362 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.2626000000000001e-05 2024-09-13T11:24:53.362 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:53.362 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:53.362 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:24:53.362 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.465482+0000", 2024-09-13T11:24:53.362 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.6086000000000001e-05 2024-09-13T11:24:53.362 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:53.362 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:53.362 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:24:53.362 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.467035+0000", 2024-09-13T11:24:53.362 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0015523729999999999 2024-09-13T11:24:53.362 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:53.363 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:53.363 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:24:53.363 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.467078+0000", 2024-09-13T11:24:53.363 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.3754999999999997e-05 2024-09-13T11:24:53.363 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:53.363 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:53.363 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:53.363 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:53.363 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:53.363 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:72250 3.0 e97/75 3:010c7171:::benchmark_data_smithi177_31116_object72249:head v 97'3353, mlcod=96'3350)", 2024-09-13T11:24:53.363 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:54.516381+0000", 2024-09-13T11:24:53.363 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 58.810110782000002, 2024-09-13T11:24:53.363 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.13663347300000001, 2024-09-13T11:24:53.364 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:53.364 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:53.364 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:53.364 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:53.364 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:53.364 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:53.364 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.516381+0000", 2024-09-13T11:24:53.364 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:53.364 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:53.364 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:53.364 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:53.364 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.516381+0000", 2024-09-13T11:24:53.364 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:53.365 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:53.365 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:53.365 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:53.365 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.516382+0000", 2024-09-13T11:24:53.365 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3239999999999999e-06 2024-09-13T11:24:53.365 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:53.365 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:53.365 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:53.365 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.516437+0000", 2024-09-13T11:24:53.365 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.5408999999999998e-05 2024-09-13T11:24:53.365 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:53.365 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:53.365 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:53.366 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.651731+0000", 2024-09-13T11:24:53.366 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.135293583 2024-09-13T11:24:53.366 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:53.366 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:53.366 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:53.366 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.651736+0000", 2024-09-13T11:24:53.366 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.3410000000000002e-06 2024-09-13T11:24:53.366 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:53.366 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:53.366 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:53.366 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.651791+0000", 2024-09-13T11:24:53.366 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.4758e-05 2024-09-13T11:24:53.366 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:53.366 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:53.367 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:24:53.367 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.651814+0000", 2024-09-13T11:24:53.367 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.3283999999999999e-05 2024-09-13T11:24:53.367 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:53.367 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:53.367 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:24:53.367 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.652975+0000", 2024-09-13T11:24:53.367 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0011604370000000001 2024-09-13T11:24:53.367 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:53.367 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:53.367 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:24:53.367 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.653014+0000", 2024-09-13T11:24:53.367 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.9337e-05 2024-09-13T11:24:53.367 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:53.368 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:53.368 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:53.368 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:53.368 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:53.368 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:72993 3.0 e97/75 3:02747b8f:::benchmark_data_smithi177_31116_object72992:head v 97'3366, mlcod=97'3363)", 2024-09-13T11:24:53.368 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:55.116009+0000", 2024-09-13T11:24:53.368 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 58.210482175999999, 2024-09-13T11:24:53.368 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.13962660099999999, 2024-09-13T11:24:53.368 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:53.368 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:53.368 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:53.368 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:53.368 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:53.369 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:53.369 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:55.116009+0000", 2024-09-13T11:24:53.369 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:53.369 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:53.369 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:53.369 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:53.369 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:55.116009+0000", 2024-09-13T11:24:53.369 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:53.369 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:53.369 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:53.369 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:53.369 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:55.116010+0000", 2024-09-13T11:24:53.369 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.291e-06 2024-09-13T11:24:53.370 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:53.370 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:53.370 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:53.370 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:55.116062+0000", 2024-09-13T11:24:53.370 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.1949e-05 2024-09-13T11:24:53.370 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:53.370 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:53.370 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:53.370 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:55.254100+0000", 2024-09-13T11:24:53.370 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.13803723000000001 2024-09-13T11:24:53.370 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:53.370 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:53.370 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:53.370 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:55.254106+0000", 2024-09-13T11:24:53.371 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.5939999999999999e-06 2024-09-13T11:24:53.371 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:53.371 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:53.371 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:53.371 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:55.254196+0000", 2024-09-13T11:24:53.371 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.9567999999999994e-05 2024-09-13T11:24:53.371 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:53.371 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:53.371 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:24:53.371 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:55.254219+0000", 2024-09-13T11:24:53.371 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.3266000000000001e-05 2024-09-13T11:24:53.371 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:53.371 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:53.372 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:24:53.372 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:55.255596+0000", 2024-09-13T11:24:53.372 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0013767409999999999 2024-09-13T11:24:53.372 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:53.372 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:53.372 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:24:53.372 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:55.255636+0000", 2024-09-13T11:24:53.372 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.9962000000000001e-05 2024-09-13T11:24:53.372 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:53.372 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:53.372 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:53.372 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:53.373 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:53.373 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:79634 3.5 3:a6c32e6f:::benchmark_data_smithi177_31116_object79633:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101)", 2024-09-13T11:24:53.373 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:00.168616+0000", 2024-09-13T11:24:53.373 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 53.157874907, 2024-09-13T11:24:53.373 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.169445552, 2024-09-13T11:24:53.373 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:53.373 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:53.373 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:53.373 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:24:53.373 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:24:53.373 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:53.373 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 79634 2024-09-13T11:24:53.374 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:53.374 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:53.374 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:53.374 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:53.374 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.168616+0000", 2024-09-13T11:24:53.374 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:53.374 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:53.374 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:53.374 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:53.374 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.168616+0000", 2024-09-13T11:24:53.374 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:53.374 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:53.374 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:53.375 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:53.375 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.168619+0000", 2024-09-13T11:24:53.375 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.565e-06 2024-09-13T11:24:53.375 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:53.375 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:53.375 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:53.375 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.168639+0000", 2024-09-13T11:24:53.375 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.0384999999999999e-05 2024-09-13T11:24:53.375 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:53.375 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:53.375 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:53.375 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.168652+0000", 2024-09-13T11:24:53.375 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.2779000000000001e-05 2024-09-13T11:24:53.375 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:53.376 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:53.376 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:53.376 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.168658+0000", 2024-09-13T11:24:53.376 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.2269999999999998e-06 2024-09-13T11:24:53.376 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:53.376 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:53.376 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:53.376 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.168722+0000", 2024-09-13T11:24:53.376 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.3888999999999998e-05 2024-09-13T11:24:53.376 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:53.376 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:53.376 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:24:53.376 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.168863+0000", 2024-09-13T11:24:53.376 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00014050799999999999 2024-09-13T11:24:53.377 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:53.377 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:53.377 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 13", 2024-09-13T11:24:53.377 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.169010+0000", 2024-09-13T11:24:53.377 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00014746200000000001 2024-09-13T11:24:53.377 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:53.377 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:53.377 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:24:53.377 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.170891+0000", 2024-09-13T11:24:53.377 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001880945 2024-09-13T11:24:53.377 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:53.377 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:53.377 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:53.377 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.337961+0000", 2024-09-13T11:24:53.378 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.16706952 2024-09-13T11:24:53.378 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:53.378 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:53.378 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:24:53.378 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.338019+0000", 2024-09-13T11:24:53.378 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.7958000000000003e-05 2024-09-13T11:24:53.378 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:53.378 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:53.378 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:24:53.378 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.338062+0000", 2024-09-13T11:24:53.378 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.3313999999999999e-05 2024-09-13T11:24:53.378 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:53.378 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:53.379 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:53.379 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:53.379 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:53.379 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPush(3.1b 102/78 [PushOp(3:df786fc7:::benchmark_data_smithi177_31116_object48840:head, version: 71'3079, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:df786fc7:::benchmark_data_smithi177_31116_object48840:head@71'3079, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:24:53.379 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:00.290536+0000", 2024-09-13T11:24:53.379 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 53.035955532999999, 2024-09-13T11:24:53.379 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.20017585399999999, 2024-09-13T11:24:53.379 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:53.379 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:53.379 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:24:53.379 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:53.379 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:53.379 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:53.379 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.290536+0000", 2024-09-13T11:24:53.380 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:53.380 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:53.380 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:53.380 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:53.380 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.290536+0000", 2024-09-13T11:24:53.380 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:53.380 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:53.380 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:53.380 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:53.380 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.290537+0000", 2024-09-13T11:24:53.380 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.327e-06 2024-09-13T11:24:53.380 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:53.380 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:53.380 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:53.381 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.290581+0000", 2024-09-13T11:24:53.381 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.3893999999999997e-05 2024-09-13T11:24:53.381 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:53.381 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:53.381 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:53.381 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.488588+0000", 2024-09-13T11:24:53.381 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.198006708 2024-09-13T11:24:53.381 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:53.381 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:53.381 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:53.381 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.488607+0000", 2024-09-13T11:24:53.381 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.9162999999999999e-05 2024-09-13T11:24:53.381 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:53.382 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:53.382 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:53.382 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.488747+0000", 2024-09-13T11:24:53.382 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00014040799999999999 2024-09-13T11:24:53.382 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:53.382 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:53.382 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:24:53.382 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.488770+0000", 2024-09-13T11:24:53.382 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.279e-05 2024-09-13T11:24:53.382 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:53.382 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:53.382 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:24:53.382 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.490712+0000", 2024-09-13T11:24:53.382 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001941564 2024-09-13T11:24:53.383 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:53.383 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:53.383 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:53.383 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:53.383 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:53.383 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPush(3.1b 102/78 [PushOp(3:d82bf6ec:::benchmark_data_smithi177_31116_object48847:head, version: 71'3080, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:d82bf6ec:::benchmark_data_smithi177_31116_object48847:head@71'3080, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:24:53.383 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:00.293979+0000", 2024-09-13T11:24:53.383 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 53.032512005999997, 2024-09-13T11:24:53.383 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.20314832099999999, 2024-09-13T11:24:53.383 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:53.383 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:53.383 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:24:53.383 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:53.383 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:53.384 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:53.384 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.293979+0000", 2024-09-13T11:24:53.384 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:53.384 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:53.384 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:53.384 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:53.384 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.293979+0000", 2024-09-13T11:24:53.384 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:53.384 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:53.384 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:53.384 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:53.384 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.293981+0000", 2024-09-13T11:24:53.384 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.4449999999999999e-06 2024-09-13T11:24:53.385 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:53.385 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:53.385 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:53.385 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.294085+0000", 2024-09-13T11:24:53.385 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000103848 2024-09-13T11:24:53.385 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:53.385 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:53.385 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:53.385 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.495571+0000", 2024-09-13T11:24:53.385 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.20148640800000001 2024-09-13T11:24:53.385 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:53.385 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:53.385 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:53.386 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.495609+0000", 2024-09-13T11:24:53.386 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.7904000000000002e-05 2024-09-13T11:24:53.386 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:53.386 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:53.386 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:53.386 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.495729+0000", 2024-09-13T11:24:53.386 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000120556 2024-09-13T11:24:53.386 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:53.386 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:53.386 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:24:53.386 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.495751+0000", 2024-09-13T11:24:53.386 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.1511000000000001e-05 2024-09-13T11:24:53.387 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:53.387 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:53.387 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:24:53.387 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.497128+0000", 2024-09-13T11:24:53.387 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001376649 2024-09-13T11:24:53.387 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:53.387 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:53.387 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:53.387 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:53.387 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:53.387 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:79801 3.5 3:a6efdf84:::benchmark_data_smithi177_31116_object79800:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102)", 2024-09-13T11:24:53.387 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:00.403201+0000", 2024-09-13T11:24:53.387 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 52.923290418999997, 2024-09-13T11:24:53.387 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.31868626300000003, 2024-09-13T11:24:53.388 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:53.388 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:53.388 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:53.388 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:24:53.388 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:24:53.388 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:53.388 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 79801 2024-09-13T11:24:53.388 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:53.388 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:53.388 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:53.388 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:53.388 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.403201+0000", 2024-09-13T11:24:53.388 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:53.389 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:53.389 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:53.389 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:53.389 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.403201+0000", 2024-09-13T11:24:53.389 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:53.389 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:53.389 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:53.389 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:53.389 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.403204+0000", 2024-09-13T11:24:53.389 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.0819999999999999e-06 2024-09-13T11:24:53.389 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:53.389 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:53.389 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:53.390 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.403260+0000", 2024-09-13T11:24:53.390 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.5624999999999997e-05 2024-09-13T11:24:53.390 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:53.390 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:53.390 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:53.390 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.403280+0000", 2024-09-13T11:24:53.390 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.0060000000000001e-05 2024-09-13T11:24:53.390 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:53.390 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:53.390 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:53.390 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.403291+0000", 2024-09-13T11:24:53.390 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.1335000000000001e-05 2024-09-13T11:24:53.390 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:53.391 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:53.391 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:53.391 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.403344+0000", 2024-09-13T11:24:53.391 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.3217000000000002e-05 2024-09-13T11:24:53.391 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:53.391 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:53.391 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:24:53.391 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.403456+0000", 2024-09-13T11:24:53.391 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000111706 2024-09-13T11:24:53.391 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:53.391 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:53.391 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 13", 2024-09-13T11:24:53.391 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.403547+0000", 2024-09-13T11:24:53.392 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.1556000000000002e-05 2024-09-13T11:24:53.392 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:53.392 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:53.392 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:24:53.392 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.407436+0000", 2024-09-13T11:24:53.392 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.003888856 2024-09-13T11:24:53.392 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:53.392 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:53.392 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:53.392 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.721722+0000", 2024-09-13T11:24:53.392 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.31428562700000001 2024-09-13T11:24:53.392 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:53.392 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:53.393 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:24:53.393 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.721814+0000", 2024-09-13T11:24:53.393 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.1564000000000006e-05 2024-09-13T11:24:53.393 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:53.393 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:53.393 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:24:53.393 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.721887+0000", 2024-09-13T11:24:53.393 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.3634999999999995e-05 2024-09-13T11:24:53.393 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:53.393 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:53.393 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:53.393 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:53.393 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:53.394 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPush(3.1b 102/78 [PushOp(3:dcbefe42:::benchmark_data_smithi177_31116_object49001:head, version: 71'3087, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:dcbefe42:::benchmark_data_smithi177_31116_object49001:head@71'3087, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:24:53.394 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:00.511027+0000", 2024-09-13T11:24:53.394 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 52.815464483, 2024-09-13T11:24:53.394 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.13427292699999999, 2024-09-13T11:24:53.394 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:53.394 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:53.394 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:24:53.394 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:53.394 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:53.394 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:53.394 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.511027+0000", 2024-09-13T11:24:53.394 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:53.394 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:53.394 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:53.395 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:53.395 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.511027+0000", 2024-09-13T11:24:53.395 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:53.395 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:53.395 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:53.395 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:53.395 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.511028+0000", 2024-09-13T11:24:53.395 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.4869999999999999e-06 2024-09-13T11:24:53.395 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:53.395 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:53.395 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:53.395 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.511095+0000", 2024-09-13T11:24:53.395 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.6382000000000002e-05 2024-09-13T11:24:53.396 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:53.396 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:53.396 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:53.396 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.643319+0000", 2024-09-13T11:24:53.396 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.13222455699999999 2024-09-13T11:24:53.396 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:53.396 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:53.396 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:53.396 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.643349+0000", 2024-09-13T11:24:53.396 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.9356000000000002e-05 2024-09-13T11:24:53.396 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:53.396 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:53.396 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:53.397 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.643518+0000", 2024-09-13T11:24:53.397 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00016963500000000001 2024-09-13T11:24:53.397 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:53.397 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:53.397 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:24:53.397 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.643541+0000", 2024-09-13T11:24:53.397 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.2367000000000001e-05 2024-09-13T11:24:53.397 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:53.397 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:53.397 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:24:53.397 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.645300+0000", 2024-09-13T11:24:53.397 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0017591429999999999 2024-09-13T11:24:53.398 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:53.398 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:53.398 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:53.398 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:53.398 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:53.398 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:79813 3.16 e102/88 3:6c2df1ae:::benchmark_data_smithi177_31116_object79812:head v 102'4016, mlcod=102'4014)", 2024-09-13T11:24:53.398 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:00.516959+0000", 2024-09-13T11:24:53.398 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 52.809532458, 2024-09-13T11:24:53.398 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.15075599100000001, 2024-09-13T11:24:53.398 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:53.398 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:53.398 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:53.398 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:53.399 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:53.399 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:53.399 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.516959+0000", 2024-09-13T11:24:53.399 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:53.399 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:53.399 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:53.399 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:53.399 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.516959+0000", 2024-09-13T11:24:53.399 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:53.399 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:53.399 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:53.399 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:53.399 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.516961+0000", 2024-09-13T11:24:53.400 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.0279999999999999e-06 2024-09-13T11:24:53.400 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:53.400 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:53.400 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:53.400 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.517002+0000", 2024-09-13T11:24:53.400 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.0754999999999999e-05 2024-09-13T11:24:53.400 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:53.400 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:53.400 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:53.400 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.664491+0000", 2024-09-13T11:24:53.400 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.14748925099999999 2024-09-13T11:24:53.400 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:53.400 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:53.401 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:53.401 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.664504+0000", 2024-09-13T11:24:53.401 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.2747e-05 2024-09-13T11:24:53.401 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:53.401 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:53.401 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:53.401 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.664642+0000", 2024-09-13T11:24:53.401 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00013850900000000001 2024-09-13T11:24:53.401 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:53.401 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:53.401 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:24:53.401 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.664678+0000", 2024-09-13T11:24:53.402 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.5403e-05 2024-09-13T11:24:53.402 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:53.402 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:53.402 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:24:53.402 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.667646+0000", 2024-09-13T11:24:53.402 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.002968133 2024-09-13T11:24:53.402 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:53.402 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:53.402 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:24:53.402 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.667715+0000", 2024-09-13T11:24:53.402 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.9165000000000006e-05 2024-09-13T11:24:53.402 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:53.402 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:53.403 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:53.403 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:53.403 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:53.403 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:83199 3.1d e106/102 3:be2c6fae:::benchmark_data_smithi177_31116_object83198:head v 106'4244, mlcod=106'4242)", 2024-09-13T11:24:53.403 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:04.298299+0000", 2024-09-13T11:24:53.403 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 49.028192292, 2024-09-13T11:24:53.403 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.25001107099999997, 2024-09-13T11:24:53.403 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:53.403 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:53.403 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:53.403 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:53.403 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:53.403 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:53.404 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.298299+0000", 2024-09-13T11:24:53.404 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:53.404 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:53.404 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:53.404 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:53.404 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.298299+0000", 2024-09-13T11:24:53.404 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:53.404 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:53.404 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:53.404 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:53.404 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.298301+0000", 2024-09-13T11:24:53.404 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5859999999999999e-06 2024-09-13T11:24:53.404 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:53.405 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:53.405 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:53.405 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.298368+0000", 2024-09-13T11:24:53.405 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.6928999999999996e-05 2024-09-13T11:24:53.405 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:53.405 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:53.405 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:53.405 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.545180+0000", 2024-09-13T11:24:53.405 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.24681209700000001 2024-09-13T11:24:53.405 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:53.405 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:53.405 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:53.406 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.545193+0000", 2024-09-13T11:24:53.406 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3427e-05 2024-09-13T11:24:53.406 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:53.406 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:53.406 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:53.406 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.545327+0000", 2024-09-13T11:24:53.406 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00013359400000000001 2024-09-13T11:24:53.406 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:53.406 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:53.406 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:24:53.406 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.545385+0000", 2024-09-13T11:24:53.406 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.8492000000000002e-05 2024-09-13T11:24:53.406 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:53.407 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:53.407 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:24:53.407 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.548225+0000", 2024-09-13T11:24:53.407 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.002839472 2024-09-13T11:24:53.407 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:53.407 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:53.407 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:24:53.407 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.548310+0000", 2024-09-13T11:24:53.407 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.5474000000000002e-05 2024-09-13T11:24:53.407 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:53.407 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:53.407 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:53.407 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:53.408 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:53.408 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:84621 3.1d e107/102 3:becd72aa:::benchmark_data_smithi177_31116_object84620:head v 107'4296, mlcod=107'4293)", 2024-09-13T11:24:53.408 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:05.742238+0000", 2024-09-13T11:24:53.408 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 47.584253191999998, 2024-09-13T11:24:53.408 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.17888174300000001, 2024-09-13T11:24:53.408 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:53.408 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:53.408 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:53.408 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:53.408 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:53.408 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:53.408 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:05.742238+0000", 2024-09-13T11:24:53.408 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:53.409 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:53.409 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:53.409 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:53.409 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:05.742238+0000", 2024-09-13T11:24:53.409 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:53.409 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:53.409 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:53.409 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:53.409 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:05.742240+0000", 2024-09-13T11:24:53.409 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.3159999999999999e-06 2024-09-13T11:24:53.409 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:53.409 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:53.409 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:53.409 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:05.742315+0000", 2024-09-13T11:24:53.410 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.4122000000000006e-05 2024-09-13T11:24:53.410 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:53.410 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:53.410 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:53.410 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:05.917757+0000", 2024-09-13T11:24:53.410 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.175442031 2024-09-13T11:24:53.410 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:53.410 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:53.410 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:53.410 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:05.917764+0000", 2024-09-13T11:24:53.410 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.661e-06 2024-09-13T11:24:53.410 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:53.410 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:53.411 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:53.411 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:05.917846+0000", 2024-09-13T11:24:53.411 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.2006000000000001e-05 2024-09-13T11:24:53.411 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:53.411 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:53.411 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:24:53.411 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:05.917879+0000", 2024-09-13T11:24:53.411 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.2796000000000002e-05 2024-09-13T11:24:53.411 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:53.411 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:53.411 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:24:53.411 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:05.921070+0000", 2024-09-13T11:24:53.411 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0031908489999999999 2024-09-13T11:24:53.412 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:53.412 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:53.412 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:24:53.412 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:05.921120+0000", 2024-09-13T11:24:53.412 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.9962e-05 2024-09-13T11:24:53.412 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:53.412 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:53.412 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:53.412 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:53.412 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:53.412 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:100995 3.1d e114/102 3:b9d3a02e:::benchmark_data_smithi177_31116_object100994:head v 114'4771, mlcod=114'4769)", 2024-09-13T11:24:53.412 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:17.338503+0000", 2024-09-13T11:24:53.412 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 35.987988528000002, 2024-09-13T11:24:53.413 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.12755466100000001, 2024-09-13T11:24:53.413 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:53.413 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:53.413 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:53.413 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:53.413 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:53.413 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:53.413 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:17.338503+0000", 2024-09-13T11:24:53.413 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:53.413 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:53.413 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:53.413 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:53.414 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:17.338503+0000", 2024-09-13T11:24:53.414 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:53.414 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:53.414 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:53.414 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:53.414 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:17.338504+0000", 2024-09-13T11:24:53.414 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.635e-06 2024-09-13T11:24:53.414 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:53.414 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:53.414 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:53.414 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:17.338575+0000", 2024-09-13T11:24:53.414 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.0869000000000002e-05 2024-09-13T11:24:53.414 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:53.415 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:53.415 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:53.415 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:17.462756+0000", 2024-09-13T11:24:53.415 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.124180943 2024-09-13T11:24:53.415 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:53.415 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:53.415 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:53.415 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:17.462764+0000", 2024-09-13T11:24:53.415 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.3170000000000004e-06 2024-09-13T11:24:53.415 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:53.415 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:53.415 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:53.416 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:17.462887+0000", 2024-09-13T11:24:53.416 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00012359200000000001 2024-09-13T11:24:53.416 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:53.416 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:53.416 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:24:53.416 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:17.462948+0000", 2024-09-13T11:24:53.416 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.0563999999999998e-05 2024-09-13T11:24:53.416 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:53.416 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:53.416 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:24:53.416 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:17.465984+0000", 2024-09-13T11:24:53.416 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0030363669999999999 2024-09-13T11:24:53.416 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:53.417 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:53.417 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:24:53.417 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:17.466057+0000", 2024-09-13T11:24:53.417 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.3374000000000006e-05 2024-09-13T11:24:53.417 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:53.417 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:53.417 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:53.417 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:53.417 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:53.417 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:107910 3.16 e116/88 3:6cf57aa3:::benchmark_data_smithi177_31116_object107909:head v 116'4911, mlcod=116'4908)", 2024-09-13T11:24:53.417 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:22.516225+0000", 2024-09-13T11:24:53.417 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 30.810266648999999, 2024-09-13T11:24:53.418 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.18319666600000001, 2024-09-13T11:24:53.418 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:53.418 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:53.418 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:53.418 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:53.418 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:53.418 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:53.418 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:22.516225+0000", 2024-09-13T11:24:53.418 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:53.418 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:53.418 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:53.418 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:53.418 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:22.516225+0000", 2024-09-13T11:24:53.419 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:53.419 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:53.419 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:53.419 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:53.419 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:22.516226+0000", 2024-09-13T11:24:53.419 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.716e-06 2024-09-13T11:24:53.419 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:53.419 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:53.419 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:53.419 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:22.516250+0000", 2024-09-13T11:24:53.419 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.3975e-05 2024-09-13T11:24:53.419 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:53.419 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:53.420 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:53.420 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:22.697686+0000", 2024-09-13T11:24:53.420 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.18143539 2024-09-13T11:24:53.420 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:53.420 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:53.420 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:53.420 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:22.697692+0000", 2024-09-13T11:24:53.420 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.0769999999999996e-06 2024-09-13T11:24:53.420 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:53.420 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:53.420 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:53.420 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:22.697744+0000", 2024-09-13T11:24:53.420 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.2525000000000003e-05 2024-09-13T11:24:53.421 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:53.421 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:53.421 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:24:53.421 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:22.697757+0000", 2024-09-13T11:24:53.421 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.2819e-05 2024-09-13T11:24:53.421 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:53.421 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:53.421 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:24:53.421 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:22.699388+0000", 2024-09-13T11:24:53.421 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0016303999999999999 2024-09-13T11:24:53.421 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:53.421 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:53.422 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:24:53.422 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:22.699421+0000", 2024-09-13T11:24:53.422 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.3763999999999997e-05 2024-09-13T11:24:53.422 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:53.422 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:53.422 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:53.422 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:53.422 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:53.422 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:127682 3.16 e131/88 3:6e6da02e:::benchmark_data_smithi177_31116_object127681:head v 131'5491, mlcod=131'5488)", 2024-09-13T11:24:53.422 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:44.556692+0000", 2024-09-13T11:24:53.422 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 8.76979972, 2024-09-13T11:24:53.422 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.18048624299999999, 2024-09-13T11:24:53.422 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:53.423 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:53.423 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:53.423 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:53.423 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:53.423 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:53.423 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:44.556692+0000", 2024-09-13T11:24:53.423 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:53.423 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:53.423 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:53.423 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:53.423 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:44.556692+0000", 2024-09-13T11:24:53.423 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:53.423 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:53.424 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:53.424 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:53.424 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:44.556693+0000", 2024-09-13T11:24:53.424 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.8109999999999999e-06 2024-09-13T11:24:53.424 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:53.424 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:53.424 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:53.424 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:44.556715+0000", 2024-09-13T11:24:53.424 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.1328999999999999e-05 2024-09-13T11:24:53.424 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:53.424 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:53.424 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:53.425 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:44.734908+0000", 2024-09-13T11:24:53.425 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.178192884 2024-09-13T11:24:53.425 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:53.425 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:53.425 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:53.425 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:44.734915+0000", 2024-09-13T11:24:53.425 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.9970000000000001e-06 2024-09-13T11:24:53.425 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:53.425 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:53.425 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:53.425 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:44.734971+0000", 2024-09-13T11:24:53.425 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.6368e-05 2024-09-13T11:24:53.425 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:53.426 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:53.426 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:24:53.426 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:44.734990+0000", 2024-09-13T11:24:53.426 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.929e-05 2024-09-13T11:24:53.426 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:53.426 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:53.426 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:24:53.426 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:44.737096+0000", 2024-09-13T11:24:53.426 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0021053090000000001 2024-09-13T11:24:53.426 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:53.426 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:53.426 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:24:53.426 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:44.737178+0000", 2024-09-13T11:24:53.427 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.2255000000000004e-05 2024-09-13T11:24:53.427 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:53.427 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:53.427 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:53.427 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:53.427 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:53.427 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-09-13T11:24:53.429 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-13T11:24:53.429 INFO:tasks.ceph.osd.5.smithi092.stderr:2024-09-13T11:24:53.422+0000 7f6f0b705640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26346) UID: 0 2024-09-13T11:24:53.529 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-13T11:24:53.527+0000 7f6a68775640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26307) UID: 0 2024-09-13T11:24:53.575 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-09-13T11:24:53.575 INFO:teuthology.orchestra.run.smithi092.stdout: "ops": [ 2024-09-13T11:24:53.576 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:53.576 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:137418 3.28 3:153c13a1:::benchmark_data_smithi177_31116_object137417:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e139)", 2024-09-13T11:24:53.576 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:24:53.458684+0000", 2024-09-13T11:24:53.576 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 0.116437122, 2024-09-13T11:24:53.576 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.11646324800000001, 2024-09-13T11:24:53.576 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:24:53.576 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:24:53.576 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "waiting for sub ops", 2024-09-13T11:24:53.576 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:24:53.577 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:24:53.577 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:53.577 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 137418 2024-09-13T11:24:53.577 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:53.577 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:24:53.577 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:53.577 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:24:53.577 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:53.458684+0000", 2024-09-13T11:24:53.577 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:53.577 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:53.577 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:53.577 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:24:53.578 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:53.458684+0000", 2024-09-13T11:24:53.578 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:53.578 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:53.578 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:53.578 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:24:53.578 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:53.458687+0000", 2024-09-13T11:24:53.578 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.5849999999999998e-06 2024-09-13T11:24:53.578 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:53.578 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:53.578 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:24:53.578 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:53.458745+0000", 2024-09-13T11:24:53.578 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.7571999999999997e-05 2024-09-13T11:24:53.578 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:53.579 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:53.579 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:24:53.579 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:53.458755+0000", 2024-09-13T11:24:53.579 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.064e-05 2024-09-13T11:24:53.579 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:53.579 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:53.579 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:24:53.579 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:53.458762+0000", 2024-09-13T11:24:53.579 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.844e-06 2024-09-13T11:24:53.579 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:53.579 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:53.579 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:24:53.579 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:53.458824+0000", 2024-09-13T11:24:53.580 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.1566e-05 2024-09-13T11:24:53.580 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:53.580 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:53.580 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:24:53.580 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:53.458995+0000", 2024-09-13T11:24:53.580 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00017173399999999999 2024-09-13T11:24:53.580 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:53.580 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:53.580 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 5,8", 2024-09-13T11:24:53.580 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:53.459151+0000", 2024-09-13T11:24:53.580 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00015526 2024-09-13T11:24:53.580 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:53.580 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:53.581 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:24:53.581 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:53.460519+0000", 2024-09-13T11:24:53.581 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.001368661 2024-09-13T11:24:53.581 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:53.581 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:53.581 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:53.581 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:53.461806+0000", 2024-09-13T11:24:53.581 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.001286918 2024-09-13T11:24:53.581 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:53.581 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:24:53.581 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:53.581 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:53.581 INFO:teuthology.orchestra.run.smithi092.stdout: ], 2024-09-13T11:24:53.581 INFO:teuthology.orchestra.run.smithi092.stdout: "num_ops": 1 2024-09-13T11:24:53.582 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-09-13T11:24:53.593 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-13T11:24:53.630 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-09-13T11:24:53.628+0000 7f97befc9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26315) UID: 0 2024-09-13T11:24:53.731 INFO:tasks.ceph.osd.5.smithi092.stderr:2024-09-13T11:24:53.726+0000 7f6f0b705640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26346) UID: 0 2024-09-13T11:24:53.738 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-09-13T11:24:53.738 INFO:teuthology.orchestra.run.smithi092.stdout: "ops": [], 2024-09-13T11:24:53.739 INFO:teuthology.orchestra.run.smithi092.stdout: "complaint_time": 30, 2024-09-13T11:24:53.739 INFO:teuthology.orchestra.run.smithi092.stdout: "num_blocked_ops": 0 2024-09-13T11:24:53.739 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-09-13T11:24:53.759 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-13T11:24:53.832 INFO:tasks.ceph.osd.15.smithi177.stderr:2024-09-13T11:24:53.830+0000 7f9d6ce77640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26578) UID: 0 2024-09-13T11:24:53.932 INFO:tasks.ceph.osd.4.smithi001.stderr:2024-09-13T11:24:53.931+0000 7ff8e0218640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26310) UID: 0 2024-09-13T11:24:53.959 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-09-13T11:24:53.960 INFO:teuthology.orchestra.run.smithi092.stdout: "size": 20, 2024-09-13T11:24:53.960 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 600, 2024-09-13T11:24:53.960 INFO:teuthology.orchestra.run.smithi092.stdout: "ops": [ 2024-09-13T11:24:53.960 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:53.960 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:3 3.4 3.125e09e4 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e40)", 2024-09-13T11:24:53.960 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:46.304641+0000", 2024-09-13T11:24:53.960 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 127.652608394, 2024-09-13T11:24:53.960 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.374372119, 2024-09-13T11:24:53.960 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:24:53.960 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:24:53.961 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "reached pg", 2024-09-13T11:24:53.961 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:24:53.961 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:24:53.961 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:53.961 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 3 2024-09-13T11:24:53.961 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:53.961 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:24:53.961 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:53.961 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:24:53.961 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.304641+0000", 2024-09-13T11:24:53.961 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:53.961 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:53.962 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:53.962 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:24:53.962 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.304641+0000", 2024-09-13T11:24:53.962 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:53.962 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:53.962 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:53.962 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:24:53.962 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.304644+0000", 2024-09-13T11:24:53.962 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.5130000000000001e-06 2024-09-13T11:24:53.962 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:53.962 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:53.962 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:24:53.962 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305058+0000", 2024-09-13T11:24:53.963 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000414277 2024-09-13T11:24:53.963 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:53.963 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:53.963 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:24:53.963 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305124+0000", 2024-09-13T11:24:53.963 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.5813000000000002e-05 2024-09-13T11:24:53.963 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:53.963 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:53.963 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:24:53.963 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305156+0000", 2024-09-13T11:24:53.963 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.1489000000000003e-05 2024-09-13T11:24:53.963 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:53.964 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:53.964 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:24:53.964 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.678939+0000", 2024-09-13T11:24:53.964 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.37378339999999999 2024-09-13T11:24:53.964 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:53.964 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:53.964 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:24:53.964 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.679013+0000", 2024-09-13T11:24:53.964 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.3627000000000005e-05 2024-09-13T11:24:53.964 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:53.964 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:24:53.965 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:53.965 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:53.965 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:53.965 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:29 3.4 3.b7bbfbe4 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-09-13T11:24:53.965 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:46.305215+0000", 2024-09-13T11:24:53.965 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 127.652034091, 2024-09-13T11:24:53.965 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.37387964600000001, 2024-09-13T11:24:53.965 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:24:53.965 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:24:53.965 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "reached pg", 2024-09-13T11:24:53.965 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:24:53.965 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:24:53.965 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:53.966 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 29 2024-09-13T11:24:53.966 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:53.966 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:24:53.966 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:53.966 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:24:53.966 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305215+0000", 2024-09-13T11:24:53.966 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:53.966 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:53.966 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:53.966 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:24:53.966 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305215+0000", 2024-09-13T11:24:53.966 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:53.966 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:53.967 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:53.967 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:24:53.967 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305218+0000", 2024-09-13T11:24:53.967 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.7030000000000002e-06 2024-09-13T11:24:53.967 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:53.967 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:53.967 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:24:53.967 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305366+0000", 2024-09-13T11:24:53.967 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00014825400000000001 2024-09-13T11:24:53.967 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:53.967 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:53.967 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:24:53.967 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305386+0000", 2024-09-13T11:24:53.968 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.9689000000000001e-05 2024-09-13T11:24:53.968 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:53.968 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:53.968 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:24:53.968 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305403+0000", 2024-09-13T11:24:53.968 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.7600000000000001e-05 2024-09-13T11:24:53.968 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:53.968 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:53.968 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:24:53.968 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.679067+0000", 2024-09-13T11:24:53.968 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.37366339100000001 2024-09-13T11:24:53.969 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:53.969 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:53.969 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:24:53.969 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.679095+0000", 2024-09-13T11:24:53.969 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.8008999999999999e-05 2024-09-13T11:24:53.969 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:53.970 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:24:53.970 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:53.970 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:53.970 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:53.970 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:30 3.4 3.547ec0f4 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-09-13T11:24:53.971 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:46.305453+0000", 2024-09-13T11:24:53.971 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 127.651795967, 2024-09-13T11:24:53.971 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.373764914, 2024-09-13T11:24:53.971 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:24:53.971 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:24:53.971 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "reached pg", 2024-09-13T11:24:53.971 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:24:53.972 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:24:53.972 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:53.972 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 30 2024-09-13T11:24:53.972 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:53.972 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:24:53.972 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:53.972 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:24:53.973 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305453+0000", 2024-09-13T11:24:53.973 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:53.973 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:53.973 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:53.973 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:24:53.973 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305453+0000", 2024-09-13T11:24:53.973 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:53.973 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:53.974 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:53.974 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:24:53.974 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305456+0000", 2024-09-13T11:24:53.974 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.9560000000000002e-06 2024-09-13T11:24:53.974 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:53.974 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:53.974 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:24:53.975 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305597+0000", 2024-09-13T11:24:53.975 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00014122700000000001 2024-09-13T11:24:53.975 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:53.975 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:53.975 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:24:53.975 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305615+0000", 2024-09-13T11:24:53.976 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.8139999999999999e-05 2024-09-13T11:24:53.976 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:53.976 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:53.976 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:24:53.976 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305632+0000", 2024-09-13T11:24:53.976 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.6101999999999999e-05 2024-09-13T11:24:53.977 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:53.977 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:53.977 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:24:53.977 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.679195+0000", 2024-09-13T11:24:53.977 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.373563486 2024-09-13T11:24:53.977 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:53.978 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:53.978 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:24:53.978 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.679218+0000", 2024-09-13T11:24:53.978 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.3003e-05 2024-09-13T11:24:53.978 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:53.978 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:24:53.978 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:53.978 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:53.978 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:53.979 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:70 3.4 3.1404a604 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-09-13T11:24:53.979 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:46.305665+0000", 2024-09-13T11:24:53.979 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 127.651583819, 2024-09-13T11:24:53.979 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.37365550199999997, 2024-09-13T11:24:53.979 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:24:53.979 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:24:53.979 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "reached pg", 2024-09-13T11:24:53.979 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:24:53.979 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:24:53.979 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:53.979 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 70 2024-09-13T11:24:53.979 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:53.979 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:24:53.979 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:53.980 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:24:53.980 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305665+0000", 2024-09-13T11:24:53.980 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:53.980 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:53.980 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:53.980 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:24:53.980 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305665+0000", 2024-09-13T11:24:53.980 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:53.980 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:53.980 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:53.980 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:24:53.980 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305668+0000", 2024-09-13T11:24:53.980 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.5890000000000001e-06 2024-09-13T11:24:53.981 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:53.981 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:53.981 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:24:53.981 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305791+0000", 2024-09-13T11:24:53.981 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000123545 2024-09-13T11:24:53.981 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:53.981 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:53.981 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:24:53.981 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305808+0000", 2024-09-13T11:24:53.981 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.668e-05 2024-09-13T11:24:53.981 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:53.981 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:53.981 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:24:53.981 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305824+0000", 2024-09-13T11:24:53.982 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.6385e-05 2024-09-13T11:24:53.982 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:53.982 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:53.982 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:24:53.982 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.679305+0000", 2024-09-13T11:24:53.982 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.37348059900000002 2024-09-13T11:24:53.982 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:53.982 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:53.982 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:24:53.982 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.679321+0000", 2024-09-13T11:24:53.982 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.5704000000000001e-05 2024-09-13T11:24:53.982 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:53.982 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:24:53.982 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:53.983 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:53.983 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:53.983 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:25408 3.4 3:294e23bd:::benchmark_data_smithi177_31116_object25407:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e56)", 2024-09-13T11:24:53.983 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:04.495455+0000", 2024-09-13T11:24:53.983 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 109.46179416699999, 2024-09-13T11:24:53.983 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.29897367600000002, 2024-09-13T11:24:53.983 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:24:53.983 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:24:53.983 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:53.983 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:24:53.983 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:24:53.983 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:53.983 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 25408 2024-09-13T11:24:53.984 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:53.984 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:24:53.984 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:53.984 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:24:53.984 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.495455+0000", 2024-09-13T11:24:53.984 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:53.984 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:53.984 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:53.984 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:24:53.984 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.495455+0000", 2024-09-13T11:24:53.984 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:53.984 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:53.984 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:53.984 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:24:53.985 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.495459+0000", 2024-09-13T11:24:53.985 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.5630000000000002e-06 2024-09-13T11:24:53.985 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:53.985 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:53.985 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:24:53.985 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.495545+0000", 2024-09-13T11:24:53.985 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.5704000000000005e-05 2024-09-13T11:24:53.985 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:53.985 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:53.985 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:24:53.985 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.495581+0000", 2024-09-13T11:24:53.985 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.5689000000000003e-05 2024-09-13T11:24:53.985 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:53.985 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:53.986 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:24:53.986 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.495597+0000", 2024-09-13T11:24:53.986 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.6053e-05 2024-09-13T11:24:53.986 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:53.986 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:53.986 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:24:53.986 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.495721+0000", 2024-09-13T11:24:53.986 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00012379099999999999 2024-09-13T11:24:53.986 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:53.986 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:53.986 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:24:53.986 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.495940+0000", 2024-09-13T11:24:53.986 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00021967699999999999 2024-09-13T11:24:53.987 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:53.987 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:53.987 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 12", 2024-09-13T11:24:53.987 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.496162+0000", 2024-09-13T11:24:53.987 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000221917 2024-09-13T11:24:53.987 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:53.987 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:53.987 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:24:53.987 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.498699+0000", 2024-09-13T11:24:53.987 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0025366690000000001 2024-09-13T11:24:53.987 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:53.987 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:53.987 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:53.987 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.794139+0000", 2024-09-13T11:24:53.988 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.295440025 2024-09-13T11:24:53.988 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:53.988 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:53.988 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:24:53.988 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.794306+0000", 2024-09-13T11:24:53.988 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00016689799999999999 2024-09-13T11:24:53.988 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:53.988 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:53.988 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:24:53.988 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.794429+0000", 2024-09-13T11:24:53.988 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00012269000000000001 2024-09-13T11:24:53.988 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:53.988 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:24:53.989 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:53.989 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:53.989 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:53.989 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "MOSDPGPushReply(3.14 75/70 [PushReplyOp(3:2f14928a:::benchmark_data_smithi177_31116_object15123:head)])", 2024-09-13T11:24:53.989 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:31.569662+0000", 2024-09-13T11:24:53.989 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 82.387587394999997, 2024-09-13T11:24:53.989 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.35759148299999999, 2024-09-13T11:24:53.989 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:24:53.989 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:24:53.989 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "reached pg", 2024-09-13T11:24:53.989 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:24:53.989 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:53.989 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:24:53.990 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:31.569662+0000", 2024-09-13T11:24:53.990 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:53.990 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:53.990 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:53.990 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:24:53.990 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:31.569662+0000", 2024-09-13T11:24:53.990 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:53.990 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:53.990 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:53.990 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:24:53.990 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:31.569663+0000", 2024-09-13T11:24:53.990 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.333e-06 2024-09-13T11:24:53.990 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:53.991 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:53.991 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:24:53.991 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:31.569667+0000", 2024-09-13T11:24:53.991 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.8929999999999998e-06 2024-09-13T11:24:53.991 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:53.991 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:53.991 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:24:53.991 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:31.923221+0000", 2024-09-13T11:24:53.991 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.35355413699999999 2024-09-13T11:24:53.991 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:53.991 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:53.991 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:24:53.991 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:31.923230+0000", 2024-09-13T11:24:53.991 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.6829999999999997e-06 2024-09-13T11:24:53.992 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:53.992 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:53.992 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:24:53.992 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:31.923807+0000", 2024-09-13T11:24:53.992 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00057686800000000002 2024-09-13T11:24:53.992 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:53.992 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:53.992 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:24:53.992 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:31.927253+0000", 2024-09-13T11:24:53.992 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.003446569 2024-09-13T11:24:53.992 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:53.992 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:24:53.992 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:53.993 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:53.993 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:53.993 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "MOSDPGPushReply(3.18 85/70 [PushReplyOp(3:1e410119:::benchmark_data_smithi177_31116_object35700:head)])", 2024-09-13T11:24:53.993 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:42.216614+0000", 2024-09-13T11:24:53.993 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 71.740635050999998, 2024-09-13T11:24:53.993 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.54260090699999997, 2024-09-13T11:24:53.993 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:24:53.993 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:24:53.993 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "reached pg", 2024-09-13T11:24:53.993 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:24:53.993 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:53.993 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:24:53.993 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.216614+0000", 2024-09-13T11:24:53.993 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:53.994 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:53.994 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:53.994 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:24:53.994 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.216614+0000", 2024-09-13T11:24:53.994 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:53.994 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:53.994 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:53.994 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:24:53.994 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.216615+0000", 2024-09-13T11:24:53.994 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.0200000000000001e-07 2024-09-13T11:24:53.994 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:53.994 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:53.994 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:24:53.995 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.216619+0000", 2024-09-13T11:24:53.995 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.805e-06 2024-09-13T11:24:53.995 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:53.995 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:53.995 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:24:53.995 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.757476+0000", 2024-09-13T11:24:53.995 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.54085762000000004 2024-09-13T11:24:53.995 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:53.995 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:53.995 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:24:53.995 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.757498+0000", 2024-09-13T11:24:53.995 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.1936000000000001e-05 2024-09-13T11:24:53.995 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:53.995 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:53.996 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:24:53.996 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.757678+0000", 2024-09-13T11:24:53.996 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00018006599999999999 2024-09-13T11:24:53.996 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:53.996 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:53.996 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:24:53.996 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.759215+0000", 2024-09-13T11:24:53.996 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0015366780000000001 2024-09-13T11:24:53.996 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:53.996 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:24:53.996 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:53.996 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:53.997 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:53.997 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "MOSDPGPushReply(3.1b 85/78 [PushReplyOp(3:dabda1cc:::benchmark_data_smithi177_31116_object12988:head)])", 2024-09-13T11:24:53.997 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:42.485171+0000", 2024-09-13T11:24:53.997 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 71.472078143000005, 2024-09-13T11:24:53.997 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.308420427, 2024-09-13T11:24:53.997 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:24:53.997 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:24:53.997 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "reached pg", 2024-09-13T11:24:53.997 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:24:53.997 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:53.997 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:24:53.997 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.485171+0000", 2024-09-13T11:24:53.997 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:53.998 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:53.998 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:53.998 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:24:53.998 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.485171+0000", 2024-09-13T11:24:53.998 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:53.998 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:53.998 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:53.998 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:24:53.998 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.485173+0000", 2024-09-13T11:24:53.998 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.1569999999999998e-06 2024-09-13T11:24:53.998 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:53.998 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:53.998 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:24:53.998 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.485182+0000", 2024-09-13T11:24:53.999 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.6759999999999996e-06 2024-09-13T11:24:53.999 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:53.999 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:53.999 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:24:53.999 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.791336+0000", 2024-09-13T11:24:53.999 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.30615446499999999 2024-09-13T11:24:53.999 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:53.999 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:53.999 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:24:53.999 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.791369+0000", 2024-09-13T11:24:53.999 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.3155999999999999e-05 2024-09-13T11:24:53.999 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:53.999 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:54.000 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:24:54.000 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.791472+0000", 2024-09-13T11:24:54.000 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00010247900000000001 2024-09-13T11:24:54.000 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:54.000 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:54.000 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:24:54.000 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.793591+0000", 2024-09-13T11:24:54.000 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.002119494 2024-09-13T11:24:54.000 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:54.000 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:24:54.000 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:54.001 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:54.001 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:54.001 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:64066 3.1b 3:dd84e0ec:::benchmark_data_smithi177_31116_object64065:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89)", 2024-09-13T11:24:54.001 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:46.633608+0000", 2024-09-13T11:24:54.001 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 67.323640703999999, 2024-09-13T11:24:54.001 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.32564315399999999, 2024-09-13T11:24:54.001 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:24:54.001 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:24:54.001 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:54.001 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:24:54.001 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:24:54.001 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:54.001 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 64066 2024-09-13T11:24:54.002 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:54.002 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:24:54.002 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:54.002 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:24:54.002 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.633608+0000", 2024-09-13T11:24:54.002 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:54.002 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:54.002 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:54.002 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:24:54.002 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.633608+0000", 2024-09-13T11:24:54.002 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:54.002 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:54.003 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:54.003 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:24:54.003 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.633612+0000", 2024-09-13T11:24:54.003 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.856e-06 2024-09-13T11:24:54.003 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:54.003 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:54.003 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:24:54.003 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.633683+0000", 2024-09-13T11:24:54.003 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.1000999999999993e-05 2024-09-13T11:24:54.003 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:54.003 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:54.003 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:24:54.004 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.633711+0000", 2024-09-13T11:24:54.004 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.8204999999999998e-05 2024-09-13T11:24:54.004 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:54.004 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:54.004 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:24:54.004 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.633732+0000", 2024-09-13T11:24:54.004 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.0662999999999999e-05 2024-09-13T11:24:54.004 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:54.004 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:54.004 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:24:54.004 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.635573+0000", 2024-09-13T11:24:54.004 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.001841365 2024-09-13T11:24:54.004 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:54.005 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:54.005 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:24:54.005 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.635995+0000", 2024-09-13T11:24:54.005 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00042113699999999998 2024-09-13T11:24:54.005 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:54.005 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:54.005 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 12,15", 2024-09-13T11:24:54.005 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.636475+0000", 2024-09-13T11:24:54.005 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00048067699999999999 2024-09-13T11:24:54.005 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:54.005 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:54.005 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:54.006 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.640247+0000", 2024-09-13T11:24:54.006 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0037716450000000001 2024-09-13T11:24:54.006 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:54.006 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:54.006 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:24:54.006 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.649308+0000", 2024-09-13T11:24:54.006 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0090613240000000008 2024-09-13T11:24:54.006 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:54.006 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:54.006 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:54.006 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.956912+0000", 2024-09-13T11:24:54.006 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.30760338500000001 2024-09-13T11:24:54.006 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:54.007 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:54.007 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:24:54.007 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.957156+0000", 2024-09-13T11:24:54.007 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000244031 2024-09-13T11:24:54.007 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:54.007 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:54.007 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:24:54.007 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.959252+0000", 2024-09-13T11:24:54.007 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0020958650000000001 2024-09-13T11:24:54.007 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:54.007 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:24:54.007 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:54.007 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:54.007 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:54.008 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop_reply(client.4665.0:64066 3.1b e89/78 ondisk, result = 0)", 2024-09-13T11:24:54.008 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:46.651596+0000", 2024-09-13T11:24:54.008 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 67.305652873, 2024-09-13T11:24:54.008 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.30767612599999999, 2024-09-13T11:24:54.008 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:24:54.008 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:24:54.008 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-09-13T11:24:54.008 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:24:54.008 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:54.008 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:24:54.008 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.651596+0000", 2024-09-13T11:24:54.009 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:54.009 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:54.009 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:54.009 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:24:54.009 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.651596+0000", 2024-09-13T11:24:54.009 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:54.009 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:54.009 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:54.009 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:24:54.009 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.651600+0000", 2024-09-13T11:24:54.009 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.6969999999999999e-06 2024-09-13T11:24:54.009 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:54.009 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:54.010 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:24:54.010 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.651611+0000", 2024-09-13T11:24:54.010 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.1516e-05 2024-09-13T11:24:54.010 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:54.010 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:54.010 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:24:54.010 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.948118+0000", 2024-09-13T11:24:54.010 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.29650663500000002 2024-09-13T11:24:54.010 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:54.010 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:54.011 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:24:54.011 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.948137+0000", 2024-09-13T11:24:54.011 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.8529000000000002e-05 2024-09-13T11:24:54.011 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:54.011 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:54.011 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:24:54.011 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.956870+0000", 2024-09-13T11:24:54.011 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0087336380000000002 2024-09-13T11:24:54.011 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:54.012 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:54.012 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:24:54.012 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.956893+0000", 2024-09-13T11:24:54.012 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.317e-05 2024-09-13T11:24:54.012 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:54.012 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:54.012 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:24:54.012 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.959272+0000", 2024-09-13T11:24:54.012 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0023789409999999999 2024-09-13T11:24:54.013 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:54.013 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:24:54.013 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:54.013 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:54.013 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:54.013 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "MOSDPGPushReply(3.1b 91/78 [PushReplyOp(3:d9ac6889:::benchmark_data_smithi177_31116_object22254:head)])", 2024-09-13T11:24:54.013 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:48.392729+0000", 2024-09-13T11:24:54.013 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 65.564519958000005, 2024-09-13T11:24:54.013 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.30924993099999998, 2024-09-13T11:24:54.013 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:24:54.013 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:24:54.013 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "reached pg", 2024-09-13T11:24:54.014 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:24:54.014 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:54.014 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:24:54.014 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:48.392729+0000", 2024-09-13T11:24:54.014 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:54.014 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:54.014 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:54.014 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:24:54.014 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:48.392729+0000", 2024-09-13T11:24:54.014 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:54.014 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:54.014 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:54.014 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:24:54.014 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:48.392730+0000", 2024-09-13T11:24:54.015 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.336e-06 2024-09-13T11:24:54.015 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:54.015 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:54.015 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:24:54.015 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:48.392739+0000", 2024-09-13T11:24:54.015 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.8580000000000006e-06 2024-09-13T11:24:54.015 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:54.015 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:54.015 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:24:54.015 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:48.700180+0000", 2024-09-13T11:24:54.015 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.30744035600000003 2024-09-13T11:24:54.015 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:54.016 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:54.016 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:24:54.016 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:48.700197+0000", 2024-09-13T11:24:54.016 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.7649999999999999e-05 2024-09-13T11:24:54.016 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:54.016 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:54.016 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:24:54.016 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:48.700398+0000", 2024-09-13T11:24:54.016 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000201113 2024-09-13T11:24:54.016 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:54.016 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:54.016 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:24:54.016 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:48.701979+0000", 2024-09-13T11:24:54.017 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.001580618 2024-09-13T11:24:54.017 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:54.017 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:24:54.017 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:54.017 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:54.017 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:54.017 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop(client.4665.0:83193 3.8 e106/67 3:10066d83:::benchmark_data_smithi177_31116_object83192:head v 106'3925, mlcod=106'3922)", 2024-09-13T11:24:54.017 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:24:04.285068+0000", 2024-09-13T11:24:54.017 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 49.672181203000001, 2024-09-13T11:24:54.017 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.64647716600000005, 2024-09-13T11:24:54.017 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:24:54.017 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:24:54.017 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:54.017 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:24:54.018 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:54.018 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:24:54.018 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:04.285068+0000", 2024-09-13T11:24:54.018 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:54.018 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:54.018 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:54.018 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:24:54.018 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:04.285068+0000", 2024-09-13T11:24:54.018 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:54.018 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:54.018 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:54.018 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:24:54.018 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:04.285071+0000", 2024-09-13T11:24:54.019 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.729e-06 2024-09-13T11:24:54.019 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:54.019 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:54.019 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:24:54.019 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:04.285215+0000", 2024-09-13T11:24:54.019 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00014410000000000001 2024-09-13T11:24:54.019 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:54.019 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:54.019 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:24:54.019 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:04.928221+0000", 2024-09-13T11:24:54.019 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.64300653299999999 2024-09-13T11:24:54.019 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:54.019 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:54.019 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:24:54.020 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:04.928240+0000", 2024-09-13T11:24:54.020 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.8678999999999999e-05 2024-09-13T11:24:54.020 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:54.020 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:54.020 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:24:54.020 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:04.928388+0000", 2024-09-13T11:24:54.020 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00014809 2024-09-13T11:24:54.020 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:54.020 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:54.020 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:24:54.020 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:04.928428+0000", 2024-09-13T11:24:54.020 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.9776e-05 2024-09-13T11:24:54.020 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:54.021 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:54.021 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:24:54.021 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:04.931463+0000", 2024-09-13T11:24:54.021 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.003035168 2024-09-13T11:24:54.021 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:54.021 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:54.021 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:24:54.021 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:04.931545+0000", 2024-09-13T11:24:54.021 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.2090999999999997e-05 2024-09-13T11:24:54.021 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:54.021 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:24:54.021 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:54.021 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:54.021 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:54.024 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop(client.4665.0:83239 3.e e106/75 3:74547f17:::benchmark_data_smithi177_31116_object83238:head v 106'4391, mlcod=106'4388)", 2024-09-13T11:24:54.024 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:24:04.581196+0000", 2024-09-13T11:24:54.024 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 49.376052696999999, 2024-09-13T11:24:54.024 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.33617203600000001, 2024-09-13T11:24:54.024 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:24:54.024 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:24:54.024 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:54.025 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:24:54.025 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:54.025 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:24:54.025 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:04.581196+0000", 2024-09-13T11:24:54.025 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:54.025 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:54.025 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:54.025 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:24:54.025 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:04.581196+0000", 2024-09-13T11:24:54.025 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:54.025 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:54.025 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:54.025 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:24:54.026 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:04.581200+0000", 2024-09-13T11:24:54.026 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.19e-06 2024-09-13T11:24:54.026 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:54.026 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:54.026 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:24:54.026 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:04.581257+0000", 2024-09-13T11:24:54.026 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.7531999999999997e-05 2024-09-13T11:24:54.026 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:54.026 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:54.026 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:24:54.026 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:04.914011+0000", 2024-09-13T11:24:54.026 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.33275413700000001 2024-09-13T11:24:54.026 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:54.027 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:54.027 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:24:54.027 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:04.914028+0000", 2024-09-13T11:24:54.027 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.7082999999999999e-05 2024-09-13T11:24:54.027 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:54.027 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:54.027 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:24:54.027 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:04.914151+0000", 2024-09-13T11:24:54.027 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00012283300000000001 2024-09-13T11:24:54.027 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:54.027 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:54.027 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:24:54.027 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:04.914185+0000", 2024-09-13T11:24:54.027 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.4255999999999999e-05 2024-09-13T11:24:54.028 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:54.028 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:54.028 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:24:54.028 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:04.917300+0000", 2024-09-13T11:24:54.028 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0031141599999999999 2024-09-13T11:24:54.028 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:54.028 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:54.028 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:24:54.028 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:04.917368+0000", 2024-09-13T11:24:54.028 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.8844999999999996e-05 2024-09-13T11:24:54.028 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:54.028 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:24:54.029 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:54.029 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:54.029 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:54.029 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "MOSDPGPushReply(3.28 121/120 [PushReplyOp(3:145f409c:::benchmark_data_smithi177_31116_object1242:head)])", 2024-09-13T11:24:54.029 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:24:27.450312+0000", 2024-09-13T11:24:54.029 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 26.506937566000001, 2024-09-13T11:24:54.029 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.27862834600000003, 2024-09-13T11:24:54.029 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:24:54.029 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:24:54.029 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "reached pg", 2024-09-13T11:24:54.029 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:24:54.029 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:54.029 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:24:54.030 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:27.450312+0000", 2024-09-13T11:24:54.030 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:54.030 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:54.030 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:54.030 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:24:54.030 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:27.450312+0000", 2024-09-13T11:24:54.030 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:54.030 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:54.030 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:54.030 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:24:54.030 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:27.450313+0000", 2024-09-13T11:24:54.030 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.372e-06 2024-09-13T11:24:54.030 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:54.031 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:54.031 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:24:54.031 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:27.450321+0000", 2024-09-13T11:24:54.031 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.089e-06 2024-09-13T11:24:54.031 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:54.031 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:54.031 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:24:54.031 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:27.726786+0000", 2024-09-13T11:24:54.031 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.27646454399999998 2024-09-13T11:24:54.031 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:54.031 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:54.031 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:24:54.031 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:27.726803+0000", 2024-09-13T11:24:54.031 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.7283e-05 2024-09-13T11:24:54.032 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:54.032 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:54.032 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:24:54.032 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:27.726977+0000", 2024-09-13T11:24:54.032 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00017421200000000001 2024-09-13T11:24:54.032 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:54.032 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:54.032 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:24:54.032 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:27.728940+0000", 2024-09-13T11:24:54.032 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0019628459999999999 2024-09-13T11:24:54.032 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:54.032 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:24:54.032 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:54.033 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:54.033 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:54.033 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:126432 3.28 3:172ceb4d:::benchmark_data_smithi177_31116_object126431:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e130)", 2024-09-13T11:24:54.033 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:24:43.334456+0000", 2024-09-13T11:24:54.033 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 10.622793498, 2024-09-13T11:24:54.033 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.27433384799999999, 2024-09-13T11:24:54.033 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:24:54.033 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:24:54.033 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:54.033 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:24:54.033 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:24:54.033 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:54.033 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 126432 2024-09-13T11:24:54.034 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:54.034 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:24:54.034 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:54.034 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:24:54.034 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:43.334456+0000", 2024-09-13T11:24:54.034 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:54.034 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:54.034 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:54.034 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:24:54.034 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:43.334456+0000", 2024-09-13T11:24:54.034 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:54.034 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:54.034 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:54.035 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:24:54.035 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:43.334458+0000", 2024-09-13T11:24:54.035 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.1459999999999999e-06 2024-09-13T11:24:54.035 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:54.035 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:54.035 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:24:54.035 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:43.334518+0000", 2024-09-13T11:24:54.035 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.0418000000000003e-05 2024-09-13T11:24:54.035 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:54.035 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:54.035 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:24:54.035 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:43.334527+0000", 2024-09-13T11:24:54.035 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.4940000000000003e-06 2024-09-13T11:24:54.036 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:54.036 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:54.036 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:24:54.036 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:43.334532+0000", 2024-09-13T11:24:54.036 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.6509999999999998e-06 2024-09-13T11:24:54.036 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:54.036 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:54.036 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:24:54.036 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:43.334587+0000", 2024-09-13T11:24:54.036 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.4598000000000001e-05 2024-09-13T11:24:54.036 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:54.036 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:54.036 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:24:54.037 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:43.334751+0000", 2024-09-13T11:24:54.037 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00016452200000000001 2024-09-13T11:24:54.037 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:54.037 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:54.037 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 5,8", 2024-09-13T11:24:54.037 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:43.334918+0000", 2024-09-13T11:24:54.037 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00016641599999999999 2024-09-13T11:24:54.037 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:54.037 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:54.037 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:54.037 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:43.337338+0000", 2024-09-13T11:24:54.037 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0024197749999999999 2024-09-13T11:24:54.037 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:54.038 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:54.038 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:24:54.038 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:43.338831+0000", 2024-09-13T11:24:54.038 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0014936940000000001 2024-09-13T11:24:54.038 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:54.038 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:54.038 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:54.038 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:43.608021+0000", 2024-09-13T11:24:54.038 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.269189709 2024-09-13T11:24:54.038 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:54.038 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:54.038 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:24:54.038 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:43.608087+0000", 2024-09-13T11:24:54.038 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.6421999999999995e-05 2024-09-13T11:24:54.039 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:54.039 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:54.039 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:24:54.039 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:43.608789+0000", 2024-09-13T11:24:54.039 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00070200299999999998 2024-09-13T11:24:54.039 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:54.039 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:24:54.039 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:54.039 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:54.039 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:54.039 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop_reply(client.4665.0:126432 3.28 e130/120 ondisk, result = 0)", 2024-09-13T11:24:54.039 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:24:43.337816+0000", 2024-09-13T11:24:54.039 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 10.619432814, 2024-09-13T11:24:54.039 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.27098083499999998, 2024-09-13T11:24:54.040 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:24:54.040 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:24:54.040 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-09-13T11:24:54.040 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:24:54.040 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:54.040 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:24:54.040 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:43.337816+0000", 2024-09-13T11:24:54.040 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:54.040 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:54.040 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:54.040 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:24:54.040 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:43.337816+0000", 2024-09-13T11:24:54.040 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:54.041 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:54.041 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:54.041 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:24:54.041 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:43.337818+0000", 2024-09-13T11:24:54.041 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.3239999999999999e-06 2024-09-13T11:24:54.041 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:54.041 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:54.041 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:24:54.041 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:43.337821+0000", 2024-09-13T11:24:54.041 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.8709999999999999e-06 2024-09-13T11:24:54.041 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:54.041 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:54.041 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:24:54.042 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:43.607950+0000", 2024-09-13T11:24:54.042 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.270128916 2024-09-13T11:24:54.042 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:54.042 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:54.042 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:24:54.042 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:43.607954+0000", 2024-09-13T11:24:54.042 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.0300000000000004e-06 2024-09-13T11:24:54.042 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:54.042 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:54.042 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:24:54.042 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:43.608002+0000", 2024-09-13T11:24:54.042 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.7757999999999999e-05 2024-09-13T11:24:54.042 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:54.042 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:54.043 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:24:54.043 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:43.608011+0000", 2024-09-13T11:24:54.043 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.9239999999999996e-06 2024-09-13T11:24:54.043 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:54.043 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:54.043 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:24:54.043 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:43.608797+0000", 2024-09-13T11:24:54.043 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000786012 2024-09-13T11:24:54.043 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:54.043 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:24:54.043 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:54.043 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:54.043 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:54.044 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:129876 3.28 3:173a39a3:::benchmark_data_smithi177_31116_object129875:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e134)", 2024-09-13T11:24:54.044 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:24:47.116837+0000", 2024-09-13T11:24:54.044 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 6.840411778, 2024-09-13T11:24:54.044 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.282398655, 2024-09-13T11:24:54.045 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:24:54.045 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:24:54.045 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:54.045 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:24:54.045 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:24:54.045 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:54.045 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 129876 2024-09-13T11:24:54.045 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:54.045 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:24:54.045 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:54.045 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:24:54.046 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:47.116837+0000", 2024-09-13T11:24:54.046 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:54.046 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:54.046 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:54.046 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:24:54.046 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:47.116837+0000", 2024-09-13T11:24:54.046 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:54.046 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:54.046 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:54.046 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:24:54.046 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:47.116841+0000", 2024-09-13T11:24:54.046 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.2370000000000001e-06 2024-09-13T11:24:54.047 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:54.047 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:54.047 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:24:54.047 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:47.116901+0000", 2024-09-13T11:24:54.047 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.0409999999999999e-05 2024-09-13T11:24:54.047 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:54.047 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:54.047 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:24:54.047 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:47.116920+0000", 2024-09-13T11:24:54.047 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.9477e-05 2024-09-13T11:24:54.047 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:54.047 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:54.047 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:24:54.047 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:47.116931+0000", 2024-09-13T11:24:54.048 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0801e-05 2024-09-13T11:24:54.048 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:54.048 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:54.048 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:24:54.048 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:47.116991+0000", 2024-09-13T11:24:54.048 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.9596000000000003e-05 2024-09-13T11:24:54.048 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:54.048 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:54.048 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:24:54.048 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:47.117123+0000", 2024-09-13T11:24:54.048 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000132342 2024-09-13T11:24:54.048 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:54.048 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:54.048 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 5,8", 2024-09-13T11:24:54.049 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:47.117248+0000", 2024-09-13T11:24:54.049 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00012444999999999999 2024-09-13T11:24:54.049 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:54.049 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:54.049 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:24:54.049 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:47.118577+0000", 2024-09-13T11:24:54.049 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.001329373 2024-09-13T11:24:54.049 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:54.049 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:54.049 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:54.049 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:47.119544+0000", 2024-09-13T11:24:54.049 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00096691099999999999 2024-09-13T11:24:54.049 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:54.049 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:54.050 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:54.050 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:47.398345+0000", 2024-09-13T11:24:54.050 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.27880143600000001 2024-09-13T11:24:54.050 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:54.050 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:54.050 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:24:54.050 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:47.398419+0000", 2024-09-13T11:24:54.050 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.3697999999999998e-05 2024-09-13T11:24:54.050 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:54.050 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:54.050 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:24:54.050 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:47.399236+0000", 2024-09-13T11:24:54.051 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00081692400000000001 2024-09-13T11:24:54.051 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:54.051 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:24:54.051 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:54.051 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:54.051 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:54.051 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop_reply(client.4665.0:129876 3.28 e134/120 ondisk, result = 0)", 2024-09-13T11:24:54.051 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:24:47.119458+0000", 2024-09-13T11:24:54.051 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 6.8377910530000001, 2024-09-13T11:24:54.052 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.27979297199999997, 2024-09-13T11:24:54.052 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:24:54.052 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:24:54.052 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-09-13T11:24:54.052 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:24:54.052 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:54.052 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:24:54.052 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:47.119458+0000", 2024-09-13T11:24:54.053 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:54.053 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:54.053 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:54.053 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:24:54.053 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:47.119458+0000", 2024-09-13T11:24:54.053 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:54.053 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:54.053 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:54.053 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:24:54.053 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:47.119459+0000", 2024-09-13T11:24:54.053 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0529999999999999e-06 2024-09-13T11:24:54.053 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:54.054 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:54.054 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:24:54.054 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:47.119463+0000", 2024-09-13T11:24:54.054 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.1049999999999997e-06 2024-09-13T11:24:54.054 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:54.054 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:54.054 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:24:54.054 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:47.398252+0000", 2024-09-13T11:24:54.054 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.27878866800000002 2024-09-13T11:24:54.054 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:54.054 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:54.055 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:24:54.055 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:47.398260+0000", 2024-09-13T11:24:54.055 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.5329999999999994e-06 2024-09-13T11:24:54.055 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:54.055 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:54.055 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:24:54.055 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:47.398323+0000", 2024-09-13T11:24:54.055 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.2880999999999996e-05 2024-09-13T11:24:54.055 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:54.055 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:54.055 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:24:54.055 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:47.398336+0000", 2024-09-13T11:24:54.055 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.277e-05 2024-09-13T11:24:54.055 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:54.056 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:54.056 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:24:54.056 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:47.399251+0000", 2024-09-13T11:24:54.056 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00091496199999999996 2024-09-13T11:24:54.056 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:54.056 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:24:54.056 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:54.056 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:54.056 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:54.056 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:131687 3.28 3:14f65078:::benchmark_data_smithi177_31116_object131686:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e135)", 2024-09-13T11:24:54.057 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:24:48.925565+0000", 2024-09-13T11:24:54.057 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 5.0316837000000003, 2024-09-13T11:24:54.057 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.41421604099999998, 2024-09-13T11:24:54.057 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:24:54.057 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:24:54.057 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:54.057 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:24:54.057 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:24:54.057 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:54.057 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 131687 2024-09-13T11:24:54.057 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:54.057 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:24:54.057 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:54.058 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:24:54.058 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:48.925565+0000", 2024-09-13T11:24:54.058 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:54.058 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:54.058 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:54.058 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:24:54.058 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:48.925565+0000", 2024-09-13T11:24:54.058 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:54.058 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:54.058 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:54.058 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:24:54.058 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:48.925568+0000", 2024-09-13T11:24:54.058 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.441e-06 2024-09-13T11:24:54.058 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:54.059 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:54.059 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:24:54.059 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:48.925617+0000", 2024-09-13T11:24:54.059 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.8739000000000002e-05 2024-09-13T11:24:54.059 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:54.059 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:54.059 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:24:54.059 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:48.925631+0000", 2024-09-13T11:24:54.059 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.4463e-05 2024-09-13T11:24:54.059 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:54.059 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:54.059 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:24:54.059 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:48.925638+0000", 2024-09-13T11:24:54.060 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.2119999999999997e-06 2024-09-13T11:24:54.060 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:54.060 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:54.060 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:24:54.060 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:48.925703+0000", 2024-09-13T11:24:54.060 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.4794000000000003e-05 2024-09-13T11:24:54.060 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:54.060 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:54.060 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:24:54.060 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:48.925857+0000", 2024-09-13T11:24:54.060 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00015427 2024-09-13T11:24:54.060 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:54.061 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:54.061 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 5,8", 2024-09-13T11:24:54.061 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:48.925995+0000", 2024-09-13T11:24:54.061 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000137365 2024-09-13T11:24:54.061 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:54.061 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:54.061 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:24:54.061 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:48.927261+0000", 2024-09-13T11:24:54.061 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.001266021 2024-09-13T11:24:54.061 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:54.061 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:54.061 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:54.061 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:48.930015+0000", 2024-09-13T11:24:54.061 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0027545130000000001 2024-09-13T11:24:54.062 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:54.062 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:54.062 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:54.062 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:49.338898+0000", 2024-09-13T11:24:54.062 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.40888254299999999 2024-09-13T11:24:54.062 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:54.062 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:54.062 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:24:54.062 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:49.338967+0000", 2024-09-13T11:24:54.062 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.9402999999999999e-05 2024-09-13T11:24:54.062 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:54.062 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:54.063 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:24:54.063 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:49.339781+0000", 2024-09-13T11:24:54.063 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00081427699999999997 2024-09-13T11:24:54.063 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:54.063 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:24:54.063 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:54.063 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:54.063 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:54.064 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop_reply(client.4665.0:131687 3.28 e135/120 ondisk, result = 0)", 2024-09-13T11:24:54.064 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:24:48.928935+0000", 2024-09-13T11:24:54.064 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 5.0283143990000001, 2024-09-13T11:24:54.064 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.41085638499999999, 2024-09-13T11:24:54.064 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:24:54.064 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:24:54.064 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-09-13T11:24:54.064 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:24:54.064 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:54.064 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:24:54.064 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:48.928935+0000", 2024-09-13T11:24:54.065 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:54.065 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:54.065 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:54.065 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:24:54.065 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:48.928935+0000", 2024-09-13T11:24:54.065 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:24:54.065 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:54.065 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:54.065 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:24:54.065 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:48.928936+0000", 2024-09-13T11:24:54.065 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.519e-06 2024-09-13T11:24:54.065 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:54.066 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:54.066 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:24:54.066 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:48.928941+0000", 2024-09-13T11:24:54.066 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.0710000000000002e-06 2024-09-13T11:24:54.066 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:54.066 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:54.066 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:24:54.066 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:49.338799+0000", 2024-09-13T11:24:54.066 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.40985808499999998 2024-09-13T11:24:54.066 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:54.066 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:54.066 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:24:54.066 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:49.338809+0000", 2024-09-13T11:24:54.066 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.4390000000000006e-06 2024-09-13T11:24:54.067 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:54.067 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:54.067 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:24:54.067 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:49.338878+0000", 2024-09-13T11:24:54.067 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.9429000000000002e-05 2024-09-13T11:24:54.067 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:54.067 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:54.067 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:24:54.067 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:49.338890+0000", 2024-09-13T11:24:54.067 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.2217e-05 2024-09-13T11:24:54.067 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:24:54.067 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:24:54.067 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:24:54.067 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:49.339791+0000", 2024-09-13T11:24:54.068 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00090062499999999995 2024-09-13T11:24:54.068 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:54.068 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:24:54.068 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:54.068 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:24:54.068 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:24:54.068 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-09-13T11:24:54.069 INFO:tasks.radosbench.radosbench.0.smithi177.stdout: 147 16 138071 138055 58.6902 73.4375 0.00454606 0.0170329 2024-09-13T11:24:54.071 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-13T11:24:54.072 INFO:tasks.ceph.osd.2.smithi162.stderr:2024-09-13T11:24:54.068+0000 7fe92b35a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26313) UID: 0 2024-09-13T11:24:54.115 INFO:tasks.ceph.osd.14.smithi162.stderr:2024-09-13T11:24:54.112+0000 7f3df89477c0 -1 osd.14 120 log_to_monitors true 2024-09-13T11:24:54.171 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-09-13T11:24:54.168+0000 7f97befc9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26315) UID: 0 2024-09-13T11:24:54.217 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-13T11:24:54.217 INFO:teuthology.orchestra.run.smithi001.stdout: "ops": [ 2024-09-13T11:24:54.217 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:54.217 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:138264 3.15 3:ab988165:::benchmark_data_smithi177_31116_object138263:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e140)", 2024-09-13T11:24:54.217 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:24:54.213501+0000", 2024-09-13T11:24:54.217 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 0.00275778, 2024-09-13T11:24:54.217 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.002788825, 2024-09-13T11:24:54.217 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:24:54.218 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:24:54.218 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "waiting for sub ops", 2024-09-13T11:24:54.218 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:24:54.218 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:24:54.218 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:54.218 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 138264 2024-09-13T11:24:54.218 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:54.218 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:24:54.218 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:54.219 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:24:54.219 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:54.213501+0000", 2024-09-13T11:24:54.219 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:54.219 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:54.219 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:54.219 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:24:54.219 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:54.213501+0000", 2024-09-13T11:24:54.219 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:54.220 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:54.220 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:54.220 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:24:54.220 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:54.213507+0000", 2024-09-13T11:24:54.220 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.4160000000000003e-06 2024-09-13T11:24:54.220 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:54.220 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:54.221 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:24:54.221 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:54.213561+0000", 2024-09-13T11:24:54.221 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.4212000000000002e-05 2024-09-13T11:24:54.221 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:54.222 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:54.222 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:24:54.222 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:54.213570+0000", 2024-09-13T11:24:54.222 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.7680000000000003e-06 2024-09-13T11:24:54.222 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:54.222 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:54.222 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:24:54.222 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:54.213576+0000", 2024-09-13T11:24:54.223 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.5099999999999998e-06 2024-09-13T11:24:54.223 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:54.223 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:54.223 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:24:54.223 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:54.215119+0000", 2024-09-13T11:24:54.223 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.001543228 2024-09-13T11:24:54.224 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:54.224 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:54.224 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:24:54.224 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:54.215236+0000", 2024-09-13T11:24:54.224 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000116412 2024-09-13T11:24:54.224 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:54.224 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:54.224 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 3", 2024-09-13T11:24:54.224 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:54.215365+0000", 2024-09-13T11:24:54.224 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00012974500000000001 2024-09-13T11:24:54.224 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:54.225 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:24:54.225 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:54.225 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:54.225 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-09-13T11:24:54.225 INFO:teuthology.orchestra.run.smithi001.stdout: "num_ops": 1 2024-09-13T11:24:54.225 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-13T11:24:54.236 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-13T11:24:54.272 INFO:tasks.ceph.osd.10.smithi162.stderr:2024-09-13T11:24:54.268+0000 7fb59c9f0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26317) UID: 0 2024-09-13T11:24:54.368 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-13T11:24:54.368 INFO:teuthology.orchestra.run.smithi001.stdout: "ops": [], 2024-09-13T11:24:54.368 INFO:teuthology.orchestra.run.smithi001.stdout: "complaint_time": 30, 2024-09-13T11:24:54.368 INFO:teuthology.orchestra.run.smithi001.stdout: "num_blocked_ops": 0 2024-09-13T11:24:54.368 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-13T11:24:54.381 INFO:tasks.ceph.osd.13.smithi092.stderr:2024-09-13T11:24:54.366+0000 7f3e5afea640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26350) UID: 0 2024-09-13T11:24:54.383 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-13T11:24:54.473 INFO:tasks.ceph.osd.9.smithi092.stderr:2024-09-13T11:24:54.470+0000 7ff5ab3d0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26348) UID: 0 2024-09-13T11:24:54.518 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-13T11:24:54.518 INFO:teuthology.orchestra.run.smithi001.stdout: "size": 20, 2024-09-13T11:24:54.518 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 600, 2024-09-13T11:24:54.518 INFO:teuthology.orchestra.run.smithi001.stdout: "ops": [ 2024-09-13T11:24:54.518 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:54.518 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDPGPushReply(3.1e 84/78 [PushReplyOp(3:79ee54b9:::benchmark_data_smithi177_31116_object11701:head)])", 2024-09-13T11:24:54.518 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:41.407335+0000", 2024-09-13T11:24:54.518 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 73.108352733999993, 2024-09-13T11:24:54.519 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.51622350400000006, 2024-09-13T11:24:54.519 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:24:54.519 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:24:54.519 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-13T11:24:54.519 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:24:54.519 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:54.519 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:24:54.519 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:41.407335+0000", 2024-09-13T11:24:54.519 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:54.519 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:54.519 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:54.519 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:24:54.519 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:41.407335+0000", 2024-09-13T11:24:54.519 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:54.520 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:54.520 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:54.520 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:24:54.520 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:41.407337+0000", 2024-09-13T11:24:54.520 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.505e-06 2024-09-13T11:24:54.520 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:54.520 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:54.520 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:24:54.520 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:41.407343+0000", 2024-09-13T11:24:54.520 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.9089999999999997e-06 2024-09-13T11:24:54.520 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:54.520 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:54.520 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:24:54.521 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:41.921977+0000", 2024-09-13T11:24:54.521 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.51463463499999995 2024-09-13T11:24:54.521 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:54.521 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:54.521 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:24:54.521 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:41.921988+0000", 2024-09-13T11:24:54.521 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.092e-05 2024-09-13T11:24:54.521 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:54.521 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:54.521 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:24:54.521 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:41.922088+0000", 2024-09-13T11:24:54.521 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.9528e-05 2024-09-13T11:24:54.521 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:54.521 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:54.522 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:24:54.522 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:41.923559+0000", 2024-09-13T11:24:54.522 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0014710070000000001 2024-09-13T11:24:54.522 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:54.522 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:24:54.522 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:54.522 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:54.522 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:54.522 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4665.0:60417 3.10 e85/84 3:092808b0:::benchmark_data_smithi177_31116_object60416:head v 85'3295, mlcod=85'3292)", 2024-09-13T11:24:54.522 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:42.174488+0000", 2024-09-13T11:24:54.523 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 72.341200180000001, 2024-09-13T11:24:54.523 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.387908896, 2024-09-13T11:24:54.523 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:24:54.523 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:24:54.523 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:54.523 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:24:54.523 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:54.523 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:24:54.523 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.174488+0000", 2024-09-13T11:24:54.523 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:54.523 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:54.523 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:54.523 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:24:54.523 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.174488+0000", 2024-09-13T11:24:54.524 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:54.524 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:54.524 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:54.524 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:24:54.524 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.174490+0000", 2024-09-13T11:24:54.524 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.9099999999999999e-06 2024-09-13T11:24:54.524 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:54.524 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:54.524 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:24:54.524 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.174527+0000", 2024-09-13T11:24:54.524 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.7543000000000003e-05 2024-09-13T11:24:54.524 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:54.524 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:54.525 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:24:54.525 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.561093+0000", 2024-09-13T11:24:54.525 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.38656578400000002 2024-09-13T11:24:54.525 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:54.525 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:54.525 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:24:54.525 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.561101+0000", 2024-09-13T11:24:54.526 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.6090000000000003e-06 2024-09-13T11:24:54.526 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:54.526 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:54.526 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:24:54.526 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.561143+0000", 2024-09-13T11:24:54.526 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.2435000000000003e-05 2024-09-13T11:24:54.526 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:54.526 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:54.526 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:24:54.526 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.561164+0000", 2024-09-13T11:24:54.526 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.1276000000000001e-05 2024-09-13T11:24:54.526 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:54.526 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:54.527 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:24:54.527 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.562360+0000", 2024-09-13T11:24:54.527 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00119596 2024-09-13T11:24:54.527 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:54.527 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:54.527 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:24:54.527 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.562397+0000", 2024-09-13T11:24:54.527 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.6378999999999998e-05 2024-09-13T11:24:54.527 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:54.527 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:24:54.527 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:54.527 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:54.527 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:54.527 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDPGPush(3.22 85/78 [PushOp(3:456e8765:::benchmark_data_smithi177_31116_object16212:head, version: 52'1006, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:456e8765:::benchmark_data_smithi177_31116_object16212:head@52'1006, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:24:54.528 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:42.673067+0000", 2024-09-13T11:24:54.528 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 71.842621038000004, 2024-09-13T11:24:54.528 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.27935889899999999, 2024-09-13T11:24:54.528 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:24:54.528 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:24:54.528 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:24:54.528 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:24:54.528 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:54.528 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:24:54.528 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.673067+0000", 2024-09-13T11:24:54.528 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:54.528 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:54.528 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:54.528 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:24:54.529 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.673067+0000", 2024-09-13T11:24:54.529 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:54.529 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:54.529 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:54.529 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:24:54.529 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.673068+0000", 2024-09-13T11:24:54.529 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.9699999999999994e-07 2024-09-13T11:24:54.529 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:54.529 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:54.529 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:24:54.529 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.673090+0000", 2024-09-13T11:24:54.529 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.2416999999999999e-05 2024-09-13T11:24:54.529 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:54.529 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:54.530 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:24:54.530 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.951363+0000", 2024-09-13T11:24:54.530 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.27827291199999998 2024-09-13T11:24:54.530 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:54.530 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:54.530 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:24:54.530 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.951386+0000", 2024-09-13T11:24:54.530 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.2226999999999999e-05 2024-09-13T11:24:54.530 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:54.530 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:54.530 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:24:54.530 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.951581+0000", 2024-09-13T11:24:54.530 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000195713 2024-09-13T11:24:54.531 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:54.531 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:54.531 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:24:54.531 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.951605+0000", 2024-09-13T11:24:54.531 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.3767000000000001e-05 2024-09-13T11:24:54.531 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:54.531 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:54.531 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:24:54.531 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.952426+0000", 2024-09-13T11:24:54.531 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00082086599999999996 2024-09-13T11:24:54.531 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:54.531 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:24:54.531 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:54.531 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:54.532 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:54.532 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "replica_scrub(pg: 3.20,from:0'0,to:89'3254,epoch:89/77,start:3:04000000::::0,end:3:045ba805:::benchmark_data_smithi177_31116_object43251:0,chunky:1,deep:0,version:9,allow_preemption:1,priority=5)", 2024-09-13T11:24:54.532 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:46.641364+0000", 2024-09-13T11:24:54.532 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 67.874324357000006, 2024-09-13T11:24:54.532 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.27141393000000003, 2024-09-13T11:24:54.532 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:24:54.532 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:24:54.532 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:24:54.532 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:24:54.532 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:54.532 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:24:54.532 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.641364+0000", 2024-09-13T11:24:54.532 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:54.533 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:54.533 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:54.533 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:24:54.533 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.641364+0000", 2024-09-13T11:24:54.533 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:54.533 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:54.533 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:54.533 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:24:54.533 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.641365+0000", 2024-09-13T11:24:54.533 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.266e-06 2024-09-13T11:24:54.533 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:54.533 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:54.533 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:24:54.533 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.641370+0000", 2024-09-13T11:24:54.534 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.8539999999999996e-06 2024-09-13T11:24:54.534 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:54.534 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:54.534 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:24:54.534 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.912491+0000", 2024-09-13T11:24:54.534 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.27112103100000001 2024-09-13T11:24:54.534 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:54.534 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:54.534 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:24:54.534 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.912520+0000", 2024-09-13T11:24:54.534 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.9379000000000001e-05 2024-09-13T11:24:54.534 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:54.534 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:54.534 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:24:54.534 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.912679+0000", 2024-09-13T11:24:54.535 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00015915800000000001 2024-09-13T11:24:54.535 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:54.535 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:54.535 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:24:54.535 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.912714+0000", 2024-09-13T11:24:54.535 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.4687999999999997e-05 2024-09-13T11:24:54.535 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:54.535 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:54.535 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:24:54.535 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.912778+0000", 2024-09-13T11:24:54.536 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.3553999999999995e-05 2024-09-13T11:24:54.536 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:54.536 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:24:54.536 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:54.536 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:54.536 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:54.536 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDPGPush(3.18 89/70 [PushOp(3:1ac6c361:::benchmark_data_smithi177_31116_object44972:head, version: 68'2767, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:1ac6c361:::benchmark_data_smithi177_31116_object44972:head@68'2767, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:24:54.536 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:46.671018+0000", 2024-09-13T11:24:54.536 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 67.844670225000002, 2024-09-13T11:24:54.536 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.271282931, 2024-09-13T11:24:54.536 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:24:54.537 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:24:54.537 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:24:54.537 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:24:54.537 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:54.537 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:24:54.537 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.671018+0000", 2024-09-13T11:24:54.537 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:54.537 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:54.537 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:54.537 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:24:54.537 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.671018+0000", 2024-09-13T11:24:54.537 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:54.537 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:54.537 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:54.538 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:24:54.538 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.671020+0000", 2024-09-13T11:24:54.538 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.018e-06 2024-09-13T11:24:54.538 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:54.538 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:54.538 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:24:54.538 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.671097+0000", 2024-09-13T11:24:54.538 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.7218999999999994e-05 2024-09-13T11:24:54.538 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:54.538 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:54.538 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:24:54.538 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.939773+0000", 2024-09-13T11:24:54.538 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.26867550200000001 2024-09-13T11:24:54.538 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:54.538 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:54.539 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:24:54.539 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.939791+0000", 2024-09-13T11:24:54.539 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.8185999999999998e-05 2024-09-13T11:24:54.539 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:54.539 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:54.539 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:24:54.539 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.939926+0000", 2024-09-13T11:24:54.539 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000135291 2024-09-13T11:24:54.539 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:54.539 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:54.539 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:24:54.539 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.939938+0000", 2024-09-13T11:24:54.539 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2393e-05 2024-09-13T11:24:54.539 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:54.540 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:54.540 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:24:54.540 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.942301+0000", 2024-09-13T11:24:54.540 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.002362322 2024-09-13T11:24:54.540 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:54.540 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:24:54.540 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:54.540 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:54.540 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:54.540 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "replica_scrub(pg: 3.20,from:0'0,to:90'3264,epoch:90/77,start:3:0650aac8:::benchmark_data_smithi177_31116_object1734:0,end:3:06af01e3:::benchmark_data_smithi177_31116_object31572:0,chunky:1,deep:0,version:9,allow_preemption:1,priority=5)", 2024-09-13T11:24:54.540 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:47.380615+0000", 2024-09-13T11:24:54.540 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 67.135072801999996, 2024-09-13T11:24:54.540 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.51951398400000004, 2024-09-13T11:24:54.541 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:24:54.541 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:24:54.541 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:24:54.541 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:24:54.541 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:54.541 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:24:54.541 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:47.380615+0000", 2024-09-13T11:24:54.541 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:54.541 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:54.541 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:54.541 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:24:54.541 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:47.380615+0000", 2024-09-13T11:24:54.541 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:54.542 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:54.542 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:54.542 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:24:54.542 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:47.380617+0000", 2024-09-13T11:24:54.542 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.671e-06 2024-09-13T11:24:54.542 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:54.542 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:54.542 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:24:54.542 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:47.380623+0000", 2024-09-13T11:24:54.542 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.1190000000000002e-06 2024-09-13T11:24:54.542 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:54.542 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:54.542 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:24:54.542 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:47.900050+0000", 2024-09-13T11:24:54.542 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.51942737400000005 2024-09-13T11:24:54.543 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:54.543 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:54.543 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:24:54.543 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:47.900057+0000", 2024-09-13T11:24:54.543 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.4359999999999999e-06 2024-09-13T11:24:54.543 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:54.543 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:54.543 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:24:54.543 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:47.900101+0000", 2024-09-13T11:24:54.543 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.4675000000000002e-05 2024-09-13T11:24:54.543 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:54.543 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:54.543 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:24:54.543 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:47.900110+0000", 2024-09-13T11:24:54.544 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.0569999999999993e-06 2024-09-13T11:24:54.544 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:54.544 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:54.544 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:24:54.544 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:47.900129+0000", 2024-09-13T11:24:54.544 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.9652000000000001e-05 2024-09-13T11:24:54.544 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:54.544 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:24:54.544 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:54.544 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:54.544 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:54.544 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDPGPush(3.21 91/78 [PushOp(3:8492321f:::benchmark_data_smithi177_31116_object13767:head, version: 50'840, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:8492321f:::benchmark_data_smithi177_31116_object13767:head@50'840, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:24:54.544 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:48.319251+0000", 2024-09-13T11:24:54.545 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 66.196437273000001, 2024-09-13T11:24:54.545 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.60903802200000001, 2024-09-13T11:24:54.545 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:24:54.545 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:24:54.545 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:24:54.545 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:24:54.545 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:54.545 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:24:54.545 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:48.319251+0000", 2024-09-13T11:24:54.545 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:54.545 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:54.545 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:54.545 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:24:54.545 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:48.319251+0000", 2024-09-13T11:24:54.546 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:54.546 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:54.546 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:54.546 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:24:54.546 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:48.319252+0000", 2024-09-13T11:24:54.546 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2640000000000001e-06 2024-09-13T11:24:54.546 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:54.546 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:54.547 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:24:54.547 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:48.319363+0000", 2024-09-13T11:24:54.547 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000110559 2024-09-13T11:24:54.547 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:54.547 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:54.547 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:24:54.547 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:48.927115+0000", 2024-09-13T11:24:54.547 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.60775277800000005 2024-09-13T11:24:54.547 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:54.547 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:54.547 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:24:54.547 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:48.927144+0000", 2024-09-13T11:24:54.547 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.8861e-05 2024-09-13T11:24:54.548 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:54.548 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:54.548 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:24:54.548 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:48.927318+0000", 2024-09-13T11:24:54.548 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00017399000000000001 2024-09-13T11:24:54.548 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:54.548 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:54.548 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:24:54.548 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:48.927340+0000", 2024-09-13T11:24:54.549 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.1433999999999998e-05 2024-09-13T11:24:54.549 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:54.549 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:54.549 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:24:54.549 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:48.928289+0000", 2024-09-13T11:24:54.549 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00094913599999999995 2024-09-13T11:24:54.549 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:54.549 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:24:54.549 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:54.549 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:54.549 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:54.549 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:69487 3.e 3:73f7f990:::benchmark_data_smithi177_31116_object69486:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94)", 2024-09-13T11:24:54.549 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:52.063559+0000", 2024-09-13T11:24:54.550 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 62.452129239999998, 2024-09-13T11:24:54.550 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.20206200199999999, 2024-09-13T11:24:54.550 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:24:54.550 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:24:54.550 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:54.550 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:24:54.550 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:24:54.550 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:54.550 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 69487 2024-09-13T11:24:54.550 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:54.550 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:24:54.550 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:54.550 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:24:54.551 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:52.063559+0000", 2024-09-13T11:24:54.551 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:54.551 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:54.551 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:54.551 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:24:54.551 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:52.063559+0000", 2024-09-13T11:24:54.551 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:54.551 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:54.551 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:54.551 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:24:54.551 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:52.063560+0000", 2024-09-13T11:24:54.551 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.167e-06 2024-09-13T11:24:54.551 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:54.552 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:54.552 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:24:54.552 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:52.063612+0000", 2024-09-13T11:24:54.552 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.1709999999999998e-05 2024-09-13T11:24:54.552 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:54.552 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:54.552 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:24:54.552 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:52.063623+0000", 2024-09-13T11:24:54.552 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.1251e-05 2024-09-13T11:24:54.552 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:54.553 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:54.553 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:24:54.553 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:52.063631+0000", 2024-09-13T11:24:54.553 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.7319999999999998e-06 2024-09-13T11:24:54.553 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:54.553 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:54.553 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:24:54.553 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:52.064267+0000", 2024-09-13T11:24:54.553 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00063608600000000001 2024-09-13T11:24:54.553 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:54.553 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:54.553 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:24:54.554 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:52.064407+0000", 2024-09-13T11:24:54.554 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000140686 2024-09-13T11:24:54.554 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:54.554 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:54.554 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 1", 2024-09-13T11:24:54.554 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:52.064540+0000", 2024-09-13T11:24:54.554 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000132244 2024-09-13T11:24:54.554 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:54.554 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:54.554 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:24:54.554 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:52.067019+0000", 2024-09-13T11:24:54.554 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00247919 2024-09-13T11:24:54.554 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:54.554 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:54.555 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:54.555 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:52.265543+0000", 2024-09-13T11:24:54.555 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.19852367800000001 2024-09-13T11:24:54.555 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:54.555 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:54.555 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:24:54.555 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:52.265576+0000", 2024-09-13T11:24:54.555 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.3550999999999998e-05 2024-09-13T11:24:54.555 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:54.555 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:54.555 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:24:54.555 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:52.265621+0000", 2024-09-13T11:24:54.556 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.4706999999999997e-05 2024-09-13T11:24:54.556 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:54.556 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:24:54.556 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:54.556 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:54.556 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:54.556 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4665.0:69556 3.18 e94/91 3:1d3dbeba:::benchmark_data_smithi177_31116_object69555:head v 94'3570, mlcod=94'3566)", 2024-09-13T11:24:54.556 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:52.142942+0000", 2024-09-13T11:24:54.556 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 62.372745737999999, 2024-09-13T11:24:54.556 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.21178485399999999, 2024-09-13T11:24:54.557 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:24:54.557 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:24:54.557 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:54.557 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:24:54.557 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:54.557 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:24:54.557 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:52.142942+0000", 2024-09-13T11:24:54.557 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:54.557 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:54.558 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:54.558 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:24:54.558 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:52.142942+0000", 2024-09-13T11:24:54.558 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:54.558 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:54.558 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:54.558 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:24:54.558 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:52.142944+0000", 2024-09-13T11:24:54.558 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.4500000000000001e-06 2024-09-13T11:24:54.558 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:54.558 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:54.559 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:24:54.559 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:52.143047+0000", 2024-09-13T11:24:54.559 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000103384 2024-09-13T11:24:54.559 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:54.559 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:54.559 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:24:54.559 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:52.353322+0000", 2024-09-13T11:24:54.559 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.21027483299999999 2024-09-13T11:24:54.559 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:54.559 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:54.559 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:24:54.559 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:52.353330+0000", 2024-09-13T11:24:54.559 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.8879999999999997e-06 2024-09-13T11:24:54.560 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:54.560 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:54.560 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:24:54.560 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:52.353402+0000", 2024-09-13T11:24:54.560 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.2155000000000003e-05 2024-09-13T11:24:54.560 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:54.560 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:54.560 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:24:54.560 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:52.353426+0000", 2024-09-13T11:24:54.560 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.3774e-05 2024-09-13T11:24:54.560 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:54.560 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:54.560 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:24:54.560 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:52.354674+0000", 2024-09-13T11:24:54.561 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.001248706 2024-09-13T11:24:54.561 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:54.561 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:54.561 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:24:54.561 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:52.354727+0000", 2024-09-13T11:24:54.561 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.2664000000000003e-05 2024-09-13T11:24:54.561 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:54.561 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:24:54.561 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:54.561 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:54.561 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:54.562 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:70749 3.1e 3:7bd7bfc6:::benchmark_data_smithi177_31116_object70748:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95)", 2024-09-13T11:24:54.562 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:53.156250+0000", 2024-09-13T11:24:54.562 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 61.359437917000001, 2024-09-13T11:24:54.562 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.22394481799999999, 2024-09-13T11:24:54.562 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:24:54.562 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:24:54.562 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:54.562 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:24:54.562 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:24:54.562 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:54.562 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 70749 2024-09-13T11:24:54.562 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:54.562 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:24:54.562 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:54.563 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:24:54.563 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:53.156250+0000", 2024-09-13T11:24:54.563 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:54.563 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:54.563 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:54.563 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:24:54.563 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:53.156250+0000", 2024-09-13T11:24:54.563 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:54.563 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:54.564 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:54.564 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:24:54.564 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:53.156252+0000", 2024-09-13T11:24:54.564 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.3460000000000001e-06 2024-09-13T11:24:54.564 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:54.564 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:54.564 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:24:54.564 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:53.156336+0000", 2024-09-13T11:24:54.564 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.3714999999999994e-05 2024-09-13T11:24:54.564 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:54.564 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:54.565 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:24:54.565 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:53.156347+0000", 2024-09-13T11:24:54.565 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0903999999999999e-05 2024-09-13T11:24:54.565 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:54.565 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:54.565 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:24:54.565 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:53.156352+0000", 2024-09-13T11:24:54.565 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.9139999999999999e-06 2024-09-13T11:24:54.565 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:54.565 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:54.565 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:24:54.565 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:53.158028+0000", 2024-09-13T11:24:54.566 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.001675958 2024-09-13T11:24:54.566 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:54.566 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:54.566 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:24:54.566 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:53.158154+0000", 2024-09-13T11:24:54.566 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00012653 2024-09-13T11:24:54.566 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:54.566 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:54.566 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 4,5", 2024-09-13T11:24:54.566 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:53.158277+0000", 2024-09-13T11:24:54.566 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000122168 2024-09-13T11:24:54.566 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:54.567 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:54.567 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:24:54.567 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:53.159585+0000", 2024-09-13T11:24:54.567 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.001308528 2024-09-13T11:24:54.567 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:54.567 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:54.567 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:54.567 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:53.163172+0000", 2024-09-13T11:24:54.567 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0035869690000000002 2024-09-13T11:24:54.567 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:54.567 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:54.568 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:54.568 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:53.379730+0000", 2024-09-13T11:24:54.568 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.21655822599999999 2024-09-13T11:24:54.568 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:54.568 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:54.568 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:24:54.568 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:53.379810+0000", 2024-09-13T11:24:54.568 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.0041000000000002e-05 2024-09-13T11:24:54.568 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:54.568 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:54.568 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:24:54.568 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:53.380195+0000", 2024-09-13T11:24:54.569 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000384519 2024-09-13T11:24:54.569 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:54.569 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:24:54.569 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:54.569 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:54.569 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:54.569 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDPGPush(3.21 96/78 [PushOp(3:862d73af:::benchmark_data_smithi177_31116_object32719:head, version: 60'2029, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:862d73af:::benchmark_data_smithi177_31116_object32719:head@60'2029, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:24:54.569 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:53.884628+0000", 2024-09-13T11:24:54.569 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 60.631060075000001, 2024-09-13T11:24:54.569 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.21020203400000001, 2024-09-13T11:24:54.569 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:24:54.569 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:24:54.570 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:24:54.570 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:24:54.570 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:54.570 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:24:54.570 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:53.884628+0000", 2024-09-13T11:24:54.570 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:54.570 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:54.570 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:54.570 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:24:54.570 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:53.884628+0000", 2024-09-13T11:24:54.570 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:54.571 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:54.571 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:54.571 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:24:54.571 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:53.884630+0000", 2024-09-13T11:24:54.571 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.68e-06 2024-09-13T11:24:54.571 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:54.571 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:54.571 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:24:54.571 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:53.884745+0000", 2024-09-13T11:24:54.571 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000115599 2024-09-13T11:24:54.571 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:54.572 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:54.572 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:24:54.572 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:54.093948+0000", 2024-09-13T11:24:54.572 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.20920306999999999 2024-09-13T11:24:54.572 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:54.572 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:54.572 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:24:54.572 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:54.093961+0000", 2024-09-13T11:24:54.572 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2815e-05 2024-09-13T11:24:54.572 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:54.572 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:54.573 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:24:54.573 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:54.094139+0000", 2024-09-13T11:24:54.573 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00017824 2024-09-13T11:24:54.573 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:54.573 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:54.573 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:24:54.573 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:54.094150+0000", 2024-09-13T11:24:54.573 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0593e-05 2024-09-13T11:24:54.573 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:54.573 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:54.573 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:24:54.574 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:54.094830+0000", 2024-09-13T11:24:54.574 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00068003699999999998 2024-09-13T11:24:54.574 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:54.574 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:24:54.574 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:54.574 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:54.574 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:54.574 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:83239 3.e 3:74547f17:::benchmark_data_smithi177_31116_object83238:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106)", 2024-09-13T11:24:54.574 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:24:04.580085+0000", 2024-09-13T11:24:54.574 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 49.935603192000002, 2024-09-13T11:24:54.574 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.33785762600000002, 2024-09-13T11:24:54.575 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:24:54.575 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:24:54.575 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:54.575 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:24:54.575 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:24:54.575 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:54.575 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 83239 2024-09-13T11:24:54.575 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:54.575 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:24:54.575 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:54.575 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:24:54.576 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:04.580085+0000", 2024-09-13T11:24:54.576 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:54.576 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:54.576 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:54.576 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:24:54.576 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:04.580085+0000", 2024-09-13T11:24:54.576 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:54.576 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:54.576 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:54.576 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:24:54.576 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:04.580089+0000", 2024-09-13T11:24:54.576 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.053e-06 2024-09-13T11:24:54.577 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:54.577 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:54.577 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:24:54.577 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:04.580166+0000", 2024-09-13T11:24:54.577 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.7102999999999997e-05 2024-09-13T11:24:54.577 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:54.577 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:54.577 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:24:54.577 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:04.580196+0000", 2024-09-13T11:24:54.577 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.0196e-05 2024-09-13T11:24:54.578 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:54.578 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:54.578 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:24:54.578 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:04.580212+0000", 2024-09-13T11:24:54.578 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.535e-05 2024-09-13T11:24:54.578 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:54.578 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:54.578 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:24:54.578 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:04.580336+0000", 2024-09-13T11:24:54.578 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00012431899999999999 2024-09-13T11:24:54.578 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:54.578 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:54.579 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:24:54.579 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:04.580575+0000", 2024-09-13T11:24:54.579 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00023924100000000001 2024-09-13T11:24:54.579 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:54.579 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:54.579 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 1", 2024-09-13T11:24:54.579 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:04.580798+0000", 2024-09-13T11:24:54.579 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00022301499999999999 2024-09-13T11:24:54.579 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:54.579 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:54.579 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:24:54.580 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:04.583404+0000", 2024-09-13T11:24:54.580 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0026056120000000002 2024-09-13T11:24:54.580 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:54.580 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:54.580 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:54.580 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:04.917768+0000", 2024-09-13T11:24:54.580 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.33436387499999998 2024-09-13T11:24:54.580 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:54.580 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:54.580 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:24:54.580 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:04.917859+0000", 2024-09-13T11:24:54.580 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.1732999999999997e-05 2024-09-13T11:24:54.581 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:54.581 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:54.581 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:24:54.581 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:04.917942+0000", 2024-09-13T11:24:54.581 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.3128999999999996e-05 2024-09-13T11:24:54.581 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:54.581 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:24:54.581 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:54.581 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:54.581 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:54.581 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:84571 3.21 3:85e7a789:::benchmark_data_smithi177_31116_object84570:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-09-13T11:24:54.582 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:24:05.695256+0000", 2024-09-13T11:24:54.582 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 48.820432160999999, 2024-09-13T11:24:54.582 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.18917198299999999, 2024-09-13T11:24:54.582 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:24:54.582 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:24:54.582 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:54.582 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:24:54.582 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:24:54.582 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:54.582 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 84571 2024-09-13T11:24:54.582 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:54.582 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:24:54.583 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:54.583 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:24:54.583 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:05.695256+0000", 2024-09-13T11:24:54.583 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:54.583 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:54.583 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:54.583 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:24:54.583 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:05.695256+0000", 2024-09-13T11:24:54.583 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:54.583 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:54.583 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:54.584 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:24:54.584 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:05.695258+0000", 2024-09-13T11:24:54.584 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.0530000000000001e-06 2024-09-13T11:24:54.584 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:54.584 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:54.584 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:24:54.584 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:05.695304+0000", 2024-09-13T11:24:54.584 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.5769999999999997e-05 2024-09-13T11:24:54.584 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:54.584 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:54.584 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:24:54.585 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:05.695313+0000", 2024-09-13T11:24:54.585 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.7049999999999994e-06 2024-09-13T11:24:54.585 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:54.585 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:54.585 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:24:54.585 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:05.695321+0000", 2024-09-13T11:24:54.585 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.7190000000000001e-06 2024-09-13T11:24:54.585 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:54.585 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:54.585 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:24:54.585 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:05.695404+0000", 2024-09-13T11:24:54.585 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.3188000000000004e-05 2024-09-13T11:24:54.586 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:54.586 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:54.586 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:24:54.586 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:05.695635+0000", 2024-09-13T11:24:54.586 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00023032900000000001 2024-09-13T11:24:54.586 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:54.586 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:54.586 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 8", 2024-09-13T11:24:54.586 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:05.695852+0000", 2024-09-13T11:24:54.586 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00021749599999999999 2024-09-13T11:24:54.586 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:54.587 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:54.587 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:24:54.587 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:05.697208+0000", 2024-09-13T11:24:54.587 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00135633 2024-09-13T11:24:54.587 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:54.587 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:54.587 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:54.587 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:05.884337+0000", 2024-09-13T11:24:54.587 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.187128867 2024-09-13T11:24:54.587 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:54.587 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:54.587 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:24:54.588 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:05.884391+0000", 2024-09-13T11:24:54.588 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.3724999999999998e-05 2024-09-13T11:24:54.588 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:54.588 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:54.588 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:24:54.588 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:05.884428+0000", 2024-09-13T11:24:54.588 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.6801000000000002e-05 2024-09-13T11:24:54.588 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:54.588 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:24:54.588 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:54.588 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:54.589 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:54.589 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4665.0:104290 3.20 e115/77 3:05d45b57:::benchmark_data_smithi177_31116_object104289:head v 115'3905, mlcod=115'3902)", 2024-09-13T11:24:54.589 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:24:20.200225+0000", 2024-09-13T11:24:54.589 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 34.315462896, 2024-09-13T11:24:54.589 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.21051286899999999, 2024-09-13T11:24:54.589 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:24:54.589 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:24:54.589 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:54.589 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:24:54.589 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:54.589 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:24:54.590 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:20.200225+0000", 2024-09-13T11:24:54.590 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:54.590 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:54.590 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:54.590 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:24:54.590 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:20.200225+0000", 2024-09-13T11:24:54.590 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:54.590 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:54.590 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:54.590 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:24:54.590 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:20.200227+0000", 2024-09-13T11:24:54.590 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.897e-06 2024-09-13T11:24:54.591 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:54.591 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:54.591 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:24:54.591 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:20.200253+0000", 2024-09-13T11:24:54.591 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.5904000000000001e-05 2024-09-13T11:24:54.591 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:54.591 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:54.591 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:24:54.591 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:20.409353+0000", 2024-09-13T11:24:54.591 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.209100339 2024-09-13T11:24:54.591 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:54.591 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:54.592 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:24:54.592 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:20.409359+0000", 2024-09-13T11:24:54.592 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.1689999999999997e-06 2024-09-13T11:24:54.592 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:54.592 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:54.592 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:24:54.592 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:20.409418+0000", 2024-09-13T11:24:54.592 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.8761999999999997e-05 2024-09-13T11:24:54.592 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:54.592 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:54.592 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:24:54.593 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:20.409440+0000", 2024-09-13T11:24:54.593 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.2098000000000001e-05 2024-09-13T11:24:54.593 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:54.593 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:54.593 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:24:54.593 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:20.410705+0000", 2024-09-13T11:24:54.593 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00126483 2024-09-13T11:24:54.593 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:54.593 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:54.593 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:24:54.593 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:20.410738+0000", 2024-09-13T11:24:54.594 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.2870000000000002e-05 2024-09-13T11:24:54.594 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:54.594 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:24:54.594 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:54.594 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:54.594 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:54.594 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:110271 3.2d 3.3c2681ad (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e119)", 2024-09-13T11:24:54.594 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:24:25.403006+0000", 2024-09-13T11:24:54.594 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 29.112681708, 2024-09-13T11:24:54.594 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.68768755999999998, 2024-09-13T11:24:54.594 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:24:54.595 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:24:54.595 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-13T11:24:54.595 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:24:54.595 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:24:54.595 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:54.595 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 110271 2024-09-13T11:24:54.595 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:54.595 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:24:54.595 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:54.595 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:24:54.595 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:25.403006+0000", 2024-09-13T11:24:54.595 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:54.596 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:54.596 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:54.596 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:24:54.596 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:25.403006+0000", 2024-09-13T11:24:54.596 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:54.596 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:54.596 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:54.596 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:24:54.596 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:25.403011+0000", 2024-09-13T11:24:54.596 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.5199999999999999e-06 2024-09-13T11:24:54.596 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:54.597 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:54.597 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:24:54.597 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:25.403065+0000", 2024-09-13T11:24:54.597 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.4478999999999998e-05 2024-09-13T11:24:54.597 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:54.597 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:54.597 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:24:54.597 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:25.403097+0000", 2024-09-13T11:24:54.597 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.1636e-05 2024-09-13T11:24:54.597 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:54.597 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:54.597 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:24:54.598 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:25.403121+0000", 2024-09-13T11:24:54.598 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.3924e-05 2024-09-13T11:24:54.598 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:54.598 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:54.598 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:24:54.598 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:25.403216+0000", 2024-09-13T11:24:54.598 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.5557999999999995e-05 2024-09-13T11:24:54.598 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:54.598 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:54.598 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for peered", 2024-09-13T11:24:54.598 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:25.403234+0000", 2024-09-13T11:24:54.598 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.7104000000000001e-05 2024-09-13T11:24:54.599 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:54.599 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:54.599 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:24:54.599 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:26.090684+0000", 2024-09-13T11:24:54.599 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.68745000700000003 2024-09-13T11:24:54.599 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:54.599 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:54.599 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:24:54.599 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:26.090694+0000", 2024-09-13T11:24:54.599 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0332e-05 2024-09-13T11:24:54.599 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:54.599 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:24:54.600 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:54.600 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:54.600 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:54.600 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDPGPushReply(3.22 125/122 [PushReplyOp(3:475b51c3:::benchmark_data_smithi177_31116_object16005:head)])", 2024-09-13T11:24:54.600 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:24:32.400814+0000", 2024-09-13T11:24:54.600 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 22.114873943999999, 2024-09-13T11:24:54.600 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.20917021799999999, 2024-09-13T11:24:54.600 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:24:54.600 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:24:54.600 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-13T11:24:54.600 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:24:54.601 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:54.601 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:24:54.601 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:32.400814+0000", 2024-09-13T11:24:54.601 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:54.601 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:54.601 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:54.601 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:24:54.601 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:32.400814+0000", 2024-09-13T11:24:54.601 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:54.601 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:54.601 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:54.601 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:24:54.602 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:32.400817+0000", 2024-09-13T11:24:54.602 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.5000000000000002e-06 2024-09-13T11:24:54.602 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:54.602 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:54.602 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:24:54.602 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:32.400826+0000", 2024-09-13T11:24:54.602 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.8830000000000001e-06 2024-09-13T11:24:54.602 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:54.602 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:54.602 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:24:54.602 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:32.608921+0000", 2024-09-13T11:24:54.603 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.20809501599999999 2024-09-13T11:24:54.603 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:54.603 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:54.603 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:24:54.603 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:32.608931+0000", 2024-09-13T11:24:54.603 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.1679999999999993e-06 2024-09-13T11:24:54.603 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:54.603 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:54.603 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:24:54.603 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:32.608982+0000", 2024-09-13T11:24:54.603 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.1827999999999999e-05 2024-09-13T11:24:54.603 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:54.604 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:54.604 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:24:54.604 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:32.609984+0000", 2024-09-13T11:24:54.604 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.001001823 2024-09-13T11:24:54.604 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:54.604 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:24:54.604 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:54.604 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:54.604 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:54.604 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:120816 3.21 3:850a51ba:::benchmark_data_smithi177_31116_object120815:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e129)", 2024-09-13T11:24:54.604 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:24:38.601204+0000", 2024-09-13T11:24:54.605 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 15.91448391, 2024-09-13T11:24:54.605 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.18932929600000001, 2024-09-13T11:24:54.605 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:24:54.605 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:24:54.605 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:54.605 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:24:54.605 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:24:54.605 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:54.605 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 120816 2024-09-13T11:24:54.605 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:54.605 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:24:54.606 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:54.606 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:24:54.606 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:38.601204+0000", 2024-09-13T11:24:54.606 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:54.606 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:54.606 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:54.606 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:24:54.606 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:38.601204+0000", 2024-09-13T11:24:54.606 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:54.606 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:54.606 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:54.606 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:24:54.607 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:38.601206+0000", 2024-09-13T11:24:54.607 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.7370000000000001e-06 2024-09-13T11:24:54.607 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:54.607 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:54.607 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:24:54.607 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:38.601241+0000", 2024-09-13T11:24:54.607 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.5524000000000001e-05 2024-09-13T11:24:54.607 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:54.607 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:54.607 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:24:54.607 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:38.601253+0000", 2024-09-13T11:24:54.607 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.1936000000000001e-05 2024-09-13T11:24:54.608 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:54.608 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:54.608 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:24:54.608 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:38.601262+0000", 2024-09-13T11:24:54.608 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.2570000000000007e-06 2024-09-13T11:24:54.608 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:54.608 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:54.608 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:24:54.608 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:38.601339+0000", 2024-09-13T11:24:54.608 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.7751999999999997e-05 2024-09-13T11:24:54.608 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:54.609 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:54.609 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:24:54.609 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:38.601527+0000", 2024-09-13T11:24:54.609 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000188092 2024-09-13T11:24:54.609 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:54.609 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:54.609 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 8", 2024-09-13T11:24:54.609 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:38.601684+0000", 2024-09-13T11:24:54.609 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00015659699999999999 2024-09-13T11:24:54.609 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:54.609 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:54.609 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:24:54.610 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:38.603108+0000", 2024-09-13T11:24:54.610 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0014244240000000001 2024-09-13T11:24:54.610 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:54.610 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:54.610 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:54.610 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:38.790425+0000", 2024-09-13T11:24:54.610 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.187316345 2024-09-13T11:24:54.610 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:54.610 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:54.610 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:24:54.610 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:38.790488+0000", 2024-09-13T11:24:54.610 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.3184999999999999e-05 2024-09-13T11:24:54.611 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:54.611 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:54.611 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:24:54.611 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:38.790533+0000", 2024-09-13T11:24:54.611 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.5447e-05 2024-09-13T11:24:54.611 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:54.611 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:24:54.611 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:54.611 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:54.611 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:54.611 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:126287 3.21 3:86b6c81d:::benchmark_data_smithi177_31116_object126286:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e130)", 2024-09-13T11:24:54.612 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:24:43.210051+0000", 2024-09-13T11:24:54.612 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 11.305636854999999, 2024-09-13T11:24:54.612 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.18801518, 2024-09-13T11:24:54.612 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:24:54.612 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:24:54.612 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:54.612 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:24:54.612 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:24:54.612 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:54.612 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 126287 2024-09-13T11:24:54.612 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:54.612 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:24:54.613 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:54.613 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:24:54.613 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:43.210051+0000", 2024-09-13T11:24:54.613 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:54.613 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:54.613 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:54.613 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:24:54.613 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:43.210051+0000", 2024-09-13T11:24:54.613 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:54.613 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:54.613 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:54.614 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:24:54.614 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:43.210054+0000", 2024-09-13T11:24:54.614 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.3810000000000002e-06 2024-09-13T11:24:54.614 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:54.614 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:54.614 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:24:54.614 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:43.210137+0000", 2024-09-13T11:24:54.614 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.3583000000000003e-05 2024-09-13T11:24:54.615 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:54.615 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:54.615 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:24:54.615 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:43.210154+0000", 2024-09-13T11:24:54.615 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.6826000000000001e-05 2024-09-13T11:24:54.615 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:54.615 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:54.615 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:24:54.615 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:43.210162+0000", 2024-09-13T11:24:54.615 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.0909999999999993e-06 2024-09-13T11:24:54.616 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:54.616 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:54.616 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:24:54.616 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:43.210220+0000", 2024-09-13T11:24:54.616 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.7507999999999999e-05 2024-09-13T11:24:54.616 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:54.616 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:54.616 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:24:54.616 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:43.210346+0000", 2024-09-13T11:24:54.616 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00012596800000000001 2024-09-13T11:24:54.616 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:54.617 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:54.617 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 8", 2024-09-13T11:24:54.617 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:43.210494+0000", 2024-09-13T11:24:54.617 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00014836299999999999 2024-09-13T11:24:54.617 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:54.617 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:54.617 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:24:54.617 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:43.212217+0000", 2024-09-13T11:24:54.617 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.001722814 2024-09-13T11:24:54.617 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:54.617 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:54.617 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:54.618 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:43.397985+0000", 2024-09-13T11:24:54.618 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.18576780900000001 2024-09-13T11:24:54.618 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:54.618 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:54.618 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:24:54.618 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:43.398016+0000", 2024-09-13T11:24:54.618 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.1795000000000003e-05 2024-09-13T11:24:54.618 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:54.618 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:54.618 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:24:54.618 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:43.398066+0000", 2024-09-13T11:24:54.618 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.0042e-05 2024-09-13T11:24:54.619 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:54.619 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:24:54.619 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:54.619 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:54.619 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:54.619 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4665.0:129901 3.20 e134/77 3:067f5dfc:::benchmark_data_smithi177_31116_object129900:head v 134'4294, mlcod=134'4291)", 2024-09-13T11:24:54.619 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:24:47.145530+0000", 2024-09-13T11:24:54.619 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 7.3701576329999998, 2024-09-13T11:24:54.619 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.333688349, 2024-09-13T11:24:54.619 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:24:54.619 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:24:54.620 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:54.620 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:24:54.620 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:54.620 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:24:54.620 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:47.145530+0000", 2024-09-13T11:24:54.620 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:54.620 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:54.620 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:54.620 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:24:54.620 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:47.145530+0000", 2024-09-13T11:24:54.620 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:54.620 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:54.621 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:54.621 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:24:54.621 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:47.145532+0000", 2024-09-13T11:24:54.621 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.7999999999999999e-06 2024-09-13T11:24:54.621 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:54.621 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:54.621 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:24:54.621 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:47.145557+0000", 2024-09-13T11:24:54.621 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.4678999999999999e-05 2024-09-13T11:24:54.621 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:54.621 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:54.621 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:24:54.622 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:47.477539+0000", 2024-09-13T11:24:54.622 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.33198164099999999 2024-09-13T11:24:54.622 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:54.622 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:54.622 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:24:54.622 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:47.477544+0000", 2024-09-13T11:24:54.622 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.9059999999999999e-06 2024-09-13T11:24:54.622 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:54.622 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:54.622 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:24:54.622 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:47.477597+0000", 2024-09-13T11:24:54.623 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.295e-05 2024-09-13T11:24:54.623 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:54.623 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:54.623 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:24:54.623 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:47.477619+0000", 2024-09-13T11:24:54.623 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.1835999999999999e-05 2024-09-13T11:24:54.623 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:54.623 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:54.623 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:24:54.623 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:47.479175+0000", 2024-09-13T11:24:54.623 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0015560470000000001 2024-09-13T11:24:54.623 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:54.624 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:54.624 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:24:54.624 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:47.479219+0000", 2024-09-13T11:24:54.624 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.3489999999999999e-05 2024-09-13T11:24:54.624 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:54.624 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:24:54.624 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:54.624 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:54.624 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:54.624 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:130006 3.21 3:85b3e472:::benchmark_data_smithi177_31116_object130005:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e134)", 2024-09-13T11:24:54.624 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:24:47.433580+0000", 2024-09-13T11:24:54.625 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 7.0821078399999999, 2024-09-13T11:24:54.625 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.189659507, 2024-09-13T11:24:54.625 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:24:54.625 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:24:54.625 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:54.625 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:24:54.625 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:24:54.625 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:54.625 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 130006 2024-09-13T11:24:54.625 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:54.626 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:24:54.626 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:54.626 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:24:54.626 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:47.433580+0000", 2024-09-13T11:24:54.626 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:54.626 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:54.626 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:54.626 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:24:54.626 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:47.433580+0000", 2024-09-13T11:24:54.626 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:24:54.626 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:54.626 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:54.627 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:24:54.627 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:47.433582+0000", 2024-09-13T11:24:54.627 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.266e-06 2024-09-13T11:24:54.627 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:54.627 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:54.627 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:24:54.627 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:47.433656+0000", 2024-09-13T11:24:54.627 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.3780000000000002e-05 2024-09-13T11:24:54.627 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:54.627 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:54.628 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:24:54.628 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:47.433796+0000", 2024-09-13T11:24:54.628 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000139627 2024-09-13T11:24:54.628 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:54.628 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:54.628 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:24:54.628 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:47.433803+0000", 2024-09-13T11:24:54.628 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.5809999999999999e-06 2024-09-13T11:24:54.628 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:54.628 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:54.628 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:24:54.629 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:47.433849+0000", 2024-09-13T11:24:54.629 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.5485000000000002e-05 2024-09-13T11:24:54.629 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:54.629 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:54.629 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:24:54.629 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:47.433946+0000", 2024-09-13T11:24:54.629 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.6788999999999998e-05 2024-09-13T11:24:54.629 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:54.629 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:54.629 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 8", 2024-09-13T11:24:54.629 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:47.434032+0000", 2024-09-13T11:24:54.630 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.5898999999999999e-05 2024-09-13T11:24:54.630 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:54.630 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:54.630 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:24:54.630 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:47.435315+0000", 2024-09-13T11:24:54.630 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0012834319999999999 2024-09-13T11:24:54.630 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:54.630 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:54.630 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:54.630 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:47.623177+0000", 2024-09-13T11:24:54.630 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.187861741 2024-09-13T11:24:54.630 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:54.631 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:54.631 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:24:54.631 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:47.623211+0000", 2024-09-13T11:24:54.631 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.3989000000000003e-05 2024-09-13T11:24:54.631 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:24:54.631 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:24:54.631 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:24:54.631 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:47.623240+0000", 2024-09-13T11:24:54.631 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.8918e-05 2024-09-13T11:24:54.632 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:54.632 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:24:54.632 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:54.632 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:24:54.632 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:24:54.632 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-13T11:24:54.633 INFO:tasks.ceph.osd.15.smithi177.stderr:2024-09-13T11:24:54.630+0000 7f9d6ce77640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26578) UID: 0 2024-09-13T11:24:54.636 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-13T11:24:54.723 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T11:24:54.734 INFO:tasks.ceph.osd.15.smithi177.stderr:2024-09-13T11:24:54.730+0000 7f9d6ce77640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26578) UID: 0 2024-09-13T11:24:54.787 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-13T11:24:54.787 INFO:teuthology.orchestra.run.smithi001.stdout: "ops": [], 2024-09-13T11:24:54.787 INFO:teuthology.orchestra.run.smithi001.stdout: "num_ops": 0 2024-09-13T11:24:54.787 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-13T11:24:54.805 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-13T11:24:54.856 INFO:tasks.ceph.osd.12.smithi001.stderr:2024-09-13T11:24:54.855+0000 7f892c6e6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26314) UID: 0 2024-09-13T11:24:54.895 INFO:teuthology.orchestra.run.smithi001.stdout:osd.0: {} 2024-09-13T11:24:54.895 INFO:teuthology.orchestra.run.smithi001.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T11:24:54.924 INFO:teuthology.orchestra.run.smithi001.stdout:osd.1: {} 2024-09-13T11:24:54.925 INFO:teuthology.orchestra.run.smithi001.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T11:24:54.935 INFO:tasks.ceph.osd.13.smithi092.stderr:2024-09-13T11:24:54.930+0000 7f3e5afea640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26350) UID: 0 2024-09-13T11:24:54.962 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-13T11:24:54.962 INFO:teuthology.orchestra.run.smithi001.stdout: "ops": [], 2024-09-13T11:24:54.962 INFO:teuthology.orchestra.run.smithi001.stdout: "complaint_time": 30, 2024-09-13T11:24:54.962 INFO:teuthology.orchestra.run.smithi001.stdout: "num_blocked_ops": 0 2024-09-13T11:24:54.962 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-13T11:24:54.964 INFO:teuthology.orchestra.run.smithi001.stdout:osd.2: {} 2024-09-13T11:24:54.964 INFO:teuthology.orchestra.run.smithi001.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T11:24:54.981 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-13T11:24:54.987 INFO:tasks.ceph.osd.7.smithi177.stderr:2024-09-13T11:24:54.982+0000 7f896ff4a640 -1 osd.7 140 handle_osd_map injecting map crc failure 2024-09-13T11:24:54.987 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-13T11:24:54.989 INFO:tasks.ceph.osd.9.smithi092.stderr:2024-09-13T11:24:54.986+0000 7ff59bdf5640 -1 osd.9 140 handle_osd_map injecting map crc failure 2024-09-13T11:24:54.995 INFO:teuthology.orchestra.run.smithi001.stdout:osd.3: {} 2024-09-13T11:24:54.996 INFO:teuthology.orchestra.run.smithi001.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T11:24:55.003 INFO:tasks.ceph.osd.14.smithi162.stderr:2024-09-13T11:24:55.000+0000 7f3df072a640 -1 osd.14 120 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-13T11:24:55.014 INFO:tasks.radosbench.radosbench.0.smithi177.stdout: 148 16 138624 138608 58.5272 34.5625 0.012215 0.0170802 2024-09-13T11:24:55.028 INFO:teuthology.orchestra.run.smithi001.stdout:osd.4: {} 2024-09-13T11:24:55.028 INFO:teuthology.orchestra.run.smithi001.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T11:24:55.035 INFO:tasks.ceph.osd.15.smithi177.stderr:2024-09-13T11:24:55.030+0000 7f9d6ce77640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26578) UID: 0 2024-09-13T11:24:55.066 INFO:teuthology.orchestra.run.smithi001.stdout:osd.5: {} 2024-09-13T11:24:55.066 INFO:teuthology.orchestra.run.smithi001.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T11:24:55.098 INFO:teuthology.orchestra.run.smithi001.stdout:osd.6: {} 2024-09-13T11:24:55.099 INFO:teuthology.orchestra.run.smithi001.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T11:24:55.110 INFO:tasks.ceph.osd.14.smithi162.stderr:2024-09-13T11:24:55.108+0000 7f3dec334640 -1 osd.14 120 handle_osd_map injecting map crc failure 2024-09-13T11:24:55.136 INFO:teuthology.orchestra.run.smithi001.stdout:osd.7: {} 2024-09-13T11:24:55.136 INFO:teuthology.orchestra.run.smithi001.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T11:24:55.137 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-13T11:24:55.135+0000 7f6a68775640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26307) UID: 0 2024-09-13T11:24:55.164 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-13T11:24:55.164 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:24:55.167 INFO:teuthology.orchestra.run.smithi001.stdout:osd.8: {} 2024-09-13T11:24:55.168 INFO:teuthology.orchestra.run.smithi001.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-13T11:24:55.187 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-09-13T11:24:55.199 INFO:teuthology.orchestra.run.smithi001.stdout:osd.9: {} 2024-09-13T11:24:55.199 INFO:teuthology.orchestra.run.smithi001.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-13T11:24:55.237 INFO:tasks.ceph.osd.9.smithi092.stderr:2024-09-13T11:24:55.230+0000 7ff5ab3d0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26348) UID: 0 2024-09-13T11:24:55.237 INFO:teuthology.orchestra.run.smithi001.stdout:osd.10: {} 2024-09-13T11:24:55.237 INFO:teuthology.orchestra.run.smithi001.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-13T11:24:55.276 INFO:teuthology.orchestra.run.smithi001.stdout:osd.11: {} 2024-09-13T11:24:55.276 INFO:teuthology.orchestra.run.smithi001.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-13T11:24:55.303 INFO:teuthology.orchestra.run.smithi001.stdout:osd.12: {} 2024-09-13T11:24:55.304 INFO:teuthology.orchestra.run.smithi001.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-13T11:24:55.331 INFO:teuthology.orchestra.run.smithi001.stdout:osd.13: {} 2024-09-13T11:24:55.332 INFO:teuthology.orchestra.run.smithi001.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-13T11:24:55.332 INFO:teuthology.orchestra.run.smithi001.stderr:Error ENXIO: problem getting command descriptions from osd.14 2024-09-13T11:24:55.337 INFO:tasks.ceph.osd.13.smithi092.stderr:2024-09-13T11:24:55.334+0000 7f3e5afea640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26350) UID: 0 2024-09-13T11:24:55.358 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-13T11:24:55.358 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:24:55.372 INFO:teuthology.orchestra.run.smithi001.stdout:osd.15: {} 2024-09-13T11:24:55.372 INFO:teuthology.orchestra.run.smithi001.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-13T11:24:55.378 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-13T11:24:55.378 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-13T11:24:55.382 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-09-13T11:24:55.468 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-13T11:24:55.467+0000 7f6a68775640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26307) UID: 0 2024-09-13T11:24:55.539 INFO:tasks.ceph.osd.11.smithi177.stderr:2024-09-13T11:24:55.534+0000 7f24792c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26580) UID: 0 2024-09-13T11:24:55.542 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-13T11:24:55.542 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:24:55.557 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-09-13T11:24:55.648 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-13T11:24:55.647+0000 7f6a68775640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26307) UID: 0 2024-09-13T11:24:55.722 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-13T11:24:55.722 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:24:55.735 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_ops_in_flight 2024-09-13T11:24:55.780 INFO:tasks.ceph.osd.8.smithi001.stderr:2024-09-13T11:24:55.779+0000 7fb9b48ae640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26312) UID: 0 2024-09-13T11:24:55.841 INFO:tasks.ceph.osd.5.smithi092.stderr:2024-09-13T11:24:55.838+0000 7f6f0b705640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26346) UID: 0 2024-09-13T11:24:55.891 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-13T11:24:55.891 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:24:55.906 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_blocked_ops 2024-09-13T11:24:55.942 INFO:tasks.ceph.osd.5.smithi092.stderr:2024-09-13T11:24:55.938+0000 7f6f0b705640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26346) UID: 0 2024-09-13T11:24:55.978 INFO:tasks.ceph.osd.4.smithi001.stderr:2024-09-13T11:24:55.975+0000 7ff8d2c41640 -1 osd.4 141 handle_osd_map injecting map crc failure 2024-09-13T11:24:55.984 INFO:teuthology.orchestra.run.smithi001.stderr:noscrub is unset 2024-09-13T11:24:55.986 INFO:tasks.ceph.osd.2.smithi162.stderr:2024-09-13T11:24:55.984+0000 7fe91fd87640 -1 osd.2 141 handle_osd_map injecting map crc failure 2024-09-13T11:24:56.014 INFO:tasks.radosbench.radosbench.0.smithi177.stdout: 149 16 139076 139060 58.3239 28.25 0.981666 0.0171012 2024-09-13T11:24:56.042 INFO:tasks.ceph.osd.8.smithi001.stderr:2024-09-13T11:24:56.039+0000 7fb9b48ae640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26312) UID: 0 2024-09-13T11:24:56.049 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-13T11:24:56.049 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:24:56.071 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_historic_ops 2024-09-13T11:24:56.123 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_ops_in_flight 2024-09-13T11:24:56.143 INFO:tasks.ceph.osd.7.smithi177.stderr:2024-09-13T11:24:56.138+0000 7f897b51d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26575) UID: 0 2024-09-13T11:24:56.244 INFO:tasks.ceph.osd.11.smithi177.stderr:2024-09-13T11:24:56.242+0000 7f24792c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26580) UID: 0 2024-09-13T11:24:56.260 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-13T11:24:56.260 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:24:56.274 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-13T11:24:56.275 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [], 2024-09-13T11:24:56.275 INFO:teuthology.orchestra.run.smithi162.stdout: "num_ops": 0 2024-09-13T11:24:56.275 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-13T11:24:56.284 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-13T11:24:56.295 INFO:tasks.thrashosds.thrasher:test_bluestore_reshard completed 2024-09-13T11:24:56.345 INFO:tasks.ceph.osd.1.smithi092.stderr:2024-09-13T11:24:56.338+0000 7fe7789c5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33718) UID: 0 2024-09-13T11:24:56.428 INFO:teuthology.orchestra.run.smithi092.stdout:ERROR: (22) Invalid argument 2024-09-13T11:24:56.428 INFO:teuthology.orchestra.run.smithi092.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:24:56.446 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-13T11:24:56.446 INFO:tasks.ceph.osd.2.smithi162.stderr:2024-09-13T11:24:56.444+0000 7fe92b35a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26313) UID: 0 2024-09-13T11:24:56.546 INFO:tasks.ceph.osd.1.smithi092.stderr:2024-09-13T11:24:56.542+0000 7fe7789c5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33718) UID: 0 2024-09-13T11:24:56.631 INFO:teuthology.orchestra.run.smithi092.stdout:ERROR: (22) Invalid argument 2024-09-13T11:24:56.631 INFO:teuthology.orchestra.run.smithi092.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:24:56.647 INFO:tasks.ceph.osd.3.smithi177.stderr:2024-09-13T11:24:56.642+0000 7f10adba4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26574) UID: 0 2024-09-13T11:24:56.653 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-13T11:24:56.748 INFO:tasks.ceph.osd.11.smithi177.stderr:2024-09-13T11:24:56.746+0000 7f24792c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26580) UID: 0 2024-09-13T11:24:56.842 INFO:teuthology.orchestra.run.smithi092.stdout:ERROR: (22) Invalid argument 2024-09-13T11:24:56.843 INFO:teuthology.orchestra.run.smithi092.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:24:56.849 INFO:tasks.ceph.osd.5.smithi092.stderr:2024-09-13T11:24:56.842+0000 7f6f0b705640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26346) UID: 0 2024-09-13T11:24:56.861 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-09-13T11:24:56.950 INFO:tasks.ceph.osd.13.smithi092.stderr:2024-09-13T11:24:56.946+0000 7f3e5afea640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26350) UID: 0 2024-09-13T11:24:56.990 INFO:tasks.ceph.osd.5.smithi092.stderr:2024-09-13T11:24:56.986+0000 7f6f00132640 -1 osd.5 142 handle_osd_map injecting map crc failure 2024-09-13T11:24:56.995 INFO:tasks.ceph.osd.9.smithi092.stderr:2024-09-13T11:24:56.990+0000 7ff59bdf5640 -1 osd.9 142 handle_osd_map injecting map crc failure 2024-09-13T11:24:57.014 INFO:tasks.radosbench.radosbench.0.smithi177.stdout: 150 8 140300 140292 58.4484 77 0.0222943 0.0171017 2024-09-13T11:24:57.042 INFO:teuthology.orchestra.run.smithi092.stdout:ERROR: (22) Invalid argument 2024-09-13T11:24:57.042 INFO:teuthology.orchestra.run.smithi092.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:24:57.050 INFO:tasks.ceph.osd.10.smithi162.stderr:2024-09-13T11:24:57.048+0000 7fb59c9f0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26317) UID: 0 2024-09-13T11:24:57.057 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-09-13T11:24:57.151 INFO:tasks.ceph.osd.13.smithi092.stderr:2024-09-13T11:24:57.146+0000 7f3e5afea640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26350) UID: 0 2024-09-13T11:24:57.227 INFO:teuthology.orchestra.run.smithi092.stdout:ERROR: (22) Invalid argument 2024-09-13T11:24:57.227 INFO:teuthology.orchestra.run.smithi092.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:24:57.242 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-09-13T11:24:57.252 INFO:tasks.ceph.osd.14.smithi162.stderr:2024-09-13T11:24:57.248+0000 7f3df5903640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33154) UID: 0 2024-09-13T11:24:57.353 INFO:tasks.ceph.osd.1.smithi092.stderr:2024-09-13T11:24:57.346+0000 7fe7789c5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33718) UID: 0 2024-09-13T11:24:57.379 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T11:24:57.418 INFO:teuthology.orchestra.run.smithi092.stdout:ERROR: (22) Invalid argument 2024-09-13T11:24:57.418 INFO:teuthology.orchestra.run.smithi092.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:24:57.434 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_ops_in_flight 2024-09-13T11:24:57.454 INFO:tasks.ceph.osd.14.smithi162.stderr:2024-09-13T11:24:57.452+0000 7f3df5903640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33154) UID: 0 2024-09-13T11:24:57.555 INFO:tasks.ceph.osd.4.smithi001.stderr:2024-09-13T11:24:57.551+0000 7ff8e0218640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26310) UID: 0 2024-09-13T11:24:57.561 INFO:teuthology.orchestra.run.smithi001.stdout:osd.0: {} 2024-09-13T11:24:57.561 INFO:teuthology.orchestra.run.smithi001.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T11:24:57.588 INFO:teuthology.orchestra.run.smithi001.stdout:osd.1: {} 2024-09-13T11:24:57.588 INFO:teuthology.orchestra.run.smithi001.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T11:24:57.613 INFO:teuthology.orchestra.run.smithi092.stdout:ERROR: (22) Invalid argument 2024-09-13T11:24:57.613 INFO:teuthology.orchestra.run.smithi092.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:24:57.615 INFO:teuthology.orchestra.run.smithi001.stdout:osd.2: {} 2024-09-13T11:24:57.615 INFO:teuthology.orchestra.run.smithi001.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T11:24:57.630 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_blocked_ops 2024-09-13T11:24:57.644 INFO:teuthology.orchestra.run.smithi001.stdout:osd.3: {} 2024-09-13T11:24:57.645 INFO:teuthology.orchestra.run.smithi001.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T11:24:57.656 INFO:tasks.ceph.osd.15.smithi177.stderr:2024-09-13T11:24:57.654+0000 7f9d6ce77640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26578) UID: 0 2024-09-13T11:24:57.668 INFO:teuthology.orchestra.run.smithi001.stdout:osd.4: {} 2024-09-13T11:24:57.668 INFO:teuthology.orchestra.run.smithi001.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T11:24:57.692 INFO:teuthology.orchestra.run.smithi001.stdout:osd.5: {} 2024-09-13T11:24:57.692 INFO:teuthology.orchestra.run.smithi001.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T11:24:57.755 INFO:teuthology.orchestra.run.smithi092.stdout:ERROR: (22) Invalid argument 2024-09-13T11:24:57.756 INFO:teuthology.orchestra.run.smithi092.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:24:57.757 INFO:tasks.ceph.osd.11.smithi177.stderr:2024-09-13T11:24:57.754+0000 7f24792c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26580) UID: 0 2024-09-13T11:24:57.771 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_historic_ops 2024-09-13T11:24:57.858 INFO:tasks.ceph.osd.4.smithi001.stderr:2024-09-13T11:24:57.855+0000 7ff8e0218640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26310) UID: 0 2024-09-13T11:24:57.892 INFO:teuthology.orchestra.run.smithi092.stdout:ERROR: (22) Invalid argument 2024-09-13T11:24:57.892 INFO:teuthology.orchestra.run.smithi092.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:24:57.909 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-13T11:24:57.958 INFO:tasks.ceph.osd.12.smithi001.stderr:2024-09-13T11:24:57.955+0000 7f892c6e6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26314) UID: 0 2024-09-13T11:24:58.006 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-13T11:24:58.011 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-13T11:24:58.007+0000 7f6a5d1a2640 -1 osd.0 143 handle_osd_map injecting map crc failure 2024-09-13T11:24:58.014 INFO:tasks.radosbench.radosbench.0.smithi177.stdout:Total time run: 150.591 2024-09-13T11:24:58.014 INFO:tasks.radosbench.radosbench.0.smithi177.stdout:Total writes made: 140300 2024-09-13T11:24:58.015 INFO:tasks.radosbench.radosbench.0.smithi177.stdout:Write size: 65536 2024-09-13T11:24:58.015 INFO:tasks.radosbench.radosbench.0.smithi177.stdout:Object size: 65536 2024-09-13T11:24:58.015 INFO:tasks.radosbench.radosbench.0.smithi177.stdout:Bandwidth (MB/sec): 58.2289 2024-09-13T11:24:58.015 INFO:tasks.radosbench.radosbench.0.smithi177.stdout:Stddev Bandwidth: 39.2697 2024-09-13T11:24:58.015 INFO:tasks.radosbench.radosbench.0.smithi177.stdout:Max bandwidth (MB/sec): 162.125 2024-09-13T11:24:58.015 INFO:tasks.radosbench.radosbench.0.smithi177.stdout:Min bandwidth (MB/sec): 0 2024-09-13T11:24:58.015 INFO:tasks.radosbench.radosbench.0.smithi177.stdout:Average IOPS: 931 2024-09-13T11:24:58.015 INFO:tasks.radosbench.radosbench.0.smithi177.stdout:Stddev IOPS: 628.328 2024-09-13T11:24:58.015 INFO:tasks.radosbench.radosbench.0.smithi177.stdout:Max IOPS: 2594 2024-09-13T11:24:58.015 INFO:tasks.radosbench.radosbench.0.smithi177.stdout:Min IOPS: 0 2024-09-13T11:24:58.016 INFO:tasks.radosbench.radosbench.0.smithi177.stdout:Average Latency(s): 0.0171254 2024-09-13T11:24:58.016 INFO:tasks.radosbench.radosbench.0.smithi177.stdout:Stddev Latency(s): 0.237731 2024-09-13T11:24:58.016 INFO:tasks.radosbench.radosbench.0.smithi177.stdout:Max latency(s): 22.1835 2024-09-13T11:24:58.016 INFO:tasks.radosbench.radosbench.0.smithi177.stdout:Min latency(s): 0.00139681 2024-09-13T11:24:58.025 INFO:tasks.radosbench.radosbench.0.smithi177.stdout:Cleaning up (deleting benchmark objects) 2024-09-13T11:24:58.052 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-13T11:24:58.052 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [ 2024-09-13T11:24:58.052 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.053 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:140315 3.29 3:967a7538:::benchmark_data_smithi177_31116_object12:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e143)", 2024-09-13T11:24:58.053 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:24:58.026180+0000", 2024-09-13T11:24:58.053 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 0.024860805999999999, 2024-09-13T11:24:58.053 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.024926692, 2024-09-13T11:24:58.053 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:24:58.053 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:24:58.053 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "waiting for sub ops", 2024-09-13T11:24:58.053 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:24:58.053 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:24:58.053 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:58.053 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 140315 2024-09-13T11:24:58.053 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.053 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:24:58.054 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.054 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:24:58.054 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:58.026180+0000", 2024-09-13T11:24:58.054 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:58.054 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.054 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.054 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:24:58.054 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:58.026180+0000", 2024-09-13T11:24:58.054 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:58.054 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.054 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.054 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:24:58.054 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:58.026182+0000", 2024-09-13T11:24:58.055 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.847e-06 2024-09-13T11:24:58.055 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.055 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.055 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:24:58.055 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:58.026192+0000", 2024-09-13T11:24:58.055 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.1959999999999997e-06 2024-09-13T11:24:58.055 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.055 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.055 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:24:58.055 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:58.026216+0000", 2024-09-13T11:24:58.055 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.4009999999999999e-05 2024-09-13T11:24:58.055 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.055 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.056 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:24:58.056 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:58.026229+0000", 2024-09-13T11:24:58.056 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3682999999999999e-05 2024-09-13T11:24:58.056 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.056 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.056 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:24:58.056 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:58.026381+0000", 2024-09-13T11:24:58.056 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00015135300000000001 2024-09-13T11:24:58.056 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.056 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.056 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:24:58.057 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:58.026785+0000", 2024-09-13T11:24:58.057 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00040434999999999999 2024-09-13T11:24:58.057 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.057 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.057 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 4,5", 2024-09-13T11:24:58.057 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:58.027184+0000", 2024-09-13T11:24:58.057 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00039868299999999999 2024-09-13T11:24:58.057 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.057 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.057 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:24:58.057 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:58.030099+0000", 2024-09-13T11:24:58.057 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.002915726 2024-09-13T11:24:58.058 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.058 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.058 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:58.058 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:58.032085+0000", 2024-09-13T11:24:58.058 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0019854579999999998 2024-09-13T11:24:58.058 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:58.058 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:24:58.058 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:58.058 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.058 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.058 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPushReply(3.29 143/120 [PushReplyOp(3:945cce9a:::benchmark_data_smithi177_31116_object49767:head)])", 2024-09-13T11:24:58.058 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:24:58.046277+0000", 2024-09-13T11:24:58.058 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 0.0047633010000000002, 2024-09-13T11:24:58.059 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0050680550000000001, 2024-09-13T11:24:58.059 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:24:58.059 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:24:58.059 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-09-13T11:24:58.059 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:24:58.059 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.059 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:24:58.059 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:58.046277+0000", 2024-09-13T11:24:58.059 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:58.059 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.059 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.059 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:24:58.059 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:58.046277+0000", 2024-09-13T11:24:58.059 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:58.060 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.060 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.060 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:24:58.060 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:58.046280+0000", 2024-09-13T11:24:58.060 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.9459999999999998e-06 2024-09-13T11:24:58.060 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.060 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.060 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:24:58.060 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:58.046294+0000", 2024-09-13T11:24:58.060 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4344e-05 2024-09-13T11:24:58.060 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.061 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.061 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:24:58.061 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:58.046344+0000", 2024-09-13T11:24:58.061 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.9638999999999997e-05 2024-09-13T11:24:58.061 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.061 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.061 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:24:58.061 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:58.046367+0000", 2024-09-13T11:24:58.061 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.3053000000000001e-05 2024-09-13T11:24:58.061 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.062 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.062 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:24:58.062 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:58.051195+0000", 2024-09-13T11:24:58.062 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0048283079999999999 2024-09-13T11:24:58.062 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:58.062 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:24:58.062 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:58.062 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.062 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.062 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:140339 3.29 3.5722b469 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e144)", 2024-09-13T11:24:58.062 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:24:58.049026+0000", 2024-09-13T11:24:58.062 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 0.0020142860000000001, 2024-09-13T11:24:58.063 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0024979949999999998, 2024-09-13T11:24:58.063 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:24:58.063 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:24:58.063 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "queued for pg", 2024-09-13T11:24:58.063 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:24:58.063 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:24:58.063 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:58.063 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 140339 2024-09-13T11:24:58.063 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.063 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:24:58.063 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.063 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:24:58.063 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:58.049026+0000", 2024-09-13T11:24:58.064 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:58.064 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.064 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.064 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:24:58.064 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:58.049026+0000", 2024-09-13T11:24:58.064 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:58.064 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.064 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.064 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:24:58.064 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:58.049029+0000", 2024-09-13T11:24:58.064 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.8830000000000002e-06 2024-09-13T11:24:58.064 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.064 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.064 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:24:58.065 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:58.049036+0000", 2024-09-13T11:24:58.065 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.3069999999999997e-06 2024-09-13T11:24:58.065 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.065 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.065 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:24:58.065 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:58.049053+0000", 2024-09-13T11:24:58.065 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.6963e-05 2024-09-13T11:24:58.065 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.065 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.065 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:24:58.065 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:58.049064+0000", 2024-09-13T11:24:58.065 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.046e-05 2024-09-13T11:24:58.065 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:58.065 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:24:58.065 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:58.066 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.066 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.066 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop_reply(client.4665.0:140315 3.29 e144/120)", 2024-09-13T11:24:58.066 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:24:58.040377+0000", 2024-09-13T11:24:58.066 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 0.010663232999999999, 2024-09-13T11:24:58.066 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.011282421000000001, 2024-09-13T11:24:58.066 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:24:58.066 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:24:58.066 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "queued for pg", 2024-09-13T11:24:58.066 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:24:58.066 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.066 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:24:58.066 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:58.040377+0000", 2024-09-13T11:24:58.066 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:58.067 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.067 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.067 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:24:58.067 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:58.040377+0000", 2024-09-13T11:24:58.067 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:58.067 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.067 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.067 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:24:58.067 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:58.040379+0000", 2024-09-13T11:24:58.067 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.719e-06 2024-09-13T11:24:58.067 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.067 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.067 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:24:58.067 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:58.040385+0000", 2024-09-13T11:24:58.068 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.913e-06 2024-09-13T11:24:58.068 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.068 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.068 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:24:58.068 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:58.049191+0000", 2024-09-13T11:24:58.068 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0088061730000000005 2024-09-13T11:24:58.068 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.068 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.068 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:24:58.068 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:58.049202+0000", 2024-09-13T11:24:58.068 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0808000000000001e-05 2024-09-13T11:24:58.068 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:58.068 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:24:58.068 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:58.069 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:58.069 INFO:teuthology.orchestra.run.smithi162.stdout: ], 2024-09-13T11:24:58.069 INFO:teuthology.orchestra.run.smithi162.stdout: "num_ops": 4 2024-09-13T11:24:58.069 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-13T11:24:58.070 INFO:tasks.ceph.osd.9.smithi092.stderr:2024-09-13T11:24:58.066+0000 7ff5ab3d0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26348) UID: 0 2024-09-13T11:24:58.071 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-13T11:24:58.171 INFO:tasks.ceph.osd.11.smithi177.stderr:2024-09-13T11:24:58.166+0000 7f24792c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26580) UID: 0 2024-09-13T11:24:58.205 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-13T11:24:58.205 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [], 2024-09-13T11:24:58.205 INFO:teuthology.orchestra.run.smithi162.stdout: "complaint_time": 30, 2024-09-13T11:24:58.205 INFO:teuthology.orchestra.run.smithi162.stdout: "num_blocked_ops": 0 2024-09-13T11:24:58.206 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-13T11:24:58.224 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-13T11:24:58.272 INFO:tasks.ceph.osd.8.smithi001.stderr:2024-09-13T11:24:58.271+0000 7fb9b48ae640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26312) UID: 0 2024-09-13T11:24:58.360 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-13T11:24:58.361 INFO:teuthology.orchestra.run.smithi162.stdout: "size": 20, 2024-09-13T11:24:58.361 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 600, 2024-09-13T11:24:58.361 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [ 2024-09-13T11:24:58.361 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.361 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.4665.0:171 3.9 e41/30 3:9e1c7eeb:::benchmark_data_smithi177_31116_object170:head v 41'13, mlcod=41'9)", 2024-09-13T11:24:58.361 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:22:47.418602+0000", 2024-09-13T11:24:58.361 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 130.939777749, 2024-09-13T11:24:58.361 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.67235367300000004, 2024-09-13T11:24:58.361 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:24:58.361 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:24:58.361 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:58.362 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:24:58.362 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.362 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:24:58.362 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:47.418602+0000", 2024-09-13T11:24:58.362 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:58.362 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.362 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.362 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:24:58.362 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:47.418602+0000", 2024-09-13T11:24:58.362 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:58.362 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.362 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.363 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:24:58.363 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:47.418605+0000", 2024-09-13T11:24:58.363 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.2200000000000001e-06 2024-09-13T11:24:58.363 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.363 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.363 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:24:58.363 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:47.418668+0000", 2024-09-13T11:24:58.363 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.2921000000000003e-05 2024-09-13T11:24:58.363 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.363 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.363 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:24:58.363 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:48.089380+0000", 2024-09-13T11:24:58.364 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.67071191100000005 2024-09-13T11:24:58.364 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.364 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.364 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:24:58.364 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:48.089423+0000", 2024-09-13T11:24:58.364 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.2916e-05 2024-09-13T11:24:58.364 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.364 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.364 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:24:58.364 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:48.089533+0000", 2024-09-13T11:24:58.364 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000109996 2024-09-13T11:24:58.364 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.364 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.365 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:24:58.365 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:48.089570+0000", 2024-09-13T11:24:58.365 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.7314999999999998e-05 2024-09-13T11:24:58.365 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.365 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.365 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:24:58.365 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:48.090852+0000", 2024-09-13T11:24:58.365 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001281641 2024-09-13T11:24:58.365 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.365 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.365 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:24:58.365 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:48.090956+0000", 2024-09-13T11:24:58.365 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00010375299999999999 2024-09-13T11:24:58.366 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:58.366 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:24:58.366 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:58.366 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.366 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.366 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPushReply(3.19 71/70 [PushReplyOp(3:980058cd:::benchmark_data_smithi177_31116_object4309:head)])", 2024-09-13T11:24:58.366 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:25.123140+0000", 2024-09-13T11:24:58.366 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 93.235240070000003, 2024-09-13T11:24:58.366 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.16436381899999999, 2024-09-13T11:24:58.366 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:24:58.366 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:24:58.366 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-09-13T11:24:58.367 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:24:58.367 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.367 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:24:58.367 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.123140+0000", 2024-09-13T11:24:58.367 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:58.367 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.367 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.367 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:24:58.367 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.123140+0000", 2024-09-13T11:24:58.367 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:58.367 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.367 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.367 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:24:58.368 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.123141+0000", 2024-09-13T11:24:58.368 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.562e-06 2024-09-13T11:24:58.368 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.368 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.368 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:24:58.368 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.123151+0000", 2024-09-13T11:24:58.368 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.3030000000000005e-06 2024-09-13T11:24:58.368 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.368 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.368 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:24:58.368 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.285828+0000", 2024-09-13T11:24:58.368 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.16267689399999999 2024-09-13T11:24:58.369 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.369 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.369 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:24:58.369 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.285838+0000", 2024-09-13T11:24:58.369 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0016e-05 2024-09-13T11:24:58.369 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.369 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.369 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:24:58.369 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.285966+0000", 2024-09-13T11:24:58.369 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00012870500000000001 2024-09-13T11:24:58.369 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.369 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.370 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:24:58.370 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:25.287504+0000", 2024-09-13T11:24:58.370 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001537339 2024-09-13T11:24:58.370 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:58.370 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:24:58.371 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:58.371 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.371 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.371 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPushReply(3.19 74/70 [PushReplyOp(3:9b6cad3a:::benchmark_data_smithi177_31116_object16698:head)])", 2024-09-13T11:24:58.372 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:30.313297+0000", 2024-09-13T11:24:58.372 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 88.045083394000002, 2024-09-13T11:24:58.372 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.162355306, 2024-09-13T11:24:58.372 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:24:58.372 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:24:58.372 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-09-13T11:24:58.372 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:24:58.372 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.372 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:24:58.372 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.313297+0000", 2024-09-13T11:24:58.372 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:58.372 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.373 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.373 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:24:58.373 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.313297+0000", 2024-09-13T11:24:58.373 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:58.373 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.373 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.373 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:24:58.373 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.313298+0000", 2024-09-13T11:24:58.373 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.61e-06 2024-09-13T11:24:58.373 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.373 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.373 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:24:58.374 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.313306+0000", 2024-09-13T11:24:58.374 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.9950000000000005e-06 2024-09-13T11:24:58.374 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.374 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.374 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:24:58.374 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.473757+0000", 2024-09-13T11:24:58.374 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.16045083099999999 2024-09-13T11:24:58.374 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.374 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.374 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:24:58.374 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.473764+0000", 2024-09-13T11:24:58.374 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.4769999999999997e-06 2024-09-13T11:24:58.374 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.374 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.375 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:24:58.375 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.473870+0000", 2024-09-13T11:24:58.375 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00010529100000000001 2024-09-13T11:24:58.375 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.375 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.375 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:24:58.375 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.475652+0000", 2024-09-13T11:24:58.375 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001782102 2024-09-13T11:24:58.375 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:58.375 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:24:58.375 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:58.375 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.375 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.376 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPushReply(3.19 74/70 [PushReplyOp(3:9edf939c:::benchmark_data_smithi177_31116_object17207:head)])", 2024-09-13T11:24:58.376 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:30.897971+0000", 2024-09-13T11:24:58.376 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 87.460408897999997, 2024-09-13T11:24:58.376 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.40616071100000001, 2024-09-13T11:24:58.376 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:24:58.376 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:24:58.376 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-09-13T11:24:58.376 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:24:58.376 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.376 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:24:58.376 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.897971+0000", 2024-09-13T11:24:58.376 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:58.377 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.377 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.377 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:24:58.377 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.897971+0000", 2024-09-13T11:24:58.377 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:58.377 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.377 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.377 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:24:58.377 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.897972+0000", 2024-09-13T11:24:58.377 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1429999999999999e-06 2024-09-13T11:24:58.377 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.377 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.378 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:24:58.378 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.897977+0000", 2024-09-13T11:24:58.378 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.5199999999999999e-06 2024-09-13T11:24:58.378 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.378 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.378 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:24:58.378 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:31.302500+0000", 2024-09-13T11:24:58.378 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.40452312400000001 2024-09-13T11:24:58.378 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.378 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.378 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:24:58.378 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:31.302513+0000", 2024-09-13T11:24:58.379 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2753e-05 2024-09-13T11:24:58.379 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.379 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.379 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:24:58.379 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:31.302645+0000", 2024-09-13T11:24:58.379 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00013292100000000001 2024-09-13T11:24:58.379 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.379 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.379 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:24:58.379 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:31.304132+0000", 2024-09-13T11:24:58.379 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0014862499999999999 2024-09-13T11:24:58.379 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:58.380 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:24:58.380 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:58.380 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.380 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.380 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPushReply(3.19 75/70 [PushReplyOp(3:9d1489b1:::benchmark_data_smithi177_31116_object18698:head)])", 2024-09-13T11:24:58.380 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:31.697212+0000", 2024-09-13T11:24:58.380 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 86.661167896999999, 2024-09-13T11:24:58.380 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.22850633200000001, 2024-09-13T11:24:58.380 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:24:58.380 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:24:58.380 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-09-13T11:24:58.380 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:24:58.380 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.381 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:24:58.381 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:31.697212+0000", 2024-09-13T11:24:58.381 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:58.381 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.381 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.381 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:24:58.381 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:31.697212+0000", 2024-09-13T11:24:58.381 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:58.381 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.381 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.381 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:24:58.382 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:31.697214+0000", 2024-09-13T11:24:58.382 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.7439999999999999e-06 2024-09-13T11:24:58.382 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.382 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.382 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:24:58.382 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:31.697224+0000", 2024-09-13T11:24:58.382 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0173000000000001e-05 2024-09-13T11:24:58.382 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.382 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.382 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:24:58.382 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:31.923931+0000", 2024-09-13T11:24:58.382 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.226706926 2024-09-13T11:24:58.382 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.383 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.383 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:24:58.383 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:31.923950+0000", 2024-09-13T11:24:58.383 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.8899e-05 2024-09-13T11:24:58.383 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.383 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.383 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:24:58.383 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:31.924449+0000", 2024-09-13T11:24:58.383 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00049907800000000002 2024-09-13T11:24:58.383 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.383 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.383 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:24:58.383 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:31.925718+0000", 2024-09-13T11:24:58.384 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001269512 2024-09-13T11:24:58.384 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:58.384 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:24:58.384 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:58.384 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.384 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.384 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPushReply(3.19 76/70 [PushReplyOp(3:9faf6d5f:::benchmark_data_smithi177_31116_object18954:head)])", 2024-09-13T11:24:58.384 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:32.042874+0000", 2024-09-13T11:24:58.384 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 86.315506028000001, 2024-09-13T11:24:58.384 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.40212404299999999, 2024-09-13T11:24:58.384 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:24:58.384 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:24:58.384 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-09-13T11:24:58.385 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:24:58.385 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.385 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:24:58.385 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.042874+0000", 2024-09-13T11:24:58.385 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:58.385 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.385 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.385 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:24:58.385 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.042874+0000", 2024-09-13T11:24:58.385 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:58.385 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.385 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.386 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:24:58.386 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.042875+0000", 2024-09-13T11:24:58.386 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.432e-06 2024-09-13T11:24:58.386 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.386 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.386 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:24:58.386 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.042886+0000", 2024-09-13T11:24:58.386 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0265000000000001e-05 2024-09-13T11:24:58.386 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.386 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.386 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:24:58.386 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.443455+0000", 2024-09-13T11:24:58.386 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.40056951400000002 2024-09-13T11:24:58.387 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.387 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.387 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:24:58.387 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.443480+0000", 2024-09-13T11:24:58.387 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.4929000000000001e-05 2024-09-13T11:24:58.387 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.387 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.387 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:24:58.387 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.443666+0000", 2024-09-13T11:24:58.387 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000186446 2024-09-13T11:24:58.387 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.387 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.387 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:24:58.388 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.444998+0000", 2024-09-13T11:24:58.388 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0013314570000000001 2024-09-13T11:24:58.388 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:58.388 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:24:58.388 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:58.388 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.388 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.388 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPush(3.1d 85/78 [PushOp(3:bc4b9336:::benchmark_data_smithi177_31116_object11944:head, version: 49'734, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:bc4b9336:::benchmark_data_smithi177_31116_object11944:head@49'734, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:24:58.388 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:42.225966+0000", 2024-09-13T11:24:58.388 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 76.132413815000007, 2024-09-13T11:24:58.388 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.23879947800000001, 2024-09-13T11:24:58.388 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:24:58.388 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:24:58.389 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:24:58.389 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:24:58.389 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.389 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:24:58.389 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.225966+0000", 2024-09-13T11:24:58.389 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:58.389 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.389 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.389 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:24:58.389 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.225966+0000", 2024-09-13T11:24:58.389 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:58.389 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.389 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.389 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:24:58.390 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.225969+0000", 2024-09-13T11:24:58.390 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.0400000000000001e-06 2024-09-13T11:24:58.390 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.390 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.390 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:24:58.390 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.226036+0000", 2024-09-13T11:24:58.390 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.6978999999999997e-05 2024-09-13T11:24:58.390 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.390 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.390 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:24:58.390 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.463110+0000", 2024-09-13T11:24:58.390 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.23707400000000001 2024-09-13T11:24:58.390 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.391 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.391 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:24:58.391 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.463140+0000", 2024-09-13T11:24:58.391 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.9575999999999999e-05 2024-09-13T11:24:58.391 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.391 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.391 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:24:58.391 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.463326+0000", 2024-09-13T11:24:58.391 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000186298 2024-09-13T11:24:58.391 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.391 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.391 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:24:58.391 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.463351+0000", 2024-09-13T11:24:58.391 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.4862999999999999e-05 2024-09-13T11:24:58.392 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.392 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.392 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:24:58.392 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.464766+0000", 2024-09-13T11:24:58.392 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001414722 2024-09-13T11:24:58.392 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:58.392 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:24:58.392 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:58.392 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.392 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.392 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPush(3.1d 89/78 [PushOp(3:bc1b7cfb:::benchmark_data_smithi177_31116_object22629:head, version: 55'1387, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:bc1b7cfb:::benchmark_data_smithi177_31116_object22629:head@55'1387, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:24:58.393 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:46.660855+0000", 2024-09-13T11:24:58.393 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 71.697525318999993, 2024-09-13T11:24:58.393 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.26622785599999998, 2024-09-13T11:24:58.393 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:24:58.393 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:24:58.393 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:24:58.393 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:24:58.393 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.393 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:24:58.393 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:46.660855+0000", 2024-09-13T11:24:58.393 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:58.393 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.394 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.394 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:24:58.394 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:46.660855+0000", 2024-09-13T11:24:58.394 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:58.394 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.394 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.394 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:24:58.394 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:46.660857+0000", 2024-09-13T11:24:58.394 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.6560000000000001e-06 2024-09-13T11:24:58.394 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.394 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.394 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:24:58.394 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:46.660931+0000", 2024-09-13T11:24:58.394 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.4041999999999993e-05 2024-09-13T11:24:58.395 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.395 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.395 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:24:58.395 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:46.924654+0000", 2024-09-13T11:24:58.395 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.26372239800000002 2024-09-13T11:24:58.395 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.395 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.395 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:24:58.395 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:46.924729+0000", 2024-09-13T11:24:58.395 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.5061999999999994e-05 2024-09-13T11:24:58.395 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.395 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.395 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:24:58.396 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:46.924986+0000", 2024-09-13T11:24:58.396 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00025762000000000002 2024-09-13T11:24:58.396 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.396 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.396 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:24:58.396 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:46.925013+0000", 2024-09-13T11:24:58.396 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.6179000000000001e-05 2024-09-13T11:24:58.396 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.396 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.396 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:24:58.396 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:46.927082+0000", 2024-09-13T11:24:58.397 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.002069899 2024-09-13T11:24:58.397 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:58.397 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:24:58.397 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:58.397 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.397 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.397 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPush(3.1d 90/78 [PushOp(3:bc0e7228:::benchmark_data_smithi177_31116_object24733:head, version: 55'1512, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:bc0e7228:::benchmark_data_smithi177_31116_object24733:head@55'1512, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:24:58.397 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:47.529126+0000", 2024-09-13T11:24:58.397 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 70.829254344999995, 2024-09-13T11:24:58.397 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.21343989499999999, 2024-09-13T11:24:58.397 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:24:58.398 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:24:58.398 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:24:58.398 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:24:58.398 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.398 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:24:58.398 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:47.529126+0000", 2024-09-13T11:24:58.398 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:58.398 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.398 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.398 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:24:58.398 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:47.529126+0000", 2024-09-13T11:24:58.398 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:58.398 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.398 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.398 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:24:58.399 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:47.529127+0000", 2024-09-13T11:24:58.399 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.252e-06 2024-09-13T11:24:58.399 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.399 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.399 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:24:58.399 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:47.529185+0000", 2024-09-13T11:24:58.399 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.8224999999999999e-05 2024-09-13T11:24:58.399 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.399 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.399 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:24:58.399 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:47.740216+0000", 2024-09-13T11:24:58.399 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.21103060800000001 2024-09-13T11:24:58.399 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.400 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.400 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:24:58.400 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:47.740248+0000", 2024-09-13T11:24:58.400 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.2821999999999998e-05 2024-09-13T11:24:58.400 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.400 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.400 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:24:58.400 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:47.740504+0000", 2024-09-13T11:24:58.400 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00025518399999999999 2024-09-13T11:24:58.400 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.400 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.400 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:24:58.400 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:47.740544+0000", 2024-09-13T11:24:58.401 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.0438999999999997e-05 2024-09-13T11:24:58.401 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.401 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.401 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:24:58.401 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:47.742565+0000", 2024-09-13T11:24:58.401 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0020213649999999998 2024-09-13T11:24:58.401 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:58.401 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:24:58.401 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:58.401 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.401 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.401 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:83199 3.1d 3:be2c6fae:::benchmark_data_smithi177_31116_object83198:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106)", 2024-09-13T11:24:58.401 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:24:04.297306+0000", 2024-09-13T11:24:58.402 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 54.061074392999998, 2024-09-13T11:24:58.402 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.25114587999999999, 2024-09-13T11:24:58.402 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:24:58.402 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:24:58.402 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:58.402 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:24:58.402 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:24:58.402 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:58.402 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 83199 2024-09-13T11:24:58.402 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.402 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:24:58.402 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.403 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:24:58.403 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.297306+0000", 2024-09-13T11:24:58.403 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:58.403 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.403 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.403 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:24:58.403 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.297306+0000", 2024-09-13T11:24:58.403 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:58.403 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.403 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.403 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:24:58.403 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.297308+0000", 2024-09-13T11:24:58.404 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.4150000000000002e-06 2024-09-13T11:24:58.404 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.404 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.404 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:24:58.404 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.297352+0000", 2024-09-13T11:24:58.404 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.4431000000000002e-05 2024-09-13T11:24:58.404 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.404 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.404 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:24:58.404 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.297369+0000", 2024-09-13T11:24:58.404 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.6685e-05 2024-09-13T11:24:58.404 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.405 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.405 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:24:58.405 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.297376+0000", 2024-09-13T11:24:58.405 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.7299999999999999e-06 2024-09-13T11:24:58.405 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.405 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.405 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:24:58.405 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.297427+0000", 2024-09-13T11:24:58.405 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.1461000000000001e-05 2024-09-13T11:24:58.405 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.405 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.405 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:24:58.405 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.297528+0000", 2024-09-13T11:24:58.406 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00010028599999999999 2024-09-13T11:24:58.406 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.406 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.406 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 15", 2024-09-13T11:24:58.406 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.297685+0000", 2024-09-13T11:24:58.406 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00015760500000000001 2024-09-13T11:24:58.406 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.406 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.406 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:24:58.406 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.300779+0000", 2024-09-13T11:24:58.406 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0030936700000000002 2024-09-13T11:24:58.406 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.406 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.407 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:58.407 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.548279+0000", 2024-09-13T11:24:58.407 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.24750059999999999 2024-09-13T11:24:58.407 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.407 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.407 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:24:58.407 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.548373+0000", 2024-09-13T11:24:58.407 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.3451999999999999e-05 2024-09-13T11:24:58.408 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.408 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.408 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:24:58.408 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.548451+0000", 2024-09-13T11:24:58.408 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.8545000000000001e-05 2024-09-13T11:24:58.408 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:58.408 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:24:58.408 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:58.409 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.409 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.409 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:84621 3.1d 3:becd72aa:::benchmark_data_smithi177_31116_object84620:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-09-13T11:24:58.409 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:24:05.741008+0000", 2024-09-13T11:24:58.409 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 52.617372392999997, 2024-09-13T11:24:58.409 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.17994101100000001, 2024-09-13T11:24:58.409 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:24:58.409 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:24:58.409 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:58.409 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:24:58.409 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:24:58.409 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:58.409 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 84621 2024-09-13T11:24:58.410 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.410 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:24:58.410 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.410 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:24:58.410 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.741008+0000", 2024-09-13T11:24:58.410 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:58.410 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.410 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.410 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:24:58.410 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.741008+0000", 2024-09-13T11:24:58.410 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:58.410 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.410 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.411 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:24:58.411 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.741010+0000", 2024-09-13T11:24:58.411 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.898e-06 2024-09-13T11:24:58.411 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.411 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.411 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:24:58.411 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.741061+0000", 2024-09-13T11:24:58.411 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.0898e-05 2024-09-13T11:24:58.411 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.411 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.411 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:24:58.411 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.741085+0000", 2024-09-13T11:24:58.411 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.3558e-05 2024-09-13T11:24:58.412 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.412 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.412 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:24:58.412 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.741093+0000", 2024-09-13T11:24:58.412 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.3890000000000005e-06 2024-09-13T11:24:58.412 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.412 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.412 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:24:58.412 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.741178+0000", 2024-09-13T11:24:58.412 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.4792999999999999e-05 2024-09-13T11:24:58.412 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.412 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.412 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:24:58.413 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.741386+0000", 2024-09-13T11:24:58.413 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000207505 2024-09-13T11:24:58.413 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.413 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.413 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 15", 2024-09-13T11:24:58.413 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.741597+0000", 2024-09-13T11:24:58.413 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00021158200000000001 2024-09-13T11:24:58.413 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.413 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.413 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:24:58.413 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.743392+0000", 2024-09-13T11:24:58.413 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00179518 2024-09-13T11:24:58.413 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.413 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.414 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:58.414 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.920866+0000", 2024-09-13T11:24:58.414 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.17747373699999999 2024-09-13T11:24:58.414 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.414 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.414 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:24:58.414 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.920913+0000", 2024-09-13T11:24:58.414 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.6451e-05 2024-09-13T11:24:58.414 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.414 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.414 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:24:58.414 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.920949+0000", 2024-09-13T11:24:58.414 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.6019999999999997e-05 2024-09-13T11:24:58.415 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:58.415 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:24:58.415 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:58.415 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.415 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.415 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:88685 3.1d 3:baceb770:::benchmark_data_smithi177_31116_object88684:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110)", 2024-09-13T11:24:58.415 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:24:08.926194+0000", 2024-09-13T11:24:58.415 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 49.432185748999999, 2024-09-13T11:24:58.415 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.16678378799999999, 2024-09-13T11:24:58.415 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:24:58.416 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:24:58.416 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:58.416 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:24:58.416 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:24:58.416 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:58.416 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 88685 2024-09-13T11:24:58.416 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.416 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:24:58.416 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.416 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:24:58.416 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:08.926194+0000", 2024-09-13T11:24:58.416 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:58.416 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.417 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.417 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:24:58.417 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:08.926194+0000", 2024-09-13T11:24:58.417 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:58.417 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.417 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.417 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:24:58.417 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:08.926196+0000", 2024-09-13T11:24:58.417 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.533e-06 2024-09-13T11:24:58.417 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.417 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.417 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:24:58.417 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:08.926253+0000", 2024-09-13T11:24:58.418 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.6906e-05 2024-09-13T11:24:58.418 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.418 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.418 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:24:58.418 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:08.926262+0000", 2024-09-13T11:24:58.418 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.3530000000000001e-06 2024-09-13T11:24:58.418 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.418 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.418 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:24:58.418 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:08.926268+0000", 2024-09-13T11:24:58.418 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.2890000000000003e-06 2024-09-13T11:24:58.418 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.418 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.418 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:24:58.418 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:08.928510+0000", 2024-09-13T11:24:58.419 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0022416519999999998 2024-09-13T11:24:58.419 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.419 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.419 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:24:58.419 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:08.928748+0000", 2024-09-13T11:24:58.419 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00023773800000000001 2024-09-13T11:24:58.419 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.419 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.419 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 15", 2024-09-13T11:24:58.419 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:08.928953+0000", 2024-09-13T11:24:58.419 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00020532999999999999 2024-09-13T11:24:58.419 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.419 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.419 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:24:58.420 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:08.930702+0000", 2024-09-13T11:24:58.420 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001749163 2024-09-13T11:24:58.420 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.420 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.420 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:58.420 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:09.092878+0000", 2024-09-13T11:24:58.420 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.16217616900000001 2024-09-13T11:24:58.420 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.420 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.420 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:24:58.420 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:09.092932+0000", 2024-09-13T11:24:58.420 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.3523999999999998e-05 2024-09-13T11:24:58.420 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.420 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.421 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:24:58.421 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:09.092978+0000", 2024-09-13T11:24:58.421 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.6131000000000003e-05 2024-09-13T11:24:58.421 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:58.421 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:24:58.421 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:58.421 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.421 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.421 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop_reply(client.4665.0:88685 3.1d e110/102 ondisk, result = 0)", 2024-09-13T11:24:58.421 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:24:08.930404+0000", 2024-09-13T11:24:58.421 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 49.427975437999997, 2024-09-13T11:24:58.422 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.162580424, 2024-09-13T11:24:58.422 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:24:58.422 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:24:58.422 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:24:58.422 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:24:58.422 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.422 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:24:58.422 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:08.930404+0000", 2024-09-13T11:24:58.422 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:58.422 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.422 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.422 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:24:58.423 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:08.930404+0000", 2024-09-13T11:24:58.423 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:58.423 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.423 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.423 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:24:58.423 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:08.930406+0000", 2024-09-13T11:24:58.423 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.8610000000000001e-06 2024-09-13T11:24:58.423 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.423 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.423 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:24:58.423 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:08.930414+0000", 2024-09-13T11:24:58.423 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.2239999999999998e-06 2024-09-13T11:24:58.423 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.424 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.424 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:24:58.424 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:09.092807+0000", 2024-09-13T11:24:58.424 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.16239347900000001 2024-09-13T11:24:58.424 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.424 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.424 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:24:58.424 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:09.092816+0000", 2024-09-13T11:24:58.424 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.5760000000000006e-06 2024-09-13T11:24:58.424 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.424 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.424 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:24:58.424 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:09.092863+0000", 2024-09-13T11:24:58.425 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.7472000000000002e-05 2024-09-13T11:24:58.425 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.425 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.425 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:24:58.425 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:09.092872+0000", 2024-09-13T11:24:58.425 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.0250000000000008e-06 2024-09-13T11:24:58.425 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.425 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.425 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:24:58.425 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:09.092985+0000", 2024-09-13T11:24:58.425 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000112787 2024-09-13T11:24:58.425 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:58.425 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:24:58.426 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:58.426 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.426 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.426 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPush(3.25 124/120 [PushOp(3:a6d4ee3f:::benchmark_data_smithi177_31116_object14712:head, version: 51'872, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:a6d4ee3f:::benchmark_data_smithi177_31116_object14712:head@51'872, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:24:58.426 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:24:31.996253+0000", 2024-09-13T11:24:58.426 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 26.362126621000002, 2024-09-13T11:24:58.426 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.17694863499999999, 2024-09-13T11:24:58.426 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:24:58.426 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:24:58.426 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:24:58.426 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:24:58.426 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.426 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:24:58.426 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:31.996253+0000", 2024-09-13T11:24:58.427 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:58.427 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.427 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.427 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:24:58.427 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:31.996253+0000", 2024-09-13T11:24:58.427 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:58.427 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.427 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.427 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:24:58.427 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:31.996257+0000", 2024-09-13T11:24:58.427 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.2629999999999999e-06 2024-09-13T11:24:58.427 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.428 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.428 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:24:58.428 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:31.996359+0000", 2024-09-13T11:24:58.428 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000102456 2024-09-13T11:24:58.428 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.428 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.428 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:24:58.428 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:32.172313+0000", 2024-09-13T11:24:58.428 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.17595388000000001 2024-09-13T11:24:58.429 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.429 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.429 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:24:58.429 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:32.172326+0000", 2024-09-13T11:24:58.429 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2632999999999999e-05 2024-09-13T11:24:58.429 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.429 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.429 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:24:58.429 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:32.172423+0000", 2024-09-13T11:24:58.429 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.7084999999999996e-05 2024-09-13T11:24:58.429 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.429 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.429 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:24:58.429 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:32.172438+0000", 2024-09-13T11:24:58.430 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5775999999999999e-05 2024-09-13T11:24:58.430 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.430 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.430 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:24:58.430 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:32.173202+0000", 2024-09-13T11:24:58.430 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00076354199999999998 2024-09-13T11:24:58.430 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:58.430 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:24:58.430 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:58.430 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.430 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.430 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPush(3.25 125/120 [PushOp(3:a4446742:::benchmark_data_smithi177_31116_object15486:head, version: 51'933, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:a4446742:::benchmark_data_smithi177_31116_object15486:head@51'933, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:24:58.430 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:24:32.725521+0000", 2024-09-13T11:24:58.431 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 25.632859311000001, 2024-09-13T11:24:58.431 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.16753559600000001, 2024-09-13T11:24:58.431 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:24:58.431 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:24:58.431 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:24:58.431 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:24:58.431 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.431 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:24:58.431 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:32.725521+0000", 2024-09-13T11:24:58.431 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:58.431 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.431 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.431 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:24:58.431 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:32.725521+0000", 2024-09-13T11:24:58.431 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:58.432 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.432 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.432 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:24:58.432 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:32.725523+0000", 2024-09-13T11:24:58.432 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.689e-06 2024-09-13T11:24:58.432 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.432 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.432 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:24:58.432 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:32.725566+0000", 2024-09-13T11:24:58.432 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.3177999999999999e-05 2024-09-13T11:24:58.432 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.432 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.432 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:24:58.432 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:32.892091+0000", 2024-09-13T11:24:58.433 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.16652452200000001 2024-09-13T11:24:58.433 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.433 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.433 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:24:58.433 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:32.892107+0000", 2024-09-13T11:24:58.433 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.6324000000000002e-05 2024-09-13T11:24:58.433 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.433 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.433 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:24:58.433 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:32.892226+0000", 2024-09-13T11:24:58.433 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000118814 2024-09-13T11:24:58.433 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.433 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.434 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:24:58.434 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:32.892241+0000", 2024-09-13T11:24:58.434 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4899000000000001e-05 2024-09-13T11:24:58.434 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.434 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.434 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:24:58.434 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:32.893056+0000", 2024-09-13T11:24:58.434 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00081517000000000004 2024-09-13T11:24:58.434 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:58.434 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:24:58.435 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:58.435 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.435 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.435 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:126039 3.29 3:96100181:::benchmark_data_smithi177_31116_object126038:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e130)", 2024-09-13T11:24:58.435 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:24:43.012785+0000", 2024-09-13T11:24:58.435 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 15.345595271000001, 2024-09-13T11:24:58.435 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.26685311900000003, 2024-09-13T11:24:58.435 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:24:58.435 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:24:58.435 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:58.435 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:24:58.435 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:24:58.436 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:58.436 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 126039 2024-09-13T11:24:58.436 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.436 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:24:58.436 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.436 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:24:58.436 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:43.012785+0000", 2024-09-13T11:24:58.436 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:58.436 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.436 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.436 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:24:58.436 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:43.012785+0000", 2024-09-13T11:24:58.437 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:58.437 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.437 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.437 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:24:58.437 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:43.012787+0000", 2024-09-13T11:24:58.437 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.875e-06 2024-09-13T11:24:58.437 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.437 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.437 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:24:58.437 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:43.012826+0000", 2024-09-13T11:24:58.437 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.9483999999999997e-05 2024-09-13T11:24:58.437 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.437 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.438 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:24:58.438 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:43.012839+0000", 2024-09-13T11:24:58.438 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3405e-05 2024-09-13T11:24:58.438 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.438 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.438 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:24:58.438 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:43.012845+0000", 2024-09-13T11:24:58.438 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.9100000000000002e-06 2024-09-13T11:24:58.438 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.438 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.438 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:24:58.438 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:43.012903+0000", 2024-09-13T11:24:58.439 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.8131000000000003e-05 2024-09-13T11:24:58.439 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.439 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.439 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:24:58.439 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:43.013066+0000", 2024-09-13T11:24:58.439 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000162058 2024-09-13T11:24:58.439 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.439 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.439 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 4,5", 2024-09-13T11:24:58.439 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:43.013235+0000", 2024-09-13T11:24:58.439 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00016929600000000001 2024-09-13T11:24:58.439 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.439 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.439 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:58.440 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:43.017063+0000", 2024-09-13T11:24:58.440 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0038283890000000002 2024-09-13T11:24:58.440 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.440 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.440 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:24:58.440 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:43.017086+0000", 2024-09-13T11:24:58.440 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.2934000000000001e-05 2024-09-13T11:24:58.440 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.440 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.440 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:58.440 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:43.278804+0000", 2024-09-13T11:24:58.440 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.26171751799999998 2024-09-13T11:24:58.440 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.441 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.441 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:24:58.441 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:43.278862+0000", 2024-09-13T11:24:58.441 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.8431999999999999e-05 2024-09-13T11:24:58.441 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.441 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.441 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:24:58.441 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:43.279638+0000", 2024-09-13T11:24:58.441 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00077568699999999995 2024-09-13T11:24:58.441 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:58.441 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:24:58.441 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:58.442 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.442 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.442 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop_reply(client.4665.0:126039 3.29 e130/120 ondisk, result = 0)", 2024-09-13T11:24:58.442 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:24:43.021691+0000", 2024-09-13T11:24:58.442 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 15.336689345, 2024-09-13T11:24:58.442 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.25795579699999999, 2024-09-13T11:24:58.442 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:24:58.442 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:24:58.442 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:24:58.442 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:24:58.442 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.442 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:24:58.442 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:43.021691+0000", 2024-09-13T11:24:58.442 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:58.443 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.443 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.443 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:24:58.443 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:43.021691+0000", 2024-09-13T11:24:58.443 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:58.443 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.443 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.443 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:24:58.443 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:43.021692+0000", 2024-09-13T11:24:58.443 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3880000000000001e-06 2024-09-13T11:24:58.443 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.443 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.443 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:24:58.444 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:43.021700+0000", 2024-09-13T11:24:58.444 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.9310000000000008e-06 2024-09-13T11:24:58.444 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.444 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.444 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:24:58.444 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:43.278719+0000", 2024-09-13T11:24:58.444 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.25701909299999998 2024-09-13T11:24:58.444 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.444 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.444 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:24:58.444 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:43.278728+0000", 2024-09-13T11:24:58.444 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.5000000000000005e-06 2024-09-13T11:24:58.444 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.444 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.445 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:24:58.445 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:43.278786+0000", 2024-09-13T11:24:58.445 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.7640999999999999e-05 2024-09-13T11:24:58.445 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.445 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.445 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:24:58.445 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:43.278796+0000", 2024-09-13T11:24:58.445 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.9199999999999999e-06 2024-09-13T11:24:58.445 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.445 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.445 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:24:58.445 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:43.279646+0000", 2024-09-13T11:24:58.445 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00085032399999999996 2024-09-13T11:24:58.446 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:58.446 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:24:58.446 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:58.446 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.446 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.446 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPushReply(3.29 134/120 [PushReplyOp(3:95648f22:::benchmark_data_smithi177_31116_object7446:head)])", 2024-09-13T11:24:58.446 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:24:47.110443+0000", 2024-09-13T11:24:58.446 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 11.247937224999999, 2024-09-13T11:24:58.446 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.171602386, 2024-09-13T11:24:58.446 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:24:58.446 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:24:58.446 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-09-13T11:24:58.446 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:24:58.447 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.447 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:24:58.447 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:47.110443+0000", 2024-09-13T11:24:58.447 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:58.447 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.447 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.447 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:24:58.447 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:47.110443+0000", 2024-09-13T11:24:58.447 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:58.447 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.447 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.447 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:24:58.447 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:47.110444+0000", 2024-09-13T11:24:58.448 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.192e-06 2024-09-13T11:24:58.448 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.448 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.448 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:24:58.448 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:47.110449+0000", 2024-09-13T11:24:58.448 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.0869999999999997e-06 2024-09-13T11:24:58.448 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.448 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.448 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:24:58.448 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:47.280321+0000", 2024-09-13T11:24:58.448 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.169872157 2024-09-13T11:24:58.448 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.449 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.449 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:24:58.449 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:47.280337+0000", 2024-09-13T11:24:58.449 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5656000000000001e-05 2024-09-13T11:24:58.449 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.449 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.449 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:24:58.449 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:47.280414+0000", 2024-09-13T11:24:58.449 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.7083999999999996e-05 2024-09-13T11:24:58.449 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.449 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.449 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:24:58.450 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:47.282045+0000", 2024-09-13T11:24:58.450 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0016312099999999999 2024-09-13T11:24:58.450 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:58.450 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:24:58.450 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:58.450 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.450 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.450 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.4665.0:138294 3.9 e140/117 3:90012dab:::benchmark_data_smithi177_31116_object138293:head v 140'5246, mlcod=140'5243)", 2024-09-13T11:24:58.450 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:24:54.252227+0000", 2024-09-13T11:24:58.451 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 4.1061528569999997, 2024-09-13T11:24:58.451 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.32919836400000002, 2024-09-13T11:24:58.451 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:24:58.451 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:24:58.451 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:58.451 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:24:58.451 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.451 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:24:58.451 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:54.252227+0000", 2024-09-13T11:24:58.451 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:58.451 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.451 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.452 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:24:58.452 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:54.252227+0000", 2024-09-13T11:24:58.452 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:58.452 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.452 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.452 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:24:58.452 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:54.252228+0000", 2024-09-13T11:24:58.452 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.8899999999999998e-07 2024-09-13T11:24:58.452 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.452 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.452 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:24:58.452 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:54.252276+0000", 2024-09-13T11:24:58.453 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.8137999999999999e-05 2024-09-13T11:24:58.453 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.453 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.453 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:24:58.453 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:54.578216+0000", 2024-09-13T11:24:58.453 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.32594014399999999 2024-09-13T11:24:58.453 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.453 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.453 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:24:58.453 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:54.578235+0000", 2024-09-13T11:24:58.453 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.8943000000000001e-05 2024-09-13T11:24:58.453 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.453 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.454 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:24:58.454 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:54.578396+0000", 2024-09-13T11:24:58.454 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000160922 2024-09-13T11:24:58.454 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.454 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.454 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:24:58.454 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:54.578457+0000", 2024-09-13T11:24:58.454 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.0665999999999998e-05 2024-09-13T11:24:58.454 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.454 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.454 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:24:58.454 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:54.581339+0000", 2024-09-13T11:24:58.454 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0028826839999999999 2024-09-13T11:24:58.455 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.455 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.455 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:24:58.455 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:54.581425+0000", 2024-09-13T11:24:58.455 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.6077999999999998e-05 2024-09-13T11:24:58.455 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:58.455 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:24:58.455 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:58.455 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.455 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.455 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPushReply(3.29 143/120 [PushReplyOp(3:973d2047:::benchmark_data_smithi177_31116_object49749:head)])", 2024-09-13T11:24:58.455 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:24:57.387106+0000", 2024-09-13T11:24:58.455 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 0.97127353000000005, 2024-09-13T11:24:58.456 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.64813637000000002, 2024-09-13T11:24:58.456 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:24:58.456 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:24:58.456 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-09-13T11:24:58.456 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:24:58.456 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.456 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:24:58.456 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:57.387106+0000", 2024-09-13T11:24:58.456 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:58.456 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.457 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.457 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:24:58.457 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:57.387106+0000", 2024-09-13T11:24:58.457 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:58.457 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.457 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.457 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:24:58.457 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:57.387116+0000", 2024-09-13T11:24:58.457 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.1530000000000003e-06 2024-09-13T11:24:58.457 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.457 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.457 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:24:58.457 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:57.387127+0000", 2024-09-13T11:24:58.458 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1197e-05 2024-09-13T11:24:58.458 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.458 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.458 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:24:58.458 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:58.034264+0000", 2024-09-13T11:24:58.458 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.64713691799999995 2024-09-13T11:24:58.458 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.458 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.458 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:24:58.458 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:58.034273+0000", 2024-09-13T11:24:58.458 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.0929999999999992e-06 2024-09-13T11:24:58.458 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.458 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.458 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:24:58.459 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:58.034321+0000", 2024-09-13T11:24:58.459 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.8396999999999998e-05 2024-09-13T11:24:58.459 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.459 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.459 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:24:58.459 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:58.035243+0000", 2024-09-13T11:24:58.459 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00092161200000000004 2024-09-13T11:24:58.459 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:58.459 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:24:58.459 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:58.459 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:58.459 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:24:58.459 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-13T11:24:58.461 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-13T11:24:58.461 INFO:tasks.ceph.osd.5.smithi092.stderr:2024-09-13T11:24:58.454+0000 7f6f0b705640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26346) UID: 0 2024-09-13T11:24:58.525 INFO:teuthology.orchestra.run.smithi001.stdout:osd.6: {} 2024-09-13T11:24:58.525 INFO:teuthology.orchestra.run.smithi001.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T11:24:58.561 INFO:tasks.ceph.osd.3.smithi177.stderr:2024-09-13T11:24:58.558+0000 7f10adba4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26574) UID: 0 2024-09-13T11:24:58.563 INFO:teuthology.orchestra.run.smithi001.stdout:osd.7: {} 2024-09-13T11:24:58.564 INFO:teuthology.orchestra.run.smithi001.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T11:24:58.585 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-13T11:24:58.586 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [], 2024-09-13T11:24:58.586 INFO:teuthology.orchestra.run.smithi162.stdout: "num_ops": 0 2024-09-13T11:24:58.586 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-13T11:24:58.600 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-13T11:24:58.608 INFO:teuthology.orchestra.run.smithi001.stdout:osd.8: {} 2024-09-13T11:24:58.608 INFO:teuthology.orchestra.run.smithi001.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-13T11:24:58.647 INFO:teuthology.orchestra.run.smithi001.stdout:osd.9: {} 2024-09-13T11:24:58.647 INFO:teuthology.orchestra.run.smithi001.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-13T11:24:58.663 INFO:tasks.ceph.osd.15.smithi177.stderr:2024-09-13T11:24:58.658+0000 7f9d6ce77640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26578) UID: 0 2024-09-13T11:24:58.681 INFO:teuthology.orchestra.run.smithi001.stdout:osd.10: {} 2024-09-13T11:24:58.681 INFO:teuthology.orchestra.run.smithi001.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-13T11:24:58.718 INFO:teuthology.orchestra.run.smithi001.stdout:osd.11: {} 2024-09-13T11:24:58.718 INFO:teuthology.orchestra.run.smithi001.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-13T11:24:58.740 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-13T11:24:58.740 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [], 2024-09-13T11:24:58.740 INFO:teuthology.orchestra.run.smithi162.stdout: "complaint_time": 30, 2024-09-13T11:24:58.740 INFO:teuthology.orchestra.run.smithi162.stdout: "num_blocked_ops": 0 2024-09-13T11:24:58.740 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-13T11:24:58.746 INFO:teuthology.orchestra.run.smithi001.stdout:osd.12: {} 2024-09-13T11:24:58.747 INFO:teuthology.orchestra.run.smithi001.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-13T11:24:58.757 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-13T11:24:58.764 INFO:tasks.ceph.osd.12.smithi001.stderr:2024-09-13T11:24:58.763+0000 7f892c6e6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26314) UID: 0 2024-09-13T11:24:58.781 INFO:teuthology.orchestra.run.smithi001.stdout:osd.13: {} 2024-09-13T11:24:58.781 INFO:teuthology.orchestra.run.smithi001.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-13T11:24:58.818 INFO:teuthology.orchestra.run.smithi001.stdout:osd.14: {} 2024-09-13T11:24:58.818 INFO:teuthology.orchestra.run.smithi001.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-13T11:24:58.854 INFO:teuthology.orchestra.run.smithi001.stdout:osd.15: {} 2024-09-13T11:24:58.855 INFO:teuthology.orchestra.run.smithi001.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-13T11:24:58.864 INFO:tasks.ceph.osd.14.smithi162.stderr:2024-09-13T11:24:58.860+0000 7f3df5903640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33154) UID: 0 2024-09-13T11:24:58.893 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-13T11:24:58.893 INFO:teuthology.orchestra.run.smithi162.stdout: "size": 20, 2024-09-13T11:24:58.893 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 600, 2024-09-13T11:24:58.893 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [ 2024-09-13T11:24:58.893 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.893 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4257.0:80 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 262144~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-13T11:24:58.894 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:22:12.516065+0000", 2024-09-13T11:24:58.894 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 166.373172205, 2024-09-13T11:24:58.894 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.46461997999999999, 2024-09-13T11:24:58.894 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:24:58.894 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:24:58.894 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:58.894 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:24:58.894 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4257", 2024-09-13T11:24:58.894 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/1488131527", 2024-09-13T11:24:58.894 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 80 2024-09-13T11:24:58.894 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.894 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:24:58.894 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.895 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:24:58.895 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.516065+0000", 2024-09-13T11:24:58.895 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:58.895 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.895 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.895 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:24:58.895 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.516065+0000", 2024-09-13T11:24:58.895 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:58.895 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.895 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.895 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:24:58.895 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.516066+0000", 2024-09-13T11:24:58.895 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0869999999999999e-06 2024-09-13T11:24:58.895 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.896 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.896 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:24:58.896 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.516112+0000", 2024-09-13T11:24:58.896 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.5503000000000001e-05 2024-09-13T11:24:58.896 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.896 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.896 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:24:58.896 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.516128+0000", 2024-09-13T11:24:58.896 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5891000000000001e-05 2024-09-13T11:24:58.896 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.896 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.896 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:24:58.896 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.516138+0000", 2024-09-13T11:24:58.897 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.8120000000000002e-06 2024-09-13T11:24:58.897 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.897 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.897 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:24:58.897 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.520398+0000", 2024-09-13T11:24:58.897 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0042605450000000001 2024-09-13T11:24:58.897 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.897 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.897 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:24:58.897 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.520526+0000", 2024-09-13T11:24:58.897 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000128355 2024-09-13T11:24:58.897 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.897 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.898 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 0", 2024-09-13T11:24:58.898 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.520730+0000", 2024-09-13T11:24:58.898 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00020337799999999999 2024-09-13T11:24:58.898 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.898 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.898 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:24:58.898 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.522417+0000", 2024-09-13T11:24:58.898 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001687184 2024-09-13T11:24:58.898 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.898 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.898 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:58.898 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.980408+0000", 2024-09-13T11:24:58.898 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.45799068399999998 2024-09-13T11:24:58.899 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.899 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.899 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:24:58.899 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.980565+0000", 2024-09-13T11:24:58.899 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00015760899999999999 2024-09-13T11:24:58.899 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.899 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.899 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:24:58.899 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.980685+0000", 2024-09-13T11:24:58.899 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000119932 2024-09-13T11:24:58.899 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:58.899 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:24:58.899 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:58.900 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.900 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.900 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop_reply(client.4257.0:80 1.0 e24/23 ondisk, result = 0)", 2024-09-13T11:24:58.900 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:22:12.522938+0000", 2024-09-13T11:24:58.900 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 166.366299172, 2024-09-13T11:24:58.900 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.45776420600000001, 2024-09-13T11:24:58.900 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:24:58.900 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:24:58.900 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:24:58.900 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:24:58.900 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.900 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:24:58.900 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.522938+0000", 2024-09-13T11:24:58.901 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:58.901 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.901 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.901 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:24:58.901 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.522938+0000", 2024-09-13T11:24:58.901 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:58.901 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.901 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.901 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:24:58.901 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.522941+0000", 2024-09-13T11:24:58.901 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.0290000000000001e-06 2024-09-13T11:24:58.901 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.901 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.901 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:24:58.902 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.522950+0000", 2024-09-13T11:24:58.902 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.7499999999999992e-06 2024-09-13T11:24:58.902 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.902 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.902 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:24:58.902 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.980215+0000", 2024-09-13T11:24:58.902 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.45726523800000002 2024-09-13T11:24:58.902 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.902 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.902 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:24:58.902 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.980229+0000", 2024-09-13T11:24:58.902 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3549e-05 2024-09-13T11:24:58.902 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.903 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.903 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:24:58.903 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.980365+0000", 2024-09-13T11:24:58.903 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00013608700000000001 2024-09-13T11:24:58.903 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.903 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.903 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:24:58.903 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.980388+0000", 2024-09-13T11:24:58.903 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.3112000000000002e-05 2024-09-13T11:24:58.903 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.903 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.903 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:24:58.903 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.980702+0000", 2024-09-13T11:24:58.903 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00031444099999999999 2024-09-13T11:24:58.904 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:58.904 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:24:58.904 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:58.904 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.904 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.904 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:25416 3.8 3:1a1fbdd1:::benchmark_data_smithi177_31116_object25415:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e56)", 2024-09-13T11:24:58.904 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:04.512188+0000", 2024-09-13T11:24:58.904 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 114.37704920500001, 2024-09-13T11:24:58.904 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.210208697, 2024-09-13T11:24:58.904 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:24:58.904 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:24:58.904 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:58.904 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:24:58.905 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:24:58.905 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:58.905 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 25416 2024-09-13T11:24:58.905 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.905 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:24:58.905 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.905 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:24:58.905 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.512188+0000", 2024-09-13T11:24:58.905 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:58.905 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.905 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.905 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:24:58.905 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.512188+0000", 2024-09-13T11:24:58.905 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:58.906 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.906 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.906 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:24:58.906 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.512192+0000", 2024-09-13T11:24:58.906 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.2660000000000003e-06 2024-09-13T11:24:58.906 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.906 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.906 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:24:58.906 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.512247+0000", 2024-09-13T11:24:58.906 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.4889000000000003e-05 2024-09-13T11:24:58.906 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.906 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.906 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:24:58.907 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.512279+0000", 2024-09-13T11:24:58.907 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.1146000000000003e-05 2024-09-13T11:24:58.907 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.907 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.907 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:24:58.907 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.512297+0000", 2024-09-13T11:24:58.907 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.825e-05 2024-09-13T11:24:58.907 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.907 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.907 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:24:58.907 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.512377+0000", 2024-09-13T11:24:58.907 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.9875000000000004e-05 2024-09-13T11:24:58.907 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.907 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.908 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:24:58.908 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.512570+0000", 2024-09-13T11:24:58.908 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00019312100000000001 2024-09-13T11:24:58.908 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.908 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.908 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 1", 2024-09-13T11:24:58.908 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.512798+0000", 2024-09-13T11:24:58.908 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00022847899999999999 2024-09-13T11:24:58.908 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.908 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.908 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:24:58.908 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.514975+0000", 2024-09-13T11:24:58.908 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0021769979999999999 2024-09-13T11:24:58.909 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.909 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.909 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:58.909 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.722207+0000", 2024-09-13T11:24:58.909 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.20723217099999999 2024-09-13T11:24:58.909 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.909 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.909 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:24:58.909 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.722314+0000", 2024-09-13T11:24:58.909 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00010648400000000001 2024-09-13T11:24:58.909 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.909 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.909 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:24:58.910 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.722397+0000", 2024-09-13T11:24:58.910 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.3017999999999997e-05 2024-09-13T11:24:58.910 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:58.910 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:24:58.910 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:58.910 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.910 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.910 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop_reply(client.4665.0:25416 3.8 e56/40 ondisk, result = 0)", 2024-09-13T11:24:58.910 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:04.515635+0000", 2024-09-13T11:24:58.910 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 114.373602553, 2024-09-13T11:24:58.910 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.20677600500000001, 2024-09-13T11:24:58.910 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:24:58.910 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:24:58.910 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:24:58.911 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:24:58.911 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.911 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:24:58.911 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.515635+0000", 2024-09-13T11:24:58.911 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:58.911 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.911 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.911 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:24:58.911 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.515635+0000", 2024-09-13T11:24:58.911 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:58.911 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.911 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.911 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:24:58.911 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.515639+0000", 2024-09-13T11:24:58.912 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.619e-06 2024-09-13T11:24:58.912 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.912 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.912 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:24:58.912 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.515650+0000", 2024-09-13T11:24:58.912 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.111e-05 2024-09-13T11:24:58.912 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.912 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.912 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:24:58.912 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.722004+0000", 2024-09-13T11:24:58.912 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.206354805 2024-09-13T11:24:58.912 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.912 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.912 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:24:58.913 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.722017+0000", 2024-09-13T11:24:58.913 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2840000000000001e-05 2024-09-13T11:24:58.913 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.913 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.913 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:24:58.913 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.722186+0000", 2024-09-13T11:24:58.913 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00016898299999999999 2024-09-13T11:24:58.913 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.913 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.913 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:24:58.913 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.722199+0000", 2024-09-13T11:24:58.913 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.289e-05 2024-09-13T11:24:58.913 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.914 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.914 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:24:58.914 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.722411+0000", 2024-09-13T11:24:58.914 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00021175799999999999 2024-09-13T11:24:58.915 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:58.915 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:24:58.915 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:58.915 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.915 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.915 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:30833 3.8 3:1c203f8d:::benchmark_data_smithi177_31116_object30832:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e59)", 2024-09-13T11:24:58.915 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:09.506656+0000", 2024-09-13T11:24:58.915 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 109.382581655, 2024-09-13T11:24:58.915 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.224862282, 2024-09-13T11:24:58.915 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:24:58.916 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:24:58.916 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:58.916 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:24:58.916 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:24:58.916 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:58.916 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 30833 2024-09-13T11:24:58.916 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.916 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:24:58.916 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.916 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:24:58.916 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.506656+0000", 2024-09-13T11:24:58.917 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:58.917 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.917 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.917 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:24:58.920 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.506656+0000", 2024-09-13T11:24:58.920 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:58.921 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.921 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.921 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:24:58.921 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.506660+0000", 2024-09-13T11:24:58.921 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.6e-06 2024-09-13T11:24:58.921 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.921 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.921 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:24:58.921 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.506717+0000", 2024-09-13T11:24:58.921 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.6131999999999997e-05 2024-09-13T11:24:58.921 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.921 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.921 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:24:58.922 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.506751+0000", 2024-09-13T11:24:58.922 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.4746000000000002e-05 2024-09-13T11:24:58.922 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.922 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.922 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:24:58.922 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.506774+0000", 2024-09-13T11:24:58.922 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.2636000000000001e-05 2024-09-13T11:24:58.922 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.922 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.922 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:24:58.922 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.506878+0000", 2024-09-13T11:24:58.922 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000104057 2024-09-13T11:24:58.922 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.923 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.923 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:24:58.923 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.507052+0000", 2024-09-13T11:24:58.923 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00017402800000000001 2024-09-13T11:24:58.923 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.923 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.923 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 1", 2024-09-13T11:24:58.923 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.507217+0000", 2024-09-13T11:24:58.923 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00016516 2024-09-13T11:24:58.923 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.923 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.923 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:24:58.923 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.508992+0000", 2024-09-13T11:24:58.924 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0017743990000000001 2024-09-13T11:24:58.924 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.924 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.924 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:58.924 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.731429+0000", 2024-09-13T11:24:58.924 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.22243718000000001 2024-09-13T11:24:58.924 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.924 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.924 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:24:58.924 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.731479+0000", 2024-09-13T11:24:58.924 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.0146999999999999e-05 2024-09-13T11:24:58.924 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.924 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.924 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:24:58.925 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.731518+0000", 2024-09-13T11:24:58.925 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.9196999999999998e-05 2024-09-13T11:24:58.925 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:58.925 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:24:58.925 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:58.925 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.925 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.925 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop_reply(client.4665.0:30833 3.8 e59/40 ondisk, result = 0)", 2024-09-13T11:24:58.925 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:09.508907+0000", 2024-09-13T11:24:58.925 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 109.38033056099999, 2024-09-13T11:24:58.925 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.222618862, 2024-09-13T11:24:58.925 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:24:58.925 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:24:58.926 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:24:58.926 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:24:58.926 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.926 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:24:58.926 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.508907+0000", 2024-09-13T11:24:58.926 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:58.926 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.926 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.926 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:24:58.926 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.508907+0000", 2024-09-13T11:24:58.926 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:58.926 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.926 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.926 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:24:58.927 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.508910+0000", 2024-09-13T11:24:58.927 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.993e-06 2024-09-13T11:24:58.927 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.927 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.927 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:24:58.927 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.508918+0000", 2024-09-13T11:24:58.927 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.7000000000000008e-06 2024-09-13T11:24:58.927 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.927 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.927 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:24:58.927 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.731341+0000", 2024-09-13T11:24:58.927 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.22242308699999999 2024-09-13T11:24:58.927 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.928 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.928 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:24:58.928 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.731350+0000", 2024-09-13T11:24:58.928 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.3179999999999996e-06 2024-09-13T11:24:58.928 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.928 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.928 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:24:58.928 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.731418+0000", 2024-09-13T11:24:58.928 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.7586e-05 2024-09-13T11:24:58.928 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.928 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.928 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:24:58.928 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.731424+0000", 2024-09-13T11:24:58.928 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.844e-06 2024-09-13T11:24:58.929 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.929 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.929 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:24:58.929 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.731526+0000", 2024-09-13T11:24:58.929 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000101334 2024-09-13T11:24:58.929 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:58.929 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:24:58.929 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:58.929 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.929 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.929 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.4665.0:54351 3.10 e74/70 3:085f283e:::benchmark_data_smithi177_31116_object54350:head v 74'3087, mlcod=49'780)", 2024-09-13T11:24:58.929 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:30.599667+0000", 2024-09-13T11:24:58.929 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 88.289570054999999, 2024-09-13T11:24:58.930 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.16617266999999999, 2024-09-13T11:24:58.930 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:24:58.930 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:24:58.930 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:58.930 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:24:58.930 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.930 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:24:58.930 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.599667+0000", 2024-09-13T11:24:58.930 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:58.930 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.930 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.931 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:24:58.931 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.599667+0000", 2024-09-13T11:24:58.931 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:58.931 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.931 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.931 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:24:58.931 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.599669+0000", 2024-09-13T11:24:58.931 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3599999999999999e-06 2024-09-13T11:24:58.931 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.931 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.931 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:24:58.931 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.599712+0000", 2024-09-13T11:24:58.931 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.3723000000000001e-05 2024-09-13T11:24:58.931 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.932 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.932 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:24:58.932 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.762241+0000", 2024-09-13T11:24:58.932 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.16252863400000001 2024-09-13T11:24:58.932 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.932 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.932 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:24:58.932 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.762255+0000", 2024-09-13T11:24:58.932 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3414999999999999e-05 2024-09-13T11:24:58.932 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.932 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.932 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:24:58.932 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.762329+0000", 2024-09-13T11:24:58.933 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.4735999999999997e-05 2024-09-13T11:24:58.933 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.933 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.933 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:24:58.933 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.762358+0000", 2024-09-13T11:24:58.933 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.8308000000000001e-05 2024-09-13T11:24:58.933 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.933 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.933 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:24:58.933 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.765770+0000", 2024-09-13T11:24:58.933 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0034126590000000002 2024-09-13T11:24:58.933 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.933 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.934 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:24:58.934 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.765840+0000", 2024-09-13T11:24:58.934 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.9834999999999998e-05 2024-09-13T11:24:58.934 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:58.934 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:24:58.934 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:58.934 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.934 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.934 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPush(3.10 76/70 [PushOp(3:0e7d2700:::benchmark_data_smithi177_31116_object17406:head, version: 52'1075, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:0e7d2700:::benchmark_data_smithi177_31116_object17406:head@52'1075, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:24:58.934 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:32.206198+0000", 2024-09-13T11:24:58.934 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 86.683039851999993, 2024-09-13T11:24:58.934 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.25193395000000002, 2024-09-13T11:24:58.934 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:24:58.934 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:24:58.935 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:24:58.935 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:24:58.935 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.935 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:24:58.935 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.206198+0000", 2024-09-13T11:24:58.935 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:58.935 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.935 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.935 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:24:58.935 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.206198+0000", 2024-09-13T11:24:58.935 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:58.935 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.935 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.936 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:24:58.936 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.206199+0000", 2024-09-13T11:24:58.936 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.46e-06 2024-09-13T11:24:58.936 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.936 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.936 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:24:58.936 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.206250+0000", 2024-09-13T11:24:58.936 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.113e-05 2024-09-13T11:24:58.936 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.936 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.936 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:24:58.936 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.456042+0000", 2024-09-13T11:24:58.937 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.24979182999999999 2024-09-13T11:24:58.937 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.937 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.937 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:24:58.937 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.456084+0000", 2024-09-13T11:24:58.937 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.1814000000000003e-05 2024-09-13T11:24:58.937 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.937 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.937 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:24:58.937 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.456342+0000", 2024-09-13T11:24:58.937 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000258404 2024-09-13T11:24:58.937 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.937 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.937 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:24:58.938 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.456367+0000", 2024-09-13T11:24:58.938 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.5099000000000001e-05 2024-09-13T11:24:58.938 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.938 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.938 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:24:58.938 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.458132+0000", 2024-09-13T11:24:58.938 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0017642129999999999 2024-09-13T11:24:58.938 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:58.938 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:24:58.938 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:58.938 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.938 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.938 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPush(3.10 76/70 [PushOp(3:0eee8bcf:::benchmark_data_smithi177_31116_object17589:head, version: 53'1086, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:0eee8bcf:::benchmark_data_smithi177_31116_object17589:head@53'1086, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:24:58.938 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:32.585183+0000", 2024-09-13T11:24:58.939 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 86.304053957999997, 2024-09-13T11:24:58.939 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.16696202199999999, 2024-09-13T11:24:58.939 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:24:58.939 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:24:58.939 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:24:58.939 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:24:58.939 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.939 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:24:58.939 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.585183+0000", 2024-09-13T11:24:58.939 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:58.939 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.939 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.939 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:24:58.940 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.585183+0000", 2024-09-13T11:24:58.940 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:58.940 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.940 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.940 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:24:58.940 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.585187+0000", 2024-09-13T11:24:58.940 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.8480000000000003e-06 2024-09-13T11:24:58.940 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.940 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.940 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:24:58.940 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.585264+0000", 2024-09-13T11:24:58.940 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.6327000000000003e-05 2024-09-13T11:24:58.940 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.941 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.941 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:24:58.941 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.750459+0000", 2024-09-13T11:24:58.941 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.16519551900000001 2024-09-13T11:24:58.941 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.941 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.941 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:24:58.941 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.750480+0000", 2024-09-13T11:24:58.941 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.0984000000000001e-05 2024-09-13T11:24:58.941 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.941 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.941 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:24:58.941 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.750633+0000", 2024-09-13T11:24:58.942 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000153148 2024-09-13T11:24:58.942 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.942 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.942 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:24:58.942 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.750652+0000", 2024-09-13T11:24:58.942 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.8981e-05 2024-09-13T11:24:58.942 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.942 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.942 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:24:58.942 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.752145+0000", 2024-09-13T11:24:58.942 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001493215 2024-09-13T11:24:58.942 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:58.942 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:24:58.943 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:58.943 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.943 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.943 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:60417 3.10 3:092808b0:::benchmark_data_smithi177_31116_object60416:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e85)", 2024-09-13T11:24:58.943 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:42.173238+0000", 2024-09-13T11:24:58.943 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 76.715998944000006, 2024-09-13T11:24:58.943 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.38965994599999998, 2024-09-13T11:24:58.943 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:24:58.943 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:24:58.943 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:58.943 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:24:58.943 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:24:58.943 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:58.944 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 60417 2024-09-13T11:24:58.944 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.944 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:24:58.944 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.944 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:24:58.944 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.173238+0000", 2024-09-13T11:24:58.944 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:58.944 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.944 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.944 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:24:58.944 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.173238+0000", 2024-09-13T11:24:58.944 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:58.944 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.944 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.945 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:24:58.945 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.173241+0000", 2024-09-13T11:24:58.945 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.6819999999999999e-06 2024-09-13T11:24:58.945 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.945 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.945 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:24:58.945 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.173335+0000", 2024-09-13T11:24:58.945 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.3957999999999997e-05 2024-09-13T11:24:58.945 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.945 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.945 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:24:58.945 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.173358+0000", 2024-09-13T11:24:58.945 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.3023999999999998e-05 2024-09-13T11:24:58.946 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.946 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.946 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:24:58.946 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.173373+0000", 2024-09-13T11:24:58.946 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4867e-05 2024-09-13T11:24:58.946 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.946 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.946 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:24:58.946 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.173466+0000", 2024-09-13T11:24:58.946 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.2542999999999998e-05 2024-09-13T11:24:58.946 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.946 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.946 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:24:58.946 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.173754+0000", 2024-09-13T11:24:58.947 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000288657 2024-09-13T11:24:58.947 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.947 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.947 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 0", 2024-09-13T11:24:58.947 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.174066+0000", 2024-09-13T11:24:58.947 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00031170999999999998 2024-09-13T11:24:58.947 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.947 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.947 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:24:58.947 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.178245+0000", 2024-09-13T11:24:58.947 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00417864 2024-09-13T11:24:58.947 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.947 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.947 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:58.948 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.562617+0000", 2024-09-13T11:24:58.948 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.38437253199999999 2024-09-13T11:24:58.948 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.948 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.948 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:24:58.948 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.562782+0000", 2024-09-13T11:24:58.948 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00016499200000000001 2024-09-13T11:24:58.948 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.948 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.948 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:24:58.948 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.562898+0000", 2024-09-13T11:24:58.948 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000116341 2024-09-13T11:24:58.948 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:58.949 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:24:58.949 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:58.949 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.949 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.949 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:83193 3.8 3:10066d83:::benchmark_data_smithi177_31116_object83192:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106)", 2024-09-13T11:24:58.949 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:24:04.283923+0000", 2024-09-13T11:24:58.949 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 54.605314260999997, 2024-09-13T11:24:58.949 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.64775463499999997, 2024-09-13T11:24:58.949 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:24:58.949 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:24:58.949 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:58.949 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:24:58.949 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:24:58.950 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:58.950 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 83193 2024-09-13T11:24:58.950 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.950 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:24:58.950 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.950 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:24:58.950 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.283923+0000", 2024-09-13T11:24:58.950 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:58.950 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.950 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.950 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:24:58.950 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.283923+0000", 2024-09-13T11:24:58.950 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:58.950 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.951 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.951 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:24:58.951 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.283925+0000", 2024-09-13T11:24:58.951 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.9489999999999999e-06 2024-09-13T11:24:58.951 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.951 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.951 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:24:58.951 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.283960+0000", 2024-09-13T11:24:58.951 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.5063e-05 2024-09-13T11:24:58.951 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.951 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.951 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:24:58.951 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.283974+0000", 2024-09-13T11:24:58.952 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3546e-05 2024-09-13T11:24:58.952 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.952 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.952 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:24:58.952 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.283983+0000", 2024-09-13T11:24:58.952 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.2499999999999995e-06 2024-09-13T11:24:58.952 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.952 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.952 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:24:58.952 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.284061+0000", 2024-09-13T11:24:58.952 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.7946000000000002e-05 2024-09-13T11:24:58.952 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.952 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.953 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:24:58.953 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.284270+0000", 2024-09-13T11:24:58.953 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00020876799999999999 2024-09-13T11:24:58.953 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.953 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.953 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 1", 2024-09-13T11:24:58.953 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.284489+0000", 2024-09-13T11:24:58.953 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00021956800000000001 2024-09-13T11:24:58.953 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.953 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.953 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:24:58.953 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.286456+0000", 2024-09-13T11:24:58.954 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0019665809999999998 2024-09-13T11:24:58.954 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.954 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.954 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:58.954 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.931497+0000", 2024-09-13T11:24:58.954 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.64504147700000003 2024-09-13T11:24:58.954 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.954 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.954 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:24:58.954 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.931589+0000", 2024-09-13T11:24:58.954 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.2041999999999997e-05 2024-09-13T11:24:58.954 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.954 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.955 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:24:58.955 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.931678+0000", 2024-09-13T11:24:58.955 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.8444999999999998e-05 2024-09-13T11:24:58.955 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:58.955 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:24:58.955 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:58.955 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.955 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.955 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:115060 3.8 3:11200ba9:::benchmark_data_smithi177_31116_object115059:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e125)", 2024-09-13T11:24:58.956 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:24:33.602178+0000", 2024-09-13T11:24:58.956 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 25.287059267, 2024-09-13T11:24:58.956 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.23668186799999999, 2024-09-13T11:24:58.956 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:24:58.956 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:24:58.956 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:58.956 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:24:58.956 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:24:58.956 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:58.956 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 115060 2024-09-13T11:24:58.956 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.956 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:24:58.956 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.957 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:24:58.957 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:33.602178+0000", 2024-09-13T11:24:58.957 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:58.957 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.957 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.957 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:24:58.957 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:33.602178+0000", 2024-09-13T11:24:58.957 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:58.957 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.957 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.957 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:24:58.957 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:33.602181+0000", 2024-09-13T11:24:58.958 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.0369999999999999e-06 2024-09-13T11:24:58.958 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.958 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.958 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:24:58.958 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:33.602217+0000", 2024-09-13T11:24:58.958 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.5837999999999998e-05 2024-09-13T11:24:58.958 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.958 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.958 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:24:58.958 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:33.602235+0000", 2024-09-13T11:24:58.958 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.7571000000000002e-05 2024-09-13T11:24:58.958 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.958 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.958 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:24:58.959 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:33.602245+0000", 2024-09-13T11:24:58.959 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0360000000000001e-05 2024-09-13T11:24:58.959 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.959 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.959 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:24:58.959 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:33.602312+0000", 2024-09-13T11:24:58.959 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.6989000000000005e-05 2024-09-13T11:24:58.959 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.959 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.959 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:24:58.959 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:33.602490+0000", 2024-09-13T11:24:58.959 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000178464 2024-09-13T11:24:58.959 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.960 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.960 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 1", 2024-09-13T11:24:58.960 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:33.602670+0000", 2024-09-13T11:24:58.960 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00018002799999999999 2024-09-13T11:24:58.960 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.960 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.960 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:24:58.960 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:33.604382+0000", 2024-09-13T11:24:58.960 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001711314 2024-09-13T11:24:58.960 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.960 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.960 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:58.960 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:33.838738+0000", 2024-09-13T11:24:58.961 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.23435634699999999 2024-09-13T11:24:58.961 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.961 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.961 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:24:58.961 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:33.838815+0000", 2024-09-13T11:24:58.961 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.7381999999999999e-05 2024-09-13T11:24:58.961 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.961 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.961 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:24:58.961 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:33.838860+0000", 2024-09-13T11:24:58.961 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.4537999999999999e-05 2024-09-13T11:24:58.961 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:58.961 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:24:58.962 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:58.962 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.962 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.962 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop_reply(client.4665.0:115060 3.8 e125/117 ondisk, result = 0)", 2024-09-13T11:24:58.962 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:24:33.604404+0000", 2024-09-13T11:24:58.962 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 25.284833009, 2024-09-13T11:24:58.962 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.23446235400000001, 2024-09-13T11:24:58.962 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:24:58.962 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:24:58.962 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:24:58.962 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:24:58.962 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.962 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:24:58.963 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:33.604404+0000", 2024-09-13T11:24:58.963 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:58.963 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.963 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.963 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:24:58.963 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:33.604404+0000", 2024-09-13T11:24:58.963 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:58.963 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.963 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.963 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:24:58.963 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:33.604406+0000", 2024-09-13T11:24:58.963 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.418e-06 2024-09-13T11:24:58.964 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.964 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.964 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:24:58.964 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:33.604411+0000", 2024-09-13T11:24:58.964 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.3519999999999997e-06 2024-09-13T11:24:58.964 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.964 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.964 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:24:58.964 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:33.838578+0000", 2024-09-13T11:24:58.964 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.23416656199999999 2024-09-13T11:24:58.964 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.964 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.964 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:24:58.965 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:33.838586+0000", 2024-09-13T11:24:58.965 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.9920000000000007e-06 2024-09-13T11:24:58.965 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.965 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.965 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:24:58.965 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:33.838655+0000", 2024-09-13T11:24:58.965 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.9463999999999998e-05 2024-09-13T11:24:58.965 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.965 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.965 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:24:58.965 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:33.838731+0000", 2024-09-13T11:24:58.965 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.5487000000000004e-05 2024-09-13T11:24:58.965 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.966 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.966 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:24:58.966 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:33.838867+0000", 2024-09-13T11:24:58.966 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000136079 2024-09-13T11:24:58.966 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:58.966 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:24:58.966 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:58.966 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.966 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.966 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPush(3.6 129/122 [PushOp(3:6261f3b6:::benchmark_data_smithi177_31116_object64090:head, version: 90'3486, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:6261f3b6:::benchmark_data_smithi177_31116_object64090:head@90'3486, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:24:58.966 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:24:38.641487+0000", 2024-09-13T11:24:58.966 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 20.247750598, 2024-09-13T11:24:58.966 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.209889098, 2024-09-13T11:24:58.967 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:24:58.967 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:24:58.967 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:24:58.967 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:24:58.967 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.967 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:24:58.967 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:38.641487+0000", 2024-09-13T11:24:58.967 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:58.967 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.967 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.967 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:24:58.967 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:38.641487+0000", 2024-09-13T11:24:58.967 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:58.968 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.968 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.968 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:24:58.968 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:38.641489+0000", 2024-09-13T11:24:58.968 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.0980000000000001e-06 2024-09-13T11:24:58.968 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.968 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.968 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:24:58.968 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:38.641560+0000", 2024-09-13T11:24:58.968 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.1404999999999998e-05 2024-09-13T11:24:58.968 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.968 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.968 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:24:58.968 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:38.850498+0000", 2024-09-13T11:24:58.969 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.20893800500000001 2024-09-13T11:24:58.969 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.969 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.969 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:24:58.969 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:38.850510+0000", 2024-09-13T11:24:58.969 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1762e-05 2024-09-13T11:24:58.969 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.969 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.969 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:24:58.969 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:38.850610+0000", 2024-09-13T11:24:58.969 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00010011200000000001 2024-09-13T11:24:58.969 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.970 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.970 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:24:58.970 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:38.850628+0000", 2024-09-13T11:24:58.970 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.7291000000000001e-05 2024-09-13T11:24:58.970 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.970 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.970 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:24:58.970 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:38.851376+0000", 2024-09-13T11:24:58.970 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00074842500000000005 2024-09-13T11:24:58.970 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:58.970 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:24:58.970 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:58.970 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.970 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.971 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.4665.0:126919 3.6 e130/122 3:634ce08e:::benchmark_data_smithi177_31116_object126918:head v 130'5137, mlcod=130'5134)", 2024-09-13T11:24:58.971 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:24:43.780120+0000", 2024-09-13T11:24:58.971 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 15.109117706999999, 2024-09-13T11:24:58.971 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.298086295, 2024-09-13T11:24:58.971 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:24:58.971 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:24:58.971 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:58.971 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:24:58.971 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.971 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:24:58.971 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:43.780120+0000", 2024-09-13T11:24:58.971 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:58.971 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.971 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.972 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:24:58.972 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:43.780120+0000", 2024-09-13T11:24:58.972 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:58.972 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.972 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.972 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:24:58.972 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:43.780123+0000", 2024-09-13T11:24:58.972 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.79e-06 2024-09-13T11:24:58.972 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.972 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.972 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:24:58.972 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:43.780160+0000", 2024-09-13T11:24:58.972 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.7945999999999999e-05 2024-09-13T11:24:58.973 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.973 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.973 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:24:58.973 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:44.075853+0000", 2024-09-13T11:24:58.973 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.29569234999999999 2024-09-13T11:24:58.973 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.973 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.973 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:24:58.973 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:44.075861+0000", 2024-09-13T11:24:58.973 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.9510000000000006e-06 2024-09-13T11:24:58.973 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.973 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.973 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:24:58.973 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:44.075924+0000", 2024-09-13T11:24:58.974 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.2946000000000003e-05 2024-09-13T11:24:58.974 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.974 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.974 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:24:58.974 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:44.075942+0000", 2024-09-13T11:24:58.974 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.8108e-05 2024-09-13T11:24:58.974 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.974 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.974 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:24:58.974 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:44.078164+0000", 2024-09-13T11:24:58.974 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.002222375 2024-09-13T11:24:58.974 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.974 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.975 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:24:58.975 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:44.078206+0000", 2024-09-13T11:24:58.975 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.1829000000000002e-05 2024-09-13T11:24:58.975 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:58.975 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:24:58.975 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:58.975 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.975 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.975 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.4665.0:127275 3.6 e130/122 3:6056789f:::benchmark_data_smithi177_31116_object127274:head v 130'5139, mlcod=130'5136)", 2024-09-13T11:24:58.975 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:24:44.138539+0000", 2024-09-13T11:24:58.975 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 14.750698058999999, 2024-09-13T11:24:58.975 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.159631157, 2024-09-13T11:24:58.975 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:24:58.975 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:24:58.975 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:58.976 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:24:58.976 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.976 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:24:58.976 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:44.138539+0000", 2024-09-13T11:24:58.976 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:58.976 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.976 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.976 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:24:58.976 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:44.138539+0000", 2024-09-13T11:24:58.976 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:58.976 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.976 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.976 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:24:58.976 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:44.138541+0000", 2024-09-13T11:24:58.977 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.9649999999999998e-06 2024-09-13T11:24:58.977 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.977 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.977 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:24:58.977 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:44.138622+0000", 2024-09-13T11:24:58.977 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.0742999999999996e-05 2024-09-13T11:24:58.977 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.977 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.977 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:24:58.977 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:44.296725+0000", 2024-09-13T11:24:58.977 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.15810305899999999 2024-09-13T11:24:58.977 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.977 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.978 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:24:58.978 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:44.296732+0000", 2024-09-13T11:24:58.978 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.3289999999999996e-06 2024-09-13T11:24:58.978 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.978 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.978 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:24:58.978 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:44.296778+0000", 2024-09-13T11:24:58.978 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.5114999999999997e-05 2024-09-13T11:24:58.978 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.978 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.978 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:24:58.978 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:44.296801+0000", 2024-09-13T11:24:58.978 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.3431999999999999e-05 2024-09-13T11:24:58.978 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.979 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.979 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:24:58.979 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:44.298135+0000", 2024-09-13T11:24:58.979 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0013335459999999999 2024-09-13T11:24:58.979 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.979 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.979 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:24:58.979 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:44.298171+0000", 2024-09-13T11:24:58.979 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.5967999999999999e-05 2024-09-13T11:24:58.979 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:58.979 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:24:58.979 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:58.979 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.980 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.980 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:137473 3.10 3:094c0efb:::benchmark_data_smithi177_31116_object137472:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e139)", 2024-09-13T11:24:58.980 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:24:53.523319+0000", 2024-09-13T11:24:58.980 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 5.3659186679999999, 2024-09-13T11:24:58.980 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.18297369299999999, 2024-09-13T11:24:58.980 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:24:58.980 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:24:58.980 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:58.980 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:24:58.980 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:24:58.980 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:58.980 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 137473 2024-09-13T11:24:58.980 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.981 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:24:58.981 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.981 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:24:58.981 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:53.523319+0000", 2024-09-13T11:24:58.981 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:58.981 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.981 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.981 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:24:58.981 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:53.523319+0000", 2024-09-13T11:24:58.981 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:58.981 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.981 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.981 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:24:58.981 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:53.523322+0000", 2024-09-13T11:24:58.982 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.2119999999999999e-06 2024-09-13T11:24:58.982 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.982 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.982 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:24:58.982 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:53.523370+0000", 2024-09-13T11:24:58.982 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.7788999999999999e-05 2024-09-13T11:24:58.982 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.982 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.982 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:24:58.982 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:53.523395+0000", 2024-09-13T11:24:58.982 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.5202e-05 2024-09-13T11:24:58.982 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.982 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.983 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:24:58.983 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:53.523409+0000", 2024-09-13T11:24:58.983 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4263999999999999e-05 2024-09-13T11:24:58.983 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.983 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.983 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:24:58.983 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:53.523477+0000", 2024-09-13T11:24:58.983 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.7578999999999998e-05 2024-09-13T11:24:58.983 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.983 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.983 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:24:58.983 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:53.523638+0000", 2024-09-13T11:24:58.983 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00016078300000000001 2024-09-13T11:24:58.983 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.984 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.984 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 0", 2024-09-13T11:24:58.984 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:53.523800+0000", 2024-09-13T11:24:58.984 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000162564 2024-09-13T11:24:58.984 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.984 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.984 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:24:58.984 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:53.526960+0000", 2024-09-13T11:24:58.984 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0031593950000000002 2024-09-13T11:24:58.984 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.984 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.984 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:58.984 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:53.706180+0000", 2024-09-13T11:24:58.985 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.17922080600000001 2024-09-13T11:24:58.985 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.985 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.985 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:24:58.985 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:53.706237+0000", 2024-09-13T11:24:58.985 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.6845000000000002e-05 2024-09-13T11:24:58.985 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.985 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.985 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:24:58.985 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:53.706292+0000", 2024-09-13T11:24:58.985 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.5254000000000003e-05 2024-09-13T11:24:58.985 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:58.985 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:24:58.986 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:58.986 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.986 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.986 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop_reply(client.4665.0:137473 3.10 e139/84 ondisk, result = 0)", 2024-09-13T11:24:58.986 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:24:53.527076+0000", 2024-09-13T11:24:58.986 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 5.3621618409999998, 2024-09-13T11:24:58.986 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.17922728099999999, 2024-09-13T11:24:58.986 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:24:58.986 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:24:58.986 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:24:58.986 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:24:58.986 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.986 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:24:58.987 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:53.527076+0000", 2024-09-13T11:24:58.987 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:58.987 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.987 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.987 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:24:58.987 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:53.527076+0000", 2024-09-13T11:24:58.987 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:58.987 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.987 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.987 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:24:58.987 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:53.527078+0000", 2024-09-13T11:24:58.988 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.9000000000000002e-06 2024-09-13T11:24:58.988 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.988 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.988 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:24:58.988 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:53.527088+0000", 2024-09-13T11:24:58.988 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.7240000000000004e-06 2024-09-13T11:24:58.988 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.988 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.988 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:24:58.988 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:53.706095+0000", 2024-09-13T11:24:58.989 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.17900633899999999 2024-09-13T11:24:58.989 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.989 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.989 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:24:58.989 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:53.706101+0000", 2024-09-13T11:24:58.989 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.7730000000000002e-06 2024-09-13T11:24:58.989 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.989 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.989 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:24:58.989 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:53.706168+0000", 2024-09-13T11:24:58.989 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.6225999999999999e-05 2024-09-13T11:24:58.989 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.989 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.990 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:24:58.990 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:53.706175+0000", 2024-09-13T11:24:58.990 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.5560000000000002e-06 2024-09-13T11:24:58.990 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.990 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.990 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:24:58.990 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:53.706303+0000", 2024-09-13T11:24:58.990 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000127763 2024-09-13T11:24:58.990 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:58.991 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:24:58.991 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:58.991 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.991 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.991 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:137780 3.8 3:1047f268:::benchmark_data_smithi177_31116_object137779:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e139)", 2024-09-13T11:24:58.991 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:24:53.795212+0000", 2024-09-13T11:24:58.991 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 5.0940259230000002, 2024-09-13T11:24:58.991 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.21530981399999999, 2024-09-13T11:24:58.991 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:24:58.991 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:24:58.991 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:58.991 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:24:58.991 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:24:58.992 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:58.992 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 137780 2024-09-13T11:24:58.992 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.992 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:24:58.992 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.992 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:24:58.992 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:53.795212+0000", 2024-09-13T11:24:58.992 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:58.992 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.993 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.993 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:24:58.993 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:53.795212+0000", 2024-09-13T11:24:58.993 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:58.993 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.993 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.993 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:24:58.993 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:53.795214+0000", 2024-09-13T11:24:58.993 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.481e-06 2024-09-13T11:24:58.993 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.993 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.993 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:24:58.993 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:53.795247+0000", 2024-09-13T11:24:58.994 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.3105999999999998e-05 2024-09-13T11:24:58.994 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.994 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.994 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:24:58.994 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:53.795258+0000", 2024-09-13T11:24:58.994 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0872e-05 2024-09-13T11:24:58.994 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.994 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.994 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:24:58.994 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:53.795265+0000", 2024-09-13T11:24:58.994 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.5729999999999996e-06 2024-09-13T11:24:58.994 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.994 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.994 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:24:58.995 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:53.795309+0000", 2024-09-13T11:24:58.995 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.4617999999999999e-05 2024-09-13T11:24:58.995 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.995 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.995 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:24:58.995 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:53.795397+0000", 2024-09-13T11:24:58.995 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.7863999999999998e-05 2024-09-13T11:24:58.995 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.995 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.995 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 1", 2024-09-13T11:24:58.995 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:53.795493+0000", 2024-09-13T11:24:58.995 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.6062000000000003e-05 2024-09-13T11:24:58.995 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.996 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.996 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:24:58.996 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:53.797245+0000", 2024-09-13T11:24:58.996 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0017521559999999999 2024-09-13T11:24:58.996 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.996 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.996 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:58.996 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:54.010409+0000", 2024-09-13T11:24:58.996 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.21316385299999999 2024-09-13T11:24:58.996 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.996 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.996 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:24:58.996 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:54.010474+0000", 2024-09-13T11:24:58.997 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.4998000000000003e-05 2024-09-13T11:24:58.997 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.997 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.997 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:24:58.997 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:54.010521+0000", 2024-09-13T11:24:58.997 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.7231000000000002e-05 2024-09-13T11:24:58.997 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:58.997 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:24:58.997 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:58.997 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.997 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.997 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop_reply(client.4665.0:137780 3.8 e139/117 ondisk, result = 0)", 2024-09-13T11:24:58.997 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:24:53.797821+0000", 2024-09-13T11:24:58.998 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 5.091416025, 2024-09-13T11:24:58.998 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.21270765599999999, 2024-09-13T11:24:58.998 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:24:58.998 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:24:58.998 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:24:58.998 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:24:58.998 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.998 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:24:58.998 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:53.797821+0000", 2024-09-13T11:24:58.998 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:58.998 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.998 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.998 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:24:58.999 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:53.797821+0000", 2024-09-13T11:24:58.999 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:58.999 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.999 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.999 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:24:58.999 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:53.797822+0000", 2024-09-13T11:24:58.999 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.012e-06 2024-09-13T11:24:58.999 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.999 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:58.999 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:24:58.999 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:53.797829+0000", 2024-09-13T11:24:58.999 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.8009999999999998e-06 2024-09-13T11:24:58.999 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:58.999 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:59.000 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:24:59.000 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:54.010334+0000", 2024-09-13T11:24:59.000 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.212504477 2024-09-13T11:24:59.000 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:59.000 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:59.000 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:24:59.000 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:54.010342+0000", 2024-09-13T11:24:59.000 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.6549999999999993e-06 2024-09-13T11:24:59.000 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:59.000 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:59.000 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:24:59.000 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:54.010394+0000", 2024-09-13T11:24:59.000 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.1875000000000001e-05 2024-09-13T11:24:59.001 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:59.001 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:59.001 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:24:59.001 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:54.010402+0000", 2024-09-13T11:24:59.001 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.6499999999999996e-06 2024-09-13T11:24:59.001 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:59.001 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:59.001 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:24:59.001 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:54.010529+0000", 2024-09-13T11:24:59.001 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000127186 2024-09-13T11:24:59.001 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:59.001 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:24:59.001 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:59.001 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:59.002 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:24:59.002 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-13T11:24:59.003 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-09-13T11:24:59.004 INFO:tasks.ceph.osd.15.smithi177.stderr:2024-09-13T11:24:58.998+0000 7f9d6ce77640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26578) UID: 0 2024-09-13T11:24:59.004 INFO:teuthology.orchestra.run.smithi001.stderr:nodeep-scrub is unset 2024-09-13T11:24:59.005 INFO:tasks.ceph.osd.8.smithi001.stderr:2024-09-13T11:24:59.003+0000 7fb9ab2df640 -1 osd.8 144 handle_osd_map injecting map crc failure 2024-09-13T11:24:59.005 INFO:tasks.ceph.osd.11.smithi177.stderr:2024-09-13T11:24:59.002+0000 7f246bcf0640 -1 osd.11 144 handle_osd_map injecting map crc failure 2024-09-13T11:24:59.038 INFO:tasks.ceph.osd.3.smithi177.stderr:2024-09-13T11:24:59.034+0000 7f10a25d1640 -1 osd.3 144 handle_osd_map injecting map crc failure 2024-09-13T11:24:59.103 INFO:tasks.ceph.osd.14.smithi162.stderr:2024-09-13T11:24:59.100+0000 7f3df5903640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33154) UID: 0 2024-09-13T11:24:59.196 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-13T11:24:59.196 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [ 2024-09-13T11:24:59.197 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:59.197 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:141072 3.18 3:19fa5b7d:::benchmark_data_smithi177_31116_object769:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e145)", 2024-09-13T11:24:59.197 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:24:59.191704+0000", 2024-09-13T11:24:59.197 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 0.0037896760000000001, 2024-09-13T11:24:59.197 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.003820169, 2024-09-13T11:24:59.197 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:24:59.198 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:24:59.198 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "waiting for sub ops", 2024-09-13T11:24:59.198 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:24:59.198 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:24:59.198 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:59.198 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 141072 2024-09-13T11:24:59.199 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:59.199 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:24:59.199 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:59.199 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:24:59.199 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:59.191704+0000", 2024-09-13T11:24:59.200 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:59.200 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:59.200 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:59.200 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:24:59.200 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:59.191704+0000", 2024-09-13T11:24:59.200 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:59.201 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:59.201 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:59.201 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:24:59.201 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:59.191706+0000", 2024-09-13T11:24:59.201 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.708e-06 2024-09-13T11:24:59.201 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:59.202 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:59.202 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:24:59.202 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:59.191711+0000", 2024-09-13T11:24:59.202 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.4630000000000003e-06 2024-09-13T11:24:59.202 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:59.203 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:59.203 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:24:59.203 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:59.191720+0000", 2024-09-13T11:24:59.203 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.5529999999999998e-06 2024-09-13T11:24:59.203 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:59.203 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:59.204 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:24:59.204 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:59.191725+0000", 2024-09-13T11:24:59.204 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.079e-06 2024-09-13T11:24:59.204 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:59.204 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:59.204 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:24:59.204 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:59.191781+0000", 2024-09-13T11:24:59.205 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.5711000000000002e-05 2024-09-13T11:24:59.205 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:59.205 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:59.205 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:24:59.205 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:59.191906+0000", 2024-09-13T11:24:59.205 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000125621 2024-09-13T11:24:59.206 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:59.206 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:59.206 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 0", 2024-09-13T11:24:59.206 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:59.192008+0000", 2024-09-13T11:24:59.206 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000101158 2024-09-13T11:24:59.206 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:59.206 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:59.206 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:24:59.206 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:59.194634+0000", 2024-09-13T11:24:59.206 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0026267690000000002 2024-09-13T11:24:59.206 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:59.207 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:59.207 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:59.207 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:59.195501+0000", 2024-09-13T11:24:59.207 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00086627700000000004 2024-09-13T11:24:59.207 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:59.207 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:24:59.207 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:59.207 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:59.207 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:59.207 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop_reply(client.4665.0:141072 3.18 e145/91 ondisk, result = 0)", 2024-09-13T11:24:59.207 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:24:59.195392+0000", 2024-09-13T11:24:59.207 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 0.000102111, 2024-09-13T11:24:59.207 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000220034, 2024-09-13T11:24:59.208 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:24:59.208 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:24:59.208 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:24:59.208 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:24:59.208 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:59.208 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:24:59.208 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:59.195392+0000", 2024-09-13T11:24:59.208 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:59.208 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:59.208 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:59.208 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:24:59.208 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:59.195392+0000", 2024-09-13T11:24:59.208 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:59.208 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:59.209 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:59.209 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:24:59.209 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:59.195393+0000", 2024-09-13T11:24:59.209 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.412e-06 2024-09-13T11:24:59.209 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:59.209 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:59.209 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:24:59.209 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:59.195402+0000", 2024-09-13T11:24:59.209 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.3559999999999993e-06 2024-09-13T11:24:59.209 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:59.209 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:59.209 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:24:59.209 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:59.195435+0000", 2024-09-13T11:24:59.209 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.3185999999999997e-05 2024-09-13T11:24:59.210 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:59.210 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:59.210 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:24:59.210 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:59.195442+0000", 2024-09-13T11:24:59.210 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.3760000000000002e-06 2024-09-13T11:24:59.210 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:59.210 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:59.210 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:24:59.210 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:59.195489+0000", 2024-09-13T11:24:59.210 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.6582999999999998e-05 2024-09-13T11:24:59.210 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:59.210 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:59.211 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:24:59.211 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:59.195496+0000", 2024-09-13T11:24:59.211 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.7279999999999998e-06 2024-09-13T11:24:59.211 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:59.211 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:24:59.211 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:59.211 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:59.211 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:59.211 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPushReply(3.1f 145/122 [PushReplyOp(3:fa4a5f8b:::benchmark_data_smithi177_31116_object82664:head)])", 2024-09-13T11:24:59.211 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:24:59.195531+0000", 2024-09-13T11:24:59.212 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 4294967295.9999628, 2024-09-13T11:24:59.212 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000139579, 2024-09-13T11:24:59.212 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:24:59.212 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:24:59.212 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-09-13T11:24:59.212 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:24:59.212 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:59.212 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:24:59.212 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:59.195531+0000", 2024-09-13T11:24:59.212 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:59.212 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:59.212 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:59.212 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:24:59.213 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:59.195531+0000", 2024-09-13T11:24:59.213 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:59.213 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:59.213 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:59.213 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:24:59.213 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:59.195533+0000", 2024-09-13T11:24:59.213 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3799999999999999e-06 2024-09-13T11:24:59.213 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:59.213 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:59.213 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:24:59.213 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:59.195539+0000", 2024-09-13T11:24:59.213 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.139e-06 2024-09-13T11:24:59.213 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:59.213 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:59.214 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:24:59.214 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:59.195566+0000", 2024-09-13T11:24:59.214 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.6798e-05 2024-09-13T11:24:59.214 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:59.214 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:59.214 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:24:59.214 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:59.195573+0000", 2024-09-13T11:24:59.214 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.1330000000000001e-06 2024-09-13T11:24:59.214 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:59.214 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:59.214 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:24:59.215 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:59.195632+0000", 2024-09-13T11:24:59.215 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.8823999999999998e-05 2024-09-13T11:24:59.215 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:59.215 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:24:59.215 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:59.215 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:59.215 INFO:teuthology.orchestra.run.smithi162.stdout: ], 2024-09-13T11:24:59.215 INFO:teuthology.orchestra.run.smithi162.stdout: "num_ops": 3 2024-09-13T11:24:59.215 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-13T11:24:59.217 INFO:tasks.ceph.osd.4.smithi001.stderr:2024-09-13T11:24:59.215+0000 7ff8e0218640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26310) UID: 0 2024-09-13T11:24:59.218 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-09-13T11:24:59.317 INFO:tasks.ceph.osd.14.smithi162.stderr:2024-09-13T11:24:59.316+0000 7f3df5903640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33154) UID: 0 2024-09-13T11:24:59.413 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-13T11:24:59.414 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [], 2024-09-13T11:24:59.414 INFO:teuthology.orchestra.run.smithi162.stdout: "complaint_time": 30, 2024-09-13T11:24:59.414 INFO:teuthology.orchestra.run.smithi162.stdout: "num_blocked_ops": 0 2024-09-13T11:24:59.414 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-13T11:24:59.418 INFO:tasks.ceph.osd.8.smithi001.stderr:2024-09-13T11:24:59.415+0000 7fb9b48ae640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26312) UID: 0 2024-09-13T11:24:59.432 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-09-13T11:24:59.521 INFO:tasks.ceph.osd.2.smithi162.stderr:2024-09-13T11:24:59.520+0000 7fe92b35a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26313) UID: 0 2024-09-13T11:24:59.609 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-13T11:24:59.609 INFO:teuthology.orchestra.run.smithi162.stdout: "size": 20, 2024-09-13T11:24:59.610 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 600, 2024-09-13T11:24:59.610 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [ 2024-09-13T11:24:59.610 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:59.610 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:25395 3.3 3:c1df8ccf:::benchmark_data_smithi177_31116_object25394:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e55)", 2024-09-13T11:24:59.610 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:04.465425+0000", 2024-09-13T11:24:59.610 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 115.141218526, 2024-09-13T11:24:59.610 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.18641877200000001, 2024-09-13T11:24:59.610 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:24:59.610 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:24:59.610 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:59.610 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:24:59.610 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:24:59.610 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:59.611 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 25395 2024-09-13T11:24:59.611 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:59.611 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:24:59.611 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:59.611 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:24:59.611 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.465425+0000", 2024-09-13T11:24:59.611 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:59.611 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:59.611 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:59.611 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:24:59.611 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.465425+0000", 2024-09-13T11:24:59.611 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:59.611 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:59.611 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:59.612 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:24:59.612 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.465427+0000", 2024-09-13T11:24:59.612 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.9970000000000001e-06 2024-09-13T11:24:59.612 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:59.612 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:59.612 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:24:59.612 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.465516+0000", 2024-09-13T11:24:59.612 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.8181000000000002e-05 2024-09-13T11:24:59.612 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:59.612 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:59.612 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:24:59.612 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.465534+0000", 2024-09-13T11:24:59.613 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.8204000000000001e-05 2024-09-13T11:24:59.613 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:59.613 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:59.613 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:24:59.613 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.465548+0000", 2024-09-13T11:24:59.613 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4195999999999999e-05 2024-09-13T11:24:59.613 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:59.613 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:59.613 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:24:59.613 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.468249+0000", 2024-09-13T11:24:59.613 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0027014729999999998 2024-09-13T11:24:59.613 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:59.613 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:59.613 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:24:59.614 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.468626+0000", 2024-09-13T11:24:59.614 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000376721 2024-09-13T11:24:59.614 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:59.614 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:59.614 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 7", 2024-09-13T11:24:59.614 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.469016+0000", 2024-09-13T11:24:59.614 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00038975000000000001 2024-09-13T11:24:59.614 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:59.614 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:59.614 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:24:59.614 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.471140+0000", 2024-09-13T11:24:59.614 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0021243400000000002 2024-09-13T11:24:59.614 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:59.615 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:59.615 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:59.615 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.651698+0000", 2024-09-13T11:24:59.615 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.18055781700000001 2024-09-13T11:24:59.615 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:59.615 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:59.615 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:24:59.615 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.651781+0000", 2024-09-13T11:24:59.615 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.3350999999999995e-05 2024-09-13T11:24:59.615 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:59.615 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:59.615 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:24:59.615 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.651844+0000", 2024-09-13T11:24:59.616 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.2742000000000003e-05 2024-09-13T11:24:59.616 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:59.616 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:24:59.616 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:59.616 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:59.616 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:59.616 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:45810 3.17 3.9bd13e57 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e69)", 2024-09-13T11:24:59.616 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:21.347960+0000", 2024-09-13T11:24:59.616 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 98.258683571999995, 2024-09-13T11:24:59.616 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.56814464200000003, 2024-09-13T11:24:59.616 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:24:59.616 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:24:59.616 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-09-13T11:24:59.617 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:24:59.617 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:24:59.617 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:59.617 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 45810 2024-09-13T11:24:59.617 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:59.617 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:24:59.617 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:59.617 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:24:59.617 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:21.347960+0000", 2024-09-13T11:24:59.617 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:59.617 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:59.617 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:59.617 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:24:59.617 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:21.347960+0000", 2024-09-13T11:24:59.618 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:59.618 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:59.618 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:59.618 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:24:59.618 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:21.347965+0000", 2024-09-13T11:24:59.618 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.6720000000000003e-06 2024-09-13T11:24:59.618 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:59.618 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:59.618 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:24:59.618 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:21.348019+0000", 2024-09-13T11:24:59.618 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.4391000000000002e-05 2024-09-13T11:24:59.618 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:59.618 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:59.619 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:24:59.619 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:21.348052+0000", 2024-09-13T11:24:59.619 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.2246000000000003e-05 2024-09-13T11:24:59.619 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:59.619 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:59.619 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:24:59.619 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:21.348072+0000", 2024-09-13T11:24:59.619 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.0622e-05 2024-09-13T11:24:59.619 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:59.619 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:59.619 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:24:59.619 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:21.348278+0000", 2024-09-13T11:24:59.619 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000205391 2024-09-13T11:24:59.619 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:59.620 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:59.620 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-09-13T11:24:59.620 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:21.348292+0000", 2024-09-13T11:24:59.620 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4759e-05 2024-09-13T11:24:59.620 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:59.620 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:59.620 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:24:59.620 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:21.916064+0000", 2024-09-13T11:24:59.620 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.56777128600000004 2024-09-13T11:24:59.620 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:59.620 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:59.620 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:24:59.620 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:21.916105+0000", 2024-09-13T11:24:59.620 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.1275e-05 2024-09-13T11:24:59.621 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:59.621 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:24:59.621 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:59.621 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:59.621 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:59.621 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:54653 3.23 3:c7a54800:::benchmark_data_smithi177_31116_object54652:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e75)", 2024-09-13T11:24:59.621 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:30.979059+0000", 2024-09-13T11:24:59.621 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 88.627585073000006, 2024-09-13T11:24:59.621 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.97284747100000002, 2024-09-13T11:24:59.621 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:24:59.621 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:24:59.621 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-09-13T11:24:59.621 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:24:59.621 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:24:59.622 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:59.622 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 54653 2024-09-13T11:24:59.622 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:59.622 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:24:59.622 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:59.622 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:24:59.622 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.979059+0000", 2024-09-13T11:24:59.622 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:59.622 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:59.622 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:59.622 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:24:59.622 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.979059+0000", 2024-09-13T11:24:59.623 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:59.623 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:59.623 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:59.623 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:24:59.623 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.979061+0000", 2024-09-13T11:24:59.623 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.4279999999999999e-06 2024-09-13T11:24:59.623 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:59.623 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:59.623 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:24:59.623 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.979114+0000", 2024-09-13T11:24:59.623 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.3254000000000002e-05 2024-09-13T11:24:59.623 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:59.623 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:59.624 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:24:59.624 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.979157+0000", 2024-09-13T11:24:59.624 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.2018000000000003e-05 2024-09-13T11:24:59.624 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:59.624 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:59.624 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:24:59.624 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.979172+0000", 2024-09-13T11:24:59.624 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5636000000000001e-05 2024-09-13T11:24:59.624 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:59.624 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:59.624 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:24:59.624 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:31.045109+0000", 2024-09-13T11:24:59.624 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.065936982000000005 2024-09-13T11:24:59.625 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:59.625 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:59.625 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-09-13T11:24:59.625 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:31.045121+0000", 2024-09-13T11:24:59.625 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1608e-05 2024-09-13T11:24:59.625 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:59.625 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:59.625 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:24:59.625 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:31.951636+0000", 2024-09-13T11:24:59.625 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.90651503899999997 2024-09-13T11:24:59.626 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:59.626 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:59.626 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:24:59.626 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:31.951906+0000", 2024-09-13T11:24:59.626 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000270506 2024-09-13T11:24:59.626 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:59.626 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:24:59.626 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:59.626 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:59.626 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:59.626 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPush(3.18 76/70 [PushOp(3:1a32c9cd:::benchmark_data_smithi177_31116_object12833:head, version: 49'818, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:1a32c9cd:::benchmark_data_smithi177_31116_object12833:head@49'818, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:24:59.626 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:32.522305+0000", 2024-09-13T11:24:59.626 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 87.084338989000003, 2024-09-13T11:24:59.627 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.40201207100000003, 2024-09-13T11:24:59.627 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:24:59.627 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:24:59.627 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:24:59.627 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:24:59.627 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:59.627 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:24:59.627 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.522305+0000", 2024-09-13T11:24:59.627 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:59.627 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:59.627 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:59.627 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:24:59.627 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.522305+0000", 2024-09-13T11:24:59.628 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:59.628 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:59.628 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:59.628 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:24:59.628 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.522309+0000", 2024-09-13T11:24:59.628 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.9829999999999998e-06 2024-09-13T11:24:59.628 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:59.628 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:59.628 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:24:59.628 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.522377+0000", 2024-09-13T11:24:59.628 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.8069000000000002e-05 2024-09-13T11:24:59.628 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:59.628 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:59.629 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:24:59.629 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.921950+0000", 2024-09-13T11:24:59.629 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.399573438 2024-09-13T11:24:59.629 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:59.629 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:59.629 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:24:59.629 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.921983+0000", 2024-09-13T11:24:59.629 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.2858000000000003e-05 2024-09-13T11:24:59.629 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:59.629 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:59.629 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:24:59.629 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.922389+0000", 2024-09-13T11:24:59.629 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00040612999999999999 2024-09-13T11:24:59.630 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:59.630 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:59.630 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:24:59.630 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.922421+0000", 2024-09-13T11:24:59.630 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.1356999999999998e-05 2024-09-13T11:24:59.630 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:59.630 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:59.630 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:24:59.630 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.924317+0000", 2024-09-13T11:24:59.630 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0018962359999999999 2024-09-13T11:24:59.630 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:59.630 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:24:59.630 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:59.631 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:59.631 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:59.631 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPush(3.17 76/70 [PushOp(3:ebcbce67:::benchmark_data_smithi177_31116_object18268:head, version: 53'1102, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:ebcbce67:::benchmark_data_smithi177_31116_object18268:head@53'1102, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:24:59.631 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:32.638908+0000", 2024-09-13T11:24:59.631 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 86.967735540999996, 2024-09-13T11:24:59.631 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.229064031, 2024-09-13T11:24:59.631 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:24:59.631 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:24:59.631 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:24:59.631 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:24:59.631 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:59.631 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:24:59.631 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.638908+0000", 2024-09-13T11:24:59.631 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:59.632 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:59.632 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:59.632 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:24:59.632 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.638908+0000", 2024-09-13T11:24:59.632 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:59.632 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:59.632 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:59.632 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:24:59.633 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.638910+0000", 2024-09-13T11:24:59.633 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.7379999999999999e-06 2024-09-13T11:24:59.633 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:59.633 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:59.633 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:24:59.633 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.638975+0000", 2024-09-13T11:24:59.633 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.4937000000000005e-05 2024-09-13T11:24:59.634 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:59.634 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:59.634 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:24:59.634 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.865860+0000", 2024-09-13T11:24:59.634 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.22688509800000001 2024-09-13T11:24:59.634 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:59.634 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:59.634 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:24:59.635 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.865890+0000", 2024-09-13T11:24:59.635 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.0273e-05 2024-09-13T11:24:59.635 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:59.635 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:59.635 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:24:59.635 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.866143+0000", 2024-09-13T11:24:59.635 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00025212800000000001 2024-09-13T11:24:59.635 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:59.636 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:59.636 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:24:59.636 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.866181+0000", 2024-09-13T11:24:59.636 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.8945000000000001e-05 2024-09-13T11:24:59.636 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:59.636 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:59.636 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:24:59.636 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.867972+0000", 2024-09-13T11:24:59.637 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001790912 2024-09-13T11:24:59.637 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:59.637 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:24:59.637 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:59.637 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:59.637 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:59.637 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:57435 3.3 3:c0db4777:::benchmark_data_smithi177_31116_object57434:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e80)", 2024-09-13T11:24:59.637 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:37.436034+0000", 2024-09-13T11:24:59.637 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 82.170610081000007, 2024-09-13T11:24:59.637 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.35949142000000001, 2024-09-13T11:24:59.637 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:24:59.637 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:24:59.638 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:59.638 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:24:59.638 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:24:59.638 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:59.638 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 57435 2024-09-13T11:24:59.638 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:59.638 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:24:59.638 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:59.638 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:24:59.638 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:37.436034+0000", 2024-09-13T11:24:59.638 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:59.638 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:59.638 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:59.639 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:24:59.639 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:37.436034+0000", 2024-09-13T11:24:59.639 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:59.639 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:59.639 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:59.639 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:24:59.639 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:37.436037+0000", 2024-09-13T11:24:59.639 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.7829999999999999e-06 2024-09-13T11:24:59.639 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:59.639 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:59.639 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:24:59.639 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:37.436090+0000", 2024-09-13T11:24:59.639 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.3665999999999997e-05 2024-09-13T11:24:59.639 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:59.640 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:59.640 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:24:59.640 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:37.436112+0000", 2024-09-13T11:24:59.640 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.1556000000000002e-05 2024-09-13T11:24:59.640 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:59.640 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:59.640 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:24:59.640 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:37.436122+0000", 2024-09-13T11:24:59.640 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0098000000000001e-05 2024-09-13T11:24:59.640 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:59.640 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:59.640 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:24:59.640 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:37.436187+0000", 2024-09-13T11:24:59.640 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.4857999999999995e-05 2024-09-13T11:24:59.641 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:59.641 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:59.641 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:24:59.641 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:37.436285+0000", 2024-09-13T11:24:59.641 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.8643999999999999e-05 2024-09-13T11:24:59.642 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:59.642 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:59.642 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 7", 2024-09-13T11:24:59.642 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:37.436423+0000", 2024-09-13T11:24:59.642 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00013809299999999999 2024-09-13T11:24:59.642 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:59.642 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:59.642 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:24:59.642 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:37.440125+0000", 2024-09-13T11:24:59.642 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0037015239999999999 2024-09-13T11:24:59.642 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:59.643 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:59.643 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:59.643 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:37.795397+0000", 2024-09-13T11:24:59.643 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.35527234400000002 2024-09-13T11:24:59.643 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:59.643 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:59.643 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:24:59.643 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:37.795485+0000", 2024-09-13T11:24:59.643 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.7538000000000001e-05 2024-09-13T11:24:59.643 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:59.643 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:59.643 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:24:59.643 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:37.795525+0000", 2024-09-13T11:24:59.643 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.0315999999999998e-05 2024-09-13T11:24:59.644 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:59.644 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:24:59.644 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:59.644 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:59.644 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:59.644 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPush(3.1f 80/78 [PushOp(3:fc3ed0c7:::benchmark_data_smithi177_31116_object672:head, version: 43'36, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:fc3ed0c7:::benchmark_data_smithi177_31116_object672:head@43'36, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:24:59.644 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:37.801754+0000", 2024-09-13T11:24:59.644 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 81.804889720999995, 2024-09-13T11:24:59.644 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.209613519, 2024-09-13T11:24:59.644 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:24:59.645 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:24:59.645 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:24:59.645 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:24:59.645 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:59.645 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:24:59.645 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:37.801754+0000", 2024-09-13T11:24:59.645 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:59.645 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:59.645 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:59.645 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:24:59.645 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:37.801754+0000", 2024-09-13T11:24:59.645 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:59.645 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:59.646 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:59.646 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:24:59.646 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:37.801756+0000", 2024-09-13T11:24:59.646 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.804e-06 2024-09-13T11:24:59.646 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:59.646 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:59.646 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:24:59.646 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:37.801947+0000", 2024-09-13T11:24:59.646 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000190636 2024-09-13T11:24:59.646 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:59.646 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:59.646 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:24:59.646 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:38.010478+0000", 2024-09-13T11:24:59.646 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.20853091100000001 2024-09-13T11:24:59.647 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:59.647 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:59.647 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:24:59.647 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:38.010506+0000", 2024-09-13T11:24:59.647 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.8793999999999999e-05 2024-09-13T11:24:59.647 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:59.647 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:59.647 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:24:59.647 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:38.010601+0000", 2024-09-13T11:24:59.647 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.4786999999999999e-05 2024-09-13T11:24:59.647 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:59.647 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:59.647 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:24:59.647 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:38.010617+0000", 2024-09-13T11:24:59.647 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5790999999999998e-05 2024-09-13T11:24:59.648 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:59.648 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:59.648 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:24:59.648 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:38.011368+0000", 2024-09-13T11:24:59.648 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00075079599999999997 2024-09-13T11:24:59.648 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:59.648 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:24:59.648 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:59.648 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:59.648 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:59.648 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:57985 3.3 3:c330bb03:::benchmark_data_smithi177_31116_object57984:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e80)", 2024-09-13T11:24:59.648 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:38.006112+0000", 2024-09-13T11:24:59.649 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 81.600531853000007, 2024-09-13T11:24:59.649 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.18277887900000001, 2024-09-13T11:24:59.649 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:24:59.649 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:24:59.649 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:59.649 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:24:59.649 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:24:59.649 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:59.649 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 57985 2024-09-13T11:24:59.649 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:59.649 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:24:59.649 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:59.649 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:24:59.650 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:38.006112+0000", 2024-09-13T11:24:59.650 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:59.650 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:59.650 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:59.650 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:24:59.650 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:38.006112+0000", 2024-09-13T11:24:59.650 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:59.650 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:59.650 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:59.650 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:24:59.650 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:38.006115+0000", 2024-09-13T11:24:59.650 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.6400000000000001e-06 2024-09-13T11:24:59.650 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:59.650 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:59.651 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:24:59.651 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:38.006162+0000", 2024-09-13T11:24:59.651 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.7599000000000003e-05 2024-09-13T11:24:59.651 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:59.651 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:59.651 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:24:59.651 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:38.006182+0000", 2024-09-13T11:24:59.651 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.9281999999999999e-05 2024-09-13T11:24:59.651 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:59.651 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:59.651 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:24:59.651 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:38.006191+0000", 2024-09-13T11:24:59.651 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.6490000000000003e-06 2024-09-13T11:24:59.651 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:59.652 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:59.652 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:24:59.652 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:38.006243+0000", 2024-09-13T11:24:59.652 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.1511999999999997e-05 2024-09-13T11:24:59.652 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:59.652 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:59.652 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:24:59.652 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:38.006333+0000", 2024-09-13T11:24:59.652 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.9843e-05 2024-09-13T11:24:59.652 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:59.652 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:59.652 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 7", 2024-09-13T11:24:59.652 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:38.006480+0000", 2024-09-13T11:24:59.652 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00014701699999999999 2024-09-13T11:24:59.653 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:59.653 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:59.653 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:24:59.653 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:38.008806+0000", 2024-09-13T11:24:59.653 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.002326172 2024-09-13T11:24:59.653 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:59.653 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:59.653 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:59.653 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:38.188815+0000", 2024-09-13T11:24:59.653 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.18000961100000001 2024-09-13T11:24:59.653 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:59.653 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:59.653 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:24:59.654 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:38.188857+0000", 2024-09-13T11:24:59.654 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.1591999999999998e-05 2024-09-13T11:24:59.654 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:59.654 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:59.654 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:24:59.654 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:38.188891+0000", 2024-09-13T11:24:59.654 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.3961999999999998e-05 2024-09-13T11:24:59.654 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:59.654 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:24:59.654 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:59.654 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:59.654 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:59.654 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:60433 3.3 3:c01f7ddc:::benchmark_data_smithi177_31116_object60432:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e85)", 2024-09-13T11:24:59.654 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:42.199382+0000", 2024-09-13T11:24:59.655 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 77.407262189999997, 2024-09-13T11:24:59.655 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.33918965200000001, 2024-09-13T11:24:59.655 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:24:59.655 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:24:59.655 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:59.655 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:24:59.655 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:24:59.655 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:59.655 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 60433 2024-09-13T11:24:59.655 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:59.655 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:24:59.655 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:59.655 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:24:59.655 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.199382+0000", 2024-09-13T11:24:59.656 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:59.656 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:59.656 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:59.656 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:24:59.656 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.199382+0000", 2024-09-13T11:24:59.656 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:59.656 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:59.656 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:59.657 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:24:59.657 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.199386+0000", 2024-09-13T11:24:59.657 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.8759999999999998e-06 2024-09-13T11:24:59.657 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:59.657 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:59.657 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:24:59.657 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.199455+0000", 2024-09-13T11:24:59.657 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.9305000000000001e-05 2024-09-13T11:24:59.657 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:59.657 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:59.657 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:24:59.657 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.199482+0000", 2024-09-13T11:24:59.657 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.6749000000000001e-05 2024-09-13T11:24:59.658 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:59.658 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:59.658 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:24:59.658 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.199503+0000", 2024-09-13T11:24:59.658 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.1137000000000001e-05 2024-09-13T11:24:59.658 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:59.658 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:59.658 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:24:59.658 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.199602+0000", 2024-09-13T11:24:59.658 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.8743e-05 2024-09-13T11:24:59.658 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:59.658 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:59.658 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:24:59.658 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.199784+0000", 2024-09-13T11:24:59.659 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00018265100000000001 2024-09-13T11:24:59.659 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:59.659 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:59.659 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 7", 2024-09-13T11:24:59.659 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.199955+0000", 2024-09-13T11:24:59.659 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00017067100000000001 2024-09-13T11:24:59.659 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:59.659 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:59.659 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:24:59.659 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.202075+0000", 2024-09-13T11:24:59.659 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0021206749999999998 2024-09-13T11:24:59.659 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:59.659 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:59.660 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:59.660 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.538395+0000", 2024-09-13T11:24:59.660 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.33631963599999998 2024-09-13T11:24:59.660 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:59.660 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:59.660 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:24:59.660 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.538498+0000", 2024-09-13T11:24:59.660 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00010267099999999999 2024-09-13T11:24:59.660 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:59.660 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:59.660 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:24:59.660 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.538571+0000", 2024-09-13T11:24:59.660 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.3538e-05 2024-09-13T11:24:59.660 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:59.661 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:24:59.661 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:59.661 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:59.661 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:59.661 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop_reply(client.4665.0:60433 3.3 e85/75 ondisk, result = 0)", 2024-09-13T11:24:59.661 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:42.203186+0000", 2024-09-13T11:24:59.661 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 77.403458012000002, 2024-09-13T11:24:59.661 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.33540096699999999, 2024-09-13T11:24:59.661 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:24:59.661 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:24:59.661 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:24:59.661 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:24:59.661 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:59.662 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:24:59.662 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.203186+0000", 2024-09-13T11:24:59.662 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:59.662 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:59.662 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:59.662 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:24:59.662 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.203186+0000", 2024-09-13T11:24:59.662 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:59.662 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:59.662 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:59.662 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:24:59.662 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.203194+0000", 2024-09-13T11:24:59.662 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.0020000000000006e-06 2024-09-13T11:24:59.663 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:59.663 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:59.663 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:24:59.663 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.203202+0000", 2024-09-13T11:24:59.663 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.9829999999999996e-06 2024-09-13T11:24:59.663 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:59.664 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:59.664 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:24:59.664 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.538182+0000", 2024-09-13T11:24:59.664 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.33498029899999998 2024-09-13T11:24:59.664 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:59.664 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:59.665 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:24:59.665 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.538196+0000", 2024-09-13T11:24:59.665 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3807e-05 2024-09-13T11:24:59.665 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:59.665 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:59.665 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:24:59.666 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.538371+0000", 2024-09-13T11:24:59.666 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00017541200000000001 2024-09-13T11:24:59.666 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:59.666 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:59.666 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:24:59.666 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.538386+0000", 2024-09-13T11:24:59.666 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4833e-05 2024-09-13T11:24:59.667 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:59.667 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:59.667 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:24:59.667 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.538587+0000", 2024-09-13T11:24:59.667 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000200631 2024-09-13T11:24:59.667 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:59.667 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:24:59.667 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:59.668 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:59.668 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:59.668 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:69556 3.18 3:1d3dbeba:::benchmark_data_smithi177_31116_object69555:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94)", 2024-09-13T11:24:59.668 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:52.142180+0000", 2024-09-13T11:24:59.668 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 67.464463941999995, 2024-09-13T11:24:59.668 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.21280995899999999, 2024-09-13T11:24:59.668 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:24:59.669 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:24:59.669 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:59.669 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:24:59.669 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:24:59.669 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:59.669 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 69556 2024-09-13T11:24:59.670 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:59.670 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:24:59.670 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:59.670 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:24:59.670 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:52.142180+0000", 2024-09-13T11:24:59.670 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:59.671 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:59.671 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:59.671 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:24:59.671 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:52.142180+0000", 2024-09-13T11:24:59.671 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:59.671 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:59.671 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:59.671 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:24:59.672 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:52.142181+0000", 2024-09-13T11:24:59.672 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.3699999999999999e-07 2024-09-13T11:24:59.672 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:59.672 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:59.672 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:24:59.672 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:52.142212+0000", 2024-09-13T11:24:59.672 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.1334000000000002e-05 2024-09-13T11:24:59.672 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:59.672 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:59.673 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:24:59.673 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:52.142219+0000", 2024-09-13T11:24:59.673 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.9990000000000002e-06 2024-09-13T11:24:59.673 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:59.673 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:59.673 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:24:59.673 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:52.142224+0000", 2024-09-13T11:24:59.673 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.0259999999999998e-06 2024-09-13T11:24:59.673 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:59.674 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:59.674 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:24:59.674 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:52.142276+0000", 2024-09-13T11:24:59.674 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.1436e-05 2024-09-13T11:24:59.674 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:59.674 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:59.674 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:24:59.674 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:52.142396+0000", 2024-09-13T11:24:59.674 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00012011 2024-09-13T11:24:59.674 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:59.674 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:59.674 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 0", 2024-09-13T11:24:59.674 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:52.142503+0000", 2024-09-13T11:24:59.675 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000107701 2024-09-13T11:24:59.675 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:59.675 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:59.675 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:24:59.675 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:52.144566+0000", 2024-09-13T11:24:59.675 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.002062749 2024-09-13T11:24:59.675 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:59.675 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:59.675 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:59.675 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:52.354839+0000", 2024-09-13T11:24:59.675 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.21027254000000001 2024-09-13T11:24:59.675 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:59.675 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:59.676 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:24:59.676 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:52.354932+0000", 2024-09-13T11:24:59.676 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.3510000000000004e-05 2024-09-13T11:24:59.676 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:59.676 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:59.676 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:24:59.676 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:52.354990+0000", 2024-09-13T11:24:59.676 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.7617000000000001e-05 2024-09-13T11:24:59.676 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:59.676 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:24:59.676 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:59.676 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:59.677 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:59.677 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:82784 3.18 3:1e10e57e:::benchmark_data_smithi177_31116_object82783:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105)", 2024-09-13T11:24:59.677 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:24:03.815559+0000", 2024-09-13T11:24:59.677 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 55.791084544, 2024-09-13T11:24:59.677 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.19711926199999999, 2024-09-13T11:24:59.677 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:24:59.677 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:24:59.677 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:59.677 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:24:59.677 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:24:59.677 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:59.677 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 82784 2024-09-13T11:24:59.677 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:59.678 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:24:59.678 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:59.678 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:24:59.678 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:03.815559+0000", 2024-09-13T11:24:59.678 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:59.678 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:59.678 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:59.678 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:24:59.678 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:03.815559+0000", 2024-09-13T11:24:59.678 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:59.678 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:59.678 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:59.678 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:24:59.679 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:03.815568+0000", 2024-09-13T11:24:59.679 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.2029999999999992e-06 2024-09-13T11:24:59.679 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:59.679 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:59.679 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:24:59.679 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:03.815618+0000", 2024-09-13T11:24:59.679 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.0285999999999999e-05 2024-09-13T11:24:59.679 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:59.679 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:59.679 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:24:59.679 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:03.815646+0000", 2024-09-13T11:24:59.679 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.7747e-05 2024-09-13T11:24:59.680 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:59.680 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:59.680 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:24:59.680 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:03.815660+0000", 2024-09-13T11:24:59.680 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4633e-05 2024-09-13T11:24:59.680 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:59.680 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:59.680 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:24:59.680 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:03.815781+0000", 2024-09-13T11:24:59.680 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000120831 2024-09-13T11:24:59.680 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:59.680 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:59.680 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:24:59.681 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:03.816015+0000", 2024-09-13T11:24:59.681 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00023364600000000001 2024-09-13T11:24:59.681 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:59.681 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:59.681 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 0", 2024-09-13T11:24:59.681 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:03.816241+0000", 2024-09-13T11:24:59.681 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00022667600000000001 2024-09-13T11:24:59.681 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:59.681 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:59.681 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:24:59.681 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:03.819236+0000", 2024-09-13T11:24:59.681 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0029943130000000002 2024-09-13T11:24:59.681 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:59.682 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:59.682 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:59.682 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.012545+0000", 2024-09-13T11:24:59.682 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.19330974100000001 2024-09-13T11:24:59.682 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:59.682 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:59.682 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:24:59.682 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.012623+0000", 2024-09-13T11:24:59.682 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.7565999999999996e-05 2024-09-13T11:24:59.682 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:59.682 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:59.682 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:24:59.682 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.012679+0000", 2024-09-13T11:24:59.682 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.5619999999999999e-05 2024-09-13T11:24:59.682 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:59.683 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:24:59.683 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:59.683 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:59.683 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:59.683 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop_reply(client.4665.0:82784 3.18 e105/91 ondisk, result = 0)", 2024-09-13T11:24:59.683 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:24:03.819384+0000", 2024-09-13T11:24:59.683 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 55.787260263999997, 2024-09-13T11:24:59.683 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.19330499100000001, 2024-09-13T11:24:59.683 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:24:59.683 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:24:59.683 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:24:59.683 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:24:59.683 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:59.684 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:24:59.684 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:03.819384+0000", 2024-09-13T11:24:59.684 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:59.684 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:59.684 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:59.684 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:24:59.684 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:03.819384+0000", 2024-09-13T11:24:59.684 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:59.684 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:59.684 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:59.685 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:24:59.685 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:03.819393+0000", 2024-09-13T11:24:59.685 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.5429999999999999e-06 2024-09-13T11:24:59.685 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:59.685 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:59.685 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:24:59.685 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:03.819403+0000", 2024-09-13T11:24:59.685 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0309000000000001e-05 2024-09-13T11:24:59.685 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:59.685 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:59.685 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:24:59.685 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.010699+0000", 2024-09-13T11:24:59.685 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.19129547699999999 2024-09-13T11:24:59.685 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:59.686 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:59.686 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:24:59.686 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.010708+0000", 2024-09-13T11:24:59.686 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.6770000000000001e-06 2024-09-13T11:24:59.686 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:59.686 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:59.686 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:24:59.686 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.012528+0000", 2024-09-13T11:24:59.686 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00182087 2024-09-13T11:24:59.686 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:59.686 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:59.686 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:24:59.686 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.012538+0000", 2024-09-13T11:24:59.686 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.6399999999999992e-06 2024-09-13T11:24:59.687 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:59.687 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:59.687 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:24:59.687 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.012689+0000", 2024-09-13T11:24:59.687 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00015047500000000001 2024-09-13T11:24:59.687 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:59.687 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:24:59.687 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:59.687 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:59.687 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:59.688 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:83144 3.18 3:1c2a5085:::benchmark_data_smithi177_31116_object83143:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106)", 2024-09-13T11:24:59.688 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:24:04.186395+0000", 2024-09-13T11:24:59.688 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 55.420248893, 2024-09-13T11:24:59.688 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.29655527599999998, 2024-09-13T11:24:59.688 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:24:59.688 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:24:59.688 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:59.688 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:24:59.688 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:24:59.688 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:59.688 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 83144 2024-09-13T11:24:59.689 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:59.689 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:24:59.689 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:59.689 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:24:59.689 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.186395+0000", 2024-09-13T11:24:59.689 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:59.689 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:59.689 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:59.689 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:24:59.689 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.186395+0000", 2024-09-13T11:24:59.689 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:59.689 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:59.689 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:59.689 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:24:59.690 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.186398+0000", 2024-09-13T11:24:59.690 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.5299999999999999e-06 2024-09-13T11:24:59.690 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:59.690 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:59.690 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:24:59.690 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.186438+0000", 2024-09-13T11:24:59.690 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.0293000000000002e-05 2024-09-13T11:24:59.691 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:59.691 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:59.691 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:24:59.691 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.186450+0000", 2024-09-13T11:24:59.691 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2687000000000001e-05 2024-09-13T11:24:59.691 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:59.692 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:59.692 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:24:59.692 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.186456+0000", 2024-09-13T11:24:59.692 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.446e-06 2024-09-13T11:24:59.692 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:59.692 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:59.693 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:24:59.693 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.186503+0000", 2024-09-13T11:24:59.693 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.7491000000000003e-05 2024-09-13T11:24:59.693 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:59.693 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:59.693 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:24:59.693 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.186618+0000", 2024-09-13T11:24:59.693 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000114999 2024-09-13T11:24:59.694 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:59.694 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:59.694 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 0", 2024-09-13T11:24:59.694 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.186712+0000", 2024-09-13T11:24:59.694 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.3805000000000001e-05 2024-09-13T11:24:59.694 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:59.694 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:59.695 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:24:59.695 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.189753+0000", 2024-09-13T11:24:59.695 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.003040713 2024-09-13T11:24:59.695 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:59.695 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:59.695 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:59.695 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.482662+0000", 2024-09-13T11:24:59.695 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.29290945699999998 2024-09-13T11:24:59.695 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:59.695 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:59.696 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:24:59.696 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.482831+0000", 2024-09-13T11:24:59.696 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00016900699999999999 2024-09-13T11:24:59.696 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:59.696 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:59.696 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:24:59.696 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.482950+0000", 2024-09-13T11:24:59.697 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000118848 2024-09-13T11:24:59.697 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:59.697 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:24:59.697 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:59.697 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:59.698 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:59.698 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop_reply(client.4665.0:83144 3.18 e106/91 ondisk, result = 0)", 2024-09-13T11:24:59.698 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:24:04.188413+0000", 2024-09-13T11:24:59.698 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 55.418230883, 2024-09-13T11:24:59.698 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.294557562, 2024-09-13T11:24:59.698 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:24:59.699 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:24:59.699 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:24:59.699 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:24:59.699 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:59.699 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:24:59.700 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.188413+0000", 2024-09-13T11:24:59.700 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:59.700 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:59.700 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:59.700 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:24:59.701 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.188413+0000", 2024-09-13T11:24:59.701 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:59.701 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:59.701 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:59.701 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:24:59.701 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.188414+0000", 2024-09-13T11:24:59.702 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4309999999999999e-06 2024-09-13T11:24:59.702 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:59.702 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:59.702 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:24:59.702 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.188419+0000", 2024-09-13T11:24:59.703 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.2889999999999999e-06 2024-09-13T11:24:59.703 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:59.703 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:59.703 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:24:59.703 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.482442+0000", 2024-09-13T11:24:59.704 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.294023168 2024-09-13T11:24:59.704 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:59.704 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:59.704 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:24:59.704 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.482459+0000", 2024-09-13T11:24:59.704 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.7013000000000001e-05 2024-09-13T11:24:59.705 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:59.705 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:59.705 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:24:59.705 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.482615+0000", 2024-09-13T11:24:59.705 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00015620400000000001 2024-09-13T11:24:59.706 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:59.706 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:59.706 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:24:59.706 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.482644+0000", 2024-09-13T11:24:59.706 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.9093000000000001e-05 2024-09-13T11:24:59.707 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:59.707 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:59.707 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:24:59.707 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.482971+0000", 2024-09-13T11:24:59.707 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00032636399999999999 2024-09-13T11:24:59.707 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:59.708 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:24:59.708 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:59.708 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:59.708 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:59.708 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPush(3.1f 106/78 [PushOp(3:fd4be819:::benchmark_data_smithi177_31116_object54560:head, version: 74'3410, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:fd4be819:::benchmark_data_smithi177_31116_object54560:head@74'3410, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:24:59.709 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:24:04.397161+0000", 2024-09-13T11:24:59.709 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 55.209482473999998, 2024-09-13T11:24:59.709 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.51611096199999995, 2024-09-13T11:24:59.709 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:24:59.709 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:24:59.709 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:24:59.710 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:24:59.710 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:59.710 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:24:59.710 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.397161+0000", 2024-09-13T11:24:59.710 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:59.710 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:59.711 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:59.711 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:24:59.711 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.397161+0000", 2024-09-13T11:24:59.711 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:59.711 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:59.712 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:59.712 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:24:59.712 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.397163+0000", 2024-09-13T11:24:59.712 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4720000000000001e-06 2024-09-13T11:24:59.712 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:59.712 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:59.713 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:24:59.713 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.397228+0000", 2024-09-13T11:24:59.713 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.5141999999999994e-05 2024-09-13T11:24:59.713 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:59.713 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:59.713 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:24:59.713 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.911583+0000", 2024-09-13T11:24:59.713 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.51435473700000001 2024-09-13T11:24:59.713 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:59.713 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:59.713 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:24:59.713 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.911615+0000", 2024-09-13T11:24:59.714 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.2620000000000003e-05 2024-09-13T11:24:59.714 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:59.714 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:59.714 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:24:59.714 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.911759+0000", 2024-09-13T11:24:59.714 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00014376300000000001 2024-09-13T11:24:59.714 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:59.714 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:59.714 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:24:59.714 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.911779+0000", 2024-09-13T11:24:59.714 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.0003000000000001e-05 2024-09-13T11:24:59.714 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:59.714 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:59.715 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:24:59.715 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.913272+0000", 2024-09-13T11:24:59.715 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0014932249999999999 2024-09-13T11:24:59.715 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:59.715 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:24:59.715 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:59.715 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:59.715 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:59.715 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:84809 3.17 3:e9e52123:::benchmark_data_smithi177_31116_object84808:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-09-13T11:24:59.715 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:24:05.914353+0000", 2024-09-13T11:24:59.715 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 53.692290456000002, 2024-09-13T11:24:59.715 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.27210915899999999, 2024-09-13T11:24:59.715 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:24:59.715 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:24:59.716 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:59.716 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:24:59.716 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:24:59.716 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:59.716 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 84809 2024-09-13T11:24:59.716 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:59.716 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:24:59.716 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:59.716 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:24:59.716 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.914353+0000", 2024-09-13T11:24:59.716 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:59.716 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:59.716 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:59.716 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:24:59.717 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.914353+0000", 2024-09-13T11:24:59.717 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:59.717 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:59.717 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:59.717 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:24:59.717 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.914356+0000", 2024-09-13T11:24:59.717 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.8710000000000001e-06 2024-09-13T11:24:59.717 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:59.717 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:59.717 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:24:59.717 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.914443+0000", 2024-09-13T11:24:59.717 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.7002000000000005e-05 2024-09-13T11:24:59.717 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:59.718 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:59.718 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:24:59.718 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.914461+0000", 2024-09-13T11:24:59.718 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.7691000000000001e-05 2024-09-13T11:24:59.718 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:59.718 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:59.718 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:24:59.718 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.914471+0000", 2024-09-13T11:24:59.718 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0447e-05 2024-09-13T11:24:59.718 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:59.718 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:59.718 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:24:59.718 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.914550+0000", 2024-09-13T11:24:59.718 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.8683000000000005e-05 2024-09-13T11:24:59.719 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:59.719 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:59.719 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:24:59.719 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.914709+0000", 2024-09-13T11:24:59.719 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00015913200000000001 2024-09-13T11:24:59.719 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:59.719 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:59.719 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 3", 2024-09-13T11:24:59.719 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.914880+0000", 2024-09-13T11:24:59.719 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00017071499999999999 2024-09-13T11:24:59.719 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:59.719 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:59.719 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:24:59.719 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.916701+0000", 2024-09-13T11:24:59.720 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0018205369999999999 2024-09-13T11:24:59.720 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:59.720 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:59.720 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:59.720 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:06.186401+0000", 2024-09-13T11:24:59.720 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.26970018699999998 2024-09-13T11:24:59.720 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:59.720 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:59.720 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:24:59.720 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:06.186434+0000", 2024-09-13T11:24:59.720 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.3448999999999998e-05 2024-09-13T11:24:59.720 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:59.720 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:59.721 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:24:59.721 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:06.186463+0000", 2024-09-13T11:24:59.721 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.8445e-05 2024-09-13T11:24:59.721 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:59.721 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:24:59.721 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:59.721 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:59.721 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:59.721 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop_reply(client.4665.0:84809 3.17 e107/85 ondisk, result = 0)", 2024-09-13T11:24:59.721 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:24:05.916779+0000", 2024-09-13T11:24:59.721 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 53.689864985, 2024-09-13T11:24:59.721 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.26968993499999999, 2024-09-13T11:24:59.721 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:24:59.721 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:24:59.722 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:24:59.722 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:24:59.722 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:59.722 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:24:59.722 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.916779+0000", 2024-09-13T11:24:59.722 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:59.722 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:59.722 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:59.722 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:24:59.722 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.916779+0000", 2024-09-13T11:24:59.722 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:59.722 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:59.722 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:59.722 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:24:59.723 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.916780+0000", 2024-09-13T11:24:59.723 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.328e-06 2024-09-13T11:24:59.723 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:59.723 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:59.723 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:24:59.723 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.916789+0000", 2024-09-13T11:24:59.723 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.4460000000000001e-06 2024-09-13T11:24:59.723 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:59.723 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:59.723 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:24:59.723 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:06.186296+0000", 2024-09-13T11:24:59.723 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.26950707099999999 2024-09-13T11:24:59.723 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:59.724 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:59.724 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:24:59.724 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:06.186306+0000", 2024-09-13T11:24:59.724 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0137e-05 2024-09-13T11:24:59.724 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:59.724 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:59.724 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:24:59.724 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:06.186388+0000", 2024-09-13T11:24:59.724 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.2149000000000002e-05 2024-09-13T11:24:59.724 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:59.724 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:59.724 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:24:59.724 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:06.186397+0000", 2024-09-13T11:24:59.724 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.5210000000000002e-06 2024-09-13T11:24:59.725 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:59.725 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:59.725 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:24:59.725 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:06.186469+0000", 2024-09-13T11:24:59.725 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.2282999999999999e-05 2024-09-13T11:24:59.725 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:59.725 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:24:59.725 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:59.725 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:59.725 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:59.725 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:99111 3.1f 3:fca4b03f:::benchmark_data_smithi177_31116_object99110:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113)", 2024-09-13T11:24:59.725 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:24:15.967965+0000", 2024-09-13T11:24:59.725 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 43.638679271999997, 2024-09-13T11:24:59.725 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.18645621300000001, 2024-09-13T11:24:59.726 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:24:59.726 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:24:59.726 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:24:59.726 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:24:59.726 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:24:59.726 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:24:59.726 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 99111 2024-09-13T11:24:59.726 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:59.726 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:24:59.726 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:59.726 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:24:59.726 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:15.967965+0000", 2024-09-13T11:24:59.726 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:59.727 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:59.727 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:59.727 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:24:59.727 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:15.967965+0000", 2024-09-13T11:24:59.727 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:59.727 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:59.727 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:59.727 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:24:59.727 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:15.967967+0000", 2024-09-13T11:24:59.727 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.79e-06 2024-09-13T11:24:59.727 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:59.727 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:59.727 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:24:59.727 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:15.968023+0000", 2024-09-13T11:24:59.728 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.5600000000000003e-05 2024-09-13T11:24:59.728 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:59.728 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:59.728 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:24:59.728 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:15.968045+0000", 2024-09-13T11:24:59.728 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.1801e-05 2024-09-13T11:24:59.728 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:59.728 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:59.728 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:24:59.728 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:15.968058+0000", 2024-09-13T11:24:59.728 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3522e-05 2024-09-13T11:24:59.728 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:59.728 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:59.729 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:24:59.729 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:15.968175+0000", 2024-09-13T11:24:59.729 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00011650900000000001 2024-09-13T11:24:59.729 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:59.729 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:59.729 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:24:59.729 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:15.968499+0000", 2024-09-13T11:24:59.729 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000324364 2024-09-13T11:24:59.729 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:59.729 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:59.729 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 14", 2024-09-13T11:24:59.729 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:15.968796+0000", 2024-09-13T11:24:59.729 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00029638100000000002 2024-09-13T11:24:59.729 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:59.730 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:59.730 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:24:59.730 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:15.971845+0000", 2024-09-13T11:24:59.730 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.003048931 2024-09-13T11:24:59.730 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:59.730 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:59.730 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:24:59.730 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:16.154118+0000", 2024-09-13T11:24:59.730 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.18227338700000001 2024-09-13T11:24:59.730 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:59.730 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:59.730 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:24:59.730 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:16.154293+0000", 2024-09-13T11:24:59.730 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00017466500000000001 2024-09-13T11:24:59.731 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:59.731 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:59.731 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:24:59.731 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:16.154421+0000", 2024-09-13T11:24:59.731 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00012826299999999999 2024-09-13T11:24:59.731 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:59.731 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:24:59.731 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:59.731 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:59.731 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:59.731 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPush(3.2a 121/120 [PushOp(3:5595108b:::benchmark_data_smithi177_31116_object899:head, version: 43'62, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:5595108b:::benchmark_data_smithi177_31116_object899:head@43'62, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:24:59.731 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:24:27.352355+0000", 2024-09-13T11:24:59.731 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 32.254289178999997, 2024-09-13T11:24:59.731 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.61628666099999996, 2024-09-13T11:24:59.732 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:24:59.732 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:24:59.732 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:24:59.732 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:24:59.732 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:59.732 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:24:59.732 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:27.352355+0000", 2024-09-13T11:24:59.732 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:59.732 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:59.732 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:59.732 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:24:59.732 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:27.352355+0000", 2024-09-13T11:24:59.732 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:24:59.733 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:59.733 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:59.733 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:24:59.733 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:27.352357+0000", 2024-09-13T11:24:59.733 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.328e-06 2024-09-13T11:24:59.733 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:59.733 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:59.733 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:24:59.733 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:27.352428+0000", 2024-09-13T11:24:59.733 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.1118999999999995e-05 2024-09-13T11:24:59.733 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:59.733 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:59.733 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:24:59.733 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:27.966950+0000", 2024-09-13T11:24:59.734 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.61452202499999997 2024-09-13T11:24:59.734 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:59.734 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:59.734 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:24:59.734 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:27.966980+0000", 2024-09-13T11:24:59.734 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.9433000000000001e-05 2024-09-13T11:24:59.734 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:59.734 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:59.734 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:24:59.734 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:27.967220+0000", 2024-09-13T11:24:59.734 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00024065499999999999 2024-09-13T11:24:59.734 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:59.734 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:59.736 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:24:59.736 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:27.967244+0000", 2024-09-13T11:24:59.736 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.4216e-05 2024-09-13T11:24:59.736 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:24:59.736 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:24:59.736 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:24:59.736 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:27.968641+0000", 2024-09-13T11:24:59.736 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0013968850000000001 2024-09-13T11:24:59.736 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:59.737 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:24:59.737 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:59.737 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:24:59.737 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:24:59.737 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-13T11:24:59.739 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-13T11:24:59.739 INFO:tasks.ceph.osd.14.smithi162.stderr:2024-09-13T11:24:59.736+0000 7f3df5903640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33154) UID: 0 2024-09-13T11:24:59.839 INFO:tasks.ceph.osd.15.smithi177.stderr:2024-09-13T11:24:59.834+0000 7f9d6ce77640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26578) UID: 0 2024-09-13T11:24:59.915 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-09-13T11:24:59.915 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [ 2024-09-13T11:24:59.915 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:59.915 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPush(3.15 145/122 [PushOp(3:ab86258b:::benchmark_data_smithi177_31116_object67550:head, version: 93'3448, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:ab86258b:::benchmark_data_smithi177_31116_object67550:head@93'3448, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:24:59.915 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:59.912792+0000", 2024-09-13T11:24:59.915 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 0.0019034340000000001, 2024-09-13T11:24:59.915 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0019480039999999999, 2024-09-13T11:24:59.915 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:24:59.915 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:24:59.915 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:24:59.915 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:24:59.916 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:59.916 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:24:59.916 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:59.912792+0000", 2024-09-13T11:24:59.916 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:59.916 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:59.916 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:59.916 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:24:59.916 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:59.912792+0000", 2024-09-13T11:24:59.916 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:24:59.916 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:59.916 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:59.916 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:24:59.916 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:59.912795+0000", 2024-09-13T11:24:59.916 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.4159999999999998e-06 2024-09-13T11:24:59.917 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:59.917 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:59.917 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:24:59.917 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:59.912838+0000", 2024-09-13T11:24:59.917 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.3004999999999999e-05 2024-09-13T11:24:59.917 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:59.917 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:59.917 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:24:59.917 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:59.912867+0000", 2024-09-13T11:24:59.917 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.9088999999999999e-05 2024-09-13T11:24:59.917 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:59.917 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:59.917 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:24:59.917 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:59.912887+0000", 2024-09-13T11:24:59.917 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.0282999999999999e-05 2024-09-13T11:24:59.918 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:59.918 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:59.918 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:24:59.918 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:59.913225+0000", 2024-09-13T11:24:59.918 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00033763 2024-09-13T11:24:59.918 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:24:59.918 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:24:59.918 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:24:59.918 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:59.913240+0000", 2024-09-13T11:24:59.918 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5184000000000001e-05 2024-09-13T11:24:59.918 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:59.918 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:24:59.918 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:59.919 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:24:59.919 INFO:teuthology.orchestra.run.smithi177.stdout: ], 2024-09-13T11:24:59.919 INFO:teuthology.orchestra.run.smithi177.stdout: "num_ops": 1 2024-09-13T11:24:59.919 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-09-13T11:24:59.940 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-13T11:24:59.941 INFO:tasks.ceph.osd.7.smithi177.stderr:2024-09-13T11:24:59.938+0000 7f897b51d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26575) UID: 0 2024-09-13T11:25:00.041 INFO:tasks.ceph.osd.11.smithi177.stderr:2024-09-13T11:25:00.038+0000 7f24792c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26580) UID: 0 2024-09-13T11:25:00.090 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-09-13T11:25:00.090 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [], 2024-09-13T11:25:00.090 INFO:teuthology.orchestra.run.smithi177.stdout: "complaint_time": 30, 2024-09-13T11:25:00.090 INFO:teuthology.orchestra.run.smithi177.stdout: "num_blocked_ops": 0 2024-09-13T11:25:00.090 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-09-13T11:25:00.110 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-13T11:25:00.145 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-13T11:25:00.143+0000 7f6a68775640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26307) UID: 0 2024-09-13T11:25:00.247 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-09-13T11:25:00.244+0000 7f97befc9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26315) UID: 0 2024-09-13T11:25:00.317 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-09-13T11:25:00.317 INFO:teuthology.orchestra.run.smithi177.stdout: "size": 20, 2024-09-13T11:25:00.317 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 600, 2024-09-13T11:25:00.317 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [ 2024-09-13T11:25:00.317 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:00.317 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:30852 3.a 3:522fb0d3:::benchmark_data_smithi177_31116_object30851:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e59)", 2024-09-13T11:25:00.318 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:09.538712+0000", 2024-09-13T11:25:00.318 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 110.77591323, 2024-09-13T11:25:00.318 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.19661263300000001, 2024-09-13T11:25:00.318 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:25:00.318 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:25:00.318 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:00.318 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:25:00.318 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:25:00.318 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:00.318 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 30852 2024-09-13T11:25:00.318 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:00.318 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:25:00.318 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:00.319 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:25:00.319 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.538712+0000", 2024-09-13T11:25:00.319 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:00.319 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:00.319 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:00.319 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:25:00.319 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.538712+0000", 2024-09-13T11:25:00.319 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:00.319 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:00.319 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:00.319 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:25:00.319 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.538714+0000", 2024-09-13T11:25:00.319 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.658e-06 2024-09-13T11:25:00.319 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:00.320 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:00.320 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:25:00.320 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.538735+0000", 2024-09-13T11:25:00.320 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.141e-05 2024-09-13T11:25:00.320 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:00.320 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:00.320 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:25:00.320 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.538747+0000", 2024-09-13T11:25:00.320 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.2258e-05 2024-09-13T11:25:00.320 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:00.320 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:00.320 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:25:00.320 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.538754+0000", 2024-09-13T11:25:00.321 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.2859999999999996e-06 2024-09-13T11:25:00.321 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:00.321 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:00.321 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:25:00.321 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.538798+0000", 2024-09-13T11:25:00.321 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.4258000000000002e-05 2024-09-13T11:25:00.321 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:00.321 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:00.321 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:25:00.321 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.538890+0000", 2024-09-13T11:25:00.321 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.1676999999999997e-05 2024-09-13T11:25:00.321 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:00.321 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:00.321 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 7", 2024-09-13T11:25:00.322 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.538991+0000", 2024-09-13T11:25:00.322 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00010173700000000001 2024-09-13T11:25:00.322 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:00.322 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:00.322 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:25:00.322 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.540171+0000", 2024-09-13T11:25:00.322 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0011800479999999999 2024-09-13T11:25:00.322 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:00.322 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:00.322 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:25:00.322 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.735031+0000", 2024-09-13T11:25:00.322 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.194859798 2024-09-13T11:25:00.322 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:00.322 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:00.323 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:25:00.323 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.735197+0000", 2024-09-13T11:25:00.323 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00016569499999999999 2024-09-13T11:25:00.323 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:00.323 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:00.323 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:25:00.323 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.735325+0000", 2024-09-13T11:25:00.323 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00012780800000000001 2024-09-13T11:25:00.323 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:00.323 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:25:00.323 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:00.323 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:00.323 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:00.324 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPush(3.17 75/70 [PushOp(3:eda94455:::benchmark_data_smithi177_31116_object13017:head, version: 50'788, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:eda94455:::benchmark_data_smithi177_31116_object13017:head@50'788, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:25:00.324 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:31.038029+0000", 2024-09-13T11:25:00.324 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 89.276596677000001, 2024-09-13T11:25:00.324 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.42064623499999998, 2024-09-13T11:25:00.324 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:25:00.324 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:25:00.324 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:25:00.324 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:25:00.324 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:00.324 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:25:00.324 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.038029+0000", 2024-09-13T11:25:00.324 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:00.325 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:00.325 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:00.325 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:25:00.325 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.038029+0000", 2024-09-13T11:25:00.325 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:00.325 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:00.325 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:00.325 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:25:00.325 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.038030+0000", 2024-09-13T11:25:00.325 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.499e-06 2024-09-13T11:25:00.325 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:00.325 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:00.326 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:25:00.326 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.038088+0000", 2024-09-13T11:25:00.326 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.7771e-05 2024-09-13T11:25:00.326 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:00.326 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:00.326 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:25:00.326 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.457025+0000", 2024-09-13T11:25:00.326 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.41893754900000002 2024-09-13T11:25:00.326 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:00.326 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:00.326 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:25:00.326 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.457073+0000", 2024-09-13T11:25:00.326 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.7450999999999997e-05 2024-09-13T11:25:00.326 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:00.327 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:00.327 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:25:00.327 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.457304+0000", 2024-09-13T11:25:00.327 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000231351 2024-09-13T11:25:00.327 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:00.327 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:00.327 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:25:00.327 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.457331+0000", 2024-09-13T11:25:00.327 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.6372000000000001e-05 2024-09-13T11:25:00.327 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:00.327 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:00.327 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:25:00.327 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.458675+0000", 2024-09-13T11:25:00.327 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001344242 2024-09-13T11:25:00.327 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:00.328 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:25:00.328 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:00.328 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:00.328 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:00.328 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPush(3.17 83/70 [PushOp(3:eb574268:::benchmark_data_smithi177_31116_object40622:head, version: 64'2475, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:eb574268:::benchmark_data_smithi177_31116_object40622:head@64'2475, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:25:00.328 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:40.760042+0000", 2024-09-13T11:25:00.328 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 79.554582960000005, 2024-09-13T11:25:00.328 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.30544642100000002, 2024-09-13T11:25:00.328 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:25:00.328 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:25:00.328 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:25:00.328 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:25:00.328 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:00.328 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:25:00.329 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:40.760042+0000", 2024-09-13T11:25:00.329 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:00.329 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:00.329 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:00.329 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:25:00.329 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:40.760042+0000", 2024-09-13T11:25:00.329 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:00.329 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:00.329 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:00.329 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:25:00.329 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:40.760044+0000", 2024-09-13T11:25:00.329 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.511e-06 2024-09-13T11:25:00.329 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:00.330 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:00.330 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:25:00.330 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:40.760149+0000", 2024-09-13T11:25:00.330 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000105473 2024-09-13T11:25:00.330 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:00.330 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:00.330 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:25:00.330 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.064465+0000", 2024-09-13T11:25:00.330 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.30431613800000001 2024-09-13T11:25:00.330 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:00.330 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:00.330 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:25:00.330 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.064479+0000", 2024-09-13T11:25:00.331 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3586e-05 2024-09-13T11:25:00.331 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:00.331 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:00.331 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:25:00.331 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.064553+0000", 2024-09-13T11:25:00.331 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.3646000000000006e-05 2024-09-13T11:25:00.331 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:00.331 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:00.331 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:25:00.331 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.064568+0000", 2024-09-13T11:25:00.331 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.52e-05 2024-09-13T11:25:00.331 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:00.331 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:00.331 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:25:00.332 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.065489+0000", 2024-09-13T11:25:00.332 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00092086700000000002 2024-09-13T11:25:00.332 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:00.332 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:25:00.332 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:00.332 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:00.332 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:00.332 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPushReply(3.1a 83/78 [PushReplyOp(3:5c3834bd:::benchmark_data_smithi177_31116_object8669:head)])", 2024-09-13T11:25:00.332 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:40.798236+0000", 2024-09-13T11:25:00.332 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 79.516389529999998, 2024-09-13T11:25:00.332 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.29701520999999997, 2024-09-13T11:25:00.332 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:25:00.332 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:25:00.333 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-13T11:25:00.333 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:25:00.333 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:00.333 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:25:00.333 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:40.798236+0000", 2024-09-13T11:25:00.333 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:00.333 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:00.333 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:00.333 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:25:00.333 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:40.798236+0000", 2024-09-13T11:25:00.333 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:00.333 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:00.334 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:00.334 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:25:00.334 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:40.798237+0000", 2024-09-13T11:25:00.334 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.0500000000000002e-07 2024-09-13T11:25:00.334 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:00.334 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:00.334 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:25:00.334 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:40.798240+0000", 2024-09-13T11:25:00.334 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.4819999999999999e-06 2024-09-13T11:25:00.334 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:00.334 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:00.334 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:25:00.334 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.093360+0000", 2024-09-13T11:25:00.335 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.29512009 2024-09-13T11:25:00.335 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:00.335 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:00.335 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:25:00.335 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.093372+0000", 2024-09-13T11:25:00.335 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.2221e-05 2024-09-13T11:25:00.335 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:00.335 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:00.335 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:25:00.335 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.093482+0000", 2024-09-13T11:25:00.335 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000109571 2024-09-13T11:25:00.335 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:00.335 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:00.336 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:25:00.336 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.095251+0000", 2024-09-13T11:25:00.336 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001768941 2024-09-13T11:25:00.336 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:00.336 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:25:00.336 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:00.336 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:00.336 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:00.336 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPushReply(3.1a 83/78 [PushReplyOp(3:58c7ffc0:::benchmark_data_smithi177_31116_object8738:head)])", 2024-09-13T11:25:00.336 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:40.805941+0000", 2024-09-13T11:25:00.336 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 79.508684275999997, 2024-09-13T11:25:00.336 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.30315678499999998, 2024-09-13T11:25:00.336 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:25:00.337 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:25:00.337 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-13T11:25:00.337 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:25:00.337 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:00.337 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:25:00.337 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:40.805941+0000", 2024-09-13T11:25:00.337 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:00.337 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:00.337 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:00.337 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:25:00.337 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:40.805941+0000", 2024-09-13T11:25:00.337 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:00.337 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:00.337 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:00.338 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:25:00.338 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:40.805942+0000", 2024-09-13T11:25:00.338 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3769999999999999e-06 2024-09-13T11:25:00.338 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:00.338 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:00.338 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:25:00.338 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:40.805947+0000", 2024-09-13T11:25:00.338 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.8980000000000004e-06 2024-09-13T11:25:00.338 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:00.338 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:00.338 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:25:00.338 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.106505+0000", 2024-09-13T11:25:00.338 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.30055777700000003 2024-09-13T11:25:00.338 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:00.339 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:00.339 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:25:00.339 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.106512+0000", 2024-09-13T11:25:00.339 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.3749999999999997e-06 2024-09-13T11:25:00.339 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:00.339 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:00.339 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:25:00.339 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.106566+0000", 2024-09-13T11:25:00.339 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.3103999999999998e-05 2024-09-13T11:25:00.339 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:00.339 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:00.339 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:25:00.339 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.109098+0000", 2024-09-13T11:25:00.340 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0025322539999999998 2024-09-13T11:25:00.340 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:00.340 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:25:00.340 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:00.340 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:00.340 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:00.340 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPushReply(3.1a 83/78 [PushReplyOp(3:58c7ffc0:::benchmark_data_smithi177_31116_object8738:head)])", 2024-09-13T11:25:00.340 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:40.805973+0000", 2024-09-13T11:25:00.340 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 79.508652272999996, 2024-09-13T11:25:00.340 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.28729672099999998, 2024-09-13T11:25:00.340 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:25:00.340 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:25:00.340 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-13T11:25:00.341 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:25:00.341 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:00.341 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:25:00.341 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:40.805973+0000", 2024-09-13T11:25:00.341 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:00.341 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:00.341 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:00.341 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:25:00.341 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:40.805973+0000", 2024-09-13T11:25:00.341 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:00.341 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:00.341 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:00.341 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:25:00.342 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:40.805974+0000", 2024-09-13T11:25:00.342 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.8599999999999998e-07 2024-09-13T11:25:00.342 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:00.342 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:00.342 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:25:00.342 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:40.805976+0000", 2024-09-13T11:25:00.342 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.2809999999999998e-06 2024-09-13T11:25:00.342 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:00.342 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:00.342 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:25:00.342 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.091588+0000", 2024-09-13T11:25:00.342 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.28561173899999998 2024-09-13T11:25:00.342 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:00.342 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:00.343 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:25:00.343 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.091596+0000", 2024-09-13T11:25:00.343 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.0520000000000002e-06 2024-09-13T11:25:00.343 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:00.343 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:00.343 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:25:00.343 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.091907+0000", 2024-09-13T11:25:00.343 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00031179599999999998 2024-09-13T11:25:00.343 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:00.343 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:00.343 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:25:00.343 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.093270+0000", 2024-09-13T11:25:00.343 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0013622669999999999 2024-09-13T11:25:00.344 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:00.344 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:25:00.344 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:00.344 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:00.344 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:00.344 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPushReply(3.1a 84/78 [PushReplyOp(3:581ea97e:::benchmark_data_smithi177_31116_object10255:head)])", 2024-09-13T11:25:00.344 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:41.514228+0000", 2024-09-13T11:25:00.344 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 78.800396785000004, 2024-09-13T11:25:00.344 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.40695564200000001, 2024-09-13T11:25:00.344 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:25:00.344 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:25:00.344 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-13T11:25:00.344 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:25:00.344 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:00.345 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:25:00.345 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.514228+0000", 2024-09-13T11:25:00.345 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:00.345 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:00.345 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:00.345 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:25:00.348 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.514228+0000", 2024-09-13T11:25:00.348 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:00.348 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:00.348 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:00.349 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:25:00.349 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.514230+0000", 2024-09-13T11:25:00.349 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.499e-06 2024-09-13T11:25:00.349 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:00.349 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:00.349 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:25:00.349 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.514236+0000", 2024-09-13T11:25:00.349 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.5389999999999998e-06 2024-09-13T11:25:00.349 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:00.349 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:00.349 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:25:00.349 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.919787+0000", 2024-09-13T11:25:00.349 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.40555152999999999 2024-09-13T11:25:00.350 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:00.350 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:00.350 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:25:00.350 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.919796+0000", 2024-09-13T11:25:00.350 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.4829999999999999e-06 2024-09-13T11:25:00.350 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:00.350 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:00.350 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:25:00.350 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.920366+0000", 2024-09-13T11:25:00.350 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00057061700000000004 2024-09-13T11:25:00.350 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:00.350 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:00.350 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:25:00.351 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.921184+0000", 2024-09-13T11:25:00.351 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00081797399999999996 2024-09-13T11:25:00.351 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:00.351 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:25:00.351 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:00.351 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:00.351 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:00.351 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:60312 3.14 3:2e846241:::benchmark_data_smithi177_31116_object60311:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e85)", 2024-09-13T11:25:00.351 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:42.065725+0000", 2024-09-13T11:25:00.351 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 78.248900141999997, 2024-09-13T11:25:00.351 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.211323176, 2024-09-13T11:25:00.351 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:25:00.351 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:25:00.351 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:00.352 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:25:00.352 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:25:00.352 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:00.352 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 60312 2024-09-13T11:25:00.352 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:00.352 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:25:00.352 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:00.352 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:25:00.352 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.065725+0000", 2024-09-13T11:25:00.352 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:00.352 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:00.352 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:00.353 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:25:00.353 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.065725+0000", 2024-09-13T11:25:00.353 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:00.353 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:00.353 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:00.353 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:25:00.353 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.065728+0000", 2024-09-13T11:25:00.353 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.3400000000000002e-06 2024-09-13T11:25:00.353 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:00.353 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:00.353 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:25:00.353 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.065759+0000", 2024-09-13T11:25:00.353 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.0074e-05 2024-09-13T11:25:00.354 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:00.354 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:00.354 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:25:00.354 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.065771+0000", 2024-09-13T11:25:00.354 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.2869e-05 2024-09-13T11:25:00.354 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:00.354 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:00.354 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:25:00.354 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.065780+0000", 2024-09-13T11:25:00.354 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.3489999999999992e-06 2024-09-13T11:25:00.354 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:00.354 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:00.355 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:25:00.355 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.065837+0000", 2024-09-13T11:25:00.355 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.7444000000000001e-05 2024-09-13T11:25:00.355 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:00.355 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:00.355 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:25:00.355 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.065966+0000", 2024-09-13T11:25:00.355 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000128973 2024-09-13T11:25:00.355 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:00.355 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:00.355 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 14", 2024-09-13T11:25:00.355 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.066114+0000", 2024-09-13T11:25:00.355 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00014788299999999999 2024-09-13T11:25:00.356 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:00.356 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:00.356 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:25:00.356 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.067604+0000", 2024-09-13T11:25:00.356 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001489637 2024-09-13T11:25:00.356 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:00.356 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:00.356 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:25:00.356 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.276963+0000", 2024-09-13T11:25:00.356 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.20935888499999999 2024-09-13T11:25:00.356 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:00.356 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:00.356 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:25:00.357 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.277003+0000", 2024-09-13T11:25:00.357 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.0241000000000003e-05 2024-09-13T11:25:00.357 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:00.357 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:00.357 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:25:00.357 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.277048+0000", 2024-09-13T11:25:00.357 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.5481e-05 2024-09-13T11:25:00.357 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:00.357 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:25:00.357 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:00.357 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:00.357 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:00.357 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop_reply(client.4665.0:60312 3.14 e85/84 ondisk, result = 0)", 2024-09-13T11:25:00.357 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:42.069120+0000", 2024-09-13T11:25:00.358 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 78.245505022000003, 2024-09-13T11:25:00.358 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.20804031000000001, 2024-09-13T11:25:00.358 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:25:00.358 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:25:00.358 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:25:00.358 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:25:00.358 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:00.358 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:25:00.358 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.069120+0000", 2024-09-13T11:25:00.358 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:00.358 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:00.358 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:00.358 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:25:00.358 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.069120+0000", 2024-09-13T11:25:00.359 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:00.359 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:00.359 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:00.359 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:25:00.359 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.069123+0000", 2024-09-13T11:25:00.359 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.289e-06 2024-09-13T11:25:00.359 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:00.359 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:00.359 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:25:00.359 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.069128+0000", 2024-09-13T11:25:00.359 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.3530000000000002e-06 2024-09-13T11:25:00.359 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:00.359 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:00.359 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:25:00.360 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.276885+0000", 2024-09-13T11:25:00.360 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.207756989 2024-09-13T11:25:00.360 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:00.360 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:00.360 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:25:00.360 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.276892+0000", 2024-09-13T11:25:00.360 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.922e-06 2024-09-13T11:25:00.360 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:00.360 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:00.360 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:25:00.360 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.276953+0000", 2024-09-13T11:25:00.360 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.0890000000000001e-05 2024-09-13T11:25:00.360 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:00.360 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:00.361 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:25:00.361 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.276959+0000", 2024-09-13T11:25:00.361 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.9179999999999999e-06 2024-09-13T11:25:00.361 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:00.361 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:00.361 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:25:00.361 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.277161+0000", 2024-09-13T11:25:00.361 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00020194899999999999 2024-09-13T11:25:00.361 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:00.361 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:25:00.361 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:00.361 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:00.361 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:00.362 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:60427 3.14 3:2afea229:::benchmark_data_smithi177_31116_object60426:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e85)", 2024-09-13T11:25:00.362 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:42.187231+0000", 2024-09-13T11:25:00.362 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 78.127393916000003, 2024-09-13T11:25:00.362 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.83900596999999999, 2024-09-13T11:25:00.362 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:25:00.362 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:25:00.362 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:00.362 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:25:00.362 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:25:00.362 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:00.362 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 60427 2024-09-13T11:25:00.362 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:00.362 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:25:00.363 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:00.363 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:25:00.363 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.187231+0000", 2024-09-13T11:25:00.363 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:00.363 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:00.363 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:00.363 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:25:00.363 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.187231+0000", 2024-09-13T11:25:00.363 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:00.363 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:00.363 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:00.363 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:25:00.363 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.187235+0000", 2024-09-13T11:25:00.364 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.3579999999999999e-06 2024-09-13T11:25:00.364 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:00.364 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:00.364 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:25:00.364 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.187261+0000", 2024-09-13T11:25:00.364 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.6089e-05 2024-09-13T11:25:00.364 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:00.364 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:00.364 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:25:00.364 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.187275+0000", 2024-09-13T11:25:00.364 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.4653e-05 2024-09-13T11:25:00.364 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:00.364 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:00.365 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:25:00.365 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.187282+0000", 2024-09-13T11:25:00.365 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.8870000000000003e-06 2024-09-13T11:25:00.365 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:00.365 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:00.365 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:25:00.365 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.187336+0000", 2024-09-13T11:25:00.365 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.3875999999999997e-05 2024-09-13T11:25:00.365 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:00.365 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:00.365 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:25:00.365 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.187480+0000", 2024-09-13T11:25:00.365 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00014330899999999999 2024-09-13T11:25:00.366 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:00.366 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:00.366 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 14", 2024-09-13T11:25:00.366 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.187658+0000", 2024-09-13T11:25:00.366 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00017861900000000001 2024-09-13T11:25:00.366 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:00.366 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:00.366 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:25:00.366 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.189353+0000", 2024-09-13T11:25:00.366 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001695144 2024-09-13T11:25:00.366 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:00.366 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:00.366 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:25:00.366 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:43.026081+0000", 2024-09-13T11:25:00.367 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.83672736199999997 2024-09-13T11:25:00.367 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:00.367 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:00.367 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:25:00.367 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:43.026169+0000", 2024-09-13T11:25:00.367 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.8411000000000004e-05 2024-09-13T11:25:00.367 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:00.367 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:00.367 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:25:00.367 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:43.026237+0000", 2024-09-13T11:25:00.367 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.8262000000000005e-05 2024-09-13T11:25:00.367 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:00.367 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:25:00.368 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:00.368 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:00.368 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:00.368 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop_reply(client.4665.0:60427 3.14 e85/84 ondisk, result = 0)", 2024-09-13T11:25:00.368 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:42.190914+0000", 2024-09-13T11:25:00.368 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 78.123711252000007, 2024-09-13T11:25:00.368 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.83534071499999996, 2024-09-13T11:25:00.368 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:25:00.368 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:25:00.368 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:25:00.368 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:25:00.368 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:00.368 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:25:00.369 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.190914+0000", 2024-09-13T11:25:00.369 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:00.369 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:00.369 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:00.369 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:25:00.369 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.190914+0000", 2024-09-13T11:25:00.369 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:00.369 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:00.369 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:00.369 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:25:00.369 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.190916+0000", 2024-09-13T11:25:00.369 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.84e-06 2024-09-13T11:25:00.369 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:00.370 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:00.370 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:25:00.370 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.190921+0000", 2024-09-13T11:25:00.370 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.2039999999999996e-06 2024-09-13T11:25:00.370 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:00.370 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:00.370 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:25:00.370 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:43.025926+0000", 2024-09-13T11:25:00.370 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.83500542 2024-09-13T11:25:00.370 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:00.370 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:00.370 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:25:00.370 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:43.025937+0000", 2024-09-13T11:25:00.371 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0597e-05 2024-09-13T11:25:00.371 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:00.371 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:00.371 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:25:00.371 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:43.026052+0000", 2024-09-13T11:25:00.371 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000114838 2024-09-13T11:25:00.371 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:00.371 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:00.371 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:25:00.371 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:43.026070+0000", 2024-09-13T11:25:00.371 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.8131e-05 2024-09-13T11:25:00.371 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:00.371 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:00.372 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:25:00.372 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:43.026255+0000", 2024-09-13T11:25:00.372 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000184685 2024-09-13T11:25:00.372 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:00.372 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:25:00.372 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:00.372 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:00.372 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:00.372 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPushReply(3.1d 89/78 [PushReplyOp(3:bc1b7cfb:::benchmark_data_smithi177_31116_object22629:head)])", 2024-09-13T11:25:00.372 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:46.663106+0000", 2024-09-13T11:25:00.372 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 73.651519069000003, 2024-09-13T11:25:00.372 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.26645533199999999, 2024-09-13T11:25:00.372 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:25:00.373 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:25:00.373 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-13T11:25:00.373 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:25:00.373 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:00.373 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:25:00.373 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.663106+0000", 2024-09-13T11:25:00.373 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:00.373 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:00.373 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:00.373 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:25:00.373 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.663106+0000", 2024-09-13T11:25:00.373 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:00.373 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:00.373 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:00.374 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:25:00.374 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.663108+0000", 2024-09-13T11:25:00.374 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.0140000000000001e-06 2024-09-13T11:25:00.374 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:00.374 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:00.374 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:25:00.374 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.663116+0000", 2024-09-13T11:25:00.374 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.5440000000000001e-06 2024-09-13T11:25:00.374 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:00.374 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:00.374 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:25:00.374 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.928111+0000", 2024-09-13T11:25:00.374 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.26499566899999999 2024-09-13T11:25:00.375 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:00.375 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:00.375 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:25:00.375 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.928135+0000", 2024-09-13T11:25:00.375 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.4003e-05 2024-09-13T11:25:00.375 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:00.375 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:00.375 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:25:00.375 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.928559+0000", 2024-09-13T11:25:00.375 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000423318 2024-09-13T11:25:00.375 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:00.375 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:00.375 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:25:00.375 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.929562+0000", 2024-09-13T11:25:00.376 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0010027840000000001 2024-09-13T11:25:00.376 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:00.376 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:25:00.376 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:00.376 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:00.376 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:00.376 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "rep_scrubmap(3.14 e84 from shard 14)", 2024-09-13T11:25:00.376 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:48.313615+0000", 2024-09-13T11:25:00.376 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 72.001010276000002, 2024-09-13T11:25:00.376 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.35633836099999999, 2024-09-13T11:25:00.376 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:25:00.376 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:25:00.376 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:25:00.376 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:25:00.377 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:00.377 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:25:00.377 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.313615+0000", 2024-09-13T11:25:00.377 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:00.377 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:00.377 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:00.377 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:25:00.377 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.313615+0000", 2024-09-13T11:25:00.377 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:00.377 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:00.377 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:00.377 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:25:00.378 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.313617+0000", 2024-09-13T11:25:00.378 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.7990000000000001e-06 2024-09-13T11:25:00.378 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:00.378 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:00.378 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:25:00.378 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.313688+0000", 2024-09-13T11:25:00.378 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.1481000000000002e-05 2024-09-13T11:25:00.378 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:00.378 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:00.378 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:25:00.378 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.669785+0000", 2024-09-13T11:25:00.378 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.356096466 2024-09-13T11:25:00.378 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:00.379 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:00.379 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:25:00.379 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.669792+0000", 2024-09-13T11:25:00.379 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.1740000000000003e-06 2024-09-13T11:25:00.379 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:00.379 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:00.379 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:25:00.379 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.669839+0000", 2024-09-13T11:25:00.379 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.7323000000000001e-05 2024-09-13T11:25:00.379 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:00.379 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:00.379 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:25:00.379 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.669847+0000", 2024-09-13T11:25:00.380 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.0469999999999994e-06 2024-09-13T11:25:00.380 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:00.380 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:00.380 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:25:00.380 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.669953+0000", 2024-09-13T11:25:00.380 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00010607099999999999 2024-09-13T11:25:00.380 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:00.380 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:25:00.380 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:00.380 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:00.380 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:00.380 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:110268 3.14 3:2997c4c5:::benchmark_data_smithi177_31116_object110267:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119)", 2024-09-13T11:25:00.380 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:25.359386+0000", 2024-09-13T11:25:00.381 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 34.955238960000003, 2024-09-13T11:25:00.381 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.29072703, 2024-09-13T11:25:00.381 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:25:00.381 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:25:00.381 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:00.381 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:25:00.381 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:25:00.381 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:00.381 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 110268 2024-09-13T11:25:00.381 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:00.381 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:25:00.381 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:00.381 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:25:00.382 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.359386+0000", 2024-09-13T11:25:00.382 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:00.382 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:00.382 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:00.382 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:25:00.382 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.359386+0000", 2024-09-13T11:25:00.382 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:00.382 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:00.382 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:00.382 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:25:00.382 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.359389+0000", 2024-09-13T11:25:00.382 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.8600000000000001e-06 2024-09-13T11:25:00.382 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:00.383 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:00.383 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:25:00.383 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.359424+0000", 2024-09-13T11:25:00.383 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.4755999999999997e-05 2024-09-13T11:25:00.383 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:00.383 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:00.383 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:25:00.383 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.359445+0000", 2024-09-13T11:25:00.383 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.1104999999999999e-05 2024-09-13T11:25:00.383 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:00.383 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:00.383 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:25:00.383 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.359460+0000", 2024-09-13T11:25:00.383 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5396999999999998e-05 2024-09-13T11:25:00.384 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:00.384 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:00.384 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:25:00.384 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.359581+0000", 2024-09-13T11:25:00.384 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000120455 2024-09-13T11:25:00.384 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:00.384 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:00.384 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:25:00.384 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.359827+0000", 2024-09-13T11:25:00.384 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00024571 2024-09-13T11:25:00.384 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:00.384 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:00.384 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 14", 2024-09-13T11:25:00.384 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.360084+0000", 2024-09-13T11:25:00.385 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00025787999999999999 2024-09-13T11:25:00.385 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:00.385 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:00.385 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:25:00.385 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.362243+0000", 2024-09-13T11:25:00.385 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0021581579999999999 2024-09-13T11:25:00.385 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:00.385 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:00.385 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:25:00.385 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.650034+0000", 2024-09-13T11:25:00.385 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.287791662 2024-09-13T11:25:00.385 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:00.385 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:00.386 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:25:00.386 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.650077+0000", 2024-09-13T11:25:00.386 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.2367999999999998e-05 2024-09-13T11:25:00.386 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:00.386 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:00.386 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:25:00.386 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.650113+0000", 2024-09-13T11:25:00.386 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.6678999999999999e-05 2024-09-13T11:25:00.386 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:00.386 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:25:00.386 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:00.386 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:00.386 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:00.387 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:110279 3.14 3:2b17a349:::benchmark_data_smithi177_31116_object110278:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119)", 2024-09-13T11:25:00.387 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:25.645137+0000", 2024-09-13T11:25:00.387 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 34.669488121999997, 2024-09-13T11:25:00.387 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.28172927800000003, 2024-09-13T11:25:00.387 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:25:00.387 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:25:00.387 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:00.387 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:25:00.387 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:25:00.387 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:00.387 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 110279 2024-09-13T11:25:00.387 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:00.388 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:25:00.388 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:00.388 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:25:00.388 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.645137+0000", 2024-09-13T11:25:00.388 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:00.388 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:00.388 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:00.388 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:25:00.388 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.645137+0000", 2024-09-13T11:25:00.388 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:00.388 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:00.388 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:00.388 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:25:00.389 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.645140+0000", 2024-09-13T11:25:00.389 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.0939999999999999e-06 2024-09-13T11:25:00.389 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:00.389 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:00.389 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:25:00.389 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.645184+0000", 2024-09-13T11:25:00.389 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.3412999999999999e-05 2024-09-13T11:25:00.389 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:00.389 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:00.389 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:25:00.389 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.645206+0000", 2024-09-13T11:25:00.389 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.1997e-05 2024-09-13T11:25:00.390 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:00.390 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:00.390 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:25:00.390 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.645216+0000", 2024-09-13T11:25:00.390 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0740999999999999e-05 2024-09-13T11:25:00.390 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:00.390 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:00.390 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:25:00.390 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.645397+0000", 2024-09-13T11:25:00.390 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00018077600000000001 2024-09-13T11:25:00.390 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:00.390 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:00.390 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:25:00.391 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.645638+0000", 2024-09-13T11:25:00.391 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000241091 2024-09-13T11:25:00.391 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:00.391 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:00.391 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 14", 2024-09-13T11:25:00.391 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.645922+0000", 2024-09-13T11:25:00.391 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000283565 2024-09-13T11:25:00.391 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:00.391 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:00.391 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:25:00.391 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.648980+0000", 2024-09-13T11:25:00.391 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00305851 2024-09-13T11:25:00.391 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:00.392 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:00.392 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:25:00.392 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.926702+0000", 2024-09-13T11:25:00.392 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.27772199800000003 2024-09-13T11:25:00.392 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:00.392 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:00.392 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:25:00.392 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.926796+0000", 2024-09-13T11:25:00.392 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.4158000000000002e-05 2024-09-13T11:25:00.392 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:00.392 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:00.392 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:25:00.392 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.926866+0000", 2024-09-13T11:25:00.393 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.9935e-05 2024-09-13T11:25:00.393 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:00.393 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:25:00.393 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:00.393 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:00.393 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:00.393 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:129865 3.14 3:2c872f8e:::benchmark_data_smithi177_31116_object129864:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e134)", 2024-09-13T11:25:00.393 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:47.107186+0000", 2024-09-13T11:25:00.393 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 13.207439565, 2024-09-13T11:25:00.393 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.21814059299999999, 2024-09-13T11:25:00.393 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:25:00.393 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:25:00.393 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:00.394 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:25:00.394 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:25:00.394 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:00.394 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 129865 2024-09-13T11:25:00.394 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:00.394 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:25:00.394 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:00.394 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:25:00.394 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:47.107186+0000", 2024-09-13T11:25:00.394 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:00.394 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:00.394 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:00.394 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:25:00.394 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:47.107186+0000", 2024-09-13T11:25:00.394 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:00.395 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:00.395 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:00.395 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:25:00.395 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:47.107187+0000", 2024-09-13T11:25:00.395 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5740000000000001e-06 2024-09-13T11:25:00.395 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:00.395 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:00.395 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:25:00.395 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:47.107207+0000", 2024-09-13T11:25:00.395 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.9304999999999999e-05 2024-09-13T11:25:00.395 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:00.395 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:00.395 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:25:00.395 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:47.107215+0000", 2024-09-13T11:25:00.396 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.7390000000000005e-06 2024-09-13T11:25:00.396 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:00.396 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:00.396 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:25:00.396 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:47.107221+0000", 2024-09-13T11:25:00.396 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.7350000000000001e-06 2024-09-13T11:25:00.396 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:00.396 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:00.396 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:25:00.396 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:47.107278+0000", 2024-09-13T11:25:00.396 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.7209000000000001e-05 2024-09-13T11:25:00.396 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:00.397 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:00.397 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:25:00.397 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:47.107383+0000", 2024-09-13T11:25:00.397 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000104718 2024-09-13T11:25:00.397 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:00.397 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:00.397 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 15", 2024-09-13T11:25:00.397 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:47.107480+0000", 2024-09-13T11:25:00.397 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.7343000000000006e-05 2024-09-13T11:25:00.397 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:00.397 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:00.397 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:25:00.397 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:47.109477+0000", 2024-09-13T11:25:00.398 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0019968490000000002 2024-09-13T11:25:00.398 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:00.398 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:00.398 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:25:00.398 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:47.324147+0000", 2024-09-13T11:25:00.398 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.21467018199999999 2024-09-13T11:25:00.398 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:00.398 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:00.398 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:25:00.398 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:47.324281+0000", 2024-09-13T11:25:00.398 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00013328600000000001 2024-09-13T11:25:00.398 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:00.398 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:00.399 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:25:00.399 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:47.325326+0000", 2024-09-13T11:25:00.399 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0010456529999999999 2024-09-13T11:25:00.399 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:00.399 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:25:00.399 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:00.399 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:00.399 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:00.399 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPush(3.15 140/122 [PushOp(3:ae90e82d:::benchmark_data_smithi177_31116_object53962:head, version: 74'3006, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:ae90e82d:::benchmark_data_smithi177_31116_object53962:head@74'3006, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:25:00.399 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:54.310786+0000", 2024-09-13T11:25:00.399 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 6.0038390919999998, 2024-09-13T11:25:00.399 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.229345255, 2024-09-13T11:25:00.399 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:25:00.400 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:25:00.400 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:25:00.400 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:25:00.400 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:00.400 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:25:00.400 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:54.310786+0000", 2024-09-13T11:25:00.400 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:00.400 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:00.400 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:00.400 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:25:00.400 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:54.310786+0000", 2024-09-13T11:25:00.400 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:00.400 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:00.401 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:00.401 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:25:00.401 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:54.310795+0000", 2024-09-13T11:25:00.401 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.1910000000000006e-06 2024-09-13T11:25:00.401 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:00.401 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:00.401 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:25:00.401 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:54.310829+0000", 2024-09-13T11:25:00.401 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.4121000000000001e-05 2024-09-13T11:25:00.401 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:00.401 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:00.401 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:25:00.401 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:54.538419+0000", 2024-09-13T11:25:00.402 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.227589611 2024-09-13T11:25:00.402 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:00.402 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:00.402 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:25:00.402 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:54.538430+0000", 2024-09-13T11:25:00.402 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0456e-05 2024-09-13T11:25:00.402 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:00.402 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:00.402 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:25:00.402 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:54.538497+0000", 2024-09-13T11:25:00.402 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.6948999999999999e-05 2024-09-13T11:25:00.402 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:00.403 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:00.404 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:25:00.404 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:54.538505+0000", 2024-09-13T11:25:00.404 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.2989999999999997e-06 2024-09-13T11:25:00.404 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:00.404 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:00.404 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:25:00.404 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:54.540131+0000", 2024-09-13T11:25:00.404 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001626628 2024-09-13T11:25:00.404 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:00.404 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:25:00.404 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:00.404 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:00.404 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:00.405 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPush(3.15 143/122 [PushOp(3:ae52bb45:::benchmark_data_smithi177_31116_object61893:head, version: 87'3272, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:ae52bb45:::benchmark_data_smithi177_31116_object61893:head@87'3272, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:25:00.405 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:57.633939+0000", 2024-09-13T11:25:00.405 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 2.6806861390000001, 2024-09-13T11:25:00.405 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.339405067, 2024-09-13T11:25:00.405 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:25:00.405 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:25:00.405 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:25:00.405 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:25:00.405 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:00.405 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:25:00.405 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:57.633939+0000", 2024-09-13T11:25:00.405 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:00.405 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:00.406 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:00.406 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:25:00.406 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:57.633939+0000", 2024-09-13T11:25:00.406 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:00.406 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:00.406 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:00.406 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:25:00.406 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:57.633941+0000", 2024-09-13T11:25:00.406 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.911e-06 2024-09-13T11:25:00.406 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:00.406 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:00.406 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:25:00.407 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:57.633994+0000", 2024-09-13T11:25:00.407 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.2753999999999997e-05 2024-09-13T11:25:00.407 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:00.407 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:00.407 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:25:00.407 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:57.968403+0000", 2024-09-13T11:25:00.407 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.334409336 2024-09-13T11:25:00.407 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:00.407 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:00.407 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:25:00.407 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:57.968451+0000", 2024-09-13T11:25:00.407 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.7525000000000003e-05 2024-09-13T11:25:00.407 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:00.407 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:00.408 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:25:00.408 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:57.968660+0000", 2024-09-13T11:25:00.408 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000209204 2024-09-13T11:25:00.408 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:00.408 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:00.408 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:25:00.408 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:57.968680+0000", 2024-09-13T11:25:00.408 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.9639999999999998e-05 2024-09-13T11:25:00.408 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:00.408 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:00.408 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:25:00.408 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:57.973344+0000", 2024-09-13T11:25:00.408 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0046646969999999998 2024-09-13T11:25:00.409 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:00.409 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:25:00.409 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:00.409 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:00.409 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:00.409 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:140658 3.17 e144/85 3:ece5a1b3:::benchmark_data_smithi177_31116_object355:head v 144'5654, mlcod=144'5652)", 2024-09-13T11:25:00.409 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:58.247792+0000", 2024-09-13T11:25:00.409 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 2.066833661, 2024-09-13T11:25:00.409 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.29148369699999999, 2024-09-13T11:25:00.409 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:25:00.409 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:25:00.409 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:00.409 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:25:00.410 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:00.410 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:25:00.410 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:58.247792+0000", 2024-09-13T11:25:00.410 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:00.410 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:00.410 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:00.410 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:25:00.410 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:58.247792+0000", 2024-09-13T11:25:00.410 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:00.410 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:00.410 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:00.410 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:25:00.411 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:58.247793+0000", 2024-09-13T11:25:00.411 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.347e-06 2024-09-13T11:25:00.411 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:00.411 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:00.411 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:25:00.411 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:58.247799+0000", 2024-09-13T11:25:00.411 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.5239999999999997e-06 2024-09-13T11:25:00.412 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:00.412 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:00.412 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:25:00.412 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:58.535226+0000", 2024-09-13T11:25:00.412 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.28742604900000002 2024-09-13T11:25:00.412 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:00.412 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:00.412 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:25:00.412 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:58.535238+0000", 2024-09-13T11:25:00.412 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.2843000000000001e-05 2024-09-13T11:25:00.412 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:00.412 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:00.413 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:25:00.413 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:58.535393+0000", 2024-09-13T11:25:00.413 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00015481699999999999 2024-09-13T11:25:00.413 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:00.413 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:00.413 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:25:00.413 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:58.535434+0000", 2024-09-13T11:25:00.413 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.1152999999999997e-05 2024-09-13T11:25:00.413 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:00.413 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:00.413 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:25:00.413 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:58.539191+0000", 2024-09-13T11:25:00.413 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0037571520000000001 2024-09-13T11:25:00.413 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:00.414 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:00.414 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:25:00.414 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:58.539275+0000", 2024-09-13T11:25:00.414 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.3812000000000003e-05 2024-09-13T11:25:00.414 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:00.414 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:25:00.414 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:00.414 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:00.414 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:00.414 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPush(3.15 144/122 [PushOp(3:a97ba025:::benchmark_data_smithi177_31116_object64775:head, version: 90'3365, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:a97ba025:::benchmark_data_smithi177_31116_object64775:head@90'3365, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:25:00.414 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:58.825014+0000", 2024-09-13T11:25:00.414 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 1.4896110330000001, 2024-09-13T11:25:00.415 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.23868276499999999, 2024-09-13T11:25:00.415 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:25:00.415 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:25:00.415 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:25:00.415 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:25:00.415 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:00.415 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:25:00.415 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:58.825014+0000", 2024-09-13T11:25:00.415 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:00.415 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:00.415 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:00.415 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:25:00.415 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:58.825014+0000", 2024-09-13T11:25:00.415 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:00.416 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:00.416 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:00.416 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:25:00.416 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:58.825017+0000", 2024-09-13T11:25:00.416 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.7630000000000001e-06 2024-09-13T11:25:00.416 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:00.416 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:00.416 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:25:00.416 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:58.825099+0000", 2024-09-13T11:25:00.416 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.2035999999999999e-05 2024-09-13T11:25:00.416 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:00.416 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:00.417 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:25:00.417 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:59.059557+0000", 2024-09-13T11:25:00.417 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.23445790399999999 2024-09-13T11:25:00.417 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:00.417 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:00.417 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:25:00.417 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:59.059609+0000", 2024-09-13T11:25:00.417 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.2550999999999999e-05 2024-09-13T11:25:00.417 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:00.417 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:00.417 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:25:00.417 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:59.060117+0000", 2024-09-13T11:25:00.417 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00050754299999999995 2024-09-13T11:25:00.418 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:00.418 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:00.418 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:25:00.418 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:59.060139+0000", 2024-09-13T11:25:00.418 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.2306e-05 2024-09-13T11:25:00.418 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:00.418 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:00.418 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:25:00.418 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:59.063697+0000", 2024-09-13T11:25:00.418 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0035576620000000001 2024-09-13T11:25:00.418 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:00.418 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:25:00.418 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:00.418 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:00.419 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:25:00.419 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-09-13T11:25:00.420 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-13T11:25:00.421 INFO:tasks.ceph.osd.13.smithi092.stderr:2024-09-13T11:25:00.414+0000 7f3e5afea640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26350) UID: 0 2024-09-13T11:25:00.521 INFO:tasks.ceph.osd.8.smithi001.stderr:2024-09-13T11:25:00.519+0000 7fb9b48ae640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26312) UID: 0 2024-09-13T11:25:00.587 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-09-13T11:25:00.587 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [ 2024-09-13T11:25:00.587 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:00.587 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:142965 3.a e146/117 3:5067123e:::benchmark_data_smithi177_31116_object2662:head v 146'5637, mlcod=146'5635)", 2024-09-13T11:25:00.587 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:25:00.583468+0000", 2024-09-13T11:25:00.588 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 0.0036220079999999999, 2024-09-13T11:25:00.588 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0036456639999999998, 2024-09-13T11:25:00.588 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:25:00.588 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:25:00.588 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:25:00.588 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:25:00.588 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:00.588 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:25:00.588 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:00.583468+0000", 2024-09-13T11:25:00.588 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:00.588 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:00.588 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:00.588 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:25:00.589 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:00.583468+0000", 2024-09-13T11:25:00.589 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:00.589 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:00.589 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:00.589 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:25:00.589 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:00.583470+0000", 2024-09-13T11:25:00.589 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.843e-06 2024-09-13T11:25:00.589 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:00.589 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:00.589 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:25:00.589 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:00.583479+0000", 2024-09-13T11:25:00.589 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.1190000000000003e-06 2024-09-13T11:25:00.589 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:00.589 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:00.590 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:25:00.590 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:00.583522+0000", 2024-09-13T11:25:00.590 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.2397000000000001e-05 2024-09-13T11:25:00.590 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:00.590 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:00.590 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:25:00.590 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:00.583531+0000", 2024-09-13T11:25:00.590 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.0950000000000002e-06 2024-09-13T11:25:00.590 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:00.590 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:00.590 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:25:00.590 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:00.583604+0000", 2024-09-13T11:25:00.590 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.3106000000000001e-05 2024-09-13T11:25:00.590 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:00.591 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:00.591 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:25:00.591 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:00.583627+0000", 2024-09-13T11:25:00.591 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.2908000000000002e-05 2024-09-13T11:25:00.591 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:00.591 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:25:00.591 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:00.591 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:00.591 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:00.591 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPushReply(3.12 146/122 [PushReplyOp(3:4d6be96c:::benchmark_data_smithi177_31116_object58804:head)])", 2024-09-13T11:25:00.591 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:25:00.583887+0000", 2024-09-13T11:25:00.591 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 0.003203614, 2024-09-13T11:25:00.591 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0032854170000000001, 2024-09-13T11:25:00.591 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:25:00.592 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:25:00.592 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-13T11:25:00.592 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:25:00.592 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:00.592 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:25:00.592 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:00.583887+0000", 2024-09-13T11:25:00.592 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:00.592 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:00.592 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:00.592 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:25:00.592 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:00.583887+0000", 2024-09-13T11:25:00.592 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:00.592 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:00.593 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:00.593 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:25:00.593 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:00.583888+0000", 2024-09-13T11:25:00.593 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.2270000000000001e-06 2024-09-13T11:25:00.593 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:00.593 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:00.593 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:25:00.593 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:00.583894+0000", 2024-09-13T11:25:00.593 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.2169999999999999e-06 2024-09-13T11:25:00.593 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:00.594 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:00.594 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:25:00.594 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:00.583933+0000", 2024-09-13T11:25:00.594 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.8689999999999997e-05 2024-09-13T11:25:00.594 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:00.594 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:00.594 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:25:00.594 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:00.583941+0000", 2024-09-13T11:25:00.594 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.9670000000000001e-06 2024-09-13T11:25:00.594 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:00.594 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:00.594 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:25:00.594 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:00.584420+0000", 2024-09-13T11:25:00.594 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00047891600000000002 2024-09-13T11:25:00.595 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:00.595 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:25:00.595 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:00.595 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:00.595 INFO:teuthology.orchestra.run.smithi177.stdout: ], 2024-09-13T11:25:00.595 INFO:teuthology.orchestra.run.smithi177.stdout: "num_ops": 2 2024-09-13T11:25:00.595 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-09-13T11:25:00.607 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-13T11:25:00.621 INFO:tasks.ceph.osd.12.smithi001.stderr:2024-09-13T11:25:00.619+0000 7f892c6e6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26314) UID: 0 2024-09-13T11:25:00.723 INFO:tasks.ceph.osd.7.smithi177.stderr:2024-09-13T11:25:00.718+0000 7f897b51d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26575) UID: 0 2024-09-13T11:25:00.769 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-09-13T11:25:00.769 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [], 2024-09-13T11:25:00.769 INFO:teuthology.orchestra.run.smithi177.stdout: "complaint_time": 30, 2024-09-13T11:25:00.769 INFO:teuthology.orchestra.run.smithi177.stdout: "num_blocked_ops": 0 2024-09-13T11:25:00.769 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-09-13T11:25:00.794 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-13T11:25:00.823 INFO:tasks.ceph.osd.1.smithi092.stderr:2024-09-13T11:25:00.818+0000 7fe7789c5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33718) UID: 0 2024-09-13T11:25:00.877 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T11:25:00.923 INFO:tasks.ceph.osd.1.smithi092.stderr:2024-09-13T11:25:00.918+0000 7fe7789c5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33718) UID: 0 2024-09-13T11:25:01.021 INFO:tasks.ceph.osd.13.smithi092.stderr:2024-09-13T11:25:01.018+0000 7f3e4da13640 -1 osd.13 146 handle_osd_map injecting map crc failure 2024-09-13T11:25:01.026 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-09-13T11:25:01.026 INFO:teuthology.orchestra.run.smithi177.stdout: "size": 20, 2024-09-13T11:25:01.026 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 600, 2024-09-13T11:25:01.026 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [ 2024-09-13T11:25:01.027 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:01.027 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPushReply(3.13 75/70 [PushReplyOp(3:cfab03d6:::benchmark_data_smithi177_31116_object16599:head)])", 2024-09-13T11:25:01.027 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:31.163480+0000", 2024-09-13T11:25:01.027 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 89.860435635000002, 2024-09-13T11:25:01.027 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.63009029299999997, 2024-09-13T11:25:01.027 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:25:01.027 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:25:01.027 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-13T11:25:01.027 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:25:01.027 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:01.027 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:25:01.027 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.163480+0000", 2024-09-13T11:25:01.027 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:01.028 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:01.028 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:01.028 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:25:01.028 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.163480+0000", 2024-09-13T11:25:01.028 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:01.028 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:01.028 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:01.028 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:25:01.028 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.163481+0000", 2024-09-13T11:25:01.028 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0410000000000001e-06 2024-09-13T11:25:01.028 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:01.028 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:01.028 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:25:01.029 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.163485+0000", 2024-09-13T11:25:01.029 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.1629999999999998e-06 2024-09-13T11:25:01.029 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:01.029 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:01.029 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:25:01.029 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.792166+0000", 2024-09-13T11:25:01.029 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.62868067400000005 2024-09-13T11:25:01.029 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:01.029 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:01.029 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:25:01.029 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.792184+0000", 2024-09-13T11:25:01.029 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.7898999999999999e-05 2024-09-13T11:25:01.029 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:01.029 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:01.030 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:25:01.030 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.792312+0000", 2024-09-13T11:25:01.030 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00012871000000000001 2024-09-13T11:25:01.030 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:01.030 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:01.030 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:25:01.030 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.793570+0000", 2024-09-13T11:25:01.030 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001257806 2024-09-13T11:25:01.030 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:01.030 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:25:01.030 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:01.030 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:01.030 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:01.030 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPush(3.12 75/69 [PushOp(3:4a36ca6d:::benchmark_data_smithi177_31116_object31707:head, version: 59'1993, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:4a36ca6d:::benchmark_data_smithi177_31116_object31707:head@59'1993, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:25:01.031 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:31.309623+0000", 2024-09-13T11:25:01.031 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 89.71429268, 2024-09-13T11:25:01.031 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.165459152, 2024-09-13T11:25:01.031 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:25:01.031 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:25:01.031 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:25:01.031 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:25:01.031 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:01.031 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:25:01.031 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.309623+0000", 2024-09-13T11:25:01.031 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:01.031 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:01.031 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:01.032 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:25:01.032 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.309623+0000", 2024-09-13T11:25:01.032 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:01.032 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:01.032 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:01.032 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:25:01.032 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.309625+0000", 2024-09-13T11:25:01.032 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.3860000000000001e-06 2024-09-13T11:25:01.032 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:01.032 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:01.032 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:25:01.032 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.309680+0000", 2024-09-13T11:25:01.032 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.4388000000000002e-05 2024-09-13T11:25:01.033 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:01.033 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:01.033 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:25:01.033 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.473157+0000", 2024-09-13T11:25:01.033 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.163476813 2024-09-13T11:25:01.033 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:01.033 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:01.033 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:25:01.033 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.473189+0000", 2024-09-13T11:25:01.033 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.2975000000000002e-05 2024-09-13T11:25:01.033 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:01.033 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:01.033 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:25:01.033 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.473395+0000", 2024-09-13T11:25:01.034 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000206009 2024-09-13T11:25:01.034 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:01.034 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:01.034 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:25:01.034 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.473424+0000", 2024-09-13T11:25:01.034 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.8255e-05 2024-09-13T11:25:01.034 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:01.034 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:01.034 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:25:01.034 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.475082+0000", 2024-09-13T11:25:01.034 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0016583259999999999 2024-09-13T11:25:01.034 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:01.034 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:25:01.034 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:01.035 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:01.035 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:01.035 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPushReply(3.13 76/70 [PushReplyOp(3:c9d43a82:::benchmark_data_smithi177_31116_object20026:head)])", 2024-09-13T11:25:01.035 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:32.427709+0000", 2024-09-13T11:25:01.035 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 88.596206120000005, 2024-09-13T11:25:01.035 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.46547316300000002, 2024-09-13T11:25:01.035 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:25:01.035 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:25:01.035 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-13T11:25:01.035 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:25:01.035 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:01.035 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:25:01.035 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:32.427709+0000", 2024-09-13T11:25:01.036 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:01.036 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:01.036 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:01.036 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:25:01.036 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:32.427709+0000", 2024-09-13T11:25:01.036 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:01.036 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:01.036 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:01.036 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:25:01.036 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:32.427711+0000", 2024-09-13T11:25:01.036 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5099999999999999e-06 2024-09-13T11:25:01.036 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:01.036 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:01.036 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:25:01.037 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:32.427717+0000", 2024-09-13T11:25:01.037 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.1569999999999997e-06 2024-09-13T11:25:01.037 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:01.037 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:01.037 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:25:01.037 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:32.892390+0000", 2024-09-13T11:25:01.037 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.46467251100000001 2024-09-13T11:25:01.037 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:01.037 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:01.037 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:25:01.037 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:32.892405+0000", 2024-09-13T11:25:01.037 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5296999999999999e-05 2024-09-13T11:25:01.037 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:01.037 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:01.038 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:25:01.038 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:32.892503+0000", 2024-09-13T11:25:01.038 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.7820999999999997e-05 2024-09-13T11:25:01.038 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:01.038 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:01.038 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:25:01.038 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:32.893183+0000", 2024-09-13T11:25:01.038 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00067986700000000001 2024-09-13T11:25:01.038 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:01.038 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:25:01.038 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:01.038 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:01.038 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:01.039 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:56066 3.a e79/75 3:561a1dd2:::benchmark_data_smithi177_31116_object56065:head v 79'3452, mlcod=79'3449)", 2024-09-13T11:25:01.039 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:36.194429+0000", 2024-09-13T11:25:01.039 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 84.829486235000005, 2024-09-13T11:25:01.039 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.13436872499999999, 2024-09-13T11:25:01.039 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:25:01.039 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:25:01.039 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:01.039 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:25:01.039 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:01.039 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:25:01.039 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:36.194429+0000", 2024-09-13T11:25:01.039 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:01.039 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:01.039 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:01.039 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:25:01.040 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:36.194429+0000", 2024-09-13T11:25:01.040 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:01.040 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:01.040 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:01.040 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:25:01.040 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:36.194431+0000", 2024-09-13T11:25:01.040 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.4279999999999999e-06 2024-09-13T11:25:01.040 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:01.040 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:01.040 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:25:01.040 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:36.194467+0000", 2024-09-13T11:25:01.040 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.6402000000000001e-05 2024-09-13T11:25:01.040 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:01.041 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:01.041 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:25:01.041 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:36.327354+0000", 2024-09-13T11:25:01.041 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.13288720800000001 2024-09-13T11:25:01.041 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:01.042 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:01.042 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:25:01.042 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:36.327361+0000", 2024-09-13T11:25:01.042 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.302e-06 2024-09-13T11:25:01.042 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:01.042 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:01.042 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:25:01.042 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:36.327441+0000", 2024-09-13T11:25:01.042 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.0406000000000002e-05 2024-09-13T11:25:01.042 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:01.042 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:01.042 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:25:01.042 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:36.327466+0000", 2024-09-13T11:25:01.042 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.5367e-05 2024-09-13T11:25:01.043 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:01.043 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:01.043 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:25:01.043 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:36.328754+0000", 2024-09-13T11:25:01.043 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001287627 2024-09-13T11:25:01.043 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:01.043 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:01.043 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:25:01.043 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:36.328798+0000", 2024-09-13T11:25:01.043 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.3985e-05 2024-09-13T11:25:01.043 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:01.043 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:25:01.043 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:01.043 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:01.044 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:01.044 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:57435 3.3 e80/75 3:c0db4777:::benchmark_data_smithi177_31116_object57434:head v 80'3155, mlcod=80'3153)", 2024-09-13T11:25:01.044 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:37.437070+0000", 2024-09-13T11:25:01.044 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 83.586845303000004, 2024-09-13T11:25:01.044 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.35861845999999997, 2024-09-13T11:25:01.044 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:25:01.044 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:25:01.044 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:01.044 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:25:01.044 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:01.044 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:25:01.044 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:37.437070+0000", 2024-09-13T11:25:01.044 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:01.044 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:01.045 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:01.045 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:25:01.045 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:37.437070+0000", 2024-09-13T11:25:01.045 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:01.045 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:01.045 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:01.045 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:25:01.045 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:37.437072+0000", 2024-09-13T11:25:01.045 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5549999999999999e-06 2024-09-13T11:25:01.045 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:01.045 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:01.045 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:25:01.045 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:37.437141+0000", 2024-09-13T11:25:01.046 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.9494999999999998e-05 2024-09-13T11:25:01.046 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:01.046 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:01.046 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:25:01.046 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:37.793400+0000", 2024-09-13T11:25:01.046 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.35625817999999998 2024-09-13T11:25:01.046 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:01.046 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:01.046 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:25:01.046 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:37.793406+0000", 2024-09-13T11:25:01.047 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.6830000000000002e-06 2024-09-13T11:25:01.047 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:01.047 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:01.047 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:25:01.047 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:37.793472+0000", 2024-09-13T11:25:01.047 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.5671000000000002e-05 2024-09-13T11:25:01.048 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:01.048 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:01.048 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:25:01.048 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:37.793492+0000", 2024-09-13T11:25:01.048 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.0051000000000002e-05 2024-09-13T11:25:01.048 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:01.048 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:01.048 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:25:01.048 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:37.795626+0000", 2024-09-13T11:25:01.048 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0021344990000000002 2024-09-13T11:25:01.049 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:01.049 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:01.049 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:25:01.049 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:37.795689+0000", 2024-09-13T11:25:01.049 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.2325999999999999e-05 2024-09-13T11:25:01.049 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:01.049 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:25:01.049 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:01.049 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:01.049 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:01.049 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:57985 3.3 e80/75 3:c330bb03:::benchmark_data_smithi177_31116_object57984:head v 80'3162, mlcod=80'3159)", 2024-09-13T11:25:01.049 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:38.007076+0000", 2024-09-13T11:25:01.049 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 83.016839950999994, 2024-09-13T11:25:01.050 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.18211438199999999, 2024-09-13T11:25:01.050 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:25:01.050 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:25:01.050 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:01.050 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:25:01.050 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:01.050 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:25:01.050 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:38.007076+0000", 2024-09-13T11:25:01.050 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:01.050 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:01.050 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:01.050 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:25:01.050 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:38.007076+0000", 2024-09-13T11:25:01.050 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:01.051 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:01.051 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:01.051 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:25:01.051 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:38.007077+0000", 2024-09-13T11:25:01.051 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.6759999999999999e-06 2024-09-13T11:25:01.051 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:01.051 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:01.051 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:25:01.051 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:38.007221+0000", 2024-09-13T11:25:01.051 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00014344299999999999 2024-09-13T11:25:01.051 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:01.051 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:01.051 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:25:01.051 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:38.186808+0000", 2024-09-13T11:25:01.051 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.17958726699999999 2024-09-13T11:25:01.052 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:01.052 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:01.052 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:25:01.052 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:38.186815+0000", 2024-09-13T11:25:01.052 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.0890000000000002e-06 2024-09-13T11:25:01.052 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:01.052 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:01.052 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:25:01.052 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:38.186887+0000", 2024-09-13T11:25:01.052 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.2006000000000001e-05 2024-09-13T11:25:01.052 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:01.052 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:01.052 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:25:01.052 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:38.186913+0000", 2024-09-13T11:25:01.053 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.5962e-05 2024-09-13T11:25:01.053 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:01.053 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:01.053 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:25:01.053 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:38.189123+0000", 2024-09-13T11:25:01.053 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0022099649999999999 2024-09-13T11:25:01.053 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:01.053 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:01.053 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:25:01.053 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:38.189190+0000", 2024-09-13T11:25:01.053 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.6973999999999999e-05 2024-09-13T11:25:01.053 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:01.053 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:25:01.053 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:01.054 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:01.054 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:01.054 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPushReply(3.23 84/78 [PushReplyOp(3:c7e44616:::benchmark_data_smithi177_31116_object32371:head)])", 2024-09-13T11:25:01.054 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:41.338210+0000", 2024-09-13T11:25:01.054 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 79.685705232000004, 2024-09-13T11:25:01.054 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.41118492299999998, 2024-09-13T11:25:01.054 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:25:01.054 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:25:01.054 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-13T11:25:01.054 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:25:01.054 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:01.054 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:25:01.054 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.338210+0000", 2024-09-13T11:25:01.054 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:01.055 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:01.055 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:01.055 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:25:01.055 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.338210+0000", 2024-09-13T11:25:01.055 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:01.055 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:01.055 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:01.055 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:25:01.055 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.338212+0000", 2024-09-13T11:25:01.055 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.314e-06 2024-09-13T11:25:01.055 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:01.055 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:01.055 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:25:01.055 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.338219+0000", 2024-09-13T11:25:01.055 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.6469999999999998e-06 2024-09-13T11:25:01.056 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:01.056 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:01.056 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:25:01.056 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.748675+0000", 2024-09-13T11:25:01.056 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.41045567500000002 2024-09-13T11:25:01.056 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:01.056 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:01.056 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:25:01.056 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.748684+0000", 2024-09-13T11:25:01.056 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.2150000000000007e-06 2024-09-13T11:25:01.056 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:01.056 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:01.056 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:25:01.057 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.748739+0000", 2024-09-13T11:25:01.057 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.4543000000000003e-05 2024-09-13T11:25:01.057 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:01.057 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:01.057 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:25:01.057 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.749395+0000", 2024-09-13T11:25:01.057 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00065652899999999999 2024-09-13T11:25:01.057 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:01.057 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:25:01.057 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:01.057 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:01.057 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:01.057 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPushReply(3.23 85/78 [PushReplyOp(3:c5c6dce1:::benchmark_data_smithi177_31116_object37293:head)])", 2024-09-13T11:25:01.057 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:42.377862+0000", 2024-09-13T11:25:01.058 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 78.646053773000006, 2024-09-13T11:25:01.058 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.62538808599999995, 2024-09-13T11:25:01.058 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:25:01.058 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:25:01.058 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-13T11:25:01.058 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:25:01.058 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:01.058 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:25:01.058 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.377862+0000", 2024-09-13T11:25:01.058 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:01.058 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:01.058 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:01.058 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:25:01.058 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.377862+0000", 2024-09-13T11:25:01.059 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:01.059 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:01.059 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:01.059 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:25:01.059 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.377863+0000", 2024-09-13T11:25:01.059 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.6899999999999996e-07 2024-09-13T11:25:01.059 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:01.059 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:01.059 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:25:01.059 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.377866+0000", 2024-09-13T11:25:01.059 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.219e-06 2024-09-13T11:25:01.059 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:01.059 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:01.059 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:25:01.060 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:43.001857+0000", 2024-09-13T11:25:01.060 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.62399084299999996 2024-09-13T11:25:01.060 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:01.060 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:01.060 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:25:01.060 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:43.001869+0000", 2024-09-13T11:25:01.060 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.2415999999999999e-05 2024-09-13T11:25:01.060 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:01.061 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:01.061 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:25:01.061 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:43.002194+0000", 2024-09-13T11:25:01.061 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000324774 2024-09-13T11:25:01.062 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:01.062 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:01.062 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:25:01.062 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:43.003250+0000", 2024-09-13T11:25:01.062 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001055865 2024-09-13T11:25:01.062 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:01.062 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:25:01.062 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:01.062 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:01.062 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:01.062 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPushReply(3.23 86/78 [PushReplyOp(3:c4ca4518:::benchmark_data_smithi177_31116_object39787:head)])", 2024-09-13T11:25:01.062 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:43.580816+0000", 2024-09-13T11:25:01.062 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 77.443100082000001, 2024-09-13T11:25:01.062 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.14999331199999999, 2024-09-13T11:25:01.063 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:25:01.063 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:25:01.063 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-13T11:25:01.063 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:25:01.063 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:01.063 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:25:01.063 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:43.580816+0000", 2024-09-13T11:25:01.063 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:01.063 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:01.063 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:01.063 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:25:01.063 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:43.580816+0000", 2024-09-13T11:25:01.063 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:01.063 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:01.064 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:01.064 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:25:01.064 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:43.580818+0000", 2024-09-13T11:25:01.064 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.7250000000000002e-06 2024-09-13T11:25:01.064 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:01.064 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:01.064 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:25:01.064 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:43.580830+0000", 2024-09-13T11:25:01.065 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.2247e-05 2024-09-13T11:25:01.065 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:01.065 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:01.065 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:25:01.065 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:43.730204+0000", 2024-09-13T11:25:01.065 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.149373531 2024-09-13T11:25:01.065 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:01.065 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:01.065 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:25:01.065 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:43.730213+0000", 2024-09-13T11:25:01.065 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.6989999999999992e-06 2024-09-13T11:25:01.065 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:01.065 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:01.066 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:25:01.066 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:43.730274+0000", 2024-09-13T11:25:01.066 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.1419000000000003e-05 2024-09-13T11:25:01.066 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:01.066 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:01.066 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:25:01.066 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:43.730809+0000", 2024-09-13T11:25:01.066 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00053469100000000001 2024-09-13T11:25:01.066 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:01.066 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:25:01.066 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:01.066 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:01.066 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:01.066 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPush(3.1c 89/77 [PushOp(3:38bcbc71:::benchmark_data_smithi177_31116_object31448:head, version: 59'1946, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:38bcbc71:::benchmark_data_smithi177_31116_object31448:head@59'1946, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:25:01.066 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:46.670723+0000", 2024-09-13T11:25:01.067 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 74.353192548999999, 2024-09-13T11:25:01.067 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.27576065, 2024-09-13T11:25:01.067 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:25:01.067 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:25:01.067 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:25:01.067 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:25:01.067 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:01.067 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:25:01.067 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.670723+0000", 2024-09-13T11:25:01.067 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:01.067 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:01.067 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:01.067 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:25:01.068 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.670723+0000", 2024-09-13T11:25:01.068 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:01.068 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:01.068 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:01.068 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:25:01.068 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.670724+0000", 2024-09-13T11:25:01.068 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.4389999999999999e-06 2024-09-13T11:25:01.068 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:01.068 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:01.068 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:25:01.068 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.670754+0000", 2024-09-13T11:25:01.068 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.9224999999999999e-05 2024-09-13T11:25:01.068 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:01.068 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:01.069 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:25:01.069 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.944369+0000", 2024-09-13T11:25:01.069 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.27361529800000001 2024-09-13T11:25:01.069 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:01.069 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:01.069 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:25:01.069 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.944422+0000", 2024-09-13T11:25:01.069 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.3318e-05 2024-09-13T11:25:01.069 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:01.069 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:01.069 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:25:01.069 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.944711+0000", 2024-09-13T11:25:01.069 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00028912399999999998 2024-09-13T11:25:01.070 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:01.070 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:01.070 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:25:01.070 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.944735+0000", 2024-09-13T11:25:01.070 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.3366999999999998e-05 2024-09-13T11:25:01.070 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:01.070 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:01.070 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:25:01.070 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.946484+0000", 2024-09-13T11:25:01.070 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0017488790000000001 2024-09-13T11:25:01.070 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:01.070 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:25:01.070 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:01.070 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:01.071 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:01.071 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPush(3.1c 91/77 [PushOp(3:3e18abc2:::benchmark_data_smithi177_31116_object35874:head, version: 62'2235, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:3e18abc2:::benchmark_data_smithi177_31116_object35874:head@62'2235, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:25:01.071 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:48.350968+0000", 2024-09-13T11:25:01.071 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 72.672947235999999, 2024-09-13T11:25:01.071 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.19257698000000001, 2024-09-13T11:25:01.071 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:25:01.071 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:25:01.071 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:25:01.071 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:25:01.071 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:01.071 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:25:01.071 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.350968+0000", 2024-09-13T11:25:01.071 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:01.072 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:01.072 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:01.072 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:25:01.072 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.350968+0000", 2024-09-13T11:25:01.072 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:01.072 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:01.072 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:01.072 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:25:01.072 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.350970+0000", 2024-09-13T11:25:01.072 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.796e-06 2024-09-13T11:25:01.072 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:01.072 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:01.072 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:25:01.073 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.351003+0000", 2024-09-13T11:25:01.073 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.2631e-05 2024-09-13T11:25:01.073 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:01.073 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:01.073 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:25:01.073 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.542140+0000", 2024-09-13T11:25:01.073 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.191136737 2024-09-13T11:25:01.073 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:01.073 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:01.073 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:25:01.073 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.542158+0000", 2024-09-13T11:25:01.073 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.8896999999999999e-05 2024-09-13T11:25:01.073 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:01.073 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:01.074 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:25:01.074 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.542263+0000", 2024-09-13T11:25:01.074 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000104158 2024-09-13T11:25:01.074 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:01.074 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:01.074 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:25:01.074 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.542279+0000", 2024-09-13T11:25:01.074 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5926999999999998e-05 2024-09-13T11:25:01.074 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:01.074 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:01.074 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:25:01.074 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.543545+0000", 2024-09-13T11:25:01.074 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0012668340000000001 2024-09-13T11:25:01.074 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:01.074 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:25:01.075 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:01.075 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:01.075 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:01.075 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:83211 3.1c e106/77 3:3ae31a0f:::benchmark_data_smithi177_31116_object83210:head v 106'4362, mlcod=106'4360)", 2024-09-13T11:25:01.075 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:04.345264+0000", 2024-09-13T11:25:01.075 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 56.678651584000001, 2024-09-13T11:25:01.075 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.17036547399999999, 2024-09-13T11:25:01.075 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:25:01.075 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:25:01.075 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:01.075 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:25:01.075 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:01.075 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:25:01.075 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.345264+0000", 2024-09-13T11:25:01.076 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:01.076 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:01.076 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:01.076 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:25:01.076 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.345264+0000", 2024-09-13T11:25:01.076 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:01.076 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:01.076 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:01.076 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:25:01.076 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.345267+0000", 2024-09-13T11:25:01.076 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.7590000000000002e-06 2024-09-13T11:25:01.076 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:01.077 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:01.077 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:25:01.077 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.345298+0000", 2024-09-13T11:25:01.077 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.1173000000000001e-05 2024-09-13T11:25:01.077 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:01.077 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:01.077 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:25:01.077 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.512265+0000", 2024-09-13T11:25:01.077 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.16696657100000001 2024-09-13T11:25:01.077 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:01.077 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:01.077 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:25:01.077 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.512279+0000", 2024-09-13T11:25:01.078 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.4953999999999999e-05 2024-09-13T11:25:01.078 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:01.078 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:01.078 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:25:01.078 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.512402+0000", 2024-09-13T11:25:01.078 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00012277999999999999 2024-09-13T11:25:01.078 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:01.078 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:01.078 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:25:01.078 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.512440+0000", 2024-09-13T11:25:01.078 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.7861999999999998e-05 2024-09-13T11:25:01.078 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:01.078 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:01.079 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:25:01.079 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.515515+0000", 2024-09-13T11:25:01.079 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00307472 2024-09-13T11:25:01.079 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:01.079 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:01.079 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:25:01.079 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.515629+0000", 2024-09-13T11:25:01.079 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00011465500000000001 2024-09-13T11:25:01.079 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:01.079 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:25:01.079 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:01.079 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:01.079 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:01.079 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:83223 3.1c e106/77 3:3bc6018d:::benchmark_data_smithi177_31116_object83222:head v 106'4363, mlcod=106'4360)", 2024-09-13T11:25:01.080 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:04.512139+0000", 2024-09-13T11:25:01.080 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 56.511776437999998, 2024-09-13T11:25:01.080 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.471923235, 2024-09-13T11:25:01.080 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:25:01.080 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:25:01.080 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:01.080 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:25:01.080 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:01.080 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:25:01.080 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.512139+0000", 2024-09-13T11:25:01.080 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:01.080 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:01.080 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:01.080 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:25:01.081 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.512139+0000", 2024-09-13T11:25:01.081 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:01.081 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:01.081 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:01.081 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:25:01.081 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.512143+0000", 2024-09-13T11:25:01.081 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.427e-06 2024-09-13T11:25:01.081 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:01.081 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:01.081 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:25:01.081 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.512227+0000", 2024-09-13T11:25:01.081 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.3965e-05 2024-09-13T11:25:01.081 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:01.081 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:01.082 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:25:01.082 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.976330+0000", 2024-09-13T11:25:01.082 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.46410342599999999 2024-09-13T11:25:01.082 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:01.082 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:01.082 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:25:01.082 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.976350+0000", 2024-09-13T11:25:01.082 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.9533000000000001e-05 2024-09-13T11:25:01.082 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:01.082 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:01.082 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:25:01.082 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.976488+0000", 2024-09-13T11:25:01.082 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00013799400000000001 2024-09-13T11:25:01.082 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:01.083 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:01.083 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:25:01.083 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.976543+0000", 2024-09-13T11:25:01.083 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.5442000000000002e-05 2024-09-13T11:25:01.083 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:01.083 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:01.083 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:25:01.083 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.983227+0000", 2024-09-13T11:25:01.083 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0066842899999999998 2024-09-13T11:25:01.083 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:01.083 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:01.083 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:25:01.083 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.984062+0000", 2024-09-13T11:25:01.083 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00083515800000000002 2024-09-13T11:25:01.083 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:01.084 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:25:01.084 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:01.084 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:01.084 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:01.084 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:110256 3.13 e119/83 3:cb00d078:::benchmark_data_smithi177_31116_object110255:head v 119'4823, mlcod=119'4821)", 2024-09-13T11:25:01.084 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:25.284530+0000", 2024-09-13T11:25:01.084 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 35.739385978000001, 2024-09-13T11:25:01.084 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.343491771, 2024-09-13T11:25:01.084 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:25:01.084 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:25:01.084 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:01.084 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:25:01.084 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:01.085 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:25:01.085 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.284530+0000", 2024-09-13T11:25:01.085 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:01.085 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:01.085 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:01.085 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:25:01.085 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.284530+0000", 2024-09-13T11:25:01.085 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:01.085 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:01.085 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:01.085 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:25:01.085 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.284532+0000", 2024-09-13T11:25:01.085 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.8870000000000001e-06 2024-09-13T11:25:01.086 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:01.086 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:01.086 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:25:01.086 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.284558+0000", 2024-09-13T11:25:01.086 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.6899000000000001e-05 2024-09-13T11:25:01.086 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:01.086 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:01.086 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:25:01.086 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.626172+0000", 2024-09-13T11:25:01.086 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.34161370800000002 2024-09-13T11:25:01.086 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:01.086 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:01.086 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:25:01.086 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.626187+0000", 2024-09-13T11:25:01.087 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.485e-05 2024-09-13T11:25:01.087 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:01.087 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:01.087 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:25:01.087 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.626245+0000", 2024-09-13T11:25:01.087 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.7998000000000003e-05 2024-09-13T11:25:01.087 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:01.087 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:01.087 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:25:01.087 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.626260+0000", 2024-09-13T11:25:01.087 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5429000000000001e-05 2024-09-13T11:25:01.087 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:01.087 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:01.087 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:25:01.088 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.627924+0000", 2024-09-13T11:25:01.088 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0016634060000000001 2024-09-13T11:25:01.088 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:01.088 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:01.088 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:25:01.088 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.628021+0000", 2024-09-13T11:25:01.088 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.7594000000000001e-05 2024-09-13T11:25:01.088 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:01.088 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:25:01.088 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:01.088 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:01.088 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:01.088 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPush(3.2b 121/119 [PushOp(3:d53a23e8:::benchmark_data_smithi177_31116_object5366:head, version: 45'342, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:d53a23e8:::benchmark_data_smithi177_31116_object5366:head@45'342, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:25:01.089 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:27.268731+0000", 2024-09-13T11:25:01.089 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 33.755184516, 2024-09-13T11:25:01.089 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.72228345000000005, 2024-09-13T11:25:01.089 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:25:01.089 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:25:01.089 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:25:01.089 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:25:01.089 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:01.089 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:25:01.089 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:27.268731+0000", 2024-09-13T11:25:01.089 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:01.089 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:01.089 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:01.089 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:25:01.090 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:27.268731+0000", 2024-09-13T11:25:01.090 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:01.090 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:01.090 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:01.090 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:25:01.090 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:27.268733+0000", 2024-09-13T11:25:01.090 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.9740000000000001e-06 2024-09-13T11:25:01.090 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:01.090 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:01.090 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:25:01.090 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:27.268778+0000", 2024-09-13T11:25:01.090 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.4679999999999999e-05 2024-09-13T11:25:01.090 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:01.091 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:01.091 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:25:01.091 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:27.988594+0000", 2024-09-13T11:25:01.091 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.71981667299999996 2024-09-13T11:25:01.091 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:01.091 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:01.091 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:25:01.091 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:27.988652+0000", 2024-09-13T11:25:01.091 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.7640999999999999e-05 2024-09-13T11:25:01.091 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:01.091 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:01.091 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:25:01.091 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:27.988889+0000", 2024-09-13T11:25:01.091 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00023706500000000001 2024-09-13T11:25:01.092 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:01.092 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:01.092 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:25:01.092 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:27.988933+0000", 2024-09-13T11:25:01.092 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.3865000000000001e-05 2024-09-13T11:25:01.092 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:01.092 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:01.092 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:25:01.092 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:27.991015+0000", 2024-09-13T11:25:01.092 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.002081552 2024-09-13T11:25:01.092 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:01.092 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:25:01.092 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:01.092 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:01.093 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:01.093 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPush(3.2b 124/119 [PushOp(3:d64c8aaf:::benchmark_data_smithi177_31116_object31700:head, version: 59'1991, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:d64c8aaf:::benchmark_data_smithi177_31116_object31700:head@59'1991, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:25:01.093 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:31.579304+0000", 2024-09-13T11:25:01.093 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 29.444611815000002, 2024-09-13T11:25:01.093 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.14279340800000001, 2024-09-13T11:25:01.093 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:25:01.093 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:25:01.093 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:25:01.093 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:25:01.093 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:01.093 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:25:01.093 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:31.579304+0000", 2024-09-13T11:25:01.093 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:01.093 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:01.094 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:01.094 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:25:01.094 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:31.579304+0000", 2024-09-13T11:25:01.094 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:01.094 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:01.094 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:01.094 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:25:01.094 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:31.579305+0000", 2024-09-13T11:25:01.094 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5090000000000001e-06 2024-09-13T11:25:01.094 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:01.094 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:01.094 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:25:01.094 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:31.579367+0000", 2024-09-13T11:25:01.094 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.1279999999999996e-05 2024-09-13T11:25:01.094 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:01.095 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:01.095 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:25:01.095 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:31.721234+0000", 2024-09-13T11:25:01.095 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.141867883 2024-09-13T11:25:01.095 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:01.095 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:01.095 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:25:01.095 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:31.721253+0000", 2024-09-13T11:25:01.095 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.8725000000000001e-05 2024-09-13T11:25:01.095 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:01.095 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:01.095 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:25:01.095 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:31.721332+0000", 2024-09-13T11:25:01.095 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.8808999999999997e-05 2024-09-13T11:25:01.096 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:01.096 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:01.096 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:25:01.096 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:31.721345+0000", 2024-09-13T11:25:01.096 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3414999999999999e-05 2024-09-13T11:25:01.096 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:01.096 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:01.096 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:25:01.096 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:31.722097+0000", 2024-09-13T11:25:01.096 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00075178699999999996 2024-09-13T11:25:01.096 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:01.096 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:25:01.096 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:01.097 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:01.097 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:01.097 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:113511 3.2b e124/119 3:d74f5e33:::benchmark_data_smithi177_31116_object113510:head v 124'5156, mlcod=60'2020)", 2024-09-13T11:25:01.097 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:32.032517+0000", 2024-09-13T11:25:01.097 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 28.991398651000001, 2024-09-13T11:25:01.097 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.13874039699999999, 2024-09-13T11:25:01.097 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:25:01.097 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:25:01.097 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:01.097 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:25:01.097 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:01.097 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:25:01.097 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:32.032517+0000", 2024-09-13T11:25:01.098 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:01.098 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:01.098 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:01.098 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:25:01.098 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:32.032517+0000", 2024-09-13T11:25:01.098 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:01.098 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:01.098 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:01.098 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:25:01.098 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:32.032518+0000", 2024-09-13T11:25:01.098 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3060000000000001e-06 2024-09-13T11:25:01.098 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:01.098 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:01.098 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:25:01.099 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:32.032545+0000", 2024-09-13T11:25:01.099 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.6800000000000001e-05 2024-09-13T11:25:01.099 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:01.099 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:01.099 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:25:01.099 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:32.169285+0000", 2024-09-13T11:25:01.099 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.13673950800000001 2024-09-13T11:25:01.099 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:01.099 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:01.099 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:25:01.099 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:32.169291+0000", 2024-09-13T11:25:01.099 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.934e-06 2024-09-13T11:25:01.100 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:01.100 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:01.100 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:25:01.100 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:32.169381+0000", 2024-09-13T11:25:01.100 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.9860999999999999e-05 2024-09-13T11:25:01.100 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:01.100 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:01.100 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:25:01.100 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:32.169406+0000", 2024-09-13T11:25:01.100 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.5052e-05 2024-09-13T11:25:01.100 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:01.100 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:01.100 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:25:01.100 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:32.171218+0000", 2024-09-13T11:25:01.101 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0018111659999999999 2024-09-13T11:25:01.101 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:01.101 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:01.101 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:25:01.101 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:32.171257+0000", 2024-09-13T11:25:01.101 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.977e-05 2024-09-13T11:25:01.101 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:01.101 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:25:01.101 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:01.101 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:01.102 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:01.102 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPush(3.2b 124/119 [PushOp(3:d70bd127:::benchmark_data_smithi177_31116_object32365:head, version: 60'2034, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:d70bd127:::benchmark_data_smithi177_31116_object32365:head@60'2034, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:25:01.102 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:32.037338+0000", 2024-09-13T11:25:01.102 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 28.986577242999999, 2024-09-13T11:25:01.102 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.24358922699999999, 2024-09-13T11:25:01.102 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:25:01.102 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:25:01.102 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:25:01.102 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:25:01.102 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:01.102 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:25:01.102 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:32.037338+0000", 2024-09-13T11:25:01.102 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:01.103 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:01.103 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:01.103 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:25:01.103 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:32.037338+0000", 2024-09-13T11:25:01.103 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:01.103 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:01.103 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:01.103 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:25:01.103 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:32.037340+0000", 2024-09-13T11:25:01.103 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3790000000000001e-06 2024-09-13T11:25:01.103 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:01.103 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:01.103 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:25:01.103 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:32.037411+0000", 2024-09-13T11:25:01.103 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.1254000000000006e-05 2024-09-13T11:25:01.104 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:01.104 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:01.104 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:25:01.104 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:32.277467+0000", 2024-09-13T11:25:01.104 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.24005563199999999 2024-09-13T11:25:01.104 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:01.104 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:01.104 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:25:01.104 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:32.277494+0000", 2024-09-13T11:25:01.104 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.7120000000000001e-05 2024-09-13T11:25:01.105 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:01.105 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:01.105 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:25:01.105 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:32.279860+0000", 2024-09-13T11:25:01.105 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0023667559999999998 2024-09-13T11:25:01.105 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:01.105 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:01.105 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:25:01.105 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:32.279883+0000", 2024-09-13T11:25:01.105 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.2138999999999999e-05 2024-09-13T11:25:01.105 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:01.105 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:01.105 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:25:01.105 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:32.280928+0000", 2024-09-13T11:25:01.106 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0010449470000000001 2024-09-13T11:25:01.106 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:01.106 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:25:01.106 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:01.106 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:01.106 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:01.106 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPush(3.2b 125/119 [PushOp(3:d4c5ede9:::benchmark_data_smithi177_31116_object33277:head, version: 60'2092, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:d4c5ede9:::benchmark_data_smithi177_31116_object33277:head@60'2092, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:25:01.106 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:32.403929+0000", 2024-09-13T11:25:01.106 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 28.619986792999999, 2024-09-13T11:25:01.106 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.194592864, 2024-09-13T11:25:01.106 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:25:01.106 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:25:01.106 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:25:01.106 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:25:01.106 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:01.107 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:25:01.107 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:32.403929+0000", 2024-09-13T11:25:01.107 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:01.107 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:01.107 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:01.107 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:25:01.107 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:32.403929+0000", 2024-09-13T11:25:01.107 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:01.107 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:01.107 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:01.107 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:25:01.107 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:32.403930+0000", 2024-09-13T11:25:01.107 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.6729999999999999e-06 2024-09-13T11:25:01.107 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:01.107 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:01.108 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:25:01.108 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:32.403979+0000", 2024-09-13T11:25:01.108 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.8242000000000003e-05 2024-09-13T11:25:01.108 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:01.108 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:01.108 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:25:01.108 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:32.597445+0000", 2024-09-13T11:25:01.108 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.19346676500000001 2024-09-13T11:25:01.108 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:01.108 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:01.108 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:25:01.108 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:32.597455+0000", 2024-09-13T11:25:01.108 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.2939999999999994e-06 2024-09-13T11:25:01.109 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:01.109 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:01.109 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:25:01.109 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:32.597514+0000", 2024-09-13T11:25:01.109 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.9509000000000002e-05 2024-09-13T11:25:01.109 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:01.109 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:01.109 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:25:01.109 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:32.597522+0000", 2024-09-13T11:25:01.110 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.0350000000000001e-06 2024-09-13T11:25:01.110 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:01.110 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:01.110 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:25:01.110 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:32.598522+0000", 2024-09-13T11:25:01.110 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00099934599999999991 2024-09-13T11:25:01.110 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:01.110 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:25:01.110 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:01.110 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:01.110 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:01.110 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPush(3.2b 128/119 [PushOp(3:d6ff590f:::benchmark_data_smithi177_31116_object64581:head, version: 90'3750, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:d6ff590f:::benchmark_data_smithi177_31116_object64581:head@90'3750, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:25:01.110 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:37.982062+0000", 2024-09-13T11:25:01.110 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 23.041853893999999, 2024-09-13T11:25:01.111 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.28899675200000002, 2024-09-13T11:25:01.111 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:25:01.111 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:25:01.111 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:25:01.111 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:25:01.111 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:01.111 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:25:01.111 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:37.982062+0000", 2024-09-13T11:25:01.111 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:01.111 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:01.111 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:01.111 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:25:01.111 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:37.982062+0000", 2024-09-13T11:25:01.111 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:01.112 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:01.112 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:01.112 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:25:01.112 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:37.982064+0000", 2024-09-13T11:25:01.112 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.2259999999999999e-06 2024-09-13T11:25:01.112 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:01.112 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:01.112 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:25:01.112 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:37.982114+0000", 2024-09-13T11:25:01.112 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.0330000000000001e-05 2024-09-13T11:25:01.112 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:01.112 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:01.112 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:25:01.112 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:38.269983+0000", 2024-09-13T11:25:01.113 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.28786869999999998 2024-09-13T11:25:01.113 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:01.113 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:01.113 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:25:01.113 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:38.270000+0000", 2024-09-13T11:25:01.113 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.6977999999999999e-05 2024-09-13T11:25:01.113 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:01.113 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:01.113 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:25:01.113 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:38.270092+0000", 2024-09-13T11:25:01.113 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.221e-05 2024-09-13T11:25:01.113 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:01.113 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:01.113 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:25:01.114 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:38.270103+0000", 2024-09-13T11:25:01.114 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.1253999999999999e-05 2024-09-13T11:25:01.114 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:01.114 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:01.114 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:25:01.114 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:38.271058+0000", 2024-09-13T11:25:01.114 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00095505399999999999 2024-09-13T11:25:01.114 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:01.114 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:25:01.114 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:01.114 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:01.114 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:25:01.114 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-09-13T11:25:01.115 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-13T11:25:01.116 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-09-13T11:25:01.036+0000 7f97b59fa640 -1 osd.6 146 handle_osd_map injecting map crc failure 2024-09-13T11:25:01.116 INFO:tasks.ceph.osd.4.smithi001.stderr:2024-09-13T11:25:01.031+0000 7ff8d0c3d640 -1 osd.4 146 handle_osd_map injecting map crc failure 2024-09-13T11:25:01.117 INFO:tasks.ceph.osd.11.smithi177.stderr:2024-09-13T11:25:01.022+0000 7f24792c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26580) UID: 0 2024-09-13T11:25:01.117 INFO:teuthology.orchestra.run.smithi001.stdout:osd.0: {} 2024-09-13T11:25:01.118 INFO:tasks.ceph.osd.7.smithi177.stderr:2024-09-13T11:25:01.026+0000 7f896ff4a640 -1 osd.7 146 handle_osd_map injecting map crc failure 2024-09-13T11:25:01.118 INFO:teuthology.orchestra.run.smithi001.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T11:25:01.119 INFO:teuthology.orchestra.run.smithi001.stdout:osd.1: {} 2024-09-13T11:25:01.120 INFO:teuthology.orchestra.run.smithi001.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T11:25:01.120 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-09-13T11:25:01.123 INFO:teuthology.orchestra.run.smithi001.stdout:osd.2: {} 2024-09-13T11:25:01.123 INFO:teuthology.orchestra.run.smithi001.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T11:25:01.125 INFO:tasks.ceph.osd.14.smithi162.stderr:2024-09-13T11:25:01.120+0000 7f3df5903640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33154) UID: 0 2024-09-13T11:25:01.151 INFO:teuthology.orchestra.run.smithi001.stdout:osd.3: {} 2024-09-13T11:25:01.151 INFO:teuthology.orchestra.run.smithi001.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T11:25:01.181 INFO:teuthology.orchestra.run.smithi001.stdout:osd.4: {} 2024-09-13T11:25:01.181 INFO:teuthology.orchestra.run.smithi001.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T11:25:01.220 INFO:teuthology.orchestra.run.smithi001.stdout:osd.5: {} 2024-09-13T11:25:01.220 INFO:teuthology.orchestra.run.smithi001.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T11:25:01.227 INFO:tasks.ceph.osd.15.smithi177.stderr:2024-09-13T11:25:01.222+0000 7f9d6ce77640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26578) UID: 0 2024-09-13T11:25:01.247 INFO:teuthology.orchestra.run.smithi001.stdout:osd.6: {} 2024-09-13T11:25:01.248 INFO:teuthology.orchestra.run.smithi001.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T11:25:01.277 INFO:teuthology.orchestra.run.smithi001.stdout:osd.7: {} 2024-09-13T11:25:01.277 INFO:teuthology.orchestra.run.smithi001.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T11:25:01.295 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 15] out_osds: [14] dead_osds: [] live_osds: [0, 4, 8, 12, 5, 9, 13, 2, 6, 10, 3, 7, 11, 15, 1, 14] 2024-09-13T11:25:01.296 INFO:tasks.thrashosds.thrasher:choose_action: min_in 2 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-13T11:25:01.296 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-09-13T11:25:01.297 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 'trim stale osdmaps' 2024-09-13T11:25:01.305 INFO:teuthology.orchestra.run.smithi001.stdout:osd.8: {} 2024-09-13T11:25:01.306 INFO:teuthology.orchestra.run.smithi001.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-13T11:25:01.327 INFO:tasks.ceph.osd.13.smithi092.stderr:2024-09-13T11:25:01.322+0000 7f3e5afea640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26350) UID: 0 2024-09-13T11:25:01.328 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-09-13T11:25:01.328 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [ 2024-09-13T11:25:01.328 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:01.328 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:144018 3.1c 3:3a867a3b:::benchmark_data_smithi177_31116_object3715:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e147)", 2024-09-13T11:25:01.329 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:25:01.318409+0000", 2024-09-13T11:25:01.329 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 0.0095501939999999997, 2024-09-13T11:25:01.329 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0095776650000000008, 2024-09-13T11:25:01.329 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:25:01.329 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:25:01.329 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "waiting for sub ops", 2024-09-13T11:25:01.329 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:25:01.329 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:25:01.329 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:01.329 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 144018 2024-09-13T11:25:01.329 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:01.329 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:25:01.329 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:01.330 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:25:01.330 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:01.318409+0000", 2024-09-13T11:25:01.330 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:01.330 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:01.330 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:01.330 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:25:01.330 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:01.318409+0000", 2024-09-13T11:25:01.330 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:01.330 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:01.330 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:01.330 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:25:01.330 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:01.318411+0000", 2024-09-13T11:25:01.330 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.6479999999999999e-06 2024-09-13T11:25:01.330 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:01.331 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:01.331 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:25:01.331 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:01.318416+0000", 2024-09-13T11:25:01.331 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.9320000000000004e-06 2024-09-13T11:25:01.331 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:01.331 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:01.331 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:25:01.331 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:01.318426+0000", 2024-09-13T11:25:01.331 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0078000000000001e-05 2024-09-13T11:25:01.331 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:01.331 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:01.331 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:25:01.331 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:01.318433+0000", 2024-09-13T11:25:01.332 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.1289999999999998e-06 2024-09-13T11:25:01.332 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:01.332 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:01.332 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:25:01.332 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:01.318492+0000", 2024-09-13T11:25:01.332 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.9611000000000002e-05 2024-09-13T11:25:01.332 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:01.332 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:01.332 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:25:01.332 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:01.318591+0000", 2024-09-13T11:25:01.332 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.8686999999999999e-05 2024-09-13T11:25:01.332 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:01.332 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:01.333 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 7", 2024-09-13T11:25:01.333 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:01.318687+0000", 2024-09-13T11:25:01.333 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.6285000000000004e-05 2024-09-13T11:25:01.333 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:01.333 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:01.333 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:25:01.333 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:01.321115+0000", 2024-09-13T11:25:01.333 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0024272069999999998 2024-09-13T11:25:01.333 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:01.333 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:25:01.333 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:01.333 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:01.333 INFO:teuthology.orchestra.run.smithi177.stdout: ], 2024-09-13T11:25:01.333 INFO:teuthology.orchestra.run.smithi177.stdout: "num_ops": 1 2024-09-13T11:25:01.334 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-09-13T11:25:01.338 INFO:teuthology.orchestra.run.smithi001.stdout:osd.9: {} 2024-09-13T11:25:01.338 INFO:teuthology.orchestra.run.smithi001.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-13T11:25:01.350 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-09-13T11:25:01.384 INFO:teuthology.orchestra.run.smithi001.stdout:osd.10: {} 2024-09-13T11:25:01.385 INFO:teuthology.orchestra.run.smithi001.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-13T11:25:01.429 INFO:tasks.ceph.osd.13.smithi092.stderr:2024-09-13T11:25:01.422+0000 7f3e5afea640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26350) UID: 0 2024-09-13T11:25:01.432 INFO:teuthology.orchestra.run.smithi001.stdout:osd.11: {} 2024-09-13T11:25:01.433 INFO:teuthology.orchestra.run.smithi001.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-13T11:25:01.464 INFO:teuthology.orchestra.run.smithi001.stdout:osd.12: {} 2024-09-13T11:25:01.465 INFO:teuthology.orchestra.run.smithi001.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-13T11:25:01.495 INFO:teuthology.orchestra.run.smithi001.stdout:osd.13: {} 2024-09-13T11:25:01.495 INFO:teuthology.orchestra.run.smithi001.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-13T11:25:01.529 INFO:teuthology.orchestra.run.smithi001.stdout:osd.14: {} 2024-09-13T11:25:01.529 INFO:teuthology.orchestra.run.smithi001.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-13T11:25:01.530 INFO:tasks.ceph.osd.14.smithi162.stderr:2024-09-13T11:25:01.528+0000 7f3df5903640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33154) UID: 0 2024-09-13T11:25:01.558 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-09-13T11:25:01.559 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:25:01.567 INFO:teuthology.orchestra.run.smithi001.stderr:no valid command found; 10 closest matches: 2024-09-13T11:25:01.568 INFO:teuthology.orchestra.run.smithi001.stderr:0 2024-09-13T11:25:01.568 INFO:teuthology.orchestra.run.smithi001.stderr:1 2024-09-13T11:25:01.568 INFO:teuthology.orchestra.run.smithi001.stderr:2 2024-09-13T11:25:01.568 INFO:teuthology.orchestra.run.smithi001.stderr:abort 2024-09-13T11:25:01.568 INFO:teuthology.orchestra.run.smithi001.stderr:assert 2024-09-13T11:25:01.568 INFO:teuthology.orchestra.run.smithi001.stderr:bench [] [] [] [] 2024-09-13T11:25:01.568 INFO:teuthology.orchestra.run.smithi001.stderr:bluefs debug_inject_read_zeros 2024-09-13T11:25:01.568 INFO:teuthology.orchestra.run.smithi001.stderr:bluefs files list 2024-09-13T11:25:01.568 INFO:teuthology.orchestra.run.smithi001.stderr:bluefs stats 2024-09-13T11:25:01.569 INFO:teuthology.orchestra.run.smithi001.stderr:bluestore allocator dump block 2024-09-13T11:25:01.569 INFO:teuthology.orchestra.run.smithi001.stderr:Error EINVAL: invalid command 2024-09-13T11:25:01.571 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-13T11:25:01.571 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-13T11:25:01.572 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 'trim stale osdmaps' 2024-09-13T11:25:01.580 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-09-13T11:25:01.580 INFO:teuthology.orchestra.run.smithi001.stdout:osd.15: {} 2024-09-13T11:25:01.581 INFO:teuthology.orchestra.run.smithi001.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-13T11:25:01.631 INFO:tasks.ceph.osd.2.smithi162.stderr:2024-09-13T11:25:01.628+0000 7fe92b35a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26313) UID: 0 2024-09-13T11:25:01.731 INFO:tasks.ceph.osd.9.smithi092.stderr:2024-09-13T11:25:01.726+0000 7ff5ab3d0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26348) UID: 0 2024-09-13T11:25:01.754 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-09-13T11:25:01.754 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:25:01.778 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_ops_in_flight 2024-09-13T11:25:01.784 INFO:teuthology.orchestra.run.smithi001.stderr:no valid command found; 10 closest matches: 2024-09-13T11:25:01.784 INFO:teuthology.orchestra.run.smithi001.stderr:0 2024-09-13T11:25:01.785 INFO:teuthology.orchestra.run.smithi001.stderr:1 2024-09-13T11:25:01.785 INFO:teuthology.orchestra.run.smithi001.stderr:2 2024-09-13T11:25:01.785 INFO:teuthology.orchestra.run.smithi001.stderr:abort 2024-09-13T11:25:01.785 INFO:teuthology.orchestra.run.smithi001.stderr:assert 2024-09-13T11:25:01.785 INFO:teuthology.orchestra.run.smithi001.stderr:bench [] [] [] [] 2024-09-13T11:25:01.785 INFO:teuthology.orchestra.run.smithi001.stderr:bluefs debug_inject_read_zeros 2024-09-13T11:25:01.785 INFO:teuthology.orchestra.run.smithi001.stderr:bluefs files list 2024-09-13T11:25:01.785 INFO:teuthology.orchestra.run.smithi001.stderr:bluefs stats 2024-09-13T11:25:01.785 INFO:teuthology.orchestra.run.smithi001.stderr:bluestore allocator dump block 2024-09-13T11:25:01.785 INFO:teuthology.orchestra.run.smithi001.stderr:Error EINVAL: invalid command 2024-09-13T11:25:01.786 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-13T11:25:01.786 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-13T11:25:01.786 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 'trim stale osdmaps' 2024-09-13T11:25:01.833 INFO:tasks.ceph.osd.14.smithi162.stderr:2024-09-13T11:25:01.828+0000 7f3df5903640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33154) UID: 0 2024-09-13T11:25:01.924 INFO:teuthology.orchestra.run.smithi001.stderr:noscrub is set 2024-09-13T11:25:01.934 INFO:tasks.ceph.osd.15.smithi177.stderr:2024-09-13T11:25:01.930+0000 7f9d6ce77640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26578) UID: 0 2024-09-13T11:25:01.947 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-09-13T11:25:01.947 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:25:01.966 INFO:teuthology.orchestra.run.smithi001.stderr:no valid command found; 10 closest matches: 2024-09-13T11:25:01.966 INFO:teuthology.orchestra.run.smithi001.stderr:0 2024-09-13T11:25:01.966 INFO:teuthology.orchestra.run.smithi001.stderr:1 2024-09-13T11:25:01.966 INFO:teuthology.orchestra.run.smithi001.stderr:2 2024-09-13T11:25:01.966 INFO:teuthology.orchestra.run.smithi001.stderr:abort 2024-09-13T11:25:01.966 INFO:teuthology.orchestra.run.smithi001.stderr:assert 2024-09-13T11:25:01.966 INFO:teuthology.orchestra.run.smithi001.stderr:bench [] [] [] [] 2024-09-13T11:25:01.967 INFO:teuthology.orchestra.run.smithi001.stderr:bluefs debug_inject_read_zeros 2024-09-13T11:25:01.967 INFO:teuthology.orchestra.run.smithi001.stderr:bluefs files list 2024-09-13T11:25:01.967 INFO:teuthology.orchestra.run.smithi001.stderr:bluefs stats 2024-09-13T11:25:01.967 INFO:teuthology.orchestra.run.smithi001.stderr:bluestore allocator dump block 2024-09-13T11:25:01.967 INFO:teuthology.orchestra.run.smithi001.stderr:Error EINVAL: invalid command 2024-09-13T11:25:01.970 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-13T11:25:01.970 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-13T11:25:01.970 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 'trim stale osdmaps' 2024-09-13T11:25:01.971 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_blocked_ops 2024-09-13T11:25:02.034 INFO:tasks.ceph.osd.13.smithi092.stderr:2024-09-13T11:25:02.030+0000 7f3e5afea640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26350) UID: 0 2024-09-13T11:25:02.134 INFO:tasks.ceph.osd.4.smithi001.stderr:2024-09-13T11:25:02.131+0000 7ff8e0218640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26310) UID: 0 2024-09-13T11:25:02.178 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-09-13T11:25:02.178 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:25:02.186 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-13T11:25:02.186 INFO:teuthology.orchestra.run.smithi001.stderr:no valid command found; 10 closest matches: 2024-09-13T11:25:02.186 INFO:teuthology.orchestra.run.smithi001.stderr:0 2024-09-13T11:25:02.186 INFO:teuthology.orchestra.run.smithi001.stderr:1 2024-09-13T11:25:02.187 INFO:teuthology.orchestra.run.smithi001.stderr:2 2024-09-13T11:25:02.187 INFO:teuthology.orchestra.run.smithi001.stderr:abort 2024-09-13T11:25:02.187 INFO:teuthology.orchestra.run.smithi001.stderr:assert 2024-09-13T11:25:02.187 INFO:teuthology.orchestra.run.smithi001.stderr:bench [] [] [] [] 2024-09-13T11:25:02.187 INFO:teuthology.orchestra.run.smithi001.stderr:bluefs debug_inject_read_zeros 2024-09-13T11:25:02.187 INFO:teuthology.orchestra.run.smithi001.stderr:bluefs files list 2024-09-13T11:25:02.187 INFO:teuthology.orchestra.run.smithi001.stderr:bluefs stats 2024-09-13T11:25:02.187 INFO:teuthology.orchestra.run.smithi001.stderr:bluestore allocator dump block 2024-09-13T11:25:02.187 INFO:teuthology.orchestra.run.smithi001.stderr:Error EINVAL: invalid command 2024-09-13T11:25:02.187 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-13T11:25:02.187 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 'trim stale osdmaps' 2024-09-13T11:25:02.200 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_historic_ops 2024-09-13T11:25:02.236 INFO:tasks.ceph.osd.13.smithi092.stderr:2024-09-13T11:25:02.230+0000 7f3e5afea640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26350) UID: 0 2024-09-13T11:25:02.337 INFO:tasks.ceph.osd.14.smithi162.stderr:2024-09-13T11:25:02.332+0000 7f3df5903640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33154) UID: 0 2024-09-13T11:25:02.340 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-09-13T11:25:02.341 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:25:02.361 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-13T11:25:02.411 INFO:teuthology.orchestra.run.smithi001.stderr:no valid command found; 10 closest matches: 2024-09-13T11:25:02.412 INFO:teuthology.orchestra.run.smithi001.stderr:0 2024-09-13T11:25:02.412 INFO:teuthology.orchestra.run.smithi001.stderr:1 2024-09-13T11:25:02.412 INFO:teuthology.orchestra.run.smithi001.stderr:2 2024-09-13T11:25:02.412 INFO:teuthology.orchestra.run.smithi001.stderr:abort 2024-09-13T11:25:02.412 INFO:teuthology.orchestra.run.smithi001.stderr:assert 2024-09-13T11:25:02.412 INFO:teuthology.orchestra.run.smithi001.stderr:bench [] [] [] [] 2024-09-13T11:25:02.412 INFO:teuthology.orchestra.run.smithi001.stderr:bluefs debug_inject_read_zeros 2024-09-13T11:25:02.412 INFO:teuthology.orchestra.run.smithi001.stderr:bluefs files list 2024-09-13T11:25:02.412 INFO:teuthology.orchestra.run.smithi001.stderr:bluefs stats 2024-09-13T11:25:02.412 INFO:teuthology.orchestra.run.smithi001.stderr:bluestore allocator dump block 2024-09-13T11:25:02.412 INFO:teuthology.orchestra.run.smithi001.stderr:Error EINVAL: invalid command 2024-09-13T11:25:02.415 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-13T11:25:02.415 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-13T11:25:02.415 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 'trim stale osdmaps' 2024-09-13T11:25:02.437 INFO:tasks.ceph.osd.11.smithi177.stderr:2024-09-13T11:25:02.434+0000 7f24792c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26580) UID: 0 2024-09-13T11:25:02.519 INFO:teuthology.orchestra.run.smithi092.stdout:ERROR: (22) Invalid argument 2024-09-13T11:25:02.519 INFO:teuthology.orchestra.run.smithi092.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:25:02.538 INFO:tasks.ceph.osd.10.smithi162.stderr:2024-09-13T11:25:02.536+0000 7fb59c9f0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26317) UID: 0 2024-09-13T11:25:02.540 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-13T11:25:02.611 INFO:teuthology.orchestra.run.smithi001.stderr:no valid command found; 10 closest matches: 2024-09-13T11:25:02.611 INFO:teuthology.orchestra.run.smithi001.stderr:0 2024-09-13T11:25:02.611 INFO:teuthology.orchestra.run.smithi001.stderr:1 2024-09-13T11:25:02.611 INFO:teuthology.orchestra.run.smithi001.stderr:2 2024-09-13T11:25:02.611 INFO:teuthology.orchestra.run.smithi001.stderr:abort 2024-09-13T11:25:02.612 INFO:teuthology.orchestra.run.smithi001.stderr:assert 2024-09-13T11:25:02.612 INFO:teuthology.orchestra.run.smithi001.stderr:bench [] [] [] [] 2024-09-13T11:25:02.612 INFO:teuthology.orchestra.run.smithi001.stderr:bluefs debug_inject_read_zeros 2024-09-13T11:25:02.612 INFO:teuthology.orchestra.run.smithi001.stderr:bluefs files list 2024-09-13T11:25:02.612 INFO:teuthology.orchestra.run.smithi001.stderr:bluefs stats 2024-09-13T11:25:02.613 INFO:teuthology.orchestra.run.smithi001.stderr:bluestore allocator dump block 2024-09-13T11:25:02.613 INFO:teuthology.orchestra.run.smithi001.stderr:Error EINVAL: invalid command 2024-09-13T11:25:02.614 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-13T11:25:02.615 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-13T11:25:02.615 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 'trim stale osdmaps' 2024-09-13T11:25:02.639 INFO:tasks.ceph.osd.15.smithi177.stderr:2024-09-13T11:25:02.634+0000 7f9d6ce77640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26578) UID: 0 2024-09-13T11:25:02.691 INFO:teuthology.orchestra.run.smithi092.stdout:ERROR: (22) Invalid argument 2024-09-13T11:25:02.691 INFO:teuthology.orchestra.run.smithi092.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:25:02.709 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-13T11:25:02.758 INFO:tasks.ceph.osd.5.smithi092.stderr:2024-09-13T11:25:02.754+0000 7f6f0b705640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26346) UID: 0 2024-09-13T11:25:02.797 INFO:teuthology.orchestra.run.smithi001.stderr:no valid command found; 10 closest matches: 2024-09-13T11:25:02.797 INFO:teuthology.orchestra.run.smithi001.stderr:0 2024-09-13T11:25:02.797 INFO:teuthology.orchestra.run.smithi001.stderr:1 2024-09-13T11:25:02.797 INFO:teuthology.orchestra.run.smithi001.stderr:2 2024-09-13T11:25:02.797 INFO:teuthology.orchestra.run.smithi001.stderr:abort 2024-09-13T11:25:02.797 INFO:teuthology.orchestra.run.smithi001.stderr:assert 2024-09-13T11:25:02.797 INFO:teuthology.orchestra.run.smithi001.stderr:bench [] [] [] [] 2024-09-13T11:25:02.797 INFO:teuthology.orchestra.run.smithi001.stderr:bluefs debug_inject_read_zeros 2024-09-13T11:25:02.798 INFO:teuthology.orchestra.run.smithi001.stderr:bluefs files list 2024-09-13T11:25:02.798 INFO:teuthology.orchestra.run.smithi001.stderr:bluefs stats 2024-09-13T11:25:02.798 INFO:teuthology.orchestra.run.smithi001.stderr:bluestore allocator dump block 2024-09-13T11:25:02.798 INFO:teuthology.orchestra.run.smithi001.stderr:Error EINVAL: invalid command 2024-09-13T11:25:02.800 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-13T11:25:02.800 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-13T11:25:02.800 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 'trim stale osdmaps' 2024-09-13T11:25:02.841 INFO:tasks.ceph.osd.14.smithi162.stderr:2024-09-13T11:25:02.840+0000 7f3df5903640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33154) UID: 0 2024-09-13T11:25:02.849 INFO:teuthology.orchestra.run.smithi092.stdout:ERROR: (22) Invalid argument 2024-09-13T11:25:02.849 INFO:teuthology.orchestra.run.smithi092.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:25:02.865 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_ops_in_flight 2024-09-13T11:25:02.943 INFO:tasks.ceph.osd.12.smithi001.stderr:2024-09-13T11:25:02.939+0000 7f892c6e6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26314) UID: 0 2024-09-13T11:25:02.954 INFO:teuthology.orchestra.run.smithi001.stderr:no valid command found; 10 closest matches: 2024-09-13T11:25:02.954 INFO:teuthology.orchestra.run.smithi001.stderr:0 2024-09-13T11:25:02.955 INFO:teuthology.orchestra.run.smithi001.stderr:1 2024-09-13T11:25:02.955 INFO:teuthology.orchestra.run.smithi001.stderr:2 2024-09-13T11:25:02.955 INFO:teuthology.orchestra.run.smithi001.stderr:abort 2024-09-13T11:25:02.955 INFO:teuthology.orchestra.run.smithi001.stderr:assert 2024-09-13T11:25:02.955 INFO:teuthology.orchestra.run.smithi001.stderr:bench [] [] [] [] 2024-09-13T11:25:02.955 INFO:teuthology.orchestra.run.smithi001.stderr:bluefs debug_inject_read_zeros 2024-09-13T11:25:02.955 INFO:teuthology.orchestra.run.smithi001.stderr:bluefs files list 2024-09-13T11:25:02.955 INFO:teuthology.orchestra.run.smithi001.stderr:bluefs stats 2024-09-13T11:25:02.955 INFO:teuthology.orchestra.run.smithi001.stderr:bluestore allocator dump block 2024-09-13T11:25:02.955 INFO:teuthology.orchestra.run.smithi001.stderr:Error EINVAL: invalid command 2024-09-13T11:25:02.957 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-13T11:25:02.957 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-13T11:25:02.957 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.8 'trim stale osdmaps' 2024-09-13T11:25:03.004 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-13T11:25:03.005 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:25:03.021 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_blocked_ops 2024-09-13T11:25:03.048 INFO:tasks.ceph.osd.8.smithi001.stderr:2024-09-13T11:25:03.047+0000 7fb9b48ae640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26312) UID: 0 2024-09-13T11:25:03.145 INFO:tasks.ceph.osd.8.smithi001.stderr:2024-09-13T11:25:03.143+0000 7fb9b48ae640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26312) UID: 0 2024-09-13T11:25:03.167 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-13T11:25:03.167 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:25:03.172 INFO:teuthology.orchestra.run.smithi001.stderr:no valid command found; 10 closest matches: 2024-09-13T11:25:03.173 INFO:teuthology.orchestra.run.smithi001.stderr:0 2024-09-13T11:25:03.173 INFO:teuthology.orchestra.run.smithi001.stderr:1 2024-09-13T11:25:03.173 INFO:teuthology.orchestra.run.smithi001.stderr:2 2024-09-13T11:25:03.173 INFO:teuthology.orchestra.run.smithi001.stderr:abort 2024-09-13T11:25:03.173 INFO:teuthology.orchestra.run.smithi001.stderr:assert 2024-09-13T11:25:03.173 INFO:teuthology.orchestra.run.smithi001.stderr:bench [] [] [] [] 2024-09-13T11:25:03.173 INFO:teuthology.orchestra.run.smithi001.stderr:bluefs debug_inject_read_zeros 2024-09-13T11:25:03.173 INFO:teuthology.orchestra.run.smithi001.stderr:bluefs files list 2024-09-13T11:25:03.173 INFO:teuthology.orchestra.run.smithi001.stderr:bluefs stats 2024-09-13T11:25:03.173 INFO:teuthology.orchestra.run.smithi001.stderr:bluestore allocator dump block 2024-09-13T11:25:03.173 INFO:teuthology.orchestra.run.smithi001.stderr:Error EINVAL: invalid command 2024-09-13T11:25:03.176 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-13T11:25:03.177 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-13T11:25:03.177 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.9 'trim stale osdmaps' 2024-09-13T11:25:03.186 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_historic_ops 2024-09-13T11:25:03.246 INFO:tasks.ceph.osd.11.smithi177.stderr:2024-09-13T11:25:03.242+0000 7f24792c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26580) UID: 0 2024-09-13T11:25:03.324 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-13T11:25:03.324 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:25:03.341 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-13T11:25:03.392 INFO:tasks.ceph.osd.8.smithi001.stderr:2024-09-13T11:25:03.391+0000 7fb9b48ae640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26312) UID: 0 2024-09-13T11:25:03.394 INFO:teuthology.orchestra.run.smithi001.stderr:no valid command found; 10 closest matches: 2024-09-13T11:25:03.394 INFO:teuthology.orchestra.run.smithi001.stderr:0 2024-09-13T11:25:03.394 INFO:teuthology.orchestra.run.smithi001.stderr:1 2024-09-13T11:25:03.394 INFO:teuthology.orchestra.run.smithi001.stderr:2 2024-09-13T11:25:03.394 INFO:teuthology.orchestra.run.smithi001.stderr:abort 2024-09-13T11:25:03.394 INFO:teuthology.orchestra.run.smithi001.stderr:assert 2024-09-13T11:25:03.394 INFO:teuthology.orchestra.run.smithi001.stderr:bench [] [] [] [] 2024-09-13T11:25:03.395 INFO:teuthology.orchestra.run.smithi001.stderr:bluefs debug_inject_read_zeros 2024-09-13T11:25:03.395 INFO:teuthology.orchestra.run.smithi001.stderr:bluefs files list 2024-09-13T11:25:03.395 INFO:teuthology.orchestra.run.smithi001.stderr:bluefs stats 2024-09-13T11:25:03.395 INFO:teuthology.orchestra.run.smithi001.stderr:bluestore allocator dump block 2024-09-13T11:25:03.395 INFO:teuthology.orchestra.run.smithi001.stderr:Error EINVAL: invalid command 2024-09-13T11:25:03.397 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-13T11:25:03.397 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-13T11:25:03.397 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.10 'trim stale osdmaps' 2024-09-13T11:25:03.448 INFO:tasks.ceph.osd.11.smithi177.stderr:2024-09-13T11:25:03.446+0000 7f24792c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26580) UID: 0 2024-09-13T11:25:03.470 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-13T11:25:03.471 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:25:03.488 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-13T11:25:03.550 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-09-13T11:25:03.548+0000 7f97befc9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26315) UID: 0 2024-09-13T11:25:03.613 INFO:teuthology.orchestra.run.smithi001.stderr:no valid command found; 10 closest matches: 2024-09-13T11:25:03.613 INFO:teuthology.orchestra.run.smithi001.stderr:0 2024-09-13T11:25:03.613 INFO:teuthology.orchestra.run.smithi001.stderr:1 2024-09-13T11:25:03.613 INFO:teuthology.orchestra.run.smithi001.stderr:2 2024-09-13T11:25:03.613 INFO:teuthology.orchestra.run.smithi001.stderr:abort 2024-09-13T11:25:03.613 INFO:teuthology.orchestra.run.smithi001.stderr:assert 2024-09-13T11:25:03.613 INFO:teuthology.orchestra.run.smithi001.stderr:bench [] [] [] [] 2024-09-13T11:25:03.613 INFO:teuthology.orchestra.run.smithi001.stderr:bluefs debug_inject_read_zeros 2024-09-13T11:25:03.613 INFO:teuthology.orchestra.run.smithi001.stderr:bluefs files list 2024-09-13T11:25:03.613 INFO:teuthology.orchestra.run.smithi001.stderr:bluefs stats 2024-09-13T11:25:03.613 INFO:teuthology.orchestra.run.smithi001.stderr:bluestore allocator dump block 2024-09-13T11:25:03.614 INFO:teuthology.orchestra.run.smithi001.stderr:Error EINVAL: invalid command 2024-09-13T11:25:03.615 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-13T11:25:03.616 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-13T11:25:03.616 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.11 'trim stale osdmaps' 2024-09-13T11:25:03.625 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T11:25:03.650 INFO:tasks.ceph.osd.15.smithi177.stderr:2024-09-13T11:25:03.646+0000 7f9d6ce77640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26578) UID: 0 2024-09-13T11:25:03.662 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-13T11:25:03.662 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:25:03.690 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-13T11:25:03.752 INFO:tasks.ceph.osd.2.smithi162.stderr:2024-09-13T11:25:03.748+0000 7fe92b35a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26313) UID: 0 2024-09-13T11:25:03.812 INFO:teuthology.orchestra.run.smithi001.stderr:no valid command found; 10 closest matches: 2024-09-13T11:25:03.812 INFO:teuthology.orchestra.run.smithi001.stderr:0 2024-09-13T11:25:03.812 INFO:teuthology.orchestra.run.smithi001.stderr:1 2024-09-13T11:25:03.812 INFO:teuthology.orchestra.run.smithi001.stderr:2 2024-09-13T11:25:03.812 INFO:teuthology.orchestra.run.smithi001.stderr:abort 2024-09-13T11:25:03.812 INFO:teuthology.orchestra.run.smithi001.stderr:assert 2024-09-13T11:25:03.813 INFO:teuthology.orchestra.run.smithi001.stderr:bench [] [] [] [] 2024-09-13T11:25:03.813 INFO:teuthology.orchestra.run.smithi001.stderr:bluefs debug_inject_read_zeros 2024-09-13T11:25:03.813 INFO:teuthology.orchestra.run.smithi001.stderr:bluefs files list 2024-09-13T11:25:03.813 INFO:teuthology.orchestra.run.smithi001.stderr:bluefs stats 2024-09-13T11:25:03.817 INFO:teuthology.orchestra.run.smithi001.stderr:bluestore allocator dump block 2024-09-13T11:25:03.817 INFO:teuthology.orchestra.run.smithi001.stderr:Error EINVAL: invalid command 2024-09-13T11:25:03.818 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-13T11:25:03.818 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-13T11:25:03.819 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.12 'trim stale osdmaps' 2024-09-13T11:25:03.846 INFO:teuthology.orchestra.run.smithi001.stdout:osd.0: {} 2024-09-13T11:25:03.846 INFO:teuthology.orchestra.run.smithi001.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T11:25:03.853 INFO:tasks.ceph.osd.1.smithi092.stderr:2024-09-13T11:25:03.846+0000 7fe7789c5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33718) UID: 0 2024-09-13T11:25:03.870 INFO:teuthology.orchestra.run.smithi001.stdout:osd.1: {} 2024-09-13T11:25:03.871 INFO:teuthology.orchestra.run.smithi001.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T11:25:03.874 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-13T11:25:03.874 INFO:teuthology.orchestra.run.smithi001.stdout: "size": 20, 2024-09-13T11:25:03.874 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 600, 2024-09-13T11:25:03.874 INFO:teuthology.orchestra.run.smithi001.stdout: "ops": [ 2024-09-13T11:25:03.874 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:03.875 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDPGPushReply(3.1e 84/78 [PushReplyOp(3:79ee54b9:::benchmark_data_smithi177_31116_object11701:head)])", 2024-09-13T11:25:03.875 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:41.407335+0000", 2024-09-13T11:25:03.875 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 82.465098201999993, 2024-09-13T11:25:03.875 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.51622350400000006, 2024-09-13T11:25:03.875 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:03.875 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:03.875 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-13T11:25:03.875 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:03.875 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:03.875 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:03.875 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:41.407335+0000", 2024-09-13T11:25:03.875 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:03.875 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:03.876 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:03.876 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:03.876 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:41.407335+0000", 2024-09-13T11:25:03.876 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:03.876 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:03.876 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:03.876 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:03.876 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:41.407337+0000", 2024-09-13T11:25:03.876 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.505e-06 2024-09-13T11:25:03.876 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:03.876 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:03.876 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:03.876 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:41.407343+0000", 2024-09-13T11:25:03.877 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.9089999999999997e-06 2024-09-13T11:25:03.877 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:03.877 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:03.877 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:03.877 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:41.921977+0000", 2024-09-13T11:25:03.877 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.51463463499999995 2024-09-13T11:25:03.877 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:03.877 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:03.877 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:03.877 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:41.921988+0000", 2024-09-13T11:25:03.877 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.092e-05 2024-09-13T11:25:03.877 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:03.877 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:03.877 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:03.878 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:41.922088+0000", 2024-09-13T11:25:03.878 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.9528e-05 2024-09-13T11:25:03.878 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:03.878 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:03.878 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:03.878 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:41.923559+0000", 2024-09-13T11:25:03.878 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0014710070000000001 2024-09-13T11:25:03.878 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:03.878 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:03.878 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:03.878 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:03.878 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:03.878 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4665.0:60417 3.10 e85/84 3:092808b0:::benchmark_data_smithi177_31116_object60416:head v 85'3295, mlcod=85'3292)", 2024-09-13T11:25:03.878 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:42.174488+0000", 2024-09-13T11:25:03.879 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 81.697945648000001, 2024-09-13T11:25:03.879 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.387908896, 2024-09-13T11:25:03.879 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:03.879 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:03.879 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:03.879 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:03.879 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:03.879 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:03.879 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.174488+0000", 2024-09-13T11:25:03.879 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:03.879 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:03.879 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:03.879 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:03.880 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.174488+0000", 2024-09-13T11:25:03.880 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:03.880 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:03.880 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:03.880 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:03.880 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.174490+0000", 2024-09-13T11:25:03.880 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.9099999999999999e-06 2024-09-13T11:25:03.880 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:03.880 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:03.880 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:03.880 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.174527+0000", 2024-09-13T11:25:03.880 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.7543000000000003e-05 2024-09-13T11:25:03.880 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:03.881 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:03.881 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:03.881 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.561093+0000", 2024-09-13T11:25:03.881 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.38656578400000002 2024-09-13T11:25:03.881 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:03.881 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:03.881 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:03.881 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.561101+0000", 2024-09-13T11:25:03.881 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.6090000000000003e-06 2024-09-13T11:25:03.881 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:03.881 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:03.881 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:03.881 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.561143+0000", 2024-09-13T11:25:03.882 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.2435000000000003e-05 2024-09-13T11:25:03.882 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:03.882 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:03.882 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:25:03.882 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.561164+0000", 2024-09-13T11:25:03.882 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.1276000000000001e-05 2024-09-13T11:25:03.882 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:03.882 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:03.882 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:25:03.882 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.562360+0000", 2024-09-13T11:25:03.882 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00119596 2024-09-13T11:25:03.882 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:03.882 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:03.883 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:03.883 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.562397+0000", 2024-09-13T11:25:03.883 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.6378999999999998e-05 2024-09-13T11:25:03.883 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:03.883 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:03.883 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:03.883 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:03.883 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:03.883 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDPGPush(3.22 85/78 [PushOp(3:456e8765:::benchmark_data_smithi177_31116_object16212:head, version: 52'1006, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:456e8765:::benchmark_data_smithi177_31116_object16212:head@52'1006, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:25:03.883 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:42.673067+0000", 2024-09-13T11:25:03.883 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 81.199366506000004, 2024-09-13T11:25:03.883 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.27935889899999999, 2024-09-13T11:25:03.883 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:03.883 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:03.884 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:25:03.884 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:03.884 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:03.884 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:03.884 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.673067+0000", 2024-09-13T11:25:03.884 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:03.884 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:03.884 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:03.884 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:03.884 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.673067+0000", 2024-09-13T11:25:03.884 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:03.884 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:03.884 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:03.884 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:03.885 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.673068+0000", 2024-09-13T11:25:03.885 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.9699999999999994e-07 2024-09-13T11:25:03.885 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:03.885 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:03.885 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:03.885 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.673090+0000", 2024-09-13T11:25:03.885 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.2416999999999999e-05 2024-09-13T11:25:03.885 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:03.885 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:03.885 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:03.885 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.951363+0000", 2024-09-13T11:25:03.885 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.27827291199999998 2024-09-13T11:25:03.886 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:03.886 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:03.886 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:03.886 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.951386+0000", 2024-09-13T11:25:03.886 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.2226999999999999e-05 2024-09-13T11:25:03.886 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:03.886 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:03.886 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:03.886 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.951581+0000", 2024-09-13T11:25:03.886 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000195713 2024-09-13T11:25:03.886 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:03.886 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:03.886 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:25:03.886 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.951605+0000", 2024-09-13T11:25:03.887 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.3767000000000001e-05 2024-09-13T11:25:03.887 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:03.887 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:03.887 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:03.887 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.952426+0000", 2024-09-13T11:25:03.887 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00082086599999999996 2024-09-13T11:25:03.887 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:03.887 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:03.887 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:03.887 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:03.887 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:03.887 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "replica_scrub(pg: 3.20,from:0'0,to:89'3254,epoch:89/77,start:3:04000000::::0,end:3:045ba805:::benchmark_data_smithi177_31116_object43251:0,chunky:1,deep:0,version:9,allow_preemption:1,priority=5)", 2024-09-13T11:25:03.887 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:46.641364+0000", 2024-09-13T11:25:03.887 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 77.231069825000006, 2024-09-13T11:25:03.888 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.27141393000000003, 2024-09-13T11:25:03.888 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:03.888 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:03.888 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:25:03.888 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:03.888 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:03.888 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:03.888 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.641364+0000", 2024-09-13T11:25:03.888 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:03.888 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:03.888 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:03.888 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:03.888 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.641364+0000", 2024-09-13T11:25:03.889 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:03.889 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:03.889 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:03.889 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:03.889 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.641365+0000", 2024-09-13T11:25:03.889 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.266e-06 2024-09-13T11:25:03.889 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:03.889 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:03.889 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:03.889 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.641370+0000", 2024-09-13T11:25:03.889 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.8539999999999996e-06 2024-09-13T11:25:03.890 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:03.890 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:03.890 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:03.890 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.912491+0000", 2024-09-13T11:25:03.890 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.27112103100000001 2024-09-13T11:25:03.890 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:03.890 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:03.890 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:03.890 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.912520+0000", 2024-09-13T11:25:03.890 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.9379000000000001e-05 2024-09-13T11:25:03.890 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:03.890 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:03.890 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:03.890 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.912679+0000", 2024-09-13T11:25:03.891 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00015915800000000001 2024-09-13T11:25:03.891 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:03.891 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:03.891 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:25:03.891 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.912714+0000", 2024-09-13T11:25:03.891 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.4687999999999997e-05 2024-09-13T11:25:03.891 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:03.891 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:03.891 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:03.891 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.912778+0000", 2024-09-13T11:25:03.891 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.3553999999999995e-05 2024-09-13T11:25:03.891 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:03.891 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:03.891 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:03.892 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:03.892 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:03.892 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDPGPush(3.18 89/70 [PushOp(3:1ac6c361:::benchmark_data_smithi177_31116_object44972:head, version: 68'2767, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:1ac6c361:::benchmark_data_smithi177_31116_object44972:head@68'2767, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:25:03.892 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:46.671018+0000", 2024-09-13T11:25:03.892 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 77.201415693000001, 2024-09-13T11:25:03.892 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.271282931, 2024-09-13T11:25:03.892 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:03.892 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:03.892 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:25:03.892 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:03.892 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:03.892 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:03.892 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.671018+0000", 2024-09-13T11:25:03.893 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:03.893 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:03.893 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:03.893 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:03.893 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.671018+0000", 2024-09-13T11:25:03.893 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:03.893 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:03.893 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:03.893 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:03.893 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.671020+0000", 2024-09-13T11:25:03.893 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.018e-06 2024-09-13T11:25:03.893 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:03.893 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:03.893 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:03.894 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.671097+0000", 2024-09-13T11:25:03.894 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.7218999999999994e-05 2024-09-13T11:25:03.894 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:03.894 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:03.894 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:03.894 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.939773+0000", 2024-09-13T11:25:03.894 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.26867550200000001 2024-09-13T11:25:03.894 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:03.894 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:03.894 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:03.894 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.939791+0000", 2024-09-13T11:25:03.894 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.8185999999999998e-05 2024-09-13T11:25:03.894 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:03.894 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:03.894 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:03.895 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.939926+0000", 2024-09-13T11:25:03.895 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000135291 2024-09-13T11:25:03.895 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:03.895 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:03.895 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:25:03.895 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.939938+0000", 2024-09-13T11:25:03.895 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2393e-05 2024-09-13T11:25:03.895 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:03.895 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:03.895 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:03.895 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.942301+0000", 2024-09-13T11:25:03.895 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.002362322 2024-09-13T11:25:03.895 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:03.895 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:03.896 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:03.896 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:03.896 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:03.896 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "replica_scrub(pg: 3.20,from:0'0,to:90'3264,epoch:90/77,start:3:0650aac8:::benchmark_data_smithi177_31116_object1734:0,end:3:06af01e3:::benchmark_data_smithi177_31116_object31572:0,chunky:1,deep:0,version:9,allow_preemption:1,priority=5)", 2024-09-13T11:25:03.896 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:47.380615+0000", 2024-09-13T11:25:03.896 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 76.491818269999996, 2024-09-13T11:25:03.896 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.51951398400000004, 2024-09-13T11:25:03.896 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:03.896 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:03.896 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:25:03.896 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:03.896 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:03.896 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:03.896 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:47.380615+0000", 2024-09-13T11:25:03.897 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:03.897 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:03.897 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:03.897 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:03.897 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:47.380615+0000", 2024-09-13T11:25:03.897 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:03.897 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:03.897 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:03.897 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:03.897 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:47.380617+0000", 2024-09-13T11:25:03.897 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.671e-06 2024-09-13T11:25:03.897 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:03.897 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:03.897 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:03.898 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:47.380623+0000", 2024-09-13T11:25:03.898 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.1190000000000002e-06 2024-09-13T11:25:03.898 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:03.898 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:03.898 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:03.898 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:47.900050+0000", 2024-09-13T11:25:03.898 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.51942737400000005 2024-09-13T11:25:03.898 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:03.898 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:03.898 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:03.898 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:47.900057+0000", 2024-09-13T11:25:03.898 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.4359999999999999e-06 2024-09-13T11:25:03.898 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:03.898 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:03.899 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:03.899 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:47.900101+0000", 2024-09-13T11:25:03.899 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.4675000000000002e-05 2024-09-13T11:25:03.899 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:03.899 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:03.899 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:25:03.899 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:47.900110+0000", 2024-09-13T11:25:03.899 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.0569999999999993e-06 2024-09-13T11:25:03.899 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:03.899 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:03.899 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:03.899 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:47.900129+0000", 2024-09-13T11:25:03.899 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.9652000000000001e-05 2024-09-13T11:25:03.899 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:03.900 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:03.900 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:03.900 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:03.900 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:03.900 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDPGPush(3.21 91/78 [PushOp(3:8492321f:::benchmark_data_smithi177_31116_object13767:head, version: 50'840, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:8492321f:::benchmark_data_smithi177_31116_object13767:head@50'840, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:25:03.900 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:48.319251+0000", 2024-09-13T11:25:03.900 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 75.553182741000001, 2024-09-13T11:25:03.900 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.60903802200000001, 2024-09-13T11:25:03.900 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:03.900 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:03.900 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:25:03.900 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:03.900 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:03.901 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:03.901 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:48.319251+0000", 2024-09-13T11:25:03.901 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:03.901 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:03.901 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:03.901 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:03.901 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:48.319251+0000", 2024-09-13T11:25:03.901 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:03.901 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:03.901 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:03.901 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:03.901 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:48.319252+0000", 2024-09-13T11:25:03.901 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2640000000000001e-06 2024-09-13T11:25:03.901 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:03.902 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:03.902 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:03.902 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:48.319363+0000", 2024-09-13T11:25:03.902 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000110559 2024-09-13T11:25:03.902 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:03.902 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:03.902 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:03.902 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:48.927115+0000", 2024-09-13T11:25:03.902 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.60775277800000005 2024-09-13T11:25:03.902 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:03.902 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:03.902 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:03.902 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:48.927144+0000", 2024-09-13T11:25:03.903 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.8861e-05 2024-09-13T11:25:03.903 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:03.903 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:03.903 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:03.903 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:48.927318+0000", 2024-09-13T11:25:03.903 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00017399000000000001 2024-09-13T11:25:03.903 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:03.903 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:03.903 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:25:03.903 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:48.927340+0000", 2024-09-13T11:25:03.903 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.1433999999999998e-05 2024-09-13T11:25:03.903 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:03.903 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:03.903 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:03.903 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:48.928289+0000", 2024-09-13T11:25:03.904 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00094913599999999995 2024-09-13T11:25:03.904 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:03.904 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:03.904 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:03.904 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:03.904 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:03.904 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:69487 3.e 3:73f7f990:::benchmark_data_smithi177_31116_object69486:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94)", 2024-09-13T11:25:03.904 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:52.063559+0000", 2024-09-13T11:25:03.904 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 71.808874708000005, 2024-09-13T11:25:03.904 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.20206200199999999, 2024-09-13T11:25:03.904 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:03.904 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:03.904 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:03.904 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:25:03.905 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:25:03.905 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:03.905 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 69487 2024-09-13T11:25:03.905 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:03.905 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:03.905 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:03.905 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:03.905 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:52.063559+0000", 2024-09-13T11:25:03.905 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:03.905 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:03.905 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:03.905 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:03.905 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:52.063559+0000", 2024-09-13T11:25:03.906 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:03.906 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:03.906 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:03.906 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:03.906 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:52.063560+0000", 2024-09-13T11:25:03.906 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.167e-06 2024-09-13T11:25:03.906 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:03.906 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:03.906 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:03.906 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:52.063612+0000", 2024-09-13T11:25:03.906 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.1709999999999998e-05 2024-09-13T11:25:03.906 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:03.906 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:03.906 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:03.907 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:52.063623+0000", 2024-09-13T11:25:03.907 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.1251e-05 2024-09-13T11:25:03.907 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:03.907 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:03.907 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:03.907 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:52.063631+0000", 2024-09-13T11:25:03.907 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.7319999999999998e-06 2024-09-13T11:25:03.907 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:03.907 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:03.907 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:03.907 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:52.064267+0000", 2024-09-13T11:25:03.907 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00063608600000000001 2024-09-13T11:25:03.907 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:03.908 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:03.908 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:25:03.908 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:52.064407+0000", 2024-09-13T11:25:03.908 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000140686 2024-09-13T11:25:03.908 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:03.908 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:03.908 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 1", 2024-09-13T11:25:03.908 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:52.064540+0000", 2024-09-13T11:25:03.908 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000132244 2024-09-13T11:25:03.908 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:03.908 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:03.908 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:25:03.908 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:52.067019+0000", 2024-09-13T11:25:03.908 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00247919 2024-09-13T11:25:03.909 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:03.909 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:03.909 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:25:03.909 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:52.265543+0000", 2024-09-13T11:25:03.909 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.19852367800000001 2024-09-13T11:25:03.909 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:03.909 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:03.909 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:25:03.909 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:52.265576+0000", 2024-09-13T11:25:03.909 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.3550999999999998e-05 2024-09-13T11:25:03.909 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:03.909 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:03.909 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:03.909 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:52.265621+0000", 2024-09-13T11:25:03.909 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.4706999999999997e-05 2024-09-13T11:25:03.910 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:03.910 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:03.910 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:03.910 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:03.910 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:03.910 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4665.0:69556 3.18 e94/91 3:1d3dbeba:::benchmark_data_smithi177_31116_object69555:head v 94'3570, mlcod=94'3566)", 2024-09-13T11:25:03.910 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:52.142942+0000", 2024-09-13T11:25:03.910 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 71.729491206000006, 2024-09-13T11:25:03.910 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.21178485399999999, 2024-09-13T11:25:03.910 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:03.910 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:03.910 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:03.910 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:03.911 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:03.911 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:03.911 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:52.142942+0000", 2024-09-13T11:25:03.911 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:03.911 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:03.911 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:03.911 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:03.911 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:52.142942+0000", 2024-09-13T11:25:03.911 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:03.911 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:03.911 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:03.911 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:03.911 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:52.142944+0000", 2024-09-13T11:25:03.912 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.4500000000000001e-06 2024-09-13T11:25:03.912 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:03.912 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:03.912 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:03.912 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:52.143047+0000", 2024-09-13T11:25:03.912 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000103384 2024-09-13T11:25:03.912 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:03.912 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:03.912 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:03.912 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:52.353322+0000", 2024-09-13T11:25:03.912 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.21027483299999999 2024-09-13T11:25:03.912 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:03.912 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:03.912 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:03.913 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:52.353330+0000", 2024-09-13T11:25:03.913 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.8879999999999997e-06 2024-09-13T11:25:03.913 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:03.913 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:03.913 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:03.913 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:52.353402+0000", 2024-09-13T11:25:03.913 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.2155000000000003e-05 2024-09-13T11:25:03.913 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:03.913 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:03.913 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:25:03.913 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:52.353426+0000", 2024-09-13T11:25:03.913 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.3774e-05 2024-09-13T11:25:03.913 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:03.913 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:03.913 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:25:03.914 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:52.354674+0000", 2024-09-13T11:25:03.914 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.001248706 2024-09-13T11:25:03.914 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:03.914 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:03.914 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:03.914 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:52.354727+0000", 2024-09-13T11:25:03.914 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.2664000000000003e-05 2024-09-13T11:25:03.914 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:03.914 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:03.914 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:03.914 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:03.914 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:03.914 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:70749 3.1e 3:7bd7bfc6:::benchmark_data_smithi177_31116_object70748:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95)", 2024-09-13T11:25:03.914 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:53.156250+0000", 2024-09-13T11:25:03.915 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 70.716183384999994, 2024-09-13T11:25:03.915 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.22394481799999999, 2024-09-13T11:25:03.915 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:03.915 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:03.915 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:03.915 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:25:03.915 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:25:03.915 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:03.915 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 70749 2024-09-13T11:25:03.915 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:03.915 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:03.915 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:03.915 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:03.915 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:53.156250+0000", 2024-09-13T11:25:03.916 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:03.916 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:03.916 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:03.916 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:03.916 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:53.156250+0000", 2024-09-13T11:25:03.916 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:03.916 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:03.916 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:03.916 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:03.916 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:53.156252+0000", 2024-09-13T11:25:03.916 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.3460000000000001e-06 2024-09-13T11:25:03.916 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:03.916 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:03.917 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:03.917 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:53.156336+0000", 2024-09-13T11:25:03.917 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.3714999999999994e-05 2024-09-13T11:25:03.917 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:03.917 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:03.917 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:03.917 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:53.156347+0000", 2024-09-13T11:25:03.917 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0903999999999999e-05 2024-09-13T11:25:03.917 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:03.917 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:03.917 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:03.917 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:53.156352+0000", 2024-09-13T11:25:03.917 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.9139999999999999e-06 2024-09-13T11:25:03.918 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:03.918 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:03.918 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:03.918 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:53.158028+0000", 2024-09-13T11:25:03.918 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.001675958 2024-09-13T11:25:03.918 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:03.918 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:03.918 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:25:03.918 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:53.158154+0000", 2024-09-13T11:25:03.918 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00012653 2024-09-13T11:25:03.918 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:03.918 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:03.918 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 4,5", 2024-09-13T11:25:03.918 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:53.158277+0000", 2024-09-13T11:25:03.919 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000122168 2024-09-13T11:25:03.919 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:03.919 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:03.919 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:25:03.919 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:53.159585+0000", 2024-09-13T11:25:03.919 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.001308528 2024-09-13T11:25:03.919 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:03.919 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:03.919 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:25:03.919 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:53.163172+0000", 2024-09-13T11:25:03.919 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0035869690000000002 2024-09-13T11:25:03.919 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:03.919 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:03.919 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:25:03.920 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:53.379730+0000", 2024-09-13T11:25:03.920 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.21655822599999999 2024-09-13T11:25:03.920 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:03.920 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:03.920 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:25:03.920 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:53.379810+0000", 2024-09-13T11:25:03.920 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.0041000000000002e-05 2024-09-13T11:25:03.920 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:03.920 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:03.920 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:03.920 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:53.380195+0000", 2024-09-13T11:25:03.920 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000384519 2024-09-13T11:25:03.920 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:03.920 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:03.921 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:03.921 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:03.921 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:03.921 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDPGPush(3.21 96/78 [PushOp(3:862d73af:::benchmark_data_smithi177_31116_object32719:head, version: 60'2029, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:862d73af:::benchmark_data_smithi177_31116_object32719:head@60'2029, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:25:03.921 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:53.884628+0000", 2024-09-13T11:25:03.921 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 69.987805542999993, 2024-09-13T11:25:03.921 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.21020203400000001, 2024-09-13T11:25:03.921 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:03.921 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:03.921 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:25:03.921 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:03.921 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:03.921 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:03.921 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:53.884628+0000", 2024-09-13T11:25:03.922 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:03.922 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:03.922 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:03.922 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:03.922 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:53.884628+0000", 2024-09-13T11:25:03.922 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:03.922 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:03.922 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:03.922 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:03.922 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:53.884630+0000", 2024-09-13T11:25:03.922 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.68e-06 2024-09-13T11:25:03.922 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:03.922 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:03.922 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:03.923 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:53.884745+0000", 2024-09-13T11:25:03.923 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000115599 2024-09-13T11:25:03.923 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:03.923 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:03.923 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:03.923 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:54.093948+0000", 2024-09-13T11:25:03.923 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.20920306999999999 2024-09-13T11:25:03.923 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:03.923 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:03.923 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:03.923 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:54.093961+0000", 2024-09-13T11:25:03.923 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2815e-05 2024-09-13T11:25:03.923 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:03.923 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:03.924 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:03.924 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:54.094139+0000", 2024-09-13T11:25:03.924 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00017824 2024-09-13T11:25:03.924 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:03.924 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:03.924 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:25:03.924 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:54.094150+0000", 2024-09-13T11:25:03.924 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0593e-05 2024-09-13T11:25:03.924 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:03.924 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:03.924 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:03.924 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:54.094830+0000", 2024-09-13T11:25:03.924 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00068003699999999998 2024-09-13T11:25:03.924 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:03.925 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:03.925 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:03.925 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:03.925 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:03.925 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:83239 3.e 3:74547f17:::benchmark_data_smithi177_31116_object83238:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106)", 2024-09-13T11:25:03.925 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:24:04.580085+0000", 2024-09-13T11:25:03.925 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 59.292348660000002, 2024-09-13T11:25:03.925 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.33785762600000002, 2024-09-13T11:25:03.925 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:03.925 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:03.925 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:03.925 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:25:03.925 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:25:03.925 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:03.925 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 83239 2024-09-13T11:25:03.926 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:03.926 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:03.926 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:03.926 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:03.926 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:04.580085+0000", 2024-09-13T11:25:03.926 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:03.926 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:03.926 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:03.926 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:03.926 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:04.580085+0000", 2024-09-13T11:25:03.926 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:03.926 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:03.926 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:03.927 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:03.927 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:04.580089+0000", 2024-09-13T11:25:03.927 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.053e-06 2024-09-13T11:25:03.927 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:03.927 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:03.927 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:03.927 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:04.580166+0000", 2024-09-13T11:25:03.927 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.7102999999999997e-05 2024-09-13T11:25:03.927 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:03.927 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:03.927 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:03.927 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:04.580196+0000", 2024-09-13T11:25:03.928 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.0196e-05 2024-09-13T11:25:03.928 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:03.928 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:03.928 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:03.928 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:04.580212+0000", 2024-09-13T11:25:03.928 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.535e-05 2024-09-13T11:25:03.928 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:03.928 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:03.928 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:03.928 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:04.580336+0000", 2024-09-13T11:25:03.928 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00012431899999999999 2024-09-13T11:25:03.928 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:03.928 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:03.928 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:25:03.929 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:04.580575+0000", 2024-09-13T11:25:03.929 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00023924100000000001 2024-09-13T11:25:03.929 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:03.929 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:03.929 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 1", 2024-09-13T11:25:03.929 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:04.580798+0000", 2024-09-13T11:25:03.929 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00022301499999999999 2024-09-13T11:25:03.929 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:03.929 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:03.929 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:25:03.929 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:04.583404+0000", 2024-09-13T11:25:03.929 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0026056120000000002 2024-09-13T11:25:03.929 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:03.929 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:03.930 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:25:03.930 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:04.917768+0000", 2024-09-13T11:25:03.930 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.33436387499999998 2024-09-13T11:25:03.930 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:03.930 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:03.930 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:25:03.930 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:04.917859+0000", 2024-09-13T11:25:03.930 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.1732999999999997e-05 2024-09-13T11:25:03.930 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:03.930 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:03.930 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:03.930 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:04.917942+0000", 2024-09-13T11:25:03.930 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.3128999999999996e-05 2024-09-13T11:25:03.931 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:03.931 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:03.931 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:03.931 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:03.931 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:03.931 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:84571 3.21 3:85e7a789:::benchmark_data_smithi177_31116_object84570:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-09-13T11:25:03.931 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:24:05.695256+0000", 2024-09-13T11:25:03.931 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 58.177177628999999, 2024-09-13T11:25:03.931 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.18917198299999999, 2024-09-13T11:25:03.931 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:03.931 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:03.931 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:03.931 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:25:03.931 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:25:03.932 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:03.932 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 84571 2024-09-13T11:25:03.932 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:03.932 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:03.932 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:03.932 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:03.932 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:05.695256+0000", 2024-09-13T11:25:03.932 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:03.932 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:03.933 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:03.933 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:03.933 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:05.695256+0000", 2024-09-13T11:25:03.933 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:03.933 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:03.933 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:03.933 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:03.933 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:05.695258+0000", 2024-09-13T11:25:03.933 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.0530000000000001e-06 2024-09-13T11:25:03.933 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:03.933 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:03.933 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:03.934 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:05.695304+0000", 2024-09-13T11:25:03.934 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.5769999999999997e-05 2024-09-13T11:25:03.934 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:03.934 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:03.934 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:03.934 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:05.695313+0000", 2024-09-13T11:25:03.934 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.7049999999999994e-06 2024-09-13T11:25:03.934 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:03.934 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:03.934 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:03.934 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:05.695321+0000", 2024-09-13T11:25:03.934 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.7190000000000001e-06 2024-09-13T11:25:03.934 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:03.934 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:03.935 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:03.935 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:05.695404+0000", 2024-09-13T11:25:03.935 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.3188000000000004e-05 2024-09-13T11:25:03.935 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:03.935 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:03.935 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:25:03.935 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:05.695635+0000", 2024-09-13T11:25:03.935 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00023032900000000001 2024-09-13T11:25:03.935 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:03.935 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:03.935 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 8", 2024-09-13T11:25:03.935 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:05.695852+0000", 2024-09-13T11:25:03.935 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00021749599999999999 2024-09-13T11:25:03.935 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:03.936 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:03.936 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:25:03.936 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:05.697208+0000", 2024-09-13T11:25:03.936 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00135633 2024-09-13T11:25:03.936 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:03.936 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:03.936 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:25:03.936 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:05.884337+0000", 2024-09-13T11:25:03.936 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.187128867 2024-09-13T11:25:03.936 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:03.936 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:03.936 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:25:03.936 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:05.884391+0000", 2024-09-13T11:25:03.936 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.3724999999999998e-05 2024-09-13T11:25:03.937 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:03.937 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:03.937 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:03.937 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:05.884428+0000", 2024-09-13T11:25:03.937 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.6801000000000002e-05 2024-09-13T11:25:03.937 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:03.937 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:03.937 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:03.937 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:03.937 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:03.937 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4665.0:104290 3.20 e115/77 3:05d45b57:::benchmark_data_smithi177_31116_object104289:head v 115'3905, mlcod=115'3902)", 2024-09-13T11:25:03.937 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:24:20.200225+0000", 2024-09-13T11:25:03.937 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 43.672208363999999, 2024-09-13T11:25:03.937 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.21051286899999999, 2024-09-13T11:25:03.938 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:03.938 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:03.938 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:03.938 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:03.938 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:03.938 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:03.938 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:20.200225+0000", 2024-09-13T11:25:03.938 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:03.938 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:03.938 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:03.938 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:03.938 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:20.200225+0000", 2024-09-13T11:25:03.938 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:03.939 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:03.939 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:03.939 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:03.939 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:20.200227+0000", 2024-09-13T11:25:03.939 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.897e-06 2024-09-13T11:25:03.939 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:03.939 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:03.939 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:03.939 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:20.200253+0000", 2024-09-13T11:25:03.939 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.5904000000000001e-05 2024-09-13T11:25:03.939 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:03.939 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:03.939 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:03.939 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:20.409353+0000", 2024-09-13T11:25:03.939 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.209100339 2024-09-13T11:25:03.940 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:03.940 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:03.940 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:03.940 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:20.409359+0000", 2024-09-13T11:25:03.940 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.1689999999999997e-06 2024-09-13T11:25:03.940 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:03.940 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:03.940 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:03.940 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:20.409418+0000", 2024-09-13T11:25:03.940 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.8761999999999997e-05 2024-09-13T11:25:03.940 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:03.940 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:03.940 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:25:03.940 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:20.409440+0000", 2024-09-13T11:25:03.941 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.2098000000000001e-05 2024-09-13T11:25:03.941 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:03.941 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:03.941 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:25:03.941 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:20.410705+0000", 2024-09-13T11:25:03.941 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00126483 2024-09-13T11:25:03.941 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:03.941 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:03.941 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:03.941 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:20.410738+0000", 2024-09-13T11:25:03.941 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.2870000000000002e-05 2024-09-13T11:25:03.941 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:03.941 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:03.942 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:03.942 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:03.942 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:03.942 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:110271 3.2d 3.3c2681ad (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e119)", 2024-09-13T11:25:03.942 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:24:25.403006+0000", 2024-09-13T11:25:03.942 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 38.469427176000003, 2024-09-13T11:25:03.942 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.68768755999999998, 2024-09-13T11:25:03.942 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:03.942 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:03.942 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-13T11:25:03.942 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:25:03.942 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:25:03.942 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:03.942 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 110271 2024-09-13T11:25:03.942 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:03.943 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:03.943 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:03.943 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:03.943 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:25.403006+0000", 2024-09-13T11:25:03.943 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:03.943 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:03.943 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:03.943 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:03.943 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:25.403006+0000", 2024-09-13T11:25:03.943 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:03.943 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:03.943 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:03.943 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:03.944 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:25.403011+0000", 2024-09-13T11:25:03.944 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.5199999999999999e-06 2024-09-13T11:25:03.944 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:03.944 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:03.944 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:03.944 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:25.403065+0000", 2024-09-13T11:25:03.944 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.4478999999999998e-05 2024-09-13T11:25:03.944 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:03.944 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:03.944 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:03.944 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:25.403097+0000", 2024-09-13T11:25:03.944 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.1636e-05 2024-09-13T11:25:03.944 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:03.944 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:03.945 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:03.945 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:25.403121+0000", 2024-09-13T11:25:03.945 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.3924e-05 2024-09-13T11:25:03.945 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:03.945 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:03.945 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:03.945 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:25.403216+0000", 2024-09-13T11:25:03.945 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.5557999999999995e-05 2024-09-13T11:25:03.945 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:03.945 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:03.945 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for peered", 2024-09-13T11:25:03.945 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:25.403234+0000", 2024-09-13T11:25:03.945 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.7104000000000001e-05 2024-09-13T11:25:03.946 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:03.946 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:03.946 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:03.946 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:26.090684+0000", 2024-09-13T11:25:03.946 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.68745000700000003 2024-09-13T11:25:03.946 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:03.946 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:03.946 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:03.946 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:26.090694+0000", 2024-09-13T11:25:03.946 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0332e-05 2024-09-13T11:25:03.946 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:03.946 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:03.946 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:03.946 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:03.946 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:03.947 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDPGPushReply(3.22 125/122 [PushReplyOp(3:475b51c3:::benchmark_data_smithi177_31116_object16005:head)])", 2024-09-13T11:25:03.947 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:24:32.400814+0000", 2024-09-13T11:25:03.947 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 31.471619411999999, 2024-09-13T11:25:03.947 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.20917021799999999, 2024-09-13T11:25:03.947 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:03.947 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:03.947 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-13T11:25:03.947 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:03.947 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:03.947 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:03.947 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:32.400814+0000", 2024-09-13T11:25:03.947 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:03.947 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:03.947 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:03.948 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:03.948 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:32.400814+0000", 2024-09-13T11:25:03.948 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:03.948 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:03.948 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:03.948 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:03.948 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:32.400817+0000", 2024-09-13T11:25:03.948 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.5000000000000002e-06 2024-09-13T11:25:03.948 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:03.948 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:03.948 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:03.948 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:32.400826+0000", 2024-09-13T11:25:03.948 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.8830000000000001e-06 2024-09-13T11:25:03.948 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:03.949 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:03.949 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:03.949 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:32.608921+0000", 2024-09-13T11:25:03.949 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.20809501599999999 2024-09-13T11:25:03.949 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:03.949 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:03.949 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:03.949 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:32.608931+0000", 2024-09-13T11:25:03.949 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.1679999999999993e-06 2024-09-13T11:25:03.949 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:03.949 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:03.949 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:03.949 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:32.608982+0000", 2024-09-13T11:25:03.949 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.1827999999999999e-05 2024-09-13T11:25:03.950 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:03.950 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:03.950 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:03.950 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:32.609984+0000", 2024-09-13T11:25:03.950 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.001001823 2024-09-13T11:25:03.950 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:03.950 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:03.950 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:03.950 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:03.950 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:03.950 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:120816 3.21 3:850a51ba:::benchmark_data_smithi177_31116_object120815:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e129)", 2024-09-13T11:25:03.950 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:24:38.601204+0000", 2024-09-13T11:25:03.950 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 25.271229378000001, 2024-09-13T11:25:03.950 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.18932929600000001, 2024-09-13T11:25:03.950 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:03.951 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:03.951 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:03.951 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:25:03.951 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:25:03.951 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:03.951 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 120816 2024-09-13T11:25:03.951 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:03.951 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:03.951 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:03.951 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:03.951 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:38.601204+0000", 2024-09-13T11:25:03.951 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:03.951 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:03.952 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:03.952 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:03.952 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:38.601204+0000", 2024-09-13T11:25:03.952 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:03.952 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:03.952 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:03.952 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:03.952 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:38.601206+0000", 2024-09-13T11:25:03.952 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.7370000000000001e-06 2024-09-13T11:25:03.952 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:03.952 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:03.953 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:03.953 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:38.601241+0000", 2024-09-13T11:25:03.953 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.5524000000000001e-05 2024-09-13T11:25:03.953 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:03.953 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:03.953 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:03.953 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:38.601253+0000", 2024-09-13T11:25:03.953 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.1936000000000001e-05 2024-09-13T11:25:03.953 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:03.953 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:03.953 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:03.953 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:38.601262+0000", 2024-09-13T11:25:03.953 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.2570000000000007e-06 2024-09-13T11:25:03.953 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:03.954 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:03.954 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:03.954 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:38.601339+0000", 2024-09-13T11:25:03.954 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.7751999999999997e-05 2024-09-13T11:25:03.954 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:03.954 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:03.954 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:25:03.954 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:38.601527+0000", 2024-09-13T11:25:03.954 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000188092 2024-09-13T11:25:03.954 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:03.954 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:03.954 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 8", 2024-09-13T11:25:03.954 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:38.601684+0000", 2024-09-13T11:25:03.954 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00015659699999999999 2024-09-13T11:25:03.954 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:03.955 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:03.955 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:25:03.955 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:38.603108+0000", 2024-09-13T11:25:03.955 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0014244240000000001 2024-09-13T11:25:03.955 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:03.955 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:03.955 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:25:03.955 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:38.790425+0000", 2024-09-13T11:25:03.955 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.187316345 2024-09-13T11:25:03.955 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:03.955 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:03.955 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:25:03.955 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:38.790488+0000", 2024-09-13T11:25:03.955 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.3184999999999999e-05 2024-09-13T11:25:03.956 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:03.956 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:03.956 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:03.956 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:38.790533+0000", 2024-09-13T11:25:03.956 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.5447e-05 2024-09-13T11:25:03.956 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:03.956 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:03.956 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:03.956 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:03.956 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:03.956 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4665.0:129901 3.20 e134/77 3:067f5dfc:::benchmark_data_smithi177_31116_object129900:head v 134'4294, mlcod=134'4291)", 2024-09-13T11:25:03.956 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:24:47.145530+0000", 2024-09-13T11:25:03.956 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 16.726903101000001, 2024-09-13T11:25:03.957 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.333688349, 2024-09-13T11:25:03.957 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:03.957 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:03.957 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:03.957 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:03.957 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:03.957 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:03.957 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:47.145530+0000", 2024-09-13T11:25:03.957 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:03.957 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:03.957 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:03.957 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:03.957 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:47.145530+0000", 2024-09-13T11:25:03.957 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:03.958 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:03.958 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:03.958 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:03.958 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:47.145532+0000", 2024-09-13T11:25:03.958 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.7999999999999999e-06 2024-09-13T11:25:03.958 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:03.958 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:03.958 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:03.958 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:47.145557+0000", 2024-09-13T11:25:03.958 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.4678999999999999e-05 2024-09-13T11:25:03.958 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:03.958 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:03.958 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:03.958 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:47.477539+0000", 2024-09-13T11:25:03.959 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.33198164099999999 2024-09-13T11:25:03.959 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:03.959 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:03.959 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:03.959 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:47.477544+0000", 2024-09-13T11:25:03.959 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.9059999999999999e-06 2024-09-13T11:25:03.959 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:03.959 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:03.959 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:03.959 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:47.477597+0000", 2024-09-13T11:25:03.959 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.295e-05 2024-09-13T11:25:03.959 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:03.959 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:03.959 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:25:03.959 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:47.477619+0000", 2024-09-13T11:25:03.960 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.1835999999999999e-05 2024-09-13T11:25:03.960 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:03.960 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:03.960 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:25:03.960 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:47.479175+0000", 2024-09-13T11:25:03.960 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0015560470000000001 2024-09-13T11:25:03.960 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:03.960 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:03.960 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:03.960 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:47.479219+0000", 2024-09-13T11:25:03.960 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.3489999999999999e-05 2024-09-13T11:25:03.960 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:03.960 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:03.960 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:03.961 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:03.961 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:03.961 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:130006 3.21 3:85b3e472:::benchmark_data_smithi177_31116_object130005:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e134)", 2024-09-13T11:25:03.961 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:24:47.433580+0000", 2024-09-13T11:25:03.961 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 16.438853307999999, 2024-09-13T11:25:03.961 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.189659507, 2024-09-13T11:25:03.961 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:03.961 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:03.961 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:03.961 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:25:03.961 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:25:03.961 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:03.961 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 130006 2024-09-13T11:25:03.961 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:03.962 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:03.962 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:03.962 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:03.962 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:47.433580+0000", 2024-09-13T11:25:03.962 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:03.962 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:03.962 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:03.962 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:03.962 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:47.433580+0000", 2024-09-13T11:25:03.962 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:03.962 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:03.962 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:03.962 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:03.962 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:47.433582+0000", 2024-09-13T11:25:03.963 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.266e-06 2024-09-13T11:25:03.963 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:03.963 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:03.963 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:03.963 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:47.433656+0000", 2024-09-13T11:25:03.963 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.3780000000000002e-05 2024-09-13T11:25:03.963 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:03.963 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:03.963 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:03.963 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:47.433796+0000", 2024-09-13T11:25:03.963 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000139627 2024-09-13T11:25:03.963 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:03.963 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:03.963 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:03.963 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:47.433803+0000", 2024-09-13T11:25:03.964 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.5809999999999999e-06 2024-09-13T11:25:03.964 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:03.964 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:03.964 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:03.964 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:47.433849+0000", 2024-09-13T11:25:03.964 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.5485000000000002e-05 2024-09-13T11:25:03.964 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:03.964 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:03.964 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:25:03.964 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:47.433946+0000", 2024-09-13T11:25:03.964 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.6788999999999998e-05 2024-09-13T11:25:03.964 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:03.964 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:03.965 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 8", 2024-09-13T11:25:03.965 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:47.434032+0000", 2024-09-13T11:25:03.965 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.5898999999999999e-05 2024-09-13T11:25:03.965 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:03.965 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:03.965 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:25:03.965 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:47.435315+0000", 2024-09-13T11:25:03.965 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0012834319999999999 2024-09-13T11:25:03.965 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:03.965 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:03.965 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:25:03.965 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:47.623177+0000", 2024-09-13T11:25:03.965 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.187861741 2024-09-13T11:25:03.965 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:03.966 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:03.966 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:25:03.966 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:47.623211+0000", 2024-09-13T11:25:03.966 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.3989000000000003e-05 2024-09-13T11:25:03.966 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:03.966 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:03.966 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:03.966 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:47.623240+0000", 2024-09-13T11:25:03.966 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.8918e-05 2024-09-13T11:25:03.966 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:03.966 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:03.966 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:03.966 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:03.966 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:03.967 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:140840 3.21 3:87ae3db6:::benchmark_data_smithi177_31116_object537:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e144)", 2024-09-13T11:25:03.967 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:24:58.810939+0000", 2024-09-13T11:25:03.967 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 5.0614939879999996, 2024-09-13T11:25:03.967 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.25628309300000002, 2024-09-13T11:25:03.967 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:03.967 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:03.967 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:03.967 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:25:03.967 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:25:03.967 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:03.967 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 140840 2024-09-13T11:25:03.967 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:03.967 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:03.967 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:03.968 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:03.968 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.810939+0000", 2024-09-13T11:25:03.968 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:03.968 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:03.968 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:03.968 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:03.968 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.810939+0000", 2024-09-13T11:25:03.968 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:03.968 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:03.968 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:03.968 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:03.968 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.810942+0000", 2024-09-13T11:25:03.968 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.6259999999999999e-06 2024-09-13T11:25:03.968 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:03.969 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:03.969 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:03.969 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.810950+0000", 2024-09-13T11:25:03.969 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.9500000000000001e-06 2024-09-13T11:25:03.969 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:03.969 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:03.969 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:03.969 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.810970+0000", 2024-09-13T11:25:03.969 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.9539999999999999e-05 2024-09-13T11:25:03.969 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:03.969 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:03.969 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:03.969 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.810978+0000", 2024-09-13T11:25:03.969 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.8680000000000005e-06 2024-09-13T11:25:03.969 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:03.970 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:03.970 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:03.970 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.811060+0000", 2024-09-13T11:25:03.970 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.1068000000000004e-05 2024-09-13T11:25:03.970 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:03.970 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:03.970 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:25:03.970 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.811174+0000", 2024-09-13T11:25:03.970 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000114385 2024-09-13T11:25:03.970 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:03.970 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:03.970 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 8", 2024-09-13T11:25:03.970 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.811255+0000", 2024-09-13T11:25:03.970 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.1298000000000007e-05 2024-09-13T11:25:03.971 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:03.971 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:03.971 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:25:03.971 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.812869+0000", 2024-09-13T11:25:03.971 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.001614234 2024-09-13T11:25:03.971 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:03.971 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:03.971 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:25:03.971 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:59.067155+0000", 2024-09-13T11:25:03.971 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.25428578299999999 2024-09-13T11:25:03.971 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:03.971 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:03.971 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:25:03.971 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:59.067192+0000", 2024-09-13T11:25:03.971 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.659e-05 2024-09-13T11:25:03.972 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:03.972 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:03.972 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:03.972 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:59.067223+0000", 2024-09-13T11:25:03.972 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.0750999999999997e-05 2024-09-13T11:25:03.972 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:03.972 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:03.972 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:03.972 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:03.972 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:03.972 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-13T11:25:03.973 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-13T11:25:03.974 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-13T11:25:03.975 INFO:teuthology.orchestra.run.smithi001.stdout:osd.2: {} 2024-09-13T11:25:03.975 INFO:teuthology.orchestra.run.smithi001.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T11:25:03.975 INFO:teuthology.orchestra.run.smithi001.stdout:osd.3: {} 2024-09-13T11:25:03.976 INFO:teuthology.orchestra.run.smithi001.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T11:25:03.976 INFO:teuthology.orchestra.run.smithi001.stdout:osd.4: {} 2024-09-13T11:25:03.976 INFO:teuthology.orchestra.run.smithi001.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T11:25:03.976 INFO:teuthology.orchestra.run.smithi001.stdout:osd.5: {} 2024-09-13T11:25:03.977 INFO:teuthology.orchestra.run.smithi001.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T11:25:03.977 INFO:tasks.ceph.osd.2.smithi162.stderr:2024-09-13T11:25:03.972+0000 7fe92b35a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26313) UID: 0 2024-09-13T11:25:03.997 INFO:teuthology.orchestra.run.smithi001.stdout:osd.6: {} 2024-09-13T11:25:03.997 INFO:teuthology.orchestra.run.smithi001.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T11:25:04.000 INFO:teuthology.orchestra.run.smithi001.stderr:no valid command found; 10 closest matches: 2024-09-13T11:25:04.001 INFO:teuthology.orchestra.run.smithi001.stderr:0 2024-09-13T11:25:04.001 INFO:teuthology.orchestra.run.smithi001.stderr:1 2024-09-13T11:25:04.001 INFO:teuthology.orchestra.run.smithi001.stderr:2 2024-09-13T11:25:04.001 INFO:teuthology.orchestra.run.smithi001.stderr:abort 2024-09-13T11:25:04.001 INFO:teuthology.orchestra.run.smithi001.stderr:assert 2024-09-13T11:25:04.001 INFO:teuthology.orchestra.run.smithi001.stderr:bench [] [] [] [] 2024-09-13T11:25:04.001 INFO:teuthology.orchestra.run.smithi001.stderr:bluefs debug_inject_read_zeros 2024-09-13T11:25:04.001 INFO:teuthology.orchestra.run.smithi001.stderr:bluefs files list 2024-09-13T11:25:04.001 INFO:teuthology.orchestra.run.smithi001.stderr:bluefs stats 2024-09-13T11:25:04.001 INFO:teuthology.orchestra.run.smithi001.stderr:bluestore allocator dump block 2024-09-13T11:25:04.001 INFO:teuthology.orchestra.run.smithi001.stderr:Error EINVAL: invalid command 2024-09-13T11:25:04.004 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-13T11:25:04.005 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-13T11:25:04.005 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.13 'trim stale osdmaps' 2024-09-13T11:25:04.024 INFO:teuthology.orchestra.run.smithi001.stdout:osd.7: {} 2024-09-13T11:25:04.025 INFO:teuthology.orchestra.run.smithi001.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T11:25:04.059 INFO:teuthology.orchestra.run.smithi001.stdout:osd.8: {} 2024-09-13T11:25:04.059 INFO:teuthology.orchestra.run.smithi001.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-13T11:25:04.076 INFO:tasks.ceph.osd.12.smithi001.stderr:2024-09-13T11:25:04.071+0000 7f892c6e6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26314) UID: 0 2024-09-13T11:25:04.099 INFO:teuthology.orchestra.run.smithi001.stdout:osd.9: {} 2024-09-13T11:25:04.099 INFO:teuthology.orchestra.run.smithi001.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-13T11:25:04.141 INFO:teuthology.orchestra.run.smithi001.stdout:osd.10: {} 2024-09-13T11:25:04.142 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-13T11:25:04.142 INFO:teuthology.orchestra.run.smithi001.stdout: "ops": [ 2024-09-13T11:25:04.142 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:04.142 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:147481 3.11 3:89520566:::benchmark_data_smithi177_31116_object7178:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e148)", 2024-09-13T11:25:04.142 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:25:04.134908+0000", 2024-09-13T11:25:04.142 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 0.0063138379999999996, 2024-09-13T11:25:04.142 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0063402370000000003, 2024-09-13T11:25:04.142 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:04.142 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:04.142 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "waiting for sub ops", 2024-09-13T11:25:04.142 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:25:04.143 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:25:04.143 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:04.143 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 147481 2024-09-13T11:25:04.143 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:04.143 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:04.143 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:04.143 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:04.143 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:04.134908+0000", 2024-09-13T11:25:04.143 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:04.143 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:04.143 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:04.143 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:04.143 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:04.134908+0000", 2024-09-13T11:25:04.143 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:04.144 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:04.144 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:04.144 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:04.144 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:04.134911+0000", 2024-09-13T11:25:04.144 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.853e-06 2024-09-13T11:25:04.144 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:04.144 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:04.144 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:04.144 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:04.134919+0000", 2024-09-13T11:25:04.144 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.9650000000000008e-06 2024-09-13T11:25:04.144 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:04.144 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:04.144 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:04.144 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:04.134940+0000", 2024-09-13T11:25:04.145 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.0659e-05 2024-09-13T11:25:04.145 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:04.145 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:04.145 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:04.145 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:04.134948+0000", 2024-09-13T11:25:04.145 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.4959999999999996e-06 2024-09-13T11:25:04.145 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:04.145 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:04.145 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:04.145 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:04.136836+0000", 2024-09-13T11:25:04.145 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0018872889999999999 2024-09-13T11:25:04.145 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:04.145 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:04.145 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:25:04.146 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:04.136984+0000", 2024-09-13T11:25:04.146 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00014861900000000001 2024-09-13T11:25:04.146 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:04.146 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:04.146 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 11", 2024-09-13T11:25:04.146 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:04.137098+0000", 2024-09-13T11:25:04.146 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00011388 2024-09-13T11:25:04.146 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:04.146 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:04.146 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:25:04.146 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:04.140219+0000", 2024-09-13T11:25:04.146 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0031207159999999999 2024-09-13T11:25:04.146 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:04.146 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:04.146 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:04.147 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:04.147 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:04.147 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4665.0:147482 3.1e e148/102 3:7959747e:::benchmark_data_smithi177_31116_object7179:head v 148'6413, mlcod=148'6411)", 2024-09-13T11:25:04.147 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:25:04.135525+0000", 2024-09-13T11:25:04.147 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 0.0056969309999999997, 2024-09-13T11:25:04.147 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0057991400000000004, 2024-09-13T11:25:04.147 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:04.147 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:04.147 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:25:04.147 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:04.147 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:04.147 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:04.147 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:04.135525+0000", 2024-09-13T11:25:04.147 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:04.147 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:04.148 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:04.148 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:04.148 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:04.135525+0000", 2024-09-13T11:25:04.148 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:04.148 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:04.148 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:04.148 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:04.148 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:04.135527+0000", 2024-09-13T11:25:04.148 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.8160000000000001e-06 2024-09-13T11:25:04.148 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:04.148 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:04.148 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:04.148 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:04.135536+0000", 2024-09-13T11:25:04.148 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.4090000000000003e-06 2024-09-13T11:25:04.149 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:04.149 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:04.149 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:04.149 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:04.135570+0000", 2024-09-13T11:25:04.149 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.4675999999999998e-05 2024-09-13T11:25:04.149 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:04.149 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:04.149 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:04.149 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:04.135578+0000", 2024-09-13T11:25:04.149 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.3499999999999999e-06 2024-09-13T11:25:04.149 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:04.149 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:04.149 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:04.149 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:04.135636+0000", 2024-09-13T11:25:04.150 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.8400000000000003e-05 2024-09-13T11:25:04.150 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:04.150 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:04.150 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:25:04.150 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:04.135658+0000", 2024-09-13T11:25:04.150 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.1554999999999999e-05 2024-09-13T11:25:04.150 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:04.150 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:04.150 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:04.150 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:04.150 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-09-13T11:25:04.150 INFO:teuthology.orchestra.run.smithi001.stdout: "num_ops": 2 2024-09-13T11:25:04.150 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-13T11:25:04.151 INFO:teuthology.orchestra.run.smithi001.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-13T11:25:04.163 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-13T11:25:04.171 INFO:teuthology.orchestra.run.smithi001.stdout:osd.11: {} 2024-09-13T11:25:04.171 INFO:teuthology.orchestra.run.smithi001.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-13T11:25:04.176 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-09-13T11:25:04.172+0000 7f97befc9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26315) UID: 0 2024-09-13T11:25:04.199 INFO:teuthology.orchestra.run.smithi001.stdout:osd.12: {} 2024-09-13T11:25:04.200 INFO:teuthology.orchestra.run.smithi001.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-13T11:25:04.273 INFO:teuthology.orchestra.run.smithi001.stdout:osd.13: {} 2024-09-13T11:25:04.273 INFO:teuthology.orchestra.run.smithi001.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-13T11:25:04.276 INFO:tasks.ceph.osd.14.smithi162.stderr:2024-09-13T11:25:04.272+0000 7f3df5903640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33154) UID: 0 2024-09-13T11:25:04.278 INFO:teuthology.orchestra.run.smithi001.stderr:no valid command found; 10 closest matches: 2024-09-13T11:25:04.278 INFO:teuthology.orchestra.run.smithi001.stderr:0 2024-09-13T11:25:04.278 INFO:teuthology.orchestra.run.smithi001.stderr:1 2024-09-13T11:25:04.278 INFO:teuthology.orchestra.run.smithi001.stderr:2 2024-09-13T11:25:04.278 INFO:teuthology.orchestra.run.smithi001.stderr:abort 2024-09-13T11:25:04.278 INFO:teuthology.orchestra.run.smithi001.stderr:assert 2024-09-13T11:25:04.278 INFO:teuthology.orchestra.run.smithi001.stderr:bench [] [] [] [] 2024-09-13T11:25:04.278 INFO:teuthology.orchestra.run.smithi001.stderr:bluefs debug_inject_read_zeros 2024-09-13T11:25:04.278 INFO:teuthology.orchestra.run.smithi001.stderr:bluefs files list 2024-09-13T11:25:04.278 INFO:teuthology.orchestra.run.smithi001.stderr:bluefs stats 2024-09-13T11:25:04.278 INFO:teuthology.orchestra.run.smithi001.stderr:bluestore allocator dump block 2024-09-13T11:25:04.278 INFO:teuthology.orchestra.run.smithi001.stderr:Error EINVAL: invalid command 2024-09-13T11:25:04.281 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-13T11:25:04.282 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-13T11:25:04.282 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.15 'trim stale osdmaps' 2024-09-13T11:25:04.303 INFO:teuthology.orchestra.run.smithi001.stdout:osd.14: {} 2024-09-13T11:25:04.304 INFO:teuthology.orchestra.run.smithi001.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-13T11:25:04.310 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-13T11:25:04.310 INFO:teuthology.orchestra.run.smithi001.stdout: "ops": [], 2024-09-13T11:25:04.310 INFO:teuthology.orchestra.run.smithi001.stdout: "complaint_time": 30, 2024-09-13T11:25:04.311 INFO:teuthology.orchestra.run.smithi001.stdout: "num_blocked_ops": 0 2024-09-13T11:25:04.311 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-13T11:25:04.329 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-13T11:25:04.332 INFO:teuthology.orchestra.run.smithi001.stdout:osd.15: {} 2024-09-13T11:25:04.333 INFO:teuthology.orchestra.run.smithi001.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-13T11:25:04.400 INFO:tasks.ceph.osd.4.smithi001.stderr:2024-09-13T11:25:04.399+0000 7ff8e0218640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26310) UID: 0 2024-09-13T11:25:04.468 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-13T11:25:04.468 INFO:teuthology.orchestra.run.smithi001.stdout: "size": 20, 2024-09-13T11:25:04.468 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 600, 2024-09-13T11:25:04.468 INFO:teuthology.orchestra.run.smithi001.stdout: "ops": [ 2024-09-13T11:25:04.468 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:04.468 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDPGPush(3.11 72/70 [PushOp(3:8ae6b6fc:::benchmark_data_smithi177_31116_object7617:head, version: 46'455, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:8ae6b6fc:::benchmark_data_smithi177_31116_object7617:head@46'455, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:25:04.468 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:26.176296+0000", 2024-09-13T11:25:04.468 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 98.289816490999996, 2024-09-13T11:25:04.468 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.13349134900000001, 2024-09-13T11:25:04.469 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:04.469 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:04.469 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:25:04.469 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:04.469 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:04.469 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:04.469 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:26.176296+0000", 2024-09-13T11:25:04.469 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:04.469 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:04.469 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:04.469 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:04.469 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:26.176296+0000", 2024-09-13T11:25:04.469 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:04.470 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:04.470 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:04.470 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:04.470 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:26.176297+0000", 2024-09-13T11:25:04.470 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2440000000000001e-06 2024-09-13T11:25:04.470 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:04.470 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:04.470 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:04.470 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:26.176433+0000", 2024-09-13T11:25:04.470 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00013554999999999999 2024-09-13T11:25:04.470 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:04.470 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:04.470 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:04.470 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:26.308747+0000", 2024-09-13T11:25:04.471 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.132314392 2024-09-13T11:25:04.471 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:04.471 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:04.471 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:04.471 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:26.308768+0000", 2024-09-13T11:25:04.471 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.0758999999999999e-05 2024-09-13T11:25:04.471 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:04.471 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:04.471 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:04.471 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:26.308874+0000", 2024-09-13T11:25:04.471 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000105334 2024-09-13T11:25:04.471 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:04.471 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:04.472 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:25:04.472 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:26.308890+0000", 2024-09-13T11:25:04.472 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.6367000000000001e-05 2024-09-13T11:25:04.472 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:04.472 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:04.472 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:04.472 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:26.309788+0000", 2024-09-13T11:25:04.472 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00089770300000000002 2024-09-13T11:25:04.472 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:04.472 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:04.472 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:04.472 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:04.473 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:04.473 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDPGPush(3.11 72/70 [PushOp(3:8da9c977:::benchmark_data_smithi177_31116_object7736:head, version: 46'461, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:8da9c977:::benchmark_data_smithi177_31116_object7736:head@46'461, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:25:04.473 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:26.393216+0000", 2024-09-13T11:25:04.473 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 98.072896631999996, 2024-09-13T11:25:04.473 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.15683955499999999, 2024-09-13T11:25:04.473 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:04.473 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:04.473 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:25:04.473 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:04.473 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:04.473 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:04.473 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:26.393216+0000", 2024-09-13T11:25:04.473 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:04.473 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:04.474 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:04.474 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:04.474 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:26.393216+0000", 2024-09-13T11:25:04.474 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:04.474 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:04.474 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:04.474 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:04.474 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:26.393217+0000", 2024-09-13T11:25:04.474 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2410000000000001e-06 2024-09-13T11:25:04.474 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:04.474 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:04.474 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:04.474 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:26.393309+0000", 2024-09-13T11:25:04.474 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.2101000000000005e-05 2024-09-13T11:25:04.475 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:04.475 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:04.475 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:04.475 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:26.547936+0000", 2024-09-13T11:25:04.475 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.15462682699999999 2024-09-13T11:25:04.475 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:04.475 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:04.475 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:04.475 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:26.548021+0000", 2024-09-13T11:25:04.475 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.4529000000000003e-05 2024-09-13T11:25:04.475 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:04.475 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:04.475 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:04.475 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:26.548321+0000", 2024-09-13T11:25:04.476 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00029988399999999999 2024-09-13T11:25:04.476 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:04.476 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:04.476 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:25:04.476 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:26.548353+0000", 2024-09-13T11:25:04.476 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.2573000000000001e-05 2024-09-13T11:25:04.476 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:04.476 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:04.476 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:04.476 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:26.550056+0000", 2024-09-13T11:25:04.476 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0017024 2024-09-13T11:25:04.476 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:04.476 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:04.476 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:04.477 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:04.477 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:04.477 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDPGPush(3.1a 86/78 [PushOp(3:5df655d4:::benchmark_data_smithi177_31116_object12486:head, version: 49'767, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:5df655d4:::benchmark_data_smithi177_31116_object12486:head@49'767, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:25:04.477 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:43.456210+0000", 2024-09-13T11:25:04.477 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 81.009902330000003, 2024-09-13T11:25:04.477 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.131081792, 2024-09-13T11:25:04.477 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:04.477 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:04.477 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:25:04.477 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:04.477 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:04.477 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:04.477 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:43.456210+0000", 2024-09-13T11:25:04.478 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:04.478 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:04.478 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:04.478 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:04.478 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:43.456210+0000", 2024-09-13T11:25:04.478 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:04.478 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:04.478 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:04.478 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:04.478 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:43.456212+0000", 2024-09-13T11:25:04.478 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.922e-06 2024-09-13T11:25:04.478 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:04.479 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:04.479 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:04.479 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:43.456262+0000", 2024-09-13T11:25:04.479 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.9242000000000001e-05 2024-09-13T11:25:04.479 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:04.479 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:04.479 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:04.479 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:43.586188+0000", 2024-09-13T11:25:04.479 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.12992674400000001 2024-09-13T11:25:04.479 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:04.479 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:04.479 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:04.479 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:43.586209+0000", 2024-09-13T11:25:04.480 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.031e-05 2024-09-13T11:25:04.480 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:04.480 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:04.480 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:04.480 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:43.586325+0000", 2024-09-13T11:25:04.480 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000115983 2024-09-13T11:25:04.480 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:04.480 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:04.480 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:25:04.480 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:43.586340+0000", 2024-09-13T11:25:04.480 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.4941e-05 2024-09-13T11:25:04.481 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:04.481 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:04.481 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:04.481 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:43.587292+0000", 2024-09-13T11:25:04.481 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00095264999999999998 2024-09-13T11:25:04.481 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:04.481 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:04.481 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:04.481 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:04.481 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:04.481 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDPGPush(3.1e 89/78 [PushOp(3:7a07f38c:::benchmark_data_smithi177_31116_object20966:head, version: 54'1402, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:7a07f38c:::benchmark_data_smithi177_31116_object20966:head@54'1402, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:25:04.481 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:46.645251+0000", 2024-09-13T11:25:04.481 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 77.820861917000002, 2024-09-13T11:25:04.481 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.17670883800000001, 2024-09-13T11:25:04.482 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:04.482 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:04.482 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:25:04.482 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:04.482 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:04.482 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:04.482 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.645251+0000", 2024-09-13T11:25:04.482 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:04.482 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:04.482 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:04.482 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:04.482 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.645251+0000", 2024-09-13T11:25:04.482 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:04.482 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:04.483 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:04.483 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:04.483 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.645253+0000", 2024-09-13T11:25:04.483 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.1399999999999998e-06 2024-09-13T11:25:04.483 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:04.483 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:04.483 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:04.483 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.645295+0000", 2024-09-13T11:25:04.483 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.2323000000000001e-05 2024-09-13T11:25:04.483 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:04.483 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:04.483 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:04.483 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.820731+0000", 2024-09-13T11:25:04.484 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.17543607999999999 2024-09-13T11:25:04.484 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:04.484 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:04.484 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:04.484 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.820750+0000", 2024-09-13T11:25:04.484 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.8793000000000001e-05 2024-09-13T11:25:04.484 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:04.484 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:04.484 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:04.484 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.820914+0000", 2024-09-13T11:25:04.484 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000164062 2024-09-13T11:25:04.484 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:04.484 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:04.485 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:25:04.485 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.820939+0000", 2024-09-13T11:25:04.485 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.4672e-05 2024-09-13T11:25:04.485 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:04.485 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:04.485 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:04.485 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.821960+0000", 2024-09-13T11:25:04.485 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.001020768 2024-09-13T11:25:04.485 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:04.485 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:04.485 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:04.485 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:04.485 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:04.486 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:64320 3.11 3:8de5de4c:::benchmark_data_smithi177_31116_object64319:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90)", 2024-09-13T11:25:04.486 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:47.132785+0000", 2024-09-13T11:25:04.486 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 77.333327616000005, 2024-09-13T11:25:04.486 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.11946314600000001, 2024-09-13T11:25:04.486 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:04.486 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:04.486 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:04.486 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:25:04.486 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:25:04.486 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:04.486 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 64320 2024-09-13T11:25:04.486 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:04.487 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:04.487 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:04.487 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:04.487 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:47.132785+0000", 2024-09-13T11:25:04.487 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:04.487 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:04.487 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:04.487 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:04.487 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:47.132785+0000", 2024-09-13T11:25:04.488 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:04.488 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:04.488 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:04.488 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:04.488 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:47.132788+0000", 2024-09-13T11:25:04.488 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.4619999999999999e-06 2024-09-13T11:25:04.488 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:04.488 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:04.488 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:04.488 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:47.132838+0000", 2024-09-13T11:25:04.488 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.0739000000000003e-05 2024-09-13T11:25:04.488 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:04.489 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:04.489 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:04.489 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:47.132853+0000", 2024-09-13T11:25:04.489 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.518e-05 2024-09-13T11:25:04.489 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:04.489 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:04.489 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:04.489 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:47.132860+0000", 2024-09-13T11:25:04.489 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.8689999999999998e-06 2024-09-13T11:25:04.489 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:04.489 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:04.489 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:04.490 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:47.132922+0000", 2024-09-13T11:25:04.490 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.1251e-05 2024-09-13T11:25:04.490 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:04.490 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:04.490 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:25:04.490 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:47.133034+0000", 2024-09-13T11:25:04.490 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000112742 2024-09-13T11:25:04.490 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:04.490 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:04.491 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 11", 2024-09-13T11:25:04.491 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:47.133141+0000", 2024-09-13T11:25:04.491 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000106968 2024-09-13T11:25:04.491 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:04.491 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:04.491 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:25:04.491 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:47.135566+0000", 2024-09-13T11:25:04.491 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0024251839999999999 2024-09-13T11:25:04.491 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:04.491 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:04.491 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:25:04.491 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:47.252135+0000", 2024-09-13T11:25:04.491 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.11656870900000001 2024-09-13T11:25:04.491 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:04.492 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:04.492 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:25:04.492 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:47.252201+0000", 2024-09-13T11:25:04.492 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.6223999999999995e-05 2024-09-13T11:25:04.492 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:04.492 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:04.492 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:04.492 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:47.252248+0000", 2024-09-13T11:25:04.492 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.6817999999999998e-05 2024-09-13T11:25:04.492 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:04.492 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:04.492 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:04.492 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:04.492 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:04.493 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDPGPush(3.1e 91/78 [PushOp(3:78b7dd1b:::benchmark_data_smithi177_31116_object23441:head, version: 55'1540, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:78b7dd1b:::benchmark_data_smithi177_31116_object23441:head@55'1540, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:25:04.493 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:48.342756+0000", 2024-09-13T11:25:04.493 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 76.123356544000004, 2024-09-13T11:25:04.493 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.114126449, 2024-09-13T11:25:04.493 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:04.493 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:04.493 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:25:04.493 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:04.493 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:04.493 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:04.493 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:48.342756+0000", 2024-09-13T11:25:04.493 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:04.493 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:04.494 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:04.494 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:04.494 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:48.342756+0000", 2024-09-13T11:25:04.494 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:04.494 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:04.494 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:04.494 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:04.494 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:48.342758+0000", 2024-09-13T11:25:04.494 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.9769999999999999e-06 2024-09-13T11:25:04.494 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:04.494 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:04.494 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:04.494 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:48.342796+0000", 2024-09-13T11:25:04.494 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.8331000000000003e-05 2024-09-13T11:25:04.495 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:04.495 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:04.495 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:04.495 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:48.455578+0000", 2024-09-13T11:25:04.495 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.11278160800000001 2024-09-13T11:25:04.495 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:04.495 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:04.495 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:04.495 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:48.455631+0000", 2024-09-13T11:25:04.495 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.3229999999999997e-05 2024-09-13T11:25:04.495 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:04.495 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:04.495 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:04.495 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:48.455799+0000", 2024-09-13T11:25:04.496 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00016763100000000001 2024-09-13T11:25:04.496 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:04.496 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:04.496 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:25:04.496 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:48.455813+0000", 2024-09-13T11:25:04.496 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.4416999999999999e-05 2024-09-13T11:25:04.496 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:04.496 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:04.496 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:04.496 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:48.456883+0000", 2024-09-13T11:25:04.496 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0010692550000000001 2024-09-13T11:25:04.496 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:04.496 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:04.496 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:04.497 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:04.497 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:04.497 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:71961 3.11 3:89bd9466:::benchmark_data_smithi177_31116_object71960:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2024-09-13T11:25:04.497 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:54.267475+0000", 2024-09-13T11:25:04.497 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 70.198637718000001, 2024-09-13T11:25:04.497 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.15027747899999999, 2024-09-13T11:25:04.497 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:04.497 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:04.497 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:04.497 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:25:04.497 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:25:04.497 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:04.497 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 71961 2024-09-13T11:25:04.498 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:04.498 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:04.498 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:04.498 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:04.498 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:54.267475+0000", 2024-09-13T11:25:04.498 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:04.498 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:04.498 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:04.498 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:04.498 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:54.267475+0000", 2024-09-13T11:25:04.498 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:04.498 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:04.498 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:04.498 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:04.499 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:54.267478+0000", 2024-09-13T11:25:04.499 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.8080000000000001e-06 2024-09-13T11:25:04.499 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:04.499 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:04.499 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:04.499 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:54.267534+0000", 2024-09-13T11:25:04.499 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.6276999999999997e-05 2024-09-13T11:25:04.499 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:04.499 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:04.499 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:04.499 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:54.267557+0000", 2024-09-13T11:25:04.499 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.2514000000000002e-05 2024-09-13T11:25:04.499 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:04.500 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:04.500 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:04.500 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:54.267570+0000", 2024-09-13T11:25:04.500 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.3251999999999999e-05 2024-09-13T11:25:04.500 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:04.500 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:04.500 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:04.500 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:54.267650+0000", 2024-09-13T11:25:04.500 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.0066000000000002e-05 2024-09-13T11:25:04.500 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:04.500 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:04.500 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:25:04.500 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:54.267886+0000", 2024-09-13T11:25:04.501 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00023637800000000001 2024-09-13T11:25:04.501 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:04.501 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:04.501 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 11", 2024-09-13T11:25:04.501 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:54.268020+0000", 2024-09-13T11:25:04.501 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00013392100000000001 2024-09-13T11:25:04.501 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:04.501 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:04.501 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:25:04.501 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:54.270076+0000", 2024-09-13T11:25:04.501 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.002055393 2024-09-13T11:25:04.501 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:04.501 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:04.501 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:25:04.502 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:54.417584+0000", 2024-09-13T11:25:04.502 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.147508008 2024-09-13T11:25:04.502 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:04.502 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:04.502 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:25:04.502 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:54.417667+0000", 2024-09-13T11:25:04.502 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.3666999999999997e-05 2024-09-13T11:25:04.502 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:04.502 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:04.502 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:04.502 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:54.417752+0000", 2024-09-13T11:25:04.502 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.5195e-05 2024-09-13T11:25:04.502 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:04.503 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:04.503 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:04.503 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:04.503 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:04.503 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4665.0:79776 3.1a e102/78 3:5d637d57:::benchmark_data_smithi177_31116_object79775:head v 102'4120, mlcod=72'3145)", 2024-09-13T11:25:04.503 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:24:00.354950+0000", 2024-09-13T11:25:04.503 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 64.111162374000003, 2024-09-13T11:25:04.503 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.14753330100000001, 2024-09-13T11:25:04.503 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:04.503 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:04.503 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:04.503 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:04.503 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:04.504 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:04.504 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:00.354950+0000", 2024-09-13T11:25:04.504 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:04.504 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:04.504 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:04.504 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:04.504 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:00.354950+0000", 2024-09-13T11:25:04.504 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:04.504 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:04.504 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:04.504 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:04.504 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:00.354960+0000", 2024-09-13T11:25:04.504 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.5219999999999996e-06 2024-09-13T11:25:04.505 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:04.505 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:04.505 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:04.505 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:00.355024+0000", 2024-09-13T11:25:04.505 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.4163000000000002e-05 2024-09-13T11:25:04.505 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:04.505 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:04.505 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:04.505 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:00.499446+0000", 2024-09-13T11:25:04.505 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.144422352 2024-09-13T11:25:04.505 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:04.506 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:04.506 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:04.506 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:00.499466+0000", 2024-09-13T11:25:04.506 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.9694000000000002e-05 2024-09-13T11:25:04.506 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:04.506 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:04.506 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:04.506 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:00.499605+0000", 2024-09-13T11:25:04.506 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00013935099999999999 2024-09-13T11:25:04.506 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:04.506 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:04.506 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:25:04.507 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:00.499655+0000", 2024-09-13T11:25:04.507 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.9787000000000003e-05 2024-09-13T11:25:04.507 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:04.507 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:04.507 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:25:04.507 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:00.502418+0000", 2024-09-13T11:25:04.507 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.002763012 2024-09-13T11:25:04.507 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:04.507 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:04.507 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:04.508 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:00.502484+0000", 2024-09-13T11:25:04.508 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.5419999999999994e-05 2024-09-13T11:25:04.508 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:04.508 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:04.508 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:04.508 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:04.508 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:04.508 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDPGPush(3.1a 102/78 [PushOp(3:5df50766:::benchmark_data_smithi177_31116_object50543:head, version: 72'3152, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:5df50766:::benchmark_data_smithi177_31116_object50543:head@72'3152, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:25:04.508 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:24:00.499311+0000", 2024-09-13T11:25:04.508 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 63.966801832000002, 2024-09-13T11:25:04.508 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.26146664800000002, 2024-09-13T11:25:04.508 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:04.508 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:04.509 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:25:04.509 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:04.509 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:04.509 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:04.509 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:00.499311+0000", 2024-09-13T11:25:04.509 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:04.509 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:04.509 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:04.509 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:04.510 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:00.499311+0000", 2024-09-13T11:25:04.510 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:04.510 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:04.510 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:04.510 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:04.510 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:00.499320+0000", 2024-09-13T11:25:04.510 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.6400000000000003e-06 2024-09-13T11:25:04.510 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:04.510 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:04.510 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:04.510 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:00.499384+0000", 2024-09-13T11:25:04.510 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.4554999999999994e-05 2024-09-13T11:25:04.510 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:04.511 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:04.511 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:04.511 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:00.759713+0000", 2024-09-13T11:25:04.511 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.260328742 2024-09-13T11:25:04.511 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:04.511 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:04.511 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:04.511 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:00.759742+0000", 2024-09-13T11:25:04.511 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.9298e-05 2024-09-13T11:25:04.511 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:04.511 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:04.511 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:04.512 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:00.759871+0000", 2024-09-13T11:25:04.512 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000128992 2024-09-13T11:25:04.512 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:04.512 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:04.512 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:25:04.512 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:00.759884+0000", 2024-09-13T11:25:04.512 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2687999999999999e-05 2024-09-13T11:25:04.512 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:04.512 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:04.512 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:04.512 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:00.760778+0000", 2024-09-13T11:25:04.513 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00089373299999999996 2024-09-13T11:25:04.513 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:04.513 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:04.513 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:04.513 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:04.513 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:04.513 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4665.0:82769 3.1e e105/102 3:7a4e88b5:::benchmark_data_smithi177_31116_object82768:head v 105'4381, mlcod=105'4379)", 2024-09-13T11:25:04.513 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:24:03.745047+0000", 2024-09-13T11:25:04.513 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 60.721065607, 2024-09-13T11:25:04.513 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.213806145, 2024-09-13T11:25:04.513 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:04.513 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:04.513 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:04.513 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:04.514 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:04.514 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:04.514 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:03.745047+0000", 2024-09-13T11:25:04.514 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:04.514 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:04.514 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:04.514 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:04.514 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:03.745047+0000", 2024-09-13T11:25:04.514 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:04.514 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:04.515 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:04.515 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:04.515 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:03.745049+0000", 2024-09-13T11:25:04.515 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.5710000000000001e-06 2024-09-13T11:25:04.515 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:04.515 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:04.515 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:04.515 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:03.745107+0000", 2024-09-13T11:25:04.515 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.8799999999999999e-05 2024-09-13T11:25:04.515 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:04.515 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:04.515 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:04.515 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:03.955145+0000", 2024-09-13T11:25:04.515 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.210037735 2024-09-13T11:25:04.516 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:04.516 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:04.516 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:04.516 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:03.955163+0000", 2024-09-13T11:25:04.516 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.7520000000000002e-05 2024-09-13T11:25:04.516 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:04.516 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:04.516 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:04.516 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:03.955473+0000", 2024-09-13T11:25:04.516 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00031014399999999999 2024-09-13T11:25:04.516 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:04.516 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:04.516 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:25:04.517 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:03.955538+0000", 2024-09-13T11:25:04.517 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.5059000000000002e-05 2024-09-13T11:25:04.517 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:04.517 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:04.517 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:25:04.517 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:03.958753+0000", 2024-09-13T11:25:04.517 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0032152769999999999 2024-09-13T11:25:04.517 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:04.517 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:04.517 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:04.517 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:03.958853+0000", 2024-09-13T11:25:04.517 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000100039 2024-09-13T11:25:04.517 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:04.517 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:04.518 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:04.518 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:04.518 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:04.518 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:98582 3.1a 3:59f9cd75:::benchmark_data_smithi177_31116_object98581:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113)", 2024-09-13T11:25:04.518 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:24:15.431226+0000", 2024-09-13T11:25:04.518 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 49.034887138000002, 2024-09-13T11:25:04.518 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.19820755800000001, 2024-09-13T11:25:04.518 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:04.518 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:04.518 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:04.518 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:25:04.518 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:25:04.518 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:04.518 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 98582 2024-09-13T11:25:04.519 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:04.519 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:04.519 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:04.519 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:04.519 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.431226+0000", 2024-09-13T11:25:04.519 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:04.519 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:04.519 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:04.519 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:04.519 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.431226+0000", 2024-09-13T11:25:04.519 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:04.520 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:04.520 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:04.520 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:04.520 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.431229+0000", 2024-09-13T11:25:04.520 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.7100000000000001e-06 2024-09-13T11:25:04.520 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:04.520 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:04.520 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:04.520 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.431334+0000", 2024-09-13T11:25:04.520 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00010506900000000001 2024-09-13T11:25:04.520 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:04.520 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:04.520 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:04.520 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.431364+0000", 2024-09-13T11:25:04.521 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.9607e-05 2024-09-13T11:25:04.521 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:04.521 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:04.521 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:04.521 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.431388+0000", 2024-09-13T11:25:04.521 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.3808999999999999e-05 2024-09-13T11:25:04.521 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:04.521 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:04.521 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:04.521 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.431483+0000", 2024-09-13T11:25:04.521 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.5402999999999994e-05 2024-09-13T11:25:04.521 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:04.521 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:04.521 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:25:04.522 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.431642+0000", 2024-09-13T11:25:04.522 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000158412 2024-09-13T11:25:04.522 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:04.522 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:04.522 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 9", 2024-09-13T11:25:04.522 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.431796+0000", 2024-09-13T11:25:04.522 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00015467899999999999 2024-09-13T11:25:04.522 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:04.522 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:04.523 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:25:04.523 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.433731+0000", 2024-09-13T11:25:04.523 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.001934945 2024-09-13T11:25:04.523 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:04.523 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:04.523 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:25:04.523 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.629245+0000", 2024-09-13T11:25:04.523 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.19551390199999999 2024-09-13T11:25:04.523 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:04.523 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:04.523 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:25:04.523 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.629346+0000", 2024-09-13T11:25:04.523 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000100869 2024-09-13T11:25:04.524 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:04.524 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:04.524 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:04.524 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.629433+0000", 2024-09-13T11:25:04.524 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.7152999999999997e-05 2024-09-13T11:25:04.524 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:04.524 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:04.524 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:04.524 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:04.524 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:04.524 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:99105 3.1a 3:5ac5090c:::benchmark_data_smithi177_31116_object99104:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113)", 2024-09-13T11:25:04.524 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:24:15.962466+0000", 2024-09-13T11:25:04.524 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 48.503646848999999, 2024-09-13T11:25:04.524 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.23832849, 2024-09-13T11:25:04.525 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:04.525 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:04.525 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:04.525 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:25:04.525 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:25:04.525 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:04.525 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 99105 2024-09-13T11:25:04.525 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:04.525 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:04.525 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:04.526 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:04.526 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.962466+0000", 2024-09-13T11:25:04.526 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:04.526 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:04.526 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:04.526 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:04.526 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.962466+0000", 2024-09-13T11:25:04.526 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:04.526 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:04.526 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:04.526 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:04.526 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.962471+0000", 2024-09-13T11:25:04.526 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.9570000000000001e-06 2024-09-13T11:25:04.527 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:04.527 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:04.527 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:04.527 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.962547+0000", 2024-09-13T11:25:04.527 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.5947999999999999e-05 2024-09-13T11:25:04.528 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:04.528 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:04.528 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:04.528 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.962576+0000", 2024-09-13T11:25:04.528 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.9547999999999999e-05 2024-09-13T11:25:04.528 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:04.528 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:04.528 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:04.528 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.962594+0000", 2024-09-13T11:25:04.528 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.8023e-05 2024-09-13T11:25:04.529 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:04.529 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:04.529 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:04.529 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.962770+0000", 2024-09-13T11:25:04.529 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00017551099999999999 2024-09-13T11:25:04.529 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:04.529 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:04.529 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:25:04.529 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.963114+0000", 2024-09-13T11:25:04.529 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00034440399999999998 2024-09-13T11:25:04.529 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:04.529 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:04.529 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 9", 2024-09-13T11:25:04.530 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.963351+0000", 2024-09-13T11:25:04.530 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000237209 2024-09-13T11:25:04.530 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:04.530 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:04.530 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:25:04.530 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.966345+0000", 2024-09-13T11:25:04.530 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0029933519999999999 2024-09-13T11:25:04.530 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:04.530 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:04.530 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:25:04.530 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:16.200724+0000", 2024-09-13T11:25:04.530 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.23437934599999999 2024-09-13T11:25:04.530 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:04.530 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:04.531 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:25:04.531 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:16.200761+0000", 2024-09-13T11:25:04.531 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.6406999999999999e-05 2024-09-13T11:25:04.531 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:04.531 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:04.531 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:04.531 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:16.200794+0000", 2024-09-13T11:25:04.531 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.3785000000000003e-05 2024-09-13T11:25:04.531 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:04.531 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:04.531 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:04.531 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:04.531 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:04.532 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4665.0:104331 3.1e e115/102 3:79890035:::benchmark_data_smithi177_31116_object104330:head v 115'5083, mlcod=115'5080)", 2024-09-13T11:25:04.532 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:24:20.242656+0000", 2024-09-13T11:25:04.532 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 44.223456757000001, 2024-09-13T11:25:04.532 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.16813846700000001, 2024-09-13T11:25:04.532 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:04.532 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:04.532 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:04.532 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:04.532 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:04.532 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:04.533 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:20.242656+0000", 2024-09-13T11:25:04.533 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:04.533 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:04.533 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:04.533 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:04.533 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:20.242656+0000", 2024-09-13T11:25:04.533 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:04.533 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:04.533 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:04.533 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:04.533 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:20.242657+0000", 2024-09-13T11:25:04.533 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.4750000000000001e-06 2024-09-13T11:25:04.533 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:04.533 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:04.534 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:04.534 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:20.242699+0000", 2024-09-13T11:25:04.534 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.193e-05 2024-09-13T11:25:04.534 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:04.534 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:04.534 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:04.534 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:20.408772+0000", 2024-09-13T11:25:04.534 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.16607291199999999 2024-09-13T11:25:04.534 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:04.534 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:04.534 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:04.534 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:20.408779+0000", 2024-09-13T11:25:04.534 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.3600000000000001e-06 2024-09-13T11:25:04.534 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:04.535 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:04.535 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:04.535 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:20.408840+0000", 2024-09-13T11:25:04.535 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.1834999999999993e-05 2024-09-13T11:25:04.535 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:04.535 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:04.535 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:25:04.535 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:20.408865+0000", 2024-09-13T11:25:04.535 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.4294000000000002e-05 2024-09-13T11:25:04.535 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:04.535 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:04.535 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:25:04.535 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:20.410751+0000", 2024-09-13T11:25:04.535 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.001885811 2024-09-13T11:25:04.536 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:04.536 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:04.536 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:04.536 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:20.410794+0000", 2024-09-13T11:25:04.536 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.3850000000000002e-05 2024-09-13T11:25:04.536 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:04.536 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:04.536 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:04.536 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:04.536 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:04.536 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:113165 3.11 3:8dbd8762:::benchmark_data_smithi177_31116_object113164:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124)", 2024-09-13T11:25:04.536 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:24:31.641595+0000", 2024-09-13T11:25:04.536 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 32.824518206999997, 2024-09-13T11:25:04.536 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.117623435, 2024-09-13T11:25:04.537 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:04.537 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:04.537 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:04.537 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:25:04.537 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:25:04.537 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:04.537 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 113165 2024-09-13T11:25:04.537 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:04.537 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:04.537 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:04.537 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:04.537 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:31.641595+0000", 2024-09-13T11:25:04.537 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:04.537 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:04.538 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:04.538 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:04.538 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:31.641595+0000", 2024-09-13T11:25:04.538 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:04.538 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:04.538 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:04.538 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:04.538 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:31.641596+0000", 2024-09-13T11:25:04.538 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.553e-06 2024-09-13T11:25:04.538 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:04.538 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:04.538 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:04.538 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:31.641655+0000", 2024-09-13T11:25:04.539 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.8737999999999999e-05 2024-09-13T11:25:04.539 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:04.539 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:04.539 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:04.539 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:31.641664+0000", 2024-09-13T11:25:04.539 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.2429999999999994e-06 2024-09-13T11:25:04.539 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:04.539 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:04.539 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:04.539 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:31.641669+0000", 2024-09-13T11:25:04.539 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.7790000000000003e-06 2024-09-13T11:25:04.539 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:04.539 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:04.539 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:04.539 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:31.641723+0000", 2024-09-13T11:25:04.540 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.3718999999999998e-05 2024-09-13T11:25:04.540 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:04.540 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:04.540 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:25:04.540 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:31.641846+0000", 2024-09-13T11:25:04.540 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000123194 2024-09-13T11:25:04.540 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:04.540 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:04.540 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 11", 2024-09-13T11:25:04.540 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:31.641967+0000", 2024-09-13T11:25:04.540 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00012108900000000001 2024-09-13T11:25:04.540 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:04.540 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:04.541 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:25:04.541 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:31.643582+0000", 2024-09-13T11:25:04.541 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.001615214 2024-09-13T11:25:04.541 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:04.541 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:04.541 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:25:04.541 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:31.759116+0000", 2024-09-13T11:25:04.541 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.115533633 2024-09-13T11:25:04.541 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:04.541 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:04.541 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:25:04.541 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:31.759174+0000", 2024-09-13T11:25:04.541 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.8140999999999998e-05 2024-09-13T11:25:04.541 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:04.542 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:04.542 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:04.542 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:31.759218+0000", 2024-09-13T11:25:04.542 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.4131999999999997e-05 2024-09-13T11:25:04.542 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:04.542 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:04.542 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:04.542 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:04.542 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:04.542 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:119446 3.1a 3:5b664280:::benchmark_data_smithi177_31116_object119445:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e128)", 2024-09-13T11:25:04.542 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:24:37.343167+0000", 2024-09-13T11:25:04.542 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 27.1229455, 2024-09-13T11:25:04.542 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.119771422, 2024-09-13T11:25:04.543 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:04.543 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:04.543 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:04.543 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:25:04.543 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:25:04.543 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:04.543 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 119446 2024-09-13T11:25:04.543 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:04.543 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:04.544 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:04.544 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:04.544 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:37.343167+0000", 2024-09-13T11:25:04.544 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:04.544 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:04.544 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:04.544 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:04.544 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:37.343167+0000", 2024-09-13T11:25:04.544 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:04.544 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:04.544 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:04.544 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:04.544 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:37.343174+0000", 2024-09-13T11:25:04.545 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.0489999999999998e-06 2024-09-13T11:25:04.545 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:04.545 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:04.545 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:04.545 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:37.343287+0000", 2024-09-13T11:25:04.545 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00011322 2024-09-13T11:25:04.545 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:04.545 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:04.545 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:04.545 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:37.343309+0000", 2024-09-13T11:25:04.545 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.1299999999999999e-05 2024-09-13T11:25:04.545 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:04.545 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:04.545 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:04.546 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:37.343320+0000", 2024-09-13T11:25:04.546 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0818999999999999e-05 2024-09-13T11:25:04.546 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:04.546 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:04.546 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:04.546 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:37.343397+0000", 2024-09-13T11:25:04.546 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.7000000000000001e-05 2024-09-13T11:25:04.546 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:04.546 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:04.546 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:25:04.546 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:37.343561+0000", 2024-09-13T11:25:04.546 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00016474799999999999 2024-09-13T11:25:04.546 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:04.546 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:04.547 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 9", 2024-09-13T11:25:04.547 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:37.343712+0000", 2024-09-13T11:25:04.547 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000150975 2024-09-13T11:25:04.547 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:04.547 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:04.547 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:25:04.547 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:37.346031+0000", 2024-09-13T11:25:04.547 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0023190350000000001 2024-09-13T11:25:04.547 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:04.547 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:04.547 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:25:04.547 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:37.462840+0000", 2024-09-13T11:25:04.547 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.116808593 2024-09-13T11:25:04.547 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:04.548 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:04.548 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:25:04.548 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:37.462896+0000", 2024-09-13T11:25:04.548 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.5591999999999999e-05 2024-09-13T11:25:04.548 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:04.548 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:04.548 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:04.548 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:37.462939+0000", 2024-09-13T11:25:04.548 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.3090999999999998e-05 2024-09-13T11:25:04.548 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:04.548 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:04.548 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:04.548 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:04.548 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:04.549 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:119844 3.1a 3:5858dd7d:::benchmark_data_smithi177_31116_object119843:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e128)", 2024-09-13T11:25:04.549 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:24:37.691795+0000", 2024-09-13T11:25:04.549 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 26.774317433, 2024-09-13T11:25:04.549 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.12927502699999999, 2024-09-13T11:25:04.549 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:04.549 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:04.549 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:04.549 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:25:04.549 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:25:04.549 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:04.549 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 119844 2024-09-13T11:25:04.549 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:04.549 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:04.549 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:04.550 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:04.550 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:37.691795+0000", 2024-09-13T11:25:04.550 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:04.550 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:04.550 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:04.550 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:04.550 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:37.691795+0000", 2024-09-13T11:25:04.550 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:04.550 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:04.550 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:04.550 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:04.551 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:37.691799+0000", 2024-09-13T11:25:04.551 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.9269999999999998e-06 2024-09-13T11:25:04.551 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:04.551 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:04.551 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:04.551 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:37.691859+0000", 2024-09-13T11:25:04.551 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.0010999999999999e-05 2024-09-13T11:25:04.551 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:04.551 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:04.551 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:04.551 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:37.691879+0000", 2024-09-13T11:25:04.551 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.9624999999999999e-05 2024-09-13T11:25:04.551 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:04.552 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:04.552 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:04.552 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:37.691888+0000", 2024-09-13T11:25:04.552 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.5629999999999998e-06 2024-09-13T11:25:04.552 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:04.552 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:04.552 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:04.552 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:37.691984+0000", 2024-09-13T11:25:04.552 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.5656999999999996e-05 2024-09-13T11:25:04.552 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:04.552 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:04.552 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:25:04.552 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:37.692099+0000", 2024-09-13T11:25:04.552 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000114609 2024-09-13T11:25:04.553 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:04.553 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:04.553 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 9", 2024-09-13T11:25:04.553 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:37.692193+0000", 2024-09-13T11:25:04.553 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.4594999999999998e-05 2024-09-13T11:25:04.553 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:04.553 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:04.553 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:25:04.553 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:37.694091+0000", 2024-09-13T11:25:04.553 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0018975730000000001 2024-09-13T11:25:04.553 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:04.553 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:04.553 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:25:04.553 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:37.820957+0000", 2024-09-13T11:25:04.554 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.126866487 2024-09-13T11:25:04.554 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:04.554 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:04.555 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:25:04.555 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:37.821022+0000", 2024-09-13T11:25:04.555 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.4976999999999998e-05 2024-09-13T11:25:04.555 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:04.555 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:04.555 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:04.555 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:37.821070+0000", 2024-09-13T11:25:04.555 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.8003000000000001e-05 2024-09-13T11:25:04.555 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:04.555 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:04.555 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:04.556 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:04.556 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:04.556 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:129921 3.1a 3:5fb9088b:::benchmark_data_smithi177_31116_object129920:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e134)", 2024-09-13T11:25:04.556 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:24:47.177556+0000", 2024-09-13T11:25:04.556 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 17.288556570000001, 2024-09-13T11:25:04.556 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.33804469999999998, 2024-09-13T11:25:04.556 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:04.557 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:04.557 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:04.557 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:25:04.557 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:25:04.557 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:04.557 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 129921 2024-09-13T11:25:04.557 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:04.557 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:04.557 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:04.557 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:04.557 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:47.177556+0000", 2024-09-13T11:25:04.557 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:04.557 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:04.558 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:04.558 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:04.558 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:47.177556+0000", 2024-09-13T11:25:04.558 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:04.558 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:04.558 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:04.558 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:04.558 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:47.177559+0000", 2024-09-13T11:25:04.558 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.0790000000000001e-06 2024-09-13T11:25:04.558 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:04.558 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:04.558 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:04.558 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:47.177589+0000", 2024-09-13T11:25:04.558 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.0173999999999999e-05 2024-09-13T11:25:04.559 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:04.559 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:04.559 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:04.559 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:47.177606+0000", 2024-09-13T11:25:04.559 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.6475999999999999e-05 2024-09-13T11:25:04.559 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:04.559 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:04.559 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:04.559 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:47.177615+0000", 2024-09-13T11:25:04.559 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.5729999999999997e-06 2024-09-13T11:25:04.559 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:04.559 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:04.559 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:04.559 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:47.177680+0000", 2024-09-13T11:25:04.560 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.4349000000000003e-05 2024-09-13T11:25:04.560 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:04.560 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:04.560 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:25:04.560 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:47.177794+0000", 2024-09-13T11:25:04.560 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000114119 2024-09-13T11:25:04.560 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:04.560 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:04.560 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 9", 2024-09-13T11:25:04.560 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:47.177895+0000", 2024-09-13T11:25:04.560 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000101078 2024-09-13T11:25:04.560 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:04.560 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:04.561 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:25:04.561 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:47.179634+0000", 2024-09-13T11:25:04.561 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.001738606 2024-09-13T11:25:04.561 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:04.561 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:04.561 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:25:04.561 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:47.515495+0000", 2024-09-13T11:25:04.561 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.33586143000000002 2024-09-13T11:25:04.561 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:04.561 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:04.561 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:25:04.561 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:47.515551+0000", 2024-09-13T11:25:04.562 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.6048999999999998e-05 2024-09-13T11:25:04.562 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:04.562 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:04.562 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:04.562 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:47.515601+0000", 2024-09-13T11:25:04.562 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.9767e-05 2024-09-13T11:25:04.562 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:04.562 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:04.562 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:04.562 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:04.562 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:04.562 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:130518 3.11 3:8cd77160:::benchmark_data_smithi177_31116_object130517:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e135)", 2024-09-13T11:25:04.562 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:24:47.970431+0000", 2024-09-13T11:25:04.563 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 16.495682022, 2024-09-13T11:25:04.563 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.127398661, 2024-09-13T11:25:04.563 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:04.563 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:04.563 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:04.563 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:25:04.563 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:25:04.563 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:04.563 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 130518 2024-09-13T11:25:04.563 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:04.563 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:04.563 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:04.564 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:04.564 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:47.970431+0000", 2024-09-13T11:25:04.564 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:04.564 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:04.564 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:04.564 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:04.564 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:47.970431+0000", 2024-09-13T11:25:04.564 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:04.564 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:04.564 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:04.564 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:04.564 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:47.970434+0000", 2024-09-13T11:25:04.564 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.8459999999999999e-06 2024-09-13T11:25:04.564 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:04.565 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:04.565 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:04.565 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:47.970557+0000", 2024-09-13T11:25:04.565 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00012314499999999999 2024-09-13T11:25:04.565 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:04.565 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:04.565 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:04.565 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:47.970573+0000", 2024-09-13T11:25:04.565 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.6651e-05 2024-09-13T11:25:04.565 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:04.566 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:04.566 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:04.566 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:47.970584+0000", 2024-09-13T11:25:04.566 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0834e-05 2024-09-13T11:25:04.566 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:04.566 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:04.566 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:04.566 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:47.970676+0000", 2024-09-13T11:25:04.566 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.2139999999999995e-05 2024-09-13T11:25:04.567 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:04.567 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:04.567 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:25:04.567 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:47.971177+0000", 2024-09-13T11:25:04.567 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00050042999999999995 2024-09-13T11:25:04.567 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:04.567 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:04.567 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 11", 2024-09-13T11:25:04.567 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:47.971420+0000", 2024-09-13T11:25:04.567 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00024278900000000001 2024-09-13T11:25:04.568 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:04.568 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:04.568 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:25:04.568 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:47.974699+0000", 2024-09-13T11:25:04.568 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.003279522 2024-09-13T11:25:04.568 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:04.568 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:04.568 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:25:04.568 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:48.097751+0000", 2024-09-13T11:25:04.568 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.12305205299999999 2024-09-13T11:25:04.568 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:04.568 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:04.568 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:25:04.568 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:48.097792+0000", 2024-09-13T11:25:04.569 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.0447000000000001e-05 2024-09-13T11:25:04.569 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:04.569 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:04.569 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:04.569 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:48.097829+0000", 2024-09-13T11:25:04.569 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.7803999999999999e-05 2024-09-13T11:25:04.569 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:04.569 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:04.569 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:04.569 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:04.569 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:04.569 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4665.0:140742 3.1e e144/102 3:79c694a0:::benchmark_data_smithi177_31116_object439:head v 144'6204, mlcod=144'6200)", 2024-09-13T11:25:04.570 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:24:58.321534+0000", 2024-09-13T11:25:04.570 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 6.1445790459999996, 2024-09-13T11:25:04.570 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.23629921500000001, 2024-09-13T11:25:04.570 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:04.570 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:04.570 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:04.570 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:04.570 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:04.570 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:04.570 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.321534+0000", 2024-09-13T11:25:04.570 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:04.570 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:04.570 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:04.570 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:04.571 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.321534+0000", 2024-09-13T11:25:04.571 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:04.571 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:04.571 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:04.571 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:04.571 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.321535+0000", 2024-09-13T11:25:04.571 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.159e-06 2024-09-13T11:25:04.571 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:04.571 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:04.571 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:04.572 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.321545+0000", 2024-09-13T11:25:04.572 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.9219999999999992e-06 2024-09-13T11:25:04.572 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:04.572 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:04.572 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:04.572 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.553719+0000", 2024-09-13T11:25:04.573 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.23217401700000001 2024-09-13T11:25:04.573 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:04.573 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:04.573 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:04.573 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.553734+0000", 2024-09-13T11:25:04.573 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.5344000000000001e-05 2024-09-13T11:25:04.573 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:04.573 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:04.573 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:04.573 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.553858+0000", 2024-09-13T11:25:04.573 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00012357500000000001 2024-09-13T11:25:04.573 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:04.574 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:04.574 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:25:04.574 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.553903+0000", 2024-09-13T11:25:04.574 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.5374999999999998e-05 2024-09-13T11:25:04.574 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:04.574 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:04.574 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:25:04.574 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.557719+0000", 2024-09-13T11:25:04.574 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0038158670000000001 2024-09-13T11:25:04.574 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:04.574 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:04.574 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:04.574 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.557833+0000", 2024-09-13T11:25:04.574 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000113956 2024-09-13T11:25:04.575 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:04.575 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:04.575 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:04.575 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:04.575 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:04.575 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4665.0:140796 3.1e e144/102 3:7987a652:::benchmark_data_smithi177_31116_object493:head v 144'6205, mlcod=144'6202)", 2024-09-13T11:25:04.575 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:24:58.553657+0000", 2024-09-13T11:25:04.575 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 5.9124560700000002, 2024-09-13T11:25:04.575 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.25308323100000002, 2024-09-13T11:25:04.575 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:04.575 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:04.575 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:04.575 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:04.576 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:04.576 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:04.576 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.553657+0000", 2024-09-13T11:25:04.576 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:04.576 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:04.576 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:04.576 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:04.576 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.553657+0000", 2024-09-13T11:25:04.576 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:04.576 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:04.576 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:04.576 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:04.576 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.553660+0000", 2024-09-13T11:25:04.577 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.7519999999999998e-06 2024-09-13T11:25:04.577 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:04.577 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:04.577 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:04.577 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.553678+0000", 2024-09-13T11:25:04.577 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.7156e-05 2024-09-13T11:25:04.577 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:04.577 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:04.577 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:04.577 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.804342+0000", 2024-09-13T11:25:04.577 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.25066436199999997 2024-09-13T11:25:04.577 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:04.577 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:04.577 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:04.578 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.804353+0000", 2024-09-13T11:25:04.578 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.1147e-05 2024-09-13T11:25:04.578 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:04.578 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:04.578 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:04.578 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.804433+0000", 2024-09-13T11:25:04.578 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.9586999999999995e-05 2024-09-13T11:25:04.578 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:04.578 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:04.578 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:25:04.579 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.804457+0000", 2024-09-13T11:25:04.579 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.4108e-05 2024-09-13T11:25:04.579 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:04.579 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:04.579 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:25:04.579 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.806684+0000", 2024-09-13T11:25:04.579 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0022271309999999998 2024-09-13T11:25:04.579 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:04.579 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:04.579 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:04.579 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.806740+0000", 2024-09-13T11:25:04.579 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.5988e-05 2024-09-13T11:25:04.579 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:04.579 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:04.580 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:04.580 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:04.580 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:04.580 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-13T11:25:04.582 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-09-13T11:25:04.582 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-13T11:25:04.583 INFO:teuthology.orchestra.run.smithi001.stderr:no valid command found; 10 closest matches: 2024-09-13T11:25:04.583 INFO:teuthology.orchestra.run.smithi001.stderr:0 2024-09-13T11:25:04.583 INFO:teuthology.orchestra.run.smithi001.stderr:1 2024-09-13T11:25:04.583 INFO:teuthology.orchestra.run.smithi001.stderr:2 2024-09-13T11:25:04.583 INFO:teuthology.orchestra.run.smithi001.stderr:abort 2024-09-13T11:25:04.583 INFO:teuthology.orchestra.run.smithi001.stderr:assert 2024-09-13T11:25:04.583 INFO:teuthology.orchestra.run.smithi001.stderr:bench [] [] [] [] 2024-09-13T11:25:04.583 INFO:teuthology.orchestra.run.smithi001.stderr:bluefs debug_inject_read_zeros 2024-09-13T11:25:04.583 INFO:teuthology.orchestra.run.smithi001.stderr:bluefs files list 2024-09-13T11:25:04.584 INFO:teuthology.orchestra.run.smithi001.stderr:bluefs stats 2024-09-13T11:25:04.584 INFO:teuthology.orchestra.run.smithi001.stderr:bluestore allocator dump block 2024-09-13T11:25:04.584 INFO:teuthology.orchestra.run.smithi001.stderr:Error EINVAL: invalid command 2024-09-13T11:25:04.584 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-13T11:25:04.584 INFO:tasks.ceph.osd.4.smithi001.stderr:2024-09-13T11:25:04.579+0000 7ff8e0218640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26310) UID: 0 2024-09-13T11:25:04.682 INFO:tasks.ceph.osd.10.smithi162.stderr:2024-09-13T11:25:04.680+0000 7fb59c9f0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26317) UID: 0 2024-09-13T11:25:04.756 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-13T11:25:04.756 INFO:teuthology.orchestra.run.smithi001.stdout: "ops": [ 2024-09-13T11:25:04.756 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:04.757 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:148173 3.26 3:6736e8af:::benchmark_data_smithi177_31116_object7870:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e148)", 2024-09-13T11:25:04.757 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:25:04.740663+0000", 2024-09-13T11:25:04.757 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 0.01547627, 2024-09-13T11:25:04.757 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.015503179000000001, 2024-09-13T11:25:04.757 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:04.757 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:04.757 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "waiting for sub ops", 2024-09-13T11:25:04.757 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:25:04.758 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:25:04.758 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:04.758 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 148173 2024-09-13T11:25:04.758 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:04.758 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:04.758 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:04.758 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:04.758 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:04.740663+0000", 2024-09-13T11:25:04.758 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:04.758 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:04.758 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:04.758 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:04.759 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:04.740663+0000", 2024-09-13T11:25:04.759 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:04.759 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:04.759 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:04.759 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:04.759 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:04.740666+0000", 2024-09-13T11:25:04.759 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.367e-06 2024-09-13T11:25:04.759 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:04.759 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:04.759 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:04.759 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:04.740672+0000", 2024-09-13T11:25:04.759 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.0610000000000001e-06 2024-09-13T11:25:04.759 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:04.760 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:04.760 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:04.760 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:04.740686+0000", 2024-09-13T11:25:04.760 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.4243999999999999e-05 2024-09-13T11:25:04.760 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:04.760 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:04.760 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:04.760 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:04.740694+0000", 2024-09-13T11:25:04.760 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.0339999999999997e-06 2024-09-13T11:25:04.760 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:04.760 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:04.760 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:04.760 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:04.740752+0000", 2024-09-13T11:25:04.761 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.7321000000000002e-05 2024-09-13T11:25:04.761 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:04.761 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:04.761 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:25:04.761 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:04.740836+0000", 2024-09-13T11:25:04.761 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.4579000000000005e-05 2024-09-13T11:25:04.761 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:04.761 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:04.761 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 12", 2024-09-13T11:25:04.761 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:04.740897+0000", 2024-09-13T11:25:04.761 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.0602e-05 2024-09-13T11:25:04.761 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:04.761 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:04.761 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:25:04.762 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:04.742060+0000", 2024-09-13T11:25:04.762 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.001163692 2024-09-13T11:25:04.762 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:04.762 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:04.762 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:04.762 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:04.762 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:04.762 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:148192 3.26 3:66dc7a0a:::benchmark_data_smithi177_31116_object7889:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e148)", 2024-09-13T11:25:04.762 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:25:04.755750+0000", 2024-09-13T11:25:04.762 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 0.000389312, 2024-09-13T11:25:04.762 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00046170899999999998, 2024-09-13T11:25:04.762 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:04.762 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:04.762 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "waiting for sub ops", 2024-09-13T11:25:04.763 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:25:04.763 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:25:04.763 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:04.763 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 148192 2024-09-13T11:25:04.763 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:04.763 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:04.763 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:04.763 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:04.763 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:04.755750+0000", 2024-09-13T11:25:04.763 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:04.763 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:04.763 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:04.763 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:04.763 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:04.755750+0000", 2024-09-13T11:25:04.764 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:04.764 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:04.764 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:04.764 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:04.764 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:04.755752+0000", 2024-09-13T11:25:04.764 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.8360000000000001e-06 2024-09-13T11:25:04.764 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:04.764 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:04.764 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:04.764 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:04.755758+0000", 2024-09-13T11:25:04.764 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.5999999999999997e-06 2024-09-13T11:25:04.764 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:04.764 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:04.764 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:04.765 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:04.755771+0000", 2024-09-13T11:25:04.765 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.3427e-05 2024-09-13T11:25:04.765 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:04.765 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:04.765 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:04.765 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:04.755778+0000", 2024-09-13T11:25:04.765 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.5119999999999997e-06 2024-09-13T11:25:04.765 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:04.765 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:04.765 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:04.765 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:04.755826+0000", 2024-09-13T11:25:04.765 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.8401e-05 2024-09-13T11:25:04.765 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:04.766 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:04.766 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:25:04.766 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:04.755923+0000", 2024-09-13T11:25:04.766 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.6367999999999996e-05 2024-09-13T11:25:04.766 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:04.766 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:04.766 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 12", 2024-09-13T11:25:04.766 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:04.755983+0000", 2024-09-13T11:25:04.766 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.0684999999999999e-05 2024-09-13T11:25:04.766 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:04.766 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:04.766 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:04.766 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:04.766 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-09-13T11:25:04.766 INFO:teuthology.orchestra.run.smithi001.stdout: "num_ops": 2 2024-09-13T11:25:04.767 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-13T11:25:04.775 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-09-13T11:25:04.783 INFO:tasks.ceph.osd.1.smithi092.stderr:2024-09-13T11:25:04.778+0000 7fe7789c5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33718) UID: 0 2024-09-13T11:25:04.884 INFO:tasks.ceph.osd.13.smithi092.stderr:2024-09-13T11:25:04.878+0000 7f3e5afea640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26350) UID: 0 2024-09-13T11:25:04.909 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-13T11:25:04.909 INFO:teuthology.orchestra.run.smithi001.stdout: "ops": [], 2024-09-13T11:25:04.909 INFO:teuthology.orchestra.run.smithi001.stdout: "complaint_time": 30, 2024-09-13T11:25:04.909 INFO:teuthology.orchestra.run.smithi001.stdout: "num_blocked_ops": 0 2024-09-13T11:25:04.909 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-13T11:25:04.926 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-09-13T11:25:04.985 INFO:tasks.ceph.osd.2.smithi162.stderr:2024-09-13T11:25:04.980+0000 7fe92b35a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26313) UID: 0 2024-09-13T11:25:05.047 INFO:tasks.ceph.osd.12.smithi001.stderr:2024-09-13T11:25:05.043+0000 7f891f10f640 -1 osd.12 148 handle_osd_map injecting map crc failure 2024-09-13T11:25:05.047 INFO:teuthology.orchestra.run.smithi001.stderr:nodeep-scrub is set 2024-09-13T11:25:05.067 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-13T11:25:05.067 INFO:teuthology.orchestra.run.smithi001.stdout: "size": 20, 2024-09-13T11:25:05.067 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 600, 2024-09-13T11:25:05.067 INFO:teuthology.orchestra.run.smithi001.stdout: "ops": [ 2024-09-13T11:25:05.067 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.068 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:54656 3.1e 3.6a49163e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e75)", 2024-09-13T11:25:05.068 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:30.979993+0000", 2024-09-13T11:25:05.068 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 94.085744817000005, 2024-09-13T11:25:05.068 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.9269557150000001, 2024-09-13T11:25:05.068 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:05.068 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:05.068 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-13T11:25:05.068 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:25:05.068 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:25:05.068 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:05.068 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 54656 2024-09-13T11:25:05.068 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.068 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:05.068 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.069 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:05.069 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:30.979993+0000", 2024-09-13T11:25:05.069 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:05.069 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.069 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.069 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:05.069 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:30.979993+0000", 2024-09-13T11:25:05.069 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:05.069 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.069 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.069 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:05.069 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:30.979997+0000", 2024-09-13T11:25:05.069 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.0319999999999997e-06 2024-09-13T11:25:05.069 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.070 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.070 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:05.070 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:30.980057+0000", 2024-09-13T11:25:05.070 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.0341999999999999e-05 2024-09-13T11:25:05.070 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.070 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.070 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:05.070 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:30.980094+0000", 2024-09-13T11:25:05.070 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.6770000000000002e-05 2024-09-13T11:25:05.070 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.070 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.070 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:05.070 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:30.980113+0000", 2024-09-13T11:25:05.070 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.8791999999999999e-05 2024-09-13T11:25:05.071 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.071 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.071 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:05.071 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.097773+0000", 2024-09-13T11:25:05.071 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.117660379 2024-09-13T11:25:05.071 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.071 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.071 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for peered", 2024-09-13T11:25:05.071 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.097784+0000", 2024-09-13T11:25:05.071 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.1025e-05 2024-09-13T11:25:05.071 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.071 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.071 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:05.072 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.906937+0000", 2024-09-13T11:25:05.072 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.8091523789999999 2024-09-13T11:25:05.072 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.072 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.072 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:05.072 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.906949+0000", 2024-09-13T11:25:05.072 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.1996e-05 2024-09-13T11:25:05.072 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:05.072 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:05.072 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:05.072 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.072 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.072 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDPGPushReply(3.17 75/70 [PushReplyOp(3:e82800c8:::benchmark_data_smithi177_31116_object13909:head)])", 2024-09-13T11:25:05.072 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:31.632344+0000", 2024-09-13T11:25:05.073 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 93.433393961999997, 2024-09-13T11:25:05.073 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.28592102899999999, 2024-09-13T11:25:05.073 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:05.073 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:05.073 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-13T11:25:05.073 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:05.073 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.073 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:05.073 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.632344+0000", 2024-09-13T11:25:05.073 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:05.073 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.073 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.073 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:05.073 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.632344+0000", 2024-09-13T11:25:05.074 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:05.074 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.074 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.074 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:05.074 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.632346+0000", 2024-09-13T11:25:05.074 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.8819999999999999e-06 2024-09-13T11:25:05.074 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.074 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.074 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:05.074 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.632353+0000", 2024-09-13T11:25:05.074 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.5059999999999998e-06 2024-09-13T11:25:05.075 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.075 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.075 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:05.075 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.917361+0000", 2024-09-13T11:25:05.075 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.285008231 2024-09-13T11:25:05.075 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.075 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.075 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:05.075 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.917372+0000", 2024-09-13T11:25:05.075 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0764999999999999e-05 2024-09-13T11:25:05.075 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.075 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.076 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:05.076 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.917458+0000", 2024-09-13T11:25:05.076 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.6136000000000003e-05 2024-09-13T11:25:05.076 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.076 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.076 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:05.076 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.918265+0000", 2024-09-13T11:25:05.076 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00080650899999999996 2024-09-13T11:25:05.076 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:05.076 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:05.076 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:05.076 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.076 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.076 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:54720 3.1e 3.a944303e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-13T11:25:05.077 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:31.973315+0000", 2024-09-13T11:25:05.077 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 93.092422581999998, 2024-09-13T11:25:05.077 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.93366527200000005, 2024-09-13T11:25:05.077 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:05.077 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:05.077 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-13T11:25:05.077 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:25:05.077 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:25:05.077 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:05.077 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 54720 2024-09-13T11:25:05.077 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.077 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:05.077 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.077 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:05.077 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.973315+0000", 2024-09-13T11:25:05.078 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:05.078 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.078 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.078 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:05.078 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.973315+0000", 2024-09-13T11:25:05.078 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:05.078 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.078 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.078 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:05.078 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.973318+0000", 2024-09-13T11:25:05.078 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.5830000000000001e-06 2024-09-13T11:25:05.078 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.078 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.078 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:05.079 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.973465+0000", 2024-09-13T11:25:05.079 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000146816 2024-09-13T11:25:05.079 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.079 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.079 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:05.079 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.973478+0000", 2024-09-13T11:25:05.079 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2931e-05 2024-09-13T11:25:05.079 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.079 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.079 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:05.079 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.973483+0000", 2024-09-13T11:25:05.079 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.8969999999999996e-06 2024-09-13T11:25:05.079 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.079 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.080 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:05.080 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.973548+0000", 2024-09-13T11:25:05.080 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.4771999999999996e-05 2024-09-13T11:25:05.080 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.080 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.080 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for peered", 2024-09-13T11:25:05.080 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.973557+0000", 2024-09-13T11:25:05.080 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.0359999999999995e-06 2024-09-13T11:25:05.080 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.080 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.080 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:05.080 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.906974+0000", 2024-09-13T11:25:05.080 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.93341706000000002 2024-09-13T11:25:05.081 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.081 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.081 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:05.081 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.906980+0000", 2024-09-13T11:25:05.081 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.1770000000000003e-06 2024-09-13T11:25:05.081 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:05.081 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:05.081 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:05.081 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.081 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.081 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:54734 3.1e 3.d7f6f7e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-13T11:25:05.081 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:31.983342+0000", 2024-09-13T11:25:05.081 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 93.082395778999995, 2024-09-13T11:25:05.081 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.92367058599999996, 2024-09-13T11:25:05.082 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:05.082 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:05.082 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-13T11:25:05.082 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:25:05.082 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:25:05.082 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:05.082 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 54734 2024-09-13T11:25:05.082 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.082 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:05.082 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.082 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:05.082 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.983342+0000", 2024-09-13T11:25:05.082 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:05.083 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.083 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.083 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:05.083 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.983342+0000", 2024-09-13T11:25:05.083 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:05.083 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.083 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.083 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:05.083 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.983345+0000", 2024-09-13T11:25:05.083 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.6220000000000001e-06 2024-09-13T11:25:05.083 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.083 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.083 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:05.083 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.983438+0000", 2024-09-13T11:25:05.084 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.3517000000000006e-05 2024-09-13T11:25:05.084 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.084 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.084 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:05.084 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.983451+0000", 2024-09-13T11:25:05.084 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2527999999999999e-05 2024-09-13T11:25:05.084 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.084 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.084 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:05.084 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.983458+0000", 2024-09-13T11:25:05.084 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.6380000000000004e-06 2024-09-13T11:25:05.084 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.085 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.085 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:05.085 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.983521+0000", 2024-09-13T11:25:05.085 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.2729000000000002e-05 2024-09-13T11:25:05.085 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.085 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.085 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for peered", 2024-09-13T11:25:05.085 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.983548+0000", 2024-09-13T11:25:05.085 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.6537999999999999e-05 2024-09-13T11:25:05.085 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.085 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.085 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:05.085 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.907005+0000", 2024-09-13T11:25:05.085 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.923456991 2024-09-13T11:25:05.086 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.086 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.086 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:05.086 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.907013+0000", 2024-09-13T11:25:05.086 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.0229999999999992e-06 2024-09-13T11:25:05.086 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:05.086 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:05.086 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:05.086 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.086 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.086 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:54760 3.1e 3.8c32c13e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-13T11:25:05.086 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:31.999685+0000", 2024-09-13T11:25:05.086 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 93.066052546999998, 2024-09-13T11:25:05.087 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.90735639599999995, 2024-09-13T11:25:05.087 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:05.087 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:05.087 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-13T11:25:05.087 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:25:05.087 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:25:05.087 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:05.087 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 54760 2024-09-13T11:25:05.087 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.087 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:05.087 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.087 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:05.087 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.999685+0000", 2024-09-13T11:25:05.087 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:05.088 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.088 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.088 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:05.088 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.999685+0000", 2024-09-13T11:25:05.088 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:05.088 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.088 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.088 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:05.088 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.999690+0000", 2024-09-13T11:25:05.088 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.7879999999999997e-06 2024-09-13T11:25:05.088 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.088 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.089 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:05.089 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.999729+0000", 2024-09-13T11:25:05.089 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.9032000000000002e-05 2024-09-13T11:25:05.089 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.089 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.089 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:05.089 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.999741+0000", 2024-09-13T11:25:05.089 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2136999999999999e-05 2024-09-13T11:25:05.090 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.090 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.090 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:05.090 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.999748+0000", 2024-09-13T11:25:05.090 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.9120000000000001e-06 2024-09-13T11:25:05.090 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.090 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.090 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:05.090 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.999803+0000", 2024-09-13T11:25:05.090 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.5064e-05 2024-09-13T11:25:05.090 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.090 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.090 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for peered", 2024-09-13T11:25:05.090 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.999815+0000", 2024-09-13T11:25:05.091 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2041e-05 2024-09-13T11:25:05.091 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.091 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.091 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:05.091 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.907034+0000", 2024-09-13T11:25:05.092 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.90721858200000005 2024-09-13T11:25:05.092 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.092 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.092 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:05.092 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.907042+0000", 2024-09-13T11:25:05.092 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.8399999999999995e-06 2024-09-13T11:25:05.092 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:05.092 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:05.092 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:05.092 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.092 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.093 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:54768 3.1e 3.6dad499e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-13T11:25:05.093 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:32.003482+0000", 2024-09-13T11:25:05.093 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 93.062255897, 2024-09-13T11:25:05.093 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.90358804699999995, 2024-09-13T11:25:05.093 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:05.093 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:05.093 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-13T11:25:05.093 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:25:05.093 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:25:05.093 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:05.093 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 54768 2024-09-13T11:25:05.094 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.094 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:05.094 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.094 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:05.094 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.003482+0000", 2024-09-13T11:25:05.094 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:05.094 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.094 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.094 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:05.094 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.003482+0000", 2024-09-13T11:25:05.094 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:05.094 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.094 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.095 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:05.095 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.003486+0000", 2024-09-13T11:25:05.095 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.1030000000000004e-06 2024-09-13T11:25:05.095 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.095 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.095 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:05.095 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.003523+0000", 2024-09-13T11:25:05.095 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.6732e-05 2024-09-13T11:25:05.095 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.095 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.095 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:05.095 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.003535+0000", 2024-09-13T11:25:05.095 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2520999999999999e-05 2024-09-13T11:25:05.095 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.096 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.096 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:05.096 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.003541+0000", 2024-09-13T11:25:05.096 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.8690000000000001e-06 2024-09-13T11:25:05.096 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.096 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.096 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:05.096 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.003593+0000", 2024-09-13T11:25:05.096 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.1786000000000002e-05 2024-09-13T11:25:05.096 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.096 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.096 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for peered", 2024-09-13T11:25:05.096 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.003602+0000", 2024-09-13T11:25:05.096 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.9679999999999995e-06 2024-09-13T11:25:05.097 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.097 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.097 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:05.097 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.907063+0000", 2024-09-13T11:25:05.097 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.90346116600000004 2024-09-13T11:25:05.097 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.097 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.097 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:05.097 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.907070+0000", 2024-09-13T11:25:05.097 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.9020000000000002e-06 2024-09-13T11:25:05.097 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:05.097 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:05.097 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:05.097 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.098 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.098 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:54785 3.1e 3.d32f1ede (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-13T11:25:05.098 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:32.016238+0000", 2024-09-13T11:25:05.098 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 93.049499686999994, 2024-09-13T11:25:05.098 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.89086243399999998, 2024-09-13T11:25:05.098 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:05.098 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:05.098 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-13T11:25:05.098 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:25:05.098 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:25:05.098 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:05.098 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 54785 2024-09-13T11:25:05.098 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.099 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:05.099 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.099 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:05.099 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.016238+0000", 2024-09-13T11:25:05.099 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:05.099 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.099 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.099 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:05.099 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.016238+0000", 2024-09-13T11:25:05.099 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:05.099 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.099 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.099 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:05.099 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.016241+0000", 2024-09-13T11:25:05.100 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.6469999999999998e-06 2024-09-13T11:25:05.100 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.100 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.100 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:05.100 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.016271+0000", 2024-09-13T11:25:05.100 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.0505999999999999e-05 2024-09-13T11:25:05.100 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.100 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.100 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:05.100 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.016287+0000", 2024-09-13T11:25:05.100 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.6073999999999999e-05 2024-09-13T11:25:05.100 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.100 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.100 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:05.101 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.016295+0000", 2024-09-13T11:25:05.101 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.0590000000000003e-06 2024-09-13T11:25:05.101 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.101 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.101 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:05.101 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.016373+0000", 2024-09-13T11:25:05.101 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.7634999999999998e-05 2024-09-13T11:25:05.101 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.101 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.101 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for peered", 2024-09-13T11:25:05.101 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.016383+0000", 2024-09-13T11:25:05.101 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0246e-05 2024-09-13T11:25:05.101 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.101 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.102 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:05.102 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.907092+0000", 2024-09-13T11:25:05.102 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.89070849200000002 2024-09-13T11:25:05.102 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.102 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.102 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:05.102 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.907101+0000", 2024-09-13T11:25:05.102 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.7749999999999998e-06 2024-09-13T11:25:05.102 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:05.102 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:05.102 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:05.102 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.102 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.102 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:54811 3.1e 3.3754367e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-13T11:25:05.103 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:32.045424+0000", 2024-09-13T11:25:05.103 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 93.020314036000002, 2024-09-13T11:25:05.103 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.86170719399999995, 2024-09-13T11:25:05.103 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:05.103 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:05.103 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-13T11:25:05.103 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:25:05.103 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:25:05.103 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:05.103 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 54811 2024-09-13T11:25:05.103 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.103 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:05.103 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.103 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:05.104 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.045424+0000", 2024-09-13T11:25:05.104 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:05.104 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.104 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.104 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:05.104 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.045424+0000", 2024-09-13T11:25:05.104 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:05.104 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.104 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.104 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:05.104 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.045426+0000", 2024-09-13T11:25:05.104 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.0260000000000002e-06 2024-09-13T11:25:05.105 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.105 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.105 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:05.105 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.045556+0000", 2024-09-13T11:25:05.105 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00013003099999999999 2024-09-13T11:25:05.105 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.106 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.106 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:05.106 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.045568+0000", 2024-09-13T11:25:05.106 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.1727e-05 2024-09-13T11:25:05.106 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.106 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.106 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:05.106 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.045576+0000", 2024-09-13T11:25:05.106 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.6459999999999999e-06 2024-09-13T11:25:05.106 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.106 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.106 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:05.106 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.045650+0000", 2024-09-13T11:25:05.106 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.4000999999999998e-05 2024-09-13T11:25:05.107 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.107 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.107 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for peered", 2024-09-13T11:25:05.107 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.045658+0000", 2024-09-13T11:25:05.107 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.3869999999999997e-06 2024-09-13T11:25:05.107 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.107 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.107 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:05.107 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.907123+0000", 2024-09-13T11:25:05.107 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.86146508399999999 2024-09-13T11:25:05.107 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.107 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.107 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:05.107 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.907131+0000", 2024-09-13T11:25:05.108 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.2919999999999996e-06 2024-09-13T11:25:05.108 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:05.108 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:05.108 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:05.108 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.108 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.108 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:54817 3.1e 3.36dd2fde (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-13T11:25:05.108 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:32.050108+0000", 2024-09-13T11:25:05.108 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 93.015630143999999, 2024-09-13T11:25:05.108 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.85705549199999997, 2024-09-13T11:25:05.108 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:05.108 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:05.108 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-13T11:25:05.108 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:25:05.109 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:25:05.109 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:05.109 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 54817 2024-09-13T11:25:05.109 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.109 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:05.109 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.109 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:05.109 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.050108+0000", 2024-09-13T11:25:05.109 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:05.109 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.109 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.109 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:05.109 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.050108+0000", 2024-09-13T11:25:05.109 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:05.110 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.110 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.110 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:05.110 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.050110+0000", 2024-09-13T11:25:05.110 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.0209999999999998e-06 2024-09-13T11:25:05.110 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.110 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.110 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:05.110 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.050166+0000", 2024-09-13T11:25:05.110 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.6026000000000002e-05 2024-09-13T11:25:05.110 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.110 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.110 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:05.110 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.050180+0000", 2024-09-13T11:25:05.111 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.4307e-05 2024-09-13T11:25:05.111 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.111 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.111 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:05.111 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.050188+0000", 2024-09-13T11:25:05.111 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.5410000000000003e-06 2024-09-13T11:25:05.111 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.111 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.111 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:05.111 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.050260+0000", 2024-09-13T11:25:05.111 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.2157999999999996e-05 2024-09-13T11:25:05.111 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.111 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.111 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for peered", 2024-09-13T11:25:05.112 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.050267+0000", 2024-09-13T11:25:05.112 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.0190000000000001e-06 2024-09-13T11:25:05.112 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.112 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.112 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:05.112 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.907155+0000", 2024-09-13T11:25:05.112 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.85688824399999997 2024-09-13T11:25:05.112 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.112 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.112 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:05.112 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.907163+0000", 2024-09-13T11:25:05.112 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.1759999999999993e-06 2024-09-13T11:25:05.112 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:05.112 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:05.113 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:05.113 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.113 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.113 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:54818 3.1e 3.5cfcc69e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-13T11:25:05.113 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:32.051374+0000", 2024-09-13T11:25:05.113 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 93.014363728000006, 2024-09-13T11:25:05.113 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.85581781000000001, 2024-09-13T11:25:05.113 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:05.113 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:05.113 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-13T11:25:05.113 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:25:05.113 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:25:05.113 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:05.114 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 54818 2024-09-13T11:25:05.114 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.114 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:05.114 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.114 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:05.114 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.051374+0000", 2024-09-13T11:25:05.114 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:05.114 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.114 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.114 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:05.114 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.051374+0000", 2024-09-13T11:25:05.114 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:05.114 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.115 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.115 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:05.115 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.051375+0000", 2024-09-13T11:25:05.115 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.4440000000000001e-06 2024-09-13T11:25:05.115 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.115 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.115 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:05.115 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.051426+0000", 2024-09-13T11:25:05.115 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.0809000000000001e-05 2024-09-13T11:25:05.115 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.115 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.115 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:05.115 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.051441+0000", 2024-09-13T11:25:05.115 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.4460999999999999e-05 2024-09-13T11:25:05.116 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.116 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.116 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:05.116 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.051449+0000", 2024-09-13T11:25:05.116 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.4880000000000007e-06 2024-09-13T11:25:05.116 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.116 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.116 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:05.116 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.051499+0000", 2024-09-13T11:25:05.116 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.9691999999999998e-05 2024-09-13T11:25:05.116 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.116 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.116 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for peered", 2024-09-13T11:25:05.117 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.051505+0000", 2024-09-13T11:25:05.117 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.2029999999999997e-06 2024-09-13T11:25:05.117 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.117 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.117 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:05.117 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.907184+0000", 2024-09-13T11:25:05.117 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.85567906599999999 2024-09-13T11:25:05.117 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.117 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.117 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:05.117 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.907192+0000", 2024-09-13T11:25:05.117 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.6469999999999998e-06 2024-09-13T11:25:05.117 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:05.117 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:05.118 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:05.118 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.118 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.118 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:54828 3.1e 3.e920c11e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-13T11:25:05.118 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:32.063344+0000", 2024-09-13T11:25:05.118 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 93.002393300999998, 2024-09-13T11:25:05.118 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.84387899099999997, 2024-09-13T11:25:05.118 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:05.118 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:05.118 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-13T11:25:05.118 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:25:05.118 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:25:05.118 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:05.118 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 54828 2024-09-13T11:25:05.119 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.119 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:05.119 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.119 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:05.119 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.063344+0000", 2024-09-13T11:25:05.119 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:05.119 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.119 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.119 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:05.119 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.063344+0000", 2024-09-13T11:25:05.119 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:05.119 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.119 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.119 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:05.119 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.063347+0000", 2024-09-13T11:25:05.120 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.2299999999999998e-06 2024-09-13T11:25:05.120 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.120 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.120 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:05.120 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.063384+0000", 2024-09-13T11:25:05.120 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.7290999999999999e-05 2024-09-13T11:25:05.120 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.120 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.120 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:05.120 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.063397+0000", 2024-09-13T11:25:05.120 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2955999999999999e-05 2024-09-13T11:25:05.120 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.120 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.121 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:05.121 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.063404+0000", 2024-09-13T11:25:05.121 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.4889999999999998e-06 2024-09-13T11:25:05.121 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.121 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.121 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:05.121 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.063470+0000", 2024-09-13T11:25:05.121 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.5516000000000001e-05 2024-09-13T11:25:05.121 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.121 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.121 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for peered", 2024-09-13T11:25:05.121 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.063482+0000", 2024-09-13T11:25:05.121 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2065000000000001e-05 2024-09-13T11:25:05.122 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.122 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.122 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:05.122 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.907216+0000", 2024-09-13T11:25:05.122 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.84373388900000001 2024-09-13T11:25:05.122 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.122 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.122 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:05.122 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.907223+0000", 2024-09-13T11:25:05.122 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.5549999999999997e-06 2024-09-13T11:25:05.122 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:05.122 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:05.122 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:05.123 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.123 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.123 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:54829 3.1e 3.192f62be (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-13T11:25:05.123 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:32.065642+0000", 2024-09-13T11:25:05.123 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 93.000096232999994, 2024-09-13T11:25:05.123 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.84161483100000001, 2024-09-13T11:25:05.123 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:05.123 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:05.123 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-13T11:25:05.123 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:25:05.123 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:25:05.123 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:05.123 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 54829 2024-09-13T11:25:05.123 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.123 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:05.124 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.124 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:05.124 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.065642+0000", 2024-09-13T11:25:05.124 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:05.124 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.124 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.124 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:05.124 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.065642+0000", 2024-09-13T11:25:05.124 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:05.124 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.124 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.124 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:05.124 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.065644+0000", 2024-09-13T11:25:05.125 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.1780000000000002e-06 2024-09-13T11:25:05.125 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.125 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.125 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:05.125 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.065695+0000", 2024-09-13T11:25:05.125 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.1632000000000003e-05 2024-09-13T11:25:05.125 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.125 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.125 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:05.125 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.065711+0000", 2024-09-13T11:25:05.125 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.5522000000000001e-05 2024-09-13T11:25:05.125 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.125 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.125 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:05.126 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.065719+0000", 2024-09-13T11:25:05.126 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.3480000000000004e-06 2024-09-13T11:25:05.126 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.126 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.126 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:05.126 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.065785+0000", 2024-09-13T11:25:05.126 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.5627e-05 2024-09-13T11:25:05.126 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.126 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.126 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for peered", 2024-09-13T11:25:05.126 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.065797+0000", 2024-09-13T11:25:05.126 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.1766000000000001e-05 2024-09-13T11:25:05.126 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.127 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.127 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:05.127 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.907248+0000", 2024-09-13T11:25:05.127 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.84145150300000005 2024-09-13T11:25:05.127 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.127 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.127 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:05.127 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.907256+0000", 2024-09-13T11:25:05.127 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.2549999999999997e-06 2024-09-13T11:25:05.127 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:05.127 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:05.127 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:05.127 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.127 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.128 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:54830 3.1e 3.b31554de (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-13T11:25:05.128 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:32.068259+0000", 2024-09-13T11:25:05.128 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 92.997478365000006, 2024-09-13T11:25:05.128 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.83902489599999996, 2024-09-13T11:25:05.128 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:05.128 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:05.128 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-13T11:25:05.128 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:25:05.128 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:25:05.128 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:05.128 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 54830 2024-09-13T11:25:05.128 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.128 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:05.129 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.129 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:05.129 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.068259+0000", 2024-09-13T11:25:05.129 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:05.129 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.129 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.129 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:05.129 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.068259+0000", 2024-09-13T11:25:05.129 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:05.129 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.129 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.129 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:05.129 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.068262+0000", 2024-09-13T11:25:05.130 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.1799999999999999e-06 2024-09-13T11:25:05.130 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.130 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.130 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:05.130 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.068301+0000", 2024-09-13T11:25:05.130 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.9289999999999998e-05 2024-09-13T11:25:05.130 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.130 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.130 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:05.130 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.068314+0000", 2024-09-13T11:25:05.130 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.3502999999999999e-05 2024-09-13T11:25:05.130 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.130 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.130 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:05.131 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.068324+0000", 2024-09-13T11:25:05.131 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.1630000000000002e-06 2024-09-13T11:25:05.131 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.131 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.131 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:05.131 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.068375+0000", 2024-09-13T11:25:05.131 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.1805000000000003e-05 2024-09-13T11:25:05.131 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.131 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.131 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for peered", 2024-09-13T11:25:05.131 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.068382+0000", 2024-09-13T11:25:05.131 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.224e-06 2024-09-13T11:25:05.131 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.132 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.132 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:05.132 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.907277+0000", 2024-09-13T11:25:05.132 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.83889509500000004 2024-09-13T11:25:05.132 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.132 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.132 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:05.132 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.907284+0000", 2024-09-13T11:25:05.132 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.6359999999999994e-06 2024-09-13T11:25:05.132 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:05.132 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:05.132 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:05.132 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.132 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.133 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:54846 3.1e 3.742e049e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-13T11:25:05.133 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:32.153567+0000", 2024-09-13T11:25:05.133 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 92.912171141000002, 2024-09-13T11:25:05.133 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.75378642399999995, 2024-09-13T11:25:05.133 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:05.133 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:05.133 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-13T11:25:05.133 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:25:05.133 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:25:05.133 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:05.133 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 54846 2024-09-13T11:25:05.133 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.133 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:05.134 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.134 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:05.134 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.153567+0000", 2024-09-13T11:25:05.134 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:05.134 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.134 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.134 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:05.134 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.153567+0000", 2024-09-13T11:25:05.134 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:05.134 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.134 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.134 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:05.134 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.153569+0000", 2024-09-13T11:25:05.134 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.4660000000000002e-06 2024-09-13T11:25:05.135 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.135 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.135 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:05.135 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.153619+0000", 2024-09-13T11:25:05.135 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.9617000000000003e-05 2024-09-13T11:25:05.135 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.135 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.135 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:05.135 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.153634+0000", 2024-09-13T11:25:05.135 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.4904e-05 2024-09-13T11:25:05.135 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.135 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.135 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:05.135 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.153642+0000", 2024-09-13T11:25:05.136 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.5599999999999994e-06 2024-09-13T11:25:05.136 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.136 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.136 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:05.136 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.153721+0000", 2024-09-13T11:25:05.136 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.8603999999999995e-05 2024-09-13T11:25:05.136 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.136 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.136 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for peered", 2024-09-13T11:25:05.136 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.153730+0000", 2024-09-13T11:25:05.136 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.3840000000000003e-06 2024-09-13T11:25:05.136 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.136 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.136 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:05.137 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.907346+0000", 2024-09-13T11:25:05.137 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.75361563099999995 2024-09-13T11:25:05.137 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.137 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.137 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:05.137 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.907353+0000", 2024-09-13T11:25:05.137 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.2579999999999998e-06 2024-09-13T11:25:05.137 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:05.137 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:05.137 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:05.137 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.137 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.137 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:54888 3.1e 3.43e14ede (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-13T11:25:05.137 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:32.423064+0000", 2024-09-13T11:25:05.137 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 92.642673470000005, 2024-09-13T11:25:05.138 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.48433585299999998, 2024-09-13T11:25:05.138 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:05.138 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:05.138 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-13T11:25:05.138 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:25:05.138 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:25:05.138 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:05.138 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 54888 2024-09-13T11:25:05.138 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.138 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:05.138 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.138 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:05.138 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.423064+0000", 2024-09-13T11:25:05.139 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:05.139 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.139 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.139 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:05.139 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.423064+0000", 2024-09-13T11:25:05.139 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:05.139 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.139 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.139 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:05.139 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.423069+0000", 2024-09-13T11:25:05.139 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.7389999999999999e-06 2024-09-13T11:25:05.139 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.139 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.139 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:05.140 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.423115+0000", 2024-09-13T11:25:05.140 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.5847000000000003e-05 2024-09-13T11:25:05.140 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.140 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.140 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:05.140 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.423126+0000", 2024-09-13T11:25:05.140 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.1418e-05 2024-09-13T11:25:05.140 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.140 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.140 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:05.140 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.423132+0000", 2024-09-13T11:25:05.140 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.57e-06 2024-09-13T11:25:05.140 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.140 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.141 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:05.141 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.423191+0000", 2024-09-13T11:25:05.141 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.9250999999999999e-05 2024-09-13T11:25:05.141 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.141 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.141 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for peered", 2024-09-13T11:25:05.141 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.423198+0000", 2024-09-13T11:25:05.141 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.9179999999999997e-06 2024-09-13T11:25:05.141 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.141 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.141 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:05.141 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.907392+0000", 2024-09-13T11:25:05.141 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.48419437500000001 2024-09-13T11:25:05.141 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.142 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.142 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:05.142 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.907400+0000", 2024-09-13T11:25:05.142 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.7349999999999996e-06 2024-09-13T11:25:05.142 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:05.142 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:05.142 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:05.142 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.142 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.142 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDPGPushReply(3.17 76/70 [PushReplyOp(3:eb8d2fae:::benchmark_data_smithi177_31116_object18267:head)])", 2024-09-13T11:25:05.142 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:32.636274+0000", 2024-09-13T11:25:05.142 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 92.429463877000003, 2024-09-13T11:25:05.142 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.269081036, 2024-09-13T11:25:05.143 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:05.143 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:05.143 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-13T11:25:05.143 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:05.143 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.143 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:05.143 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.636274+0000", 2024-09-13T11:25:05.143 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:05.143 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.143 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.143 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:05.143 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.636274+0000", 2024-09-13T11:25:05.143 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:05.143 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.143 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.144 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:05.144 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.636276+0000", 2024-09-13T11:25:05.144 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.1679999999999998e-06 2024-09-13T11:25:05.144 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.144 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.144 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:05.144 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.636284+0000", 2024-09-13T11:25:05.144 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.7440000000000007e-06 2024-09-13T11:25:05.144 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.144 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.144 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:05.144 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.904155+0000", 2024-09-13T11:25:05.144 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.26787080800000002 2024-09-13T11:25:05.144 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.145 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.145 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:05.145 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.904165+0000", 2024-09-13T11:25:05.145 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0750999999999999e-05 2024-09-13T11:25:05.145 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.145 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.145 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:05.145 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.904238+0000", 2024-09-13T11:25:05.145 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.2335000000000005e-05 2024-09-13T11:25:05.145 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.145 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.145 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:05.145 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.905355+0000", 2024-09-13T11:25:05.145 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0011172300000000001 2024-09-13T11:25:05.146 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:05.146 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:05.146 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:05.146 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.146 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.146 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDPGPushReply(3.17 76/70 [PushReplyOp(3:ebcbce67:::benchmark_data_smithi177_31116_object18268:head)])", 2024-09-13T11:25:05.146 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:32.641093+0000", 2024-09-13T11:25:05.146 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 92.424644329000003, 2024-09-13T11:25:05.146 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.27471406799999998, 2024-09-13T11:25:05.146 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:05.146 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:05.146 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-13T11:25:05.147 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:05.147 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.147 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:05.147 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.641093+0000", 2024-09-13T11:25:05.147 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:05.147 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.147 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.147 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:05.147 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.641093+0000", 2024-09-13T11:25:05.147 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:05.147 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.147 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.147 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:05.147 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.641094+0000", 2024-09-13T11:25:05.148 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.0299999999999999e-07 2024-09-13T11:25:05.148 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.148 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.148 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:05.148 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.641098+0000", 2024-09-13T11:25:05.148 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.2200000000000003e-06 2024-09-13T11:25:05.148 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.148 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.148 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:05.148 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.910434+0000", 2024-09-13T11:25:05.148 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.26933570600000001 2024-09-13T11:25:05.148 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.148 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.149 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:05.149 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.910442+0000", 2024-09-13T11:25:05.149 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.2700000000000004e-06 2024-09-13T11:25:05.149 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.149 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.149 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:05.149 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.914989+0000", 2024-09-13T11:25:05.149 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0045465059999999996 2024-09-13T11:25:05.149 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.149 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.149 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:05.149 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.915808+0000", 2024-09-13T11:25:05.149 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00081876300000000002 2024-09-13T11:25:05.150 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:05.150 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:05.150 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:05.150 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.150 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.150 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4665.0:59977 3.21 e83/78 3:86b4d8f6:::benchmark_data_smithi177_31116_object59976:head v 83'3136, mlcod=0'0)", 2024-09-13T11:25:05.150 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:40.571147+0000", 2024-09-13T11:25:05.150 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 84.494591021000005, 2024-09-13T11:25:05.150 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.51907542799999995, 2024-09-13T11:25:05.150 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:05.150 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:05.150 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:05.150 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:05.150 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.150 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:05.151 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:40.571147+0000", 2024-09-13T11:25:05.151 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:05.151 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.151 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.151 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:05.151 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:40.571147+0000", 2024-09-13T11:25:05.151 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:05.151 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.151 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.151 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:05.151 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:40.571151+0000", 2024-09-13T11:25:05.151 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.5650000000000003e-06 2024-09-13T11:25:05.151 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.151 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.152 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:05.152 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:40.571204+0000", 2024-09-13T11:25:05.152 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.2720999999999999e-05 2024-09-13T11:25:05.152 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.152 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.152 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:05.152 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:41.088332+0000", 2024-09-13T11:25:05.152 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.51712822000000003 2024-09-13T11:25:05.152 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.152 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.152 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:05.152 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:41.088341+0000", 2024-09-13T11:25:05.152 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.3100000000000001e-06 2024-09-13T11:25:05.152 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.153 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.153 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:05.153 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:41.088444+0000", 2024-09-13T11:25:05.153 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000103708 2024-09-13T11:25:05.153 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.153 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.153 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:25:05.153 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:41.088475+0000", 2024-09-13T11:25:05.153 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.0497e-05 2024-09-13T11:25:05.153 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.153 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.153 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:25:05.153 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:41.090192+0000", 2024-09-13T11:25:05.154 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.001717288 2024-09-13T11:25:05.154 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.154 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.154 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:05.154 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:41.090222+0000", 2024-09-13T11:25:05.154 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.0119000000000001e-05 2024-09-13T11:25:05.154 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:05.154 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:05.154 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:05.154 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.154 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.154 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:110286 3.26 3.c41c0fa6 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e119)", 2024-09-13T11:25:05.154 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:24:25.654272+0000", 2024-09-13T11:25:05.154 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 39.411465329000002, 2024-09-13T11:25:05.154 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.43337235400000002, 2024-09-13T11:25:05.155 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:05.155 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:05.155 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-13T11:25:05.155 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:25:05.155 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:25:05.155 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:05.155 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 110286 2024-09-13T11:25:05.155 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.155 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:05.155 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.155 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:05.155 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:25.654272+0000", 2024-09-13T11:25:05.155 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:05.155 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.155 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.156 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:05.156 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:25.654272+0000", 2024-09-13T11:25:05.156 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:05.156 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.156 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.156 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:05.156 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:25.654275+0000", 2024-09-13T11:25:05.156 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.3199999999999998e-06 2024-09-13T11:25:05.156 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.156 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.156 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:05.156 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:25.654517+0000", 2024-09-13T11:25:05.156 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00024270199999999999 2024-09-13T11:25:05.156 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.157 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.157 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:05.157 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:25.654550+0000", 2024-09-13T11:25:05.157 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.2549000000000003e-05 2024-09-13T11:25:05.157 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.157 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.157 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:05.157 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:25.654566+0000", 2024-09-13T11:25:05.157 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.5517000000000001e-05 2024-09-13T11:25:05.157 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.157 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.157 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:05.157 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:25.654655+0000", 2024-09-13T11:25:05.157 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.9961000000000001e-05 2024-09-13T11:25:05.158 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.158 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.158 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for peered", 2024-09-13T11:25:05.158 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:25.654665+0000", 2024-09-13T11:25:05.158 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.0520000000000007e-06 2024-09-13T11:25:05.158 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.158 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.158 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:05.158 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:26.087629+0000", 2024-09-13T11:25:05.158 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.43296476900000003 2024-09-13T11:25:05.158 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.158 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.158 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:05.158 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:26.087645+0000", 2024-09-13T11:25:05.158 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.5483999999999999e-05 2024-09-13T11:25:05.159 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:05.159 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:05.159 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:05.159 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.159 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.159 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4665.0:140840 3.21 e144/102 3:87ae3db6:::benchmark_data_smithi177_31116_object537:head v 144'4366, mlcod=144'4364)", 2024-09-13T11:25:05.159 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:24:58.811357+0000", 2024-09-13T11:25:05.159 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 6.254380544, 2024-09-13T11:25:05.159 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.255611747, 2024-09-13T11:25:05.159 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:05.159 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:05.159 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:05.159 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:05.159 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.160 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:05.160 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.811357+0000", 2024-09-13T11:25:05.160 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:05.160 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.160 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.160 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:05.160 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.811357+0000", 2024-09-13T11:25:05.160 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:05.160 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.160 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.160 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:05.160 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.811360+0000", 2024-09-13T11:25:05.160 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.6809999999999998e-06 2024-09-13T11:25:05.160 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.161 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.161 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:05.161 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.811369+0000", 2024-09-13T11:25:05.161 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.1220000000000001e-06 2024-09-13T11:25:05.161 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.161 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.161 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:05.161 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:59.064607+0000", 2024-09-13T11:25:05.161 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.25323846500000002 2024-09-13T11:25:05.161 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.161 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.161 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:05.161 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:59.064614+0000", 2024-09-13T11:25:05.161 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.4869999999999999e-06 2024-09-13T11:25:05.161 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.162 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.162 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:05.162 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:59.064942+0000", 2024-09-13T11:25:05.162 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00032829499999999999 2024-09-13T11:25:05.162 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.162 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.162 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:25:05.162 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:59.064980+0000", 2024-09-13T11:25:05.162 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.8046000000000001e-05 2024-09-13T11:25:05.162 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.162 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.162 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:25:05.162 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:59.066914+0000", 2024-09-13T11:25:05.162 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.001933431 2024-09-13T11:25:05.162 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.163 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.163 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:05.163 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:59.066969+0000", 2024-09-13T11:25:05.163 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.5220000000000003e-05 2024-09-13T11:25:05.163 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:05.163 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:05.163 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:05.163 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:05.163 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:05.163 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-13T11:25:05.165 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_ops_in_flight 2024-09-13T11:25:05.166 INFO:tasks.ceph.osd.1.smithi092.stderr:2024-09-13T11:25:05.158+0000 7fe7789c5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33718) UID: 0 2024-09-13T11:25:05.266 INFO:tasks.ceph.osd.13.smithi092.stderr:2024-09-13T11:25:05.262+0000 7f3e5afea640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26350) UID: 0 2024-09-13T11:25:05.298 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-13T11:25:05.298 INFO:teuthology.orchestra.run.smithi001.stdout: "ops": [ 2024-09-13T11:25:05.298 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.298 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:148933 3.1b 3:dfca852f:::benchmark_data_smithi177_31116_object8630:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e149)", 2024-09-13T11:25:05.299 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:25:05.275217+0000", 2024-09-13T11:25:05.299 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 0.022841594, 2024-09-13T11:25:05.299 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.022860608000000001, 2024-09-13T11:25:05.299 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:05.299 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:05.299 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "waiting for sub ops", 2024-09-13T11:25:05.299 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:25:05.299 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:25:05.299 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:05.299 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 148933 2024-09-13T11:25:05.299 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.299 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:05.299 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.300 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:05.300 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:05.275217+0000", 2024-09-13T11:25:05.300 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:05.300 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.300 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.300 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:05.300 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:05.275217+0000", 2024-09-13T11:25:05.300 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:05.300 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.300 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.300 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:05.300 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:05.275219+0000", 2024-09-13T11:25:05.300 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.8029999999999999e-06 2024-09-13T11:25:05.300 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.301 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.301 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:05.301 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:05.275224+0000", 2024-09-13T11:25:05.301 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.057e-06 2024-09-13T11:25:05.301 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.301 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.301 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:05.301 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:05.275240+0000", 2024-09-13T11:25:05.301 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.6039999999999999e-05 2024-09-13T11:25:05.301 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.301 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.301 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:05.301 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:05.275248+0000", 2024-09-13T11:25:05.301 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.1580000000000005e-06 2024-09-13T11:25:05.302 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.302 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.302 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:05.302 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:05.275304+0000", 2024-09-13T11:25:05.302 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.5909999999999998e-05 2024-09-13T11:25:05.302 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.302 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.302 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:25:05.302 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:05.275431+0000", 2024-09-13T11:25:05.302 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000126494 2024-09-13T11:25:05.302 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.302 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.302 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 15", 2024-09-13T11:25:05.303 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:05.275531+0000", 2024-09-13T11:25:05.303 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.9773999999999997e-05 2024-09-13T11:25:05.303 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.303 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.303 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:25:05.303 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:05.277862+0000", 2024-09-13T11:25:05.303 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.002331606 2024-09-13T11:25:05.303 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:05.303 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:05.303 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:05.303 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:05.303 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-09-13T11:25:05.303 INFO:teuthology.orchestra.run.smithi001.stdout: "num_ops": 1 2024-09-13T11:25:05.303 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-13T11:25:05.316 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_blocked_ops 2024-09-13T11:25:05.368 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-09-13T11:25:05.364+0000 7f97befc9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26315) UID: 0 2024-09-13T11:25:05.450 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-13T11:25:05.450 INFO:teuthology.orchestra.run.smithi001.stdout: "ops": [], 2024-09-13T11:25:05.450 INFO:teuthology.orchestra.run.smithi001.stdout: "complaint_time": 30, 2024-09-13T11:25:05.450 INFO:teuthology.orchestra.run.smithi001.stdout: "num_blocked_ops": 0 2024-09-13T11:25:05.450 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-13T11:25:05.467 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_historic_ops 2024-09-13T11:25:05.468 INFO:tasks.ceph.osd.11.smithi177.stderr:2024-09-13T11:25:05.466+0000 7f24792c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26580) UID: 0 2024-09-13T11:25:05.569 INFO:tasks.ceph.osd.3.smithi177.stderr:2024-09-13T11:25:05.566+0000 7f10adba4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26574) UID: 0 2024-09-13T11:25:05.608 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-13T11:25:05.609 INFO:teuthology.orchestra.run.smithi001.stdout: "size": 20, 2024-09-13T11:25:05.609 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 600, 2024-09-13T11:25:05.609 INFO:teuthology.orchestra.run.smithi001.stdout: "ops": [ 2024-09-13T11:25:05.609 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.609 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop_reply(client.4665.0:167 3.9 e41/30 ondisk, result = 0)", 2024-09-13T11:25:05.609 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:22:47.419696+0000", 2024-09-13T11:25:05.609 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 138.18713154100001, 2024-09-13T11:25:05.609 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.64615778199999996, 2024-09-13T11:25:05.609 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:05.609 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:05.609 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:25:05.609 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:05.609 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.610 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:05.610 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:47.419696+0000", 2024-09-13T11:25:05.610 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:05.610 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.610 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.610 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:05.610 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:47.419696+0000", 2024-09-13T11:25:05.610 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:05.610 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.610 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.610 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:05.610 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:47.419697+0000", 2024-09-13T11:25:05.610 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.15e-06 2024-09-13T11:25:05.610 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.611 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.611 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:05.611 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:47.419700+0000", 2024-09-13T11:25:05.611 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.2449999999999998e-06 2024-09-13T11:25:05.611 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.611 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.611 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:05.611 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:48.065637+0000", 2024-09-13T11:25:05.611 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.64593674300000004 2024-09-13T11:25:05.611 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.611 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.611 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:05.611 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:48.065648+0000", 2024-09-13T11:25:05.611 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0865e-05 2024-09-13T11:25:05.612 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.612 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.612 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:05.612 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:48.065720+0000", 2024-09-13T11:25:05.612 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.2020999999999994e-05 2024-09-13T11:25:05.612 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.612 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.612 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:25:05.612 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:48.065728+0000", 2024-09-13T11:25:05.612 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.7660000000000004e-06 2024-09-13T11:25:05.612 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.612 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.612 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:05.613 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:48.065853+0000", 2024-09-13T11:25:05.613 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00012499200000000001 2024-09-13T11:25:05.613 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:05.613 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:05.613 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:05.613 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.613 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.613 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:25383 3.0 3:0a6bb869:::benchmark_data_smithi177_31116_object25382:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e55)", 2024-09-13T11:25:05.613 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:04.447376+0000", 2024-09-13T11:25:05.613 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 121.15945087199999, 2024-09-13T11:25:05.613 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.21239864, 2024-09-13T11:25:05.613 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:05.613 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:05.613 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:05.614 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:25:05.614 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:25:05.614 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:05.614 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 25383 2024-09-13T11:25:05.614 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.614 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:05.614 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.614 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:05.614 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.447376+0000", 2024-09-13T11:25:05.614 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:05.614 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.614 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.614 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:05.614 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.447376+0000", 2024-09-13T11:25:05.615 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:05.615 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.615 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.615 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:05.615 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.447378+0000", 2024-09-13T11:25:05.615 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.6729999999999999e-06 2024-09-13T11:25:05.615 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.615 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.615 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:05.615 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.447433+0000", 2024-09-13T11:25:05.615 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.5371000000000002e-05 2024-09-13T11:25:05.615 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.615 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.615 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:05.615 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.447446+0000", 2024-09-13T11:25:05.616 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.3179999999999999e-05 2024-09-13T11:25:05.616 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.616 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.616 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:05.616 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.447456+0000", 2024-09-13T11:25:05.616 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0008e-05 2024-09-13T11:25:05.616 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.616 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.616 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:05.616 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.447514+0000", 2024-09-13T11:25:05.617 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.8023999999999999e-05 2024-09-13T11:25:05.617 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.617 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.617 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:25:05.617 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.447642+0000", 2024-09-13T11:25:05.617 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000127296 2024-09-13T11:25:05.617 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.617 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.617 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 15", 2024-09-13T11:25:05.617 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.447767+0000", 2024-09-13T11:25:05.618 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000125547 2024-09-13T11:25:05.618 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.618 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.618 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:25:05.618 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.449188+0000", 2024-09-13T11:25:05.618 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.001420604 2024-09-13T11:25:05.618 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.618 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.618 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:25:05.618 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.659668+0000", 2024-09-13T11:25:05.618 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.210479685 2024-09-13T11:25:05.619 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.619 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.619 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:25:05.619 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.659716+0000", 2024-09-13T11:25:05.619 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.8609999999999997e-05 2024-09-13T11:25:05.619 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.619 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.619 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:05.619 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.659775+0000", 2024-09-13T11:25:05.619 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.8641999999999999e-05 2024-09-13T11:25:05.619 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:05.619 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:05.619 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:05.620 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.620 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.620 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4665.0:25408 3.4 e56/40 3:294e23bd:::benchmark_data_smithi177_31116_object25407:head v 56'1645, mlcod=55'1642)", 2024-09-13T11:25:05.620 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:04.494686+0000", 2024-09-13T11:25:05.620 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 121.112141213, 2024-09-13T11:25:05.620 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.29725566399999998, 2024-09-13T11:25:05.620 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:05.620 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:05.620 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:05.620 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:05.620 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.620 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:05.620 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.494686+0000", 2024-09-13T11:25:05.620 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:05.621 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.621 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.621 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:05.621 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.494686+0000", 2024-09-13T11:25:05.621 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:05.621 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.621 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.621 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:05.621 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.494686+0000", 2024-09-13T11:25:05.621 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.2399999999999999e-07 2024-09-13T11:25:05.621 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.621 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.621 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:05.621 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.494809+0000", 2024-09-13T11:25:05.622 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00012308600000000001 2024-09-13T11:25:05.622 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.622 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.622 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:05.622 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.789410+0000", 2024-09-13T11:25:05.622 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.29460066899999998 2024-09-13T11:25:05.622 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.622 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.622 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:05.622 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.789425+0000", 2024-09-13T11:25:05.622 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.4600000000000001e-05 2024-09-13T11:25:05.622 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.622 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.622 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:05.622 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.789507+0000", 2024-09-13T11:25:05.623 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.2785000000000001e-05 2024-09-13T11:25:05.623 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.623 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.623 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:25:05.623 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.789525+0000", 2024-09-13T11:25:05.623 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.8117999999999998e-05 2024-09-13T11:25:05.623 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.623 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.623 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:25:05.623 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.791852+0000", 2024-09-13T11:25:05.624 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0023264459999999998 2024-09-13T11:25:05.624 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.624 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.624 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:05.624 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.791942+0000", 2024-09-13T11:25:05.624 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.9635999999999994e-05 2024-09-13T11:25:05.624 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:05.624 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:05.624 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:05.624 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.624 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.624 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4665.0:30815 3.f e59/30 3:fde17cb1:::benchmark_data_smithi177_31116_object30814:head v 59'1919, mlcod=59'1914)", 2024-09-13T11:25:05.624 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:09.494342+0000", 2024-09-13T11:25:05.625 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 116.11248522699999, 2024-09-13T11:25:05.625 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.22787855700000001, 2024-09-13T11:25:05.625 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:05.625 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:05.625 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:05.625 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:05.625 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.625 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:05.625 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.494342+0000", 2024-09-13T11:25:05.625 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:05.625 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.625 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.625 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:05.626 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.494342+0000", 2024-09-13T11:25:05.626 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:05.626 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.626 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.626 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:05.626 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.494343+0000", 2024-09-13T11:25:05.626 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.251e-06 2024-09-13T11:25:05.626 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.626 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.626 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:05.626 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.494387+0000", 2024-09-13T11:25:05.626 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.4276000000000001e-05 2024-09-13T11:25:05.626 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.626 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.627 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:05.627 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.720454+0000", 2024-09-13T11:25:05.627 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.226066614 2024-09-13T11:25:05.627 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.627 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.627 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:05.627 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.720462+0000", 2024-09-13T11:25:05.627 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.9240000000000007e-06 2024-09-13T11:25:05.627 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.627 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.627 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:05.627 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.720537+0000", 2024-09-13T11:25:05.627 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.4967000000000002e-05 2024-09-13T11:25:05.627 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.628 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.628 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:25:05.628 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.720563+0000", 2024-09-13T11:25:05.628 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.5630999999999999e-05 2024-09-13T11:25:05.628 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.628 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.628 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:25:05.628 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.722183+0000", 2024-09-13T11:25:05.628 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0016209670000000001 2024-09-13T11:25:05.628 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.628 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.628 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:05.628 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.722220+0000", 2024-09-13T11:25:05.629 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.6927000000000001e-05 2024-09-13T11:25:05.629 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:05.629 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:05.629 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:05.629 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.629 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.629 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4665.0:30828 3.f e59/30 3:f3a9c6a9:::benchmark_data_smithi177_31116_object30827:head v 59'1920, mlcod=59'1916)", 2024-09-13T11:25:05.629 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:09.502607+0000", 2024-09-13T11:25:05.629 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 116.10422017, 2024-09-13T11:25:05.629 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.26374408300000002, 2024-09-13T11:25:05.629 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:05.629 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:05.629 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:05.629 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:05.630 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.630 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:05.630 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.502607+0000", 2024-09-13T11:25:05.630 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:05.630 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.630 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.630 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:05.630 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.502607+0000", 2024-09-13T11:25:05.630 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:05.630 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.630 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.630 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:05.630 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.502609+0000", 2024-09-13T11:25:05.631 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.84e-06 2024-09-13T11:25:05.631 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.631 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.631 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:05.631 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.502663+0000", 2024-09-13T11:25:05.631 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.3971000000000002e-05 2024-09-13T11:25:05.631 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.631 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.631 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:05.631 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.763912+0000", 2024-09-13T11:25:05.631 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.26124904300000001 2024-09-13T11:25:05.631 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.631 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.631 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:05.631 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.763920+0000", 2024-09-13T11:25:05.632 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.1149999999999994e-06 2024-09-13T11:25:05.632 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.632 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.632 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:05.632 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.764024+0000", 2024-09-13T11:25:05.632 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00010380899999999999 2024-09-13T11:25:05.632 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.632 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.632 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:25:05.632 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.764068+0000", 2024-09-13T11:25:05.632 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.4638999999999997e-05 2024-09-13T11:25:05.632 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.632 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.632 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:25:05.633 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.766308+0000", 2024-09-13T11:25:05.633 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0022392039999999998 2024-09-13T11:25:05.633 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.633 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.633 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:05.633 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.766351+0000", 2024-09-13T11:25:05.633 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.3461999999999998e-05 2024-09-13T11:25:05.633 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:05.633 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:05.633 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:05.633 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.633 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.633 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDPGPush(3.1b 84/78 [PushOp(3:dcddd139:::benchmark_data_smithi177_31116_object10384:head, version: 48'673, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:dcddd139:::benchmark_data_smithi177_31116_object10384:head@48'673, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:25:05.633 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:41.400602+0000", 2024-09-13T11:25:05.634 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 84.206224684999995, 2024-09-13T11:25:05.634 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.43365821999999998, 2024-09-13T11:25:05.634 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:05.634 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:05.634 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:25:05.634 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:05.634 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.634 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:05.634 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:41.400602+0000", 2024-09-13T11:25:05.634 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:05.634 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.634 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.634 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:05.634 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:41.400602+0000", 2024-09-13T11:25:05.635 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:05.635 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.635 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.635 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:05.635 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:41.400604+0000", 2024-09-13T11:25:05.635 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.99e-06 2024-09-13T11:25:05.635 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.635 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.635 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:05.635 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:41.400653+0000", 2024-09-13T11:25:05.635 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.8195000000000002e-05 2024-09-13T11:25:05.635 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.635 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.635 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:05.635 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:41.833180+0000", 2024-09-13T11:25:05.635 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.43252769200000002 2024-09-13T11:25:05.636 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.636 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.636 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:05.636 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:41.833213+0000", 2024-09-13T11:25:05.636 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.2737999999999997e-05 2024-09-13T11:25:05.636 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.636 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.636 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:05.636 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:41.833408+0000", 2024-09-13T11:25:05.637 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00019473799999999999 2024-09-13T11:25:05.637 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.637 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.637 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:25:05.637 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:41.833417+0000", 2024-09-13T11:25:05.637 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.5349999999999993e-06 2024-09-13T11:25:05.637 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.637 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.637 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:05.637 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:41.834261+0000", 2024-09-13T11:25:05.638 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00084333200000000002 2024-09-13T11:25:05.638 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:05.638 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:05.638 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:05.638 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.639 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.639 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDPGPushReply(3.20 85/77 [PushReplyOp(3:066982a6:::benchmark_data_smithi177_31116_object16400:head)])", 2024-09-13T11:25:05.639 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:42.671551+0000", 2024-09-13T11:25:05.639 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 82.935275861999997, 2024-09-13T11:25:05.639 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.28249085499999999, 2024-09-13T11:25:05.639 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:05.639 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:05.639 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-13T11:25:05.639 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:05.639 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.639 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:05.639 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.671551+0000", 2024-09-13T11:25:05.639 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:05.640 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.640 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.640 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:05.640 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.671551+0000", 2024-09-13T11:25:05.640 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:05.640 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.640 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.640 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:05.640 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.671552+0000", 2024-09-13T11:25:05.640 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.097e-06 2024-09-13T11:25:05.640 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.640 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.640 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:05.640 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.671556+0000", 2024-09-13T11:25:05.641 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.8519999999999997e-06 2024-09-13T11:25:05.641 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.641 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.641 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:05.641 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.951769+0000", 2024-09-13T11:25:05.641 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.28021293000000003 2024-09-13T11:25:05.641 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.641 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.641 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:05.641 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.951780+0000", 2024-09-13T11:25:05.641 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.1059e-05 2024-09-13T11:25:05.641 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.641 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.641 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:05.642 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.951992+0000", 2024-09-13T11:25:05.642 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000211478 2024-09-13T11:25:05.642 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.642 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.642 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:05.642 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.954042+0000", 2024-09-13T11:25:05.642 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0020504389999999998 2024-09-13T11:25:05.642 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:05.642 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:05.642 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:05.642 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.642 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.642 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDPGPush(3.1b 89/78 [PushOp(3:dc4c79e7:::benchmark_data_smithi177_31116_object19857:head, version: 54'1266, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:dc4c79e7:::benchmark_data_smithi177_31116_object19857:head@54'1266, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:25:05.643 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:46.649128+0000", 2024-09-13T11:25:05.643 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 78.957698754000006, 2024-09-13T11:25:05.643 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.29963603, 2024-09-13T11:25:05.643 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:05.643 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:05.643 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:25:05.643 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:05.643 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.643 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:05.643 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.649128+0000", 2024-09-13T11:25:05.643 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:05.643 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.643 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.644 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:05.644 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.649128+0000", 2024-09-13T11:25:05.644 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:05.644 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.644 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.644 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:05.644 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.649130+0000", 2024-09-13T11:25:05.644 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2950000000000001e-06 2024-09-13T11:25:05.644 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.644 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.644 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:05.644 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.649197+0000", 2024-09-13T11:25:05.644 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.7033999999999995e-05 2024-09-13T11:25:05.644 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.645 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.645 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:05.645 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.947691+0000", 2024-09-13T11:25:05.645 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.29849387300000002 2024-09-13T11:25:05.645 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.645 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.645 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:05.645 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.947702+0000", 2024-09-13T11:25:05.645 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.1239e-05 2024-09-13T11:25:05.645 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.645 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.645 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:05.645 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.947799+0000", 2024-09-13T11:25:05.645 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.7283999999999999e-05 2024-09-13T11:25:05.646 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.646 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.646 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:25:05.646 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.947809+0000", 2024-09-13T11:25:05.646 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.008e-05 2024-09-13T11:25:05.646 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.646 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.646 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:05.646 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.948764+0000", 2024-09-13T11:25:05.646 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00095522499999999998 2024-09-13T11:25:05.646 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:05.646 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:05.646 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:05.646 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.647 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.647 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:71927 3.0 3:0197244a:::benchmark_data_smithi177_31116_object71926:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2024-09-13T11:25:05.647 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:54.199022+0000", 2024-09-13T11:25:05.647 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 71.407805397999994, 2024-09-13T11:25:05.647 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.26819617800000001, 2024-09-13T11:25:05.647 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:05.647 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:05.647 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:05.647 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:25:05.647 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:25:05.647 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:05.647 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 71927 2024-09-13T11:25:05.647 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.647 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:05.648 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.648 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:05.648 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:54.199022+0000", 2024-09-13T11:25:05.648 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:05.648 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.648 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.648 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:05.648 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:54.199022+0000", 2024-09-13T11:25:05.648 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:05.648 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.648 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.648 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:05.648 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:54.199025+0000", 2024-09-13T11:25:05.648 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.2200000000000001e-06 2024-09-13T11:25:05.649 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.649 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.649 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:05.649 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:54.199076+0000", 2024-09-13T11:25:05.649 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.1184000000000003e-05 2024-09-13T11:25:05.649 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.649 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.649 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:05.649 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:54.199095+0000", 2024-09-13T11:25:05.649 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.9253999999999999e-05 2024-09-13T11:25:05.649 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.649 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.649 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:05.649 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:54.199107+0000", 2024-09-13T11:25:05.650 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.1916000000000001e-05 2024-09-13T11:25:05.650 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.650 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.650 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:05.650 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:54.199196+0000", 2024-09-13T11:25:05.650 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.8645000000000003e-05 2024-09-13T11:25:05.650 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.650 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.650 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:25:05.650 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:54.199421+0000", 2024-09-13T11:25:05.650 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000224868 2024-09-13T11:25:05.650 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.650 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.650 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 15", 2024-09-13T11:25:05.651 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:54.199647+0000", 2024-09-13T11:25:05.651 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00022666099999999999 2024-09-13T11:25:05.651 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.651 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.651 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:25:05.651 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:54.201333+0000", 2024-09-13T11:25:05.651 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.001685761 2024-09-13T11:25:05.651 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.651 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.651 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:25:05.651 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:54.467097+0000", 2024-09-13T11:25:05.651 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.26576393399999998 2024-09-13T11:25:05.651 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.652 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.652 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:25:05.652 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:54.467163+0000", 2024-09-13T11:25:05.652 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.6088999999999997e-05 2024-09-13T11:25:05.652 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.652 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.652 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:05.652 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:54.467218+0000", 2024-09-13T11:25:05.652 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.4645999999999998e-05 2024-09-13T11:25:05.652 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:05.652 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:05.652 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:05.652 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.652 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.653 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4665.0:98479 3.2 e113/75 3:41084616:::benchmark_data_smithi177_31116_object98478:head v 113'3785, mlcod=113'3782)", 2024-09-13T11:25:05.653 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:24:15.343730+0000", 2024-09-13T11:25:05.653 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 50.263097127999998, 2024-09-13T11:25:05.653 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.196889648, 2024-09-13T11:25:05.653 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:05.653 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:05.653 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:05.653 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:05.653 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.653 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:05.653 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.343730+0000", 2024-09-13T11:25:05.653 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:05.653 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.653 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.654 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:05.654 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.343730+0000", 2024-09-13T11:25:05.654 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:05.654 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.654 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.654 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:05.654 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.343732+0000", 2024-09-13T11:25:05.654 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.6819999999999999e-06 2024-09-13T11:25:05.654 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.654 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.654 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:05.654 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.343781+0000", 2024-09-13T11:25:05.654 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.9691999999999998e-05 2024-09-13T11:25:05.654 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.655 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.655 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:05.655 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.538204+0000", 2024-09-13T11:25:05.655 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.19442271699999999 2024-09-13T11:25:05.655 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.655 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.655 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:05.655 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.538215+0000", 2024-09-13T11:25:05.655 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0648e-05 2024-09-13T11:25:05.655 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.655 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.655 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:05.655 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.538342+0000", 2024-09-13T11:25:05.655 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000127448 2024-09-13T11:25:05.656 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.656 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.656 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:25:05.656 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.538395+0000", 2024-09-13T11:25:05.656 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.2895000000000001e-05 2024-09-13T11:25:05.656 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.656 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.656 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:25:05.656 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.540544+0000", 2024-09-13T11:25:05.656 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0021487379999999999 2024-09-13T11:25:05.656 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.656 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.656 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:05.657 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.540620+0000", 2024-09-13T11:25:05.657 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.5828000000000007e-05 2024-09-13T11:25:05.657 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:05.657 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:05.657 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:05.657 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.657 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.657 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4665.0:98954 3.2 e113/75 3:416baeb7:::benchmark_data_smithi177_31116_object98953:head v 113'3790, mlcod=113'3787)", 2024-09-13T11:25:05.657 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:24:15.819065+0000", 2024-09-13T11:25:05.657 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 49.787762246, 2024-09-13T11:25:05.657 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.187742985, 2024-09-13T11:25:05.657 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:05.657 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:05.657 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:05.658 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:05.658 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.658 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:05.658 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.819065+0000", 2024-09-13T11:25:05.658 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:05.658 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.658 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.658 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:05.658 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.819065+0000", 2024-09-13T11:25:05.659 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:05.659 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.659 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.659 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:05.659 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.819068+0000", 2024-09-13T11:25:05.659 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.0469999999999998e-06 2024-09-13T11:25:05.659 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.659 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.659 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:05.659 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.819124+0000", 2024-09-13T11:25:05.659 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.6335999999999997e-05 2024-09-13T11:25:05.660 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.660 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.660 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:05.660 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:16.003736+0000", 2024-09-13T11:25:05.660 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.18461164899999999 2024-09-13T11:25:05.660 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.660 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.660 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:05.660 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:16.003749+0000", 2024-09-13T11:25:05.660 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.3157e-05 2024-09-13T11:25:05.660 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.660 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.660 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:05.661 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:16.003903+0000", 2024-09-13T11:25:05.661 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00015374799999999999 2024-09-13T11:25:05.661 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.661 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.661 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:25:05.661 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:16.003961+0000", 2024-09-13T11:25:05.661 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.8026000000000003e-05 2024-09-13T11:25:05.661 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.661 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.661 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:25:05.661 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:16.006699+0000", 2024-09-13T11:25:05.661 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0027384290000000001 2024-09-13T11:25:05.662 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.662 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.662 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:05.662 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:16.006808+0000", 2024-09-13T11:25:05.662 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000108593 2024-09-13T11:25:05.662 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:05.662 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:05.663 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:05.663 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.663 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.663 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4665.0:99149 3.2 e113/75 3:4350ea1a:::benchmark_data_smithi177_31116_object99148:head v 113'3791, mlcod=113'3788)", 2024-09-13T11:25:05.663 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:24:16.003634+0000", 2024-09-13T11:25:05.663 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 49.603192725, 2024-09-13T11:25:05.663 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.19660476800000001, 2024-09-13T11:25:05.663 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:05.663 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:05.663 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:05.663 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:05.663 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.663 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:05.663 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:16.003634+0000", 2024-09-13T11:25:05.664 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:05.664 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.664 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.664 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:05.664 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:16.003634+0000", 2024-09-13T11:25:05.664 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:05.664 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.664 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.664 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:05.664 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:16.003637+0000", 2024-09-13T11:25:05.664 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.9749999999999999e-06 2024-09-13T11:25:05.664 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.664 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.664 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:05.665 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:16.003701+0000", 2024-09-13T11:25:05.665 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.3764999999999997e-05 2024-09-13T11:25:05.665 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.665 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.665 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:05.665 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:16.198858+0000", 2024-09-13T11:25:05.665 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.19515649900000001 2024-09-13T11:25:05.665 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.665 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.665 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:05.665 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:16.198863+0000", 2024-09-13T11:25:05.665 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.2229999999999998e-06 2024-09-13T11:25:05.665 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.666 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.666 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:05.666 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:16.198918+0000", 2024-09-13T11:25:05.666 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.5427000000000003e-05 2024-09-13T11:25:05.666 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.666 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.666 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:25:05.666 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:16.198935+0000", 2024-09-13T11:25:05.666 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.6776e-05 2024-09-13T11:25:05.666 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.667 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.667 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:25:05.667 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:16.200208+0000", 2024-09-13T11:25:05.667 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0012725429999999999 2024-09-13T11:25:05.667 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.667 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.667 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:05.667 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:16.200239+0000", 2024-09-13T11:25:05.667 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.1560000000000003e-05 2024-09-13T11:25:05.667 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:05.667 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:05.667 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:05.667 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.668 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.668 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4665.0:101036 3.4 e114/67 3:207f4fe8:::benchmark_data_smithi177_31116_object101035:head v 114'4508, mlcod=114'4505)", 2024-09-13T11:25:05.668 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:24:17.391594+0000", 2024-09-13T11:25:05.668 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 48.215233349000002, 2024-09-13T11:25:05.668 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.19061766999999999, 2024-09-13T11:25:05.668 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:05.668 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:05.668 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:05.668 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:05.668 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.668 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:05.668 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:17.391594+0000", 2024-09-13T11:25:05.668 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:05.669 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.669 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.669 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:05.669 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:17.391594+0000", 2024-09-13T11:25:05.669 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:05.669 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.669 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.669 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:05.669 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:17.391595+0000", 2024-09-13T11:25:05.669 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.6729999999999999e-06 2024-09-13T11:25:05.669 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.669 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.669 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:05.669 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:17.391668+0000", 2024-09-13T11:25:05.670 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.2096999999999998e-05 2024-09-13T11:25:05.670 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.670 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.670 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:05.670 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:17.579638+0000", 2024-09-13T11:25:05.670 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.187969993 2024-09-13T11:25:05.670 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.670 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.670 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:05.670 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:17.579644+0000", 2024-09-13T11:25:05.670 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.2049999999999999e-06 2024-09-13T11:25:05.670 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.670 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.670 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:05.671 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:17.579707+0000", 2024-09-13T11:25:05.671 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.2948999999999996e-05 2024-09-13T11:25:05.671 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.671 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.671 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:25:05.671 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:17.579731+0000", 2024-09-13T11:25:05.671 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.4315999999999999e-05 2024-09-13T11:25:05.671 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.671 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.671 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:25:05.671 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:17.582172+0000", 2024-09-13T11:25:05.671 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.002440705 2024-09-13T11:25:05.671 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.671 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.672 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:05.672 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:17.582211+0000", 2024-09-13T11:25:05.672 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.9731999999999998e-05 2024-09-13T11:25:05.672 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:05.672 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:05.672 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:05.672 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.672 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.672 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:108129 3.9 3:94038175:::benchmark_data_smithi177_31116_object108128:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116)", 2024-09-13T11:25:05.672 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:24:22.802908+0000", 2024-09-13T11:25:05.672 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 42.803918629000002, 2024-09-13T11:25:05.672 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.179351336, 2024-09-13T11:25:05.672 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:05.672 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:05.673 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:05.673 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:25:05.673 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:25:05.673 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:05.673 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 108129 2024-09-13T11:25:05.673 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.673 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:05.673 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.673 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:05.673 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:22.802908+0000", 2024-09-13T11:25:05.673 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:05.673 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.673 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.673 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:05.674 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:22.802908+0000", 2024-09-13T11:25:05.674 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:05.674 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.674 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.674 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:05.674 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:22.802910+0000", 2024-09-13T11:25:05.674 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.765e-06 2024-09-13T11:25:05.674 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.674 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.674 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:05.674 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:22.802971+0000", 2024-09-13T11:25:05.674 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.0866000000000003e-05 2024-09-13T11:25:05.674 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.674 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.675 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:05.675 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:22.802985+0000", 2024-09-13T11:25:05.675 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.433e-05 2024-09-13T11:25:05.675 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.675 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.675 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:05.675 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:22.802993+0000", 2024-09-13T11:25:05.675 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.7940000000000003e-06 2024-09-13T11:25:05.675 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.675 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.675 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:05.675 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:22.803050+0000", 2024-09-13T11:25:05.675 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.6567999999999998e-05 2024-09-13T11:25:05.675 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.676 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.676 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:25:05.676 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:22.803158+0000", 2024-09-13T11:25:05.676 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00010788700000000001 2024-09-13T11:25:05.676 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.676 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.676 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 2", 2024-09-13T11:25:05.676 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:22.803262+0000", 2024-09-13T11:25:05.676 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00010450099999999999 2024-09-13T11:25:05.676 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.676 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.676 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:25:05.676 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:22.806107+0000", 2024-09-13T11:25:05.677 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0028447149999999998 2024-09-13T11:25:05.677 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.677 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.677 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:25:05.677 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:22.982142+0000", 2024-09-13T11:25:05.677 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.17603523700000001 2024-09-13T11:25:05.677 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.677 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.677 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:25:05.677 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:22.982196+0000", 2024-09-13T11:25:05.677 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.3832999999999998e-05 2024-09-13T11:25:05.677 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.677 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.677 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:05.678 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:22.982260+0000", 2024-09-13T11:25:05.678 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.3839999999999999e-05 2024-09-13T11:25:05.678 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:05.678 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:05.678 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:05.678 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.678 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.678 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDPGPush(3.25 121/120 [PushOp(3:a5de2229:::benchmark_data_smithi177_31116_object1005:head, version: 43'62, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:a5de2229:::benchmark_data_smithi177_31116_object1005:head@43'62, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:25:05.678 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:24:27.379880+0000", 2024-09-13T11:25:05.678 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 38.226947559999999, 2024-09-13T11:25:05.678 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.72647719499999996, 2024-09-13T11:25:05.678 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:05.678 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:05.678 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:25:05.679 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:05.679 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.679 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:05.679 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:27.379880+0000", 2024-09-13T11:25:05.679 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:05.679 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.679 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.679 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:05.679 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:27.379880+0000", 2024-09-13T11:25:05.679 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:05.679 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.679 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.679 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:05.679 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:27.379882+0000", 2024-09-13T11:25:05.680 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.1050000000000002e-06 2024-09-13T11:25:05.680 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.680 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.680 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:05.680 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:27.379976+0000", 2024-09-13T11:25:05.680 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.4468000000000004e-05 2024-09-13T11:25:05.680 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.680 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.680 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:05.680 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:28.105075+0000", 2024-09-13T11:25:05.680 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.72509911299999996 2024-09-13T11:25:05.680 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.680 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.681 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:05.681 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:28.105111+0000", 2024-09-13T11:25:05.681 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.5403e-05 2024-09-13T11:25:05.681 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.681 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.681 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:05.681 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:28.105267+0000", 2024-09-13T11:25:05.681 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00015663600000000001 2024-09-13T11:25:05.681 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.681 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.681 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:25:05.681 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:28.105291+0000", 2024-09-13T11:25:05.681 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.4164000000000001e-05 2024-09-13T11:25:05.681 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.682 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.682 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:05.682 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:28.106357+0000", 2024-09-13T11:25:05.682 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.001065306 2024-09-13T11:25:05.682 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:05.682 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:05.682 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:05.682 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.682 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.682 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:113646 3.20 3:059047a0:::benchmark_data_smithi177_31116_object113645:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124)", 2024-09-13T11:25:05.682 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:24:32.178229+0000", 2024-09-13T11:25:05.682 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 33.428597600000003, 2024-09-13T11:25:05.682 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.23346934699999999, 2024-09-13T11:25:05.682 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:05.683 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:05.683 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:05.683 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:25:05.683 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:25:05.683 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:05.683 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 113646 2024-09-13T11:25:05.683 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.683 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:05.683 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.683 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:05.683 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:32.178229+0000", 2024-09-13T11:25:05.683 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:05.683 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.683 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.684 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:05.684 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:32.178229+0000", 2024-09-13T11:25:05.684 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:05.684 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.684 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.684 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:05.684 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:32.178232+0000", 2024-09-13T11:25:05.684 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.069e-06 2024-09-13T11:25:05.684 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.684 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.684 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:05.684 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:32.178266+0000", 2024-09-13T11:25:05.684 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.4242999999999997e-05 2024-09-13T11:25:05.684 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.685 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.685 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:05.685 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:32.178278+0000", 2024-09-13T11:25:05.685 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2558000000000001e-05 2024-09-13T11:25:05.685 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.685 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.685 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:05.685 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:32.178288+0000", 2024-09-13T11:25:05.685 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.6469999999999993e-06 2024-09-13T11:25:05.685 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.685 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.685 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:05.685 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:32.178349+0000", 2024-09-13T11:25:05.685 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.0643999999999997e-05 2024-09-13T11:25:05.686 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.686 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.686 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:25:05.686 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:32.178477+0000", 2024-09-13T11:25:05.686 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000128038 2024-09-13T11:25:05.686 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.686 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.686 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 0", 2024-09-13T11:25:05.686 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:32.178602+0000", 2024-09-13T11:25:05.686 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00012553600000000001 2024-09-13T11:25:05.686 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.686 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.686 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:25:05.687 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:32.180157+0000", 2024-09-13T11:25:05.687 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.001554386 2024-09-13T11:25:05.687 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.687 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.687 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:25:05.687 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:32.411634+0000", 2024-09-13T11:25:05.687 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.231477561 2024-09-13T11:25:05.687 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.687 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.687 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:25:05.687 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:32.411672+0000", 2024-09-13T11:25:05.687 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.7357999999999997e-05 2024-09-13T11:25:05.687 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.687 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.688 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:05.688 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:32.411699+0000", 2024-09-13T11:25:05.688 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.7307000000000001e-05 2024-09-13T11:25:05.688 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:05.688 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:05.688 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:05.688 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.688 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.688 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop_reply(client.4665.0:113646 3.20 e124/77 ondisk, result = 0)", 2024-09-13T11:25:05.688 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:24:32.180524+0000", 2024-09-13T11:25:05.688 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 33.426302819, 2024-09-13T11:25:05.688 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.231180894, 2024-09-13T11:25:05.688 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:05.688 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:05.689 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:25:05.689 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:05.689 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.689 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:05.689 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:32.180524+0000", 2024-09-13T11:25:05.689 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:05.689 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.689 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.689 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:05.689 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:32.180524+0000", 2024-09-13T11:25:05.689 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:05.689 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.689 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.689 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:05.690 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:32.180526+0000", 2024-09-13T11:25:05.690 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.4419999999999999e-06 2024-09-13T11:25:05.690 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.690 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.690 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:05.690 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:32.180530+0000", 2024-09-13T11:25:05.690 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.7330000000000003e-06 2024-09-13T11:25:05.690 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.690 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.690 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:05.690 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:32.411549+0000", 2024-09-13T11:25:05.690 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.23101862300000001 2024-09-13T11:25:05.690 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.690 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.691 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:05.691 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:32.411558+0000", 2024-09-13T11:25:05.691 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.5949999999999999e-06 2024-09-13T11:25:05.691 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.691 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.691 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:05.691 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:32.411622+0000", 2024-09-13T11:25:05.691 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.4251000000000005e-05 2024-09-13T11:25:05.691 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.691 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.691 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:25:05.691 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:32.411629+0000", 2024-09-13T11:25:05.691 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.328e-06 2024-09-13T11:25:05.691 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.692 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.692 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:05.692 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:32.411705+0000", 2024-09-13T11:25:05.692 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.5921999999999996e-05 2024-09-13T11:25:05.692 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:05.692 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:05.692 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:05.692 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.692 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.692 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop_reply(client.4665.0:129892 3.0 e134/75 ondisk, result = 0)", 2024-09-13T11:25:05.692 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:24:47.136509+0000", 2024-09-13T11:25:05.692 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 18.470317772000001, 2024-09-13T11:25:05.692 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.211101067, 2024-09-13T11:25:05.693 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:05.693 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:05.693 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:25:05.693 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:05.693 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.693 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:05.693 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:47.136509+0000", 2024-09-13T11:25:05.693 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:05.693 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.693 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.693 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:05.693 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:47.136509+0000", 2024-09-13T11:25:05.693 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:05.693 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.693 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.694 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:05.694 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:47.136510+0000", 2024-09-13T11:25:05.694 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.17e-06 2024-09-13T11:25:05.694 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.694 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.694 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:05.694 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:47.136514+0000", 2024-09-13T11:25:05.694 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.5609999999999999e-06 2024-09-13T11:25:05.694 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.694 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.694 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:05.694 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:47.347445+0000", 2024-09-13T11:25:05.694 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.210931324 2024-09-13T11:25:05.695 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.695 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.695 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:05.695 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:47.347450+0000", 2024-09-13T11:25:05.695 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.3819999999999997e-06 2024-09-13T11:25:05.695 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.695 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.695 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:05.695 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:47.347491+0000", 2024-09-13T11:25:05.695 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.1025000000000001e-05 2024-09-13T11:25:05.695 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.695 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.695 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:25:05.695 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:47.347496+0000", 2024-09-13T11:25:05.696 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.463e-06 2024-09-13T11:25:05.696 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.696 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.696 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:05.696 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:47.347610+0000", 2024-09-13T11:25:05.696 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00011414200000000001 2024-09-13T11:25:05.696 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:05.696 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:05.696 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:05.696 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.696 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.696 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:138294 3.9 3:90012dab:::benchmark_data_smithi177_31116_object138293:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e140)", 2024-09-13T11:25:05.696 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:24:54.250266+0000", 2024-09-13T11:25:05.696 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 11.356560887000001, 2024-09-13T11:25:05.697 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.33165556200000001, 2024-09-13T11:25:05.697 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:05.697 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:05.697 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:05.697 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:25:05.697 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:25:05.697 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:05.697 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 138294 2024-09-13T11:25:05.697 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.697 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:05.697 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.697 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:05.697 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:54.250266+0000", 2024-09-13T11:25:05.697 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:05.698 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.698 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.698 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:05.698 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:54.250266+0000", 2024-09-13T11:25:05.698 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:05.698 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.698 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.698 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:05.698 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:54.250269+0000", 2024-09-13T11:25:05.699 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.3159999999999999e-06 2024-09-13T11:25:05.699 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.699 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.699 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:05.699 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:54.250362+0000", 2024-09-13T11:25:05.699 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.3051000000000001e-05 2024-09-13T11:25:05.699 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.699 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.699 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:05.699 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:54.250374+0000", 2024-09-13T11:25:05.699 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2327000000000001e-05 2024-09-13T11:25:05.699 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.699 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.699 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:05.700 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:54.250382+0000", 2024-09-13T11:25:05.700 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.7249999999999997e-06 2024-09-13T11:25:05.700 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.700 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.700 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:05.700 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:54.250459+0000", 2024-09-13T11:25:05.700 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.7218999999999994e-05 2024-09-13T11:25:05.700 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.700 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.700 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:25:05.700 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:54.250611+0000", 2024-09-13T11:25:05.700 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00015241000000000001 2024-09-13T11:25:05.700 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.701 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.701 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 2", 2024-09-13T11:25:05.701 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:54.250777+0000", 2024-09-13T11:25:05.701 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00016575199999999999 2024-09-13T11:25:05.701 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.701 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.701 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:25:05.701 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:54.252776+0000", 2024-09-13T11:25:05.701 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0019987260000000001 2024-09-13T11:25:05.701 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.701 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.701 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:25:05.701 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:54.581758+0000", 2024-09-13T11:25:05.701 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.32898232799999999 2024-09-13T11:25:05.702 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.702 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.702 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:25:05.702 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:54.581854+0000", 2024-09-13T11:25:05.702 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.5468000000000001e-05 2024-09-13T11:25:05.702 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.702 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.702 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:05.702 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:54.581922+0000", 2024-09-13T11:25:05.702 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.8239999999999997e-05 2024-09-13T11:25:05.703 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:05.703 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:05.703 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:05.703 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.703 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.703 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop_reply(client.4665.0:140790 3.20 e144/77 ondisk, result = 0)", 2024-09-13T11:25:05.703 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:24:58.385972+0000", 2024-09-13T11:25:05.703 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 7.220855319, 2024-09-13T11:25:05.703 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.51829568500000001, 2024-09-13T11:25:05.703 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:05.703 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:05.703 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:25:05.703 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:05.703 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.704 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:05.704 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.385972+0000", 2024-09-13T11:25:05.704 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:05.704 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.704 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.704 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:05.704 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.385972+0000", 2024-09-13T11:25:05.704 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:05.704 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.704 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.704 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:05.704 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.385974+0000", 2024-09-13T11:25:05.704 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.7400000000000001e-06 2024-09-13T11:25:05.705 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.705 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.705 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:05.705 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.385979+0000", 2024-09-13T11:25:05.705 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.091e-06 2024-09-13T11:25:05.705 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.705 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.705 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:05.705 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.904091+0000", 2024-09-13T11:25:05.705 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.51811194900000002 2024-09-13T11:25:05.705 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.705 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.705 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:05.705 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.904098+0000", 2024-09-13T11:25:05.706 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.0609999999999998e-06 2024-09-13T11:25:05.706 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.706 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.706 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:05.706 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.904156+0000", 2024-09-13T11:25:05.706 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.7902000000000002e-05 2024-09-13T11:25:05.706 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.706 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.706 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:25:05.706 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.904163+0000", 2024-09-13T11:25:05.706 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.1430000000000001e-06 2024-09-13T11:25:05.706 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:05.706 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:05.706 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:05.707 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.904267+0000", 2024-09-13T11:25:05.707 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000104799 2024-09-13T11:25:05.707 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:05.707 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:05.707 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:05.707 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:05.707 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:05.707 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-13T11:25:05.709 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-13T11:25:05.709 INFO:tasks.ceph.osd.13.smithi092.stderr:2024-09-13T11:25:05.702+0000 7f3e5afea640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26350) UID: 0 2024-09-13T11:25:05.809 INFO:tasks.ceph.osd.5.smithi092.stderr:2024-09-13T11:25:05.806+0000 7f6f0b705640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26346) UID: 0 2024-09-13T11:25:05.857 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-09-13T11:25:05.857 INFO:teuthology.orchestra.run.smithi092.stdout: "ops": [ 2024-09-13T11:25:05.857 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:05.858 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop(client.4665.0:149875 3.22 e149/122 3:458c3f75:::benchmark_data_smithi177_31116_object9572:head v 149'4567, mlcod=149'4564)", 2024-09-13T11:25:05.858 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:25:05.856976+0000", 2024-09-13T11:25:05.858 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 0.00024429399999999999, 2024-09-13T11:25:05.858 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000260886, 2024-09-13T11:25:05.858 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:25:05.858 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:25:05.858 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-09-13T11:25:05.858 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:25:05.858 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:05.858 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:25:05.858 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:05.856976+0000", 2024-09-13T11:25:05.858 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:05.858 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:05.859 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:05.859 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:25:05.859 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:05.856976+0000", 2024-09-13T11:25:05.859 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:05.859 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:05.859 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:05.859 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:25:05.859 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:05.856977+0000", 2024-09-13T11:25:05.859 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.2780000000000001e-06 2024-09-13T11:25:05.859 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:05.859 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:05.859 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:25:05.859 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:05.856983+0000", 2024-09-13T11:25:05.860 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.0730000000000001e-06 2024-09-13T11:25:05.860 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:05.860 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:05.860 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:25:05.860 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:05.857001+0000", 2024-09-13T11:25:05.860 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.7804000000000001e-05 2024-09-13T11:25:05.860 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:05.860 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:05.860 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:25:05.860 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:05.857008+0000", 2024-09-13T11:25:05.860 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.3889999999999999e-06 2024-09-13T11:25:05.860 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:05.860 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:05.861 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:25:05.861 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:05.857042+0000", 2024-09-13T11:25:05.861 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.3917000000000001e-05 2024-09-13T11:25:05.861 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:05.861 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:05.861 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:25:05.861 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:05.857053+0000", 2024-09-13T11:25:05.861 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0847e-05 2024-09-13T11:25:05.861 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:05.861 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:25:05.861 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:05.861 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:05.861 INFO:teuthology.orchestra.run.smithi092.stdout: ], 2024-09-13T11:25:05.861 INFO:teuthology.orchestra.run.smithi092.stdout: "num_ops": 1 2024-09-13T11:25:05.862 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-09-13T11:25:05.877 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-13T11:25:05.911 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-13T11:25:05.907+0000 7f6a68775640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26307) UID: 0 2024-09-13T11:25:06.012 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-13T11:25:06.011+0000 7f6a68775640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26307) UID: 0 2024-09-13T11:25:06.062 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-09-13T11:25:06.062 INFO:teuthology.orchestra.run.smithi092.stdout: "ops": [], 2024-09-13T11:25:06.062 INFO:teuthology.orchestra.run.smithi092.stdout: "complaint_time": 30, 2024-09-13T11:25:06.063 INFO:teuthology.orchestra.run.smithi092.stdout: "num_blocked_ops": 0 2024-09-13T11:25:06.063 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-09-13T11:25:06.080 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-13T11:25:06.123 INFO:tasks.ceph.osd.3.smithi177.stderr:2024-09-13T11:25:06.110+0000 7f10adba4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26574) UID: 0 2024-09-13T11:25:06.214 INFO:tasks.ceph.osd.1.smithi092.stderr:2024-09-13T11:25:06.210+0000 7fe7789c5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33718) UID: 0 2024-09-13T11:25:06.215 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-09-13T11:25:06.216 INFO:teuthology.orchestra.run.smithi092.stdout: "size": 20, 2024-09-13T11:25:06.216 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 600, 2024-09-13T11:25:06.216 INFO:teuthology.orchestra.run.smithi092.stdout: "ops": [ 2024-09-13T11:25:06.216 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:06.216 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "MOSDPGPush(3.23 83/78 [PushOp(3:c61f6bab:::benchmark_data_smithi177_31116_object29479:head, version: 59'1860, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:c61f6bab:::benchmark_data_smithi177_31116_object29479:head@59'1860, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:25:06.216 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:40.770864+0000", 2024-09-13T11:25:06.216 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 85.441528457000004, 2024-09-13T11:25:06.216 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.31229662800000002, 2024-09-13T11:25:06.216 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:25:06.216 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:25:06.216 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-09-13T11:25:06.216 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:25:06.217 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:06.217 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:25:06.217 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:40.770864+0000", 2024-09-13T11:25:06.217 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:06.217 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:06.217 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:06.217 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:25:06.217 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:40.770864+0000", 2024-09-13T11:25:06.217 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:06.217 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:06.217 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:06.217 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:25:06.217 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:40.770865+0000", 2024-09-13T11:25:06.218 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.288e-06 2024-09-13T11:25:06.218 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:06.218 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:06.218 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:25:06.218 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:40.770955+0000", 2024-09-13T11:25:06.218 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.9871999999999996e-05 2024-09-13T11:25:06.218 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:06.218 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:06.218 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:25:06.218 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:41.082105+0000", 2024-09-13T11:25:06.218 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.31114963400000001 2024-09-13T11:25:06.218 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:06.218 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:06.218 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:25:06.218 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:41.082136+0000", 2024-09-13T11:25:06.219 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.0972000000000001e-05 2024-09-13T11:25:06.219 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:06.219 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:06.219 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:25:06.219 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:41.082404+0000", 2024-09-13T11:25:06.219 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00026844800000000002 2024-09-13T11:25:06.219 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:06.219 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:06.219 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:25:06.219 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:41.082429+0000", 2024-09-13T11:25:06.219 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.5021e-05 2024-09-13T11:25:06.219 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:06.219 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:06.219 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:25:06.220 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:41.083161+0000", 2024-09-13T11:25:06.220 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000731393 2024-09-13T11:25:06.220 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:06.220 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:25:06.220 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:06.220 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:06.220 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:06.220 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "MOSDPGPush(3.1e 85/78 [PushOp(3:7f40e925:::benchmark_data_smithi177_31116_object14084:head, version: 50'927, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:7f40e925:::benchmark_data_smithi177_31116_object14084:head@50'927, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:25:06.220 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:42.751000+0000", 2024-09-13T11:25:06.220 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 83.461392149999995, 2024-09-13T11:25:06.220 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.19208357500000001, 2024-09-13T11:25:06.220 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:25:06.221 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:25:06.221 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-09-13T11:25:06.221 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:25:06.221 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:06.221 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:25:06.221 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.751000+0000", 2024-09-13T11:25:06.221 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:06.221 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:06.221 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:06.221 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:25:06.221 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.751000+0000", 2024-09-13T11:25:06.222 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:06.222 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:06.222 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:06.222 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:25:06.222 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.751003+0000", 2024-09-13T11:25:06.222 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.0800000000000002e-06 2024-09-13T11:25:06.222 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:06.222 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:06.222 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:25:06.222 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.751069+0000", 2024-09-13T11:25:06.222 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.5417000000000001e-05 2024-09-13T11:25:06.222 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:06.222 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:06.223 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:25:06.223 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.941261+0000", 2024-09-13T11:25:06.223 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.19019233899999999 2024-09-13T11:25:06.223 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:06.223 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:06.223 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:25:06.223 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.941300+0000", 2024-09-13T11:25:06.223 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.9091999999999998e-05 2024-09-13T11:25:06.223 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:06.223 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:06.223 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:25:06.223 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.941519+0000", 2024-09-13T11:25:06.223 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000218521 2024-09-13T11:25:06.224 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:06.224 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:06.224 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:25:06.224 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.941544+0000", 2024-09-13T11:25:06.224 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.5412999999999999e-05 2024-09-13T11:25:06.224 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:06.224 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:06.224 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:25:06.224 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.943084+0000", 2024-09-13T11:25:06.224 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0015397130000000001 2024-09-13T11:25:06.224 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:06.224 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:25:06.225 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:06.225 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:06.225 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:06.225 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:82769 3.1e 3:7a4e88b5:::benchmark_data_smithi177_31116_object82768:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105)", 2024-09-13T11:25:06.225 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:24:03.744632+0000", 2024-09-13T11:25:06.225 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 62.467760005000002, 2024-09-13T11:25:06.225 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.21475722699999999, 2024-09-13T11:25:06.225 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:25:06.225 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:25:06.225 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:06.225 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:25:06.225 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:25:06.225 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:06.226 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 82769 2024-09-13T11:25:06.226 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:06.226 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:25:06.226 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:06.226 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:25:06.226 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:03.744632+0000", 2024-09-13T11:25:06.226 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:06.226 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:06.226 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:06.226 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:25:06.226 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:03.744632+0000", 2024-09-13T11:25:06.226 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:06.226 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:06.226 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:06.227 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:25:06.227 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:03.744634+0000", 2024-09-13T11:25:06.227 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.672e-06 2024-09-13T11:25:06.227 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:06.227 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:06.227 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:25:06.227 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:03.744724+0000", 2024-09-13T11:25:06.227 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.0237999999999999e-05 2024-09-13T11:25:06.227 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:06.227 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:06.227 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:25:06.227 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:03.744736+0000", 2024-09-13T11:25:06.227 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.2038999999999999e-05 2024-09-13T11:25:06.227 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:06.228 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:06.228 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:25:06.228 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:03.744742+0000", 2024-09-13T11:25:06.228 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.2129999999999996e-06 2024-09-13T11:25:06.228 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:06.228 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:06.228 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:25:06.228 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:03.744785+0000", 2024-09-13T11:25:06.228 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.2898000000000002e-05 2024-09-13T11:25:06.228 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:06.228 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:06.228 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:25:06.228 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:03.744894+0000", 2024-09-13T11:25:06.229 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000109073 2024-09-13T11:25:06.229 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:06.229 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:06.229 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 4", 2024-09-13T11:25:06.229 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:03.745009+0000", 2024-09-13T11:25:06.229 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00011428499999999999 2024-09-13T11:25:06.229 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:06.229 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:06.229 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:25:06.229 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:03.746451+0000", 2024-09-13T11:25:06.230 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0014419179999999999 2024-09-13T11:25:06.230 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:06.230 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:06.230 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:25:06.230 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:03.959257+0000", 2024-09-13T11:25:06.230 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.212806361 2024-09-13T11:25:06.230 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:06.230 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:06.230 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:25:06.230 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:03.959333+0000", 2024-09-13T11:25:06.230 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.6001000000000006e-05 2024-09-13T11:25:06.230 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:06.230 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:06.230 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:25:06.231 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:03.959390+0000", 2024-09-13T11:25:06.231 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.6529e-05 2024-09-13T11:25:06.231 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:06.231 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:25:06.231 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:06.231 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:06.231 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:06.231 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:104331 3.1e 3:79890035:::benchmark_data_smithi177_31116_object104330:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-09-13T11:25:06.231 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:24:20.241529+0000", 2024-09-13T11:25:06.231 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 45.970863487000003, 2024-09-13T11:25:06.231 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.16976961700000001, 2024-09-13T11:25:06.231 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:25:06.231 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:25:06.231 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:06.232 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:25:06.232 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:25:06.232 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:06.232 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 104331 2024-09-13T11:25:06.232 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:06.232 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:25:06.232 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:06.232 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:25:06.232 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:20.241529+0000", 2024-09-13T11:25:06.232 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:06.232 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:06.232 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:06.232 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:25:06.232 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:20.241529+0000", 2024-09-13T11:25:06.233 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:06.233 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:06.233 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:06.233 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:25:06.233 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:20.241534+0000", 2024-09-13T11:25:06.233 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.775e-06 2024-09-13T11:25:06.233 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:06.233 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:06.233 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:25:06.233 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:20.241660+0000", 2024-09-13T11:25:06.233 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000126499 2024-09-13T11:25:06.234 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:06.234 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:06.234 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:25:06.234 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:20.241687+0000", 2024-09-13T11:25:06.234 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.6826999999999999e-05 2024-09-13T11:25:06.234 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:06.235 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:06.235 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:25:06.235 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:20.241703+0000", 2024-09-13T11:25:06.235 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.5858999999999998e-05 2024-09-13T11:25:06.235 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:06.235 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:06.235 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:25:06.235 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:20.241832+0000", 2024-09-13T11:25:06.235 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00012925799999999999 2024-09-13T11:25:06.235 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:06.235 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:06.235 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:25:06.235 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:20.242133+0000", 2024-09-13T11:25:06.236 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00030115900000000001 2024-09-13T11:25:06.236 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:06.236 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:06.236 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 4", 2024-09-13T11:25:06.236 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:20.242408+0000", 2024-09-13T11:25:06.236 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00027487799999999999 2024-09-13T11:25:06.236 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:06.236 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:06.236 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:25:06.236 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:20.246683+0000", 2024-09-13T11:25:06.236 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0042750469999999997 2024-09-13T11:25:06.236 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:06.236 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:06.237 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:25:06.237 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:20.411076+0000", 2024-09-13T11:25:06.237 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.16439300800000001 2024-09-13T11:25:06.237 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:06.237 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:06.237 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:25:06.237 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:20.411188+0000", 2024-09-13T11:25:06.237 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00011191 2024-09-13T11:25:06.237 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:06.237 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:06.237 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:25:06.237 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:20.411298+0000", 2024-09-13T11:25:06.238 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00011039699999999999 2024-09-13T11:25:06.238 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:06.238 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:25:06.238 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:06.238 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:06.238 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:06.238 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:110289 3.29 3.27287969 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e119)", 2024-09-13T11:25:06.238 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:24:25.774634+0000", 2024-09-13T11:25:06.238 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 40.437758381999998, 2024-09-13T11:25:06.238 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.31721633999999999, 2024-09-13T11:25:06.238 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:25:06.238 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:25:06.238 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "reached pg", 2024-09-13T11:25:06.238 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:25:06.239 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:25:06.239 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:06.239 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 110289 2024-09-13T11:25:06.239 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:06.239 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:25:06.239 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:06.239 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:25:06.239 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:25.774634+0000", 2024-09-13T11:25:06.239 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:06.239 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:06.239 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:06.239 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:25:06.239 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:25.774634+0000", 2024-09-13T11:25:06.239 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:06.240 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:06.240 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:06.240 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:25:06.240 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:25.774639+0000", 2024-09-13T11:25:06.240 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.6029999999999998e-06 2024-09-13T11:25:06.240 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:06.240 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:06.240 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:25:06.240 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:25.774695+0000", 2024-09-13T11:25:06.240 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.6694999999999998e-05 2024-09-13T11:25:06.240 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:06.240 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:06.240 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:25:06.240 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:25.774728+0000", 2024-09-13T11:25:06.241 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.2502000000000002e-05 2024-09-13T11:25:06.241 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:06.241 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:06.241 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:25:06.241 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:25.774750+0000", 2024-09-13T11:25:06.241 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.2002999999999999e-05 2024-09-13T11:25:06.241 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:06.241 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:06.241 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:25:06.241 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:25.774942+0000", 2024-09-13T11:25:06.241 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000192265 2024-09-13T11:25:06.241 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:06.241 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:06.241 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-09-13T11:25:06.242 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:25.774960+0000", 2024-09-13T11:25:06.242 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.8275999999999999e-05 2024-09-13T11:25:06.242 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:06.242 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:06.242 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:25:06.242 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:26.091821+0000", 2024-09-13T11:25:06.242 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.31686065000000002 2024-09-13T11:25:06.242 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:06.243 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:06.243 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:25:06.243 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:26.091850+0000", 2024-09-13T11:25:06.243 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.9346e-05 2024-09-13T11:25:06.243 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:06.243 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:25:06.243 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:06.243 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:06.243 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:06.243 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "MOSDPGPush(3.22 125/122 [PushOp(3:44e4f9f2:::benchmark_data_smithi177_31116_object16663:head, version: 52'1032, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:44e4f9f2:::benchmark_data_smithi177_31116_object16663:head@52'1032, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:25:06.243 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:24:32.743254+0000", 2024-09-13T11:25:06.243 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 33.469138293, 2024-09-13T11:25:06.244 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.215485287, 2024-09-13T11:25:06.244 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:25:06.244 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:25:06.244 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-09-13T11:25:06.244 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:25:06.244 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:06.244 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:25:06.244 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:32.743254+0000", 2024-09-13T11:25:06.244 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:06.244 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:06.244 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:06.244 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:25:06.245 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:32.743254+0000", 2024-09-13T11:25:06.245 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:06.245 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:06.245 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:06.245 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:25:06.245 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:32.743255+0000", 2024-09-13T11:25:06.245 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.2500000000000004e-07 2024-09-13T11:25:06.245 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:06.245 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:06.245 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:25:06.245 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:32.743310+0000", 2024-09-13T11:25:06.245 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.5442000000000002e-05 2024-09-13T11:25:06.245 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:06.246 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:06.246 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:25:06.246 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:32.957754+0000", 2024-09-13T11:25:06.246 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.21444387000000001 2024-09-13T11:25:06.246 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:06.246 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:06.246 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:25:06.246 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:32.957765+0000", 2024-09-13T11:25:06.246 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0458999999999999e-05 2024-09-13T11:25:06.246 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:06.246 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:06.246 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:25:06.246 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:32.957841+0000", 2024-09-13T11:25:06.247 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.5993000000000002e-05 2024-09-13T11:25:06.247 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:06.247 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:06.247 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:25:06.247 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:32.957852+0000", 2024-09-13T11:25:06.247 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.1453e-05 2024-09-13T11:25:06.247 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:06.247 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:06.247 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:25:06.247 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:32.958739+0000", 2024-09-13T11:25:06.247 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00088724500000000005 2024-09-13T11:25:06.247 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:06.248 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:25:06.248 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:06.248 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:06.248 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:06.248 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop(client.4665.0:115089 3.28 e125/120 3:15b13357:::benchmark_data_smithi177_31116_object115088:head v 125'4843, mlcod=47'620)", 2024-09-13T11:25:06.248 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:24:33.642654+0000", 2024-09-13T11:25:06.248 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 32.569737883000002, 2024-09-13T11:25:06.248 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.19312220099999999, 2024-09-13T11:25:06.248 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:25:06.248 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:25:06.248 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:06.249 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:25:06.249 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:06.249 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:25:06.249 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:33.642654+0000", 2024-09-13T11:25:06.249 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:06.249 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:06.249 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:06.249 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:25:06.249 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:33.642654+0000", 2024-09-13T11:25:06.249 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:06.249 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:06.250 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:06.250 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:25:06.250 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:33.642656+0000", 2024-09-13T11:25:06.250 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.869e-06 2024-09-13T11:25:06.250 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:06.250 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:06.250 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:25:06.250 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:33.642688+0000", 2024-09-13T11:25:06.251 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.2169999999999999e-05 2024-09-13T11:25:06.251 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:06.251 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:06.251 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:25:06.251 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:33.832657+0000", 2024-09-13T11:25:06.251 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.18996885099999999 2024-09-13T11:25:06.251 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:06.251 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:06.251 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:25:06.251 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:33.832678+0000", 2024-09-13T11:25:06.251 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.0222000000000001e-05 2024-09-13T11:25:06.251 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:06.251 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:06.252 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:25:06.252 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:33.832760+0000", 2024-09-13T11:25:06.252 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.2113000000000005e-05 2024-09-13T11:25:06.252 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:06.252 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:06.252 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:25:06.252 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:33.832788+0000", 2024-09-13T11:25:06.252 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.8095000000000001e-05 2024-09-13T11:25:06.252 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:06.252 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:06.252 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:25:06.252 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:33.835729+0000", 2024-09-13T11:25:06.252 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0029408920000000001 2024-09-13T11:25:06.253 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:06.253 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:06.253 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:25:06.253 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:33.835777+0000", 2024-09-13T11:25:06.253 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.7988999999999997e-05 2024-09-13T11:25:06.253 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:06.253 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:25:06.253 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:06.253 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:06.253 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:06.253 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop(client.4665.0:115281 3.28 e125/120 3:17aeecb4:::benchmark_data_smithi177_31116_object115280:head v 125'4845, mlcod=47'620)", 2024-09-13T11:25:06.253 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:24:33.852156+0000", 2024-09-13T11:25:06.254 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 32.360236483000001, 2024-09-13T11:25:06.254 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.16574401999999999, 2024-09-13T11:25:06.254 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:25:06.254 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:25:06.254 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:06.254 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:25:06.254 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:06.254 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:25:06.254 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:33.852156+0000", 2024-09-13T11:25:06.254 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:06.254 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:06.254 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:06.255 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:25:06.255 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:33.852156+0000", 2024-09-13T11:25:06.255 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:06.255 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:06.255 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:06.255 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:25:06.255 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:33.852158+0000", 2024-09-13T11:25:06.255 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.683e-06 2024-09-13T11:25:06.255 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:06.255 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:06.255 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:25:06.255 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:33.852184+0000", 2024-09-13T11:25:06.255 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.6241000000000002e-05 2024-09-13T11:25:06.256 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:06.256 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:06.256 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:25:06.256 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:34.015751+0000", 2024-09-13T11:25:06.256 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.16356685800000001 2024-09-13T11:25:06.256 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:06.256 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:06.256 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:25:06.256 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:34.015757+0000", 2024-09-13T11:25:06.256 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.8980000000000001e-06 2024-09-13T11:25:06.256 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:06.256 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:06.257 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:25:06.257 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:34.015800+0000", 2024-09-13T11:25:06.257 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.3371999999999997e-05 2024-09-13T11:25:06.257 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:06.257 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:06.257 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:25:06.257 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:34.015822+0000", 2024-09-13T11:25:06.257 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.2565000000000002e-05 2024-09-13T11:25:06.257 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:06.257 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:06.257 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:25:06.258 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:34.017861+0000", 2024-09-13T11:25:06.258 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0020388469999999999 2024-09-13T11:25:06.258 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:06.258 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:06.258 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:25:06.258 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:34.017900+0000", 2024-09-13T11:25:06.258 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.8556000000000002e-05 2024-09-13T11:25:06.258 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:06.258 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:25:06.258 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:06.258 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:06.258 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:06.258 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "MOSDPGPush(3.29 134/120 [PushOp(3:95648f22:::benchmark_data_smithi177_31116_object7446:head, version: 46'480, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:95648f22:::benchmark_data_smithi177_31116_object7446:head@46'480, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:25:06.259 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:24:47.108593+0000", 2024-09-13T11:25:06.259 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 19.103799652999999, 2024-09-13T11:25:06.259 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.155295241, 2024-09-13T11:25:06.259 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:25:06.259 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:25:06.259 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-09-13T11:25:06.259 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:25:06.259 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:06.259 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:25:06.259 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:47.108593+0000", 2024-09-13T11:25:06.259 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:06.259 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:06.260 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:06.260 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:25:06.260 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:47.108593+0000", 2024-09-13T11:25:06.260 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:06.260 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:06.260 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:06.260 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:25:06.260 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:47.108594+0000", 2024-09-13T11:25:06.260 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.4190000000000001e-06 2024-09-13T11:25:06.260 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:06.260 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:06.260 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:25:06.260 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:47.108656+0000", 2024-09-13T11:25:06.261 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.1730999999999996e-05 2024-09-13T11:25:06.261 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:06.261 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:06.261 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:25:06.261 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:47.262441+0000", 2024-09-13T11:25:06.261 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.15378476499999999 2024-09-13T11:25:06.261 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:06.264 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:06.264 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:25:06.265 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:47.262449+0000", 2024-09-13T11:25:06.265 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.1049999999999995e-06 2024-09-13T11:25:06.265 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:06.265 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:06.265 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:25:06.265 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:47.262557+0000", 2024-09-13T11:25:06.265 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000108076 2024-09-13T11:25:06.265 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:06.265 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:06.265 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:25:06.265 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:47.262574+0000", 2024-09-13T11:25:06.265 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.6878999999999999e-05 2024-09-13T11:25:06.266 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:06.266 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:06.266 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:25:06.266 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:47.263888+0000", 2024-09-13T11:25:06.266 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0013142659999999999 2024-09-13T11:25:06.266 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:06.266 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:25:06.266 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:06.266 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:06.266 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:06.266 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop(client.4665.0:137296 3.2d e139/120 3:b4062804:::benchmark_data_smithi177_31116_object137295:head v 139'5552, mlcod=46'430)", 2024-09-13T11:25:06.266 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:24:53.347825+0000", 2024-09-13T11:25:06.267 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 12.864567088999999, 2024-09-13T11:25:06.267 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.162969905, 2024-09-13T11:25:06.267 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:25:06.267 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:25:06.267 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:06.267 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:25:06.267 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:06.267 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:25:06.267 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:53.347825+0000", 2024-09-13T11:25:06.267 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:06.267 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:06.267 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:06.267 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:25:06.268 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:53.347825+0000", 2024-09-13T11:25:06.268 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:06.268 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:06.268 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:06.268 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:25:06.268 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:53.347827+0000", 2024-09-13T11:25:06.268 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.517e-06 2024-09-13T11:25:06.268 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:06.268 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:06.268 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:25:06.268 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:53.347889+0000", 2024-09-13T11:25:06.268 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.1798999999999996e-05 2024-09-13T11:25:06.269 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:06.269 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:06.269 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:25:06.269 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:53.508273+0000", 2024-09-13T11:25:06.269 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.160384001 2024-09-13T11:25:06.269 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:06.269 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:06.269 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:25:06.269 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:53.508283+0000", 2024-09-13T11:25:06.269 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0074999999999999e-05 2024-09-13T11:25:06.269 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:06.269 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:06.269 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:25:06.269 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:53.508340+0000", 2024-09-13T11:25:06.270 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.7766000000000002e-05 2024-09-13T11:25:06.270 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:06.270 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:06.270 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:25:06.270 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:53.508358+0000", 2024-09-13T11:25:06.270 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.7940000000000001e-05 2024-09-13T11:25:06.270 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:06.270 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:06.270 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:25:06.270 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:53.510750+0000", 2024-09-13T11:25:06.270 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0023914639999999998 2024-09-13T11:25:06.271 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:06.271 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:06.271 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:25:06.271 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:53.510795+0000", 2024-09-13T11:25:06.271 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.5343000000000003e-05 2024-09-13T11:25:06.271 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:06.271 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:25:06.271 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:06.271 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:06.271 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:06.271 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "MOSDPGPush(3.29 139/120 [PushOp(3:961aa4aa:::benchmark_data_smithi177_31116_object26561:head, version: 58'1688, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:961aa4aa:::benchmark_data_smithi177_31116_object26561:head@58'1688, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:25:06.271 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:24:53.385067+0000", 2024-09-13T11:25:06.271 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 12.827324924999999, 2024-09-13T11:25:06.272 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.185187613, 2024-09-13T11:25:06.272 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:25:06.272 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:25:06.272 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-09-13T11:25:06.272 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:25:06.272 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:06.272 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:25:06.272 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:53.385067+0000", 2024-09-13T11:25:06.272 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:06.272 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:06.272 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:06.272 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:25:06.272 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:53.385067+0000", 2024-09-13T11:25:06.273 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:06.273 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:06.273 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:06.273 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:25:06.273 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:53.385069+0000", 2024-09-13T11:25:06.273 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.4729999999999999e-06 2024-09-13T11:25:06.273 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:06.273 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:06.273 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:25:06.273 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:53.385139+0000", 2024-09-13T11:25:06.273 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.9881000000000004e-05 2024-09-13T11:25:06.273 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:06.273 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:06.273 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:25:06.274 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:53.569032+0000", 2024-09-13T11:25:06.274 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.183893626 2024-09-13T11:25:06.274 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:06.274 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:06.274 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:25:06.274 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:53.569053+0000", 2024-09-13T11:25:06.274 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.0305999999999998e-05 2024-09-13T11:25:06.274 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:06.274 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:06.274 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:25:06.274 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:53.569205+0000", 2024-09-13T11:25:06.274 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00015209699999999999 2024-09-13T11:25:06.274 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:06.275 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:06.275 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:25:06.275 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:53.569214+0000", 2024-09-13T11:25:06.275 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.3740000000000004e-06 2024-09-13T11:25:06.275 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:06.275 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:06.275 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:25:06.275 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:53.570255+0000", 2024-09-13T11:25:06.275 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0010408559999999999 2024-09-13T11:25:06.275 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:06.275 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:25:06.275 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:06.275 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:06.276 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:06.276 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:138370 3.23 3:c5730cd2:::benchmark_data_smithi177_31116_object138369:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e140)", 2024-09-13T11:25:06.276 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:24:54.415176+0000", 2024-09-13T11:25:06.276 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 11.797216285999999, 2024-09-13T11:25:06.276 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.24549895499999999, 2024-09-13T11:25:06.276 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:25:06.276 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:25:06.276 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:06.276 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:25:06.276 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:25:06.276 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:06.277 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 138370 2024-09-13T11:25:06.277 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:06.277 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:25:06.277 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:06.277 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:25:06.277 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:54.415176+0000", 2024-09-13T11:25:06.277 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:06.277 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:06.277 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:06.277 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:25:06.277 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:54.415176+0000", 2024-09-13T11:25:06.277 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:06.277 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:06.277 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:06.278 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:25:06.278 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:54.415180+0000", 2024-09-13T11:25:06.278 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.1670000000000001e-06 2024-09-13T11:25:06.278 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:06.278 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:06.278 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:25:06.278 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:54.415269+0000", 2024-09-13T11:25:06.278 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.8659000000000006e-05 2024-09-13T11:25:06.278 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:06.278 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:06.278 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:25:06.278 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:54.415292+0000", 2024-09-13T11:25:06.279 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.3152000000000001e-05 2024-09-13T11:25:06.279 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:06.279 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:06.279 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:25:06.279 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:54.415304+0000", 2024-09-13T11:25:06.279 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.1841999999999999e-05 2024-09-13T11:25:06.279 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:06.279 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:06.279 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:25:06.279 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:54.415432+0000", 2024-09-13T11:25:06.279 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00012802000000000001 2024-09-13T11:25:06.279 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:06.280 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:06.280 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:25:06.280 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:54.415710+0000", 2024-09-13T11:25:06.280 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00027840299999999998 2024-09-13T11:25:06.280 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:06.280 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:06.280 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 9", 2024-09-13T11:25:06.280 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:54.415950+0000", 2024-09-13T11:25:06.280 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00023928299999999999 2024-09-13T11:25:06.280 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:06.280 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:06.280 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:25:06.280 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:54.419265+0000", 2024-09-13T11:25:06.281 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0033150440000000001 2024-09-13T11:25:06.281 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:06.281 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:06.281 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:25:06.281 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:54.660448+0000", 2024-09-13T11:25:06.281 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.24118348100000001 2024-09-13T11:25:06.281 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:06.281 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:06.281 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:25:06.281 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:54.660571+0000", 2024-09-13T11:25:06.281 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00012332900000000001 2024-09-13T11:25:06.281 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:06.281 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:06.281 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:25:06.282 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:54.660675+0000", 2024-09-13T11:25:06.282 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000103575 2024-09-13T11:25:06.282 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:06.282 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:25:06.282 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:06.282 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:06.282 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:06.282 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop(client.4665.0:138550 3.27 e140/120 3:e494a291:::benchmark_data_smithi177_31116_object138549:head v 140'5229, mlcod=49'686)", 2024-09-13T11:25:06.282 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:24:54.831360+0000", 2024-09-13T11:25:06.282 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 11.381032277999999, 2024-09-13T11:25:06.282 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.176014844, 2024-09-13T11:25:06.282 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:25:06.282 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:25:06.283 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:06.283 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:25:06.283 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:06.283 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:25:06.283 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:54.831360+0000", 2024-09-13T11:25:06.283 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:06.283 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:06.283 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:06.283 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:25:06.283 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:54.831360+0000", 2024-09-13T11:25:06.283 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:06.283 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:06.283 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:06.284 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:25:06.284 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:54.831367+0000", 2024-09-13T11:25:06.284 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.8419999999999999e-06 2024-09-13T11:25:06.284 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:06.284 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:06.284 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:25:06.284 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:54.831426+0000", 2024-09-13T11:25:06.284 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.9072e-05 2024-09-13T11:25:06.284 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:06.284 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:06.284 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:25:06.284 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:54.992898+0000", 2024-09-13T11:25:06.284 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.161471579 2024-09-13T11:25:06.285 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:06.285 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:06.285 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:25:06.285 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:54.992915+0000", 2024-09-13T11:25:06.285 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.7299000000000002e-05 2024-09-13T11:25:06.285 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:06.285 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:06.285 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:25:06.285 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:54.993246+0000", 2024-09-13T11:25:06.285 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00033146199999999998 2024-09-13T11:25:06.285 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:06.285 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:06.285 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:25:06.286 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:54.993391+0000", 2024-09-13T11:25:06.286 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00014489400000000001 2024-09-13T11:25:06.286 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:06.286 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:06.286 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:25:06.286 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:55.007321+0000", 2024-09-13T11:25:06.286 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.013929777000000001 2024-09-13T11:25:06.286 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:06.286 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:06.286 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:25:06.286 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:55.007375+0000", 2024-09-13T11:25:06.286 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.3919000000000003e-05 2024-09-13T11:25:06.286 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:06.287 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:25:06.287 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:06.287 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:06.287 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:06.287 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:140742 3.1e 3:79c694a0:::benchmark_data_smithi177_31116_object439:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e144)", 2024-09-13T11:25:06.287 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:24:58.321262+0000", 2024-09-13T11:25:06.287 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 7.8911298690000002, 2024-09-13T11:25:06.287 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.23727013499999999, 2024-09-13T11:25:06.287 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:25:06.287 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:25:06.287 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:06.287 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:25:06.287 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:25:06.288 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:06.288 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 140742 2024-09-13T11:25:06.288 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:06.288 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:25:06.288 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:06.288 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:25:06.288 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:58.321262+0000", 2024-09-13T11:25:06.288 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:06.288 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:06.288 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:06.288 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:25:06.288 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:58.321262+0000", 2024-09-13T11:25:06.288 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:06.289 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:06.289 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:06.289 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:25:06.289 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:58.321264+0000", 2024-09-13T11:25:06.289 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.581e-06 2024-09-13T11:25:06.289 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:06.289 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:06.289 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:25:06.289 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:58.321272+0000", 2024-09-13T11:25:06.289 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.2179999999999999e-06 2024-09-13T11:25:06.289 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:06.289 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:06.290 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:25:06.290 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:58.321282+0000", 2024-09-13T11:25:06.290 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0173000000000001e-05 2024-09-13T11:25:06.290 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:06.290 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:06.290 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:25:06.290 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:58.321290+0000", 2024-09-13T11:25:06.290 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.943e-06 2024-09-13T11:25:06.290 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:06.290 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:06.290 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:25:06.290 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:58.321355+0000", 2024-09-13T11:25:06.291 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.4461000000000005e-05 2024-09-13T11:25:06.291 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:06.291 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:06.291 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:25:06.291 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:58.321450+0000", 2024-09-13T11:25:06.291 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.4769e-05 2024-09-13T11:25:06.291 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:06.291 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:06.291 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 4", 2024-09-13T11:25:06.291 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:58.321516+0000", 2024-09-13T11:25:06.291 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.5998000000000001e-05 2024-09-13T11:25:06.291 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:06.291 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:06.292 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:25:06.292 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:58.324045+0000", 2024-09-13T11:25:06.292 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0025290579999999998 2024-09-13T11:25:06.292 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:06.292 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:06.292 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:25:06.292 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:58.558244+0000", 2024-09-13T11:25:06.292 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.234199455 2024-09-13T11:25:06.292 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:06.292 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:06.293 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:25:06.293 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:58.558401+0000", 2024-09-13T11:25:06.293 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00015700800000000001 2024-09-13T11:25:06.293 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:06.293 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:06.293 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:25:06.293 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:58.558533+0000", 2024-09-13T11:25:06.293 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000131471 2024-09-13T11:25:06.293 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:06.293 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:25:06.293 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:06.293 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:06.294 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:06.294 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "MOSDPGPush(3.12 144/122 [PushOp(3:4e80bd1d:::benchmark_data_smithi177_31116_object55197:head, version: 79'3121, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:4e80bd1d:::benchmark_data_smithi177_31116_object55197:head@79'3121, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:25:06.294 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:24:58.397928+0000", 2024-09-13T11:25:06.294 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 7.814463935, 2024-09-13T11:25:06.294 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.16235765499999999, 2024-09-13T11:25:06.294 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:25:06.294 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:25:06.294 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-09-13T11:25:06.294 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:25:06.294 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:06.294 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:25:06.294 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:58.397928+0000", 2024-09-13T11:25:06.294 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:06.295 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:06.295 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:06.295 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:25:06.295 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:58.397928+0000", 2024-09-13T11:25:06.295 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:06.295 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:06.295 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:06.295 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:25:06.295 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:58.397930+0000", 2024-09-13T11:25:06.295 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.818e-06 2024-09-13T11:25:06.295 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:06.295 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:06.295 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:25:06.296 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:58.397994+0000", 2024-09-13T11:25:06.296 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.4071000000000004e-05 2024-09-13T11:25:06.296 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:06.296 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:06.296 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:25:06.296 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:58.555948+0000", 2024-09-13T11:25:06.296 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.15795406000000001 2024-09-13T11:25:06.296 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:06.296 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:06.296 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:25:06.296 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:58.556000+0000", 2024-09-13T11:25:06.296 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.1635999999999998e-05 2024-09-13T11:25:06.297 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:06.297 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:06.297 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:25:06.297 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:58.556278+0000", 2024-09-13T11:25:06.297 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00027833600000000001 2024-09-13T11:25:06.297 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:06.297 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:06.297 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:25:06.297 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:58.556329+0000", 2024-09-13T11:25:06.297 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.0429000000000001e-05 2024-09-13T11:25:06.297 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:06.297 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:06.298 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:25:06.298 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:58.560286+0000", 2024-09-13T11:25:06.298 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0039573050000000004 2024-09-13T11:25:06.298 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:06.298 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:25:06.298 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:06.298 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:06.298 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:06.298 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:140796 3.1e 3:7987a652:::benchmark_data_smithi177_31116_object493:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e144)", 2024-09-13T11:25:06.298 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:24:58.552900+0000", 2024-09-13T11:25:06.298 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 7.6594923340000003, 2024-09-13T11:25:06.298 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.254397398, 2024-09-13T11:25:06.299 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:25:06.299 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:25:06.299 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:06.299 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:25:06.299 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:25:06.299 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:06.299 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 140796 2024-09-13T11:25:06.299 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:06.299 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:25:06.299 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:06.299 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:25:06.299 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:58.552900+0000", 2024-09-13T11:25:06.299 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:06.299 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:06.300 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:06.300 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:25:06.300 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:58.552900+0000", 2024-09-13T11:25:06.300 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:06.300 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:06.300 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:06.300 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:25:06.300 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:58.552904+0000", 2024-09-13T11:25:06.300 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.76e-06 2024-09-13T11:25:06.300 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:06.300 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:06.300 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:25:06.300 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:58.552915+0000", 2024-09-13T11:25:06.301 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0763e-05 2024-09-13T11:25:06.301 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:06.301 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:06.301 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:25:06.301 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:58.552943+0000", 2024-09-13T11:25:06.301 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.8469000000000001e-05 2024-09-13T11:25:06.301 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:06.301 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:06.301 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:25:06.301 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:58.552959+0000", 2024-09-13T11:25:06.301 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.5549999999999999e-05 2024-09-13T11:25:06.301 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:06.302 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:06.302 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:25:06.302 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:58.553106+0000", 2024-09-13T11:25:06.302 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00014729500000000001 2024-09-13T11:25:06.302 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:06.302 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:06.302 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:25:06.302 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:58.553348+0000", 2024-09-13T11:25:06.302 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000242353 2024-09-13T11:25:06.302 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:06.302 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:06.302 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 4", 2024-09-13T11:25:06.302 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:58.553556+0000", 2024-09-13T11:25:06.303 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00020765700000000001 2024-09-13T11:25:06.303 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:06.303 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:06.303 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:25:06.303 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:58.557140+0000", 2024-09-13T11:25:06.303 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0035844840000000002 2024-09-13T11:25:06.303 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:06.303 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:06.303 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:25:06.303 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:58.807126+0000", 2024-09-13T11:25:06.303 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.24998542000000001 2024-09-13T11:25:06.303 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:06.303 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:06.304 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:25:06.304 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:58.807214+0000", 2024-09-13T11:25:06.304 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.8561999999999997e-05 2024-09-13T11:25:06.304 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:06.304 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:06.304 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:25:06.304 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:58.807297+0000", 2024-09-13T11:25:06.304 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.3084999999999995e-05 2024-09-13T11:25:06.304 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:06.304 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:25:06.304 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:06.304 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:06.305 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:06.305 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "MOSDPGPush(3.12 144/122 [PushOp(3:4cd0194d:::benchmark_data_smithi177_31116_object55212:head, version: 79'3122, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:4cd0194d:::benchmark_data_smithi177_31116_object55212:head@79'3122, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:25:06.305 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:24:58.572241+0000", 2024-09-13T11:25:06.305 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 7.6401517089999995, 2024-09-13T11:25:06.305 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.20343498300000001, 2024-09-13T11:25:06.305 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:25:06.305 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:25:06.305 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-09-13T11:25:06.305 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:25:06.305 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:06.305 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:25:06.305 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:58.572241+0000", 2024-09-13T11:25:06.305 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:06.306 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:06.306 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:06.306 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:25:06.306 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:58.572241+0000", 2024-09-13T11:25:06.306 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:06.306 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:06.306 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:06.306 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:25:06.306 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:58.572244+0000", 2024-09-13T11:25:06.306 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.4929999999999998e-06 2024-09-13T11:25:06.306 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:06.306 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:06.306 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:25:06.306 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:58.572321+0000", 2024-09-13T11:25:06.307 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.6983000000000005e-05 2024-09-13T11:25:06.307 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:06.307 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:06.307 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:25:06.307 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:58.771433+0000", 2024-09-13T11:25:06.307 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.19911215299999999 2024-09-13T11:25:06.307 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:06.307 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:06.307 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:25:06.307 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:58.771462+0000", 2024-09-13T11:25:06.307 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.8345999999999999e-05 2024-09-13T11:25:06.307 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:06.308 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:06.308 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:25:06.308 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:58.771615+0000", 2024-09-13T11:25:06.308 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000152981 2024-09-13T11:25:06.308 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:06.308 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:06.308 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:25:06.308 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:58.771638+0000", 2024-09-13T11:25:06.308 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.3516e-05 2024-09-13T11:25:06.308 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:06.308 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:06.308 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:25:06.308 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:58.775676+0000", 2024-09-13T11:25:06.309 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0040375109999999997 2024-09-13T11:25:06.309 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:06.309 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:25:06.309 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:06.309 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:06.309 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:06.309 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop(client.4665.0:147266 3.2d e148/120 3:b77e61ed:::benchmark_data_smithi177_31116_object6963:head v 148'5717, mlcod=46'430)", 2024-09-13T11:25:06.309 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:25:03.948902+0000", 2024-09-13T11:25:06.309 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 2.2634904950000001, 2024-09-13T11:25:06.309 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.177609762, 2024-09-13T11:25:06.309 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:25:06.309 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:25:06.310 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:06.310 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:25:06.310 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:06.310 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:25:06.310 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:03.948902+0000", 2024-09-13T11:25:06.310 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:06.310 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:06.310 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:06.310 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:25:06.310 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:03.948902+0000", 2024-09-13T11:25:06.310 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:06.310 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:06.311 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:06.311 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:25:06.311 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:03.948903+0000", 2024-09-13T11:25:06.311 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.6020000000000001e-06 2024-09-13T11:25:06.311 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:06.311 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:06.311 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:25:06.311 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:03.948911+0000", 2024-09-13T11:25:06.311 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.7020000000000001e-06 2024-09-13T11:25:06.311 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:06.311 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:06.311 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:25:06.311 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:04.123318+0000", 2024-09-13T11:25:06.312 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.174406905 2024-09-13T11:25:06.312 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:06.312 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:06.312 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:25:06.312 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:04.123322+0000", 2024-09-13T11:25:06.312 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.3919999999999996e-06 2024-09-13T11:25:06.312 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:06.312 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:06.312 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:25:06.312 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:04.123378+0000", 2024-09-13T11:25:06.312 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.5930000000000002e-05 2024-09-13T11:25:06.313 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:06.313 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:06.313 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:25:06.313 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:04.123405+0000", 2024-09-13T11:25:06.313 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.6570000000000001e-05 2024-09-13T11:25:06.313 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:06.313 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:06.313 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:25:06.313 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:04.126478+0000", 2024-09-13T11:25:06.313 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0030730699999999998 2024-09-13T11:25:06.313 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:06.313 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:06.313 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:25:06.313 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:04.126512+0000", 2024-09-13T11:25:06.314 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.3590999999999997e-05 2024-09-13T11:25:06.314 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:06.314 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:25:06.314 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:06.314 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:06.314 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:06.314 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop(client.4665.0:147469 3.2d e148/120 3:b79b6f50:::benchmark_data_smithi177_31116_object7166:head v 148'5719, mlcod=46'431)", 2024-09-13T11:25:06.314 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:25:04.133079+0000", 2024-09-13T11:25:06.314 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 2.0793134150000001, 2024-09-13T11:25:06.314 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.16502599900000001, 2024-09-13T11:25:06.314 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:25:06.314 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:25:06.314 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:06.315 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:25:06.315 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:06.315 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:25:06.315 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:04.133079+0000", 2024-09-13T11:25:06.315 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:06.315 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:06.315 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:06.315 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:25:06.315 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:04.133079+0000", 2024-09-13T11:25:06.315 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:06.315 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:06.315 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:06.315 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:25:06.316 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:04.133081+0000", 2024-09-13T11:25:06.316 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.624e-06 2024-09-13T11:25:06.316 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:06.316 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:06.316 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:25:06.316 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:04.133087+0000", 2024-09-13T11:25:06.316 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.7530000000000004e-06 2024-09-13T11:25:06.316 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:06.316 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:06.316 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:25:06.316 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:04.296266+0000", 2024-09-13T11:25:06.316 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.163178985 2024-09-13T11:25:06.317 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:06.317 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:06.317 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:25:06.317 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:04.296272+0000", 2024-09-13T11:25:06.317 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.0170000000000002e-06 2024-09-13T11:25:06.317 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:06.317 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:06.317 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:25:06.317 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:04.296327+0000", 2024-09-13T11:25:06.317 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.4496000000000001e-05 2024-09-13T11:25:06.317 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:06.317 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:06.318 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:25:06.318 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:04.296343+0000", 2024-09-13T11:25:06.318 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.649e-05 2024-09-13T11:25:06.318 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:06.318 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:06.318 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:25:06.318 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:04.298072+0000", 2024-09-13T11:25:06.318 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.001728222 2024-09-13T11:25:06.318 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:06.318 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:06.318 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:25:06.318 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:04.298105+0000", 2024-09-13T11:25:06.318 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.3411999999999998e-05 2024-09-13T11:25:06.319 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:06.319 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:25:06.319 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:06.319 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:06.319 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:06.319 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop(client.4665.0:147472 3.2d e148/120 3:b6aeefb0:::benchmark_data_smithi177_31116_object7169:head v 148'5720, mlcod=46'431)", 2024-09-13T11:25:06.319 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:25:04.137580+0000", 2024-09-13T11:25:06.319 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 2.0748122599999999, 2024-09-13T11:25:06.319 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.214816644, 2024-09-13T11:25:06.319 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:25:06.319 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:25:06.319 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:06.319 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:25:06.319 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:06.320 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:25:06.320 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:04.137580+0000", 2024-09-13T11:25:06.320 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:06.320 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:06.320 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:06.320 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:25:06.320 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:04.137580+0000", 2024-09-13T11:25:06.320 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:06.320 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:06.320 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:06.320 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:25:06.320 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:04.137581+0000", 2024-09-13T11:25:06.320 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.2309999999999999e-06 2024-09-13T11:25:06.321 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:06.321 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:06.321 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:25:06.321 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:04.137587+0000", 2024-09-13T11:25:06.321 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.879e-06 2024-09-13T11:25:06.321 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:06.321 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:06.321 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:25:06.321 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:04.350285+0000", 2024-09-13T11:25:06.321 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.212697737 2024-09-13T11:25:06.321 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:06.321 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:06.321 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:25:06.321 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:04.350291+0000", 2024-09-13T11:25:06.322 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.9070000000000004e-06 2024-09-13T11:25:06.322 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:06.322 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:06.322 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:25:06.322 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:04.350368+0000", 2024-09-13T11:25:06.322 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.7150000000000005e-05 2024-09-13T11:25:06.322 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:06.322 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:06.322 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:25:06.322 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:04.350407+0000", 2024-09-13T11:25:06.322 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.9433e-05 2024-09-13T11:25:06.322 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:06.322 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:06.322 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:25:06.323 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:04.352361+0000", 2024-09-13T11:25:06.323 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0019532379999999999 2024-09-13T11:25:06.323 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:06.323 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:06.323 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:25:06.323 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:04.352397+0000", 2024-09-13T11:25:06.323 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.6069000000000003e-05 2024-09-13T11:25:06.323 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:06.323 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:25:06.323 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:06.323 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:06.323 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:25:06.323 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-09-13T11:25:06.325 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-09-13T11:25:06.326 INFO:tasks.ceph.osd.2.smithi162.stderr:2024-09-13T11:25:06.320+0000 7fe92b35a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26313) UID: 0 2024-09-13T11:25:06.354 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T11:25:06.426 INFO:tasks.ceph.osd.14.smithi162.stderr:2024-09-13T11:25:06.424+0000 7f3df5903640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33154) UID: 0 2024-09-13T11:25:06.512 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-09-13T11:25:06.513 INFO:teuthology.orchestra.run.smithi092.stdout: "ops": [], 2024-09-13T11:25:06.513 INFO:teuthology.orchestra.run.smithi092.stdout: "num_ops": 0 2024-09-13T11:25:06.513 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-09-13T11:25:06.517 INFO:teuthology.orchestra.run.smithi001.stdout:osd.0: {} 2024-09-13T11:25:06.517 INFO:teuthology.orchestra.run.smithi001.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T11:25:06.527 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-13T11:25:06.523+0000 7f6a68775640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26307) UID: 0 2024-09-13T11:25:06.531 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-09-13T11:25:06.547 INFO:teuthology.orchestra.run.smithi001.stdout:osd.1: {} 2024-09-13T11:25:06.548 INFO:teuthology.orchestra.run.smithi001.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T11:25:06.577 INFO:teuthology.orchestra.run.smithi001.stdout:osd.2: {} 2024-09-13T11:25:06.577 INFO:teuthology.orchestra.run.smithi001.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T11:25:06.602 INFO:teuthology.orchestra.run.smithi001.stdout:osd.3: {} 2024-09-13T11:25:06.603 INFO:teuthology.orchestra.run.smithi001.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T11:25:06.628 INFO:teuthology.orchestra.run.smithi001.stdout:osd.4: {} 2024-09-13T11:25:06.628 INFO:teuthology.orchestra.run.smithi001.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T11:25:06.628 INFO:tasks.ceph.osd.14.smithi162.stderr:2024-09-13T11:25:06.624+0000 7f3df5903640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33154) UID: 0 2024-09-13T11:25:06.653 INFO:teuthology.orchestra.run.smithi001.stdout:osd.5: {} 2024-09-13T11:25:06.653 INFO:teuthology.orchestra.run.smithi001.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T11:25:06.668 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-09-13T11:25:06.668 INFO:teuthology.orchestra.run.smithi092.stdout: "ops": [], 2024-09-13T11:25:06.668 INFO:teuthology.orchestra.run.smithi092.stdout: "complaint_time": 30, 2024-09-13T11:25:06.668 INFO:teuthology.orchestra.run.smithi092.stdout: "num_blocked_ops": 0 2024-09-13T11:25:06.668 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-09-13T11:25:06.684 INFO:teuthology.orchestra.run.smithi001.stdout:osd.6: {} 2024-09-13T11:25:06.684 INFO:teuthology.orchestra.run.smithi001.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T11:25:06.686 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-09-13T11:25:06.709 INFO:teuthology.orchestra.run.smithi001.stdout:osd.7: {} 2024-09-13T11:25:06.710 INFO:teuthology.orchestra.run.smithi001.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T11:25:06.733 INFO:teuthology.orchestra.run.smithi001.stdout:osd.8: {} 2024-09-13T11:25:06.733 INFO:teuthology.orchestra.run.smithi001.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-13T11:25:06.733 INFO:tasks.ceph.osd.5.smithi092.stderr:2024-09-13T11:25:06.729+0000 7f6f0b705640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26346) UID: 0 2024-09-13T11:25:06.760 INFO:teuthology.orchestra.run.smithi001.stdout:osd.9: {} 2024-09-13T11:25:06.760 INFO:teuthology.orchestra.run.smithi001.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-13T11:25:06.787 INFO:teuthology.orchestra.run.smithi001.stdout:osd.10: {} 2024-09-13T11:25:06.787 INFO:teuthology.orchestra.run.smithi001.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-13T11:25:06.811 INFO:teuthology.orchestra.run.smithi001.stdout:osd.11: {} 2024-09-13T11:25:06.811 INFO:teuthology.orchestra.run.smithi001.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-13T11:25:06.821 INFO:teuthology.orchestra.run.smithi092.stdout:ERROR: (22) Invalid argument 2024-09-13T11:25:06.821 INFO:teuthology.orchestra.run.smithi092.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:25:06.830 INFO:tasks.ceph.osd.13.smithi092.stderr:2024-09-13T11:25:06.825+0000 7f3e5afea640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26350) UID: 0 2024-09-13T11:25:06.836 INFO:teuthology.orchestra.run.smithi001.stdout:osd.12: {} 2024-09-13T11:25:06.836 INFO:teuthology.orchestra.run.smithi001.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-13T11:25:06.840 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_ops_in_flight 2024-09-13T11:25:06.931 INFO:tasks.ceph.osd.5.smithi092.stderr:2024-09-13T11:25:06.925+0000 7f6f0b705640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26346) UID: 0 2024-09-13T11:25:06.940 INFO:tasks.ceph.osd.4.smithi001.stderr:2024-09-13T11:25:06.939+0000 7ff8d2c41640 -1 osd.4 149 handle_osd_map injecting map crc failure 2024-09-13T11:25:06.988 INFO:teuthology.orchestra.run.smithi001.stdout:osd.13: {} 2024-09-13T11:25:06.988 INFO:teuthology.orchestra.run.smithi001.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-13T11:25:07.014 INFO:teuthology.orchestra.run.smithi001.stdout:osd.14: {} 2024-09-13T11:25:07.014 INFO:teuthology.orchestra.run.smithi001.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-13T11:25:07.025 INFO:teuthology.orchestra.run.smithi092.stdout:ERROR: (22) Invalid argument 2024-09-13T11:25:07.025 INFO:teuthology.orchestra.run.smithi092.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:25:07.032 INFO:tasks.ceph.osd.5.smithi092.stderr:2024-09-13T11:25:07.025+0000 7f6f0b705640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26346) UID: 0 2024-09-13T11:25:07.040 INFO:teuthology.orchestra.run.smithi001.stdout:osd.15: {} 2024-09-13T11:25:07.040 INFO:teuthology.orchestra.run.smithi001.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-13T11:25:07.045 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_blocked_ops 2024-09-13T11:25:07.133 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-09-13T11:25:07.128+0000 7f97befc9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26315) UID: 0 2024-09-13T11:25:07.166 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-13T11:25:07.234 INFO:tasks.ceph.osd.4.smithi001.stderr:2024-09-13T11:25:07.231+0000 7ff8e0218640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26310) UID: 0 2024-09-13T11:25:07.236 INFO:teuthology.orchestra.run.smithi092.stdout:ERROR: (22) Invalid argument 2024-09-13T11:25:07.236 INFO:teuthology.orchestra.run.smithi092.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:25:07.256 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_historic_ops 2024-09-13T11:25:07.335 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-09-13T11:25:07.332+0000 7f97befc9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26315) UID: 0 2024-09-13T11:25:07.415 INFO:teuthology.orchestra.run.smithi092.stdout:ERROR: (22) Invalid argument 2024-09-13T11:25:07.415 INFO:teuthology.orchestra.run.smithi092.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:25:07.433 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-13T11:25:07.436 INFO:tasks.ceph.osd.13.smithi092.stderr:2024-09-13T11:25:07.429+0000 7f3e5afea640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26350) UID: 0 2024-09-13T11:25:07.537 INFO:tasks.ceph.osd.10.smithi162.stderr:2024-09-13T11:25:07.532+0000 7fb59c9f0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26317) UID: 0 2024-09-13T11:25:07.573 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-13T11:25:07.573 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:25:07.589 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-13T11:25:07.639 INFO:tasks.ceph.osd.5.smithi092.stderr:2024-09-13T11:25:07.633+0000 7f6f0b705640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26346) UID: 0 2024-09-13T11:25:07.739 INFO:tasks.ceph.osd.5.smithi092.stderr:2024-09-13T11:25:07.733+0000 7f6f0b705640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26346) UID: 0 2024-09-13T11:25:07.766 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-13T11:25:07.766 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:25:07.782 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-13T11:25:07.840 INFO:tasks.ceph.osd.5.smithi092.stderr:2024-09-13T11:25:07.833+0000 7f6f0b705640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26346) UID: 0 2024-09-13T11:25:07.910 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-13T11:25:07.911 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:25:07.927 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-13T11:25:07.942 INFO:tasks.ceph.osd.7.smithi177.stderr:2024-09-13T11:25:07.938+0000 7f897b51d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26575) UID: 0 2024-09-13T11:25:08.042 INFO:tasks.ceph.osd.10.smithi162.stderr:2024-09-13T11:25:08.040+0000 7fb59c9f0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26317) UID: 0 2024-09-13T11:25:08.055 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-13T11:25:08.055 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:25:08.061 INFO:teuthology.orchestra.run.smithi001.stderr:noscrub is unset 2024-09-13T11:25:08.066 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-09-13T11:25:08.064+0000 7f97b59fa640 -1 osd.6 150 handle_osd_map injecting map crc failure 2024-09-13T11:25:08.068 INFO:tasks.ceph.osd.10.smithi162.stderr:2024-09-13T11:25:08.064+0000 7fb593421640 -1 osd.10 150 handle_osd_map injecting map crc failure 2024-09-13T11:25:08.075 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-13T11:25:08.143 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-13T11:25:08.139+0000 7f6a68775640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26307) UID: 0 2024-09-13T11:25:08.243 INFO:tasks.ceph.osd.10.smithi162.stderr:2024-09-13T11:25:08.240+0000 7fb59c9f0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26317) UID: 0 2024-09-13T11:25:08.250 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-13T11:25:08.250 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:25:08.269 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-13T11:25:08.345 INFO:tasks.ceph.osd.9.smithi092.stderr:2024-09-13T11:25:08.337+0000 7ff5ab3d0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26348) UID: 0 2024-09-13T11:25:08.445 INFO:tasks.ceph.osd.15.smithi177.stderr:2024-09-13T11:25:08.442+0000 7f9d6ce77640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26578) UID: 0 2024-09-13T11:25:08.447 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-13T11:25:08.448 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:25:08.463 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-09-13T11:25:08.547 INFO:tasks.ceph.osd.7.smithi177.stderr:2024-09-13T11:25:08.542+0000 7f897b51d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26575) UID: 0 2024-09-13T11:25:08.603 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-13T11:25:08.603 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:25:08.620 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-09-13T11:25:08.665 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-09-13T11:25:08.664+0000 7f97befc9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26315) UID: 0 2024-09-13T11:25:08.749 INFO:tasks.ceph.osd.5.smithi092.stderr:2024-09-13T11:25:08.745+0000 7f6f0b705640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26346) UID: 0 2024-09-13T11:25:08.760 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-13T11:25:08.760 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:25:08.778 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-09-13T11:25:08.850 INFO:tasks.ceph.osd.7.smithi177.stderr:2024-09-13T11:25:08.846+0000 7f897b51d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26575) UID: 0 2024-09-13T11:25:08.951 INFO:tasks.ceph.osd.8.smithi001.stderr:2024-09-13T11:25:08.947+0000 7fb9b48ae640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26312) UID: 0 2024-09-13T11:25:08.952 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-13T11:25:08.953 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:25:08.970 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-13T11:25:09.052 INFO:tasks.ceph.osd.1.smithi092.stderr:2024-09-13T11:25:09.045+0000 7fe7789c5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33718) UID: 0 2024-09-13T11:25:09.065 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T11:25:09.116 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-09-13T11:25:09.116 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:25:09.140 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-13T11:25:09.188 INFO:tasks.ceph.osd.11.smithi177.stderr:2024-09-13T11:25:09.186+0000 7f24792c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26580) UID: 0 2024-09-13T11:25:09.243 INFO:teuthology.orchestra.run.smithi001.stdout:osd.0: {} 2024-09-13T11:25:09.243 INFO:teuthology.orchestra.run.smithi001.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T11:25:09.254 INFO:tasks.ceph.osd.1.smithi092.stderr:2024-09-13T11:25:09.249+0000 7fe7789c5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33718) UID: 0 2024-09-13T11:25:09.269 INFO:teuthology.orchestra.run.smithi001.stdout:osd.1: {} 2024-09-13T11:25:09.270 INFO:teuthology.orchestra.run.smithi001.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T11:25:09.292 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-09-13T11:25:09.293 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:25:09.295 INFO:teuthology.orchestra.run.smithi001.stdout:osd.2: {} 2024-09-13T11:25:09.295 INFO:teuthology.orchestra.run.smithi001.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T11:25:09.313 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-13T11:25:09.318 INFO:teuthology.orchestra.run.smithi001.stdout:osd.3: {} 2024-09-13T11:25:09.319 INFO:teuthology.orchestra.run.smithi001.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T11:25:09.343 INFO:teuthology.orchestra.run.smithi001.stdout:osd.4: {} 2024-09-13T11:25:09.343 INFO:teuthology.orchestra.run.smithi001.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T11:25:09.354 INFO:tasks.ceph.osd.12.smithi001.stderr:2024-09-13T11:25:09.351+0000 7f892c6e6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26314) UID: 0 2024-09-13T11:25:09.373 INFO:teuthology.orchestra.run.smithi001.stdout:osd.5: {} 2024-09-13T11:25:09.373 INFO:teuthology.orchestra.run.smithi001.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T11:25:09.401 INFO:teuthology.orchestra.run.smithi001.stdout:osd.6: {} 2024-09-13T11:25:09.401 INFO:teuthology.orchestra.run.smithi001.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T11:25:09.427 INFO:teuthology.orchestra.run.smithi001.stdout:osd.7: {} 2024-09-13T11:25:09.428 INFO:teuthology.orchestra.run.smithi001.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T11:25:09.453 INFO:teuthology.orchestra.run.smithi001.stdout:osd.8: {} 2024-09-13T11:25:09.454 INFO:teuthology.orchestra.run.smithi001.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-13T11:25:09.455 INFO:tasks.ceph.osd.11.smithi177.stderr:2024-09-13T11:25:09.450+0000 7f24792c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26580) UID: 0 2024-09-13T11:25:09.480 INFO:teuthology.orchestra.run.smithi001.stdout:osd.9: {} 2024-09-13T11:25:09.480 INFO:teuthology.orchestra.run.smithi001.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-13T11:25:09.509 INFO:teuthology.orchestra.run.smithi001.stdout:osd.10: {} 2024-09-13T11:25:09.510 INFO:teuthology.orchestra.run.smithi001.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-13T11:25:09.519 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-09-13T11:25:09.519 INFO:teuthology.orchestra.run.smithi177.stdout: "size": 20, 2024-09-13T11:25:09.519 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 600, 2024-09-13T11:25:09.519 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [ 2024-09-13T11:25:09.519 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:09.519 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPush(3.17 75/70 [PushOp(3:eda94455:::benchmark_data_smithi177_31116_object13017:head, version: 50'788, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:eda94455:::benchmark_data_smithi177_31116_object13017:head@50'788, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:25:09.519 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:31.038029+0000", 2024-09-13T11:25:09.519 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 98.478749132999994, 2024-09-13T11:25:09.519 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.42064623499999998, 2024-09-13T11:25:09.519 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:25:09.520 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:25:09.520 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:25:09.520 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:25:09.520 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:09.520 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:25:09.520 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.038029+0000", 2024-09-13T11:25:09.520 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:09.520 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:09.520 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:09.520 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:25:09.520 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.038029+0000", 2024-09-13T11:25:09.520 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:09.520 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:09.520 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:09.521 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:25:09.521 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.038030+0000", 2024-09-13T11:25:09.521 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.499e-06 2024-09-13T11:25:09.521 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:09.521 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:09.521 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:25:09.521 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.038088+0000", 2024-09-13T11:25:09.521 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.7771e-05 2024-09-13T11:25:09.521 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:09.521 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:09.521 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:25:09.521 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.457025+0000", 2024-09-13T11:25:09.522 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.41893754900000002 2024-09-13T11:25:09.522 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:09.522 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:09.522 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:25:09.522 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.457073+0000", 2024-09-13T11:25:09.522 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.7450999999999997e-05 2024-09-13T11:25:09.522 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:09.522 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:09.522 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:25:09.522 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.457304+0000", 2024-09-13T11:25:09.522 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000231351 2024-09-13T11:25:09.522 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:09.522 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:09.522 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:25:09.523 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.457331+0000", 2024-09-13T11:25:09.523 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.6372000000000001e-05 2024-09-13T11:25:09.523 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:09.523 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:09.523 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:25:09.523 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.458675+0000", 2024-09-13T11:25:09.523 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001344242 2024-09-13T11:25:09.523 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:09.523 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:25:09.523 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:09.523 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:09.523 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:09.523 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPush(3.17 83/70 [PushOp(3:eb574268:::benchmark_data_smithi177_31116_object40622:head, version: 64'2475, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:eb574268:::benchmark_data_smithi177_31116_object40622:head@64'2475, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:25:09.524 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:40.760042+0000", 2024-09-13T11:25:09.524 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 88.756735415999998, 2024-09-13T11:25:09.524 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.30544642100000002, 2024-09-13T11:25:09.524 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:25:09.524 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:25:09.524 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:25:09.524 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:25:09.524 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:09.524 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:25:09.524 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:40.760042+0000", 2024-09-13T11:25:09.524 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:09.524 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:09.524 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:09.524 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:25:09.525 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:40.760042+0000", 2024-09-13T11:25:09.525 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:09.525 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:09.525 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:09.525 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:25:09.525 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:40.760044+0000", 2024-09-13T11:25:09.525 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.511e-06 2024-09-13T11:25:09.525 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:09.525 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:09.525 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:25:09.525 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:40.760149+0000", 2024-09-13T11:25:09.525 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000105473 2024-09-13T11:25:09.525 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:09.525 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:09.526 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:25:09.526 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.064465+0000", 2024-09-13T11:25:09.526 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.30431613800000001 2024-09-13T11:25:09.526 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:09.526 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:09.526 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:25:09.526 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.064479+0000", 2024-09-13T11:25:09.526 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3586e-05 2024-09-13T11:25:09.526 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:09.526 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:09.526 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:25:09.527 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.064553+0000", 2024-09-13T11:25:09.527 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.3646000000000006e-05 2024-09-13T11:25:09.527 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:09.527 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:09.527 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:25:09.527 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.064568+0000", 2024-09-13T11:25:09.527 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.52e-05 2024-09-13T11:25:09.528 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:09.528 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:09.528 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:25:09.528 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.065489+0000", 2024-09-13T11:25:09.528 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00092086700000000002 2024-09-13T11:25:09.528 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:09.528 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:25:09.528 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:09.528 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:09.528 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:09.528 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPushReply(3.1a 83/78 [PushReplyOp(3:5c3834bd:::benchmark_data_smithi177_31116_object8669:head)])", 2024-09-13T11:25:09.529 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:40.798236+0000", 2024-09-13T11:25:09.529 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 88.718541986000005, 2024-09-13T11:25:09.529 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.29701520999999997, 2024-09-13T11:25:09.529 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:25:09.529 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:25:09.529 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-13T11:25:09.529 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:25:09.529 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:09.530 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:25:09.530 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:40.798236+0000", 2024-09-13T11:25:09.530 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:09.530 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:09.530 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:09.530 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:25:09.530 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:40.798236+0000", 2024-09-13T11:25:09.530 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:09.530 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:09.530 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:09.530 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:25:09.530 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:40.798237+0000", 2024-09-13T11:25:09.530 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.0500000000000002e-07 2024-09-13T11:25:09.531 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:09.531 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:09.531 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:25:09.531 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:40.798240+0000", 2024-09-13T11:25:09.531 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.4819999999999999e-06 2024-09-13T11:25:09.531 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:09.531 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:09.531 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:25:09.531 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.093360+0000", 2024-09-13T11:25:09.531 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.29512009 2024-09-13T11:25:09.532 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:09.532 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:09.532 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:25:09.532 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.093372+0000", 2024-09-13T11:25:09.532 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.2221e-05 2024-09-13T11:25:09.532 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:09.532 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:09.532 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:25:09.532 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.093482+0000", 2024-09-13T11:25:09.532 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000109571 2024-09-13T11:25:09.532 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:09.532 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:09.532 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:25:09.533 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.095251+0000", 2024-09-13T11:25:09.533 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001768941 2024-09-13T11:25:09.533 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:09.533 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:25:09.533 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:09.533 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:09.533 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:09.533 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPushReply(3.1a 83/78 [PushReplyOp(3:58c7ffc0:::benchmark_data_smithi177_31116_object8738:head)])", 2024-09-13T11:25:09.534 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:40.805941+0000", 2024-09-13T11:25:09.534 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 88.710836732000004, 2024-09-13T11:25:09.534 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.30315678499999998, 2024-09-13T11:25:09.534 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:25:09.535 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:25:09.535 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-13T11:25:09.535 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:25:09.535 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:09.535 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:25:09.535 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:40.805941+0000", 2024-09-13T11:25:09.535 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:09.536 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:09.536 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:09.536 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:25:09.536 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:40.805941+0000", 2024-09-13T11:25:09.536 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:09.536 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:09.536 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:09.536 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:25:09.536 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:40.805942+0000", 2024-09-13T11:25:09.536 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3769999999999999e-06 2024-09-13T11:25:09.536 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:09.536 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:09.536 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:25:09.537 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:40.805947+0000", 2024-09-13T11:25:09.537 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.8980000000000004e-06 2024-09-13T11:25:09.537 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:09.537 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:09.537 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:25:09.537 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.106505+0000", 2024-09-13T11:25:09.537 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.30055777700000003 2024-09-13T11:25:09.537 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:09.537 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:09.537 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:25:09.537 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.106512+0000", 2024-09-13T11:25:09.537 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.3749999999999997e-06 2024-09-13T11:25:09.538 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:09.538 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:09.538 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:25:09.538 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.106566+0000", 2024-09-13T11:25:09.538 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.3103999999999998e-05 2024-09-13T11:25:09.538 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:09.538 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:09.538 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:25:09.538 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.109098+0000", 2024-09-13T11:25:09.538 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0025322539999999998 2024-09-13T11:25:09.538 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:09.539 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:25:09.539 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:09.539 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:09.539 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:09.539 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPushReply(3.1a 83/78 [PushReplyOp(3:58c7ffc0:::benchmark_data_smithi177_31116_object8738:head)])", 2024-09-13T11:25:09.539 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:40.805973+0000", 2024-09-13T11:25:09.539 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 88.710804729000003, 2024-09-13T11:25:09.539 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.28729672099999998, 2024-09-13T11:25:09.539 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:25:09.539 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:25:09.539 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-13T11:25:09.539 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:25:09.539 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:09.540 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:25:09.540 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:40.805973+0000", 2024-09-13T11:25:09.540 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:09.540 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:09.540 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:09.540 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:25:09.540 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:40.805973+0000", 2024-09-13T11:25:09.540 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:09.540 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:09.540 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:09.540 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:25:09.540 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:40.805974+0000", 2024-09-13T11:25:09.540 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.8599999999999998e-07 2024-09-13T11:25:09.541 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:09.541 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:09.541 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:25:09.541 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:40.805976+0000", 2024-09-13T11:25:09.541 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.2809999999999998e-06 2024-09-13T11:25:09.541 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:09.541 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:09.541 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:25:09.541 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.091588+0000", 2024-09-13T11:25:09.541 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.28561173899999998 2024-09-13T11:25:09.541 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:09.541 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:09.542 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:25:09.542 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.091596+0000", 2024-09-13T11:25:09.542 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.0520000000000002e-06 2024-09-13T11:25:09.542 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:09.542 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:09.542 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:25:09.542 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.091907+0000", 2024-09-13T11:25:09.542 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00031179599999999998 2024-09-13T11:25:09.542 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:09.542 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:09.542 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:25:09.542 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.093270+0000", 2024-09-13T11:25:09.542 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0013622669999999999 2024-09-13T11:25:09.542 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:09.543 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:25:09.543 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:09.543 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:09.543 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:09.543 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPushReply(3.1a 84/78 [PushReplyOp(3:581ea97e:::benchmark_data_smithi177_31116_object10255:head)])", 2024-09-13T11:25:09.543 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:41.514228+0000", 2024-09-13T11:25:09.543 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 88.002549240999997, 2024-09-13T11:25:09.543 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.40695564200000001, 2024-09-13T11:25:09.543 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:25:09.543 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:25:09.543 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-13T11:25:09.543 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:25:09.543 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:09.544 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:25:09.544 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.514228+0000", 2024-09-13T11:25:09.544 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:09.544 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:09.544 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:09.544 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:25:09.544 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.514228+0000", 2024-09-13T11:25:09.544 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:09.544 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:09.544 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:09.544 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:25:09.544 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.514230+0000", 2024-09-13T11:25:09.544 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.499e-06 2024-09-13T11:25:09.545 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:09.545 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:09.545 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:25:09.545 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.514236+0000", 2024-09-13T11:25:09.545 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.5389999999999998e-06 2024-09-13T11:25:09.545 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:09.545 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:09.545 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:25:09.545 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.919787+0000", 2024-09-13T11:25:09.545 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.40555152999999999 2024-09-13T11:25:09.545 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:09.545 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:09.545 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:25:09.546 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.919796+0000", 2024-09-13T11:25:09.546 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.4829999999999999e-06 2024-09-13T11:25:09.546 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:09.546 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:09.546 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:25:09.546 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.920366+0000", 2024-09-13T11:25:09.546 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00057061700000000004 2024-09-13T11:25:09.546 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:09.546 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:09.546 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:25:09.546 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.921184+0000", 2024-09-13T11:25:09.546 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00081797399999999996 2024-09-13T11:25:09.547 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:09.547 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:25:09.547 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:09.547 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:09.547 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:09.547 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:60312 3.14 3:2e846241:::benchmark_data_smithi177_31116_object60311:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e85)", 2024-09-13T11:25:09.547 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:42.065725+0000", 2024-09-13T11:25:09.547 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 87.451052598000004, 2024-09-13T11:25:09.547 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.211323176, 2024-09-13T11:25:09.547 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:25:09.547 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:25:09.547 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:09.547 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:25:09.547 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:25:09.548 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:09.548 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 60312 2024-09-13T11:25:09.548 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:09.548 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:25:09.548 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:09.548 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:25:09.548 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.065725+0000", 2024-09-13T11:25:09.548 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:09.548 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:09.548 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:09.548 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:25:09.548 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.065725+0000", 2024-09-13T11:25:09.548 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:09.549 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:09.549 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:09.549 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:25:09.549 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.065728+0000", 2024-09-13T11:25:09.549 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.3400000000000002e-06 2024-09-13T11:25:09.549 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:09.549 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:09.549 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:25:09.549 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.065759+0000", 2024-09-13T11:25:09.549 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.0074e-05 2024-09-13T11:25:09.549 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:09.549 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:09.549 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:25:09.550 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.065771+0000", 2024-09-13T11:25:09.550 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.2869e-05 2024-09-13T11:25:09.550 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:09.550 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:09.550 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:25:09.550 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.065780+0000", 2024-09-13T11:25:09.551 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.3489999999999992e-06 2024-09-13T11:25:09.551 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:09.551 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:09.551 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:25:09.551 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.065837+0000", 2024-09-13T11:25:09.551 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.7444000000000001e-05 2024-09-13T11:25:09.551 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:09.551 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:09.552 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:25:09.552 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.065966+0000", 2024-09-13T11:25:09.552 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000128973 2024-09-13T11:25:09.552 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:09.552 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:09.552 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 14", 2024-09-13T11:25:09.552 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.066114+0000", 2024-09-13T11:25:09.552 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00014788299999999999 2024-09-13T11:25:09.552 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:09.552 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:09.552 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:25:09.552 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.067604+0000", 2024-09-13T11:25:09.553 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001489637 2024-09-13T11:25:09.553 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:09.553 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:09.553 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:25:09.553 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.276963+0000", 2024-09-13T11:25:09.553 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.20935888499999999 2024-09-13T11:25:09.553 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:09.553 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:09.553 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:25:09.554 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.277003+0000", 2024-09-13T11:25:09.554 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.0241000000000003e-05 2024-09-13T11:25:09.554 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:09.554 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:09.554 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:25:09.554 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.277048+0000", 2024-09-13T11:25:09.554 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.5481e-05 2024-09-13T11:25:09.554 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:09.554 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:25:09.554 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:09.554 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:09.554 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:09.555 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop_reply(client.4665.0:60312 3.14 e85/84 ondisk, result = 0)", 2024-09-13T11:25:09.555 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:42.069120+0000", 2024-09-13T11:25:09.555 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 87.447657477999996, 2024-09-13T11:25:09.555 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.20804031000000001, 2024-09-13T11:25:09.555 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:25:09.555 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:25:09.555 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:25:09.555 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:25:09.555 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:09.555 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:25:09.555 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.069120+0000", 2024-09-13T11:25:09.556 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:09.556 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:09.556 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:09.556 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:25:09.556 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.069120+0000", 2024-09-13T11:25:09.556 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:09.556 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:09.556 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:09.556 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:25:09.556 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.069123+0000", 2024-09-13T11:25:09.557 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.289e-06 2024-09-13T11:25:09.557 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:09.557 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:09.557 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:25:09.557 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.069128+0000", 2024-09-13T11:25:09.557 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.3530000000000002e-06 2024-09-13T11:25:09.557 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:09.557 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:09.557 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:25:09.557 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.276885+0000", 2024-09-13T11:25:09.557 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.207756989 2024-09-13T11:25:09.557 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:09.558 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:09.558 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:25:09.558 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.276892+0000", 2024-09-13T11:25:09.558 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.922e-06 2024-09-13T11:25:09.558 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:09.558 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:09.558 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:25:09.558 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.276953+0000", 2024-09-13T11:25:09.558 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.0890000000000001e-05 2024-09-13T11:25:09.558 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:09.558 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:09.558 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:25:09.559 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.276959+0000", 2024-09-13T11:25:09.559 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.9179999999999999e-06 2024-09-13T11:25:09.559 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:09.559 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:09.559 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:25:09.559 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.277161+0000", 2024-09-13T11:25:09.559 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00020194899999999999 2024-09-13T11:25:09.559 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:09.559 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:25:09.559 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:09.559 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:09.560 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:09.560 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:60427 3.14 3:2afea229:::benchmark_data_smithi177_31116_object60426:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e85)", 2024-09-13T11:25:09.560 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:42.187231+0000", 2024-09-13T11:25:09.560 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 87.329546371999996, 2024-09-13T11:25:09.560 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.83900596999999999, 2024-09-13T11:25:09.560 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:25:09.560 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:25:09.560 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:09.560 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:25:09.560 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:25:09.560 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:09.560 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 60427 2024-09-13T11:25:09.561 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:09.561 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:25:09.561 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:09.561 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:25:09.561 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.187231+0000", 2024-09-13T11:25:09.561 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:09.561 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:09.561 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:09.561 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:25:09.561 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.187231+0000", 2024-09-13T11:25:09.561 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:09.561 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:09.562 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:09.562 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:25:09.562 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.187235+0000", 2024-09-13T11:25:09.562 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.3579999999999999e-06 2024-09-13T11:25:09.562 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:09.562 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:09.562 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:25:09.562 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.187261+0000", 2024-09-13T11:25:09.562 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.6089e-05 2024-09-13T11:25:09.562 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:09.562 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:09.563 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:25:09.563 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.187275+0000", 2024-09-13T11:25:09.563 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.4653e-05 2024-09-13T11:25:09.563 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:09.563 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:09.563 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:25:09.563 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.187282+0000", 2024-09-13T11:25:09.563 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.8870000000000003e-06 2024-09-13T11:25:09.563 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:09.563 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:09.563 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:25:09.563 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.187336+0000", 2024-09-13T11:25:09.564 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.3875999999999997e-05 2024-09-13T11:25:09.564 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:09.564 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:09.564 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:25:09.564 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.187480+0000", 2024-09-13T11:25:09.564 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00014330899999999999 2024-09-13T11:25:09.564 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:09.565 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:09.565 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 14", 2024-09-13T11:25:09.565 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.187658+0000", 2024-09-13T11:25:09.565 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00017861900000000001 2024-09-13T11:25:09.565 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:09.565 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:09.565 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:25:09.565 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.189353+0000", 2024-09-13T11:25:09.565 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001695144 2024-09-13T11:25:09.565 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:09.565 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:09.566 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:25:09.566 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:43.026081+0000", 2024-09-13T11:25:09.566 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.83672736199999997 2024-09-13T11:25:09.566 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:09.566 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:09.566 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:25:09.566 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:43.026169+0000", 2024-09-13T11:25:09.566 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.8411000000000004e-05 2024-09-13T11:25:09.566 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:09.566 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:09.566 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:25:09.566 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:43.026237+0000", 2024-09-13T11:25:09.567 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.8262000000000005e-05 2024-09-13T11:25:09.567 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:09.567 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:25:09.567 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:09.567 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:09.567 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:09.567 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop_reply(client.4665.0:60427 3.14 e85/84 ondisk, result = 0)", 2024-09-13T11:25:09.567 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:42.190914+0000", 2024-09-13T11:25:09.567 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 87.325863708, 2024-09-13T11:25:09.567 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.83534071499999996, 2024-09-13T11:25:09.567 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:25:09.567 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:25:09.568 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:25:09.568 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:25:09.568 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:09.568 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:25:09.568 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.190914+0000", 2024-09-13T11:25:09.568 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:09.568 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:09.568 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:09.568 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:25:09.568 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.190914+0000", 2024-09-13T11:25:09.568 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:09.569 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:09.569 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:09.569 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:25:09.569 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.190916+0000", 2024-09-13T11:25:09.569 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.84e-06 2024-09-13T11:25:09.569 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:09.569 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:09.569 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:25:09.569 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.190921+0000", 2024-09-13T11:25:09.569 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.2039999999999996e-06 2024-09-13T11:25:09.569 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:09.569 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:09.570 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:25:09.570 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:43.025926+0000", 2024-09-13T11:25:09.570 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.83500542 2024-09-13T11:25:09.570 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:09.570 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:09.570 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:25:09.570 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:43.025937+0000", 2024-09-13T11:25:09.570 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0597e-05 2024-09-13T11:25:09.570 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:09.570 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:09.570 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:25:09.570 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:43.026052+0000", 2024-09-13T11:25:09.571 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000114838 2024-09-13T11:25:09.571 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:09.571 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:09.571 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:25:09.571 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:43.026070+0000", 2024-09-13T11:25:09.571 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.8131e-05 2024-09-13T11:25:09.571 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:09.571 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:09.571 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:25:09.571 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:43.026255+0000", 2024-09-13T11:25:09.571 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000184685 2024-09-13T11:25:09.571 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:09.572 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:25:09.572 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:09.572 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:09.572 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:09.572 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPushReply(3.1d 89/78 [PushReplyOp(3:bc1b7cfb:::benchmark_data_smithi177_31116_object22629:head)])", 2024-09-13T11:25:09.572 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:46.663106+0000", 2024-09-13T11:25:09.572 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 82.853671524999996, 2024-09-13T11:25:09.572 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.26645533199999999, 2024-09-13T11:25:09.572 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:25:09.572 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:25:09.572 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-13T11:25:09.573 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:25:09.573 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:09.573 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:25:09.573 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.663106+0000", 2024-09-13T11:25:09.573 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:09.573 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:09.573 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:09.573 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:25:09.573 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.663106+0000", 2024-09-13T11:25:09.573 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:09.573 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:09.573 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:09.574 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:25:09.574 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.663108+0000", 2024-09-13T11:25:09.574 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.0140000000000001e-06 2024-09-13T11:25:09.574 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:09.574 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:09.574 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:25:09.574 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.663116+0000", 2024-09-13T11:25:09.574 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.5440000000000001e-06 2024-09-13T11:25:09.574 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:09.574 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:09.574 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:25:09.575 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.928111+0000", 2024-09-13T11:25:09.575 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.26499566899999999 2024-09-13T11:25:09.575 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:09.575 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:09.575 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:25:09.575 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.928135+0000", 2024-09-13T11:25:09.575 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.4003e-05 2024-09-13T11:25:09.575 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:09.575 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:09.575 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:25:09.575 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.928559+0000", 2024-09-13T11:25:09.575 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000423318 2024-09-13T11:25:09.576 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:09.576 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:09.576 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:25:09.576 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.929562+0000", 2024-09-13T11:25:09.576 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0010027840000000001 2024-09-13T11:25:09.576 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:09.576 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:25:09.576 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:09.576 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:09.576 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:09.577 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "rep_scrubmap(3.14 e84 from shard 14)", 2024-09-13T11:25:09.577 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:48.313615+0000", 2024-09-13T11:25:09.577 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 81.203162731999996, 2024-09-13T11:25:09.577 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.35633836099999999, 2024-09-13T11:25:09.577 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:25:09.577 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:25:09.577 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:25:09.577 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:25:09.577 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:09.577 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:25:09.577 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.313615+0000", 2024-09-13T11:25:09.577 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:09.578 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:09.578 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:09.578 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:25:09.578 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.313615+0000", 2024-09-13T11:25:09.578 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:09.578 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:09.578 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:09.578 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:25:09.578 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.313617+0000", 2024-09-13T11:25:09.578 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.7990000000000001e-06 2024-09-13T11:25:09.578 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:09.578 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:09.579 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:25:09.579 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.313688+0000", 2024-09-13T11:25:09.579 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.1481000000000002e-05 2024-09-13T11:25:09.579 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:09.579 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:09.579 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:25:09.579 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.669785+0000", 2024-09-13T11:25:09.579 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.356096466 2024-09-13T11:25:09.579 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:09.579 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:09.579 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:25:09.579 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.669792+0000", 2024-09-13T11:25:09.580 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.1740000000000003e-06 2024-09-13T11:25:09.580 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:09.580 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:09.580 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:25:09.580 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.669839+0000", 2024-09-13T11:25:09.580 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.7323000000000001e-05 2024-09-13T11:25:09.580 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:09.580 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:09.580 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:25:09.580 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.669847+0000", 2024-09-13T11:25:09.580 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.0469999999999994e-06 2024-09-13T11:25:09.580 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:09.581 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:09.581 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:25:09.581 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.669953+0000", 2024-09-13T11:25:09.581 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00010607099999999999 2024-09-13T11:25:09.581 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:09.581 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:25:09.581 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:09.581 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:09.581 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:09.581 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:110268 3.14 3:2997c4c5:::benchmark_data_smithi177_31116_object110267:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119)", 2024-09-13T11:25:09.581 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:25.359386+0000", 2024-09-13T11:25:09.582 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 44.157391416000003, 2024-09-13T11:25:09.582 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.29072703, 2024-09-13T11:25:09.582 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:25:09.582 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:25:09.582 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:09.582 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:25:09.582 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:25:09.582 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:09.582 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 110268 2024-09-13T11:25:09.582 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:09.582 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:25:09.582 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:09.583 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:25:09.583 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.359386+0000", 2024-09-13T11:25:09.583 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:09.583 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:09.583 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:09.583 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:25:09.583 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.359386+0000", 2024-09-13T11:25:09.583 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:09.583 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:09.583 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:09.583 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:25:09.584 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.359389+0000", 2024-09-13T11:25:09.584 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.8600000000000001e-06 2024-09-13T11:25:09.584 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:09.584 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:09.584 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:25:09.584 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.359424+0000", 2024-09-13T11:25:09.584 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.4755999999999997e-05 2024-09-13T11:25:09.584 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:09.584 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:09.584 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:25:09.584 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.359445+0000", 2024-09-13T11:25:09.584 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.1104999999999999e-05 2024-09-13T11:25:09.585 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:09.585 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:09.585 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:25:09.585 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.359460+0000", 2024-09-13T11:25:09.585 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5396999999999998e-05 2024-09-13T11:25:09.585 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:09.585 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:09.585 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:25:09.585 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.359581+0000", 2024-09-13T11:25:09.585 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000120455 2024-09-13T11:25:09.585 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:09.586 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:09.586 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:25:09.586 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.359827+0000", 2024-09-13T11:25:09.586 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00024571 2024-09-13T11:25:09.586 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:09.586 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:09.586 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 14", 2024-09-13T11:25:09.586 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.360084+0000", 2024-09-13T11:25:09.586 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00025787999999999999 2024-09-13T11:25:09.586 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:09.586 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:09.587 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:25:09.587 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.362243+0000", 2024-09-13T11:25:09.587 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0021581579999999999 2024-09-13T11:25:09.587 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:09.587 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:09.587 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:25:09.587 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.650034+0000", 2024-09-13T11:25:09.587 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.287791662 2024-09-13T11:25:09.587 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:09.587 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:09.587 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:25:09.588 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.650077+0000", 2024-09-13T11:25:09.588 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.2367999999999998e-05 2024-09-13T11:25:09.588 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:09.588 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:09.588 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:25:09.588 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.650113+0000", 2024-09-13T11:25:09.588 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.6678999999999999e-05 2024-09-13T11:25:09.588 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:09.588 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:25:09.588 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:09.588 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:09.589 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:09.589 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:110279 3.14 3:2b17a349:::benchmark_data_smithi177_31116_object110278:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119)", 2024-09-13T11:25:09.589 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:25.645137+0000", 2024-09-13T11:25:09.589 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 43.871640577999997, 2024-09-13T11:25:09.589 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.28172927800000003, 2024-09-13T11:25:09.589 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:25:09.589 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:25:09.589 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:09.589 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:25:09.589 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:25:09.589 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:09.590 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 110279 2024-09-13T11:25:09.590 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:09.590 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:25:09.590 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:09.590 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:25:09.590 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.645137+0000", 2024-09-13T11:25:09.590 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:09.590 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:09.590 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:09.590 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:25:09.590 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.645137+0000", 2024-09-13T11:25:09.590 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:09.591 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:09.591 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:09.591 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:25:09.591 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.645140+0000", 2024-09-13T11:25:09.591 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.0939999999999999e-06 2024-09-13T11:25:09.591 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:09.591 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:09.591 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:25:09.591 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.645184+0000", 2024-09-13T11:25:09.591 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.3412999999999999e-05 2024-09-13T11:25:09.591 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:09.591 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:09.592 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:25:09.592 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.645206+0000", 2024-09-13T11:25:09.592 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.1997e-05 2024-09-13T11:25:09.592 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:09.592 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:09.592 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:25:09.592 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.645216+0000", 2024-09-13T11:25:09.592 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0740999999999999e-05 2024-09-13T11:25:09.592 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:09.592 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:09.592 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:25:09.593 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.645397+0000", 2024-09-13T11:25:09.593 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00018077600000000001 2024-09-13T11:25:09.593 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:09.593 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:09.593 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:25:09.593 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.645638+0000", 2024-09-13T11:25:09.593 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000241091 2024-09-13T11:25:09.593 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:09.593 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:09.593 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 14", 2024-09-13T11:25:09.593 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.645922+0000", 2024-09-13T11:25:09.593 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000283565 2024-09-13T11:25:09.594 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:09.594 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:09.594 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:25:09.594 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.648980+0000", 2024-09-13T11:25:09.594 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00305851 2024-09-13T11:25:09.594 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:09.594 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:09.594 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:25:09.594 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.926702+0000", 2024-09-13T11:25:09.594 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.27772199800000003 2024-09-13T11:25:09.594 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:09.595 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:09.595 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:25:09.595 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.926796+0000", 2024-09-13T11:25:09.595 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.4158000000000002e-05 2024-09-13T11:25:09.595 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:09.595 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:09.595 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:25:09.595 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.926866+0000", 2024-09-13T11:25:09.595 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.9935e-05 2024-09-13T11:25:09.595 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:09.595 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:25:09.595 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:09.596 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:09.596 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:09.596 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:129865 3.14 3:2c872f8e:::benchmark_data_smithi177_31116_object129864:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e134)", 2024-09-13T11:25:09.596 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:47.107186+0000", 2024-09-13T11:25:09.596 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 22.409592021000002, 2024-09-13T11:25:09.596 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.21814059299999999, 2024-09-13T11:25:09.596 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:25:09.596 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:25:09.596 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:09.596 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:25:09.596 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:25:09.597 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:09.597 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 129865 2024-09-13T11:25:09.597 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:09.597 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:25:09.597 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:09.597 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:25:09.597 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:47.107186+0000", 2024-09-13T11:25:09.597 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:09.597 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:09.597 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:09.597 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:25:09.597 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:47.107186+0000", 2024-09-13T11:25:09.598 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:09.598 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:09.598 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:09.598 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:25:09.598 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:47.107187+0000", 2024-09-13T11:25:09.598 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5740000000000001e-06 2024-09-13T11:25:09.598 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:09.598 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:09.598 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:25:09.598 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:47.107207+0000", 2024-09-13T11:25:09.598 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.9304999999999999e-05 2024-09-13T11:25:09.599 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:09.599 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:09.599 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:25:09.599 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:47.107215+0000", 2024-09-13T11:25:09.599 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.7390000000000005e-06 2024-09-13T11:25:09.599 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:09.599 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:09.599 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:25:09.599 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:47.107221+0000", 2024-09-13T11:25:09.599 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.7350000000000001e-06 2024-09-13T11:25:09.599 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:09.599 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:09.600 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:25:09.600 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:47.107278+0000", 2024-09-13T11:25:09.600 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.7209000000000001e-05 2024-09-13T11:25:09.600 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:09.600 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:09.600 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:25:09.600 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:47.107383+0000", 2024-09-13T11:25:09.600 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000104718 2024-09-13T11:25:09.600 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:09.600 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:09.600 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 15", 2024-09-13T11:25:09.600 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:47.107480+0000", 2024-09-13T11:25:09.601 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.7343000000000006e-05 2024-09-13T11:25:09.601 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:09.601 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:09.601 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:25:09.601 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:47.109477+0000", 2024-09-13T11:25:09.601 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0019968490000000002 2024-09-13T11:25:09.601 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:09.601 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:09.601 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:25:09.601 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:47.324147+0000", 2024-09-13T11:25:09.601 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.21467018199999999 2024-09-13T11:25:09.601 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:09.602 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:09.602 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:25:09.602 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:47.324281+0000", 2024-09-13T11:25:09.602 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00013328600000000001 2024-09-13T11:25:09.602 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:09.602 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:09.602 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:25:09.602 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:47.325326+0000", 2024-09-13T11:25:09.602 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0010456529999999999 2024-09-13T11:25:09.602 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:09.602 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:25:09.603 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:09.603 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:09.603 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:09.603 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPush(3.15 140/122 [PushOp(3:ae90e82d:::benchmark_data_smithi177_31116_object53962:head, version: 74'3006, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:ae90e82d:::benchmark_data_smithi177_31116_object53962:head@74'3006, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:25:09.603 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:54.310786+0000", 2024-09-13T11:25:09.603 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 15.205991548, 2024-09-13T11:25:09.603 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.229345255, 2024-09-13T11:25:09.603 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:25:09.603 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:25:09.603 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:25:09.603 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:25:09.604 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:09.604 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:25:09.604 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:54.310786+0000", 2024-09-13T11:25:09.604 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:09.604 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:09.604 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:09.604 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:25:09.604 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:54.310786+0000", 2024-09-13T11:25:09.604 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:09.604 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:09.604 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:09.604 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:25:09.605 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:54.310795+0000", 2024-09-13T11:25:09.605 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.1910000000000006e-06 2024-09-13T11:25:09.605 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:09.605 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:09.605 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:25:09.605 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:54.310829+0000", 2024-09-13T11:25:09.605 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.4121000000000001e-05 2024-09-13T11:25:09.605 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:09.605 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:09.605 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:25:09.605 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:54.538419+0000", 2024-09-13T11:25:09.605 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.227589611 2024-09-13T11:25:09.606 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:09.606 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:09.606 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:25:09.606 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:54.538430+0000", 2024-09-13T11:25:09.606 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0456e-05 2024-09-13T11:25:09.606 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:09.606 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:09.606 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:25:09.606 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:54.538497+0000", 2024-09-13T11:25:09.606 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.6948999999999999e-05 2024-09-13T11:25:09.606 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:09.606 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:09.607 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:25:09.607 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:54.538505+0000", 2024-09-13T11:25:09.607 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.2989999999999997e-06 2024-09-13T11:25:09.607 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:09.607 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:09.607 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:25:09.607 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:54.540131+0000", 2024-09-13T11:25:09.607 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001626628 2024-09-13T11:25:09.607 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:09.607 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:25:09.607 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:09.607 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:09.608 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:09.608 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPush(3.15 143/122 [PushOp(3:ae52bb45:::benchmark_data_smithi177_31116_object61893:head, version: 87'3272, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:ae52bb45:::benchmark_data_smithi177_31116_object61893:head@87'3272, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:25:09.608 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:57.633939+0000", 2024-09-13T11:25:09.608 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 11.882838594999999, 2024-09-13T11:25:09.608 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.339405067, 2024-09-13T11:25:09.608 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:25:09.608 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:25:09.608 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:25:09.608 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:25:09.608 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:09.608 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:25:09.609 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:57.633939+0000", 2024-09-13T11:25:09.609 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:09.609 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:09.609 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:09.609 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:25:09.609 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:57.633939+0000", 2024-09-13T11:25:09.609 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:09.609 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:09.609 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:09.609 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:25:09.609 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:57.633941+0000", 2024-09-13T11:25:09.610 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.911e-06 2024-09-13T11:25:09.610 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:09.610 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:09.610 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:25:09.610 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:57.633994+0000", 2024-09-13T11:25:09.610 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.2753999999999997e-05 2024-09-13T11:25:09.610 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:09.610 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:09.610 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:25:09.610 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:57.968403+0000", 2024-09-13T11:25:09.610 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.334409336 2024-09-13T11:25:09.610 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:09.611 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:09.611 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:25:09.611 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:57.968451+0000", 2024-09-13T11:25:09.611 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.7525000000000003e-05 2024-09-13T11:25:09.611 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:09.611 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:09.611 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:25:09.611 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:57.968660+0000", 2024-09-13T11:25:09.611 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000209204 2024-09-13T11:25:09.611 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:09.611 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:09.611 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:25:09.612 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:57.968680+0000", 2024-09-13T11:25:09.612 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.9639999999999998e-05 2024-09-13T11:25:09.612 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:09.612 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:09.612 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:25:09.612 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:57.973344+0000", 2024-09-13T11:25:09.612 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0046646969999999998 2024-09-13T11:25:09.612 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:09.612 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:25:09.612 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:09.612 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:09.613 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:09.613 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:140658 3.17 e144/85 3:ece5a1b3:::benchmark_data_smithi177_31116_object355:head v 144'5654, mlcod=144'5652)", 2024-09-13T11:25:09.613 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:58.247792+0000", 2024-09-13T11:25:09.613 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 11.268986117000001, 2024-09-13T11:25:09.613 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.29148369699999999, 2024-09-13T11:25:09.613 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:25:09.613 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:25:09.613 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:09.613 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:25:09.613 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:09.614 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:25:09.614 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:58.247792+0000", 2024-09-13T11:25:09.614 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:09.614 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:09.614 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:09.614 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:25:09.614 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:58.247792+0000", 2024-09-13T11:25:09.614 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:09.614 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:09.614 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:09.614 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:25:09.615 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:58.247793+0000", 2024-09-13T11:25:09.615 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.347e-06 2024-09-13T11:25:09.615 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:09.615 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:09.615 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:25:09.615 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:58.247799+0000", 2024-09-13T11:25:09.615 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.5239999999999997e-06 2024-09-13T11:25:09.615 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:09.615 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:09.615 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:25:09.615 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:58.535226+0000", 2024-09-13T11:25:09.616 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.28742604900000002 2024-09-13T11:25:09.616 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:09.616 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:09.616 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:25:09.616 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:58.535238+0000", 2024-09-13T11:25:09.616 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.2843000000000001e-05 2024-09-13T11:25:09.616 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:09.616 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:09.616 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:25:09.616 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:58.535393+0000", 2024-09-13T11:25:09.616 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00015481699999999999 2024-09-13T11:25:09.617 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:09.617 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:09.617 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:25:09.617 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:58.535434+0000", 2024-09-13T11:25:09.617 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.1152999999999997e-05 2024-09-13T11:25:09.617 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:09.617 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:09.617 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:25:09.617 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:58.539191+0000", 2024-09-13T11:25:09.617 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0037571520000000001 2024-09-13T11:25:09.618 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:09.618 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:09.618 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:25:09.618 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:58.539275+0000", 2024-09-13T11:25:09.618 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.3812000000000003e-05 2024-09-13T11:25:09.618 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:09.618 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:25:09.618 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:09.618 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:09.618 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:09.618 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPush(3.15 144/122 [PushOp(3:a97ba025:::benchmark_data_smithi177_31116_object64775:head, version: 90'3365, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:a97ba025:::benchmark_data_smithi177_31116_object64775:head@90'3365, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:25:09.618 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:58.825014+0000", 2024-09-13T11:25:09.619 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 10.691763489, 2024-09-13T11:25:09.619 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.23868276499999999, 2024-09-13T11:25:09.619 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:25:09.619 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:25:09.619 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:25:09.619 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:25:09.619 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:09.619 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:25:09.619 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:58.825014+0000", 2024-09-13T11:25:09.619 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:09.619 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:09.619 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:09.620 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:25:09.620 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:58.825014+0000", 2024-09-13T11:25:09.620 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:09.620 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:09.620 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:09.620 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:25:09.620 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:58.825017+0000", 2024-09-13T11:25:09.620 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.7630000000000001e-06 2024-09-13T11:25:09.620 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:09.620 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:09.620 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:25:09.621 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:58.825099+0000", 2024-09-13T11:25:09.621 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.2035999999999999e-05 2024-09-13T11:25:09.621 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:09.621 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:09.621 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:25:09.621 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:59.059557+0000", 2024-09-13T11:25:09.621 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.23445790399999999 2024-09-13T11:25:09.621 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:09.621 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:09.621 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:25:09.622 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:59.059609+0000", 2024-09-13T11:25:09.622 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.2550999999999999e-05 2024-09-13T11:25:09.622 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:09.622 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:09.622 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:25:09.622 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:59.060117+0000", 2024-09-13T11:25:09.622 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00050754299999999995 2024-09-13T11:25:09.622 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:09.622 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:09.622 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:25:09.622 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:59.060139+0000", 2024-09-13T11:25:09.622 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.2306e-05 2024-09-13T11:25:09.623 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:09.623 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:09.623 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:25:09.623 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:59.063697+0000", 2024-09-13T11:25:09.623 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0035576620000000001 2024-09-13T11:25:09.623 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:09.623 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:25:09.623 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:09.623 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:09.623 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:09.623 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:147472 3.2d 3:b6aeefb0:::benchmark_data_smithi177_31116_object7169:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e148)", 2024-09-13T11:25:09.624 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:25:04.123331+0000", 2024-09-13T11:25:09.624 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 5.3934472280000003, 2024-09-13T11:25:09.624 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.23047060799999999, 2024-09-13T11:25:09.624 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:25:09.624 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:25:09.624 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:09.624 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:25:09.624 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:25:09.624 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:09.624 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 147472 2024-09-13T11:25:09.624 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:09.624 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:25:09.625 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:09.625 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:25:09.625 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:04.123331+0000", 2024-09-13T11:25:09.625 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:09.625 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:09.625 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:09.625 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:25:09.625 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:04.123331+0000", 2024-09-13T11:25:09.625 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:09.625 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:09.626 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:09.626 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:25:09.626 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:04.123332+0000", 2024-09-13T11:25:09.626 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.556e-06 2024-09-13T11:25:09.626 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:09.626 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:09.626 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:25:09.626 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:04.123337+0000", 2024-09-13T11:25:09.626 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.5889999999999996e-06 2024-09-13T11:25:09.626 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:09.627 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:09.627 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:25:09.627 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:04.123344+0000", 2024-09-13T11:25:09.627 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.7889999999999995e-06 2024-09-13T11:25:09.627 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:09.627 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:09.627 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:25:09.627 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:04.123350+0000", 2024-09-13T11:25:09.627 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.3820000000000003e-06 2024-09-13T11:25:09.627 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:09.627 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:09.628 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:25:09.628 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:04.123402+0000", 2024-09-13T11:25:09.628 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.2281999999999999e-05 2024-09-13T11:25:09.628 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:09.628 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:09.628 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for degraded object", 2024-09-13T11:25:09.628 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:04.124030+0000", 2024-09-13T11:25:09.628 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00062812999999999996 2024-09-13T11:25:09.628 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:09.628 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:09.628 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:25:09.629 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:04.137378+0000", 2024-09-13T11:25:09.629 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.013347816 2024-09-13T11:25:09.629 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:09.629 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:09.629 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:25:09.629 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:04.137472+0000", 2024-09-13T11:25:09.629 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.3777999999999996e-05 2024-09-13T11:25:09.629 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:09.629 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:09.629 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 0,5", 2024-09-13T11:25:09.629 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:04.137594+0000", 2024-09-13T11:25:09.629 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00012196099999999999 2024-09-13T11:25:09.630 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:09.630 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:09.630 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:25:09.630 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:04.139184+0000", 2024-09-13T11:25:09.630 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001590508 2024-09-13T11:25:09.630 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:09.630 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:09.630 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:25:09.630 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:04.143627+0000", 2024-09-13T11:25:09.630 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0044427989999999999 2024-09-13T11:25:09.630 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:09.631 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:09.631 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:25:09.631 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:04.352741+0000", 2024-09-13T11:25:09.631 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.209114142 2024-09-13T11:25:09.631 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:09.631 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:09.631 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:25:09.631 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:04.352808+0000", 2024-09-13T11:25:09.631 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.7034999999999998e-05 2024-09-13T11:25:09.631 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:09.631 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:09.631 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:25:09.632 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:04.353801+0000", 2024-09-13T11:25:09.632 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00099284099999999995 2024-09-13T11:25:09.632 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:09.632 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:25:09.632 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:09.632 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:09.632 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:25:09.632 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-09-13T11:25:09.633 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 15] out_osds: [14] dead_osds: [] live_osds: [0, 4, 8, 12, 5, 9, 13, 2, 6, 10, 3, 7, 11, 15, 1, 14] 2024-09-13T11:25:09.633 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 0 1 2024-09-13T11:25:09.634 INFO:teuthology.orchestra.run.smithi001.stdout:osd.11: {} 2024-09-13T11:25:09.635 INFO:tasks.ceph.osd.5.smithi092.stderr:2024-09-13T11:25:09.629+0000 7f6f0b705640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26346) UID: 0 2024-09-13T11:25:09.635 INFO:teuthology.orchestra.run.smithi001.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-13T11:25:09.636 INFO:teuthology.orchestra.run.smithi001.stdout:osd.12: {} 2024-09-13T11:25:09.636 INFO:teuthology.orchestra.run.smithi001.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-13T11:25:09.636 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-13T11:25:09.637 INFO:teuthology.orchestra.run.smithi001.stdout:osd.13: {} 2024-09-13T11:25:09.637 INFO:teuthology.orchestra.run.smithi001.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-13T11:25:09.638 INFO:teuthology.orchestra.run.smithi001.stdout:osd.14: {} 2024-09-13T11:25:09.638 INFO:teuthology.orchestra.run.smithi001.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-13T11:25:09.658 INFO:teuthology.orchestra.run.smithi001.stdout:osd.15: {} 2024-09-13T11:25:09.659 INFO:teuthology.orchestra.run.smithi001.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-13T11:25:09.734 INFO:tasks.ceph.osd.14.smithi162.stderr:2024-09-13T11:25:09.732+0000 7f3df5903640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33154) UID: 0 2024-09-13T11:25:09.830 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-09-13T11:25:09.830 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [ 2024-09-13T11:25:09.830 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:09.830 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:154930 3.12 3:4ec1e631:::benchmark_data_smithi177_31116_object14627:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e151)", 2024-09-13T11:25:09.830 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:25:09.827875+0000", 2024-09-13T11:25:09.830 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 0.0022455560000000001, 2024-09-13T11:25:09.831 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0022734790000000001, 2024-09-13T11:25:09.831 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:25:09.831 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:25:09.831 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "waiting for sub ops", 2024-09-13T11:25:09.831 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:25:09.831 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:25:09.831 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:09.831 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 154930 2024-09-13T11:25:09.831 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:09.831 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:25:09.831 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:09.831 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:25:09.831 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:09.827875+0000", 2024-09-13T11:25:09.832 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:09.832 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:09.832 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:09.832 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:25:09.832 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:09.827875+0000", 2024-09-13T11:25:09.832 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:09.832 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:09.832 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:09.832 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:25:09.832 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:09.827876+0000", 2024-09-13T11:25:09.832 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.6589999999999999e-06 2024-09-13T11:25:09.832 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:09.832 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:09.833 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:25:09.833 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:09.827881+0000", 2024-09-13T11:25:09.833 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.95e-06 2024-09-13T11:25:09.833 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:09.833 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:09.833 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:25:09.833 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:09.827892+0000", 2024-09-13T11:25:09.833 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0927000000000001e-05 2024-09-13T11:25:09.833 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:09.833 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:09.833 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:25:09.833 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:09.827898+0000", 2024-09-13T11:25:09.834 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.4700000000000001e-06 2024-09-13T11:25:09.834 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:09.834 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:09.834 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:25:09.834 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:09.827942+0000", 2024-09-13T11:25:09.834 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.4079000000000003e-05 2024-09-13T11:25:09.834 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:09.834 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:09.834 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:25:09.835 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:09.828041+0000", 2024-09-13T11:25:09.835 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.8728999999999996e-05 2024-09-13T11:25:09.835 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:09.835 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:09.835 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 5", 2024-09-13T11:25:09.835 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:09.828126+0000", 2024-09-13T11:25:09.835 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.5813e-05 2024-09-13T11:25:09.835 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:09.835 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:25:09.835 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:09.835 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:09.835 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:09.835 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:154931 3.13 e151/83)", 2024-09-13T11:25:09.835 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:25:09.830102+0000", 2024-09-13T11:25:09.836 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 1.8541000000000001e-05, 2024-09-13T11:25:09.836 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00011228800000000001, 2024-09-13T11:25:09.836 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:25:09.836 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:25:09.836 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-13T11:25:09.836 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:25:09.836 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:09.836 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:25:09.836 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:09.830102+0000", 2024-09-13T11:25:09.836 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:09.836 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:09.836 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:09.836 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:25:09.837 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:09.830102+0000", 2024-09-13T11:25:09.837 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:09.837 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:09.837 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:09.837 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:25:09.837 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:09.830103+0000", 2024-09-13T11:25:09.837 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5099999999999999e-06 2024-09-13T11:25:09.837 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:09.837 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:09.837 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:25:09.837 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:09.830110+0000", 2024-09-13T11:25:09.837 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.7789999999999998e-06 2024-09-13T11:25:09.837 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:09.838 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:09.838 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:25:09.838 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:09.830152+0000", 2024-09-13T11:25:09.838 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.2323000000000001e-05 2024-09-13T11:25:09.838 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:09.838 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:09.838 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:25:09.838 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:09.830162+0000", 2024-09-13T11:25:09.838 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.3770000000000002e-06 2024-09-13T11:25:09.839 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:09.839 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:09.839 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:25:09.839 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:09.830212+0000", 2024-09-13T11:25:09.839 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.011e-05 2024-09-13T11:25:09.839 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:09.839 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:25:09.839 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:09.839 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:09.839 INFO:teuthology.orchestra.run.smithi177.stdout: ], 2024-09-13T11:25:09.839 INFO:teuthology.orchestra.run.smithi177.stdout: "num_ops": 2 2024-09-13T11:25:09.839 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-09-13T11:25:09.840 INFO:tasks.ceph.osd.10.smithi162.stderr:2024-09-13T11:25:09.836+0000 7fb59c9f0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26317) UID: 0 2024-09-13T11:25:09.848 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-13T11:25:09.941 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-13T11:25:09.939+0000 7f6a68775640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26307) UID: 0 2024-09-13T11:25:09.995 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-09-13T11:25:09.995 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [], 2024-09-13T11:25:09.995 INFO:teuthology.orchestra.run.smithi177.stdout: "complaint_time": 30, 2024-09-13T11:25:09.995 INFO:teuthology.orchestra.run.smithi177.stdout: "num_blocked_ops": 0 2024-09-13T11:25:09.995 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-09-13T11:25:10.017 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-13T11:25:10.060 INFO:tasks.ceph.osd.3.smithi177.stderr:2024-09-13T11:25:10.058+0000 7f10adba4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26574) UID: 0 2024-09-13T11:25:10.092 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-13T11:25:10.143 INFO:tasks.ceph.osd.14.smithi162.stderr:2024-09-13T11:25:10.140+0000 7f3df5903640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33154) UID: 0 2024-09-13T11:25:10.167 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-09-13T11:25:10.167 INFO:teuthology.orchestra.run.smithi177.stdout: "size": 20, 2024-09-13T11:25:10.167 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 600, 2024-09-13T11:25:10.167 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [ 2024-09-13T11:25:10.168 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.168 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPushReply(3.13 75/70 [PushReplyOp(3:cfab03d6:::benchmark_data_smithi177_31116_object16599:head)])", 2024-09-13T11:25:10.168 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:31.163480+0000", 2024-09-13T11:25:10.168 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 99.001785096999996, 2024-09-13T11:25:10.168 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.63009029299999997, 2024-09-13T11:25:10.168 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:25:10.168 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:25:10.168 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-13T11:25:10.168 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:25:10.168 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.168 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:25:10.168 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.163480+0000", 2024-09-13T11:25:10.169 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:10.169 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.169 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.169 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:25:10.169 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.163480+0000", 2024-09-13T11:25:10.169 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:10.169 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.169 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.169 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:25:10.169 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.163481+0000", 2024-09-13T11:25:10.169 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0410000000000001e-06 2024-09-13T11:25:10.169 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.169 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.170 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:25:10.170 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.163485+0000", 2024-09-13T11:25:10.170 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.1629999999999998e-06 2024-09-13T11:25:10.170 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.170 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.170 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:25:10.170 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.792166+0000", 2024-09-13T11:25:10.170 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.62868067400000005 2024-09-13T11:25:10.170 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.170 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.170 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:25:10.170 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.792184+0000", 2024-09-13T11:25:10.170 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.7898999999999999e-05 2024-09-13T11:25:10.170 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.171 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.171 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:25:10.171 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.792312+0000", 2024-09-13T11:25:10.171 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00012871000000000001 2024-09-13T11:25:10.171 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.171 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.171 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:25:10.171 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.793570+0000", 2024-09-13T11:25:10.171 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001257806 2024-09-13T11:25:10.171 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:10.171 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:25:10.171 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:10.171 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.172 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.172 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPush(3.12 75/69 [PushOp(3:4a36ca6d:::benchmark_data_smithi177_31116_object31707:head, version: 59'1993, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:4a36ca6d:::benchmark_data_smithi177_31116_object31707:head@59'1993, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:25:10.172 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:31.309623+0000", 2024-09-13T11:25:10.172 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 98.855642141999994, 2024-09-13T11:25:10.172 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.165459152, 2024-09-13T11:25:10.172 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:25:10.172 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:25:10.172 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:25:10.172 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:25:10.172 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.172 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:25:10.172 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.309623+0000", 2024-09-13T11:25:10.172 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:10.172 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.173 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.173 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:25:10.173 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.309623+0000", 2024-09-13T11:25:10.173 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:10.173 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.173 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.173 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:25:10.173 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.309625+0000", 2024-09-13T11:25:10.173 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.3860000000000001e-06 2024-09-13T11:25:10.173 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.174 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.174 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:25:10.174 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.309680+0000", 2024-09-13T11:25:10.174 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.4388000000000002e-05 2024-09-13T11:25:10.174 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.174 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.174 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:25:10.174 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.473157+0000", 2024-09-13T11:25:10.174 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.163476813 2024-09-13T11:25:10.174 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.174 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.174 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:25:10.174 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.473189+0000", 2024-09-13T11:25:10.174 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.2975000000000002e-05 2024-09-13T11:25:10.175 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.175 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.175 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:25:10.175 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.473395+0000", 2024-09-13T11:25:10.175 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000206009 2024-09-13T11:25:10.175 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.175 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.175 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:25:10.175 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.473424+0000", 2024-09-13T11:25:10.175 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.8255e-05 2024-09-13T11:25:10.175 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.175 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.175 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:25:10.175 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.475082+0000", 2024-09-13T11:25:10.176 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0016583259999999999 2024-09-13T11:25:10.176 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:10.176 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:25:10.176 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:10.176 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.176 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.176 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPushReply(3.13 76/70 [PushReplyOp(3:c9d43a82:::benchmark_data_smithi177_31116_object20026:head)])", 2024-09-13T11:25:10.176 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:32.427709+0000", 2024-09-13T11:25:10.176 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 97.737555581999999, 2024-09-13T11:25:10.176 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.46547316300000002, 2024-09-13T11:25:10.176 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:25:10.176 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:25:10.176 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-13T11:25:10.176 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:25:10.177 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.177 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:25:10.177 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:32.427709+0000", 2024-09-13T11:25:10.177 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:10.177 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.177 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.177 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:25:10.177 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:32.427709+0000", 2024-09-13T11:25:10.177 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:10.177 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.177 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.178 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:25:10.178 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:32.427711+0000", 2024-09-13T11:25:10.178 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5099999999999999e-06 2024-09-13T11:25:10.178 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.178 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.178 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:25:10.178 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:32.427717+0000", 2024-09-13T11:25:10.178 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.1569999999999997e-06 2024-09-13T11:25:10.178 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.178 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.178 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:25:10.178 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:32.892390+0000", 2024-09-13T11:25:10.178 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.46467251100000001 2024-09-13T11:25:10.178 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.178 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.179 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:25:10.179 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:32.892405+0000", 2024-09-13T11:25:10.179 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5296999999999999e-05 2024-09-13T11:25:10.179 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.179 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.179 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:25:10.179 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:32.892503+0000", 2024-09-13T11:25:10.179 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.7820999999999997e-05 2024-09-13T11:25:10.179 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.179 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.179 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:25:10.179 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:32.893183+0000", 2024-09-13T11:25:10.179 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00067986700000000001 2024-09-13T11:25:10.179 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:10.179 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:25:10.180 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:10.180 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.180 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.180 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:57435 3.3 e80/75 3:c0db4777:::benchmark_data_smithi177_31116_object57434:head v 80'3155, mlcod=80'3153)", 2024-09-13T11:25:10.180 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:37.437070+0000", 2024-09-13T11:25:10.180 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 92.728194764999998, 2024-09-13T11:25:10.180 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.35861845999999997, 2024-09-13T11:25:10.180 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:25:10.180 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:25:10.180 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:10.180 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:25:10.180 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.180 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:25:10.181 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:37.437070+0000", 2024-09-13T11:25:10.181 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:10.181 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.181 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.181 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:25:10.181 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:37.437070+0000", 2024-09-13T11:25:10.181 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:10.181 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.181 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.181 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:25:10.181 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:37.437072+0000", 2024-09-13T11:25:10.181 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5549999999999999e-06 2024-09-13T11:25:10.181 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.181 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.182 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:25:10.182 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:37.437141+0000", 2024-09-13T11:25:10.182 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.9494999999999998e-05 2024-09-13T11:25:10.182 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.182 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.182 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:25:10.182 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:37.793400+0000", 2024-09-13T11:25:10.182 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.35625817999999998 2024-09-13T11:25:10.182 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.182 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.182 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:25:10.182 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:37.793406+0000", 2024-09-13T11:25:10.182 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.6830000000000002e-06 2024-09-13T11:25:10.182 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.183 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.183 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:25:10.183 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:37.793472+0000", 2024-09-13T11:25:10.183 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.5671000000000002e-05 2024-09-13T11:25:10.183 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.183 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.183 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:25:10.183 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:37.793492+0000", 2024-09-13T11:25:10.183 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.0051000000000002e-05 2024-09-13T11:25:10.183 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.183 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.183 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:25:10.183 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:37.795626+0000", 2024-09-13T11:25:10.183 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0021344990000000002 2024-09-13T11:25:10.183 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.184 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.184 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:25:10.184 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:37.795689+0000", 2024-09-13T11:25:10.184 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.2325999999999999e-05 2024-09-13T11:25:10.184 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:10.184 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:25:10.184 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:10.184 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.184 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.184 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:57985 3.3 e80/75 3:c330bb03:::benchmark_data_smithi177_31116_object57984:head v 80'3162, mlcod=80'3159)", 2024-09-13T11:25:10.184 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:38.007076+0000", 2024-09-13T11:25:10.185 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 92.158189413000002, 2024-09-13T11:25:10.185 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.18211438199999999, 2024-09-13T11:25:10.185 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:25:10.185 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:25:10.185 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:10.185 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:25:10.185 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.185 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:25:10.185 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:38.007076+0000", 2024-09-13T11:25:10.185 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:10.185 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.185 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.185 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:25:10.186 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:38.007076+0000", 2024-09-13T11:25:10.186 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:10.186 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.186 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.186 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:25:10.186 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:38.007077+0000", 2024-09-13T11:25:10.186 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.6759999999999999e-06 2024-09-13T11:25:10.186 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.186 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.187 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:25:10.187 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:38.007221+0000", 2024-09-13T11:25:10.187 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00014344299999999999 2024-09-13T11:25:10.187 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.187 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.187 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:25:10.187 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:38.186808+0000", 2024-09-13T11:25:10.187 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.17958726699999999 2024-09-13T11:25:10.187 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.187 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.187 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:25:10.188 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:38.186815+0000", 2024-09-13T11:25:10.188 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.0890000000000002e-06 2024-09-13T11:25:10.188 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.188 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.188 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:25:10.188 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:38.186887+0000", 2024-09-13T11:25:10.188 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.2006000000000001e-05 2024-09-13T11:25:10.188 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.188 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.188 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:25:10.188 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:38.186913+0000", 2024-09-13T11:25:10.188 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.5962e-05 2024-09-13T11:25:10.189 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.189 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.189 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:25:10.189 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:38.189123+0000", 2024-09-13T11:25:10.189 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0022099649999999999 2024-09-13T11:25:10.189 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.189 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.189 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:25:10.189 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:38.189190+0000", 2024-09-13T11:25:10.189 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.6973999999999999e-05 2024-09-13T11:25:10.189 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:10.189 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:25:10.189 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:10.189 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.190 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.190 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPushReply(3.23 84/78 [PushReplyOp(3:c7e44616:::benchmark_data_smithi177_31116_object32371:head)])", 2024-09-13T11:25:10.190 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:41.338210+0000", 2024-09-13T11:25:10.190 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 88.827054693999997, 2024-09-13T11:25:10.190 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.41118492299999998, 2024-09-13T11:25:10.190 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:25:10.190 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:25:10.190 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-13T11:25:10.190 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:25:10.190 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.190 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:25:10.190 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.338210+0000", 2024-09-13T11:25:10.190 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:10.191 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.191 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.191 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:25:10.191 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.338210+0000", 2024-09-13T11:25:10.191 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:10.191 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.191 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.191 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:25:10.191 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.338212+0000", 2024-09-13T11:25:10.191 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.314e-06 2024-09-13T11:25:10.192 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.192 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.192 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:25:10.192 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.338219+0000", 2024-09-13T11:25:10.192 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.6469999999999998e-06 2024-09-13T11:25:10.192 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.192 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.192 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:25:10.192 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.748675+0000", 2024-09-13T11:25:10.192 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.41045567500000002 2024-09-13T11:25:10.192 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.192 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.192 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:25:10.192 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.748684+0000", 2024-09-13T11:25:10.193 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.2150000000000007e-06 2024-09-13T11:25:10.193 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.193 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.193 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:25:10.193 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.748739+0000", 2024-09-13T11:25:10.193 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.4543000000000003e-05 2024-09-13T11:25:10.193 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.193 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.193 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:25:10.193 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.749395+0000", 2024-09-13T11:25:10.193 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00065652899999999999 2024-09-13T11:25:10.193 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:10.193 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:25:10.193 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:10.194 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.194 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.194 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPushReply(3.23 85/78 [PushReplyOp(3:c5c6dce1:::benchmark_data_smithi177_31116_object37293:head)])", 2024-09-13T11:25:10.194 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:42.377862+0000", 2024-09-13T11:25:10.194 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 87.787403234999999, 2024-09-13T11:25:10.194 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.62538808599999995, 2024-09-13T11:25:10.194 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:25:10.194 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:25:10.194 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-13T11:25:10.194 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:25:10.194 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.194 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:25:10.194 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.377862+0000", 2024-09-13T11:25:10.194 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:10.195 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.195 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.195 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:25:10.195 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.377862+0000", 2024-09-13T11:25:10.195 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:10.195 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.195 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.195 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:25:10.195 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.377863+0000", 2024-09-13T11:25:10.195 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.6899999999999996e-07 2024-09-13T11:25:10.195 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.195 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.195 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:25:10.195 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.377866+0000", 2024-09-13T11:25:10.196 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.219e-06 2024-09-13T11:25:10.196 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.196 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.196 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:25:10.196 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:43.001857+0000", 2024-09-13T11:25:10.196 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.62399084299999996 2024-09-13T11:25:10.196 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.196 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.196 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:25:10.196 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:43.001869+0000", 2024-09-13T11:25:10.196 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.2415999999999999e-05 2024-09-13T11:25:10.196 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.196 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.197 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:25:10.197 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:43.002194+0000", 2024-09-13T11:25:10.197 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000324774 2024-09-13T11:25:10.197 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.197 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.197 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:25:10.197 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:43.003250+0000", 2024-09-13T11:25:10.197 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001055865 2024-09-13T11:25:10.197 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:10.197 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:25:10.197 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:10.197 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.197 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.198 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPushReply(3.23 86/78 [PushReplyOp(3:c4ca4518:::benchmark_data_smithi177_31116_object39787:head)])", 2024-09-13T11:25:10.198 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:43.580816+0000", 2024-09-13T11:25:10.198 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 86.584449543999995, 2024-09-13T11:25:10.198 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.14999331199999999, 2024-09-13T11:25:10.198 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:25:10.198 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:25:10.198 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-13T11:25:10.198 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:25:10.198 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.198 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:25:10.198 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:43.580816+0000", 2024-09-13T11:25:10.198 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:10.198 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.199 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.199 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:25:10.199 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:43.580816+0000", 2024-09-13T11:25:10.199 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:10.199 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.199 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.199 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:25:10.199 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:43.580818+0000", 2024-09-13T11:25:10.199 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.7250000000000002e-06 2024-09-13T11:25:10.199 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.199 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.199 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:25:10.199 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:43.580830+0000", 2024-09-13T11:25:10.199 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.2247e-05 2024-09-13T11:25:10.199 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.200 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.200 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:25:10.200 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:43.730204+0000", 2024-09-13T11:25:10.200 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.149373531 2024-09-13T11:25:10.200 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.200 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.200 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:25:10.200 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:43.730213+0000", 2024-09-13T11:25:10.200 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.6989999999999992e-06 2024-09-13T11:25:10.200 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.200 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.201 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:25:10.201 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:43.730274+0000", 2024-09-13T11:25:10.201 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.1419000000000003e-05 2024-09-13T11:25:10.201 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.201 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.201 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:25:10.201 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:43.730809+0000", 2024-09-13T11:25:10.201 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00053469100000000001 2024-09-13T11:25:10.201 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:10.201 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:25:10.201 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:10.201 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.201 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.201 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPush(3.1c 89/77 [PushOp(3:38bcbc71:::benchmark_data_smithi177_31116_object31448:head, version: 59'1946, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:38bcbc71:::benchmark_data_smithi177_31116_object31448:head@59'1946, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:25:10.202 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:46.670723+0000", 2024-09-13T11:25:10.202 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 83.494542010999993, 2024-09-13T11:25:10.202 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.27576065, 2024-09-13T11:25:10.202 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:25:10.202 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:25:10.202 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:25:10.202 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:25:10.202 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.202 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:25:10.202 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.670723+0000", 2024-09-13T11:25:10.202 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:10.202 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.202 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.202 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:25:10.203 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.670723+0000", 2024-09-13T11:25:10.203 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:10.203 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.203 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.203 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:25:10.203 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.670724+0000", 2024-09-13T11:25:10.203 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.4389999999999999e-06 2024-09-13T11:25:10.203 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.203 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.203 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:25:10.203 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.670754+0000", 2024-09-13T11:25:10.203 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.9224999999999999e-05 2024-09-13T11:25:10.203 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.203 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.204 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:25:10.204 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.944369+0000", 2024-09-13T11:25:10.204 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.27361529800000001 2024-09-13T11:25:10.204 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.204 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.204 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:25:10.204 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.944422+0000", 2024-09-13T11:25:10.204 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.3318e-05 2024-09-13T11:25:10.204 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.204 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.204 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:25:10.204 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.944711+0000", 2024-09-13T11:25:10.204 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00028912399999999998 2024-09-13T11:25:10.204 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.205 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.205 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:25:10.205 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.944735+0000", 2024-09-13T11:25:10.205 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.3366999999999998e-05 2024-09-13T11:25:10.205 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.205 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.205 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:25:10.205 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.946484+0000", 2024-09-13T11:25:10.205 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0017488790000000001 2024-09-13T11:25:10.205 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:10.205 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:25:10.205 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:10.205 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.206 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.206 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPush(3.1c 91/77 [PushOp(3:3e18abc2:::benchmark_data_smithi177_31116_object35874:head, version: 62'2235, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:3e18abc2:::benchmark_data_smithi177_31116_object35874:head@62'2235, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:25:10.206 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:48.350968+0000", 2024-09-13T11:25:10.206 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 81.814296698000007, 2024-09-13T11:25:10.206 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.19257698000000001, 2024-09-13T11:25:10.206 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:25:10.206 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:25:10.206 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:25:10.206 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:25:10.206 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.206 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:25:10.206 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.350968+0000", 2024-09-13T11:25:10.206 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:10.207 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.207 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.207 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:25:10.207 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.350968+0000", 2024-09-13T11:25:10.207 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:10.207 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.207 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.207 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:25:10.207 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.350970+0000", 2024-09-13T11:25:10.207 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.796e-06 2024-09-13T11:25:10.207 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.207 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.207 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:25:10.207 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.351003+0000", 2024-09-13T11:25:10.208 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.2631e-05 2024-09-13T11:25:10.208 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.208 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.208 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:25:10.208 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.542140+0000", 2024-09-13T11:25:10.208 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.191136737 2024-09-13T11:25:10.208 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.208 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.208 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:25:10.208 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.542158+0000", 2024-09-13T11:25:10.208 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.8896999999999999e-05 2024-09-13T11:25:10.209 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.209 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.209 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:25:10.209 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.542263+0000", 2024-09-13T11:25:10.209 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000104158 2024-09-13T11:25:10.209 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.209 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.209 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:25:10.209 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.542279+0000", 2024-09-13T11:25:10.209 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5926999999999998e-05 2024-09-13T11:25:10.209 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.209 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.209 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:25:10.210 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.543545+0000", 2024-09-13T11:25:10.210 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0012668340000000001 2024-09-13T11:25:10.210 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:10.210 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:25:10.210 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:10.210 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.210 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.210 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:83211 3.1c e106/77 3:3ae31a0f:::benchmark_data_smithi177_31116_object83210:head v 106'4362, mlcod=106'4360)", 2024-09-13T11:25:10.210 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:04.345264+0000", 2024-09-13T11:25:10.210 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 65.820001046000002, 2024-09-13T11:25:10.210 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.17036547399999999, 2024-09-13T11:25:10.210 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:25:10.210 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:25:10.210 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:10.211 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:25:10.211 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.211 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:25:10.211 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.345264+0000", 2024-09-13T11:25:10.211 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:10.211 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.211 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.211 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:25:10.211 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.345264+0000", 2024-09-13T11:25:10.211 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:10.211 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.211 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.211 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:25:10.211 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.345267+0000", 2024-09-13T11:25:10.212 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.7590000000000002e-06 2024-09-13T11:25:10.212 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.212 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.212 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:25:10.212 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.345298+0000", 2024-09-13T11:25:10.212 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.1173000000000001e-05 2024-09-13T11:25:10.212 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.212 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.212 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:25:10.212 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.512265+0000", 2024-09-13T11:25:10.212 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.16696657100000001 2024-09-13T11:25:10.212 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.212 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.212 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:25:10.213 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.512279+0000", 2024-09-13T11:25:10.213 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.4953999999999999e-05 2024-09-13T11:25:10.213 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.213 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.213 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:25:10.213 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.512402+0000", 2024-09-13T11:25:10.213 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00012277999999999999 2024-09-13T11:25:10.213 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.213 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.213 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:25:10.213 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.512440+0000", 2024-09-13T11:25:10.213 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.7861999999999998e-05 2024-09-13T11:25:10.213 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.213 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.214 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:25:10.214 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.515515+0000", 2024-09-13T11:25:10.214 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00307472 2024-09-13T11:25:10.214 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.214 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.214 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:25:10.214 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.515629+0000", 2024-09-13T11:25:10.214 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00011465500000000001 2024-09-13T11:25:10.214 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:10.214 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:25:10.214 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:10.214 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.214 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.214 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:83223 3.1c e106/77 3:3bc6018d:::benchmark_data_smithi177_31116_object83222:head v 106'4363, mlcod=106'4360)", 2024-09-13T11:25:10.215 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:04.512139+0000", 2024-09-13T11:25:10.215 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 65.653125900000006, 2024-09-13T11:25:10.215 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.471923235, 2024-09-13T11:25:10.215 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:25:10.215 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:25:10.215 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:10.215 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:25:10.215 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.215 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:25:10.215 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.512139+0000", 2024-09-13T11:25:10.215 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:10.215 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.215 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.216 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:25:10.216 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.512139+0000", 2024-09-13T11:25:10.216 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:10.216 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.216 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.216 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:25:10.216 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.512143+0000", 2024-09-13T11:25:10.216 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.427e-06 2024-09-13T11:25:10.216 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.216 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.216 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:25:10.216 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.512227+0000", 2024-09-13T11:25:10.216 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.3965e-05 2024-09-13T11:25:10.217 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.217 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.217 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:25:10.217 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.976330+0000", 2024-09-13T11:25:10.217 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.46410342599999999 2024-09-13T11:25:10.217 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.217 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.217 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:25:10.217 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.976350+0000", 2024-09-13T11:25:10.217 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.9533000000000001e-05 2024-09-13T11:25:10.217 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.217 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.217 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:25:10.218 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.976488+0000", 2024-09-13T11:25:10.218 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00013799400000000001 2024-09-13T11:25:10.218 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.218 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.218 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:25:10.218 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.976543+0000", 2024-09-13T11:25:10.218 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.5442000000000002e-05 2024-09-13T11:25:10.218 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.218 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.218 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:25:10.218 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.983227+0000", 2024-09-13T11:25:10.218 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0066842899999999998 2024-09-13T11:25:10.218 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.218 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.219 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:25:10.219 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.984062+0000", 2024-09-13T11:25:10.219 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00083515800000000002 2024-09-13T11:25:10.219 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:10.219 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:25:10.219 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:10.219 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.219 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.219 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:110256 3.13 e119/83 3:cb00d078:::benchmark_data_smithi177_31116_object110255:head v 119'4823, mlcod=119'4821)", 2024-09-13T11:25:10.219 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:25.284530+0000", 2024-09-13T11:25:10.219 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 44.880735440000002, 2024-09-13T11:25:10.219 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.343491771, 2024-09-13T11:25:10.219 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:25:10.219 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:25:10.220 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:10.220 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:25:10.220 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.220 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:25:10.220 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.284530+0000", 2024-09-13T11:25:10.220 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:10.220 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.220 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.220 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:25:10.220 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.284530+0000", 2024-09-13T11:25:10.220 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:10.220 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.220 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.220 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:25:10.221 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.284532+0000", 2024-09-13T11:25:10.221 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.8870000000000001e-06 2024-09-13T11:25:10.221 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.221 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.221 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:25:10.221 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.284558+0000", 2024-09-13T11:25:10.221 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.6899000000000001e-05 2024-09-13T11:25:10.221 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.221 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.221 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:25:10.221 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.626172+0000", 2024-09-13T11:25:10.221 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.34161370800000002 2024-09-13T11:25:10.221 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.222 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.222 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:25:10.222 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.626187+0000", 2024-09-13T11:25:10.222 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.485e-05 2024-09-13T11:25:10.222 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.222 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.222 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:25:10.222 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.626245+0000", 2024-09-13T11:25:10.222 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.7998000000000003e-05 2024-09-13T11:25:10.222 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.222 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.222 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:25:10.222 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.626260+0000", 2024-09-13T11:25:10.222 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5429000000000001e-05 2024-09-13T11:25:10.223 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.223 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.223 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:25:10.223 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.627924+0000", 2024-09-13T11:25:10.223 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0016634060000000001 2024-09-13T11:25:10.223 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.223 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.223 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:25:10.223 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.628021+0000", 2024-09-13T11:25:10.223 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.7594000000000001e-05 2024-09-13T11:25:10.223 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:10.223 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:25:10.223 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:10.223 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.223 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.224 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPush(3.2b 121/119 [PushOp(3:d53a23e8:::benchmark_data_smithi177_31116_object5366:head, version: 45'342, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:d53a23e8:::benchmark_data_smithi177_31116_object5366:head@45'342, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:25:10.224 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:27.268731+0000", 2024-09-13T11:25:10.224 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 42.896533978000001, 2024-09-13T11:25:10.224 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.72228345000000005, 2024-09-13T11:25:10.224 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:25:10.224 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:25:10.224 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:25:10.224 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:25:10.224 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.224 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:25:10.225 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:27.268731+0000", 2024-09-13T11:25:10.225 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:10.225 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.225 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.225 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:25:10.225 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:27.268731+0000", 2024-09-13T11:25:10.225 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:10.225 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.225 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.225 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:25:10.225 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:27.268733+0000", 2024-09-13T11:25:10.225 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.9740000000000001e-06 2024-09-13T11:25:10.225 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.226 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.226 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:25:10.226 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:27.268778+0000", 2024-09-13T11:25:10.226 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.4679999999999999e-05 2024-09-13T11:25:10.226 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.226 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.226 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:25:10.226 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:27.988594+0000", 2024-09-13T11:25:10.226 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.71981667299999996 2024-09-13T11:25:10.226 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.226 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.226 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:25:10.226 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:27.988652+0000", 2024-09-13T11:25:10.226 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.7640999999999999e-05 2024-09-13T11:25:10.227 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.227 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.227 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:25:10.227 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:27.988889+0000", 2024-09-13T11:25:10.227 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00023706500000000001 2024-09-13T11:25:10.227 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.227 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.227 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:25:10.227 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:27.988933+0000", 2024-09-13T11:25:10.227 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.3865000000000001e-05 2024-09-13T11:25:10.227 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.227 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.227 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:25:10.227 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:27.991015+0000", 2024-09-13T11:25:10.228 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.002081552 2024-09-13T11:25:10.228 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:10.228 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:25:10.228 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:10.228 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.228 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.228 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPush(3.2b 124/119 [PushOp(3:d64c8aaf:::benchmark_data_smithi177_31116_object31700:head, version: 59'1991, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:d64c8aaf:::benchmark_data_smithi177_31116_object31700:head@59'1991, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:25:10.228 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:31.579304+0000", 2024-09-13T11:25:10.228 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 38.585961277000003, 2024-09-13T11:25:10.228 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.14279340800000001, 2024-09-13T11:25:10.228 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:25:10.228 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:25:10.228 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:25:10.229 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:25:10.229 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.229 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:25:10.229 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:31.579304+0000", 2024-09-13T11:25:10.229 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:10.229 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.229 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.229 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:25:10.229 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:31.579304+0000", 2024-09-13T11:25:10.229 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:10.229 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.229 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.229 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:25:10.229 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:31.579305+0000", 2024-09-13T11:25:10.230 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5090000000000001e-06 2024-09-13T11:25:10.230 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.230 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.230 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:25:10.230 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:31.579367+0000", 2024-09-13T11:25:10.230 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.1279999999999996e-05 2024-09-13T11:25:10.230 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.230 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.230 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:25:10.230 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:31.721234+0000", 2024-09-13T11:25:10.230 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.141867883 2024-09-13T11:25:10.230 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.230 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.230 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:25:10.231 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:31.721253+0000", 2024-09-13T11:25:10.231 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.8725000000000001e-05 2024-09-13T11:25:10.231 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.231 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.231 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:25:10.231 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:31.721332+0000", 2024-09-13T11:25:10.231 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.8808999999999997e-05 2024-09-13T11:25:10.231 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.231 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.231 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:25:10.231 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:31.721345+0000", 2024-09-13T11:25:10.231 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3414999999999999e-05 2024-09-13T11:25:10.231 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.231 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.231 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:25:10.232 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:31.722097+0000", 2024-09-13T11:25:10.232 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00075178699999999996 2024-09-13T11:25:10.232 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:10.232 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:25:10.232 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:10.232 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.232 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.232 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:113511 3.2b e124/119 3:d74f5e33:::benchmark_data_smithi177_31116_object113510:head v 124'5156, mlcod=60'2020)", 2024-09-13T11:25:10.232 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:32.032517+0000", 2024-09-13T11:25:10.232 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 38.132748112999998, 2024-09-13T11:25:10.232 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.13874039699999999, 2024-09-13T11:25:10.232 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:25:10.232 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:25:10.233 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:10.233 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:25:10.233 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.233 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:25:10.233 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:32.032517+0000", 2024-09-13T11:25:10.233 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:10.233 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.233 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.233 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:25:10.233 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:32.032517+0000", 2024-09-13T11:25:10.233 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:10.233 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.233 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.233 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:25:10.234 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:32.032518+0000", 2024-09-13T11:25:10.234 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3060000000000001e-06 2024-09-13T11:25:10.234 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.234 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.234 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:25:10.234 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:32.032545+0000", 2024-09-13T11:25:10.234 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.6800000000000001e-05 2024-09-13T11:25:10.234 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.234 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.234 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:25:10.234 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:32.169285+0000", 2024-09-13T11:25:10.234 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.13673950800000001 2024-09-13T11:25:10.234 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.234 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.235 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:25:10.235 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:32.169291+0000", 2024-09-13T11:25:10.235 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.934e-06 2024-09-13T11:25:10.235 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.235 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.235 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:25:10.235 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:32.169381+0000", 2024-09-13T11:25:10.235 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.9860999999999999e-05 2024-09-13T11:25:10.235 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.235 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.235 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:25:10.235 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:32.169406+0000", 2024-09-13T11:25:10.235 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.5052e-05 2024-09-13T11:25:10.235 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.236 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.236 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:25:10.236 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:32.171218+0000", 2024-09-13T11:25:10.236 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0018111659999999999 2024-09-13T11:25:10.236 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.236 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.236 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:25:10.236 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:32.171257+0000", 2024-09-13T11:25:10.236 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.977e-05 2024-09-13T11:25:10.237 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:10.237 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:25:10.237 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:10.237 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.237 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.237 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPush(3.2b 124/119 [PushOp(3:d70bd127:::benchmark_data_smithi177_31116_object32365:head, version: 60'2034, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:d70bd127:::benchmark_data_smithi177_31116_object32365:head@60'2034, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:25:10.237 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:32.037338+0000", 2024-09-13T11:25:10.237 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 38.127926705, 2024-09-13T11:25:10.237 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.24358922699999999, 2024-09-13T11:25:10.237 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:25:10.237 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:25:10.237 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:25:10.237 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:25:10.238 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.238 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:25:10.238 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:32.037338+0000", 2024-09-13T11:25:10.238 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:10.238 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.238 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.238 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:25:10.238 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:32.037338+0000", 2024-09-13T11:25:10.238 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:10.238 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.238 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.238 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:25:10.239 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:32.037340+0000", 2024-09-13T11:25:10.239 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3790000000000001e-06 2024-09-13T11:25:10.239 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.239 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.239 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:25:10.239 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:32.037411+0000", 2024-09-13T11:25:10.239 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.1254000000000006e-05 2024-09-13T11:25:10.240 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.240 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.240 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:25:10.240 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:32.277467+0000", 2024-09-13T11:25:10.240 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.24005563199999999 2024-09-13T11:25:10.240 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.240 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.240 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:25:10.240 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:32.277494+0000", 2024-09-13T11:25:10.240 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.7120000000000001e-05 2024-09-13T11:25:10.240 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.240 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.240 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:25:10.241 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:32.279860+0000", 2024-09-13T11:25:10.241 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0023667559999999998 2024-09-13T11:25:10.241 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.241 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.241 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:25:10.241 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:32.279883+0000", 2024-09-13T11:25:10.241 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.2138999999999999e-05 2024-09-13T11:25:10.241 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.241 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.241 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:25:10.241 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:32.280928+0000", 2024-09-13T11:25:10.241 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0010449470000000001 2024-09-13T11:25:10.241 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:10.241 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:25:10.242 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:10.242 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.242 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.242 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPush(3.2b 125/119 [PushOp(3:d4c5ede9:::benchmark_data_smithi177_31116_object33277:head, version: 60'2092, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:d4c5ede9:::benchmark_data_smithi177_31116_object33277:head@60'2092, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:25:10.242 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:32.403929+0000", 2024-09-13T11:25:10.242 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 37.761336255000003, 2024-09-13T11:25:10.242 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.194592864, 2024-09-13T11:25:10.242 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:25:10.242 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:25:10.242 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:25:10.242 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:25:10.242 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.243 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:25:10.243 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:32.403929+0000", 2024-09-13T11:25:10.243 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:10.243 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.243 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.243 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:25:10.243 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:32.403929+0000", 2024-09-13T11:25:10.243 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:10.243 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.243 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.243 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:25:10.243 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:32.403930+0000", 2024-09-13T11:25:10.243 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.6729999999999999e-06 2024-09-13T11:25:10.243 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.243 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.244 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:25:10.244 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:32.403979+0000", 2024-09-13T11:25:10.244 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.8242000000000003e-05 2024-09-13T11:25:10.244 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.244 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.244 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:25:10.244 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:32.597445+0000", 2024-09-13T11:25:10.244 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.19346676500000001 2024-09-13T11:25:10.244 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.244 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.244 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:25:10.244 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:32.597455+0000", 2024-09-13T11:25:10.244 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.2939999999999994e-06 2024-09-13T11:25:10.245 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.245 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.245 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:25:10.245 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:32.597514+0000", 2024-09-13T11:25:10.245 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.9509000000000002e-05 2024-09-13T11:25:10.245 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.245 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.245 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:25:10.245 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:32.597522+0000", 2024-09-13T11:25:10.245 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.0350000000000001e-06 2024-09-13T11:25:10.245 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.245 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.245 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:25:10.245 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:32.598522+0000", 2024-09-13T11:25:10.246 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00099934599999999991 2024-09-13T11:25:10.246 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:10.246 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:25:10.246 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:10.246 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.246 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.246 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPush(3.2b 128/119 [PushOp(3:d6ff590f:::benchmark_data_smithi177_31116_object64581:head, version: 90'3750, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:d6ff590f:::benchmark_data_smithi177_31116_object64581:head@90'3750, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:25:10.246 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:37.982062+0000", 2024-09-13T11:25:10.246 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 32.183203356, 2024-09-13T11:25:10.246 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.28899675200000002, 2024-09-13T11:25:10.246 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:25:10.247 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:25:10.247 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:25:10.247 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:25:10.247 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.247 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:25:10.247 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:37.982062+0000", 2024-09-13T11:25:10.247 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:10.247 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.247 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.247 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:25:10.247 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:37.982062+0000", 2024-09-13T11:25:10.247 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:10.247 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.247 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.248 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:25:10.248 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:37.982064+0000", 2024-09-13T11:25:10.248 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.2259999999999999e-06 2024-09-13T11:25:10.248 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.248 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.248 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:25:10.248 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:37.982114+0000", 2024-09-13T11:25:10.248 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.0330000000000001e-05 2024-09-13T11:25:10.248 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.248 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.248 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:25:10.248 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:38.269983+0000", 2024-09-13T11:25:10.248 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.28786869999999998 2024-09-13T11:25:10.249 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.249 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.249 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:25:10.249 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:38.270000+0000", 2024-09-13T11:25:10.249 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.6977999999999999e-05 2024-09-13T11:25:10.249 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.249 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.250 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:25:10.250 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:38.270092+0000", 2024-09-13T11:25:10.250 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.221e-05 2024-09-13T11:25:10.250 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.250 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.250 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:25:10.251 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:38.270103+0000", 2024-09-13T11:25:10.251 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.1253999999999999e-05 2024-09-13T11:25:10.251 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.251 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.251 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:25:10.251 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:38.271058+0000", 2024-09-13T11:25:10.251 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00095505399999999999 2024-09-13T11:25:10.252 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:10.252 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:25:10.252 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:10.252 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.252 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.252 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:150366 3.2c e149/144 3:348937c7:::benchmark_data_smithi177_31116_object10063:head v 149'5822, mlcod=149'5819)", 2024-09-13T11:25:10.252 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:25:06.284607+0000", 2024-09-13T11:25:10.252 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 3.8806580660000001, 2024-09-13T11:25:10.253 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.16600746599999999, 2024-09-13T11:25:10.253 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:25:10.253 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:25:10.253 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:10.253 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:25:10.253 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.253 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:25:10.253 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:06.284607+0000", 2024-09-13T11:25:10.253 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:10.253 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.253 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.253 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:25:10.253 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:06.284607+0000", 2024-09-13T11:25:10.254 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:10.254 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.254 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.254 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:25:10.254 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:06.284608+0000", 2024-09-13T11:25:10.254 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.018e-06 2024-09-13T11:25:10.254 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.254 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.254 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:25:10.254 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:06.284612+0000", 2024-09-13T11:25:10.254 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.3239999999999996e-06 2024-09-13T11:25:10.254 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.254 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.254 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:25:10.254 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:06.449402+0000", 2024-09-13T11:25:10.255 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.16478921099999999 2024-09-13T11:25:10.255 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.255 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.255 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:25:10.255 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:06.449408+0000", 2024-09-13T11:25:10.255 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.1990000000000003e-06 2024-09-13T11:25:10.255 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.255 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.255 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:25:10.255 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:06.449467+0000", 2024-09-13T11:25:10.255 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.8792999999999998e-05 2024-09-13T11:25:10.255 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.255 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.255 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:25:10.255 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:06.449490+0000", 2024-09-13T11:25:10.256 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.3895000000000001e-05 2024-09-13T11:25:10.256 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.256 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.256 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:25:10.256 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:06.450581+0000", 2024-09-13T11:25:10.256 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001090653 2024-09-13T11:25:10.256 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.256 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.256 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:25:10.256 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:06.450614+0000", 2024-09-13T11:25:10.256 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.3373e-05 2024-09-13T11:25:10.256 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:10.256 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:25:10.256 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:10.257 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:10.257 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:25:10.257 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-09-13T11:25:10.258 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-09-13T11:25:10.259 INFO:tasks.ceph.osd.14.smithi162.stderr:2024-09-13T11:25:10.256+0000 7f3df5903640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33154) UID: 0 2024-09-13T11:25:10.359 INFO:tasks.ceph.osd.10.smithi162.stderr:2024-09-13T11:25:10.356+0000 7fb59c9f0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26317) UID: 0 2024-09-13T11:25:10.459 INFO:tasks.ceph.osd.5.smithi092.stderr:2024-09-13T11:25:10.453+0000 7f6f0b705640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26346) UID: 0 2024-09-13T11:25:10.460 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-09-13T11:25:10.460 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [ 2024-09-13T11:25:10.460 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.460 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:155896 3.16 3:6b940528:::benchmark_data_smithi177_31116_object15593:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e151)", 2024-09-13T11:25:10.460 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:25:10.458776+0000", 2024-09-13T11:25:10.460 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 0.001236123, 2024-09-13T11:25:10.460 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001262057, 2024-09-13T11:25:10.460 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:25:10.461 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:25:10.461 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "waiting for sub ops", 2024-09-13T11:25:10.461 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:25:10.461 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:25:10.461 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:10.461 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 155896 2024-09-13T11:25:10.461 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.461 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:25:10.461 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.461 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:25:10.461 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:10.458776+0000", 2024-09-13T11:25:10.461 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:10.461 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.462 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.462 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:25:10.462 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:10.458776+0000", 2024-09-13T11:25:10.462 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:10.462 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.462 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.462 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:25:10.462 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:10.458778+0000", 2024-09-13T11:25:10.462 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.1399999999999998e-06 2024-09-13T11:25:10.462 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.462 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.462 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:25:10.462 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:10.458784+0000", 2024-09-13T11:25:10.462 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.1e-06 2024-09-13T11:25:10.462 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.463 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.463 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:25:10.463 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:10.458796+0000", 2024-09-13T11:25:10.463 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.2294e-05 2024-09-13T11:25:10.463 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.463 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.463 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:25:10.463 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:10.458804+0000", 2024-09-13T11:25:10.463 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.7440000000000007e-06 2024-09-13T11:25:10.463 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.463 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.463 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:25:10.463 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:10.458867+0000", 2024-09-13T11:25:10.463 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.2959000000000005e-05 2024-09-13T11:25:10.463 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.464 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.464 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:25:10.464 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:10.458988+0000", 2024-09-13T11:25:10.464 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00012061999999999999 2024-09-13T11:25:10.464 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.464 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.464 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 15", 2024-09-13T11:25:10.464 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:10.459091+0000", 2024-09-13T11:25:10.464 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00010280699999999999 2024-09-13T11:25:10.464 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:10.464 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:25:10.464 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:10.464 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:10.465 INFO:teuthology.orchestra.run.smithi177.stdout: ], 2024-09-13T11:25:10.465 INFO:teuthology.orchestra.run.smithi177.stdout: "num_ops": 1 2024-09-13T11:25:10.465 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-09-13T11:25:10.483 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-09-13T11:25:10.561 INFO:tasks.ceph.osd.14.smithi162.stderr:2024-09-13T11:25:10.556+0000 7f3df5903640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33154) UID: 0 2024-09-13T11:25:10.640 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-09-13T11:25:10.641 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [], 2024-09-13T11:25:10.641 INFO:teuthology.orchestra.run.smithi177.stdout: "complaint_time": 30, 2024-09-13T11:25:10.641 INFO:teuthology.orchestra.run.smithi177.stdout: "num_blocked_ops": 0 2024-09-13T11:25:10.641 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-09-13T11:25:10.661 INFO:tasks.ceph.osd.15.smithi177.stderr:2024-09-13T11:25:10.658+0000 7f9d6ce77640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26578) UID: 0 2024-09-13T11:25:10.664 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-09-13T11:25:10.762 INFO:tasks.ceph.osd.10.smithi162.stderr:2024-09-13T11:25:10.760+0000 7fb59c9f0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26317) UID: 0 2024-09-13T11:25:10.863 INFO:tasks.ceph.osd.15.smithi177.stderr:2024-09-13T11:25:10.858+0000 7f9d6ce77640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26578) UID: 0 2024-09-13T11:25:10.864 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-09-13T11:25:10.864 INFO:teuthology.orchestra.run.smithi177.stdout: "size": 20, 2024-09-13T11:25:10.864 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 600, 2024-09-13T11:25:10.864 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [ 2024-09-13T11:25:10.864 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.864 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:79 3.1 e32/30 3:841df5b5:::benchmark_data_smithi177_31116_object78:head v 32'6, mlcod=32'4)", 2024-09-13T11:25:10.864 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:22:27.057292+0000", 2024-09-13T11:25:10.864 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 163.80542982200001, 2024-09-13T11:25:10.865 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.89296786800000005, 2024-09-13T11:25:10.865 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:25:10.865 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:25:10.865 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:10.865 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:25:10.865 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.865 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:25:10.865 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.057292+0000", 2024-09-13T11:25:10.865 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:10.865 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.865 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.865 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:25:10.865 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.057292+0000", 2024-09-13T11:25:10.866 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:10.866 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.866 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.866 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:25:10.866 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.057293+0000", 2024-09-13T11:25:10.866 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.6109999999999999e-06 2024-09-13T11:25:10.866 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.866 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.866 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:25:10.866 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.057330+0000", 2024-09-13T11:25:10.866 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.6721999999999998e-05 2024-09-13T11:25:10.866 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.866 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.867 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:25:10.867 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.948197+0000", 2024-09-13T11:25:10.867 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.890867042 2024-09-13T11:25:10.867 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.867 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.867 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:25:10.867 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.948213+0000", 2024-09-13T11:25:10.867 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5826000000000001e-05 2024-09-13T11:25:10.867 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.867 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.868 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:25:10.868 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.948377+0000", 2024-09-13T11:25:10.868 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00016348600000000001 2024-09-13T11:25:10.868 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.868 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.868 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:25:10.868 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.948447+0000", 2024-09-13T11:25:10.868 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.0891999999999998e-05 2024-09-13T11:25:10.868 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.868 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.868 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:25:10.869 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.950150+0000", 2024-09-13T11:25:10.869 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001702329 2024-09-13T11:25:10.869 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.869 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.869 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:25:10.869 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.950260+0000", 2024-09-13T11:25:10.869 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00010996 2024-09-13T11:25:10.869 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:10.869 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:25:10.869 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:10.869 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.869 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.870 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:30828 3.f 3:f3a9c6a9:::benchmark_data_smithi177_31116_object30827:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e59)", 2024-09-13T11:25:10.870 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:09.503661+0000", 2024-09-13T11:25:10.870 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 121.359060473, 2024-09-13T11:25:10.870 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.26477667100000002, 2024-09-13T11:25:10.870 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:25:10.870 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:25:10.870 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:10.870 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:25:10.870 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:25:10.870 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:10.870 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 30828 2024-09-13T11:25:10.871 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.871 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:25:10.871 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.871 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:25:10.871 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.503661+0000", 2024-09-13T11:25:10.871 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:10.871 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.871 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.871 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:25:10.871 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.503661+0000", 2024-09-13T11:25:10.871 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:10.872 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.872 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.872 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:25:10.872 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.503663+0000", 2024-09-13T11:25:10.872 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.968e-06 2024-09-13T11:25:10.872 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.872 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.872 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:25:10.872 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.503687+0000", 2024-09-13T11:25:10.872 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.3983999999999999e-05 2024-09-13T11:25:10.872 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.872 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.873 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:25:10.873 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.503701+0000", 2024-09-13T11:25:10.873 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3416e-05 2024-09-13T11:25:10.873 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.873 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.873 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:25:10.873 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.503708+0000", 2024-09-13T11:25:10.873 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.9530000000000002e-06 2024-09-13T11:25:10.873 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.873 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.873 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:25:10.873 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.503759+0000", 2024-09-13T11:25:10.873 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.1075999999999997e-05 2024-09-13T11:25:10.873 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.874 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.874 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:25:10.874 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.503861+0000", 2024-09-13T11:25:10.874 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000102372 2024-09-13T11:25:10.874 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.874 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.874 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 12", 2024-09-13T11:25:10.874 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.503952+0000", 2024-09-13T11:25:10.874 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.1135000000000001e-05 2024-09-13T11:25:10.874 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.874 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.874 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:25:10.874 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.505340+0000", 2024-09-13T11:25:10.875 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0013877169999999999 2024-09-13T11:25:10.875 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.875 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.875 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:25:10.875 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.768281+0000", 2024-09-13T11:25:10.875 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.26294092400000002 2024-09-13T11:25:10.875 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.875 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.875 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:25:10.875 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.768369+0000", 2024-09-13T11:25:10.875 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.7818999999999994e-05 2024-09-13T11:25:10.875 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.875 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.876 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:25:10.876 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.768438+0000", 2024-09-13T11:25:10.876 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.9307000000000006e-05 2024-09-13T11:25:10.876 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:10.876 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:25:10.876 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:10.876 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.876 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.876 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:45783 3.16 3.17c10a76 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e69)", 2024-09-13T11:25:10.876 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:20.947774+0000", 2024-09-13T11:25:10.876 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 109.914947571, 2024-09-13T11:25:10.876 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.96682922400000004, 2024-09-13T11:25:10.877 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:25:10.877 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:25:10.877 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-13T11:25:10.877 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:25:10.877 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:25:10.877 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:10.877 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 45783 2024-09-13T11:25:10.877 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.877 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:25:10.877 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.877 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:25:10.877 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:20.947774+0000", 2024-09-13T11:25:10.877 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:10.878 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.878 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.878 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:25:10.878 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:20.947774+0000", 2024-09-13T11:25:10.878 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:10.878 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.878 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.878 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:25:10.878 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:20.947777+0000", 2024-09-13T11:25:10.878 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.57e-06 2024-09-13T11:25:10.878 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.878 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.879 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:25:10.879 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:20.947802+0000", 2024-09-13T11:25:10.879 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.5528999999999999e-05 2024-09-13T11:25:10.879 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.879 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.879 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:25:10.879 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:20.947813+0000", 2024-09-13T11:25:10.879 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0473e-05 2024-09-13T11:25:10.879 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.879 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.879 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:25:10.879 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:20.947821+0000", 2024-09-13T11:25:10.880 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.3440000000000001e-06 2024-09-13T11:25:10.880 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.880 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.880 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:25:10.880 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:20.947933+0000", 2024-09-13T11:25:10.880 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00011210200000000001 2024-09-13T11:25:10.880 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.880 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.880 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for peered", 2024-09-13T11:25:10.880 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:20.947942+0000", 2024-09-13T11:25:10.880 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.6610000000000006e-06 2024-09-13T11:25:10.880 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.880 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.881 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:25:10.881 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:21.914579+0000", 2024-09-13T11:25:10.881 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.96663672300000003 2024-09-13T11:25:10.881 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.881 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.881 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:25:10.881 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:21.914603+0000", 2024-09-13T11:25:10.881 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.4822000000000001e-05 2024-09-13T11:25:10.881 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:10.881 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:25:10.881 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:10.881 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.881 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.881 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:54625 3.1f 3:f812cdfb:::benchmark_data_smithi177_31116_object54624:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e75)", 2024-09-13T11:25:10.882 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:30.979327+0000", 2024-09-13T11:25:10.882 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 99.883394781999996, 2024-09-13T11:25:10.882 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.96163670899999998, 2024-09-13T11:25:10.882 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:25:10.882 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:25:10.882 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-13T11:25:10.882 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:25:10.882 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:25:10.882 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:10.882 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 54625 2024-09-13T11:25:10.882 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.882 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:25:10.882 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.883 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:25:10.883 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.979327+0000", 2024-09-13T11:25:10.883 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:10.883 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.883 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.883 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:25:10.883 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.979327+0000", 2024-09-13T11:25:10.883 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:10.883 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.883 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.883 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:25:10.883 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.979329+0000", 2024-09-13T11:25:10.883 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.686e-06 2024-09-13T11:25:10.883 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.884 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.884 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:25:10.884 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.979364+0000", 2024-09-13T11:25:10.884 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.5750000000000002e-05 2024-09-13T11:25:10.884 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.884 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.884 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:25:10.884 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.979408+0000", 2024-09-13T11:25:10.884 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.3795999999999999e-05 2024-09-13T11:25:10.884 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.884 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.884 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:25:10.884 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.979427+0000", 2024-09-13T11:25:10.885 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.9040999999999999e-05 2024-09-13T11:25:10.885 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.885 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.885 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:25:10.885 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.122291+0000", 2024-09-13T11:25:10.885 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.14286373599999999 2024-09-13T11:25:10.885 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.885 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.885 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for peered", 2024-09-13T11:25:10.885 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.122298+0000", 2024-09-13T11:25:10.885 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.3719999999999999e-06 2024-09-13T11:25:10.885 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.885 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.886 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:25:10.886 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.940341+0000", 2024-09-13T11:25:10.886 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.81804249699999998 2024-09-13T11:25:10.886 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.886 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.886 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:25:10.886 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.940964+0000", 2024-09-13T11:25:10.886 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00062283100000000001 2024-09-13T11:25:10.886 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:10.886 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:25:10.886 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:10.886 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.886 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.886 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:54649 3.1c 3:3a457a93:::benchmark_data_smithi177_31116_object54648:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e75)", 2024-09-13T11:25:10.887 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:30.979460+0000", 2024-09-13T11:25:10.887 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 99.883261591999997, 2024-09-13T11:25:10.887 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.95579424000000002, 2024-09-13T11:25:10.887 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:25:10.887 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:25:10.887 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-13T11:25:10.887 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:25:10.887 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:25:10.887 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:10.887 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 54649 2024-09-13T11:25:10.887 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.887 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:25:10.887 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.888 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:25:10.888 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.979460+0000", 2024-09-13T11:25:10.888 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:10.888 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.888 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.888 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:25:10.888 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.979460+0000", 2024-09-13T11:25:10.888 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:10.888 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.888 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.889 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:25:10.889 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.979462+0000", 2024-09-13T11:25:10.889 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5799999999999999e-06 2024-09-13T11:25:10.889 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.889 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.889 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:25:10.889 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.979524+0000", 2024-09-13T11:25:10.889 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.2521999999999995e-05 2024-09-13T11:25:10.889 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.889 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.889 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:25:10.889 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.979561+0000", 2024-09-13T11:25:10.890 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.6349e-05 2024-09-13T11:25:10.890 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.890 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.890 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:25:10.890 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.979571+0000", 2024-09-13T11:25:10.890 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0457e-05 2024-09-13T11:25:10.890 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.890 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.890 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:25:10.890 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.121029+0000", 2024-09-13T11:25:10.890 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.14145766200000001 2024-09-13T11:25:10.890 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.890 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.891 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for peered", 2024-09-13T11:25:10.891 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.121039+0000", 2024-09-13T11:25:10.891 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0084e-05 2024-09-13T11:25:10.891 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.891 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.891 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:25:10.891 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.935157+0000", 2024-09-13T11:25:10.891 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.81411807899999999 2024-09-13T11:25:10.891 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.891 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.891 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:25:10.892 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.935254+0000", 2024-09-13T11:25:10.892 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.7507e-05 2024-09-13T11:25:10.892 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:10.892 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:25:10.892 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:10.892 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.892 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.892 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:54654 3.1c 3:3f0bb31a:::benchmark_data_smithi177_31116_object54653:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e75)", 2024-09-13T11:25:10.892 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:30.979665+0000", 2024-09-13T11:25:10.892 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 99.883056427, 2024-09-13T11:25:10.892 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.95623908099999999, 2024-09-13T11:25:10.892 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:25:10.892 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:25:10.892 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-13T11:25:10.893 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:25:10.893 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:25:10.893 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:10.893 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 54654 2024-09-13T11:25:10.893 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.893 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:25:10.893 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.893 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:25:10.893 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.979665+0000", 2024-09-13T11:25:10.893 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:10.893 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.893 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.893 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:25:10.894 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.979665+0000", 2024-09-13T11:25:10.894 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:10.894 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.894 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.894 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:25:10.894 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.979667+0000", 2024-09-13T11:25:10.894 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3540000000000001e-06 2024-09-13T11:25:10.894 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.894 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.894 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:25:10.894 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.979703+0000", 2024-09-13T11:25:10.894 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.6758000000000003e-05 2024-09-13T11:25:10.894 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.894 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.895 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:25:10.895 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.979714+0000", 2024-09-13T11:25:10.895 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0804e-05 2024-09-13T11:25:10.895 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.895 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.895 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:25:10.895 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.979724+0000", 2024-09-13T11:25:10.895 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0040999999999999e-05 2024-09-13T11:25:10.895 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.895 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.895 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:25:10.895 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.121061+0000", 2024-09-13T11:25:10.895 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.141336713 2024-09-13T11:25:10.895 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.896 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.896 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for peered", 2024-09-13T11:25:10.896 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.121067+0000", 2024-09-13T11:25:10.896 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.4150000000000004e-06 2024-09-13T11:25:10.896 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.896 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.896 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:25:10.896 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.935776+0000", 2024-09-13T11:25:10.896 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.81470863100000002 2024-09-13T11:25:10.896 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.896 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.896 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:25:10.896 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.935904+0000", 2024-09-13T11:25:10.897 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00012836499999999999 2024-09-13T11:25:10.897 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:10.897 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:25:10.897 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:10.897 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.897 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.897 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPushReply(3.11 75/70 [PushReplyOp(3:8df10771:::benchmark_data_smithi177_31116_object24734:head)])", 2024-09-13T11:25:10.897 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:31.149798+0000", 2024-09-13T11:25:10.897 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 99.712923416999999, 2024-09-13T11:25:10.897 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.32402702500000002, 2024-09-13T11:25:10.897 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:25:10.897 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:25:10.897 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-13T11:25:10.897 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:25:10.898 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.898 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:25:10.898 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.149798+0000", 2024-09-13T11:25:10.898 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:10.898 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.898 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.898 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:25:10.898 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.149798+0000", 2024-09-13T11:25:10.898 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:10.898 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.898 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.898 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:25:10.898 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.149799+0000", 2024-09-13T11:25:10.899 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.5700000000000001e-07 2024-09-13T11:25:10.899 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.899 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.899 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:25:10.899 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.149803+0000", 2024-09-13T11:25:10.899 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.0040000000000001e-06 2024-09-13T11:25:10.899 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.899 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.899 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:25:10.899 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.472172+0000", 2024-09-13T11:25:10.899 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.32236856699999999 2024-09-13T11:25:10.899 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.899 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.899 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:25:10.900 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.472187+0000", 2024-09-13T11:25:10.900 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5016e-05 2024-09-13T11:25:10.900 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.900 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.900 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:25:10.900 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.472302+0000", 2024-09-13T11:25:10.900 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000115121 2024-09-13T11:25:10.900 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.900 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.900 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:25:10.900 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.473825+0000", 2024-09-13T11:25:10.900 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0015234599999999999 2024-09-13T11:25:10.900 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:10.901 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:25:10.901 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:10.901 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.901 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.901 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:54668 3.1c 3:3d2c5e40:::benchmark_data_smithi177_31116_object54667:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e75)", 2024-09-13T11:25:10.901 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:31.245525+0000", 2024-09-13T11:25:10.901 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 99.617197160000003, 2024-09-13T11:25:10.901 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.69828973000000005, 2024-09-13T11:25:10.901 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:25:10.901 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:25:10.901 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:10.901 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:25:10.901 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:25:10.902 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:10.902 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 54668 2024-09-13T11:25:10.902 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.902 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:25:10.902 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.902 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:25:10.902 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.245525+0000", 2024-09-13T11:25:10.902 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:10.902 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.902 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.902 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:25:10.902 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.245525+0000", 2024-09-13T11:25:10.902 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:10.902 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.903 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.903 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:25:10.903 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.245526+0000", 2024-09-13T11:25:10.903 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.719e-06 2024-09-13T11:25:10.903 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.903 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.903 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:25:10.903 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.245547+0000", 2024-09-13T11:25:10.903 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.1084999999999999e-05 2024-09-13T11:25:10.903 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.903 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.903 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:25:10.903 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.245558+0000", 2024-09-13T11:25:10.903 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0934000000000001e-05 2024-09-13T11:25:10.904 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.904 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.904 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:25:10.904 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.245564+0000", 2024-09-13T11:25:10.904 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.6269999999999997e-06 2024-09-13T11:25:10.904 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.904 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.904 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:25:10.904 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.245620+0000", 2024-09-13T11:25:10.904 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.6289999999999998e-05 2024-09-13T11:25:10.904 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.904 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.904 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for peered", 2024-09-13T11:25:10.905 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.245632+0000", 2024-09-13T11:25:10.905 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.1748e-05 2024-09-13T11:25:10.905 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.905 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.905 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:25:10.905 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.935976+0000", 2024-09-13T11:25:10.905 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.69034455299999997 2024-09-13T11:25:10.905 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.905 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.905 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:25:10.905 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.936302+0000", 2024-09-13T11:25:10.905 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000325397 2024-09-13T11:25:10.905 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.905 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.906 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 13", 2024-09-13T11:25:10.906 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.936459+0000", 2024-09-13T11:25:10.906 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00015754300000000001 2024-09-13T11:25:10.906 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.906 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.906 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:25:10.906 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.938813+0000", 2024-09-13T11:25:10.906 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0023530999999999999 2024-09-13T11:25:10.906 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.906 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.906 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:25:10.906 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.943695+0000", 2024-09-13T11:25:10.906 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.004882506 2024-09-13T11:25:10.906 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.907 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.907 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:25:10.907 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.943767+0000", 2024-09-13T11:25:10.907 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.2454999999999997e-05 2024-09-13T11:25:10.907 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.907 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.907 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:25:10.907 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.943814+0000", 2024-09-13T11:25:10.907 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.6773000000000001e-05 2024-09-13T11:25:10.907 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:10.907 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:25:10.907 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:10.907 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.908 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.908 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPush(3.16 84/70 [PushOp(3:6f7be541:::benchmark_data_smithi177_31116_object43332:head, version: 66'2788, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:6f7be541:::benchmark_data_smithi177_31116_object43332:head@66'2788, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:25:10.908 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:41.207515+0000", 2024-09-13T11:25:10.908 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 89.655206389, 2024-09-13T11:25:10.908 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.52643771500000003, 2024-09-13T11:25:10.908 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:25:10.908 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:25:10.908 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:25:10.908 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:25:10.908 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.908 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:25:10.908 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.207515+0000", 2024-09-13T11:25:10.908 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:10.909 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.909 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.909 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:25:10.909 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.207515+0000", 2024-09-13T11:25:10.909 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:10.909 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.909 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.909 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:25:10.909 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.207516+0000", 2024-09-13T11:25:10.910 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.096e-06 2024-09-13T11:25:10.910 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.910 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.910 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:25:10.910 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.207717+0000", 2024-09-13T11:25:10.910 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000200199 2024-09-13T11:25:10.910 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.910 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.910 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:25:10.910 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.732785+0000", 2024-09-13T11:25:10.910 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.52506869599999995 2024-09-13T11:25:10.910 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.911 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.911 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:25:10.911 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.732801+0000", 2024-09-13T11:25:10.911 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5305999999999999e-05 2024-09-13T11:25:10.911 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.911 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.911 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:25:10.911 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.732889+0000", 2024-09-13T11:25:10.911 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.8157000000000003e-05 2024-09-13T11:25:10.911 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.911 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.911 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:25:10.911 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.732901+0000", 2024-09-13T11:25:10.912 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.1866e-05 2024-09-13T11:25:10.912 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.912 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.912 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:25:10.912 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.733953+0000", 2024-09-13T11:25:10.912 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001052395 2024-09-13T11:25:10.912 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:10.912 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:25:10.912 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:10.912 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.912 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.913 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPushReply(3.1f 85/78 [PushReplyOp(3:fd221d21:::benchmark_data_smithi177_31116_object7639:head)])", 2024-09-13T11:25:10.913 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:42.198715+0000", 2024-09-13T11:25:10.913 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 88.664006819999997, 2024-09-13T11:25:10.913 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.48930391099999998, 2024-09-13T11:25:10.913 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:25:10.913 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:25:10.913 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-13T11:25:10.913 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:25:10.913 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.913 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:25:10.913 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.198715+0000", 2024-09-13T11:25:10.913 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:10.914 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.914 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.914 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:25:10.914 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.198715+0000", 2024-09-13T11:25:10.914 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:10.914 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.914 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.914 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:25:10.914 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.198716+0000", 2024-09-13T11:25:10.914 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.3500000000000005e-07 2024-09-13T11:25:10.914 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.914 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.914 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:25:10.915 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.198719+0000", 2024-09-13T11:25:10.915 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.4539999999999999e-06 2024-09-13T11:25:10.915 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.915 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.915 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:25:10.915 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.684732+0000", 2024-09-13T11:25:10.915 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.48601262099999998 2024-09-13T11:25:10.915 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.915 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.915 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:25:10.915 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.684746+0000", 2024-09-13T11:25:10.915 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.4518999999999999e-05 2024-09-13T11:25:10.915 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.916 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.916 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:25:10.916 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.684911+0000", 2024-09-13T11:25:10.916 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000164728 2024-09-13T11:25:10.916 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.916 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.916 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:25:10.916 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.688019+0000", 2024-09-13T11:25:10.916 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0031077539999999999 2024-09-13T11:25:10.916 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:10.916 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:25:10.916 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:10.916 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.917 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.917 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPushReply(3.1c 85/77 [PushReplyOp(3:3a669437:::benchmark_data_smithi177_31116_object25280:head)])", 2024-09-13T11:25:10.917 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:42.352466+0000", 2024-09-13T11:25:10.917 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 88.510255925999999, 2024-09-13T11:25:10.917 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.36670725300000001, 2024-09-13T11:25:10.917 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:25:10.917 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:25:10.917 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-13T11:25:10.917 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:25:10.917 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.917 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:25:10.917 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.352466+0000", 2024-09-13T11:25:10.917 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:10.918 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.918 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.918 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:25:10.918 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.352466+0000", 2024-09-13T11:25:10.918 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:10.918 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.918 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.918 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:25:10.918 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.352467+0000", 2024-09-13T11:25:10.918 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.1429999999999999e-06 2024-09-13T11:25:10.918 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.918 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.918 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:25:10.919 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.352474+0000", 2024-09-13T11:25:10.919 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.6100000000000002e-06 2024-09-13T11:25:10.919 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.919 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.919 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:25:10.919 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.717011+0000", 2024-09-13T11:25:10.919 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.36453790200000002 2024-09-13T11:25:10.919 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.919 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.919 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:25:10.919 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.717032+0000", 2024-09-13T11:25:10.919 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.0999e-05 2024-09-13T11:25:10.919 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.920 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.920 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:25:10.920 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.717162+0000", 2024-09-13T11:25:10.920 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000129159 2024-09-13T11:25:10.920 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.920 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.920 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:25:10.920 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.719173+0000", 2024-09-13T11:25:10.920 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0020114400000000002 2024-09-13T11:25:10.920 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:10.920 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:25:10.920 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:10.921 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.921 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.921 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPush(3.16 85/70 [PushOp(3:6aef2f27:::benchmark_data_smithi177_31116_object44090:head, version: 68'2844, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:6aef2f27:::benchmark_data_smithi177_31116_object44090:head@68'2844, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:25:10.921 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:42.574221+0000", 2024-09-13T11:25:10.921 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 88.288500818000003, 2024-09-13T11:25:10.921 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.25345585700000001, 2024-09-13T11:25:10.921 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:25:10.921 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:25:10.921 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:25:10.921 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:25:10.921 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.921 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:25:10.921 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.574221+0000", 2024-09-13T11:25:10.922 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:10.922 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.922 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.922 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:25:10.922 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.574221+0000", 2024-09-13T11:25:10.922 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:10.922 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.922 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.922 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:25:10.922 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.574252+0000", 2024-09-13T11:25:10.922 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.0973000000000003e-05 2024-09-13T11:25:10.922 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.922 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.922 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:25:10.923 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.574434+0000", 2024-09-13T11:25:10.923 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000182265 2024-09-13T11:25:10.923 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.923 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.923 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:25:10.923 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.825391+0000", 2024-09-13T11:25:10.923 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.250957023 2024-09-13T11:25:10.923 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.923 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.923 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:25:10.923 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.825442+0000", 2024-09-13T11:25:10.923 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.1267999999999998e-05 2024-09-13T11:25:10.924 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.924 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.924 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:25:10.924 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.825692+0000", 2024-09-13T11:25:10.924 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00024924700000000002 2024-09-13T11:25:10.924 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.924 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.924 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:25:10.924 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.825715+0000", 2024-09-13T11:25:10.924 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.3176e-05 2024-09-13T11:25:10.924 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.924 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.925 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:25:10.925 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.827677+0000", 2024-09-13T11:25:10.925 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0019619049999999999 2024-09-13T11:25:10.925 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:10.925 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:25:10.925 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:10.925 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.925 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.925 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPushReply(3.1f 85/78 [PushReplyOp(3:fecb0d6a:::benchmark_data_smithi177_31116_object7644:head)])", 2024-09-13T11:25:10.925 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:42.705351+0000", 2024-09-13T11:25:10.925 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 88.157371084000005, 2024-09-13T11:25:10.925 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.290549681, 2024-09-13T11:25:10.925 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:25:10.926 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:25:10.926 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-13T11:25:10.926 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:25:10.926 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.926 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:25:10.926 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.705351+0000", 2024-09-13T11:25:10.926 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:10.926 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.926 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.926 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:25:10.926 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.705351+0000", 2024-09-13T11:25:10.926 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:10.927 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.927 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.927 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:25:10.927 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.705353+0000", 2024-09-13T11:25:10.927 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.4169999999999999e-06 2024-09-13T11:25:10.927 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.927 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.927 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:25:10.927 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.705365+0000", 2024-09-13T11:25:10.927 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.1894e-05 2024-09-13T11:25:10.927 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.927 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.927 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:25:10.928 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.992081+0000", 2024-09-13T11:25:10.928 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.28671641199999998 2024-09-13T11:25:10.928 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.928 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.928 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:25:10.928 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.992111+0000", 2024-09-13T11:25:10.928 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.9427000000000001e-05 2024-09-13T11:25:10.928 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.928 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.928 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:25:10.928 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.993573+0000", 2024-09-13T11:25:10.928 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0014623590000000001 2024-09-13T11:25:10.929 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.929 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.929 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:25:10.929 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.995900+0000", 2024-09-13T11:25:10.929 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0023271720000000002 2024-09-13T11:25:10.929 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:10.929 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:25:10.929 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:10.929 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.929 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.929 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPushReply(3.21 89/78 [PushReplyOp(3:8534d087:::benchmark_data_smithi177_31116_object5474:head)])", 2024-09-13T11:25:10.929 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:46.632834+0000", 2024-09-13T11:25:10.930 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 84.229887644000002, 2024-09-13T11:25:10.930 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.324805556, 2024-09-13T11:25:10.930 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:25:10.930 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:25:10.930 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-13T11:25:10.930 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:25:10.930 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.930 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:25:10.930 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.632834+0000", 2024-09-13T11:25:10.930 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:10.930 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.930 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.930 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:25:10.931 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.632834+0000", 2024-09-13T11:25:10.931 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:10.931 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.931 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.931 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:25:10.931 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.632835+0000", 2024-09-13T11:25:10.931 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.4270000000000001e-06 2024-09-13T11:25:10.931 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.931 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.931 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:25:10.931 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.632840+0000", 2024-09-13T11:25:10.931 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.7809999999999996e-06 2024-09-13T11:25:10.931 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.932 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.932 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:25:10.932 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.956282+0000", 2024-09-13T11:25:10.932 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.323441639 2024-09-13T11:25:10.932 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.932 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.932 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:25:10.932 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.956293+0000", 2024-09-13T11:25:10.932 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.083e-05 2024-09-13T11:25:10.932 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.932 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.932 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:25:10.933 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.956395+0000", 2024-09-13T11:25:10.933 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00010184200000000001 2024-09-13T11:25:10.933 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.933 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.933 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:25:10.933 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.957640+0000", 2024-09-13T11:25:10.933 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0012450370000000001 2024-09-13T11:25:10.933 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:10.933 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:25:10.933 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:10.933 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.933 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.933 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPushReply(3.1f 89/78 [PushReplyOp(3:ff150976:::benchmark_data_smithi177_31116_object12907:head)])", 2024-09-13T11:25:10.934 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:46.697504+0000", 2024-09-13T11:25:10.934 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 84.165218045000003, 2024-09-13T11:25:10.934 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.26836818099999998, 2024-09-13T11:25:10.934 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:25:10.934 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:25:10.934 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-13T11:25:10.934 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:25:10.934 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.934 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:25:10.934 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.697504+0000", 2024-09-13T11:25:10.934 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:10.934 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.934 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.935 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:25:10.935 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.697504+0000", 2024-09-13T11:25:10.935 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:10.935 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.935 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.935 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:25:10.935 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.697505+0000", 2024-09-13T11:25:10.935 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0529999999999999e-06 2024-09-13T11:25:10.935 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.935 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.935 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:25:10.935 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.697509+0000", 2024-09-13T11:25:10.935 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.9249999999999997e-06 2024-09-13T11:25:10.936 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.936 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.936 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:25:10.936 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.964086+0000", 2024-09-13T11:25:10.936 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.26657776100000002 2024-09-13T11:25:10.936 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.936 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.936 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:25:10.936 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.964110+0000", 2024-09-13T11:25:10.936 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.3189000000000001e-05 2024-09-13T11:25:10.936 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.936 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.936 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:25:10.937 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.964354+0000", 2024-09-13T11:25:10.937 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00024481399999999999 2024-09-13T11:25:10.937 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.937 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.937 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:25:10.937 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.965872+0000", 2024-09-13T11:25:10.937 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001517439 2024-09-13T11:25:10.937 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:10.937 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:25:10.937 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:10.937 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.937 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.937 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:83223 3.1c 3:3bc6018d:::benchmark_data_smithi177_31116_object83222:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106)", 2024-09-13T11:25:10.938 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:04.511163+0000", 2024-09-13T11:25:10.938 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 66.351558267000001, 2024-09-13T11:25:10.938 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.47558093200000001, 2024-09-13T11:25:10.938 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:25:10.938 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:25:10.938 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:10.938 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:25:10.938 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:25:10.938 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:10.938 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 83223 2024-09-13T11:25:10.938 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.938 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:25:10.939 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.939 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:25:10.939 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.511163+0000", 2024-09-13T11:25:10.939 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:10.939 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.939 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.939 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:25:10.939 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.511163+0000", 2024-09-13T11:25:10.939 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:10.939 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.939 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.939 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:25:10.940 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.511167+0000", 2024-09-13T11:25:10.940 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.8330000000000004e-06 2024-09-13T11:25:10.940 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.940 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.940 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:25:10.940 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.511219+0000", 2024-09-13T11:25:10.940 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.1771999999999998e-05 2024-09-13T11:25:10.940 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.940 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.940 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:25:10.940 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.511252+0000", 2024-09-13T11:25:10.940 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.3315999999999997e-05 2024-09-13T11:25:10.941 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.941 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.941 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:25:10.941 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.511268+0000", 2024-09-13T11:25:10.941 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5812999999999999e-05 2024-09-13T11:25:10.941 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.941 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.941 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:25:10.941 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.511418+0000", 2024-09-13T11:25:10.941 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000150023 2024-09-13T11:25:10.941 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.941 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.941 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:25:10.942 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.511670+0000", 2024-09-13T11:25:10.942 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00025185499999999999 2024-09-13T11:25:10.942 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.942 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.942 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 7", 2024-09-13T11:25:10.942 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.511905+0000", 2024-09-13T11:25:10.942 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00023493799999999999 2024-09-13T11:25:10.942 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.942 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.942 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:25:10.942 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.514960+0000", 2024-09-13T11:25:10.942 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.003054977 2024-09-13T11:25:10.943 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.943 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.943 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:25:10.943 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.986432+0000", 2024-09-13T11:25:10.943 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.47147209200000001 2024-09-13T11:25:10.943 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.943 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.943 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:25:10.943 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.986635+0000", 2024-09-13T11:25:10.943 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00020309899999999999 2024-09-13T11:25:10.943 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.943 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.943 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:25:10.944 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.986744+0000", 2024-09-13T11:25:10.944 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000109214 2024-09-13T11:25:10.944 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:10.944 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:25:10.944 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:10.944 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.944 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.944 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop_reply(client.4665.0:84803 3.1f e107/78)", 2024-09-13T11:25:10.944 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:05.912627+0000", 2024-09-13T11:25:10.944 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 64.950094359999994, 2024-09-13T11:25:10.944 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.33726566600000002, 2024-09-13T11:25:10.944 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:25:10.945 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:25:10.945 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-13T11:25:10.945 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:25:10.945 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.945 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:25:10.945 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:05.912627+0000", 2024-09-13T11:25:10.945 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:10.945 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.945 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.945 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:25:10.945 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:05.912627+0000", 2024-09-13T11:25:10.945 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:10.946 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.946 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.946 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:25:10.946 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:05.912629+0000", 2024-09-13T11:25:10.946 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.4440000000000001e-06 2024-09-13T11:25:10.946 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.946 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.946 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:25:10.946 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:05.912634+0000", 2024-09-13T11:25:10.946 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.6869999999999999e-06 2024-09-13T11:25:10.946 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.946 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.947 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:25:10.947 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:06.249722+0000", 2024-09-13T11:25:10.947 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.33708766099999998 2024-09-13T11:25:10.947 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.947 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.947 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:25:10.947 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:06.249736+0000", 2024-09-13T11:25:10.947 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3623e-05 2024-09-13T11:25:10.947 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.947 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.947 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:25:10.947 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:06.249878+0000", 2024-09-13T11:25:10.947 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00014264599999999999 2024-09-13T11:25:10.948 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.948 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.948 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:25:10.948 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:06.249893+0000", 2024-09-13T11:25:10.948 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.4605e-05 2024-09-13T11:25:10.948 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:10.948 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:25:10.948 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:10.948 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.948 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.948 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop_reply(client.4665.0:110260 3.f e119/75 ondisk, result = 0)", 2024-09-13T11:25:10.949 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:25.333188+0000", 2024-09-13T11:25:10.949 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 45.529533733000001, 2024-09-13T11:25:10.949 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.29195486199999998, 2024-09-13T11:25:10.949 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:25:10.949 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:25:10.949 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:25:10.949 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:25:10.949 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.949 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:25:10.949 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.333188+0000", 2024-09-13T11:25:10.949 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:10.949 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.949 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.950 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:25:10.950 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.333188+0000", 2024-09-13T11:25:10.950 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:10.950 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.950 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.950 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:25:10.950 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.333190+0000", 2024-09-13T11:25:10.950 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.2120000000000002e-06 2024-09-13T11:25:10.950 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.950 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.950 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:25:10.950 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.333197+0000", 2024-09-13T11:25:10.950 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.7469999999999999e-06 2024-09-13T11:25:10.951 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.951 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.951 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:25:10.951 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.624892+0000", 2024-09-13T11:25:10.951 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.291694913 2024-09-13T11:25:10.951 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.951 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.951 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:25:10.951 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.624907+0000", 2024-09-13T11:25:10.951 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5053e-05 2024-09-13T11:25:10.951 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.951 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.951 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:25:10.952 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.624999+0000", 2024-09-13T11:25:10.952 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.2603999999999996e-05 2024-09-13T11:25:10.952 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.952 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.952 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:25:10.952 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.625011+0000", 2024-09-13T11:25:10.952 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.1253999999999999e-05 2024-09-13T11:25:10.952 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.952 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.952 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:25:10.952 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.625143+0000", 2024-09-13T11:25:10.952 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00013207900000000001 2024-09-13T11:25:10.953 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:10.953 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:25:10.953 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:10.953 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.953 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.953 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:110265 3.1 e119/75 3:8236484d:::benchmark_data_smithi177_31116_object110264:head v 119'3935, mlcod=118'3933)", 2024-09-13T11:25:10.953 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:25.343348+0000", 2024-09-13T11:25:10.953 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 45.519374118000002, 2024-09-13T11:25:10.953 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.60013013500000001, 2024-09-13T11:25:10.953 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:25:10.953 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:25:10.953 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:10.954 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:25:10.954 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.954 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:25:10.954 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.343348+0000", 2024-09-13T11:25:10.954 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:10.954 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.954 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.954 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:25:10.954 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.343348+0000", 2024-09-13T11:25:10.954 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:10.954 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.954 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.954 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:25:10.955 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.343350+0000", 2024-09-13T11:25:10.955 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.576e-06 2024-09-13T11:25:10.955 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.955 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.955 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:25:10.955 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.343443+0000", 2024-09-13T11:25:10.955 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.2989e-05 2024-09-13T11:25:10.955 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.955 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.955 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:25:10.955 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.940863+0000", 2024-09-13T11:25:10.955 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.59742003300000002 2024-09-13T11:25:10.956 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.956 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.956 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:25:10.956 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.940875+0000", 2024-09-13T11:25:10.956 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.2147000000000001e-05 2024-09-13T11:25:10.956 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.956 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.956 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:25:10.956 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.940993+0000", 2024-09-13T11:25:10.956 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000117903 2024-09-13T11:25:10.956 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.956 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.956 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:25:10.957 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.941017+0000", 2024-09-13T11:25:10.957 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.3442999999999999e-05 2024-09-13T11:25:10.957 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.957 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.957 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:25:10.957 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.943347+0000", 2024-09-13T11:25:10.957 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0023299869999999999 2024-09-13T11:25:10.957 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.957 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.957 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:25:10.957 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.943478+0000", 2024-09-13T11:25:10.957 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00013105699999999999 2024-09-13T11:25:10.957 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:10.958 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:25:10.958 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:10.958 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.958 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.958 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop_reply(client.4665.0:129919 3.16 e134/88 ondisk, result = 0)", 2024-09-13T11:25:10.958 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:47.175742+0000", 2024-09-13T11:25:10.958 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 23.686980047999999, 2024-09-13T11:25:10.958 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.27593726299999999, 2024-09-13T11:25:10.958 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:25:10.958 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:25:10.958 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:25:10.958 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:25:10.958 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.959 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:25:10.959 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:47.175742+0000", 2024-09-13T11:25:10.959 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:10.959 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.959 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.959 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:25:10.959 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:47.175742+0000", 2024-09-13T11:25:10.959 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:10.959 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.959 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.959 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:25:10.959 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:47.175744+0000", 2024-09-13T11:25:10.960 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.0030000000000001e-06 2024-09-13T11:25:10.960 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.960 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.960 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:25:10.960 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:47.175751+0000", 2024-09-13T11:25:10.960 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.9779999999999999e-06 2024-09-13T11:25:10.960 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.960 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.960 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:25:10.960 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:47.451490+0000", 2024-09-13T11:25:10.960 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.27573892500000002 2024-09-13T11:25:10.960 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.960 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.961 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:25:10.961 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:47.451496+0000", 2024-09-13T11:25:10.961 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.0079999999999999e-06 2024-09-13T11:25:10.961 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.961 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.961 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:25:10.961 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:47.451560+0000", 2024-09-13T11:25:10.961 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.3955999999999995e-05 2024-09-13T11:25:10.961 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.961 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.961 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:25:10.961 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:47.451567+0000", 2024-09-13T11:25:10.961 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.1980000000000004e-06 2024-09-13T11:25:10.961 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:10.962 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:10.962 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:25:10.962 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:47.451679+0000", 2024-09-13T11:25:10.962 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00011219500000000001 2024-09-13T11:25:10.962 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:10.962 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:25:10.962 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:10.962 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:10.962 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:25:10.962 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-09-13T11:25:10.963 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_ops_in_flight 2024-09-13T11:25:10.965 INFO:tasks.ceph.osd.5.smithi092.stderr:2024-09-13T11:25:10.957+0000 7f6f0b705640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26346) UID: 0 2024-09-13T11:25:11.065 INFO:tasks.ceph.osd.4.smithi001.stderr:2024-09-13T11:25:11.063+0000 7ff8e0218640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26310) UID: 0 2024-09-13T11:25:11.079 INFO:teuthology.orchestra.run.smithi001.stderr:reweighted osd.0 to 1 (10000) 2024-09-13T11:25:11.085 INFO:teuthology.orchestra.run.smithi001.stderr:nodeep-scrub is unset 2024-09-13T11:25:11.106 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 1 1 2024-09-13T11:25:11.122 INFO:tasks.ceph.osd.10.smithi162.stderr:2024-09-13T11:25:11.120+0000 7fb593421640 -1 osd.10 151 handle_osd_map injecting map crc failure 2024-09-13T11:25:11.159 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-09-13T11:25:11.159 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [ 2024-09-13T11:25:11.159 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:11.160 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPush(3.14 152/122 [PushOp(3:2eeee97e:::benchmark_data_smithi177_31116_object98978:head, version: 113'4497, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:2eeee97e:::benchmark_data_smithi177_31116_object98978:head@113'4497, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:25:11.160 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:25:11.143142+0000", 2024-09-13T11:25:11.160 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 0.016152176000000001, 2024-09-13T11:25:11.160 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.016198856000000001, 2024-09-13T11:25:11.160 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:25:11.160 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:25:11.161 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:25:11.161 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:25:11.161 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:11.161 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:25:11.161 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:11.143142+0000", 2024-09-13T11:25:11.161 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:11.161 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:11.161 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:11.161 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:25:11.162 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:11.143142+0000", 2024-09-13T11:25:11.162 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:11.162 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:11.162 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:11.162 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:25:11.162 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:11.143143+0000", 2024-09-13T11:25:11.162 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.6530000000000001e-06 2024-09-13T11:25:11.162 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:11.162 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:11.162 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:25:11.162 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:11.143177+0000", 2024-09-13T11:25:11.162 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.4137000000000002e-05 2024-09-13T11:25:11.162 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:11.162 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:11.163 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:25:11.163 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:11.157387+0000", 2024-09-13T11:25:11.163 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.014209576999999999 2024-09-13T11:25:11.163 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:11.163 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:11.163 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:25:11.163 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:11.157407+0000", 2024-09-13T11:25:11.163 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.035e-05 2024-09-13T11:25:11.163 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:11.163 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:11.163 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:25:11.163 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:11.157509+0000", 2024-09-13T11:25:11.163 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000101847 2024-09-13T11:25:11.163 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:11.164 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:11.164 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:25:11.164 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:11.157524+0000", 2024-09-13T11:25:11.164 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5238999999999999e-05 2024-09-13T11:25:11.164 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:11.164 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:25:11.164 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:11.164 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:11.164 INFO:teuthology.orchestra.run.smithi177.stdout: ], 2024-09-13T11:25:11.164 INFO:teuthology.orchestra.run.smithi177.stdout: "num_ops": 1 2024-09-13T11:25:11.164 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-09-13T11:25:11.165 INFO:tasks.ceph.osd.5.smithi092.stderr:2024-09-13T11:25:11.161+0000 7f6f0b705640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26346) UID: 0 2024-09-13T11:25:11.181 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_blocked_ops 2024-09-13T11:25:11.267 INFO:tasks.ceph.osd.11.smithi177.stderr:2024-09-13T11:25:11.262+0000 7f24792c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26580) UID: 0 2024-09-13T11:25:11.338 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-09-13T11:25:11.338 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [], 2024-09-13T11:25:11.338 INFO:teuthology.orchestra.run.smithi177.stdout: "complaint_time": 30, 2024-09-13T11:25:11.338 INFO:teuthology.orchestra.run.smithi177.stdout: "num_blocked_ops": 0 2024-09-13T11:25:11.338 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-09-13T11:25:11.357 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_historic_ops 2024-09-13T11:25:11.368 INFO:tasks.ceph.osd.13.smithi092.stderr:2024-09-13T11:25:11.361+0000 7f3e5afea640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26350) UID: 0 2024-09-13T11:25:11.469 INFO:tasks.ceph.osd.11.smithi177.stderr:2024-09-13T11:25:11.466+0000 7f24792c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26580) UID: 0 2024-09-13T11:25:11.549 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-09-13T11:25:11.549 INFO:teuthology.orchestra.run.smithi177.stdout: "size": 20, 2024-09-13T11:25:11.549 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 600, 2024-09-13T11:25:11.549 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [ 2024-09-13T11:25:11.549 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:11.550 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:25383 3.0 e55/30 3:0a6bb869:::benchmark_data_smithi177_31116_object25382:head v 55'1620, mlcod=55'1617)", 2024-09-13T11:25:11.550 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:04.449956+0000", 2024-09-13T11:25:11.550 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 127.098069353, 2024-09-13T11:25:11.550 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.21143479600000001, 2024-09-13T11:25:11.550 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:25:11.550 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:25:11.550 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:11.550 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:25:11.550 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:11.550 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:25:11.550 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.449956+0000", 2024-09-13T11:25:11.550 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:11.550 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:11.551 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:11.551 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:25:11.551 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.449956+0000", 2024-09-13T11:25:11.551 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:11.551 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:11.551 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:11.551 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:25:11.551 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.449959+0000", 2024-09-13T11:25:11.551 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.393e-06 2024-09-13T11:25:11.551 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:11.551 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:11.551 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:25:11.552 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.450014+0000", 2024-09-13T11:25:11.552 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.4200000000000003e-05 2024-09-13T11:25:11.552 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:11.552 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:11.552 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:25:11.552 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.658637+0000", 2024-09-13T11:25:11.552 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.20862334799999999 2024-09-13T11:25:11.552 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:11.552 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:11.552 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:25:11.552 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.658650+0000", 2024-09-13T11:25:11.552 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3148e-05 2024-09-13T11:25:11.553 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:11.553 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:11.553 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:25:11.553 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.658867+0000", 2024-09-13T11:25:11.553 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00021673499999999999 2024-09-13T11:25:11.553 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:11.553 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:11.553 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:25:11.553 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.658931+0000", 2024-09-13T11:25:11.553 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.4045000000000001e-05 2024-09-13T11:25:11.553 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:11.553 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:11.553 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:25:11.554 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.661317+0000", 2024-09-13T11:25:11.554 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.002385823 2024-09-13T11:25:11.554 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:11.554 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:11.554 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:25:11.554 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.661391+0000", 2024-09-13T11:25:11.554 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.4103999999999994e-05 2024-09-13T11:25:11.554 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:11.554 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:25:11.554 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:11.554 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:11.554 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:11.555 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPush(3.16 85/70 [PushOp(3:6ac421eb:::benchmark_data_smithi177_31116_object43910:head, version: 66'2836, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:6ac421eb:::benchmark_data_smithi177_31116_object43910:head@66'2836, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:25:11.555 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:42.145291+0000", 2024-09-13T11:25:11.555 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 89.402734147000004, 2024-09-13T11:25:11.555 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.148514919, 2024-09-13T11:25:11.555 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:25:11.555 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:25:11.555 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:25:11.555 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:25:11.555 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:11.555 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:25:11.555 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.145291+0000", 2024-09-13T11:25:11.555 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:11.555 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:11.555 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:11.556 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:25:11.556 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.145291+0000", 2024-09-13T11:25:11.556 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:11.556 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:11.556 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:11.556 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:25:11.556 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.145293+0000", 2024-09-13T11:25:11.556 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.699e-06 2024-09-13T11:25:11.556 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:11.556 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:11.556 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:25:11.556 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.145417+0000", 2024-09-13T11:25:11.556 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00012415200000000001 2024-09-13T11:25:11.557 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:11.557 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:11.557 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:25:11.557 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.292964+0000", 2024-09-13T11:25:11.557 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.14754715400000001 2024-09-13T11:25:11.557 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:11.557 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:11.557 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:25:11.557 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.292981+0000", 2024-09-13T11:25:11.557 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.6790999999999999e-05 2024-09-13T11:25:11.557 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:11.557 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:11.557 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:25:11.558 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.293083+0000", 2024-09-13T11:25:11.558 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00010199799999999999 2024-09-13T11:25:11.558 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:11.558 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:11.558 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:25:11.558 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.293098+0000", 2024-09-13T11:25:11.558 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.4612e-05 2024-09-13T11:25:11.558 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:11.558 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:11.558 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:25:11.558 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.293806+0000", 2024-09-13T11:25:11.558 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00070851300000000002 2024-09-13T11:25:11.558 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:11.559 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:25:11.559 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:11.559 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:11.559 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:11.559 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:60390 3.16 e85/70 3:6a388186:::benchmark_data_smithi177_31116_object60389:head v 85'3357, mlcod=66'2835)", 2024-09-13T11:25:11.559 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:42.149102+0000", 2024-09-13T11:25:11.559 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 89.398923089999997, 2024-09-13T11:25:11.559 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.14821161799999999, 2024-09-13T11:25:11.559 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:25:11.559 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:25:11.559 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:11.559 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:25:11.559 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:11.560 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:25:11.560 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.149102+0000", 2024-09-13T11:25:11.560 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:11.560 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:11.560 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:11.560 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:25:11.560 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.149102+0000", 2024-09-13T11:25:11.560 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:11.560 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:11.560 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:11.560 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:25:11.560 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.149104+0000", 2024-09-13T11:25:11.561 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.708e-06 2024-09-13T11:25:11.561 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:11.561 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:11.561 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:25:11.561 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.149160+0000", 2024-09-13T11:25:11.561 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.5618000000000002e-05 2024-09-13T11:25:11.561 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:11.561 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:11.561 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:25:11.561 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.296071+0000", 2024-09-13T11:25:11.561 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.14691148600000001 2024-09-13T11:25:11.561 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:11.561 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:11.561 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:25:11.562 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.296076+0000", 2024-09-13T11:25:11.562 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.5580000000000002e-06 2024-09-13T11:25:11.562 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:11.562 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:11.562 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:25:11.562 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.296127+0000", 2024-09-13T11:25:11.562 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.1405e-05 2024-09-13T11:25:11.562 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:11.562 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:11.562 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:25:11.562 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.296150+0000", 2024-09-13T11:25:11.562 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.3113e-05 2024-09-13T11:25:11.562 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:11.562 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:11.563 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:25:11.563 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.297274+0000", 2024-09-13T11:25:11.563 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0011239900000000001 2024-09-13T11:25:11.563 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:11.563 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:11.563 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:25:11.563 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.297314+0000", 2024-09-13T11:25:11.563 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.9740000000000002e-05 2024-09-13T11:25:11.563 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:11.563 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:25:11.564 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:11.564 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:11.564 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:11.564 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPush(3.16 85/70 [PushOp(3:69aac9fd:::benchmark_data_smithi177_31116_object43914:head, version: 66'2837, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:69aac9fd:::benchmark_data_smithi177_31116_object43914:head@66'2837, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:25:11.564 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:42.295995+0000", 2024-09-13T11:25:11.564 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 89.252030032999997, 2024-09-13T11:25:11.564 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.215089431, 2024-09-13T11:25:11.564 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:25:11.564 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:25:11.564 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:25:11.564 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:25:11.564 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:11.565 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:25:11.565 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.295995+0000", 2024-09-13T11:25:11.565 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:11.565 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:11.565 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:11.565 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:25:11.565 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.295995+0000", 2024-09-13T11:25:11.565 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:11.565 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:11.565 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:11.565 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:25:11.565 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.295997+0000", 2024-09-13T11:25:11.565 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3170000000000001e-06 2024-09-13T11:25:11.566 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:11.566 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:11.566 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:25:11.566 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.296058+0000", 2024-09-13T11:25:11.566 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.1690000000000001e-05 2024-09-13T11:25:11.566 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:11.566 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:11.566 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:25:11.566 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.509606+0000", 2024-09-13T11:25:11.566 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.21354748300000001 2024-09-13T11:25:11.566 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:11.566 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:11.566 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:25:11.567 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.509638+0000", 2024-09-13T11:25:11.567 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.1644999999999999e-05 2024-09-13T11:25:11.567 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:11.567 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:11.567 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:25:11.567 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.509814+0000", 2024-09-13T11:25:11.567 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00017679499999999999 2024-09-13T11:25:11.567 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:11.567 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:11.567 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:25:11.567 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.509840+0000", 2024-09-13T11:25:11.567 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.5871e-05 2024-09-13T11:25:11.567 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:11.568 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:11.568 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:25:11.568 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.511085+0000", 2024-09-13T11:25:11.568 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00124463 2024-09-13T11:25:11.568 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:11.568 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:25:11.568 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:11.568 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:11.568 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:11.568 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPush(3.1d 91/78 [PushOp(3:be6a0ae4:::benchmark_data_smithi177_31116_object25671:head, version: 58'1569, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:be6a0ae4:::benchmark_data_smithi177_31116_object25671:head@58'1569, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:25:11.568 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:48.481971+0000", 2024-09-13T11:25:11.568 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 83.066054788000002, 2024-09-13T11:25:11.568 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.47945454199999998, 2024-09-13T11:25:11.569 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:25:11.569 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:25:11.569 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:25:11.569 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:25:11.569 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:11.569 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:25:11.569 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.481971+0000", 2024-09-13T11:25:11.569 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:11.569 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:11.569 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:11.569 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:25:11.569 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.481971+0000", 2024-09-13T11:25:11.569 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:11.570 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:11.570 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:11.570 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:25:11.570 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.481972+0000", 2024-09-13T11:25:11.570 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5179999999999999e-06 2024-09-13T11:25:11.570 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:11.570 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:11.570 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:25:11.570 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.482000+0000", 2024-09-13T11:25:11.570 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.8249e-05 2024-09-13T11:25:11.570 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:11.570 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:11.570 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:25:11.570 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.959926+0000", 2024-09-13T11:25:11.571 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.47792585100000001 2024-09-13T11:25:11.571 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:11.571 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:11.571 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:25:11.571 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.959944+0000", 2024-09-13T11:25:11.571 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.7620000000000001e-05 2024-09-13T11:25:11.571 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:11.571 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:11.571 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:25:11.571 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.960059+0000", 2024-09-13T11:25:11.571 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000115188 2024-09-13T11:25:11.571 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:11.571 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:11.571 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:25:11.572 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.960074+0000", 2024-09-13T11:25:11.572 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.4738e-05 2024-09-13T11:25:11.572 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:11.572 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:11.572 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:25:11.572 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.961425+0000", 2024-09-13T11:25:11.572 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0013513780000000001 2024-09-13T11:25:11.572 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:11.572 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:25:11.572 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:11.572 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:11.572 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:11.572 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:71884 3.16 e96/88 3:6ca846b2:::benchmark_data_smithi177_31116_object71883:head v 96'3755, mlcod=96'3753)", 2024-09-13T11:25:11.573 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:54.108299+0000", 2024-09-13T11:25:11.573 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 77.439726614999998, 2024-09-13T11:25:11.573 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.20685140099999999, 2024-09-13T11:25:11.573 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:25:11.573 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:25:11.573 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:11.573 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:25:11.573 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:11.573 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:25:11.573 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.108299+0000", 2024-09-13T11:25:11.573 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:11.573 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:11.573 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:11.573 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:25:11.574 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.108299+0000", 2024-09-13T11:25:11.574 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:11.574 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:11.574 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:11.574 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:25:11.574 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.108300+0000", 2024-09-13T11:25:11.574 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.173e-06 2024-09-13T11:25:11.574 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:11.574 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:11.574 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:25:11.574 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.108323+0000", 2024-09-13T11:25:11.574 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.2591000000000001e-05 2024-09-13T11:25:11.574 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:11.575 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:11.575 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:25:11.575 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.312377+0000", 2024-09-13T11:25:11.575 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.20405456599999999 2024-09-13T11:25:11.575 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:11.575 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:11.575 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:25:11.575 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.312382+0000", 2024-09-13T11:25:11.575 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.1429999999999997e-06 2024-09-13T11:25:11.575 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:11.575 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:11.575 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:25:11.575 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.312480+0000", 2024-09-13T11:25:11.576 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.7742e-05 2024-09-13T11:25:11.576 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:11.576 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:11.576 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:25:11.576 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.312521+0000", 2024-09-13T11:25:11.576 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.1369000000000003e-05 2024-09-13T11:25:11.576 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:11.576 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:11.576 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:25:11.576 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.315105+0000", 2024-09-13T11:25:11.576 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0025835060000000002 2024-09-13T11:25:11.576 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:11.577 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:11.577 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:25:11.577 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.315150+0000", 2024-09-13T11:25:11.577 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.5311e-05 2024-09-13T11:25:11.577 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:11.577 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:25:11.577 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:11.577 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:11.577 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:11.577 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:71927 3.0 e97/75 3:0197244a:::benchmark_data_smithi177_31116_object71926:head v 97'3351, mlcod=96'3348)", 2024-09-13T11:25:11.577 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:54.200161+0000", 2024-09-13T11:25:11.577 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 77.347864387000001, 2024-09-13T11:25:11.577 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.26691738999999998, 2024-09-13T11:25:11.578 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:25:11.578 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:25:11.578 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:11.578 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:25:11.578 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:11.578 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:25:11.578 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.200161+0000", 2024-09-13T11:25:11.578 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:11.578 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:11.578 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:11.578 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:25:11.578 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.200161+0000", 2024-09-13T11:25:11.578 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:11.579 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:11.579 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:11.579 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:25:11.579 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.200162+0000", 2024-09-13T11:25:11.579 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.78e-07 2024-09-13T11:25:11.579 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:11.579 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:11.579 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:25:11.579 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.200200+0000", 2024-09-13T11:25:11.579 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.8121000000000003e-05 2024-09-13T11:25:11.579 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:11.579 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:11.579 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:25:11.579 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.465397+0000", 2024-09-13T11:25:11.580 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.26519681099999998 2024-09-13T11:25:11.580 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:11.580 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:11.580 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:25:11.580 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.465404+0000", 2024-09-13T11:25:11.580 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.7399999999999998e-06 2024-09-13T11:25:11.580 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:11.580 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:11.580 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:25:11.580 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.465456+0000", 2024-09-13T11:25:11.580 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.2626000000000001e-05 2024-09-13T11:25:11.580 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:11.581 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:11.581 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:25:11.581 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.465482+0000", 2024-09-13T11:25:11.581 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.6086000000000001e-05 2024-09-13T11:25:11.581 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:11.581 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:11.581 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:25:11.581 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.467035+0000", 2024-09-13T11:25:11.581 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0015523729999999999 2024-09-13T11:25:11.581 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:11.581 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:11.581 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:25:11.581 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.467078+0000", 2024-09-13T11:25:11.582 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.3754999999999997e-05 2024-09-13T11:25:11.582 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:11.582 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:25:11.582 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:11.582 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:11.582 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:11.582 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:72250 3.0 e97/75 3:010c7171:::benchmark_data_smithi177_31116_object72249:head v 97'3353, mlcod=96'3350)", 2024-09-13T11:25:11.582 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:54.516381+0000", 2024-09-13T11:25:11.582 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 77.031644873999994, 2024-09-13T11:25:11.582 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.13663347300000001, 2024-09-13T11:25:11.582 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:25:11.582 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:25:11.583 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:11.583 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:25:11.583 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:11.583 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:25:11.583 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.516381+0000", 2024-09-13T11:25:11.583 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:11.583 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:11.583 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:11.583 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:25:11.583 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.516381+0000", 2024-09-13T11:25:11.583 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:11.583 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:11.583 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:11.583 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:25:11.584 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.516382+0000", 2024-09-13T11:25:11.584 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3239999999999999e-06 2024-09-13T11:25:11.584 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:11.584 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:11.584 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:25:11.584 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.516437+0000", 2024-09-13T11:25:11.585 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.5408999999999998e-05 2024-09-13T11:25:11.585 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:11.585 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:11.585 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:25:11.585 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.651731+0000", 2024-09-13T11:25:11.585 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.135293583 2024-09-13T11:25:11.585 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:11.585 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:11.585 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:25:11.585 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.651736+0000", 2024-09-13T11:25:11.585 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.3410000000000002e-06 2024-09-13T11:25:11.585 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:11.585 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:11.586 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:25:11.586 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.651791+0000", 2024-09-13T11:25:11.586 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.4758e-05 2024-09-13T11:25:11.586 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:11.586 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:11.586 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:25:11.586 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.651814+0000", 2024-09-13T11:25:11.586 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.3283999999999999e-05 2024-09-13T11:25:11.586 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:11.586 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:11.586 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:25:11.586 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.652975+0000", 2024-09-13T11:25:11.587 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0011604370000000001 2024-09-13T11:25:11.587 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:11.587 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:11.587 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:25:11.587 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.653014+0000", 2024-09-13T11:25:11.587 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.9337e-05 2024-09-13T11:25:11.587 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:11.587 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:25:11.587 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:11.587 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:11.587 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:11.587 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:72993 3.0 e97/75 3:02747b8f:::benchmark_data_smithi177_31116_object72992:head v 97'3366, mlcod=97'3363)", 2024-09-13T11:25:11.587 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:55.116009+0000", 2024-09-13T11:25:11.588 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 76.432016267999998, 2024-09-13T11:25:11.588 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.13962660099999999, 2024-09-13T11:25:11.588 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:25:11.588 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:25:11.588 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:11.588 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:25:11.588 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:11.588 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:25:11.588 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:55.116009+0000", 2024-09-13T11:25:11.588 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:11.588 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:11.588 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:11.588 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:25:11.589 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:55.116009+0000", 2024-09-13T11:25:11.589 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:11.589 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:11.589 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:11.589 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:25:11.589 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:55.116010+0000", 2024-09-13T11:25:11.589 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.291e-06 2024-09-13T11:25:11.589 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:11.589 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:11.589 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:25:11.589 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:55.116062+0000", 2024-09-13T11:25:11.589 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.1949e-05 2024-09-13T11:25:11.589 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:11.590 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:11.590 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:25:11.590 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:55.254100+0000", 2024-09-13T11:25:11.590 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.13803723000000001 2024-09-13T11:25:11.590 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:11.590 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:11.590 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:25:11.590 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:55.254106+0000", 2024-09-13T11:25:11.590 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.5939999999999999e-06 2024-09-13T11:25:11.590 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:11.590 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:11.590 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:25:11.590 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:55.254196+0000", 2024-09-13T11:25:11.590 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.9567999999999994e-05 2024-09-13T11:25:11.591 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:11.591 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:11.591 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:25:11.591 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:55.254219+0000", 2024-09-13T11:25:11.591 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.3266000000000001e-05 2024-09-13T11:25:11.591 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:11.591 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:11.591 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:25:11.591 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:55.255596+0000", 2024-09-13T11:25:11.591 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0013767409999999999 2024-09-13T11:25:11.592 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:11.592 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:11.592 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:25:11.592 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:55.255636+0000", 2024-09-13T11:25:11.592 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.9962000000000001e-05 2024-09-13T11:25:11.592 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:11.592 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:25:11.592 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:11.592 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:11.592 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:11.592 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:79634 3.5 3:a6c32e6f:::benchmark_data_smithi177_31116_object79633:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101)", 2024-09-13T11:25:11.593 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:00.168616+0000", 2024-09-13T11:25:11.593 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 71.379408999000006, 2024-09-13T11:25:11.593 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.169445552, 2024-09-13T11:25:11.593 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:25:11.593 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:25:11.593 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:11.593 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:25:11.593 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:25:11.593 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:11.593 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 79634 2024-09-13T11:25:11.593 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:11.594 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:25:11.594 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:11.594 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:25:11.594 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.168616+0000", 2024-09-13T11:25:11.594 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:11.594 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:11.594 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:11.594 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:25:11.594 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.168616+0000", 2024-09-13T11:25:11.594 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:11.594 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:11.594 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:11.594 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:25:11.594 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.168619+0000", 2024-09-13T11:25:11.595 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.565e-06 2024-09-13T11:25:11.595 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:11.595 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:11.595 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:25:11.595 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.168639+0000", 2024-09-13T11:25:11.595 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.0384999999999999e-05 2024-09-13T11:25:11.595 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:11.595 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:11.595 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:25:11.595 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.168652+0000", 2024-09-13T11:25:11.595 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.2779000000000001e-05 2024-09-13T11:25:11.595 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:11.595 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:11.595 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:25:11.596 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.168658+0000", 2024-09-13T11:25:11.596 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.2269999999999998e-06 2024-09-13T11:25:11.596 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:11.596 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:11.596 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:25:11.596 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.168722+0000", 2024-09-13T11:25:11.596 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.3888999999999998e-05 2024-09-13T11:25:11.596 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:11.596 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:11.596 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:25:11.596 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.168863+0000", 2024-09-13T11:25:11.596 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00014050799999999999 2024-09-13T11:25:11.596 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:11.596 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:11.597 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 13", 2024-09-13T11:25:11.597 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.169010+0000", 2024-09-13T11:25:11.597 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00014746200000000001 2024-09-13T11:25:11.597 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:11.597 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:11.597 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:25:11.597 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.170891+0000", 2024-09-13T11:25:11.597 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001880945 2024-09-13T11:25:11.597 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:11.597 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:11.597 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:25:11.597 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.337961+0000", 2024-09-13T11:25:11.597 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.16706952 2024-09-13T11:25:11.597 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:11.598 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:11.598 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:25:11.598 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.338019+0000", 2024-09-13T11:25:11.598 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.7958000000000003e-05 2024-09-13T11:25:11.598 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:11.598 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:11.598 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:25:11.598 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.338062+0000", 2024-09-13T11:25:11.598 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.3313999999999999e-05 2024-09-13T11:25:11.598 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:11.598 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:25:11.598 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:11.598 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:11.598 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:11.599 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPush(3.1b 102/78 [PushOp(3:df786fc7:::benchmark_data_smithi177_31116_object48840:head, version: 71'3079, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:df786fc7:::benchmark_data_smithi177_31116_object48840:head@71'3079, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:25:11.599 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:00.290536+0000", 2024-09-13T11:25:11.599 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 71.257489625000005, 2024-09-13T11:25:11.599 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.20017585399999999, 2024-09-13T11:25:11.599 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:25:11.599 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:25:11.599 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:25:11.599 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:25:11.599 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:11.599 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:25:11.599 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.290536+0000", 2024-09-13T11:25:11.599 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:11.599 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:11.600 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:11.600 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:25:11.600 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.290536+0000", 2024-09-13T11:25:11.600 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:11.600 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:11.600 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:11.600 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:25:11.600 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.290537+0000", 2024-09-13T11:25:11.600 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.327e-06 2024-09-13T11:25:11.600 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:11.600 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:11.600 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:25:11.600 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.290581+0000", 2024-09-13T11:25:11.600 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.3893999999999997e-05 2024-09-13T11:25:11.601 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:11.601 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:11.601 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:25:11.601 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.488588+0000", 2024-09-13T11:25:11.601 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.198006708 2024-09-13T11:25:11.601 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:11.601 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:11.601 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:25:11.601 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.488607+0000", 2024-09-13T11:25:11.601 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.9162999999999999e-05 2024-09-13T11:25:11.601 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:11.601 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:11.601 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:25:11.601 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.488747+0000", 2024-09-13T11:25:11.602 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00014040799999999999 2024-09-13T11:25:11.602 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:11.602 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:11.602 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:25:11.602 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.488770+0000", 2024-09-13T11:25:11.602 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.279e-05 2024-09-13T11:25:11.602 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:11.602 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:11.602 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:25:11.602 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.490712+0000", 2024-09-13T11:25:11.602 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001941564 2024-09-13T11:25:11.603 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:11.603 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:25:11.603 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:11.603 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:11.603 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:11.603 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPush(3.1b 102/78 [PushOp(3:d82bf6ec:::benchmark_data_smithi177_31116_object48847:head, version: 71'3080, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:d82bf6ec:::benchmark_data_smithi177_31116_object48847:head@71'3080, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:25:11.603 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:00.293979+0000", 2024-09-13T11:25:11.603 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 71.254046098000003, 2024-09-13T11:25:11.603 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.20314832099999999, 2024-09-13T11:25:11.603 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:25:11.603 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:25:11.604 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:25:11.604 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:25:11.604 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:11.604 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:25:11.604 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.293979+0000", 2024-09-13T11:25:11.604 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:11.604 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:11.604 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:11.604 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:25:11.604 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.293979+0000", 2024-09-13T11:25:11.604 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:11.604 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:11.605 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:11.605 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:25:11.605 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.293981+0000", 2024-09-13T11:25:11.605 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.4449999999999999e-06 2024-09-13T11:25:11.605 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:11.605 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:11.605 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:25:11.605 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.294085+0000", 2024-09-13T11:25:11.605 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000103848 2024-09-13T11:25:11.605 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:11.605 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:11.605 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:25:11.605 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.495571+0000", 2024-09-13T11:25:11.606 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.20148640800000001 2024-09-13T11:25:11.606 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:11.606 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:11.606 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:25:11.606 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.495609+0000", 2024-09-13T11:25:11.606 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.7904000000000002e-05 2024-09-13T11:25:11.606 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:11.606 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:11.606 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:25:11.606 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.495729+0000", 2024-09-13T11:25:11.606 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000120556 2024-09-13T11:25:11.606 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:11.606 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:11.606 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:25:11.606 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.495751+0000", 2024-09-13T11:25:11.607 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.1511000000000001e-05 2024-09-13T11:25:11.607 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:11.607 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:11.607 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:25:11.607 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.497128+0000", 2024-09-13T11:25:11.607 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001376649 2024-09-13T11:25:11.607 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:11.607 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:25:11.607 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:11.607 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:11.607 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:11.607 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:79801 3.5 3:a6efdf84:::benchmark_data_smithi177_31116_object79800:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102)", 2024-09-13T11:25:11.608 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:00.403201+0000", 2024-09-13T11:25:11.608 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 71.144824510999996, 2024-09-13T11:25:11.608 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.31868626300000003, 2024-09-13T11:25:11.608 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:25:11.608 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:25:11.608 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:11.608 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:25:11.608 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:25:11.608 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:11.608 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 79801 2024-09-13T11:25:11.608 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:11.608 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:25:11.608 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:11.609 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:25:11.609 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.403201+0000", 2024-09-13T11:25:11.609 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:11.609 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:11.609 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:11.609 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:25:11.609 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.403201+0000", 2024-09-13T11:25:11.609 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:11.609 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:11.609 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:11.609 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:25:11.609 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.403204+0000", 2024-09-13T11:25:11.609 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.0819999999999999e-06 2024-09-13T11:25:11.609 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:11.610 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:11.610 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:25:11.610 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.403260+0000", 2024-09-13T11:25:11.610 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.5624999999999997e-05 2024-09-13T11:25:11.610 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:11.610 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:11.610 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:25:11.610 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.403280+0000", 2024-09-13T11:25:11.610 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.0060000000000001e-05 2024-09-13T11:25:11.610 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:11.610 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:11.610 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:25:11.610 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.403291+0000", 2024-09-13T11:25:11.611 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.1335000000000001e-05 2024-09-13T11:25:11.611 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:11.611 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:11.611 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:25:11.611 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.403344+0000", 2024-09-13T11:25:11.611 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.3217000000000002e-05 2024-09-13T11:25:11.611 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:11.611 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:11.611 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:25:11.611 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.403456+0000", 2024-09-13T11:25:11.611 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000111706 2024-09-13T11:25:11.611 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:11.611 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:11.611 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 13", 2024-09-13T11:25:11.612 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.403547+0000", 2024-09-13T11:25:11.612 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.1556000000000002e-05 2024-09-13T11:25:11.612 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:11.612 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:11.612 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:25:11.612 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.407436+0000", 2024-09-13T11:25:11.612 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.003888856 2024-09-13T11:25:11.612 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:11.612 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:11.612 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:25:11.612 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.721722+0000", 2024-09-13T11:25:11.612 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.31428562700000001 2024-09-13T11:25:11.612 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:11.612 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:11.613 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:25:11.613 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.721814+0000", 2024-09-13T11:25:11.613 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.1564000000000006e-05 2024-09-13T11:25:11.613 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:11.613 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:11.613 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:25:11.613 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.721887+0000", 2024-09-13T11:25:11.613 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.3634999999999995e-05 2024-09-13T11:25:11.613 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:11.613 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:25:11.613 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:11.613 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:11.613 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:11.614 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPush(3.1b 102/78 [PushOp(3:dcbefe42:::benchmark_data_smithi177_31116_object49001:head, version: 71'3087, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:dcbefe42:::benchmark_data_smithi177_31116_object49001:head@71'3087, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:25:11.614 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:00.511027+0000", 2024-09-13T11:25:11.614 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 71.036998574999998, 2024-09-13T11:25:11.614 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.13427292699999999, 2024-09-13T11:25:11.614 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:25:11.614 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:25:11.614 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:25:11.614 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:25:11.614 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:11.614 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:25:11.614 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.511027+0000", 2024-09-13T11:25:11.614 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:11.614 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:11.614 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:11.615 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:25:11.615 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.511027+0000", 2024-09-13T11:25:11.615 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:11.615 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:11.615 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:11.615 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:25:11.615 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.511028+0000", 2024-09-13T11:25:11.615 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.4869999999999999e-06 2024-09-13T11:25:11.615 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:11.615 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:11.616 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:25:11.616 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.511095+0000", 2024-09-13T11:25:11.616 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.6382000000000002e-05 2024-09-13T11:25:11.616 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:11.616 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:11.616 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:25:11.616 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.643319+0000", 2024-09-13T11:25:11.616 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.13222455699999999 2024-09-13T11:25:11.616 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:11.616 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:11.616 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:25:11.617 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.643349+0000", 2024-09-13T11:25:11.617 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.9356000000000002e-05 2024-09-13T11:25:11.617 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:11.617 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:11.617 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:25:11.617 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.643518+0000", 2024-09-13T11:25:11.617 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00016963500000000001 2024-09-13T11:25:11.617 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:11.617 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:11.617 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:25:11.617 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.643541+0000", 2024-09-13T11:25:11.617 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.2367000000000001e-05 2024-09-13T11:25:11.617 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:11.617 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:11.618 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:25:11.618 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.645300+0000", 2024-09-13T11:25:11.618 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0017591429999999999 2024-09-13T11:25:11.618 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:11.618 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:25:11.618 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:11.618 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:11.618 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:11.618 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:79813 3.16 e102/88 3:6c2df1ae:::benchmark_data_smithi177_31116_object79812:head v 102'4016, mlcod=102'4014)", 2024-09-13T11:25:11.618 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:00.516959+0000", 2024-09-13T11:25:11.618 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 71.031066550000006, 2024-09-13T11:25:11.618 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.15075599100000001, 2024-09-13T11:25:11.618 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:25:11.619 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:25:11.619 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:11.619 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:25:11.619 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:11.619 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:25:11.619 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.516959+0000", 2024-09-13T11:25:11.619 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:11.619 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:11.619 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:11.619 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:25:11.619 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.516959+0000", 2024-09-13T11:25:11.619 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:11.620 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:11.620 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:11.620 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:25:11.620 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.516961+0000", 2024-09-13T11:25:11.620 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.0279999999999999e-06 2024-09-13T11:25:11.620 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:11.620 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:11.620 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:25:11.620 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.517002+0000", 2024-09-13T11:25:11.620 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.0754999999999999e-05 2024-09-13T11:25:11.620 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:11.620 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:11.620 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:25:11.621 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.664491+0000", 2024-09-13T11:25:11.621 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.14748925099999999 2024-09-13T11:25:11.621 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:11.621 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:11.621 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:25:11.621 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.664504+0000", 2024-09-13T11:25:11.621 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.2747e-05 2024-09-13T11:25:11.621 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:11.621 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:11.621 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:25:11.621 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.664642+0000", 2024-09-13T11:25:11.621 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00013850900000000001 2024-09-13T11:25:11.621 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:11.622 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:11.622 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:25:11.622 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.664678+0000", 2024-09-13T11:25:11.622 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.5403e-05 2024-09-13T11:25:11.622 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:11.622 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:11.622 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:25:11.622 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.667646+0000", 2024-09-13T11:25:11.622 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.002968133 2024-09-13T11:25:11.622 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:11.622 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:11.622 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:25:11.622 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.667715+0000", 2024-09-13T11:25:11.623 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.9165000000000006e-05 2024-09-13T11:25:11.623 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:11.623 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:25:11.623 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:11.623 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:11.623 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:11.623 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:83199 3.1d e106/102 3:be2c6fae:::benchmark_data_smithi177_31116_object83198:head v 106'4244, mlcod=106'4242)", 2024-09-13T11:25:11.623 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:04.298299+0000", 2024-09-13T11:25:11.623 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 67.249726383999999, 2024-09-13T11:25:11.623 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.25001107099999997, 2024-09-13T11:25:11.623 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:25:11.623 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:25:11.623 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:11.623 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:25:11.623 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:11.624 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:25:11.624 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.298299+0000", 2024-09-13T11:25:11.624 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:11.624 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:11.624 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:11.624 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:25:11.624 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.298299+0000", 2024-09-13T11:25:11.624 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:11.624 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:11.624 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:11.624 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:25:11.624 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.298301+0000", 2024-09-13T11:25:11.624 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5859999999999999e-06 2024-09-13T11:25:11.625 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:11.625 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:11.625 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:25:11.625 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.298368+0000", 2024-09-13T11:25:11.625 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.6928999999999996e-05 2024-09-13T11:25:11.625 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:11.625 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:11.625 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:25:11.625 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.545180+0000", 2024-09-13T11:25:11.625 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.24681209700000001 2024-09-13T11:25:11.625 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:11.625 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:11.625 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:25:11.626 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.545193+0000", 2024-09-13T11:25:11.626 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3427e-05 2024-09-13T11:25:11.626 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:11.626 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:11.626 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:25:11.626 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.545327+0000", 2024-09-13T11:25:11.626 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00013359400000000001 2024-09-13T11:25:11.626 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:11.626 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:11.626 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:25:11.626 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.545385+0000", 2024-09-13T11:25:11.627 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.8492000000000002e-05 2024-09-13T11:25:11.627 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:11.627 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:11.627 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:25:11.627 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.548225+0000", 2024-09-13T11:25:11.627 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.002839472 2024-09-13T11:25:11.627 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:11.627 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:11.627 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:25:11.627 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.548310+0000", 2024-09-13T11:25:11.627 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.5474000000000002e-05 2024-09-13T11:25:11.627 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:11.627 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:25:11.627 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:11.628 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:11.628 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:11.628 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:84621 3.1d e107/102 3:becd72aa:::benchmark_data_smithi177_31116_object84620:head v 107'4296, mlcod=107'4293)", 2024-09-13T11:25:11.628 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:05.742238+0000", 2024-09-13T11:25:11.628 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 65.805787284000004, 2024-09-13T11:25:11.628 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.17888174300000001, 2024-09-13T11:25:11.628 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:25:11.628 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:25:11.628 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:11.628 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:25:11.628 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:11.628 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:25:11.628 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:05.742238+0000", 2024-09-13T11:25:11.629 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:11.629 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:11.629 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:11.629 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:25:11.629 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:05.742238+0000", 2024-09-13T11:25:11.629 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:11.629 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:11.629 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:11.629 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:25:11.629 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:05.742240+0000", 2024-09-13T11:25:11.629 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.3159999999999999e-06 2024-09-13T11:25:11.629 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:11.630 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:11.630 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:25:11.630 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:05.742315+0000", 2024-09-13T11:25:11.630 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.4122000000000006e-05 2024-09-13T11:25:11.630 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:11.630 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:11.630 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:25:11.630 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:05.917757+0000", 2024-09-13T11:25:11.630 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.175442031 2024-09-13T11:25:11.630 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:11.630 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:11.630 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:25:11.631 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:05.917764+0000", 2024-09-13T11:25:11.631 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.661e-06 2024-09-13T11:25:11.631 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:11.631 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:11.631 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:25:11.631 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:05.917846+0000", 2024-09-13T11:25:11.631 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.2006000000000001e-05 2024-09-13T11:25:11.631 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:11.631 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:11.631 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:25:11.631 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:05.917879+0000", 2024-09-13T11:25:11.631 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.2796000000000002e-05 2024-09-13T11:25:11.631 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:11.632 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:11.632 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:25:11.632 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:05.921070+0000", 2024-09-13T11:25:11.632 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0031908489999999999 2024-09-13T11:25:11.632 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:11.632 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:11.632 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:25:11.632 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:05.921120+0000", 2024-09-13T11:25:11.632 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.9962e-05 2024-09-13T11:25:11.632 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:11.632 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:25:11.632 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:11.633 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:11.633 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:11.633 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:107910 3.16 e116/88 3:6cf57aa3:::benchmark_data_smithi177_31116_object107909:head v 116'4911, mlcod=116'4908)", 2024-09-13T11:25:11.633 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:22.516225+0000", 2024-09-13T11:25:11.633 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 49.031800740999998, 2024-09-13T11:25:11.633 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.18319666600000001, 2024-09-13T11:25:11.633 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:25:11.633 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:25:11.633 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:11.633 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:25:11.633 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:11.633 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:25:11.633 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:22.516225+0000", 2024-09-13T11:25:11.634 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:11.634 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:11.634 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:11.634 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:25:11.634 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:22.516225+0000", 2024-09-13T11:25:11.634 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:11.634 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:11.634 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:11.634 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:25:11.634 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:22.516226+0000", 2024-09-13T11:25:11.634 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.716e-06 2024-09-13T11:25:11.635 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:11.635 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:11.635 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:25:11.635 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:22.516250+0000", 2024-09-13T11:25:11.635 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.3975e-05 2024-09-13T11:25:11.635 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:11.635 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:11.635 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:25:11.635 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:22.697686+0000", 2024-09-13T11:25:11.635 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.18143539 2024-09-13T11:25:11.635 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:11.635 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:11.635 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:25:11.636 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:22.697692+0000", 2024-09-13T11:25:11.636 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.0769999999999996e-06 2024-09-13T11:25:11.636 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:11.636 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:11.636 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:25:11.636 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:22.697744+0000", 2024-09-13T11:25:11.636 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.2525000000000003e-05 2024-09-13T11:25:11.636 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:11.636 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:11.636 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:25:11.636 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:22.697757+0000", 2024-09-13T11:25:11.636 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.2819e-05 2024-09-13T11:25:11.636 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:11.637 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:11.637 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:25:11.637 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:22.699388+0000", 2024-09-13T11:25:11.637 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0016303999999999999 2024-09-13T11:25:11.637 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:11.637 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:11.637 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:25:11.637 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:22.699421+0000", 2024-09-13T11:25:11.637 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.3763999999999997e-05 2024-09-13T11:25:11.637 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:11.637 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:25:11.637 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:11.638 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:11.638 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:11.638 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:127682 3.16 e131/88 3:6e6da02e:::benchmark_data_smithi177_31116_object127681:head v 131'5491, mlcod=131'5488)", 2024-09-13T11:25:11.638 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:44.556692+0000", 2024-09-13T11:25:11.638 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 26.991333812000001, 2024-09-13T11:25:11.638 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.18048624299999999, 2024-09-13T11:25:11.638 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:25:11.638 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:25:11.638 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:11.639 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:25:11.639 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:11.639 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:25:11.639 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:44.556692+0000", 2024-09-13T11:25:11.639 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:11.639 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:11.639 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:11.639 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:25:11.639 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:44.556692+0000", 2024-09-13T11:25:11.639 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:11.639 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:11.639 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:11.640 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:25:11.640 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:44.556693+0000", 2024-09-13T11:25:11.640 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.8109999999999999e-06 2024-09-13T11:25:11.640 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:11.640 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:11.640 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:25:11.640 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:44.556715+0000", 2024-09-13T11:25:11.640 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.1328999999999999e-05 2024-09-13T11:25:11.640 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:11.640 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:11.640 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:25:11.640 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:44.734908+0000", 2024-09-13T11:25:11.641 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.178192884 2024-09-13T11:25:11.641 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:11.641 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:11.641 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:25:11.641 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:44.734915+0000", 2024-09-13T11:25:11.641 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.9970000000000001e-06 2024-09-13T11:25:11.641 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:11.641 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:11.641 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:25:11.642 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:44.734971+0000", 2024-09-13T11:25:11.642 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.6368e-05 2024-09-13T11:25:11.642 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:11.642 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:11.642 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:25:11.642 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:44.734990+0000", 2024-09-13T11:25:11.642 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.929e-05 2024-09-13T11:25:11.642 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:11.642 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:11.642 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:25:11.642 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:44.737096+0000", 2024-09-13T11:25:11.642 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0021053090000000001 2024-09-13T11:25:11.642 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:11.643 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:11.643 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:25:11.643 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:44.737178+0000", 2024-09-13T11:25:11.643 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.2255000000000004e-05 2024-09-13T11:25:11.643 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:11.643 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:25:11.643 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:11.643 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:11.643 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:11.643 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:137062 3.5 3:a16e178f:::benchmark_data_smithi177_31116_object137061:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e139)", 2024-09-13T11:25:11.643 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:53.140525+0000", 2024-09-13T11:25:11.643 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 18.407500791, 2024-09-13T11:25:11.644 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.20697954599999999, 2024-09-13T11:25:11.644 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:25:11.644 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:25:11.644 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:11.644 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:25:11.644 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:25:11.644 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:11.644 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 137062 2024-09-13T11:25:11.644 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:11.644 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:25:11.644 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:11.644 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:25:11.645 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:53.140525+0000", 2024-09-13T11:25:11.645 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:11.645 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:11.645 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:11.645 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:25:11.645 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:53.140525+0000", 2024-09-13T11:25:11.645 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:11.645 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:11.645 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:11.645 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:25:11.645 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:53.140527+0000", 2024-09-13T11:25:11.645 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.1739999999999999e-06 2024-09-13T11:25:11.645 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:11.646 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:11.646 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:25:11.646 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:53.140553+0000", 2024-09-13T11:25:11.646 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.6631e-05 2024-09-13T11:25:11.646 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:11.646 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:11.646 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:25:11.646 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:53.140566+0000", 2024-09-13T11:25:11.646 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.2268e-05 2024-09-13T11:25:11.646 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:11.647 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:11.647 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:25:11.647 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:53.140574+0000", 2024-09-13T11:25:11.647 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.5669999999999995e-06 2024-09-13T11:25:11.647 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:11.647 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:11.647 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:25:11.647 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:53.140622+0000", 2024-09-13T11:25:11.647 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.8143999999999998e-05 2024-09-13T11:25:11.647 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:11.647 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:11.647 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:25:11.647 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:53.140759+0000", 2024-09-13T11:25:11.647 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000136499 2024-09-13T11:25:11.648 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:11.648 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:11.648 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 13", 2024-09-13T11:25:11.648 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:53.140887+0000", 2024-09-13T11:25:11.648 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00012796300000000001 2024-09-13T11:25:11.648 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:11.648 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:11.648 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:25:11.648 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:53.142675+0000", 2024-09-13T11:25:11.648 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001788073 2024-09-13T11:25:11.648 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:11.648 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:11.649 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:25:11.649 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:53.347407+0000", 2024-09-13T11:25:11.649 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.20473206899999999 2024-09-13T11:25:11.649 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:11.649 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:11.649 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:25:11.649 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:53.347464+0000", 2024-09-13T11:25:11.649 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.6536000000000002e-05 2024-09-13T11:25:11.649 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:11.649 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:11.649 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:25:11.649 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:53.347504+0000", 2024-09-13T11:25:11.649 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.0621999999999998e-05 2024-09-13T11:25:11.650 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:11.650 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:25:11.650 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:11.650 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:11.650 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:25:11.650 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-09-13T11:25:11.652 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-13T11:25:11.652 INFO:tasks.ceph.osd.1.smithi092.stderr:2024-09-13T11:25:11.645+0000 7fe7789c5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33718) UID: 0 2024-09-13T11:25:11.682 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T11:25:11.752 INFO:tasks.ceph.osd.8.smithi001.stderr:2024-09-13T11:25:11.751+0000 7fb9b48ae640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26312) UID: 0 2024-09-13T11:25:11.783 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-09-13T11:25:11.784 INFO:teuthology.orchestra.run.smithi092.stdout: "ops": [ 2024-09-13T11:25:11.784 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:11.784 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:157798 3.28 3:1493eca6:::benchmark_data_smithi177_31116_object17495:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e152)", 2024-09-13T11:25:11.784 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:25:11.777330+0000", 2024-09-13T11:25:11.784 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 0.0061731889999999999, 2024-09-13T11:25:11.784 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0061972269999999996, 2024-09-13T11:25:11.784 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:25:11.784 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:25:11.784 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "waiting for sub ops", 2024-09-13T11:25:11.784 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:25:11.785 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:25:11.785 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:11.785 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 157798 2024-09-13T11:25:11.785 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:11.785 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:25:11.785 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:11.785 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:25:11.785 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:11.777330+0000", 2024-09-13T11:25:11.785 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:11.785 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:11.785 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:11.785 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:25:11.786 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:11.777330+0000", 2024-09-13T11:25:11.786 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:11.787 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:11.788 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:11.788 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:25:11.788 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:11.777332+0000", 2024-09-13T11:25:11.788 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.1660000000000001e-06 2024-09-13T11:25:11.788 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:11.788 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:11.788 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:25:11.788 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:11.777339+0000", 2024-09-13T11:25:11.788 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.3620000000000002e-06 2024-09-13T11:25:11.788 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:11.788 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:11.789 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:25:11.789 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:11.777355+0000", 2024-09-13T11:25:11.789 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.6263e-05 2024-09-13T11:25:11.789 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:11.789 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:11.789 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:25:11.789 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:11.777364+0000", 2024-09-13T11:25:11.789 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.0559999999999994e-06 2024-09-13T11:25:11.789 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:11.789 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:11.789 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:25:11.789 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:11.777462+0000", 2024-09-13T11:25:11.789 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.7714000000000007e-05 2024-09-13T11:25:11.789 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:11.790 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:11.790 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:25:11.790 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:11.777599+0000", 2024-09-13T11:25:11.790 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000137016 2024-09-13T11:25:11.790 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:11.790 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:11.790 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 5,8", 2024-09-13T11:25:11.790 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:11.777769+0000", 2024-09-13T11:25:11.790 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00017054900000000001 2024-09-13T11:25:11.790 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:11.790 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:11.790 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:25:11.790 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:11.779150+0000", 2024-09-13T11:25:11.790 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.001380961 2024-09-13T11:25:11.791 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:11.791 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:11.791 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:25:11.791 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:11.780616+0000", 2024-09-13T11:25:11.791 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.001466188 2024-09-13T11:25:11.791 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:11.791 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:25:11.791 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:11.791 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:11.791 INFO:teuthology.orchestra.run.smithi092.stdout: ], 2024-09-13T11:25:11.791 INFO:teuthology.orchestra.run.smithi092.stdout: "num_ops": 1 2024-09-13T11:25:11.791 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-09-13T11:25:11.805 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-13T11:25:11.845 INFO:teuthology.orchestra.run.smithi001.stdout:osd.0: {} 2024-09-13T11:25:11.845 INFO:teuthology.orchestra.run.smithi001.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T11:25:11.853 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-13T11:25:11.851+0000 7f6a68775640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26307) UID: 0 2024-09-13T11:25:11.870 INFO:teuthology.orchestra.run.smithi001.stdout:osd.1: {} 2024-09-13T11:25:11.870 INFO:teuthology.orchestra.run.smithi001.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T11:25:11.902 INFO:teuthology.orchestra.run.smithi001.stdout:osd.2: {} 2024-09-13T11:25:11.903 INFO:teuthology.orchestra.run.smithi001.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T11:25:11.928 INFO:teuthology.orchestra.run.smithi001.stdout:osd.3: {} 2024-09-13T11:25:11.928 INFO:teuthology.orchestra.run.smithi001.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T11:25:11.929 INFO:tasks.ceph.osd.1.smithi092.stderr:2024-09-13T11:25:11.925+0000 7fe76f3f6640 -1 osd.1 152 handle_osd_map injecting map crc failure 2024-09-13T11:25:11.930 INFO:tasks.ceph.osd.8.smithi001.stderr:2024-09-13T11:25:11.927+0000 7fb9ab2df640 -1 osd.8 152 handle_osd_map injecting map crc failure 2024-09-13T11:25:11.932 INFO:teuthology.orchestra.run.smithi001.stderr:reweighted osd.1 to 1 (10000) 2024-09-13T11:25:11.944 INFO:teuthology.orchestra.run.smithi092.stdout:ERROR: (22) Invalid argument 2024-09-13T11:25:11.945 INFO:teuthology.orchestra.run.smithi092.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:25:11.954 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-13T11:25:11.951+0000 7f6a68775640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26307) UID: 0 2024-09-13T11:25:11.957 INFO:teuthology.orchestra.run.smithi001.stdout:osd.4: {} 2024-09-13T11:25:11.958 INFO:teuthology.orchestra.run.smithi001.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T11:25:11.959 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 2 1 2024-09-13T11:25:11.962 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-13T11:25:11.989 INFO:teuthology.orchestra.run.smithi001.stdout:osd.5: {} 2024-09-13T11:25:11.990 INFO:teuthology.orchestra.run.smithi001.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T11:25:12.025 INFO:teuthology.orchestra.run.smithi001.stdout:osd.6: {} 2024-09-13T11:25:12.025 INFO:teuthology.orchestra.run.smithi001.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T11:25:12.055 INFO:tasks.ceph.osd.14.smithi162.stderr:2024-09-13T11:25:12.052+0000 7f3df5903640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33154) UID: 0 2024-09-13T11:25:12.057 INFO:teuthology.orchestra.run.smithi001.stdout:osd.7: {} 2024-09-13T11:25:12.058 INFO:teuthology.orchestra.run.smithi001.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T11:25:12.084 INFO:teuthology.orchestra.run.smithi001.stdout:osd.8: {} 2024-09-13T11:25:12.084 INFO:teuthology.orchestra.run.smithi001.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-13T11:25:12.101 INFO:teuthology.orchestra.run.smithi092.stdout:ERROR: (22) Invalid argument 2024-09-13T11:25:12.101 INFO:teuthology.orchestra.run.smithi092.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:25:12.119 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_ops_in_flight 2024-09-13T11:25:12.121 INFO:teuthology.orchestra.run.smithi001.stdout:osd.9: {} 2024-09-13T11:25:12.121 INFO:teuthology.orchestra.run.smithi001.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-13T11:25:12.154 INFO:teuthology.orchestra.run.smithi001.stdout:osd.10: {} 2024-09-13T11:25:12.154 INFO:teuthology.orchestra.run.smithi001.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-13T11:25:12.156 INFO:tasks.ceph.osd.5.smithi092.stderr:2024-09-13T11:25:12.149+0000 7f6f0b705640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26346) UID: 0 2024-09-13T11:25:12.183 INFO:teuthology.orchestra.run.smithi001.stdout:osd.11: {} 2024-09-13T11:25:12.184 INFO:teuthology.orchestra.run.smithi001.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-13T11:25:12.214 INFO:teuthology.orchestra.run.smithi001.stdout:osd.12: {} 2024-09-13T11:25:12.215 INFO:teuthology.orchestra.run.smithi001.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-13T11:25:12.243 INFO:teuthology.orchestra.run.smithi001.stdout:osd.13: {} 2024-09-13T11:25:12.243 INFO:teuthology.orchestra.run.smithi001.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-13T11:25:12.256 INFO:tasks.ceph.osd.2.smithi162.stderr:2024-09-13T11:25:12.252+0000 7fe92b35a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26313) UID: 0 2024-09-13T11:25:12.260 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-13T11:25:12.260 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [], 2024-09-13T11:25:12.260 INFO:teuthology.orchestra.run.smithi162.stdout: "num_ops": 0 2024-09-13T11:25:12.260 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-13T11:25:12.277 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_blocked_ops 2024-09-13T11:25:12.279 INFO:teuthology.orchestra.run.smithi001.stdout:osd.14: {} 2024-09-13T11:25:12.280 INFO:teuthology.orchestra.run.smithi001.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-13T11:25:12.307 INFO:teuthology.orchestra.run.smithi001.stdout:osd.15: {} 2024-09-13T11:25:12.308 INFO:teuthology.orchestra.run.smithi001.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-13T11:25:12.376 INFO:tasks.ceph.osd.4.smithi001.stderr:2024-09-13T11:25:12.375+0000 7ff8e0218640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26310) UID: 0 2024-09-13T11:25:12.444 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-13T11:25:12.444 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:25:12.458 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_historic_ops 2024-09-13T11:25:12.459 INFO:tasks.ceph.osd.2.smithi162.stderr:2024-09-13T11:25:12.456+0000 7fe92b35a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26313) UID: 0 2024-09-13T11:25:12.560 INFO:tasks.ceph.osd.13.smithi092.stderr:2024-09-13T11:25:12.553+0000 7f3e5afea640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26350) UID: 0 2024-09-13T11:25:12.608 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-13T11:25:12.608 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:25:12.624 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-13T11:25:12.661 INFO:tasks.ceph.osd.7.smithi177.stderr:2024-09-13T11:25:12.658+0000 7f897b51d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26575) UID: 0 2024-09-13T11:25:12.763 INFO:tasks.ceph.osd.11.smithi177.stderr:2024-09-13T11:25:12.758+0000 7f24792c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26580) UID: 0 2024-09-13T11:25:12.772 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-13T11:25:12.772 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:25:12.786 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-13T11:25:12.864 INFO:tasks.ceph.osd.8.smithi001.stderr:2024-09-13T11:25:12.863+0000 7fb9b48ae640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26312) UID: 0 2024-09-13T11:25:12.908 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-13T11:25:12.908 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:25:12.920 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-13T11:25:12.937 INFO:teuthology.orchestra.run.smithi001.stderr:reweighted osd.2 to 1 (10000) 2024-09-13T11:25:12.965 INFO:tasks.ceph.osd.3.smithi177.stderr:2024-09-13T11:25:12.962+0000 7f10adba4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26574) UID: 0 2024-09-13T11:25:12.966 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 3 1 2024-09-13T11:25:13.066 INFO:tasks.ceph.osd.14.smithi162.stderr:2024-09-13T11:25:13.064+0000 7f3df5903640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33154) UID: 0 2024-09-13T11:25:13.109 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-13T11:25:13.109 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:25:13.113 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-13T11:25:13.131 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-13T11:25:13.167 INFO:tasks.ceph.osd.10.smithi162.stderr:2024-09-13T11:25:13.164+0000 7fb59c9f0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26317) UID: 0 2024-09-13T11:25:13.268 INFO:tasks.ceph.osd.1.smithi092.stderr:2024-09-13T11:25:13.261+0000 7fe7789c5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33718) UID: 0 2024-09-13T11:25:13.334 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-13T11:25:13.334 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:25:13.350 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-13T11:25:13.369 INFO:tasks.ceph.osd.13.smithi092.stderr:2024-09-13T11:25:13.365+0000 7f3e5afea640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26350) UID: 0 2024-09-13T11:25:13.470 INFO:tasks.ceph.osd.2.smithi162.stderr:2024-09-13T11:25:13.468+0000 7fe92b35a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26313) UID: 0 2024-09-13T11:25:13.488 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-13T11:25:13.488 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:25:13.506 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-13T11:25:13.570 INFO:tasks.ceph.osd.8.smithi001.stderr:2024-09-13T11:25:13.567+0000 7fb9b48ae640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26312) UID: 0 2024-09-13T11:25:13.640 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-13T11:25:13.640 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:25:13.659 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-09-13T11:25:13.704 INFO:tasks.ceph.osd.8.smithi001.stderr:2024-09-13T11:25:13.703+0000 7fb9b48ae640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26312) UID: 0 2024-09-13T11:25:13.773 INFO:tasks.ceph.osd.14.smithi162.stderr:2024-09-13T11:25:13.768+0000 7f3df5903640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33154) UID: 0 2024-09-13T11:25:13.849 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-13T11:25:13.849 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:25:13.865 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-09-13T11:25:13.873 INFO:tasks.ceph.osd.1.smithi092.stderr:2024-09-13T11:25:13.869+0000 7fe7789c5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33718) UID: 0 2024-09-13T11:25:13.974 INFO:tasks.ceph.osd.14.smithi162.stderr:2024-09-13T11:25:13.972+0000 7f3df5903640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33154) UID: 0 2024-09-13T11:25:14.001 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-13T11:25:14.001 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:25:14.018 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-09-13T11:25:14.075 INFO:tasks.ceph.osd.7.smithi177.stderr:2024-09-13T11:25:14.070+0000 7f897b51d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26575) UID: 0 2024-09-13T11:25:14.096 INFO:teuthology.orchestra.run.smithi001.stderr:reweighted osd.3 to 1 (10000) 2024-09-13T11:25:14.099 INFO:teuthology.orchestra.run.smithi001.stderr:noscrub is set 2024-09-13T11:25:14.134 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 4 1 2024-09-13T11:25:14.155 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-13T11:25:14.155 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:25:14.176 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_ops_in_flight 2024-09-13T11:25:14.176 INFO:tasks.ceph.osd.11.smithi177.stderr:2024-09-13T11:25:14.170+0000 7f24792c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26580) UID: 0 2024-09-13T11:25:14.276 INFO:tasks.ceph.osd.15.smithi177.stderr:2024-09-13T11:25:14.274+0000 7f9d6ce77640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26578) UID: 0 2024-09-13T11:25:14.332 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T11:25:14.375 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-13T11:25:14.375 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:25:14.378 INFO:tasks.ceph.osd.2.smithi162.stderr:2024-09-13T11:25:14.376+0000 7fe92b35a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26313) UID: 0 2024-09-13T11:25:14.396 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_blocked_ops 2024-09-13T11:25:14.478 INFO:tasks.ceph.osd.13.smithi092.stderr:2024-09-13T11:25:14.473+0000 7f3e5afea640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26350) UID: 0 2024-09-13T11:25:14.557 INFO:teuthology.orchestra.run.smithi001.stdout:osd.0: {} 2024-09-13T11:25:14.558 INFO:teuthology.orchestra.run.smithi001.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T11:25:14.579 INFO:tasks.ceph.osd.1.smithi092.stderr:2024-09-13T11:25:14.573+0000 7fe7789c5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33718) UID: 0 2024-09-13T11:25:14.587 INFO:teuthology.orchestra.run.smithi001.stdout:osd.1: {} 2024-09-13T11:25:14.587 INFO:teuthology.orchestra.run.smithi001.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T11:25:14.620 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-13T11:25:14.620 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:25:14.636 INFO:teuthology.orchestra.run.smithi001.stdout:osd.2: {} 2024-09-13T11:25:14.636 INFO:teuthology.orchestra.run.smithi001.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T11:25:14.643 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_historic_ops 2024-09-13T11:25:14.664 INFO:teuthology.orchestra.run.smithi001.stdout:osd.3: {} 2024-09-13T11:25:14.664 INFO:teuthology.orchestra.run.smithi001.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T11:25:14.680 INFO:tasks.ceph.osd.5.smithi092.stderr:2024-09-13T11:25:14.673+0000 7f6f0b705640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26346) UID: 0 2024-09-13T11:25:14.690 INFO:teuthology.orchestra.run.smithi001.stdout:osd.4: {} 2024-09-13T11:25:14.690 INFO:teuthology.orchestra.run.smithi001.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T11:25:14.727 INFO:teuthology.orchestra.run.smithi001.stdout:osd.5: {} 2024-09-13T11:25:14.727 INFO:teuthology.orchestra.run.smithi001.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T11:25:14.767 INFO:teuthology.orchestra.run.smithi001.stdout:osd.6: {} 2024-09-13T11:25:14.767 INFO:teuthology.orchestra.run.smithi001.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T11:25:14.781 INFO:tasks.ceph.osd.1.smithi092.stderr:2024-09-13T11:25:14.777+0000 7fe7789c5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33718) UID: 0 2024-09-13T11:25:14.793 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-13T11:25:14.793 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:25:14.794 INFO:teuthology.orchestra.run.smithi001.stdout:osd.7: {} 2024-09-13T11:25:14.794 INFO:teuthology.orchestra.run.smithi001.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T11:25:14.813 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-13T11:25:14.818 INFO:teuthology.orchestra.run.smithi001.stdout:osd.8: {} 2024-09-13T11:25:14.818 INFO:teuthology.orchestra.run.smithi001.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-13T11:25:14.846 INFO:teuthology.orchestra.run.smithi001.stdout:osd.9: {} 2024-09-13T11:25:14.846 INFO:teuthology.orchestra.run.smithi001.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-13T11:25:14.876 INFO:teuthology.orchestra.run.smithi001.stdout:osd.10: {} 2024-09-13T11:25:14.876 INFO:teuthology.orchestra.run.smithi001.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-13T11:25:14.883 INFO:tasks.ceph.osd.13.smithi092.stderr:2024-09-13T11:25:14.877+0000 7f3e5afea640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26350) UID: 0 2024-09-13T11:25:14.899 INFO:teuthology.orchestra.run.smithi001.stdout:osd.11: {} 2024-09-13T11:25:14.900 INFO:teuthology.orchestra.run.smithi001.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-13T11:25:14.929 INFO:teuthology.orchestra.run.smithi001.stdout:osd.12: {} 2024-09-13T11:25:14.929 INFO:teuthology.orchestra.run.smithi001.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-13T11:25:14.955 INFO:teuthology.orchestra.run.smithi001.stdout:osd.13: {} 2024-09-13T11:25:14.955 INFO:teuthology.orchestra.run.smithi001.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-13T11:25:14.961 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-09-13T11:25:14.961 INFO:teuthology.orchestra.run.smithi092.stdout: "ops": [ 2024-09-13T11:25:14.961 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:14.961 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop(client.4665.0:161439 3.27 e155/120 3:e7b2fd0b:::benchmark_data_smithi177_31116_object21136:head v 155'5574, mlcod=66'2683)", 2024-09-13T11:25:14.961 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:25:14.960577+0000", 2024-09-13T11:25:14.961 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 0.00072656000000000001, 2024-09-13T11:25:14.961 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00074339299999999996, 2024-09-13T11:25:14.962 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:25:14.962 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:25:14.962 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-09-13T11:25:14.962 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:25:14.962 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:14.962 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:25:14.962 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:14.960577+0000", 2024-09-13T11:25:14.962 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:14.962 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:14.962 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:14.962 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:25:14.962 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:14.960577+0000", 2024-09-13T11:25:14.962 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:14.962 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:14.963 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:14.963 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:25:14.963 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:14.960579+0000", 2024-09-13T11:25:14.963 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.2449999999999999e-06 2024-09-13T11:25:14.963 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:14.963 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:14.963 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:25:14.963 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:14.960584+0000", 2024-09-13T11:25:14.963 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.614e-06 2024-09-13T11:25:14.963 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:14.963 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:14.963 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:25:14.963 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:14.960598+0000", 2024-09-13T11:25:14.964 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.3424999999999999e-05 2024-09-13T11:25:14.964 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:14.964 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:14.964 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:25:14.964 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:14.960603+0000", 2024-09-13T11:25:14.964 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.3650000000000003e-06 2024-09-13T11:25:14.964 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:14.964 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:14.964 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:25:14.964 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:14.960649+0000", 2024-09-13T11:25:14.964 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.5904e-05 2024-09-13T11:25:14.964 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:14.964 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:14.965 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:25:14.965 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:14.960668+0000", 2024-09-13T11:25:14.965 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.8706999999999999e-05 2024-09-13T11:25:14.965 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:14.965 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:25:14.965 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:14.965 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:14.965 INFO:teuthology.orchestra.run.smithi092.stdout: ], 2024-09-13T11:25:14.965 INFO:teuthology.orchestra.run.smithi092.stdout: "num_ops": 1 2024-09-13T11:25:14.965 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-09-13T11:25:14.982 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-13T11:25:14.983 INFO:tasks.ceph.osd.7.smithi177.stderr:2024-09-13T11:25:14.982+0000 7f897b51d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26575) UID: 0 2024-09-13T11:25:14.984 INFO:teuthology.orchestra.run.smithi001.stdout:osd.14: {} 2024-09-13T11:25:14.984 INFO:teuthology.orchestra.run.smithi001.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-13T11:25:15.013 INFO:teuthology.orchestra.run.smithi001.stdout:osd.15: {} 2024-09-13T11:25:15.013 INFO:teuthology.orchestra.run.smithi001.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-13T11:25:15.084 INFO:tasks.ceph.osd.2.smithi162.stderr:2024-09-13T11:25:15.080+0000 7fe92b35a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26313) UID: 0 2024-09-13T11:25:15.105 INFO:teuthology.orchestra.run.smithi001.stderr:reweighted osd.4 to 1 (10000) 2024-09-13T11:25:15.111 INFO:tasks.ceph.osd.11.smithi177.stderr:2024-09-13T11:25:15.110+0000 7f2469cec640 -1 osd.11 155 handle_osd_map injecting map crc failure 2024-09-13T11:25:15.129 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 5 1 2024-09-13T11:25:15.170 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-09-13T11:25:15.170 INFO:teuthology.orchestra.run.smithi092.stdout: "ops": [], 2024-09-13T11:25:15.170 INFO:teuthology.orchestra.run.smithi092.stdout: "complaint_time": 30, 2024-09-13T11:25:15.170 INFO:teuthology.orchestra.run.smithi092.stdout: "num_blocked_ops": 0 2024-09-13T11:25:15.170 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-09-13T11:25:15.185 INFO:tasks.ceph.osd.8.smithi001.stderr:2024-09-13T11:25:15.183+0000 7fb9b48ae640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26312) UID: 0 2024-09-13T11:25:15.190 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-13T11:25:15.286 INFO:tasks.ceph.osd.8.smithi001.stderr:2024-09-13T11:25:15.283+0000 7fb9b48ae640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26312) UID: 0 2024-09-13T11:25:15.335 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-09-13T11:25:15.335 INFO:teuthology.orchestra.run.smithi092.stdout: "size": 20, 2024-09-13T11:25:15.335 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 600, 2024-09-13T11:25:15.336 INFO:teuthology.orchestra.run.smithi092.stdout: "ops": [ 2024-09-13T11:25:15.336 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.336 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "MOSDPGPush(3.23 83/78 [PushOp(3:c61f6bab:::benchmark_data_smithi177_31116_object29479:head, version: 59'1860, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:c61f6bab:::benchmark_data_smithi177_31116_object29479:head@59'1860, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:25:15.336 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:40.770864+0000", 2024-09-13T11:25:15.336 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 94.562911018999998, 2024-09-13T11:25:15.336 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.31229662800000002, 2024-09-13T11:25:15.336 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:25:15.336 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:25:15.336 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-09-13T11:25:15.336 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:25:15.336 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.337 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:25:15.337 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:40.770864+0000", 2024-09-13T11:25:15.337 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:15.337 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.337 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.337 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:25:15.337 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:40.770864+0000", 2024-09-13T11:25:15.337 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:15.337 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.337 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.337 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:25:15.338 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:40.770865+0000", 2024-09-13T11:25:15.338 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.288e-06 2024-09-13T11:25:15.338 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.338 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.338 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:25:15.338 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:40.770955+0000", 2024-09-13T11:25:15.338 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.9871999999999996e-05 2024-09-13T11:25:15.338 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.338 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.338 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:25:15.338 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:41.082105+0000", 2024-09-13T11:25:15.338 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.31114963400000001 2024-09-13T11:25:15.338 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.339 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.339 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:25:15.339 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:41.082136+0000", 2024-09-13T11:25:15.339 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.0972000000000001e-05 2024-09-13T11:25:15.339 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.339 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.339 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:25:15.339 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:41.082404+0000", 2024-09-13T11:25:15.339 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00026844800000000002 2024-09-13T11:25:15.339 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.339 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.339 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:25:15.340 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:41.082429+0000", 2024-09-13T11:25:15.340 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.5021e-05 2024-09-13T11:25:15.340 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.340 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.340 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:25:15.340 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:41.083161+0000", 2024-09-13T11:25:15.340 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000731393 2024-09-13T11:25:15.340 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:15.340 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:25:15.340 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:15.340 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.341 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.341 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "MOSDPGPush(3.1e 85/78 [PushOp(3:7f40e925:::benchmark_data_smithi177_31116_object14084:head, version: 50'927, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:7f40e925:::benchmark_data_smithi177_31116_object14084:head@50'927, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:25:15.341 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:42.751000+0000", 2024-09-13T11:25:15.341 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 92.582774712000003, 2024-09-13T11:25:15.341 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.19208357500000001, 2024-09-13T11:25:15.341 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:25:15.341 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:25:15.341 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-09-13T11:25:15.341 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:25:15.341 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.341 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:25:15.342 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.751000+0000", 2024-09-13T11:25:15.342 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:15.342 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.342 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.342 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:25:15.342 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.751000+0000", 2024-09-13T11:25:15.342 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:15.342 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.342 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.342 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:25:15.342 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.751003+0000", 2024-09-13T11:25:15.342 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.0800000000000002e-06 2024-09-13T11:25:15.343 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.343 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.343 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:25:15.343 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.751069+0000", 2024-09-13T11:25:15.343 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.5417000000000001e-05 2024-09-13T11:25:15.343 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.343 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.343 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:25:15.343 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.941261+0000", 2024-09-13T11:25:15.343 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.19019233899999999 2024-09-13T11:25:15.343 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.343 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.343 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:25:15.344 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.941300+0000", 2024-09-13T11:25:15.344 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.9091999999999998e-05 2024-09-13T11:25:15.344 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.344 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.344 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:25:15.344 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.941519+0000", 2024-09-13T11:25:15.344 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000218521 2024-09-13T11:25:15.344 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.344 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.344 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:25:15.344 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.941544+0000", 2024-09-13T11:25:15.345 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.5412999999999999e-05 2024-09-13T11:25:15.345 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.345 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.345 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:25:15.345 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.943084+0000", 2024-09-13T11:25:15.345 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0015397130000000001 2024-09-13T11:25:15.345 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:15.345 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:25:15.345 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:15.345 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.345 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.346 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:82769 3.1e 3:7a4e88b5:::benchmark_data_smithi177_31116_object82768:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105)", 2024-09-13T11:25:15.346 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:24:03.744632+0000", 2024-09-13T11:25:15.346 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 71.589142566999996, 2024-09-13T11:25:15.346 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.21475722699999999, 2024-09-13T11:25:15.346 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:25:15.346 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:25:15.346 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:15.346 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:25:15.346 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:25:15.346 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:15.346 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 82769 2024-09-13T11:25:15.346 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.347 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:25:15.347 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.347 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:25:15.347 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:03.744632+0000", 2024-09-13T11:25:15.347 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:15.347 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.347 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.347 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:25:15.347 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:03.744632+0000", 2024-09-13T11:25:15.347 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:15.347 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.348 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.348 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:25:15.348 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:03.744634+0000", 2024-09-13T11:25:15.348 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.672e-06 2024-09-13T11:25:15.348 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.348 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.348 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:25:15.348 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:03.744724+0000", 2024-09-13T11:25:15.348 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.0237999999999999e-05 2024-09-13T11:25:15.348 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.348 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.349 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:25:15.349 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:03.744736+0000", 2024-09-13T11:25:15.349 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.2038999999999999e-05 2024-09-13T11:25:15.349 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.349 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.349 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:25:15.349 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:03.744742+0000", 2024-09-13T11:25:15.349 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.2129999999999996e-06 2024-09-13T11:25:15.349 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.349 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.349 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:25:15.349 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:03.744785+0000", 2024-09-13T11:25:15.350 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.2898000000000002e-05 2024-09-13T11:25:15.350 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.350 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.350 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:25:15.350 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:03.744894+0000", 2024-09-13T11:25:15.350 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000109073 2024-09-13T11:25:15.350 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.351 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.351 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 4", 2024-09-13T11:25:15.351 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:03.745009+0000", 2024-09-13T11:25:15.351 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00011428499999999999 2024-09-13T11:25:15.351 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.351 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.351 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:25:15.351 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:03.746451+0000", 2024-09-13T11:25:15.352 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0014419179999999999 2024-09-13T11:25:15.352 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.352 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.352 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:25:15.352 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:03.959257+0000", 2024-09-13T11:25:15.352 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.212806361 2024-09-13T11:25:15.352 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.352 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.352 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:25:15.352 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:03.959333+0000", 2024-09-13T11:25:15.352 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.6001000000000006e-05 2024-09-13T11:25:15.353 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.353 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.353 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:25:15.353 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:03.959390+0000", 2024-09-13T11:25:15.353 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.6529e-05 2024-09-13T11:25:15.353 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:15.353 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:25:15.353 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:15.353 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.353 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.353 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:104331 3.1e 3:79890035:::benchmark_data_smithi177_31116_object104330:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-09-13T11:25:15.353 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:24:20.241529+0000", 2024-09-13T11:25:15.353 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 55.092246049000003, 2024-09-13T11:25:15.353 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.16976961700000001, 2024-09-13T11:25:15.354 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:25:15.354 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:25:15.354 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:15.354 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:25:15.354 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:25:15.354 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:15.354 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 104331 2024-09-13T11:25:15.354 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.354 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:25:15.354 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.354 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:25:15.354 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:20.241529+0000", 2024-09-13T11:25:15.354 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:15.355 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.355 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.355 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:25:15.355 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:20.241529+0000", 2024-09-13T11:25:15.355 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:15.355 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.355 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.355 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:25:15.355 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:20.241534+0000", 2024-09-13T11:25:15.355 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.775e-06 2024-09-13T11:25:15.355 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.355 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.355 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:25:15.355 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:20.241660+0000", 2024-09-13T11:25:15.356 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000126499 2024-09-13T11:25:15.356 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.356 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.356 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:25:15.356 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:20.241687+0000", 2024-09-13T11:25:15.356 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.6826999999999999e-05 2024-09-13T11:25:15.356 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.356 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.356 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:25:15.356 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:20.241703+0000", 2024-09-13T11:25:15.356 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.5858999999999998e-05 2024-09-13T11:25:15.356 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.356 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.357 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:25:15.357 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:20.241832+0000", 2024-09-13T11:25:15.357 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00012925799999999999 2024-09-13T11:25:15.357 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.357 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.357 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:25:15.357 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:20.242133+0000", 2024-09-13T11:25:15.357 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00030115900000000001 2024-09-13T11:25:15.357 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.357 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.358 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 4", 2024-09-13T11:25:15.358 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:20.242408+0000", 2024-09-13T11:25:15.358 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00027487799999999999 2024-09-13T11:25:15.358 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.358 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.358 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:25:15.358 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:20.246683+0000", 2024-09-13T11:25:15.358 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0042750469999999997 2024-09-13T11:25:15.358 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.358 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.358 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:25:15.358 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:20.411076+0000", 2024-09-13T11:25:15.358 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.16439300800000001 2024-09-13T11:25:15.359 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.359 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.359 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:25:15.359 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:20.411188+0000", 2024-09-13T11:25:15.359 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00011191 2024-09-13T11:25:15.359 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.359 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.359 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:25:15.359 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:20.411298+0000", 2024-09-13T11:25:15.359 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00011039699999999999 2024-09-13T11:25:15.359 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:15.359 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:25:15.359 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:15.360 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.360 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.360 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:110289 3.29 3.27287969 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e119)", 2024-09-13T11:25:15.360 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:24:25.774634+0000", 2024-09-13T11:25:15.360 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 49.559140943999999, 2024-09-13T11:25:15.360 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.31721633999999999, 2024-09-13T11:25:15.360 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:25:15.360 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:25:15.360 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "reached pg", 2024-09-13T11:25:15.360 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:25:15.360 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:25:15.360 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:15.360 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 110289 2024-09-13T11:25:15.360 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.361 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:25:15.361 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.361 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:25:15.361 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:25.774634+0000", 2024-09-13T11:25:15.361 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:15.361 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.361 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.361 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:25:15.361 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:25.774634+0000", 2024-09-13T11:25:15.361 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:15.361 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.361 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.361 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:25:15.362 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:25.774639+0000", 2024-09-13T11:25:15.362 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.6029999999999998e-06 2024-09-13T11:25:15.362 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.362 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.362 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:25:15.362 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:25.774695+0000", 2024-09-13T11:25:15.362 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.6694999999999998e-05 2024-09-13T11:25:15.362 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.362 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.362 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:25:15.362 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:25.774728+0000", 2024-09-13T11:25:15.362 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.2502000000000002e-05 2024-09-13T11:25:15.362 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.362 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.362 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:25:15.363 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:25.774750+0000", 2024-09-13T11:25:15.363 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.2002999999999999e-05 2024-09-13T11:25:15.363 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.363 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.363 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:25:15.363 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:25.774942+0000", 2024-09-13T11:25:15.363 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000192265 2024-09-13T11:25:15.363 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.363 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.363 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-09-13T11:25:15.363 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:25.774960+0000", 2024-09-13T11:25:15.363 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.8275999999999999e-05 2024-09-13T11:25:15.363 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.363 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.363 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:25:15.364 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:26.091821+0000", 2024-09-13T11:25:15.364 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.31686065000000002 2024-09-13T11:25:15.364 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.364 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.364 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:25:15.364 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:26.091850+0000", 2024-09-13T11:25:15.364 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.9346e-05 2024-09-13T11:25:15.364 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:15.364 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:25:15.364 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:15.364 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.364 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.364 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "MOSDPGPush(3.22 125/122 [PushOp(3:44e4f9f2:::benchmark_data_smithi177_31116_object16663:head, version: 52'1032, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:44e4f9f2:::benchmark_data_smithi177_31116_object16663:head@52'1032, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:25:15.365 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:24:32.743254+0000", 2024-09-13T11:25:15.365 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 42.590520855000001, 2024-09-13T11:25:15.365 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.215485287, 2024-09-13T11:25:15.365 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:25:15.365 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:25:15.365 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-09-13T11:25:15.365 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:25:15.365 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.365 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:25:15.365 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:32.743254+0000", 2024-09-13T11:25:15.365 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:15.365 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.365 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.366 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:25:15.366 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:32.743254+0000", 2024-09-13T11:25:15.366 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:15.366 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.366 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.366 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:25:15.366 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:32.743255+0000", 2024-09-13T11:25:15.366 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.2500000000000004e-07 2024-09-13T11:25:15.366 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.366 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.366 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:25:15.366 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:32.743310+0000", 2024-09-13T11:25:15.366 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.5442000000000002e-05 2024-09-13T11:25:15.366 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.367 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.367 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:25:15.367 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:32.957754+0000", 2024-09-13T11:25:15.367 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.21444387000000001 2024-09-13T11:25:15.367 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.367 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.367 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:25:15.367 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:32.957765+0000", 2024-09-13T11:25:15.367 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0458999999999999e-05 2024-09-13T11:25:15.367 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.367 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.367 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:25:15.367 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:32.957841+0000", 2024-09-13T11:25:15.367 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.5993000000000002e-05 2024-09-13T11:25:15.368 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.368 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.368 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:25:15.368 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:32.957852+0000", 2024-09-13T11:25:15.368 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.1453e-05 2024-09-13T11:25:15.368 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.368 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.368 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:25:15.368 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:32.958739+0000", 2024-09-13T11:25:15.368 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00088724500000000005 2024-09-13T11:25:15.368 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:15.368 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:25:15.368 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:15.369 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.369 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.369 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop(client.4665.0:115089 3.28 e125/120 3:15b13357:::benchmark_data_smithi177_31116_object115088:head v 125'4843, mlcod=47'620)", 2024-09-13T11:25:15.369 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:24:33.642654+0000", 2024-09-13T11:25:15.369 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 41.691120445000003, 2024-09-13T11:25:15.369 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.19312220099999999, 2024-09-13T11:25:15.369 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:25:15.369 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:25:15.369 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:15.369 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:25:15.369 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.369 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:25:15.369 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:33.642654+0000", 2024-09-13T11:25:15.370 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:15.370 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.370 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.370 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:25:15.370 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:33.642654+0000", 2024-09-13T11:25:15.370 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:15.370 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.370 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.370 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:25:15.370 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:33.642656+0000", 2024-09-13T11:25:15.370 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.869e-06 2024-09-13T11:25:15.370 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.370 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.370 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:25:15.371 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:33.642688+0000", 2024-09-13T11:25:15.371 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.2169999999999999e-05 2024-09-13T11:25:15.371 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.371 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.371 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:25:15.371 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:33.832657+0000", 2024-09-13T11:25:15.371 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.18996885099999999 2024-09-13T11:25:15.371 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.371 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.371 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:25:15.371 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:33.832678+0000", 2024-09-13T11:25:15.371 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.0222000000000001e-05 2024-09-13T11:25:15.371 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.371 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.372 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:25:15.372 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:33.832760+0000", 2024-09-13T11:25:15.372 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.2113000000000005e-05 2024-09-13T11:25:15.372 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.372 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.372 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:25:15.372 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:33.832788+0000", 2024-09-13T11:25:15.372 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.8095000000000001e-05 2024-09-13T11:25:15.372 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.372 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.372 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:25:15.372 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:33.835729+0000", 2024-09-13T11:25:15.372 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0029408920000000001 2024-09-13T11:25:15.373 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.373 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.373 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:25:15.373 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:33.835777+0000", 2024-09-13T11:25:15.373 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.7988999999999997e-05 2024-09-13T11:25:15.373 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:15.373 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:25:15.373 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:15.373 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.373 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.373 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop(client.4665.0:115281 3.28 e125/120 3:17aeecb4:::benchmark_data_smithi177_31116_object115280:head v 125'4845, mlcod=47'620)", 2024-09-13T11:25:15.373 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:24:33.852156+0000", 2024-09-13T11:25:15.373 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 41.481619045000002, 2024-09-13T11:25:15.373 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.16574401999999999, 2024-09-13T11:25:15.374 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:25:15.374 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:25:15.374 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:15.374 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:25:15.374 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.374 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:25:15.374 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:33.852156+0000", 2024-09-13T11:25:15.374 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:15.374 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.374 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.374 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:25:15.374 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:33.852156+0000", 2024-09-13T11:25:15.374 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:15.375 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.375 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.375 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:25:15.375 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:33.852158+0000", 2024-09-13T11:25:15.375 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.683e-06 2024-09-13T11:25:15.375 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.375 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.375 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:25:15.375 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:33.852184+0000", 2024-09-13T11:25:15.375 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.6241000000000002e-05 2024-09-13T11:25:15.375 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.375 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.375 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:25:15.375 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:34.015751+0000", 2024-09-13T11:25:15.376 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.16356685800000001 2024-09-13T11:25:15.376 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.376 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.376 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:25:15.376 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:34.015757+0000", 2024-09-13T11:25:15.376 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.8980000000000001e-06 2024-09-13T11:25:15.376 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.376 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.376 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:25:15.376 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:34.015800+0000", 2024-09-13T11:25:15.376 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.3371999999999997e-05 2024-09-13T11:25:15.376 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.376 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.376 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:25:15.377 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:34.015822+0000", 2024-09-13T11:25:15.377 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.2565000000000002e-05 2024-09-13T11:25:15.377 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.377 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.377 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:25:15.377 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:34.017861+0000", 2024-09-13T11:25:15.377 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0020388469999999999 2024-09-13T11:25:15.377 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.377 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.377 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:25:15.377 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:34.017900+0000", 2024-09-13T11:25:15.377 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.8556000000000002e-05 2024-09-13T11:25:15.377 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:15.377 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:25:15.378 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:15.378 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.378 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.378 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop(client.4665.0:137296 3.2d e139/120 3:b4062804:::benchmark_data_smithi177_31116_object137295:head v 139'5552, mlcod=46'430)", 2024-09-13T11:25:15.378 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:24:53.347825+0000", 2024-09-13T11:25:15.378 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 21.985949650999999, 2024-09-13T11:25:15.378 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.162969905, 2024-09-13T11:25:15.378 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:25:15.378 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:25:15.378 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:15.378 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:25:15.379 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.379 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:25:15.379 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:53.347825+0000", 2024-09-13T11:25:15.379 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:15.379 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.379 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.379 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:25:15.379 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:53.347825+0000", 2024-09-13T11:25:15.379 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:15.379 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.379 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.379 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:25:15.379 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:53.347827+0000", 2024-09-13T11:25:15.380 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.517e-06 2024-09-13T11:25:15.380 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.380 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.380 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:25:15.380 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:53.347889+0000", 2024-09-13T11:25:15.380 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.1798999999999996e-05 2024-09-13T11:25:15.380 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.380 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.380 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:25:15.380 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:53.508273+0000", 2024-09-13T11:25:15.380 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.160384001 2024-09-13T11:25:15.380 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.381 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.381 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:25:15.381 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:53.508283+0000", 2024-09-13T11:25:15.381 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0074999999999999e-05 2024-09-13T11:25:15.381 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.381 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.381 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:25:15.381 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:53.508340+0000", 2024-09-13T11:25:15.381 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.7766000000000002e-05 2024-09-13T11:25:15.381 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.381 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.381 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:25:15.381 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:53.508358+0000", 2024-09-13T11:25:15.381 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.7940000000000001e-05 2024-09-13T11:25:15.382 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.382 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.382 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:25:15.382 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:53.510750+0000", 2024-09-13T11:25:15.382 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0023914639999999998 2024-09-13T11:25:15.382 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.382 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.382 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:25:15.382 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:53.510795+0000", 2024-09-13T11:25:15.382 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.5343000000000003e-05 2024-09-13T11:25:15.382 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:15.382 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:25:15.382 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:15.382 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.383 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.383 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "MOSDPGPush(3.29 139/120 [PushOp(3:961aa4aa:::benchmark_data_smithi177_31116_object26561:head, version: 58'1688, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:961aa4aa:::benchmark_data_smithi177_31116_object26561:head@58'1688, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:25:15.383 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:24:53.385067+0000", 2024-09-13T11:25:15.383 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 21.948707487, 2024-09-13T11:25:15.383 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.185187613, 2024-09-13T11:25:15.383 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:25:15.383 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:25:15.383 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-09-13T11:25:15.383 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:25:15.383 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.383 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:25:15.383 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:53.385067+0000", 2024-09-13T11:25:15.383 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:15.383 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.384 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.384 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:25:15.384 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:53.385067+0000", 2024-09-13T11:25:15.384 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:15.384 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.384 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.384 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:25:15.384 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:53.385069+0000", 2024-09-13T11:25:15.384 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.4729999999999999e-06 2024-09-13T11:25:15.384 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.384 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.384 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:25:15.384 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:53.385139+0000", 2024-09-13T11:25:15.384 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.9881000000000004e-05 2024-09-13T11:25:15.385 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.385 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.385 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:25:15.385 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:53.569032+0000", 2024-09-13T11:25:15.385 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.183893626 2024-09-13T11:25:15.385 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.385 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.385 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:25:15.385 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:53.569053+0000", 2024-09-13T11:25:15.385 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.0305999999999998e-05 2024-09-13T11:25:15.385 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.385 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.385 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:25:15.385 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:53.569205+0000", 2024-09-13T11:25:15.386 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00015209699999999999 2024-09-13T11:25:15.386 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.386 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.386 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:25:15.386 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:53.569214+0000", 2024-09-13T11:25:15.386 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.3740000000000004e-06 2024-09-13T11:25:15.386 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.386 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.386 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:25:15.386 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:53.570255+0000", 2024-09-13T11:25:15.386 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0010408559999999999 2024-09-13T11:25:15.386 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:15.386 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:25:15.386 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:15.387 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.387 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.387 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:138370 3.23 3:c5730cd2:::benchmark_data_smithi177_31116_object138369:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e140)", 2024-09-13T11:25:15.387 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:24:54.415176+0000", 2024-09-13T11:25:15.387 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 20.918598847999998, 2024-09-13T11:25:15.387 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.24549895499999999, 2024-09-13T11:25:15.387 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:25:15.387 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:25:15.387 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:15.387 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:25:15.387 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:25:15.387 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:15.387 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 138370 2024-09-13T11:25:15.387 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.388 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:25:15.388 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.388 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:25:15.388 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:54.415176+0000", 2024-09-13T11:25:15.388 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:15.388 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.388 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.388 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:25:15.388 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:54.415176+0000", 2024-09-13T11:25:15.388 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:15.388 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.388 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.388 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:25:15.388 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:54.415180+0000", 2024-09-13T11:25:15.388 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.1670000000000001e-06 2024-09-13T11:25:15.389 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.389 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.389 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:25:15.389 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:54.415269+0000", 2024-09-13T11:25:15.389 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.8659000000000006e-05 2024-09-13T11:25:15.389 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.389 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.389 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:25:15.389 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:54.415292+0000", 2024-09-13T11:25:15.389 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.3152000000000001e-05 2024-09-13T11:25:15.389 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.389 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.389 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:25:15.389 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:54.415304+0000", 2024-09-13T11:25:15.390 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.1841999999999999e-05 2024-09-13T11:25:15.390 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.390 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.390 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:25:15.390 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:54.415432+0000", 2024-09-13T11:25:15.390 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00012802000000000001 2024-09-13T11:25:15.390 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.390 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.390 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:25:15.390 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:54.415710+0000", 2024-09-13T11:25:15.390 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00027840299999999998 2024-09-13T11:25:15.390 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.390 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.390 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 9", 2024-09-13T11:25:15.391 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:54.415950+0000", 2024-09-13T11:25:15.391 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00023928299999999999 2024-09-13T11:25:15.391 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.391 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.391 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:25:15.391 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:54.419265+0000", 2024-09-13T11:25:15.391 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0033150440000000001 2024-09-13T11:25:15.391 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.391 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.391 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:25:15.391 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:54.660448+0000", 2024-09-13T11:25:15.391 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.24118348100000001 2024-09-13T11:25:15.391 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.391 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.392 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:25:15.392 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:54.660571+0000", 2024-09-13T11:25:15.392 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00012332900000000001 2024-09-13T11:25:15.392 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.392 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.392 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:25:15.392 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:54.660675+0000", 2024-09-13T11:25:15.392 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000103575 2024-09-13T11:25:15.392 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:15.392 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:25:15.392 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:15.392 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.392 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.392 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop(client.4665.0:138550 3.27 e140/120 3:e494a291:::benchmark_data_smithi177_31116_object138549:head v 140'5229, mlcod=49'686)", 2024-09-13T11:25:15.393 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:24:54.831360+0000", 2024-09-13T11:25:15.393 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 20.50241484, 2024-09-13T11:25:15.393 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.176014844, 2024-09-13T11:25:15.393 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:25:15.393 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:25:15.393 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:15.393 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:25:15.393 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.393 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:25:15.393 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:54.831360+0000", 2024-09-13T11:25:15.393 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:15.393 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.393 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.393 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:25:15.394 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:54.831360+0000", 2024-09-13T11:25:15.394 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:15.394 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.394 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.394 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:25:15.394 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:54.831367+0000", 2024-09-13T11:25:15.394 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.8419999999999999e-06 2024-09-13T11:25:15.394 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.394 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.394 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:25:15.394 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:54.831426+0000", 2024-09-13T11:25:15.394 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.9072e-05 2024-09-13T11:25:15.394 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.394 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.395 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:25:15.395 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:54.992898+0000", 2024-09-13T11:25:15.395 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.161471579 2024-09-13T11:25:15.395 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.395 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.395 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:25:15.395 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:54.992915+0000", 2024-09-13T11:25:15.395 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.7299000000000002e-05 2024-09-13T11:25:15.395 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.395 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.395 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:25:15.395 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:54.993246+0000", 2024-09-13T11:25:15.395 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00033146199999999998 2024-09-13T11:25:15.395 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.396 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.396 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:25:15.396 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:54.993391+0000", 2024-09-13T11:25:15.396 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00014489400000000001 2024-09-13T11:25:15.396 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.396 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.396 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:25:15.396 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:55.007321+0000", 2024-09-13T11:25:15.396 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.013929777000000001 2024-09-13T11:25:15.396 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.396 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.396 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:25:15.396 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:55.007375+0000", 2024-09-13T11:25:15.396 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.3919000000000003e-05 2024-09-13T11:25:15.397 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:15.397 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:25:15.397 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:15.397 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.397 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.397 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:140742 3.1e 3:79c694a0:::benchmark_data_smithi177_31116_object439:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e144)", 2024-09-13T11:25:15.397 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:24:58.321262+0000", 2024-09-13T11:25:15.397 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 17.012512431000001, 2024-09-13T11:25:15.397 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.23727013499999999, 2024-09-13T11:25:15.397 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:25:15.397 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:25:15.397 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:15.397 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:25:15.398 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:25:15.398 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:15.398 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 140742 2024-09-13T11:25:15.398 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.398 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:25:15.398 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.398 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:25:15.398 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:58.321262+0000", 2024-09-13T11:25:15.398 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:15.398 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.398 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.398 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:25:15.398 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:58.321262+0000", 2024-09-13T11:25:15.398 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:15.399 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.399 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.399 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:25:15.399 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:58.321264+0000", 2024-09-13T11:25:15.399 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.581e-06 2024-09-13T11:25:15.399 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.399 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.399 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:25:15.399 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:58.321272+0000", 2024-09-13T11:25:15.399 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.2179999999999999e-06 2024-09-13T11:25:15.399 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.399 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.399 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:25:15.399 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:58.321282+0000", 2024-09-13T11:25:15.400 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0173000000000001e-05 2024-09-13T11:25:15.400 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.400 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.400 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:25:15.400 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:58.321290+0000", 2024-09-13T11:25:15.400 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.943e-06 2024-09-13T11:25:15.400 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.400 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.400 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:25:15.400 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:58.321355+0000", 2024-09-13T11:25:15.400 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.4461000000000005e-05 2024-09-13T11:25:15.400 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.400 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.401 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:25:15.401 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:58.321450+0000", 2024-09-13T11:25:15.401 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.4769e-05 2024-09-13T11:25:15.401 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.401 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.401 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 4", 2024-09-13T11:25:15.401 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:58.321516+0000", 2024-09-13T11:25:15.401 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.5998000000000001e-05 2024-09-13T11:25:15.401 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.401 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.401 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:25:15.401 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:58.324045+0000", 2024-09-13T11:25:15.401 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0025290579999999998 2024-09-13T11:25:15.401 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.401 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.402 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:25:15.402 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:58.558244+0000", 2024-09-13T11:25:15.402 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.234199455 2024-09-13T11:25:15.402 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.402 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.402 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:25:15.402 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:58.558401+0000", 2024-09-13T11:25:15.402 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00015700800000000001 2024-09-13T11:25:15.402 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.402 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.402 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:25:15.402 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:58.558533+0000", 2024-09-13T11:25:15.402 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000131471 2024-09-13T11:25:15.402 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:15.403 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:25:15.403 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:15.403 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.403 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.403 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "MOSDPGPush(3.12 144/122 [PushOp(3:4e80bd1d:::benchmark_data_smithi177_31116_object55197:head, version: 79'3121, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:4e80bd1d:::benchmark_data_smithi177_31116_object55197:head@79'3121, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:25:15.403 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:24:58.397928+0000", 2024-09-13T11:25:15.403 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 16.935846497, 2024-09-13T11:25:15.403 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.16235765499999999, 2024-09-13T11:25:15.403 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:25:15.403 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:25:15.403 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-09-13T11:25:15.403 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:25:15.403 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.403 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:25:15.404 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:58.397928+0000", 2024-09-13T11:25:15.404 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:15.404 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.404 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.404 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:25:15.404 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:58.397928+0000", 2024-09-13T11:25:15.404 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:15.404 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.404 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.404 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:25:15.404 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:58.397930+0000", 2024-09-13T11:25:15.404 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.818e-06 2024-09-13T11:25:15.404 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.405 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.405 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:25:15.405 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:58.397994+0000", 2024-09-13T11:25:15.405 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.4071000000000004e-05 2024-09-13T11:25:15.406 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.406 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.406 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:25:15.406 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:58.555948+0000", 2024-09-13T11:25:15.406 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.15795406000000001 2024-09-13T11:25:15.406 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.406 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.406 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:25:15.406 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:58.556000+0000", 2024-09-13T11:25:15.406 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.1635999999999998e-05 2024-09-13T11:25:15.406 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.406 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.406 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:25:15.407 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:58.556278+0000", 2024-09-13T11:25:15.407 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00027833600000000001 2024-09-13T11:25:15.407 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.407 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.407 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:25:15.407 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:58.556329+0000", 2024-09-13T11:25:15.407 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.0429000000000001e-05 2024-09-13T11:25:15.407 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.407 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.407 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:25:15.407 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:58.560286+0000", 2024-09-13T11:25:15.407 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0039573050000000004 2024-09-13T11:25:15.407 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:15.407 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:25:15.408 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:15.408 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.408 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.408 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:140796 3.1e 3:7987a652:::benchmark_data_smithi177_31116_object493:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e144)", 2024-09-13T11:25:15.408 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:24:58.552900+0000", 2024-09-13T11:25:15.408 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 16.780874896, 2024-09-13T11:25:15.408 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.254397398, 2024-09-13T11:25:15.408 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:25:15.408 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:25:15.408 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:15.408 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:25:15.408 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:25:15.408 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:15.408 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 140796 2024-09-13T11:25:15.409 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.409 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:25:15.409 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.409 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:25:15.409 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:58.552900+0000", 2024-09-13T11:25:15.409 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:15.409 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.409 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.409 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:25:15.409 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:58.552900+0000", 2024-09-13T11:25:15.409 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:15.409 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.409 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.409 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:25:15.410 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:58.552904+0000", 2024-09-13T11:25:15.410 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.76e-06 2024-09-13T11:25:15.410 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.410 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.410 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:25:15.410 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:58.552915+0000", 2024-09-13T11:25:15.410 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0763e-05 2024-09-13T11:25:15.410 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.410 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.410 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:25:15.410 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:58.552943+0000", 2024-09-13T11:25:15.410 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.8469000000000001e-05 2024-09-13T11:25:15.410 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.410 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.411 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:25:15.411 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:58.552959+0000", 2024-09-13T11:25:15.411 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.5549999999999999e-05 2024-09-13T11:25:15.411 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.411 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.411 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:25:15.411 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:58.553106+0000", 2024-09-13T11:25:15.411 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00014729500000000001 2024-09-13T11:25:15.411 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.411 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.411 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:25:15.411 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:58.553348+0000", 2024-09-13T11:25:15.412 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000242353 2024-09-13T11:25:15.412 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.412 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.412 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 4", 2024-09-13T11:25:15.412 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:58.553556+0000", 2024-09-13T11:25:15.412 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00020765700000000001 2024-09-13T11:25:15.412 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.412 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.412 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:25:15.412 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:58.557140+0000", 2024-09-13T11:25:15.412 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0035844840000000002 2024-09-13T11:25:15.412 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.412 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.412 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:25:15.413 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:58.807126+0000", 2024-09-13T11:25:15.413 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.24998542000000001 2024-09-13T11:25:15.413 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.413 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.413 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:25:15.413 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:58.807214+0000", 2024-09-13T11:25:15.413 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.8561999999999997e-05 2024-09-13T11:25:15.413 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.413 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.413 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:25:15.413 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:58.807297+0000", 2024-09-13T11:25:15.413 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.3084999999999995e-05 2024-09-13T11:25:15.413 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:15.414 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:25:15.414 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:15.414 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.414 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.414 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "MOSDPGPush(3.12 144/122 [PushOp(3:4cd0194d:::benchmark_data_smithi177_31116_object55212:head, version: 79'3122, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:4cd0194d:::benchmark_data_smithi177_31116_object55212:head@79'3122, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:25:15.414 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:24:58.572241+0000", 2024-09-13T11:25:15.414 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 16.761534270999999, 2024-09-13T11:25:15.414 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.20343498300000001, 2024-09-13T11:25:15.414 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:25:15.414 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:25:15.414 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-09-13T11:25:15.414 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:25:15.414 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.414 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:25:15.415 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:58.572241+0000", 2024-09-13T11:25:15.415 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:15.415 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.415 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.415 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:25:15.415 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:58.572241+0000", 2024-09-13T11:25:15.415 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:15.415 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.415 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.415 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:25:15.415 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:58.572244+0000", 2024-09-13T11:25:15.415 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.4929999999999998e-06 2024-09-13T11:25:15.415 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.416 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.416 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:25:15.416 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:58.572321+0000", 2024-09-13T11:25:15.416 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.6983000000000005e-05 2024-09-13T11:25:15.416 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.416 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.416 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:25:15.416 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:58.771433+0000", 2024-09-13T11:25:15.416 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.19911215299999999 2024-09-13T11:25:15.416 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.416 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.416 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:25:15.416 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:58.771462+0000", 2024-09-13T11:25:15.416 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.8345999999999999e-05 2024-09-13T11:25:15.417 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.417 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.417 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:25:15.417 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:58.771615+0000", 2024-09-13T11:25:15.417 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000152981 2024-09-13T11:25:15.417 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.417 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.417 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:25:15.417 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:58.771638+0000", 2024-09-13T11:25:15.417 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.3516e-05 2024-09-13T11:25:15.417 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.417 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.417 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:25:15.418 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:58.775676+0000", 2024-09-13T11:25:15.418 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0040375109999999997 2024-09-13T11:25:15.418 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:15.418 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:25:15.418 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:15.418 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.418 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.418 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop(client.4665.0:147266 3.2d e148/120 3:b77e61ed:::benchmark_data_smithi177_31116_object6963:head v 148'5717, mlcod=46'430)", 2024-09-13T11:25:15.418 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:25:03.948902+0000", 2024-09-13T11:25:15.418 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 11.384873057, 2024-09-13T11:25:15.418 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.177609762, 2024-09-13T11:25:15.418 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:25:15.418 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:25:15.419 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:15.419 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:25:15.419 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.419 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:25:15.419 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:03.948902+0000", 2024-09-13T11:25:15.419 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:15.419 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.419 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.419 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:25:15.419 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:03.948902+0000", 2024-09-13T11:25:15.419 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:15.419 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.419 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.419 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:25:15.420 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:03.948903+0000", 2024-09-13T11:25:15.420 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.6020000000000001e-06 2024-09-13T11:25:15.420 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.420 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.420 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:25:15.420 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:03.948911+0000", 2024-09-13T11:25:15.420 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.7020000000000001e-06 2024-09-13T11:25:15.420 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.420 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.420 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:25:15.420 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:04.123318+0000", 2024-09-13T11:25:15.420 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.174406905 2024-09-13T11:25:15.420 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.420 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.421 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:25:15.421 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:04.123322+0000", 2024-09-13T11:25:15.421 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.3919999999999996e-06 2024-09-13T11:25:15.421 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.421 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.421 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:25:15.421 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:04.123378+0000", 2024-09-13T11:25:15.421 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.5930000000000002e-05 2024-09-13T11:25:15.421 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.421 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.421 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:25:15.421 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:04.123405+0000", 2024-09-13T11:25:15.421 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.6570000000000001e-05 2024-09-13T11:25:15.421 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.422 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.422 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:25:15.422 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:04.126478+0000", 2024-09-13T11:25:15.422 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0030730699999999998 2024-09-13T11:25:15.422 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.422 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.422 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:25:15.422 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:04.126512+0000", 2024-09-13T11:25:15.422 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.3590999999999997e-05 2024-09-13T11:25:15.422 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:15.422 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:25:15.422 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:15.422 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.422 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.423 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop(client.4665.0:147469 3.2d e148/120 3:b79b6f50:::benchmark_data_smithi177_31116_object7166:head v 148'5719, mlcod=46'431)", 2024-09-13T11:25:15.423 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:25:04.133079+0000", 2024-09-13T11:25:15.423 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 11.200695977000001, 2024-09-13T11:25:15.423 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.16502599900000001, 2024-09-13T11:25:15.423 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:25:15.423 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:25:15.423 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:15.423 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:25:15.423 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.423 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:25:15.423 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:04.133079+0000", 2024-09-13T11:25:15.423 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:15.423 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.424 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.424 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:25:15.424 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:04.133079+0000", 2024-09-13T11:25:15.424 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:15.424 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.424 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.424 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:25:15.424 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:04.133081+0000", 2024-09-13T11:25:15.424 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.624e-06 2024-09-13T11:25:15.424 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.424 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.424 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:25:15.424 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:04.133087+0000", 2024-09-13T11:25:15.424 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.7530000000000004e-06 2024-09-13T11:25:15.425 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.425 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.425 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:25:15.425 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:04.296266+0000", 2024-09-13T11:25:15.425 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.163178985 2024-09-13T11:25:15.425 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.425 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.425 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:25:15.425 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:04.296272+0000", 2024-09-13T11:25:15.425 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.0170000000000002e-06 2024-09-13T11:25:15.425 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.425 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.425 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:25:15.425 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:04.296327+0000", 2024-09-13T11:25:15.426 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.4496000000000001e-05 2024-09-13T11:25:15.426 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.426 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.426 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:25:15.426 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:04.296343+0000", 2024-09-13T11:25:15.426 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.649e-05 2024-09-13T11:25:15.426 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.426 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.426 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:25:15.426 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:04.298072+0000", 2024-09-13T11:25:15.426 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.001728222 2024-09-13T11:25:15.426 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.426 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.426 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:25:15.427 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:04.298105+0000", 2024-09-13T11:25:15.427 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.3411999999999998e-05 2024-09-13T11:25:15.427 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:15.427 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:25:15.427 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:15.427 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.427 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.427 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop(client.4665.0:147472 3.2d e148/120 3:b6aeefb0:::benchmark_data_smithi177_31116_object7169:head v 148'5720, mlcod=46'431)", 2024-09-13T11:25:15.427 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:25:04.137580+0000", 2024-09-13T11:25:15.427 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 11.196194822000001, 2024-09-13T11:25:15.427 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.214816644, 2024-09-13T11:25:15.427 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:25:15.427 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:25:15.427 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:15.428 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:25:15.428 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.428 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:25:15.428 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:04.137580+0000", 2024-09-13T11:25:15.428 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:15.428 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.428 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.428 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:25:15.428 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:04.137580+0000", 2024-09-13T11:25:15.428 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:15.428 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.428 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.428 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:25:15.429 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:04.137581+0000", 2024-09-13T11:25:15.429 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.2309999999999999e-06 2024-09-13T11:25:15.429 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.429 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.429 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:25:15.429 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:04.137587+0000", 2024-09-13T11:25:15.429 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.879e-06 2024-09-13T11:25:15.429 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.429 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.429 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:25:15.429 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:04.350285+0000", 2024-09-13T11:25:15.429 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.212697737 2024-09-13T11:25:15.429 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.430 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.430 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:25:15.430 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:04.350291+0000", 2024-09-13T11:25:15.430 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.9070000000000004e-06 2024-09-13T11:25:15.430 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.430 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.430 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:25:15.430 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:04.350368+0000", 2024-09-13T11:25:15.430 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.7150000000000005e-05 2024-09-13T11:25:15.430 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.430 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.430 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:25:15.430 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:04.350407+0000", 2024-09-13T11:25:15.430 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.9433e-05 2024-09-13T11:25:15.431 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.431 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.431 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:25:15.431 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:04.352361+0000", 2024-09-13T11:25:15.431 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0019532379999999999 2024-09-13T11:25:15.431 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.431 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.431 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:25:15.431 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:04.352397+0000", 2024-09-13T11:25:15.431 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.6069000000000003e-05 2024-09-13T11:25:15.431 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:15.431 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:25:15.431 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:15.431 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.432 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.432 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "MOSDPGPush(3.29 149/120 [PushOp(3:97fa78fe:::benchmark_data_smithi177_31116_object84063:head, version: 107'4048, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:97fa78fe:::benchmark_data_smithi177_31116_object84063:head@107'4048, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:25:15.432 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:25:06.346022+0000", 2024-09-13T11:25:15.432 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 8.9877525990000002, 2024-09-13T11:25:15.432 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.18646221599999999, 2024-09-13T11:25:15.432 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:25:15.432 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:25:15.432 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-09-13T11:25:15.432 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:25:15.432 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.432 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:25:15.432 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:06.346022+0000", 2024-09-13T11:25:15.432 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:15.432 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.433 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.433 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:25:15.433 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:06.346022+0000", 2024-09-13T11:25:15.433 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:15.433 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.433 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.433 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:25:15.433 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:06.346024+0000", 2024-09-13T11:25:15.433 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.4980000000000001e-06 2024-09-13T11:25:15.433 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.433 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.433 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:25:15.433 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:06.346058+0000", 2024-09-13T11:25:15.433 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.4137999999999997e-05 2024-09-13T11:25:15.434 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.434 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.434 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:25:15.434 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:06.531182+0000", 2024-09-13T11:25:15.434 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.18512414399999999 2024-09-13T11:25:15.434 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.434 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.434 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:25:15.434 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:06.531206+0000", 2024-09-13T11:25:15.434 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.4187999999999999e-05 2024-09-13T11:25:15.434 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.434 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.434 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:25:15.434 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:06.531316+0000", 2024-09-13T11:25:15.435 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000109675 2024-09-13T11:25:15.435 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.435 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.435 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:25:15.435 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:06.531330+0000", 2024-09-13T11:25:15.435 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.3947999999999999e-05 2024-09-13T11:25:15.435 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.435 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.435 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:25:15.435 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:06.532485+0000", 2024-09-13T11:25:15.435 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0011546250000000001 2024-09-13T11:25:15.435 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:15.435 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:25:15.435 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:15.436 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:15.436 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:25:15.436 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-09-13T11:25:15.437 INFO:tasks.ceph.osd.12.smithi001.stderr:2024-09-13T11:25:15.435+0000 7f892c6e6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26314) UID: 0 2024-09-13T11:25:15.438 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-09-13T11:25:15.538 INFO:tasks.ceph.osd.10.smithi162.stderr:2024-09-13T11:25:15.536+0000 7fb59c9f0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26317) UID: 0 2024-09-13T11:25:15.570 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-09-13T11:25:15.571 INFO:teuthology.orchestra.run.smithi092.stdout: "ops": [ 2024-09-13T11:25:15.571 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.571 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:162201 3.24 3:2593bc17:::benchmark_data_smithi177_31116_object21898:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e156)", 2024-09-13T11:25:15.571 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:25:15.568168+0000", 2024-09-13T11:25:15.571 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 0.0023547899999999998, 2024-09-13T11:25:15.571 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0023735840000000002, 2024-09-13T11:25:15.571 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:25:15.571 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:25:15.571 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "waiting for sub ops", 2024-09-13T11:25:15.571 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:25:15.571 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:25:15.571 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:15.572 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 162201 2024-09-13T11:25:15.572 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.572 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:25:15.572 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.572 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:25:15.572 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:15.568168+0000", 2024-09-13T11:25:15.572 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:15.572 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.572 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.572 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:25:15.572 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:15.568168+0000", 2024-09-13T11:25:15.572 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:15.572 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.572 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.573 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:25:15.573 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:15.568170+0000", 2024-09-13T11:25:15.573 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.889e-06 2024-09-13T11:25:15.573 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.573 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.573 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:25:15.573 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:15.568176+0000", 2024-09-13T11:25:15.573 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.7959999999999998e-06 2024-09-13T11:25:15.573 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.573 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.573 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:25:15.573 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:15.568191+0000", 2024-09-13T11:25:15.573 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.4378e-05 2024-09-13T11:25:15.574 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.574 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.574 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:25:15.574 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:15.568199+0000", 2024-09-13T11:25:15.574 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.5809999999999997e-06 2024-09-13T11:25:15.574 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.574 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.574 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:25:15.574 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:15.568301+0000", 2024-09-13T11:25:15.574 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000101573 2024-09-13T11:25:15.574 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.574 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.574 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:25:15.574 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:15.568447+0000", 2024-09-13T11:25:15.575 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00014599 2024-09-13T11:25:15.575 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.575 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.575 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1", 2024-09-13T11:25:15.575 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:15.568593+0000", 2024-09-13T11:25:15.575 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00014614900000000001 2024-09-13T11:25:15.575 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:15.575 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:25:15.575 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:15.575 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:15.575 INFO:teuthology.orchestra.run.smithi092.stdout: ], 2024-09-13T11:25:15.575 INFO:teuthology.orchestra.run.smithi092.stdout: "num_ops": 1 2024-09-13T11:25:15.576 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-09-13T11:25:15.586 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-09-13T11:25:15.638 INFO:tasks.ceph.osd.2.smithi162.stderr:2024-09-13T11:25:15.636+0000 7fe92b35a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26313) UID: 0 2024-09-13T11:25:15.728 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-09-13T11:25:15.728 INFO:teuthology.orchestra.run.smithi092.stdout: "ops": [], 2024-09-13T11:25:15.729 INFO:teuthology.orchestra.run.smithi092.stdout: "complaint_time": 30, 2024-09-13T11:25:15.729 INFO:teuthology.orchestra.run.smithi092.stdout: "num_blocked_ops": 0 2024-09-13T11:25:15.729 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-09-13T11:25:15.739 INFO:tasks.ceph.osd.15.smithi177.stderr:2024-09-13T11:25:15.734+0000 7f9d6ce77640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26578) UID: 0 2024-09-13T11:25:15.749 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-09-13T11:25:15.840 INFO:tasks.ceph.osd.15.smithi177.stderr:2024-09-13T11:25:15.838+0000 7f9d6ce77640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26578) UID: 0 2024-09-13T11:25:15.880 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-09-13T11:25:15.880 INFO:teuthology.orchestra.run.smithi092.stdout: "size": 20, 2024-09-13T11:25:15.880 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 600, 2024-09-13T11:25:15.881 INFO:teuthology.orchestra.run.smithi092.stdout: "ops": [ 2024-09-13T11:25:15.881 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.881 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "MOSDPGPush(3.19 76/70 [PushOp(3:9eacf3d0:::benchmark_data_smithi177_31116_object19747:head, version: 54'1236, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:9eacf3d0:::benchmark_data_smithi177_31116_object19747:head@54'1236, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:25:15.881 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:32.704676+0000", 2024-09-13T11:25:15.881 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 103.173761594, 2024-09-13T11:25:15.881 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.23525014899999999, 2024-09-13T11:25:15.881 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:25:15.881 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:25:15.881 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-09-13T11:25:15.881 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:25:15.881 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.882 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:25:15.882 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:32.704676+0000", 2024-09-13T11:25:15.882 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:15.882 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.882 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.882 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:25:15.882 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:32.704676+0000", 2024-09-13T11:25:15.882 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:15.882 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.882 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.882 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:25:15.882 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:32.704677+0000", 2024-09-13T11:25:15.882 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.531e-06 2024-09-13T11:25:15.882 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.883 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.883 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:25:15.883 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:32.704749+0000", 2024-09-13T11:25:15.883 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.1451000000000004e-05 2024-09-13T11:25:15.883 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.883 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.883 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:25:15.883 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:32.938053+0000", 2024-09-13T11:25:15.883 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.233304752 2024-09-13T11:25:15.883 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.883 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.883 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:25:15.883 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:32.938087+0000", 2024-09-13T11:25:15.883 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.4016000000000001e-05 2024-09-13T11:25:15.883 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.884 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.884 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:25:15.884 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:32.938351+0000", 2024-09-13T11:25:15.884 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00026397500000000001 2024-09-13T11:25:15.884 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.884 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.884 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:25:15.884 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:32.938381+0000", 2024-09-13T11:25:15.884 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.0001999999999998e-05 2024-09-13T11:25:15.884 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.884 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.884 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:25:15.884 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:32.939926+0000", 2024-09-13T11:25:15.884 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0015444219999999999 2024-09-13T11:25:15.885 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:15.885 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:25:15.885 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:15.885 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.885 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.885 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "MOSDPGPush(3.23 80/78 [PushOp(3:c59ecc6d:::benchmark_data_smithi177_31116_object17005:head, version: 52'1054, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:c59ecc6d:::benchmark_data_smithi177_31116_object17005:head@52'1054, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:25:15.885 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:37.433040+0000", 2024-09-13T11:25:15.885 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 98.445397618000001, 2024-09-13T11:25:15.885 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.26658219100000002, 2024-09-13T11:25:15.885 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:25:15.885 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:25:15.885 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-09-13T11:25:15.885 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:25:15.885 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.886 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:25:15.886 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:37.433040+0000", 2024-09-13T11:25:15.886 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:15.886 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.886 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.886 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:25:15.886 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:37.433040+0000", 2024-09-13T11:25:15.886 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:15.886 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.886 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.886 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:25:15.886 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:37.433042+0000", 2024-09-13T11:25:15.886 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.3999999999999999e-06 2024-09-13T11:25:15.886 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.887 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.887 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:25:15.887 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:37.433090+0000", 2024-09-13T11:25:15.887 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.8065999999999997e-05 2024-09-13T11:25:15.887 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.887 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.887 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:25:15.887 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:37.698797+0000", 2024-09-13T11:25:15.887 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.26570674500000002 2024-09-13T11:25:15.887 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.887 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.887 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:25:15.887 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:37.698827+0000", 2024-09-13T11:25:15.887 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.0505e-05 2024-09-13T11:25:15.888 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.888 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.888 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:25:15.888 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:37.698941+0000", 2024-09-13T11:25:15.888 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00011403799999999999 2024-09-13T11:25:15.888 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.888 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.888 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:25:15.888 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:37.698956+0000", 2024-09-13T11:25:15.888 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.4765e-05 2024-09-13T11:25:15.888 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.888 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.888 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:25:15.888 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:37.699622+0000", 2024-09-13T11:25:15.889 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00066567199999999997 2024-09-13T11:25:15.889 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:15.889 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:25:15.889 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:15.889 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.889 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.889 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "MOSDPGPush(3.1a 81/78 [PushOp(3:5a9c05b8:::benchmark_data_smithi177_31116_object4600:head, version: 44'280, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:5a9c05b8:::benchmark_data_smithi177_31116_object4600:head@44'280, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:25:15.889 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:38.261429+0000", 2024-09-13T11:25:15.889 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 97.617008182000006, 2024-09-13T11:25:15.889 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.13814789799999999, 2024-09-13T11:25:15.889 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:25:15.889 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:25:15.890 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-09-13T11:25:15.890 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:25:15.890 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.890 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:25:15.890 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:38.261429+0000", 2024-09-13T11:25:15.890 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:15.890 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.890 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.890 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:25:15.890 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:38.261429+0000", 2024-09-13T11:25:15.890 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:15.890 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.890 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.890 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:25:15.890 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:38.261430+0000", 2024-09-13T11:25:15.891 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.376e-06 2024-09-13T11:25:15.891 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.891 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.891 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:25:15.891 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:38.261501+0000", 2024-09-13T11:25:15.891 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.0724999999999998e-05 2024-09-13T11:25:15.891 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.891 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.891 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:25:15.891 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:38.398898+0000", 2024-09-13T11:25:15.891 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.13739662499999999 2024-09-13T11:25:15.891 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.891 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.891 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:25:15.892 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:38.398909+0000", 2024-09-13T11:25:15.892 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0926e-05 2024-09-13T11:25:15.892 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.892 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.892 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:25:15.892 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:38.398974+0000", 2024-09-13T11:25:15.892 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.5467000000000002e-05 2024-09-13T11:25:15.892 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.892 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.892 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:25:15.892 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:38.398982+0000", 2024-09-13T11:25:15.892 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.841e-06 2024-09-13T11:25:15.893 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.893 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.893 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:25:15.893 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:38.399577+0000", 2024-09-13T11:25:15.893 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00059493800000000004 2024-09-13T11:25:15.893 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:15.893 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:25:15.893 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:15.893 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.893 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.893 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "MOSDPGPush(3.1a 85/78 [PushOp(3:5dc85525:::benchmark_data_smithi177_31116_object11755:head, version: 49'723, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:5dc85525:::benchmark_data_smithi177_31116_object11755:head@49'723, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:25:15.893 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:42.422061+0000", 2024-09-13T11:25:15.893 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 93.456376164999995, 2024-09-13T11:25:15.894 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.52875617500000005, 2024-09-13T11:25:15.894 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:25:15.894 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:25:15.894 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-09-13T11:25:15.894 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:25:15.894 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.894 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:25:15.894 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.422061+0000", 2024-09-13T11:25:15.894 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:15.894 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.894 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.894 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:25:15.894 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.422061+0000", 2024-09-13T11:25:15.895 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:15.895 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.895 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.895 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:25:15.895 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.422064+0000", 2024-09-13T11:25:15.895 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.5409999999999999e-06 2024-09-13T11:25:15.895 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.895 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.895 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:25:15.895 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.422139+0000", 2024-09-13T11:25:15.895 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.5488999999999995e-05 2024-09-13T11:25:15.895 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.895 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.895 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:25:15.896 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.948781+0000", 2024-09-13T11:25:15.896 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.52664230999999995 2024-09-13T11:25:15.896 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.896 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.896 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:25:15.896 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.948841+0000", 2024-09-13T11:25:15.896 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.9450999999999997e-05 2024-09-13T11:25:15.896 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.896 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.896 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:25:15.896 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.949035+0000", 2024-09-13T11:25:15.896 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00019380600000000001 2024-09-13T11:25:15.896 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.897 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.897 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:25:15.897 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.949060+0000", 2024-09-13T11:25:15.897 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.5381999999999999e-05 2024-09-13T11:25:15.897 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.897 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.897 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:25:15.897 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.950817+0000", 2024-09-13T11:25:15.897 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0017571959999999999 2024-09-13T11:25:15.897 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:15.897 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:25:15.897 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:15.897 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.898 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.898 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "MOSDPGPush(3.1a 89/78 [PushOp(3:59270734:::benchmark_data_smithi177_31116_object20039:head, version: 54'1256, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:59270734:::benchmark_data_smithi177_31116_object20039:head@54'1256, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:25:15.898 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:46.607562+0000", 2024-09-13T11:25:15.898 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 89.270875387999993, 2024-09-13T11:25:15.898 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.32123945300000001, 2024-09-13T11:25:15.898 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:25:15.898 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:25:15.898 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-09-13T11:25:15.898 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:25:15.898 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.898 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:25:15.898 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.607562+0000", 2024-09-13T11:25:15.898 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:15.898 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.899 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.899 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:25:15.899 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.607562+0000", 2024-09-13T11:25:15.899 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:15.899 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.900 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.900 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:25:15.900 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.607564+0000", 2024-09-13T11:25:15.900 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.8610000000000001e-06 2024-09-13T11:25:15.900 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.900 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.900 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:25:15.900 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.607663+0000", 2024-09-13T11:25:15.900 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.9210999999999996e-05 2024-09-13T11:25:15.900 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.900 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.900 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:25:15.900 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.925193+0000", 2024-09-13T11:25:15.901 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.317530586 2024-09-13T11:25:15.901 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.901 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.901 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:25:15.901 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.925250+0000", 2024-09-13T11:25:15.901 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.6925000000000001e-05 2024-09-13T11:25:15.901 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.901 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.901 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:25:15.901 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.927013+0000", 2024-09-13T11:25:15.901 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.001762853 2024-09-13T11:25:15.901 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.901 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.902 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:25:15.902 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.927067+0000", 2024-09-13T11:25:15.902 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.3601999999999999e-05 2024-09-13T11:25:15.902 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.902 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.902 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:25:15.902 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.928801+0000", 2024-09-13T11:25:15.902 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.001734415 2024-09-13T11:25:15.902 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:15.902 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:25:15.902 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:15.902 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.903 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.903 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "MOSDPGPush(3.1a 91/78 [PushOp(3:5e5010d2:::benchmark_data_smithi177_31116_object25733:head, version: 58'1609, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:5e5010d2:::benchmark_data_smithi177_31116_object25733:head@58'1609, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:25:15.903 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:48.608485+0000", 2024-09-13T11:25:15.903 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 87.269952270000005, 2024-09-13T11:25:15.903 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.35035599699999997, 2024-09-13T11:25:15.903 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:25:15.903 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:25:15.903 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-09-13T11:25:15.903 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:25:15.903 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.903 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:25:15.903 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:48.608485+0000", 2024-09-13T11:25:15.903 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:15.904 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.904 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.904 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:25:15.904 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:48.608485+0000", 2024-09-13T11:25:15.904 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:15.904 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.904 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.904 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:25:15.904 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:48.608488+0000", 2024-09-13T11:25:15.904 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.9399999999999998e-06 2024-09-13T11:25:15.904 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.904 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.904 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:25:15.904 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:48.608552+0000", 2024-09-13T11:25:15.905 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.3731000000000004e-05 2024-09-13T11:25:15.905 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.905 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.905 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:25:15.905 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:48.957330+0000", 2024-09-13T11:25:15.905 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.34877806099999997 2024-09-13T11:25:15.905 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.905 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.905 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:25:15.905 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:48.957361+0000", 2024-09-13T11:25:15.905 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.0852000000000002e-05 2024-09-13T11:25:15.905 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.906 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.906 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:25:15.906 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:48.957544+0000", 2024-09-13T11:25:15.906 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000183537 2024-09-13T11:25:15.906 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.906 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.906 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:25:15.906 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:48.957582+0000", 2024-09-13T11:25:15.906 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.7472999999999998e-05 2024-09-13T11:25:15.906 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.906 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.906 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:25:15.906 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:48.958841+0000", 2024-09-13T11:25:15.906 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0012594030000000001 2024-09-13T11:25:15.906 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:15.907 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:25:15.907 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:15.907 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.907 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.907 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop(client.4665.0:72923 3.23 e97/90 3:c6230123:::benchmark_data_smithi177_31116_object72922:head v 97'3418, mlcod=97'3414)", 2024-09-13T11:25:15.907 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:55.047734+0000", 2024-09-13T11:25:15.907 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 80.830702752999997, 2024-09-13T11:25:15.907 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.20075528000000001, 2024-09-13T11:25:15.907 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:25:15.908 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:25:15.908 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:15.908 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:25:15.908 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.908 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:25:15.908 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:55.047734+0000", 2024-09-13T11:25:15.908 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:15.908 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.908 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.908 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:25:15.908 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:55.047734+0000", 2024-09-13T11:25:15.908 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:15.908 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.908 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.909 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:25:15.909 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:55.047736+0000", 2024-09-13T11:25:15.909 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.466e-06 2024-09-13T11:25:15.909 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.909 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.909 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:25:15.909 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:55.047772+0000", 2024-09-13T11:25:15.909 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.6576999999999999e-05 2024-09-13T11:25:15.909 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.909 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.909 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:25:15.909 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:55.246478+0000", 2024-09-13T11:25:15.909 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.19870581300000001 2024-09-13T11:25:15.909 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.910 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.910 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:25:15.910 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:55.246488+0000", 2024-09-13T11:25:15.910 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.5419999999999994e-06 2024-09-13T11:25:15.910 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.910 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.910 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:25:15.910 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:55.246631+0000", 2024-09-13T11:25:15.910 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00014287300000000001 2024-09-13T11:25:15.910 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.911 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.911 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:25:15.911 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:55.246665+0000", 2024-09-13T11:25:15.911 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.4684999999999997e-05 2024-09-13T11:25:15.911 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.911 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.911 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:25:15.911 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:55.248453+0000", 2024-09-13T11:25:15.911 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0017879459999999999 2024-09-13T11:25:15.911 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.911 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.911 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:25:15.912 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:55.248490+0000", 2024-09-13T11:25:15.912 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.6378000000000003e-05 2024-09-13T11:25:15.912 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:15.912 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:25:15.912 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:15.912 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.912 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.912 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:79823 3.19 3:9e038f3c:::benchmark_data_smithi177_31116_object79822:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102)", 2024-09-13T11:25:15.912 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:24:00.563188+0000", 2024-09-13T11:25:15.912 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 75.315249128000005, 2024-09-13T11:25:15.912 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.164156422, 2024-09-13T11:25:15.912 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:25:15.912 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:25:15.912 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:15.913 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:25:15.913 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:25:15.913 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:15.913 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 79823 2024-09-13T11:25:15.913 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.913 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:25:15.913 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.913 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:25:15.913 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:00.563188+0000", 2024-09-13T11:25:15.913 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:15.913 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.913 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.913 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:25:15.913 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:00.563188+0000", 2024-09-13T11:25:15.914 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:15.914 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.914 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.914 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:25:15.914 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:00.563201+0000", 2024-09-13T11:25:15.914 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.3183000000000001e-05 2024-09-13T11:25:15.914 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.914 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.914 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:25:15.914 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:00.563263+0000", 2024-09-13T11:25:15.914 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.1904999999999998e-05 2024-09-13T11:25:15.914 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.914 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.915 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:25:15.915 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:00.563297+0000", 2024-09-13T11:25:15.915 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.3979000000000001e-05 2024-09-13T11:25:15.915 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.915 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.915 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:25:15.915 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:00.563318+0000", 2024-09-13T11:25:15.915 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.0703000000000002e-05 2024-09-13T11:25:15.915 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.915 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.915 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:25:15.915 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:00.563513+0000", 2024-09-13T11:25:15.915 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00019524200000000001 2024-09-13T11:25:15.915 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.916 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.916 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:25:15.916 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:00.563758+0000", 2024-09-13T11:25:15.916 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000244711 2024-09-13T11:25:15.916 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.916 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.916 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1", 2024-09-13T11:25:15.916 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:00.563978+0000", 2024-09-13T11:25:15.916 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00022011700000000001 2024-09-13T11:25:15.916 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.916 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.916 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:25:15.916 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:00.566893+0000", 2024-09-13T11:25:15.916 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0029147650000000002 2024-09-13T11:25:15.917 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.917 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.917 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:25:15.917 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:00.727176+0000", 2024-09-13T11:25:15.917 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.16028374100000001 2024-09-13T11:25:15.917 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.917 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.917 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:25:15.917 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:00.727271+0000", 2024-09-13T11:25:15.917 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.4481000000000006e-05 2024-09-13T11:25:15.917 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.917 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.917 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:25:15.917 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:00.727344+0000", 2024-09-13T11:25:15.917 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.3595000000000003e-05 2024-09-13T11:25:15.918 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:15.918 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:25:15.918 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:15.918 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.918 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.918 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop_reply(client.4665.0:79823 3.19 e102/84 ondisk, result = 0)", 2024-09-13T11:25:15.918 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:24:00.567585+0000", 2024-09-13T11:25:15.918 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 75.310851817, 2024-09-13T11:25:15.918 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.15977266400000001, 2024-09-13T11:25:15.918 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:25:15.918 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:25:15.918 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-09-13T11:25:15.918 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:25:15.918 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.919 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:25:15.919 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:00.567585+0000", 2024-09-13T11:25:15.919 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:15.919 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.919 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.919 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:25:15.919 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:00.567585+0000", 2024-09-13T11:25:15.919 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:15.919 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.919 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.919 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:25:15.919 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:00.567589+0000", 2024-09-13T11:25:15.919 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.2969999999999999e-06 2024-09-13T11:25:15.920 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.920 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.920 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:25:15.920 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:00.567599+0000", 2024-09-13T11:25:15.920 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0142999999999999e-05 2024-09-13T11:25:15.920 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.920 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.920 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:25:15.920 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:00.726959+0000", 2024-09-13T11:25:15.920 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.15936018399999999 2024-09-13T11:25:15.920 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.920 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.920 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:25:15.921 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:00.726972+0000", 2024-09-13T11:25:15.921 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.2622999999999999e-05 2024-09-13T11:25:15.921 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.921 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.921 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:25:15.921 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:00.727151+0000", 2024-09-13T11:25:15.921 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00017909600000000001 2024-09-13T11:25:15.921 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.921 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.921 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:25:15.921 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:00.727167+0000", 2024-09-13T11:25:15.921 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.6639000000000001e-05 2024-09-13T11:25:15.921 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.921 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.921 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:25:15.922 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:00.727358+0000", 2024-09-13T11:25:15.922 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00019068199999999999 2024-09-13T11:25:15.922 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:15.922 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:25:15.922 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:15.922 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.922 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.922 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:98229 3.19 3:98f5bd4f:::benchmark_data_smithi177_31116_object98228:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113)", 2024-09-13T11:25:15.922 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:24:15.139305+0000", 2024-09-13T11:25:15.922 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 60.739131905000001, 2024-09-13T11:25:15.922 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.116582746, 2024-09-13T11:25:15.922 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:25:15.922 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:25:15.923 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:15.923 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:25:15.923 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:25:15.923 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:15.923 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 98229 2024-09-13T11:25:15.923 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.923 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:25:15.923 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.923 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:25:15.923 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:15.139305+0000", 2024-09-13T11:25:15.923 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:15.923 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.923 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.923 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:25:15.924 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:15.139305+0000", 2024-09-13T11:25:15.924 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:15.924 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.924 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.924 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:25:15.924 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:15.139311+0000", 2024-09-13T11:25:15.924 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.4299999999999997e-06 2024-09-13T11:25:15.924 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.924 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.924 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:25:15.924 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:15.139427+0000", 2024-09-13T11:25:15.924 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000116258 2024-09-13T11:25:15.924 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.925 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.925 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:25:15.925 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:15.139458+0000", 2024-09-13T11:25:15.925 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.0892000000000002e-05 2024-09-13T11:25:15.925 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.925 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.925 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:25:15.925 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:15.139477+0000", 2024-09-13T11:25:15.925 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.9544000000000001e-05 2024-09-13T11:25:15.925 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.925 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.925 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:25:15.925 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:15.139609+0000", 2024-09-13T11:25:15.926 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00013150799999999999 2024-09-13T11:25:15.926 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.926 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.926 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:25:15.926 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:15.139873+0000", 2024-09-13T11:25:15.926 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000264288 2024-09-13T11:25:15.926 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.926 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.926 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1", 2024-09-13T11:25:15.926 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:15.140160+0000", 2024-09-13T11:25:15.926 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00028688999999999999 2024-09-13T11:25:15.926 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.926 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.926 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:25:15.926 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:15.143356+0000", 2024-09-13T11:25:15.927 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0031956179999999999 2024-09-13T11:25:15.927 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.927 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.927 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:25:15.927 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:15.255687+0000", 2024-09-13T11:25:15.927 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.11233159099999999 2024-09-13T11:25:15.927 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.927 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.927 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:25:15.927 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:15.255799+0000", 2024-09-13T11:25:15.927 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000111492 2024-09-13T11:25:15.927 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.927 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.928 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:25:15.928 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:15.255888+0000", 2024-09-13T11:25:15.928 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.9234999999999995e-05 2024-09-13T11:25:15.928 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:15.928 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:25:15.928 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:15.928 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.928 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.928 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop(client.4665.0:98582 3.1a e113/105 3:59f9cd75:::benchmark_data_smithi177_31116_object98581:head v 113'4743, mlcod=113'4741)", 2024-09-13T11:25:15.928 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:24:15.432027+0000", 2024-09-13T11:25:15.928 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 60.446410137999997, 2024-09-13T11:25:15.928 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.196879784, 2024-09-13T11:25:15.928 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:25:15.928 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:25:15.929 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:15.929 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:25:15.929 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.929 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:25:15.929 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:15.432027+0000", 2024-09-13T11:25:15.929 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:15.929 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.929 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.929 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:25:15.929 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:15.432027+0000", 2024-09-13T11:25:15.929 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:15.929 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.929 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.929 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:25:15.930 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:15.432029+0000", 2024-09-13T11:25:15.930 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.838e-06 2024-09-13T11:25:15.930 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.930 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.930 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:25:15.930 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:15.432088+0000", 2024-09-13T11:25:15.930 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.8767999999999997e-05 2024-09-13T11:25:15.930 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.930 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.930 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:25:15.930 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:15.624731+0000", 2024-09-13T11:25:15.930 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.192643221 2024-09-13T11:25:15.930 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.930 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.931 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:25:15.931 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:15.624746+0000", 2024-09-13T11:25:15.931 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.4705000000000001e-05 2024-09-13T11:25:15.931 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.931 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.931 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:25:15.931 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:15.624882+0000", 2024-09-13T11:25:15.931 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00013602300000000001 2024-09-13T11:25:15.931 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.931 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.931 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:25:15.931 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:15.624941+0000", 2024-09-13T11:25:15.931 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.9351000000000001e-05 2024-09-13T11:25:15.932 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.932 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.932 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:25:15.932 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:15.628836+0000", 2024-09-13T11:25:15.932 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0038955109999999999 2024-09-13T11:25:15.932 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.932 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.932 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:25:15.932 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:15.628907+0000", 2024-09-13T11:25:15.932 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.0366999999999999e-05 2024-09-13T11:25:15.932 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:15.932 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:25:15.932 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:15.933 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.933 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.933 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop(client.4665.0:98872 3.1a e113/105 3:5b894ab8:::benchmark_data_smithi177_31116_object98871:head v 113'4746, mlcod=113'4743)", 2024-09-13T11:25:15.933 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:24:15.736076+0000", 2024-09-13T11:25:15.933 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 60.142361501000003, 2024-09-13T11:25:15.933 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.111852691, 2024-09-13T11:25:15.933 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:25:15.933 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:25:15.933 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:15.933 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:25:15.933 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.933 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:25:15.933 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:15.736076+0000", 2024-09-13T11:25:15.934 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:15.934 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.934 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.934 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:25:15.934 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:15.736076+0000", 2024-09-13T11:25:15.934 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:15.934 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.934 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.934 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:25:15.934 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:15.736079+0000", 2024-09-13T11:25:15.934 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.2260000000000001e-06 2024-09-13T11:25:15.934 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.934 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.934 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:25:15.935 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:15.736171+0000", 2024-09-13T11:25:15.935 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.2260000000000001e-05 2024-09-13T11:25:15.935 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.935 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.935 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:25:15.935 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:15.843102+0000", 2024-09-13T11:25:15.935 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.10693101100000001 2024-09-13T11:25:15.935 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.935 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.935 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:25:15.935 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:15.843120+0000", 2024-09-13T11:25:15.935 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.7771999999999999e-05 2024-09-13T11:25:15.935 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.935 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.936 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:25:15.936 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:15.843247+0000", 2024-09-13T11:25:15.936 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00012656000000000001 2024-09-13T11:25:15.936 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.936 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.936 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:25:15.936 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:15.843295+0000", 2024-09-13T11:25:15.936 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.8516000000000001e-05 2024-09-13T11:25:15.936 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.936 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.936 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:25:15.936 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:15.847762+0000", 2024-09-13T11:25:15.936 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0044674149999999998 2024-09-13T11:25:15.936 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.937 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.937 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:25:15.937 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:15.847928+0000", 2024-09-13T11:25:15.937 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00016593099999999999 2024-09-13T11:25:15.937 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:15.937 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:25:15.937 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:15.937 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.937 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.937 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop(client.4665.0:99105 3.1a e113/105 3:5ac5090c:::benchmark_data_smithi177_31116_object99104:head v 113'4749, mlcod=113'4746)", 2024-09-13T11:25:15.937 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:24:15.963602+0000", 2024-09-13T11:25:15.937 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 59.914834956, 2024-09-13T11:25:15.937 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.23699906300000001, 2024-09-13T11:25:15.938 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:25:15.938 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:25:15.938 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:15.938 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:25:15.938 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.938 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:25:15.938 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:15.963602+0000", 2024-09-13T11:25:15.938 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:15.938 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.938 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.938 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:25:15.938 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:15.963602+0000", 2024-09-13T11:25:15.939 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:15.939 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.939 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.939 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:25:15.939 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:15.963605+0000", 2024-09-13T11:25:15.939 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.0400000000000001e-06 2024-09-13T11:25:15.939 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.939 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.939 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:25:15.939 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:15.963718+0000", 2024-09-13T11:25:15.939 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000112796 2024-09-13T11:25:15.939 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.939 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.940 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:25:15.940 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:16.197994+0000", 2024-09-13T11:25:15.940 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.234276285 2024-09-13T11:25:15.940 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.940 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.940 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:25:15.940 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:16.198004+0000", 2024-09-13T11:25:15.940 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.5559999999999996e-06 2024-09-13T11:25:15.940 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.940 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.940 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:25:15.940 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:16.198110+0000", 2024-09-13T11:25:15.940 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00010614 2024-09-13T11:25:15.940 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.941 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.941 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:25:15.941 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:16.198152+0000", 2024-09-13T11:25:15.941 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.1899e-05 2024-09-13T11:25:15.941 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.941 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.941 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:25:15.941 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:16.200528+0000", 2024-09-13T11:25:15.941 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00237587 2024-09-13T11:25:15.941 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.941 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.942 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:25:15.942 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:16.200601+0000", 2024-09-13T11:25:15.942 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.3477000000000001e-05 2024-09-13T11:25:15.942 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:15.942 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:25:15.942 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:15.942 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.942 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.942 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop(client.4665.0:119446 3.1a e128/105 3:5b664280:::benchmark_data_smithi177_31116_object119445:head v 128'5409, mlcod=128'5405)", 2024-09-13T11:25:15.942 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:24:37.343796+0000", 2024-09-13T11:25:15.942 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 38.534641460000003, 2024-09-13T11:25:15.942 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.118847789, 2024-09-13T11:25:15.942 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:25:15.942 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:25:15.943 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:15.943 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:25:15.943 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.943 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:25:15.943 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:37.343796+0000", 2024-09-13T11:25:15.943 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:15.943 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.943 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.943 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:25:15.943 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:37.343796+0000", 2024-09-13T11:25:15.943 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:15.943 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.944 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.944 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:25:15.944 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:37.343799+0000", 2024-09-13T11:25:15.944 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.8590000000000001e-06 2024-09-13T11:25:15.944 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.944 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.944 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:25:15.944 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:37.343848+0000", 2024-09-13T11:25:15.944 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.8930000000000001e-05 2024-09-13T11:25:15.944 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.944 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.944 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:25:15.944 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:37.460590+0000", 2024-09-13T11:25:15.945 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.116742045 2024-09-13T11:25:15.945 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.945 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.945 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:25:15.945 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:37.460596+0000", 2024-09-13T11:25:15.945 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.4559999999999997e-06 2024-09-13T11:25:15.945 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.945 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.945 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:25:15.945 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:37.460645+0000", 2024-09-13T11:25:15.945 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.8550000000000001e-05 2024-09-13T11:25:15.945 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.945 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.946 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:25:15.946 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:37.460660+0000", 2024-09-13T11:25:15.946 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.5662e-05 2024-09-13T11:25:15.946 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.946 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.946 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:25:15.946 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:37.462612+0000", 2024-09-13T11:25:15.946 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0019521499999999999 2024-09-13T11:25:15.946 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.946 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.946 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:25:15.947 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:37.462644+0000", 2024-09-13T11:25:15.947 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.1136999999999997e-05 2024-09-13T11:25:15.947 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:15.947 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:25:15.947 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:15.947 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.947 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.947 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop(client.4665.0:119844 3.1a e128/105 3:5858dd7d:::benchmark_data_smithi177_31116_object119843:head v 128'5419, mlcod=128'5416)", 2024-09-13T11:25:15.947 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:24:37.692299+0000", 2024-09-13T11:25:15.947 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 38.186138030999999, 2024-09-13T11:25:15.947 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.12845056299999999, 2024-09-13T11:25:15.947 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:25:15.947 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:25:15.947 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:15.948 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:25:15.948 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.948 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:25:15.948 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:37.692299+0000", 2024-09-13T11:25:15.948 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:15.948 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.948 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.948 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:25:15.948 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:37.692299+0000", 2024-09-13T11:25:15.948 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:15.948 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.948 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.948 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:25:15.949 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:37.692301+0000", 2024-09-13T11:25:15.949 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.6300000000000001e-06 2024-09-13T11:25:15.949 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.949 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.949 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:25:15.949 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:37.692352+0000", 2024-09-13T11:25:15.949 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.1388000000000003e-05 2024-09-13T11:25:15.949 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.949 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.949 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:25:15.949 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:37.817572+0000", 2024-09-13T11:25:15.949 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.12522014300000001 2024-09-13T11:25:15.949 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.950 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.950 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:25:15.950 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:37.817579+0000", 2024-09-13T11:25:15.950 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.8549999999999996e-06 2024-09-13T11:25:15.950 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.950 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.950 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:25:15.950 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:37.817641+0000", 2024-09-13T11:25:15.950 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.2092000000000001e-05 2024-09-13T11:25:15.950 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.950 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.950 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:25:15.950 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:37.817659+0000", 2024-09-13T11:25:15.950 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.7849000000000001e-05 2024-09-13T11:25:15.951 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.951 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.951 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:25:15.951 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:37.820701+0000", 2024-09-13T11:25:15.951 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0030416459999999998 2024-09-13T11:25:15.951 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.951 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.951 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:25:15.951 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:37.820750+0000", 2024-09-13T11:25:15.951 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.8959999999999999e-05 2024-09-13T11:25:15.951 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:15.951 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:25:15.951 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:15.952 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.952 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.952 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:127499 3.19 3:9b8540f0:::benchmark_data_smithi177_31116_object127498:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e131)", 2024-09-13T11:25:15.952 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:24:44.319661+0000", 2024-09-13T11:25:15.952 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 31.558775743999998, 2024-09-13T11:25:15.952 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.116384459, 2024-09-13T11:25:15.952 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:25:15.952 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:25:15.952 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:15.952 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:25:15.952 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:25:15.952 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:15.952 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 127499 2024-09-13T11:25:15.952 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.953 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:25:15.953 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.953 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:25:15.953 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:44.319661+0000", 2024-09-13T11:25:15.953 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:15.953 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.953 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.953 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:25:15.953 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:44.319661+0000", 2024-09-13T11:25:15.953 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:15.953 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.953 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.954 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:25:15.954 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:44.319664+0000", 2024-09-13T11:25:15.954 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.835e-06 2024-09-13T11:25:15.954 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.954 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.954 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:25:15.954 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:44.319706+0000", 2024-09-13T11:25:15.954 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.2035e-05 2024-09-13T11:25:15.954 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.954 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.954 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:25:15.954 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:44.319725+0000", 2024-09-13T11:25:15.954 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.9004999999999998e-05 2024-09-13T11:25:15.955 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.955 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.955 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:25:15.955 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:44.319735+0000", 2024-09-13T11:25:15.955 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.842e-06 2024-09-13T11:25:15.955 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.955 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.955 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:25:15.955 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:44.319813+0000", 2024-09-13T11:25:15.955 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.7713000000000006e-05 2024-09-13T11:25:15.955 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.955 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.955 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:25:15.955 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:44.319967+0000", 2024-09-13T11:25:15.955 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00015394399999999999 2024-09-13T11:25:15.956 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.956 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.956 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1", 2024-09-13T11:25:15.956 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:44.320103+0000", 2024-09-13T11:25:15.956 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00013657900000000001 2024-09-13T11:25:15.956 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.956 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.956 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:25:15.956 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:44.324061+0000", 2024-09-13T11:25:15.956 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0039573359999999997 2024-09-13T11:25:15.956 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.957 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.957 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:25:15.957 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:44.435934+0000", 2024-09-13T11:25:15.957 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.111873032 2024-09-13T11:25:15.957 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.957 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.957 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:25:15.957 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:44.435999+0000", 2024-09-13T11:25:15.957 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.4820999999999995e-05 2024-09-13T11:25:15.957 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.957 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.957 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:25:15.958 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:44.436046+0000", 2024-09-13T11:25:15.958 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.7317000000000001e-05 2024-09-13T11:25:15.958 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:15.958 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:25:15.958 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:15.958 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.958 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.958 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop(client.4665.0:129921 3.1a e134/105 3:5fb9088b:::benchmark_data_smithi177_31116_object129920:head v 134'5758, mlcod=134'5755)", 2024-09-13T11:25:15.958 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:24:47.178021+0000", 2024-09-13T11:25:15.958 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 28.70041595, 2024-09-13T11:25:15.958 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.33723278600000001, 2024-09-13T11:25:15.958 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:25:15.959 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:25:15.959 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:15.959 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:25:15.959 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.959 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:25:15.959 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:47.178021+0000", 2024-09-13T11:25:15.959 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:15.959 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.959 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.959 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:25:15.959 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:47.178021+0000", 2024-09-13T11:25:15.959 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:15.959 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.959 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.960 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:25:15.960 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:47.178024+0000", 2024-09-13T11:25:15.960 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.3609999999999999e-06 2024-09-13T11:25:15.960 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.960 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.960 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:25:15.960 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:47.178102+0000", 2024-09-13T11:25:15.960 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.8325000000000006e-05 2024-09-13T11:25:15.960 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.960 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.960 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:25:15.960 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:47.512707+0000", 2024-09-13T11:25:15.960 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.33460516000000001 2024-09-13T11:25:15.961 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.961 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.961 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:25:15.961 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:47.512716+0000", 2024-09-13T11:25:15.961 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.0389999999999993e-06 2024-09-13T11:25:15.961 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.961 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.961 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:25:15.961 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:47.512788+0000", 2024-09-13T11:25:15.961 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.1840000000000003e-05 2024-09-13T11:25:15.962 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.962 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.962 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:25:15.962 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:47.512809+0000", 2024-09-13T11:25:15.962 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.1005e-05 2024-09-13T11:25:15.962 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.962 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.962 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:25:15.962 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:47.515182+0000", 2024-09-13T11:25:15.962 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0023731939999999999 2024-09-13T11:25:15.962 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.962 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.962 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:25:15.963 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:47.515254+0000", 2024-09-13T11:25:15.963 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.1861999999999997e-05 2024-09-13T11:25:15.963 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:15.963 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:25:15.963 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:15.963 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.963 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.963 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop(client.4665.0:138370 3.23 e140/90 3:c5730cd2:::benchmark_data_smithi177_31116_object138369:head v 140'4458, mlcod=140'4455)", 2024-09-13T11:25:15.963 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:24:54.416118+0000", 2024-09-13T11:25:15.963 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 21.462319472000001, 2024-09-13T11:25:15.963 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.244024991, 2024-09-13T11:25:15.963 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:25:15.963 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:25:15.964 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:15.964 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:25:15.964 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.964 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:25:15.964 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:54.416118+0000", 2024-09-13T11:25:15.964 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:15.964 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.964 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.964 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:25:15.964 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:54.416118+0000", 2024-09-13T11:25:15.964 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:15.964 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.964 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.964 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:25:15.965 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:54.416120+0000", 2024-09-13T11:25:15.965 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.6280000000000001e-06 2024-09-13T11:25:15.965 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.965 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.965 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:25:15.965 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:54.416161+0000", 2024-09-13T11:25:15.965 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.0877000000000002e-05 2024-09-13T11:25:15.965 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.965 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.965 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:25:15.965 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:54.657128+0000", 2024-09-13T11:25:15.965 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.24096696300000001 2024-09-13T11:25:15.965 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.966 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.966 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:25:15.966 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:54.657142+0000", 2024-09-13T11:25:15.966 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.358e-05 2024-09-13T11:25:15.966 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.966 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.966 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:25:15.966 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:54.657290+0000", 2024-09-13T11:25:15.966 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00014794400000000001 2024-09-13T11:25:15.966 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.966 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.966 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:25:15.966 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:54.657325+0000", 2024-09-13T11:25:15.967 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.5623000000000001e-05 2024-09-13T11:25:15.967 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.967 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.967 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:25:15.967 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:54.660079+0000", 2024-09-13T11:25:15.967 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0027536489999999999 2024-09-13T11:25:15.967 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.967 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.967 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:25:15.967 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:54.660143+0000", 2024-09-13T11:25:15.967 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.3726999999999995e-05 2024-09-13T11:25:15.967 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:15.967 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:25:15.967 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:15.967 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.968 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.968 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:154584 3.19 3:9c2bc229:::benchmark_data_smithi177_31116_object14281:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e151)", 2024-09-13T11:25:15.968 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:25:09.527283+0000", 2024-09-13T11:25:15.968 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 6.3511539690000003, 2024-09-13T11:25:15.968 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.120590691, 2024-09-13T11:25:15.968 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:25:15.968 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:25:15.968 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:15.968 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:25:15.968 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:25:15.968 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:15.968 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 154584 2024-09-13T11:25:15.968 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.969 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:25:15.969 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.969 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:25:15.969 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:09.527283+0000", 2024-09-13T11:25:15.969 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:15.969 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.969 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.969 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:25:15.969 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:09.527283+0000", 2024-09-13T11:25:15.969 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:15.969 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.969 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.969 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:25:15.969 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:09.527286+0000", 2024-09-13T11:25:15.970 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.926e-06 2024-09-13T11:25:15.970 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.970 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.970 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:25:15.970 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:09.527294+0000", 2024-09-13T11:25:15.970 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.9249999999999995e-06 2024-09-13T11:25:15.970 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.970 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.970 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:25:15.970 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:09.527315+0000", 2024-09-13T11:25:15.970 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.0639e-05 2024-09-13T11:25:15.970 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.970 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.970 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:25:15.971 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:09.527326+0000", 2024-09-13T11:25:15.971 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.1321999999999999e-05 2024-09-13T11:25:15.971 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.971 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.971 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:25:15.971 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:09.527406+0000", 2024-09-13T11:25:15.971 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.0184000000000003e-05 2024-09-13T11:25:15.971 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.971 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.971 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:25:15.971 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:09.527571+0000", 2024-09-13T11:25:15.971 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00016465300000000001 2024-09-13T11:25:15.971 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.971 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.972 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1", 2024-09-13T11:25:15.972 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:09.527694+0000", 2024-09-13T11:25:15.972 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00012302400000000001 2024-09-13T11:25:15.972 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.972 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.972 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:25:15.972 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:09.530208+0000", 2024-09-13T11:25:15.972 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0025140050000000001 2024-09-13T11:25:15.972 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.972 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.972 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:25:15.972 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:09.647770+0000", 2024-09-13T11:25:15.973 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.117561969 2024-09-13T11:25:15.973 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.973 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.973 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:25:15.973 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:09.647827+0000", 2024-09-13T11:25:15.973 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.7447000000000001e-05 2024-09-13T11:25:15.973 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.973 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.973 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:25:15.973 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:09.647874+0000", 2024-09-13T11:25:15.973 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.6597000000000001e-05 2024-09-13T11:25:15.973 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:15.973 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:25:15.973 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:15.974 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.974 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.974 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop_reply(client.4665.0:154584 3.19 e151/84 ondisk, result = 0)", 2024-09-13T11:25:15.974 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:25:09.530631+0000", 2024-09-13T11:25:15.974 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 6.3478060520000001, 2024-09-13T11:25:15.974 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.11725072, 2024-09-13T11:25:15.974 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:25:15.974 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:25:15.974 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-09-13T11:25:15.974 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:25:15.974 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.974 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:25:15.974 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:09.530631+0000", 2024-09-13T11:25:15.975 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:15.975 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.975 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.975 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:25:15.975 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:09.530631+0000", 2024-09-13T11:25:15.975 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:15.975 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.975 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.975 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:25:15.975 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:09.530633+0000", 2024-09-13T11:25:15.975 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.8390000000000001e-06 2024-09-13T11:25:15.975 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.975 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.975 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:25:15.976 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:09.530640+0000", 2024-09-13T11:25:15.976 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.5560000000000004e-06 2024-09-13T11:25:15.976 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.976 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.976 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:25:15.976 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:09.647658+0000", 2024-09-13T11:25:15.976 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.117018048 2024-09-13T11:25:15.976 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.976 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.976 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:25:15.976 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:09.647666+0000", 2024-09-13T11:25:15.976 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.4200000000000007e-06 2024-09-13T11:25:15.976 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.977 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.977 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:25:15.977 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:09.647752+0000", 2024-09-13T11:25:15.977 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.6335000000000006e-05 2024-09-13T11:25:15.977 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.977 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.977 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:25:15.977 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:09.647764+0000", 2024-09-13T11:25:15.977 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.1240000000000001e-05 2024-09-13T11:25:15.977 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:15.977 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:15.977 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:25:15.977 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:09.647882+0000", 2024-09-13T11:25:15.977 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000118282 2024-09-13T11:25:15.978 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:15.978 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:25:15.978 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:15.978 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:15.978 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:25:15.978 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-09-13T11:25:15.980 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_ops_in_flight 2024-09-13T11:25:15.981 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-13T11:25:15.975+0000 7f6a68775640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26307) UID: 0 2024-09-13T11:25:16.085 INFO:tasks.ceph.osd.14.smithi162.stderr:2024-09-13T11:25:16.080+0000 7f3df5903640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33154) UID: 0 2024-09-13T11:25:16.116 INFO:teuthology.orchestra.run.smithi001.stderr:reweighted osd.5 to 1 (10000) 2024-09-13T11:25:16.121 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-09-13T11:25:16.121 INFO:teuthology.orchestra.run.smithi092.stdout: "ops": [], 2024-09-13T11:25:16.121 INFO:teuthology.orchestra.run.smithi092.stdout: "num_ops": 0 2024-09-13T11:25:16.121 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-09-13T11:25:16.122 INFO:tasks.ceph.osd.14.smithi162.stderr:2024-09-13T11:25:16.120+0000 7f3dea330640 -1 osd.14 156 handle_osd_map injecting map crc failure 2024-09-13T11:25:16.130 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-13T11:25:16.140 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_blocked_ops 2024-09-13T11:25:16.141 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 6 1 2024-09-13T11:25:16.182 INFO:tasks.ceph.osd.3.smithi177.stderr:2024-09-13T11:25:16.178+0000 7f10adba4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26574) UID: 0 2024-09-13T11:25:16.279 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-09-13T11:25:16.279 INFO:teuthology.orchestra.run.smithi092.stdout: "ops": [], 2024-09-13T11:25:16.279 INFO:teuthology.orchestra.run.smithi092.stdout: "complaint_time": 30, 2024-09-13T11:25:16.279 INFO:teuthology.orchestra.run.smithi092.stdout: "num_blocked_ops": 0 2024-09-13T11:25:16.279 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-09-13T11:25:16.281 INFO:tasks.ceph.osd.3.smithi177.stderr:2024-09-13T11:25:16.278+0000 7f10adba4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26574) UID: 0 2024-09-13T11:25:16.299 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_historic_ops 2024-09-13T11:25:16.382 INFO:tasks.ceph.osd.3.smithi177.stderr:2024-09-13T11:25:16.378+0000 7f10adba4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26574) UID: 0 2024-09-13T11:25:16.433 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-09-13T11:25:16.434 INFO:teuthology.orchestra.run.smithi092.stdout: "size": 20, 2024-09-13T11:25:16.434 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 600, 2024-09-13T11:25:16.434 INFO:teuthology.orchestra.run.smithi092.stdout: "ops": [ 2024-09-13T11:25:16.434 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:16.434 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:79 3.1 3:841df5b5:::benchmark_data_smithi177_31116_object78:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-09-13T11:25:16.434 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:27.056083+0000", 2024-09-13T11:25:16.434 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 169.37535580400001, 2024-09-13T11:25:16.434 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.89469730199999997, 2024-09-13T11:25:16.434 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:25:16.434 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:25:16.434 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:16.435 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:25:16.435 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:25:16.435 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:16.435 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 79 2024-09-13T11:25:16.435 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:16.435 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:25:16.435 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:16.435 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:25:16.435 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.056083+0000", 2024-09-13T11:25:16.435 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:16.435 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:16.435 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:16.436 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:25:16.436 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.056083+0000", 2024-09-13T11:25:16.436 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:16.436 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:16.436 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:16.436 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:25:16.436 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.056087+0000", 2024-09-13T11:25:16.436 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.0659999999999997e-06 2024-09-13T11:25:16.436 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:16.436 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:16.436 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:25:16.436 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.056148+0000", 2024-09-13T11:25:16.436 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.1376000000000003e-05 2024-09-13T11:25:16.437 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:16.437 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:16.437 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:25:16.437 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.056180+0000", 2024-09-13T11:25:16.437 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.2370999999999999e-05 2024-09-13T11:25:16.437 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:16.437 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:16.437 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:25:16.437 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.056195+0000", 2024-09-13T11:25:16.437 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.5119e-05 2024-09-13T11:25:16.437 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:16.437 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:16.437 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:25:16.438 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.056304+0000", 2024-09-13T11:25:16.438 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000108365 2024-09-13T11:25:16.438 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:16.438 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:16.438 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:25:16.438 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.056478+0000", 2024-09-13T11:25:16.438 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000174542 2024-09-13T11:25:16.438 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:16.438 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:16.438 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 11", 2024-09-13T11:25:16.438 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.056690+0000", 2024-09-13T11:25:16.438 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00021167500000000001 2024-09-13T11:25:16.439 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:16.439 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:16.439 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:25:16.439 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.058063+0000", 2024-09-13T11:25:16.439 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0013731819999999999 2024-09-13T11:25:16.439 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:16.439 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:16.439 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:25:16.439 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.950477+0000", 2024-09-13T11:25:16.439 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.89241369400000004 2024-09-13T11:25:16.439 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:16.439 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:16.440 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:25:16.440 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.950652+0000", 2024-09-13T11:25:16.440 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00017526400000000001 2024-09-13T11:25:16.440 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:16.440 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:16.440 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:25:16.440 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.950780+0000", 2024-09-13T11:25:16.440 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00012764800000000001 2024-09-13T11:25:16.440 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:16.440 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:25:16.440 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:16.440 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:16.441 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:16.441 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:7 3.b 3:de78393b:::benchmark_data_smithi177_31116_object6:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T11:25:16.441 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:48.313952+0000", 2024-09-13T11:25:16.441 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 148.117486055, 2024-09-13T11:25:16.441 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.86421288600000001, 2024-09-13T11:25:16.441 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:25:16.441 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:25:16.441 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:16.441 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:25:16.441 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:25:16.441 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:16.441 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 7 2024-09-13T11:25:16.441 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:16.441 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:25:16.442 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:16.442 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:25:16.442 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.313952+0000", 2024-09-13T11:25:16.442 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:16.442 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:16.442 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:16.442 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:25:16.442 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.313952+0000", 2024-09-13T11:25:16.442 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:16.442 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:16.442 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:16.442 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:25:16.443 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.313957+0000", 2024-09-13T11:25:16.443 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.3599999999999998e-06 2024-09-13T11:25:16.443 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:16.443 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:16.443 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:25:16.443 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314043+0000", 2024-09-13T11:25:16.443 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.6766000000000002e-05 2024-09-13T11:25:16.443 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:16.443 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:16.443 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:25:16.443 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314065+0000", 2024-09-13T11:25:16.443 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.1416e-05 2024-09-13T11:25:16.443 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:16.444 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:16.444 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:25:16.444 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314080+0000", 2024-09-13T11:25:16.444 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.5519000000000002e-05 2024-09-13T11:25:16.444 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:16.444 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:16.444 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:25:16.444 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315478+0000", 2024-09-13T11:25:16.444 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.001397765 2024-09-13T11:25:16.444 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:16.444 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:16.444 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-09-13T11:25:16.445 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315507+0000", 2024-09-13T11:25:16.445 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.9128999999999998e-05 2024-09-13T11:25:16.445 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:16.445 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:16.445 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:25:16.445 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.167978+0000", 2024-09-13T11:25:16.445 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.85247068400000003 2024-09-13T11:25:16.445 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:16.446 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:16.446 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:25:16.446 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.168307+0000", 2024-09-13T11:25:16.446 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00032886699999999999 2024-09-13T11:25:16.446 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:16.446 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:16.446 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1", 2024-09-13T11:25:16.446 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.168647+0000", 2024-09-13T11:25:16.446 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00033998399999999999 2024-09-13T11:25:16.446 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:16.447 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:16.447 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:25:16.447 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.178011+0000", 2024-09-13T11:25:16.447 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0093639169999999994 2024-09-13T11:25:16.447 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:16.447 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:16.447 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:25:16.447 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.178026+0000", 2024-09-13T11:25:16.447 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.5274999999999999e-05 2024-09-13T11:25:16.448 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:16.448 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:16.448 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:25:16.448 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.178080+0000", 2024-09-13T11:25:16.448 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.3601999999999999e-05 2024-09-13T11:25:16.448 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:16.448 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:16.449 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:25:16.449 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.178165+0000", 2024-09-13T11:25:16.449 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.5601999999999998e-05 2024-09-13T11:25:16.449 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:16.449 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:25:16.449 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:16.449 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:16.449 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:16.449 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:9 3.b 3:dafadff6:::benchmark_data_smithi177_31116_object8:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T11:25:16.449 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:48.314114+0000", 2024-09-13T11:25:16.449 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 148.11732424100001, 2024-09-13T11:25:16.449 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.86613615899999996, 2024-09-13T11:25:16.449 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:25:16.450 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:25:16.450 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:16.450 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:25:16.450 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:25:16.450 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:16.450 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 9 2024-09-13T11:25:16.450 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:16.450 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:25:16.450 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:16.450 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:25:16.450 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314114+0000", 2024-09-13T11:25:16.450 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:16.450 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:16.451 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:16.451 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:25:16.451 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314114+0000", 2024-09-13T11:25:16.451 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:16.451 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:16.451 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:16.451 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:25:16.451 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314117+0000", 2024-09-13T11:25:16.451 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.3860000000000001e-06 2024-09-13T11:25:16.451 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:16.451 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:16.451 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:25:16.451 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314215+0000", 2024-09-13T11:25:16.452 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.8900000000000005e-05 2024-09-13T11:25:16.452 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:16.452 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:16.452 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:25:16.452 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314229+0000", 2024-09-13T11:25:16.452 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.3444999999999999e-05 2024-09-13T11:25:16.452 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:16.452 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:16.452 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:25:16.452 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314239+0000", 2024-09-13T11:25:16.452 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.8470000000000008e-06 2024-09-13T11:25:16.452 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:16.452 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:16.453 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:25:16.453 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315648+0000", 2024-09-13T11:25:16.453 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.001409291 2024-09-13T11:25:16.453 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:16.453 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:16.453 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-09-13T11:25:16.453 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315671+0000", 2024-09-13T11:25:16.453 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.3169000000000001e-05 2024-09-13T11:25:16.453 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:16.453 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:16.453 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:25:16.453 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.176036+0000", 2024-09-13T11:25:16.454 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.86036497499999998 2024-09-13T11:25:16.454 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:16.454 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:16.454 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:25:16.454 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.176189+0000", 2024-09-13T11:25:16.454 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00015271100000000001 2024-09-13T11:25:16.454 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:16.454 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:16.454 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1", 2024-09-13T11:25:16.454 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.176339+0000", 2024-09-13T11:25:16.454 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00015057499999999999 2024-09-13T11:25:16.454 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:16.454 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:16.454 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:25:16.455 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.180148+0000", 2024-09-13T11:25:16.455 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.003808891 2024-09-13T11:25:16.455 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:16.455 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:16.455 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:25:16.455 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.180168+0000", 2024-09-13T11:25:16.455 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.9507000000000001e-05 2024-09-13T11:25:16.455 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:16.455 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:16.455 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:25:16.455 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.180213+0000", 2024-09-13T11:25:16.455 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.5504999999999999e-05 2024-09-13T11:25:16.455 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:16.456 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:16.456 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:25:16.456 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.180250+0000", 2024-09-13T11:25:16.456 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.6956999999999999e-05 2024-09-13T11:25:16.456 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:16.456 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:25:16.456 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:16.456 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:16.456 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:16.456 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:10 3.b 3:dbe028a8:::benchmark_data_smithi177_31116_object9:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T11:25:16.456 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:48.314259+0000", 2024-09-13T11:25:16.456 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 148.11717911100001, 2024-09-13T11:25:16.457 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.86637313599999999, 2024-09-13T11:25:16.457 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:25:16.457 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:25:16.457 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:16.457 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:25:16.457 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:25:16.457 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:16.457 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 10 2024-09-13T11:25:16.457 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:16.457 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:25:16.457 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:16.457 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:25:16.457 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314259+0000", 2024-09-13T11:25:16.457 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:16.458 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:16.458 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:16.458 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:25:16.458 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314259+0000", 2024-09-13T11:25:16.458 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:16.458 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:16.459 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:16.459 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:25:16.459 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314261+0000", 2024-09-13T11:25:16.459 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.1500000000000002e-06 2024-09-13T11:25:16.459 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:16.459 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:16.459 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:25:16.459 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314349+0000", 2024-09-13T11:25:16.459 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.8070000000000002e-05 2024-09-13T11:25:16.459 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:16.459 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:16.459 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:25:16.459 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314365+0000", 2024-09-13T11:25:16.460 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.5835999999999999e-05 2024-09-13T11:25:16.460 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:16.460 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:16.460 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:25:16.460 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314377+0000", 2024-09-13T11:25:16.460 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.1589e-05 2024-09-13T11:25:16.460 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:16.460 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:16.460 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:25:16.460 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315788+0000", 2024-09-13T11:25:16.460 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0014109579999999999 2024-09-13T11:25:16.460 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:16.460 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:16.461 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-09-13T11:25:16.461 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315809+0000", 2024-09-13T11:25:16.461 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.0985999999999999e-05 2024-09-13T11:25:16.461 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:16.461 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:16.461 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:25:16.461 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.176822+0000", 2024-09-13T11:25:16.461 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.86101310799999997 2024-09-13T11:25:16.461 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:16.461 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:16.461 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:25:16.461 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.177021+0000", 2024-09-13T11:25:16.461 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00019911199999999999 2024-09-13T11:25:16.462 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:16.462 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:16.462 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1", 2024-09-13T11:25:16.462 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.177226+0000", 2024-09-13T11:25:16.462 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00020516600000000001 2024-09-13T11:25:16.462 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:16.462 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:16.462 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:25:16.462 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.180264+0000", 2024-09-13T11:25:16.462 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0030381739999999998 2024-09-13T11:25:16.462 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:16.462 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:16.462 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:25:16.462 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.180387+0000", 2024-09-13T11:25:16.463 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00012257699999999999 2024-09-13T11:25:16.463 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:16.463 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:16.463 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:25:16.463 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.180518+0000", 2024-09-13T11:25:16.463 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00013135800000000001 2024-09-13T11:25:16.463 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:16.463 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:16.463 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:25:16.463 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.180632+0000", 2024-09-13T11:25:16.463 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000114052 2024-09-13T11:25:16.463 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:16.463 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:25:16.464 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:16.464 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:16.464 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:16.464 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:39 3.b 3:d91854b3:::benchmark_data_smithi177_31116_object38:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T11:25:16.464 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:48.314393+0000", 2024-09-13T11:25:16.464 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 148.117045569, 2024-09-13T11:25:16.464 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.86850121700000005, 2024-09-13T11:25:16.464 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:25:16.464 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:25:16.464 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:16.464 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:25:16.464 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:25:16.464 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:16.465 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 39 2024-09-13T11:25:16.465 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:16.465 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:25:16.465 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:16.465 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:25:16.465 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314393+0000", 2024-09-13T11:25:16.465 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:16.465 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:16.465 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:16.465 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:25:16.465 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314393+0000", 2024-09-13T11:25:16.465 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:16.466 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:16.466 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:16.466 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:25:16.466 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314394+0000", 2024-09-13T11:25:16.466 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.513e-06 2024-09-13T11:25:16.466 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:16.466 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:16.466 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:25:16.466 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314463+0000", 2024-09-13T11:25:16.466 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.8261000000000003e-05 2024-09-13T11:25:16.466 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:16.466 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:16.466 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:25:16.466 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314477+0000", 2024-09-13T11:25:16.467 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.4619e-05 2024-09-13T11:25:16.467 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:16.467 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:16.467 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:25:16.467 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314485+0000", 2024-09-13T11:25:16.467 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.1699999999999997e-06 2024-09-13T11:25:16.467 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:16.467 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:16.467 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:25:16.467 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315917+0000", 2024-09-13T11:25:16.467 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0014316089999999999 2024-09-13T11:25:16.467 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:16.467 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:16.467 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-09-13T11:25:16.468 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315933+0000", 2024-09-13T11:25:16.468 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.5999999999999999e-05 2024-09-13T11:25:16.468 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:16.468 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:16.468 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:25:16.468 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.178205+0000", 2024-09-13T11:25:16.468 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.86227247100000004 2024-09-13T11:25:16.468 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:16.468 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:16.468 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:25:16.468 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.178316+0000", 2024-09-13T11:25:16.468 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000110788 2024-09-13T11:25:16.469 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:16.469 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:16.469 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1", 2024-09-13T11:25:16.469 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.178448+0000", 2024-09-13T11:25:16.469 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000132056 2024-09-13T11:25:16.469 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:16.469 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:16.469 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:25:16.469 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.182721+0000", 2024-09-13T11:25:16.470 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0042729780000000002 2024-09-13T11:25:16.470 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:16.470 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:16.470 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:25:16.470 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.182801+0000", 2024-09-13T11:25:16.470 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.0209000000000004e-05 2024-09-13T11:25:16.470 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:16.470 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:16.470 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:25:16.470 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.182855+0000", 2024-09-13T11:25:16.470 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.3749000000000003e-05 2024-09-13T11:25:16.471 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:16.471 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:16.471 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:25:16.471 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.182894+0000", 2024-09-13T11:25:16.471 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.8794000000000002e-05 2024-09-13T11:25:16.471 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:16.471 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:25:16.471 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:16.471 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:16.471 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:16.471 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:50 3.b 3:d1c144b8:::benchmark_data_smithi177_31116_object49:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T11:25:16.471 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:48.314498+0000", 2024-09-13T11:25:16.471 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 148.11694015399999, 2024-09-13T11:25:16.472 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.86878673299999998, 2024-09-13T11:25:16.472 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:25:16.472 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:25:16.472 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:16.472 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:25:16.472 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:25:16.472 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:16.472 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 50 2024-09-13T11:25:16.472 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:16.472 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:25:16.472 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:16.472 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:25:16.472 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314498+0000", 2024-09-13T11:25:16.473 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:16.473 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:16.473 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:16.473 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:25:16.473 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314498+0000", 2024-09-13T11:25:16.473 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:16.473 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:16.473 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:16.473 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:25:16.473 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314499+0000", 2024-09-13T11:25:16.473 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.1969999999999999e-06 2024-09-13T11:25:16.473 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:16.473 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:16.474 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:25:16.474 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314565+0000", 2024-09-13T11:25:16.474 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.5628000000000003e-05 2024-09-13T11:25:16.474 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:16.474 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:16.474 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:25:16.474 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314579+0000", 2024-09-13T11:25:16.474 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.4222e-05 2024-09-13T11:25:16.474 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:16.474 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:16.474 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:25:16.474 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314590+0000", 2024-09-13T11:25:16.474 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.1240000000000001e-05 2024-09-13T11:25:16.474 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:16.475 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:16.475 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:25:16.475 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316060+0000", 2024-09-13T11:25:16.475 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.001469515 2024-09-13T11:25:16.475 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:16.475 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:16.475 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-09-13T11:25:16.475 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316076+0000", 2024-09-13T11:25:16.475 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.5608999999999999e-05 2024-09-13T11:25:16.475 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:16.475 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:16.475 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:25:16.475 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.178857+0000", 2024-09-13T11:25:16.475 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.86278099900000005 2024-09-13T11:25:16.475 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:16.476 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:16.476 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:25:16.476 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.178961+0000", 2024-09-13T11:25:16.476 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000104664 2024-09-13T11:25:16.476 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:16.476 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:16.476 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1", 2024-09-13T11:25:16.476 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.179123+0000", 2024-09-13T11:25:16.476 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000161738 2024-09-13T11:25:16.476 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:16.476 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:16.476 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:25:16.477 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.182755+0000", 2024-09-13T11:25:16.477 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0036323200000000001 2024-09-13T11:25:16.477 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:16.477 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:16.477 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:25:16.477 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.183023+0000", 2024-09-13T11:25:16.477 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00026749100000000002 2024-09-13T11:25:16.477 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:16.477 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:16.477 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:25:16.477 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.183168+0000", 2024-09-13T11:25:16.477 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00014499099999999999 2024-09-13T11:25:16.477 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:16.477 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:16.477 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:25:16.478 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.183285+0000", 2024-09-13T11:25:16.478 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000117119 2024-09-13T11:25:16.478 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:16.478 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:25:16.478 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:16.478 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:16.478 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:16.478 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:60 3.b 3:d67218b0:::benchmark_data_smithi177_31116_object59:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T11:25:16.478 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:48.314603+0000", 2024-09-13T11:25:16.478 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 148.11683559100001, 2024-09-13T11:25:16.478 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.87106201100000002, 2024-09-13T11:25:16.478 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:25:16.478 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:25:16.478 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:16.479 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:25:16.479 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:25:16.479 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:16.479 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 60 2024-09-13T11:25:16.479 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:16.479 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:25:16.479 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:16.479 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:25:16.479 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314603+0000", 2024-09-13T11:25:16.479 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:16.479 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:16.479 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:16.479 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:25:16.480 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314603+0000", 2024-09-13T11:25:16.480 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:16.480 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:16.480 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:16.480 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:25:16.480 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314604+0000", 2024-09-13T11:25:16.480 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.2359999999999999e-06 2024-09-13T11:25:16.480 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:16.480 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:16.480 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:25:16.480 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314667+0000", 2024-09-13T11:25:16.480 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.3077999999999995e-05 2024-09-13T11:25:16.480 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:16.480 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:16.481 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:25:16.481 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314699+0000", 2024-09-13T11:25:16.481 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.1965999999999999e-05 2024-09-13T11:25:16.481 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:16.481 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:16.481 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:25:16.481 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314710+0000", 2024-09-13T11:25:16.481 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.1148999999999999e-05 2024-09-13T11:25:16.481 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:16.481 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:16.481 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:25:16.481 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316291+0000", 2024-09-13T11:25:16.481 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0015804440000000001 2024-09-13T11:25:16.482 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:16.482 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:16.482 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-09-13T11:25:16.482 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316310+0000", 2024-09-13T11:25:16.482 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.9799000000000001e-05 2024-09-13T11:25:16.482 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:16.482 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:16.482 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:25:16.482 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.179555+0000", 2024-09-13T11:25:16.482 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.86324469500000001 2024-09-13T11:25:16.482 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:16.482 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:16.482 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:25:16.482 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.179666+0000", 2024-09-13T11:25:16.483 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000110443 2024-09-13T11:25:16.483 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:16.483 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:16.483 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1", 2024-09-13T11:25:16.483 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.179791+0000", 2024-09-13T11:25:16.483 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00012560499999999999 2024-09-13T11:25:16.483 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:16.483 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:16.483 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:25:16.483 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.182761+0000", 2024-09-13T11:25:16.483 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0029697019999999998 2024-09-13T11:25:16.483 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:16.483 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:16.483 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:25:16.484 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.185572+0000", 2024-09-13T11:25:16.484 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.002811512 2024-09-13T11:25:16.484 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:16.484 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:16.484 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:25:16.484 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.185625+0000", 2024-09-13T11:25:16.484 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.2716000000000001e-05 2024-09-13T11:25:16.484 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:16.484 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:16.484 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:25:16.484 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.185665+0000", 2024-09-13T11:25:16.484 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.9666000000000003e-05 2024-09-13T11:25:16.484 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:16.485 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:25:16.485 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:16.485 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:16.485 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:16.485 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:62 3.b 3:d20ae5a8:::benchmark_data_smithi177_31116_object61:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T11:25:16.485 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:48.314723+0000", 2024-09-13T11:25:16.485 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 148.11671504899999, 2024-09-13T11:25:16.485 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.87338985099999999, 2024-09-13T11:25:16.485 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:25:16.485 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:25:16.485 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:16.485 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:25:16.485 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:25:16.486 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:16.486 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 62 2024-09-13T11:25:16.486 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:16.486 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:25:16.486 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:16.486 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:25:16.486 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314723+0000", 2024-09-13T11:25:16.486 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:16.486 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:16.486 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:16.486 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:25:16.486 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314723+0000", 2024-09-13T11:25:16.486 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:16.486 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:16.487 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:16.487 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:25:16.487 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314725+0000", 2024-09-13T11:25:16.487 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.3069999999999999e-06 2024-09-13T11:25:16.487 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:16.487 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:16.487 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:25:16.487 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314797+0000", 2024-09-13T11:25:16.487 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.2652999999999997e-05 2024-09-13T11:25:16.487 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:16.487 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:16.487 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:25:16.487 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314809+0000", 2024-09-13T11:25:16.487 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.1496e-05 2024-09-13T11:25:16.488 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:16.488 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:16.488 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:25:16.488 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314816+0000", 2024-09-13T11:25:16.488 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.6639999999999998e-06 2024-09-13T11:25:16.488 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:16.488 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:16.488 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:25:16.488 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316424+0000", 2024-09-13T11:25:16.488 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0016077559999999999 2024-09-13T11:25:16.488 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:16.488 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:16.488 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-09-13T11:25:16.489 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316441+0000", 2024-09-13T11:25:16.489 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.7305000000000001e-05 2024-09-13T11:25:16.489 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:16.489 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:16.489 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:25:16.489 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.180688+0000", 2024-09-13T11:25:16.489 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.86424692800000003 2024-09-13T11:25:16.489 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:16.489 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:16.489 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:25:16.489 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.180807+0000", 2024-09-13T11:25:16.489 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000118977 2024-09-13T11:25:16.489 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:16.489 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:16.490 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1", 2024-09-13T11:25:16.490 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.180938+0000", 2024-09-13T11:25:16.490 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00013049700000000001 2024-09-13T11:25:16.490 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:16.490 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:16.490 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:25:16.490 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.185680+0000", 2024-09-13T11:25:16.490 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0047417029999999999 2024-09-13T11:25:16.490 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:16.490 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:16.490 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:25:16.490 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.187856+0000", 2024-09-13T11:25:16.491 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0021760550000000001 2024-09-13T11:25:16.491 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:16.491 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:16.491 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:25:16.491 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.188007+0000", 2024-09-13T11:25:16.491 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00015185099999999999 2024-09-13T11:25:16.491 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:16.491 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:16.491 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:25:16.491 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.188113+0000", 2024-09-13T11:25:16.491 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000105659 2024-09-13T11:25:16.491 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:16.491 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:25:16.491 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:16.492 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:16.492 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:16.492 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:73 3.b 3:d0060d5f:::benchmark_data_smithi177_31116_object72:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T11:25:16.492 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:48.314829+0000", 2024-09-13T11:25:16.492 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 148.1166097, 2024-09-13T11:25:16.492 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.87364757199999998, 2024-09-13T11:25:16.492 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:25:16.492 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:25:16.492 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:16.492 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:25:16.492 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:25:16.492 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:16.492 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 73 2024-09-13T11:25:16.493 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:16.493 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:25:16.493 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:16.493 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:25:16.493 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314829+0000", 2024-09-13T11:25:16.493 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:16.493 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:16.493 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:16.493 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:25:16.493 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314829+0000", 2024-09-13T11:25:16.493 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:16.493 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:16.493 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:16.494 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:25:16.494 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314830+0000", 2024-09-13T11:25:16.494 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.243e-06 2024-09-13T11:25:16.494 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:16.494 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:16.494 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:25:16.494 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314965+0000", 2024-09-13T11:25:16.494 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00013470800000000001 2024-09-13T11:25:16.494 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:16.494 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:16.494 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:25:16.494 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314979+0000", 2024-09-13T11:25:16.494 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.4375999999999999e-05 2024-09-13T11:25:16.495 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:16.495 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:16.495 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:25:16.495 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314988+0000", 2024-09-13T11:25:16.495 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.8570000000000001e-06 2024-09-13T11:25:16.495 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:16.495 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:16.495 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:25:16.495 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316545+0000", 2024-09-13T11:25:16.495 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.001556878 2024-09-13T11:25:16.495 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:16.495 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:16.495 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-09-13T11:25:16.496 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316556+0000", 2024-09-13T11:25:16.496 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.1453e-05 2024-09-13T11:25:16.496 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:16.496 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:16.496 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:25:16.496 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.182124+0000", 2024-09-13T11:25:16.496 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.86556744200000002 2024-09-13T11:25:16.496 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:16.496 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:16.496 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:25:16.496 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.182246+0000", 2024-09-13T11:25:16.496 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00012210500000000001 2024-09-13T11:25:16.496 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:16.497 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:16.497 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1", 2024-09-13T11:25:16.497 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.182375+0000", 2024-09-13T11:25:16.497 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00012920999999999999 2024-09-13T11:25:16.497 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:16.497 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:16.497 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:25:16.497 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.185684+0000", 2024-09-13T11:25:16.497 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0033093969999999999 2024-09-13T11:25:16.497 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:16.497 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:16.497 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:25:16.497 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.188244+0000", 2024-09-13T11:25:16.498 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0025601550000000002 2024-09-13T11:25:16.498 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:16.498 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:16.498 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:25:16.498 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.188379+0000", 2024-09-13T11:25:16.498 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00013463400000000001 2024-09-13T11:25:16.498 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:16.498 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:16.498 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:25:16.498 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.188476+0000", 2024-09-13T11:25:16.498 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.7114000000000006e-05 2024-09-13T11:25:16.498 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:16.499 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:25:16.499 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:16.499 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:16.499 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:16.499 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:81 3.b 3:d705fd70:::benchmark_data_smithi177_31116_object80:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T11:25:16.499 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:48.315000+0000", 2024-09-13T11:25:16.499 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 148.11643786299999, 2024-09-13T11:25:16.499 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.87561418300000005, 2024-09-13T11:25:16.499 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:25:16.499 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:25:16.499 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:16.499 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:25:16.500 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:25:16.500 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:16.500 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 81 2024-09-13T11:25:16.500 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:16.500 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:25:16.500 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:16.500 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:25:16.500 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315000+0000", 2024-09-13T11:25:16.500 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:16.500 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:16.500 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:16.500 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:25:16.500 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315000+0000", 2024-09-13T11:25:16.501 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:16.501 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:16.501 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:16.501 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:25:16.501 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315002+0000", 2024-09-13T11:25:16.501 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.3799999999999999e-06 2024-09-13T11:25:16.501 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:16.501 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:16.501 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:25:16.501 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315119+0000", 2024-09-13T11:25:16.501 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000116925 2024-09-13T11:25:16.501 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:16.501 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:16.501 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:25:16.502 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315133+0000", 2024-09-13T11:25:16.502 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.3971999999999999e-05 2024-09-13T11:25:16.502 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:16.502 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:16.502 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:25:16.502 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315142+0000", 2024-09-13T11:25:16.502 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.4399999999999994e-06 2024-09-13T11:25:16.502 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:16.502 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:16.502 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:25:16.502 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316652+0000", 2024-09-13T11:25:16.502 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.001509907 2024-09-13T11:25:16.502 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:16.503 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:16.503 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-09-13T11:25:16.503 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316668+0000", 2024-09-13T11:25:16.503 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.5945999999999999e-05 2024-09-13T11:25:16.503 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:16.503 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:16.503 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:25:16.503 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.183405+0000", 2024-09-13T11:25:16.503 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.866737384 2024-09-13T11:25:16.503 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:16.503 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:16.503 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:25:16.504 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.183701+0000", 2024-09-13T11:25:16.504 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00029511099999999997 2024-09-13T11:25:16.504 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:16.504 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:16.504 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1", 2024-09-13T11:25:16.504 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.184078+0000", 2024-09-13T11:25:16.504 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00037741000000000001 2024-09-13T11:25:16.504 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:16.504 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:16.504 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:25:16.504 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.188504+0000", 2024-09-13T11:25:16.504 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0044260280000000003 2024-09-13T11:25:16.505 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:16.505 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:16.505 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:25:16.505 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.190562+0000", 2024-09-13T11:25:16.505 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0020580099999999999 2024-09-13T11:25:16.505 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:16.505 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:16.505 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:25:16.505 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.190592+0000", 2024-09-13T11:25:16.505 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.0057999999999999e-05 2024-09-13T11:25:16.505 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:16.505 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:16.505 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:25:16.506 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.190615+0000", 2024-09-13T11:25:16.506 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.2612e-05 2024-09-13T11:25:16.506 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:16.506 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:25:16.506 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:16.506 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:16.506 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:16.506 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:87 3.b 3:dd7de0ad:::benchmark_data_smithi177_31116_object86:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T11:25:16.506 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:48.315155+0000", 2024-09-13T11:25:16.506 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 148.11628306599999, 2024-09-13T11:25:16.506 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.87555916099999997, 2024-09-13T11:25:16.506 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:25:16.506 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:25:16.506 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:16.507 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:25:16.507 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:25:16.507 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:16.507 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 87 2024-09-13T11:25:16.507 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:16.507 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:25:16.507 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:16.507 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:25:16.507 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315155+0000", 2024-09-13T11:25:16.507 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:16.507 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:16.507 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:16.507 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:25:16.507 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315155+0000", 2024-09-13T11:25:16.508 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:16.508 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:16.508 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:16.508 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:25:16.508 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315157+0000", 2024-09-13T11:25:16.508 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.3829999999999999e-06 2024-09-13T11:25:16.508 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:16.508 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:16.508 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:25:16.508 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315271+0000", 2024-09-13T11:25:16.509 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000114539 2024-09-13T11:25:16.509 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:16.509 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:16.509 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:25:16.509 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315285+0000", 2024-09-13T11:25:16.510 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.4042e-05 2024-09-13T11:25:16.510 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:16.510 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:16.510 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:25:16.510 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315295+0000", 2024-09-13T11:25:16.510 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.2739999999999996e-06 2024-09-13T11:25:16.510 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:16.510 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:16.510 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:25:16.510 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316739+0000", 2024-09-13T11:25:16.510 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0014446960000000001 2024-09-13T11:25:16.511 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:16.511 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:16.512 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-09-13T11:25:16.512 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316747+0000", 2024-09-13T11:25:16.512 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.0790000000000001e-06 2024-09-13T11:25:16.512 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:16.513 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:16.513 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:25:16.513 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.184922+0000", 2024-09-13T11:25:16.513 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.86817446399999998 2024-09-13T11:25:16.513 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:16.513 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:16.513 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:25:16.513 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.185037+0000", 2024-09-13T11:25:16.513 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00011565099999999999 2024-09-13T11:25:16.513 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:16.513 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:16.513 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1", 2024-09-13T11:25:16.513 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.185171+0000", 2024-09-13T11:25:16.513 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00013316900000000001 2024-09-13T11:25:16.514 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:16.514 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:16.514 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:25:16.514 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.188520+0000", 2024-09-13T11:25:16.514 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0033489150000000001 2024-09-13T11:25:16.514 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:16.514 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:16.514 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:25:16.514 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.190652+0000", 2024-09-13T11:25:16.514 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0021325649999999999 2024-09-13T11:25:16.514 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:16.514 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:16.514 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:25:16.515 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.190686+0000", 2024-09-13T11:25:16.515 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.3827e-05 2024-09-13T11:25:16.515 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:16.515 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:16.515 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:25:16.515 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.190714+0000", 2024-09-13T11:25:16.515 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.8557000000000001e-05 2024-09-13T11:25:16.515 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:16.515 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:25:16.515 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:16.515 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:16.515 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:16.515 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:99 3.b 3:d011b600:::benchmark_data_smithi177_31116_object98:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T11:25:16.515 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:48.315307+0000", 2024-09-13T11:25:16.516 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 148.116131239, 2024-09-13T11:25:16.516 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.87551770799999995, 2024-09-13T11:25:16.516 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:25:16.516 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:25:16.516 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:16.516 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:25:16.516 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:25:16.516 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:16.516 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 99 2024-09-13T11:25:16.516 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:16.516 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:25:16.516 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:16.516 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:25:16.517 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315307+0000", 2024-09-13T11:25:16.517 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:16.517 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:16.517 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:16.517 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:25:16.517 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315307+0000", 2024-09-13T11:25:16.517 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:16.517 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:16.517 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:16.517 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:25:16.517 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315308+0000", 2024-09-13T11:25:16.517 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.316e-06 2024-09-13T11:25:16.517 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:16.518 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:16.518 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:25:16.518 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315453+0000", 2024-09-13T11:25:16.518 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00014473200000000001 2024-09-13T11:25:16.518 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:16.518 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:16.518 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:25:16.518 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315471+0000", 2024-09-13T11:25:16.518 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.8019000000000001e-05 2024-09-13T11:25:16.518 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:16.518 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:16.518 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:25:16.518 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315489+0000", 2024-09-13T11:25:16.518 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.7861999999999999e-05 2024-09-13T11:25:16.519 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:16.519 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:16.519 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:25:16.519 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316837+0000", 2024-09-13T11:25:16.519 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.001347787 2024-09-13T11:25:16.519 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:16.519 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:16.519 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-09-13T11:25:16.519 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316858+0000", 2024-09-13T11:25:16.519 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.1103000000000001e-05 2024-09-13T11:25:16.519 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:16.519 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:16.519 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:25:16.519 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.185713+0000", 2024-09-13T11:25:16.520 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.86885540800000005 2024-09-13T11:25:16.520 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:16.520 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:16.520 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:25:16.520 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.185820+0000", 2024-09-13T11:25:16.520 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000107081 2024-09-13T11:25:16.520 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:16.520 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:16.520 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1", 2024-09-13T11:25:16.520 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.185948+0000", 2024-09-13T11:25:16.520 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00012775800000000001 2024-09-13T11:25:16.520 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:16.521 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:16.521 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:25:16.521 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.190527+0000", 2024-09-13T11:25:16.521 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0045783910000000002 2024-09-13T11:25:16.521 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:16.521 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:16.521 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:25:16.521 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.190759+0000", 2024-09-13T11:25:16.521 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00023266199999999999 2024-09-13T11:25:16.521 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:16.521 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:16.521 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:25:16.521 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.190789+0000", 2024-09-13T11:25:16.522 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.9666999999999999e-05 2024-09-13T11:25:16.522 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:16.522 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:16.522 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:25:16.522 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.190825+0000", 2024-09-13T11:25:16.522 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.5921999999999999e-05 2024-09-13T11:25:16.522 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:16.522 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:25:16.522 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:16.522 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:16.522 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:16.522 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:108 3.b 3:d5c90217:::benchmark_data_smithi177_31116_object107:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T11:25:16.522 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:48.315513+0000", 2024-09-13T11:25:16.523 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 148.11592553400001, 2024-09-13T11:25:16.523 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.87544772500000001, 2024-09-13T11:25:16.523 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:25:16.523 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:25:16.523 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:16.523 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:25:16.523 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:25:16.523 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:16.523 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 108 2024-09-13T11:25:16.523 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:16.523 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:25:16.523 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:16.524 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:25:16.524 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315513+0000", 2024-09-13T11:25:16.524 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:16.524 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:16.524 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:16.524 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:25:16.524 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315513+0000", 2024-09-13T11:25:16.524 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:16.524 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:16.524 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:16.524 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:25:16.525 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315515+0000", 2024-09-13T11:25:16.525 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.9369999999999998e-06 2024-09-13T11:25:16.525 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:16.525 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:16.525 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:25:16.525 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315644+0000", 2024-09-13T11:25:16.525 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000129011 2024-09-13T11:25:16.525 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:16.525 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:16.525 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:25:16.525 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315657+0000", 2024-09-13T11:25:16.525 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.3689999999999999e-05 2024-09-13T11:25:16.525 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:16.526 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:16.526 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:25:16.526 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315668+0000", 2024-09-13T11:25:16.526 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0589e-05 2024-09-13T11:25:16.526 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:16.526 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:16.526 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:25:16.526 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316925+0000", 2024-09-13T11:25:16.526 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0012568340000000001 2024-09-13T11:25:16.526 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:16.526 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:16.526 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-09-13T11:25:16.526 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316932+0000", 2024-09-13T11:25:16.526 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.2230000000000001e-06 2024-09-13T11:25:16.527 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:16.527 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:16.527 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:25:16.527 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.186619+0000", 2024-09-13T11:25:16.527 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.86968679599999998 2024-09-13T11:25:16.527 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:16.527 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:16.527 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:25:16.527 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.186914+0000", 2024-09-13T11:25:16.527 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00029528900000000001 2024-09-13T11:25:16.527 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:16.527 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:16.527 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1", 2024-09-13T11:25:16.527 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.187217+0000", 2024-09-13T11:25:16.528 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00030232099999999998 2024-09-13T11:25:16.528 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:16.528 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:16.528 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:25:16.528 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.190532+0000", 2024-09-13T11:25:16.528 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.003315391 2024-09-13T11:25:16.528 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:16.528 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:16.528 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:25:16.528 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.190875+0000", 2024-09-13T11:25:16.528 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00034326300000000001 2024-09-13T11:25:16.528 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:16.528 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:16.529 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:25:16.529 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.190925+0000", 2024-09-13T11:25:16.529 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.9429999999999999e-05 2024-09-13T11:25:16.529 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:16.529 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:16.529 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:25:16.529 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.190961+0000", 2024-09-13T11:25:16.529 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.5951000000000002e-05 2024-09-13T11:25:16.529 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:16.529 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:25:16.529 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:16.529 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:16.529 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:16.530 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:172 3.b 3:dbec1eeb:::benchmark_data_smithi177_31116_object171:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T11:25:16.530 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:48.315684+0000", 2024-09-13T11:25:16.530 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 148.115754377, 2024-09-13T11:25:16.530 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.87615312999999995, 2024-09-13T11:25:16.530 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:25:16.530 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:25:16.530 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:16.530 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:25:16.530 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:25:16.530 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:16.530 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 172 2024-09-13T11:25:16.530 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:16.530 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:25:16.531 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:16.531 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:25:16.531 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315684+0000", 2024-09-13T11:25:16.531 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:16.531 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:16.531 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:16.531 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:25:16.531 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315684+0000", 2024-09-13T11:25:16.531 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:16.531 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:16.531 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:16.531 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:25:16.531 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315686+0000", 2024-09-13T11:25:16.532 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.886e-06 2024-09-13T11:25:16.532 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:16.532 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:16.532 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:25:16.532 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315820+0000", 2024-09-13T11:25:16.532 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00013388500000000001 2024-09-13T11:25:16.532 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:16.532 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:16.532 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:25:16.532 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315837+0000", 2024-09-13T11:25:16.532 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.6756e-05 2024-09-13T11:25:16.532 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:16.532 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:16.533 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:25:16.533 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315851+0000", 2024-09-13T11:25:16.533 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.4566000000000001e-05 2024-09-13T11:25:16.533 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:16.533 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:16.533 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:25:16.533 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.317021+0000", 2024-09-13T11:25:16.533 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0011701769999999999 2024-09-13T11:25:16.533 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:16.533 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:16.533 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-09-13T11:25:16.533 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.317037+0000", 2024-09-13T11:25:16.533 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.609e-05 2024-09-13T11:25:16.534 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:16.534 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:16.534 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:25:16.534 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.188634+0000", 2024-09-13T11:25:16.534 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.87159711900000003 2024-09-13T11:25:16.534 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:16.534 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:16.534 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:25:16.534 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.188974+0000", 2024-09-13T11:25:16.534 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00033922299999999999 2024-09-13T11:25:16.534 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:16.534 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:16.534 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1", 2024-09-13T11:25:16.535 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.189510+0000", 2024-09-13T11:25:16.535 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00053618999999999997 2024-09-13T11:25:16.535 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:16.535 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:16.535 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:25:16.535 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.191027+0000", 2024-09-13T11:25:16.535 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0015168390000000001 2024-09-13T11:25:16.535 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:16.535 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:16.535 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:25:16.535 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.191766+0000", 2024-09-13T11:25:16.535 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000739064 2024-09-13T11:25:16.535 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:16.536 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:16.536 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:25:16.536 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.191810+0000", 2024-09-13T11:25:16.536 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.3755999999999999e-05 2024-09-13T11:25:16.536 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:16.536 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:16.536 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:25:16.536 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.191837+0000", 2024-09-13T11:25:16.536 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.7579000000000001e-05 2024-09-13T11:25:16.536 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:16.536 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:25:16.536 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:16.536 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:16.536 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:16.537 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:177 3.b 3:de89c779:::benchmark_data_smithi177_31116_object176:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T11:25:16.537 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:48.315898+0000", 2024-09-13T11:25:16.537 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 148.11554006200001, 2024-09-13T11:25:16.537 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.87600712599999997, 2024-09-13T11:25:16.537 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:25:16.537 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:25:16.537 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:16.537 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:25:16.537 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:25:16.537 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:16.537 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 177 2024-09-13T11:25:16.537 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:16.537 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:25:16.538 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:16.538 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:25:16.538 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315898+0000", 2024-09-13T11:25:16.538 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:16.538 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:16.538 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:16.538 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:25:16.538 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315898+0000", 2024-09-13T11:25:16.538 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:16.538 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:16.538 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:16.538 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:25:16.538 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315900+0000", 2024-09-13T11:25:16.538 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.4929999999999999e-06 2024-09-13T11:25:16.539 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:16.539 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:16.539 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:25:16.539 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316027+0000", 2024-09-13T11:25:16.539 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000127568 2024-09-13T11:25:16.539 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:16.539 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:16.539 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:25:16.539 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316048+0000", 2024-09-13T11:25:16.539 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.0795999999999999e-05 2024-09-13T11:25:16.539 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:16.539 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:16.539 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:25:16.540 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316061+0000", 2024-09-13T11:25:16.540 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.3283e-05 2024-09-13T11:25:16.540 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:16.540 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:16.540 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:25:16.540 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.317107+0000", 2024-09-13T11:25:16.540 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0010452949999999999 2024-09-13T11:25:16.540 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:16.540 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:16.540 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-09-13T11:25:16.540 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.317114+0000", 2024-09-13T11:25:16.540 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.4569999999999999e-06 2024-09-13T11:25:16.540 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:16.541 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:16.541 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:25:16.541 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.190017+0000", 2024-09-13T11:25:16.541 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.87290258499999995 2024-09-13T11:25:16.541 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:16.541 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:16.541 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:25:16.541 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.190138+0000", 2024-09-13T11:25:16.541 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000121418 2024-09-13T11:25:16.541 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:16.541 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:16.541 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1", 2024-09-13T11:25:16.541 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.190216+0000", 2024-09-13T11:25:16.542 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.7410999999999995e-05 2024-09-13T11:25:16.542 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:16.542 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:16.542 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:25:16.542 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.191755+0000", 2024-09-13T11:25:16.542 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0015395420000000001 2024-09-13T11:25:16.542 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:16.542 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:16.542 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:25:16.542 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.191843+0000", 2024-09-13T11:25:16.542 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.8022999999999994e-05 2024-09-13T11:25:16.542 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:16.542 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:16.543 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:25:16.543 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.191876+0000", 2024-09-13T11:25:16.543 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.2619000000000001e-05 2024-09-13T11:25:16.543 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:16.543 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:16.543 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:25:16.543 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.191905+0000", 2024-09-13T11:25:16.543 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.9635999999999999e-05 2024-09-13T11:25:16.543 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:16.543 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:25:16.543 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:16.543 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:16.543 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:16.543 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:187 3.b 3:da03c8dd:::benchmark_data_smithi177_31116_object186:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T11:25:16.544 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:48.316083+0000", 2024-09-13T11:25:16.544 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 148.11535510600001, 2024-09-13T11:25:16.544 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.87835499500000003, 2024-09-13T11:25:16.544 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:25:16.544 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:25:16.544 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:16.544 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:25:16.544 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:25:16.544 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:16.544 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 187 2024-09-13T11:25:16.544 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:16.544 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:25:16.544 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:16.545 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:25:16.545 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316083+0000", 2024-09-13T11:25:16.545 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:16.545 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:16.545 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:16.545 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:25:16.545 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316083+0000", 2024-09-13T11:25:16.545 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:16.545 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:16.545 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:16.545 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:25:16.545 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316086+0000", 2024-09-13T11:25:16.545 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.5560000000000002e-06 2024-09-13T11:25:16.546 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:16.546 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:16.546 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:25:16.546 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316199+0000", 2024-09-13T11:25:16.546 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000113522 2024-09-13T11:25:16.546 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:16.546 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:16.546 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:25:16.546 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316215+0000", 2024-09-13T11:25:16.546 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.5265999999999999e-05 2024-09-13T11:25:16.546 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:16.546 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:16.546 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:25:16.547 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316226+0000", 2024-09-13T11:25:16.547 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0967e-05 2024-09-13T11:25:16.547 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:16.547 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:16.547 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:25:16.547 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.317198+0000", 2024-09-13T11:25:16.547 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00097194100000000004 2024-09-13T11:25:16.547 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:16.547 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:16.547 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-09-13T11:25:16.547 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.317213+0000", 2024-09-13T11:25:16.547 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.5472999999999999e-05 2024-09-13T11:25:16.547 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:16.548 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:16.548 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:25:16.548 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.191077+0000", 2024-09-13T11:25:16.548 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.87386435200000001 2024-09-13T11:25:16.548 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:16.548 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:16.548 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:25:16.548 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.191181+0000", 2024-09-13T11:25:16.548 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000103618 2024-09-13T11:25:16.548 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:16.548 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:16.548 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1", 2024-09-13T11:25:16.548 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.191305+0000", 2024-09-13T11:25:16.549 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00012358599999999999 2024-09-13T11:25:16.549 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:16.549 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:16.549 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:25:16.549 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.194279+0000", 2024-09-13T11:25:16.549 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0029746970000000001 2024-09-13T11:25:16.549 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:16.549 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:16.549 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:25:16.549 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.194336+0000", 2024-09-13T11:25:16.549 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.7244000000000003e-05 2024-09-13T11:25:16.549 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:16.550 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:16.550 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:25:16.550 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.194397+0000", 2024-09-13T11:25:16.550 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.0278000000000001e-05 2024-09-13T11:25:16.550 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:16.550 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:16.550 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:25:16.550 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.194438+0000", 2024-09-13T11:25:16.550 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.1495000000000002e-05 2024-09-13T11:25:16.550 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:16.550 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:25:16.550 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:16.550 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:16.551 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:16.551 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:200 3.b 3:d90039f1:::benchmark_data_smithi177_31116_object199:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T11:25:16.551 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:48.316280+0000", 2024-09-13T11:25:16.551 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 148.11515864099999, 2024-09-13T11:25:16.551 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.87879273000000002, 2024-09-13T11:25:16.551 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:25:16.551 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:25:16.551 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:16.551 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:25:16.551 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:25:16.551 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:16.551 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 200 2024-09-13T11:25:16.551 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:16.551 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:25:16.552 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:16.552 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:25:16.552 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316280+0000", 2024-09-13T11:25:16.552 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:16.552 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:16.552 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:16.552 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:25:16.552 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316280+0000", 2024-09-13T11:25:16.552 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:16.552 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:16.552 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:16.552 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:25:16.553 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316282+0000", 2024-09-13T11:25:16.553 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.497e-06 2024-09-13T11:25:16.553 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:16.553 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:16.553 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:25:16.553 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316411+0000", 2024-09-13T11:25:16.553 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00012924799999999999 2024-09-13T11:25:16.553 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:16.553 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:16.553 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:25:16.553 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316426+0000", 2024-09-13T11:25:16.553 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.4183e-05 2024-09-13T11:25:16.553 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:16.554 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:16.554 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:25:16.554 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316435+0000", 2024-09-13T11:25:16.554 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.0990000000000005e-06 2024-09-13T11:25:16.554 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:16.554 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:16.554 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:25:16.554 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.317250+0000", 2024-09-13T11:25:16.554 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00081501100000000003 2024-09-13T11:25:16.554 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:16.554 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:16.554 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-09-13T11:25:16.554 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.317257+0000", 2024-09-13T11:25:16.555 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.2169999999999997e-06 2024-09-13T11:25:16.555 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:16.555 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:16.555 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:25:16.555 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.191936+0000", 2024-09-13T11:25:16.555 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.87467922099999995 2024-09-13T11:25:16.555 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:16.555 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:16.555 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:25:16.555 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.192056+0000", 2024-09-13T11:25:16.555 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000119638 2024-09-13T11:25:16.555 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:16.555 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:16.556 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1", 2024-09-13T11:25:16.556 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.192184+0000", 2024-09-13T11:25:16.556 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00012813399999999999 2024-09-13T11:25:16.556 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:16.556 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:16.556 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:25:16.556 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.194324+0000", 2024-09-13T11:25:16.556 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0021399420000000001 2024-09-13T11:25:16.556 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:16.556 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:16.556 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:25:16.556 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.194802+0000", 2024-09-13T11:25:16.556 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00047798799999999999 2024-09-13T11:25:16.557 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:16.557 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:16.557 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:25:16.557 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.194951+0000", 2024-09-13T11:25:16.557 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0001494 2024-09-13T11:25:16.557 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:16.557 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:16.557 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:25:16.557 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.195072+0000", 2024-09-13T11:25:16.557 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00012115199999999999 2024-09-13T11:25:16.557 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:16.557 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:25:16.557 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:16.558 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:16.558 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:16.558 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop(client.4665.0:79801 3.5 e102/67 3:a6efdf84:::benchmark_data_smithi177_31116_object79800:head v 102'3851, mlcod=102'3848)", 2024-09-13T11:25:16.558 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:24:00.403754+0000", 2024-09-13T11:25:16.558 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 76.027684289000007, 2024-09-13T11:25:16.558 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.31749079899999999, 2024-09-13T11:25:16.558 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:25:16.558 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:25:16.558 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:16.558 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:25:16.558 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:16.558 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:25:16.558 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:00.403754+0000", 2024-09-13T11:25:16.558 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:16.559 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:16.559 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:16.559 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:25:16.559 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:00.403754+0000", 2024-09-13T11:25:16.559 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:16.559 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:16.559 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:16.559 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:25:16.559 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:00.403758+0000", 2024-09-13T11:25:16.559 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.507e-06 2024-09-13T11:25:16.559 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:16.559 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:16.559 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:25:16.559 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:00.403819+0000", 2024-09-13T11:25:16.560 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.1042000000000003e-05 2024-09-13T11:25:16.560 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:16.560 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:16.560 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:25:16.560 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:00.717963+0000", 2024-09-13T11:25:16.560 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.314144594 2024-09-13T11:25:16.560 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:16.560 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:16.560 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:25:16.560 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:00.717976+0000", 2024-09-13T11:25:16.560 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.3029e-05 2024-09-13T11:25:16.560 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:16.560 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:16.561 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:25:16.561 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:00.718143+0000", 2024-09-13T11:25:16.561 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00016639000000000001 2024-09-13T11:25:16.561 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:16.561 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:16.561 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:25:16.561 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:00.718184+0000", 2024-09-13T11:25:16.561 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.1801999999999997e-05 2024-09-13T11:25:16.561 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:16.561 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:16.561 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:25:16.561 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:00.721162+0000", 2024-09-13T11:25:16.561 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0029772800000000001 2024-09-13T11:25:16.562 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:16.562 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:16.562 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:25:16.562 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:00.721245+0000", 2024-09-13T11:25:16.562 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.3154999999999999e-05 2024-09-13T11:25:16.562 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:16.562 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:25:16.562 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:16.562 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:16.562 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:16.562 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "MOSDPGPushReply(3.26 121/121 [PushReplyOp(3:670b9aab:::benchmark_data_smithi177_31116_object93:head)])", 2024-09-13T11:25:16.563 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:24:27.300318+0000", 2024-09-13T11:25:16.563 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 49.131120420999999, 2024-09-13T11:25:16.563 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.811436728, 2024-09-13T11:25:16.563 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:25:16.563 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:25:16.563 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "reached pg", 2024-09-13T11:25:16.563 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:25:16.563 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:16.563 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:25:16.563 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:27.300318+0000", 2024-09-13T11:25:16.563 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:16.563 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:16.563 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:16.564 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:25:16.564 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:27.300318+0000", 2024-09-13T11:25:16.564 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:16.564 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:16.564 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:16.564 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:25:16.564 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:27.300320+0000", 2024-09-13T11:25:16.564 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.778e-06 2024-09-13T11:25:16.564 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:16.564 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:16.564 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:25:16.564 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:27.300326+0000", 2024-09-13T11:25:16.564 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.7700000000000004e-06 2024-09-13T11:25:16.565 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:16.565 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:16.565 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:25:16.565 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:28.109665+0000", 2024-09-13T11:25:16.565 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.80933853499999997 2024-09-13T11:25:16.565 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:16.565 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:16.565 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:25:16.565 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:28.109678+0000", 2024-09-13T11:25:16.565 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.2870999999999999e-05 2024-09-13T11:25:16.565 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:16.565 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:16.565 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:25:16.565 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:28.109800+0000", 2024-09-13T11:25:16.566 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000122328 2024-09-13T11:25:16.566 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:16.566 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:16.566 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:25:16.566 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:28.111755+0000", 2024-09-13T11:25:16.566 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0019544459999999999 2024-09-13T11:25:16.566 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:16.566 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:25:16.566 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:16.566 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:16.566 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:16.566 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "MOSDPGPushReply(3.6 125/122 [PushReplyOp(3:61422348:::benchmark_data_smithi177_31116_object23374:head)])", 2024-09-13T11:25:16.566 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:24:32.775743+0000", 2024-09-13T11:25:16.567 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 43.655695563999998, 2024-09-13T11:25:16.567 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.304090953, 2024-09-13T11:25:16.567 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:25:16.567 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:25:16.567 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "reached pg", 2024-09-13T11:25:16.567 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:25:16.567 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:16.567 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:25:16.567 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:32.775743+0000", 2024-09-13T11:25:16.567 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:16.567 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:16.567 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:16.567 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:25:16.568 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:32.775743+0000", 2024-09-13T11:25:16.568 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:16.568 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:16.568 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:16.568 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:25:16.568 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:32.775744+0000", 2024-09-13T11:25:16.568 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.522e-06 2024-09-13T11:25:16.568 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:16.568 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:16.568 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:25:16.568 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:32.775753+0000", 2024-09-13T11:25:16.568 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.2889999999999998e-06 2024-09-13T11:25:16.569 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:16.569 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:16.569 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:25:16.569 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:33.078899+0000", 2024-09-13T11:25:16.569 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.30314683399999998 2024-09-13T11:25:16.569 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:16.569 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:16.569 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:25:16.569 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:33.078906+0000", 2024-09-13T11:25:16.569 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.9029999999999998e-06 2024-09-13T11:25:16.569 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:16.569 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:16.569 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:25:16.570 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:33.078954+0000", 2024-09-13T11:25:16.570 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.7754999999999999e-05 2024-09-13T11:25:16.570 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:16.570 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:16.570 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:25:16.570 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:33.079834+0000", 2024-09-13T11:25:16.570 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00087965000000000005 2024-09-13T11:25:16.570 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:16.570 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:25:16.570 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:16.570 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:16.570 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:25:16.570 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-09-13T11:25:16.572 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-13T11:25:16.573 INFO:tasks.ceph.osd.1.smithi092.stderr:2024-09-13T11:25:16.565+0000 7fe7789c5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33718) UID: 0 2024-09-13T11:25:16.673 INFO:tasks.ceph.osd.12.smithi001.stderr:2024-09-13T11:25:16.671+0000 7f892c6e6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26314) UID: 0 2024-09-13T11:25:16.714 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-13T11:25:16.714 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [], 2024-09-13T11:25:16.714 INFO:teuthology.orchestra.run.smithi162.stdout: "num_ops": 0 2024-09-13T11:25:16.715 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-13T11:25:16.732 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-13T11:25:16.774 INFO:tasks.ceph.osd.4.smithi001.stderr:2024-09-13T11:25:16.771+0000 7ff8e0218640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26310) UID: 0 2024-09-13T11:25:16.865 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-13T11:25:16.865 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [], 2024-09-13T11:25:16.865 INFO:teuthology.orchestra.run.smithi162.stdout: "complaint_time": 30, 2024-09-13T11:25:16.865 INFO:teuthology.orchestra.run.smithi162.stdout: "num_blocked_ops": 0 2024-09-13T11:25:16.865 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-13T11:25:16.875 INFO:tasks.ceph.osd.12.smithi001.stderr:2024-09-13T11:25:16.871+0000 7f892c6e6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26314) UID: 0 2024-09-13T11:25:16.881 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-13T11:25:16.928 INFO:teuthology.orchestra.run.smithi001.stderr:nodeep-scrub is set 2024-09-13T11:25:16.928 INFO:teuthology.orchestra.run.smithi001.stderr:reweighted osd.6 to 1 (10000) 2024-09-13T11:25:16.933 INFO:tasks.ceph.osd.7.smithi177.stderr:2024-09-13T11:25:16.930+0000 7f896ff4a640 -1 osd.7 157 handle_osd_map injecting map crc failure 2024-09-13T11:25:16.956 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 7 1 2024-09-13T11:25:16.976 INFO:tasks.ceph.osd.11.smithi177.stderr:2024-09-13T11:25:16.974+0000 7f24792c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26580) UID: 0 2024-09-13T11:25:17.005 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-13T11:25:17.005 INFO:teuthology.orchestra.run.smithi162.stdout: "size": 20, 2024-09-13T11:25:17.005 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 600, 2024-09-13T11:25:17.005 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [ 2024-09-13T11:25:17.005 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:17.006 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.4665.0:171 3.9 e41/30 3:9e1c7eeb:::benchmark_data_smithi177_31116_object170:head v 41'13, mlcod=41'9)", 2024-09-13T11:25:17.006 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:22:47.418602+0000", 2024-09-13T11:25:17.006 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 149.584888294, 2024-09-13T11:25:17.006 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.67235367300000004, 2024-09-13T11:25:17.006 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:25:17.006 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:25:17.006 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:17.006 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:25:17.006 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:17.006 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:25:17.006 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:47.418602+0000", 2024-09-13T11:25:17.006 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:17.007 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:17.007 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:17.007 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:25:17.007 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:47.418602+0000", 2024-09-13T11:25:17.007 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:17.007 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:17.007 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:17.007 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:25:17.007 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:47.418605+0000", 2024-09-13T11:25:17.007 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.2200000000000001e-06 2024-09-13T11:25:17.007 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:17.007 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:17.008 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:25:17.008 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:47.418668+0000", 2024-09-13T11:25:17.008 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.2921000000000003e-05 2024-09-13T11:25:17.008 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:17.008 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:17.008 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:25:17.008 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:48.089380+0000", 2024-09-13T11:25:17.008 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.67071191100000005 2024-09-13T11:25:17.008 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:17.008 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:17.008 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:25:17.008 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:48.089423+0000", 2024-09-13T11:25:17.008 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.2916e-05 2024-09-13T11:25:17.009 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:17.009 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:17.009 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:25:17.009 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:48.089533+0000", 2024-09-13T11:25:17.009 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000109996 2024-09-13T11:25:17.009 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:17.009 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:17.009 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:25:17.009 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:48.089570+0000", 2024-09-13T11:25:17.009 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.7314999999999998e-05 2024-09-13T11:25:17.009 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:17.009 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:17.009 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:25:17.010 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:48.090852+0000", 2024-09-13T11:25:17.010 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001281641 2024-09-13T11:25:17.010 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:17.010 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:17.010 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:25:17.010 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:48.090956+0000", 2024-09-13T11:25:17.010 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00010375299999999999 2024-09-13T11:25:17.010 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:17.010 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:25:17.011 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:17.011 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:17.011 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:17.011 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPushReply(3.19 74/70 [PushReplyOp(3:9edf939c:::benchmark_data_smithi177_31116_object17207:head)])", 2024-09-13T11:25:17.011 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:30.897971+0000", 2024-09-13T11:25:17.011 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 106.10551944300001, 2024-09-13T11:25:17.011 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.40616071100000001, 2024-09-13T11:25:17.011 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:25:17.011 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:25:17.011 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-09-13T11:25:17.011 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:25:17.011 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:17.011 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:25:17.012 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.897971+0000", 2024-09-13T11:25:17.012 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:17.012 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:17.012 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:17.012 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:25:17.012 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.897971+0000", 2024-09-13T11:25:17.012 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:17.012 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:17.012 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:17.012 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:25:17.012 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.897972+0000", 2024-09-13T11:25:17.012 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1429999999999999e-06 2024-09-13T11:25:17.013 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:17.013 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:17.013 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:25:17.013 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.897977+0000", 2024-09-13T11:25:17.013 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.5199999999999999e-06 2024-09-13T11:25:17.013 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:17.013 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:17.014 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:25:17.014 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:31.302500+0000", 2024-09-13T11:25:17.014 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.40452312400000001 2024-09-13T11:25:17.014 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:17.014 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:17.014 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:25:17.014 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:31.302513+0000", 2024-09-13T11:25:17.014 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2753e-05 2024-09-13T11:25:17.014 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:17.014 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:17.014 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:25:17.014 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:31.302645+0000", 2024-09-13T11:25:17.015 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00013292100000000001 2024-09-13T11:25:17.015 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:17.015 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:17.015 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:25:17.015 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:31.304132+0000", 2024-09-13T11:25:17.015 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0014862499999999999 2024-09-13T11:25:17.015 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:17.015 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:25:17.015 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:17.015 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:17.015 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:17.015 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPushReply(3.19 75/70 [PushReplyOp(3:9d1489b1:::benchmark_data_smithi177_31116_object18698:head)])", 2024-09-13T11:25:17.015 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:31.697212+0000", 2024-09-13T11:25:17.016 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 105.30627844200001, 2024-09-13T11:25:17.016 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.22850633200000001, 2024-09-13T11:25:17.016 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:25:17.016 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:25:17.016 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-09-13T11:25:17.016 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:25:17.016 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:17.016 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:25:17.016 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:31.697212+0000", 2024-09-13T11:25:17.016 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:17.016 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:17.016 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:17.016 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:25:17.017 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:31.697212+0000", 2024-09-13T11:25:17.017 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:17.017 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:17.017 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:17.017 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:25:17.017 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:31.697214+0000", 2024-09-13T11:25:17.017 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.7439999999999999e-06 2024-09-13T11:25:17.017 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:17.017 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:17.017 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:25:17.017 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:31.697224+0000", 2024-09-13T11:25:17.017 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0173000000000001e-05 2024-09-13T11:25:17.017 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:17.017 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:17.018 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:25:17.018 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:31.923931+0000", 2024-09-13T11:25:17.018 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.226706926 2024-09-13T11:25:17.018 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:17.018 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:17.018 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:25:17.018 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:31.923950+0000", 2024-09-13T11:25:17.018 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.8899e-05 2024-09-13T11:25:17.018 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:17.018 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:17.018 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:25:17.018 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:31.924449+0000", 2024-09-13T11:25:17.018 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00049907800000000002 2024-09-13T11:25:17.018 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:17.019 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:17.019 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:25:17.019 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:31.925718+0000", 2024-09-13T11:25:17.019 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001269512 2024-09-13T11:25:17.019 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:17.019 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:25:17.019 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:17.019 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:17.019 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:17.019 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPushReply(3.19 76/70 [PushReplyOp(3:9faf6d5f:::benchmark_data_smithi177_31116_object18954:head)])", 2024-09-13T11:25:17.019 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:32.042874+0000", 2024-09-13T11:25:17.019 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 104.960616573, 2024-09-13T11:25:17.019 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.40212404299999999, 2024-09-13T11:25:17.019 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:25:17.020 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:25:17.020 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-09-13T11:25:17.020 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:25:17.020 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:17.020 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:25:17.020 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.042874+0000", 2024-09-13T11:25:17.020 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:17.020 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:17.020 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:17.020 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:25:17.020 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.042874+0000", 2024-09-13T11:25:17.020 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:17.020 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:17.021 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:17.021 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:25:17.021 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.042875+0000", 2024-09-13T11:25:17.021 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.432e-06 2024-09-13T11:25:17.021 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:17.021 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:17.021 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:25:17.021 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.042886+0000", 2024-09-13T11:25:17.021 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0265000000000001e-05 2024-09-13T11:25:17.021 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:17.021 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:17.021 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:25:17.021 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.443455+0000", 2024-09-13T11:25:17.022 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.40056951400000002 2024-09-13T11:25:17.022 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:17.022 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:17.022 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:25:17.022 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.443480+0000", 2024-09-13T11:25:17.022 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.4929000000000001e-05 2024-09-13T11:25:17.022 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:17.022 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:17.022 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:25:17.022 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.443666+0000", 2024-09-13T11:25:17.022 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000186446 2024-09-13T11:25:17.022 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:17.022 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:17.022 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:25:17.023 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.444998+0000", 2024-09-13T11:25:17.023 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0013314570000000001 2024-09-13T11:25:17.023 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:17.023 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:25:17.023 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:17.023 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:17.023 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:17.023 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPush(3.1d 85/78 [PushOp(3:bc4b9336:::benchmark_data_smithi177_31116_object11944:head, version: 49'734, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:bc4b9336:::benchmark_data_smithi177_31116_object11944:head@49'734, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:25:17.023 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:42.225966+0000", 2024-09-13T11:25:17.023 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 94.777524360000001, 2024-09-13T11:25:17.023 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.23879947800000001, 2024-09-13T11:25:17.023 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:25:17.023 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:25:17.023 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:25:17.024 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:25:17.024 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:17.024 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:25:17.024 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.225966+0000", 2024-09-13T11:25:17.024 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:17.024 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:17.024 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:17.024 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:25:17.024 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.225966+0000", 2024-09-13T11:25:17.024 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:17.024 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:17.024 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:17.024 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:25:17.025 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.225969+0000", 2024-09-13T11:25:17.025 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.0400000000000001e-06 2024-09-13T11:25:17.025 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:17.025 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:17.025 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:25:17.025 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.226036+0000", 2024-09-13T11:25:17.025 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.6978999999999997e-05 2024-09-13T11:25:17.025 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:17.025 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:17.025 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:25:17.025 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.463110+0000", 2024-09-13T11:25:17.025 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.23707400000000001 2024-09-13T11:25:17.025 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:17.025 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:17.025 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:25:17.026 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.463140+0000", 2024-09-13T11:25:17.026 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.9575999999999999e-05 2024-09-13T11:25:17.026 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:17.026 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:17.026 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:25:17.026 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.463326+0000", 2024-09-13T11:25:17.026 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000186298 2024-09-13T11:25:17.026 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:17.026 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:17.026 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:25:17.026 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.463351+0000", 2024-09-13T11:25:17.026 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.4862999999999999e-05 2024-09-13T11:25:17.026 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:17.027 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:17.027 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:25:17.027 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.464766+0000", 2024-09-13T11:25:17.027 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001414722 2024-09-13T11:25:17.027 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:17.027 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:25:17.027 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:17.027 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:17.027 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:17.027 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPush(3.1d 89/78 [PushOp(3:bc1b7cfb:::benchmark_data_smithi177_31116_object22629:head, version: 55'1387, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:bc1b7cfb:::benchmark_data_smithi177_31116_object22629:head@55'1387, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:25:17.027 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:46.660855+0000", 2024-09-13T11:25:17.027 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 90.342635864000002, 2024-09-13T11:25:17.027 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.26622785599999998, 2024-09-13T11:25:17.027 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:25:17.028 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:25:17.028 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:25:17.028 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:25:17.028 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:17.028 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:25:17.028 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:46.660855+0000", 2024-09-13T11:25:17.028 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:17.028 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:17.028 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:17.028 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:25:17.028 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:46.660855+0000", 2024-09-13T11:25:17.028 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:17.028 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:17.028 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:17.029 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:25:17.029 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:46.660857+0000", 2024-09-13T11:25:17.029 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.6560000000000001e-06 2024-09-13T11:25:17.029 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:17.029 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:17.029 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:25:17.029 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:46.660931+0000", 2024-09-13T11:25:17.029 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.4041999999999993e-05 2024-09-13T11:25:17.029 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:17.029 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:17.029 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:25:17.029 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:46.924654+0000", 2024-09-13T11:25:17.029 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.26372239800000002 2024-09-13T11:25:17.029 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:17.030 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:17.030 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:25:17.030 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:46.924729+0000", 2024-09-13T11:25:17.030 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.5061999999999994e-05 2024-09-13T11:25:17.030 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:17.030 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:17.030 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:25:17.030 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:46.924986+0000", 2024-09-13T11:25:17.030 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00025762000000000002 2024-09-13T11:25:17.030 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:17.030 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:17.030 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:25:17.030 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:46.925013+0000", 2024-09-13T11:25:17.030 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.6179000000000001e-05 2024-09-13T11:25:17.031 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:17.031 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:17.031 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:25:17.031 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:46.927082+0000", 2024-09-13T11:25:17.031 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.002069899 2024-09-13T11:25:17.031 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:17.031 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:25:17.031 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:17.031 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:17.031 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:17.031 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPush(3.1d 90/78 [PushOp(3:bc0e7228:::benchmark_data_smithi177_31116_object24733:head, version: 55'1512, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:bc0e7228:::benchmark_data_smithi177_31116_object24733:head@55'1512, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:25:17.031 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:47.529126+0000", 2024-09-13T11:25:17.031 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 89.474364890000004, 2024-09-13T11:25:17.031 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.21343989499999999, 2024-09-13T11:25:17.032 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:25:17.032 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:25:17.032 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:25:17.032 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:25:17.032 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:17.032 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:25:17.032 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:47.529126+0000", 2024-09-13T11:25:17.032 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:17.032 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:17.032 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:17.032 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:25:17.032 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:47.529126+0000", 2024-09-13T11:25:17.032 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:17.032 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:17.033 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:17.033 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:25:17.033 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:47.529127+0000", 2024-09-13T11:25:17.033 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.252e-06 2024-09-13T11:25:17.033 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:17.033 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:17.033 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:25:17.033 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:47.529185+0000", 2024-09-13T11:25:17.033 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.8224999999999999e-05 2024-09-13T11:25:17.033 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:17.033 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:17.033 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:25:17.033 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:47.740216+0000", 2024-09-13T11:25:17.033 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.21103060800000001 2024-09-13T11:25:17.034 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:17.034 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:17.034 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:25:17.034 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:47.740248+0000", 2024-09-13T11:25:17.034 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.2821999999999998e-05 2024-09-13T11:25:17.034 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:17.034 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:17.034 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:25:17.034 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:47.740504+0000", 2024-09-13T11:25:17.034 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00025518399999999999 2024-09-13T11:25:17.034 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:17.034 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:17.034 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:25:17.034 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:47.740544+0000", 2024-09-13T11:25:17.035 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.0438999999999997e-05 2024-09-13T11:25:17.035 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:17.035 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:17.035 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:25:17.035 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:47.742565+0000", 2024-09-13T11:25:17.035 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0020213649999999998 2024-09-13T11:25:17.035 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:17.035 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:25:17.035 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:17.035 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:17.035 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:17.035 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:83199 3.1d 3:be2c6fae:::benchmark_data_smithi177_31116_object83198:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106)", 2024-09-13T11:25:17.035 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:24:04.297306+0000", 2024-09-13T11:25:17.036 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 72.706184938000007, 2024-09-13T11:25:17.036 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.25114587999999999, 2024-09-13T11:25:17.036 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:25:17.036 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:25:17.036 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:17.036 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:25:17.036 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:25:17.036 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:17.036 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 83199 2024-09-13T11:25:17.036 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:17.036 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:25:17.036 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:17.037 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:25:17.037 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.297306+0000", 2024-09-13T11:25:17.037 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:17.037 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:17.037 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:17.037 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:25:17.037 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.297306+0000", 2024-09-13T11:25:17.037 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:17.037 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:17.037 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:17.037 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:25:17.037 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.297308+0000", 2024-09-13T11:25:17.038 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.4150000000000002e-06 2024-09-13T11:25:17.038 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:17.038 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:17.038 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:25:17.038 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.297352+0000", 2024-09-13T11:25:17.038 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.4431000000000002e-05 2024-09-13T11:25:17.038 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:17.038 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:17.038 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:25:17.038 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.297369+0000", 2024-09-13T11:25:17.038 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.6685e-05 2024-09-13T11:25:17.038 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:17.039 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:17.039 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:25:17.039 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.297376+0000", 2024-09-13T11:25:17.039 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.7299999999999999e-06 2024-09-13T11:25:17.039 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:17.039 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:17.039 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:25:17.039 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.297427+0000", 2024-09-13T11:25:17.039 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.1461000000000001e-05 2024-09-13T11:25:17.039 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:17.039 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:17.039 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:25:17.040 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.297528+0000", 2024-09-13T11:25:17.040 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00010028599999999999 2024-09-13T11:25:17.040 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:17.040 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:17.040 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 15", 2024-09-13T11:25:17.040 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.297685+0000", 2024-09-13T11:25:17.040 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00015760500000000001 2024-09-13T11:25:17.040 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:17.040 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:17.040 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:25:17.040 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.300779+0000", 2024-09-13T11:25:17.040 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0030936700000000002 2024-09-13T11:25:17.041 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:17.041 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:17.041 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:25:17.041 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.548279+0000", 2024-09-13T11:25:17.041 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.24750059999999999 2024-09-13T11:25:17.041 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:17.041 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:17.041 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:25:17.041 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.548373+0000", 2024-09-13T11:25:17.041 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.3451999999999999e-05 2024-09-13T11:25:17.041 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:17.041 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:17.042 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:25:17.042 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.548451+0000", 2024-09-13T11:25:17.042 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.8545000000000001e-05 2024-09-13T11:25:17.042 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:17.042 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:25:17.042 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:17.042 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:17.042 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:17.042 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:84621 3.1d 3:becd72aa:::benchmark_data_smithi177_31116_object84620:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-09-13T11:25:17.042 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:24:05.741008+0000", 2024-09-13T11:25:17.042 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 71.262482938000005, 2024-09-13T11:25:17.042 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.17994101100000001, 2024-09-13T11:25:17.043 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:25:17.043 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:25:17.043 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:17.043 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:25:17.043 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:25:17.043 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:17.043 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 84621 2024-09-13T11:25:17.043 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:17.043 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:25:17.043 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:17.043 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:25:17.043 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.741008+0000", 2024-09-13T11:25:17.044 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:17.044 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:17.044 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:17.044 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:25:17.044 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.741008+0000", 2024-09-13T11:25:17.044 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:17.044 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:17.044 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:17.044 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:25:17.044 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.741010+0000", 2024-09-13T11:25:17.044 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.898e-06 2024-09-13T11:25:17.044 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:17.045 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:17.045 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:25:17.045 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.741061+0000", 2024-09-13T11:25:17.045 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.0898e-05 2024-09-13T11:25:17.045 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:17.045 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:17.045 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:25:17.045 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.741085+0000", 2024-09-13T11:25:17.045 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.3558e-05 2024-09-13T11:25:17.045 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:17.045 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:17.045 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:25:17.045 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.741093+0000", 2024-09-13T11:25:17.046 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.3890000000000005e-06 2024-09-13T11:25:17.046 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:17.046 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:17.046 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:25:17.046 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.741178+0000", 2024-09-13T11:25:17.046 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.4792999999999999e-05 2024-09-13T11:25:17.046 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:17.046 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:17.046 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:25:17.046 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.741386+0000", 2024-09-13T11:25:17.046 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000207505 2024-09-13T11:25:17.046 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:17.046 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:17.047 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 15", 2024-09-13T11:25:17.047 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.741597+0000", 2024-09-13T11:25:17.047 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00021158200000000001 2024-09-13T11:25:17.047 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:17.047 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:17.047 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:25:17.047 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.743392+0000", 2024-09-13T11:25:17.047 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00179518 2024-09-13T11:25:17.047 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:17.047 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:17.047 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:25:17.047 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.920866+0000", 2024-09-13T11:25:17.048 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.17747373699999999 2024-09-13T11:25:17.048 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:17.048 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:17.048 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:25:17.048 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.920913+0000", 2024-09-13T11:25:17.048 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.6451e-05 2024-09-13T11:25:17.048 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:17.048 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:17.048 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:25:17.048 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.920949+0000", 2024-09-13T11:25:17.048 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.6019999999999997e-05 2024-09-13T11:25:17.048 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:17.049 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:25:17.049 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:17.049 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:17.049 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:17.049 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:88685 3.1d 3:baceb770:::benchmark_data_smithi177_31116_object88684:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110)", 2024-09-13T11:25:17.049 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:24:08.926194+0000", 2024-09-13T11:25:17.049 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 68.077296294000007, 2024-09-13T11:25:17.049 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.16678378799999999, 2024-09-13T11:25:17.049 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:25:17.049 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:25:17.049 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:17.049 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:25:17.050 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:25:17.050 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:17.050 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 88685 2024-09-13T11:25:17.050 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:17.050 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:25:17.050 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:17.050 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:25:17.050 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:08.926194+0000", 2024-09-13T11:25:17.050 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:17.050 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:17.050 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:17.050 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:25:17.050 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:08.926194+0000", 2024-09-13T11:25:17.051 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:17.051 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:17.051 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:17.051 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:25:17.051 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:08.926196+0000", 2024-09-13T11:25:17.051 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.533e-06 2024-09-13T11:25:17.051 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:17.051 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:17.051 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:25:17.051 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:08.926253+0000", 2024-09-13T11:25:17.051 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.6906e-05 2024-09-13T11:25:17.051 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:17.051 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:17.051 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:25:17.052 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:08.926262+0000", 2024-09-13T11:25:17.052 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.3530000000000001e-06 2024-09-13T11:25:17.052 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:17.052 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:17.052 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:25:17.052 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:08.926268+0000", 2024-09-13T11:25:17.052 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.2890000000000003e-06 2024-09-13T11:25:17.052 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:17.052 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:17.052 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:25:17.052 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:08.928510+0000", 2024-09-13T11:25:17.052 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0022416519999999998 2024-09-13T11:25:17.052 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:17.053 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:17.053 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:25:17.053 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:08.928748+0000", 2024-09-13T11:25:17.053 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00023773800000000001 2024-09-13T11:25:17.053 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:17.053 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:17.053 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 15", 2024-09-13T11:25:17.053 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:08.928953+0000", 2024-09-13T11:25:17.053 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00020532999999999999 2024-09-13T11:25:17.053 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:17.053 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:17.053 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:25:17.053 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:08.930702+0000", 2024-09-13T11:25:17.053 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001749163 2024-09-13T11:25:17.054 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:17.054 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:17.054 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:25:17.054 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:09.092878+0000", 2024-09-13T11:25:17.054 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.16217616900000001 2024-09-13T11:25:17.054 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:17.054 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:17.054 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:25:17.054 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:09.092932+0000", 2024-09-13T11:25:17.054 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.3523999999999998e-05 2024-09-13T11:25:17.054 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:17.054 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:17.054 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:25:17.055 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:09.092978+0000", 2024-09-13T11:25:17.055 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.6131000000000003e-05 2024-09-13T11:25:17.055 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:17.055 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:25:17.055 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:17.055 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:17.055 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:17.055 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPush(3.25 124/120 [PushOp(3:a6d4ee3f:::benchmark_data_smithi177_31116_object14712:head, version: 51'872, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:a6d4ee3f:::benchmark_data_smithi177_31116_object14712:head@51'872, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:25:17.055 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:24:31.996253+0000", 2024-09-13T11:25:17.055 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 45.007237166000003, 2024-09-13T11:25:17.055 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.17694863499999999, 2024-09-13T11:25:17.055 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:25:17.055 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:25:17.055 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:25:17.056 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:25:17.056 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:17.056 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:25:17.056 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:31.996253+0000", 2024-09-13T11:25:17.056 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:17.056 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:17.056 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:17.056 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:25:17.056 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:31.996253+0000", 2024-09-13T11:25:17.056 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:17.056 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:17.056 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:17.057 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:25:17.057 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:31.996257+0000", 2024-09-13T11:25:17.057 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.2629999999999999e-06 2024-09-13T11:25:17.057 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:17.057 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:17.057 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:25:17.057 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:31.996359+0000", 2024-09-13T11:25:17.057 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000102456 2024-09-13T11:25:17.057 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:17.057 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:17.057 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:25:17.057 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:32.172313+0000", 2024-09-13T11:25:17.057 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.17595388000000001 2024-09-13T11:25:17.058 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:17.058 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:17.058 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:25:17.058 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:32.172326+0000", 2024-09-13T11:25:17.058 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2632999999999999e-05 2024-09-13T11:25:17.058 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:17.058 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:17.058 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:25:17.058 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:32.172423+0000", 2024-09-13T11:25:17.058 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.7084999999999996e-05 2024-09-13T11:25:17.058 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:17.058 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:17.058 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:25:17.059 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:32.172438+0000", 2024-09-13T11:25:17.059 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5775999999999999e-05 2024-09-13T11:25:17.059 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:17.059 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:17.059 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:25:17.059 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:32.173202+0000", 2024-09-13T11:25:17.059 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00076354199999999998 2024-09-13T11:25:17.059 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:17.059 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:25:17.059 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:17.059 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:17.059 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:17.059 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPush(3.25 125/120 [PushOp(3:a4446742:::benchmark_data_smithi177_31116_object15486:head, version: 51'933, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:a4446742:::benchmark_data_smithi177_31116_object15486:head@51'933, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:25:17.059 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:24:32.725521+0000", 2024-09-13T11:25:17.059 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 44.277969855999999, 2024-09-13T11:25:17.060 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.16753559600000001, 2024-09-13T11:25:17.060 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:25:17.060 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:25:17.060 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:25:17.060 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:25:17.060 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:17.060 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:25:17.060 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:32.725521+0000", 2024-09-13T11:25:17.060 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:17.060 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:17.060 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:17.060 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:25:17.060 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:32.725521+0000", 2024-09-13T11:25:17.061 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:17.061 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:17.061 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:17.061 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:25:17.061 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:32.725523+0000", 2024-09-13T11:25:17.061 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.689e-06 2024-09-13T11:25:17.061 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:17.061 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:17.061 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:25:17.061 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:32.725566+0000", 2024-09-13T11:25:17.061 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.3177999999999999e-05 2024-09-13T11:25:17.061 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:17.061 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:17.061 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:25:17.062 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:32.892091+0000", 2024-09-13T11:25:17.062 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.16652452200000001 2024-09-13T11:25:17.062 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:17.062 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:17.062 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:25:17.062 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:32.892107+0000", 2024-09-13T11:25:17.062 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.6324000000000002e-05 2024-09-13T11:25:17.062 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:17.062 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:17.062 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:25:17.062 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:32.892226+0000", 2024-09-13T11:25:17.062 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000118814 2024-09-13T11:25:17.062 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:17.062 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:17.063 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:25:17.063 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:32.892241+0000", 2024-09-13T11:25:17.063 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4899000000000001e-05 2024-09-13T11:25:17.063 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:17.063 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:17.063 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:25:17.063 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:32.893056+0000", 2024-09-13T11:25:17.063 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00081517000000000004 2024-09-13T11:25:17.063 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:17.063 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:25:17.063 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:17.063 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:17.063 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:17.063 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:126039 3.29 3:96100181:::benchmark_data_smithi177_31116_object126038:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e130)", 2024-09-13T11:25:17.064 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:24:43.012785+0000", 2024-09-13T11:25:17.064 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 33.990705816000002, 2024-09-13T11:25:17.064 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.26685311900000003, 2024-09-13T11:25:17.064 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:25:17.064 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:25:17.064 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:17.064 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:25:17.064 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:25:17.064 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:17.064 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 126039 2024-09-13T11:25:17.064 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:17.064 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:25:17.064 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:17.064 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:25:17.065 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:43.012785+0000", 2024-09-13T11:25:17.065 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:17.065 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:17.065 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:17.065 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:25:17.065 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:43.012785+0000", 2024-09-13T11:25:17.065 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:17.065 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:17.065 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:17.065 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:25:17.065 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:43.012787+0000", 2024-09-13T11:25:17.065 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.875e-06 2024-09-13T11:25:17.065 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:17.065 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:17.066 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:25:17.066 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:43.012826+0000", 2024-09-13T11:25:17.066 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.9483999999999997e-05 2024-09-13T11:25:17.066 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:17.066 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:17.066 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:25:17.066 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:43.012839+0000", 2024-09-13T11:25:17.066 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3405e-05 2024-09-13T11:25:17.066 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:17.066 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:17.066 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:25:17.066 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:43.012845+0000", 2024-09-13T11:25:17.066 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.9100000000000002e-06 2024-09-13T11:25:17.066 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:17.067 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:17.067 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:25:17.067 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:43.012903+0000", 2024-09-13T11:25:17.067 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.8131000000000003e-05 2024-09-13T11:25:17.067 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:17.067 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:17.067 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:25:17.067 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:43.013066+0000", 2024-09-13T11:25:17.067 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000162058 2024-09-13T11:25:17.067 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:17.067 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:17.067 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 4,5", 2024-09-13T11:25:17.067 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:43.013235+0000", 2024-09-13T11:25:17.067 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00016929600000000001 2024-09-13T11:25:17.068 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:17.068 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:17.068 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:25:17.068 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:43.017063+0000", 2024-09-13T11:25:17.068 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0038283890000000002 2024-09-13T11:25:17.068 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:17.068 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:17.068 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:25:17.068 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:43.017086+0000", 2024-09-13T11:25:17.068 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.2934000000000001e-05 2024-09-13T11:25:17.068 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:17.068 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:17.068 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:25:17.068 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:43.278804+0000", 2024-09-13T11:25:17.069 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.26171751799999998 2024-09-13T11:25:17.069 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:17.069 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:17.069 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:25:17.069 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:43.278862+0000", 2024-09-13T11:25:17.069 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.8431999999999999e-05 2024-09-13T11:25:17.069 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:17.069 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:17.069 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:25:17.069 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:43.279638+0000", 2024-09-13T11:25:17.069 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00077568699999999995 2024-09-13T11:25:17.069 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:17.069 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:25:17.069 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:17.070 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:17.070 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:17.070 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop_reply(client.4665.0:126039 3.29 e130/120 ondisk, result = 0)", 2024-09-13T11:25:17.070 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:24:43.021691+0000", 2024-09-13T11:25:17.070 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 33.981799889999998, 2024-09-13T11:25:17.070 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.25795579699999999, 2024-09-13T11:25:17.070 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:25:17.070 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:25:17.070 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:25:17.070 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:25:17.070 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:17.070 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:25:17.070 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:43.021691+0000", 2024-09-13T11:25:17.070 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:17.071 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:17.071 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:17.071 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:25:17.071 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:43.021691+0000", 2024-09-13T11:25:17.071 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:17.071 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:17.071 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:17.071 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:25:17.071 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:43.021692+0000", 2024-09-13T11:25:17.071 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3880000000000001e-06 2024-09-13T11:25:17.071 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:17.071 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:17.071 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:25:17.072 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:43.021700+0000", 2024-09-13T11:25:17.072 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.9310000000000008e-06 2024-09-13T11:25:17.072 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:17.072 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:17.072 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:25:17.072 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:43.278719+0000", 2024-09-13T11:25:17.072 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.25701909299999998 2024-09-13T11:25:17.072 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:17.072 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:17.072 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:25:17.072 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:43.278728+0000", 2024-09-13T11:25:17.072 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.5000000000000005e-06 2024-09-13T11:25:17.072 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:17.072 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:17.073 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:25:17.073 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:43.278786+0000", 2024-09-13T11:25:17.073 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.7640999999999999e-05 2024-09-13T11:25:17.073 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:17.073 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:17.073 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:25:17.073 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:43.278796+0000", 2024-09-13T11:25:17.073 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.9199999999999999e-06 2024-09-13T11:25:17.073 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:17.073 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:17.073 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:25:17.073 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:43.279646+0000", 2024-09-13T11:25:17.073 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00085032399999999996 2024-09-13T11:25:17.073 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:17.074 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:25:17.074 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:17.074 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:17.074 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:17.074 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPushReply(3.29 134/120 [PushReplyOp(3:95648f22:::benchmark_data_smithi177_31116_object7446:head)])", 2024-09-13T11:25:17.074 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:24:47.110443+0000", 2024-09-13T11:25:17.074 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 29.893047769999999, 2024-09-13T11:25:17.074 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.171602386, 2024-09-13T11:25:17.074 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:25:17.074 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:25:17.074 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-09-13T11:25:17.074 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:25:17.074 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:17.075 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:25:17.075 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:47.110443+0000", 2024-09-13T11:25:17.075 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:17.075 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:17.075 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:17.075 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:25:17.075 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:47.110443+0000", 2024-09-13T11:25:17.075 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:17.075 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:17.075 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:17.075 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:25:17.075 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:47.110444+0000", 2024-09-13T11:25:17.075 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.192e-06 2024-09-13T11:25:17.075 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:17.075 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:17.076 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:25:17.076 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:47.110449+0000", 2024-09-13T11:25:17.076 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.0869999999999997e-06 2024-09-13T11:25:17.076 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:17.076 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:17.076 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:25:17.076 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:47.280321+0000", 2024-09-13T11:25:17.076 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.169872157 2024-09-13T11:25:17.076 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:17.076 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:17.076 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:25:17.076 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:47.280337+0000", 2024-09-13T11:25:17.076 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5656000000000001e-05 2024-09-13T11:25:17.077 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:17.077 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:17.077 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:25:17.077 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:47.280414+0000", 2024-09-13T11:25:17.077 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.7083999999999996e-05 2024-09-13T11:25:17.077 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:17.077 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:17.077 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:25:17.077 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:47.282045+0000", 2024-09-13T11:25:17.077 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0016312099999999999 2024-09-13T11:25:17.077 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:17.077 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:25:17.077 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:17.077 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:17.078 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:17.078 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.4665.0:138294 3.9 e140/117 3:90012dab:::benchmark_data_smithi177_31116_object138293:head v 140'5246, mlcod=140'5243)", 2024-09-13T11:25:17.078 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:24:54.252227+0000", 2024-09-13T11:25:17.078 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 22.751263401999999, 2024-09-13T11:25:17.078 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.32919836400000002, 2024-09-13T11:25:17.078 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:25:17.078 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:25:17.078 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:17.078 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:25:17.078 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:17.078 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:25:17.078 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:54.252227+0000", 2024-09-13T11:25:17.078 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:17.078 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:17.079 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:17.079 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:25:17.079 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:54.252227+0000", 2024-09-13T11:25:17.079 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:17.079 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:17.079 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:17.079 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:25:17.079 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:54.252228+0000", 2024-09-13T11:25:17.079 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.8899999999999998e-07 2024-09-13T11:25:17.079 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:17.079 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:17.079 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:25:17.079 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:54.252276+0000", 2024-09-13T11:25:17.080 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.8137999999999999e-05 2024-09-13T11:25:17.080 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:17.080 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:17.080 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:25:17.080 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:54.578216+0000", 2024-09-13T11:25:17.080 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.32594014399999999 2024-09-13T11:25:17.080 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:17.080 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:17.080 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:25:17.080 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:54.578235+0000", 2024-09-13T11:25:17.080 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.8943000000000001e-05 2024-09-13T11:25:17.080 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:17.080 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:17.081 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:25:17.081 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:54.578396+0000", 2024-09-13T11:25:17.081 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000160922 2024-09-13T11:25:17.081 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:17.081 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:17.081 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:25:17.081 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:54.578457+0000", 2024-09-13T11:25:17.081 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.0665999999999998e-05 2024-09-13T11:25:17.081 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:17.081 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:17.081 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:25:17.081 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:54.581339+0000", 2024-09-13T11:25:17.081 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0028826839999999999 2024-09-13T11:25:17.081 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:17.082 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:17.082 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:25:17.082 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:54.581425+0000", 2024-09-13T11:25:17.082 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.6077999999999998e-05 2024-09-13T11:25:17.082 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:17.082 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:25:17.082 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:17.082 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:17.082 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:17.082 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPushReply(3.29 143/120 [PushReplyOp(3:973d2047:::benchmark_data_smithi177_31116_object49749:head)])", 2024-09-13T11:25:17.082 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:24:57.387106+0000", 2024-09-13T11:25:17.082 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 19.616384074999999, 2024-09-13T11:25:17.082 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.64813637000000002, 2024-09-13T11:25:17.082 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:25:17.083 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:25:17.083 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-09-13T11:25:17.083 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:25:17.083 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:17.083 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:25:17.083 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:57.387106+0000", 2024-09-13T11:25:17.083 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:17.083 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:17.083 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:17.083 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:25:17.083 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:57.387106+0000", 2024-09-13T11:25:17.083 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:17.083 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:17.083 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:17.083 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:25:17.084 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:57.387116+0000", 2024-09-13T11:25:17.084 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.1530000000000003e-06 2024-09-13T11:25:17.084 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:17.084 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:17.084 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:25:17.084 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:57.387127+0000", 2024-09-13T11:25:17.084 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1197e-05 2024-09-13T11:25:17.084 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:17.084 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:17.084 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:25:17.084 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:58.034264+0000", 2024-09-13T11:25:17.084 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.64713691799999995 2024-09-13T11:25:17.084 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:17.084 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:17.085 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:25:17.085 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:58.034273+0000", 2024-09-13T11:25:17.085 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.0929999999999992e-06 2024-09-13T11:25:17.085 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:17.085 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:17.085 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:25:17.085 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:58.034321+0000", 2024-09-13T11:25:17.085 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.8396999999999998e-05 2024-09-13T11:25:17.085 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:17.085 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:17.085 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:25:17.085 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:58.035243+0000", 2024-09-13T11:25:17.085 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00092161200000000004 2024-09-13T11:25:17.085 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:17.086 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:25:17.086 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:17.086 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:17.086 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:17.086 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPushReply(3.29 144/120 [PushReplyOp(3:94a6fc3a:::benchmark_data_smithi177_31116_object50245:head)])", 2024-09-13T11:25:17.086 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:24:58.320977+0000", 2024-09-13T11:25:17.086 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 18.682513491999998, 2024-09-13T11:25:17.086 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.31110569599999999, 2024-09-13T11:25:17.086 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:25:17.086 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:25:17.086 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-09-13T11:25:17.086 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:25:17.086 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:17.087 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:25:17.087 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:58.320977+0000", 2024-09-13T11:25:17.087 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:17.087 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:17.087 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:17.087 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:25:17.087 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:58.320977+0000", 2024-09-13T11:25:17.087 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:17.087 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:17.087 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:17.087 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:25:17.087 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:58.320978+0000", 2024-09-13T11:25:17.087 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.8700000000000004e-07 2024-09-13T11:25:17.087 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:17.088 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:17.088 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:25:17.088 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:58.320985+0000", 2024-09-13T11:25:17.088 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.742e-06 2024-09-13T11:25:17.088 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:17.088 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:17.088 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:25:17.088 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:58.631147+0000", 2024-09-13T11:25:17.088 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.31016212900000001 2024-09-13T11:25:17.088 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:17.088 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:17.088 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:25:17.088 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:58.631159+0000", 2024-09-13T11:25:17.088 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1946999999999999e-05 2024-09-13T11:25:17.089 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:17.089 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:17.089 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:25:17.089 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:58.631257+0000", 2024-09-13T11:25:17.089 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.8096000000000004e-05 2024-09-13T11:25:17.089 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:17.089 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:17.089 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:25:17.089 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:58.632083+0000", 2024-09-13T11:25:17.089 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00082589499999999999 2024-09-13T11:25:17.089 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:17.089 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:25:17.089 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:17.090 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:17.090 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:17.090 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:147568 3.1d 3:b82a2ed2:::benchmark_data_smithi177_31116_object7265:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e148)", 2024-09-13T11:25:17.090 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:25:04.213894+0000", 2024-09-13T11:25:17.090 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 12.789596721000001, 2024-09-13T11:25:17.090 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.16897441999999999, 2024-09-13T11:25:17.090 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:25:17.090 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:25:17.090 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:17.090 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:25:17.090 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:25:17.090 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:17.090 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 147568 2024-09-13T11:25:17.090 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:17.091 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:25:17.091 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:17.091 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:25:17.091 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:04.213894+0000", 2024-09-13T11:25:17.091 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:17.091 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:17.091 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:17.091 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:25:17.091 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:04.213894+0000", 2024-09-13T11:25:17.091 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:17.092 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:17.092 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:17.092 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:25:17.092 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:04.213898+0000", 2024-09-13T11:25:17.092 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.9049999999999999e-06 2024-09-13T11:25:17.092 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:17.092 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:17.092 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:25:17.092 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:04.213908+0000", 2024-09-13T11:25:17.092 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0069e-05 2024-09-13T11:25:17.092 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:17.092 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:17.092 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:25:17.093 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:04.213931+0000", 2024-09-13T11:25:17.093 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.3538999999999999e-05 2024-09-13T11:25:17.093 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:17.093 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:17.093 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:25:17.093 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:04.213941+0000", 2024-09-13T11:25:17.093 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0073e-05 2024-09-13T11:25:17.093 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:17.093 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:17.093 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:25:17.093 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:04.214039+0000", 2024-09-13T11:25:17.093 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.7666999999999998e-05 2024-09-13T11:25:17.093 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:17.094 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:17.094 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:25:17.094 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:04.214226+0000", 2024-09-13T11:25:17.094 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000187166 2024-09-13T11:25:17.094 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:17.094 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:17.094 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 15", 2024-09-13T11:25:17.094 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:04.214365+0000", 2024-09-13T11:25:17.094 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000139016 2024-09-13T11:25:17.094 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:17.094 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:17.094 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:25:17.094 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:04.217864+0000", 2024-09-13T11:25:17.095 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.003499177 2024-09-13T11:25:17.095 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:17.095 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:17.095 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:25:17.095 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:04.382775+0000", 2024-09-13T11:25:17.095 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.16491091499999999 2024-09-13T11:25:17.095 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:17.095 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:17.095 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:25:17.095 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:04.382823+0000", 2024-09-13T11:25:17.095 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.8232000000000002e-05 2024-09-13T11:25:17.095 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:17.095 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:17.095 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:25:17.096 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:04.382868+0000", 2024-09-13T11:25:17.096 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.4660999999999998e-05 2024-09-13T11:25:17.096 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:17.096 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:25:17.096 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:17.096 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:17.096 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:17.096 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop_reply(client.4665.0:147568 3.1d e148/102 ondisk, result = 0)", 2024-09-13T11:25:17.096 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:25:04.217811+0000", 2024-09-13T11:25:17.096 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 12.785679308000001, 2024-09-13T11:25:17.096 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.16506296300000001, 2024-09-13T11:25:17.096 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:25:17.096 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:25:17.096 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:25:17.097 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:25:17.097 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:17.097 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:25:17.097 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:04.217811+0000", 2024-09-13T11:25:17.097 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:17.097 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:17.097 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:17.097 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:25:17.097 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:04.217811+0000", 2024-09-13T11:25:17.097 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:17.097 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:17.097 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:17.097 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:25:17.097 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:04.217813+0000", 2024-09-13T11:25:17.098 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.748e-06 2024-09-13T11:25:17.098 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:17.098 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:17.098 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:25:17.098 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:04.217823+0000", 2024-09-13T11:25:17.098 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.8519999999999999e-06 2024-09-13T11:25:17.098 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:17.098 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:17.098 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:25:17.098 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:04.382674+0000", 2024-09-13T11:25:17.098 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.16485156200000001 2024-09-13T11:25:17.098 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:17.098 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:17.098 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:25:17.099 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:04.382687+0000", 2024-09-13T11:25:17.099 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2286000000000001e-05 2024-09-13T11:25:17.099 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:17.099 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:17.099 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:25:17.099 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:04.382762+0000", 2024-09-13T11:25:17.099 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.5902000000000006e-05 2024-09-13T11:25:17.099 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:17.099 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:17.099 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:25:17.099 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:04.382770+0000", 2024-09-13T11:25:17.099 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.4159999999999998e-06 2024-09-13T11:25:17.099 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:17.099 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:17.099 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:25:17.100 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:04.382874+0000", 2024-09-13T11:25:17.100 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00010419700000000001 2024-09-13T11:25:17.100 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:17.100 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:25:17.100 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:17.100 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:17.100 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:25:17.100 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-13T11:25:17.100 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T11:25:17.102 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-13T11:25:17.103 INFO:tasks.ceph.osd.2.smithi162.stderr:2024-09-13T11:25:17.100+0000 7fe92b35a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26313) UID: 0 2024-09-13T11:25:17.203 INFO:tasks.ceph.osd.2.smithi162.stderr:2024-09-13T11:25:17.200+0000 7fe92b35a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26313) UID: 0 2024-09-13T11:25:17.292 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-13T11:25:17.292 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [ 2024-09-13T11:25:17.292 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:17.292 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:164890 3.10 3:0d7f38cf:::benchmark_data_smithi177_31116_object24587:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e158)", 2024-09-13T11:25:17.292 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:25:17.291268+0000", 2024-09-13T11:25:17.292 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 0.000446861, 2024-09-13T11:25:17.292 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00050198899999999999, 2024-09-13T11:25:17.292 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:25:17.293 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:25:17.293 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "waiting for sub ops", 2024-09-13T11:25:17.293 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:25:17.293 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:25:17.293 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:17.293 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 164890 2024-09-13T11:25:17.293 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:17.293 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:25:17.293 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:17.293 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:25:17.293 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:17.291268+0000", 2024-09-13T11:25:17.293 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:17.293 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:17.294 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:17.294 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:25:17.294 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:17.291268+0000", 2024-09-13T11:25:17.294 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:17.294 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:17.294 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:17.294 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:25:17.294 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:17.291270+0000", 2024-09-13T11:25:17.294 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.413e-06 2024-09-13T11:25:17.294 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:17.294 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:17.294 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:25:17.294 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:17.291279+0000", 2024-09-13T11:25:17.295 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.2470000000000008e-06 2024-09-13T11:25:17.295 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:17.295 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:17.295 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:25:17.295 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:17.291300+0000", 2024-09-13T11:25:17.295 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.1690000000000001e-05 2024-09-13T11:25:17.295 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:17.295 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:17.295 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:25:17.295 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:17.291312+0000", 2024-09-13T11:25:17.295 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1586999999999999e-05 2024-09-13T11:25:17.295 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:17.296 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:17.296 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:25:17.296 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:17.291386+0000", 2024-09-13T11:25:17.296 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.3911000000000004e-05 2024-09-13T11:25:17.296 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:17.296 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:17.296 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:25:17.296 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:17.291531+0000", 2024-09-13T11:25:17.296 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00014468700000000001 2024-09-13T11:25:17.296 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:17.296 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:17.296 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 0", 2024-09-13T11:25:17.297 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:17.291643+0000", 2024-09-13T11:25:17.297 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000112037 2024-09-13T11:25:17.297 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:17.297 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:25:17.297 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:17.297 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:17.297 INFO:teuthology.orchestra.run.smithi162.stdout: ], 2024-09-13T11:25:17.297 INFO:teuthology.orchestra.run.smithi162.stdout: "num_ops": 1 2024-09-13T11:25:17.297 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-13T11:25:17.298 INFO:teuthology.orchestra.run.smithi001.stdout:osd.0: {} 2024-09-13T11:25:17.298 INFO:teuthology.orchestra.run.smithi001.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T11:25:17.304 INFO:tasks.ceph.osd.12.smithi001.stderr:2024-09-13T11:25:17.303+0000 7f892c6e6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26314) UID: 0 2024-09-13T11:25:17.308 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-13T11:25:17.328 INFO:teuthology.orchestra.run.smithi001.stdout:osd.1: {} 2024-09-13T11:25:17.328 INFO:teuthology.orchestra.run.smithi001.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T11:25:17.359 INFO:teuthology.orchestra.run.smithi001.stdout:osd.2: {} 2024-09-13T11:25:17.359 INFO:teuthology.orchestra.run.smithi001.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T11:25:17.389 INFO:teuthology.orchestra.run.smithi001.stdout:osd.3: {} 2024-09-13T11:25:17.389 INFO:teuthology.orchestra.run.smithi001.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T11:25:17.404 INFO:tasks.ceph.osd.3.smithi177.stderr:2024-09-13T11:25:17.402+0000 7f10adba4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26574) UID: 0 2024-09-13T11:25:17.416 INFO:teuthology.orchestra.run.smithi001.stdout:osd.4: {} 2024-09-13T11:25:17.417 INFO:teuthology.orchestra.run.smithi001.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T11:25:17.443 INFO:teuthology.orchestra.run.smithi001.stdout:osd.5: {} 2024-09-13T11:25:17.443 INFO:teuthology.orchestra.run.smithi001.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T11:25:17.468 INFO:teuthology.orchestra.run.smithi001.stdout:osd.6: {} 2024-09-13T11:25:17.469 INFO:teuthology.orchestra.run.smithi001.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T11:25:17.480 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-13T11:25:17.480 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:25:17.495 INFO:teuthology.orchestra.run.smithi001.stdout:osd.7: {} 2024-09-13T11:25:17.495 INFO:teuthology.orchestra.run.smithi001.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T11:25:17.497 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-13T11:25:17.519 INFO:teuthology.orchestra.run.smithi001.stdout:osd.8: {} 2024-09-13T11:25:17.519 INFO:teuthology.orchestra.run.smithi001.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-13T11:25:17.545 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-09-13T11:25:17.544+0000 7f97befc9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26315) UID: 0 2024-09-13T11:25:17.546 INFO:teuthology.orchestra.run.smithi001.stdout:osd.9: {} 2024-09-13T11:25:17.546 INFO:teuthology.orchestra.run.smithi001.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-13T11:25:17.572 INFO:teuthology.orchestra.run.smithi001.stdout:osd.10: {} 2024-09-13T11:25:17.572 INFO:teuthology.orchestra.run.smithi001.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-13T11:25:17.599 INFO:teuthology.orchestra.run.smithi001.stdout:osd.11: {} 2024-09-13T11:25:17.600 INFO:teuthology.orchestra.run.smithi001.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-13T11:25:17.606 INFO:tasks.ceph.osd.8.smithi001.stderr:2024-09-13T11:25:17.603+0000 7fb9b48ae640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26312) UID: 0 2024-09-13T11:25:17.626 INFO:teuthology.orchestra.run.smithi001.stdout:osd.12: {} 2024-09-13T11:25:17.626 INFO:teuthology.orchestra.run.smithi001.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-13T11:25:17.631 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-13T11:25:17.631 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:25:17.647 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-09-13T11:25:17.650 INFO:teuthology.orchestra.run.smithi001.stdout:osd.13: {} 2024-09-13T11:25:17.651 INFO:teuthology.orchestra.run.smithi001.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-13T11:25:17.676 INFO:teuthology.orchestra.run.smithi001.stdout:osd.14: {} 2024-09-13T11:25:17.676 INFO:teuthology.orchestra.run.smithi001.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-13T11:25:17.704 INFO:teuthology.orchestra.run.smithi001.stdout:osd.15: {} 2024-09-13T11:25:17.705 INFO:teuthology.orchestra.run.smithi001.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-13T11:25:17.737 INFO:tasks.ceph.osd.10.smithi162.stderr:2024-09-13T11:25:17.736+0000 7fb59c9f0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26317) UID: 0 2024-09-13T11:25:17.808 INFO:tasks.ceph.osd.12.smithi001.stderr:2024-09-13T11:25:17.807+0000 7f892c6e6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26314) UID: 0 2024-09-13T11:25:17.824 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-13T11:25:17.824 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:25:17.843 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-09-13T11:25:17.908 INFO:tasks.ceph.osd.13.smithi092.stderr:2024-09-13T11:25:17.901+0000 7f3e5afea640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26350) UID: 0 2024-09-13T11:25:18.009 INFO:tasks.ceph.osd.15.smithi177.stderr:2024-09-13T11:25:18.006+0000 7f9d6ce77640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26578) UID: 0 2024-09-13T11:25:18.013 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-13T11:25:18.013 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:25:18.027 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-09-13T11:25:18.110 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-09-13T11:25:18.108+0000 7f97befc9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26315) UID: 0 2024-09-13T11:25:18.130 INFO:teuthology.orchestra.run.smithi001.stderr:reweighted osd.7 to 1 (10000) 2024-09-13T11:25:18.143 INFO:tasks.ceph.osd.9.smithi092.stderr:2024-09-13T11:25:18.137+0000 7ff59ddf9640 -1 osd.9 158 handle_osd_map injecting map crc failure 2024-09-13T11:25:18.152 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 8 1 2024-09-13T11:25:18.160 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-13T11:25:18.160 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:25:18.177 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-13T11:25:18.219 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-09-13T11:25:18.216+0000 7f97befc9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26315) UID: 0 2024-09-13T11:25:18.311 INFO:tasks.ceph.osd.12.smithi001.stderr:2024-09-13T11:25:18.307+0000 7f892c6e6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26314) UID: 0 2024-09-13T11:25:18.315 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-09-13T11:25:18.315 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:25:18.335 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-13T11:25:18.412 INFO:tasks.ceph.osd.1.smithi092.stderr:2024-09-13T11:25:18.405+0000 7fe7789c5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33718) UID: 0 2024-09-13T11:25:18.477 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-09-13T11:25:18.477 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:25:18.494 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-13T11:25:18.513 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-13T11:25:18.511+0000 7f6a68775640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26307) UID: 0 2024-09-13T11:25:18.614 INFO:tasks.ceph.osd.10.smithi162.stderr:2024-09-13T11:25:18.612+0000 7fb59c9f0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26317) UID: 0 2024-09-13T11:25:18.619 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-09-13T11:25:18.619 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:25:18.632 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-13T11:25:18.715 INFO:tasks.ceph.osd.9.smithi092.stderr:2024-09-13T11:25:18.709+0000 7ff5ab3d0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26348) UID: 0 2024-09-13T11:25:18.751 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-09-13T11:25:18.752 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:25:18.769 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-13T11:25:18.816 INFO:tasks.ceph.osd.10.smithi162.stderr:2024-09-13T11:25:18.812+0000 7fb59c9f0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26317) UID: 0 2024-09-13T11:25:18.903 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-09-13T11:25:18.903 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:25:18.918 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-13T11:25:18.919 INFO:tasks.ceph.osd.2.smithi162.stderr:2024-09-13T11:25:18.916+0000 7fe92b35a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26313) UID: 0 2024-09-13T11:25:18.956 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-13T11:25:19.019 INFO:tasks.ceph.osd.13.smithi092.stderr:2024-09-13T11:25:19.013+0000 7f3e5afea640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26350) UID: 0 2024-09-13T11:25:19.043 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-09-13T11:25:19.043 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:25:19.061 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-09-13T11:25:19.120 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-09-13T11:25:19.116+0000 7f97befc9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26315) UID: 0 2024-09-13T11:25:19.139 INFO:teuthology.orchestra.run.smithi001.stderr:reweighted osd.8 to 1 (10000) 2024-09-13T11:25:19.149 INFO:tasks.ceph.osd.3.smithi177.stderr:2024-09-13T11:25:19.146+0000 7f10a25d1640 -1 osd.3 159 handle_osd_map injecting map crc failure 2024-09-13T11:25:19.151 INFO:tasks.ceph.osd.8.smithi001.stderr:2024-09-13T11:25:19.147+0000 7fb9a92db640 -1 osd.8 159 handle_osd_map injecting map crc failure 2024-09-13T11:25:19.151 INFO:tasks.ceph.osd.5.smithi092.stderr:2024-09-13T11:25:19.145+0000 7f6f00132640 -1 osd.5 159 handle_osd_map injecting map crc failure 2024-09-13T11:25:19.162 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 9 1 2024-09-13T11:25:19.177 INFO:tasks.ceph.osd.2.smithi162.stderr:2024-09-13T11:25:19.172+0000 7fe921d8b640 -1 osd.2 159 handle_osd_map injecting map crc failure 2024-09-13T11:25:19.189 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-09-13T11:25:19.189 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:25:19.202 INFO:tasks.ceph.osd.1.smithi092.stderr:2024-09-13T11:25:19.197+0000 7fe76d3f2640 -1 osd.1 159 handle_osd_map injecting map crc failure 2024-09-13T11:25:19.206 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-09-13T11:25:19.221 INFO:tasks.ceph.osd.12.smithi001.stderr:2024-09-13T11:25:19.219+0000 7f892c6e6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26314) UID: 0 2024-09-13T11:25:19.321 INFO:tasks.ceph.osd.12.smithi001.stderr:2024-09-13T11:25:19.319+0000 7f892c6e6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26314) UID: 0 2024-09-13T11:25:19.328 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-09-13T11:25:19.328 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:25:19.347 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-09-13T11:25:19.423 INFO:tasks.ceph.osd.13.smithi092.stderr:2024-09-13T11:25:19.417+0000 7f3e5afea640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26350) UID: 0 2024-09-13T11:25:19.482 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-09-13T11:25:19.482 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:25:19.500 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_ops_in_flight 2024-09-13T11:25:19.544 INFO:tasks.ceph.osd.15.smithi177.stderr:2024-09-13T11:25:19.542+0000 7f9d6ce77640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26578) UID: 0 2024-09-13T11:25:19.625 INFO:tasks.ceph.osd.15.smithi177.stderr:2024-09-13T11:25:19.622+0000 7f9d6ce77640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26578) UID: 0 2024-09-13T11:25:19.630 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-09-13T11:25:19.630 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:25:19.646 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_blocked_ops 2024-09-13T11:25:19.726 INFO:tasks.ceph.osd.10.smithi162.stderr:2024-09-13T11:25:19.724+0000 7fb59c9f0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26317) UID: 0 2024-09-13T11:25:19.727 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T11:25:19.812 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-09-13T11:25:19.812 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:25:19.827 INFO:tasks.ceph.osd.4.smithi001.stderr:2024-09-13T11:25:19.823+0000 7ff8e0218640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26310) UID: 0 2024-09-13T11:25:19.829 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_historic_ops 2024-09-13T11:25:19.920 INFO:teuthology.orchestra.run.smithi001.stdout:osd.0: {} 2024-09-13T11:25:19.920 INFO:teuthology.orchestra.run.smithi001.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T11:25:19.928 INFO:tasks.ceph.osd.2.smithi162.stderr:2024-09-13T11:25:19.924+0000 7fe92b35a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26313) UID: 0 2024-09-13T11:25:19.944 INFO:teuthology.orchestra.run.smithi001.stdout:osd.1: {} 2024-09-13T11:25:19.945 INFO:teuthology.orchestra.run.smithi001.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T11:25:19.970 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-09-13T11:25:19.970 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:25:19.975 INFO:teuthology.orchestra.run.smithi001.stdout:osd.2: {} 2024-09-13T11:25:19.975 INFO:teuthology.orchestra.run.smithi001.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T11:25:19.990 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-13T11:25:20.002 INFO:teuthology.orchestra.run.smithi001.stdout:osd.3: {} 2024-09-13T11:25:20.002 INFO:teuthology.orchestra.run.smithi001.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T11:25:20.027 INFO:teuthology.orchestra.run.smithi001.stdout:osd.4: {} 2024-09-13T11:25:20.028 INFO:teuthology.orchestra.run.smithi001.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T11:25:20.037 INFO:tasks.ceph.osd.1.smithi092.stderr:2024-09-13T11:25:20.033+0000 7fe7789c5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33718) UID: 0 2024-09-13T11:25:20.050 INFO:teuthology.orchestra.run.smithi001.stdout:osd.5: {} 2024-09-13T11:25:20.051 INFO:teuthology.orchestra.run.smithi001.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T11:25:20.078 INFO:teuthology.orchestra.run.smithi001.stdout:osd.6: {} 2024-09-13T11:25:20.078 INFO:teuthology.orchestra.run.smithi001.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T11:25:20.104 INFO:teuthology.orchestra.run.smithi001.stdout:osd.7: {} 2024-09-13T11:25:20.105 INFO:teuthology.orchestra.run.smithi001.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T11:25:20.128 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-09-13T11:25:20.128 INFO:teuthology.orchestra.run.smithi092.stdout: "ops": [ 2024-09-13T11:25:20.128 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:20.128 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop(client.4665.0:168063 3.24 e160/133)", 2024-09-13T11:25:20.128 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:25:20.123608+0000", 2024-09-13T11:25:20.128 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 0.0038878480000000002, 2024-09-13T11:25:20.128 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0039051210000000001, 2024-09-13T11:25:20.128 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:25:20.128 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:25:20.128 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "reached pg", 2024-09-13T11:25:20.129 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:25:20.129 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:20.129 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:25:20.129 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:20.123608+0000", 2024-09-13T11:25:20.129 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:20.129 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:20.129 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:20.129 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:25:20.129 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:20.123608+0000", 2024-09-13T11:25:20.129 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:20.130 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:20.130 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:20.130 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:25:20.130 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:20.123609+0000", 2024-09-13T11:25:20.130 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.3289999999999999e-06 2024-09-13T11:25:20.130 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:20.130 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:20.130 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:25:20.130 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:20.123617+0000", 2024-09-13T11:25:20.130 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.4540000000000001e-06 2024-09-13T11:25:20.130 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:20.130 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:20.130 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:25:20.131 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:20.127449+0000", 2024-09-13T11:25:20.131 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0038326829999999999 2024-09-13T11:25:20.131 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:20.131 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:20.131 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:25:20.131 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:20.127457+0000", 2024-09-13T11:25:20.131 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.7789999999999996e-06 2024-09-13T11:25:20.131 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:20.131 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:20.131 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:25:20.131 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:20.127515+0000", 2024-09-13T11:25:20.131 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.7812999999999997e-05 2024-09-13T11:25:20.131 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:20.132 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:25:20.132 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:20.132 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:20.132 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:20.132 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:167989 3.2b 3:d56b7ee0:::benchmark_data_smithi177_31116_object27686:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160)", 2024-09-13T11:25:20.132 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:25:20.058615+0000", 2024-09-13T11:25:20.132 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 0.068880417999999999, 2024-09-13T11:25:20.132 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.068956775999999997, 2024-09-13T11:25:20.132 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:25:20.132 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:25:20.132 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "waiting for sub ops", 2024-09-13T11:25:20.132 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:25:20.132 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:25:20.133 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:20.133 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 167989 2024-09-13T11:25:20.133 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:20.133 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:25:20.133 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:20.133 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:25:20.133 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:20.058615+0000", 2024-09-13T11:25:20.133 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:20.133 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:20.133 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:20.133 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:25:20.133 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:20.058615+0000", 2024-09-13T11:25:20.133 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:20.134 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:20.134 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:20.134 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:25:20.134 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:20.058619+0000", 2024-09-13T11:25:20.134 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.061e-06 2024-09-13T11:25:20.134 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:20.134 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:20.134 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:25:20.134 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:20.058628+0000", 2024-09-13T11:25:20.134 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.0629999999999995e-06 2024-09-13T11:25:20.134 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:20.134 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:20.135 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:25:20.135 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:20.058646+0000", 2024-09-13T11:25:20.135 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.8828999999999999e-05 2024-09-13T11:25:20.135 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:20.135 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:20.135 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:25:20.135 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:20.058655+0000", 2024-09-13T11:25:20.135 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.4679999999999992e-06 2024-09-13T11:25:20.135 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:20.135 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:20.135 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:25:20.135 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:20.058724+0000", 2024-09-13T11:25:20.135 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.9170000000000004e-05 2024-09-13T11:25:20.136 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:20.136 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:20.136 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:25:20.136 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:20.058898+0000", 2024-09-13T11:25:20.136 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00017405400000000001 2024-09-13T11:25:20.136 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:20.136 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:20.136 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 7", 2024-09-13T11:25:20.136 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:20.059045+0000", 2024-09-13T11:25:20.136 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00014687599999999999 2024-09-13T11:25:20.136 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:20.136 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:20.136 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:25:20.136 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:20.060569+0000", 2024-09-13T11:25:20.137 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0015240150000000001 2024-09-13T11:25:20.137 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:20.137 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:25:20.137 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:20.137 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:20.137 INFO:teuthology.orchestra.run.smithi092.stdout: ], 2024-09-13T11:25:20.137 INFO:teuthology.orchestra.run.smithi092.stdout: "num_ops": 2 2024-09-13T11:25:20.137 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-09-13T11:25:20.138 INFO:teuthology.orchestra.run.smithi001.stdout:osd.8: {} 2024-09-13T11:25:20.138 INFO:tasks.ceph.osd.13.smithi092.stderr:2024-09-13T11:25:20.121+0000 7f3e5afea640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26350) UID: 0 2024-09-13T11:25:20.138 INFO:teuthology.orchestra.run.smithi001.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-13T11:25:20.149 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-13T11:25:20.155 INFO:teuthology.orchestra.run.smithi001.stdout:osd.9: {} 2024-09-13T11:25:20.155 INFO:teuthology.orchestra.run.smithi001.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-13T11:25:20.156 INFO:teuthology.orchestra.run.smithi001.stderr:reweighted osd.9 to 1 (10000) 2024-09-13T11:25:20.159 INFO:teuthology.orchestra.run.smithi001.stderr:noscrub is unset 2024-09-13T11:25:20.160 INFO:tasks.ceph.osd.12.smithi001.stderr:2024-09-13T11:25:20.159+0000 7f891f10f640 -1 osd.12 160 handle_osd_map injecting map crc failure 2024-09-13T11:25:20.177 INFO:tasks.ceph.osd.9.smithi092.stderr:2024-09-13T11:25:20.169+0000 7ff59bdf5640 -1 osd.9 160 handle_osd_map injecting map crc failure 2024-09-13T11:25:20.188 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 10 1 2024-09-13T11:25:20.198 INFO:teuthology.orchestra.run.smithi001.stdout:osd.10: {} 2024-09-13T11:25:20.199 INFO:teuthology.orchestra.run.smithi001.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-13T11:25:20.230 INFO:teuthology.orchestra.run.smithi001.stdout:osd.11: {} 2024-09-13T11:25:20.231 INFO:teuthology.orchestra.run.smithi001.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-13T11:25:20.231 INFO:tasks.ceph.osd.4.smithi001.stderr:2024-09-13T11:25:20.227+0000 7ff8e0218640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26310) UID: 0 2024-09-13T11:25:20.258 INFO:teuthology.orchestra.run.smithi001.stdout:osd.12: {} 2024-09-13T11:25:20.259 INFO:teuthology.orchestra.run.smithi001.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-13T11:25:20.295 INFO:teuthology.orchestra.run.smithi001.stdout:osd.13: {} 2024-09-13T11:25:20.296 INFO:teuthology.orchestra.run.smithi001.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-13T11:25:20.324 INFO:teuthology.orchestra.run.smithi001.stdout:osd.14: {} 2024-09-13T11:25:20.325 INFO:teuthology.orchestra.run.smithi001.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-13T11:25:20.330 INFO:tasks.ceph.osd.11.smithi177.stderr:2024-09-13T11:25:20.326+0000 7f24792c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26580) UID: 0 2024-09-13T11:25:20.333 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-09-13T11:25:20.333 INFO:teuthology.orchestra.run.smithi092.stdout: "ops": [], 2024-09-13T11:25:20.333 INFO:teuthology.orchestra.run.smithi092.stdout: "complaint_time": 30, 2024-09-13T11:25:20.333 INFO:teuthology.orchestra.run.smithi092.stdout: "num_blocked_ops": 0 2024-09-13T11:25:20.333 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-09-13T11:25:20.354 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-13T11:25:20.354 INFO:teuthology.orchestra.run.smithi001.stdout:osd.15: {} 2024-09-13T11:25:20.355 INFO:teuthology.orchestra.run.smithi001.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-13T11:25:20.431 INFO:tasks.ceph.osd.13.smithi092.stderr:2024-09-13T11:25:20.425+0000 7f3e5afea640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26350) UID: 0 2024-09-13T11:25:20.493 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-09-13T11:25:20.493 INFO:teuthology.orchestra.run.smithi092.stdout: "size": 20, 2024-09-13T11:25:20.493 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 600, 2024-09-13T11:25:20.494 INFO:teuthology.orchestra.run.smithi092.stdout: "ops": [ 2024-09-13T11:25:20.494 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:20.494 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:3 3.4 3.125e09e4 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e40)", 2024-09-13T11:25:20.494 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:46.304641+0000", 2024-09-13T11:25:20.494 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 154.18672570300001, 2024-09-13T11:25:20.494 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.374372119, 2024-09-13T11:25:20.494 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:25:20.494 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:25:20.494 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "reached pg", 2024-09-13T11:25:20.494 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:25:20.494 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:25:20.494 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:20.495 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 3 2024-09-13T11:25:20.495 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:20.495 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:25:20.495 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:20.495 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:25:20.495 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.304641+0000", 2024-09-13T11:25:20.495 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:20.495 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:20.495 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:20.495 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:25:20.495 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.304641+0000", 2024-09-13T11:25:20.495 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:20.495 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:20.495 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:20.496 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:25:20.496 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.304644+0000", 2024-09-13T11:25:20.496 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.5130000000000001e-06 2024-09-13T11:25:20.496 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:20.496 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:20.496 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:25:20.496 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305058+0000", 2024-09-13T11:25:20.496 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000414277 2024-09-13T11:25:20.497 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:20.497 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:20.497 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:25:20.497 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305124+0000", 2024-09-13T11:25:20.497 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.5813000000000002e-05 2024-09-13T11:25:20.497 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:20.497 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:20.497 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:25:20.497 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305156+0000", 2024-09-13T11:25:20.497 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.1489000000000003e-05 2024-09-13T11:25:20.497 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:20.498 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:20.498 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:25:20.498 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.678939+0000", 2024-09-13T11:25:20.498 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.37378339999999999 2024-09-13T11:25:20.498 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:20.498 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:20.498 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:25:20.498 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.679013+0000", 2024-09-13T11:25:20.498 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.3627000000000005e-05 2024-09-13T11:25:20.498 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:20.498 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:25:20.498 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:20.498 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:20.499 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:20.499 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:29 3.4 3.b7bbfbe4 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-09-13T11:25:20.499 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:46.305215+0000", 2024-09-13T11:25:20.499 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 154.1861514, 2024-09-13T11:25:20.499 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.37387964600000001, 2024-09-13T11:25:20.499 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:25:20.499 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:25:20.499 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "reached pg", 2024-09-13T11:25:20.499 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:25:20.499 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:25:20.499 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:20.499 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 29 2024-09-13T11:25:20.499 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:20.499 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:25:20.500 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:20.500 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:25:20.500 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305215+0000", 2024-09-13T11:25:20.500 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:20.500 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:20.500 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:20.500 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:25:20.500 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305215+0000", 2024-09-13T11:25:20.500 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:20.500 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:20.500 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:20.500 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:25:20.500 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305218+0000", 2024-09-13T11:25:20.501 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.7030000000000002e-06 2024-09-13T11:25:20.501 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:20.501 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:20.501 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:25:20.501 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305366+0000", 2024-09-13T11:25:20.501 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00014825400000000001 2024-09-13T11:25:20.501 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:20.501 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:20.501 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:25:20.501 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305386+0000", 2024-09-13T11:25:20.501 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.9689000000000001e-05 2024-09-13T11:25:20.501 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:20.501 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:20.502 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:25:20.502 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305403+0000", 2024-09-13T11:25:20.502 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.7600000000000001e-05 2024-09-13T11:25:20.502 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:20.502 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:20.502 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:25:20.502 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.679067+0000", 2024-09-13T11:25:20.502 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.37366339100000001 2024-09-13T11:25:20.502 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:20.502 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:20.502 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:25:20.502 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.679095+0000", 2024-09-13T11:25:20.502 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.8008999999999999e-05 2024-09-13T11:25:20.502 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:20.503 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:25:20.503 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:20.503 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:20.503 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:20.503 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:30 3.4 3.547ec0f4 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-09-13T11:25:20.503 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:46.305453+0000", 2024-09-13T11:25:20.503 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 154.18591327600001, 2024-09-13T11:25:20.503 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.373764914, 2024-09-13T11:25:20.504 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:25:20.504 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:25:20.504 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "reached pg", 2024-09-13T11:25:20.504 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:25:20.504 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:25:20.504 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:20.504 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 30 2024-09-13T11:25:20.504 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:20.504 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:25:20.504 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:20.504 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:25:20.505 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305453+0000", 2024-09-13T11:25:20.505 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:20.505 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:20.505 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:20.505 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:25:20.505 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305453+0000", 2024-09-13T11:25:20.505 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:20.505 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:20.505 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:20.505 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:25:20.505 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305456+0000", 2024-09-13T11:25:20.505 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.9560000000000002e-06 2024-09-13T11:25:20.505 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:20.505 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:20.506 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:25:20.506 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305597+0000", 2024-09-13T11:25:20.506 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00014122700000000001 2024-09-13T11:25:20.506 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:20.506 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:20.506 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:25:20.506 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305615+0000", 2024-09-13T11:25:20.506 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.8139999999999999e-05 2024-09-13T11:25:20.506 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:20.506 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:20.506 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:25:20.506 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305632+0000", 2024-09-13T11:25:20.506 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.6101999999999999e-05 2024-09-13T11:25:20.507 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:20.507 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:20.507 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:25:20.507 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.679195+0000", 2024-09-13T11:25:20.507 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.373563486 2024-09-13T11:25:20.507 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:20.507 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:20.507 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:25:20.507 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.679218+0000", 2024-09-13T11:25:20.507 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.3003e-05 2024-09-13T11:25:20.507 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:20.507 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:25:20.507 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:20.507 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:20.508 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:20.508 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:70 3.4 3.1404a604 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-09-13T11:25:20.508 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:46.305665+0000", 2024-09-13T11:25:20.508 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 154.18570112800001, 2024-09-13T11:25:20.508 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.37365550199999997, 2024-09-13T11:25:20.508 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:25:20.508 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:25:20.508 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "reached pg", 2024-09-13T11:25:20.508 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:25:20.508 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:25:20.508 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:20.508 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 70 2024-09-13T11:25:20.508 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:20.509 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:25:20.509 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:20.509 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:25:20.509 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305665+0000", 2024-09-13T11:25:20.509 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:20.509 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:20.509 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:20.509 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:25:20.509 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305665+0000", 2024-09-13T11:25:20.509 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:20.509 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:20.509 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:20.509 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:25:20.509 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305668+0000", 2024-09-13T11:25:20.510 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.5890000000000001e-06 2024-09-13T11:25:20.510 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:20.510 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:20.510 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:25:20.510 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305791+0000", 2024-09-13T11:25:20.510 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000123545 2024-09-13T11:25:20.510 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:20.510 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:20.510 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:25:20.510 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305808+0000", 2024-09-13T11:25:20.510 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.668e-05 2024-09-13T11:25:20.511 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:20.511 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:20.511 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:25:20.511 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305824+0000", 2024-09-13T11:25:20.511 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.6385e-05 2024-09-13T11:25:20.511 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:20.511 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:20.511 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:25:20.511 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.679305+0000", 2024-09-13T11:25:20.511 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.37348059900000002 2024-09-13T11:25:20.511 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:20.511 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:20.512 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:25:20.512 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.679321+0000", 2024-09-13T11:25:20.512 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.5704000000000001e-05 2024-09-13T11:25:20.512 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:20.512 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:25:20.512 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:20.512 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:20.512 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:20.512 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:25408 3.4 3:294e23bd:::benchmark_data_smithi177_31116_object25407:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e56)", 2024-09-13T11:25:20.512 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:04.495455+0000", 2024-09-13T11:25:20.512 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 135.995911476, 2024-09-13T11:25:20.512 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.29897367600000002, 2024-09-13T11:25:20.512 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:25:20.513 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:25:20.513 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:20.513 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:25:20.513 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:25:20.513 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:20.513 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 25408 2024-09-13T11:25:20.513 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:20.513 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:25:20.513 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:20.513 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:25:20.513 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.495455+0000", 2024-09-13T11:25:20.513 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:20.513 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:20.514 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:20.514 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:25:20.514 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.495455+0000", 2024-09-13T11:25:20.514 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:20.514 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:20.514 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:20.514 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:25:20.514 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.495459+0000", 2024-09-13T11:25:20.514 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.5630000000000002e-06 2024-09-13T11:25:20.514 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:20.514 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:20.514 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:25:20.514 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.495545+0000", 2024-09-13T11:25:20.515 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.5704000000000005e-05 2024-09-13T11:25:20.515 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:20.515 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:20.515 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:25:20.515 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.495581+0000", 2024-09-13T11:25:20.515 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.5689000000000003e-05 2024-09-13T11:25:20.516 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:20.516 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:20.516 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:25:20.516 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.495597+0000", 2024-09-13T11:25:20.516 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.6053e-05 2024-09-13T11:25:20.516 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:20.516 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:20.516 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:25:20.516 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.495721+0000", 2024-09-13T11:25:20.516 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00012379099999999999 2024-09-13T11:25:20.516 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:20.516 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:20.516 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:25:20.517 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.495940+0000", 2024-09-13T11:25:20.517 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00021967699999999999 2024-09-13T11:25:20.517 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:20.517 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:20.517 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 12", 2024-09-13T11:25:20.517 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.496162+0000", 2024-09-13T11:25:20.517 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000221917 2024-09-13T11:25:20.517 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:20.517 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:20.517 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:25:20.517 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.498699+0000", 2024-09-13T11:25:20.517 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0025366690000000001 2024-09-13T11:25:20.518 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:20.518 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:20.518 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:25:20.518 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.794139+0000", 2024-09-13T11:25:20.518 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.295440025 2024-09-13T11:25:20.518 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:20.518 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:20.518 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:25:20.518 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.794306+0000", 2024-09-13T11:25:20.518 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00016689799999999999 2024-09-13T11:25:20.518 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:20.518 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:20.518 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:25:20.519 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.794429+0000", 2024-09-13T11:25:20.519 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00012269000000000001 2024-09-13T11:25:20.519 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:20.519 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:25:20.519 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:20.519 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:20.519 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:20.519 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "MOSDPGPushReply(3.14 75/70 [PushReplyOp(3:2f14928a:::benchmark_data_smithi177_31116_object15123:head)])", 2024-09-13T11:25:20.519 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:31.569662+0000", 2024-09-13T11:25:20.519 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 108.92170470400001, 2024-09-13T11:25:20.519 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.35759148299999999, 2024-09-13T11:25:20.519 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:25:20.519 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:25:20.520 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "reached pg", 2024-09-13T11:25:20.520 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:25:20.520 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:20.520 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:25:20.520 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:31.569662+0000", 2024-09-13T11:25:20.520 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:20.520 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:20.520 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:20.520 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:25:20.520 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:31.569662+0000", 2024-09-13T11:25:20.520 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:20.520 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:20.520 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:20.520 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:25:20.521 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:31.569663+0000", 2024-09-13T11:25:20.521 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.333e-06 2024-09-13T11:25:20.521 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:20.521 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:20.521 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:25:20.521 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:31.569667+0000", 2024-09-13T11:25:20.521 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.8929999999999998e-06 2024-09-13T11:25:20.521 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:20.521 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:20.521 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:25:20.522 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:31.923221+0000", 2024-09-13T11:25:20.522 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.35355413699999999 2024-09-13T11:25:20.522 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:20.522 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:20.522 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:25:20.522 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:31.923230+0000", 2024-09-13T11:25:20.522 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.6829999999999997e-06 2024-09-13T11:25:20.522 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:20.522 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:20.522 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:25:20.522 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:31.923807+0000", 2024-09-13T11:25:20.522 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00057686800000000002 2024-09-13T11:25:20.522 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:20.522 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:20.523 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:25:20.523 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:31.927253+0000", 2024-09-13T11:25:20.523 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.003446569 2024-09-13T11:25:20.523 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:20.523 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:25:20.523 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:20.523 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:20.523 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:20.523 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "MOSDPGPushReply(3.18 85/70 [PushReplyOp(3:1e410119:::benchmark_data_smithi177_31116_object35700:head)])", 2024-09-13T11:25:20.524 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:42.216614+0000", 2024-09-13T11:25:20.524 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 98.274752359999994, 2024-09-13T11:25:20.524 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.54260090699999997, 2024-09-13T11:25:20.524 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:25:20.524 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:25:20.524 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "reached pg", 2024-09-13T11:25:20.524 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:25:20.524 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:20.524 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:25:20.524 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.216614+0000", 2024-09-13T11:25:20.524 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:20.524 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:20.525 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:20.525 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:25:20.525 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.216614+0000", 2024-09-13T11:25:20.525 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:20.525 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:20.525 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:20.525 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:25:20.525 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.216615+0000", 2024-09-13T11:25:20.525 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.0200000000000001e-07 2024-09-13T11:25:20.525 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:20.525 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:20.525 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:25:20.526 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.216619+0000", 2024-09-13T11:25:20.526 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.805e-06 2024-09-13T11:25:20.526 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:20.526 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:20.526 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:25:20.526 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.757476+0000", 2024-09-13T11:25:20.526 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.54085762000000004 2024-09-13T11:25:20.526 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:20.526 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:20.526 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:25:20.526 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.757498+0000", 2024-09-13T11:25:20.527 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.1936000000000001e-05 2024-09-13T11:25:20.527 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:20.527 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:20.527 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:25:20.527 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.757678+0000", 2024-09-13T11:25:20.527 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00018006599999999999 2024-09-13T11:25:20.527 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:20.527 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:20.527 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:25:20.527 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.759215+0000", 2024-09-13T11:25:20.527 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0015366780000000001 2024-09-13T11:25:20.527 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:20.527 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:25:20.528 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:20.528 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:20.528 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:20.528 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "MOSDPGPushReply(3.1b 85/78 [PushReplyOp(3:dabda1cc:::benchmark_data_smithi177_31116_object12988:head)])", 2024-09-13T11:25:20.528 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:42.485171+0000", 2024-09-13T11:25:20.528 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 98.006195452, 2024-09-13T11:25:20.528 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.308420427, 2024-09-13T11:25:20.528 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:25:20.528 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:25:20.528 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "reached pg", 2024-09-13T11:25:20.528 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:25:20.528 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:20.528 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:25:20.528 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.485171+0000", 2024-09-13T11:25:20.529 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:20.529 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:20.529 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:20.529 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:25:20.529 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.485171+0000", 2024-09-13T11:25:20.529 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:20.529 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:20.529 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:20.529 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:25:20.529 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.485173+0000", 2024-09-13T11:25:20.529 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.1569999999999998e-06 2024-09-13T11:25:20.529 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:20.529 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:20.529 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:25:20.530 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.485182+0000", 2024-09-13T11:25:20.530 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.6759999999999996e-06 2024-09-13T11:25:20.530 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:20.530 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:20.530 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:25:20.530 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.791336+0000", 2024-09-13T11:25:20.530 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.30615446499999999 2024-09-13T11:25:20.530 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:20.530 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:20.530 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:25:20.530 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.791369+0000", 2024-09-13T11:25:20.530 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.3155999999999999e-05 2024-09-13T11:25:20.530 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:20.531 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:20.531 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:25:20.531 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.791472+0000", 2024-09-13T11:25:20.531 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00010247900000000001 2024-09-13T11:25:20.531 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:20.531 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:20.531 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:25:20.531 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.793591+0000", 2024-09-13T11:25:20.531 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.002119494 2024-09-13T11:25:20.531 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:20.531 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:25:20.531 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:20.531 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:20.532 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:20.532 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:64066 3.1b 3:dd84e0ec:::benchmark_data_smithi177_31116_object64065:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89)", 2024-09-13T11:25:20.532 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:46.633608+0000", 2024-09-13T11:25:20.532 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 93.857758012999994, 2024-09-13T11:25:20.532 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.32564315399999999, 2024-09-13T11:25:20.532 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:25:20.532 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:25:20.532 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:20.532 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:25:20.532 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:25:20.532 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:20.532 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 64066 2024-09-13T11:25:20.532 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:20.532 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:25:20.533 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:20.533 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:25:20.533 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.633608+0000", 2024-09-13T11:25:20.533 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:20.533 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:20.533 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:20.533 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:25:20.533 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.633608+0000", 2024-09-13T11:25:20.533 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:20.533 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:20.533 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:20.533 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:25:20.533 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.633612+0000", 2024-09-13T11:25:20.534 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.856e-06 2024-09-13T11:25:20.534 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:20.534 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:20.534 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:25:20.534 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.633683+0000", 2024-09-13T11:25:20.534 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.1000999999999993e-05 2024-09-13T11:25:20.534 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:20.534 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:20.534 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:25:20.534 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.633711+0000", 2024-09-13T11:25:20.534 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.8204999999999998e-05 2024-09-13T11:25:20.534 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:20.534 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:20.535 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:25:20.535 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.633732+0000", 2024-09-13T11:25:20.535 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.0662999999999999e-05 2024-09-13T11:25:20.535 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:20.535 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:20.535 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:25:20.535 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.635573+0000", 2024-09-13T11:25:20.535 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.001841365 2024-09-13T11:25:20.535 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:20.535 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:20.535 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:25:20.535 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.635995+0000", 2024-09-13T11:25:20.535 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00042113699999999998 2024-09-13T11:25:20.536 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:20.536 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:20.536 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 12,15", 2024-09-13T11:25:20.536 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.636475+0000", 2024-09-13T11:25:20.536 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00048067699999999999 2024-09-13T11:25:20.536 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:20.536 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:20.536 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:25:20.536 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.640247+0000", 2024-09-13T11:25:20.536 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0037716450000000001 2024-09-13T11:25:20.536 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:20.536 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:20.537 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:25:20.537 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.649308+0000", 2024-09-13T11:25:20.537 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0090613240000000008 2024-09-13T11:25:20.537 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:20.537 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:20.537 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:25:20.537 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.956912+0000", 2024-09-13T11:25:20.537 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.30760338500000001 2024-09-13T11:25:20.537 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:20.537 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:20.537 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:25:20.537 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.957156+0000", 2024-09-13T11:25:20.537 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000244031 2024-09-13T11:25:20.538 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:20.538 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:20.538 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:25:20.538 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.959252+0000", 2024-09-13T11:25:20.538 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0020958650000000001 2024-09-13T11:25:20.538 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:20.538 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:25:20.538 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:20.538 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:20.538 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:20.538 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop_reply(client.4665.0:64066 3.1b e89/78 ondisk, result = 0)", 2024-09-13T11:25:20.538 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:46.651596+0000", 2024-09-13T11:25:20.539 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 93.839770181999995, 2024-09-13T11:25:20.539 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.30767612599999999, 2024-09-13T11:25:20.539 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:25:20.539 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:25:20.539 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-09-13T11:25:20.539 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:25:20.539 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:20.539 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:25:20.539 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.651596+0000", 2024-09-13T11:25:20.539 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:20.539 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:20.539 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:20.540 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:25:20.540 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.651596+0000", 2024-09-13T11:25:20.540 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:20.540 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:20.540 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:20.540 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:25:20.540 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.651600+0000", 2024-09-13T11:25:20.540 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.6969999999999999e-06 2024-09-13T11:25:20.540 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:20.540 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:20.540 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:25:20.540 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.651611+0000", 2024-09-13T11:25:20.540 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.1516e-05 2024-09-13T11:25:20.540 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:20.541 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:20.541 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:25:20.541 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.948118+0000", 2024-09-13T11:25:20.541 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.29650663500000002 2024-09-13T11:25:20.541 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:20.541 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:20.541 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:25:20.541 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.948137+0000", 2024-09-13T11:25:20.541 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.8529000000000002e-05 2024-09-13T11:25:20.541 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:20.541 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:20.541 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:25:20.541 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.956870+0000", 2024-09-13T11:25:20.542 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0087336380000000002 2024-09-13T11:25:20.542 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:20.542 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:20.542 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:25:20.542 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.956893+0000", 2024-09-13T11:25:20.542 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.317e-05 2024-09-13T11:25:20.542 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:20.542 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:20.542 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:25:20.542 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.959272+0000", 2024-09-13T11:25:20.542 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0023789409999999999 2024-09-13T11:25:20.542 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:20.542 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:25:20.542 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:20.543 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:20.543 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:20.543 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "MOSDPGPushReply(3.1b 91/78 [PushReplyOp(3:d9ac6889:::benchmark_data_smithi177_31116_object22254:head)])", 2024-09-13T11:25:20.543 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:48.392729+0000", 2024-09-13T11:25:20.543 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 92.098637267000001, 2024-09-13T11:25:20.543 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.30924993099999998, 2024-09-13T11:25:20.543 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:25:20.543 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:25:20.543 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "reached pg", 2024-09-13T11:25:20.543 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:25:20.543 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:20.543 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:25:20.543 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:48.392729+0000", 2024-09-13T11:25:20.543 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:20.544 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:20.544 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:20.544 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:25:20.544 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:48.392729+0000", 2024-09-13T11:25:20.544 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:20.544 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:20.544 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:20.544 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:25:20.544 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:48.392730+0000", 2024-09-13T11:25:20.544 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.336e-06 2024-09-13T11:25:20.544 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:20.544 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:20.544 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:25:20.544 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:48.392739+0000", 2024-09-13T11:25:20.545 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.8580000000000006e-06 2024-09-13T11:25:20.545 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:20.545 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:20.545 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:25:20.545 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:48.700180+0000", 2024-09-13T11:25:20.545 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.30744035600000003 2024-09-13T11:25:20.545 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:20.545 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:20.545 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:25:20.545 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:48.700197+0000", 2024-09-13T11:25:20.545 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.7649999999999999e-05 2024-09-13T11:25:20.545 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:20.545 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:20.545 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:25:20.546 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:48.700398+0000", 2024-09-13T11:25:20.546 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000201113 2024-09-13T11:25:20.546 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:20.546 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:20.546 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:25:20.546 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:48.701979+0000", 2024-09-13T11:25:20.546 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.001580618 2024-09-13T11:25:20.546 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:20.546 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:25:20.546 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:20.546 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:20.546 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:20.546 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop(client.4665.0:83193 3.8 e106/67 3:10066d83:::benchmark_data_smithi177_31116_object83192:head v 106'3925, mlcod=106'3922)", 2024-09-13T11:25:20.547 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:24:04.285068+0000", 2024-09-13T11:25:20.547 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 76.206298512000004, 2024-09-13T11:25:20.547 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.64647716600000005, 2024-09-13T11:25:20.547 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:25:20.547 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:25:20.547 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:20.547 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:25:20.547 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:20.547 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:25:20.547 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:04.285068+0000", 2024-09-13T11:25:20.547 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:20.547 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:20.547 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:20.547 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:25:20.548 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:04.285068+0000", 2024-09-13T11:25:20.548 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:20.548 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:20.548 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:20.548 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:25:20.548 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:04.285071+0000", 2024-09-13T11:25:20.548 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.729e-06 2024-09-13T11:25:20.548 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:20.548 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:20.548 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:25:20.548 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:04.285215+0000", 2024-09-13T11:25:20.548 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00014410000000000001 2024-09-13T11:25:20.548 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:20.548 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:20.549 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:25:20.549 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:04.928221+0000", 2024-09-13T11:25:20.549 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.64300653299999999 2024-09-13T11:25:20.549 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:20.549 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:20.549 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:25:20.549 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:04.928240+0000", 2024-09-13T11:25:20.549 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.8678999999999999e-05 2024-09-13T11:25:20.549 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:20.549 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:20.549 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:25:20.549 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:04.928388+0000", 2024-09-13T11:25:20.549 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00014809 2024-09-13T11:25:20.550 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:20.550 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:20.550 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:25:20.550 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:04.928428+0000", 2024-09-13T11:25:20.550 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.9776e-05 2024-09-13T11:25:20.550 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:20.550 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:20.550 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:25:20.550 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:04.931463+0000", 2024-09-13T11:25:20.550 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.003035168 2024-09-13T11:25:20.550 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:20.550 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:20.550 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:25:20.550 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:04.931545+0000", 2024-09-13T11:25:20.551 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.2090999999999997e-05 2024-09-13T11:25:20.551 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:20.551 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:25:20.551 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:20.551 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:20.551 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:20.551 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop(client.4665.0:83239 3.e e106/75 3:74547f17:::benchmark_data_smithi177_31116_object83238:head v 106'4391, mlcod=106'4388)", 2024-09-13T11:25:20.551 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:24:04.581196+0000", 2024-09-13T11:25:20.551 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 75.910170006000001, 2024-09-13T11:25:20.551 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.33617203600000001, 2024-09-13T11:25:20.551 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:25:20.551 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:25:20.551 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:20.551 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:25:20.552 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:20.552 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:25:20.552 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:04.581196+0000", 2024-09-13T11:25:20.552 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:20.552 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:20.552 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:20.552 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:25:20.552 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:04.581196+0000", 2024-09-13T11:25:20.552 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:20.552 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:20.552 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:20.552 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:25:20.553 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:04.581200+0000", 2024-09-13T11:25:20.553 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.19e-06 2024-09-13T11:25:20.553 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:20.553 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:20.553 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:25:20.553 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:04.581257+0000", 2024-09-13T11:25:20.553 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.7531999999999997e-05 2024-09-13T11:25:20.553 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:20.553 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:20.553 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:25:20.553 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:04.914011+0000", 2024-09-13T11:25:20.553 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.33275413700000001 2024-09-13T11:25:20.553 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:20.554 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:20.554 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:25:20.554 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:04.914028+0000", 2024-09-13T11:25:20.554 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.7082999999999999e-05 2024-09-13T11:25:20.554 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:20.554 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:20.554 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:25:20.554 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:04.914151+0000", 2024-09-13T11:25:20.554 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00012283300000000001 2024-09-13T11:25:20.554 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:20.554 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:20.554 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:25:20.554 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:04.914185+0000", 2024-09-13T11:25:20.554 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.4255999999999999e-05 2024-09-13T11:25:20.555 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:20.555 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:20.555 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:25:20.555 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:04.917300+0000", 2024-09-13T11:25:20.555 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0031141599999999999 2024-09-13T11:25:20.555 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:20.555 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:20.555 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:25:20.555 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:04.917368+0000", 2024-09-13T11:25:20.555 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.8844999999999996e-05 2024-09-13T11:25:20.555 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:20.555 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:25:20.556 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:20.556 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:20.556 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:20.556 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "MOSDPGPushReply(3.28 121/120 [PushReplyOp(3:145f409c:::benchmark_data_smithi177_31116_object1242:head)])", 2024-09-13T11:25:20.556 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:24:27.450312+0000", 2024-09-13T11:25:20.556 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 53.041054875, 2024-09-13T11:25:20.556 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.27862834600000003, 2024-09-13T11:25:20.556 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:25:20.556 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:25:20.556 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "reached pg", 2024-09-13T11:25:20.556 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:25:20.556 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:20.556 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:25:20.557 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:27.450312+0000", 2024-09-13T11:25:20.557 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:20.557 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:20.557 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:20.557 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:25:20.557 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:27.450312+0000", 2024-09-13T11:25:20.557 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:20.557 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:20.557 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:20.557 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:25:20.557 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:27.450313+0000", 2024-09-13T11:25:20.557 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.372e-06 2024-09-13T11:25:20.558 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:20.558 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:20.558 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:25:20.558 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:27.450321+0000", 2024-09-13T11:25:20.558 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.089e-06 2024-09-13T11:25:20.558 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:20.558 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:20.558 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:25:20.558 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:27.726786+0000", 2024-09-13T11:25:20.558 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.27646454399999998 2024-09-13T11:25:20.558 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:20.558 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:20.558 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:25:20.559 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:27.726803+0000", 2024-09-13T11:25:20.559 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.7283e-05 2024-09-13T11:25:20.559 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:20.559 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:20.559 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:25:20.559 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:27.726977+0000", 2024-09-13T11:25:20.559 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00017421200000000001 2024-09-13T11:25:20.559 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:20.559 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:20.559 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:25:20.559 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:27.728940+0000", 2024-09-13T11:25:20.559 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0019628459999999999 2024-09-13T11:25:20.559 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:20.559 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:25:20.559 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:20.560 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:20.560 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:20.560 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:126432 3.28 3:172ceb4d:::benchmark_data_smithi177_31116_object126431:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e130)", 2024-09-13T11:25:20.560 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:24:43.334456+0000", 2024-09-13T11:25:20.560 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 37.156910807000003, 2024-09-13T11:25:20.560 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.27433384799999999, 2024-09-13T11:25:20.560 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:25:20.560 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:25:20.560 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:20.560 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:25:20.560 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:25:20.560 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:20.560 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 126432 2024-09-13T11:25:20.561 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:20.561 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:25:20.561 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:20.561 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:25:20.561 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:43.334456+0000", 2024-09-13T11:25:20.561 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:20.561 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:20.561 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:20.561 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:25:20.561 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:43.334456+0000", 2024-09-13T11:25:20.561 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:20.561 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:20.561 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:20.562 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:25:20.562 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:43.334458+0000", 2024-09-13T11:25:20.562 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.1459999999999999e-06 2024-09-13T11:25:20.562 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:20.562 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:20.562 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:25:20.562 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:43.334518+0000", 2024-09-13T11:25:20.562 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.0418000000000003e-05 2024-09-13T11:25:20.562 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:20.562 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:20.562 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:25:20.562 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:43.334527+0000", 2024-09-13T11:25:20.562 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.4940000000000003e-06 2024-09-13T11:25:20.563 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:20.563 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:20.563 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:25:20.563 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:43.334532+0000", 2024-09-13T11:25:20.563 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.6509999999999998e-06 2024-09-13T11:25:20.563 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:20.563 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:20.563 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:25:20.563 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:43.334587+0000", 2024-09-13T11:25:20.563 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.4598000000000001e-05 2024-09-13T11:25:20.563 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:20.563 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:20.563 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:25:20.563 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:43.334751+0000", 2024-09-13T11:25:20.564 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00016452200000000001 2024-09-13T11:25:20.564 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:20.564 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:20.564 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 5,8", 2024-09-13T11:25:20.564 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:43.334918+0000", 2024-09-13T11:25:20.564 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00016641599999999999 2024-09-13T11:25:20.564 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:20.564 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:20.564 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:25:20.564 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:43.337338+0000", 2024-09-13T11:25:20.564 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0024197749999999999 2024-09-13T11:25:20.564 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:20.564 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:20.564 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:25:20.565 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:43.338831+0000", 2024-09-13T11:25:20.565 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0014936940000000001 2024-09-13T11:25:20.565 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:20.565 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:20.565 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:25:20.565 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:43.608021+0000", 2024-09-13T11:25:20.565 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.269189709 2024-09-13T11:25:20.565 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:20.565 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:20.565 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:25:20.565 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:43.608087+0000", 2024-09-13T11:25:20.565 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.6421999999999995e-05 2024-09-13T11:25:20.565 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:20.565 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:20.565 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:25:20.566 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:43.608789+0000", 2024-09-13T11:25:20.566 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00070200299999999998 2024-09-13T11:25:20.566 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:20.566 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:25:20.566 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:20.566 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:20.566 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:20.566 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop_reply(client.4665.0:126432 3.28 e130/120 ondisk, result = 0)", 2024-09-13T11:25:20.566 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:24:43.337816+0000", 2024-09-13T11:25:20.566 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 37.153550123000002, 2024-09-13T11:25:20.566 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.27098083499999998, 2024-09-13T11:25:20.566 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:25:20.566 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:25:20.567 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-09-13T11:25:20.567 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:25:20.567 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:20.567 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:25:20.567 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:43.337816+0000", 2024-09-13T11:25:20.567 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:20.567 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:20.567 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:20.567 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:25:20.567 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:43.337816+0000", 2024-09-13T11:25:20.567 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:20.567 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:20.567 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:20.568 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:25:20.568 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:43.337818+0000", 2024-09-13T11:25:20.568 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.3239999999999999e-06 2024-09-13T11:25:20.568 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:20.568 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:20.568 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:25:20.568 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:43.337821+0000", 2024-09-13T11:25:20.568 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.8709999999999999e-06 2024-09-13T11:25:20.568 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:20.568 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:20.568 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:25:20.568 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:43.607950+0000", 2024-09-13T11:25:20.568 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.270128916 2024-09-13T11:25:20.568 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:20.569 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:20.569 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:25:20.569 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:43.607954+0000", 2024-09-13T11:25:20.569 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.0300000000000004e-06 2024-09-13T11:25:20.569 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:20.569 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:20.569 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:25:20.569 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:43.608002+0000", 2024-09-13T11:25:20.569 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.7757999999999999e-05 2024-09-13T11:25:20.569 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:20.569 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:20.569 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:25:20.569 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:43.608011+0000", 2024-09-13T11:25:20.570 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.9239999999999996e-06 2024-09-13T11:25:20.570 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:20.570 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:20.570 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:25:20.570 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:43.608797+0000", 2024-09-13T11:25:20.570 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000786012 2024-09-13T11:25:20.570 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:20.570 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:25:20.570 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:20.570 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:20.570 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:20.570 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:129876 3.28 3:173a39a3:::benchmark_data_smithi177_31116_object129875:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e134)", 2024-09-13T11:25:20.570 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:24:47.116837+0000", 2024-09-13T11:25:20.571 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 33.374529086999999, 2024-09-13T11:25:20.571 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.282398655, 2024-09-13T11:25:20.571 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:25:20.571 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:25:20.571 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:20.571 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:25:20.571 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:25:20.571 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:20.571 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 129876 2024-09-13T11:25:20.571 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:20.571 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:25:20.571 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:20.571 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:25:20.572 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:47.116837+0000", 2024-09-13T11:25:20.572 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:20.572 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:20.572 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:20.572 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:25:20.572 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:47.116837+0000", 2024-09-13T11:25:20.572 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:20.572 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:20.572 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:20.572 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:25:20.572 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:47.116841+0000", 2024-09-13T11:25:20.572 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.2370000000000001e-06 2024-09-13T11:25:20.572 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:20.572 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:20.573 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:25:20.573 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:47.116901+0000", 2024-09-13T11:25:20.573 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.0409999999999999e-05 2024-09-13T11:25:20.573 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:20.573 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:20.573 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:25:20.573 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:47.116920+0000", 2024-09-13T11:25:20.573 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.9477e-05 2024-09-13T11:25:20.573 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:20.573 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:20.573 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:25:20.573 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:47.116931+0000", 2024-09-13T11:25:20.573 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0801e-05 2024-09-13T11:25:20.574 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:20.574 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:20.574 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:25:20.574 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:47.116991+0000", 2024-09-13T11:25:20.574 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.9596000000000003e-05 2024-09-13T11:25:20.574 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:20.574 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:20.574 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:25:20.574 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:47.117123+0000", 2024-09-13T11:25:20.574 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000132342 2024-09-13T11:25:20.574 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:20.574 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:20.575 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 5,8", 2024-09-13T11:25:20.575 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:47.117248+0000", 2024-09-13T11:25:20.575 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00012444999999999999 2024-09-13T11:25:20.575 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:20.575 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:20.575 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:25:20.575 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:47.118577+0000", 2024-09-13T11:25:20.575 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.001329373 2024-09-13T11:25:20.575 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:20.575 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:20.575 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:25:20.575 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:47.119544+0000", 2024-09-13T11:25:20.575 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00096691099999999999 2024-09-13T11:25:20.575 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:20.576 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:20.576 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:25:20.576 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:47.398345+0000", 2024-09-13T11:25:20.576 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.27880143600000001 2024-09-13T11:25:20.576 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:20.576 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:20.576 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:25:20.576 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:47.398419+0000", 2024-09-13T11:25:20.576 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.3697999999999998e-05 2024-09-13T11:25:20.576 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:20.576 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:20.576 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:25:20.576 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:47.399236+0000", 2024-09-13T11:25:20.577 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00081692400000000001 2024-09-13T11:25:20.577 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:20.577 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:25:20.577 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:20.577 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:20.577 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:20.577 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop_reply(client.4665.0:129876 3.28 e134/120 ondisk, result = 0)", 2024-09-13T11:25:20.577 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:24:47.119458+0000", 2024-09-13T11:25:20.577 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 33.371908361999999, 2024-09-13T11:25:20.577 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.27979297199999997, 2024-09-13T11:25:20.577 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:25:20.577 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:25:20.577 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-09-13T11:25:20.578 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:25:20.578 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:20.578 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:25:20.578 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:47.119458+0000", 2024-09-13T11:25:20.578 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:20.578 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:20.578 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:20.578 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:25:20.578 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:47.119458+0000", 2024-09-13T11:25:20.578 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:20.578 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:20.578 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:20.578 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:25:20.579 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:47.119459+0000", 2024-09-13T11:25:20.579 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0529999999999999e-06 2024-09-13T11:25:20.579 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:20.579 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:20.579 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:25:20.579 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:47.119463+0000", 2024-09-13T11:25:20.579 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.1049999999999997e-06 2024-09-13T11:25:20.579 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:20.579 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:20.579 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:25:20.579 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:47.398252+0000", 2024-09-13T11:25:20.579 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.27878866800000002 2024-09-13T11:25:20.579 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:20.579 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:20.579 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:25:20.580 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:47.398260+0000", 2024-09-13T11:25:20.580 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.5329999999999994e-06 2024-09-13T11:25:20.580 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:20.580 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:20.580 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:25:20.580 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:47.398323+0000", 2024-09-13T11:25:20.580 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.2880999999999996e-05 2024-09-13T11:25:20.580 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:20.580 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:20.580 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:25:20.580 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:47.398336+0000", 2024-09-13T11:25:20.580 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.277e-05 2024-09-13T11:25:20.580 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:20.581 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:20.581 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:25:20.581 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:47.399251+0000", 2024-09-13T11:25:20.581 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00091496199999999996 2024-09-13T11:25:20.581 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:20.581 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:25:20.581 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:20.581 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:20.581 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:20.581 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:131687 3.28 3:14f65078:::benchmark_data_smithi177_31116_object131686:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e135)", 2024-09-13T11:25:20.581 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:24:48.925565+0000", 2024-09-13T11:25:20.581 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 31.565801009000001, 2024-09-13T11:25:20.581 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.41421604099999998, 2024-09-13T11:25:20.581 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:25:20.582 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:25:20.582 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:20.582 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:25:20.582 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:25:20.582 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:20.582 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 131687 2024-09-13T11:25:20.582 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:20.582 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:25:20.582 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:20.582 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:25:20.582 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:48.925565+0000", 2024-09-13T11:25:20.582 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:20.582 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:20.582 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:20.583 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:25:20.583 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:48.925565+0000", 2024-09-13T11:25:20.583 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:20.583 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:20.583 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:20.583 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:25:20.583 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:48.925568+0000", 2024-09-13T11:25:20.583 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.441e-06 2024-09-13T11:25:20.583 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:20.583 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:20.583 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:25:20.583 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:48.925617+0000", 2024-09-13T11:25:20.583 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.8739000000000002e-05 2024-09-13T11:25:20.583 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:20.584 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:20.584 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:25:20.584 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:48.925631+0000", 2024-09-13T11:25:20.584 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.4463e-05 2024-09-13T11:25:20.584 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:20.584 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:20.584 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:25:20.584 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:48.925638+0000", 2024-09-13T11:25:20.584 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.2119999999999997e-06 2024-09-13T11:25:20.584 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:20.584 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:20.584 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:25:20.584 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:48.925703+0000", 2024-09-13T11:25:20.584 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.4794000000000003e-05 2024-09-13T11:25:20.585 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:20.585 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:20.585 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:25:20.585 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:48.925857+0000", 2024-09-13T11:25:20.585 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00015427 2024-09-13T11:25:20.585 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:20.585 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:20.585 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 5,8", 2024-09-13T11:25:20.585 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:48.925995+0000", 2024-09-13T11:25:20.585 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000137365 2024-09-13T11:25:20.585 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:20.585 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:20.585 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:25:20.586 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:48.927261+0000", 2024-09-13T11:25:20.586 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.001266021 2024-09-13T11:25:20.586 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:20.586 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:20.586 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:25:20.586 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:48.930015+0000", 2024-09-13T11:25:20.586 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0027545130000000001 2024-09-13T11:25:20.586 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:20.586 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:20.586 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:25:20.586 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:49.338898+0000", 2024-09-13T11:25:20.586 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.40888254299999999 2024-09-13T11:25:20.586 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:20.586 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:20.587 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:25:20.587 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:49.338967+0000", 2024-09-13T11:25:20.587 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.9402999999999999e-05 2024-09-13T11:25:20.587 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:20.587 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:20.587 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:25:20.587 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:49.339781+0000", 2024-09-13T11:25:20.587 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00081427699999999997 2024-09-13T11:25:20.587 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:20.587 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:25:20.587 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:20.587 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:20.587 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:20.587 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop_reply(client.4665.0:131687 3.28 e135/120 ondisk, result = 0)", 2024-09-13T11:25:20.588 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:24:48.928935+0000", 2024-09-13T11:25:20.588 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 31.562431707999998, 2024-09-13T11:25:20.588 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.41085638499999999, 2024-09-13T11:25:20.588 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:25:20.588 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:25:20.588 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-09-13T11:25:20.588 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:25:20.588 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:20.588 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:25:20.588 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:48.928935+0000", 2024-09-13T11:25:20.588 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:20.588 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:20.588 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:20.588 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:25:20.589 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:48.928935+0000", 2024-09-13T11:25:20.589 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:20.589 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:20.589 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:20.589 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:25:20.589 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:48.928936+0000", 2024-09-13T11:25:20.589 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.519e-06 2024-09-13T11:25:20.589 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:20.589 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:20.589 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:25:20.589 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:48.928941+0000", 2024-09-13T11:25:20.589 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.0710000000000002e-06 2024-09-13T11:25:20.589 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:20.589 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:20.589 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:25:20.590 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:49.338799+0000", 2024-09-13T11:25:20.590 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.40985808499999998 2024-09-13T11:25:20.590 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:20.590 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:20.590 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:25:20.590 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:49.338809+0000", 2024-09-13T11:25:20.590 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.4390000000000006e-06 2024-09-13T11:25:20.590 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:20.590 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:20.590 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:25:20.590 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:49.338878+0000", 2024-09-13T11:25:20.590 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.9429000000000002e-05 2024-09-13T11:25:20.590 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:20.590 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:20.590 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:25:20.591 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:49.338890+0000", 2024-09-13T11:25:20.591 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.2217e-05 2024-09-13T11:25:20.591 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:20.591 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:20.591 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:25:20.591 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:49.339791+0000", 2024-09-13T11:25:20.591 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00090062499999999995 2024-09-13T11:25:20.591 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:20.591 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:25:20.591 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:20.591 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:20.591 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:25:20.591 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-09-13T11:25:20.593 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_ops_in_flight 2024-09-13T11:25:20.593 INFO:tasks.ceph.osd.13.smithi092.stderr:2024-09-13T11:25:20.585+0000 7f3e5afea640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26350) UID: 0 2024-09-13T11:25:20.693 INFO:tasks.ceph.osd.13.smithi092.stderr:2024-09-13T11:25:20.689+0000 7f3e5afea640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26350) UID: 0 2024-09-13T11:25:20.744 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-13T11:25:20.745 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [], 2024-09-13T11:25:20.745 INFO:teuthology.orchestra.run.smithi162.stdout: "num_ops": 0 2024-09-13T11:25:20.745 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-13T11:25:20.760 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_blocked_ops 2024-09-13T11:25:20.794 INFO:tasks.ceph.osd.11.smithi177.stderr:2024-09-13T11:25:20.790+0000 7f24792c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26580) UID: 0 2024-09-13T11:25:20.885 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-13T11:25:20.885 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [], 2024-09-13T11:25:20.885 INFO:teuthology.orchestra.run.smithi162.stdout: "complaint_time": 30, 2024-09-13T11:25:20.885 INFO:teuthology.orchestra.run.smithi162.stdout: "num_blocked_ops": 0 2024-09-13T11:25:20.885 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-13T11:25:20.895 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-09-13T11:25:20.892+0000 7f97befc9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26315) UID: 0 2024-09-13T11:25:20.901 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_historic_ops 2024-09-13T11:25:20.997 INFO:tasks.ceph.osd.3.smithi177.stderr:2024-09-13T11:25:20.994+0000 7f10adba4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26574) UID: 0 2024-09-13T11:25:21.084 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-13T11:25:21.085 INFO:teuthology.orchestra.run.smithi162.stdout: "size": 20, 2024-09-13T11:25:21.085 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 600, 2024-09-13T11:25:21.085 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [] 2024-09-13T11:25:21.085 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-13T11:25:21.097 INFO:tasks.ceph.osd.14.smithi162.stderr:2024-09-13T11:25:21.096+0000 7f3df5903640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33154) UID: 0 2024-09-13T11:25:21.102 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-13T11:25:21.169 INFO:teuthology.orchestra.run.smithi001.stderr:reweighted osd.10 to 1 (10000) 2024-09-13T11:25:21.425 INFO:tasks.ceph.osd.7.smithi177.stderr:2024-09-13T11:25:21.422+0000 7f897b51d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26575) UID: 0 2024-09-13T11:25:21.425 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 11 1 2024-09-13T11:25:21.426 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-13T11:25:21.427 INFO:teuthology.orchestra.run.smithi001.stdout: "ops": [ 2024-09-13T11:25:21.427 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:21.427 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:169884 3.e 3:74eaabdb:::benchmark_data_smithi177_31116_object29581:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e162)", 2024-09-13T11:25:21.427 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:25:21.267535+0000", 2024-09-13T11:25:21.427 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 0.0035911879999999999, 2024-09-13T11:25:21.427 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0036152900000000002, 2024-09-13T11:25:21.427 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:21.427 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:21.427 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "waiting for sub ops", 2024-09-13T11:25:21.427 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:25:21.427 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:25:21.427 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:21.428 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 169884 2024-09-13T11:25:21.428 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:21.428 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:21.428 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:21.428 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:21.428 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:21.267535+0000", 2024-09-13T11:25:21.428 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:21.428 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:21.428 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:21.428 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:21.428 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:21.267535+0000", 2024-09-13T11:25:21.429 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:21.429 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:21.429 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:21.431 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:21.432 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:21.267537+0000", 2024-09-13T11:25:21.432 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.666e-06 2024-09-13T11:25:21.432 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:21.432 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:21.432 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:21.432 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:21.267542+0000", 2024-09-13T11:25:21.433 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.8420000000000004e-06 2024-09-13T11:25:21.433 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:21.433 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:21.433 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:21.433 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:21.267552+0000", 2024-09-13T11:25:21.433 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0057999999999999e-05 2024-09-13T11:25:21.433 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:21.433 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:21.433 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:21.433 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:21.267557+0000", 2024-09-13T11:25:21.433 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.1649999999999997e-06 2024-09-13T11:25:21.433 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:21.433 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:21.434 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:21.434 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:21.267606+0000", 2024-09-13T11:25:21.434 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.9397999999999997e-05 2024-09-13T11:25:21.434 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:21.434 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:21.434 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:25:21.434 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:21.267730+0000", 2024-09-13T11:25:21.434 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00012417400000000001 2024-09-13T11:25:21.434 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:21.434 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:21.434 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 1", 2024-09-13T11:25:21.434 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:21.267833+0000", 2024-09-13T11:25:21.434 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00010224499999999999 2024-09-13T11:25:21.434 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:21.435 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:21.435 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:25:21.435 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:21.270802+0000", 2024-09-13T11:25:21.435 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0029693900000000001 2024-09-13T11:25:21.435 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:21.435 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:21.435 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:21.435 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:21.435 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:21.435 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:169886 3.21 3:8579c7f7:::benchmark_data_smithi177_31116_object29583:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e162)", 2024-09-13T11:25:21.435 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:25:21.269068+0000", 2024-09-13T11:25:21.435 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 0.0020578940000000002, 2024-09-13T11:25:21.435 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0021569340000000001, 2024-09-13T11:25:21.436 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:21.436 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:21.436 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "waiting for sub ops", 2024-09-13T11:25:21.436 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:25:21.436 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:25:21.436 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:21.436 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 169886 2024-09-13T11:25:21.436 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:21.436 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:21.436 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:21.436 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:21.436 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:21.269068+0000", 2024-09-13T11:25:21.436 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:21.437 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:21.437 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:21.437 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:21.437 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:21.269068+0000", 2024-09-13T11:25:21.437 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:21.437 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:21.437 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:21.437 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:21.437 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:21.269070+0000", 2024-09-13T11:25:21.437 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.5799999999999999e-06 2024-09-13T11:25:21.437 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:21.437 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:21.437 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:21.438 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:21.269075+0000", 2024-09-13T11:25:21.438 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.0420000000000002e-06 2024-09-13T11:25:21.438 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:21.438 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:21.438 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:21.438 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:21.269088+0000", 2024-09-13T11:25:21.438 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.3397999999999999e-05 2024-09-13T11:25:21.438 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:21.438 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:21.439 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:21.439 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:21.269096+0000", 2024-09-13T11:25:21.439 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.0979999999999994e-06 2024-09-13T11:25:21.439 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:21.439 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:21.439 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:21.439 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:21.269155+0000", 2024-09-13T11:25:21.439 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.8842999999999999e-05 2024-09-13T11:25:21.439 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:21.439 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:21.439 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:25:21.439 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:21.269276+0000", 2024-09-13T11:25:21.439 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000120766 2024-09-13T11:25:21.440 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:21.440 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:21.440 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 8", 2024-09-13T11:25:21.440 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:21.269369+0000", 2024-09-13T11:25:21.440 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.3318000000000003e-05 2024-09-13T11:25:21.440 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:21.440 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:21.440 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:25:21.440 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:21.271019+0000", 2024-09-13T11:25:21.440 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0016497 2024-09-13T11:25:21.440 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:21.440 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:21.440 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:21.441 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:21.441 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:21.441 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4665.0:169888 3.20 e162/77)", 2024-09-13T11:25:21.441 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:25:21.271165+0000", 2024-09-13T11:25:21.441 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 4294967295.9999609, 2024-09-13T11:25:21.441 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00014123, 2024-09-13T11:25:21.441 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:21.441 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:21.441 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "queued for pg", 2024-09-13T11:25:21.441 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:21.441 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:21.441 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:21.441 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:21.271165+0000", 2024-09-13T11:25:21.441 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:21.442 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:21.442 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:21.442 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:21.442 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:21.271165+0000", 2024-09-13T11:25:21.442 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:21.442 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:21.442 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:21.442 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:21.442 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:21.271167+0000", 2024-09-13T11:25:21.442 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.4980000000000001e-06 2024-09-13T11:25:21.442 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:21.442 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:21.442 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:21.443 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:21.271174+0000", 2024-09-13T11:25:21.443 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.5130000000000002e-06 2024-09-13T11:25:21.443 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:21.443 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:21.443 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:21.443 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:21.271202+0000", 2024-09-13T11:25:21.443 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.8419999999999999e-05 2024-09-13T11:25:21.443 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:21.443 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:21.443 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:21.443 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:21.271209+0000", 2024-09-13T11:25:21.443 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.5780000000000004e-06 2024-09-13T11:25:21.443 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:21.443 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:21.444 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:21.444 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:21.444 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-09-13T11:25:21.444 INFO:teuthology.orchestra.run.smithi001.stdout: "num_ops": 3 2024-09-13T11:25:21.444 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-13T11:25:21.444 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-13T11:25:21.525 INFO:tasks.ceph.osd.5.smithi092.stderr:2024-09-13T11:25:21.521+0000 7f6f0b705640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26346) UID: 0 2024-09-13T11:25:21.626 INFO:tasks.ceph.osd.9.smithi092.stderr:2024-09-13T11:25:21.621+0000 7ff5ab3d0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26348) UID: 0 2024-09-13T11:25:21.630 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-13T11:25:21.630 INFO:teuthology.orchestra.run.smithi001.stdout: "ops": [], 2024-09-13T11:25:21.630 INFO:teuthology.orchestra.run.smithi001.stdout: "complaint_time": 30, 2024-09-13T11:25:21.631 INFO:teuthology.orchestra.run.smithi001.stdout: "num_blocked_ops": 0 2024-09-13T11:25:21.631 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-13T11:25:21.650 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-13T11:25:21.726 INFO:tasks.ceph.osd.8.smithi001.stderr:2024-09-13T11:25:21.723+0000 7fb9b48ae640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26312) UID: 0 2024-09-13T11:25:21.802 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-13T11:25:21.802 INFO:teuthology.orchestra.run.smithi001.stdout: "size": 20, 2024-09-13T11:25:21.802 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 600, 2024-09-13T11:25:21.802 INFO:teuthology.orchestra.run.smithi001.stdout: "ops": [ 2024-09-13T11:25:21.802 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:21.802 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDPGPushReply(3.1e 84/78 [PushReplyOp(3:79ee54b9:::benchmark_data_smithi177_31116_object11701:head)])", 2024-09-13T11:25:21.802 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:41.407335+0000", 2024-09-13T11:25:21.803 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 100.39269034500001, 2024-09-13T11:25:21.803 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.51622350400000006, 2024-09-13T11:25:21.803 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:21.803 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:21.803 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-13T11:25:21.803 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:21.803 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:21.803 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:21.803 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:41.407335+0000", 2024-09-13T11:25:21.803 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:21.803 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:21.803 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:21.803 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:21.803 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:41.407335+0000", 2024-09-13T11:25:21.804 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:21.804 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:21.804 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:21.804 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:21.804 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:41.407337+0000", 2024-09-13T11:25:21.804 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.505e-06 2024-09-13T11:25:21.804 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:21.804 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:21.804 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:21.804 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:41.407343+0000", 2024-09-13T11:25:21.804 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.9089999999999997e-06 2024-09-13T11:25:21.804 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:21.804 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:21.805 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:21.805 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:41.921977+0000", 2024-09-13T11:25:21.805 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.51463463499999995 2024-09-13T11:25:21.805 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:21.805 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:21.805 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:21.805 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:41.921988+0000", 2024-09-13T11:25:21.805 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.092e-05 2024-09-13T11:25:21.805 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:21.805 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:21.805 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:21.805 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:41.922088+0000", 2024-09-13T11:25:21.805 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.9528e-05 2024-09-13T11:25:21.805 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:21.806 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:21.806 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:21.806 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:41.923559+0000", 2024-09-13T11:25:21.806 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0014710070000000001 2024-09-13T11:25:21.806 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:21.806 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:21.806 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:21.806 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:21.806 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:21.806 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4665.0:60417 3.10 e85/84 3:092808b0:::benchmark_data_smithi177_31116_object60416:head v 85'3295, mlcod=85'3292)", 2024-09-13T11:25:21.806 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:42.174488+0000", 2024-09-13T11:25:21.806 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 99.625537790999999, 2024-09-13T11:25:21.806 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.387908896, 2024-09-13T11:25:21.806 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:21.807 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:21.807 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:21.807 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:21.807 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:21.807 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:21.807 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.174488+0000", 2024-09-13T11:25:21.807 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:21.807 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:21.807 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:21.807 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:21.807 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.174488+0000", 2024-09-13T11:25:21.807 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:21.807 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:21.807 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:21.808 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:21.808 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.174490+0000", 2024-09-13T11:25:21.808 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.9099999999999999e-06 2024-09-13T11:25:21.808 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:21.808 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:21.808 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:21.808 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.174527+0000", 2024-09-13T11:25:21.808 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.7543000000000003e-05 2024-09-13T11:25:21.808 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:21.808 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:21.808 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:21.808 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.561093+0000", 2024-09-13T11:25:21.808 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.38656578400000002 2024-09-13T11:25:21.808 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:21.809 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:21.809 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:21.809 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.561101+0000", 2024-09-13T11:25:21.809 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.6090000000000003e-06 2024-09-13T11:25:21.809 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:21.809 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:21.809 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:21.809 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.561143+0000", 2024-09-13T11:25:21.809 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.2435000000000003e-05 2024-09-13T11:25:21.809 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:21.809 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:21.809 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:25:21.809 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.561164+0000", 2024-09-13T11:25:21.810 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.1276000000000001e-05 2024-09-13T11:25:21.810 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:21.810 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:21.810 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:25:21.810 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.562360+0000", 2024-09-13T11:25:21.810 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00119596 2024-09-13T11:25:21.810 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:21.810 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:21.810 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:21.810 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.562397+0000", 2024-09-13T11:25:21.810 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.6378999999999998e-05 2024-09-13T11:25:21.810 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:21.810 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:21.810 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:21.810 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:21.811 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:21.811 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDPGPush(3.22 85/78 [PushOp(3:456e8765:::benchmark_data_smithi177_31116_object16212:head, version: 52'1006, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:456e8765:::benchmark_data_smithi177_31116_object16212:head@52'1006, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:25:21.811 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:42.673067+0000", 2024-09-13T11:25:21.811 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 99.126958649000002, 2024-09-13T11:25:21.811 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.27935889899999999, 2024-09-13T11:25:21.811 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:21.811 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:21.811 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:25:21.811 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:21.811 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:21.811 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:21.811 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.673067+0000", 2024-09-13T11:25:21.811 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:21.812 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:21.812 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:21.812 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:21.812 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.673067+0000", 2024-09-13T11:25:21.812 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:21.812 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:21.812 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:21.812 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:21.812 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.673068+0000", 2024-09-13T11:25:21.812 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.9699999999999994e-07 2024-09-13T11:25:21.812 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:21.812 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:21.812 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:21.812 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.673090+0000", 2024-09-13T11:25:21.813 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.2416999999999999e-05 2024-09-13T11:25:21.813 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:21.813 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:21.813 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:21.813 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.951363+0000", 2024-09-13T11:25:21.813 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.27827291199999998 2024-09-13T11:25:21.813 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:21.813 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:21.813 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:21.813 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.951386+0000", 2024-09-13T11:25:21.813 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.2226999999999999e-05 2024-09-13T11:25:21.813 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:21.813 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:21.813 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:21.813 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.951581+0000", 2024-09-13T11:25:21.814 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000195713 2024-09-13T11:25:21.814 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:21.814 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:21.814 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:25:21.814 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.951605+0000", 2024-09-13T11:25:21.814 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.3767000000000001e-05 2024-09-13T11:25:21.814 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:21.814 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:21.814 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:21.814 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.952426+0000", 2024-09-13T11:25:21.814 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00082086599999999996 2024-09-13T11:25:21.814 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:21.814 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:21.814 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:21.815 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:21.815 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:21.815 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "replica_scrub(pg: 3.20,from:0'0,to:89'3254,epoch:89/77,start:3:04000000::::0,end:3:045ba805:::benchmark_data_smithi177_31116_object43251:0,chunky:1,deep:0,version:9,allow_preemption:1,priority=5)", 2024-09-13T11:25:21.815 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:46.641364+0000", 2024-09-13T11:25:21.815 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 95.158661968000004, 2024-09-13T11:25:21.815 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.27141393000000003, 2024-09-13T11:25:21.815 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:21.815 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:21.815 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:25:21.815 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:21.815 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:21.815 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:21.815 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.641364+0000", 2024-09-13T11:25:21.815 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:21.816 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:21.816 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:21.816 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:21.816 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.641364+0000", 2024-09-13T11:25:21.816 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:21.816 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:21.816 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:21.816 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:21.816 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.641365+0000", 2024-09-13T11:25:21.816 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.266e-06 2024-09-13T11:25:21.816 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:21.816 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:21.816 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:21.816 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.641370+0000", 2024-09-13T11:25:21.817 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.8539999999999996e-06 2024-09-13T11:25:21.817 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:21.817 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:21.817 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:21.817 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.912491+0000", 2024-09-13T11:25:21.817 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.27112103100000001 2024-09-13T11:25:21.817 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:21.817 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:21.817 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:21.817 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.912520+0000", 2024-09-13T11:25:21.817 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.9379000000000001e-05 2024-09-13T11:25:21.817 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:21.817 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:21.817 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:21.818 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.912679+0000", 2024-09-13T11:25:21.818 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00015915800000000001 2024-09-13T11:25:21.818 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:21.818 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:21.818 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:25:21.818 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.912714+0000", 2024-09-13T11:25:21.818 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.4687999999999997e-05 2024-09-13T11:25:21.818 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:21.818 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:21.818 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:21.818 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.912778+0000", 2024-09-13T11:25:21.818 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.3553999999999995e-05 2024-09-13T11:25:21.818 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:21.818 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:21.818 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:21.819 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:21.819 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:21.819 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDPGPush(3.18 89/70 [PushOp(3:1ac6c361:::benchmark_data_smithi177_31116_object44972:head, version: 68'2767, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:1ac6c361:::benchmark_data_smithi177_31116_object44972:head@68'2767, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:25:21.819 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:46.671018+0000", 2024-09-13T11:25:21.819 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 95.129007836, 2024-09-13T11:25:21.819 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.271282931, 2024-09-13T11:25:21.819 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:21.819 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:21.819 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:25:21.819 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:21.819 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:21.819 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:21.819 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.671018+0000", 2024-09-13T11:25:21.819 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:21.820 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:21.820 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:21.820 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:21.820 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.671018+0000", 2024-09-13T11:25:21.820 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:21.820 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:21.820 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:21.820 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:21.820 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.671020+0000", 2024-09-13T11:25:21.820 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.018e-06 2024-09-13T11:25:21.820 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:21.820 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:21.820 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:21.821 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.671097+0000", 2024-09-13T11:25:21.821 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.7218999999999994e-05 2024-09-13T11:25:21.821 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:21.821 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:21.821 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:21.821 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.939773+0000", 2024-09-13T11:25:21.821 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.26867550200000001 2024-09-13T11:25:21.821 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:21.821 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:21.821 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:21.821 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.939791+0000", 2024-09-13T11:25:21.821 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.8185999999999998e-05 2024-09-13T11:25:21.821 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:21.821 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:21.822 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:21.822 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.939926+0000", 2024-09-13T11:25:21.822 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000135291 2024-09-13T11:25:21.822 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:21.822 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:21.822 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:25:21.822 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.939938+0000", 2024-09-13T11:25:21.822 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2393e-05 2024-09-13T11:25:21.822 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:21.822 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:21.822 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:21.822 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.942301+0000", 2024-09-13T11:25:21.822 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.002362322 2024-09-13T11:25:21.822 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:21.823 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:21.823 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:21.823 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:21.823 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:21.823 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "replica_scrub(pg: 3.20,from:0'0,to:90'3264,epoch:90/77,start:3:0650aac8:::benchmark_data_smithi177_31116_object1734:0,end:3:06af01e3:::benchmark_data_smithi177_31116_object31572:0,chunky:1,deep:0,version:9,allow_preemption:1,priority=5)", 2024-09-13T11:25:21.823 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:47.380615+0000", 2024-09-13T11:25:21.823 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 94.419410412999994, 2024-09-13T11:25:21.823 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.51951398400000004, 2024-09-13T11:25:21.823 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:21.823 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:21.823 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:25:21.823 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:21.823 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:21.823 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:21.824 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:47.380615+0000", 2024-09-13T11:25:21.824 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:21.824 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:21.824 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:21.824 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:21.824 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:47.380615+0000", 2024-09-13T11:25:21.824 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:21.824 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:21.824 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:21.824 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:21.824 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:47.380617+0000", 2024-09-13T11:25:21.824 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.671e-06 2024-09-13T11:25:21.824 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:21.825 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:21.825 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:21.825 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:47.380623+0000", 2024-09-13T11:25:21.825 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.1190000000000002e-06 2024-09-13T11:25:21.825 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:21.825 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:21.825 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:21.825 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:47.900050+0000", 2024-09-13T11:25:21.825 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.51942737400000005 2024-09-13T11:25:21.825 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:21.825 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:21.825 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:21.825 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:47.900057+0000", 2024-09-13T11:25:21.826 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.4359999999999999e-06 2024-09-13T11:25:21.826 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:21.826 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:21.826 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:21.826 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:47.900101+0000", 2024-09-13T11:25:21.826 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.4675000000000002e-05 2024-09-13T11:25:21.826 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:21.826 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:21.826 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:25:21.826 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:47.900110+0000", 2024-09-13T11:25:21.826 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.0569999999999993e-06 2024-09-13T11:25:21.826 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:21.826 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:21.826 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:21.826 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:47.900129+0000", 2024-09-13T11:25:21.827 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.9652000000000001e-05 2024-09-13T11:25:21.827 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:21.827 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:21.827 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:21.827 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:21.827 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:21.827 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDPGPush(3.21 91/78 [PushOp(3:8492321f:::benchmark_data_smithi177_31116_object13767:head, version: 50'840, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:8492321f:::benchmark_data_smithi177_31116_object13767:head@50'840, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:25:21.827 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:48.319251+0000", 2024-09-13T11:25:21.827 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 93.480774883999999, 2024-09-13T11:25:21.827 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.60903802200000001, 2024-09-13T11:25:21.827 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:21.827 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:21.827 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:25:21.827 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:21.828 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:21.828 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:21.828 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:48.319251+0000", 2024-09-13T11:25:21.828 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:21.828 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:21.828 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:21.828 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:21.828 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:48.319251+0000", 2024-09-13T11:25:21.828 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:21.828 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:21.828 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:21.828 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:21.829 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:48.319252+0000", 2024-09-13T11:25:21.829 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2640000000000001e-06 2024-09-13T11:25:21.829 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:21.829 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:21.829 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:21.829 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:48.319363+0000", 2024-09-13T11:25:21.829 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000110559 2024-09-13T11:25:21.829 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:21.829 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:21.829 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:21.829 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:48.927115+0000", 2024-09-13T11:25:21.829 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.60775277800000005 2024-09-13T11:25:21.829 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:21.829 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:21.829 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:21.830 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:48.927144+0000", 2024-09-13T11:25:21.830 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.8861e-05 2024-09-13T11:25:21.830 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:21.830 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:21.830 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:21.830 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:48.927318+0000", 2024-09-13T11:25:21.830 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00017399000000000001 2024-09-13T11:25:21.830 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:21.830 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:21.830 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:25:21.831 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:48.927340+0000", 2024-09-13T11:25:21.831 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.1433999999999998e-05 2024-09-13T11:25:21.831 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:21.831 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:21.831 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:21.831 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:48.928289+0000", 2024-09-13T11:25:21.831 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00094913599999999995 2024-09-13T11:25:21.831 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:21.831 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:21.831 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:21.831 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:21.831 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:21.831 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:69487 3.e 3:73f7f990:::benchmark_data_smithi177_31116_object69486:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94)", 2024-09-13T11:25:21.832 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:52.063559+0000", 2024-09-13T11:25:21.832 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 89.736466851000003, 2024-09-13T11:25:21.832 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.20206200199999999, 2024-09-13T11:25:21.832 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:21.832 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:21.832 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:21.832 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:25:21.832 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:25:21.832 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:21.832 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 69487 2024-09-13T11:25:21.832 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:21.832 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:21.832 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:21.832 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:21.833 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:52.063559+0000", 2024-09-13T11:25:21.833 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:21.833 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:21.833 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:21.833 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:21.833 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:52.063559+0000", 2024-09-13T11:25:21.833 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:21.833 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:21.833 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:21.833 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:21.833 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:52.063560+0000", 2024-09-13T11:25:21.833 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.167e-06 2024-09-13T11:25:21.833 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:21.833 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:21.834 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:21.834 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:52.063612+0000", 2024-09-13T11:25:21.834 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.1709999999999998e-05 2024-09-13T11:25:21.834 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:21.834 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:21.834 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:21.834 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:52.063623+0000", 2024-09-13T11:25:21.834 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.1251e-05 2024-09-13T11:25:21.834 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:21.834 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:21.834 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:21.834 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:52.063631+0000", 2024-09-13T11:25:21.834 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.7319999999999998e-06 2024-09-13T11:25:21.834 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:21.835 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:21.835 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:21.835 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:52.064267+0000", 2024-09-13T11:25:21.835 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00063608600000000001 2024-09-13T11:25:21.835 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:21.835 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:21.835 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:25:21.835 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:52.064407+0000", 2024-09-13T11:25:21.835 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000140686 2024-09-13T11:25:21.835 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:21.835 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:21.835 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 1", 2024-09-13T11:25:21.835 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:52.064540+0000", 2024-09-13T11:25:21.835 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000132244 2024-09-13T11:25:21.836 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:21.836 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:21.836 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:25:21.836 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:52.067019+0000", 2024-09-13T11:25:21.836 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00247919 2024-09-13T11:25:21.836 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:21.836 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:21.836 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:25:21.836 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:52.265543+0000", 2024-09-13T11:25:21.836 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.19852367800000001 2024-09-13T11:25:21.836 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:21.836 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:21.836 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:25:21.836 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:52.265576+0000", 2024-09-13T11:25:21.837 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.3550999999999998e-05 2024-09-13T11:25:21.837 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:21.837 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:21.837 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:21.837 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:52.265621+0000", 2024-09-13T11:25:21.837 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.4706999999999997e-05 2024-09-13T11:25:21.837 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:21.837 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:21.837 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:21.837 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:21.837 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:21.837 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4665.0:69556 3.18 e94/91 3:1d3dbeba:::benchmark_data_smithi177_31116_object69555:head v 94'3570, mlcod=94'3566)", 2024-09-13T11:25:21.837 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:52.142942+0000", 2024-09-13T11:25:21.837 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 89.657083349000004, 2024-09-13T11:25:21.838 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.21178485399999999, 2024-09-13T11:25:21.838 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:21.838 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:21.838 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:21.838 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:21.838 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:21.838 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:21.838 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:52.142942+0000", 2024-09-13T11:25:21.838 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:21.838 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:21.840 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:21.840 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:21.840 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:52.142942+0000", 2024-09-13T11:25:21.840 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:21.840 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:21.840 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:21.840 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:21.840 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:52.142944+0000", 2024-09-13T11:25:21.840 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.4500000000000001e-06 2024-09-13T11:25:21.840 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:21.840 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:21.840 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:21.840 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:52.143047+0000", 2024-09-13T11:25:21.840 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000103384 2024-09-13T11:25:21.841 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:21.841 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:21.841 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:21.841 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:52.353322+0000", 2024-09-13T11:25:21.841 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.21027483299999999 2024-09-13T11:25:21.841 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:21.841 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:21.841 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:21.841 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:52.353330+0000", 2024-09-13T11:25:21.841 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.8879999999999997e-06 2024-09-13T11:25:21.841 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:21.841 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:21.841 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:21.841 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:52.353402+0000", 2024-09-13T11:25:21.842 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.2155000000000003e-05 2024-09-13T11:25:21.842 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:21.842 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:21.842 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:25:21.842 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:52.353426+0000", 2024-09-13T11:25:21.842 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.3774e-05 2024-09-13T11:25:21.842 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:21.842 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:21.842 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:25:21.842 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:52.354674+0000", 2024-09-13T11:25:21.843 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.001248706 2024-09-13T11:25:21.843 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:21.843 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:21.843 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:21.843 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:52.354727+0000", 2024-09-13T11:25:21.843 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.2664000000000003e-05 2024-09-13T11:25:21.843 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:21.843 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:21.843 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:21.843 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:21.843 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:21.843 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:70749 3.1e 3:7bd7bfc6:::benchmark_data_smithi177_31116_object70748:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95)", 2024-09-13T11:25:21.843 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:53.156250+0000", 2024-09-13T11:25:21.844 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 88.643775528000006, 2024-09-13T11:25:21.844 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.22394481799999999, 2024-09-13T11:25:21.844 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:21.844 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:21.844 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:21.844 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:25:21.844 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:25:21.844 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:21.844 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 70749 2024-09-13T11:25:21.844 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:21.844 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:21.844 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:21.844 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:21.844 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:53.156250+0000", 2024-09-13T11:25:21.845 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:21.845 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:21.845 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:21.845 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:21.845 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:53.156250+0000", 2024-09-13T11:25:21.845 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:21.845 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:21.845 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:21.845 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:21.845 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:53.156252+0000", 2024-09-13T11:25:21.845 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.3460000000000001e-06 2024-09-13T11:25:21.846 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:21.846 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:21.846 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:21.846 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:53.156336+0000", 2024-09-13T11:25:21.846 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.3714999999999994e-05 2024-09-13T11:25:21.846 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:21.846 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:21.846 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:21.846 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:53.156347+0000", 2024-09-13T11:25:21.846 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0903999999999999e-05 2024-09-13T11:25:21.846 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:21.846 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:21.846 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:21.846 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:53.156352+0000", 2024-09-13T11:25:21.847 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.9139999999999999e-06 2024-09-13T11:25:21.847 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:21.847 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:21.847 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:21.847 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:53.158028+0000", 2024-09-13T11:25:21.847 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.001675958 2024-09-13T11:25:21.847 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:21.847 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:21.847 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:25:21.847 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:53.158154+0000", 2024-09-13T11:25:21.847 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00012653 2024-09-13T11:25:21.847 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:21.847 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:21.848 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 4,5", 2024-09-13T11:25:21.848 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:53.158277+0000", 2024-09-13T11:25:21.848 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000122168 2024-09-13T11:25:21.848 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:21.848 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:21.848 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:25:21.848 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:53.159585+0000", 2024-09-13T11:25:21.848 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.001308528 2024-09-13T11:25:21.848 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:21.848 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:21.848 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:25:21.848 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:53.163172+0000", 2024-09-13T11:25:21.848 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0035869690000000002 2024-09-13T11:25:21.849 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:21.849 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:21.849 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:25:21.849 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:53.379730+0000", 2024-09-13T11:25:21.849 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.21655822599999999 2024-09-13T11:25:21.849 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:21.849 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:21.849 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:25:21.849 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:53.379810+0000", 2024-09-13T11:25:21.849 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.0041000000000002e-05 2024-09-13T11:25:21.849 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:21.849 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:21.849 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:21.849 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:53.380195+0000", 2024-09-13T11:25:21.850 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000384519 2024-09-13T11:25:21.850 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:21.850 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:21.850 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:21.850 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:21.850 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:21.850 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDPGPush(3.21 96/78 [PushOp(3:862d73af:::benchmark_data_smithi177_31116_object32719:head, version: 60'2029, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:862d73af:::benchmark_data_smithi177_31116_object32719:head@60'2029, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:25:21.850 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:53.884628+0000", 2024-09-13T11:25:21.850 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 87.915397686000006, 2024-09-13T11:25:21.850 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.21020203400000001, 2024-09-13T11:25:21.850 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:21.850 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:21.851 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:25:21.851 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:21.851 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:21.851 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:21.851 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:53.884628+0000", 2024-09-13T11:25:21.851 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:21.851 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:21.851 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:21.851 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:21.851 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:53.884628+0000", 2024-09-13T11:25:21.851 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:21.851 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:21.851 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:21.852 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:21.852 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:53.884630+0000", 2024-09-13T11:25:21.852 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.68e-06 2024-09-13T11:25:21.852 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:21.852 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:21.852 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:21.852 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:53.884745+0000", 2024-09-13T11:25:21.852 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000115599 2024-09-13T11:25:21.852 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:21.852 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:21.852 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:21.852 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:54.093948+0000", 2024-09-13T11:25:21.852 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.20920306999999999 2024-09-13T11:25:21.853 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:21.853 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:21.853 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:21.853 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:54.093961+0000", 2024-09-13T11:25:21.853 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2815e-05 2024-09-13T11:25:21.853 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:21.853 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:21.853 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:21.853 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:54.094139+0000", 2024-09-13T11:25:21.853 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00017824 2024-09-13T11:25:21.853 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:21.853 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:21.853 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:25:21.853 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:54.094150+0000", 2024-09-13T11:25:21.854 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0593e-05 2024-09-13T11:25:21.854 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:21.854 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:21.854 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:21.854 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:54.094830+0000", 2024-09-13T11:25:21.854 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00068003699999999998 2024-09-13T11:25:21.854 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:21.854 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:21.854 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:21.854 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:21.854 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:21.854 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:83239 3.e 3:74547f17:::benchmark_data_smithi177_31116_object83238:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106)", 2024-09-13T11:25:21.854 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:24:04.580085+0000", 2024-09-13T11:25:21.854 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 77.219940803, 2024-09-13T11:25:21.855 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.33785762600000002, 2024-09-13T11:25:21.855 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:21.855 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:21.855 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:21.855 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:25:21.855 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:25:21.855 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:21.855 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 83239 2024-09-13T11:25:21.855 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:21.855 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:21.855 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:21.855 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:21.855 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:04.580085+0000", 2024-09-13T11:25:21.855 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:21.856 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:21.856 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:21.856 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:21.856 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:04.580085+0000", 2024-09-13T11:25:21.856 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:21.856 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:21.856 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:21.856 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:21.856 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:04.580089+0000", 2024-09-13T11:25:21.856 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.053e-06 2024-09-13T11:25:21.856 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:21.856 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:21.856 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:21.856 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:04.580166+0000", 2024-09-13T11:25:21.857 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.7102999999999997e-05 2024-09-13T11:25:21.857 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:21.857 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:21.857 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:21.857 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:04.580196+0000", 2024-09-13T11:25:21.857 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.0196e-05 2024-09-13T11:25:21.857 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:21.857 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:21.857 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:21.857 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:04.580212+0000", 2024-09-13T11:25:21.857 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.535e-05 2024-09-13T11:25:21.857 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:21.857 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:21.857 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:21.858 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:04.580336+0000", 2024-09-13T11:25:21.858 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00012431899999999999 2024-09-13T11:25:21.858 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:21.858 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:21.858 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:25:21.858 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:04.580575+0000", 2024-09-13T11:25:21.858 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00023924100000000001 2024-09-13T11:25:21.858 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:21.858 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:21.858 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 1", 2024-09-13T11:25:21.858 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:04.580798+0000", 2024-09-13T11:25:21.858 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00022301499999999999 2024-09-13T11:25:21.858 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:21.858 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:21.859 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:25:21.859 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:04.583404+0000", 2024-09-13T11:25:21.859 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0026056120000000002 2024-09-13T11:25:21.859 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:21.859 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:21.859 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:25:21.859 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:04.917768+0000", 2024-09-13T11:25:21.859 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.33436387499999998 2024-09-13T11:25:21.859 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:21.859 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:21.859 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:25:21.860 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:04.917859+0000", 2024-09-13T11:25:21.860 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.1732999999999997e-05 2024-09-13T11:25:21.860 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:21.860 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:21.860 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:21.860 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:04.917942+0000", 2024-09-13T11:25:21.860 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.3128999999999996e-05 2024-09-13T11:25:21.860 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:21.860 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:21.860 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:21.861 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:21.861 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:21.861 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:84571 3.21 3:85e7a789:::benchmark_data_smithi177_31116_object84570:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-09-13T11:25:21.861 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:24:05.695256+0000", 2024-09-13T11:25:21.861 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 76.104769771999997, 2024-09-13T11:25:21.861 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.18917198299999999, 2024-09-13T11:25:21.861 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:21.861 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:21.861 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:21.861 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:25:21.861 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:25:21.861 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:21.861 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 84571 2024-09-13T11:25:21.861 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:21.862 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:21.862 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:21.862 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:21.862 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:05.695256+0000", 2024-09-13T11:25:21.862 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:21.862 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:21.862 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:21.862 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:21.862 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:05.695256+0000", 2024-09-13T11:25:21.862 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:21.862 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:21.862 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:21.862 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:21.862 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:05.695258+0000", 2024-09-13T11:25:21.863 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.0530000000000001e-06 2024-09-13T11:25:21.863 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:21.863 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:21.863 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:21.863 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:05.695304+0000", 2024-09-13T11:25:21.863 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.5769999999999997e-05 2024-09-13T11:25:21.863 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:21.863 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:21.863 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:21.863 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:05.695313+0000", 2024-09-13T11:25:21.863 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.7049999999999994e-06 2024-09-13T11:25:21.864 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:21.864 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:21.864 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:21.864 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:05.695321+0000", 2024-09-13T11:25:21.864 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.7190000000000001e-06 2024-09-13T11:25:21.864 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:21.864 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:21.864 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:21.864 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:05.695404+0000", 2024-09-13T11:25:21.864 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.3188000000000004e-05 2024-09-13T11:25:21.864 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:21.864 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:21.864 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:25:21.865 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:05.695635+0000", 2024-09-13T11:25:21.865 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00023032900000000001 2024-09-13T11:25:21.865 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:21.865 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:21.865 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 8", 2024-09-13T11:25:21.865 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:05.695852+0000", 2024-09-13T11:25:21.865 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00021749599999999999 2024-09-13T11:25:21.865 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:21.865 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:21.865 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:25:21.866 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:05.697208+0000", 2024-09-13T11:25:21.866 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00135633 2024-09-13T11:25:21.866 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:21.866 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:21.866 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:25:21.866 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:05.884337+0000", 2024-09-13T11:25:21.866 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.187128867 2024-09-13T11:25:21.866 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:21.866 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:21.866 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:25:21.866 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:05.884391+0000", 2024-09-13T11:25:21.866 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.3724999999999998e-05 2024-09-13T11:25:21.866 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:21.866 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:21.867 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:21.867 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:05.884428+0000", 2024-09-13T11:25:21.867 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.6801000000000002e-05 2024-09-13T11:25:21.867 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:21.867 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:21.867 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:21.867 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:21.867 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:21.867 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4665.0:104290 3.20 e115/77 3:05d45b57:::benchmark_data_smithi177_31116_object104289:head v 115'3905, mlcod=115'3902)", 2024-09-13T11:25:21.867 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:24:20.200225+0000", 2024-09-13T11:25:21.867 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 61.599800506999998, 2024-09-13T11:25:21.867 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.21051286899999999, 2024-09-13T11:25:21.867 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:21.867 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:21.868 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:21.868 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:21.868 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:21.868 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:21.868 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:20.200225+0000", 2024-09-13T11:25:21.868 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:21.868 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:21.868 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:21.868 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:21.868 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:20.200225+0000", 2024-09-13T11:25:21.868 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:21.868 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:21.868 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:21.868 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:21.869 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:20.200227+0000", 2024-09-13T11:25:21.869 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.897e-06 2024-09-13T11:25:21.869 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:21.869 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:21.869 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:21.869 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:20.200253+0000", 2024-09-13T11:25:21.869 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.5904000000000001e-05 2024-09-13T11:25:21.869 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:21.869 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:21.869 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:21.869 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:20.409353+0000", 2024-09-13T11:25:21.869 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.209100339 2024-09-13T11:25:21.869 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:21.869 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:21.870 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:21.870 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:20.409359+0000", 2024-09-13T11:25:21.870 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.1689999999999997e-06 2024-09-13T11:25:21.870 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:21.870 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:21.870 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:21.870 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:20.409418+0000", 2024-09-13T11:25:21.870 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.8761999999999997e-05 2024-09-13T11:25:21.870 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:21.870 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:21.870 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:25:21.870 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:20.409440+0000", 2024-09-13T11:25:21.871 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.2098000000000001e-05 2024-09-13T11:25:21.871 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:21.871 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:21.871 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:25:21.871 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:20.410705+0000", 2024-09-13T11:25:21.871 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00126483 2024-09-13T11:25:21.871 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:21.871 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:21.871 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:21.871 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:20.410738+0000", 2024-09-13T11:25:21.871 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.2870000000000002e-05 2024-09-13T11:25:21.871 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:21.872 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:21.872 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:21.872 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:21.872 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:21.872 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:110271 3.2d 3.3c2681ad (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e119)", 2024-09-13T11:25:21.872 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:24:25.403006+0000", 2024-09-13T11:25:21.872 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 56.397019319000002, 2024-09-13T11:25:21.872 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.68768755999999998, 2024-09-13T11:25:21.872 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:21.873 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:21.873 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-13T11:25:21.873 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:25:21.873 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:25:21.873 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:21.873 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 110271 2024-09-13T11:25:21.873 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:21.873 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:21.874 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:21.874 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:21.874 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:25.403006+0000", 2024-09-13T11:25:21.874 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:21.874 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:21.874 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:21.874 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:21.874 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:25.403006+0000", 2024-09-13T11:25:21.874 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:21.874 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:21.874 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:21.874 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:21.874 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:25.403011+0000", 2024-09-13T11:25:21.874 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.5199999999999999e-06 2024-09-13T11:25:21.875 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:21.875 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:21.875 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:21.875 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:25.403065+0000", 2024-09-13T11:25:21.875 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.4478999999999998e-05 2024-09-13T11:25:21.875 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:21.876 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:21.876 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:21.876 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:25.403097+0000", 2024-09-13T11:25:21.876 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.1636e-05 2024-09-13T11:25:21.876 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:21.876 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:21.876 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:21.876 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:25.403121+0000", 2024-09-13T11:25:21.876 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.3924e-05 2024-09-13T11:25:21.876 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:21.876 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:21.876 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:21.876 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:25.403216+0000", 2024-09-13T11:25:21.877 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.5557999999999995e-05 2024-09-13T11:25:21.877 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:21.877 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:21.877 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for peered", 2024-09-13T11:25:21.877 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:25.403234+0000", 2024-09-13T11:25:21.877 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.7104000000000001e-05 2024-09-13T11:25:21.877 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:21.877 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:21.877 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:21.877 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:26.090684+0000", 2024-09-13T11:25:21.877 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.68745000700000003 2024-09-13T11:25:21.877 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:21.877 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:21.877 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:21.878 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:26.090694+0000", 2024-09-13T11:25:21.878 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0332e-05 2024-09-13T11:25:21.878 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:21.878 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:21.878 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:21.878 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:21.878 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:21.878 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDPGPushReply(3.22 125/122 [PushReplyOp(3:475b51c3:::benchmark_data_smithi177_31116_object16005:head)])", 2024-09-13T11:25:21.878 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:24:32.400814+0000", 2024-09-13T11:25:21.878 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 49.399211555000001, 2024-09-13T11:25:21.878 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.20917021799999999, 2024-09-13T11:25:21.879 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:21.879 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:21.879 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-13T11:25:21.880 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:21.880 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:21.880 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:21.880 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:32.400814+0000", 2024-09-13T11:25:21.880 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:21.881 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:21.881 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:21.881 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:21.881 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:32.400814+0000", 2024-09-13T11:25:21.881 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:21.881 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:21.881 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:21.881 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:21.881 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:32.400817+0000", 2024-09-13T11:25:21.881 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.5000000000000002e-06 2024-09-13T11:25:21.881 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:21.881 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:21.881 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:21.881 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:32.400826+0000", 2024-09-13T11:25:21.881 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.8830000000000001e-06 2024-09-13T11:25:21.882 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:21.882 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:21.882 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:21.882 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:32.608921+0000", 2024-09-13T11:25:21.882 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.20809501599999999 2024-09-13T11:25:21.882 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:21.882 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:21.882 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:21.882 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:32.608931+0000", 2024-09-13T11:25:21.882 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.1679999999999993e-06 2024-09-13T11:25:21.882 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:21.882 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:21.882 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:21.883 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:32.608982+0000", 2024-09-13T11:25:21.883 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.1827999999999999e-05 2024-09-13T11:25:21.883 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:21.883 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:21.883 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:21.883 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:32.609984+0000", 2024-09-13T11:25:21.883 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.001001823 2024-09-13T11:25:21.883 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:21.883 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:21.883 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:21.883 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:21.883 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:21.884 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:120816 3.21 3:850a51ba:::benchmark_data_smithi177_31116_object120815:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e129)", 2024-09-13T11:25:21.884 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:24:38.601204+0000", 2024-09-13T11:25:21.884 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 43.198821520999999, 2024-09-13T11:25:21.884 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.18932929600000001, 2024-09-13T11:25:21.884 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:21.884 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:21.884 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:21.884 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:25:21.884 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:25:21.884 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:21.884 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 120816 2024-09-13T11:25:21.884 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:21.884 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:21.884 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:21.885 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:21.885 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:38.601204+0000", 2024-09-13T11:25:21.885 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:21.885 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:21.885 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:21.885 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:21.885 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:38.601204+0000", 2024-09-13T11:25:21.885 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:21.885 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:21.885 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:21.885 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:21.885 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:38.601206+0000", 2024-09-13T11:25:21.885 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.7370000000000001e-06 2024-09-13T11:25:21.885 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:21.886 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:21.886 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:21.886 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:38.601241+0000", 2024-09-13T11:25:21.886 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.5524000000000001e-05 2024-09-13T11:25:21.886 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:21.886 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:21.886 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:21.886 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:38.601253+0000", 2024-09-13T11:25:21.886 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.1936000000000001e-05 2024-09-13T11:25:21.886 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:21.886 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:21.886 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:21.886 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:38.601262+0000", 2024-09-13T11:25:21.886 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.2570000000000007e-06 2024-09-13T11:25:21.887 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:21.887 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:21.887 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:21.887 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:38.601339+0000", 2024-09-13T11:25:21.887 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.7751999999999997e-05 2024-09-13T11:25:21.887 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:21.887 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:21.887 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:25:21.887 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:38.601527+0000", 2024-09-13T11:25:21.887 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000188092 2024-09-13T11:25:21.887 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:21.887 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:21.887 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 8", 2024-09-13T11:25:21.887 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:38.601684+0000", 2024-09-13T11:25:21.888 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00015659699999999999 2024-09-13T11:25:21.888 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:21.888 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:21.888 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:25:21.888 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:38.603108+0000", 2024-09-13T11:25:21.888 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0014244240000000001 2024-09-13T11:25:21.888 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:21.888 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:21.888 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:25:21.888 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:38.790425+0000", 2024-09-13T11:25:21.888 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.187316345 2024-09-13T11:25:21.888 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:21.888 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:21.889 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:25:21.889 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:38.790488+0000", 2024-09-13T11:25:21.889 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.3184999999999999e-05 2024-09-13T11:25:21.889 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:21.889 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:21.889 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:21.889 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:38.790533+0000", 2024-09-13T11:25:21.889 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.5447e-05 2024-09-13T11:25:21.889 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:21.889 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:21.889 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:21.889 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:21.890 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:21.890 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4665.0:129901 3.20 e134/77 3:067f5dfc:::benchmark_data_smithi177_31116_object129900:head v 134'4294, mlcod=134'4291)", 2024-09-13T11:25:21.890 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:24:47.145530+0000", 2024-09-13T11:25:21.890 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 34.654495244000003, 2024-09-13T11:25:21.890 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.333688349, 2024-09-13T11:25:21.890 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:21.890 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:21.890 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:21.890 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:21.890 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:21.890 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:21.890 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:47.145530+0000", 2024-09-13T11:25:21.891 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:21.891 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:21.891 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:21.891 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:21.891 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:47.145530+0000", 2024-09-13T11:25:21.891 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:21.891 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:21.891 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:21.891 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:21.891 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:47.145532+0000", 2024-09-13T11:25:21.891 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.7999999999999999e-06 2024-09-13T11:25:21.891 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:21.891 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:21.891 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:21.891 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:47.145557+0000", 2024-09-13T11:25:21.892 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.4678999999999999e-05 2024-09-13T11:25:21.892 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:21.892 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:21.892 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:21.892 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:47.477539+0000", 2024-09-13T11:25:21.892 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.33198164099999999 2024-09-13T11:25:21.892 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:21.892 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:21.892 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:21.892 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:47.477544+0000", 2024-09-13T11:25:21.892 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.9059999999999999e-06 2024-09-13T11:25:21.892 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:21.892 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:21.892 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:21.893 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:47.477597+0000", 2024-09-13T11:25:21.893 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.295e-05 2024-09-13T11:25:21.893 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:21.893 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:21.893 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:25:21.893 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:47.477619+0000", 2024-09-13T11:25:21.893 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.1835999999999999e-05 2024-09-13T11:25:21.893 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:21.893 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:21.893 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:25:21.893 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:47.479175+0000", 2024-09-13T11:25:21.893 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0015560470000000001 2024-09-13T11:25:21.893 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:21.893 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:21.894 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:21.894 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:47.479219+0000", 2024-09-13T11:25:21.894 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.3489999999999999e-05 2024-09-13T11:25:21.894 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:21.894 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:21.894 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:21.894 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:21.894 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:21.894 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:130006 3.21 3:85b3e472:::benchmark_data_smithi177_31116_object130005:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e134)", 2024-09-13T11:25:21.894 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:24:47.433580+0000", 2024-09-13T11:25:21.894 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 34.366445450999997, 2024-09-13T11:25:21.894 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.189659507, 2024-09-13T11:25:21.894 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:21.894 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:21.895 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:21.895 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:25:21.895 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:25:21.895 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:21.895 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 130006 2024-09-13T11:25:21.895 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:21.895 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:21.895 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:21.895 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:21.895 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:47.433580+0000", 2024-09-13T11:25:21.895 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:21.895 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:21.895 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:21.895 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:21.895 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:47.433580+0000", 2024-09-13T11:25:21.896 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:21.896 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:21.896 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:21.896 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:21.896 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:47.433582+0000", 2024-09-13T11:25:21.896 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.266e-06 2024-09-13T11:25:21.896 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:21.896 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:21.896 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:21.896 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:47.433656+0000", 2024-09-13T11:25:21.896 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.3780000000000002e-05 2024-09-13T11:25:21.896 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:21.896 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:21.896 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:21.897 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:47.433796+0000", 2024-09-13T11:25:21.897 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000139627 2024-09-13T11:25:21.897 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:21.897 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:21.897 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:21.897 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:47.433803+0000", 2024-09-13T11:25:21.897 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.5809999999999999e-06 2024-09-13T11:25:21.897 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:21.897 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:21.897 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:21.897 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:47.433849+0000", 2024-09-13T11:25:21.897 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.5485000000000002e-05 2024-09-13T11:25:21.897 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:21.897 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:21.898 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:25:21.898 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:47.433946+0000", 2024-09-13T11:25:21.898 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.6788999999999998e-05 2024-09-13T11:25:21.898 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:21.898 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:21.898 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 8", 2024-09-13T11:25:21.898 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:47.434032+0000", 2024-09-13T11:25:21.898 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.5898999999999999e-05 2024-09-13T11:25:21.898 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:21.898 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:21.898 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:25:21.898 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:47.435315+0000", 2024-09-13T11:25:21.898 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0012834319999999999 2024-09-13T11:25:21.898 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:21.899 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:21.899 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:25:21.899 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:47.623177+0000", 2024-09-13T11:25:21.899 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.187861741 2024-09-13T11:25:21.899 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:21.899 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:21.899 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:25:21.899 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:47.623211+0000", 2024-09-13T11:25:21.899 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.3989000000000003e-05 2024-09-13T11:25:21.899 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:21.899 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:21.899 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:21.899 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:47.623240+0000", 2024-09-13T11:25:21.899 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.8918e-05 2024-09-13T11:25:21.900 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:21.900 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:21.900 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:21.900 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:21.900 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:21.900 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:140840 3.21 3:87ae3db6:::benchmark_data_smithi177_31116_object537:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e144)", 2024-09-13T11:25:21.900 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:24:58.810939+0000", 2024-09-13T11:25:21.900 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 22.989086131000001, 2024-09-13T11:25:21.900 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.25628309300000002, 2024-09-13T11:25:21.900 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:21.900 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:21.900 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:21.900 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:25:21.900 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:25:21.901 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:21.901 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 140840 2024-09-13T11:25:21.901 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:21.901 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:21.901 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:21.901 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:21.901 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.810939+0000", 2024-09-13T11:25:21.901 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:21.901 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:21.901 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:21.901 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:21.901 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.810939+0000", 2024-09-13T11:25:21.901 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:21.901 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:21.901 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:21.902 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:21.902 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.810942+0000", 2024-09-13T11:25:21.902 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.6259999999999999e-06 2024-09-13T11:25:21.902 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:21.902 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:21.902 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:21.902 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.810950+0000", 2024-09-13T11:25:21.902 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.9500000000000001e-06 2024-09-13T11:25:21.902 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:21.902 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:21.902 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:21.902 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.810970+0000", 2024-09-13T11:25:21.902 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.9539999999999999e-05 2024-09-13T11:25:21.902 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:21.903 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:21.903 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:21.903 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.810978+0000", 2024-09-13T11:25:21.903 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.8680000000000005e-06 2024-09-13T11:25:21.903 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:21.903 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:21.903 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:21.903 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.811060+0000", 2024-09-13T11:25:21.903 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.1068000000000004e-05 2024-09-13T11:25:21.903 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:21.903 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:21.903 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:25:21.903 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.811174+0000", 2024-09-13T11:25:21.903 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000114385 2024-09-13T11:25:21.904 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:21.904 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:21.904 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 8", 2024-09-13T11:25:21.904 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.811255+0000", 2024-09-13T11:25:21.904 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.1298000000000007e-05 2024-09-13T11:25:21.904 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:21.904 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:21.904 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:25:21.904 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.812869+0000", 2024-09-13T11:25:21.904 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.001614234 2024-09-13T11:25:21.904 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:21.904 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:21.904 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:25:21.904 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:59.067155+0000", 2024-09-13T11:25:21.904 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.25428578299999999 2024-09-13T11:25:21.905 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:21.905 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:21.905 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:25:21.905 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:59.067192+0000", 2024-09-13T11:25:21.905 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.659e-05 2024-09-13T11:25:21.905 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:21.905 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:21.905 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:21.905 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:59.067223+0000", 2024-09-13T11:25:21.905 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.0750999999999997e-05 2024-09-13T11:25:21.905 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:21.905 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:21.905 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:21.905 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:21.906 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:21.906 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-13T11:25:21.907 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-13T11:25:21.908 INFO:tasks.ceph.osd.4.smithi001.stderr:2024-09-13T11:25:21.903+0000 7ff8e0218640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26310) UID: 0 2024-09-13T11:25:22.008 INFO:tasks.ceph.osd.11.smithi177.stderr:2024-09-13T11:25:22.006+0000 7f24792c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26580) UID: 0 2024-09-13T11:25:22.095 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-13T11:25:22.096 INFO:teuthology.orchestra.run.smithi001.stdout: "ops": [ 2024-09-13T11:25:22.096 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:22.096 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:171176 3.11 3:8f13f92a:::benchmark_data_smithi177_31116_object30873:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e162)", 2024-09-13T11:25:22.096 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:25:22.091297+0000", 2024-09-13T11:25:22.096 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 0.0039484979999999999, 2024-09-13T11:25:22.096 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00396686, 2024-09-13T11:25:22.096 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:22.096 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:22.096 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "waiting for sub ops", 2024-09-13T11:25:22.096 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:25:22.096 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:25:22.096 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:22.096 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 171176 2024-09-13T11:25:22.097 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:22.097 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:22.097 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:22.097 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:22.097 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:22.091297+0000", 2024-09-13T11:25:22.097 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:22.097 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:22.097 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:22.097 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:22.097 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:22.091297+0000", 2024-09-13T11:25:22.097 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:22.097 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:22.098 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:22.098 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:22.098 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:22.091300+0000", 2024-09-13T11:25:22.098 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.8310000000000001e-06 2024-09-13T11:25:22.098 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:22.098 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:22.098 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:22.098 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:22.091308+0000", 2024-09-13T11:25:22.098 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.0199999999999994e-06 2024-09-13T11:25:22.098 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:22.098 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:22.098 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:22.098 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:22.091325+0000", 2024-09-13T11:25:22.098 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.7825e-05 2024-09-13T11:25:22.099 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:22.099 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:22.099 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:22.099 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:22.091333+0000", 2024-09-13T11:25:22.099 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.8509999999999999e-06 2024-09-13T11:25:22.099 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:22.099 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:22.099 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:22.099 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:22.091406+0000", 2024-09-13T11:25:22.099 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.2467999999999998e-05 2024-09-13T11:25:22.099 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:22.099 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:22.099 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:25:22.099 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:22.091516+0000", 2024-09-13T11:25:22.100 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000110122 2024-09-13T11:25:22.100 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:22.100 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:22.100 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 11", 2024-09-13T11:25:22.100 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:22.091589+0000", 2024-09-13T11:25:22.100 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.3208999999999997e-05 2024-09-13T11:25:22.100 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:22.100 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:22.100 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:25:22.100 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:22.093587+0000", 2024-09-13T11:25:22.100 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0019976730000000002 2024-09-13T11:25:22.100 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:22.100 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:22.101 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:22.101 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:22.101 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:22.101 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:171177 3.1a 3:5accb86f:::benchmark_data_smithi177_31116_object30874:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e162)", 2024-09-13T11:25:22.101 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:25:22.094559+0000", 2024-09-13T11:25:22.101 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 0.00068581999999999996, 2024-09-13T11:25:22.101 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00075745599999999999, 2024-09-13T11:25:22.101 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:22.101 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:22.101 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "waiting for sub ops", 2024-09-13T11:25:22.101 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:25:22.101 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:25:22.101 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:22.101 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 171177 2024-09-13T11:25:22.102 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:22.102 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:22.102 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:22.102 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:22.102 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:22.094559+0000", 2024-09-13T11:25:22.102 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:22.102 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:22.102 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:22.102 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:22.102 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:22.094559+0000", 2024-09-13T11:25:22.102 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:22.102 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:22.102 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:22.102 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:22.103 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:22.094561+0000", 2024-09-13T11:25:22.103 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2240000000000001e-06 2024-09-13T11:25:22.103 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:22.103 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:22.103 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:22.103 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:22.094564+0000", 2024-09-13T11:25:22.103 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.3330000000000001e-06 2024-09-13T11:25:22.103 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:22.103 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:22.103 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:22.103 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:22.094572+0000", 2024-09-13T11:25:22.103 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.5499999999999995e-06 2024-09-13T11:25:22.103 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:22.103 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:22.104 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:22.104 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:22.094579+0000", 2024-09-13T11:25:22.104 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.3470000000000004e-06 2024-09-13T11:25:22.104 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:22.104 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:22.104 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:22.104 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:22.094633+0000", 2024-09-13T11:25:22.104 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.3937000000000002e-05 2024-09-13T11:25:22.104 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:22.104 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:22.104 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:25:22.104 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:22.094727+0000", 2024-09-13T11:25:22.104 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.4685000000000006e-05 2024-09-13T11:25:22.105 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:22.105 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:22.105 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 9", 2024-09-13T11:25:22.105 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:22.094796+0000", 2024-09-13T11:25:22.105 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.8844999999999996e-05 2024-09-13T11:25:22.105 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:22.105 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:22.105 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:22.105 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:22.105 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:22.105 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:171178 3.11 3:8f83badf:::benchmark_data_smithi177_31116_object30875:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e162)", 2024-09-13T11:25:22.105 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:25:22.094597+0000", 2024-09-13T11:25:22.105 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 0.00064799599999999997, 2024-09-13T11:25:22.105 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00075423299999999999, 2024-09-13T11:25:22.106 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:22.106 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:22.106 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "waiting for sub ops", 2024-09-13T11:25:22.106 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:25:22.106 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:25:22.106 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:22.106 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 171178 2024-09-13T11:25:22.106 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:22.106 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:22.106 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:22.106 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:22.106 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:22.094597+0000", 2024-09-13T11:25:22.106 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:22.107 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:22.107 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:22.107 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:22.107 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:22.094597+0000", 2024-09-13T11:25:22.107 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:22.107 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:22.107 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:22.107 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:22.107 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:22.094598+0000", 2024-09-13T11:25:22.107 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.9300000000000006e-07 2024-09-13T11:25:22.107 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:22.107 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:22.107 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:22.107 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:22.094602+0000", 2024-09-13T11:25:22.108 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.4479999999999999e-06 2024-09-13T11:25:22.108 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:22.108 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:22.108 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:22.108 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:22.094608+0000", 2024-09-13T11:25:22.108 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.7320000000000001e-06 2024-09-13T11:25:22.108 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:22.108 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:22.108 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:22.108 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:22.094614+0000", 2024-09-13T11:25:22.108 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.144e-06 2024-09-13T11:25:22.108 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:22.108 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:22.109 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:22.109 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:22.094664+0000", 2024-09-13T11:25:22.109 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.9472999999999999e-05 2024-09-13T11:25:22.109 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:22.109 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:22.109 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:25:22.109 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:22.094763+0000", 2024-09-13T11:25:22.109 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.9468000000000004e-05 2024-09-13T11:25:22.109 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:22.109 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:22.109 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 11", 2024-09-13T11:25:22.109 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:22.094867+0000", 2024-09-13T11:25:22.109 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00010395099999999999 2024-09-13T11:25:22.109 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:22.110 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:22.110 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:22.110 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:22.110 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-09-13T11:25:22.110 INFO:teuthology.orchestra.run.smithi001.stdout: "num_ops": 3 2024-09-13T11:25:22.110 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-13T11:25:22.111 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-13T11:25:22.112 INFO:tasks.ceph.osd.10.smithi162.stderr:2024-09-13T11:25:22.108+0000 7fb59c9f0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26317) UID: 0 2024-09-13T11:25:22.179 INFO:teuthology.orchestra.run.smithi001.stderr:reweighted osd.11 to 1 (10000) 2024-09-13T11:25:22.197 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-13T11:25:22.206 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 12 1 2024-09-13T11:25:22.211 INFO:tasks.ceph.osd.7.smithi177.stderr:2024-09-13T11:25:22.210+0000 7f897b51d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26575) UID: 0 2024-09-13T11:25:22.261 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-13T11:25:22.261 INFO:teuthology.orchestra.run.smithi001.stdout: "ops": [], 2024-09-13T11:25:22.261 INFO:teuthology.orchestra.run.smithi001.stdout: "complaint_time": 30, 2024-09-13T11:25:22.262 INFO:teuthology.orchestra.run.smithi001.stdout: "num_blocked_ops": 0 2024-09-13T11:25:22.262 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-13T11:25:22.287 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-13T11:25:22.312 INFO:tasks.ceph.osd.2.smithi162.stderr:2024-09-13T11:25:22.308+0000 7fe92b35a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26313) UID: 0 2024-09-13T11:25:22.380 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T11:25:22.412 INFO:tasks.ceph.osd.7.smithi177.stderr:2024-09-13T11:25:22.410+0000 7f897b51d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26575) UID: 0 2024-09-13T11:25:22.502 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-13T11:25:22.502 INFO:teuthology.orchestra.run.smithi001.stdout: "size": 20, 2024-09-13T11:25:22.502 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 600, 2024-09-13T11:25:22.502 INFO:teuthology.orchestra.run.smithi001.stdout: "ops": [ 2024-09-13T11:25:22.502 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:22.503 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDPGPush(3.11 72/70 [PushOp(3:8da9c977:::benchmark_data_smithi177_31116_object7736:head, version: 46'461, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:8da9c977:::benchmark_data_smithi177_31116_object7736:head@46'461, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:25:22.503 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:26.393216+0000", 2024-09-13T11:25:22.503 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 116.10600034300001, 2024-09-13T11:25:22.503 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.15683955499999999, 2024-09-13T11:25:22.503 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:22.503 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:22.503 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:25:22.503 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:22.503 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:22.503 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:22.503 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:26.393216+0000", 2024-09-13T11:25:22.503 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:22.504 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:22.504 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:22.504 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:22.504 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:26.393216+0000", 2024-09-13T11:25:22.504 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:22.504 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:22.504 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:22.504 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:22.504 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:26.393217+0000", 2024-09-13T11:25:22.504 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2410000000000001e-06 2024-09-13T11:25:22.504 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:22.504 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:22.504 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:22.505 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:26.393309+0000", 2024-09-13T11:25:22.505 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.2101000000000005e-05 2024-09-13T11:25:22.505 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:22.505 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:22.505 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:22.505 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:26.547936+0000", 2024-09-13T11:25:22.505 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.15462682699999999 2024-09-13T11:25:22.505 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:22.505 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:22.505 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:22.505 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:26.548021+0000", 2024-09-13T11:25:22.505 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.4529000000000003e-05 2024-09-13T11:25:22.505 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:22.506 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:22.506 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:22.506 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:26.548321+0000", 2024-09-13T11:25:22.506 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00029988399999999999 2024-09-13T11:25:22.506 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:22.506 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:22.506 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:25:22.506 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:26.548353+0000", 2024-09-13T11:25:22.506 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.2573000000000001e-05 2024-09-13T11:25:22.506 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:22.506 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:22.506 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:22.506 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:26.550056+0000", 2024-09-13T11:25:22.506 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0017024 2024-09-13T11:25:22.507 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:22.507 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:22.507 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:22.507 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:22.507 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:22.507 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDPGPush(3.1e 89/78 [PushOp(3:7a07f38c:::benchmark_data_smithi177_31116_object20966:head, version: 54'1402, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:7a07f38c:::benchmark_data_smithi177_31116_object20966:head@54'1402, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:25:22.507 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:46.645251+0000", 2024-09-13T11:25:22.507 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 95.853965627999997, 2024-09-13T11:25:22.507 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.17670883800000001, 2024-09-13T11:25:22.507 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:22.507 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:22.507 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:25:22.508 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:22.508 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:22.508 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:22.508 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.645251+0000", 2024-09-13T11:25:22.508 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:22.508 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:22.508 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:22.508 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:22.508 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.645251+0000", 2024-09-13T11:25:22.508 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:22.508 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:22.508 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:22.509 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:22.509 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.645253+0000", 2024-09-13T11:25:22.509 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.1399999999999998e-06 2024-09-13T11:25:22.509 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:22.509 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:22.509 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:22.509 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.645295+0000", 2024-09-13T11:25:22.509 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.2323000000000001e-05 2024-09-13T11:25:22.509 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:22.509 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:22.509 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:22.509 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.820731+0000", 2024-09-13T11:25:22.509 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.17543607999999999 2024-09-13T11:25:22.510 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:22.510 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:22.510 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:22.510 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.820750+0000", 2024-09-13T11:25:22.510 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.8793000000000001e-05 2024-09-13T11:25:22.510 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:22.510 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:22.510 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:22.510 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.820914+0000", 2024-09-13T11:25:22.510 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000164062 2024-09-13T11:25:22.510 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:22.511 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:22.511 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:25:22.511 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.820939+0000", 2024-09-13T11:25:22.511 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.4672e-05 2024-09-13T11:25:22.511 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:22.511 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:22.511 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:22.511 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.821960+0000", 2024-09-13T11:25:22.511 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.001020768 2024-09-13T11:25:22.511 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:22.511 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:22.511 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:22.511 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:22.512 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:22.512 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:71961 3.11 3:89bd9466:::benchmark_data_smithi177_31116_object71960:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2024-09-13T11:25:22.512 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:54.267475+0000", 2024-09-13T11:25:22.512 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 88.231741428999996, 2024-09-13T11:25:22.512 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.15027747899999999, 2024-09-13T11:25:22.512 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:22.512 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:22.512 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:22.512 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:25:22.512 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:25:22.512 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:22.512 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 71961 2024-09-13T11:25:22.512 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:22.513 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:22.513 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:22.513 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:22.513 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:54.267475+0000", 2024-09-13T11:25:22.513 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:22.513 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:22.513 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:22.513 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:22.513 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:54.267475+0000", 2024-09-13T11:25:22.513 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:22.513 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:22.513 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:22.513 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:22.514 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:54.267478+0000", 2024-09-13T11:25:22.514 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.8080000000000001e-06 2024-09-13T11:25:22.514 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:22.514 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:22.514 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:22.514 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:54.267534+0000", 2024-09-13T11:25:22.514 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.6276999999999997e-05 2024-09-13T11:25:22.514 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:22.514 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:22.514 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:22.514 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:54.267557+0000", 2024-09-13T11:25:22.514 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.2514000000000002e-05 2024-09-13T11:25:22.514 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:22.515 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:22.515 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:22.515 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:54.267570+0000", 2024-09-13T11:25:22.515 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.3251999999999999e-05 2024-09-13T11:25:22.515 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:22.515 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:22.515 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:22.515 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:54.267650+0000", 2024-09-13T11:25:22.515 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.0066000000000002e-05 2024-09-13T11:25:22.515 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:22.515 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:22.515 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:25:22.515 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:54.267886+0000", 2024-09-13T11:25:22.516 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00023637800000000001 2024-09-13T11:25:22.516 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:22.516 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:22.516 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 11", 2024-09-13T11:25:22.516 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:54.268020+0000", 2024-09-13T11:25:22.516 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00013392100000000001 2024-09-13T11:25:22.516 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:22.516 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:22.516 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:25:22.516 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:54.270076+0000", 2024-09-13T11:25:22.516 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.002055393 2024-09-13T11:25:22.516 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:22.517 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:22.517 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:25:22.517 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:54.417584+0000", 2024-09-13T11:25:22.517 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.147508008 2024-09-13T11:25:22.517 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:22.517 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:22.517 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:25:22.517 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:54.417667+0000", 2024-09-13T11:25:22.517 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.3666999999999997e-05 2024-09-13T11:25:22.517 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:22.517 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:22.517 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:22.517 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:54.417752+0000", 2024-09-13T11:25:22.518 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.5195e-05 2024-09-13T11:25:22.518 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:22.518 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:22.518 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:22.518 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:22.518 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:22.518 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4665.0:79776 3.1a e102/78 3:5d637d57:::benchmark_data_smithi177_31116_object79775:head v 102'4120, mlcod=72'3145)", 2024-09-13T11:25:22.518 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:24:00.354950+0000", 2024-09-13T11:25:22.518 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 82.144266084999998, 2024-09-13T11:25:22.518 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.14753330100000001, 2024-09-13T11:25:22.518 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:22.519 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:22.519 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:22.519 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:22.519 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:22.519 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:22.519 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:00.354950+0000", 2024-09-13T11:25:22.519 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:22.519 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:22.519 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:22.519 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:22.519 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:00.354950+0000", 2024-09-13T11:25:22.519 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:22.519 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:22.519 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:22.520 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:22.520 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:00.354960+0000", 2024-09-13T11:25:22.520 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.5219999999999996e-06 2024-09-13T11:25:22.520 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:22.520 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:22.520 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:22.520 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:00.355024+0000", 2024-09-13T11:25:22.520 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.4163000000000002e-05 2024-09-13T11:25:22.520 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:22.520 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:22.520 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:22.520 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:00.499446+0000", 2024-09-13T11:25:22.521 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.144422352 2024-09-13T11:25:22.521 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:22.521 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:22.521 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:22.521 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:00.499466+0000", 2024-09-13T11:25:22.521 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.9694000000000002e-05 2024-09-13T11:25:22.521 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:22.521 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:22.521 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:22.521 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:00.499605+0000", 2024-09-13T11:25:22.521 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00013935099999999999 2024-09-13T11:25:22.521 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:22.521 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:22.522 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:25:22.522 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:00.499655+0000", 2024-09-13T11:25:22.522 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.9787000000000003e-05 2024-09-13T11:25:22.522 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:22.522 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:22.522 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:25:22.522 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:00.502418+0000", 2024-09-13T11:25:22.522 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.002763012 2024-09-13T11:25:22.522 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:22.522 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:22.522 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:22.522 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:00.502484+0000", 2024-09-13T11:25:22.522 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.5419999999999994e-05 2024-09-13T11:25:22.523 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:22.523 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:22.523 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:22.523 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:22.523 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:22.523 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDPGPush(3.1a 102/78 [PushOp(3:5df50766:::benchmark_data_smithi177_31116_object50543:head, version: 72'3152, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:5df50766:::benchmark_data_smithi177_31116_object50543:head@72'3152, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:25:22.523 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:24:00.499311+0000", 2024-09-13T11:25:22.523 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 81.999905542999997, 2024-09-13T11:25:22.523 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.26146664800000002, 2024-09-13T11:25:22.523 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:22.523 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:22.523 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:25:22.524 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:22.524 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:22.524 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:22.524 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:00.499311+0000", 2024-09-13T11:25:22.524 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:22.524 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:22.524 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:22.524 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:22.524 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:00.499311+0000", 2024-09-13T11:25:22.524 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:22.524 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:22.524 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:22.525 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:22.525 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:00.499320+0000", 2024-09-13T11:25:22.525 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.6400000000000003e-06 2024-09-13T11:25:22.525 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:22.525 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:22.525 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:22.525 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:00.499384+0000", 2024-09-13T11:25:22.525 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.4554999999999994e-05 2024-09-13T11:25:22.525 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:22.525 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:22.525 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:22.525 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:00.759713+0000", 2024-09-13T11:25:22.525 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.260328742 2024-09-13T11:25:22.526 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:22.526 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:22.526 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:22.526 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:00.759742+0000", 2024-09-13T11:25:22.526 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.9298e-05 2024-09-13T11:25:22.526 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:22.526 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:22.526 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:22.526 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:00.759871+0000", 2024-09-13T11:25:22.526 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000128992 2024-09-13T11:25:22.526 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:22.526 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:22.527 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:25:22.527 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:00.759884+0000", 2024-09-13T11:25:22.527 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2687999999999999e-05 2024-09-13T11:25:22.527 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:22.527 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:22.527 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:22.527 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:00.760778+0000", 2024-09-13T11:25:22.527 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00089373299999999996 2024-09-13T11:25:22.527 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:22.527 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:22.527 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:22.527 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:22.527 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:22.528 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4665.0:82769 3.1e e105/102 3:7a4e88b5:::benchmark_data_smithi177_31116_object82768:head v 105'4381, mlcod=105'4379)", 2024-09-13T11:25:22.528 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:24:03.745047+0000", 2024-09-13T11:25:22.528 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 78.754169317999995, 2024-09-13T11:25:22.528 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.213806145, 2024-09-13T11:25:22.528 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:22.528 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:22.528 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:22.528 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:22.528 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:22.528 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:22.528 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:03.745047+0000", 2024-09-13T11:25:22.528 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:22.528 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:22.528 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:22.529 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:22.529 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:03.745047+0000", 2024-09-13T11:25:22.529 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:22.529 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:22.529 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:22.529 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:22.529 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:03.745049+0000", 2024-09-13T11:25:22.529 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.5710000000000001e-06 2024-09-13T11:25:22.529 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:22.529 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:22.529 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:22.529 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:03.745107+0000", 2024-09-13T11:25:22.529 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.8799999999999999e-05 2024-09-13T11:25:22.530 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:22.530 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:22.530 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:22.530 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:03.955145+0000", 2024-09-13T11:25:22.530 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.210037735 2024-09-13T11:25:22.530 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:22.530 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:22.530 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:22.530 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:03.955163+0000", 2024-09-13T11:25:22.530 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.7520000000000002e-05 2024-09-13T11:25:22.530 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:22.530 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:22.530 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:22.530 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:03.955473+0000", 2024-09-13T11:25:22.531 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00031014399999999999 2024-09-13T11:25:22.531 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:22.531 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:22.531 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:25:22.531 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:03.955538+0000", 2024-09-13T11:25:22.531 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.5059000000000002e-05 2024-09-13T11:25:22.531 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:22.531 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:22.532 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:25:22.532 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:03.958753+0000", 2024-09-13T11:25:22.532 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0032152769999999999 2024-09-13T11:25:22.532 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:22.532 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:22.532 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:22.532 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:03.958853+0000", 2024-09-13T11:25:22.532 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000100039 2024-09-13T11:25:22.532 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:22.532 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:22.532 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:22.532 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:22.533 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:22.533 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:98582 3.1a 3:59f9cd75:::benchmark_data_smithi177_31116_object98581:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113)", 2024-09-13T11:25:22.533 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:24:15.431226+0000", 2024-09-13T11:25:22.533 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 67.067990848999997, 2024-09-13T11:25:22.533 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.19820755800000001, 2024-09-13T11:25:22.533 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:22.533 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:22.533 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:22.533 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:25:22.533 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:25:22.533 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:22.533 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 98582 2024-09-13T11:25:22.533 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:22.534 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:22.534 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:22.534 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:22.534 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.431226+0000", 2024-09-13T11:25:22.534 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:22.534 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:22.534 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:22.534 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:22.534 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.431226+0000", 2024-09-13T11:25:22.534 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:22.534 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:22.534 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:22.534 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:22.535 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.431229+0000", 2024-09-13T11:25:22.535 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.7100000000000001e-06 2024-09-13T11:25:22.535 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:22.535 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:22.535 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:22.535 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.431334+0000", 2024-09-13T11:25:22.535 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00010506900000000001 2024-09-13T11:25:22.535 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:22.535 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:22.535 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:22.535 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.431364+0000", 2024-09-13T11:25:22.535 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.9607e-05 2024-09-13T11:25:22.536 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:22.536 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:22.536 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:22.536 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.431388+0000", 2024-09-13T11:25:22.536 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.3808999999999999e-05 2024-09-13T11:25:22.536 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:22.536 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:22.536 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:22.536 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.431483+0000", 2024-09-13T11:25:22.536 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.5402999999999994e-05 2024-09-13T11:25:22.536 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:22.536 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:22.536 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:25:22.537 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.431642+0000", 2024-09-13T11:25:22.537 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000158412 2024-09-13T11:25:22.537 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:22.537 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:22.537 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 9", 2024-09-13T11:25:22.537 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.431796+0000", 2024-09-13T11:25:22.537 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00015467899999999999 2024-09-13T11:25:22.537 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:22.537 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:22.537 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:25:22.537 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.433731+0000", 2024-09-13T11:25:22.537 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.001934945 2024-09-13T11:25:22.537 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:22.537 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:22.538 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:25:22.538 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.629245+0000", 2024-09-13T11:25:22.538 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.19551390199999999 2024-09-13T11:25:22.538 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:22.538 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:22.538 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:25:22.538 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.629346+0000", 2024-09-13T11:25:22.538 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000100869 2024-09-13T11:25:22.538 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:22.538 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:22.538 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:22.538 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.629433+0000", 2024-09-13T11:25:22.538 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.7152999999999997e-05 2024-09-13T11:25:22.539 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:22.539 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:22.539 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:22.539 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:22.539 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:22.539 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:99105 3.1a 3:5ac5090c:::benchmark_data_smithi177_31116_object99104:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113)", 2024-09-13T11:25:22.539 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:24:15.962466+0000", 2024-09-13T11:25:22.539 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 66.536750560000002, 2024-09-13T11:25:22.539 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.23832849, 2024-09-13T11:25:22.539 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:22.539 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:22.539 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:22.539 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:25:22.539 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:25:22.540 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:22.540 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 99105 2024-09-13T11:25:22.540 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:22.540 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:22.540 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:22.540 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:22.540 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.962466+0000", 2024-09-13T11:25:22.540 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:22.540 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:22.540 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:22.540 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:22.540 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.962466+0000", 2024-09-13T11:25:22.540 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:22.540 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:22.541 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:22.541 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:22.541 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.962471+0000", 2024-09-13T11:25:22.541 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.9570000000000001e-06 2024-09-13T11:25:22.541 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:22.541 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:22.541 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:22.541 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.962547+0000", 2024-09-13T11:25:22.541 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.5947999999999999e-05 2024-09-13T11:25:22.541 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:22.541 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:22.541 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:22.541 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.962576+0000", 2024-09-13T11:25:22.541 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.9547999999999999e-05 2024-09-13T11:25:22.542 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:22.542 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:22.542 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:22.542 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.962594+0000", 2024-09-13T11:25:22.542 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.8023e-05 2024-09-13T11:25:22.542 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:22.542 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:22.542 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:22.542 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.962770+0000", 2024-09-13T11:25:22.542 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00017551099999999999 2024-09-13T11:25:22.542 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:22.542 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:22.542 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:25:22.542 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.963114+0000", 2024-09-13T11:25:22.543 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00034440399999999998 2024-09-13T11:25:22.543 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:22.543 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:22.543 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 9", 2024-09-13T11:25:22.543 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.963351+0000", 2024-09-13T11:25:22.543 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000237209 2024-09-13T11:25:22.543 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:22.543 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:22.543 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:25:22.543 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.966345+0000", 2024-09-13T11:25:22.543 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0029933519999999999 2024-09-13T11:25:22.543 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:22.543 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:22.543 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:25:22.544 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:16.200724+0000", 2024-09-13T11:25:22.544 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.23437934599999999 2024-09-13T11:25:22.544 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:22.544 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:22.544 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:25:22.544 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:16.200761+0000", 2024-09-13T11:25:22.544 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.6406999999999999e-05 2024-09-13T11:25:22.544 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:22.544 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:22.544 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:22.544 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:16.200794+0000", 2024-09-13T11:25:22.544 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.3785000000000003e-05 2024-09-13T11:25:22.544 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:22.545 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:22.545 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:22.545 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:22.545 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:22.545 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4665.0:104331 3.1e e115/102 3:79890035:::benchmark_data_smithi177_31116_object104330:head v 115'5083, mlcod=115'5080)", 2024-09-13T11:25:22.545 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:24:20.242656+0000", 2024-09-13T11:25:22.545 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 62.256560468000004, 2024-09-13T11:25:22.545 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.16813846700000001, 2024-09-13T11:25:22.545 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:22.545 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:22.545 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:22.545 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:22.545 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:22.545 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:22.546 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:20.242656+0000", 2024-09-13T11:25:22.546 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:22.546 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:22.546 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:22.546 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:22.546 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:20.242656+0000", 2024-09-13T11:25:22.546 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:22.546 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:22.546 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:22.546 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:22.546 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:20.242657+0000", 2024-09-13T11:25:22.546 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.4750000000000001e-06 2024-09-13T11:25:22.546 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:22.546 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:22.547 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:22.547 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:20.242699+0000", 2024-09-13T11:25:22.547 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.193e-05 2024-09-13T11:25:22.547 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:22.547 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:22.547 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:22.547 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:20.408772+0000", 2024-09-13T11:25:22.547 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.16607291199999999 2024-09-13T11:25:22.547 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:22.547 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:22.547 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:22.547 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:20.408779+0000", 2024-09-13T11:25:22.547 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.3600000000000001e-06 2024-09-13T11:25:22.547 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:22.548 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:22.548 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:22.548 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:20.408840+0000", 2024-09-13T11:25:22.548 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.1834999999999993e-05 2024-09-13T11:25:22.548 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:22.548 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:22.548 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:25:22.548 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:20.408865+0000", 2024-09-13T11:25:22.548 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.4294000000000002e-05 2024-09-13T11:25:22.548 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:22.548 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:22.548 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:25:22.548 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:20.410751+0000", 2024-09-13T11:25:22.549 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.001885811 2024-09-13T11:25:22.549 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:22.549 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:22.549 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:22.549 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:20.410794+0000", 2024-09-13T11:25:22.549 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.3850000000000002e-05 2024-09-13T11:25:22.549 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:22.549 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:22.549 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:22.549 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:22.549 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:22.549 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:129921 3.1a 3:5fb9088b:::benchmark_data_smithi177_31116_object129920:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e134)", 2024-09-13T11:25:22.549 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:24:47.177556+0000", 2024-09-13T11:25:22.549 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 35.321660281, 2024-09-13T11:25:22.550 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.33804469999999998, 2024-09-13T11:25:22.550 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:22.550 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:22.550 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:22.550 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:25:22.550 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:25:22.550 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:22.550 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 129921 2024-09-13T11:25:22.550 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:22.550 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:22.550 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:22.550 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:22.550 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:47.177556+0000", 2024-09-13T11:25:22.551 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:22.551 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:22.551 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:22.551 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:22.551 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:47.177556+0000", 2024-09-13T11:25:22.551 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:22.551 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:22.551 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:22.551 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:22.551 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:47.177559+0000", 2024-09-13T11:25:22.551 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.0790000000000001e-06 2024-09-13T11:25:22.551 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:22.551 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:22.552 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:22.552 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:47.177589+0000", 2024-09-13T11:25:22.552 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.0173999999999999e-05 2024-09-13T11:25:22.552 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:22.552 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:22.552 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:22.552 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:47.177606+0000", 2024-09-13T11:25:22.552 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.6475999999999999e-05 2024-09-13T11:25:22.552 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:22.552 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:22.552 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:22.552 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:47.177615+0000", 2024-09-13T11:25:22.552 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.5729999999999997e-06 2024-09-13T11:25:22.553 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:22.553 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:22.553 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:22.553 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:47.177680+0000", 2024-09-13T11:25:22.553 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.4349000000000003e-05 2024-09-13T11:25:22.553 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:22.553 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:22.553 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:25:22.553 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:47.177794+0000", 2024-09-13T11:25:22.553 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000114119 2024-09-13T11:25:22.553 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:22.553 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:22.553 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 9", 2024-09-13T11:25:22.553 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:47.177895+0000", 2024-09-13T11:25:22.554 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000101078 2024-09-13T11:25:22.554 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:22.554 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:22.554 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:25:22.554 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:47.179634+0000", 2024-09-13T11:25:22.554 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.001738606 2024-09-13T11:25:22.554 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:22.554 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:22.554 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:25:22.554 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:47.515495+0000", 2024-09-13T11:25:22.554 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.33586143000000002 2024-09-13T11:25:22.554 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:22.554 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:22.554 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:25:22.555 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:47.515551+0000", 2024-09-13T11:25:22.555 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.6048999999999998e-05 2024-09-13T11:25:22.555 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:22.555 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:22.555 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:22.555 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:47.515601+0000", 2024-09-13T11:25:22.555 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.9767e-05 2024-09-13T11:25:22.555 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:22.555 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:22.555 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:22.555 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:22.555 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:22.555 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4665.0:140742 3.1e e144/102 3:79c694a0:::benchmark_data_smithi177_31116_object439:head v 144'6204, mlcod=144'6200)", 2024-09-13T11:25:22.555 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:24:58.321534+0000", 2024-09-13T11:25:22.556 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 24.177682756999999, 2024-09-13T11:25:22.556 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.23629921500000001, 2024-09-13T11:25:22.556 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:22.556 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:22.556 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:22.556 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:22.556 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:22.556 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:22.556 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.321534+0000", 2024-09-13T11:25:22.556 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:22.556 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:22.556 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:22.556 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:22.557 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.321534+0000", 2024-09-13T11:25:22.557 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:22.557 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:22.557 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:22.557 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:22.557 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.321535+0000", 2024-09-13T11:25:22.557 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.159e-06 2024-09-13T11:25:22.557 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:22.557 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:22.557 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:22.557 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.321545+0000", 2024-09-13T11:25:22.557 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.9219999999999992e-06 2024-09-13T11:25:22.557 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:22.557 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:22.558 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:22.558 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.553719+0000", 2024-09-13T11:25:22.558 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.23217401700000001 2024-09-13T11:25:22.558 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:22.558 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:22.558 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:22.558 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.553734+0000", 2024-09-13T11:25:22.558 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.5344000000000001e-05 2024-09-13T11:25:22.558 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:22.558 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:22.558 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:22.558 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.553858+0000", 2024-09-13T11:25:22.558 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00012357500000000001 2024-09-13T11:25:22.558 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:22.559 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:22.559 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:25:22.559 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.553903+0000", 2024-09-13T11:25:22.559 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.5374999999999998e-05 2024-09-13T11:25:22.559 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:22.559 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:22.559 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:25:22.559 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.557719+0000", 2024-09-13T11:25:22.559 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0038158670000000001 2024-09-13T11:25:22.559 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:22.559 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:22.559 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:22.559 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.557833+0000", 2024-09-13T11:25:22.559 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000113956 2024-09-13T11:25:22.560 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:22.560 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:22.560 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:22.560 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:22.560 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:22.560 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4665.0:140796 3.1e e144/102 3:7987a652:::benchmark_data_smithi177_31116_object493:head v 144'6205, mlcod=144'6202)", 2024-09-13T11:25:22.560 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:24:58.553657+0000", 2024-09-13T11:25:22.560 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 23.945559781, 2024-09-13T11:25:22.560 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.25308323100000002, 2024-09-13T11:25:22.560 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:22.560 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:22.560 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:22.560 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:22.560 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:22.561 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:22.561 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.553657+0000", 2024-09-13T11:25:22.561 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:22.561 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:22.561 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:22.561 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:22.561 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.553657+0000", 2024-09-13T11:25:22.561 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:22.561 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:22.561 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:22.561 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:22.561 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.553660+0000", 2024-09-13T11:25:22.561 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.7519999999999998e-06 2024-09-13T11:25:22.561 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:22.562 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:22.562 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:22.562 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.553678+0000", 2024-09-13T11:25:22.562 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.7156e-05 2024-09-13T11:25:22.562 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:22.562 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:22.562 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:22.562 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.804342+0000", 2024-09-13T11:25:22.562 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.25066436199999997 2024-09-13T11:25:22.562 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:22.562 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:22.562 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:22.562 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.804353+0000", 2024-09-13T11:25:22.562 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.1147e-05 2024-09-13T11:25:22.563 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:22.563 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:22.563 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:22.563 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.804433+0000", 2024-09-13T11:25:22.563 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.9586999999999995e-05 2024-09-13T11:25:22.563 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:22.563 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:22.563 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:25:22.563 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.804457+0000", 2024-09-13T11:25:22.563 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.4108e-05 2024-09-13T11:25:22.563 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:22.563 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:22.563 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:25:22.564 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.806684+0000", 2024-09-13T11:25:22.564 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0022271309999999998 2024-09-13T11:25:22.564 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:22.564 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:22.564 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:22.564 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.806740+0000", 2024-09-13T11:25:22.564 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.5988e-05 2024-09-13T11:25:22.564 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:22.564 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:22.564 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:22.564 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:22.564 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:22.564 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:149955 3.11 3:8fa29838:::benchmark_data_smithi177_31116_object9652:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e149)", 2024-09-13T11:25:22.564 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:25:05.905492+0000", 2024-09-13T11:25:22.565 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 16.593724407, 2024-09-13T11:25:22.565 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.204181955, 2024-09-13T11:25:22.565 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:22.565 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:22.565 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:22.565 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:25:22.565 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:25:22.565 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:22.565 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 149955 2024-09-13T11:25:22.565 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:22.565 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:22.565 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:22.565 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:22.566 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:05.905492+0000", 2024-09-13T11:25:22.566 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:22.566 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:22.566 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:22.566 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:22.566 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:05.905492+0000", 2024-09-13T11:25:22.566 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:22.566 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:22.566 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:22.566 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:22.566 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:05.905494+0000", 2024-09-13T11:25:22.566 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.9369999999999998e-06 2024-09-13T11:25:22.566 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:22.567 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:22.567 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:22.567 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:05.905500+0000", 2024-09-13T11:25:22.567 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.7039999999999999e-06 2024-09-13T11:25:22.567 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:22.567 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:22.567 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:22.567 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:05.905512+0000", 2024-09-13T11:25:22.567 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.257e-05 2024-09-13T11:25:22.567 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:22.567 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:22.567 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:22.567 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:05.905519+0000", 2024-09-13T11:25:22.568 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.5420000000000002e-06 2024-09-13T11:25:22.568 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:22.568 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:22.568 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:22.568 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:05.905584+0000", 2024-09-13T11:25:22.568 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.4819000000000004e-05 2024-09-13T11:25:22.568 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:22.568 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:22.568 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:25:22.568 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:05.905696+0000", 2024-09-13T11:25:22.568 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00011248599999999999 2024-09-13T11:25:22.568 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:22.568 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:22.568 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 11", 2024-09-13T11:25:22.569 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:05.905759+0000", 2024-09-13T11:25:22.569 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.2451000000000002e-05 2024-09-13T11:25:22.569 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:22.569 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:22.569 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:25:22.569 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:05.907584+0000", 2024-09-13T11:25:22.569 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0018250110000000001 2024-09-13T11:25:22.569 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:22.569 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:22.569 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:25:22.569 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:06.109588+0000", 2024-09-13T11:25:22.569 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.20200470300000001 2024-09-13T11:25:22.569 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:22.569 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:22.570 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:25:22.570 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:06.109637+0000", 2024-09-13T11:25:22.570 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.8439000000000002e-05 2024-09-13T11:25:22.570 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:22.570 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:22.570 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:22.570 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:06.109674+0000", 2024-09-13T11:25:22.570 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.7292999999999997e-05 2024-09-13T11:25:22.570 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:22.570 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:22.570 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:22.570 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:22.570 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:22.570 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:150379 3.11 3:8a0baa7f:::benchmark_data_smithi177_31116_object10076:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e149)", 2024-09-13T11:25:22.570 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:25:06.299792+0000", 2024-09-13T11:25:22.571 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 16.199424523000001, 2024-09-13T11:25:22.571 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.13920049000000001, 2024-09-13T11:25:22.571 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:22.571 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:22.571 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:22.571 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:25:22.571 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:25:22.571 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:22.571 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 150379 2024-09-13T11:25:22.571 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:22.571 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:22.571 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:22.571 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:22.571 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:06.299792+0000", 2024-09-13T11:25:22.572 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:22.572 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:22.572 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:22.572 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:22.572 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:06.299792+0000", 2024-09-13T11:25:22.572 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:22.572 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:22.572 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:22.572 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:22.572 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:06.299794+0000", 2024-09-13T11:25:22.572 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.7549999999999999e-06 2024-09-13T11:25:22.572 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:22.572 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:22.573 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:22.573 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:06.299798+0000", 2024-09-13T11:25:22.573 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.809e-06 2024-09-13T11:25:22.573 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:22.573 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:22.573 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:22.573 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:06.299809+0000", 2024-09-13T11:25:22.573 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0774999999999999e-05 2024-09-13T11:25:22.573 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:22.573 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:22.573 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:22.573 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:06.299815+0000", 2024-09-13T11:25:22.573 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.6559999999999997e-06 2024-09-13T11:25:22.573 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:22.574 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:22.574 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:22.574 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:06.299899+0000", 2024-09-13T11:25:22.574 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.3876999999999997e-05 2024-09-13T11:25:22.574 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:22.574 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:22.574 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:25:22.574 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:06.300042+0000", 2024-09-13T11:25:22.574 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00014310500000000001 2024-09-13T11:25:22.574 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:22.574 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:22.574 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 11", 2024-09-13T11:25:22.574 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:06.300164+0000", 2024-09-13T11:25:22.575 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000121951 2024-09-13T11:25:22.575 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:22.575 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:22.575 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:25:22.575 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:06.302456+0000", 2024-09-13T11:25:22.575 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0022923380000000001 2024-09-13T11:25:22.575 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:22.575 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:22.575 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:25:22.575 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:06.438889+0000", 2024-09-13T11:25:22.575 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.13643327799999999 2024-09-13T11:25:22.575 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:22.575 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:22.575 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:25:22.576 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:06.438946+0000", 2024-09-13T11:25:22.576 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.6718999999999997e-05 2024-09-13T11:25:22.576 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:22.576 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:22.576 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:22.576 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:06.438992+0000", 2024-09-13T11:25:22.576 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.6227000000000003e-05 2024-09-13T11:25:22.576 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:22.576 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:22.576 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:22.576 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:22.576 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:22.576 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:150414 3.11 3:8f72eed4:::benchmark_data_smithi177_31116_object10111:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e149)", 2024-09-13T11:25:22.576 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:25:06.348925+0000", 2024-09-13T11:25:22.577 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 16.150291294999999, 2024-09-13T11:25:22.577 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.17250813300000001, 2024-09-13T11:25:22.577 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:22.577 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:22.577 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:22.577 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:25:22.577 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:25:22.577 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:22.577 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 150414 2024-09-13T11:25:22.577 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:22.577 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:22.577 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:22.577 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:22.578 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:06.348925+0000", 2024-09-13T11:25:22.578 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:22.578 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:22.578 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:22.578 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:22.578 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:06.348925+0000", 2024-09-13T11:25:22.578 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:22.578 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:22.578 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:22.578 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:22.578 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:06.348927+0000", 2024-09-13T11:25:22.578 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.7430000000000001e-06 2024-09-13T11:25:22.578 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:22.579 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:22.579 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:22.579 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:06.348931+0000", 2024-09-13T11:25:22.579 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.5210000000000004e-06 2024-09-13T11:25:22.579 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:22.579 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:22.579 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:22.579 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:06.348943+0000", 2024-09-13T11:25:22.579 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.1168e-05 2024-09-13T11:25:22.579 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:22.579 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:22.579 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:22.579 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:06.348950+0000", 2024-09-13T11:25:22.580 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.1430000000000001e-06 2024-09-13T11:25:22.580 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:22.580 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:22.580 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:22.580 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:06.349032+0000", 2024-09-13T11:25:22.580 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.2517999999999998e-05 2024-09-13T11:25:22.580 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:22.580 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:22.580 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:25:22.580 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:06.349179+0000", 2024-09-13T11:25:22.580 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000146935 2024-09-13T11:25:22.580 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:22.580 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:22.580 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 11", 2024-09-13T11:25:22.581 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:06.349290+0000", 2024-09-13T11:25:22.581 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000110927 2024-09-13T11:25:22.581 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:22.581 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:22.581 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:25:22.581 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:06.351686+0000", 2024-09-13T11:25:22.581 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0023963560000000001 2024-09-13T11:25:22.581 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:22.581 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:22.581 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:25:22.581 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:06.521374+0000", 2024-09-13T11:25:22.581 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.169687216 2024-09-13T11:25:22.581 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:22.581 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:22.582 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:25:22.582 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:06.521406+0000", 2024-09-13T11:25:22.582 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.1841000000000002e-05 2024-09-13T11:25:22.582 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:22.582 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:22.582 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:22.582 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:06.521433+0000", 2024-09-13T11:25:22.582 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.7764999999999999e-05 2024-09-13T11:25:22.582 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:22.582 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:22.582 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:22.582 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:22.582 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:22.582 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDPGPush(3.29 151/120 [PushOp(3:97da7246:::benchmark_data_smithi177_31116_object96568:head, version: 113'4425, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:97da7246:::benchmark_data_smithi177_31116_object96568:head@113'4425, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:25:22.583 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:25:09.375401+0000", 2024-09-13T11:25:22.583 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 13.123815016, 2024-09-13T11:25:22.583 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.15027211300000001, 2024-09-13T11:25:22.583 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:22.583 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:22.583 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:25:22.583 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:22.583 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:22.583 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:22.583 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:09.375401+0000", 2024-09-13T11:25:22.583 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:22.583 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:22.583 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:22.583 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:22.584 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:09.375401+0000", 2024-09-13T11:25:22.584 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:22.584 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:22.584 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:22.584 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:22.584 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:09.375402+0000", 2024-09-13T11:25:22.584 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.049e-06 2024-09-13T11:25:22.584 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:22.584 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:22.584 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:22.584 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:09.375505+0000", 2024-09-13T11:25:22.584 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00010225000000000001 2024-09-13T11:25:22.584 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:22.585 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:22.585 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:22.585 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:09.524742+0000", 2024-09-13T11:25:22.585 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.14923771899999999 2024-09-13T11:25:22.585 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:22.585 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:22.585 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:22.585 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:09.524761+0000", 2024-09-13T11:25:22.585 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.8377999999999999e-05 2024-09-13T11:25:22.585 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:22.585 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:22.585 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:22.585 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:09.524842+0000", 2024-09-13T11:25:22.586 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.0841999999999996e-05 2024-09-13T11:25:22.586 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:22.586 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:22.586 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:25:22.586 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:09.524850+0000", 2024-09-13T11:25:22.586 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.8709999999999997e-06 2024-09-13T11:25:22.586 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:22.586 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:22.586 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:22.586 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:09.525674+0000", 2024-09-13T11:25:22.586 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00082400400000000003 2024-09-13T11:25:22.586 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:22.586 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:22.587 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:22.587 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:22.587 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:22.587 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:163098 3.1a 3:58bec031:::benchmark_data_smithi177_31116_object22795:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e157)", 2024-09-13T11:25:22.587 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:25:16.139554+0000", 2024-09-13T11:25:22.587 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 6.3596624469999998, 2024-09-13T11:25:22.587 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.14248129000000001, 2024-09-13T11:25:22.587 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:22.587 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:22.587 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:22.587 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:25:22.587 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:25:22.587 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:22.587 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 163098 2024-09-13T11:25:22.588 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:22.588 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:22.588 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:22.588 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:22.588 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:16.139554+0000", 2024-09-13T11:25:22.588 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:22.588 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:22.588 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:22.588 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:22.588 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:16.139554+0000", 2024-09-13T11:25:22.588 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:22.588 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:22.588 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:22.589 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:22.589 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:16.139557+0000", 2024-09-13T11:25:22.589 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.4249999999999998e-06 2024-09-13T11:25:22.589 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:22.589 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:22.589 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:22.589 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:16.139566+0000", 2024-09-13T11:25:22.589 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.8659999999999995e-06 2024-09-13T11:25:22.589 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:22.589 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:22.589 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:22.589 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:16.139587+0000", 2024-09-13T11:25:22.589 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.1058999999999999e-05 2024-09-13T11:25:22.590 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:22.590 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:22.590 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:22.590 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:16.139597+0000", 2024-09-13T11:25:22.590 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.004e-05 2024-09-13T11:25:22.590 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:22.590 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:22.590 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:22.590 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:16.139668+0000", 2024-09-13T11:25:22.590 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.0257000000000002e-05 2024-09-13T11:25:22.590 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:22.590 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:22.590 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:25:22.590 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:16.139831+0000", 2024-09-13T11:25:22.591 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00016312999999999999 2024-09-13T11:25:22.591 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:22.591 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:22.591 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 9", 2024-09-13T11:25:22.591 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:16.139973+0000", 2024-09-13T11:25:22.591 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00014215199999999999 2024-09-13T11:25:22.591 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:22.591 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:22.591 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:25:22.591 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:16.142075+0000", 2024-09-13T11:25:22.591 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0021017760000000001 2024-09-13T11:25:22.591 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:22.592 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:22.592 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:25:22.592 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:16.281933+0000", 2024-09-13T11:25:22.592 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.13985831900000001 2024-09-13T11:25:22.592 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:22.592 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:22.592 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:25:22.592 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:16.281987+0000", 2024-09-13T11:25:22.592 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.3896e-05 2024-09-13T11:25:22.592 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:22.592 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:22.592 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:22.592 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:16.282035+0000", 2024-09-13T11:25:22.592 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.8369999999999999e-05 2024-09-13T11:25:22.593 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:22.593 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:22.593 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:22.593 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:22.593 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:22.593 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:168772 3.1a 3:5be647e9:::benchmark_data_smithi177_31116_object28469:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e161)", 2024-09-13T11:25:22.593 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:25:20.636095+0000", 2024-09-13T11:25:22.593 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 1.8631218860000001, 2024-09-13T11:25:22.593 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.17968553000000001, 2024-09-13T11:25:22.593 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:22.593 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:22.593 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:22.593 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:25:22.594 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:25:22.594 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:22.594 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 168772 2024-09-13T11:25:22.594 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:22.594 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:22.594 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:22.594 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:22.594 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:20.636095+0000", 2024-09-13T11:25:22.594 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:22.594 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:22.594 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:22.594 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:22.594 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:20.636095+0000", 2024-09-13T11:25:22.594 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:22.595 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:22.595 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:22.595 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:22.595 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:20.636096+0000", 2024-09-13T11:25:22.595 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.852e-06 2024-09-13T11:25:22.595 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:22.595 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:22.595 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:22.595 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:20.636101+0000", 2024-09-13T11:25:22.595 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.0329999999999999e-06 2024-09-13T11:25:22.595 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:22.595 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:22.595 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:22.595 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:20.636114+0000", 2024-09-13T11:25:22.596 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2816999999999999e-05 2024-09-13T11:25:22.596 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:22.596 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:22.596 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:22.596 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:20.636120+0000", 2024-09-13T11:25:22.596 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.7429999999999999e-06 2024-09-13T11:25:22.596 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:22.596 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:22.596 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:22.596 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:20.636194+0000", 2024-09-13T11:25:22.596 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.3520000000000001e-05 2024-09-13T11:25:22.596 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:22.596 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:22.597 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:25:22.597 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:20.636369+0000", 2024-09-13T11:25:22.597 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00017552 2024-09-13T11:25:22.597 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:22.597 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:22.597 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 9", 2024-09-13T11:25:22.597 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:20.636519+0000", 2024-09-13T11:25:22.597 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000149949 2024-09-13T11:25:22.597 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:22.597 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:22.597 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:25:22.597 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:20.638735+0000", 2024-09-13T11:25:22.597 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0022156509999999999 2024-09-13T11:25:22.598 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:22.598 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:22.598 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:25:22.598 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:20.815699+0000", 2024-09-13T11:25:22.598 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.176964343 2024-09-13T11:25:22.598 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:22.598 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:22.598 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:25:22.598 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:20.815748+0000", 2024-09-13T11:25:22.598 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.9518000000000003e-05 2024-09-13T11:25:22.598 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:22.598 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:22.598 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:22.599 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:20.815780+0000", 2024-09-13T11:25:22.599 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.1584000000000001e-05 2024-09-13T11:25:22.599 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:22.599 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:22.599 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:22.599 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:22.599 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:22.599 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:168782 3.1a 3:590b82e1:::benchmark_data_smithi177_31116_object28479:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e161)", 2024-09-13T11:25:22.599 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:25:20.641567+0000", 2024-09-13T11:25:22.599 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 1.8576490539999999, 2024-09-13T11:25:22.599 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.19100501, 2024-09-13T11:25:22.599 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:22.599 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:22.600 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:22.600 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:25:22.600 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:25:22.600 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:22.600 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 168782 2024-09-13T11:25:22.600 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:22.600 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:22.600 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:22.600 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:22.600 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:20.641567+0000", 2024-09-13T11:25:22.600 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:22.600 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:22.600 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:22.601 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:22.601 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:20.641567+0000", 2024-09-13T11:25:22.601 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:22.601 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:22.601 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:22.601 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:22.601 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:20.641569+0000", 2024-09-13T11:25:22.601 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.869e-06 2024-09-13T11:25:22.601 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:22.601 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:22.601 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:22.601 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:20.641574+0000", 2024-09-13T11:25:22.601 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.1680000000000003e-06 2024-09-13T11:25:22.601 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:22.602 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:22.602 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:22.602 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:20.641587+0000", 2024-09-13T11:25:22.602 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2858e-05 2024-09-13T11:25:22.602 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:22.602 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:22.602 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:22.602 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:20.641595+0000", 2024-09-13T11:25:22.602 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.6149999999999999e-06 2024-09-13T11:25:22.602 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:22.602 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:22.602 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:22.602 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:20.641655+0000", 2024-09-13T11:25:22.602 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.9673999999999998e-05 2024-09-13T11:25:22.603 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:22.603 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:22.603 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:25:22.603 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:20.641748+0000", 2024-09-13T11:25:22.603 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.3919999999999995e-05 2024-09-13T11:25:22.603 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:22.603 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:22.603 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 9", 2024-09-13T11:25:22.603 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:20.642990+0000", 2024-09-13T11:25:22.603 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0012415709999999999 2024-09-13T11:25:22.603 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:22.603 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:22.603 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:25:22.603 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:20.645016+0000", 2024-09-13T11:25:22.604 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.002025448 2024-09-13T11:25:22.604 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:22.604 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:22.604 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:25:22.604 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:20.832409+0000", 2024-09-13T11:25:22.604 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.18739319600000001 2024-09-13T11:25:22.604 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:22.604 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:22.604 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:25:22.604 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:20.832525+0000", 2024-09-13T11:25:22.604 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000116273 2024-09-13T11:25:22.604 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:22.604 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:22.604 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:22.605 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:20.832572+0000", 2024-09-13T11:25:22.605 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.7417999999999999e-05 2024-09-13T11:25:22.605 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:22.605 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:22.605 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:22.605 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:22.605 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:22.605 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop_reply(client.4665.0:168772 3.1a e161/105 ondisk, result = 0)", 2024-09-13T11:25:22.605 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:25:20.645700+0000", 2024-09-13T11:25:22.605 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 1.853516873, 2024-09-13T11:25:22.605 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.17008606200000001, 2024-09-13T11:25:22.605 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:22.605 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:22.605 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:25:22.606 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:22.606 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:22.606 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:22.606 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:20.645700+0000", 2024-09-13T11:25:22.606 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:22.606 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:22.606 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:22.606 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:22.606 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:20.645700+0000", 2024-09-13T11:25:22.606 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:22.606 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:22.606 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:22.606 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:22.606 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:20.645701+0000", 2024-09-13T11:25:22.607 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.474e-06 2024-09-13T11:25:22.607 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:22.607 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:22.607 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:22.607 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:20.645714+0000", 2024-09-13T11:25:22.607 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2931999999999999e-05 2024-09-13T11:25:22.607 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:22.607 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:22.607 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:22.607 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:20.815603+0000", 2024-09-13T11:25:22.607 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.16988872199999999 2024-09-13T11:25:22.607 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:22.607 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:22.607 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:22.608 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:20.815610+0000", 2024-09-13T11:25:22.608 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.2139999999999999e-06 2024-09-13T11:25:22.608 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:22.608 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:22.608 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:22.608 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:20.815689+0000", 2024-09-13T11:25:22.608 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.9302999999999996e-05 2024-09-13T11:25:22.608 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:22.608 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:22.608 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:25:22.608 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:20.815695+0000", 2024-09-13T11:25:22.608 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.818e-06 2024-09-13T11:25:22.608 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:22.608 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:22.609 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:22.609 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:20.815786+0000", 2024-09-13T11:25:22.609 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.0599000000000005e-05 2024-09-13T11:25:22.609 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:22.609 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:22.609 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:22.609 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:22.609 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:22.609 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-13T11:25:22.611 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-09-13T11:25:22.611 INFO:teuthology.orchestra.run.smithi001.stdout:osd.0: {} 2024-09-13T11:25:22.612 INFO:teuthology.orchestra.run.smithi001.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T11:25:22.612 INFO:teuthology.orchestra.run.smithi001.stdout:osd.1: {} 2024-09-13T11:25:22.613 INFO:teuthology.orchestra.run.smithi001.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T11:25:22.613 INFO:tasks.ceph.osd.11.smithi177.stderr:2024-09-13T11:25:22.606+0000 7f24792c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26580) UID: 0 2024-09-13T11:25:22.625 INFO:teuthology.orchestra.run.smithi001.stdout:osd.2: {} 2024-09-13T11:25:22.626 INFO:teuthology.orchestra.run.smithi001.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T11:25:22.656 INFO:teuthology.orchestra.run.smithi001.stdout:osd.3: {} 2024-09-13T11:25:22.656 INFO:teuthology.orchestra.run.smithi001.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T11:25:22.678 INFO:teuthology.orchestra.run.smithi001.stdout:osd.4: {} 2024-09-13T11:25:22.679 INFO:teuthology.orchestra.run.smithi001.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T11:25:22.705 INFO:teuthology.orchestra.run.smithi001.stdout:osd.5: {} 2024-09-13T11:25:22.705 INFO:teuthology.orchestra.run.smithi001.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T11:25:22.712 INFO:tasks.ceph.osd.1.smithi092.stderr:2024-09-13T11:25:22.705+0000 7fe7789c5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33718) UID: 0 2024-09-13T11:25:22.730 INFO:teuthology.orchestra.run.smithi001.stdout:osd.6: {} 2024-09-13T11:25:22.730 INFO:teuthology.orchestra.run.smithi001.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T11:25:22.734 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-13T11:25:22.734 INFO:teuthology.orchestra.run.smithi001.stdout: "ops": [ 2024-09-13T11:25:22.734 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:22.734 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:171763 3.27 3:e7a18f3f:::benchmark_data_smithi177_31116_object31460:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e163)", 2024-09-13T11:25:22.734 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:25:22.676389+0000", 2024-09-13T11:25:22.735 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 0.057372424999999998, 2024-09-13T11:25:22.735 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.057397615999999999, 2024-09-13T11:25:22.735 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:22.735 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:22.735 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "waiting for sub ops", 2024-09-13T11:25:22.735 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:25:22.735 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:25:22.735 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:22.735 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 171763 2024-09-13T11:25:22.736 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:22.736 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:22.736 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:22.736 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:22.736 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:22.676389+0000", 2024-09-13T11:25:22.736 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:22.736 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:22.736 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:22.737 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:22.737 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:22.676389+0000", 2024-09-13T11:25:22.737 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:22.737 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:22.737 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:22.737 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:22.737 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:22.676391+0000", 2024-09-13T11:25:22.737 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.4559999999999999e-06 2024-09-13T11:25:22.737 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:22.737 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:22.738 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:22.738 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:22.676399+0000", 2024-09-13T11:25:22.738 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.6899999999999992e-06 2024-09-13T11:25:22.738 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:22.738 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:22.738 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:22.738 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:22.676415+0000", 2024-09-13T11:25:22.738 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.6659000000000001e-05 2024-09-13T11:25:22.738 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:22.738 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:22.739 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:22.739 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:22.676423+0000", 2024-09-13T11:25:22.739 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.3660000000000003e-06 2024-09-13T11:25:22.739 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:22.739 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:22.739 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:22.739 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:22.676494+0000", 2024-09-13T11:25:22.739 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.1456000000000002e-05 2024-09-13T11:25:22.739 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:22.740 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:22.740 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:25:22.740 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:22.676679+0000", 2024-09-13T11:25:22.740 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00018507800000000001 2024-09-13T11:25:22.740 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:22.740 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:22.740 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 5,11", 2024-09-13T11:25:22.740 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:22.676839+0000", 2024-09-13T11:25:22.740 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00015936699999999999 2024-09-13T11:25:22.740 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:22.741 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:22.741 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:25:22.741 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:22.678185+0000", 2024-09-13T11:25:22.741 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.001346482 2024-09-13T11:25:22.741 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:22.741 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:22.741 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:25:22.741 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:22.678685+0000", 2024-09-13T11:25:22.741 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000499414 2024-09-13T11:25:22.742 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:22.742 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:22.742 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:22.742 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:22.742 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-09-13T11:25:22.742 INFO:teuthology.orchestra.run.smithi001.stdout: "num_ops": 1 2024-09-13T11:25:22.742 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-13T11:25:22.754 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-09-13T11:25:22.759 INFO:teuthology.orchestra.run.smithi001.stdout:osd.7: {} 2024-09-13T11:25:22.759 INFO:teuthology.orchestra.run.smithi001.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T11:25:22.781 INFO:teuthology.orchestra.run.smithi001.stdout:osd.8: {} 2024-09-13T11:25:22.781 INFO:teuthology.orchestra.run.smithi001.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-13T11:25:22.810 INFO:teuthology.orchestra.run.smithi001.stdout:osd.9: {} 2024-09-13T11:25:22.811 INFO:teuthology.orchestra.run.smithi001.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-13T11:25:22.812 INFO:tasks.ceph.osd.15.smithi177.stderr:2024-09-13T11:25:22.810+0000 7f9d6ce77640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26578) UID: 0 2024-09-13T11:25:22.834 INFO:teuthology.orchestra.run.smithi001.stdout:osd.10: {} 2024-09-13T11:25:22.835 INFO:teuthology.orchestra.run.smithi001.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-13T11:25:22.857 INFO:teuthology.orchestra.run.smithi001.stdout:osd.11: {} 2024-09-13T11:25:22.857 INFO:teuthology.orchestra.run.smithi001.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-13T11:25:22.877 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-13T11:25:22.877 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:25:22.879 INFO:teuthology.orchestra.run.smithi001.stdout:osd.12: {} 2024-09-13T11:25:22.879 INFO:teuthology.orchestra.run.smithi001.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-13T11:25:22.894 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-09-13T11:25:22.911 INFO:teuthology.orchestra.run.smithi001.stdout:osd.13: {} 2024-09-13T11:25:22.911 INFO:teuthology.orchestra.run.smithi001.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-13T11:25:22.913 INFO:tasks.ceph.osd.15.smithi177.stderr:2024-09-13T11:25:22.910+0000 7f9d6ce77640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26578) UID: 0 2024-09-13T11:25:22.943 INFO:teuthology.orchestra.run.smithi001.stdout:osd.14: {} 2024-09-13T11:25:22.943 INFO:teuthology.orchestra.run.smithi001.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-13T11:25:22.970 INFO:teuthology.orchestra.run.smithi001.stdout:osd.15: {} 2024-09-13T11:25:22.970 INFO:teuthology.orchestra.run.smithi001.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-13T11:25:23.040 INFO:tasks.ceph.osd.4.smithi001.stderr:2024-09-13T11:25:23.039+0000 7ff8e0218640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26310) UID: 0 2024-09-13T11:25:23.047 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-13T11:25:23.047 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:25:23.065 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_ops_in_flight 2024-09-13T11:25:23.156 INFO:tasks.ceph.osd.4.smithi001.stderr:2024-09-13T11:25:23.151+0000 7ff8e0218640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26310) UID: 0 2024-09-13T11:25:23.190 INFO:teuthology.orchestra.run.smithi001.stderr:reweighted osd.12 to 1 (10000) 2024-09-13T11:25:23.193 INFO:tasks.ceph.osd.5.smithi092.stderr:2024-09-13T11:25:23.189+0000 7f6f02136640 -1 osd.5 163 handle_osd_map injecting map crc failure 2024-09-13T11:25:23.197 INFO:tasks.ceph.osd.2.smithi162.stderr:2024-09-13T11:25:23.192+0000 7fe921d8b640 -1 osd.2 163 handle_osd_map injecting map crc failure 2024-09-13T11:25:23.197 INFO:teuthology.orchestra.run.smithi001.stderr:nodeep-scrub is unset 2024-09-13T11:25:23.215 INFO:tasks.ceph.osd.13.smithi092.stderr:2024-09-13T11:25:23.209+0000 7f3e5afea640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26350) UID: 0 2024-09-13T11:25:23.231 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 13 1 2024-09-13T11:25:23.259 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-13T11:25:23.260 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:25:23.282 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_blocked_ops 2024-09-13T11:25:23.317 INFO:tasks.ceph.osd.11.smithi177.stderr:2024-09-13T11:25:23.314+0000 7f24792c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26580) UID: 0 2024-09-13T11:25:23.418 INFO:tasks.ceph.osd.10.smithi162.stderr:2024-09-13T11:25:23.416+0000 7fb59c9f0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26317) UID: 0 2024-09-13T11:25:23.475 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-13T11:25:23.475 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:25:23.498 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_historic_ops 2024-09-13T11:25:23.518 INFO:tasks.ceph.osd.1.smithi092.stderr:2024-09-13T11:25:23.513+0000 7fe7789c5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33718) UID: 0 2024-09-13T11:25:23.619 INFO:tasks.ceph.osd.5.smithi092.stderr:2024-09-13T11:25:23.613+0000 7f6f0b705640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26346) UID: 0 2024-09-13T11:25:23.674 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-13T11:25:23.674 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:25:23.695 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-13T11:25:23.741 INFO:tasks.ceph.osd.5.smithi092.stderr:2024-09-13T11:25:23.737+0000 7f6f0b705640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26346) UID: 0 2024-09-13T11:25:23.821 INFO:tasks.ceph.osd.11.smithi177.stderr:2024-09-13T11:25:23.818+0000 7f24792c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26580) UID: 0 2024-09-13T11:25:23.851 INFO:teuthology.orchestra.run.smithi092.stdout:ERROR: (22) Invalid argument 2024-09-13T11:25:23.852 INFO:teuthology.orchestra.run.smithi092.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:25:23.872 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-13T11:25:23.921 INFO:tasks.ceph.osd.4.smithi001.stderr:2024-09-13T11:25:23.919+0000 7ff8e0218640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26310) UID: 0 2024-09-13T11:25:24.022 INFO:tasks.ceph.osd.4.smithi001.stderr:2024-09-13T11:25:24.019+0000 7ff8e0218640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26310) UID: 0 2024-09-13T11:25:24.041 INFO:teuthology.orchestra.run.smithi092.stdout:ERROR: (22) Invalid argument 2024-09-13T11:25:24.041 INFO:teuthology.orchestra.run.smithi092.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:25:24.059 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-13T11:25:24.123 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-13T11:25:24.119+0000 7f6a68775640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26307) UID: 0 2024-09-13T11:25:24.192 INFO:teuthology.orchestra.run.smithi092.stdout:ERROR: (22) Invalid argument 2024-09-13T11:25:24.192 INFO:teuthology.orchestra.run.smithi092.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:25:24.201 INFO:teuthology.orchestra.run.smithi001.stderr:reweighted osd.13 to 1 (10000) 2024-09-13T11:25:24.204 INFO:tasks.ceph.osd.3.smithi177.stderr:2024-09-13T11:25:24.202+0000 7f10a45d5640 -1 osd.3 164 handle_osd_map injecting map crc failure 2024-09-13T11:25:24.210 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-09-13T11:25:24.227 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 15 1 2024-09-13T11:25:24.251 INFO:tasks.ceph.osd.14.smithi162.stderr:2024-09-13T11:25:24.248+0000 7f3dea330640 -1 osd.14 164 handle_osd_map injecting map crc failure 2024-09-13T11:25:24.253 INFO:tasks.ceph.osd.9.smithi092.stderr:2024-09-13T11:25:24.249+0000 7ff5ab3d0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26348) UID: 0 2024-09-13T11:25:24.324 INFO:tasks.ceph.osd.15.smithi177.stderr:2024-09-13T11:25:24.322+0000 7f9d6ce77640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26578) UID: 0 2024-09-13T11:25:24.346 INFO:teuthology.orchestra.run.smithi092.stdout:ERROR: (22) Invalid argument 2024-09-13T11:25:24.346 INFO:teuthology.orchestra.run.smithi092.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:25:24.364 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-09-13T11:25:24.449 INFO:tasks.ceph.osd.9.smithi092.stderr:2024-09-13T11:25:24.445+0000 7ff5ab3d0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26348) UID: 0 2024-09-13T11:25:24.526 INFO:tasks.ceph.osd.2.smithi162.stderr:2024-09-13T11:25:24.524+0000 7fe92b35a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26313) UID: 0 2024-09-13T11:25:24.540 INFO:teuthology.orchestra.run.smithi092.stdout:ERROR: (22) Invalid argument 2024-09-13T11:25:24.540 INFO:teuthology.orchestra.run.smithi092.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:25:24.558 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-09-13T11:25:24.627 INFO:tasks.ceph.osd.3.smithi177.stderr:2024-09-13T11:25:24.622+0000 7f10adba4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26574) UID: 0 2024-09-13T11:25:24.728 INFO:tasks.ceph.osd.10.smithi162.stderr:2024-09-13T11:25:24.724+0000 7fb59c9f0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26317) UID: 0 2024-09-13T11:25:24.739 INFO:teuthology.orchestra.run.smithi092.stdout:ERROR: (22) Invalid argument 2024-09-13T11:25:24.739 INFO:teuthology.orchestra.run.smithi092.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:25:24.758 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_ops_in_flight 2024-09-13T11:25:24.829 INFO:tasks.ceph.osd.12.smithi001.stderr:2024-09-13T11:25:24.827+0000 7f892c6e6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26314) UID: 0 2024-09-13T11:25:24.892 INFO:teuthology.orchestra.run.smithi092.stdout:ERROR: (22) Invalid argument 2024-09-13T11:25:24.892 INFO:teuthology.orchestra.run.smithi092.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:25:24.910 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_blocked_ops 2024-09-13T11:25:24.930 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-09-13T11:25:24.928+0000 7f97befc9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26315) UID: 0 2024-09-13T11:25:24.998 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T11:25:25.044 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-13T11:25:25.043+0000 7f6a68775640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26307) UID: 0 2024-09-13T11:25:25.044 INFO:teuthology.orchestra.run.smithi092.stdout:ERROR: (22) Invalid argument 2024-09-13T11:25:25.044 INFO:teuthology.orchestra.run.smithi092.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:25:25.058 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_historic_ops 2024-09-13T11:25:25.132 INFO:tasks.ceph.osd.11.smithi177.stderr:2024-09-13T11:25:25.130+0000 7f24792c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26580) UID: 0 2024-09-13T11:25:25.187 INFO:teuthology.orchestra.run.smithi001.stdout:osd.0: {} 2024-09-13T11:25:25.187 INFO:teuthology.orchestra.run.smithi001.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T11:25:25.198 INFO:teuthology.orchestra.run.smithi092.stdout:ERROR: (22) Invalid argument 2024-09-13T11:25:25.198 INFO:teuthology.orchestra.run.smithi092.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:25:25.211 INFO:teuthology.orchestra.run.smithi001.stderr:reweighted osd.15 to 1 (10000) 2024-09-13T11:25:25.215 INFO:teuthology.orchestra.run.smithi001.stdout:osd.1: {} 2024-09-13T11:25:25.215 INFO:teuthology.orchestra.run.smithi001.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T11:25:25.216 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-13T11:25:25.223 INFO:tasks.ceph.osd.15.smithi177.stderr:2024-09-13T11:25:25.218+0000 7f9d618a4640 -1 osd.15 165 handle_osd_map injecting map crc failure 2024-09-13T11:25:25.237 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-13T11:25:25.238 INFO:tasks.ceph.ceph_manager.ceph:waiting for recovery to complete 2024-09-13T11:25:25.239 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-13T11:25:25.249 INFO:teuthology.orchestra.run.smithi001.stdout:osd.2: {} 2024-09-13T11:25:25.249 INFO:teuthology.orchestra.run.smithi001.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T11:25:25.261 INFO:tasks.ceph.osd.10.smithi162.stderr:2024-09-13T11:25:25.260+0000 7fb59c9f0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26317) UID: 0 2024-09-13T11:25:25.285 INFO:teuthology.orchestra.run.smithi001.stdout:osd.3: {} 2024-09-13T11:25:25.285 INFO:teuthology.orchestra.run.smithi001.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T11:25:25.325 INFO:teuthology.orchestra.run.smithi001.stdout:osd.4: {} 2024-09-13T11:25:25.325 INFO:teuthology.orchestra.run.smithi001.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T11:25:25.334 INFO:tasks.ceph.osd.2.smithi162.stderr:2024-09-13T11:25:25.332+0000 7fe92b35a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26313) UID: 0 2024-09-13T11:25:25.353 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-13T11:25:25.353 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [], 2024-09-13T11:25:25.353 INFO:teuthology.orchestra.run.smithi162.stdout: "num_ops": 0 2024-09-13T11:25:25.353 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-13T11:25:25.364 INFO:teuthology.orchestra.run.smithi001.stdout:osd.5: {} 2024-09-13T11:25:25.364 INFO:teuthology.orchestra.run.smithi001.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T11:25:25.369 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-13T11:25:25.392 INFO:teuthology.orchestra.run.smithi001.stdout:osd.6: {} 2024-09-13T11:25:25.393 INFO:teuthology.orchestra.run.smithi001.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T11:25:25.425 INFO:teuthology.orchestra.run.smithi001.stdout:osd.7: {} 2024-09-13T11:25:25.426 INFO:teuthology.orchestra.run.smithi001.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T11:25:25.434 INFO:tasks.ceph.osd.15.smithi177.stderr:2024-09-13T11:25:25.430+0000 7f9d6ce77640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26578) UID: 0 2024-09-13T11:25:25.459 INFO:teuthology.orchestra.run.smithi001.stdout:osd.8: {} 2024-09-13T11:25:25.460 INFO:teuthology.orchestra.run.smithi001.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-13T11:25:25.494 INFO:teuthology.orchestra.run.smithi001.stdout:osd.9: {} 2024-09-13T11:25:25.495 INFO:teuthology.orchestra.run.smithi001.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-13T11:25:25.532 INFO:teuthology.orchestra.run.smithi001.stdout:osd.10: {} 2024-09-13T11:25:25.533 INFO:teuthology.orchestra.run.smithi001.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-13T11:25:25.535 INFO:tasks.ceph.osd.7.smithi177.stderr:2024-09-13T11:25:25.530+0000 7f897b51d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26575) UID: 0 2024-09-13T11:25:25.556 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-13T11:25:25.557 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [], 2024-09-13T11:25:25.557 INFO:teuthology.orchestra.run.smithi162.stdout: "complaint_time": 30, 2024-09-13T11:25:25.557 INFO:teuthology.orchestra.run.smithi162.stdout: "num_blocked_ops": 0 2024-09-13T11:25:25.557 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-13T11:25:25.562 INFO:teuthology.orchestra.run.smithi001.stdout:osd.11: {} 2024-09-13T11:25:25.562 INFO:teuthology.orchestra.run.smithi001.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-13T11:25:25.573 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-13T11:25:25.590 INFO:teuthology.orchestra.run.smithi001.stdout:osd.12: {} 2024-09-13T11:25:25.591 INFO:teuthology.orchestra.run.smithi001.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-13T11:25:25.621 INFO:teuthology.orchestra.run.smithi001.stdout:osd.13: {} 2024-09-13T11:25:25.622 INFO:teuthology.orchestra.run.smithi001.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-13T11:25:25.636 INFO:tasks.ceph.osd.9.smithi092.stderr:2024-09-13T11:25:25.629+0000 7ff5ab3d0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26348) UID: 0 2024-09-13T11:25:25.659 INFO:teuthology.orchestra.run.smithi001.stdout:osd.14: {} 2024-09-13T11:25:25.659 INFO:teuthology.orchestra.run.smithi001.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-13T11:25:25.686 INFO:teuthology.orchestra.run.smithi001.stdout:osd.15: {} 2024-09-13T11:25:25.686 INFO:teuthology.orchestra.run.smithi001.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-13T11:25:25.706 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-13T11:25:25.706 INFO:teuthology.orchestra.run.smithi162.stdout: "size": 20, 2024-09-13T11:25:25.706 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 600, 2024-09-13T11:25:25.707 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [ 2024-09-13T11:25:25.707 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:25.707 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.4665.0:171 3.9 e41/30 3:9e1c7eeb:::benchmark_data_smithi177_31116_object170:head v 41'13, mlcod=41'9)", 2024-09-13T11:25:25.707 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:22:47.418602+0000", 2024-09-13T11:25:25.707 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 158.286172972, 2024-09-13T11:25:25.707 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.67235367300000004, 2024-09-13T11:25:25.707 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:25:25.707 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:25:25.707 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:25.707 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:25:25.707 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:25.707 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:25:25.707 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:47.418602+0000", 2024-09-13T11:25:25.707 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:25.708 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:25.708 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:25.708 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:25:25.708 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:47.418602+0000", 2024-09-13T11:25:25.708 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:25.708 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:25.708 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:25.708 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:25:25.708 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:47.418605+0000", 2024-09-13T11:25:25.708 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.2200000000000001e-06 2024-09-13T11:25:25.708 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:25.708 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:25.709 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:25:25.709 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:47.418668+0000", 2024-09-13T11:25:25.709 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.2921000000000003e-05 2024-09-13T11:25:25.709 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:25.709 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:25.709 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:25:25.709 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:48.089380+0000", 2024-09-13T11:25:25.709 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.67071191100000005 2024-09-13T11:25:25.709 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:25.709 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:25.709 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:25:25.709 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:48.089423+0000", 2024-09-13T11:25:25.709 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.2916e-05 2024-09-13T11:25:25.710 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:25.710 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:25.710 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:25:25.710 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:48.089533+0000", 2024-09-13T11:25:25.710 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000109996 2024-09-13T11:25:25.710 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:25.710 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:25.710 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:25:25.710 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:48.089570+0000", 2024-09-13T11:25:25.710 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.7314999999999998e-05 2024-09-13T11:25:25.710 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:25.710 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:25.710 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:25:25.711 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:48.090852+0000", 2024-09-13T11:25:25.711 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001281641 2024-09-13T11:25:25.711 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:25.711 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:25.711 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:25:25.711 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:48.090956+0000", 2024-09-13T11:25:25.711 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00010375299999999999 2024-09-13T11:25:25.711 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:25.711 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:25:25.711 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:25.711 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:25.711 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:25.711 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPushReply(3.19 74/70 [PushReplyOp(3:9edf939c:::benchmark_data_smithi177_31116_object17207:head)])", 2024-09-13T11:25:25.712 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:30.897971+0000", 2024-09-13T11:25:25.712 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 114.806804121, 2024-09-13T11:25:25.712 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.40616071100000001, 2024-09-13T11:25:25.712 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:25:25.712 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:25:25.712 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-09-13T11:25:25.712 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:25:25.712 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:25.712 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:25:25.712 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.897971+0000", 2024-09-13T11:25:25.712 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:25.712 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:25.712 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:25.713 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:25:25.713 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.897971+0000", 2024-09-13T11:25:25.713 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:25.713 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:25.713 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:25.713 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:25:25.713 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.897972+0000", 2024-09-13T11:25:25.713 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1429999999999999e-06 2024-09-13T11:25:25.713 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:25.713 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:25.713 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:25:25.713 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.897977+0000", 2024-09-13T11:25:25.713 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.5199999999999999e-06 2024-09-13T11:25:25.714 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:25.714 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:25.714 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:25:25.714 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:31.302500+0000", 2024-09-13T11:25:25.714 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.40452312400000001 2024-09-13T11:25:25.714 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:25.714 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:25.714 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:25:25.714 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:31.302513+0000", 2024-09-13T11:25:25.714 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2753e-05 2024-09-13T11:25:25.714 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:25.714 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:25.714 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:25:25.715 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:31.302645+0000", 2024-09-13T11:25:25.715 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00013292100000000001 2024-09-13T11:25:25.715 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:25.715 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:25.715 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:25:25.715 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:31.304132+0000", 2024-09-13T11:25:25.715 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0014862499999999999 2024-09-13T11:25:25.715 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:25.715 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:25:25.715 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:25.715 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:25.715 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:25.715 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPushReply(3.19 75/70 [PushReplyOp(3:9d1489b1:::benchmark_data_smithi177_31116_object18698:head)])", 2024-09-13T11:25:25.715 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:31.697212+0000", 2024-09-13T11:25:25.716 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 114.00756312, 2024-09-13T11:25:25.716 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.22850633200000001, 2024-09-13T11:25:25.716 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:25:25.716 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:25:25.716 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-09-13T11:25:25.716 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:25:25.716 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:25.716 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:25:25.716 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:31.697212+0000", 2024-09-13T11:25:25.716 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:25.716 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:25.716 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:25.717 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:25:25.717 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:31.697212+0000", 2024-09-13T11:25:25.717 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:25.717 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:25.717 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:25.717 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:25:25.717 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:31.697214+0000", 2024-09-13T11:25:25.717 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.7439999999999999e-06 2024-09-13T11:25:25.717 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:25.717 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:25.717 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:25:25.717 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:31.697224+0000", 2024-09-13T11:25:25.717 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0173000000000001e-05 2024-09-13T11:25:25.717 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:25.718 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:25.718 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:25:25.718 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:31.923931+0000", 2024-09-13T11:25:25.718 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.226706926 2024-09-13T11:25:25.718 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:25.718 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:25.718 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:25:25.718 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:31.923950+0000", 2024-09-13T11:25:25.718 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.8899e-05 2024-09-13T11:25:25.718 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:25.718 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:25.718 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:25:25.718 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:31.924449+0000", 2024-09-13T11:25:25.719 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00049907800000000002 2024-09-13T11:25:25.719 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:25.719 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:25.719 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:25:25.719 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:31.925718+0000", 2024-09-13T11:25:25.719 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001269512 2024-09-13T11:25:25.719 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:25.719 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:25:25.719 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:25.719 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:25.719 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:25.719 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPushReply(3.19 76/70 [PushReplyOp(3:9faf6d5f:::benchmark_data_smithi177_31116_object18954:head)])", 2024-09-13T11:25:25.719 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:32.042874+0000", 2024-09-13T11:25:25.719 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 113.661901251, 2024-09-13T11:25:25.720 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.40212404299999999, 2024-09-13T11:25:25.720 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:25:25.720 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:25:25.720 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-09-13T11:25:25.720 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:25:25.720 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:25.720 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:25:25.720 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.042874+0000", 2024-09-13T11:25:25.720 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:25.720 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:25.720 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:25.720 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:25:25.721 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.042874+0000", 2024-09-13T11:25:25.721 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:25.721 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:25.721 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:25.721 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:25:25.721 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.042875+0000", 2024-09-13T11:25:25.721 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.432e-06 2024-09-13T11:25:25.721 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:25.721 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:25.721 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:25:25.721 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.042886+0000", 2024-09-13T11:25:25.721 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0265000000000001e-05 2024-09-13T11:25:25.721 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:25.722 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:25.722 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:25:25.722 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.443455+0000", 2024-09-13T11:25:25.722 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.40056951400000002 2024-09-13T11:25:25.722 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:25.722 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:25.722 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:25:25.722 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.443480+0000", 2024-09-13T11:25:25.722 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.4929000000000001e-05 2024-09-13T11:25:25.722 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:25.722 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:25.722 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:25:25.723 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.443666+0000", 2024-09-13T11:25:25.723 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000186446 2024-09-13T11:25:25.723 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:25.723 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:25.723 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:25:25.723 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.444998+0000", 2024-09-13T11:25:25.723 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0013314570000000001 2024-09-13T11:25:25.723 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:25.723 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:25:25.723 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:25.723 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:25.723 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:25.723 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPush(3.1d 85/78 [PushOp(3:bc4b9336:::benchmark_data_smithi177_31116_object11944:head, version: 49'734, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:bc4b9336:::benchmark_data_smithi177_31116_object11944:head@49'734, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:25:25.723 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:42.225966+0000", 2024-09-13T11:25:25.724 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 103.47880903799999, 2024-09-13T11:25:25.724 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.23879947800000001, 2024-09-13T11:25:25.724 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:25:25.724 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:25:25.724 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:25:25.724 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:25:25.724 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:25.724 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:25:25.724 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.225966+0000", 2024-09-13T11:25:25.724 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:25.724 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:25.724 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:25.724 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:25:25.724 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.225966+0000", 2024-09-13T11:25:25.725 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:25.725 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:25.725 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:25.725 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:25:25.725 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.225969+0000", 2024-09-13T11:25:25.725 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.0400000000000001e-06 2024-09-13T11:25:25.725 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:25.725 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:25.725 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:25:25.725 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.226036+0000", 2024-09-13T11:25:25.725 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.6978999999999997e-05 2024-09-13T11:25:25.725 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:25.725 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:25.725 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:25:25.726 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.463110+0000", 2024-09-13T11:25:25.726 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.23707400000000001 2024-09-13T11:25:25.726 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:25.726 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:25.726 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:25:25.726 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.463140+0000", 2024-09-13T11:25:25.726 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.9575999999999999e-05 2024-09-13T11:25:25.726 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:25.726 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:25.726 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:25:25.726 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.463326+0000", 2024-09-13T11:25:25.726 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000186298 2024-09-13T11:25:25.726 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:25.727 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:25.727 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:25:25.727 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.463351+0000", 2024-09-13T11:25:25.727 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.4862999999999999e-05 2024-09-13T11:25:25.727 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:25.727 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:25.727 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:25:25.727 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.464766+0000", 2024-09-13T11:25:25.727 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001414722 2024-09-13T11:25:25.727 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:25.727 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:25:25.727 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:25.728 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:25.728 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:25.728 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPush(3.1d 89/78 [PushOp(3:bc1b7cfb:::benchmark_data_smithi177_31116_object22629:head, version: 55'1387, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:bc1b7cfb:::benchmark_data_smithi177_31116_object22629:head@55'1387, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:25:25.728 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:46.660855+0000", 2024-09-13T11:25:25.728 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 99.043920541999995, 2024-09-13T11:25:25.728 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.26622785599999998, 2024-09-13T11:25:25.728 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:25:25.728 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:25:25.728 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:25:25.728 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:25:25.728 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:25.728 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:25:25.728 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:46.660855+0000", 2024-09-13T11:25:25.728 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:25.729 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:25.729 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:25.729 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:25:25.729 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:46.660855+0000", 2024-09-13T11:25:25.729 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:25.729 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:25.729 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:25.729 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:25:25.729 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:46.660857+0000", 2024-09-13T11:25:25.729 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.6560000000000001e-06 2024-09-13T11:25:25.729 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:25.729 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:25.729 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:25:25.729 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:46.660931+0000", 2024-09-13T11:25:25.730 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.4041999999999993e-05 2024-09-13T11:25:25.730 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:25.730 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:25.730 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:25:25.730 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:46.924654+0000", 2024-09-13T11:25:25.730 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.26372239800000002 2024-09-13T11:25:25.730 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:25.730 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:25.730 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:25:25.730 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:46.924729+0000", 2024-09-13T11:25:25.730 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.5061999999999994e-05 2024-09-13T11:25:25.730 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:25.730 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:25.730 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:25:25.731 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:46.924986+0000", 2024-09-13T11:25:25.731 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00025762000000000002 2024-09-13T11:25:25.731 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:25.731 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:25.731 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:25:25.731 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:46.925013+0000", 2024-09-13T11:25:25.731 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.6179000000000001e-05 2024-09-13T11:25:25.731 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:25.731 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:25.731 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:25:25.731 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:46.927082+0000", 2024-09-13T11:25:25.731 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.002069899 2024-09-13T11:25:25.731 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:25.731 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:25:25.732 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:25.732 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:25.732 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:25.732 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPush(3.1d 90/78 [PushOp(3:bc0e7228:::benchmark_data_smithi177_31116_object24733:head, version: 55'1512, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:bc0e7228:::benchmark_data_smithi177_31116_object24733:head@55'1512, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:25:25.732 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:47.529126+0000", 2024-09-13T11:25:25.732 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 98.175649567999997, 2024-09-13T11:25:25.732 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.21343989499999999, 2024-09-13T11:25:25.732 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:25:25.732 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:25:25.732 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:25:25.732 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:25:25.732 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:25.732 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:25:25.733 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:47.529126+0000", 2024-09-13T11:25:25.733 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:25.733 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:25.733 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:25.733 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:25:25.733 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:47.529126+0000", 2024-09-13T11:25:25.733 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:25.733 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:25.733 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:25.733 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:25:25.733 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:47.529127+0000", 2024-09-13T11:25:25.733 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.252e-06 2024-09-13T11:25:25.733 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:25.733 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:25.734 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:25:25.734 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:47.529185+0000", 2024-09-13T11:25:25.734 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.8224999999999999e-05 2024-09-13T11:25:25.734 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:25.734 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:25.734 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:25:25.734 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:47.740216+0000", 2024-09-13T11:25:25.734 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.21103060800000001 2024-09-13T11:25:25.734 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:25.734 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:25.734 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:25:25.734 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:47.740248+0000", 2024-09-13T11:25:25.734 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.2821999999999998e-05 2024-09-13T11:25:25.734 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:25.735 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:25.735 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:25:25.735 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:47.740504+0000", 2024-09-13T11:25:25.735 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00025518399999999999 2024-09-13T11:25:25.735 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:25.735 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:25.735 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:25:25.735 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:47.740544+0000", 2024-09-13T11:25:25.735 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.0438999999999997e-05 2024-09-13T11:25:25.735 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:25.735 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:25.735 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:25:25.735 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:47.742565+0000", 2024-09-13T11:25:25.735 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0020213649999999998 2024-09-13T11:25:25.736 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:25.736 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:25:25.736 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:25.736 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:25.736 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:25.736 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:83199 3.1d 3:be2c6fae:::benchmark_data_smithi177_31116_object83198:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106)", 2024-09-13T11:25:25.736 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:24:04.297306+0000", 2024-09-13T11:25:25.736 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 81.407469616, 2024-09-13T11:25:25.736 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.25114587999999999, 2024-09-13T11:25:25.736 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:25:25.736 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:25:25.736 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:25.736 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:25:25.737 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:25:25.737 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:25.737 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 83199 2024-09-13T11:25:25.737 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:25.737 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:25:25.737 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:25.737 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:25:25.737 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.297306+0000", 2024-09-13T11:25:25.738 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:25.738 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:25.738 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:25.738 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:25:25.738 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.297306+0000", 2024-09-13T11:25:25.738 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:25.738 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:25.738 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:25.738 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:25:25.738 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.297308+0000", 2024-09-13T11:25:25.738 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.4150000000000002e-06 2024-09-13T11:25:25.738 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:25.738 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:25.738 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:25:25.739 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.297352+0000", 2024-09-13T11:25:25.739 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.4431000000000002e-05 2024-09-13T11:25:25.739 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:25.739 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:25.739 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:25:25.739 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.297369+0000", 2024-09-13T11:25:25.739 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.6685e-05 2024-09-13T11:25:25.739 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:25.739 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:25.739 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:25:25.739 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.297376+0000", 2024-09-13T11:25:25.739 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.7299999999999999e-06 2024-09-13T11:25:25.739 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:25.740 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:25.740 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:25:25.740 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.297427+0000", 2024-09-13T11:25:25.740 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.1461000000000001e-05 2024-09-13T11:25:25.740 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:25.740 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:25.740 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:25:25.740 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.297528+0000", 2024-09-13T11:25:25.740 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00010028599999999999 2024-09-13T11:25:25.740 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:25.740 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:25.740 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 15", 2024-09-13T11:25:25.740 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.297685+0000", 2024-09-13T11:25:25.741 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00015760500000000001 2024-09-13T11:25:25.741 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:25.741 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:25.741 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:25:25.741 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.300779+0000", 2024-09-13T11:25:25.741 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0030936700000000002 2024-09-13T11:25:25.741 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:25.741 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:25.741 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:25:25.741 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.548279+0000", 2024-09-13T11:25:25.741 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.24750059999999999 2024-09-13T11:25:25.741 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:25.741 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:25.741 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:25:25.742 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.548373+0000", 2024-09-13T11:25:25.742 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.3451999999999999e-05 2024-09-13T11:25:25.742 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:25.742 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:25.742 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:25:25.742 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.548451+0000", 2024-09-13T11:25:25.742 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.8545000000000001e-05 2024-09-13T11:25:25.742 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:25.742 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:25:25.742 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:25.742 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:25.742 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:25.742 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:84621 3.1d 3:becd72aa:::benchmark_data_smithi177_31116_object84620:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-09-13T11:25:25.742 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:24:05.741008+0000", 2024-09-13T11:25:25.743 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 79.963767615999998, 2024-09-13T11:25:25.743 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.17994101100000001, 2024-09-13T11:25:25.743 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:25:25.743 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:25:25.743 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:25.743 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:25:25.743 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:25:25.743 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:25.743 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 84621 2024-09-13T11:25:25.743 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:25.743 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:25:25.743 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:25.743 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:25:25.744 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.741008+0000", 2024-09-13T11:25:25.744 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:25.744 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:25.744 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:25.744 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:25:25.744 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.741008+0000", 2024-09-13T11:25:25.744 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:25.744 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:25.744 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:25.744 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:25:25.744 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.741010+0000", 2024-09-13T11:25:25.744 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.898e-06 2024-09-13T11:25:25.744 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:25.744 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:25.745 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:25:25.745 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.741061+0000", 2024-09-13T11:25:25.745 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.0898e-05 2024-09-13T11:25:25.745 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:25.745 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:25.745 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:25:25.745 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.741085+0000", 2024-09-13T11:25:25.745 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.3558e-05 2024-09-13T11:25:25.745 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:25.745 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:25.745 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:25:25.745 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.741093+0000", 2024-09-13T11:25:25.745 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.3890000000000005e-06 2024-09-13T11:25:25.745 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:25.746 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:25.746 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:25:25.746 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.741178+0000", 2024-09-13T11:25:25.746 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.4792999999999999e-05 2024-09-13T11:25:25.746 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:25.746 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:25.746 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:25:25.746 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.741386+0000", 2024-09-13T11:25:25.746 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000207505 2024-09-13T11:25:25.746 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:25.746 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:25.746 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 15", 2024-09-13T11:25:25.746 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.741597+0000", 2024-09-13T11:25:25.746 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00021158200000000001 2024-09-13T11:25:25.747 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:25.747 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:25.747 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:25:25.747 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.743392+0000", 2024-09-13T11:25:25.747 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00179518 2024-09-13T11:25:25.747 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:25.747 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:25.747 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:25:25.747 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.920866+0000", 2024-09-13T11:25:25.747 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.17747373699999999 2024-09-13T11:25:25.747 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:25.747 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:25.747 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:25:25.748 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.920913+0000", 2024-09-13T11:25:25.748 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.6451e-05 2024-09-13T11:25:25.748 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:25.748 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:25.748 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:25:25.748 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.920949+0000", 2024-09-13T11:25:25.748 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.6019999999999997e-05 2024-09-13T11:25:25.748 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:25.748 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:25:25.748 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:25.748 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:25.748 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:25.748 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:88685 3.1d 3:baceb770:::benchmark_data_smithi177_31116_object88684:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110)", 2024-09-13T11:25:25.749 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:24:08.926194+0000", 2024-09-13T11:25:25.749 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 76.778580972, 2024-09-13T11:25:25.749 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.16678378799999999, 2024-09-13T11:25:25.749 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:25:25.749 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:25:25.749 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:25.749 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:25:25.749 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:25:25.749 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:25.749 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 88685 2024-09-13T11:25:25.749 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:25.749 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:25:25.749 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:25.750 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:25:25.750 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:08.926194+0000", 2024-09-13T11:25:25.750 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:25.750 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:25.750 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:25.750 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:25:25.750 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:08.926194+0000", 2024-09-13T11:25:25.750 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:25.750 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:25.750 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:25.750 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:25:25.750 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:08.926196+0000", 2024-09-13T11:25:25.750 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.533e-06 2024-09-13T11:25:25.750 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:25.751 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:25.751 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:25:25.751 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:08.926253+0000", 2024-09-13T11:25:25.751 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.6906e-05 2024-09-13T11:25:25.751 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:25.751 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:25.751 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:25:25.751 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:08.926262+0000", 2024-09-13T11:25:25.751 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.3530000000000001e-06 2024-09-13T11:25:25.751 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:25.751 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:25.751 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:25:25.751 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:08.926268+0000", 2024-09-13T11:25:25.751 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.2890000000000003e-06 2024-09-13T11:25:25.752 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:25.752 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:25.752 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:25:25.752 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:08.928510+0000", 2024-09-13T11:25:25.752 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0022416519999999998 2024-09-13T11:25:25.752 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:25.752 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:25.752 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:25:25.752 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:08.928748+0000", 2024-09-13T11:25:25.752 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00023773800000000001 2024-09-13T11:25:25.752 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:25.752 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:25.752 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 15", 2024-09-13T11:25:25.753 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:08.928953+0000", 2024-09-13T11:25:25.753 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00020532999999999999 2024-09-13T11:25:25.753 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:25.753 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:25.753 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:25:25.753 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:08.930702+0000", 2024-09-13T11:25:25.753 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001749163 2024-09-13T11:25:25.753 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:25.753 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:25.753 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:25:25.753 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:09.092878+0000", 2024-09-13T11:25:25.753 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.16217616900000001 2024-09-13T11:25:25.753 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:25.754 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:25.754 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:25:25.754 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:09.092932+0000", 2024-09-13T11:25:25.754 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.3523999999999998e-05 2024-09-13T11:25:25.754 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:25.754 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:25.754 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:25:25.754 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:09.092978+0000", 2024-09-13T11:25:25.754 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.6131000000000003e-05 2024-09-13T11:25:25.754 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:25.754 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:25:25.754 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:25.754 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:25.754 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:25.755 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPush(3.25 124/120 [PushOp(3:a6d4ee3f:::benchmark_data_smithi177_31116_object14712:head, version: 51'872, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:a6d4ee3f:::benchmark_data_smithi177_31116_object14712:head@51'872, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:25:25.755 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:24:31.996253+0000", 2024-09-13T11:25:25.755 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 53.708521844000003, 2024-09-13T11:25:25.755 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.17694863499999999, 2024-09-13T11:25:25.755 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:25:25.755 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:25:25.755 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:25:25.755 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:25:25.755 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:25.755 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:25:25.755 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:31.996253+0000", 2024-09-13T11:25:25.755 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:25.755 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:25.756 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:25.756 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:25:25.756 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:31.996253+0000", 2024-09-13T11:25:25.756 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:25.756 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:25.756 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:25.756 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:25:25.756 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:31.996257+0000", 2024-09-13T11:25:25.756 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.2629999999999999e-06 2024-09-13T11:25:25.756 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:25.756 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:25.756 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:25:25.756 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:31.996359+0000", 2024-09-13T11:25:25.756 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000102456 2024-09-13T11:25:25.757 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:25.757 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:25.757 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:25:25.757 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:32.172313+0000", 2024-09-13T11:25:25.757 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.17595388000000001 2024-09-13T11:25:25.757 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:25.757 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:25.757 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:25:25.757 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:32.172326+0000", 2024-09-13T11:25:25.757 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2632999999999999e-05 2024-09-13T11:25:25.757 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:25.757 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:25.757 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:25:25.757 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:32.172423+0000", 2024-09-13T11:25:25.758 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.7084999999999996e-05 2024-09-13T11:25:25.758 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:25.758 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:25.758 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:25:25.758 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:32.172438+0000", 2024-09-13T11:25:25.758 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5775999999999999e-05 2024-09-13T11:25:25.758 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:25.758 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:25.758 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:25:25.758 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:32.173202+0000", 2024-09-13T11:25:25.758 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00076354199999999998 2024-09-13T11:25:25.758 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:25.758 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:25:25.758 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:25.759 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:25.759 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:25.759 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPush(3.25 125/120 [PushOp(3:a4446742:::benchmark_data_smithi177_31116_object15486:head, version: 51'933, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:a4446742:::benchmark_data_smithi177_31116_object15486:head@51'933, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:25:25.759 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:24:32.725521+0000", 2024-09-13T11:25:25.759 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 52.979254533999999, 2024-09-13T11:25:25.759 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.16753559600000001, 2024-09-13T11:25:25.759 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:25:25.759 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:25:25.759 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:25:25.759 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:25:25.759 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:25.759 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:25:25.759 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:32.725521+0000", 2024-09-13T11:25:25.760 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:25.760 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:25.760 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:25.760 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:25:25.760 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:32.725521+0000", 2024-09-13T11:25:25.760 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:25.760 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:25.760 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:25.760 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:25:25.760 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:32.725523+0000", 2024-09-13T11:25:25.760 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.689e-06 2024-09-13T11:25:25.760 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:25.760 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:25.761 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:25:25.761 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:32.725566+0000", 2024-09-13T11:25:25.761 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.3177999999999999e-05 2024-09-13T11:25:25.761 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:25.761 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:25.761 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:25:25.761 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:32.892091+0000", 2024-09-13T11:25:25.761 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.16652452200000001 2024-09-13T11:25:25.761 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:25.761 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:25.761 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:25:25.761 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:32.892107+0000", 2024-09-13T11:25:25.761 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.6324000000000002e-05 2024-09-13T11:25:25.762 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:25.762 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:25.762 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:25:25.762 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:32.892226+0000", 2024-09-13T11:25:25.762 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000118814 2024-09-13T11:25:25.762 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:25.762 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:25.762 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:25:25.762 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:32.892241+0000", 2024-09-13T11:25:25.762 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4899000000000001e-05 2024-09-13T11:25:25.762 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:25.762 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:25.762 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:25:25.762 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:32.893056+0000", 2024-09-13T11:25:25.763 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00081517000000000004 2024-09-13T11:25:25.763 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:25.763 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:25:25.763 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:25.763 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:25.763 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:25.763 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:126039 3.29 3:96100181:::benchmark_data_smithi177_31116_object126038:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e130)", 2024-09-13T11:25:25.763 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:24:43.012785+0000", 2024-09-13T11:25:25.763 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 42.691990494000002, 2024-09-13T11:25:25.763 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.26685311900000003, 2024-09-13T11:25:25.763 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:25:25.763 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:25:25.763 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:25.764 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:25:25.764 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:25:25.764 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:25.764 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 126039 2024-09-13T11:25:25.764 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:25.764 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:25:25.764 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:25.764 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:25:25.764 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:43.012785+0000", 2024-09-13T11:25:25.764 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:25.764 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:25.764 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:25.764 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:25:25.764 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:43.012785+0000", 2024-09-13T11:25:25.765 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:25.765 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:25.765 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:25.765 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:25:25.765 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:43.012787+0000", 2024-09-13T11:25:25.765 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.875e-06 2024-09-13T11:25:25.765 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:25.765 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:25.765 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:25:25.765 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:43.012826+0000", 2024-09-13T11:25:25.765 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.9483999999999997e-05 2024-09-13T11:25:25.765 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:25.765 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:25.766 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:25:25.766 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:43.012839+0000", 2024-09-13T11:25:25.766 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3405e-05 2024-09-13T11:25:25.766 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:25.766 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:25.766 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:25:25.766 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:43.012845+0000", 2024-09-13T11:25:25.766 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.9100000000000002e-06 2024-09-13T11:25:25.766 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:25.766 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:25.766 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:25:25.766 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:43.012903+0000", 2024-09-13T11:25:25.766 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.8131000000000003e-05 2024-09-13T11:25:25.766 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:25.767 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:25.767 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:25:25.767 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:43.013066+0000", 2024-09-13T11:25:25.767 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000162058 2024-09-13T11:25:25.767 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:25.767 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:25.767 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 4,5", 2024-09-13T11:25:25.767 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:43.013235+0000", 2024-09-13T11:25:25.767 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00016929600000000001 2024-09-13T11:25:25.767 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:25.767 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:25.767 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:25:25.767 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:43.017063+0000", 2024-09-13T11:25:25.768 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0038283890000000002 2024-09-13T11:25:25.768 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:25.768 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:25.768 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:25:25.768 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:43.017086+0000", 2024-09-13T11:25:25.768 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.2934000000000001e-05 2024-09-13T11:25:25.768 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:25.768 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:25.768 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:25:25.768 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:43.278804+0000", 2024-09-13T11:25:25.768 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.26171751799999998 2024-09-13T11:25:25.768 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:25.769 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:25.769 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:25:25.769 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:43.278862+0000", 2024-09-13T11:25:25.769 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.8431999999999999e-05 2024-09-13T11:25:25.769 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:25.769 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:25.769 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:25:25.769 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:43.279638+0000", 2024-09-13T11:25:25.769 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00077568699999999995 2024-09-13T11:25:25.769 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:25.769 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:25:25.769 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:25.769 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:25.769 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:25.770 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop_reply(client.4665.0:126039 3.29 e130/120 ondisk, result = 0)", 2024-09-13T11:25:25.770 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:24:43.021691+0000", 2024-09-13T11:25:25.770 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 42.683084567999998, 2024-09-13T11:25:25.770 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.25795579699999999, 2024-09-13T11:25:25.770 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:25:25.770 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:25:25.770 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:25:25.770 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:25:25.770 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:25.770 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:25:25.770 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:43.021691+0000", 2024-09-13T11:25:25.770 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:25.770 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:25.770 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:25.771 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:25:25.771 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:43.021691+0000", 2024-09-13T11:25:25.771 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:25.771 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:25.771 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:25.771 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:25:25.771 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:43.021692+0000", 2024-09-13T11:25:25.771 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3880000000000001e-06 2024-09-13T11:25:25.771 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:25.771 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:25.771 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:25:25.771 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:43.021700+0000", 2024-09-13T11:25:25.771 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.9310000000000008e-06 2024-09-13T11:25:25.772 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:25.772 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:25.772 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:25:25.772 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:43.278719+0000", 2024-09-13T11:25:25.772 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.25701909299999998 2024-09-13T11:25:25.772 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:25.772 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:25.772 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:25:25.772 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:43.278728+0000", 2024-09-13T11:25:25.772 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.5000000000000005e-06 2024-09-13T11:25:25.772 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:25.772 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:25.772 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:25:25.773 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:43.278786+0000", 2024-09-13T11:25:25.773 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.7640999999999999e-05 2024-09-13T11:25:25.773 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:25.773 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:25.773 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:25:25.773 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:43.278796+0000", 2024-09-13T11:25:25.773 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.9199999999999999e-06 2024-09-13T11:25:25.773 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:25.773 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:25.774 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:25:25.774 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:43.279646+0000", 2024-09-13T11:25:25.774 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00085032399999999996 2024-09-13T11:25:25.774 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:25.774 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:25:25.774 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:25.774 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:25.774 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:25.774 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPushReply(3.29 134/120 [PushReplyOp(3:95648f22:::benchmark_data_smithi177_31116_object7446:head)])", 2024-09-13T11:25:25.774 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:24:47.110443+0000", 2024-09-13T11:25:25.774 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 38.594332448000003, 2024-09-13T11:25:25.775 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.171602386, 2024-09-13T11:25:25.775 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:25:25.775 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:25:25.775 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-09-13T11:25:25.775 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:25:25.775 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:25.775 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:25:25.775 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:47.110443+0000", 2024-09-13T11:25:25.775 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:25.775 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:25.775 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:25.776 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:25:25.776 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:47.110443+0000", 2024-09-13T11:25:25.776 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:25.776 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:25.776 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:25.776 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:25:25.776 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:47.110444+0000", 2024-09-13T11:25:25.776 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.192e-06 2024-09-13T11:25:25.776 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:25.776 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:25.776 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:25:25.777 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:47.110449+0000", 2024-09-13T11:25:25.777 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.0869999999999997e-06 2024-09-13T11:25:25.777 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:25.777 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:25.777 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:25:25.777 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:47.280321+0000", 2024-09-13T11:25:25.777 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.169872157 2024-09-13T11:25:25.777 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:25.777 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:25.777 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:25:25.777 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:47.280337+0000", 2024-09-13T11:25:25.778 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5656000000000001e-05 2024-09-13T11:25:25.778 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:25.778 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:25.778 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:25:25.778 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:47.280414+0000", 2024-09-13T11:25:25.778 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.7083999999999996e-05 2024-09-13T11:25:25.778 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:25.778 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:25.778 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:25:25.779 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:47.282045+0000", 2024-09-13T11:25:25.779 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0016312099999999999 2024-09-13T11:25:25.779 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:25.779 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:25:25.779 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:25.779 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:25.779 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:25.779 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.4665.0:138294 3.9 e140/117 3:90012dab:::benchmark_data_smithi177_31116_object138293:head v 140'5246, mlcod=140'5243)", 2024-09-13T11:25:25.779 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:24:54.252227+0000", 2024-09-13T11:25:25.779 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 31.45254808, 2024-09-13T11:25:25.780 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.32919836400000002, 2024-09-13T11:25:25.780 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:25:25.780 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:25:25.780 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:25.780 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:25:25.780 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:25.780 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:25:25.780 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:54.252227+0000", 2024-09-13T11:25:25.780 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:25.780 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:25.780 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:25.781 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:25:25.781 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:54.252227+0000", 2024-09-13T11:25:25.781 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:25.781 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:25.781 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:25.781 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:25:25.781 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:54.252228+0000", 2024-09-13T11:25:25.781 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.8899999999999998e-07 2024-09-13T11:25:25.781 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:25.781 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:25.782 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:25:25.782 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:54.252276+0000", 2024-09-13T11:25:25.782 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.8137999999999999e-05 2024-09-13T11:25:25.782 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:25.782 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:25.782 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:25:25.782 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:54.578216+0000", 2024-09-13T11:25:25.782 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.32594014399999999 2024-09-13T11:25:25.782 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:25.782 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:25.782 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:25:25.783 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:54.578235+0000", 2024-09-13T11:25:25.783 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.8943000000000001e-05 2024-09-13T11:25:25.783 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:25.783 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:25.783 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:25:25.783 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:54.578396+0000", 2024-09-13T11:25:25.783 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000160922 2024-09-13T11:25:25.783 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:25.783 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:25.783 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:25:25.783 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:54.578457+0000", 2024-09-13T11:25:25.783 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.0665999999999998e-05 2024-09-13T11:25:25.784 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:25.784 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:25.784 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:25:25.784 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:54.581339+0000", 2024-09-13T11:25:25.784 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0028826839999999999 2024-09-13T11:25:25.784 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:25.784 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:25.784 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:25:25.784 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:54.581425+0000", 2024-09-13T11:25:25.784 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.6077999999999998e-05 2024-09-13T11:25:25.784 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:25.785 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:25:25.785 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:25.785 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:25.785 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:25.785 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPushReply(3.29 143/120 [PushReplyOp(3:973d2047:::benchmark_data_smithi177_31116_object49749:head)])", 2024-09-13T11:25:25.785 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:24:57.387106+0000", 2024-09-13T11:25:25.785 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 28.317668753, 2024-09-13T11:25:25.785 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.64813637000000002, 2024-09-13T11:25:25.785 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:25:25.785 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:25:25.786 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-09-13T11:25:25.786 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:25:25.786 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:25.786 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:25:25.786 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:57.387106+0000", 2024-09-13T11:25:25.786 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:25.786 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:25.786 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:25.786 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:25:25.786 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:57.387106+0000", 2024-09-13T11:25:25.786 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:25.786 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:25.786 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:25.787 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:25:25.787 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:57.387116+0000", 2024-09-13T11:25:25.787 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.1530000000000003e-06 2024-09-13T11:25:25.787 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:25.787 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:25.787 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:25:25.787 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:57.387127+0000", 2024-09-13T11:25:25.787 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1197e-05 2024-09-13T11:25:25.787 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:25.787 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:25.787 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:25:25.787 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:58.034264+0000", 2024-09-13T11:25:25.788 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.64713691799999995 2024-09-13T11:25:25.788 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:25.788 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:25.788 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:25:25.788 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:58.034273+0000", 2024-09-13T11:25:25.788 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.0929999999999992e-06 2024-09-13T11:25:25.788 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:25.788 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:25.788 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:25:25.788 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:58.034321+0000", 2024-09-13T11:25:25.788 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.8396999999999998e-05 2024-09-13T11:25:25.789 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:25.789 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:25.789 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:25:25.789 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:58.035243+0000", 2024-09-13T11:25:25.789 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00092161200000000004 2024-09-13T11:25:25.789 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:25.789 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:25:25.789 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:25.789 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:25.789 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:25.789 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPushReply(3.29 144/120 [PushReplyOp(3:94a6fc3a:::benchmark_data_smithi177_31116_object50245:head)])", 2024-09-13T11:25:25.789 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:24:58.320977+0000", 2024-09-13T11:25:25.790 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 27.383798169999999, 2024-09-13T11:25:25.790 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.31110569599999999, 2024-09-13T11:25:25.790 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:25:25.790 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:25:25.790 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-09-13T11:25:25.790 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:25:25.790 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:25.790 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:25:25.790 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:58.320977+0000", 2024-09-13T11:25:25.790 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:25.791 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:25.791 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:25.791 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:25:25.791 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:58.320977+0000", 2024-09-13T11:25:25.791 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:25.791 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:25.791 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:25.791 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:25:25.791 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:58.320978+0000", 2024-09-13T11:25:25.791 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.8700000000000004e-07 2024-09-13T11:25:25.791 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:25.791 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:25.791 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:25:25.791 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:58.320985+0000", 2024-09-13T11:25:25.792 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.742e-06 2024-09-13T11:25:25.792 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:25.792 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:25.792 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:25:25.792 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:58.631147+0000", 2024-09-13T11:25:25.792 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.31016212900000001 2024-09-13T11:25:25.792 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:25.792 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:25.792 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:25:25.792 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:58.631159+0000", 2024-09-13T11:25:25.793 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1946999999999999e-05 2024-09-13T11:25:25.793 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:25.793 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:25.793 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:25:25.793 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:58.631257+0000", 2024-09-13T11:25:25.793 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.8096000000000004e-05 2024-09-13T11:25:25.793 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:25.793 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:25.793 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:25:25.793 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:58.632083+0000", 2024-09-13T11:25:25.793 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00082589499999999999 2024-09-13T11:25:25.793 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:25.794 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:25:25.794 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:25.794 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:25.794 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:25.794 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:147568 3.1d 3:b82a2ed2:::benchmark_data_smithi177_31116_object7265:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e148)", 2024-09-13T11:25:25.794 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:25:04.213894+0000", 2024-09-13T11:25:25.794 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 21.490881398999999, 2024-09-13T11:25:25.794 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.16897441999999999, 2024-09-13T11:25:25.794 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:25:25.794 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:25:25.794 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:25.794 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:25:25.794 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:25:25.795 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:25.795 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 147568 2024-09-13T11:25:25.795 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:25.795 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:25:25.795 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:25.795 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:25:25.795 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:04.213894+0000", 2024-09-13T11:25:25.795 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:25.795 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:25.795 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:25.795 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:25:25.795 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:04.213894+0000", 2024-09-13T11:25:25.796 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:25.796 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:25.796 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:25.796 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:25:25.796 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:04.213898+0000", 2024-09-13T11:25:25.796 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.9049999999999999e-06 2024-09-13T11:25:25.796 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:25.796 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:25.796 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:25:25.796 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:04.213908+0000", 2024-09-13T11:25:25.796 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0069e-05 2024-09-13T11:25:25.797 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:25.797 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:25.797 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:25:25.797 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:04.213931+0000", 2024-09-13T11:25:25.797 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.3538999999999999e-05 2024-09-13T11:25:25.797 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:25.797 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:25.797 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:25:25.797 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:04.213941+0000", 2024-09-13T11:25:25.797 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0073e-05 2024-09-13T11:25:25.797 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:25.798 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:25.798 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:25:25.798 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:04.214039+0000", 2024-09-13T11:25:25.798 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.7666999999999998e-05 2024-09-13T11:25:25.798 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:25.798 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:25.798 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:25:25.798 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:04.214226+0000", 2024-09-13T11:25:25.798 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000187166 2024-09-13T11:25:25.798 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:25.798 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:25.798 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 15", 2024-09-13T11:25:25.799 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:04.214365+0000", 2024-09-13T11:25:25.799 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000139016 2024-09-13T11:25:25.799 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:25.799 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:25.799 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:25:25.799 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:04.217864+0000", 2024-09-13T11:25:25.799 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.003499177 2024-09-13T11:25:25.799 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:25.799 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:25.799 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:25:25.799 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:04.382775+0000", 2024-09-13T11:25:25.799 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.16491091499999999 2024-09-13T11:25:25.800 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:25.800 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:25.800 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:25:25.800 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:04.382823+0000", 2024-09-13T11:25:25.800 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.8232000000000002e-05 2024-09-13T11:25:25.800 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:25.800 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:25.800 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:25:25.800 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:04.382868+0000", 2024-09-13T11:25:25.800 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.4660999999999998e-05 2024-09-13T11:25:25.801 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:25.801 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:25:25.801 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:25.801 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:25.801 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:25.801 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.4665.0:170721 3.9 e162/117 3:93244b1b:::benchmark_data_smithi177_31116_object30418:head v 162'5743, mlcod=162'5740)", 2024-09-13T11:25:25.801 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:25:21.721723+0000", 2024-09-13T11:25:25.801 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 3.9830524550000002, 2024-09-13T11:25:25.801 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.19722754300000001, 2024-09-13T11:25:25.801 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:25:25.801 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:25:25.802 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:25.802 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:25:25.802 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:25.802 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:25:25.802 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:21.721723+0000", 2024-09-13T11:25:25.802 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:25.802 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:25.802 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:25.802 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:25:25.802 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:21.721723+0000", 2024-09-13T11:25:25.802 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:25.802 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:25.803 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:25.803 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:25:25.803 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:21.721726+0000", 2024-09-13T11:25:25.803 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.1930000000000002e-06 2024-09-13T11:25:25.803 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:25.803 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:25.803 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:25:25.803 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:21.721741+0000", 2024-09-13T11:25:25.803 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5299999999999999e-05 2024-09-13T11:25:25.803 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:25.803 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:25.803 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:25:25.804 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:21.915450+0000", 2024-09-13T11:25:25.804 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.193708519 2024-09-13T11:25:25.804 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:25.804 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:25.804 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:25:25.804 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:21.915462+0000", 2024-09-13T11:25:25.804 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2414e-05 2024-09-13T11:25:25.804 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:25.804 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:25.804 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:25:25.805 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:21.915603+0000", 2024-09-13T11:25:25.805 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00014116500000000001 2024-09-13T11:25:25.805 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:25.805 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:25.805 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:25:25.805 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:21.915668+0000", 2024-09-13T11:25:25.805 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.4488999999999999e-05 2024-09-13T11:25:25.805 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:25.805 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:25.805 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:25:25.805 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:21.918850+0000", 2024-09-13T11:25:25.806 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0031825310000000002 2024-09-13T11:25:25.806 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:25.806 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:25.806 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:25:25.806 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:21.918950+0000", 2024-09-13T11:25:25.806 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.9932000000000005e-05 2024-09-13T11:25:25.806 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:25.806 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:25:25.806 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:25.806 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:25.806 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:25:25.807 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-13T11:25:25.808 INFO:tasks.ceph.osd.11.smithi177.stderr:2024-09-13T11:25:25.806+0000 7f24792c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26580) UID: 0 2024-09-13T11:25:25.809 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-13T11:25:25.844 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-13T11:25:25.845 INFO:teuthology.orchestra.run.smithi001.stderr:dumped all 2024-09-13T11:25:25.863 INFO:teuthology.orchestra.run.smithi001.stdout:{"pg_ready":true,"pg_map":{"version":249,"stamp":"2024-09-13T11:25:25.214714+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":7237501515,"num_objects":110432,"num_object_clones":0,"num_object_copies":220864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4616,"num_objects_unfound":0,"num_objects_dirty":110432,"num_whiteouts":0,"num_read":87,"num_read_kb":74,"num_write":86415,"num_write_kb":3637107,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19908,"num_bytes_recovered":1304559635,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":279767,"ondisk_log_size":279767,"up":110,"acting":110,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":123,"num_osds":15,"num_per_pool_osds":15,"num_per_pool_omap_osds":15,"kb":1415577600,"kb_used":16197728,"kb_used_data":14848816,"kb_used_omap":26,"kb_used_meta":1348453,"kb_avail":1399379872,"statfs":{"total":1449551462400,"available":1432964988928,"internally_reserved":0,"allocated":15205187584,"data_stored":15194649130,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":26763,"internal_metadata":1380816757},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-155648000,"num_objects":-2375,"num_object_clones":0,"num_object_copies":-4750,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1360,"num_objects_unfound":0,"num_objects_dirty":-2375,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2375,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":905,"num_bytes_recovered":59310080,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.040244"},"pg_stats":[{"pgid":"3.2d","version":"165'6141","reported_seq":9113,"reported_epoch":165,"state":"active+recovering+remapped","last_fresh":"2024-09-13T11:25:25.101919+0000","last_change":"2024-09-13T11:25:19.656321+0000","last_active":"2024-09-13T11:25:25.101919+0000","last_peered":"2024-09-13T11:25:25.101919+0000","last_clean":"2024-09-13T11:24:24.974950+0000","last_became_active":"2024-09-13T11:24:27.227288+0000","last_became_peered":"2024-09-13T11:24:27.227288+0000","last_unstale":"2024-09-13T11:25:25.101919+0000","last_undegraded":"2024-09-13T11:25:25.101919+0000","last_fullsized":"2024-09-13T11:25:25.101919+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":6,"last_scrub":"76'3395","last_scrub_stamp":"2024-09-13T11:23:32.776747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:32.776747+0000","objects_scrubbed":1704,"log_size":6141,"log_dups_size":0,"ondisk_log_size":6141,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":179,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110985216,"num_objects":1694,"num_object_clones":0,"num_object_copies":3388,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1674,"num_objects_unfound":0,"num_objects_dirty":1694,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1904,"num_write_kb":87840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":882,"num_bytes_recovered":57802752,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[3,0],"avail_no_missing":["3","7"],"object_location_counts":[{"shards":"0,3","objects":857},{"shards":"3,7","objects":837}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2c","version":"161'6126","reported_seq":11448,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-13T11:25:21.049707+0000","last_change":"2024-09-13T11:24:59.456762+0000","last_active":"2024-09-13T11:25:21.049707+0000","last_peered":"2024-09-13T11:25:21.049707+0000","last_clean":"2024-09-13T11:25:21.049707+0000","last_became_active":"2024-09-13T11:24:59.021895+0000","last_became_peered":"2024-09-13T11:24:59.021895+0000","last_unstale":"2024-09-13T11:25:21.049707+0000","last_undegraded":"2024-09-13T11:25:21.049707+0000","last_fullsized":"2024-09-13T11:25:21.049707+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":6,"last_scrub":"145'5672","last_scrub_stamp":"2024-09-13T11:24:59.456653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:24:59.456653+0000","objects_scrubbed":2192,"log_size":6126,"log_dups_size":0,"ondisk_log_size":6126,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:26:20.877991+0000","scrub_duration":279,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":113901568,"num_objects":1738,"num_object_clones":0,"num_object_copies":3476,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1738,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":454,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,7],"acting":[10,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.2b","version":"161'6016","reported_seq":9695,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-13T11:25:20.289715+0000","last_change":"2024-09-13T11:24:45.271477+0000","last_active":"2024-09-13T11:25:20.289715+0000","last_peered":"2024-09-13T11:25:20.289715+0000","last_clean":"2024-09-13T11:25:20.289715+0000","last_became_active":"2024-09-13T11:24:26.180578+0000","last_became_peered":"2024-09-13T11:24:26.180578+0000","last_unstale":"2024-09-13T11:25:20.289715+0000","last_undegraded":"2024-09-13T11:25:20.289715+0000","last_fullsized":"2024-09-13T11:25:20.289715+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":6,"last_scrub":"131'5382","last_scrub_stamp":"2024-09-13T11:24:45.271376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:24:45.271376+0000","objects_scrubbed":1940,"log_size":6016,"log_dups_size":0,"ondisk_log_size":6016,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:59.210895+0000","scrub_duration":279,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":107347968,"num_objects":1638,"num_object_clones":0,"num_object_copies":3276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1638,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":634,"num_write_kb":10624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2a","version":"161'6119","reported_seq":9985,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-13T11:25:21.063315+0000","last_change":"2024-09-13T11:24:46.625117+0000","last_active":"2024-09-13T11:25:21.063315+0000","last_peered":"2024-09-13T11:25:21.063315+0000","last_clean":"2024-09-13T11:25:21.063315+0000","last_became_active":"2024-09-13T11:24:45.320540+0000","last_became_peered":"2024-09-13T11:24:45.320540+0000","last_unstale":"2024-09-13T11:25:21.063315+0000","last_undegraded":"2024-09-13T11:25:21.063315+0000","last_fullsized":"2024-09-13T11:25:21.063315+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":6,"last_scrub":"133'5491","last_scrub_stamp":"2024-09-13T11:24:46.624970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:24:46.624970+0000","objects_scrubbed":2094,"log_size":6119,"log_dups_size":0,"ondisk_log_size":6119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:26:04.813280+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":118882304,"num_objects":1814,"num_object_clones":0,"num_object_copies":3628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1814,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":630,"num_write_kb":11200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,7],"acting":[10,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.29","version":"161'5774","reported_seq":11455,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-13T11:25:20.906211+0000","last_change":"2024-09-13T11:25:13.333062+0000","last_active":"2024-09-13T11:25:20.906211+0000","last_peered":"2024-09-13T11:25:20.906211+0000","last_clean":"2024-09-13T11:25:20.906211+0000","last_became_active":"2024-09-13T11:25:12.944028+0000","last_became_peered":"2024-09-13T11:25:12.944028+0000","last_unstale":"2024-09-13T11:25:20.906211+0000","last_undegraded":"2024-09-13T11:25:20.906211+0000","last_fullsized":"2024-09-13T11:25:20.906211+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":6,"last_scrub":"154'5579","last_scrub_stamp":"2024-09-13T11:25:13.332959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:13.332959+0000","objects_scrubbed":1937,"log_size":5774,"log_dups_size":0,"ondisk_log_size":5774,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:26:24.089228+0000","scrub_duration":243,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":114032640,"num_objects":1740,"num_object_clones":0,"num_object_copies":3480,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1740,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":197,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.28","version":"161'5646","reported_seq":8437,"reported_epoch":161,"state":"active+recovering+remapped","last_fresh":"2024-09-13T11:25:20.348299+0000","last_change":"2024-09-13T11:25:11.182075+0000","last_active":"2024-09-13T11:25:20.348299+0000","last_peered":"2024-09-13T11:25:20.348299+0000","last_clean":"2024-09-13T11:24:25.005266+0000","last_became_active":"2024-09-13T11:24:27.218508+0000","last_became_peered":"2024-09-13T11:24:27.218508+0000","last_unstale":"2024-09-13T11:25:20.348299+0000","last_undegraded":"2024-09-13T11:25:20.348299+0000","last_fullsized":"2024-09-13T11:25:20.348299+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":6,"last_scrub":"68'2771","last_scrub_stamp":"2024-09-13T11:23:20.644747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:20.644747+0000","objects_scrubbed":1356,"log_size":5646,"log_dups_size":0,"ondisk_log_size":5646,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":411,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":113311744,"num_objects":1729,"num_object_clones":0,"num_object_copies":3458,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1724,"num_objects_unfound":0,"num_objects_dirty":1729,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1899,"num_write_kb":94400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":824,"num_bytes_recovered":54001664,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5],"acting":[1,5],"avail_no_missing":["1","6"],"object_location_counts":[{"shards":"1,5","objects":867},{"shards":"1,6","objects":862}],"blocked_by":[],"up_primary":8,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.27","version":"161'5701","reported_seq":9371,"reported_epoch":161,"state":"active+recovering+remapped","last_fresh":"2024-09-13T11:25:21.021350+0000","last_change":"2024-09-13T11:25:08.115804+0000","last_active":"2024-09-13T11:25:21.021350+0000","last_peered":"2024-09-13T11:25:21.021350+0000","last_clean":"2024-09-13T11:24:24.942332+0000","last_became_active":"2024-09-13T11:24:27.215734+0000","last_became_peered":"2024-09-13T11:24:27.215734+0000","last_unstale":"2024-09-13T11:25:21.021350+0000","last_undegraded":"2024-09-13T11:25:21.021350+0000","last_fullsized":"2024-09-13T11:25:21.021350+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":6,"last_scrub":"68'2736","last_scrub_stamp":"2024-09-13T11:23:20.443329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:20.443329+0000","objects_scrubbed":99,"log_size":5701,"log_dups_size":0,"ondisk_log_size":5701,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":183,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":116195328,"num_objects":1773,"num_object_clones":0,"num_object_copies":3546,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1218,"num_objects_unfound":0,"num_objects_dirty":1773,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1979,"num_write_kb":98144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1116,"num_bytes_recovered":73138176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,11],"acting":[8,5],"avail_no_missing":["8","13"],"object_location_counts":[{"shards":"5,8","objects":1164},{"shards":"8,13","objects":609}],"blocked_by":[],"up_primary":5,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.26","version":"161'5836","reported_seq":11397,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-13T11:25:21.072063+0000","last_change":"2024-09-13T11:25:01.697082+0000","last_active":"2024-09-13T11:25:21.072063+0000","last_peered":"2024-09-13T11:25:21.072063+0000","last_clean":"2024-09-13T11:25:21.072063+0000","last_became_active":"2024-09-13T11:24:54.976919+0000","last_became_peered":"2024-09-13T11:24:54.976919+0000","last_unstale":"2024-09-13T11:25:21.072063+0000","last_undegraded":"2024-09-13T11:25:21.072063+0000","last_fullsized":"2024-09-13T11:25:21.072063+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":6,"last_scrub":"147'5433","last_scrub_stamp":"2024-09-13T11:25:01.696886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:01.696886+0000","objects_scrubbed":2158,"log_size":5836,"log_dups_size":0,"ondisk_log_size":5836,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:26:22.070621+0000","scrub_duration":403,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":114491392,"num_objects":1747,"num_object_clones":0,"num_object_copies":3494,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1747,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":411,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,12],"acting":[8,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.25","version":"161'5678","reported_seq":11398,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-13T11:25:21.054355+0000","last_change":"2024-09-13T11:24:57.073162+0000","last_active":"2024-09-13T11:25:21.054355+0000","last_peered":"2024-09-13T11:25:21.054355+0000","last_clean":"2024-09-13T11:25:21.054355+0000","last_became_active":"2024-09-13T11:24:55.999019+0000","last_became_peered":"2024-09-13T11:24:55.999019+0000","last_unstale":"2024-09-13T11:25:21.054355+0000","last_undegraded":"2024-09-13T11:25:21.054355+0000","last_fullsized":"2024-09-13T11:25:21.054355+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":6,"last_scrub":"142'5214","last_scrub_stamp":"2024-09-13T11:24:57.073061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:24:57.073061+0000","objects_scrubbed":2249,"log_size":5678,"log_dups_size":0,"ondisk_log_size":5678,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:26:12.348519+0000","scrub_duration":571,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117178368,"num_objects":1788,"num_object_clones":0,"num_object_copies":3576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1788,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":467,"num_write_kb":192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,2],"acting":[12,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[]},{"pgid":"3.24","version":"162'5742","reported_seq":9650,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-13T11:25:21.944050+0000","last_change":"2024-09-13T11:24:48.048846+0000","last_active":"2024-09-13T11:25:21.944050+0000","last_peered":"2024-09-13T11:25:21.944050+0000","last_clean":"2024-09-13T11:25:21.944050+0000","last_became_active":"2024-09-13T11:24:46.936994+0000","last_became_peered":"2024-09-13T11:24:46.936994+0000","last_unstale":"2024-09-13T11:25:21.944050+0000","last_undegraded":"2024-09-13T11:25:21.944050+0000","last_fullsized":"2024-09-13T11:25:21.944050+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":6,"last_scrub":"135'5093","last_scrub_stamp":"2024-09-13T11:24:48.048764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:24:48.048764+0000","objects_scrubbed":2102,"log_size":5742,"log_dups_size":0,"ondisk_log_size":5742,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:53.485370+0000","scrub_duration":287,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":114163712,"num_objects":1742,"num_object_clones":0,"num_object_copies":3484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1742,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":650,"num_write_kb":9280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1],"acting":[9,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.23","version":"161'4933","reported_seq":7774,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-13T11:25:20.951111+0000","last_change":"2024-09-13T11:23:48.133184+0000","last_active":"2024-09-13T11:25:20.951111+0000","last_peered":"2024-09-13T11:25:20.951111+0000","last_clean":"2024-09-13T11:25:20.951111+0000","last_became_active":"2024-09-13T11:23:47.926677+0000","last_became_peered":"2024-09-13T11:23:47.926677+0000","last_unstale":"2024-09-13T11:25:20.951111+0000","last_undegraded":"2024-09-13T11:25:20.951111+0000","last_fullsized":"2024-09-13T11:25:20.951111+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":6,"last_scrub":"91'3278","last_scrub_stamp":"2024-09-13T11:23:48.133084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:48.133084+0000","objects_scrubbed":1029,"log_size":4933,"log_dups_size":0,"ondisk_log_size":4933,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":139,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":118816768,"num_objects":1813,"num_object_clones":0,"num_object_copies":3626,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1813,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1656,"num_write_kb":78080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9],"acting":[5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.22","version":"161'4886","reported_seq":10457,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-13T11:25:21.072230+0000","last_change":"2024-09-13T11:24:45.485711+0000","last_active":"2024-09-13T11:25:21.072230+0000","last_peered":"2024-09-13T11:25:21.072230+0000","last_clean":"2024-09-13T11:25:21.072230+0000","last_became_active":"2024-09-13T11:24:29.253848+0000","last_became_peered":"2024-09-13T11:24:29.253848+0000","last_unstale":"2024-09-13T11:25:21.072230+0000","last_undegraded":"2024-09-13T11:25:21.072230+0000","last_fullsized":"2024-09-13T11:25:21.072230+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":6,"last_scrub":"90'3246","last_scrub_stamp":"2024-09-13T11:23:47.300339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:47.300339+0000","objects_scrubbed":1033,"log_size":4886,"log_dups_size":0,"ondisk_log_size":4886,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":203,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":115146752,"num_objects":1757,"num_object_clones":0,"num_object_copies":3514,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1757,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1642,"num_write_kb":75712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1768,"num_bytes_recovered":115867648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.21","version":"161'4806","reported_seq":7662,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-13T11:25:21.069900+0000","last_change":"2024-09-13T11:24:12.564117+0000","last_active":"2024-09-13T11:25:21.069900+0000","last_peered":"2024-09-13T11:25:21.069900+0000","last_clean":"2024-09-13T11:25:21.069900+0000","last_became_active":"2024-09-13T11:24:01.250308+0000","last_became_peered":"2024-09-13T11:24:01.250308+0000","last_unstale":"2024-09-13T11:25:21.069900+0000","last_undegraded":"2024-09-13T11:25:21.069900+0000","last_fullsized":"2024-09-13T11:25:21.069900+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":6,"last_scrub":"112'3670","last_scrub_stamp":"2024-09-13T11:24:12.564031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:24:12.564031+0000","objects_scrubbed":1431,"log_size":4806,"log_dups_size":0,"ondisk_log_size":4806,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:36.766299+0000","scrub_duration":183,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109707264,"num_objects":1674,"num_object_clones":0,"num_object_copies":3348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1674,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1139,"num_write_kb":44224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8],"acting":[0,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.20","version":"161'4939","reported_seq":7082,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-13T11:25:21.042354+0000","last_change":"2024-09-13T11:23:47.969503+0000","last_active":"2024-09-13T11:25:21.042354+0000","last_peered":"2024-09-13T11:25:21.042354+0000","last_clean":"2024-09-13T11:25:21.042354+0000","last_became_active":"2024-09-13T11:23:33.954946+0000","last_became_peered":"2024-09-13T11:23:33.954946+0000","last_unstale":"2024-09-13T11:25:21.042354+0000","last_undegraded":"2024-09-13T11:25:21.042354+0000","last_fullsized":"2024-09-13T11:25:21.042354+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":6,"last_scrub":"91'3270","last_scrub_stamp":"2024-09-13T11:23:47.969372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:47.969372+0000","objects_scrubbed":1061,"log_size":4939,"log_dups_size":0,"ondisk_log_size":4939,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:21.467746+0000","scrub_duration":1327,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112918528,"num_objects":1723,"num_object_clones":0,"num_object_copies":3446,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1723,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1674,"num_write_kb":74752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,0],"acting":[12,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[]},{"pgid":"3.1f","version":"161'6881","reported_seq":18822,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-13T11:25:21.059193+0000","last_change":"2024-09-13T11:25:20.773752+0000","last_active":"2024-09-13T11:25:21.059193+0000","last_peered":"2024-09-13T11:25:21.059193+0000","last_clean":"2024-09-13T11:25:21.059193+0000","last_became_active":"2024-09-13T11:24:29.290326+0000","last_became_peered":"2024-09-13T11:24:29.290326+0000","last_unstale":"2024-09-13T11:25:21.059193+0000","last_undegraded":"2024-09-13T11:25:21.059193+0000","last_fullsized":"2024-09-13T11:25:21.059193+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":5,"last_scrub":"161'6865","last_scrub_stamp":"2024-09-13T11:25:20.773641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:20.773641+0000","objects_scrubbed":3425,"log_size":6881,"log_dups_size":0,"ondisk_log_size":6881,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:26:21.317280+0000","scrub_duration":535,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":223084544,"num_objects":3404,"num_object_clones":0,"num_object_copies":6808,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3404,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2461,"num_write_kb":102848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3306,"num_bytes_recovered":216662016,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5],"acting":[10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1e","version":"161'7120","reported_seq":17627,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-13T11:25:21.008189+0000","last_change":"2024-09-13T11:24:08.338485+0000","last_active":"2024-09-13T11:25:21.008189+0000","last_peered":"2024-09-13T11:25:21.008189+0000","last_clean":"2024-09-13T11:25:21.008189+0000","last_became_active":"2024-09-13T11:24:01.261448+0000","last_became_peered":"2024-09-13T11:24:01.261448+0000","last_unstale":"2024-09-13T11:25:21.008189+0000","last_undegraded":"2024-09-13T11:25:21.008189+0000","last_fullsized":"2024-09-13T11:25:21.008189+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":5,"last_scrub":"110'4537","last_scrub_stamp":"2024-09-13T11:24:08.338354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:24:08.338354+0000","objects_scrubbed":2760,"log_size":7120,"log_dups_size":0,"ondisk_log_size":7120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":367,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":228589568,"num_objects":3488,"num_object_clones":0,"num_object_copies":6976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3488,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2592,"num_write_kb":106240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"161'6918","reported_seq":12354,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-13T11:25:21.052521+0000","last_change":"2024-09-13T11:24:12.492592+0000","last_active":"2024-09-13T11:25:21.052521+0000","last_peered":"2024-09-13T11:25:21.052521+0000","last_clean":"2024-09-13T11:25:21.052521+0000","last_became_active":"2024-09-13T11:24:01.258225+0000","last_became_peered":"2024-09-13T11:24:01.258225+0000","last_unstale":"2024-09-13T11:25:21.052521+0000","last_undegraded":"2024-09-13T11:25:21.052521+0000","last_fullsized":"2024-09-13T11:25:21.052521+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":5,"last_scrub":"112'4579","last_scrub_stamp":"2024-09-13T11:24:12.492507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:24:12.492507+0000","objects_scrubbed":2875,"log_size":6918,"log_dups_size":0,"ondisk_log_size":6918,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:34.926575+0000","scrub_duration":375,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":228065304,"num_objects":3481,"num_object_clones":0,"num_object_copies":6962,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3481,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2352,"num_write_kb":94593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,15],"acting":[2,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1c","version":"162'7099","reported_seq":10894,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-13T11:25:22.041539+0000","last_change":"2024-09-13T11:24:00.529312+0000","last_active":"2024-09-13T11:25:22.041539+0000","last_peered":"2024-09-13T11:25:22.041539+0000","last_clean":"2024-09-13T11:25:22.041539+0000","last_became_active":"2024-09-13T11:23:33.989492+0000","last_became_peered":"2024-09-13T11:23:33.989492+0000","last_unstale":"2024-09-13T11:25:22.041539+0000","last_undegraded":"2024-09-13T11:25:22.041539+0000","last_fullsized":"2024-09-13T11:25:22.041539+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":5,"last_scrub":"102'4277","last_scrub_stamp":"2024-09-13T11:24:00.529243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:24:00.529243+0000","objects_scrubbed":2544,"log_size":7099,"log_dups_size":0,"ondisk_log_size":7099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":259,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":226754560,"num_objects":3460,"num_object_clones":0,"num_object_copies":6920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3460,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2824,"num_write_kb":119680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7],"acting":[11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1b","version":"161'7049","reported_seq":12664,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-13T11:25:21.114543+0000","last_change":"2024-09-13T11:24:12.392093+0000","last_active":"2024-09-13T11:25:21.114543+0000","last_peered":"2024-09-13T11:25:21.114543+0000","last_clean":"2024-09-13T11:25:21.114543+0000","last_became_active":"2024-09-13T11:24:04.950821+0000","last_became_peered":"2024-09-13T11:24:04.950821+0000","last_unstale":"2024-09-13T11:25:21.114543+0000","last_undegraded":"2024-09-13T11:25:21.114543+0000","last_fullsized":"2024-09-13T11:25:21.114543+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":5,"last_scrub":"112'4665","last_scrub_stamp":"2024-09-13T11:24:12.391890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:24:12.391890+0000","objects_scrubbed":2934,"log_size":7049,"log_dups_size":0,"ondisk_log_size":7049,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:41.846147+0000","scrub_duration":675,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":228524032,"num_objects":3487,"num_object_clones":0,"num_object_copies":6974,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3487,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2405,"num_write_kb":94656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,15],"acting":[12,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[]},{"pgid":"3.1a","version":"161'6928","reported_seq":12204,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-13T11:25:21.040785+0000","last_change":"2024-09-13T11:24:09.556174+0000","last_active":"2024-09-13T11:25:21.040785+0000","last_peered":"2024-09-13T11:25:21.040785+0000","last_clean":"2024-09-13T11:25:21.040785+0000","last_became_active":"2024-09-13T11:24:03.935185+0000","last_became_peered":"2024-09-13T11:24:03.935185+0000","last_unstale":"2024-09-13T11:25:21.040785+0000","last_undegraded":"2024-09-13T11:25:21.040785+0000","last_fullsized":"2024-09-13T11:25:21.040785+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":5,"last_scrub":"110'4433","last_scrub_stamp":"2024-09-13T11:24:09.556022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:24:09.556022+0000","objects_scrubbed":2654,"log_size":6928,"log_dups_size":0,"ondisk_log_size":6928,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:28.469324+0000","scrub_duration":387,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":223608832,"num_objects":3412,"num_object_clones":0,"num_object_copies":6824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3412,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2506,"num_write_kb":104448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9],"acting":[4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.19","version":"162'6603","reported_seq":11067,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-13T11:25:22.000325+0000","last_change":"2024-09-13T11:25:10.784789+0000","last_active":"2024-09-13T11:25:22.000325+0000","last_peered":"2024-09-13T11:25:22.000325+0000","last_clean":"2024-09-13T11:25:22.000325+0000","last_became_active":"2024-09-13T11:23:41.924941+0000","last_became_peered":"2024-09-13T11:23:41.924941+0000","last_unstale":"2024-09-13T11:25:22.000325+0000","last_undegraded":"2024-09-13T11:25:22.000325+0000","last_fullsized":"2024-09-13T11:25:22.000325+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":5,"last_scrub":"151'6156","last_scrub_stamp":"2024-09-13T11:25:10.784716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:10.784716+0000","objects_scrubbed":3781,"log_size":6603,"log_dups_size":0,"ondisk_log_size":6603,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:26:30.390310+0000","scrub_duration":1131,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":217317376,"num_objects":3316,"num_object_clones":0,"num_object_copies":6632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3316,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3201,"num_write_kb":145216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1],"acting":[9,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.18","version":"161'6633","reported_seq":11358,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-13T11:25:21.075816+0000","last_change":"2024-09-13T11:24:13.454659+0000","last_active":"2024-09-13T11:25:21.075816+0000","last_peered":"2024-09-13T11:25:21.075816+0000","last_clean":"2024-09-13T11:25:21.075816+0000","last_became_active":"2024-09-13T11:23:48.926014+0000","last_became_peered":"2024-09-13T11:23:48.926014+0000","last_unstale":"2024-09-13T11:25:21.075816+0000","last_undegraded":"2024-09-13T11:25:21.075816+0000","last_fullsized":"2024-09-13T11:25:21.075816+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":5,"last_scrub":"112'4376","last_scrub_stamp":"2024-09-13T11:24:13.454575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:24:13.454575+0000","objects_scrubbed":3036,"log_size":6633,"log_dups_size":0,"ondisk_log_size":6633,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:34.811860+0000","scrub_duration":375,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":227672064,"num_objects":3474,"num_object_clones":0,"num_object_copies":6948,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3474,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2266,"num_write_kb":86528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0],"acting":[10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.17","version":"161'6534","reported_seq":10729,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-13T11:25:21.058643+0000","last_change":"2024-09-13T11:25:12.658562+0000","last_active":"2024-09-13T11:25:21.058643+0000","last_peered":"2024-09-13T11:25:21.058643+0000","last_clean":"2024-09-13T11:25:21.058643+0000","last_became_active":"2024-09-13T11:23:42.935601+0000","last_became_peered":"2024-09-13T11:23:42.935601+0000","last_unstale":"2024-09-13T11:25:21.058643+0000","last_undegraded":"2024-09-13T11:25:21.058643+0000","last_fullsized":"2024-09-13T11:25:21.058643+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":5,"last_scrub":"153'6197","last_scrub_stamp":"2024-09-13T11:25:12.658464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:12.658464+0000","objects_scrubbed":3743,"log_size":6534,"log_dups_size":0,"ondisk_log_size":6534,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:26:25.625277+0000","scrub_duration":467,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":223150080,"num_objects":3405,"num_object_clones":0,"num_object_copies":6810,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3405,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3183,"num_write_kb":146688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3],"acting":[10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"162'6868","reported_seq":11702,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-13T11:25:22.064255+0000","last_change":"2024-09-13T11:25:01.903487+0000","last_active":"2024-09-13T11:25:22.064255+0000","last_peered":"2024-09-13T11:25:22.064255+0000","last_clean":"2024-09-13T11:25:22.064255+0000","last_became_active":"2024-09-13T11:23:46.105237+0000","last_became_peered":"2024-09-13T11:23:46.105237+0000","last_unstale":"2024-09-13T11:25:22.064255+0000","last_undegraded":"2024-09-13T11:25:22.064255+0000","last_fullsized":"2024-09-13T11:25:22.064255+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":5,"last_scrub":"147'6035","last_scrub_stamp":"2024-09-13T11:25:01.903395+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:01.903395+0000","objects_scrubbed":4359,"log_size":6868,"log_dups_size":0,"ondisk_log_size":6868,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:26:16.436377+0000","scrub_duration":743,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":230096896,"num_objects":3511,"num_object_clones":0,"num_object_copies":7022,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3511,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3393,"num_write_kb":155200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,15],"acting":[11,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.15","version":"161'6520","reported_seq":17835,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-13T11:25:21.107228+0000","last_change":"2024-09-13T11:25:12.198448+0000","last_active":"2024-09-13T11:25:21.107228+0000","last_peered":"2024-09-13T11:25:21.107228+0000","last_clean":"2024-09-13T11:25:21.107228+0000","last_became_active":"2024-09-13T11:24:29.226615+0000","last_became_peered":"2024-09-13T11:24:29.226615+0000","last_unstale":"2024-09-13T11:25:21.107228+0000","last_undegraded":"2024-09-13T11:25:21.107228+0000","last_fullsized":"2024-09-13T11:25:21.107228+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":5,"last_scrub":"91'3384","last_scrub_stamp":"2024-09-13T11:23:48.332600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:48.332600+0000","objects_scrubbed":2032,"log_size":6520,"log_dups_size":0,"ondisk_log_size":6520,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":235,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":223150080,"num_objects":3405,"num_object_clones":0,"num_object_copies":6810,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3405,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3137,"num_write_kb":144320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3385,"num_bytes_recovered":221839360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.14","version":"165'6857","reported_seq":18106,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-13T11:25:25.064840+0000","last_change":"2024-09-13T11:25:15.378521+0000","last_active":"2024-09-13T11:25:25.064840+0000","last_peered":"2024-09-13T11:25:25.064840+0000","last_clean":"2024-09-13T11:25:25.064840+0000","last_became_active":"2024-09-13T11:24:29.253106+0000","last_became_peered":"2024-09-13T11:24:29.253106+0000","last_unstale":"2024-09-13T11:25:25.064840+0000","last_undegraded":"2024-09-13T11:25:25.064840+0000","last_fullsized":"2024-09-13T11:25:25.064840+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":5,"last_scrub":"91'3371","last_scrub_stamp":"2024-09-13T11:23:48.696960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:48.696960+0000","objects_scrubbed":1935,"log_size":6857,"log_dups_size":0,"ondisk_log_size":6857,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":735,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":222494720,"num_objects":3395,"num_object_clones":0,"num_object_copies":6790,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3395,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3496,"num_write_kb":158592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3437,"num_bytes_recovered":225247232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,15],"acting":[3,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":294,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-13T11:25:20.171181+0000","last_change":"2024-09-13T11:24:23.212494+0000","last_active":"2024-09-13T11:25:20.171181+0000","last_peered":"2024-09-13T11:25:20.171181+0000","last_clean":"2024-09-13T11:25:20.171181+0000","last_became_active":"2024-09-13T11:22:14.192897+0000","last_became_peered":"2024-09-13T11:22:14.192897+0000","last_unstale":"2024-09-13T11:25:20.171181+0000","last_undegraded":"2024-09-13T11:25:20.171181+0000","last_fullsized":"2024-09-13T11:25:20.171181+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:24:23.212406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:24:23.212406+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:29.886382+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00084655899999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,10],"acting":[15,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"3.6","version":"162'5853","reported_seq":9708,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-13T11:25:21.885529+0000","last_change":"2024-09-13T11:24:49.416851+0000","last_active":"2024-09-13T11:25:21.885529+0000","last_peered":"2024-09-13T11:25:21.885529+0000","last_clean":"2024-09-13T11:25:21.885529+0000","last_became_active":"2024-09-13T11:24:29.322558+0000","last_became_peered":"2024-09-13T11:24:29.322558+0000","last_unstale":"2024-09-13T11:25:21.885529+0000","last_undegraded":"2024-09-13T11:25:21.885529+0000","last_fullsized":"2024-09-13T11:25:21.885529+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"136'5216","last_scrub_stamp":"2024-09-13T11:24:49.416706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:24:49.416706+0000","objects_scrubbed":2083,"log_size":5853,"log_dups_size":0,"ondisk_log_size":5853,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:26:09.525086+0000","scrub_duration":247,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":113967104,"num_objects":1739,"num_object_clones":0,"num_object_copies":3478,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1739,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":638,"num_write_kb":9408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,6],"acting":[13,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":303,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-13T11:25:21.181055+0000","last_change":"2024-09-13T11:24:59.556253+0000","last_active":"2024-09-13T11:25:21.181055+0000","last_peered":"2024-09-13T11:25:21.181055+0000","last_clean":"2024-09-13T11:25:21.181055+0000","last_became_active":"2024-09-13T11:22:14.211923+0000","last_became_peered":"2024-09-13T11:22:14.211923+0000","last_unstale":"2024-09-13T11:25:21.181055+0000","last_undegraded":"2024-09-13T11:25:21.181055+0000","last_fullsized":"2024-09-13T11:25:21.181055+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:24:59.556180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:24:59.556180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:26:06.229322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011689529999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0],"acting":[9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"162'5680","reported_seq":6029,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-13T11:25:22.049662+0000","last_change":"2024-09-13T11:24:33.336210+0000","last_active":"2024-09-13T11:25:22.049662+0000","last_peered":"2024-09-13T11:25:22.049662+0000","last_clean":"2024-09-13T11:25:22.049662+0000","last_became_active":"2024-09-13T11:24:24.092975+0000","last_became_peered":"2024-09-13T11:24:24.092975+0000","last_unstale":"2024-09-13T11:25:22.049662+0000","last_undegraded":"2024-09-13T11:25:22.049662+0000","last_fullsized":"2024-09-13T11:25:22.049662+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"125'4820","last_scrub_stamp":"2024-09-13T11:24:33.336091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:24:33.336091+0000","objects_scrubbed":1766,"log_size":5680,"log_dups_size":0,"ondisk_log_size":5680,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:54.538352+0000","scrub_duration":263,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":113246208,"num_objects":1728,"num_object_clones":0,"num_object_copies":3456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1728,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":862,"num_write_kb":26368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,8],"acting":[13,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":294,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-13T11:25:20.165033+0000","last_change":"2024-09-13T11:23:59.530794+0000","last_active":"2024-09-13T11:25:20.165033+0000","last_peered":"2024-09-13T11:25:20.165033+0000","last_clean":"2024-09-13T11:25:20.165033+0000","last_became_active":"2024-09-13T11:22:14.581458+0000","last_became_peered":"2024-09-13T11:22:14.581458+0000","last_unstale":"2024-09-13T11:25:20.165033+0000","last_undegraded":"2024-09-13T11:25:20.165033+0000","last_fullsized":"2024-09-13T11:25:20.165033+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:23:59.530670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:23:59.530670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001026047,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5],"acting":[8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"161'5722","reported_seq":6041,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-13T11:25:20.283916+0000","last_change":"2024-09-13T11:24:24.579036+0000","last_active":"2024-09-13T11:25:20.283916+0000","last_peered":"2024-09-13T11:25:20.283916+0000","last_clean":"2024-09-13T11:25:20.283916+0000","last_became_active":"2024-09-13T11:24:24.083077+0000","last_became_peered":"2024-09-13T11:24:24.083077+0000","last_unstale":"2024-09-13T11:25:20.283916+0000","last_undegraded":"2024-09-13T11:25:20.283916+0000","last_fullsized":"2024-09-13T11:25:20.283916+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'4737","last_scrub_stamp":"2024-09-13T11:24:24.578900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:24:24.578900+0000","objects_scrubbed":1690,"log_size":5722,"log_dups_size":0,"ondisk_log_size":5722,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:38.103421+0000","scrub_duration":407,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112984064,"num_objects":1724,"num_object_clones":0,"num_object_copies":3448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1724,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":988,"num_write_kb":32704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,12],"acting":[1,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":282,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-13T11:25:20.170533+0000","last_change":"2024-09-13T11:24:08.271026+0000","last_active":"2024-09-13T11:25:20.170533+0000","last_peered":"2024-09-13T11:25:20.170533+0000","last_clean":"2024-09-13T11:25:20.170533+0000","last_became_active":"2024-09-13T11:22:47.719636+0000","last_became_peered":"2024-09-13T11:22:47.719636+0000","last_unstale":"2024-09-13T11:25:20.170533+0000","last_undegraded":"2024-09-13T11:25:20.170533+0000","last_fullsized":"2024-09-13T11:25:20.170533+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:24:08.270972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:24:08.270972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:36.805605+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00087476000000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,9],"acting":[1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"161'5667","reported_seq":6005,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-13T11:25:21.015508+0000","last_change":"2024-09-13T11:24:24.616937+0000","last_active":"2024-09-13T11:25:21.015508+0000","last_peered":"2024-09-13T11:25:21.015508+0000","last_clean":"2024-09-13T11:25:21.015508+0000","last_became_active":"2024-09-13T11:24:24.093492+0000","last_became_peered":"2024-09-13T11:24:24.093492+0000","last_unstale":"2024-09-13T11:25:21.015508+0000","last_undegraded":"2024-09-13T11:25:21.015508+0000","last_fullsized":"2024-09-13T11:25:21.015508+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'4744","last_scrub_stamp":"2024-09-13T11:24:24.616633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:24:24.616633+0000","objects_scrubbed":1624,"log_size":5667,"log_dups_size":0,"ondisk_log_size":5667,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:42.911074+0000","scrub_duration":415,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":111804416,"num_objects":1706,"num_object_clones":0,"num_object_copies":3412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1706,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":930,"num_write_kb":32384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,13],"acting":[15,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"3.3","version":"161'4854","reported_seq":5212,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-13T11:25:20.987507+0000","last_change":"2024-09-13T11:25:08.394545+0000","last_active":"2024-09-13T11:25:20.987507+0000","last_peered":"2024-09-13T11:25:20.987507+0000","last_clean":"2024-09-13T11:25:20.987507+0000","last_became_active":"2024-09-13T11:23:31.930448+0000","last_became_peered":"2024-09-13T11:23:31.930448+0000","last_unstale":"2024-09-13T11:25:20.987507+0000","last_undegraded":"2024-09-13T11:25:20.987507+0000","last_fullsized":"2024-09-13T11:25:20.987507+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"151'4587","last_scrub_stamp":"2024-09-13T11:25:08.394441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:08.394441+0000","objects_scrubbed":1942,"log_size":4854,"log_dups_size":0,"ondisk_log_size":4854,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:26:27.641900+0000","scrub_duration":219,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109576192,"num_objects":1672,"num_object_clones":0,"num_object_copies":3344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1672,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1732,"num_write_kb":80896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,7],"acting":[10,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"28'2","reported_seq":232,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-13T11:25:20.166058+0000","last_change":"2024-09-13T11:25:13.869564+0000","last_active":"2024-09-13T11:25:20.166058+0000","last_peered":"2024-09-13T11:25:20.166058+0000","last_clean":"2024-09-13T11:25:20.166058+0000","last_became_active":"2024-09-13T11:23:12.546348+0000","last_became_peered":"2024-09-13T11:23:12.546348+0000","last_unstale":"2024-09-13T11:25:20.166058+0000","last_undegraded":"2024-09-13T11:25:20.166058+0000","last_fullsized":"2024-09-13T11:25:20.166058+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2","last_scrub_stamp":"2024-09-13T11:25:13.869413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:25:13.869413+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:26:38.761718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0025826730000000002,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8],"acting":[2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"161'4873","reported_seq":5229,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-13T11:25:21.086865+0000","last_change":"2024-09-13T11:25:09.842875+0000","last_active":"2024-09-13T11:25:21.086865+0000","last_peered":"2024-09-13T11:25:21.086865+0000","last_clean":"2024-09-13T11:25:21.086865+0000","last_became_active":"2024-09-13T11:23:31.921388+0000","last_became_peered":"2024-09-13T11:23:31.921388+0000","last_unstale":"2024-09-13T11:25:21.086865+0000","last_undegraded":"2024-09-13T11:25:21.086865+0000","last_fullsized":"2024-09-13T11:25:21.086865+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"151'4644","last_scrub_stamp":"2024-09-13T11:25:09.842728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:09.842728+0000","objects_scrubbed":1944,"log_size":4873,"log_dups_size":0,"ondisk_log_size":4873,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:26:25.553642+0000","scrub_duration":395,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112394240,"num_objects":1715,"num_object_clones":0,"num_object_copies":3430,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1715,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1767,"num_write_kb":84032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,15],"acting":[12,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":135,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-13T11:25:20.170278+0000","last_change":"2024-09-13T11:24:29.096863+0000","last_active":"2024-09-13T11:25:20.170278+0000","last_peered":"2024-09-13T11:25:20.170278+0000","last_clean":"2024-09-13T11:25:20.170278+0000","last_became_active":"2024-09-13T11:24:29.096575+0000","last_became_peered":"2024-09-13T11:24:29.096575+0000","last_unstale":"2024-09-13T11:25:20.170278+0000","last_undegraded":"2024-09-13T11:25:20.170278+0000","last_fullsized":"2024-09-13T11:25:20.170278+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:24:22.064711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:24:22.064711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:34.178168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00081225399999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"162'4866","reported_seq":5227,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-13T11:25:22.016980+0000","last_change":"2024-09-13T11:25:00.861990+0000","last_active":"2024-09-13T11:25:22.016980+0000","last_peered":"2024-09-13T11:25:22.016980+0000","last_clean":"2024-09-13T11:25:22.016980+0000","last_became_active":"2024-09-13T11:23:31.926528+0000","last_became_peered":"2024-09-13T11:23:31.926528+0000","last_unstale":"2024-09-13T11:25:22.016980+0000","last_undegraded":"2024-09-13T11:25:22.016980+0000","last_fullsized":"2024-09-13T11:25:22.016980+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"146'4447","last_scrub_stamp":"2024-09-13T11:25:00.861834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:00.861834+0000","objects_scrubbed":2126,"log_size":4866,"log_dups_size":0,"ondisk_log_size":4866,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:26:15.411903+0000","scrub_duration":787,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":111673344,"num_objects":1704,"num_object_clones":0,"num_object_copies":3408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1704,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1796,"num_write_kb":85056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,11],"acting":[13,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":302,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-13T11:25:24.217757+0000","last_change":"2024-09-13T11:24:11.895017+0000","last_active":"2024-09-13T11:25:24.217757+0000","last_peered":"2024-09-13T11:25:24.217757+0000","last_clean":"2024-09-13T11:25:24.217757+0000","last_became_active":"2024-09-13T11:22:14.211435+0000","last_became_peered":"2024-09-13T11:22:14.211435+0000","last_unstale":"2024-09-13T11:25:24.217757+0000","last_undegraded":"2024-09-13T11:25:24.217757+0000","last_fullsized":"2024-09-13T11:25:24.217757+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:24:11.894937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:24:11.894937+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0010010360000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2","version":"161'4935","reported_seq":8731,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-13T11:25:21.085454+0000","last_change":"2024-09-13T11:24:46.503641+0000","last_active":"2024-09-13T11:25:21.085454+0000","last_peered":"2024-09-13T11:25:21.085454+0000","last_clean":"2024-09-13T11:25:21.085454+0000","last_became_active":"2024-09-13T11:24:29.175407+0000","last_became_peered":"2024-09-13T11:24:29.175407+0000","last_unstale":"2024-09-13T11:25:21.085454+0000","last_undegraded":"2024-09-13T11:25:21.085454+0000","last_fullsized":"2024-09-13T11:25:21.085454+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'3057","last_scrub_stamp":"2024-09-13T11:23:32.977559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:32.977559+0000","objects_scrubbed":841,"log_size":4935,"log_dups_size":0,"ondisk_log_size":4935,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":151,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":114360320,"num_objects":1745,"num_object_clones":0,"num_object_copies":3490,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1745,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1878,"num_write_kb":89024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1731,"num_bytes_recovered":113442816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,3],"acting":[12,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[]},{"pgid":"2.3","version":"26'1","reported_seq":97,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-13T11:25:20.172567+0000","last_change":"2024-09-13T11:24:29.113426+0000","last_active":"2024-09-13T11:25:20.172567+0000","last_peered":"2024-09-13T11:25:20.172567+0000","last_clean":"2024-09-13T11:25:20.172567+0000","last_became_active":"2024-09-13T11:24:29.110028+0000","last_became_peered":"2024-09-13T11:24:29.110028+0000","last_unstale":"2024-09-13T11:25:20.172567+0000","last_undegraded":"2024-09-13T11:25:20.172567+0000","last_fullsized":"2024-09-13T11:25:20.172567+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"26'1","last_scrub_stamp":"2024-09-13T11:24:12.081872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:24:12.081872+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:22.133933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,8],"acting":[15,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"1.0","version":"26'112","reported_seq":428,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-13T11:25:20.162571+0000","last_change":"2024-09-13T11:23:49.325249+0000","last_active":"2024-09-13T11:25:20.162571+0000","last_peered":"2024-09-13T11:25:20.162571+0000","last_clean":"2024-09-13T11:25:20.162571+0000","last_became_active":"2024-09-13T11:22:12.129721+0000","last_became_peered":"2024-09-13T11:22:12.129721+0000","last_unstale":"2024-09-13T11:25:20.162571+0000","last_undegraded":"2024-09-13T11:25:20.162571+0000","last_fullsized":"2024-09-13T11:25:20.162571+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"26'112","last_scrub_stamp":"2024-09-13T11:23:49.325175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:11.098388+0000","last_clean_scrub_stamp":"2024-09-13T11:23:49.325175+0000","objects_scrubbed":2,"log_size":112,"log_dups_size":0,"ondisk_log_size":112,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:22.575734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.8","version":"161'5671","reported_seq":6010,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-13T11:25:20.986355+0000","last_change":"2024-09-13T11:24:24.506212+0000","last_active":"2024-09-13T11:25:20.986355+0000","last_peered":"2024-09-13T11:25:20.986355+0000","last_clean":"2024-09-13T11:25:20.986355+0000","last_became_active":"2024-09-13T11:24:24.096220+0000","last_became_peered":"2024-09-13T11:24:24.096220+0000","last_unstale":"2024-09-13T11:25:20.986355+0000","last_undegraded":"2024-09-13T11:25:20.986355+0000","last_fullsized":"2024-09-13T11:25:20.986355+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'4749","last_scrub_stamp":"2024-09-13T11:24:24.506109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:24:24.506109+0000","objects_scrubbed":1690,"log_size":5671,"log_dups_size":0,"ondisk_log_size":5671,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:49.430042+0000","scrub_duration":215,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":114229248,"num_objects":1743,"num_object_clones":0,"num_object_copies":3486,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1743,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":923,"num_write_kb":31232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.9","version":"161'5726","reported_seq":6064,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-13T11:25:21.076731+0000","last_change":"2024-09-13T11:24:24.928964+0000","last_active":"2024-09-13T11:25:21.076731+0000","last_peered":"2024-09-13T11:25:21.076731+0000","last_clean":"2024-09-13T11:25:21.076731+0000","last_became_active":"2024-09-13T11:24:24.082504+0000","last_became_peered":"2024-09-13T11:24:24.082504+0000","last_unstale":"2024-09-13T11:25:21.076731+0000","last_undegraded":"2024-09-13T11:25:21.076731+0000","last_fullsized":"2024-09-13T11:25:21.076731+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'4799","last_scrub_stamp":"2024-09-13T11:24:24.928724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:24:24.928724+0000","objects_scrubbed":1705,"log_size":5726,"log_dups_size":0,"ondisk_log_size":5726,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:43.845801+0000","scrub_duration":255,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":114491392,"num_objects":1747,"num_object_clones":0,"num_object_copies":3494,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1747,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":930,"num_write_kb":31104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,2],"acting":[12,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[]},{"pgid":"3.a","version":"165'6176","reported_seq":6526,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-13T11:25:25.021479+0000","last_change":"2024-09-13T11:24:25.137871+0000","last_active":"2024-09-13T11:25:25.021479+0000","last_peered":"2024-09-13T11:25:25.021479+0000","last_clean":"2024-09-13T11:25:25.021479+0000","last_became_active":"2024-09-13T11:24:24.089253+0000","last_became_peered":"2024-09-13T11:24:24.089253+0000","last_unstale":"2024-09-13T11:25:25.021479+0000","last_undegraded":"2024-09-13T11:25:25.021479+0000","last_fullsized":"2024-09-13T11:25:25.021479+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'5157","last_scrub_stamp":"2024-09-13T11:24:25.137777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:24:25.137777+0000","objects_scrubbed":1777,"log_size":6176,"log_dups_size":0,"ondisk_log_size":6176,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:36.149013+0000","scrub_duration":223,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":106758144,"num_objects":1629,"num_object_clones":0,"num_object_copies":3258,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1629,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1022,"num_write_kb":27968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.b","version":"162'6059","reported_seq":6402,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-13T11:25:21.990278+0000","last_change":"2024-09-13T11:24:25.503864+0000","last_active":"2024-09-13T11:25:21.990278+0000","last_peered":"2024-09-13T11:25:21.990278+0000","last_clean":"2024-09-13T11:25:21.990278+0000","last_became_active":"2024-09-13T11:24:24.085684+0000","last_became_peered":"2024-09-13T11:24:24.085684+0000","last_unstale":"2024-09-13T11:25:21.990278+0000","last_undegraded":"2024-09-13T11:25:21.990278+0000","last_fullsized":"2024-09-13T11:25:21.990278+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'5112","last_scrub_stamp":"2024-09-13T11:24:25.503716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:24:25.503716+0000","objects_scrubbed":1718,"log_size":6059,"log_dups_size":0,"ondisk_log_size":6059,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:37.636639+0000","scrub_duration":395,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":113311744,"num_objects":1729,"num_object_clones":0,"num_object_copies":3458,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1729,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":947,"num_write_kb":30656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,1],"acting":[13,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"3.c","version":"162'6080","reported_seq":6423,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-13T11:25:21.979106+0000","last_change":"2024-09-13T11:24:24.595021+0000","last_active":"2024-09-13T11:25:21.979106+0000","last_peered":"2024-09-13T11:25:21.979106+0000","last_clean":"2024-09-13T11:25:21.979106+0000","last_became_active":"2024-09-13T11:24:24.090847+0000","last_became_peered":"2024-09-13T11:24:24.090847+0000","last_unstale":"2024-09-13T11:25:21.979106+0000","last_undegraded":"2024-09-13T11:25:21.979106+0000","last_fullsized":"2024-09-13T11:25:21.979106+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'5154","last_scrub_stamp":"2024-09-13T11:24:24.594848+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:24:24.594848+0000","objects_scrubbed":1720,"log_size":6080,"log_dups_size":0,"ondisk_log_size":6080,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:35.181590+0000","scrub_duration":343,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112001024,"num_objects":1709,"num_object_clones":0,"num_object_copies":3418,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1709,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":931,"num_write_kb":29440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,13],"acting":[11,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.d","version":"161'6052","reported_seq":6395,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-13T11:25:21.177675+0000","last_change":"2024-09-13T11:24:24.967444+0000","last_active":"2024-09-13T11:25:21.177675+0000","last_peered":"2024-09-13T11:25:21.177675+0000","last_clean":"2024-09-13T11:25:21.177675+0000","last_became_active":"2024-09-13T11:24:24.087224+0000","last_became_peered":"2024-09-13T11:24:24.087224+0000","last_unstale":"2024-09-13T11:25:21.177675+0000","last_undegraded":"2024-09-13T11:25:21.177675+0000","last_fullsized":"2024-09-13T11:25:21.177675+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'5110","last_scrub_stamp":"2024-09-13T11:24:24.967337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:24:24.967337+0000","objects_scrubbed":1725,"log_size":6052,"log_dups_size":0,"ondisk_log_size":6052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:53.620949+0000","scrub_duration":255,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":114425856,"num_objects":1746,"num_object_clones":0,"num_object_copies":3492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1746,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":945,"num_write_kb":30912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.e","version":"161'7073","reported_seq":12311,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-13T11:25:21.101573+0000","last_change":"2024-09-13T11:23:32.368856+0000","last_active":"2024-09-13T11:25:21.101573+0000","last_peered":"2024-09-13T11:25:21.101573+0000","last_clean":"2024-09-13T11:25:21.101573+0000","last_became_active":"2024-09-13T11:23:31.925869+0000","last_became_peered":"2024-09-13T11:23:31.925869+0000","last_unstale":"2024-09-13T11:25:21.101573+0000","last_undegraded":"2024-09-13T11:25:21.101573+0000","last_fullsized":"2024-09-13T11:25:21.101573+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'3527","last_scrub_stamp":"2024-09-13T11:23:32.368727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:32.368727+0000","objects_scrubbed":1774,"log_size":7073,"log_dups_size":0,"ondisk_log_size":7073,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":219,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":223543296,"num_objects":3411,"num_object_clones":0,"num_object_copies":6822,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3411,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3547,"num_write_kb":165888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"162'7116","reported_seq":7478,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-13T11:25:22.048153+0000","last_change":"2024-09-13T11:25:00.289940+0000","last_active":"2024-09-13T11:25:22.048153+0000","last_peered":"2024-09-13T11:25:22.048153+0000","last_clean":"2024-09-13T11:25:22.048153+0000","last_became_active":"2024-09-13T11:23:31.935239+0000","last_became_peered":"2024-09-13T11:23:31.935239+0000","last_unstale":"2024-09-13T11:25:22.048153+0000","last_undegraded":"2024-09-13T11:25:22.048153+0000","last_fullsized":"2024-09-13T11:25:22.048153+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"146'6154","last_scrub_stamp":"2024-09-13T11:25:00.289673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:00.289673+0000","objects_scrubbed":4394,"log_size":7116,"log_dups_size":0,"ondisk_log_size":7116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:26:24.274856+0000","scrub_duration":1187,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":223477760,"num_objects":3410,"num_object_clones":0,"num_object_copies":6820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3410,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3696,"num_write_kb":170560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,12],"acting":[11,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.10","version":"161'6745","reported_seq":11257,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-13T11:25:21.026164+0000","last_change":"2024-09-13T11:25:10.161564+0000","last_active":"2024-09-13T11:25:21.026164+0000","last_peered":"2024-09-13T11:25:21.026164+0000","last_clean":"2024-09-13T11:25:21.026164+0000","last_became_active":"2024-09-13T11:23:41.931183+0000","last_became_peered":"2024-09-13T11:23:41.931183+0000","last_unstale":"2024-09-13T11:25:21.026164+0000","last_undegraded":"2024-09-13T11:25:21.026164+0000","last_fullsized":"2024-09-13T11:25:21.026164+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":5,"last_scrub":"151'6298","last_scrub_stamp":"2024-09-13T11:25:10.161464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:10.161464+0000","objects_scrubbed":3964,"log_size":6745,"log_dups_size":0,"ondisk_log_size":6745,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:26:33.015478+0000","scrub_duration":607,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":229507072,"num_objects":3502,"num_object_clones":0,"num_object_copies":7004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3502,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3279,"num_write_kb":152000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.11","version":"161'6673","reported_seq":9916,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-13T11:25:21.041751+0000","last_change":"2024-09-13T11:23:46.102466+0000","last_active":"2024-09-13T11:25:21.041751+0000","last_peered":"2024-09-13T11:25:21.041751+0000","last_clean":"2024-09-13T11:25:21.041751+0000","last_became_active":"2024-09-13T11:23:39.063172+0000","last_became_peered":"2024-09-13T11:23:39.063172+0000","last_unstale":"2024-09-13T11:25:21.041751+0000","last_undegraded":"2024-09-13T11:25:21.041751+0000","last_fullsized":"2024-09-13T11:25:21.041751+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":5,"last_scrub":"88'3345","last_scrub_stamp":"2024-09-13T11:23:46.101973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:46.101973+0000","objects_scrubbed":2011,"log_size":6673,"log_dups_size":0,"ondisk_log_size":6673,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:22.342075+0000","scrub_duration":955,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":232062976,"num_objects":3541,"num_object_clones":0,"num_object_copies":7082,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3541,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3332,"num_write_kb":155584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11],"acting":[4,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.12","version":"161'6700","reported_seq":16788,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-13T11:25:21.177685+0000","last_change":"2024-09-13T11:25:19.637097+0000","last_active":"2024-09-13T11:25:21.177685+0000","last_peered":"2024-09-13T11:25:21.177685+0000","last_clean":"2024-09-13T11:25:21.177685+0000","last_became_active":"2024-09-13T11:24:29.268229+0000","last_became_peered":"2024-09-13T11:24:29.268229+0000","last_unstale":"2024-09-13T11:25:21.177685+0000","last_undegraded":"2024-09-13T11:25:21.177685+0000","last_fullsized":"2024-09-13T11:25:21.177685+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'3109","last_scrub_stamp":"2024-09-13T11:23:35.111320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:35.111320+0000","objects_scrubbed":1732,"log_size":6700,"log_dups_size":0,"ondisk_log_size":6700,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:25.006155+0000","scrub_duration":239,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":229834752,"num_objects":3507,"num_object_clones":0,"num_object_copies":7014,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3507,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3591,"num_write_kb":171712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3457,"num_bytes_recovered":226557952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.13","version":"162'6778","reported_seq":9999,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-13T11:25:22.046470+0000","last_change":"2024-09-13T11:25:01.153044+0000","last_active":"2024-09-13T11:25:22.046470+0000","last_peered":"2024-09-13T11:25:22.046470+0000","last_clean":"2024-09-13T11:25:22.046470+0000","last_became_active":"2024-09-13T11:23:41.082433+0000","last_became_peered":"2024-09-13T11:23:41.082433+0000","last_unstale":"2024-09-13T11:25:22.046470+0000","last_undegraded":"2024-09-13T11:25:22.046470+0000","last_fullsized":"2024-09-13T11:25:22.046470+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":5,"last_scrub":"147'5921","last_scrub_stamp":"2024-09-13T11:25:01.152971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:01.152971+0000","objects_scrubbed":4311,"log_size":6778,"log_dups_size":0,"ondisk_log_size":6778,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:26:07.027631+0000","scrub_duration":1011,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":223674368,"num_objects":3413,"num_object_clones":0,"num_object_copies":6826,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3413,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3382,"num_write_kb":154048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7],"acting":[11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":46,"stat_sum":{"num_bytes":7236911128,"num_objects":110428,"num_object_clones":0,"num_object_copies":220856,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4616,"num_objects_unfound":0,"num_objects_dirty":110428,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":86268,"num_write_kb":3634433,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19906,"num_bytes_recovered":1304559616,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16128548864,"data_stored":16128540720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":43282,"internal_metadata":0},"log_size":279652,"ondisk_log_size":279652,"up":92,"acting":92,"num_store_stats":16},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":11},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":112,"ondisk_log_size":112,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":13,"up_from":23,"seq":98784247850,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":886968,"kb_used_data":779432,"kb_used_omap":1,"kb_used_meta":107518,"kb_avail":93484872,"statfs":{"total":96636764160,"available":95728508928,"internally_reserved":0,"allocated":798138368,"data_stored":797420163,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":110098885},"hb_peers":[0,1,2,4,6,8,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":23,"seq":98784247849,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":736032,"kb_used_data":656272,"kb_used_omap":1,"kb_used_meta":79742,"kb_avail":93635808,"statfs":{"total":96636764160,"available":95883067392,"internally_reserved":0,"allocated":672022528,"data_stored":671332275,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":81656256},"hb_peers":[0,1,3,8,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":40,"seq":171798691872,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1249956,"kb_used_data":1149908,"kb_used_omap":4,"kb_used_meta":100027,"kb_avail":93121884,"statfs":{"total":96636764160,"available":95356809216,"internally_reserved":0,"allocated":1177505792,"data_stored":1176800520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4433,"internal_metadata":102428335},"hb_peers":[0,2,3,4,5,6,7,8,9,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":21,"seq":90194313258,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1218360,"kb_used_data":1119144,"kb_used_omap":1,"kb_used_meta":99198,"kb_avail":93153480,"statfs":{"total":96636764160,"available":95389163520,"internally_reserved":0,"allocated":1146003456,"data_stored":1145255743,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":101579207},"hb_peers":[0,2,4,5,7,9,10,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":21,"seq":90194313260,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1269500,"kb_used_data":1179884,"kb_used_omap":1,"kb_used_meta":89470,"kb_avail":93102340,"statfs":{"total":96636764160,"available":95336796160,"internally_reserved":0,"allocated":1208201216,"data_stored":1207481214,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":91617735},"hb_peers":[0,2,4,6,7,8,9,11,12,13],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":22,"seq":94489280552,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1359040,"kb_used_data":1259824,"kb_used_omap":1,"kb_used_meta":99198,"kb_avail":93012800,"statfs":{"total":96636764160,"available":95245107200,"internally_reserved":0,"allocated":1290059776,"data_stored":1289397131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":101579205},"hb_peers":[1,3,4,5,7,8,10,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":22,"seq":94489280553,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":637552,"kb_used_data":561952,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":93734288,"statfs":{"total":96636764160,"available":95983910912,"internally_reserved":0,"allocated":575438848,"data_stored":574734506,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,1,3,4,5,7,8,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":15,"up_from":22,"seq":94489280553,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1298296,"kb_used_data":1190760,"kb_used_omap":1,"kb_used_meta":107518,"kb_avail":93073544,"statfs":{"total":96636764160,"available":95307309056,"internally_reserved":0,"allocated":1219338240,"data_stored":1218683375,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":110098886},"hb_peers":[0,4,5,6,7,8,10,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":22,"seq":94489280554,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":863736,"kb_used_data":788136,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":93508104,"statfs":{"total":96636764160,"available":95752298496,"internally_reserved":0,"allocated":807051264,"data_stored":806340007,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[3,5,6,7,9,10,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":22,"seq":94489280553,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":616404,"kb_used_data":563012,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":93755436,"statfs":{"total":96636764160,"available":96005566464,"internally_reserved":0,"allocated":576524288,"data_stored":575844106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54655430},"hb_peers":[0,1,3,5,7,8,10,11,12,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":22,"seq":94489280553,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1556884,"kb_used_data":1445124,"kb_used_omap":1,"kb_used_meta":111678,"kb_avail":92814956,"statfs":{"total":96636764160,"available":95042514944,"internally_reserved":0,"allocated":1479806976,"data_stored":1479123475,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":114358730},"hb_peers":[0,3,5,6,8,9,10,11,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":22,"seq":94489280553,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":573576,"kb_used_data":506296,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":93798264,"statfs":{"total":96636764160,"available":96049422336,"internally_reserved":0,"allocated":518447104,"data_stored":517716093,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[1,5,6,7,9,10,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":22,"seq":94489280552,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1073248,"kb_used_data":1005968,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":93298592,"statfs":{"total":96636764160,"available":95537758208,"internally_reserved":0,"allocated":1030111232,"data_stored":1029419532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":68876736},"hb_peers":[0,1,3,5,7,9,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":11,"up_from":22,"seq":94489280554,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1513220,"kb_used_data":1405684,"kb_used_omap":1,"kb_used_meta":107518,"kb_avail":92858620,"statfs":{"total":96636764160,"available":95087226880,"internally_reserved":0,"allocated":1439420416,"data_stored":1438701096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":110098884},"hb_peers":[0,1,2,4,7,8,10,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":12,"up_from":22,"seq":94489280553,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1344956,"kb_used_data":1237420,"kb_used_omap":1,"kb_used_meta":107518,"kb_avail":93026884,"statfs":{"total":96636764160,"available":95259529216,"internally_reserved":0,"allocated":1267118080,"data_stored":1266399894,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":110098885},"hb_peers":[0,1,2,3,4,5,10,11,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":14,"up_from":0,"seq":609885356039,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1287585792,"data_stored":1287585792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1167720448,"data_stored":1167720448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":570429440,"data_stored":570425368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1139474432,"data_stored":1139474432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":798097408,"data_stored":798097408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1203109888,"data_stored":1203109888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":570163200,"data_stored":570163200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1473183744,"data_stored":1473183744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":513146880,"data_stored":513146880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":668794880,"data_stored":668794880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":1016266752,"data_stored":1016266752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":1436155904,"data_stored":1436155904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":1253769216,"data_stored":1253769216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":787087360,"data_stored":787087360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":1030029312,"data_stored":1030029312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":43282,"internal_metadata":0},{"poolid":3,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":1213534208,"data_stored":1213530136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T11:25:25.866 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-13T11:25:25.947 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-13T11:25:25.948 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [ 2024-09-13T11:25:25.948 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:25.948 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.4665.0:175734 3.6 e166/122)", 2024-09-13T11:25:25.948 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:25:25.947236+0000", 2024-09-13T11:25:25.948 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 9.7468999999999998e-05, 2024-09-13T11:25:25.948 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00013797500000000001, 2024-09-13T11:25:25.948 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:25:25.948 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:25:25.948 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:25:25.949 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:25:25.949 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:25.949 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:25:25.949 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:25.947236+0000", 2024-09-13T11:25:25.949 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:25.949 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:25.949 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:25.949 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:25:25.949 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:25.947236+0000", 2024-09-13T11:25:25.949 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:25.949 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:25.949 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:25.949 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:25:25.950 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:25.947237+0000", 2024-09-13T11:25:25.950 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5600000000000001e-06 2024-09-13T11:25:25.950 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:25.950 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:25.950 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:25:25.950 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:25.947244+0000", 2024-09-13T11:25:25.950 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.3409999999999997e-06 2024-09-13T11:25:25.950 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:25.950 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:25.950 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:25:25.950 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:25.947278+0000", 2024-09-13T11:25:25.950 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.3875999999999999e-05 2024-09-13T11:25:25.951 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:25.951 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:25.951 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:25:25.951 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:25.947285+0000", 2024-09-13T11:25:25.951 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.9090000000000003e-06 2024-09-13T11:25:25.951 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:25.951 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:25.951 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:25:25.951 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:25.947335+0000", 2024-09-13T11:25:25.951 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.9944999999999997e-05 2024-09-13T11:25:25.951 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:25.951 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:25.952 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:25:25.952 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:25.947350+0000", 2024-09-13T11:25:25.952 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4800000000000001e-05 2024-09-13T11:25:25.952 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:25.952 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:25:25.952 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:25.952 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:25.952 INFO:teuthology.orchestra.run.smithi162.stdout: ], 2024-09-13T11:25:25.952 INFO:teuthology.orchestra.run.smithi162.stdout: "num_ops": 1 2024-09-13T11:25:25.952 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-13T11:25:25.953 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-13T11:25:25.947+0000 7f6a68775640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26307) UID: 0 2024-09-13T11:25:25.964 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-13T11:25:26.010 INFO:tasks.ceph.osd.11.smithi177.stderr:2024-09-13T11:25:26.006+0000 7f24792c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26580) UID: 0 2024-09-13T11:25:26.099 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-13T11:25:26.099 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [], 2024-09-13T11:25:26.100 INFO:teuthology.orchestra.run.smithi162.stdout: "complaint_time": 30, 2024-09-13T11:25:26.100 INFO:teuthology.orchestra.run.smithi162.stdout: "num_blocked_ops": 0 2024-09-13T11:25:26.100 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-13T11:25:26.111 INFO:tasks.ceph.osd.12.smithi001.stderr:2024-09-13T11:25:26.107+0000 7f892c6e6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26314) UID: 0 2024-09-13T11:25:26.117 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-13T11:25:26.212 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-13T11:25:26.211+0000 7f6a68775640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26307) UID: 0 2024-09-13T11:25:26.225 INFO:teuthology.orchestra.run.smithi001.stderr:noscrub is set 2024-09-13T11:25:26.231 INFO:tasks.ceph.osd.9.smithi092.stderr:2024-09-13T11:25:26.225+0000 7ff59ddf9640 -1 osd.9 166 handle_osd_map injecting map crc failure 2024-09-13T11:25:26.258 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-13T11:25:26.258 INFO:teuthology.orchestra.run.smithi162.stdout: "size": 20, 2024-09-13T11:25:26.258 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 600, 2024-09-13T11:25:26.258 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [ 2024-09-13T11:25:26.258 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.259 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4257.0:80 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 262144~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-13T11:25:26.259 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:22:12.516065+0000", 2024-09-13T11:25:26.259 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 193.73984301499999, 2024-09-13T11:25:26.259 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.46461997999999999, 2024-09-13T11:25:26.259 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:25:26.259 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:25:26.259 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:26.259 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:25:26.259 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4257", 2024-09-13T11:25:26.259 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/1488131527", 2024-09-13T11:25:26.259 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 80 2024-09-13T11:25:26.259 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.259 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:25:26.260 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.260 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:25:26.260 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.516065+0000", 2024-09-13T11:25:26.260 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:26.260 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.260 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.260 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:25:26.260 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.516065+0000", 2024-09-13T11:25:26.260 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:26.260 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.260 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.260 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:25:26.260 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.516066+0000", 2024-09-13T11:25:26.261 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0869999999999999e-06 2024-09-13T11:25:26.261 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.261 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.261 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:25:26.261 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.516112+0000", 2024-09-13T11:25:26.261 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.5503000000000001e-05 2024-09-13T11:25:26.261 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.261 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.261 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:25:26.261 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.516128+0000", 2024-09-13T11:25:26.261 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5891000000000001e-05 2024-09-13T11:25:26.261 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.262 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.262 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:25:26.262 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.516138+0000", 2024-09-13T11:25:26.262 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.8120000000000002e-06 2024-09-13T11:25:26.262 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.262 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.262 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:25:26.262 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.520398+0000", 2024-09-13T11:25:26.262 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0042605450000000001 2024-09-13T11:25:26.262 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.262 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.262 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:25:26.262 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.520526+0000", 2024-09-13T11:25:26.262 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000128355 2024-09-13T11:25:26.263 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.263 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.263 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 0", 2024-09-13T11:25:26.263 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.520730+0000", 2024-09-13T11:25:26.263 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00020337799999999999 2024-09-13T11:25:26.263 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.263 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.263 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:25:26.263 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.522417+0000", 2024-09-13T11:25:26.263 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001687184 2024-09-13T11:25:26.263 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.263 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.263 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:25:26.264 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.980408+0000", 2024-09-13T11:25:26.264 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.45799068399999998 2024-09-13T11:25:26.264 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.264 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.264 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:25:26.264 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.980565+0000", 2024-09-13T11:25:26.264 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00015760899999999999 2024-09-13T11:25:26.264 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.264 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.264 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:25:26.264 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.980685+0000", 2024-09-13T11:25:26.264 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000119932 2024-09-13T11:25:26.264 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:26.264 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:25:26.265 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:26.265 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.265 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.265 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop_reply(client.4257.0:80 1.0 e24/23 ondisk, result = 0)", 2024-09-13T11:25:26.265 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:22:12.522938+0000", 2024-09-13T11:25:26.265 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 193.73296998199999, 2024-09-13T11:25:26.265 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.45776420600000001, 2024-09-13T11:25:26.265 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:25:26.265 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:25:26.265 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:25:26.265 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:25:26.265 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.265 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:25:26.266 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.522938+0000", 2024-09-13T11:25:26.266 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:26.266 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.266 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.266 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:25:26.266 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.522938+0000", 2024-09-13T11:25:26.266 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:26.266 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.266 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.266 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:25:26.266 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.522941+0000", 2024-09-13T11:25:26.266 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.0290000000000001e-06 2024-09-13T11:25:26.266 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.266 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.267 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:25:26.267 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.522950+0000", 2024-09-13T11:25:26.267 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.7499999999999992e-06 2024-09-13T11:25:26.267 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.267 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.267 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:25:26.267 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.980215+0000", 2024-09-13T11:25:26.267 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.45726523800000002 2024-09-13T11:25:26.267 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.267 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.267 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:25:26.267 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.980229+0000", 2024-09-13T11:25:26.267 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3549e-05 2024-09-13T11:25:26.267 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.268 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.268 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:25:26.268 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.980365+0000", 2024-09-13T11:25:26.268 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00013608700000000001 2024-09-13T11:25:26.268 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.268 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.268 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:25:26.268 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.980388+0000", 2024-09-13T11:25:26.268 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.3112000000000002e-05 2024-09-13T11:25:26.268 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.268 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.268 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:25:26.268 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.980702+0000", 2024-09-13T11:25:26.268 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00031444099999999999 2024-09-13T11:25:26.269 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:26.269 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:25:26.269 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:26.269 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.269 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.269 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:25416 3.8 3:1a1fbdd1:::benchmark_data_smithi177_31116_object25415:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e56)", 2024-09-13T11:25:26.269 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:04.512188+0000", 2024-09-13T11:25:26.269 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 141.74372001500001, 2024-09-13T11:25:26.269 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.210208697, 2024-09-13T11:25:26.269 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:25:26.269 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:25:26.269 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:26.269 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:25:26.270 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:25:26.270 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:26.270 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 25416 2024-09-13T11:25:26.270 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.270 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:25:26.270 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.270 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:25:26.270 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.512188+0000", 2024-09-13T11:25:26.270 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:26.270 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.270 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.270 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:25:26.270 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.512188+0000", 2024-09-13T11:25:26.272 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:26.272 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.272 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.272 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:25:26.272 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.512192+0000", 2024-09-13T11:25:26.272 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.2660000000000003e-06 2024-09-13T11:25:26.273 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.273 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.273 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:25:26.273 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.512247+0000", 2024-09-13T11:25:26.273 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.4889000000000003e-05 2024-09-13T11:25:26.273 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.273 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.273 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:25:26.273 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.512279+0000", 2024-09-13T11:25:26.273 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.1146000000000003e-05 2024-09-13T11:25:26.273 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.273 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.273 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:25:26.273 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.512297+0000", 2024-09-13T11:25:26.274 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.825e-05 2024-09-13T11:25:26.274 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.274 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.274 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:25:26.274 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.512377+0000", 2024-09-13T11:25:26.274 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.9875000000000004e-05 2024-09-13T11:25:26.274 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.274 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.274 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:25:26.274 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.512570+0000", 2024-09-13T11:25:26.274 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00019312100000000001 2024-09-13T11:25:26.274 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.274 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.275 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 1", 2024-09-13T11:25:26.275 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.512798+0000", 2024-09-13T11:25:26.275 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00022847899999999999 2024-09-13T11:25:26.275 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.275 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.275 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:25:26.275 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.514975+0000", 2024-09-13T11:25:26.276 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0021769979999999999 2024-09-13T11:25:26.276 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.276 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.276 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:25:26.276 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.722207+0000", 2024-09-13T11:25:26.276 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.20723217099999999 2024-09-13T11:25:26.276 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.276 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.276 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:25:26.276 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.722314+0000", 2024-09-13T11:25:26.276 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00010648400000000001 2024-09-13T11:25:26.276 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.276 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.276 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:25:26.277 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.722397+0000", 2024-09-13T11:25:26.277 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.3017999999999997e-05 2024-09-13T11:25:26.277 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:26.277 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:25:26.277 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:26.277 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.277 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.277 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop_reply(client.4665.0:25416 3.8 e56/40 ondisk, result = 0)", 2024-09-13T11:25:26.277 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:04.515635+0000", 2024-09-13T11:25:26.277 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 141.740273363, 2024-09-13T11:25:26.278 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.20677600500000001, 2024-09-13T11:25:26.278 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:25:26.278 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:25:26.278 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:25:26.279 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:25:26.279 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.279 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:25:26.279 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.515635+0000", 2024-09-13T11:25:26.279 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:26.279 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.279 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.279 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:25:26.279 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.515635+0000", 2024-09-13T11:25:26.279 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:26.280 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.280 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.280 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:25:26.280 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.515639+0000", 2024-09-13T11:25:26.280 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.619e-06 2024-09-13T11:25:26.280 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.280 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.280 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:25:26.280 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.515650+0000", 2024-09-13T11:25:26.280 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.111e-05 2024-09-13T11:25:26.280 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.280 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.281 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:25:26.281 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.722004+0000", 2024-09-13T11:25:26.281 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.206354805 2024-09-13T11:25:26.281 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.281 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.281 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:25:26.281 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.722017+0000", 2024-09-13T11:25:26.281 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2840000000000001e-05 2024-09-13T11:25:26.281 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.281 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.281 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:25:26.281 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.722186+0000", 2024-09-13T11:25:26.281 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00016898299999999999 2024-09-13T11:25:26.282 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.282 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.282 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:25:26.282 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.722199+0000", 2024-09-13T11:25:26.282 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.289e-05 2024-09-13T11:25:26.282 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.282 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.282 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:25:26.282 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.722411+0000", 2024-09-13T11:25:26.282 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00021175799999999999 2024-09-13T11:25:26.282 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:26.282 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:25:26.282 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:26.282 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.283 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.283 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:30833 3.8 3:1c203f8d:::benchmark_data_smithi177_31116_object30832:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e59)", 2024-09-13T11:25:26.283 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:09.506656+0000", 2024-09-13T11:25:26.283 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 136.74925246500001, 2024-09-13T11:25:26.283 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.224862282, 2024-09-13T11:25:26.283 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:25:26.283 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:25:26.283 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:26.283 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:25:26.283 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:25:26.283 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:26.283 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 30833 2024-09-13T11:25:26.283 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.284 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:25:26.284 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.284 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:25:26.284 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.506656+0000", 2024-09-13T11:25:26.284 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:26.284 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.284 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.284 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:25:26.284 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.506656+0000", 2024-09-13T11:25:26.284 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:26.284 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.284 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.284 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:25:26.285 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.506660+0000", 2024-09-13T11:25:26.285 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.6e-06 2024-09-13T11:25:26.285 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.285 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.285 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:25:26.285 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.506717+0000", 2024-09-13T11:25:26.285 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.6131999999999997e-05 2024-09-13T11:25:26.285 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.285 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.285 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:25:26.285 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.506751+0000", 2024-09-13T11:25:26.285 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.4746000000000002e-05 2024-09-13T11:25:26.285 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.285 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.286 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:25:26.286 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.506774+0000", 2024-09-13T11:25:26.286 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.2636000000000001e-05 2024-09-13T11:25:26.286 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.286 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.286 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:25:26.286 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.506878+0000", 2024-09-13T11:25:26.286 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000104057 2024-09-13T11:25:26.286 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.286 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.286 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:25:26.286 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.507052+0000", 2024-09-13T11:25:26.286 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00017402800000000001 2024-09-13T11:25:26.286 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.287 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.287 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 1", 2024-09-13T11:25:26.287 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.507217+0000", 2024-09-13T11:25:26.287 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00016516 2024-09-13T11:25:26.287 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.287 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.287 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:25:26.287 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.508992+0000", 2024-09-13T11:25:26.287 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0017743990000000001 2024-09-13T11:25:26.287 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.287 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.287 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:25:26.287 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.731429+0000", 2024-09-13T11:25:26.287 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.22243718000000001 2024-09-13T11:25:26.288 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.288 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.288 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:25:26.288 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.731479+0000", 2024-09-13T11:25:26.288 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.0146999999999999e-05 2024-09-13T11:25:26.288 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.288 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.288 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:25:26.288 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.731518+0000", 2024-09-13T11:25:26.288 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.9196999999999998e-05 2024-09-13T11:25:26.288 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:26.288 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:25:26.288 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:26.289 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.289 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.289 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop_reply(client.4665.0:30833 3.8 e59/40 ondisk, result = 0)", 2024-09-13T11:25:26.289 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:09.508907+0000", 2024-09-13T11:25:26.289 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 136.74700137100001, 2024-09-13T11:25:26.289 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.222618862, 2024-09-13T11:25:26.289 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:25:26.289 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:25:26.289 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:25:26.289 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:25:26.289 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.289 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:25:26.289 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.508907+0000", 2024-09-13T11:25:26.289 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:26.290 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.290 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.290 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:25:26.290 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.508907+0000", 2024-09-13T11:25:26.290 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:26.290 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.290 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.290 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:25:26.290 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.508910+0000", 2024-09-13T11:25:26.290 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.993e-06 2024-09-13T11:25:26.290 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.290 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.290 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:25:26.290 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.508918+0000", 2024-09-13T11:25:26.291 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.7000000000000008e-06 2024-09-13T11:25:26.291 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.291 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.291 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:25:26.291 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.731341+0000", 2024-09-13T11:25:26.291 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.22242308699999999 2024-09-13T11:25:26.291 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.291 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.291 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:25:26.291 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.731350+0000", 2024-09-13T11:25:26.291 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.3179999999999996e-06 2024-09-13T11:25:26.291 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.291 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.292 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:25:26.292 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.731418+0000", 2024-09-13T11:25:26.292 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.7586e-05 2024-09-13T11:25:26.292 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.292 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.292 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:25:26.292 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.731424+0000", 2024-09-13T11:25:26.292 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.844e-06 2024-09-13T11:25:26.292 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.292 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.293 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:25:26.293 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.731526+0000", 2024-09-13T11:25:26.293 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000101334 2024-09-13T11:25:26.293 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:26.293 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:25:26.293 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:26.293 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.293 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.293 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.4665.0:54351 3.10 e74/70 3:085f283e:::benchmark_data_smithi177_31116_object54350:head v 74'3087, mlcod=49'780)", 2024-09-13T11:25:26.293 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:30.599667+0000", 2024-09-13T11:25:26.293 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 115.656240865, 2024-09-13T11:25:26.293 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.16617266999999999, 2024-09-13T11:25:26.293 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:25:26.294 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:25:26.294 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:26.294 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:25:26.294 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.294 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:25:26.294 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.599667+0000", 2024-09-13T11:25:26.294 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:26.294 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.294 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.294 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:25:26.294 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.599667+0000", 2024-09-13T11:25:26.294 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:26.294 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.294 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.295 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:25:26.295 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.599669+0000", 2024-09-13T11:25:26.295 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3599999999999999e-06 2024-09-13T11:25:26.295 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.295 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.295 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:25:26.295 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.599712+0000", 2024-09-13T11:25:26.295 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.3723000000000001e-05 2024-09-13T11:25:26.295 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.295 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.295 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:25:26.295 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.762241+0000", 2024-09-13T11:25:26.295 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.16252863400000001 2024-09-13T11:25:26.295 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.296 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.296 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:25:26.296 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.762255+0000", 2024-09-13T11:25:26.296 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3414999999999999e-05 2024-09-13T11:25:26.296 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.296 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.296 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:25:26.296 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.762329+0000", 2024-09-13T11:25:26.296 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.4735999999999997e-05 2024-09-13T11:25:26.296 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.297 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.297 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:25:26.297 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.762358+0000", 2024-09-13T11:25:26.297 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.8308000000000001e-05 2024-09-13T11:25:26.297 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.297 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.297 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:25:26.297 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.765770+0000", 2024-09-13T11:25:26.297 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0034126590000000002 2024-09-13T11:25:26.297 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.297 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.297 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:25:26.298 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.765840+0000", 2024-09-13T11:25:26.298 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.9834999999999998e-05 2024-09-13T11:25:26.298 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:26.298 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:25:26.298 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:26.298 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.298 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.298 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPush(3.10 76/70 [PushOp(3:0e7d2700:::benchmark_data_smithi177_31116_object17406:head, version: 52'1075, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:0e7d2700:::benchmark_data_smithi177_31116_object17406:head@52'1075, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:25:26.298 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:32.206198+0000", 2024-09-13T11:25:26.298 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 114.049710662, 2024-09-13T11:25:26.298 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.25193395000000002, 2024-09-13T11:25:26.298 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:25:26.298 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:25:26.299 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:25:26.299 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:25:26.299 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.299 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:25:26.299 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.206198+0000", 2024-09-13T11:25:26.299 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:26.299 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.299 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.299 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:25:26.299 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.206198+0000", 2024-09-13T11:25:26.299 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:26.299 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.299 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.300 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:25:26.300 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.206199+0000", 2024-09-13T11:25:26.300 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.46e-06 2024-09-13T11:25:26.300 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.300 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.300 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:25:26.300 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.206250+0000", 2024-09-13T11:25:26.300 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.113e-05 2024-09-13T11:25:26.300 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.300 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.300 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:25:26.300 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.456042+0000", 2024-09-13T11:25:26.300 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.24979182999999999 2024-09-13T11:25:26.300 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.301 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.301 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:25:26.301 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.456084+0000", 2024-09-13T11:25:26.301 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.1814000000000003e-05 2024-09-13T11:25:26.301 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.301 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.301 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:25:26.301 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.456342+0000", 2024-09-13T11:25:26.301 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000258404 2024-09-13T11:25:26.301 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.301 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.301 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:25:26.301 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.456367+0000", 2024-09-13T11:25:26.301 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.5099000000000001e-05 2024-09-13T11:25:26.302 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.302 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.302 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:25:26.302 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.458132+0000", 2024-09-13T11:25:26.302 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0017642129999999999 2024-09-13T11:25:26.302 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:26.302 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:25:26.302 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:26.302 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.302 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.302 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPush(3.10 76/70 [PushOp(3:0eee8bcf:::benchmark_data_smithi177_31116_object17589:head, version: 53'1086, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:0eee8bcf:::benchmark_data_smithi177_31116_object17589:head@53'1086, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:25:26.302 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:32.585183+0000", 2024-09-13T11:25:26.302 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 113.670724768, 2024-09-13T11:25:26.302 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.16696202199999999, 2024-09-13T11:25:26.303 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:25:26.303 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:25:26.303 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:25:26.303 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:25:26.303 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.303 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:25:26.303 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.585183+0000", 2024-09-13T11:25:26.303 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:26.303 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.303 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.303 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:25:26.303 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.585183+0000", 2024-09-13T11:25:26.303 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:26.304 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.304 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.304 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:25:26.304 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.585187+0000", 2024-09-13T11:25:26.304 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.8480000000000003e-06 2024-09-13T11:25:26.304 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.304 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.304 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:25:26.304 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.585264+0000", 2024-09-13T11:25:26.304 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.6327000000000003e-05 2024-09-13T11:25:26.304 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.304 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.304 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:25:26.305 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.750459+0000", 2024-09-13T11:25:26.305 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.16519551900000001 2024-09-13T11:25:26.305 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.305 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.305 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:25:26.305 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.750480+0000", 2024-09-13T11:25:26.305 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.0984000000000001e-05 2024-09-13T11:25:26.305 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.305 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.305 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:25:26.305 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.750633+0000", 2024-09-13T11:25:26.305 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000153148 2024-09-13T11:25:26.305 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.305 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.306 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:25:26.306 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.750652+0000", 2024-09-13T11:25:26.306 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.8981e-05 2024-09-13T11:25:26.306 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.306 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.306 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:25:26.306 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.752145+0000", 2024-09-13T11:25:26.306 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001493215 2024-09-13T11:25:26.306 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:26.306 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:25:26.306 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:26.306 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.306 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.306 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:60417 3.10 3:092808b0:::benchmark_data_smithi177_31116_object60416:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e85)", 2024-09-13T11:25:26.307 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:42.173238+0000", 2024-09-13T11:25:26.307 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 104.08266975399999, 2024-09-13T11:25:26.307 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.38965994599999998, 2024-09-13T11:25:26.307 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:25:26.307 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:25:26.307 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:26.307 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:25:26.307 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:25:26.307 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:26.307 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 60417 2024-09-13T11:25:26.307 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.307 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:25:26.307 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.307 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:25:26.308 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.173238+0000", 2024-09-13T11:25:26.308 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:26.308 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.308 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.308 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:25:26.308 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.173238+0000", 2024-09-13T11:25:26.308 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:26.308 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.308 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.308 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:25:26.308 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.173241+0000", 2024-09-13T11:25:26.308 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.6819999999999999e-06 2024-09-13T11:25:26.308 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.309 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.309 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:25:26.309 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.173335+0000", 2024-09-13T11:25:26.309 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.3957999999999997e-05 2024-09-13T11:25:26.309 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.309 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.309 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:25:26.309 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.173358+0000", 2024-09-13T11:25:26.309 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.3023999999999998e-05 2024-09-13T11:25:26.309 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.309 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.309 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:25:26.309 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.173373+0000", 2024-09-13T11:25:26.309 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4867e-05 2024-09-13T11:25:26.310 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.310 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.310 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:25:26.310 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.173466+0000", 2024-09-13T11:25:26.310 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.2542999999999998e-05 2024-09-13T11:25:26.310 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.310 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.310 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:25:26.310 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.173754+0000", 2024-09-13T11:25:26.310 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000288657 2024-09-13T11:25:26.310 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.310 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.310 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 0", 2024-09-13T11:25:26.310 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.174066+0000", 2024-09-13T11:25:26.311 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00031170999999999998 2024-09-13T11:25:26.311 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.311 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.311 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:25:26.311 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.178245+0000", 2024-09-13T11:25:26.311 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00417864 2024-09-13T11:25:26.311 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.311 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.311 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:25:26.311 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.562617+0000", 2024-09-13T11:25:26.311 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.38437253199999999 2024-09-13T11:25:26.311 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.311 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.312 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:25:26.312 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.562782+0000", 2024-09-13T11:25:26.312 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00016499200000000001 2024-09-13T11:25:26.312 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.312 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.312 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:25:26.312 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.562898+0000", 2024-09-13T11:25:26.312 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000116341 2024-09-13T11:25:26.312 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:26.312 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:25:26.312 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:26.312 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.312 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.312 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:83193 3.8 3:10066d83:::benchmark_data_smithi177_31116_object83192:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106)", 2024-09-13T11:25:26.313 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:24:04.283923+0000", 2024-09-13T11:25:26.313 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 81.971985071000006, 2024-09-13T11:25:26.313 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.64775463499999997, 2024-09-13T11:25:26.313 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:25:26.313 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:25:26.313 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:26.313 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:25:26.313 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:25:26.313 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:26.313 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 83193 2024-09-13T11:25:26.313 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.313 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:25:26.313 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.313 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:25:26.314 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.283923+0000", 2024-09-13T11:25:26.314 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:26.314 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.314 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.314 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:25:26.314 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.283923+0000", 2024-09-13T11:25:26.314 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:26.314 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.314 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.314 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:25:26.314 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.283925+0000", 2024-09-13T11:25:26.314 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.9489999999999999e-06 2024-09-13T11:25:26.314 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.314 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.315 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:25:26.315 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.283960+0000", 2024-09-13T11:25:26.315 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.5063e-05 2024-09-13T11:25:26.315 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.315 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.315 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:25:26.315 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.283974+0000", 2024-09-13T11:25:26.315 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3546e-05 2024-09-13T11:25:26.315 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.315 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.315 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:25:26.315 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.283983+0000", 2024-09-13T11:25:26.315 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.2499999999999995e-06 2024-09-13T11:25:26.315 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.316 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.316 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:25:26.316 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.284061+0000", 2024-09-13T11:25:26.316 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.7946000000000002e-05 2024-09-13T11:25:26.316 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.316 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.316 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:25:26.316 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.284270+0000", 2024-09-13T11:25:26.316 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00020876799999999999 2024-09-13T11:25:26.316 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.316 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.316 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 1", 2024-09-13T11:25:26.316 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.284489+0000", 2024-09-13T11:25:26.317 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00021956800000000001 2024-09-13T11:25:26.317 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.317 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.317 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:25:26.317 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.286456+0000", 2024-09-13T11:25:26.317 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0019665809999999998 2024-09-13T11:25:26.317 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.317 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.317 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:25:26.317 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.931497+0000", 2024-09-13T11:25:26.317 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.64504147700000003 2024-09-13T11:25:26.320 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-13T11:25:26.321 INFO:teuthology.orchestra.run.smithi001.stderr:dumped all 2024-09-13T11:25:26.321 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.321 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.321 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:25:26.321 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.931589+0000", 2024-09-13T11:25:26.321 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.2041999999999997e-05 2024-09-13T11:25:26.321 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.321 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.321 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:25:26.322 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.931678+0000", 2024-09-13T11:25:26.322 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.8444999999999998e-05 2024-09-13T11:25:26.322 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:26.322 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:25:26.322 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:26.322 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.322 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.322 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:115060 3.8 3:11200ba9:::benchmark_data_smithi177_31116_object115059:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e125)", 2024-09-13T11:25:26.322 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:24:33.602178+0000", 2024-09-13T11:25:26.322 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 52.653730076999999, 2024-09-13T11:25:26.322 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.23668186799999999, 2024-09-13T11:25:26.322 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:25:26.322 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:25:26.322 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:26.323 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:25:26.323 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:25:26.323 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:26.323 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 115060 2024-09-13T11:25:26.323 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.323 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:25:26.323 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.323 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:25:26.323 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:33.602178+0000", 2024-09-13T11:25:26.323 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:26.323 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.323 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.323 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:25:26.324 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:33.602178+0000", 2024-09-13T11:25:26.324 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:26.324 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.324 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.324 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:25:26.324 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:33.602181+0000", 2024-09-13T11:25:26.324 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.0369999999999999e-06 2024-09-13T11:25:26.324 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.324 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.324 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:25:26.325 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:33.602217+0000", 2024-09-13T11:25:26.325 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.5837999999999998e-05 2024-09-13T11:25:26.325 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.325 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.325 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:25:26.325 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:33.602235+0000", 2024-09-13T11:25:26.325 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.7571000000000002e-05 2024-09-13T11:25:26.325 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.326 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.326 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:25:26.326 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:33.602245+0000", 2024-09-13T11:25:26.326 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0360000000000001e-05 2024-09-13T11:25:26.326 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.326 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.326 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:25:26.326 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:33.602312+0000", 2024-09-13T11:25:26.326 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.6989000000000005e-05 2024-09-13T11:25:26.326 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.326 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.327 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:25:26.327 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:33.602490+0000", 2024-09-13T11:25:26.327 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000178464 2024-09-13T11:25:26.327 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.327 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.327 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 1", 2024-09-13T11:25:26.327 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:33.602670+0000", 2024-09-13T11:25:26.327 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00018002799999999999 2024-09-13T11:25:26.327 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.327 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.327 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:25:26.327 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:33.604382+0000", 2024-09-13T11:25:26.327 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001711314 2024-09-13T11:25:26.328 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.328 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.328 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:25:26.328 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:33.838738+0000", 2024-09-13T11:25:26.328 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.23435634699999999 2024-09-13T11:25:26.329 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.329 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.329 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:25:26.329 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:33.838815+0000", 2024-09-13T11:25:26.329 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.7381999999999999e-05 2024-09-13T11:25:26.329 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.329 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.329 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:25:26.329 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:33.838860+0000", 2024-09-13T11:25:26.329 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.4537999999999999e-05 2024-09-13T11:25:26.329 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:26.329 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:25:26.330 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:26.330 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.330 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.330 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop_reply(client.4665.0:115060 3.8 e125/117 ondisk, result = 0)", 2024-09-13T11:25:26.330 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:24:33.604404+0000", 2024-09-13T11:25:26.330 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 52.651503818999998, 2024-09-13T11:25:26.330 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.23446235400000001, 2024-09-13T11:25:26.330 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:25:26.330 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:25:26.330 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:25:26.330 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:25:26.330 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.330 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:25:26.330 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:33.604404+0000", 2024-09-13T11:25:26.331 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:26.331 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.331 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.331 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:25:26.331 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:33.604404+0000", 2024-09-13T11:25:26.331 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:26.331 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.331 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.331 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:25:26.331 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:33.604406+0000", 2024-09-13T11:25:26.331 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.418e-06 2024-09-13T11:25:26.331 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.331 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.331 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:25:26.332 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:33.604411+0000", 2024-09-13T11:25:26.332 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.3519999999999997e-06 2024-09-13T11:25:26.332 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.332 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.332 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:25:26.332 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:33.838578+0000", 2024-09-13T11:25:26.332 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.23416656199999999 2024-09-13T11:25:26.332 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.332 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.332 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:25:26.332 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:33.838586+0000", 2024-09-13T11:25:26.332 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.9920000000000007e-06 2024-09-13T11:25:26.332 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.332 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.333 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:25:26.333 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:33.838655+0000", 2024-09-13T11:25:26.333 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.9463999999999998e-05 2024-09-13T11:25:26.333 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.333 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.333 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:25:26.333 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:33.838731+0000", 2024-09-13T11:25:26.333 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.5487000000000004e-05 2024-09-13T11:25:26.333 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.333 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.333 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:25:26.333 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:33.838867+0000", 2024-09-13T11:25:26.333 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000136079 2024-09-13T11:25:26.333 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:26.334 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:25:26.334 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:26.334 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.334 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.334 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPush(3.6 129/122 [PushOp(3:6261f3b6:::benchmark_data_smithi177_31116_object64090:head, version: 90'3486, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:6261f3b6:::benchmark_data_smithi177_31116_object64090:head@90'3486, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:25:26.334 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:24:38.641487+0000", 2024-09-13T11:25:26.334 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 47.614421407999998, 2024-09-13T11:25:26.334 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.209889098, 2024-09-13T11:25:26.334 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:25:26.334 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:25:26.334 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:25:26.334 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:25:26.334 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.334 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:25:26.334 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:38.641487+0000", 2024-09-13T11:25:26.335 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:26.335 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.335 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.335 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:25:26.336 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:38.641487+0000", 2024-09-13T11:25:26.336 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:26.336 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.336 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.336 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:25:26.336 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:38.641489+0000", 2024-09-13T11:25:26.336 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.0980000000000001e-06 2024-09-13T11:25:26.337 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.337 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.337 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:25:26.337 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:38.641560+0000", 2024-09-13T11:25:26.337 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.1404999999999998e-05 2024-09-13T11:25:26.337 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.337 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.337 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:25:26.337 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:38.850498+0000", 2024-09-13T11:25:26.338 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.20893800500000001 2024-09-13T11:25:26.338 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.338 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.338 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:25:26.338 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:38.850510+0000", 2024-09-13T11:25:26.338 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1762e-05 2024-09-13T11:25:26.338 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.339 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.339 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:25:26.339 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:38.850610+0000", 2024-09-13T11:25:26.339 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00010011200000000001 2024-09-13T11:25:26.339 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.340 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.340 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:25:26.341 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:38.850628+0000", 2024-09-13T11:25:26.341 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.7291000000000001e-05 2024-09-13T11:25:26.341 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.341 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.341 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:25:26.341 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:38.851376+0000", 2024-09-13T11:25:26.341 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00074842500000000005 2024-09-13T11:25:26.341 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:26.341 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:25:26.341 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:26.341 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.341 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.342 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.4665.0:126919 3.6 e130/122 3:634ce08e:::benchmark_data_smithi177_31116_object126918:head v 130'5137, mlcod=130'5134)", 2024-09-13T11:25:26.342 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:24:43.780120+0000", 2024-09-13T11:25:26.342 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 42.475788516999998, 2024-09-13T11:25:26.342 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.298086295, 2024-09-13T11:25:26.342 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:25:26.342 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:25:26.342 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:26.342 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:25:26.342 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.342 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:25:26.342 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:43.780120+0000", 2024-09-13T11:25:26.342 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:26.342 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.343 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.343 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:25:26.343 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:43.780120+0000", 2024-09-13T11:25:26.343 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:26.343 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.343 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.343 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:25:26.343 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:43.780123+0000", 2024-09-13T11:25:26.343 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.79e-06 2024-09-13T11:25:26.343 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.343 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.343 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:25:26.343 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:43.780160+0000", 2024-09-13T11:25:26.343 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.7945999999999999e-05 2024-09-13T11:25:26.344 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.344 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.344 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:25:26.344 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:44.075853+0000", 2024-09-13T11:25:26.344 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.29569234999999999 2024-09-13T11:25:26.344 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.344 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.344 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:25:26.344 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:44.075861+0000", 2024-09-13T11:25:26.344 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.9510000000000006e-06 2024-09-13T11:25:26.344 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.344 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.344 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:25:26.345 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:44.075924+0000", 2024-09-13T11:25:26.345 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.2946000000000003e-05 2024-09-13T11:25:26.345 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.345 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.345 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:25:26.345 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:44.075942+0000", 2024-09-13T11:25:26.345 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.8108e-05 2024-09-13T11:25:26.345 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.345 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.345 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:25:26.345 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:44.078164+0000", 2024-09-13T11:25:26.345 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.002222375 2024-09-13T11:25:26.346 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.346 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.346 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:25:26.346 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:44.078206+0000", 2024-09-13T11:25:26.347 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.1829000000000002e-05 2024-09-13T11:25:26.347 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:26.347 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:25:26.347 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:26.347 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.347 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.347 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:137473 3.10 3:094c0efb:::benchmark_data_smithi177_31116_object137472:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e139)", 2024-09-13T11:25:26.347 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:24:53.523319+0000", 2024-09-13T11:25:26.347 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 32.732589478000001, 2024-09-13T11:25:26.347 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.18297369299999999, 2024-09-13T11:25:26.347 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:25:26.347 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:25:26.347 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:26.347 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:25:26.348 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:25:26.348 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:26.348 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 137473 2024-09-13T11:25:26.348 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.348 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:25:26.348 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.348 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:25:26.348 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:53.523319+0000", 2024-09-13T11:25:26.348 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:26.348 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.348 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.348 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:25:26.348 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:53.523319+0000", 2024-09-13T11:25:26.349 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:26.349 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.349 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.349 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:25:26.349 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:53.523322+0000", 2024-09-13T11:25:26.349 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.2119999999999999e-06 2024-09-13T11:25:26.349 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.349 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.349 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:25:26.349 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:53.523370+0000", 2024-09-13T11:25:26.349 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.7788999999999999e-05 2024-09-13T11:25:26.349 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.349 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.349 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:25:26.349 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:53.523395+0000", 2024-09-13T11:25:26.350 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.5202e-05 2024-09-13T11:25:26.350 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.350 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.350 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:25:26.350 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:53.523409+0000", 2024-09-13T11:25:26.350 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4263999999999999e-05 2024-09-13T11:25:26.350 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.350 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.350 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:25:26.350 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:53.523477+0000", 2024-09-13T11:25:26.350 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.7578999999999998e-05 2024-09-13T11:25:26.350 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.350 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.350 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:25:26.351 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:53.523638+0000", 2024-09-13T11:25:26.351 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00016078300000000001 2024-09-13T11:25:26.351 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.351 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.351 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 0", 2024-09-13T11:25:26.351 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:53.523800+0000", 2024-09-13T11:25:26.351 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000162564 2024-09-13T11:25:26.351 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.351 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.351 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:25:26.351 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:53.526960+0000", 2024-09-13T11:25:26.351 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0031593950000000002 2024-09-13T11:25:26.351 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.351 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.351 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:25:26.352 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:53.706180+0000", 2024-09-13T11:25:26.352 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.17922080600000001 2024-09-13T11:25:26.352 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.352 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.352 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:25:26.352 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:53.706237+0000", 2024-09-13T11:25:26.352 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.6845000000000002e-05 2024-09-13T11:25:26.352 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.352 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.352 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:25:26.352 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:53.706292+0000", 2024-09-13T11:25:26.352 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.5254000000000003e-05 2024-09-13T11:25:26.352 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:26.352 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:25:26.353 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:26.353 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.353 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.353 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop_reply(client.4665.0:137473 3.10 e139/84 ondisk, result = 0)", 2024-09-13T11:25:26.353 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:24:53.527076+0000", 2024-09-13T11:25:26.353 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 32.728832650999998, 2024-09-13T11:25:26.353 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.17922728099999999, 2024-09-13T11:25:26.353 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:25:26.353 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:25:26.353 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:25:26.353 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:25:26.353 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.353 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:25:26.353 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:53.527076+0000", 2024-09-13T11:25:26.354 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:26.354 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.354 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.354 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:25:26.354 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:53.527076+0000", 2024-09-13T11:25:26.354 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:26.354 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.354 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.354 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:25:26.356 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:53.527078+0000", 2024-09-13T11:25:26.356 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.9000000000000002e-06 2024-09-13T11:25:26.356 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.356 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.356 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:25:26.356 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:53.527088+0000", 2024-09-13T11:25:26.356 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.7240000000000004e-06 2024-09-13T11:25:26.356 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.356 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.356 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:25:26.356 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:53.706095+0000", 2024-09-13T11:25:26.357 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.17900633899999999 2024-09-13T11:25:26.357 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.357 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.357 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:25:26.357 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:53.706101+0000", 2024-09-13T11:25:26.357 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.7730000000000002e-06 2024-09-13T11:25:26.357 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.357 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.357 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:25:26.357 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:53.706168+0000", 2024-09-13T11:25:26.357 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.6225999999999999e-05 2024-09-13T11:25:26.357 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.357 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.357 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:25:26.358 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:53.706175+0000", 2024-09-13T11:25:26.358 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.5560000000000002e-06 2024-09-13T11:25:26.358 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.358 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.358 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:25:26.358 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:53.706303+0000", 2024-09-13T11:25:26.358 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000127763 2024-09-13T11:25:26.358 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:26.358 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:25:26.358 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:26.358 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.358 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.358 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:137780 3.8 3:1047f268:::benchmark_data_smithi177_31116_object137779:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e139)", 2024-09-13T11:25:26.358 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:24:53.795212+0000", 2024-09-13T11:25:26.358 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 32.460696732999999, 2024-09-13T11:25:26.359 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.21530981399999999, 2024-09-13T11:25:26.359 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:25:26.359 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:25:26.359 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:26.359 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:25:26.359 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:25:26.359 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:26.359 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 137780 2024-09-13T11:25:26.359 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.359 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:25:26.359 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.359 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:25:26.359 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:53.795212+0000", 2024-09-13T11:25:26.359 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:26.360 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.360 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.360 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:25:26.360 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:53.795212+0000", 2024-09-13T11:25:26.360 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:26.360 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.360 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.360 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:25:26.360 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:53.795214+0000", 2024-09-13T11:25:26.360 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.481e-06 2024-09-13T11:25:26.360 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.360 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.360 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:25:26.360 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:53.795247+0000", 2024-09-13T11:25:26.361 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.3105999999999998e-05 2024-09-13T11:25:26.361 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.361 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.361 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:25:26.361 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:53.795258+0000", 2024-09-13T11:25:26.361 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0872e-05 2024-09-13T11:25:26.361 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.361 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.361 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:25:26.361 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:53.795265+0000", 2024-09-13T11:25:26.361 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.5729999999999996e-06 2024-09-13T11:25:26.361 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.361 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.361 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:25:26.362 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:53.795309+0000", 2024-09-13T11:25:26.362 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.4617999999999999e-05 2024-09-13T11:25:26.362 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.362 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.362 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:25:26.362 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:53.795397+0000", 2024-09-13T11:25:26.362 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.7863999999999998e-05 2024-09-13T11:25:26.362 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.362 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.362 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 1", 2024-09-13T11:25:26.362 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:53.795493+0000", 2024-09-13T11:25:26.362 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.6062000000000003e-05 2024-09-13T11:25:26.362 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.362 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.362 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:25:26.363 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:53.797245+0000", 2024-09-13T11:25:26.363 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0017521559999999999 2024-09-13T11:25:26.363 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.363 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.363 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:25:26.363 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:54.010409+0000", 2024-09-13T11:25:26.363 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.21316385299999999 2024-09-13T11:25:26.363 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.363 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.363 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:25:26.363 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:54.010474+0000", 2024-09-13T11:25:26.363 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.4998000000000003e-05 2024-09-13T11:25:26.363 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.364 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.364 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:25:26.364 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:54.010521+0000", 2024-09-13T11:25:26.364 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.7231000000000002e-05 2024-09-13T11:25:26.364 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:26.364 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:25:26.364 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:26.364 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.364 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.364 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop_reply(client.4665.0:137780 3.8 e139/117 ondisk, result = 0)", 2024-09-13T11:25:26.364 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:24:53.797821+0000", 2024-09-13T11:25:26.364 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 32.458086835000003, 2024-09-13T11:25:26.364 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.21270765599999999, 2024-09-13T11:25:26.365 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:25:26.365 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:25:26.365 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:25:26.365 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:25:26.365 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.365 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:25:26.365 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:53.797821+0000", 2024-09-13T11:25:26.365 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:26.365 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.365 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.365 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:25:26.365 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:53.797821+0000", 2024-09-13T11:25:26.365 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:26.366 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.366 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.366 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:25:26.366 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:53.797822+0000", 2024-09-13T11:25:26.366 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.012e-06 2024-09-13T11:25:26.366 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.366 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.366 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:25:26.366 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:53.797829+0000", 2024-09-13T11:25:26.366 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.8009999999999998e-06 2024-09-13T11:25:26.366 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.366 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.366 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:25:26.366 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:54.010334+0000", 2024-09-13T11:25:26.366 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.212504477 2024-09-13T11:25:26.367 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.367 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.367 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:25:26.367 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:54.010342+0000", 2024-09-13T11:25:26.367 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.6549999999999993e-06 2024-09-13T11:25:26.367 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.367 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.367 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:25:26.367 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:54.010394+0000", 2024-09-13T11:25:26.367 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.1875000000000001e-05 2024-09-13T11:25:26.367 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.367 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.367 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:25:26.367 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:54.010402+0000", 2024-09-13T11:25:26.368 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.6499999999999996e-06 2024-09-13T11:25:26.368 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.368 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.368 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:25:26.368 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:54.010529+0000", 2024-09-13T11:25:26.368 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000127186 2024-09-13T11:25:26.368 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:26.368 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:25:26.368 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:26.368 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.368 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.368 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.4665.0:148704 3.6 e149/122 3:60557309:::benchmark_data_smithi177_31116_object8401:head v 149'5494, mlcod=148'5491)", 2024-09-13T11:25:26.368 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:25:05.107526+0000", 2024-09-13T11:25:26.368 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 21.148382220999999, 2024-09-13T11:25:26.369 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.16494699299999999, 2024-09-13T11:25:26.369 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:25:26.369 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:25:26.369 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:26.369 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:25:26.369 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.369 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:25:26.369 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:05.107526+0000", 2024-09-13T11:25:26.369 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:26.369 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.369 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.369 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:25:26.369 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:05.107526+0000", 2024-09-13T11:25:26.369 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:26.370 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.370 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.370 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:25:26.370 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:05.107528+0000", 2024-09-13T11:25:26.370 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.1100000000000001e-06 2024-09-13T11:25:26.370 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.370 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.370 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:25:26.370 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:05.107538+0000", 2024-09-13T11:25:26.370 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.7070000000000004e-06 2024-09-13T11:25:26.370 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.370 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.370 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:25:26.370 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:05.270952+0000", 2024-09-13T11:25:26.371 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.16341366199999999 2024-09-13T11:25:26.371 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.371 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.371 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:25:26.371 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:05.270959+0000", 2024-09-13T11:25:26.371 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.9789999999999993e-06 2024-09-13T11:25:26.371 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.371 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.371 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:25:26.371 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:05.271024+0000", 2024-09-13T11:25:26.371 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.4381999999999994e-05 2024-09-13T11:25:26.371 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.371 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.371 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:25:26.372 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:05.271048+0000", 2024-09-13T11:25:26.372 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.4239999999999998e-05 2024-09-13T11:25:26.372 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.372 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.372 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:25:26.372 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:05.272416+0000", 2024-09-13T11:25:26.372 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0013679720000000001 2024-09-13T11:25:26.372 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.372 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.372 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:25:26.372 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:05.272473+0000", 2024-09-13T11:25:26.372 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.6941000000000003e-05 2024-09-13T11:25:26.372 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:26.372 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:25:26.373 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:26.373 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:26.373 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:25:26.373 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-13T11:25:26.373 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-09-13T11:25:26.374 INFO:teuthology.orchestra.run.smithi001.stdout:{"pg_ready":true,"pg_map":{"version":251,"stamp":"2024-09-13T11:25:26.228933+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":7007666763,"num_objects":106925,"num_object_clones":0,"num_object_copies":213850,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3138,"num_objects_unfound":0,"num_objects_dirty":106925,"num_whiteouts":0,"num_read":87,"num_read_kb":74,"num_write":89922,"num_write_kb":3637107,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20647,"num_bytes_recovered":1352990739,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":283274,"ondisk_log_size":283274,"up":110,"acting":110,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":123,"num_osds":15,"num_per_pool_osds":15,"num_per_pool_omap_osds":15,"kb":1415577600,"kb_used":15703904,"kb_used_data":14359280,"kb_used_omap":26,"kb_used_meta":1344293,"kb_avail":1399873696,"statfs":{"total":1449551462400,"available":1433470664704,"internally_reserved":0,"allocated":14703902720,"data_stored":14693139520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":26763,"internal_metadata":1376556917},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-256507904,"num_objects":-3914,"num_object_clones":0,"num_object_copies":-7828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-2388,"num_objects_unfound":0,"num_objects_dirty":-3914,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3914,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1194,"num_bytes_recovered":78249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.048216"},"pg_stats":[{"pgid":"3.2d","version":"165'6141","reported_seq":9113,"reported_epoch":165,"state":"active+recovering+remapped","last_fresh":"2024-09-13T11:25:25.101919+0000","last_change":"2024-09-13T11:25:19.656321+0000","last_active":"2024-09-13T11:25:25.101919+0000","last_peered":"2024-09-13T11:25:25.101919+0000","last_clean":"2024-09-13T11:24:24.974950+0000","last_became_active":"2024-09-13T11:24:27.227288+0000","last_became_peered":"2024-09-13T11:24:27.227288+0000","last_unstale":"2024-09-13T11:25:25.101919+0000","last_undegraded":"2024-09-13T11:25:25.101919+0000","last_fullsized":"2024-09-13T11:25:25.101919+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":6,"last_scrub":"76'3395","last_scrub_stamp":"2024-09-13T11:23:32.776747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:32.776747+0000","objects_scrubbed":1704,"log_size":6141,"log_dups_size":0,"ondisk_log_size":6141,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":179,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110985216,"num_objects":1694,"num_object_clones":0,"num_object_copies":3388,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1674,"num_objects_unfound":0,"num_objects_dirty":1694,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1904,"num_write_kb":87840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":882,"num_bytes_recovered":57802752,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[3,0],"avail_no_missing":["3","7"],"object_location_counts":[{"shards":"0,3","objects":857},{"shards":"3,7","objects":837}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2c","version":"161'6126","reported_seq":11448,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-13T11:25:21.049707+0000","last_change":"2024-09-13T11:24:59.456762+0000","last_active":"2024-09-13T11:25:21.049707+0000","last_peered":"2024-09-13T11:25:21.049707+0000","last_clean":"2024-09-13T11:25:21.049707+0000","last_became_active":"2024-09-13T11:24:59.021895+0000","last_became_peered":"2024-09-13T11:24:59.021895+0000","last_unstale":"2024-09-13T11:25:21.049707+0000","last_undegraded":"2024-09-13T11:25:21.049707+0000","last_fullsized":"2024-09-13T11:25:21.049707+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":6,"last_scrub":"145'5672","last_scrub_stamp":"2024-09-13T11:24:59.456653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:24:59.456653+0000","objects_scrubbed":2192,"log_size":6126,"log_dups_size":0,"ondisk_log_size":6126,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:26:20.877991+0000","scrub_duration":279,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":113901568,"num_objects":1738,"num_object_clones":0,"num_object_copies":3476,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1738,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":454,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,7],"acting":[10,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.2b","version":"166'6105","reported_seq":9794,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-13T11:25:25.244306+0000","last_change":"2024-09-13T11:24:45.271477+0000","last_active":"2024-09-13T11:25:25.244306+0000","last_peered":"2024-09-13T11:25:25.244306+0000","last_clean":"2024-09-13T11:25:25.244306+0000","last_became_active":"2024-09-13T11:24:26.180578+0000","last_became_peered":"2024-09-13T11:24:26.180578+0000","last_unstale":"2024-09-13T11:25:25.244306+0000","last_undegraded":"2024-09-13T11:25:25.244306+0000","last_fullsized":"2024-09-13T11:25:25.244306+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":6,"last_scrub":"131'5382","last_scrub_stamp":"2024-09-13T11:24:45.271376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:24:45.271376+0000","objects_scrubbed":1940,"log_size":6105,"log_dups_size":0,"ondisk_log_size":6105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:59.210895+0000","scrub_duration":279,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":101515264,"num_objects":1549,"num_object_clones":0,"num_object_copies":3098,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1549,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":723,"num_write_kb":10624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2a","version":"161'6119","reported_seq":9985,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-13T11:25:21.063315+0000","last_change":"2024-09-13T11:24:46.625117+0000","last_active":"2024-09-13T11:25:21.063315+0000","last_peered":"2024-09-13T11:25:21.063315+0000","last_clean":"2024-09-13T11:25:21.063315+0000","last_became_active":"2024-09-13T11:24:45.320540+0000","last_became_peered":"2024-09-13T11:24:45.320540+0000","last_unstale":"2024-09-13T11:25:21.063315+0000","last_undegraded":"2024-09-13T11:25:21.063315+0000","last_fullsized":"2024-09-13T11:25:21.063315+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":6,"last_scrub":"133'5491","last_scrub_stamp":"2024-09-13T11:24:46.624970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:24:46.624970+0000","objects_scrubbed":2094,"log_size":6119,"log_dups_size":0,"ondisk_log_size":6119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:26:04.813280+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":118882304,"num_objects":1814,"num_object_clones":0,"num_object_copies":3628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1814,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":630,"num_write_kb":11200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,7],"acting":[10,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.29","version":"166'5881","reported_seq":11572,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-13T11:25:25.960499+0000","last_change":"2024-09-13T11:25:13.333062+0000","last_active":"2024-09-13T11:25:25.960499+0000","last_peered":"2024-09-13T11:25:25.960499+0000","last_clean":"2024-09-13T11:25:25.960499+0000","last_became_active":"2024-09-13T11:25:12.944028+0000","last_became_peered":"2024-09-13T11:25:12.944028+0000","last_unstale":"2024-09-13T11:25:25.960499+0000","last_undegraded":"2024-09-13T11:25:25.960499+0000","last_fullsized":"2024-09-13T11:25:25.960499+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":6,"last_scrub":"154'5579","last_scrub_stamp":"2024-09-13T11:25:13.332959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:13.332959+0000","objects_scrubbed":1937,"log_size":5881,"log_dups_size":0,"ondisk_log_size":5881,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:26:24.089228+0000","scrub_duration":243,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":107020288,"num_objects":1633,"num_object_clones":0,"num_object_copies":3266,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1633,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":304,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.28","version":"166'5729","reported_seq":9507,"reported_epoch":166,"state":"active+recovering+remapped","last_fresh":"2024-09-13T11:25:25.386902+0000","last_change":"2024-09-13T11:25:11.182075+0000","last_active":"2024-09-13T11:25:25.386902+0000","last_peered":"2024-09-13T11:25:25.386902+0000","last_clean":"2024-09-13T11:24:25.005266+0000","last_became_active":"2024-09-13T11:24:27.218508+0000","last_became_peered":"2024-09-13T11:24:27.218508+0000","last_unstale":"2024-09-13T11:25:25.386902+0000","last_undegraded":"2024-09-13T11:25:25.386902+0000","last_fullsized":"2024-09-13T11:25:25.386902+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":6,"last_scrub":"68'2771","last_scrub_stamp":"2024-09-13T11:23:20.644747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:20.644747+0000","objects_scrubbed":1356,"log_size":5729,"log_dups_size":0,"ondisk_log_size":5729,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":411,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":107872256,"num_objects":1646,"num_object_clones":0,"num_object_copies":3292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1072,"num_objects_unfound":0,"num_objects_dirty":1646,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1982,"num_write_kb":94400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1150,"num_bytes_recovered":75366400,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5],"acting":[1,5],"avail_no_missing":["1","6"],"object_location_counts":[{"shards":"1,5","objects":1110},{"shards":"1,6","objects":536}],"blocked_by":[],"up_primary":8,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.27","version":"166'5812","reported_seq":10730,"reported_epoch":166,"state":"active+recovering+remapped","last_fresh":"2024-09-13T11:25:26.113633+0000","last_change":"2024-09-13T11:25:08.115804+0000","last_active":"2024-09-13T11:25:26.113633+0000","last_peered":"2024-09-13T11:25:26.113633+0000","last_clean":"2024-09-13T11:24:24.942332+0000","last_became_active":"2024-09-13T11:24:27.215734+0000","last_became_peered":"2024-09-13T11:24:27.215734+0000","last_unstale":"2024-09-13T11:25:26.113633+0000","last_undegraded":"2024-09-13T11:25:26.113633+0000","last_fullsized":"2024-09-13T11:25:26.113633+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":6,"last_scrub":"68'2736","last_scrub_stamp":"2024-09-13T11:23:20.443329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:20.443329+0000","objects_scrubbed":99,"log_size":5812,"log_dups_size":0,"ondisk_log_size":5812,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":183,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":108920832,"num_objects":1662,"num_object_clones":0,"num_object_copies":3324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":392,"num_objects_unfound":0,"num_objects_dirty":1662,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2090,"num_write_kb":98144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1529,"num_bytes_recovered":100204544,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,11],"acting":[8,5],"avail_no_missing":["8","13"],"object_location_counts":[{"shards":"5,8","objects":1466},{"shards":"8,13","objects":196}],"blocked_by":[],"up_primary":5,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.26","version":"166'5927","reported_seq":11498,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-13T11:25:26.025249+0000","last_change":"2024-09-13T11:25:01.697082+0000","last_active":"2024-09-13T11:25:26.025249+0000","last_peered":"2024-09-13T11:25:26.025249+0000","last_clean":"2024-09-13T11:25:26.025249+0000","last_became_active":"2024-09-13T11:24:54.976919+0000","last_became_peered":"2024-09-13T11:24:54.976919+0000","last_unstale":"2024-09-13T11:25:26.025249+0000","last_undegraded":"2024-09-13T11:25:26.025249+0000","last_fullsized":"2024-09-13T11:25:26.025249+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":6,"last_scrub":"147'5433","last_scrub_stamp":"2024-09-13T11:25:01.696886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:01.696886+0000","objects_scrubbed":2158,"log_size":5927,"log_dups_size":0,"ondisk_log_size":5927,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:26:22.070621+0000","scrub_duration":403,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":108527616,"num_objects":1656,"num_object_clones":0,"num_object_copies":3312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1656,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":502,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,12],"acting":[8,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.25","version":"166'5781","reported_seq":11511,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-13T11:25:26.095773+0000","last_change":"2024-09-13T11:24:57.073162+0000","last_active":"2024-09-13T11:25:26.095773+0000","last_peered":"2024-09-13T11:25:26.095773+0000","last_clean":"2024-09-13T11:25:26.095773+0000","last_became_active":"2024-09-13T11:24:55.999019+0000","last_became_peered":"2024-09-13T11:24:55.999019+0000","last_unstale":"2024-09-13T11:25:26.095773+0000","last_undegraded":"2024-09-13T11:25:26.095773+0000","last_fullsized":"2024-09-13T11:25:26.095773+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":6,"last_scrub":"142'5214","last_scrub_stamp":"2024-09-13T11:24:57.073061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:24:57.073061+0000","objects_scrubbed":2249,"log_size":5781,"log_dups_size":0,"ondisk_log_size":5781,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:26:12.348519+0000","scrub_duration":571,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110428160,"num_objects":1685,"num_object_clones":0,"num_object_copies":3370,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1685,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":570,"num_write_kb":192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,2],"acting":[12,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[]},{"pgid":"3.24","version":"162'5742","reported_seq":9650,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-13T11:25:21.944050+0000","last_change":"2024-09-13T11:24:48.048846+0000","last_active":"2024-09-13T11:25:21.944050+0000","last_peered":"2024-09-13T11:25:21.944050+0000","last_clean":"2024-09-13T11:25:21.944050+0000","last_became_active":"2024-09-13T11:24:46.936994+0000","last_became_peered":"2024-09-13T11:24:46.936994+0000","last_unstale":"2024-09-13T11:25:21.944050+0000","last_undegraded":"2024-09-13T11:25:21.944050+0000","last_fullsized":"2024-09-13T11:25:21.944050+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":6,"last_scrub":"135'5093","last_scrub_stamp":"2024-09-13T11:24:48.048764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:24:48.048764+0000","objects_scrubbed":2102,"log_size":5742,"log_dups_size":0,"ondisk_log_size":5742,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:53.485370+0000","scrub_duration":287,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":114163712,"num_objects":1742,"num_object_clones":0,"num_object_copies":3484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1742,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":650,"num_write_kb":9280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1],"acting":[9,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.23","version":"166'5040","reported_seq":7891,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-13T11:25:25.955199+0000","last_change":"2024-09-13T11:23:48.133184+0000","last_active":"2024-09-13T11:25:25.955199+0000","last_peered":"2024-09-13T11:25:25.955199+0000","last_clean":"2024-09-13T11:25:25.955199+0000","last_became_active":"2024-09-13T11:23:47.926677+0000","last_became_peered":"2024-09-13T11:23:47.926677+0000","last_unstale":"2024-09-13T11:25:25.955199+0000","last_undegraded":"2024-09-13T11:25:25.955199+0000","last_fullsized":"2024-09-13T11:25:25.955199+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":6,"last_scrub":"91'3278","last_scrub_stamp":"2024-09-13T11:23:48.133084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:48.133084+0000","objects_scrubbed":1029,"log_size":5040,"log_dups_size":0,"ondisk_log_size":5040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":139,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":111804416,"num_objects":1706,"num_object_clones":0,"num_object_copies":3412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1706,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1763,"num_write_kb":78080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9],"acting":[5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.22","version":"166'4987","reported_seq":10567,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-13T11:25:26.093268+0000","last_change":"2024-09-13T11:24:45.485711+0000","last_active":"2024-09-13T11:25:26.093268+0000","last_peered":"2024-09-13T11:25:26.093268+0000","last_clean":"2024-09-13T11:25:26.093268+0000","last_became_active":"2024-09-13T11:24:29.253848+0000","last_became_peered":"2024-09-13T11:24:29.253848+0000","last_unstale":"2024-09-13T11:25:26.093268+0000","last_undegraded":"2024-09-13T11:25:26.093268+0000","last_fullsized":"2024-09-13T11:25:26.093268+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":6,"last_scrub":"90'3246","last_scrub_stamp":"2024-09-13T11:23:47.300339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:47.300339+0000","objects_scrubbed":1033,"log_size":4987,"log_dups_size":0,"ondisk_log_size":4987,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":203,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":108527616,"num_objects":1656,"num_object_clones":0,"num_object_copies":3312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1656,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1743,"num_write_kb":75712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1768,"num_bytes_recovered":115867648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.21","version":"166'4903","reported_seq":7769,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-13T11:25:26.085283+0000","last_change":"2024-09-13T11:24:12.564117+0000","last_active":"2024-09-13T11:25:26.085283+0000","last_peered":"2024-09-13T11:25:26.085283+0000","last_clean":"2024-09-13T11:25:26.085283+0000","last_became_active":"2024-09-13T11:24:01.250308+0000","last_became_peered":"2024-09-13T11:24:01.250308+0000","last_unstale":"2024-09-13T11:25:26.085283+0000","last_undegraded":"2024-09-13T11:25:26.085283+0000","last_fullsized":"2024-09-13T11:25:26.085283+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":6,"last_scrub":"112'3670","last_scrub_stamp":"2024-09-13T11:24:12.564031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:24:12.564031+0000","objects_scrubbed":1431,"log_size":4903,"log_dups_size":0,"ondisk_log_size":4903,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:36.766299+0000","scrub_duration":183,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":103350272,"num_objects":1577,"num_object_clones":0,"num_object_copies":3154,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1577,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1236,"num_write_kb":44224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8],"acting":[0,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.20","version":"166'5035","reported_seq":7188,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-13T11:25:25.998756+0000","last_change":"2024-09-13T11:23:47.969503+0000","last_active":"2024-09-13T11:25:25.998756+0000","last_peered":"2024-09-13T11:25:25.998756+0000","last_clean":"2024-09-13T11:25:25.998756+0000","last_became_active":"2024-09-13T11:23:33.954946+0000","last_became_peered":"2024-09-13T11:23:33.954946+0000","last_unstale":"2024-09-13T11:25:25.998756+0000","last_undegraded":"2024-09-13T11:25:25.998756+0000","last_fullsized":"2024-09-13T11:25:25.998756+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":6,"last_scrub":"91'3270","last_scrub_stamp":"2024-09-13T11:23:47.969372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:47.969372+0000","objects_scrubbed":1061,"log_size":5035,"log_dups_size":0,"ondisk_log_size":5035,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1327,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":106627072,"num_objects":1627,"num_object_clones":0,"num_object_copies":3254,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1627,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1770,"num_write_kb":74752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,0],"acting":[12,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[]},{"pgid":"3.1f","version":"161'6881","reported_seq":18822,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-13T11:25:21.059193+0000","last_change":"2024-09-13T11:25:20.773752+0000","last_active":"2024-09-13T11:25:21.059193+0000","last_peered":"2024-09-13T11:25:21.059193+0000","last_clean":"2024-09-13T11:25:21.059193+0000","last_became_active":"2024-09-13T11:24:29.290326+0000","last_became_peered":"2024-09-13T11:24:29.290326+0000","last_unstale":"2024-09-13T11:25:21.059193+0000","last_undegraded":"2024-09-13T11:25:21.059193+0000","last_fullsized":"2024-09-13T11:25:21.059193+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":5,"last_scrub":"161'6865","last_scrub_stamp":"2024-09-13T11:25:20.773641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:20.773641+0000","objects_scrubbed":3425,"log_size":6881,"log_dups_size":0,"ondisk_log_size":6881,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:26:21.317280+0000","scrub_duration":535,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":223084544,"num_objects":3404,"num_object_clones":0,"num_object_copies":6808,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3404,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2461,"num_write_kb":102848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3306,"num_bytes_recovered":216662016,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5],"acting":[10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1e","version":"166'7310","reported_seq":17827,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-13T11:25:25.982804+0000","last_change":"2024-09-13T11:24:08.338485+0000","last_active":"2024-09-13T11:25:25.982804+0000","last_peered":"2024-09-13T11:25:25.982804+0000","last_clean":"2024-09-13T11:25:25.982804+0000","last_became_active":"2024-09-13T11:24:01.261448+0000","last_became_peered":"2024-09-13T11:24:01.261448+0000","last_unstale":"2024-09-13T11:25:25.982804+0000","last_undegraded":"2024-09-13T11:25:25.982804+0000","last_fullsized":"2024-09-13T11:25:25.982804+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":5,"last_scrub":"110'4537","last_scrub_stamp":"2024-09-13T11:24:08.338354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:24:08.338354+0000","objects_scrubbed":2760,"log_size":7310,"log_dups_size":0,"ondisk_log_size":7310,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":367,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":216137728,"num_objects":3298,"num_object_clones":0,"num_object_copies":6596,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3298,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2782,"num_write_kb":106240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"166'7123","reported_seq":12569,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-13T11:25:26.004350+0000","last_change":"2024-09-13T11:24:12.492592+0000","last_active":"2024-09-13T11:25:26.004350+0000","last_peered":"2024-09-13T11:25:26.004350+0000","last_clean":"2024-09-13T11:25:26.004350+0000","last_became_active":"2024-09-13T11:24:01.258225+0000","last_became_peered":"2024-09-13T11:24:01.258225+0000","last_unstale":"2024-09-13T11:25:26.004350+0000","last_undegraded":"2024-09-13T11:25:26.004350+0000","last_fullsized":"2024-09-13T11:25:26.004350+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":5,"last_scrub":"112'4579","last_scrub_stamp":"2024-09-13T11:24:12.492507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:24:12.492507+0000","objects_scrubbed":2875,"log_size":7123,"log_dups_size":0,"ondisk_log_size":7123,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:34.926575+0000","scrub_duration":375,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":214630424,"num_objects":3276,"num_object_clones":0,"num_object_copies":6552,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3276,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2557,"num_write_kb":94593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,15],"acting":[2,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1c","version":"162'7099","reported_seq":10894,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-13T11:25:22.041539+0000","last_change":"2024-09-13T11:24:00.529312+0000","last_active":"2024-09-13T11:25:22.041539+0000","last_peered":"2024-09-13T11:25:22.041539+0000","last_clean":"2024-09-13T11:25:22.041539+0000","last_became_active":"2024-09-13T11:23:33.989492+0000","last_became_peered":"2024-09-13T11:23:33.989492+0000","last_unstale":"2024-09-13T11:25:22.041539+0000","last_undegraded":"2024-09-13T11:25:22.041539+0000","last_fullsized":"2024-09-13T11:25:22.041539+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":5,"last_scrub":"102'4277","last_scrub_stamp":"2024-09-13T11:24:00.529243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:24:00.529243+0000","objects_scrubbed":2544,"log_size":7099,"log_dups_size":0,"ondisk_log_size":7099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":259,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":226754560,"num_objects":3460,"num_object_clones":0,"num_object_copies":6920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3460,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2824,"num_write_kb":119680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7],"acting":[11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1b","version":"166'7246","reported_seq":12871,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-13T11:25:26.099182+0000","last_change":"2024-09-13T11:24:12.392093+0000","last_active":"2024-09-13T11:25:26.099182+0000","last_peered":"2024-09-13T11:25:26.099182+0000","last_clean":"2024-09-13T11:25:26.099182+0000","last_became_active":"2024-09-13T11:24:04.950821+0000","last_became_peered":"2024-09-13T11:24:04.950821+0000","last_unstale":"2024-09-13T11:25:26.099182+0000","last_undegraded":"2024-09-13T11:25:26.099182+0000","last_fullsized":"2024-09-13T11:25:26.099182+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":5,"last_scrub":"112'4665","last_scrub_stamp":"2024-09-13T11:24:12.391890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:24:12.391890+0000","objects_scrubbed":2934,"log_size":7246,"log_dups_size":0,"ondisk_log_size":7246,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:41.846147+0000","scrub_duration":675,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":215613440,"num_objects":3290,"num_object_clones":0,"num_object_copies":6580,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3290,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2602,"num_write_kb":94656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,15],"acting":[12,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[]},{"pgid":"3.1a","version":"166'7125","reported_seq":12411,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-13T11:25:25.999241+0000","last_change":"2024-09-13T11:24:09.556174+0000","last_active":"2024-09-13T11:25:25.999241+0000","last_peered":"2024-09-13T11:25:25.999241+0000","last_clean":"2024-09-13T11:25:25.999241+0000","last_became_active":"2024-09-13T11:24:03.935185+0000","last_became_peered":"2024-09-13T11:24:03.935185+0000","last_unstale":"2024-09-13T11:25:25.999241+0000","last_undegraded":"2024-09-13T11:25:25.999241+0000","last_fullsized":"2024-09-13T11:25:25.999241+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":5,"last_scrub":"110'4433","last_scrub_stamp":"2024-09-13T11:24:09.556022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:24:09.556022+0000","objects_scrubbed":2654,"log_size":7125,"log_dups_size":0,"ondisk_log_size":7125,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:28.469324+0000","scrub_duration":387,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":210698240,"num_objects":3215,"num_object_clones":0,"num_object_copies":6430,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3215,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2703,"num_write_kb":104448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9],"acting":[4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.19","version":"162'6603","reported_seq":11067,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-13T11:25:22.000325+0000","last_change":"2024-09-13T11:25:10.784789+0000","last_active":"2024-09-13T11:25:22.000325+0000","last_peered":"2024-09-13T11:25:22.000325+0000","last_clean":"2024-09-13T11:25:22.000325+0000","last_became_active":"2024-09-13T11:23:41.924941+0000","last_became_peered":"2024-09-13T11:23:41.924941+0000","last_unstale":"2024-09-13T11:25:22.000325+0000","last_undegraded":"2024-09-13T11:25:22.000325+0000","last_fullsized":"2024-09-13T11:25:22.000325+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":5,"last_scrub":"151'6156","last_scrub_stamp":"2024-09-13T11:25:10.784716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:10.784716+0000","objects_scrubbed":3781,"log_size":6603,"log_dups_size":0,"ondisk_log_size":6603,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:26:30.390310+0000","scrub_duration":1131,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":217317376,"num_objects":3316,"num_object_clones":0,"num_object_copies":6632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3316,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3201,"num_write_kb":145216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1],"acting":[9,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.18","version":"161'6633","reported_seq":11358,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-13T11:25:21.075816+0000","last_change":"2024-09-13T11:24:13.454659+0000","last_active":"2024-09-13T11:25:21.075816+0000","last_peered":"2024-09-13T11:25:21.075816+0000","last_clean":"2024-09-13T11:25:21.075816+0000","last_became_active":"2024-09-13T11:23:48.926014+0000","last_became_peered":"2024-09-13T11:23:48.926014+0000","last_unstale":"2024-09-13T11:25:21.075816+0000","last_undegraded":"2024-09-13T11:25:21.075816+0000","last_fullsized":"2024-09-13T11:25:21.075816+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":5,"last_scrub":"112'4376","last_scrub_stamp":"2024-09-13T11:24:13.454575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:24:13.454575+0000","objects_scrubbed":3036,"log_size":6633,"log_dups_size":0,"ondisk_log_size":6633,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:34.811860+0000","scrub_duration":375,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":227672064,"num_objects":3474,"num_object_clones":0,"num_object_copies":6948,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3474,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2266,"num_write_kb":86528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0],"acting":[10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.17","version":"161'6534","reported_seq":10729,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-13T11:25:21.058643+0000","last_change":"2024-09-13T11:25:12.658562+0000","last_active":"2024-09-13T11:25:21.058643+0000","last_peered":"2024-09-13T11:25:21.058643+0000","last_clean":"2024-09-13T11:25:21.058643+0000","last_became_active":"2024-09-13T11:23:42.935601+0000","last_became_peered":"2024-09-13T11:23:42.935601+0000","last_unstale":"2024-09-13T11:25:21.058643+0000","last_undegraded":"2024-09-13T11:25:21.058643+0000","last_fullsized":"2024-09-13T11:25:21.058643+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":5,"last_scrub":"153'6197","last_scrub_stamp":"2024-09-13T11:25:12.658464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:12.658464+0000","objects_scrubbed":3743,"log_size":6534,"log_dups_size":0,"ondisk_log_size":6534,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:26:25.625277+0000","scrub_duration":467,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":223150080,"num_objects":3405,"num_object_clones":0,"num_object_copies":6810,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3405,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3183,"num_write_kb":146688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3],"acting":[10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"162'6868","reported_seq":11702,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-13T11:25:22.064255+0000","last_change":"2024-09-13T11:25:01.903487+0000","last_active":"2024-09-13T11:25:22.064255+0000","last_peered":"2024-09-13T11:25:22.064255+0000","last_clean":"2024-09-13T11:25:22.064255+0000","last_became_active":"2024-09-13T11:23:46.105237+0000","last_became_peered":"2024-09-13T11:23:46.105237+0000","last_unstale":"2024-09-13T11:25:22.064255+0000","last_undegraded":"2024-09-13T11:25:22.064255+0000","last_fullsized":"2024-09-13T11:25:22.064255+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":5,"last_scrub":"147'6035","last_scrub_stamp":"2024-09-13T11:25:01.903395+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:01.903395+0000","objects_scrubbed":4359,"log_size":6868,"log_dups_size":0,"ondisk_log_size":6868,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:26:16.436377+0000","scrub_duration":743,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":230096896,"num_objects":3511,"num_object_clones":0,"num_object_copies":7022,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3511,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3393,"num_write_kb":155200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,15],"acting":[11,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.15","version":"166'6726","reported_seq":18051,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-13T11:25:26.070608+0000","last_change":"2024-09-13T11:25:12.198448+0000","last_active":"2024-09-13T11:25:26.070608+0000","last_peered":"2024-09-13T11:25:26.070608+0000","last_clean":"2024-09-13T11:25:26.070608+0000","last_became_active":"2024-09-13T11:24:29.226615+0000","last_became_peered":"2024-09-13T11:24:29.226615+0000","last_unstale":"2024-09-13T11:25:26.070608+0000","last_undegraded":"2024-09-13T11:25:26.070608+0000","last_fullsized":"2024-09-13T11:25:26.070608+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":5,"last_scrub":"91'3384","last_scrub_stamp":"2024-09-13T11:23:48.332600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:48.332600+0000","objects_scrubbed":2032,"log_size":6726,"log_dups_size":0,"ondisk_log_size":6726,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":235,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":209649664,"num_objects":3199,"num_object_clones":0,"num_object_copies":6398,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3199,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3343,"num_write_kb":144320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3385,"num_bytes_recovered":221839360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.14","version":"165'6857","reported_seq":18106,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-13T11:25:25.064840+0000","last_change":"2024-09-13T11:25:15.378521+0000","last_active":"2024-09-13T11:25:25.064840+0000","last_peered":"2024-09-13T11:25:25.064840+0000","last_clean":"2024-09-13T11:25:25.064840+0000","last_became_active":"2024-09-13T11:24:29.253106+0000","last_became_peered":"2024-09-13T11:24:29.253106+0000","last_unstale":"2024-09-13T11:25:25.064840+0000","last_undegraded":"2024-09-13T11:25:25.064840+0000","last_fullsized":"2024-09-13T11:25:25.064840+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":5,"last_scrub":"91'3371","last_scrub_stamp":"2024-09-13T11:23:48.696960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:48.696960+0000","objects_scrubbed":1935,"log_size":6857,"log_dups_size":0,"ondisk_log_size":6857,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":735,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":222494720,"num_objects":3395,"num_object_clones":0,"num_object_copies":6790,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3395,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3496,"num_write_kb":158592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3437,"num_bytes_recovered":225247232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,15],"acting":[3,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":304,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-13T11:25:25.226545+0000","last_change":"2024-09-13T11:24:23.212494+0000","last_active":"2024-09-13T11:25:25.226545+0000","last_peered":"2024-09-13T11:25:25.226545+0000","last_clean":"2024-09-13T11:25:25.226545+0000","last_became_active":"2024-09-13T11:22:14.192897+0000","last_became_peered":"2024-09-13T11:22:14.192897+0000","last_unstale":"2024-09-13T11:25:25.226545+0000","last_undegraded":"2024-09-13T11:25:25.226545+0000","last_fullsized":"2024-09-13T11:25:25.226545+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:24:23.212406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:24:23.212406+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:29.886382+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00084655899999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,10],"acting":[15,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"3.6","version":"162'5853","reported_seq":9708,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-13T11:25:21.885529+0000","last_change":"2024-09-13T11:24:49.416851+0000","last_active":"2024-09-13T11:25:21.885529+0000","last_peered":"2024-09-13T11:25:21.885529+0000","last_clean":"2024-09-13T11:25:21.885529+0000","last_became_active":"2024-09-13T11:24:29.322558+0000","last_became_peered":"2024-09-13T11:24:29.322558+0000","last_unstale":"2024-09-13T11:25:21.885529+0000","last_undegraded":"2024-09-13T11:25:21.885529+0000","last_fullsized":"2024-09-13T11:25:21.885529+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"136'5216","last_scrub_stamp":"2024-09-13T11:24:49.416706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:24:49.416706+0000","objects_scrubbed":2083,"log_size":5853,"log_dups_size":0,"ondisk_log_size":5853,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:26:09.525086+0000","scrub_duration":247,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":113967104,"num_objects":1739,"num_object_clones":0,"num_object_copies":3478,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1739,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":638,"num_write_kb":9408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,6],"acting":[13,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":303,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-13T11:25:21.181055+0000","last_change":"2024-09-13T11:24:59.556253+0000","last_active":"2024-09-13T11:25:21.181055+0000","last_peered":"2024-09-13T11:25:21.181055+0000","last_clean":"2024-09-13T11:25:21.181055+0000","last_became_active":"2024-09-13T11:22:14.211923+0000","last_became_peered":"2024-09-13T11:22:14.211923+0000","last_unstale":"2024-09-13T11:25:21.181055+0000","last_undegraded":"2024-09-13T11:25:21.181055+0000","last_fullsized":"2024-09-13T11:25:21.181055+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:24:59.556180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:24:59.556180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:26:06.229322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011689529999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0],"acting":[9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"162'5680","reported_seq":6029,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-13T11:25:22.049662+0000","last_change":"2024-09-13T11:24:33.336210+0000","last_active":"2024-09-13T11:25:22.049662+0000","last_peered":"2024-09-13T11:25:22.049662+0000","last_clean":"2024-09-13T11:25:22.049662+0000","last_became_active":"2024-09-13T11:24:24.092975+0000","last_became_peered":"2024-09-13T11:24:24.092975+0000","last_unstale":"2024-09-13T11:25:22.049662+0000","last_undegraded":"2024-09-13T11:25:22.049662+0000","last_fullsized":"2024-09-13T11:25:22.049662+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"125'4820","last_scrub_stamp":"2024-09-13T11:24:33.336091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:24:33.336091+0000","objects_scrubbed":1766,"log_size":5680,"log_dups_size":0,"ondisk_log_size":5680,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:54.538352+0000","scrub_duration":263,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":113246208,"num_objects":1728,"num_object_clones":0,"num_object_copies":3456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1728,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":862,"num_write_kb":26368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,8],"acting":[13,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":304,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-13T11:25:25.221085+0000","last_change":"2024-09-13T11:23:59.530794+0000","last_active":"2024-09-13T11:25:25.221085+0000","last_peered":"2024-09-13T11:25:25.221085+0000","last_clean":"2024-09-13T11:25:25.221085+0000","last_became_active":"2024-09-13T11:22:14.581458+0000","last_became_peered":"2024-09-13T11:22:14.581458+0000","last_unstale":"2024-09-13T11:25:25.221085+0000","last_undegraded":"2024-09-13T11:25:25.221085+0000","last_fullsized":"2024-09-13T11:25:25.221085+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:23:59.530670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:23:59.530670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001026047,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5],"acting":[8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"166'5817","reported_seq":6146,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-13T11:25:25.378158+0000","last_change":"2024-09-13T11:24:24.579036+0000","last_active":"2024-09-13T11:25:25.378158+0000","last_peered":"2024-09-13T11:25:25.378158+0000","last_clean":"2024-09-13T11:25:25.378158+0000","last_became_active":"2024-09-13T11:24:24.083077+0000","last_became_peered":"2024-09-13T11:24:24.083077+0000","last_unstale":"2024-09-13T11:25:25.378158+0000","last_undegraded":"2024-09-13T11:25:25.378158+0000","last_fullsized":"2024-09-13T11:25:25.378158+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'4737","last_scrub_stamp":"2024-09-13T11:24:24.578900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:24:24.578900+0000","objects_scrubbed":1690,"log_size":5817,"log_dups_size":0,"ondisk_log_size":5817,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:38.103421+0000","scrub_duration":407,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":106758144,"num_objects":1629,"num_object_clones":0,"num_object_copies":3258,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1629,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1083,"num_write_kb":32704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,12],"acting":[1,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":292,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-13T11:25:25.217725+0000","last_change":"2024-09-13T11:24:08.271026+0000","last_active":"2024-09-13T11:25:25.217725+0000","last_peered":"2024-09-13T11:25:25.217725+0000","last_clean":"2024-09-13T11:25:25.217725+0000","last_became_active":"2024-09-13T11:22:47.719636+0000","last_became_peered":"2024-09-13T11:22:47.719636+0000","last_unstale":"2024-09-13T11:25:25.217725+0000","last_undegraded":"2024-09-13T11:25:25.217725+0000","last_fullsized":"2024-09-13T11:25:25.217725+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:24:08.270972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:24:08.270972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:36.805605+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00087476000000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,9],"acting":[1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"166'5772","reported_seq":6120,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-13T11:25:26.041019+0000","last_change":"2024-09-13T11:24:24.616937+0000","last_active":"2024-09-13T11:25:26.041019+0000","last_peered":"2024-09-13T11:25:26.041019+0000","last_clean":"2024-09-13T11:25:26.041019+0000","last_became_active":"2024-09-13T11:24:24.093492+0000","last_became_peered":"2024-09-13T11:24:24.093492+0000","last_unstale":"2024-09-13T11:25:26.041019+0000","last_undegraded":"2024-09-13T11:25:26.041019+0000","last_fullsized":"2024-09-13T11:25:26.041019+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'4744","last_scrub_stamp":"2024-09-13T11:24:24.616633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:24:24.616633+0000","objects_scrubbed":1624,"log_size":5772,"log_dups_size":0,"ondisk_log_size":5772,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:42.911074+0000","scrub_duration":415,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":104923136,"num_objects":1601,"num_object_clones":0,"num_object_copies":3202,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1601,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1035,"num_write_kb":32384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,13],"acting":[15,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"3.3","version":"161'4854","reported_seq":5212,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-13T11:25:20.987507+0000","last_change":"2024-09-13T11:25:08.394545+0000","last_active":"2024-09-13T11:25:20.987507+0000","last_peered":"2024-09-13T11:25:20.987507+0000","last_clean":"2024-09-13T11:25:20.987507+0000","last_became_active":"2024-09-13T11:23:31.930448+0000","last_became_peered":"2024-09-13T11:23:31.930448+0000","last_unstale":"2024-09-13T11:25:20.987507+0000","last_undegraded":"2024-09-13T11:25:20.987507+0000","last_fullsized":"2024-09-13T11:25:20.987507+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"151'4587","last_scrub_stamp":"2024-09-13T11:25:08.394441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:08.394441+0000","objects_scrubbed":1942,"log_size":4854,"log_dups_size":0,"ondisk_log_size":4854,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:26:27.641900+0000","scrub_duration":219,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109576192,"num_objects":1672,"num_object_clones":0,"num_object_copies":3344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1672,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1732,"num_write_kb":80896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,7],"acting":[10,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"28'2","reported_seq":242,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-13T11:25:25.216396+0000","last_change":"2024-09-13T11:25:13.869564+0000","last_active":"2024-09-13T11:25:25.216396+0000","last_peered":"2024-09-13T11:25:25.216396+0000","last_clean":"2024-09-13T11:25:25.216396+0000","last_became_active":"2024-09-13T11:23:12.546348+0000","last_became_peered":"2024-09-13T11:23:12.546348+0000","last_unstale":"2024-09-13T11:25:25.216396+0000","last_undegraded":"2024-09-13T11:25:25.216396+0000","last_fullsized":"2024-09-13T11:25:25.216396+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2","last_scrub_stamp":"2024-09-13T11:25:13.869413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:25:13.869413+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:26:38.761718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0025826730000000002,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8],"acting":[2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"166'4962","reported_seq":5328,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-13T11:25:26.067735+0000","last_change":"2024-09-13T11:25:09.842875+0000","last_active":"2024-09-13T11:25:26.067735+0000","last_peered":"2024-09-13T11:25:26.067735+0000","last_clean":"2024-09-13T11:25:26.067735+0000","last_became_active":"2024-09-13T11:23:31.921388+0000","last_became_peered":"2024-09-13T11:23:31.921388+0000","last_unstale":"2024-09-13T11:25:26.067735+0000","last_undegraded":"2024-09-13T11:25:26.067735+0000","last_fullsized":"2024-09-13T11:25:26.067735+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"151'4644","last_scrub_stamp":"2024-09-13T11:25:09.842728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:09.842728+0000","objects_scrubbed":1944,"log_size":4962,"log_dups_size":0,"ondisk_log_size":4962,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:26:25.553642+0000","scrub_duration":395,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":106561536,"num_objects":1626,"num_object_clones":0,"num_object_copies":3252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1626,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1856,"num_write_kb":84032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,15],"acting":[12,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":145,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-13T11:25:25.217562+0000","last_change":"2024-09-13T11:24:29.096863+0000","last_active":"2024-09-13T11:25:25.217562+0000","last_peered":"2024-09-13T11:25:25.217562+0000","last_clean":"2024-09-13T11:25:25.217562+0000","last_became_active":"2024-09-13T11:24:29.096575+0000","last_became_peered":"2024-09-13T11:24:29.096575+0000","last_unstale":"2024-09-13T11:25:25.217562+0000","last_undegraded":"2024-09-13T11:25:25.217562+0000","last_fullsized":"2024-09-13T11:25:25.217562+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:24:22.064711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:24:22.064711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:34.178168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00081225399999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"162'4866","reported_seq":5227,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-13T11:25:22.016980+0000","last_change":"2024-09-13T11:25:00.861990+0000","last_active":"2024-09-13T11:25:22.016980+0000","last_peered":"2024-09-13T11:25:22.016980+0000","last_clean":"2024-09-13T11:25:22.016980+0000","last_became_active":"2024-09-13T11:23:31.926528+0000","last_became_peered":"2024-09-13T11:23:31.926528+0000","last_unstale":"2024-09-13T11:25:22.016980+0000","last_undegraded":"2024-09-13T11:25:22.016980+0000","last_fullsized":"2024-09-13T11:25:22.016980+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"146'4447","last_scrub_stamp":"2024-09-13T11:25:00.861834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:00.861834+0000","objects_scrubbed":2126,"log_size":4866,"log_dups_size":0,"ondisk_log_size":4866,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:26:15.411903+0000","scrub_duration":787,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":111673344,"num_objects":1704,"num_object_clones":0,"num_object_copies":3408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1704,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1796,"num_write_kb":85056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,11],"acting":[13,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":302,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-13T11:25:24.217757+0000","last_change":"2024-09-13T11:24:11.895017+0000","last_active":"2024-09-13T11:25:24.217757+0000","last_peered":"2024-09-13T11:25:24.217757+0000","last_clean":"2024-09-13T11:25:24.217757+0000","last_became_active":"2024-09-13T11:22:14.211435+0000","last_became_peered":"2024-09-13T11:22:14.211435+0000","last_unstale":"2024-09-13T11:25:24.217757+0000","last_undegraded":"2024-09-13T11:25:24.217757+0000","last_fullsized":"2024-09-13T11:25:24.217757+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:24:11.894937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:24:11.894937+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0010010360000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2","version":"166'5033","reported_seq":8863,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-13T11:25:25.866101+0000","last_change":"2024-09-13T11:25:23.765128+0000","last_active":"2024-09-13T11:25:25.866101+0000","last_peered":"2024-09-13T11:25:25.866101+0000","last_clean":"2024-09-13T11:25:25.866101+0000","last_became_active":"2024-09-13T11:24:29.175407+0000","last_became_peered":"2024-09-13T11:24:29.175407+0000","last_unstale":"2024-09-13T11:25:25.866101+0000","last_undegraded":"2024-09-13T11:25:25.866101+0000","last_fullsized":"2024-09-13T11:25:25.866101+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"164'4994","last_scrub_stamp":"2024-09-13T11:25:23.765047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:23.765047+0000","objects_scrubbed":1689,"log_size":5033,"log_dups_size":0,"ondisk_log_size":5033,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:26:45.252942+0000","scrub_duration":283,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":107937792,"num_objects":1647,"num_object_clones":0,"num_object_copies":3294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1647,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1976,"num_write_kb":89024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1731,"num_bytes_recovered":113442816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,3],"acting":[12,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[]},{"pgid":"2.3","version":"26'1","reported_seq":115,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-13T11:25:25.226319+0000","last_change":"2024-09-13T11:25:23.034357+0000","last_active":"2024-09-13T11:25:25.226319+0000","last_peered":"2024-09-13T11:25:25.226319+0000","last_clean":"2024-09-13T11:25:25.226319+0000","last_became_active":"2024-09-13T11:24:29.110028+0000","last_became_peered":"2024-09-13T11:24:29.110028+0000","last_unstale":"2024-09-13T11:25:25.226319+0000","last_undegraded":"2024-09-13T11:25:25.226319+0000","last_fullsized":"2024-09-13T11:25:25.226319+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"26'1","last_scrub_stamp":"2024-09-13T11:25:23.034247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:25:23.034247+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:26:42.429349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,8],"acting":[15,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"1.0","version":"26'112","reported_seq":446,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-13T11:25:25.232244+0000","last_change":"2024-09-13T11:25:23.642366+0000","last_active":"2024-09-13T11:25:25.232244+0000","last_peered":"2024-09-13T11:25:25.232244+0000","last_clean":"2024-09-13T11:25:25.232244+0000","last_became_active":"2024-09-13T11:22:12.129721+0000","last_became_peered":"2024-09-13T11:22:12.129721+0000","last_unstale":"2024-09-13T11:25:25.232244+0000","last_undegraded":"2024-09-13T11:25:25.232244+0000","last_fullsized":"2024-09-13T11:25:25.232244+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"26'112","last_scrub_stamp":"2024-09-13T11:25:23.642204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:11.098388+0000","last_clean_scrub_stamp":"2024-09-13T11:25:23.642204+0000","objects_scrubbed":2,"log_size":112,"log_dups_size":0,"ondisk_log_size":112,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:26:42.628319+0000","scrub_duration":67,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.8","version":"166'5776","reported_seq":6125,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-13T11:25:25.960544+0000","last_change":"2024-09-13T11:24:24.506212+0000","last_active":"2024-09-13T11:25:25.960544+0000","last_peered":"2024-09-13T11:25:25.960544+0000","last_clean":"2024-09-13T11:25:25.960544+0000","last_became_active":"2024-09-13T11:24:24.096220+0000","last_became_peered":"2024-09-13T11:24:24.096220+0000","last_unstale":"2024-09-13T11:25:25.960544+0000","last_undegraded":"2024-09-13T11:25:25.960544+0000","last_fullsized":"2024-09-13T11:25:25.960544+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'4749","last_scrub_stamp":"2024-09-13T11:24:24.506109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:24:24.506109+0000","objects_scrubbed":1690,"log_size":5776,"log_dups_size":0,"ondisk_log_size":5776,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:49.430042+0000","scrub_duration":215,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":107347968,"num_objects":1638,"num_object_clones":0,"num_object_copies":3276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1638,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1028,"num_write_kb":31232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.9","version":"166'5817","reported_seq":6165,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-13T11:25:25.911883+0000","last_change":"2024-09-13T11:24:24.928964+0000","last_active":"2024-09-13T11:25:25.911883+0000","last_peered":"2024-09-13T11:25:25.911883+0000","last_clean":"2024-09-13T11:25:25.911883+0000","last_became_active":"2024-09-13T11:24:24.082504+0000","last_became_peered":"2024-09-13T11:24:24.082504+0000","last_unstale":"2024-09-13T11:25:25.911883+0000","last_undegraded":"2024-09-13T11:25:25.911883+0000","last_fullsized":"2024-09-13T11:25:25.911883+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'4799","last_scrub_stamp":"2024-09-13T11:24:24.928724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:24:24.928724+0000","objects_scrubbed":1705,"log_size":5817,"log_dups_size":0,"ondisk_log_size":5817,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:43.845801+0000","scrub_duration":255,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":108527616,"num_objects":1656,"num_object_clones":0,"num_object_copies":3312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1656,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1021,"num_write_kb":31104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,2],"acting":[12,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[]},{"pgid":"3.a","version":"165'6176","reported_seq":6526,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-13T11:25:25.021479+0000","last_change":"2024-09-13T11:24:25.137871+0000","last_active":"2024-09-13T11:25:25.021479+0000","last_peered":"2024-09-13T11:25:25.021479+0000","last_clean":"2024-09-13T11:25:25.021479+0000","last_became_active":"2024-09-13T11:24:24.089253+0000","last_became_peered":"2024-09-13T11:24:24.089253+0000","last_unstale":"2024-09-13T11:25:25.021479+0000","last_undegraded":"2024-09-13T11:25:25.021479+0000","last_fullsized":"2024-09-13T11:25:25.021479+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'5157","last_scrub_stamp":"2024-09-13T11:24:25.137777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:24:25.137777+0000","objects_scrubbed":1777,"log_size":6176,"log_dups_size":0,"ondisk_log_size":6176,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:36.149013+0000","scrub_duration":223,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":106758144,"num_objects":1629,"num_object_clones":0,"num_object_copies":3258,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1629,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1022,"num_write_kb":27968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.b","version":"162'6059","reported_seq":6402,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-13T11:25:21.990278+0000","last_change":"2024-09-13T11:24:25.503864+0000","last_active":"2024-09-13T11:25:21.990278+0000","last_peered":"2024-09-13T11:25:21.990278+0000","last_clean":"2024-09-13T11:25:21.990278+0000","last_became_active":"2024-09-13T11:24:24.085684+0000","last_became_peered":"2024-09-13T11:24:24.085684+0000","last_unstale":"2024-09-13T11:25:21.990278+0000","last_undegraded":"2024-09-13T11:25:21.990278+0000","last_fullsized":"2024-09-13T11:25:21.990278+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'5112","last_scrub_stamp":"2024-09-13T11:24:25.503716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:24:25.503716+0000","objects_scrubbed":1718,"log_size":6059,"log_dups_size":0,"ondisk_log_size":6059,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:37.636639+0000","scrub_duration":395,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":113311744,"num_objects":1729,"num_object_clones":0,"num_object_copies":3458,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1729,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":947,"num_write_kb":30656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,1],"acting":[13,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"3.c","version":"162'6080","reported_seq":6423,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-13T11:25:21.979106+0000","last_change":"2024-09-13T11:24:24.595021+0000","last_active":"2024-09-13T11:25:21.979106+0000","last_peered":"2024-09-13T11:25:21.979106+0000","last_clean":"2024-09-13T11:25:21.979106+0000","last_became_active":"2024-09-13T11:24:24.090847+0000","last_became_peered":"2024-09-13T11:24:24.090847+0000","last_unstale":"2024-09-13T11:25:21.979106+0000","last_undegraded":"2024-09-13T11:25:21.979106+0000","last_fullsized":"2024-09-13T11:25:21.979106+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'5154","last_scrub_stamp":"2024-09-13T11:24:24.594848+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:24:24.594848+0000","objects_scrubbed":1720,"log_size":6080,"log_dups_size":0,"ondisk_log_size":6080,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:35.181590+0000","scrub_duration":343,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112001024,"num_objects":1709,"num_object_clones":0,"num_object_copies":3418,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1709,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":931,"num_write_kb":29440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,13],"acting":[11,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.d","version":"166'6161","reported_seq":6512,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-13T11:25:26.002410+0000","last_change":"2024-09-13T11:24:24.967444+0000","last_active":"2024-09-13T11:25:26.002410+0000","last_peered":"2024-09-13T11:25:26.002410+0000","last_clean":"2024-09-13T11:25:26.002410+0000","last_became_active":"2024-09-13T11:24:24.087224+0000","last_became_peered":"2024-09-13T11:24:24.087224+0000","last_unstale":"2024-09-13T11:25:26.002410+0000","last_undegraded":"2024-09-13T11:25:26.002410+0000","last_fullsized":"2024-09-13T11:25:26.002410+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'5110","last_scrub_stamp":"2024-09-13T11:24:24.967337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:24:24.967337+0000","objects_scrubbed":1725,"log_size":6161,"log_dups_size":0,"ondisk_log_size":6161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:53.620949+0000","scrub_duration":255,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":107282432,"num_objects":1637,"num_object_clones":0,"num_object_copies":3274,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1637,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1054,"num_write_kb":30912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.e","version":"166'7290","reported_seq":12538,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-13T11:25:26.050702+0000","last_change":"2024-09-13T11:23:32.368856+0000","last_active":"2024-09-13T11:25:26.050702+0000","last_peered":"2024-09-13T11:25:26.050702+0000","last_clean":"2024-09-13T11:25:26.050702+0000","last_became_active":"2024-09-13T11:23:31.925869+0000","last_became_peered":"2024-09-13T11:23:31.925869+0000","last_unstale":"2024-09-13T11:25:26.050702+0000","last_undegraded":"2024-09-13T11:25:26.050702+0000","last_fullsized":"2024-09-13T11:25:26.050702+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'3527","last_scrub_stamp":"2024-09-13T11:23:32.368727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:32.368727+0000","objects_scrubbed":1774,"log_size":7290,"log_dups_size":0,"ondisk_log_size":7290,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":219,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":209321984,"num_objects":3194,"num_object_clones":0,"num_object_copies":6388,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3194,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3764,"num_write_kb":165888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"162'7116","reported_seq":7478,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-13T11:25:22.048153+0000","last_change":"2024-09-13T11:25:00.289940+0000","last_active":"2024-09-13T11:25:22.048153+0000","last_peered":"2024-09-13T11:25:22.048153+0000","last_clean":"2024-09-13T11:25:22.048153+0000","last_became_active":"2024-09-13T11:23:31.935239+0000","last_became_peered":"2024-09-13T11:23:31.935239+0000","last_unstale":"2024-09-13T11:25:22.048153+0000","last_undegraded":"2024-09-13T11:25:22.048153+0000","last_fullsized":"2024-09-13T11:25:22.048153+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"146'6154","last_scrub_stamp":"2024-09-13T11:25:00.289673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:00.289673+0000","objects_scrubbed":4394,"log_size":7116,"log_dups_size":0,"ondisk_log_size":7116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:26:24.274856+0000","scrub_duration":1187,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":223477760,"num_objects":3410,"num_object_clones":0,"num_object_copies":6820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3410,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3696,"num_write_kb":170560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,12],"acting":[11,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.10","version":"166'6955","reported_seq":11477,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-13T11:25:26.029727+0000","last_change":"2024-09-13T11:25:10.161564+0000","last_active":"2024-09-13T11:25:26.029727+0000","last_peered":"2024-09-13T11:25:26.029727+0000","last_clean":"2024-09-13T11:25:26.029727+0000","last_became_active":"2024-09-13T11:23:41.931183+0000","last_became_peered":"2024-09-13T11:23:41.931183+0000","last_unstale":"2024-09-13T11:25:26.029727+0000","last_undegraded":"2024-09-13T11:25:26.029727+0000","last_fullsized":"2024-09-13T11:25:26.029727+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":5,"last_scrub":"151'6298","last_scrub_stamp":"2024-09-13T11:25:10.161464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:10.161464+0000","objects_scrubbed":3964,"log_size":6955,"log_dups_size":0,"ondisk_log_size":6955,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:26:33.015478+0000","scrub_duration":607,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":215744512,"num_objects":3292,"num_object_clones":0,"num_object_copies":6584,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3292,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3489,"num_write_kb":152000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.11","version":"166'6880","reported_seq":10174,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-13T11:25:26.042414+0000","last_change":"2024-09-13T11:25:24.287326+0000","last_active":"2024-09-13T11:25:26.042414+0000","last_peered":"2024-09-13T11:25:26.042414+0000","last_clean":"2024-09-13T11:25:26.042414+0000","last_became_active":"2024-09-13T11:23:39.063172+0000","last_became_peered":"2024-09-13T11:23:39.063172+0000","last_unstale":"2024-09-13T11:25:26.042414+0000","last_undegraded":"2024-09-13T11:25:26.042414+0000","last_fullsized":"2024-09-13T11:25:26.042414+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":5,"last_scrub":"165'6821","last_scrub_stamp":"2024-09-13T11:25:24.287082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:24.287082+0000","objects_scrubbed":3406,"log_size":6880,"log_dups_size":0,"ondisk_log_size":6880,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:26:49.405332+0000","scrub_duration":947,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":218497024,"num_objects":3334,"num_object_clones":0,"num_object_copies":6668,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3334,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3539,"num_write_kb":155584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11],"acting":[4,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.12","version":"166'6901","reported_seq":17004,"reported_epoch":166,"state":"active+clean+scrubbing","last_fresh":"2024-09-13T11:25:26.193477+0000","last_change":"2024-09-13T11:25:26.136263+0000","last_active":"2024-09-13T11:25:26.193477+0000","last_peered":"2024-09-13T11:25:26.193477+0000","last_clean":"2024-09-13T11:25:26.193477+0000","last_became_active":"2024-09-13T11:24:29.268229+0000","last_became_peered":"2024-09-13T11:24:29.268229+0000","last_unstale":"2024-09-13T11:25:26.193477+0000","last_undegraded":"2024-09-13T11:25:26.193477+0000","last_fullsized":"2024-09-13T11:25:26.193477+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'3109","last_scrub_stamp":"2024-09-13T11:23:35.111320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:35.111320+0000","objects_scrubbed":399,"log_size":6901,"log_dups_size":0,"ondisk_log_size":6901,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"scrubbing for 0s","scrub_duration":239,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":216662016,"num_objects":3306,"num_object_clones":0,"num_object_copies":6612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3306,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3792,"num_write_kb":171712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3457,"num_bytes_recovered":226557952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.13","version":"162'6778","reported_seq":9999,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-13T11:25:22.046470+0000","last_change":"2024-09-13T11:25:01.153044+0000","last_active":"2024-09-13T11:25:22.046470+0000","last_peered":"2024-09-13T11:25:22.046470+0000","last_clean":"2024-09-13T11:25:22.046470+0000","last_became_active":"2024-09-13T11:23:41.082433+0000","last_became_peered":"2024-09-13T11:23:41.082433+0000","last_unstale":"2024-09-13T11:25:22.046470+0000","last_undegraded":"2024-09-13T11:25:22.046470+0000","last_fullsized":"2024-09-13T11:25:22.046470+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":5,"last_scrub":"147'5921","last_scrub_stamp":"2024-09-13T11:25:01.152971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:01.152971+0000","objects_scrubbed":4311,"log_size":6778,"log_dups_size":0,"ondisk_log_size":6778,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:26:07.027631+0000","scrub_duration":1011,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":223674368,"num_objects":3413,"num_object_clones":0,"num_object_copies":6826,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3413,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3382,"num_write_kb":154048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7],"acting":[11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":46,"stat_sum":{"num_bytes":7007076376,"num_objects":106921,"num_object_clones":0,"num_object_copies":213842,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3138,"num_objects_unfound":0,"num_objects_dirty":106921,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":89775,"num_write_kb":3634433,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20645,"num_bytes_recovered":1352990720,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":15639519232,"data_stored":15639511088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":43282,"internal_metadata":0},"log_size":283159,"ondisk_log_size":283159,"up":92,"acting":92,"num_store_stats":16},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":11},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":112,"ondisk_log_size":112,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":13,"up_from":23,"seq":98784247850,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":886968,"kb_used_data":779432,"kb_used_omap":1,"kb_used_meta":107518,"kb_avail":93484872,"statfs":{"total":96636764160,"available":95728508928,"internally_reserved":0,"allocated":798138368,"data_stored":797420163,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":110098885},"hb_peers":[0,1,2,4,6,8,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":23,"seq":98784247849,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":736032,"kb_used_data":656272,"kb_used_omap":1,"kb_used_meta":79742,"kb_avail":93635808,"statfs":{"total":96636764160,"available":95883067392,"internally_reserved":0,"allocated":672022528,"data_stored":671332275,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":81656256},"hb_peers":[0,1,3,8,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":40,"seq":171798691873,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1171488,"kb_used_data":1067280,"kb_used_omap":4,"kb_used_meta":104187,"kb_avail":93200352,"statfs":{"total":96636764160,"available":95437160448,"internally_reserved":0,"allocated":1092894720,"data_stored":1092165338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4433,"internal_metadata":106688175},"hb_peers":[0,2,3,4,5,6,7,8,9,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":21,"seq":90194313258,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1218360,"kb_used_data":1119144,"kb_used_omap":1,"kb_used_meta":99198,"kb_avail":93153480,"statfs":{"total":96636764160,"available":95389163520,"internally_reserved":0,"allocated":1146003456,"data_stored":1145255743,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":101579207},"hb_peers":[0,2,4,5,7,9,10,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":21,"seq":90194313261,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1257128,"kb_used_data":1175960,"kb_used_omap":1,"kb_used_meta":81150,"kb_avail":93114712,"statfs":{"total":96636764160,"available":95349465088,"internally_reserved":0,"allocated":1204183040,"data_stored":1203447594,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":83098055},"hb_peers":[0,2,4,6,7,8,9,11,12,13],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":22,"seq":94489280553,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1307644,"kb_used_data":1200108,"kb_used_omap":1,"kb_used_meta":107518,"kb_avail":93064196,"statfs":{"total":96636764160,"available":95297736704,"internally_reserved":0,"allocated":1228910592,"data_stored":1228223837,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":110098885},"hb_peers":[1,3,4,5,7,8,10,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":22,"seq":94489280554,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":602472,"kb_used_data":526872,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":93769368,"statfs":{"total":96636764160,"available":96019832832,"internally_reserved":0,"allocated":539516928,"data_stored":538792320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,1,3,4,5,7,8,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":15,"up_from":22,"seq":94489280554,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1227888,"kb_used_data":1116192,"kb_used_omap":1,"kb_used_meta":111678,"kb_avail":93143952,"statfs":{"total":96636764160,"available":95379406848,"internally_reserved":0,"allocated":1142980608,"data_stored":1142305477,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":114358726},"hb_peers":[0,4,5,6,7,8,10,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":22,"seq":94489280555,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":806068,"kb_used_data":742948,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":93565772,"statfs":{"total":96636764160,"available":95811350528,"internally_reserved":0,"allocated":760778752,"data_stored":760043385,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[3,5,6,7,9,10,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":22,"seq":94489280554,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":589264,"kb_used_data":535808,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":93782576,"statfs":{"total":96636764160,"available":96033357824,"internally_reserved":0,"allocated":548667392,"data_stored":547963100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54655430},"hb_peers":[0,1,3,5,7,8,10,11,12,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":22,"seq":94489280554,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1467536,"kb_used_data":1364160,"kb_used_omap":1,"kb_used_meta":103358,"kb_avail":92904304,"statfs":{"total":96636764160,"available":95134007296,"internally_reserved":0,"allocated":1396899840,"data_stored":1396192229,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":105839050},"hb_peers":[0,3,5,6,8,9,10,11,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":22,"seq":94489280554,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":566340,"kb_used_data":494900,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":93805500,"statfs":{"total":96636764160,"available":96056832000,"internally_reserved":0,"allocated":506777600,"data_stored":506022479,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73136581},"hb_peers":[1,5,6,7,9,10,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":22,"seq":94489280552,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1073248,"kb_used_data":1005968,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":93298592,"statfs":{"total":96636764160,"available":95537758208,"internally_reserved":0,"allocated":1030111232,"data_stored":1029419532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":68876736},"hb_peers":[0,1,3,5,7,9,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":11,"up_from":22,"seq":94489280554,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1513220,"kb_used_data":1405684,"kb_used_omap":1,"kb_used_meta":107518,"kb_avail":92858620,"statfs":{"total":96636764160,"available":95087226880,"internally_reserved":0,"allocated":1439420416,"data_stored":1438701096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":110098884},"hb_peers":[0,1,2,4,7,8,10,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":12,"up_from":22,"seq":94489280554,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1280248,"kb_used_data":1168552,"kb_used_omap":1,"kb_used_meta":111678,"kb_avail":93091592,"statfs":{"total":96636764160,"available":95325790208,"internally_reserved":0,"allocated":1196597248,"data_stored":1195854952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":114358725},"hb_peers":[0,1,2,3,4,5,10,11,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":14,"up_from":0,"seq":609885356039,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1226309632,"data_stored":1226309632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1082654720,"data_stored":1082654720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":534515712,"data_stored":534511640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1139474432,"data_stored":1139474432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":752025600,"data_stored":752025600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1200947200,"data_stored":1200947200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":542900224,"data_stored":542900224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1394081792,"data_stored":1394081792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":504627200,"data_stored":504627200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":668794880,"data_stored":668794880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":1016266752,"data_stored":1016266752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":1436155904,"data_stored":1436155904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":1183645696,"data_stored":1183645696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":787087360,"data_stored":787087360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":1030029312,"data_stored":1030029312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":43282,"internal_metadata":0},{"poolid":3,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":1140002816,"data_stored":1139998744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T11:25:26.377 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-13T11:25:26.377 INFO:tasks.ceph.osd.4.smithi001.stderr:2024-09-13T11:25:26.315+0000 7ff8e0218640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26310) UID: 0 2024-09-13T11:25:26.419 INFO:tasks.ceph.osd.3.smithi177.stderr:2024-09-13T11:25:26.414+0000 7f10adba4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26574) UID: 0 2024-09-13T11:25:26.520 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-13T11:25:26.519+0000 7f6a68775640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26307) UID: 0 2024-09-13T11:25:26.520 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-13T11:25:26.520 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [ 2024-09-13T11:25:26.520 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.520 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:176577 3.17 3:ec0ba30c:::benchmark_data_smithi177_31116_object36274:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e167)", 2024-09-13T11:25:26.520 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:25:26.514985+0000", 2024-09-13T11:25:26.520 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 0.0055109729999999997, 2024-09-13T11:25:26.521 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0055305629999999996, 2024-09-13T11:25:26.521 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:25:26.521 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:25:26.521 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "waiting for sub ops", 2024-09-13T11:25:26.521 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:25:26.521 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:25:26.521 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:26.521 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 176577 2024-09-13T11:25:26.521 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.521 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:25:26.521 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.521 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:25:26.521 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:26.514985+0000", 2024-09-13T11:25:26.521 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:26.522 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.522 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.522 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:25:26.522 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:26.514985+0000", 2024-09-13T11:25:26.522 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:26.522 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.522 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.522 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:25:26.522 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:26.514987+0000", 2024-09-13T11:25:26.522 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.793e-06 2024-09-13T11:25:26.522 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.522 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.522 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:25:26.523 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:26.514992+0000", 2024-09-13T11:25:26.523 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.6999999999999996e-06 2024-09-13T11:25:26.523 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.523 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.523 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:25:26.523 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:26.515007+0000", 2024-09-13T11:25:26.523 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4589999999999999e-05 2024-09-13T11:25:26.523 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.523 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.523 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:25:26.523 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:26.515019+0000", 2024-09-13T11:25:26.523 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1463e-05 2024-09-13T11:25:26.524 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.524 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.524 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:25:26.524 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:26.515125+0000", 2024-09-13T11:25:26.524 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000106304 2024-09-13T11:25:26.524 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.524 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.524 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:25:26.524 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:26.515351+0000", 2024-09-13T11:25:26.524 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00022625799999999999 2024-09-13T11:25:26.524 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.524 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.525 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 3", 2024-09-13T11:25:26.525 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:26.515561+0000", 2024-09-13T11:25:26.525 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00021035800000000001 2024-09-13T11:25:26.525 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.525 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.525 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:25:26.525 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:26.519045+0000", 2024-09-13T11:25:26.525 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0034838120000000002 2024-09-13T11:25:26.525 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:26.525 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:25:26.525 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:26.525 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.525 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.525 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:176492 3.1f 3:f99090f8:::benchmark_data_smithi177_31116_object36189:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e167)", 2024-09-13T11:25:26.526 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:25:26.458721+0000", 2024-09-13T11:25:26.526 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 0.061774846000000001, 2024-09-13T11:25:26.526 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.061839499999999999, 2024-09-13T11:25:26.526 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:25:26.526 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:25:26.526 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "waiting for sub ops", 2024-09-13T11:25:26.526 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:25:26.526 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:25:26.526 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:26.526 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 176492 2024-09-13T11:25:26.526 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.526 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:25:26.526 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.527 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:25:26.527 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:26.458721+0000", 2024-09-13T11:25:26.527 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:26.527 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.527 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.527 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:25:26.527 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:26.458721+0000", 2024-09-13T11:25:26.527 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:26.527 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.527 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.527 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:25:26.527 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:26.458724+0000", 2024-09-13T11:25:26.527 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.1860000000000001e-06 2024-09-13T11:25:26.528 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.528 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.528 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:25:26.528 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:26.458733+0000", 2024-09-13T11:25:26.528 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.8179999999999993e-06 2024-09-13T11:25:26.528 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.528 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.528 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:25:26.528 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:26.458758+0000", 2024-09-13T11:25:26.528 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.5049999999999999e-05 2024-09-13T11:25:26.528 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.528 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.528 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:25:26.529 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:26.458773+0000", 2024-09-13T11:25:26.529 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4598e-05 2024-09-13T11:25:26.529 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.529 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.529 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:25:26.529 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:26.458837+0000", 2024-09-13T11:25:26.529 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.4633000000000003e-05 2024-09-13T11:25:26.529 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.529 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.529 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:25:26.529 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:26.458943+0000", 2024-09-13T11:25:26.529 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000105215 2024-09-13T11:25:26.529 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.529 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.530 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 5", 2024-09-13T11:25:26.530 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:26.459020+0000", 2024-09-13T11:25:26.530 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.7714999999999997e-05 2024-09-13T11:25:26.530 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.530 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.530 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:25:26.530 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:26.461558+0000", 2024-09-13T11:25:26.530 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.002537287 2024-09-13T11:25:26.530 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:26.530 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:25:26.530 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:26.530 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.530 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.530 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:176492 3.1f 3:f99090f8:::benchmark_data_smithi177_31116_object36189:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e167)", 2024-09-13T11:25:26.531 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:25:26.466595+0000", 2024-09-13T11:25:26.531 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 0.053901143999999998, 2024-09-13T11:25:26.531 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.054006244000000002, 2024-09-13T11:25:26.531 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:25:26.531 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:25:26.531 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "waiting for ondisk", 2024-09-13T11:25:26.531 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:25:26.531 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:25:26.531 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:26.531 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 176492 2024-09-13T11:25:26.531 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.531 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:25:26.531 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.531 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:25:26.532 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:26.466595+0000", 2024-09-13T11:25:26.532 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:26.532 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.532 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.532 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:25:26.532 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:26.466595+0000", 2024-09-13T11:25:26.532 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:26.532 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.532 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.532 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:25:26.532 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:26.466596+0000", 2024-09-13T11:25:26.532 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.19e-06 2024-09-13T11:25:26.532 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.532 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.533 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:25:26.533 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:26.466601+0000", 2024-09-13T11:25:26.533 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.9080000000000003e-06 2024-09-13T11:25:26.533 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.533 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.533 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:25:26.533 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:26.466612+0000", 2024-09-13T11:25:26.533 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0766e-05 2024-09-13T11:25:26.533 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.533 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.533 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:25:26.533 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:26.466622+0000", 2024-09-13T11:25:26.533 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0383999999999999e-05 2024-09-13T11:25:26.533 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.533 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.534 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:25:26.534 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:26.466674+0000", 2024-09-13T11:25:26.534 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.1439e-05 2024-09-13T11:25:26.534 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.534 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.534 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for ondisk", 2024-09-13T11:25:26.534 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:26.466740+0000", 2024-09-13T11:25:26.534 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.6248999999999995e-05 2024-09-13T11:25:26.534 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:26.534 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:25:26.534 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:26.534 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:26.534 INFO:teuthology.orchestra.run.smithi162.stdout: ], 2024-09-13T11:25:26.534 INFO:teuthology.orchestra.run.smithi162.stdout: "num_ops": 3 2024-09-13T11:25:26.535 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-13T11:25:26.538 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-09-13T11:25:26.621 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-09-13T11:25:26.620+0000 7f97befc9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26315) UID: 0 2024-09-13T11:25:26.665 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-13T11:25:26.666 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [], 2024-09-13T11:25:26.666 INFO:teuthology.orchestra.run.smithi162.stdout: "complaint_time": 30, 2024-09-13T11:25:26.666 INFO:teuthology.orchestra.run.smithi162.stdout: "num_blocked_ops": 0 2024-09-13T11:25:26.666 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-13T11:25:26.683 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-09-13T11:25:26.722 INFO:tasks.ceph.osd.5.smithi092.stderr:2024-09-13T11:25:26.717+0000 7f6f0b705640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26346) UID: 0 2024-09-13T11:25:26.823 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-09-13T11:25:26.820+0000 7f97befc9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26315) UID: 0 2024-09-13T11:25:26.844 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-13T11:25:26.845 INFO:teuthology.orchestra.run.smithi001.stdout:{"fsid":"2fd68ac7-5581-4d90-80a4-30bcf5252a7a","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noscrub flag(s) set","count":7},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":209,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":167,"num_osds":16,"num_up_osds":16,"osd_up_since":1726226695,"num_in_osds":15,"osd_in_since":1726226668,"num_remapped_pgs":3},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":52},{"state_name":"active+recovering+remapped","count":3}],"num_pgs":55,"num_pools":3,"num_objects":107235,"data_bytes":7027982923,"bytes_used":16172290048,"bytes_avail":1433379172352,"bytes_total":1449551462400,"misplaced_objects":3138,"misplaced_total":214470,"misplaced_ratio":0.014631416981395999,"recovering_objects_per_sec":298,"recovering_bytes_per_sec":19558719,"recovering_keys_per_sec":0,"num_objects_recovered":1194,"num_bytes_recovered":78249984,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":900},"fsmap":{"epoch":1,"btime":"2024-09-13T11:21:57:222574+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-13T11:24:06.142821+0000","services":{}},"progress_events":{}} 2024-09-13T11:25:26.869 INFO:tasks.ceph.ceph_manager.ceph:making progress, resetting timeout 2024-09-13T11:25:26.869 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-13T11:25:26.870 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-13T11:25:26.870 INFO:teuthology.orchestra.run.smithi162.stdout: "size": 20, 2024-09-13T11:25:26.871 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 600, 2024-09-13T11:25:26.871 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [ 2024-09-13T11:25:26.871 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.871 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:45810 3.17 3.9bd13e57 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e69)", 2024-09-13T11:25:26.871 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:21.347960+0000", 2024-09-13T11:25:26.871 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 125.519338834, 2024-09-13T11:25:26.871 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.56814464200000003, 2024-09-13T11:25:26.871 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:25:26.871 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:25:26.871 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-09-13T11:25:26.871 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:25:26.871 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:25:26.871 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:26.872 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 45810 2024-09-13T11:25:26.872 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.872 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:25:26.872 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.872 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:25:26.872 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:21.347960+0000", 2024-09-13T11:25:26.872 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:26.872 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.872 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.872 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:25:26.872 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:21.347960+0000", 2024-09-13T11:25:26.872 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:26.872 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.872 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.873 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:25:26.873 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:21.347965+0000", 2024-09-13T11:25:26.873 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.6720000000000003e-06 2024-09-13T11:25:26.873 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.873 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.873 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:25:26.873 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:21.348019+0000", 2024-09-13T11:25:26.873 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.4391000000000002e-05 2024-09-13T11:25:26.873 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.873 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.873 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:25:26.873 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:21.348052+0000", 2024-09-13T11:25:26.873 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.2246000000000003e-05 2024-09-13T11:25:26.874 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.874 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.874 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:25:26.874 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:21.348072+0000", 2024-09-13T11:25:26.874 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.0622e-05 2024-09-13T11:25:26.874 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.874 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.874 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:25:26.874 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:21.348278+0000", 2024-09-13T11:25:26.874 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000205391 2024-09-13T11:25:26.874 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.874 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.874 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-09-13T11:25:26.874 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:21.348292+0000", 2024-09-13T11:25:26.875 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4759e-05 2024-09-13T11:25:26.875 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.875 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.875 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:25:26.875 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:21.916064+0000", 2024-09-13T11:25:26.875 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.56777128600000004 2024-09-13T11:25:26.875 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.875 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.875 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:25:26.875 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:21.916105+0000", 2024-09-13T11:25:26.875 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.1275e-05 2024-09-13T11:25:26.875 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:26.875 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:25:26.875 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:26.876 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.876 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.876 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:54653 3.23 3:c7a54800:::benchmark_data_smithi177_31116_object54652:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e75)", 2024-09-13T11:25:26.876 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:30.979059+0000", 2024-09-13T11:25:26.876 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 115.88824033500001, 2024-09-13T11:25:26.876 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.97284747100000002, 2024-09-13T11:25:26.876 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:25:26.876 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:25:26.876 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-09-13T11:25:26.876 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:25:26.876 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:25:26.876 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:26.876 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 54653 2024-09-13T11:25:26.877 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.877 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:25:26.877 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.877 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:25:26.877 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.979059+0000", 2024-09-13T11:25:26.877 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:26.877 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.877 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.877 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:25:26.877 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.979059+0000", 2024-09-13T11:25:26.877 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:26.877 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.877 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.877 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:25:26.878 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.979061+0000", 2024-09-13T11:25:26.878 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.4279999999999999e-06 2024-09-13T11:25:26.878 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.878 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.878 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:25:26.878 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.979114+0000", 2024-09-13T11:25:26.878 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.3254000000000002e-05 2024-09-13T11:25:26.878 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.878 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.878 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:25:26.878 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.979157+0000", 2024-09-13T11:25:26.878 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.2018000000000003e-05 2024-09-13T11:25:26.878 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.878 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.878 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:25:26.879 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.979172+0000", 2024-09-13T11:25:26.879 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5636000000000001e-05 2024-09-13T11:25:26.879 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.879 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.879 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:25:26.879 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:31.045109+0000", 2024-09-13T11:25:26.879 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.065936982000000005 2024-09-13T11:25:26.879 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.879 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.879 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-09-13T11:25:26.879 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:31.045121+0000", 2024-09-13T11:25:26.879 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1608e-05 2024-09-13T11:25:26.879 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.880 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.880 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:25:26.880 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:31.951636+0000", 2024-09-13T11:25:26.880 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.90651503899999997 2024-09-13T11:25:26.880 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.880 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.880 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:25:26.880 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:31.951906+0000", 2024-09-13T11:25:26.880 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000270506 2024-09-13T11:25:26.880 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:26.880 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:25:26.880 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:26.880 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.880 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.880 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPush(3.18 76/70 [PushOp(3:1a32c9cd:::benchmark_data_smithi177_31116_object12833:head, version: 49'818, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:1a32c9cd:::benchmark_data_smithi177_31116_object12833:head@49'818, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:25:26.881 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:32.522305+0000", 2024-09-13T11:25:26.881 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 114.344994251, 2024-09-13T11:25:26.881 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.40201207100000003, 2024-09-13T11:25:26.881 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:25:26.881 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:25:26.881 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:25:26.881 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:25:26.881 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.881 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:25:26.881 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.522305+0000", 2024-09-13T11:25:26.881 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:26.881 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.881 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.881 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:25:26.882 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.522305+0000", 2024-09-13T11:25:26.882 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:26.882 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.882 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.882 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:25:26.882 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.522309+0000", 2024-09-13T11:25:26.882 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.9829999999999998e-06 2024-09-13T11:25:26.882 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.882 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.882 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:25:26.882 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.522377+0000", 2024-09-13T11:25:26.882 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.8069000000000002e-05 2024-09-13T11:25:26.882 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.882 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.883 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:25:26.883 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.921950+0000", 2024-09-13T11:25:26.883 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.399573438 2024-09-13T11:25:26.883 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.883 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.883 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:25:26.883 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.921983+0000", 2024-09-13T11:25:26.883 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.2858000000000003e-05 2024-09-13T11:25:26.883 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.883 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.883 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:25:26.883 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.922389+0000", 2024-09-13T11:25:26.883 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00040612999999999999 2024-09-13T11:25:26.883 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.884 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.884 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:25:26.884 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.922421+0000", 2024-09-13T11:25:26.884 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.1356999999999998e-05 2024-09-13T11:25:26.884 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.884 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.884 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:25:26.884 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.924317+0000", 2024-09-13T11:25:26.884 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0018962359999999999 2024-09-13T11:25:26.884 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:26.884 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:25:26.884 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:26.884 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.884 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.885 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPush(3.17 76/70 [PushOp(3:ebcbce67:::benchmark_data_smithi177_31116_object18268:head, version: 53'1102, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:ebcbce67:::benchmark_data_smithi177_31116_object18268:head@53'1102, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:25:26.885 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:32.638908+0000", 2024-09-13T11:25:26.885 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 114.228390803, 2024-09-13T11:25:26.885 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.229064031, 2024-09-13T11:25:26.885 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:25:26.885 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:25:26.885 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:25:26.885 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:25:26.885 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.885 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:25:26.885 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.638908+0000", 2024-09-13T11:25:26.885 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:26.885 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.885 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.886 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:25:26.886 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.638908+0000", 2024-09-13T11:25:26.886 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:26.886 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.886 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.886 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:25:26.886 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.638910+0000", 2024-09-13T11:25:26.886 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.7379999999999999e-06 2024-09-13T11:25:26.886 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.886 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.886 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:25:26.886 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.638975+0000", 2024-09-13T11:25:26.886 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.4937000000000005e-05 2024-09-13T11:25:26.886 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.886 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.887 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:25:26.887 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.865860+0000", 2024-09-13T11:25:26.887 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.22688509800000001 2024-09-13T11:25:26.887 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.887 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.887 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:25:26.887 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.865890+0000", 2024-09-13T11:25:26.887 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.0273e-05 2024-09-13T11:25:26.887 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.887 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.887 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:25:26.887 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.866143+0000", 2024-09-13T11:25:26.887 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00025212800000000001 2024-09-13T11:25:26.887 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.888 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.888 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:25:26.888 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.866181+0000", 2024-09-13T11:25:26.888 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.8945000000000001e-05 2024-09-13T11:25:26.888 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.888 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.888 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:25:26.888 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.867972+0000", 2024-09-13T11:25:26.888 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001790912 2024-09-13T11:25:26.888 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:26.888 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:25:26.888 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:26.888 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.888 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.889 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:57435 3.3 3:c0db4777:::benchmark_data_smithi177_31116_object57434:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e80)", 2024-09-13T11:25:26.889 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:37.436034+0000", 2024-09-13T11:25:26.889 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 109.43126534300001, 2024-09-13T11:25:26.889 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.35949142000000001, 2024-09-13T11:25:26.889 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:25:26.889 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:25:26.889 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:26.889 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:25:26.889 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:25:26.889 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:26.889 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 57435 2024-09-13T11:25:26.889 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.889 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:25:26.889 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.890 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:25:26.890 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:37.436034+0000", 2024-09-13T11:25:26.890 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:26.890 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.890 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.890 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:25:26.890 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:37.436034+0000", 2024-09-13T11:25:26.890 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:26.890 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.890 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.890 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:25:26.890 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:37.436037+0000", 2024-09-13T11:25:26.890 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.7829999999999999e-06 2024-09-13T11:25:26.890 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.891 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.891 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:25:26.891 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:37.436090+0000", 2024-09-13T11:25:26.891 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.3665999999999997e-05 2024-09-13T11:25:26.891 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.891 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.891 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:25:26.891 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:37.436112+0000", 2024-09-13T11:25:26.891 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.1556000000000002e-05 2024-09-13T11:25:26.891 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.891 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.891 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:25:26.891 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:37.436122+0000", 2024-09-13T11:25:26.891 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0098000000000001e-05 2024-09-13T11:25:26.891 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.892 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.892 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:25:26.892 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:37.436187+0000", 2024-09-13T11:25:26.892 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.4857999999999995e-05 2024-09-13T11:25:26.892 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.892 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.892 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:25:26.892 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:37.436285+0000", 2024-09-13T11:25:26.892 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.8643999999999999e-05 2024-09-13T11:25:26.892 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.892 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.892 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 7", 2024-09-13T11:25:26.892 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:37.436423+0000", 2024-09-13T11:25:26.892 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00013809299999999999 2024-09-13T11:25:26.893 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.893 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.893 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:25:26.893 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:37.440125+0000", 2024-09-13T11:25:26.893 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0037015239999999999 2024-09-13T11:25:26.893 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.893 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.893 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:25:26.893 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:37.795397+0000", 2024-09-13T11:25:26.893 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.35527234400000002 2024-09-13T11:25:26.893 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.893 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.893 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:25:26.894 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:37.795485+0000", 2024-09-13T11:25:26.894 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.7538000000000001e-05 2024-09-13T11:25:26.894 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.894 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.894 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:25:26.894 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:37.795525+0000", 2024-09-13T11:25:26.894 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.0315999999999998e-05 2024-09-13T11:25:26.894 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:26.894 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:25:26.894 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:26.894 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.894 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.894 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPush(3.1f 80/78 [PushOp(3:fc3ed0c7:::benchmark_data_smithi177_31116_object672:head, version: 43'36, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:fc3ed0c7:::benchmark_data_smithi177_31116_object672:head@43'36, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:25:26.894 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:37.801754+0000", 2024-09-13T11:25:26.895 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 109.065544983, 2024-09-13T11:25:26.895 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.209613519, 2024-09-13T11:25:26.895 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:25:26.895 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:25:26.895 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:25:26.895 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:25:26.895 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.895 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:25:26.895 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:37.801754+0000", 2024-09-13T11:25:26.895 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:26.895 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.895 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.895 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:25:26.896 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:37.801754+0000", 2024-09-13T11:25:26.896 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:26.896 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.896 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.896 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:25:26.896 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:37.801756+0000", 2024-09-13T11:25:26.896 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.804e-06 2024-09-13T11:25:26.896 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.896 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.896 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:25:26.896 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:37.801947+0000", 2024-09-13T11:25:26.896 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000190636 2024-09-13T11:25:26.896 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.896 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.897 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:25:26.897 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:38.010478+0000", 2024-09-13T11:25:26.897 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.20853091100000001 2024-09-13T11:25:26.897 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.897 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.897 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:25:26.897 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:38.010506+0000", 2024-09-13T11:25:26.897 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.8793999999999999e-05 2024-09-13T11:25:26.897 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.897 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.897 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:25:26.897 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:38.010601+0000", 2024-09-13T11:25:26.897 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.4786999999999999e-05 2024-09-13T11:25:26.897 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.898 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.898 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:25:26.898 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:38.010617+0000", 2024-09-13T11:25:26.898 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5790999999999998e-05 2024-09-13T11:25:26.898 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.898 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.898 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:25:26.898 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:38.011368+0000", 2024-09-13T11:25:26.898 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00075079599999999997 2024-09-13T11:25:26.898 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:26.898 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:25:26.898 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:26.898 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.898 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.899 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:60433 3.3 3:c01f7ddc:::benchmark_data_smithi177_31116_object60432:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e85)", 2024-09-13T11:25:26.899 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:42.199382+0000", 2024-09-13T11:25:26.899 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 104.667917452, 2024-09-13T11:25:26.899 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.33918965200000001, 2024-09-13T11:25:26.899 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:25:26.899 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:25:26.899 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:26.899 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:25:26.899 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:25:26.899 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:26.899 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 60433 2024-09-13T11:25:26.899 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.899 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:25:26.899 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.900 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:25:26.900 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.199382+0000", 2024-09-13T11:25:26.900 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:26.900 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.900 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.900 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:25:26.900 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.199382+0000", 2024-09-13T11:25:26.900 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:26.900 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.900 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.900 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:25:26.900 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.199386+0000", 2024-09-13T11:25:26.900 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.8759999999999998e-06 2024-09-13T11:25:26.901 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.901 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.901 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:25:26.901 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.199455+0000", 2024-09-13T11:25:26.901 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.9305000000000001e-05 2024-09-13T11:25:26.901 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.901 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.901 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:25:26.901 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.199482+0000", 2024-09-13T11:25:26.901 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.6749000000000001e-05 2024-09-13T11:25:26.901 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.901 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.901 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:25:26.901 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.199503+0000", 2024-09-13T11:25:26.902 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.1137000000000001e-05 2024-09-13T11:25:26.902 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.902 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.902 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:25:26.902 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.199602+0000", 2024-09-13T11:25:26.902 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.8743e-05 2024-09-13T11:25:26.902 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.902 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.902 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:25:26.902 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.199784+0000", 2024-09-13T11:25:26.902 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00018265100000000001 2024-09-13T11:25:26.902 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.902 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.902 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 7", 2024-09-13T11:25:26.902 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.199955+0000", 2024-09-13T11:25:26.903 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00017067100000000001 2024-09-13T11:25:26.903 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.903 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.903 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:25:26.903 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.202075+0000", 2024-09-13T11:25:26.903 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0021206749999999998 2024-09-13T11:25:26.903 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.903 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.903 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:25:26.903 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.538395+0000", 2024-09-13T11:25:26.903 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.33631963599999998 2024-09-13T11:25:26.903 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.903 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.903 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:25:26.904 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.538498+0000", 2024-09-13T11:25:26.904 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00010267099999999999 2024-09-13T11:25:26.904 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.904 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.904 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:25:26.904 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.538571+0000", 2024-09-13T11:25:26.904 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.3538e-05 2024-09-13T11:25:26.904 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:26.904 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:25:26.904 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:26.904 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.904 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.904 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop_reply(client.4665.0:60433 3.3 e85/75 ondisk, result = 0)", 2024-09-13T11:25:26.904 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:42.203186+0000", 2024-09-13T11:25:26.905 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 104.664113274, 2024-09-13T11:25:26.905 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.33540096699999999, 2024-09-13T11:25:26.905 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:25:26.905 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:25:26.905 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:25:26.905 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:25:26.905 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.905 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:25:26.905 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.203186+0000", 2024-09-13T11:25:26.905 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:26.905 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.905 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.905 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:25:26.905 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.203186+0000", 2024-09-13T11:25:26.905 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:26.906 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.906 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.906 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:25:26.906 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.203194+0000", 2024-09-13T11:25:26.906 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.0020000000000006e-06 2024-09-13T11:25:26.906 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.906 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.906 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:25:26.906 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.203202+0000", 2024-09-13T11:25:26.906 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.9829999999999996e-06 2024-09-13T11:25:26.906 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.906 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.906 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:25:26.906 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.538182+0000", 2024-09-13T11:25:26.907 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.33498029899999998 2024-09-13T11:25:26.907 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.907 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.907 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:25:26.907 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.538196+0000", 2024-09-13T11:25:26.907 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3807e-05 2024-09-13T11:25:26.907 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.907 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.907 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:25:26.907 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.538371+0000", 2024-09-13T11:25:26.907 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00017541200000000001 2024-09-13T11:25:26.907 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.907 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.907 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:25:26.907 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.538386+0000", 2024-09-13T11:25:26.908 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4833e-05 2024-09-13T11:25:26.908 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.908 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.908 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:25:26.908 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.538587+0000", 2024-09-13T11:25:26.908 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000200631 2024-09-13T11:25:26.908 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:26.908 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:25:26.908 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:26.908 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.908 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.908 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:69556 3.18 3:1d3dbeba:::benchmark_data_smithi177_31116_object69555:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94)", 2024-09-13T11:25:26.908 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:52.142180+0000", 2024-09-13T11:25:26.908 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 94.725119203999995, 2024-09-13T11:25:26.909 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.21280995899999999, 2024-09-13T11:25:26.909 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:25:26.909 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:25:26.909 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:26.909 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:25:26.909 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:25:26.909 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:26.909 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 69556 2024-09-13T11:25:26.909 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.909 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:25:26.909 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.909 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:25:26.909 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:52.142180+0000", 2024-09-13T11:25:26.910 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:26.910 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.910 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.910 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:25:26.910 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:52.142180+0000", 2024-09-13T11:25:26.910 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:26.910 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.910 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.910 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:25:26.910 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:52.142181+0000", 2024-09-13T11:25:26.910 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.3699999999999999e-07 2024-09-13T11:25:26.910 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.910 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.910 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:25:26.910 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:52.142212+0000", 2024-09-13T11:25:26.911 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.1334000000000002e-05 2024-09-13T11:25:26.911 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.911 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.911 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:25:26.911 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:52.142219+0000", 2024-09-13T11:25:26.911 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.9990000000000002e-06 2024-09-13T11:25:26.911 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.911 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.911 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:25:26.911 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:52.142224+0000", 2024-09-13T11:25:26.911 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.0259999999999998e-06 2024-09-13T11:25:26.911 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.911 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.911 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:25:26.912 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:52.142276+0000", 2024-09-13T11:25:26.912 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.1436e-05 2024-09-13T11:25:26.912 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.912 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.912 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:25:26.912 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:52.142396+0000", 2024-09-13T11:25:26.912 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00012011 2024-09-13T11:25:26.912 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.912 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.912 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 0", 2024-09-13T11:25:26.912 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:52.142503+0000", 2024-09-13T11:25:26.912 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000107701 2024-09-13T11:25:26.912 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.912 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.912 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:25:26.913 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:52.144566+0000", 2024-09-13T11:25:26.913 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.002062749 2024-09-13T11:25:26.913 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.913 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.913 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:25:26.913 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:52.354839+0000", 2024-09-13T11:25:26.913 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.21027254000000001 2024-09-13T11:25:26.913 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.913 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.913 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:25:26.913 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:52.354932+0000", 2024-09-13T11:25:26.913 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.3510000000000004e-05 2024-09-13T11:25:26.913 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.913 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.914 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:25:26.914 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:52.354990+0000", 2024-09-13T11:25:26.914 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.7617000000000001e-05 2024-09-13T11:25:26.914 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:26.914 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:25:26.914 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:26.914 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.914 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.914 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:82784 3.18 3:1e10e57e:::benchmark_data_smithi177_31116_object82783:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105)", 2024-09-13T11:25:26.914 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:24:03.815559+0000", 2024-09-13T11:25:26.914 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 83.051739806, 2024-09-13T11:25:26.914 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.19711926199999999, 2024-09-13T11:25:26.914 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:25:26.914 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:25:26.915 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:26.915 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:25:26.915 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:25:26.915 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:26.915 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 82784 2024-09-13T11:25:26.915 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.915 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:25:26.915 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.915 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:25:26.915 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:03.815559+0000", 2024-09-13T11:25:26.915 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:26.915 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.915 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.915 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:25:26.915 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:03.815559+0000", 2024-09-13T11:25:26.916 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:26.916 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.916 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.916 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:25:26.916 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:03.815568+0000", 2024-09-13T11:25:26.916 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.2029999999999992e-06 2024-09-13T11:25:26.916 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.916 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.916 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:25:26.916 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:03.815618+0000", 2024-09-13T11:25:26.916 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.0285999999999999e-05 2024-09-13T11:25:26.916 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.916 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.916 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:25:26.917 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:03.815646+0000", 2024-09-13T11:25:26.917 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.7747e-05 2024-09-13T11:25:26.917 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.917 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.917 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:25:26.917 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:03.815660+0000", 2024-09-13T11:25:26.917 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4633e-05 2024-09-13T11:25:26.917 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.917 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.917 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:25:26.917 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:03.815781+0000", 2024-09-13T11:25:26.917 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000120831 2024-09-13T11:25:26.917 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.917 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.918 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:25:26.918 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:03.816015+0000", 2024-09-13T11:25:26.918 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00023364600000000001 2024-09-13T11:25:26.918 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.918 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.918 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 0", 2024-09-13T11:25:26.918 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:03.816241+0000", 2024-09-13T11:25:26.918 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00022667600000000001 2024-09-13T11:25:26.918 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.918 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.918 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:25:26.918 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:03.819236+0000", 2024-09-13T11:25:26.918 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0029943130000000002 2024-09-13T11:25:26.918 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.919 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.919 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:25:26.919 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.012545+0000", 2024-09-13T11:25:26.919 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.19330974100000001 2024-09-13T11:25:26.919 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.919 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.919 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:25:26.919 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.012623+0000", 2024-09-13T11:25:26.919 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.7565999999999996e-05 2024-09-13T11:25:26.919 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.919 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.919 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:25:26.919 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.012679+0000", 2024-09-13T11:25:26.919 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.5619999999999999e-05 2024-09-13T11:25:26.919 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:26.920 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:25:26.920 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:26.920 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.920 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.920 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:83144 3.18 3:1c2a5085:::benchmark_data_smithi177_31116_object83143:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106)", 2024-09-13T11:25:26.920 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:24:04.186395+0000", 2024-09-13T11:25:26.920 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 82.680904154999993, 2024-09-13T11:25:26.920 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.29655527599999998, 2024-09-13T11:25:26.920 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:25:26.920 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:25:26.920 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:26.920 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:25:26.920 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:25:26.920 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:26.920 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 83144 2024-09-13T11:25:26.921 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.921 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:25:26.921 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.921 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:25:26.921 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.186395+0000", 2024-09-13T11:25:26.921 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:26.921 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.921 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.921 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:25:26.921 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.186395+0000", 2024-09-13T11:25:26.921 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:26.921 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.921 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.921 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:25:26.922 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.186398+0000", 2024-09-13T11:25:26.922 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.5299999999999999e-06 2024-09-13T11:25:26.922 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.922 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.922 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:25:26.922 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.186438+0000", 2024-09-13T11:25:26.922 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.0293000000000002e-05 2024-09-13T11:25:26.922 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.922 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.922 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:25:26.922 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.186450+0000", 2024-09-13T11:25:26.922 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2687000000000001e-05 2024-09-13T11:25:26.922 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.922 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.923 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:25:26.923 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.186456+0000", 2024-09-13T11:25:26.923 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.446e-06 2024-09-13T11:25:26.923 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.923 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.923 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:25:26.923 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.186503+0000", 2024-09-13T11:25:26.923 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.7491000000000003e-05 2024-09-13T11:25:26.923 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.923 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.923 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:25:26.923 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.186618+0000", 2024-09-13T11:25:26.923 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000114999 2024-09-13T11:25:26.923 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.924 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.924 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 0", 2024-09-13T11:25:26.924 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.186712+0000", 2024-09-13T11:25:26.924 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.3805000000000001e-05 2024-09-13T11:25:26.924 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.924 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.924 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:25:26.924 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.189753+0000", 2024-09-13T11:25:26.924 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.003040713 2024-09-13T11:25:26.924 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.924 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.924 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:25:26.924 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.482662+0000", 2024-09-13T11:25:26.924 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.29290945699999998 2024-09-13T11:25:26.925 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.925 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.925 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:25:26.925 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.482831+0000", 2024-09-13T11:25:26.925 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00016900699999999999 2024-09-13T11:25:26.925 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.925 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.925 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:25:26.925 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.482950+0000", 2024-09-13T11:25:26.925 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000118848 2024-09-13T11:25:26.925 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:26.925 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:25:26.925 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:26.925 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.926 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.926 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop_reply(client.4665.0:83144 3.18 e106/91 ondisk, result = 0)", 2024-09-13T11:25:26.926 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:24:04.188413+0000", 2024-09-13T11:25:26.926 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 82.678886145000007, 2024-09-13T11:25:26.926 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.294557562, 2024-09-13T11:25:26.926 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:25:26.926 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:25:26.926 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:25:26.926 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:25:26.926 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.926 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:25:26.926 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.188413+0000", 2024-09-13T11:25:26.926 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:26.926 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.926 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.927 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:25:26.927 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.188413+0000", 2024-09-13T11:25:26.927 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:26.927 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.927 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.927 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:25:26.927 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.188414+0000", 2024-09-13T11:25:26.927 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4309999999999999e-06 2024-09-13T11:25:26.927 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.927 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.927 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:25:26.927 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.188419+0000", 2024-09-13T11:25:26.927 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.2889999999999999e-06 2024-09-13T11:25:26.928 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.928 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.928 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:25:26.928 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.482442+0000", 2024-09-13T11:25:26.928 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.294023168 2024-09-13T11:25:26.928 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.928 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.928 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:25:26.928 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.482459+0000", 2024-09-13T11:25:26.928 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.7013000000000001e-05 2024-09-13T11:25:26.928 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.928 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.928 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:25:26.929 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.482615+0000", 2024-09-13T11:25:26.929 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00015620400000000001 2024-09-13T11:25:26.929 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.929 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.929 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:25:26.929 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.482644+0000", 2024-09-13T11:25:26.929 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.9093000000000001e-05 2024-09-13T11:25:26.929 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.929 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.929 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:25:26.929 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.482971+0000", 2024-09-13T11:25:26.929 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00032636399999999999 2024-09-13T11:25:26.929 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:26.930 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:25:26.930 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:26.930 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.930 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.930 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPush(3.1f 106/78 [PushOp(3:fd4be819:::benchmark_data_smithi177_31116_object54560:head, version: 74'3410, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:fd4be819:::benchmark_data_smithi177_31116_object54560:head@74'3410, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:25:26.930 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:24:04.397161+0000", 2024-09-13T11:25:26.930 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 82.470137735999998, 2024-09-13T11:25:26.930 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.51611096199999995, 2024-09-13T11:25:26.930 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:25:26.930 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:25:26.930 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:25:26.930 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:25:26.930 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.930 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:25:26.931 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.397161+0000", 2024-09-13T11:25:26.931 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:26.931 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.931 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.931 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:25:26.931 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.397161+0000", 2024-09-13T11:25:26.931 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:26.931 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.931 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.931 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:25:26.931 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.397163+0000", 2024-09-13T11:25:26.931 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4720000000000001e-06 2024-09-13T11:25:26.931 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.931 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.931 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:25:26.932 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.397228+0000", 2024-09-13T11:25:26.932 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.5141999999999994e-05 2024-09-13T11:25:26.932 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.932 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.932 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:25:26.932 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.911583+0000", 2024-09-13T11:25:26.932 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.51435473700000001 2024-09-13T11:25:26.932 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.932 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.932 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:25:26.932 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.911615+0000", 2024-09-13T11:25:26.932 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.2620000000000003e-05 2024-09-13T11:25:26.932 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.932 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.933 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:25:26.933 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.911759+0000", 2024-09-13T11:25:26.933 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00014376300000000001 2024-09-13T11:25:26.933 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.933 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.933 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:25:26.933 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.911779+0000", 2024-09-13T11:25:26.933 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.0003000000000001e-05 2024-09-13T11:25:26.933 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.933 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.933 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:25:26.933 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.913272+0000", 2024-09-13T11:25:26.933 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0014932249999999999 2024-09-13T11:25:26.933 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:26.934 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:25:26.934 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:26.934 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.934 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.934 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:84809 3.17 3:e9e52123:::benchmark_data_smithi177_31116_object84808:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-09-13T11:25:26.934 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:24:05.914353+0000", 2024-09-13T11:25:26.934 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 80.952945717999995, 2024-09-13T11:25:26.934 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.27210915899999999, 2024-09-13T11:25:26.934 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:25:26.934 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:25:26.934 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:26.934 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:25:26.934 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:25:26.935 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:26.935 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 84809 2024-09-13T11:25:26.935 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.935 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:25:26.935 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.935 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:25:26.935 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.914353+0000", 2024-09-13T11:25:26.935 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:26.935 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.935 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.935 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:25:26.935 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.914353+0000", 2024-09-13T11:25:26.935 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:26.935 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.935 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.936 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:25:26.936 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.914356+0000", 2024-09-13T11:25:26.936 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.8710000000000001e-06 2024-09-13T11:25:26.936 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.936 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.936 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:25:26.936 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.914443+0000", 2024-09-13T11:25:26.936 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.7002000000000005e-05 2024-09-13T11:25:26.936 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.936 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.936 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:25:26.936 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.914461+0000", 2024-09-13T11:25:26.936 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.7691000000000001e-05 2024-09-13T11:25:26.936 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.936 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.937 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:25:26.937 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.914471+0000", 2024-09-13T11:25:26.937 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0447e-05 2024-09-13T11:25:26.937 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.937 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.937 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:25:26.937 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.914550+0000", 2024-09-13T11:25:26.937 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.8683000000000005e-05 2024-09-13T11:25:26.937 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.937 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.937 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:25:26.937 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.914709+0000", 2024-09-13T11:25:26.937 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00015913200000000001 2024-09-13T11:25:26.938 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.938 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.938 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 3", 2024-09-13T11:25:26.938 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.914880+0000", 2024-09-13T11:25:26.938 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00017071499999999999 2024-09-13T11:25:26.938 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.938 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.938 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:25:26.938 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.916701+0000", 2024-09-13T11:25:26.938 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0018205369999999999 2024-09-13T11:25:26.938 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.938 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.938 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:25:26.939 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:06.186401+0000", 2024-09-13T11:25:26.939 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.26970018699999998 2024-09-13T11:25:26.939 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.939 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.939 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:25:26.939 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:06.186434+0000", 2024-09-13T11:25:26.939 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.3448999999999998e-05 2024-09-13T11:25:26.939 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.939 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.939 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:25:26.939 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:06.186463+0000", 2024-09-13T11:25:26.939 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.8445e-05 2024-09-13T11:25:26.939 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:26.939 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:25:26.939 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:26.940 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.940 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.940 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop_reply(client.4665.0:84809 3.17 e107/85 ondisk, result = 0)", 2024-09-13T11:25:26.940 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:24:05.916779+0000", 2024-09-13T11:25:26.940 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 80.950520247, 2024-09-13T11:25:26.940 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.26968993499999999, 2024-09-13T11:25:26.940 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:25:26.940 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:25:26.940 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:25:26.940 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:25:26.940 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.940 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:25:26.940 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.916779+0000", 2024-09-13T11:25:26.940 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:26.940 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.941 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.941 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:25:26.941 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.916779+0000", 2024-09-13T11:25:26.941 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:26.941 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.941 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.941 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:25:26.941 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.916780+0000", 2024-09-13T11:25:26.941 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.328e-06 2024-09-13T11:25:26.941 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.941 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.941 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:25:26.941 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.916789+0000", 2024-09-13T11:25:26.941 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.4460000000000001e-06 2024-09-13T11:25:26.942 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.942 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.942 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:25:26.942 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:06.186296+0000", 2024-09-13T11:25:26.942 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.26950707099999999 2024-09-13T11:25:26.942 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.942 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.942 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:25:26.942 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:06.186306+0000", 2024-09-13T11:25:26.942 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0137e-05 2024-09-13T11:25:26.942 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.942 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.942 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:25:26.942 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:06.186388+0000", 2024-09-13T11:25:26.943 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.2149000000000002e-05 2024-09-13T11:25:26.943 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.943 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.943 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:25:26.943 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:06.186397+0000", 2024-09-13T11:25:26.943 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.5210000000000002e-06 2024-09-13T11:25:26.943 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.943 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.943 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:25:26.943 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:06.186469+0000", 2024-09-13T11:25:26.943 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.2282999999999999e-05 2024-09-13T11:25:26.943 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:26.943 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:25:26.943 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:26.943 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.944 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.944 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPush(3.2a 121/120 [PushOp(3:5595108b:::benchmark_data_smithi177_31116_object899:head, version: 43'62, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:5595108b:::benchmark_data_smithi177_31116_object899:head@43'62, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:25:26.944 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:24:27.352355+0000", 2024-09-13T11:25:26.944 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 59.514944440999997, 2024-09-13T11:25:26.944 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.61628666099999996, 2024-09-13T11:25:26.944 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:25:26.944 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:25:26.944 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:25:26.944 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:25:26.944 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.944 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:25:26.944 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:27.352355+0000", 2024-09-13T11:25:26.944 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:26.944 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.945 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.945 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:25:26.945 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:27.352355+0000", 2024-09-13T11:25:26.945 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:26.945 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.945 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.945 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:25:26.945 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:27.352357+0000", 2024-09-13T11:25:26.945 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.328e-06 2024-09-13T11:25:26.945 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.945 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.945 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:25:26.945 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:27.352428+0000", 2024-09-13T11:25:26.945 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.1118999999999995e-05 2024-09-13T11:25:26.945 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.946 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.946 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:25:26.946 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:27.966950+0000", 2024-09-13T11:25:26.946 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.61452202499999997 2024-09-13T11:25:26.946 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.946 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.946 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:25:26.946 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:27.966980+0000", 2024-09-13T11:25:26.946 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.9433000000000001e-05 2024-09-13T11:25:26.946 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.946 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.946 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:25:26.946 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:27.967220+0000", 2024-09-13T11:25:26.946 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00024065499999999999 2024-09-13T11:25:26.947 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.947 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.947 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:25:26.947 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:27.967244+0000", 2024-09-13T11:25:26.947 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.4216e-05 2024-09-13T11:25:26.947 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.947 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.947 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:25:26.947 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:27.968641+0000", 2024-09-13T11:25:26.947 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0013968850000000001 2024-09-13T11:25:26.947 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:26.947 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:25:26.947 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:26.948 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.948 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.948 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:162057 3.1f 3:f8710094:::benchmark_data_smithi177_31116_object21754:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e156)", 2024-09-13T11:25:26.948 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:25:15.442781+0000", 2024-09-13T11:25:26.948 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 11.424517798, 2024-09-13T11:25:26.948 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.25670282500000002, 2024-09-13T11:25:26.948 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:25:26.948 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:25:26.948 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:26.948 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:25:26.949 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:25:26.949 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:26.949 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 162057 2024-09-13T11:25:26.949 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.949 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:25:26.949 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.949 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:25:26.949 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:15.442781+0000", 2024-09-13T11:25:26.949 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:26.949 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.949 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.949 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:25:26.949 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:15.442781+0000", 2024-09-13T11:25:26.949 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:26.950 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.950 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.950 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:25:26.950 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:15.442785+0000", 2024-09-13T11:25:26.950 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.4000000000000001e-06 2024-09-13T11:25:26.950 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.950 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.950 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:25:26.950 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:15.442795+0000", 2024-09-13T11:25:26.950 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0193999999999999e-05 2024-09-13T11:25:26.950 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.950 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.950 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:25:26.950 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:15.442820+0000", 2024-09-13T11:25:26.951 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.4683000000000001e-05 2024-09-13T11:25:26.951 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.951 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.951 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:25:26.951 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:15.442840+0000", 2024-09-13T11:25:26.951 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.0596000000000001e-05 2024-09-13T11:25:26.951 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.951 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.951 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:25:26.951 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:15.442952+0000", 2024-09-13T11:25:26.951 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00011215899999999999 2024-09-13T11:25:26.951 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.951 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.952 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:25:26.952 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:15.443143+0000", 2024-09-13T11:25:26.952 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00019058899999999999 2024-09-13T11:25:26.952 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.952 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.952 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 5", 2024-09-13T11:25:26.952 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:15.443327+0000", 2024-09-13T11:25:26.952 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00018364499999999999 2024-09-13T11:25:26.952 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.952 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.952 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:25:26.952 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:15.446727+0000", 2024-09-13T11:25:26.952 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0034000689999999999 2024-09-13T11:25:26.952 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.953 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.953 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:25:26.953 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:15.699356+0000", 2024-09-13T11:25:26.953 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.25262958400000002 2024-09-13T11:25:26.953 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.953 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.953 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:25:26.953 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:15.699428+0000", 2024-09-13T11:25:26.953 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.1810000000000005e-05 2024-09-13T11:25:26.953 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.953 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.953 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:25:26.953 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:15.699484+0000", 2024-09-13T11:25:26.953 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.6096e-05 2024-09-13T11:25:26.954 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:26.954 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:25:26.954 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:26.954 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.954 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.954 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:162073 3.1f 3:fd1a00ae:::benchmark_data_smithi177_31116_object21770:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e156)", 2024-09-13T11:25:26.954 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:25:15.454353+0000", 2024-09-13T11:25:26.954 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 11.412945842999999, 2024-09-13T11:25:26.954 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.27516647, 2024-09-13T11:25:26.954 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:25:26.954 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:25:26.954 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:26.954 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:25:26.955 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:25:26.955 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:26.955 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 162073 2024-09-13T11:25:26.955 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.955 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:25:26.955 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.955 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:25:26.955 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:15.454353+0000", 2024-09-13T11:25:26.955 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:26.955 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.955 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.955 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:25:26.955 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:15.454353+0000", 2024-09-13T11:25:26.956 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:26.956 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.956 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.956 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:25:26.956 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:15.454358+0000", 2024-09-13T11:25:26.956 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.3499999999999999e-06 2024-09-13T11:25:26.956 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.956 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.956 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:25:26.956 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:15.454371+0000", 2024-09-13T11:25:26.956 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3594000000000001e-05 2024-09-13T11:25:26.956 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.956 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.956 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:25:26.957 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:15.454408+0000", 2024-09-13T11:25:26.957 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.6619999999999998e-05 2024-09-13T11:25:26.957 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.957 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.957 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:25:26.957 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:15.454431+0000", 2024-09-13T11:25:26.957 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.3546000000000001e-05 2024-09-13T11:25:26.957 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.957 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.957 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:25:26.957 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:15.454587+0000", 2024-09-13T11:25:26.957 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000155248 2024-09-13T11:25:26.957 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.958 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.958 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:25:26.958 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:15.454897+0000", 2024-09-13T11:25:26.958 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000309955 2024-09-13T11:25:26.958 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.958 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.958 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 5", 2024-09-13T11:25:26.958 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:15.455203+0000", 2024-09-13T11:25:26.958 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00030604999999999999 2024-09-13T11:25:26.958 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.958 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.958 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:25:26.958 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:15.460534+0000", 2024-09-13T11:25:26.958 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0053310199999999997 2024-09-13T11:25:26.959 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.959 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.959 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:25:26.959 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:15.729356+0000", 2024-09-13T11:25:26.959 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.26882267199999998 2024-09-13T11:25:26.959 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.959 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.959 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:25:26.959 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:15.729442+0000", 2024-09-13T11:25:26.959 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.5209000000000004e-05 2024-09-13T11:25:26.959 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.959 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.959 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:25:26.959 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:15.729520+0000", 2024-09-13T11:25:26.960 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.8206000000000003e-05 2024-09-13T11:25:26.960 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:26.960 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:25:26.960 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:26.960 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.960 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.960 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:162102 3.1f 3:fc0cda6c:::benchmark_data_smithi177_31116_object21799:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e156)", 2024-09-13T11:25:26.960 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:25:15.477044+0000", 2024-09-13T11:25:26.960 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 11.390254989000001, 2024-09-13T11:25:26.960 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.28798895800000002, 2024-09-13T11:25:26.960 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:25:26.960 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:25:26.960 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:26.960 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:25:26.961 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:25:26.961 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:26.961 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 162102 2024-09-13T11:25:26.961 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.961 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:25:26.961 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.961 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:25:26.961 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:15.477044+0000", 2024-09-13T11:25:26.961 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:26.961 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.961 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.961 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:25:26.961 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:15.477044+0000", 2024-09-13T11:25:26.961 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:26.961 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.962 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.962 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:25:26.962 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:15.477048+0000", 2024-09-13T11:25:26.962 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.3569999999999998e-06 2024-09-13T11:25:26.962 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.962 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.962 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:25:26.962 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:15.477058+0000", 2024-09-13T11:25:26.962 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.9899999999999992e-06 2024-09-13T11:25:26.962 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.962 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.962 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:25:26.962 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:15.477087+0000", 2024-09-13T11:25:26.962 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.9026999999999998e-05 2024-09-13T11:25:26.963 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.963 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.963 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:25:26.963 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:15.477101+0000", 2024-09-13T11:25:26.963 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4583000000000001e-05 2024-09-13T11:25:26.963 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.963 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.963 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:25:26.963 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:15.477212+0000", 2024-09-13T11:25:26.963 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00011071200000000001 2024-09-13T11:25:26.963 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.963 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.963 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:25:26.963 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:15.477492+0000", 2024-09-13T11:25:26.964 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00027995400000000001 2024-09-13T11:25:26.964 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.964 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.964 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 5", 2024-09-13T11:25:26.964 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:15.477682+0000", 2024-09-13T11:25:26.964 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000189929 2024-09-13T11:25:26.964 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.964 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.964 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:25:26.964 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:15.481903+0000", 2024-09-13T11:25:26.964 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0042214649999999998 2024-09-13T11:25:26.964 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.964 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.965 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:25:26.965 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:15.764891+0000", 2024-09-13T11:25:26.965 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.28298753599999998 2024-09-13T11:25:26.965 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.965 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.965 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:25:26.965 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:15.764972+0000", 2024-09-13T11:25:26.965 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.1230000000000007e-05 2024-09-13T11:25:26.965 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.965 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.965 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:25:26.965 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:15.765033+0000", 2024-09-13T11:25:26.965 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.1174999999999996e-05 2024-09-13T11:25:26.965 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:26.965 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:25:26.966 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:26.966 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.966 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.966 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop_reply(client.4665.0:162057 3.1f e156/122 ondisk, result = 0)", 2024-09-13T11:25:26.966 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:25:15.479734+0000", 2024-09-13T11:25:26.966 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 11.387564891, 2024-09-13T11:25:26.966 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.21975831800000001, 2024-09-13T11:25:26.966 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:25:26.966 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:25:26.966 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:25:26.966 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:25:26.966 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.966 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:25:26.966 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:15.479734+0000", 2024-09-13T11:25:26.967 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:26.967 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.967 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.967 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:25:26.967 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:15.479734+0000", 2024-09-13T11:25:26.967 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:26.967 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.967 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.967 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:25:26.967 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:15.479745+0000", 2024-09-13T11:25:26.967 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.084e-05 2024-09-13T11:25:26.967 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.967 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.967 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:25:26.968 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:15.479757+0000", 2024-09-13T11:25:26.968 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1600000000000001e-05 2024-09-13T11:25:26.968 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.968 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.968 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:25:26.968 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:15.698724+0000", 2024-09-13T11:25:26.968 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.218967044 2024-09-13T11:25:26.968 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.968 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.968 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:25:26.968 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:15.698740+0000", 2024-09-13T11:25:26.968 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.6707000000000001e-05 2024-09-13T11:25:26.968 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.968 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.969 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:25:26.969 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:15.699336+0000", 2024-09-13T11:25:26.969 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00059547500000000004 2024-09-13T11:25:26.969 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.969 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.969 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:25:26.969 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:15.699348+0000", 2024-09-13T11:25:26.969 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1703e-05 2024-09-13T11:25:26.969 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:26.969 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:26.969 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:25:26.969 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:15.699493+0000", 2024-09-13T11:25:26.969 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00014494900000000001 2024-09-13T11:25:26.969 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:26.969 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:25:26.970 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:26.970 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:26.970 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:25:26.970 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-13T11:25:26.971 INFO:tasks.ceph.osd.12.smithi001.stderr:2024-09-13T11:25:26.935+0000 7f891f10f640 -1 osd.12 167 handle_osd_map injecting map crc failure 2024-09-13T11:25:26.971 INFO:tasks.ceph.osd.7.smithi177.stderr:2024-09-13T11:25:26.966+0000 7f897b51d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26575) UID: 0 2024-09-13T11:25:26.972 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-13T11:25:27.072 INFO:tasks.ceph.osd.3.smithi177.stderr:2024-09-13T11:25:27.070+0000 7f10adba4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26574) UID: 0 2024-09-13T11:25:27.126 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-09-13T11:25:27.126 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [], 2024-09-13T11:25:27.126 INFO:teuthology.orchestra.run.smithi177.stdout: "num_ops": 0 2024-09-13T11:25:27.126 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-09-13T11:25:27.147 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-13T11:25:27.188 INFO:tasks.ceph.osd.7.smithi177.stderr:2024-09-13T11:25:27.186+0000 7f897b51d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26575) UID: 0 2024-09-13T11:25:27.274 INFO:tasks.ceph.osd.9.smithi092.stderr:2024-09-13T11:25:27.269+0000 7ff5ab3d0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26348) UID: 0 2024-09-13T11:25:27.323 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-09-13T11:25:27.323 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [], 2024-09-13T11:25:27.323 INFO:teuthology.orchestra.run.smithi177.stdout: "complaint_time": 30, 2024-09-13T11:25:27.324 INFO:teuthology.orchestra.run.smithi177.stdout: "num_blocked_ops": 0 2024-09-13T11:25:27.324 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-09-13T11:25:27.344 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-13T11:25:27.357 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-13T11:25:27.358 INFO:teuthology.orchestra.run.smithi001.stderr:dumped all 2024-09-13T11:25:27.377 INFO:tasks.ceph.osd.13.smithi092.stderr:2024-09-13T11:25:27.373+0000 7f3e5afea640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26350) UID: 0 2024-09-13T11:25:27.377 INFO:teuthology.orchestra.run.smithi001.stdout:{"pg_ready":true,"pg_map":{"version":252,"stamp":"2024-09-13T11:25:26.931782+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":6946718283,"num_objects":105995,"num_object_clones":0,"num_object_copies":211990,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3138,"num_objects_unfound":0,"num_objects_dirty":105995,"num_whiteouts":0,"num_read":87,"num_read_kb":74,"num_write":90852,"num_write_kb":3637107,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20647,"num_bytes_recovered":1352990739,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":284204,"ondisk_log_size":284204,"up":110,"acting":110,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":123,"num_osds":15,"num_per_pool_osds":15,"num_per_pool_omap_osds":15,"kb":1415577600,"kb_used":15648668,"kb_used_data":14299884,"kb_used_omap":26,"kb_used_meta":1348453,"kb_avail":1399928932,"statfs":{"total":1449551462400,"available":1433527226368,"internally_reserved":0,"allocated":14643081216,"data_stored":14632293906,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":26763,"internal_metadata":1380816757},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-317456384,"num_objects":-4844,"num_object_clones":0,"num_object_copies":-9688,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-2388,"num_objects_unfound":0,"num_objects_dirty":-4844,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4844,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1194,"num_bytes_recovered":78249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"3.740153"},"pg_stats":[{"pgid":"3.2d","version":"165'6141","reported_seq":9113,"reported_epoch":165,"state":"active+recovering+remapped","last_fresh":"2024-09-13T11:25:25.101919+0000","last_change":"2024-09-13T11:25:19.656321+0000","last_active":"2024-09-13T11:25:25.101919+0000","last_peered":"2024-09-13T11:25:25.101919+0000","last_clean":"2024-09-13T11:24:24.974950+0000","last_became_active":"2024-09-13T11:24:27.227288+0000","last_became_peered":"2024-09-13T11:24:27.227288+0000","last_unstale":"2024-09-13T11:25:25.101919+0000","last_undegraded":"2024-09-13T11:25:25.101919+0000","last_fullsized":"2024-09-13T11:25:25.101919+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":6,"last_scrub":"76'3395","last_scrub_stamp":"2024-09-13T11:23:32.776747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:32.776747+0000","objects_scrubbed":1704,"log_size":6141,"log_dups_size":0,"ondisk_log_size":6141,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":179,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110985216,"num_objects":1694,"num_object_clones":0,"num_object_copies":3388,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1674,"num_objects_unfound":0,"num_objects_dirty":1694,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1904,"num_write_kb":87840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":882,"num_bytes_recovered":57802752,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[3,0],"avail_no_missing":["3","7"],"object_location_counts":[{"shards":"0,3","objects":857},{"shards":"3,7","objects":837}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2c","version":"166'6222","reported_seq":11554,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-13T11:25:25.967987+0000","last_change":"2024-09-13T11:24:59.456762+0000","last_active":"2024-09-13T11:25:25.967987+0000","last_peered":"2024-09-13T11:25:25.967987+0000","last_clean":"2024-09-13T11:25:25.967987+0000","last_became_active":"2024-09-13T11:24:59.021895+0000","last_became_peered":"2024-09-13T11:24:59.021895+0000","last_unstale":"2024-09-13T11:25:25.967987+0000","last_undegraded":"2024-09-13T11:25:25.967987+0000","last_fullsized":"2024-09-13T11:25:25.967987+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":6,"last_scrub":"145'5672","last_scrub_stamp":"2024-09-13T11:24:59.456653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:24:59.456653+0000","objects_scrubbed":2192,"log_size":6222,"log_dups_size":0,"ondisk_log_size":6222,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:26:20.877991+0000","scrub_duration":279,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":107610112,"num_objects":1642,"num_object_clones":0,"num_object_copies":3284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1642,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":550,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,7],"acting":[10,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.2b","version":"166'6105","reported_seq":9794,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-13T11:25:25.244306+0000","last_change":"2024-09-13T11:24:45.271477+0000","last_active":"2024-09-13T11:25:25.244306+0000","last_peered":"2024-09-13T11:25:25.244306+0000","last_clean":"2024-09-13T11:25:25.244306+0000","last_became_active":"2024-09-13T11:24:26.180578+0000","last_became_peered":"2024-09-13T11:24:26.180578+0000","last_unstale":"2024-09-13T11:25:25.244306+0000","last_undegraded":"2024-09-13T11:25:25.244306+0000","last_fullsized":"2024-09-13T11:25:25.244306+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":6,"last_scrub":"131'5382","last_scrub_stamp":"2024-09-13T11:24:45.271376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:24:45.271376+0000","objects_scrubbed":1940,"log_size":6105,"log_dups_size":0,"ondisk_log_size":6105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:59.210895+0000","scrub_duration":279,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":101515264,"num_objects":1549,"num_object_clones":0,"num_object_copies":3098,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1549,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":723,"num_write_kb":10624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2a","version":"166'6221","reported_seq":10097,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-13T11:25:26.035064+0000","last_change":"2024-09-13T11:24:46.625117+0000","last_active":"2024-09-13T11:25:26.035064+0000","last_peered":"2024-09-13T11:25:26.035064+0000","last_clean":"2024-09-13T11:25:26.035064+0000","last_became_active":"2024-09-13T11:24:45.320540+0000","last_became_peered":"2024-09-13T11:24:45.320540+0000","last_unstale":"2024-09-13T11:25:26.035064+0000","last_undegraded":"2024-09-13T11:25:26.035064+0000","last_fullsized":"2024-09-13T11:25:26.035064+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":6,"last_scrub":"133'5491","last_scrub_stamp":"2024-09-13T11:24:46.624970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:24:46.624970+0000","objects_scrubbed":2094,"log_size":6221,"log_dups_size":0,"ondisk_log_size":6221,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:26:04.813280+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112197632,"num_objects":1712,"num_object_clones":0,"num_object_copies":3424,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1712,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":732,"num_write_kb":11200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,7],"acting":[10,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.29","version":"166'5881","reported_seq":11572,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-13T11:25:25.960499+0000","last_change":"2024-09-13T11:25:13.333062+0000","last_active":"2024-09-13T11:25:25.960499+0000","last_peered":"2024-09-13T11:25:25.960499+0000","last_clean":"2024-09-13T11:25:25.960499+0000","last_became_active":"2024-09-13T11:25:12.944028+0000","last_became_peered":"2024-09-13T11:25:12.944028+0000","last_unstale":"2024-09-13T11:25:25.960499+0000","last_undegraded":"2024-09-13T11:25:25.960499+0000","last_fullsized":"2024-09-13T11:25:25.960499+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":6,"last_scrub":"154'5579","last_scrub_stamp":"2024-09-13T11:25:13.332959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:13.332959+0000","objects_scrubbed":1937,"log_size":5881,"log_dups_size":0,"ondisk_log_size":5881,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:26:24.089228+0000","scrub_duration":243,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":107020288,"num_objects":1633,"num_object_clones":0,"num_object_copies":3266,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1633,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":304,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.28","version":"166'5729","reported_seq":9507,"reported_epoch":166,"state":"active+recovering+remapped","last_fresh":"2024-09-13T11:25:25.386902+0000","last_change":"2024-09-13T11:25:11.182075+0000","last_active":"2024-09-13T11:25:25.386902+0000","last_peered":"2024-09-13T11:25:25.386902+0000","last_clean":"2024-09-13T11:24:25.005266+0000","last_became_active":"2024-09-13T11:24:27.218508+0000","last_became_peered":"2024-09-13T11:24:27.218508+0000","last_unstale":"2024-09-13T11:25:25.386902+0000","last_undegraded":"2024-09-13T11:25:25.386902+0000","last_fullsized":"2024-09-13T11:25:25.386902+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":6,"last_scrub":"68'2771","last_scrub_stamp":"2024-09-13T11:23:20.644747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:20.644747+0000","objects_scrubbed":1356,"log_size":5729,"log_dups_size":0,"ondisk_log_size":5729,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":411,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":107872256,"num_objects":1646,"num_object_clones":0,"num_object_copies":3292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1072,"num_objects_unfound":0,"num_objects_dirty":1646,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1982,"num_write_kb":94400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1150,"num_bytes_recovered":75366400,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5],"acting":[1,5],"avail_no_missing":["1","6"],"object_location_counts":[{"shards":"1,5","objects":1110},{"shards":"1,6","objects":536}],"blocked_by":[],"up_primary":8,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.27","version":"166'5812","reported_seq":10730,"reported_epoch":166,"state":"active+recovering+remapped","last_fresh":"2024-09-13T11:25:26.113633+0000","last_change":"2024-09-13T11:25:08.115804+0000","last_active":"2024-09-13T11:25:26.113633+0000","last_peered":"2024-09-13T11:25:26.113633+0000","last_clean":"2024-09-13T11:24:24.942332+0000","last_became_active":"2024-09-13T11:24:27.215734+0000","last_became_peered":"2024-09-13T11:24:27.215734+0000","last_unstale":"2024-09-13T11:25:26.113633+0000","last_undegraded":"2024-09-13T11:25:26.113633+0000","last_fullsized":"2024-09-13T11:25:26.113633+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":6,"last_scrub":"68'2736","last_scrub_stamp":"2024-09-13T11:23:20.443329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:20.443329+0000","objects_scrubbed":99,"log_size":5812,"log_dups_size":0,"ondisk_log_size":5812,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":183,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":108920832,"num_objects":1662,"num_object_clones":0,"num_object_copies":3324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":392,"num_objects_unfound":0,"num_objects_dirty":1662,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2090,"num_write_kb":98144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1529,"num_bytes_recovered":100204544,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,11],"acting":[8,5],"avail_no_missing":["8","13"],"object_location_counts":[{"shards":"5,8","objects":1466},{"shards":"8,13","objects":196}],"blocked_by":[],"up_primary":5,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.26","version":"166'5927","reported_seq":11498,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-13T11:25:26.025249+0000","last_change":"2024-09-13T11:25:01.697082+0000","last_active":"2024-09-13T11:25:26.025249+0000","last_peered":"2024-09-13T11:25:26.025249+0000","last_clean":"2024-09-13T11:25:26.025249+0000","last_became_active":"2024-09-13T11:24:54.976919+0000","last_became_peered":"2024-09-13T11:24:54.976919+0000","last_unstale":"2024-09-13T11:25:26.025249+0000","last_undegraded":"2024-09-13T11:25:26.025249+0000","last_fullsized":"2024-09-13T11:25:26.025249+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":6,"last_scrub":"147'5433","last_scrub_stamp":"2024-09-13T11:25:01.696886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:01.696886+0000","objects_scrubbed":2158,"log_size":5927,"log_dups_size":0,"ondisk_log_size":5927,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:26:22.070621+0000","scrub_duration":403,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":108527616,"num_objects":1656,"num_object_clones":0,"num_object_copies":3312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1656,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":502,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,12],"acting":[8,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.25","version":"166'5781","reported_seq":11511,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-13T11:25:26.095773+0000","last_change":"2024-09-13T11:24:57.073162+0000","last_active":"2024-09-13T11:25:26.095773+0000","last_peered":"2024-09-13T11:25:26.095773+0000","last_clean":"2024-09-13T11:25:26.095773+0000","last_became_active":"2024-09-13T11:24:55.999019+0000","last_became_peered":"2024-09-13T11:24:55.999019+0000","last_unstale":"2024-09-13T11:25:26.095773+0000","last_undegraded":"2024-09-13T11:25:26.095773+0000","last_fullsized":"2024-09-13T11:25:26.095773+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":6,"last_scrub":"142'5214","last_scrub_stamp":"2024-09-13T11:24:57.073061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:24:57.073061+0000","objects_scrubbed":2249,"log_size":5781,"log_dups_size":0,"ondisk_log_size":5781,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:26:12.348519+0000","scrub_duration":571,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110428160,"num_objects":1685,"num_object_clones":0,"num_object_copies":3370,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1685,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":570,"num_write_kb":192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,2],"acting":[12,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[]},{"pgid":"3.24","version":"162'5742","reported_seq":9650,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-13T11:25:21.944050+0000","last_change":"2024-09-13T11:24:48.048846+0000","last_active":"2024-09-13T11:25:21.944050+0000","last_peered":"2024-09-13T11:25:21.944050+0000","last_clean":"2024-09-13T11:25:21.944050+0000","last_became_active":"2024-09-13T11:24:46.936994+0000","last_became_peered":"2024-09-13T11:24:46.936994+0000","last_unstale":"2024-09-13T11:25:21.944050+0000","last_undegraded":"2024-09-13T11:25:21.944050+0000","last_fullsized":"2024-09-13T11:25:21.944050+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":6,"last_scrub":"135'5093","last_scrub_stamp":"2024-09-13T11:24:48.048764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:24:48.048764+0000","objects_scrubbed":2102,"log_size":5742,"log_dups_size":0,"ondisk_log_size":5742,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:53.485370+0000","scrub_duration":287,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":114163712,"num_objects":1742,"num_object_clones":0,"num_object_copies":3484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1742,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":650,"num_write_kb":9280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1],"acting":[9,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.23","version":"166'5040","reported_seq":7891,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-13T11:25:25.955199+0000","last_change":"2024-09-13T11:23:48.133184+0000","last_active":"2024-09-13T11:25:25.955199+0000","last_peered":"2024-09-13T11:25:25.955199+0000","last_clean":"2024-09-13T11:25:25.955199+0000","last_became_active":"2024-09-13T11:23:47.926677+0000","last_became_peered":"2024-09-13T11:23:47.926677+0000","last_unstale":"2024-09-13T11:25:25.955199+0000","last_undegraded":"2024-09-13T11:25:25.955199+0000","last_fullsized":"2024-09-13T11:25:25.955199+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":6,"last_scrub":"91'3278","last_scrub_stamp":"2024-09-13T11:23:48.133084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:48.133084+0000","objects_scrubbed":1029,"log_size":5040,"log_dups_size":0,"ondisk_log_size":5040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":139,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":111804416,"num_objects":1706,"num_object_clones":0,"num_object_copies":3412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1706,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1763,"num_write_kb":78080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9],"acting":[5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.22","version":"166'4987","reported_seq":10567,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-13T11:25:26.093268+0000","last_change":"2024-09-13T11:24:45.485711+0000","last_active":"2024-09-13T11:25:26.093268+0000","last_peered":"2024-09-13T11:25:26.093268+0000","last_clean":"2024-09-13T11:25:26.093268+0000","last_became_active":"2024-09-13T11:24:29.253848+0000","last_became_peered":"2024-09-13T11:24:29.253848+0000","last_unstale":"2024-09-13T11:25:26.093268+0000","last_undegraded":"2024-09-13T11:25:26.093268+0000","last_fullsized":"2024-09-13T11:25:26.093268+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":6,"last_scrub":"90'3246","last_scrub_stamp":"2024-09-13T11:23:47.300339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:47.300339+0000","objects_scrubbed":1033,"log_size":4987,"log_dups_size":0,"ondisk_log_size":4987,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":203,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":108527616,"num_objects":1656,"num_object_clones":0,"num_object_copies":3312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1656,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1743,"num_write_kb":75712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1768,"num_bytes_recovered":115867648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.21","version":"166'4903","reported_seq":7769,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-13T11:25:26.085283+0000","last_change":"2024-09-13T11:24:12.564117+0000","last_active":"2024-09-13T11:25:26.085283+0000","last_peered":"2024-09-13T11:25:26.085283+0000","last_clean":"2024-09-13T11:25:26.085283+0000","last_became_active":"2024-09-13T11:24:01.250308+0000","last_became_peered":"2024-09-13T11:24:01.250308+0000","last_unstale":"2024-09-13T11:25:26.085283+0000","last_undegraded":"2024-09-13T11:25:26.085283+0000","last_fullsized":"2024-09-13T11:25:26.085283+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":6,"last_scrub":"112'3670","last_scrub_stamp":"2024-09-13T11:24:12.564031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:24:12.564031+0000","objects_scrubbed":1431,"log_size":4903,"log_dups_size":0,"ondisk_log_size":4903,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:36.766299+0000","scrub_duration":183,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":103350272,"num_objects":1577,"num_object_clones":0,"num_object_copies":3154,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1577,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1236,"num_write_kb":44224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8],"acting":[0,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.20","version":"166'5035","reported_seq":7188,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-13T11:25:25.998756+0000","last_change":"2024-09-13T11:23:47.969503+0000","last_active":"2024-09-13T11:25:25.998756+0000","last_peered":"2024-09-13T11:25:25.998756+0000","last_clean":"2024-09-13T11:25:25.998756+0000","last_became_active":"2024-09-13T11:23:33.954946+0000","last_became_peered":"2024-09-13T11:23:33.954946+0000","last_unstale":"2024-09-13T11:25:25.998756+0000","last_undegraded":"2024-09-13T11:25:25.998756+0000","last_fullsized":"2024-09-13T11:25:25.998756+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":6,"last_scrub":"91'3270","last_scrub_stamp":"2024-09-13T11:23:47.969372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:47.969372+0000","objects_scrubbed":1061,"log_size":5035,"log_dups_size":0,"ondisk_log_size":5035,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1327,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":106627072,"num_objects":1627,"num_object_clones":0,"num_object_copies":3254,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1627,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1770,"num_write_kb":74752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,0],"acting":[12,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[]},{"pgid":"3.1f","version":"166'7087","reported_seq":19037,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-13T11:25:26.064403+0000","last_change":"2024-09-13T11:25:20.773752+0000","last_active":"2024-09-13T11:25:26.064403+0000","last_peered":"2024-09-13T11:25:26.064403+0000","last_clean":"2024-09-13T11:25:26.064403+0000","last_became_active":"2024-09-13T11:24:29.290326+0000","last_became_peered":"2024-09-13T11:24:29.290326+0000","last_unstale":"2024-09-13T11:25:26.064403+0000","last_undegraded":"2024-09-13T11:25:26.064403+0000","last_fullsized":"2024-09-13T11:25:26.064403+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":5,"last_scrub":"161'6865","last_scrub_stamp":"2024-09-13T11:25:20.773641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:20.773641+0000","objects_scrubbed":3425,"log_size":7087,"log_dups_size":0,"ondisk_log_size":7087,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:26:21.317280+0000","scrub_duration":535,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":209584128,"num_objects":3198,"num_object_clones":0,"num_object_copies":6396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3198,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2667,"num_write_kb":102848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3306,"num_bytes_recovered":216662016,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5],"acting":[10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1e","version":"166'7310","reported_seq":17827,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-13T11:25:25.982804+0000","last_change":"2024-09-13T11:24:08.338485+0000","last_active":"2024-09-13T11:25:25.982804+0000","last_peered":"2024-09-13T11:25:25.982804+0000","last_clean":"2024-09-13T11:25:25.982804+0000","last_became_active":"2024-09-13T11:24:01.261448+0000","last_became_peered":"2024-09-13T11:24:01.261448+0000","last_unstale":"2024-09-13T11:25:25.982804+0000","last_undegraded":"2024-09-13T11:25:25.982804+0000","last_fullsized":"2024-09-13T11:25:25.982804+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":5,"last_scrub":"110'4537","last_scrub_stamp":"2024-09-13T11:24:08.338354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:24:08.338354+0000","objects_scrubbed":2760,"log_size":7310,"log_dups_size":0,"ondisk_log_size":7310,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":367,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":216137728,"num_objects":3298,"num_object_clones":0,"num_object_copies":6596,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3298,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2782,"num_write_kb":106240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"166'7123","reported_seq":12569,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-13T11:25:26.004350+0000","last_change":"2024-09-13T11:24:12.492592+0000","last_active":"2024-09-13T11:25:26.004350+0000","last_peered":"2024-09-13T11:25:26.004350+0000","last_clean":"2024-09-13T11:25:26.004350+0000","last_became_active":"2024-09-13T11:24:01.258225+0000","last_became_peered":"2024-09-13T11:24:01.258225+0000","last_unstale":"2024-09-13T11:25:26.004350+0000","last_undegraded":"2024-09-13T11:25:26.004350+0000","last_fullsized":"2024-09-13T11:25:26.004350+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":5,"last_scrub":"112'4579","last_scrub_stamp":"2024-09-13T11:24:12.492507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:24:12.492507+0000","objects_scrubbed":2875,"log_size":7123,"log_dups_size":0,"ondisk_log_size":7123,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:34.926575+0000","scrub_duration":375,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":214630424,"num_objects":3276,"num_object_clones":0,"num_object_copies":6552,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3276,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2557,"num_write_kb":94593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,15],"acting":[2,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1c","version":"162'7099","reported_seq":10894,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-13T11:25:22.041539+0000","last_change":"2024-09-13T11:24:00.529312+0000","last_active":"2024-09-13T11:25:22.041539+0000","last_peered":"2024-09-13T11:25:22.041539+0000","last_clean":"2024-09-13T11:25:22.041539+0000","last_became_active":"2024-09-13T11:23:33.989492+0000","last_became_peered":"2024-09-13T11:23:33.989492+0000","last_unstale":"2024-09-13T11:25:22.041539+0000","last_undegraded":"2024-09-13T11:25:22.041539+0000","last_fullsized":"2024-09-13T11:25:22.041539+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":5,"last_scrub":"102'4277","last_scrub_stamp":"2024-09-13T11:24:00.529243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:24:00.529243+0000","objects_scrubbed":2544,"log_size":7099,"log_dups_size":0,"ondisk_log_size":7099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":259,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":226754560,"num_objects":3460,"num_object_clones":0,"num_object_copies":6920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3460,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2824,"num_write_kb":119680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7],"acting":[11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1b","version":"166'7246","reported_seq":12871,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-13T11:25:26.099182+0000","last_change":"2024-09-13T11:24:12.392093+0000","last_active":"2024-09-13T11:25:26.099182+0000","last_peered":"2024-09-13T11:25:26.099182+0000","last_clean":"2024-09-13T11:25:26.099182+0000","last_became_active":"2024-09-13T11:24:04.950821+0000","last_became_peered":"2024-09-13T11:24:04.950821+0000","last_unstale":"2024-09-13T11:25:26.099182+0000","last_undegraded":"2024-09-13T11:25:26.099182+0000","last_fullsized":"2024-09-13T11:25:26.099182+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":5,"last_scrub":"112'4665","last_scrub_stamp":"2024-09-13T11:24:12.391890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:24:12.391890+0000","objects_scrubbed":2934,"log_size":7246,"log_dups_size":0,"ondisk_log_size":7246,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:41.846147+0000","scrub_duration":675,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":215613440,"num_objects":3290,"num_object_clones":0,"num_object_copies":6580,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3290,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2602,"num_write_kb":94656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,15],"acting":[12,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[]},{"pgid":"3.1a","version":"166'7125","reported_seq":12411,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-13T11:25:25.999241+0000","last_change":"2024-09-13T11:24:09.556174+0000","last_active":"2024-09-13T11:25:25.999241+0000","last_peered":"2024-09-13T11:25:25.999241+0000","last_clean":"2024-09-13T11:25:25.999241+0000","last_became_active":"2024-09-13T11:24:03.935185+0000","last_became_peered":"2024-09-13T11:24:03.935185+0000","last_unstale":"2024-09-13T11:25:25.999241+0000","last_undegraded":"2024-09-13T11:25:25.999241+0000","last_fullsized":"2024-09-13T11:25:25.999241+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":5,"last_scrub":"110'4433","last_scrub_stamp":"2024-09-13T11:24:09.556022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:24:09.556022+0000","objects_scrubbed":2654,"log_size":7125,"log_dups_size":0,"ondisk_log_size":7125,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:28.469324+0000","scrub_duration":387,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":210698240,"num_objects":3215,"num_object_clones":0,"num_object_copies":6430,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3215,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2703,"num_write_kb":104448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9],"acting":[4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.19","version":"162'6603","reported_seq":11067,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-13T11:25:22.000325+0000","last_change":"2024-09-13T11:25:10.784789+0000","last_active":"2024-09-13T11:25:22.000325+0000","last_peered":"2024-09-13T11:25:22.000325+0000","last_clean":"2024-09-13T11:25:22.000325+0000","last_became_active":"2024-09-13T11:23:41.924941+0000","last_became_peered":"2024-09-13T11:23:41.924941+0000","last_unstale":"2024-09-13T11:25:22.000325+0000","last_undegraded":"2024-09-13T11:25:22.000325+0000","last_fullsized":"2024-09-13T11:25:22.000325+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":5,"last_scrub":"151'6156","last_scrub_stamp":"2024-09-13T11:25:10.784716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:10.784716+0000","objects_scrubbed":3781,"log_size":6603,"log_dups_size":0,"ondisk_log_size":6603,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:26:30.390310+0000","scrub_duration":1131,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":217317376,"num_objects":3316,"num_object_clones":0,"num_object_copies":6632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3316,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3201,"num_write_kb":145216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1],"acting":[9,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.18","version":"166'6844","reported_seq":11579,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-13T11:25:26.060441+0000","last_change":"2024-09-13T11:24:13.454659+0000","last_active":"2024-09-13T11:25:26.060441+0000","last_peered":"2024-09-13T11:25:26.060441+0000","last_clean":"2024-09-13T11:25:26.060441+0000","last_became_active":"2024-09-13T11:23:48.926014+0000","last_became_peered":"2024-09-13T11:23:48.926014+0000","last_unstale":"2024-09-13T11:25:26.060441+0000","last_undegraded":"2024-09-13T11:25:26.060441+0000","last_fullsized":"2024-09-13T11:25:26.060441+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":5,"last_scrub":"112'4376","last_scrub_stamp":"2024-09-13T11:24:13.454575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:24:13.454575+0000","objects_scrubbed":3036,"log_size":6844,"log_dups_size":0,"ondisk_log_size":6844,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:34.811860+0000","scrub_duration":375,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":213843968,"num_objects":3263,"num_object_clones":0,"num_object_copies":6526,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3263,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2477,"num_write_kb":86528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0],"acting":[10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.17","version":"166'6739","reported_seq":10945,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-13T11:25:26.063961+0000","last_change":"2024-09-13T11:25:12.658562+0000","last_active":"2024-09-13T11:25:26.063961+0000","last_peered":"2024-09-13T11:25:26.063961+0000","last_clean":"2024-09-13T11:25:26.063961+0000","last_became_active":"2024-09-13T11:23:42.935601+0000","last_became_peered":"2024-09-13T11:23:42.935601+0000","last_unstale":"2024-09-13T11:25:26.063961+0000","last_undegraded":"2024-09-13T11:25:26.063961+0000","last_fullsized":"2024-09-13T11:25:26.063961+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":5,"last_scrub":"153'6197","last_scrub_stamp":"2024-09-13T11:25:12.658464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:12.658464+0000","objects_scrubbed":3743,"log_size":6739,"log_dups_size":0,"ondisk_log_size":6739,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:26:25.625277+0000","scrub_duration":467,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":209715200,"num_objects":3200,"num_object_clones":0,"num_object_copies":6400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3200,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3388,"num_write_kb":146688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3],"acting":[10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"162'6868","reported_seq":11702,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-13T11:25:22.064255+0000","last_change":"2024-09-13T11:25:01.903487+0000","last_active":"2024-09-13T11:25:22.064255+0000","last_peered":"2024-09-13T11:25:22.064255+0000","last_clean":"2024-09-13T11:25:22.064255+0000","last_became_active":"2024-09-13T11:23:46.105237+0000","last_became_peered":"2024-09-13T11:23:46.105237+0000","last_unstale":"2024-09-13T11:25:22.064255+0000","last_undegraded":"2024-09-13T11:25:22.064255+0000","last_fullsized":"2024-09-13T11:25:22.064255+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":5,"last_scrub":"147'6035","last_scrub_stamp":"2024-09-13T11:25:01.903395+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:01.903395+0000","objects_scrubbed":4359,"log_size":6868,"log_dups_size":0,"ondisk_log_size":6868,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:26:16.436377+0000","scrub_duration":743,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":230096896,"num_objects":3511,"num_object_clones":0,"num_object_copies":7022,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3511,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3393,"num_write_kb":155200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,15],"acting":[11,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.15","version":"166'6726","reported_seq":18051,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-13T11:25:26.070608+0000","last_change":"2024-09-13T11:25:12.198448+0000","last_active":"2024-09-13T11:25:26.070608+0000","last_peered":"2024-09-13T11:25:26.070608+0000","last_clean":"2024-09-13T11:25:26.070608+0000","last_became_active":"2024-09-13T11:24:29.226615+0000","last_became_peered":"2024-09-13T11:24:29.226615+0000","last_unstale":"2024-09-13T11:25:26.070608+0000","last_undegraded":"2024-09-13T11:25:26.070608+0000","last_fullsized":"2024-09-13T11:25:26.070608+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":5,"last_scrub":"91'3384","last_scrub_stamp":"2024-09-13T11:23:48.332600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:48.332600+0000","objects_scrubbed":2032,"log_size":6726,"log_dups_size":0,"ondisk_log_size":6726,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":235,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":209649664,"num_objects":3199,"num_object_clones":0,"num_object_copies":6398,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3199,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3343,"num_write_kb":144320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3385,"num_bytes_recovered":221839360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.14","version":"165'6857","reported_seq":18106,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-13T11:25:25.064840+0000","last_change":"2024-09-13T11:25:15.378521+0000","last_active":"2024-09-13T11:25:25.064840+0000","last_peered":"2024-09-13T11:25:25.064840+0000","last_clean":"2024-09-13T11:25:25.064840+0000","last_became_active":"2024-09-13T11:24:29.253106+0000","last_became_peered":"2024-09-13T11:24:29.253106+0000","last_unstale":"2024-09-13T11:25:25.064840+0000","last_undegraded":"2024-09-13T11:25:25.064840+0000","last_fullsized":"2024-09-13T11:25:25.064840+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":5,"last_scrub":"91'3371","last_scrub_stamp":"2024-09-13T11:23:48.696960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:48.696960+0000","objects_scrubbed":1935,"log_size":6857,"log_dups_size":0,"ondisk_log_size":6857,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":735,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":222494720,"num_objects":3395,"num_object_clones":0,"num_object_copies":6790,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3395,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3496,"num_write_kb":158592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3437,"num_bytes_recovered":225247232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,15],"acting":[3,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":304,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-13T11:25:25.226545+0000","last_change":"2024-09-13T11:24:23.212494+0000","last_active":"2024-09-13T11:25:25.226545+0000","last_peered":"2024-09-13T11:25:25.226545+0000","last_clean":"2024-09-13T11:25:25.226545+0000","last_became_active":"2024-09-13T11:22:14.192897+0000","last_became_peered":"2024-09-13T11:22:14.192897+0000","last_unstale":"2024-09-13T11:25:25.226545+0000","last_undegraded":"2024-09-13T11:25:25.226545+0000","last_fullsized":"2024-09-13T11:25:25.226545+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:24:23.212406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:24:23.212406+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:29.886382+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00084655899999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,10],"acting":[15,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"3.6","version":"162'5853","reported_seq":9708,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-13T11:25:21.885529+0000","last_change":"2024-09-13T11:24:49.416851+0000","last_active":"2024-09-13T11:25:21.885529+0000","last_peered":"2024-09-13T11:25:21.885529+0000","last_clean":"2024-09-13T11:25:21.885529+0000","last_became_active":"2024-09-13T11:24:29.322558+0000","last_became_peered":"2024-09-13T11:24:29.322558+0000","last_unstale":"2024-09-13T11:25:21.885529+0000","last_undegraded":"2024-09-13T11:25:21.885529+0000","last_fullsized":"2024-09-13T11:25:21.885529+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"136'5216","last_scrub_stamp":"2024-09-13T11:24:49.416706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:24:49.416706+0000","objects_scrubbed":2083,"log_size":5853,"log_dups_size":0,"ondisk_log_size":5853,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:26:09.525086+0000","scrub_duration":247,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":113967104,"num_objects":1739,"num_object_clones":0,"num_object_copies":3478,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1739,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":638,"num_write_kb":9408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,6],"acting":[13,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":303,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-13T11:25:21.181055+0000","last_change":"2024-09-13T11:24:59.556253+0000","last_active":"2024-09-13T11:25:21.181055+0000","last_peered":"2024-09-13T11:25:21.181055+0000","last_clean":"2024-09-13T11:25:21.181055+0000","last_became_active":"2024-09-13T11:22:14.211923+0000","last_became_peered":"2024-09-13T11:22:14.211923+0000","last_unstale":"2024-09-13T11:25:21.181055+0000","last_undegraded":"2024-09-13T11:25:21.181055+0000","last_fullsized":"2024-09-13T11:25:21.181055+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:24:59.556180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:24:59.556180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:26:06.229322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011689529999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0],"acting":[9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"162'5680","reported_seq":6029,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-13T11:25:22.049662+0000","last_change":"2024-09-13T11:24:33.336210+0000","last_active":"2024-09-13T11:25:22.049662+0000","last_peered":"2024-09-13T11:25:22.049662+0000","last_clean":"2024-09-13T11:25:22.049662+0000","last_became_active":"2024-09-13T11:24:24.092975+0000","last_became_peered":"2024-09-13T11:24:24.092975+0000","last_unstale":"2024-09-13T11:25:22.049662+0000","last_undegraded":"2024-09-13T11:25:22.049662+0000","last_fullsized":"2024-09-13T11:25:22.049662+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"125'4820","last_scrub_stamp":"2024-09-13T11:24:33.336091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:24:33.336091+0000","objects_scrubbed":1766,"log_size":5680,"log_dups_size":0,"ondisk_log_size":5680,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:54.538352+0000","scrub_duration":263,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":113246208,"num_objects":1728,"num_object_clones":0,"num_object_copies":3456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1728,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":862,"num_write_kb":26368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,8],"acting":[13,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":304,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-13T11:25:25.221085+0000","last_change":"2024-09-13T11:23:59.530794+0000","last_active":"2024-09-13T11:25:25.221085+0000","last_peered":"2024-09-13T11:25:25.221085+0000","last_clean":"2024-09-13T11:25:25.221085+0000","last_became_active":"2024-09-13T11:22:14.581458+0000","last_became_peered":"2024-09-13T11:22:14.581458+0000","last_unstale":"2024-09-13T11:25:25.221085+0000","last_undegraded":"2024-09-13T11:25:25.221085+0000","last_fullsized":"2024-09-13T11:25:25.221085+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:23:59.530670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:23:59.530670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001026047,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5],"acting":[8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"166'5817","reported_seq":6146,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-13T11:25:25.378158+0000","last_change":"2024-09-13T11:24:24.579036+0000","last_active":"2024-09-13T11:25:25.378158+0000","last_peered":"2024-09-13T11:25:25.378158+0000","last_clean":"2024-09-13T11:25:25.378158+0000","last_became_active":"2024-09-13T11:24:24.083077+0000","last_became_peered":"2024-09-13T11:24:24.083077+0000","last_unstale":"2024-09-13T11:25:25.378158+0000","last_undegraded":"2024-09-13T11:25:25.378158+0000","last_fullsized":"2024-09-13T11:25:25.378158+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'4737","last_scrub_stamp":"2024-09-13T11:24:24.578900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:24:24.578900+0000","objects_scrubbed":1690,"log_size":5817,"log_dups_size":0,"ondisk_log_size":5817,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:38.103421+0000","scrub_duration":407,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":106758144,"num_objects":1629,"num_object_clones":0,"num_object_copies":3258,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1629,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1083,"num_write_kb":32704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,12],"acting":[1,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":292,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-13T11:25:25.217725+0000","last_change":"2024-09-13T11:24:08.271026+0000","last_active":"2024-09-13T11:25:25.217725+0000","last_peered":"2024-09-13T11:25:25.217725+0000","last_clean":"2024-09-13T11:25:25.217725+0000","last_became_active":"2024-09-13T11:22:47.719636+0000","last_became_peered":"2024-09-13T11:22:47.719636+0000","last_unstale":"2024-09-13T11:25:25.217725+0000","last_undegraded":"2024-09-13T11:25:25.217725+0000","last_fullsized":"2024-09-13T11:25:25.217725+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:24:08.270972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:24:08.270972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:36.805605+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00087476000000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,9],"acting":[1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"166'5772","reported_seq":6120,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-13T11:25:26.041019+0000","last_change":"2024-09-13T11:24:24.616937+0000","last_active":"2024-09-13T11:25:26.041019+0000","last_peered":"2024-09-13T11:25:26.041019+0000","last_clean":"2024-09-13T11:25:26.041019+0000","last_became_active":"2024-09-13T11:24:24.093492+0000","last_became_peered":"2024-09-13T11:24:24.093492+0000","last_unstale":"2024-09-13T11:25:26.041019+0000","last_undegraded":"2024-09-13T11:25:26.041019+0000","last_fullsized":"2024-09-13T11:25:26.041019+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'4744","last_scrub_stamp":"2024-09-13T11:24:24.616633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:24:24.616633+0000","objects_scrubbed":1624,"log_size":5772,"log_dups_size":0,"ondisk_log_size":5772,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:42.911074+0000","scrub_duration":415,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":104923136,"num_objects":1601,"num_object_clones":0,"num_object_copies":3202,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1601,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1035,"num_write_kb":32384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,13],"acting":[15,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"3.3","version":"166'4964","reported_seq":5331,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-13T11:25:26.038212+0000","last_change":"2024-09-13T11:25:08.394545+0000","last_active":"2024-09-13T11:25:26.038212+0000","last_peered":"2024-09-13T11:25:26.038212+0000","last_clean":"2024-09-13T11:25:26.038212+0000","last_became_active":"2024-09-13T11:23:31.930448+0000","last_became_peered":"2024-09-13T11:23:31.930448+0000","last_unstale":"2024-09-13T11:25:26.038212+0000","last_undegraded":"2024-09-13T11:25:26.038212+0000","last_fullsized":"2024-09-13T11:25:26.038212+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"151'4587","last_scrub_stamp":"2024-09-13T11:25:08.394441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:08.394441+0000","objects_scrubbed":1942,"log_size":4964,"log_dups_size":0,"ondisk_log_size":4964,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:26:27.641900+0000","scrub_duration":219,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":102367232,"num_objects":1562,"num_object_clones":0,"num_object_copies":3124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1562,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1842,"num_write_kb":80896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,7],"acting":[10,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"28'2","reported_seq":242,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-13T11:25:25.216396+0000","last_change":"2024-09-13T11:25:13.869564+0000","last_active":"2024-09-13T11:25:25.216396+0000","last_peered":"2024-09-13T11:25:25.216396+0000","last_clean":"2024-09-13T11:25:25.216396+0000","last_became_active":"2024-09-13T11:23:12.546348+0000","last_became_peered":"2024-09-13T11:23:12.546348+0000","last_unstale":"2024-09-13T11:25:25.216396+0000","last_undegraded":"2024-09-13T11:25:25.216396+0000","last_fullsized":"2024-09-13T11:25:25.216396+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2","last_scrub_stamp":"2024-09-13T11:25:13.869413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:25:13.869413+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:26:38.761718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0025826730000000002,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8],"acting":[2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"166'4962","reported_seq":5328,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-13T11:25:26.067735+0000","last_change":"2024-09-13T11:25:09.842875+0000","last_active":"2024-09-13T11:25:26.067735+0000","last_peered":"2024-09-13T11:25:26.067735+0000","last_clean":"2024-09-13T11:25:26.067735+0000","last_became_active":"2024-09-13T11:23:31.921388+0000","last_became_peered":"2024-09-13T11:23:31.921388+0000","last_unstale":"2024-09-13T11:25:26.067735+0000","last_undegraded":"2024-09-13T11:25:26.067735+0000","last_fullsized":"2024-09-13T11:25:26.067735+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"151'4644","last_scrub_stamp":"2024-09-13T11:25:09.842728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:09.842728+0000","objects_scrubbed":1944,"log_size":4962,"log_dups_size":0,"ondisk_log_size":4962,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:26:25.553642+0000","scrub_duration":395,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":106561536,"num_objects":1626,"num_object_clones":0,"num_object_copies":3252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1626,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1856,"num_write_kb":84032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,15],"acting":[12,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":145,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-13T11:25:25.217562+0000","last_change":"2024-09-13T11:24:29.096863+0000","last_active":"2024-09-13T11:25:25.217562+0000","last_peered":"2024-09-13T11:25:25.217562+0000","last_clean":"2024-09-13T11:25:25.217562+0000","last_became_active":"2024-09-13T11:24:29.096575+0000","last_became_peered":"2024-09-13T11:24:29.096575+0000","last_unstale":"2024-09-13T11:25:25.217562+0000","last_undegraded":"2024-09-13T11:25:25.217562+0000","last_fullsized":"2024-09-13T11:25:25.217562+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:24:22.064711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:24:22.064711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:34.178168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00081225399999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"162'4866","reported_seq":5227,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-13T11:25:22.016980+0000","last_change":"2024-09-13T11:25:00.861990+0000","last_active":"2024-09-13T11:25:22.016980+0000","last_peered":"2024-09-13T11:25:22.016980+0000","last_clean":"2024-09-13T11:25:22.016980+0000","last_became_active":"2024-09-13T11:23:31.926528+0000","last_became_peered":"2024-09-13T11:23:31.926528+0000","last_unstale":"2024-09-13T11:25:22.016980+0000","last_undegraded":"2024-09-13T11:25:22.016980+0000","last_fullsized":"2024-09-13T11:25:22.016980+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"146'4447","last_scrub_stamp":"2024-09-13T11:25:00.861834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:00.861834+0000","objects_scrubbed":2126,"log_size":4866,"log_dups_size":0,"ondisk_log_size":4866,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:26:15.411903+0000","scrub_duration":787,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":111673344,"num_objects":1704,"num_object_clones":0,"num_object_copies":3408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1704,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1796,"num_write_kb":85056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,11],"acting":[13,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":302,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-13T11:25:24.217757+0000","last_change":"2024-09-13T11:24:11.895017+0000","last_active":"2024-09-13T11:25:24.217757+0000","last_peered":"2024-09-13T11:25:24.217757+0000","last_clean":"2024-09-13T11:25:24.217757+0000","last_became_active":"2024-09-13T11:22:14.211435+0000","last_became_peered":"2024-09-13T11:22:14.211435+0000","last_unstale":"2024-09-13T11:25:24.217757+0000","last_undegraded":"2024-09-13T11:25:24.217757+0000","last_fullsized":"2024-09-13T11:25:24.217757+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:24:11.894937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:24:11.894937+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0010010360000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2","version":"166'5033","reported_seq":8863,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-13T11:25:25.866101+0000","last_change":"2024-09-13T11:25:23.765128+0000","last_active":"2024-09-13T11:25:25.866101+0000","last_peered":"2024-09-13T11:25:25.866101+0000","last_clean":"2024-09-13T11:25:25.866101+0000","last_became_active":"2024-09-13T11:24:29.175407+0000","last_became_peered":"2024-09-13T11:24:29.175407+0000","last_unstale":"2024-09-13T11:25:25.866101+0000","last_undegraded":"2024-09-13T11:25:25.866101+0000","last_fullsized":"2024-09-13T11:25:25.866101+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"164'4994","last_scrub_stamp":"2024-09-13T11:25:23.765047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:23.765047+0000","objects_scrubbed":1689,"log_size":5033,"log_dups_size":0,"ondisk_log_size":5033,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:26:45.252942+0000","scrub_duration":283,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":107937792,"num_objects":1647,"num_object_clones":0,"num_object_copies":3294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1647,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1976,"num_write_kb":89024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1731,"num_bytes_recovered":113442816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,3],"acting":[12,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[]},{"pgid":"2.3","version":"26'1","reported_seq":115,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-13T11:25:25.226319+0000","last_change":"2024-09-13T11:25:23.034357+0000","last_active":"2024-09-13T11:25:25.226319+0000","last_peered":"2024-09-13T11:25:25.226319+0000","last_clean":"2024-09-13T11:25:25.226319+0000","last_became_active":"2024-09-13T11:24:29.110028+0000","last_became_peered":"2024-09-13T11:24:29.110028+0000","last_unstale":"2024-09-13T11:25:25.226319+0000","last_undegraded":"2024-09-13T11:25:25.226319+0000","last_fullsized":"2024-09-13T11:25:25.226319+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"26'1","last_scrub_stamp":"2024-09-13T11:25:23.034247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:25:23.034247+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:26:42.429349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,8],"acting":[15,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"1.0","version":"26'112","reported_seq":446,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-13T11:25:25.232244+0000","last_change":"2024-09-13T11:25:23.642366+0000","last_active":"2024-09-13T11:25:25.232244+0000","last_peered":"2024-09-13T11:25:25.232244+0000","last_clean":"2024-09-13T11:25:25.232244+0000","last_became_active":"2024-09-13T11:22:12.129721+0000","last_became_peered":"2024-09-13T11:22:12.129721+0000","last_unstale":"2024-09-13T11:25:25.232244+0000","last_undegraded":"2024-09-13T11:25:25.232244+0000","last_fullsized":"2024-09-13T11:25:25.232244+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"26'112","last_scrub_stamp":"2024-09-13T11:25:23.642204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:11.098388+0000","last_clean_scrub_stamp":"2024-09-13T11:25:23.642204+0000","objects_scrubbed":2,"log_size":112,"log_dups_size":0,"ondisk_log_size":112,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:26:42.628319+0000","scrub_duration":67,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.8","version":"166'5776","reported_seq":6125,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-13T11:25:25.960544+0000","last_change":"2024-09-13T11:24:24.506212+0000","last_active":"2024-09-13T11:25:25.960544+0000","last_peered":"2024-09-13T11:25:25.960544+0000","last_clean":"2024-09-13T11:25:25.960544+0000","last_became_active":"2024-09-13T11:24:24.096220+0000","last_became_peered":"2024-09-13T11:24:24.096220+0000","last_unstale":"2024-09-13T11:25:25.960544+0000","last_undegraded":"2024-09-13T11:25:25.960544+0000","last_fullsized":"2024-09-13T11:25:25.960544+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'4749","last_scrub_stamp":"2024-09-13T11:24:24.506109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:24:24.506109+0000","objects_scrubbed":1690,"log_size":5776,"log_dups_size":0,"ondisk_log_size":5776,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:49.430042+0000","scrub_duration":215,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":107347968,"num_objects":1638,"num_object_clones":0,"num_object_copies":3276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1638,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1028,"num_write_kb":31232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.9","version":"166'5817","reported_seq":6165,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-13T11:25:25.911883+0000","last_change":"2024-09-13T11:24:24.928964+0000","last_active":"2024-09-13T11:25:25.911883+0000","last_peered":"2024-09-13T11:25:25.911883+0000","last_clean":"2024-09-13T11:25:25.911883+0000","last_became_active":"2024-09-13T11:24:24.082504+0000","last_became_peered":"2024-09-13T11:24:24.082504+0000","last_unstale":"2024-09-13T11:25:25.911883+0000","last_undegraded":"2024-09-13T11:25:25.911883+0000","last_fullsized":"2024-09-13T11:25:25.911883+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'4799","last_scrub_stamp":"2024-09-13T11:24:24.928724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:24:24.928724+0000","objects_scrubbed":1705,"log_size":5817,"log_dups_size":0,"ondisk_log_size":5817,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:43.845801+0000","scrub_duration":255,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":108527616,"num_objects":1656,"num_object_clones":0,"num_object_copies":3312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1656,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1021,"num_write_kb":31104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,2],"acting":[12,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[]},{"pgid":"3.a","version":"165'6176","reported_seq":6526,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-13T11:25:25.021479+0000","last_change":"2024-09-13T11:24:25.137871+0000","last_active":"2024-09-13T11:25:25.021479+0000","last_peered":"2024-09-13T11:25:25.021479+0000","last_clean":"2024-09-13T11:25:25.021479+0000","last_became_active":"2024-09-13T11:24:24.089253+0000","last_became_peered":"2024-09-13T11:24:24.089253+0000","last_unstale":"2024-09-13T11:25:25.021479+0000","last_undegraded":"2024-09-13T11:25:25.021479+0000","last_fullsized":"2024-09-13T11:25:25.021479+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'5157","last_scrub_stamp":"2024-09-13T11:24:25.137777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:24:25.137777+0000","objects_scrubbed":1777,"log_size":6176,"log_dups_size":0,"ondisk_log_size":6176,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:36.149013+0000","scrub_duration":223,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":106758144,"num_objects":1629,"num_object_clones":0,"num_object_copies":3258,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1629,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1022,"num_write_kb":27968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.b","version":"162'6059","reported_seq":6402,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-13T11:25:21.990278+0000","last_change":"2024-09-13T11:24:25.503864+0000","last_active":"2024-09-13T11:25:21.990278+0000","last_peered":"2024-09-13T11:25:21.990278+0000","last_clean":"2024-09-13T11:25:21.990278+0000","last_became_active":"2024-09-13T11:24:24.085684+0000","last_became_peered":"2024-09-13T11:24:24.085684+0000","last_unstale":"2024-09-13T11:25:21.990278+0000","last_undegraded":"2024-09-13T11:25:21.990278+0000","last_fullsized":"2024-09-13T11:25:21.990278+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'5112","last_scrub_stamp":"2024-09-13T11:24:25.503716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:24:25.503716+0000","objects_scrubbed":1718,"log_size":6059,"log_dups_size":0,"ondisk_log_size":6059,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:37.636639+0000","scrub_duration":395,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":113311744,"num_objects":1729,"num_object_clones":0,"num_object_copies":3458,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1729,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":947,"num_write_kb":30656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,1],"acting":[13,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"3.c","version":"162'6080","reported_seq":6423,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-13T11:25:21.979106+0000","last_change":"2024-09-13T11:24:24.595021+0000","last_active":"2024-09-13T11:25:21.979106+0000","last_peered":"2024-09-13T11:25:21.979106+0000","last_clean":"2024-09-13T11:25:21.979106+0000","last_became_active":"2024-09-13T11:24:24.090847+0000","last_became_peered":"2024-09-13T11:24:24.090847+0000","last_unstale":"2024-09-13T11:25:21.979106+0000","last_undegraded":"2024-09-13T11:25:21.979106+0000","last_fullsized":"2024-09-13T11:25:21.979106+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'5154","last_scrub_stamp":"2024-09-13T11:24:24.594848+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:24:24.594848+0000","objects_scrubbed":1720,"log_size":6080,"log_dups_size":0,"ondisk_log_size":6080,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:35.181590+0000","scrub_duration":343,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112001024,"num_objects":1709,"num_object_clones":0,"num_object_copies":3418,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1709,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":931,"num_write_kb":29440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,13],"acting":[11,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.d","version":"166'6161","reported_seq":6512,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-13T11:25:26.002410+0000","last_change":"2024-09-13T11:24:24.967444+0000","last_active":"2024-09-13T11:25:26.002410+0000","last_peered":"2024-09-13T11:25:26.002410+0000","last_clean":"2024-09-13T11:25:26.002410+0000","last_became_active":"2024-09-13T11:24:24.087224+0000","last_became_peered":"2024-09-13T11:24:24.087224+0000","last_unstale":"2024-09-13T11:25:26.002410+0000","last_undegraded":"2024-09-13T11:25:26.002410+0000","last_fullsized":"2024-09-13T11:25:26.002410+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'5110","last_scrub_stamp":"2024-09-13T11:24:24.967337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:24:24.967337+0000","objects_scrubbed":1725,"log_size":6161,"log_dups_size":0,"ondisk_log_size":6161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:53.620949+0000","scrub_duration":255,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":107282432,"num_objects":1637,"num_object_clones":0,"num_object_copies":3274,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1637,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1054,"num_write_kb":30912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.e","version":"166'7290","reported_seq":12538,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-13T11:25:26.050702+0000","last_change":"2024-09-13T11:23:32.368856+0000","last_active":"2024-09-13T11:25:26.050702+0000","last_peered":"2024-09-13T11:25:26.050702+0000","last_clean":"2024-09-13T11:25:26.050702+0000","last_became_active":"2024-09-13T11:23:31.925869+0000","last_became_peered":"2024-09-13T11:23:31.925869+0000","last_unstale":"2024-09-13T11:25:26.050702+0000","last_undegraded":"2024-09-13T11:25:26.050702+0000","last_fullsized":"2024-09-13T11:25:26.050702+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'3527","last_scrub_stamp":"2024-09-13T11:23:32.368727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:32.368727+0000","objects_scrubbed":1774,"log_size":7290,"log_dups_size":0,"ondisk_log_size":7290,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":219,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":209321984,"num_objects":3194,"num_object_clones":0,"num_object_copies":6388,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3194,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3764,"num_write_kb":165888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"162'7116","reported_seq":7478,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-13T11:25:22.048153+0000","last_change":"2024-09-13T11:25:00.289940+0000","last_active":"2024-09-13T11:25:22.048153+0000","last_peered":"2024-09-13T11:25:22.048153+0000","last_clean":"2024-09-13T11:25:22.048153+0000","last_became_active":"2024-09-13T11:23:31.935239+0000","last_became_peered":"2024-09-13T11:23:31.935239+0000","last_unstale":"2024-09-13T11:25:22.048153+0000","last_undegraded":"2024-09-13T11:25:22.048153+0000","last_fullsized":"2024-09-13T11:25:22.048153+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"146'6154","last_scrub_stamp":"2024-09-13T11:25:00.289673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:00.289673+0000","objects_scrubbed":4394,"log_size":7116,"log_dups_size":0,"ondisk_log_size":7116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:26:24.274856+0000","scrub_duration":1187,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":223477760,"num_objects":3410,"num_object_clones":0,"num_object_copies":6820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3410,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3696,"num_write_kb":170560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,12],"acting":[11,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.10","version":"166'6955","reported_seq":11477,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-13T11:25:26.029727+0000","last_change":"2024-09-13T11:25:10.161564+0000","last_active":"2024-09-13T11:25:26.029727+0000","last_peered":"2024-09-13T11:25:26.029727+0000","last_clean":"2024-09-13T11:25:26.029727+0000","last_became_active":"2024-09-13T11:23:41.931183+0000","last_became_peered":"2024-09-13T11:23:41.931183+0000","last_unstale":"2024-09-13T11:25:26.029727+0000","last_undegraded":"2024-09-13T11:25:26.029727+0000","last_fullsized":"2024-09-13T11:25:26.029727+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":5,"last_scrub":"151'6298","last_scrub_stamp":"2024-09-13T11:25:10.161464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:10.161464+0000","objects_scrubbed":3964,"log_size":6955,"log_dups_size":0,"ondisk_log_size":6955,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:26:33.015478+0000","scrub_duration":607,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":215744512,"num_objects":3292,"num_object_clones":0,"num_object_copies":6584,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3292,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3489,"num_write_kb":152000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.11","version":"166'6880","reported_seq":10174,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-13T11:25:26.042414+0000","last_change":"2024-09-13T11:25:24.287326+0000","last_active":"2024-09-13T11:25:26.042414+0000","last_peered":"2024-09-13T11:25:26.042414+0000","last_clean":"2024-09-13T11:25:26.042414+0000","last_became_active":"2024-09-13T11:23:39.063172+0000","last_became_peered":"2024-09-13T11:23:39.063172+0000","last_unstale":"2024-09-13T11:25:26.042414+0000","last_undegraded":"2024-09-13T11:25:26.042414+0000","last_fullsized":"2024-09-13T11:25:26.042414+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":5,"last_scrub":"165'6821","last_scrub_stamp":"2024-09-13T11:25:24.287082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:24.287082+0000","objects_scrubbed":3406,"log_size":6880,"log_dups_size":0,"ondisk_log_size":6880,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:26:49.405332+0000","scrub_duration":947,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":218497024,"num_objects":3334,"num_object_clones":0,"num_object_copies":6668,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3334,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3539,"num_write_kb":155584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11],"acting":[4,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.12","version":"166'6901","reported_seq":17004,"reported_epoch":166,"state":"active+clean+scrubbing","last_fresh":"2024-09-13T11:25:26.193477+0000","last_change":"2024-09-13T11:25:26.136263+0000","last_active":"2024-09-13T11:25:26.193477+0000","last_peered":"2024-09-13T11:25:26.193477+0000","last_clean":"2024-09-13T11:25:26.193477+0000","last_became_active":"2024-09-13T11:24:29.268229+0000","last_became_peered":"2024-09-13T11:24:29.268229+0000","last_unstale":"2024-09-13T11:25:26.193477+0000","last_undegraded":"2024-09-13T11:25:26.193477+0000","last_fullsized":"2024-09-13T11:25:26.193477+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'3109","last_scrub_stamp":"2024-09-13T11:23:35.111320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:35.111320+0000","objects_scrubbed":399,"log_size":6901,"log_dups_size":0,"ondisk_log_size":6901,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"scrubbing for 0s","scrub_duration":239,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":216662016,"num_objects":3306,"num_object_clones":0,"num_object_copies":6612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3306,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3792,"num_write_kb":171712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3457,"num_bytes_recovered":226557952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.13","version":"162'6778","reported_seq":9999,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-13T11:25:22.046470+0000","last_change":"2024-09-13T11:25:01.153044+0000","last_active":"2024-09-13T11:25:22.046470+0000","last_peered":"2024-09-13T11:25:22.046470+0000","last_clean":"2024-09-13T11:25:22.046470+0000","last_became_active":"2024-09-13T11:23:41.082433+0000","last_became_peered":"2024-09-13T11:23:41.082433+0000","last_unstale":"2024-09-13T11:25:22.046470+0000","last_undegraded":"2024-09-13T11:25:22.046470+0000","last_fullsized":"2024-09-13T11:25:22.046470+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":5,"last_scrub":"147'5921","last_scrub_stamp":"2024-09-13T11:25:01.152971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:01.152971+0000","objects_scrubbed":4311,"log_size":6778,"log_dups_size":0,"ondisk_log_size":6778,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:26:07.027631+0000","scrub_duration":1011,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":223674368,"num_objects":3413,"num_object_clones":0,"num_object_copies":6826,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3413,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3382,"num_write_kb":154048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7],"acting":[11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":46,"stat_sum":{"num_bytes":6946127896,"num_objects":105991,"num_object_clones":0,"num_object_copies":211982,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3138,"num_objects_unfound":0,"num_objects_dirty":105991,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":90705,"num_write_kb":3634433,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20645,"num_bytes_recovered":1352990720,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":15578505216,"data_stored":15578497072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":43282,"internal_metadata":0},"log_size":284089,"ondisk_log_size":284089,"up":92,"acting":92,"num_store_stats":16},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":11},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":112,"ondisk_log_size":112,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":13,"up_from":23,"seq":98784247850,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":886968,"kb_used_data":779432,"kb_used_omap":1,"kb_used_meta":107518,"kb_avail":93484872,"statfs":{"total":96636764160,"available":95728508928,"internally_reserved":0,"allocated":798138368,"data_stored":797420163,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":110098885},"hb_peers":[0,1,2,4,6,8,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":23,"seq":98784247849,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":736032,"kb_used_data":656272,"kb_used_omap":1,"kb_used_meta":79742,"kb_avail":93635808,"statfs":{"total":96636764160,"available":95883067392,"internally_reserved":0,"allocated":672022528,"data_stored":671332275,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":81656256},"hb_peers":[0,1,3,8,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":40,"seq":171798691873,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1171488,"kb_used_data":1067280,"kb_used_omap":4,"kb_used_meta":104187,"kb_avail":93200352,"statfs":{"total":96636764160,"available":95437160448,"internally_reserved":0,"allocated":1092894720,"data_stored":1092165338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4433,"internal_metadata":106688175},"hb_peers":[0,2,3,4,5,6,7,8,9,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":21,"seq":90194313258,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1218360,"kb_used_data":1119144,"kb_used_omap":1,"kb_used_meta":99198,"kb_avail":93153480,"statfs":{"total":96636764160,"available":95389163520,"internally_reserved":0,"allocated":1146003456,"data_stored":1145255743,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":101579207},"hb_peers":[0,2,4,5,7,9,10,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":21,"seq":90194313261,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1257128,"kb_used_data":1175960,"kb_used_omap":1,"kb_used_meta":81150,"kb_avail":93114712,"statfs":{"total":96636764160,"available":95349465088,"internally_reserved":0,"allocated":1204183040,"data_stored":1203447594,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":83098055},"hb_peers":[0,2,4,6,7,8,9,11,12,13],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":22,"seq":94489280553,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1307644,"kb_used_data":1200108,"kb_used_omap":1,"kb_used_meta":107518,"kb_avail":93064196,"statfs":{"total":96636764160,"available":95297736704,"internally_reserved":0,"allocated":1228910592,"data_stored":1228223837,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":110098885},"hb_peers":[1,3,4,5,7,8,10,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":22,"seq":94489280554,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":602472,"kb_used_data":526872,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":93769368,"statfs":{"total":96636764160,"available":96019832832,"internally_reserved":0,"allocated":539516928,"data_stored":538792320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,1,3,4,5,7,8,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":15,"up_from":22,"seq":94489280554,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1227888,"kb_used_data":1116192,"kb_used_omap":1,"kb_used_meta":111678,"kb_avail":93143952,"statfs":{"total":96636764160,"available":95379406848,"internally_reserved":0,"allocated":1142980608,"data_stored":1142305477,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":114358726},"hb_peers":[0,4,5,6,7,8,10,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":22,"seq":94489280555,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":806068,"kb_used_data":742948,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":93565772,"statfs":{"total":96636764160,"available":95811350528,"internally_reserved":0,"allocated":760778752,"data_stored":760043385,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[3,5,6,7,9,10,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":22,"seq":94489280554,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":589264,"kb_used_data":535808,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":93782576,"statfs":{"total":96636764160,"available":96033357824,"internally_reserved":0,"allocated":548667392,"data_stored":547963100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54655430},"hb_peers":[0,1,3,5,7,8,10,11,12,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":22,"seq":94489280554,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1467536,"kb_used_data":1364160,"kb_used_omap":1,"kb_used_meta":103358,"kb_avail":92904304,"statfs":{"total":96636764160,"available":95134007296,"internally_reserved":0,"allocated":1396899840,"data_stored":1396192229,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":105839050},"hb_peers":[0,3,5,6,8,9,10,11,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":22,"seq":94489280554,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":566340,"kb_used_data":494900,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":93805500,"statfs":{"total":96636764160,"available":96056832000,"internally_reserved":0,"allocated":506777600,"data_stored":506022479,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73136581},"hb_peers":[1,5,6,7,9,10,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":22,"seq":94489280553,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1018012,"kb_used_data":946572,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":93353828,"statfs":{"total":96636764160,"available":95594319872,"internally_reserved":0,"allocated":969289728,"data_stored":968573918,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":73136576},"hb_peers":[0,1,3,5,7,9,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":11,"up_from":22,"seq":94489280554,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1513220,"kb_used_data":1405684,"kb_used_omap":1,"kb_used_meta":107518,"kb_avail":92858620,"statfs":{"total":96636764160,"available":95087226880,"internally_reserved":0,"allocated":1439420416,"data_stored":1438701096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":110098884},"hb_peers":[0,1,2,4,7,8,10,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":12,"up_from":22,"seq":94489280554,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1280248,"kb_used_data":1168552,"kb_used_omap":1,"kb_used_meta":111678,"kb_avail":93091592,"statfs":{"total":96636764160,"available":95325790208,"internally_reserved":0,"allocated":1196597248,"data_stored":1195854952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":114358725},"hb_peers":[0,1,2,3,4,5,10,11,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":14,"up_from":0,"seq":609885356039,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1226309632,"data_stored":1226309632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1082654720,"data_stored":1082654720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":534515712,"data_stored":534511640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1139474432,"data_stored":1139474432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":752025600,"data_stored":752025600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1200947200,"data_stored":1200947200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":542900224,"data_stored":542900224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1394081792,"data_stored":1394081792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":504627200,"data_stored":504627200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":668794880,"data_stored":668794880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":955252736,"data_stored":955252736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":1436155904,"data_stored":1436155904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":1183645696,"data_stored":1183645696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":787087360,"data_stored":787087360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":1030029312,"data_stored":1030029312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":43282,"internal_metadata":0},{"poolid":3,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":1140002816,"data_stored":1139998744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T11:25:27.478 INFO:tasks.ceph.osd.14.smithi162.stderr:2024-09-13T11:25:27.476+0000 7f3df5903640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33154) UID: 0 2024-09-13T11:25:27.487 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-09-13T11:25:27.487 INFO:teuthology.orchestra.run.smithi177.stdout: "size": 20, 2024-09-13T11:25:27.487 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 600, 2024-09-13T11:25:27.487 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [ 2024-09-13T11:25:27.487 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:27.487 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPush(3.17 75/70 [PushOp(3:eda94455:::benchmark_data_smithi177_31116_object13017:head, version: 50'788, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:eda94455:::benchmark_data_smithi177_31116_object13017:head@50'788, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:25:27.487 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:31.038029+0000", 2024-09-13T11:25:27.487 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 116.44798163900001, 2024-09-13T11:25:27.487 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.42064623499999998, 2024-09-13T11:25:27.487 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:25:27.488 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:25:27.488 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:25:27.488 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:25:27.488 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:27.488 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:25:27.488 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.038029+0000", 2024-09-13T11:25:27.488 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:27.488 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:27.488 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:27.488 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:25:27.488 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.038029+0000", 2024-09-13T11:25:27.488 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:27.488 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:27.489 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:27.489 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:25:27.489 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.038030+0000", 2024-09-13T11:25:27.489 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.499e-06 2024-09-13T11:25:27.489 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:27.489 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:27.489 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:25:27.489 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.038088+0000", 2024-09-13T11:25:27.489 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.7771e-05 2024-09-13T11:25:27.489 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:27.489 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:27.489 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:25:27.489 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.457025+0000", 2024-09-13T11:25:27.490 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.41893754900000002 2024-09-13T11:25:27.490 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:27.490 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:27.490 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:25:27.490 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.457073+0000", 2024-09-13T11:25:27.490 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.7450999999999997e-05 2024-09-13T11:25:27.490 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:27.490 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:27.490 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:25:27.490 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.457304+0000", 2024-09-13T11:25:27.490 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000231351 2024-09-13T11:25:27.490 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:27.490 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:27.490 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:25:27.491 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.457331+0000", 2024-09-13T11:25:27.491 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.6372000000000001e-05 2024-09-13T11:25:27.491 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:27.491 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:27.491 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:25:27.491 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.458675+0000", 2024-09-13T11:25:27.491 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001344242 2024-09-13T11:25:27.491 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:27.491 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:25:27.491 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:27.491 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:27.491 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:27.491 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPush(3.17 83/70 [PushOp(3:eb574268:::benchmark_data_smithi177_31116_object40622:head, version: 64'2475, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:eb574268:::benchmark_data_smithi177_31116_object40622:head@64'2475, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:25:27.492 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:40.760042+0000", 2024-09-13T11:25:27.492 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 106.725967922, 2024-09-13T11:25:27.492 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.30544642100000002, 2024-09-13T11:25:27.492 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:25:27.492 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:25:27.492 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:25:27.492 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:25:27.492 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:27.492 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:25:27.492 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:40.760042+0000", 2024-09-13T11:25:27.492 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:27.492 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:27.492 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:27.493 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:25:27.493 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:40.760042+0000", 2024-09-13T11:25:27.493 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:27.493 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:27.493 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:27.493 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:25:27.493 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:40.760044+0000", 2024-09-13T11:25:27.493 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.511e-06 2024-09-13T11:25:27.493 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:27.493 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:27.493 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:25:27.493 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:40.760149+0000", 2024-09-13T11:25:27.493 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000105473 2024-09-13T11:25:27.493 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:27.494 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:27.494 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:25:27.494 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.064465+0000", 2024-09-13T11:25:27.494 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.30431613800000001 2024-09-13T11:25:27.494 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:27.494 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:27.494 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:25:27.494 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.064479+0000", 2024-09-13T11:25:27.494 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3586e-05 2024-09-13T11:25:27.494 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:27.494 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:27.494 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:25:27.494 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.064553+0000", 2024-09-13T11:25:27.494 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.3646000000000006e-05 2024-09-13T11:25:27.495 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:27.495 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:27.495 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:25:27.495 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.064568+0000", 2024-09-13T11:25:27.495 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.52e-05 2024-09-13T11:25:27.495 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:27.495 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:27.495 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:25:27.495 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.065489+0000", 2024-09-13T11:25:27.495 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00092086700000000002 2024-09-13T11:25:27.496 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:27.496 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:25:27.496 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:27.496 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:27.496 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:27.496 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPushReply(3.1a 83/78 [PushReplyOp(3:5c3834bd:::benchmark_data_smithi177_31116_object8669:head)])", 2024-09-13T11:25:27.496 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:40.798236+0000", 2024-09-13T11:25:27.496 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 106.687774492, 2024-09-13T11:25:27.496 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.29701520999999997, 2024-09-13T11:25:27.496 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:25:27.496 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:25:27.496 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-13T11:25:27.496 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:25:27.496 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:27.497 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:25:27.497 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:40.798236+0000", 2024-09-13T11:25:27.497 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:27.497 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:27.497 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:27.497 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:25:27.497 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:40.798236+0000", 2024-09-13T11:25:27.497 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:27.497 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:27.498 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:27.498 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:25:27.498 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:40.798237+0000", 2024-09-13T11:25:27.498 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.0500000000000002e-07 2024-09-13T11:25:27.498 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:27.498 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:27.498 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:25:27.498 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:40.798240+0000", 2024-09-13T11:25:27.498 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.4819999999999999e-06 2024-09-13T11:25:27.498 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:27.498 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:27.498 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:25:27.498 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.093360+0000", 2024-09-13T11:25:27.498 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.29512009 2024-09-13T11:25:27.499 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:27.499 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:27.499 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:25:27.499 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.093372+0000", 2024-09-13T11:25:27.499 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.2221e-05 2024-09-13T11:25:27.499 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:27.499 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:27.499 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:25:27.499 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.093482+0000", 2024-09-13T11:25:27.499 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000109571 2024-09-13T11:25:27.499 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:27.499 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:27.499 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:25:27.499 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.095251+0000", 2024-09-13T11:25:27.500 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001768941 2024-09-13T11:25:27.500 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:27.500 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:25:27.500 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:27.500 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:27.500 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:27.500 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPushReply(3.1a 83/78 [PushReplyOp(3:58c7ffc0:::benchmark_data_smithi177_31116_object8738:head)])", 2024-09-13T11:25:27.500 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:40.805941+0000", 2024-09-13T11:25:27.500 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 106.680069238, 2024-09-13T11:25:27.500 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.30315678499999998, 2024-09-13T11:25:27.500 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:25:27.500 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:25:27.501 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-13T11:25:27.501 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:25:27.501 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:27.501 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:25:27.501 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:40.805941+0000", 2024-09-13T11:25:27.501 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:27.501 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:27.501 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:27.501 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:25:27.501 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:40.805941+0000", 2024-09-13T11:25:27.501 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:27.501 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:27.501 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:27.502 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:25:27.502 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:40.805942+0000", 2024-09-13T11:25:27.502 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3769999999999999e-06 2024-09-13T11:25:27.502 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:27.502 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:27.502 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:25:27.502 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:40.805947+0000", 2024-09-13T11:25:27.502 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.8980000000000004e-06 2024-09-13T11:25:27.502 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:27.502 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:27.502 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:25:27.502 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.106505+0000", 2024-09-13T11:25:27.502 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.30055777700000003 2024-09-13T11:25:27.502 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:27.503 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:27.503 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:25:27.503 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.106512+0000", 2024-09-13T11:25:27.503 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.3749999999999997e-06 2024-09-13T11:25:27.503 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:27.503 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:27.503 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:25:27.503 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.106566+0000", 2024-09-13T11:25:27.503 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.3103999999999998e-05 2024-09-13T11:25:27.503 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:27.503 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:27.503 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:25:27.503 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.109098+0000", 2024-09-13T11:25:27.503 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0025322539999999998 2024-09-13T11:25:27.504 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:27.504 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:25:27.504 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:27.504 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:27.504 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:27.504 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPushReply(3.1a 83/78 [PushReplyOp(3:58c7ffc0:::benchmark_data_smithi177_31116_object8738:head)])", 2024-09-13T11:25:27.504 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:40.805973+0000", 2024-09-13T11:25:27.504 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 106.680037235, 2024-09-13T11:25:27.504 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.28729672099999998, 2024-09-13T11:25:27.504 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:25:27.504 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:25:27.504 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-13T11:25:27.504 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:25:27.504 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:27.505 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:25:27.505 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:40.805973+0000", 2024-09-13T11:25:27.505 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:27.505 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:27.505 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:27.505 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:25:27.505 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:40.805973+0000", 2024-09-13T11:25:27.505 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:27.505 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:27.505 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:27.505 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:25:27.505 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:40.805974+0000", 2024-09-13T11:25:27.505 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.8599999999999998e-07 2024-09-13T11:25:27.506 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:27.506 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:27.506 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:25:27.506 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:40.805976+0000", 2024-09-13T11:25:27.506 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.2809999999999998e-06 2024-09-13T11:25:27.506 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:27.506 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:27.506 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:25:27.506 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.091588+0000", 2024-09-13T11:25:27.506 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.28561173899999998 2024-09-13T11:25:27.506 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:27.506 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:27.506 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:25:27.506 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.091596+0000", 2024-09-13T11:25:27.507 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.0520000000000002e-06 2024-09-13T11:25:27.507 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:27.507 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:27.507 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:25:27.507 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.091907+0000", 2024-09-13T11:25:27.507 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00031179599999999998 2024-09-13T11:25:27.507 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:27.507 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:27.507 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:25:27.507 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.093270+0000", 2024-09-13T11:25:27.507 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0013622669999999999 2024-09-13T11:25:27.507 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:27.507 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:25:27.507 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:27.508 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:27.508 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:27.508 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPushReply(3.1a 84/78 [PushReplyOp(3:581ea97e:::benchmark_data_smithi177_31116_object10255:head)])", 2024-09-13T11:25:27.508 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:41.514228+0000", 2024-09-13T11:25:27.508 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 105.97178174699999, 2024-09-13T11:25:27.508 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.40695564200000001, 2024-09-13T11:25:27.508 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:25:27.508 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:25:27.508 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-13T11:25:27.508 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:25:27.508 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:27.508 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:25:27.508 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.514228+0000", 2024-09-13T11:25:27.508 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:27.509 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:27.509 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:27.509 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:25:27.509 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.514228+0000", 2024-09-13T11:25:27.509 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:27.509 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:27.509 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:27.509 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:25:27.509 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.514230+0000", 2024-09-13T11:25:27.509 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.499e-06 2024-09-13T11:25:27.509 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:27.509 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:27.509 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:25:27.509 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.514236+0000", 2024-09-13T11:25:27.510 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.5389999999999998e-06 2024-09-13T11:25:27.510 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:27.510 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:27.510 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:25:27.510 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.919787+0000", 2024-09-13T11:25:27.510 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.40555152999999999 2024-09-13T11:25:27.510 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:27.510 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:27.510 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:25:27.510 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.919796+0000", 2024-09-13T11:25:27.510 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.4829999999999999e-06 2024-09-13T11:25:27.510 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:27.510 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:27.510 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:25:27.510 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.920366+0000", 2024-09-13T11:25:27.511 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00057061700000000004 2024-09-13T11:25:27.511 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:27.511 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:27.511 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:25:27.511 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.921184+0000", 2024-09-13T11:25:27.511 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00081797399999999996 2024-09-13T11:25:27.511 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:27.511 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:25:27.511 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:27.511 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:27.511 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:27.511 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:60312 3.14 3:2e846241:::benchmark_data_smithi177_31116_object60311:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e85)", 2024-09-13T11:25:27.511 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:42.065725+0000", 2024-09-13T11:25:27.511 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 105.420285104, 2024-09-13T11:25:27.512 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.211323176, 2024-09-13T11:25:27.512 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:25:27.512 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:25:27.512 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:27.512 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:25:27.512 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:25:27.512 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:27.512 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 60312 2024-09-13T11:25:27.512 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:27.512 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:25:27.512 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:27.512 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:25:27.512 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.065725+0000", 2024-09-13T11:25:27.512 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:27.513 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:27.513 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:27.513 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:25:27.513 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.065725+0000", 2024-09-13T11:25:27.513 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:27.513 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:27.513 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:27.513 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:25:27.513 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.065728+0000", 2024-09-13T11:25:27.513 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.3400000000000002e-06 2024-09-13T11:25:27.513 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:27.513 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:27.513 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:25:27.513 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.065759+0000", 2024-09-13T11:25:27.514 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.0074e-05 2024-09-13T11:25:27.514 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:27.514 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:27.514 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:25:27.514 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.065771+0000", 2024-09-13T11:25:27.514 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.2869e-05 2024-09-13T11:25:27.514 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:27.514 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:27.514 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:25:27.514 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.065780+0000", 2024-09-13T11:25:27.514 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.3489999999999992e-06 2024-09-13T11:25:27.514 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:27.514 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:27.514 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:25:27.514 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.065837+0000", 2024-09-13T11:25:27.515 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.7444000000000001e-05 2024-09-13T11:25:27.515 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:27.515 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:27.515 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:25:27.515 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.065966+0000", 2024-09-13T11:25:27.515 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000128973 2024-09-13T11:25:27.515 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:27.515 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:27.515 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 14", 2024-09-13T11:25:27.515 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.066114+0000", 2024-09-13T11:25:27.515 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00014788299999999999 2024-09-13T11:25:27.515 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:27.515 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:27.515 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:25:27.516 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.067604+0000", 2024-09-13T11:25:27.516 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001489637 2024-09-13T11:25:27.516 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:27.516 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:27.516 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:25:27.516 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.276963+0000", 2024-09-13T11:25:27.516 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.20935888499999999 2024-09-13T11:25:27.516 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:27.516 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:27.516 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:25:27.516 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.277003+0000", 2024-09-13T11:25:27.516 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.0241000000000003e-05 2024-09-13T11:25:27.516 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:27.516 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:27.517 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:25:27.517 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.277048+0000", 2024-09-13T11:25:27.517 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.5481e-05 2024-09-13T11:25:27.517 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:27.517 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:25:27.517 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:27.517 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:27.517 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:27.517 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop_reply(client.4665.0:60312 3.14 e85/84 ondisk, result = 0)", 2024-09-13T11:25:27.517 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:42.069120+0000", 2024-09-13T11:25:27.517 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 105.41688998399999, 2024-09-13T11:25:27.517 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.20804031000000001, 2024-09-13T11:25:27.517 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:25:27.517 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:25:27.518 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:25:27.518 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:25:27.518 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:27.518 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:25:27.518 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.069120+0000", 2024-09-13T11:25:27.518 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:27.518 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:27.518 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:27.518 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:25:27.518 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.069120+0000", 2024-09-13T11:25:27.518 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:27.518 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:27.518 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:27.519 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:25:27.519 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.069123+0000", 2024-09-13T11:25:27.519 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.289e-06 2024-09-13T11:25:27.519 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:27.519 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:27.519 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:25:27.519 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.069128+0000", 2024-09-13T11:25:27.519 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.3530000000000002e-06 2024-09-13T11:25:27.519 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:27.519 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:27.519 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:25:27.519 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.276885+0000", 2024-09-13T11:25:27.519 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.207756989 2024-09-13T11:25:27.519 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:27.519 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:27.520 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:25:27.520 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.276892+0000", 2024-09-13T11:25:27.520 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.922e-06 2024-09-13T11:25:27.520 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:27.520 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:27.520 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:25:27.520 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.276953+0000", 2024-09-13T11:25:27.520 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.0890000000000001e-05 2024-09-13T11:25:27.520 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:27.520 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:27.520 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:25:27.520 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.276959+0000", 2024-09-13T11:25:27.520 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.9179999999999999e-06 2024-09-13T11:25:27.521 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:27.521 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:27.521 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:25:27.521 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.277161+0000", 2024-09-13T11:25:27.521 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00020194899999999999 2024-09-13T11:25:27.521 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:27.521 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:25:27.521 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:27.521 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:27.521 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:27.521 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:60427 3.14 3:2afea229:::benchmark_data_smithi177_31116_object60426:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e85)", 2024-09-13T11:25:27.521 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:42.187231+0000", 2024-09-13T11:25:27.521 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 105.29877887799999, 2024-09-13T11:25:27.521 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.83900596999999999, 2024-09-13T11:25:27.522 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:25:27.522 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:25:27.522 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:27.522 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:25:27.522 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:25:27.522 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:27.522 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 60427 2024-09-13T11:25:27.522 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:27.522 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:25:27.522 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:27.522 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:25:27.522 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.187231+0000", 2024-09-13T11:25:27.522 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:27.522 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:27.522 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:27.523 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:25:27.523 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.187231+0000", 2024-09-13T11:25:27.523 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:27.523 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:27.523 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:27.523 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:25:27.523 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.187235+0000", 2024-09-13T11:25:27.523 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.3579999999999999e-06 2024-09-13T11:25:27.523 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:27.523 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:27.523 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:25:27.523 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.187261+0000", 2024-09-13T11:25:27.523 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.6089e-05 2024-09-13T11:25:27.523 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:27.524 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:27.524 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:25:27.524 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.187275+0000", 2024-09-13T11:25:27.524 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.4653e-05 2024-09-13T11:25:27.524 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:27.524 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:27.524 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:25:27.524 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.187282+0000", 2024-09-13T11:25:27.524 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.8870000000000003e-06 2024-09-13T11:25:27.524 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:27.524 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:27.524 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:25:27.524 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.187336+0000", 2024-09-13T11:25:27.524 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.3875999999999997e-05 2024-09-13T11:25:27.525 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:27.525 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:27.525 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:25:27.525 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.187480+0000", 2024-09-13T11:25:27.525 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00014330899999999999 2024-09-13T11:25:27.525 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:27.525 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:27.525 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 14", 2024-09-13T11:25:27.525 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.187658+0000", 2024-09-13T11:25:27.525 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00017861900000000001 2024-09-13T11:25:27.525 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:27.525 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:27.525 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:25:27.525 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.189353+0000", 2024-09-13T11:25:27.526 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001695144 2024-09-13T11:25:27.526 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:27.526 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:27.526 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:25:27.526 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:43.026081+0000", 2024-09-13T11:25:27.526 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.83672736199999997 2024-09-13T11:25:27.526 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:27.526 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:27.526 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:25:27.526 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:43.026169+0000", 2024-09-13T11:25:27.526 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.8411000000000004e-05 2024-09-13T11:25:27.526 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:27.526 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:27.526 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:25:27.526 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:43.026237+0000", 2024-09-13T11:25:27.527 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.8262000000000005e-05 2024-09-13T11:25:27.527 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:27.527 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:25:27.527 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:27.527 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:27.527 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:27.527 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop_reply(client.4665.0:60427 3.14 e85/84 ondisk, result = 0)", 2024-09-13T11:25:27.527 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:42.190914+0000", 2024-09-13T11:25:27.527 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 105.295096214, 2024-09-13T11:25:27.527 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.83534071499999996, 2024-09-13T11:25:27.527 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:25:27.527 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:25:27.527 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:25:27.527 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:25:27.528 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:27.528 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:25:27.528 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.190914+0000", 2024-09-13T11:25:27.528 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:27.528 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:27.528 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:27.528 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:25:27.528 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.190914+0000", 2024-09-13T11:25:27.528 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:27.528 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:27.528 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:27.528 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:25:27.528 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.190916+0000", 2024-09-13T11:25:27.528 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.84e-06 2024-09-13T11:25:27.529 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:27.529 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:27.529 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:25:27.529 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.190921+0000", 2024-09-13T11:25:27.529 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.2039999999999996e-06 2024-09-13T11:25:27.529 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:27.529 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:27.529 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:25:27.529 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:43.025926+0000", 2024-09-13T11:25:27.529 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.83500542 2024-09-13T11:25:27.529 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:27.529 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:27.529 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:25:27.529 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:43.025937+0000", 2024-09-13T11:25:27.530 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0597e-05 2024-09-13T11:25:27.530 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:27.530 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:27.530 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:25:27.530 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:43.026052+0000", 2024-09-13T11:25:27.530 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000114838 2024-09-13T11:25:27.530 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:27.530 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:27.530 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:25:27.530 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:43.026070+0000", 2024-09-13T11:25:27.530 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.8131e-05 2024-09-13T11:25:27.530 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:27.530 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:27.530 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:25:27.531 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:43.026255+0000", 2024-09-13T11:25:27.531 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000184685 2024-09-13T11:25:27.531 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:27.531 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:25:27.531 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:27.531 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:27.531 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:27.531 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPushReply(3.1d 89/78 [PushReplyOp(3:bc1b7cfb:::benchmark_data_smithi177_31116_object22629:head)])", 2024-09-13T11:25:27.531 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:46.663106+0000", 2024-09-13T11:25:27.531 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 100.82290403099999, 2024-09-13T11:25:27.531 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.26645533199999999, 2024-09-13T11:25:27.531 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:25:27.531 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:25:27.531 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-13T11:25:27.532 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:25:27.532 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:27.532 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:25:27.532 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.663106+0000", 2024-09-13T11:25:27.532 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:27.532 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:27.532 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:27.532 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:25:27.532 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.663106+0000", 2024-09-13T11:25:27.532 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:27.532 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:27.532 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:27.532 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:25:27.532 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.663108+0000", 2024-09-13T11:25:27.533 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.0140000000000001e-06 2024-09-13T11:25:27.533 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:27.533 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:27.533 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:25:27.533 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.663116+0000", 2024-09-13T11:25:27.533 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.5440000000000001e-06 2024-09-13T11:25:27.533 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:27.533 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:27.533 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:25:27.533 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.928111+0000", 2024-09-13T11:25:27.533 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.26499566899999999 2024-09-13T11:25:27.533 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:27.533 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:27.533 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:25:27.534 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.928135+0000", 2024-09-13T11:25:27.534 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.4003e-05 2024-09-13T11:25:27.534 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:27.534 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:27.534 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:25:27.534 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.928559+0000", 2024-09-13T11:25:27.534 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000423318 2024-09-13T11:25:27.534 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:27.534 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:27.534 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:25:27.534 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.929562+0000", 2024-09-13T11:25:27.534 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0010027840000000001 2024-09-13T11:25:27.534 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:27.534 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:25:27.535 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:27.535 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:27.535 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:27.535 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "rep_scrubmap(3.14 e84 from shard 14)", 2024-09-13T11:25:27.535 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:48.313615+0000", 2024-09-13T11:25:27.535 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 99.172395238000007, 2024-09-13T11:25:27.535 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.35633836099999999, 2024-09-13T11:25:27.535 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:25:27.535 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:25:27.535 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:25:27.535 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:25:27.535 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:27.535 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:25:27.535 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.313615+0000", 2024-09-13T11:25:27.536 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:27.536 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:27.536 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:27.536 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:25:27.536 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.313615+0000", 2024-09-13T11:25:27.536 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:27.536 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:27.536 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:27.536 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:25:27.536 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.313617+0000", 2024-09-13T11:25:27.536 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.7990000000000001e-06 2024-09-13T11:25:27.536 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:27.536 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:27.536 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:25:27.536 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.313688+0000", 2024-09-13T11:25:27.537 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.1481000000000002e-05 2024-09-13T11:25:27.537 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:27.537 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:27.537 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:25:27.537 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.669785+0000", 2024-09-13T11:25:27.537 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.356096466 2024-09-13T11:25:27.537 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:27.537 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:27.537 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:25:27.537 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.669792+0000", 2024-09-13T11:25:27.537 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.1740000000000003e-06 2024-09-13T11:25:27.537 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:27.537 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:27.537 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:25:27.538 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.669839+0000", 2024-09-13T11:25:27.538 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.7323000000000001e-05 2024-09-13T11:25:27.538 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:27.538 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:27.538 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:25:27.538 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.669847+0000", 2024-09-13T11:25:27.538 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.0469999999999994e-06 2024-09-13T11:25:27.538 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:27.538 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:27.538 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:25:27.538 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.669953+0000", 2024-09-13T11:25:27.538 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00010607099999999999 2024-09-13T11:25:27.538 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:27.538 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:25:27.539 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:27.539 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:27.539 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:27.539 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:110268 3.14 3:2997c4c5:::benchmark_data_smithi177_31116_object110267:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119)", 2024-09-13T11:25:27.539 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:25.359386+0000", 2024-09-13T11:25:27.539 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 62.126623922, 2024-09-13T11:25:27.539 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.29072703, 2024-09-13T11:25:27.539 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:25:27.539 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:25:27.539 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:27.539 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:25:27.539 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:25:27.539 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:27.539 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 110268 2024-09-13T11:25:27.540 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:27.540 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:25:27.540 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:27.540 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:25:27.540 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.359386+0000", 2024-09-13T11:25:27.540 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:27.540 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:27.540 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:27.540 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:25:27.540 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.359386+0000", 2024-09-13T11:25:27.540 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:27.540 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:27.540 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:27.540 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:25:27.541 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.359389+0000", 2024-09-13T11:25:27.541 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.8600000000000001e-06 2024-09-13T11:25:27.541 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:27.541 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:27.541 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:25:27.541 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.359424+0000", 2024-09-13T11:25:27.541 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.4755999999999997e-05 2024-09-13T11:25:27.541 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:27.541 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:27.541 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:25:27.541 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.359445+0000", 2024-09-13T11:25:27.541 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.1104999999999999e-05 2024-09-13T11:25:27.541 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:27.541 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:27.541 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:25:27.542 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.359460+0000", 2024-09-13T11:25:27.542 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5396999999999998e-05 2024-09-13T11:25:27.542 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:27.542 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:27.542 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:25:27.542 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.359581+0000", 2024-09-13T11:25:27.542 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000120455 2024-09-13T11:25:27.542 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:27.542 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:27.542 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:25:27.542 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.359827+0000", 2024-09-13T11:25:27.542 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00024571 2024-09-13T11:25:27.542 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:27.542 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:27.543 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 14", 2024-09-13T11:25:27.543 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.360084+0000", 2024-09-13T11:25:27.543 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00025787999999999999 2024-09-13T11:25:27.543 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:27.543 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:27.543 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:25:27.543 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.362243+0000", 2024-09-13T11:25:27.543 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0021581579999999999 2024-09-13T11:25:27.543 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:27.543 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:27.543 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:25:27.543 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.650034+0000", 2024-09-13T11:25:27.543 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.287791662 2024-09-13T11:25:27.544 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:27.544 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:27.544 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:25:27.544 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.650077+0000", 2024-09-13T11:25:27.544 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.2367999999999998e-05 2024-09-13T11:25:27.544 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:27.544 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:27.544 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:25:27.544 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.650113+0000", 2024-09-13T11:25:27.544 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.6678999999999999e-05 2024-09-13T11:25:27.544 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:27.544 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:25:27.544 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:27.544 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:27.545 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:27.545 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:110279 3.14 3:2b17a349:::benchmark_data_smithi177_31116_object110278:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119)", 2024-09-13T11:25:27.545 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:25.645137+0000", 2024-09-13T11:25:27.545 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 61.840873084000002, 2024-09-13T11:25:27.545 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.28172927800000003, 2024-09-13T11:25:27.545 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:25:27.545 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:25:27.545 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:27.545 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:25:27.545 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:25:27.545 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:27.545 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 110279 2024-09-13T11:25:27.545 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:27.546 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:25:27.546 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:27.546 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:25:27.546 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.645137+0000", 2024-09-13T11:25:27.546 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:27.546 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:27.546 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:27.546 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:25:27.546 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.645137+0000", 2024-09-13T11:25:27.546 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:27.546 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:27.546 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:27.546 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:25:27.546 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.645140+0000", 2024-09-13T11:25:27.546 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.0939999999999999e-06 2024-09-13T11:25:27.547 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:27.547 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:27.547 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:25:27.547 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.645184+0000", 2024-09-13T11:25:27.547 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.3412999999999999e-05 2024-09-13T11:25:27.547 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:27.547 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:27.547 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:25:27.547 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.645206+0000", 2024-09-13T11:25:27.547 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.1997e-05 2024-09-13T11:25:27.547 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:27.547 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:27.547 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:25:27.547 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.645216+0000", 2024-09-13T11:25:27.548 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0740999999999999e-05 2024-09-13T11:25:27.548 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:27.548 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:27.548 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:25:27.548 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.645397+0000", 2024-09-13T11:25:27.548 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00018077600000000001 2024-09-13T11:25:27.548 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:27.548 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:27.548 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:25:27.548 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.645638+0000", 2024-09-13T11:25:27.548 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000241091 2024-09-13T11:25:27.548 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:27.548 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:27.549 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 14", 2024-09-13T11:25:27.549 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.645922+0000", 2024-09-13T11:25:27.549 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000283565 2024-09-13T11:25:27.549 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:27.549 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:27.549 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:25:27.549 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.648980+0000", 2024-09-13T11:25:27.549 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00305851 2024-09-13T11:25:27.549 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:27.549 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:27.549 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:25:27.549 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.926702+0000", 2024-09-13T11:25:27.549 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.27772199800000003 2024-09-13T11:25:27.550 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:27.550 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:27.550 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:25:27.550 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.926796+0000", 2024-09-13T11:25:27.551 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.4158000000000002e-05 2024-09-13T11:25:27.551 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:27.551 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:27.551 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:25:27.551 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.926866+0000", 2024-09-13T11:25:27.551 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.9935e-05 2024-09-13T11:25:27.551 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:27.551 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:25:27.551 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:27.551 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:27.551 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:27.551 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:129865 3.14 3:2c872f8e:::benchmark_data_smithi177_31116_object129864:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e134)", 2024-09-13T11:25:27.551 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:47.107186+0000", 2024-09-13T11:25:27.551 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 40.378824526999999, 2024-09-13T11:25:27.552 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.21814059299999999, 2024-09-13T11:25:27.552 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:25:27.552 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:25:27.552 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:27.552 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:25:27.552 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:25:27.552 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:27.552 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 129865 2024-09-13T11:25:27.552 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:27.552 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:25:27.552 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:27.552 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:25:27.552 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:47.107186+0000", 2024-09-13T11:25:27.552 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:27.553 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:27.553 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:27.553 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:25:27.553 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:47.107186+0000", 2024-09-13T11:25:27.553 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:27.553 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:27.553 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:27.553 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:25:27.553 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:47.107187+0000", 2024-09-13T11:25:27.553 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5740000000000001e-06 2024-09-13T11:25:27.553 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:27.553 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:27.553 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:25:27.554 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:47.107207+0000", 2024-09-13T11:25:27.554 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.9304999999999999e-05 2024-09-13T11:25:27.554 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:27.554 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:27.554 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:25:27.554 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:47.107215+0000", 2024-09-13T11:25:27.554 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.7390000000000005e-06 2024-09-13T11:25:27.554 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:27.554 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:27.554 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:25:27.554 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:47.107221+0000", 2024-09-13T11:25:27.554 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.7350000000000001e-06 2024-09-13T11:25:27.554 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:27.554 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:27.554 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:25:27.555 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:47.107278+0000", 2024-09-13T11:25:27.555 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.7209000000000001e-05 2024-09-13T11:25:27.555 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:27.555 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:27.555 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:25:27.555 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:47.107383+0000", 2024-09-13T11:25:27.555 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000104718 2024-09-13T11:25:27.555 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:27.555 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:27.555 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 15", 2024-09-13T11:25:27.555 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:47.107480+0000", 2024-09-13T11:25:27.555 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.7343000000000006e-05 2024-09-13T11:25:27.555 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:27.556 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:27.556 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:25:27.556 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:47.109477+0000", 2024-09-13T11:25:27.556 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0019968490000000002 2024-09-13T11:25:27.556 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:27.556 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:27.556 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:25:27.556 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:47.324147+0000", 2024-09-13T11:25:27.556 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.21467018199999999 2024-09-13T11:25:27.556 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:27.556 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:27.556 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:25:27.556 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:47.324281+0000", 2024-09-13T11:25:27.556 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00013328600000000001 2024-09-13T11:25:27.557 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:27.557 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:27.557 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:25:27.557 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:47.325326+0000", 2024-09-13T11:25:27.557 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0010456529999999999 2024-09-13T11:25:27.557 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:27.557 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:25:27.557 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:27.557 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:27.557 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:27.557 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPush(3.15 140/122 [PushOp(3:ae90e82d:::benchmark_data_smithi177_31116_object53962:head, version: 74'3006, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:ae90e82d:::benchmark_data_smithi177_31116_object53962:head@74'3006, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:25:27.557 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:54.310786+0000", 2024-09-13T11:25:27.557 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 33.175224053999997, 2024-09-13T11:25:27.557 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.229345255, 2024-09-13T11:25:27.558 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:25:27.558 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:25:27.558 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:25:27.558 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:25:27.558 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:27.558 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:25:27.558 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:54.310786+0000", 2024-09-13T11:25:27.558 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:27.558 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:27.558 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:27.558 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:25:27.558 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:54.310786+0000", 2024-09-13T11:25:27.558 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:27.558 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:27.558 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:27.559 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:25:27.559 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:54.310795+0000", 2024-09-13T11:25:27.559 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.1910000000000006e-06 2024-09-13T11:25:27.559 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:27.559 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:27.559 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:25:27.559 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:54.310829+0000", 2024-09-13T11:25:27.559 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.4121000000000001e-05 2024-09-13T11:25:27.559 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:27.559 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:27.559 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:25:27.559 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:54.538419+0000", 2024-09-13T11:25:27.559 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.227589611 2024-09-13T11:25:27.560 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:27.560 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:27.560 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:25:27.560 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:54.538430+0000", 2024-09-13T11:25:27.560 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0456e-05 2024-09-13T11:25:27.560 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:27.560 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:27.560 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:25:27.560 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:54.538497+0000", 2024-09-13T11:25:27.560 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.6948999999999999e-05 2024-09-13T11:25:27.560 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:27.560 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:27.560 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:25:27.560 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:54.538505+0000", 2024-09-13T11:25:27.560 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.2989999999999997e-06 2024-09-13T11:25:27.561 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:27.561 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:27.561 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:25:27.561 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:54.540131+0000", 2024-09-13T11:25:27.561 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001626628 2024-09-13T11:25:27.561 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:27.561 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:25:27.561 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:27.561 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:27.561 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:27.561 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPush(3.15 143/122 [PushOp(3:ae52bb45:::benchmark_data_smithi177_31116_object61893:head, version: 87'3272, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:ae52bb45:::benchmark_data_smithi177_31116_object61893:head@87'3272, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:25:27.561 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:57.633939+0000", 2024-09-13T11:25:27.561 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 29.852071101, 2024-09-13T11:25:27.561 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.339405067, 2024-09-13T11:25:27.562 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:25:27.562 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:25:27.562 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:25:27.562 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:25:27.562 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:27.562 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:25:27.562 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:57.633939+0000", 2024-09-13T11:25:27.562 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:27.562 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:27.562 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:27.562 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:25:27.562 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:57.633939+0000", 2024-09-13T11:25:27.562 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:27.563 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:27.563 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:27.563 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:25:27.563 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:57.633941+0000", 2024-09-13T11:25:27.563 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.911e-06 2024-09-13T11:25:27.563 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:27.563 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:27.563 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:25:27.563 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:57.633994+0000", 2024-09-13T11:25:27.563 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.2753999999999997e-05 2024-09-13T11:25:27.563 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:27.563 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:27.563 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:25:27.563 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:57.968403+0000", 2024-09-13T11:25:27.564 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.334409336 2024-09-13T11:25:27.564 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:27.564 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:27.564 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:25:27.564 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:57.968451+0000", 2024-09-13T11:25:27.564 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.7525000000000003e-05 2024-09-13T11:25:27.564 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:27.564 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:27.564 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:25:27.564 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:57.968660+0000", 2024-09-13T11:25:27.564 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000209204 2024-09-13T11:25:27.564 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:27.564 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:27.564 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:25:27.565 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:57.968680+0000", 2024-09-13T11:25:27.565 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.9639999999999998e-05 2024-09-13T11:25:27.565 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:27.565 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:27.565 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:25:27.565 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:57.973344+0000", 2024-09-13T11:25:27.565 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0046646969999999998 2024-09-13T11:25:27.565 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:27.565 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:25:27.565 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:27.565 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:27.565 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:27.565 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:140658 3.17 e144/85 3:ece5a1b3:::benchmark_data_smithi177_31116_object355:head v 144'5654, mlcod=144'5652)", 2024-09-13T11:25:27.565 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:58.247792+0000", 2024-09-13T11:25:27.566 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 29.238218623000002, 2024-09-13T11:25:27.566 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.29148369699999999, 2024-09-13T11:25:27.566 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:25:27.566 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:25:27.566 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:27.566 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:25:27.566 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:27.566 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:25:27.566 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:58.247792+0000", 2024-09-13T11:25:27.566 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:27.566 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:27.566 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:27.566 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:25:27.566 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:58.247792+0000", 2024-09-13T11:25:27.566 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:27.567 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:27.567 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:27.567 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:25:27.567 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:58.247793+0000", 2024-09-13T11:25:27.567 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.347e-06 2024-09-13T11:25:27.567 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:27.567 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:27.567 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:25:27.567 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:58.247799+0000", 2024-09-13T11:25:27.567 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.5239999999999997e-06 2024-09-13T11:25:27.567 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:27.567 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:27.567 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:25:27.567 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:58.535226+0000", 2024-09-13T11:25:27.568 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.28742604900000002 2024-09-13T11:25:27.568 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:27.568 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:27.568 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:25:27.568 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:58.535238+0000", 2024-09-13T11:25:27.568 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.2843000000000001e-05 2024-09-13T11:25:27.568 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:27.568 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:27.568 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:25:27.568 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:58.535393+0000", 2024-09-13T11:25:27.568 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00015481699999999999 2024-09-13T11:25:27.568 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:27.568 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:27.568 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:25:27.569 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:58.535434+0000", 2024-09-13T11:25:27.569 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.1152999999999997e-05 2024-09-13T11:25:27.569 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:27.569 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:27.569 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:25:27.569 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:58.539191+0000", 2024-09-13T11:25:27.569 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0037571520000000001 2024-09-13T11:25:27.569 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:27.569 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:27.569 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:25:27.569 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:58.539275+0000", 2024-09-13T11:25:27.569 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.3812000000000003e-05 2024-09-13T11:25:27.569 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:27.569 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:25:27.570 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:27.570 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:27.570 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:27.570 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPush(3.15 144/122 [PushOp(3:a97ba025:::benchmark_data_smithi177_31116_object64775:head, version: 90'3365, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:a97ba025:::benchmark_data_smithi177_31116_object64775:head@90'3365, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:25:27.570 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:58.825014+0000", 2024-09-13T11:25:27.570 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 28.660995995, 2024-09-13T11:25:27.570 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.23868276499999999, 2024-09-13T11:25:27.570 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:25:27.570 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:25:27.570 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:25:27.570 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:25:27.570 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:27.570 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:25:27.570 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:58.825014+0000", 2024-09-13T11:25:27.571 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:27.571 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:27.571 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:27.571 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:25:27.571 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:58.825014+0000", 2024-09-13T11:25:27.571 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:27.571 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:27.571 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:27.571 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:25:27.571 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:58.825017+0000", 2024-09-13T11:25:27.571 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.7630000000000001e-06 2024-09-13T11:25:27.571 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:27.571 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:27.571 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:25:27.572 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:58.825099+0000", 2024-09-13T11:25:27.572 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.2035999999999999e-05 2024-09-13T11:25:27.572 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:27.572 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:27.572 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:25:27.572 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:59.059557+0000", 2024-09-13T11:25:27.572 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.23445790399999999 2024-09-13T11:25:27.572 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:27.572 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:27.572 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:25:27.572 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:59.059609+0000", 2024-09-13T11:25:27.572 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.2550999999999999e-05 2024-09-13T11:25:27.572 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:27.572 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:27.573 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:25:27.573 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:59.060117+0000", 2024-09-13T11:25:27.573 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00050754299999999995 2024-09-13T11:25:27.573 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:27.573 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:27.573 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:25:27.573 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:59.060139+0000", 2024-09-13T11:25:27.573 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.2306e-05 2024-09-13T11:25:27.573 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:27.573 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:27.573 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:25:27.573 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:59.063697+0000", 2024-09-13T11:25:27.573 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0035576620000000001 2024-09-13T11:25:27.573 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:27.574 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:25:27.574 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:27.574 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:27.574 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:27.574 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:147472 3.2d 3:b6aeefb0:::benchmark_data_smithi177_31116_object7169:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e148)", 2024-09-13T11:25:27.574 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:25:04.123331+0000", 2024-09-13T11:25:27.574 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 23.362679734, 2024-09-13T11:25:27.574 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.23047060799999999, 2024-09-13T11:25:27.574 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:25:27.574 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:25:27.574 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:27.574 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:25:27.574 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:25:27.575 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:27.575 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 147472 2024-09-13T11:25:27.575 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:27.575 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:25:27.575 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:27.575 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:25:27.575 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:04.123331+0000", 2024-09-13T11:25:27.575 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:27.575 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:27.575 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:27.575 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:25:27.575 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:04.123331+0000", 2024-09-13T11:25:27.575 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:27.575 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:27.576 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:27.576 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:25:27.576 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:04.123332+0000", 2024-09-13T11:25:27.576 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.556e-06 2024-09-13T11:25:27.576 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:27.576 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:27.576 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:25:27.576 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:04.123337+0000", 2024-09-13T11:25:27.576 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.5889999999999996e-06 2024-09-13T11:25:27.576 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:27.576 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:27.576 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:25:27.576 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:04.123344+0000", 2024-09-13T11:25:27.576 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.7889999999999995e-06 2024-09-13T11:25:27.577 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:27.577 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:27.577 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:25:27.577 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:04.123350+0000", 2024-09-13T11:25:27.577 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.3820000000000003e-06 2024-09-13T11:25:27.577 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:27.577 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:27.577 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:25:27.577 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:04.123402+0000", 2024-09-13T11:25:27.577 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.2281999999999999e-05 2024-09-13T11:25:27.577 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:27.577 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:27.577 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for degraded object", 2024-09-13T11:25:27.577 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:04.124030+0000", 2024-09-13T11:25:27.578 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00062812999999999996 2024-09-13T11:25:27.578 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:27.578 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:27.578 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:25:27.578 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:04.137378+0000", 2024-09-13T11:25:27.578 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.013347816 2024-09-13T11:25:27.578 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:27.578 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:27.578 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:25:27.578 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:04.137472+0000", 2024-09-13T11:25:27.578 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.3777999999999996e-05 2024-09-13T11:25:27.578 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:27.578 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:27.578 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 0,5", 2024-09-13T11:25:27.579 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:04.137594+0000", 2024-09-13T11:25:27.579 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00012196099999999999 2024-09-13T11:25:27.579 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:27.579 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:27.579 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:25:27.579 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:04.139184+0000", 2024-09-13T11:25:27.579 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001590508 2024-09-13T11:25:27.579 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:27.579 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:27.579 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:25:27.579 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:04.143627+0000", 2024-09-13T11:25:27.579 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0044427989999999999 2024-09-13T11:25:27.579 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:27.579 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:27.579 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:25:27.580 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:04.352741+0000", 2024-09-13T11:25:27.580 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.209114142 2024-09-13T11:25:27.580 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:27.580 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:27.580 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:25:27.580 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:04.352808+0000", 2024-09-13T11:25:27.580 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.7034999999999998e-05 2024-09-13T11:25:27.580 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:27.580 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:27.580 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:25:27.580 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:04.353801+0000", 2024-09-13T11:25:27.580 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00099284099999999995 2024-09-13T11:25:27.580 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:27.580 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:25:27.581 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:27.581 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:27.581 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:25:27.581 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-09-13T11:25:27.585 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-13T11:25:27.585 INFO:tasks.ceph.osd.9.smithi092.stderr:2024-09-13T11:25:27.577+0000 7ff5ab3d0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26348) UID: 0 2024-09-13T11:25:27.683 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-13T11:25:27.679+0000 7f6a68775640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26307) UID: 0 2024-09-13T11:25:27.709 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-09-13T11:25:27.709 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [], 2024-09-13T11:25:27.709 INFO:teuthology.orchestra.run.smithi177.stdout: "num_ops": 0 2024-09-13T11:25:27.709 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-09-13T11:25:27.726 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-13T11:25:27.784 INFO:tasks.ceph.osd.7.smithi177.stderr:2024-09-13T11:25:27.782+0000 7f897b51d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26575) UID: 0 2024-09-13T11:25:27.810 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T11:25:27.863 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-09-13T11:25:27.863 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [], 2024-09-13T11:25:27.864 INFO:teuthology.orchestra.run.smithi177.stdout: "complaint_time": 30, 2024-09-13T11:25:27.864 INFO:teuthology.orchestra.run.smithi177.stdout: "num_blocked_ops": 0 2024-09-13T11:25:27.864 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-09-13T11:25:27.879 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-13T11:25:27.885 INFO:tasks.ceph.osd.2.smithi162.stderr:2024-09-13T11:25:27.884+0000 7fe92b35a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26313) UID: 0 2024-09-13T11:25:27.986 INFO:tasks.ceph.osd.4.smithi001.stderr:2024-09-13T11:25:27.983+0000 7ff8e0218640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26310) UID: 0 2024-09-13T11:25:27.986 INFO:teuthology.orchestra.run.smithi001.stdout:osd.0: {} 2024-09-13T11:25:27.986 INFO:teuthology.orchestra.run.smithi001.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T11:25:28.016 INFO:teuthology.orchestra.run.smithi001.stdout:osd.1: {} 2024-09-13T11:25:28.016 INFO:teuthology.orchestra.run.smithi001.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T11:25:28.046 INFO:teuthology.orchestra.run.smithi001.stdout:osd.2: {} 2024-09-13T11:25:28.046 INFO:teuthology.orchestra.run.smithi001.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T11:25:28.059 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-09-13T11:25:28.060 INFO:teuthology.orchestra.run.smithi177.stdout: "size": 20, 2024-09-13T11:25:28.060 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 600, 2024-09-13T11:25:28.060 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [ 2024-09-13T11:25:28.060 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:28.060 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPushReply(3.13 75/70 [PushReplyOp(3:cfab03d6:::benchmark_data_smithi177_31116_object16599:head)])", 2024-09-13T11:25:28.060 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:31.163480+0000", 2024-09-13T11:25:28.060 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 116.89560028, 2024-09-13T11:25:28.060 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.63009029299999997, 2024-09-13T11:25:28.060 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:25:28.060 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:25:28.060 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-13T11:25:28.060 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:25:28.060 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:28.061 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:25:28.061 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.163480+0000", 2024-09-13T11:25:28.061 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:28.061 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:28.061 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:28.061 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:25:28.061 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.163480+0000", 2024-09-13T11:25:28.061 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:28.061 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:28.061 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:28.061 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:25:28.061 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.163481+0000", 2024-09-13T11:25:28.061 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0410000000000001e-06 2024-09-13T11:25:28.061 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:28.062 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:28.062 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:25:28.062 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.163485+0000", 2024-09-13T11:25:28.062 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.1629999999999998e-06 2024-09-13T11:25:28.062 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:28.062 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:28.062 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:25:28.062 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.792166+0000", 2024-09-13T11:25:28.062 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.62868067400000005 2024-09-13T11:25:28.062 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:28.062 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:28.062 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:25:28.062 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.792184+0000", 2024-09-13T11:25:28.062 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.7898999999999999e-05 2024-09-13T11:25:28.063 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:28.063 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:28.063 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:25:28.063 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.792312+0000", 2024-09-13T11:25:28.063 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00012871000000000001 2024-09-13T11:25:28.063 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:28.063 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:28.063 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:25:28.063 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.793570+0000", 2024-09-13T11:25:28.063 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001257806 2024-09-13T11:25:28.063 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:28.063 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:25:28.063 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:28.063 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:28.064 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:28.064 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPush(3.12 75/69 [PushOp(3:4a36ca6d:::benchmark_data_smithi177_31116_object31707:head, version: 59'1993, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:4a36ca6d:::benchmark_data_smithi177_31116_object31707:head@59'1993, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:25:28.064 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:31.309623+0000", 2024-09-13T11:25:28.064 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 116.74945732499999, 2024-09-13T11:25:28.064 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.165459152, 2024-09-13T11:25:28.064 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:25:28.064 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:25:28.064 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:25:28.064 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:25:28.064 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:28.064 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:25:28.064 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.309623+0000", 2024-09-13T11:25:28.064 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:28.065 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:28.065 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:28.065 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:25:28.065 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.309623+0000", 2024-09-13T11:25:28.065 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:28.065 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:28.065 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:28.065 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:25:28.065 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.309625+0000", 2024-09-13T11:25:28.065 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.3860000000000001e-06 2024-09-13T11:25:28.065 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:28.065 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:28.065 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:25:28.065 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.309680+0000", 2024-09-13T11:25:28.065 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.4388000000000002e-05 2024-09-13T11:25:28.066 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:28.066 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:28.066 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:25:28.066 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.473157+0000", 2024-09-13T11:25:28.066 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.163476813 2024-09-13T11:25:28.066 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:28.066 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:28.066 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:25:28.066 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.473189+0000", 2024-09-13T11:25:28.066 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.2975000000000002e-05 2024-09-13T11:25:28.066 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:28.066 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:28.067 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:25:28.067 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.473395+0000", 2024-09-13T11:25:28.067 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000206009 2024-09-13T11:25:28.067 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:28.067 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:28.067 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:25:28.067 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.473424+0000", 2024-09-13T11:25:28.067 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.8255e-05 2024-09-13T11:25:28.068 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:28.068 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:28.068 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:25:28.068 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.475082+0000", 2024-09-13T11:25:28.068 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0016583259999999999 2024-09-13T11:25:28.068 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:28.068 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:25:28.068 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:28.068 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:28.068 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:28.068 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPushReply(3.13 76/70 [PushReplyOp(3:c9d43a82:::benchmark_data_smithi177_31116_object20026:head)])", 2024-09-13T11:25:28.068 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:32.427709+0000", 2024-09-13T11:25:28.068 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 115.631370765, 2024-09-13T11:25:28.069 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.46547316300000002, 2024-09-13T11:25:28.069 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:25:28.069 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:25:28.069 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-13T11:25:28.069 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:25:28.069 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:28.069 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:25:28.069 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:32.427709+0000", 2024-09-13T11:25:28.069 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:28.069 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:28.069 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:28.069 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:25:28.069 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:32.427709+0000", 2024-09-13T11:25:28.069 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:28.070 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:28.070 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:28.070 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:25:28.070 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:32.427711+0000", 2024-09-13T11:25:28.070 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5099999999999999e-06 2024-09-13T11:25:28.070 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:28.070 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:28.070 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:25:28.070 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:32.427717+0000", 2024-09-13T11:25:28.070 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.1569999999999997e-06 2024-09-13T11:25:28.070 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:28.070 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:28.070 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:25:28.070 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:32.892390+0000", 2024-09-13T11:25:28.071 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.46467251100000001 2024-09-13T11:25:28.071 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:28.071 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:28.071 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:25:28.071 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:32.892405+0000", 2024-09-13T11:25:28.071 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5296999999999999e-05 2024-09-13T11:25:28.071 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:28.071 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:28.071 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:25:28.071 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:32.892503+0000", 2024-09-13T11:25:28.071 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.7820999999999997e-05 2024-09-13T11:25:28.071 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:28.071 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:28.071 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:25:28.072 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:32.893183+0000", 2024-09-13T11:25:28.072 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00067986700000000001 2024-09-13T11:25:28.072 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:28.072 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:25:28.072 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:28.072 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:28.072 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:28.072 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:57435 3.3 e80/75 3:c0db4777:::benchmark_data_smithi177_31116_object57434:head v 80'3155, mlcod=80'3153)", 2024-09-13T11:25:28.072 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:37.437070+0000", 2024-09-13T11:25:28.072 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 110.622009948, 2024-09-13T11:25:28.072 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.35861845999999997, 2024-09-13T11:25:28.072 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:25:28.072 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:25:28.073 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:28.073 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:25:28.073 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:28.073 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:25:28.073 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:37.437070+0000", 2024-09-13T11:25:28.073 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:28.073 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:28.073 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:28.073 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:25:28.073 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:37.437070+0000", 2024-09-13T11:25:28.073 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:28.073 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:28.073 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:28.073 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:25:28.074 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:37.437072+0000", 2024-09-13T11:25:28.074 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5549999999999999e-06 2024-09-13T11:25:28.074 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:28.074 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:28.074 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:25:28.074 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:37.437141+0000", 2024-09-13T11:25:28.074 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.9494999999999998e-05 2024-09-13T11:25:28.074 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:28.074 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:28.074 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:25:28.074 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:37.793400+0000", 2024-09-13T11:25:28.074 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.35625817999999998 2024-09-13T11:25:28.074 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:28.074 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:28.075 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:25:28.075 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:37.793406+0000", 2024-09-13T11:25:28.075 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.6830000000000002e-06 2024-09-13T11:25:28.075 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:28.075 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:28.075 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:25:28.075 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:37.793472+0000", 2024-09-13T11:25:28.075 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.5671000000000002e-05 2024-09-13T11:25:28.075 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:28.075 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:28.075 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:25:28.075 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:37.793492+0000", 2024-09-13T11:25:28.075 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.0051000000000002e-05 2024-09-13T11:25:28.075 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:28.076 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:28.076 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:25:28.076 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:37.795626+0000", 2024-09-13T11:25:28.076 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0021344990000000002 2024-09-13T11:25:28.076 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:28.076 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:28.076 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:25:28.076 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:37.795689+0000", 2024-09-13T11:25:28.076 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.2325999999999999e-05 2024-09-13T11:25:28.076 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:28.076 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:25:28.076 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:28.076 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:28.076 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:28.077 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:57985 3.3 e80/75 3:c330bb03:::benchmark_data_smithi177_31116_object57984:head v 80'3162, mlcod=80'3159)", 2024-09-13T11:25:28.077 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:38.007076+0000", 2024-09-13T11:25:28.077 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 110.052004596, 2024-09-13T11:25:28.077 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.18211438199999999, 2024-09-13T11:25:28.077 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:25:28.077 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:25:28.077 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:28.077 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:25:28.077 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:28.077 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:25:28.077 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:38.007076+0000", 2024-09-13T11:25:28.077 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:28.077 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:28.077 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:28.078 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:25:28.078 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:38.007076+0000", 2024-09-13T11:25:28.078 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:28.078 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:28.078 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:28.078 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:25:28.078 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:38.007077+0000", 2024-09-13T11:25:28.078 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.6759999999999999e-06 2024-09-13T11:25:28.078 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:28.078 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:28.078 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:25:28.078 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:38.007221+0000", 2024-09-13T11:25:28.078 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00014344299999999999 2024-09-13T11:25:28.078 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:28.079 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:28.079 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:25:28.079 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:38.186808+0000", 2024-09-13T11:25:28.079 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.17958726699999999 2024-09-13T11:25:28.079 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:28.079 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:28.079 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:25:28.079 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:38.186815+0000", 2024-09-13T11:25:28.079 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.0890000000000002e-06 2024-09-13T11:25:28.079 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:28.079 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:28.079 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:25:28.079 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:38.186887+0000", 2024-09-13T11:25:28.079 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.2006000000000001e-05 2024-09-13T11:25:28.080 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:28.080 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:28.080 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:25:28.080 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:38.186913+0000", 2024-09-13T11:25:28.080 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.5962e-05 2024-09-13T11:25:28.080 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:28.080 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:28.080 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:25:28.080 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:38.189123+0000", 2024-09-13T11:25:28.080 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0022099649999999999 2024-09-13T11:25:28.080 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:28.080 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:28.080 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:25:28.080 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:38.189190+0000", 2024-09-13T11:25:28.081 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.6973999999999999e-05 2024-09-13T11:25:28.081 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:28.081 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:25:28.081 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:28.081 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:28.081 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:28.081 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPushReply(3.23 84/78 [PushReplyOp(3:c7e44616:::benchmark_data_smithi177_31116_object32371:head)])", 2024-09-13T11:25:28.081 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:41.338210+0000", 2024-09-13T11:25:28.081 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 106.720869877, 2024-09-13T11:25:28.081 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.41118492299999998, 2024-09-13T11:25:28.081 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:25:28.081 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:25:28.081 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-13T11:25:28.081 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:25:28.081 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:28.082 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:25:28.082 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.338210+0000", 2024-09-13T11:25:28.082 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:28.082 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:28.082 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:28.082 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:25:28.082 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.338210+0000", 2024-09-13T11:25:28.082 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:28.082 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:28.082 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:28.082 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:25:28.082 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.338212+0000", 2024-09-13T11:25:28.082 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.314e-06 2024-09-13T11:25:28.082 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:28.083 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:28.083 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:25:28.083 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.338219+0000", 2024-09-13T11:25:28.083 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.6469999999999998e-06 2024-09-13T11:25:28.083 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:28.083 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:28.083 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:25:28.083 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.748675+0000", 2024-09-13T11:25:28.083 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.41045567500000002 2024-09-13T11:25:28.083 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:28.083 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:28.083 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:25:28.083 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.748684+0000", 2024-09-13T11:25:28.083 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.2150000000000007e-06 2024-09-13T11:25:28.084 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:28.084 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:28.084 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:25:28.084 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.748739+0000", 2024-09-13T11:25:28.084 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.4543000000000003e-05 2024-09-13T11:25:28.084 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:28.084 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:28.084 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:25:28.084 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.749395+0000", 2024-09-13T11:25:28.084 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00065652899999999999 2024-09-13T11:25:28.084 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:28.084 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:25:28.084 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:28.084 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:28.085 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:28.085 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPushReply(3.23 85/78 [PushReplyOp(3:c5c6dce1:::benchmark_data_smithi177_31116_object37293:head)])", 2024-09-13T11:25:28.085 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:42.377862+0000", 2024-09-13T11:25:28.085 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 105.681218418, 2024-09-13T11:25:28.085 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.62538808599999995, 2024-09-13T11:25:28.085 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:25:28.085 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:25:28.085 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-13T11:25:28.085 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:25:28.085 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:28.085 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:25:28.085 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.377862+0000", 2024-09-13T11:25:28.085 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:28.085 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:28.086 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:28.086 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:25:28.086 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.377862+0000", 2024-09-13T11:25:28.086 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:28.086 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:28.086 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:28.086 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:25:28.086 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.377863+0000", 2024-09-13T11:25:28.086 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.6899999999999996e-07 2024-09-13T11:25:28.086 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:28.086 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:28.086 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:25:28.086 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.377866+0000", 2024-09-13T11:25:28.086 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.219e-06 2024-09-13T11:25:28.087 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:28.087 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:28.087 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:25:28.087 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:43.001857+0000", 2024-09-13T11:25:28.087 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.62399084299999996 2024-09-13T11:25:28.087 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:28.087 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:28.087 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:25:28.087 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:43.001869+0000", 2024-09-13T11:25:28.087 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.2415999999999999e-05 2024-09-13T11:25:28.087 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:28.087 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:28.087 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:25:28.087 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:43.002194+0000", 2024-09-13T11:25:28.088 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000324774 2024-09-13T11:25:28.088 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:28.088 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:28.088 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:25:28.088 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:43.003250+0000", 2024-09-13T11:25:28.088 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001055865 2024-09-13T11:25:28.088 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:28.088 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:25:28.088 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:28.088 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:28.088 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:28.088 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPushReply(3.23 86/78 [PushReplyOp(3:c4ca4518:::benchmark_data_smithi177_31116_object39787:head)])", 2024-09-13T11:25:28.088 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:43.580816+0000", 2024-09-13T11:25:28.088 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 104.478264727, 2024-09-13T11:25:28.089 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.14999331199999999, 2024-09-13T11:25:28.089 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:25:28.089 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:25:28.089 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-13T11:25:28.089 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:25:28.089 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:28.089 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:25:28.089 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:43.580816+0000", 2024-09-13T11:25:28.089 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:28.089 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:28.089 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:28.089 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:25:28.089 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:43.580816+0000", 2024-09-13T11:25:28.089 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:28.090 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:28.090 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:28.090 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:25:28.090 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:43.580818+0000", 2024-09-13T11:25:28.090 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.7250000000000002e-06 2024-09-13T11:25:28.090 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:28.090 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:28.090 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:25:28.090 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:43.580830+0000", 2024-09-13T11:25:28.090 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.2247e-05 2024-09-13T11:25:28.090 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:28.090 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:28.090 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:25:28.090 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:43.730204+0000", 2024-09-13T11:25:28.091 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.149373531 2024-09-13T11:25:28.091 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:28.091 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:28.091 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:25:28.091 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:43.730213+0000", 2024-09-13T11:25:28.091 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.6989999999999992e-06 2024-09-13T11:25:28.091 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:28.091 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:28.091 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:25:28.091 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:43.730274+0000", 2024-09-13T11:25:28.091 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.1419000000000003e-05 2024-09-13T11:25:28.091 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:28.091 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:28.091 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:25:28.092 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:43.730809+0000", 2024-09-13T11:25:28.092 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00053469100000000001 2024-09-13T11:25:28.092 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:28.092 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:25:28.092 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:28.092 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:28.092 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:28.092 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPush(3.1c 89/77 [PushOp(3:38bcbc71:::benchmark_data_smithi177_31116_object31448:head, version: 59'1946, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:38bcbc71:::benchmark_data_smithi177_31116_object31448:head@59'1946, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:25:28.092 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:46.670723+0000", 2024-09-13T11:25:28.092 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 101.38835719399999, 2024-09-13T11:25:28.092 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.27576065, 2024-09-13T11:25:28.092 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:25:28.092 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:25:28.092 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:25:28.093 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:25:28.093 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:28.093 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:25:28.093 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.670723+0000", 2024-09-13T11:25:28.093 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:28.093 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:28.093 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:28.093 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:25:28.093 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.670723+0000", 2024-09-13T11:25:28.093 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:28.093 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:28.093 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:28.093 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:25:28.093 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.670724+0000", 2024-09-13T11:25:28.094 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.4389999999999999e-06 2024-09-13T11:25:28.094 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:28.094 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:28.094 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:25:28.094 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.670754+0000", 2024-09-13T11:25:28.094 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.9224999999999999e-05 2024-09-13T11:25:28.094 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:28.094 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:28.094 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:25:28.094 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.944369+0000", 2024-09-13T11:25:28.094 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.27361529800000001 2024-09-13T11:25:28.094 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:28.094 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:28.094 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:25:28.095 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.944422+0000", 2024-09-13T11:25:28.095 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.3318e-05 2024-09-13T11:25:28.095 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:28.095 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:28.095 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:25:28.095 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.944711+0000", 2024-09-13T11:25:28.095 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00028912399999999998 2024-09-13T11:25:28.095 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:28.095 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:28.095 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:25:28.095 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.944735+0000", 2024-09-13T11:25:28.095 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.3366999999999998e-05 2024-09-13T11:25:28.095 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:28.095 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:28.096 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:25:28.096 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.946484+0000", 2024-09-13T11:25:28.096 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0017488790000000001 2024-09-13T11:25:28.096 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:28.096 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:25:28.096 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:28.096 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:28.096 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:28.096 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPush(3.1c 91/77 [PushOp(3:3e18abc2:::benchmark_data_smithi177_31116_object35874:head, version: 62'2235, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:3e18abc2:::benchmark_data_smithi177_31116_object35874:head@62'2235, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:25:28.096 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:48.350968+0000", 2024-09-13T11:25:28.096 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 99.708111880999994, 2024-09-13T11:25:28.096 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.19257698000000001, 2024-09-13T11:25:28.096 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:25:28.096 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:25:28.097 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:25:28.097 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:25:28.097 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:28.097 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:25:28.097 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.350968+0000", 2024-09-13T11:25:28.097 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:28.097 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:28.097 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:28.097 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:25:28.097 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.350968+0000", 2024-09-13T11:25:28.097 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:28.097 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:28.097 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:28.097 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:25:28.098 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.350970+0000", 2024-09-13T11:25:28.098 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.796e-06 2024-09-13T11:25:28.098 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:28.098 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:28.098 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:25:28.098 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.351003+0000", 2024-09-13T11:25:28.098 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.2631e-05 2024-09-13T11:25:28.098 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:28.098 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:28.098 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:25:28.098 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.542140+0000", 2024-09-13T11:25:28.098 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.191136737 2024-09-13T11:25:28.098 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:28.098 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:28.098 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:25:28.099 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.542158+0000", 2024-09-13T11:25:28.099 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.8896999999999999e-05 2024-09-13T11:25:28.099 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:28.099 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:28.099 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:25:28.099 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.542263+0000", 2024-09-13T11:25:28.099 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000104158 2024-09-13T11:25:28.099 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:28.099 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:28.099 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:25:28.099 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.542279+0000", 2024-09-13T11:25:28.099 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5926999999999998e-05 2024-09-13T11:25:28.099 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:28.099 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:28.100 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:25:28.100 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.543545+0000", 2024-09-13T11:25:28.100 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0012668340000000001 2024-09-13T11:25:28.100 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:28.100 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:25:28.100 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:28.100 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:28.100 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:28.100 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:83211 3.1c e106/77 3:3ae31a0f:::benchmark_data_smithi177_31116_object83210:head v 106'4362, mlcod=106'4360)", 2024-09-13T11:25:28.100 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:04.345264+0000", 2024-09-13T11:25:28.100 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 83.713816229000003, 2024-09-13T11:25:28.100 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.17036547399999999, 2024-09-13T11:25:28.100 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:25:28.101 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:25:28.101 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:28.101 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:25:28.101 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:28.101 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:25:28.101 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.345264+0000", 2024-09-13T11:25:28.101 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:28.101 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:28.101 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:28.101 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:25:28.101 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.345264+0000", 2024-09-13T11:25:28.101 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:28.101 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:28.101 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:28.102 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:25:28.102 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.345267+0000", 2024-09-13T11:25:28.102 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.7590000000000002e-06 2024-09-13T11:25:28.102 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:28.102 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:28.102 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:25:28.102 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.345298+0000", 2024-09-13T11:25:28.102 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.1173000000000001e-05 2024-09-13T11:25:28.102 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:28.102 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:28.102 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:25:28.102 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.512265+0000", 2024-09-13T11:25:28.102 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.16696657100000001 2024-09-13T11:25:28.102 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:28.103 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:28.103 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:25:28.103 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.512279+0000", 2024-09-13T11:25:28.103 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.4953999999999999e-05 2024-09-13T11:25:28.103 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:28.103 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:28.103 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:25:28.103 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.512402+0000", 2024-09-13T11:25:28.103 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00012277999999999999 2024-09-13T11:25:28.103 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:28.103 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:28.103 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:25:28.103 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.512440+0000", 2024-09-13T11:25:28.103 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.7861999999999998e-05 2024-09-13T11:25:28.104 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:28.104 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:28.104 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:25:28.104 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.515515+0000", 2024-09-13T11:25:28.104 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00307472 2024-09-13T11:25:28.104 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:28.104 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:28.104 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:25:28.104 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.515629+0000", 2024-09-13T11:25:28.104 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00011465500000000001 2024-09-13T11:25:28.104 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:28.104 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:25:28.104 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:28.104 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:28.105 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:28.105 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:83223 3.1c e106/77 3:3bc6018d:::benchmark_data_smithi177_31116_object83222:head v 106'4363, mlcod=106'4360)", 2024-09-13T11:25:28.105 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:04.512139+0000", 2024-09-13T11:25:28.105 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 83.546941082999993, 2024-09-13T11:25:28.105 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.471923235, 2024-09-13T11:25:28.105 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:25:28.105 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:25:28.105 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:28.105 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:25:28.105 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:28.105 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:25:28.105 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.512139+0000", 2024-09-13T11:25:28.105 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:28.105 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:28.106 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:28.106 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:25:28.106 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.512139+0000", 2024-09-13T11:25:28.106 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:28.106 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:28.106 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:28.106 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:25:28.106 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.512143+0000", 2024-09-13T11:25:28.106 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.427e-06 2024-09-13T11:25:28.106 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:28.106 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:28.106 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:25:28.106 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.512227+0000", 2024-09-13T11:25:28.106 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.3965e-05 2024-09-13T11:25:28.107 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:28.107 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:28.107 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:25:28.107 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.976330+0000", 2024-09-13T11:25:28.107 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.46410342599999999 2024-09-13T11:25:28.107 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:28.107 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:28.107 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:25:28.107 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.976350+0000", 2024-09-13T11:25:28.107 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.9533000000000001e-05 2024-09-13T11:25:28.107 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:28.107 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:28.107 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:25:28.107 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.976488+0000", 2024-09-13T11:25:28.108 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00013799400000000001 2024-09-13T11:25:28.108 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:28.108 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:28.108 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:25:28.108 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.976543+0000", 2024-09-13T11:25:28.108 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.5442000000000002e-05 2024-09-13T11:25:28.108 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:28.108 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:28.108 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:25:28.108 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.983227+0000", 2024-09-13T11:25:28.108 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0066842899999999998 2024-09-13T11:25:28.108 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:28.108 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:28.108 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:25:28.109 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.984062+0000", 2024-09-13T11:25:28.109 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00083515800000000002 2024-09-13T11:25:28.109 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:28.109 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:25:28.109 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:28.109 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:28.109 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:28.109 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:110256 3.13 e119/83 3:cb00d078:::benchmark_data_smithi177_31116_object110255:head v 119'4823, mlcod=119'4821)", 2024-09-13T11:25:28.109 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:25.284530+0000", 2024-09-13T11:25:28.109 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 62.774550623000003, 2024-09-13T11:25:28.109 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.343491771, 2024-09-13T11:25:28.109 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:25:28.109 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:25:28.109 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:28.110 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:25:28.110 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:28.110 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:25:28.110 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.284530+0000", 2024-09-13T11:25:28.110 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:28.110 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:28.110 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:28.110 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:25:28.110 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.284530+0000", 2024-09-13T11:25:28.110 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:28.110 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:28.110 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:28.110 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:25:28.110 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.284532+0000", 2024-09-13T11:25:28.111 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.8870000000000001e-06 2024-09-13T11:25:28.111 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:28.111 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:28.111 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:25:28.111 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.284558+0000", 2024-09-13T11:25:28.111 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.6899000000000001e-05 2024-09-13T11:25:28.111 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:28.111 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:28.111 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:25:28.111 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.626172+0000", 2024-09-13T11:25:28.111 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.34161370800000002 2024-09-13T11:25:28.111 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:28.111 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:28.111 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:25:28.112 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.626187+0000", 2024-09-13T11:25:28.112 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.485e-05 2024-09-13T11:25:28.112 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:28.112 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:28.112 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:25:28.112 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.626245+0000", 2024-09-13T11:25:28.112 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.7998000000000003e-05 2024-09-13T11:25:28.112 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:28.112 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:28.112 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:25:28.112 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.626260+0000", 2024-09-13T11:25:28.112 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5429000000000001e-05 2024-09-13T11:25:28.112 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:28.112 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:28.113 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:25:28.113 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.627924+0000", 2024-09-13T11:25:28.113 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0016634060000000001 2024-09-13T11:25:28.113 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:28.113 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:28.113 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:25:28.113 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.628021+0000", 2024-09-13T11:25:28.113 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.7594000000000001e-05 2024-09-13T11:25:28.113 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:28.113 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:25:28.113 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:28.113 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:28.113 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:28.113 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPush(3.2b 121/119 [PushOp(3:d53a23e8:::benchmark_data_smithi177_31116_object5366:head, version: 45'342, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:d53a23e8:::benchmark_data_smithi177_31116_object5366:head@45'342, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:25:28.114 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:27.268731+0000", 2024-09-13T11:25:28.114 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 60.790349161000002, 2024-09-13T11:25:28.114 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.72228345000000005, 2024-09-13T11:25:28.114 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:25:28.114 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:25:28.114 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:25:28.114 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:25:28.114 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:28.114 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:25:28.114 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:27.268731+0000", 2024-09-13T11:25:28.114 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:28.114 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:28.114 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:28.114 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:25:28.115 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:27.268731+0000", 2024-09-13T11:25:28.115 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:28.115 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:28.115 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:28.115 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:25:28.115 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:27.268733+0000", 2024-09-13T11:25:28.115 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.9740000000000001e-06 2024-09-13T11:25:28.115 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:28.115 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:28.115 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:25:28.115 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:27.268778+0000", 2024-09-13T11:25:28.115 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.4679999999999999e-05 2024-09-13T11:25:28.115 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:28.115 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:28.116 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:25:28.116 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:27.988594+0000", 2024-09-13T11:25:28.116 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.71981667299999996 2024-09-13T11:25:28.116 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:28.116 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:28.116 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:25:28.116 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:27.988652+0000", 2024-09-13T11:25:28.116 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.7640999999999999e-05 2024-09-13T11:25:28.116 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:28.116 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:28.116 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:25:28.116 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:27.988889+0000", 2024-09-13T11:25:28.116 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00023706500000000001 2024-09-13T11:25:28.116 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:28.117 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:28.117 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:25:28.117 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:27.988933+0000", 2024-09-13T11:25:28.117 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.3865000000000001e-05 2024-09-13T11:25:28.117 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:28.117 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:28.117 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:25:28.117 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:27.991015+0000", 2024-09-13T11:25:28.117 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.002081552 2024-09-13T11:25:28.117 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:28.117 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:25:28.117 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:28.117 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:28.117 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:28.118 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPush(3.2b 124/119 [PushOp(3:d64c8aaf:::benchmark_data_smithi177_31116_object31700:head, version: 59'1991, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:d64c8aaf:::benchmark_data_smithi177_31116_object31700:head@59'1991, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:25:28.118 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:31.579304+0000", 2024-09-13T11:25:28.118 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 56.479776459999997, 2024-09-13T11:25:28.118 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.14279340800000001, 2024-09-13T11:25:28.118 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:25:28.118 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:25:28.118 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:25:28.118 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:25:28.118 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:28.118 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:25:28.118 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:31.579304+0000", 2024-09-13T11:25:28.118 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:28.118 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:28.118 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:28.119 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:25:28.119 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:31.579304+0000", 2024-09-13T11:25:28.119 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:28.119 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:28.119 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:28.119 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:25:28.119 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:31.579305+0000", 2024-09-13T11:25:28.119 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5090000000000001e-06 2024-09-13T11:25:28.119 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:28.119 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:28.119 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:25:28.119 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:31.579367+0000", 2024-09-13T11:25:28.119 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.1279999999999996e-05 2024-09-13T11:25:28.119 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:28.120 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:28.120 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:25:28.120 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:31.721234+0000", 2024-09-13T11:25:28.120 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.141867883 2024-09-13T11:25:28.120 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:28.120 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:28.120 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:25:28.120 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:31.721253+0000", 2024-09-13T11:25:28.120 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.8725000000000001e-05 2024-09-13T11:25:28.120 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:28.120 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:28.120 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:25:28.120 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:31.721332+0000", 2024-09-13T11:25:28.120 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.8808999999999997e-05 2024-09-13T11:25:28.120 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:28.121 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:28.121 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:25:28.121 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:31.721345+0000", 2024-09-13T11:25:28.121 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3414999999999999e-05 2024-09-13T11:25:28.121 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:28.121 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:28.121 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:25:28.121 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:31.722097+0000", 2024-09-13T11:25:28.121 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00075178699999999996 2024-09-13T11:25:28.121 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:28.121 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:25:28.121 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:28.121 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:28.121 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:28.122 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:113511 3.2b e124/119 3:d74f5e33:::benchmark_data_smithi177_31116_object113510:head v 124'5156, mlcod=60'2020)", 2024-09-13T11:25:28.122 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:32.032517+0000", 2024-09-13T11:25:28.122 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 56.026563295999999, 2024-09-13T11:25:28.122 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.13874039699999999, 2024-09-13T11:25:28.122 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:25:28.122 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:25:28.122 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:28.122 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:25:28.122 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:28.122 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:25:28.122 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:32.032517+0000", 2024-09-13T11:25:28.122 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:28.122 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:28.122 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:28.123 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:25:28.123 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:32.032517+0000", 2024-09-13T11:25:28.123 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:28.123 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:28.123 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:28.123 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:25:28.123 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:32.032518+0000", 2024-09-13T11:25:28.123 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3060000000000001e-06 2024-09-13T11:25:28.123 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:28.123 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:28.123 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:25:28.123 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:32.032545+0000", 2024-09-13T11:25:28.123 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.6800000000000001e-05 2024-09-13T11:25:28.123 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:28.124 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:28.124 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:25:28.124 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:32.169285+0000", 2024-09-13T11:25:28.124 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.13673950800000001 2024-09-13T11:25:28.124 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:28.124 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:28.124 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:25:28.124 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:32.169291+0000", 2024-09-13T11:25:28.124 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.934e-06 2024-09-13T11:25:28.124 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:28.124 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:28.124 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:25:28.124 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:32.169381+0000", 2024-09-13T11:25:28.124 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.9860999999999999e-05 2024-09-13T11:25:28.125 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:28.125 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:28.125 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:25:28.125 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:32.169406+0000", 2024-09-13T11:25:28.125 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.5052e-05 2024-09-13T11:25:28.125 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:28.125 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:28.125 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:25:28.125 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:32.171218+0000", 2024-09-13T11:25:28.125 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0018111659999999999 2024-09-13T11:25:28.125 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:28.125 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:28.125 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:25:28.125 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:32.171257+0000", 2024-09-13T11:25:28.126 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.977e-05 2024-09-13T11:25:28.126 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:28.126 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:25:28.126 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:28.126 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:28.126 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:28.126 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPush(3.2b 124/119 [PushOp(3:d70bd127:::benchmark_data_smithi177_31116_object32365:head, version: 60'2034, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:d70bd127:::benchmark_data_smithi177_31116_object32365:head@60'2034, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:25:28.126 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:32.037338+0000", 2024-09-13T11:25:28.126 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 56.021741888000001, 2024-09-13T11:25:28.126 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.24358922699999999, 2024-09-13T11:25:28.126 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:25:28.126 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:25:28.126 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:25:28.126 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:25:28.127 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:28.127 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:25:28.127 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:32.037338+0000", 2024-09-13T11:25:28.127 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:28.127 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:28.127 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:28.127 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:25:28.127 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:32.037338+0000", 2024-09-13T11:25:28.127 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:28.127 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:28.127 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:28.127 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:25:28.127 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:32.037340+0000", 2024-09-13T11:25:28.127 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3790000000000001e-06 2024-09-13T11:25:28.128 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:28.128 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:28.128 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:25:28.128 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:32.037411+0000", 2024-09-13T11:25:28.128 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.1254000000000006e-05 2024-09-13T11:25:28.128 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:28.128 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:28.128 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:25:28.128 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:32.277467+0000", 2024-09-13T11:25:28.128 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.24005563199999999 2024-09-13T11:25:28.128 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:28.128 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:28.128 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:25:28.128 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:32.277494+0000", 2024-09-13T11:25:28.129 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.7120000000000001e-05 2024-09-13T11:25:28.129 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:28.129 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:28.129 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:25:28.129 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:32.279860+0000", 2024-09-13T11:25:28.129 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0023667559999999998 2024-09-13T11:25:28.129 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:28.129 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:28.129 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:25:28.129 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:32.279883+0000", 2024-09-13T11:25:28.129 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.2138999999999999e-05 2024-09-13T11:25:28.129 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:28.129 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:28.129 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:25:28.130 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:32.280928+0000", 2024-09-13T11:25:28.130 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0010449470000000001 2024-09-13T11:25:28.130 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:28.130 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:25:28.130 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:28.130 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:28.130 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:28.130 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPush(3.2b 125/119 [PushOp(3:d4c5ede9:::benchmark_data_smithi177_31116_object33277:head, version: 60'2092, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:d4c5ede9:::benchmark_data_smithi177_31116_object33277:head@60'2092, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:25:28.130 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:32.403929+0000", 2024-09-13T11:25:28.130 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 55.655151437999997, 2024-09-13T11:25:28.130 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.194592864, 2024-09-13T11:25:28.130 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:25:28.130 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:25:28.130 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:25:28.131 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:25:28.131 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:28.131 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:25:28.131 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:32.403929+0000", 2024-09-13T11:25:28.131 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:28.131 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:28.131 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:28.131 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:25:28.131 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:32.403929+0000", 2024-09-13T11:25:28.131 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:28.131 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:28.132 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:28.132 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:25:28.132 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:32.403930+0000", 2024-09-13T11:25:28.132 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.6729999999999999e-06 2024-09-13T11:25:28.132 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:28.132 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:28.132 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:25:28.132 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:32.403979+0000", 2024-09-13T11:25:28.132 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.8242000000000003e-05 2024-09-13T11:25:28.132 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:28.132 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:28.132 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:25:28.132 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:32.597445+0000", 2024-09-13T11:25:28.132 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.19346676500000001 2024-09-13T11:25:28.133 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:28.133 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:28.133 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:25:28.133 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:32.597455+0000", 2024-09-13T11:25:28.133 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.2939999999999994e-06 2024-09-13T11:25:28.133 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:28.133 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:28.133 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:25:28.133 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:32.597514+0000", 2024-09-13T11:25:28.133 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.9509000000000002e-05 2024-09-13T11:25:28.133 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:28.133 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:28.133 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:25:28.133 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:32.597522+0000", 2024-09-13T11:25:28.134 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.0350000000000001e-06 2024-09-13T11:25:28.134 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:28.134 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:28.134 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:25:28.134 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:32.598522+0000", 2024-09-13T11:25:28.134 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00099934599999999991 2024-09-13T11:25:28.134 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:28.134 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:25:28.134 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:28.134 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:28.134 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:28.134 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPush(3.2b 128/119 [PushOp(3:d6ff590f:::benchmark_data_smithi177_31116_object64581:head, version: 90'3750, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:d6ff590f:::benchmark_data_smithi177_31116_object64581:head@90'3750, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:25:28.134 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:37.982062+0000", 2024-09-13T11:25:28.134 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 50.077018539000001, 2024-09-13T11:25:28.135 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.28899675200000002, 2024-09-13T11:25:28.135 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:25:28.135 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:25:28.135 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:25:28.135 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:25:28.135 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:28.135 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:25:28.135 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:37.982062+0000", 2024-09-13T11:25:28.135 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:28.135 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:28.135 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:28.135 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:25:28.135 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:37.982062+0000", 2024-09-13T11:25:28.135 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:28.136 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:28.136 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:28.136 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:25:28.136 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:37.982064+0000", 2024-09-13T11:25:28.136 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.2259999999999999e-06 2024-09-13T11:25:28.136 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:28.136 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:28.136 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:25:28.136 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:37.982114+0000", 2024-09-13T11:25:28.136 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.0330000000000001e-05 2024-09-13T11:25:28.136 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:28.136 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:28.136 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:25:28.136 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:38.269983+0000", 2024-09-13T11:25:28.137 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.28786869999999998 2024-09-13T11:25:28.137 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:28.137 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:28.137 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:25:28.137 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:38.270000+0000", 2024-09-13T11:25:28.137 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.6977999999999999e-05 2024-09-13T11:25:28.137 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:28.137 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:28.137 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:25:28.137 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:38.270092+0000", 2024-09-13T11:25:28.137 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.221e-05 2024-09-13T11:25:28.137 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:28.137 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:28.137 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:25:28.138 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:38.270103+0000", 2024-09-13T11:25:28.138 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.1253999999999999e-05 2024-09-13T11:25:28.138 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:28.138 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:28.138 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:25:28.138 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:38.271058+0000", 2024-09-13T11:25:28.138 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00095505399999999999 2024-09-13T11:25:28.138 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:28.138 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:25:28.138 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:28.138 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:28.138 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:28.138 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:150366 3.2c e149/144 3:348937c7:::benchmark_data_smithi177_31116_object10063:head v 149'5822, mlcod=149'5819)", 2024-09-13T11:25:28.138 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:25:06.284607+0000", 2024-09-13T11:25:28.139 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 21.774473249, 2024-09-13T11:25:28.139 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.16600746599999999, 2024-09-13T11:25:28.139 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:25:28.139 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:25:28.139 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:28.139 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:25:28.139 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:28.139 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:25:28.139 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:06.284607+0000", 2024-09-13T11:25:28.139 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:28.139 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:28.139 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:28.139 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:25:28.139 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:06.284607+0000", 2024-09-13T11:25:28.140 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:28.140 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:28.140 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:28.140 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:25:28.140 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:06.284608+0000", 2024-09-13T11:25:28.140 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.018e-06 2024-09-13T11:25:28.140 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:28.140 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:28.140 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:25:28.140 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:06.284612+0000", 2024-09-13T11:25:28.140 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.3239999999999996e-06 2024-09-13T11:25:28.140 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:28.140 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:28.140 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:25:28.141 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:06.449402+0000", 2024-09-13T11:25:28.141 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.16478921099999999 2024-09-13T11:25:28.141 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:28.141 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:28.141 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:25:28.141 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:06.449408+0000", 2024-09-13T11:25:28.141 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.1990000000000003e-06 2024-09-13T11:25:28.141 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:28.141 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:28.141 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:25:28.141 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:06.449467+0000", 2024-09-13T11:25:28.141 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.8792999999999998e-05 2024-09-13T11:25:28.141 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:28.141 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:28.141 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:25:28.142 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:06.449490+0000", 2024-09-13T11:25:28.142 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.3895000000000001e-05 2024-09-13T11:25:28.142 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:28.142 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:28.142 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:25:28.142 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:06.450581+0000", 2024-09-13T11:25:28.142 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001090653 2024-09-13T11:25:28.142 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:28.142 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:28.142 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:25:28.142 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:06.450614+0000", 2024-09-13T11:25:28.142 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.3373e-05 2024-09-13T11:25:28.142 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:28.142 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:25:28.143 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:28.143 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:28.143 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:25:28.143 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-09-13T11:25:28.144 INFO:teuthology.orchestra.run.smithi001.stdout:osd.3: {} 2024-09-13T11:25:28.144 INFO:teuthology.orchestra.run.smithi001.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T11:25:28.145 INFO:teuthology.orchestra.run.smithi001.stdout:osd.4: {} 2024-09-13T11:25:28.145 INFO:teuthology.orchestra.run.smithi001.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T11:25:28.145 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-09-13T11:25:28.146 INFO:teuthology.orchestra.run.smithi001.stdout:osd.5: {} 2024-09-13T11:25:28.146 INFO:teuthology.orchestra.run.smithi001.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T11:25:28.146 INFO:tasks.ceph.osd.8.smithi001.stderr:2024-09-13T11:25:28.143+0000 7fb9b48ae640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26312) UID: 0 2024-09-13T11:25:28.159 INFO:teuthology.orchestra.run.smithi001.stdout:osd.6: {} 2024-09-13T11:25:28.159 INFO:teuthology.orchestra.run.smithi001.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T11:25:28.187 INFO:teuthology.orchestra.run.smithi001.stdout:osd.7: {} 2024-09-13T11:25:28.188 INFO:teuthology.orchestra.run.smithi001.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T11:25:28.215 INFO:teuthology.orchestra.run.smithi001.stdout:osd.8: {} 2024-09-13T11:25:28.216 INFO:teuthology.orchestra.run.smithi001.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-13T11:25:28.245 INFO:teuthology.orchestra.run.smithi001.stdout:osd.9: {} 2024-09-13T11:25:28.245 INFO:teuthology.orchestra.run.smithi001.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-13T11:25:28.246 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-09-13T11:25:28.244+0000 7f97befc9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26315) UID: 0 2024-09-13T11:25:28.251 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-13T11:25:28.281 INFO:teuthology.orchestra.run.smithi001.stdout:osd.10: {} 2024-09-13T11:25:28.281 INFO:teuthology.orchestra.run.smithi001.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-13T11:25:28.298 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-09-13T11:25:28.299 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [ 2024-09-13T11:25:28.299 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:28.299 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:178648 3.16 3:6cd11cef:::benchmark_data_smithi177_31116_object38345:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e168)", 2024-09-13T11:25:28.299 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:25:28.283354+0000", 2024-09-13T11:25:28.299 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 0.015410963999999999, 2024-09-13T11:25:28.299 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.015429755, 2024-09-13T11:25:28.300 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:25:28.300 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:25:28.301 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "waiting for sub ops", 2024-09-13T11:25:28.301 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:25:28.301 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:25:28.301 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:28.301 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 178648 2024-09-13T11:25:28.302 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:28.302 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:25:28.302 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:28.302 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:25:28.302 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:28.283354+0000", 2024-09-13T11:25:28.302 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:28.303 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:28.303 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:28.303 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:25:28.303 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:28.283354+0000", 2024-09-13T11:25:28.303 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:28.303 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:28.303 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:28.303 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:25:28.303 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:28.283356+0000", 2024-09-13T11:25:28.303 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.9010000000000001e-06 2024-09-13T11:25:28.303 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:28.303 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:28.303 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:25:28.304 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:28.283360+0000", 2024-09-13T11:25:28.304 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.6200000000000001e-06 2024-09-13T11:25:28.304 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:28.304 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:28.304 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:25:28.304 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:28.283373+0000", 2024-09-13T11:25:28.304 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.2947e-05 2024-09-13T11:25:28.304 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:28.304 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:28.304 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:25:28.304 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:28.283378+0000", 2024-09-13T11:25:28.305 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.8770000000000001e-06 2024-09-13T11:25:28.305 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:28.305 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:28.305 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:25:28.305 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:28.283419+0000", 2024-09-13T11:25:28.305 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.1489000000000002e-05 2024-09-13T11:25:28.305 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:28.305 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:28.305 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:25:28.305 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:28.283515+0000", 2024-09-13T11:25:28.305 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.5917999999999999e-05 2024-09-13T11:25:28.305 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:28.305 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:28.306 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 15", 2024-09-13T11:25:28.306 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:28.283582+0000", 2024-09-13T11:25:28.306 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.7291000000000003e-05 2024-09-13T11:25:28.306 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:28.306 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:28.306 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:25:28.306 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:28.289219+0000", 2024-09-13T11:25:28.306 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0056370789999999997 2024-09-13T11:25:28.306 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:28.306 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:25:28.306 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:28.306 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:28.306 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:28.307 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:178666 3.13 3:cfdf51cc:::benchmark_data_smithi177_31116_object38363:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e168)", 2024-09-13T11:25:28.307 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:25:28.297097+0000", 2024-09-13T11:25:28.307 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 0.0016677930000000001, 2024-09-13T11:25:28.307 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0017343079999999999, 2024-09-13T11:25:28.307 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:25:28.307 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:25:28.307 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "waiting for sub ops", 2024-09-13T11:25:28.307 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:25:28.307 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:25:28.307 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:28.307 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 178666 2024-09-13T11:25:28.307 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:28.307 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:25:28.308 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:28.308 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:25:28.308 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:28.297097+0000", 2024-09-13T11:25:28.308 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:28.308 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:28.308 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:28.308 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:25:28.308 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:28.297097+0000", 2024-09-13T11:25:28.308 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:28.308 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:28.308 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:28.308 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:25:28.308 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:28.297100+0000", 2024-09-13T11:25:28.309 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.057e-06 2024-09-13T11:25:28.309 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:28.309 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:28.309 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:25:28.309 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:28.297104+0000", 2024-09-13T11:25:28.309 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.853e-06 2024-09-13T11:25:28.309 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:28.310 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:28.310 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:25:28.310 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:28.297115+0000", 2024-09-13T11:25:28.310 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0538e-05 2024-09-13T11:25:28.310 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:28.310 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:28.310 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:25:28.310 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:28.297120+0000", 2024-09-13T11:25:28.310 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.3730000000000001e-06 2024-09-13T11:25:28.310 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:28.310 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:28.310 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:25:28.310 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:28.297178+0000", 2024-09-13T11:25:28.311 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.7850999999999999e-05 2024-09-13T11:25:28.311 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:28.311 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:28.311 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:25:28.311 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:28.297299+0000", 2024-09-13T11:25:28.311 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000120748 2024-09-13T11:25:28.311 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:28.311 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:28.311 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 7", 2024-09-13T11:25:28.311 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:28.297397+0000", 2024-09-13T11:25:28.311 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.8121999999999993e-05 2024-09-13T11:25:28.311 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:28.311 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:25:28.311 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:28.312 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:28.312 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:28.312 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPush(3.27 168/120 [PushOp(3:e628cd75:::benchmark_data_smithi177_31116_object105502:head, version: 116'4630, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:e628cd75:::benchmark_data_smithi177_31116_object105502:head@116'4630, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:25:28.312 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:25:28.297534+0000", 2024-09-13T11:25:28.312 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 0.0012308340000000001, 2024-09-13T11:25:28.312 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001351152, 2024-09-13T11:25:28.312 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:25:28.312 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:25:28.312 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:25:28.312 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:25:28.312 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:28.312 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:25:28.312 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:28.297534+0000", 2024-09-13T11:25:28.313 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:28.313 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:28.313 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:28.313 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:25:28.313 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:28.297534+0000", 2024-09-13T11:25:28.313 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:28.313 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:28.313 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:28.313 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:25:28.313 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:28.297536+0000", 2024-09-13T11:25:28.313 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.2470000000000001e-06 2024-09-13T11:25:28.314 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:28.314 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:28.314 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:25:28.314 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:28.297597+0000", 2024-09-13T11:25:28.314 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.1260999999999995e-05 2024-09-13T11:25:28.314 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:28.314 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:28.314 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:25:28.314 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:28.297634+0000", 2024-09-13T11:25:28.314 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.7333999999999999e-05 2024-09-13T11:25:28.314 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:28.314 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:28.314 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:25:28.314 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:28.297646+0000", 2024-09-13T11:25:28.315 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.1559999999999999e-05 2024-09-13T11:25:28.315 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:28.315 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:28.315 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:25:28.315 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:28.297727+0000", 2024-09-13T11:25:28.315 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.0819e-05 2024-09-13T11:25:28.315 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:28.315 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:28.315 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:25:28.315 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:28.297740+0000", 2024-09-13T11:25:28.315 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3784999999999999e-05 2024-09-13T11:25:28.315 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:28.315 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:25:28.316 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:28.316 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:28.316 INFO:teuthology.orchestra.run.smithi177.stdout: ], 2024-09-13T11:25:28.316 INFO:teuthology.orchestra.run.smithi177.stdout: "num_ops": 3 2024-09-13T11:25:28.316 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-09-13T11:25:28.317 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-09-13T11:25:28.318 INFO:teuthology.orchestra.run.smithi001.stdout:osd.11: {} 2024-09-13T11:25:28.318 INFO:teuthology.orchestra.run.smithi001.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-13T11:25:28.346 INFO:tasks.ceph.osd.12.smithi001.stderr:2024-09-13T11:25:28.343+0000 7f892c6e6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26314) UID: 0 2024-09-13T11:25:28.348 INFO:teuthology.orchestra.run.smithi001.stdout:osd.12: {} 2024-09-13T11:25:28.348 INFO:teuthology.orchestra.run.smithi001.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-13T11:25:28.374 INFO:teuthology.orchestra.run.smithi001.stdout:osd.13: {} 2024-09-13T11:25:28.375 INFO:teuthology.orchestra.run.smithi001.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-13T11:25:28.407 INFO:teuthology.orchestra.run.smithi001.stdout:osd.14: {} 2024-09-13T11:25:28.408 INFO:teuthology.orchestra.run.smithi001.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-13T11:25:28.437 INFO:teuthology.orchestra.run.smithi001.stdout:osd.15: {} 2024-09-13T11:25:28.437 INFO:teuthology.orchestra.run.smithi001.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-13T11:25:28.447 INFO:tasks.ceph.osd.11.smithi177.stderr:2024-09-13T11:25:28.442+0000 7f24792c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26580) UID: 0 2024-09-13T11:25:28.461 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-09-13T11:25:28.461 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:25:28.481 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-09-13T11:25:28.547 INFO:tasks.ceph.osd.14.smithi162.stderr:2024-09-13T11:25:28.544+0000 7f3df5903640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33154) UID: 0 2024-09-13T11:25:28.648 INFO:tasks.ceph.osd.11.smithi177.stderr:2024-09-13T11:25:28.646+0000 7f24792c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26580) UID: 0 2024-09-13T11:25:28.667 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-09-13T11:25:28.667 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:25:28.686 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_ops_in_flight 2024-09-13T11:25:28.749 INFO:tasks.ceph.osd.5.smithi092.stderr:2024-09-13T11:25:28.745+0000 7f6f0b705640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26346) UID: 0 2024-09-13T11:25:28.850 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-13T11:25:28.847+0000 7f6a68775640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26307) UID: 0 2024-09-13T11:25:28.859 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-09-13T11:25:28.860 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:25:28.875 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_blocked_ops 2024-09-13T11:25:28.951 INFO:tasks.ceph.osd.1.smithi092.stderr:2024-09-13T11:25:28.945+0000 7fe7789c5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33718) UID: 0 2024-09-13T11:25:29.005 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-09-13T11:25:29.005 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:25:29.020 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_historic_ops 2024-09-13T11:25:29.051 INFO:tasks.ceph.osd.12.smithi001.stderr:2024-09-13T11:25:29.047+0000 7f892c6e6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26314) UID: 0 2024-09-13T11:25:29.149 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-09-13T11:25:29.149 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:25:29.152 INFO:tasks.ceph.osd.1.smithi092.stderr:2024-09-13T11:25:29.145+0000 7fe7789c5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33718) UID: 0 2024-09-13T11:25:29.163 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-13T11:25:29.240 INFO:teuthology.orchestra.run.smithi001.stderr:nodeep-scrub is set 2024-09-13T11:25:29.246 INFO:tasks.ceph.osd.7.smithi177.stderr:2024-09-13T11:25:29.242+0000 7f896ff4a640 -1 osd.7 168 handle_osd_map injecting map crc failure 2024-09-13T11:25:29.249 INFO:tasks.ceph.osd.4.smithi001.stderr:2024-09-13T11:25:29.247+0000 7ff8d2c41640 -1 osd.4 168 handle_osd_map injecting map crc failure 2024-09-13T11:25:29.253 INFO:tasks.ceph.osd.15.smithi177.stderr:2024-09-13T11:25:29.250+0000 7f9d6ce77640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26578) UID: 0 2024-09-13T11:25:29.268 INFO:tasks.ceph.osd.10.smithi162.stderr:2024-09-13T11:25:29.264+0000 7fb593421640 -1 osd.10 168 handle_osd_map injecting map crc failure 2024-09-13T11:25:29.307 INFO:teuthology.orchestra.run.smithi092.stdout:ERROR: (22) Invalid argument 2024-09-13T11:25:29.307 INFO:teuthology.orchestra.run.smithi092.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:25:29.323 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-13T11:25:29.354 INFO:tasks.ceph.osd.14.smithi162.stderr:2024-09-13T11:25:29.352+0000 7f3df5903640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33154) UID: 0 2024-09-13T11:25:29.444 INFO:teuthology.orchestra.run.smithi092.stdout:ERROR: (22) Invalid argument 2024-09-13T11:25:29.444 INFO:teuthology.orchestra.run.smithi092.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:25:29.455 INFO:tasks.ceph.osd.14.smithi162.stderr:2024-09-13T11:25:29.452+0000 7f3df5903640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33154) UID: 0 2024-09-13T11:25:29.458 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-13T11:25:29.557 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-09-13T11:25:29.552+0000 7f97befc9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26315) UID: 0 2024-09-13T11:25:29.591 INFO:teuthology.orchestra.run.smithi092.stdout:ERROR: (22) Invalid argument 2024-09-13T11:25:29.591 INFO:teuthology.orchestra.run.smithi092.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:25:29.606 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_ops_in_flight 2024-09-13T11:25:29.658 INFO:tasks.ceph.osd.11.smithi177.stderr:2024-09-13T11:25:29.654+0000 7f24792c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26580) UID: 0 2024-09-13T11:25:29.740 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-13T11:25:29.741 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:25:29.756 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_blocked_ops 2024-09-13T11:25:29.758 INFO:tasks.ceph.osd.4.smithi001.stderr:2024-09-13T11:25:29.755+0000 7ff8e0218640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26310) UID: 0 2024-09-13T11:25:29.860 INFO:tasks.ceph.osd.9.smithi092.stderr:2024-09-13T11:25:29.853+0000 7ff5ab3d0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26348) UID: 0 2024-09-13T11:25:29.887 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-13T11:25:29.887 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:25:29.902 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_historic_ops 2024-09-13T11:25:29.961 INFO:tasks.ceph.osd.3.smithi177.stderr:2024-09-13T11:25:29.958+0000 7f10adba4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26574) UID: 0 2024-09-13T11:25:30.034 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-13T11:25:30.034 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:25:30.050 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-13T11:25:30.062 INFO:tasks.ceph.osd.1.smithi092.stderr:2024-09-13T11:25:30.057+0000 7fe7789c5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33718) UID: 0 2024-09-13T11:25:30.162 INFO:tasks.ceph.osd.8.smithi001.stderr:2024-09-13T11:25:30.159+0000 7fb9b48ae640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26312) UID: 0 2024-09-13T11:25:30.204 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-13T11:25:30.204 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:25:30.220 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-13T11:25:30.264 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-13T11:25:30.263+0000 7f6a68775640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26307) UID: 0 2024-09-13T11:25:30.290 INFO:tasks.ceph.osd.8.smithi001.stderr:2024-09-13T11:25:30.287+0000 7fb9ab2df640 -1 osd.8 169 handle_osd_map injecting map crc failure 2024-09-13T11:25:30.310 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-09-13T11:25:30.308+0000 7f97b59fa640 -1 osd.6 169 handle_osd_map injecting map crc failure 2024-09-13T11:25:30.364 INFO:tasks.ceph.osd.14.smithi162.stderr:2024-09-13T11:25:30.360+0000 7f3df5903640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33154) UID: 0 2024-09-13T11:25:30.381 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-13T11:25:30.401 INFO:tasks.ceph.osd.14.smithi162.stderr:2024-09-13T11:25:30.400+0000 7f3dea330640 -1 osd.14 168 handle_osd_map injecting map crc failure 2024-09-13T11:25:30.421 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-13T11:25:30.421 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:25:30.444 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-13T11:25:30.467 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T11:25:30.491 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-13T11:25:30.487+0000 7f6a68775640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26307) UID: 0 2024-09-13T11:25:30.566 INFO:tasks.ceph.osd.10.smithi162.stderr:2024-09-13T11:25:30.564+0000 7fb59c9f0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26317) UID: 0 2024-09-13T11:25:30.667 INFO:tasks.ceph.osd.7.smithi177.stderr:2024-09-13T11:25:30.666+0000 7f897b51d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26575) UID: 0 2024-09-13T11:25:30.714 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-13T11:25:30.714 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:25:30.738 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-13T11:25:30.755 INFO:teuthology.orchestra.run.smithi001.stdout:osd.0: {} 2024-09-13T11:25:30.755 INFO:teuthology.orchestra.run.smithi001.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T11:25:30.768 INFO:tasks.ceph.osd.8.smithi001.stderr:2024-09-13T11:25:30.767+0000 7fb9b48ae640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26312) UID: 0 2024-09-13T11:25:30.789 INFO:teuthology.orchestra.run.smithi001.stdout:osd.1: {} 2024-09-13T11:25:30.789 INFO:teuthology.orchestra.run.smithi001.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T11:25:30.825 INFO:teuthology.orchestra.run.smithi001.stdout:osd.2: {} 2024-09-13T11:25:30.825 INFO:teuthology.orchestra.run.smithi001.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T11:25:30.854 INFO:teuthology.orchestra.run.smithi001.stdout:osd.3: {} 2024-09-13T11:25:30.855 INFO:teuthology.orchestra.run.smithi001.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T11:25:30.870 INFO:tasks.ceph.osd.15.smithi177.stderr:2024-09-13T11:25:30.866+0000 7f9d6ce77640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26578) UID: 0 2024-09-13T11:25:30.902 INFO:teuthology.orchestra.run.smithi001.stdout:osd.4: {} 2024-09-13T11:25:30.903 INFO:teuthology.orchestra.run.smithi001.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T11:25:30.951 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-13T11:25:30.951 INFO:teuthology.orchestra.run.smithi001.stdout: "ops": [ 2024-09-13T11:25:30.952 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:30.952 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:181349 3.1a 3:5c715662:::benchmark_data_smithi177_31116_object41046:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e170)", 2024-09-13T11:25:30.952 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:25:30.948213+0000", 2024-09-13T11:25:30.952 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 0.0029002350000000001, 2024-09-13T11:25:30.952 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0029280140000000001, 2024-09-13T11:25:30.952 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:30.952 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:30.952 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "waiting for sub ops", 2024-09-13T11:25:30.952 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:25:30.952 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:25:30.952 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:30.952 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 181349 2024-09-13T11:25:30.953 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:30.953 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:30.953 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:30.953 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:30.953 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:30.948213+0000", 2024-09-13T11:25:30.953 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:30.953 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:30.953 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:30.953 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:30.953 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:30.948213+0000", 2024-09-13T11:25:30.953 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:30.953 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:30.953 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:30.954 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:30.954 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:30.948219+0000", 2024-09-13T11:25:30.954 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.9139999999999996e-06 2024-09-13T11:25:30.954 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:30.954 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:30.954 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:30.954 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:30.948227+0000", 2024-09-13T11:25:30.954 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.8730000000000007e-06 2024-09-13T11:25:30.954 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:30.954 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:30.954 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:30.954 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:30.948248+0000", 2024-09-13T11:25:30.954 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.1033e-05 2024-09-13T11:25:30.955 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:30.955 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:30.955 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:30.955 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:30.948258+0000", 2024-09-13T11:25:30.955 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.5789999999999993e-06 2024-09-13T11:25:30.955 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:30.955 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:30.955 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:30.955 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:30.948333+0000", 2024-09-13T11:25:30.955 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.5087999999999997e-05 2024-09-13T11:25:30.955 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:30.955 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:30.955 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:25:30.956 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:30.948460+0000", 2024-09-13T11:25:30.956 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000126919 2024-09-13T11:25:30.956 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:30.956 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:30.956 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 9", 2024-09-13T11:25:30.956 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:30.948570+0000", 2024-09-13T11:25:30.956 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000109922 2024-09-13T11:25:30.956 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:30.956 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:30.956 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:30.956 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:30.956 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:30.956 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:181350 3.1a 3.7e0e0bba (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e170)", 2024-09-13T11:25:30.957 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:25:30.948282+0000", 2024-09-13T11:25:30.957 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 0.0028315559999999998, 2024-09-13T11:25:30.957 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0029290739999999998, 2024-09-13T11:25:30.957 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:30.957 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:30.957 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "queued for pg", 2024-09-13T11:25:30.957 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:25:30.957 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:25:30.957 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:30.957 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 181350 2024-09-13T11:25:30.957 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:30.957 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:30.957 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:30.958 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:30.958 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:30.948282+0000", 2024-09-13T11:25:30.958 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:30.958 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:30.958 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:30.958 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:30.958 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:30.948282+0000", 2024-09-13T11:25:30.958 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:30.958 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:30.958 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:30.958 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:30.958 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:30.948283+0000", 2024-09-13T11:25:30.958 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.0599999999999999e-07 2024-09-13T11:25:30.958 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:30.959 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:30.959 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:30.959 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:30.948286+0000", 2024-09-13T11:25:30.959 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.3100000000000001e-06 2024-09-13T11:25:30.959 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:30.959 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:30.959 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:30.959 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:30.948292+0000", 2024-09-13T11:25:30.959 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.0979999999999999e-06 2024-09-13T11:25:30.959 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:30.959 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:30.959 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:30.959 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:30.948297+0000", 2024-09-13T11:25:30.959 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.1599999999999997e-06 2024-09-13T11:25:30.960 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:30.960 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:30.960 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:30.960 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:30.951214+0000", 2024-09-13T11:25:30.960 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0029161719999999999 2024-09-13T11:25:30.960 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:30.960 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:30.960 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:30.960 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:30.960 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-09-13T11:25:30.960 INFO:teuthology.orchestra.run.smithi001.stdout: "num_ops": 2 2024-09-13T11:25:30.960 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-13T11:25:30.961 INFO:teuthology.orchestra.run.smithi001.stdout:osd.5: {} 2024-09-13T11:25:30.961 INFO:teuthology.orchestra.run.smithi001.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T11:25:30.971 INFO:tasks.ceph.osd.12.smithi001.stderr:2024-09-13T11:25:30.967+0000 7f892c6e6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26314) UID: 0 2024-09-13T11:25:30.977 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-13T11:25:30.982 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-13T11:25:30.985 INFO:teuthology.orchestra.run.smithi001.stderr:dumped all 2024-09-13T11:25:30.989 INFO:teuthology.orchestra.run.smithi001.stdout:osd.6: {} 2024-09-13T11:25:30.990 INFO:teuthology.orchestra.run.smithi001.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T11:25:31.014 INFO:teuthology.orchestra.run.smithi001.stdout:{"pg_ready":true,"pg_map":{"version":256,"stamp":"2024-09-13T11:25:30.247882+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":6824821323,"num_objects":104135,"num_object_clones":0,"num_object_copies":208270,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2396,"num_objects_unfound":0,"num_objects_dirty":104135,"num_whiteouts":0,"num_read":87,"num_read_kb":74,"num_write":92712,"num_write_kb":3637107,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21018,"num_bytes_recovered":1377304595,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":286064,"ondisk_log_size":286064,"up":110,"acting":110,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":122,"num_osds":15,"num_per_pool_osds":15,"num_per_pool_omap_osds":15,"kb":1415577600,"kb_used":15443464,"kb_used_data":14102872,"kb_used_omap":26,"kb_used_meta":1340133,"kb_avail":1400134136,"statfs":{"total":1449551462400,"available":1433737355264,"internally_reserved":0,"allocated":14441340928,"data_stored":14430460982,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":26763,"internal_metadata":1372297077},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,2,0,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-203161600,"num_objects":-3100,"num_object_clones":0,"num_object_copies":-6200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-742,"num_objects_unfound":0,"num_objects_dirty":-3100,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3100,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":371,"num_bytes_recovered":24313856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.085889"},"pg_stats":[{"pgid":"3.2d","version":"169'6213","reported_seq":10306,"reported_epoch":169,"state":"active+recovering+remapped","last_fresh":"2024-09-13T11:25:30.100854+0000","last_change":"2024-09-13T11:25:19.656321+0000","last_active":"2024-09-13T11:25:30.100854+0000","last_peered":"2024-09-13T11:25:30.100854+0000","last_clean":"2024-09-13T11:24:24.974950+0000","last_became_active":"2024-09-13T11:24:27.227288+0000","last_became_peered":"2024-09-13T11:24:27.227288+0000","last_unstale":"2024-09-13T11:25:30.100854+0000","last_undegraded":"2024-09-13T11:25:30.100854+0000","last_fullsized":"2024-09-13T11:25:30.100854+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":6,"last_scrub":"76'3395","last_scrub_stamp":"2024-09-13T11:23:32.776747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:32.776747+0000","objects_scrubbed":1704,"log_size":6213,"log_dups_size":0,"ondisk_log_size":6213,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":179,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":106266624,"num_objects":1622,"num_object_clones":0,"num_object_copies":3244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":932,"num_objects_unfound":0,"num_objects_dirty":1622,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1976,"num_write_kb":87840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1253,"num_bytes_recovered":82116608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[3,0],"avail_no_missing":["3","7"],"object_location_counts":[{"shards":"0,3","objects":1156},{"shards":"3,7","objects":466}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2c","version":"166'6222","reported_seq":11554,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-13T11:25:25.967987+0000","last_change":"2024-09-13T11:24:59.456762+0000","last_active":"2024-09-13T11:25:25.967987+0000","last_peered":"2024-09-13T11:25:25.967987+0000","last_clean":"2024-09-13T11:25:25.967987+0000","last_became_active":"2024-09-13T11:24:59.021895+0000","last_became_peered":"2024-09-13T11:24:59.021895+0000","last_unstale":"2024-09-13T11:25:25.967987+0000","last_undegraded":"2024-09-13T11:25:25.967987+0000","last_fullsized":"2024-09-13T11:25:25.967987+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":6,"last_scrub":"145'5672","last_scrub_stamp":"2024-09-13T11:24:59.456653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:24:59.456653+0000","objects_scrubbed":2192,"log_size":6222,"log_dups_size":0,"ondisk_log_size":6222,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:26:20.877991+0000","scrub_duration":279,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":107610112,"num_objects":1642,"num_object_clones":0,"num_object_copies":3284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1642,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":550,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,7],"acting":[10,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.2b","version":"166'6105","reported_seq":9794,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-13T11:25:25.244306+0000","last_change":"2024-09-13T11:24:45.271477+0000","last_active":"2024-09-13T11:25:25.244306+0000","last_peered":"2024-09-13T11:25:25.244306+0000","last_clean":"2024-09-13T11:25:25.244306+0000","last_became_active":"2024-09-13T11:24:26.180578+0000","last_became_peered":"2024-09-13T11:24:26.180578+0000","last_unstale":"2024-09-13T11:25:25.244306+0000","last_undegraded":"2024-09-13T11:25:25.244306+0000","last_fullsized":"2024-09-13T11:25:25.244306+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":6,"last_scrub":"131'5382","last_scrub_stamp":"2024-09-13T11:24:45.271376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:24:45.271376+0000","objects_scrubbed":1940,"log_size":6105,"log_dups_size":0,"ondisk_log_size":6105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:59.210895+0000","scrub_duration":279,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":101515264,"num_objects":1549,"num_object_clones":0,"num_object_copies":3098,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1549,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":723,"num_write_kb":10624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2a","version":"166'6221","reported_seq":10097,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-13T11:25:26.035064+0000","last_change":"2024-09-13T11:24:46.625117+0000","last_active":"2024-09-13T11:25:26.035064+0000","last_peered":"2024-09-13T11:25:26.035064+0000","last_clean":"2024-09-13T11:25:26.035064+0000","last_became_active":"2024-09-13T11:24:45.320540+0000","last_became_peered":"2024-09-13T11:24:45.320540+0000","last_unstale":"2024-09-13T11:25:26.035064+0000","last_undegraded":"2024-09-13T11:25:26.035064+0000","last_fullsized":"2024-09-13T11:25:26.035064+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":6,"last_scrub":"133'5491","last_scrub_stamp":"2024-09-13T11:24:46.624970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:24:46.624970+0000","objects_scrubbed":2094,"log_size":6221,"log_dups_size":0,"ondisk_log_size":6221,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:26:04.813280+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112197632,"num_objects":1712,"num_object_clones":0,"num_object_copies":3424,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1712,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":732,"num_write_kb":11200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,7],"acting":[10,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.29","version":"166'5881","reported_seq":11572,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-13T11:25:25.960499+0000","last_change":"2024-09-13T11:25:13.333062+0000","last_active":"2024-09-13T11:25:25.960499+0000","last_peered":"2024-09-13T11:25:25.960499+0000","last_clean":"2024-09-13T11:25:25.960499+0000","last_became_active":"2024-09-13T11:25:12.944028+0000","last_became_peered":"2024-09-13T11:25:12.944028+0000","last_unstale":"2024-09-13T11:25:25.960499+0000","last_undegraded":"2024-09-13T11:25:25.960499+0000","last_fullsized":"2024-09-13T11:25:25.960499+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":6,"last_scrub":"154'5579","last_scrub_stamp":"2024-09-13T11:25:13.332959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:13.332959+0000","objects_scrubbed":1937,"log_size":5881,"log_dups_size":0,"ondisk_log_size":5881,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:26:24.089228+0000","scrub_duration":243,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":107020288,"num_objects":1633,"num_object_clones":0,"num_object_copies":3266,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1633,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":304,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.28","version":"166'5729","reported_seq":9507,"reported_epoch":166,"state":"active+recovering+remapped","last_fresh":"2024-09-13T11:25:25.386902+0000","last_change":"2024-09-13T11:25:11.182075+0000","last_active":"2024-09-13T11:25:25.386902+0000","last_peered":"2024-09-13T11:25:25.386902+0000","last_clean":"2024-09-13T11:24:25.005266+0000","last_became_active":"2024-09-13T11:24:27.218508+0000","last_became_peered":"2024-09-13T11:24:27.218508+0000","last_unstale":"2024-09-13T11:25:25.386902+0000","last_undegraded":"2024-09-13T11:25:25.386902+0000","last_fullsized":"2024-09-13T11:25:25.386902+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":6,"last_scrub":"68'2771","last_scrub_stamp":"2024-09-13T11:23:20.644747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:20.644747+0000","objects_scrubbed":1356,"log_size":5729,"log_dups_size":0,"ondisk_log_size":5729,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":411,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":107872256,"num_objects":1646,"num_object_clones":0,"num_object_copies":3292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1072,"num_objects_unfound":0,"num_objects_dirty":1646,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1982,"num_write_kb":94400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1150,"num_bytes_recovered":75366400,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5],"acting":[1,5],"avail_no_missing":["1","6"],"object_location_counts":[{"shards":"1,5","objects":1110},{"shards":"1,6","objects":536}],"blocked_by":[],"up_primary":8,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.27","version":"166'5812","reported_seq":10730,"reported_epoch":166,"state":"active+recovering+remapped","last_fresh":"2024-09-13T11:25:26.113633+0000","last_change":"2024-09-13T11:25:08.115804+0000","last_active":"2024-09-13T11:25:26.113633+0000","last_peered":"2024-09-13T11:25:26.113633+0000","last_clean":"2024-09-13T11:24:24.942332+0000","last_became_active":"2024-09-13T11:24:27.215734+0000","last_became_peered":"2024-09-13T11:24:27.215734+0000","last_unstale":"2024-09-13T11:25:26.113633+0000","last_undegraded":"2024-09-13T11:25:26.113633+0000","last_fullsized":"2024-09-13T11:25:26.113633+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":6,"last_scrub":"68'2736","last_scrub_stamp":"2024-09-13T11:23:20.443329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:20.443329+0000","objects_scrubbed":99,"log_size":5812,"log_dups_size":0,"ondisk_log_size":5812,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":183,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":108920832,"num_objects":1662,"num_object_clones":0,"num_object_copies":3324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":392,"num_objects_unfound":0,"num_objects_dirty":1662,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2090,"num_write_kb":98144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1529,"num_bytes_recovered":100204544,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,11],"acting":[8,5],"avail_no_missing":["8","13"],"object_location_counts":[{"shards":"5,8","objects":1466},{"shards":"8,13","objects":196}],"blocked_by":[],"up_primary":5,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.26","version":"166'5927","reported_seq":11498,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-13T11:25:26.025249+0000","last_change":"2024-09-13T11:25:01.697082+0000","last_active":"2024-09-13T11:25:26.025249+0000","last_peered":"2024-09-13T11:25:26.025249+0000","last_clean":"2024-09-13T11:25:26.025249+0000","last_became_active":"2024-09-13T11:24:54.976919+0000","last_became_peered":"2024-09-13T11:24:54.976919+0000","last_unstale":"2024-09-13T11:25:26.025249+0000","last_undegraded":"2024-09-13T11:25:26.025249+0000","last_fullsized":"2024-09-13T11:25:26.025249+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":6,"last_scrub":"147'5433","last_scrub_stamp":"2024-09-13T11:25:01.696886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:01.696886+0000","objects_scrubbed":2158,"log_size":5927,"log_dups_size":0,"ondisk_log_size":5927,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:26:22.070621+0000","scrub_duration":403,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":108527616,"num_objects":1656,"num_object_clones":0,"num_object_copies":3312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1656,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":502,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,12],"acting":[8,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.25","version":"166'5781","reported_seq":11511,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-13T11:25:26.095773+0000","last_change":"2024-09-13T11:24:57.073162+0000","last_active":"2024-09-13T11:25:26.095773+0000","last_peered":"2024-09-13T11:25:26.095773+0000","last_clean":"2024-09-13T11:25:26.095773+0000","last_became_active":"2024-09-13T11:24:55.999019+0000","last_became_peered":"2024-09-13T11:24:55.999019+0000","last_unstale":"2024-09-13T11:25:26.095773+0000","last_undegraded":"2024-09-13T11:25:26.095773+0000","last_fullsized":"2024-09-13T11:25:26.095773+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":6,"last_scrub":"142'5214","last_scrub_stamp":"2024-09-13T11:24:57.073061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:24:57.073061+0000","objects_scrubbed":2249,"log_size":5781,"log_dups_size":0,"ondisk_log_size":5781,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:26:12.348519+0000","scrub_duration":571,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110428160,"num_objects":1685,"num_object_clones":0,"num_object_copies":3370,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1685,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":570,"num_write_kb":192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,2],"acting":[12,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[]},{"pgid":"3.24","version":"168'5832","reported_seq":9752,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-09-13T11:25:26.955529+0000","last_change":"2024-09-13T11:24:48.048846+0000","last_active":"2024-09-13T11:25:26.955529+0000","last_peered":"2024-09-13T11:25:26.955529+0000","last_clean":"2024-09-13T11:25:26.955529+0000","last_became_active":"2024-09-13T11:24:46.936994+0000","last_became_peered":"2024-09-13T11:24:46.936994+0000","last_unstale":"2024-09-13T11:25:26.955529+0000","last_undegraded":"2024-09-13T11:25:26.955529+0000","last_fullsized":"2024-09-13T11:25:26.955529+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":6,"last_scrub":"135'5093","last_scrub_stamp":"2024-09-13T11:24:48.048764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:24:48.048764+0000","objects_scrubbed":2102,"log_size":5832,"log_dups_size":0,"ondisk_log_size":5832,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:53.485370+0000","scrub_duration":287,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":108265472,"num_objects":1652,"num_object_clones":0,"num_object_copies":3304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1652,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":740,"num_write_kb":9280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1],"acting":[9,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.23","version":"166'5040","reported_seq":7891,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-13T11:25:25.955199+0000","last_change":"2024-09-13T11:23:48.133184+0000","last_active":"2024-09-13T11:25:25.955199+0000","last_peered":"2024-09-13T11:25:25.955199+0000","last_clean":"2024-09-13T11:25:25.955199+0000","last_became_active":"2024-09-13T11:23:47.926677+0000","last_became_peered":"2024-09-13T11:23:47.926677+0000","last_unstale":"2024-09-13T11:25:25.955199+0000","last_undegraded":"2024-09-13T11:25:25.955199+0000","last_fullsized":"2024-09-13T11:25:25.955199+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":6,"last_scrub":"91'3278","last_scrub_stamp":"2024-09-13T11:23:48.133084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:48.133084+0000","objects_scrubbed":1029,"log_size":5040,"log_dups_size":0,"ondisk_log_size":5040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":139,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":111804416,"num_objects":1706,"num_object_clones":0,"num_object_copies":3412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1706,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1763,"num_write_kb":78080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9],"acting":[5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.22","version":"166'4987","reported_seq":10567,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-13T11:25:26.093268+0000","last_change":"2024-09-13T11:24:45.485711+0000","last_active":"2024-09-13T11:25:26.093268+0000","last_peered":"2024-09-13T11:25:26.093268+0000","last_clean":"2024-09-13T11:25:26.093268+0000","last_became_active":"2024-09-13T11:24:29.253848+0000","last_became_peered":"2024-09-13T11:24:29.253848+0000","last_unstale":"2024-09-13T11:25:26.093268+0000","last_undegraded":"2024-09-13T11:25:26.093268+0000","last_fullsized":"2024-09-13T11:25:26.093268+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":6,"last_scrub":"90'3246","last_scrub_stamp":"2024-09-13T11:23:47.300339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:47.300339+0000","objects_scrubbed":1033,"log_size":4987,"log_dups_size":0,"ondisk_log_size":4987,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":203,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":108527616,"num_objects":1656,"num_object_clones":0,"num_object_copies":3312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1656,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1743,"num_write_kb":75712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1768,"num_bytes_recovered":115867648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.21","version":"166'4903","reported_seq":7769,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-13T11:25:26.085283+0000","last_change":"2024-09-13T11:24:12.564117+0000","last_active":"2024-09-13T11:25:26.085283+0000","last_peered":"2024-09-13T11:25:26.085283+0000","last_clean":"2024-09-13T11:25:26.085283+0000","last_became_active":"2024-09-13T11:24:01.250308+0000","last_became_peered":"2024-09-13T11:24:01.250308+0000","last_unstale":"2024-09-13T11:25:26.085283+0000","last_undegraded":"2024-09-13T11:25:26.085283+0000","last_fullsized":"2024-09-13T11:25:26.085283+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":6,"last_scrub":"112'3670","last_scrub_stamp":"2024-09-13T11:24:12.564031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:24:12.564031+0000","objects_scrubbed":1431,"log_size":4903,"log_dups_size":0,"ondisk_log_size":4903,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:36.766299+0000","scrub_duration":183,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":103350272,"num_objects":1577,"num_object_clones":0,"num_object_copies":3154,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1577,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1236,"num_write_kb":44224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8],"acting":[0,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.20","version":"166'5035","reported_seq":7188,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-13T11:25:25.998756+0000","last_change":"2024-09-13T11:23:47.969503+0000","last_active":"2024-09-13T11:25:25.998756+0000","last_peered":"2024-09-13T11:25:25.998756+0000","last_clean":"2024-09-13T11:25:25.998756+0000","last_became_active":"2024-09-13T11:23:33.954946+0000","last_became_peered":"2024-09-13T11:23:33.954946+0000","last_unstale":"2024-09-13T11:25:25.998756+0000","last_undegraded":"2024-09-13T11:25:25.998756+0000","last_fullsized":"2024-09-13T11:25:25.998756+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":6,"last_scrub":"91'3270","last_scrub_stamp":"2024-09-13T11:23:47.969372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:47.969372+0000","objects_scrubbed":1061,"log_size":5035,"log_dups_size":0,"ondisk_log_size":5035,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1327,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":106627072,"num_objects":1627,"num_object_clones":0,"num_object_copies":3254,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1627,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1770,"num_write_kb":74752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,0],"acting":[12,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[]},{"pgid":"3.1f","version":"166'7087","reported_seq":19037,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-13T11:25:26.064403+0000","last_change":"2024-09-13T11:25:20.773752+0000","last_active":"2024-09-13T11:25:26.064403+0000","last_peered":"2024-09-13T11:25:26.064403+0000","last_clean":"2024-09-13T11:25:26.064403+0000","last_became_active":"2024-09-13T11:24:29.290326+0000","last_became_peered":"2024-09-13T11:24:29.290326+0000","last_unstale":"2024-09-13T11:25:26.064403+0000","last_undegraded":"2024-09-13T11:25:26.064403+0000","last_fullsized":"2024-09-13T11:25:26.064403+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":5,"last_scrub":"161'6865","last_scrub_stamp":"2024-09-13T11:25:20.773641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:20.773641+0000","objects_scrubbed":3425,"log_size":7087,"log_dups_size":0,"ondisk_log_size":7087,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:26:21.317280+0000","scrub_duration":535,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":209584128,"num_objects":3198,"num_object_clones":0,"num_object_copies":6396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3198,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2667,"num_write_kb":102848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3306,"num_bytes_recovered":216662016,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5],"acting":[10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1e","version":"166'7310","reported_seq":17827,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-13T11:25:25.982804+0000","last_change":"2024-09-13T11:24:08.338485+0000","last_active":"2024-09-13T11:25:25.982804+0000","last_peered":"2024-09-13T11:25:25.982804+0000","last_clean":"2024-09-13T11:25:25.982804+0000","last_became_active":"2024-09-13T11:24:01.261448+0000","last_became_peered":"2024-09-13T11:24:01.261448+0000","last_unstale":"2024-09-13T11:25:25.982804+0000","last_undegraded":"2024-09-13T11:25:25.982804+0000","last_fullsized":"2024-09-13T11:25:25.982804+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":5,"last_scrub":"110'4537","last_scrub_stamp":"2024-09-13T11:24:08.338354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:24:08.338354+0000","objects_scrubbed":2760,"log_size":7310,"log_dups_size":0,"ondisk_log_size":7310,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":367,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":216137728,"num_objects":3298,"num_object_clones":0,"num_object_copies":6596,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3298,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2782,"num_write_kb":106240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"166'7123","reported_seq":12569,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-13T11:25:26.004350+0000","last_change":"2024-09-13T11:24:12.492592+0000","last_active":"2024-09-13T11:25:26.004350+0000","last_peered":"2024-09-13T11:25:26.004350+0000","last_clean":"2024-09-13T11:25:26.004350+0000","last_became_active":"2024-09-13T11:24:01.258225+0000","last_became_peered":"2024-09-13T11:24:01.258225+0000","last_unstale":"2024-09-13T11:25:26.004350+0000","last_undegraded":"2024-09-13T11:25:26.004350+0000","last_fullsized":"2024-09-13T11:25:26.004350+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":5,"last_scrub":"112'4579","last_scrub_stamp":"2024-09-13T11:24:12.492507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:24:12.492507+0000","objects_scrubbed":2875,"log_size":7123,"log_dups_size":0,"ondisk_log_size":7123,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:34.926575+0000","scrub_duration":375,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":214630424,"num_objects":3276,"num_object_clones":0,"num_object_copies":6552,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3276,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2557,"num_write_kb":94593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,15],"acting":[2,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1c","version":"168'7295","reported_seq":11102,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-09-13T11:25:27.053429+0000","last_change":"2024-09-13T11:24:00.529312+0000","last_active":"2024-09-13T11:25:27.053429+0000","last_peered":"2024-09-13T11:25:27.053429+0000","last_clean":"2024-09-13T11:25:27.053429+0000","last_became_active":"2024-09-13T11:23:33.989492+0000","last_became_peered":"2024-09-13T11:23:33.989492+0000","last_unstale":"2024-09-13T11:25:27.053429+0000","last_undegraded":"2024-09-13T11:25:27.053429+0000","last_fullsized":"2024-09-13T11:25:27.053429+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":5,"last_scrub":"102'4277","last_scrub_stamp":"2024-09-13T11:24:00.529243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:24:00.529243+0000","objects_scrubbed":2544,"log_size":7295,"log_dups_size":0,"ondisk_log_size":7295,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":259,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":213909504,"num_objects":3264,"num_object_clones":0,"num_object_copies":6528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3264,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3020,"num_write_kb":119680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7],"acting":[11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1b","version":"166'7246","reported_seq":12871,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-13T11:25:26.099182+0000","last_change":"2024-09-13T11:24:12.392093+0000","last_active":"2024-09-13T11:25:26.099182+0000","last_peered":"2024-09-13T11:25:26.099182+0000","last_clean":"2024-09-13T11:25:26.099182+0000","last_became_active":"2024-09-13T11:24:04.950821+0000","last_became_peered":"2024-09-13T11:24:04.950821+0000","last_unstale":"2024-09-13T11:25:26.099182+0000","last_undegraded":"2024-09-13T11:25:26.099182+0000","last_fullsized":"2024-09-13T11:25:26.099182+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":5,"last_scrub":"112'4665","last_scrub_stamp":"2024-09-13T11:24:12.391890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:24:12.391890+0000","objects_scrubbed":2934,"log_size":7246,"log_dups_size":0,"ondisk_log_size":7246,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:41.846147+0000","scrub_duration":675,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":215613440,"num_objects":3290,"num_object_clones":0,"num_object_copies":6580,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3290,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2602,"num_write_kb":94656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,15],"acting":[12,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[]},{"pgid":"3.1a","version":"166'7125","reported_seq":12411,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-13T11:25:25.999241+0000","last_change":"2024-09-13T11:24:09.556174+0000","last_active":"2024-09-13T11:25:25.999241+0000","last_peered":"2024-09-13T11:25:25.999241+0000","last_clean":"2024-09-13T11:25:25.999241+0000","last_became_active":"2024-09-13T11:24:03.935185+0000","last_became_peered":"2024-09-13T11:24:03.935185+0000","last_unstale":"2024-09-13T11:25:25.999241+0000","last_undegraded":"2024-09-13T11:25:25.999241+0000","last_fullsized":"2024-09-13T11:25:25.999241+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":5,"last_scrub":"110'4433","last_scrub_stamp":"2024-09-13T11:24:09.556022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:24:09.556022+0000","objects_scrubbed":2654,"log_size":7125,"log_dups_size":0,"ondisk_log_size":7125,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:28.469324+0000","scrub_duration":387,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":210698240,"num_objects":3215,"num_object_clones":0,"num_object_copies":6430,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3215,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2703,"num_write_kb":104448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9],"acting":[4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.19","version":"168'6802","reported_seq":11279,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-09-13T11:25:26.985429+0000","last_change":"2024-09-13T11:25:10.784789+0000","last_active":"2024-09-13T11:25:26.985429+0000","last_peered":"2024-09-13T11:25:26.985429+0000","last_clean":"2024-09-13T11:25:26.985429+0000","last_became_active":"2024-09-13T11:23:41.924941+0000","last_became_peered":"2024-09-13T11:23:41.924941+0000","last_unstale":"2024-09-13T11:25:26.985429+0000","last_undegraded":"2024-09-13T11:25:26.985429+0000","last_fullsized":"2024-09-13T11:25:26.985429+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":5,"last_scrub":"151'6156","last_scrub_stamp":"2024-09-13T11:25:10.784716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:10.784716+0000","objects_scrubbed":3781,"log_size":6802,"log_dups_size":0,"ondisk_log_size":6802,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:26:30.390310+0000","scrub_duration":1131,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":204275712,"num_objects":3117,"num_object_clones":0,"num_object_copies":6234,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3117,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3400,"num_write_kb":145216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1],"acting":[9,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.18","version":"166'6844","reported_seq":11579,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-13T11:25:26.060441+0000","last_change":"2024-09-13T11:24:13.454659+0000","last_active":"2024-09-13T11:25:26.060441+0000","last_peered":"2024-09-13T11:25:26.060441+0000","last_clean":"2024-09-13T11:25:26.060441+0000","last_became_active":"2024-09-13T11:23:48.926014+0000","last_became_peered":"2024-09-13T11:23:48.926014+0000","last_unstale":"2024-09-13T11:25:26.060441+0000","last_undegraded":"2024-09-13T11:25:26.060441+0000","last_fullsized":"2024-09-13T11:25:26.060441+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":5,"last_scrub":"112'4376","last_scrub_stamp":"2024-09-13T11:24:13.454575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:24:13.454575+0000","objects_scrubbed":3036,"log_size":6844,"log_dups_size":0,"ondisk_log_size":6844,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:34.811860+0000","scrub_duration":375,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":213843968,"num_objects":3263,"num_object_clones":0,"num_object_copies":6526,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3263,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2477,"num_write_kb":86528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0],"acting":[10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.17","version":"166'6739","reported_seq":10945,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-13T11:25:26.063961+0000","last_change":"2024-09-13T11:25:12.658562+0000","last_active":"2024-09-13T11:25:26.063961+0000","last_peered":"2024-09-13T11:25:26.063961+0000","last_clean":"2024-09-13T11:25:26.063961+0000","last_became_active":"2024-09-13T11:23:42.935601+0000","last_became_peered":"2024-09-13T11:23:42.935601+0000","last_unstale":"2024-09-13T11:25:26.063961+0000","last_undegraded":"2024-09-13T11:25:26.063961+0000","last_fullsized":"2024-09-13T11:25:26.063961+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":5,"last_scrub":"153'6197","last_scrub_stamp":"2024-09-13T11:25:12.658464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:12.658464+0000","objects_scrubbed":3743,"log_size":6739,"log_dups_size":0,"ondisk_log_size":6739,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:26:25.625277+0000","scrub_duration":467,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":209715200,"num_objects":3200,"num_object_clones":0,"num_object_copies":6400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3200,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3388,"num_write_kb":146688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3],"acting":[10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"168'7080","reported_seq":11926,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-09-13T11:25:26.981846+0000","last_change":"2024-09-13T11:25:01.903487+0000","last_active":"2024-09-13T11:25:26.981846+0000","last_peered":"2024-09-13T11:25:26.981846+0000","last_clean":"2024-09-13T11:25:26.981846+0000","last_became_active":"2024-09-13T11:23:46.105237+0000","last_became_peered":"2024-09-13T11:23:46.105237+0000","last_unstale":"2024-09-13T11:25:26.981846+0000","last_undegraded":"2024-09-13T11:25:26.981846+0000","last_fullsized":"2024-09-13T11:25:26.981846+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":5,"last_scrub":"147'6035","last_scrub_stamp":"2024-09-13T11:25:01.903395+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:01.903395+0000","objects_scrubbed":4359,"log_size":7080,"log_dups_size":0,"ondisk_log_size":7080,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:26:16.436377+0000","scrub_duration":743,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":216203264,"num_objects":3299,"num_object_clones":0,"num_object_copies":6598,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3299,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3605,"num_write_kb":155200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,15],"acting":[11,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.15","version":"166'6726","reported_seq":18051,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-13T11:25:26.070608+0000","last_change":"2024-09-13T11:25:12.198448+0000","last_active":"2024-09-13T11:25:26.070608+0000","last_peered":"2024-09-13T11:25:26.070608+0000","last_clean":"2024-09-13T11:25:26.070608+0000","last_became_active":"2024-09-13T11:24:29.226615+0000","last_became_peered":"2024-09-13T11:24:29.226615+0000","last_unstale":"2024-09-13T11:25:26.070608+0000","last_undegraded":"2024-09-13T11:25:26.070608+0000","last_fullsized":"2024-09-13T11:25:26.070608+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":5,"last_scrub":"91'3384","last_scrub_stamp":"2024-09-13T11:23:48.332600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:48.332600+0000","objects_scrubbed":2032,"log_size":6726,"log_dups_size":0,"ondisk_log_size":6726,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":235,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":209649664,"num_objects":3199,"num_object_clones":0,"num_object_copies":6398,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3199,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3343,"num_write_kb":144320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3385,"num_bytes_recovered":221839360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.14","version":"169'7021","reported_seq":18278,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-09-13T11:25:29.995610+0000","last_change":"2024-09-13T11:25:15.378521+0000","last_active":"2024-09-13T11:25:29.995610+0000","last_peered":"2024-09-13T11:25:29.995610+0000","last_clean":"2024-09-13T11:25:29.995610+0000","last_became_active":"2024-09-13T11:24:29.253106+0000","last_became_peered":"2024-09-13T11:24:29.253106+0000","last_unstale":"2024-09-13T11:25:29.995610+0000","last_undegraded":"2024-09-13T11:25:29.995610+0000","last_fullsized":"2024-09-13T11:25:29.995610+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":5,"last_scrub":"91'3371","last_scrub_stamp":"2024-09-13T11:23:48.696960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:48.696960+0000","objects_scrubbed":1935,"log_size":7021,"log_dups_size":0,"ondisk_log_size":7021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":735,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":211746816,"num_objects":3231,"num_object_clones":0,"num_object_copies":6462,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3231,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3660,"num_write_kb":158592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3437,"num_bytes_recovered":225247232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,15],"acting":[3,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":304,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-13T11:25:25.226545+0000","last_change":"2024-09-13T11:24:23.212494+0000","last_active":"2024-09-13T11:25:25.226545+0000","last_peered":"2024-09-13T11:25:25.226545+0000","last_clean":"2024-09-13T11:25:25.226545+0000","last_became_active":"2024-09-13T11:22:14.192897+0000","last_became_peered":"2024-09-13T11:22:14.192897+0000","last_unstale":"2024-09-13T11:25:25.226545+0000","last_undegraded":"2024-09-13T11:25:25.226545+0000","last_fullsized":"2024-09-13T11:25:25.226545+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:24:23.212406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:24:23.212406+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:29.886382+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00084655899999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,10],"acting":[15,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"3.6","version":"168'5938","reported_seq":9805,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-09-13T11:25:27.033999+0000","last_change":"2024-09-13T11:24:49.416851+0000","last_active":"2024-09-13T11:25:27.033999+0000","last_peered":"2024-09-13T11:25:27.033999+0000","last_clean":"2024-09-13T11:25:27.033999+0000","last_became_active":"2024-09-13T11:24:29.322558+0000","last_became_peered":"2024-09-13T11:24:29.322558+0000","last_unstale":"2024-09-13T11:25:27.033999+0000","last_undegraded":"2024-09-13T11:25:27.033999+0000","last_fullsized":"2024-09-13T11:25:27.033999+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"136'5216","last_scrub_stamp":"2024-09-13T11:24:49.416706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:24:49.416706+0000","objects_scrubbed":2083,"log_size":5938,"log_dups_size":0,"ondisk_log_size":5938,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:26:09.525086+0000","scrub_duration":247,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":108396544,"num_objects":1654,"num_object_clones":0,"num_object_copies":3308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1654,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":723,"num_write_kb":9408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,6],"acting":[13,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":315,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-09-13T11:25:26.955759+0000","last_change":"2024-09-13T11:24:59.556253+0000","last_active":"2024-09-13T11:25:26.955759+0000","last_peered":"2024-09-13T11:25:26.955759+0000","last_clean":"2024-09-13T11:25:26.955759+0000","last_became_active":"2024-09-13T11:22:14.211923+0000","last_became_peered":"2024-09-13T11:22:14.211923+0000","last_unstale":"2024-09-13T11:25:26.955759+0000","last_undegraded":"2024-09-13T11:25:26.955759+0000","last_fullsized":"2024-09-13T11:25:26.955759+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:24:59.556180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:24:59.556180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:26:06.229322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011689529999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0],"acting":[9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"168'5770","reported_seq":6131,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-09-13T11:25:27.024109+0000","last_change":"2024-09-13T11:24:33.336210+0000","last_active":"2024-09-13T11:25:27.024109+0000","last_peered":"2024-09-13T11:25:27.024109+0000","last_clean":"2024-09-13T11:25:27.024109+0000","last_became_active":"2024-09-13T11:24:24.092975+0000","last_became_peered":"2024-09-13T11:24:24.092975+0000","last_unstale":"2024-09-13T11:25:27.024109+0000","last_undegraded":"2024-09-13T11:25:27.024109+0000","last_fullsized":"2024-09-13T11:25:27.024109+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"125'4820","last_scrub_stamp":"2024-09-13T11:24:33.336091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:24:33.336091+0000","objects_scrubbed":1766,"log_size":5770,"log_dups_size":0,"ondisk_log_size":5770,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:54.538352+0000","scrub_duration":263,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":107347968,"num_objects":1638,"num_object_clones":0,"num_object_copies":3276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1638,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":952,"num_write_kb":26368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,8],"acting":[13,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":304,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-13T11:25:25.221085+0000","last_change":"2024-09-13T11:23:59.530794+0000","last_active":"2024-09-13T11:25:25.221085+0000","last_peered":"2024-09-13T11:25:25.221085+0000","last_clean":"2024-09-13T11:25:25.221085+0000","last_became_active":"2024-09-13T11:22:14.581458+0000","last_became_peered":"2024-09-13T11:22:14.581458+0000","last_unstale":"2024-09-13T11:25:25.221085+0000","last_undegraded":"2024-09-13T11:25:25.221085+0000","last_fullsized":"2024-09-13T11:25:25.221085+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:23:59.530670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:23:59.530670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001026047,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5],"acting":[8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"166'5817","reported_seq":6146,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-13T11:25:25.378158+0000","last_change":"2024-09-13T11:24:24.579036+0000","last_active":"2024-09-13T11:25:25.378158+0000","last_peered":"2024-09-13T11:25:25.378158+0000","last_clean":"2024-09-13T11:25:25.378158+0000","last_became_active":"2024-09-13T11:24:24.083077+0000","last_became_peered":"2024-09-13T11:24:24.083077+0000","last_unstale":"2024-09-13T11:25:25.378158+0000","last_undegraded":"2024-09-13T11:25:25.378158+0000","last_fullsized":"2024-09-13T11:25:25.378158+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'4737","last_scrub_stamp":"2024-09-13T11:24:24.578900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:24:24.578900+0000","objects_scrubbed":1690,"log_size":5817,"log_dups_size":0,"ondisk_log_size":5817,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:38.103421+0000","scrub_duration":407,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":106758144,"num_objects":1629,"num_object_clones":0,"num_object_copies":3258,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1629,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1083,"num_write_kb":32704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,12],"acting":[1,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":292,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-13T11:25:25.217725+0000","last_change":"2024-09-13T11:24:08.271026+0000","last_active":"2024-09-13T11:25:25.217725+0000","last_peered":"2024-09-13T11:25:25.217725+0000","last_clean":"2024-09-13T11:25:25.217725+0000","last_became_active":"2024-09-13T11:22:47.719636+0000","last_became_peered":"2024-09-13T11:22:47.719636+0000","last_unstale":"2024-09-13T11:25:25.217725+0000","last_undegraded":"2024-09-13T11:25:25.217725+0000","last_fullsized":"2024-09-13T11:25:25.217725+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:24:08.270972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:24:08.270972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:36.805605+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00087476000000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,9],"acting":[1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"166'5772","reported_seq":6120,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-13T11:25:26.041019+0000","last_change":"2024-09-13T11:24:24.616937+0000","last_active":"2024-09-13T11:25:26.041019+0000","last_peered":"2024-09-13T11:25:26.041019+0000","last_clean":"2024-09-13T11:25:26.041019+0000","last_became_active":"2024-09-13T11:24:24.093492+0000","last_became_peered":"2024-09-13T11:24:24.093492+0000","last_unstale":"2024-09-13T11:25:26.041019+0000","last_undegraded":"2024-09-13T11:25:26.041019+0000","last_fullsized":"2024-09-13T11:25:26.041019+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'4744","last_scrub_stamp":"2024-09-13T11:24:24.616633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:24:24.616633+0000","objects_scrubbed":1624,"log_size":5772,"log_dups_size":0,"ondisk_log_size":5772,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:42.911074+0000","scrub_duration":415,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":104923136,"num_objects":1601,"num_object_clones":0,"num_object_copies":3202,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1601,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1035,"num_write_kb":32384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,13],"acting":[15,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"3.3","version":"166'4964","reported_seq":5331,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-13T11:25:26.038212+0000","last_change":"2024-09-13T11:25:08.394545+0000","last_active":"2024-09-13T11:25:26.038212+0000","last_peered":"2024-09-13T11:25:26.038212+0000","last_clean":"2024-09-13T11:25:26.038212+0000","last_became_active":"2024-09-13T11:23:31.930448+0000","last_became_peered":"2024-09-13T11:23:31.930448+0000","last_unstale":"2024-09-13T11:25:26.038212+0000","last_undegraded":"2024-09-13T11:25:26.038212+0000","last_fullsized":"2024-09-13T11:25:26.038212+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"151'4587","last_scrub_stamp":"2024-09-13T11:25:08.394441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:08.394441+0000","objects_scrubbed":1942,"log_size":4964,"log_dups_size":0,"ondisk_log_size":4964,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:26:27.641900+0000","scrub_duration":219,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":102367232,"num_objects":1562,"num_object_clones":0,"num_object_copies":3124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1562,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1842,"num_write_kb":80896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,7],"acting":[10,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"28'2","reported_seq":242,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-13T11:25:25.216396+0000","last_change":"2024-09-13T11:25:13.869564+0000","last_active":"2024-09-13T11:25:25.216396+0000","last_peered":"2024-09-13T11:25:25.216396+0000","last_clean":"2024-09-13T11:25:25.216396+0000","last_became_active":"2024-09-13T11:23:12.546348+0000","last_became_peered":"2024-09-13T11:23:12.546348+0000","last_unstale":"2024-09-13T11:25:25.216396+0000","last_undegraded":"2024-09-13T11:25:25.216396+0000","last_fullsized":"2024-09-13T11:25:25.216396+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2","last_scrub_stamp":"2024-09-13T11:25:13.869413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:25:13.869413+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:26:38.761718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0025826730000000002,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8],"acting":[2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"166'4962","reported_seq":5328,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-13T11:25:26.067735+0000","last_change":"2024-09-13T11:25:09.842875+0000","last_active":"2024-09-13T11:25:26.067735+0000","last_peered":"2024-09-13T11:25:26.067735+0000","last_clean":"2024-09-13T11:25:26.067735+0000","last_became_active":"2024-09-13T11:23:31.921388+0000","last_became_peered":"2024-09-13T11:23:31.921388+0000","last_unstale":"2024-09-13T11:25:26.067735+0000","last_undegraded":"2024-09-13T11:25:26.067735+0000","last_fullsized":"2024-09-13T11:25:26.067735+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"151'4644","last_scrub_stamp":"2024-09-13T11:25:09.842728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:09.842728+0000","objects_scrubbed":1944,"log_size":4962,"log_dups_size":0,"ondisk_log_size":4962,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:26:25.553642+0000","scrub_duration":395,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":106561536,"num_objects":1626,"num_object_clones":0,"num_object_copies":3252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1626,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1856,"num_write_kb":84032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,15],"acting":[12,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":145,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-13T11:25:25.217562+0000","last_change":"2024-09-13T11:24:29.096863+0000","last_active":"2024-09-13T11:25:25.217562+0000","last_peered":"2024-09-13T11:25:25.217562+0000","last_clean":"2024-09-13T11:25:25.217562+0000","last_became_active":"2024-09-13T11:24:29.096575+0000","last_became_peered":"2024-09-13T11:24:29.096575+0000","last_unstale":"2024-09-13T11:25:25.217562+0000","last_undegraded":"2024-09-13T11:25:25.217562+0000","last_fullsized":"2024-09-13T11:25:25.217562+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:24:22.064711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:24:22.064711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:34.178168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00081225399999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"168'4968","reported_seq":5340,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-09-13T11:25:26.935370+0000","last_change":"2024-09-13T11:25:00.861990+0000","last_active":"2024-09-13T11:25:26.935370+0000","last_peered":"2024-09-13T11:25:26.935370+0000","last_clean":"2024-09-13T11:25:26.935370+0000","last_became_active":"2024-09-13T11:23:31.926528+0000","last_became_peered":"2024-09-13T11:23:31.926528+0000","last_unstale":"2024-09-13T11:25:26.935370+0000","last_undegraded":"2024-09-13T11:25:26.935370+0000","last_fullsized":"2024-09-13T11:25:26.935370+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"146'4447","last_scrub_stamp":"2024-09-13T11:25:00.861834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:00.861834+0000","objects_scrubbed":2126,"log_size":4968,"log_dups_size":0,"ondisk_log_size":4968,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:26:15.411903+0000","scrub_duration":787,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":104988672,"num_objects":1602,"num_object_clones":0,"num_object_copies":3204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1602,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1898,"num_write_kb":85056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,11],"acting":[13,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":310,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-09-13T11:25:29.252224+0000","last_change":"2024-09-13T11:24:11.895017+0000","last_active":"2024-09-13T11:25:29.252224+0000","last_peered":"2024-09-13T11:25:29.252224+0000","last_clean":"2024-09-13T11:25:29.252224+0000","last_became_active":"2024-09-13T11:22:14.211435+0000","last_became_peered":"2024-09-13T11:22:14.211435+0000","last_unstale":"2024-09-13T11:25:29.252224+0000","last_undegraded":"2024-09-13T11:25:29.252224+0000","last_fullsized":"2024-09-13T11:25:29.252224+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:24:11.894937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:24:11.894937+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0010010360000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2","version":"166'5033","reported_seq":8863,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-13T11:25:25.866101+0000","last_change":"2024-09-13T11:25:23.765128+0000","last_active":"2024-09-13T11:25:25.866101+0000","last_peered":"2024-09-13T11:25:25.866101+0000","last_clean":"2024-09-13T11:25:25.866101+0000","last_became_active":"2024-09-13T11:24:29.175407+0000","last_became_peered":"2024-09-13T11:24:29.175407+0000","last_unstale":"2024-09-13T11:25:25.866101+0000","last_undegraded":"2024-09-13T11:25:25.866101+0000","last_fullsized":"2024-09-13T11:25:25.866101+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"164'4994","last_scrub_stamp":"2024-09-13T11:25:23.765047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:23.765047+0000","objects_scrubbed":1689,"log_size":5033,"log_dups_size":0,"ondisk_log_size":5033,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:26:45.252942+0000","scrub_duration":283,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":107937792,"num_objects":1647,"num_object_clones":0,"num_object_copies":3294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1647,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1976,"num_write_kb":89024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1731,"num_bytes_recovered":113442816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,3],"acting":[12,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[]},{"pgid":"2.3","version":"26'1","reported_seq":115,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-13T11:25:25.226319+0000","last_change":"2024-09-13T11:25:23.034357+0000","last_active":"2024-09-13T11:25:25.226319+0000","last_peered":"2024-09-13T11:25:25.226319+0000","last_clean":"2024-09-13T11:25:25.226319+0000","last_became_active":"2024-09-13T11:24:29.110028+0000","last_became_peered":"2024-09-13T11:24:29.110028+0000","last_unstale":"2024-09-13T11:25:25.226319+0000","last_undegraded":"2024-09-13T11:25:25.226319+0000","last_fullsized":"2024-09-13T11:25:25.226319+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"26'1","last_scrub_stamp":"2024-09-13T11:25:23.034247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:25:23.034247+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:26:42.429349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,8],"acting":[15,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"1.0","version":"26'112","reported_seq":446,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-13T11:25:25.232244+0000","last_change":"2024-09-13T11:25:23.642366+0000","last_active":"2024-09-13T11:25:25.232244+0000","last_peered":"2024-09-13T11:25:25.232244+0000","last_clean":"2024-09-13T11:25:25.232244+0000","last_became_active":"2024-09-13T11:22:12.129721+0000","last_became_peered":"2024-09-13T11:22:12.129721+0000","last_unstale":"2024-09-13T11:25:25.232244+0000","last_undegraded":"2024-09-13T11:25:25.232244+0000","last_fullsized":"2024-09-13T11:25:25.232244+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"26'112","last_scrub_stamp":"2024-09-13T11:25:23.642204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:11.098388+0000","last_clean_scrub_stamp":"2024-09-13T11:25:23.642204+0000","objects_scrubbed":2,"log_size":112,"log_dups_size":0,"ondisk_log_size":112,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:26:42.628319+0000","scrub_duration":67,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.8","version":"166'5776","reported_seq":6125,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-13T11:25:25.960544+0000","last_change":"2024-09-13T11:24:24.506212+0000","last_active":"2024-09-13T11:25:25.960544+0000","last_peered":"2024-09-13T11:25:25.960544+0000","last_clean":"2024-09-13T11:25:25.960544+0000","last_became_active":"2024-09-13T11:24:24.096220+0000","last_became_peered":"2024-09-13T11:24:24.096220+0000","last_unstale":"2024-09-13T11:25:25.960544+0000","last_undegraded":"2024-09-13T11:25:25.960544+0000","last_fullsized":"2024-09-13T11:25:25.960544+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'4749","last_scrub_stamp":"2024-09-13T11:24:24.506109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:24:24.506109+0000","objects_scrubbed":1690,"log_size":5776,"log_dups_size":0,"ondisk_log_size":5776,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:49.430042+0000","scrub_duration":215,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":107347968,"num_objects":1638,"num_object_clones":0,"num_object_copies":3276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1638,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1028,"num_write_kb":31232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.9","version":"166'5817","reported_seq":6165,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-13T11:25:25.911883+0000","last_change":"2024-09-13T11:24:24.928964+0000","last_active":"2024-09-13T11:25:25.911883+0000","last_peered":"2024-09-13T11:25:25.911883+0000","last_clean":"2024-09-13T11:25:25.911883+0000","last_became_active":"2024-09-13T11:24:24.082504+0000","last_became_peered":"2024-09-13T11:24:24.082504+0000","last_unstale":"2024-09-13T11:25:25.911883+0000","last_undegraded":"2024-09-13T11:25:25.911883+0000","last_fullsized":"2024-09-13T11:25:25.911883+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'4799","last_scrub_stamp":"2024-09-13T11:24:24.928724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:24:24.928724+0000","objects_scrubbed":1705,"log_size":5817,"log_dups_size":0,"ondisk_log_size":5817,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:43.845801+0000","scrub_duration":255,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":108527616,"num_objects":1656,"num_object_clones":0,"num_object_copies":3312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1656,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1021,"num_write_kb":31104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,2],"acting":[12,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[]},{"pgid":"3.a","version":"169'6243","reported_seq":6602,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-09-13T11:25:30.011322+0000","last_change":"2024-09-13T11:24:25.137871+0000","last_active":"2024-09-13T11:25:30.011322+0000","last_peered":"2024-09-13T11:25:30.011322+0000","last_clean":"2024-09-13T11:25:30.011322+0000","last_became_active":"2024-09-13T11:24:24.089253+0000","last_became_peered":"2024-09-13T11:24:24.089253+0000","last_unstale":"2024-09-13T11:25:30.011322+0000","last_undegraded":"2024-09-13T11:25:30.011322+0000","last_fullsized":"2024-09-13T11:25:30.011322+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'5157","last_scrub_stamp":"2024-09-13T11:24:25.137777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:24:25.137777+0000","objects_scrubbed":1777,"log_size":6243,"log_dups_size":0,"ondisk_log_size":6243,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:36.149013+0000","scrub_duration":223,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":102367232,"num_objects":1562,"num_object_clones":0,"num_object_copies":3124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1562,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1089,"num_write_kb":27968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.b","version":"168'6162","reported_seq":6518,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-09-13T11:25:26.980603+0000","last_change":"2024-09-13T11:24:25.503864+0000","last_active":"2024-09-13T11:25:26.980603+0000","last_peered":"2024-09-13T11:25:26.980603+0000","last_clean":"2024-09-13T11:25:26.980603+0000","last_became_active":"2024-09-13T11:24:24.085684+0000","last_became_peered":"2024-09-13T11:24:24.085684+0000","last_unstale":"2024-09-13T11:25:26.980603+0000","last_undegraded":"2024-09-13T11:25:26.980603+0000","last_fullsized":"2024-09-13T11:25:26.980603+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'5112","last_scrub_stamp":"2024-09-13T11:24:25.503716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:24:25.503716+0000","objects_scrubbed":1718,"log_size":6162,"log_dups_size":0,"ondisk_log_size":6162,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:37.636639+0000","scrub_duration":395,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":106561536,"num_objects":1626,"num_object_clones":0,"num_object_copies":3252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1626,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1050,"num_write_kb":30656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,1],"acting":[13,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"3.c","version":"168'6166","reported_seq":6520,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-09-13T11:25:26.972326+0000","last_change":"2024-09-13T11:24:24.595021+0000","last_active":"2024-09-13T11:25:26.972326+0000","last_peered":"2024-09-13T11:25:26.972326+0000","last_clean":"2024-09-13T11:25:26.972326+0000","last_became_active":"2024-09-13T11:24:24.090847+0000","last_became_peered":"2024-09-13T11:24:24.090847+0000","last_unstale":"2024-09-13T11:25:26.972326+0000","last_undegraded":"2024-09-13T11:25:26.972326+0000","last_fullsized":"2024-09-13T11:25:26.972326+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'5154","last_scrub_stamp":"2024-09-13T11:24:24.594848+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:24:24.594848+0000","objects_scrubbed":1720,"log_size":6166,"log_dups_size":0,"ondisk_log_size":6166,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:35.181590+0000","scrub_duration":343,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":106364928,"num_objects":1623,"num_object_clones":0,"num_object_copies":3246,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1623,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1017,"num_write_kb":29440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,13],"acting":[11,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.d","version":"166'6161","reported_seq":6512,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-13T11:25:26.002410+0000","last_change":"2024-09-13T11:24:24.967444+0000","last_active":"2024-09-13T11:25:26.002410+0000","last_peered":"2024-09-13T11:25:26.002410+0000","last_clean":"2024-09-13T11:25:26.002410+0000","last_became_active":"2024-09-13T11:24:24.087224+0000","last_became_peered":"2024-09-13T11:24:24.087224+0000","last_unstale":"2024-09-13T11:25:26.002410+0000","last_undegraded":"2024-09-13T11:25:26.002410+0000","last_fullsized":"2024-09-13T11:25:26.002410+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'5110","last_scrub_stamp":"2024-09-13T11:24:24.967337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:24:24.967337+0000","objects_scrubbed":1725,"log_size":6161,"log_dups_size":0,"ondisk_log_size":6161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:53.620949+0000","scrub_duration":255,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":107282432,"num_objects":1637,"num_object_clones":0,"num_object_copies":3274,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1637,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1054,"num_write_kb":30912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.e","version":"166'7290","reported_seq":12538,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-13T11:25:26.050702+0000","last_change":"2024-09-13T11:23:32.368856+0000","last_active":"2024-09-13T11:25:26.050702+0000","last_peered":"2024-09-13T11:25:26.050702+0000","last_clean":"2024-09-13T11:25:26.050702+0000","last_became_active":"2024-09-13T11:23:31.925869+0000","last_became_peered":"2024-09-13T11:23:31.925869+0000","last_unstale":"2024-09-13T11:25:26.050702+0000","last_undegraded":"2024-09-13T11:25:26.050702+0000","last_fullsized":"2024-09-13T11:25:26.050702+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'3527","last_scrub_stamp":"2024-09-13T11:23:32.368727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:32.368727+0000","objects_scrubbed":1774,"log_size":7290,"log_dups_size":0,"ondisk_log_size":7290,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":219,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":209321984,"num_objects":3194,"num_object_clones":0,"num_object_copies":6388,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3194,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3764,"num_write_kb":165888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"168'7308","reported_seq":7683,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-09-13T11:25:27.069976+0000","last_change":"2024-09-13T11:25:00.289940+0000","last_active":"2024-09-13T11:25:27.069976+0000","last_peered":"2024-09-13T11:25:27.069976+0000","last_clean":"2024-09-13T11:25:27.069976+0000","last_became_active":"2024-09-13T11:23:31.935239+0000","last_became_peered":"2024-09-13T11:23:31.935239+0000","last_unstale":"2024-09-13T11:25:27.069976+0000","last_undegraded":"2024-09-13T11:25:27.069976+0000","last_fullsized":"2024-09-13T11:25:27.069976+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"146'6154","last_scrub_stamp":"2024-09-13T11:25:00.289673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:00.289673+0000","objects_scrubbed":4394,"log_size":7308,"log_dups_size":0,"ondisk_log_size":7308,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:26:24.274856+0000","scrub_duration":1187,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":210894848,"num_objects":3218,"num_object_clones":0,"num_object_copies":6436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3218,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3888,"num_write_kb":170560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,12],"acting":[11,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.10","version":"166'6955","reported_seq":11477,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-13T11:25:26.029727+0000","last_change":"2024-09-13T11:25:10.161564+0000","last_active":"2024-09-13T11:25:26.029727+0000","last_peered":"2024-09-13T11:25:26.029727+0000","last_clean":"2024-09-13T11:25:26.029727+0000","last_became_active":"2024-09-13T11:23:41.931183+0000","last_became_peered":"2024-09-13T11:23:41.931183+0000","last_unstale":"2024-09-13T11:25:26.029727+0000","last_undegraded":"2024-09-13T11:25:26.029727+0000","last_fullsized":"2024-09-13T11:25:26.029727+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":5,"last_scrub":"151'6298","last_scrub_stamp":"2024-09-13T11:25:10.161464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:10.161464+0000","objects_scrubbed":3964,"log_size":6955,"log_dups_size":0,"ondisk_log_size":6955,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:26:33.015478+0000","scrub_duration":607,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":215744512,"num_objects":3292,"num_object_clones":0,"num_object_copies":6584,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3292,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3489,"num_write_kb":152000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.11","version":"166'6880","reported_seq":10174,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-13T11:25:26.042414+0000","last_change":"2024-09-13T11:25:24.287326+0000","last_active":"2024-09-13T11:25:26.042414+0000","last_peered":"2024-09-13T11:25:26.042414+0000","last_clean":"2024-09-13T11:25:26.042414+0000","last_became_active":"2024-09-13T11:23:39.063172+0000","last_became_peered":"2024-09-13T11:23:39.063172+0000","last_unstale":"2024-09-13T11:25:26.042414+0000","last_undegraded":"2024-09-13T11:25:26.042414+0000","last_fullsized":"2024-09-13T11:25:26.042414+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":5,"last_scrub":"165'6821","last_scrub_stamp":"2024-09-13T11:25:24.287082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:24.287082+0000","objects_scrubbed":3406,"log_size":6880,"log_dups_size":0,"ondisk_log_size":6880,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:26:49.405332+0000","scrub_duration":947,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":218497024,"num_objects":3334,"num_object_clones":0,"num_object_copies":6668,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3334,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3539,"num_write_kb":155584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11],"acting":[4,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.12","version":"166'6901","reported_seq":17004,"reported_epoch":166,"state":"active+clean+scrubbing","last_fresh":"2024-09-13T11:25:26.193477+0000","last_change":"2024-09-13T11:25:26.136263+0000","last_active":"2024-09-13T11:25:26.193477+0000","last_peered":"2024-09-13T11:25:26.193477+0000","last_clean":"2024-09-13T11:25:26.193477+0000","last_became_active":"2024-09-13T11:24:29.268229+0000","last_became_peered":"2024-09-13T11:24:29.268229+0000","last_unstale":"2024-09-13T11:25:26.193477+0000","last_undegraded":"2024-09-13T11:25:26.193477+0000","last_fullsized":"2024-09-13T11:25:26.193477+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'3109","last_scrub_stamp":"2024-09-13T11:23:35.111320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:35.111320+0000","objects_scrubbed":399,"log_size":6901,"log_dups_size":0,"ondisk_log_size":6901,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"scrubbing for 0s","scrub_duration":239,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":216662016,"num_objects":3306,"num_object_clones":0,"num_object_copies":6612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3306,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3792,"num_write_kb":171712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3457,"num_bytes_recovered":226557952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.13","version":"168'6980","reported_seq":10213,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-09-13T11:25:27.021692+0000","last_change":"2024-09-13T11:25:01.153044+0000","last_active":"2024-09-13T11:25:27.021692+0000","last_peered":"2024-09-13T11:25:27.021692+0000","last_clean":"2024-09-13T11:25:27.021692+0000","last_became_active":"2024-09-13T11:23:41.082433+0000","last_became_peered":"2024-09-13T11:23:41.082433+0000","last_unstale":"2024-09-13T11:25:27.021692+0000","last_undegraded":"2024-09-13T11:25:27.021692+0000","last_fullsized":"2024-09-13T11:25:27.021692+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":5,"last_scrub":"147'5921","last_scrub_stamp":"2024-09-13T11:25:01.152971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:01.152971+0000","objects_scrubbed":4311,"log_size":6980,"log_dups_size":0,"ondisk_log_size":6980,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:26:07.027631+0000","scrub_duration":1011,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":210436096,"num_objects":3211,"num_object_clones":0,"num_object_copies":6422,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3211,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3584,"num_write_kb":154048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7],"acting":[11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":46,"stat_sum":{"num_bytes":6824230936,"num_objects":104131,"num_object_clones":0,"num_object_copies":208262,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2396,"num_objects_unfound":0,"num_objects_dirty":104131,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":92565,"num_write_kb":3634433,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21016,"num_bytes_recovered":1377304576,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":15303843840,"data_stored":15303835696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":43282,"internal_metadata":0},"log_size":285949,"ondisk_log_size":285949,"up":92,"acting":92,"num_store_stats":16},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":11},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":112,"ondisk_log_size":112,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":13,"up_from":23,"seq":98784247851,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":848884,"kb_used_data":741284,"kb_used_omap":1,"kb_used_meta":107518,"kb_avail":93522956,"statfs":{"total":96636764160,"available":95767506944,"internally_reserved":0,"allocated":759074816,"data_stored":758332501,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":110098885},"hb_peers":[0,1,2,4,6,8,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":23,"seq":98784247850,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":681240,"kb_used_data":618056,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":93690600,"statfs":{"total":96636764160,"available":95939174400,"internally_reserved":0,"allocated":632889344,"data_stored":632178825,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":64616896},"hb_peers":[0,1,3,8,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":40,"seq":171798691873,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1171488,"kb_used_data":1067280,"kb_used_omap":4,"kb_used_meta":104187,"kb_avail":93200352,"statfs":{"total":96636764160,"available":95437160448,"internally_reserved":0,"allocated":1092894720,"data_stored":1092165338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4433,"internal_metadata":106688175},"hb_peers":[0,2,3,4,5,6,7,8,9,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":21,"seq":90194313259,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1159400,"kb_used_data":1056024,"kb_used_omap":1,"kb_used_meta":103358,"kb_avail":93212440,"statfs":{"total":96636764160,"available":95449538560,"internally_reserved":0,"allocated":1081368576,"data_stored":1080601551,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":105839047},"hb_peers":[0,2,4,5,7,9,10,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":21,"seq":90194313261,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1257128,"kb_used_data":1175960,"kb_used_omap":1,"kb_used_meta":81150,"kb_avail":93114712,"statfs":{"total":96636764160,"available":95349465088,"internally_reserved":0,"allocated":1204183040,"data_stored":1203447594,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":83098055},"hb_peers":[0,2,4,6,7,8,9,11,12,13],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":22,"seq":94489280553,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1307644,"kb_used_data":1200108,"kb_used_omap":1,"kb_used_meta":107518,"kb_avail":93064196,"statfs":{"total":96636764160,"available":95297736704,"internally_reserved":0,"allocated":1228910592,"data_stored":1228223837,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":110098885},"hb_peers":[1,3,4,5,7,8,10,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":22,"seq":94489280554,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":602472,"kb_used_data":526872,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":93769368,"statfs":{"total":96636764160,"available":96019832832,"internally_reserved":0,"allocated":539516928,"data_stored":538792320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,1,3,4,5,7,8,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":15,"up_from":22,"seq":94489280554,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1227888,"kb_used_data":1116192,"kb_used_omap":1,"kb_used_meta":111678,"kb_avail":93143952,"statfs":{"total":96636764160,"available":95379406848,"internally_reserved":0,"allocated":1142980608,"data_stored":1142305477,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":114358726},"hb_peers":[0,4,5,6,7,8,10,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":22,"seq":94489280555,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":806068,"kb_used_data":742948,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":93565772,"statfs":{"total":96636764160,"available":95811350528,"internally_reserved":0,"allocated":760778752,"data_stored":760043385,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[3,5,6,7,9,10,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":22,"seq":94489280554,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":589264,"kb_used_data":535808,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":93782576,"statfs":{"total":96636764160,"available":96033357824,"internally_reserved":0,"allocated":548667392,"data_stored":547963100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54655430},"hb_peers":[0,1,3,5,7,8,10,11,12,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":22,"seq":94489280554,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1467536,"kb_used_data":1364160,"kb_used_omap":1,"kb_used_meta":103358,"kb_avail":92904304,"statfs":{"total":96636764160,"available":95134007296,"internally_reserved":0,"allocated":1396899840,"data_stored":1396192229,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":105839050},"hb_peers":[0,3,5,6,8,9,10,11,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":22,"seq":94489280554,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":566340,"kb_used_data":494900,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":93805500,"statfs":{"total":96636764160,"available":96056832000,"internally_reserved":0,"allocated":506777600,"data_stored":506022479,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73136581},"hb_peers":[1,5,6,7,9,10,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":22,"seq":94489280553,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1018012,"kb_used_data":946572,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":93353828,"statfs":{"total":96636764160,"available":95594319872,"internally_reserved":0,"allocated":969289728,"data_stored":968573918,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":73136576},"hb_peers":[0,1,3,5,7,9,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":11,"up_from":22,"seq":94489280555,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1459852,"kb_used_data":1348156,"kb_used_omap":1,"kb_used_meta":111678,"kb_avail":92911988,"statfs":{"total":96636764160,"available":95141875712,"internally_reserved":0,"allocated":1380511744,"data_stored":1379763476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":114358724},"hb_peers":[0,1,2,4,7,8,10,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1,0,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":12,"up_from":22,"seq":94489280554,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1280248,"kb_used_data":1168552,"kb_used_omap":1,"kb_used_meta":111678,"kb_avail":93091592,"statfs":{"total":96636764160,"available":95325790208,"internally_reserved":0,"allocated":1196597248,"data_stored":1195854952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":114358725},"hb_peers":[0,1,2,3,4,5,10,11,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":14,"up_from":0,"seq":609885356040,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1226309632,"data_stored":1226309632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1082654720,"data_stored":1082654720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":534515712,"data_stored":534511640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1077346304,"data_stored":1077346304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":752025600,"data_stored":752025600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1200947200,"data_stored":1200947200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":542900224,"data_stored":542900224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1394081792,"data_stored":1394081792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":504627200,"data_stored":504627200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":629866496,"data_stored":629866496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":955252736,"data_stored":955252736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":1378156544,"data_stored":1378156544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":1183645696,"data_stored":1183645696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":750125056,"data_stored":750125056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":951386112,"data_stored":951386112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":43282,"internal_metadata":0},{"poolid":3,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":1140002816,"data_stored":1139998744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T11:25:31.017 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-13T11:25:31.029 INFO:teuthology.orchestra.run.smithi001.stdout:osd.7: {} 2024-09-13T11:25:31.030 INFO:teuthology.orchestra.run.smithi001.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T11:25:31.069 INFO:teuthology.orchestra.run.smithi001.stdout:osd.8: {} 2024-09-13T11:25:31.069 INFO:teuthology.orchestra.run.smithi001.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-13T11:25:31.073 INFO:tasks.ceph.osd.10.smithi162.stderr:2024-09-13T11:25:31.068+0000 7fb59c9f0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26317) UID: 0 2024-09-13T11:25:31.121 INFO:teuthology.orchestra.run.smithi001.stdout:osd.9: {} 2024-09-13T11:25:31.121 INFO:teuthology.orchestra.run.smithi001.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-13T11:25:31.160 INFO:teuthology.orchestra.run.smithi001.stdout:osd.10: {} 2024-09-13T11:25:31.161 INFO:teuthology.orchestra.run.smithi001.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-13T11:25:31.167 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-13T11:25:31.167 INFO:teuthology.orchestra.run.smithi001.stdout: "ops": [], 2024-09-13T11:25:31.167 INFO:teuthology.orchestra.run.smithi001.stdout: "complaint_time": 30, 2024-09-13T11:25:31.167 INFO:teuthology.orchestra.run.smithi001.stdout: "num_blocked_ops": 0 2024-09-13T11:25:31.167 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-13T11:25:31.173 INFO:tasks.ceph.osd.3.smithi177.stderr:2024-09-13T11:25:31.170+0000 7f10adba4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26574) UID: 0 2024-09-13T11:25:31.189 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-13T11:25:31.196 INFO:teuthology.orchestra.run.smithi001.stdout:osd.11: {} 2024-09-13T11:25:31.196 INFO:teuthology.orchestra.run.smithi001.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-13T11:25:31.226 INFO:teuthology.orchestra.run.smithi001.stdout:osd.12: {} 2024-09-13T11:25:31.227 INFO:teuthology.orchestra.run.smithi001.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-13T11:25:31.263 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-13T11:25:31.264 INFO:teuthology.orchestra.run.smithi001.stdout:osd.13: {} 2024-09-13T11:25:31.265 INFO:teuthology.orchestra.run.smithi001.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-13T11:25:31.274 INFO:tasks.ceph.osd.1.smithi092.stderr:2024-09-13T11:25:31.269+0000 7fe7789c5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33718) UID: 0 2024-09-13T11:25:31.317 INFO:teuthology.orchestra.run.smithi001.stdout:osd.14: {} 2024-09-13T11:25:31.317 INFO:teuthology.orchestra.run.smithi001.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-13T11:25:31.353 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-13T11:25:31.353 INFO:teuthology.orchestra.run.smithi001.stdout: "size": 20, 2024-09-13T11:25:31.353 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 600, 2024-09-13T11:25:31.353 INFO:teuthology.orchestra.run.smithi001.stdout: "ops": [ 2024-09-13T11:25:31.354 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:31.354 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDPGPush(3.11 72/70 [PushOp(3:8da9c977:::benchmark_data_smithi177_31116_object7736:head, version: 46'461, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:8da9c977:::benchmark_data_smithi177_31116_object7736:head@46'461, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:25:31.354 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:26.393216+0000", 2024-09-13T11:25:31.354 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 124.95816048499999, 2024-09-13T11:25:31.354 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.15683955499999999, 2024-09-13T11:25:31.354 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:31.354 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:31.354 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:25:31.354 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:31.354 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:31.354 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:31.354 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:26.393216+0000", 2024-09-13T11:25:31.354 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:31.354 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:31.355 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:31.355 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:31.355 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:26.393216+0000", 2024-09-13T11:25:31.355 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:31.355 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:31.355 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:31.355 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:31.355 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:26.393217+0000", 2024-09-13T11:25:31.355 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2410000000000001e-06 2024-09-13T11:25:31.355 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:31.355 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:31.355 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:31.355 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:26.393309+0000", 2024-09-13T11:25:31.355 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.2101000000000005e-05 2024-09-13T11:25:31.356 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:31.356 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:31.356 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:31.356 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:26.547936+0000", 2024-09-13T11:25:31.356 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.15462682699999999 2024-09-13T11:25:31.356 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:31.356 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:31.356 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:31.356 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:26.548021+0000", 2024-09-13T11:25:31.356 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.4529000000000003e-05 2024-09-13T11:25:31.356 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:31.356 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:31.356 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:31.357 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:26.548321+0000", 2024-09-13T11:25:31.357 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00029988399999999999 2024-09-13T11:25:31.357 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:31.357 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:31.357 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:25:31.357 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:26.548353+0000", 2024-09-13T11:25:31.357 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.2573000000000001e-05 2024-09-13T11:25:31.357 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:31.357 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:31.357 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:31.357 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:26.550056+0000", 2024-09-13T11:25:31.357 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0017024 2024-09-13T11:25:31.357 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:31.357 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:31.357 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:31.358 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:31.358 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:31.358 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDPGPush(3.1e 89/78 [PushOp(3:7a07f38c:::benchmark_data_smithi177_31116_object20966:head, version: 54'1402, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:7a07f38c:::benchmark_data_smithi177_31116_object20966:head@54'1402, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:25:31.358 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:46.645251+0000", 2024-09-13T11:25:31.358 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 104.70612577, 2024-09-13T11:25:31.358 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.17670883800000001, 2024-09-13T11:25:31.358 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:31.358 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:31.358 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:25:31.358 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:31.358 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:31.358 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:31.358 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.645251+0000", 2024-09-13T11:25:31.359 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:31.359 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:31.359 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:31.359 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:31.359 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.645251+0000", 2024-09-13T11:25:31.359 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:31.359 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:31.359 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:31.359 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:31.359 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.645253+0000", 2024-09-13T11:25:31.359 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.1399999999999998e-06 2024-09-13T11:25:31.359 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:31.359 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:31.359 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:31.360 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.645295+0000", 2024-09-13T11:25:31.360 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.2323000000000001e-05 2024-09-13T11:25:31.360 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:31.360 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:31.360 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:31.360 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.820731+0000", 2024-09-13T11:25:31.360 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.17543607999999999 2024-09-13T11:25:31.360 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:31.360 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:31.360 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:31.360 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.820750+0000", 2024-09-13T11:25:31.360 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.8793000000000001e-05 2024-09-13T11:25:31.360 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:31.361 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:31.361 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:31.361 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.820914+0000", 2024-09-13T11:25:31.361 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000164062 2024-09-13T11:25:31.361 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:31.361 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:31.361 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:25:31.361 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.820939+0000", 2024-09-13T11:25:31.361 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.4672e-05 2024-09-13T11:25:31.361 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:31.361 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:31.361 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:31.361 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.821960+0000", 2024-09-13T11:25:31.361 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.001020768 2024-09-13T11:25:31.362 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:31.362 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:31.362 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:31.362 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:31.362 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:31.362 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:71961 3.11 3:89bd9466:::benchmark_data_smithi177_31116_object71960:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2024-09-13T11:25:31.362 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:54.267475+0000", 2024-09-13T11:25:31.362 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 97.083901570999998, 2024-09-13T11:25:31.362 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.15027747899999999, 2024-09-13T11:25:31.362 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:31.362 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:31.362 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:31.362 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:25:31.362 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:25:31.363 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:31.363 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 71961 2024-09-13T11:25:31.363 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:31.363 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:31.363 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:31.363 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:31.363 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:54.267475+0000", 2024-09-13T11:25:31.363 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:31.363 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:31.363 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:31.363 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:31.363 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:54.267475+0000", 2024-09-13T11:25:31.363 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:31.364 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:31.364 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:31.364 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:31.364 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:54.267478+0000", 2024-09-13T11:25:31.364 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.8080000000000001e-06 2024-09-13T11:25:31.364 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:31.364 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:31.364 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:31.364 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:54.267534+0000", 2024-09-13T11:25:31.365 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.6276999999999997e-05 2024-09-13T11:25:31.365 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:31.365 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:31.365 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:31.365 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:54.267557+0000", 2024-09-13T11:25:31.365 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.2514000000000002e-05 2024-09-13T11:25:31.365 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:31.365 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:31.365 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:31.365 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:54.267570+0000", 2024-09-13T11:25:31.365 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.3251999999999999e-05 2024-09-13T11:25:31.365 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:31.365 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:31.366 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:31.366 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:54.267650+0000", 2024-09-13T11:25:31.366 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.0066000000000002e-05 2024-09-13T11:25:31.366 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:31.366 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:31.366 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:25:31.366 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:54.267886+0000", 2024-09-13T11:25:31.366 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00023637800000000001 2024-09-13T11:25:31.366 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:31.366 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:31.366 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 11", 2024-09-13T11:25:31.366 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:54.268020+0000", 2024-09-13T11:25:31.367 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00013392100000000001 2024-09-13T11:25:31.367 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:31.367 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:31.367 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:25:31.367 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:54.270076+0000", 2024-09-13T11:25:31.367 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.002055393 2024-09-13T11:25:31.367 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:31.367 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:31.367 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:25:31.367 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:54.417584+0000", 2024-09-13T11:25:31.367 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.147508008 2024-09-13T11:25:31.367 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:31.367 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:31.367 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:25:31.368 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:54.417667+0000", 2024-09-13T11:25:31.368 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.3666999999999997e-05 2024-09-13T11:25:31.368 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:31.368 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:31.368 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:31.368 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:54.417752+0000", 2024-09-13T11:25:31.368 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.5195e-05 2024-09-13T11:25:31.368 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:31.368 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:31.368 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:31.368 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:31.368 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:31.368 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4665.0:79776 3.1a e102/78 3:5d637d57:::benchmark_data_smithi177_31116_object79775:head v 102'4120, mlcod=72'3145)", 2024-09-13T11:25:31.368 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:24:00.354950+0000", 2024-09-13T11:25:31.369 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 90.996426227000001, 2024-09-13T11:25:31.369 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.14753330100000001, 2024-09-13T11:25:31.369 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:31.369 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:31.369 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:31.369 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:31.369 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:31.369 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:31.369 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:00.354950+0000", 2024-09-13T11:25:31.369 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:31.369 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:31.369 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:31.369 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:31.369 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:00.354950+0000", 2024-09-13T11:25:31.370 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:31.370 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:31.370 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:31.370 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:31.370 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:00.354960+0000", 2024-09-13T11:25:31.370 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.5219999999999996e-06 2024-09-13T11:25:31.370 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:31.370 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:31.370 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:31.370 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:00.355024+0000", 2024-09-13T11:25:31.370 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.4163000000000002e-05 2024-09-13T11:25:31.370 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:31.370 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:31.370 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:31.371 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:00.499446+0000", 2024-09-13T11:25:31.371 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.144422352 2024-09-13T11:25:31.371 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:31.371 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:31.371 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:31.371 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:00.499466+0000", 2024-09-13T11:25:31.371 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.9694000000000002e-05 2024-09-13T11:25:31.371 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:31.371 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:31.371 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:31.371 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:00.499605+0000", 2024-09-13T11:25:31.371 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00013935099999999999 2024-09-13T11:25:31.371 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:31.372 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:31.372 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:25:31.372 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:00.499655+0000", 2024-09-13T11:25:31.372 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.9787000000000003e-05 2024-09-13T11:25:31.372 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:31.372 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:31.372 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:25:31.372 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:00.502418+0000", 2024-09-13T11:25:31.372 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.002763012 2024-09-13T11:25:31.372 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:31.372 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:31.372 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:31.372 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:00.502484+0000", 2024-09-13T11:25:31.372 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.5419999999999994e-05 2024-09-13T11:25:31.373 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:31.373 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:31.373 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:31.373 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:31.373 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:31.373 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDPGPush(3.1a 102/78 [PushOp(3:5df50766:::benchmark_data_smithi177_31116_object50543:head, version: 72'3152, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:5df50766:::benchmark_data_smithi177_31116_object50543:head@72'3152, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:25:31.373 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:24:00.499311+0000", 2024-09-13T11:25:31.373 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 90.852065684999999, 2024-09-13T11:25:31.373 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.26146664800000002, 2024-09-13T11:25:31.373 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:31.373 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:31.373 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:25:31.373 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:31.373 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:31.374 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:31.374 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:00.499311+0000", 2024-09-13T11:25:31.374 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:31.374 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:31.374 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:31.374 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:31.374 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:00.499311+0000", 2024-09-13T11:25:31.374 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:31.374 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:31.374 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:31.374 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:31.374 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:00.499320+0000", 2024-09-13T11:25:31.374 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.6400000000000003e-06 2024-09-13T11:25:31.374 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:31.374 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:31.375 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:31.375 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:00.499384+0000", 2024-09-13T11:25:31.375 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.4554999999999994e-05 2024-09-13T11:25:31.375 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:31.375 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:31.375 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:31.375 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:00.759713+0000", 2024-09-13T11:25:31.375 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.260328742 2024-09-13T11:25:31.375 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:31.375 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:31.375 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:31.375 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:00.759742+0000", 2024-09-13T11:25:31.375 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.9298e-05 2024-09-13T11:25:31.375 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:31.376 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:31.376 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:31.376 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:00.759871+0000", 2024-09-13T11:25:31.376 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000128992 2024-09-13T11:25:31.376 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:31.376 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:31.376 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:25:31.376 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:00.759884+0000", 2024-09-13T11:25:31.376 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2687999999999999e-05 2024-09-13T11:25:31.376 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:31.376 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:31.376 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:31.376 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:00.760778+0000", 2024-09-13T11:25:31.376 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00089373299999999996 2024-09-13T11:25:31.377 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:31.377 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:31.377 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:31.377 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:31.377 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:31.377 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4665.0:82769 3.1e e105/102 3:7a4e88b5:::benchmark_data_smithi177_31116_object82768:head v 105'4381, mlcod=105'4379)", 2024-09-13T11:25:31.377 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:24:03.745047+0000", 2024-09-13T11:25:31.377 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 87.606329459999998, 2024-09-13T11:25:31.377 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.213806145, 2024-09-13T11:25:31.377 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:31.377 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:31.377 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:31.377 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:31.378 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:31.378 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:31.378 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:03.745047+0000", 2024-09-13T11:25:31.378 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:31.378 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:31.378 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:31.378 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:31.378 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:03.745047+0000", 2024-09-13T11:25:31.378 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:31.378 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:31.378 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:31.378 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:31.378 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:03.745049+0000", 2024-09-13T11:25:31.378 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.5710000000000001e-06 2024-09-13T11:25:31.378 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:31.379 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:31.379 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:31.379 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:03.745107+0000", 2024-09-13T11:25:31.379 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.8799999999999999e-05 2024-09-13T11:25:31.379 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:31.379 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:31.379 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:31.379 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:03.955145+0000", 2024-09-13T11:25:31.379 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.210037735 2024-09-13T11:25:31.379 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:31.379 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:31.379 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:31.380 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:03.955163+0000", 2024-09-13T11:25:31.380 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.7520000000000002e-05 2024-09-13T11:25:31.380 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:31.380 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:31.380 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:31.380 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:03.955473+0000", 2024-09-13T11:25:31.380 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00031014399999999999 2024-09-13T11:25:31.380 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:31.380 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:31.380 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:25:31.380 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:03.955538+0000", 2024-09-13T11:25:31.380 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.5059000000000002e-05 2024-09-13T11:25:31.380 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:31.380 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:31.381 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:25:31.381 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:03.958753+0000", 2024-09-13T11:25:31.381 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0032152769999999999 2024-09-13T11:25:31.381 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:31.381 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:31.381 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:31.381 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:03.958853+0000", 2024-09-13T11:25:31.381 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000100039 2024-09-13T11:25:31.381 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:31.381 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:31.381 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:31.381 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:31.381 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:31.381 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:98582 3.1a 3:59f9cd75:::benchmark_data_smithi177_31116_object98581:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113)", 2024-09-13T11:25:31.381 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:24:15.431226+0000", 2024-09-13T11:25:31.382 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 75.920150991, 2024-09-13T11:25:31.382 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.19820755800000001, 2024-09-13T11:25:31.382 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:31.382 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:31.382 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:31.382 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:25:31.382 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:25:31.382 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:31.382 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 98582 2024-09-13T11:25:31.382 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:31.382 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:31.382 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:31.382 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:31.382 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.431226+0000", 2024-09-13T11:25:31.383 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:31.383 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:31.383 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:31.383 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:31.383 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.431226+0000", 2024-09-13T11:25:31.383 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:31.383 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:31.383 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:31.383 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:31.383 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.431229+0000", 2024-09-13T11:25:31.383 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.7100000000000001e-06 2024-09-13T11:25:31.383 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:31.383 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:31.383 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:31.383 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.431334+0000", 2024-09-13T11:25:31.384 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00010506900000000001 2024-09-13T11:25:31.384 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:31.384 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:31.384 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:31.384 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.431364+0000", 2024-09-13T11:25:31.384 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.9607e-05 2024-09-13T11:25:31.384 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:31.384 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:31.384 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:31.384 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.431388+0000", 2024-09-13T11:25:31.384 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.3808999999999999e-05 2024-09-13T11:25:31.384 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:31.384 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:31.385 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:31.385 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.431483+0000", 2024-09-13T11:25:31.385 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.5402999999999994e-05 2024-09-13T11:25:31.385 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:31.385 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:31.385 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:25:31.385 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.431642+0000", 2024-09-13T11:25:31.385 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000158412 2024-09-13T11:25:31.385 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:31.385 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:31.385 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 9", 2024-09-13T11:25:31.385 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.431796+0000", 2024-09-13T11:25:31.385 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00015467899999999999 2024-09-13T11:25:31.386 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:31.386 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:31.386 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:25:31.386 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.433731+0000", 2024-09-13T11:25:31.386 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.001934945 2024-09-13T11:25:31.386 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:31.386 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:31.386 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:25:31.386 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.629245+0000", 2024-09-13T11:25:31.386 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.19551390199999999 2024-09-13T11:25:31.386 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:31.386 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:31.386 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:25:31.386 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.629346+0000", 2024-09-13T11:25:31.387 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000100869 2024-09-13T11:25:31.387 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:31.387 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:31.387 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:31.387 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.629433+0000", 2024-09-13T11:25:31.387 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.7152999999999997e-05 2024-09-13T11:25:31.387 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:31.387 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:31.387 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:31.387 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:31.387 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:31.387 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:99105 3.1a 3:5ac5090c:::benchmark_data_smithi177_31116_object99104:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113)", 2024-09-13T11:25:31.387 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:24:15.962466+0000", 2024-09-13T11:25:31.387 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 75.388910702000004, 2024-09-13T11:25:31.387 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.23832849, 2024-09-13T11:25:31.388 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:31.388 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:31.388 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:31.388 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:25:31.388 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:25:31.388 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:31.388 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 99105 2024-09-13T11:25:31.388 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:31.388 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:31.388 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:31.388 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:31.388 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.962466+0000", 2024-09-13T11:25:31.388 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:31.388 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:31.389 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:31.389 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:31.389 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.962466+0000", 2024-09-13T11:25:31.389 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:31.389 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:31.389 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:31.389 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:31.389 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.962471+0000", 2024-09-13T11:25:31.389 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.9570000000000001e-06 2024-09-13T11:25:31.389 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:31.389 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:31.389 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:31.389 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.962547+0000", 2024-09-13T11:25:31.389 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.5947999999999999e-05 2024-09-13T11:25:31.389 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:31.390 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:31.390 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:31.390 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.962576+0000", 2024-09-13T11:25:31.390 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.9547999999999999e-05 2024-09-13T11:25:31.390 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:31.390 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:31.390 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:31.390 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.962594+0000", 2024-09-13T11:25:31.390 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.8023e-05 2024-09-13T11:25:31.390 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:31.390 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:31.390 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:31.390 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.962770+0000", 2024-09-13T11:25:31.390 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00017551099999999999 2024-09-13T11:25:31.390 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:31.391 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:31.391 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:25:31.391 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.963114+0000", 2024-09-13T11:25:31.391 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00034440399999999998 2024-09-13T11:25:31.391 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:31.391 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:31.391 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 9", 2024-09-13T11:25:31.391 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.963351+0000", 2024-09-13T11:25:31.391 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000237209 2024-09-13T11:25:31.391 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:31.391 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:31.391 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:25:31.391 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.966345+0000", 2024-09-13T11:25:31.391 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0029933519999999999 2024-09-13T11:25:31.392 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:31.392 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:31.392 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:25:31.392 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:16.200724+0000", 2024-09-13T11:25:31.392 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.23437934599999999 2024-09-13T11:25:31.392 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:31.392 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:31.392 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:25:31.392 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:16.200761+0000", 2024-09-13T11:25:31.392 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.6406999999999999e-05 2024-09-13T11:25:31.392 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:31.392 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:31.392 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:31.393 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:16.200794+0000", 2024-09-13T11:25:31.393 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.3785000000000003e-05 2024-09-13T11:25:31.393 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:31.393 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:31.393 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:31.393 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:31.393 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:31.393 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4665.0:104331 3.1e e115/102 3:79890035:::benchmark_data_smithi177_31116_object104330:head v 115'5083, mlcod=115'5080)", 2024-09-13T11:25:31.393 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:24:20.242656+0000", 2024-09-13T11:25:31.393 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 71.108720610000006, 2024-09-13T11:25:31.393 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.16813846700000001, 2024-09-13T11:25:31.393 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:31.393 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:31.394 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:31.394 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:31.394 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:31.394 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:31.394 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:20.242656+0000", 2024-09-13T11:25:31.394 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:31.394 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:31.394 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:31.394 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:31.394 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:20.242656+0000", 2024-09-13T11:25:31.394 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:31.394 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:31.394 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:31.395 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:31.395 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:20.242657+0000", 2024-09-13T11:25:31.395 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.4750000000000001e-06 2024-09-13T11:25:31.395 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:31.395 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:31.395 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:31.395 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:20.242699+0000", 2024-09-13T11:25:31.395 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.193e-05 2024-09-13T11:25:31.395 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:31.395 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:31.395 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:31.395 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:20.408772+0000", 2024-09-13T11:25:31.395 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.16607291199999999 2024-09-13T11:25:31.396 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:31.396 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:31.396 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:31.396 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:20.408779+0000", 2024-09-13T11:25:31.396 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.3600000000000001e-06 2024-09-13T11:25:31.396 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:31.396 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:31.396 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:31.396 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:20.408840+0000", 2024-09-13T11:25:31.396 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.1834999999999993e-05 2024-09-13T11:25:31.396 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:31.396 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:31.396 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:25:31.396 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:20.408865+0000", 2024-09-13T11:25:31.397 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.4294000000000002e-05 2024-09-13T11:25:31.397 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:31.397 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:31.397 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:25:31.397 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:20.410751+0000", 2024-09-13T11:25:31.397 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.001885811 2024-09-13T11:25:31.397 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:31.397 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:31.397 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:31.397 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:20.410794+0000", 2024-09-13T11:25:31.397 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.3850000000000002e-05 2024-09-13T11:25:31.397 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:31.398 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:31.398 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:31.398 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:31.398 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:31.398 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:129921 3.1a 3:5fb9088b:::benchmark_data_smithi177_31116_object129920:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e134)", 2024-09-13T11:25:31.398 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:24:47.177556+0000", 2024-09-13T11:25:31.398 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 44.173820423000002, 2024-09-13T11:25:31.398 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.33804469999999998, 2024-09-13T11:25:31.398 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:31.398 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:31.398 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:31.398 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:25:31.398 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:25:31.399 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:31.399 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 129921 2024-09-13T11:25:31.399 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:31.399 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:31.399 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:31.399 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:31.399 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:47.177556+0000", 2024-09-13T11:25:31.399 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:31.399 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:31.399 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:31.399 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:31.399 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:47.177556+0000", 2024-09-13T11:25:31.399 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:31.399 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:31.400 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:31.400 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:31.400 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:47.177559+0000", 2024-09-13T11:25:31.400 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.0790000000000001e-06 2024-09-13T11:25:31.400 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:31.400 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:31.400 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:31.400 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:47.177589+0000", 2024-09-13T11:25:31.400 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.0173999999999999e-05 2024-09-13T11:25:31.400 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:31.400 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:31.400 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:31.400 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:47.177606+0000", 2024-09-13T11:25:31.401 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.6475999999999999e-05 2024-09-13T11:25:31.401 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:31.401 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:31.401 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:31.401 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:47.177615+0000", 2024-09-13T11:25:31.401 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.5729999999999997e-06 2024-09-13T11:25:31.401 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:31.401 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:31.401 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:31.401 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:47.177680+0000", 2024-09-13T11:25:31.401 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.4349000000000003e-05 2024-09-13T11:25:31.401 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:31.401 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:31.402 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:25:31.402 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:47.177794+0000", 2024-09-13T11:25:31.402 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000114119 2024-09-13T11:25:31.402 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:31.402 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:31.402 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 9", 2024-09-13T11:25:31.402 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:47.177895+0000", 2024-09-13T11:25:31.402 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000101078 2024-09-13T11:25:31.402 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:31.402 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:31.402 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:25:31.402 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:47.179634+0000", 2024-09-13T11:25:31.402 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.001738606 2024-09-13T11:25:31.402 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:31.403 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:31.403 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:25:31.403 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:47.515495+0000", 2024-09-13T11:25:31.403 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.33586143000000002 2024-09-13T11:25:31.403 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:31.403 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:31.403 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:25:31.403 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:47.515551+0000", 2024-09-13T11:25:31.403 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.6048999999999998e-05 2024-09-13T11:25:31.403 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:31.403 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:31.403 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:31.403 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:47.515601+0000", 2024-09-13T11:25:31.403 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.9767e-05 2024-09-13T11:25:31.403 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:31.404 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:31.404 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:31.404 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:31.404 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:31.404 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4665.0:140742 3.1e e144/102 3:79c694a0:::benchmark_data_smithi177_31116_object439:head v 144'6204, mlcod=144'6200)", 2024-09-13T11:25:31.404 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:24:58.321534+0000", 2024-09-13T11:25:31.404 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 33.029842899000002, 2024-09-13T11:25:31.404 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.23629921500000001, 2024-09-13T11:25:31.404 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:31.404 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:31.404 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:31.404 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:31.404 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:31.405 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:31.405 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.321534+0000", 2024-09-13T11:25:31.405 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:31.405 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:31.405 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:31.405 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:31.405 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.321534+0000", 2024-09-13T11:25:31.405 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:31.405 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:31.405 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:31.405 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:31.405 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.321535+0000", 2024-09-13T11:25:31.405 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.159e-06 2024-09-13T11:25:31.405 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:31.406 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:31.406 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:31.406 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.321545+0000", 2024-09-13T11:25:31.406 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.9219999999999992e-06 2024-09-13T11:25:31.406 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:31.406 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:31.406 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:31.406 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.553719+0000", 2024-09-13T11:25:31.406 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.23217401700000001 2024-09-13T11:25:31.406 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:31.406 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:31.406 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:31.406 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.553734+0000", 2024-09-13T11:25:31.407 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.5344000000000001e-05 2024-09-13T11:25:31.407 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:31.407 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:31.407 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:31.407 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.553858+0000", 2024-09-13T11:25:31.407 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00012357500000000001 2024-09-13T11:25:31.407 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:31.407 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:31.407 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:25:31.407 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.553903+0000", 2024-09-13T11:25:31.407 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.5374999999999998e-05 2024-09-13T11:25:31.407 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:31.407 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:31.407 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:25:31.408 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.557719+0000", 2024-09-13T11:25:31.408 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0038158670000000001 2024-09-13T11:25:31.408 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:31.408 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:31.408 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:31.408 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.557833+0000", 2024-09-13T11:25:31.408 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000113956 2024-09-13T11:25:31.408 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:31.408 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:31.408 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:31.408 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:31.408 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:31.408 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4665.0:140796 3.1e e144/102 3:7987a652:::benchmark_data_smithi177_31116_object493:head v 144'6205, mlcod=144'6202)", 2024-09-13T11:25:31.409 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:24:58.553657+0000", 2024-09-13T11:25:31.409 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 32.797719923000002, 2024-09-13T11:25:31.409 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.25308323100000002, 2024-09-13T11:25:31.409 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:31.409 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:31.409 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:31.409 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:31.409 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:31.409 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:31.409 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.553657+0000", 2024-09-13T11:25:31.409 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:31.409 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:31.409 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:31.409 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:31.409 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.553657+0000", 2024-09-13T11:25:31.410 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:31.410 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:31.410 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:31.410 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:31.410 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.553660+0000", 2024-09-13T11:25:31.410 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.7519999999999998e-06 2024-09-13T11:25:31.410 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:31.410 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:31.410 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:31.410 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.553678+0000", 2024-09-13T11:25:31.410 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.7156e-05 2024-09-13T11:25:31.410 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:31.410 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:31.411 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:31.411 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.804342+0000", 2024-09-13T11:25:31.411 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.25066436199999997 2024-09-13T11:25:31.411 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:31.411 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:31.411 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:31.411 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.804353+0000", 2024-09-13T11:25:31.411 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.1147e-05 2024-09-13T11:25:31.411 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:31.411 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:31.411 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:31.411 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.804433+0000", 2024-09-13T11:25:31.412 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.9586999999999995e-05 2024-09-13T11:25:31.412 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:31.412 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:31.412 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:25:31.412 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.804457+0000", 2024-09-13T11:25:31.412 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.4108e-05 2024-09-13T11:25:31.412 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:31.412 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:31.412 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:25:31.412 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.806684+0000", 2024-09-13T11:25:31.412 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0022271309999999998 2024-09-13T11:25:31.412 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:31.412 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:31.413 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:31.413 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.806740+0000", 2024-09-13T11:25:31.413 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.5988e-05 2024-09-13T11:25:31.413 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:31.413 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:31.413 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:31.413 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:31.413 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:31.413 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:149955 3.11 3:8fa29838:::benchmark_data_smithi177_31116_object9652:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e149)", 2024-09-13T11:25:31.413 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:25:05.905492+0000", 2024-09-13T11:25:31.413 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 25.445884548999999, 2024-09-13T11:25:31.413 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.204181955, 2024-09-13T11:25:31.413 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:31.414 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:31.414 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:31.414 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:25:31.414 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:25:31.414 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:31.414 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 149955 2024-09-13T11:25:31.414 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:31.414 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:31.414 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:31.414 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:31.414 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:05.905492+0000", 2024-09-13T11:25:31.414 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:31.414 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:31.414 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:31.415 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:31.415 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:05.905492+0000", 2024-09-13T11:25:31.415 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:31.415 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:31.415 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:31.415 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:31.415 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:05.905494+0000", 2024-09-13T11:25:31.415 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.9369999999999998e-06 2024-09-13T11:25:31.415 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:31.415 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:31.415 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:31.415 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:05.905500+0000", 2024-09-13T11:25:31.415 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.7039999999999999e-06 2024-09-13T11:25:31.416 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:31.416 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:31.416 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:31.416 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:05.905512+0000", 2024-09-13T11:25:31.416 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.257e-05 2024-09-13T11:25:31.416 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:31.416 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:31.416 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:31.416 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:05.905519+0000", 2024-09-13T11:25:31.416 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.5420000000000002e-06 2024-09-13T11:25:31.416 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:31.416 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:31.416 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:31.416 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:05.905584+0000", 2024-09-13T11:25:31.417 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.4819000000000004e-05 2024-09-13T11:25:31.417 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:31.417 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:31.417 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:25:31.417 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:05.905696+0000", 2024-09-13T11:25:31.417 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00011248599999999999 2024-09-13T11:25:31.417 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:31.417 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:31.417 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 11", 2024-09-13T11:25:31.417 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:05.905759+0000", 2024-09-13T11:25:31.417 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.2451000000000002e-05 2024-09-13T11:25:31.417 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:31.417 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:31.417 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:25:31.418 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:05.907584+0000", 2024-09-13T11:25:31.418 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0018250110000000001 2024-09-13T11:25:31.418 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:31.418 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:31.418 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:25:31.418 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:06.109588+0000", 2024-09-13T11:25:31.418 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.20200470300000001 2024-09-13T11:25:31.418 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:31.418 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:31.418 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:25:31.418 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:06.109637+0000", 2024-09-13T11:25:31.418 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.8439000000000002e-05 2024-09-13T11:25:31.418 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:31.419 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:31.419 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:31.419 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:06.109674+0000", 2024-09-13T11:25:31.419 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.7292999999999997e-05 2024-09-13T11:25:31.419 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:31.419 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:31.419 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:31.419 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:31.419 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:31.419 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:150414 3.11 3:8f72eed4:::benchmark_data_smithi177_31116_object10111:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e149)", 2024-09-13T11:25:31.419 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:25:06.348925+0000", 2024-09-13T11:25:31.419 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 25.002451437000001, 2024-09-13T11:25:31.419 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.17250813300000001, 2024-09-13T11:25:31.420 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:31.420 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:31.420 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:31.420 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:25:31.420 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:25:31.420 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:31.420 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 150414 2024-09-13T11:25:31.420 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:31.420 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:31.420 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:31.420 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:31.420 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:06.348925+0000", 2024-09-13T11:25:31.420 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:31.420 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:31.420 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:31.421 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:31.421 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:06.348925+0000", 2024-09-13T11:25:31.421 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:31.421 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:31.421 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:31.421 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:31.421 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:06.348927+0000", 2024-09-13T11:25:31.421 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.7430000000000001e-06 2024-09-13T11:25:31.421 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:31.421 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:31.421 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:31.421 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:06.348931+0000", 2024-09-13T11:25:31.422 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.5210000000000004e-06 2024-09-13T11:25:31.422 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:31.422 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:31.422 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:31.422 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:06.348943+0000", 2024-09-13T11:25:31.422 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.1168e-05 2024-09-13T11:25:31.422 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:31.422 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:31.422 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:31.422 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:06.348950+0000", 2024-09-13T11:25:31.422 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.1430000000000001e-06 2024-09-13T11:25:31.422 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:31.422 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:31.423 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:31.423 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:06.349032+0000", 2024-09-13T11:25:31.423 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.2517999999999998e-05 2024-09-13T11:25:31.423 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:31.423 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:31.423 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:25:31.423 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:06.349179+0000", 2024-09-13T11:25:31.423 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000146935 2024-09-13T11:25:31.423 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:31.423 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:31.423 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 11", 2024-09-13T11:25:31.423 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:06.349290+0000", 2024-09-13T11:25:31.423 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000110927 2024-09-13T11:25:31.423 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:31.424 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:31.424 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:25:31.424 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:06.351686+0000", 2024-09-13T11:25:31.424 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0023963560000000001 2024-09-13T11:25:31.424 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:31.424 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:31.424 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:25:31.424 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:06.521374+0000", 2024-09-13T11:25:31.424 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.169687216 2024-09-13T11:25:31.424 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:31.424 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:31.424 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:25:31.425 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:06.521406+0000", 2024-09-13T11:25:31.425 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.1841000000000002e-05 2024-09-13T11:25:31.425 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:31.425 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:31.425 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:31.425 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:06.521433+0000", 2024-09-13T11:25:31.425 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.7764999999999999e-05 2024-09-13T11:25:31.425 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:31.425 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:31.425 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:31.425 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:31.425 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:31.426 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDPGPush(3.29 151/120 [PushOp(3:97da7246:::benchmark_data_smithi177_31116_object96568:head, version: 113'4425, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:97da7246:::benchmark_data_smithi177_31116_object96568:head@113'4425, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:25:31.426 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:25:09.375401+0000", 2024-09-13T11:25:31.426 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 21.975975158000001, 2024-09-13T11:25:31.426 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.15027211300000001, 2024-09-13T11:25:31.426 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:31.426 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:31.426 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:25:31.426 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:31.426 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:31.426 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:31.426 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:09.375401+0000", 2024-09-13T11:25:31.426 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:31.426 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:31.426 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:31.427 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:31.427 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:09.375401+0000", 2024-09-13T11:25:31.427 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:31.427 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:31.427 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:31.427 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:31.427 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:09.375402+0000", 2024-09-13T11:25:31.427 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.049e-06 2024-09-13T11:25:31.427 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:31.427 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:31.427 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:31.427 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:09.375505+0000", 2024-09-13T11:25:31.427 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00010225000000000001 2024-09-13T11:25:31.428 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:31.428 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:31.428 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:31.428 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:09.524742+0000", 2024-09-13T11:25:31.428 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.14923771899999999 2024-09-13T11:25:31.428 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:31.428 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:31.428 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:31.428 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:09.524761+0000", 2024-09-13T11:25:31.428 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.8377999999999999e-05 2024-09-13T11:25:31.428 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:31.428 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:31.429 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:31.429 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:09.524842+0000", 2024-09-13T11:25:31.429 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.0841999999999996e-05 2024-09-13T11:25:31.429 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:31.429 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:31.429 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:25:31.429 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:09.524850+0000", 2024-09-13T11:25:31.429 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.8709999999999997e-06 2024-09-13T11:25:31.429 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:31.429 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:31.429 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:31.429 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:09.525674+0000", 2024-09-13T11:25:31.429 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00082400400000000003 2024-09-13T11:25:31.430 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:31.430 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:31.430 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:31.430 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:31.430 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:31.430 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:163098 3.1a 3:58bec031:::benchmark_data_smithi177_31116_object22795:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e157)", 2024-09-13T11:25:31.430 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:25:16.139554+0000", 2024-09-13T11:25:31.430 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 15.211822589000001, 2024-09-13T11:25:31.430 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.14248129000000001, 2024-09-13T11:25:31.430 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:31.430 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:31.430 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:31.430 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:25:31.430 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:25:31.431 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:31.431 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 163098 2024-09-13T11:25:31.431 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:31.431 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:31.431 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:31.431 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:31.431 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:16.139554+0000", 2024-09-13T11:25:31.431 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:31.431 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:31.431 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:31.431 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:31.431 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:16.139554+0000", 2024-09-13T11:25:31.431 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:31.431 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:31.431 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:31.432 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:31.432 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:16.139557+0000", 2024-09-13T11:25:31.432 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.4249999999999998e-06 2024-09-13T11:25:31.432 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:31.432 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:31.432 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:31.432 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:16.139566+0000", 2024-09-13T11:25:31.432 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.8659999999999995e-06 2024-09-13T11:25:31.432 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:31.432 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:31.432 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:31.432 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:16.139587+0000", 2024-09-13T11:25:31.432 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.1058999999999999e-05 2024-09-13T11:25:31.432 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:31.433 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:31.433 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:31.433 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:16.139597+0000", 2024-09-13T11:25:31.433 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.004e-05 2024-09-13T11:25:31.433 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:31.433 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:31.433 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:31.433 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:16.139668+0000", 2024-09-13T11:25:31.433 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.0257000000000002e-05 2024-09-13T11:25:31.433 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:31.433 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:31.433 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:25:31.433 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:16.139831+0000", 2024-09-13T11:25:31.434 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00016312999999999999 2024-09-13T11:25:31.434 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:31.434 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:31.434 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 9", 2024-09-13T11:25:31.434 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:16.139973+0000", 2024-09-13T11:25:31.434 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00014215199999999999 2024-09-13T11:25:31.434 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:31.434 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:31.434 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:25:31.434 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:16.142075+0000", 2024-09-13T11:25:31.434 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0021017760000000001 2024-09-13T11:25:31.434 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:31.434 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:31.434 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:25:31.435 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:16.281933+0000", 2024-09-13T11:25:31.435 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.13985831900000001 2024-09-13T11:25:31.435 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:31.435 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:31.435 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:25:31.435 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:16.281987+0000", 2024-09-13T11:25:31.435 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.3896e-05 2024-09-13T11:25:31.435 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:31.435 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:31.435 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:31.435 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:16.282035+0000", 2024-09-13T11:25:31.435 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.8369999999999999e-05 2024-09-13T11:25:31.435 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:31.436 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:31.436 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:31.436 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:31.436 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:31.436 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:168772 3.1a 3:5be647e9:::benchmark_data_smithi177_31116_object28469:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e161)", 2024-09-13T11:25:31.436 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:25:20.636095+0000", 2024-09-13T11:25:31.436 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 10.715282028000001, 2024-09-13T11:25:31.436 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.17968553000000001, 2024-09-13T11:25:31.436 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:31.436 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:31.436 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:31.436 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:25:31.436 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:25:31.436 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:31.437 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 168772 2024-09-13T11:25:31.437 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:31.437 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:31.437 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:31.437 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:31.437 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:20.636095+0000", 2024-09-13T11:25:31.437 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:31.437 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:31.437 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:31.437 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:31.437 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:20.636095+0000", 2024-09-13T11:25:31.437 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:31.437 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:31.438 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:31.438 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:31.438 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:20.636096+0000", 2024-09-13T11:25:31.438 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.852e-06 2024-09-13T11:25:31.438 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:31.438 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:31.438 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:31.438 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:20.636101+0000", 2024-09-13T11:25:31.438 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.0329999999999999e-06 2024-09-13T11:25:31.438 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:31.438 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:31.438 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:31.438 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:20.636114+0000", 2024-09-13T11:25:31.438 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2816999999999999e-05 2024-09-13T11:25:31.439 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:31.439 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:31.439 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:31.439 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:20.636120+0000", 2024-09-13T11:25:31.439 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.7429999999999999e-06 2024-09-13T11:25:31.439 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:31.439 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:31.439 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:31.439 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:20.636194+0000", 2024-09-13T11:25:31.439 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.3520000000000001e-05 2024-09-13T11:25:31.439 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:31.439 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:31.439 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:25:31.440 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:20.636369+0000", 2024-09-13T11:25:31.440 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00017552 2024-09-13T11:25:31.440 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:31.440 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:31.440 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 9", 2024-09-13T11:25:31.440 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:20.636519+0000", 2024-09-13T11:25:31.440 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000149949 2024-09-13T11:25:31.440 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:31.440 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:31.440 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:25:31.440 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:20.638735+0000", 2024-09-13T11:25:31.440 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0022156509999999999 2024-09-13T11:25:31.441 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:31.441 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:31.441 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:25:31.441 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:20.815699+0000", 2024-09-13T11:25:31.441 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.176964343 2024-09-13T11:25:31.441 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:31.441 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:31.441 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:25:31.441 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:20.815748+0000", 2024-09-13T11:25:31.441 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.9518000000000003e-05 2024-09-13T11:25:31.441 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:31.442 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:31.442 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:31.442 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:20.815780+0000", 2024-09-13T11:25:31.442 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.1584000000000001e-05 2024-09-13T11:25:31.442 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:31.442 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:31.442 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:31.442 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:31.442 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:31.442 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:168782 3.1a 3:590b82e1:::benchmark_data_smithi177_31116_object28479:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e161)", 2024-09-13T11:25:31.442 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:25:20.641567+0000", 2024-09-13T11:25:31.442 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 10.709809196, 2024-09-13T11:25:31.442 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.19100501, 2024-09-13T11:25:31.442 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:31.443 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:31.443 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:31.443 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:25:31.443 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:25:31.443 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:31.443 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 168782 2024-09-13T11:25:31.443 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:31.443 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:31.443 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:31.443 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:31.443 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:20.641567+0000", 2024-09-13T11:25:31.443 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:31.443 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:31.443 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:31.443 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:31.443 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:20.641567+0000", 2024-09-13T11:25:31.444 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:31.444 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:31.444 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:31.444 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:31.444 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:20.641569+0000", 2024-09-13T11:25:31.444 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.869e-06 2024-09-13T11:25:31.444 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:31.444 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:31.444 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:31.444 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:20.641574+0000", 2024-09-13T11:25:31.444 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.1680000000000003e-06 2024-09-13T11:25:31.444 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:31.444 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:31.444 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:31.444 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:20.641587+0000", 2024-09-13T11:25:31.444 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2858e-05 2024-09-13T11:25:31.445 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:31.445 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:31.445 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:31.445 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:20.641595+0000", 2024-09-13T11:25:31.445 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.6149999999999999e-06 2024-09-13T11:25:31.445 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:31.445 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:31.445 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:31.445 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:20.641655+0000", 2024-09-13T11:25:31.445 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.9673999999999998e-05 2024-09-13T11:25:31.445 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:31.445 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:31.445 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:25:31.445 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:20.641748+0000", 2024-09-13T11:25:31.445 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.3919999999999995e-05 2024-09-13T11:25:31.446 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:31.446 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:31.446 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 9", 2024-09-13T11:25:31.446 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:20.642990+0000", 2024-09-13T11:25:31.446 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0012415709999999999 2024-09-13T11:25:31.446 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:31.446 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:31.446 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:25:31.446 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:20.645016+0000", 2024-09-13T11:25:31.446 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.002025448 2024-09-13T11:25:31.446 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:31.446 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:31.446 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:25:31.446 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:20.832409+0000", 2024-09-13T11:25:31.446 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.18739319600000001 2024-09-13T11:25:31.447 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:31.447 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:31.447 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:25:31.447 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:20.832525+0000", 2024-09-13T11:25:31.447 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000116273 2024-09-13T11:25:31.447 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:31.447 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:31.447 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:31.447 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:20.832572+0000", 2024-09-13T11:25:31.447 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.7417999999999999e-05 2024-09-13T11:25:31.447 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:31.447 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:31.447 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:31.447 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:31.448 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:31.448 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop_reply(client.4665.0:168772 3.1a e161/105 ondisk, result = 0)", 2024-09-13T11:25:31.448 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:25:20.645700+0000", 2024-09-13T11:25:31.448 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 10.705677014999999, 2024-09-13T11:25:31.448 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.17008606200000001, 2024-09-13T11:25:31.448 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:31.448 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:31.448 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:25:31.448 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:31.448 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:31.448 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:31.448 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:20.645700+0000", 2024-09-13T11:25:31.448 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:31.448 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:31.449 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:31.449 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:31.449 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:20.645700+0000", 2024-09-13T11:25:31.449 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:31.449 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:31.449 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:31.449 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:31.449 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:20.645701+0000", 2024-09-13T11:25:31.449 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.474e-06 2024-09-13T11:25:31.449 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:31.449 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:31.449 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:31.449 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:20.645714+0000", 2024-09-13T11:25:31.450 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2931999999999999e-05 2024-09-13T11:25:31.450 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:31.450 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:31.450 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:31.450 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:20.815603+0000", 2024-09-13T11:25:31.450 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.16988872199999999 2024-09-13T11:25:31.450 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:31.450 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:31.450 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:31.450 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:20.815610+0000", 2024-09-13T11:25:31.450 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.2139999999999999e-06 2024-09-13T11:25:31.450 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:31.451 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:31.451 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:31.451 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:20.815689+0000", 2024-09-13T11:25:31.451 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.9302999999999996e-05 2024-09-13T11:25:31.451 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:31.451 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:31.451 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:25:31.451 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:20.815695+0000", 2024-09-13T11:25:31.451 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.818e-06 2024-09-13T11:25:31.451 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:31.451 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:31.451 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:31.451 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:20.815786+0000", 2024-09-13T11:25:31.451 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.0599000000000005e-05 2024-09-13T11:25:31.452 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:31.452 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:31.452 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:31.452 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:31.452 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:31.452 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:174971 3.1a 3:5a8f64c5:::benchmark_data_smithi177_31116_object34668:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e166)", 2024-09-13T11:25:31.452 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:25:25.422480+0000", 2024-09-13T11:25:31.452 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 5.9288965850000004, 2024-09-13T11:25:31.452 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.18399536399999999, 2024-09-13T11:25:31.452 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:31.452 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:31.452 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:31.452 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:25:31.453 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:25:31.453 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:31.453 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 174971 2024-09-13T11:25:31.453 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:31.453 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:31.453 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:31.453 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:31.453 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:25.422480+0000", 2024-09-13T11:25:31.453 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:31.453 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:31.453 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:31.453 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:31.453 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:25.422480+0000", 2024-09-13T11:25:31.454 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:31.454 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:31.454 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:31.454 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:31.454 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:25.422483+0000", 2024-09-13T11:25:31.454 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.4529999999999998e-06 2024-09-13T11:25:31.454 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:31.454 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:31.454 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:31.454 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:25.422492+0000", 2024-09-13T11:25:31.454 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.8820000000000007e-06 2024-09-13T11:25:31.454 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:31.454 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:31.454 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:31.455 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:25.422515+0000", 2024-09-13T11:25:31.455 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.2756e-05 2024-09-13T11:25:31.455 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:31.455 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:31.455 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:31.455 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:25.422526+0000", 2024-09-13T11:25:31.455 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0684e-05 2024-09-13T11:25:31.455 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:31.455 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:31.455 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:31.455 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:25.422590+0000", 2024-09-13T11:25:31.455 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.4163000000000002e-05 2024-09-13T11:25:31.455 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:31.456 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:31.456 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:25:31.456 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:25.422675+0000", 2024-09-13T11:25:31.456 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.5311999999999999e-05 2024-09-13T11:25:31.456 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:31.456 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:31.456 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 9", 2024-09-13T11:25:31.456 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:25.422745+0000", 2024-09-13T11:25:31.456 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.0217999999999998e-05 2024-09-13T11:25:31.456 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:31.456 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:31.456 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:25:31.456 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:25.424910+0000", 2024-09-13T11:25:31.456 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0021641500000000001 2024-09-13T11:25:31.456 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:31.457 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:31.457 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:25:31.457 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:25.606366+0000", 2024-09-13T11:25:31.457 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.181456267 2024-09-13T11:25:31.457 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:31.457 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:31.457 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:25:31.457 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:25.606428+0000", 2024-09-13T11:25:31.457 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.2312999999999998e-05 2024-09-13T11:25:31.457 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:31.457 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:31.457 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:31.457 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:25.606475+0000", 2024-09-13T11:25:31.457 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.7166000000000002e-05 2024-09-13T11:25:31.458 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:31.458 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:31.458 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:31.458 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:31.458 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:31.458 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-13T11:25:31.459 INFO:teuthology.orchestra.run.smithi001.stdout:osd.15: {} 2024-09-13T11:25:31.459 INFO:teuthology.orchestra.run.smithi001.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-13T11:25:31.460 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-09-13T11:25:31.461 INFO:tasks.ceph.osd.11.smithi177.stderr:2024-09-13T11:25:31.458+0000 7f24792c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26580) UID: 0 2024-09-13T11:25:31.543 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-13T11:25:31.543 INFO:teuthology.orchestra.run.smithi001.stdout:{"fsid":"2fd68ac7-5581-4d90-80a4-30bcf5252a7a","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noscrub,nodeep-scrub flag(s) set","count":20},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":214,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":170,"num_osds":16,"num_up_osds":16,"osd_up_since":1726226695,"num_in_osds":15,"osd_in_since":1726226668,"num_remapped_pgs":2},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":51},{"state_name":"active+recovering+remapped","count":3},{"state_name":"active+clean+scrubbing","count":1}],"num_pgs":55,"num_pools":3,"num_objects":104135,"data_bytes":6824821323,"bytes_used":15814107136,"bytes_avail":1433737355264,"bytes_total":1449551462400,"misplaced_objects":2396,"misplaced_total":208270,"misplaced_ratio":0.011504297306381139,"recovering_objects_per_sec":224,"recovering_bytes_per_sec":14701081,"recovering_keys_per_sec":0,"num_objects_recovered":1110,"num_bytes_recovered":72744960,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":1272},"fsmap":{"epoch":1,"btime":"2024-09-13T11:21:57:222574+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-13T11:24:06.142821+0000","services":{}},"progress_events":{}} 2024-09-13T11:25:31.560 INFO:tasks.ceph.osd.1.smithi092.stderr:2024-09-13T11:25:31.553+0000 7fe7789c5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33718) UID: 0 2024-09-13T11:25:31.576 INFO:tasks.ceph.ceph_manager.ceph:making progress, resetting timeout 2024-09-13T11:25:31.576 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-13T11:25:31.618 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-13T11:25:31.618 INFO:teuthology.orchestra.run.smithi001.stdout: "ops": [], 2024-09-13T11:25:31.618 INFO:teuthology.orchestra.run.smithi001.stdout: "num_ops": 0 2024-09-13T11:25:31.618 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-13T11:25:31.637 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-09-13T11:25:31.661 INFO:tasks.ceph.osd.1.smithi092.stderr:2024-09-13T11:25:31.653+0000 7fe7789c5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33718) UID: 0 2024-09-13T11:25:31.761 INFO:tasks.ceph.osd.15.smithi177.stderr:2024-09-13T11:25:31.758+0000 7f9d6ce77640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26578) UID: 0 2024-09-13T11:25:31.820 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-13T11:25:31.820 INFO:teuthology.orchestra.run.smithi001.stdout: "ops": [], 2024-09-13T11:25:31.820 INFO:teuthology.orchestra.run.smithi001.stdout: "complaint_time": 30, 2024-09-13T11:25:31.820 INFO:teuthology.orchestra.run.smithi001.stdout: "num_blocked_ops": 0 2024-09-13T11:25:31.820 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-13T11:25:31.837 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-09-13T11:25:31.862 INFO:tasks.ceph.osd.15.smithi177.stderr:2024-09-13T11:25:31.858+0000 7f9d6ce77640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26578) UID: 0 2024-09-13T11:25:31.963 INFO:tasks.ceph.osd.11.smithi177.stderr:2024-09-13T11:25:31.962+0000 7f24792c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26580) UID: 0 2024-09-13T11:25:31.979 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-13T11:25:31.980 INFO:teuthology.orchestra.run.smithi001.stderr:dumped all 2024-09-13T11:25:31.994 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-13T11:25:31.994 INFO:teuthology.orchestra.run.smithi001.stdout: "size": 20, 2024-09-13T11:25:31.994 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 600, 2024-09-13T11:25:31.994 INFO:teuthology.orchestra.run.smithi001.stdout: "ops": [ 2024-09-13T11:25:31.994 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:31.995 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:54656 3.1e 3.6a49163e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e75)", 2024-09-13T11:25:31.995 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:30.979993+0000", 2024-09-13T11:25:31.995 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 121.012212873, 2024-09-13T11:25:31.995 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.9269557150000001, 2024-09-13T11:25:31.995 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:31.995 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:31.995 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-13T11:25:31.995 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:25:31.995 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:25:31.995 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:31.995 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 54656 2024-09-13T11:25:31.995 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:31.995 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:31.996 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:31.996 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:31.996 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:30.979993+0000", 2024-09-13T11:25:31.996 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:31.996 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:31.996 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:31.996 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:31.996 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:30.979993+0000", 2024-09-13T11:25:31.996 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:31.996 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:31.996 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:31.996 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:31.996 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:30.979997+0000", 2024-09-13T11:25:31.997 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.0319999999999997e-06 2024-09-13T11:25:31.997 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:31.997 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:31.997 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:31.997 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:30.980057+0000", 2024-09-13T11:25:31.997 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.0341999999999999e-05 2024-09-13T11:25:31.997 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:31.997 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:31.997 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:31.997 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:30.980094+0000", 2024-09-13T11:25:31.997 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.6770000000000002e-05 2024-09-13T11:25:31.997 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:31.997 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:31.998 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:31.998 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:30.980113+0000", 2024-09-13T11:25:31.998 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.8791999999999999e-05 2024-09-13T11:25:31.998 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:31.998 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:31.998 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:31.998 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.097773+0000", 2024-09-13T11:25:31.998 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.117660379 2024-09-13T11:25:31.998 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:31.998 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:31.998 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for peered", 2024-09-13T11:25:31.998 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.097784+0000", 2024-09-13T11:25:31.998 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.1025e-05 2024-09-13T11:25:31.998 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:31.999 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:31.999 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:31.999 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.906937+0000", 2024-09-13T11:25:31.999 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.8091523789999999 2024-09-13T11:25:31.999 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:31.999 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:31.999 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:31.999 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.906949+0000", 2024-09-13T11:25:31.999 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.1996e-05 2024-09-13T11:25:31.999 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:31.999 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:31.999 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:31.999 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:31.999 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.000 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDPGPushReply(3.17 75/70 [PushReplyOp(3:e82800c8:::benchmark_data_smithi177_31116_object13909:head)])", 2024-09-13T11:25:32.000 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:31.632344+0000", 2024-09-13T11:25:32.000 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 120.359862018, 2024-09-13T11:25:32.000 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.28592102899999999, 2024-09-13T11:25:32.000 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:32.000 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:32.000 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-13T11:25:32.000 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:32.000 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.000 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:32.000 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.632344+0000", 2024-09-13T11:25:32.000 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:32.000 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.000 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.001 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:32.001 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.632344+0000", 2024-09-13T11:25:32.001 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:32.001 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.001 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.001 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:32.001 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.632346+0000", 2024-09-13T11:25:32.001 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.8819999999999999e-06 2024-09-13T11:25:32.001 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.001 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.001 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:32.001 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.632353+0000", 2024-09-13T11:25:32.001 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.5059999999999998e-06 2024-09-13T11:25:32.001 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.002 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.002 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:32.002 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.917361+0000", 2024-09-13T11:25:32.002 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.285008231 2024-09-13T11:25:32.002 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.002 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.002 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:32.002 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.917372+0000", 2024-09-13T11:25:32.002 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0764999999999999e-05 2024-09-13T11:25:32.002 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.002 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.002 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:32.002 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.917458+0000", 2024-09-13T11:25:32.003 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.6136000000000003e-05 2024-09-13T11:25:32.003 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.003 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.003 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:32.003 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.918265+0000", 2024-09-13T11:25:32.003 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00080650899999999996 2024-09-13T11:25:32.003 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:32.003 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:32.003 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:32.003 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.003 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.003 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:54720 3.1e 3.a944303e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-13T11:25:32.004 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:31.973315+0000", 2024-09-13T11:25:32.004 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 120.018890638, 2024-09-13T11:25:32.004 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.93366527200000005, 2024-09-13T11:25:32.004 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:32.004 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:32.004 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-13T11:25:32.004 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:25:32.004 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:25:32.004 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:32.004 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 54720 2024-09-13T11:25:32.004 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.004 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:32.004 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.004 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:32.005 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.973315+0000", 2024-09-13T11:25:32.005 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:32.005 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.005 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.005 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:32.005 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.973315+0000", 2024-09-13T11:25:32.005 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:32.005 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.005 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.005 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:32.005 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.973318+0000", 2024-09-13T11:25:32.005 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.5830000000000001e-06 2024-09-13T11:25:32.006 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.006 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.006 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:32.006 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.973465+0000", 2024-09-13T11:25:32.006 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000146816 2024-09-13T11:25:32.006 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.006 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.006 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:32.006 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.973478+0000", 2024-09-13T11:25:32.006 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2931e-05 2024-09-13T11:25:32.006 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.006 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.006 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:32.006 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.973483+0000", 2024-09-13T11:25:32.007 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.8969999999999996e-06 2024-09-13T11:25:32.007 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.007 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.007 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:32.007 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.973548+0000", 2024-09-13T11:25:32.007 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.4771999999999996e-05 2024-09-13T11:25:32.007 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.007 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.007 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for peered", 2024-09-13T11:25:32.007 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.973557+0000", 2024-09-13T11:25:32.007 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.0359999999999995e-06 2024-09-13T11:25:32.008 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.008 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.008 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:32.008 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.906974+0000", 2024-09-13T11:25:32.008 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.93341706000000002 2024-09-13T11:25:32.008 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.008 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.008 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:32.008 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.906980+0000", 2024-09-13T11:25:32.009 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.1770000000000003e-06 2024-09-13T11:25:32.009 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:32.009 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:32.009 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:32.009 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.009 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.009 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:54734 3.1e 3.d7f6f7e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-13T11:25:32.009 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:31.983342+0000", 2024-09-13T11:25:32.009 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 120.008863835, 2024-09-13T11:25:32.009 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.92367058599999996, 2024-09-13T11:25:32.009 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:32.009 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:32.009 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-13T11:25:32.009 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:25:32.010 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:25:32.010 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:32.010 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 54734 2024-09-13T11:25:32.010 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.010 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:32.010 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.010 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:32.010 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.983342+0000", 2024-09-13T11:25:32.010 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:32.010 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.011 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.011 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:32.011 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.983342+0000", 2024-09-13T11:25:32.011 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:32.011 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.011 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.011 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:32.011 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.983345+0000", 2024-09-13T11:25:32.011 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.6220000000000001e-06 2024-09-13T11:25:32.011 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.011 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.011 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:32.011 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.983438+0000", 2024-09-13T11:25:32.012 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.3517000000000006e-05 2024-09-13T11:25:32.012 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.012 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.012 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:32.012 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.983451+0000", 2024-09-13T11:25:32.012 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2527999999999999e-05 2024-09-13T11:25:32.012 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.012 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.012 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:32.012 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.983458+0000", 2024-09-13T11:25:32.012 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.6380000000000004e-06 2024-09-13T11:25:32.012 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.012 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.012 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:32.013 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.983521+0000", 2024-09-13T11:25:32.013 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.2729000000000002e-05 2024-09-13T11:25:32.013 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.013 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.013 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for peered", 2024-09-13T11:25:32.013 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.983548+0000", 2024-09-13T11:25:32.013 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.6537999999999999e-05 2024-09-13T11:25:32.013 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.013 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.013 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:32.013 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.907005+0000", 2024-09-13T11:25:32.013 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.923456991 2024-09-13T11:25:32.013 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.014 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.014 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:32.014 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.907013+0000", 2024-09-13T11:25:32.014 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.0229999999999992e-06 2024-09-13T11:25:32.014 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:32.014 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:32.014 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:32.014 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.014 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.014 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:54760 3.1e 3.8c32c13e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-13T11:25:32.014 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:31.999685+0000", 2024-09-13T11:25:32.014 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 119.992520603, 2024-09-13T11:25:32.015 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.90735639599999995, 2024-09-13T11:25:32.015 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:32.015 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:32.015 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-13T11:25:32.015 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:25:32.015 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:25:32.015 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:32.015 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 54760 2024-09-13T11:25:32.015 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.015 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:32.015 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.015 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:32.015 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.999685+0000", 2024-09-13T11:25:32.016 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:32.016 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.016 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.016 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:32.016 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.999685+0000", 2024-09-13T11:25:32.016 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:32.016 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.016 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.016 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:32.016 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.999690+0000", 2024-09-13T11:25:32.016 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.7879999999999997e-06 2024-09-13T11:25:32.016 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.017 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.017 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:32.017 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.999729+0000", 2024-09-13T11:25:32.017 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.9032000000000002e-05 2024-09-13T11:25:32.017 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.017 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.017 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:32.017 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.999741+0000", 2024-09-13T11:25:32.017 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2136999999999999e-05 2024-09-13T11:25:32.017 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.017 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.017 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:32.017 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.999748+0000", 2024-09-13T11:25:32.018 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.9120000000000001e-06 2024-09-13T11:25:32.018 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.018 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.018 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:32.018 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.999803+0000", 2024-09-13T11:25:32.018 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.5064e-05 2024-09-13T11:25:32.018 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.018 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.018 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for peered", 2024-09-13T11:25:32.018 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.999815+0000", 2024-09-13T11:25:32.018 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2041e-05 2024-09-13T11:25:32.018 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.018 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.018 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:32.018 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.907034+0000", 2024-09-13T11:25:32.019 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.90721858200000005 2024-09-13T11:25:32.019 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.019 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.019 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:32.019 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.907042+0000", 2024-09-13T11:25:32.019 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.8399999999999995e-06 2024-09-13T11:25:32.019 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:32.019 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:32.019 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:32.019 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.019 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.019 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:54768 3.1e 3.6dad499e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-13T11:25:32.019 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:32.003482+0000", 2024-09-13T11:25:32.020 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 119.988723953, 2024-09-13T11:25:32.020 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.90358804699999995, 2024-09-13T11:25:32.020 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:32.020 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:32.020 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-13T11:25:32.020 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:25:32.020 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:25:32.020 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:32.020 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 54768 2024-09-13T11:25:32.020 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.020 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:32.020 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.020 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:32.020 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.003482+0000", 2024-09-13T11:25:32.021 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:32.021 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.021 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.021 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:32.021 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.003482+0000", 2024-09-13T11:25:32.021 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:32.021 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.021 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.021 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:32.021 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.003486+0000", 2024-09-13T11:25:32.021 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.1030000000000004e-06 2024-09-13T11:25:32.021 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.021 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.021 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:32.022 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.003523+0000", 2024-09-13T11:25:32.022 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.6732e-05 2024-09-13T11:25:32.022 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.022 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.022 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:32.022 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.003535+0000", 2024-09-13T11:25:32.022 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2520999999999999e-05 2024-09-13T11:25:32.022 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.022 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.022 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:32.022 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.003541+0000", 2024-09-13T11:25:32.022 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.8690000000000001e-06 2024-09-13T11:25:32.022 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.022 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.023 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:32.023 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.003593+0000", 2024-09-13T11:25:32.023 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.1786000000000002e-05 2024-09-13T11:25:32.023 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.023 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.023 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for peered", 2024-09-13T11:25:32.023 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.003602+0000", 2024-09-13T11:25:32.023 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.9679999999999995e-06 2024-09-13T11:25:32.023 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.023 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.023 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:32.023 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.907063+0000", 2024-09-13T11:25:32.023 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.90346116600000004 2024-09-13T11:25:32.023 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.024 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.024 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:32.024 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.907070+0000", 2024-09-13T11:25:32.024 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.9020000000000002e-06 2024-09-13T11:25:32.024 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:32.024 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:32.024 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:32.024 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.024 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.024 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:54785 3.1e 3.d32f1ede (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-13T11:25:32.024 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:32.016238+0000", 2024-09-13T11:25:32.024 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 119.975967743, 2024-09-13T11:25:32.025 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.89086243399999998, 2024-09-13T11:25:32.025 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:32.025 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:32.025 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-13T11:25:32.025 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:25:32.025 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:25:32.025 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:32.025 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 54785 2024-09-13T11:25:32.025 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.025 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:32.025 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.025 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:32.025 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.016238+0000", 2024-09-13T11:25:32.025 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:32.026 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.026 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.026 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:32.026 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.016238+0000", 2024-09-13T11:25:32.026 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:32.026 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.026 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.026 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:32.026 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.016241+0000", 2024-09-13T11:25:32.026 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.6469999999999998e-06 2024-09-13T11:25:32.026 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.026 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.026 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:32.026 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.016271+0000", 2024-09-13T11:25:32.027 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.0505999999999999e-05 2024-09-13T11:25:32.027 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.027 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.027 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:32.027 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.016287+0000", 2024-09-13T11:25:32.027 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.6073999999999999e-05 2024-09-13T11:25:32.027 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.027 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.027 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:32.027 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.016295+0000", 2024-09-13T11:25:32.027 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.0590000000000003e-06 2024-09-13T11:25:32.027 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.027 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.027 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:32.028 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.016373+0000", 2024-09-13T11:25:32.028 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.7634999999999998e-05 2024-09-13T11:25:32.028 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.028 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.028 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for peered", 2024-09-13T11:25:32.028 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.016383+0000", 2024-09-13T11:25:32.028 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0246e-05 2024-09-13T11:25:32.028 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.028 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.029 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:32.029 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.907092+0000", 2024-09-13T11:25:32.029 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.89070849200000002 2024-09-13T11:25:32.029 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.029 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.029 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:32.029 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.907101+0000", 2024-09-13T11:25:32.029 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.7749999999999998e-06 2024-09-13T11:25:32.029 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:32.029 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:32.029 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:32.029 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.029 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.030 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:54811 3.1e 3.3754367e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-13T11:25:32.030 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:32.045424+0000", 2024-09-13T11:25:32.030 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 119.94678209200001, 2024-09-13T11:25:32.030 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.86170719399999995, 2024-09-13T11:25:32.030 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:32.030 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:32.030 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-13T11:25:32.030 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:25:32.030 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:25:32.030 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:32.030 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 54811 2024-09-13T11:25:32.030 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.030 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:32.030 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.031 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:32.031 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.045424+0000", 2024-09-13T11:25:32.031 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:32.031 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.031 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.031 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:32.031 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.045424+0000", 2024-09-13T11:25:32.031 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:32.031 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.031 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.031 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:32.031 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.045426+0000", 2024-09-13T11:25:32.032 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.0260000000000002e-06 2024-09-13T11:25:32.032 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.032 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.032 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:32.032 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.045556+0000", 2024-09-13T11:25:32.032 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00013003099999999999 2024-09-13T11:25:32.032 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.032 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.032 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:32.032 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.045568+0000", 2024-09-13T11:25:32.032 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.1727e-05 2024-09-13T11:25:32.032 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.032 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.033 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:32.033 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.045576+0000", 2024-09-13T11:25:32.033 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.6459999999999999e-06 2024-09-13T11:25:32.033 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.033 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.033 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:32.033 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.045650+0000", 2024-09-13T11:25:32.033 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.4000999999999998e-05 2024-09-13T11:25:32.033 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.033 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.033 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for peered", 2024-09-13T11:25:32.033 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.045658+0000", 2024-09-13T11:25:32.033 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.3869999999999997e-06 2024-09-13T11:25:32.034 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.034 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.034 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:32.034 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.907123+0000", 2024-09-13T11:25:32.034 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.86146508399999999 2024-09-13T11:25:32.034 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.034 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.034 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:32.034 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.907131+0000", 2024-09-13T11:25:32.034 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.2919999999999996e-06 2024-09-13T11:25:32.034 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:32.034 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:32.034 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:32.034 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.035 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.035 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:54817 3.1e 3.36dd2fde (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-13T11:25:32.035 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:32.050108+0000", 2024-09-13T11:25:32.035 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 119.9420982, 2024-09-13T11:25:32.035 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.85705549199999997, 2024-09-13T11:25:32.035 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:32.035 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:32.035 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-13T11:25:32.035 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:25:32.035 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:25:32.035 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:32.035 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 54817 2024-09-13T11:25:32.035 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.036 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:32.036 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.036 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:32.036 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.050108+0000", 2024-09-13T11:25:32.036 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:32.036 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.036 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.036 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:32.036 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.050108+0000", 2024-09-13T11:25:32.036 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:32.036 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.036 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.036 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:32.037 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.050110+0000", 2024-09-13T11:25:32.037 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.0209999999999998e-06 2024-09-13T11:25:32.037 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.037 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.037 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:32.037 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.050166+0000", 2024-09-13T11:25:32.037 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.6026000000000002e-05 2024-09-13T11:25:32.037 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.037 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.037 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:32.037 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.050180+0000", 2024-09-13T11:25:32.037 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.4307e-05 2024-09-13T11:25:32.037 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.037 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.038 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:32.038 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.050188+0000", 2024-09-13T11:25:32.038 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.5410000000000003e-06 2024-09-13T11:25:32.038 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.038 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.038 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:32.038 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.050260+0000", 2024-09-13T11:25:32.038 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.2157999999999996e-05 2024-09-13T11:25:32.038 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.038 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.038 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for peered", 2024-09-13T11:25:32.038 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.050267+0000", 2024-09-13T11:25:32.038 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.0190000000000001e-06 2024-09-13T11:25:32.038 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.039 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.039 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:32.039 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.907155+0000", 2024-09-13T11:25:32.039 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.85688824399999997 2024-09-13T11:25:32.039 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.039 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.039 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:32.039 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.907163+0000", 2024-09-13T11:25:32.039 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.1759999999999993e-06 2024-09-13T11:25:32.039 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:32.039 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:32.039 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:32.039 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.039 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.040 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:54818 3.1e 3.5cfcc69e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-13T11:25:32.040 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:32.051374+0000", 2024-09-13T11:25:32.040 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 119.940831784, 2024-09-13T11:25:32.040 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.85581781000000001, 2024-09-13T11:25:32.040 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:32.040 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:32.040 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-13T11:25:32.040 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:25:32.040 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:25:32.040 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:32.040 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 54818 2024-09-13T11:25:32.040 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.040 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:32.041 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.041 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:32.041 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.051374+0000", 2024-09-13T11:25:32.041 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:32.041 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.041 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.041 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:32.041 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.051374+0000", 2024-09-13T11:25:32.041 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:32.041 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.041 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.041 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:32.041 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.051375+0000", 2024-09-13T11:25:32.042 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.4440000000000001e-06 2024-09-13T11:25:32.042 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.042 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.042 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:32.042 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.051426+0000", 2024-09-13T11:25:32.042 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.0809000000000001e-05 2024-09-13T11:25:32.042 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.042 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.042 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:32.042 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.051441+0000", 2024-09-13T11:25:32.042 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.4460999999999999e-05 2024-09-13T11:25:32.042 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.042 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.042 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:32.042 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.051449+0000", 2024-09-13T11:25:32.043 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.4880000000000007e-06 2024-09-13T11:25:32.043 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.043 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.043 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:32.043 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.051499+0000", 2024-09-13T11:25:32.043 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.9691999999999998e-05 2024-09-13T11:25:32.043 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.043 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.043 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for peered", 2024-09-13T11:25:32.043 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.051505+0000", 2024-09-13T11:25:32.043 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.2029999999999997e-06 2024-09-13T11:25:32.043 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.043 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.044 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:32.044 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.907184+0000", 2024-09-13T11:25:32.044 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.85567906599999999 2024-09-13T11:25:32.044 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.044 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.044 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:32.044 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.907192+0000", 2024-09-13T11:25:32.044 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.6469999999999998e-06 2024-09-13T11:25:32.044 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:32.044 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:32.044 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:32.044 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.044 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.044 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:54828 3.1e 3.e920c11e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-13T11:25:32.045 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:32.063344+0000", 2024-09-13T11:25:32.045 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 119.928861357, 2024-09-13T11:25:32.045 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.84387899099999997, 2024-09-13T11:25:32.045 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:32.045 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:32.045 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-13T11:25:32.045 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:25:32.045 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:25:32.045 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:32.045 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 54828 2024-09-13T11:25:32.045 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.045 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:32.045 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.045 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:32.046 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.063344+0000", 2024-09-13T11:25:32.046 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:32.046 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.046 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.046 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:32.046 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.063344+0000", 2024-09-13T11:25:32.046 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:32.046 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.046 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.046 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:32.046 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.063347+0000", 2024-09-13T11:25:32.046 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.2299999999999998e-06 2024-09-13T11:25:32.046 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.047 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.047 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:32.047 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.063384+0000", 2024-09-13T11:25:32.047 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.7290999999999999e-05 2024-09-13T11:25:32.047 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.047 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.047 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:32.047 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.063397+0000", 2024-09-13T11:25:32.047 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2955999999999999e-05 2024-09-13T11:25:32.047 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.047 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.047 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:32.047 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.063404+0000", 2024-09-13T11:25:32.048 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.4889999999999998e-06 2024-09-13T11:25:32.048 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.048 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.048 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:32.048 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.063470+0000", 2024-09-13T11:25:32.048 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.5516000000000001e-05 2024-09-13T11:25:32.048 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.048 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.048 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for peered", 2024-09-13T11:25:32.048 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.063482+0000", 2024-09-13T11:25:32.048 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2065000000000001e-05 2024-09-13T11:25:32.048 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.048 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.048 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:32.049 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.907216+0000", 2024-09-13T11:25:32.049 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.84373388900000001 2024-09-13T11:25:32.049 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.049 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.049 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:32.049 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.907223+0000", 2024-09-13T11:25:32.049 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.5549999999999997e-06 2024-09-13T11:25:32.049 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:32.049 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:32.049 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:32.049 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.049 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.049 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:54829 3.1e 3.192f62be (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-13T11:25:32.050 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:32.065642+0000", 2024-09-13T11:25:32.050 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 119.926564289, 2024-09-13T11:25:32.050 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.84161483100000001, 2024-09-13T11:25:32.050 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:32.050 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:32.050 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-13T11:25:32.050 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:25:32.050 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:25:32.050 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:32.050 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 54829 2024-09-13T11:25:32.050 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.050 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:32.050 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.050 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:32.050 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.065642+0000", 2024-09-13T11:25:32.051 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:32.051 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.051 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.051 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:32.051 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.065642+0000", 2024-09-13T11:25:32.051 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:32.051 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.051 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.051 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:32.051 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.065644+0000", 2024-09-13T11:25:32.051 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.1780000000000002e-06 2024-09-13T11:25:32.051 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.051 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.051 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:32.052 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.065695+0000", 2024-09-13T11:25:32.052 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.1632000000000003e-05 2024-09-13T11:25:32.052 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.052 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.052 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:32.052 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.065711+0000", 2024-09-13T11:25:32.052 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.5522000000000001e-05 2024-09-13T11:25:32.052 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.052 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.052 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:32.052 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.065719+0000", 2024-09-13T11:25:32.052 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.3480000000000004e-06 2024-09-13T11:25:32.052 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.053 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.053 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:32.053 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.065785+0000", 2024-09-13T11:25:32.053 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.5627e-05 2024-09-13T11:25:32.053 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.053 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.053 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for peered", 2024-09-13T11:25:32.053 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.065797+0000", 2024-09-13T11:25:32.053 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.1766000000000001e-05 2024-09-13T11:25:32.053 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.053 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.053 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:32.053 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.907248+0000", 2024-09-13T11:25:32.053 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.84145150300000005 2024-09-13T11:25:32.054 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.054 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.054 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:32.054 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.907256+0000", 2024-09-13T11:25:32.054 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.2549999999999997e-06 2024-09-13T11:25:32.054 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:32.054 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:32.054 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:32.054 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.054 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.054 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:54830 3.1e 3.b31554de (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-13T11:25:32.054 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:32.068259+0000", 2024-09-13T11:25:32.054 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 119.923946421, 2024-09-13T11:25:32.054 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.83902489599999996, 2024-09-13T11:25:32.055 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:32.055 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:32.055 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-13T11:25:32.055 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:25:32.055 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:25:32.055 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:32.055 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 54830 2024-09-13T11:25:32.055 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.055 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:32.055 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.055 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:32.055 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.068259+0000", 2024-09-13T11:25:32.055 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:32.056 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.056 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.056 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:32.056 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.068259+0000", 2024-09-13T11:25:32.056 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:32.056 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.056 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.056 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:32.056 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.068262+0000", 2024-09-13T11:25:32.056 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.1799999999999999e-06 2024-09-13T11:25:32.056 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.056 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.056 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:32.057 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.068301+0000", 2024-09-13T11:25:32.057 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.9289999999999998e-05 2024-09-13T11:25:32.057 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.057 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.057 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:32.057 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.068314+0000", 2024-09-13T11:25:32.057 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.3502999999999999e-05 2024-09-13T11:25:32.057 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.057 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.057 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:32.057 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.068324+0000", 2024-09-13T11:25:32.057 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.1630000000000002e-06 2024-09-13T11:25:32.057 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.058 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.058 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:32.058 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.068375+0000", 2024-09-13T11:25:32.058 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.1805000000000003e-05 2024-09-13T11:25:32.058 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.058 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.058 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for peered", 2024-09-13T11:25:32.058 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.068382+0000", 2024-09-13T11:25:32.058 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.224e-06 2024-09-13T11:25:32.058 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.058 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.058 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:32.058 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.907277+0000", 2024-09-13T11:25:32.059 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.83889509500000004 2024-09-13T11:25:32.059 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.059 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.059 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:32.059 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.907284+0000", 2024-09-13T11:25:32.059 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.6359999999999994e-06 2024-09-13T11:25:32.059 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:32.059 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:32.059 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:32.059 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.059 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.059 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:54846 3.1e 3.742e049e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-13T11:25:32.059 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:32.153567+0000", 2024-09-13T11:25:32.059 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 119.83863919700001, 2024-09-13T11:25:32.060 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.75378642399999995, 2024-09-13T11:25:32.060 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:32.060 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:32.060 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-13T11:25:32.060 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:25:32.060 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:25:32.060 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:32.060 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 54846 2024-09-13T11:25:32.060 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.060 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:32.060 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.060 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:32.060 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.153567+0000", 2024-09-13T11:25:32.061 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:32.061 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.061 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.061 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:32.061 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.153567+0000", 2024-09-13T11:25:32.061 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:32.061 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.061 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.061 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:32.061 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.153569+0000", 2024-09-13T11:25:32.061 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.4660000000000002e-06 2024-09-13T11:25:32.061 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.061 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.061 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:32.062 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.153619+0000", 2024-09-13T11:25:32.062 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.9617000000000003e-05 2024-09-13T11:25:32.062 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.062 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.062 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:32.062 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.153634+0000", 2024-09-13T11:25:32.062 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.4904e-05 2024-09-13T11:25:32.062 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.062 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.062 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:32.062 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.153642+0000", 2024-09-13T11:25:32.062 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.5599999999999994e-06 2024-09-13T11:25:32.062 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.062 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.063 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:32.063 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.153721+0000", 2024-09-13T11:25:32.063 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.8603999999999995e-05 2024-09-13T11:25:32.063 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.063 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.063 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for peered", 2024-09-13T11:25:32.063 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.153730+0000", 2024-09-13T11:25:32.063 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.3840000000000003e-06 2024-09-13T11:25:32.063 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.063 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.063 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:32.063 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.907346+0000", 2024-09-13T11:25:32.063 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.75361563099999995 2024-09-13T11:25:32.063 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.063 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.064 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:32.064 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.907353+0000", 2024-09-13T11:25:32.064 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.2579999999999998e-06 2024-09-13T11:25:32.064 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:32.064 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:32.064 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:32.064 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.064 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.064 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:54888 3.1e 3.43e14ede (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-13T11:25:32.064 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:32.423064+0000", 2024-09-13T11:25:32.064 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 119.569141526, 2024-09-13T11:25:32.065 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.48433585299999998, 2024-09-13T11:25:32.065 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:32.065 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:32.065 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-13T11:25:32.065 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:25:32.065 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:25:32.065 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:32.065 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 54888 2024-09-13T11:25:32.065 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.065 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:32.065 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.065 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:32.065 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.423064+0000", 2024-09-13T11:25:32.066 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:32.066 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.066 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.066 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:32.066 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.423064+0000", 2024-09-13T11:25:32.066 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:32.066 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.066 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.066 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:32.066 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.423069+0000", 2024-09-13T11:25:32.066 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.7389999999999999e-06 2024-09-13T11:25:32.066 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.066 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.067 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:32.067 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.423115+0000", 2024-09-13T11:25:32.067 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.5847000000000003e-05 2024-09-13T11:25:32.067 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.067 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.067 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:32.067 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.423126+0000", 2024-09-13T11:25:32.067 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.1418e-05 2024-09-13T11:25:32.067 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.067 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.067 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:32.067 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.423132+0000", 2024-09-13T11:25:32.067 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.57e-06 2024-09-13T11:25:32.068 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.068 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.068 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:32.068 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.423191+0000", 2024-09-13T11:25:32.068 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.9250999999999999e-05 2024-09-13T11:25:32.068 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.068 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.068 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for peered", 2024-09-13T11:25:32.068 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.423198+0000", 2024-09-13T11:25:32.068 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.9179999999999997e-06 2024-09-13T11:25:32.068 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.068 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.068 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:32.069 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.907392+0000", 2024-09-13T11:25:32.069 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.48419437500000001 2024-09-13T11:25:32.069 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.069 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.069 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:32.069 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.907400+0000", 2024-09-13T11:25:32.069 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.7349999999999996e-06 2024-09-13T11:25:32.069 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:32.069 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:32.069 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:32.069 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.069 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.070 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4665.0:59977 3.21 e83/78 3:86b4d8f6:::benchmark_data_smithi177_31116_object59976:head v 83'3136, mlcod=0'0)", 2024-09-13T11:25:32.070 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:40.571147+0000", 2024-09-13T11:25:32.070 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 111.421059077, 2024-09-13T11:25:32.070 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.51907542799999995, 2024-09-13T11:25:32.070 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:32.070 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:32.070 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:32.070 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:32.070 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.070 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:32.070 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:40.571147+0000", 2024-09-13T11:25:32.070 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:32.070 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.071 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.071 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:32.071 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:40.571147+0000", 2024-09-13T11:25:32.071 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:32.071 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.071 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.071 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:32.071 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:40.571151+0000", 2024-09-13T11:25:32.071 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.5650000000000003e-06 2024-09-13T11:25:32.072 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.072 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.072 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:32.072 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:40.571204+0000", 2024-09-13T11:25:32.072 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.2720999999999999e-05 2024-09-13T11:25:32.072 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.072 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.072 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:32.072 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:41.088332+0000", 2024-09-13T11:25:32.072 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.51712822000000003 2024-09-13T11:25:32.072 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.072 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.073 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:32.073 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:41.088341+0000", 2024-09-13T11:25:32.073 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.3100000000000001e-06 2024-09-13T11:25:32.073 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.073 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.073 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:32.073 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:41.088444+0000", 2024-09-13T11:25:32.073 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000103708 2024-09-13T11:25:32.073 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.073 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.073 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:25:32.073 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:41.088475+0000", 2024-09-13T11:25:32.074 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.0497e-05 2024-09-13T11:25:32.074 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.074 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.074 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:25:32.074 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:41.090192+0000", 2024-09-13T11:25:32.074 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.001717288 2024-09-13T11:25:32.074 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.074 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.074 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:32.074 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:41.090222+0000", 2024-09-13T11:25:32.074 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.0119000000000001e-05 2024-09-13T11:25:32.074 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:32.074 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:32.075 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:32.075 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.075 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.075 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:110286 3.26 3.c41c0fa6 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e119)", 2024-09-13T11:25:32.075 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:24:25.654272+0000", 2024-09-13T11:25:32.075 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 66.337933384999999, 2024-09-13T11:25:32.075 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.43337235400000002, 2024-09-13T11:25:32.075 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:32.075 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:32.075 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-13T11:25:32.075 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:25:32.075 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:25:32.076 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:32.076 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 110286 2024-09-13T11:25:32.076 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.076 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:32.076 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.076 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:32.076 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:25.654272+0000", 2024-09-13T11:25:32.076 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:32.076 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.076 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.076 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:32.076 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:25.654272+0000", 2024-09-13T11:25:32.077 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:32.077 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.077 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.077 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:32.077 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:25.654275+0000", 2024-09-13T11:25:32.080 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.3199999999999998e-06 2024-09-13T11:25:32.080 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.080 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.080 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:32.080 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:25.654517+0000", 2024-09-13T11:25:32.080 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00024270199999999999 2024-09-13T11:25:32.081 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.081 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.081 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:32.081 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:25.654550+0000", 2024-09-13T11:25:32.081 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.2549000000000003e-05 2024-09-13T11:25:32.081 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.081 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.081 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:32.081 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:25.654566+0000", 2024-09-13T11:25:32.081 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.5517000000000001e-05 2024-09-13T11:25:32.081 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.081 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.081 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:32.082 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:25.654655+0000", 2024-09-13T11:25:32.082 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.9961000000000001e-05 2024-09-13T11:25:32.082 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.082 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.082 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for peered", 2024-09-13T11:25:32.082 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:25.654665+0000", 2024-09-13T11:25:32.082 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.0520000000000007e-06 2024-09-13T11:25:32.082 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.082 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.082 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:32.082 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:26.087629+0000", 2024-09-13T11:25:32.082 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.43296476900000003 2024-09-13T11:25:32.083 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.083 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.083 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:32.083 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:26.087645+0000", 2024-09-13T11:25:32.083 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.5483999999999999e-05 2024-09-13T11:25:32.083 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:32.083 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:32.083 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:32.083 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.083 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.083 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:177913 3.27 3:e5268e7a:::benchmark_data_smithi177_31116_object37610:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e168)", 2024-09-13T11:25:32.083 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:25:27.607015+0000", 2024-09-13T11:25:32.083 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 4.3851910350000001, 2024-09-13T11:25:32.084 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.301660555, 2024-09-13T11:25:32.084 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:32.084 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:32.084 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:32.084 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:25:32.084 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:25:32.084 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:32.084 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 177913 2024-09-13T11:25:32.084 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.084 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:32.084 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.084 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:32.084 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:27.607015+0000", 2024-09-13T11:25:32.085 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:32.085 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.085 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.085 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:32.085 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:27.607015+0000", 2024-09-13T11:25:32.085 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:32.085 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.085 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.085 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:32.085 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:27.607018+0000", 2024-09-13T11:25:32.085 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.7000000000000002e-06 2024-09-13T11:25:32.085 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.086 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.086 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:32.086 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:27.607029+0000", 2024-09-13T11:25:32.086 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0933e-05 2024-09-13T11:25:32.086 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.086 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.086 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:32.086 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:27.607062+0000", 2024-09-13T11:25:32.086 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.2786000000000001e-05 2024-09-13T11:25:32.086 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.086 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.086 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:32.087 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:27.607078+0000", 2024-09-13T11:25:32.087 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.6232999999999999e-05 2024-09-13T11:25:32.087 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.087 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.087 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:32.087 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:27.607204+0000", 2024-09-13T11:25:32.087 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000125458 2024-09-13T11:25:32.087 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.087 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.087 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:25:32.087 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:27.607544+0000", 2024-09-13T11:25:32.087 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00033990600000000001 2024-09-13T11:25:32.087 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.088 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.088 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 5,11", 2024-09-13T11:25:32.088 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:27.607882+0000", 2024-09-13T11:25:32.088 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00033788399999999999 2024-09-13T11:25:32.088 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.088 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.088 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:25:32.088 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:27.610120+0000", 2024-09-13T11:25:32.088 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0022379930000000002 2024-09-13T11:25:32.088 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.088 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.088 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:25:32.088 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:27.613170+0000", 2024-09-13T11:25:32.089 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.003050545 2024-09-13T11:25:32.089 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.089 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.089 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:25:32.089 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:27.908398+0000", 2024-09-13T11:25:32.089 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.29522760599999998 2024-09-13T11:25:32.089 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.089 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.089 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:25:32.089 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:27.908458+0000", 2024-09-13T11:25:32.089 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.9718e-05 2024-09-13T11:25:32.089 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.090 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.090 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:32.090 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:27.908675+0000", 2024-09-13T11:25:32.090 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00021779300000000001 2024-09-13T11:25:32.090 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:32.090 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:32.090 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:32.090 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.090 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.090 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop_reply(client.4665.0:177913 3.27 e168/120 ondisk, result = 0)", 2024-09-13T11:25:32.090 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:25:27.610053+0000", 2024-09-13T11:25:32.090 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 4.3821528760000001, 2024-09-13T11:25:32.090 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.29862817200000003, 2024-09-13T11:25:32.091 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:32.091 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:32.091 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:25:32.091 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:32.091 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.091 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:32.091 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:27.610053+0000", 2024-09-13T11:25:32.091 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:32.091 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.091 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.091 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:32.091 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:27.610053+0000", 2024-09-13T11:25:32.091 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:32.092 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.092 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.092 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:32.092 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:27.610056+0000", 2024-09-13T11:25:32.092 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.49e-06 2024-09-13T11:25:32.092 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.092 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.092 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:32.092 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:27.610067+0000", 2024-09-13T11:25:32.092 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0599e-05 2024-09-13T11:25:32.092 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.092 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.092 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:32.093 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:27.908314+0000", 2024-09-13T11:25:32.093 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.29824728700000003 2024-09-13T11:25:32.093 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.093 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.093 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:32.093 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:27.908320+0000", 2024-09-13T11:25:32.093 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.9100000000000002e-06 2024-09-13T11:25:32.093 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.093 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.093 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:32.093 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:27.908378+0000", 2024-09-13T11:25:32.093 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.7575999999999999e-05 2024-09-13T11:25:32.093 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.093 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.094 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:25:32.094 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:27.908390+0000", 2024-09-13T11:25:32.094 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.187e-05 2024-09-13T11:25:32.094 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.094 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.094 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:32.094 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:27.908681+0000", 2024-09-13T11:25:32.094 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00029144000000000002 2024-09-13T11:25:32.094 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:32.094 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:32.094 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:32.094 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.094 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.094 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDPGPushReply(3.27 168/120 [PushReplyOp(3:e535cc18:::benchmark_data_smithi177_31116_object103295:head)])", 2024-09-13T11:25:32.095 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:25:27.615014+0000", 2024-09-13T11:25:32.095 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 4.3771915420000003, 2024-09-13T11:25:32.095 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.31810488799999997, 2024-09-13T11:25:32.095 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:32.095 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:32.095 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-13T11:25:32.095 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:32.095 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.095 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:32.095 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:27.615014+0000", 2024-09-13T11:25:32.095 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:32.095 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.095 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.096 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:32.096 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:27.615014+0000", 2024-09-13T11:25:32.096 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:32.096 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.096 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.096 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:32.096 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:27.615017+0000", 2024-09-13T11:25:32.096 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.6199999999999999e-06 2024-09-13T11:25:32.096 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.096 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.096 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:32.096 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:27.615030+0000", 2024-09-13T11:25:32.096 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2833000000000001e-05 2024-09-13T11:25:32.096 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.097 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.097 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:32.097 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:27.932588+0000", 2024-09-13T11:25:32.097 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.31755857700000001 2024-09-13T11:25:32.097 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.097 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.097 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:32.097 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:27.932595+0000", 2024-09-13T11:25:32.097 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.2160000000000003e-06 2024-09-13T11:25:32.097 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.097 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.097 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:32.097 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:27.932658+0000", 2024-09-13T11:25:32.097 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.3618e-05 2024-09-13T11:25:32.098 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.098 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.098 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:32.098 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:27.933119+0000", 2024-09-13T11:25:32.098 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00046102399999999998 2024-09-13T11:25:32.098 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:32.098 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:32.098 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:32.098 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:32.098 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:32.098 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-13T11:25:32.100 INFO:tasks.ceph.osd.1.smithi092.stderr:2024-09-13T11:25:32.093+0000 7fe7789c5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33718) UID: 0 2024-09-13T11:25:32.100 INFO:teuthology.orchestra.run.smithi001.stdout:{"pg_ready":true,"pg_map":{"version":256,"stamp":"2024-09-13T11:25:30.247882+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":6824821323,"num_objects":104135,"num_object_clones":0,"num_object_copies":208270,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2396,"num_objects_unfound":0,"num_objects_dirty":104135,"num_whiteouts":0,"num_read":87,"num_read_kb":74,"num_write":92712,"num_write_kb":3637107,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21018,"num_bytes_recovered":1377304595,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":286064,"ondisk_log_size":286064,"up":110,"acting":110,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":122,"num_osds":15,"num_per_pool_osds":15,"num_per_pool_omap_osds":15,"kb":1415577600,"kb_used":15443464,"kb_used_data":14102872,"kb_used_omap":26,"kb_used_meta":1340133,"kb_avail":1400134136,"statfs":{"total":1449551462400,"available":1433737355264,"internally_reserved":0,"allocated":14441340928,"data_stored":14430460982,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":26763,"internal_metadata":1372297077},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,2,0,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-203161600,"num_objects":-3100,"num_object_clones":0,"num_object_copies":-6200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-742,"num_objects_unfound":0,"num_objects_dirty":-3100,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3100,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":371,"num_bytes_recovered":24313856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.085889"},"pg_stats":[{"pgid":"3.2d","version":"169'6213","reported_seq":10306,"reported_epoch":169,"state":"active+recovering+remapped","last_fresh":"2024-09-13T11:25:30.100854+0000","last_change":"2024-09-13T11:25:19.656321+0000","last_active":"2024-09-13T11:25:30.100854+0000","last_peered":"2024-09-13T11:25:30.100854+0000","last_clean":"2024-09-13T11:24:24.974950+0000","last_became_active":"2024-09-13T11:24:27.227288+0000","last_became_peered":"2024-09-13T11:24:27.227288+0000","last_unstale":"2024-09-13T11:25:30.100854+0000","last_undegraded":"2024-09-13T11:25:30.100854+0000","last_fullsized":"2024-09-13T11:25:30.100854+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":6,"last_scrub":"76'3395","last_scrub_stamp":"2024-09-13T11:23:32.776747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:32.776747+0000","objects_scrubbed":1704,"log_size":6213,"log_dups_size":0,"ondisk_log_size":6213,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":179,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":106266624,"num_objects":1622,"num_object_clones":0,"num_object_copies":3244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":932,"num_objects_unfound":0,"num_objects_dirty":1622,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1976,"num_write_kb":87840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1253,"num_bytes_recovered":82116608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[3,0],"avail_no_missing":["3","7"],"object_location_counts":[{"shards":"0,3","objects":1156},{"shards":"3,7","objects":466}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2c","version":"166'6222","reported_seq":11554,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-13T11:25:25.967987+0000","last_change":"2024-09-13T11:24:59.456762+0000","last_active":"2024-09-13T11:25:25.967987+0000","last_peered":"2024-09-13T11:25:25.967987+0000","last_clean":"2024-09-13T11:25:25.967987+0000","last_became_active":"2024-09-13T11:24:59.021895+0000","last_became_peered":"2024-09-13T11:24:59.021895+0000","last_unstale":"2024-09-13T11:25:25.967987+0000","last_undegraded":"2024-09-13T11:25:25.967987+0000","last_fullsized":"2024-09-13T11:25:25.967987+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":6,"last_scrub":"145'5672","last_scrub_stamp":"2024-09-13T11:24:59.456653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:24:59.456653+0000","objects_scrubbed":2192,"log_size":6222,"log_dups_size":0,"ondisk_log_size":6222,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:26:20.877991+0000","scrub_duration":279,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":107610112,"num_objects":1642,"num_object_clones":0,"num_object_copies":3284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1642,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":550,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,7],"acting":[10,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.2b","version":"166'6105","reported_seq":9794,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-13T11:25:25.244306+0000","last_change":"2024-09-13T11:24:45.271477+0000","last_active":"2024-09-13T11:25:25.244306+0000","last_peered":"2024-09-13T11:25:25.244306+0000","last_clean":"2024-09-13T11:25:25.244306+0000","last_became_active":"2024-09-13T11:24:26.180578+0000","last_became_peered":"2024-09-13T11:24:26.180578+0000","last_unstale":"2024-09-13T11:25:25.244306+0000","last_undegraded":"2024-09-13T11:25:25.244306+0000","last_fullsized":"2024-09-13T11:25:25.244306+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":6,"last_scrub":"131'5382","last_scrub_stamp":"2024-09-13T11:24:45.271376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:24:45.271376+0000","objects_scrubbed":1940,"log_size":6105,"log_dups_size":0,"ondisk_log_size":6105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:59.210895+0000","scrub_duration":279,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":101515264,"num_objects":1549,"num_object_clones":0,"num_object_copies":3098,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1549,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":723,"num_write_kb":10624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2a","version":"166'6221","reported_seq":10097,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-13T11:25:26.035064+0000","last_change":"2024-09-13T11:24:46.625117+0000","last_active":"2024-09-13T11:25:26.035064+0000","last_peered":"2024-09-13T11:25:26.035064+0000","last_clean":"2024-09-13T11:25:26.035064+0000","last_became_active":"2024-09-13T11:24:45.320540+0000","last_became_peered":"2024-09-13T11:24:45.320540+0000","last_unstale":"2024-09-13T11:25:26.035064+0000","last_undegraded":"2024-09-13T11:25:26.035064+0000","last_fullsized":"2024-09-13T11:25:26.035064+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":6,"last_scrub":"133'5491","last_scrub_stamp":"2024-09-13T11:24:46.624970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:24:46.624970+0000","objects_scrubbed":2094,"log_size":6221,"log_dups_size":0,"ondisk_log_size":6221,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:26:04.813280+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112197632,"num_objects":1712,"num_object_clones":0,"num_object_copies":3424,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1712,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":732,"num_write_kb":11200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,7],"acting":[10,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.29","version":"166'5881","reported_seq":11572,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-13T11:25:25.960499+0000","last_change":"2024-09-13T11:25:13.333062+0000","last_active":"2024-09-13T11:25:25.960499+0000","last_peered":"2024-09-13T11:25:25.960499+0000","last_clean":"2024-09-13T11:25:25.960499+0000","last_became_active":"2024-09-13T11:25:12.944028+0000","last_became_peered":"2024-09-13T11:25:12.944028+0000","last_unstale":"2024-09-13T11:25:25.960499+0000","last_undegraded":"2024-09-13T11:25:25.960499+0000","last_fullsized":"2024-09-13T11:25:25.960499+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":6,"last_scrub":"154'5579","last_scrub_stamp":"2024-09-13T11:25:13.332959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:13.332959+0000","objects_scrubbed":1937,"log_size":5881,"log_dups_size":0,"ondisk_log_size":5881,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:26:24.089228+0000","scrub_duration":243,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":107020288,"num_objects":1633,"num_object_clones":0,"num_object_copies":3266,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1633,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":304,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.28","version":"166'5729","reported_seq":9507,"reported_epoch":166,"state":"active+recovering+remapped","last_fresh":"2024-09-13T11:25:25.386902+0000","last_change":"2024-09-13T11:25:11.182075+0000","last_active":"2024-09-13T11:25:25.386902+0000","last_peered":"2024-09-13T11:25:25.386902+0000","last_clean":"2024-09-13T11:24:25.005266+0000","last_became_active":"2024-09-13T11:24:27.218508+0000","last_became_peered":"2024-09-13T11:24:27.218508+0000","last_unstale":"2024-09-13T11:25:25.386902+0000","last_undegraded":"2024-09-13T11:25:25.386902+0000","last_fullsized":"2024-09-13T11:25:25.386902+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":6,"last_scrub":"68'2771","last_scrub_stamp":"2024-09-13T11:23:20.644747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:20.644747+0000","objects_scrubbed":1356,"log_size":5729,"log_dups_size":0,"ondisk_log_size":5729,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":411,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":107872256,"num_objects":1646,"num_object_clones":0,"num_object_copies":3292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1072,"num_objects_unfound":0,"num_objects_dirty":1646,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1982,"num_write_kb":94400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1150,"num_bytes_recovered":75366400,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5],"acting":[1,5],"avail_no_missing":["1","6"],"object_location_counts":[{"shards":"1,5","objects":1110},{"shards":"1,6","objects":536}],"blocked_by":[],"up_primary":8,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.27","version":"166'5812","reported_seq":10730,"reported_epoch":166,"state":"active+recovering+remapped","last_fresh":"2024-09-13T11:25:26.113633+0000","last_change":"2024-09-13T11:25:08.115804+0000","last_active":"2024-09-13T11:25:26.113633+0000","last_peered":"2024-09-13T11:25:26.113633+0000","last_clean":"2024-09-13T11:24:24.942332+0000","last_became_active":"2024-09-13T11:24:27.215734+0000","last_became_peered":"2024-09-13T11:24:27.215734+0000","last_unstale":"2024-09-13T11:25:26.113633+0000","last_undegraded":"2024-09-13T11:25:26.113633+0000","last_fullsized":"2024-09-13T11:25:26.113633+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":6,"last_scrub":"68'2736","last_scrub_stamp":"2024-09-13T11:23:20.443329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:20.443329+0000","objects_scrubbed":99,"log_size":5812,"log_dups_size":0,"ondisk_log_size":5812,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":183,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":108920832,"num_objects":1662,"num_object_clones":0,"num_object_copies":3324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":392,"num_objects_unfound":0,"num_objects_dirty":1662,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2090,"num_write_kb":98144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1529,"num_bytes_recovered":100204544,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,11],"acting":[8,5],"avail_no_missing":["8","13"],"object_location_counts":[{"shards":"5,8","objects":1466},{"shards":"8,13","objects":196}],"blocked_by":[],"up_primary":5,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.26","version":"166'5927","reported_seq":11498,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-13T11:25:26.025249+0000","last_change":"2024-09-13T11:25:01.697082+0000","last_active":"2024-09-13T11:25:26.025249+0000","last_peered":"2024-09-13T11:25:26.025249+0000","last_clean":"2024-09-13T11:25:26.025249+0000","last_became_active":"2024-09-13T11:24:54.976919+0000","last_became_peered":"2024-09-13T11:24:54.976919+0000","last_unstale":"2024-09-13T11:25:26.025249+0000","last_undegraded":"2024-09-13T11:25:26.025249+0000","last_fullsized":"2024-09-13T11:25:26.025249+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":6,"last_scrub":"147'5433","last_scrub_stamp":"2024-09-13T11:25:01.696886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:01.696886+0000","objects_scrubbed":2158,"log_size":5927,"log_dups_size":0,"ondisk_log_size":5927,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:26:22.070621+0000","scrub_duration":403,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":108527616,"num_objects":1656,"num_object_clones":0,"num_object_copies":3312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1656,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":502,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,12],"acting":[8,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.25","version":"166'5781","reported_seq":11511,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-13T11:25:26.095773+0000","last_change":"2024-09-13T11:24:57.073162+0000","last_active":"2024-09-13T11:25:26.095773+0000","last_peered":"2024-09-13T11:25:26.095773+0000","last_clean":"2024-09-13T11:25:26.095773+0000","last_became_active":"2024-09-13T11:24:55.999019+0000","last_became_peered":"2024-09-13T11:24:55.999019+0000","last_unstale":"2024-09-13T11:25:26.095773+0000","last_undegraded":"2024-09-13T11:25:26.095773+0000","last_fullsized":"2024-09-13T11:25:26.095773+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":6,"last_scrub":"142'5214","last_scrub_stamp":"2024-09-13T11:24:57.073061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:24:57.073061+0000","objects_scrubbed":2249,"log_size":5781,"log_dups_size":0,"ondisk_log_size":5781,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:26:12.348519+0000","scrub_duration":571,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110428160,"num_objects":1685,"num_object_clones":0,"num_object_copies":3370,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1685,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":570,"num_write_kb":192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,2],"acting":[12,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[]},{"pgid":"3.24","version":"168'5832","reported_seq":9752,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-09-13T11:25:26.955529+0000","last_change":"2024-09-13T11:24:48.048846+0000","last_active":"2024-09-13T11:25:26.955529+0000","last_peered":"2024-09-13T11:25:26.955529+0000","last_clean":"2024-09-13T11:25:26.955529+0000","last_became_active":"2024-09-13T11:24:46.936994+0000","last_became_peered":"2024-09-13T11:24:46.936994+0000","last_unstale":"2024-09-13T11:25:26.955529+0000","last_undegraded":"2024-09-13T11:25:26.955529+0000","last_fullsized":"2024-09-13T11:25:26.955529+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":6,"last_scrub":"135'5093","last_scrub_stamp":"2024-09-13T11:24:48.048764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:24:48.048764+0000","objects_scrubbed":2102,"log_size":5832,"log_dups_size":0,"ondisk_log_size":5832,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:53.485370+0000","scrub_duration":287,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":108265472,"num_objects":1652,"num_object_clones":0,"num_object_copies":3304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1652,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":740,"num_write_kb":9280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1],"acting":[9,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.23","version":"166'5040","reported_seq":7891,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-13T11:25:25.955199+0000","last_change":"2024-09-13T11:23:48.133184+0000","last_active":"2024-09-13T11:25:25.955199+0000","last_peered":"2024-09-13T11:25:25.955199+0000","last_clean":"2024-09-13T11:25:25.955199+0000","last_became_active":"2024-09-13T11:23:47.926677+0000","last_became_peered":"2024-09-13T11:23:47.926677+0000","last_unstale":"2024-09-13T11:25:25.955199+0000","last_undegraded":"2024-09-13T11:25:25.955199+0000","last_fullsized":"2024-09-13T11:25:25.955199+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":6,"last_scrub":"91'3278","last_scrub_stamp":"2024-09-13T11:23:48.133084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:48.133084+0000","objects_scrubbed":1029,"log_size":5040,"log_dups_size":0,"ondisk_log_size":5040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":139,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":111804416,"num_objects":1706,"num_object_clones":0,"num_object_copies":3412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1706,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1763,"num_write_kb":78080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9],"acting":[5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.22","version":"166'4987","reported_seq":10567,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-13T11:25:26.093268+0000","last_change":"2024-09-13T11:24:45.485711+0000","last_active":"2024-09-13T11:25:26.093268+0000","last_peered":"2024-09-13T11:25:26.093268+0000","last_clean":"2024-09-13T11:25:26.093268+0000","last_became_active":"2024-09-13T11:24:29.253848+0000","last_became_peered":"2024-09-13T11:24:29.253848+0000","last_unstale":"2024-09-13T11:25:26.093268+0000","last_undegraded":"2024-09-13T11:25:26.093268+0000","last_fullsized":"2024-09-13T11:25:26.093268+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":6,"last_scrub":"90'3246","last_scrub_stamp":"2024-09-13T11:23:47.300339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:47.300339+0000","objects_scrubbed":1033,"log_size":4987,"log_dups_size":0,"ondisk_log_size":4987,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":203,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":108527616,"num_objects":1656,"num_object_clones":0,"num_object_copies":3312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1656,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1743,"num_write_kb":75712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1768,"num_bytes_recovered":115867648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.21","version":"166'4903","reported_seq":7769,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-13T11:25:26.085283+0000","last_change":"2024-09-13T11:24:12.564117+0000","last_active":"2024-09-13T11:25:26.085283+0000","last_peered":"2024-09-13T11:25:26.085283+0000","last_clean":"2024-09-13T11:25:26.085283+0000","last_became_active":"2024-09-13T11:24:01.250308+0000","last_became_peered":"2024-09-13T11:24:01.250308+0000","last_unstale":"2024-09-13T11:25:26.085283+0000","last_undegraded":"2024-09-13T11:25:26.085283+0000","last_fullsized":"2024-09-13T11:25:26.085283+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":6,"last_scrub":"112'3670","last_scrub_stamp":"2024-09-13T11:24:12.564031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:24:12.564031+0000","objects_scrubbed":1431,"log_size":4903,"log_dups_size":0,"ondisk_log_size":4903,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:36.766299+0000","scrub_duration":183,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":103350272,"num_objects":1577,"num_object_clones":0,"num_object_copies":3154,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1577,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1236,"num_write_kb":44224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8],"acting":[0,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.20","version":"166'5035","reported_seq":7188,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-13T11:25:25.998756+0000","last_change":"2024-09-13T11:23:47.969503+0000","last_active":"2024-09-13T11:25:25.998756+0000","last_peered":"2024-09-13T11:25:25.998756+0000","last_clean":"2024-09-13T11:25:25.998756+0000","last_became_active":"2024-09-13T11:23:33.954946+0000","last_became_peered":"2024-09-13T11:23:33.954946+0000","last_unstale":"2024-09-13T11:25:25.998756+0000","last_undegraded":"2024-09-13T11:25:25.998756+0000","last_fullsized":"2024-09-13T11:25:25.998756+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":6,"last_scrub":"91'3270","last_scrub_stamp":"2024-09-13T11:23:47.969372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:47.969372+0000","objects_scrubbed":1061,"log_size":5035,"log_dups_size":0,"ondisk_log_size":5035,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1327,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":106627072,"num_objects":1627,"num_object_clones":0,"num_object_copies":3254,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1627,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1770,"num_write_kb":74752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,0],"acting":[12,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[]},{"pgid":"3.1f","version":"166'7087","reported_seq":19037,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-13T11:25:26.064403+0000","last_change":"2024-09-13T11:25:20.773752+0000","last_active":"2024-09-13T11:25:26.064403+0000","last_peered":"2024-09-13T11:25:26.064403+0000","last_clean":"2024-09-13T11:25:26.064403+0000","last_became_active":"2024-09-13T11:24:29.290326+0000","last_became_peered":"2024-09-13T11:24:29.290326+0000","last_unstale":"2024-09-13T11:25:26.064403+0000","last_undegraded":"2024-09-13T11:25:26.064403+0000","last_fullsized":"2024-09-13T11:25:26.064403+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":5,"last_scrub":"161'6865","last_scrub_stamp":"2024-09-13T11:25:20.773641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:20.773641+0000","objects_scrubbed":3425,"log_size":7087,"log_dups_size":0,"ondisk_log_size":7087,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:26:21.317280+0000","scrub_duration":535,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":209584128,"num_objects":3198,"num_object_clones":0,"num_object_copies":6396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3198,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2667,"num_write_kb":102848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3306,"num_bytes_recovered":216662016,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5],"acting":[10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1e","version":"166'7310","reported_seq":17827,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-13T11:25:25.982804+0000","last_change":"2024-09-13T11:24:08.338485+0000","last_active":"2024-09-13T11:25:25.982804+0000","last_peered":"2024-09-13T11:25:25.982804+0000","last_clean":"2024-09-13T11:25:25.982804+0000","last_became_active":"2024-09-13T11:24:01.261448+0000","last_became_peered":"2024-09-13T11:24:01.261448+0000","last_unstale":"2024-09-13T11:25:25.982804+0000","last_undegraded":"2024-09-13T11:25:25.982804+0000","last_fullsized":"2024-09-13T11:25:25.982804+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":5,"last_scrub":"110'4537","last_scrub_stamp":"2024-09-13T11:24:08.338354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:24:08.338354+0000","objects_scrubbed":2760,"log_size":7310,"log_dups_size":0,"ondisk_log_size":7310,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":367,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":216137728,"num_objects":3298,"num_object_clones":0,"num_object_copies":6596,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3298,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2782,"num_write_kb":106240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"166'7123","reported_seq":12569,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-13T11:25:26.004350+0000","last_change":"2024-09-13T11:24:12.492592+0000","last_active":"2024-09-13T11:25:26.004350+0000","last_peered":"2024-09-13T11:25:26.004350+0000","last_clean":"2024-09-13T11:25:26.004350+0000","last_became_active":"2024-09-13T11:24:01.258225+0000","last_became_peered":"2024-09-13T11:24:01.258225+0000","last_unstale":"2024-09-13T11:25:26.004350+0000","last_undegraded":"2024-09-13T11:25:26.004350+0000","last_fullsized":"2024-09-13T11:25:26.004350+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":5,"last_scrub":"112'4579","last_scrub_stamp":"2024-09-13T11:24:12.492507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:24:12.492507+0000","objects_scrubbed":2875,"log_size":7123,"log_dups_size":0,"ondisk_log_size":7123,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:34.926575+0000","scrub_duration":375,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":214630424,"num_objects":3276,"num_object_clones":0,"num_object_copies":6552,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3276,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2557,"num_write_kb":94593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,15],"acting":[2,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1c","version":"168'7295","reported_seq":11102,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-09-13T11:25:27.053429+0000","last_change":"2024-09-13T11:24:00.529312+0000","last_active":"2024-09-13T11:25:27.053429+0000","last_peered":"2024-09-13T11:25:27.053429+0000","last_clean":"2024-09-13T11:25:27.053429+0000","last_became_active":"2024-09-13T11:23:33.989492+0000","last_became_peered":"2024-09-13T11:23:33.989492+0000","last_unstale":"2024-09-13T11:25:27.053429+0000","last_undegraded":"2024-09-13T11:25:27.053429+0000","last_fullsized":"2024-09-13T11:25:27.053429+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":5,"last_scrub":"102'4277","last_scrub_stamp":"2024-09-13T11:24:00.529243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:24:00.529243+0000","objects_scrubbed":2544,"log_size":7295,"log_dups_size":0,"ondisk_log_size":7295,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":259,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":213909504,"num_objects":3264,"num_object_clones":0,"num_object_copies":6528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3264,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3020,"num_write_kb":119680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7],"acting":[11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1b","version":"166'7246","reported_seq":12871,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-13T11:25:26.099182+0000","last_change":"2024-09-13T11:24:12.392093+0000","last_active":"2024-09-13T11:25:26.099182+0000","last_peered":"2024-09-13T11:25:26.099182+0000","last_clean":"2024-09-13T11:25:26.099182+0000","last_became_active":"2024-09-13T11:24:04.950821+0000","last_became_peered":"2024-09-13T11:24:04.950821+0000","last_unstale":"2024-09-13T11:25:26.099182+0000","last_undegraded":"2024-09-13T11:25:26.099182+0000","last_fullsized":"2024-09-13T11:25:26.099182+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":5,"last_scrub":"112'4665","last_scrub_stamp":"2024-09-13T11:24:12.391890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:24:12.391890+0000","objects_scrubbed":2934,"log_size":7246,"log_dups_size":0,"ondisk_log_size":7246,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:41.846147+0000","scrub_duration":675,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":215613440,"num_objects":3290,"num_object_clones":0,"num_object_copies":6580,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3290,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2602,"num_write_kb":94656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,15],"acting":[12,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[]},{"pgid":"3.1a","version":"166'7125","reported_seq":12411,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-13T11:25:25.999241+0000","last_change":"2024-09-13T11:24:09.556174+0000","last_active":"2024-09-13T11:25:25.999241+0000","last_peered":"2024-09-13T11:25:25.999241+0000","last_clean":"2024-09-13T11:25:25.999241+0000","last_became_active":"2024-09-13T11:24:03.935185+0000","last_became_peered":"2024-09-13T11:24:03.935185+0000","last_unstale":"2024-09-13T11:25:25.999241+0000","last_undegraded":"2024-09-13T11:25:25.999241+0000","last_fullsized":"2024-09-13T11:25:25.999241+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":5,"last_scrub":"110'4433","last_scrub_stamp":"2024-09-13T11:24:09.556022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:24:09.556022+0000","objects_scrubbed":2654,"log_size":7125,"log_dups_size":0,"ondisk_log_size":7125,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:28.469324+0000","scrub_duration":387,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":210698240,"num_objects":3215,"num_object_clones":0,"num_object_copies":6430,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3215,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2703,"num_write_kb":104448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9],"acting":[4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.19","version":"168'6802","reported_seq":11279,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-09-13T11:25:26.985429+0000","last_change":"2024-09-13T11:25:10.784789+0000","last_active":"2024-09-13T11:25:26.985429+0000","last_peered":"2024-09-13T11:25:26.985429+0000","last_clean":"2024-09-13T11:25:26.985429+0000","last_became_active":"2024-09-13T11:23:41.924941+0000","last_became_peered":"2024-09-13T11:23:41.924941+0000","last_unstale":"2024-09-13T11:25:26.985429+0000","last_undegraded":"2024-09-13T11:25:26.985429+0000","last_fullsized":"2024-09-13T11:25:26.985429+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":5,"last_scrub":"151'6156","last_scrub_stamp":"2024-09-13T11:25:10.784716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:10.784716+0000","objects_scrubbed":3781,"log_size":6802,"log_dups_size":0,"ondisk_log_size":6802,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:26:30.390310+0000","scrub_duration":1131,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":204275712,"num_objects":3117,"num_object_clones":0,"num_object_copies":6234,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3117,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3400,"num_write_kb":145216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1],"acting":[9,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.18","version":"166'6844","reported_seq":11579,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-13T11:25:26.060441+0000","last_change":"2024-09-13T11:24:13.454659+0000","last_active":"2024-09-13T11:25:26.060441+0000","last_peered":"2024-09-13T11:25:26.060441+0000","last_clean":"2024-09-13T11:25:26.060441+0000","last_became_active":"2024-09-13T11:23:48.926014+0000","last_became_peered":"2024-09-13T11:23:48.926014+0000","last_unstale":"2024-09-13T11:25:26.060441+0000","last_undegraded":"2024-09-13T11:25:26.060441+0000","last_fullsized":"2024-09-13T11:25:26.060441+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":5,"last_scrub":"112'4376","last_scrub_stamp":"2024-09-13T11:24:13.454575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:24:13.454575+0000","objects_scrubbed":3036,"log_size":6844,"log_dups_size":0,"ondisk_log_size":6844,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:34.811860+0000","scrub_duration":375,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":213843968,"num_objects":3263,"num_object_clones":0,"num_object_copies":6526,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3263,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2477,"num_write_kb":86528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0],"acting":[10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.17","version":"166'6739","reported_seq":10945,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-13T11:25:26.063961+0000","last_change":"2024-09-13T11:25:12.658562+0000","last_active":"2024-09-13T11:25:26.063961+0000","last_peered":"2024-09-13T11:25:26.063961+0000","last_clean":"2024-09-13T11:25:26.063961+0000","last_became_active":"2024-09-13T11:23:42.935601+0000","last_became_peered":"2024-09-13T11:23:42.935601+0000","last_unstale":"2024-09-13T11:25:26.063961+0000","last_undegraded":"2024-09-13T11:25:26.063961+0000","last_fullsized":"2024-09-13T11:25:26.063961+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":5,"last_scrub":"153'6197","last_scrub_stamp":"2024-09-13T11:25:12.658464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:12.658464+0000","objects_scrubbed":3743,"log_size":6739,"log_dups_size":0,"ondisk_log_size":6739,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:26:25.625277+0000","scrub_duration":467,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":209715200,"num_objects":3200,"num_object_clones":0,"num_object_copies":6400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3200,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3388,"num_write_kb":146688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3],"acting":[10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"168'7080","reported_seq":11926,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-09-13T11:25:26.981846+0000","last_change":"2024-09-13T11:25:01.903487+0000","last_active":"2024-09-13T11:25:26.981846+0000","last_peered":"2024-09-13T11:25:26.981846+0000","last_clean":"2024-09-13T11:25:26.981846+0000","last_became_active":"2024-09-13T11:23:46.105237+0000","last_became_peered":"2024-09-13T11:23:46.105237+0000","last_unstale":"2024-09-13T11:25:26.981846+0000","last_undegraded":"2024-09-13T11:25:26.981846+0000","last_fullsized":"2024-09-13T11:25:26.981846+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":5,"last_scrub":"147'6035","last_scrub_stamp":"2024-09-13T11:25:01.903395+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:01.903395+0000","objects_scrubbed":4359,"log_size":7080,"log_dups_size":0,"ondisk_log_size":7080,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:26:16.436377+0000","scrub_duration":743,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":216203264,"num_objects":3299,"num_object_clones":0,"num_object_copies":6598,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3299,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3605,"num_write_kb":155200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,15],"acting":[11,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.15","version":"166'6726","reported_seq":18051,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-13T11:25:26.070608+0000","last_change":"2024-09-13T11:25:12.198448+0000","last_active":"2024-09-13T11:25:26.070608+0000","last_peered":"2024-09-13T11:25:26.070608+0000","last_clean":"2024-09-13T11:25:26.070608+0000","last_became_active":"2024-09-13T11:24:29.226615+0000","last_became_peered":"2024-09-13T11:24:29.226615+0000","last_unstale":"2024-09-13T11:25:26.070608+0000","last_undegraded":"2024-09-13T11:25:26.070608+0000","last_fullsized":"2024-09-13T11:25:26.070608+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":5,"last_scrub":"91'3384","last_scrub_stamp":"2024-09-13T11:23:48.332600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:48.332600+0000","objects_scrubbed":2032,"log_size":6726,"log_dups_size":0,"ondisk_log_size":6726,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":235,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":209649664,"num_objects":3199,"num_object_clones":0,"num_object_copies":6398,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3199,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3343,"num_write_kb":144320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3385,"num_bytes_recovered":221839360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.14","version":"169'7021","reported_seq":18278,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-09-13T11:25:29.995610+0000","last_change":"2024-09-13T11:25:15.378521+0000","last_active":"2024-09-13T11:25:29.995610+0000","last_peered":"2024-09-13T11:25:29.995610+0000","last_clean":"2024-09-13T11:25:29.995610+0000","last_became_active":"2024-09-13T11:24:29.253106+0000","last_became_peered":"2024-09-13T11:24:29.253106+0000","last_unstale":"2024-09-13T11:25:29.995610+0000","last_undegraded":"2024-09-13T11:25:29.995610+0000","last_fullsized":"2024-09-13T11:25:29.995610+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":5,"last_scrub":"91'3371","last_scrub_stamp":"2024-09-13T11:23:48.696960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:48.696960+0000","objects_scrubbed":1935,"log_size":7021,"log_dups_size":0,"ondisk_log_size":7021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":735,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":211746816,"num_objects":3231,"num_object_clones":0,"num_object_copies":6462,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3231,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3660,"num_write_kb":158592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3437,"num_bytes_recovered":225247232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,15],"acting":[3,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":304,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-13T11:25:25.226545+0000","last_change":"2024-09-13T11:24:23.212494+0000","last_active":"2024-09-13T11:25:25.226545+0000","last_peered":"2024-09-13T11:25:25.226545+0000","last_clean":"2024-09-13T11:25:25.226545+0000","last_became_active":"2024-09-13T11:22:14.192897+0000","last_became_peered":"2024-09-13T11:22:14.192897+0000","last_unstale":"2024-09-13T11:25:25.226545+0000","last_undegraded":"2024-09-13T11:25:25.226545+0000","last_fullsized":"2024-09-13T11:25:25.226545+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:24:23.212406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:24:23.212406+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:29.886382+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00084655899999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,10],"acting":[15,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"3.6","version":"168'5938","reported_seq":9805,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-09-13T11:25:27.033999+0000","last_change":"2024-09-13T11:24:49.416851+0000","last_active":"2024-09-13T11:25:27.033999+0000","last_peered":"2024-09-13T11:25:27.033999+0000","last_clean":"2024-09-13T11:25:27.033999+0000","last_became_active":"2024-09-13T11:24:29.322558+0000","last_became_peered":"2024-09-13T11:24:29.322558+0000","last_unstale":"2024-09-13T11:25:27.033999+0000","last_undegraded":"2024-09-13T11:25:27.033999+0000","last_fullsized":"2024-09-13T11:25:27.033999+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"136'5216","last_scrub_stamp":"2024-09-13T11:24:49.416706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:24:49.416706+0000","objects_scrubbed":2083,"log_size":5938,"log_dups_size":0,"ondisk_log_size":5938,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:26:09.525086+0000","scrub_duration":247,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":108396544,"num_objects":1654,"num_object_clones":0,"num_object_copies":3308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1654,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":723,"num_write_kb":9408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,6],"acting":[13,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":315,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-09-13T11:25:26.955759+0000","last_change":"2024-09-13T11:24:59.556253+0000","last_active":"2024-09-13T11:25:26.955759+0000","last_peered":"2024-09-13T11:25:26.955759+0000","last_clean":"2024-09-13T11:25:26.955759+0000","last_became_active":"2024-09-13T11:22:14.211923+0000","last_became_peered":"2024-09-13T11:22:14.211923+0000","last_unstale":"2024-09-13T11:25:26.955759+0000","last_undegraded":"2024-09-13T11:25:26.955759+0000","last_fullsized":"2024-09-13T11:25:26.955759+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:24:59.556180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:24:59.556180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:26:06.229322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011689529999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0],"acting":[9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"168'5770","reported_seq":6131,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-09-13T11:25:27.024109+0000","last_change":"2024-09-13T11:24:33.336210+0000","last_active":"2024-09-13T11:25:27.024109+0000","last_peered":"2024-09-13T11:25:27.024109+0000","last_clean":"2024-09-13T11:25:27.024109+0000","last_became_active":"2024-09-13T11:24:24.092975+0000","last_became_peered":"2024-09-13T11:24:24.092975+0000","last_unstale":"2024-09-13T11:25:27.024109+0000","last_undegraded":"2024-09-13T11:25:27.024109+0000","last_fullsized":"2024-09-13T11:25:27.024109+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"125'4820","last_scrub_stamp":"2024-09-13T11:24:33.336091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:24:33.336091+0000","objects_scrubbed":1766,"log_size":5770,"log_dups_size":0,"ondisk_log_size":5770,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:54.538352+0000","scrub_duration":263,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":107347968,"num_objects":1638,"num_object_clones":0,"num_object_copies":3276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1638,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":952,"num_write_kb":26368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,8],"acting":[13,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":304,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-13T11:25:25.221085+0000","last_change":"2024-09-13T11:23:59.530794+0000","last_active":"2024-09-13T11:25:25.221085+0000","last_peered":"2024-09-13T11:25:25.221085+0000","last_clean":"2024-09-13T11:25:25.221085+0000","last_became_active":"2024-09-13T11:22:14.581458+0000","last_became_peered":"2024-09-13T11:22:14.581458+0000","last_unstale":"2024-09-13T11:25:25.221085+0000","last_undegraded":"2024-09-13T11:25:25.221085+0000","last_fullsized":"2024-09-13T11:25:25.221085+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:23:59.530670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:23:59.530670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001026047,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5],"acting":[8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"166'5817","reported_seq":6146,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-13T11:25:25.378158+0000","last_change":"2024-09-13T11:24:24.579036+0000","last_active":"2024-09-13T11:25:25.378158+0000","last_peered":"2024-09-13T11:25:25.378158+0000","last_clean":"2024-09-13T11:25:25.378158+0000","last_became_active":"2024-09-13T11:24:24.083077+0000","last_became_peered":"2024-09-13T11:24:24.083077+0000","last_unstale":"2024-09-13T11:25:25.378158+0000","last_undegraded":"2024-09-13T11:25:25.378158+0000","last_fullsized":"2024-09-13T11:25:25.378158+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'4737","last_scrub_stamp":"2024-09-13T11:24:24.578900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:24:24.578900+0000","objects_scrubbed":1690,"log_size":5817,"log_dups_size":0,"ondisk_log_size":5817,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:38.103421+0000","scrub_duration":407,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":106758144,"num_objects":1629,"num_object_clones":0,"num_object_copies":3258,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1629,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1083,"num_write_kb":32704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,12],"acting":[1,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":292,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-13T11:25:25.217725+0000","last_change":"2024-09-13T11:24:08.271026+0000","last_active":"2024-09-13T11:25:25.217725+0000","last_peered":"2024-09-13T11:25:25.217725+0000","last_clean":"2024-09-13T11:25:25.217725+0000","last_became_active":"2024-09-13T11:22:47.719636+0000","last_became_peered":"2024-09-13T11:22:47.719636+0000","last_unstale":"2024-09-13T11:25:25.217725+0000","last_undegraded":"2024-09-13T11:25:25.217725+0000","last_fullsized":"2024-09-13T11:25:25.217725+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:24:08.270972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:24:08.270972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:36.805605+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00087476000000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,9],"acting":[1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"166'5772","reported_seq":6120,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-13T11:25:26.041019+0000","last_change":"2024-09-13T11:24:24.616937+0000","last_active":"2024-09-13T11:25:26.041019+0000","last_peered":"2024-09-13T11:25:26.041019+0000","last_clean":"2024-09-13T11:25:26.041019+0000","last_became_active":"2024-09-13T11:24:24.093492+0000","last_became_peered":"2024-09-13T11:24:24.093492+0000","last_unstale":"2024-09-13T11:25:26.041019+0000","last_undegraded":"2024-09-13T11:25:26.041019+0000","last_fullsized":"2024-09-13T11:25:26.041019+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'4744","last_scrub_stamp":"2024-09-13T11:24:24.616633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:24:24.616633+0000","objects_scrubbed":1624,"log_size":5772,"log_dups_size":0,"ondisk_log_size":5772,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:42.911074+0000","scrub_duration":415,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":104923136,"num_objects":1601,"num_object_clones":0,"num_object_copies":3202,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1601,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1035,"num_write_kb":32384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,13],"acting":[15,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"3.3","version":"166'4964","reported_seq":5331,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-13T11:25:26.038212+0000","last_change":"2024-09-13T11:25:08.394545+0000","last_active":"2024-09-13T11:25:26.038212+0000","last_peered":"2024-09-13T11:25:26.038212+0000","last_clean":"2024-09-13T11:25:26.038212+0000","last_became_active":"2024-09-13T11:23:31.930448+0000","last_became_peered":"2024-09-13T11:23:31.930448+0000","last_unstale":"2024-09-13T11:25:26.038212+0000","last_undegraded":"2024-09-13T11:25:26.038212+0000","last_fullsized":"2024-09-13T11:25:26.038212+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"151'4587","last_scrub_stamp":"2024-09-13T11:25:08.394441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:08.394441+0000","objects_scrubbed":1942,"log_size":4964,"log_dups_size":0,"ondisk_log_size":4964,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:26:27.641900+0000","scrub_duration":219,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":102367232,"num_objects":1562,"num_object_clones":0,"num_object_copies":3124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1562,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1842,"num_write_kb":80896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,7],"acting":[10,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"28'2","reported_seq":242,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-13T11:25:25.216396+0000","last_change":"2024-09-13T11:25:13.869564+0000","last_active":"2024-09-13T11:25:25.216396+0000","last_peered":"2024-09-13T11:25:25.216396+0000","last_clean":"2024-09-13T11:25:25.216396+0000","last_became_active":"2024-09-13T11:23:12.546348+0000","last_became_peered":"2024-09-13T11:23:12.546348+0000","last_unstale":"2024-09-13T11:25:25.216396+0000","last_undegraded":"2024-09-13T11:25:25.216396+0000","last_fullsized":"2024-09-13T11:25:25.216396+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2","last_scrub_stamp":"2024-09-13T11:25:13.869413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:25:13.869413+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:26:38.761718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0025826730000000002,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8],"acting":[2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"166'4962","reported_seq":5328,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-13T11:25:26.067735+0000","last_change":"2024-09-13T11:25:09.842875+0000","last_active":"2024-09-13T11:25:26.067735+0000","last_peered":"2024-09-13T11:25:26.067735+0000","last_clean":"2024-09-13T11:25:26.067735+0000","last_became_active":"2024-09-13T11:23:31.921388+0000","last_became_peered":"2024-09-13T11:23:31.921388+0000","last_unstale":"2024-09-13T11:25:26.067735+0000","last_undegraded":"2024-09-13T11:25:26.067735+0000","last_fullsized":"2024-09-13T11:25:26.067735+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"151'4644","last_scrub_stamp":"2024-09-13T11:25:09.842728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:09.842728+0000","objects_scrubbed":1944,"log_size":4962,"log_dups_size":0,"ondisk_log_size":4962,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:26:25.553642+0000","scrub_duration":395,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":106561536,"num_objects":1626,"num_object_clones":0,"num_object_copies":3252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1626,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1856,"num_write_kb":84032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,15],"acting":[12,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":145,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-13T11:25:25.217562+0000","last_change":"2024-09-13T11:24:29.096863+0000","last_active":"2024-09-13T11:25:25.217562+0000","last_peered":"2024-09-13T11:25:25.217562+0000","last_clean":"2024-09-13T11:25:25.217562+0000","last_became_active":"2024-09-13T11:24:29.096575+0000","last_became_peered":"2024-09-13T11:24:29.096575+0000","last_unstale":"2024-09-13T11:25:25.217562+0000","last_undegraded":"2024-09-13T11:25:25.217562+0000","last_fullsized":"2024-09-13T11:25:25.217562+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:24:22.064711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:24:22.064711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:34.178168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00081225399999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"168'4968","reported_seq":5340,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-09-13T11:25:26.935370+0000","last_change":"2024-09-13T11:25:00.861990+0000","last_active":"2024-09-13T11:25:26.935370+0000","last_peered":"2024-09-13T11:25:26.935370+0000","last_clean":"2024-09-13T11:25:26.935370+0000","last_became_active":"2024-09-13T11:23:31.926528+0000","last_became_peered":"2024-09-13T11:23:31.926528+0000","last_unstale":"2024-09-13T11:25:26.935370+0000","last_undegraded":"2024-09-13T11:25:26.935370+0000","last_fullsized":"2024-09-13T11:25:26.935370+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"146'4447","last_scrub_stamp":"2024-09-13T11:25:00.861834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:00.861834+0000","objects_scrubbed":2126,"log_size":4968,"log_dups_size":0,"ondisk_log_size":4968,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:26:15.411903+0000","scrub_duration":787,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":104988672,"num_objects":1602,"num_object_clones":0,"num_object_copies":3204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1602,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1898,"num_write_kb":85056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,11],"acting":[13,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":310,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-09-13T11:25:29.252224+0000","last_change":"2024-09-13T11:24:11.895017+0000","last_active":"2024-09-13T11:25:29.252224+0000","last_peered":"2024-09-13T11:25:29.252224+0000","last_clean":"2024-09-13T11:25:29.252224+0000","last_became_active":"2024-09-13T11:22:14.211435+0000","last_became_peered":"2024-09-13T11:22:14.211435+0000","last_unstale":"2024-09-13T11:25:29.252224+0000","last_undegraded":"2024-09-13T11:25:29.252224+0000","last_fullsized":"2024-09-13T11:25:29.252224+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:24:11.894937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:24:11.894937+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0010010360000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2","version":"166'5033","reported_seq":8863,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-13T11:25:25.866101+0000","last_change":"2024-09-13T11:25:23.765128+0000","last_active":"2024-09-13T11:25:25.866101+0000","last_peered":"2024-09-13T11:25:25.866101+0000","last_clean":"2024-09-13T11:25:25.866101+0000","last_became_active":"2024-09-13T11:24:29.175407+0000","last_became_peered":"2024-09-13T11:24:29.175407+0000","last_unstale":"2024-09-13T11:25:25.866101+0000","last_undegraded":"2024-09-13T11:25:25.866101+0000","last_fullsized":"2024-09-13T11:25:25.866101+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"164'4994","last_scrub_stamp":"2024-09-13T11:25:23.765047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:23.765047+0000","objects_scrubbed":1689,"log_size":5033,"log_dups_size":0,"ondisk_log_size":5033,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:26:45.252942+0000","scrub_duration":283,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":107937792,"num_objects":1647,"num_object_clones":0,"num_object_copies":3294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1647,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1976,"num_write_kb":89024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1731,"num_bytes_recovered":113442816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,3],"acting":[12,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[]},{"pgid":"2.3","version":"26'1","reported_seq":115,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-13T11:25:25.226319+0000","last_change":"2024-09-13T11:25:23.034357+0000","last_active":"2024-09-13T11:25:25.226319+0000","last_peered":"2024-09-13T11:25:25.226319+0000","last_clean":"2024-09-13T11:25:25.226319+0000","last_became_active":"2024-09-13T11:24:29.110028+0000","last_became_peered":"2024-09-13T11:24:29.110028+0000","last_unstale":"2024-09-13T11:25:25.226319+0000","last_undegraded":"2024-09-13T11:25:25.226319+0000","last_fullsized":"2024-09-13T11:25:25.226319+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"26'1","last_scrub_stamp":"2024-09-13T11:25:23.034247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:25:23.034247+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:26:42.429349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,8],"acting":[15,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"1.0","version":"26'112","reported_seq":446,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-13T11:25:25.232244+0000","last_change":"2024-09-13T11:25:23.642366+0000","last_active":"2024-09-13T11:25:25.232244+0000","last_peered":"2024-09-13T11:25:25.232244+0000","last_clean":"2024-09-13T11:25:25.232244+0000","last_became_active":"2024-09-13T11:22:12.129721+0000","last_became_peered":"2024-09-13T11:22:12.129721+0000","last_unstale":"2024-09-13T11:25:25.232244+0000","last_undegraded":"2024-09-13T11:25:25.232244+0000","last_fullsized":"2024-09-13T11:25:25.232244+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"26'112","last_scrub_stamp":"2024-09-13T11:25:23.642204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:11.098388+0000","last_clean_scrub_stamp":"2024-09-13T11:25:23.642204+0000","objects_scrubbed":2,"log_size":112,"log_dups_size":0,"ondisk_log_size":112,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:26:42.628319+0000","scrub_duration":67,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.8","version":"166'5776","reported_seq":6125,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-13T11:25:25.960544+0000","last_change":"2024-09-13T11:24:24.506212+0000","last_active":"2024-09-13T11:25:25.960544+0000","last_peered":"2024-09-13T11:25:25.960544+0000","last_clean":"2024-09-13T11:25:25.960544+0000","last_became_active":"2024-09-13T11:24:24.096220+0000","last_became_peered":"2024-09-13T11:24:24.096220+0000","last_unstale":"2024-09-13T11:25:25.960544+0000","last_undegraded":"2024-09-13T11:25:25.960544+0000","last_fullsized":"2024-09-13T11:25:25.960544+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'4749","last_scrub_stamp":"2024-09-13T11:24:24.506109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:24:24.506109+0000","objects_scrubbed":1690,"log_size":5776,"log_dups_size":0,"ondisk_log_size":5776,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:49.430042+0000","scrub_duration":215,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":107347968,"num_objects":1638,"num_object_clones":0,"num_object_copies":3276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1638,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1028,"num_write_kb":31232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.9","version":"166'5817","reported_seq":6165,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-13T11:25:25.911883+0000","last_change":"2024-09-13T11:24:24.928964+0000","last_active":"2024-09-13T11:25:25.911883+0000","last_peered":"2024-09-13T11:25:25.911883+0000","last_clean":"2024-09-13T11:25:25.911883+0000","last_became_active":"2024-09-13T11:24:24.082504+0000","last_became_peered":"2024-09-13T11:24:24.082504+0000","last_unstale":"2024-09-13T11:25:25.911883+0000","last_undegraded":"2024-09-13T11:25:25.911883+0000","last_fullsized":"2024-09-13T11:25:25.911883+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'4799","last_scrub_stamp":"2024-09-13T11:24:24.928724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:24:24.928724+0000","objects_scrubbed":1705,"log_size":5817,"log_dups_size":0,"ondisk_log_size":5817,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:43.845801+0000","scrub_duration":255,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":108527616,"num_objects":1656,"num_object_clones":0,"num_object_copies":3312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1656,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1021,"num_write_kb":31104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,2],"acting":[12,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[]},{"pgid":"3.a","version":"169'6243","reported_seq":6602,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-09-13T11:25:30.011322+0000","last_change":"2024-09-13T11:24:25.137871+0000","last_active":"2024-09-13T11:25:30.011322+0000","last_peered":"2024-09-13T11:25:30.011322+0000","last_clean":"2024-09-13T11:25:30.011322+0000","last_became_active":"2024-09-13T11:24:24.089253+0000","last_became_peered":"2024-09-13T11:24:24.089253+0000","last_unstale":"2024-09-13T11:25:30.011322+0000","last_undegraded":"2024-09-13T11:25:30.011322+0000","last_fullsized":"2024-09-13T11:25:30.011322+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'5157","last_scrub_stamp":"2024-09-13T11:24:25.137777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:24:25.137777+0000","objects_scrubbed":1777,"log_size":6243,"log_dups_size":0,"ondisk_log_size":6243,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:36.149013+0000","scrub_duration":223,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":102367232,"num_objects":1562,"num_object_clones":0,"num_object_copies":3124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1562,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1089,"num_write_kb":27968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.b","version":"168'6162","reported_seq":6518,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-09-13T11:25:26.980603+0000","last_change":"2024-09-13T11:24:25.503864+0000","last_active":"2024-09-13T11:25:26.980603+0000","last_peered":"2024-09-13T11:25:26.980603+0000","last_clean":"2024-09-13T11:25:26.980603+0000","last_became_active":"2024-09-13T11:24:24.085684+0000","last_became_peered":"2024-09-13T11:24:24.085684+0000","last_unstale":"2024-09-13T11:25:26.980603+0000","last_undegraded":"2024-09-13T11:25:26.980603+0000","last_fullsized":"2024-09-13T11:25:26.980603+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'5112","last_scrub_stamp":"2024-09-13T11:24:25.503716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:24:25.503716+0000","objects_scrubbed":1718,"log_size":6162,"log_dups_size":0,"ondisk_log_size":6162,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:37.636639+0000","scrub_duration":395,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":106561536,"num_objects":1626,"num_object_clones":0,"num_object_copies":3252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1626,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1050,"num_write_kb":30656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,1],"acting":[13,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"3.c","version":"168'6166","reported_seq":6520,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-09-13T11:25:26.972326+0000","last_change":"2024-09-13T11:24:24.595021+0000","last_active":"2024-09-13T11:25:26.972326+0000","last_peered":"2024-09-13T11:25:26.972326+0000","last_clean":"2024-09-13T11:25:26.972326+0000","last_became_active":"2024-09-13T11:24:24.090847+0000","last_became_peered":"2024-09-13T11:24:24.090847+0000","last_unstale":"2024-09-13T11:25:26.972326+0000","last_undegraded":"2024-09-13T11:25:26.972326+0000","last_fullsized":"2024-09-13T11:25:26.972326+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'5154","last_scrub_stamp":"2024-09-13T11:24:24.594848+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:24:24.594848+0000","objects_scrubbed":1720,"log_size":6166,"log_dups_size":0,"ondisk_log_size":6166,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:35.181590+0000","scrub_duration":343,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":106364928,"num_objects":1623,"num_object_clones":0,"num_object_copies":3246,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1623,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1017,"num_write_kb":29440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,13],"acting":[11,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.d","version":"166'6161","reported_seq":6512,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-13T11:25:26.002410+0000","last_change":"2024-09-13T11:24:24.967444+0000","last_active":"2024-09-13T11:25:26.002410+0000","last_peered":"2024-09-13T11:25:26.002410+0000","last_clean":"2024-09-13T11:25:26.002410+0000","last_became_active":"2024-09-13T11:24:24.087224+0000","last_became_peered":"2024-09-13T11:24:24.087224+0000","last_unstale":"2024-09-13T11:25:26.002410+0000","last_undegraded":"2024-09-13T11:25:26.002410+0000","last_fullsized":"2024-09-13T11:25:26.002410+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'5110","last_scrub_stamp":"2024-09-13T11:24:24.967337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:24:24.967337+0000","objects_scrubbed":1725,"log_size":6161,"log_dups_size":0,"ondisk_log_size":6161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:53.620949+0000","scrub_duration":255,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":107282432,"num_objects":1637,"num_object_clones":0,"num_object_copies":3274,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1637,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1054,"num_write_kb":30912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.e","version":"166'7290","reported_seq":12538,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-13T11:25:26.050702+0000","last_change":"2024-09-13T11:23:32.368856+0000","last_active":"2024-09-13T11:25:26.050702+0000","last_peered":"2024-09-13T11:25:26.050702+0000","last_clean":"2024-09-13T11:25:26.050702+0000","last_became_active":"2024-09-13T11:23:31.925869+0000","last_became_peered":"2024-09-13T11:23:31.925869+0000","last_unstale":"2024-09-13T11:25:26.050702+0000","last_undegraded":"2024-09-13T11:25:26.050702+0000","last_fullsized":"2024-09-13T11:25:26.050702+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'3527","last_scrub_stamp":"2024-09-13T11:23:32.368727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:32.368727+0000","objects_scrubbed":1774,"log_size":7290,"log_dups_size":0,"ondisk_log_size":7290,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":219,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":209321984,"num_objects":3194,"num_object_clones":0,"num_object_copies":6388,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3194,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3764,"num_write_kb":165888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"168'7308","reported_seq":7683,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-09-13T11:25:27.069976+0000","last_change":"2024-09-13T11:25:00.289940+0000","last_active":"2024-09-13T11:25:27.069976+0000","last_peered":"2024-09-13T11:25:27.069976+0000","last_clean":"2024-09-13T11:25:27.069976+0000","last_became_active":"2024-09-13T11:23:31.935239+0000","last_became_peered":"2024-09-13T11:23:31.935239+0000","last_unstale":"2024-09-13T11:25:27.069976+0000","last_undegraded":"2024-09-13T11:25:27.069976+0000","last_fullsized":"2024-09-13T11:25:27.069976+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"146'6154","last_scrub_stamp":"2024-09-13T11:25:00.289673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:00.289673+0000","objects_scrubbed":4394,"log_size":7308,"log_dups_size":0,"ondisk_log_size":7308,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:26:24.274856+0000","scrub_duration":1187,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":210894848,"num_objects":3218,"num_object_clones":0,"num_object_copies":6436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3218,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3888,"num_write_kb":170560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,12],"acting":[11,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.10","version":"166'6955","reported_seq":11477,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-13T11:25:26.029727+0000","last_change":"2024-09-13T11:25:10.161564+0000","last_active":"2024-09-13T11:25:26.029727+0000","last_peered":"2024-09-13T11:25:26.029727+0000","last_clean":"2024-09-13T11:25:26.029727+0000","last_became_active":"2024-09-13T11:23:41.931183+0000","last_became_peered":"2024-09-13T11:23:41.931183+0000","last_unstale":"2024-09-13T11:25:26.029727+0000","last_undegraded":"2024-09-13T11:25:26.029727+0000","last_fullsized":"2024-09-13T11:25:26.029727+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":5,"last_scrub":"151'6298","last_scrub_stamp":"2024-09-13T11:25:10.161464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:10.161464+0000","objects_scrubbed":3964,"log_size":6955,"log_dups_size":0,"ondisk_log_size":6955,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:26:33.015478+0000","scrub_duration":607,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":215744512,"num_objects":3292,"num_object_clones":0,"num_object_copies":6584,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3292,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3489,"num_write_kb":152000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.11","version":"166'6880","reported_seq":10174,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-13T11:25:26.042414+0000","last_change":"2024-09-13T11:25:24.287326+0000","last_active":"2024-09-13T11:25:26.042414+0000","last_peered":"2024-09-13T11:25:26.042414+0000","last_clean":"2024-09-13T11:25:26.042414+0000","last_became_active":"2024-09-13T11:23:39.063172+0000","last_became_peered":"2024-09-13T11:23:39.063172+0000","last_unstale":"2024-09-13T11:25:26.042414+0000","last_undegraded":"2024-09-13T11:25:26.042414+0000","last_fullsized":"2024-09-13T11:25:26.042414+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":5,"last_scrub":"165'6821","last_scrub_stamp":"2024-09-13T11:25:24.287082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:24.287082+0000","objects_scrubbed":3406,"log_size":6880,"log_dups_size":0,"ondisk_log_size":6880,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:26:49.405332+0000","scrub_duration":947,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":218497024,"num_objects":3334,"num_object_clones":0,"num_object_copies":6668,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3334,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3539,"num_write_kb":155584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11],"acting":[4,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.12","version":"166'6901","reported_seq":17004,"reported_epoch":166,"state":"active+clean+scrubbing","last_fresh":"2024-09-13T11:25:26.193477+0000","last_change":"2024-09-13T11:25:26.136263+0000","last_active":"2024-09-13T11:25:26.193477+0000","last_peered":"2024-09-13T11:25:26.193477+0000","last_clean":"2024-09-13T11:25:26.193477+0000","last_became_active":"2024-09-13T11:24:29.268229+0000","last_became_peered":"2024-09-13T11:24:29.268229+0000","last_unstale":"2024-09-13T11:25:26.193477+0000","last_undegraded":"2024-09-13T11:25:26.193477+0000","last_fullsized":"2024-09-13T11:25:26.193477+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'3109","last_scrub_stamp":"2024-09-13T11:23:35.111320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:35.111320+0000","objects_scrubbed":399,"log_size":6901,"log_dups_size":0,"ondisk_log_size":6901,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"scrubbing for 0s","scrub_duration":239,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":216662016,"num_objects":3306,"num_object_clones":0,"num_object_copies":6612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3306,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3792,"num_write_kb":171712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3457,"num_bytes_recovered":226557952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.13","version":"168'6980","reported_seq":10213,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-09-13T11:25:27.021692+0000","last_change":"2024-09-13T11:25:01.153044+0000","last_active":"2024-09-13T11:25:27.021692+0000","last_peered":"2024-09-13T11:25:27.021692+0000","last_clean":"2024-09-13T11:25:27.021692+0000","last_became_active":"2024-09-13T11:23:41.082433+0000","last_became_peered":"2024-09-13T11:23:41.082433+0000","last_unstale":"2024-09-13T11:25:27.021692+0000","last_undegraded":"2024-09-13T11:25:27.021692+0000","last_fullsized":"2024-09-13T11:25:27.021692+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":5,"last_scrub":"147'5921","last_scrub_stamp":"2024-09-13T11:25:01.152971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:01.152971+0000","objects_scrubbed":4311,"log_size":6980,"log_dups_size":0,"ondisk_log_size":6980,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:26:07.027631+0000","scrub_duration":1011,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":210436096,"num_objects":3211,"num_object_clones":0,"num_object_copies":6422,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3211,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3584,"num_write_kb":154048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7],"acting":[11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":46,"stat_sum":{"num_bytes":6824230936,"num_objects":104131,"num_object_clones":0,"num_object_copies":208262,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2396,"num_objects_unfound":0,"num_objects_dirty":104131,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":92565,"num_write_kb":3634433,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21016,"num_bytes_recovered":1377304576,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":15303843840,"data_stored":15303835696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":43282,"internal_metadata":0},"log_size":285949,"ondisk_log_size":285949,"up":92,"acting":92,"num_store_stats":16},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":11},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":112,"ondisk_log_size":112,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":13,"up_from":23,"seq":98784247851,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":848884,"kb_used_data":741284,"kb_used_omap":1,"kb_used_meta":107518,"kb_avail":93522956,"statfs":{"total":96636764160,"available":95767506944,"internally_reserved":0,"allocated":759074816,"data_stored":758332501,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":110098885},"hb_peers":[0,1,2,4,6,8,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":23,"seq":98784247850,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":681240,"kb_used_data":618056,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":93690600,"statfs":{"total":96636764160,"available":95939174400,"internally_reserved":0,"allocated":632889344,"data_stored":632178825,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":64616896},"hb_peers":[0,1,3,8,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":40,"seq":171798691873,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1171488,"kb_used_data":1067280,"kb_used_omap":4,"kb_used_meta":104187,"kb_avail":93200352,"statfs":{"total":96636764160,"available":95437160448,"internally_reserved":0,"allocated":1092894720,"data_stored":1092165338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4433,"internal_metadata":106688175},"hb_peers":[0,2,3,4,5,6,7,8,9,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":21,"seq":90194313259,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1159400,"kb_used_data":1056024,"kb_used_omap":1,"kb_used_meta":103358,"kb_avail":93212440,"statfs":{"total":96636764160,"available":95449538560,"internally_reserved":0,"allocated":1081368576,"data_stored":1080601551,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":105839047},"hb_peers":[0,2,4,5,7,9,10,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":21,"seq":90194313261,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1257128,"kb_used_data":1175960,"kb_used_omap":1,"kb_used_meta":81150,"kb_avail":93114712,"statfs":{"total":96636764160,"available":95349465088,"internally_reserved":0,"allocated":1204183040,"data_stored":1203447594,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":83098055},"hb_peers":[0,2,4,6,7,8,9,11,12,13],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":22,"seq":94489280553,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1307644,"kb_used_data":1200108,"kb_used_omap":1,"kb_used_meta":107518,"kb_avail":93064196,"statfs":{"total":96636764160,"available":95297736704,"internally_reserved":0,"allocated":1228910592,"data_stored":1228223837,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":110098885},"hb_peers":[1,3,4,5,7,8,10,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":22,"seq":94489280554,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":602472,"kb_used_data":526872,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":93769368,"statfs":{"total":96636764160,"available":96019832832,"internally_reserved":0,"allocated":539516928,"data_stored":538792320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,1,3,4,5,7,8,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":15,"up_from":22,"seq":94489280554,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1227888,"kb_used_data":1116192,"kb_used_omap":1,"kb_used_meta":111678,"kb_avail":93143952,"statfs":{"total":96636764160,"available":95379406848,"internally_reserved":0,"allocated":1142980608,"data_stored":1142305477,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":114358726},"hb_peers":[0,4,5,6,7,8,10,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":22,"seq":94489280555,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":806068,"kb_used_data":742948,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":93565772,"statfs":{"total":96636764160,"available":95811350528,"internally_reserved":0,"allocated":760778752,"data_stored":760043385,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[3,5,6,7,9,10,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":22,"seq":94489280554,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":589264,"kb_used_data":535808,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":93782576,"statfs":{"total":96636764160,"available":96033357824,"internally_reserved":0,"allocated":548667392,"data_stored":547963100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54655430},"hb_peers":[0,1,3,5,7,8,10,11,12,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":22,"seq":94489280554,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1467536,"kb_used_data":1364160,"kb_used_omap":1,"kb_used_meta":103358,"kb_avail":92904304,"statfs":{"total":96636764160,"available":95134007296,"internally_reserved":0,"allocated":1396899840,"data_stored":1396192229,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":105839050},"hb_peers":[0,3,5,6,8,9,10,11,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":22,"seq":94489280554,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":566340,"kb_used_data":494900,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":93805500,"statfs":{"total":96636764160,"available":96056832000,"internally_reserved":0,"allocated":506777600,"data_stored":506022479,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73136581},"hb_peers":[1,5,6,7,9,10,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":22,"seq":94489280553,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1018012,"kb_used_data":946572,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":93353828,"statfs":{"total":96636764160,"available":95594319872,"internally_reserved":0,"allocated":969289728,"data_stored":968573918,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":73136576},"hb_peers":[0,1,3,5,7,9,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":11,"up_from":22,"seq":94489280555,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1459852,"kb_used_data":1348156,"kb_used_omap":1,"kb_used_meta":111678,"kb_avail":92911988,"statfs":{"total":96636764160,"available":95141875712,"internally_reserved":0,"allocated":1380511744,"data_stored":1379763476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":114358724},"hb_peers":[0,1,2,4,7,8,10,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1,0,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":12,"up_from":22,"seq":94489280554,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1280248,"kb_used_data":1168552,"kb_used_omap":1,"kb_used_meta":111678,"kb_avail":93091592,"statfs":{"total":96636764160,"available":95325790208,"internally_reserved":0,"allocated":1196597248,"data_stored":1195854952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":114358725},"hb_peers":[0,1,2,3,4,5,10,11,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":14,"up_from":0,"seq":609885356040,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1226309632,"data_stored":1226309632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1082654720,"data_stored":1082654720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":534515712,"data_stored":534511640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1077346304,"data_stored":1077346304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":752025600,"data_stored":752025600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1200947200,"data_stored":1200947200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":542900224,"data_stored":542900224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1394081792,"data_stored":1394081792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":504627200,"data_stored":504627200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":629866496,"data_stored":629866496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":955252736,"data_stored":955252736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":1378156544,"data_stored":1378156544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":1183645696,"data_stored":1183645696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":750125056,"data_stored":750125056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":951386112,"data_stored":951386112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":43282,"internal_metadata":0},{"poolid":3,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":1140002816,"data_stored":1139998744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T11:25:32.104 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_ops_in_flight 2024-09-13T11:25:32.201 INFO:tasks.ceph.osd.8.smithi001.stderr:2024-09-13T11:25:32.199+0000 7fb9b48ae640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26312) UID: 0 2024-09-13T11:25:32.240 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-13T11:25:32.240 INFO:teuthology.orchestra.run.smithi001.stdout: "ops": [ 2024-09-13T11:25:32.240 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.241 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:183414 3.20 3:04e8e109:::benchmark_data_smithi177_31116_object43111:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e170)", 2024-09-13T11:25:32.241 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:25:32.234645+0000", 2024-09-13T11:25:32.241 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 0.0054157440000000001, 2024-09-13T11:25:32.241 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0054339849999999997, 2024-09-13T11:25:32.241 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:32.241 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:32.241 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "waiting for sub ops", 2024-09-13T11:25:32.241 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:25:32.241 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:25:32.241 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:32.241 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 183414 2024-09-13T11:25:32.241 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.241 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:32.241 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.242 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:32.242 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:32.234645+0000", 2024-09-13T11:25:32.242 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:32.242 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.242 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.242 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:32.242 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:32.234645+0000", 2024-09-13T11:25:32.242 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:32.242 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.242 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.242 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:32.242 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:32.234647+0000", 2024-09-13T11:25:32.242 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.928e-06 2024-09-13T11:25:32.242 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.243 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.243 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:32.243 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:32.234651+0000", 2024-09-13T11:25:32.243 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.673e-06 2024-09-13T11:25:32.243 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.243 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.243 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:32.243 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:32.234665+0000", 2024-09-13T11:25:32.243 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.3916e-05 2024-09-13T11:25:32.243 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.243 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.243 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:32.243 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:32.234674+0000", 2024-09-13T11:25:32.243 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.5979999999999997e-06 2024-09-13T11:25:32.244 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.244 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.244 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:32.244 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:32.234723+0000", 2024-09-13T11:25:32.244 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.9014999999999997e-05 2024-09-13T11:25:32.244 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.244 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.244 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:25:32.244 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:32.234806+0000", 2024-09-13T11:25:32.244 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.3203999999999998e-05 2024-09-13T11:25:32.244 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.244 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.244 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 0", 2024-09-13T11:25:32.244 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:32.234869+0000", 2024-09-13T11:25:32.245 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.2989000000000003e-05 2024-09-13T11:25:32.245 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.245 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.245 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:25:32.245 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:32.236426+0000", 2024-09-13T11:25:32.245 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0015570650000000001 2024-09-13T11:25:32.245 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:32.245 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:32.245 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:32.245 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:32.245 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-09-13T11:25:32.245 INFO:teuthology.orchestra.run.smithi001.stdout: "num_ops": 1 2024-09-13T11:25:32.245 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-13T11:25:32.253 INFO:teuthology.orchestra.run.smithi001.stderr:noscrub is unset 2024-09-13T11:25:32.257 INFO:tasks.ceph.osd.5.smithi092.stderr:2024-09-13T11:25:32.253+0000 7f6f02136640 -1 osd.5 170 handle_osd_map injecting map crc failure 2024-09-13T11:25:32.259 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_blocked_ops 2024-09-13T11:25:32.259 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-09-13T11:25:32.256+0000 7f97b59fa640 -1 osd.6 170 handle_osd_map injecting map crc failure 2024-09-13T11:25:32.260 INFO:tasks.ceph.osd.5.smithi092.stderr:2024-09-13T11:25:32.253+0000 7f6f02136640 -1 osd.5 170 handle_osd_map injecting map crc failure 2024-09-13T11:25:32.301 INFO:tasks.ceph.osd.3.smithi177.stderr:2024-09-13T11:25:32.298+0000 7f10adba4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26574) UID: 0 2024-09-13T11:25:32.379 INFO:tasks.ceph.osd.14.smithi162.stderr:2024-09-13T11:25:32.376+0000 7f3dea330640 -1 osd.14 170 handle_osd_map injecting map crc failure 2024-09-13T11:25:32.403 INFO:tasks.ceph.osd.10.smithi162.stderr:2024-09-13T11:25:32.400+0000 7fb59c9f0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26317) UID: 0 2024-09-13T11:25:32.462 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-13T11:25:32.462 INFO:teuthology.orchestra.run.smithi001.stdout: "ops": [], 2024-09-13T11:25:32.462 INFO:teuthology.orchestra.run.smithi001.stdout: "complaint_time": 30, 2024-09-13T11:25:32.462 INFO:teuthology.orchestra.run.smithi001.stdout: "num_blocked_ops": 0 2024-09-13T11:25:32.462 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-13T11:25:32.478 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_historic_ops 2024-09-13T11:25:32.503 INFO:tasks.ceph.osd.13.smithi092.stderr:2024-09-13T11:25:32.497+0000 7f3e5afea640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26350) UID: 0 2024-09-13T11:25:32.604 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-13T11:25:32.605 INFO:teuthology.orchestra.run.smithi001.stdout: "size": 20, 2024-09-13T11:25:32.605 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 600, 2024-09-13T11:25:32.605 INFO:teuthology.orchestra.run.smithi001.stdout: "ops": [ 2024-09-13T11:25:32.605 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.605 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop_reply(client.4665.0:167 3.9 e41/30 ondisk, result = 0)", 2024-09-13T11:25:32.605 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:22:47.419696+0000", 2024-09-13T11:25:32.605 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 165.182493943, 2024-09-13T11:25:32.605 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.64615778199999996, 2024-09-13T11:25:32.605 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:32.605 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:32.606 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:25:32.606 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:32.606 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.606 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:32.606 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:47.419696+0000", 2024-09-13T11:25:32.606 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:32.606 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.606 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.606 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:32.606 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:47.419696+0000", 2024-09-13T11:25:32.606 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:32.607 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.607 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.607 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:32.607 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:47.419697+0000", 2024-09-13T11:25:32.607 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.15e-06 2024-09-13T11:25:32.607 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.607 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.607 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:32.607 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:47.419700+0000", 2024-09-13T11:25:32.607 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.2449999999999998e-06 2024-09-13T11:25:32.607 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.607 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.608 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:32.608 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:48.065637+0000", 2024-09-13T11:25:32.608 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.64593674300000004 2024-09-13T11:25:32.608 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.608 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.608 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:32.608 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:48.065648+0000", 2024-09-13T11:25:32.608 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0865e-05 2024-09-13T11:25:32.608 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.608 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.608 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:32.609 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:48.065720+0000", 2024-09-13T11:25:32.609 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.2020999999999994e-05 2024-09-13T11:25:32.609 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.609 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.609 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:25:32.609 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:48.065728+0000", 2024-09-13T11:25:32.609 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.7660000000000004e-06 2024-09-13T11:25:32.610 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.610 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.610 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:32.610 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:48.065853+0000", 2024-09-13T11:25:32.610 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00012499200000000001 2024-09-13T11:25:32.610 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:32.610 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:32.610 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:32.610 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.610 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.611 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:25383 3.0 3:0a6bb869:::benchmark_data_smithi177_31116_object25382:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e55)", 2024-09-13T11:25:32.611 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:04.447376+0000", 2024-09-13T11:25:32.611 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 148.15481327399999, 2024-09-13T11:25:32.611 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.21239864, 2024-09-13T11:25:32.611 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:32.611 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:32.611 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:32.611 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:25:32.611 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:25:32.611 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:32.611 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 25383 2024-09-13T11:25:32.612 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.612 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:32.612 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.612 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:32.612 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.447376+0000", 2024-09-13T11:25:32.612 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:32.612 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.612 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.612 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:32.613 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.447376+0000", 2024-09-13T11:25:32.613 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:32.613 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.613 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.613 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:32.613 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.447378+0000", 2024-09-13T11:25:32.613 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.6729999999999999e-06 2024-09-13T11:25:32.613 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.613 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.613 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:32.613 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.447433+0000", 2024-09-13T11:25:32.614 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.5371000000000002e-05 2024-09-13T11:25:32.614 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.614 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.614 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:32.614 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.447446+0000", 2024-09-13T11:25:32.614 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.3179999999999999e-05 2024-09-13T11:25:32.614 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.614 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.614 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:32.614 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.447456+0000", 2024-09-13T11:25:32.614 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0008e-05 2024-09-13T11:25:32.615 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.615 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.615 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:32.615 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.447514+0000", 2024-09-13T11:25:32.615 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.8023999999999999e-05 2024-09-13T11:25:32.615 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.615 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.615 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:25:32.615 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.447642+0000", 2024-09-13T11:25:32.615 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000127296 2024-09-13T11:25:32.615 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.616 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.616 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 15", 2024-09-13T11:25:32.616 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.447767+0000", 2024-09-13T11:25:32.616 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000125547 2024-09-13T11:25:32.616 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.616 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.616 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:25:32.616 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.449188+0000", 2024-09-13T11:25:32.616 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.001420604 2024-09-13T11:25:32.616 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.616 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.617 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:25:32.617 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.659668+0000", 2024-09-13T11:25:32.617 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.210479685 2024-09-13T11:25:32.617 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.617 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.617 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:25:32.617 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.659716+0000", 2024-09-13T11:25:32.617 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.8609999999999997e-05 2024-09-13T11:25:32.617 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.617 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.617 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:32.618 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.659775+0000", 2024-09-13T11:25:32.618 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.8641999999999999e-05 2024-09-13T11:25:32.618 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:32.618 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:32.618 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:32.618 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.618 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.618 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4665.0:25408 3.4 e56/40 3:294e23bd:::benchmark_data_smithi177_31116_object25407:head v 56'1645, mlcod=55'1642)", 2024-09-13T11:25:32.618 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:04.494686+0000", 2024-09-13T11:25:32.619 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 148.10750361500001, 2024-09-13T11:25:32.619 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.29725566399999998, 2024-09-13T11:25:32.619 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:32.619 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:32.619 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:32.619 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:32.619 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.619 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:32.620 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.494686+0000", 2024-09-13T11:25:32.620 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:32.620 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.620 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.620 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:32.620 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.494686+0000", 2024-09-13T11:25:32.620 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:32.620 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.620 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.620 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:32.621 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.494686+0000", 2024-09-13T11:25:32.621 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.2399999999999999e-07 2024-09-13T11:25:32.621 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.621 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.621 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:32.621 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.494809+0000", 2024-09-13T11:25:32.621 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00012308600000000001 2024-09-13T11:25:32.621 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.621 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.621 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:32.622 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.789410+0000", 2024-09-13T11:25:32.622 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.29460066899999998 2024-09-13T11:25:32.622 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.622 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.622 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:32.622 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.789425+0000", 2024-09-13T11:25:32.622 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.4600000000000001e-05 2024-09-13T11:25:32.622 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.622 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.622 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:32.623 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.789507+0000", 2024-09-13T11:25:32.623 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.2785000000000001e-05 2024-09-13T11:25:32.623 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.623 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.624 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:25:32.624 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.789525+0000", 2024-09-13T11:25:32.624 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.8117999999999998e-05 2024-09-13T11:25:32.624 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.624 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.624 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:25:32.624 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.791852+0000", 2024-09-13T11:25:32.624 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0023264459999999998 2024-09-13T11:25:32.624 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.625 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.625 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:32.625 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.791942+0000", 2024-09-13T11:25:32.625 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.9635999999999994e-05 2024-09-13T11:25:32.625 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:32.625 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:32.625 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:32.625 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.625 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.625 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4665.0:30815 3.f e59/30 3:fde17cb1:::benchmark_data_smithi177_31116_object30814:head v 59'1919, mlcod=59'1914)", 2024-09-13T11:25:32.625 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:09.494342+0000", 2024-09-13T11:25:32.625 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 143.10784762899999, 2024-09-13T11:25:32.625 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.22787855700000001, 2024-09-13T11:25:32.626 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:32.626 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:32.626 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:32.626 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:32.626 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.626 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:32.626 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.494342+0000", 2024-09-13T11:25:32.626 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:32.626 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.626 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.626 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:32.626 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.494342+0000", 2024-09-13T11:25:32.626 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:32.627 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.627 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.627 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:32.627 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.494343+0000", 2024-09-13T11:25:32.627 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.251e-06 2024-09-13T11:25:32.627 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.627 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.627 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:32.627 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.494387+0000", 2024-09-13T11:25:32.627 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.4276000000000001e-05 2024-09-13T11:25:32.627 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.627 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.627 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:32.628 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.720454+0000", 2024-09-13T11:25:32.628 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.226066614 2024-09-13T11:25:32.628 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.628 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.628 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:32.628 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.720462+0000", 2024-09-13T11:25:32.628 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.9240000000000007e-06 2024-09-13T11:25:32.628 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.628 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.628 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:32.628 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.720537+0000", 2024-09-13T11:25:32.628 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.4967000000000002e-05 2024-09-13T11:25:32.628 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.628 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.629 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:25:32.629 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.720563+0000", 2024-09-13T11:25:32.629 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.5630999999999999e-05 2024-09-13T11:25:32.629 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.629 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.629 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:25:32.629 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.722183+0000", 2024-09-13T11:25:32.629 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0016209670000000001 2024-09-13T11:25:32.629 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.629 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.629 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:32.629 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.722220+0000", 2024-09-13T11:25:32.629 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.6927000000000001e-05 2024-09-13T11:25:32.629 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:32.630 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:32.630 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:32.630 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.630 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.630 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4665.0:30828 3.f e59/30 3:f3a9c6a9:::benchmark_data_smithi177_31116_object30827:head v 59'1920, mlcod=59'1916)", 2024-09-13T11:25:32.630 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:09.502607+0000", 2024-09-13T11:25:32.630 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 143.099582572, 2024-09-13T11:25:32.630 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.26374408300000002, 2024-09-13T11:25:32.630 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:32.630 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:32.630 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:32.630 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:32.630 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.631 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:32.631 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.502607+0000", 2024-09-13T11:25:32.631 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:32.631 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.631 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.631 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:32.631 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.502607+0000", 2024-09-13T11:25:32.631 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:32.631 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.631 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.631 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:32.631 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.502609+0000", 2024-09-13T11:25:32.631 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.84e-06 2024-09-13T11:25:32.632 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.632 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.632 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:32.632 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.502663+0000", 2024-09-13T11:25:32.632 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.3971000000000002e-05 2024-09-13T11:25:32.632 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.632 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.632 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:32.632 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.763912+0000", 2024-09-13T11:25:32.632 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.26124904300000001 2024-09-13T11:25:32.632 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.632 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.632 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:32.633 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.763920+0000", 2024-09-13T11:25:32.633 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.1149999999999994e-06 2024-09-13T11:25:32.633 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.633 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.633 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:32.633 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.764024+0000", 2024-09-13T11:25:32.633 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00010380899999999999 2024-09-13T11:25:32.633 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.633 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.633 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:25:32.633 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.764068+0000", 2024-09-13T11:25:32.633 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.4638999999999997e-05 2024-09-13T11:25:32.633 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.634 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.634 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:25:32.634 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.766308+0000", 2024-09-13T11:25:32.634 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0022392039999999998 2024-09-13T11:25:32.634 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.634 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.634 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:32.634 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.766351+0000", 2024-09-13T11:25:32.634 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.3461999999999998e-05 2024-09-13T11:25:32.634 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:32.634 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:32.634 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:32.634 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.634 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.635 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDPGPush(3.1b 84/78 [PushOp(3:dcddd139:::benchmark_data_smithi177_31116_object10384:head, version: 48'673, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:dcddd139:::benchmark_data_smithi177_31116_object10384:head@48'673, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:25:32.635 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:41.400602+0000", 2024-09-13T11:25:32.635 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 111.20158708699999, 2024-09-13T11:25:32.635 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.43365821999999998, 2024-09-13T11:25:32.635 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:32.635 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:32.635 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:25:32.635 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:32.635 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.635 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:32.635 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:41.400602+0000", 2024-09-13T11:25:32.635 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:32.635 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.635 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.636 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:32.636 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:41.400602+0000", 2024-09-13T11:25:32.636 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:32.636 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.636 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.636 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:32.636 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:41.400604+0000", 2024-09-13T11:25:32.636 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.99e-06 2024-09-13T11:25:32.636 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.636 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.636 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:32.636 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:41.400653+0000", 2024-09-13T11:25:32.636 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.8195000000000002e-05 2024-09-13T11:25:32.637 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.637 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.637 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:32.637 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:41.833180+0000", 2024-09-13T11:25:32.637 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.43252769200000002 2024-09-13T11:25:32.637 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.637 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.637 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:32.637 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:41.833213+0000", 2024-09-13T11:25:32.637 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.2737999999999997e-05 2024-09-13T11:25:32.637 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.637 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.637 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:32.638 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:41.833408+0000", 2024-09-13T11:25:32.638 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00019473799999999999 2024-09-13T11:25:32.638 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.638 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.638 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:25:32.638 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:41.833417+0000", 2024-09-13T11:25:32.638 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.5349999999999993e-06 2024-09-13T11:25:32.638 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.638 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.638 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:32.638 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:41.834261+0000", 2024-09-13T11:25:32.638 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00084333200000000002 2024-09-13T11:25:32.638 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:32.639 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:32.639 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:32.639 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.639 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.639 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDPGPushReply(3.20 85/77 [PushReplyOp(3:066982a6:::benchmark_data_smithi177_31116_object16400:head)])", 2024-09-13T11:25:32.639 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:42.671551+0000", 2024-09-13T11:25:32.639 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 109.930638264, 2024-09-13T11:25:32.639 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.28249085499999999, 2024-09-13T11:25:32.639 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:32.639 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:32.639 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-13T11:25:32.639 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:32.639 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.639 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:32.640 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.671551+0000", 2024-09-13T11:25:32.640 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:32.640 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.640 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.640 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:32.640 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.671551+0000", 2024-09-13T11:25:32.640 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:32.640 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.641 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.641 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:32.641 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.671552+0000", 2024-09-13T11:25:32.641 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.097e-06 2024-09-13T11:25:32.641 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.641 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.641 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:32.641 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.671556+0000", 2024-09-13T11:25:32.641 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.8519999999999997e-06 2024-09-13T11:25:32.641 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.641 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.641 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:32.641 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.951769+0000", 2024-09-13T11:25:32.641 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.28021293000000003 2024-09-13T11:25:32.642 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.642 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.642 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:32.642 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.951780+0000", 2024-09-13T11:25:32.642 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.1059e-05 2024-09-13T11:25:32.642 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.642 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.642 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:32.642 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.951992+0000", 2024-09-13T11:25:32.642 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000211478 2024-09-13T11:25:32.642 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.642 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.642 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:32.642 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.954042+0000", 2024-09-13T11:25:32.643 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0020504389999999998 2024-09-13T11:25:32.643 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:32.643 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:32.643 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:32.643 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.643 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.643 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDPGPush(3.1b 89/78 [PushOp(3:dc4c79e7:::benchmark_data_smithi177_31116_object19857:head, version: 54'1266, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:dc4c79e7:::benchmark_data_smithi177_31116_object19857:head@54'1266, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:25:32.643 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:46.649128+0000", 2024-09-13T11:25:32.643 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 105.953061156, 2024-09-13T11:25:32.643 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.29963603, 2024-09-13T11:25:32.643 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:32.643 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:32.643 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:25:32.644 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:32.644 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.644 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:32.644 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.649128+0000", 2024-09-13T11:25:32.644 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:32.644 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.644 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.644 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:32.644 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.649128+0000", 2024-09-13T11:25:32.644 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:32.644 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.644 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.644 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:32.645 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.649130+0000", 2024-09-13T11:25:32.645 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2950000000000001e-06 2024-09-13T11:25:32.645 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.645 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.645 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:32.645 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.649197+0000", 2024-09-13T11:25:32.645 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.7033999999999995e-05 2024-09-13T11:25:32.645 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.645 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.645 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:32.645 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.947691+0000", 2024-09-13T11:25:32.646 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.29849387300000002 2024-09-13T11:25:32.646 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.646 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.646 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:32.646 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.947702+0000", 2024-09-13T11:25:32.646 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.1239e-05 2024-09-13T11:25:32.646 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.646 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.646 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:32.646 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.947799+0000", 2024-09-13T11:25:32.646 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.7283999999999999e-05 2024-09-13T11:25:32.646 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.646 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.646 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:25:32.647 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.947809+0000", 2024-09-13T11:25:32.647 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.008e-05 2024-09-13T11:25:32.647 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.647 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.647 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:32.647 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.948764+0000", 2024-09-13T11:25:32.647 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00095522499999999998 2024-09-13T11:25:32.647 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:32.647 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:32.647 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:32.647 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.647 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.647 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:71927 3.0 3:0197244a:::benchmark_data_smithi177_31116_object71926:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2024-09-13T11:25:32.647 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:54.199022+0000", 2024-09-13T11:25:32.648 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 98.403167800000006, 2024-09-13T11:25:32.648 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.26819617800000001, 2024-09-13T11:25:32.648 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:32.648 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:32.648 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:32.648 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:25:32.648 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:25:32.648 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:32.648 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 71927 2024-09-13T11:25:32.648 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.648 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:32.648 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.648 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:32.648 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:54.199022+0000", 2024-09-13T11:25:32.649 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:32.649 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.649 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.649 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:32.649 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:54.199022+0000", 2024-09-13T11:25:32.649 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:32.649 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.649 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.649 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:32.649 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:54.199025+0000", 2024-09-13T11:25:32.649 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.2200000000000001e-06 2024-09-13T11:25:32.649 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.649 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.649 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:32.650 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:54.199076+0000", 2024-09-13T11:25:32.650 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.1184000000000003e-05 2024-09-13T11:25:32.650 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.650 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.650 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:32.650 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:54.199095+0000", 2024-09-13T11:25:32.650 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.9253999999999999e-05 2024-09-13T11:25:32.650 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.650 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.650 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:32.650 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:54.199107+0000", 2024-09-13T11:25:32.650 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.1916000000000001e-05 2024-09-13T11:25:32.650 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.650 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.651 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:32.651 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:54.199196+0000", 2024-09-13T11:25:32.651 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.8645000000000003e-05 2024-09-13T11:25:32.651 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.651 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.651 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:25:32.651 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:54.199421+0000", 2024-09-13T11:25:32.651 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000224868 2024-09-13T11:25:32.651 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.651 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.651 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 15", 2024-09-13T11:25:32.651 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:54.199647+0000", 2024-09-13T11:25:32.651 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00022666099999999999 2024-09-13T11:25:32.651 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.651 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.652 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:25:32.652 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:54.201333+0000", 2024-09-13T11:25:32.652 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.001685761 2024-09-13T11:25:32.652 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.652 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.652 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:25:32.652 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:54.467097+0000", 2024-09-13T11:25:32.652 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.26576393399999998 2024-09-13T11:25:32.652 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.652 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.652 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:25:32.652 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:54.467163+0000", 2024-09-13T11:25:32.652 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.6088999999999997e-05 2024-09-13T11:25:32.653 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.653 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.653 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:32.653 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:54.467218+0000", 2024-09-13T11:25:32.653 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.4645999999999998e-05 2024-09-13T11:25:32.653 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:32.653 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:32.653 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:32.653 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.653 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.653 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4665.0:98479 3.2 e113/75 3:41084616:::benchmark_data_smithi177_31116_object98478:head v 113'3785, mlcod=113'3782)", 2024-09-13T11:25:32.653 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:24:15.343730+0000", 2024-09-13T11:25:32.653 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 77.258459529999996, 2024-09-13T11:25:32.653 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.196889648, 2024-09-13T11:25:32.654 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:32.654 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:32.654 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:32.654 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:32.654 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.654 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:32.654 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.343730+0000", 2024-09-13T11:25:32.654 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:32.654 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.654 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.654 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:32.654 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.343730+0000", 2024-09-13T11:25:32.654 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:32.655 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.655 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.655 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:32.655 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.343732+0000", 2024-09-13T11:25:32.655 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.6819999999999999e-06 2024-09-13T11:25:32.655 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.655 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.655 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:32.655 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.343781+0000", 2024-09-13T11:25:32.655 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.9691999999999998e-05 2024-09-13T11:25:32.655 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.655 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.655 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:32.656 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.538204+0000", 2024-09-13T11:25:32.656 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.19442271699999999 2024-09-13T11:25:32.656 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.656 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.656 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:32.656 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.538215+0000", 2024-09-13T11:25:32.656 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0648e-05 2024-09-13T11:25:32.656 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.656 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.656 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:32.656 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.538342+0000", 2024-09-13T11:25:32.656 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000127448 2024-09-13T11:25:32.656 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.657 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.657 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:25:32.657 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.538395+0000", 2024-09-13T11:25:32.657 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.2895000000000001e-05 2024-09-13T11:25:32.657 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.657 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.657 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:25:32.657 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.540544+0000", 2024-09-13T11:25:32.657 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0021487379999999999 2024-09-13T11:25:32.657 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.657 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.657 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:32.657 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.540620+0000", 2024-09-13T11:25:32.658 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.5828000000000007e-05 2024-09-13T11:25:32.658 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:32.658 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:32.658 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:32.658 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.658 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.658 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4665.0:98954 3.2 e113/75 3:416baeb7:::benchmark_data_smithi177_31116_object98953:head v 113'3790, mlcod=113'3787)", 2024-09-13T11:25:32.658 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:24:15.819065+0000", 2024-09-13T11:25:32.658 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 76.783124647999998, 2024-09-13T11:25:32.658 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.187742985, 2024-09-13T11:25:32.658 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:32.658 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:32.658 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:32.659 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:32.659 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.659 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:32.659 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.819065+0000", 2024-09-13T11:25:32.659 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:32.659 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.659 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.659 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:32.659 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.819065+0000", 2024-09-13T11:25:32.659 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:32.659 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.659 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.659 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:32.660 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.819068+0000", 2024-09-13T11:25:32.660 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.0469999999999998e-06 2024-09-13T11:25:32.660 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.660 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.660 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:32.660 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.819124+0000", 2024-09-13T11:25:32.660 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.6335999999999997e-05 2024-09-13T11:25:32.660 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.660 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.660 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:32.660 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:16.003736+0000", 2024-09-13T11:25:32.660 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.18461164899999999 2024-09-13T11:25:32.660 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.661 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.661 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:32.661 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:16.003749+0000", 2024-09-13T11:25:32.661 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.3157e-05 2024-09-13T11:25:32.661 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.661 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.661 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:32.661 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:16.003903+0000", 2024-09-13T11:25:32.661 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00015374799999999999 2024-09-13T11:25:32.661 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.661 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.661 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:25:32.661 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:16.003961+0000", 2024-09-13T11:25:32.661 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.8026000000000003e-05 2024-09-13T11:25:32.662 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.662 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.662 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:25:32.662 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:16.006699+0000", 2024-09-13T11:25:32.662 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0027384290000000001 2024-09-13T11:25:32.662 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.662 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.662 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:32.662 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:16.006808+0000", 2024-09-13T11:25:32.662 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000108593 2024-09-13T11:25:32.663 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:32.663 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:32.663 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:32.663 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.663 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.663 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4665.0:99149 3.2 e113/75 3:4350ea1a:::benchmark_data_smithi177_31116_object99148:head v 113'3791, mlcod=113'3788)", 2024-09-13T11:25:32.663 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:24:16.003634+0000", 2024-09-13T11:25:32.663 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 76.598555126999997, 2024-09-13T11:25:32.663 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.19660476800000001, 2024-09-13T11:25:32.663 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:32.663 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:32.663 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:32.663 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:32.664 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.664 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:32.664 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:16.003634+0000", 2024-09-13T11:25:32.664 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:32.664 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.664 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.664 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:32.664 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:16.003634+0000", 2024-09-13T11:25:32.665 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:32.665 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.665 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.665 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:32.665 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:16.003637+0000", 2024-09-13T11:25:32.665 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.9749999999999999e-06 2024-09-13T11:25:32.665 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.665 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.665 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:32.665 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:16.003701+0000", 2024-09-13T11:25:32.665 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.3764999999999997e-05 2024-09-13T11:25:32.665 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.665 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.665 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:32.666 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:16.198858+0000", 2024-09-13T11:25:32.666 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.19515649900000001 2024-09-13T11:25:32.666 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.666 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.666 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:32.666 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:16.198863+0000", 2024-09-13T11:25:32.666 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.2229999999999998e-06 2024-09-13T11:25:32.666 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.666 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.666 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:32.666 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:16.198918+0000", 2024-09-13T11:25:32.666 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.5427000000000003e-05 2024-09-13T11:25:32.667 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.667 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.667 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:25:32.667 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:16.198935+0000", 2024-09-13T11:25:32.667 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.6776e-05 2024-09-13T11:25:32.667 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.667 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.667 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:25:32.667 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:16.200208+0000", 2024-09-13T11:25:32.667 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0012725429999999999 2024-09-13T11:25:32.667 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.667 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.667 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:32.668 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:16.200239+0000", 2024-09-13T11:25:32.668 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.1560000000000003e-05 2024-09-13T11:25:32.668 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:32.668 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:32.668 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:32.668 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.668 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.668 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4665.0:101036 3.4 e114/67 3:207f4fe8:::benchmark_data_smithi177_31116_object101035:head v 114'4508, mlcod=114'4505)", 2024-09-13T11:25:32.668 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:24:17.391594+0000", 2024-09-13T11:25:32.668 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 75.210595751, 2024-09-13T11:25:32.668 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.19061766999999999, 2024-09-13T11:25:32.668 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:32.668 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:32.669 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:32.669 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:32.669 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.669 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:32.669 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:17.391594+0000", 2024-09-13T11:25:32.669 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:32.669 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.669 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.669 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:32.669 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:17.391594+0000", 2024-09-13T11:25:32.669 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:32.669 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.670 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.670 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:32.670 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:17.391595+0000", 2024-09-13T11:25:32.670 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.6729999999999999e-06 2024-09-13T11:25:32.670 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.670 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.670 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:32.670 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:17.391668+0000", 2024-09-13T11:25:32.670 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.2096999999999998e-05 2024-09-13T11:25:32.670 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.670 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.670 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:32.670 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:17.579638+0000", 2024-09-13T11:25:32.671 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.187969993 2024-09-13T11:25:32.671 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.671 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.671 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:32.671 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:17.579644+0000", 2024-09-13T11:25:32.671 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.2049999999999999e-06 2024-09-13T11:25:32.671 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.671 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.671 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:32.671 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:17.579707+0000", 2024-09-13T11:25:32.671 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.2948999999999996e-05 2024-09-13T11:25:32.671 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.671 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.672 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:25:32.672 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:17.579731+0000", 2024-09-13T11:25:32.672 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.4315999999999999e-05 2024-09-13T11:25:32.672 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.672 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.672 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:25:32.672 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:17.582172+0000", 2024-09-13T11:25:32.672 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.002440705 2024-09-13T11:25:32.672 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.672 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.672 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:32.672 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:17.582211+0000", 2024-09-13T11:25:32.672 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.9731999999999998e-05 2024-09-13T11:25:32.672 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:32.673 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:32.673 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:32.673 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.673 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.673 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDPGPush(3.25 121/120 [PushOp(3:a5de2229:::benchmark_data_smithi177_31116_object1005:head, version: 43'62, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:a5de2229:::benchmark_data_smithi177_31116_object1005:head@43'62, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:25:32.673 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:24:27.379880+0000", 2024-09-13T11:25:32.673 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 65.222309961999997, 2024-09-13T11:25:32.673 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.72647719499999996, 2024-09-13T11:25:32.673 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:32.673 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:32.673 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:25:32.673 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:32.674 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.674 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:32.674 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:27.379880+0000", 2024-09-13T11:25:32.674 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:32.674 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.674 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.674 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:32.674 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:27.379880+0000", 2024-09-13T11:25:32.674 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:32.674 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.674 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.674 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:32.674 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:27.379882+0000", 2024-09-13T11:25:32.674 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.1050000000000002e-06 2024-09-13T11:25:32.675 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.675 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.675 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:32.675 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:27.379976+0000", 2024-09-13T11:25:32.675 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.4468000000000004e-05 2024-09-13T11:25:32.675 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.675 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.675 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:32.675 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:28.105075+0000", 2024-09-13T11:25:32.675 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.72509911299999996 2024-09-13T11:25:32.675 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.675 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.675 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:32.676 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:28.105111+0000", 2024-09-13T11:25:32.676 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.5403e-05 2024-09-13T11:25:32.676 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.676 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.676 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:32.676 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:28.105267+0000", 2024-09-13T11:25:32.676 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00015663600000000001 2024-09-13T11:25:32.676 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.676 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.676 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:25:32.676 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:28.105291+0000", 2024-09-13T11:25:32.676 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.4164000000000001e-05 2024-09-13T11:25:32.676 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.677 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.677 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:32.677 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:28.106357+0000", 2024-09-13T11:25:32.677 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.001065306 2024-09-13T11:25:32.677 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:32.677 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:32.677 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:32.677 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.677 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.677 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:113646 3.20 3:059047a0:::benchmark_data_smithi177_31116_object113645:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124)", 2024-09-13T11:25:32.677 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:24:32.178229+0000", 2024-09-13T11:25:32.677 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 60.423960002000001, 2024-09-13T11:25:32.677 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.23346934699999999, 2024-09-13T11:25:32.678 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:32.678 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:32.678 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:32.678 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:25:32.678 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:25:32.678 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:32.678 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 113646 2024-09-13T11:25:32.678 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.678 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:32.678 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.678 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:32.678 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:32.178229+0000", 2024-09-13T11:25:32.678 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:32.679 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.679 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.679 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:32.679 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:32.178229+0000", 2024-09-13T11:25:32.679 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:32.679 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.679 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.679 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:32.679 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:32.178232+0000", 2024-09-13T11:25:32.679 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.069e-06 2024-09-13T11:25:32.679 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.679 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.679 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:32.680 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:32.178266+0000", 2024-09-13T11:25:32.680 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.4242999999999997e-05 2024-09-13T11:25:32.680 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.680 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.680 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:32.680 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:32.178278+0000", 2024-09-13T11:25:32.680 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2558000000000001e-05 2024-09-13T11:25:32.680 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.680 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.680 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:32.680 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:32.178288+0000", 2024-09-13T11:25:32.680 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.6469999999999993e-06 2024-09-13T11:25:32.680 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.680 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.681 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:32.681 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:32.178349+0000", 2024-09-13T11:25:32.681 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.0643999999999997e-05 2024-09-13T11:25:32.681 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.681 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.681 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:25:32.681 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:32.178477+0000", 2024-09-13T11:25:32.681 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000128038 2024-09-13T11:25:32.681 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.681 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.681 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 0", 2024-09-13T11:25:32.681 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:32.178602+0000", 2024-09-13T11:25:32.681 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00012553600000000001 2024-09-13T11:25:32.682 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.682 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.682 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:25:32.682 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:32.180157+0000", 2024-09-13T11:25:32.682 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.001554386 2024-09-13T11:25:32.682 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.682 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.682 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:25:32.682 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:32.411634+0000", 2024-09-13T11:25:32.682 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.231477561 2024-09-13T11:25:32.682 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.682 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.682 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:25:32.682 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:32.411672+0000", 2024-09-13T11:25:32.683 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.7357999999999997e-05 2024-09-13T11:25:32.683 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.683 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.683 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:32.683 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:32.411699+0000", 2024-09-13T11:25:32.683 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.7307000000000001e-05 2024-09-13T11:25:32.683 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:32.683 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:32.683 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:32.683 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.683 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.683 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop_reply(client.4665.0:113646 3.20 e124/77 ondisk, result = 0)", 2024-09-13T11:25:32.683 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:24:32.180524+0000", 2024-09-13T11:25:32.683 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 60.421665220999998, 2024-09-13T11:25:32.684 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.231180894, 2024-09-13T11:25:32.684 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:32.684 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:32.684 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:25:32.684 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:32.684 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.684 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:32.684 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:32.180524+0000", 2024-09-13T11:25:32.684 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:32.684 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.684 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.684 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:32.684 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:32.180524+0000", 2024-09-13T11:25:32.684 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:32.685 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.685 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.685 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:32.685 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:32.180526+0000", 2024-09-13T11:25:32.685 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.4419999999999999e-06 2024-09-13T11:25:32.685 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.685 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.685 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:32.685 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:32.180530+0000", 2024-09-13T11:25:32.685 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.7330000000000003e-06 2024-09-13T11:25:32.685 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.685 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.685 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:32.685 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:32.411549+0000", 2024-09-13T11:25:32.686 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.23101862300000001 2024-09-13T11:25:32.686 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.686 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.686 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:32.686 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:32.411558+0000", 2024-09-13T11:25:32.686 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.5949999999999999e-06 2024-09-13T11:25:32.686 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.686 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.686 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:32.686 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:32.411622+0000", 2024-09-13T11:25:32.686 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.4251000000000005e-05 2024-09-13T11:25:32.686 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.686 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.687 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:25:32.687 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:32.411629+0000", 2024-09-13T11:25:32.687 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.328e-06 2024-09-13T11:25:32.687 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.687 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.687 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:32.687 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:32.411705+0000", 2024-09-13T11:25:32.687 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.5921999999999996e-05 2024-09-13T11:25:32.687 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:32.687 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:32.687 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:32.687 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.687 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.687 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop_reply(client.4665.0:129892 3.0 e134/75 ondisk, result = 0)", 2024-09-13T11:25:32.688 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:24:47.136509+0000", 2024-09-13T11:25:32.688 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 45.465680173999999, 2024-09-13T11:25:32.688 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.211101067, 2024-09-13T11:25:32.688 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:32.688 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:32.688 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:25:32.688 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:32.688 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.688 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:32.688 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:47.136509+0000", 2024-09-13T11:25:32.688 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:32.688 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.688 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.688 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:32.689 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:47.136509+0000", 2024-09-13T11:25:32.689 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:32.689 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.689 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.689 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:32.689 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:47.136510+0000", 2024-09-13T11:25:32.689 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.17e-06 2024-09-13T11:25:32.689 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.689 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.689 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:32.689 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:47.136514+0000", 2024-09-13T11:25:32.689 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.5609999999999999e-06 2024-09-13T11:25:32.689 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.690 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.690 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:32.690 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:47.347445+0000", 2024-09-13T11:25:32.690 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.210931324 2024-09-13T11:25:32.690 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.690 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.690 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:32.690 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:47.347450+0000", 2024-09-13T11:25:32.690 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.3819999999999997e-06 2024-09-13T11:25:32.690 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.690 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.690 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:32.690 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:47.347491+0000", 2024-09-13T11:25:32.690 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.1025000000000001e-05 2024-09-13T11:25:32.691 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.691 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.691 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:25:32.691 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:47.347496+0000", 2024-09-13T11:25:32.691 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.463e-06 2024-09-13T11:25:32.691 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.691 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.691 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:32.691 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:47.347610+0000", 2024-09-13T11:25:32.691 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00011414200000000001 2024-09-13T11:25:32.691 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:32.691 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:32.691 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:32.691 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.692 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.692 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:138294 3.9 3:90012dab:::benchmark_data_smithi177_31116_object138293:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e140)", 2024-09-13T11:25:32.692 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:24:54.250266+0000", 2024-09-13T11:25:32.692 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 38.351923288999998, 2024-09-13T11:25:32.692 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.33165556200000001, 2024-09-13T11:25:32.692 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:32.692 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:32.692 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:32.692 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:25:32.692 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:25:32.692 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:32.692 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 138294 2024-09-13T11:25:32.692 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.693 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:32.693 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.693 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:32.693 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:54.250266+0000", 2024-09-13T11:25:32.693 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:32.693 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.693 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.693 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:32.693 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:54.250266+0000", 2024-09-13T11:25:32.693 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:32.693 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.693 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.693 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:32.693 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:54.250269+0000", 2024-09-13T11:25:32.694 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.3159999999999999e-06 2024-09-13T11:25:32.694 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.694 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.694 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:32.694 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:54.250362+0000", 2024-09-13T11:25:32.694 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.3051000000000001e-05 2024-09-13T11:25:32.696 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.696 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.696 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:32.696 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:54.250374+0000", 2024-09-13T11:25:32.696 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2327000000000001e-05 2024-09-13T11:25:32.696 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.696 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.696 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:32.696 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:54.250382+0000", 2024-09-13T11:25:32.697 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.7249999999999997e-06 2024-09-13T11:25:32.697 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.697 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.697 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:32.697 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:54.250459+0000", 2024-09-13T11:25:32.697 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.7218999999999994e-05 2024-09-13T11:25:32.697 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.697 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.697 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:25:32.697 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:54.250611+0000", 2024-09-13T11:25:32.697 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00015241000000000001 2024-09-13T11:25:32.697 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.697 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.698 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 2", 2024-09-13T11:25:32.698 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:54.250777+0000", 2024-09-13T11:25:32.698 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00016575199999999999 2024-09-13T11:25:32.698 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.698 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.698 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:25:32.698 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:54.252776+0000", 2024-09-13T11:25:32.698 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0019987260000000001 2024-09-13T11:25:32.698 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.698 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.698 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:25:32.698 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:54.581758+0000", 2024-09-13T11:25:32.698 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.32898232799999999 2024-09-13T11:25:32.698 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.698 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.699 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:25:32.699 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:54.581854+0000", 2024-09-13T11:25:32.699 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.5468000000000001e-05 2024-09-13T11:25:32.699 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.699 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.699 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:32.699 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:54.581922+0000", 2024-09-13T11:25:32.699 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.8239999999999997e-05 2024-09-13T11:25:32.699 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:32.699 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:32.699 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:32.699 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.699 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.700 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop_reply(client.4665.0:140790 3.20 e144/77 ondisk, result = 0)", 2024-09-13T11:25:32.700 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:24:58.385972+0000", 2024-09-13T11:25:32.700 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 34.216217721, 2024-09-13T11:25:32.700 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.51829568500000001, 2024-09-13T11:25:32.700 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:32.700 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:32.700 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:25:32.700 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:32.700 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.700 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:32.700 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.385972+0000", 2024-09-13T11:25:32.700 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:32.700 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.700 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.701 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:32.701 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.385972+0000", 2024-09-13T11:25:32.701 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:32.701 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.701 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.701 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:32.701 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.385974+0000", 2024-09-13T11:25:32.701 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.7400000000000001e-06 2024-09-13T11:25:32.701 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.701 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.701 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:32.701 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.385979+0000", 2024-09-13T11:25:32.701 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.091e-06 2024-09-13T11:25:32.701 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.701 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.702 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:32.702 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.904091+0000", 2024-09-13T11:25:32.702 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.51811194900000002 2024-09-13T11:25:32.702 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.702 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.702 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:32.702 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.904098+0000", 2024-09-13T11:25:32.702 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.0609999999999998e-06 2024-09-13T11:25:32.702 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.702 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.702 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:32.702 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.904156+0000", 2024-09-13T11:25:32.702 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.7902000000000002e-05 2024-09-13T11:25:32.703 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.703 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.703 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:25:32.703 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.904163+0000", 2024-09-13T11:25:32.703 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.1430000000000001e-06 2024-09-13T11:25:32.703 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.703 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.703 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:32.703 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.904267+0000", 2024-09-13T11:25:32.703 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000104799 2024-09-13T11:25:32.703 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:32.703 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:32.703 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:32.704 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.704 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.704 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:170721 3.9 3:93244b1b:::benchmark_data_smithi177_31116_object30418:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e162)", 2024-09-13T11:25:32.704 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:25:21.721042+0000", 2024-09-13T11:25:32.704 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 10.88114772, 2024-09-13T11:25:32.704 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.19807348799999999, 2024-09-13T11:25:32.704 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:32.704 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:32.704 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:32.705 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:25:32.705 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:25:32.705 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:32.705 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 170721 2024-09-13T11:25:32.705 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.705 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:32.705 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.705 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:32.705 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:21.721042+0000", 2024-09-13T11:25:32.705 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:32.705 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.706 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.706 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:32.706 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:21.721042+0000", 2024-09-13T11:25:32.706 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:32.706 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.706 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.706 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:32.706 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:21.721044+0000", 2024-09-13T11:25:32.706 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.781e-06 2024-09-13T11:25:32.706 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.706 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.706 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:32.707 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:21.721050+0000", 2024-09-13T11:25:32.707 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.1840000000000004e-06 2024-09-13T11:25:32.707 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.707 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.707 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:32.707 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:21.721065+0000", 2024-09-13T11:25:32.707 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.5073e-05 2024-09-13T11:25:32.707 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.707 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.707 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:32.707 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:21.721073+0000", 2024-09-13T11:25:32.708 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.6030000000000005e-06 2024-09-13T11:25:32.708 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.708 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.708 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:32.708 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:21.721140+0000", 2024-09-13T11:25:32.708 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.6979999999999999e-05 2024-09-13T11:25:32.708 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.708 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.708 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:25:32.708 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:21.721260+0000", 2024-09-13T11:25:32.708 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000119119 2024-09-13T11:25:32.709 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.709 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.709 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 2", 2024-09-13T11:25:32.709 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:21.721363+0000", 2024-09-13T11:25:32.709 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000103052 2024-09-13T11:25:32.709 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.709 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.709 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:25:32.709 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:21.723031+0000", 2024-09-13T11:25:32.709 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0016684650000000001 2024-09-13T11:25:32.709 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.710 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.710 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:25:32.710 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:21.919011+0000", 2024-09-13T11:25:32.710 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.19598026800000001 2024-09-13T11:25:32.710 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.710 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.710 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:25:32.710 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:21.919070+0000", 2024-09-13T11:25:32.710 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.9110000000000002e-05 2024-09-13T11:25:32.710 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:32.710 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:32.711 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:32.711 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:21.919115+0000", 2024-09-13T11:25:32.711 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.4852999999999999e-05 2024-09-13T11:25:32.711 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:32.711 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:32.711 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:32.711 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:32.711 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:32.711 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-13T11:25:32.712 INFO:tasks.ceph.osd.4.smithi001.stderr:2024-09-13T11:25:32.603+0000 7ff8e0218640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26310) UID: 0 2024-09-13T11:25:32.713 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-13T11:25:32.714 INFO:tasks.ceph.osd.10.smithi162.stderr:2024-09-13T11:25:32.712+0000 7fb59c9f0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26317) UID: 0 2024-09-13T11:25:32.815 INFO:tasks.ceph.osd.4.smithi001.stderr:2024-09-13T11:25:32.815+0000 7ff8e0218640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26310) UID: 0 2024-09-13T11:25:32.859 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-09-13T11:25:32.859 INFO:teuthology.orchestra.run.smithi092.stdout: "ops": [ 2024-09-13T11:25:32.859 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:32.860 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:184221 3.27 3:e690195f:::benchmark_data_smithi177_31116_object43918:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e171)", 2024-09-13T11:25:32.860 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:25:32.845735+0000", 2024-09-13T11:25:32.860 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 0.013400916000000001, 2024-09-13T11:25:32.860 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.013428028999999999, 2024-09-13T11:25:32.860 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:25:32.860 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:25:32.860 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "waiting for sub ops", 2024-09-13T11:25:32.860 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:25:32.860 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:25:32.860 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:32.860 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 184221 2024-09-13T11:25:32.860 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:32.860 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:25:32.861 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:32.861 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:25:32.861 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:32.845735+0000", 2024-09-13T11:25:32.861 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:32.861 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:32.861 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:32.861 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:25:32.861 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:32.845735+0000", 2024-09-13T11:25:32.861 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:32.861 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:32.861 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:32.861 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:25:32.861 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:32.845738+0000", 2024-09-13T11:25:32.862 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.413e-06 2024-09-13T11:25:32.862 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:32.862 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:32.862 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:25:32.862 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:32.845744+0000", 2024-09-13T11:25:32.862 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.8310000000000003e-06 2024-09-13T11:25:32.862 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:32.862 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:32.862 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:25:32.862 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:32.845759+0000", 2024-09-13T11:25:32.862 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.4749999999999999e-05 2024-09-13T11:25:32.862 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:32.862 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:32.863 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:25:32.863 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:32.845769+0000", 2024-09-13T11:25:32.863 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0129000000000001e-05 2024-09-13T11:25:32.863 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:32.863 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:32.863 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:25:32.863 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:32.845844+0000", 2024-09-13T11:25:32.863 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.4838000000000004e-05 2024-09-13T11:25:32.863 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:32.863 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:32.863 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:25:32.863 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:32.845964+0000", 2024-09-13T11:25:32.863 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00011958199999999999 2024-09-13T11:25:32.863 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:32.864 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:32.864 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 11", 2024-09-13T11:25:32.864 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:32.846047+0000", 2024-09-13T11:25:32.864 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.3011999999999997e-05 2024-09-13T11:25:32.864 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:32.864 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:32.864 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:25:32.864 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:32.847723+0000", 2024-09-13T11:25:32.864 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.001676294 2024-09-13T11:25:32.864 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:32.864 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:32.864 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:25:32.865 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:32.859134+0000", 2024-09-13T11:25:32.865 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.011410461 2024-09-13T11:25:32.865 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:32.865 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:32.865 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:25:32.865 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:32.859174+0000", 2024-09-13T11:25:32.865 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.0054e-05 2024-09-13T11:25:32.865 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:32.865 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:25:32.865 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:32.865 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:32.865 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:32.865 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:184233 3.29 3:964ca41b:::benchmark_data_smithi177_31116_object43930:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e171)", 2024-09-13T11:25:32.866 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:25:32.856064+0000", 2024-09-13T11:25:32.866 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 0.0030720840000000001, 2024-09-13T11:25:32.866 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0032181079999999999, 2024-09-13T11:25:32.866 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:25:32.866 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:25:32.866 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "waiting for sub ops", 2024-09-13T11:25:32.866 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:25:32.866 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:25:32.866 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:32.866 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 184233 2024-09-13T11:25:32.866 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:32.866 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:25:32.866 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:32.866 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:25:32.867 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:32.856064+0000", 2024-09-13T11:25:32.867 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:32.867 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:32.867 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:32.867 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:25:32.867 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:32.856064+0000", 2024-09-13T11:25:32.867 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:32.867 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:32.867 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:32.867 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:25:32.867 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:32.856066+0000", 2024-09-13T11:25:32.867 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.671e-06 2024-09-13T11:25:32.867 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:32.868 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:32.868 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:25:32.868 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:32.856074+0000", 2024-09-13T11:25:32.868 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.3920000000000003e-06 2024-09-13T11:25:32.868 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:32.868 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:32.868 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:25:32.868 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:32.856087+0000", 2024-09-13T11:25:32.868 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.3151e-05 2024-09-13T11:25:32.868 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:32.868 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:32.868 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:25:32.868 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:32.856097+0000", 2024-09-13T11:25:32.869 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.4679999999999998e-06 2024-09-13T11:25:32.869 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:32.869 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:32.869 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:25:32.869 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:32.856170+0000", 2024-09-13T11:25:32.869 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.3257999999999996e-05 2024-09-13T11:25:32.869 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:32.869 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:32.869 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:25:32.869 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:32.856312+0000", 2024-09-13T11:25:32.869 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00014185000000000001 2024-09-13T11:25:32.869 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:32.869 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:32.869 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 4", 2024-09-13T11:25:32.870 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:32.856434+0000", 2024-09-13T11:25:32.870 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000121747 2024-09-13T11:25:32.870 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:32.870 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:32.870 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:25:32.870 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:32.858191+0000", 2024-09-13T11:25:32.870 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.001757689 2024-09-13T11:25:32.870 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:32.870 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:32.870 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:25:32.870 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:32.859265+0000", 2024-09-13T11:25:32.870 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0010739009999999999 2024-09-13T11:25:32.870 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:32.871 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:25:32.871 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:32.871 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:32.871 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:32.871 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop(client.4665.0:184234 3.28 e171/120 3:14fc98f3:::benchmark_data_smithi177_31116_object43931:head v 171'5874, mlcod=171'5872)", 2024-09-13T11:25:32.871 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:25:32.856723+0000", 2024-09-13T11:25:32.871 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 0.0024127340000000001, 2024-09-13T11:25:32.871 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0026359700000000001, 2024-09-13T11:25:32.871 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:25:32.871 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:25:32.871 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-09-13T11:25:32.871 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:25:32.871 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:32.872 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:25:32.872 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:32.856723+0000", 2024-09-13T11:25:32.872 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:32.872 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:32.872 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:32.872 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:25:32.872 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:32.856723+0000", 2024-09-13T11:25:32.872 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:32.872 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:32.872 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:32.872 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:25:32.872 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:32.856725+0000", 2024-09-13T11:25:32.872 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.19e-06 2024-09-13T11:25:32.873 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:32.873 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:32.873 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:25:32.873 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:32.856733+0000", 2024-09-13T11:25:32.873 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.276e-06 2024-09-13T11:25:32.873 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:32.873 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:32.873 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:25:32.873 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:32.856773+0000", 2024-09-13T11:25:32.873 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.0524e-05 2024-09-13T11:25:32.873 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:32.873 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:32.873 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:25:32.873 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:32.856782+0000", 2024-09-13T11:25:32.874 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.4100000000000008e-06 2024-09-13T11:25:32.874 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:32.874 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:32.874 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:25:32.874 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:32.856849+0000", 2024-09-13T11:25:32.874 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.6905999999999999e-05 2024-09-13T11:25:32.874 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:32.874 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:32.874 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:25:32.874 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:32.856880+0000", 2024-09-13T11:25:32.874 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.1195000000000002e-05 2024-09-13T11:25:32.874 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:32.874 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:25:32.874 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:32.875 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:32.875 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:32.875 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:184236 3.27 3:e587ca0c:::benchmark_data_smithi177_31116_object43933:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e171)", 2024-09-13T11:25:32.875 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:25:32.857226+0000", 2024-09-13T11:25:32.875 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 0.001910665, 2024-09-13T11:25:32.875 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0021858400000000001, 2024-09-13T11:25:32.875 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:25:32.875 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:25:32.875 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "waiting for sub ops", 2024-09-13T11:25:32.875 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:25:32.875 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:25:32.875 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:32.875 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 184236 2024-09-13T11:25:32.876 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:32.876 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:25:32.876 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:32.876 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:25:32.876 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:32.857226+0000", 2024-09-13T11:25:32.876 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:32.876 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:32.876 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:32.876 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:25:32.876 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:32.857226+0000", 2024-09-13T11:25:32.876 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:32.876 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:32.876 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:32.877 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:25:32.877 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:32.857227+0000", 2024-09-13T11:25:32.877 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.5179999999999999e-06 2024-09-13T11:25:32.877 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:32.877 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:32.877 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:25:32.877 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:32.857231+0000", 2024-09-13T11:25:32.877 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.9759999999999997e-06 2024-09-13T11:25:32.877 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:32.877 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:32.877 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:25:32.877 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:32.857241+0000", 2024-09-13T11:25:32.877 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.8980000000000008e-06 2024-09-13T11:25:32.877 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:32.878 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:32.878 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:25:32.878 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:32.857247+0000", 2024-09-13T11:25:32.878 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.6470000000000003e-06 2024-09-13T11:25:32.878 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:32.878 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:32.878 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:25:32.878 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:32.857301+0000", 2024-09-13T11:25:32.878 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.4889999999999998e-05 2024-09-13T11:25:32.878 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:32.878 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:32.878 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:25:32.878 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:32.857400+0000", 2024-09-13T11:25:32.879 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.8281000000000003e-05 2024-09-13T11:25:32.879 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:32.879 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:32.879 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 11", 2024-09-13T11:25:32.879 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:32.857491+0000", 2024-09-13T11:25:32.879 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.1273000000000005e-05 2024-09-13T11:25:32.879 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:32.879 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:25:32.879 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:32.879 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:32.879 INFO:teuthology.orchestra.run.smithi092.stdout: ], 2024-09-13T11:25:32.879 INFO:teuthology.orchestra.run.smithi092.stdout: "num_ops": 4 2024-09-13T11:25:32.879 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-09-13T11:25:32.881 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-13T11:25:32.916 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-09-13T11:25:32.912+0000 7f97befc9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26315) UID: 0 2024-09-13T11:25:33.017 INFO:tasks.ceph.osd.4.smithi001.stderr:2024-09-13T11:25:33.015+0000 7ff8e0218640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26310) UID: 0 2024-09-13T11:25:33.020 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-09-13T11:25:33.020 INFO:teuthology.orchestra.run.smithi092.stdout: "ops": [], 2024-09-13T11:25:33.020 INFO:teuthology.orchestra.run.smithi092.stdout: "complaint_time": 30, 2024-09-13T11:25:33.020 INFO:teuthology.orchestra.run.smithi092.stdout: "num_blocked_ops": 0 2024-09-13T11:25:33.020 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-09-13T11:25:33.037 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-13T11:25:33.118 INFO:tasks.ceph.osd.10.smithi162.stderr:2024-09-13T11:25:33.116+0000 7fb59c9f0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26317) UID: 0 2024-09-13T11:25:33.174 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-09-13T11:25:33.174 INFO:teuthology.orchestra.run.smithi092.stdout: "size": 20, 2024-09-13T11:25:33.174 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 600, 2024-09-13T11:25:33.175 INFO:teuthology.orchestra.run.smithi092.stdout: "ops": [ 2024-09-13T11:25:33.175 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.175 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "MOSDPGPush(3.23 83/78 [PushOp(3:c61f6bab:::benchmark_data_smithi177_31116_object29479:head, version: 59'1860, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:c61f6bab:::benchmark_data_smithi177_31116_object29479:head@59'1860, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:25:33.175 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:40.770864+0000", 2024-09-13T11:25:33.175 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 112.401703438, 2024-09-13T11:25:33.175 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.31229662800000002, 2024-09-13T11:25:33.175 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:25:33.175 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:25:33.175 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-09-13T11:25:33.175 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:25:33.175 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.175 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:25:33.175 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:40.770864+0000", 2024-09-13T11:25:33.176 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:33.176 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.176 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.176 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:25:33.176 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:40.770864+0000", 2024-09-13T11:25:33.176 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:33.176 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.176 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.176 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:25:33.176 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:40.770865+0000", 2024-09-13T11:25:33.176 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.288e-06 2024-09-13T11:25:33.177 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.177 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.177 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:25:33.177 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:40.770955+0000", 2024-09-13T11:25:33.177 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.9871999999999996e-05 2024-09-13T11:25:33.177 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.177 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.177 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:25:33.177 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:41.082105+0000", 2024-09-13T11:25:33.178 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.31114963400000001 2024-09-13T11:25:33.178 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.178 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.178 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:25:33.178 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:41.082136+0000", 2024-09-13T11:25:33.178 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.0972000000000001e-05 2024-09-13T11:25:33.178 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.178 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.178 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:25:33.178 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:41.082404+0000", 2024-09-13T11:25:33.178 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00026844800000000002 2024-09-13T11:25:33.178 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.178 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.179 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:25:33.179 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:41.082429+0000", 2024-09-13T11:25:33.179 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.5021e-05 2024-09-13T11:25:33.179 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.179 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.179 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:25:33.179 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:41.083161+0000", 2024-09-13T11:25:33.179 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000731393 2024-09-13T11:25:33.179 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:33.179 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:25:33.179 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:33.179 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.179 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.179 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "MOSDPGPush(3.1e 85/78 [PushOp(3:7f40e925:::benchmark_data_smithi177_31116_object14084:head, version: 50'927, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:7f40e925:::benchmark_data_smithi177_31116_object14084:head@50'927, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:25:33.180 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:42.751000+0000", 2024-09-13T11:25:33.180 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 110.421567131, 2024-09-13T11:25:33.180 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.19208357500000001, 2024-09-13T11:25:33.180 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:25:33.180 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:25:33.180 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-09-13T11:25:33.180 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:25:33.180 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.180 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:25:33.180 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.751000+0000", 2024-09-13T11:25:33.180 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:33.180 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.180 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.180 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:25:33.181 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.751000+0000", 2024-09-13T11:25:33.181 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:33.181 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.181 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.181 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:25:33.181 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.751003+0000", 2024-09-13T11:25:33.181 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.0800000000000002e-06 2024-09-13T11:25:33.181 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.181 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.181 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:25:33.181 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.751069+0000", 2024-09-13T11:25:33.181 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.5417000000000001e-05 2024-09-13T11:25:33.181 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.181 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.182 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:25:33.182 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.941261+0000", 2024-09-13T11:25:33.182 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.19019233899999999 2024-09-13T11:25:33.182 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.182 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.182 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:25:33.182 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.941300+0000", 2024-09-13T11:25:33.182 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.9091999999999998e-05 2024-09-13T11:25:33.182 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.182 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.182 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:25:33.182 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.941519+0000", 2024-09-13T11:25:33.182 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000218521 2024-09-13T11:25:33.182 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.183 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.183 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:25:33.183 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.941544+0000", 2024-09-13T11:25:33.183 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.5412999999999999e-05 2024-09-13T11:25:33.183 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.183 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.183 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:25:33.183 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.943084+0000", 2024-09-13T11:25:33.183 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0015397130000000001 2024-09-13T11:25:33.183 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:33.183 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:25:33.183 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:33.183 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.183 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.183 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:82769 3.1e 3:7a4e88b5:::benchmark_data_smithi177_31116_object82768:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105)", 2024-09-13T11:25:33.184 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:24:03.744632+0000", 2024-09-13T11:25:33.184 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 89.427934985999997, 2024-09-13T11:25:33.184 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.21475722699999999, 2024-09-13T11:25:33.184 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:25:33.184 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:25:33.184 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:33.184 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:25:33.184 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:25:33.184 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:33.184 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 82769 2024-09-13T11:25:33.184 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.184 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:25:33.184 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.184 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:25:33.185 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:03.744632+0000", 2024-09-13T11:25:33.185 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:33.185 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.185 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.185 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:25:33.185 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:03.744632+0000", 2024-09-13T11:25:33.185 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:33.185 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.185 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.185 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:25:33.185 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:03.744634+0000", 2024-09-13T11:25:33.185 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.672e-06 2024-09-13T11:25:33.185 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.185 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.186 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:25:33.186 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:03.744724+0000", 2024-09-13T11:25:33.186 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.0237999999999999e-05 2024-09-13T11:25:33.186 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.186 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.186 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:25:33.186 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:03.744736+0000", 2024-09-13T11:25:33.186 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.2038999999999999e-05 2024-09-13T11:25:33.186 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.186 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.186 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:25:33.186 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:03.744742+0000", 2024-09-13T11:25:33.186 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.2129999999999996e-06 2024-09-13T11:25:33.186 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.187 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.187 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:25:33.187 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:03.744785+0000", 2024-09-13T11:25:33.187 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.2898000000000002e-05 2024-09-13T11:25:33.187 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.187 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.187 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:25:33.187 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:03.744894+0000", 2024-09-13T11:25:33.187 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000109073 2024-09-13T11:25:33.187 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.187 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.188 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 4", 2024-09-13T11:25:33.188 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:03.745009+0000", 2024-09-13T11:25:33.188 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00011428499999999999 2024-09-13T11:25:33.188 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.188 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.188 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:25:33.188 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:03.746451+0000", 2024-09-13T11:25:33.188 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0014419179999999999 2024-09-13T11:25:33.188 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.188 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.188 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:25:33.188 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:03.959257+0000", 2024-09-13T11:25:33.189 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.212806361 2024-09-13T11:25:33.189 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.189 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.189 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:25:33.189 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:03.959333+0000", 2024-09-13T11:25:33.189 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.6001000000000006e-05 2024-09-13T11:25:33.189 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.189 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.189 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:25:33.189 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:03.959390+0000", 2024-09-13T11:25:33.189 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.6529e-05 2024-09-13T11:25:33.189 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:33.189 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:25:33.189 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:33.190 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.190 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.190 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:110289 3.29 3.27287969 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e119)", 2024-09-13T11:25:33.190 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:24:25.774634+0000", 2024-09-13T11:25:33.190 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 67.397933363000007, 2024-09-13T11:25:33.190 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.31721633999999999, 2024-09-13T11:25:33.190 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:25:33.190 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:25:33.190 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "reached pg", 2024-09-13T11:25:33.190 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:25:33.190 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:25:33.190 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:33.190 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 110289 2024-09-13T11:25:33.190 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.191 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:25:33.191 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.191 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:25:33.191 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:25.774634+0000", 2024-09-13T11:25:33.191 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:33.191 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.191 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.191 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:25:33.191 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:25.774634+0000", 2024-09-13T11:25:33.191 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:33.191 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.191 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.192 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:25:33.192 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:25.774639+0000", 2024-09-13T11:25:33.192 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.6029999999999998e-06 2024-09-13T11:25:33.192 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.192 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.192 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:25:33.192 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:25.774695+0000", 2024-09-13T11:25:33.192 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.6694999999999998e-05 2024-09-13T11:25:33.192 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.192 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.193 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:25:33.193 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:25.774728+0000", 2024-09-13T11:25:33.193 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.2502000000000002e-05 2024-09-13T11:25:33.193 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.193 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.193 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:25:33.193 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:25.774750+0000", 2024-09-13T11:25:33.193 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.2002999999999999e-05 2024-09-13T11:25:33.193 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.194 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.194 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:25:33.194 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:25.774942+0000", 2024-09-13T11:25:33.194 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000192265 2024-09-13T11:25:33.194 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.194 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.194 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-09-13T11:25:33.194 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:25.774960+0000", 2024-09-13T11:25:33.194 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.8275999999999999e-05 2024-09-13T11:25:33.194 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.194 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.195 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:25:33.195 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:26.091821+0000", 2024-09-13T11:25:33.195 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.31686065000000002 2024-09-13T11:25:33.195 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.195 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.195 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:25:33.195 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:26.091850+0000", 2024-09-13T11:25:33.195 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.9346e-05 2024-09-13T11:25:33.195 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:33.195 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:25:33.195 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:33.195 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.196 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.196 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "MOSDPGPush(3.22 125/122 [PushOp(3:44e4f9f2:::benchmark_data_smithi177_31116_object16663:head, version: 52'1032, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:44e4f9f2:::benchmark_data_smithi177_31116_object16663:head@52'1032, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:25:33.196 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:24:32.743254+0000", 2024-09-13T11:25:33.196 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 60.429313274000002, 2024-09-13T11:25:33.196 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.215485287, 2024-09-13T11:25:33.196 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:25:33.196 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:25:33.196 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-09-13T11:25:33.196 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:25:33.196 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.196 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:25:33.196 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:32.743254+0000", 2024-09-13T11:25:33.196 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:33.197 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.197 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.197 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:25:33.197 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:32.743254+0000", 2024-09-13T11:25:33.197 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:33.197 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.197 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.197 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:25:33.197 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:32.743255+0000", 2024-09-13T11:25:33.197 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.2500000000000004e-07 2024-09-13T11:25:33.197 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.197 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.197 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:25:33.198 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:32.743310+0000", 2024-09-13T11:25:33.198 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.5442000000000002e-05 2024-09-13T11:25:33.198 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.198 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.198 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:25:33.198 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:32.957754+0000", 2024-09-13T11:25:33.198 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.21444387000000001 2024-09-13T11:25:33.198 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.198 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.198 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:25:33.198 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:32.957765+0000", 2024-09-13T11:25:33.198 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0458999999999999e-05 2024-09-13T11:25:33.198 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.198 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.199 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:25:33.199 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:32.957841+0000", 2024-09-13T11:25:33.199 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.5993000000000002e-05 2024-09-13T11:25:33.199 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.199 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.199 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:25:33.199 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:32.957852+0000", 2024-09-13T11:25:33.199 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.1453e-05 2024-09-13T11:25:33.199 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.199 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.199 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:25:33.199 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:32.958739+0000", 2024-09-13T11:25:33.199 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00088724500000000005 2024-09-13T11:25:33.199 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:33.200 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:25:33.200 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:33.200 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.200 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.200 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop(client.4665.0:115089 3.28 e125/120 3:15b13357:::benchmark_data_smithi177_31116_object115088:head v 125'4843, mlcod=47'620)", 2024-09-13T11:25:33.200 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:24:33.642654+0000", 2024-09-13T11:25:33.200 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 59.529912864000003, 2024-09-13T11:25:33.200 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.19312220099999999, 2024-09-13T11:25:33.200 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:25:33.200 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:25:33.200 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:33.200 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:25:33.200 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.200 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:25:33.201 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:33.642654+0000", 2024-09-13T11:25:33.201 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:33.201 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.201 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.201 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:25:33.201 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:33.642654+0000", 2024-09-13T11:25:33.201 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:33.201 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.201 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.201 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:25:33.201 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:33.642656+0000", 2024-09-13T11:25:33.201 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.869e-06 2024-09-13T11:25:33.201 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.201 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.202 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:25:33.202 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:33.642688+0000", 2024-09-13T11:25:33.202 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.2169999999999999e-05 2024-09-13T11:25:33.202 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.202 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.202 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:25:33.202 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:33.832657+0000", 2024-09-13T11:25:33.202 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.18996885099999999 2024-09-13T11:25:33.202 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.202 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.202 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:25:33.202 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:33.832678+0000", 2024-09-13T11:25:33.203 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.0222000000000001e-05 2024-09-13T11:25:33.203 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.203 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.203 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:25:33.203 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:33.832760+0000", 2024-09-13T11:25:33.203 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.2113000000000005e-05 2024-09-13T11:25:33.203 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.203 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.203 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:25:33.203 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:33.832788+0000", 2024-09-13T11:25:33.203 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.8095000000000001e-05 2024-09-13T11:25:33.203 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.203 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.203 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:25:33.204 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:33.835729+0000", 2024-09-13T11:25:33.204 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0029408920000000001 2024-09-13T11:25:33.204 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.204 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.204 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:25:33.204 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:33.835777+0000", 2024-09-13T11:25:33.204 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.7988999999999997e-05 2024-09-13T11:25:33.204 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:33.204 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:25:33.204 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:33.204 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.204 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.204 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "MOSDPGPush(3.29 139/120 [PushOp(3:961aa4aa:::benchmark_data_smithi177_31116_object26561:head, version: 58'1688, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:961aa4aa:::benchmark_data_smithi177_31116_object26561:head@58'1688, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:25:33.205 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:24:53.385067+0000", 2024-09-13T11:25:33.205 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 39.787499906000001, 2024-09-13T11:25:33.205 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.185187613, 2024-09-13T11:25:33.205 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:25:33.205 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:25:33.205 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-09-13T11:25:33.205 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:25:33.205 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.205 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:25:33.205 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:53.385067+0000", 2024-09-13T11:25:33.205 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:33.205 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.205 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.206 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:25:33.206 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:53.385067+0000", 2024-09-13T11:25:33.206 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:33.206 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.206 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.206 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:25:33.206 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:53.385069+0000", 2024-09-13T11:25:33.206 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.4729999999999999e-06 2024-09-13T11:25:33.206 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.206 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.206 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:25:33.206 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:53.385139+0000", 2024-09-13T11:25:33.206 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.9881000000000004e-05 2024-09-13T11:25:33.207 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.207 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.207 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:25:33.207 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:53.569032+0000", 2024-09-13T11:25:33.207 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.183893626 2024-09-13T11:25:33.207 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.207 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.207 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:25:33.207 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:53.569053+0000", 2024-09-13T11:25:33.207 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.0305999999999998e-05 2024-09-13T11:25:33.207 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.207 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.207 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:25:33.207 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:53.569205+0000", 2024-09-13T11:25:33.208 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00015209699999999999 2024-09-13T11:25:33.208 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.208 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.208 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:25:33.208 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:53.569214+0000", 2024-09-13T11:25:33.208 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.3740000000000004e-06 2024-09-13T11:25:33.208 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.208 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.208 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:25:33.208 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:53.570255+0000", 2024-09-13T11:25:33.208 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0010408559999999999 2024-09-13T11:25:33.208 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:33.208 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:25:33.208 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:33.209 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.209 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.209 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:138370 3.23 3:c5730cd2:::benchmark_data_smithi177_31116_object138369:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e140)", 2024-09-13T11:25:33.209 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:24:54.415176+0000", 2024-09-13T11:25:33.209 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 38.757391267000003, 2024-09-13T11:25:33.209 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.24549895499999999, 2024-09-13T11:25:33.209 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:25:33.209 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:25:33.209 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:33.209 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:25:33.209 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:25:33.209 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:33.209 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 138370 2024-09-13T11:25:33.210 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.210 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:25:33.210 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.210 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:25:33.210 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:54.415176+0000", 2024-09-13T11:25:33.210 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:33.210 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.210 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.210 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:25:33.210 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:54.415176+0000", 2024-09-13T11:25:33.210 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:33.210 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.210 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.210 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:25:33.211 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:54.415180+0000", 2024-09-13T11:25:33.211 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.1670000000000001e-06 2024-09-13T11:25:33.211 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.211 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.211 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:25:33.211 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:54.415269+0000", 2024-09-13T11:25:33.211 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.8659000000000006e-05 2024-09-13T11:25:33.211 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.211 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.211 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:25:33.211 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:54.415292+0000", 2024-09-13T11:25:33.211 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.3152000000000001e-05 2024-09-13T11:25:33.211 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.212 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.212 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:25:33.212 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:54.415304+0000", 2024-09-13T11:25:33.212 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.1841999999999999e-05 2024-09-13T11:25:33.212 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.212 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.212 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:25:33.212 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:54.415432+0000", 2024-09-13T11:25:33.212 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00012802000000000001 2024-09-13T11:25:33.212 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.212 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.212 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:25:33.212 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:54.415710+0000", 2024-09-13T11:25:33.212 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00027840299999999998 2024-09-13T11:25:33.213 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.213 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.213 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 9", 2024-09-13T11:25:33.213 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:54.415950+0000", 2024-09-13T11:25:33.213 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00023928299999999999 2024-09-13T11:25:33.213 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.213 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.213 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:25:33.213 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:54.419265+0000", 2024-09-13T11:25:33.213 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0033150440000000001 2024-09-13T11:25:33.213 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.213 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.213 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:25:33.213 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:54.660448+0000", 2024-09-13T11:25:33.214 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.24118348100000001 2024-09-13T11:25:33.214 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.214 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.214 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:25:33.214 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:54.660571+0000", 2024-09-13T11:25:33.214 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00012332900000000001 2024-09-13T11:25:33.214 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.214 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.214 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:25:33.214 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:54.660675+0000", 2024-09-13T11:25:33.214 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000103575 2024-09-13T11:25:33.214 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:33.214 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:25:33.214 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:33.215 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.215 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.215 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop(client.4665.0:138550 3.27 e140/120 3:e494a291:::benchmark_data_smithi177_31116_object138549:head v 140'5229, mlcod=49'686)", 2024-09-13T11:25:33.215 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:24:54.831360+0000", 2024-09-13T11:25:33.215 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 38.341207259000001, 2024-09-13T11:25:33.215 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.176014844, 2024-09-13T11:25:33.215 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:25:33.215 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:25:33.215 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:33.215 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:25:33.215 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.215 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:25:33.215 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:54.831360+0000", 2024-09-13T11:25:33.216 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:33.216 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.216 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.216 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:25:33.216 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:54.831360+0000", 2024-09-13T11:25:33.216 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:33.216 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.216 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.216 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:25:33.216 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:54.831367+0000", 2024-09-13T11:25:33.216 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.8419999999999999e-06 2024-09-13T11:25:33.216 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.216 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.216 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:25:33.217 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:54.831426+0000", 2024-09-13T11:25:33.217 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.9072e-05 2024-09-13T11:25:33.217 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.217 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.217 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:25:33.217 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:54.992898+0000", 2024-09-13T11:25:33.217 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.161471579 2024-09-13T11:25:33.217 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.217 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.217 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:25:33.217 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:54.992915+0000", 2024-09-13T11:25:33.217 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.7299000000000002e-05 2024-09-13T11:25:33.217 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.217 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.218 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:25:33.218 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:54.993246+0000", 2024-09-13T11:25:33.218 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00033146199999999998 2024-09-13T11:25:33.218 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.218 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.218 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:25:33.218 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:54.993391+0000", 2024-09-13T11:25:33.218 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00014489400000000001 2024-09-13T11:25:33.218 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.218 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.218 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:25:33.218 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:55.007321+0000", 2024-09-13T11:25:33.218 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.013929777000000001 2024-09-13T11:25:33.218 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.219 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.219 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:25:33.219 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:55.007375+0000", 2024-09-13T11:25:33.219 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.3919000000000003e-05 2024-09-13T11:25:33.219 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:33.219 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:25:33.219 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:33.219 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.219 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.219 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:140742 3.1e 3:79c694a0:::benchmark_data_smithi177_31116_object439:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e144)", 2024-09-13T11:25:33.219 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:24:58.321262+0000", 2024-09-13T11:25:33.219 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 34.851304849999998, 2024-09-13T11:25:33.219 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.23727013499999999, 2024-09-13T11:25:33.219 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:25:33.220 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:25:33.220 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:33.220 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:25:33.220 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:25:33.220 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:33.220 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 140742 2024-09-13T11:25:33.220 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.220 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:25:33.220 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.220 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:25:33.220 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:58.321262+0000", 2024-09-13T11:25:33.220 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:33.220 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.220 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.221 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:25:33.221 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:58.321262+0000", 2024-09-13T11:25:33.221 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:33.221 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.221 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.221 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:25:33.221 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:58.321264+0000", 2024-09-13T11:25:33.221 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.581e-06 2024-09-13T11:25:33.221 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.221 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.221 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:25:33.221 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:58.321272+0000", 2024-09-13T11:25:33.221 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.2179999999999999e-06 2024-09-13T11:25:33.222 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.222 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.222 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:25:33.222 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:58.321282+0000", 2024-09-13T11:25:33.222 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0173000000000001e-05 2024-09-13T11:25:33.222 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.222 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.222 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:25:33.222 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:58.321290+0000", 2024-09-13T11:25:33.222 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.943e-06 2024-09-13T11:25:33.222 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.222 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.222 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:25:33.222 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:58.321355+0000", 2024-09-13T11:25:33.223 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.4461000000000005e-05 2024-09-13T11:25:33.223 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.223 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.223 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:25:33.223 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:58.321450+0000", 2024-09-13T11:25:33.223 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.4769e-05 2024-09-13T11:25:33.223 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.223 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.223 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 4", 2024-09-13T11:25:33.223 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:58.321516+0000", 2024-09-13T11:25:33.223 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.5998000000000001e-05 2024-09-13T11:25:33.223 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.223 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.223 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:25:33.224 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:58.324045+0000", 2024-09-13T11:25:33.224 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0025290579999999998 2024-09-13T11:25:33.224 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.224 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.224 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:25:33.224 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:58.558244+0000", 2024-09-13T11:25:33.224 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.234199455 2024-09-13T11:25:33.225 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.225 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.225 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:25:33.225 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:58.558401+0000", 2024-09-13T11:25:33.225 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00015700800000000001 2024-09-13T11:25:33.225 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.225 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.225 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:25:33.225 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:58.558533+0000", 2024-09-13T11:25:33.225 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000131471 2024-09-13T11:25:33.225 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:33.225 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:25:33.226 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:33.226 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.226 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.226 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:140796 3.1e 3:7987a652:::benchmark_data_smithi177_31116_object493:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e144)", 2024-09-13T11:25:33.226 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:24:58.552900+0000", 2024-09-13T11:25:33.226 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 34.619667315000001, 2024-09-13T11:25:33.226 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.254397398, 2024-09-13T11:25:33.226 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:25:33.226 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:25:33.226 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:33.226 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:25:33.226 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:25:33.226 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:33.226 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 140796 2024-09-13T11:25:33.227 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.227 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:25:33.227 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.227 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:25:33.227 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:58.552900+0000", 2024-09-13T11:25:33.227 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:33.227 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.227 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.227 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:25:33.227 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:58.552900+0000", 2024-09-13T11:25:33.227 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:33.227 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.227 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.227 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:25:33.228 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:58.552904+0000", 2024-09-13T11:25:33.228 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.76e-06 2024-09-13T11:25:33.228 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.228 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.228 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:25:33.228 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:58.552915+0000", 2024-09-13T11:25:33.228 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0763e-05 2024-09-13T11:25:33.228 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.228 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.228 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:25:33.228 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:58.552943+0000", 2024-09-13T11:25:33.228 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.8469000000000001e-05 2024-09-13T11:25:33.228 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.228 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.229 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:25:33.229 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:58.552959+0000", 2024-09-13T11:25:33.229 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.5549999999999999e-05 2024-09-13T11:25:33.229 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.229 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.229 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:25:33.229 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:58.553106+0000", 2024-09-13T11:25:33.229 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00014729500000000001 2024-09-13T11:25:33.229 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.229 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.229 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:25:33.229 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:58.553348+0000", 2024-09-13T11:25:33.229 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000242353 2024-09-13T11:25:33.230 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.230 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.230 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 4", 2024-09-13T11:25:33.230 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:58.553556+0000", 2024-09-13T11:25:33.230 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00020765700000000001 2024-09-13T11:25:33.230 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.230 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.230 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:25:33.230 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:58.557140+0000", 2024-09-13T11:25:33.230 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0035844840000000002 2024-09-13T11:25:33.230 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.230 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.230 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:25:33.230 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:58.807126+0000", 2024-09-13T11:25:33.230 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.24998542000000001 2024-09-13T11:25:33.231 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.231 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.231 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:25:33.231 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:58.807214+0000", 2024-09-13T11:25:33.231 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.8561999999999997e-05 2024-09-13T11:25:33.231 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.231 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.231 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:25:33.231 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:58.807297+0000", 2024-09-13T11:25:33.231 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.3084999999999995e-05 2024-09-13T11:25:33.231 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:33.231 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:25:33.231 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:33.232 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.232 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.232 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "MOSDPGPush(3.12 144/122 [PushOp(3:4cd0194d:::benchmark_data_smithi177_31116_object55212:head, version: 79'3122, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:4cd0194d:::benchmark_data_smithi177_31116_object55212:head@79'3122, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:25:33.232 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:24:58.572241+0000", 2024-09-13T11:25:33.232 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 34.600326690000003, 2024-09-13T11:25:33.232 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.20343498300000001, 2024-09-13T11:25:33.232 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:25:33.232 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:25:33.232 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-09-13T11:25:33.232 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:25:33.232 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.232 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:25:33.232 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:58.572241+0000", 2024-09-13T11:25:33.232 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:33.233 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.233 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.233 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:25:33.233 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:58.572241+0000", 2024-09-13T11:25:33.233 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:33.233 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.233 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.233 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:25:33.233 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:58.572244+0000", 2024-09-13T11:25:33.233 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.4929999999999998e-06 2024-09-13T11:25:33.233 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.233 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.233 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:25:33.233 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:58.572321+0000", 2024-09-13T11:25:33.234 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.6983000000000005e-05 2024-09-13T11:25:33.234 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.234 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.234 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:25:33.234 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:58.771433+0000", 2024-09-13T11:25:33.234 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.19911215299999999 2024-09-13T11:25:33.234 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.234 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.234 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:25:33.235 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:58.771462+0000", 2024-09-13T11:25:33.235 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.8345999999999999e-05 2024-09-13T11:25:33.235 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.235 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.235 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:25:33.235 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:58.771615+0000", 2024-09-13T11:25:33.235 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000152981 2024-09-13T11:25:33.235 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.235 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.235 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:25:33.235 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:58.771638+0000", 2024-09-13T11:25:33.235 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.3516e-05 2024-09-13T11:25:33.236 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.236 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.236 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:25:33.236 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:58.775676+0000", 2024-09-13T11:25:33.236 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0040375109999999997 2024-09-13T11:25:33.236 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:33.236 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:25:33.236 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:33.236 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.236 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.236 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop(client.4665.0:147266 3.2d e148/120 3:b77e61ed:::benchmark_data_smithi177_31116_object6963:head v 148'5717, mlcod=46'430)", 2024-09-13T11:25:33.236 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:25:03.948902+0000", 2024-09-13T11:25:33.236 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 29.223665476000001, 2024-09-13T11:25:33.237 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.177609762, 2024-09-13T11:25:33.237 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:25:33.237 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:25:33.237 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:33.237 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:25:33.237 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.237 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:25:33.237 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:03.948902+0000", 2024-09-13T11:25:33.237 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:33.237 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.237 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.237 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:25:33.237 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:03.948902+0000", 2024-09-13T11:25:33.237 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:33.238 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.238 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.238 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:25:33.238 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:03.948903+0000", 2024-09-13T11:25:33.238 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.6020000000000001e-06 2024-09-13T11:25:33.238 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.238 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.238 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:25:33.238 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:03.948911+0000", 2024-09-13T11:25:33.238 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.7020000000000001e-06 2024-09-13T11:25:33.238 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.238 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.238 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:25:33.238 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:04.123318+0000", 2024-09-13T11:25:33.239 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.174406905 2024-09-13T11:25:33.239 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.239 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.239 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:25:33.239 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:04.123322+0000", 2024-09-13T11:25:33.239 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.3919999999999996e-06 2024-09-13T11:25:33.239 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.239 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.239 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:25:33.239 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:04.123378+0000", 2024-09-13T11:25:33.239 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.5930000000000002e-05 2024-09-13T11:25:33.239 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.239 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.239 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:25:33.240 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:04.123405+0000", 2024-09-13T11:25:33.240 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.6570000000000001e-05 2024-09-13T11:25:33.240 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.240 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.240 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:25:33.240 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:04.126478+0000", 2024-09-13T11:25:33.240 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0030730699999999998 2024-09-13T11:25:33.240 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.240 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.240 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:25:33.240 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:04.126512+0000", 2024-09-13T11:25:33.240 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.3590999999999997e-05 2024-09-13T11:25:33.240 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:33.240 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:25:33.241 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:33.241 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.241 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.241 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop(client.4665.0:147472 3.2d e148/120 3:b6aeefb0:::benchmark_data_smithi177_31116_object7169:head v 148'5720, mlcod=46'431)", 2024-09-13T11:25:33.241 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:25:04.137580+0000", 2024-09-13T11:25:33.241 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 29.034987241, 2024-09-13T11:25:33.241 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.214816644, 2024-09-13T11:25:33.241 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:25:33.241 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:25:33.241 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:33.241 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:25:33.241 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.241 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:25:33.242 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:04.137580+0000", 2024-09-13T11:25:33.242 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:33.242 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.242 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.242 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:25:33.242 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:04.137580+0000", 2024-09-13T11:25:33.242 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:33.242 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.242 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.242 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:25:33.242 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:04.137581+0000", 2024-09-13T11:25:33.242 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.2309999999999999e-06 2024-09-13T11:25:33.242 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.242 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.243 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:25:33.243 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:04.137587+0000", 2024-09-13T11:25:33.243 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.879e-06 2024-09-13T11:25:33.243 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.243 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.243 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:25:33.243 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:04.350285+0000", 2024-09-13T11:25:33.243 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.212697737 2024-09-13T11:25:33.243 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.243 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.243 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:25:33.243 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:04.350291+0000", 2024-09-13T11:25:33.243 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.9070000000000004e-06 2024-09-13T11:25:33.243 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.244 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.244 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:25:33.244 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:04.350368+0000", 2024-09-13T11:25:33.244 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.7150000000000005e-05 2024-09-13T11:25:33.244 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.244 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.244 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:25:33.244 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:04.350407+0000", 2024-09-13T11:25:33.244 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.9433e-05 2024-09-13T11:25:33.244 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.244 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.244 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:25:33.244 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:04.352361+0000", 2024-09-13T11:25:33.245 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0019532379999999999 2024-09-13T11:25:33.245 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.245 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.245 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:25:33.245 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:04.352397+0000", 2024-09-13T11:25:33.245 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.6069000000000003e-05 2024-09-13T11:25:33.245 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:33.245 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:25:33.245 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:33.245 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.245 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.245 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "MOSDPGPush(3.29 149/120 [PushOp(3:97fa78fe:::benchmark_data_smithi177_31116_object84063:head, version: 107'4048, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:97fa78fe:::benchmark_data_smithi177_31116_object84063:head@107'4048, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:25:33.245 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:25:06.346022+0000", 2024-09-13T11:25:33.245 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 26.826545018000001, 2024-09-13T11:25:33.246 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.18646221599999999, 2024-09-13T11:25:33.246 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:25:33.246 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:25:33.246 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-09-13T11:25:33.246 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:25:33.246 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.246 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:25:33.246 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:06.346022+0000", 2024-09-13T11:25:33.246 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:33.246 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.246 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.246 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:25:33.246 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:06.346022+0000", 2024-09-13T11:25:33.246 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:33.247 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.247 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.247 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:25:33.247 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:06.346024+0000", 2024-09-13T11:25:33.247 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.4980000000000001e-06 2024-09-13T11:25:33.247 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.247 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.247 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:25:33.247 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:06.346058+0000", 2024-09-13T11:25:33.247 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.4137999999999997e-05 2024-09-13T11:25:33.247 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.247 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.247 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:25:33.247 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:06.531182+0000", 2024-09-13T11:25:33.248 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.18512414399999999 2024-09-13T11:25:33.248 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.248 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.248 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:25:33.248 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:06.531206+0000", 2024-09-13T11:25:33.248 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.4187999999999999e-05 2024-09-13T11:25:33.248 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.248 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.248 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:25:33.248 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:06.531316+0000", 2024-09-13T11:25:33.248 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000109675 2024-09-13T11:25:33.248 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.248 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.249 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:25:33.249 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:06.531330+0000", 2024-09-13T11:25:33.249 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.3947999999999999e-05 2024-09-13T11:25:33.249 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.249 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.249 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:25:33.249 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:06.532485+0000", 2024-09-13T11:25:33.249 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0011546250000000001 2024-09-13T11:25:33.249 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:33.249 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:25:33.249 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:33.249 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.249 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.250 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop(client.4665.0:162073 3.1f e156/122 3:fd1a00ae:::benchmark_data_smithi177_31116_object21770:head v 156'6684, mlcod=156'6679)", 2024-09-13T11:25:33.250 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:25:15.455533+0000", 2024-09-13T11:25:33.250 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 17.717033947000001, 2024-09-13T11:25:33.250 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.24315087499999999, 2024-09-13T11:25:33.250 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:25:33.250 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:25:33.250 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:33.250 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:25:33.250 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.250 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:25:33.250 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:15.455533+0000", 2024-09-13T11:25:33.250 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:33.250 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.251 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.251 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:25:33.251 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:15.455533+0000", 2024-09-13T11:25:33.251 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:33.251 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.251 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.251 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:25:33.251 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:15.455535+0000", 2024-09-13T11:25:33.251 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.42e-06 2024-09-13T11:25:33.251 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.251 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.251 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:25:33.251 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:15.455542+0000", 2024-09-13T11:25:33.251 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.9820000000000002e-06 2024-09-13T11:25:33.252 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.252 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.252 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:25:33.252 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:15.696032+0000", 2024-09-13T11:25:33.252 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.240490483 2024-09-13T11:25:33.252 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.252 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.252 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:25:33.252 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:15.696039+0000", 2024-09-13T11:25:33.252 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.832e-06 2024-09-13T11:25:33.252 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.252 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.252 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:25:33.252 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:15.696098+0000", 2024-09-13T11:25:33.253 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.8901999999999999e-05 2024-09-13T11:25:33.253 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.253 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.253 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:25:33.253 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:15.696115+0000", 2024-09-13T11:25:33.253 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.7479999999999999e-05 2024-09-13T11:25:33.253 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.253 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.253 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:25:33.253 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:15.698637+0000", 2024-09-13T11:25:33.253 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0025215889999999999 2024-09-13T11:25:33.253 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.253 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.253 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:25:33.253 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:15.698684+0000", 2024-09-13T11:25:33.254 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.7187000000000001e-05 2024-09-13T11:25:33.254 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:33.254 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:25:33.254 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:33.254 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.254 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.254 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop(client.4665.0:162102 3.1f e156/122 3:fc0cda6c:::benchmark_data_smithi177_31116_object21799:head v 156'6685, mlcod=156'6680)", 2024-09-13T11:25:33.254 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:25:15.477994+0000", 2024-09-13T11:25:33.254 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 17.694573146, 2024-09-13T11:25:33.254 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.25110996400000002, 2024-09-13T11:25:33.254 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:25:33.254 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:25:33.254 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:33.255 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:25:33.255 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.255 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:25:33.255 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:15.477994+0000", 2024-09-13T11:25:33.255 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:33.255 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.255 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.255 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:25:33.255 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:15.477994+0000", 2024-09-13T11:25:33.255 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:33.255 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.255 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.255 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:25:33.255 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:15.477995+0000", 2024-09-13T11:25:33.256 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0210000000000001e-06 2024-09-13T11:25:33.256 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.256 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.256 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:25:33.256 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:15.478001+0000", 2024-09-13T11:25:33.256 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.0020000000000003e-06 2024-09-13T11:25:33.256 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.256 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.256 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:25:33.256 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:15.726384+0000", 2024-09-13T11:25:33.256 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.248382609 2024-09-13T11:25:33.256 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.256 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.256 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:25:33.257 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:15.726390+0000", 2024-09-13T11:25:33.257 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.1029999999999998e-06 2024-09-13T11:25:33.257 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.257 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.257 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:25:33.257 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:15.727107+0000", 2024-09-13T11:25:33.257 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000717525 2024-09-13T11:25:33.257 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.257 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.257 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:25:33.257 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:15.727130+0000", 2024-09-13T11:25:33.257 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.2572999999999999e-05 2024-09-13T11:25:33.257 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.258 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.258 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:25:33.258 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:15.729066+0000", 2024-09-13T11:25:33.258 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0019362170000000001 2024-09-13T11:25:33.258 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.258 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.258 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:25:33.258 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:15.729104+0000", 2024-09-13T11:25:33.258 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.7914000000000003e-05 2024-09-13T11:25:33.258 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:33.258 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:25:33.258 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:33.258 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.258 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.259 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "MOSDPGPush(3.28 162/120 [PushOp(3:1627afc8:::benchmark_data_smithi177_31116_object59077:head, version: 82'3191, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:1627afc8:::benchmark_data_smithi177_31116_object59077:head@82'3191, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:25:33.259 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:25:21.908022+0000", 2024-09-13T11:25:33.259 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 11.264544845, 2024-09-13T11:25:33.259 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.19387715999999999, 2024-09-13T11:25:33.259 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:25:33.259 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:25:33.259 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-09-13T11:25:33.259 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:25:33.259 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.259 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:25:33.259 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:21.908022+0000", 2024-09-13T11:25:33.259 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:33.259 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.260 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.260 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:25:33.260 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:21.908022+0000", 2024-09-13T11:25:33.260 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:33.260 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.260 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.260 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:25:33.260 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:21.908037+0000", 2024-09-13T11:25:33.260 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.4474000000000001e-05 2024-09-13T11:25:33.260 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.260 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.260 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:25:33.260 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:21.908079+0000", 2024-09-13T11:25:33.261 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.1984000000000003e-05 2024-09-13T11:25:33.261 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.261 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.261 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:25:33.261 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:22.100612+0000", 2024-09-13T11:25:33.261 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.19253292499999999 2024-09-13T11:25:33.261 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.261 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.261 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:25:33.261 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:22.100627+0000", 2024-09-13T11:25:33.261 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.4711e-05 2024-09-13T11:25:33.261 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.261 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.261 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:25:33.262 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:22.100725+0000", 2024-09-13T11:25:33.262 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.8301000000000006e-05 2024-09-13T11:25:33.262 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.262 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.262 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:25:33.262 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:22.100735+0000", 2024-09-13T11:25:33.262 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0061000000000001e-05 2024-09-13T11:25:33.262 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.262 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.262 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:25:33.262 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:22.101900+0000", 2024-09-13T11:25:33.262 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0011647039999999999 2024-09-13T11:25:33.262 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:33.262 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:25:33.263 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:33.263 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.263 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.263 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop(client.4665.0:175609 3.22 e166/122 3:44988182:::benchmark_data_smithi177_31116_object35306:head v 166'4981, mlcod=166'4978)", 2024-09-13T11:25:33.263 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:25:25.854560+0000", 2024-09-13T11:25:33.263 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 7.3180070500000003, 2024-09-13T11:25:33.263 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.191045139, 2024-09-13T11:25:33.263 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:25:33.263 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:25:33.263 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:33.263 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:25:33.263 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.263 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:25:33.264 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:25.854560+0000", 2024-09-13T11:25:33.264 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:33.264 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.264 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.264 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:25:33.264 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:25.854560+0000", 2024-09-13T11:25:33.264 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:33.264 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.264 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.264 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:25:33.264 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:25.854562+0000", 2024-09-13T11:25:33.264 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.728e-06 2024-09-13T11:25:33.265 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.265 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.265 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:25:33.265 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:25.854570+0000", 2024-09-13T11:25:33.265 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.6249999999999998e-06 2024-09-13T11:25:33.265 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.265 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.265 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:25:33.265 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:26.042889+0000", 2024-09-13T11:25:33.265 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.18831914699999999 2024-09-13T11:25:33.265 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.265 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.265 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:25:33.266 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:26.042902+0000", 2024-09-13T11:25:33.266 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.2894000000000001e-05 2024-09-13T11:25:33.266 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.266 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.266 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:25:33.266 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:26.042991+0000", 2024-09-13T11:25:33.266 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.9438999999999995e-05 2024-09-13T11:25:33.266 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.266 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.266 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:25:33.266 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:26.043011+0000", 2024-09-13T11:25:33.266 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.9806e-05 2024-09-13T11:25:33.266 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.267 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.267 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:25:33.267 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:26.045553+0000", 2024-09-13T11:25:33.267 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.002541892 2024-09-13T11:25:33.267 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.267 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.267 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:25:33.267 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:26.045605+0000", 2024-09-13T11:25:33.267 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.2608000000000002e-05 2024-09-13T11:25:33.267 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:33.267 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:25:33.267 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:33.268 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.268 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.268 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "MOSDPGPush(3.28 168/120 [PushOp(3:14eb8ed9:::benchmark_data_smithi177_31116_object87538:head, version: 110'4071, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:14eb8ed9:::benchmark_data_smithi177_31116_object87538:head@110'4071, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:25:33.268 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:25:27.274802+0000", 2024-09-13T11:25:33.268 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 5.8977655980000003, 2024-09-13T11:25:33.268 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.369268239, 2024-09-13T11:25:33.268 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:25:33.268 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:25:33.268 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-09-13T11:25:33.268 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:25:33.268 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.268 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:25:33.268 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:27.274802+0000", 2024-09-13T11:25:33.269 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:33.269 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.269 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.269 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:25:33.269 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:27.274802+0000", 2024-09-13T11:25:33.269 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:33.269 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.269 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.269 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:25:33.269 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:27.274803+0000", 2024-09-13T11:25:33.269 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.7579999999999999e-06 2024-09-13T11:25:33.269 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.269 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.269 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:25:33.270 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:27.274831+0000", 2024-09-13T11:25:33.270 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.7464e-05 2024-09-13T11:25:33.270 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.270 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.270 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:25:33.270 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:27.641355+0000", 2024-09-13T11:25:33.270 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.36652425199999999 2024-09-13T11:25:33.270 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.270 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.270 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:25:33.270 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:27.641385+0000", 2024-09-13T11:25:33.270 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.9433000000000001e-05 2024-09-13T11:25:33.270 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.270 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.271 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:25:33.271 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:27.641611+0000", 2024-09-13T11:25:33.271 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00022646299999999999 2024-09-13T11:25:33.271 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.271 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.271 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:25:33.271 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:27.641638+0000", 2024-09-13T11:25:33.271 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.6482000000000002e-05 2024-09-13T11:25:33.271 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.271 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.271 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:25:33.271 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:27.644070+0000", 2024-09-13T11:25:33.271 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0024323869999999998 2024-09-13T11:25:33.272 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:33.272 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:25:33.272 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:33.272 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:33.272 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:25:33.272 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-09-13T11:25:33.274 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-09-13T11:25:33.274 INFO:tasks.ceph.osd.9.smithi092.stderr:2024-09-13T11:25:33.269+0000 7ff59ddf9640 -1 osd.9 171 handle_osd_map injecting map crc failure 2024-09-13T11:25:33.274 INFO:tasks.ceph.osd.5.smithi092.stderr:2024-09-13T11:25:33.273+0000 7f6f0b705640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26346) UID: 0 2024-09-13T11:25:33.374 INFO:tasks.ceph.osd.3.smithi177.stderr:2024-09-13T11:25:33.370+0000 7f10adba4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26574) UID: 0 2024-09-13T11:25:33.458 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-09-13T11:25:33.459 INFO:teuthology.orchestra.run.smithi092.stdout: "ops": [ 2024-09-13T11:25:33.459 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.459 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:185024 3.19 3:9eb415f7:::benchmark_data_smithi177_31116_object44721:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e172)", 2024-09-13T11:25:33.459 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:25:33.444032+0000", 2024-09-13T11:25:33.459 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 0.014493826, 2024-09-13T11:25:33.459 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.014521024, 2024-09-13T11:25:33.459 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:25:33.459 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:25:33.459 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "waiting for sub ops", 2024-09-13T11:25:33.459 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:25:33.459 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:25:33.459 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:33.460 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 185024 2024-09-13T11:25:33.460 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.460 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:25:33.460 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.460 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:25:33.460 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:33.444032+0000", 2024-09-13T11:25:33.460 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:33.460 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.460 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.460 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:25:33.460 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:33.444032+0000", 2024-09-13T11:25:33.460 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:33.460 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.461 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.461 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:25:33.461 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:33.444034+0000", 2024-09-13T11:25:33.461 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.844e-06 2024-09-13T11:25:33.461 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.461 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.461 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:25:33.461 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:33.444043+0000", 2024-09-13T11:25:33.461 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.0040000000000005e-06 2024-09-13T11:25:33.461 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.461 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.461 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:25:33.461 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:33.444057+0000", 2024-09-13T11:25:33.461 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.4151999999999999e-05 2024-09-13T11:25:33.462 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.462 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.462 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:25:33.462 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:33.444065+0000", 2024-09-13T11:25:33.462 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.9449999999999993e-06 2024-09-13T11:25:33.462 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.462 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.462 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:25:33.462 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:33.445435+0000", 2024-09-13T11:25:33.462 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0013692789999999999 2024-09-13T11:25:33.462 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.462 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.462 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:25:33.462 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:33.445574+0000", 2024-09-13T11:25:33.463 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00013935299999999999 2024-09-13T11:25:33.463 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.463 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.463 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1", 2024-09-13T11:25:33.463 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:33.445695+0000", 2024-09-13T11:25:33.463 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00012053900000000001 2024-09-13T11:25:33.463 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.463 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.463 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:25:33.463 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:33.448740+0000", 2024-09-13T11:25:33.463 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.003045037 2024-09-13T11:25:33.463 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:33.463 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:25:33.463 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:33.464 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.464 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.464 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop(client.4665.0:185041 3.23 e172/90 3:c7d4611c:::benchmark_data_smithi177_31116_object44738:head v 172'5206, mlcod=172'5204)", 2024-09-13T11:25:33.464 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:25:33.457358+0000", 2024-09-13T11:25:33.464 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 0.0011684709999999999, 2024-09-13T11:25:33.464 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0012768689999999999, 2024-09-13T11:25:33.464 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:25:33.464 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:25:33.464 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-09-13T11:25:33.464 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:25:33.464 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.464 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:25:33.464 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:33.457358+0000", 2024-09-13T11:25:33.465 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:33.465 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.465 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.465 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:25:33.465 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:33.457358+0000", 2024-09-13T11:25:33.465 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:33.465 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.465 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.465 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:25:33.465 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:33.457363+0000", 2024-09-13T11:25:33.465 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.8879999999999996e-06 2024-09-13T11:25:33.465 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.465 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.465 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:25:33.466 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:33.457372+0000", 2024-09-13T11:25:33.466 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.5969999999999998e-06 2024-09-13T11:25:33.466 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.466 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.466 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:25:33.466 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:33.457430+0000", 2024-09-13T11:25:33.466 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.7738999999999997e-05 2024-09-13T11:25:33.466 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.466 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.466 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:25:33.466 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:33.457440+0000", 2024-09-13T11:25:33.466 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.9709999999999999e-06 2024-09-13T11:25:33.466 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.466 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.467 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:25:33.467 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:33.457503+0000", 2024-09-13T11:25:33.467 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.3398999999999994e-05 2024-09-13T11:25:33.467 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.467 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.467 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:25:33.467 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:33.457523+0000", 2024-09-13T11:25:33.467 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.9276999999999999e-05 2024-09-13T11:25:33.467 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:33.467 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:25:33.467 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:33.467 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:33.467 INFO:teuthology.orchestra.run.smithi092.stdout: ], 2024-09-13T11:25:33.467 INFO:teuthology.orchestra.run.smithi092.stdout: "num_ops": 2 2024-09-13T11:25:33.468 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-09-13T11:25:33.468 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T11:25:33.474 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-09-13T11:25:33.472+0000 7f97befc9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26315) UID: 0 2024-09-13T11:25:33.477 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-09-13T11:25:33.575 INFO:tasks.ceph.osd.10.smithi162.stderr:2024-09-13T11:25:33.572+0000 7fb59c9f0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26317) UID: 0 2024-09-13T11:25:33.611 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-09-13T11:25:33.611 INFO:teuthology.orchestra.run.smithi092.stdout: "ops": [], 2024-09-13T11:25:33.611 INFO:teuthology.orchestra.run.smithi092.stdout: "complaint_time": 30, 2024-09-13T11:25:33.611 INFO:teuthology.orchestra.run.smithi092.stdout: "num_blocked_ops": 0 2024-09-13T11:25:33.611 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-09-13T11:25:33.626 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-09-13T11:25:33.636 INFO:teuthology.orchestra.run.smithi001.stdout:osd.0: {} 2024-09-13T11:25:33.636 INFO:teuthology.orchestra.run.smithi001.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T11:25:33.659 INFO:teuthology.orchestra.run.smithi001.stdout:osd.1: {} 2024-09-13T11:25:33.659 INFO:teuthology.orchestra.run.smithi001.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T11:25:33.676 INFO:tasks.ceph.osd.10.smithi162.stderr:2024-09-13T11:25:33.672+0000 7fb59c9f0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26317) UID: 0 2024-09-13T11:25:33.685 INFO:teuthology.orchestra.run.smithi001.stdout:osd.2: {} 2024-09-13T11:25:33.685 INFO:teuthology.orchestra.run.smithi001.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T11:25:33.710 INFO:teuthology.orchestra.run.smithi001.stdout:osd.3: {} 2024-09-13T11:25:33.710 INFO:teuthology.orchestra.run.smithi001.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T11:25:33.731 INFO:teuthology.orchestra.run.smithi001.stdout:osd.4: {} 2024-09-13T11:25:33.731 INFO:teuthology.orchestra.run.smithi001.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T11:25:33.743 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-09-13T11:25:33.743 INFO:teuthology.orchestra.run.smithi092.stdout: "size": 20, 2024-09-13T11:25:33.743 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 600, 2024-09-13T11:25:33.743 INFO:teuthology.orchestra.run.smithi092.stdout: "ops": [ 2024-09-13T11:25:33.743 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.744 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "MOSDPGPush(3.19 76/70 [PushOp(3:9eacf3d0:::benchmark_data_smithi177_31116_object19747:head, version: 54'1236, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:9eacf3d0:::benchmark_data_smithi177_31116_object19747:head@54'1236, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:25:33.744 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:32.704676+0000", 2024-09-13T11:25:33.744 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 121.03769912, 2024-09-13T11:25:33.744 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.23525014899999999, 2024-09-13T11:25:33.744 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:25:33.744 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:25:33.744 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-09-13T11:25:33.744 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:25:33.744 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.744 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:25:33.744 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:32.704676+0000", 2024-09-13T11:25:33.744 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:33.744 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.744 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.744 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:25:33.745 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:32.704676+0000", 2024-09-13T11:25:33.745 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:33.745 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.745 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.745 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:25:33.745 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:32.704677+0000", 2024-09-13T11:25:33.745 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.531e-06 2024-09-13T11:25:33.745 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.745 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.745 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:25:33.745 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:32.704749+0000", 2024-09-13T11:25:33.745 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.1451000000000004e-05 2024-09-13T11:25:33.745 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.745 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.746 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:25:33.746 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:32.938053+0000", 2024-09-13T11:25:33.746 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.233304752 2024-09-13T11:25:33.746 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.746 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.746 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:25:33.746 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:32.938087+0000", 2024-09-13T11:25:33.746 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.4016000000000001e-05 2024-09-13T11:25:33.746 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.746 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.746 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:25:33.746 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:32.938351+0000", 2024-09-13T11:25:33.746 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00026397500000000001 2024-09-13T11:25:33.746 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.746 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.746 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:25:33.747 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:32.938381+0000", 2024-09-13T11:25:33.747 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.0001999999999998e-05 2024-09-13T11:25:33.747 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.747 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.747 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:25:33.747 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:32.939926+0000", 2024-09-13T11:25:33.747 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0015444219999999999 2024-09-13T11:25:33.747 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:33.747 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:25:33.747 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:33.747 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.747 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.747 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "MOSDPGPush(3.23 80/78 [PushOp(3:c59ecc6d:::benchmark_data_smithi177_31116_object17005:head, version: 52'1054, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:c59ecc6d:::benchmark_data_smithi177_31116_object17005:head@52'1054, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:25:33.747 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:37.433040+0000", 2024-09-13T11:25:33.747 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 116.309335144, 2024-09-13T11:25:33.748 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.26658219100000002, 2024-09-13T11:25:33.748 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:25:33.748 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:25:33.748 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-09-13T11:25:33.748 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:25:33.748 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.748 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:25:33.748 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:37.433040+0000", 2024-09-13T11:25:33.748 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:33.748 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.748 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.748 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:25:33.748 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:37.433040+0000", 2024-09-13T11:25:33.748 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:33.748 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.749 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.749 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:25:33.749 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:37.433042+0000", 2024-09-13T11:25:33.749 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.3999999999999999e-06 2024-09-13T11:25:33.749 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.749 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.749 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:25:33.749 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:37.433090+0000", 2024-09-13T11:25:33.749 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.8065999999999997e-05 2024-09-13T11:25:33.749 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.749 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.749 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:25:33.749 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:37.698797+0000", 2024-09-13T11:25:33.749 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.26570674500000002 2024-09-13T11:25:33.749 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.750 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.750 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:25:33.750 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:37.698827+0000", 2024-09-13T11:25:33.750 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.0505e-05 2024-09-13T11:25:33.750 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.750 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.750 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:25:33.750 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:37.698941+0000", 2024-09-13T11:25:33.750 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00011403799999999999 2024-09-13T11:25:33.750 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.750 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.750 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:25:33.750 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:37.698956+0000", 2024-09-13T11:25:33.750 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.4765e-05 2024-09-13T11:25:33.750 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.751 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.751 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:25:33.751 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:37.699622+0000", 2024-09-13T11:25:33.751 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00066567199999999997 2024-09-13T11:25:33.751 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:33.751 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:25:33.751 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:33.751 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.751 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.751 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "MOSDPGPush(3.1a 81/78 [PushOp(3:5a9c05b8:::benchmark_data_smithi177_31116_object4600:head, version: 44'280, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:5a9c05b8:::benchmark_data_smithi177_31116_object4600:head@44'280, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:25:33.751 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:38.261429+0000", 2024-09-13T11:25:33.751 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 115.48094570799999, 2024-09-13T11:25:33.751 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.13814789799999999, 2024-09-13T11:25:33.751 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:25:33.751 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:25:33.751 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-09-13T11:25:33.752 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:25:33.752 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.752 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:25:33.752 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:38.261429+0000", 2024-09-13T11:25:33.752 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:33.752 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.752 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.752 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:25:33.752 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:38.261429+0000", 2024-09-13T11:25:33.752 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:33.752 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.752 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.752 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:25:33.752 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:38.261430+0000", 2024-09-13T11:25:33.753 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.376e-06 2024-09-13T11:25:33.753 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.753 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.753 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:25:33.753 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:38.261501+0000", 2024-09-13T11:25:33.753 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.0724999999999998e-05 2024-09-13T11:25:33.753 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.753 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.753 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:25:33.753 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:38.398898+0000", 2024-09-13T11:25:33.753 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.13739662499999999 2024-09-13T11:25:33.753 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.753 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.753 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:25:33.754 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:38.398909+0000", 2024-09-13T11:25:33.754 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0926e-05 2024-09-13T11:25:33.754 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.754 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.754 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:25:33.754 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:38.398974+0000", 2024-09-13T11:25:33.754 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.5467000000000002e-05 2024-09-13T11:25:33.754 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.754 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.754 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:25:33.754 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:38.398982+0000", 2024-09-13T11:25:33.754 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.841e-06 2024-09-13T11:25:33.754 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.755 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.755 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:25:33.755 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:38.399577+0000", 2024-09-13T11:25:33.755 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00059493800000000004 2024-09-13T11:25:33.755 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:33.755 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:25:33.755 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:33.755 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.755 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.755 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "MOSDPGPush(3.1a 85/78 [PushOp(3:5dc85525:::benchmark_data_smithi177_31116_object11755:head, version: 49'723, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:5dc85525:::benchmark_data_smithi177_31116_object11755:head@49'723, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:25:33.755 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:42.422061+0000", 2024-09-13T11:25:33.755 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 111.320313691, 2024-09-13T11:25:33.756 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.52875617500000005, 2024-09-13T11:25:33.756 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:25:33.756 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:25:33.756 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-09-13T11:25:33.756 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:25:33.756 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.756 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:25:33.756 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.422061+0000", 2024-09-13T11:25:33.756 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:33.756 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.756 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.757 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:25:33.757 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.422061+0000", 2024-09-13T11:25:33.757 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:33.757 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.757 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.757 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:25:33.757 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.422064+0000", 2024-09-13T11:25:33.757 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.5409999999999999e-06 2024-09-13T11:25:33.757 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.758 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.758 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:25:33.758 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.422139+0000", 2024-09-13T11:25:33.758 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.5488999999999995e-05 2024-09-13T11:25:33.758 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.758 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.758 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:25:33.758 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.948781+0000", 2024-09-13T11:25:33.758 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.52664230999999995 2024-09-13T11:25:33.758 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.758 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.758 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:25:33.759 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.948841+0000", 2024-09-13T11:25:33.759 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.9450999999999997e-05 2024-09-13T11:25:33.759 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.759 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.759 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:25:33.759 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.949035+0000", 2024-09-13T11:25:33.759 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00019380600000000001 2024-09-13T11:25:33.759 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.759 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.759 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:25:33.759 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.949060+0000", 2024-09-13T11:25:33.759 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.5381999999999999e-05 2024-09-13T11:25:33.759 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.760 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.760 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:25:33.760 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.950817+0000", 2024-09-13T11:25:33.760 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0017571959999999999 2024-09-13T11:25:33.760 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:33.760 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:25:33.760 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:33.760 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.760 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.760 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "MOSDPGPush(3.1a 89/78 [PushOp(3:59270734:::benchmark_data_smithi177_31116_object20039:head, version: 54'1256, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:59270734:::benchmark_data_smithi177_31116_object20039:head@54'1256, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:25:33.760 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:46.607562+0000", 2024-09-13T11:25:33.760 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 107.13481291399999, 2024-09-13T11:25:33.760 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.32123945300000001, 2024-09-13T11:25:33.760 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:25:33.761 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:25:33.761 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-09-13T11:25:33.761 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:25:33.761 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.761 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:25:33.761 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.607562+0000", 2024-09-13T11:25:33.761 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:33.761 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.761 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.761 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:25:33.761 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.607562+0000", 2024-09-13T11:25:33.761 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:33.761 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.761 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.762 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:25:33.762 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.607564+0000", 2024-09-13T11:25:33.762 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.8610000000000001e-06 2024-09-13T11:25:33.762 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.762 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.762 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:25:33.762 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.607663+0000", 2024-09-13T11:25:33.762 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.9210999999999996e-05 2024-09-13T11:25:33.762 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.762 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.762 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:25:33.762 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.925193+0000", 2024-09-13T11:25:33.762 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.317530586 2024-09-13T11:25:33.762 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.763 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.763 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:25:33.763 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.925250+0000", 2024-09-13T11:25:33.763 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.6925000000000001e-05 2024-09-13T11:25:33.763 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.763 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.763 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:25:33.763 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.927013+0000", 2024-09-13T11:25:33.763 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.001762853 2024-09-13T11:25:33.763 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.763 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.763 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:25:33.763 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.927067+0000", 2024-09-13T11:25:33.763 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.3601999999999999e-05 2024-09-13T11:25:33.763 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.763 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.764 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:25:33.764 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.928801+0000", 2024-09-13T11:25:33.764 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.001734415 2024-09-13T11:25:33.764 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:33.764 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:25:33.764 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:33.764 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.764 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.764 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "MOSDPGPush(3.1a 91/78 [PushOp(3:5e5010d2:::benchmark_data_smithi177_31116_object25733:head, version: 58'1609, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:5e5010d2:::benchmark_data_smithi177_31116_object25733:head@58'1609, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:25:33.764 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:48.608485+0000", 2024-09-13T11:25:33.764 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 105.13388979600001, 2024-09-13T11:25:33.764 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.35035599699999997, 2024-09-13T11:25:33.764 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:25:33.764 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:25:33.764 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-09-13T11:25:33.765 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:25:33.765 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.765 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:25:33.765 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:48.608485+0000", 2024-09-13T11:25:33.765 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:33.765 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.765 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.765 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:25:33.765 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:48.608485+0000", 2024-09-13T11:25:33.765 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:33.765 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.765 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.765 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:25:33.765 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:48.608488+0000", 2024-09-13T11:25:33.765 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.9399999999999998e-06 2024-09-13T11:25:33.766 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.766 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.766 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:25:33.766 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:48.608552+0000", 2024-09-13T11:25:33.766 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.3731000000000004e-05 2024-09-13T11:25:33.766 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.766 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.766 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:25:33.766 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:48.957330+0000", 2024-09-13T11:25:33.766 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.34877806099999997 2024-09-13T11:25:33.766 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.766 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.766 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:25:33.766 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:48.957361+0000", 2024-09-13T11:25:33.766 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.0852000000000002e-05 2024-09-13T11:25:33.766 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.767 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.767 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:25:33.767 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:48.957544+0000", 2024-09-13T11:25:33.767 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000183537 2024-09-13T11:25:33.767 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.767 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.767 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:25:33.767 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:48.957582+0000", 2024-09-13T11:25:33.767 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.7472999999999998e-05 2024-09-13T11:25:33.767 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.767 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.767 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:25:33.767 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:48.958841+0000", 2024-09-13T11:25:33.767 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0012594030000000001 2024-09-13T11:25:33.768 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:33.768 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:25:33.768 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:33.768 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.768 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.768 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop(client.4665.0:72923 3.23 e97/90 3:c6230123:::benchmark_data_smithi177_31116_object72922:head v 97'3418, mlcod=97'3414)", 2024-09-13T11:25:33.768 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:55.047734+0000", 2024-09-13T11:25:33.768 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 98.694640278999998, 2024-09-13T11:25:33.768 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.20075528000000001, 2024-09-13T11:25:33.768 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:25:33.768 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:25:33.768 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:33.768 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:25:33.768 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.768 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:25:33.768 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:55.047734+0000", 2024-09-13T11:25:33.769 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:33.769 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.769 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.769 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:25:33.769 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:55.047734+0000", 2024-09-13T11:25:33.769 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:33.769 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.769 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.769 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:25:33.769 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:55.047736+0000", 2024-09-13T11:25:33.769 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.466e-06 2024-09-13T11:25:33.769 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.769 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.769 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:25:33.770 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:55.047772+0000", 2024-09-13T11:25:33.770 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.6576999999999999e-05 2024-09-13T11:25:33.770 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.770 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.770 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:25:33.770 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:55.246478+0000", 2024-09-13T11:25:33.770 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.19870581300000001 2024-09-13T11:25:33.770 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.770 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.770 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:25:33.770 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:55.246488+0000", 2024-09-13T11:25:33.770 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.5419999999999994e-06 2024-09-13T11:25:33.770 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.770 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.770 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:25:33.770 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:55.246631+0000", 2024-09-13T11:25:33.771 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00014287300000000001 2024-09-13T11:25:33.771 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.771 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.771 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:25:33.771 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:55.246665+0000", 2024-09-13T11:25:33.771 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.4684999999999997e-05 2024-09-13T11:25:33.771 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.771 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.771 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:25:33.771 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:55.248453+0000", 2024-09-13T11:25:33.771 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0017879459999999999 2024-09-13T11:25:33.771 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.771 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.771 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:25:33.771 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:55.248490+0000", 2024-09-13T11:25:33.771 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.6378000000000003e-05 2024-09-13T11:25:33.772 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:33.772 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:25:33.772 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:33.772 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.772 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.772 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:79823 3.19 3:9e038f3c:::benchmark_data_smithi177_31116_object79822:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102)", 2024-09-13T11:25:33.772 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:24:00.563188+0000", 2024-09-13T11:25:33.772 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 93.179186654000006, 2024-09-13T11:25:33.772 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.164156422, 2024-09-13T11:25:33.772 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:25:33.772 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:25:33.772 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:33.772 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:25:33.772 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:25:33.773 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:33.773 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 79823 2024-09-13T11:25:33.773 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.773 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:25:33.773 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.773 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:25:33.773 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:00.563188+0000", 2024-09-13T11:25:33.773 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:33.774 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.774 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.774 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:25:33.774 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:00.563188+0000", 2024-09-13T11:25:33.774 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:33.774 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.774 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.774 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:25:33.774 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:00.563201+0000", 2024-09-13T11:25:33.774 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.3183000000000001e-05 2024-09-13T11:25:33.774 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.774 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.775 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:25:33.775 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:00.563263+0000", 2024-09-13T11:25:33.775 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.1904999999999998e-05 2024-09-13T11:25:33.775 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.775 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.775 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:25:33.775 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:00.563297+0000", 2024-09-13T11:25:33.775 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.3979000000000001e-05 2024-09-13T11:25:33.775 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.775 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.775 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:25:33.775 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:00.563318+0000", 2024-09-13T11:25:33.775 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.0703000000000002e-05 2024-09-13T11:25:33.775 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.775 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.776 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:25:33.776 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:00.563513+0000", 2024-09-13T11:25:33.776 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00019524200000000001 2024-09-13T11:25:33.776 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.776 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.776 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:25:33.776 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:00.563758+0000", 2024-09-13T11:25:33.776 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000244711 2024-09-13T11:25:33.776 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.776 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.776 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1", 2024-09-13T11:25:33.776 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:00.563978+0000", 2024-09-13T11:25:33.776 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00022011700000000001 2024-09-13T11:25:33.776 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.776 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.777 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:25:33.777 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:00.566893+0000", 2024-09-13T11:25:33.777 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0029147650000000002 2024-09-13T11:25:33.777 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.777 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.777 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:25:33.777 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:00.727176+0000", 2024-09-13T11:25:33.777 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.16028374100000001 2024-09-13T11:25:33.777 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.777 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.777 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:25:33.777 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:00.727271+0000", 2024-09-13T11:25:33.777 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.4481000000000006e-05 2024-09-13T11:25:33.777 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.777 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.778 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:25:33.778 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:00.727344+0000", 2024-09-13T11:25:33.778 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.3595000000000003e-05 2024-09-13T11:25:33.778 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:33.778 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:25:33.778 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:33.778 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.778 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.778 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop_reply(client.4665.0:79823 3.19 e102/84 ondisk, result = 0)", 2024-09-13T11:25:33.778 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:24:00.567585+0000", 2024-09-13T11:25:33.778 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 93.174789343, 2024-09-13T11:25:33.778 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.15977266400000001, 2024-09-13T11:25:33.778 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:25:33.778 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:25:33.778 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-09-13T11:25:33.779 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:25:33.779 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.779 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:25:33.779 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:00.567585+0000", 2024-09-13T11:25:33.779 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:33.779 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.779 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.779 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:25:33.779 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:00.567585+0000", 2024-09-13T11:25:33.779 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:33.779 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.779 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.779 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:25:33.779 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:00.567589+0000", 2024-09-13T11:25:33.779 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.2969999999999999e-06 2024-09-13T11:25:33.779 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.780 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.780 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:25:33.780 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:00.567599+0000", 2024-09-13T11:25:33.780 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0142999999999999e-05 2024-09-13T11:25:33.780 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.780 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.780 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:25:33.780 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:00.726959+0000", 2024-09-13T11:25:33.780 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.15936018399999999 2024-09-13T11:25:33.780 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.780 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.780 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:25:33.780 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:00.726972+0000", 2024-09-13T11:25:33.780 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.2622999999999999e-05 2024-09-13T11:25:33.781 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.781 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.781 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:25:33.781 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:00.727151+0000", 2024-09-13T11:25:33.781 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00017909600000000001 2024-09-13T11:25:33.781 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.781 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.781 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:25:33.781 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:00.727167+0000", 2024-09-13T11:25:33.781 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.6639000000000001e-05 2024-09-13T11:25:33.781 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.781 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.781 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:25:33.781 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:00.727358+0000", 2024-09-13T11:25:33.781 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00019068199999999999 2024-09-13T11:25:33.781 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:33.782 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:25:33.782 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:33.782 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.782 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.782 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop(client.4665.0:98582 3.1a e113/105 3:59f9cd75:::benchmark_data_smithi177_31116_object98581:head v 113'4743, mlcod=113'4741)", 2024-09-13T11:25:33.782 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:24:15.432027+0000", 2024-09-13T11:25:33.782 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 78.310347664000005, 2024-09-13T11:25:33.782 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.196879784, 2024-09-13T11:25:33.782 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:25:33.782 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:25:33.782 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:33.782 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:25:33.782 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.782 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:25:33.782 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:15.432027+0000", 2024-09-13T11:25:33.783 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:33.783 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.783 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.783 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:25:33.783 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:15.432027+0000", 2024-09-13T11:25:33.783 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:33.783 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.783 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.783 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:25:33.783 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:15.432029+0000", 2024-09-13T11:25:33.783 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.838e-06 2024-09-13T11:25:33.783 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.783 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.783 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:25:33.783 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:15.432088+0000", 2024-09-13T11:25:33.783 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.8767999999999997e-05 2024-09-13T11:25:33.784 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.784 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.784 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:25:33.784 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:15.624731+0000", 2024-09-13T11:25:33.784 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.192643221 2024-09-13T11:25:33.784 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.784 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.784 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:25:33.784 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:15.624746+0000", 2024-09-13T11:25:33.784 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.4705000000000001e-05 2024-09-13T11:25:33.784 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.784 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.784 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:25:33.784 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:15.624882+0000", 2024-09-13T11:25:33.784 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00013602300000000001 2024-09-13T11:25:33.785 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.785 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.785 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:25:33.785 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:15.624941+0000", 2024-09-13T11:25:33.785 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.9351000000000001e-05 2024-09-13T11:25:33.785 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.785 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.785 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:25:33.785 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:15.628836+0000", 2024-09-13T11:25:33.785 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0038955109999999999 2024-09-13T11:25:33.785 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.785 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.785 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:25:33.785 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:15.628907+0000", 2024-09-13T11:25:33.785 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.0366999999999999e-05 2024-09-13T11:25:33.786 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:33.786 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:25:33.786 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:33.786 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.786 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.786 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop(client.4665.0:99105 3.1a e113/105 3:5ac5090c:::benchmark_data_smithi177_31116_object99104:head v 113'4749, mlcod=113'4746)", 2024-09-13T11:25:33.786 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:24:15.963602+0000", 2024-09-13T11:25:33.786 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 77.778772481999994, 2024-09-13T11:25:33.786 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.23699906300000001, 2024-09-13T11:25:33.786 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:25:33.786 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:25:33.786 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:33.786 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:25:33.786 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.786 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:25:33.786 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:15.963602+0000", 2024-09-13T11:25:33.787 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:33.787 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.787 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.787 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:25:33.787 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:15.963602+0000", 2024-09-13T11:25:33.787 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:33.787 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.787 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.787 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:25:33.787 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:15.963605+0000", 2024-09-13T11:25:33.787 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.0400000000000001e-06 2024-09-13T11:25:33.787 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.787 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.787 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:25:33.787 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:15.963718+0000", 2024-09-13T11:25:33.788 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000112796 2024-09-13T11:25:33.788 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.788 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.788 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:25:33.788 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:16.197994+0000", 2024-09-13T11:25:33.788 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.234276285 2024-09-13T11:25:33.788 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.788 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.788 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:25:33.788 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:16.198004+0000", 2024-09-13T11:25:33.788 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.5559999999999996e-06 2024-09-13T11:25:33.788 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.788 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.788 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:25:33.788 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:16.198110+0000", 2024-09-13T11:25:33.789 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00010614 2024-09-13T11:25:33.789 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.789 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.789 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:25:33.789 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:16.198152+0000", 2024-09-13T11:25:33.789 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.1899e-05 2024-09-13T11:25:33.789 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.789 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.789 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:25:33.789 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:16.200528+0000", 2024-09-13T11:25:33.789 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00237587 2024-09-13T11:25:33.789 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.789 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.789 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:25:33.790 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:16.200601+0000", 2024-09-13T11:25:33.790 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.3477000000000001e-05 2024-09-13T11:25:33.790 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:33.790 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:25:33.790 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:33.790 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.790 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.790 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop(client.4665.0:119446 3.1a e128/105 3:5b664280:::benchmark_data_smithi177_31116_object119445:head v 128'5409, mlcod=128'5405)", 2024-09-13T11:25:33.790 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:24:37.343796+0000", 2024-09-13T11:25:33.790 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 56.398578985999997, 2024-09-13T11:25:33.790 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.118847789, 2024-09-13T11:25:33.790 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:25:33.790 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:25:33.790 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:33.790 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:25:33.790 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.791 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:25:33.791 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:37.343796+0000", 2024-09-13T11:25:33.791 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:33.791 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.791 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.791 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:25:33.791 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:37.343796+0000", 2024-09-13T11:25:33.791 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:33.791 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.791 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.791 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:25:33.791 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:37.343799+0000", 2024-09-13T11:25:33.791 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.8590000000000001e-06 2024-09-13T11:25:33.791 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.791 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.791 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:25:33.792 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:37.343848+0000", 2024-09-13T11:25:33.792 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.8930000000000001e-05 2024-09-13T11:25:33.792 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.792 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.792 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:25:33.792 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:37.460590+0000", 2024-09-13T11:25:33.792 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.116742045 2024-09-13T11:25:33.792 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.792 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.792 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:25:33.792 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:37.460596+0000", 2024-09-13T11:25:33.792 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.4559999999999997e-06 2024-09-13T11:25:33.792 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.792 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.792 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:25:33.793 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:37.460645+0000", 2024-09-13T11:25:33.793 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.8550000000000001e-05 2024-09-13T11:25:33.793 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.793 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.793 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:25:33.793 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:37.460660+0000", 2024-09-13T11:25:33.793 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.5662e-05 2024-09-13T11:25:33.793 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.793 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.793 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:25:33.793 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:37.462612+0000", 2024-09-13T11:25:33.793 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0019521499999999999 2024-09-13T11:25:33.793 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.793 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.793 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:25:33.793 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:37.462644+0000", 2024-09-13T11:25:33.794 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.1136999999999997e-05 2024-09-13T11:25:33.794 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:33.794 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:25:33.794 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:33.794 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.794 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.794 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop(client.4665.0:119844 3.1a e128/105 3:5858dd7d:::benchmark_data_smithi177_31116_object119843:head v 128'5419, mlcod=128'5416)", 2024-09-13T11:25:33.794 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:24:37.692299+0000", 2024-09-13T11:25:33.794 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 56.050075557, 2024-09-13T11:25:33.794 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.12845056299999999, 2024-09-13T11:25:33.794 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:25:33.794 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:25:33.794 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:33.794 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:25:33.794 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.795 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:25:33.795 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:37.692299+0000", 2024-09-13T11:25:33.795 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:33.795 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.795 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.795 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:25:33.795 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:37.692299+0000", 2024-09-13T11:25:33.795 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:33.795 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.795 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.795 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:25:33.795 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:37.692301+0000", 2024-09-13T11:25:33.795 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.6300000000000001e-06 2024-09-13T11:25:33.795 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.795 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.795 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:25:33.796 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:37.692352+0000", 2024-09-13T11:25:33.796 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.1388000000000003e-05 2024-09-13T11:25:33.796 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.796 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.796 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:25:33.796 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:37.817572+0000", 2024-09-13T11:25:33.796 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.12522014300000001 2024-09-13T11:25:33.796 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.796 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.796 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:25:33.796 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:37.817579+0000", 2024-09-13T11:25:33.796 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.8549999999999996e-06 2024-09-13T11:25:33.796 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.796 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.796 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:25:33.797 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:37.817641+0000", 2024-09-13T11:25:33.797 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.2092000000000001e-05 2024-09-13T11:25:33.797 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.797 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.797 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:25:33.797 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:37.817659+0000", 2024-09-13T11:25:33.797 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.7849000000000001e-05 2024-09-13T11:25:33.797 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.797 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.797 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:25:33.797 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:37.820701+0000", 2024-09-13T11:25:33.797 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0030416459999999998 2024-09-13T11:25:33.797 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.797 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.797 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:25:33.798 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:37.820750+0000", 2024-09-13T11:25:33.798 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.8959999999999999e-05 2024-09-13T11:25:33.798 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:33.798 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:25:33.798 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:33.798 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.798 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.798 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop(client.4665.0:129921 3.1a e134/105 3:5fb9088b:::benchmark_data_smithi177_31116_object129920:head v 134'5758, mlcod=134'5755)", 2024-09-13T11:25:33.798 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:24:47.178021+0000", 2024-09-13T11:25:33.798 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 46.564353476000001, 2024-09-13T11:25:33.798 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.33723278600000001, 2024-09-13T11:25:33.798 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:25:33.798 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:25:33.798 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:33.799 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:25:33.799 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.799 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:25:33.799 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:47.178021+0000", 2024-09-13T11:25:33.799 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:33.799 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.799 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.799 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:25:33.799 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:47.178021+0000", 2024-09-13T11:25:33.799 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:33.799 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.799 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.799 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:25:33.799 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:47.178024+0000", 2024-09-13T11:25:33.799 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.3609999999999999e-06 2024-09-13T11:25:33.800 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.800 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.800 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:25:33.800 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:47.178102+0000", 2024-09-13T11:25:33.800 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.8325000000000006e-05 2024-09-13T11:25:33.800 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.800 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.800 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:25:33.800 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:47.512707+0000", 2024-09-13T11:25:33.800 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.33460516000000001 2024-09-13T11:25:33.800 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.800 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.800 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:25:33.800 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:47.512716+0000", 2024-09-13T11:25:33.801 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.0389999999999993e-06 2024-09-13T11:25:33.801 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.801 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.801 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:25:33.801 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:47.512788+0000", 2024-09-13T11:25:33.801 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.1840000000000003e-05 2024-09-13T11:25:33.801 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.801 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.801 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:25:33.801 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:47.512809+0000", 2024-09-13T11:25:33.801 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.1005e-05 2024-09-13T11:25:33.801 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.801 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.802 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:25:33.802 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:47.515182+0000", 2024-09-13T11:25:33.802 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0023731939999999999 2024-09-13T11:25:33.802 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.802 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.802 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:25:33.802 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:47.515254+0000", 2024-09-13T11:25:33.802 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.1861999999999997e-05 2024-09-13T11:25:33.802 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:33.802 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:25:33.802 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:33.802 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.802 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.802 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop(client.4665.0:138370 3.23 e140/90 3:c5730cd2:::benchmark_data_smithi177_31116_object138369:head v 140'4458, mlcod=140'4455)", 2024-09-13T11:25:33.802 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:24:54.416118+0000", 2024-09-13T11:25:33.802 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 39.326256997999998, 2024-09-13T11:25:33.803 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.244024991, 2024-09-13T11:25:33.803 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:25:33.803 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:25:33.803 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:33.803 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:25:33.803 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.803 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:25:33.803 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:54.416118+0000", 2024-09-13T11:25:33.803 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:33.803 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.803 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.803 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:25:33.803 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:54.416118+0000", 2024-09-13T11:25:33.803 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:33.803 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.804 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.804 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:25:33.804 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:54.416120+0000", 2024-09-13T11:25:33.804 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.6280000000000001e-06 2024-09-13T11:25:33.804 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.804 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.804 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:25:33.804 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:54.416161+0000", 2024-09-13T11:25:33.804 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.0877000000000002e-05 2024-09-13T11:25:33.804 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.804 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.804 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:25:33.804 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:54.657128+0000", 2024-09-13T11:25:33.804 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.24096696300000001 2024-09-13T11:25:33.804 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.805 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.805 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:25:33.805 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:54.657142+0000", 2024-09-13T11:25:33.805 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.358e-05 2024-09-13T11:25:33.805 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.805 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.805 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:25:33.805 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:54.657290+0000", 2024-09-13T11:25:33.805 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00014794400000000001 2024-09-13T11:25:33.805 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.805 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.805 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:25:33.805 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:54.657325+0000", 2024-09-13T11:25:33.805 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.5623000000000001e-05 2024-09-13T11:25:33.806 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.806 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.806 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:25:33.806 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:54.660079+0000", 2024-09-13T11:25:33.806 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0027536489999999999 2024-09-13T11:25:33.806 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.806 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.806 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:25:33.806 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:54.660143+0000", 2024-09-13T11:25:33.806 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.3726999999999995e-05 2024-09-13T11:25:33.806 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:33.806 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:25:33.806 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:33.806 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.806 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.807 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:154584 3.19 3:9c2bc229:::benchmark_data_smithi177_31116_object14281:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e151)", 2024-09-13T11:25:33.807 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:25:09.527283+0000", 2024-09-13T11:25:33.807 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 24.215091494999999, 2024-09-13T11:25:33.807 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.120590691, 2024-09-13T11:25:33.807 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:25:33.807 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:25:33.807 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:33.807 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:25:33.807 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:25:33.807 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:33.807 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 154584 2024-09-13T11:25:33.807 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.807 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:25:33.808 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.808 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:25:33.808 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:09.527283+0000", 2024-09-13T11:25:33.808 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:33.808 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.808 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.808 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:25:33.808 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:09.527283+0000", 2024-09-13T11:25:33.808 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:33.808 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.808 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.808 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:25:33.808 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:09.527286+0000", 2024-09-13T11:25:33.808 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.926e-06 2024-09-13T11:25:33.809 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.809 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.809 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:25:33.809 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:09.527294+0000", 2024-09-13T11:25:33.809 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.9249999999999995e-06 2024-09-13T11:25:33.809 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.809 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.809 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:25:33.809 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:09.527315+0000", 2024-09-13T11:25:33.809 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.0639e-05 2024-09-13T11:25:33.809 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.809 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.809 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:25:33.809 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:09.527326+0000", 2024-09-13T11:25:33.810 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.1321999999999999e-05 2024-09-13T11:25:33.810 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.810 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.810 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:25:33.810 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:09.527406+0000", 2024-09-13T11:25:33.810 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.0184000000000003e-05 2024-09-13T11:25:33.810 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.810 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.810 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:25:33.810 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:09.527571+0000", 2024-09-13T11:25:33.810 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00016465300000000001 2024-09-13T11:25:33.810 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.810 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.810 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1", 2024-09-13T11:25:33.810 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:09.527694+0000", 2024-09-13T11:25:33.811 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00012302400000000001 2024-09-13T11:25:33.811 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.811 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.811 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:25:33.811 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:09.530208+0000", 2024-09-13T11:25:33.811 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0025140050000000001 2024-09-13T11:25:33.811 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.811 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.811 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:25:33.811 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:09.647770+0000", 2024-09-13T11:25:33.811 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.117561969 2024-09-13T11:25:33.811 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.811 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.811 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:25:33.812 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:09.647827+0000", 2024-09-13T11:25:33.812 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.7447000000000001e-05 2024-09-13T11:25:33.812 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.812 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.812 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:25:33.812 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:09.647874+0000", 2024-09-13T11:25:33.812 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.6597000000000001e-05 2024-09-13T11:25:33.812 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:33.812 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:25:33.812 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:33.813 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.813 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.813 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop_reply(client.4665.0:154584 3.19 e151/84 ondisk, result = 0)", 2024-09-13T11:25:33.813 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:25:09.530631+0000", 2024-09-13T11:25:33.813 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 24.211743578, 2024-09-13T11:25:33.813 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.11725072, 2024-09-13T11:25:33.813 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:25:33.813 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:25:33.813 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-09-13T11:25:33.813 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:25:33.813 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.813 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:25:33.813 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:09.530631+0000", 2024-09-13T11:25:33.814 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:33.814 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.814 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.814 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:25:33.814 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:09.530631+0000", 2024-09-13T11:25:33.814 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:33.814 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.814 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.814 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:25:33.814 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:09.530633+0000", 2024-09-13T11:25:33.814 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.8390000000000001e-06 2024-09-13T11:25:33.814 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.814 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.814 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:25:33.815 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:09.530640+0000", 2024-09-13T11:25:33.815 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.5560000000000004e-06 2024-09-13T11:25:33.815 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.815 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.815 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:25:33.815 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:09.647658+0000", 2024-09-13T11:25:33.815 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.117018048 2024-09-13T11:25:33.815 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.815 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.815 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:25:33.815 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:09.647666+0000", 2024-09-13T11:25:33.815 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.4200000000000007e-06 2024-09-13T11:25:33.815 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.815 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.816 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:25:33.816 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:09.647752+0000", 2024-09-13T11:25:33.816 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.6335000000000006e-05 2024-09-13T11:25:33.816 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.816 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.816 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:25:33.816 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:09.647764+0000", 2024-09-13T11:25:33.816 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.1240000000000001e-05 2024-09-13T11:25:33.816 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.816 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.816 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:25:33.816 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:09.647882+0000", 2024-09-13T11:25:33.816 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000118282 2024-09-13T11:25:33.816 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:33.816 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:25:33.817 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:33.817 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.817 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.817 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop(client.4665.0:168782 3.1a e161/105 3:590b82e1:::benchmark_data_smithi177_31116_object28479:head v 161'6914, mlcod=161'6910)", 2024-09-13T11:25:33.817 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:25:20.643397+0000", 2024-09-13T11:25:33.817 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 13.098977745999999, 2024-09-13T11:25:33.817 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.17239671500000001, 2024-09-13T11:25:33.817 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:25:33.817 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:25:33.817 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:33.817 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:25:33.817 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.817 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:25:33.817 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:20.643397+0000", 2024-09-13T11:25:33.817 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:33.818 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.818 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.818 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:25:33.818 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:20.643397+0000", 2024-09-13T11:25:33.818 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:33.818 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.818 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.818 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:25:33.818 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:20.643399+0000", 2024-09-13T11:25:33.818 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.807e-06 2024-09-13T11:25:33.818 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.818 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.818 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:25:33.818 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:20.643410+0000", 2024-09-13T11:25:33.818 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0966e-05 2024-09-13T11:25:33.819 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.819 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.819 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:25:33.819 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:20.811600+0000", 2024-09-13T11:25:33.819 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.16819072400000001 2024-09-13T11:25:33.819 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.819 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.819 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:25:33.819 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:20.811613+0000", 2024-09-13T11:25:33.819 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.2166e-05 2024-09-13T11:25:33.819 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.819 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.819 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:25:33.819 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:20.811710+0000", 2024-09-13T11:25:33.820 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.7501e-05 2024-09-13T11:25:33.820 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.820 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.820 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:25:33.820 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:20.811727+0000", 2024-09-13T11:25:33.820 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.6645000000000001e-05 2024-09-13T11:25:33.820 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.820 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.820 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:25:33.820 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:20.815721+0000", 2024-09-13T11:25:33.820 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0039940260000000003 2024-09-13T11:25:33.820 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.820 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.820 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:25:33.821 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:20.815794+0000", 2024-09-13T11:25:33.821 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.2879999999999993e-05 2024-09-13T11:25:33.821 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:33.821 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:25:33.821 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:33.821 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.821 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.821 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:170854 3.24 3:25558d38:::benchmark_data_smithi177_31116_object30551:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e162)", 2024-09-13T11:25:33.821 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:25:21.815003+0000", 2024-09-13T11:25:33.821 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 11.927371695, 2024-09-13T11:25:33.821 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.12909341099999999, 2024-09-13T11:25:33.821 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:25:33.821 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:25:33.822 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:33.822 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:25:33.822 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:25:33.822 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:33.822 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 170854 2024-09-13T11:25:33.822 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.822 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:25:33.822 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.822 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:25:33.822 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:21.815003+0000", 2024-09-13T11:25:33.822 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:33.822 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.822 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.822 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:25:33.823 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:21.815003+0000", 2024-09-13T11:25:33.823 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:33.823 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.823 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.823 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:25:33.823 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:21.815006+0000", 2024-09-13T11:25:33.823 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.1089999999999998e-06 2024-09-13T11:25:33.823 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.823 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.823 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:25:33.823 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:21.815015+0000", 2024-09-13T11:25:33.823 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.7069999999999998e-06 2024-09-13T11:25:33.824 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.824 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.824 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:25:33.824 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:21.815033+0000", 2024-09-13T11:25:33.824 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.7842999999999998e-05 2024-09-13T11:25:33.824 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.824 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.824 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:25:33.824 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:21.815041+0000", 2024-09-13T11:25:33.824 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.1340000000000004e-06 2024-09-13T11:25:33.824 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.825 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.825 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:25:33.825 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:21.815103+0000", 2024-09-13T11:25:33.825 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.2279000000000005e-05 2024-09-13T11:25:33.825 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.825 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.825 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:25:33.825 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:21.815205+0000", 2024-09-13T11:25:33.825 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00010176800000000001 2024-09-13T11:25:33.825 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.825 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.825 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1", 2024-09-13T11:25:33.825 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:21.815280+0000", 2024-09-13T11:25:33.825 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.5364000000000006e-05 2024-09-13T11:25:33.826 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.826 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.826 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:25:33.826 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:21.816637+0000", 2024-09-13T11:25:33.826 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0013567360000000001 2024-09-13T11:25:33.826 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.826 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.826 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:25:33.826 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:21.943996+0000", 2024-09-13T11:25:33.826 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.12735875299999999 2024-09-13T11:25:33.826 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.826 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.827 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:25:33.827 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:21.944048+0000", 2024-09-13T11:25:33.827 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.2142000000000003e-05 2024-09-13T11:25:33.827 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.827 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.827 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:25:33.827 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:21.944096+0000", 2024-09-13T11:25:33.827 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.8575999999999997e-05 2024-09-13T11:25:33.827 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:33.827 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:25:33.827 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:33.827 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.827 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.827 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop(client.4665.0:174971 3.1a e166/105 3:5a8f64c5:::benchmark_data_smithi177_31116_object34668:head v 166'7101, mlcod=166'7098)", 2024-09-13T11:25:33.828 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:25:25.423119+0000", 2024-09-13T11:25:33.828 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 8.3192555590000001, 2024-09-13T11:25:33.828 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.18331898799999999, 2024-09-13T11:25:33.828 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:25:33.828 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:25:33.828 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:33.828 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:25:33.828 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.828 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:25:33.828 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:25.423119+0000", 2024-09-13T11:25:33.828 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:33.828 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.828 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.828 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:25:33.829 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:25.423119+0000", 2024-09-13T11:25:33.829 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:33.829 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.829 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.829 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:25:33.829 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:25.423120+0000", 2024-09-13T11:25:33.829 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.184e-06 2024-09-13T11:25:33.829 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.829 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.829 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:25:33.829 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:25.423127+0000", 2024-09-13T11:25:33.829 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.7139999999999996e-06 2024-09-13T11:25:33.829 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.829 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.830 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:25:33.830 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:25.604412+0000", 2024-09-13T11:25:33.830 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.18128530200000001 2024-09-13T11:25:33.830 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.830 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.830 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:25:33.830 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:25.604418+0000", 2024-09-13T11:25:33.830 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.6099999999999997e-06 2024-09-13T11:25:33.830 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.830 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.830 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:25:33.830 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:25.604467+0000", 2024-09-13T11:25:33.830 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.8729000000000001e-05 2024-09-13T11:25:33.830 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.830 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.831 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:25:33.831 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:25.604482+0000", 2024-09-13T11:25:33.831 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.5182e-05 2024-09-13T11:25:33.831 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.831 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.831 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:25:33.831 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:25.606402+0000", 2024-09-13T11:25:33.831 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0019203530000000001 2024-09-13T11:25:33.831 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:33.831 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:33.831 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:25:33.831 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:25.606438+0000", 2024-09-13T11:25:33.831 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.5914000000000002e-05 2024-09-13T11:25:33.831 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:33.832 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:25:33.832 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:33.832 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:33.832 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:25:33.832 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-09-13T11:25:33.833 INFO:teuthology.orchestra.run.smithi001.stdout:osd.5: {} 2024-09-13T11:25:33.833 INFO:teuthology.orchestra.run.smithi001.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T11:25:33.834 INFO:teuthology.orchestra.run.smithi001.stdout:osd.6: {} 2024-09-13T11:25:33.834 INFO:teuthology.orchestra.run.smithi001.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T11:25:33.834 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_ops_in_flight 2024-09-13T11:25:33.835 INFO:teuthology.orchestra.run.smithi001.stdout:osd.7: {} 2024-09-13T11:25:33.835 INFO:teuthology.orchestra.run.smithi001.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T11:25:33.836 INFO:tasks.ceph.osd.11.smithi177.stderr:2024-09-13T11:25:33.830+0000 7f24792c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26580) UID: 0 2024-09-13T11:25:33.842 INFO:teuthology.orchestra.run.smithi001.stdout:osd.8: {} 2024-09-13T11:25:33.842 INFO:teuthology.orchestra.run.smithi001.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-13T11:25:33.870 INFO:teuthology.orchestra.run.smithi001.stdout:osd.9: {} 2024-09-13T11:25:33.871 INFO:teuthology.orchestra.run.smithi001.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-13T11:25:33.900 INFO:teuthology.orchestra.run.smithi001.stdout:osd.10: {} 2024-09-13T11:25:33.900 INFO:teuthology.orchestra.run.smithi001.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-13T11:25:33.933 INFO:teuthology.orchestra.run.smithi001.stdout:osd.11: {} 2024-09-13T11:25:33.933 INFO:teuthology.orchestra.run.smithi001.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-13T11:25:33.934 INFO:tasks.ceph.osd.14.smithi162.stderr:2024-09-13T11:25:33.932+0000 7f3df5903640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33154) UID: 0 2024-09-13T11:25:33.965 INFO:teuthology.orchestra.run.smithi001.stdout:osd.12: {} 2024-09-13T11:25:33.966 INFO:teuthology.orchestra.run.smithi001.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-13T11:25:33.970 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-09-13T11:25:33.970 INFO:teuthology.orchestra.run.smithi092.stdout: "ops": [], 2024-09-13T11:25:33.970 INFO:teuthology.orchestra.run.smithi092.stdout: "num_ops": 0 2024-09-13T11:25:33.970 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-09-13T11:25:33.986 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_blocked_ops 2024-09-13T11:25:33.993 INFO:teuthology.orchestra.run.smithi001.stdout:osd.13: {} 2024-09-13T11:25:33.993 INFO:teuthology.orchestra.run.smithi001.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-13T11:25:34.021 INFO:teuthology.orchestra.run.smithi001.stdout:osd.14: {} 2024-09-13T11:25:34.022 INFO:teuthology.orchestra.run.smithi001.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-13T11:25:34.035 INFO:tasks.ceph.osd.4.smithi001.stderr:2024-09-13T11:25:34.031+0000 7ff8e0218640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26310) UID: 0 2024-09-13T11:25:34.056 INFO:teuthology.orchestra.run.smithi001.stdout:osd.15: {} 2024-09-13T11:25:34.056 INFO:teuthology.orchestra.run.smithi001.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-13T11:25:34.114 INFO:teuthology.orchestra.run.smithi092.stdout:ERROR: (22) Invalid argument 2024-09-13T11:25:34.114 INFO:teuthology.orchestra.run.smithi092.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:25:34.131 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_historic_ops 2024-09-13T11:25:34.136 INFO:tasks.ceph.osd.12.smithi001.stderr:2024-09-13T11:25:34.135+0000 7f892c6e6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26314) UID: 0 2024-09-13T11:25:34.237 INFO:tasks.ceph.osd.13.smithi092.stderr:2024-09-13T11:25:34.237+0000 7f3e5afea640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26350) UID: 0 2024-09-13T11:25:34.260 INFO:teuthology.orchestra.run.smithi092.stdout:ERROR: (22) Invalid argument 2024-09-13T11:25:34.260 INFO:teuthology.orchestra.run.smithi092.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:25:34.267 INFO:tasks.ceph.osd.8.smithi001.stderr:2024-09-13T11:25:34.267+0000 7fb9ab2df640 -1 osd.8 172 handle_osd_map injecting map crc failure 2024-09-13T11:25:34.276 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-13T11:25:34.280 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-13T11:25:34.338 INFO:tasks.ceph.osd.8.smithi001.stderr:2024-09-13T11:25:34.335+0000 7fb9b48ae640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26312) UID: 0 2024-09-13T11:25:34.428 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-13T11:25:34.428 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:25:34.438 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-09-13T11:25:34.436+0000 7f97befc9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26315) UID: 0 2024-09-13T11:25:34.444 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-13T11:25:34.540 INFO:tasks.ceph.osd.2.smithi162.stderr:2024-09-13T11:25:34.536+0000 7fe92b35a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26313) UID: 0 2024-09-13T11:25:34.623 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-13T11:25:34.623 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:25:34.640 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-13T11:25:34.641 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-13T11:25:34.639+0000 7f6a68775640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26307) UID: 0 2024-09-13T11:25:34.746 INFO:tasks.ceph.osd.5.smithi092.stderr:2024-09-13T11:25:34.745+0000 7f6f0b705640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26346) UID: 0 2024-09-13T11:25:34.820 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-13T11:25:34.820 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:25:34.835 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-13T11:25:34.847 INFO:tasks.ceph.osd.7.smithi177.stderr:2024-09-13T11:25:34.846+0000 7f897b51d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26575) UID: 0 2024-09-13T11:25:34.948 INFO:tasks.ceph.osd.11.smithi177.stderr:2024-09-13T11:25:34.946+0000 7f24792c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26580) UID: 0 2024-09-13T11:25:34.967 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-13T11:25:34.968 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:25:34.983 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-13T11:25:35.049 INFO:tasks.ceph.osd.14.smithi162.stderr:2024-09-13T11:25:35.048+0000 7f3df5903640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33154) UID: 0 2024-09-13T11:25:35.104 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-13T11:25:35.105 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-13T11:25:35.105 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:25:35.119 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-13T11:25:35.150 INFO:tasks.ceph.osd.15.smithi177.stderr:2024-09-13T11:25:35.146+0000 7f9d6ce77640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26578) UID: 0 2024-09-13T11:25:35.252 INFO:tasks.ceph.osd.1.smithi092.stderr:2024-09-13T11:25:35.249+0000 7fe7789c5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33718) UID: 0 2024-09-13T11:25:35.277 INFO:teuthology.orchestra.run.smithi001.stderr:nodeep-scrub is unset 2024-09-13T11:25:35.284 INFO:tasks.ceph.osd.5.smithi092.stderr:2024-09-13T11:25:35.281+0000 7f6f00132640 -1 osd.5 173 handle_osd_map injecting map crc failure 2024-09-13T11:25:35.284 INFO:tasks.ceph.osd.9.smithi092.stderr:2024-09-13T11:25:35.281+0000 7ff59bdf5640 -1 osd.9 173 handle_osd_map injecting map crc failure 2024-09-13T11:25:35.291 INFO:tasks.ceph.osd.2.smithi162.stderr:2024-09-13T11:25:35.288+0000 7fe921d8b640 -1 osd.2 173 handle_osd_map injecting map crc failure 2024-09-13T11:25:35.298 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-13T11:25:35.299 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:25:35.305 INFO:tasks.ceph.osd.10.smithi162.stderr:2024-09-13T11:25:35.300+0000 7fb593421640 -1 osd.10 173 handle_osd_map injecting map crc failure 2024-09-13T11:25:35.315 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-09-13T11:25:35.352 INFO:tasks.ceph.osd.7.smithi177.stderr:2024-09-13T11:25:35.350+0000 7f897b51d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26575) UID: 0 2024-09-13T11:25:35.446 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-13T11:25:35.446 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:25:35.452 INFO:tasks.ceph.osd.4.smithi001.stderr:2024-09-13T11:25:35.451+0000 7ff8e0218640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26310) UID: 0 2024-09-13T11:25:35.463 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-09-13T11:25:35.489 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-13T11:25:35.490 INFO:teuthology.orchestra.run.smithi001.stderr:dumped all 2024-09-13T11:25:35.516 INFO:teuthology.orchestra.run.smithi001.stdout:{"pg_ready":true,"pg_map":{"version":262,"stamp":"2024-09-13T11:25:35.283501+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":6450807371,"num_objects":98428,"num_object_clones":0,"num_object_copies":196856,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":754,"num_objects_unfound":0,"num_objects_dirty":98428,"num_whiteouts":0,"num_read":87,"num_read_kb":74,"num_write":98419,"num_write_kb":3637107,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21839,"num_bytes_recovered":1431109651,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":291771,"ondisk_log_size":291771,"up":110,"acting":110,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":120,"num_osds":15,"num_per_pool_osds":15,"num_per_pool_omap_osds":15,"kb":1415577600,"kb_used":14716428,"kb_used_data":13314908,"kb_used_omap":26,"kb_used_meta":1401189,"kb_avail":1400861172,"statfs":{"total":1449551462400,"available":1434481840128,"internally_reserved":0,"allocated":13634465792,"data_stored":13623341883,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":26763,"internal_metadata":1434818421},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,1,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-374013952,"num_objects":-5707,"num_object_clones":0,"num_object_copies":-11414,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1642,"num_objects_unfound":0,"num_objects_dirty":-5707,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5707,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":821,"num_bytes_recovered":53805056,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.035619"},"pg_stats":[{"pgid":"3.2d","version":"173'6306","reported_seq":11475,"reported_epoch":173,"state":"active+recovering+remapped","last_fresh":"2024-09-13T11:25:35.084123+0000","last_change":"2024-09-13T11:25:19.656321+0000","last_active":"2024-09-13T11:25:35.084123+0000","last_peered":"2024-09-13T11:25:35.084123+0000","last_clean":"2024-09-13T11:24:24.974950+0000","last_became_active":"2024-09-13T11:24:27.227288+0000","last_became_peered":"2024-09-13T11:24:27.227288+0000","last_unstale":"2024-09-13T11:25:35.084123+0000","last_undegraded":"2024-09-13T11:25:35.084123+0000","last_fullsized":"2024-09-13T11:25:35.084123+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":6,"last_scrub":"76'3395","last_scrub_stamp":"2024-09-13T11:23:32.776747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:32.776747+0000","objects_scrubbed":1704,"log_size":6306,"log_dups_size":0,"ondisk_log_size":6306,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":179,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":100171776,"num_objects":1529,"num_object_clones":0,"num_object_copies":3058,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":220,"num_objects_unfound":0,"num_objects_dirty":1529,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2069,"num_write_kb":87840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1609,"num_bytes_recovered":105447424,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[3,0],"avail_no_missing":["3","7"],"object_location_counts":[{"shards":"0,3","objects":1419},{"shards":"3,7","objects":110}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2c","version":"170'6316","reported_seq":11656,"reported_epoch":170,"state":"active+clean","last_fresh":"2024-09-13T11:25:31.028026+0000","last_change":"2024-09-13T11:24:59.456762+0000","last_active":"2024-09-13T11:25:31.028026+0000","last_peered":"2024-09-13T11:25:31.028026+0000","last_clean":"2024-09-13T11:25:31.028026+0000","last_became_active":"2024-09-13T11:24:59.021895+0000","last_became_peered":"2024-09-13T11:24:59.021895+0000","last_unstale":"2024-09-13T11:25:31.028026+0000","last_undegraded":"2024-09-13T11:25:31.028026+0000","last_fullsized":"2024-09-13T11:25:31.028026+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":6,"last_scrub":"145'5672","last_scrub_stamp":"2024-09-13T11:24:59.456653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:24:59.456653+0000","objects_scrubbed":2192,"log_size":6316,"log_dups_size":0,"ondisk_log_size":6316,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:26:20.877991+0000","scrub_duration":279,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":101449728,"num_objects":1548,"num_object_clones":0,"num_object_copies":3096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":644,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,7],"acting":[10,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.2b","version":"169'6189","reported_seq":9886,"reported_epoch":170,"state":"active+clean","last_fresh":"2024-09-13T11:25:30.250841+0000","last_change":"2024-09-13T11:24:45.271477+0000","last_active":"2024-09-13T11:25:30.250841+0000","last_peered":"2024-09-13T11:25:30.250841+0000","last_clean":"2024-09-13T11:25:30.250841+0000","last_became_active":"2024-09-13T11:24:26.180578+0000","last_became_peered":"2024-09-13T11:24:26.180578+0000","last_unstale":"2024-09-13T11:25:30.250841+0000","last_undegraded":"2024-09-13T11:25:30.250841+0000","last_fullsized":"2024-09-13T11:25:30.250841+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":6,"last_scrub":"131'5382","last_scrub_stamp":"2024-09-13T11:24:45.271376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:24:45.271376+0000","objects_scrubbed":1940,"log_size":6189,"log_dups_size":0,"ondisk_log_size":6189,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:59.210895+0000","scrub_duration":279,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96010240,"num_objects":1465,"num_object_clones":0,"num_object_copies":2930,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1465,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":807,"num_write_kb":10624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2a","version":"170'6324","reported_seq":10208,"reported_epoch":170,"state":"active+clean","last_fresh":"2024-09-13T11:25:31.058262+0000","last_change":"2024-09-13T11:24:46.625117+0000","last_active":"2024-09-13T11:25:31.058262+0000","last_peered":"2024-09-13T11:25:31.058262+0000","last_clean":"2024-09-13T11:25:31.058262+0000","last_became_active":"2024-09-13T11:24:45.320540+0000","last_became_peered":"2024-09-13T11:24:45.320540+0000","last_unstale":"2024-09-13T11:25:31.058262+0000","last_undegraded":"2024-09-13T11:25:31.058262+0000","last_fullsized":"2024-09-13T11:25:31.058262+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":6,"last_scrub":"133'5491","last_scrub_stamp":"2024-09-13T11:24:46.624970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:24:46.624970+0000","objects_scrubbed":2094,"log_size":6324,"log_dups_size":0,"ondisk_log_size":6324,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:26:04.813280+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":105447424,"num_objects":1609,"num_object_clones":0,"num_object_copies":3218,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1609,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":835,"num_write_kb":11200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,7],"acting":[10,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.29","version":"170'5973","reported_seq":11672,"reported_epoch":170,"state":"active+clean","last_fresh":"2024-09-13T11:25:30.906236+0000","last_change":"2024-09-13T11:25:13.333062+0000","last_active":"2024-09-13T11:25:30.906236+0000","last_peered":"2024-09-13T11:25:30.906236+0000","last_clean":"2024-09-13T11:25:30.906236+0000","last_became_active":"2024-09-13T11:25:12.944028+0000","last_became_peered":"2024-09-13T11:25:12.944028+0000","last_unstale":"2024-09-13T11:25:30.906236+0000","last_undegraded":"2024-09-13T11:25:30.906236+0000","last_fullsized":"2024-09-13T11:25:30.906236+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":6,"last_scrub":"154'5579","last_scrub_stamp":"2024-09-13T11:25:13.332959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:13.332959+0000","objects_scrubbed":1937,"log_size":5973,"log_dups_size":0,"ondisk_log_size":5973,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:26:24.089228+0000","scrub_duration":243,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":100990976,"num_objects":1541,"num_object_clones":0,"num_object_copies":3082,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1541,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":396,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.28","version":"169'5802","reported_seq":10396,"reported_epoch":170,"state":"active+recovering+remapped","last_fresh":"2024-09-13T11:25:30.257977+0000","last_change":"2024-09-13T11:25:11.182075+0000","last_active":"2024-09-13T11:25:30.257977+0000","last_peered":"2024-09-13T11:25:30.257977+0000","last_clean":"2024-09-13T11:24:25.005266+0000","last_became_active":"2024-09-13T11:24:27.218508+0000","last_became_peered":"2024-09-13T11:24:27.218508+0000","last_unstale":"2024-09-13T11:25:30.257977+0000","last_undegraded":"2024-09-13T11:25:30.257977+0000","last_fullsized":"2024-09-13T11:25:30.257977+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":6,"last_scrub":"68'2771","last_scrub_stamp":"2024-09-13T11:23:20.644747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:20.644747+0000","objects_scrubbed":1356,"log_size":5802,"log_dups_size":0,"ondisk_log_size":5802,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":411,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":103088128,"num_objects":1573,"num_object_clones":0,"num_object_copies":3146,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":534,"num_objects_unfound":0,"num_objects_dirty":1573,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2055,"num_write_kb":94400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1419,"num_bytes_recovered":92995584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5],"acting":[1,5],"avail_no_missing":["1","6"],"object_location_counts":[{"shards":"1,5","objects":1306},{"shards":"1,6","objects":267}],"blocked_by":[],"up_primary":8,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.27","version":"170'5897","reported_seq":11417,"reported_epoch":170,"state":"active+clean","last_fresh":"2024-09-13T11:25:30.837477+0000","last_change":"2024-09-13T11:25:30.270960+0000","last_active":"2024-09-13T11:25:30.837477+0000","last_peered":"2024-09-13T11:25:30.837477+0000","last_clean":"2024-09-13T11:25:30.837477+0000","last_became_active":"2024-09-13T11:25:30.257911+0000","last_became_peered":"2024-09-13T11:25:30.257911+0000","last_unstale":"2024-09-13T11:25:30.837477+0000","last_undegraded":"2024-09-13T11:25:30.837477+0000","last_fullsized":"2024-09-13T11:25:30.837477+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":6,"last_scrub":"68'2736","last_scrub_stamp":"2024-09-13T11:23:20.443329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:20.443329+0000","objects_scrubbed":99,"log_size":5897,"log_dups_size":0,"ondisk_log_size":5897,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":183,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":103350272,"num_objects":1577,"num_object_clones":0,"num_object_copies":3154,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1577,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2175,"num_write_kb":98144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1725,"num_bytes_recovered":113049600,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,11],"acting":[5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.26","version":"170'6024","reported_seq":11603,"reported_epoch":170,"state":"active+clean","last_fresh":"2024-09-13T11:25:31.029851+0000","last_change":"2024-09-13T11:25:01.697082+0000","last_active":"2024-09-13T11:25:31.029851+0000","last_peered":"2024-09-13T11:25:31.029851+0000","last_clean":"2024-09-13T11:25:31.029851+0000","last_became_active":"2024-09-13T11:24:54.976919+0000","last_became_peered":"2024-09-13T11:24:54.976919+0000","last_unstale":"2024-09-13T11:25:31.029851+0000","last_undegraded":"2024-09-13T11:25:31.029851+0000","last_fullsized":"2024-09-13T11:25:31.029851+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":6,"last_scrub":"147'5433","last_scrub_stamp":"2024-09-13T11:25:01.696886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:01.696886+0000","objects_scrubbed":2158,"log_size":6024,"log_dups_size":0,"ondisk_log_size":6024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:26:22.070621+0000","scrub_duration":403,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":102170624,"num_objects":1559,"num_object_clones":0,"num_object_copies":3118,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1559,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":599,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,12],"acting":[8,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.25","version":"170'5865","reported_seq":11603,"reported_epoch":170,"state":"active+clean","last_fresh":"2024-09-13T11:25:31.073510+0000","last_change":"2024-09-13T11:24:57.073162+0000","last_active":"2024-09-13T11:25:31.073510+0000","last_peered":"2024-09-13T11:25:31.073510+0000","last_clean":"2024-09-13T11:25:31.073510+0000","last_became_active":"2024-09-13T11:24:55.999019+0000","last_became_peered":"2024-09-13T11:24:55.999019+0000","last_unstale":"2024-09-13T11:25:31.073510+0000","last_undegraded":"2024-09-13T11:25:31.073510+0000","last_fullsized":"2024-09-13T11:25:31.073510+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":6,"last_scrub":"142'5214","last_scrub_stamp":"2024-09-13T11:24:57.073061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:24:57.073061+0000","objects_scrubbed":2249,"log_size":5865,"log_dups_size":0,"ondisk_log_size":5865,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:26:12.348519+0000","scrub_duration":571,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":104923136,"num_objects":1601,"num_object_clones":0,"num_object_copies":3202,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1601,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":654,"num_write_kb":192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,2],"acting":[12,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[]},{"pgid":"3.24","version":"170'5920","reported_seq":9844,"reported_epoch":170,"state":"active+clean","last_fresh":"2024-09-13T11:25:31.989043+0000","last_change":"2024-09-13T11:24:48.048846+0000","last_active":"2024-09-13T11:25:31.989043+0000","last_peered":"2024-09-13T11:25:31.989043+0000","last_clean":"2024-09-13T11:25:31.989043+0000","last_became_active":"2024-09-13T11:24:46.936994+0000","last_became_peered":"2024-09-13T11:24:46.936994+0000","last_unstale":"2024-09-13T11:25:31.989043+0000","last_undegraded":"2024-09-13T11:25:31.989043+0000","last_fullsized":"2024-09-13T11:25:31.989043+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":6,"last_scrub":"135'5093","last_scrub_stamp":"2024-09-13T11:24:48.048764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:24:48.048764+0000","objects_scrubbed":2102,"log_size":5920,"log_dups_size":0,"ondisk_log_size":5920,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:53.485370+0000","scrub_duration":287,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":102498304,"num_objects":1564,"num_object_clones":0,"num_object_copies":3128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1564,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":828,"num_write_kb":9280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1],"acting":[9,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.23","version":"170'5149","reported_seq":8008,"reported_epoch":170,"state":"active+clean","last_fresh":"2024-09-13T11:25:30.931382+0000","last_change":"2024-09-13T11:23:48.133184+0000","last_active":"2024-09-13T11:25:30.931382+0000","last_peered":"2024-09-13T11:25:30.931382+0000","last_clean":"2024-09-13T11:25:30.931382+0000","last_became_active":"2024-09-13T11:23:47.926677+0000","last_became_peered":"2024-09-13T11:23:47.926677+0000","last_unstale":"2024-09-13T11:25:30.931382+0000","last_undegraded":"2024-09-13T11:25:30.931382+0000","last_fullsized":"2024-09-13T11:25:30.931382+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":6,"last_scrub":"91'3278","last_scrub_stamp":"2024-09-13T11:23:48.133084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:48.133084+0000","objects_scrubbed":1029,"log_size":5149,"log_dups_size":0,"ondisk_log_size":5149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":139,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":104660992,"num_objects":1597,"num_object_clones":0,"num_object_copies":3194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1597,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1872,"num_write_kb":78080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9],"acting":[5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.22","version":"170'5058","reported_seq":10647,"reported_epoch":170,"state":"active+clean","last_fresh":"2024-09-13T11:25:31.095934+0000","last_change":"2024-09-13T11:24:45.485711+0000","last_active":"2024-09-13T11:25:31.095934+0000","last_peered":"2024-09-13T11:25:31.095934+0000","last_clean":"2024-09-13T11:25:31.095934+0000","last_became_active":"2024-09-13T11:24:29.253848+0000","last_became_peered":"2024-09-13T11:24:29.253848+0000","last_unstale":"2024-09-13T11:25:31.095934+0000","last_undegraded":"2024-09-13T11:25:31.095934+0000","last_fullsized":"2024-09-13T11:25:31.095934+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":6,"last_scrub":"90'3246","last_scrub_stamp":"2024-09-13T11:23:47.300339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:47.300339+0000","objects_scrubbed":1033,"log_size":5058,"log_dups_size":0,"ondisk_log_size":5058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":203,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":103874560,"num_objects":1585,"num_object_clones":0,"num_object_copies":3170,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1585,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1814,"num_write_kb":75712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1768,"num_bytes_recovered":115867648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.21","version":"170'4975","reported_seq":7849,"reported_epoch":170,"state":"active+clean","last_fresh":"2024-09-13T11:25:31.076527+0000","last_change":"2024-09-13T11:24:12.564117+0000","last_active":"2024-09-13T11:25:31.076527+0000","last_peered":"2024-09-13T11:25:31.076527+0000","last_clean":"2024-09-13T11:25:31.076527+0000","last_became_active":"2024-09-13T11:24:01.250308+0000","last_became_peered":"2024-09-13T11:24:01.250308+0000","last_unstale":"2024-09-13T11:25:31.076527+0000","last_undegraded":"2024-09-13T11:25:31.076527+0000","last_fullsized":"2024-09-13T11:25:31.076527+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":6,"last_scrub":"112'3670","last_scrub_stamp":"2024-09-13T11:24:12.564031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:24:12.564031+0000","objects_scrubbed":1431,"log_size":4975,"log_dups_size":0,"ondisk_log_size":4975,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:36.766299+0000","scrub_duration":183,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":98631680,"num_objects":1505,"num_object_clones":0,"num_object_copies":3010,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1505,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1308,"num_write_kb":44224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8],"acting":[0,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.20","version":"170'5120","reported_seq":7281,"reported_epoch":170,"state":"active+clean","last_fresh":"2024-09-13T11:25:30.882336+0000","last_change":"2024-09-13T11:23:47.969503+0000","last_active":"2024-09-13T11:25:30.882336+0000","last_peered":"2024-09-13T11:25:30.882336+0000","last_clean":"2024-09-13T11:25:30.882336+0000","last_became_active":"2024-09-13T11:23:33.954946+0000","last_became_peered":"2024-09-13T11:23:33.954946+0000","last_unstale":"2024-09-13T11:25:30.882336+0000","last_undegraded":"2024-09-13T11:25:30.882336+0000","last_fullsized":"2024-09-13T11:25:30.882336+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":6,"last_scrub":"91'3270","last_scrub_stamp":"2024-09-13T11:23:47.969372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:47.969372+0000","objects_scrubbed":1061,"log_size":5120,"log_dups_size":0,"ondisk_log_size":5120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:33.468204+0000","scrub_duration":1327,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":101056512,"num_objects":1542,"num_object_clones":0,"num_object_copies":3084,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1542,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1855,"num_write_kb":74752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,0],"acting":[12,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[]},{"pgid":"3.1f","version":"170'7278","reported_seq":19237,"reported_epoch":170,"state":"active+clean","last_fresh":"2024-09-13T11:25:31.028780+0000","last_change":"2024-09-13T11:25:20.773752+0000","last_active":"2024-09-13T11:25:31.028780+0000","last_peered":"2024-09-13T11:25:31.028780+0000","last_clean":"2024-09-13T11:25:31.028780+0000","last_became_active":"2024-09-13T11:24:29.290326+0000","last_became_peered":"2024-09-13T11:24:29.290326+0000","last_unstale":"2024-09-13T11:25:31.028780+0000","last_undegraded":"2024-09-13T11:25:31.028780+0000","last_fullsized":"2024-09-13T11:25:31.028780+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":5,"last_scrub":"161'6865","last_scrub_stamp":"2024-09-13T11:25:20.773641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:20.773641+0000","objects_scrubbed":3425,"log_size":7278,"log_dups_size":0,"ondisk_log_size":7278,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:26:21.317280+0000","scrub_duration":535,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":197066752,"num_objects":3007,"num_object_clones":0,"num_object_copies":6014,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3007,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2858,"num_write_kb":102848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3306,"num_bytes_recovered":216662016,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5],"acting":[10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1e","version":"170'7490","reported_seq":18014,"reported_epoch":170,"state":"active+clean","last_fresh":"2024-09-13T11:25:31.022824+0000","last_change":"2024-09-13T11:24:08.338485+0000","last_active":"2024-09-13T11:25:31.022824+0000","last_peered":"2024-09-13T11:25:31.022824+0000","last_clean":"2024-09-13T11:25:31.022824+0000","last_became_active":"2024-09-13T11:24:01.261448+0000","last_became_peered":"2024-09-13T11:24:01.261448+0000","last_unstale":"2024-09-13T11:25:31.022824+0000","last_undegraded":"2024-09-13T11:25:31.022824+0000","last_fullsized":"2024-09-13T11:25:31.022824+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":5,"last_scrub":"110'4537","last_scrub_stamp":"2024-09-13T11:24:08.338354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:24:08.338354+0000","objects_scrubbed":2760,"log_size":7490,"log_dups_size":0,"ondisk_log_size":7490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":367,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":204341248,"num_objects":3118,"num_object_clones":0,"num_object_copies":6236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3118,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2962,"num_write_kb":106240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"170'7300","reported_seq":12754,"reported_epoch":170,"state":"active+clean","last_fresh":"2024-09-13T11:25:31.032564+0000","last_change":"2024-09-13T11:24:12.492592+0000","last_active":"2024-09-13T11:25:31.032564+0000","last_peered":"2024-09-13T11:25:31.032564+0000","last_clean":"2024-09-13T11:25:31.032564+0000","last_became_active":"2024-09-13T11:24:01.258225+0000","last_became_peered":"2024-09-13T11:24:01.258225+0000","last_unstale":"2024-09-13T11:25:31.032564+0000","last_undegraded":"2024-09-13T11:25:31.032564+0000","last_fullsized":"2024-09-13T11:25:31.032564+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":5,"last_scrub":"112'4579","last_scrub_stamp":"2024-09-13T11:24:12.492507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:24:12.492507+0000","objects_scrubbed":2875,"log_size":7300,"log_dups_size":0,"ondisk_log_size":7300,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:34.926575+0000","scrub_duration":375,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":203030552,"num_objects":3099,"num_object_clones":0,"num_object_copies":6198,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3099,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2734,"num_write_kb":94593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,15],"acting":[2,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1c","version":"170'7480","reported_seq":11291,"reported_epoch":170,"state":"active+clean","last_fresh":"2024-09-13T11:25:32.078405+0000","last_change":"2024-09-13T11:24:00.529312+0000","last_active":"2024-09-13T11:25:32.078405+0000","last_peered":"2024-09-13T11:25:32.078405+0000","last_clean":"2024-09-13T11:25:32.078405+0000","last_became_active":"2024-09-13T11:23:33.989492+0000","last_became_peered":"2024-09-13T11:23:33.989492+0000","last_unstale":"2024-09-13T11:25:32.078405+0000","last_undegraded":"2024-09-13T11:25:32.078405+0000","last_fullsized":"2024-09-13T11:25:32.078405+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":5,"last_scrub":"102'4277","last_scrub_stamp":"2024-09-13T11:24:00.529243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:24:00.529243+0000","objects_scrubbed":2544,"log_size":7480,"log_dups_size":0,"ondisk_log_size":7480,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:37.048397+0000","scrub_duration":259,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":201785344,"num_objects":3079,"num_object_clones":0,"num_object_copies":6158,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3079,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3205,"num_write_kb":119680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7],"acting":[11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1b","version":"170'7432","reported_seq":13065,"reported_epoch":170,"state":"active+clean","last_fresh":"2024-09-13T11:25:31.057509+0000","last_change":"2024-09-13T11:24:12.392093+0000","last_active":"2024-09-13T11:25:31.057509+0000","last_peered":"2024-09-13T11:25:31.057509+0000","last_clean":"2024-09-13T11:25:31.057509+0000","last_became_active":"2024-09-13T11:24:04.950821+0000","last_became_peered":"2024-09-13T11:24:04.950821+0000","last_unstale":"2024-09-13T11:25:31.057509+0000","last_undegraded":"2024-09-13T11:25:31.057509+0000","last_fullsized":"2024-09-13T11:25:31.057509+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":5,"last_scrub":"112'4665","last_scrub_stamp":"2024-09-13T11:24:12.391890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:24:12.391890+0000","objects_scrubbed":2934,"log_size":7432,"log_dups_size":0,"ondisk_log_size":7432,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:41.846147+0000","scrub_duration":675,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":203423744,"num_objects":3104,"num_object_clones":0,"num_object_copies":6208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3104,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2788,"num_write_kb":94656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,15],"acting":[12,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[]},{"pgid":"3.1a","version":"170'7314","reported_seq":12608,"reported_epoch":170,"state":"active+clean","last_fresh":"2024-09-13T11:25:31.026995+0000","last_change":"2024-09-13T11:24:09.556174+0000","last_active":"2024-09-13T11:25:31.026995+0000","last_peered":"2024-09-13T11:25:31.026995+0000","last_clean":"2024-09-13T11:25:31.026995+0000","last_became_active":"2024-09-13T11:24:03.935185+0000","last_became_peered":"2024-09-13T11:24:03.935185+0000","last_unstale":"2024-09-13T11:25:31.026995+0000","last_undegraded":"2024-09-13T11:25:31.026995+0000","last_fullsized":"2024-09-13T11:25:31.026995+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":5,"last_scrub":"110'4433","last_scrub_stamp":"2024-09-13T11:24:09.556022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:24:09.556022+0000","objects_scrubbed":2654,"log_size":7314,"log_dups_size":0,"ondisk_log_size":7314,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":387,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":198311936,"num_objects":3026,"num_object_clones":0,"num_object_copies":6052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3026,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2892,"num_write_kb":104448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9],"acting":[4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.19","version":"170'6950","reported_seq":11431,"reported_epoch":170,"state":"active+clean","last_fresh":"2024-09-13T11:25:31.961853+0000","last_change":"2024-09-13T11:25:10.784789+0000","last_active":"2024-09-13T11:25:31.961853+0000","last_peered":"2024-09-13T11:25:31.961853+0000","last_clean":"2024-09-13T11:25:31.961853+0000","last_became_active":"2024-09-13T11:23:41.924941+0000","last_became_peered":"2024-09-13T11:23:41.924941+0000","last_unstale":"2024-09-13T11:25:31.961853+0000","last_undegraded":"2024-09-13T11:25:31.961853+0000","last_fullsized":"2024-09-13T11:25:31.961853+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":5,"last_scrub":"151'6156","last_scrub_stamp":"2024-09-13T11:25:10.784716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:10.784716+0000","objects_scrubbed":3781,"log_size":6950,"log_dups_size":0,"ondisk_log_size":6950,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:26:30.390310+0000","scrub_duration":1131,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":194576384,"num_objects":2969,"num_object_clones":0,"num_object_copies":5938,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2969,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3548,"num_write_kb":145216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1],"acting":[9,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.18","version":"170'7021","reported_seq":11764,"reported_epoch":170,"state":"active+clean","last_fresh":"2024-09-13T11:25:31.087593+0000","last_change":"2024-09-13T11:24:13.454659+0000","last_active":"2024-09-13T11:25:31.087593+0000","last_peered":"2024-09-13T11:25:31.087593+0000","last_clean":"2024-09-13T11:25:31.087593+0000","last_became_active":"2024-09-13T11:23:48.926014+0000","last_became_peered":"2024-09-13T11:23:48.926014+0000","last_unstale":"2024-09-13T11:25:31.087593+0000","last_undegraded":"2024-09-13T11:25:31.087593+0000","last_fullsized":"2024-09-13T11:25:31.087593+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":5,"last_scrub":"112'4376","last_scrub_stamp":"2024-09-13T11:24:13.454575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:24:13.454575+0000","objects_scrubbed":3036,"log_size":7021,"log_dups_size":0,"ondisk_log_size":7021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:34.811860+0000","scrub_duration":375,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":202244096,"num_objects":3086,"num_object_clones":0,"num_object_copies":6172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3086,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2654,"num_write_kb":86528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0],"acting":[10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.17","version":"170'6908","reported_seq":11121,"reported_epoch":170,"state":"active+clean","last_fresh":"2024-09-13T11:25:31.068524+0000","last_change":"2024-09-13T11:25:12.658562+0000","last_active":"2024-09-13T11:25:31.068524+0000","last_peered":"2024-09-13T11:25:31.068524+0000","last_clean":"2024-09-13T11:25:31.068524+0000","last_became_active":"2024-09-13T11:23:42.935601+0000","last_became_peered":"2024-09-13T11:23:42.935601+0000","last_unstale":"2024-09-13T11:25:31.068524+0000","last_undegraded":"2024-09-13T11:25:31.068524+0000","last_fullsized":"2024-09-13T11:25:31.068524+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":5,"last_scrub":"153'6197","last_scrub_stamp":"2024-09-13T11:25:12.658464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:12.658464+0000","objects_scrubbed":3743,"log_size":6908,"log_dups_size":0,"ondisk_log_size":6908,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:26:25.625277+0000","scrub_duration":467,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":198639616,"num_objects":3031,"num_object_clones":0,"num_object_copies":6062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3031,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3557,"num_write_kb":146688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3],"acting":[10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"170'7270","reported_seq":12120,"reported_epoch":170,"state":"active+clean","last_fresh":"2024-09-13T11:25:32.071492+0000","last_change":"2024-09-13T11:25:01.903487+0000","last_active":"2024-09-13T11:25:32.071492+0000","last_peered":"2024-09-13T11:25:32.071492+0000","last_clean":"2024-09-13T11:25:32.071492+0000","last_became_active":"2024-09-13T11:23:46.105237+0000","last_became_peered":"2024-09-13T11:23:46.105237+0000","last_unstale":"2024-09-13T11:25:32.071492+0000","last_undegraded":"2024-09-13T11:25:32.071492+0000","last_fullsized":"2024-09-13T11:25:32.071492+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":5,"last_scrub":"147'6035","last_scrub_stamp":"2024-09-13T11:25:01.903395+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:01.903395+0000","objects_scrubbed":4359,"log_size":7270,"log_dups_size":0,"ondisk_log_size":7270,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:26:16.436377+0000","scrub_duration":743,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":203751424,"num_objects":3109,"num_object_clones":0,"num_object_copies":6218,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3109,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3795,"num_write_kb":155200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,15],"acting":[11,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.15","version":"170'6904","reported_seq":18237,"reported_epoch":170,"state":"active+clean","last_fresh":"2024-09-13T11:25:31.062797+0000","last_change":"2024-09-13T11:25:12.198448+0000","last_active":"2024-09-13T11:25:31.062797+0000","last_peered":"2024-09-13T11:25:31.062797+0000","last_clean":"2024-09-13T11:25:31.062797+0000","last_became_active":"2024-09-13T11:24:29.226615+0000","last_became_peered":"2024-09-13T11:24:29.226615+0000","last_unstale":"2024-09-13T11:25:31.062797+0000","last_undegraded":"2024-09-13T11:25:31.062797+0000","last_fullsized":"2024-09-13T11:25:31.062797+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":5,"last_scrub":"91'3384","last_scrub_stamp":"2024-09-13T11:23:48.332600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:48.332600+0000","objects_scrubbed":2032,"log_size":6904,"log_dups_size":0,"ondisk_log_size":6904,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":235,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":197984256,"num_objects":3021,"num_object_clones":0,"num_object_copies":6042,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3021,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3521,"num_write_kb":144320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3385,"num_bytes_recovered":221839360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.14","version":"173'7203","reported_seq":18468,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-09-13T11:25:34.992634+0000","last_change":"2024-09-13T11:25:15.378521+0000","last_active":"2024-09-13T11:25:34.992634+0000","last_peered":"2024-09-13T11:25:34.992634+0000","last_clean":"2024-09-13T11:25:34.992634+0000","last_became_active":"2024-09-13T11:24:29.253106+0000","last_became_peered":"2024-09-13T11:24:29.253106+0000","last_unstale":"2024-09-13T11:25:34.992634+0000","last_undegraded":"2024-09-13T11:25:34.992634+0000","last_fullsized":"2024-09-13T11:25:34.992634+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":5,"last_scrub":"91'3371","last_scrub_stamp":"2024-09-13T11:23:48.696960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:48.696960+0000","objects_scrubbed":1935,"log_size":7203,"log_dups_size":0,"ondisk_log_size":7203,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":735,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":199819264,"num_objects":3049,"num_object_clones":0,"num_object_copies":6098,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3049,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3842,"num_write_kb":158592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3437,"num_bytes_recovered":225247232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,15],"acting":[3,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":312,"reported_epoch":170,"state":"active+clean","last_fresh":"2024-09-13T11:25:30.278347+0000","last_change":"2024-09-13T11:24:23.212494+0000","last_active":"2024-09-13T11:25:30.278347+0000","last_peered":"2024-09-13T11:25:30.278347+0000","last_clean":"2024-09-13T11:25:30.278347+0000","last_became_active":"2024-09-13T11:22:14.192897+0000","last_became_peered":"2024-09-13T11:22:14.192897+0000","last_unstale":"2024-09-13T11:25:30.278347+0000","last_undegraded":"2024-09-13T11:25:30.278347+0000","last_fullsized":"2024-09-13T11:25:30.278347+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:24:23.212406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:24:23.212406+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00084655899999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,10],"acting":[15,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"3.6","version":"170'6056","reported_seq":9927,"reported_epoch":170,"state":"active+clean","last_fresh":"2024-09-13T11:25:31.914698+0000","last_change":"2024-09-13T11:24:49.416851+0000","last_active":"2024-09-13T11:25:31.914698+0000","last_peered":"2024-09-13T11:25:31.914698+0000","last_clean":"2024-09-13T11:25:31.914698+0000","last_became_active":"2024-09-13T11:24:29.322558+0000","last_became_peered":"2024-09-13T11:24:29.322558+0000","last_unstale":"2024-09-13T11:25:31.914698+0000","last_undegraded":"2024-09-13T11:25:31.914698+0000","last_fullsized":"2024-09-13T11:25:31.914698+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"136'5216","last_scrub_stamp":"2024-09-13T11:24:49.416706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:24:49.416706+0000","objects_scrubbed":2083,"log_size":6056,"log_dups_size":0,"ondisk_log_size":6056,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:26:09.525086+0000","scrub_duration":247,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":100663296,"num_objects":1536,"num_object_clones":0,"num_object_copies":3072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1536,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":841,"num_write_kb":9408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,6],"acting":[13,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":319,"reported_epoch":170,"state":"active+clean","last_fresh":"2024-09-13T11:25:30.290690+0000","last_change":"2024-09-13T11:24:59.556253+0000","last_active":"2024-09-13T11:25:30.290690+0000","last_peered":"2024-09-13T11:25:30.290690+0000","last_clean":"2024-09-13T11:25:30.290690+0000","last_became_active":"2024-09-13T11:22:14.211923+0000","last_became_peered":"2024-09-13T11:22:14.211923+0000","last_unstale":"2024-09-13T11:25:30.290690+0000","last_undegraded":"2024-09-13T11:25:30.290690+0000","last_fullsized":"2024-09-13T11:25:30.290690+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:24:59.556180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:24:59.556180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:26:06.229322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011689529999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0],"acting":[9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"170'5865","reported_seq":6230,"reported_epoch":170,"state":"active+clean","last_fresh":"2024-09-13T11:25:32.025740+0000","last_change":"2024-09-13T11:24:33.336210+0000","last_active":"2024-09-13T11:25:32.025740+0000","last_peered":"2024-09-13T11:25:32.025740+0000","last_clean":"2024-09-13T11:25:32.025740+0000","last_became_active":"2024-09-13T11:24:24.092975+0000","last_became_peered":"2024-09-13T11:24:24.092975+0000","last_unstale":"2024-09-13T11:25:32.025740+0000","last_undegraded":"2024-09-13T11:25:32.025740+0000","last_fullsized":"2024-09-13T11:25:32.025740+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"125'4820","last_scrub_stamp":"2024-09-13T11:24:33.336091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:24:33.336091+0000","objects_scrubbed":1766,"log_size":5865,"log_dups_size":0,"ondisk_log_size":5865,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:54.538352+0000","scrub_duration":263,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":101122048,"num_objects":1543,"num_object_clones":0,"num_object_copies":3086,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1543,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1047,"num_write_kb":26368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,8],"acting":[13,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":312,"reported_epoch":170,"state":"active+clean","last_fresh":"2024-09-13T11:25:30.292381+0000","last_change":"2024-09-13T11:23:59.530794+0000","last_active":"2024-09-13T11:25:30.292381+0000","last_peered":"2024-09-13T11:25:30.292381+0000","last_clean":"2024-09-13T11:25:30.292381+0000","last_became_active":"2024-09-13T11:22:14.581458+0000","last_became_peered":"2024-09-13T11:22:14.581458+0000","last_unstale":"2024-09-13T11:25:30.292381+0000","last_undegraded":"2024-09-13T11:25:30.292381+0000","last_fullsized":"2024-09-13T11:25:30.292381+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:23:59.530670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:23:59.530670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001026047,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5],"acting":[8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"170'5900","reported_seq":6237,"reported_epoch":170,"state":"active+clean","last_fresh":"2024-09-13T11:25:30.298801+0000","last_change":"2024-09-13T11:24:24.579036+0000","last_active":"2024-09-13T11:25:30.298801+0000","last_peered":"2024-09-13T11:25:30.298801+0000","last_clean":"2024-09-13T11:25:30.298801+0000","last_became_active":"2024-09-13T11:24:24.083077+0000","last_became_peered":"2024-09-13T11:24:24.083077+0000","last_unstale":"2024-09-13T11:25:30.298801+0000","last_undegraded":"2024-09-13T11:25:30.298801+0000","last_fullsized":"2024-09-13T11:25:30.298801+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'4737","last_scrub_stamp":"2024-09-13T11:24:24.578900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:24:24.578900+0000","objects_scrubbed":1690,"log_size":5900,"log_dups_size":0,"ondisk_log_size":5900,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:38.103421+0000","scrub_duration":407,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":101318656,"num_objects":1546,"num_object_clones":0,"num_object_copies":3092,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1546,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1166,"num_write_kb":32704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,12],"acting":[1,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":300,"reported_epoch":170,"state":"active+clean","last_fresh":"2024-09-13T11:25:30.251294+0000","last_change":"2024-09-13T11:24:08.271026+0000","last_active":"2024-09-13T11:25:30.251294+0000","last_peered":"2024-09-13T11:25:30.251294+0000","last_clean":"2024-09-13T11:25:30.251294+0000","last_became_active":"2024-09-13T11:22:47.719636+0000","last_became_peered":"2024-09-13T11:22:47.719636+0000","last_unstale":"2024-09-13T11:25:30.251294+0000","last_undegraded":"2024-09-13T11:25:30.251294+0000","last_fullsized":"2024-09-13T11:25:30.251294+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:24:08.270972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:24:08.270972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:36.805605+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00087476000000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,9],"acting":[1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"170'5861","reported_seq":6217,"reported_epoch":170,"state":"active+clean","last_fresh":"2024-09-13T11:25:30.904229+0000","last_change":"2024-09-13T11:24:24.616937+0000","last_active":"2024-09-13T11:25:30.904229+0000","last_peered":"2024-09-13T11:25:30.904229+0000","last_clean":"2024-09-13T11:25:30.904229+0000","last_became_active":"2024-09-13T11:24:24.093492+0000","last_became_peered":"2024-09-13T11:24:24.093492+0000","last_unstale":"2024-09-13T11:25:30.904229+0000","last_undegraded":"2024-09-13T11:25:30.904229+0000","last_fullsized":"2024-09-13T11:25:30.904229+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'4744","last_scrub_stamp":"2024-09-13T11:24:24.616633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:24:24.616633+0000","objects_scrubbed":1624,"log_size":5861,"log_dups_size":0,"ondisk_log_size":5861,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:42.911074+0000","scrub_duration":415,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":99090432,"num_objects":1512,"num_object_clones":0,"num_object_copies":3024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1512,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1124,"num_write_kb":32384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,13],"acting":[15,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"3.3","version":"170'5057","reported_seq":5433,"reported_epoch":170,"state":"active+clean","last_fresh":"2024-09-13T11:25:31.000561+0000","last_change":"2024-09-13T11:25:08.394545+0000","last_active":"2024-09-13T11:25:31.000561+0000","last_peered":"2024-09-13T11:25:31.000561+0000","last_clean":"2024-09-13T11:25:31.000561+0000","last_became_active":"2024-09-13T11:23:31.930448+0000","last_became_peered":"2024-09-13T11:23:31.930448+0000","last_unstale":"2024-09-13T11:25:31.000561+0000","last_undegraded":"2024-09-13T11:25:31.000561+0000","last_fullsized":"2024-09-13T11:25:31.000561+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"151'4587","last_scrub_stamp":"2024-09-13T11:25:08.394441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:08.394441+0000","objects_scrubbed":1942,"log_size":5057,"log_dups_size":0,"ondisk_log_size":5057,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:26:27.641900+0000","scrub_duration":219,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96272384,"num_objects":1469,"num_object_clones":0,"num_object_copies":2938,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1469,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1935,"num_write_kb":80896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,7],"acting":[10,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"28'2","reported_seq":250,"reported_epoch":170,"state":"active+clean","last_fresh":"2024-09-13T11:25:30.267837+0000","last_change":"2024-09-13T11:25:13.869564+0000","last_active":"2024-09-13T11:25:30.267837+0000","last_peered":"2024-09-13T11:25:30.267837+0000","last_clean":"2024-09-13T11:25:30.267837+0000","last_became_active":"2024-09-13T11:23:12.546348+0000","last_became_peered":"2024-09-13T11:23:12.546348+0000","last_unstale":"2024-09-13T11:25:30.267837+0000","last_undegraded":"2024-09-13T11:25:30.267837+0000","last_fullsized":"2024-09-13T11:25:30.267837+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2","last_scrub_stamp":"2024-09-13T11:25:13.869413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:25:13.869413+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:26:38.761718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0025826730000000002,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8],"acting":[2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"170'5068","reported_seq":5442,"reported_epoch":170,"state":"active+clean","last_fresh":"2024-09-13T11:25:31.101943+0000","last_change":"2024-09-13T11:25:09.842875+0000","last_active":"2024-09-13T11:25:31.101943+0000","last_peered":"2024-09-13T11:25:31.101943+0000","last_clean":"2024-09-13T11:25:31.101943+0000","last_became_active":"2024-09-13T11:23:31.921388+0000","last_became_peered":"2024-09-13T11:23:31.921388+0000","last_unstale":"2024-09-13T11:25:31.101943+0000","last_undegraded":"2024-09-13T11:25:31.101943+0000","last_fullsized":"2024-09-13T11:25:31.101943+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"151'4644","last_scrub_stamp":"2024-09-13T11:25:09.842728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:09.842728+0000","objects_scrubbed":1944,"log_size":5068,"log_dups_size":0,"ondisk_log_size":5068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:26:25.553642+0000","scrub_duration":395,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":99614720,"num_objects":1520,"num_object_clones":0,"num_object_copies":3040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1520,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1962,"num_write_kb":84032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,15],"acting":[12,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":153,"reported_epoch":170,"state":"active+clean","last_fresh":"2024-09-13T11:25:30.250985+0000","last_change":"2024-09-13T11:24:29.096863+0000","last_active":"2024-09-13T11:25:30.250985+0000","last_peered":"2024-09-13T11:25:30.250985+0000","last_clean":"2024-09-13T11:25:30.250985+0000","last_became_active":"2024-09-13T11:24:29.096575+0000","last_became_peered":"2024-09-13T11:24:29.096575+0000","last_unstale":"2024-09-13T11:25:30.250985+0000","last_undegraded":"2024-09-13T11:25:30.250985+0000","last_fullsized":"2024-09-13T11:25:30.250985+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:24:22.064711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:24:22.064711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:34.178168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00081225399999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"170'5051","reported_seq":5428,"reported_epoch":170,"state":"active+clean","last_fresh":"2024-09-13T11:25:32.076900+0000","last_change":"2024-09-13T11:25:00.861990+0000","last_active":"2024-09-13T11:25:32.076900+0000","last_peered":"2024-09-13T11:25:32.076900+0000","last_clean":"2024-09-13T11:25:32.076900+0000","last_became_active":"2024-09-13T11:23:31.926528+0000","last_became_peered":"2024-09-13T11:23:31.926528+0000","last_unstale":"2024-09-13T11:25:32.076900+0000","last_undegraded":"2024-09-13T11:25:32.076900+0000","last_fullsized":"2024-09-13T11:25:32.076900+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"146'4447","last_scrub_stamp":"2024-09-13T11:25:00.861834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:00.861834+0000","objects_scrubbed":2126,"log_size":5051,"log_dups_size":0,"ondisk_log_size":5051,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:26:15.411903+0000","scrub_duration":787,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":99549184,"num_objects":1519,"num_object_clones":0,"num_object_copies":3038,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1519,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1981,"num_write_kb":85056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,11],"acting":[13,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":318,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-09-13T11:25:34.282870+0000","last_change":"2024-09-13T11:24:11.895017+0000","last_active":"2024-09-13T11:25:34.282870+0000","last_peered":"2024-09-13T11:25:34.282870+0000","last_clean":"2024-09-13T11:25:34.282870+0000","last_became_active":"2024-09-13T11:22:14.211435+0000","last_became_peered":"2024-09-13T11:22:14.211435+0000","last_unstale":"2024-09-13T11:25:34.282870+0000","last_undegraded":"2024-09-13T11:25:34.282870+0000","last_fullsized":"2024-09-13T11:25:34.282870+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:24:11.894937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:24:11.894937+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0010010360000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2","version":"170'5117","reported_seq":8956,"reported_epoch":170,"state":"active+clean","last_fresh":"2024-09-13T11:25:31.008997+0000","last_change":"2024-09-13T11:25:23.765128+0000","last_active":"2024-09-13T11:25:31.008997+0000","last_peered":"2024-09-13T11:25:31.008997+0000","last_clean":"2024-09-13T11:25:31.008997+0000","last_became_active":"2024-09-13T11:24:29.175407+0000","last_became_peered":"2024-09-13T11:24:29.175407+0000","last_unstale":"2024-09-13T11:25:31.008997+0000","last_undegraded":"2024-09-13T11:25:31.008997+0000","last_fullsized":"2024-09-13T11:25:31.008997+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"164'4994","last_scrub_stamp":"2024-09-13T11:25:23.765047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:23.765047+0000","objects_scrubbed":1689,"log_size":5117,"log_dups_size":0,"ondisk_log_size":5117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:26:45.252942+0000","scrub_duration":283,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":102432768,"num_objects":1563,"num_object_clones":0,"num_object_copies":3126,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1563,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2060,"num_write_kb":89024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1731,"num_bytes_recovered":113442816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,3],"acting":[12,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[]},{"pgid":"2.3","version":"26'1","reported_seq":123,"reported_epoch":170,"state":"active+clean","last_fresh":"2024-09-13T11:25:30.278115+0000","last_change":"2024-09-13T11:25:23.034357+0000","last_active":"2024-09-13T11:25:30.278115+0000","last_peered":"2024-09-13T11:25:30.278115+0000","last_clean":"2024-09-13T11:25:30.278115+0000","last_became_active":"2024-09-13T11:24:29.110028+0000","last_became_peered":"2024-09-13T11:24:29.110028+0000","last_unstale":"2024-09-13T11:25:30.278115+0000","last_undegraded":"2024-09-13T11:25:30.278115+0000","last_fullsized":"2024-09-13T11:25:30.278115+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"26'1","last_scrub_stamp":"2024-09-13T11:25:23.034247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:25:23.034247+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:26:42.429349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,8],"acting":[15,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"1.0","version":"26'112","reported_seq":454,"reported_epoch":170,"state":"active+clean","last_fresh":"2024-09-13T11:25:30.316418+0000","last_change":"2024-09-13T11:25:23.642366+0000","last_active":"2024-09-13T11:25:30.316418+0000","last_peered":"2024-09-13T11:25:30.316418+0000","last_clean":"2024-09-13T11:25:30.316418+0000","last_became_active":"2024-09-13T11:22:12.129721+0000","last_became_peered":"2024-09-13T11:22:12.129721+0000","last_unstale":"2024-09-13T11:25:30.316418+0000","last_undegraded":"2024-09-13T11:25:30.316418+0000","last_fullsized":"2024-09-13T11:25:30.316418+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"26'112","last_scrub_stamp":"2024-09-13T11:25:23.642204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:11.098388+0000","last_clean_scrub_stamp":"2024-09-13T11:25:23.642204+0000","objects_scrubbed":2,"log_size":112,"log_dups_size":0,"ondisk_log_size":112,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:26:42.628319+0000","scrub_duration":67,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.8","version":"170'5877","reported_seq":6234,"reported_epoch":170,"state":"active+clean","last_fresh":"2024-09-13T11:25:31.051573+0000","last_change":"2024-09-13T11:24:24.506212+0000","last_active":"2024-09-13T11:25:31.051573+0000","last_peered":"2024-09-13T11:25:31.051573+0000","last_clean":"2024-09-13T11:25:31.051573+0000","last_became_active":"2024-09-13T11:24:24.096220+0000","last_became_peered":"2024-09-13T11:24:24.096220+0000","last_unstale":"2024-09-13T11:25:31.051573+0000","last_undegraded":"2024-09-13T11:25:31.051573+0000","last_fullsized":"2024-09-13T11:25:31.051573+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'4749","last_scrub_stamp":"2024-09-13T11:24:24.506109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:24:24.506109+0000","objects_scrubbed":1690,"log_size":5877,"log_dups_size":0,"ondisk_log_size":5877,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:49.430042+0000","scrub_duration":215,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":100728832,"num_objects":1537,"num_object_clones":0,"num_object_copies":3074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1537,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1129,"num_write_kb":31232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.9","version":"170'5907","reported_seq":6263,"reported_epoch":170,"state":"active+clean","last_fresh":"2024-09-13T11:25:31.088005+0000","last_change":"2024-09-13T11:24:24.928964+0000","last_active":"2024-09-13T11:25:31.088005+0000","last_peered":"2024-09-13T11:25:31.088005+0000","last_clean":"2024-09-13T11:25:31.088005+0000","last_became_active":"2024-09-13T11:24:24.082504+0000","last_became_peered":"2024-09-13T11:24:24.082504+0000","last_unstale":"2024-09-13T11:25:31.088005+0000","last_undegraded":"2024-09-13T11:25:31.088005+0000","last_fullsized":"2024-09-13T11:25:31.088005+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'4799","last_scrub_stamp":"2024-09-13T11:24:24.928724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:24:24.928724+0000","objects_scrubbed":1705,"log_size":5907,"log_dups_size":0,"ondisk_log_size":5907,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:43.845801+0000","scrub_duration":255,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":102629376,"num_objects":1566,"num_object_clones":0,"num_object_copies":3132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1566,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1111,"num_write_kb":31104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,2],"acting":[12,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[]},{"pgid":"3.a","version":"173'6337","reported_seq":6703,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-09-13T11:25:34.984376+0000","last_change":"2024-09-13T11:24:25.137871+0000","last_active":"2024-09-13T11:25:34.984376+0000","last_peered":"2024-09-13T11:25:34.984376+0000","last_clean":"2024-09-13T11:25:34.984376+0000","last_became_active":"2024-09-13T11:24:24.089253+0000","last_became_peered":"2024-09-13T11:24:24.089253+0000","last_unstale":"2024-09-13T11:25:34.984376+0000","last_undegraded":"2024-09-13T11:25:34.984376+0000","last_fullsized":"2024-09-13T11:25:34.984376+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'5157","last_scrub_stamp":"2024-09-13T11:24:25.137777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:24:25.137777+0000","objects_scrubbed":1777,"log_size":6337,"log_dups_size":0,"ondisk_log_size":6337,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:36.149013+0000","scrub_duration":223,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96206848,"num_objects":1468,"num_object_clones":0,"num_object_copies":2936,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1468,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1183,"num_write_kb":27968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.b","version":"170'6256","reported_seq":6616,"reported_epoch":170,"state":"active+clean","last_fresh":"2024-09-13T11:25:32.023235+0000","last_change":"2024-09-13T11:24:25.503864+0000","last_active":"2024-09-13T11:25:32.023235+0000","last_peered":"2024-09-13T11:25:32.023235+0000","last_clean":"2024-09-13T11:25:32.023235+0000","last_became_active":"2024-09-13T11:24:24.085684+0000","last_became_peered":"2024-09-13T11:24:24.085684+0000","last_unstale":"2024-09-13T11:25:32.023235+0000","last_undegraded":"2024-09-13T11:25:32.023235+0000","last_fullsized":"2024-09-13T11:25:32.023235+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'5112","last_scrub_stamp":"2024-09-13T11:24:25.503716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:24:25.503716+0000","objects_scrubbed":1718,"log_size":6256,"log_dups_size":0,"ondisk_log_size":6256,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:37.636639+0000","scrub_duration":395,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":100401152,"num_objects":1532,"num_object_clones":0,"num_object_copies":3064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1532,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1144,"num_write_kb":30656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,1],"acting":[13,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"3.c","version":"170'6255","reported_seq":6614,"reported_epoch":170,"state":"active+clean","last_fresh":"2024-09-13T11:25:32.014973+0000","last_change":"2024-09-13T11:24:24.595021+0000","last_active":"2024-09-13T11:25:32.014973+0000","last_peered":"2024-09-13T11:25:32.014973+0000","last_clean":"2024-09-13T11:25:32.014973+0000","last_became_active":"2024-09-13T11:24:24.090847+0000","last_became_peered":"2024-09-13T11:24:24.090847+0000","last_unstale":"2024-09-13T11:25:32.014973+0000","last_undegraded":"2024-09-13T11:25:32.014973+0000","last_fullsized":"2024-09-13T11:25:32.014973+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'5154","last_scrub_stamp":"2024-09-13T11:24:24.594848+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:24:24.594848+0000","objects_scrubbed":1720,"log_size":6255,"log_dups_size":0,"ondisk_log_size":6255,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:35.181590+0000","scrub_duration":343,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":100532224,"num_objects":1534,"num_object_clones":0,"num_object_copies":3068,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1534,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1106,"num_write_kb":29440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,13],"acting":[11,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.d","version":"170'6234","reported_seq":6593,"reported_epoch":170,"state":"active+clean","last_fresh":"2024-09-13T11:25:31.170448+0000","last_change":"2024-09-13T11:24:24.967444+0000","last_active":"2024-09-13T11:25:31.170448+0000","last_peered":"2024-09-13T11:25:31.170448+0000","last_clean":"2024-09-13T11:25:31.170448+0000","last_became_active":"2024-09-13T11:24:24.087224+0000","last_became_peered":"2024-09-13T11:24:24.087224+0000","last_unstale":"2024-09-13T11:25:31.170448+0000","last_undegraded":"2024-09-13T11:25:31.170448+0000","last_fullsized":"2024-09-13T11:25:31.170448+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'5110","last_scrub_stamp":"2024-09-13T11:24:24.967337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:24:24.967337+0000","objects_scrubbed":1725,"log_size":6234,"log_dups_size":0,"ondisk_log_size":6234,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:53.620949+0000","scrub_duration":255,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":102498304,"num_objects":1564,"num_object_clones":0,"num_object_copies":3128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1564,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1127,"num_write_kb":30912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.e","version":"170'7460","reported_seq":12716,"reported_epoch":170,"state":"active+clean","last_fresh":"2024-09-13T11:25:31.108008+0000","last_change":"2024-09-13T11:23:32.368856+0000","last_active":"2024-09-13T11:25:31.108008+0000","last_peered":"2024-09-13T11:25:31.108008+0000","last_clean":"2024-09-13T11:25:31.108008+0000","last_became_active":"2024-09-13T11:23:31.925869+0000","last_became_peered":"2024-09-13T11:23:31.925869+0000","last_unstale":"2024-09-13T11:25:31.108008+0000","last_undegraded":"2024-09-13T11:25:31.108008+0000","last_fullsized":"2024-09-13T11:25:31.108008+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'3527","last_scrub_stamp":"2024-09-13T11:23:32.368727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:32.368727+0000","objects_scrubbed":1774,"log_size":7460,"log_dups_size":0,"ondisk_log_size":7460,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":219,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":198180864,"num_objects":3024,"num_object_clones":0,"num_object_copies":6048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3934,"num_write_kb":165888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"170'7488","reported_seq":7867,"reported_epoch":170,"state":"active+clean","last_fresh":"2024-09-13T11:25:32.067943+0000","last_change":"2024-09-13T11:25:00.289940+0000","last_active":"2024-09-13T11:25:32.067943+0000","last_peered":"2024-09-13T11:25:32.067943+0000","last_clean":"2024-09-13T11:25:32.067943+0000","last_became_active":"2024-09-13T11:23:31.935239+0000","last_became_peered":"2024-09-13T11:23:31.935239+0000","last_unstale":"2024-09-13T11:25:32.067943+0000","last_undegraded":"2024-09-13T11:25:32.067943+0000","last_fullsized":"2024-09-13T11:25:32.067943+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"146'6154","last_scrub_stamp":"2024-09-13T11:25:00.289673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:00.289673+0000","objects_scrubbed":4394,"log_size":7488,"log_dups_size":0,"ondisk_log_size":7488,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:26:24.274856+0000","scrub_duration":1187,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":199098368,"num_objects":3038,"num_object_clones":0,"num_object_copies":6076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3038,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4068,"num_write_kb":170560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,12],"acting":[11,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.10","version":"170'7117","reported_seq":11647,"reported_epoch":170,"state":"active+clean","last_fresh":"2024-09-13T11:25:31.047614+0000","last_change":"2024-09-13T11:25:10.161564+0000","last_active":"2024-09-13T11:25:31.047614+0000","last_peered":"2024-09-13T11:25:31.047614+0000","last_clean":"2024-09-13T11:25:31.047614+0000","last_became_active":"2024-09-13T11:23:41.931183+0000","last_became_peered":"2024-09-13T11:23:41.931183+0000","last_unstale":"2024-09-13T11:25:31.047614+0000","last_undegraded":"2024-09-13T11:25:31.047614+0000","last_fullsized":"2024-09-13T11:25:31.047614+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":5,"last_scrub":"151'6298","last_scrub_stamp":"2024-09-13T11:25:10.161464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:10.161464+0000","objects_scrubbed":3964,"log_size":7117,"log_dups_size":0,"ondisk_log_size":7117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:26:33.015478+0000","scrub_duration":607,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":205127680,"num_objects":3130,"num_object_clones":0,"num_object_copies":6260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3130,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3651,"num_write_kb":152000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.11","version":"170'7054","reported_seq":10357,"reported_epoch":170,"state":"active+clean","last_fresh":"2024-09-13T11:25:31.066004+0000","last_change":"2024-09-13T11:25:24.287326+0000","last_active":"2024-09-13T11:25:31.066004+0000","last_peered":"2024-09-13T11:25:31.066004+0000","last_clean":"2024-09-13T11:25:31.066004+0000","last_became_active":"2024-09-13T11:23:39.063172+0000","last_became_peered":"2024-09-13T11:23:39.063172+0000","last_unstale":"2024-09-13T11:25:31.066004+0000","last_undegraded":"2024-09-13T11:25:31.066004+0000","last_fullsized":"2024-09-13T11:25:31.066004+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":5,"last_scrub":"165'6821","last_scrub_stamp":"2024-09-13T11:25:24.287082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:24.287082+0000","objects_scrubbed":3406,"log_size":7054,"log_dups_size":0,"ondisk_log_size":7054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:26:49.405332+0000","scrub_duration":947,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":207093760,"num_objects":3160,"num_object_clones":0,"num_object_copies":6320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3160,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3713,"num_write_kb":155584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11],"acting":[4,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.12","version":"170'7086","reported_seq":17200,"reported_epoch":170,"state":"active+clean","last_fresh":"2024-09-13T11:25:31.168446+0000","last_change":"2024-09-13T11:25:26.257543+0000","last_active":"2024-09-13T11:25:31.168446+0000","last_peered":"2024-09-13T11:25:31.168446+0000","last_clean":"2024-09-13T11:25:31.168446+0000","last_became_active":"2024-09-13T11:24:29.268229+0000","last_became_peered":"2024-09-13T11:24:29.268229+0000","last_unstale":"2024-09-13T11:25:31.168446+0000","last_undegraded":"2024-09-13T11:25:31.168446+0000","last_fullsized":"2024-09-13T11:25:31.168446+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'3109","last_scrub_stamp":"2024-09-13T11:23:35.111320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:35.111320+0000","objects_scrubbed":599,"log_size":7086,"log_dups_size":0,"ondisk_log_size":7086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:31.236137+0000","scrub_duration":239,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":204537856,"num_objects":3121,"num_object_clones":0,"num_object_copies":6242,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3121,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3977,"num_write_kb":171712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3457,"num_bytes_recovered":226557952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.13","version":"170'7142","reported_seq":10379,"reported_epoch":170,"state":"active+clean","last_fresh":"2024-09-13T11:25:32.046674+0000","last_change":"2024-09-13T11:25:01.153044+0000","last_active":"2024-09-13T11:25:32.046674+0000","last_peered":"2024-09-13T11:25:32.046674+0000","last_clean":"2024-09-13T11:25:32.046674+0000","last_became_active":"2024-09-13T11:23:41.082433+0000","last_became_peered":"2024-09-13T11:23:41.082433+0000","last_unstale":"2024-09-13T11:25:32.046674+0000","last_undegraded":"2024-09-13T11:25:32.046674+0000","last_fullsized":"2024-09-13T11:25:32.046674+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":5,"last_scrub":"147'5921","last_scrub_stamp":"2024-09-13T11:25:01.152971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:01.152971+0000","objects_scrubbed":4311,"log_size":7142,"log_dups_size":0,"ondisk_log_size":7142,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:26:07.027631+0000","scrub_duration":1011,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":199819264,"num_objects":3049,"num_object_clones":0,"num_object_copies":6098,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3049,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3746,"num_write_kb":154048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7],"acting":[11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":46,"stat_sum":{"num_bytes":6450216984,"num_objects":98424,"num_object_clones":0,"num_object_copies":196848,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":754,"num_objects_unfound":0,"num_objects_dirty":98424,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":98272,"num_write_kb":3634433,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21837,"num_bytes_recovered":1431109632,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":14407245824,"data_stored":14407237680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":43282,"internal_metadata":0},"log_size":291656,"ondisk_log_size":291656,"up":92,"acting":92,"num_store_stats":16},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":11},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":112,"ondisk_log_size":112,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":13,"up_from":23,"seq":98784247852,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":698916,"kb_used_data":587220,"kb_used_omap":1,"kb_used_meta":111678,"kb_avail":93672924,"statfs":{"total":96636764160,"available":95921074176,"internally_reserved":0,"allocated":601313280,"data_stored":600551624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":114358725},"hb_peers":[0,1,2,4,6,8,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":23,"seq":98784247851,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":651516,"kb_used_data":584236,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":93720324,"statfs":{"total":96636764160,"available":95969611776,"internally_reserved":0,"allocated":598257664,"data_stored":597532634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":68876736},"hb_peers":[0,1,3,8,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":40,"seq":171798691874,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1102284,"kb_used_data":993916,"kb_used_omap":4,"kb_used_meta":108347,"kb_avail":93269556,"statfs":{"total":96636764160,"available":95508025344,"internally_reserved":0,"allocated":1017769984,"data_stored":1017025142,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4433,"internal_metadata":110948015},"hb_peers":[0,2,3,4,5,6,7,8,9,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":21,"seq":90194313260,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1095640,"kb_used_data":988104,"kb_used_omap":1,"kb_used_meta":107518,"kb_avail":93276200,"statfs":{"total":96636764160,"available":95514828800,"internally_reserved":0,"allocated":1011818496,"data_stored":1011032022,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":110098887},"hb_peers":[0,2,4,5,7,9,10,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":21,"seq":90194313262,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1276568,"kb_used_data":1169032,"kb_used_omap":1,"kb_used_meta":107518,"kb_avail":93095272,"statfs":{"total":96636764160,"available":95329558528,"internally_reserved":0,"allocated":1197088768,"data_stored":1196334010,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":110098887},"hb_peers":[0,2,4,6,7,8,9,11,12,13],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":22,"seq":94489280554,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1274284,"kb_used_data":1162588,"kb_used_omap":1,"kb_used_meta":111678,"kb_avail":93097556,"statfs":{"total":96636764160,"available":95331897344,"internally_reserved":0,"allocated":1190490112,"data_stored":1189784016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":114358725},"hb_peers":[1,3,4,5,7,8,10,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":22,"seq":94489280555,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":571992,"kb_used_data":496392,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":93799848,"statfs":{"total":96636764160,"available":96051044352,"internally_reserved":0,"allocated":508305408,"data_stored":507561459,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,1,3,4,5,7,8,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":15,"up_from":22,"seq":94489280555,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1147552,"kb_used_data":1048336,"kb_used_omap":1,"kb_used_meta":99198,"kb_avail":93224288,"statfs":{"total":96636764160,"available":95461670912,"internally_reserved":0,"allocated":1073496064,"data_stored":1072801592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":101579206},"hb_peers":[0,4,5,6,7,8,10,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":22,"seq":94489280556,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":771680,"kb_used_data":704400,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":93600160,"statfs":{"total":96636764160,"available":95846563840,"internally_reserved":0,"allocated":721305600,"data_stored":720554715,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[3,5,6,7,9,10,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":22,"seq":94489280555,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":566652,"kb_used_data":513260,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":93805188,"statfs":{"total":96636764160,"available":96056512512,"internally_reserved":0,"allocated":525578240,"data_stored":524858451,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54655430},"hb_peers":[0,1,3,5,7,8,10,11,12,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":22,"seq":94489280555,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1404924,"kb_used_data":1297388,"kb_used_omap":1,"kb_used_meta":107518,"kb_avail":92966916,"statfs":{"total":96636764160,"available":95198121984,"internally_reserved":0,"allocated":1328525312,"data_stored":1327802183,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":110098890},"hb_peers":[0,3,5,6,8,9,10,11,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":22,"seq":94489280555,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":561904,"kb_used_data":486304,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":93809936,"statfs":{"total":96636764160,"available":96061374464,"internally_reserved":0,"allocated":497975296,"data_stored":497204678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[1,5,6,7,9,10,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":22,"seq":94489280554,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":989628,"kb_used_data":895980,"kb_used_omap":1,"kb_used_meta":93630,"kb_avail":93382212,"statfs":{"total":96636764160,"available":95623385088,"internally_reserved":0,"allocated":917483520,"data_stored":916757043,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":95877568},"hb_peers":[0,1,3,5,7,9,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":11,"up_from":22,"seq":94489280556,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1386084,"kb_used_data":1282708,"kb_used_omap":1,"kb_used_meta":103358,"kb_avail":92985756,"statfs":{"total":96636764160,"available":95217414144,"internally_reserved":0,"allocated":1313492992,"data_stored":1312735043,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":105839044},"hb_peers":[0,1,2,4,7,8,10,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":12,"up_from":22,"seq":94489280555,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1216804,"kb_used_data":1105044,"kb_used_omap":1,"kb_used_meta":111678,"kb_avail":93155036,"statfs":{"total":96636764160,"available":95390756864,"internally_reserved":0,"allocated":1131565056,"data_stored":1130807271,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":114358725},"hb_peers":[0,1,2,3,4,5,10,11,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":14,"up_from":0,"seq":609885356041,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1186856960,"data_stored":1186856960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1011089408,"data_stored":1011089408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":501747712,"data_stored":501743640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1003945984,"data_stored":1003945984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":710475776,"data_stored":710475776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1187708928,"data_stored":1187708928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":519110656,"data_stored":519110656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1324744704,"data_stored":1324744704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":495255552,"data_stored":495255552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":595656704,"data_stored":595656704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":901054464,"data_stored":901054464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":1310130176,"data_stored":1310130176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":1117650944,"data_stored":1117650944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":599261184,"data_stored":599261184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":872742912,"data_stored":872742912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":43282,"internal_metadata":0},{"poolid":3,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":1069813760,"data_stored":1069809688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T11:25:35.519 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-13T11:25:35.560 INFO:tasks.ceph.osd.12.smithi001.stderr:2024-09-13T11:25:35.559+0000 7f892c6e6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26314) UID: 0 2024-09-13T11:25:35.594 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-13T11:25:35.594 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:25:35.612 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-09-13T11:25:35.654 INFO:tasks.ceph.osd.3.smithi177.stderr:2024-09-13T11:25:35.650+0000 7f10adba4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26574) UID: 0 2024-09-13T11:25:35.748 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-13T11:25:35.748 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:25:35.759 INFO:tasks.ceph.osd.9.smithi092.stderr:2024-09-13T11:25:35.753+0000 7ff5ab3d0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26348) UID: 0 2024-09-13T11:25:35.765 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-13T11:25:35.857 INFO:tasks.ceph.osd.13.smithi092.stderr:2024-09-13T11:25:35.857+0000 7f3e5afea640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26350) UID: 0 2024-09-13T11:25:35.920 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-09-13T11:25:35.921 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:25:35.943 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-13T11:25:35.958 INFO:tasks.ceph.osd.14.smithi162.stderr:2024-09-13T11:25:35.956+0000 7f3df5903640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33154) UID: 0 2024-09-13T11:25:36.058 INFO:tasks.ceph.osd.12.smithi001.stderr:2024-09-13T11:25:36.055+0000 7f892c6e6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26314) UID: 0 2024-09-13T11:25:36.059 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-13T11:25:36.059 INFO:teuthology.orchestra.run.smithi001.stdout:{"fsid":"2fd68ac7-5581-4d90-80a4-30bcf5252a7a","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":218,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":174,"num_osds":16,"num_up_osds":16,"osd_up_since":1726226695,"num_in_osds":15,"osd_in_since":1726226668,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":53},{"state_name":"active+recovering+remapped","count":2}],"num_pgs":55,"num_pools":3,"num_objects":98797,"data_bytes":6474990155,"bytes_used":15134912512,"bytes_avail":1434416549888,"bytes_total":1449551462400,"misplaced_objects":1466,"misplaced_total":197594,"misplaced_ratio":0.0074192536210613679,"recovering_objects_per_sec":169,"recovering_bytes_per_sec":11129285,"recovering_keys_per_sec":0,"num_objects_recovered":836,"num_bytes_recovered":54788096,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":1145},"fsmap":{"epoch":1,"btime":"2024-09-13T11:21:57:222574+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-13T11:24:06.142821+0000","services":{}},"progress_events":{}} 2024-09-13T11:25:36.080 INFO:tasks.ceph.ceph_manager.ceph:making progress, resetting timeout 2024-09-13T11:25:36.081 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-13T11:25:36.081 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T11:25:36.104 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-09-13T11:25:36.104 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:25:36.127 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-13T11:25:36.173 INFO:tasks.ceph.osd.12.smithi001.stderr:2024-09-13T11:25:36.171+0000 7f892c6e6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26314) UID: 0 2024-09-13T11:25:36.260 INFO:tasks.ceph.osd.14.smithi162.stderr:2024-09-13T11:25:36.256+0000 7f3df5903640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33154) UID: 0 2024-09-13T11:25:36.291 INFO:teuthology.orchestra.run.smithi001.stdout:osd.0: {} 2024-09-13T11:25:36.291 INFO:teuthology.orchestra.run.smithi001.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T11:25:36.291 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-09-13T11:25:36.291 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:25:36.307 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-13T11:25:36.317 INFO:teuthology.orchestra.run.smithi001.stdout:osd.1: {} 2024-09-13T11:25:36.317 INFO:teuthology.orchestra.run.smithi001.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T11:25:36.347 INFO:teuthology.orchestra.run.smithi001.stdout:osd.2: {} 2024-09-13T11:25:36.347 INFO:teuthology.orchestra.run.smithi001.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T11:25:36.361 INFO:tasks.ceph.osd.2.smithi162.stderr:2024-09-13T11:25:36.360+0000 7fe92b35a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26313) UID: 0 2024-09-13T11:25:36.370 INFO:teuthology.orchestra.run.smithi001.stdout:osd.3: {} 2024-09-13T11:25:36.371 INFO:teuthology.orchestra.run.smithi001.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T11:25:36.393 INFO:teuthology.orchestra.run.smithi001.stdout:osd.4: {} 2024-09-13T11:25:36.393 INFO:teuthology.orchestra.run.smithi001.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T11:25:36.418 INFO:teuthology.orchestra.run.smithi001.stdout:osd.5: {} 2024-09-13T11:25:36.418 INFO:teuthology.orchestra.run.smithi001.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T11:25:36.443 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-09-13T11:25:36.443 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:25:36.460 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-13T11:25:36.462 INFO:tasks.ceph.osd.10.smithi162.stderr:2024-09-13T11:25:36.460+0000 7fb59c9f0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26317) UID: 0 2024-09-13T11:25:36.475 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-13T11:25:36.476 INFO:teuthology.orchestra.run.smithi001.stderr:dumped all 2024-09-13T11:25:36.493 INFO:teuthology.orchestra.run.smithi001.stdout:{"pg_ready":true,"pg_map":{"version":263,"stamp":"2024-09-13T11:25:36.164744+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":6175097419,"num_objects":94221,"num_object_clones":0,"num_object_copies":188442,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":220,"num_objects_unfound":0,"num_objects_dirty":94221,"num_whiteouts":0,"num_read":87,"num_read_kb":74,"num_write":102626,"num_write_kb":3637107,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22106,"num_bytes_recovered":1448607763,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":295978,"ondisk_log_size":295978,"up":110,"acting":110,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":118,"num_osds":15,"num_per_pool_osds":15,"num_per_pool_omap_osds":15,"kb":1415577600,"kb_used":14196448,"kb_used_data":12801904,"kb_used_omap":26,"kb_used_meta":1394277,"kb_avail":1401381152,"statfs":{"total":1449551462400,"available":1435014299648,"internally_reserved":0,"allocated":13109149696,"data_stored":13097842594,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":26763,"internal_metadata":1427740533},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,0,4,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-315359232,"num_objects":-4812,"num_object_clones":0,"num_object_copies":-9624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1246,"num_objects_unfound":0,"num_objects_dirty":-4812,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4812,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":623,"num_bytes_recovered":40828928,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.001116"},"pg_stats":[{"pgid":"3.2d","version":"173'6306","reported_seq":11475,"reported_epoch":173,"state":"active+recovering+remapped","last_fresh":"2024-09-13T11:25:35.084123+0000","last_change":"2024-09-13T11:25:19.656321+0000","last_active":"2024-09-13T11:25:35.084123+0000","last_peered":"2024-09-13T11:25:35.084123+0000","last_clean":"2024-09-13T11:24:24.974950+0000","last_became_active":"2024-09-13T11:24:27.227288+0000","last_became_peered":"2024-09-13T11:24:27.227288+0000","last_unstale":"2024-09-13T11:25:35.084123+0000","last_undegraded":"2024-09-13T11:25:35.084123+0000","last_fullsized":"2024-09-13T11:25:35.084123+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":6,"last_scrub":"76'3395","last_scrub_stamp":"2024-09-13T11:23:32.776747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:32.776747+0000","objects_scrubbed":1704,"log_size":6306,"log_dups_size":0,"ondisk_log_size":6306,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":179,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":100171776,"num_objects":1529,"num_object_clones":0,"num_object_copies":3058,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":220,"num_objects_unfound":0,"num_objects_dirty":1529,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2069,"num_write_kb":87840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1609,"num_bytes_recovered":105447424,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[3,0],"avail_no_missing":["3","7"],"object_location_counts":[{"shards":"0,3","objects":1419},{"shards":"3,7","objects":110}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2c","version":"174'6425","reported_seq":11773,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-13T11:25:35.967991+0000","last_change":"2024-09-13T11:24:59.456762+0000","last_active":"2024-09-13T11:25:35.967991+0000","last_peered":"2024-09-13T11:25:35.967991+0000","last_clean":"2024-09-13T11:25:35.967991+0000","last_became_active":"2024-09-13T11:24:59.021895+0000","last_became_peered":"2024-09-13T11:24:59.021895+0000","last_unstale":"2024-09-13T11:25:35.967991+0000","last_undegraded":"2024-09-13T11:25:35.967991+0000","last_fullsized":"2024-09-13T11:25:35.967991+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":6,"last_scrub":"145'5672","last_scrub_stamp":"2024-09-13T11:24:59.456653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:24:59.456653+0000","objects_scrubbed":2192,"log_size":6425,"log_dups_size":0,"ondisk_log_size":6425,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:26:20.877991+0000","scrub_duration":279,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":94306304,"num_objects":1439,"num_object_clones":0,"num_object_copies":2878,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1439,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":753,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,7],"acting":[10,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.2b","version":"174'6281","reported_seq":9986,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-13T11:25:35.337136+0000","last_change":"2024-09-13T11:24:45.271477+0000","last_active":"2024-09-13T11:25:35.337136+0000","last_peered":"2024-09-13T11:25:35.337136+0000","last_clean":"2024-09-13T11:25:35.337136+0000","last_became_active":"2024-09-13T11:24:26.180578+0000","last_became_peered":"2024-09-13T11:24:26.180578+0000","last_unstale":"2024-09-13T11:25:35.337136+0000","last_undegraded":"2024-09-13T11:25:35.337136+0000","last_fullsized":"2024-09-13T11:25:35.337136+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":6,"last_scrub":"131'5382","last_scrub_stamp":"2024-09-13T11:24:45.271376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:24:45.271376+0000","objects_scrubbed":1940,"log_size":6281,"log_dups_size":0,"ondisk_log_size":6281,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:59.210895+0000","scrub_duration":279,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":89980928,"num_objects":1373,"num_object_clones":0,"num_object_copies":2746,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1373,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":899,"num_write_kb":10624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2a","version":"174'6430","reported_seq":10322,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-13T11:25:36.070639+0000","last_change":"2024-09-13T11:24:46.625117+0000","last_active":"2024-09-13T11:25:36.070639+0000","last_peered":"2024-09-13T11:25:36.070639+0000","last_clean":"2024-09-13T11:25:36.070639+0000","last_became_active":"2024-09-13T11:24:45.320540+0000","last_became_peered":"2024-09-13T11:24:45.320540+0000","last_unstale":"2024-09-13T11:25:36.070639+0000","last_undegraded":"2024-09-13T11:25:36.070639+0000","last_fullsized":"2024-09-13T11:25:36.070639+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":6,"last_scrub":"133'5491","last_scrub_stamp":"2024-09-13T11:24:46.624970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:24:46.624970+0000","objects_scrubbed":2094,"log_size":6430,"log_dups_size":0,"ondisk_log_size":6430,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:26:04.813280+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":98500608,"num_objects":1503,"num_object_clones":0,"num_object_copies":3006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":941,"num_write_kb":11200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,7],"acting":[10,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.29","version":"174'6086","reported_seq":11793,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-13T11:25:35.973234+0000","last_change":"2024-09-13T11:25:13.333062+0000","last_active":"2024-09-13T11:25:35.973234+0000","last_peered":"2024-09-13T11:25:35.973234+0000","last_clean":"2024-09-13T11:25:35.973234+0000","last_became_active":"2024-09-13T11:25:12.944028+0000","last_became_peered":"2024-09-13T11:25:12.944028+0000","last_unstale":"2024-09-13T11:25:35.973234+0000","last_undegraded":"2024-09-13T11:25:35.973234+0000","last_fullsized":"2024-09-13T11:25:35.973234+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":6,"last_scrub":"154'5579","last_scrub_stamp":"2024-09-13T11:25:13.332959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:13.332959+0000","objects_scrubbed":1937,"log_size":6086,"log_dups_size":0,"ondisk_log_size":6086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:26:24.089228+0000","scrub_duration":243,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":93585408,"num_objects":1428,"num_object_clones":0,"num_object_copies":2856,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1428,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":509,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.28","version":"174'5935","reported_seq":11365,"reported_epoch":174,"state":"active+clean+scrubbing","last_fresh":"2024-09-13T11:25:36.085306+0000","last_change":"2024-09-13T11:25:34.995273+0000","last_active":"2024-09-13T11:25:36.085306+0000","last_peered":"2024-09-13T11:25:36.085306+0000","last_clean":"2024-09-13T11:25:36.085306+0000","last_became_active":"2024-09-13T11:25:34.278640+0000","last_became_peered":"2024-09-13T11:25:34.278640+0000","last_unstale":"2024-09-13T11:25:36.085306+0000","last_undegraded":"2024-09-13T11:25:36.085306+0000","last_fullsized":"2024-09-13T11:25:36.085306+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":6,"last_scrub":"68'2771","last_scrub_stamp":"2024-09-13T11:23:20.644747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:20.644747+0000","objects_scrubbed":1399,"log_size":5935,"log_dups_size":0,"ondisk_log_size":5935,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"scrubbing for 1s","scrub_duration":411,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":94371840,"num_objects":1440,"num_object_clones":0,"num_object_copies":2880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1440,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2188,"num_write_kb":94400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1686,"num_bytes_recovered":110493696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5],"acting":[8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.27","version":"174'6006","reported_seq":11536,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-13T11:25:35.917658+0000","last_change":"2024-09-13T11:25:31.204736+0000","last_active":"2024-09-13T11:25:35.917658+0000","last_peered":"2024-09-13T11:25:35.917658+0000","last_clean":"2024-09-13T11:25:35.917658+0000","last_became_active":"2024-09-13T11:25:30.257911+0000","last_became_peered":"2024-09-13T11:25:30.257911+0000","last_unstale":"2024-09-13T11:25:35.917658+0000","last_undegraded":"2024-09-13T11:25:35.917658+0000","last_fullsized":"2024-09-13T11:25:35.917658+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":6,"last_scrub":"68'2736","last_scrub_stamp":"2024-09-13T11:23:20.443329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:20.443329+0000","objects_scrubbed":0,"log_size":6006,"log_dups_size":0,"ondisk_log_size":6006,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"user requested scrub scheduled @ 2024-09-13T11:25:36.159492+0000","scrub_duration":183,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96206848,"num_objects":1468,"num_object_clones":0,"num_object_copies":2936,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1468,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2284,"num_write_kb":98144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1725,"num_bytes_recovered":113049600,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,11],"acting":[5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.26","version":"174'6127","reported_seq":11714,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-13T11:25:35.931196+0000","last_change":"2024-09-13T11:25:01.697082+0000","last_active":"2024-09-13T11:25:35.931196+0000","last_peered":"2024-09-13T11:25:35.931196+0000","last_clean":"2024-09-13T11:25:35.931196+0000","last_became_active":"2024-09-13T11:24:54.976919+0000","last_became_peered":"2024-09-13T11:24:54.976919+0000","last_unstale":"2024-09-13T11:25:35.931196+0000","last_undegraded":"2024-09-13T11:25:35.931196+0000","last_fullsized":"2024-09-13T11:25:35.931196+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":6,"last_scrub":"147'5433","last_scrub_stamp":"2024-09-13T11:25:01.696886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:01.696886+0000","objects_scrubbed":2158,"log_size":6127,"log_dups_size":0,"ondisk_log_size":6127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:26:22.070621+0000","scrub_duration":403,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":95420416,"num_objects":1456,"num_object_clones":0,"num_object_copies":2912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1456,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":702,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,12],"acting":[8,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.25","version":"174'5965","reported_seq":11711,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-13T11:25:36.091753+0000","last_change":"2024-09-13T11:24:57.073162+0000","last_active":"2024-09-13T11:25:36.091753+0000","last_peered":"2024-09-13T11:25:36.091753+0000","last_clean":"2024-09-13T11:25:36.091753+0000","last_became_active":"2024-09-13T11:24:55.999019+0000","last_became_peered":"2024-09-13T11:24:55.999019+0000","last_unstale":"2024-09-13T11:25:36.091753+0000","last_undegraded":"2024-09-13T11:25:36.091753+0000","last_fullsized":"2024-09-13T11:25:36.091753+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":6,"last_scrub":"142'5214","last_scrub_stamp":"2024-09-13T11:24:57.073061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:24:57.073061+0000","objects_scrubbed":2249,"log_size":5965,"log_dups_size":0,"ondisk_log_size":5965,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:26:12.348519+0000","scrub_duration":571,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":98369536,"num_objects":1501,"num_object_clones":0,"num_object_copies":3002,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1501,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":754,"num_write_kb":192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,2],"acting":[12,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[]},{"pgid":"3.24","version":"170'5920","reported_seq":9844,"reported_epoch":170,"state":"active+clean","last_fresh":"2024-09-13T11:25:31.989043+0000","last_change":"2024-09-13T11:24:48.048846+0000","last_active":"2024-09-13T11:25:31.989043+0000","last_peered":"2024-09-13T11:25:31.989043+0000","last_clean":"2024-09-13T11:25:31.989043+0000","last_became_active":"2024-09-13T11:24:46.936994+0000","last_became_peered":"2024-09-13T11:24:46.936994+0000","last_unstale":"2024-09-13T11:25:31.989043+0000","last_undegraded":"2024-09-13T11:25:31.989043+0000","last_fullsized":"2024-09-13T11:25:31.989043+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":6,"last_scrub":"135'5093","last_scrub_stamp":"2024-09-13T11:24:48.048764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:24:48.048764+0000","objects_scrubbed":2102,"log_size":5920,"log_dups_size":0,"ondisk_log_size":5920,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:53.485370+0000","scrub_duration":287,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":102498304,"num_objects":1564,"num_object_clones":0,"num_object_copies":3128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1564,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":828,"num_write_kb":9280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1],"acting":[9,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.23","version":"174'5247","reported_seq":8114,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-13T11:25:35.945765+0000","last_change":"2024-09-13T11:23:48.133184+0000","last_active":"2024-09-13T11:25:35.945765+0000","last_peered":"2024-09-13T11:25:35.945765+0000","last_clean":"2024-09-13T11:25:35.945765+0000","last_became_active":"2024-09-13T11:23:47.926677+0000","last_became_peered":"2024-09-13T11:23:47.926677+0000","last_unstale":"2024-09-13T11:25:35.945765+0000","last_undegraded":"2024-09-13T11:25:35.945765+0000","last_fullsized":"2024-09-13T11:25:35.945765+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":6,"last_scrub":"91'3278","last_scrub_stamp":"2024-09-13T11:23:48.133084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:48.133084+0000","objects_scrubbed":1029,"log_size":5247,"log_dups_size":0,"ondisk_log_size":5247,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":139,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":98238464,"num_objects":1499,"num_object_clones":0,"num_object_copies":2998,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1499,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1970,"num_write_kb":78080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9],"acting":[5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.22","version":"174'5159","reported_seq":10755,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-13T11:25:35.960754+0000","last_change":"2024-09-13T11:24:45.485711+0000","last_active":"2024-09-13T11:25:35.960754+0000","last_peered":"2024-09-13T11:25:35.960754+0000","last_clean":"2024-09-13T11:25:35.960754+0000","last_became_active":"2024-09-13T11:24:29.253848+0000","last_became_peered":"2024-09-13T11:24:29.253848+0000","last_unstale":"2024-09-13T11:25:35.960754+0000","last_undegraded":"2024-09-13T11:25:35.960754+0000","last_fullsized":"2024-09-13T11:25:35.960754+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":6,"last_scrub":"90'3246","last_scrub_stamp":"2024-09-13T11:23:47.300339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:47.300339+0000","objects_scrubbed":1033,"log_size":5159,"log_dups_size":0,"ondisk_log_size":5159,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":203,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":97255424,"num_objects":1484,"num_object_clones":0,"num_object_copies":2968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1484,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1915,"num_write_kb":75712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1768,"num_bytes_recovered":115867648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.21","version":"174'5079","reported_seq":7961,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-13T11:25:36.054055+0000","last_change":"2024-09-13T11:24:12.564117+0000","last_active":"2024-09-13T11:25:36.054055+0000","last_peered":"2024-09-13T11:25:36.054055+0000","last_clean":"2024-09-13T11:25:36.054055+0000","last_became_active":"2024-09-13T11:24:01.250308+0000","last_became_peered":"2024-09-13T11:24:01.250308+0000","last_unstale":"2024-09-13T11:25:36.054055+0000","last_undegraded":"2024-09-13T11:25:36.054055+0000","last_fullsized":"2024-09-13T11:25:36.054055+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":6,"last_scrub":"112'3670","last_scrub_stamp":"2024-09-13T11:24:12.564031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:24:12.564031+0000","objects_scrubbed":1431,"log_size":5079,"log_dups_size":0,"ondisk_log_size":5079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:36.766299+0000","scrub_duration":183,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":91815936,"num_objects":1401,"num_object_clones":0,"num_object_copies":2802,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1401,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1412,"num_write_kb":44224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8],"acting":[0,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.20","version":"174'5244","reported_seq":7436,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-13T11:25:35.940585+0000","last_change":"2024-09-13T11:25:35.773483+0000","last_active":"2024-09-13T11:25:35.940585+0000","last_peered":"2024-09-13T11:25:35.940585+0000","last_clean":"2024-09-13T11:25:35.940585+0000","last_became_active":"2024-09-13T11:23:33.954946+0000","last_became_peered":"2024-09-13T11:23:33.954946+0000","last_unstale":"2024-09-13T11:25:35.940585+0000","last_undegraded":"2024-09-13T11:25:35.940585+0000","last_fullsized":"2024-09-13T11:25:35.940585+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":6,"last_scrub":"174'5240","last_scrub_stamp":"2024-09-13T11:25:35.773409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:35.773409+0000","objects_scrubbed":1424,"log_size":5244,"log_dups_size":0,"ondisk_log_size":5244,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:26:53.605828+0000","scrub_duration":215,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92930048,"num_objects":1418,"num_object_clones":0,"num_object_copies":2836,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1418,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1979,"num_write_kb":74752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,0],"acting":[12,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[]},{"pgid":"3.1f","version":"174'7463","reported_seq":19430,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-13T11:25:36.005567+0000","last_change":"2024-09-13T11:25:20.773752+0000","last_active":"2024-09-13T11:25:36.005567+0000","last_peered":"2024-09-13T11:25:36.005567+0000","last_clean":"2024-09-13T11:25:36.005567+0000","last_became_active":"2024-09-13T11:24:29.290326+0000","last_became_peered":"2024-09-13T11:24:29.290326+0000","last_unstale":"2024-09-13T11:25:36.005567+0000","last_undegraded":"2024-09-13T11:25:36.005567+0000","last_fullsized":"2024-09-13T11:25:36.005567+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":5,"last_scrub":"161'6865","last_scrub_stamp":"2024-09-13T11:25:20.773641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:20.773641+0000","objects_scrubbed":3425,"log_size":7463,"log_dups_size":0,"ondisk_log_size":7463,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:26:21.317280+0000","scrub_duration":535,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":184942592,"num_objects":2822,"num_object_clones":0,"num_object_copies":5644,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2822,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3043,"num_write_kb":102848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3306,"num_bytes_recovered":216662016,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5],"acting":[10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1e","version":"174'7708","reported_seq":18241,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-13T11:25:35.989155+0000","last_change":"2024-09-13T11:24:08.338485+0000","last_active":"2024-09-13T11:25:35.989155+0000","last_peered":"2024-09-13T11:25:35.989155+0000","last_clean":"2024-09-13T11:25:35.989155+0000","last_became_active":"2024-09-13T11:24:01.261448+0000","last_became_peered":"2024-09-13T11:24:01.261448+0000","last_unstale":"2024-09-13T11:25:35.989155+0000","last_undegraded":"2024-09-13T11:25:35.989155+0000","last_fullsized":"2024-09-13T11:25:35.989155+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":5,"last_scrub":"110'4537","last_scrub_stamp":"2024-09-13T11:24:08.338354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:24:08.338354+0000","objects_scrubbed":2760,"log_size":7708,"log_dups_size":0,"ondisk_log_size":7708,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":367,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":190054400,"num_objects":2900,"num_object_clones":0,"num_object_copies":5800,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2900,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3180,"num_write_kb":106240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"174'7502","reported_seq":12964,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-13T11:25:36.042351+0000","last_change":"2024-09-13T11:24:12.492592+0000","last_active":"2024-09-13T11:25:36.042351+0000","last_peered":"2024-09-13T11:25:36.042351+0000","last_clean":"2024-09-13T11:25:36.042351+0000","last_became_active":"2024-09-13T11:24:01.258225+0000","last_became_peered":"2024-09-13T11:24:01.258225+0000","last_unstale":"2024-09-13T11:25:36.042351+0000","last_undegraded":"2024-09-13T11:25:36.042351+0000","last_fullsized":"2024-09-13T11:25:36.042351+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":5,"last_scrub":"112'4579","last_scrub_stamp":"2024-09-13T11:24:12.492507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:24:12.492507+0000","objects_scrubbed":2875,"log_size":7502,"log_dups_size":0,"ondisk_log_size":7502,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":375,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":189792280,"num_objects":2897,"num_object_clones":0,"num_object_copies":5794,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2897,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2936,"num_write_kb":94593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,15],"acting":[2,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1c","version":"170'7480","reported_seq":11291,"reported_epoch":170,"state":"active+clean","last_fresh":"2024-09-13T11:25:32.078405+0000","last_change":"2024-09-13T11:24:00.529312+0000","last_active":"2024-09-13T11:25:32.078405+0000","last_peered":"2024-09-13T11:25:32.078405+0000","last_clean":"2024-09-13T11:25:32.078405+0000","last_became_active":"2024-09-13T11:23:33.989492+0000","last_became_peered":"2024-09-13T11:23:33.989492+0000","last_unstale":"2024-09-13T11:25:32.078405+0000","last_undegraded":"2024-09-13T11:25:32.078405+0000","last_fullsized":"2024-09-13T11:25:32.078405+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":5,"last_scrub":"102'4277","last_scrub_stamp":"2024-09-13T11:24:00.529243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:24:00.529243+0000","objects_scrubbed":2544,"log_size":7480,"log_dups_size":0,"ondisk_log_size":7480,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:37.048397+0000","scrub_duration":259,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":201785344,"num_objects":3079,"num_object_clones":0,"num_object_copies":6158,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3079,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3205,"num_write_kb":119680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7],"acting":[11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1b","version":"174'7639","reported_seq":13280,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-13T11:25:36.109056+0000","last_change":"2024-09-13T11:24:12.392093+0000","last_active":"2024-09-13T11:25:36.109056+0000","last_peered":"2024-09-13T11:25:36.109056+0000","last_clean":"2024-09-13T11:25:36.109056+0000","last_became_active":"2024-09-13T11:24:04.950821+0000","last_became_peered":"2024-09-13T11:24:04.950821+0000","last_unstale":"2024-09-13T11:25:36.109056+0000","last_undegraded":"2024-09-13T11:25:36.109056+0000","last_fullsized":"2024-09-13T11:25:36.109056+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":5,"last_scrub":"112'4665","last_scrub_stamp":"2024-09-13T11:24:12.391890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:24:12.391890+0000","objects_scrubbed":2934,"log_size":7639,"log_dups_size":0,"ondisk_log_size":7639,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:41.846147+0000","scrub_duration":675,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":189857792,"num_objects":2897,"num_object_clones":0,"num_object_copies":5794,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2897,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2995,"num_write_kb":94656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,15],"acting":[12,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[]},{"pgid":"3.1a","version":"174'7491","reported_seq":12793,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-13T11:25:36.043187+0000","last_change":"2024-09-13T11:24:09.556174+0000","last_active":"2024-09-13T11:25:36.043187+0000","last_peered":"2024-09-13T11:25:36.043187+0000","last_clean":"2024-09-13T11:25:36.043187+0000","last_became_active":"2024-09-13T11:24:03.935185+0000","last_became_peered":"2024-09-13T11:24:03.935185+0000","last_unstale":"2024-09-13T11:25:36.043187+0000","last_undegraded":"2024-09-13T11:25:36.043187+0000","last_fullsized":"2024-09-13T11:25:36.043187+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":5,"last_scrub":"110'4433","last_scrub_stamp":"2024-09-13T11:24:09.556022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:24:09.556022+0000","objects_scrubbed":2654,"log_size":7491,"log_dups_size":0,"ondisk_log_size":7491,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:36.486646+0000","scrub_duration":387,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":186712064,"num_objects":2849,"num_object_clones":0,"num_object_copies":5698,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2849,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3069,"num_write_kb":104448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9],"acting":[4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.19","version":"170'6950","reported_seq":11431,"reported_epoch":170,"state":"active+clean","last_fresh":"2024-09-13T11:25:31.961853+0000","last_change":"2024-09-13T11:25:10.784789+0000","last_active":"2024-09-13T11:25:31.961853+0000","last_peered":"2024-09-13T11:25:31.961853+0000","last_clean":"2024-09-13T11:25:31.961853+0000","last_became_active":"2024-09-13T11:23:41.924941+0000","last_became_peered":"2024-09-13T11:23:41.924941+0000","last_unstale":"2024-09-13T11:25:31.961853+0000","last_undegraded":"2024-09-13T11:25:31.961853+0000","last_fullsized":"2024-09-13T11:25:31.961853+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":5,"last_scrub":"151'6156","last_scrub_stamp":"2024-09-13T11:25:10.784716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:10.784716+0000","objects_scrubbed":3781,"log_size":6950,"log_dups_size":0,"ondisk_log_size":6950,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:26:30.390310+0000","scrub_duration":1131,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":194576384,"num_objects":2969,"num_object_clones":0,"num_object_copies":5938,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2969,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3548,"num_write_kb":145216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1],"acting":[9,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.18","version":"174'7232","reported_seq":11982,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-13T11:25:36.068112+0000","last_change":"2024-09-13T11:24:13.454659+0000","last_active":"2024-09-13T11:25:36.068112+0000","last_peered":"2024-09-13T11:25:36.068112+0000","last_clean":"2024-09-13T11:25:36.068112+0000","last_became_active":"2024-09-13T11:23:48.926014+0000","last_became_peered":"2024-09-13T11:23:48.926014+0000","last_unstale":"2024-09-13T11:25:36.068112+0000","last_undegraded":"2024-09-13T11:25:36.068112+0000","last_fullsized":"2024-09-13T11:25:36.068112+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":5,"last_scrub":"112'4376","last_scrub_stamp":"2024-09-13T11:24:13.454575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:24:13.454575+0000","objects_scrubbed":3036,"log_size":7232,"log_dups_size":0,"ondisk_log_size":7232,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":375,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":188416000,"num_objects":2875,"num_object_clones":0,"num_object_copies":5750,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2875,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2865,"num_write_kb":86528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0],"acting":[10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.17","version":"174'7116","reported_seq":11338,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-13T11:25:36.055838+0000","last_change":"2024-09-13T11:25:12.658562+0000","last_active":"2024-09-13T11:25:36.055838+0000","last_peered":"2024-09-13T11:25:36.055838+0000","last_clean":"2024-09-13T11:25:36.055838+0000","last_became_active":"2024-09-13T11:23:42.935601+0000","last_became_peered":"2024-09-13T11:23:42.935601+0000","last_unstale":"2024-09-13T11:25:36.055838+0000","last_undegraded":"2024-09-13T11:25:36.055838+0000","last_fullsized":"2024-09-13T11:25:36.055838+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":5,"last_scrub":"153'6197","last_scrub_stamp":"2024-09-13T11:25:12.658464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:12.658464+0000","objects_scrubbed":3743,"log_size":7116,"log_dups_size":0,"ondisk_log_size":7116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:26:25.625277+0000","scrub_duration":467,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":185008128,"num_objects":2823,"num_object_clones":0,"num_object_copies":5646,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2823,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3765,"num_write_kb":146688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3],"acting":[10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"170'7270","reported_seq":12120,"reported_epoch":170,"state":"active+clean","last_fresh":"2024-09-13T11:25:32.071492+0000","last_change":"2024-09-13T11:25:01.903487+0000","last_active":"2024-09-13T11:25:32.071492+0000","last_peered":"2024-09-13T11:25:32.071492+0000","last_clean":"2024-09-13T11:25:32.071492+0000","last_became_active":"2024-09-13T11:23:46.105237+0000","last_became_peered":"2024-09-13T11:23:46.105237+0000","last_unstale":"2024-09-13T11:25:32.071492+0000","last_undegraded":"2024-09-13T11:25:32.071492+0000","last_fullsized":"2024-09-13T11:25:32.071492+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":5,"last_scrub":"147'6035","last_scrub_stamp":"2024-09-13T11:25:01.903395+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:01.903395+0000","objects_scrubbed":4359,"log_size":7270,"log_dups_size":0,"ondisk_log_size":7270,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:26:16.436377+0000","scrub_duration":743,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":203751424,"num_objects":3109,"num_object_clones":0,"num_object_copies":6218,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3109,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3795,"num_write_kb":155200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,15],"acting":[11,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.15","version":"174'7111","reported_seq":18452,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-13T11:25:36.097434+0000","last_change":"2024-09-13T11:25:12.198448+0000","last_active":"2024-09-13T11:25:36.097434+0000","last_peered":"2024-09-13T11:25:36.097434+0000","last_clean":"2024-09-13T11:25:36.097434+0000","last_became_active":"2024-09-13T11:24:29.226615+0000","last_became_peered":"2024-09-13T11:24:29.226615+0000","last_unstale":"2024-09-13T11:25:36.097434+0000","last_undegraded":"2024-09-13T11:25:36.097434+0000","last_fullsized":"2024-09-13T11:25:36.097434+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":5,"last_scrub":"91'3384","last_scrub_stamp":"2024-09-13T11:23:48.332600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:48.332600+0000","objects_scrubbed":2032,"log_size":7111,"log_dups_size":0,"ondisk_log_size":7111,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":235,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":184418304,"num_objects":2814,"num_object_clones":0,"num_object_copies":5628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2814,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3728,"num_write_kb":144320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3385,"num_bytes_recovered":221839360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.14","version":"173'7203","reported_seq":18468,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-09-13T11:25:34.992634+0000","last_change":"2024-09-13T11:25:15.378521+0000","last_active":"2024-09-13T11:25:34.992634+0000","last_peered":"2024-09-13T11:25:34.992634+0000","last_clean":"2024-09-13T11:25:34.992634+0000","last_became_active":"2024-09-13T11:24:29.253106+0000","last_became_peered":"2024-09-13T11:24:29.253106+0000","last_unstale":"2024-09-13T11:25:34.992634+0000","last_undegraded":"2024-09-13T11:25:34.992634+0000","last_fullsized":"2024-09-13T11:25:34.992634+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":5,"last_scrub":"91'3371","last_scrub_stamp":"2024-09-13T11:23:48.696960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:48.696960+0000","objects_scrubbed":1935,"log_size":7203,"log_dups_size":0,"ondisk_log_size":7203,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":735,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":199819264,"num_objects":3049,"num_object_clones":0,"num_object_copies":6098,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3049,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3842,"num_write_kb":158592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3437,"num_bytes_recovered":225247232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,15],"acting":[3,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":320,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-13T11:25:35.285144+0000","last_change":"2024-09-13T11:24:23.212494+0000","last_active":"2024-09-13T11:25:35.285144+0000","last_peered":"2024-09-13T11:25:35.285144+0000","last_clean":"2024-09-13T11:25:35.285144+0000","last_became_active":"2024-09-13T11:22:14.192897+0000","last_became_peered":"2024-09-13T11:22:14.192897+0000","last_unstale":"2024-09-13T11:25:35.285144+0000","last_undegraded":"2024-09-13T11:25:35.285144+0000","last_fullsized":"2024-09-13T11:25:35.285144+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:24:23.212406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:24:23.212406+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:36.980197+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00084655899999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,10],"acting":[15,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"3.6","version":"170'6056","reported_seq":9927,"reported_epoch":170,"state":"active+clean","last_fresh":"2024-09-13T11:25:31.914698+0000","last_change":"2024-09-13T11:24:49.416851+0000","last_active":"2024-09-13T11:25:31.914698+0000","last_peered":"2024-09-13T11:25:31.914698+0000","last_clean":"2024-09-13T11:25:31.914698+0000","last_became_active":"2024-09-13T11:24:29.322558+0000","last_became_peered":"2024-09-13T11:24:29.322558+0000","last_unstale":"2024-09-13T11:25:31.914698+0000","last_undegraded":"2024-09-13T11:25:31.914698+0000","last_fullsized":"2024-09-13T11:25:31.914698+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"136'5216","last_scrub_stamp":"2024-09-13T11:24:49.416706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:24:49.416706+0000","objects_scrubbed":2083,"log_size":6056,"log_dups_size":0,"ondisk_log_size":6056,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:26:09.525086+0000","scrub_duration":247,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":100663296,"num_objects":1536,"num_object_clones":0,"num_object_copies":3072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1536,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":841,"num_write_kb":9408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,6],"acting":[13,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":319,"reported_epoch":170,"state":"active+clean","last_fresh":"2024-09-13T11:25:30.290690+0000","last_change":"2024-09-13T11:24:59.556253+0000","last_active":"2024-09-13T11:25:30.290690+0000","last_peered":"2024-09-13T11:25:30.290690+0000","last_clean":"2024-09-13T11:25:30.290690+0000","last_became_active":"2024-09-13T11:22:14.211923+0000","last_became_peered":"2024-09-13T11:22:14.211923+0000","last_unstale":"2024-09-13T11:25:30.290690+0000","last_undegraded":"2024-09-13T11:25:30.290690+0000","last_fullsized":"2024-09-13T11:25:30.290690+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:24:59.556180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:24:59.556180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:26:06.229322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011689529999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0],"acting":[9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"170'5865","reported_seq":6230,"reported_epoch":170,"state":"active+clean","last_fresh":"2024-09-13T11:25:32.025740+0000","last_change":"2024-09-13T11:24:33.336210+0000","last_active":"2024-09-13T11:25:32.025740+0000","last_peered":"2024-09-13T11:25:32.025740+0000","last_clean":"2024-09-13T11:25:32.025740+0000","last_became_active":"2024-09-13T11:24:24.092975+0000","last_became_peered":"2024-09-13T11:24:24.092975+0000","last_unstale":"2024-09-13T11:25:32.025740+0000","last_undegraded":"2024-09-13T11:25:32.025740+0000","last_fullsized":"2024-09-13T11:25:32.025740+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"125'4820","last_scrub_stamp":"2024-09-13T11:24:33.336091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:24:33.336091+0000","objects_scrubbed":1766,"log_size":5865,"log_dups_size":0,"ondisk_log_size":5865,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:54.538352+0000","scrub_duration":263,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":101122048,"num_objects":1543,"num_object_clones":0,"num_object_copies":3086,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1543,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1047,"num_write_kb":26368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,8],"acting":[13,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":328,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-13T11:25:35.283473+0000","last_change":"2024-09-13T11:25:33.957283+0000","last_active":"2024-09-13T11:25:35.283473+0000","last_peered":"2024-09-13T11:25:35.283473+0000","last_clean":"2024-09-13T11:25:35.283473+0000","last_became_active":"2024-09-13T11:22:14.581458+0000","last_became_peered":"2024-09-13T11:22:14.581458+0000","last_unstale":"2024-09-13T11:25:35.283473+0000","last_undegraded":"2024-09-13T11:25:35.283473+0000","last_fullsized":"2024-09-13T11:25:35.283473+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:25:33.957179+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:25:33.957179+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:26:52.126360+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.001026047,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5],"acting":[8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"174'5998","reported_seq":6343,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-13T11:25:35.354435+0000","last_change":"2024-09-13T11:24:24.579036+0000","last_active":"2024-09-13T11:25:35.354435+0000","last_peered":"2024-09-13T11:25:35.354435+0000","last_clean":"2024-09-13T11:25:35.354435+0000","last_became_active":"2024-09-13T11:24:24.083077+0000","last_became_peered":"2024-09-13T11:24:24.083077+0000","last_unstale":"2024-09-13T11:25:35.354435+0000","last_undegraded":"2024-09-13T11:25:35.354435+0000","last_fullsized":"2024-09-13T11:25:35.354435+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'4737","last_scrub_stamp":"2024-09-13T11:24:24.578900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:24:24.578900+0000","objects_scrubbed":1690,"log_size":5998,"log_dups_size":0,"ondisk_log_size":5998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:38.103421+0000","scrub_duration":407,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":94896128,"num_objects":1448,"num_object_clones":0,"num_object_copies":2896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1448,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1264,"num_write_kb":32704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,12],"acting":[1,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":308,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-13T11:25:35.298684+0000","last_change":"2024-09-13T11:24:08.271026+0000","last_active":"2024-09-13T11:25:35.298684+0000","last_peered":"2024-09-13T11:25:35.298684+0000","last_clean":"2024-09-13T11:25:35.298684+0000","last_became_active":"2024-09-13T11:22:47.719636+0000","last_became_peered":"2024-09-13T11:22:47.719636+0000","last_unstale":"2024-09-13T11:25:35.298684+0000","last_undegraded":"2024-09-13T11:25:35.298684+0000","last_fullsized":"2024-09-13T11:25:35.298684+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:24:08.270972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:24:08.270972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:36.805605+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00087476000000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,9],"acting":[1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"174'5958","reported_seq":6322,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-13T11:25:36.019857+0000","last_change":"2024-09-13T11:24:24.616937+0000","last_active":"2024-09-13T11:25:36.019857+0000","last_peered":"2024-09-13T11:25:36.019857+0000","last_clean":"2024-09-13T11:25:36.019857+0000","last_became_active":"2024-09-13T11:24:24.093492+0000","last_became_peered":"2024-09-13T11:24:24.093492+0000","last_unstale":"2024-09-13T11:25:36.019857+0000","last_undegraded":"2024-09-13T11:25:36.019857+0000","last_fullsized":"2024-09-13T11:25:36.019857+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'4744","last_scrub_stamp":"2024-09-13T11:24:24.616633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:24:24.616633+0000","objects_scrubbed":1624,"log_size":5958,"log_dups_size":0,"ondisk_log_size":5958,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:42.911074+0000","scrub_duration":415,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92733440,"num_objects":1415,"num_object_clones":0,"num_object_copies":2830,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1415,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1221,"num_write_kb":32384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,13],"acting":[15,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"3.3","version":"174'5160","reported_seq":5544,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-13T11:25:36.042110+0000","last_change":"2024-09-13T11:25:08.394545+0000","last_active":"2024-09-13T11:25:36.042110+0000","last_peered":"2024-09-13T11:25:36.042110+0000","last_clean":"2024-09-13T11:25:36.042110+0000","last_became_active":"2024-09-13T11:23:31.930448+0000","last_became_peered":"2024-09-13T11:23:31.930448+0000","last_unstale":"2024-09-13T11:25:36.042110+0000","last_undegraded":"2024-09-13T11:25:36.042110+0000","last_fullsized":"2024-09-13T11:25:36.042110+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"151'4587","last_scrub_stamp":"2024-09-13T11:25:08.394441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:08.394441+0000","objects_scrubbed":1942,"log_size":5160,"log_dups_size":0,"ondisk_log_size":5160,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:26:27.641900+0000","scrub_duration":219,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":89522176,"num_objects":1366,"num_object_clones":0,"num_object_copies":2732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1366,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2038,"num_write_kb":80896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,7],"acting":[10,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"28'2","reported_seq":258,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-13T11:25:35.304129+0000","last_change":"2024-09-13T11:25:13.869564+0000","last_active":"2024-09-13T11:25:35.304129+0000","last_peered":"2024-09-13T11:25:35.304129+0000","last_clean":"2024-09-13T11:25:35.304129+0000","last_became_active":"2024-09-13T11:23:12.546348+0000","last_became_peered":"2024-09-13T11:23:12.546348+0000","last_unstale":"2024-09-13T11:25:35.304129+0000","last_undegraded":"2024-09-13T11:25:35.304129+0000","last_fullsized":"2024-09-13T11:25:35.304129+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2","last_scrub_stamp":"2024-09-13T11:25:13.869413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:25:13.869413+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:26:38.761718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0025826730000000002,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8],"acting":[2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"174'5170","reported_seq":5552,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-13T11:25:36.073635+0000","last_change":"2024-09-13T11:25:09.842875+0000","last_active":"2024-09-13T11:25:36.073635+0000","last_peered":"2024-09-13T11:25:36.073635+0000","last_clean":"2024-09-13T11:25:36.073635+0000","last_became_active":"2024-09-13T11:23:31.921388+0000","last_became_peered":"2024-09-13T11:23:31.921388+0000","last_unstale":"2024-09-13T11:25:36.073635+0000","last_undegraded":"2024-09-13T11:25:36.073635+0000","last_fullsized":"2024-09-13T11:25:36.073635+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"151'4644","last_scrub_stamp":"2024-09-13T11:25:09.842728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:09.842728+0000","objects_scrubbed":1944,"log_size":5170,"log_dups_size":0,"ondisk_log_size":5170,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:26:25.553642+0000","scrub_duration":395,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92930048,"num_objects":1418,"num_object_clones":0,"num_object_copies":2836,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1418,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2064,"num_write_kb":84032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,15],"acting":[12,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":169,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-13T11:25:35.298317+0000","last_change":"2024-09-13T11:25:34.822952+0000","last_active":"2024-09-13T11:25:35.298317+0000","last_peered":"2024-09-13T11:25:35.298317+0000","last_clean":"2024-09-13T11:25:35.298317+0000","last_became_active":"2024-09-13T11:24:29.096575+0000","last_became_peered":"2024-09-13T11:24:29.096575+0000","last_unstale":"2024-09-13T11:25:35.298317+0000","last_undegraded":"2024-09-13T11:25:35.298317+0000","last_fullsized":"2024-09-13T11:25:35.298317+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:25:34.822831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:25:34.822831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:27:01.304133+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00081225399999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"170'5051","reported_seq":5428,"reported_epoch":170,"state":"active+clean","last_fresh":"2024-09-13T11:25:32.076900+0000","last_change":"2024-09-13T11:25:00.861990+0000","last_active":"2024-09-13T11:25:32.076900+0000","last_peered":"2024-09-13T11:25:32.076900+0000","last_clean":"2024-09-13T11:25:32.076900+0000","last_became_active":"2024-09-13T11:23:31.926528+0000","last_became_peered":"2024-09-13T11:23:31.926528+0000","last_unstale":"2024-09-13T11:25:32.076900+0000","last_undegraded":"2024-09-13T11:25:32.076900+0000","last_fullsized":"2024-09-13T11:25:32.076900+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"146'4447","last_scrub_stamp":"2024-09-13T11:25:00.861834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:00.861834+0000","objects_scrubbed":2126,"log_size":5051,"log_dups_size":0,"ondisk_log_size":5051,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:26:15.411903+0000","scrub_duration":787,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":99549184,"num_objects":1519,"num_object_clones":0,"num_object_copies":3038,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1519,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1981,"num_write_kb":85056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,11],"acting":[13,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":318,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-09-13T11:25:34.282870+0000","last_change":"2024-09-13T11:24:11.895017+0000","last_active":"2024-09-13T11:25:34.282870+0000","last_peered":"2024-09-13T11:25:34.282870+0000","last_clean":"2024-09-13T11:25:34.282870+0000","last_became_active":"2024-09-13T11:22:14.211435+0000","last_became_peered":"2024-09-13T11:22:14.211435+0000","last_unstale":"2024-09-13T11:25:34.282870+0000","last_undegraded":"2024-09-13T11:25:34.282870+0000","last_fullsized":"2024-09-13T11:25:34.282870+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:24:11.894937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:24:11.894937+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0010010360000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2","version":"174'5195","reported_seq":9042,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-13T11:25:35.957500+0000","last_change":"2024-09-13T11:25:23.765128+0000","last_active":"2024-09-13T11:25:35.957500+0000","last_peered":"2024-09-13T11:25:35.957500+0000","last_clean":"2024-09-13T11:25:35.957500+0000","last_became_active":"2024-09-13T11:24:29.175407+0000","last_became_peered":"2024-09-13T11:24:29.175407+0000","last_unstale":"2024-09-13T11:25:35.957500+0000","last_undegraded":"2024-09-13T11:25:35.957500+0000","last_fullsized":"2024-09-13T11:25:35.957500+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"164'4994","last_scrub_stamp":"2024-09-13T11:25:23.765047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:23.765047+0000","objects_scrubbed":1689,"log_size":5195,"log_dups_size":0,"ondisk_log_size":5195,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:26:45.252942+0000","scrub_duration":283,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":97320960,"num_objects":1485,"num_object_clones":0,"num_object_copies":2970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1485,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2138,"num_write_kb":89024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1731,"num_bytes_recovered":113442816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,3],"acting":[12,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[]},{"pgid":"2.3","version":"26'1","reported_seq":131,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-13T11:25:35.286688+0000","last_change":"2024-09-13T11:25:23.034357+0000","last_active":"2024-09-13T11:25:35.286688+0000","last_peered":"2024-09-13T11:25:35.286688+0000","last_clean":"2024-09-13T11:25:35.286688+0000","last_became_active":"2024-09-13T11:24:29.110028+0000","last_became_peered":"2024-09-13T11:24:29.110028+0000","last_unstale":"2024-09-13T11:25:35.286688+0000","last_undegraded":"2024-09-13T11:25:35.286688+0000","last_fullsized":"2024-09-13T11:25:35.286688+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"26'1","last_scrub_stamp":"2024-09-13T11:25:23.034247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:25:23.034247+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:26:42.429349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,8],"acting":[15,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"1.0","version":"26'112","reported_seq":462,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-13T11:25:35.310912+0000","last_change":"2024-09-13T11:25:23.642366+0000","last_active":"2024-09-13T11:25:35.310912+0000","last_peered":"2024-09-13T11:25:35.310912+0000","last_clean":"2024-09-13T11:25:35.310912+0000","last_became_active":"2024-09-13T11:22:12.129721+0000","last_became_peered":"2024-09-13T11:22:12.129721+0000","last_unstale":"2024-09-13T11:25:35.310912+0000","last_undegraded":"2024-09-13T11:25:35.310912+0000","last_fullsized":"2024-09-13T11:25:35.310912+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"26'112","last_scrub_stamp":"2024-09-13T11:25:23.642204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:11.098388+0000","last_clean_scrub_stamp":"2024-09-13T11:25:23.642204+0000","objects_scrubbed":2,"log_size":112,"log_dups_size":0,"ondisk_log_size":112,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:26:42.628319+0000","scrub_duration":67,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.8","version":"174'5968","reported_seq":6331,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-13T11:25:35.943739+0000","last_change":"2024-09-13T11:24:24.506212+0000","last_active":"2024-09-13T11:25:35.943739+0000","last_peered":"2024-09-13T11:25:35.943739+0000","last_clean":"2024-09-13T11:25:35.943739+0000","last_became_active":"2024-09-13T11:24:24.096220+0000","last_became_peered":"2024-09-13T11:24:24.096220+0000","last_unstale":"2024-09-13T11:25:35.943739+0000","last_undegraded":"2024-09-13T11:25:35.943739+0000","last_fullsized":"2024-09-13T11:25:35.943739+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'4749","last_scrub_stamp":"2024-09-13T11:24:24.506109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:24:24.506109+0000","objects_scrubbed":1690,"log_size":5968,"log_dups_size":0,"ondisk_log_size":5968,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:49.430042+0000","scrub_duration":215,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":94765056,"num_objects":1446,"num_object_clones":0,"num_object_copies":2892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1446,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1220,"num_write_kb":31232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.9","version":"174'6001","reported_seq":6365,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-13T11:25:35.990096+0000","last_change":"2024-09-13T11:24:24.928964+0000","last_active":"2024-09-13T11:25:35.990096+0000","last_peered":"2024-09-13T11:25:35.990096+0000","last_clean":"2024-09-13T11:25:35.990096+0000","last_became_active":"2024-09-13T11:24:24.082504+0000","last_became_peered":"2024-09-13T11:24:24.082504+0000","last_unstale":"2024-09-13T11:25:35.990096+0000","last_undegraded":"2024-09-13T11:25:35.990096+0000","last_fullsized":"2024-09-13T11:25:35.990096+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'4799","last_scrub_stamp":"2024-09-13T11:24:24.928724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:24:24.928724+0000","objects_scrubbed":1705,"log_size":6001,"log_dups_size":0,"ondisk_log_size":6001,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:43.845801+0000","scrub_duration":255,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96468992,"num_objects":1472,"num_object_clones":0,"num_object_copies":2944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1472,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1205,"num_write_kb":31104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,2],"acting":[12,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[]},{"pgid":"3.a","version":"173'6337","reported_seq":6703,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-09-13T11:25:34.984376+0000","last_change":"2024-09-13T11:24:25.137871+0000","last_active":"2024-09-13T11:25:34.984376+0000","last_peered":"2024-09-13T11:25:34.984376+0000","last_clean":"2024-09-13T11:25:34.984376+0000","last_became_active":"2024-09-13T11:24:24.089253+0000","last_became_peered":"2024-09-13T11:24:24.089253+0000","last_unstale":"2024-09-13T11:25:34.984376+0000","last_undegraded":"2024-09-13T11:25:34.984376+0000","last_fullsized":"2024-09-13T11:25:34.984376+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'5157","last_scrub_stamp":"2024-09-13T11:24:25.137777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:24:25.137777+0000","objects_scrubbed":1777,"log_size":6337,"log_dups_size":0,"ondisk_log_size":6337,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:36.149013+0000","scrub_duration":223,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96206848,"num_objects":1468,"num_object_clones":0,"num_object_copies":2936,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1468,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1183,"num_write_kb":27968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.b","version":"170'6256","reported_seq":6616,"reported_epoch":170,"state":"active+clean","last_fresh":"2024-09-13T11:25:32.023235+0000","last_change":"2024-09-13T11:24:25.503864+0000","last_active":"2024-09-13T11:25:32.023235+0000","last_peered":"2024-09-13T11:25:32.023235+0000","last_clean":"2024-09-13T11:25:32.023235+0000","last_became_active":"2024-09-13T11:24:24.085684+0000","last_became_peered":"2024-09-13T11:24:24.085684+0000","last_unstale":"2024-09-13T11:25:32.023235+0000","last_undegraded":"2024-09-13T11:25:32.023235+0000","last_fullsized":"2024-09-13T11:25:32.023235+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'5112","last_scrub_stamp":"2024-09-13T11:24:25.503716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:24:25.503716+0000","objects_scrubbed":1718,"log_size":6256,"log_dups_size":0,"ondisk_log_size":6256,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:37.636639+0000","scrub_duration":395,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":100401152,"num_objects":1532,"num_object_clones":0,"num_object_copies":3064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1532,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1144,"num_write_kb":30656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,1],"acting":[13,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"3.c","version":"170'6255","reported_seq":6614,"reported_epoch":170,"state":"active+clean","last_fresh":"2024-09-13T11:25:32.014973+0000","last_change":"2024-09-13T11:24:24.595021+0000","last_active":"2024-09-13T11:25:32.014973+0000","last_peered":"2024-09-13T11:25:32.014973+0000","last_clean":"2024-09-13T11:25:32.014973+0000","last_became_active":"2024-09-13T11:24:24.090847+0000","last_became_peered":"2024-09-13T11:24:24.090847+0000","last_unstale":"2024-09-13T11:25:32.014973+0000","last_undegraded":"2024-09-13T11:25:32.014973+0000","last_fullsized":"2024-09-13T11:25:32.014973+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'5154","last_scrub_stamp":"2024-09-13T11:24:24.594848+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:24:24.594848+0000","objects_scrubbed":1720,"log_size":6255,"log_dups_size":0,"ondisk_log_size":6255,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:35.181590+0000","scrub_duration":343,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":100532224,"num_objects":1534,"num_object_clones":0,"num_object_copies":3068,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1534,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1106,"num_write_kb":29440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,13],"acting":[11,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.d","version":"170'6234","reported_seq":6593,"reported_epoch":170,"state":"active+clean","last_fresh":"2024-09-13T11:25:31.170448+0000","last_change":"2024-09-13T11:24:24.967444+0000","last_active":"2024-09-13T11:25:31.170448+0000","last_peered":"2024-09-13T11:25:31.170448+0000","last_clean":"2024-09-13T11:25:31.170448+0000","last_became_active":"2024-09-13T11:24:24.087224+0000","last_became_peered":"2024-09-13T11:24:24.087224+0000","last_unstale":"2024-09-13T11:25:31.170448+0000","last_undegraded":"2024-09-13T11:25:31.170448+0000","last_fullsized":"2024-09-13T11:25:31.170448+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'5110","last_scrub_stamp":"2024-09-13T11:24:24.967337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:24:24.967337+0000","objects_scrubbed":1725,"log_size":6234,"log_dups_size":0,"ondisk_log_size":6234,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:53.620949+0000","scrub_duration":255,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":102498304,"num_objects":1564,"num_object_clones":0,"num_object_copies":3128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1564,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1127,"num_write_kb":30912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.e","version":"174'7669","reported_seq":12933,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-13T11:25:36.052006+0000","last_change":"2024-09-13T11:23:32.368856+0000","last_active":"2024-09-13T11:25:36.052006+0000","last_peered":"2024-09-13T11:25:36.052006+0000","last_clean":"2024-09-13T11:25:36.052006+0000","last_became_active":"2024-09-13T11:23:31.925869+0000","last_became_peered":"2024-09-13T11:23:31.925869+0000","last_unstale":"2024-09-13T11:25:36.052006+0000","last_undegraded":"2024-09-13T11:25:36.052006+0000","last_fullsized":"2024-09-13T11:25:36.052006+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'3527","last_scrub_stamp":"2024-09-13T11:23:32.368727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:32.368727+0000","objects_scrubbed":1774,"log_size":7669,"log_dups_size":0,"ondisk_log_size":7669,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":219,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":184483840,"num_objects":2815,"num_object_clones":0,"num_object_copies":5630,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2815,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4143,"num_write_kb":165888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"170'7488","reported_seq":7867,"reported_epoch":170,"state":"active+clean","last_fresh":"2024-09-13T11:25:32.067943+0000","last_change":"2024-09-13T11:25:00.289940+0000","last_active":"2024-09-13T11:25:32.067943+0000","last_peered":"2024-09-13T11:25:32.067943+0000","last_clean":"2024-09-13T11:25:32.067943+0000","last_became_active":"2024-09-13T11:23:31.935239+0000","last_became_peered":"2024-09-13T11:23:31.935239+0000","last_unstale":"2024-09-13T11:25:32.067943+0000","last_undegraded":"2024-09-13T11:25:32.067943+0000","last_fullsized":"2024-09-13T11:25:32.067943+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"146'6154","last_scrub_stamp":"2024-09-13T11:25:00.289673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:00.289673+0000","objects_scrubbed":4394,"log_size":7488,"log_dups_size":0,"ondisk_log_size":7488,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:26:24.274856+0000","scrub_duration":1187,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":199098368,"num_objects":3038,"num_object_clones":0,"num_object_copies":6076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3038,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4068,"num_write_kb":170560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,12],"acting":[11,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.10","version":"174'7299","reported_seq":11837,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-13T11:25:36.026699+0000","last_change":"2024-09-13T11:25:10.161564+0000","last_active":"2024-09-13T11:25:36.026699+0000","last_peered":"2024-09-13T11:25:36.026699+0000","last_clean":"2024-09-13T11:25:36.026699+0000","last_became_active":"2024-09-13T11:23:41.931183+0000","last_became_peered":"2024-09-13T11:23:41.931183+0000","last_unstale":"2024-09-13T11:25:36.026699+0000","last_undegraded":"2024-09-13T11:25:36.026699+0000","last_fullsized":"2024-09-13T11:25:36.026699+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":5,"last_scrub":"151'6298","last_scrub_stamp":"2024-09-13T11:25:10.161464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:10.161464+0000","objects_scrubbed":3964,"log_size":7299,"log_dups_size":0,"ondisk_log_size":7299,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:26:33.015478+0000","scrub_duration":607,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":193200128,"num_objects":2948,"num_object_clones":0,"num_object_copies":5896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2948,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3833,"num_write_kb":152000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.11","version":"174'7300","reported_seq":10611,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-13T11:25:36.022837+0000","last_change":"2024-09-13T11:25:24.287326+0000","last_active":"2024-09-13T11:25:36.022837+0000","last_peered":"2024-09-13T11:25:36.022837+0000","last_clean":"2024-09-13T11:25:36.022837+0000","last_became_active":"2024-09-13T11:23:39.063172+0000","last_became_peered":"2024-09-13T11:23:39.063172+0000","last_unstale":"2024-09-13T11:25:36.022837+0000","last_undegraded":"2024-09-13T11:25:36.022837+0000","last_fullsized":"2024-09-13T11:25:36.022837+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":5,"last_scrub":"165'6821","last_scrub_stamp":"2024-09-13T11:25:24.287082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:24.287082+0000","objects_scrubbed":3406,"log_size":7300,"log_dups_size":0,"ondisk_log_size":7300,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:26:49.405332+0000","scrub_duration":947,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":190971904,"num_objects":2914,"num_object_clones":0,"num_object_copies":5828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2914,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3959,"num_write_kb":155584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11],"acting":[4,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.12","version":"170'7086","reported_seq":17200,"reported_epoch":170,"state":"active+clean","last_fresh":"2024-09-13T11:25:31.168446+0000","last_change":"2024-09-13T11:25:26.257543+0000","last_active":"2024-09-13T11:25:31.168446+0000","last_peered":"2024-09-13T11:25:31.168446+0000","last_clean":"2024-09-13T11:25:31.168446+0000","last_became_active":"2024-09-13T11:24:29.268229+0000","last_became_peered":"2024-09-13T11:24:29.268229+0000","last_unstale":"2024-09-13T11:25:31.168446+0000","last_undegraded":"2024-09-13T11:25:31.168446+0000","last_fullsized":"2024-09-13T11:25:31.168446+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'3109","last_scrub_stamp":"2024-09-13T11:23:35.111320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:35.111320+0000","objects_scrubbed":599,"log_size":7086,"log_dups_size":0,"ondisk_log_size":7086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:31.236137+0000","scrub_duration":239,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":204537856,"num_objects":3121,"num_object_clones":0,"num_object_copies":6242,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3121,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3977,"num_write_kb":171712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3457,"num_bytes_recovered":226557952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.13","version":"170'7142","reported_seq":10379,"reported_epoch":170,"state":"active+clean","last_fresh":"2024-09-13T11:25:32.046674+0000","last_change":"2024-09-13T11:25:01.153044+0000","last_active":"2024-09-13T11:25:32.046674+0000","last_peered":"2024-09-13T11:25:32.046674+0000","last_clean":"2024-09-13T11:25:32.046674+0000","last_became_active":"2024-09-13T11:23:41.082433+0000","last_became_peered":"2024-09-13T11:23:41.082433+0000","last_unstale":"2024-09-13T11:25:32.046674+0000","last_undegraded":"2024-09-13T11:25:32.046674+0000","last_fullsized":"2024-09-13T11:25:32.046674+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":5,"last_scrub":"147'5921","last_scrub_stamp":"2024-09-13T11:25:01.152971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:01.152971+0000","objects_scrubbed":4311,"log_size":7142,"log_dups_size":0,"ondisk_log_size":7142,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:26:07.027631+0000","scrub_duration":1011,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":199819264,"num_objects":3049,"num_object_clones":0,"num_object_copies":6098,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3049,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3746,"num_write_kb":154048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7],"acting":[11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":46,"stat_sum":{"num_bytes":6174507032,"num_objects":94217,"num_object_clones":0,"num_object_copies":188434,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":220,"num_objects_unfound":0,"num_objects_dirty":94217,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":102479,"num_write_kb":3634433,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22104,"num_bytes_recovered":1448607744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":13885644800,"data_stored":13885636656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":43282,"internal_metadata":0},"log_size":295863,"ondisk_log_size":295863,"up":92,"acting":92,"num_store_stats":16},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":11},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":112,"ondisk_log_size":112,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":13,"up_from":23,"seq":98784247852,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":698916,"kb_used_data":587220,"kb_used_omap":1,"kb_used_meta":111678,"kb_avail":93672924,"statfs":{"total":96636764160,"available":95921074176,"internally_reserved":0,"allocated":601313280,"data_stored":600551624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":114358725},"hb_peers":[0,1,2,4,6,8,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":23,"seq":98784247851,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":651516,"kb_used_data":584236,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":93720324,"statfs":{"total":96636764160,"available":95969611776,"internally_reserved":0,"allocated":598257664,"data_stored":597532634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":68876736},"hb_peers":[0,1,3,8,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":40,"seq":171798691875,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1040956,"kb_used_data":928428,"kb_used_omap":4,"kb_used_meta":112507,"kb_avail":93330884,"statfs":{"total":96636764160,"available":95570825216,"internally_reserved":0,"allocated":950710272,"data_stored":949945981,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4433,"internal_metadata":115207855},"hb_peers":[0,2,3,4,5,6,7,8,9,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":21,"seq":90194313260,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1095640,"kb_used_data":988104,"kb_used_omap":1,"kb_used_meta":107518,"kb_avail":93276200,"statfs":{"total":96636764160,"available":95514828800,"internally_reserved":0,"allocated":1011818496,"data_stored":1011032022,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":110098887},"hb_peers":[0,2,4,5,7,9,10,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":21,"seq":90194313263,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1246916,"kb_used_data":1129652,"kb_used_omap":1,"kb_used_meta":117246,"kb_avail":93124924,"statfs":{"total":96636764160,"available":95359922176,"internally_reserved":0,"allocated":1156763648,"data_stored":1155993293,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":120060359},"hb_peers":[0,2,4,6,7,8,9,11,12,13],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":22,"seq":94489280555,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1203548,"kb_used_data":1104332,"kb_used_omap":1,"kb_used_meta":99198,"kb_avail":93168292,"statfs":{"total":96636764160,"available":95404331008,"internally_reserved":0,"allocated":1130835968,"data_stored":1130110386,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":101579205},"hb_peers":[1,3,4,5,7,8,10,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,4],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":22,"seq":94489280556,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":523844,"kb_used_data":460724,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":93847996,"statfs":{"total":96636764160,"available":96100347904,"internally_reserved":0,"allocated":471781376,"data_stored":471021793,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[0,1,3,4,5,7,8,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":15,"up_from":22,"seq":94489280556,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1075024,"kb_used_data":971648,"kb_used_omap":1,"kb_used_meta":103358,"kb_avail":93296816,"statfs":{"total":96636764160,"available":95535939584,"internally_reserved":0,"allocated":994967552,"data_stored":994253594,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":105839046},"hb_peers":[0,4,5,6,7,8,10,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":22,"seq":94489280557,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722000,"kb_used_data":654720,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":93649840,"statfs":{"total":96636764160,"available":95897436160,"internally_reserved":0,"allocated":670433280,"data_stored":669662909,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[3,5,6,7,9,10,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":22,"seq":94489280556,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":542568,"kb_used_data":485016,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":93829272,"statfs":{"total":96636764160,"available":96081174528,"internally_reserved":0,"allocated":496656384,"data_stored":495920961,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58915270},"hb_peers":[0,1,3,5,7,8,10,11,12,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":22,"seq":94489280555,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1404924,"kb_used_data":1297388,"kb_used_omap":1,"kb_used_meta":107518,"kb_avail":92966916,"statfs":{"total":96636764160,"available":95198121984,"internally_reserved":0,"allocated":1328525312,"data_stored":1327802183,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":110098890},"hb_peers":[0,3,5,6,8,9,10,11,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":22,"seq":94489280556,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":538080,"kb_used_data":462480,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":93833760,"statfs":{"total":96636764160,"available":96085770240,"internally_reserved":0,"allocated":473579520,"data_stored":472789416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[1,5,6,7,9,10,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":22,"seq":94489280555,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":931500,"kb_used_data":833692,"kb_used_omap":1,"kb_used_meta":97790,"kb_avail":93440340,"statfs":{"total":96636764160,"available":95682908160,"internally_reserved":0,"allocated":853700608,"data_stored":852954682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":100137408},"hb_peers":[0,1,3,5,7,9,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":11,"up_from":22,"seq":94489280556,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1386084,"kb_used_data":1282708,"kb_used_omap":1,"kb_used_meta":103358,"kb_avail":92985756,"statfs":{"total":96636764160,"available":95217414144,"internally_reserved":0,"allocated":1313492992,"data_stored":1312735043,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":105839044},"hb_peers":[0,1,2,4,7,8,10,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":12,"up_from":22,"seq":94489280556,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1134932,"kb_used_data":1031556,"kb_used_omap":1,"kb_used_meta":103358,"kb_avail":93236908,"statfs":{"total":96636764160,"available":95474593792,"internally_reserved":0,"allocated":1056313344,"data_stored":1055536073,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":105839045},"hb_peers":[0,1,2,3,4,5,10,11,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":14,"up_from":0,"seq":609885356041,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1128267776,"data_stored":1128267776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":941752320,"data_stored":941752320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":465768448,"data_stored":465764376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1003945984,"data_stored":1003945984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":661127168,"data_stored":661127168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1141702656,"data_stored":1141702656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":489881600,"data_stored":489881600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1324744704,"data_stored":1324744704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":471072768,"data_stored":471072768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":595656704,"data_stored":595656704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":840695808,"data_stored":840695808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":1310130176,"data_stored":1310130176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":1046806528,"data_stored":1046806528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":599261184,"data_stored":599261184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":872742912,"data_stored":872742912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":43282,"internal_metadata":0},{"poolid":3,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":992088064,"data_stored":992083992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T11:25:36.563 INFO:tasks.ceph.osd.13.smithi092.stderr:2024-09-13T11:25:36.561+0000 7f3e5afea640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26350) UID: 0 2024-09-13T11:25:36.587 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-09-13T11:25:36.588 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:25:36.605 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-13T11:25:36.664 INFO:tasks.ceph.osd.13.smithi092.stderr:2024-09-13T11:25:36.661+0000 7f3e5afea640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26350) UID: 0 2024-09-13T11:25:36.746 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-09-13T11:25:36.746 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:25:36.766 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-09-13T11:25:36.766 INFO:tasks.ceph.osd.5.smithi092.stderr:2024-09-13T11:25:36.765+0000 7f6f0b705640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26346) UID: 0 2024-09-13T11:25:36.866 INFO:tasks.ceph.osd.1.smithi092.stderr:2024-09-13T11:25:36.865+0000 7fe7789c5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33718) UID: 0 2024-09-13T11:25:36.911 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-09-13T11:25:36.912 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:25:36.930 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-09-13T11:25:36.968 INFO:tasks.ceph.osd.8.smithi001.stderr:2024-09-13T11:25:36.967+0000 7fb9b48ae640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26312) UID: 0 2024-09-13T11:25:37.071 INFO:tasks.ceph.osd.9.smithi092.stderr:2024-09-13T11:25:37.065+0000 7ff5ab3d0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26348) UID: 0 2024-09-13T11:25:37.074 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-09-13T11:25:37.074 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:25:37.093 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-09-13T11:25:37.169 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-09-13T11:25:37.168+0000 7f97befc9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26315) UID: 0 2024-09-13T11:25:37.230 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-09-13T11:25:37.230 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:25:37.250 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_ops_in_flight 2024-09-13T11:25:37.256 INFO:teuthology.orchestra.run.smithi001.stdout:osd.6: {} 2024-09-13T11:25:37.256 INFO:teuthology.orchestra.run.smithi001.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T11:25:37.270 INFO:tasks.ceph.osd.8.smithi001.stderr:2024-09-13T11:25:37.267+0000 7fb9b48ae640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26312) UID: 0 2024-09-13T11:25:37.283 INFO:teuthology.orchestra.run.smithi001.stdout:osd.7: {} 2024-09-13T11:25:37.283 INFO:teuthology.orchestra.run.smithi001.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T11:25:37.304 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-13T11:25:37.310 INFO:teuthology.orchestra.run.smithi001.stdout:osd.8: {} 2024-09-13T11:25:37.310 INFO:teuthology.orchestra.run.smithi001.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-13T11:25:37.335 INFO:teuthology.orchestra.run.smithi001.stdout:osd.9: {} 2024-09-13T11:25:37.335 INFO:teuthology.orchestra.run.smithi001.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-13T11:25:37.364 INFO:teuthology.orchestra.run.smithi001.stdout:osd.10: {} 2024-09-13T11:25:37.364 INFO:teuthology.orchestra.run.smithi001.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-13T11:25:37.370 INFO:tasks.ceph.osd.15.smithi177.stderr:2024-09-13T11:25:37.366+0000 7f9d6ce77640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26578) UID: 0 2024-09-13T11:25:37.384 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-09-13T11:25:37.384 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:25:37.389 INFO:teuthology.orchestra.run.smithi001.stdout:osd.11: {} 2024-09-13T11:25:37.389 INFO:teuthology.orchestra.run.smithi001.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-13T11:25:37.400 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_blocked_ops 2024-09-13T11:25:37.416 INFO:teuthology.orchestra.run.smithi001.stdout:osd.12: {} 2024-09-13T11:25:37.416 INFO:teuthology.orchestra.run.smithi001.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-13T11:25:37.444 INFO:teuthology.orchestra.run.smithi001.stdout:osd.13: {} 2024-09-13T11:25:37.444 INFO:teuthology.orchestra.run.smithi001.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-13T11:25:37.471 INFO:teuthology.orchestra.run.smithi001.stdout:osd.14: {} 2024-09-13T11:25:37.472 INFO:teuthology.orchestra.run.smithi001.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-13T11:25:37.491 INFO:tasks.ceph.osd.7.smithi177.stderr:2024-09-13T11:25:37.490+0000 7f897b51d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26575) UID: 0 2024-09-13T11:25:37.497 INFO:teuthology.orchestra.run.smithi001.stdout:osd.15: {} 2024-09-13T11:25:37.497 INFO:teuthology.orchestra.run.smithi001.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-13T11:25:37.572 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-09-13T11:25:37.572 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [], 2024-09-13T11:25:37.572 INFO:teuthology.orchestra.run.smithi177.stdout: "complaint_time": 30, 2024-09-13T11:25:37.572 INFO:teuthology.orchestra.run.smithi177.stdout: "num_blocked_ops": 0 2024-09-13T11:25:37.572 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-09-13T11:25:37.573 INFO:tasks.ceph.osd.8.smithi001.stderr:2024-09-13T11:25:37.571+0000 7fb9b48ae640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26312) UID: 0 2024-09-13T11:25:37.590 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_historic_ops 2024-09-13T11:25:37.673 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-09-13T11:25:37.672+0000 7f97befc9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26315) UID: 0 2024-09-13T11:25:37.774 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-09-13T11:25:37.774 INFO:teuthology.orchestra.run.smithi177.stdout: "size": 20, 2024-09-13T11:25:37.774 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 600, 2024-09-13T11:25:37.774 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [ 2024-09-13T11:25:37.774 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:37.774 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:25383 3.0 e55/30 3:0a6bb869:::benchmark_data_smithi177_31116_object25382:head v 55'1620, mlcod=55'1617)", 2024-09-13T11:25:37.774 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:04.449956+0000", 2024-09-13T11:25:37.774 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 153.32161939599999, 2024-09-13T11:25:37.774 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.21143479600000001, 2024-09-13T11:25:37.775 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:25:37.775 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:25:37.775 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:37.775 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:25:37.775 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:37.775 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:25:37.775 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.449956+0000", 2024-09-13T11:25:37.775 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:37.775 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:37.775 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:37.775 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:25:37.775 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.449956+0000", 2024-09-13T11:25:37.775 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:37.776 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:37.776 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:37.776 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:25:37.776 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.449959+0000", 2024-09-13T11:25:37.776 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.393e-06 2024-09-13T11:25:37.776 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:37.776 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:37.776 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:25:37.776 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.450014+0000", 2024-09-13T11:25:37.776 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.4200000000000003e-05 2024-09-13T11:25:37.776 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:37.776 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:37.776 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:25:37.777 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.658637+0000", 2024-09-13T11:25:37.777 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.20862334799999999 2024-09-13T11:25:37.777 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:37.777 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:37.777 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:25:37.777 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.658650+0000", 2024-09-13T11:25:37.777 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3148e-05 2024-09-13T11:25:37.777 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:37.777 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:37.777 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:25:37.777 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.658867+0000", 2024-09-13T11:25:37.777 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00021673499999999999 2024-09-13T11:25:37.777 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:37.778 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:37.778 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:25:37.778 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.658931+0000", 2024-09-13T11:25:37.778 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.4045000000000001e-05 2024-09-13T11:25:37.778 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:37.778 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:37.778 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:25:37.778 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.661317+0000", 2024-09-13T11:25:37.778 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.002385823 2024-09-13T11:25:37.778 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:37.778 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:37.778 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:25:37.778 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.661391+0000", 2024-09-13T11:25:37.778 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.4103999999999994e-05 2024-09-13T11:25:37.779 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:37.779 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:25:37.779 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:37.779 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:37.779 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:37.779 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPush(3.16 85/70 [PushOp(3:6ac421eb:::benchmark_data_smithi177_31116_object43910:head, version: 66'2836, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:6ac421eb:::benchmark_data_smithi177_31116_object43910:head@66'2836, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:25:37.779 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:42.145291+0000", 2024-09-13T11:25:37.779 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 115.62628419000001, 2024-09-13T11:25:37.779 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.148514919, 2024-09-13T11:25:37.779 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:25:37.779 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:25:37.779 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:25:37.779 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:25:37.780 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:37.780 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:25:37.780 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.145291+0000", 2024-09-13T11:25:37.780 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:37.780 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:37.780 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:37.780 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:25:37.780 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.145291+0000", 2024-09-13T11:25:37.780 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:37.780 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:37.780 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:37.780 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:25:37.780 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.145293+0000", 2024-09-13T11:25:37.781 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.699e-06 2024-09-13T11:25:37.781 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:37.781 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:37.781 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:25:37.781 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.145417+0000", 2024-09-13T11:25:37.781 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00012415200000000001 2024-09-13T11:25:37.781 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:37.781 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:37.781 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:25:37.781 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.292964+0000", 2024-09-13T11:25:37.781 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.14754715400000001 2024-09-13T11:25:37.781 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:37.781 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:37.781 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:25:37.782 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.292981+0000", 2024-09-13T11:25:37.782 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.6790999999999999e-05 2024-09-13T11:25:37.782 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:37.782 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:37.782 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:25:37.782 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.293083+0000", 2024-09-13T11:25:37.782 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00010199799999999999 2024-09-13T11:25:37.782 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:37.782 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:37.782 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:25:37.782 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.293098+0000", 2024-09-13T11:25:37.782 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.4612e-05 2024-09-13T11:25:37.782 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:37.782 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:37.782 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:25:37.783 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.293806+0000", 2024-09-13T11:25:37.783 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00070851300000000002 2024-09-13T11:25:37.783 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:37.783 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:25:37.783 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:37.783 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:37.783 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:37.783 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:60390 3.16 e85/70 3:6a388186:::benchmark_data_smithi177_31116_object60389:head v 85'3357, mlcod=66'2835)", 2024-09-13T11:25:37.783 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:42.149102+0000", 2024-09-13T11:25:37.783 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 115.622473133, 2024-09-13T11:25:37.783 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.14821161799999999, 2024-09-13T11:25:37.783 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:25:37.783 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:25:37.784 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:37.784 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:25:37.784 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:37.784 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:25:37.784 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.149102+0000", 2024-09-13T11:25:37.784 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:37.784 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:37.784 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:37.784 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:25:37.784 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.149102+0000", 2024-09-13T11:25:37.784 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:37.784 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:37.784 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:37.785 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:25:37.785 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.149104+0000", 2024-09-13T11:25:37.785 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.708e-06 2024-09-13T11:25:37.785 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:37.785 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:37.785 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:25:37.785 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.149160+0000", 2024-09-13T11:25:37.785 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.5618000000000002e-05 2024-09-13T11:25:37.785 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:37.785 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:37.785 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:25:37.785 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.296071+0000", 2024-09-13T11:25:37.785 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.14691148600000001 2024-09-13T11:25:37.786 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:37.786 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:37.786 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:25:37.786 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.296076+0000", 2024-09-13T11:25:37.786 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.5580000000000002e-06 2024-09-13T11:25:37.786 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:37.786 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:37.786 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:25:37.786 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.296127+0000", 2024-09-13T11:25:37.786 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.1405e-05 2024-09-13T11:25:37.786 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:37.786 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:37.786 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:25:37.786 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.296150+0000", 2024-09-13T11:25:37.787 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.3113e-05 2024-09-13T11:25:37.787 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:37.787 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:37.787 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:25:37.787 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.297274+0000", 2024-09-13T11:25:37.787 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0011239900000000001 2024-09-13T11:25:37.787 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:37.787 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:37.787 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:25:37.787 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.297314+0000", 2024-09-13T11:25:37.787 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.9740000000000002e-05 2024-09-13T11:25:37.787 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:37.787 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:25:37.788 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:37.788 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:37.788 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:37.788 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPush(3.16 85/70 [PushOp(3:69aac9fd:::benchmark_data_smithi177_31116_object43914:head, version: 66'2837, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:69aac9fd:::benchmark_data_smithi177_31116_object43914:head@66'2837, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:25:37.788 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:42.295995+0000", 2024-09-13T11:25:37.788 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 115.475580076, 2024-09-13T11:25:37.788 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.215089431, 2024-09-13T11:25:37.788 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:25:37.788 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:25:37.788 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:25:37.788 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:25:37.788 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:37.788 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:25:37.789 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.295995+0000", 2024-09-13T11:25:37.789 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:37.789 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:37.789 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:37.789 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:25:37.789 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.295995+0000", 2024-09-13T11:25:37.789 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:37.789 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:37.789 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:37.790 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:25:37.790 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.295997+0000", 2024-09-13T11:25:37.790 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3170000000000001e-06 2024-09-13T11:25:37.790 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:37.790 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:37.790 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:25:37.790 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.296058+0000", 2024-09-13T11:25:37.790 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.1690000000000001e-05 2024-09-13T11:25:37.790 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:37.790 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:37.790 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:25:37.790 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.509606+0000", 2024-09-13T11:25:37.791 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.21354748300000001 2024-09-13T11:25:37.791 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:37.791 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:37.791 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:25:37.791 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.509638+0000", 2024-09-13T11:25:37.791 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.1644999999999999e-05 2024-09-13T11:25:37.791 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:37.791 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:37.791 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:25:37.791 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.509814+0000", 2024-09-13T11:25:37.791 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00017679499999999999 2024-09-13T11:25:37.791 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:37.792 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:37.792 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:25:37.792 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.509840+0000", 2024-09-13T11:25:37.792 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.5871e-05 2024-09-13T11:25:37.792 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:37.792 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:37.792 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:25:37.792 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.511085+0000", 2024-09-13T11:25:37.792 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00124463 2024-09-13T11:25:37.792 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:37.792 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:25:37.792 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:37.792 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:37.792 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:37.793 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPush(3.1d 91/78 [PushOp(3:be6a0ae4:::benchmark_data_smithi177_31116_object25671:head, version: 58'1569, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:be6a0ae4:::benchmark_data_smithi177_31116_object25671:head@58'1569, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:25:37.793 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:48.481971+0000", 2024-09-13T11:25:37.793 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 109.28960483100001, 2024-09-13T11:25:37.793 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.47945454199999998, 2024-09-13T11:25:37.793 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:25:37.793 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:25:37.793 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:25:37.793 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:25:37.793 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:37.793 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:25:37.793 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.481971+0000", 2024-09-13T11:25:37.793 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:37.794 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:37.794 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:37.794 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:25:37.794 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.481971+0000", 2024-09-13T11:25:37.794 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:37.794 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:37.794 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:37.794 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:25:37.794 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.481972+0000", 2024-09-13T11:25:37.794 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5179999999999999e-06 2024-09-13T11:25:37.794 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:37.794 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:37.794 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:25:37.794 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.482000+0000", 2024-09-13T11:25:37.795 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.8249e-05 2024-09-13T11:25:37.795 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:37.795 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:37.795 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:25:37.795 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.959926+0000", 2024-09-13T11:25:37.795 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.47792585100000001 2024-09-13T11:25:37.795 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:37.795 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:37.795 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:25:37.795 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.959944+0000", 2024-09-13T11:25:37.795 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.7620000000000001e-05 2024-09-13T11:25:37.795 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:37.795 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:37.795 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:25:37.796 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.960059+0000", 2024-09-13T11:25:37.796 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000115188 2024-09-13T11:25:37.796 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:37.796 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:37.796 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:25:37.796 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.960074+0000", 2024-09-13T11:25:37.796 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.4738e-05 2024-09-13T11:25:37.796 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:37.796 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:37.796 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:25:37.796 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.961425+0000", 2024-09-13T11:25:37.796 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0013513780000000001 2024-09-13T11:25:37.796 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:37.796 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:25:37.797 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:37.797 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:37.797 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:37.797 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:71884 3.16 e96/88 3:6ca846b2:::benchmark_data_smithi177_31116_object71883:head v 96'3755, mlcod=96'3753)", 2024-09-13T11:25:37.797 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:54.108299+0000", 2024-09-13T11:25:37.797 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 103.663276658, 2024-09-13T11:25:37.797 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.20685140099999999, 2024-09-13T11:25:37.797 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:25:37.797 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:25:37.797 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:37.797 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:25:37.797 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:37.797 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:25:37.798 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.108299+0000", 2024-09-13T11:25:37.798 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:37.798 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:37.798 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:37.798 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:25:37.798 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.108299+0000", 2024-09-13T11:25:37.798 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:37.798 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:37.798 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:37.798 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:25:37.798 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.108300+0000", 2024-09-13T11:25:37.798 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.173e-06 2024-09-13T11:25:37.798 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:37.799 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:37.799 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:25:37.799 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.108323+0000", 2024-09-13T11:25:37.799 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.2591000000000001e-05 2024-09-13T11:25:37.799 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:37.799 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:37.799 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:25:37.799 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.312377+0000", 2024-09-13T11:25:37.799 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.20405456599999999 2024-09-13T11:25:37.799 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:37.799 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:37.799 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:25:37.799 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.312382+0000", 2024-09-13T11:25:37.799 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.1429999999999997e-06 2024-09-13T11:25:37.800 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:37.800 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:37.800 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:25:37.800 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.312480+0000", 2024-09-13T11:25:37.800 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.7742e-05 2024-09-13T11:25:37.800 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:37.800 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:37.800 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:25:37.800 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.312521+0000", 2024-09-13T11:25:37.800 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.1369000000000003e-05 2024-09-13T11:25:37.800 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:37.800 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:37.800 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:25:37.801 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.315105+0000", 2024-09-13T11:25:37.801 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0025835060000000002 2024-09-13T11:25:37.801 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:37.801 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:37.801 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:25:37.801 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.315150+0000", 2024-09-13T11:25:37.801 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.5311e-05 2024-09-13T11:25:37.801 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:37.801 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:25:37.801 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:37.801 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:37.801 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:37.801 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:71927 3.0 e97/75 3:0197244a:::benchmark_data_smithi177_31116_object71926:head v 97'3351, mlcod=96'3348)", 2024-09-13T11:25:37.802 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:54.200161+0000", 2024-09-13T11:25:37.802 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 103.57141443, 2024-09-13T11:25:37.802 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.26691738999999998, 2024-09-13T11:25:37.802 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:25:37.802 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:25:37.802 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:37.802 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:25:37.802 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:37.802 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:25:37.802 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.200161+0000", 2024-09-13T11:25:37.802 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:37.802 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:37.802 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:37.803 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:25:37.803 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.200161+0000", 2024-09-13T11:25:37.803 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:37.803 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:37.803 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:37.803 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:25:37.803 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.200162+0000", 2024-09-13T11:25:37.803 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.78e-07 2024-09-13T11:25:37.803 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:37.803 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:37.803 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:25:37.803 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.200200+0000", 2024-09-13T11:25:37.804 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.8121000000000003e-05 2024-09-13T11:25:37.804 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:37.804 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:37.804 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:25:37.804 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.465397+0000", 2024-09-13T11:25:37.804 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.26519681099999998 2024-09-13T11:25:37.804 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:37.804 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:37.804 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:25:37.804 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.465404+0000", 2024-09-13T11:25:37.804 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.7399999999999998e-06 2024-09-13T11:25:37.804 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:37.804 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:37.804 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:25:37.805 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.465456+0000", 2024-09-13T11:25:37.805 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.2626000000000001e-05 2024-09-13T11:25:37.805 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:37.805 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:37.805 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:25:37.805 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.465482+0000", 2024-09-13T11:25:37.805 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.6086000000000001e-05 2024-09-13T11:25:37.805 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:37.805 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:37.805 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:25:37.805 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.467035+0000", 2024-09-13T11:25:37.805 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0015523729999999999 2024-09-13T11:25:37.805 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:37.806 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:37.806 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:25:37.806 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.467078+0000", 2024-09-13T11:25:37.806 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.3754999999999997e-05 2024-09-13T11:25:37.806 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:37.806 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:25:37.806 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:37.806 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:37.806 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:37.806 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:72993 3.0 e97/75 3:02747b8f:::benchmark_data_smithi177_31116_object72992:head v 97'3366, mlcod=97'3363)", 2024-09-13T11:25:37.806 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:55.116009+0000", 2024-09-13T11:25:37.806 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 102.655566311, 2024-09-13T11:25:37.806 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.13962660099999999, 2024-09-13T11:25:37.807 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:25:37.807 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:25:37.807 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:37.807 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:25:37.807 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:37.807 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:25:37.807 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:55.116009+0000", 2024-09-13T11:25:37.807 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:37.807 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:37.807 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:37.807 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:25:37.807 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:55.116009+0000", 2024-09-13T11:25:37.807 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:37.808 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:37.808 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:37.808 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:25:37.808 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:55.116010+0000", 2024-09-13T11:25:37.808 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.291e-06 2024-09-13T11:25:37.808 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:37.808 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:37.808 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:25:37.808 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:55.116062+0000", 2024-09-13T11:25:37.808 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.1949e-05 2024-09-13T11:25:37.808 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:37.808 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:37.809 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:25:37.809 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:55.254100+0000", 2024-09-13T11:25:37.809 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.13803723000000001 2024-09-13T11:25:37.809 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:37.809 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:37.809 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:25:37.809 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:55.254106+0000", 2024-09-13T11:25:37.809 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.5939999999999999e-06 2024-09-13T11:25:37.809 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:37.809 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:37.809 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:25:37.809 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:55.254196+0000", 2024-09-13T11:25:37.809 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.9567999999999994e-05 2024-09-13T11:25:37.810 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:37.810 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:37.810 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:25:37.810 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:55.254219+0000", 2024-09-13T11:25:37.810 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.3266000000000001e-05 2024-09-13T11:25:37.810 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:37.810 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:37.810 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:25:37.810 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:55.255596+0000", 2024-09-13T11:25:37.810 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0013767409999999999 2024-09-13T11:25:37.810 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:37.810 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:37.810 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:25:37.811 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:55.255636+0000", 2024-09-13T11:25:37.811 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.9962000000000001e-05 2024-09-13T11:25:37.811 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:37.811 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:25:37.811 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:37.811 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:37.811 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:37.811 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:79634 3.5 3:a6c32e6f:::benchmark_data_smithi177_31116_object79633:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101)", 2024-09-13T11:25:37.811 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:00.168616+0000", 2024-09-13T11:25:37.811 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 97.602959041999995, 2024-09-13T11:25:37.811 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.169445552, 2024-09-13T11:25:37.811 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:25:37.811 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:25:37.811 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:37.812 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:25:37.812 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:25:37.812 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:37.812 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 79634 2024-09-13T11:25:37.812 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:37.812 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:25:37.812 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:37.812 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:25:37.812 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.168616+0000", 2024-09-13T11:25:37.812 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:37.812 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:37.812 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:37.812 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:25:37.813 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.168616+0000", 2024-09-13T11:25:37.813 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:37.813 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:37.813 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:37.813 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:25:37.813 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.168619+0000", 2024-09-13T11:25:37.813 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.565e-06 2024-09-13T11:25:37.813 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:37.813 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:37.813 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:25:37.813 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.168639+0000", 2024-09-13T11:25:37.813 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.0384999999999999e-05 2024-09-13T11:25:37.813 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:37.814 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:37.814 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:25:37.814 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.168652+0000", 2024-09-13T11:25:37.814 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.2779000000000001e-05 2024-09-13T11:25:37.814 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:37.814 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:37.814 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:25:37.814 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.168658+0000", 2024-09-13T11:25:37.814 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.2269999999999998e-06 2024-09-13T11:25:37.814 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:37.814 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:37.814 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:25:37.814 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.168722+0000", 2024-09-13T11:25:37.814 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.3888999999999998e-05 2024-09-13T11:25:37.814 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:37.815 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:37.815 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:25:37.815 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.168863+0000", 2024-09-13T11:25:37.815 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00014050799999999999 2024-09-13T11:25:37.815 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:37.815 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:37.815 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 13", 2024-09-13T11:25:37.815 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.169010+0000", 2024-09-13T11:25:37.815 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00014746200000000001 2024-09-13T11:25:37.815 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:37.815 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:37.815 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:25:37.816 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.170891+0000", 2024-09-13T11:25:37.816 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001880945 2024-09-13T11:25:37.816 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:37.816 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:37.816 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:25:37.816 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.337961+0000", 2024-09-13T11:25:37.816 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.16706952 2024-09-13T11:25:37.816 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:37.816 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:37.816 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:25:37.816 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.338019+0000", 2024-09-13T11:25:37.816 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.7958000000000003e-05 2024-09-13T11:25:37.816 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:37.817 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:37.817 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:25:37.817 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.338062+0000", 2024-09-13T11:25:37.817 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.3313999999999999e-05 2024-09-13T11:25:37.817 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:37.817 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:25:37.817 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:37.817 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:37.817 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:37.817 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPush(3.1b 102/78 [PushOp(3:df786fc7:::benchmark_data_smithi177_31116_object48840:head, version: 71'3079, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:df786fc7:::benchmark_data_smithi177_31116_object48840:head@71'3079, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:25:37.817 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:00.290536+0000", 2024-09-13T11:25:37.818 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 97.481039667999994, 2024-09-13T11:25:37.818 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.20017585399999999, 2024-09-13T11:25:37.818 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:25:37.818 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:25:37.818 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:25:37.818 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:25:37.818 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:37.818 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:25:37.818 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.290536+0000", 2024-09-13T11:25:37.818 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:37.818 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:37.818 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:37.818 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:25:37.819 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.290536+0000", 2024-09-13T11:25:37.819 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:37.819 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:37.819 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:37.819 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:25:37.819 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.290537+0000", 2024-09-13T11:25:37.819 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.327e-06 2024-09-13T11:25:37.819 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:37.819 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:37.819 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:25:37.819 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.290581+0000", 2024-09-13T11:25:37.819 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.3893999999999997e-05 2024-09-13T11:25:37.819 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:37.820 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:37.820 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:25:37.820 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.488588+0000", 2024-09-13T11:25:37.820 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.198006708 2024-09-13T11:25:37.820 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:37.820 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:37.820 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:25:37.820 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.488607+0000", 2024-09-13T11:25:37.820 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.9162999999999999e-05 2024-09-13T11:25:37.820 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:37.820 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:37.820 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:25:37.821 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.488747+0000", 2024-09-13T11:25:37.821 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00014040799999999999 2024-09-13T11:25:37.821 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:37.821 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:37.821 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:25:37.821 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.488770+0000", 2024-09-13T11:25:37.821 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.279e-05 2024-09-13T11:25:37.821 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:37.821 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:37.821 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:25:37.821 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.490712+0000", 2024-09-13T11:25:37.821 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001941564 2024-09-13T11:25:37.821 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:37.821 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:25:37.822 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:37.822 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:37.822 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:37.822 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPush(3.1b 102/78 [PushOp(3:d82bf6ec:::benchmark_data_smithi177_31116_object48847:head, version: 71'3080, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:d82bf6ec:::benchmark_data_smithi177_31116_object48847:head@71'3080, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:25:37.822 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:00.293979+0000", 2024-09-13T11:25:37.822 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 97.477596141000006, 2024-09-13T11:25:37.822 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.20314832099999999, 2024-09-13T11:25:37.822 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:25:37.822 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:25:37.822 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:25:37.822 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:25:37.822 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:37.823 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:25:37.823 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.293979+0000", 2024-09-13T11:25:37.823 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:37.823 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:37.823 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:37.823 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:25:37.823 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.293979+0000", 2024-09-13T11:25:37.823 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:37.823 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:37.823 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:37.823 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:25:37.823 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.293981+0000", 2024-09-13T11:25:37.823 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.4449999999999999e-06 2024-09-13T11:25:37.823 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:37.824 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:37.824 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:25:37.824 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.294085+0000", 2024-09-13T11:25:37.824 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000103848 2024-09-13T11:25:37.824 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:37.824 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:37.824 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:25:37.824 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.495571+0000", 2024-09-13T11:25:37.824 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.20148640800000001 2024-09-13T11:25:37.824 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:37.824 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:37.824 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:25:37.824 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.495609+0000", 2024-09-13T11:25:37.824 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.7904000000000002e-05 2024-09-13T11:25:37.825 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:37.825 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:37.825 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:25:37.825 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.495729+0000", 2024-09-13T11:25:37.825 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000120556 2024-09-13T11:25:37.825 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:37.825 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:37.825 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:25:37.825 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.495751+0000", 2024-09-13T11:25:37.825 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.1511000000000001e-05 2024-09-13T11:25:37.825 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:37.825 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:37.825 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:25:37.826 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.497128+0000", 2024-09-13T11:25:37.826 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001376649 2024-09-13T11:25:37.826 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:37.826 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:25:37.826 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:37.826 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:37.826 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:37.826 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:79801 3.5 3:a6efdf84:::benchmark_data_smithi177_31116_object79800:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102)", 2024-09-13T11:25:37.826 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:00.403201+0000", 2024-09-13T11:25:37.826 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 97.368374553999999, 2024-09-13T11:25:37.826 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.31868626300000003, 2024-09-13T11:25:37.826 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:25:37.827 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:25:37.827 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:37.827 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:25:37.827 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:25:37.827 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:37.827 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 79801 2024-09-13T11:25:37.827 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:37.827 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:25:37.827 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:37.827 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:25:37.827 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.403201+0000", 2024-09-13T11:25:37.827 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:37.828 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:37.828 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:37.828 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:25:37.828 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.403201+0000", 2024-09-13T11:25:37.828 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:37.828 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:37.828 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:37.828 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:25:37.828 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.403204+0000", 2024-09-13T11:25:37.828 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.0819999999999999e-06 2024-09-13T11:25:37.828 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:37.828 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:37.828 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:25:37.828 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.403260+0000", 2024-09-13T11:25:37.829 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.5624999999999997e-05 2024-09-13T11:25:37.829 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:37.829 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:37.829 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:25:37.829 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.403280+0000", 2024-09-13T11:25:37.829 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.0060000000000001e-05 2024-09-13T11:25:37.829 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:37.829 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:37.829 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:25:37.829 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.403291+0000", 2024-09-13T11:25:37.829 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.1335000000000001e-05 2024-09-13T11:25:37.829 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:37.829 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:37.830 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:25:37.830 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.403344+0000", 2024-09-13T11:25:37.830 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.3217000000000002e-05 2024-09-13T11:25:37.830 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:37.830 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:37.830 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:25:37.830 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.403456+0000", 2024-09-13T11:25:37.830 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000111706 2024-09-13T11:25:37.830 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:37.831 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:37.831 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 13", 2024-09-13T11:25:37.831 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.403547+0000", 2024-09-13T11:25:37.831 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.1556000000000002e-05 2024-09-13T11:25:37.831 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:37.831 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:37.831 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:25:37.831 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.407436+0000", 2024-09-13T11:25:37.831 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.003888856 2024-09-13T11:25:37.831 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:37.831 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:37.831 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:25:37.831 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.721722+0000", 2024-09-13T11:25:37.832 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.31428562700000001 2024-09-13T11:25:37.832 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:37.832 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:37.832 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:25:37.832 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.721814+0000", 2024-09-13T11:25:37.832 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.1564000000000006e-05 2024-09-13T11:25:37.832 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:37.832 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:37.832 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:25:37.832 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.721887+0000", 2024-09-13T11:25:37.832 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.3634999999999995e-05 2024-09-13T11:25:37.832 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:37.832 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:25:37.833 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:37.833 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:37.833 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:37.833 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:79813 3.16 e102/88 3:6c2df1ae:::benchmark_data_smithi177_31116_object79812:head v 102'4016, mlcod=102'4014)", 2024-09-13T11:25:37.833 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:00.516959+0000", 2024-09-13T11:25:37.833 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 97.254616592999994, 2024-09-13T11:25:37.833 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.15075599100000001, 2024-09-13T11:25:37.833 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:25:37.833 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:25:37.833 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:37.833 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:25:37.833 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:37.833 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:25:37.833 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.516959+0000", 2024-09-13T11:25:37.834 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:37.834 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:37.834 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:37.834 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:25:37.834 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.516959+0000", 2024-09-13T11:25:37.834 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:37.834 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:37.834 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:37.834 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:25:37.834 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.516961+0000", 2024-09-13T11:25:37.834 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.0279999999999999e-06 2024-09-13T11:25:37.834 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:37.834 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:37.835 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:25:37.835 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.517002+0000", 2024-09-13T11:25:37.835 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.0754999999999999e-05 2024-09-13T11:25:37.835 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:37.835 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:37.835 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:25:37.835 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.664491+0000", 2024-09-13T11:25:37.835 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.14748925099999999 2024-09-13T11:25:37.835 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:37.835 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:37.835 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:25:37.835 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.664504+0000", 2024-09-13T11:25:37.835 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.2747e-05 2024-09-13T11:25:37.835 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:37.836 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:37.836 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:25:37.836 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.664642+0000", 2024-09-13T11:25:37.836 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00013850900000000001 2024-09-13T11:25:37.836 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:37.836 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:37.836 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:25:37.836 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.664678+0000", 2024-09-13T11:25:37.836 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.5403e-05 2024-09-13T11:25:37.836 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:37.836 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:37.836 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:25:37.836 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.667646+0000", 2024-09-13T11:25:37.837 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.002968133 2024-09-13T11:25:37.837 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:37.837 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:37.837 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:25:37.837 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.667715+0000", 2024-09-13T11:25:37.837 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.9165000000000006e-05 2024-09-13T11:25:37.837 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:37.837 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:25:37.837 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:37.837 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:37.837 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:37.837 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:83199 3.1d e106/102 3:be2c6fae:::benchmark_data_smithi177_31116_object83198:head v 106'4244, mlcod=106'4242)", 2024-09-13T11:25:37.837 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:04.298299+0000", 2024-09-13T11:25:37.838 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 93.473276427000002, 2024-09-13T11:25:37.838 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.25001107099999997, 2024-09-13T11:25:37.838 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:25:37.838 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:25:37.838 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:37.838 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:25:37.838 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:37.838 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:25:37.838 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.298299+0000", 2024-09-13T11:25:37.838 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:37.838 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:37.838 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:37.838 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:25:37.838 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.298299+0000", 2024-09-13T11:25:37.839 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:37.839 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:37.839 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:37.839 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:25:37.839 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.298301+0000", 2024-09-13T11:25:37.839 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5859999999999999e-06 2024-09-13T11:25:37.839 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:37.839 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:37.839 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:25:37.839 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.298368+0000", 2024-09-13T11:25:37.839 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.6928999999999996e-05 2024-09-13T11:25:37.839 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:37.839 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:37.839 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:25:37.840 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.545180+0000", 2024-09-13T11:25:37.840 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.24681209700000001 2024-09-13T11:25:37.840 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:37.840 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:37.840 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:25:37.840 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.545193+0000", 2024-09-13T11:25:37.840 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3427e-05 2024-09-13T11:25:37.840 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:37.840 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:37.840 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:25:37.840 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.545327+0000", 2024-09-13T11:25:37.840 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00013359400000000001 2024-09-13T11:25:37.840 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:37.841 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:37.841 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:25:37.841 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.545385+0000", 2024-09-13T11:25:37.841 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.8492000000000002e-05 2024-09-13T11:25:37.841 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:37.841 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:37.841 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:25:37.841 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.548225+0000", 2024-09-13T11:25:37.841 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.002839472 2024-09-13T11:25:37.842 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:37.842 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:37.842 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:25:37.842 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.548310+0000", 2024-09-13T11:25:37.842 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.5474000000000002e-05 2024-09-13T11:25:37.842 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:37.842 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:25:37.842 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:37.842 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:37.842 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:37.842 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:84621 3.1d e107/102 3:becd72aa:::benchmark_data_smithi177_31116_object84620:head v 107'4296, mlcod=107'4293)", 2024-09-13T11:25:37.842 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:05.742238+0000", 2024-09-13T11:25:37.842 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 92.029337326999993, 2024-09-13T11:25:37.842 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.17888174300000001, 2024-09-13T11:25:37.843 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:25:37.843 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:25:37.843 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:37.843 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:25:37.843 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:37.843 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:25:37.843 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:05.742238+0000", 2024-09-13T11:25:37.843 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:37.843 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:37.843 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:37.843 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:25:37.843 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:05.742238+0000", 2024-09-13T11:25:37.843 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:37.844 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:37.844 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:37.844 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:25:37.844 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:05.742240+0000", 2024-09-13T11:25:37.844 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.3159999999999999e-06 2024-09-13T11:25:37.844 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:37.844 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:37.844 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:25:37.844 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:05.742315+0000", 2024-09-13T11:25:37.844 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.4122000000000006e-05 2024-09-13T11:25:37.844 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:37.844 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:37.845 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:25:37.845 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:05.917757+0000", 2024-09-13T11:25:37.845 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.175442031 2024-09-13T11:25:37.845 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:37.845 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:37.845 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:25:37.845 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:05.917764+0000", 2024-09-13T11:25:37.845 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.661e-06 2024-09-13T11:25:37.845 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:37.845 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:37.845 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:25:37.845 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:05.917846+0000", 2024-09-13T11:25:37.845 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.2006000000000001e-05 2024-09-13T11:25:37.846 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:37.846 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:37.846 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:25:37.846 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:05.917879+0000", 2024-09-13T11:25:37.846 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.2796000000000002e-05 2024-09-13T11:25:37.846 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:37.846 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:37.846 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:25:37.846 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:05.921070+0000", 2024-09-13T11:25:37.846 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0031908489999999999 2024-09-13T11:25:37.846 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:37.847 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:37.847 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:25:37.847 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:05.921120+0000", 2024-09-13T11:25:37.847 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.9962e-05 2024-09-13T11:25:37.847 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:37.847 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:25:37.847 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:37.847 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:37.847 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:37.847 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:107910 3.16 e116/88 3:6cf57aa3:::benchmark_data_smithi177_31116_object107909:head v 116'4911, mlcod=116'4908)", 2024-09-13T11:25:37.847 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:22.516225+0000", 2024-09-13T11:25:37.847 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 75.255350784000001, 2024-09-13T11:25:37.847 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.18319666600000001, 2024-09-13T11:25:37.847 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:25:37.848 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:25:37.848 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:37.848 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:25:37.848 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:37.848 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:25:37.848 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:22.516225+0000", 2024-09-13T11:25:37.848 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:37.848 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:37.848 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:37.848 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:25:37.848 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:22.516225+0000", 2024-09-13T11:25:37.848 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:37.848 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:37.848 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:37.849 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:25:37.849 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:22.516226+0000", 2024-09-13T11:25:37.849 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.716e-06 2024-09-13T11:25:37.849 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:37.849 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:37.849 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:25:37.849 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:22.516250+0000", 2024-09-13T11:25:37.849 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.3975e-05 2024-09-13T11:25:37.849 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:37.849 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:37.849 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:25:37.849 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:22.697686+0000", 2024-09-13T11:25:37.849 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.18143539 2024-09-13T11:25:37.850 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:37.850 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:37.850 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:25:37.850 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:22.697692+0000", 2024-09-13T11:25:37.850 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.0769999999999996e-06 2024-09-13T11:25:37.850 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:37.850 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:37.850 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:25:37.850 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:22.697744+0000", 2024-09-13T11:25:37.850 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.2525000000000003e-05 2024-09-13T11:25:37.850 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:37.850 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:37.850 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:25:37.850 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:22.697757+0000", 2024-09-13T11:25:37.850 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.2819e-05 2024-09-13T11:25:37.851 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:37.851 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:37.851 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:25:37.851 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:22.699388+0000", 2024-09-13T11:25:37.851 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0016303999999999999 2024-09-13T11:25:37.851 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:37.851 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:37.851 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:25:37.851 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:22.699421+0000", 2024-09-13T11:25:37.851 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.3763999999999997e-05 2024-09-13T11:25:37.851 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:37.851 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:25:37.851 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:37.852 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:37.852 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:37.852 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:127682 3.16 e131/88 3:6e6da02e:::benchmark_data_smithi177_31116_object127681:head v 131'5491, mlcod=131'5488)", 2024-09-13T11:25:37.852 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:44.556692+0000", 2024-09-13T11:25:37.852 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 53.214883854999997, 2024-09-13T11:25:37.852 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.18048624299999999, 2024-09-13T11:25:37.852 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:25:37.852 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:25:37.852 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:37.852 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:25:37.852 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:37.852 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:25:37.852 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:44.556692+0000", 2024-09-13T11:25:37.853 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:37.853 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:37.853 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:37.853 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:25:37.853 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:44.556692+0000", 2024-09-13T11:25:37.853 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:37.853 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:37.853 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:37.853 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:25:37.853 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:44.556693+0000", 2024-09-13T11:25:37.853 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.8109999999999999e-06 2024-09-13T11:25:37.853 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:37.853 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:37.853 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:25:37.854 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:44.556715+0000", 2024-09-13T11:25:37.854 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.1328999999999999e-05 2024-09-13T11:25:37.854 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:37.854 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:37.854 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:25:37.854 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:44.734908+0000", 2024-09-13T11:25:37.854 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.178192884 2024-09-13T11:25:37.854 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:37.854 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:37.854 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:25:37.854 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:44.734915+0000", 2024-09-13T11:25:37.854 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.9970000000000001e-06 2024-09-13T11:25:37.854 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:37.854 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:37.855 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:25:37.855 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:44.734971+0000", 2024-09-13T11:25:37.855 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.6368e-05 2024-09-13T11:25:37.855 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:37.855 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:37.855 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:25:37.855 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:44.734990+0000", 2024-09-13T11:25:37.855 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.929e-05 2024-09-13T11:25:37.855 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:37.855 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:37.855 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:25:37.855 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:44.737096+0000", 2024-09-13T11:25:37.855 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0021053090000000001 2024-09-13T11:25:37.855 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:37.855 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:37.856 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:25:37.856 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:44.737178+0000", 2024-09-13T11:25:37.856 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.2255000000000004e-05 2024-09-13T11:25:37.856 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:37.856 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:25:37.856 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:37.856 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:37.856 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:37.856 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:137062 3.5 3:a16e178f:::benchmark_data_smithi177_31116_object137061:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e139)", 2024-09-13T11:25:37.856 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:53.140525+0000", 2024-09-13T11:25:37.856 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 44.631050834, 2024-09-13T11:25:37.856 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.20697954599999999, 2024-09-13T11:25:37.857 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:25:37.857 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:25:37.857 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:37.857 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:25:37.857 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:25:37.857 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:37.857 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 137062 2024-09-13T11:25:37.857 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:37.857 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:25:37.857 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:37.857 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:25:37.857 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:53.140525+0000", 2024-09-13T11:25:37.858 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:37.858 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:37.858 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:37.858 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:25:37.858 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:53.140525+0000", 2024-09-13T11:25:37.858 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:37.858 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:37.858 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:37.858 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:25:37.858 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:53.140527+0000", 2024-09-13T11:25:37.858 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.1739999999999999e-06 2024-09-13T11:25:37.858 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:37.859 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:37.859 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:25:37.859 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:53.140553+0000", 2024-09-13T11:25:37.859 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.6631e-05 2024-09-13T11:25:37.859 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:37.859 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:37.859 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:25:37.859 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:53.140566+0000", 2024-09-13T11:25:37.859 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.2268e-05 2024-09-13T11:25:37.859 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:37.859 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:37.859 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:25:37.859 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:53.140574+0000", 2024-09-13T11:25:37.860 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.5669999999999995e-06 2024-09-13T11:25:37.860 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:37.860 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:37.860 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:25:37.860 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:53.140622+0000", 2024-09-13T11:25:37.860 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.8143999999999998e-05 2024-09-13T11:25:37.860 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:37.860 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:37.860 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:25:37.860 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:53.140759+0000", 2024-09-13T11:25:37.860 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000136499 2024-09-13T11:25:37.860 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:37.860 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:37.861 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 13", 2024-09-13T11:25:37.861 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:53.140887+0000", 2024-09-13T11:25:37.861 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00012796300000000001 2024-09-13T11:25:37.861 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:37.861 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:37.861 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:25:37.861 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:53.142675+0000", 2024-09-13T11:25:37.861 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001788073 2024-09-13T11:25:37.861 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:37.861 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:37.861 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:25:37.861 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:53.347407+0000", 2024-09-13T11:25:37.861 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.20473206899999999 2024-09-13T11:25:37.861 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:37.862 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:37.862 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:25:37.862 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:53.347464+0000", 2024-09-13T11:25:37.862 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.6536000000000002e-05 2024-09-13T11:25:37.862 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:37.862 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:37.862 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:25:37.862 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:53.347504+0000", 2024-09-13T11:25:37.862 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.0621999999999998e-05 2024-09-13T11:25:37.862 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:37.862 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:25:37.862 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:37.862 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:37.862 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:37.863 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:161601 3.0 e155/75 3:03e8e5c6:::benchmark_data_smithi177_31116_object21298:head v 155'4743, mlcod=155'4741)", 2024-09-13T11:25:37.863 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:25:15.060319+0000", 2024-09-13T11:25:37.863 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 22.711256828, 2024-09-13T11:25:37.863 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.156824982, 2024-09-13T11:25:37.863 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:25:37.863 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:25:37.863 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:37.863 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:25:37.863 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:37.863 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:25:37.863 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:15.060319+0000", 2024-09-13T11:25:37.863 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:37.863 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:37.863 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:37.864 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:25:37.864 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:15.060319+0000", 2024-09-13T11:25:37.864 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:37.864 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:37.864 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:37.864 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:25:37.864 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:15.060320+0000", 2024-09-13T11:25:37.864 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.082e-06 2024-09-13T11:25:37.864 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:37.864 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:37.864 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:25:37.864 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:15.060325+0000", 2024-09-13T11:25:37.864 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.8110000000000002e-06 2024-09-13T11:25:37.864 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:37.865 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:37.865 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:25:37.865 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:15.212041+0000", 2024-09-13T11:25:37.865 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.15171630799999999 2024-09-13T11:25:37.865 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:37.865 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:37.865 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:25:37.865 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:15.212048+0000", 2024-09-13T11:25:37.865 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.7499999999999997e-06 2024-09-13T11:25:37.865 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:37.865 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:37.865 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:25:37.865 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:15.212106+0000", 2024-09-13T11:25:37.865 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.8870999999999999e-05 2024-09-13T11:25:37.866 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:37.866 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:37.866 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:25:37.866 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:15.212122+0000", 2024-09-13T11:25:37.866 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5175e-05 2024-09-13T11:25:37.866 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:37.866 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:37.866 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:25:37.866 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:15.217112+0000", 2024-09-13T11:25:37.866 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.004989991 2024-09-13T11:25:37.866 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:37.866 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:37.866 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:25:37.866 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:15.217144+0000", 2024-09-13T11:25:37.867 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.1993999999999999e-05 2024-09-13T11:25:37.867 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:37.867 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:25:37.867 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:37.867 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:37.867 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:37.867 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:176711 3.16 e167/88 3:68b3a0bf:::benchmark_data_smithi177_31116_object36408:head v 167'7062, mlcod=167'7058)", 2024-09-13T11:25:37.867 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:25:26.616425+0000", 2024-09-13T11:25:37.867 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 11.155150196999999, 2024-09-13T11:25:37.867 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.16947335599999999, 2024-09-13T11:25:37.867 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:25:37.867 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:25:37.867 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:37.867 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:25:37.868 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:37.868 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:25:37.868 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:26.616425+0000", 2024-09-13T11:25:37.868 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:37.868 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:37.868 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:37.868 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:25:37.868 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:26.616425+0000", 2024-09-13T11:25:37.868 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:37.868 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:37.868 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:37.868 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:25:37.868 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:26.616426+0000", 2024-09-13T11:25:37.869 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.1209999999999999e-06 2024-09-13T11:25:37.869 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:37.869 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:37.869 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:25:37.869 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:26.616432+0000", 2024-09-13T11:25:37.869 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.186e-06 2024-09-13T11:25:37.869 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:37.869 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:37.869 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:25:37.869 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:26.782875+0000", 2024-09-13T11:25:37.869 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.16644309199999999 2024-09-13T11:25:37.869 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:37.869 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:37.869 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:25:37.870 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:26.782880+0000", 2024-09-13T11:25:37.870 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.4580000000000001e-06 2024-09-13T11:25:37.870 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:37.870 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:37.870 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:25:37.870 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:26.782935+0000", 2024-09-13T11:25:37.870 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.5194e-05 2024-09-13T11:25:37.870 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:37.870 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:37.870 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:25:37.870 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:26.782951+0000", 2024-09-13T11:25:37.870 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5330999999999999e-05 2024-09-13T11:25:37.871 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:37.871 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:37.871 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:25:37.871 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:26.785854+0000", 2024-09-13T11:25:37.871 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0029030739999999998 2024-09-13T11:25:37.871 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:37.871 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:37.871 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:25:37.871 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:26.785899+0000", 2024-09-13T11:25:37.871 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.49e-05 2024-09-13T11:25:37.871 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:37.871 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:25:37.871 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:37.872 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:37.872 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:25:37.872 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-09-13T11:25:37.872 INFO:tasks.ceph.osd.4.smithi001.stderr:2024-09-13T11:25:37.771+0000 7ff8e0218640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26310) UID: 0 2024-09-13T11:25:37.873 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-13T11:25:37.875 INFO:tasks.ceph.osd.5.smithi092.stderr:2024-09-13T11:25:37.873+0000 7f6f0b705640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26346) UID: 0 2024-09-13T11:25:37.975 INFO:tasks.ceph.osd.5.smithi092.stderr:2024-09-13T11:25:37.973+0000 7f6f0b705640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26346) UID: 0 2024-09-13T11:25:38.027 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-09-13T11:25:38.027 INFO:teuthology.orchestra.run.smithi092.stdout: "ops": [], 2024-09-13T11:25:38.027 INFO:teuthology.orchestra.run.smithi092.stdout: "num_ops": 0 2024-09-13T11:25:38.028 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-09-13T11:25:38.044 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-13T11:25:38.076 INFO:tasks.ceph.osd.10.smithi162.stderr:2024-09-13T11:25:38.072+0000 7fb59c9f0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26317) UID: 0 2024-09-13T11:25:38.177 INFO:tasks.ceph.osd.4.smithi001.stderr:2024-09-13T11:25:38.175+0000 7ff8e0218640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26310) UID: 0 2024-09-13T11:25:38.237 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-09-13T11:25:38.237 INFO:teuthology.orchestra.run.smithi092.stdout: "ops": [], 2024-09-13T11:25:38.237 INFO:teuthology.orchestra.run.smithi092.stdout: "complaint_time": 30, 2024-09-13T11:25:38.237 INFO:teuthology.orchestra.run.smithi092.stdout: "num_blocked_ops": 0 2024-09-13T11:25:38.237 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-09-13T11:25:38.258 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-13T11:25:38.277 INFO:tasks.ceph.osd.4.smithi001.stderr:2024-09-13T11:25:38.275+0000 7ff8e0218640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26310) UID: 0 2024-09-13T11:25:38.318 INFO:teuthology.orchestra.run.smithi001.stderr:noscrub is set 2024-09-13T11:25:38.319 INFO:tasks.ceph.osd.12.smithi001.stderr:2024-09-13T11:25:38.315+0000 7f891f10f640 -1 osd.12 174 handle_osd_map injecting map crc failure 2024-09-13T11:25:38.378 INFO:tasks.ceph.osd.10.smithi162.stderr:2024-09-13T11:25:38.376+0000 7fb59c9f0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26317) UID: 0 2024-09-13T11:25:38.403 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-09-13T11:25:38.404 INFO:teuthology.orchestra.run.smithi092.stdout: "size": 20, 2024-09-13T11:25:38.404 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 600, 2024-09-13T11:25:38.404 INFO:teuthology.orchestra.run.smithi092.stdout: "ops": [ 2024-09-13T11:25:38.404 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:38.404 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:3 3.4 3.125e09e4 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e40)", 2024-09-13T11:25:38.404 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:46.304641+0000", 2024-09-13T11:25:38.404 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 172.09782227599999, 2024-09-13T11:25:38.404 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.374372119, 2024-09-13T11:25:38.404 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:25:38.404 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:25:38.404 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "reached pg", 2024-09-13T11:25:38.404 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:25:38.404 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:25:38.405 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:38.405 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 3 2024-09-13T11:25:38.405 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:38.405 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:25:38.405 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:38.405 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:25:38.405 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.304641+0000", 2024-09-13T11:25:38.405 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:38.405 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:38.405 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:38.405 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:25:38.405 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.304641+0000", 2024-09-13T11:25:38.405 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:38.406 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:38.406 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:38.406 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:25:38.406 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.304644+0000", 2024-09-13T11:25:38.406 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.5130000000000001e-06 2024-09-13T11:25:38.406 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:38.406 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:38.406 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:25:38.406 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305058+0000", 2024-09-13T11:25:38.406 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000414277 2024-09-13T11:25:38.406 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:38.406 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:38.407 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:25:38.407 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305124+0000", 2024-09-13T11:25:38.407 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.5813000000000002e-05 2024-09-13T11:25:38.407 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:38.407 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:38.407 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:25:38.407 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305156+0000", 2024-09-13T11:25:38.407 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.1489000000000003e-05 2024-09-13T11:25:38.407 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:38.407 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:38.407 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:25:38.407 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.678939+0000", 2024-09-13T11:25:38.407 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.37378339999999999 2024-09-13T11:25:38.407 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:38.408 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:38.408 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:25:38.408 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.679013+0000", 2024-09-13T11:25:38.408 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.3627000000000005e-05 2024-09-13T11:25:38.408 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:38.408 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:25:38.408 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:38.408 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:38.408 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:38.408 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:29 3.4 3.b7bbfbe4 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-09-13T11:25:38.408 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:46.305215+0000", 2024-09-13T11:25:38.408 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 172.09724797300001, 2024-09-13T11:25:38.408 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.37387964600000001, 2024-09-13T11:25:38.408 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:25:38.409 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:25:38.409 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "reached pg", 2024-09-13T11:25:38.409 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:25:38.409 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:25:38.409 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:38.409 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 29 2024-09-13T11:25:38.409 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:38.409 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:25:38.409 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:38.409 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:25:38.409 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305215+0000", 2024-09-13T11:25:38.409 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:38.409 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:38.409 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:38.410 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:25:38.410 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305215+0000", 2024-09-13T11:25:38.410 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:38.410 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:38.410 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:38.410 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:25:38.410 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305218+0000", 2024-09-13T11:25:38.410 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.7030000000000002e-06 2024-09-13T11:25:38.410 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:38.410 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:38.410 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:25:38.410 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305366+0000", 2024-09-13T11:25:38.410 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00014825400000000001 2024-09-13T11:25:38.411 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:38.411 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:38.411 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:25:38.411 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305386+0000", 2024-09-13T11:25:38.411 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.9689000000000001e-05 2024-09-13T11:25:38.411 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:38.411 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:38.411 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:25:38.411 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305403+0000", 2024-09-13T11:25:38.411 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.7600000000000001e-05 2024-09-13T11:25:38.411 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:38.411 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:38.411 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:25:38.411 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.679067+0000", 2024-09-13T11:25:38.412 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.37366339100000001 2024-09-13T11:25:38.412 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:38.412 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:38.412 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:25:38.412 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.679095+0000", 2024-09-13T11:25:38.412 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.8008999999999999e-05 2024-09-13T11:25:38.412 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:38.412 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:25:38.412 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:38.412 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:38.412 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:38.412 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:30 3.4 3.547ec0f4 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-09-13T11:25:38.413 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:46.305453+0000", 2024-09-13T11:25:38.413 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 172.09700984899999, 2024-09-13T11:25:38.413 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.373764914, 2024-09-13T11:25:38.413 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:25:38.413 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:25:38.413 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "reached pg", 2024-09-13T11:25:38.413 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:25:38.413 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:25:38.413 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:38.413 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 30 2024-09-13T11:25:38.413 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:38.413 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:25:38.414 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:38.414 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:25:38.414 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305453+0000", 2024-09-13T11:25:38.414 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:38.414 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:38.414 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:38.414 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:25:38.414 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305453+0000", 2024-09-13T11:25:38.414 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:38.414 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:38.414 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:38.415 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:25:38.415 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305456+0000", 2024-09-13T11:25:38.415 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.9560000000000002e-06 2024-09-13T11:25:38.415 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:38.415 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:38.415 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:25:38.415 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305597+0000", 2024-09-13T11:25:38.415 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00014122700000000001 2024-09-13T11:25:38.415 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:38.415 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:38.415 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:25:38.415 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305615+0000", 2024-09-13T11:25:38.415 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.8139999999999999e-05 2024-09-13T11:25:38.416 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:38.416 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:38.416 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:25:38.416 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305632+0000", 2024-09-13T11:25:38.416 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.6101999999999999e-05 2024-09-13T11:25:38.416 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:38.416 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:38.416 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:25:38.416 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.679195+0000", 2024-09-13T11:25:38.416 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.373563486 2024-09-13T11:25:38.416 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:38.416 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:38.416 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:25:38.416 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.679218+0000", 2024-09-13T11:25:38.417 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.3003e-05 2024-09-13T11:25:38.417 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:38.417 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:25:38.417 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:38.417 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:38.417 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:38.417 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:70 3.4 3.1404a604 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-09-13T11:25:38.417 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:46.305665+0000", 2024-09-13T11:25:38.417 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 172.09679770100001, 2024-09-13T11:25:38.417 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.37365550199999997, 2024-09-13T11:25:38.417 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:25:38.417 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:25:38.417 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "reached pg", 2024-09-13T11:25:38.418 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:25:38.418 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:25:38.418 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:38.418 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 70 2024-09-13T11:25:38.418 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:38.418 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:25:38.418 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:38.418 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:25:38.418 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305665+0000", 2024-09-13T11:25:38.418 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:38.418 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:38.418 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:38.418 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:25:38.419 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305665+0000", 2024-09-13T11:25:38.419 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:38.419 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:38.419 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:38.419 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:25:38.419 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305668+0000", 2024-09-13T11:25:38.419 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.5890000000000001e-06 2024-09-13T11:25:38.419 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:38.419 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:38.419 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:25:38.419 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305791+0000", 2024-09-13T11:25:38.419 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000123545 2024-09-13T11:25:38.419 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:38.419 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:38.420 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:25:38.420 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305808+0000", 2024-09-13T11:25:38.420 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.668e-05 2024-09-13T11:25:38.420 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:38.420 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:38.420 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:25:38.420 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305824+0000", 2024-09-13T11:25:38.420 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.6385e-05 2024-09-13T11:25:38.420 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:38.420 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:38.420 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:25:38.420 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.679305+0000", 2024-09-13T11:25:38.420 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.37348059900000002 2024-09-13T11:25:38.421 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:38.421 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:38.421 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:25:38.421 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.679321+0000", 2024-09-13T11:25:38.421 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.5704000000000001e-05 2024-09-13T11:25:38.421 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:38.421 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:25:38.421 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:38.421 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:38.421 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:38.421 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:25408 3.4 3:294e23bd:::benchmark_data_smithi177_31116_object25407:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e56)", 2024-09-13T11:25:38.421 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:04.495455+0000", 2024-09-13T11:25:38.421 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 153.90700804900001, 2024-09-13T11:25:38.422 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.29897367600000002, 2024-09-13T11:25:38.422 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:25:38.422 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:25:38.422 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:38.422 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:25:38.422 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:25:38.422 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:38.422 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 25408 2024-09-13T11:25:38.422 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:38.422 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:25:38.422 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:38.422 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:25:38.422 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.495455+0000", 2024-09-13T11:25:38.422 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:38.422 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:38.422 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:38.423 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:25:38.423 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.495455+0000", 2024-09-13T11:25:38.423 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:38.423 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:38.423 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:38.423 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:25:38.423 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.495459+0000", 2024-09-13T11:25:38.423 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.5630000000000002e-06 2024-09-13T11:25:38.423 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:38.423 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:38.423 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:25:38.423 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.495545+0000", 2024-09-13T11:25:38.423 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.5704000000000005e-05 2024-09-13T11:25:38.423 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:38.424 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:38.424 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:25:38.424 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.495581+0000", 2024-09-13T11:25:38.424 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.5689000000000003e-05 2024-09-13T11:25:38.424 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:38.424 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:38.424 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:25:38.424 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.495597+0000", 2024-09-13T11:25:38.424 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.6053e-05 2024-09-13T11:25:38.424 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:38.424 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:38.424 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:25:38.424 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.495721+0000", 2024-09-13T11:25:38.424 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00012379099999999999 2024-09-13T11:25:38.424 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:38.425 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:38.425 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:25:38.425 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.495940+0000", 2024-09-13T11:25:38.425 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00021967699999999999 2024-09-13T11:25:38.425 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:38.425 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:38.425 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 12", 2024-09-13T11:25:38.425 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.496162+0000", 2024-09-13T11:25:38.425 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000221917 2024-09-13T11:25:38.425 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:38.425 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:38.425 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:25:38.425 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.498699+0000", 2024-09-13T11:25:38.425 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0025366690000000001 2024-09-13T11:25:38.426 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:38.426 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:38.426 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:25:38.426 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.794139+0000", 2024-09-13T11:25:38.426 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.295440025 2024-09-13T11:25:38.426 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:38.426 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:38.426 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:25:38.426 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.794306+0000", 2024-09-13T11:25:38.426 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00016689799999999999 2024-09-13T11:25:38.426 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:38.426 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:38.427 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:25:38.427 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.794429+0000", 2024-09-13T11:25:38.427 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00012269000000000001 2024-09-13T11:25:38.427 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:38.427 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:25:38.427 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:38.427 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:38.427 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:38.427 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "MOSDPGPushReply(3.14 75/70 [PushReplyOp(3:2f14928a:::benchmark_data_smithi177_31116_object15123:head)])", 2024-09-13T11:25:38.427 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:31.569662+0000", 2024-09-13T11:25:38.427 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 126.832801277, 2024-09-13T11:25:38.427 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.35759148299999999, 2024-09-13T11:25:38.427 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:25:38.427 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:25:38.428 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "reached pg", 2024-09-13T11:25:38.428 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:25:38.428 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:38.428 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:25:38.428 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:31.569662+0000", 2024-09-13T11:25:38.428 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:38.428 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:38.428 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:38.428 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:25:38.428 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:31.569662+0000", 2024-09-13T11:25:38.428 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:38.428 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:38.428 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:38.429 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:25:38.429 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:31.569663+0000", 2024-09-13T11:25:38.429 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.333e-06 2024-09-13T11:25:38.429 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:38.429 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:38.429 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:25:38.429 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:31.569667+0000", 2024-09-13T11:25:38.429 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.8929999999999998e-06 2024-09-13T11:25:38.429 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:38.429 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:38.429 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:25:38.429 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:31.923221+0000", 2024-09-13T11:25:38.430 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.35355413699999999 2024-09-13T11:25:38.430 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:38.430 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:38.430 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:25:38.430 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:31.923230+0000", 2024-09-13T11:25:38.430 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.6829999999999997e-06 2024-09-13T11:25:38.430 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:38.430 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:38.430 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:25:38.430 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:31.923807+0000", 2024-09-13T11:25:38.430 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00057686800000000002 2024-09-13T11:25:38.430 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:38.430 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:38.431 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:25:38.431 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:31.927253+0000", 2024-09-13T11:25:38.431 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.003446569 2024-09-13T11:25:38.431 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:38.431 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:25:38.431 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:38.431 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:38.431 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:38.431 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "MOSDPGPushReply(3.18 85/70 [PushReplyOp(3:1e410119:::benchmark_data_smithi177_31116_object35700:head)])", 2024-09-13T11:25:38.431 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:42.216614+0000", 2024-09-13T11:25:38.431 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 116.185848933, 2024-09-13T11:25:38.431 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.54260090699999997, 2024-09-13T11:25:38.431 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:25:38.431 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:25:38.432 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "reached pg", 2024-09-13T11:25:38.432 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:25:38.432 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:38.432 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:25:38.432 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.216614+0000", 2024-09-13T11:25:38.432 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:38.432 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:38.432 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:38.432 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:25:38.432 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.216614+0000", 2024-09-13T11:25:38.432 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:38.432 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:38.432 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:38.433 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:25:38.433 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.216615+0000", 2024-09-13T11:25:38.433 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.0200000000000001e-07 2024-09-13T11:25:38.433 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:38.433 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:38.433 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:25:38.433 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.216619+0000", 2024-09-13T11:25:38.433 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.805e-06 2024-09-13T11:25:38.433 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:38.433 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:38.433 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:25:38.433 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.757476+0000", 2024-09-13T11:25:38.433 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.54085762000000004 2024-09-13T11:25:38.433 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:38.434 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:38.434 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:25:38.434 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.757498+0000", 2024-09-13T11:25:38.434 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.1936000000000001e-05 2024-09-13T11:25:38.434 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:38.434 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:38.434 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:25:38.434 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.757678+0000", 2024-09-13T11:25:38.434 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00018006599999999999 2024-09-13T11:25:38.434 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:38.434 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:38.434 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:25:38.434 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.759215+0000", 2024-09-13T11:25:38.435 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0015366780000000001 2024-09-13T11:25:38.437 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:38.437 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:25:38.437 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:38.437 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:38.437 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:38.437 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "MOSDPGPushReply(3.1b 85/78 [PushReplyOp(3:dabda1cc:::benchmark_data_smithi177_31116_object12988:head)])", 2024-09-13T11:25:38.437 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:42.485171+0000", 2024-09-13T11:25:38.438 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 115.91729202499999, 2024-09-13T11:25:38.438 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.308420427, 2024-09-13T11:25:38.438 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:25:38.438 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:25:38.438 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "reached pg", 2024-09-13T11:25:38.438 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:25:38.438 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:38.438 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:25:38.438 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.485171+0000", 2024-09-13T11:25:38.438 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:38.438 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:38.438 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:38.438 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:25:38.438 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.485171+0000", 2024-09-13T11:25:38.439 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:38.439 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:38.439 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:38.439 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:25:38.439 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.485173+0000", 2024-09-13T11:25:38.439 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.1569999999999998e-06 2024-09-13T11:25:38.439 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:38.439 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:38.439 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:25:38.439 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.485182+0000", 2024-09-13T11:25:38.439 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.6759999999999996e-06 2024-09-13T11:25:38.439 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:38.440 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:38.440 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:25:38.440 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.791336+0000", 2024-09-13T11:25:38.440 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.30615446499999999 2024-09-13T11:25:38.440 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:38.440 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:38.440 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:25:38.440 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.791369+0000", 2024-09-13T11:25:38.440 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.3155999999999999e-05 2024-09-13T11:25:38.440 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:38.440 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:38.440 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:25:38.440 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.791472+0000", 2024-09-13T11:25:38.440 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00010247900000000001 2024-09-13T11:25:38.441 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:38.441 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:38.441 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:25:38.441 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.793591+0000", 2024-09-13T11:25:38.441 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.002119494 2024-09-13T11:25:38.441 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:38.441 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:25:38.441 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:38.441 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:38.441 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:38.441 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:64066 3.1b 3:dd84e0ec:::benchmark_data_smithi177_31116_object64065:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89)", 2024-09-13T11:25:38.441 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:46.633608+0000", 2024-09-13T11:25:38.441 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 111.768854586, 2024-09-13T11:25:38.442 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.32564315399999999, 2024-09-13T11:25:38.442 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:25:38.442 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:25:38.442 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:38.442 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:25:38.442 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:25:38.442 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:38.442 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 64066 2024-09-13T11:25:38.442 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:38.442 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:25:38.442 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:38.442 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:25:38.442 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.633608+0000", 2024-09-13T11:25:38.442 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:38.443 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:38.443 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:38.443 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:25:38.443 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.633608+0000", 2024-09-13T11:25:38.443 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:38.443 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:38.443 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:38.443 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:25:38.443 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.633612+0000", 2024-09-13T11:25:38.443 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.856e-06 2024-09-13T11:25:38.443 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:38.443 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:38.443 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:25:38.444 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.633683+0000", 2024-09-13T11:25:38.444 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.1000999999999993e-05 2024-09-13T11:25:38.444 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:38.444 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:38.444 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:25:38.444 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.633711+0000", 2024-09-13T11:25:38.444 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.8204999999999998e-05 2024-09-13T11:25:38.444 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:38.444 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:38.444 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:25:38.444 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.633732+0000", 2024-09-13T11:25:38.444 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.0662999999999999e-05 2024-09-13T11:25:38.444 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:38.445 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:38.445 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:25:38.445 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.635573+0000", 2024-09-13T11:25:38.445 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.001841365 2024-09-13T11:25:38.445 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:38.445 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:38.445 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:25:38.445 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.635995+0000", 2024-09-13T11:25:38.445 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00042113699999999998 2024-09-13T11:25:38.445 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:38.445 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:38.445 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 12,15", 2024-09-13T11:25:38.445 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.636475+0000", 2024-09-13T11:25:38.445 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00048067699999999999 2024-09-13T11:25:38.446 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:38.446 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:38.446 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:25:38.446 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.640247+0000", 2024-09-13T11:25:38.446 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0037716450000000001 2024-09-13T11:25:38.446 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:38.446 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:38.446 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:25:38.446 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.649308+0000", 2024-09-13T11:25:38.446 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0090613240000000008 2024-09-13T11:25:38.446 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:38.446 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:38.446 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:25:38.447 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.956912+0000", 2024-09-13T11:25:38.447 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.30760338500000001 2024-09-13T11:25:38.447 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:38.447 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:38.447 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:25:38.447 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.957156+0000", 2024-09-13T11:25:38.447 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000244031 2024-09-13T11:25:38.447 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:38.447 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:38.447 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:25:38.447 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.959252+0000", 2024-09-13T11:25:38.447 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0020958650000000001 2024-09-13T11:25:38.447 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:38.447 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:25:38.448 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:38.448 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:38.448 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:38.448 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop_reply(client.4665.0:64066 3.1b e89/78 ondisk, result = 0)", 2024-09-13T11:25:38.448 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:46.651596+0000", 2024-09-13T11:25:38.448 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 111.750866755, 2024-09-13T11:25:38.448 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.30767612599999999, 2024-09-13T11:25:38.448 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:25:38.448 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:25:38.448 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-09-13T11:25:38.448 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:25:38.448 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:38.448 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:25:38.448 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.651596+0000", 2024-09-13T11:25:38.449 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:38.449 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:38.449 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:38.449 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:25:38.449 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.651596+0000", 2024-09-13T11:25:38.449 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:38.449 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:38.449 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:38.449 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:25:38.449 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.651600+0000", 2024-09-13T11:25:38.449 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.6969999999999999e-06 2024-09-13T11:25:38.449 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:38.449 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:38.449 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:25:38.450 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.651611+0000", 2024-09-13T11:25:38.450 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.1516e-05 2024-09-13T11:25:38.450 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:38.450 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:38.450 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:25:38.450 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.948118+0000", 2024-09-13T11:25:38.450 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.29650663500000002 2024-09-13T11:25:38.450 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:38.450 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:38.450 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:25:38.450 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.948137+0000", 2024-09-13T11:25:38.450 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.8529000000000002e-05 2024-09-13T11:25:38.450 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:38.450 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:38.451 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:25:38.451 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.956870+0000", 2024-09-13T11:25:38.451 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0087336380000000002 2024-09-13T11:25:38.451 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:38.451 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:38.451 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:25:38.451 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.956893+0000", 2024-09-13T11:25:38.451 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.317e-05 2024-09-13T11:25:38.451 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:38.451 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:38.451 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:25:38.451 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.959272+0000", 2024-09-13T11:25:38.451 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0023789409999999999 2024-09-13T11:25:38.451 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:38.452 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:25:38.452 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:38.452 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:38.452 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:38.452 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "MOSDPGPushReply(3.1b 91/78 [PushReplyOp(3:d9ac6889:::benchmark_data_smithi177_31116_object22254:head)])", 2024-09-13T11:25:38.452 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:48.392729+0000", 2024-09-13T11:25:38.452 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 110.00973384, 2024-09-13T11:25:38.452 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.30924993099999998, 2024-09-13T11:25:38.452 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:25:38.452 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:25:38.452 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "reached pg", 2024-09-13T11:25:38.452 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:25:38.452 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:38.452 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:25:38.453 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:48.392729+0000", 2024-09-13T11:25:38.453 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:38.453 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:38.453 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:38.453 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:25:38.453 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:48.392729+0000", 2024-09-13T11:25:38.453 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:38.453 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:38.453 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:38.453 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:25:38.453 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:48.392730+0000", 2024-09-13T11:25:38.453 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.336e-06 2024-09-13T11:25:38.453 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:38.454 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:38.454 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:25:38.454 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:48.392739+0000", 2024-09-13T11:25:38.454 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.8580000000000006e-06 2024-09-13T11:25:38.454 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:38.454 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:38.454 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:25:38.454 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:48.700180+0000", 2024-09-13T11:25:38.454 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.30744035600000003 2024-09-13T11:25:38.454 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:38.454 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:38.454 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:25:38.454 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:48.700197+0000", 2024-09-13T11:25:38.455 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.7649999999999999e-05 2024-09-13T11:25:38.455 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:38.455 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:38.455 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:25:38.455 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:48.700398+0000", 2024-09-13T11:25:38.455 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000201113 2024-09-13T11:25:38.455 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:38.455 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:38.455 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:25:38.455 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:48.701979+0000", 2024-09-13T11:25:38.455 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.001580618 2024-09-13T11:25:38.455 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:38.455 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:25:38.455 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:38.456 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:38.456 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:38.456 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop(client.4665.0:83193 3.8 e106/67 3:10066d83:::benchmark_data_smithi177_31116_object83192:head v 106'3925, mlcod=106'3922)", 2024-09-13T11:25:38.456 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:24:04.285068+0000", 2024-09-13T11:25:38.456 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 94.117395084999998, 2024-09-13T11:25:38.456 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.64647716600000005, 2024-09-13T11:25:38.456 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:25:38.456 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:25:38.456 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:38.456 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:25:38.456 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:38.456 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:25:38.457 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:04.285068+0000", 2024-09-13T11:25:38.457 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:38.457 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:38.457 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:38.457 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:25:38.457 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:04.285068+0000", 2024-09-13T11:25:38.457 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:38.457 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:38.457 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:38.457 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:25:38.457 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:04.285071+0000", 2024-09-13T11:25:38.457 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.729e-06 2024-09-13T11:25:38.457 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:38.458 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:38.458 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:25:38.458 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:04.285215+0000", 2024-09-13T11:25:38.458 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00014410000000000001 2024-09-13T11:25:38.458 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:38.458 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:38.458 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:25:38.458 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:04.928221+0000", 2024-09-13T11:25:38.458 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.64300653299999999 2024-09-13T11:25:38.458 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:38.458 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:38.458 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:25:38.458 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:04.928240+0000", 2024-09-13T11:25:38.458 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.8678999999999999e-05 2024-09-13T11:25:38.459 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:38.459 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:38.459 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:25:38.459 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:04.928388+0000", 2024-09-13T11:25:38.459 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00014809 2024-09-13T11:25:38.459 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:38.459 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:38.459 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:25:38.459 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:04.928428+0000", 2024-09-13T11:25:38.459 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.9776e-05 2024-09-13T11:25:38.459 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:38.459 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:38.459 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:25:38.459 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:04.931463+0000", 2024-09-13T11:25:38.460 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.003035168 2024-09-13T11:25:38.460 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:38.460 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:38.460 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:25:38.460 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:04.931545+0000", 2024-09-13T11:25:38.460 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.2090999999999997e-05 2024-09-13T11:25:38.460 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:38.460 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:25:38.460 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:38.460 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:38.460 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:38.460 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop(client.4665.0:83239 3.e e106/75 3:74547f17:::benchmark_data_smithi177_31116_object83238:head v 106'4391, mlcod=106'4388)", 2024-09-13T11:25:38.460 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:24:04.581196+0000", 2024-09-13T11:25:38.461 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 93.821266578999996, 2024-09-13T11:25:38.461 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.33617203600000001, 2024-09-13T11:25:38.461 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:25:38.461 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:25:38.461 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:38.461 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:25:38.461 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:38.461 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:25:38.461 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:04.581196+0000", 2024-09-13T11:25:38.461 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:38.461 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:38.461 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:38.461 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:25:38.461 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:04.581196+0000", 2024-09-13T11:25:38.462 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:38.462 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:38.462 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:38.462 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:25:38.462 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:04.581200+0000", 2024-09-13T11:25:38.462 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.19e-06 2024-09-13T11:25:38.462 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:38.462 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:38.462 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:25:38.462 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:04.581257+0000", 2024-09-13T11:25:38.462 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.7531999999999997e-05 2024-09-13T11:25:38.462 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:38.462 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:38.462 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:25:38.463 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:04.914011+0000", 2024-09-13T11:25:38.463 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.33275413700000001 2024-09-13T11:25:38.463 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:38.463 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:38.463 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:25:38.463 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:04.914028+0000", 2024-09-13T11:25:38.463 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.7082999999999999e-05 2024-09-13T11:25:38.463 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:38.463 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:38.463 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:25:38.463 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:04.914151+0000", 2024-09-13T11:25:38.463 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00012283300000000001 2024-09-13T11:25:38.463 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:38.463 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:38.464 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:25:38.464 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:04.914185+0000", 2024-09-13T11:25:38.464 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.4255999999999999e-05 2024-09-13T11:25:38.464 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:38.464 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:38.464 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:25:38.464 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:04.917300+0000", 2024-09-13T11:25:38.464 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0031141599999999999 2024-09-13T11:25:38.464 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:38.464 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:38.464 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:25:38.464 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:04.917368+0000", 2024-09-13T11:25:38.465 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.8844999999999996e-05 2024-09-13T11:25:38.465 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:38.465 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:25:38.465 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:38.465 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:38.465 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:38.465 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:131687 3.28 3:14f65078:::benchmark_data_smithi177_31116_object131686:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e135)", 2024-09-13T11:25:38.465 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:24:48.925565+0000", 2024-09-13T11:25:38.465 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 49.476897581999999, 2024-09-13T11:25:38.465 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.41421604099999998, 2024-09-13T11:25:38.465 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:25:38.465 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:25:38.465 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:38.465 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:25:38.466 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:25:38.466 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:38.466 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 131687 2024-09-13T11:25:38.466 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:38.466 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:25:38.466 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:38.466 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:25:38.466 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:48.925565+0000", 2024-09-13T11:25:38.466 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:38.466 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:38.466 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:38.466 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:25:38.466 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:48.925565+0000", 2024-09-13T11:25:38.467 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:38.467 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:38.467 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:38.467 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:25:38.467 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:48.925568+0000", 2024-09-13T11:25:38.467 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.441e-06 2024-09-13T11:25:38.467 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:38.467 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:38.467 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:25:38.467 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:48.925617+0000", 2024-09-13T11:25:38.467 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.8739000000000002e-05 2024-09-13T11:25:38.467 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:38.467 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:38.467 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:25:38.468 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:48.925631+0000", 2024-09-13T11:25:38.468 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.4463e-05 2024-09-13T11:25:38.468 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:38.468 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:38.468 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:25:38.468 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:48.925638+0000", 2024-09-13T11:25:38.468 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.2119999999999997e-06 2024-09-13T11:25:38.468 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:38.468 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:38.468 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:25:38.468 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:48.925703+0000", 2024-09-13T11:25:38.468 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.4794000000000003e-05 2024-09-13T11:25:38.468 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:38.469 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:38.469 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:25:38.469 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:48.925857+0000", 2024-09-13T11:25:38.469 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00015427 2024-09-13T11:25:38.469 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:38.469 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:38.469 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 5,8", 2024-09-13T11:25:38.469 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:48.925995+0000", 2024-09-13T11:25:38.469 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000137365 2024-09-13T11:25:38.469 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:38.469 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:38.469 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:25:38.469 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:48.927261+0000", 2024-09-13T11:25:38.470 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.001266021 2024-09-13T11:25:38.470 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:38.470 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:38.470 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:25:38.470 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:48.930015+0000", 2024-09-13T11:25:38.470 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0027545130000000001 2024-09-13T11:25:38.470 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:38.470 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:38.470 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:25:38.470 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:49.338898+0000", 2024-09-13T11:25:38.470 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.40888254299999999 2024-09-13T11:25:38.470 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:38.471 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:38.471 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:25:38.471 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:49.338967+0000", 2024-09-13T11:25:38.471 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.9402999999999999e-05 2024-09-13T11:25:38.471 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:38.471 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:38.471 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:25:38.471 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:49.339781+0000", 2024-09-13T11:25:38.471 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00081427699999999997 2024-09-13T11:25:38.471 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:38.471 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:25:38.471 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:38.471 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:38.472 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:38.472 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop_reply(client.4665.0:131687 3.28 e135/120 ondisk, result = 0)", 2024-09-13T11:25:38.472 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:24:48.928935+0000", 2024-09-13T11:25:38.472 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 49.473528281, 2024-09-13T11:25:38.472 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.41085638499999999, 2024-09-13T11:25:38.472 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:25:38.472 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:25:38.472 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-09-13T11:25:38.472 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:25:38.472 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:38.472 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:25:38.472 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:48.928935+0000", 2024-09-13T11:25:38.472 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:38.473 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:38.473 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:38.473 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:25:38.473 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:48.928935+0000", 2024-09-13T11:25:38.473 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:38.473 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:38.473 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:38.473 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:25:38.473 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:48.928936+0000", 2024-09-13T11:25:38.473 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.519e-06 2024-09-13T11:25:38.473 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:38.473 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:38.473 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:25:38.473 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:48.928941+0000", 2024-09-13T11:25:38.474 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.0710000000000002e-06 2024-09-13T11:25:38.474 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:38.474 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:38.474 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:25:38.474 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:49.338799+0000", 2024-09-13T11:25:38.474 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.40985808499999998 2024-09-13T11:25:38.474 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:38.474 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:38.474 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:25:38.474 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:49.338809+0000", 2024-09-13T11:25:38.474 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.4390000000000006e-06 2024-09-13T11:25:38.474 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:38.474 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:38.475 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:25:38.475 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:49.338878+0000", 2024-09-13T11:25:38.475 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.9429000000000002e-05 2024-09-13T11:25:38.475 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:38.475 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:38.475 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:25:38.475 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:49.338890+0000", 2024-09-13T11:25:38.475 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.2217e-05 2024-09-13T11:25:38.475 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:38.475 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:38.475 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:25:38.475 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:49.339791+0000", 2024-09-13T11:25:38.475 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00090062499999999995 2024-09-13T11:25:38.476 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:38.476 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:25:38.476 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:38.476 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:38.476 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:38.476 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "MOSDPGPushReply(3.28 163/120 [PushReplyOp(3:169fe9fd:::benchmark_data_smithi177_31116_object59744:head)])", 2024-09-13T11:25:38.476 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:25:22.263786+0000", 2024-09-13T11:25:38.476 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 16.138677012999999, 2024-09-13T11:25:38.476 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.28927712799999999, 2024-09-13T11:25:38.476 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:25:38.476 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:25:38.476 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "reached pg", 2024-09-13T11:25:38.476 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:25:38.477 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:38.477 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:25:38.477 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:22.263786+0000", 2024-09-13T11:25:38.477 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:38.477 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:38.477 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:38.477 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:25:38.477 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:22.263786+0000", 2024-09-13T11:25:38.477 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:38.477 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:38.477 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:38.477 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:25:38.477 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:22.263788+0000", 2024-09-13T11:25:38.478 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.542e-06 2024-09-13T11:25:38.478 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:38.478 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:38.478 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:25:38.478 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:22.263797+0000", 2024-09-13T11:25:38.478 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.234e-06 2024-09-13T11:25:38.478 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:38.478 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:38.478 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:25:38.478 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:22.552310+0000", 2024-09-13T11:25:38.478 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.28851281499999998 2024-09-13T11:25:38.478 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:38.478 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:38.478 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:25:38.479 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:22.552315+0000", 2024-09-13T11:25:38.479 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.5990000000000001e-06 2024-09-13T11:25:38.479 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:38.479 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:38.479 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:25:38.479 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:22.552362+0000", 2024-09-13T11:25:38.479 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.7144000000000001e-05 2024-09-13T11:25:38.479 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:38.479 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:38.479 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:25:38.479 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:22.553063+0000", 2024-09-13T11:25:38.479 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00070079400000000003 2024-09-13T11:25:38.479 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:38.480 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:25:38.480 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:38.480 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:38.480 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:38.480 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:181661 3.4 3:2019f47a:::benchmark_data_smithi177_31116_object41358:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e170)", 2024-09-13T11:25:38.480 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:25:31.131054+0000", 2024-09-13T11:25:38.480 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 7.2714085150000001, 2024-09-13T11:25:38.480 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.44761152399999998, 2024-09-13T11:25:38.480 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:25:38.480 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:25:38.480 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:38.480 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:25:38.481 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:25:38.481 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:38.481 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 181661 2024-09-13T11:25:38.481 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:38.481 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:25:38.481 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:38.481 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:25:38.481 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:31.131054+0000", 2024-09-13T11:25:38.481 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:38.481 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:38.481 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:38.481 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:25:38.481 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:31.131054+0000", 2024-09-13T11:25:38.481 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:38.482 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:38.482 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:38.482 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:25:38.482 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:31.131056+0000", 2024-09-13T11:25:38.482 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.8640000000000001e-06 2024-09-13T11:25:38.482 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:38.482 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:38.482 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:25:38.482 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:31.131061+0000", 2024-09-13T11:25:38.482 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.6249999999999998e-06 2024-09-13T11:25:38.482 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:38.482 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:38.482 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:25:38.483 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:31.131069+0000", 2024-09-13T11:25:38.483 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.1049999999999995e-06 2024-09-13T11:25:38.483 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:38.483 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:38.483 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:25:38.483 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:31.131074+0000", 2024-09-13T11:25:38.483 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.344e-06 2024-09-13T11:25:38.483 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:38.483 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:38.483 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:25:38.483 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:31.131121+0000", 2024-09-13T11:25:38.483 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.6794e-05 2024-09-13T11:25:38.483 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:38.483 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:38.484 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:25:38.484 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:31.131225+0000", 2024-09-13T11:25:38.484 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000103502 2024-09-13T11:25:38.484 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:38.484 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:38.484 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 12", 2024-09-13T11:25:38.484 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:31.131289+0000", 2024-09-13T11:25:38.484 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.4790999999999997e-05 2024-09-13T11:25:38.484 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:38.484 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:38.484 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:25:38.484 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:31.133191+0000", 2024-09-13T11:25:38.484 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0019013859999999999 2024-09-13T11:25:38.484 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:38.485 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:38.485 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:25:38.485 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:31.578585+0000", 2024-09-13T11:25:38.485 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.44539403 2024-09-13T11:25:38.485 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:38.485 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:38.485 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:25:38.485 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:31.578620+0000", 2024-09-13T11:25:38.485 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.4657999999999999e-05 2024-09-13T11:25:38.485 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:38.485 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:38.485 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:25:38.486 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:31.578666+0000", 2024-09-13T11:25:38.486 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.6424999999999997e-05 2024-09-13T11:25:38.486 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:38.486 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:25:38.486 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:38.486 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:38.486 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:38.486 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop_reply(client.4665.0:181661 3.4 e170/117 ondisk, result = 0)", 2024-09-13T11:25:38.486 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:25:31.135385+0000", 2024-09-13T11:25:38.486 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 7.2670776950000002, 2024-09-13T11:25:38.486 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.44328512599999997, 2024-09-13T11:25:38.486 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:25:38.486 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:25:38.487 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-09-13T11:25:38.487 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:25:38.487 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:38.487 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:25:38.487 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:31.135385+0000", 2024-09-13T11:25:38.487 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:38.487 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:38.487 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:38.487 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:25:38.487 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:31.135385+0000", 2024-09-13T11:25:38.487 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:38.487 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:38.487 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:38.487 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:25:38.488 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:31.135387+0000", 2024-09-13T11:25:38.488 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.217e-06 2024-09-13T11:25:38.488 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:38.488 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:38.488 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:25:38.488 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:31.135391+0000", 2024-09-13T11:25:38.488 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.0820000000000001e-06 2024-09-13T11:25:38.488 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:38.488 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:38.488 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:25:38.488 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:31.578504+0000", 2024-09-13T11:25:38.488 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.44311382100000002 2024-09-13T11:25:38.488 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:38.488 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:38.489 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:25:38.489 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:31.578530+0000", 2024-09-13T11:25:38.489 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.5367e-05 2024-09-13T11:25:38.489 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:38.489 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:38.489 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:25:38.489 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:31.578576+0000", 2024-09-13T11:25:38.489 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.5840999999999997e-05 2024-09-13T11:25:38.489 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:38.489 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:38.489 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:25:38.489 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:31.578581+0000", 2024-09-13T11:25:38.489 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.5620000000000003e-06 2024-09-13T11:25:38.490 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:38.490 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:38.490 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:25:38.490 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:31.578670+0000", 2024-09-13T11:25:38.490 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.9235999999999997e-05 2024-09-13T11:25:38.490 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:38.490 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:25:38.490 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:38.490 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:38.490 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:38.490 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop(client.4665.0:184026 3.b e171/117 3:d3052347:::benchmark_data_smithi177_31116_object43723:head v 171'6273, mlcod=171'6269)", 2024-09-13T11:25:38.490 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:25:32.645695+0000", 2024-09-13T11:25:38.490 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 5.7567679609999995, 2024-09-13T11:25:38.491 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.34822244200000002, 2024-09-13T11:25:38.491 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:25:38.491 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:25:38.491 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:38.491 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:25:38.491 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:38.491 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:25:38.491 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:32.645695+0000", 2024-09-13T11:25:38.491 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:38.491 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:38.491 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:38.491 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:25:38.491 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:32.645695+0000", 2024-09-13T11:25:38.492 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:38.492 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:38.492 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:38.492 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:25:38.492 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:32.645696+0000", 2024-09-13T11:25:38.492 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.4780000000000001e-06 2024-09-13T11:25:38.492 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:38.492 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:38.493 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:25:38.493 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:32.645704+0000", 2024-09-13T11:25:38.493 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.1980000000000004e-06 2024-09-13T11:25:38.493 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:38.493 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:38.493 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:25:38.493 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:32.992228+0000", 2024-09-13T11:25:38.493 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.346523992 2024-09-13T11:25:38.493 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:38.493 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:38.493 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:25:38.493 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:32.992232+0000", 2024-09-13T11:25:38.493 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.3109999999999999e-06 2024-09-13T11:25:38.494 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:38.494 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:38.494 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:25:38.494 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:32.992275+0000", 2024-09-13T11:25:38.494 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.2589999999999997e-05 2024-09-13T11:25:38.494 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:38.494 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:38.494 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:25:38.494 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:32.992289+0000", 2024-09-13T11:25:38.494 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.4116e-05 2024-09-13T11:25:38.494 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:38.494 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:38.494 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:25:38.495 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:32.993877+0000", 2024-09-13T11:25:38.495 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0015879259999999999 2024-09-13T11:25:38.495 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:38.495 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:38.495 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:25:38.495 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:32.993917+0000", 2024-09-13T11:25:38.495 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.0831000000000003e-05 2024-09-13T11:25:38.495 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:38.495 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:25:38.495 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:38.495 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:38.495 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:38.496 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop(client.4665.0:184072 3.b e171/117 3:d319cf93:::benchmark_data_smithi177_31116_object43769:head v 171'6274, mlcod=171'6270)", 2024-09-13T11:25:38.496 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:25:32.701516+0000", 2024-09-13T11:25:38.496 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 5.7009472729999997, 2024-09-13T11:25:38.496 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.30463955399999998, 2024-09-13T11:25:38.496 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:25:38.496 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:25:38.496 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:38.496 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:25:38.496 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:38.496 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:25:38.496 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:32.701516+0000", 2024-09-13T11:25:38.496 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:38.497 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:38.497 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:38.497 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:25:38.497 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:32.701516+0000", 2024-09-13T11:25:38.497 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:38.497 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:38.497 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:38.497 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:25:38.497 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:32.701519+0000", 2024-09-13T11:25:38.497 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.0029999999999999e-06 2024-09-13T11:25:38.497 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:38.497 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:38.498 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:25:38.498 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:32.701530+0000", 2024-09-13T11:25:38.498 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.1759999999999999e-05 2024-09-13T11:25:38.498 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:38.498 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:38.498 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:25:38.498 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:33.003551+0000", 2024-09-13T11:25:38.498 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.302020819 2024-09-13T11:25:38.498 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:38.498 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:38.498 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:25:38.498 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:33.003558+0000", 2024-09-13T11:25:38.499 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.2299999999999996e-06 2024-09-13T11:25:38.499 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:38.499 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:38.499 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:25:38.499 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:33.003630+0000", 2024-09-13T11:25:38.499 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.2125000000000005e-05 2024-09-13T11:25:38.499 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:38.499 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:38.499 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:25:38.499 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:33.003648+0000", 2024-09-13T11:25:38.499 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.8672e-05 2024-09-13T11:25:38.499 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:38.499 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:38.499 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:25:38.500 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:33.006114+0000", 2024-09-13T11:25:38.500 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0024656790000000001 2024-09-13T11:25:38.500 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:38.500 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:38.500 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:25:38.500 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:33.006155+0000", 2024-09-13T11:25:38.500 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.1266000000000001e-05 2024-09-13T11:25:38.500 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:38.500 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:25:38.500 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:38.500 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:38.500 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:25:38.500 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-09-13T11:25:38.502 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_ops_in_flight 2024-09-13T11:25:38.503 INFO:tasks.ceph.osd.5.smithi092.stderr:2024-09-13T11:25:38.501+0000 7f6f0b705640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26346) UID: 0 2024-09-13T11:25:38.603 INFO:tasks.ceph.osd.2.smithi162.stderr:2024-09-13T11:25:38.600+0000 7fe92b35a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26313) UID: 0 2024-09-13T11:25:38.635 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-13T11:25:38.635 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [], 2024-09-13T11:25:38.636 INFO:teuthology.orchestra.run.smithi162.stdout: "num_ops": 0 2024-09-13T11:25:38.636 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-13T11:25:38.650 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_blocked_ops 2024-09-13T11:25:38.704 INFO:tasks.ceph.osd.11.smithi177.stderr:2024-09-13T11:25:38.702+0000 7f24792c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26580) UID: 0 2024-09-13T11:25:38.805 INFO:tasks.ceph.osd.5.smithi092.stderr:2024-09-13T11:25:38.805+0000 7f6f0b705640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26346) UID: 0 2024-09-13T11:25:38.822 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-13T11:25:38.822 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [], 2024-09-13T11:25:38.822 INFO:teuthology.orchestra.run.smithi162.stdout: "complaint_time": 30, 2024-09-13T11:25:38.823 INFO:teuthology.orchestra.run.smithi162.stdout: "num_blocked_ops": 0 2024-09-13T11:25:38.823 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-13T11:25:38.838 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_historic_ops 2024-09-13T11:25:38.906 INFO:tasks.ceph.osd.9.smithi092.stderr:2024-09-13T11:25:38.905+0000 7ff5ab3d0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26348) UID: 0 2024-09-13T11:25:38.968 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-13T11:25:38.968 INFO:teuthology.orchestra.run.smithi162.stdout: "size": 20, 2024-09-13T11:25:38.968 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 600, 2024-09-13T11:25:38.968 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [] 2024-09-13T11:25:38.969 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-13T11:25:38.982 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-13T11:25:39.032 INFO:tasks.ceph.osd.4.smithi001.stderr:2024-09-13T11:25:39.031+0000 7ff8e0218640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26310) UID: 0 2024-09-13T11:25:39.108 INFO:tasks.ceph.osd.15.smithi177.stderr:2024-09-13T11:25:39.106+0000 7f9d6ce77640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26578) UID: 0 2024-09-13T11:25:39.126 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-13T11:25:39.126 INFO:teuthology.orchestra.run.smithi001.stdout: "ops": [], 2024-09-13T11:25:39.127 INFO:teuthology.orchestra.run.smithi001.stdout: "num_ops": 0 2024-09-13T11:25:39.127 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-13T11:25:39.141 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-13T11:25:39.209 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-09-13T11:25:39.204+0000 7f97befc9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26315) UID: 0 2024-09-13T11:25:39.310 INFO:tasks.ceph.osd.5.smithi092.stderr:2024-09-13T11:25:39.309+0000 7f6f0b705640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26346) UID: 0 2024-09-13T11:25:39.318 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-13T11:25:39.318 INFO:teuthology.orchestra.run.smithi001.stdout: "ops": [], 2024-09-13T11:25:39.318 INFO:teuthology.orchestra.run.smithi001.stdout: "complaint_time": 30, 2024-09-13T11:25:39.318 INFO:teuthology.orchestra.run.smithi001.stdout: "num_blocked_ops": 0 2024-09-13T11:25:39.318 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-13T11:25:39.337 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-13T11:25:39.341 INFO:tasks.ceph.osd.15.smithi177.stderr:2024-09-13T11:25:39.338+0000 7f9d618a4640 -1 osd.15 175 handle_osd_map injecting map crc failure 2024-09-13T11:25:39.370 INFO:tasks.ceph.osd.11.smithi177.stderr:2024-09-13T11:25:39.366+0000 7f246bcf0640 -1 osd.11 175 handle_osd_map injecting map crc failure 2024-09-13T11:25:39.410 INFO:tasks.ceph.osd.12.smithi001.stderr:2024-09-13T11:25:39.407+0000 7f892c6e6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26314) UID: 0 2024-09-13T11:25:39.483 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-13T11:25:39.483 INFO:teuthology.orchestra.run.smithi001.stdout: "size": 20, 2024-09-13T11:25:39.483 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 600, 2024-09-13T11:25:39.483 INFO:teuthology.orchestra.run.smithi001.stdout: "ops": [ 2024-09-13T11:25:39.483 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:39.483 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDPGPushReply(3.1e 84/78 [PushReplyOp(3:79ee54b9:::benchmark_data_smithi177_31116_object11701:head)])", 2024-09-13T11:25:39.484 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:41.407335+0000", 2024-09-13T11:25:39.484 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 118.073385395, 2024-09-13T11:25:39.484 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.51622350400000006, 2024-09-13T11:25:39.484 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:39.484 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:39.484 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-13T11:25:39.484 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:39.484 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:39.484 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:39.484 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:41.407335+0000", 2024-09-13T11:25:39.484 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:39.484 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:39.485 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:39.485 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:39.485 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:41.407335+0000", 2024-09-13T11:25:39.485 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:39.485 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:39.485 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:39.485 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:39.485 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:41.407337+0000", 2024-09-13T11:25:39.485 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.505e-06 2024-09-13T11:25:39.485 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:39.485 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:39.486 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:39.486 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:41.407343+0000", 2024-09-13T11:25:39.486 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.9089999999999997e-06 2024-09-13T11:25:39.486 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:39.486 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:39.486 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:39.486 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:41.921977+0000", 2024-09-13T11:25:39.486 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.51463463499999995 2024-09-13T11:25:39.486 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:39.486 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:39.486 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:39.486 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:41.921988+0000", 2024-09-13T11:25:39.487 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.092e-05 2024-09-13T11:25:39.487 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:39.487 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:39.487 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:39.487 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:41.922088+0000", 2024-09-13T11:25:39.487 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.9528e-05 2024-09-13T11:25:39.488 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:39.488 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:39.488 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:39.488 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:41.923559+0000", 2024-09-13T11:25:39.489 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0014710070000000001 2024-09-13T11:25:39.489 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:39.489 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:39.489 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:39.489 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:39.489 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:39.489 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4665.0:60417 3.10 e85/84 3:092808b0:::benchmark_data_smithi177_31116_object60416:head v 85'3295, mlcod=85'3292)", 2024-09-13T11:25:39.490 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:42.174488+0000", 2024-09-13T11:25:39.490 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 117.306232841, 2024-09-13T11:25:39.490 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.387908896, 2024-09-13T11:25:39.490 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:39.490 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:39.490 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:39.490 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:39.490 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:39.490 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:39.490 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.174488+0000", 2024-09-13T11:25:39.491 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:39.491 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:39.491 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:39.491 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:39.491 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.174488+0000", 2024-09-13T11:25:39.491 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:39.491 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:39.491 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:39.491 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:39.491 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.174490+0000", 2024-09-13T11:25:39.491 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.9099999999999999e-06 2024-09-13T11:25:39.492 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:39.492 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:39.492 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:39.492 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.174527+0000", 2024-09-13T11:25:39.492 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.7543000000000003e-05 2024-09-13T11:25:39.492 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:39.492 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:39.492 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:39.492 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.561093+0000", 2024-09-13T11:25:39.492 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.38656578400000002 2024-09-13T11:25:39.492 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:39.492 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:39.493 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:39.493 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.561101+0000", 2024-09-13T11:25:39.493 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.6090000000000003e-06 2024-09-13T11:25:39.493 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:39.493 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:39.493 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:39.493 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.561143+0000", 2024-09-13T11:25:39.493 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.2435000000000003e-05 2024-09-13T11:25:39.493 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:39.493 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:39.493 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:25:39.494 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.561164+0000", 2024-09-13T11:25:39.494 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.1276000000000001e-05 2024-09-13T11:25:39.494 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:39.494 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:39.494 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:25:39.494 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.562360+0000", 2024-09-13T11:25:39.494 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00119596 2024-09-13T11:25:39.494 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:39.494 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:39.494 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:39.494 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.562397+0000", 2024-09-13T11:25:39.495 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.6378999999999998e-05 2024-09-13T11:25:39.495 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:39.495 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:39.495 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:39.495 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:39.495 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:39.495 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDPGPush(3.22 85/78 [PushOp(3:456e8765:::benchmark_data_smithi177_31116_object16212:head, version: 52'1006, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:456e8765:::benchmark_data_smithi177_31116_object16212:head@52'1006, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:25:39.495 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:42.673067+0000", 2024-09-13T11:25:39.495 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 116.807653699, 2024-09-13T11:25:39.495 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.27935889899999999, 2024-09-13T11:25:39.496 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:39.496 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:39.496 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:25:39.496 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:39.496 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:39.496 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:39.496 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.673067+0000", 2024-09-13T11:25:39.496 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:39.496 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:39.496 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:39.497 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:39.497 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.673067+0000", 2024-09-13T11:25:39.497 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:39.497 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:39.497 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:39.497 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:39.497 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.673068+0000", 2024-09-13T11:25:39.497 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.9699999999999994e-07 2024-09-13T11:25:39.497 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:39.497 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:39.497 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:39.497 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.673090+0000", 2024-09-13T11:25:39.498 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.2416999999999999e-05 2024-09-13T11:25:39.498 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:39.498 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:39.498 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:39.498 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.951363+0000", 2024-09-13T11:25:39.498 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.27827291199999998 2024-09-13T11:25:39.498 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:39.498 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:39.498 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:39.498 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.951386+0000", 2024-09-13T11:25:39.498 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.2226999999999999e-05 2024-09-13T11:25:39.498 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:39.499 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:39.499 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:39.499 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.951581+0000", 2024-09-13T11:25:39.499 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000195713 2024-09-13T11:25:39.499 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:39.499 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:39.499 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:25:39.499 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.951605+0000", 2024-09-13T11:25:39.499 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.3767000000000001e-05 2024-09-13T11:25:39.499 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:39.499 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:39.499 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:39.499 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.952426+0000", 2024-09-13T11:25:39.500 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00082086599999999996 2024-09-13T11:25:39.500 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:39.500 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:39.500 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:39.500 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:39.500 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:39.500 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "replica_scrub(pg: 3.20,from:0'0,to:89'3254,epoch:89/77,start:3:04000000::::0,end:3:045ba805:::benchmark_data_smithi177_31116_object43251:0,chunky:1,deep:0,version:9,allow_preemption:1,priority=5)", 2024-09-13T11:25:39.500 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:46.641364+0000", 2024-09-13T11:25:39.500 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 112.839357018, 2024-09-13T11:25:39.500 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.27141393000000003, 2024-09-13T11:25:39.500 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:39.500 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:39.501 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:25:39.501 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:39.501 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:39.501 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:39.501 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.641364+0000", 2024-09-13T11:25:39.501 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:39.501 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:39.501 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:39.501 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:39.501 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.641364+0000", 2024-09-13T11:25:39.501 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:39.502 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:39.502 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:39.502 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:39.502 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.641365+0000", 2024-09-13T11:25:39.502 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.266e-06 2024-09-13T11:25:39.502 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:39.502 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:39.502 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:39.502 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.641370+0000", 2024-09-13T11:25:39.502 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.8539999999999996e-06 2024-09-13T11:25:39.502 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:39.502 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:39.503 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:39.503 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.912491+0000", 2024-09-13T11:25:39.503 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.27112103100000001 2024-09-13T11:25:39.503 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:39.503 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:39.503 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:39.503 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.912520+0000", 2024-09-13T11:25:39.503 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.9379000000000001e-05 2024-09-13T11:25:39.503 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:39.503 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:39.503 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:39.503 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.912679+0000", 2024-09-13T11:25:39.504 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00015915800000000001 2024-09-13T11:25:39.504 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:39.504 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:39.504 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:25:39.504 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.912714+0000", 2024-09-13T11:25:39.504 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.4687999999999997e-05 2024-09-13T11:25:39.504 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:39.504 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:39.504 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:39.504 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.912778+0000", 2024-09-13T11:25:39.504 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.3553999999999995e-05 2024-09-13T11:25:39.505 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:39.505 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:39.505 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:39.505 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:39.505 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:39.505 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDPGPush(3.18 89/70 [PushOp(3:1ac6c361:::benchmark_data_smithi177_31116_object44972:head, version: 68'2767, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:1ac6c361:::benchmark_data_smithi177_31116_object44972:head@68'2767, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:25:39.505 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:46.671018+0000", 2024-09-13T11:25:39.505 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 112.809702886, 2024-09-13T11:25:39.505 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.271282931, 2024-09-13T11:25:39.505 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:39.505 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:39.506 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:25:39.506 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:39.506 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:39.506 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:39.506 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.671018+0000", 2024-09-13T11:25:39.506 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:39.506 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:39.506 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:39.506 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:39.506 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.671018+0000", 2024-09-13T11:25:39.506 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:39.507 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:39.507 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:39.507 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:39.507 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.671020+0000", 2024-09-13T11:25:39.507 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.018e-06 2024-09-13T11:25:39.507 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:39.507 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:39.507 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:39.507 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.671097+0000", 2024-09-13T11:25:39.507 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.7218999999999994e-05 2024-09-13T11:25:39.508 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:39.508 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:39.508 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:39.508 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.939773+0000", 2024-09-13T11:25:39.508 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.26867550200000001 2024-09-13T11:25:39.508 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:39.508 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:39.508 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:39.508 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.939791+0000", 2024-09-13T11:25:39.508 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.8185999999999998e-05 2024-09-13T11:25:39.509 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:39.509 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:39.509 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:39.509 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.939926+0000", 2024-09-13T11:25:39.509 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000135291 2024-09-13T11:25:39.509 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:39.509 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:39.509 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:25:39.510 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.939938+0000", 2024-09-13T11:25:39.510 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2393e-05 2024-09-13T11:25:39.510 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:39.510 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:39.510 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:39.510 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.942301+0000", 2024-09-13T11:25:39.510 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.002362322 2024-09-13T11:25:39.510 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:39.510 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:39.510 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:39.510 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:39.511 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:39.511 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "replica_scrub(pg: 3.20,from:0'0,to:90'3264,epoch:90/77,start:3:0650aac8:::benchmark_data_smithi177_31116_object1734:0,end:3:06af01e3:::benchmark_data_smithi177_31116_object31572:0,chunky:1,deep:0,version:9,allow_preemption:1,priority=5)", 2024-09-13T11:25:39.511 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:47.380615+0000", 2024-09-13T11:25:39.511 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 112.10010546300001, 2024-09-13T11:25:39.511 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.51951398400000004, 2024-09-13T11:25:39.511 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:39.511 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:39.511 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:25:39.511 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:39.511 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:39.511 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:39.511 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:47.380615+0000", 2024-09-13T11:25:39.512 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:39.512 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:39.512 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:39.512 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:39.512 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:47.380615+0000", 2024-09-13T11:25:39.512 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:39.512 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:39.512 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:39.512 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:39.512 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:47.380617+0000", 2024-09-13T11:25:39.512 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.671e-06 2024-09-13T11:25:39.512 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:39.513 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:39.513 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:39.513 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:47.380623+0000", 2024-09-13T11:25:39.513 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.1190000000000002e-06 2024-09-13T11:25:39.513 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:39.513 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:39.513 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:39.513 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:47.900050+0000", 2024-09-13T11:25:39.513 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.51942737400000005 2024-09-13T11:25:39.513 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:39.513 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:39.514 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:39.514 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:47.900057+0000", 2024-09-13T11:25:39.514 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.4359999999999999e-06 2024-09-13T11:25:39.514 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:39.514 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:39.514 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:39.514 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:47.900101+0000", 2024-09-13T11:25:39.514 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.4675000000000002e-05 2024-09-13T11:25:39.514 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:39.514 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:39.514 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:25:39.515 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:47.900110+0000", 2024-09-13T11:25:39.515 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.0569999999999993e-06 2024-09-13T11:25:39.515 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:39.515 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:39.515 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:39.515 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:47.900129+0000", 2024-09-13T11:25:39.515 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.9652000000000001e-05 2024-09-13T11:25:39.515 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:39.515 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:39.515 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:39.515 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:39.515 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:39.515 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDPGPush(3.21 91/78 [PushOp(3:8492321f:::benchmark_data_smithi177_31116_object13767:head, version: 50'840, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:8492321f:::benchmark_data_smithi177_31116_object13767:head@50'840, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:25:39.516 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:48.319251+0000", 2024-09-13T11:25:39.516 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 111.161469934, 2024-09-13T11:25:39.516 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.60903802200000001, 2024-09-13T11:25:39.516 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:39.516 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:39.516 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:25:39.516 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:39.516 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:39.516 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:39.516 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:48.319251+0000", 2024-09-13T11:25:39.516 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:39.517 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:39.517 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:39.517 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:39.517 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:48.319251+0000", 2024-09-13T11:25:39.517 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:39.517 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:39.517 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:39.517 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:39.517 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:48.319252+0000", 2024-09-13T11:25:39.517 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2640000000000001e-06 2024-09-13T11:25:39.517 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:39.518 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:39.518 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:39.518 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:48.319363+0000", 2024-09-13T11:25:39.518 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000110559 2024-09-13T11:25:39.518 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:39.518 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:39.518 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:39.518 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:48.927115+0000", 2024-09-13T11:25:39.518 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.60775277800000005 2024-09-13T11:25:39.518 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:39.518 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:39.518 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:39.519 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:48.927144+0000", 2024-09-13T11:25:39.519 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.8861e-05 2024-09-13T11:25:39.519 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:39.519 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:39.519 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:39.519 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:48.927318+0000", 2024-09-13T11:25:39.519 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00017399000000000001 2024-09-13T11:25:39.519 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:39.519 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:39.519 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:25:39.519 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:48.927340+0000", 2024-09-13T11:25:39.519 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.1433999999999998e-05 2024-09-13T11:25:39.519 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:39.520 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:39.520 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:39.520 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:48.928289+0000", 2024-09-13T11:25:39.520 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00094913599999999995 2024-09-13T11:25:39.520 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:39.520 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:39.520 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:39.520 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:39.520 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:39.520 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4665.0:69556 3.18 e94/91 3:1d3dbeba:::benchmark_data_smithi177_31116_object69555:head v 94'3570, mlcod=94'3566)", 2024-09-13T11:25:39.520 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:52.142942+0000", 2024-09-13T11:25:39.521 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 107.337778399, 2024-09-13T11:25:39.521 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.21178485399999999, 2024-09-13T11:25:39.521 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:39.521 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:39.521 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:39.521 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:39.521 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:39.521 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:39.521 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:52.142942+0000", 2024-09-13T11:25:39.521 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:39.521 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:39.522 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:39.522 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:39.522 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:52.142942+0000", 2024-09-13T11:25:39.522 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:39.522 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:39.522 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:39.522 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:39.522 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:52.142944+0000", 2024-09-13T11:25:39.522 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.4500000000000001e-06 2024-09-13T11:25:39.522 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:39.522 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:39.522 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:39.523 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:52.143047+0000", 2024-09-13T11:25:39.523 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000103384 2024-09-13T11:25:39.523 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:39.523 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:39.523 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:39.523 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:52.353322+0000", 2024-09-13T11:25:39.523 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.21027483299999999 2024-09-13T11:25:39.523 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:39.523 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:39.523 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:39.523 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:52.353330+0000", 2024-09-13T11:25:39.523 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.8879999999999997e-06 2024-09-13T11:25:39.523 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:39.524 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:39.524 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:39.524 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:52.353402+0000", 2024-09-13T11:25:39.524 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.2155000000000003e-05 2024-09-13T11:25:39.524 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:39.524 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:39.524 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:25:39.524 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:52.353426+0000", 2024-09-13T11:25:39.524 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.3774e-05 2024-09-13T11:25:39.524 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:39.524 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:39.524 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:25:39.525 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:52.354674+0000", 2024-09-13T11:25:39.525 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.001248706 2024-09-13T11:25:39.525 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:39.525 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:39.525 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:39.525 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:52.354727+0000", 2024-09-13T11:25:39.525 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.2664000000000003e-05 2024-09-13T11:25:39.525 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:39.525 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:39.525 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:39.525 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:39.525 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:39.526 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:70749 3.1e 3:7bd7bfc6:::benchmark_data_smithi177_31116_object70748:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95)", 2024-09-13T11:25:39.526 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:53.156250+0000", 2024-09-13T11:25:39.526 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 106.324470578, 2024-09-13T11:25:39.526 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.22394481799999999, 2024-09-13T11:25:39.526 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:39.526 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:39.526 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:39.526 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:25:39.526 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:25:39.526 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:39.526 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 70749 2024-09-13T11:25:39.526 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:39.527 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:39.527 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:39.527 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:39.527 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:53.156250+0000", 2024-09-13T11:25:39.527 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:39.527 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:39.527 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:39.527 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:39.527 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:53.156250+0000", 2024-09-13T11:25:39.527 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:39.528 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:39.528 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:39.528 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:39.528 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:53.156252+0000", 2024-09-13T11:25:39.528 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.3460000000000001e-06 2024-09-13T11:25:39.528 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:39.528 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:39.528 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:39.528 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:53.156336+0000", 2024-09-13T11:25:39.529 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.3714999999999994e-05 2024-09-13T11:25:39.529 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:39.529 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:39.529 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:39.529 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:53.156347+0000", 2024-09-13T11:25:39.529 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0903999999999999e-05 2024-09-13T11:25:39.529 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:39.529 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:39.529 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:39.529 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:53.156352+0000", 2024-09-13T11:25:39.529 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.9139999999999999e-06 2024-09-13T11:25:39.529 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:39.530 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:39.530 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:39.530 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:53.158028+0000", 2024-09-13T11:25:39.530 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.001675958 2024-09-13T11:25:39.530 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:39.530 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:39.530 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:25:39.530 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:53.158154+0000", 2024-09-13T11:25:39.530 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00012653 2024-09-13T11:25:39.530 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:39.530 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:39.530 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 4,5", 2024-09-13T11:25:39.530 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:53.158277+0000", 2024-09-13T11:25:39.531 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000122168 2024-09-13T11:25:39.531 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:39.531 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:39.531 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:25:39.531 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:53.159585+0000", 2024-09-13T11:25:39.531 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.001308528 2024-09-13T11:25:39.531 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:39.531 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:39.531 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:25:39.531 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:53.163172+0000", 2024-09-13T11:25:39.531 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0035869690000000002 2024-09-13T11:25:39.531 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:39.531 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:39.531 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:25:39.532 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:53.379730+0000", 2024-09-13T11:25:39.532 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.21655822599999999 2024-09-13T11:25:39.532 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:39.532 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:39.532 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:25:39.532 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:53.379810+0000", 2024-09-13T11:25:39.532 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.0041000000000002e-05 2024-09-13T11:25:39.532 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:39.532 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:39.532 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:39.532 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:53.380195+0000", 2024-09-13T11:25:39.532 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000384519 2024-09-13T11:25:39.533 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:39.533 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:39.533 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:39.533 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:39.533 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:39.533 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDPGPush(3.21 96/78 [PushOp(3:862d73af:::benchmark_data_smithi177_31116_object32719:head, version: 60'2029, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:862d73af:::benchmark_data_smithi177_31116_object32719:head@60'2029, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:25:39.533 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:53.884628+0000", 2024-09-13T11:25:39.533 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 105.596092736, 2024-09-13T11:25:39.533 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.21020203400000001, 2024-09-13T11:25:39.533 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:39.533 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:39.534 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:25:39.534 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:39.534 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:39.534 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:39.534 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:53.884628+0000", 2024-09-13T11:25:39.534 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:39.534 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:39.534 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:39.534 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:39.534 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:53.884628+0000", 2024-09-13T11:25:39.535 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:39.535 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:39.535 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:39.535 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:39.535 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:53.884630+0000", 2024-09-13T11:25:39.535 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.68e-06 2024-09-13T11:25:39.535 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:39.535 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:39.535 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:39.535 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:53.884745+0000", 2024-09-13T11:25:39.535 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000115599 2024-09-13T11:25:39.535 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:39.536 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:39.536 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:39.536 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:54.093948+0000", 2024-09-13T11:25:39.536 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.20920306999999999 2024-09-13T11:25:39.536 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:39.536 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:39.536 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:39.536 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:54.093961+0000", 2024-09-13T11:25:39.536 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2815e-05 2024-09-13T11:25:39.536 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:39.537 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:39.537 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:39.537 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:54.094139+0000", 2024-09-13T11:25:39.537 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00017824 2024-09-13T11:25:39.537 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:39.537 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:39.537 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:25:39.537 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:54.094150+0000", 2024-09-13T11:25:39.537 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0593e-05 2024-09-13T11:25:39.537 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:39.537 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:39.537 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:39.538 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:54.094830+0000", 2024-09-13T11:25:39.538 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00068003699999999998 2024-09-13T11:25:39.538 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:39.538 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:39.538 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:39.538 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:39.538 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:39.538 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:83239 3.e 3:74547f17:::benchmark_data_smithi177_31116_object83238:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106)", 2024-09-13T11:25:39.538 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:24:04.580085+0000", 2024-09-13T11:25:39.538 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 94.900635852999997, 2024-09-13T11:25:39.539 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.33785762600000002, 2024-09-13T11:25:39.539 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:39.539 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:39.539 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:39.539 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:25:39.539 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:25:39.539 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:39.539 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 83239 2024-09-13T11:25:39.539 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:39.539 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:39.539 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:39.539 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:39.540 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:04.580085+0000", 2024-09-13T11:25:39.540 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:39.540 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:39.540 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:39.540 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:39.540 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:04.580085+0000", 2024-09-13T11:25:39.540 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:39.540 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:39.540 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:39.540 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:39.541 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:04.580089+0000", 2024-09-13T11:25:39.541 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.053e-06 2024-09-13T11:25:39.541 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:39.541 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:39.541 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:39.541 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:04.580166+0000", 2024-09-13T11:25:39.541 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.7102999999999997e-05 2024-09-13T11:25:39.541 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:39.541 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:39.541 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:39.541 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:04.580196+0000", 2024-09-13T11:25:39.542 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.0196e-05 2024-09-13T11:25:39.542 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:39.542 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:39.542 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:39.542 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:04.580212+0000", 2024-09-13T11:25:39.542 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.535e-05 2024-09-13T11:25:39.542 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:39.542 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:39.542 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:39.542 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:04.580336+0000", 2024-09-13T11:25:39.542 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00012431899999999999 2024-09-13T11:25:39.542 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:39.543 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:39.543 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:25:39.543 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:04.580575+0000", 2024-09-13T11:25:39.543 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00023924100000000001 2024-09-13T11:25:39.543 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:39.543 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:39.543 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 1", 2024-09-13T11:25:39.543 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:04.580798+0000", 2024-09-13T11:25:39.543 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00022301499999999999 2024-09-13T11:25:39.543 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:39.543 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:39.543 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:25:39.544 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:04.583404+0000", 2024-09-13T11:25:39.544 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0026056120000000002 2024-09-13T11:25:39.544 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:39.544 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:39.544 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:25:39.544 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:04.917768+0000", 2024-09-13T11:25:39.544 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.33436387499999998 2024-09-13T11:25:39.544 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:39.544 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:39.544 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:25:39.544 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:04.917859+0000", 2024-09-13T11:25:39.545 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.1732999999999997e-05 2024-09-13T11:25:39.545 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:39.545 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:39.545 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:39.545 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:04.917942+0000", 2024-09-13T11:25:39.545 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.3128999999999996e-05 2024-09-13T11:25:39.545 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:39.545 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:39.545 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:39.545 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:39.545 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:39.546 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4665.0:104290 3.20 e115/77 3:05d45b57:::benchmark_data_smithi177_31116_object104289:head v 115'3905, mlcod=115'3902)", 2024-09-13T11:25:39.546 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:24:20.200225+0000", 2024-09-13T11:25:39.546 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 79.280495556999995, 2024-09-13T11:25:39.546 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.21051286899999999, 2024-09-13T11:25:39.546 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:39.546 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:39.546 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:39.546 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:39.546 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:39.546 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:39.547 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:20.200225+0000", 2024-09-13T11:25:39.547 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:39.547 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:39.547 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:39.547 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:39.547 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:20.200225+0000", 2024-09-13T11:25:39.547 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:39.547 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:39.547 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:39.547 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:39.548 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:20.200227+0000", 2024-09-13T11:25:39.548 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.897e-06 2024-09-13T11:25:39.548 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:39.548 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:39.548 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:39.548 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:20.200253+0000", 2024-09-13T11:25:39.548 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.5904000000000001e-05 2024-09-13T11:25:39.548 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:39.548 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:39.548 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:39.548 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:20.409353+0000", 2024-09-13T11:25:39.549 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.209100339 2024-09-13T11:25:39.549 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:39.549 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:39.549 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:39.549 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:20.409359+0000", 2024-09-13T11:25:39.549 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.1689999999999997e-06 2024-09-13T11:25:39.550 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:39.550 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:39.550 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:39.550 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:20.409418+0000", 2024-09-13T11:25:39.550 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.8761999999999997e-05 2024-09-13T11:25:39.550 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:39.550 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:39.550 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:25:39.550 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:20.409440+0000", 2024-09-13T11:25:39.550 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.2098000000000001e-05 2024-09-13T11:25:39.550 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:39.551 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:39.551 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:25:39.551 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:20.410705+0000", 2024-09-13T11:25:39.551 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00126483 2024-09-13T11:25:39.551 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:39.551 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:39.551 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:39.551 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:20.410738+0000", 2024-09-13T11:25:39.551 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.2870000000000002e-05 2024-09-13T11:25:39.551 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:39.551 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:39.552 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:39.552 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:39.552 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:39.552 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:110271 3.2d 3.3c2681ad (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e119)", 2024-09-13T11:25:39.552 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:24:25.403006+0000", 2024-09-13T11:25:39.552 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 74.077714369000006, 2024-09-13T11:25:39.552 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.68768755999999998, 2024-09-13T11:25:39.552 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:39.552 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:39.552 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-13T11:25:39.552 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:25:39.553 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:25:39.553 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:39.553 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 110271 2024-09-13T11:25:39.553 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:39.553 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:39.553 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:39.553 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:39.553 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:25.403006+0000", 2024-09-13T11:25:39.553 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:39.553 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:39.553 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:39.554 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:39.554 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:25.403006+0000", 2024-09-13T11:25:39.554 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:39.554 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:39.554 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:39.554 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:39.554 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:25.403011+0000", 2024-09-13T11:25:39.554 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.5199999999999999e-06 2024-09-13T11:25:39.554 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:39.554 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:39.554 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:39.554 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:25.403065+0000", 2024-09-13T11:25:39.555 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.4478999999999998e-05 2024-09-13T11:25:39.555 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:39.555 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:39.555 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:39.555 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:25.403097+0000", 2024-09-13T11:25:39.555 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.1636e-05 2024-09-13T11:25:39.555 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:39.555 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:39.555 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:39.555 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:25.403121+0000", 2024-09-13T11:25:39.555 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.3924e-05 2024-09-13T11:25:39.556 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:39.556 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:39.556 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:39.556 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:25.403216+0000", 2024-09-13T11:25:39.556 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.5557999999999995e-05 2024-09-13T11:25:39.556 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:39.556 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:39.556 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for peered", 2024-09-13T11:25:39.556 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:25.403234+0000", 2024-09-13T11:25:39.556 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.7104000000000001e-05 2024-09-13T11:25:39.556 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:39.557 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:39.557 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:39.557 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:26.090684+0000", 2024-09-13T11:25:39.557 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.68745000700000003 2024-09-13T11:25:39.558 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:39.558 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:39.558 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:39.558 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:26.090694+0000", 2024-09-13T11:25:39.558 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0332e-05 2024-09-13T11:25:39.558 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:39.558 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:39.558 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:39.558 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:39.558 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:39.559 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDPGPushReply(3.22 125/122 [PushReplyOp(3:475b51c3:::benchmark_data_smithi177_31116_object16005:head)])", 2024-09-13T11:25:39.559 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:24:32.400814+0000", 2024-09-13T11:25:39.559 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 67.079906605000005, 2024-09-13T11:25:39.559 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.20917021799999999, 2024-09-13T11:25:39.559 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:39.559 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:39.559 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-13T11:25:39.559 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:39.559 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:39.559 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:39.560 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:32.400814+0000", 2024-09-13T11:25:39.560 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:39.560 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:39.560 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:39.560 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:39.560 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:32.400814+0000", 2024-09-13T11:25:39.560 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:39.560 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:39.560 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:39.560 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:39.560 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:32.400817+0000", 2024-09-13T11:25:39.560 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.5000000000000002e-06 2024-09-13T11:25:39.561 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:39.561 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:39.561 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:39.561 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:32.400826+0000", 2024-09-13T11:25:39.561 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.8830000000000001e-06 2024-09-13T11:25:39.561 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:39.561 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:39.561 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:39.561 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:32.608921+0000", 2024-09-13T11:25:39.561 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.20809501599999999 2024-09-13T11:25:39.561 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:39.561 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:39.562 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:39.562 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:32.608931+0000", 2024-09-13T11:25:39.562 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.1679999999999993e-06 2024-09-13T11:25:39.562 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:39.562 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:39.562 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:39.562 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:32.608982+0000", 2024-09-13T11:25:39.562 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.1827999999999999e-05 2024-09-13T11:25:39.562 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:39.562 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:39.562 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:39.562 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:32.609984+0000", 2024-09-13T11:25:39.562 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.001001823 2024-09-13T11:25:39.563 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:39.563 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:39.563 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:39.563 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:39.563 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:39.563 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4665.0:129901 3.20 e134/77 3:067f5dfc:::benchmark_data_smithi177_31116_object129900:head v 134'4294, mlcod=134'4291)", 2024-09-13T11:25:39.563 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:24:47.145530+0000", 2024-09-13T11:25:39.563 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 52.335190294, 2024-09-13T11:25:39.563 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.333688349, 2024-09-13T11:25:39.563 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:39.564 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:39.564 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:39.564 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:39.564 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:39.564 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:39.564 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:47.145530+0000", 2024-09-13T11:25:39.564 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:39.564 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:39.564 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:39.564 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:39.564 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:47.145530+0000", 2024-09-13T11:25:39.565 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:39.565 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:39.565 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:39.565 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:39.565 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:47.145532+0000", 2024-09-13T11:25:39.565 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.7999999999999999e-06 2024-09-13T11:25:39.565 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:39.565 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:39.565 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:39.565 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:47.145557+0000", 2024-09-13T11:25:39.565 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.4678999999999999e-05 2024-09-13T11:25:39.566 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:39.566 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:39.566 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:39.566 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:47.477539+0000", 2024-09-13T11:25:39.566 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.33198164099999999 2024-09-13T11:25:39.566 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:39.566 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:39.566 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:39.566 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:47.477544+0000", 2024-09-13T11:25:39.566 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.9059999999999999e-06 2024-09-13T11:25:39.566 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:39.566 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:39.567 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:39.567 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:47.477597+0000", 2024-09-13T11:25:39.567 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.295e-05 2024-09-13T11:25:39.567 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:39.567 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:39.567 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:25:39.567 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:47.477619+0000", 2024-09-13T11:25:39.567 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.1835999999999999e-05 2024-09-13T11:25:39.567 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:39.567 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:39.567 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:25:39.568 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:47.479175+0000", 2024-09-13T11:25:39.568 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0015560470000000001 2024-09-13T11:25:39.568 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:39.568 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:39.568 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:39.568 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:47.479219+0000", 2024-09-13T11:25:39.568 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.3489999999999999e-05 2024-09-13T11:25:39.568 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:39.568 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:39.568 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:39.568 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:39.568 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:39.569 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:140840 3.21 3:87ae3db6:::benchmark_data_smithi177_31116_object537:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e144)", 2024-09-13T11:25:39.569 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:24:58.810939+0000", 2024-09-13T11:25:39.569 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 40.669781180999998, 2024-09-13T11:25:39.569 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.25628309300000002, 2024-09-13T11:25:39.569 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:39.569 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:39.569 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:39.569 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:25:39.569 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:25:39.569 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:39.569 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 140840 2024-09-13T11:25:39.570 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:39.570 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:39.570 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:39.570 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:39.570 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.810939+0000", 2024-09-13T11:25:39.570 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:39.570 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:39.570 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:39.570 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:39.570 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.810939+0000", 2024-09-13T11:25:39.570 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:39.570 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:39.570 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:39.571 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:39.571 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.810942+0000", 2024-09-13T11:25:39.571 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.6259999999999999e-06 2024-09-13T11:25:39.571 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:39.571 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:39.571 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:39.571 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.810950+0000", 2024-09-13T11:25:39.571 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.9500000000000001e-06 2024-09-13T11:25:39.571 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:39.571 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:39.571 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:39.571 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.810970+0000", 2024-09-13T11:25:39.571 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.9539999999999999e-05 2024-09-13T11:25:39.571 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:39.572 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:39.572 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:39.572 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.810978+0000", 2024-09-13T11:25:39.572 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.8680000000000005e-06 2024-09-13T11:25:39.572 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:39.572 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:39.572 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:39.572 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.811060+0000", 2024-09-13T11:25:39.572 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.1068000000000004e-05 2024-09-13T11:25:39.572 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:39.572 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:39.572 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:25:39.573 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.811174+0000", 2024-09-13T11:25:39.573 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000114385 2024-09-13T11:25:39.573 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:39.573 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:39.573 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 8", 2024-09-13T11:25:39.573 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.811255+0000", 2024-09-13T11:25:39.573 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.1298000000000007e-05 2024-09-13T11:25:39.573 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:39.573 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:39.573 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:25:39.573 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.812869+0000", 2024-09-13T11:25:39.574 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.001614234 2024-09-13T11:25:39.574 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:39.574 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:39.574 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:25:39.574 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:59.067155+0000", 2024-09-13T11:25:39.574 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.25428578299999999 2024-09-13T11:25:39.574 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:39.574 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:39.574 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:25:39.574 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:59.067192+0000", 2024-09-13T11:25:39.574 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.659e-05 2024-09-13T11:25:39.575 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:39.575 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:39.575 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:39.575 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:59.067223+0000", 2024-09-13T11:25:39.575 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.0750999999999997e-05 2024-09-13T11:25:39.575 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:39.575 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:39.575 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:39.575 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:39.575 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:39.575 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop_reply(client.4665.0:188185 3.22 e174/122 ondisk, result = 0)", 2024-09-13T11:25:39.575 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:25:36.235023+0000", 2024-09-13T11:25:39.576 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 3.2456973979999999, 2024-09-13T11:25:39.576 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.23136179600000001, 2024-09-13T11:25:39.576 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:39.576 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:39.576 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:25:39.576 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:39.576 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:39.576 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:39.576 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:36.235023+0000", 2024-09-13T11:25:39.576 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:39.576 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:39.576 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:39.576 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:39.577 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:36.235023+0000", 2024-09-13T11:25:39.577 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:39.577 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:39.577 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:39.577 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:39.577 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:36.235024+0000", 2024-09-13T11:25:39.577 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.212e-06 2024-09-13T11:25:39.577 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:39.577 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:39.577 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:39.577 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:36.235029+0000", 2024-09-13T11:25:39.578 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.4839999999999998e-06 2024-09-13T11:25:39.578 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:39.578 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:39.578 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:39.578 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:36.466056+0000", 2024-09-13T11:25:39.578 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.23102687 2024-09-13T11:25:39.578 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:39.578 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:39.578 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:39.578 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:36.466061+0000", 2024-09-13T11:25:39.578 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.8629999999999999e-06 2024-09-13T11:25:39.578 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:39.578 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:39.579 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:39.579 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:36.466133+0000", 2024-09-13T11:25:39.579 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.1999e-05 2024-09-13T11:25:39.579 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:39.579 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:39.579 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:25:39.579 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:36.466148+0000", 2024-09-13T11:25:39.579 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.4800000000000001e-05 2024-09-13T11:25:39.579 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:39.579 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:39.579 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:39.579 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:36.466385+0000", 2024-09-13T11:25:39.579 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00023756800000000001 2024-09-13T11:25:39.580 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:39.580 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:39.580 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:39.580 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:39.580 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:39.580 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:188435 3.22 3:446e4bc0:::benchmark_data_smithi177_31116_object48132:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e174)", 2024-09-13T11:25:39.580 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:25:36.478076+0000", 2024-09-13T11:25:39.580 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 3.0026446120000001, 2024-09-13T11:25:39.580 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.22579131699999999, 2024-09-13T11:25:39.580 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:39.580 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:39.580 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:39.581 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:25:39.581 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:25:39.581 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:39.581 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 188435 2024-09-13T11:25:39.581 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:39.581 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:39.581 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:39.581 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:39.581 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:36.478076+0000", 2024-09-13T11:25:39.581 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:39.581 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:39.582 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:39.582 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:39.582 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:36.478076+0000", 2024-09-13T11:25:39.582 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:39.582 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:39.582 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:39.582 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:39.582 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:36.478078+0000", 2024-09-13T11:25:39.582 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.385e-06 2024-09-13T11:25:39.582 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:39.582 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:39.583 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:39.583 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:36.478085+0000", 2024-09-13T11:25:39.583 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.8900000000000001e-06 2024-09-13T11:25:39.583 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:39.583 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:39.583 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:39.583 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:36.478102+0000", 2024-09-13T11:25:39.583 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.6422999999999998e-05 2024-09-13T11:25:39.583 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:39.583 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:39.583 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:39.583 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:36.478109+0000", 2024-09-13T11:25:39.583 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.7179999999999996e-06 2024-09-13T11:25:39.584 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:39.584 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:39.584 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:39.584 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:36.478173+0000", 2024-09-13T11:25:39.584 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.3398000000000005e-05 2024-09-13T11:25:39.584 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:39.584 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:39.584 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:25:39.584 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:36.478260+0000", 2024-09-13T11:25:39.584 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.7022999999999997e-05 2024-09-13T11:25:39.584 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:39.585 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:39.585 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 5", 2024-09-13T11:25:39.585 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:36.478323+0000", 2024-09-13T11:25:39.585 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.2811000000000006e-05 2024-09-13T11:25:39.585 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:39.585 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:39.585 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:25:39.585 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:36.479692+0000", 2024-09-13T11:25:39.585 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.001369264 2024-09-13T11:25:39.585 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:39.585 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:39.586 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:25:39.586 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:36.703793+0000", 2024-09-13T11:25:39.586 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.224101257 2024-09-13T11:25:39.586 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:39.586 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:39.586 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:25:39.586 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:36.703823+0000", 2024-09-13T11:25:39.586 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.9714999999999999e-05 2024-09-13T11:25:39.586 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:39.586 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:39.586 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:39.587 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:36.703867+0000", 2024-09-13T11:25:39.587 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.4433e-05 2024-09-13T11:25:39.587 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:39.587 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:39.587 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:39.587 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:39.587 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:39.587 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop_reply(client.4665.0:188435 3.22 e174/122 ondisk, result = 0)", 2024-09-13T11:25:39.587 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:25:36.481530+0000", 2024-09-13T11:25:39.587 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 2.9991910719999999, 2024-09-13T11:25:39.587 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.22234125699999999, 2024-09-13T11:25:39.587 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:39.588 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:39.588 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:25:39.588 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:39.588 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:39.588 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:39.588 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:36.481530+0000", 2024-09-13T11:25:39.588 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:39.588 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:39.588 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:39.588 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:39.588 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:36.481530+0000", 2024-09-13T11:25:39.589 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:39.589 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:39.589 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:39.589 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:39.589 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:36.481531+0000", 2024-09-13T11:25:39.589 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.6160000000000001e-06 2024-09-13T11:25:39.589 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:39.589 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:39.589 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:39.589 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:36.481536+0000", 2024-09-13T11:25:39.589 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.2460000000000003e-06 2024-09-13T11:25:39.589 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:39.590 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:39.590 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:39.590 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:36.703742+0000", 2024-09-13T11:25:39.590 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.22220578199999999 2024-09-13T11:25:39.590 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:39.590 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:39.590 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:39.590 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:36.703747+0000", 2024-09-13T11:25:39.590 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.4889999999999997e-06 2024-09-13T11:25:39.590 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:39.590 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:39.591 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:39.591 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:36.703786+0000", 2024-09-13T11:25:39.591 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.9097000000000002e-05 2024-09-13T11:25:39.591 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:39.591 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:39.591 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:25:39.591 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:36.703790+0000", 2024-09-13T11:25:39.591 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.233e-06 2024-09-13T11:25:39.591 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:39.591 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:39.591 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:39.592 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:36.703871+0000", 2024-09-13T11:25:39.592 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.0793999999999999e-05 2024-09-13T11:25:39.592 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:39.592 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:39.592 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:39.592 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:39.592 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:39.592 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4665.0:191286 3.18 e175/91 3:1ffc0b26:::benchmark_data_smithi177_31116_object50983:head v 175'7339, mlcod=175'7337)", 2024-09-13T11:25:39.592 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:25:39.153114+0000", 2024-09-13T11:25:39.592 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 0.32760656900000001, 2024-09-13T11:25:39.593 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.23200507400000001, 2024-09-13T11:25:39.593 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:39.593 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:39.593 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:39.593 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:39.593 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:39.593 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:39.593 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:39.153114+0000", 2024-09-13T11:25:39.593 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:39.593 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:39.593 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:39.593 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:39.594 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:39.153114+0000", 2024-09-13T11:25:39.594 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:39.594 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:39.594 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:39.594 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:39.594 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:39.153117+0000", 2024-09-13T11:25:39.594 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.9170000000000002e-06 2024-09-13T11:25:39.594 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:39.594 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:39.594 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:39.594 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:39.153130+0000", 2024-09-13T11:25:39.594 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2493e-05 2024-09-13T11:25:39.595 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:39.595 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:39.595 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:39.595 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:39.381951+0000", 2024-09-13T11:25:39.595 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.22882103000000001 2024-09-13T11:25:39.595 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:39.595 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:39.595 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:39.595 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:39.381956+0000", 2024-09-13T11:25:39.595 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.1359999999999996e-06 2024-09-13T11:25:39.596 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:39.596 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:39.596 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:39.596 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:39.382030+0000", 2024-09-13T11:25:39.596 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.4270999999999994e-05 2024-09-13T11:25:39.596 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:39.596 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:39.596 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:25:39.596 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:39.382054+0000", 2024-09-13T11:25:39.596 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.4142999999999999e-05 2024-09-13T11:25:39.596 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:39.596 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:39.597 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:25:39.597 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:39.385086+0000", 2024-09-13T11:25:39.597 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00303163 2024-09-13T11:25:39.597 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:39.597 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:39.597 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:39.597 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:39.385119+0000", 2024-09-13T11:25:39.597 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.3454000000000002e-05 2024-09-13T11:25:39.597 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:39.597 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:39.597 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:39.598 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:39.598 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:39.598 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-13T11:25:39.598 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-13T11:25:39.599 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T11:25:39.600 INFO:tasks.ceph.osd.13.smithi092.stderr:2024-09-13T11:25:39.597+0000 7f3e5afea640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26350) UID: 0 2024-09-13T11:25:39.601 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-13T11:25:39.701 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-09-13T11:25:39.696+0000 7f97befc9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26315) UID: 0 2024-09-13T11:25:39.801 INFO:tasks.ceph.osd.12.smithi001.stderr:2024-09-13T11:25:39.799+0000 7f892c6e6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26314) UID: 0 2024-09-13T11:25:39.814 INFO:teuthology.orchestra.run.smithi001.stdout:osd.0: {} 2024-09-13T11:25:39.814 INFO:teuthology.orchestra.run.smithi001.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T11:25:39.826 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-13T11:25:39.826 INFO:teuthology.orchestra.run.smithi001.stdout: "ops": [ 2024-09-13T11:25:39.826 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:39.826 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:191672 3.11 3:8b81ea05:::benchmark_data_smithi177_31116_object51369:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e176)", 2024-09-13T11:25:39.826 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:25:39.675914+0000", 2024-09-13T11:25:39.826 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 0.15011490599999999, 2024-09-13T11:25:39.826 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.15013621599999999, 2024-09-13T11:25:39.827 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:39.827 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:39.827 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "waiting for sub ops", 2024-09-13T11:25:39.827 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:25:39.827 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:25:39.827 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:39.827 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 191672 2024-09-13T11:25:39.827 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:39.827 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:39.827 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:39.827 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:39.827 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:39.675914+0000", 2024-09-13T11:25:39.827 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:39.828 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:39.828 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:39.828 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:39.828 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:39.675914+0000", 2024-09-13T11:25:39.828 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:39.829 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:39.829 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:39.829 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:39.829 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:39.675916+0000", 2024-09-13T11:25:39.829 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.8059999999999999e-06 2024-09-13T11:25:39.829 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:39.829 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:39.829 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:39.829 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:39.675923+0000", 2024-09-13T11:25:39.829 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.3489999999999997e-06 2024-09-13T11:25:39.829 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:39.829 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:39.830 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:39.830 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:39.675940+0000", 2024-09-13T11:25:39.830 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.7489000000000001e-05 2024-09-13T11:25:39.830 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:39.830 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:39.830 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:39.830 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:39.675948+0000", 2024-09-13T11:25:39.830 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.0690000000000002e-06 2024-09-13T11:25:39.830 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:39.830 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:39.830 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:39.830 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:39.676017+0000", 2024-09-13T11:25:39.830 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.8714000000000007e-05 2024-09-13T11:25:39.830 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:39.831 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:39.831 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:25:39.831 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:39.676142+0000", 2024-09-13T11:25:39.831 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000125535 2024-09-13T11:25:39.831 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:39.831 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:39.831 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 11", 2024-09-13T11:25:39.832 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:39.676236+0000", 2024-09-13T11:25:39.832 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.3249000000000001e-05 2024-09-13T11:25:39.832 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:39.832 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:39.832 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:25:39.832 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:39.679269+0000", 2024-09-13T11:25:39.832 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0030331780000000001 2024-09-13T11:25:39.832 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:39.832 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:39.832 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:39.832 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:39.833 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:39.833 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4665.0:191736 3.1e e176/102 3:79fe8e59:::benchmark_data_smithi177_31116_object51433:head v 176'7837, mlcod=176'7834)", 2024-09-13T11:25:39.833 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:25:39.825935+0000", 2024-09-13T11:25:39.833 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 9.3623999999999997e-05, 2024-09-13T11:25:39.833 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00016244299999999999, 2024-09-13T11:25:39.833 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:39.833 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:39.833 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:25:39.833 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:39.833 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:39.833 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:39.833 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:39.825935+0000", 2024-09-13T11:25:39.833 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:39.833 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:39.834 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:39.834 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:39.834 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:39.825935+0000", 2024-09-13T11:25:39.834 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:39.834 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:39.834 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:39.834 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:39.834 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:39.825937+0000", 2024-09-13T11:25:39.834 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.3590000000000002e-06 2024-09-13T11:25:39.834 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:39.834 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:39.834 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:39.834 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:39.825946+0000", 2024-09-13T11:25:39.835 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.8689999999999993e-06 2024-09-13T11:25:39.835 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:39.835 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:39.835 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:39.835 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:39.825976+0000", 2024-09-13T11:25:39.835 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.0108999999999999e-05 2024-09-13T11:25:39.835 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:39.835 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:39.835 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:39.835 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:39.825983+0000", 2024-09-13T11:25:39.835 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.861e-06 2024-09-13T11:25:39.835 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:39.835 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:39.835 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:39.835 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:39.826048+0000", 2024-09-13T11:25:39.836 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.5092999999999995e-05 2024-09-13T11:25:39.836 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:39.836 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:39.836 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:25:39.836 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:39.826076+0000", 2024-09-13T11:25:39.836 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.7676999999999999e-05 2024-09-13T11:25:39.836 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:39.836 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:39.836 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:39.836 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:39.836 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-09-13T11:25:39.836 INFO:teuthology.orchestra.run.smithi001.stdout: "num_ops": 2 2024-09-13T11:25:39.836 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-13T11:25:39.844 INFO:teuthology.orchestra.run.smithi001.stdout:osd.1: {} 2024-09-13T11:25:39.844 INFO:teuthology.orchestra.run.smithi001.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T11:25:39.845 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-13T11:25:39.871 INFO:teuthology.orchestra.run.smithi001.stdout:osd.2: {} 2024-09-13T11:25:39.872 INFO:teuthology.orchestra.run.smithi001.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T11:25:39.899 INFO:teuthology.orchestra.run.smithi001.stdout:osd.3: {} 2024-09-13T11:25:39.899 INFO:teuthology.orchestra.run.smithi001.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T11:25:39.904 INFO:tasks.ceph.osd.7.smithi177.stderr:2024-09-13T11:25:39.902+0000 7f897b51d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26575) UID: 0 2024-09-13T11:25:39.923 INFO:teuthology.orchestra.run.smithi001.stdout:osd.4: {} 2024-09-13T11:25:39.923 INFO:teuthology.orchestra.run.smithi001.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T11:25:39.951 INFO:teuthology.orchestra.run.smithi001.stdout:osd.5: {} 2024-09-13T11:25:39.951 INFO:teuthology.orchestra.run.smithi001.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T11:25:39.982 INFO:teuthology.orchestra.run.smithi001.stdout:osd.6: {} 2024-09-13T11:25:39.982 INFO:teuthology.orchestra.run.smithi001.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T11:25:40.003 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-13T11:25:40.003+0000 7f6a68775640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26307) UID: 0 2024-09-13T11:25:40.004 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-13T11:25:40.004 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:25:40.014 INFO:teuthology.orchestra.run.smithi001.stdout:osd.7: {} 2024-09-13T11:25:40.015 INFO:teuthology.orchestra.run.smithi001.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T11:25:40.017 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-13T11:25:40.018 INFO:teuthology.orchestra.run.smithi001.stderr:dumped all 2024-09-13T11:25:40.024 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-13T11:25:40.037 INFO:teuthology.orchestra.run.smithi001.stdout:osd.8: {} 2024-09-13T11:25:40.037 INFO:teuthology.orchestra.run.smithi001.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-13T11:25:40.038 INFO:teuthology.orchestra.run.smithi001.stdout:{"pg_ready":true,"pg_map":{"version":266,"stamp":"2024-09-13T11:25:39.326962+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":6054314571,"num_objects":92378,"num_object_clones":0,"num_object_copies":184756,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":220,"num_objects_unfound":0,"num_objects_dirty":92378,"num_whiteouts":0,"num_read":87,"num_read_kb":74,"num_write":104469,"num_write_kb":3637107,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22106,"num_bytes_recovered":1448607763,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":297821,"ondisk_log_size":297821,"up":110,"acting":110,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":118,"num_osds":15,"num_per_pool_osds":15,"num_per_pool_omap_osds":15,"kb":1415577600,"kb_used":13982232,"kb_used_data":12579368,"kb_used_omap":26,"kb_used_meta":1402597,"kb_avail":1401595368,"statfs":{"total":1449551462400,"available":1435233656832,"internally_reserved":0,"allocated":12881272832,"data_stored":12869895461,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":26763,"internal_metadata":1436260213},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,4],"upper_bound":16},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-420675584,"num_objects":-6419,"num_object_clones":0,"num_object_copies":-12838,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1246,"num_objects_unfound":0,"num_objects_dirty":-6419,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6419,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":623,"num_bytes_recovered":40828928,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.162834"},"pg_stats":[{"pgid":"3.2d","version":"173'6306","reported_seq":11475,"reported_epoch":173,"state":"active+recovering+remapped","last_fresh":"2024-09-13T11:25:35.084123+0000","last_change":"2024-09-13T11:25:19.656321+0000","last_active":"2024-09-13T11:25:35.084123+0000","last_peered":"2024-09-13T11:25:35.084123+0000","last_clean":"2024-09-13T11:24:24.974950+0000","last_became_active":"2024-09-13T11:24:27.227288+0000","last_became_peered":"2024-09-13T11:24:27.227288+0000","last_unstale":"2024-09-13T11:25:35.084123+0000","last_undegraded":"2024-09-13T11:25:35.084123+0000","last_fullsized":"2024-09-13T11:25:35.084123+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":6,"last_scrub":"76'3395","last_scrub_stamp":"2024-09-13T11:23:32.776747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:32.776747+0000","objects_scrubbed":1704,"log_size":6306,"log_dups_size":0,"ondisk_log_size":6306,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":179,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":100171776,"num_objects":1529,"num_object_clones":0,"num_object_copies":3058,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":220,"num_objects_unfound":0,"num_objects_dirty":1529,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2069,"num_write_kb":87840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1609,"num_bytes_recovered":105447424,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[3,0],"avail_no_missing":["3","7"],"object_location_counts":[{"shards":"0,3","objects":1419},{"shards":"3,7","objects":110}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2c","version":"174'6425","reported_seq":11773,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-13T11:25:35.967991+0000","last_change":"2024-09-13T11:24:59.456762+0000","last_active":"2024-09-13T11:25:35.967991+0000","last_peered":"2024-09-13T11:25:35.967991+0000","last_clean":"2024-09-13T11:25:35.967991+0000","last_became_active":"2024-09-13T11:24:59.021895+0000","last_became_peered":"2024-09-13T11:24:59.021895+0000","last_unstale":"2024-09-13T11:25:35.967991+0000","last_undegraded":"2024-09-13T11:25:35.967991+0000","last_fullsized":"2024-09-13T11:25:35.967991+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":6,"last_scrub":"145'5672","last_scrub_stamp":"2024-09-13T11:24:59.456653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:24:59.456653+0000","objects_scrubbed":2192,"log_size":6425,"log_dups_size":0,"ondisk_log_size":6425,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:26:20.877991+0000","scrub_duration":279,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":94306304,"num_objects":1439,"num_object_clones":0,"num_object_copies":2878,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1439,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":753,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,7],"acting":[10,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.2b","version":"174'6281","reported_seq":9986,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-13T11:25:35.337136+0000","last_change":"2024-09-13T11:24:45.271477+0000","last_active":"2024-09-13T11:25:35.337136+0000","last_peered":"2024-09-13T11:25:35.337136+0000","last_clean":"2024-09-13T11:25:35.337136+0000","last_became_active":"2024-09-13T11:24:26.180578+0000","last_became_peered":"2024-09-13T11:24:26.180578+0000","last_unstale":"2024-09-13T11:25:35.337136+0000","last_undegraded":"2024-09-13T11:25:35.337136+0000","last_fullsized":"2024-09-13T11:25:35.337136+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":6,"last_scrub":"131'5382","last_scrub_stamp":"2024-09-13T11:24:45.271376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:24:45.271376+0000","objects_scrubbed":1940,"log_size":6281,"log_dups_size":0,"ondisk_log_size":6281,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:59.210895+0000","scrub_duration":279,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":89980928,"num_objects":1373,"num_object_clones":0,"num_object_copies":2746,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1373,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":899,"num_write_kb":10624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2a","version":"174'6430","reported_seq":10322,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-13T11:25:36.070639+0000","last_change":"2024-09-13T11:24:46.625117+0000","last_active":"2024-09-13T11:25:36.070639+0000","last_peered":"2024-09-13T11:25:36.070639+0000","last_clean":"2024-09-13T11:25:36.070639+0000","last_became_active":"2024-09-13T11:24:45.320540+0000","last_became_peered":"2024-09-13T11:24:45.320540+0000","last_unstale":"2024-09-13T11:25:36.070639+0000","last_undegraded":"2024-09-13T11:25:36.070639+0000","last_fullsized":"2024-09-13T11:25:36.070639+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":6,"last_scrub":"133'5491","last_scrub_stamp":"2024-09-13T11:24:46.624970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:24:46.624970+0000","objects_scrubbed":2094,"log_size":6430,"log_dups_size":0,"ondisk_log_size":6430,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:26:04.813280+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":98500608,"num_objects":1503,"num_object_clones":0,"num_object_copies":3006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":941,"num_write_kb":11200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,7],"acting":[10,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.29","version":"174'6086","reported_seq":11793,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-13T11:25:35.973234+0000","last_change":"2024-09-13T11:25:13.333062+0000","last_active":"2024-09-13T11:25:35.973234+0000","last_peered":"2024-09-13T11:25:35.973234+0000","last_clean":"2024-09-13T11:25:35.973234+0000","last_became_active":"2024-09-13T11:25:12.944028+0000","last_became_peered":"2024-09-13T11:25:12.944028+0000","last_unstale":"2024-09-13T11:25:35.973234+0000","last_undegraded":"2024-09-13T11:25:35.973234+0000","last_fullsized":"2024-09-13T11:25:35.973234+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":6,"last_scrub":"154'5579","last_scrub_stamp":"2024-09-13T11:25:13.332959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:13.332959+0000","objects_scrubbed":1937,"log_size":6086,"log_dups_size":0,"ondisk_log_size":6086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:26:24.089228+0000","scrub_duration":243,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":93585408,"num_objects":1428,"num_object_clones":0,"num_object_copies":2856,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1428,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":509,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.28","version":"174'5935","reported_seq":11365,"reported_epoch":174,"state":"active+clean+scrubbing","last_fresh":"2024-09-13T11:25:36.085306+0000","last_change":"2024-09-13T11:25:34.995273+0000","last_active":"2024-09-13T11:25:36.085306+0000","last_peered":"2024-09-13T11:25:36.085306+0000","last_clean":"2024-09-13T11:25:36.085306+0000","last_became_active":"2024-09-13T11:25:34.278640+0000","last_became_peered":"2024-09-13T11:25:34.278640+0000","last_unstale":"2024-09-13T11:25:36.085306+0000","last_undegraded":"2024-09-13T11:25:36.085306+0000","last_fullsized":"2024-09-13T11:25:36.085306+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":6,"last_scrub":"68'2771","last_scrub_stamp":"2024-09-13T11:23:20.644747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:20.644747+0000","objects_scrubbed":1399,"log_size":5935,"log_dups_size":0,"ondisk_log_size":5935,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"scrubbing for 1s","scrub_duration":411,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":94371840,"num_objects":1440,"num_object_clones":0,"num_object_copies":2880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1440,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2188,"num_write_kb":94400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1686,"num_bytes_recovered":110493696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5],"acting":[8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.27","version":"174'6006","reported_seq":11536,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-13T11:25:35.917658+0000","last_change":"2024-09-13T11:25:31.204736+0000","last_active":"2024-09-13T11:25:35.917658+0000","last_peered":"2024-09-13T11:25:35.917658+0000","last_clean":"2024-09-13T11:25:35.917658+0000","last_became_active":"2024-09-13T11:25:30.257911+0000","last_became_peered":"2024-09-13T11:25:30.257911+0000","last_unstale":"2024-09-13T11:25:35.917658+0000","last_undegraded":"2024-09-13T11:25:35.917658+0000","last_fullsized":"2024-09-13T11:25:35.917658+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":6,"last_scrub":"68'2736","last_scrub_stamp":"2024-09-13T11:23:20.443329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:20.443329+0000","objects_scrubbed":0,"log_size":6006,"log_dups_size":0,"ondisk_log_size":6006,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"user requested scrub scheduled @ 2024-09-13T11:25:36.159492+0000","scrub_duration":183,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96206848,"num_objects":1468,"num_object_clones":0,"num_object_copies":2936,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1468,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2284,"num_write_kb":98144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1725,"num_bytes_recovered":113049600,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,11],"acting":[5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.26","version":"174'6127","reported_seq":11714,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-13T11:25:35.931196+0000","last_change":"2024-09-13T11:25:01.697082+0000","last_active":"2024-09-13T11:25:35.931196+0000","last_peered":"2024-09-13T11:25:35.931196+0000","last_clean":"2024-09-13T11:25:35.931196+0000","last_became_active":"2024-09-13T11:24:54.976919+0000","last_became_peered":"2024-09-13T11:24:54.976919+0000","last_unstale":"2024-09-13T11:25:35.931196+0000","last_undegraded":"2024-09-13T11:25:35.931196+0000","last_fullsized":"2024-09-13T11:25:35.931196+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":6,"last_scrub":"147'5433","last_scrub_stamp":"2024-09-13T11:25:01.696886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:01.696886+0000","objects_scrubbed":2158,"log_size":6127,"log_dups_size":0,"ondisk_log_size":6127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:26:22.070621+0000","scrub_duration":403,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":95420416,"num_objects":1456,"num_object_clones":0,"num_object_copies":2912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1456,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":702,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,12],"acting":[8,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.25","version":"174'5965","reported_seq":11711,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-13T11:25:36.091753+0000","last_change":"2024-09-13T11:24:57.073162+0000","last_active":"2024-09-13T11:25:36.091753+0000","last_peered":"2024-09-13T11:25:36.091753+0000","last_clean":"2024-09-13T11:25:36.091753+0000","last_became_active":"2024-09-13T11:24:55.999019+0000","last_became_peered":"2024-09-13T11:24:55.999019+0000","last_unstale":"2024-09-13T11:25:36.091753+0000","last_undegraded":"2024-09-13T11:25:36.091753+0000","last_fullsized":"2024-09-13T11:25:36.091753+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":6,"last_scrub":"142'5214","last_scrub_stamp":"2024-09-13T11:24:57.073061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:24:57.073061+0000","objects_scrubbed":2249,"log_size":5965,"log_dups_size":0,"ondisk_log_size":5965,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:26:12.348519+0000","scrub_duration":571,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":98369536,"num_objects":1501,"num_object_clones":0,"num_object_copies":3002,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1501,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":754,"num_write_kb":192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,2],"acting":[12,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[]},{"pgid":"3.24","version":"174'6026","reported_seq":9958,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-13T11:25:37.032528+0000","last_change":"2024-09-13T11:24:48.048846+0000","last_active":"2024-09-13T11:25:37.032528+0000","last_peered":"2024-09-13T11:25:37.032528+0000","last_clean":"2024-09-13T11:25:37.032528+0000","last_became_active":"2024-09-13T11:24:46.936994+0000","last_became_peered":"2024-09-13T11:24:46.936994+0000","last_unstale":"2024-09-13T11:25:37.032528+0000","last_undegraded":"2024-09-13T11:25:37.032528+0000","last_fullsized":"2024-09-13T11:25:37.032528+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":6,"last_scrub":"135'5093","last_scrub_stamp":"2024-09-13T11:24:48.048764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:24:48.048764+0000","objects_scrubbed":2102,"log_size":6026,"log_dups_size":0,"ondisk_log_size":6026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:53.485370+0000","scrub_duration":287,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":95551488,"num_objects":1458,"num_object_clones":0,"num_object_copies":2916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1458,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":934,"num_write_kb":9280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1],"acting":[9,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.23","version":"174'5247","reported_seq":8114,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-13T11:25:35.945765+0000","last_change":"2024-09-13T11:23:48.133184+0000","last_active":"2024-09-13T11:25:35.945765+0000","last_peered":"2024-09-13T11:25:35.945765+0000","last_clean":"2024-09-13T11:25:35.945765+0000","last_became_active":"2024-09-13T11:23:47.926677+0000","last_became_peered":"2024-09-13T11:23:47.926677+0000","last_unstale":"2024-09-13T11:25:35.945765+0000","last_undegraded":"2024-09-13T11:25:35.945765+0000","last_fullsized":"2024-09-13T11:25:35.945765+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":6,"last_scrub":"91'3278","last_scrub_stamp":"2024-09-13T11:23:48.133084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:48.133084+0000","objects_scrubbed":1029,"log_size":5247,"log_dups_size":0,"ondisk_log_size":5247,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":139,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":98238464,"num_objects":1499,"num_object_clones":0,"num_object_copies":2998,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1499,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1970,"num_write_kb":78080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9],"acting":[5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.22","version":"174'5159","reported_seq":10755,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-13T11:25:35.960754+0000","last_change":"2024-09-13T11:24:45.485711+0000","last_active":"2024-09-13T11:25:35.960754+0000","last_peered":"2024-09-13T11:25:35.960754+0000","last_clean":"2024-09-13T11:25:35.960754+0000","last_became_active":"2024-09-13T11:24:29.253848+0000","last_became_peered":"2024-09-13T11:24:29.253848+0000","last_unstale":"2024-09-13T11:25:35.960754+0000","last_undegraded":"2024-09-13T11:25:35.960754+0000","last_fullsized":"2024-09-13T11:25:35.960754+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":6,"last_scrub":"90'3246","last_scrub_stamp":"2024-09-13T11:23:47.300339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:47.300339+0000","objects_scrubbed":1033,"log_size":5159,"log_dups_size":0,"ondisk_log_size":5159,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":203,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":97255424,"num_objects":1484,"num_object_clones":0,"num_object_copies":2968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1484,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1915,"num_write_kb":75712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1768,"num_bytes_recovered":115867648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.21","version":"174'5079","reported_seq":7961,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-13T11:25:36.054055+0000","last_change":"2024-09-13T11:24:12.564117+0000","last_active":"2024-09-13T11:25:36.054055+0000","last_peered":"2024-09-13T11:25:36.054055+0000","last_clean":"2024-09-13T11:25:36.054055+0000","last_became_active":"2024-09-13T11:24:01.250308+0000","last_became_peered":"2024-09-13T11:24:01.250308+0000","last_unstale":"2024-09-13T11:25:36.054055+0000","last_undegraded":"2024-09-13T11:25:36.054055+0000","last_fullsized":"2024-09-13T11:25:36.054055+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":6,"last_scrub":"112'3670","last_scrub_stamp":"2024-09-13T11:24:12.564031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:24:12.564031+0000","objects_scrubbed":1431,"log_size":5079,"log_dups_size":0,"ondisk_log_size":5079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:36.766299+0000","scrub_duration":183,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":91815936,"num_objects":1401,"num_object_clones":0,"num_object_copies":2802,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1401,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1412,"num_write_kb":44224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8],"acting":[0,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.20","version":"174'5244","reported_seq":7436,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-13T11:25:35.940585+0000","last_change":"2024-09-13T11:25:35.773483+0000","last_active":"2024-09-13T11:25:35.940585+0000","last_peered":"2024-09-13T11:25:35.940585+0000","last_clean":"2024-09-13T11:25:35.940585+0000","last_became_active":"2024-09-13T11:23:33.954946+0000","last_became_peered":"2024-09-13T11:23:33.954946+0000","last_unstale":"2024-09-13T11:25:35.940585+0000","last_undegraded":"2024-09-13T11:25:35.940585+0000","last_fullsized":"2024-09-13T11:25:35.940585+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":6,"last_scrub":"174'5240","last_scrub_stamp":"2024-09-13T11:25:35.773409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:35.773409+0000","objects_scrubbed":1424,"log_size":5244,"log_dups_size":0,"ondisk_log_size":5244,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:26:53.605828+0000","scrub_duration":215,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92930048,"num_objects":1418,"num_object_clones":0,"num_object_copies":2836,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1418,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1979,"num_write_kb":74752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,0],"acting":[12,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[]},{"pgid":"3.1f","version":"174'7463","reported_seq":19430,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-13T11:25:36.005567+0000","last_change":"2024-09-13T11:25:20.773752+0000","last_active":"2024-09-13T11:25:36.005567+0000","last_peered":"2024-09-13T11:25:36.005567+0000","last_clean":"2024-09-13T11:25:36.005567+0000","last_became_active":"2024-09-13T11:24:29.290326+0000","last_became_peered":"2024-09-13T11:24:29.290326+0000","last_unstale":"2024-09-13T11:25:36.005567+0000","last_undegraded":"2024-09-13T11:25:36.005567+0000","last_fullsized":"2024-09-13T11:25:36.005567+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":5,"last_scrub":"161'6865","last_scrub_stamp":"2024-09-13T11:25:20.773641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:20.773641+0000","objects_scrubbed":3425,"log_size":7463,"log_dups_size":0,"ondisk_log_size":7463,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:26:21.317280+0000","scrub_duration":535,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":184942592,"num_objects":2822,"num_object_clones":0,"num_object_copies":5644,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2822,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3043,"num_write_kb":102848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3306,"num_bytes_recovered":216662016,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5],"acting":[10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1e","version":"174'7708","reported_seq":18241,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-13T11:25:35.989155+0000","last_change":"2024-09-13T11:24:08.338485+0000","last_active":"2024-09-13T11:25:35.989155+0000","last_peered":"2024-09-13T11:25:35.989155+0000","last_clean":"2024-09-13T11:25:35.989155+0000","last_became_active":"2024-09-13T11:24:01.261448+0000","last_became_peered":"2024-09-13T11:24:01.261448+0000","last_unstale":"2024-09-13T11:25:35.989155+0000","last_undegraded":"2024-09-13T11:25:35.989155+0000","last_fullsized":"2024-09-13T11:25:35.989155+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":5,"last_scrub":"110'4537","last_scrub_stamp":"2024-09-13T11:24:08.338354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:24:08.338354+0000","objects_scrubbed":2760,"log_size":7708,"log_dups_size":0,"ondisk_log_size":7708,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":367,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":190054400,"num_objects":2900,"num_object_clones":0,"num_object_copies":5800,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2900,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3180,"num_write_kb":106240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"174'7502","reported_seq":12964,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-13T11:25:36.042351+0000","last_change":"2024-09-13T11:24:12.492592+0000","last_active":"2024-09-13T11:25:36.042351+0000","last_peered":"2024-09-13T11:25:36.042351+0000","last_clean":"2024-09-13T11:25:36.042351+0000","last_became_active":"2024-09-13T11:24:01.258225+0000","last_became_peered":"2024-09-13T11:24:01.258225+0000","last_unstale":"2024-09-13T11:25:36.042351+0000","last_undegraded":"2024-09-13T11:25:36.042351+0000","last_fullsized":"2024-09-13T11:25:36.042351+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":5,"last_scrub":"112'4579","last_scrub_stamp":"2024-09-13T11:24:12.492507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:24:12.492507+0000","objects_scrubbed":2875,"log_size":7502,"log_dups_size":0,"ondisk_log_size":7502,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":375,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":189792280,"num_objects":2897,"num_object_clones":0,"num_object_copies":5794,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2897,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2936,"num_write_kb":94593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,15],"acting":[2,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1c","version":"174'7684","reported_seq":11503,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-13T11:25:37.001667+0000","last_change":"2024-09-13T11:24:00.529312+0000","last_active":"2024-09-13T11:25:37.001667+0000","last_peered":"2024-09-13T11:25:37.001667+0000","last_clean":"2024-09-13T11:25:37.001667+0000","last_became_active":"2024-09-13T11:23:33.989492+0000","last_became_peered":"2024-09-13T11:23:33.989492+0000","last_unstale":"2024-09-13T11:25:37.001667+0000","last_undegraded":"2024-09-13T11:25:37.001667+0000","last_fullsized":"2024-09-13T11:25:37.001667+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":5,"last_scrub":"102'4277","last_scrub_stamp":"2024-09-13T11:24:00.529243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:24:00.529243+0000","objects_scrubbed":2544,"log_size":7684,"log_dups_size":0,"ondisk_log_size":7684,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:37.048397+0000","scrub_duration":259,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":188416000,"num_objects":2875,"num_object_clones":0,"num_object_copies":5750,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2875,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3409,"num_write_kb":119680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7],"acting":[11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1b","version":"174'7639","reported_seq":13280,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-13T11:25:36.109056+0000","last_change":"2024-09-13T11:24:12.392093+0000","last_active":"2024-09-13T11:25:36.109056+0000","last_peered":"2024-09-13T11:25:36.109056+0000","last_clean":"2024-09-13T11:25:36.109056+0000","last_became_active":"2024-09-13T11:24:04.950821+0000","last_became_peered":"2024-09-13T11:24:04.950821+0000","last_unstale":"2024-09-13T11:25:36.109056+0000","last_undegraded":"2024-09-13T11:25:36.109056+0000","last_fullsized":"2024-09-13T11:25:36.109056+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":5,"last_scrub":"112'4665","last_scrub_stamp":"2024-09-13T11:24:12.391890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:24:12.391890+0000","objects_scrubbed":2934,"log_size":7639,"log_dups_size":0,"ondisk_log_size":7639,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:41.846147+0000","scrub_duration":675,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":189857792,"num_objects":2897,"num_object_clones":0,"num_object_copies":5794,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2897,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2995,"num_write_kb":94656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,15],"acting":[12,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[]},{"pgid":"3.1a","version":"174'7491","reported_seq":12793,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-13T11:25:36.043187+0000","last_change":"2024-09-13T11:24:09.556174+0000","last_active":"2024-09-13T11:25:36.043187+0000","last_peered":"2024-09-13T11:25:36.043187+0000","last_clean":"2024-09-13T11:25:36.043187+0000","last_became_active":"2024-09-13T11:24:03.935185+0000","last_became_peered":"2024-09-13T11:24:03.935185+0000","last_unstale":"2024-09-13T11:25:36.043187+0000","last_undegraded":"2024-09-13T11:25:36.043187+0000","last_fullsized":"2024-09-13T11:25:36.043187+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":5,"last_scrub":"110'4433","last_scrub_stamp":"2024-09-13T11:24:09.556022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:24:09.556022+0000","objects_scrubbed":2654,"log_size":7491,"log_dups_size":0,"ondisk_log_size":7491,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:36.486646+0000","scrub_duration":387,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":186712064,"num_objects":2849,"num_object_clones":0,"num_object_copies":5698,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2849,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3069,"num_write_kb":104448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9],"acting":[4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.19","version":"174'7134","reported_seq":11623,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-13T11:25:36.965624+0000","last_change":"2024-09-13T11:25:10.784789+0000","last_active":"2024-09-13T11:25:36.965624+0000","last_peered":"2024-09-13T11:25:36.965624+0000","last_clean":"2024-09-13T11:25:36.965624+0000","last_became_active":"2024-09-13T11:23:41.924941+0000","last_became_peered":"2024-09-13T11:23:41.924941+0000","last_unstale":"2024-09-13T11:25:36.965624+0000","last_undegraded":"2024-09-13T11:25:36.965624+0000","last_fullsized":"2024-09-13T11:25:36.965624+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":5,"last_scrub":"151'6156","last_scrub_stamp":"2024-09-13T11:25:10.784716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:10.784716+0000","objects_scrubbed":3781,"log_size":7134,"log_dups_size":0,"ondisk_log_size":7134,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:26:30.390310+0000","scrub_duration":1131,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":182517760,"num_objects":2785,"num_object_clones":0,"num_object_copies":5570,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2785,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3732,"num_write_kb":145216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1],"acting":[9,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.18","version":"174'7232","reported_seq":11982,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-13T11:25:36.068112+0000","last_change":"2024-09-13T11:24:13.454659+0000","last_active":"2024-09-13T11:25:36.068112+0000","last_peered":"2024-09-13T11:25:36.068112+0000","last_clean":"2024-09-13T11:25:36.068112+0000","last_became_active":"2024-09-13T11:23:48.926014+0000","last_became_peered":"2024-09-13T11:23:48.926014+0000","last_unstale":"2024-09-13T11:25:36.068112+0000","last_undegraded":"2024-09-13T11:25:36.068112+0000","last_fullsized":"2024-09-13T11:25:36.068112+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":5,"last_scrub":"112'4376","last_scrub_stamp":"2024-09-13T11:24:13.454575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:24:13.454575+0000","objects_scrubbed":3036,"log_size":7232,"log_dups_size":0,"ondisk_log_size":7232,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":375,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":188416000,"num_objects":2875,"num_object_clones":0,"num_object_copies":5750,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2875,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2865,"num_write_kb":86528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0],"acting":[10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.17","version":"174'7116","reported_seq":11338,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-13T11:25:36.055838+0000","last_change":"2024-09-13T11:25:12.658562+0000","last_active":"2024-09-13T11:25:36.055838+0000","last_peered":"2024-09-13T11:25:36.055838+0000","last_clean":"2024-09-13T11:25:36.055838+0000","last_became_active":"2024-09-13T11:23:42.935601+0000","last_became_peered":"2024-09-13T11:23:42.935601+0000","last_unstale":"2024-09-13T11:25:36.055838+0000","last_undegraded":"2024-09-13T11:25:36.055838+0000","last_fullsized":"2024-09-13T11:25:36.055838+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":5,"last_scrub":"153'6197","last_scrub_stamp":"2024-09-13T11:25:12.658464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:12.658464+0000","objects_scrubbed":3743,"log_size":7116,"log_dups_size":0,"ondisk_log_size":7116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:26:25.625277+0000","scrub_duration":467,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":185008128,"num_objects":2823,"num_object_clones":0,"num_object_copies":5646,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2823,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3765,"num_write_kb":146688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3],"acting":[10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"174'7497","reported_seq":12355,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-13T11:25:37.015499+0000","last_change":"2024-09-13T11:25:01.903487+0000","last_active":"2024-09-13T11:25:37.015499+0000","last_peered":"2024-09-13T11:25:37.015499+0000","last_clean":"2024-09-13T11:25:37.015499+0000","last_became_active":"2024-09-13T11:23:46.105237+0000","last_became_peered":"2024-09-13T11:23:46.105237+0000","last_unstale":"2024-09-13T11:25:37.015499+0000","last_undegraded":"2024-09-13T11:25:37.015499+0000","last_fullsized":"2024-09-13T11:25:37.015499+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":5,"last_scrub":"147'6035","last_scrub_stamp":"2024-09-13T11:25:01.903395+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:01.903395+0000","objects_scrubbed":4359,"log_size":7497,"log_dups_size":0,"ondisk_log_size":7497,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:26:16.436377+0000","scrub_duration":743,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":188874752,"num_objects":2882,"num_object_clones":0,"num_object_copies":5764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2882,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4022,"num_write_kb":155200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,15],"acting":[11,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.15","version":"174'7111","reported_seq":18452,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-13T11:25:36.097434+0000","last_change":"2024-09-13T11:25:12.198448+0000","last_active":"2024-09-13T11:25:36.097434+0000","last_peered":"2024-09-13T11:25:36.097434+0000","last_clean":"2024-09-13T11:25:36.097434+0000","last_became_active":"2024-09-13T11:24:29.226615+0000","last_became_peered":"2024-09-13T11:24:29.226615+0000","last_unstale":"2024-09-13T11:25:36.097434+0000","last_undegraded":"2024-09-13T11:25:36.097434+0000","last_fullsized":"2024-09-13T11:25:36.097434+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":5,"last_scrub":"91'3384","last_scrub_stamp":"2024-09-13T11:23:48.332600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:48.332600+0000","objects_scrubbed":2032,"log_size":7111,"log_dups_size":0,"ondisk_log_size":7111,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":235,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":184418304,"num_objects":2814,"num_object_clones":0,"num_object_copies":5628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2814,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3728,"num_write_kb":144320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3385,"num_bytes_recovered":221839360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.14","version":"173'7203","reported_seq":18468,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-09-13T11:25:34.992634+0000","last_change":"2024-09-13T11:25:15.378521+0000","last_active":"2024-09-13T11:25:34.992634+0000","last_peered":"2024-09-13T11:25:34.992634+0000","last_clean":"2024-09-13T11:25:34.992634+0000","last_became_active":"2024-09-13T11:24:29.253106+0000","last_became_peered":"2024-09-13T11:24:29.253106+0000","last_unstale":"2024-09-13T11:25:34.992634+0000","last_undegraded":"2024-09-13T11:25:34.992634+0000","last_fullsized":"2024-09-13T11:25:34.992634+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":5,"last_scrub":"91'3371","last_scrub_stamp":"2024-09-13T11:23:48.696960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:48.696960+0000","objects_scrubbed":1935,"log_size":7203,"log_dups_size":0,"ondisk_log_size":7203,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":735,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":199819264,"num_objects":3049,"num_object_clones":0,"num_object_copies":6098,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3049,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3842,"num_write_kb":158592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3437,"num_bytes_recovered":225247232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,15],"acting":[3,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":320,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-13T11:25:35.285144+0000","last_change":"2024-09-13T11:24:23.212494+0000","last_active":"2024-09-13T11:25:35.285144+0000","last_peered":"2024-09-13T11:25:35.285144+0000","last_clean":"2024-09-13T11:25:35.285144+0000","last_became_active":"2024-09-13T11:22:14.192897+0000","last_became_peered":"2024-09-13T11:22:14.192897+0000","last_unstale":"2024-09-13T11:25:35.285144+0000","last_undegraded":"2024-09-13T11:25:35.285144+0000","last_fullsized":"2024-09-13T11:25:35.285144+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:24:23.212406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:24:23.212406+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:36.980197+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00084655899999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,10],"acting":[15,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"3.6","version":"174'6156","reported_seq":10035,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-13T11:25:37.015602+0000","last_change":"2024-09-13T11:24:49.416851+0000","last_active":"2024-09-13T11:25:37.015602+0000","last_peered":"2024-09-13T11:25:37.015602+0000","last_clean":"2024-09-13T11:25:37.015602+0000","last_became_active":"2024-09-13T11:24:29.322558+0000","last_became_peered":"2024-09-13T11:24:29.322558+0000","last_unstale":"2024-09-13T11:25:37.015602+0000","last_undegraded":"2024-09-13T11:25:37.015602+0000","last_fullsized":"2024-09-13T11:25:37.015602+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"136'5216","last_scrub_stamp":"2024-09-13T11:24:49.416706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:24:49.416706+0000","objects_scrubbed":2083,"log_size":6156,"log_dups_size":0,"ondisk_log_size":6156,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:26:09.525086+0000","scrub_duration":247,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":94109696,"num_objects":1436,"num_object_clones":0,"num_object_copies":2872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1436,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":941,"num_write_kb":9408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,6],"acting":[13,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":327,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-13T11:25:35.293272+0000","last_change":"2024-09-13T11:24:59.556253+0000","last_active":"2024-09-13T11:25:35.293272+0000","last_peered":"2024-09-13T11:25:35.293272+0000","last_clean":"2024-09-13T11:25:35.293272+0000","last_became_active":"2024-09-13T11:22:14.211923+0000","last_became_peered":"2024-09-13T11:22:14.211923+0000","last_unstale":"2024-09-13T11:25:35.293272+0000","last_undegraded":"2024-09-13T11:25:35.293272+0000","last_fullsized":"2024-09-13T11:25:35.293272+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:24:59.556180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:24:59.556180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:26:06.229322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011689529999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0],"acting":[9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"174'5957","reported_seq":6330,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-13T11:25:37.061360+0000","last_change":"2024-09-13T11:24:33.336210+0000","last_active":"2024-09-13T11:25:37.061360+0000","last_peered":"2024-09-13T11:25:37.061360+0000","last_clean":"2024-09-13T11:25:37.061360+0000","last_became_active":"2024-09-13T11:24:24.092975+0000","last_became_peered":"2024-09-13T11:24:24.092975+0000","last_unstale":"2024-09-13T11:25:37.061360+0000","last_undegraded":"2024-09-13T11:25:37.061360+0000","last_fullsized":"2024-09-13T11:25:37.061360+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"125'4820","last_scrub_stamp":"2024-09-13T11:24:33.336091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:24:33.336091+0000","objects_scrubbed":1766,"log_size":5957,"log_dups_size":0,"ondisk_log_size":5957,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:54.538352+0000","scrub_duration":263,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":95092736,"num_objects":1451,"num_object_clones":0,"num_object_copies":2902,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1451,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1139,"num_write_kb":26368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,8],"acting":[13,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":328,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-13T11:25:35.283473+0000","last_change":"2024-09-13T11:25:33.957283+0000","last_active":"2024-09-13T11:25:35.283473+0000","last_peered":"2024-09-13T11:25:35.283473+0000","last_clean":"2024-09-13T11:25:35.283473+0000","last_became_active":"2024-09-13T11:22:14.581458+0000","last_became_peered":"2024-09-13T11:22:14.581458+0000","last_unstale":"2024-09-13T11:25:35.283473+0000","last_undegraded":"2024-09-13T11:25:35.283473+0000","last_fullsized":"2024-09-13T11:25:35.283473+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:25:33.957179+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:25:33.957179+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:26:52.126360+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.001026047,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5],"acting":[8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"174'5998","reported_seq":6343,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-13T11:25:35.354435+0000","last_change":"2024-09-13T11:24:24.579036+0000","last_active":"2024-09-13T11:25:35.354435+0000","last_peered":"2024-09-13T11:25:35.354435+0000","last_clean":"2024-09-13T11:25:35.354435+0000","last_became_active":"2024-09-13T11:24:24.083077+0000","last_became_peered":"2024-09-13T11:24:24.083077+0000","last_unstale":"2024-09-13T11:25:35.354435+0000","last_undegraded":"2024-09-13T11:25:35.354435+0000","last_fullsized":"2024-09-13T11:25:35.354435+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'4737","last_scrub_stamp":"2024-09-13T11:24:24.578900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:24:24.578900+0000","objects_scrubbed":1690,"log_size":5998,"log_dups_size":0,"ondisk_log_size":5998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:38.103421+0000","scrub_duration":407,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":94896128,"num_objects":1448,"num_object_clones":0,"num_object_copies":2896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1448,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1264,"num_write_kb":32704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,12],"acting":[1,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":308,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-13T11:25:35.298684+0000","last_change":"2024-09-13T11:24:08.271026+0000","last_active":"2024-09-13T11:25:35.298684+0000","last_peered":"2024-09-13T11:25:35.298684+0000","last_clean":"2024-09-13T11:25:35.298684+0000","last_became_active":"2024-09-13T11:22:47.719636+0000","last_became_peered":"2024-09-13T11:22:47.719636+0000","last_unstale":"2024-09-13T11:25:35.298684+0000","last_undegraded":"2024-09-13T11:25:35.298684+0000","last_fullsized":"2024-09-13T11:25:35.298684+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:24:08.270972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:24:08.270972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:36.805605+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00087476000000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,9],"acting":[1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"174'5958","reported_seq":6322,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-13T11:25:36.019857+0000","last_change":"2024-09-13T11:24:24.616937+0000","last_active":"2024-09-13T11:25:36.019857+0000","last_peered":"2024-09-13T11:25:36.019857+0000","last_clean":"2024-09-13T11:25:36.019857+0000","last_became_active":"2024-09-13T11:24:24.093492+0000","last_became_peered":"2024-09-13T11:24:24.093492+0000","last_unstale":"2024-09-13T11:25:36.019857+0000","last_undegraded":"2024-09-13T11:25:36.019857+0000","last_fullsized":"2024-09-13T11:25:36.019857+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'4744","last_scrub_stamp":"2024-09-13T11:24:24.616633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:24:24.616633+0000","objects_scrubbed":1624,"log_size":5958,"log_dups_size":0,"ondisk_log_size":5958,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:42.911074+0000","scrub_duration":415,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92733440,"num_objects":1415,"num_object_clones":0,"num_object_copies":2830,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1415,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1221,"num_write_kb":32384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,13],"acting":[15,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"3.3","version":"174'5160","reported_seq":5544,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-13T11:25:36.042110+0000","last_change":"2024-09-13T11:25:08.394545+0000","last_active":"2024-09-13T11:25:36.042110+0000","last_peered":"2024-09-13T11:25:36.042110+0000","last_clean":"2024-09-13T11:25:36.042110+0000","last_became_active":"2024-09-13T11:23:31.930448+0000","last_became_peered":"2024-09-13T11:23:31.930448+0000","last_unstale":"2024-09-13T11:25:36.042110+0000","last_undegraded":"2024-09-13T11:25:36.042110+0000","last_fullsized":"2024-09-13T11:25:36.042110+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"151'4587","last_scrub_stamp":"2024-09-13T11:25:08.394441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:08.394441+0000","objects_scrubbed":1942,"log_size":5160,"log_dups_size":0,"ondisk_log_size":5160,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:26:27.641900+0000","scrub_duration":219,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":89522176,"num_objects":1366,"num_object_clones":0,"num_object_copies":2732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1366,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2038,"num_write_kb":80896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,7],"acting":[10,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"28'2","reported_seq":258,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-13T11:25:35.304129+0000","last_change":"2024-09-13T11:25:13.869564+0000","last_active":"2024-09-13T11:25:35.304129+0000","last_peered":"2024-09-13T11:25:35.304129+0000","last_clean":"2024-09-13T11:25:35.304129+0000","last_became_active":"2024-09-13T11:23:12.546348+0000","last_became_peered":"2024-09-13T11:23:12.546348+0000","last_unstale":"2024-09-13T11:25:35.304129+0000","last_undegraded":"2024-09-13T11:25:35.304129+0000","last_fullsized":"2024-09-13T11:25:35.304129+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2","last_scrub_stamp":"2024-09-13T11:25:13.869413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:25:13.869413+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:26:38.761718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0025826730000000002,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8],"acting":[2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"174'5170","reported_seq":5552,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-13T11:25:36.073635+0000","last_change":"2024-09-13T11:25:09.842875+0000","last_active":"2024-09-13T11:25:36.073635+0000","last_peered":"2024-09-13T11:25:36.073635+0000","last_clean":"2024-09-13T11:25:36.073635+0000","last_became_active":"2024-09-13T11:23:31.921388+0000","last_became_peered":"2024-09-13T11:23:31.921388+0000","last_unstale":"2024-09-13T11:25:36.073635+0000","last_undegraded":"2024-09-13T11:25:36.073635+0000","last_fullsized":"2024-09-13T11:25:36.073635+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"151'4644","last_scrub_stamp":"2024-09-13T11:25:09.842728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:09.842728+0000","objects_scrubbed":1944,"log_size":5170,"log_dups_size":0,"ondisk_log_size":5170,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:26:25.553642+0000","scrub_duration":395,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92930048,"num_objects":1418,"num_object_clones":0,"num_object_copies":2836,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1418,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2064,"num_write_kb":84032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,15],"acting":[12,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":169,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-13T11:25:35.298317+0000","last_change":"2024-09-13T11:25:34.822952+0000","last_active":"2024-09-13T11:25:35.298317+0000","last_peered":"2024-09-13T11:25:35.298317+0000","last_clean":"2024-09-13T11:25:35.298317+0000","last_became_active":"2024-09-13T11:24:29.096575+0000","last_became_peered":"2024-09-13T11:24:29.096575+0000","last_unstale":"2024-09-13T11:25:35.298317+0000","last_undegraded":"2024-09-13T11:25:35.298317+0000","last_fullsized":"2024-09-13T11:25:35.298317+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:25:34.822831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:25:34.822831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:27:01.304133+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00081225399999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"174'5139","reported_seq":5524,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-13T11:25:37.061952+0000","last_change":"2024-09-13T11:25:00.861990+0000","last_active":"2024-09-13T11:25:37.061952+0000","last_peered":"2024-09-13T11:25:37.061952+0000","last_clean":"2024-09-13T11:25:37.061952+0000","last_became_active":"2024-09-13T11:23:31.926528+0000","last_became_peered":"2024-09-13T11:23:31.926528+0000","last_unstale":"2024-09-13T11:25:37.061952+0000","last_undegraded":"2024-09-13T11:25:37.061952+0000","last_fullsized":"2024-09-13T11:25:37.061952+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"146'4447","last_scrub_stamp":"2024-09-13T11:25:00.861834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:00.861834+0000","objects_scrubbed":2126,"log_size":5139,"log_dups_size":0,"ondisk_log_size":5139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:26:15.411903+0000","scrub_duration":787,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":93782016,"num_objects":1431,"num_object_clones":0,"num_object_copies":2862,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1431,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2069,"num_write_kb":85056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,11],"acting":[13,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":318,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-09-13T11:25:34.282870+0000","last_change":"2024-09-13T11:24:11.895017+0000","last_active":"2024-09-13T11:25:34.282870+0000","last_peered":"2024-09-13T11:25:34.282870+0000","last_clean":"2024-09-13T11:25:34.282870+0000","last_became_active":"2024-09-13T11:22:14.211435+0000","last_became_peered":"2024-09-13T11:22:14.211435+0000","last_unstale":"2024-09-13T11:25:34.282870+0000","last_undegraded":"2024-09-13T11:25:34.282870+0000","last_fullsized":"2024-09-13T11:25:34.282870+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:24:11.894937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:24:11.894937+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0010010360000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2","version":"174'5195","reported_seq":9042,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-13T11:25:35.957500+0000","last_change":"2024-09-13T11:25:23.765128+0000","last_active":"2024-09-13T11:25:35.957500+0000","last_peered":"2024-09-13T11:25:35.957500+0000","last_clean":"2024-09-13T11:25:35.957500+0000","last_became_active":"2024-09-13T11:24:29.175407+0000","last_became_peered":"2024-09-13T11:24:29.175407+0000","last_unstale":"2024-09-13T11:25:35.957500+0000","last_undegraded":"2024-09-13T11:25:35.957500+0000","last_fullsized":"2024-09-13T11:25:35.957500+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"164'4994","last_scrub_stamp":"2024-09-13T11:25:23.765047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:23.765047+0000","objects_scrubbed":1689,"log_size":5195,"log_dups_size":0,"ondisk_log_size":5195,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:26:45.252942+0000","scrub_duration":283,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":97320960,"num_objects":1485,"num_object_clones":0,"num_object_copies":2970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1485,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2138,"num_write_kb":89024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1731,"num_bytes_recovered":113442816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,3],"acting":[12,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[]},{"pgid":"2.3","version":"26'1","reported_seq":131,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-13T11:25:35.286688+0000","last_change":"2024-09-13T11:25:23.034357+0000","last_active":"2024-09-13T11:25:35.286688+0000","last_peered":"2024-09-13T11:25:35.286688+0000","last_clean":"2024-09-13T11:25:35.286688+0000","last_became_active":"2024-09-13T11:24:29.110028+0000","last_became_peered":"2024-09-13T11:24:29.110028+0000","last_unstale":"2024-09-13T11:25:35.286688+0000","last_undegraded":"2024-09-13T11:25:35.286688+0000","last_fullsized":"2024-09-13T11:25:35.286688+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"26'1","last_scrub_stamp":"2024-09-13T11:25:23.034247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:25:23.034247+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:26:42.429349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,8],"acting":[15,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"1.0","version":"26'112","reported_seq":462,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-13T11:25:35.310912+0000","last_change":"2024-09-13T11:25:23.642366+0000","last_active":"2024-09-13T11:25:35.310912+0000","last_peered":"2024-09-13T11:25:35.310912+0000","last_clean":"2024-09-13T11:25:35.310912+0000","last_became_active":"2024-09-13T11:22:12.129721+0000","last_became_peered":"2024-09-13T11:22:12.129721+0000","last_unstale":"2024-09-13T11:25:35.310912+0000","last_undegraded":"2024-09-13T11:25:35.310912+0000","last_fullsized":"2024-09-13T11:25:35.310912+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"26'112","last_scrub_stamp":"2024-09-13T11:25:23.642204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:11.098388+0000","last_clean_scrub_stamp":"2024-09-13T11:25:23.642204+0000","objects_scrubbed":2,"log_size":112,"log_dups_size":0,"ondisk_log_size":112,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:26:42.628319+0000","scrub_duration":67,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.8","version":"174'5968","reported_seq":6331,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-13T11:25:35.943739+0000","last_change":"2024-09-13T11:24:24.506212+0000","last_active":"2024-09-13T11:25:35.943739+0000","last_peered":"2024-09-13T11:25:35.943739+0000","last_clean":"2024-09-13T11:25:35.943739+0000","last_became_active":"2024-09-13T11:24:24.096220+0000","last_became_peered":"2024-09-13T11:24:24.096220+0000","last_unstale":"2024-09-13T11:25:35.943739+0000","last_undegraded":"2024-09-13T11:25:35.943739+0000","last_fullsized":"2024-09-13T11:25:35.943739+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'4749","last_scrub_stamp":"2024-09-13T11:24:24.506109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:24:24.506109+0000","objects_scrubbed":1690,"log_size":5968,"log_dups_size":0,"ondisk_log_size":5968,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:49.430042+0000","scrub_duration":215,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":94765056,"num_objects":1446,"num_object_clones":0,"num_object_copies":2892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1446,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1220,"num_write_kb":31232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.9","version":"174'6001","reported_seq":6365,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-13T11:25:35.990096+0000","last_change":"2024-09-13T11:24:24.928964+0000","last_active":"2024-09-13T11:25:35.990096+0000","last_peered":"2024-09-13T11:25:35.990096+0000","last_clean":"2024-09-13T11:25:35.990096+0000","last_became_active":"2024-09-13T11:24:24.082504+0000","last_became_peered":"2024-09-13T11:24:24.082504+0000","last_unstale":"2024-09-13T11:25:35.990096+0000","last_undegraded":"2024-09-13T11:25:35.990096+0000","last_fullsized":"2024-09-13T11:25:35.990096+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'4799","last_scrub_stamp":"2024-09-13T11:24:24.928724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:24:24.928724+0000","objects_scrubbed":1705,"log_size":6001,"log_dups_size":0,"ondisk_log_size":6001,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:43.845801+0000","scrub_duration":255,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96468992,"num_objects":1472,"num_object_clones":0,"num_object_copies":2944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1472,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1205,"num_write_kb":31104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,2],"acting":[12,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[]},{"pgid":"3.a","version":"173'6337","reported_seq":6703,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-09-13T11:25:34.984376+0000","last_change":"2024-09-13T11:24:25.137871+0000","last_active":"2024-09-13T11:25:34.984376+0000","last_peered":"2024-09-13T11:25:34.984376+0000","last_clean":"2024-09-13T11:25:34.984376+0000","last_became_active":"2024-09-13T11:24:24.089253+0000","last_became_peered":"2024-09-13T11:24:24.089253+0000","last_unstale":"2024-09-13T11:25:34.984376+0000","last_undegraded":"2024-09-13T11:25:34.984376+0000","last_fullsized":"2024-09-13T11:25:34.984376+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'5157","last_scrub_stamp":"2024-09-13T11:24:25.137777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:24:25.137777+0000","objects_scrubbed":1777,"log_size":6337,"log_dups_size":0,"ondisk_log_size":6337,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:36.149013+0000","scrub_duration":223,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96206848,"num_objects":1468,"num_object_clones":0,"num_object_copies":2936,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1468,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1183,"num_write_kb":27968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.b","version":"174'6346","reported_seq":6713,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-13T11:25:37.072480+0000","last_change":"2024-09-13T11:24:25.503864+0000","last_active":"2024-09-13T11:25:37.072480+0000","last_peered":"2024-09-13T11:25:37.072480+0000","last_clean":"2024-09-13T11:25:37.072480+0000","last_became_active":"2024-09-13T11:24:24.085684+0000","last_became_peered":"2024-09-13T11:24:24.085684+0000","last_unstale":"2024-09-13T11:25:37.072480+0000","last_undegraded":"2024-09-13T11:25:37.072480+0000","last_fullsized":"2024-09-13T11:25:37.072480+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'5112","last_scrub_stamp":"2024-09-13T11:24:25.503716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:24:25.503716+0000","objects_scrubbed":1718,"log_size":6346,"log_dups_size":0,"ondisk_log_size":6346,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:37.636639+0000","scrub_duration":395,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":94502912,"num_objects":1442,"num_object_clones":0,"num_object_copies":2884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1442,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1234,"num_write_kb":30656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,1],"acting":[13,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"3.c","version":"174'6358","reported_seq":6748,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-13T11:25:37.081785+0000","last_change":"2024-09-13T11:25:36.279464+0000","last_active":"2024-09-13T11:25:37.081785+0000","last_peered":"2024-09-13T11:25:37.081785+0000","last_clean":"2024-09-13T11:25:37.081785+0000","last_became_active":"2024-09-13T11:24:24.090847+0000","last_became_peered":"2024-09-13T11:24:24.090847+0000","last_unstale":"2024-09-13T11:25:37.081785+0000","last_undegraded":"2024-09-13T11:25:37.081785+0000","last_fullsized":"2024-09-13T11:25:37.081785+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"174'6334","last_scrub_stamp":"2024-09-13T11:25:36.279390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:36.279390+0000","objects_scrubbed":1456,"log_size":6358,"log_dups_size":0,"ondisk_log_size":6358,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:27:04.013051+0000","scrub_duration":235,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":93782016,"num_objects":1431,"num_object_clones":0,"num_object_copies":2862,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1431,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1209,"num_write_kb":29440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,13],"acting":[11,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.d","version":"174'6325","reported_seq":6692,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-13T11:25:36.063101+0000","last_change":"2024-09-13T11:24:24.967444+0000","last_active":"2024-09-13T11:25:36.063101+0000","last_peered":"2024-09-13T11:25:36.063101+0000","last_clean":"2024-09-13T11:25:36.063101+0000","last_became_active":"2024-09-13T11:24:24.087224+0000","last_became_peered":"2024-09-13T11:24:24.087224+0000","last_unstale":"2024-09-13T11:25:36.063101+0000","last_undegraded":"2024-09-13T11:25:36.063101+0000","last_fullsized":"2024-09-13T11:25:36.063101+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'5110","last_scrub_stamp":"2024-09-13T11:24:24.967337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:24:24.967337+0000","objects_scrubbed":1725,"log_size":6325,"log_dups_size":0,"ondisk_log_size":6325,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:53.620949+0000","scrub_duration":255,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96534528,"num_objects":1473,"num_object_clones":0,"num_object_copies":2946,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1473,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1218,"num_write_kb":30912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.e","version":"174'7669","reported_seq":12933,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-13T11:25:36.052006+0000","last_change":"2024-09-13T11:23:32.368856+0000","last_active":"2024-09-13T11:25:36.052006+0000","last_peered":"2024-09-13T11:25:36.052006+0000","last_clean":"2024-09-13T11:25:36.052006+0000","last_became_active":"2024-09-13T11:23:31.925869+0000","last_became_peered":"2024-09-13T11:23:31.925869+0000","last_unstale":"2024-09-13T11:25:36.052006+0000","last_undegraded":"2024-09-13T11:25:36.052006+0000","last_fullsized":"2024-09-13T11:25:36.052006+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'3527","last_scrub_stamp":"2024-09-13T11:23:32.368727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:32.368727+0000","objects_scrubbed":1774,"log_size":7669,"log_dups_size":0,"ondisk_log_size":7669,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":219,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":184483840,"num_objects":2815,"num_object_clones":0,"num_object_copies":5630,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2815,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4143,"num_write_kb":165888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"174'7673","reported_seq":8060,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-13T11:25:37.077063+0000","last_change":"2024-09-13T11:25:00.289940+0000","last_active":"2024-09-13T11:25:37.077063+0000","last_peered":"2024-09-13T11:25:37.077063+0000","last_clean":"2024-09-13T11:25:37.077063+0000","last_became_active":"2024-09-13T11:23:31.935239+0000","last_became_peered":"2024-09-13T11:23:31.935239+0000","last_unstale":"2024-09-13T11:25:37.077063+0000","last_undegraded":"2024-09-13T11:25:37.077063+0000","last_fullsized":"2024-09-13T11:25:37.077063+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"146'6154","last_scrub_stamp":"2024-09-13T11:25:00.289673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:00.289673+0000","objects_scrubbed":4394,"log_size":7673,"log_dups_size":0,"ondisk_log_size":7673,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:26:24.274856+0000","scrub_duration":1187,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":186974208,"num_objects":2853,"num_object_clones":0,"num_object_copies":5706,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2853,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4253,"num_write_kb":170560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,12],"acting":[11,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.10","version":"174'7299","reported_seq":11837,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-13T11:25:36.026699+0000","last_change":"2024-09-13T11:25:10.161564+0000","last_active":"2024-09-13T11:25:36.026699+0000","last_peered":"2024-09-13T11:25:36.026699+0000","last_clean":"2024-09-13T11:25:36.026699+0000","last_became_active":"2024-09-13T11:23:41.931183+0000","last_became_peered":"2024-09-13T11:23:41.931183+0000","last_unstale":"2024-09-13T11:25:36.026699+0000","last_undegraded":"2024-09-13T11:25:36.026699+0000","last_fullsized":"2024-09-13T11:25:36.026699+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":5,"last_scrub":"151'6298","last_scrub_stamp":"2024-09-13T11:25:10.161464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:10.161464+0000","objects_scrubbed":3964,"log_size":7299,"log_dups_size":0,"ondisk_log_size":7299,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:26:33.015478+0000","scrub_duration":607,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":193200128,"num_objects":2948,"num_object_clones":0,"num_object_copies":5896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2948,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3833,"num_write_kb":152000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.11","version":"174'7300","reported_seq":10611,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-13T11:25:36.022837+0000","last_change":"2024-09-13T11:25:24.287326+0000","last_active":"2024-09-13T11:25:36.022837+0000","last_peered":"2024-09-13T11:25:36.022837+0000","last_clean":"2024-09-13T11:25:36.022837+0000","last_became_active":"2024-09-13T11:23:39.063172+0000","last_became_peered":"2024-09-13T11:23:39.063172+0000","last_unstale":"2024-09-13T11:25:36.022837+0000","last_undegraded":"2024-09-13T11:25:36.022837+0000","last_fullsized":"2024-09-13T11:25:36.022837+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":5,"last_scrub":"165'6821","last_scrub_stamp":"2024-09-13T11:25:24.287082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:24.287082+0000","objects_scrubbed":3406,"log_size":7300,"log_dups_size":0,"ondisk_log_size":7300,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:26:49.405332+0000","scrub_duration":947,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":190971904,"num_objects":2914,"num_object_clones":0,"num_object_copies":5828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2914,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3959,"num_write_kb":155584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11],"acting":[4,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.12","version":"174'7285","reported_seq":17407,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-13T11:25:36.141943+0000","last_change":"2024-09-13T11:25:26.257543+0000","last_active":"2024-09-13T11:25:36.141943+0000","last_peered":"2024-09-13T11:25:36.141943+0000","last_clean":"2024-09-13T11:25:36.141943+0000","last_became_active":"2024-09-13T11:24:29.268229+0000","last_became_peered":"2024-09-13T11:24:29.268229+0000","last_unstale":"2024-09-13T11:25:36.141943+0000","last_undegraded":"2024-09-13T11:25:36.141943+0000","last_fullsized":"2024-09-13T11:25:36.141943+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'3109","last_scrub_stamp":"2024-09-13T11:23:35.111320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:35.111320+0000","objects_scrubbed":599,"log_size":7285,"log_dups_size":0,"ondisk_log_size":7285,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:37.168999+0000","scrub_duration":239,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":191496192,"num_objects":2922,"num_object_clones":0,"num_object_copies":5844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2922,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4176,"num_write_kb":171712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3457,"num_bytes_recovered":226557952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.13","version":"174'7316","reported_seq":10560,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-13T11:25:37.068509+0000","last_change":"2024-09-13T11:25:01.153044+0000","last_active":"2024-09-13T11:25:37.068509+0000","last_peered":"2024-09-13T11:25:37.068509+0000","last_clean":"2024-09-13T11:25:37.068509+0000","last_became_active":"2024-09-13T11:23:41.082433+0000","last_became_peered":"2024-09-13T11:23:41.082433+0000","last_unstale":"2024-09-13T11:25:37.068509+0000","last_undegraded":"2024-09-13T11:25:37.068509+0000","last_fullsized":"2024-09-13T11:25:37.068509+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":5,"last_scrub":"147'5921","last_scrub_stamp":"2024-09-13T11:25:01.152971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:01.152971+0000","objects_scrubbed":4311,"log_size":7316,"log_dups_size":0,"ondisk_log_size":7316,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:26:07.027631+0000","scrub_duration":1011,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":188416000,"num_objects":2875,"num_object_clones":0,"num_object_copies":5750,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2875,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3920,"num_write_kb":154048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7],"acting":[11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":46,"stat_sum":{"num_bytes":6053724184,"num_objects":92374,"num_object_clones":0,"num_object_copies":184748,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":220,"num_objects_unfound":0,"num_objects_dirty":92374,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":104322,"num_write_kb":3634433,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22104,"num_bytes_recovered":1448607744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":13651156992,"data_stored":13651148848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":43282,"internal_metadata":0},"log_size":297706,"ondisk_log_size":297706,"up":92,"acting":92,"num_store_stats":16},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":11},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":112,"ondisk_log_size":112,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":13,"up_from":23,"seq":98784247853,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":670164,"kb_used_data":558468,"kb_used_omap":1,"kb_used_meta":111678,"kb_avail":93701676,"statfs":{"total":96636764160,"available":95950516224,"internally_reserved":0,"allocated":571871232,"data_stored":571090090,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":114358725},"hb_peers":[0,1,2,4,6,8,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":23,"seq":98784247852,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":615844,"kb_used_data":548564,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":93755996,"statfs":{"total":96636764160,"available":96006139904,"internally_reserved":0,"allocated":561729536,"data_stored":560992695,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":68876736},"hb_peers":[0,1,3,8,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":40,"seq":171798691875,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1040956,"kb_used_data":928428,"kb_used_omap":4,"kb_used_meta":112507,"kb_avail":93330884,"statfs":{"total":96636764160,"available":95570825216,"internally_reserved":0,"allocated":950710272,"data_stored":949945981,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4433,"internal_metadata":115207855},"hb_peers":[0,2,3,4,5,6,7,8,9,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":21,"seq":90194313260,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1095640,"kb_used_data":988104,"kb_used_omap":1,"kb_used_meta":107518,"kb_avail":93276200,"statfs":{"total":96636764160,"available":95514828800,"internally_reserved":0,"allocated":1011818496,"data_stored":1011032022,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":110098887},"hb_peers":[0,2,4,5,7,9,10,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":21,"seq":90194313263,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1246916,"kb_used_data":1129652,"kb_used_omap":1,"kb_used_meta":117246,"kb_avail":93124924,"statfs":{"total":96636764160,"available":95359922176,"internally_reserved":0,"allocated":1156763648,"data_stored":1155993293,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":120060359},"hb_peers":[0,2,4,6,7,8,9,11,12,13],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":22,"seq":94489280555,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1203548,"kb_used_data":1104332,"kb_used_omap":1,"kb_used_meta":99198,"kb_avail":93168292,"statfs":{"total":96636764160,"available":95404331008,"internally_reserved":0,"allocated":1130835968,"data_stored":1130110386,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":101579205},"hb_peers":[1,3,4,5,7,8,10,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,4],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":22,"seq":94489280556,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":523844,"kb_used_data":460724,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":93847996,"statfs":{"total":96636764160,"available":96100347904,"internally_reserved":0,"allocated":471781376,"data_stored":471021793,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[0,1,3,4,5,7,8,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":15,"up_from":22,"seq":94489280556,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1075024,"kb_used_data":971648,"kb_used_omap":1,"kb_used_meta":103358,"kb_avail":93296816,"statfs":{"total":96636764160,"available":95535939584,"internally_reserved":0,"allocated":994967552,"data_stored":994253594,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":105839046},"hb_peers":[0,4,5,6,7,8,10,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":22,"seq":94489280557,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722000,"kb_used_data":654720,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":93649840,"statfs":{"total":96636764160,"available":95897436160,"internally_reserved":0,"allocated":670433280,"data_stored":669662909,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[3,5,6,7,9,10,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":22,"seq":94489280556,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":542568,"kb_used_data":485016,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":93829272,"statfs":{"total":96636764160,"available":96081174528,"internally_reserved":0,"allocated":496656384,"data_stored":495920961,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58915270},"hb_peers":[0,1,3,5,7,8,10,11,12,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":22,"seq":94489280556,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1334124,"kb_used_data":1222428,"kb_used_omap":1,"kb_used_meta":111678,"kb_avail":93037716,"statfs":{"total":96636764160,"available":95270621184,"internally_reserved":0,"allocated":1251766272,"data_stored":1251023657,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":114358730},"hb_peers":[0,3,5,6,8,9,10,11,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":22,"seq":94489280556,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":538080,"kb_used_data":462480,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":93833760,"statfs":{"total":96636764160,"available":96085770240,"internally_reserved":0,"allocated":473579520,"data_stored":472789416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[1,5,6,7,9,10,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":22,"seq":94489280555,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":931500,"kb_used_data":833692,"kb_used_omap":1,"kb_used_meta":97790,"kb_avail":93440340,"statfs":{"total":96636764160,"available":95682908160,"internally_reserved":0,"allocated":853700608,"data_stored":852954682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":100137408},"hb_peers":[0,1,3,5,7,9,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":11,"up_from":22,"seq":94489280557,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1307092,"kb_used_data":1199556,"kb_used_omap":1,"kb_used_meta":107518,"kb_avail":93064748,"statfs":{"total":96636764160,"available":95298301952,"internally_reserved":0,"allocated":1228345344,"data_stored":1227567909,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":110098884},"hb_peers":[0,1,2,4,7,8,10,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":12,"up_from":22,"seq":94489280556,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1134932,"kb_used_data":1031556,"kb_used_omap":1,"kb_used_meta":103358,"kb_avail":93236908,"statfs":{"total":96636764160,"available":95474593792,"internally_reserved":0,"allocated":1056313344,"data_stored":1055536073,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":105839045},"hb_peers":[0,1,2,3,4,5,10,11,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":14,"up_from":0,"seq":609885356041,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1128267776,"data_stored":1128267776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":941752320,"data_stored":941752320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":465768448,"data_stored":465764376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1003945984,"data_stored":1003945984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":661127168,"data_stored":661127168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1141702656,"data_stored":1141702656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":489881600,"data_stored":489881600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1248526336,"data_stored":1248526336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":471072768,"data_stored":471072768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":559087616,"data_stored":559087616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":840695808,"data_stored":840695808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":1224736768,"data_stored":1224736768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":1046806528,"data_stored":1046806528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":562954240,"data_stored":562954240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":872742912,"data_stored":872742912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":43282,"internal_metadata":0},{"poolid":3,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":992088064,"data_stored":992083992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T11:25:40.041 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-13T11:25:40.069 INFO:teuthology.orchestra.run.smithi001.stdout:osd.9: {} 2024-09-13T11:25:40.069 INFO:teuthology.orchestra.run.smithi001.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-13T11:25:40.104 INFO:teuthology.orchestra.run.smithi001.stdout:osd.10: {} 2024-09-13T11:25:40.104 INFO:teuthology.orchestra.run.smithi001.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-13T11:25:40.104 INFO:tasks.ceph.osd.15.smithi177.stderr:2024-09-13T11:25:40.102+0000 7f9d6ce77640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26578) UID: 0 2024-09-13T11:25:40.146 INFO:teuthology.orchestra.run.smithi001.stdout:osd.11: {} 2024-09-13T11:25:40.146 INFO:teuthology.orchestra.run.smithi001.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-13T11:25:40.182 INFO:teuthology.orchestra.run.smithi001.stdout:osd.12: {} 2024-09-13T11:25:40.182 INFO:teuthology.orchestra.run.smithi001.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-13T11:25:40.205 INFO:tasks.ceph.osd.1.smithi092.stderr:2024-09-13T11:25:40.205+0000 7fe7789c5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33718) UID: 0 2024-09-13T11:25:40.206 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-13T11:25:40.206 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:25:40.215 INFO:teuthology.orchestra.run.smithi001.stdout:osd.13: {} 2024-09-13T11:25:40.215 INFO:teuthology.orchestra.run.smithi001.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-13T11:25:40.226 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-09-13T11:25:40.245 INFO:teuthology.orchestra.run.smithi001.stdout:osd.14: {} 2024-09-13T11:25:40.245 INFO:teuthology.orchestra.run.smithi001.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-13T11:25:40.272 INFO:teuthology.orchestra.run.smithi001.stdout:osd.15: {} 2024-09-13T11:25:40.273 INFO:teuthology.orchestra.run.smithi001.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-13T11:25:40.307 INFO:tasks.ceph.osd.2.smithi162.stderr:2024-09-13T11:25:40.304+0000 7fe92b35a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26313) UID: 0 2024-09-13T11:25:40.339 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-13T11:25:40.389 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-13T11:25:40.389 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:25:40.407 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-09-13T11:25:40.408 INFO:tasks.ceph.osd.5.smithi092.stderr:2024-09-13T11:25:40.405+0000 7f6f0b705640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26346) UID: 0 2024-09-13T11:25:40.509 INFO:tasks.ceph.osd.10.smithi162.stderr:2024-09-13T11:25:40.508+0000 7fb59c9f0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26317) UID: 0 2024-09-13T11:25:40.540 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-13T11:25:40.540 INFO:teuthology.orchestra.run.smithi001.stdout:{"fsid":"2fd68ac7-5581-4d90-80a4-30bcf5252a7a","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noscrub flag(s) set","count":7},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":223,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":176,"num_osds":16,"num_up_osds":16,"osd_up_since":1726226695,"num_in_osds":15,"osd_in_since":1726226668,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":53},{"state_name":"active+recovering+remapped","count":1},{"state_name":"active+clean+scrubbing","count":1}],"num_pgs":55,"num_pools":3,"num_objects":92129,"data_bytes":6037996107,"bytes_used":14255321088,"bytes_avail":1435296141312,"bytes_total":1449551462400,"misplaced_objects":220,"misplaced_total":184258,"misplaced_ratio":0.0011939780090959415,"recovering_objects_per_sec":105,"recovering_bytes_per_sec":6923345,"recovering_keys_per_sec":0,"num_objects_recovered":623,"num_bytes_recovered":40828928,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":1130},"fsmap":{"epoch":1,"btime":"2024-09-13T11:21:57:222574+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-13T11:24:06.142821+0000","services":{}},"progress_events":{}} 2024-09-13T11:25:40.565 INFO:tasks.ceph.ceph_manager.ceph:making progress, resetting timeout 2024-09-13T11:25:40.565 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-13T11:25:40.590 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-13T11:25:40.590 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:25:40.610 INFO:tasks.ceph.osd.11.smithi177.stderr:2024-09-13T11:25:40.606+0000 7f24792c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26580) UID: 0 2024-09-13T11:25:40.610 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-09-13T11:25:40.711 INFO:tasks.ceph.osd.9.smithi092.stderr:2024-09-13T11:25:40.709+0000 7ff5ab3d0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26348) UID: 0 2024-09-13T11:25:40.789 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-13T11:25:40.789 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:25:40.806 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_ops_in_flight 2024-09-13T11:25:40.852 INFO:tasks.ceph.osd.4.smithi001.stderr:2024-09-13T11:25:40.851+0000 7ff8e0218640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26310) UID: 0 2024-09-13T11:25:40.913 INFO:tasks.ceph.osd.13.smithi092.stderr:2024-09-13T11:25:40.913+0000 7f3e5afea640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26350) UID: 0 2024-09-13T11:25:40.918 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-13T11:25:40.919 INFO:teuthology.orchestra.run.smithi001.stderr:dumped all 2024-09-13T11:25:40.930 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-13T11:25:40.930 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:25:40.940 INFO:teuthology.orchestra.run.smithi001.stdout:{"pg_ready":true,"pg_map":{"version":267,"stamp":"2024-09-13T11:25:40.165765+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":6037996107,"num_objects":92129,"num_object_clones":0,"num_object_copies":184258,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":220,"num_objects_unfound":0,"num_objects_dirty":92129,"num_whiteouts":0,"num_read":87,"num_read_kb":74,"num_write":104718,"num_write_kb":3637107,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22106,"num_bytes_recovered":1448607763,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":298070,"ondisk_log_size":298070,"up":110,"acting":110,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":118,"num_osds":15,"num_per_pool_osds":15,"num_per_pool_omap_osds":15,"kb":1415577600,"kb_used":13921212,"kb_used_data":12514188,"kb_used_omap":26,"kb_used_meta":1406757,"kb_avail":1401656388,"statfs":{"total":1449551462400,"available":1435296141312,"internally_reserved":0,"allocated":12814528512,"data_stored":12803136456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":26763,"internal_metadata":1440520053},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,4],"upper_bound":16},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-436994048,"num_objects":-6668,"num_object_clones":0,"num_object_copies":-13336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1246,"num_objects_unfound":0,"num_objects_dirty":-6668,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6668,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":623,"num_bytes_recovered":40828928,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.897283"},"pg_stats":[{"pgid":"3.2d","version":"173'6306","reported_seq":11475,"reported_epoch":173,"state":"active+recovering+remapped","last_fresh":"2024-09-13T11:25:35.084123+0000","last_change":"2024-09-13T11:25:19.656321+0000","last_active":"2024-09-13T11:25:35.084123+0000","last_peered":"2024-09-13T11:25:35.084123+0000","last_clean":"2024-09-13T11:24:24.974950+0000","last_became_active":"2024-09-13T11:24:27.227288+0000","last_became_peered":"2024-09-13T11:24:27.227288+0000","last_unstale":"2024-09-13T11:25:35.084123+0000","last_undegraded":"2024-09-13T11:25:35.084123+0000","last_fullsized":"2024-09-13T11:25:35.084123+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":6,"last_scrub":"76'3395","last_scrub_stamp":"2024-09-13T11:23:32.776747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:32.776747+0000","objects_scrubbed":1704,"log_size":6306,"log_dups_size":0,"ondisk_log_size":6306,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":179,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":100171776,"num_objects":1529,"num_object_clones":0,"num_object_copies":3058,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":220,"num_objects_unfound":0,"num_objects_dirty":1529,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2069,"num_write_kb":87840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1609,"num_bytes_recovered":105447424,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[3,0],"avail_no_missing":["3","7"],"object_location_counts":[{"shards":"0,3","objects":1419},{"shards":"3,7","objects":110}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2c","version":"174'6425","reported_seq":11773,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-13T11:25:35.967991+0000","last_change":"2024-09-13T11:24:59.456762+0000","last_active":"2024-09-13T11:25:35.967991+0000","last_peered":"2024-09-13T11:25:35.967991+0000","last_clean":"2024-09-13T11:25:35.967991+0000","last_became_active":"2024-09-13T11:24:59.021895+0000","last_became_peered":"2024-09-13T11:24:59.021895+0000","last_unstale":"2024-09-13T11:25:35.967991+0000","last_undegraded":"2024-09-13T11:25:35.967991+0000","last_fullsized":"2024-09-13T11:25:35.967991+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":6,"last_scrub":"145'5672","last_scrub_stamp":"2024-09-13T11:24:59.456653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:24:59.456653+0000","objects_scrubbed":2192,"log_size":6425,"log_dups_size":0,"ondisk_log_size":6425,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:26:20.877991+0000","scrub_duration":279,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":94306304,"num_objects":1439,"num_object_clones":0,"num_object_copies":2878,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1439,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":753,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,7],"acting":[10,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.2b","version":"174'6281","reported_seq":9986,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-13T11:25:35.337136+0000","last_change":"2024-09-13T11:24:45.271477+0000","last_active":"2024-09-13T11:25:35.337136+0000","last_peered":"2024-09-13T11:25:35.337136+0000","last_clean":"2024-09-13T11:25:35.337136+0000","last_became_active":"2024-09-13T11:24:26.180578+0000","last_became_peered":"2024-09-13T11:24:26.180578+0000","last_unstale":"2024-09-13T11:25:35.337136+0000","last_undegraded":"2024-09-13T11:25:35.337136+0000","last_fullsized":"2024-09-13T11:25:35.337136+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":6,"last_scrub":"131'5382","last_scrub_stamp":"2024-09-13T11:24:45.271376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:24:45.271376+0000","objects_scrubbed":1940,"log_size":6281,"log_dups_size":0,"ondisk_log_size":6281,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:59.210895+0000","scrub_duration":279,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":89980928,"num_objects":1373,"num_object_clones":0,"num_object_copies":2746,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1373,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":899,"num_write_kb":10624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2a","version":"174'6430","reported_seq":10322,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-13T11:25:36.070639+0000","last_change":"2024-09-13T11:24:46.625117+0000","last_active":"2024-09-13T11:25:36.070639+0000","last_peered":"2024-09-13T11:25:36.070639+0000","last_clean":"2024-09-13T11:25:36.070639+0000","last_became_active":"2024-09-13T11:24:45.320540+0000","last_became_peered":"2024-09-13T11:24:45.320540+0000","last_unstale":"2024-09-13T11:25:36.070639+0000","last_undegraded":"2024-09-13T11:25:36.070639+0000","last_fullsized":"2024-09-13T11:25:36.070639+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":6,"last_scrub":"133'5491","last_scrub_stamp":"2024-09-13T11:24:46.624970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:24:46.624970+0000","objects_scrubbed":2094,"log_size":6430,"log_dups_size":0,"ondisk_log_size":6430,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:26:04.813280+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":98500608,"num_objects":1503,"num_object_clones":0,"num_object_copies":3006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":941,"num_write_kb":11200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,7],"acting":[10,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.29","version":"174'6086","reported_seq":11793,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-13T11:25:35.973234+0000","last_change":"2024-09-13T11:25:13.333062+0000","last_active":"2024-09-13T11:25:35.973234+0000","last_peered":"2024-09-13T11:25:35.973234+0000","last_clean":"2024-09-13T11:25:35.973234+0000","last_became_active":"2024-09-13T11:25:12.944028+0000","last_became_peered":"2024-09-13T11:25:12.944028+0000","last_unstale":"2024-09-13T11:25:35.973234+0000","last_undegraded":"2024-09-13T11:25:35.973234+0000","last_fullsized":"2024-09-13T11:25:35.973234+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":6,"last_scrub":"154'5579","last_scrub_stamp":"2024-09-13T11:25:13.332959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:13.332959+0000","objects_scrubbed":1937,"log_size":6086,"log_dups_size":0,"ondisk_log_size":6086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:26:24.089228+0000","scrub_duration":243,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":93585408,"num_objects":1428,"num_object_clones":0,"num_object_copies":2856,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1428,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":509,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.28","version":"174'5935","reported_seq":11365,"reported_epoch":174,"state":"active+clean+scrubbing","last_fresh":"2024-09-13T11:25:36.085306+0000","last_change":"2024-09-13T11:25:34.995273+0000","last_active":"2024-09-13T11:25:36.085306+0000","last_peered":"2024-09-13T11:25:36.085306+0000","last_clean":"2024-09-13T11:25:36.085306+0000","last_became_active":"2024-09-13T11:25:34.278640+0000","last_became_peered":"2024-09-13T11:25:34.278640+0000","last_unstale":"2024-09-13T11:25:36.085306+0000","last_undegraded":"2024-09-13T11:25:36.085306+0000","last_fullsized":"2024-09-13T11:25:36.085306+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":6,"last_scrub":"68'2771","last_scrub_stamp":"2024-09-13T11:23:20.644747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:20.644747+0000","objects_scrubbed":1399,"log_size":5935,"log_dups_size":0,"ondisk_log_size":5935,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"scrubbing for 1s","scrub_duration":411,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":94371840,"num_objects":1440,"num_object_clones":0,"num_object_copies":2880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1440,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2188,"num_write_kb":94400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1686,"num_bytes_recovered":110493696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5],"acting":[8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.27","version":"174'6006","reported_seq":11536,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-13T11:25:35.917658+0000","last_change":"2024-09-13T11:25:31.204736+0000","last_active":"2024-09-13T11:25:35.917658+0000","last_peered":"2024-09-13T11:25:35.917658+0000","last_clean":"2024-09-13T11:25:35.917658+0000","last_became_active":"2024-09-13T11:25:30.257911+0000","last_became_peered":"2024-09-13T11:25:30.257911+0000","last_unstale":"2024-09-13T11:25:35.917658+0000","last_undegraded":"2024-09-13T11:25:35.917658+0000","last_fullsized":"2024-09-13T11:25:35.917658+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":6,"last_scrub":"68'2736","last_scrub_stamp":"2024-09-13T11:23:20.443329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:20.443329+0000","objects_scrubbed":0,"log_size":6006,"log_dups_size":0,"ondisk_log_size":6006,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"user requested scrub scheduled @ 2024-09-13T11:25:36.159492+0000","scrub_duration":183,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96206848,"num_objects":1468,"num_object_clones":0,"num_object_copies":2936,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1468,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2284,"num_write_kb":98144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1725,"num_bytes_recovered":113049600,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,11],"acting":[5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.26","version":"174'6127","reported_seq":11714,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-13T11:25:35.931196+0000","last_change":"2024-09-13T11:25:01.697082+0000","last_active":"2024-09-13T11:25:35.931196+0000","last_peered":"2024-09-13T11:25:35.931196+0000","last_clean":"2024-09-13T11:25:35.931196+0000","last_became_active":"2024-09-13T11:24:54.976919+0000","last_became_peered":"2024-09-13T11:24:54.976919+0000","last_unstale":"2024-09-13T11:25:35.931196+0000","last_undegraded":"2024-09-13T11:25:35.931196+0000","last_fullsized":"2024-09-13T11:25:35.931196+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":6,"last_scrub":"147'5433","last_scrub_stamp":"2024-09-13T11:25:01.696886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:01.696886+0000","objects_scrubbed":2158,"log_size":6127,"log_dups_size":0,"ondisk_log_size":6127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:26:22.070621+0000","scrub_duration":403,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":95420416,"num_objects":1456,"num_object_clones":0,"num_object_copies":2912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1456,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":702,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,12],"acting":[8,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.25","version":"174'5965","reported_seq":11711,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-13T11:25:36.091753+0000","last_change":"2024-09-13T11:24:57.073162+0000","last_active":"2024-09-13T11:25:36.091753+0000","last_peered":"2024-09-13T11:25:36.091753+0000","last_clean":"2024-09-13T11:25:36.091753+0000","last_became_active":"2024-09-13T11:24:55.999019+0000","last_became_peered":"2024-09-13T11:24:55.999019+0000","last_unstale":"2024-09-13T11:25:36.091753+0000","last_undegraded":"2024-09-13T11:25:36.091753+0000","last_fullsized":"2024-09-13T11:25:36.091753+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":6,"last_scrub":"142'5214","last_scrub_stamp":"2024-09-13T11:24:57.073061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:24:57.073061+0000","objects_scrubbed":2249,"log_size":5965,"log_dups_size":0,"ondisk_log_size":5965,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:26:12.348519+0000","scrub_duration":571,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":98369536,"num_objects":1501,"num_object_clones":0,"num_object_copies":3002,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1501,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":754,"num_write_kb":192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,2],"acting":[12,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[]},{"pgid":"3.24","version":"174'6026","reported_seq":9958,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-13T11:25:37.032528+0000","last_change":"2024-09-13T11:24:48.048846+0000","last_active":"2024-09-13T11:25:37.032528+0000","last_peered":"2024-09-13T11:25:37.032528+0000","last_clean":"2024-09-13T11:25:37.032528+0000","last_became_active":"2024-09-13T11:24:46.936994+0000","last_became_peered":"2024-09-13T11:24:46.936994+0000","last_unstale":"2024-09-13T11:25:37.032528+0000","last_undegraded":"2024-09-13T11:25:37.032528+0000","last_fullsized":"2024-09-13T11:25:37.032528+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":6,"last_scrub":"135'5093","last_scrub_stamp":"2024-09-13T11:24:48.048764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:24:48.048764+0000","objects_scrubbed":2102,"log_size":6026,"log_dups_size":0,"ondisk_log_size":6026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:53.485370+0000","scrub_duration":287,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":95551488,"num_objects":1458,"num_object_clones":0,"num_object_copies":2916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1458,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":934,"num_write_kb":9280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1],"acting":[9,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.23","version":"174'5247","reported_seq":8114,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-13T11:25:35.945765+0000","last_change":"2024-09-13T11:23:48.133184+0000","last_active":"2024-09-13T11:25:35.945765+0000","last_peered":"2024-09-13T11:25:35.945765+0000","last_clean":"2024-09-13T11:25:35.945765+0000","last_became_active":"2024-09-13T11:23:47.926677+0000","last_became_peered":"2024-09-13T11:23:47.926677+0000","last_unstale":"2024-09-13T11:25:35.945765+0000","last_undegraded":"2024-09-13T11:25:35.945765+0000","last_fullsized":"2024-09-13T11:25:35.945765+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":6,"last_scrub":"91'3278","last_scrub_stamp":"2024-09-13T11:23:48.133084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:48.133084+0000","objects_scrubbed":1029,"log_size":5247,"log_dups_size":0,"ondisk_log_size":5247,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":139,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":98238464,"num_objects":1499,"num_object_clones":0,"num_object_copies":2998,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1499,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1970,"num_write_kb":78080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9],"acting":[5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.22","version":"174'5159","reported_seq":10755,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-13T11:25:35.960754+0000","last_change":"2024-09-13T11:24:45.485711+0000","last_active":"2024-09-13T11:25:35.960754+0000","last_peered":"2024-09-13T11:25:35.960754+0000","last_clean":"2024-09-13T11:25:35.960754+0000","last_became_active":"2024-09-13T11:24:29.253848+0000","last_became_peered":"2024-09-13T11:24:29.253848+0000","last_unstale":"2024-09-13T11:25:35.960754+0000","last_undegraded":"2024-09-13T11:25:35.960754+0000","last_fullsized":"2024-09-13T11:25:35.960754+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":6,"last_scrub":"90'3246","last_scrub_stamp":"2024-09-13T11:23:47.300339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:47.300339+0000","objects_scrubbed":1033,"log_size":5159,"log_dups_size":0,"ondisk_log_size":5159,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":203,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":97255424,"num_objects":1484,"num_object_clones":0,"num_object_copies":2968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1484,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1915,"num_write_kb":75712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1768,"num_bytes_recovered":115867648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.21","version":"174'5079","reported_seq":7961,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-13T11:25:36.054055+0000","last_change":"2024-09-13T11:24:12.564117+0000","last_active":"2024-09-13T11:25:36.054055+0000","last_peered":"2024-09-13T11:25:36.054055+0000","last_clean":"2024-09-13T11:25:36.054055+0000","last_became_active":"2024-09-13T11:24:01.250308+0000","last_became_peered":"2024-09-13T11:24:01.250308+0000","last_unstale":"2024-09-13T11:25:36.054055+0000","last_undegraded":"2024-09-13T11:25:36.054055+0000","last_fullsized":"2024-09-13T11:25:36.054055+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":6,"last_scrub":"112'3670","last_scrub_stamp":"2024-09-13T11:24:12.564031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:24:12.564031+0000","objects_scrubbed":1431,"log_size":5079,"log_dups_size":0,"ondisk_log_size":5079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:36.766299+0000","scrub_duration":183,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":91815936,"num_objects":1401,"num_object_clones":0,"num_object_copies":2802,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1401,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1412,"num_write_kb":44224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8],"acting":[0,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.20","version":"174'5244","reported_seq":7436,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-13T11:25:35.940585+0000","last_change":"2024-09-13T11:25:35.773483+0000","last_active":"2024-09-13T11:25:35.940585+0000","last_peered":"2024-09-13T11:25:35.940585+0000","last_clean":"2024-09-13T11:25:35.940585+0000","last_became_active":"2024-09-13T11:23:33.954946+0000","last_became_peered":"2024-09-13T11:23:33.954946+0000","last_unstale":"2024-09-13T11:25:35.940585+0000","last_undegraded":"2024-09-13T11:25:35.940585+0000","last_fullsized":"2024-09-13T11:25:35.940585+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":6,"last_scrub":"174'5240","last_scrub_stamp":"2024-09-13T11:25:35.773409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:35.773409+0000","objects_scrubbed":1424,"log_size":5244,"log_dups_size":0,"ondisk_log_size":5244,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:26:53.605828+0000","scrub_duration":215,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92930048,"num_objects":1418,"num_object_clones":0,"num_object_copies":2836,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1418,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1979,"num_write_kb":74752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,0],"acting":[12,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[]},{"pgid":"3.1f","version":"174'7463","reported_seq":19430,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-13T11:25:36.005567+0000","last_change":"2024-09-13T11:25:20.773752+0000","last_active":"2024-09-13T11:25:36.005567+0000","last_peered":"2024-09-13T11:25:36.005567+0000","last_clean":"2024-09-13T11:25:36.005567+0000","last_became_active":"2024-09-13T11:24:29.290326+0000","last_became_peered":"2024-09-13T11:24:29.290326+0000","last_unstale":"2024-09-13T11:25:36.005567+0000","last_undegraded":"2024-09-13T11:25:36.005567+0000","last_fullsized":"2024-09-13T11:25:36.005567+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":5,"last_scrub":"161'6865","last_scrub_stamp":"2024-09-13T11:25:20.773641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:20.773641+0000","objects_scrubbed":3425,"log_size":7463,"log_dups_size":0,"ondisk_log_size":7463,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:26:21.317280+0000","scrub_duration":535,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":184942592,"num_objects":2822,"num_object_clones":0,"num_object_copies":5644,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2822,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3043,"num_write_kb":102848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3306,"num_bytes_recovered":216662016,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5],"acting":[10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1e","version":"174'7708","reported_seq":18241,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-13T11:25:35.989155+0000","last_change":"2024-09-13T11:24:08.338485+0000","last_active":"2024-09-13T11:25:35.989155+0000","last_peered":"2024-09-13T11:25:35.989155+0000","last_clean":"2024-09-13T11:25:35.989155+0000","last_became_active":"2024-09-13T11:24:01.261448+0000","last_became_peered":"2024-09-13T11:24:01.261448+0000","last_unstale":"2024-09-13T11:25:35.989155+0000","last_undegraded":"2024-09-13T11:25:35.989155+0000","last_fullsized":"2024-09-13T11:25:35.989155+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":5,"last_scrub":"110'4537","last_scrub_stamp":"2024-09-13T11:24:08.338354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:24:08.338354+0000","objects_scrubbed":2760,"log_size":7708,"log_dups_size":0,"ondisk_log_size":7708,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":367,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":190054400,"num_objects":2900,"num_object_clones":0,"num_object_copies":5800,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2900,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3180,"num_write_kb":106240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"174'7502","reported_seq":12964,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-13T11:25:36.042351+0000","last_change":"2024-09-13T11:24:12.492592+0000","last_active":"2024-09-13T11:25:36.042351+0000","last_peered":"2024-09-13T11:25:36.042351+0000","last_clean":"2024-09-13T11:25:36.042351+0000","last_became_active":"2024-09-13T11:24:01.258225+0000","last_became_peered":"2024-09-13T11:24:01.258225+0000","last_unstale":"2024-09-13T11:25:36.042351+0000","last_undegraded":"2024-09-13T11:25:36.042351+0000","last_fullsized":"2024-09-13T11:25:36.042351+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":5,"last_scrub":"112'4579","last_scrub_stamp":"2024-09-13T11:24:12.492507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:24:12.492507+0000","objects_scrubbed":2875,"log_size":7502,"log_dups_size":0,"ondisk_log_size":7502,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":375,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":189792280,"num_objects":2897,"num_object_clones":0,"num_object_copies":5794,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2897,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2936,"num_write_kb":94593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,15],"acting":[2,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1c","version":"174'7684","reported_seq":11503,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-13T11:25:37.001667+0000","last_change":"2024-09-13T11:24:00.529312+0000","last_active":"2024-09-13T11:25:37.001667+0000","last_peered":"2024-09-13T11:25:37.001667+0000","last_clean":"2024-09-13T11:25:37.001667+0000","last_became_active":"2024-09-13T11:23:33.989492+0000","last_became_peered":"2024-09-13T11:23:33.989492+0000","last_unstale":"2024-09-13T11:25:37.001667+0000","last_undegraded":"2024-09-13T11:25:37.001667+0000","last_fullsized":"2024-09-13T11:25:37.001667+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":5,"last_scrub":"102'4277","last_scrub_stamp":"2024-09-13T11:24:00.529243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:24:00.529243+0000","objects_scrubbed":2544,"log_size":7684,"log_dups_size":0,"ondisk_log_size":7684,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:37.048397+0000","scrub_duration":259,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":188416000,"num_objects":2875,"num_object_clones":0,"num_object_copies":5750,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2875,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3409,"num_write_kb":119680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7],"acting":[11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1b","version":"174'7639","reported_seq":13280,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-13T11:25:36.109056+0000","last_change":"2024-09-13T11:24:12.392093+0000","last_active":"2024-09-13T11:25:36.109056+0000","last_peered":"2024-09-13T11:25:36.109056+0000","last_clean":"2024-09-13T11:25:36.109056+0000","last_became_active":"2024-09-13T11:24:04.950821+0000","last_became_peered":"2024-09-13T11:24:04.950821+0000","last_unstale":"2024-09-13T11:25:36.109056+0000","last_undegraded":"2024-09-13T11:25:36.109056+0000","last_fullsized":"2024-09-13T11:25:36.109056+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":5,"last_scrub":"112'4665","last_scrub_stamp":"2024-09-13T11:24:12.391890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:24:12.391890+0000","objects_scrubbed":2934,"log_size":7639,"log_dups_size":0,"ondisk_log_size":7639,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:41.846147+0000","scrub_duration":675,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":189857792,"num_objects":2897,"num_object_clones":0,"num_object_copies":5794,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2897,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2995,"num_write_kb":94656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,15],"acting":[12,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[]},{"pgid":"3.1a","version":"174'7491","reported_seq":12793,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-13T11:25:36.043187+0000","last_change":"2024-09-13T11:24:09.556174+0000","last_active":"2024-09-13T11:25:36.043187+0000","last_peered":"2024-09-13T11:25:36.043187+0000","last_clean":"2024-09-13T11:25:36.043187+0000","last_became_active":"2024-09-13T11:24:03.935185+0000","last_became_peered":"2024-09-13T11:24:03.935185+0000","last_unstale":"2024-09-13T11:25:36.043187+0000","last_undegraded":"2024-09-13T11:25:36.043187+0000","last_fullsized":"2024-09-13T11:25:36.043187+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":5,"last_scrub":"110'4433","last_scrub_stamp":"2024-09-13T11:24:09.556022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:24:09.556022+0000","objects_scrubbed":2654,"log_size":7491,"log_dups_size":0,"ondisk_log_size":7491,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:36.486646+0000","scrub_duration":387,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":186712064,"num_objects":2849,"num_object_clones":0,"num_object_copies":5698,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2849,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3069,"num_write_kb":104448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9],"acting":[4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.19","version":"174'7134","reported_seq":11623,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-13T11:25:36.965624+0000","last_change":"2024-09-13T11:25:10.784789+0000","last_active":"2024-09-13T11:25:36.965624+0000","last_peered":"2024-09-13T11:25:36.965624+0000","last_clean":"2024-09-13T11:25:36.965624+0000","last_became_active":"2024-09-13T11:23:41.924941+0000","last_became_peered":"2024-09-13T11:23:41.924941+0000","last_unstale":"2024-09-13T11:25:36.965624+0000","last_undegraded":"2024-09-13T11:25:36.965624+0000","last_fullsized":"2024-09-13T11:25:36.965624+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":5,"last_scrub":"151'6156","last_scrub_stamp":"2024-09-13T11:25:10.784716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:10.784716+0000","objects_scrubbed":3781,"log_size":7134,"log_dups_size":0,"ondisk_log_size":7134,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:26:30.390310+0000","scrub_duration":1131,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":182517760,"num_objects":2785,"num_object_clones":0,"num_object_copies":5570,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2785,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3732,"num_write_kb":145216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1],"acting":[9,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.18","version":"174'7232","reported_seq":11982,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-13T11:25:36.068112+0000","last_change":"2024-09-13T11:24:13.454659+0000","last_active":"2024-09-13T11:25:36.068112+0000","last_peered":"2024-09-13T11:25:36.068112+0000","last_clean":"2024-09-13T11:25:36.068112+0000","last_became_active":"2024-09-13T11:23:48.926014+0000","last_became_peered":"2024-09-13T11:23:48.926014+0000","last_unstale":"2024-09-13T11:25:36.068112+0000","last_undegraded":"2024-09-13T11:25:36.068112+0000","last_fullsized":"2024-09-13T11:25:36.068112+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":5,"last_scrub":"112'4376","last_scrub_stamp":"2024-09-13T11:24:13.454575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:24:13.454575+0000","objects_scrubbed":3036,"log_size":7232,"log_dups_size":0,"ondisk_log_size":7232,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":375,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":188416000,"num_objects":2875,"num_object_clones":0,"num_object_copies":5750,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2875,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2865,"num_write_kb":86528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0],"acting":[10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.17","version":"174'7116","reported_seq":11338,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-13T11:25:36.055838+0000","last_change":"2024-09-13T11:25:12.658562+0000","last_active":"2024-09-13T11:25:36.055838+0000","last_peered":"2024-09-13T11:25:36.055838+0000","last_clean":"2024-09-13T11:25:36.055838+0000","last_became_active":"2024-09-13T11:23:42.935601+0000","last_became_peered":"2024-09-13T11:23:42.935601+0000","last_unstale":"2024-09-13T11:25:36.055838+0000","last_undegraded":"2024-09-13T11:25:36.055838+0000","last_fullsized":"2024-09-13T11:25:36.055838+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":5,"last_scrub":"153'6197","last_scrub_stamp":"2024-09-13T11:25:12.658464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:12.658464+0000","objects_scrubbed":3743,"log_size":7116,"log_dups_size":0,"ondisk_log_size":7116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:26:25.625277+0000","scrub_duration":467,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":185008128,"num_objects":2823,"num_object_clones":0,"num_object_copies":5646,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2823,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3765,"num_write_kb":146688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3],"acting":[10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"174'7497","reported_seq":12355,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-13T11:25:37.015499+0000","last_change":"2024-09-13T11:25:01.903487+0000","last_active":"2024-09-13T11:25:37.015499+0000","last_peered":"2024-09-13T11:25:37.015499+0000","last_clean":"2024-09-13T11:25:37.015499+0000","last_became_active":"2024-09-13T11:23:46.105237+0000","last_became_peered":"2024-09-13T11:23:46.105237+0000","last_unstale":"2024-09-13T11:25:37.015499+0000","last_undegraded":"2024-09-13T11:25:37.015499+0000","last_fullsized":"2024-09-13T11:25:37.015499+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":5,"last_scrub":"147'6035","last_scrub_stamp":"2024-09-13T11:25:01.903395+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:01.903395+0000","objects_scrubbed":4359,"log_size":7497,"log_dups_size":0,"ondisk_log_size":7497,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:26:16.436377+0000","scrub_duration":743,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":188874752,"num_objects":2882,"num_object_clones":0,"num_object_copies":5764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2882,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4022,"num_write_kb":155200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,15],"acting":[11,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.15","version":"174'7111","reported_seq":18452,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-13T11:25:36.097434+0000","last_change":"2024-09-13T11:25:12.198448+0000","last_active":"2024-09-13T11:25:36.097434+0000","last_peered":"2024-09-13T11:25:36.097434+0000","last_clean":"2024-09-13T11:25:36.097434+0000","last_became_active":"2024-09-13T11:24:29.226615+0000","last_became_peered":"2024-09-13T11:24:29.226615+0000","last_unstale":"2024-09-13T11:25:36.097434+0000","last_undegraded":"2024-09-13T11:25:36.097434+0000","last_fullsized":"2024-09-13T11:25:36.097434+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":5,"last_scrub":"91'3384","last_scrub_stamp":"2024-09-13T11:23:48.332600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:48.332600+0000","objects_scrubbed":2032,"log_size":7111,"log_dups_size":0,"ondisk_log_size":7111,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":235,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":184418304,"num_objects":2814,"num_object_clones":0,"num_object_copies":5628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2814,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3728,"num_write_kb":144320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3385,"num_bytes_recovered":221839360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.14","version":"176'7360","reported_seq":18668,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-09-13T11:25:40.066572+0000","last_change":"2024-09-13T11:25:38.085809+0000","last_active":"2024-09-13T11:25:40.066572+0000","last_peered":"2024-09-13T11:25:40.066572+0000","last_clean":"2024-09-13T11:25:40.066572+0000","last_became_active":"2024-09-13T11:24:29.253106+0000","last_became_peered":"2024-09-13T11:24:29.253106+0000","last_unstale":"2024-09-13T11:25:40.066572+0000","last_undegraded":"2024-09-13T11:25:40.066572+0000","last_fullsized":"2024-09-13T11:25:40.066572+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":5,"last_scrub":"174'7321","last_scrub_stamp":"2024-09-13T11:25:38.085667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:38.085667+0000","objects_scrubbed":2938,"log_size":7360,"log_dups_size":0,"ondisk_log_size":7360,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:26:55.536565+0000","scrub_duration":431,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":189530112,"num_objects":2892,"num_object_clones":0,"num_object_copies":5784,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2892,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3999,"num_write_kb":158592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3437,"num_bytes_recovered":225247232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,15],"acting":[3,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":320,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-13T11:25:35.285144+0000","last_change":"2024-09-13T11:24:23.212494+0000","last_active":"2024-09-13T11:25:35.285144+0000","last_peered":"2024-09-13T11:25:35.285144+0000","last_clean":"2024-09-13T11:25:35.285144+0000","last_became_active":"2024-09-13T11:22:14.192897+0000","last_became_peered":"2024-09-13T11:22:14.192897+0000","last_unstale":"2024-09-13T11:25:35.285144+0000","last_undegraded":"2024-09-13T11:25:35.285144+0000","last_fullsized":"2024-09-13T11:25:35.285144+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:24:23.212406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:24:23.212406+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:36.980197+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00084655899999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,10],"acting":[15,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"3.6","version":"174'6156","reported_seq":10035,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-13T11:25:37.015602+0000","last_change":"2024-09-13T11:24:49.416851+0000","last_active":"2024-09-13T11:25:37.015602+0000","last_peered":"2024-09-13T11:25:37.015602+0000","last_clean":"2024-09-13T11:25:37.015602+0000","last_became_active":"2024-09-13T11:24:29.322558+0000","last_became_peered":"2024-09-13T11:24:29.322558+0000","last_unstale":"2024-09-13T11:25:37.015602+0000","last_undegraded":"2024-09-13T11:25:37.015602+0000","last_fullsized":"2024-09-13T11:25:37.015602+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"136'5216","last_scrub_stamp":"2024-09-13T11:24:49.416706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:24:49.416706+0000","objects_scrubbed":2083,"log_size":6156,"log_dups_size":0,"ondisk_log_size":6156,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:26:09.525086+0000","scrub_duration":247,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":94109696,"num_objects":1436,"num_object_clones":0,"num_object_copies":2872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1436,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":941,"num_write_kb":9408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,6],"acting":[13,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":327,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-13T11:25:35.293272+0000","last_change":"2024-09-13T11:24:59.556253+0000","last_active":"2024-09-13T11:25:35.293272+0000","last_peered":"2024-09-13T11:25:35.293272+0000","last_clean":"2024-09-13T11:25:35.293272+0000","last_became_active":"2024-09-13T11:22:14.211923+0000","last_became_peered":"2024-09-13T11:22:14.211923+0000","last_unstale":"2024-09-13T11:25:35.293272+0000","last_undegraded":"2024-09-13T11:25:35.293272+0000","last_fullsized":"2024-09-13T11:25:35.293272+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:24:59.556180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:24:59.556180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:26:06.229322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011689529999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0],"acting":[9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"174'5957","reported_seq":6330,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-13T11:25:37.061360+0000","last_change":"2024-09-13T11:24:33.336210+0000","last_active":"2024-09-13T11:25:37.061360+0000","last_peered":"2024-09-13T11:25:37.061360+0000","last_clean":"2024-09-13T11:25:37.061360+0000","last_became_active":"2024-09-13T11:24:24.092975+0000","last_became_peered":"2024-09-13T11:24:24.092975+0000","last_unstale":"2024-09-13T11:25:37.061360+0000","last_undegraded":"2024-09-13T11:25:37.061360+0000","last_fullsized":"2024-09-13T11:25:37.061360+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"125'4820","last_scrub_stamp":"2024-09-13T11:24:33.336091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:24:33.336091+0000","objects_scrubbed":1766,"log_size":5957,"log_dups_size":0,"ondisk_log_size":5957,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:54.538352+0000","scrub_duration":263,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":95092736,"num_objects":1451,"num_object_clones":0,"num_object_copies":2902,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1451,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1139,"num_write_kb":26368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,8],"acting":[13,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":328,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-13T11:25:35.283473+0000","last_change":"2024-09-13T11:25:33.957283+0000","last_active":"2024-09-13T11:25:35.283473+0000","last_peered":"2024-09-13T11:25:35.283473+0000","last_clean":"2024-09-13T11:25:35.283473+0000","last_became_active":"2024-09-13T11:22:14.581458+0000","last_became_peered":"2024-09-13T11:22:14.581458+0000","last_unstale":"2024-09-13T11:25:35.283473+0000","last_undegraded":"2024-09-13T11:25:35.283473+0000","last_fullsized":"2024-09-13T11:25:35.283473+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:25:33.957179+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:25:33.957179+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:26:52.126360+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.001026047,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5],"acting":[8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"174'5998","reported_seq":6343,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-13T11:25:35.354435+0000","last_change":"2024-09-13T11:24:24.579036+0000","last_active":"2024-09-13T11:25:35.354435+0000","last_peered":"2024-09-13T11:25:35.354435+0000","last_clean":"2024-09-13T11:25:35.354435+0000","last_became_active":"2024-09-13T11:24:24.083077+0000","last_became_peered":"2024-09-13T11:24:24.083077+0000","last_unstale":"2024-09-13T11:25:35.354435+0000","last_undegraded":"2024-09-13T11:25:35.354435+0000","last_fullsized":"2024-09-13T11:25:35.354435+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'4737","last_scrub_stamp":"2024-09-13T11:24:24.578900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:24:24.578900+0000","objects_scrubbed":1690,"log_size":5998,"log_dups_size":0,"ondisk_log_size":5998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:38.103421+0000","scrub_duration":407,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":94896128,"num_objects":1448,"num_object_clones":0,"num_object_copies":2896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1448,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1264,"num_write_kb":32704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,12],"acting":[1,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":308,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-13T11:25:35.298684+0000","last_change":"2024-09-13T11:24:08.271026+0000","last_active":"2024-09-13T11:25:35.298684+0000","last_peered":"2024-09-13T11:25:35.298684+0000","last_clean":"2024-09-13T11:25:35.298684+0000","last_became_active":"2024-09-13T11:22:47.719636+0000","last_became_peered":"2024-09-13T11:22:47.719636+0000","last_unstale":"2024-09-13T11:25:35.298684+0000","last_undegraded":"2024-09-13T11:25:35.298684+0000","last_fullsized":"2024-09-13T11:25:35.298684+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:24:08.270972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:24:08.270972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:36.805605+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00087476000000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,9],"acting":[1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"174'5958","reported_seq":6322,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-13T11:25:36.019857+0000","last_change":"2024-09-13T11:24:24.616937+0000","last_active":"2024-09-13T11:25:36.019857+0000","last_peered":"2024-09-13T11:25:36.019857+0000","last_clean":"2024-09-13T11:25:36.019857+0000","last_became_active":"2024-09-13T11:24:24.093492+0000","last_became_peered":"2024-09-13T11:24:24.093492+0000","last_unstale":"2024-09-13T11:25:36.019857+0000","last_undegraded":"2024-09-13T11:25:36.019857+0000","last_fullsized":"2024-09-13T11:25:36.019857+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'4744","last_scrub_stamp":"2024-09-13T11:24:24.616633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:24:24.616633+0000","objects_scrubbed":1624,"log_size":5958,"log_dups_size":0,"ondisk_log_size":5958,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:42.911074+0000","scrub_duration":415,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92733440,"num_objects":1415,"num_object_clones":0,"num_object_copies":2830,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1415,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1221,"num_write_kb":32384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,13],"acting":[15,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"3.3","version":"174'5160","reported_seq":5544,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-13T11:25:36.042110+0000","last_change":"2024-09-13T11:25:08.394545+0000","last_active":"2024-09-13T11:25:36.042110+0000","last_peered":"2024-09-13T11:25:36.042110+0000","last_clean":"2024-09-13T11:25:36.042110+0000","last_became_active":"2024-09-13T11:23:31.930448+0000","last_became_peered":"2024-09-13T11:23:31.930448+0000","last_unstale":"2024-09-13T11:25:36.042110+0000","last_undegraded":"2024-09-13T11:25:36.042110+0000","last_fullsized":"2024-09-13T11:25:36.042110+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"151'4587","last_scrub_stamp":"2024-09-13T11:25:08.394441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:08.394441+0000","objects_scrubbed":1942,"log_size":5160,"log_dups_size":0,"ondisk_log_size":5160,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:26:27.641900+0000","scrub_duration":219,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":89522176,"num_objects":1366,"num_object_clones":0,"num_object_copies":2732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1366,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2038,"num_write_kb":80896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,7],"acting":[10,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"28'2","reported_seq":258,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-13T11:25:35.304129+0000","last_change":"2024-09-13T11:25:13.869564+0000","last_active":"2024-09-13T11:25:35.304129+0000","last_peered":"2024-09-13T11:25:35.304129+0000","last_clean":"2024-09-13T11:25:35.304129+0000","last_became_active":"2024-09-13T11:23:12.546348+0000","last_became_peered":"2024-09-13T11:23:12.546348+0000","last_unstale":"2024-09-13T11:25:35.304129+0000","last_undegraded":"2024-09-13T11:25:35.304129+0000","last_fullsized":"2024-09-13T11:25:35.304129+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2","last_scrub_stamp":"2024-09-13T11:25:13.869413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:25:13.869413+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:26:38.761718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0025826730000000002,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8],"acting":[2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"174'5170","reported_seq":5552,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-13T11:25:36.073635+0000","last_change":"2024-09-13T11:25:09.842875+0000","last_active":"2024-09-13T11:25:36.073635+0000","last_peered":"2024-09-13T11:25:36.073635+0000","last_clean":"2024-09-13T11:25:36.073635+0000","last_became_active":"2024-09-13T11:23:31.921388+0000","last_became_peered":"2024-09-13T11:23:31.921388+0000","last_unstale":"2024-09-13T11:25:36.073635+0000","last_undegraded":"2024-09-13T11:25:36.073635+0000","last_fullsized":"2024-09-13T11:25:36.073635+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"151'4644","last_scrub_stamp":"2024-09-13T11:25:09.842728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:09.842728+0000","objects_scrubbed":1944,"log_size":5170,"log_dups_size":0,"ondisk_log_size":5170,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:26:25.553642+0000","scrub_duration":395,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92930048,"num_objects":1418,"num_object_clones":0,"num_object_copies":2836,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1418,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2064,"num_write_kb":84032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,15],"acting":[12,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":169,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-13T11:25:35.298317+0000","last_change":"2024-09-13T11:25:34.822952+0000","last_active":"2024-09-13T11:25:35.298317+0000","last_peered":"2024-09-13T11:25:35.298317+0000","last_clean":"2024-09-13T11:25:35.298317+0000","last_became_active":"2024-09-13T11:24:29.096575+0000","last_became_peered":"2024-09-13T11:24:29.096575+0000","last_unstale":"2024-09-13T11:25:35.298317+0000","last_undegraded":"2024-09-13T11:25:35.298317+0000","last_fullsized":"2024-09-13T11:25:35.298317+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:25:34.822831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:25:34.822831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:27:01.304133+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00081225399999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"174'5139","reported_seq":5524,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-13T11:25:37.061952+0000","last_change":"2024-09-13T11:25:00.861990+0000","last_active":"2024-09-13T11:25:37.061952+0000","last_peered":"2024-09-13T11:25:37.061952+0000","last_clean":"2024-09-13T11:25:37.061952+0000","last_became_active":"2024-09-13T11:23:31.926528+0000","last_became_peered":"2024-09-13T11:23:31.926528+0000","last_unstale":"2024-09-13T11:25:37.061952+0000","last_undegraded":"2024-09-13T11:25:37.061952+0000","last_fullsized":"2024-09-13T11:25:37.061952+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"146'4447","last_scrub_stamp":"2024-09-13T11:25:00.861834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:00.861834+0000","objects_scrubbed":2126,"log_size":5139,"log_dups_size":0,"ondisk_log_size":5139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:26:15.411903+0000","scrub_duration":787,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":93782016,"num_objects":1431,"num_object_clones":0,"num_object_copies":2862,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1431,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2069,"num_write_kb":85056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,11],"acting":[13,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":324,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-09-13T11:25:39.356955+0000","last_change":"2024-09-13T11:24:11.895017+0000","last_active":"2024-09-13T11:25:39.356955+0000","last_peered":"2024-09-13T11:25:39.356955+0000","last_clean":"2024-09-13T11:25:39.356955+0000","last_became_active":"2024-09-13T11:22:14.211435+0000","last_became_peered":"2024-09-13T11:22:14.211435+0000","last_unstale":"2024-09-13T11:25:39.356955+0000","last_undegraded":"2024-09-13T11:25:39.356955+0000","last_fullsized":"2024-09-13T11:25:39.356955+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:24:11.894937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:24:11.894937+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:43.628381+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0010010360000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2","version":"174'5195","reported_seq":9042,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-13T11:25:35.957500+0000","last_change":"2024-09-13T11:25:23.765128+0000","last_active":"2024-09-13T11:25:35.957500+0000","last_peered":"2024-09-13T11:25:35.957500+0000","last_clean":"2024-09-13T11:25:35.957500+0000","last_became_active":"2024-09-13T11:24:29.175407+0000","last_became_peered":"2024-09-13T11:24:29.175407+0000","last_unstale":"2024-09-13T11:25:35.957500+0000","last_undegraded":"2024-09-13T11:25:35.957500+0000","last_fullsized":"2024-09-13T11:25:35.957500+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"164'4994","last_scrub_stamp":"2024-09-13T11:25:23.765047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:23.765047+0000","objects_scrubbed":1689,"log_size":5195,"log_dups_size":0,"ondisk_log_size":5195,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:26:45.252942+0000","scrub_duration":283,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":97320960,"num_objects":1485,"num_object_clones":0,"num_object_copies":2970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1485,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2138,"num_write_kb":89024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1731,"num_bytes_recovered":113442816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,3],"acting":[12,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[]},{"pgid":"2.3","version":"26'1","reported_seq":131,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-13T11:25:35.286688+0000","last_change":"2024-09-13T11:25:23.034357+0000","last_active":"2024-09-13T11:25:35.286688+0000","last_peered":"2024-09-13T11:25:35.286688+0000","last_clean":"2024-09-13T11:25:35.286688+0000","last_became_active":"2024-09-13T11:24:29.110028+0000","last_became_peered":"2024-09-13T11:24:29.110028+0000","last_unstale":"2024-09-13T11:25:35.286688+0000","last_undegraded":"2024-09-13T11:25:35.286688+0000","last_fullsized":"2024-09-13T11:25:35.286688+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"26'1","last_scrub_stamp":"2024-09-13T11:25:23.034247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:25:23.034247+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:26:42.429349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,8],"acting":[15,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"1.0","version":"26'112","reported_seq":462,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-13T11:25:35.310912+0000","last_change":"2024-09-13T11:25:23.642366+0000","last_active":"2024-09-13T11:25:35.310912+0000","last_peered":"2024-09-13T11:25:35.310912+0000","last_clean":"2024-09-13T11:25:35.310912+0000","last_became_active":"2024-09-13T11:22:12.129721+0000","last_became_peered":"2024-09-13T11:22:12.129721+0000","last_unstale":"2024-09-13T11:25:35.310912+0000","last_undegraded":"2024-09-13T11:25:35.310912+0000","last_fullsized":"2024-09-13T11:25:35.310912+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"26'112","last_scrub_stamp":"2024-09-13T11:25:23.642204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:11.098388+0000","last_clean_scrub_stamp":"2024-09-13T11:25:23.642204+0000","objects_scrubbed":2,"log_size":112,"log_dups_size":0,"ondisk_log_size":112,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:26:42.628319+0000","scrub_duration":67,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.8","version":"174'5968","reported_seq":6331,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-13T11:25:35.943739+0000","last_change":"2024-09-13T11:24:24.506212+0000","last_active":"2024-09-13T11:25:35.943739+0000","last_peered":"2024-09-13T11:25:35.943739+0000","last_clean":"2024-09-13T11:25:35.943739+0000","last_became_active":"2024-09-13T11:24:24.096220+0000","last_became_peered":"2024-09-13T11:24:24.096220+0000","last_unstale":"2024-09-13T11:25:35.943739+0000","last_undegraded":"2024-09-13T11:25:35.943739+0000","last_fullsized":"2024-09-13T11:25:35.943739+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'4749","last_scrub_stamp":"2024-09-13T11:24:24.506109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:24:24.506109+0000","objects_scrubbed":1690,"log_size":5968,"log_dups_size":0,"ondisk_log_size":5968,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:49.430042+0000","scrub_duration":215,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":94765056,"num_objects":1446,"num_object_clones":0,"num_object_copies":2892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1446,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1220,"num_write_kb":31232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.9","version":"174'6001","reported_seq":6365,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-13T11:25:35.990096+0000","last_change":"2024-09-13T11:24:24.928964+0000","last_active":"2024-09-13T11:25:35.990096+0000","last_peered":"2024-09-13T11:25:35.990096+0000","last_clean":"2024-09-13T11:25:35.990096+0000","last_became_active":"2024-09-13T11:24:24.082504+0000","last_became_peered":"2024-09-13T11:24:24.082504+0000","last_unstale":"2024-09-13T11:25:35.990096+0000","last_undegraded":"2024-09-13T11:25:35.990096+0000","last_fullsized":"2024-09-13T11:25:35.990096+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'4799","last_scrub_stamp":"2024-09-13T11:24:24.928724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:24:24.928724+0000","objects_scrubbed":1705,"log_size":6001,"log_dups_size":0,"ondisk_log_size":6001,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:43.845801+0000","scrub_duration":255,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96468992,"num_objects":1472,"num_object_clones":0,"num_object_copies":2944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1472,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1205,"num_write_kb":31104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,2],"acting":[12,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[]},{"pgid":"3.a","version":"176'6429","reported_seq":6802,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-09-13T11:25:40.045542+0000","last_change":"2024-09-13T11:24:25.137871+0000","last_active":"2024-09-13T11:25:40.045542+0000","last_peered":"2024-09-13T11:25:40.045542+0000","last_clean":"2024-09-13T11:25:40.045542+0000","last_became_active":"2024-09-13T11:24:24.089253+0000","last_became_peered":"2024-09-13T11:24:24.089253+0000","last_unstale":"2024-09-13T11:25:40.045542+0000","last_undegraded":"2024-09-13T11:25:40.045542+0000","last_fullsized":"2024-09-13T11:25:40.045542+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'5157","last_scrub_stamp":"2024-09-13T11:24:25.137777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:24:25.137777+0000","objects_scrubbed":1777,"log_size":6429,"log_dups_size":0,"ondisk_log_size":6429,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":223,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90177536,"num_objects":1376,"num_object_clones":0,"num_object_copies":2752,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1376,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1275,"num_write_kb":27968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.b","version":"174'6346","reported_seq":6713,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-13T11:25:37.072480+0000","last_change":"2024-09-13T11:24:25.503864+0000","last_active":"2024-09-13T11:25:37.072480+0000","last_peered":"2024-09-13T11:25:37.072480+0000","last_clean":"2024-09-13T11:25:37.072480+0000","last_became_active":"2024-09-13T11:24:24.085684+0000","last_became_peered":"2024-09-13T11:24:24.085684+0000","last_unstale":"2024-09-13T11:25:37.072480+0000","last_undegraded":"2024-09-13T11:25:37.072480+0000","last_fullsized":"2024-09-13T11:25:37.072480+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'5112","last_scrub_stamp":"2024-09-13T11:24:25.503716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:24:25.503716+0000","objects_scrubbed":1718,"log_size":6346,"log_dups_size":0,"ondisk_log_size":6346,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:37.636639+0000","scrub_duration":395,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":94502912,"num_objects":1442,"num_object_clones":0,"num_object_copies":2884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1442,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1234,"num_write_kb":30656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,1],"acting":[13,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"3.c","version":"174'6358","reported_seq":6748,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-13T11:25:37.081785+0000","last_change":"2024-09-13T11:25:36.279464+0000","last_active":"2024-09-13T11:25:37.081785+0000","last_peered":"2024-09-13T11:25:37.081785+0000","last_clean":"2024-09-13T11:25:37.081785+0000","last_became_active":"2024-09-13T11:24:24.090847+0000","last_became_peered":"2024-09-13T11:24:24.090847+0000","last_unstale":"2024-09-13T11:25:37.081785+0000","last_undegraded":"2024-09-13T11:25:37.081785+0000","last_fullsized":"2024-09-13T11:25:37.081785+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"174'6334","last_scrub_stamp":"2024-09-13T11:25:36.279390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:36.279390+0000","objects_scrubbed":1456,"log_size":6358,"log_dups_size":0,"ondisk_log_size":6358,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:27:04.013051+0000","scrub_duration":235,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":93782016,"num_objects":1431,"num_object_clones":0,"num_object_copies":2862,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1431,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1209,"num_write_kb":29440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,13],"acting":[11,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.d","version":"174'6325","reported_seq":6692,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-13T11:25:36.063101+0000","last_change":"2024-09-13T11:24:24.967444+0000","last_active":"2024-09-13T11:25:36.063101+0000","last_peered":"2024-09-13T11:25:36.063101+0000","last_clean":"2024-09-13T11:25:36.063101+0000","last_became_active":"2024-09-13T11:24:24.087224+0000","last_became_peered":"2024-09-13T11:24:24.087224+0000","last_unstale":"2024-09-13T11:25:36.063101+0000","last_undegraded":"2024-09-13T11:25:36.063101+0000","last_fullsized":"2024-09-13T11:25:36.063101+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'5110","last_scrub_stamp":"2024-09-13T11:24:24.967337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:24:24.967337+0000","objects_scrubbed":1725,"log_size":6325,"log_dups_size":0,"ondisk_log_size":6325,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:53.620949+0000","scrub_duration":255,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96534528,"num_objects":1473,"num_object_clones":0,"num_object_copies":2946,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1473,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1218,"num_write_kb":30912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.e","version":"174'7669","reported_seq":12933,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-13T11:25:36.052006+0000","last_change":"2024-09-13T11:23:32.368856+0000","last_active":"2024-09-13T11:25:36.052006+0000","last_peered":"2024-09-13T11:25:36.052006+0000","last_clean":"2024-09-13T11:25:36.052006+0000","last_became_active":"2024-09-13T11:23:31.925869+0000","last_became_peered":"2024-09-13T11:23:31.925869+0000","last_unstale":"2024-09-13T11:25:36.052006+0000","last_undegraded":"2024-09-13T11:25:36.052006+0000","last_fullsized":"2024-09-13T11:25:36.052006+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'3527","last_scrub_stamp":"2024-09-13T11:23:32.368727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:32.368727+0000","objects_scrubbed":1774,"log_size":7669,"log_dups_size":0,"ondisk_log_size":7669,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":219,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":184483840,"num_objects":2815,"num_object_clones":0,"num_object_copies":5630,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2815,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4143,"num_write_kb":165888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"174'7673","reported_seq":8060,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-13T11:25:37.077063+0000","last_change":"2024-09-13T11:25:00.289940+0000","last_active":"2024-09-13T11:25:37.077063+0000","last_peered":"2024-09-13T11:25:37.077063+0000","last_clean":"2024-09-13T11:25:37.077063+0000","last_became_active":"2024-09-13T11:23:31.935239+0000","last_became_peered":"2024-09-13T11:23:31.935239+0000","last_unstale":"2024-09-13T11:25:37.077063+0000","last_undegraded":"2024-09-13T11:25:37.077063+0000","last_fullsized":"2024-09-13T11:25:37.077063+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"146'6154","last_scrub_stamp":"2024-09-13T11:25:00.289673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:00.289673+0000","objects_scrubbed":4394,"log_size":7673,"log_dups_size":0,"ondisk_log_size":7673,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:26:24.274856+0000","scrub_duration":1187,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":186974208,"num_objects":2853,"num_object_clones":0,"num_object_copies":5706,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2853,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4253,"num_write_kb":170560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,12],"acting":[11,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.10","version":"174'7299","reported_seq":11837,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-13T11:25:36.026699+0000","last_change":"2024-09-13T11:25:10.161564+0000","last_active":"2024-09-13T11:25:36.026699+0000","last_peered":"2024-09-13T11:25:36.026699+0000","last_clean":"2024-09-13T11:25:36.026699+0000","last_became_active":"2024-09-13T11:23:41.931183+0000","last_became_peered":"2024-09-13T11:23:41.931183+0000","last_unstale":"2024-09-13T11:25:36.026699+0000","last_undegraded":"2024-09-13T11:25:36.026699+0000","last_fullsized":"2024-09-13T11:25:36.026699+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":5,"last_scrub":"151'6298","last_scrub_stamp":"2024-09-13T11:25:10.161464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:10.161464+0000","objects_scrubbed":3964,"log_size":7299,"log_dups_size":0,"ondisk_log_size":7299,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:26:33.015478+0000","scrub_duration":607,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":193200128,"num_objects":2948,"num_object_clones":0,"num_object_copies":5896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2948,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3833,"num_write_kb":152000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.11","version":"174'7300","reported_seq":10611,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-13T11:25:36.022837+0000","last_change":"2024-09-13T11:25:24.287326+0000","last_active":"2024-09-13T11:25:36.022837+0000","last_peered":"2024-09-13T11:25:36.022837+0000","last_clean":"2024-09-13T11:25:36.022837+0000","last_became_active":"2024-09-13T11:23:39.063172+0000","last_became_peered":"2024-09-13T11:23:39.063172+0000","last_unstale":"2024-09-13T11:25:36.022837+0000","last_undegraded":"2024-09-13T11:25:36.022837+0000","last_fullsized":"2024-09-13T11:25:36.022837+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":5,"last_scrub":"165'6821","last_scrub_stamp":"2024-09-13T11:25:24.287082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:24.287082+0000","objects_scrubbed":3406,"log_size":7300,"log_dups_size":0,"ondisk_log_size":7300,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:26:49.405332+0000","scrub_duration":947,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":190971904,"num_objects":2914,"num_object_clones":0,"num_object_copies":5828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2914,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3959,"num_write_kb":155584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11],"acting":[4,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.12","version":"174'7285","reported_seq":17407,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-13T11:25:36.141943+0000","last_change":"2024-09-13T11:25:26.257543+0000","last_active":"2024-09-13T11:25:36.141943+0000","last_peered":"2024-09-13T11:25:36.141943+0000","last_clean":"2024-09-13T11:25:36.141943+0000","last_became_active":"2024-09-13T11:24:29.268229+0000","last_became_peered":"2024-09-13T11:24:29.268229+0000","last_unstale":"2024-09-13T11:25:36.141943+0000","last_undegraded":"2024-09-13T11:25:36.141943+0000","last_fullsized":"2024-09-13T11:25:36.141943+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'3109","last_scrub_stamp":"2024-09-13T11:23:35.111320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:35.111320+0000","objects_scrubbed":599,"log_size":7285,"log_dups_size":0,"ondisk_log_size":7285,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:37.168999+0000","scrub_duration":239,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":191496192,"num_objects":2922,"num_object_clones":0,"num_object_copies":5844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2922,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4176,"num_write_kb":171712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3457,"num_bytes_recovered":226557952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.13","version":"174'7316","reported_seq":10560,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-13T11:25:37.068509+0000","last_change":"2024-09-13T11:25:01.153044+0000","last_active":"2024-09-13T11:25:37.068509+0000","last_peered":"2024-09-13T11:25:37.068509+0000","last_clean":"2024-09-13T11:25:37.068509+0000","last_became_active":"2024-09-13T11:23:41.082433+0000","last_became_peered":"2024-09-13T11:23:41.082433+0000","last_unstale":"2024-09-13T11:25:37.068509+0000","last_undegraded":"2024-09-13T11:25:37.068509+0000","last_fullsized":"2024-09-13T11:25:37.068509+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":5,"last_scrub":"147'5921","last_scrub_stamp":"2024-09-13T11:25:01.152971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:01.152971+0000","objects_scrubbed":4311,"log_size":7316,"log_dups_size":0,"ondisk_log_size":7316,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:26:07.027631+0000","scrub_duration":1011,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":188416000,"num_objects":2875,"num_object_clones":0,"num_object_copies":5750,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2875,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3920,"num_write_kb":154048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7],"acting":[11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":46,"stat_sum":{"num_bytes":6037405720,"num_objects":92125,"num_object_clones":0,"num_object_copies":184250,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":220,"num_objects_unfound":0,"num_objects_dirty":92125,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":104571,"num_write_kb":3634433,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22104,"num_bytes_recovered":1448607744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":13508485120,"data_stored":13508476976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":43282,"internal_metadata":0},"log_size":297955,"ondisk_log_size":297955,"up":92,"acting":92,"num_store_stats":16},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":11},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":112,"ondisk_log_size":112,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":13,"up_from":23,"seq":98784247853,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":670164,"kb_used_data":558468,"kb_used_omap":1,"kb_used_meta":111678,"kb_avail":93701676,"statfs":{"total":96636764160,"available":95950516224,"internally_reserved":0,"allocated":571871232,"data_stored":571090090,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":114358725},"hb_peers":[0,1,2,4,6,8,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":23,"seq":98784247852,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":615844,"kb_used_data":548564,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":93755996,"statfs":{"total":96636764160,"available":96006139904,"internally_reserved":0,"allocated":561729536,"data_stored":560992695,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":68876736},"hb_peers":[0,1,3,8,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":40,"seq":171798691875,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1040956,"kb_used_data":928428,"kb_used_omap":4,"kb_used_meta":112507,"kb_avail":93330884,"statfs":{"total":96636764160,"available":95570825216,"internally_reserved":0,"allocated":950710272,"data_stored":949945981,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4433,"internal_metadata":115207855},"hb_peers":[0,2,3,4,5,6,7,8,9,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":21,"seq":90194313261,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1034620,"kb_used_data":922924,"kb_used_omap":1,"kb_used_meta":111678,"kb_avail":93337220,"statfs":{"total":96636764160,"available":95577313280,"internally_reserved":0,"allocated":945074176,"data_stored":944273017,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":114358727},"hb_peers":[0,2,4,5,7,9,10,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":21,"seq":90194313263,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1246916,"kb_used_data":1129652,"kb_used_omap":1,"kb_used_meta":117246,"kb_avail":93124924,"statfs":{"total":96636764160,"available":95359922176,"internally_reserved":0,"allocated":1156763648,"data_stored":1155993293,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":120060359},"hb_peers":[0,2,4,6,7,8,9,11,12,13],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":22,"seq":94489280555,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1203548,"kb_used_data":1104332,"kb_used_omap":1,"kb_used_meta":99198,"kb_avail":93168292,"statfs":{"total":96636764160,"available":95404331008,"internally_reserved":0,"allocated":1130835968,"data_stored":1130110386,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":101579205},"hb_peers":[1,3,4,5,7,8,10,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,4],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":22,"seq":94489280556,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":523844,"kb_used_data":460724,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":93847996,"statfs":{"total":96636764160,"available":96100347904,"internally_reserved":0,"allocated":471781376,"data_stored":471021793,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[0,1,3,4,5,7,8,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":15,"up_from":22,"seq":94489280556,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1075024,"kb_used_data":971648,"kb_used_omap":1,"kb_used_meta":103358,"kb_avail":93296816,"statfs":{"total":96636764160,"available":95535939584,"internally_reserved":0,"allocated":994967552,"data_stored":994253594,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":105839046},"hb_peers":[0,4,5,6,7,8,10,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":22,"seq":94489280557,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722000,"kb_used_data":654720,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":93649840,"statfs":{"total":96636764160,"available":95897436160,"internally_reserved":0,"allocated":670433280,"data_stored":669662909,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[3,5,6,7,9,10,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":22,"seq":94489280556,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":542568,"kb_used_data":485016,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":93829272,"statfs":{"total":96636764160,"available":96081174528,"internally_reserved":0,"allocated":496656384,"data_stored":495920961,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58915270},"hb_peers":[0,1,3,5,7,8,10,11,12,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":22,"seq":94489280556,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1334124,"kb_used_data":1222428,"kb_used_omap":1,"kb_used_meta":111678,"kb_avail":93037716,"statfs":{"total":96636764160,"available":95270621184,"internally_reserved":0,"allocated":1251766272,"data_stored":1251023657,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":114358730},"hb_peers":[0,3,5,6,8,9,10,11,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":22,"seq":94489280556,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":538080,"kb_used_data":462480,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":93833760,"statfs":{"total":96636764160,"available":96085770240,"internally_reserved":0,"allocated":473579520,"data_stored":472789416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[1,5,6,7,9,10,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":22,"seq":94489280555,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":931500,"kb_used_data":833692,"kb_used_omap":1,"kb_used_meta":97790,"kb_avail":93440340,"statfs":{"total":96636764160,"available":95682908160,"internally_reserved":0,"allocated":853700608,"data_stored":852954682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":100137408},"hb_peers":[0,1,3,5,7,9,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":11,"up_from":22,"seq":94489280557,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1307092,"kb_used_data":1199556,"kb_used_omap":1,"kb_used_meta":107518,"kb_avail":93064748,"statfs":{"total":96636764160,"available":95298301952,"internally_reserved":0,"allocated":1228345344,"data_stored":1227567909,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":110098884},"hb_peers":[0,1,2,4,7,8,10,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":12,"up_from":22,"seq":94489280556,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1134932,"kb_used_data":1031556,"kb_used_omap":1,"kb_used_meta":103358,"kb_avail":93236908,"statfs":{"total":96636764160,"available":95474593792,"internally_reserved":0,"allocated":1056313344,"data_stored":1055536073,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":105839045},"hb_peers":[0,1,2,3,4,5,10,11,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":14,"up_from":0,"seq":609885356042,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1128267776,"data_stored":1128267776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":941752320,"data_stored":941752320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":465768448,"data_stored":465764376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":939917312,"data_stored":939917312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":661127168,"data_stored":661127168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1141702656,"data_stored":1141702656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":489881600,"data_stored":489881600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1248526336,"data_stored":1248526336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":471072768,"data_stored":471072768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":559087616,"data_stored":559087616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":840695808,"data_stored":840695808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":1224736768,"data_stored":1224736768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":1046806528,"data_stored":1046806528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":562954240,"data_stored":562954240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":794099712,"data_stored":794099712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":43282,"internal_metadata":0},{"poolid":3,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":992088064,"data_stored":992083992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T11:25:40.948 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_blocked_ops 2024-09-13T11:25:41.036 INFO:tasks.ceph.osd.12.smithi001.stderr:2024-09-13T11:25:41.035+0000 7f892c6e6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26314) UID: 0 2024-09-13T11:25:41.115 INFO:tasks.ceph.osd.15.smithi177.stderr:2024-09-13T11:25:41.114+0000 7f9d6ce77640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26578) UID: 0 2024-09-13T11:25:41.124 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-13T11:25:41.125 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:25:41.141 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_historic_ops 2024-09-13T11:25:41.216 INFO:tasks.ceph.osd.2.smithi162.stderr:2024-09-13T11:25:41.212+0000 7fe92b35a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26313) UID: 0 2024-09-13T11:25:41.317 INFO:tasks.ceph.osd.8.smithi001.stderr:2024-09-13T11:25:41.315+0000 7fb9b48ae640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26312) UID: 0 2024-09-13T11:25:41.324 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-13T11:25:41.324 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:25:41.339 INFO:teuthology.orchestra.run.smithi001.stderr:nodeep-scrub is set 2024-09-13T11:25:41.344 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-13T11:25:41.361 INFO:tasks.ceph.osd.1.smithi092.stderr:2024-09-13T11:25:41.357+0000 7fe76f3f6640 -1 osd.1 176 handle_osd_map injecting map crc failure 2024-09-13T11:25:41.418 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-13T11:25:41.415+0000 7f6a68775640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26307) UID: 0 2024-09-13T11:25:41.491 INFO:teuthology.orchestra.run.smithi092.stdout:ERROR: (22) Invalid argument 2024-09-13T11:25:41.491 INFO:teuthology.orchestra.run.smithi092.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:25:41.513 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-13T11:25:41.557 INFO:tasks.ceph.osd.13.smithi092.stderr:2024-09-13T11:25:41.557+0000 7f3e5afea640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26350) UID: 0 2024-09-13T11:25:41.621 INFO:tasks.ceph.osd.14.smithi162.stderr:2024-09-13T11:25:41.616+0000 7f3df5903640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33154) UID: 0 2024-09-13T11:25:41.696 INFO:teuthology.orchestra.run.smithi092.stdout:ERROR: (22) Invalid argument 2024-09-13T11:25:41.696 INFO:teuthology.orchestra.run.smithi092.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:25:41.712 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-13T11:25:41.721 INFO:tasks.ceph.osd.10.smithi162.stderr:2024-09-13T11:25:41.720+0000 7fb59c9f0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26317) UID: 0 2024-09-13T11:25:41.822 INFO:tasks.ceph.osd.7.smithi177.stderr:2024-09-13T11:25:41.818+0000 7f897b51d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26575) UID: 0 2024-09-13T11:25:41.851 INFO:teuthology.orchestra.run.smithi092.stdout:ERROR: (22) Invalid argument 2024-09-13T11:25:41.851 INFO:teuthology.orchestra.run.smithi092.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:25:41.866 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-09-13T11:25:41.922 INFO:tasks.ceph.osd.5.smithi092.stderr:2024-09-13T11:25:41.921+0000 7f6f0b705640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26346) UID: 0 2024-09-13T11:25:41.955 INFO:tasks.ceph.osd.11.smithi177.stderr:2024-09-13T11:25:41.950+0000 7f2469cec640 -1 osd.11 177 handle_osd_map injecting map crc failure 2024-09-13T11:25:42.010 INFO:teuthology.orchestra.run.smithi092.stdout:ERROR: (22) Invalid argument 2024-09-13T11:25:42.010 INFO:teuthology.orchestra.run.smithi092.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:25:42.023 INFO:tasks.ceph.osd.5.smithi092.stderr:2024-09-13T11:25:42.021+0000 7f6f0b705640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26346) UID: 0 2024-09-13T11:25:42.025 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-09-13T11:25:42.125 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-13T11:25:42.123+0000 7f6a68775640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26307) UID: 0 2024-09-13T11:25:42.207 INFO:teuthology.orchestra.run.smithi092.stdout:ERROR: (22) Invalid argument 2024-09-13T11:25:42.207 INFO:teuthology.orchestra.run.smithi092.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:25:42.223 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-09-13T11:25:42.226 INFO:tasks.ceph.osd.15.smithi177.stderr:2024-09-13T11:25:42.222+0000 7f9d6ce77640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26578) UID: 0 2024-09-13T11:25:42.301 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T11:25:42.329 INFO:tasks.ceph.osd.10.smithi162.stderr:2024-09-13T11:25:42.328+0000 7fb59c9f0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26317) UID: 0 2024-09-13T11:25:42.361 INFO:teuthology.orchestra.run.smithi092.stdout:ERROR: (22) Invalid argument 2024-09-13T11:25:42.361 INFO:teuthology.orchestra.run.smithi092.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:25:42.379 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_ops_in_flight 2024-09-13T11:25:42.429 INFO:tasks.ceph.osd.11.smithi177.stderr:2024-09-13T11:25:42.426+0000 7f24792c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26580) UID: 0 2024-09-13T11:25:42.487 INFO:teuthology.orchestra.run.smithi001.stdout:osd.0: {} 2024-09-13T11:25:42.488 INFO:teuthology.orchestra.run.smithi001.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T11:25:42.509 INFO:teuthology.orchestra.run.smithi092.stdout:ERROR: (22) Invalid argument 2024-09-13T11:25:42.509 INFO:teuthology.orchestra.run.smithi092.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:25:42.511 INFO:teuthology.orchestra.run.smithi001.stdout:osd.1: {} 2024-09-13T11:25:42.511 INFO:teuthology.orchestra.run.smithi001.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T11:25:42.525 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_blocked_ops 2024-09-13T11:25:42.530 INFO:tasks.ceph.osd.10.smithi162.stderr:2024-09-13T11:25:42.528+0000 7fb59c9f0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26317) UID: 0 2024-09-13T11:25:42.539 INFO:teuthology.orchestra.run.smithi001.stdout:osd.2: {} 2024-09-13T11:25:42.539 INFO:teuthology.orchestra.run.smithi001.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T11:25:42.563 INFO:teuthology.orchestra.run.smithi001.stdout:osd.3: {} 2024-09-13T11:25:42.563 INFO:teuthology.orchestra.run.smithi001.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T11:25:42.588 INFO:teuthology.orchestra.run.smithi001.stdout:osd.4: {} 2024-09-13T11:25:42.588 INFO:teuthology.orchestra.run.smithi001.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T11:25:42.612 INFO:teuthology.orchestra.run.smithi001.stdout:osd.5: {} 2024-09-13T11:25:42.612 INFO:teuthology.orchestra.run.smithi001.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T11:25:42.631 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-13T11:25:42.627+0000 7f6a68775640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26307) UID: 0 2024-09-13T11:25:42.642 INFO:teuthology.orchestra.run.smithi001.stdout:osd.6: {} 2024-09-13T11:25:42.642 INFO:teuthology.orchestra.run.smithi001.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T11:25:42.652 INFO:teuthology.orchestra.run.smithi092.stdout:ERROR: (22) Invalid argument 2024-09-13T11:25:42.653 INFO:teuthology.orchestra.run.smithi092.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:25:42.667 INFO:teuthology.orchestra.run.smithi001.stdout:osd.7: {} 2024-09-13T11:25:42.667 INFO:teuthology.orchestra.run.smithi001.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T11:25:42.670 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_historic_ops 2024-09-13T11:25:42.692 INFO:teuthology.orchestra.run.smithi001.stdout:osd.8: {} 2024-09-13T11:25:42.692 INFO:teuthology.orchestra.run.smithi001.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-13T11:25:42.718 INFO:teuthology.orchestra.run.smithi001.stdout:osd.9: {} 2024-09-13T11:25:42.718 INFO:teuthology.orchestra.run.smithi001.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-13T11:25:42.733 INFO:tasks.ceph.osd.5.smithi092.stderr:2024-09-13T11:25:42.733+0000 7f6f0b705640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26346) UID: 0 2024-09-13T11:25:42.748 INFO:teuthology.orchestra.run.smithi001.stdout:osd.10: {} 2024-09-13T11:25:42.748 INFO:teuthology.orchestra.run.smithi001.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-13T11:25:42.776 INFO:teuthology.orchestra.run.smithi001.stdout:osd.11: {} 2024-09-13T11:25:42.776 INFO:teuthology.orchestra.run.smithi001.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-13T11:25:42.801 INFO:teuthology.orchestra.run.smithi092.stdout:ERROR: (22) Invalid argument 2024-09-13T11:25:42.801 INFO:teuthology.orchestra.run.smithi092.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:25:42.802 INFO:teuthology.orchestra.run.smithi001.stdout:osd.12: {} 2024-09-13T11:25:42.802 INFO:teuthology.orchestra.run.smithi001.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-13T11:25:42.818 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-13T11:25:42.828 INFO:teuthology.orchestra.run.smithi001.stdout:osd.13: {} 2024-09-13T11:25:42.828 INFO:teuthology.orchestra.run.smithi001.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-13T11:25:42.854 INFO:teuthology.orchestra.run.smithi001.stdout:osd.14: {} 2024-09-13T11:25:42.854 INFO:teuthology.orchestra.run.smithi001.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-13T11:25:42.865 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-09-13T11:25:42.864+0000 7f97befc9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26315) UID: 0 2024-09-13T11:25:42.880 INFO:teuthology.orchestra.run.smithi001.stdout:osd.15: {} 2024-09-13T11:25:42.881 INFO:teuthology.orchestra.run.smithi001.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-13T11:25:42.949 INFO:tasks.ceph.osd.4.smithi001.stderr:2024-09-13T11:25:42.947+0000 7ff8e0218640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26310) UID: 0 2024-09-13T11:25:42.964 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-13T11:25:42.964 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [], 2024-09-13T11:25:42.964 INFO:teuthology.orchestra.run.smithi162.stdout: "num_ops": 0 2024-09-13T11:25:42.964 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-13T11:25:42.981 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-13T11:25:43.038 INFO:tasks.ceph.osd.3.smithi177.stderr:2024-09-13T11:25:43.034+0000 7f10adba4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26574) UID: 0 2024-09-13T11:25:43.139 INFO:tasks.ceph.osd.10.smithi162.stderr:2024-09-13T11:25:43.136+0000 7fb59c9f0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26317) UID: 0 2024-09-13T11:25:43.157 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-13T11:25:43.157 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [], 2024-09-13T11:25:43.157 INFO:teuthology.orchestra.run.smithi162.stdout: "complaint_time": 30, 2024-09-13T11:25:43.157 INFO:teuthology.orchestra.run.smithi162.stdout: "num_blocked_ops": 0 2024-09-13T11:25:43.157 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-13T11:25:43.171 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-13T11:25:43.261 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-09-13T11:25:43.260+0000 7f97befc9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26315) UID: 0 2024-09-13T11:25:43.340 INFO:tasks.ceph.osd.14.smithi162.stderr:2024-09-13T11:25:43.336+0000 7f3df5903640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33154) UID: 0 2024-09-13T11:25:43.349 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-13T11:25:43.349 INFO:teuthology.orchestra.run.smithi162.stdout: "size": 20, 2024-09-13T11:25:43.350 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 600, 2024-09-13T11:25:43.350 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [ 2024-09-13T11:25:43.350 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.350 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.4665.0:171 3.9 e41/30 3:9e1c7eeb:::benchmark_data_smithi177_31116_object170:head v 41'13, mlcod=41'9)", 2024-09-13T11:25:43.350 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:22:47.418602+0000", 2024-09-13T11:25:43.350 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 175.924930346, 2024-09-13T11:25:43.350 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.67235367300000004, 2024-09-13T11:25:43.351 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:25:43.351 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:25:43.351 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:43.351 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:25:43.351 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.351 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:25:43.352 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:47.418602+0000", 2024-09-13T11:25:43.352 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:43.352 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.352 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.352 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:25:43.352 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:47.418602+0000", 2024-09-13T11:25:43.353 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:43.353 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.353 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.353 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:25:43.353 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:47.418605+0000", 2024-09-13T11:25:43.353 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.2200000000000001e-06 2024-09-13T11:25:43.354 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.354 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.354 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:25:43.354 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:47.418668+0000", 2024-09-13T11:25:43.354 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.2921000000000003e-05 2024-09-13T11:25:43.354 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.354 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.355 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:25:43.355 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:48.089380+0000", 2024-09-13T11:25:43.355 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.67071191100000005 2024-09-13T11:25:43.355 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.355 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.355 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:25:43.356 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:48.089423+0000", 2024-09-13T11:25:43.356 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.2916e-05 2024-09-13T11:25:43.356 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.356 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.356 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:25:43.357 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:48.089533+0000", 2024-09-13T11:25:43.357 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000109996 2024-09-13T11:25:43.357 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.357 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.357 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:25:43.357 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:48.089570+0000", 2024-09-13T11:25:43.358 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.7314999999999998e-05 2024-09-13T11:25:43.358 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.358 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.358 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:25:43.358 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:48.090852+0000", 2024-09-13T11:25:43.358 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001281641 2024-09-13T11:25:43.358 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.358 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.359 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:25:43.359 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:48.090956+0000", 2024-09-13T11:25:43.359 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00010375299999999999 2024-09-13T11:25:43.359 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:43.359 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:25:43.359 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:43.359 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.359 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.359 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPushReply(3.19 74/70 [PushReplyOp(3:9edf939c:::benchmark_data_smithi177_31116_object17207:head)])", 2024-09-13T11:25:43.359 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:30.897971+0000", 2024-09-13T11:25:43.359 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 132.44556149499999, 2024-09-13T11:25:43.359 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.40616071100000001, 2024-09-13T11:25:43.360 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:25:43.360 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:25:43.360 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-09-13T11:25:43.360 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:25:43.360 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.360 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:25:43.360 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.897971+0000", 2024-09-13T11:25:43.360 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:43.360 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.360 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.360 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:25:43.360 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.897971+0000", 2024-09-13T11:25:43.361 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:43.361 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.361 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.361 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:25:43.361 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.897972+0000", 2024-09-13T11:25:43.362 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1429999999999999e-06 2024-09-13T11:25:43.362 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.363 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.363 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:25:43.363 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.897977+0000", 2024-09-13T11:25:43.363 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.5199999999999999e-06 2024-09-13T11:25:43.363 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.363 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.363 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:25:43.363 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:31.302500+0000", 2024-09-13T11:25:43.363 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.40452312400000001 2024-09-13T11:25:43.363 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.363 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.363 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:25:43.364 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:31.302513+0000", 2024-09-13T11:25:43.364 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2753e-05 2024-09-13T11:25:43.364 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.364 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.364 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:25:43.364 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:31.302645+0000", 2024-09-13T11:25:43.364 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00013292100000000001 2024-09-13T11:25:43.364 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.364 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.364 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:25:43.364 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:31.304132+0000", 2024-09-13T11:25:43.364 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0014862499999999999 2024-09-13T11:25:43.365 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:43.365 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:25:43.365 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:43.365 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.365 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.365 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPushReply(3.19 75/70 [PushReplyOp(3:9d1489b1:::benchmark_data_smithi177_31116_object18698:head)])", 2024-09-13T11:25:43.365 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:31.697212+0000", 2024-09-13T11:25:43.365 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 131.64632049400001, 2024-09-13T11:25:43.365 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.22850633200000001, 2024-09-13T11:25:43.365 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:25:43.365 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:25:43.365 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-09-13T11:25:43.365 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:25:43.366 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.366 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:25:43.366 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:31.697212+0000", 2024-09-13T11:25:43.366 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:43.366 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.366 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.366 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:25:43.366 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:31.697212+0000", 2024-09-13T11:25:43.366 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:43.366 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.366 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.366 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:25:43.366 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:31.697214+0000", 2024-09-13T11:25:43.367 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.7439999999999999e-06 2024-09-13T11:25:43.367 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.367 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.367 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:25:43.367 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:31.697224+0000", 2024-09-13T11:25:43.367 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0173000000000001e-05 2024-09-13T11:25:43.367 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.367 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.367 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:25:43.368 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:31.923931+0000", 2024-09-13T11:25:43.368 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.226706926 2024-09-13T11:25:43.368 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.368 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.368 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:25:43.368 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:31.923950+0000", 2024-09-13T11:25:43.368 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.8899e-05 2024-09-13T11:25:43.368 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.368 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.368 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:25:43.368 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:31.924449+0000", 2024-09-13T11:25:43.368 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00049907800000000002 2024-09-13T11:25:43.368 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.369 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.369 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:25:43.369 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:31.925718+0000", 2024-09-13T11:25:43.369 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001269512 2024-09-13T11:25:43.369 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:43.369 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:25:43.369 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:43.369 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.369 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.369 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPushReply(3.19 76/70 [PushReplyOp(3:9faf6d5f:::benchmark_data_smithi177_31116_object18954:head)])", 2024-09-13T11:25:43.370 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:32.042874+0000", 2024-09-13T11:25:43.370 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 131.30065862500001, 2024-09-13T11:25:43.370 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.40212404299999999, 2024-09-13T11:25:43.370 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:25:43.370 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:25:43.370 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-09-13T11:25:43.371 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:25:43.371 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.371 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:25:43.371 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.042874+0000", 2024-09-13T11:25:43.371 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:43.372 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.372 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.372 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:25:43.372 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.042874+0000", 2024-09-13T11:25:43.372 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:43.372 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.373 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.373 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:25:43.373 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.042875+0000", 2024-09-13T11:25:43.373 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.432e-06 2024-09-13T11:25:43.373 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.373 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.373 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:25:43.374 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.042886+0000", 2024-09-13T11:25:43.374 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0265000000000001e-05 2024-09-13T11:25:43.374 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.374 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.374 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:25:43.374 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.443455+0000", 2024-09-13T11:25:43.374 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.40056951400000002 2024-09-13T11:25:43.375 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.375 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.375 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:25:43.375 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.443480+0000", 2024-09-13T11:25:43.375 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.4929000000000001e-05 2024-09-13T11:25:43.375 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.376 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.376 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:25:43.376 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.443666+0000", 2024-09-13T11:25:43.376 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000186446 2024-09-13T11:25:43.376 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.376 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.376 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:25:43.376 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.444998+0000", 2024-09-13T11:25:43.376 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0013314570000000001 2024-09-13T11:25:43.377 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:43.377 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:25:43.377 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:43.377 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.377 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.377 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPush(3.1d 85/78 [PushOp(3:bc4b9336:::benchmark_data_smithi177_31116_object11944:head, version: 49'734, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:bc4b9336:::benchmark_data_smithi177_31116_object11944:head@49'734, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:25:43.377 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:42.225966+0000", 2024-09-13T11:25:43.378 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 121.117566412, 2024-09-13T11:25:43.378 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.23879947800000001, 2024-09-13T11:25:43.378 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:25:43.378 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:25:43.378 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:25:43.378 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:25:43.378 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.379 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:25:43.379 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.225966+0000", 2024-09-13T11:25:43.379 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:43.379 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.379 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.379 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:25:43.379 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.225966+0000", 2024-09-13T11:25:43.380 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:43.380 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.380 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.380 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:25:43.380 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.225969+0000", 2024-09-13T11:25:43.380 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.0400000000000001e-06 2024-09-13T11:25:43.381 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.381 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.381 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:25:43.381 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.226036+0000", 2024-09-13T11:25:43.381 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.6978999999999997e-05 2024-09-13T11:25:43.382 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.382 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.382 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:25:43.382 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.463110+0000", 2024-09-13T11:25:43.382 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.23707400000000001 2024-09-13T11:25:43.382 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.383 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.383 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:25:43.383 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.463140+0000", 2024-09-13T11:25:43.383 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.9575999999999999e-05 2024-09-13T11:25:43.383 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.383 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.384 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:25:43.384 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.463326+0000", 2024-09-13T11:25:43.384 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000186298 2024-09-13T11:25:43.384 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.384 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.384 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:25:43.385 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.463351+0000", 2024-09-13T11:25:43.385 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.4862999999999999e-05 2024-09-13T11:25:43.385 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.385 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.385 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:25:43.385 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.464766+0000", 2024-09-13T11:25:43.385 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001414722 2024-09-13T11:25:43.386 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:43.386 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:25:43.386 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:43.386 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.386 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.386 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPush(3.1d 89/78 [PushOp(3:bc1b7cfb:::benchmark_data_smithi177_31116_object22629:head, version: 55'1387, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:bc1b7cfb:::benchmark_data_smithi177_31116_object22629:head@55'1387, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:25:43.386 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:46.660855+0000", 2024-09-13T11:25:43.387 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 116.682677916, 2024-09-13T11:25:43.387 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.26622785599999998, 2024-09-13T11:25:43.387 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:25:43.387 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:25:43.387 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:25:43.387 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:25:43.388 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.388 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:25:43.388 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:46.660855+0000", 2024-09-13T11:25:43.388 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:43.388 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.388 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.389 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:25:43.389 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:46.660855+0000", 2024-09-13T11:25:43.389 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:43.389 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.389 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.390 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:25:43.390 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:46.660857+0000", 2024-09-13T11:25:43.390 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.6560000000000001e-06 2024-09-13T11:25:43.390 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.390 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.390 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:25:43.391 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:46.660931+0000", 2024-09-13T11:25:43.391 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.4041999999999993e-05 2024-09-13T11:25:43.391 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.391 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.391 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:25:43.391 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:46.924654+0000", 2024-09-13T11:25:43.392 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.26372239800000002 2024-09-13T11:25:43.392 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.392 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.392 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:25:43.392 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:46.924729+0000", 2024-09-13T11:25:43.392 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.5061999999999994e-05 2024-09-13T11:25:43.393 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.393 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.393 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:25:43.393 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:46.924986+0000", 2024-09-13T11:25:43.393 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00025762000000000002 2024-09-13T11:25:43.393 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.394 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.394 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:25:43.394 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:46.925013+0000", 2024-09-13T11:25:43.394 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.6179000000000001e-05 2024-09-13T11:25:43.395 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.395 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.395 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:25:43.395 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:46.927082+0000", 2024-09-13T11:25:43.395 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.002069899 2024-09-13T11:25:43.395 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:43.395 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:25:43.395 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:43.395 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.396 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.396 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPush(3.1d 90/78 [PushOp(3:bc0e7228:::benchmark_data_smithi177_31116_object24733:head, version: 55'1512, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:bc0e7228:::benchmark_data_smithi177_31116_object24733:head@55'1512, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:25:43.396 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:47.529126+0000", 2024-09-13T11:25:43.396 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 115.81440694200001, 2024-09-13T11:25:43.396 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.21343989499999999, 2024-09-13T11:25:43.396 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:25:43.396 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:25:43.396 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:25:43.396 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:25:43.396 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.396 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:25:43.396 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:47.529126+0000", 2024-09-13T11:25:43.396 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:43.396 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.397 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.397 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:25:43.397 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:47.529126+0000", 2024-09-13T11:25:43.397 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:43.397 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.397 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.397 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:25:43.397 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:47.529127+0000", 2024-09-13T11:25:43.397 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.252e-06 2024-09-13T11:25:43.397 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.397 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.397 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:25:43.398 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:47.529185+0000", 2024-09-13T11:25:43.398 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.8224999999999999e-05 2024-09-13T11:25:43.398 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.398 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.398 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:25:43.398 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:47.740216+0000", 2024-09-13T11:25:43.398 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.21103060800000001 2024-09-13T11:25:43.398 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.398 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.398 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:25:43.398 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:47.740248+0000", 2024-09-13T11:25:43.398 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.2821999999999998e-05 2024-09-13T11:25:43.398 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.398 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.399 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:25:43.399 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:47.740504+0000", 2024-09-13T11:25:43.399 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00025518399999999999 2024-09-13T11:25:43.399 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.399 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.399 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:25:43.399 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:47.740544+0000", 2024-09-13T11:25:43.399 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.0438999999999997e-05 2024-09-13T11:25:43.399 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.399 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.399 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:25:43.399 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:47.742565+0000", 2024-09-13T11:25:43.399 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0020213649999999998 2024-09-13T11:25:43.399 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:43.399 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:25:43.400 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:43.400 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.400 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.400 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:83199 3.1d 3:be2c6fae:::benchmark_data_smithi177_31116_object83198:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106)", 2024-09-13T11:25:43.400 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:24:04.297306+0000", 2024-09-13T11:25:43.400 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 99.046226989999994, 2024-09-13T11:25:43.400 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.25114587999999999, 2024-09-13T11:25:43.400 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:25:43.400 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:25:43.400 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:43.400 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:25:43.400 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:25:43.400 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:43.401 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 83199 2024-09-13T11:25:43.401 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.401 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:25:43.401 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.401 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:25:43.401 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.297306+0000", 2024-09-13T11:25:43.401 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:43.401 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.401 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.401 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:25:43.401 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.297306+0000", 2024-09-13T11:25:43.401 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:43.401 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.402 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.402 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:25:43.402 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.297308+0000", 2024-09-13T11:25:43.402 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.4150000000000002e-06 2024-09-13T11:25:43.402 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.402 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.402 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:25:43.402 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.297352+0000", 2024-09-13T11:25:43.402 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.4431000000000002e-05 2024-09-13T11:25:43.402 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.402 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.402 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:25:43.402 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.297369+0000", 2024-09-13T11:25:43.402 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.6685e-05 2024-09-13T11:25:43.402 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.403 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.403 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:25:43.403 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.297376+0000", 2024-09-13T11:25:43.403 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.7299999999999999e-06 2024-09-13T11:25:43.403 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.403 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.403 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:25:43.403 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.297427+0000", 2024-09-13T11:25:43.403 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.1461000000000001e-05 2024-09-13T11:25:43.403 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.403 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.403 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:25:43.403 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.297528+0000", 2024-09-13T11:25:43.404 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00010028599999999999 2024-09-13T11:25:43.404 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.404 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.404 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 15", 2024-09-13T11:25:43.404 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.297685+0000", 2024-09-13T11:25:43.404 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00015760500000000001 2024-09-13T11:25:43.404 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.404 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.404 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:25:43.404 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.300779+0000", 2024-09-13T11:25:43.404 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0030936700000000002 2024-09-13T11:25:43.404 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.405 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.405 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:25:43.405 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.548279+0000", 2024-09-13T11:25:43.405 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.24750059999999999 2024-09-13T11:25:43.405 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.405 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.405 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:25:43.405 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.548373+0000", 2024-09-13T11:25:43.405 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.3451999999999999e-05 2024-09-13T11:25:43.405 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.406 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.406 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:25:43.406 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.548451+0000", 2024-09-13T11:25:43.406 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.8545000000000001e-05 2024-09-13T11:25:43.406 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:43.406 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:25:43.407 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:43.407 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.407 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.407 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:84621 3.1d 3:becd72aa:::benchmark_data_smithi177_31116_object84620:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-09-13T11:25:43.407 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:24:05.741008+0000", 2024-09-13T11:25:43.407 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 97.602524990000006, 2024-09-13T11:25:43.408 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.17994101100000001, 2024-09-13T11:25:43.408 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:25:43.408 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:25:43.408 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:43.408 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:25:43.408 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:25:43.409 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:43.409 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 84621 2024-09-13T11:25:43.409 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.409 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:25:43.409 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.409 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:25:43.409 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.741008+0000", 2024-09-13T11:25:43.409 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:43.409 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.409 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.409 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:25:43.409 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.741008+0000", 2024-09-13T11:25:43.410 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:43.410 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.410 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.410 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:25:43.410 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.741010+0000", 2024-09-13T11:25:43.410 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.898e-06 2024-09-13T11:25:43.410 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.410 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.410 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:25:43.410 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.741061+0000", 2024-09-13T11:25:43.410 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.0898e-05 2024-09-13T11:25:43.410 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.410 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.410 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:25:43.411 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.741085+0000", 2024-09-13T11:25:43.411 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.3558e-05 2024-09-13T11:25:43.411 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.411 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.411 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:25:43.411 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.741093+0000", 2024-09-13T11:25:43.411 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.3890000000000005e-06 2024-09-13T11:25:43.411 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.411 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.411 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:25:43.411 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.741178+0000", 2024-09-13T11:25:43.411 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.4792999999999999e-05 2024-09-13T11:25:43.411 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.411 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.412 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:25:43.412 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.741386+0000", 2024-09-13T11:25:43.412 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000207505 2024-09-13T11:25:43.412 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.412 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.412 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 15", 2024-09-13T11:25:43.412 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.741597+0000", 2024-09-13T11:25:43.412 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00021158200000000001 2024-09-13T11:25:43.412 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.412 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.412 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:25:43.412 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.743392+0000", 2024-09-13T11:25:43.412 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00179518 2024-09-13T11:25:43.412 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.413 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.413 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:25:43.413 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.920866+0000", 2024-09-13T11:25:43.413 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.17747373699999999 2024-09-13T11:25:43.413 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.413 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.413 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:25:43.413 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.920913+0000", 2024-09-13T11:25:43.413 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.6451e-05 2024-09-13T11:25:43.413 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.413 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.413 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:25:43.413 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.920949+0000", 2024-09-13T11:25:43.414 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.6019999999999997e-05 2024-09-13T11:25:43.414 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:43.414 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:25:43.414 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:43.414 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.414 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.414 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPush(3.25 124/120 [PushOp(3:a6d4ee3f:::benchmark_data_smithi177_31116_object14712:head, version: 51'872, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:a6d4ee3f:::benchmark_data_smithi177_31116_object14712:head@51'872, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:25:43.414 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:24:31.996253+0000", 2024-09-13T11:25:43.414 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 71.347279217999997, 2024-09-13T11:25:43.414 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.17694863499999999, 2024-09-13T11:25:43.414 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:25:43.414 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:25:43.414 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:25:43.414 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:25:43.415 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.415 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:25:43.415 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:31.996253+0000", 2024-09-13T11:25:43.415 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:43.415 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.415 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.415 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:25:43.415 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:31.996253+0000", 2024-09-13T11:25:43.415 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:43.415 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.415 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.415 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:25:43.415 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:31.996257+0000", 2024-09-13T11:25:43.415 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.2629999999999999e-06 2024-09-13T11:25:43.416 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.416 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.416 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:25:43.416 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:31.996359+0000", 2024-09-13T11:25:43.416 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000102456 2024-09-13T11:25:43.416 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.416 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.416 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:25:43.416 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:32.172313+0000", 2024-09-13T11:25:43.416 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.17595388000000001 2024-09-13T11:25:43.416 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.416 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.416 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:25:43.416 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:32.172326+0000", 2024-09-13T11:25:43.417 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2632999999999999e-05 2024-09-13T11:25:43.417 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.417 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.417 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:25:43.417 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:32.172423+0000", 2024-09-13T11:25:43.417 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.7084999999999996e-05 2024-09-13T11:25:43.417 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.417 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.417 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:25:43.417 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:32.172438+0000", 2024-09-13T11:25:43.417 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5775999999999999e-05 2024-09-13T11:25:43.417 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.417 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.417 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:25:43.418 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:32.173202+0000", 2024-09-13T11:25:43.418 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00076354199999999998 2024-09-13T11:25:43.418 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:43.418 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:25:43.418 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:43.418 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.418 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.418 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:126039 3.29 3:96100181:::benchmark_data_smithi177_31116_object126038:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e130)", 2024-09-13T11:25:43.418 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:24:43.012785+0000", 2024-09-13T11:25:43.418 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 60.330747868000003, 2024-09-13T11:25:43.418 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.26685311900000003, 2024-09-13T11:25:43.418 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:25:43.418 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:25:43.419 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:43.419 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:25:43.419 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:25:43.419 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:43.419 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 126039 2024-09-13T11:25:43.419 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.419 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:25:43.419 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.419 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:25:43.419 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:43.012785+0000", 2024-09-13T11:25:43.419 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:43.419 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.419 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.419 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:25:43.420 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:43.012785+0000", 2024-09-13T11:25:43.420 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:43.420 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.420 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.420 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:25:43.420 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:43.012787+0000", 2024-09-13T11:25:43.420 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.875e-06 2024-09-13T11:25:43.420 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.420 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.420 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:25:43.420 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:43.012826+0000", 2024-09-13T11:25:43.420 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.9483999999999997e-05 2024-09-13T11:25:43.420 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.421 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.421 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:25:43.421 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:43.012839+0000", 2024-09-13T11:25:43.421 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3405e-05 2024-09-13T11:25:43.421 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.421 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.421 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:25:43.421 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:43.012845+0000", 2024-09-13T11:25:43.421 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.9100000000000002e-06 2024-09-13T11:25:43.421 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.421 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.421 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:25:43.421 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:43.012903+0000", 2024-09-13T11:25:43.421 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.8131000000000003e-05 2024-09-13T11:25:43.422 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.422 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.422 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:25:43.422 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:43.013066+0000", 2024-09-13T11:25:43.422 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000162058 2024-09-13T11:25:43.422 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.422 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.422 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 4,5", 2024-09-13T11:25:43.422 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:43.013235+0000", 2024-09-13T11:25:43.422 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00016929600000000001 2024-09-13T11:25:43.422 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.422 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.422 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:25:43.422 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:43.017063+0000", 2024-09-13T11:25:43.422 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0038283890000000002 2024-09-13T11:25:43.423 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.423 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.423 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:25:43.423 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:43.017086+0000", 2024-09-13T11:25:43.423 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.2934000000000001e-05 2024-09-13T11:25:43.423 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.423 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.423 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:25:43.423 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:43.278804+0000", 2024-09-13T11:25:43.423 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.26171751799999998 2024-09-13T11:25:43.423 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.424 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.424 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:25:43.424 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:43.278862+0000", 2024-09-13T11:25:43.424 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.8431999999999999e-05 2024-09-13T11:25:43.424 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.424 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.424 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:25:43.424 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:43.279638+0000", 2024-09-13T11:25:43.424 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00077568699999999995 2024-09-13T11:25:43.424 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:43.424 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:25:43.424 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:43.424 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.424 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.425 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop_reply(client.4665.0:126039 3.29 e130/120 ondisk, result = 0)", 2024-09-13T11:25:43.425 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:24:43.021691+0000", 2024-09-13T11:25:43.425 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 60.321841941999999, 2024-09-13T11:25:43.425 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.25795579699999999, 2024-09-13T11:25:43.425 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:25:43.425 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:25:43.425 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:25:43.425 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:25:43.425 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.425 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:25:43.425 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:43.021691+0000", 2024-09-13T11:25:43.425 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:43.425 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.425 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.426 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:25:43.426 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:43.021691+0000", 2024-09-13T11:25:43.426 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:43.426 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.426 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.426 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:25:43.426 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:43.021692+0000", 2024-09-13T11:25:43.426 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3880000000000001e-06 2024-09-13T11:25:43.426 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.426 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.426 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:25:43.426 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:43.021700+0000", 2024-09-13T11:25:43.427 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.9310000000000008e-06 2024-09-13T11:25:43.427 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.427 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.427 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:25:43.427 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:43.278719+0000", 2024-09-13T11:25:43.427 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.25701909299999998 2024-09-13T11:25:43.427 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.427 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.427 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:25:43.427 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:43.278728+0000", 2024-09-13T11:25:43.427 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.5000000000000005e-06 2024-09-13T11:25:43.427 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.427 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.428 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:25:43.428 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:43.278786+0000", 2024-09-13T11:25:43.428 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.7640999999999999e-05 2024-09-13T11:25:43.428 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.428 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.428 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:25:43.428 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:43.278796+0000", 2024-09-13T11:25:43.428 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.9199999999999999e-06 2024-09-13T11:25:43.428 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.428 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.428 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:25:43.428 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:43.279646+0000", 2024-09-13T11:25:43.428 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00085032399999999996 2024-09-13T11:25:43.428 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:43.429 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:25:43.429 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:43.429 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.429 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.429 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPushReply(3.29 134/120 [PushReplyOp(3:95648f22:::benchmark_data_smithi177_31116_object7446:head)])", 2024-09-13T11:25:43.429 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:24:47.110443+0000", 2024-09-13T11:25:43.429 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 56.233089821999997, 2024-09-13T11:25:43.429 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.171602386, 2024-09-13T11:25:43.429 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:25:43.429 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:25:43.429 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-09-13T11:25:43.429 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:25:43.429 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.430 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:25:43.430 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:47.110443+0000", 2024-09-13T11:25:43.430 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:43.430 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.430 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.430 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:25:43.430 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:47.110443+0000", 2024-09-13T11:25:43.430 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:43.430 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.430 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.430 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:25:43.430 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:47.110444+0000", 2024-09-13T11:25:43.430 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.192e-06 2024-09-13T11:25:43.431 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.431 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.431 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:25:43.431 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:47.110449+0000", 2024-09-13T11:25:43.431 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.0869999999999997e-06 2024-09-13T11:25:43.431 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.431 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.431 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:25:43.431 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:47.280321+0000", 2024-09-13T11:25:43.431 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.169872157 2024-09-13T11:25:43.431 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.431 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.431 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:25:43.431 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:47.280337+0000", 2024-09-13T11:25:43.432 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5656000000000001e-05 2024-09-13T11:25:43.432 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.432 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.432 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:25:43.432 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:47.280414+0000", 2024-09-13T11:25:43.432 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.7083999999999996e-05 2024-09-13T11:25:43.432 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.432 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.432 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:25:43.432 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:47.282045+0000", 2024-09-13T11:25:43.432 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0016312099999999999 2024-09-13T11:25:43.432 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:43.432 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:25:43.432 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:43.433 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.433 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.433 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.4665.0:138294 3.9 e140/117 3:90012dab:::benchmark_data_smithi177_31116_object138293:head v 140'5246, mlcod=140'5243)", 2024-09-13T11:25:43.433 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:24:54.252227+0000", 2024-09-13T11:25:43.433 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 49.091305454, 2024-09-13T11:25:43.433 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.32919836400000002, 2024-09-13T11:25:43.433 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:25:43.433 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:25:43.433 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:43.433 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:25:43.433 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.433 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:25:43.433 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:54.252227+0000", 2024-09-13T11:25:43.434 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:43.434 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.434 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.434 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:25:43.434 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:54.252227+0000", 2024-09-13T11:25:43.434 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:43.434 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.434 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.434 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:25:43.434 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:54.252228+0000", 2024-09-13T11:25:43.434 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.8899999999999998e-07 2024-09-13T11:25:43.435 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.435 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.435 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:25:43.435 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:54.252276+0000", 2024-09-13T11:25:43.435 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.8137999999999999e-05 2024-09-13T11:25:43.435 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.435 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.435 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:25:43.435 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:54.578216+0000", 2024-09-13T11:25:43.436 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.32594014399999999 2024-09-13T11:25:43.436 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.436 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.436 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:25:43.436 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:54.578235+0000", 2024-09-13T11:25:43.436 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.8943000000000001e-05 2024-09-13T11:25:43.436 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.436 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.436 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:25:43.436 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:54.578396+0000", 2024-09-13T11:25:43.436 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000160922 2024-09-13T11:25:43.436 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.436 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.436 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:25:43.437 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:54.578457+0000", 2024-09-13T11:25:43.437 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.0665999999999998e-05 2024-09-13T11:25:43.437 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.437 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.437 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:25:43.437 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:54.581339+0000", 2024-09-13T11:25:43.437 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0028826839999999999 2024-09-13T11:25:43.437 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.437 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.437 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:25:43.437 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:54.581425+0000", 2024-09-13T11:25:43.437 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.6077999999999998e-05 2024-09-13T11:25:43.437 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:43.437 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:25:43.438 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:43.438 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.438 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.438 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPushReply(3.29 143/120 [PushReplyOp(3:973d2047:::benchmark_data_smithi177_31116_object49749:head)])", 2024-09-13T11:25:43.438 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:24:57.387106+0000", 2024-09-13T11:25:43.438 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 45.956426127, 2024-09-13T11:25:43.438 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.64813637000000002, 2024-09-13T11:25:43.438 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:25:43.438 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:25:43.438 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-09-13T11:25:43.438 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:25:43.438 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.438 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:25:43.438 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:57.387106+0000", 2024-09-13T11:25:43.439 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:43.439 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.439 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.439 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:25:43.439 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:57.387106+0000", 2024-09-13T11:25:43.439 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:43.439 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.439 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.439 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:25:43.439 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:57.387116+0000", 2024-09-13T11:25:43.439 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.1530000000000003e-06 2024-09-13T11:25:43.439 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.439 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.439 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:25:43.440 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:57.387127+0000", 2024-09-13T11:25:43.440 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1197e-05 2024-09-13T11:25:43.440 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.440 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.440 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:25:43.440 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:58.034264+0000", 2024-09-13T11:25:43.440 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.64713691799999995 2024-09-13T11:25:43.440 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.440 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.440 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:25:43.440 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:58.034273+0000", 2024-09-13T11:25:43.440 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.0929999999999992e-06 2024-09-13T11:25:43.440 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.440 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.441 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:25:43.441 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:58.034321+0000", 2024-09-13T11:25:43.441 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.8396999999999998e-05 2024-09-13T11:25:43.441 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.441 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.441 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:25:43.441 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:58.035243+0000", 2024-09-13T11:25:43.441 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00092161200000000004 2024-09-13T11:25:43.441 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:43.441 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:25:43.441 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:43.441 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.441 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.442 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPushReply(3.29 144/120 [PushReplyOp(3:94a6fc3a:::benchmark_data_smithi177_31116_object50245:head)])", 2024-09-13T11:25:43.442 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:24:58.320977+0000", 2024-09-13T11:25:43.442 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 45.022555543999999, 2024-09-13T11:25:43.442 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.31110569599999999, 2024-09-13T11:25:43.442 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:25:43.442 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:25:43.442 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-09-13T11:25:43.442 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:25:43.442 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.442 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:25:43.442 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:58.320977+0000", 2024-09-13T11:25:43.442 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:43.442 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.443 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.443 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:25:43.443 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:58.320977+0000", 2024-09-13T11:25:43.443 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:43.443 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.443 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.443 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:25:43.443 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:58.320978+0000", 2024-09-13T11:25:43.443 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.8700000000000004e-07 2024-09-13T11:25:43.443 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.443 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.443 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:25:43.443 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:58.320985+0000", 2024-09-13T11:25:43.444 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.742e-06 2024-09-13T11:25:43.444 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.444 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.444 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:25:43.444 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:58.631147+0000", 2024-09-13T11:25:43.444 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.31016212900000001 2024-09-13T11:25:43.444 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.444 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.444 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:25:43.444 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:58.631159+0000", 2024-09-13T11:25:43.444 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1946999999999999e-05 2024-09-13T11:25:43.444 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.444 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.444 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:25:43.445 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:58.631257+0000", 2024-09-13T11:25:43.445 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.8096000000000004e-05 2024-09-13T11:25:43.445 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.445 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.445 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:25:43.445 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:58.632083+0000", 2024-09-13T11:25:43.445 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00082589499999999999 2024-09-13T11:25:43.445 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:43.445 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:25:43.445 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:43.445 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.445 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.445 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:147568 3.1d 3:b82a2ed2:::benchmark_data_smithi177_31116_object7265:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e148)", 2024-09-13T11:25:43.445 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:25:04.213894+0000", 2024-09-13T11:25:43.446 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 39.129638773000003, 2024-09-13T11:25:43.446 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.16897441999999999, 2024-09-13T11:25:43.446 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:25:43.446 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:25:43.446 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:43.446 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:25:43.446 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:25:43.446 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:43.446 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 147568 2024-09-13T11:25:43.446 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.446 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:25:43.446 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.446 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:25:43.446 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:04.213894+0000", 2024-09-13T11:25:43.446 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:43.447 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.447 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.447 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:25:43.447 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:04.213894+0000", 2024-09-13T11:25:43.447 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:43.447 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.447 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.447 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:25:43.447 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:04.213898+0000", 2024-09-13T11:25:43.447 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.9049999999999999e-06 2024-09-13T11:25:43.447 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.447 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.447 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:25:43.448 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:04.213908+0000", 2024-09-13T11:25:43.448 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0069e-05 2024-09-13T11:25:43.448 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.448 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.448 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:25:43.448 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:04.213931+0000", 2024-09-13T11:25:43.448 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.3538999999999999e-05 2024-09-13T11:25:43.448 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.448 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.448 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:25:43.448 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:04.213941+0000", 2024-09-13T11:25:43.448 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0073e-05 2024-09-13T11:25:43.448 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.448 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.449 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:25:43.449 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:04.214039+0000", 2024-09-13T11:25:43.449 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.7666999999999998e-05 2024-09-13T11:25:43.449 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.449 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.449 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:25:43.449 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:04.214226+0000", 2024-09-13T11:25:43.449 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000187166 2024-09-13T11:25:43.449 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.449 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.449 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 15", 2024-09-13T11:25:43.449 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:04.214365+0000", 2024-09-13T11:25:43.449 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000139016 2024-09-13T11:25:43.449 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.450 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.450 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:25:43.450 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:04.217864+0000", 2024-09-13T11:25:43.450 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.003499177 2024-09-13T11:25:43.450 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.450 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.450 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:25:43.450 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:04.382775+0000", 2024-09-13T11:25:43.450 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.16491091499999999 2024-09-13T11:25:43.450 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.450 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.450 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:25:43.450 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:04.382823+0000", 2024-09-13T11:25:43.450 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.8232000000000002e-05 2024-09-13T11:25:43.451 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.451 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.451 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:25:43.451 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:04.382868+0000", 2024-09-13T11:25:43.451 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.4660999999999998e-05 2024-09-13T11:25:43.451 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:43.451 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:25:43.451 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:43.451 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.451 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.451 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.4665.0:170721 3.9 e162/117 3:93244b1b:::benchmark_data_smithi177_31116_object30418:head v 162'5743, mlcod=162'5740)", 2024-09-13T11:25:43.451 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:25:21.721723+0000", 2024-09-13T11:25:43.451 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 21.621809829, 2024-09-13T11:25:43.451 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.19722754300000001, 2024-09-13T11:25:43.451 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:25:43.452 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:25:43.452 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:43.452 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:25:43.452 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.452 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:25:43.452 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:21.721723+0000", 2024-09-13T11:25:43.452 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:43.452 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.452 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.452 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:25:43.452 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:21.721723+0000", 2024-09-13T11:25:43.452 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:43.452 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.452 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.453 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:25:43.453 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:21.721726+0000", 2024-09-13T11:25:43.453 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.1930000000000002e-06 2024-09-13T11:25:43.453 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.453 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.453 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:25:43.453 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:21.721741+0000", 2024-09-13T11:25:43.453 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5299999999999999e-05 2024-09-13T11:25:43.453 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.453 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.453 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:25:43.453 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:21.915450+0000", 2024-09-13T11:25:43.453 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.193708519 2024-09-13T11:25:43.453 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.454 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.454 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:25:43.454 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:21.915462+0000", 2024-09-13T11:25:43.454 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2414e-05 2024-09-13T11:25:43.454 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.454 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.454 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:25:43.454 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:21.915603+0000", 2024-09-13T11:25:43.454 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00014116500000000001 2024-09-13T11:25:43.454 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.454 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.454 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:25:43.454 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:21.915668+0000", 2024-09-13T11:25:43.455 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.4488999999999999e-05 2024-09-13T11:25:43.455 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.455 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.455 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:25:43.455 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:21.918850+0000", 2024-09-13T11:25:43.455 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0031825310000000002 2024-09-13T11:25:43.455 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.455 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.455 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:25:43.455 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:21.918950+0000", 2024-09-13T11:25:43.455 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.9932000000000005e-05 2024-09-13T11:25:43.455 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:43.455 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:25:43.455 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:43.456 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.456 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.456 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:191058 3.1d 3:bd6e7029:::benchmark_data_smithi177_31116_object50755:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e175)", 2024-09-13T11:25:43.456 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:25:38.512403+0000", 2024-09-13T11:25:43.456 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 4.8311292840000002, 2024-09-13T11:25:43.456 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.38726333200000002, 2024-09-13T11:25:43.456 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:25:43.456 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:25:43.456 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:43.456 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:25:43.456 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:25:43.456 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:43.456 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 191058 2024-09-13T11:25:43.457 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.457 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:25:43.457 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.457 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:25:43.457 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:38.512403+0000", 2024-09-13T11:25:43.457 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:43.457 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.457 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.457 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:25:43.457 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:38.512403+0000", 2024-09-13T11:25:43.457 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:43.457 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.457 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.457 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:25:43.458 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:38.512406+0000", 2024-09-13T11:25:43.458 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.4059999999999999e-06 2024-09-13T11:25:43.458 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.458 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.458 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:25:43.458 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:38.512412+0000", 2024-09-13T11:25:43.458 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.8170000000000001e-06 2024-09-13T11:25:43.458 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.458 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.458 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:25:43.458 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:38.512430+0000", 2024-09-13T11:25:43.458 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.7397999999999998e-05 2024-09-13T11:25:43.458 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.459 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.459 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:25:43.459 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:38.512438+0000", 2024-09-13T11:25:43.459 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.3459999999999994e-06 2024-09-13T11:25:43.459 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.459 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.459 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:25:43.459 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:38.512526+0000", 2024-09-13T11:25:43.459 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.8146999999999995e-05 2024-09-13T11:25:43.459 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.459 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.459 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:25:43.460 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:38.512722+0000", 2024-09-13T11:25:43.460 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00019587599999999999 2024-09-13T11:25:43.460 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.460 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.460 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 15", 2024-09-13T11:25:43.460 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:38.512882+0000", 2024-09-13T11:25:43.460 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00015957300000000001 2024-09-13T11:25:43.460 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.460 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.460 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:25:43.460 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:38.517854+0000", 2024-09-13T11:25:43.460 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0049725500000000001 2024-09-13T11:25:43.460 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.460 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.461 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:25:43.461 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:38.899562+0000", 2024-09-13T11:25:43.461 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.38170719800000003 2024-09-13T11:25:43.461 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.461 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.461 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:25:43.461 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:38.899622+0000", 2024-09-13T11:25:43.461 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.0396999999999998e-05 2024-09-13T11:25:43.462 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.462 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.462 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:25:43.462 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:38.899667+0000", 2024-09-13T11:25:43.462 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.4623999999999998e-05 2024-09-13T11:25:43.462 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:43.462 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:25:43.462 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:43.462 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.462 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.462 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.4665.0:191547 3.25 e176/141 3:a7ac3e5b:::benchmark_data_smithi177_31116_object51244:head v 176'6020, mlcod=175'6018)", 2024-09-13T11:25:43.462 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:25:39.496290+0000", 2024-09-13T11:25:43.463 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 3.8472427229999999, 2024-09-13T11:25:43.463 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.179206057, 2024-09-13T11:25:43.463 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:25:43.463 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:25:43.463 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:43.463 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:25:43.463 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.463 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:25:43.463 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:39.496290+0000", 2024-09-13T11:25:43.463 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:43.463 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.463 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.463 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:25:43.464 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:39.496290+0000", 2024-09-13T11:25:43.464 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:43.464 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.464 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.464 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:25:43.464 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:39.496293+0000", 2024-09-13T11:25:43.464 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.8499999999999998e-06 2024-09-13T11:25:43.464 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.464 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.464 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:25:43.464 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:39.496305+0000", 2024-09-13T11:25:43.464 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2286999999999999e-05 2024-09-13T11:25:43.464 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.465 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.465 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:25:43.465 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:39.670614+0000", 2024-09-13T11:25:43.465 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.17430884699999999 2024-09-13T11:25:43.465 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.465 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.465 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:25:43.465 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:39.670625+0000", 2024-09-13T11:25:43.465 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1233000000000001e-05 2024-09-13T11:25:43.465 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.465 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.465 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:25:43.466 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:39.670722+0000", 2024-09-13T11:25:43.466 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.6885999999999994e-05 2024-09-13T11:25:43.466 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.466 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.466 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:25:43.466 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:39.670756+0000", 2024-09-13T11:25:43.466 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.4125999999999998e-05 2024-09-13T11:25:43.466 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.466 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.466 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:25:43.466 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:39.675411+0000", 2024-09-13T11:25:43.466 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0046546160000000003 2024-09-13T11:25:43.466 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.467 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.467 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:25:43.467 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:39.675496+0000", 2024-09-13T11:25:43.467 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.5211999999999997e-05 2024-09-13T11:25:43.467 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:43.467 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:25:43.467 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:43.467 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:43.467 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:25:43.467 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-13T11:25:43.468 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-13T11:25:43.470 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-13T11:25:43.471 INFO:tasks.ceph.osd.7.smithi177.stderr:2024-09-13T11:25:43.466+0000 7f897b51d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26575) UID: 0 2024-09-13T11:25:43.571 INFO:tasks.ceph.osd.9.smithi092.stderr:2024-09-13T11:25:43.569+0000 7ff5ab3d0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26348) UID: 0 2024-09-13T11:25:43.652 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-13T11:25:43.652 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [], 2024-09-13T11:25:43.652 INFO:teuthology.orchestra.run.smithi162.stdout: "num_ops": 0 2024-09-13T11:25:43.652 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-13T11:25:43.669 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-13T11:25:43.671 INFO:tasks.ceph.osd.15.smithi177.stderr:2024-09-13T11:25:43.666+0000 7f9d6ce77640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26578) UID: 0 2024-09-13T11:25:43.772 INFO:tasks.ceph.osd.12.smithi001.stderr:2024-09-13T11:25:43.771+0000 7f892c6e6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26314) UID: 0 2024-09-13T11:25:43.793 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-13T11:25:43.793 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [], 2024-09-13T11:25:43.793 INFO:teuthology.orchestra.run.smithi162.stdout: "complaint_time": 30, 2024-09-13T11:25:43.793 INFO:teuthology.orchestra.run.smithi162.stdout: "num_blocked_ops": 0 2024-09-13T11:25:43.793 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-13T11:25:43.809 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-13T11:25:43.873 INFO:tasks.ceph.osd.12.smithi001.stderr:2024-09-13T11:25:43.871+0000 7f892c6e6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26314) UID: 0 2024-09-13T11:25:43.935 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-13T11:25:43.936 INFO:teuthology.orchestra.run.smithi162.stdout: "size": 20, 2024-09-13T11:25:43.936 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 600, 2024-09-13T11:25:43.936 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [ 2024-09-13T11:25:43.936 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.936 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4257.0:80 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 262144~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-13T11:25:43.936 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:22:12.516065+0000", 2024-09-13T11:25:43.936 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 211.417929915, 2024-09-13T11:25:43.936 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.46461997999999999, 2024-09-13T11:25:43.936 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:25:43.936 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:25:43.936 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:43.937 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:25:43.937 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4257", 2024-09-13T11:25:43.937 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/1488131527", 2024-09-13T11:25:43.937 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 80 2024-09-13T11:25:43.937 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.937 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:25:43.937 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.937 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:25:43.937 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.516065+0000", 2024-09-13T11:25:43.937 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:43.937 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.938 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.938 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:25:43.938 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.516065+0000", 2024-09-13T11:25:43.938 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:43.938 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.938 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.938 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:25:43.938 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.516066+0000", 2024-09-13T11:25:43.938 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0869999999999999e-06 2024-09-13T11:25:43.938 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.938 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.938 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:25:43.938 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.516112+0000", 2024-09-13T11:25:43.938 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.5503000000000001e-05 2024-09-13T11:25:43.939 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.939 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.939 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:25:43.939 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.516128+0000", 2024-09-13T11:25:43.939 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5891000000000001e-05 2024-09-13T11:25:43.939 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.939 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.939 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:25:43.939 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.516138+0000", 2024-09-13T11:25:43.939 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.8120000000000002e-06 2024-09-13T11:25:43.939 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.939 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.939 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:25:43.940 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.520398+0000", 2024-09-13T11:25:43.940 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0042605450000000001 2024-09-13T11:25:43.940 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.940 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.940 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:25:43.940 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.520526+0000", 2024-09-13T11:25:43.940 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000128355 2024-09-13T11:25:43.940 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.940 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.940 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 0", 2024-09-13T11:25:43.940 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.520730+0000", 2024-09-13T11:25:43.940 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00020337799999999999 2024-09-13T11:25:43.940 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.941 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.941 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:25:43.941 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.522417+0000", 2024-09-13T11:25:43.941 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001687184 2024-09-13T11:25:43.941 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.941 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.941 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:25:43.941 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.980408+0000", 2024-09-13T11:25:43.941 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.45799068399999998 2024-09-13T11:25:43.941 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.942 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.942 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:25:43.942 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.980565+0000", 2024-09-13T11:25:43.942 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00015760899999999999 2024-09-13T11:25:43.942 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.942 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.942 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:25:43.942 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.980685+0000", 2024-09-13T11:25:43.942 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000119932 2024-09-13T11:25:43.942 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:43.942 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:25:43.942 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:43.942 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.943 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.943 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop_reply(client.4257.0:80 1.0 e24/23 ondisk, result = 0)", 2024-09-13T11:25:43.943 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:22:12.522938+0000", 2024-09-13T11:25:43.943 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 211.411056882, 2024-09-13T11:25:43.943 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.45776420600000001, 2024-09-13T11:25:43.943 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:25:43.943 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:25:43.943 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:25:43.943 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:25:43.943 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.943 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:25:43.943 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.522938+0000", 2024-09-13T11:25:43.943 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:43.944 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.944 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.944 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:25:43.944 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.522938+0000", 2024-09-13T11:25:43.944 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:43.944 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.944 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.944 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:25:43.944 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.522941+0000", 2024-09-13T11:25:43.944 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.0290000000000001e-06 2024-09-13T11:25:43.944 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.945 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.945 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:25:43.945 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.522950+0000", 2024-09-13T11:25:43.945 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.7499999999999992e-06 2024-09-13T11:25:43.945 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.945 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.945 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:25:43.945 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.980215+0000", 2024-09-13T11:25:43.945 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.45726523800000002 2024-09-13T11:25:43.945 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.945 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.945 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:25:43.945 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.980229+0000", 2024-09-13T11:25:43.945 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3549e-05 2024-09-13T11:25:43.946 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.946 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.946 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:25:43.946 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.980365+0000", 2024-09-13T11:25:43.946 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00013608700000000001 2024-09-13T11:25:43.946 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.946 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.946 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:25:43.946 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.980388+0000", 2024-09-13T11:25:43.946 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.3112000000000002e-05 2024-09-13T11:25:43.947 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.947 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.947 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:25:43.947 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.980702+0000", 2024-09-13T11:25:43.947 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00031444099999999999 2024-09-13T11:25:43.947 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:43.947 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:25:43.947 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:43.948 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.948 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.948 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:25416 3.8 3:1a1fbdd1:::benchmark_data_smithi177_31116_object25415:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e56)", 2024-09-13T11:25:43.948 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:04.512188+0000", 2024-09-13T11:25:43.948 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 159.42180691499999, 2024-09-13T11:25:43.948 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.210208697, 2024-09-13T11:25:43.948 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:25:43.948 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:25:43.948 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:43.948 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:25:43.948 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:25:43.948 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:43.948 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 25416 2024-09-13T11:25:43.949 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.949 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:25:43.949 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.949 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:25:43.949 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.512188+0000", 2024-09-13T11:25:43.949 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:43.949 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.949 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.949 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:25:43.949 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.512188+0000", 2024-09-13T11:25:43.949 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:43.949 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.950 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.950 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:25:43.950 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.512192+0000", 2024-09-13T11:25:43.950 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.2660000000000003e-06 2024-09-13T11:25:43.950 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.950 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.950 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:25:43.950 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.512247+0000", 2024-09-13T11:25:43.950 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.4889000000000003e-05 2024-09-13T11:25:43.950 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.950 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.950 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:25:43.951 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.512279+0000", 2024-09-13T11:25:43.951 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.1146000000000003e-05 2024-09-13T11:25:43.951 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.951 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.951 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:25:43.951 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.512297+0000", 2024-09-13T11:25:43.951 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.825e-05 2024-09-13T11:25:43.951 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.951 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.952 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:25:43.952 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.512377+0000", 2024-09-13T11:25:43.952 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.9875000000000004e-05 2024-09-13T11:25:43.952 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.952 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.952 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:25:43.952 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.512570+0000", 2024-09-13T11:25:43.952 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00019312100000000001 2024-09-13T11:25:43.952 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.952 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.952 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 1", 2024-09-13T11:25:43.952 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.512798+0000", 2024-09-13T11:25:43.953 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00022847899999999999 2024-09-13T11:25:43.953 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.953 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.953 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:25:43.953 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.514975+0000", 2024-09-13T11:25:43.953 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0021769979999999999 2024-09-13T11:25:43.953 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.953 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.953 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:25:43.954 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.722207+0000", 2024-09-13T11:25:43.954 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.20723217099999999 2024-09-13T11:25:43.954 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.954 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.954 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:25:43.954 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.722314+0000", 2024-09-13T11:25:43.954 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00010648400000000001 2024-09-13T11:25:43.954 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.955 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.955 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:25:43.955 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.722397+0000", 2024-09-13T11:25:43.955 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.3017999999999997e-05 2024-09-13T11:25:43.955 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:43.956 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:25:43.956 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:43.956 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.956 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.956 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop_reply(client.4665.0:25416 3.8 e56/40 ondisk, result = 0)", 2024-09-13T11:25:43.956 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:04.515635+0000", 2024-09-13T11:25:43.956 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 159.41836026300001, 2024-09-13T11:25:43.956 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.20677600500000001, 2024-09-13T11:25:43.957 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:25:43.957 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:25:43.957 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:25:43.957 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:25:43.957 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.957 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:25:43.957 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.515635+0000", 2024-09-13T11:25:43.957 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:43.957 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.957 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.957 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:25:43.957 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.515635+0000", 2024-09-13T11:25:43.958 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:43.958 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.958 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.958 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:25:43.958 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.515639+0000", 2024-09-13T11:25:43.958 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.619e-06 2024-09-13T11:25:43.958 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.958 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.958 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:25:43.958 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.515650+0000", 2024-09-13T11:25:43.958 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.111e-05 2024-09-13T11:25:43.958 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.958 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.958 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:25:43.959 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.722004+0000", 2024-09-13T11:25:43.959 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.206354805 2024-09-13T11:25:43.959 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.959 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.959 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:25:43.959 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.722017+0000", 2024-09-13T11:25:43.959 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2840000000000001e-05 2024-09-13T11:25:43.959 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.959 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.959 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:25:43.959 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.722186+0000", 2024-09-13T11:25:43.959 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00016898299999999999 2024-09-13T11:25:43.959 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.959 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.960 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:25:43.960 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.722199+0000", 2024-09-13T11:25:43.960 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.289e-05 2024-09-13T11:25:43.960 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.960 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.960 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:25:43.960 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.722411+0000", 2024-09-13T11:25:43.960 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00021175799999999999 2024-09-13T11:25:43.960 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:43.960 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:25:43.960 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:43.960 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.960 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.961 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:30833 3.8 3:1c203f8d:::benchmark_data_smithi177_31116_object30832:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e59)", 2024-09-13T11:25:43.961 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:09.506656+0000", 2024-09-13T11:25:43.961 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 154.42733936499999, 2024-09-13T11:25:43.961 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.224862282, 2024-09-13T11:25:43.961 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:25:43.961 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:25:43.961 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:43.961 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:25:43.961 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:25:43.961 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:43.961 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 30833 2024-09-13T11:25:43.961 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.961 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:25:43.962 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.962 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:25:43.962 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.506656+0000", 2024-09-13T11:25:43.962 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:43.962 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.962 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.962 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:25:43.962 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.506656+0000", 2024-09-13T11:25:43.962 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:43.962 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.962 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.962 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:25:43.962 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.506660+0000", 2024-09-13T11:25:43.962 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.6e-06 2024-09-13T11:25:43.963 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.963 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.963 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:25:43.963 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.506717+0000", 2024-09-13T11:25:43.963 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.6131999999999997e-05 2024-09-13T11:25:43.963 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.963 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.963 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:25:43.963 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.506751+0000", 2024-09-13T11:25:43.963 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.4746000000000002e-05 2024-09-13T11:25:43.963 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.963 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.963 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:25:43.964 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.506774+0000", 2024-09-13T11:25:43.964 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.2636000000000001e-05 2024-09-13T11:25:43.964 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.964 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.964 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:25:43.964 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.506878+0000", 2024-09-13T11:25:43.964 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000104057 2024-09-13T11:25:43.964 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.964 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.964 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:25:43.964 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.507052+0000", 2024-09-13T11:25:43.964 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00017402800000000001 2024-09-13T11:25:43.964 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.964 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.965 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 1", 2024-09-13T11:25:43.965 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.507217+0000", 2024-09-13T11:25:43.968 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00016516 2024-09-13T11:25:43.968 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.968 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.968 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:25:43.969 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.508992+0000", 2024-09-13T11:25:43.969 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0017743990000000001 2024-09-13T11:25:43.969 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.969 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.969 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:25:43.969 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.731429+0000", 2024-09-13T11:25:43.969 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.22243718000000001 2024-09-13T11:25:43.969 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.969 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.969 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:25:43.969 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.731479+0000", 2024-09-13T11:25:43.969 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.0146999999999999e-05 2024-09-13T11:25:43.969 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.970 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.970 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:25:43.970 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.731518+0000", 2024-09-13T11:25:43.970 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.9196999999999998e-05 2024-09-13T11:25:43.970 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:43.970 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:25:43.970 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:43.970 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.970 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.970 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop_reply(client.4665.0:30833 3.8 e59/40 ondisk, result = 0)", 2024-09-13T11:25:43.970 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:09.508907+0000", 2024-09-13T11:25:43.970 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 154.42508827099999, 2024-09-13T11:25:43.970 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.222618862, 2024-09-13T11:25:43.970 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:25:43.971 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:25:43.971 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:25:43.971 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:25:43.971 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.971 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:25:43.971 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.508907+0000", 2024-09-13T11:25:43.971 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:43.971 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.971 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.971 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:25:43.971 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.508907+0000", 2024-09-13T11:25:43.971 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:43.971 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.971 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.972 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:25:43.972 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.508910+0000", 2024-09-13T11:25:43.972 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.993e-06 2024-09-13T11:25:43.972 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.972 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.972 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:25:43.972 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.508918+0000", 2024-09-13T11:25:43.972 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.7000000000000008e-06 2024-09-13T11:25:43.972 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.972 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.972 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:25:43.972 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.731341+0000", 2024-09-13T11:25:43.973 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.22242308699999999 2024-09-13T11:25:43.973 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.973 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.973 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:25:43.973 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.731350+0000", 2024-09-13T11:25:43.973 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.3179999999999996e-06 2024-09-13T11:25:43.973 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.973 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.973 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:25:43.973 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.731418+0000", 2024-09-13T11:25:43.973 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.7586e-05 2024-09-13T11:25:43.973 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.973 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.974 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:25:43.974 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.731424+0000", 2024-09-13T11:25:43.974 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.844e-06 2024-09-13T11:25:43.974 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.974 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.974 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:25:43.974 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.731526+0000", 2024-09-13T11:25:43.974 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000101334 2024-09-13T11:25:43.974 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:43.974 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:25:43.974 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:43.974 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.974 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.974 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPush(3.10 76/70 [PushOp(3:0e7d2700:::benchmark_data_smithi177_31116_object17406:head, version: 52'1075, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:0e7d2700:::benchmark_data_smithi177_31116_object17406:head@52'1075, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:25:43.975 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:32.206198+0000", 2024-09-13T11:25:43.975 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 131.72779756200001, 2024-09-13T11:25:43.975 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.25193395000000002, 2024-09-13T11:25:43.975 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:25:43.975 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:25:43.975 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:25:43.975 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:25:43.975 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.975 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:25:43.975 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.206198+0000", 2024-09-13T11:25:43.975 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:43.975 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.975 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.976 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:25:43.976 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.206198+0000", 2024-09-13T11:25:43.976 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:43.976 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.976 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.976 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:25:43.976 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.206199+0000", 2024-09-13T11:25:43.976 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.46e-06 2024-09-13T11:25:43.976 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.976 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.976 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:25:43.976 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.206250+0000", 2024-09-13T11:25:43.976 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.113e-05 2024-09-13T11:25:43.977 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.977 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.977 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:25:43.977 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.456042+0000", 2024-09-13T11:25:43.977 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.24979182999999999 2024-09-13T11:25:43.977 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.977 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.977 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:25:43.977 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.456084+0000", 2024-09-13T11:25:43.977 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.1814000000000003e-05 2024-09-13T11:25:43.978 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.978 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.978 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:25:43.978 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.456342+0000", 2024-09-13T11:25:43.978 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000258404 2024-09-13T11:25:43.978 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.978 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.978 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:25:43.978 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.456367+0000", 2024-09-13T11:25:43.978 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.5099000000000001e-05 2024-09-13T11:25:43.978 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.978 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.978 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:25:43.978 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.458132+0000", 2024-09-13T11:25:43.979 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0017642129999999999 2024-09-13T11:25:43.979 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:43.979 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:25:43.979 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:43.979 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.979 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.979 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPush(3.10 76/70 [PushOp(3:0eee8bcf:::benchmark_data_smithi177_31116_object17589:head, version: 53'1086, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:0eee8bcf:::benchmark_data_smithi177_31116_object17589:head@53'1086, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:25:43.979 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:32.585183+0000", 2024-09-13T11:25:43.979 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 131.348811668, 2024-09-13T11:25:43.979 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.16696202199999999, 2024-09-13T11:25:43.979 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:25:43.979 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:25:43.979 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:25:43.979 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:25:43.980 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.980 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:25:43.980 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.585183+0000", 2024-09-13T11:25:43.980 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:43.980 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.980 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.980 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:25:43.980 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.585183+0000", 2024-09-13T11:25:43.980 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:43.980 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.980 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.980 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:25:43.980 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.585187+0000", 2024-09-13T11:25:43.980 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.8480000000000003e-06 2024-09-13T11:25:43.981 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.981 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.981 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:25:43.981 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.585264+0000", 2024-09-13T11:25:43.981 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.6327000000000003e-05 2024-09-13T11:25:43.981 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.981 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.981 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:25:43.981 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.750459+0000", 2024-09-13T11:25:43.981 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.16519551900000001 2024-09-13T11:25:43.981 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.981 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.981 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:25:43.981 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.750480+0000", 2024-09-13T11:25:43.982 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.0984000000000001e-05 2024-09-13T11:25:43.982 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.982 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.982 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:25:43.982 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.750633+0000", 2024-09-13T11:25:43.982 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000153148 2024-09-13T11:25:43.982 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.982 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.982 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:25:43.982 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.750652+0000", 2024-09-13T11:25:43.982 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.8981e-05 2024-09-13T11:25:43.982 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.982 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.983 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:25:43.983 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.752145+0000", 2024-09-13T11:25:43.983 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001493215 2024-09-13T11:25:43.983 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:43.983 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:25:43.983 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:43.983 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.983 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.983 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:60417 3.10 3:092808b0:::benchmark_data_smithi177_31116_object60416:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e85)", 2024-09-13T11:25:43.983 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:42.173238+0000", 2024-09-13T11:25:43.983 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 121.76075665400001, 2024-09-13T11:25:43.983 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.38965994599999998, 2024-09-13T11:25:43.983 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:25:43.983 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:25:43.984 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:43.984 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:25:43.984 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:25:43.984 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:43.984 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 60417 2024-09-13T11:25:43.984 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.984 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:25:43.984 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.984 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:25:43.984 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.173238+0000", 2024-09-13T11:25:43.984 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:43.984 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.984 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.985 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:25:43.985 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.173238+0000", 2024-09-13T11:25:43.985 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:43.985 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.985 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.985 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:25:43.985 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.173241+0000", 2024-09-13T11:25:43.985 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.6819999999999999e-06 2024-09-13T11:25:43.985 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.985 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.985 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:25:43.985 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.173335+0000", 2024-09-13T11:25:43.986 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.3957999999999997e-05 2024-09-13T11:25:43.986 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.986 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.986 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:25:43.986 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.173358+0000", 2024-09-13T11:25:43.986 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.3023999999999998e-05 2024-09-13T11:25:43.986 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.986 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.986 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:25:43.986 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.173373+0000", 2024-09-13T11:25:43.986 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4867e-05 2024-09-13T11:25:43.986 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.986 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.986 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:25:43.987 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.173466+0000", 2024-09-13T11:25:43.987 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.2542999999999998e-05 2024-09-13T11:25:43.987 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.987 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.987 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:25:43.987 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.173754+0000", 2024-09-13T11:25:43.987 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000288657 2024-09-13T11:25:43.987 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.987 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.987 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 0", 2024-09-13T11:25:43.987 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.174066+0000", 2024-09-13T11:25:43.987 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00031170999999999998 2024-09-13T11:25:43.987 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.987 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.988 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:25:43.988 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.178245+0000", 2024-09-13T11:25:43.988 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00417864 2024-09-13T11:25:43.988 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.988 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.988 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:25:43.988 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.562617+0000", 2024-09-13T11:25:43.988 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.38437253199999999 2024-09-13T11:25:43.988 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.988 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.988 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:25:43.988 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.562782+0000", 2024-09-13T11:25:43.988 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00016499200000000001 2024-09-13T11:25:43.988 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.989 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.989 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:25:43.989 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.562898+0000", 2024-09-13T11:25:43.989 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000116341 2024-09-13T11:25:43.989 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:43.989 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:25:43.989 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:43.989 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.989 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.989 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:83193 3.8 3:10066d83:::benchmark_data_smithi177_31116_object83192:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106)", 2024-09-13T11:25:43.989 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:24:04.283923+0000", 2024-09-13T11:25:43.989 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 99.650071971000003, 2024-09-13T11:25:43.989 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.64775463499999997, 2024-09-13T11:25:43.989 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:25:43.990 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:25:43.990 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:43.990 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:25:43.990 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:25:43.990 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:43.990 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 83193 2024-09-13T11:25:43.990 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.990 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:25:43.990 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.990 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:25:43.990 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.283923+0000", 2024-09-13T11:25:43.990 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:43.990 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.990 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.991 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:25:43.991 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.283923+0000", 2024-09-13T11:25:43.991 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:43.991 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.991 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.991 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:25:43.991 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.283925+0000", 2024-09-13T11:25:43.991 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.9489999999999999e-06 2024-09-13T11:25:43.991 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.991 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.991 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:25:43.991 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.283960+0000", 2024-09-13T11:25:43.991 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.5063e-05 2024-09-13T11:25:43.991 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.992 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.992 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:25:43.992 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.283974+0000", 2024-09-13T11:25:43.992 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3546e-05 2024-09-13T11:25:43.992 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.992 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.992 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:25:43.992 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.283983+0000", 2024-09-13T11:25:43.992 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.2499999999999995e-06 2024-09-13T11:25:43.992 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.992 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.993 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:25:43.993 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.284061+0000", 2024-09-13T11:25:43.993 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.7946000000000002e-05 2024-09-13T11:25:43.993 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.993 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.993 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:25:43.993 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.284270+0000", 2024-09-13T11:25:43.993 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00020876799999999999 2024-09-13T11:25:43.993 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.993 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.993 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 1", 2024-09-13T11:25:43.993 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.284489+0000", 2024-09-13T11:25:43.993 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00021956800000000001 2024-09-13T11:25:43.993 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.994 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.994 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:25:43.994 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.286456+0000", 2024-09-13T11:25:43.994 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0019665809999999998 2024-09-13T11:25:43.994 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.994 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.994 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:25:43.994 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.931497+0000", 2024-09-13T11:25:43.994 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.64504147700000003 2024-09-13T11:25:43.994 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.994 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.994 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:25:43.994 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.931589+0000", 2024-09-13T11:25:43.994 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.2041999999999997e-05 2024-09-13T11:25:43.995 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.995 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.995 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:25:43.995 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.931678+0000", 2024-09-13T11:25:43.995 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.8444999999999998e-05 2024-09-13T11:25:43.995 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:43.995 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:25:43.995 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:43.995 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.995 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.995 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:115060 3.8 3:11200ba9:::benchmark_data_smithi177_31116_object115059:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e125)", 2024-09-13T11:25:43.995 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:24:33.602178+0000", 2024-09-13T11:25:43.995 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 70.331816977000003, 2024-09-13T11:25:43.995 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.23668186799999999, 2024-09-13T11:25:43.996 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:25:43.996 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:25:43.996 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:43.996 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:25:43.996 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:25:43.996 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:43.996 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 115060 2024-09-13T11:25:43.996 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.996 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:25:43.996 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.996 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:25:43.996 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:33.602178+0000", 2024-09-13T11:25:43.997 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:43.997 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.997 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.997 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:25:43.997 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:33.602178+0000", 2024-09-13T11:25:43.997 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:43.997 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.997 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.997 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:25:43.997 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:33.602181+0000", 2024-09-13T11:25:43.997 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.0369999999999999e-06 2024-09-13T11:25:43.997 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.997 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.997 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:25:43.998 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:33.602217+0000", 2024-09-13T11:25:43.998 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.5837999999999998e-05 2024-09-13T11:25:43.998 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.998 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.998 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:25:43.998 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:33.602235+0000", 2024-09-13T11:25:43.998 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.7571000000000002e-05 2024-09-13T11:25:43.998 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.998 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.998 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:25:43.998 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:33.602245+0000", 2024-09-13T11:25:43.998 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0360000000000001e-05 2024-09-13T11:25:43.998 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.998 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.999 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:25:43.999 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:33.602312+0000", 2024-09-13T11:25:43.999 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.6989000000000005e-05 2024-09-13T11:25:43.999 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.999 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.999 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:25:43.999 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:33.602490+0000", 2024-09-13T11:25:43.999 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000178464 2024-09-13T11:25:43.999 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:43.999 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:43.999 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 1", 2024-09-13T11:25:43.999 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:33.602670+0000", 2024-09-13T11:25:43.999 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00018002799999999999 2024-09-13T11:25:44.000 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.000 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.000 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:25:44.000 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:33.604382+0000", 2024-09-13T11:25:44.000 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001711314 2024-09-13T11:25:44.000 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.000 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.000 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:25:44.000 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:33.838738+0000", 2024-09-13T11:25:44.000 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.23435634699999999 2024-09-13T11:25:44.000 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.001 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.001 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:25:44.001 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:33.838815+0000", 2024-09-13T11:25:44.001 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.7381999999999999e-05 2024-09-13T11:25:44.001 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.001 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.001 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:25:44.001 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:33.838860+0000", 2024-09-13T11:25:44.001 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.4537999999999999e-05 2024-09-13T11:25:44.001 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:44.001 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:25:44.001 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:44.001 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.001 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.002 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop_reply(client.4665.0:115060 3.8 e125/117 ondisk, result = 0)", 2024-09-13T11:25:44.002 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:24:33.604404+0000", 2024-09-13T11:25:44.002 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 70.329590718999995, 2024-09-13T11:25:44.002 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.23446235400000001, 2024-09-13T11:25:44.002 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:25:44.002 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:25:44.002 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:25:44.002 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:25:44.002 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.002 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:25:44.002 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:33.604404+0000", 2024-09-13T11:25:44.002 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:44.002 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.002 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.003 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:25:44.003 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:33.604404+0000", 2024-09-13T11:25:44.003 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:44.003 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.003 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.003 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:25:44.003 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:33.604406+0000", 2024-09-13T11:25:44.003 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.418e-06 2024-09-13T11:25:44.003 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.003 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.003 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:25:44.003 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:33.604411+0000", 2024-09-13T11:25:44.003 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.3519999999999997e-06 2024-09-13T11:25:44.003 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.004 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.004 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:25:44.004 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:33.838578+0000", 2024-09-13T11:25:44.004 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.23416656199999999 2024-09-13T11:25:44.004 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.004 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.004 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:25:44.004 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:33.838586+0000", 2024-09-13T11:25:44.004 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.9920000000000007e-06 2024-09-13T11:25:44.004 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.004 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.004 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:25:44.004 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:33.838655+0000", 2024-09-13T11:25:44.004 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.9463999999999998e-05 2024-09-13T11:25:44.005 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.005 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.005 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:25:44.005 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:33.838731+0000", 2024-09-13T11:25:44.005 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.5487000000000004e-05 2024-09-13T11:25:44.005 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.005 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.005 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:25:44.005 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:33.838867+0000", 2024-09-13T11:25:44.005 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000136079 2024-09-13T11:25:44.005 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:44.005 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:25:44.005 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:44.006 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.006 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.006 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPush(3.6 129/122 [PushOp(3:6261f3b6:::benchmark_data_smithi177_31116_object64090:head, version: 90'3486, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:6261f3b6:::benchmark_data_smithi177_31116_object64090:head@90'3486, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:25:44.006 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:24:38.641487+0000", 2024-09-13T11:25:44.006 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 65.292508307999995, 2024-09-13T11:25:44.006 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.209889098, 2024-09-13T11:25:44.006 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:25:44.006 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:25:44.006 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:25:44.006 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:25:44.006 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.006 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:25:44.006 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:38.641487+0000", 2024-09-13T11:25:44.007 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:44.007 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.007 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.007 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:25:44.007 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:38.641487+0000", 2024-09-13T11:25:44.007 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:44.007 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.007 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.007 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:25:44.007 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:38.641489+0000", 2024-09-13T11:25:44.007 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.0980000000000001e-06 2024-09-13T11:25:44.007 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.007 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.007 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:25:44.007 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:38.641560+0000", 2024-09-13T11:25:44.008 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.1404999999999998e-05 2024-09-13T11:25:44.008 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.008 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.008 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:25:44.008 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:38.850498+0000", 2024-09-13T11:25:44.008 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.20893800500000001 2024-09-13T11:25:44.008 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.008 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.008 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:25:44.008 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:38.850510+0000", 2024-09-13T11:25:44.008 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1762e-05 2024-09-13T11:25:44.008 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.008 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.008 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:25:44.009 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:38.850610+0000", 2024-09-13T11:25:44.009 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00010011200000000001 2024-09-13T11:25:44.009 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.009 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.009 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:25:44.009 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:38.850628+0000", 2024-09-13T11:25:44.009 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.7291000000000001e-05 2024-09-13T11:25:44.009 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.009 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.009 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:25:44.009 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:38.851376+0000", 2024-09-13T11:25:44.009 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00074842500000000005 2024-09-13T11:25:44.009 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:44.009 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:25:44.010 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:44.010 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.010 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.010 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.4665.0:126919 3.6 e130/122 3:634ce08e:::benchmark_data_smithi177_31116_object126918:head v 130'5137, mlcod=130'5134)", 2024-09-13T11:25:44.010 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:24:43.780120+0000", 2024-09-13T11:25:44.010 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 60.153875417000002, 2024-09-13T11:25:44.010 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.298086295, 2024-09-13T11:25:44.010 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:25:44.010 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:25:44.010 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:44.010 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:25:44.010 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.010 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:25:44.010 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:43.780120+0000", 2024-09-13T11:25:44.011 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:44.011 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.011 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.011 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:25:44.011 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:43.780120+0000", 2024-09-13T11:25:44.011 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:44.011 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.011 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.011 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:25:44.011 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:43.780123+0000", 2024-09-13T11:25:44.011 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.79e-06 2024-09-13T11:25:44.011 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.011 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.011 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:25:44.012 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:43.780160+0000", 2024-09-13T11:25:44.012 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.7945999999999999e-05 2024-09-13T11:25:44.012 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.012 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.012 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:25:44.012 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:44.075853+0000", 2024-09-13T11:25:44.012 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.29569234999999999 2024-09-13T11:25:44.012 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.012 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.012 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:25:44.012 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:44.075861+0000", 2024-09-13T11:25:44.012 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.9510000000000006e-06 2024-09-13T11:25:44.012 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.013 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.013 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:25:44.013 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:44.075924+0000", 2024-09-13T11:25:44.013 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.2946000000000003e-05 2024-09-13T11:25:44.013 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.013 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.013 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:25:44.013 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:44.075942+0000", 2024-09-13T11:25:44.013 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.8108e-05 2024-09-13T11:25:44.013 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.013 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.013 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:25:44.013 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:44.078164+0000", 2024-09-13T11:25:44.013 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.002222375 2024-09-13T11:25:44.014 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.014 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.014 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:25:44.014 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:44.078206+0000", 2024-09-13T11:25:44.014 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.1829000000000002e-05 2024-09-13T11:25:44.014 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:44.014 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:25:44.014 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:44.014 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.014 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.014 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:137473 3.10 3:094c0efb:::benchmark_data_smithi177_31116_object137472:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e139)", 2024-09-13T11:25:44.014 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:24:53.523319+0000", 2024-09-13T11:25:44.014 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 50.410676377999998, 2024-09-13T11:25:44.015 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.18297369299999999, 2024-09-13T11:25:44.015 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:25:44.015 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:25:44.015 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:44.015 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:25:44.015 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:25:44.015 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:44.015 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 137473 2024-09-13T11:25:44.015 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.015 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:25:44.015 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.016 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:25:44.016 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:53.523319+0000", 2024-09-13T11:25:44.016 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:44.016 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.016 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.016 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:25:44.016 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:53.523319+0000", 2024-09-13T11:25:44.016 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:44.016 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.016 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.016 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:25:44.017 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:53.523322+0000", 2024-09-13T11:25:44.017 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.2119999999999999e-06 2024-09-13T11:25:44.017 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.017 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.017 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:25:44.017 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:53.523370+0000", 2024-09-13T11:25:44.017 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.7788999999999999e-05 2024-09-13T11:25:44.017 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.017 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.017 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:25:44.017 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:53.523395+0000", 2024-09-13T11:25:44.017 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.5202e-05 2024-09-13T11:25:44.017 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.017 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.018 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:25:44.018 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:53.523409+0000", 2024-09-13T11:25:44.018 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4263999999999999e-05 2024-09-13T11:25:44.018 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.018 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.018 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:25:44.018 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:53.523477+0000", 2024-09-13T11:25:44.018 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.7578999999999998e-05 2024-09-13T11:25:44.018 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.018 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.018 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:25:44.018 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:53.523638+0000", 2024-09-13T11:25:44.018 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00016078300000000001 2024-09-13T11:25:44.018 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.019 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.019 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 0", 2024-09-13T11:25:44.019 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:53.523800+0000", 2024-09-13T11:25:44.019 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000162564 2024-09-13T11:25:44.019 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.019 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.019 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:25:44.019 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:53.526960+0000", 2024-09-13T11:25:44.019 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0031593950000000002 2024-09-13T11:25:44.019 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.019 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.019 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:25:44.019 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:53.706180+0000", 2024-09-13T11:25:44.019 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.17922080600000001 2024-09-13T11:25:44.020 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.020 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.020 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:25:44.020 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:53.706237+0000", 2024-09-13T11:25:44.020 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.6845000000000002e-05 2024-09-13T11:25:44.020 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.020 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.020 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:25:44.020 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:53.706292+0000", 2024-09-13T11:25:44.020 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.5254000000000003e-05 2024-09-13T11:25:44.020 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:44.020 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:25:44.020 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:44.021 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.021 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.021 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop_reply(client.4665.0:137473 3.10 e139/84 ondisk, result = 0)", 2024-09-13T11:25:44.021 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:24:53.527076+0000", 2024-09-13T11:25:44.021 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 50.406919551000001, 2024-09-13T11:25:44.021 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.17922728099999999, 2024-09-13T11:25:44.021 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:25:44.021 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:25:44.021 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:25:44.021 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:25:44.021 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.021 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:25:44.021 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:53.527076+0000", 2024-09-13T11:25:44.022 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:44.022 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.022 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.022 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:25:44.022 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:53.527076+0000", 2024-09-13T11:25:44.022 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:44.022 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.022 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.022 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:25:44.022 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:53.527078+0000", 2024-09-13T11:25:44.022 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.9000000000000002e-06 2024-09-13T11:25:44.022 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.022 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.023 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:25:44.023 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:53.527088+0000", 2024-09-13T11:25:44.023 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.7240000000000004e-06 2024-09-13T11:25:44.023 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.023 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.023 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:25:44.023 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:53.706095+0000", 2024-09-13T11:25:44.023 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.17900633899999999 2024-09-13T11:25:44.023 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.023 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.023 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:25:44.023 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:53.706101+0000", 2024-09-13T11:25:44.023 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.7730000000000002e-06 2024-09-13T11:25:44.023 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.023 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.024 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:25:44.024 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:53.706168+0000", 2024-09-13T11:25:44.024 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.6225999999999999e-05 2024-09-13T11:25:44.024 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.024 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.024 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:25:44.024 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:53.706175+0000", 2024-09-13T11:25:44.024 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.5560000000000002e-06 2024-09-13T11:25:44.024 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.024 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.024 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:25:44.024 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:53.706303+0000", 2024-09-13T11:25:44.024 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000127763 2024-09-13T11:25:44.025 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:44.025 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:25:44.025 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:44.025 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.025 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.025 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:137780 3.8 3:1047f268:::benchmark_data_smithi177_31116_object137779:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e139)", 2024-09-13T11:25:44.025 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:24:53.795212+0000", 2024-09-13T11:25:44.025 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 50.138783633000003, 2024-09-13T11:25:44.025 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.21530981399999999, 2024-09-13T11:25:44.025 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:25:44.025 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:25:44.025 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:44.025 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:25:44.025 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:25:44.025 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:44.026 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 137780 2024-09-13T11:25:44.026 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.026 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:25:44.026 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.026 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:25:44.026 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:53.795212+0000", 2024-09-13T11:25:44.026 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:44.026 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.026 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.026 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:25:44.026 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:53.795212+0000", 2024-09-13T11:25:44.026 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:44.026 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.027 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.027 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:25:44.027 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:53.795214+0000", 2024-09-13T11:25:44.027 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.481e-06 2024-09-13T11:25:44.027 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.027 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.027 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:25:44.027 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:53.795247+0000", 2024-09-13T11:25:44.027 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.3105999999999998e-05 2024-09-13T11:25:44.027 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.027 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.027 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:25:44.027 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:53.795258+0000", 2024-09-13T11:25:44.028 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0872e-05 2024-09-13T11:25:44.028 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.028 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.028 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:25:44.028 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:53.795265+0000", 2024-09-13T11:25:44.028 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.5729999999999996e-06 2024-09-13T11:25:44.028 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.028 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.028 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:25:44.028 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:53.795309+0000", 2024-09-13T11:25:44.028 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.4617999999999999e-05 2024-09-13T11:25:44.028 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.028 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.028 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:25:44.029 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:53.795397+0000", 2024-09-13T11:25:44.029 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.7863999999999998e-05 2024-09-13T11:25:44.029 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.029 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.029 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 1", 2024-09-13T11:25:44.029 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:53.795493+0000", 2024-09-13T11:25:44.029 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.6062000000000003e-05 2024-09-13T11:25:44.029 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.029 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.029 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:25:44.029 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:53.797245+0000", 2024-09-13T11:25:44.029 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0017521559999999999 2024-09-13T11:25:44.029 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.029 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.030 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:25:44.030 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:54.010409+0000", 2024-09-13T11:25:44.030 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.21316385299999999 2024-09-13T11:25:44.030 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.030 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.030 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:25:44.030 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:54.010474+0000", 2024-09-13T11:25:44.030 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.4998000000000003e-05 2024-09-13T11:25:44.030 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.030 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.030 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:25:44.030 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:54.010521+0000", 2024-09-13T11:25:44.030 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.7231000000000002e-05 2024-09-13T11:25:44.030 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:44.031 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:25:44.031 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:44.031 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.031 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.031 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop_reply(client.4665.0:137780 3.8 e139/117 ondisk, result = 0)", 2024-09-13T11:25:44.031 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:24:53.797821+0000", 2024-09-13T11:25:44.031 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 50.136173735, 2024-09-13T11:25:44.031 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.21270765599999999, 2024-09-13T11:25:44.031 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:25:44.031 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:25:44.031 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:25:44.031 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:25:44.031 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.031 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:25:44.031 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:53.797821+0000", 2024-09-13T11:25:44.032 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:44.032 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.032 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.032 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:25:44.032 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:53.797821+0000", 2024-09-13T11:25:44.032 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:44.032 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.032 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.032 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:25:44.032 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:53.797822+0000", 2024-09-13T11:25:44.032 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.012e-06 2024-09-13T11:25:44.032 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.032 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.033 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:25:44.033 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:53.797829+0000", 2024-09-13T11:25:44.033 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.8009999999999998e-06 2024-09-13T11:25:44.033 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.033 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.033 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:25:44.033 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:54.010334+0000", 2024-09-13T11:25:44.033 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.212504477 2024-09-13T11:25:44.033 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.033 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.033 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:25:44.033 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:54.010342+0000", 2024-09-13T11:25:44.033 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.6549999999999993e-06 2024-09-13T11:25:44.033 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.034 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.034 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:25:44.034 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:54.010394+0000", 2024-09-13T11:25:44.034 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.1875000000000001e-05 2024-09-13T11:25:44.034 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.034 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.034 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:25:44.034 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:54.010402+0000", 2024-09-13T11:25:44.034 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.6499999999999996e-06 2024-09-13T11:25:44.034 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.034 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.034 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:25:44.034 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:54.010529+0000", 2024-09-13T11:25:44.034 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000127186 2024-09-13T11:25:44.035 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:44.035 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:25:44.035 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:44.035 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.035 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.035 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.4665.0:196211 3.6 e178/122 3:63573deb:::benchmark_data_smithi177_31116_object55908:head v 178'6255, mlcod=178'6252)", 2024-09-13T11:25:44.035 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:25:43.175295+0000", 2024-09-13T11:25:44.035 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 0.75870040500000002, 2024-09-13T11:25:44.035 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.167837029, 2024-09-13T11:25:44.035 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:25:44.035 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:25:44.035 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:44.035 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:25:44.035 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.036 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:25:44.036 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:43.175295+0000", 2024-09-13T11:25:44.036 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:44.036 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.036 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.036 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:25:44.036 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:43.175295+0000", 2024-09-13T11:25:44.036 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:44.036 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.036 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.036 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:25:44.036 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:43.175300+0000", 2024-09-13T11:25:44.036 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.9099999999999996e-06 2024-09-13T11:25:44.036 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.037 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.037 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:25:44.037 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:43.175318+0000", 2024-09-13T11:25:44.037 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.8346e-05 2024-09-13T11:25:44.037 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.037 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.037 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:25:44.037 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:43.340757+0000", 2024-09-13T11:25:44.037 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.165439058 2024-09-13T11:25:44.037 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.037 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.037 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:25:44.037 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:43.340775+0000", 2024-09-13T11:25:44.037 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.7546000000000001e-05 2024-09-13T11:25:44.038 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.038 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.038 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:25:44.038 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:43.340855+0000", 2024-09-13T11:25:44.038 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.0207e-05 2024-09-13T11:25:44.038 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.038 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.038 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:25:44.038 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:43.340879+0000", 2024-09-13T11:25:44.038 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.4494e-05 2024-09-13T11:25:44.038 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.038 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.038 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:25:44.038 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:43.343057+0000", 2024-09-13T11:25:44.039 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0021772380000000002 2024-09-13T11:25:44.039 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.039 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.039 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:25:44.039 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:43.343132+0000", 2024-09-13T11:25:44.039 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.5229999999999996e-05 2024-09-13T11:25:44.039 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:44.039 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:25:44.039 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:44.039 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.039 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.039 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.4665.0:196314 3.6 e178/122 3:62e79b1c:::benchmark_data_smithi177_31116_object56011:head v 178'6256, mlcod=178'6253)", 2024-09-13T11:25:44.039 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:25:43.340693+0000", 2024-09-13T11:25:44.039 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 0.59330215399999997, 2024-09-13T11:25:44.039 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.24109011699999999, 2024-09-13T11:25:44.040 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:25:44.040 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:25:44.040 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:44.040 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:25:44.040 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.040 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:25:44.040 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:43.340693+0000", 2024-09-13T11:25:44.040 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:44.040 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.040 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.040 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:25:44.040 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:43.340693+0000", 2024-09-13T11:25:44.040 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:44.041 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.041 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.041 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:25:44.041 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:43.340704+0000", 2024-09-13T11:25:44.041 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1049e-05 2024-09-13T11:25:44.041 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.041 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.041 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:25:44.041 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:43.340718+0000", 2024-09-13T11:25:44.041 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4214e-05 2024-09-13T11:25:44.041 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.041 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.041 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:25:44.042 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:43.579964+0000", 2024-09-13T11:25:44.042 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.23924590200000001 2024-09-13T11:25:44.042 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.042 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.042 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:25:44.042 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:43.579972+0000", 2024-09-13T11:25:44.042 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.2789999999999998e-06 2024-09-13T11:25:44.042 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.042 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.042 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:25:44.042 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:43.580043+0000", 2024-09-13T11:25:44.042 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.0856000000000001e-05 2024-09-13T11:25:44.042 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.042 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.042 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:25:44.043 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:43.580065+0000", 2024-09-13T11:25:44.043 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.1795999999999999e-05 2024-09-13T11:25:44.043 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.043 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.043 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:25:44.043 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:43.581741+0000", 2024-09-13T11:25:44.043 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0016759190000000001 2024-09-13T11:25:44.044 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.044 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.044 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:25:44.044 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:43.581783+0000", 2024-09-13T11:25:44.044 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.2101999999999998e-05 2024-09-13T11:25:44.044 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:44.044 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:25:44.045 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:44.045 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:44.045 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:25:44.045 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-13T11:25:44.046 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-13T11:25:44.048 INFO:tasks.ceph.osd.7.smithi177.stderr:2024-09-13T11:25:44.046+0000 7f897b51d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26575) UID: 0 2024-09-13T11:25:44.049 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-09-13T11:25:44.149 INFO:tasks.ceph.osd.7.smithi177.stderr:2024-09-13T11:25:44.146+0000 7f897b51d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26575) UID: 0 2024-09-13T11:25:44.184 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-13T11:25:44.184 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [ 2024-09-13T11:25:44.185 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.185 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:196982 3.17 3:ed71ee8b:::benchmark_data_smithi177_31116_object56679:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e178)", 2024-09-13T11:25:44.185 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:25:44.065734+0000", 2024-09-13T11:25:44.185 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 0.11872276399999999, 2024-09-13T11:25:44.185 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.11876062499999999, 2024-09-13T11:25:44.185 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:25:44.185 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:25:44.185 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "waiting for sub ops", 2024-09-13T11:25:44.185 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:25:44.185 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:25:44.185 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:44.185 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 196982 2024-09-13T11:25:44.185 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.186 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:25:44.186 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.186 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:25:44.186 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:44.065734+0000", 2024-09-13T11:25:44.186 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:44.186 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.186 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.186 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:25:44.186 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:44.065734+0000", 2024-09-13T11:25:44.186 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:44.186 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.186 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.186 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:25:44.186 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:44.065737+0000", 2024-09-13T11:25:44.187 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.4020000000000002e-06 2024-09-13T11:25:44.187 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.187 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.187 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:25:44.187 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:44.065746+0000", 2024-09-13T11:25:44.187 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.7909999999999993e-06 2024-09-13T11:25:44.187 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.187 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.187 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:25:44.187 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:44.065765+0000", 2024-09-13T11:25:44.187 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.893e-05 2024-09-13T11:25:44.187 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.187 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.187 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:25:44.188 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:44.065775+0000", 2024-09-13T11:25:44.188 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0000000000000001e-05 2024-09-13T11:25:44.188 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.188 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.188 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:25:44.188 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:44.065859+0000", 2024-09-13T11:25:44.188 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.4515e-05 2024-09-13T11:25:44.188 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.188 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.188 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:25:44.188 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:44.066060+0000", 2024-09-13T11:25:44.188 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000200385 2024-09-13T11:25:44.188 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.189 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.189 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 3", 2024-09-13T11:25:44.189 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:44.066228+0000", 2024-09-13T11:25:44.189 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00016846600000000001 2024-09-13T11:25:44.189 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.189 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.189 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:25:44.189 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:44.070493+0000", 2024-09-13T11:25:44.189 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0042646369999999999 2024-09-13T11:25:44.189 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:44.189 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:25:44.189 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:44.189 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.190 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.190 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:197056 3.3 3:c07087d0:::benchmark_data_smithi177_31116_object56753:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e178)", 2024-09-13T11:25:44.190 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:25:44.135772+0000", 2024-09-13T11:25:44.190 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 0.048684074000000001, 2024-09-13T11:25:44.190 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.048778780000000001, 2024-09-13T11:25:44.190 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:25:44.190 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:25:44.190 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "waiting for sub ops", 2024-09-13T11:25:44.190 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:25:44.190 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:25:44.190 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:44.191 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 197056 2024-09-13T11:25:44.191 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.191 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:25:44.191 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.191 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:25:44.191 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:44.135772+0000", 2024-09-13T11:25:44.191 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:44.191 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.191 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.191 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:25:44.191 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:44.135772+0000", 2024-09-13T11:25:44.191 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:44.191 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.191 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.191 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:25:44.192 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:44.135775+0000", 2024-09-13T11:25:44.192 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.2840000000000001e-06 2024-09-13T11:25:44.192 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.192 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.192 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:25:44.192 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:44.135781+0000", 2024-09-13T11:25:44.192 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.528e-06 2024-09-13T11:25:44.192 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.192 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.192 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:25:44.193 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:44.135798+0000", 2024-09-13T11:25:44.193 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.6946999999999999e-05 2024-09-13T11:25:44.193 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.193 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.193 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:25:44.193 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:44.135806+0000", 2024-09-13T11:25:44.194 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.8669999999999994e-06 2024-09-13T11:25:44.194 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.194 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.194 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:25:44.194 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:44.135953+0000", 2024-09-13T11:25:44.194 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00014687399999999999 2024-09-13T11:25:44.194 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.194 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.194 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:25:44.194 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:44.136311+0000", 2024-09-13T11:25:44.194 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00035766999999999998 2024-09-13T11:25:44.195 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.195 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.195 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 7", 2024-09-13T11:25:44.195 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:44.136615+0000", 2024-09-13T11:25:44.195 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00030461000000000003 2024-09-13T11:25:44.195 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.195 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.195 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:25:44.195 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:44.139759+0000", 2024-09-13T11:25:44.195 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0031439240000000002 2024-09-13T11:25:44.195 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:44.195 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:25:44.195 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:44.195 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:44.196 INFO:teuthology.orchestra.run.smithi162.stdout: ], 2024-09-13T11:25:44.196 INFO:teuthology.orchestra.run.smithi162.stdout: "num_ops": 2 2024-09-13T11:25:44.196 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-13T11:25:44.201 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-09-13T11:25:44.250 INFO:tasks.ceph.osd.4.smithi001.stderr:2024-09-13T11:25:44.247+0000 7ff8e0218640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26310) UID: 0 2024-09-13T11:25:44.329 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-13T11:25:44.329 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [], 2024-09-13T11:25:44.329 INFO:teuthology.orchestra.run.smithi162.stdout: "complaint_time": 30, 2024-09-13T11:25:44.329 INFO:teuthology.orchestra.run.smithi162.stdout: "num_blocked_ops": 0 2024-09-13T11:25:44.329 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-13T11:25:44.345 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-09-13T11:25:44.350 INFO:tasks.ceph.osd.5.smithi092.stderr:2024-09-13T11:25:44.349+0000 7f6f0b705640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26346) UID: 0 2024-09-13T11:25:44.354 INFO:teuthology.orchestra.run.smithi001.stderr:noscrub is unset 2024-09-13T11:25:44.356 INFO:tasks.ceph.osd.7.smithi177.stderr:2024-09-13T11:25:44.354+0000 7f8971f4e640 -1 osd.7 178 handle_osd_map injecting map crc failure 2024-09-13T11:25:44.409 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-13T11:25:44.410 INFO:teuthology.orchestra.run.smithi001.stderr:dumped all 2024-09-13T11:25:44.429 INFO:teuthology.orchestra.run.smithi001.stdout:{"pg_ready":true,"pg_map":{"version":272,"stamp":"2024-09-13T11:25:44.372329+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":5708481099,"num_objects":87101,"num_object_clones":0,"num_object_copies":174202,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":87101,"num_whiteouts":0,"num_read":87,"num_read_kb":74,"num_write":105276,"num_write_kb":3444563,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18805,"num_bytes_recovered":1232273427,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":303101,"ondisk_log_size":303101,"up":110,"acting":110,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":118,"num_osds":15,"num_per_pool_osds":15,"num_per_pool_omap_osds":15,"kb":1415577600,"kb_used":13259272,"kb_used_data":11843672,"kb_used_omap":26,"kb_used_meta":1415077,"kb_avail":1402318328,"statfs":{"total":1449551462400,"available":1435973967872,"internally_reserved":0,"allocated":12127920128,"data_stored":12116379620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":26763,"internal_metadata":1449039733},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,0,1,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-345833472,"num_objects":-5277,"num_object_clones":0,"num_object_copies":-10554,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-220,"num_objects_unfound":0,"num_objects_dirty":-5277,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":807,"num_write_kb":-192544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-3301,"num_bytes_recovered":-216334336,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.045366"},"pg_stats":[{"pgid":"3.2d","version":"176'6415","reported_seq":11929,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-09-13T11:25:41.096358+0000","last_change":"2024-09-13T11:25:40.139391+0000","last_active":"2024-09-13T11:25:41.096358+0000","last_peered":"2024-09-13T11:25:41.096358+0000","last_clean":"2024-09-13T11:25:41.096358+0000","last_became_active":"2024-09-13T11:25:39.334408+0000","last_became_peered":"2024-09-13T11:25:39.334408+0000","last_unstale":"2024-09-13T11:25:41.096358+0000","last_undegraded":"2024-09-13T11:25:41.096358+0000","last_fullsized":"2024-09-13T11:25:41.096358+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":6,"last_scrub":"76'3395","last_scrub_stamp":"2024-09-13T11:23:32.776747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:32.776747+0000","objects_scrubbed":0,"log_size":6415,"log_dups_size":0,"ondisk_log_size":6415,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"user requested scrub scheduled @ 2024-09-13T11:25:45.101522+0000","scrub_duration":179,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":93028352,"num_objects":1420,"num_object_clones":0,"num_object_copies":2840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1420,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2178,"num_write_kb":87840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1719,"num_bytes_recovered":112656384,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"176'6490","reported_seq":11842,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-09-13T11:25:40.892986+0000","last_change":"2024-09-13T11:24:59.456762+0000","last_active":"2024-09-13T11:25:40.892986+0000","last_peered":"2024-09-13T11:25:40.892986+0000","last_clean":"2024-09-13T11:25:40.892986+0000","last_became_active":"2024-09-13T11:24:59.021895+0000","last_became_peered":"2024-09-13T11:24:59.021895+0000","last_unstale":"2024-09-13T11:25:40.892986+0000","last_undegraded":"2024-09-13T11:25:40.892986+0000","last_fullsized":"2024-09-13T11:25:40.892986+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":6,"last_scrub":"145'5672","last_scrub_stamp":"2024-09-13T11:24:59.456653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:24:59.456653+0000","objects_scrubbed":2192,"log_size":6490,"log_dups_size":0,"ondisk_log_size":6490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:26:20.877991+0000","scrub_duration":279,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90046464,"num_objects":1374,"num_object_clones":0,"num_object_copies":2748,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1374,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":818,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,7],"acting":[10,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.2b","version":"176'6378","reported_seq":10087,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-09-13T11:25:40.338938+0000","last_change":"2024-09-13T11:24:45.271477+0000","last_active":"2024-09-13T11:25:40.338938+0000","last_peered":"2024-09-13T11:25:40.338938+0000","last_clean":"2024-09-13T11:25:40.338938+0000","last_became_active":"2024-09-13T11:24:26.180578+0000","last_became_peered":"2024-09-13T11:24:26.180578+0000","last_unstale":"2024-09-13T11:25:40.338938+0000","last_undegraded":"2024-09-13T11:25:40.338938+0000","last_fullsized":"2024-09-13T11:25:40.338938+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":6,"last_scrub":"131'5382","last_scrub_stamp":"2024-09-13T11:24:45.271376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:24:45.271376+0000","objects_scrubbed":1940,"log_size":6378,"log_dups_size":0,"ondisk_log_size":6378,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:59.210895+0000","scrub_duration":279,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":83623936,"num_objects":1276,"num_object_clones":0,"num_object_copies":2552,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1276,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":996,"num_write_kb":10624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2a","version":"176'6505","reported_seq":10401,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-09-13T11:25:41.017731+0000","last_change":"2024-09-13T11:24:46.625117+0000","last_active":"2024-09-13T11:25:41.017731+0000","last_peered":"2024-09-13T11:25:41.017731+0000","last_clean":"2024-09-13T11:25:41.017731+0000","last_became_active":"2024-09-13T11:24:45.320540+0000","last_became_peered":"2024-09-13T11:24:45.320540+0000","last_unstale":"2024-09-13T11:25:41.017731+0000","last_undegraded":"2024-09-13T11:25:41.017731+0000","last_fullsized":"2024-09-13T11:25:41.017731+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":6,"last_scrub":"133'5491","last_scrub_stamp":"2024-09-13T11:24:46.624970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:24:46.624970+0000","objects_scrubbed":2094,"log_size":6505,"log_dups_size":0,"ondisk_log_size":6505,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:26:04.813280+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":93585408,"num_objects":1428,"num_object_clones":0,"num_object_copies":2856,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1428,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1016,"num_write_kb":11200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,7],"acting":[10,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.29","version":"176'6160","reported_seq":11871,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-09-13T11:25:40.974307+0000","last_change":"2024-09-13T11:25:13.333062+0000","last_active":"2024-09-13T11:25:40.974307+0000","last_peered":"2024-09-13T11:25:40.974307+0000","last_clean":"2024-09-13T11:25:40.974307+0000","last_became_active":"2024-09-13T11:25:12.944028+0000","last_became_peered":"2024-09-13T11:25:12.944028+0000","last_unstale":"2024-09-13T11:25:40.974307+0000","last_undegraded":"2024-09-13T11:25:40.974307+0000","last_fullsized":"2024-09-13T11:25:40.974307+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":6,"last_scrub":"154'5579","last_scrub_stamp":"2024-09-13T11:25:13.332959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:13.332959+0000","objects_scrubbed":1937,"log_size":6160,"log_dups_size":0,"ondisk_log_size":6160,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:26:24.089228+0000","scrub_duration":243,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88735744,"num_objects":1354,"num_object_clones":0,"num_object_copies":2708,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1354,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":583,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.28","version":"176'6011","reported_seq":11450,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-09-13T11:25:40.948798+0000","last_change":"2024-09-13T11:25:36.143261+0000","last_active":"2024-09-13T11:25:40.948798+0000","last_peered":"2024-09-13T11:25:40.948798+0000","last_clean":"2024-09-13T11:25:40.948798+0000","last_became_active":"2024-09-13T11:25:34.278640+0000","last_became_peered":"2024-09-13T11:25:34.278640+0000","last_unstale":"2024-09-13T11:25:40.948798+0000","last_undegraded":"2024-09-13T11:25:40.948798+0000","last_fullsized":"2024-09-13T11:25:40.948798+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":6,"last_scrub":"174'5936","last_scrub_stamp":"2024-09-13T11:25:36.143113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:36.143113+0000","objects_scrubbed":1447,"log_size":6011,"log_dups_size":0,"ondisk_log_size":6011,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:27:02.442082+0000","scrub_duration":1147,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88670208,"num_objects":1353,"num_object_clones":0,"num_object_copies":2706,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1353,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":94,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5],"acting":[8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.27","version":"176'6077","reported_seq":11634,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-09-13T11:25:40.864270+0000","last_change":"2024-09-13T11:25:37.344464+0000","last_active":"2024-09-13T11:25:40.864270+0000","last_peered":"2024-09-13T11:25:40.864270+0000","last_clean":"2024-09-13T11:25:40.864270+0000","last_became_active":"2024-09-13T11:25:30.257911+0000","last_became_peered":"2024-09-13T11:25:30.257911+0000","last_unstale":"2024-09-13T11:25:40.864270+0000","last_undegraded":"2024-09-13T11:25:40.864270+0000","last_fullsized":"2024-09-13T11:25:40.864270+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":6,"last_scrub":"174'6026","last_scrub_stamp":"2024-09-13T11:25:37.344335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:37.344335+0000","objects_scrubbed":1463,"log_size":6077,"log_dups_size":0,"ondisk_log_size":6077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:26:49.139373+0000","scrub_duration":195,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92471296,"num_objects":1411,"num_object_clones":0,"num_object_copies":2822,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1411,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":52,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,11],"acting":[5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.26","version":"176'6222","reported_seq":11812,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-09-13T11:25:41.079371+0000","last_change":"2024-09-13T11:25:01.697082+0000","last_active":"2024-09-13T11:25:41.079371+0000","last_peered":"2024-09-13T11:25:41.079371+0000","last_clean":"2024-09-13T11:25:41.079371+0000","last_became_active":"2024-09-13T11:24:54.976919+0000","last_became_peered":"2024-09-13T11:24:54.976919+0000","last_unstale":"2024-09-13T11:25:41.079371+0000","last_undegraded":"2024-09-13T11:25:41.079371+0000","last_fullsized":"2024-09-13T11:25:41.079371+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":6,"last_scrub":"147'5433","last_scrub_stamp":"2024-09-13T11:25:01.696886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:01.696886+0000","objects_scrubbed":2158,"log_size":6222,"log_dups_size":0,"ondisk_log_size":6222,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:26:22.070621+0000","scrub_duration":403,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":89194496,"num_objects":1361,"num_object_clones":0,"num_object_copies":2722,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1361,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":797,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,12],"acting":[8,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.25","version":"176'6048","reported_seq":11798,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-09-13T11:25:41.084847+0000","last_change":"2024-09-13T11:24:57.073162+0000","last_active":"2024-09-13T11:25:41.084847+0000","last_peered":"2024-09-13T11:25:41.084847+0000","last_clean":"2024-09-13T11:25:41.084847+0000","last_became_active":"2024-09-13T11:24:55.999019+0000","last_became_peered":"2024-09-13T11:24:55.999019+0000","last_unstale":"2024-09-13T11:25:41.084847+0000","last_undegraded":"2024-09-13T11:25:41.084847+0000","last_fullsized":"2024-09-13T11:25:41.084847+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":6,"last_scrub":"142'5214","last_scrub_stamp":"2024-09-13T11:24:57.073061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:24:57.073061+0000","objects_scrubbed":2249,"log_size":6048,"log_dups_size":0,"ondisk_log_size":6048,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:26:12.348519+0000","scrub_duration":571,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92930048,"num_objects":1418,"num_object_clones":0,"num_object_copies":2836,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1418,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":837,"num_write_kb":192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,2],"acting":[12,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[]},{"pgid":"3.24","version":"178'6109","reported_seq":10049,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-09-13T11:25:41.970866+0000","last_change":"2024-09-13T11:24:48.048846+0000","last_active":"2024-09-13T11:25:41.970866+0000","last_peered":"2024-09-13T11:25:41.970866+0000","last_clean":"2024-09-13T11:25:41.970866+0000","last_became_active":"2024-09-13T11:24:46.936994+0000","last_became_peered":"2024-09-13T11:24:46.936994+0000","last_unstale":"2024-09-13T11:25:41.970866+0000","last_undegraded":"2024-09-13T11:25:41.970866+0000","last_fullsized":"2024-09-13T11:25:41.970866+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":6,"last_scrub":"135'5093","last_scrub_stamp":"2024-09-13T11:24:48.048764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:24:48.048764+0000","objects_scrubbed":2102,"log_size":6109,"log_dups_size":0,"ondisk_log_size":6109,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:53.485370+0000","scrub_duration":287,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90112000,"num_objects":1375,"num_object_clones":0,"num_object_copies":2750,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1375,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1017,"num_write_kb":9280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1],"acting":[9,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.23","version":"176'5332","reported_seq":8210,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-09-13T11:25:40.967157+0000","last_change":"2024-09-13T11:25:38.462609+0000","last_active":"2024-09-13T11:25:40.967157+0000","last_peered":"2024-09-13T11:25:40.967157+0000","last_clean":"2024-09-13T11:25:40.967157+0000","last_became_active":"2024-09-13T11:23:47.926677+0000","last_became_peered":"2024-09-13T11:23:47.926677+0000","last_unstale":"2024-09-13T11:25:40.967157+0000","last_undegraded":"2024-09-13T11:25:40.967157+0000","last_fullsized":"2024-09-13T11:25:40.967157+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":6,"last_scrub":"91'3278","last_scrub_stamp":"2024-09-13T11:23:48.133084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:48.133084+0000","objects_scrubbed":349,"log_size":5332,"log_dups_size":0,"ondisk_log_size":5332,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:43.448448+0000","scrub_duration":139,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92667904,"num_objects":1414,"num_object_clones":0,"num_object_copies":2828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1414,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2055,"num_write_kb":78080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9],"acting":[5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.22","version":"176'5240","reported_seq":10841,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-09-13T11:25:41.046364+0000","last_change":"2024-09-13T11:24:45.485711+0000","last_active":"2024-09-13T11:25:41.046364+0000","last_peered":"2024-09-13T11:25:41.046364+0000","last_clean":"2024-09-13T11:25:41.046364+0000","last_became_active":"2024-09-13T11:24:29.253848+0000","last_became_peered":"2024-09-13T11:24:29.253848+0000","last_unstale":"2024-09-13T11:25:41.046364+0000","last_undegraded":"2024-09-13T11:25:41.046364+0000","last_fullsized":"2024-09-13T11:25:41.046364+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":6,"last_scrub":"90'3246","last_scrub_stamp":"2024-09-13T11:23:47.300339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:47.300339+0000","objects_scrubbed":1033,"log_size":5240,"log_dups_size":0,"ondisk_log_size":5240,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":203,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":91947008,"num_objects":1403,"num_object_clones":0,"num_object_copies":2806,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1403,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1996,"num_write_kb":75712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1768,"num_bytes_recovered":115867648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.21","version":"176'5161","reported_seq":8047,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-09-13T11:25:41.069159+0000","last_change":"2024-09-13T11:24:12.564117+0000","last_active":"2024-09-13T11:25:41.069159+0000","last_peered":"2024-09-13T11:25:41.069159+0000","last_clean":"2024-09-13T11:25:41.069159+0000","last_became_active":"2024-09-13T11:24:01.250308+0000","last_became_peered":"2024-09-13T11:24:01.250308+0000","last_unstale":"2024-09-13T11:25:41.069159+0000","last_undegraded":"2024-09-13T11:25:41.069159+0000","last_fullsized":"2024-09-13T11:25:41.069159+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":6,"last_scrub":"112'3670","last_scrub_stamp":"2024-09-13T11:24:12.564031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:24:12.564031+0000","objects_scrubbed":1431,"log_size":5161,"log_dups_size":0,"ondisk_log_size":5161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":183,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":86441984,"num_objects":1319,"num_object_clones":0,"num_object_copies":2638,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1319,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1494,"num_write_kb":44224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8],"acting":[0,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.20","version":"176'5315","reported_seq":7511,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-09-13T11:25:41.080869+0000","last_change":"2024-09-13T11:25:35.773483+0000","last_active":"2024-09-13T11:25:41.080869+0000","last_peered":"2024-09-13T11:25:41.080869+0000","last_clean":"2024-09-13T11:25:41.080869+0000","last_became_active":"2024-09-13T11:23:33.954946+0000","last_became_peered":"2024-09-13T11:23:33.954946+0000","last_unstale":"2024-09-13T11:25:41.080869+0000","last_undegraded":"2024-09-13T11:25:41.080869+0000","last_fullsized":"2024-09-13T11:25:41.080869+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":6,"last_scrub":"174'5240","last_scrub_stamp":"2024-09-13T11:25:35.773409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:35.773409+0000","objects_scrubbed":1424,"log_size":5315,"log_dups_size":0,"ondisk_log_size":5315,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:26:53.605828+0000","scrub_duration":215,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88276992,"num_objects":1347,"num_object_clones":0,"num_object_copies":2694,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1347,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2050,"num_write_kb":74752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,0],"acting":[12,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[]},{"pgid":"3.1f","version":"176'7607","reported_seq":19577,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-09-13T11:25:41.073573+0000","last_change":"2024-09-13T11:25:20.773752+0000","last_active":"2024-09-13T11:25:41.073573+0000","last_peered":"2024-09-13T11:25:41.073573+0000","last_clean":"2024-09-13T11:25:41.073573+0000","last_became_active":"2024-09-13T11:24:29.290326+0000","last_became_peered":"2024-09-13T11:24:29.290326+0000","last_unstale":"2024-09-13T11:25:41.073573+0000","last_undegraded":"2024-09-13T11:25:41.073573+0000","last_fullsized":"2024-09-13T11:25:41.073573+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":5,"last_scrub":"161'6865","last_scrub_stamp":"2024-09-13T11:25:20.773641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:20.773641+0000","objects_scrubbed":3425,"log_size":7607,"log_dups_size":0,"ondisk_log_size":7607,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:26:21.317280+0000","scrub_duration":535,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":175505408,"num_objects":2678,"num_object_clones":0,"num_object_copies":5356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2678,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3187,"num_write_kb":102848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3306,"num_bytes_recovered":216662016,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5],"acting":[10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1e","version":"176'7886","reported_seq":18422,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-09-13T11:25:41.000936+0000","last_change":"2024-09-13T11:24:08.338485+0000","last_active":"2024-09-13T11:25:41.000936+0000","last_peered":"2024-09-13T11:25:41.000936+0000","last_clean":"2024-09-13T11:25:41.000936+0000","last_became_active":"2024-09-13T11:24:01.261448+0000","last_became_peered":"2024-09-13T11:24:01.261448+0000","last_unstale":"2024-09-13T11:25:41.000936+0000","last_undegraded":"2024-09-13T11:25:41.000936+0000","last_fullsized":"2024-09-13T11:25:41.000936+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":5,"last_scrub":"110'4537","last_scrub_stamp":"2024-09-13T11:24:08.338354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:24:08.338354+0000","objects_scrubbed":2760,"log_size":7886,"log_dups_size":0,"ondisk_log_size":7886,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":367,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":178388992,"num_objects":2722,"num_object_clones":0,"num_object_copies":5444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2722,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3358,"num_write_kb":106240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"176'7690","reported_seq":13156,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-09-13T11:25:41.009773+0000","last_change":"2024-09-13T11:24:12.492592+0000","last_active":"2024-09-13T11:25:41.009773+0000","last_peered":"2024-09-13T11:25:41.009773+0000","last_clean":"2024-09-13T11:25:41.009773+0000","last_became_active":"2024-09-13T11:24:01.258225+0000","last_became_peered":"2024-09-13T11:24:01.258225+0000","last_unstale":"2024-09-13T11:25:41.009773+0000","last_undegraded":"2024-09-13T11:25:41.009773+0000","last_fullsized":"2024-09-13T11:25:41.009773+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":5,"last_scrub":"112'4579","last_scrub_stamp":"2024-09-13T11:24:12.492507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:24:12.492507+0000","objects_scrubbed":2875,"log_size":7690,"log_dups_size":0,"ondisk_log_size":7690,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:44.732421+0000","scrub_duration":375,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":177471512,"num_objects":2709,"num_object_clones":0,"num_object_copies":5418,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2709,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":3124,"num_write_kb":94593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,15],"acting":[2,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1c","version":"178'7857","reported_seq":11684,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-09-13T11:25:42.080785+0000","last_change":"2024-09-13T11:24:00.529312+0000","last_active":"2024-09-13T11:25:42.080785+0000","last_peered":"2024-09-13T11:25:42.080785+0000","last_clean":"2024-09-13T11:25:42.080785+0000","last_became_active":"2024-09-13T11:23:33.989492+0000","last_became_peered":"2024-09-13T11:23:33.989492+0000","last_unstale":"2024-09-13T11:25:42.080785+0000","last_undegraded":"2024-09-13T11:25:42.080785+0000","last_fullsized":"2024-09-13T11:25:42.080785+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":5,"last_scrub":"102'4277","last_scrub_stamp":"2024-09-13T11:24:00.529243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:24:00.529243+0000","objects_scrubbed":2544,"log_size":7857,"log_dups_size":0,"ondisk_log_size":7857,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:44.044520+0000","scrub_duration":259,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":177078272,"num_objects":2702,"num_object_clones":0,"num_object_copies":5404,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2702,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3582,"num_write_kb":119680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7],"acting":[11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1b","version":"176'7816","reported_seq":13461,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-09-13T11:25:41.073751+0000","last_change":"2024-09-13T11:24:12.392093+0000","last_active":"2024-09-13T11:25:41.073751+0000","last_peered":"2024-09-13T11:25:41.073751+0000","last_clean":"2024-09-13T11:25:41.073751+0000","last_became_active":"2024-09-13T11:24:04.950821+0000","last_became_peered":"2024-09-13T11:24:04.950821+0000","last_unstale":"2024-09-13T11:25:41.073751+0000","last_undegraded":"2024-09-13T11:25:41.073751+0000","last_fullsized":"2024-09-13T11:25:41.073751+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":5,"last_scrub":"112'4665","last_scrub_stamp":"2024-09-13T11:24:12.391890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:24:12.391890+0000","objects_scrubbed":2934,"log_size":7816,"log_dups_size":0,"ondisk_log_size":7816,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:41.846147+0000","scrub_duration":675,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":178257920,"num_objects":2720,"num_object_clones":0,"num_object_copies":5440,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2720,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3172,"num_write_kb":94656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,15],"acting":[12,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[]},{"pgid":"3.1a","version":"176'7649","reported_seq":12955,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-09-13T11:25:41.042532+0000","last_change":"2024-09-13T11:24:09.556174+0000","last_active":"2024-09-13T11:25:41.042532+0000","last_peered":"2024-09-13T11:25:41.042532+0000","last_clean":"2024-09-13T11:25:41.042532+0000","last_became_active":"2024-09-13T11:24:03.935185+0000","last_became_peered":"2024-09-13T11:24:03.935185+0000","last_unstale":"2024-09-13T11:25:41.042532+0000","last_undegraded":"2024-09-13T11:25:41.042532+0000","last_fullsized":"2024-09-13T11:25:41.042532+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":5,"last_scrub":"110'4433","last_scrub_stamp":"2024-09-13T11:24:09.556022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:24:09.556022+0000","objects_scrubbed":2654,"log_size":7649,"log_dups_size":0,"ondisk_log_size":7649,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:44.397973+0000","scrub_duration":387,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":176357376,"num_objects":2691,"num_object_clones":0,"num_object_copies":5382,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2691,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3227,"num_write_kb":104448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9],"acting":[4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.19","version":"178'7282","reported_seq":11776,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-09-13T11:25:42.006712+0000","last_change":"2024-09-13T11:25:10.784789+0000","last_active":"2024-09-13T11:25:42.006712+0000","last_peered":"2024-09-13T11:25:42.006712+0000","last_clean":"2024-09-13T11:25:42.006712+0000","last_became_active":"2024-09-13T11:23:41.924941+0000","last_became_peered":"2024-09-13T11:23:41.924941+0000","last_unstale":"2024-09-13T11:25:42.006712+0000","last_undegraded":"2024-09-13T11:25:42.006712+0000","last_fullsized":"2024-09-13T11:25:42.006712+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":5,"last_scrub":"151'6156","last_scrub_stamp":"2024-09-13T11:25:10.784716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:10.784716+0000","objects_scrubbed":3781,"log_size":7282,"log_dups_size":0,"ondisk_log_size":7282,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:26:30.390310+0000","scrub_duration":1131,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":172818432,"num_objects":2637,"num_object_clones":0,"num_object_copies":5274,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2637,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3880,"num_write_kb":145216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1],"acting":[9,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.18","version":"176'7401","reported_seq":12192,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-09-13T11:25:41.085040+0000","last_change":"2024-09-13T11:25:37.928304+0000","last_active":"2024-09-13T11:25:41.085040+0000","last_peered":"2024-09-13T11:25:41.085040+0000","last_clean":"2024-09-13T11:25:41.085040+0000","last_became_active":"2024-09-13T11:23:48.926014+0000","last_became_peered":"2024-09-13T11:23:48.926014+0000","last_unstale":"2024-09-13T11:25:41.085040+0000","last_undegraded":"2024-09-13T11:25:41.085040+0000","last_fullsized":"2024-09-13T11:25:41.085040+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":5,"last_scrub":"174'7306","last_scrub_stamp":"2024-09-13T11:25:37.928052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:37.928052+0000","objects_scrubbed":2815,"log_size":7401,"log_dups_size":0,"ondisk_log_size":7401,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:26:56.053559+0000","scrub_duration":631,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":177340416,"num_objects":2706,"num_object_clones":0,"num_object_copies":5412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2706,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3034,"num_write_kb":86528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0],"acting":[10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.17","version":"176'7261","reported_seq":11487,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-09-13T11:25:41.056573+0000","last_change":"2024-09-13T11:25:12.658562+0000","last_active":"2024-09-13T11:25:41.056573+0000","last_peered":"2024-09-13T11:25:41.056573+0000","last_clean":"2024-09-13T11:25:41.056573+0000","last_became_active":"2024-09-13T11:23:42.935601+0000","last_became_peered":"2024-09-13T11:23:42.935601+0000","last_unstale":"2024-09-13T11:25:41.056573+0000","last_undegraded":"2024-09-13T11:25:41.056573+0000","last_fullsized":"2024-09-13T11:25:41.056573+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":5,"last_scrub":"153'6197","last_scrub_stamp":"2024-09-13T11:25:12.658464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:12.658464+0000","objects_scrubbed":3743,"log_size":7261,"log_dups_size":0,"ondisk_log_size":7261,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:26:25.625277+0000","scrub_duration":467,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":175505408,"num_objects":2678,"num_object_clones":0,"num_object_copies":5356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2678,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3910,"num_write_kb":146688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3],"acting":[10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"178'7648","reported_seq":12514,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-09-13T11:25:42.030599+0000","last_change":"2024-09-13T11:25:01.903487+0000","last_active":"2024-09-13T11:25:42.030599+0000","last_peered":"2024-09-13T11:25:42.030599+0000","last_clean":"2024-09-13T11:25:42.030599+0000","last_became_active":"2024-09-13T11:23:46.105237+0000","last_became_peered":"2024-09-13T11:23:46.105237+0000","last_unstale":"2024-09-13T11:25:42.030599+0000","last_undegraded":"2024-09-13T11:25:42.030599+0000","last_fullsized":"2024-09-13T11:25:42.030599+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":5,"last_scrub":"147'6035","last_scrub_stamp":"2024-09-13T11:25:01.903395+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:01.903395+0000","objects_scrubbed":4359,"log_size":7648,"log_dups_size":0,"ondisk_log_size":7648,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:26:16.436377+0000","scrub_duration":743,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":178978816,"num_objects":2731,"num_object_clones":0,"num_object_copies":5462,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2731,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4173,"num_write_kb":155200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,15],"acting":[11,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.15","version":"176'7269","reported_seq":18613,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-09-13T11:25:41.083915+0000","last_change":"2024-09-13T11:25:12.198448+0000","last_active":"2024-09-13T11:25:41.083915+0000","last_peered":"2024-09-13T11:25:41.083915+0000","last_clean":"2024-09-13T11:25:41.083915+0000","last_became_active":"2024-09-13T11:24:29.226615+0000","last_became_peered":"2024-09-13T11:24:29.226615+0000","last_unstale":"2024-09-13T11:25:41.083915+0000","last_undegraded":"2024-09-13T11:25:41.083915+0000","last_fullsized":"2024-09-13T11:25:41.083915+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":5,"last_scrub":"91'3384","last_scrub_stamp":"2024-09-13T11:23:48.332600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:48.332600+0000","objects_scrubbed":2032,"log_size":7269,"log_dups_size":0,"ondisk_log_size":7269,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":235,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":174063616,"num_objects":2656,"num_object_clones":0,"num_object_copies":5312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2656,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3886,"num_write_kb":144320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3385,"num_bytes_recovered":221839360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.14","version":"176'7360","reported_seq":18668,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-09-13T11:25:40.066572+0000","last_change":"2024-09-13T11:25:38.085809+0000","last_active":"2024-09-13T11:25:40.066572+0000","last_peered":"2024-09-13T11:25:40.066572+0000","last_clean":"2024-09-13T11:25:40.066572+0000","last_became_active":"2024-09-13T11:24:29.253106+0000","last_became_peered":"2024-09-13T11:24:29.253106+0000","last_unstale":"2024-09-13T11:25:40.066572+0000","last_undegraded":"2024-09-13T11:25:40.066572+0000","last_fullsized":"2024-09-13T11:25:40.066572+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":5,"last_scrub":"174'7321","last_scrub_stamp":"2024-09-13T11:25:38.085667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:38.085667+0000","objects_scrubbed":2938,"log_size":7360,"log_dups_size":0,"ondisk_log_size":7360,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:26:55.536565+0000","scrub_duration":431,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":189530112,"num_objects":2892,"num_object_clones":0,"num_object_copies":5784,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2892,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3999,"num_write_kb":158592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3437,"num_bytes_recovered":225247232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,15],"acting":[3,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":324,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-09-13T11:25:39.347308+0000","last_change":"2024-09-13T11:24:23.212494+0000","last_active":"2024-09-13T11:25:39.347308+0000","last_peered":"2024-09-13T11:25:39.347308+0000","last_clean":"2024-09-13T11:25:39.347308+0000","last_became_active":"2024-09-13T11:22:14.192897+0000","last_became_peered":"2024-09-13T11:22:14.192897+0000","last_unstale":"2024-09-13T11:25:39.347308+0000","last_undegraded":"2024-09-13T11:25:39.347308+0000","last_fullsized":"2024-09-13T11:25:39.347308+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:24:23.212406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:24:23.212406+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00084655899999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,10],"acting":[15,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"3.6","version":"178'6228","reported_seq":10115,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-09-13T11:25:41.965662+0000","last_change":"2024-09-13T11:24:49.416851+0000","last_active":"2024-09-13T11:25:41.965662+0000","last_peered":"2024-09-13T11:25:41.965662+0000","last_clean":"2024-09-13T11:25:41.965662+0000","last_became_active":"2024-09-13T11:24:29.322558+0000","last_became_peered":"2024-09-13T11:24:29.322558+0000","last_unstale":"2024-09-13T11:25:41.965662+0000","last_undegraded":"2024-09-13T11:25:41.965662+0000","last_fullsized":"2024-09-13T11:25:41.965662+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"136'5216","last_scrub_stamp":"2024-09-13T11:24:49.416706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:24:49.416706+0000","objects_scrubbed":2083,"log_size":6228,"log_dups_size":0,"ondisk_log_size":6228,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:26:09.525086+0000","scrub_duration":247,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":89391104,"num_objects":1364,"num_object_clones":0,"num_object_copies":2728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1364,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1013,"num_write_kb":9408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,6],"acting":[13,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":335,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-09-13T11:25:41.952807+0000","last_change":"2024-09-13T11:24:59.556253+0000","last_active":"2024-09-13T11:25:41.952807+0000","last_peered":"2024-09-13T11:25:41.952807+0000","last_clean":"2024-09-13T11:25:41.952807+0000","last_became_active":"2024-09-13T11:22:14.211923+0000","last_became_peered":"2024-09-13T11:22:14.211923+0000","last_unstale":"2024-09-13T11:25:41.952807+0000","last_undegraded":"2024-09-13T11:25:41.952807+0000","last_fullsized":"2024-09-13T11:25:41.952807+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:24:59.556180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:24:59.556180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:26:06.229322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011689529999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0],"acting":[9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"178'6051","reported_seq":6432,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-09-13T11:25:42.071673+0000","last_change":"2024-09-13T11:24:33.336210+0000","last_active":"2024-09-13T11:25:42.071673+0000","last_peered":"2024-09-13T11:25:42.071673+0000","last_clean":"2024-09-13T11:25:42.071673+0000","last_became_active":"2024-09-13T11:24:24.092975+0000","last_became_peered":"2024-09-13T11:24:24.092975+0000","last_unstale":"2024-09-13T11:25:42.071673+0000","last_undegraded":"2024-09-13T11:25:42.071673+0000","last_fullsized":"2024-09-13T11:25:42.071673+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"125'4820","last_scrub_stamp":"2024-09-13T11:24:33.336091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:24:33.336091+0000","objects_scrubbed":1766,"log_size":6051,"log_dups_size":0,"ondisk_log_size":6051,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:54.538352+0000","scrub_duration":263,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88932352,"num_objects":1357,"num_object_clones":0,"num_object_copies":2714,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1357,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1233,"num_write_kb":26368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,8],"acting":[13,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":332,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-09-13T11:25:39.326833+0000","last_change":"2024-09-13T11:25:33.957283+0000","last_active":"2024-09-13T11:25:39.326833+0000","last_peered":"2024-09-13T11:25:39.326833+0000","last_clean":"2024-09-13T11:25:39.326833+0000","last_became_active":"2024-09-13T11:22:14.581458+0000","last_became_peered":"2024-09-13T11:22:14.581458+0000","last_unstale":"2024-09-13T11:25:39.326833+0000","last_undegraded":"2024-09-13T11:25:39.326833+0000","last_fullsized":"2024-09-13T11:25:39.326833+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:25:33.957179+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:25:33.957179+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:26:52.126360+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.001026047,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5],"acting":[8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"176'6070","reported_seq":6419,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-09-13T11:25:40.370851+0000","last_change":"2024-09-13T11:24:24.579036+0000","last_active":"2024-09-13T11:25:40.370851+0000","last_peered":"2024-09-13T11:25:40.370851+0000","last_clean":"2024-09-13T11:25:40.370851+0000","last_became_active":"2024-09-13T11:24:24.083077+0000","last_became_peered":"2024-09-13T11:24:24.083077+0000","last_unstale":"2024-09-13T11:25:40.370851+0000","last_undegraded":"2024-09-13T11:25:40.370851+0000","last_fullsized":"2024-09-13T11:25:40.370851+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'4737","last_scrub_stamp":"2024-09-13T11:24:24.578900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:24:24.578900+0000","objects_scrubbed":1690,"log_size":6070,"log_dups_size":0,"ondisk_log_size":6070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":407,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90177536,"num_objects":1376,"num_object_clones":0,"num_object_copies":2752,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1376,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1336,"num_write_kb":32704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,12],"acting":[1,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":312,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-09-13T11:25:39.354101+0000","last_change":"2024-09-13T11:24:08.271026+0000","last_active":"2024-09-13T11:25:39.354101+0000","last_peered":"2024-09-13T11:25:39.354101+0000","last_clean":"2024-09-13T11:25:39.354101+0000","last_became_active":"2024-09-13T11:22:47.719636+0000","last_became_peered":"2024-09-13T11:22:47.719636+0000","last_unstale":"2024-09-13T11:25:39.354101+0000","last_undegraded":"2024-09-13T11:25:39.354101+0000","last_fullsized":"2024-09-13T11:25:39.354101+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:24:08.270972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:24:08.270972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00087476000000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,9],"acting":[1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"176'6032","reported_seq":6399,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-09-13T11:25:41.056263+0000","last_change":"2024-09-13T11:24:24.616937+0000","last_active":"2024-09-13T11:25:41.056263+0000","last_peered":"2024-09-13T11:25:41.056263+0000","last_clean":"2024-09-13T11:25:41.056263+0000","last_became_active":"2024-09-13T11:24:24.093492+0000","last_became_peered":"2024-09-13T11:24:24.093492+0000","last_unstale":"2024-09-13T11:25:41.056263+0000","last_undegraded":"2024-09-13T11:25:41.056263+0000","last_fullsized":"2024-09-13T11:25:41.056263+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'4744","last_scrub_stamp":"2024-09-13T11:24:24.616633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:24:24.616633+0000","objects_scrubbed":1624,"log_size":6032,"log_dups_size":0,"ondisk_log_size":6032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:42.911074+0000","scrub_duration":415,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":87883776,"num_objects":1341,"num_object_clones":0,"num_object_copies":2682,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1341,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1295,"num_write_kb":32384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,13],"acting":[15,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"3.3","version":"176'5241","reported_seq":5629,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-09-13T11:25:41.079715+0000","last_change":"2024-09-13T11:25:08.394545+0000","last_active":"2024-09-13T11:25:41.079715+0000","last_peered":"2024-09-13T11:25:41.079715+0000","last_clean":"2024-09-13T11:25:41.079715+0000","last_became_active":"2024-09-13T11:23:31.930448+0000","last_became_peered":"2024-09-13T11:23:31.930448+0000","last_unstale":"2024-09-13T11:25:41.079715+0000","last_undegraded":"2024-09-13T11:25:41.079715+0000","last_fullsized":"2024-09-13T11:25:41.079715+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"151'4587","last_scrub_stamp":"2024-09-13T11:25:08.394441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:08.394441+0000","objects_scrubbed":1942,"log_size":5241,"log_dups_size":0,"ondisk_log_size":5241,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:26:27.641900+0000","scrub_duration":219,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":84213760,"num_objects":1285,"num_object_clones":0,"num_object_copies":2570,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1285,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2119,"num_write_kb":80896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,7],"acting":[10,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"28'2","reported_seq":262,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-09-13T11:25:39.345241+0000","last_change":"2024-09-13T11:25:13.869564+0000","last_active":"2024-09-13T11:25:39.345241+0000","last_peered":"2024-09-13T11:25:39.345241+0000","last_clean":"2024-09-13T11:25:39.345241+0000","last_became_active":"2024-09-13T11:23:12.546348+0000","last_became_peered":"2024-09-13T11:23:12.546348+0000","last_unstale":"2024-09-13T11:25:39.345241+0000","last_undegraded":"2024-09-13T11:25:39.345241+0000","last_fullsized":"2024-09-13T11:25:39.345241+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2","last_scrub_stamp":"2024-09-13T11:25:13.869413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:25:13.869413+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:26:38.761718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0025826730000000002,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8],"acting":[2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"176'5253","reported_seq":5639,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-09-13T11:25:41.091261+0000","last_change":"2024-09-13T11:25:09.842875+0000","last_active":"2024-09-13T11:25:41.091261+0000","last_peered":"2024-09-13T11:25:41.091261+0000","last_clean":"2024-09-13T11:25:41.091261+0000","last_became_active":"2024-09-13T11:23:31.921388+0000","last_became_peered":"2024-09-13T11:23:31.921388+0000","last_unstale":"2024-09-13T11:25:41.091261+0000","last_undegraded":"2024-09-13T11:25:41.091261+0000","last_fullsized":"2024-09-13T11:25:41.091261+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"151'4644","last_scrub_stamp":"2024-09-13T11:25:09.842728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:09.842728+0000","objects_scrubbed":1944,"log_size":5253,"log_dups_size":0,"ondisk_log_size":5253,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:26:25.553642+0000","scrub_duration":395,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":87490560,"num_objects":1335,"num_object_clones":0,"num_object_copies":2670,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1335,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2147,"num_write_kb":84032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,15],"acting":[12,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":173,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-09-13T11:25:39.353655+0000","last_change":"2024-09-13T11:25:34.822952+0000","last_active":"2024-09-13T11:25:39.353655+0000","last_peered":"2024-09-13T11:25:39.353655+0000","last_clean":"2024-09-13T11:25:39.353655+0000","last_became_active":"2024-09-13T11:24:29.096575+0000","last_became_peered":"2024-09-13T11:24:29.096575+0000","last_unstale":"2024-09-13T11:25:39.353655+0000","last_undegraded":"2024-09-13T11:25:39.353655+0000","last_fullsized":"2024-09-13T11:25:39.353655+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:25:34.822831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:25:34.822831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:27:01.304133+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00081225399999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"178'5232","reported_seq":5624,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-09-13T11:25:41.988732+0000","last_change":"2024-09-13T11:25:00.861990+0000","last_active":"2024-09-13T11:25:41.988732+0000","last_peered":"2024-09-13T11:25:41.988732+0000","last_clean":"2024-09-13T11:25:41.988732+0000","last_became_active":"2024-09-13T11:23:31.926528+0000","last_became_peered":"2024-09-13T11:23:31.926528+0000","last_unstale":"2024-09-13T11:25:41.988732+0000","last_undegraded":"2024-09-13T11:25:41.988732+0000","last_fullsized":"2024-09-13T11:25:41.988732+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"146'4447","last_scrub_stamp":"2024-09-13T11:25:00.861834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:00.861834+0000","objects_scrubbed":2126,"log_size":5232,"log_dups_size":0,"ondisk_log_size":5232,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:26:15.411903+0000","scrub_duration":787,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":87687168,"num_objects":1338,"num_object_clones":0,"num_object_copies":2676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1338,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2162,"num_write_kb":85056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,11],"acting":[13,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":324,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-09-13T11:25:39.356955+0000","last_change":"2024-09-13T11:24:11.895017+0000","last_active":"2024-09-13T11:25:39.356955+0000","last_peered":"2024-09-13T11:25:39.356955+0000","last_clean":"2024-09-13T11:25:39.356955+0000","last_became_active":"2024-09-13T11:22:14.211435+0000","last_became_peered":"2024-09-13T11:22:14.211435+0000","last_unstale":"2024-09-13T11:25:39.356955+0000","last_undegraded":"2024-09-13T11:25:39.356955+0000","last_fullsized":"2024-09-13T11:25:39.356955+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:24:11.894937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:24:11.894937+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:43.628381+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0010010360000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2","version":"176'5264","reported_seq":9115,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-09-13T11:25:40.989461+0000","last_change":"2024-09-13T11:25:23.765128+0000","last_active":"2024-09-13T11:25:40.989461+0000","last_peered":"2024-09-13T11:25:40.989461+0000","last_clean":"2024-09-13T11:25:40.989461+0000","last_became_active":"2024-09-13T11:24:29.175407+0000","last_became_peered":"2024-09-13T11:24:29.175407+0000","last_unstale":"2024-09-13T11:25:40.989461+0000","last_undegraded":"2024-09-13T11:25:40.989461+0000","last_fullsized":"2024-09-13T11:25:40.989461+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"164'4994","last_scrub_stamp":"2024-09-13T11:25:23.765047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:23.765047+0000","objects_scrubbed":1689,"log_size":5264,"log_dups_size":0,"ondisk_log_size":5264,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:26:45.252942+0000","scrub_duration":283,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92798976,"num_objects":1416,"num_object_clones":0,"num_object_copies":2832,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1416,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2207,"num_write_kb":89024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1731,"num_bytes_recovered":113442816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,3],"acting":[12,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[]},{"pgid":"2.3","version":"26'1","reported_seq":135,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-09-13T11:25:39.347207+0000","last_change":"2024-09-13T11:25:23.034357+0000","last_active":"2024-09-13T11:25:39.347207+0000","last_peered":"2024-09-13T11:25:39.347207+0000","last_clean":"2024-09-13T11:25:39.347207+0000","last_became_active":"2024-09-13T11:24:29.110028+0000","last_became_peered":"2024-09-13T11:24:29.110028+0000","last_unstale":"2024-09-13T11:25:39.347207+0000","last_undegraded":"2024-09-13T11:25:39.347207+0000","last_fullsized":"2024-09-13T11:25:39.347207+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"26'1","last_scrub_stamp":"2024-09-13T11:25:23.034247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:25:23.034247+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:26:42.429349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,8],"acting":[15,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"1.0","version":"26'112","reported_seq":466,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-09-13T11:25:39.335558+0000","last_change":"2024-09-13T11:25:23.642366+0000","last_active":"2024-09-13T11:25:39.335558+0000","last_peered":"2024-09-13T11:25:39.335558+0000","last_clean":"2024-09-13T11:25:39.335558+0000","last_became_active":"2024-09-13T11:22:12.129721+0000","last_became_peered":"2024-09-13T11:22:12.129721+0000","last_unstale":"2024-09-13T11:25:39.335558+0000","last_undegraded":"2024-09-13T11:25:39.335558+0000","last_fullsized":"2024-09-13T11:25:39.335558+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"26'112","last_scrub_stamp":"2024-09-13T11:25:23.642204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:11.098388+0000","last_clean_scrub_stamp":"2024-09-13T11:25:23.642204+0000","objects_scrubbed":2,"log_size":112,"log_dups_size":0,"ondisk_log_size":112,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:26:42.628319+0000","scrub_duration":67,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.8","version":"176'6044","reported_seq":6413,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-09-13T11:25:41.054086+0000","last_change":"2024-09-13T11:24:24.506212+0000","last_active":"2024-09-13T11:25:41.054086+0000","last_peered":"2024-09-13T11:25:41.054086+0000","last_clean":"2024-09-13T11:25:41.054086+0000","last_became_active":"2024-09-13T11:24:24.096220+0000","last_became_peered":"2024-09-13T11:24:24.096220+0000","last_unstale":"2024-09-13T11:25:41.054086+0000","last_undegraded":"2024-09-13T11:25:41.054086+0000","last_fullsized":"2024-09-13T11:25:41.054086+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'4749","last_scrub_stamp":"2024-09-13T11:24:24.506109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:24:24.506109+0000","objects_scrubbed":1690,"log_size":6044,"log_dups_size":0,"ondisk_log_size":6044,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:49.430042+0000","scrub_duration":215,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":89784320,"num_objects":1370,"num_object_clones":0,"num_object_copies":2740,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1370,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1296,"num_write_kb":31232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.9","version":"176'6109","reported_seq":6477,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-09-13T11:25:41.101105+0000","last_change":"2024-09-13T11:24:24.928964+0000","last_active":"2024-09-13T11:25:41.101105+0000","last_peered":"2024-09-13T11:25:41.101105+0000","last_clean":"2024-09-13T11:25:41.101105+0000","last_became_active":"2024-09-13T11:24:24.082504+0000","last_became_peered":"2024-09-13T11:24:24.082504+0000","last_unstale":"2024-09-13T11:25:41.101105+0000","last_undegraded":"2024-09-13T11:25:41.101105+0000","last_fullsized":"2024-09-13T11:25:41.101105+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'4799","last_scrub_stamp":"2024-09-13T11:24:24.928724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:24:24.928724+0000","objects_scrubbed":1705,"log_size":6109,"log_dups_size":0,"ondisk_log_size":6109,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:43.845801+0000","scrub_duration":255,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":89391104,"num_objects":1364,"num_object_clones":0,"num_object_copies":2728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1364,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1313,"num_write_kb":31104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,2],"acting":[12,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[]},{"pgid":"3.a","version":"176'6429","reported_seq":6802,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-09-13T11:25:40.045542+0000","last_change":"2024-09-13T11:24:25.137871+0000","last_active":"2024-09-13T11:25:40.045542+0000","last_peered":"2024-09-13T11:25:40.045542+0000","last_clean":"2024-09-13T11:25:40.045542+0000","last_became_active":"2024-09-13T11:24:24.089253+0000","last_became_peered":"2024-09-13T11:24:24.089253+0000","last_unstale":"2024-09-13T11:25:40.045542+0000","last_undegraded":"2024-09-13T11:25:40.045542+0000","last_fullsized":"2024-09-13T11:25:40.045542+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'5157","last_scrub_stamp":"2024-09-13T11:24:25.137777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:24:25.137777+0000","objects_scrubbed":1777,"log_size":6429,"log_dups_size":0,"ondisk_log_size":6429,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":223,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90177536,"num_objects":1376,"num_object_clones":0,"num_object_copies":2752,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1376,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1275,"num_write_kb":27968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.b","version":"177'6442","reported_seq":6831,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-09-13T11:25:41.965533+0000","last_change":"2024-09-13T11:25:38.359869+0000","last_active":"2024-09-13T11:25:41.965533+0000","last_peered":"2024-09-13T11:25:41.965533+0000","last_clean":"2024-09-13T11:25:41.965533+0000","last_became_active":"2024-09-13T11:24:24.085684+0000","last_became_peered":"2024-09-13T11:24:24.085684+0000","last_unstale":"2024-09-13T11:25:41.965533+0000","last_undegraded":"2024-09-13T11:25:41.965533+0000","last_fullsized":"2024-09-13T11:25:41.965533+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'5112","last_scrub_stamp":"2024-09-13T11:24:25.503716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:24:25.503716+0000","objects_scrubbed":949,"log_size":6442,"log_dups_size":0,"ondisk_log_size":6442,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:55.125774+0000","scrub_duration":395,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88211456,"num_objects":1346,"num_object_clones":0,"num_object_copies":2692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1346,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1330,"num_write_kb":30656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,1],"acting":[13,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"3.c","version":"178'6465","reported_seq":6863,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-09-13T11:25:42.020852+0000","last_change":"2024-09-13T11:25:36.279464+0000","last_active":"2024-09-13T11:25:42.020852+0000","last_peered":"2024-09-13T11:25:42.020852+0000","last_clean":"2024-09-13T11:25:42.020852+0000","last_became_active":"2024-09-13T11:24:24.090847+0000","last_became_peered":"2024-09-13T11:24:24.090847+0000","last_unstale":"2024-09-13T11:25:42.020852+0000","last_undegraded":"2024-09-13T11:25:42.020852+0000","last_fullsized":"2024-09-13T11:25:42.020852+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"174'6334","last_scrub_stamp":"2024-09-13T11:25:36.279390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:36.279390+0000","objects_scrubbed":1456,"log_size":6465,"log_dups_size":0,"ondisk_log_size":6465,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:27:04.013051+0000","scrub_duration":235,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":86769664,"num_objects":1324,"num_object_clones":0,"num_object_copies":2648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1324,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1316,"num_write_kb":29440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,13],"acting":[11,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.d","version":"176'6419","reported_seq":6790,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-09-13T11:25:41.161098+0000","last_change":"2024-09-13T11:24:24.967444+0000","last_active":"2024-09-13T11:25:41.161098+0000","last_peered":"2024-09-13T11:25:41.161098+0000","last_clean":"2024-09-13T11:25:41.161098+0000","last_became_active":"2024-09-13T11:24:24.087224+0000","last_became_peered":"2024-09-13T11:24:24.087224+0000","last_unstale":"2024-09-13T11:25:41.161098+0000","last_undegraded":"2024-09-13T11:25:41.161098+0000","last_fullsized":"2024-09-13T11:25:41.161098+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'5110","last_scrub_stamp":"2024-09-13T11:24:24.967337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:24:24.967337+0000","objects_scrubbed":1725,"log_size":6419,"log_dups_size":0,"ondisk_log_size":6419,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:53.620949+0000","scrub_duration":255,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90374144,"num_objects":1379,"num_object_clones":0,"num_object_copies":2758,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1379,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1312,"num_write_kb":30912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.e","version":"176'7834","reported_seq":13113,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-09-13T11:25:41.096357+0000","last_change":"2024-09-13T11:25:38.368658+0000","last_active":"2024-09-13T11:25:41.096357+0000","last_peered":"2024-09-13T11:25:41.096357+0000","last_clean":"2024-09-13T11:25:41.096357+0000","last_became_active":"2024-09-13T11:23:31.925869+0000","last_became_peered":"2024-09-13T11:23:31.925869+0000","last_unstale":"2024-09-13T11:25:41.096357+0000","last_undegraded":"2024-09-13T11:25:41.096357+0000","last_fullsized":"2024-09-13T11:25:41.096357+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'3527","last_scrub_stamp":"2024-09-13T11:23:32.368727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:32.368727+0000","objects_scrubbed":799,"log_size":7834,"log_dups_size":0,"ondisk_log_size":7834,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:43.357323+0000","scrub_duration":219,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":173670400,"num_objects":2650,"num_object_clones":0,"num_object_copies":5300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2650,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4308,"num_write_kb":165888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"178'7849","reported_seq":8244,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-09-13T11:25:41.996866+0000","last_change":"2024-09-13T11:25:00.289940+0000","last_active":"2024-09-13T11:25:41.996866+0000","last_peered":"2024-09-13T11:25:41.996866+0000","last_clean":"2024-09-13T11:25:41.996866+0000","last_became_active":"2024-09-13T11:23:31.935239+0000","last_became_peered":"2024-09-13T11:23:31.935239+0000","last_unstale":"2024-09-13T11:25:41.996866+0000","last_undegraded":"2024-09-13T11:25:41.996866+0000","last_fullsized":"2024-09-13T11:25:41.996866+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"146'6154","last_scrub_stamp":"2024-09-13T11:25:00.289673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:00.289673+0000","objects_scrubbed":4394,"log_size":7849,"log_dups_size":0,"ondisk_log_size":7849,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:26:24.274856+0000","scrub_duration":1187,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":175439872,"num_objects":2677,"num_object_clones":0,"num_object_copies":5354,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2677,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4429,"num_write_kb":170560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,12],"acting":[11,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.10","version":"176'7446","reported_seq":11989,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-09-13T11:25:41.061481+0000","last_change":"2024-09-13T11:25:10.161564+0000","last_active":"2024-09-13T11:25:41.061481+0000","last_peered":"2024-09-13T11:25:41.061481+0000","last_clean":"2024-09-13T11:25:41.061481+0000","last_became_active":"2024-09-13T11:23:41.931183+0000","last_became_peered":"2024-09-13T11:23:41.931183+0000","last_unstale":"2024-09-13T11:25:41.061481+0000","last_undegraded":"2024-09-13T11:25:41.061481+0000","last_fullsized":"2024-09-13T11:25:41.061481+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":5,"last_scrub":"151'6298","last_scrub_stamp":"2024-09-13T11:25:10.161464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:10.161464+0000","objects_scrubbed":3964,"log_size":7446,"log_dups_size":0,"ondisk_log_size":7446,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:26:33.015478+0000","scrub_duration":607,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":183566336,"num_objects":2801,"num_object_clones":0,"num_object_copies":5602,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2801,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3980,"num_write_kb":152000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.11","version":"176'7457","reported_seq":10772,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-09-13T11:25:41.027672+0000","last_change":"2024-09-13T11:25:24.287326+0000","last_active":"2024-09-13T11:25:41.027672+0000","last_peered":"2024-09-13T11:25:41.027672+0000","last_clean":"2024-09-13T11:25:41.027672+0000","last_became_active":"2024-09-13T11:23:39.063172+0000","last_became_peered":"2024-09-13T11:23:39.063172+0000","last_unstale":"2024-09-13T11:25:41.027672+0000","last_undegraded":"2024-09-13T11:25:41.027672+0000","last_fullsized":"2024-09-13T11:25:41.027672+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":5,"last_scrub":"165'6821","last_scrub_stamp":"2024-09-13T11:25:24.287082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:24.287082+0000","objects_scrubbed":3406,"log_size":7457,"log_dups_size":0,"ondisk_log_size":7457,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:26:49.405332+0000","scrub_duration":947,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180682752,"num_objects":2757,"num_object_clones":0,"num_object_copies":5514,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2757,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4116,"num_write_kb":155584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11],"acting":[4,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.12","version":"176'7449","reported_seq":17583,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-09-13T11:25:41.162018+0000","last_change":"2024-09-13T11:25:38.338725+0000","last_active":"2024-09-13T11:25:41.162018+0000","last_peered":"2024-09-13T11:25:41.162018+0000","last_clean":"2024-09-13T11:25:41.162018+0000","last_became_active":"2024-09-13T11:24:29.268229+0000","last_became_peered":"2024-09-13T11:24:29.268229+0000","last_unstale":"2024-09-13T11:25:41.162018+0000","last_undegraded":"2024-09-13T11:25:41.162018+0000","last_fullsized":"2024-09-13T11:25:41.162018+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'3109","last_scrub_stamp":"2024-09-13T11:23:35.111320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:23:35.111320+0000","objects_scrubbed":499,"log_size":7449,"log_dups_size":0,"ondisk_log_size":7449,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:25:43.335690+0000","scrub_duration":239,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180748288,"num_objects":2758,"num_object_clones":0,"num_object_copies":5516,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2758,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4340,"num_write_kb":171712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3457,"num_bytes_recovered":226557952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.13","version":"178'7483","reported_seq":10736,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-09-13T11:25:42.039637+0000","last_change":"2024-09-13T11:25:01.153044+0000","last_active":"2024-09-13T11:25:42.039637+0000","last_peered":"2024-09-13T11:25:42.039637+0000","last_clean":"2024-09-13T11:25:42.039637+0000","last_became_active":"2024-09-13T11:23:41.082433+0000","last_became_peered":"2024-09-13T11:23:41.082433+0000","last_unstale":"2024-09-13T11:25:42.039637+0000","last_undegraded":"2024-09-13T11:25:42.039637+0000","last_fullsized":"2024-09-13T11:25:42.039637+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":5,"last_scrub":"147'5921","last_scrub_stamp":"2024-09-13T11:25:01.152971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:01.152971+0000","objects_scrubbed":4311,"log_size":7483,"log_dups_size":0,"ondisk_log_size":7483,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:26:07.027631+0000","scrub_duration":1011,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":177471488,"num_objects":2708,"num_object_clones":0,"num_object_copies":5416,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2708,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4087,"num_write_kb":154048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7],"acting":[11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":46,"stat_sum":{"num_bytes":5707890712,"num_objects":87097,"num_object_clones":0,"num_object_copies":174194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":87097,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":105129,"num_write_kb":3441889,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18803,"num_bytes_recovered":1232273408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12818784256,"data_stored":12818776112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":43282,"internal_metadata":0},"log_size":302986,"ondisk_log_size":302986,"up":92,"acting":92,"num_store_stats":16},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":11},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":112,"ondisk_log_size":112,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":13,"up_from":23,"seq":98784247854,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":624940,"kb_used_data":525724,"kb_used_omap":1,"kb_used_meta":99198,"kb_avail":93746900,"statfs":{"total":96636764160,"available":95996825600,"internally_reserved":0,"allocated":538341376,"data_stored":537550437,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":101579205},"hb_peers":[0,1,2,4,6,8,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":23,"seq":98784247853,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":589012,"kb_used_data":517508,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":93782828,"statfs":{"total":96636764160,"available":96033615872,"internally_reserved":0,"allocated":529928192,"data_stored":529171720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":73136576},"hb_peers":[0,1,3,8,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":40,"seq":171798691876,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":988380,"kb_used_data":871628,"kb_used_omap":4,"kb_used_meta":116667,"kb_avail":93383460,"statfs":{"total":96636764160,"available":95624663040,"internally_reserved":0,"allocated":892547072,"data_stored":891771948,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4433,"internal_metadata":119467695},"hb_peers":[0,2,3,4,5,6,7,8,9,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":21,"seq":90194313261,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1034620,"kb_used_data":922924,"kb_used_omap":1,"kb_used_meta":111678,"kb_avail":93337220,"statfs":{"total":96636764160,"available":95577313280,"internally_reserved":0,"allocated":945074176,"data_stored":944273017,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":114358727},"hb_peers":[0,2,4,5,7,9,10,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":21,"seq":90194313264,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1193448,"kb_used_data":1071960,"kb_used_omap":1,"kb_used_meta":121406,"kb_avail":93178392,"statfs":{"total":96636764160,"available":95414673408,"internally_reserved":0,"allocated":1097687040,"data_stored":1096902000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":124320199},"hb_peers":[0,2,4,6,7,8,9,11,12,13],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":22,"seq":94489280556,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1149172,"kb_used_data":1045796,"kb_used_omap":1,"kb_used_meta":103358,"kb_avail":93222668,"statfs":{"total":96636764160,"available":95460012032,"internally_reserved":0,"allocated":1070895104,"data_stored":1070159725,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":105839045},"hb_peers":[1,3,4,5,7,8,10,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":22,"seq":94489280557,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":489880,"kb_used_data":426760,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":93881960,"statfs":{"total":96636764160,"available":96135127040,"internally_reserved":0,"allocated":437002240,"data_stored":436236683,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[0,1,3,4,5,7,8,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":15,"up_from":22,"seq":94489280557,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1016100,"kb_used_data":908564,"kb_used_omap":1,"kb_used_meta":107518,"kb_avail":93355740,"statfs":{"total":96636764160,"available":95596277760,"internally_reserved":0,"allocated":930369536,"data_stored":929649625,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":110098886},"hb_peers":[0,4,5,6,7,8,10,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":22,"seq":94489280558,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":689128,"kb_used_data":617688,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":93682712,"statfs":{"total":96636764160,"available":95931097088,"internally_reserved":0,"allocated":632512512,"data_stored":631732344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73136581},"hb_peers":[3,5,6,7,9,10,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":22,"seq":94489280557,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":512960,"kb_used_data":455408,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":93858880,"statfs":{"total":96636764160,"available":96111493120,"internally_reserved":0,"allocated":466337792,"data_stored":465592572,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58915270},"hb_peers":[0,1,3,5,7,8,10,11,12,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":22,"seq":94489280557,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1270148,"kb_used_data":1154228,"kb_used_omap":1,"kb_used_meta":115838,"kb_avail":93101692,"statfs":{"total":96636764160,"available":95336132608,"internally_reserved":0,"allocated":1181929472,"data_stored":1181177060,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":118618570},"hb_peers":[0,3,5,6,8,9,10,11,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":22,"seq":94489280557,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":511480,"kb_used_data":431720,"kb_used_omap":1,"kb_used_meta":79742,"kb_avail":93860360,"statfs":{"total":96636764160,"available":96113008640,"internally_reserved":0,"allocated":442081280,"data_stored":441281379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":81656261},"hb_peers":[1,5,6,7,9,10,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":22,"seq":94489280556,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":870156,"kb_used_data":788988,"kb_used_omap":1,"kb_used_meta":81150,"kb_avail":93501684,"statfs":{"total":96636764160,"available":95745724416,"internally_reserved":0,"allocated":807923712,"data_stored":807166953,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":83098048},"hb_peers":[0,1,3,5,7,9,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":11,"up_from":22,"seq":94489280558,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1240448,"kb_used_data":1128752,"kb_used_omap":1,"kb_used_meta":111678,"kb_avail":93131392,"statfs":{"total":96636764160,"available":95366545408,"internally_reserved":0,"allocated":1155842048,"data_stored":1155048826,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":114358724},"hb_peers":[0,1,2,4,7,8,10,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":12,"up_from":22,"seq":94489280557,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1079400,"kb_used_data":976024,"kb_used_omap":1,"kb_used_meta":103358,"kb_avail":93292440,"statfs":{"total":96636764160,"available":95531458560,"internally_reserved":0,"allocated":999448576,"data_stored":998665331,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":105839045},"hb_peers":[0,1,2,3,4,5,10,11,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":14,"up_from":0,"seq":609885356042,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1068105728,"data_stored":1068105728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":883294208,"data_stored":883294208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":431362048,"data_stored":431357976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":939917312,"data_stored":939917312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":623771648,"data_stored":623771648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1083179008,"data_stored":1083179008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":460980224,"data_stored":460980224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1177419776,"data_stored":1177419776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":439287808,"data_stored":439287808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":527368192,"data_stored":527368192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":796196864,"data_stored":796196864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":1151926272,"data_stored":1151926272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":986710016,"data_stored":986710016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":527564800,"data_stored":527564800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":794099712,"data_stored":794099712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":43282,"internal_metadata":0},{"poolid":3,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":927600640,"data_stored":927596568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T11:25:44.432 INFO:tasks.ceph.ceph_manager.ceph:recovered! 2024-09-13T11:25:44.432 INFO:tasks.thrashosds.thrasher:choose_action: min_in 2 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-13T11:25:44.432 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-09-13T11:25:44.433 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 'trim stale osdmaps' 2024-09-13T11:25:44.451 INFO:tasks.ceph.osd.15.smithi177.stderr:2024-09-13T11:25:44.450+0000 7f9d6ce77640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26578) UID: 0 2024-09-13T11:25:44.483 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-13T11:25:44.483 INFO:teuthology.orchestra.run.smithi162.stdout: "size": 20, 2024-09-13T11:25:44.483 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 600, 2024-09-13T11:25:44.484 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [ 2024-09-13T11:25:44.484 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.484 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:45810 3.17 3.9bd13e57 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e69)", 2024-09-13T11:25:44.484 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:21.347960+0000", 2024-09-13T11:25:44.484 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 143.133524241, 2024-09-13T11:25:44.485 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.56814464200000003, 2024-09-13T11:25:44.485 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:25:44.485 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:25:44.485 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-09-13T11:25:44.485 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:25:44.485 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:25:44.486 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:44.486 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 45810 2024-09-13T11:25:44.486 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.486 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:25:44.486 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.487 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:25:44.487 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:21.347960+0000", 2024-09-13T11:25:44.487 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:44.487 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.487 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.487 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:25:44.488 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:21.347960+0000", 2024-09-13T11:25:44.488 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:44.488 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.488 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.488 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:25:44.489 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:21.347965+0000", 2024-09-13T11:25:44.489 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.6720000000000003e-06 2024-09-13T11:25:44.489 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.489 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.489 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:25:44.489 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:21.348019+0000", 2024-09-13T11:25:44.490 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.4391000000000002e-05 2024-09-13T11:25:44.490 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.490 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.490 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:25:44.490 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:21.348052+0000", 2024-09-13T11:25:44.491 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.2246000000000003e-05 2024-09-13T11:25:44.491 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.491 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.491 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:25:44.491 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:21.348072+0000", 2024-09-13T11:25:44.492 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.0622e-05 2024-09-13T11:25:44.492 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.492 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.492 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:25:44.492 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:21.348278+0000", 2024-09-13T11:25:44.493 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000205391 2024-09-13T11:25:44.493 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.493 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.493 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-09-13T11:25:44.493 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:21.348292+0000", 2024-09-13T11:25:44.493 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4759e-05 2024-09-13T11:25:44.494 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.494 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.494 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:25:44.494 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:21.916064+0000", 2024-09-13T11:25:44.494 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.56777128600000004 2024-09-13T11:25:44.495 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.495 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.495 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:25:44.495 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:21.916105+0000", 2024-09-13T11:25:44.495 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.1275e-05 2024-09-13T11:25:44.495 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:44.496 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:25:44.496 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:44.496 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.496 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.496 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:54653 3.23 3:c7a54800:::benchmark_data_smithi177_31116_object54652:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e75)", 2024-09-13T11:25:44.497 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:30.979059+0000", 2024-09-13T11:25:44.497 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 133.50242574200001, 2024-09-13T11:25:44.497 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.97284747100000002, 2024-09-13T11:25:44.497 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:25:44.497 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:25:44.497 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-09-13T11:25:44.498 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:25:44.498 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:25:44.498 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:44.498 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 54653 2024-09-13T11:25:44.499 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.499 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:25:44.499 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.499 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:25:44.500 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.979059+0000", 2024-09-13T11:25:44.500 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:44.500 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.500 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.500 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:25:44.501 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.979059+0000", 2024-09-13T11:25:44.501 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:44.501 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.501 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.501 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:25:44.502 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.979061+0000", 2024-09-13T11:25:44.502 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.4279999999999999e-06 2024-09-13T11:25:44.502 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.502 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.502 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:25:44.503 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.979114+0000", 2024-09-13T11:25:44.503 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.3254000000000002e-05 2024-09-13T11:25:44.503 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.503 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.503 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:25:44.503 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.979157+0000", 2024-09-13T11:25:44.504 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.2018000000000003e-05 2024-09-13T11:25:44.504 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.504 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.504 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:25:44.505 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.979172+0000", 2024-09-13T11:25:44.505 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5636000000000001e-05 2024-09-13T11:25:44.505 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.505 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.505 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:25:44.506 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:31.045109+0000", 2024-09-13T11:25:44.506 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.065936982000000005 2024-09-13T11:25:44.506 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.506 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.506 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-09-13T11:25:44.506 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:31.045121+0000", 2024-09-13T11:25:44.507 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1608e-05 2024-09-13T11:25:44.507 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.507 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.507 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:25:44.507 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:31.951636+0000", 2024-09-13T11:25:44.507 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.90651503899999997 2024-09-13T11:25:44.508 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.508 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.508 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:25:44.508 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:31.951906+0000", 2024-09-13T11:25:44.508 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000270506 2024-09-13T11:25:44.509 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:44.509 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:25:44.509 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:44.509 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.509 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.510 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPush(3.18 76/70 [PushOp(3:1a32c9cd:::benchmark_data_smithi177_31116_object12833:head, version: 49'818, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:1a32c9cd:::benchmark_data_smithi177_31116_object12833:head@49'818, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:25:44.510 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:32.522305+0000", 2024-09-13T11:25:44.510 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 131.95917965800001, 2024-09-13T11:25:44.510 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.40201207100000003, 2024-09-13T11:25:44.510 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:25:44.511 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:25:44.511 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:25:44.511 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:25:44.511 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.511 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:25:44.511 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.522305+0000", 2024-09-13T11:25:44.512 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:44.512 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.512 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.512 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:25:44.512 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.522305+0000", 2024-09-13T11:25:44.513 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:44.513 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.513 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.513 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:25:44.513 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.522309+0000", 2024-09-13T11:25:44.514 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.9829999999999998e-06 2024-09-13T11:25:44.514 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.514 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.514 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:25:44.514 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.522377+0000", 2024-09-13T11:25:44.514 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.8069000000000002e-05 2024-09-13T11:25:44.515 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.515 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.515 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:25:44.515 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.921950+0000", 2024-09-13T11:25:44.515 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.399573438 2024-09-13T11:25:44.516 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.516 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.516 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:25:44.516 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.921983+0000", 2024-09-13T11:25:44.517 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.2858000000000003e-05 2024-09-13T11:25:44.517 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.517 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.517 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:25:44.517 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.922389+0000", 2024-09-13T11:25:44.517 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00040612999999999999 2024-09-13T11:25:44.518 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.518 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.518 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:25:44.518 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.922421+0000", 2024-09-13T11:25:44.518 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.1356999999999998e-05 2024-09-13T11:25:44.519 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.519 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.519 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:25:44.520 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.924317+0000", 2024-09-13T11:25:44.520 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0018962359999999999 2024-09-13T11:25:44.520 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:44.520 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:25:44.520 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:44.521 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.521 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.521 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPush(3.17 76/70 [PushOp(3:ebcbce67:::benchmark_data_smithi177_31116_object18268:head, version: 53'1102, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:ebcbce67:::benchmark_data_smithi177_31116_object18268:head@53'1102, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:25:44.521 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:32.638908+0000", 2024-09-13T11:25:44.521 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 131.84257621, 2024-09-13T11:25:44.521 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.229064031, 2024-09-13T11:25:44.522 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:25:44.522 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:25:44.522 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:25:44.522 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:25:44.523 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.523 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:25:44.523 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.638908+0000", 2024-09-13T11:25:44.523 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:44.523 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.524 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.524 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:25:44.524 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.638908+0000", 2024-09-13T11:25:44.524 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:44.524 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.525 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.525 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:25:44.525 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.638910+0000", 2024-09-13T11:25:44.525 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.7379999999999999e-06 2024-09-13T11:25:44.525 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.525 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.526 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:25:44.526 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.638975+0000", 2024-09-13T11:25:44.526 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.4937000000000005e-05 2024-09-13T11:25:44.526 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.526 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.526 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:25:44.527 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.865860+0000", 2024-09-13T11:25:44.527 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.22688509800000001 2024-09-13T11:25:44.527 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.527 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.527 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:25:44.528 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.865890+0000", 2024-09-13T11:25:44.528 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.0273e-05 2024-09-13T11:25:44.528 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.528 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.528 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:25:44.529 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.866143+0000", 2024-09-13T11:25:44.529 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00025212800000000001 2024-09-13T11:25:44.529 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.529 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.529 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:25:44.529 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.866181+0000", 2024-09-13T11:25:44.530 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.8945000000000001e-05 2024-09-13T11:25:44.530 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.530 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.530 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:25:44.530 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.867972+0000", 2024-09-13T11:25:44.530 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001790912 2024-09-13T11:25:44.531 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:44.531 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:25:44.531 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:44.531 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.531 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.532 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:57435 3.3 3:c0db4777:::benchmark_data_smithi177_31116_object57434:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e80)", 2024-09-13T11:25:44.532 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:37.436034+0000", 2024-09-13T11:25:44.532 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 127.04545075, 2024-09-13T11:25:44.532 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.35949142000000001, 2024-09-13T11:25:44.532 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:25:44.533 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:25:44.533 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:44.533 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:25:44.533 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:25:44.533 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:44.533 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 57435 2024-09-13T11:25:44.534 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.534 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:25:44.534 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.534 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:25:44.534 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:37.436034+0000", 2024-09-13T11:25:44.535 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:44.535 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.535 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.535 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:25:44.535 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:37.436034+0000", 2024-09-13T11:25:44.535 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:44.536 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.536 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.536 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:25:44.536 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:37.436037+0000", 2024-09-13T11:25:44.536 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.7829999999999999e-06 2024-09-13T11:25:44.536 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.537 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.537 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:25:44.537 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:37.436090+0000", 2024-09-13T11:25:44.537 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.3665999999999997e-05 2024-09-13T11:25:44.537 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.538 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.538 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:25:44.538 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:37.436112+0000", 2024-09-13T11:25:44.538 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.1556000000000002e-05 2024-09-13T11:25:44.538 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.538 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.539 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:25:44.539 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:37.436122+0000", 2024-09-13T11:25:44.539 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0098000000000001e-05 2024-09-13T11:25:44.539 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.539 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.540 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:25:44.540 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:37.436187+0000", 2024-09-13T11:25:44.540 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.4857999999999995e-05 2024-09-13T11:25:44.540 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.540 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.541 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:25:44.541 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:37.436285+0000", 2024-09-13T11:25:44.541 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.8643999999999999e-05 2024-09-13T11:25:44.541 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.541 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.541 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 7", 2024-09-13T11:25:44.542 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:37.436423+0000", 2024-09-13T11:25:44.542 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00013809299999999999 2024-09-13T11:25:44.542 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.542 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.542 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:25:44.543 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:37.440125+0000", 2024-09-13T11:25:44.543 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0037015239999999999 2024-09-13T11:25:44.543 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.543 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.543 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:25:44.543 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:37.795397+0000", 2024-09-13T11:25:44.544 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.35527234400000002 2024-09-13T11:25:44.544 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.544 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.544 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:25:44.544 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:37.795485+0000", 2024-09-13T11:25:44.545 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.7538000000000001e-05 2024-09-13T11:25:44.545 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.545 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.545 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:25:44.545 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:37.795525+0000", 2024-09-13T11:25:44.545 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.0315999999999998e-05 2024-09-13T11:25:44.546 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:44.546 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:25:44.546 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:44.546 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.546 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.547 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPush(3.1f 80/78 [PushOp(3:fc3ed0c7:::benchmark_data_smithi177_31116_object672:head, version: 43'36, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:fc3ed0c7:::benchmark_data_smithi177_31116_object672:head@43'36, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:25:44.547 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:37.801754+0000", 2024-09-13T11:25:44.547 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 126.67973039, 2024-09-13T11:25:44.547 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.209613519, 2024-09-13T11:25:44.547 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:25:44.547 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:25:44.548 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:25:44.548 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:25:44.548 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.548 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:25:44.548 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:37.801754+0000", 2024-09-13T11:25:44.549 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:44.549 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.549 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.549 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:25:44.549 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:37.801754+0000", 2024-09-13T11:25:44.549 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:44.550 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.550 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.550 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:25:44.550 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:37.801756+0000", 2024-09-13T11:25:44.550 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.804e-06 2024-09-13T11:25:44.550 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.550 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.550 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:25:44.550 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:37.801947+0000", 2024-09-13T11:25:44.550 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000190636 2024-09-13T11:25:44.550 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.550 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.551 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:25:44.551 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:38.010478+0000", 2024-09-13T11:25:44.551 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.20853091100000001 2024-09-13T11:25:44.551 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.551 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.551 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:25:44.551 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:38.010506+0000", 2024-09-13T11:25:44.551 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.8793999999999999e-05 2024-09-13T11:25:44.551 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.551 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.551 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:25:44.551 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:38.010601+0000", 2024-09-13T11:25:44.552 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.4786999999999999e-05 2024-09-13T11:25:44.552 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.552 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.552 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:25:44.552 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:38.010617+0000", 2024-09-13T11:25:44.552 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5790999999999998e-05 2024-09-13T11:25:44.552 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.552 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.552 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:25:44.552 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:38.011368+0000", 2024-09-13T11:25:44.552 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00075079599999999997 2024-09-13T11:25:44.552 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:44.552 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:25:44.552 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:44.553 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.553 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.553 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:60433 3.3 3:c01f7ddc:::benchmark_data_smithi177_31116_object60432:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e85)", 2024-09-13T11:25:44.553 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:42.199382+0000", 2024-09-13T11:25:44.553 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 122.28210285900001, 2024-09-13T11:25:44.553 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.33918965200000001, 2024-09-13T11:25:44.553 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:25:44.553 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:25:44.553 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:44.553 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:25:44.553 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:25:44.553 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:44.553 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 60433 2024-09-13T11:25:44.553 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.554 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:25:44.554 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.554 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:25:44.554 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.199382+0000", 2024-09-13T11:25:44.554 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:44.554 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.554 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.554 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:25:44.554 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.199382+0000", 2024-09-13T11:25:44.554 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:44.554 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.554 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.554 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:25:44.554 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.199386+0000", 2024-09-13T11:25:44.555 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.8759999999999998e-06 2024-09-13T11:25:44.555 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.555 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.555 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:25:44.555 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.199455+0000", 2024-09-13T11:25:44.555 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.9305000000000001e-05 2024-09-13T11:25:44.555 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.555 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.555 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:25:44.555 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.199482+0000", 2024-09-13T11:25:44.555 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.6749000000000001e-05 2024-09-13T11:25:44.555 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.555 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.555 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:25:44.556 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.199503+0000", 2024-09-13T11:25:44.556 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.1137000000000001e-05 2024-09-13T11:25:44.556 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.556 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.556 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:25:44.556 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.199602+0000", 2024-09-13T11:25:44.556 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.8743e-05 2024-09-13T11:25:44.556 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.556 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.556 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:25:44.556 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.199784+0000", 2024-09-13T11:25:44.556 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00018265100000000001 2024-09-13T11:25:44.556 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.556 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.557 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 7", 2024-09-13T11:25:44.557 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.199955+0000", 2024-09-13T11:25:44.557 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00017067100000000001 2024-09-13T11:25:44.557 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.557 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.557 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:25:44.557 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.202075+0000", 2024-09-13T11:25:44.557 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0021206749999999998 2024-09-13T11:25:44.557 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.557 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.557 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:25:44.557 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.538395+0000", 2024-09-13T11:25:44.557 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.33631963599999998 2024-09-13T11:25:44.557 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.557 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.558 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:25:44.558 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.538498+0000", 2024-09-13T11:25:44.558 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00010267099999999999 2024-09-13T11:25:44.558 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.558 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.558 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:25:44.558 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.538571+0000", 2024-09-13T11:25:44.558 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.3538e-05 2024-09-13T11:25:44.558 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:44.558 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:25:44.558 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:44.558 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.558 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.558 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop_reply(client.4665.0:60433 3.3 e85/75 ondisk, result = 0)", 2024-09-13T11:25:44.559 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:42.203186+0000", 2024-09-13T11:25:44.559 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 122.278298681, 2024-09-13T11:25:44.559 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.33540096699999999, 2024-09-13T11:25:44.559 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:25:44.559 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:25:44.559 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:25:44.559 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:25:44.559 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.559 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:25:44.559 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.203186+0000", 2024-09-13T11:25:44.559 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:44.559 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.559 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.559 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:25:44.560 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.203186+0000", 2024-09-13T11:25:44.560 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:44.560 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.560 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.560 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:25:44.560 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.203194+0000", 2024-09-13T11:25:44.560 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.0020000000000006e-06 2024-09-13T11:25:44.560 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.560 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.560 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:25:44.560 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.203202+0000", 2024-09-13T11:25:44.560 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.9829999999999996e-06 2024-09-13T11:25:44.560 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.561 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.561 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:25:44.561 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.538182+0000", 2024-09-13T11:25:44.561 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.33498029899999998 2024-09-13T11:25:44.561 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.561 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.561 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:25:44.561 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.538196+0000", 2024-09-13T11:25:44.561 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3807e-05 2024-09-13T11:25:44.561 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.561 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.561 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:25:44.561 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.538371+0000", 2024-09-13T11:25:44.562 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00017541200000000001 2024-09-13T11:25:44.562 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.562 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.562 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:25:44.562 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.538386+0000", 2024-09-13T11:25:44.562 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4833e-05 2024-09-13T11:25:44.562 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.562 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.562 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:25:44.562 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.538587+0000", 2024-09-13T11:25:44.562 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000200631 2024-09-13T11:25:44.562 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:44.562 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:25:44.562 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:44.563 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.563 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.563 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:69556 3.18 3:1d3dbeba:::benchmark_data_smithi177_31116_object69555:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94)", 2024-09-13T11:25:44.563 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:52.142180+0000", 2024-09-13T11:25:44.563 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 112.339304611, 2024-09-13T11:25:44.563 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.21280995899999999, 2024-09-13T11:25:44.563 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:25:44.563 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:25:44.563 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:44.563 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:25:44.563 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:25:44.563 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:44.563 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 69556 2024-09-13T11:25:44.563 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.564 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:25:44.564 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.564 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:25:44.564 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:52.142180+0000", 2024-09-13T11:25:44.564 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:44.564 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.564 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.564 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:25:44.564 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:52.142180+0000", 2024-09-13T11:25:44.564 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:44.564 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.564 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.564 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:25:44.564 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:52.142181+0000", 2024-09-13T11:25:44.565 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.3699999999999999e-07 2024-09-13T11:25:44.565 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.565 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.565 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:25:44.565 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:52.142212+0000", 2024-09-13T11:25:44.565 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.1334000000000002e-05 2024-09-13T11:25:44.565 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.565 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.565 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:25:44.565 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:52.142219+0000", 2024-09-13T11:25:44.566 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.9990000000000002e-06 2024-09-13T11:25:44.566 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.566 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.566 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:25:44.566 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:52.142224+0000", 2024-09-13T11:25:44.566 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.0259999999999998e-06 2024-09-13T11:25:44.566 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.566 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.566 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:25:44.566 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:52.142276+0000", 2024-09-13T11:25:44.566 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.1436e-05 2024-09-13T11:25:44.566 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.566 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.566 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:25:44.567 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:52.142396+0000", 2024-09-13T11:25:44.567 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00012011 2024-09-13T11:25:44.567 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.567 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.567 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 0", 2024-09-13T11:25:44.567 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:52.142503+0000", 2024-09-13T11:25:44.567 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000107701 2024-09-13T11:25:44.567 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.567 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.567 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:25:44.567 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:52.144566+0000", 2024-09-13T11:25:44.567 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.002062749 2024-09-13T11:25:44.567 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.567 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.568 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:25:44.568 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:52.354839+0000", 2024-09-13T11:25:44.568 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.21027254000000001 2024-09-13T11:25:44.568 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.568 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.568 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:25:44.568 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:52.354932+0000", 2024-09-13T11:25:44.568 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.3510000000000004e-05 2024-09-13T11:25:44.568 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.568 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.568 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:25:44.568 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:52.354990+0000", 2024-09-13T11:25:44.568 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.7617000000000001e-05 2024-09-13T11:25:44.568 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:44.569 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:25:44.569 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:44.569 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.569 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.569 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:83144 3.18 3:1c2a5085:::benchmark_data_smithi177_31116_object83143:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106)", 2024-09-13T11:25:44.569 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:24:04.186395+0000", 2024-09-13T11:25:44.569 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 100.295089562, 2024-09-13T11:25:44.569 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.29655527599999998, 2024-09-13T11:25:44.569 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:25:44.569 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:25:44.569 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:44.569 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:25:44.569 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:25:44.570 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:44.570 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 83144 2024-09-13T11:25:44.570 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.570 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:25:44.570 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.570 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:25:44.570 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.186395+0000", 2024-09-13T11:25:44.570 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:44.570 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.570 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.570 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:25:44.570 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.186395+0000", 2024-09-13T11:25:44.570 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:44.571 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.571 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.571 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:25:44.571 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.186398+0000", 2024-09-13T11:25:44.571 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.5299999999999999e-06 2024-09-13T11:25:44.571 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.571 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.571 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:25:44.571 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.186438+0000", 2024-09-13T11:25:44.571 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.0293000000000002e-05 2024-09-13T11:25:44.571 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.571 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.571 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:25:44.572 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.186450+0000", 2024-09-13T11:25:44.572 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2687000000000001e-05 2024-09-13T11:25:44.572 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.572 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.572 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:25:44.572 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.186456+0000", 2024-09-13T11:25:44.572 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.446e-06 2024-09-13T11:25:44.572 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.572 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.572 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:25:44.572 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.186503+0000", 2024-09-13T11:25:44.572 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.7491000000000003e-05 2024-09-13T11:25:44.573 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.573 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.573 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:25:44.573 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.186618+0000", 2024-09-13T11:25:44.573 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000114999 2024-09-13T11:25:44.573 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.573 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.573 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 0", 2024-09-13T11:25:44.573 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.186712+0000", 2024-09-13T11:25:44.573 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.3805000000000001e-05 2024-09-13T11:25:44.573 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.573 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.574 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:25:44.574 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.189753+0000", 2024-09-13T11:25:44.574 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.003040713 2024-09-13T11:25:44.574 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.574 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.574 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:25:44.574 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.482662+0000", 2024-09-13T11:25:44.574 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.29290945699999998 2024-09-13T11:25:44.574 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.574 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.574 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:25:44.574 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.482831+0000", 2024-09-13T11:25:44.574 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00016900699999999999 2024-09-13T11:25:44.575 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.575 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.575 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:25:44.575 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.482950+0000", 2024-09-13T11:25:44.575 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000118848 2024-09-13T11:25:44.575 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:44.575 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:25:44.575 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:44.575 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.575 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.575 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop_reply(client.4665.0:83144 3.18 e106/91 ondisk, result = 0)", 2024-09-13T11:25:44.575 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:24:04.188413+0000", 2024-09-13T11:25:44.575 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 100.293071552, 2024-09-13T11:25:44.575 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.294557562, 2024-09-13T11:25:44.576 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:25:44.576 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:25:44.576 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:25:44.576 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:25:44.576 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.576 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:25:44.576 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.188413+0000", 2024-09-13T11:25:44.576 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:44.576 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.576 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.576 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:25:44.577 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.188413+0000", 2024-09-13T11:25:44.577 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:44.577 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.577 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.580 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:25:44.580 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.188414+0000", 2024-09-13T11:25:44.580 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4309999999999999e-06 2024-09-13T11:25:44.580 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.580 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.581 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:25:44.581 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.188419+0000", 2024-09-13T11:25:44.581 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.2889999999999999e-06 2024-09-13T11:25:44.581 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.581 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.581 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:25:44.581 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.482442+0000", 2024-09-13T11:25:44.581 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.294023168 2024-09-13T11:25:44.581 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.581 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.581 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:25:44.581 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.482459+0000", 2024-09-13T11:25:44.581 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.7013000000000001e-05 2024-09-13T11:25:44.581 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.582 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.582 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:25:44.582 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.482615+0000", 2024-09-13T11:25:44.582 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00015620400000000001 2024-09-13T11:25:44.582 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.582 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.582 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:25:44.582 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.482644+0000", 2024-09-13T11:25:44.582 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.9093000000000001e-05 2024-09-13T11:25:44.582 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.582 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.582 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:25:44.582 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.482971+0000", 2024-09-13T11:25:44.583 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00032636399999999999 2024-09-13T11:25:44.583 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:44.583 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:25:44.583 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:44.583 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.583 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.583 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPush(3.1f 106/78 [PushOp(3:fd4be819:::benchmark_data_smithi177_31116_object54560:head, version: 74'3410, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:fd4be819:::benchmark_data_smithi177_31116_object54560:head@74'3410, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:25:44.583 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:24:04.397161+0000", 2024-09-13T11:25:44.583 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 100.08432314300001, 2024-09-13T11:25:44.583 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.51611096199999995, 2024-09-13T11:25:44.583 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:25:44.583 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:25:44.583 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:25:44.583 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:25:44.584 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.584 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:25:44.584 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.397161+0000", 2024-09-13T11:25:44.584 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:44.584 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.584 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.584 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:25:44.584 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.397161+0000", 2024-09-13T11:25:44.584 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:44.584 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.584 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.584 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:25:44.584 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.397163+0000", 2024-09-13T11:25:44.585 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4720000000000001e-06 2024-09-13T11:25:44.585 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.585 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.585 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:25:44.585 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.397228+0000", 2024-09-13T11:25:44.585 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.5141999999999994e-05 2024-09-13T11:25:44.585 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.585 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.585 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:25:44.585 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.911583+0000", 2024-09-13T11:25:44.585 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.51435473700000001 2024-09-13T11:25:44.585 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.585 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.585 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:25:44.585 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.911615+0000", 2024-09-13T11:25:44.586 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.2620000000000003e-05 2024-09-13T11:25:44.586 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.586 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.586 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:25:44.586 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.911759+0000", 2024-09-13T11:25:44.586 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00014376300000000001 2024-09-13T11:25:44.586 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.586 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.586 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:25:44.586 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.911779+0000", 2024-09-13T11:25:44.586 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.0003000000000001e-05 2024-09-13T11:25:44.586 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.586 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.586 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:25:44.587 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.913272+0000", 2024-09-13T11:25:44.587 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0014932249999999999 2024-09-13T11:25:44.587 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:44.587 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:25:44.587 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:44.587 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.587 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.587 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:84809 3.17 3:e9e52123:::benchmark_data_smithi177_31116_object84808:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-09-13T11:25:44.587 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:24:05.914353+0000", 2024-09-13T11:25:44.587 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 98.567131125000003, 2024-09-13T11:25:44.587 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.27210915899999999, 2024-09-13T11:25:44.587 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:25:44.587 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:25:44.588 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:44.588 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:25:44.588 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:25:44.588 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:44.588 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 84809 2024-09-13T11:25:44.588 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.588 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:25:44.588 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.588 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:25:44.588 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.914353+0000", 2024-09-13T11:25:44.588 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:44.588 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.588 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.588 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:25:44.589 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.914353+0000", 2024-09-13T11:25:44.589 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:44.589 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.589 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.589 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:25:44.589 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.914356+0000", 2024-09-13T11:25:44.589 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.8710000000000001e-06 2024-09-13T11:25:44.589 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.589 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.589 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:25:44.589 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.914443+0000", 2024-09-13T11:25:44.589 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.7002000000000005e-05 2024-09-13T11:25:44.589 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.589 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.590 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:25:44.590 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.914461+0000", 2024-09-13T11:25:44.590 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.7691000000000001e-05 2024-09-13T11:25:44.590 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.590 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.590 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:25:44.590 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.914471+0000", 2024-09-13T11:25:44.590 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0447e-05 2024-09-13T11:25:44.590 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.590 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.590 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:25:44.590 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.914550+0000", 2024-09-13T11:25:44.590 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.8683000000000005e-05 2024-09-13T11:25:44.590 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.590 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.591 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:25:44.591 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.914709+0000", 2024-09-13T11:25:44.591 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00015913200000000001 2024-09-13T11:25:44.591 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.591 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.591 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 3", 2024-09-13T11:25:44.591 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.914880+0000", 2024-09-13T11:25:44.591 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00017071499999999999 2024-09-13T11:25:44.591 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.591 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.591 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:25:44.591 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.916701+0000", 2024-09-13T11:25:44.591 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0018205369999999999 2024-09-13T11:25:44.592 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.592 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.592 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:25:44.592 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:06.186401+0000", 2024-09-13T11:25:44.592 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.26970018699999998 2024-09-13T11:25:44.592 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.592 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.592 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:25:44.592 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:06.186434+0000", 2024-09-13T11:25:44.592 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.3448999999999998e-05 2024-09-13T11:25:44.592 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.592 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.592 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:25:44.593 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:06.186463+0000", 2024-09-13T11:25:44.593 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.8445e-05 2024-09-13T11:25:44.593 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:44.593 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:25:44.593 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:44.593 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.593 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.593 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop_reply(client.4665.0:84809 3.17 e107/85 ondisk, result = 0)", 2024-09-13T11:25:44.593 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:24:05.916779+0000", 2024-09-13T11:25:44.593 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 98.564705653999994, 2024-09-13T11:25:44.593 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.26968993499999999, 2024-09-13T11:25:44.593 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:25:44.593 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:25:44.593 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:25:44.593 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:25:44.594 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.594 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:25:44.594 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.916779+0000", 2024-09-13T11:25:44.594 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:44.594 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.594 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.594 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:25:44.594 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.916779+0000", 2024-09-13T11:25:44.594 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:44.594 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.594 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.594 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:25:44.594 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.916780+0000", 2024-09-13T11:25:44.595 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.328e-06 2024-09-13T11:25:44.595 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.595 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.595 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:25:44.595 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.916789+0000", 2024-09-13T11:25:44.595 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.4460000000000001e-06 2024-09-13T11:25:44.595 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.595 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.595 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:25:44.595 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:06.186296+0000", 2024-09-13T11:25:44.595 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.26950707099999999 2024-09-13T11:25:44.595 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.595 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.595 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:25:44.596 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:06.186306+0000", 2024-09-13T11:25:44.596 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0137e-05 2024-09-13T11:25:44.596 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.596 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.596 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:25:44.596 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:06.186388+0000", 2024-09-13T11:25:44.596 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.2149000000000002e-05 2024-09-13T11:25:44.596 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.596 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.596 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:25:44.596 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:06.186397+0000", 2024-09-13T11:25:44.596 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.5210000000000002e-06 2024-09-13T11:25:44.596 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.596 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.597 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:25:44.597 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:06.186469+0000", 2024-09-13T11:25:44.597 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.2282999999999999e-05 2024-09-13T11:25:44.597 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:44.597 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:25:44.597 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:44.597 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.597 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.597 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPush(3.2a 121/120 [PushOp(3:5595108b:::benchmark_data_smithi177_31116_object899:head, version: 43'62, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:5595108b:::benchmark_data_smithi177_31116_object899:head@43'62, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:25:44.597 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:24:27.352355+0000", 2024-09-13T11:25:44.597 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 77.129129848000005, 2024-09-13T11:25:44.597 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.61628666099999996, 2024-09-13T11:25:44.597 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:25:44.598 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:25:44.598 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:25:44.598 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:25:44.598 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.598 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:25:44.598 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:27.352355+0000", 2024-09-13T11:25:44.598 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:44.598 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.598 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.598 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:25:44.598 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:27.352355+0000", 2024-09-13T11:25:44.598 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:44.598 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.598 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.598 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:25:44.599 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:27.352357+0000", 2024-09-13T11:25:44.599 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.328e-06 2024-09-13T11:25:44.599 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.599 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.599 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:25:44.599 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:27.352428+0000", 2024-09-13T11:25:44.599 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.1118999999999995e-05 2024-09-13T11:25:44.599 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.599 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.599 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:25:44.599 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:27.966950+0000", 2024-09-13T11:25:44.599 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.61452202499999997 2024-09-13T11:25:44.599 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.600 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.600 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:25:44.600 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:27.966980+0000", 2024-09-13T11:25:44.600 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.9433000000000001e-05 2024-09-13T11:25:44.600 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.600 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.600 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:25:44.600 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:27.967220+0000", 2024-09-13T11:25:44.600 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00024065499999999999 2024-09-13T11:25:44.600 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.600 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.600 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:25:44.600 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:27.967244+0000", 2024-09-13T11:25:44.600 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.4216e-05 2024-09-13T11:25:44.601 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.601 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.601 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:25:44.601 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:27.968641+0000", 2024-09-13T11:25:44.601 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0013968850000000001 2024-09-13T11:25:44.601 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:44.601 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:25:44.601 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:44.601 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.601 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.601 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:162057 3.1f 3:f8710094:::benchmark_data_smithi177_31116_object21754:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e156)", 2024-09-13T11:25:44.601 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:25:15.442781+0000", 2024-09-13T11:25:44.601 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 29.038703205000001, 2024-09-13T11:25:44.601 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.25670282500000002, 2024-09-13T11:25:44.602 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:25:44.602 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:25:44.602 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:44.602 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:25:44.602 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:25:44.602 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:44.602 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 162057 2024-09-13T11:25:44.602 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.602 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:25:44.602 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.602 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:25:44.602 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:15.442781+0000", 2024-09-13T11:25:44.602 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:44.602 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.602 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.603 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:25:44.603 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:15.442781+0000", 2024-09-13T11:25:44.603 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:44.603 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.603 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.603 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:25:44.603 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:15.442785+0000", 2024-09-13T11:25:44.603 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.4000000000000001e-06 2024-09-13T11:25:44.603 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.603 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.603 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:25:44.603 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:15.442795+0000", 2024-09-13T11:25:44.603 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0193999999999999e-05 2024-09-13T11:25:44.603 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.604 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.604 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:25:44.604 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:15.442820+0000", 2024-09-13T11:25:44.604 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.4683000000000001e-05 2024-09-13T11:25:44.604 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.604 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.604 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:25:44.604 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:15.442840+0000", 2024-09-13T11:25:44.604 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.0596000000000001e-05 2024-09-13T11:25:44.604 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.604 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.604 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:25:44.604 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:15.442952+0000", 2024-09-13T11:25:44.604 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00011215899999999999 2024-09-13T11:25:44.604 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.605 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.605 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:25:44.605 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:15.443143+0000", 2024-09-13T11:25:44.605 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00019058899999999999 2024-09-13T11:25:44.605 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.605 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.605 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 5", 2024-09-13T11:25:44.605 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:15.443327+0000", 2024-09-13T11:25:44.605 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00018364499999999999 2024-09-13T11:25:44.605 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.605 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.605 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:25:44.605 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:15.446727+0000", 2024-09-13T11:25:44.605 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0034000689999999999 2024-09-13T11:25:44.606 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.606 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.606 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:25:44.606 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:15.699356+0000", 2024-09-13T11:25:44.606 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.25262958400000002 2024-09-13T11:25:44.606 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.606 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.606 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:25:44.606 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:15.699428+0000", 2024-09-13T11:25:44.606 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.1810000000000005e-05 2024-09-13T11:25:44.606 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.606 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.606 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:25:44.606 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:15.699484+0000", 2024-09-13T11:25:44.606 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.6096e-05 2024-09-13T11:25:44.607 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:44.607 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:25:44.607 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:44.607 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.607 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.607 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:162073 3.1f 3:fd1a00ae:::benchmark_data_smithi177_31116_object21770:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e156)", 2024-09-13T11:25:44.607 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:25:15.454353+0000", 2024-09-13T11:25:44.607 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 29.02713125, 2024-09-13T11:25:44.607 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.27516647, 2024-09-13T11:25:44.607 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:25:44.607 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:25:44.607 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:44.607 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:25:44.608 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:25:44.608 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:44.608 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 162073 2024-09-13T11:25:44.608 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.608 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:25:44.608 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.608 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:25:44.608 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:15.454353+0000", 2024-09-13T11:25:44.608 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:44.608 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.608 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.608 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:25:44.608 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:15.454353+0000", 2024-09-13T11:25:44.608 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:44.609 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.609 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.609 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:25:44.609 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:15.454358+0000", 2024-09-13T11:25:44.609 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.3499999999999999e-06 2024-09-13T11:25:44.609 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.609 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.609 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:25:44.609 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:15.454371+0000", 2024-09-13T11:25:44.609 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3594000000000001e-05 2024-09-13T11:25:44.609 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.609 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.609 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:25:44.610 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:15.454408+0000", 2024-09-13T11:25:44.610 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.6619999999999998e-05 2024-09-13T11:25:44.610 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.610 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.610 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:25:44.610 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:15.454431+0000", 2024-09-13T11:25:44.610 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.3546000000000001e-05 2024-09-13T11:25:44.610 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.610 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.610 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:25:44.610 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:15.454587+0000", 2024-09-13T11:25:44.610 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000155248 2024-09-13T11:25:44.610 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.610 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.611 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:25:44.611 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:15.454897+0000", 2024-09-13T11:25:44.611 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000309955 2024-09-13T11:25:44.611 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.611 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.611 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 5", 2024-09-13T11:25:44.611 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:15.455203+0000", 2024-09-13T11:25:44.611 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00030604999999999999 2024-09-13T11:25:44.611 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.611 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.611 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:25:44.611 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:15.460534+0000", 2024-09-13T11:25:44.611 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0053310199999999997 2024-09-13T11:25:44.611 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.611 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.612 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:25:44.612 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:15.729356+0000", 2024-09-13T11:25:44.612 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.26882267199999998 2024-09-13T11:25:44.612 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.612 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.612 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:25:44.612 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:15.729442+0000", 2024-09-13T11:25:44.612 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.5209000000000004e-05 2024-09-13T11:25:44.612 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.612 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.612 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:25:44.612 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:15.729520+0000", 2024-09-13T11:25:44.612 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.8206000000000003e-05 2024-09-13T11:25:44.612 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:44.613 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:25:44.613 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:44.613 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.613 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.613 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:162102 3.1f 3:fc0cda6c:::benchmark_data_smithi177_31116_object21799:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e156)", 2024-09-13T11:25:44.613 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:25:15.477044+0000", 2024-09-13T11:25:44.613 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 29.004440396, 2024-09-13T11:25:44.613 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.28798895800000002, 2024-09-13T11:25:44.613 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:25:44.613 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:25:44.613 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:44.613 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:25:44.613 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:25:44.614 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:44.614 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 162102 2024-09-13T11:25:44.614 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.614 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:25:44.614 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.614 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:25:44.614 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:15.477044+0000", 2024-09-13T11:25:44.614 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:44.614 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.614 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.614 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:25:44.614 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:15.477044+0000", 2024-09-13T11:25:44.614 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:44.614 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.614 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.615 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:25:44.615 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:15.477048+0000", 2024-09-13T11:25:44.615 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.3569999999999998e-06 2024-09-13T11:25:44.615 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.615 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.615 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:25:44.615 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:15.477058+0000", 2024-09-13T11:25:44.615 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.9899999999999992e-06 2024-09-13T11:25:44.615 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.615 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.615 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:25:44.615 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:15.477087+0000", 2024-09-13T11:25:44.615 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.9026999999999998e-05 2024-09-13T11:25:44.615 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.616 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.616 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:25:44.616 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:15.477101+0000", 2024-09-13T11:25:44.616 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4583000000000001e-05 2024-09-13T11:25:44.616 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.616 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.616 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:25:44.616 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:15.477212+0000", 2024-09-13T11:25:44.616 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00011071200000000001 2024-09-13T11:25:44.616 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.616 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.616 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:25:44.616 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:15.477492+0000", 2024-09-13T11:25:44.617 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00027995400000000001 2024-09-13T11:25:44.617 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.617 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.617 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 5", 2024-09-13T11:25:44.617 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:15.477682+0000", 2024-09-13T11:25:44.617 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000189929 2024-09-13T11:25:44.617 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.617 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.617 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:25:44.617 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:15.481903+0000", 2024-09-13T11:25:44.617 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0042214649999999998 2024-09-13T11:25:44.618 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.618 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.618 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:25:44.618 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:15.764891+0000", 2024-09-13T11:25:44.618 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.28298753599999998 2024-09-13T11:25:44.618 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.618 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.618 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:25:44.618 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:15.764972+0000", 2024-09-13T11:25:44.618 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.1230000000000007e-05 2024-09-13T11:25:44.618 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.618 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.618 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:25:44.619 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:15.765033+0000", 2024-09-13T11:25:44.619 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.1174999999999996e-05 2024-09-13T11:25:44.619 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:44.619 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:25:44.619 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:44.619 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.619 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.619 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop_reply(client.4665.0:162057 3.1f e156/122 ondisk, result = 0)", 2024-09-13T11:25:44.619 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:25:15.479734+0000", 2024-09-13T11:25:44.619 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 29.001750298000001, 2024-09-13T11:25:44.619 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.21975831800000001, 2024-09-13T11:25:44.619 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:25:44.619 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:25:44.620 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:25:44.620 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:25:44.620 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.620 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:25:44.620 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:15.479734+0000", 2024-09-13T11:25:44.620 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:44.620 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.620 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.620 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:25:44.620 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:15.479734+0000", 2024-09-13T11:25:44.620 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:44.620 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.620 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.621 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:25:44.621 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:15.479745+0000", 2024-09-13T11:25:44.621 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.084e-05 2024-09-13T11:25:44.621 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.621 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.621 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:25:44.621 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:15.479757+0000", 2024-09-13T11:25:44.621 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1600000000000001e-05 2024-09-13T11:25:44.621 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.621 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.621 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:25:44.621 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:15.698724+0000", 2024-09-13T11:25:44.621 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.218967044 2024-09-13T11:25:44.622 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.622 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.622 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:25:44.622 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:15.698740+0000", 2024-09-13T11:25:44.622 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.6707000000000001e-05 2024-09-13T11:25:44.622 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.622 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.622 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:25:44.622 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:15.699336+0000", 2024-09-13T11:25:44.622 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00059547500000000004 2024-09-13T11:25:44.624 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.624 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.624 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:25:44.624 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:15.699348+0000", 2024-09-13T11:25:44.624 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1703e-05 2024-09-13T11:25:44.624 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.624 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.624 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:25:44.624 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:15.699493+0000", 2024-09-13T11:25:44.625 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00014494900000000001 2024-09-13T11:25:44.625 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:44.625 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:25:44.625 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:44.625 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.625 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.625 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:191286 3.18 3:1ffc0b26:::benchmark_data_smithi177_31116_object50983:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e175)", 2024-09-13T11:25:44.625 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:25:39.152634+0000", 2024-09-13T11:25:44.625 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 5.3288502150000001, 2024-09-13T11:25:44.625 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.236097576, 2024-09-13T11:25:44.625 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:25:44.625 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:25:44.626 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:44.626 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:25:44.626 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:25:44.626 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:44.626 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 191286 2024-09-13T11:25:44.626 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.626 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:25:44.626 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.626 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:25:44.626 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:39.152634+0000", 2024-09-13T11:25:44.626 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:44.626 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.627 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.627 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:25:44.627 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:39.152634+0000", 2024-09-13T11:25:44.627 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:44.627 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.627 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.627 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:25:44.627 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:39.152638+0000", 2024-09-13T11:25:44.627 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.5130000000000001e-06 2024-09-13T11:25:44.627 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.627 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.627 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:25:44.627 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:39.152649+0000", 2024-09-13T11:25:44.628 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0677e-05 2024-09-13T11:25:44.628 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.628 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.628 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:25:44.628 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:39.152678+0000", 2024-09-13T11:25:44.628 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.9436000000000001e-05 2024-09-13T11:25:44.628 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.628 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.628 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:25:44.628 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:39.152693+0000", 2024-09-13T11:25:44.628 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4823e-05 2024-09-13T11:25:44.628 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.628 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.628 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:25:44.629 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:39.152821+0000", 2024-09-13T11:25:44.629 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00012815299999999999 2024-09-13T11:25:44.629 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.629 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.629 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:25:44.629 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:39.153031+0000", 2024-09-13T11:25:44.629 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00021011399999999999 2024-09-13T11:25:44.629 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.629 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.629 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 0", 2024-09-13T11:25:44.629 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:39.153195+0000", 2024-09-13T11:25:44.629 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00016374599999999999 2024-09-13T11:25:44.629 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.630 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.630 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:25:44.630 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:39.157900+0000", 2024-09-13T11:25:44.630 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0047053049999999999 2024-09-13T11:25:44.630 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.630 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.630 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:25:44.630 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:39.388451+0000", 2024-09-13T11:25:44.630 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.230550957 2024-09-13T11:25:44.630 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.630 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.630 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:25:44.630 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:39.388611+0000", 2024-09-13T11:25:44.631 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00016029299999999999 2024-09-13T11:25:44.631 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:44.631 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:44.631 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:25:44.631 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:39.388732+0000", 2024-09-13T11:25:44.631 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000120559 2024-09-13T11:25:44.631 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:44.631 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:25:44.631 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:44.631 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:44.631 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:25:44.631 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-13T11:25:44.633 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-13T11:25:44.634 INFO:tasks.ceph.osd.13.smithi092.stderr:2024-09-13T11:25:44.633+0000 7f3e5afea640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26350) UID: 0 2024-09-13T11:25:44.634 INFO:teuthology.orchestra.run.smithi001.stderr:no valid command found; 10 closest matches: 2024-09-13T11:25:44.634 INFO:teuthology.orchestra.run.smithi001.stderr:0 2024-09-13T11:25:44.634 INFO:teuthology.orchestra.run.smithi001.stderr:1 2024-09-13T11:25:44.635 INFO:teuthology.orchestra.run.smithi001.stderr:2 2024-09-13T11:25:44.635 INFO:teuthology.orchestra.run.smithi001.stderr:abort 2024-09-13T11:25:44.635 INFO:teuthology.orchestra.run.smithi001.stderr:assert 2024-09-13T11:25:44.635 INFO:teuthology.orchestra.run.smithi001.stderr:bench [] [] [] [] 2024-09-13T11:25:44.635 INFO:teuthology.orchestra.run.smithi001.stderr:bluefs debug_inject_read_zeros 2024-09-13T11:25:44.635 INFO:teuthology.orchestra.run.smithi001.stderr:bluefs files list 2024-09-13T11:25:44.635 INFO:teuthology.orchestra.run.smithi001.stderr:bluefs stats 2024-09-13T11:25:44.635 INFO:teuthology.orchestra.run.smithi001.stderr:bluestore allocator dump block 2024-09-13T11:25:44.635 INFO:teuthology.orchestra.run.smithi001.stderr:Error EINVAL: invalid command 2024-09-13T11:25:44.636 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-13T11:25:44.637 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-13T11:25:44.637 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 'trim stale osdmaps' 2024-09-13T11:25:44.734 INFO:tasks.ceph.osd.2.smithi162.stderr:2024-09-13T11:25:44.732+0000 7fe92b35a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26313) UID: 0 2024-09-13T11:25:44.781 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-09-13T11:25:44.781 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [ 2024-09-13T11:25:44.781 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:44.781 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:197664 3.14 3:2ffeeb4c:::benchmark_data_smithi177_31116_object57361:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e179)", 2024-09-13T11:25:44.781 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:25:44.765356+0000", 2024-09-13T11:25:44.781 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 0.01594949, 2024-09-13T11:25:44.781 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.015983758000000001, 2024-09-13T11:25:44.781 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:25:44.782 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:25:44.782 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "waiting for sub ops", 2024-09-13T11:25:44.782 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:25:44.782 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:25:44.782 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:44.782 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 197664 2024-09-13T11:25:44.782 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:44.782 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:25:44.782 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:44.782 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:25:44.782 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:44.765356+0000", 2024-09-13T11:25:44.782 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:44.782 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:44.783 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:44.783 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:25:44.783 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:44.765356+0000", 2024-09-13T11:25:44.783 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:44.783 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:44.783 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:44.783 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:25:44.783 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:44.765358+0000", 2024-09-13T11:25:44.783 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.4530000000000001e-06 2024-09-13T11:25:44.783 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:44.783 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:44.783 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:25:44.783 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:44.765365+0000", 2024-09-13T11:25:44.784 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.1060000000000002e-06 2024-09-13T11:25:44.784 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:44.784 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:44.784 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:25:44.784 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:44.765383+0000", 2024-09-13T11:25:44.784 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.7306e-05 2024-09-13T11:25:44.784 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:44.784 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:44.784 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:25:44.784 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:44.765393+0000", 2024-09-13T11:25:44.784 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0276e-05 2024-09-13T11:25:44.784 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:44.784 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:44.785 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:25:44.785 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:44.765456+0000", 2024-09-13T11:25:44.785 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.3417999999999995e-05 2024-09-13T11:25:44.785 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:44.785 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:44.785 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:25:44.785 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:44.765597+0000", 2024-09-13T11:25:44.785 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00014018799999999999 2024-09-13T11:25:44.785 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:44.785 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:44.785 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 15", 2024-09-13T11:25:44.785 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:44.765722+0000", 2024-09-13T11:25:44.785 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00012528499999999999 2024-09-13T11:25:44.786 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:44.786 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:44.786 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:25:44.786 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:44.768654+0000", 2024-09-13T11:25:44.786 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.002931725 2024-09-13T11:25:44.786 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:44.786 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:25:44.786 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:44.786 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:44.786 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:44.786 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:197677 3.14 3:2caa43aa:::benchmark_data_smithi177_31116_object57374:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e179)", 2024-09-13T11:25:44.786 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:25:44.777442+0000", 2024-09-13T11:25:44.786 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 0.003862889, 2024-09-13T11:25:44.786 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.003975396, 2024-09-13T11:25:44.787 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:25:44.788 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:25:44.788 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "waiting for sub ops", 2024-09-13T11:25:44.788 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:25:44.788 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:25:44.788 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:44.789 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 197677 2024-09-13T11:25:44.789 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:44.790 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:25:44.790 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:44.790 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:25:44.790 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:44.777442+0000", 2024-09-13T11:25:44.790 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:44.790 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:44.790 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:44.790 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:25:44.790 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:44.777442+0000", 2024-09-13T11:25:44.790 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:44.790 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:44.790 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:44.790 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:25:44.791 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:44.777445+0000", 2024-09-13T11:25:44.791 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.813e-06 2024-09-13T11:25:44.791 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:44.791 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:44.791 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:25:44.791 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:44.777453+0000", 2024-09-13T11:25:44.791 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.2930000000000003e-06 2024-09-13T11:25:44.791 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:44.791 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:44.791 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:25:44.791 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:44.777469+0000", 2024-09-13T11:25:44.791 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.6614000000000001e-05 2024-09-13T11:25:44.791 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:44.791 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:44.792 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:25:44.792 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:44.777479+0000", 2024-09-13T11:25:44.792 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.7280000000000007e-06 2024-09-13T11:25:44.792 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:44.792 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:44.792 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:25:44.792 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:44.777557+0000", 2024-09-13T11:25:44.792 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.8541999999999994e-05 2024-09-13T11:25:44.792 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:44.792 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:44.792 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:25:44.792 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:44.777747+0000", 2024-09-13T11:25:44.792 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00018976099999999999 2024-09-13T11:25:44.793 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:44.793 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:44.793 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 15", 2024-09-13T11:25:44.793 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:44.777911+0000", 2024-09-13T11:25:44.793 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00016429199999999999 2024-09-13T11:25:44.793 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:44.793 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:25:44.793 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:44.793 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:44.793 INFO:teuthology.orchestra.run.smithi177.stdout: ], 2024-09-13T11:25:44.793 INFO:teuthology.orchestra.run.smithi177.stdout: "num_ops": 2 2024-09-13T11:25:44.793 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-09-13T11:25:44.800 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-13T11:25:44.810 INFO:teuthology.orchestra.run.smithi001.stderr:no valid command found; 10 closest matches: 2024-09-13T11:25:44.810 INFO:teuthology.orchestra.run.smithi001.stderr:0 2024-09-13T11:25:44.811 INFO:teuthology.orchestra.run.smithi001.stderr:1 2024-09-13T11:25:44.811 INFO:teuthology.orchestra.run.smithi001.stderr:2 2024-09-13T11:25:44.811 INFO:teuthology.orchestra.run.smithi001.stderr:abort 2024-09-13T11:25:44.811 INFO:teuthology.orchestra.run.smithi001.stderr:assert 2024-09-13T11:25:44.811 INFO:teuthology.orchestra.run.smithi001.stderr:bench [] [] [] [] 2024-09-13T11:25:44.811 INFO:teuthology.orchestra.run.smithi001.stderr:bluefs debug_inject_read_zeros 2024-09-13T11:25:44.811 INFO:teuthology.orchestra.run.smithi001.stderr:bluefs files list 2024-09-13T11:25:44.811 INFO:teuthology.orchestra.run.smithi001.stderr:bluefs stats 2024-09-13T11:25:44.811 INFO:teuthology.orchestra.run.smithi001.stderr:bluestore allocator dump block 2024-09-13T11:25:44.811 INFO:teuthology.orchestra.run.smithi001.stderr:Error EINVAL: invalid command 2024-09-13T11:25:44.813 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-13T11:25:44.813 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-13T11:25:44.814 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 'trim stale osdmaps' 2024-09-13T11:25:44.847 INFO:tasks.ceph.osd.7.smithi177.stderr:2024-09-13T11:25:44.846+0000 7f897b51d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26575) UID: 0 2024-09-13T11:25:44.902 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T11:25:44.935 INFO:tasks.ceph.osd.10.smithi162.stderr:2024-09-13T11:25:44.932+0000 7fb59c9f0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26317) UID: 0 2024-09-13T11:25:44.939 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-09-13T11:25:44.939 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [], 2024-09-13T11:25:44.939 INFO:teuthology.orchestra.run.smithi177.stdout: "complaint_time": 30, 2024-09-13T11:25:44.939 INFO:teuthology.orchestra.run.smithi177.stdout: "num_blocked_ops": 0 2024-09-13T11:25:44.940 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-09-13T11:25:44.957 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-13T11:25:44.984 INFO:teuthology.orchestra.run.smithi001.stderr:no valid command found; 10 closest matches: 2024-09-13T11:25:44.984 INFO:teuthology.orchestra.run.smithi001.stderr:0 2024-09-13T11:25:44.984 INFO:teuthology.orchestra.run.smithi001.stderr:1 2024-09-13T11:25:44.985 INFO:teuthology.orchestra.run.smithi001.stderr:2 2024-09-13T11:25:44.985 INFO:teuthology.orchestra.run.smithi001.stderr:abort 2024-09-13T11:25:44.985 INFO:teuthology.orchestra.run.smithi001.stderr:assert 2024-09-13T11:25:44.985 INFO:teuthology.orchestra.run.smithi001.stderr:bench [] [] [] [] 2024-09-13T11:25:44.985 INFO:teuthology.orchestra.run.smithi001.stderr:bluefs debug_inject_read_zeros 2024-09-13T11:25:44.985 INFO:teuthology.orchestra.run.smithi001.stderr:bluefs files list 2024-09-13T11:25:44.985 INFO:teuthology.orchestra.run.smithi001.stderr:bluefs stats 2024-09-13T11:25:44.985 INFO:teuthology.orchestra.run.smithi001.stderr:bluestore allocator dump block 2024-09-13T11:25:44.985 INFO:teuthology.orchestra.run.smithi001.stderr:Error EINVAL: invalid command 2024-09-13T11:25:44.989 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-13T11:25:44.989 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-13T11:25:44.989 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 'trim stale osdmaps' 2024-09-13T11:25:45.036 INFO:tasks.ceph.osd.13.smithi092.stderr:2024-09-13T11:25:45.033+0000 7f3e5afea640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26350) UID: 0 2024-09-13T11:25:45.071 INFO:teuthology.orchestra.run.smithi001.stdout:osd.0: {} 2024-09-13T11:25:45.071 INFO:teuthology.orchestra.run.smithi001.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T11:25:45.110 INFO:teuthology.orchestra.run.smithi001.stdout:osd.1: {} 2024-09-13T11:25:45.110 INFO:teuthology.orchestra.run.smithi001.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T11:25:45.137 INFO:tasks.ceph.osd.11.smithi177.stderr:2024-09-13T11:25:45.134+0000 7f24792c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26580) UID: 0 2024-09-13T11:25:45.138 INFO:teuthology.orchestra.run.smithi001.stdout:osd.2: {} 2024-09-13T11:25:45.138 INFO:teuthology.orchestra.run.smithi001.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T11:25:45.142 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-09-13T11:25:45.142 INFO:teuthology.orchestra.run.smithi177.stdout: "size": 20, 2024-09-13T11:25:45.142 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 600, 2024-09-13T11:25:45.142 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [ 2024-09-13T11:25:45.142 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:45.142 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPush(3.17 75/70 [PushOp(3:eda94455:::benchmark_data_smithi177_31116_object13017:head, version: 50'788, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:eda94455:::benchmark_data_smithi177_31116_object13017:head@50'788, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:25:45.143 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:31.038029+0000", 2024-09-13T11:25:45.143 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 134.102739383, 2024-09-13T11:25:45.143 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.42064623499999998, 2024-09-13T11:25:45.143 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:25:45.143 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:25:45.143 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:25:45.143 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:25:45.143 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:45.143 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:25:45.143 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.038029+0000", 2024-09-13T11:25:45.143 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:45.143 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:45.143 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:45.143 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:25:45.144 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.038029+0000", 2024-09-13T11:25:45.144 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:45.144 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:45.144 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:45.144 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:25:45.144 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.038030+0000", 2024-09-13T11:25:45.144 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.499e-06 2024-09-13T11:25:45.144 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:45.144 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:45.144 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:25:45.144 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.038088+0000", 2024-09-13T11:25:45.144 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.7771e-05 2024-09-13T11:25:45.144 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:45.145 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:45.145 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:25:45.145 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.457025+0000", 2024-09-13T11:25:45.145 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.41893754900000002 2024-09-13T11:25:45.145 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:45.145 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:45.145 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:25:45.145 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.457073+0000", 2024-09-13T11:25:45.145 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.7450999999999997e-05 2024-09-13T11:25:45.145 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:45.145 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:45.145 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:25:45.145 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.457304+0000", 2024-09-13T11:25:45.146 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000231351 2024-09-13T11:25:45.146 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:45.146 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:45.146 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:25:45.146 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.457331+0000", 2024-09-13T11:25:45.146 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.6372000000000001e-05 2024-09-13T11:25:45.146 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:45.146 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:45.146 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:25:45.146 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.458675+0000", 2024-09-13T11:25:45.146 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001344242 2024-09-13T11:25:45.146 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:45.146 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:25:45.146 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:45.147 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:45.147 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:45.147 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPush(3.17 83/70 [PushOp(3:eb574268:::benchmark_data_smithi177_31116_object40622:head, version: 64'2475, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:eb574268:::benchmark_data_smithi177_31116_object40622:head@64'2475, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:25:45.147 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:40.760042+0000", 2024-09-13T11:25:45.147 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 124.380725666, 2024-09-13T11:25:45.147 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.30544642100000002, 2024-09-13T11:25:45.147 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:25:45.147 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:25:45.147 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:25:45.147 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:25:45.147 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:45.147 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:25:45.147 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:40.760042+0000", 2024-09-13T11:25:45.147 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:45.148 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:45.148 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:45.148 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:25:45.148 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:40.760042+0000", 2024-09-13T11:25:45.148 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:45.148 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:45.148 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:45.148 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:25:45.148 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:40.760044+0000", 2024-09-13T11:25:45.148 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.511e-06 2024-09-13T11:25:45.148 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:45.148 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:45.148 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:25:45.149 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:40.760149+0000", 2024-09-13T11:25:45.149 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000105473 2024-09-13T11:25:45.149 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:45.149 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:45.149 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:25:45.149 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.064465+0000", 2024-09-13T11:25:45.149 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.30431613800000001 2024-09-13T11:25:45.149 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:45.149 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:45.149 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:25:45.149 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.064479+0000", 2024-09-13T11:25:45.149 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3586e-05 2024-09-13T11:25:45.149 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:45.149 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:45.150 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:25:45.150 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.064553+0000", 2024-09-13T11:25:45.150 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.3646000000000006e-05 2024-09-13T11:25:45.150 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:45.150 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:45.150 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:25:45.150 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.064568+0000", 2024-09-13T11:25:45.150 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.52e-05 2024-09-13T11:25:45.150 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:45.150 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:45.150 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:25:45.150 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.065489+0000", 2024-09-13T11:25:45.150 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00092086700000000002 2024-09-13T11:25:45.150 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:45.150 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:25:45.151 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:45.151 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:45.151 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:45.151 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPushReply(3.1a 83/78 [PushReplyOp(3:5c3834bd:::benchmark_data_smithi177_31116_object8669:head)])", 2024-09-13T11:25:45.151 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:40.798236+0000", 2024-09-13T11:25:45.151 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 124.342532236, 2024-09-13T11:25:45.151 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.29701520999999997, 2024-09-13T11:25:45.151 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:25:45.151 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:25:45.151 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-13T11:25:45.151 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:25:45.151 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:45.151 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:25:45.152 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:40.798236+0000", 2024-09-13T11:25:45.152 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:45.152 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:45.152 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:45.152 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:25:45.152 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:40.798236+0000", 2024-09-13T11:25:45.152 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:45.152 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:45.152 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:45.152 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:25:45.152 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:40.798237+0000", 2024-09-13T11:25:45.152 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.0500000000000002e-07 2024-09-13T11:25:45.152 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:45.152 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:45.153 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:25:45.153 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:40.798240+0000", 2024-09-13T11:25:45.153 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.4819999999999999e-06 2024-09-13T11:25:45.153 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:45.153 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:45.153 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:25:45.153 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.093360+0000", 2024-09-13T11:25:45.153 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.29512009 2024-09-13T11:25:45.153 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:45.153 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:45.153 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:25:45.153 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.093372+0000", 2024-09-13T11:25:45.153 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.2221e-05 2024-09-13T11:25:45.153 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:45.154 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:45.154 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:25:45.154 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.093482+0000", 2024-09-13T11:25:45.154 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000109571 2024-09-13T11:25:45.154 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:45.154 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:45.154 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:25:45.154 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.095251+0000", 2024-09-13T11:25:45.154 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001768941 2024-09-13T11:25:45.154 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:45.154 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:25:45.154 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:45.154 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:45.154 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:45.154 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPushReply(3.1a 83/78 [PushReplyOp(3:58c7ffc0:::benchmark_data_smithi177_31116_object8738:head)])", 2024-09-13T11:25:45.155 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:40.805941+0000", 2024-09-13T11:25:45.155 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 124.334826982, 2024-09-13T11:25:45.155 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.30315678499999998, 2024-09-13T11:25:45.155 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:25:45.155 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:25:45.155 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-13T11:25:45.155 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:25:45.155 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:45.155 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:25:45.155 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:40.805941+0000", 2024-09-13T11:25:45.155 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:45.155 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:45.155 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:45.156 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:25:45.156 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:40.805941+0000", 2024-09-13T11:25:45.156 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:45.156 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:45.156 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:45.156 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:25:45.156 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:40.805942+0000", 2024-09-13T11:25:45.156 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3769999999999999e-06 2024-09-13T11:25:45.156 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:45.156 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:45.156 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:25:45.156 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:40.805947+0000", 2024-09-13T11:25:45.156 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.8980000000000004e-06 2024-09-13T11:25:45.157 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:45.157 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:45.157 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:25:45.157 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.106505+0000", 2024-09-13T11:25:45.157 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.30055777700000003 2024-09-13T11:25:45.157 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:45.157 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:45.157 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:25:45.157 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.106512+0000", 2024-09-13T11:25:45.157 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.3749999999999997e-06 2024-09-13T11:25:45.157 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:45.157 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:45.157 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:25:45.157 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.106566+0000", 2024-09-13T11:25:45.158 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.3103999999999998e-05 2024-09-13T11:25:45.158 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:45.158 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:45.158 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:25:45.158 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.109098+0000", 2024-09-13T11:25:45.158 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0025322539999999998 2024-09-13T11:25:45.158 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:45.158 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:25:45.158 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:45.158 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:45.158 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:45.158 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPushReply(3.1a 83/78 [PushReplyOp(3:58c7ffc0:::benchmark_data_smithi177_31116_object8738:head)])", 2024-09-13T11:25:45.158 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:40.805973+0000", 2024-09-13T11:25:45.158 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 124.33479497899999, 2024-09-13T11:25:45.159 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.28729672099999998, 2024-09-13T11:25:45.159 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:25:45.159 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:25:45.159 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-13T11:25:45.159 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:25:45.159 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:45.159 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:25:45.159 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:40.805973+0000", 2024-09-13T11:25:45.159 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:45.159 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:45.159 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:45.159 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:25:45.159 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:40.805973+0000", 2024-09-13T11:25:45.159 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:45.160 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:45.160 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:45.160 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:25:45.160 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:40.805974+0000", 2024-09-13T11:25:45.160 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.8599999999999998e-07 2024-09-13T11:25:45.160 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:45.160 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:45.160 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:25:45.160 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:40.805976+0000", 2024-09-13T11:25:45.160 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.2809999999999998e-06 2024-09-13T11:25:45.160 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:45.160 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:45.160 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:25:45.161 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.091588+0000", 2024-09-13T11:25:45.161 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.28561173899999998 2024-09-13T11:25:45.161 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:45.161 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:45.161 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:25:45.161 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.091596+0000", 2024-09-13T11:25:45.161 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.0520000000000002e-06 2024-09-13T11:25:45.161 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:45.161 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:45.161 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:25:45.161 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.091907+0000", 2024-09-13T11:25:45.161 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00031179599999999998 2024-09-13T11:25:45.161 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:45.161 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:45.162 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:25:45.162 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.093270+0000", 2024-09-13T11:25:45.162 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0013622669999999999 2024-09-13T11:25:45.162 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:45.162 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:25:45.162 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:45.162 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:45.162 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:45.162 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPushReply(3.1a 84/78 [PushReplyOp(3:581ea97e:::benchmark_data_smithi177_31116_object10255:head)])", 2024-09-13T11:25:45.162 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:41.514228+0000", 2024-09-13T11:25:45.162 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 123.626539491, 2024-09-13T11:25:45.162 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.40695564200000001, 2024-09-13T11:25:45.162 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:25:45.163 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:25:45.163 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-13T11:25:45.163 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:25:45.163 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:45.163 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:25:45.163 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.514228+0000", 2024-09-13T11:25:45.163 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:45.163 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:45.163 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:45.163 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:25:45.163 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.514228+0000", 2024-09-13T11:25:45.163 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:45.163 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:45.163 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:45.164 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:25:45.164 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.514230+0000", 2024-09-13T11:25:45.164 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.499e-06 2024-09-13T11:25:45.164 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:45.164 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:45.164 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:25:45.164 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.514236+0000", 2024-09-13T11:25:45.164 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.5389999999999998e-06 2024-09-13T11:25:45.164 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:45.164 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:45.164 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:25:45.164 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.919787+0000", 2024-09-13T11:25:45.164 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.40555152999999999 2024-09-13T11:25:45.164 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:45.165 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:45.165 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:25:45.165 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.919796+0000", 2024-09-13T11:25:45.165 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.4829999999999999e-06 2024-09-13T11:25:45.165 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:45.165 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:45.165 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:25:45.165 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.920366+0000", 2024-09-13T11:25:45.165 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00057061700000000004 2024-09-13T11:25:45.165 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:45.165 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:45.165 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:25:45.165 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.921184+0000", 2024-09-13T11:25:45.165 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00081797399999999996 2024-09-13T11:25:45.166 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:45.166 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:25:45.166 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:45.166 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:45.166 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:45.166 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:60312 3.14 3:2e846241:::benchmark_data_smithi177_31116_object60311:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e85)", 2024-09-13T11:25:45.166 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:42.065725+0000", 2024-09-13T11:25:45.166 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 123.075042848, 2024-09-13T11:25:45.166 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.211323176, 2024-09-13T11:25:45.166 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:25:45.166 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:25:45.166 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:45.166 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:25:45.166 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:25:45.167 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:45.167 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 60312 2024-09-13T11:25:45.167 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:45.167 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:25:45.167 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:45.167 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:25:45.167 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.065725+0000", 2024-09-13T11:25:45.167 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:45.167 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:45.167 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:45.167 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:25:45.167 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.065725+0000", 2024-09-13T11:25:45.167 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:45.167 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:45.168 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:45.168 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:25:45.168 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.065728+0000", 2024-09-13T11:25:45.168 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.3400000000000002e-06 2024-09-13T11:25:45.168 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:45.168 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:45.168 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:25:45.168 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.065759+0000", 2024-09-13T11:25:45.168 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.0074e-05 2024-09-13T11:25:45.168 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:45.168 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:45.168 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:25:45.168 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.065771+0000", 2024-09-13T11:25:45.168 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.2869e-05 2024-09-13T11:25:45.169 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:45.169 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:45.169 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:25:45.169 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.065780+0000", 2024-09-13T11:25:45.169 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.3489999999999992e-06 2024-09-13T11:25:45.169 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:45.169 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:45.169 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:25:45.169 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.065837+0000", 2024-09-13T11:25:45.169 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.7444000000000001e-05 2024-09-13T11:25:45.169 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:45.169 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:45.169 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:25:45.169 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.065966+0000", 2024-09-13T11:25:45.170 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000128973 2024-09-13T11:25:45.170 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:45.170 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:45.170 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 14", 2024-09-13T11:25:45.170 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.066114+0000", 2024-09-13T11:25:45.170 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00014788299999999999 2024-09-13T11:25:45.170 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:45.170 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:45.170 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:25:45.170 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.067604+0000", 2024-09-13T11:25:45.170 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001489637 2024-09-13T11:25:45.170 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:45.170 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:45.170 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:25:45.170 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.276963+0000", 2024-09-13T11:25:45.171 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.20935888499999999 2024-09-13T11:25:45.171 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:45.171 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:45.171 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:25:45.171 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.277003+0000", 2024-09-13T11:25:45.171 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.0241000000000003e-05 2024-09-13T11:25:45.171 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:45.171 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:45.171 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:25:45.171 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.277048+0000", 2024-09-13T11:25:45.171 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.5481e-05 2024-09-13T11:25:45.171 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:45.171 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:25:45.171 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:45.172 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:45.172 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:45.172 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop_reply(client.4665.0:60312 3.14 e85/84 ondisk, result = 0)", 2024-09-13T11:25:45.172 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:42.069120+0000", 2024-09-13T11:25:45.172 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 123.071647728, 2024-09-13T11:25:45.172 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.20804031000000001, 2024-09-13T11:25:45.172 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:25:45.172 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:25:45.172 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:25:45.172 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:25:45.172 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:45.172 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:25:45.172 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.069120+0000", 2024-09-13T11:25:45.172 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:45.173 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:45.173 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:45.173 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:25:45.173 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.069120+0000", 2024-09-13T11:25:45.173 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:45.173 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:45.173 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:45.173 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:25:45.173 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.069123+0000", 2024-09-13T11:25:45.173 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.289e-06 2024-09-13T11:25:45.173 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:45.173 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:45.173 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:25:45.173 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.069128+0000", 2024-09-13T11:25:45.174 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.3530000000000002e-06 2024-09-13T11:25:45.174 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:45.174 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:45.174 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:25:45.174 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.276885+0000", 2024-09-13T11:25:45.174 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.207756989 2024-09-13T11:25:45.174 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:45.174 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:45.174 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:25:45.174 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.276892+0000", 2024-09-13T11:25:45.174 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.922e-06 2024-09-13T11:25:45.174 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:45.174 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:45.175 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:25:45.175 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.276953+0000", 2024-09-13T11:25:45.175 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.0890000000000001e-05 2024-09-13T11:25:45.175 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:45.175 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:45.175 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:25:45.175 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.276959+0000", 2024-09-13T11:25:45.175 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.9179999999999999e-06 2024-09-13T11:25:45.175 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:45.175 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:45.175 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:25:45.175 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.277161+0000", 2024-09-13T11:25:45.175 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00020194899999999999 2024-09-13T11:25:45.175 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:45.176 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:25:45.176 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:45.176 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:45.176 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:45.176 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:60427 3.14 3:2afea229:::benchmark_data_smithi177_31116_object60426:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e85)", 2024-09-13T11:25:45.176 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:42.187231+0000", 2024-09-13T11:25:45.176 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 122.953536622, 2024-09-13T11:25:45.176 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.83900596999999999, 2024-09-13T11:25:45.176 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:25:45.176 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:25:45.176 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:45.176 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:25:45.176 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:25:45.176 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:45.177 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 60427 2024-09-13T11:25:45.177 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:45.177 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:25:45.177 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:45.177 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:25:45.177 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.187231+0000", 2024-09-13T11:25:45.177 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:45.177 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:45.177 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:45.177 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:25:45.177 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.187231+0000", 2024-09-13T11:25:45.177 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:45.177 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:45.177 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:45.178 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:25:45.178 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.187235+0000", 2024-09-13T11:25:45.178 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.3579999999999999e-06 2024-09-13T11:25:45.178 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:45.178 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:45.178 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:25:45.178 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.187261+0000", 2024-09-13T11:25:45.178 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.6089e-05 2024-09-13T11:25:45.178 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:45.178 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:45.178 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:25:45.178 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.187275+0000", 2024-09-13T11:25:45.178 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.4653e-05 2024-09-13T11:25:45.178 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:45.178 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:45.179 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:25:45.179 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.187282+0000", 2024-09-13T11:25:45.179 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.8870000000000003e-06 2024-09-13T11:25:45.179 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:45.179 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:45.179 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:25:45.179 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.187336+0000", 2024-09-13T11:25:45.179 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.3875999999999997e-05 2024-09-13T11:25:45.179 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:45.179 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:45.179 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:25:45.179 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.187480+0000", 2024-09-13T11:25:45.179 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00014330899999999999 2024-09-13T11:25:45.180 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:45.180 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:45.180 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 14", 2024-09-13T11:25:45.180 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.187658+0000", 2024-09-13T11:25:45.180 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00017861900000000001 2024-09-13T11:25:45.180 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:45.180 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:45.180 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:25:45.180 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.189353+0000", 2024-09-13T11:25:45.180 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001695144 2024-09-13T11:25:45.180 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:45.180 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:45.180 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:25:45.180 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:43.026081+0000", 2024-09-13T11:25:45.181 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.83672736199999997 2024-09-13T11:25:45.181 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:45.181 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:45.181 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:25:45.181 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:43.026169+0000", 2024-09-13T11:25:45.181 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.8411000000000004e-05 2024-09-13T11:25:45.181 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:45.181 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:45.181 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:25:45.181 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:43.026237+0000", 2024-09-13T11:25:45.181 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.8262000000000005e-05 2024-09-13T11:25:45.181 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:45.181 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:25:45.181 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:45.182 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:45.182 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:45.182 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop_reply(client.4665.0:60427 3.14 e85/84 ondisk, result = 0)", 2024-09-13T11:25:45.182 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:42.190914+0000", 2024-09-13T11:25:45.182 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 122.94985395800001, 2024-09-13T11:25:45.182 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.83534071499999996, 2024-09-13T11:25:45.182 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:25:45.182 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:25:45.182 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:25:45.182 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:25:45.182 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:45.183 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:25:45.183 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.190914+0000", 2024-09-13T11:25:45.183 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:45.183 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:45.183 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:45.183 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:25:45.183 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.190914+0000", 2024-09-13T11:25:45.183 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:45.183 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:45.183 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:45.183 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:25:45.183 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.190916+0000", 2024-09-13T11:25:45.183 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.84e-06 2024-09-13T11:25:45.183 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:45.184 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:45.184 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:25:45.184 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.190921+0000", 2024-09-13T11:25:45.184 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.2039999999999996e-06 2024-09-13T11:25:45.184 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:45.184 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:45.184 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:25:45.184 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:43.025926+0000", 2024-09-13T11:25:45.184 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.83500542 2024-09-13T11:25:45.184 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:45.184 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:45.184 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:25:45.184 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:43.025937+0000", 2024-09-13T11:25:45.185 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0597e-05 2024-09-13T11:25:45.185 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:45.185 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:45.185 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:25:45.185 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:43.026052+0000", 2024-09-13T11:25:45.185 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000114838 2024-09-13T11:25:45.185 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:45.185 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:45.185 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:25:45.185 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:43.026070+0000", 2024-09-13T11:25:45.185 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.8131e-05 2024-09-13T11:25:45.185 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:45.185 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:45.185 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:25:45.186 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:43.026255+0000", 2024-09-13T11:25:45.186 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000184685 2024-09-13T11:25:45.186 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:45.186 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:25:45.186 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:45.186 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:45.186 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:45.186 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPushReply(3.1d 89/78 [PushReplyOp(3:bc1b7cfb:::benchmark_data_smithi177_31116_object22629:head)])", 2024-09-13T11:25:45.186 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:46.663106+0000", 2024-09-13T11:25:45.186 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 118.477661775, 2024-09-13T11:25:45.186 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.26645533199999999, 2024-09-13T11:25:45.186 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:25:45.186 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:25:45.186 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-13T11:25:45.187 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:25:45.187 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:45.187 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:25:45.187 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.663106+0000", 2024-09-13T11:25:45.187 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:45.187 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:45.187 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:45.187 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:25:45.187 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.663106+0000", 2024-09-13T11:25:45.187 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:45.187 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:45.187 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:45.187 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:25:45.187 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.663108+0000", 2024-09-13T11:25:45.188 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.0140000000000001e-06 2024-09-13T11:25:45.188 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:45.188 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:45.188 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:25:45.188 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.663116+0000", 2024-09-13T11:25:45.188 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.5440000000000001e-06 2024-09-13T11:25:45.188 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:45.188 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:45.188 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:25:45.188 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.928111+0000", 2024-09-13T11:25:45.188 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.26499566899999999 2024-09-13T11:25:45.188 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:45.188 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:45.188 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:25:45.189 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.928135+0000", 2024-09-13T11:25:45.189 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.4003e-05 2024-09-13T11:25:45.189 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:45.189 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:45.189 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:25:45.189 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.928559+0000", 2024-09-13T11:25:45.189 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000423318 2024-09-13T11:25:45.189 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:45.189 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:45.189 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:25:45.189 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.929562+0000", 2024-09-13T11:25:45.189 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0010027840000000001 2024-09-13T11:25:45.189 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:45.190 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:25:45.190 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:45.190 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:45.190 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:45.190 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "rep_scrubmap(3.14 e84 from shard 14)", 2024-09-13T11:25:45.190 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:48.313615+0000", 2024-09-13T11:25:45.190 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 116.827152982, 2024-09-13T11:25:45.190 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.35633836099999999, 2024-09-13T11:25:45.190 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:25:45.190 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:25:45.190 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:25:45.190 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:25:45.190 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:45.191 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:25:45.191 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.313615+0000", 2024-09-13T11:25:45.191 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:45.191 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:45.191 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:45.191 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:25:45.191 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.313615+0000", 2024-09-13T11:25:45.191 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:45.191 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:45.191 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:45.191 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:25:45.191 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.313617+0000", 2024-09-13T11:25:45.191 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.7990000000000001e-06 2024-09-13T11:25:45.191 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:45.192 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:45.192 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:25:45.192 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.313688+0000", 2024-09-13T11:25:45.192 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.1481000000000002e-05 2024-09-13T11:25:45.192 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:45.192 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:45.192 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:25:45.192 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.669785+0000", 2024-09-13T11:25:45.192 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.356096466 2024-09-13T11:25:45.192 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:45.192 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:45.192 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:25:45.192 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.669792+0000", 2024-09-13T11:25:45.192 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.1740000000000003e-06 2024-09-13T11:25:45.193 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:45.193 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:45.193 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:25:45.193 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.669839+0000", 2024-09-13T11:25:45.193 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.7323000000000001e-05 2024-09-13T11:25:45.193 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:45.193 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:45.193 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:25:45.193 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.669847+0000", 2024-09-13T11:25:45.193 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.0469999999999994e-06 2024-09-13T11:25:45.193 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:45.193 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:45.193 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:25:45.193 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.669953+0000", 2024-09-13T11:25:45.194 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00010607099999999999 2024-09-13T11:25:45.194 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:45.194 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:25:45.194 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:45.194 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:45.194 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:45.194 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:110268 3.14 3:2997c4c5:::benchmark_data_smithi177_31116_object110267:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119)", 2024-09-13T11:25:45.194 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:25.359386+0000", 2024-09-13T11:25:45.194 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 79.781381666000001, 2024-09-13T11:25:45.194 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.29072703, 2024-09-13T11:25:45.194 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:25:45.194 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:25:45.194 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:45.195 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:25:45.195 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:25:45.195 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:45.195 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 110268 2024-09-13T11:25:45.195 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:45.195 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:25:45.195 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:45.195 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:25:45.195 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.359386+0000", 2024-09-13T11:25:45.195 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:45.195 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:45.195 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:45.195 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:25:45.196 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.359386+0000", 2024-09-13T11:25:45.196 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:45.196 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:45.196 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:45.196 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:25:45.196 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.359389+0000", 2024-09-13T11:25:45.196 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.8600000000000001e-06 2024-09-13T11:25:45.196 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:45.196 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:45.196 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:25:45.196 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.359424+0000", 2024-09-13T11:25:45.196 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.4755999999999997e-05 2024-09-13T11:25:45.196 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:45.196 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:45.197 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:25:45.197 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.359445+0000", 2024-09-13T11:25:45.197 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.1104999999999999e-05 2024-09-13T11:25:45.200 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:45.200 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:45.200 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:25:45.200 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.359460+0000", 2024-09-13T11:25:45.200 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5396999999999998e-05 2024-09-13T11:25:45.200 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:45.200 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:45.201 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:25:45.201 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.359581+0000", 2024-09-13T11:25:45.201 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000120455 2024-09-13T11:25:45.201 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:45.201 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:45.201 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:25:45.201 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.359827+0000", 2024-09-13T11:25:45.201 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00024571 2024-09-13T11:25:45.201 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:45.201 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:45.201 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 14", 2024-09-13T11:25:45.201 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.360084+0000", 2024-09-13T11:25:45.201 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00025787999999999999 2024-09-13T11:25:45.202 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:45.202 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:45.202 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:25:45.202 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.362243+0000", 2024-09-13T11:25:45.202 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0021581579999999999 2024-09-13T11:25:45.202 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:45.202 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:45.202 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:25:45.202 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.650034+0000", 2024-09-13T11:25:45.202 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.287791662 2024-09-13T11:25:45.202 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:45.202 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:45.202 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:25:45.202 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.650077+0000", 2024-09-13T11:25:45.203 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.2367999999999998e-05 2024-09-13T11:25:45.203 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:45.203 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:45.203 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:25:45.203 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.650113+0000", 2024-09-13T11:25:45.203 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.6678999999999999e-05 2024-09-13T11:25:45.203 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:45.203 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:25:45.203 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:45.203 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:45.203 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:45.203 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:110279 3.14 3:2b17a349:::benchmark_data_smithi177_31116_object110278:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119)", 2024-09-13T11:25:45.203 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:25.645137+0000", 2024-09-13T11:25:45.203 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 79.495630828000003, 2024-09-13T11:25:45.204 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.28172927800000003, 2024-09-13T11:25:45.204 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:25:45.204 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:25:45.204 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:45.204 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:25:45.204 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:25:45.204 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:45.204 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 110279 2024-09-13T11:25:45.204 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:45.204 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:25:45.204 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:45.204 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:25:45.204 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.645137+0000", 2024-09-13T11:25:45.205 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:45.205 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:45.205 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:45.205 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:25:45.205 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.645137+0000", 2024-09-13T11:25:45.205 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:45.205 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:45.205 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:45.205 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:25:45.205 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.645140+0000", 2024-09-13T11:25:45.205 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.0939999999999999e-06 2024-09-13T11:25:45.205 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:45.205 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:45.206 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:25:45.206 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.645184+0000", 2024-09-13T11:25:45.206 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.3412999999999999e-05 2024-09-13T11:25:45.206 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:45.206 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:45.206 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:25:45.206 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.645206+0000", 2024-09-13T11:25:45.206 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.1997e-05 2024-09-13T11:25:45.206 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:45.206 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:45.206 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:25:45.206 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.645216+0000", 2024-09-13T11:25:45.206 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0740999999999999e-05 2024-09-13T11:25:45.206 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:45.206 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:45.207 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:25:45.207 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.645397+0000", 2024-09-13T11:25:45.207 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00018077600000000001 2024-09-13T11:25:45.207 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:45.207 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:45.207 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:25:45.207 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.645638+0000", 2024-09-13T11:25:45.207 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000241091 2024-09-13T11:25:45.207 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:45.207 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:45.207 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 14", 2024-09-13T11:25:45.207 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.645922+0000", 2024-09-13T11:25:45.207 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000283565 2024-09-13T11:25:45.208 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:45.208 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:45.208 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:25:45.208 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.648980+0000", 2024-09-13T11:25:45.208 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00305851 2024-09-13T11:25:45.208 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:45.208 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:45.208 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:25:45.208 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.926702+0000", 2024-09-13T11:25:45.208 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.27772199800000003 2024-09-13T11:25:45.208 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:45.208 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:45.208 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:25:45.208 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.926796+0000", 2024-09-13T11:25:45.209 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.4158000000000002e-05 2024-09-13T11:25:45.209 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:45.209 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:45.209 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:25:45.209 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.926866+0000", 2024-09-13T11:25:45.209 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.9935e-05 2024-09-13T11:25:45.209 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:45.209 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:25:45.209 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:45.209 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:45.209 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:45.209 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:129865 3.14 3:2c872f8e:::benchmark_data_smithi177_31116_object129864:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e134)", 2024-09-13T11:25:45.209 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:47.107186+0000", 2024-09-13T11:25:45.210 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 58.033582271, 2024-09-13T11:25:45.210 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.21814059299999999, 2024-09-13T11:25:45.210 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:25:45.210 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:25:45.210 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:45.210 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:25:45.210 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:25:45.210 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:45.210 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 129865 2024-09-13T11:25:45.210 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:45.210 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:25:45.210 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:45.210 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:25:45.210 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:47.107186+0000", 2024-09-13T11:25:45.211 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:45.211 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:45.211 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:45.211 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:25:45.211 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:47.107186+0000", 2024-09-13T11:25:45.211 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:45.211 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:45.211 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:45.211 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:25:45.211 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:47.107187+0000", 2024-09-13T11:25:45.211 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5740000000000001e-06 2024-09-13T11:25:45.211 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:45.211 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:45.211 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:25:45.212 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:47.107207+0000", 2024-09-13T11:25:45.212 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.9304999999999999e-05 2024-09-13T11:25:45.212 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:45.212 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:45.212 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:25:45.212 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:47.107215+0000", 2024-09-13T11:25:45.212 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.7390000000000005e-06 2024-09-13T11:25:45.212 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:45.212 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:45.212 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:25:45.212 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:47.107221+0000", 2024-09-13T11:25:45.212 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.7350000000000001e-06 2024-09-13T11:25:45.212 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:45.212 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:45.213 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:25:45.213 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:47.107278+0000", 2024-09-13T11:25:45.213 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.7209000000000001e-05 2024-09-13T11:25:45.213 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:45.213 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:45.213 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:25:45.213 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:47.107383+0000", 2024-09-13T11:25:45.213 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000104718 2024-09-13T11:25:45.213 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:45.213 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:45.213 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 15", 2024-09-13T11:25:45.213 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:47.107480+0000", 2024-09-13T11:25:45.213 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.7343000000000006e-05 2024-09-13T11:25:45.213 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:45.213 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:45.214 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:25:45.214 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:47.109477+0000", 2024-09-13T11:25:45.214 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0019968490000000002 2024-09-13T11:25:45.214 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:45.214 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:45.214 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:25:45.214 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:47.324147+0000", 2024-09-13T11:25:45.214 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.21467018199999999 2024-09-13T11:25:45.214 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:45.214 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:45.214 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:25:45.214 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:47.324281+0000", 2024-09-13T11:25:45.214 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00013328600000000001 2024-09-13T11:25:45.215 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:45.215 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:45.215 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:25:45.215 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:47.325326+0000", 2024-09-13T11:25:45.215 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0010456529999999999 2024-09-13T11:25:45.215 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:45.215 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:25:45.215 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:45.215 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:45.215 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:45.215 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPush(3.15 140/122 [PushOp(3:ae90e82d:::benchmark_data_smithi177_31116_object53962:head, version: 74'3006, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:ae90e82d:::benchmark_data_smithi177_31116_object53962:head@74'3006, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:25:45.215 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:54.310786+0000", 2024-09-13T11:25:45.215 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 50.829981797999999, 2024-09-13T11:25:45.215 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.229345255, 2024-09-13T11:25:45.216 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:25:45.216 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:25:45.216 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:25:45.216 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:25:45.216 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:45.216 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:25:45.216 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:54.310786+0000", 2024-09-13T11:25:45.216 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:45.216 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:45.216 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:45.216 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:25:45.216 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:54.310786+0000", 2024-09-13T11:25:45.216 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:45.216 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:45.217 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:45.217 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:25:45.217 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:54.310795+0000", 2024-09-13T11:25:45.217 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.1910000000000006e-06 2024-09-13T11:25:45.217 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:45.217 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:45.217 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:25:45.217 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:54.310829+0000", 2024-09-13T11:25:45.217 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.4121000000000001e-05 2024-09-13T11:25:45.217 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:45.217 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:45.217 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:25:45.217 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:54.538419+0000", 2024-09-13T11:25:45.217 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.227589611 2024-09-13T11:25:45.218 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:45.218 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:45.218 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:25:45.218 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:54.538430+0000", 2024-09-13T11:25:45.218 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0456e-05 2024-09-13T11:25:45.218 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:45.218 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:45.218 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:25:45.218 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:54.538497+0000", 2024-09-13T11:25:45.218 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.6948999999999999e-05 2024-09-13T11:25:45.218 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:45.218 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:45.218 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:25:45.218 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:54.538505+0000", 2024-09-13T11:25:45.218 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.2989999999999997e-06 2024-09-13T11:25:45.219 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:45.219 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:45.219 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:25:45.219 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:54.540131+0000", 2024-09-13T11:25:45.219 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001626628 2024-09-13T11:25:45.219 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:45.219 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:25:45.219 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:45.219 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:45.219 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:45.219 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPush(3.15 143/122 [PushOp(3:ae52bb45:::benchmark_data_smithi177_31116_object61893:head, version: 87'3272, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:ae52bb45:::benchmark_data_smithi177_31116_object61893:head@87'3272, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:25:45.219 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:57.633939+0000", 2024-09-13T11:25:45.220 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 47.506828845000001, 2024-09-13T11:25:45.220 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.339405067, 2024-09-13T11:25:45.220 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:25:45.220 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:25:45.220 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:25:45.220 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:25:45.220 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:45.220 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:25:45.220 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:57.633939+0000", 2024-09-13T11:25:45.220 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:45.220 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:45.221 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:45.221 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:25:45.221 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:57.633939+0000", 2024-09-13T11:25:45.221 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:45.221 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:45.221 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:45.221 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:25:45.221 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:57.633941+0000", 2024-09-13T11:25:45.221 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.911e-06 2024-09-13T11:25:45.221 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:45.221 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:45.221 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:25:45.221 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:57.633994+0000", 2024-09-13T11:25:45.222 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.2753999999999997e-05 2024-09-13T11:25:45.222 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:45.222 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:45.222 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:25:45.222 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:57.968403+0000", 2024-09-13T11:25:45.222 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.334409336 2024-09-13T11:25:45.222 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:45.222 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:45.222 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:25:45.222 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:57.968451+0000", 2024-09-13T11:25:45.222 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.7525000000000003e-05 2024-09-13T11:25:45.222 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:45.222 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:45.222 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:25:45.223 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:57.968660+0000", 2024-09-13T11:25:45.223 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000209204 2024-09-13T11:25:45.223 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:45.223 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:45.223 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:25:45.223 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:57.968680+0000", 2024-09-13T11:25:45.223 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.9639999999999998e-05 2024-09-13T11:25:45.223 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:45.223 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:45.223 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:25:45.223 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:57.973344+0000", 2024-09-13T11:25:45.223 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0046646969999999998 2024-09-13T11:25:45.223 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:45.223 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:25:45.224 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:45.224 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:45.224 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:45.224 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:140658 3.17 e144/85 3:ece5a1b3:::benchmark_data_smithi177_31116_object355:head v 144'5654, mlcod=144'5652)", 2024-09-13T11:25:45.224 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:58.247792+0000", 2024-09-13T11:25:45.224 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 46.892976367000003, 2024-09-13T11:25:45.224 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.29148369699999999, 2024-09-13T11:25:45.224 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:25:45.224 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:25:45.224 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:45.224 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:25:45.224 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:45.224 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:25:45.225 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:58.247792+0000", 2024-09-13T11:25:45.225 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:45.225 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:45.225 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:45.225 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:25:45.225 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:58.247792+0000", 2024-09-13T11:25:45.225 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:45.225 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:45.225 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:45.225 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:25:45.225 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:58.247793+0000", 2024-09-13T11:25:45.225 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.347e-06 2024-09-13T11:25:45.225 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:45.225 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:45.226 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:25:45.226 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:58.247799+0000", 2024-09-13T11:25:45.226 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.5239999999999997e-06 2024-09-13T11:25:45.226 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:45.226 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:45.226 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:25:45.226 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:58.535226+0000", 2024-09-13T11:25:45.226 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.28742604900000002 2024-09-13T11:25:45.226 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:45.226 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:45.226 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:25:45.226 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:58.535238+0000", 2024-09-13T11:25:45.226 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.2843000000000001e-05 2024-09-13T11:25:45.226 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:45.227 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:45.227 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:25:45.227 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:58.535393+0000", 2024-09-13T11:25:45.227 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00015481699999999999 2024-09-13T11:25:45.227 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:45.227 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:45.227 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:25:45.227 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:58.535434+0000", 2024-09-13T11:25:45.227 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.1152999999999997e-05 2024-09-13T11:25:45.227 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:45.227 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:45.227 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:25:45.227 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:58.539191+0000", 2024-09-13T11:25:45.228 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0037571520000000001 2024-09-13T11:25:45.228 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:45.228 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:45.228 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:25:45.228 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:58.539275+0000", 2024-09-13T11:25:45.228 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.3812000000000003e-05 2024-09-13T11:25:45.228 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:45.228 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:25:45.228 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:45.228 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:45.228 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:45.228 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPush(3.15 144/122 [PushOp(3:a97ba025:::benchmark_data_smithi177_31116_object64775:head, version: 90'3365, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:a97ba025:::benchmark_data_smithi177_31116_object64775:head@90'3365, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:25:45.228 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:58.825014+0000", 2024-09-13T11:25:45.229 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 46.315753739000002, 2024-09-13T11:25:45.229 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.23868276499999999, 2024-09-13T11:25:45.229 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:25:45.229 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:25:45.229 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:25:45.229 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:25:45.229 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:45.229 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:25:45.229 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:58.825014+0000", 2024-09-13T11:25:45.229 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:45.229 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:45.229 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:45.229 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:25:45.230 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:58.825014+0000", 2024-09-13T11:25:45.230 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:45.230 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:45.230 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:45.230 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:25:45.230 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:58.825017+0000", 2024-09-13T11:25:45.230 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.7630000000000001e-06 2024-09-13T11:25:45.230 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:45.230 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:45.230 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:25:45.230 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:58.825099+0000", 2024-09-13T11:25:45.230 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.2035999999999999e-05 2024-09-13T11:25:45.230 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:45.230 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:45.231 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:25:45.231 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:59.059557+0000", 2024-09-13T11:25:45.231 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.23445790399999999 2024-09-13T11:25:45.231 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:45.231 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:45.231 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:25:45.231 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:59.059609+0000", 2024-09-13T11:25:45.231 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.2550999999999999e-05 2024-09-13T11:25:45.231 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:45.231 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:45.231 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:25:45.231 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:59.060117+0000", 2024-09-13T11:25:45.231 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00050754299999999995 2024-09-13T11:25:45.232 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:45.232 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:45.232 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:25:45.232 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:59.060139+0000", 2024-09-13T11:25:45.232 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.2306e-05 2024-09-13T11:25:45.232 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:45.232 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:45.232 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:25:45.232 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:59.063697+0000", 2024-09-13T11:25:45.232 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0035576620000000001 2024-09-13T11:25:45.232 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:45.232 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:25:45.232 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:45.232 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:45.233 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:45.233 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:147472 3.2d 3:b6aeefb0:::benchmark_data_smithi177_31116_object7169:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e148)", 2024-09-13T11:25:45.233 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:25:04.123331+0000", 2024-09-13T11:25:45.233 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 41.017437477999998, 2024-09-13T11:25:45.233 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.23047060799999999, 2024-09-13T11:25:45.233 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:25:45.233 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:25:45.233 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:45.233 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:25:45.233 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:25:45.233 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:45.233 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 147472 2024-09-13T11:25:45.233 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:45.233 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:25:45.234 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:45.234 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:25:45.234 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:04.123331+0000", 2024-09-13T11:25:45.234 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:45.234 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:45.234 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:45.234 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:25:45.234 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:04.123331+0000", 2024-09-13T11:25:45.234 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:45.234 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:45.234 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:45.234 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:25:45.234 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:04.123332+0000", 2024-09-13T11:25:45.234 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.556e-06 2024-09-13T11:25:45.235 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:45.235 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:45.235 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:25:45.235 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:04.123337+0000", 2024-09-13T11:25:45.235 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.5889999999999996e-06 2024-09-13T11:25:45.235 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:45.235 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:45.235 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:25:45.235 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:04.123344+0000", 2024-09-13T11:25:45.235 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.7889999999999995e-06 2024-09-13T11:25:45.235 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:45.235 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:45.235 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:25:45.235 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:04.123350+0000", 2024-09-13T11:25:45.236 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.3820000000000003e-06 2024-09-13T11:25:45.236 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:45.236 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:45.236 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:25:45.236 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:04.123402+0000", 2024-09-13T11:25:45.236 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.2281999999999999e-05 2024-09-13T11:25:45.236 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:45.236 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:45.236 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for degraded object", 2024-09-13T11:25:45.236 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:04.124030+0000", 2024-09-13T11:25:45.236 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00062812999999999996 2024-09-13T11:25:45.236 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:45.236 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:45.236 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:25:45.237 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:04.137378+0000", 2024-09-13T11:25:45.237 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.013347816 2024-09-13T11:25:45.237 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:45.237 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:45.237 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:25:45.237 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:04.137472+0000", 2024-09-13T11:25:45.237 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.3777999999999996e-05 2024-09-13T11:25:45.237 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:45.237 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:45.237 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 0,5", 2024-09-13T11:25:45.237 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:04.137594+0000", 2024-09-13T11:25:45.237 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00012196099999999999 2024-09-13T11:25:45.237 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:45.237 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:45.238 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:25:45.238 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:04.139184+0000", 2024-09-13T11:25:45.238 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001590508 2024-09-13T11:25:45.238 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:45.238 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:45.238 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:25:45.238 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:04.143627+0000", 2024-09-13T11:25:45.238 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0044427989999999999 2024-09-13T11:25:45.238 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:45.238 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:45.238 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:25:45.238 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:04.352741+0000", 2024-09-13T11:25:45.238 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.209114142 2024-09-13T11:25:45.238 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:45.239 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:45.239 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:25:45.239 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:04.352808+0000", 2024-09-13T11:25:45.239 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.7034999999999998e-05 2024-09-13T11:25:45.239 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:45.239 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:45.239 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:25:45.239 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:04.353801+0000", 2024-09-13T11:25:45.239 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00099284099999999995 2024-09-13T11:25:45.239 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:45.239 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:25:45.239 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:45.239 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:45.239 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:25:45.240 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-09-13T11:25:45.241 INFO:teuthology.orchestra.run.smithi001.stdout:osd.3: {} 2024-09-13T11:25:45.241 INFO:teuthology.orchestra.run.smithi001.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T11:25:45.242 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-13T11:25:45.243 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-13T11:25:45.243 INFO:teuthology.orchestra.run.smithi001.stderr:no valid command found; 10 closest matches: 2024-09-13T11:25:45.243 INFO:teuthology.orchestra.run.smithi001.stderr:0 2024-09-13T11:25:45.243 INFO:teuthology.orchestra.run.smithi001.stderr:1 2024-09-13T11:25:45.243 INFO:teuthology.orchestra.run.smithi001.stderr:2 2024-09-13T11:25:45.243 INFO:teuthology.orchestra.run.smithi001.stderr:abort 2024-09-13T11:25:45.243 INFO:teuthology.orchestra.run.smithi001.stderr:assert 2024-09-13T11:25:45.243 INFO:teuthology.orchestra.run.smithi001.stderr:bench [] [] [] [] 2024-09-13T11:25:45.244 INFO:teuthology.orchestra.run.smithi001.stderr:bluefs debug_inject_read_zeros 2024-09-13T11:25:45.244 INFO:teuthology.orchestra.run.smithi001.stderr:bluefs files list 2024-09-13T11:25:45.244 INFO:teuthology.orchestra.run.smithi001.stderr:bluefs stats 2024-09-13T11:25:45.244 INFO:teuthology.orchestra.run.smithi001.stderr:bluestore allocator dump block 2024-09-13T11:25:45.244 INFO:teuthology.orchestra.run.smithi001.stderr:Error EINVAL: invalid command 2024-09-13T11:25:45.244 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-13T11:25:45.244 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 'trim stale osdmaps' 2024-09-13T11:25:45.245 INFO:teuthology.orchestra.run.smithi001.stdout:osd.4: {} 2024-09-13T11:25:45.245 INFO:teuthology.orchestra.run.smithi001.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T11:25:45.245 INFO:teuthology.orchestra.run.smithi001.stdout:osd.5: {} 2024-09-13T11:25:45.245 INFO:teuthology.orchestra.run.smithi001.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T11:25:45.246 INFO:tasks.ceph.osd.12.smithi001.stderr:2024-09-13T11:25:45.239+0000 7f892c6e6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26314) UID: 0 2024-09-13T11:25:45.250 INFO:teuthology.orchestra.run.smithi001.stdout:osd.6: {} 2024-09-13T11:25:45.250 INFO:teuthology.orchestra.run.smithi001.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T11:25:45.276 INFO:teuthology.orchestra.run.smithi001.stdout:osd.7: {} 2024-09-13T11:25:45.276 INFO:teuthology.orchestra.run.smithi001.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T11:25:45.304 INFO:teuthology.orchestra.run.smithi001.stdout:osd.8: {} 2024-09-13T11:25:45.304 INFO:teuthology.orchestra.run.smithi001.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-13T11:25:45.336 INFO:teuthology.orchestra.run.smithi001.stdout:osd.9: {} 2024-09-13T11:25:45.336 INFO:teuthology.orchestra.run.smithi001.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-13T11:25:45.341 INFO:tasks.ceph.osd.5.smithi092.stderr:2024-09-13T11:25:45.341+0000 7f6f0b705640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26346) UID: 0 2024-09-13T11:25:45.366 INFO:teuthology.orchestra.run.smithi001.stdout:osd.10: {} 2024-09-13T11:25:45.366 INFO:teuthology.orchestra.run.smithi001.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-13T11:25:45.390 INFO:teuthology.orchestra.run.smithi001.stdout:osd.11: {} 2024-09-13T11:25:45.390 INFO:teuthology.orchestra.run.smithi001.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-13T11:25:45.418 INFO:teuthology.orchestra.run.smithi001.stdout:osd.12: {} 2024-09-13T11:25:45.418 INFO:teuthology.orchestra.run.smithi001.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-13T11:25:45.425 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-09-13T11:25:45.425 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:25:45.443 INFO:tasks.ceph.osd.1.smithi092.stderr:2024-09-13T11:25:45.441+0000 7fe7789c5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33718) UID: 0 2024-09-13T11:25:45.444 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-13T11:25:45.444 INFO:teuthology.orchestra.run.smithi001.stdout:osd.13: {} 2024-09-13T11:25:45.445 INFO:teuthology.orchestra.run.smithi001.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-13T11:25:45.455 INFO:teuthology.orchestra.run.smithi001.stderr:no valid command found; 10 closest matches: 2024-09-13T11:25:45.455 INFO:teuthology.orchestra.run.smithi001.stderr:0 2024-09-13T11:25:45.455 INFO:teuthology.orchestra.run.smithi001.stderr:1 2024-09-13T11:25:45.455 INFO:teuthology.orchestra.run.smithi001.stderr:2 2024-09-13T11:25:45.455 INFO:teuthology.orchestra.run.smithi001.stderr:abort 2024-09-13T11:25:45.455 INFO:teuthology.orchestra.run.smithi001.stderr:assert 2024-09-13T11:25:45.455 INFO:teuthology.orchestra.run.smithi001.stderr:bench [] [] [] [] 2024-09-13T11:25:45.455 INFO:teuthology.orchestra.run.smithi001.stderr:bluefs debug_inject_read_zeros 2024-09-13T11:25:45.456 INFO:teuthology.orchestra.run.smithi001.stderr:bluefs files list 2024-09-13T11:25:45.456 INFO:teuthology.orchestra.run.smithi001.stderr:bluefs stats 2024-09-13T11:25:45.456 INFO:teuthology.orchestra.run.smithi001.stderr:bluestore allocator dump block 2024-09-13T11:25:45.456 INFO:teuthology.orchestra.run.smithi001.stderr:Error EINVAL: invalid command 2024-09-13T11:25:45.457 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-13T11:25:45.458 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-13T11:25:45.458 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 'trim stale osdmaps' 2024-09-13T11:25:45.472 INFO:teuthology.orchestra.run.smithi001.stdout:osd.14: {} 2024-09-13T11:25:45.472 INFO:teuthology.orchestra.run.smithi001.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-13T11:25:45.497 INFO:teuthology.orchestra.run.smithi001.stdout:osd.15: {} 2024-09-13T11:25:45.497 INFO:teuthology.orchestra.run.smithi001.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-13T11:25:45.543 INFO:tasks.ceph.osd.10.smithi162.stderr:2024-09-13T11:25:45.540+0000 7fb59c9f0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26317) UID: 0 2024-09-13T11:25:45.597 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-09-13T11:25:45.597 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:25:45.617 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-13T11:25:45.652 INFO:teuthology.orchestra.run.smithi001.stderr:no valid command found; 10 closest matches: 2024-09-13T11:25:45.652 INFO:teuthology.orchestra.run.smithi001.stderr:0 2024-09-13T11:25:45.652 INFO:teuthology.orchestra.run.smithi001.stderr:1 2024-09-13T11:25:45.652 INFO:teuthology.orchestra.run.smithi001.stderr:2 2024-09-13T11:25:45.652 INFO:teuthology.orchestra.run.smithi001.stderr:abort 2024-09-13T11:25:45.652 INFO:teuthology.orchestra.run.smithi001.stderr:assert 2024-09-13T11:25:45.652 INFO:teuthology.orchestra.run.smithi001.stderr:bench [] [] [] [] 2024-09-13T11:25:45.652 INFO:teuthology.orchestra.run.smithi001.stderr:bluefs debug_inject_read_zeros 2024-09-13T11:25:45.653 INFO:teuthology.orchestra.run.smithi001.stderr:bluefs files list 2024-09-13T11:25:45.653 INFO:teuthology.orchestra.run.smithi001.stderr:bluefs stats 2024-09-13T11:25:45.653 INFO:teuthology.orchestra.run.smithi001.stderr:bluestore allocator dump block 2024-09-13T11:25:45.653 INFO:teuthology.orchestra.run.smithi001.stderr:Error EINVAL: invalid command 2024-09-13T11:25:45.654 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-13T11:25:45.655 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-13T11:25:45.655 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 'trim stale osdmaps' 2024-09-13T11:25:45.663 INFO:tasks.ceph.osd.11.smithi177.stderr:2024-09-13T11:25:45.662+0000 7f24792c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26580) UID: 0 2024-09-13T11:25:45.746 INFO:tasks.ceph.osd.15.smithi177.stderr:2024-09-13T11:25:45.742+0000 7f9d6ce77640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26578) UID: 0 2024-09-13T11:25:45.765 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-09-13T11:25:45.766 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:25:45.784 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-09-13T11:25:45.840 INFO:teuthology.orchestra.run.smithi001.stderr:no valid command found; 10 closest matches: 2024-09-13T11:25:45.841 INFO:teuthology.orchestra.run.smithi001.stderr:0 2024-09-13T11:25:45.841 INFO:teuthology.orchestra.run.smithi001.stderr:1 2024-09-13T11:25:45.841 INFO:teuthology.orchestra.run.smithi001.stderr:2 2024-09-13T11:25:45.841 INFO:teuthology.orchestra.run.smithi001.stderr:abort 2024-09-13T11:25:45.841 INFO:teuthology.orchestra.run.smithi001.stderr:assert 2024-09-13T11:25:45.841 INFO:teuthology.orchestra.run.smithi001.stderr:bench [] [] [] [] 2024-09-13T11:25:45.841 INFO:teuthology.orchestra.run.smithi001.stderr:bluefs debug_inject_read_zeros 2024-09-13T11:25:45.841 INFO:teuthology.orchestra.run.smithi001.stderr:bluefs files list 2024-09-13T11:25:45.841 INFO:teuthology.orchestra.run.smithi001.stderr:bluefs stats 2024-09-13T11:25:45.841 INFO:teuthology.orchestra.run.smithi001.stderr:bluestore allocator dump block 2024-09-13T11:25:45.841 INFO:teuthology.orchestra.run.smithi001.stderr:Error EINVAL: invalid command 2024-09-13T11:25:45.844 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-13T11:25:45.844 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-13T11:25:45.844 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 'trim stale osdmaps' 2024-09-13T11:25:45.846 INFO:tasks.ceph.osd.13.smithi092.stderr:2024-09-13T11:25:45.845+0000 7f3e5afea640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26350) UID: 0 2024-09-13T11:25:45.946 INFO:tasks.ceph.osd.8.smithi001.stderr:2024-09-13T11:25:45.943+0000 7fb9b48ae640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26312) UID: 0 2024-09-13T11:25:45.951 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-09-13T11:25:45.952 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:25:45.967 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-09-13T11:25:46.023 INFO:teuthology.orchestra.run.smithi001.stderr:no valid command found; 10 closest matches: 2024-09-13T11:25:46.023 INFO:teuthology.orchestra.run.smithi001.stderr:0 2024-09-13T11:25:46.023 INFO:teuthology.orchestra.run.smithi001.stderr:1 2024-09-13T11:25:46.023 INFO:teuthology.orchestra.run.smithi001.stderr:2 2024-09-13T11:25:46.023 INFO:teuthology.orchestra.run.smithi001.stderr:abort 2024-09-13T11:25:46.023 INFO:teuthology.orchestra.run.smithi001.stderr:assert 2024-09-13T11:25:46.023 INFO:teuthology.orchestra.run.smithi001.stderr:bench [] [] [] [] 2024-09-13T11:25:46.023 INFO:teuthology.orchestra.run.smithi001.stderr:bluefs debug_inject_read_zeros 2024-09-13T11:25:46.024 INFO:teuthology.orchestra.run.smithi001.stderr:bluefs files list 2024-09-13T11:25:46.024 INFO:teuthology.orchestra.run.smithi001.stderr:bluefs stats 2024-09-13T11:25:46.024 INFO:teuthology.orchestra.run.smithi001.stderr:bluestore allocator dump block 2024-09-13T11:25:46.024 INFO:teuthology.orchestra.run.smithi001.stderr:Error EINVAL: invalid command 2024-09-13T11:25:46.026 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-13T11:25:46.026 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-13T11:25:46.026 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.8 'trim stale osdmaps' 2024-09-13T11:25:46.047 INFO:tasks.ceph.osd.9.smithi092.stderr:2024-09-13T11:25:46.045+0000 7ff5ab3d0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26348) UID: 0 2024-09-13T11:25:46.117 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-09-13T11:25:46.118 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:25:46.139 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-09-13T11:25:46.184 INFO:tasks.ceph.osd.3.smithi177.stderr:2024-09-13T11:25:46.182+0000 7f10adba4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26574) UID: 0 2024-09-13T11:25:46.246 INFO:teuthology.orchestra.run.smithi001.stderr:no valid command found; 10 closest matches: 2024-09-13T11:25:46.247 INFO:teuthology.orchestra.run.smithi001.stderr:0 2024-09-13T11:25:46.247 INFO:teuthology.orchestra.run.smithi001.stderr:1 2024-09-13T11:25:46.247 INFO:teuthology.orchestra.run.smithi001.stderr:2 2024-09-13T11:25:46.247 INFO:teuthology.orchestra.run.smithi001.stderr:abort 2024-09-13T11:25:46.247 INFO:teuthology.orchestra.run.smithi001.stderr:assert 2024-09-13T11:25:46.247 INFO:teuthology.orchestra.run.smithi001.stderr:bench [] [] [] [] 2024-09-13T11:25:46.247 INFO:teuthology.orchestra.run.smithi001.stderr:bluefs debug_inject_read_zeros 2024-09-13T11:25:46.247 INFO:teuthology.orchestra.run.smithi001.stderr:bluefs files list 2024-09-13T11:25:46.247 INFO:teuthology.orchestra.run.smithi001.stderr:bluefs stats 2024-09-13T11:25:46.247 INFO:teuthology.orchestra.run.smithi001.stderr:bluestore allocator dump block 2024-09-13T11:25:46.247 INFO:teuthology.orchestra.run.smithi001.stderr:Error EINVAL: invalid command 2024-09-13T11:25:46.249 INFO:tasks.ceph.osd.2.smithi162.stderr:2024-09-13T11:25:46.244+0000 7fe92b35a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26313) UID: 0 2024-09-13T11:25:46.249 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-13T11:25:46.250 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-13T11:25:46.250 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.9 'trim stale osdmaps' 2024-09-13T11:25:46.279 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-09-13T11:25:46.279 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:25:46.297 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_ops_in_flight 2024-09-13T11:25:46.374 INFO:tasks.ceph.osd.9.smithi092.stderr:2024-09-13T11:25:46.373+0000 7ff59ddf9640 -1 osd.9 179 handle_osd_map injecting map crc failure 2024-09-13T11:25:46.376 INFO:tasks.ceph.osd.9.smithi092.stderr:2024-09-13T11:25:46.373+0000 7ff59bdf5640 -1 osd.9 179 handle_osd_map injecting map crc failure 2024-09-13T11:25:46.379 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-13T11:25:46.375+0000 7f6a5f1a6640 -1 osd.0 179 handle_osd_map injecting map crc failure 2024-09-13T11:25:46.379 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-13T11:25:46.383 INFO:tasks.ceph.osd.15.smithi177.stderr:2024-09-13T11:25:46.382+0000 7f9d6ce77640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26578) UID: 0 2024-09-13T11:25:46.444 INFO:teuthology.orchestra.run.smithi001.stderr:no valid command found; 10 closest matches: 2024-09-13T11:25:46.444 INFO:teuthology.orchestra.run.smithi001.stderr:0 2024-09-13T11:25:46.444 INFO:teuthology.orchestra.run.smithi001.stderr:1 2024-09-13T11:25:46.444 INFO:teuthology.orchestra.run.smithi001.stderr:2 2024-09-13T11:25:46.444 INFO:teuthology.orchestra.run.smithi001.stderr:abort 2024-09-13T11:25:46.444 INFO:teuthology.orchestra.run.smithi001.stderr:assert 2024-09-13T11:25:46.444 INFO:teuthology.orchestra.run.smithi001.stderr:bench [] [] [] [] 2024-09-13T11:25:46.445 INFO:teuthology.orchestra.run.smithi001.stderr:bluefs debug_inject_read_zeros 2024-09-13T11:25:46.445 INFO:teuthology.orchestra.run.smithi001.stderr:bluefs files list 2024-09-13T11:25:46.445 INFO:teuthology.orchestra.run.smithi001.stderr:bluefs stats 2024-09-13T11:25:46.445 INFO:teuthology.orchestra.run.smithi001.stderr:bluestore allocator dump block 2024-09-13T11:25:46.445 INFO:teuthology.orchestra.run.smithi001.stderr:Error EINVAL: invalid command 2024-09-13T11:25:46.447 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-13T11:25:46.447 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-13T11:25:46.448 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.10 'trim stale osdmaps' 2024-09-13T11:25:46.450 INFO:tasks.ceph.osd.15.smithi177.stderr:2024-09-13T11:25:46.446+0000 7f9d6ce77640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26578) UID: 0 2024-09-13T11:25:46.485 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-09-13T11:25:46.485 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:25:46.502 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_blocked_ops 2024-09-13T11:25:46.550 INFO:tasks.ceph.osd.13.smithi092.stderr:2024-09-13T11:25:46.549+0000 7f3e5afea640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26350) UID: 0 2024-09-13T11:25:46.620 INFO:teuthology.orchestra.run.smithi001.stderr:no valid command found; 10 closest matches: 2024-09-13T11:25:46.620 INFO:teuthology.orchestra.run.smithi001.stderr:0 2024-09-13T11:25:46.620 INFO:teuthology.orchestra.run.smithi001.stderr:1 2024-09-13T11:25:46.621 INFO:teuthology.orchestra.run.smithi001.stderr:2 2024-09-13T11:25:46.621 INFO:teuthology.orchestra.run.smithi001.stderr:abort 2024-09-13T11:25:46.621 INFO:teuthology.orchestra.run.smithi001.stderr:assert 2024-09-13T11:25:46.621 INFO:teuthology.orchestra.run.smithi001.stderr:bench [] [] [] [] 2024-09-13T11:25:46.621 INFO:teuthology.orchestra.run.smithi001.stderr:bluefs debug_inject_read_zeros 2024-09-13T11:25:46.621 INFO:teuthology.orchestra.run.smithi001.stderr:bluefs files list 2024-09-13T11:25:46.621 INFO:teuthology.orchestra.run.smithi001.stderr:bluefs stats 2024-09-13T11:25:46.621 INFO:teuthology.orchestra.run.smithi001.stderr:bluestore allocator dump block 2024-09-13T11:25:46.621 INFO:teuthology.orchestra.run.smithi001.stderr:Error EINVAL: invalid command 2024-09-13T11:25:46.623 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-13T11:25:46.624 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-13T11:25:46.624 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.11 'trim stale osdmaps' 2024-09-13T11:25:46.652 INFO:tasks.ceph.osd.9.smithi092.stderr:2024-09-13T11:25:46.649+0000 7ff5ab3d0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26348) UID: 0 2024-09-13T11:25:46.681 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-09-13T11:25:46.681 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:25:46.697 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_historic_ops 2024-09-13T11:25:46.753 INFO:tasks.ceph.osd.14.smithi162.stderr:2024-09-13T11:25:46.752+0000 7f3df5903640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33154) UID: 0 2024-09-13T11:25:46.805 INFO:teuthology.orchestra.run.smithi001.stderr:no valid command found; 10 closest matches: 2024-09-13T11:25:46.805 INFO:teuthology.orchestra.run.smithi001.stderr:0 2024-09-13T11:25:46.805 INFO:teuthology.orchestra.run.smithi001.stderr:1 2024-09-13T11:25:46.806 INFO:teuthology.orchestra.run.smithi001.stderr:2 2024-09-13T11:25:46.806 INFO:teuthology.orchestra.run.smithi001.stderr:abort 2024-09-13T11:25:46.806 INFO:teuthology.orchestra.run.smithi001.stderr:assert 2024-09-13T11:25:46.806 INFO:teuthology.orchestra.run.smithi001.stderr:bench [] [] [] [] 2024-09-13T11:25:46.806 INFO:teuthology.orchestra.run.smithi001.stderr:bluefs debug_inject_read_zeros 2024-09-13T11:25:46.806 INFO:teuthology.orchestra.run.smithi001.stderr:bluefs files list 2024-09-13T11:25:46.806 INFO:teuthology.orchestra.run.smithi001.stderr:bluefs stats 2024-09-13T11:25:46.806 INFO:teuthology.orchestra.run.smithi001.stderr:bluestore allocator dump block 2024-09-13T11:25:46.806 INFO:teuthology.orchestra.run.smithi001.stderr:Error EINVAL: invalid command 2024-09-13T11:25:46.808 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-13T11:25:46.808 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-13T11:25:46.808 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.12 'trim stale osdmaps' 2024-09-13T11:25:46.830 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-09-13T11:25:46.830 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:25:46.846 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-13T11:25:46.853 INFO:tasks.ceph.osd.3.smithi177.stderr:2024-09-13T11:25:46.850+0000 7f10adba4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26574) UID: 0 2024-09-13T11:25:46.955 INFO:tasks.ceph.osd.5.smithi092.stderr:2024-09-13T11:25:46.953+0000 7f6f0b705640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26346) UID: 0 2024-09-13T11:25:46.970 INFO:teuthology.orchestra.run.smithi001.stderr:no valid command found; 10 closest matches: 2024-09-13T11:25:46.970 INFO:teuthology.orchestra.run.smithi001.stderr:0 2024-09-13T11:25:46.970 INFO:teuthology.orchestra.run.smithi001.stderr:1 2024-09-13T11:25:46.970 INFO:teuthology.orchestra.run.smithi001.stderr:2 2024-09-13T11:25:46.970 INFO:teuthology.orchestra.run.smithi001.stderr:abort 2024-09-13T11:25:46.970 INFO:teuthology.orchestra.run.smithi001.stderr:assert 2024-09-13T11:25:46.970 INFO:teuthology.orchestra.run.smithi001.stderr:bench [] [] [] [] 2024-09-13T11:25:46.970 INFO:teuthology.orchestra.run.smithi001.stderr:bluefs debug_inject_read_zeros 2024-09-13T11:25:46.970 INFO:teuthology.orchestra.run.smithi001.stderr:bluefs files list 2024-09-13T11:25:46.971 INFO:teuthology.orchestra.run.smithi001.stderr:bluefs stats 2024-09-13T11:25:46.971 INFO:teuthology.orchestra.run.smithi001.stderr:bluestore allocator dump block 2024-09-13T11:25:46.971 INFO:teuthology.orchestra.run.smithi001.stderr:Error EINVAL: invalid command 2024-09-13T11:25:46.973 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-13T11:25:46.973 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-13T11:25:46.974 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.13 'trim stale osdmaps' 2024-09-13T11:25:46.980 INFO:teuthology.orchestra.run.smithi092.stdout:ERROR: (22) Invalid argument 2024-09-13T11:25:46.981 INFO:teuthology.orchestra.run.smithi092.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:25:46.997 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-13T11:25:47.056 INFO:tasks.ceph.osd.4.smithi001.stderr:2024-09-13T11:25:47.055+0000 7ff8e0218640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26310) UID: 0 2024-09-13T11:25:47.151 INFO:teuthology.orchestra.run.smithi001.stderr:no valid command found; 10 closest matches: 2024-09-13T11:25:47.151 INFO:teuthology.orchestra.run.smithi001.stderr:0 2024-09-13T11:25:47.151 INFO:teuthology.orchestra.run.smithi001.stderr:1 2024-09-13T11:25:47.151 INFO:teuthology.orchestra.run.smithi001.stderr:2 2024-09-13T11:25:47.151 INFO:teuthology.orchestra.run.smithi001.stderr:abort 2024-09-13T11:25:47.151 INFO:teuthology.orchestra.run.smithi001.stderr:assert 2024-09-13T11:25:47.151 INFO:teuthology.orchestra.run.smithi001.stderr:bench [] [] [] [] 2024-09-13T11:25:47.152 INFO:teuthology.orchestra.run.smithi001.stderr:bluefs debug_inject_read_zeros 2024-09-13T11:25:47.152 INFO:teuthology.orchestra.run.smithi001.stderr:bluefs files list 2024-09-13T11:25:47.152 INFO:teuthology.orchestra.run.smithi001.stderr:bluefs stats 2024-09-13T11:25:47.152 INFO:teuthology.orchestra.run.smithi001.stderr:bluestore allocator dump block 2024-09-13T11:25:47.152 INFO:teuthology.orchestra.run.smithi001.stderr:Error EINVAL: invalid command 2024-09-13T11:25:47.154 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-13T11:25:47.154 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-13T11:25:47.154 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.15 'trim stale osdmaps' 2024-09-13T11:25:47.157 INFO:tasks.ceph.osd.11.smithi177.stderr:2024-09-13T11:25:47.154+0000 7f24792c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26580) UID: 0 2024-09-13T11:25:47.174 INFO:teuthology.orchestra.run.smithi092.stdout:ERROR: (22) Invalid argument 2024-09-13T11:25:47.174 INFO:teuthology.orchestra.run.smithi092.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:25:47.190 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-13T11:25:47.258 INFO:tasks.ceph.osd.15.smithi177.stderr:2024-09-13T11:25:47.254+0000 7f9d6ce77640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26578) UID: 0 2024-09-13T11:25:47.326 INFO:teuthology.orchestra.run.smithi092.stdout:ERROR: (22) Invalid argument 2024-09-13T11:25:47.327 INFO:teuthology.orchestra.run.smithi092.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:25:47.344 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_ops_in_flight 2024-09-13T11:25:47.360 INFO:teuthology.orchestra.run.smithi001.stderr:no valid command found; 10 closest matches: 2024-09-13T11:25:47.360 INFO:teuthology.orchestra.run.smithi001.stderr:0 2024-09-13T11:25:47.360 INFO:teuthology.orchestra.run.smithi001.stderr:1 2024-09-13T11:25:47.360 INFO:teuthology.orchestra.run.smithi001.stderr:2 2024-09-13T11:25:47.360 INFO:teuthology.orchestra.run.smithi001.stderr:abort 2024-09-13T11:25:47.360 INFO:teuthology.orchestra.run.smithi001.stderr:assert 2024-09-13T11:25:47.361 INFO:teuthology.orchestra.run.smithi001.stderr:bench [] [] [] [] 2024-09-13T11:25:47.361 INFO:teuthology.orchestra.run.smithi001.stderr:bluefs debug_inject_read_zeros 2024-09-13T11:25:47.361 INFO:teuthology.orchestra.run.smithi001.stderr:bluefs files list 2024-09-13T11:25:47.361 INFO:teuthology.orchestra.run.smithi001.stderr:bluefs stats 2024-09-13T11:25:47.361 INFO:teuthology.orchestra.run.smithi001.stderr:bluestore allocator dump block 2024-09-13T11:25:47.361 INFO:teuthology.orchestra.run.smithi001.stderr:Error EINVAL: invalid command 2024-09-13T11:25:47.363 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-13T11:25:47.363 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-13T11:25:47.376 INFO:teuthology.orchestra.run.smithi001.stderr:nodeep-scrub is unset 2024-09-13T11:25:47.389 INFO:tasks.ceph.osd.14.smithi162.stderr:2024-09-13T11:25:47.388+0000 7f3df5903640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33154) UID: 0 2024-09-13T11:25:47.390 INFO:tasks.ceph.osd.7.smithi177.stderr:2024-09-13T11:25:47.382+0000 7f896ff4a640 -1 osd.7 180 handle_osd_map injecting map crc failure 2024-09-13T11:25:47.401 INFO:tasks.ceph.osd.3.smithi177.stderr:2024-09-13T11:25:47.398+0000 7f10a25d1640 -1 osd.3 180 handle_osd_map injecting map crc failure 2024-09-13T11:25:47.460 INFO:tasks.ceph.osd.9.smithi092.stderr:2024-09-13T11:25:47.457+0000 7ff5ab3d0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26348) UID: 0 2024-09-13T11:25:47.494 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-13T11:25:47.494 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:25:47.510 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_blocked_ops 2024-09-13T11:25:47.522 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T11:25:47.568 INFO:tasks.ceph.osd.4.smithi001.stderr:2024-09-13T11:25:47.567+0000 7ff8e0218640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26310) UID: 0 2024-09-13T11:25:47.661 INFO:tasks.ceph.osd.14.smithi162.stderr:2024-09-13T11:25:47.660+0000 7f3df5903640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33154) UID: 0 2024-09-13T11:25:47.681 INFO:teuthology.orchestra.run.smithi001.stdout:osd.0: {} 2024-09-13T11:25:47.681 INFO:teuthology.orchestra.run.smithi001.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T11:25:47.686 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-13T11:25:47.686 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:25:47.702 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_historic_ops 2024-09-13T11:25:47.711 INFO:teuthology.orchestra.run.smithi001.stdout:osd.1: {} 2024-09-13T11:25:47.711 INFO:teuthology.orchestra.run.smithi001.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T11:25:47.737 INFO:teuthology.orchestra.run.smithi001.stdout:osd.2: {} 2024-09-13T11:25:47.737 INFO:teuthology.orchestra.run.smithi001.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T11:25:47.760 INFO:teuthology.orchestra.run.smithi001.stdout:osd.3: {} 2024-09-13T11:25:47.760 INFO:teuthology.orchestra.run.smithi001.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T11:25:47.762 INFO:tasks.ceph.osd.12.smithi001.stderr:2024-09-13T11:25:47.759+0000 7f892c6e6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26314) UID: 0 2024-09-13T11:25:47.784 INFO:teuthology.orchestra.run.smithi001.stdout:osd.4: {} 2024-09-13T11:25:47.785 INFO:teuthology.orchestra.run.smithi001.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T11:25:47.814 INFO:teuthology.orchestra.run.smithi001.stdout:osd.5: {} 2024-09-13T11:25:47.814 INFO:teuthology.orchestra.run.smithi001.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T11:25:47.842 INFO:teuthology.orchestra.run.smithi001.stdout:osd.6: {} 2024-09-13T11:25:47.843 INFO:teuthology.orchestra.run.smithi001.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T11:25:47.864 INFO:tasks.ceph.osd.10.smithi162.stderr:2024-09-13T11:25:47.860+0000 7fb59c9f0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26317) UID: 0 2024-09-13T11:25:47.865 INFO:teuthology.orchestra.run.smithi001.stdout:osd.7: {} 2024-09-13T11:25:47.866 INFO:teuthology.orchestra.run.smithi001.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T11:25:47.874 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-13T11:25:47.874 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:25:47.886 INFO:teuthology.orchestra.run.smithi001.stdout:osd.8: {} 2024-09-13T11:25:47.886 INFO:teuthology.orchestra.run.smithi001.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-13T11:25:47.888 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-13T11:25:47.913 INFO:teuthology.orchestra.run.smithi001.stdout:osd.9: {} 2024-09-13T11:25:47.913 INFO:teuthology.orchestra.run.smithi001.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-13T11:25:47.940 INFO:teuthology.orchestra.run.smithi001.stdout:osd.10: {} 2024-09-13T11:25:47.940 INFO:teuthology.orchestra.run.smithi001.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-13T11:25:47.964 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-13T11:25:47.963+0000 7f6a68775640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26307) UID: 0 2024-09-13T11:25:47.979 INFO:teuthology.orchestra.run.smithi001.stdout:osd.11: {} 2024-09-13T11:25:47.979 INFO:teuthology.orchestra.run.smithi001.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-13T11:25:48.015 INFO:teuthology.orchestra.run.smithi001.stdout:osd.12: {} 2024-09-13T11:25:48.016 INFO:teuthology.orchestra.run.smithi001.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-13T11:25:48.028 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-13T11:25:48.028 INFO:teuthology.orchestra.run.smithi001.stdout: "ops": [ 2024-09-13T11:25:48.028 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.028 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:201246 3.15 3:abe123a6:::benchmark_data_smithi177_31116_object60943:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e181)", 2024-09-13T11:25:48.028 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:25:47.844013+0000", 2024-09-13T11:25:48.029 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 0.18352165500000001, 2024-09-13T11:25:48.029 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.18353987799999999, 2024-09-13T11:25:48.029 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:48.029 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:48.029 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "waiting for sub ops", 2024-09-13T11:25:48.029 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:25:48.029 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:25:48.029 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:48.029 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 201246 2024-09-13T11:25:48.029 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.029 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:48.030 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.030 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:48.030 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:47.844013+0000", 2024-09-13T11:25:48.030 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:48.030 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.030 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.030 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:48.030 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:47.844013+0000", 2024-09-13T11:25:48.030 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:48.031 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.031 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.031 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:48.031 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:47.844014+0000", 2024-09-13T11:25:48.031 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.835e-06 2024-09-13T11:25:48.031 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.031 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.031 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:48.032 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:47.844019+0000", 2024-09-13T11:25:48.032 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.1699999999999999e-06 2024-09-13T11:25:48.032 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.032 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.032 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:48.032 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:47.844029+0000", 2024-09-13T11:25:48.032 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0631e-05 2024-09-13T11:25:48.032 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.032 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.032 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:48.032 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:47.844037+0000", 2024-09-13T11:25:48.032 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.6539999999999999e-06 2024-09-13T11:25:48.032 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.033 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.033 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:48.033 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:47.847663+0000", 2024-09-13T11:25:48.033 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00362583 2024-09-13T11:25:48.033 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.033 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.033 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:25:48.033 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:47.847786+0000", 2024-09-13T11:25:48.033 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00012374900000000001 2024-09-13T11:25:48.033 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.033 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.033 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 3", 2024-09-13T11:25:48.034 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:47.847933+0000", 2024-09-13T11:25:48.034 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00014651700000000001 2024-09-13T11:25:48.034 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.034 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.034 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:25:48.034 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:47.851682+0000", 2024-09-13T11:25:48.034 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0037493259999999999 2024-09-13T11:25:48.034 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:48.034 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:48.034 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:48.034 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.034 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.034 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:201251 3.15 3:a896c150:::benchmark_data_smithi177_31116_object60948:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e181)", 2024-09-13T11:25:48.034 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:25:47.851517+0000", 2024-09-13T11:25:48.035 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 0.17601682799999999, 2024-09-13T11:25:48.035 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.17608054000000001, 2024-09-13T11:25:48.035 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:48.035 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:48.035 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "waiting for sub ops", 2024-09-13T11:25:48.035 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:25:48.035 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:25:48.035 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:48.035 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 201251 2024-09-13T11:25:48.035 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.035 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:48.035 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.035 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:48.036 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:47.851517+0000", 2024-09-13T11:25:48.036 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:48.036 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.036 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.036 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:48.036 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:47.851517+0000", 2024-09-13T11:25:48.036 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:48.036 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.036 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.036 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:48.037 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:47.851519+0000", 2024-09-13T11:25:48.037 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.0650000000000001e-06 2024-09-13T11:25:48.037 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.037 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.037 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:48.037 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:47.851525+0000", 2024-09-13T11:25:48.037 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.7799999999999997e-06 2024-09-13T11:25:48.037 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.037 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.037 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:48.037 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:47.851536+0000", 2024-09-13T11:25:48.038 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.1155000000000001e-05 2024-09-13T11:25:48.038 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.038 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.038 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:48.038 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:47.851545+0000", 2024-09-13T11:25:48.038 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.4509999999999992e-06 2024-09-13T11:25:48.038 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.038 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.038 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:48.038 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:47.851724+0000", 2024-09-13T11:25:48.038 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000179011 2024-09-13T11:25:48.038 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.038 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.038 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:25:48.039 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:47.851868+0000", 2024-09-13T11:25:48.039 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00014447599999999999 2024-09-13T11:25:48.039 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.039 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.039 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 3", 2024-09-13T11:25:48.039 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:47.851996+0000", 2024-09-13T11:25:48.039 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00012794099999999999 2024-09-13T11:25:48.039 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.039 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.039 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:25:48.039 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:47.854695+0000", 2024-09-13T11:25:48.039 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.002698523 2024-09-13T11:25:48.039 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:48.039 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:48.040 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:48.040 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.040 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.040 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:201255 3.15 3:ac3fab9c:::benchmark_data_smithi177_31116_object60952:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e181)", 2024-09-13T11:25:48.040 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:25:47.859692+0000", 2024-09-13T11:25:48.040 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 0.16784212700000001, 2024-09-13T11:25:48.040 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.16794515700000001, 2024-09-13T11:25:48.040 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:48.040 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:48.040 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "waiting for sub ops", 2024-09-13T11:25:48.040 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:25:48.040 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:25:48.040 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:48.040 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 201255 2024-09-13T11:25:48.041 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.041 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:48.041 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.041 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:48.041 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:47.859692+0000", 2024-09-13T11:25:48.041 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:48.041 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.041 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.041 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:48.041 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:47.859692+0000", 2024-09-13T11:25:48.041 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:48.041 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.041 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.041 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:48.042 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:47.859693+0000", 2024-09-13T11:25:48.042 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.3149999999999999e-06 2024-09-13T11:25:48.042 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.042 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.042 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:48.042 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:47.859698+0000", 2024-09-13T11:25:48.042 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.685e-06 2024-09-13T11:25:48.042 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.042 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.042 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:48.042 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:47.859710+0000", 2024-09-13T11:25:48.042 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.1649e-05 2024-09-13T11:25:48.042 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.042 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.043 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:48.043 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:47.859717+0000", 2024-09-13T11:25:48.043 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.9890000000000003e-06 2024-09-13T11:25:48.043 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.043 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.043 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:48.043 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:47.859773+0000", 2024-09-13T11:25:48.043 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.588e-05 2024-09-13T11:25:48.043 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.043 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.043 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:25:48.043 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:47.859929+0000", 2024-09-13T11:25:48.043 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00015672099999999999 2024-09-13T11:25:48.043 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.043 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.044 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 3", 2024-09-13T11:25:48.044 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:47.860052+0000", 2024-09-13T11:25:48.044 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00012284399999999999 2024-09-13T11:25:48.044 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.044 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.044 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:25:48.044 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:47.862862+0000", 2024-09-13T11:25:48.044 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0028093720000000001 2024-09-13T11:25:48.044 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:48.044 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:48.044 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:48.044 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.044 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.045 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:201269 3.e 3:70120e20:::benchmark_data_smithi177_31116_object60966:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e181)", 2024-09-13T11:25:48.045 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:25:47.936894+0000", 2024-09-13T11:25:48.045 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 0.090639834000000002, 2024-09-13T11:25:48.045 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.090780932999999994, 2024-09-13T11:25:48.045 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:48.045 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:48.045 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "waiting for sub ops", 2024-09-13T11:25:48.045 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:25:48.045 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:25:48.045 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:48.045 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 201269 2024-09-13T11:25:48.045 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.045 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:48.046 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.046 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:48.046 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:47.936894+0000", 2024-09-13T11:25:48.046 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:48.046 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.046 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.046 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:48.046 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:47.936894+0000", 2024-09-13T11:25:48.046 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:48.046 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.046 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.046 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:48.046 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:47.936896+0000", 2024-09-13T11:25:48.046 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.6160000000000001e-06 2024-09-13T11:25:48.046 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.047 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.047 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:48.047 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:47.936901+0000", 2024-09-13T11:25:48.047 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.5129999999999998e-06 2024-09-13T11:25:48.047 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.047 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.047 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:48.047 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:47.936911+0000", 2024-09-13T11:25:48.047 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0747000000000001e-05 2024-09-13T11:25:48.047 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.047 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.047 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:48.047 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:47.936919+0000", 2024-09-13T11:25:48.047 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.6809999999999998e-06 2024-09-13T11:25:48.048 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.048 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.048 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:48.048 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:47.936972+0000", 2024-09-13T11:25:48.048 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.3464999999999997e-05 2024-09-13T11:25:48.048 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.048 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.048 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:25:48.048 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:47.937133+0000", 2024-09-13T11:25:48.048 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00016052899999999999 2024-09-13T11:25:48.048 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.048 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.048 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 1", 2024-09-13T11:25:48.048 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:47.937260+0000", 2024-09-13T11:25:48.049 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00012673 2024-09-13T11:25:48.049 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.049 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.049 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:25:48.049 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:47.940195+0000", 2024-09-13T11:25:48.049 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0029351770000000002 2024-09-13T11:25:48.049 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:48.049 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:48.049 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:48.049 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.049 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.049 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:201281 3.15 3:acb822cb:::benchmark_data_smithi177_31116_object60978:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e181)", 2024-09-13T11:25:48.049 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:25:47.963875+0000", 2024-09-13T11:25:48.049 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 0.063659508000000004, 2024-09-13T11:25:48.050 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.063836766000000003, 2024-09-13T11:25:48.050 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:48.050 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:48.050 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "waiting for sub ops", 2024-09-13T11:25:48.050 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:25:48.050 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:25:48.050 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:48.050 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 201281 2024-09-13T11:25:48.050 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.050 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:48.050 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.050 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:48.050 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:47.963875+0000", 2024-09-13T11:25:48.050 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:48.051 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.051 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.051 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:48.051 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:47.963875+0000", 2024-09-13T11:25:48.051 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:48.051 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.051 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.051 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:48.051 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:47.963876+0000", 2024-09-13T11:25:48.051 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.482e-06 2024-09-13T11:25:48.051 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.051 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.051 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:48.051 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:47.963880+0000", 2024-09-13T11:25:48.052 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.2649999999999998e-06 2024-09-13T11:25:48.052 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.052 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.052 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:48.052 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:47.963889+0000", 2024-09-13T11:25:48.052 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.1999999999999994e-06 2024-09-13T11:25:48.052 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.052 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.052 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:48.052 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:47.963894+0000", 2024-09-13T11:25:48.052 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.1510000000000003e-06 2024-09-13T11:25:48.052 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.052 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.053 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:48.053 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:47.963951+0000", 2024-09-13T11:25:48.053 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.7592e-05 2024-09-13T11:25:48.053 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.053 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.053 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:25:48.053 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:47.964109+0000", 2024-09-13T11:25:48.053 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000157818 2024-09-13T11:25:48.053 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.053 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.053 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 3", 2024-09-13T11:25:48.053 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:47.964259+0000", 2024-09-13T11:25:48.053 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00014941100000000001 2024-09-13T11:25:48.053 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.054 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.054 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:25:48.054 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:47.966743+0000", 2024-09-13T11:25:48.054 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0024845710000000001 2024-09-13T11:25:48.054 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:48.054 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:48.054 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:48.054 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.054 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.054 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:201314 3.e 3:74d02078:::benchmark_data_smithi177_31116_object61011:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e181)", 2024-09-13T11:25:48.054 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:25:48.021686+0000", 2024-09-13T11:25:48.054 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 0.0058481540000000004, 2024-09-13T11:25:48.054 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0060612089999999997, 2024-09-13T11:25:48.054 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:48.055 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:48.055 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "waiting for sub ops", 2024-09-13T11:25:48.055 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:25:48.055 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:25:48.055 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:48.055 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 201314 2024-09-13T11:25:48.055 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.055 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:48.055 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.055 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:48.055 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:48.021686+0000", 2024-09-13T11:25:48.055 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:48.055 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.055 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.055 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:48.056 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:48.021686+0000", 2024-09-13T11:25:48.056 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:48.056 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.056 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.056 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:48.056 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:48.021688+0000", 2024-09-13T11:25:48.056 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.8989999999999999e-06 2024-09-13T11:25:48.056 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.056 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.056 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:48.056 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:48.021693+0000", 2024-09-13T11:25:48.056 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.1839999999999998e-06 2024-09-13T11:25:48.056 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.056 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.057 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:48.057 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:48.021702+0000", 2024-09-13T11:25:48.057 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.2949999999999999e-06 2024-09-13T11:25:48.057 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.057 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.057 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:48.057 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:48.021708+0000", 2024-09-13T11:25:48.057 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.8830000000000003e-06 2024-09-13T11:25:48.057 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.057 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.057 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:48.057 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:48.021761+0000", 2024-09-13T11:25:48.057 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.2528000000000003e-05 2024-09-13T11:25:48.058 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.058 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.058 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:25:48.058 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:48.021839+0000", 2024-09-13T11:25:48.058 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.8435000000000004e-05 2024-09-13T11:25:48.058 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.058 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.058 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 1", 2024-09-13T11:25:48.058 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:48.021898+0000", 2024-09-13T11:25:48.058 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.8582999999999998e-05 2024-09-13T11:25:48.058 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.058 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.058 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:25:48.058 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:48.024454+0000", 2024-09-13T11:25:48.059 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0025561889999999999 2024-09-13T11:25:48.059 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:48.059 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:48.059 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:48.059 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.059 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.059 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:201234 3.21 3:85402570:::benchmark_data_smithi177_31116_object60931:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e181)", 2024-09-13T11:25:48.059 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:25:47.835618+0000", 2024-09-13T11:25:48.059 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 0.19191654399999999, 2024-09-13T11:25:48.059 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.19216804800000001, 2024-09-13T11:25:48.059 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:48.059 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:48.059 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "waiting for sub ops", 2024-09-13T11:25:48.059 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:25:48.060 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:25:48.060 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:48.060 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 201234 2024-09-13T11:25:48.060 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.060 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:48.060 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.060 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:48.060 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:47.835618+0000", 2024-09-13T11:25:48.060 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:48.060 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.060 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.060 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:48.060 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:47.835618+0000", 2024-09-13T11:25:48.060 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:48.061 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.061 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.061 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:48.061 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:47.835619+0000", 2024-09-13T11:25:48.061 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.638e-06 2024-09-13T11:25:48.061 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.061 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.061 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:48.061 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:47.835627+0000", 2024-09-13T11:25:48.061 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.0150000000000003e-06 2024-09-13T11:25:48.061 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.061 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.061 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:48.062 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:47.835639+0000", 2024-09-13T11:25:48.062 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.1939e-05 2024-09-13T11:25:48.062 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.062 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.062 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:48.062 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:47.835644+0000", 2024-09-13T11:25:48.062 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.006e-06 2024-09-13T11:25:48.062 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.062 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.062 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:48.062 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:47.835696+0000", 2024-09-13T11:25:48.062 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.1668000000000001e-05 2024-09-13T11:25:48.062 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.062 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.062 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:25:48.063 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:47.835775+0000", 2024-09-13T11:25:48.063 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.8737000000000002e-05 2024-09-13T11:25:48.063 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.063 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.063 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 8", 2024-09-13T11:25:48.063 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:47.835857+0000", 2024-09-13T11:25:48.063 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.2508000000000004e-05 2024-09-13T11:25:48.063 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.063 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.063 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:25:48.063 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:47.837366+0000", 2024-09-13T11:25:48.063 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.001509292 2024-09-13T11:25:48.063 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:48.063 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:48.064 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:48.064 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.064 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.064 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:201238 3.15 3:ae4e8f1e:::benchmark_data_smithi177_31116_object60935:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e181)", 2024-09-13T11:25:48.064 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:25:47.836319+0000", 2024-09-13T11:25:48.064 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 0.19121550600000001, 2024-09-13T11:25:48.064 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.191524262, 2024-09-13T11:25:48.064 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:48.064 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:48.064 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "waiting for sub ops", 2024-09-13T11:25:48.064 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:25:48.064 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:25:48.064 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:48.064 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 201238 2024-09-13T11:25:48.065 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.065 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:48.065 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.065 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:48.065 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:47.836319+0000", 2024-09-13T11:25:48.065 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:48.065 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.065 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.065 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:48.065 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:47.836319+0000", 2024-09-13T11:25:48.065 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:48.068 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.068 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.068 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:48.068 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:47.836320+0000", 2024-09-13T11:25:48.068 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.387e-06 2024-09-13T11:25:48.068 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.068 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.069 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:48.069 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:47.836325+0000", 2024-09-13T11:25:48.069 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.7720000000000002e-06 2024-09-13T11:25:48.069 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.069 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.069 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:48.069 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:47.836337+0000", 2024-09-13T11:25:48.069 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2499e-05 2024-09-13T11:25:48.069 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.069 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.069 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:48.069 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:47.836345+0000", 2024-09-13T11:25:48.069 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.5939999999999996e-06 2024-09-13T11:25:48.069 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.070 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.070 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:48.070 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:47.836399+0000", 2024-09-13T11:25:48.070 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.4527000000000002e-05 2024-09-13T11:25:48.070 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.070 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.070 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:25:48.070 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:47.836518+0000", 2024-09-13T11:25:48.070 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000118047 2024-09-13T11:25:48.070 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.070 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.070 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 3", 2024-09-13T11:25:48.070 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:47.836626+0000", 2024-09-13T11:25:48.070 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000108739 2024-09-13T11:25:48.070 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.071 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.071 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:25:48.071 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:47.839388+0000", 2024-09-13T11:25:48.071 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0027619789999999999 2024-09-13T11:25:48.071 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:48.071 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:48.071 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:48.071 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:48.071 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-09-13T11:25:48.071 INFO:teuthology.orchestra.run.smithi001.stdout: "num_ops": 8 2024-09-13T11:25:48.071 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-13T11:25:48.073 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-13T11:25:48.073 INFO:teuthology.orchestra.run.smithi001.stdout:osd.13: {} 2024-09-13T11:25:48.073 INFO:teuthology.orchestra.run.smithi001.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-13T11:25:48.074 INFO:tasks.ceph.osd.15.smithi177.stderr:2024-09-13T11:25:48.070+0000 7f9d6ce77640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26578) UID: 0 2024-09-13T11:25:48.079 INFO:teuthology.orchestra.run.smithi001.stdout:osd.14: {} 2024-09-13T11:25:48.080 INFO:teuthology.orchestra.run.smithi001.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-13T11:25:48.109 INFO:teuthology.orchestra.run.smithi001.stdout:osd.15: {} 2024-09-13T11:25:48.109 INFO:teuthology.orchestra.run.smithi001.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-13T11:25:48.173 INFO:tasks.ceph.osd.2.smithi162.stderr:2024-09-13T11:25:48.172+0000 7fe92b35a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26313) UID: 0 2024-09-13T11:25:48.214 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-13T11:25:48.214 INFO:teuthology.orchestra.run.smithi001.stdout: "ops": [], 2024-09-13T11:25:48.214 INFO:teuthology.orchestra.run.smithi001.stdout: "complaint_time": 30, 2024-09-13T11:25:48.214 INFO:teuthology.orchestra.run.smithi001.stdout: "num_blocked_ops": 0 2024-09-13T11:25:48.214 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-13T11:25:48.232 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-13T11:25:48.275 INFO:tasks.ceph.osd.12.smithi001.stderr:2024-09-13T11:25:48.275+0000 7f892c6e6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26314) UID: 0 2024-09-13T11:25:48.371 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-13T11:25:48.371 INFO:teuthology.orchestra.run.smithi001.stdout: "size": 20, 2024-09-13T11:25:48.371 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 600, 2024-09-13T11:25:48.371 INFO:teuthology.orchestra.run.smithi001.stdout: "ops": [ 2024-09-13T11:25:48.371 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.371 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDPGPushReply(3.1e 84/78 [PushReplyOp(3:79ee54b9:::benchmark_data_smithi177_31116_object11701:head)])", 2024-09-13T11:25:48.372 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:41.407335+0000", 2024-09-13T11:25:48.372 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 126.961511716, 2024-09-13T11:25:48.372 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.51622350400000006, 2024-09-13T11:25:48.372 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:48.372 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:48.372 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-13T11:25:48.372 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:48.372 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.372 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:48.372 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:41.407335+0000", 2024-09-13T11:25:48.372 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:48.372 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.373 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.373 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:48.373 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:41.407335+0000", 2024-09-13T11:25:48.373 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:48.373 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.373 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.373 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:48.373 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:41.407337+0000", 2024-09-13T11:25:48.373 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.505e-06 2024-09-13T11:25:48.373 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.373 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.373 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:48.373 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:41.407343+0000", 2024-09-13T11:25:48.374 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.9089999999999997e-06 2024-09-13T11:25:48.374 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.374 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.374 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:48.374 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:41.921977+0000", 2024-09-13T11:25:48.374 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.51463463499999995 2024-09-13T11:25:48.374 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.374 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.374 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:48.374 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:41.921988+0000", 2024-09-13T11:25:48.374 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.092e-05 2024-09-13T11:25:48.374 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.375 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.375 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:48.375 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:41.922088+0000", 2024-09-13T11:25:48.375 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.9528e-05 2024-09-13T11:25:48.375 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.375 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.375 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:48.375 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:41.923559+0000", 2024-09-13T11:25:48.375 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0014710070000000001 2024-09-13T11:25:48.375 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:48.375 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:48.375 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:48.376 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.376 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.376 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4665.0:60417 3.10 e85/84 3:092808b0:::benchmark_data_smithi177_31116_object60416:head v 85'3295, mlcod=85'3292)", 2024-09-13T11:25:48.376 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:42.174488+0000", 2024-09-13T11:25:48.376 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 126.194359162, 2024-09-13T11:25:48.376 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.387908896, 2024-09-13T11:25:48.376 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:48.376 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:48.376 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:48.376 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:48.376 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.376 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:48.377 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.174488+0000", 2024-09-13T11:25:48.377 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:48.377 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.377 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.377 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:48.377 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.174488+0000", 2024-09-13T11:25:48.377 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:48.377 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.377 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.377 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:48.377 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.174490+0000", 2024-09-13T11:25:48.377 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.9099999999999999e-06 2024-09-13T11:25:48.377 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.378 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.378 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:48.378 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.174527+0000", 2024-09-13T11:25:48.378 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.7543000000000003e-05 2024-09-13T11:25:48.378 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.378 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.378 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:48.378 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.561093+0000", 2024-09-13T11:25:48.378 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.38656578400000002 2024-09-13T11:25:48.378 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.378 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.378 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:48.379 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.561101+0000", 2024-09-13T11:25:48.379 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.6090000000000003e-06 2024-09-13T11:25:48.379 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.379 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.379 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:48.379 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.561143+0000", 2024-09-13T11:25:48.379 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.2435000000000003e-05 2024-09-13T11:25:48.379 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.379 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.379 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:25:48.379 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.561164+0000", 2024-09-13T11:25:48.379 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.1276000000000001e-05 2024-09-13T11:25:48.379 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.380 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.380 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:25:48.380 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.562360+0000", 2024-09-13T11:25:48.380 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00119596 2024-09-13T11:25:48.380 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.380 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.380 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:48.380 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.562397+0000", 2024-09-13T11:25:48.380 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.6378999999999998e-05 2024-09-13T11:25:48.380 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:48.380 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:48.380 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:48.381 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.381 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.381 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDPGPush(3.22 85/78 [PushOp(3:456e8765:::benchmark_data_smithi177_31116_object16212:head, version: 52'1006, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:456e8765:::benchmark_data_smithi177_31116_object16212:head@52'1006, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:25:48.381 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:42.673067+0000", 2024-09-13T11:25:48.381 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 125.69578002, 2024-09-13T11:25:48.381 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.27935889899999999, 2024-09-13T11:25:48.381 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:48.381 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:48.381 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:25:48.381 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:48.381 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.381 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:48.381 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.673067+0000", 2024-09-13T11:25:48.382 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:48.382 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.382 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.382 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:48.382 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.673067+0000", 2024-09-13T11:25:48.382 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:48.382 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.382 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.382 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:48.382 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.673068+0000", 2024-09-13T11:25:48.382 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.9699999999999994e-07 2024-09-13T11:25:48.382 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.383 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.383 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:48.383 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.673090+0000", 2024-09-13T11:25:48.383 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.2416999999999999e-05 2024-09-13T11:25:48.383 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.383 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.383 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:48.383 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.951363+0000", 2024-09-13T11:25:48.383 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.27827291199999998 2024-09-13T11:25:48.383 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.383 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.383 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:48.383 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.951386+0000", 2024-09-13T11:25:48.384 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.2226999999999999e-05 2024-09-13T11:25:48.384 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.385 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.385 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:48.385 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.951581+0000", 2024-09-13T11:25:48.385 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000195713 2024-09-13T11:25:48.385 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.385 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.385 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:25:48.385 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.951605+0000", 2024-09-13T11:25:48.385 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.3767000000000001e-05 2024-09-13T11:25:48.385 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.386 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.386 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:48.386 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.952426+0000", 2024-09-13T11:25:48.386 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00082086599999999996 2024-09-13T11:25:48.386 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:48.386 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:48.386 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:48.386 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.386 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.386 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "replica_scrub(pg: 3.20,from:0'0,to:89'3254,epoch:89/77,start:3:04000000::::0,end:3:045ba805:::benchmark_data_smithi177_31116_object43251:0,chunky:1,deep:0,version:9,allow_preemption:1,priority=5)", 2024-09-13T11:25:48.386 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:46.641364+0000", 2024-09-13T11:25:48.386 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 121.727483339, 2024-09-13T11:25:48.387 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.27141393000000003, 2024-09-13T11:25:48.387 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:48.387 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:48.387 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:25:48.387 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:48.387 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.387 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:48.387 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.641364+0000", 2024-09-13T11:25:48.387 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:48.388 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.388 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.388 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:48.388 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.641364+0000", 2024-09-13T11:25:48.388 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:48.388 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.388 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.388 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:48.388 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.641365+0000", 2024-09-13T11:25:48.388 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.266e-06 2024-09-13T11:25:48.388 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.388 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.388 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:48.389 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.641370+0000", 2024-09-13T11:25:48.389 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.8539999999999996e-06 2024-09-13T11:25:48.389 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.389 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.389 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:48.389 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.912491+0000", 2024-09-13T11:25:48.389 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.27112103100000001 2024-09-13T11:25:48.389 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.389 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.389 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:48.389 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.912520+0000", 2024-09-13T11:25:48.390 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.9379000000000001e-05 2024-09-13T11:25:48.390 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.390 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.390 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:48.390 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.912679+0000", 2024-09-13T11:25:48.390 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00015915800000000001 2024-09-13T11:25:48.390 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.390 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.390 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:25:48.390 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.912714+0000", 2024-09-13T11:25:48.390 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.4687999999999997e-05 2024-09-13T11:25:48.390 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.390 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.390 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:48.391 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.912778+0000", 2024-09-13T11:25:48.391 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.3553999999999995e-05 2024-09-13T11:25:48.391 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:48.391 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:48.391 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:48.391 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.391 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.391 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDPGPush(3.18 89/70 [PushOp(3:1ac6c361:::benchmark_data_smithi177_31116_object44972:head, version: 68'2767, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:1ac6c361:::benchmark_data_smithi177_31116_object44972:head@68'2767, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:25:48.391 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:46.671018+0000", 2024-09-13T11:25:48.391 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 121.697829207, 2024-09-13T11:25:48.391 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.271282931, 2024-09-13T11:25:48.391 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:48.391 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:48.392 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:25:48.392 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:48.392 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.392 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:48.392 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.671018+0000", 2024-09-13T11:25:48.392 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:48.392 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.392 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.392 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:48.392 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.671018+0000", 2024-09-13T11:25:48.392 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:48.392 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.392 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.392 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:48.393 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.671020+0000", 2024-09-13T11:25:48.393 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.018e-06 2024-09-13T11:25:48.393 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.393 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.393 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:48.393 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.671097+0000", 2024-09-13T11:25:48.393 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.7218999999999994e-05 2024-09-13T11:25:48.393 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.393 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.393 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:48.393 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.939773+0000", 2024-09-13T11:25:48.393 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.26867550200000001 2024-09-13T11:25:48.393 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.394 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.394 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:48.394 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.939791+0000", 2024-09-13T11:25:48.394 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.8185999999999998e-05 2024-09-13T11:25:48.394 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.394 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.394 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:48.394 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.939926+0000", 2024-09-13T11:25:48.394 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000135291 2024-09-13T11:25:48.394 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.394 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.394 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:25:48.395 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.939938+0000", 2024-09-13T11:25:48.395 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2393e-05 2024-09-13T11:25:48.395 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.395 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.395 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:48.395 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.942301+0000", 2024-09-13T11:25:48.395 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.002362322 2024-09-13T11:25:48.395 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:48.395 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:48.395 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:48.395 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.395 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.396 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "replica_scrub(pg: 3.20,from:0'0,to:90'3264,epoch:90/77,start:3:0650aac8:::benchmark_data_smithi177_31116_object1734:0,end:3:06af01e3:::benchmark_data_smithi177_31116_object31572:0,chunky:1,deep:0,version:9,allow_preemption:1,priority=5)", 2024-09-13T11:25:48.396 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:47.380615+0000", 2024-09-13T11:25:48.396 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 120.98823178400001, 2024-09-13T11:25:48.396 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.51951398400000004, 2024-09-13T11:25:48.396 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:48.396 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:48.396 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:25:48.396 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:48.396 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.396 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:48.396 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:47.380615+0000", 2024-09-13T11:25:48.396 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:48.396 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.396 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.397 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:48.397 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:47.380615+0000", 2024-09-13T11:25:48.397 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:48.397 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.397 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.397 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:48.397 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:47.380617+0000", 2024-09-13T11:25:48.397 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.671e-06 2024-09-13T11:25:48.397 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.397 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.397 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:48.397 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:47.380623+0000", 2024-09-13T11:25:48.398 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.1190000000000002e-06 2024-09-13T11:25:48.398 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.398 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.398 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:48.398 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:47.900050+0000", 2024-09-13T11:25:48.398 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.51942737400000005 2024-09-13T11:25:48.398 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.398 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.398 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:48.398 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:47.900057+0000", 2024-09-13T11:25:48.398 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.4359999999999999e-06 2024-09-13T11:25:48.398 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.399 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.399 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:48.399 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:47.900101+0000", 2024-09-13T11:25:48.399 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.4675000000000002e-05 2024-09-13T11:25:48.399 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.399 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.399 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:25:48.399 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:47.900110+0000", 2024-09-13T11:25:48.399 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.0569999999999993e-06 2024-09-13T11:25:48.399 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.399 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.399 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:48.399 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:47.900129+0000", 2024-09-13T11:25:48.399 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.9652000000000001e-05 2024-09-13T11:25:48.400 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:48.400 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:48.400 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:48.400 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.400 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.400 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDPGPush(3.21 91/78 [PushOp(3:8492321f:::benchmark_data_smithi177_31116_object13767:head, version: 50'840, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:8492321f:::benchmark_data_smithi177_31116_object13767:head@50'840, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:25:48.400 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:48.319251+0000", 2024-09-13T11:25:48.400 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 120.049596255, 2024-09-13T11:25:48.400 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.60903802200000001, 2024-09-13T11:25:48.400 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:48.400 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:48.400 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:25:48.401 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:48.401 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.401 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:48.401 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:48.319251+0000", 2024-09-13T11:25:48.401 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:48.401 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.401 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.401 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:48.401 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:48.319251+0000", 2024-09-13T11:25:48.401 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:48.401 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.401 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.402 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:48.402 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:48.319252+0000", 2024-09-13T11:25:48.402 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2640000000000001e-06 2024-09-13T11:25:48.402 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.402 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.402 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:48.402 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:48.319363+0000", 2024-09-13T11:25:48.402 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000110559 2024-09-13T11:25:48.402 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.402 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.402 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:48.402 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:48.927115+0000", 2024-09-13T11:25:48.402 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.60775277800000005 2024-09-13T11:25:48.403 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.403 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.403 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:48.403 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:48.927144+0000", 2024-09-13T11:25:48.403 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.8861e-05 2024-09-13T11:25:48.403 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.403 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.403 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:48.403 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:48.927318+0000", 2024-09-13T11:25:48.403 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00017399000000000001 2024-09-13T11:25:48.403 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.403 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.403 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:25:48.404 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:48.927340+0000", 2024-09-13T11:25:48.404 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.1433999999999998e-05 2024-09-13T11:25:48.404 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.404 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.404 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:48.404 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:48.928289+0000", 2024-09-13T11:25:48.404 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00094913599999999995 2024-09-13T11:25:48.404 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:48.404 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:48.404 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:48.404 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.405 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.405 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:70749 3.1e 3:7bd7bfc6:::benchmark_data_smithi177_31116_object70748:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95)", 2024-09-13T11:25:48.405 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:53.156250+0000", 2024-09-13T11:25:48.405 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 115.212596899, 2024-09-13T11:25:48.405 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.22394481799999999, 2024-09-13T11:25:48.405 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:48.405 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:48.405 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:48.405 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:25:48.405 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:25:48.405 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:48.405 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 70749 2024-09-13T11:25:48.405 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.405 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:48.406 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.406 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:48.406 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:53.156250+0000", 2024-09-13T11:25:48.406 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:48.406 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.406 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.406 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:48.406 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:53.156250+0000", 2024-09-13T11:25:48.406 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:48.406 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.406 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.406 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:48.406 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:53.156252+0000", 2024-09-13T11:25:48.407 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.3460000000000001e-06 2024-09-13T11:25:48.407 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.407 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.407 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:48.407 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:53.156336+0000", 2024-09-13T11:25:48.407 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.3714999999999994e-05 2024-09-13T11:25:48.407 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.407 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.407 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:48.407 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:53.156347+0000", 2024-09-13T11:25:48.407 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0903999999999999e-05 2024-09-13T11:25:48.407 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.408 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.408 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:48.408 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:53.156352+0000", 2024-09-13T11:25:48.408 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.9139999999999999e-06 2024-09-13T11:25:48.408 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.408 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.408 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:48.408 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:53.158028+0000", 2024-09-13T11:25:48.408 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.001675958 2024-09-13T11:25:48.408 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.408 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.408 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:25:48.408 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:53.158154+0000", 2024-09-13T11:25:48.409 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00012653 2024-09-13T11:25:48.409 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.409 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.409 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 4,5", 2024-09-13T11:25:48.409 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:53.158277+0000", 2024-09-13T11:25:48.409 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000122168 2024-09-13T11:25:48.409 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.409 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.409 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:25:48.409 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:53.159585+0000", 2024-09-13T11:25:48.409 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.001308528 2024-09-13T11:25:48.409 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.409 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.410 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:25:48.410 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:53.163172+0000", 2024-09-13T11:25:48.410 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0035869690000000002 2024-09-13T11:25:48.410 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.410 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.410 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:25:48.410 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:53.379730+0000", 2024-09-13T11:25:48.410 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.21655822599999999 2024-09-13T11:25:48.410 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.411 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.411 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:25:48.411 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:53.379810+0000", 2024-09-13T11:25:48.411 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.0041000000000002e-05 2024-09-13T11:25:48.411 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.411 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.411 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:48.411 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:53.380195+0000", 2024-09-13T11:25:48.411 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000384519 2024-09-13T11:25:48.411 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:48.411 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:48.411 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:48.411 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.412 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.412 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:83239 3.e 3:74547f17:::benchmark_data_smithi177_31116_object83238:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106)", 2024-09-13T11:25:48.412 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:24:04.580085+0000", 2024-09-13T11:25:48.412 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 103.788762174, 2024-09-13T11:25:48.412 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.33785762600000002, 2024-09-13T11:25:48.412 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:48.412 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:48.412 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:48.412 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:25:48.412 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:25:48.412 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:48.413 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 83239 2024-09-13T11:25:48.413 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.413 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:48.413 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.413 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:48.413 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:04.580085+0000", 2024-09-13T11:25:48.413 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:48.413 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.413 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.413 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:48.413 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:04.580085+0000", 2024-09-13T11:25:48.413 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:48.413 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.414 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.414 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:48.414 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:04.580089+0000", 2024-09-13T11:25:48.414 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.053e-06 2024-09-13T11:25:48.414 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.414 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.414 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:48.414 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:04.580166+0000", 2024-09-13T11:25:48.414 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.7102999999999997e-05 2024-09-13T11:25:48.414 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.414 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.414 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:48.414 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:04.580196+0000", 2024-09-13T11:25:48.415 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.0196e-05 2024-09-13T11:25:48.415 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.415 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.415 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:48.415 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:04.580212+0000", 2024-09-13T11:25:48.415 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.535e-05 2024-09-13T11:25:48.415 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.415 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.415 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:48.415 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:04.580336+0000", 2024-09-13T11:25:48.415 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00012431899999999999 2024-09-13T11:25:48.415 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.415 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.416 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:25:48.416 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:04.580575+0000", 2024-09-13T11:25:48.416 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00023924100000000001 2024-09-13T11:25:48.416 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.416 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.416 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 1", 2024-09-13T11:25:48.416 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:04.580798+0000", 2024-09-13T11:25:48.416 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00022301499999999999 2024-09-13T11:25:48.416 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.416 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.416 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:25:48.416 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:04.583404+0000", 2024-09-13T11:25:48.416 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0026056120000000002 2024-09-13T11:25:48.416 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.417 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.417 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:25:48.417 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:04.917768+0000", 2024-09-13T11:25:48.417 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.33436387499999998 2024-09-13T11:25:48.417 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.417 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.417 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:25:48.417 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:04.917859+0000", 2024-09-13T11:25:48.417 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.1732999999999997e-05 2024-09-13T11:25:48.417 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.417 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.417 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:48.417 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:04.917942+0000", 2024-09-13T11:25:48.418 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.3128999999999996e-05 2024-09-13T11:25:48.418 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:48.418 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:48.418 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:48.418 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.418 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.418 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:110271 3.2d 3.3c2681ad (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e119)", 2024-09-13T11:25:48.418 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:24:25.403006+0000", 2024-09-13T11:25:48.418 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 82.965840689999993, 2024-09-13T11:25:48.418 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.68768755999999998, 2024-09-13T11:25:48.418 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:48.418 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:48.418 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-13T11:25:48.418 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:25:48.419 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:25:48.419 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:48.419 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 110271 2024-09-13T11:25:48.419 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.419 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:48.419 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.419 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:48.419 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:25.403006+0000", 2024-09-13T11:25:48.419 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:48.419 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.419 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.419 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:48.420 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:25.403006+0000", 2024-09-13T11:25:48.420 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:48.420 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.420 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.420 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:48.420 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:25.403011+0000", 2024-09-13T11:25:48.420 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.5199999999999999e-06 2024-09-13T11:25:48.420 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.420 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.420 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:48.420 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:25.403065+0000", 2024-09-13T11:25:48.420 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.4478999999999998e-05 2024-09-13T11:25:48.421 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.421 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.421 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:48.421 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:25.403097+0000", 2024-09-13T11:25:48.421 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.1636e-05 2024-09-13T11:25:48.421 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.421 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.421 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:48.421 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:25.403121+0000", 2024-09-13T11:25:48.421 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.3924e-05 2024-09-13T11:25:48.421 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.421 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.421 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:48.422 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:25.403216+0000", 2024-09-13T11:25:48.422 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.5557999999999995e-05 2024-09-13T11:25:48.422 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.422 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.422 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for peered", 2024-09-13T11:25:48.422 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:25.403234+0000", 2024-09-13T11:25:48.422 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.7104000000000001e-05 2024-09-13T11:25:48.422 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.422 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.422 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:48.422 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:26.090684+0000", 2024-09-13T11:25:48.422 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.68745000700000003 2024-09-13T11:25:48.423 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.423 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.423 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:48.423 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:26.090694+0000", 2024-09-13T11:25:48.423 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0332e-05 2024-09-13T11:25:48.423 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:48.423 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:48.423 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:48.423 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.423 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.423 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4665.0:129901 3.20 e134/77 3:067f5dfc:::benchmark_data_smithi177_31116_object129900:head v 134'4294, mlcod=134'4291)", 2024-09-13T11:25:48.423 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:24:47.145530+0000", 2024-09-13T11:25:48.424 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 61.223316615000002, 2024-09-13T11:25:48.424 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.333688349, 2024-09-13T11:25:48.424 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:48.424 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:48.424 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:48.424 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:48.424 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.424 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:48.424 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:47.145530+0000", 2024-09-13T11:25:48.424 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:48.424 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.425 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.425 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:48.425 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:47.145530+0000", 2024-09-13T11:25:48.425 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:48.425 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.425 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.425 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:48.425 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:47.145532+0000", 2024-09-13T11:25:48.425 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.7999999999999999e-06 2024-09-13T11:25:48.425 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.425 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.425 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:48.426 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:47.145557+0000", 2024-09-13T11:25:48.426 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.4678999999999999e-05 2024-09-13T11:25:48.426 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.426 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.426 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:48.426 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:47.477539+0000", 2024-09-13T11:25:48.426 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.33198164099999999 2024-09-13T11:25:48.426 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.426 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.426 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:48.426 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:47.477544+0000", 2024-09-13T11:25:48.426 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.9059999999999999e-06 2024-09-13T11:25:48.427 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.427 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.427 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:48.427 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:47.477597+0000", 2024-09-13T11:25:48.427 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.295e-05 2024-09-13T11:25:48.427 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.427 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.427 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:25:48.428 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:47.477619+0000", 2024-09-13T11:25:48.428 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.1835999999999999e-05 2024-09-13T11:25:48.428 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.428 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.428 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:25:48.428 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:47.479175+0000", 2024-09-13T11:25:48.428 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0015560470000000001 2024-09-13T11:25:48.428 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.428 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.428 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:48.428 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:47.479219+0000", 2024-09-13T11:25:48.428 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.3489999999999999e-05 2024-09-13T11:25:48.428 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:48.429 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:48.429 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:48.429 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.429 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.429 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:140840 3.21 3:87ae3db6:::benchmark_data_smithi177_31116_object537:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e144)", 2024-09-13T11:25:48.429 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:24:58.810939+0000", 2024-09-13T11:25:48.429 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 49.557907501999999, 2024-09-13T11:25:48.429 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.25628309300000002, 2024-09-13T11:25:48.429 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:48.429 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:48.429 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:48.429 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:25:48.429 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:25:48.429 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:48.430 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 140840 2024-09-13T11:25:48.430 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.430 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:48.430 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.430 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:48.430 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.810939+0000", 2024-09-13T11:25:48.430 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:48.430 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.430 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.430 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:48.430 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.810939+0000", 2024-09-13T11:25:48.430 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:48.430 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.431 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.431 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:48.431 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.810942+0000", 2024-09-13T11:25:48.431 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.6259999999999999e-06 2024-09-13T11:25:48.431 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.431 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.431 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:48.431 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.810950+0000", 2024-09-13T11:25:48.431 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.9500000000000001e-06 2024-09-13T11:25:48.431 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.431 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.431 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:48.431 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.810970+0000", 2024-09-13T11:25:48.432 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.9539999999999999e-05 2024-09-13T11:25:48.432 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.432 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.432 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:48.432 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.810978+0000", 2024-09-13T11:25:48.432 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.8680000000000005e-06 2024-09-13T11:25:48.432 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.432 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.432 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:48.432 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.811060+0000", 2024-09-13T11:25:48.432 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.1068000000000004e-05 2024-09-13T11:25:48.432 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.432 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.433 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:25:48.433 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.811174+0000", 2024-09-13T11:25:48.433 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000114385 2024-09-13T11:25:48.433 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.433 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.433 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 8", 2024-09-13T11:25:48.433 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.811255+0000", 2024-09-13T11:25:48.433 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.1298000000000007e-05 2024-09-13T11:25:48.433 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.433 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.433 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:25:48.433 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.812869+0000", 2024-09-13T11:25:48.433 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.001614234 2024-09-13T11:25:48.433 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.433 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.434 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:25:48.434 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:59.067155+0000", 2024-09-13T11:25:48.434 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.25428578299999999 2024-09-13T11:25:48.434 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.434 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.434 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:25:48.434 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:59.067192+0000", 2024-09-13T11:25:48.434 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.659e-05 2024-09-13T11:25:48.434 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.434 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.434 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:48.434 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:59.067223+0000", 2024-09-13T11:25:48.434 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.0750999999999997e-05 2024-09-13T11:25:48.435 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:48.435 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:48.435 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:48.435 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.435 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.435 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop_reply(client.4665.0:188185 3.22 e174/122 ondisk, result = 0)", 2024-09-13T11:25:48.435 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:25:36.235023+0000", 2024-09-13T11:25:48.435 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 12.133823719, 2024-09-13T11:25:48.435 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.23136179600000001, 2024-09-13T11:25:48.435 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:48.435 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:48.435 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:25:48.435 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:48.436 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.436 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:48.436 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:36.235023+0000", 2024-09-13T11:25:48.436 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:48.436 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.436 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.436 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:48.436 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:36.235023+0000", 2024-09-13T11:25:48.436 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:48.436 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.436 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.436 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:48.436 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:36.235024+0000", 2024-09-13T11:25:48.436 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.212e-06 2024-09-13T11:25:48.437 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.437 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.437 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:48.437 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:36.235029+0000", 2024-09-13T11:25:48.437 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.4839999999999998e-06 2024-09-13T11:25:48.437 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.437 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.437 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:48.437 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:36.466056+0000", 2024-09-13T11:25:48.437 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.23102687 2024-09-13T11:25:48.437 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.437 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.437 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:48.437 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:36.466061+0000", 2024-09-13T11:25:48.438 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.8629999999999999e-06 2024-09-13T11:25:48.438 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.438 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.438 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:48.438 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:36.466133+0000", 2024-09-13T11:25:48.438 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.1999e-05 2024-09-13T11:25:48.438 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.438 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.438 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:25:48.438 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:36.466148+0000", 2024-09-13T11:25:48.438 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.4800000000000001e-05 2024-09-13T11:25:48.438 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.438 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.438 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:48.439 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:36.466385+0000", 2024-09-13T11:25:48.439 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00023756800000000001 2024-09-13T11:25:48.439 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:48.439 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:48.439 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:48.439 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.439 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.439 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:188435 3.22 3:446e4bc0:::benchmark_data_smithi177_31116_object48132:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e174)", 2024-09-13T11:25:48.439 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:25:36.478076+0000", 2024-09-13T11:25:48.439 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 11.890770933000001, 2024-09-13T11:25:48.439 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.22579131699999999, 2024-09-13T11:25:48.439 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:48.439 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:48.440 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:48.440 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:25:48.440 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:25:48.440 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:48.440 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 188435 2024-09-13T11:25:48.440 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.440 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:48.440 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.440 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:48.440 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:36.478076+0000", 2024-09-13T11:25:48.440 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:48.440 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.440 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.441 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:48.441 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:36.478076+0000", 2024-09-13T11:25:48.441 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:48.441 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.441 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.441 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:48.441 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:36.478078+0000", 2024-09-13T11:25:48.441 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.385e-06 2024-09-13T11:25:48.441 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.441 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.441 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:48.441 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:36.478085+0000", 2024-09-13T11:25:48.441 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.8900000000000001e-06 2024-09-13T11:25:48.442 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.442 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.442 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:48.442 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:36.478102+0000", 2024-09-13T11:25:48.442 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.6422999999999998e-05 2024-09-13T11:25:48.442 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.442 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.442 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:48.442 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:36.478109+0000", 2024-09-13T11:25:48.442 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.7179999999999996e-06 2024-09-13T11:25:48.442 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.442 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.442 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:48.442 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:36.478173+0000", 2024-09-13T11:25:48.443 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.3398000000000005e-05 2024-09-13T11:25:48.443 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.443 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.443 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:25:48.443 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:36.478260+0000", 2024-09-13T11:25:48.443 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.7022999999999997e-05 2024-09-13T11:25:48.443 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.443 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.443 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 5", 2024-09-13T11:25:48.443 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:36.478323+0000", 2024-09-13T11:25:48.443 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.2811000000000006e-05 2024-09-13T11:25:48.443 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.443 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.443 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:25:48.444 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:36.479692+0000", 2024-09-13T11:25:48.444 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.001369264 2024-09-13T11:25:48.444 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.444 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.444 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:25:48.444 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:36.703793+0000", 2024-09-13T11:25:48.444 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.224101257 2024-09-13T11:25:48.444 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.444 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.444 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:25:48.444 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:36.703823+0000", 2024-09-13T11:25:48.444 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.9714999999999999e-05 2024-09-13T11:25:48.444 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.445 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.445 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:48.445 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:36.703867+0000", 2024-09-13T11:25:48.445 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.4433e-05 2024-09-13T11:25:48.445 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:48.445 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:48.445 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:48.445 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.445 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.445 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop_reply(client.4665.0:188435 3.22 e174/122 ondisk, result = 0)", 2024-09-13T11:25:48.445 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:25:36.481530+0000", 2024-09-13T11:25:48.445 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 11.887317393, 2024-09-13T11:25:48.446 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.22234125699999999, 2024-09-13T11:25:48.446 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:48.446 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:48.446 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:25:48.446 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:48.446 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.446 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:48.446 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:36.481530+0000", 2024-09-13T11:25:48.446 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:48.446 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.446 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.446 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:48.446 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:36.481530+0000", 2024-09-13T11:25:48.447 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:48.447 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.447 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.447 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:48.447 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:36.481531+0000", 2024-09-13T11:25:48.447 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.6160000000000001e-06 2024-09-13T11:25:48.447 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.447 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.447 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:48.447 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:36.481536+0000", 2024-09-13T11:25:48.447 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.2460000000000003e-06 2024-09-13T11:25:48.447 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.447 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.447 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:48.448 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:36.703742+0000", 2024-09-13T11:25:48.448 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.22220578199999999 2024-09-13T11:25:48.448 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.448 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.448 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:48.448 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:36.703747+0000", 2024-09-13T11:25:48.448 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.4889999999999997e-06 2024-09-13T11:25:48.448 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.448 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.448 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:48.448 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:36.703786+0000", 2024-09-13T11:25:48.448 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.9097000000000002e-05 2024-09-13T11:25:48.448 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.449 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.449 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:25:48.449 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:36.703790+0000", 2024-09-13T11:25:48.449 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.233e-06 2024-09-13T11:25:48.449 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.449 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.449 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:48.449 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:36.703871+0000", 2024-09-13T11:25:48.449 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.0793999999999999e-05 2024-09-13T11:25:48.449 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:48.449 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:48.449 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:48.449 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.449 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.450 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4665.0:191286 3.18 e175/91 3:1ffc0b26:::benchmark_data_smithi177_31116_object50983:head v 175'7339, mlcod=175'7337)", 2024-09-13T11:25:48.450 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:25:39.153114+0000", 2024-09-13T11:25:48.450 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 9.21573289, 2024-09-13T11:25:48.450 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.23200507400000001, 2024-09-13T11:25:48.450 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:48.450 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:48.450 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:48.450 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:48.450 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.450 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:48.450 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:39.153114+0000", 2024-09-13T11:25:48.450 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:48.450 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.450 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.451 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:48.451 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:39.153114+0000", 2024-09-13T11:25:48.451 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:48.451 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.451 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.451 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:48.451 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:39.153117+0000", 2024-09-13T11:25:48.451 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.9170000000000002e-06 2024-09-13T11:25:48.451 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.451 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.451 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:48.451 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:39.153130+0000", 2024-09-13T11:25:48.451 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2493e-05 2024-09-13T11:25:48.451 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.452 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.452 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:48.452 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:39.381951+0000", 2024-09-13T11:25:48.452 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.22882103000000001 2024-09-13T11:25:48.452 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.452 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.452 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:48.452 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:39.381956+0000", 2024-09-13T11:25:48.452 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.1359999999999996e-06 2024-09-13T11:25:48.452 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.452 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.452 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:48.452 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:39.382030+0000", 2024-09-13T11:25:48.452 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.4270999999999994e-05 2024-09-13T11:25:48.453 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.453 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.453 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:25:48.453 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:39.382054+0000", 2024-09-13T11:25:48.453 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.4142999999999999e-05 2024-09-13T11:25:48.453 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.453 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.453 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:25:48.453 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:39.385086+0000", 2024-09-13T11:25:48.453 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00303163 2024-09-13T11:25:48.453 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.453 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.453 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:48.453 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:39.385119+0000", 2024-09-13T11:25:48.454 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.3454000000000002e-05 2024-09-13T11:25:48.454 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:48.454 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:48.454 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:48.454 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.454 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.454 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4665.0:196438 3.20 e178/77 3:0573f1a4:::benchmark_data_smithi177_31116_object56135:head v 178'5356, mlcod=178'5353)", 2024-09-13T11:25:48.454 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:25:43.522272+0000", 2024-09-13T11:25:48.454 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 4.8465748, 2024-09-13T11:25:48.454 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.23723141, 2024-09-13T11:25:48.454 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:48.454 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:48.454 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:48.455 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:48.455 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.455 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:48.455 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:43.522272+0000", 2024-09-13T11:25:48.455 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:48.455 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.455 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.455 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:48.455 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:43.522272+0000", 2024-09-13T11:25:48.455 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:48.455 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.455 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.455 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:48.455 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:43.522274+0000", 2024-09-13T11:25:48.456 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.573e-06 2024-09-13T11:25:48.456 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.456 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.456 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:48.456 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:43.522281+0000", 2024-09-13T11:25:48.456 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.5490000000000001e-06 2024-09-13T11:25:48.456 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.456 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.456 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:48.456 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:43.756836+0000", 2024-09-13T11:25:48.456 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.23455496000000001 2024-09-13T11:25:48.456 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.456 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.456 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:48.457 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:43.756843+0000", 2024-09-13T11:25:48.457 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.4620000000000001e-06 2024-09-13T11:25:48.457 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.457 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.457 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:48.457 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:43.756914+0000", 2024-09-13T11:25:48.457 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.1276e-05 2024-09-13T11:25:48.457 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.457 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.457 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:25:48.457 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:43.756938+0000", 2024-09-13T11:25:48.457 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.4281999999999999e-05 2024-09-13T11:25:48.457 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.457 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.458 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:25:48.458 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:43.759408+0000", 2024-09-13T11:25:48.458 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0024699449999999999 2024-09-13T11:25:48.458 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.458 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.458 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:48.458 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:43.759504+0000", 2024-09-13T11:25:48.458 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.5363000000000002e-05 2024-09-13T11:25:48.458 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:48.458 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:48.458 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:48.458 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.458 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.459 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:201234 3.21 3:85402570:::benchmark_data_smithi177_31116_object60931:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e181)", 2024-09-13T11:25:48.459 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:25:47.835618+0000", 2024-09-13T11:25:48.459 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 0.53322933699999997, 2024-09-13T11:25:48.459 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.26153554899999998, 2024-09-13T11:25:48.459 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:48.459 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:48.459 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:48.459 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:25:48.459 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:25:48.459 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:48.459 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 201234 2024-09-13T11:25:48.459 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.459 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:48.459 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.460 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:48.460 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:47.835618+0000", 2024-09-13T11:25:48.460 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:48.460 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.460 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.460 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:48.460 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:47.835618+0000", 2024-09-13T11:25:48.460 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:48.460 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.460 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.460 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:48.460 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:47.835619+0000", 2024-09-13T11:25:48.460 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.638e-06 2024-09-13T11:25:48.461 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.461 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.461 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:48.461 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:47.835627+0000", 2024-09-13T11:25:48.461 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.0150000000000003e-06 2024-09-13T11:25:48.461 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.461 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.461 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:48.461 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:47.835639+0000", 2024-09-13T11:25:48.461 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.1939e-05 2024-09-13T11:25:48.461 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.461 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.461 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:48.461 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:47.835644+0000", 2024-09-13T11:25:48.462 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.006e-06 2024-09-13T11:25:48.462 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.462 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.462 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:48.462 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:47.835696+0000", 2024-09-13T11:25:48.462 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.1668000000000001e-05 2024-09-13T11:25:48.462 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.462 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.462 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:25:48.462 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:47.835775+0000", 2024-09-13T11:25:48.462 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.8737000000000002e-05 2024-09-13T11:25:48.462 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.462 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.462 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 8", 2024-09-13T11:25:48.462 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:47.835857+0000", 2024-09-13T11:25:48.463 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.2508000000000004e-05 2024-09-13T11:25:48.463 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.463 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.463 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:25:48.463 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:47.837366+0000", 2024-09-13T11:25:48.463 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.001509292 2024-09-13T11:25:48.463 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.463 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.463 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:25:48.463 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:48.097051+0000", 2024-09-13T11:25:48.463 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.25968466699999998 2024-09-13T11:25:48.463 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.463 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.463 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:25:48.464 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:48.097109+0000", 2024-09-13T11:25:48.464 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.7778000000000001e-05 2024-09-13T11:25:48.464 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.464 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.464 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:48.464 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:48.097153+0000", 2024-09-13T11:25:48.464 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.4301000000000002e-05 2024-09-13T11:25:48.464 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:48.464 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:48.464 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:48.464 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.464 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.464 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:201238 3.15 3:ae4e8f1e:::benchmark_data_smithi177_31116_object60935:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e181)", 2024-09-13T11:25:48.464 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:25:47.836319+0000", 2024-09-13T11:25:48.465 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 0.53252829899999998, 2024-09-13T11:25:48.465 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.216262386, 2024-09-13T11:25:48.465 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:48.465 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:48.465 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:48.465 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:25:48.465 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:25:48.465 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:48.465 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 201238 2024-09-13T11:25:48.465 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.465 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:48.465 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.465 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:48.465 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:47.836319+0000", 2024-09-13T11:25:48.466 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:48.466 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.466 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.466 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:48.466 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:47.836319+0000", 2024-09-13T11:25:48.466 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:48.466 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.466 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.466 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:48.466 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:47.836320+0000", 2024-09-13T11:25:48.466 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.387e-06 2024-09-13T11:25:48.466 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.466 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.467 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:48.467 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:47.836325+0000", 2024-09-13T11:25:48.467 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.7720000000000002e-06 2024-09-13T11:25:48.467 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.467 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.467 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:48.467 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:47.836337+0000", 2024-09-13T11:25:48.467 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2499e-05 2024-09-13T11:25:48.467 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.467 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.467 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:48.467 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:47.836345+0000", 2024-09-13T11:25:48.467 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.5939999999999996e-06 2024-09-13T11:25:48.467 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.468 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.468 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:48.468 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:47.836399+0000", 2024-09-13T11:25:48.468 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.4527000000000002e-05 2024-09-13T11:25:48.468 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.468 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.468 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:25:48.468 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:47.836518+0000", 2024-09-13T11:25:48.468 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000118047 2024-09-13T11:25:48.468 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.468 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.468 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 3", 2024-09-13T11:25:48.468 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:47.836626+0000", 2024-09-13T11:25:48.469 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000108739 2024-09-13T11:25:48.469 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.469 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.469 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:25:48.469 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:47.839388+0000", 2024-09-13T11:25:48.469 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0027619789999999999 2024-09-13T11:25:48.469 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.469 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.469 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:25:48.469 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:48.052519+0000", 2024-09-13T11:25:48.469 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.21313048300000001 2024-09-13T11:25:48.469 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.469 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.469 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:25:48.470 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:48.052547+0000", 2024-09-13T11:25:48.470 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.8031e-05 2024-09-13T11:25:48.470 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.470 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.470 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:48.470 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:48.052581+0000", 2024-09-13T11:25:48.470 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.4328000000000001e-05 2024-09-13T11:25:48.470 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:48.470 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:48.470 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:48.470 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.470 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.470 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:201251 3.15 3:a896c150:::benchmark_data_smithi177_31116_object60948:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e181)", 2024-09-13T11:25:48.471 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:25:47.851517+0000", 2024-09-13T11:25:48.471 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 0.51732962100000002, 2024-09-13T11:25:48.471 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.215458226, 2024-09-13T11:25:48.471 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:48.471 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:48.471 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:48.471 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:25:48.471 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:25:48.471 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:48.471 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 201251 2024-09-13T11:25:48.471 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.471 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:48.471 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.471 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:48.472 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:47.851517+0000", 2024-09-13T11:25:48.472 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:48.472 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.472 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.472 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:48.472 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:47.851517+0000", 2024-09-13T11:25:48.472 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:48.472 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.472 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.472 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:48.472 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:47.851519+0000", 2024-09-13T11:25:48.472 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.0650000000000001e-06 2024-09-13T11:25:48.472 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.472 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.473 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:48.473 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:47.851525+0000", 2024-09-13T11:25:48.473 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.7799999999999997e-06 2024-09-13T11:25:48.473 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.473 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.473 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:48.473 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:47.851536+0000", 2024-09-13T11:25:48.473 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.1155000000000001e-05 2024-09-13T11:25:48.473 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.473 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.473 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:48.473 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:47.851545+0000", 2024-09-13T11:25:48.473 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.4509999999999992e-06 2024-09-13T11:25:48.473 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.474 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.474 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:48.474 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:47.851724+0000", 2024-09-13T11:25:48.474 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000179011 2024-09-13T11:25:48.474 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.474 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.474 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:25:48.474 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:47.851868+0000", 2024-09-13T11:25:48.474 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00014447599999999999 2024-09-13T11:25:48.474 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.474 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.474 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 3", 2024-09-13T11:25:48.474 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:47.851996+0000", 2024-09-13T11:25:48.475 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00012794099999999999 2024-09-13T11:25:48.475 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.475 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.475 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:25:48.475 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:47.854695+0000", 2024-09-13T11:25:48.475 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.002698523 2024-09-13T11:25:48.475 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.475 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.475 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:25:48.475 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:48.066888+0000", 2024-09-13T11:25:48.475 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.21219289899999999 2024-09-13T11:25:48.475 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.475 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.475 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:25:48.476 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:48.066924+0000", 2024-09-13T11:25:48.476 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.659e-05 2024-09-13T11:25:48.476 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.476 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.476 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:48.476 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:48.066976+0000", 2024-09-13T11:25:48.476 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.1335000000000002e-05 2024-09-13T11:25:48.476 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:48.476 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:48.476 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:48.476 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:48.476 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:48.476 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-13T11:25:48.477 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-09-13T11:25:48.388+0000 7f97b59fa640 -1 osd.6 181 handle_osd_map injecting map crc failure 2024-09-13T11:25:48.478 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-13T11:25:48.479 INFO:tasks.ceph.osd.10.smithi162.stderr:2024-09-13T11:25:48.476+0000 7fb59c9f0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26317) UID: 0 2024-09-13T11:25:48.578 INFO:tasks.ceph.osd.11.smithi177.stderr:2024-09-13T11:25:48.574+0000 7f24792c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26580) UID: 0 2024-09-13T11:25:48.658 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-13T11:25:48.658 INFO:teuthology.orchestra.run.smithi001.stdout: "ops": [ 2024-09-13T11:25:48.658 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.658 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:202145 3.1a 3:5fe5b9e3:::benchmark_data_smithi177_31116_object61842:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e182)", 2024-09-13T11:25:48.658 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:25:48.598830+0000", 2024-09-13T11:25:48.658 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 0.058713577000000003, 2024-09-13T11:25:48.658 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.058731495000000002, 2024-09-13T11:25:48.658 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:48.658 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:48.658 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "waiting for sub ops", 2024-09-13T11:25:48.658 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:25:48.659 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:25:48.659 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:48.659 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 202145 2024-09-13T11:25:48.659 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.659 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:48.659 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.659 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:48.659 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:48.598830+0000", 2024-09-13T11:25:48.659 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:48.659 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.659 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.659 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:48.659 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:48.598830+0000", 2024-09-13T11:25:48.659 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:48.659 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.660 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.660 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:48.660 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:48.598833+0000", 2024-09-13T11:25:48.660 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.79e-06 2024-09-13T11:25:48.660 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.660 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.660 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:48.660 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:48.598840+0000", 2024-09-13T11:25:48.660 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.5100000000000001e-06 2024-09-13T11:25:48.660 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.660 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.660 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:48.660 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:48.598858+0000", 2024-09-13T11:25:48.661 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.7294000000000001e-05 2024-09-13T11:25:48.661 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.661 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.661 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:48.661 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:48.598865+0000", 2024-09-13T11:25:48.661 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.7219999999999999e-06 2024-09-13T11:25:48.661 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.661 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.661 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:48.661 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:48.598931+0000", 2024-09-13T11:25:48.661 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.5734000000000005e-05 2024-09-13T11:25:48.661 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.661 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.661 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:25:48.662 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:48.599037+0000", 2024-09-13T11:25:48.662 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000105806 2024-09-13T11:25:48.662 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.662 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.662 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 9", 2024-09-13T11:25:48.662 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:48.599110+0000", 2024-09-13T11:25:48.662 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.3464e-05 2024-09-13T11:25:48.662 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.662 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.662 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:25:48.662 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:48.601703+0000", 2024-09-13T11:25:48.662 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0025923729999999998 2024-09-13T11:25:48.662 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:48.662 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:48.662 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:48.663 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:48.663 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-09-13T11:25:48.663 INFO:teuthology.orchestra.run.smithi001.stdout: "num_ops": 1 2024-09-13T11:25:48.663 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-13T11:25:48.673 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-13T11:25:48.720 INFO:tasks.ceph.osd.12.smithi001.stderr:2024-09-13T11:25:48.719+0000 7f892c6e6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26314) UID: 0 2024-09-13T11:25:48.780 INFO:tasks.ceph.osd.5.smithi092.stderr:2024-09-13T11:25:48.777+0000 7f6f0b705640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26346) UID: 0 2024-09-13T11:25:48.802 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-13T11:25:48.803 INFO:teuthology.orchestra.run.smithi001.stdout: "ops": [], 2024-09-13T11:25:48.803 INFO:teuthology.orchestra.run.smithi001.stdout: "complaint_time": 30, 2024-09-13T11:25:48.803 INFO:teuthology.orchestra.run.smithi001.stdout: "num_blocked_ops": 0 2024-09-13T11:25:48.803 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-13T11:25:48.822 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-13T11:25:48.881 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-09-13T11:25:48.880+0000 7f97befc9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26315) UID: 0 2024-09-13T11:25:48.982 INFO:tasks.ceph.osd.14.smithi162.stderr:2024-09-13T11:25:48.980+0000 7f3df5903640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33154) UID: 0 2024-09-13T11:25:48.997 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-13T11:25:48.998 INFO:teuthology.orchestra.run.smithi001.stdout: "size": 20, 2024-09-13T11:25:48.998 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 600, 2024-09-13T11:25:48.998 INFO:teuthology.orchestra.run.smithi001.stdout: "ops": [ 2024-09-13T11:25:48.998 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.998 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDPGPush(3.1e 89/78 [PushOp(3:7a07f38c:::benchmark_data_smithi177_31116_object20966:head, version: 54'1402, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:7a07f38c:::benchmark_data_smithi177_31116_object20966:head@54'1402, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:25:48.998 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:46.645251+0000", 2024-09-13T11:25:48.998 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 122.350341882, 2024-09-13T11:25:48.998 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.17670883800000001, 2024-09-13T11:25:48.998 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:48.998 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:48.998 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:25:48.998 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:48.999 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.999 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:48.999 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.645251+0000", 2024-09-13T11:25:48.999 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:48.999 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.999 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.999 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:48.999 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.645251+0000", 2024-09-13T11:25:48.999 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:48.999 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:48.999 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:48.999 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:48.999 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.645253+0000", 2024-09-13T11:25:48.999 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.1399999999999998e-06 2024-09-13T11:25:49.000 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.000 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.000 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:49.000 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.645295+0000", 2024-09-13T11:25:49.000 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.2323000000000001e-05 2024-09-13T11:25:49.000 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.000 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.000 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:49.000 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.820731+0000", 2024-09-13T11:25:49.000 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.17543607999999999 2024-09-13T11:25:49.000 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.000 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.000 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:49.001 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.820750+0000", 2024-09-13T11:25:49.001 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.8793000000000001e-05 2024-09-13T11:25:49.001 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.001 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.001 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:49.001 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.820914+0000", 2024-09-13T11:25:49.001 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000164062 2024-09-13T11:25:49.001 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.001 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.001 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:25:49.001 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.820939+0000", 2024-09-13T11:25:49.001 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.4672e-05 2024-09-13T11:25:49.001 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.001 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.002 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:49.002 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.821960+0000", 2024-09-13T11:25:49.002 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.001020768 2024-09-13T11:25:49.002 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:49.002 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:49.002 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:49.002 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.002 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.002 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDPGPush(3.1a 102/78 [PushOp(3:5df50766:::benchmark_data_smithi177_31116_object50543:head, version: 72'3152, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:5df50766:::benchmark_data_smithi177_31116_object50543:head@72'3152, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:25:49.002 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:24:00.499311+0000", 2024-09-13T11:25:49.002 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 108.49628179699999, 2024-09-13T11:25:49.002 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.26146664800000002, 2024-09-13T11:25:49.002 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:49.002 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:49.003 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:25:49.003 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:49.003 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.003 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:49.003 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:00.499311+0000", 2024-09-13T11:25:49.003 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:49.003 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.003 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.003 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:49.003 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:00.499311+0000", 2024-09-13T11:25:49.003 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:49.003 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.003 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.004 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:49.004 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:00.499320+0000", 2024-09-13T11:25:49.004 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.6400000000000003e-06 2024-09-13T11:25:49.004 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.004 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.004 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:49.004 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:00.499384+0000", 2024-09-13T11:25:49.004 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.4554999999999994e-05 2024-09-13T11:25:49.004 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.004 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.004 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:49.004 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:00.759713+0000", 2024-09-13T11:25:49.004 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.260328742 2024-09-13T11:25:49.005 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.005 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.005 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:49.005 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:00.759742+0000", 2024-09-13T11:25:49.005 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.9298e-05 2024-09-13T11:25:49.005 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.005 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.005 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:49.005 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:00.759871+0000", 2024-09-13T11:25:49.005 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000128992 2024-09-13T11:25:49.005 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.005 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.005 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:25:49.005 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:00.759884+0000", 2024-09-13T11:25:49.005 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2687999999999999e-05 2024-09-13T11:25:49.006 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.006 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.006 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:49.006 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:00.760778+0000", 2024-09-13T11:25:49.006 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00089373299999999996 2024-09-13T11:25:49.006 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:49.006 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:49.006 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:49.006 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.006 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.006 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4665.0:82769 3.1e e105/102 3:7a4e88b5:::benchmark_data_smithi177_31116_object82768:head v 105'4381, mlcod=105'4379)", 2024-09-13T11:25:49.006 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:24:03.745047+0000", 2024-09-13T11:25:49.006 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 105.25054557199999, 2024-09-13T11:25:49.007 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.213806145, 2024-09-13T11:25:49.007 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:49.007 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:49.007 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:49.007 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:49.007 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.007 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:49.007 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:03.745047+0000", 2024-09-13T11:25:49.007 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:49.007 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.007 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.007 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:49.007 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:03.745047+0000", 2024-09-13T11:25:49.007 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:49.008 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.008 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.008 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:49.008 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:03.745049+0000", 2024-09-13T11:25:49.008 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.5710000000000001e-06 2024-09-13T11:25:49.008 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.008 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.008 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:49.008 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:03.745107+0000", 2024-09-13T11:25:49.008 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.8799999999999999e-05 2024-09-13T11:25:49.009 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.009 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.009 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:49.009 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:03.955145+0000", 2024-09-13T11:25:49.009 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.210037735 2024-09-13T11:25:49.009 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.009 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.009 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:49.009 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:03.955163+0000", 2024-09-13T11:25:49.009 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.7520000000000002e-05 2024-09-13T11:25:49.009 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.009 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.009 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:49.010 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:03.955473+0000", 2024-09-13T11:25:49.010 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00031014399999999999 2024-09-13T11:25:49.010 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.010 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.010 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:25:49.010 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:03.955538+0000", 2024-09-13T11:25:49.010 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.5059000000000002e-05 2024-09-13T11:25:49.010 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.010 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.010 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:25:49.010 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:03.958753+0000", 2024-09-13T11:25:49.010 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0032152769999999999 2024-09-13T11:25:49.011 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.011 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.011 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:49.011 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:03.958853+0000", 2024-09-13T11:25:49.011 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000100039 2024-09-13T11:25:49.011 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:49.011 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:49.011 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:49.011 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.011 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.011 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:98582 3.1a 3:59f9cd75:::benchmark_data_smithi177_31116_object98581:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113)", 2024-09-13T11:25:49.011 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:24:15.431226+0000", 2024-09-13T11:25:49.011 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 93.564367102999995, 2024-09-13T11:25:49.012 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.19820755800000001, 2024-09-13T11:25:49.012 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:49.012 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:49.012 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:49.012 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:25:49.012 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:25:49.012 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:49.012 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 98582 2024-09-13T11:25:49.012 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.012 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:49.012 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.012 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:49.012 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.431226+0000", 2024-09-13T11:25:49.013 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:49.013 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.013 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.013 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:49.013 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.431226+0000", 2024-09-13T11:25:49.013 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:49.013 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.013 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.013 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:49.013 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.431229+0000", 2024-09-13T11:25:49.013 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.7100000000000001e-06 2024-09-13T11:25:49.013 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.013 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.014 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:49.014 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.431334+0000", 2024-09-13T11:25:49.014 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00010506900000000001 2024-09-13T11:25:49.014 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.014 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.014 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:49.014 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.431364+0000", 2024-09-13T11:25:49.014 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.9607e-05 2024-09-13T11:25:49.014 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.014 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.014 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:49.014 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.431388+0000", 2024-09-13T11:25:49.014 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.3808999999999999e-05 2024-09-13T11:25:49.014 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.015 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.015 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:49.015 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.431483+0000", 2024-09-13T11:25:49.015 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.5402999999999994e-05 2024-09-13T11:25:49.015 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.015 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.015 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:25:49.015 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.431642+0000", 2024-09-13T11:25:49.015 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000158412 2024-09-13T11:25:49.015 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.015 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.015 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 9", 2024-09-13T11:25:49.015 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.431796+0000", 2024-09-13T11:25:49.015 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00015467899999999999 2024-09-13T11:25:49.016 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.016 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.016 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:25:49.016 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.433731+0000", 2024-09-13T11:25:49.016 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.001934945 2024-09-13T11:25:49.016 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.016 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.016 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:25:49.016 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.629245+0000", 2024-09-13T11:25:49.016 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.19551390199999999 2024-09-13T11:25:49.016 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.016 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.016 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:25:49.016 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.629346+0000", 2024-09-13T11:25:49.017 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000100869 2024-09-13T11:25:49.017 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.017 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.017 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:49.017 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.629433+0000", 2024-09-13T11:25:49.017 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.7152999999999997e-05 2024-09-13T11:25:49.017 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:49.017 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:49.017 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:49.017 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.017 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.017 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:99105 3.1a 3:5ac5090c:::benchmark_data_smithi177_31116_object99104:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113)", 2024-09-13T11:25:49.017 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:24:15.962466+0000", 2024-09-13T11:25:49.017 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 93.033126813999999, 2024-09-13T11:25:49.018 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.23832849, 2024-09-13T11:25:49.018 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:49.018 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:49.018 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:49.018 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:25:49.018 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:25:49.018 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:49.018 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 99105 2024-09-13T11:25:49.018 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.018 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:49.018 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.018 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:49.018 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.962466+0000", 2024-09-13T11:25:49.018 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:49.019 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.019 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.019 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:49.019 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.962466+0000", 2024-09-13T11:25:49.019 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:49.019 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.019 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.019 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:49.019 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.962471+0000", 2024-09-13T11:25:49.019 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.9570000000000001e-06 2024-09-13T11:25:49.019 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.019 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.019 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:49.019 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.962547+0000", 2024-09-13T11:25:49.020 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.5947999999999999e-05 2024-09-13T11:25:49.020 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.020 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.020 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:49.020 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.962576+0000", 2024-09-13T11:25:49.020 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.9547999999999999e-05 2024-09-13T11:25:49.020 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.020 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.020 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:49.020 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.962594+0000", 2024-09-13T11:25:49.020 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.8023e-05 2024-09-13T11:25:49.020 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.020 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.020 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:49.020 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.962770+0000", 2024-09-13T11:25:49.021 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00017551099999999999 2024-09-13T11:25:49.021 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.021 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.021 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:25:49.021 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.963114+0000", 2024-09-13T11:25:49.021 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00034440399999999998 2024-09-13T11:25:49.021 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.021 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.021 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 9", 2024-09-13T11:25:49.021 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.963351+0000", 2024-09-13T11:25:49.021 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000237209 2024-09-13T11:25:49.021 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.021 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.021 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:25:49.022 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.966345+0000", 2024-09-13T11:25:49.022 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0029933519999999999 2024-09-13T11:25:49.022 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.022 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.022 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:25:49.022 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:16.200724+0000", 2024-09-13T11:25:49.022 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.23437934599999999 2024-09-13T11:25:49.022 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.022 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.022 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:25:49.022 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:16.200761+0000", 2024-09-13T11:25:49.022 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.6406999999999999e-05 2024-09-13T11:25:49.022 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.022 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.023 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:49.023 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:16.200794+0000", 2024-09-13T11:25:49.023 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.3785000000000003e-05 2024-09-13T11:25:49.023 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:49.023 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:49.023 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:49.023 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.023 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.023 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4665.0:104331 3.1e e115/102 3:79890035:::benchmark_data_smithi177_31116_object104330:head v 115'5083, mlcod=115'5080)", 2024-09-13T11:25:49.023 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:24:20.242656+0000", 2024-09-13T11:25:49.024 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 88.752936722000001, 2024-09-13T11:25:49.024 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.16813846700000001, 2024-09-13T11:25:49.024 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:49.024 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:49.024 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:49.024 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:49.024 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.024 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:49.024 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:20.242656+0000", 2024-09-13T11:25:49.024 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:49.024 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.024 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.024 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:49.024 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:20.242656+0000", 2024-09-13T11:25:49.025 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:49.025 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.025 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.025 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:49.025 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:20.242657+0000", 2024-09-13T11:25:49.025 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.4750000000000001e-06 2024-09-13T11:25:49.025 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.025 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.025 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:49.025 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:20.242699+0000", 2024-09-13T11:25:49.025 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.193e-05 2024-09-13T11:25:49.025 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.025 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.025 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:49.026 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:20.408772+0000", 2024-09-13T11:25:49.026 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.16607291199999999 2024-09-13T11:25:49.026 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.026 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.026 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:49.026 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:20.408779+0000", 2024-09-13T11:25:49.026 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.3600000000000001e-06 2024-09-13T11:25:49.026 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.026 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.026 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:49.026 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:20.408840+0000", 2024-09-13T11:25:49.026 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.1834999999999993e-05 2024-09-13T11:25:49.026 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.026 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.027 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:25:49.027 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:20.408865+0000", 2024-09-13T11:25:49.027 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.4294000000000002e-05 2024-09-13T11:25:49.027 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.027 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.027 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:25:49.027 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:20.410751+0000", 2024-09-13T11:25:49.027 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.001885811 2024-09-13T11:25:49.027 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.027 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.027 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:49.028 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:20.410794+0000", 2024-09-13T11:25:49.028 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.3850000000000002e-05 2024-09-13T11:25:49.028 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:49.028 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:49.028 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:49.028 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.028 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.028 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:129921 3.1a 3:5fb9088b:::benchmark_data_smithi177_31116_object129920:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e134)", 2024-09-13T11:25:49.028 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:24:47.177556+0000", 2024-09-13T11:25:49.028 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 61.818036534999997, 2024-09-13T11:25:49.028 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.33804469999999998, 2024-09-13T11:25:49.028 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:49.028 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:49.029 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:49.029 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:25:49.029 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:25:49.029 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:49.029 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 129921 2024-09-13T11:25:49.029 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.029 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:49.029 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.029 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:49.029 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:47.177556+0000", 2024-09-13T11:25:49.029 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:49.029 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.029 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.029 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:49.029 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:47.177556+0000", 2024-09-13T11:25:49.030 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:49.030 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.030 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.030 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:49.030 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:47.177559+0000", 2024-09-13T11:25:49.030 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.0790000000000001e-06 2024-09-13T11:25:49.030 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.030 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.030 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:49.030 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:47.177589+0000", 2024-09-13T11:25:49.030 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.0173999999999999e-05 2024-09-13T11:25:49.030 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.030 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.030 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:49.031 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:47.177606+0000", 2024-09-13T11:25:49.031 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.6475999999999999e-05 2024-09-13T11:25:49.031 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.031 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.031 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:49.031 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:47.177615+0000", 2024-09-13T11:25:49.031 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.5729999999999997e-06 2024-09-13T11:25:49.031 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.031 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.031 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:49.031 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:47.177680+0000", 2024-09-13T11:25:49.031 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.4349000000000003e-05 2024-09-13T11:25:49.031 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.031 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.032 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:25:49.032 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:47.177794+0000", 2024-09-13T11:25:49.032 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000114119 2024-09-13T11:25:49.032 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.032 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.032 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 9", 2024-09-13T11:25:49.032 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:47.177895+0000", 2024-09-13T11:25:49.032 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000101078 2024-09-13T11:25:49.032 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.032 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.032 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:25:49.032 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:47.179634+0000", 2024-09-13T11:25:49.032 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.001738606 2024-09-13T11:25:49.032 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.033 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.033 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:25:49.033 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:47.515495+0000", 2024-09-13T11:25:49.033 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.33586143000000002 2024-09-13T11:25:49.033 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.033 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.033 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:25:49.033 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:47.515551+0000", 2024-09-13T11:25:49.033 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.6048999999999998e-05 2024-09-13T11:25:49.033 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.033 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.034 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:49.034 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:47.515601+0000", 2024-09-13T11:25:49.034 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.9767e-05 2024-09-13T11:25:49.034 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:49.034 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:49.034 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:49.034 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.034 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.034 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4665.0:140742 3.1e e144/102 3:79c694a0:::benchmark_data_smithi177_31116_object439:head v 144'6204, mlcod=144'6200)", 2024-09-13T11:25:49.034 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:24:58.321534+0000", 2024-09-13T11:25:49.034 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 50.674059010999997, 2024-09-13T11:25:49.034 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.23629921500000001, 2024-09-13T11:25:49.034 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:49.035 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:49.035 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:49.035 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:49.035 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.035 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:49.035 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.321534+0000", 2024-09-13T11:25:49.035 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:49.035 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.035 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.035 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:49.035 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.321534+0000", 2024-09-13T11:25:49.035 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:49.035 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.035 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.036 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:49.036 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.321535+0000", 2024-09-13T11:25:49.036 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.159e-06 2024-09-13T11:25:49.036 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.036 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.036 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:49.036 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.321545+0000", 2024-09-13T11:25:49.036 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.9219999999999992e-06 2024-09-13T11:25:49.036 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.036 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.036 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:49.036 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.553719+0000", 2024-09-13T11:25:49.036 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.23217401700000001 2024-09-13T11:25:49.037 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.037 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.037 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:49.037 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.553734+0000", 2024-09-13T11:25:49.037 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.5344000000000001e-05 2024-09-13T11:25:49.037 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.037 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.037 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:49.037 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.553858+0000", 2024-09-13T11:25:49.037 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00012357500000000001 2024-09-13T11:25:49.037 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.037 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.037 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:25:49.037 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.553903+0000", 2024-09-13T11:25:49.037 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.5374999999999998e-05 2024-09-13T11:25:49.038 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.038 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.038 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:25:49.038 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.557719+0000", 2024-09-13T11:25:49.038 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0038158670000000001 2024-09-13T11:25:49.038 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.038 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.038 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:49.038 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.557833+0000", 2024-09-13T11:25:49.039 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000113956 2024-09-13T11:25:49.039 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:49.039 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:49.039 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:49.039 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.039 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.039 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4665.0:140796 3.1e e144/102 3:7987a652:::benchmark_data_smithi177_31116_object493:head v 144'6205, mlcod=144'6202)", 2024-09-13T11:25:49.039 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:24:58.553657+0000", 2024-09-13T11:25:49.039 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 50.441936034999998, 2024-09-13T11:25:49.039 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.25308323100000002, 2024-09-13T11:25:49.039 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:49.039 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:49.039 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:49.040 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:49.040 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.040 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:49.040 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.553657+0000", 2024-09-13T11:25:49.040 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:49.040 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.040 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.040 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:49.040 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.553657+0000", 2024-09-13T11:25:49.040 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:49.040 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.040 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.040 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:49.041 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.553660+0000", 2024-09-13T11:25:49.041 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.7519999999999998e-06 2024-09-13T11:25:49.041 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.041 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.041 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:49.041 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.553678+0000", 2024-09-13T11:25:49.041 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.7156e-05 2024-09-13T11:25:49.041 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.041 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.041 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:49.041 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.804342+0000", 2024-09-13T11:25:49.041 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.25066436199999997 2024-09-13T11:25:49.041 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.041 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.042 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:49.042 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.804353+0000", 2024-09-13T11:25:49.042 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.1147e-05 2024-09-13T11:25:49.042 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.042 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.042 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:49.042 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.804433+0000", 2024-09-13T11:25:49.042 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.9586999999999995e-05 2024-09-13T11:25:49.042 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.042 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.042 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:25:49.042 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.804457+0000", 2024-09-13T11:25:49.042 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.4108e-05 2024-09-13T11:25:49.043 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.043 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.043 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:25:49.043 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.806684+0000", 2024-09-13T11:25:49.043 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0022271309999999998 2024-09-13T11:25:49.043 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.043 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.043 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:49.043 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.806740+0000", 2024-09-13T11:25:49.043 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.5988e-05 2024-09-13T11:25:49.043 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:49.043 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:49.043 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:49.043 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.043 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.044 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:149955 3.11 3:8fa29838:::benchmark_data_smithi177_31116_object9652:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e149)", 2024-09-13T11:25:49.044 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:25:05.905492+0000", 2024-09-13T11:25:49.044 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 43.090100661000001, 2024-09-13T11:25:49.044 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.204181955, 2024-09-13T11:25:49.044 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:49.044 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:49.044 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:49.044 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:25:49.044 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:25:49.044 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:49.044 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 149955 2024-09-13T11:25:49.044 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.044 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:49.045 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.045 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:49.045 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:05.905492+0000", 2024-09-13T11:25:49.045 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:49.045 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.045 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.045 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:49.045 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:05.905492+0000", 2024-09-13T11:25:49.045 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:49.045 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.045 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.045 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:49.045 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:05.905494+0000", 2024-09-13T11:25:49.046 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.9369999999999998e-06 2024-09-13T11:25:49.046 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.046 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.046 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:49.046 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:05.905500+0000", 2024-09-13T11:25:49.046 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.7039999999999999e-06 2024-09-13T11:25:49.046 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.046 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.046 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:49.046 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:05.905512+0000", 2024-09-13T11:25:49.046 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.257e-05 2024-09-13T11:25:49.046 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.046 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.046 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:49.047 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:05.905519+0000", 2024-09-13T11:25:49.047 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.5420000000000002e-06 2024-09-13T11:25:49.047 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.047 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.047 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:49.047 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:05.905584+0000", 2024-09-13T11:25:49.047 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.4819000000000004e-05 2024-09-13T11:25:49.047 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.047 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.047 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:25:49.047 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:05.905696+0000", 2024-09-13T11:25:49.047 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00011248599999999999 2024-09-13T11:25:49.047 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.047 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.047 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 11", 2024-09-13T11:25:49.048 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:05.905759+0000", 2024-09-13T11:25:49.048 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.2451000000000002e-05 2024-09-13T11:25:49.048 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.048 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.048 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:25:49.048 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:05.907584+0000", 2024-09-13T11:25:49.048 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0018250110000000001 2024-09-13T11:25:49.048 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.048 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.048 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:25:49.048 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:06.109588+0000", 2024-09-13T11:25:49.048 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.20200470300000001 2024-09-13T11:25:49.048 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.048 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.049 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:25:49.049 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:06.109637+0000", 2024-09-13T11:25:49.049 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.8439000000000002e-05 2024-09-13T11:25:49.049 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.049 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.049 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:49.049 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:06.109674+0000", 2024-09-13T11:25:49.049 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.7292999999999997e-05 2024-09-13T11:25:49.049 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:49.049 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:49.049 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:49.049 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.049 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.050 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:150414 3.11 3:8f72eed4:::benchmark_data_smithi177_31116_object10111:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e149)", 2024-09-13T11:25:49.050 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:25:06.348925+0000", 2024-09-13T11:25:49.050 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 42.646667549, 2024-09-13T11:25:49.050 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.17250813300000001, 2024-09-13T11:25:49.050 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:49.050 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:49.050 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:49.050 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:25:49.050 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:25:49.050 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:49.050 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 150414 2024-09-13T11:25:49.050 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.050 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:49.050 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.051 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:49.051 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:06.348925+0000", 2024-09-13T11:25:49.051 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:49.051 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.051 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.051 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:49.051 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:06.348925+0000", 2024-09-13T11:25:49.051 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:49.051 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.051 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.051 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:49.051 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:06.348927+0000", 2024-09-13T11:25:49.051 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.7430000000000001e-06 2024-09-13T11:25:49.051 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.051 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.052 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:49.052 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:06.348931+0000", 2024-09-13T11:25:49.052 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.5210000000000004e-06 2024-09-13T11:25:49.052 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.052 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.052 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:49.052 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:06.348943+0000", 2024-09-13T11:25:49.052 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.1168e-05 2024-09-13T11:25:49.052 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.052 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.052 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:49.053 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:06.348950+0000", 2024-09-13T11:25:49.053 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.1430000000000001e-06 2024-09-13T11:25:49.053 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.053 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.053 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:49.053 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:06.349032+0000", 2024-09-13T11:25:49.053 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.2517999999999998e-05 2024-09-13T11:25:49.053 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.053 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.053 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:25:49.053 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:06.349179+0000", 2024-09-13T11:25:49.053 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000146935 2024-09-13T11:25:49.053 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.053 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.054 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 11", 2024-09-13T11:25:49.054 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:06.349290+0000", 2024-09-13T11:25:49.054 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000110927 2024-09-13T11:25:49.054 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.054 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.054 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:25:49.054 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:06.351686+0000", 2024-09-13T11:25:49.054 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0023963560000000001 2024-09-13T11:25:49.054 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.054 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.054 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:25:49.054 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:06.521374+0000", 2024-09-13T11:25:49.054 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.169687216 2024-09-13T11:25:49.054 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.055 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.055 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:25:49.055 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:06.521406+0000", 2024-09-13T11:25:49.055 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.1841000000000002e-05 2024-09-13T11:25:49.055 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.055 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.055 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:49.055 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:06.521433+0000", 2024-09-13T11:25:49.055 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.7764999999999999e-05 2024-09-13T11:25:49.055 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:49.055 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:49.055 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:49.055 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.055 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.056 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:168772 3.1a 3:5be647e9:::benchmark_data_smithi177_31116_object28469:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e161)", 2024-09-13T11:25:49.056 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:25:20.636095+0000", 2024-09-13T11:25:49.056 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 28.359498139999999, 2024-09-13T11:25:49.056 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.17968553000000001, 2024-09-13T11:25:49.056 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:49.056 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:49.056 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:49.056 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:25:49.056 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:25:49.056 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:49.056 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 168772 2024-09-13T11:25:49.056 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.056 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:49.056 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.057 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:49.057 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:20.636095+0000", 2024-09-13T11:25:49.057 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:49.057 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.057 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.057 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:49.057 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:20.636095+0000", 2024-09-13T11:25:49.057 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:49.057 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.057 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.057 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:49.057 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:20.636096+0000", 2024-09-13T11:25:49.057 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.852e-06 2024-09-13T11:25:49.058 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.058 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.058 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:49.058 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:20.636101+0000", 2024-09-13T11:25:49.058 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.0329999999999999e-06 2024-09-13T11:25:49.058 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.058 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.058 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:49.058 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:20.636114+0000", 2024-09-13T11:25:49.058 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2816999999999999e-05 2024-09-13T11:25:49.058 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.058 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.058 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:49.058 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:20.636120+0000", 2024-09-13T11:25:49.059 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.7429999999999999e-06 2024-09-13T11:25:49.059 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.059 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.059 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:49.059 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:20.636194+0000", 2024-09-13T11:25:49.059 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.3520000000000001e-05 2024-09-13T11:25:49.059 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.059 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.059 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:25:49.059 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:20.636369+0000", 2024-09-13T11:25:49.059 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00017552 2024-09-13T11:25:49.059 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.059 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.059 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 9", 2024-09-13T11:25:49.060 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:20.636519+0000", 2024-09-13T11:25:49.060 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000149949 2024-09-13T11:25:49.060 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.060 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.060 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:25:49.060 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:20.638735+0000", 2024-09-13T11:25:49.060 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0022156509999999999 2024-09-13T11:25:49.060 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.060 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.060 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:25:49.060 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:20.815699+0000", 2024-09-13T11:25:49.060 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.176964343 2024-09-13T11:25:49.060 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.060 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.061 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:25:49.061 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:20.815748+0000", 2024-09-13T11:25:49.061 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.9518000000000003e-05 2024-09-13T11:25:49.061 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.061 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.061 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:49.061 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:20.815780+0000", 2024-09-13T11:25:49.061 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.1584000000000001e-05 2024-09-13T11:25:49.061 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:49.061 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:49.061 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:49.061 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.061 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.061 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:168782 3.1a 3:590b82e1:::benchmark_data_smithi177_31116_object28479:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e161)", 2024-09-13T11:25:49.062 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:25:20.641567+0000", 2024-09-13T11:25:49.062 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 28.354025308000001, 2024-09-13T11:25:49.062 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.19100501, 2024-09-13T11:25:49.062 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:49.062 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:49.062 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:49.062 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:25:49.062 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:25:49.062 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:49.062 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 168782 2024-09-13T11:25:49.062 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.062 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:49.062 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.062 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:49.063 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:20.641567+0000", 2024-09-13T11:25:49.063 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:49.063 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.063 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.063 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:49.063 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:20.641567+0000", 2024-09-13T11:25:49.063 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:49.063 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.063 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.063 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:49.063 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:20.641569+0000", 2024-09-13T11:25:49.063 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.869e-06 2024-09-13T11:25:49.063 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.063 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.064 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:49.064 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:20.641574+0000", 2024-09-13T11:25:49.064 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.1680000000000003e-06 2024-09-13T11:25:49.064 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.064 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.064 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:49.064 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:20.641587+0000", 2024-09-13T11:25:49.064 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2858e-05 2024-09-13T11:25:49.064 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.064 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.064 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:49.064 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:20.641595+0000", 2024-09-13T11:25:49.064 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.6149999999999999e-06 2024-09-13T11:25:49.064 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.065 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.065 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:49.065 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:20.641655+0000", 2024-09-13T11:25:49.065 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.9673999999999998e-05 2024-09-13T11:25:49.065 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.065 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.065 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:25:49.065 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:20.641748+0000", 2024-09-13T11:25:49.065 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.3919999999999995e-05 2024-09-13T11:25:49.065 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.065 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.065 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 9", 2024-09-13T11:25:49.065 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:20.642990+0000", 2024-09-13T11:25:49.065 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0012415709999999999 2024-09-13T11:25:49.065 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.066 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.066 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:25:49.066 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:20.645016+0000", 2024-09-13T11:25:49.066 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.002025448 2024-09-13T11:25:49.066 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.066 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.066 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:25:49.066 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:20.832409+0000", 2024-09-13T11:25:49.066 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.18739319600000001 2024-09-13T11:25:49.066 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.066 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.066 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:25:49.066 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:20.832525+0000", 2024-09-13T11:25:49.066 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000116273 2024-09-13T11:25:49.067 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.067 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.067 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:49.067 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:20.832572+0000", 2024-09-13T11:25:49.067 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.7417999999999999e-05 2024-09-13T11:25:49.067 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:49.067 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:49.067 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:49.067 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.067 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.067 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop_reply(client.4665.0:168772 3.1a e161/105 ondisk, result = 0)", 2024-09-13T11:25:49.067 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:25:20.645700+0000", 2024-09-13T11:25:49.067 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 28.349893127000001, 2024-09-13T11:25:49.067 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.17008606200000001, 2024-09-13T11:25:49.068 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:49.068 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:49.068 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:25:49.068 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:49.068 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.068 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:49.068 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:20.645700+0000", 2024-09-13T11:25:49.068 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:49.068 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.068 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.068 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:49.068 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:20.645700+0000", 2024-09-13T11:25:49.068 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:49.068 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.069 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.069 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:49.069 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:20.645701+0000", 2024-09-13T11:25:49.069 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.474e-06 2024-09-13T11:25:49.069 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.069 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.069 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:49.069 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:20.645714+0000", 2024-09-13T11:25:49.069 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2931999999999999e-05 2024-09-13T11:25:49.069 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.069 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.069 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:49.069 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:20.815603+0000", 2024-09-13T11:25:49.069 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.16988872199999999 2024-09-13T11:25:49.070 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.070 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.070 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:49.070 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:20.815610+0000", 2024-09-13T11:25:49.070 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.2139999999999999e-06 2024-09-13T11:25:49.070 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.070 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.070 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:49.070 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:20.815689+0000", 2024-09-13T11:25:49.070 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.9302999999999996e-05 2024-09-13T11:25:49.070 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.070 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.070 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:25:49.070 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:20.815695+0000", 2024-09-13T11:25:49.071 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.818e-06 2024-09-13T11:25:49.071 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.071 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.071 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:49.071 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:20.815786+0000", 2024-09-13T11:25:49.071 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.0599000000000005e-05 2024-09-13T11:25:49.071 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:49.071 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:49.071 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:49.071 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.071 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.071 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:174971 3.1a 3:5a8f64c5:::benchmark_data_smithi177_31116_object34668:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e166)", 2024-09-13T11:25:49.071 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:25:25.422480+0000", 2024-09-13T11:25:49.071 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 23.573112696999999, 2024-09-13T11:25:49.072 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.18399536399999999, 2024-09-13T11:25:49.072 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:49.072 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:49.072 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:49.072 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:25:49.072 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:25:49.072 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:49.072 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 174971 2024-09-13T11:25:49.072 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.072 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:49.072 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.072 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:49.072 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:25.422480+0000", 2024-09-13T11:25:49.072 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:49.073 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.073 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.073 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:49.073 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:25.422480+0000", 2024-09-13T11:25:49.073 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:49.073 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.073 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.073 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:49.073 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:25.422483+0000", 2024-09-13T11:25:49.073 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.4529999999999998e-06 2024-09-13T11:25:49.073 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.073 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.073 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:49.073 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:25.422492+0000", 2024-09-13T11:25:49.074 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.8820000000000007e-06 2024-09-13T11:25:49.074 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.074 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.074 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:49.074 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:25.422515+0000", 2024-09-13T11:25:49.074 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.2756e-05 2024-09-13T11:25:49.074 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.074 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.074 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:49.074 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:25.422526+0000", 2024-09-13T11:25:49.074 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0684e-05 2024-09-13T11:25:49.074 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.074 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.074 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:49.075 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:25.422590+0000", 2024-09-13T11:25:49.075 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.4163000000000002e-05 2024-09-13T11:25:49.075 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.075 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.075 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:25:49.075 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:25.422675+0000", 2024-09-13T11:25:49.075 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.5311999999999999e-05 2024-09-13T11:25:49.075 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.075 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.075 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 9", 2024-09-13T11:25:49.075 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:25.422745+0000", 2024-09-13T11:25:49.075 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.0217999999999998e-05 2024-09-13T11:25:49.075 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.075 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.076 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:25:49.076 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:25.424910+0000", 2024-09-13T11:25:49.076 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0021641500000000001 2024-09-13T11:25:49.076 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.076 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.076 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:25:49.076 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:25.606366+0000", 2024-09-13T11:25:49.076 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.181456267 2024-09-13T11:25:49.076 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.076 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.076 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:25:49.076 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:25.606428+0000", 2024-09-13T11:25:49.076 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.2312999999999998e-05 2024-09-13T11:25:49.076 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.077 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.077 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:49.077 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:25.606475+0000", 2024-09-13T11:25:49.077 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.7166000000000002e-05 2024-09-13T11:25:49.077 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:49.077 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:49.077 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:49.077 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.077 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.077 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4665.0:191081 3.1e e175/102 3:7e784823:::benchmark_data_smithi177_31116_object50778:head v 175'7812, mlcod=175'7809)", 2024-09-13T11:25:49.077 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:25:38.547539+0000", 2024-09-13T11:25:49.077 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 10.448053411, 2024-09-13T11:25:49.077 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.213922061, 2024-09-13T11:25:49.077 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:49.078 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:49.078 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:49.078 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:49.078 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.078 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:49.078 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:38.547539+0000", 2024-09-13T11:25:49.078 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:49.078 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.078 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.078 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:49.078 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:38.547539+0000", 2024-09-13T11:25:49.078 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:49.078 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.078 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.078 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:49.079 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:38.547543+0000", 2024-09-13T11:25:49.079 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.3629999999999998e-06 2024-09-13T11:25:49.079 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.079 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.079 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:49.079 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:38.547558+0000", 2024-09-13T11:25:49.079 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.5043e-05 2024-09-13T11:25:49.079 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.079 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.079 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:49.079 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:38.756778+0000", 2024-09-13T11:25:49.079 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.209219936 2024-09-13T11:25:49.079 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.079 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.080 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:49.080 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:38.756790+0000", 2024-09-13T11:25:49.080 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2775999999999999e-05 2024-09-13T11:25:49.080 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.080 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.080 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:49.080 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:38.756934+0000", 2024-09-13T11:25:49.080 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00014401600000000001 2024-09-13T11:25:49.080 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.080 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.080 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:25:49.080 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:38.757001+0000", 2024-09-13T11:25:49.080 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.6716000000000003e-05 2024-09-13T11:25:49.080 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.081 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.081 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:25:49.081 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:38.761376+0000", 2024-09-13T11:25:49.081 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0043747320000000001 2024-09-13T11:25:49.081 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.081 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.081 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:49.081 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:38.761461+0000", 2024-09-13T11:25:49.081 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.5478999999999999e-05 2024-09-13T11:25:49.081 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:49.082 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:49.082 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:49.082 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.082 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.082 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4665.0:191087 3.1e e175/102 3:7a697202:::benchmark_data_smithi177_31116_object50784:head v 175'7813, mlcod=175'7809)", 2024-09-13T11:25:49.082 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:25:38.561827+0000", 2024-09-13T11:25:49.082 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 10.433765201, 2024-09-13T11:25:49.082 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.209395781, 2024-09-13T11:25:49.082 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:49.082 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:49.082 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:49.082 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:49.082 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.082 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:49.082 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:38.561827+0000", 2024-09-13T11:25:49.083 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:49.083 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.083 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.083 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:49.083 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:38.561827+0000", 2024-09-13T11:25:49.083 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:49.083 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.083 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.083 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:49.083 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:38.561837+0000", 2024-09-13T11:25:49.083 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.2329999999999995e-06 2024-09-13T11:25:49.083 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.083 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.083 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:49.084 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:38.561849+0000", 2024-09-13T11:25:49.084 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2593e-05 2024-09-13T11:25:49.084 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.084 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.084 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:49.084 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:38.766804+0000", 2024-09-13T11:25:49.084 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.20495488000000001 2024-09-13T11:25:49.084 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.084 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.084 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:49.084 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:38.766823+0000", 2024-09-13T11:25:49.084 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.9143e-05 2024-09-13T11:25:49.084 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.084 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.085 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:49.085 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:38.766973+0000", 2024-09-13T11:25:49.085 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000149226 2024-09-13T11:25:49.085 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.085 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.085 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:25:49.085 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:38.767014+0000", 2024-09-13T11:25:49.085 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.1183000000000002e-05 2024-09-13T11:25:49.085 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.085 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.085 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:25:49.085 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:38.771145+0000", 2024-09-13T11:25:49.085 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0041315270000000003 2024-09-13T11:25:49.085 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.086 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.086 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:49.086 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:38.771223+0000", 2024-09-13T11:25:49.086 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.7996000000000003e-05 2024-09-13T11:25:49.086 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:49.086 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:49.086 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:49.086 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.086 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.086 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:191672 3.11 3:8b81ea05:::benchmark_data_smithi177_31116_object51369:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e176)", 2024-09-13T11:25:49.087 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:25:39.675914+0000", 2024-09-13T11:25:49.087 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 9.3196791559999994, 2024-09-13T11:25:49.087 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.165205352, 2024-09-13T11:25:49.087 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:49.087 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:49.087 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:49.087 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:25:49.087 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:25:49.087 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:49.087 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 191672 2024-09-13T11:25:49.087 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.087 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:49.087 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.087 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:49.088 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:39.675914+0000", 2024-09-13T11:25:49.088 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:49.088 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.088 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.088 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:49.088 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:39.675914+0000", 2024-09-13T11:25:49.088 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:49.088 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.088 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.088 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:49.088 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:39.675916+0000", 2024-09-13T11:25:49.088 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.8059999999999999e-06 2024-09-13T11:25:49.088 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.089 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.089 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:49.089 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:39.675923+0000", 2024-09-13T11:25:49.089 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.3489999999999997e-06 2024-09-13T11:25:49.089 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.089 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.089 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:49.089 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:39.675940+0000", 2024-09-13T11:25:49.089 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.7489000000000001e-05 2024-09-13T11:25:49.089 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.089 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.089 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:49.089 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:39.675948+0000", 2024-09-13T11:25:49.089 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.0690000000000002e-06 2024-09-13T11:25:49.090 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.090 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.090 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:49.090 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:39.676017+0000", 2024-09-13T11:25:49.090 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.8714000000000007e-05 2024-09-13T11:25:49.090 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.090 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.090 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:25:49.090 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:39.676142+0000", 2024-09-13T11:25:49.090 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000125535 2024-09-13T11:25:49.090 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.090 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.090 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 11", 2024-09-13T11:25:49.090 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:39.676236+0000", 2024-09-13T11:25:49.091 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.3249000000000001e-05 2024-09-13T11:25:49.091 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.091 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.091 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:25:49.091 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:39.679269+0000", 2024-09-13T11:25:49.091 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0030331780000000001 2024-09-13T11:25:49.091 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.092 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.092 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:25:49.092 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:39.841008+0000", 2024-09-13T11:25:49.092 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.16173915799999999 2024-09-13T11:25:49.092 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.092 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.092 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:25:49.092 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:39.841068+0000", 2024-09-13T11:25:49.092 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.9579e-05 2024-09-13T11:25:49.092 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.092 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.092 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:49.092 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:39.841119+0000", 2024-09-13T11:25:49.092 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.1226e-05 2024-09-13T11:25:49.093 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:49.093 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:49.093 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:49.093 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.093 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.093 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:202145 3.1a 3:5fe5b9e3:::benchmark_data_smithi177_31116_object61842:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e182)", 2024-09-13T11:25:49.093 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:25:48.598830+0000", 2024-09-13T11:25:49.093 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 0.39676255300000002, 2024-09-13T11:25:49.093 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.16940577600000001, 2024-09-13T11:25:49.093 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:49.093 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:49.093 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:49.093 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:25:49.093 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:25:49.094 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:49.094 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 202145 2024-09-13T11:25:49.094 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.094 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:49.094 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.094 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:49.094 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:48.598830+0000", 2024-09-13T11:25:49.094 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:49.094 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.094 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.094 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:49.094 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:48.598830+0000", 2024-09-13T11:25:49.094 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:49.094 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.095 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.095 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:49.095 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:48.598833+0000", 2024-09-13T11:25:49.095 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.79e-06 2024-09-13T11:25:49.095 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.095 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.095 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:49.095 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:48.598840+0000", 2024-09-13T11:25:49.095 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.5100000000000001e-06 2024-09-13T11:25:49.095 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.095 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.095 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:49.095 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:48.598858+0000", 2024-09-13T11:25:49.095 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.7294000000000001e-05 2024-09-13T11:25:49.096 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.096 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.096 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:49.096 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:48.598865+0000", 2024-09-13T11:25:49.096 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.7219999999999999e-06 2024-09-13T11:25:49.096 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.096 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.096 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:49.096 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:48.598931+0000", 2024-09-13T11:25:49.096 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.5734000000000005e-05 2024-09-13T11:25:49.096 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.096 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.096 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:25:49.096 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:48.599037+0000", 2024-09-13T11:25:49.097 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000105806 2024-09-13T11:25:49.097 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.097 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.097 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 9", 2024-09-13T11:25:49.097 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:48.599110+0000", 2024-09-13T11:25:49.097 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.3464e-05 2024-09-13T11:25:49.097 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.097 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.097 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:25:49.097 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:48.601703+0000", 2024-09-13T11:25:49.097 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0025923729999999998 2024-09-13T11:25:49.097 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.097 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.097 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:25:49.098 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:48.768165+0000", 2024-09-13T11:25:49.098 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.16646239400000001 2024-09-13T11:25:49.098 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.098 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.098 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:25:49.098 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:48.768198+0000", 2024-09-13T11:25:49.098 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.2568999999999999e-05 2024-09-13T11:25:49.098 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.098 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.098 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:49.098 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:48.768236+0000", 2024-09-13T11:25:49.098 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.8120000000000001e-05 2024-09-13T11:25:49.098 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:49.098 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:49.099 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:49.099 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.099 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.099 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop_reply(client.4665.0:202145 3.1a e182/105 ondisk, result = 0)", 2024-09-13T11:25:49.099 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:25:48.602776+0000", 2024-09-13T11:25:49.099 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 0.39281619499999998, 2024-09-13T11:25:49.099 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.16546329100000001, 2024-09-13T11:25:49.099 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:49.099 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:49.099 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:25:49.099 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:49.099 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.099 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:49.099 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:48.602776+0000", 2024-09-13T11:25:49.099 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:49.100 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.100 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.100 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:49.100 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:48.602776+0000", 2024-09-13T11:25:49.100 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:49.100 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.100 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.100 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:49.100 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:48.602778+0000", 2024-09-13T11:25:49.100 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.195e-06 2024-09-13T11:25:49.100 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.100 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.100 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:49.100 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:48.602785+0000", 2024-09-13T11:25:49.101 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.4869999999999996e-06 2024-09-13T11:25:49.101 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.101 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.101 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:49.101 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:48.768090+0000", 2024-09-13T11:25:49.101 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.165304597 2024-09-13T11:25:49.101 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.101 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.101 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:49.101 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:48.768099+0000", 2024-09-13T11:25:49.101 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.3449999999999995e-06 2024-09-13T11:25:49.101 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.101 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.101 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:49.102 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:48.768157+0000", 2024-09-13T11:25:49.102 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.7479999999999999e-05 2024-09-13T11:25:49.102 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.102 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.102 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:25:49.102 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:48.768162+0000", 2024-09-13T11:25:49.102 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.3890000000000004e-06 2024-09-13T11:25:49.102 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.102 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.102 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:49.102 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:48.768240+0000", 2024-09-13T11:25:49.102 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.7798000000000003e-05 2024-09-13T11:25:49.102 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:49.102 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:49.102 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:49.103 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:49.103 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:49.103 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-13T11:25:49.104 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-09-13T11:25:49.105 INFO:tasks.ceph.osd.5.smithi092.stderr:2024-09-13T11:25:49.101+0000 7f6f0b705640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26346) UID: 0 2024-09-13T11:25:49.205 INFO:tasks.ceph.osd.9.smithi092.stderr:2024-09-13T11:25:49.205+0000 7ff5ab3d0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26348) UID: 0 2024-09-13T11:25:49.243 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-13T11:25:49.244 INFO:teuthology.orchestra.run.smithi001.stdout: "ops": [], 2024-09-13T11:25:49.244 INFO:teuthology.orchestra.run.smithi001.stdout: "num_ops": 0 2024-09-13T11:25:49.244 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-13T11:25:49.260 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-09-13T11:25:49.306 INFO:tasks.ceph.osd.10.smithi162.stderr:2024-09-13T11:25:49.304+0000 7fb59c9f0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26317) UID: 0 2024-09-13T11:25:49.399 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-13T11:25:49.399 INFO:teuthology.orchestra.run.smithi001.stdout: "ops": [], 2024-09-13T11:25:49.399 INFO:teuthology.orchestra.run.smithi001.stdout: "complaint_time": 30, 2024-09-13T11:25:49.399 INFO:teuthology.orchestra.run.smithi001.stdout: "num_blocked_ops": 0 2024-09-13T11:25:49.400 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-13T11:25:49.400 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-13T11:25:49.407 INFO:tasks.ceph.osd.7.smithi177.stderr:2024-09-13T11:25:49.406+0000 7f897b51d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26575) UID: 0 2024-09-13T11:25:49.418 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-09-13T11:25:49.507 INFO:tasks.ceph.osd.5.smithi092.stderr:2024-09-13T11:25:49.505+0000 7f6f0b705640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26346) UID: 0 2024-09-13T11:25:49.608 INFO:tasks.ceph.osd.10.smithi162.stderr:2024-09-13T11:25:49.604+0000 7fb59c9f0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26317) UID: 0 2024-09-13T11:25:49.610 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-13T11:25:49.610 INFO:teuthology.orchestra.run.smithi001.stdout: "size": 20, 2024-09-13T11:25:49.610 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 600, 2024-09-13T11:25:49.610 INFO:teuthology.orchestra.run.smithi001.stdout: "ops": [ 2024-09-13T11:25:49.610 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.611 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:54656 3.1e 3.6a49163e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e75)", 2024-09-13T11:25:49.611 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:30.979993+0000", 2024-09-13T11:25:49.611 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 138.62905619899999, 2024-09-13T11:25:49.611 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.9269557150000001, 2024-09-13T11:25:49.611 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:49.611 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:49.611 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-13T11:25:49.611 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:25:49.611 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:25:49.611 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:49.611 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 54656 2024-09-13T11:25:49.611 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.611 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:49.611 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.612 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:49.612 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:30.979993+0000", 2024-09-13T11:25:49.612 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:49.612 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.612 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.612 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:49.612 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:30.979993+0000", 2024-09-13T11:25:49.612 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:49.612 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.612 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.612 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:49.612 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:30.979997+0000", 2024-09-13T11:25:49.612 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.0319999999999997e-06 2024-09-13T11:25:49.613 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.613 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.613 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:49.613 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:30.980057+0000", 2024-09-13T11:25:49.613 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.0341999999999999e-05 2024-09-13T11:25:49.613 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.613 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.613 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:49.613 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:30.980094+0000", 2024-09-13T11:25:49.613 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.6770000000000002e-05 2024-09-13T11:25:49.613 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.613 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.613 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:49.614 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:30.980113+0000", 2024-09-13T11:25:49.614 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.8791999999999999e-05 2024-09-13T11:25:49.614 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.614 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.614 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:49.614 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.097773+0000", 2024-09-13T11:25:49.614 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.117660379 2024-09-13T11:25:49.614 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.614 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.614 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for peered", 2024-09-13T11:25:49.614 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.097784+0000", 2024-09-13T11:25:49.614 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.1025e-05 2024-09-13T11:25:49.614 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.614 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.615 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:49.615 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.906937+0000", 2024-09-13T11:25:49.615 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.8091523789999999 2024-09-13T11:25:49.615 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.615 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.615 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:49.615 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.906949+0000", 2024-09-13T11:25:49.615 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.1996e-05 2024-09-13T11:25:49.615 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:49.615 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:49.615 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:49.615 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.615 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.615 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:54720 3.1e 3.a944303e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-13T11:25:49.616 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:31.973315+0000", 2024-09-13T11:25:49.616 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 137.635733964, 2024-09-13T11:25:49.616 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.93366527200000005, 2024-09-13T11:25:49.616 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:49.616 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:49.616 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-13T11:25:49.616 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:25:49.616 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:25:49.616 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:49.616 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 54720 2024-09-13T11:25:49.616 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.616 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:49.616 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.617 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:49.617 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.973315+0000", 2024-09-13T11:25:49.617 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:49.617 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.617 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.617 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:49.617 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.973315+0000", 2024-09-13T11:25:49.617 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:49.617 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.617 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.617 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:49.617 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.973318+0000", 2024-09-13T11:25:49.617 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.5830000000000001e-06 2024-09-13T11:25:49.617 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.618 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.618 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:49.618 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.973465+0000", 2024-09-13T11:25:49.618 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000146816 2024-09-13T11:25:49.618 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.618 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.618 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:49.618 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.973478+0000", 2024-09-13T11:25:49.618 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2931e-05 2024-09-13T11:25:49.618 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.618 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.618 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:49.618 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.973483+0000", 2024-09-13T11:25:49.618 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.8969999999999996e-06 2024-09-13T11:25:49.619 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.619 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.619 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:49.619 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.973548+0000", 2024-09-13T11:25:49.619 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.4771999999999996e-05 2024-09-13T11:25:49.619 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.619 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.619 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for peered", 2024-09-13T11:25:49.619 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.973557+0000", 2024-09-13T11:25:49.619 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.0359999999999995e-06 2024-09-13T11:25:49.619 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.619 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.619 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:49.619 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.906974+0000", 2024-09-13T11:25:49.619 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.93341706000000002 2024-09-13T11:25:49.620 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.620 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.620 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:49.620 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.906980+0000", 2024-09-13T11:25:49.620 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.1770000000000003e-06 2024-09-13T11:25:49.620 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:49.620 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:49.620 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:49.620 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.620 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.620 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:54734 3.1e 3.d7f6f7e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-13T11:25:49.620 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:31.983342+0000", 2024-09-13T11:25:49.620 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 137.62570716100001, 2024-09-13T11:25:49.620 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.92367058599999996, 2024-09-13T11:25:49.621 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:49.621 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:49.621 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-13T11:25:49.621 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:25:49.621 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:25:49.621 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:49.621 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 54734 2024-09-13T11:25:49.621 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.621 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:49.621 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.621 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:49.621 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.983342+0000", 2024-09-13T11:25:49.621 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:49.621 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.622 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.622 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:49.622 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.983342+0000", 2024-09-13T11:25:49.622 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:49.622 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.622 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.622 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:49.622 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.983345+0000", 2024-09-13T11:25:49.622 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.6220000000000001e-06 2024-09-13T11:25:49.622 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.622 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.622 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:49.622 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.983438+0000", 2024-09-13T11:25:49.622 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.3517000000000006e-05 2024-09-13T11:25:49.623 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.623 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.623 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:49.623 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.983451+0000", 2024-09-13T11:25:49.623 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2527999999999999e-05 2024-09-13T11:25:49.623 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.623 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.623 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:49.623 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.983458+0000", 2024-09-13T11:25:49.623 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.6380000000000004e-06 2024-09-13T11:25:49.623 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.623 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.623 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:49.623 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.983521+0000", 2024-09-13T11:25:49.623 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.2729000000000002e-05 2024-09-13T11:25:49.624 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.624 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.624 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for peered", 2024-09-13T11:25:49.624 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.983548+0000", 2024-09-13T11:25:49.624 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.6537999999999999e-05 2024-09-13T11:25:49.624 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.624 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.624 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:49.624 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.907005+0000", 2024-09-13T11:25:49.624 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.923456991 2024-09-13T11:25:49.624 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.624 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.624 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:49.624 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.907013+0000", 2024-09-13T11:25:49.625 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.0229999999999992e-06 2024-09-13T11:25:49.625 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:49.625 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:49.625 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:49.625 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.625 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.625 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:54760 3.1e 3.8c32c13e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-13T11:25:49.625 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:31.999685+0000", 2024-09-13T11:25:49.625 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 137.60936392900001, 2024-09-13T11:25:49.625 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.90735639599999995, 2024-09-13T11:25:49.625 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:49.625 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:49.625 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-13T11:25:49.626 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:25:49.626 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:25:49.626 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:49.626 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 54760 2024-09-13T11:25:49.626 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.626 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:49.626 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.626 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:49.626 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.999685+0000", 2024-09-13T11:25:49.626 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:49.626 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.626 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.626 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:49.626 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.999685+0000", 2024-09-13T11:25:49.627 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:49.627 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.627 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.627 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:49.627 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.999690+0000", 2024-09-13T11:25:49.627 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.7879999999999997e-06 2024-09-13T11:25:49.627 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.627 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.627 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:49.627 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.999729+0000", 2024-09-13T11:25:49.627 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.9032000000000002e-05 2024-09-13T11:25:49.627 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.627 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.627 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:49.628 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.999741+0000", 2024-09-13T11:25:49.628 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2136999999999999e-05 2024-09-13T11:25:49.628 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.628 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.628 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:49.628 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.999748+0000", 2024-09-13T11:25:49.628 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.9120000000000001e-06 2024-09-13T11:25:49.628 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.628 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.628 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:49.628 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.999803+0000", 2024-09-13T11:25:49.628 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.5064e-05 2024-09-13T11:25:49.628 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.628 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.629 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for peered", 2024-09-13T11:25:49.629 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.999815+0000", 2024-09-13T11:25:49.629 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2041e-05 2024-09-13T11:25:49.629 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.629 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.629 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:49.629 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.907034+0000", 2024-09-13T11:25:49.629 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.90721858200000005 2024-09-13T11:25:49.629 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.629 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.629 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:49.629 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.907042+0000", 2024-09-13T11:25:49.629 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.8399999999999995e-06 2024-09-13T11:25:49.629 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:49.630 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:49.630 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:49.630 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.630 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.630 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:54768 3.1e 3.6dad499e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-13T11:25:49.630 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:32.003482+0000", 2024-09-13T11:25:49.630 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 137.60556727900001, 2024-09-13T11:25:49.630 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.90358804699999995, 2024-09-13T11:25:49.630 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:49.630 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:49.630 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-13T11:25:49.630 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:25:49.630 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:25:49.630 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:49.631 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 54768 2024-09-13T11:25:49.631 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.631 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:49.631 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.631 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:49.631 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.003482+0000", 2024-09-13T11:25:49.631 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:49.631 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.631 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.631 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:49.631 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.003482+0000", 2024-09-13T11:25:49.631 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:49.631 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.631 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.632 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:49.632 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.003486+0000", 2024-09-13T11:25:49.632 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.1030000000000004e-06 2024-09-13T11:25:49.632 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.632 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.632 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:49.632 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.003523+0000", 2024-09-13T11:25:49.632 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.6732e-05 2024-09-13T11:25:49.632 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.632 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.632 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:49.632 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.003535+0000", 2024-09-13T11:25:49.632 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2520999999999999e-05 2024-09-13T11:25:49.632 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.633 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.633 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:49.633 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.003541+0000", 2024-09-13T11:25:49.633 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.8690000000000001e-06 2024-09-13T11:25:49.633 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.633 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.633 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:49.633 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.003593+0000", 2024-09-13T11:25:49.633 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.1786000000000002e-05 2024-09-13T11:25:49.633 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.633 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.633 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for peered", 2024-09-13T11:25:49.633 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.003602+0000", 2024-09-13T11:25:49.633 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.9679999999999995e-06 2024-09-13T11:25:49.633 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.634 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.634 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:49.634 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.907063+0000", 2024-09-13T11:25:49.634 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.90346116600000004 2024-09-13T11:25:49.634 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.634 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.634 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:49.634 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.907070+0000", 2024-09-13T11:25:49.634 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.9020000000000002e-06 2024-09-13T11:25:49.634 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:49.634 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:49.634 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:49.634 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.634 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.635 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:54785 3.1e 3.d32f1ede (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-13T11:25:49.635 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:32.016238+0000", 2024-09-13T11:25:49.635 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 137.59281106899999, 2024-09-13T11:25:49.635 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.89086243399999998, 2024-09-13T11:25:49.635 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:49.635 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:49.635 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-13T11:25:49.635 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:25:49.635 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:25:49.635 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:49.635 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 54785 2024-09-13T11:25:49.635 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.635 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:49.635 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.636 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:49.636 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.016238+0000", 2024-09-13T11:25:49.636 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:49.636 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.636 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.636 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:49.636 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.016238+0000", 2024-09-13T11:25:49.636 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:49.636 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.636 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.637 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:49.637 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.016241+0000", 2024-09-13T11:25:49.637 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.6469999999999998e-06 2024-09-13T11:25:49.637 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.637 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.637 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:49.637 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.016271+0000", 2024-09-13T11:25:49.637 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.0505999999999999e-05 2024-09-13T11:25:49.637 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.637 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.637 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:49.637 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.016287+0000", 2024-09-13T11:25:49.637 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.6073999999999999e-05 2024-09-13T11:25:49.637 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.638 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.638 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:49.638 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.016295+0000", 2024-09-13T11:25:49.638 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.0590000000000003e-06 2024-09-13T11:25:49.638 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.638 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.638 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:49.638 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.016373+0000", 2024-09-13T11:25:49.638 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.7634999999999998e-05 2024-09-13T11:25:49.638 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.638 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.638 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for peered", 2024-09-13T11:25:49.638 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.016383+0000", 2024-09-13T11:25:49.638 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0246e-05 2024-09-13T11:25:49.639 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.639 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.639 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:49.639 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.907092+0000", 2024-09-13T11:25:49.639 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.89070849200000002 2024-09-13T11:25:49.639 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.639 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.639 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:49.639 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.907101+0000", 2024-09-13T11:25:49.639 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.7749999999999998e-06 2024-09-13T11:25:49.639 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:49.639 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:49.639 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:49.639 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.640 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.640 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:54811 3.1e 3.3754367e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-13T11:25:49.640 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:32.045424+0000", 2024-09-13T11:25:49.640 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 137.56362541799999, 2024-09-13T11:25:49.640 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.86170719399999995, 2024-09-13T11:25:49.640 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:49.640 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:49.640 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-13T11:25:49.640 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:25:49.640 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:25:49.640 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:49.640 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 54811 2024-09-13T11:25:49.640 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.641 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:49.641 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.641 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:49.641 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.045424+0000", 2024-09-13T11:25:49.641 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:49.641 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.641 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.641 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:49.641 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.045424+0000", 2024-09-13T11:25:49.641 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:49.641 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.641 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.641 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:49.641 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.045426+0000", 2024-09-13T11:25:49.642 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.0260000000000002e-06 2024-09-13T11:25:49.642 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.642 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.642 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:49.642 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.045556+0000", 2024-09-13T11:25:49.642 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00013003099999999999 2024-09-13T11:25:49.642 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.642 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.642 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:49.642 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.045568+0000", 2024-09-13T11:25:49.642 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.1727e-05 2024-09-13T11:25:49.642 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.642 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.642 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:49.642 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.045576+0000", 2024-09-13T11:25:49.643 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.6459999999999999e-06 2024-09-13T11:25:49.643 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.643 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.643 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:49.643 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.045650+0000", 2024-09-13T11:25:49.643 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.4000999999999998e-05 2024-09-13T11:25:49.643 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.643 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.643 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for peered", 2024-09-13T11:25:49.643 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.045658+0000", 2024-09-13T11:25:49.643 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.3869999999999997e-06 2024-09-13T11:25:49.643 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.643 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.643 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:49.644 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.907123+0000", 2024-09-13T11:25:49.644 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.86146508399999999 2024-09-13T11:25:49.644 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.644 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.644 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:49.644 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.907131+0000", 2024-09-13T11:25:49.644 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.2919999999999996e-06 2024-09-13T11:25:49.644 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:49.644 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:49.644 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:49.644 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.644 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.644 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:54817 3.1e 3.36dd2fde (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-13T11:25:49.644 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:32.050108+0000", 2024-09-13T11:25:49.645 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 137.55894152600001, 2024-09-13T11:25:49.645 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.85705549199999997, 2024-09-13T11:25:49.645 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:49.645 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:49.645 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-13T11:25:49.645 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:25:49.645 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:25:49.645 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:49.645 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 54817 2024-09-13T11:25:49.645 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.645 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:49.645 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.645 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:49.645 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.050108+0000", 2024-09-13T11:25:49.646 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:49.646 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.646 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.646 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:49.646 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.050108+0000", 2024-09-13T11:25:49.646 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:49.646 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.646 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.646 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:49.646 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.050110+0000", 2024-09-13T11:25:49.646 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.0209999999999998e-06 2024-09-13T11:25:49.646 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.646 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.647 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:49.647 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.050166+0000", 2024-09-13T11:25:49.647 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.6026000000000002e-05 2024-09-13T11:25:49.647 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.647 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.647 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:49.647 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.050180+0000", 2024-09-13T11:25:49.647 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.4307e-05 2024-09-13T11:25:49.647 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.647 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.647 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:49.647 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.050188+0000", 2024-09-13T11:25:49.647 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.5410000000000003e-06 2024-09-13T11:25:49.647 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.648 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.648 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:49.648 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.050260+0000", 2024-09-13T11:25:49.648 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.2157999999999996e-05 2024-09-13T11:25:49.648 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.648 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.648 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for peered", 2024-09-13T11:25:49.648 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.050267+0000", 2024-09-13T11:25:49.648 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.0190000000000001e-06 2024-09-13T11:25:49.648 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.648 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.648 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:49.648 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.907155+0000", 2024-09-13T11:25:49.648 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.85688824399999997 2024-09-13T11:25:49.649 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.649 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.649 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:49.649 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.907163+0000", 2024-09-13T11:25:49.649 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.1759999999999993e-06 2024-09-13T11:25:49.649 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:49.649 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:49.649 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:49.649 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.649 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.649 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:54818 3.1e 3.5cfcc69e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-13T11:25:49.649 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:32.051374+0000", 2024-09-13T11:25:49.649 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 137.55767510999999, 2024-09-13T11:25:49.649 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.85581781000000001, 2024-09-13T11:25:49.650 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:49.650 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:49.650 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-13T11:25:49.650 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:25:49.650 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:25:49.650 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:49.650 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 54818 2024-09-13T11:25:49.650 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.650 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:49.650 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.650 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:49.650 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.051374+0000", 2024-09-13T11:25:49.650 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:49.650 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.650 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.651 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:49.651 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.051374+0000", 2024-09-13T11:25:49.651 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:49.651 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.651 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.651 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:49.651 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.051375+0000", 2024-09-13T11:25:49.651 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.4440000000000001e-06 2024-09-13T11:25:49.651 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.651 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.651 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:49.651 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.051426+0000", 2024-09-13T11:25:49.651 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.0809000000000001e-05 2024-09-13T11:25:49.652 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.652 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.652 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:49.652 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.051441+0000", 2024-09-13T11:25:49.652 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.4460999999999999e-05 2024-09-13T11:25:49.652 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.652 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.652 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:49.652 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.051449+0000", 2024-09-13T11:25:49.652 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.4880000000000007e-06 2024-09-13T11:25:49.652 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.652 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.652 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:49.652 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.051499+0000", 2024-09-13T11:25:49.653 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.9691999999999998e-05 2024-09-13T11:25:49.653 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.653 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.653 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for peered", 2024-09-13T11:25:49.653 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.051505+0000", 2024-09-13T11:25:49.653 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.2029999999999997e-06 2024-09-13T11:25:49.653 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.653 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.653 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:49.653 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.907184+0000", 2024-09-13T11:25:49.653 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.85567906599999999 2024-09-13T11:25:49.653 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.653 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.653 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:49.654 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.907192+0000", 2024-09-13T11:25:49.654 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.6469999999999998e-06 2024-09-13T11:25:49.654 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:49.654 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:49.654 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:49.654 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.654 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.654 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:54828 3.1e 3.e920c11e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-13T11:25:49.654 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:32.063344+0000", 2024-09-13T11:25:49.654 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 137.545704683, 2024-09-13T11:25:49.654 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.84387899099999997, 2024-09-13T11:25:49.654 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:49.654 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:49.654 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-13T11:25:49.655 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:25:49.655 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:25:49.655 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:49.655 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 54828 2024-09-13T11:25:49.655 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.655 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:49.655 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.655 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:49.655 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.063344+0000", 2024-09-13T11:25:49.655 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:49.655 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.655 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.655 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:49.655 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.063344+0000", 2024-09-13T11:25:49.656 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:49.656 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.656 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.656 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:49.656 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.063347+0000", 2024-09-13T11:25:49.656 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.2299999999999998e-06 2024-09-13T11:25:49.656 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.656 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.656 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:49.656 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.063384+0000", 2024-09-13T11:25:49.656 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.7290999999999999e-05 2024-09-13T11:25:49.656 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.656 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.656 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:49.657 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.063397+0000", 2024-09-13T11:25:49.657 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2955999999999999e-05 2024-09-13T11:25:49.657 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.657 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.657 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:49.657 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.063404+0000", 2024-09-13T11:25:49.657 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.4889999999999998e-06 2024-09-13T11:25:49.657 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.657 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.657 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:49.657 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.063470+0000", 2024-09-13T11:25:49.657 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.5516000000000001e-05 2024-09-13T11:25:49.657 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.657 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.658 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for peered", 2024-09-13T11:25:49.658 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.063482+0000", 2024-09-13T11:25:49.658 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2065000000000001e-05 2024-09-13T11:25:49.658 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.658 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.658 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:49.658 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.907216+0000", 2024-09-13T11:25:49.658 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.84373388900000001 2024-09-13T11:25:49.658 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.658 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.658 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:49.658 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.907223+0000", 2024-09-13T11:25:49.658 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.5549999999999997e-06 2024-09-13T11:25:49.658 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:49.659 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:49.659 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:49.659 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.659 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.659 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:54829 3.1e 3.192f62be (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-13T11:25:49.659 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:32.065642+0000", 2024-09-13T11:25:49.659 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 137.54340761500001, 2024-09-13T11:25:49.659 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.84161483100000001, 2024-09-13T11:25:49.659 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:49.659 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:49.659 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-13T11:25:49.659 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:25:49.659 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:25:49.659 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:49.660 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 54829 2024-09-13T11:25:49.660 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.660 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:49.660 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.660 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:49.660 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.065642+0000", 2024-09-13T11:25:49.660 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:49.660 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.660 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.660 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:49.660 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.065642+0000", 2024-09-13T11:25:49.660 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:49.660 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.660 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.661 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:49.661 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.065644+0000", 2024-09-13T11:25:49.661 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.1780000000000002e-06 2024-09-13T11:25:49.661 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.661 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.661 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:49.661 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.065695+0000", 2024-09-13T11:25:49.661 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.1632000000000003e-05 2024-09-13T11:25:49.661 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.661 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.661 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:49.661 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.065711+0000", 2024-09-13T11:25:49.661 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.5522000000000001e-05 2024-09-13T11:25:49.661 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.662 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.662 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:49.662 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.065719+0000", 2024-09-13T11:25:49.662 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.3480000000000004e-06 2024-09-13T11:25:49.662 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.662 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.662 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:49.662 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.065785+0000", 2024-09-13T11:25:49.662 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.5627e-05 2024-09-13T11:25:49.662 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.662 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.662 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for peered", 2024-09-13T11:25:49.662 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.065797+0000", 2024-09-13T11:25:49.662 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.1766000000000001e-05 2024-09-13T11:25:49.662 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.663 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.663 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:49.663 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.907248+0000", 2024-09-13T11:25:49.663 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.84145150300000005 2024-09-13T11:25:49.663 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.663 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.663 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:49.663 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.907256+0000", 2024-09-13T11:25:49.663 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.2549999999999997e-06 2024-09-13T11:25:49.663 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:49.663 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:49.663 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:49.663 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.663 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.664 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:54830 3.1e 3.b31554de (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-13T11:25:49.664 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:32.068259+0000", 2024-09-13T11:25:49.664 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 137.54078974699999, 2024-09-13T11:25:49.664 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.83902489599999996, 2024-09-13T11:25:49.664 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:49.664 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:49.664 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-13T11:25:49.664 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:25:49.664 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:25:49.664 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:49.664 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 54830 2024-09-13T11:25:49.664 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.664 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:49.664 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.665 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:49.665 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.068259+0000", 2024-09-13T11:25:49.665 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:49.665 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.665 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.665 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:49.665 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.068259+0000", 2024-09-13T11:25:49.665 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:49.665 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.665 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.665 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:49.665 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.068262+0000", 2024-09-13T11:25:49.665 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.1799999999999999e-06 2024-09-13T11:25:49.665 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.666 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.666 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:49.666 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.068301+0000", 2024-09-13T11:25:49.666 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.9289999999999998e-05 2024-09-13T11:25:49.666 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.666 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.666 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:49.666 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.068314+0000", 2024-09-13T11:25:49.666 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.3502999999999999e-05 2024-09-13T11:25:49.666 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.666 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.666 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:49.666 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.068324+0000", 2024-09-13T11:25:49.666 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.1630000000000002e-06 2024-09-13T11:25:49.666 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.667 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.667 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:49.667 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.068375+0000", 2024-09-13T11:25:49.667 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.1805000000000003e-05 2024-09-13T11:25:49.667 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.667 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.667 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for peered", 2024-09-13T11:25:49.667 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.068382+0000", 2024-09-13T11:25:49.667 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.224e-06 2024-09-13T11:25:49.667 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.667 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.667 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:49.667 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.907277+0000", 2024-09-13T11:25:49.668 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.83889509500000004 2024-09-13T11:25:49.668 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.668 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.668 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:49.668 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.907284+0000", 2024-09-13T11:25:49.668 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.6359999999999994e-06 2024-09-13T11:25:49.668 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:49.668 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:49.668 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:49.668 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.668 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.668 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:54846 3.1e 3.742e049e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-13T11:25:49.668 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:32.153567+0000", 2024-09-13T11:25:49.668 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 137.455482523, 2024-09-13T11:25:49.669 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.75378642399999995, 2024-09-13T11:25:49.669 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:49.669 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:49.669 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-13T11:25:49.669 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:25:49.669 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:25:49.669 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:49.669 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 54846 2024-09-13T11:25:49.669 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.669 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:49.669 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.669 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:49.669 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.153567+0000", 2024-09-13T11:25:49.669 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:49.670 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.670 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.670 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:49.670 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.153567+0000", 2024-09-13T11:25:49.670 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:49.670 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.670 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.670 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:49.670 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.153569+0000", 2024-09-13T11:25:49.670 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.4660000000000002e-06 2024-09-13T11:25:49.671 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.671 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.671 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:49.671 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.153619+0000", 2024-09-13T11:25:49.671 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.9617000000000003e-05 2024-09-13T11:25:49.671 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.671 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.671 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:49.671 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.153634+0000", 2024-09-13T11:25:49.671 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.4904e-05 2024-09-13T11:25:49.671 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.671 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.671 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:49.671 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.153642+0000", 2024-09-13T11:25:49.672 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.5599999999999994e-06 2024-09-13T11:25:49.672 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.672 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.672 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:49.672 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.153721+0000", 2024-09-13T11:25:49.672 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.8603999999999995e-05 2024-09-13T11:25:49.672 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.672 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.672 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for peered", 2024-09-13T11:25:49.672 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.153730+0000", 2024-09-13T11:25:49.672 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.3840000000000003e-06 2024-09-13T11:25:49.672 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.672 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.672 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:49.673 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.907346+0000", 2024-09-13T11:25:49.673 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.75361563099999995 2024-09-13T11:25:49.673 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.673 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.673 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:49.673 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.907353+0000", 2024-09-13T11:25:49.673 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.2579999999999998e-06 2024-09-13T11:25:49.673 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:49.673 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:49.673 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:49.673 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.673 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.673 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:54888 3.1e 3.43e14ede (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-13T11:25:49.673 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:32.423064+0000", 2024-09-13T11:25:49.674 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 137.18598485199999, 2024-09-13T11:25:49.674 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.48433585299999998, 2024-09-13T11:25:49.674 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:49.674 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:49.674 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-13T11:25:49.674 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:25:49.674 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:25:49.674 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:49.674 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 54888 2024-09-13T11:25:49.674 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.674 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:49.674 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.674 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:49.674 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.423064+0000", 2024-09-13T11:25:49.675 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:49.675 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.675 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.675 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:49.675 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.423064+0000", 2024-09-13T11:25:49.675 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:49.675 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.675 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.675 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:49.675 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.423069+0000", 2024-09-13T11:25:49.675 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.7389999999999999e-06 2024-09-13T11:25:49.675 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.675 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.675 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:49.676 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.423115+0000", 2024-09-13T11:25:49.676 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.5847000000000003e-05 2024-09-13T11:25:49.676 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.676 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.676 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:49.676 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.423126+0000", 2024-09-13T11:25:49.676 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.1418e-05 2024-09-13T11:25:49.676 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.676 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.676 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:49.676 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.423132+0000", 2024-09-13T11:25:49.676 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.57e-06 2024-09-13T11:25:49.676 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.676 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.677 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:49.677 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.423191+0000", 2024-09-13T11:25:49.677 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.9250999999999999e-05 2024-09-13T11:25:49.677 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.677 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.677 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for peered", 2024-09-13T11:25:49.677 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.423198+0000", 2024-09-13T11:25:49.677 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.9179999999999997e-06 2024-09-13T11:25:49.677 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.677 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.677 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:49.677 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.907392+0000", 2024-09-13T11:25:49.677 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.48419437500000001 2024-09-13T11:25:49.677 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.677 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.678 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:49.678 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.907400+0000", 2024-09-13T11:25:49.678 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.7349999999999996e-06 2024-09-13T11:25:49.678 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:49.678 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:49.678 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:49.678 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.678 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.678 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4665.0:59977 3.21 e83/78 3:86b4d8f6:::benchmark_data_smithi177_31116_object59976:head v 83'3136, mlcod=0'0)", 2024-09-13T11:25:49.678 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:40.571147+0000", 2024-09-13T11:25:49.678 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 129.037902403, 2024-09-13T11:25:49.678 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.51907542799999995, 2024-09-13T11:25:49.678 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:49.678 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:49.679 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:49.679 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:49.679 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.679 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:49.679 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:40.571147+0000", 2024-09-13T11:25:49.679 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:49.679 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.679 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.679 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:49.679 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:40.571147+0000", 2024-09-13T11:25:49.679 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:49.679 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.679 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.679 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:49.680 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:40.571151+0000", 2024-09-13T11:25:49.680 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.5650000000000003e-06 2024-09-13T11:25:49.680 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.680 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.680 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:49.680 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:40.571204+0000", 2024-09-13T11:25:49.680 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.2720999999999999e-05 2024-09-13T11:25:49.680 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.680 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.680 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:49.680 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:41.088332+0000", 2024-09-13T11:25:49.680 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.51712822000000003 2024-09-13T11:25:49.680 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.680 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.681 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:49.681 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:41.088341+0000", 2024-09-13T11:25:49.681 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.3100000000000001e-06 2024-09-13T11:25:49.681 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.681 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.681 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:49.681 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:41.088444+0000", 2024-09-13T11:25:49.681 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000103708 2024-09-13T11:25:49.681 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.681 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.681 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:25:49.681 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:41.088475+0000", 2024-09-13T11:25:49.681 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.0497e-05 2024-09-13T11:25:49.681 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.682 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.682 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:25:49.682 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:41.090192+0000", 2024-09-13T11:25:49.682 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.001717288 2024-09-13T11:25:49.682 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.682 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.682 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:49.682 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:41.090222+0000", 2024-09-13T11:25:49.682 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.0119000000000001e-05 2024-09-13T11:25:49.682 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:49.682 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:49.682 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:49.682 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.682 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.683 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:110286 3.26 3.c41c0fa6 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e119)", 2024-09-13T11:25:49.683 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:24:25.654272+0000", 2024-09-13T11:25:49.683 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 83.954776710999994, 2024-09-13T11:25:49.683 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.43337235400000002, 2024-09-13T11:25:49.683 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:49.683 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:49.683 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-13T11:25:49.683 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:25:49.683 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:25:49.683 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:49.684 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 110286 2024-09-13T11:25:49.684 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.684 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:49.684 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.684 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:49.684 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:25.654272+0000", 2024-09-13T11:25:49.684 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:49.684 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.684 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.684 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:49.684 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:25.654272+0000", 2024-09-13T11:25:49.684 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:49.684 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.685 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.685 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:49.685 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:25.654275+0000", 2024-09-13T11:25:49.685 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.3199999999999998e-06 2024-09-13T11:25:49.685 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.685 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.685 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:49.685 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:25.654517+0000", 2024-09-13T11:25:49.685 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00024270199999999999 2024-09-13T11:25:49.685 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.685 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.685 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:49.685 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:25.654550+0000", 2024-09-13T11:25:49.685 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.2549000000000003e-05 2024-09-13T11:25:49.685 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.686 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.686 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:49.686 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:25.654566+0000", 2024-09-13T11:25:49.686 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.5517000000000001e-05 2024-09-13T11:25:49.686 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.686 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.686 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:49.686 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:25.654655+0000", 2024-09-13T11:25:49.686 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.9961000000000001e-05 2024-09-13T11:25:49.686 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.686 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.686 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for peered", 2024-09-13T11:25:49.686 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:25.654665+0000", 2024-09-13T11:25:49.686 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.0520000000000007e-06 2024-09-13T11:25:49.687 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.687 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.687 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:49.687 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:26.087629+0000", 2024-09-13T11:25:49.687 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.43296476900000003 2024-09-13T11:25:49.687 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.687 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.687 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:49.687 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:26.087645+0000", 2024-09-13T11:25:49.687 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.5483999999999999e-05 2024-09-13T11:25:49.687 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:49.687 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:49.687 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:49.687 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.688 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.688 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:177913 3.27 3:e5268e7a:::benchmark_data_smithi177_31116_object37610:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e168)", 2024-09-13T11:25:49.688 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:25:27.607015+0000", 2024-09-13T11:25:49.688 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 22.002034361, 2024-09-13T11:25:49.688 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.301660555, 2024-09-13T11:25:49.688 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:49.688 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:49.688 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:49.688 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:25:49.688 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:25:49.688 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:49.688 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 177913 2024-09-13T11:25:49.688 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.688 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:49.689 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.689 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:49.689 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:27.607015+0000", 2024-09-13T11:25:49.689 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:49.689 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.689 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.689 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:49.689 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:27.607015+0000", 2024-09-13T11:25:49.689 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:49.689 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.689 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.689 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:49.689 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:27.607018+0000", 2024-09-13T11:25:49.689 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.7000000000000002e-06 2024-09-13T11:25:49.690 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.690 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.690 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:49.690 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:27.607029+0000", 2024-09-13T11:25:49.690 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0933e-05 2024-09-13T11:25:49.690 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.690 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.690 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:49.690 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:27.607062+0000", 2024-09-13T11:25:49.690 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.2786000000000001e-05 2024-09-13T11:25:49.690 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.691 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.691 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:49.691 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:27.607078+0000", 2024-09-13T11:25:49.691 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.6232999999999999e-05 2024-09-13T11:25:49.691 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.691 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.691 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:49.691 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:27.607204+0000", 2024-09-13T11:25:49.691 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000125458 2024-09-13T11:25:49.691 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.691 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.691 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:25:49.691 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:27.607544+0000", 2024-09-13T11:25:49.691 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00033990600000000001 2024-09-13T11:25:49.692 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.692 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.692 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 5,11", 2024-09-13T11:25:49.692 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:27.607882+0000", 2024-09-13T11:25:49.692 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00033788399999999999 2024-09-13T11:25:49.692 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.692 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.692 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:25:49.692 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:27.610120+0000", 2024-09-13T11:25:49.692 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0022379930000000002 2024-09-13T11:25:49.692 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.692 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.692 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:25:49.692 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:27.613170+0000", 2024-09-13T11:25:49.693 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.003050545 2024-09-13T11:25:49.693 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.693 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.693 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:25:49.693 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:27.908398+0000", 2024-09-13T11:25:49.693 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.29522760599999998 2024-09-13T11:25:49.693 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.693 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.693 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:25:49.693 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:27.908458+0000", 2024-09-13T11:25:49.693 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.9718e-05 2024-09-13T11:25:49.693 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.693 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.693 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:49.694 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:27.908675+0000", 2024-09-13T11:25:49.694 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00021779300000000001 2024-09-13T11:25:49.694 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:49.694 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:49.694 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:49.694 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.694 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.694 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop_reply(client.4665.0:177913 3.27 e168/120 ondisk, result = 0)", 2024-09-13T11:25:49.694 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:25:27.610053+0000", 2024-09-13T11:25:49.694 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 21.998996202000001, 2024-09-13T11:25:49.694 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.29862817200000003, 2024-09-13T11:25:49.694 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:49.694 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:49.694 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:25:49.694 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:49.695 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.695 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:49.695 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:27.610053+0000", 2024-09-13T11:25:49.695 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:49.695 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.695 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.695 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:49.695 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:27.610053+0000", 2024-09-13T11:25:49.695 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:49.695 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.695 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.695 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:49.695 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:27.610056+0000", 2024-09-13T11:25:49.696 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.49e-06 2024-09-13T11:25:49.696 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.696 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.696 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:49.696 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:27.610067+0000", 2024-09-13T11:25:49.696 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0599e-05 2024-09-13T11:25:49.696 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.696 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.696 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:49.696 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:27.908314+0000", 2024-09-13T11:25:49.696 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.29824728700000003 2024-09-13T11:25:49.696 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.696 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.696 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:49.697 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:27.908320+0000", 2024-09-13T11:25:49.697 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.9100000000000002e-06 2024-09-13T11:25:49.697 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.697 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.697 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:49.697 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:27.908378+0000", 2024-09-13T11:25:49.697 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.7575999999999999e-05 2024-09-13T11:25:49.697 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.697 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.697 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:25:49.697 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:27.908390+0000", 2024-09-13T11:25:49.697 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.187e-05 2024-09-13T11:25:49.697 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.697 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.698 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:49.698 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:27.908681+0000", 2024-09-13T11:25:49.698 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00029144000000000002 2024-09-13T11:25:49.698 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:49.698 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:49.698 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:49.698 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.698 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.698 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDPGPushReply(3.27 168/120 [PushReplyOp(3:e535cc18:::benchmark_data_smithi177_31116_object103295:head)])", 2024-09-13T11:25:49.698 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:25:27.615014+0000", 2024-09-13T11:25:49.698 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 21.994034868, 2024-09-13T11:25:49.698 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.31810488799999997, 2024-09-13T11:25:49.698 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:49.698 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:49.698 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-13T11:25:49.699 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:49.699 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.699 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:49.699 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:27.615014+0000", 2024-09-13T11:25:49.699 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:49.699 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.699 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.699 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:49.699 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:27.615014+0000", 2024-09-13T11:25:49.699 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:49.699 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.699 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.699 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:49.699 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:27.615017+0000", 2024-09-13T11:25:49.700 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.6199999999999999e-06 2024-09-13T11:25:49.700 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.700 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.700 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:49.700 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:27.615030+0000", 2024-09-13T11:25:49.700 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2833000000000001e-05 2024-09-13T11:25:49.700 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.700 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.700 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:49.700 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:27.932588+0000", 2024-09-13T11:25:49.700 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.31755857700000001 2024-09-13T11:25:49.700 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.700 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.701 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:49.701 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:27.932595+0000", 2024-09-13T11:25:49.701 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.2160000000000003e-06 2024-09-13T11:25:49.701 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.701 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.704 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:49.704 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:27.932658+0000", 2024-09-13T11:25:49.704 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.3618e-05 2024-09-13T11:25:49.704 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.704 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.705 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:49.705 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:27.933119+0000", 2024-09-13T11:25:49.705 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00046102399999999998 2024-09-13T11:25:49.705 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:49.705 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:49.705 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:49.705 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.705 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.705 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:191082 3.26 3:67bd986b:::benchmark_data_smithi177_31116_object50779:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e175)", 2024-09-13T11:25:49.705 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:25:38.550314+0000", 2024-09-13T11:25:49.705 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 11.058735241000001, 2024-09-13T11:25:49.705 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.29180858100000001, 2024-09-13T11:25:49.705 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:49.706 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:49.706 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:49.706 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:25:49.706 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:25:49.706 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:49.706 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 191082 2024-09-13T11:25:49.706 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.706 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:49.706 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.706 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:49.706 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:38.550314+0000", 2024-09-13T11:25:49.706 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:49.706 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.706 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.707 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:49.707 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:38.550314+0000", 2024-09-13T11:25:49.707 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:49.707 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.707 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.707 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:49.707 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:38.550317+0000", 2024-09-13T11:25:49.707 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.14e-06 2024-09-13T11:25:49.707 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.707 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.707 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:49.707 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:38.550327+0000", 2024-09-13T11:25:49.707 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0386e-05 2024-09-13T11:25:49.707 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.707 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.708 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:49.708 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:38.550353+0000", 2024-09-13T11:25:49.708 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.5431000000000001e-05 2024-09-13T11:25:49.708 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.708 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.708 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:49.708 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:38.550367+0000", 2024-09-13T11:25:49.708 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.3859999999999999e-05 2024-09-13T11:25:49.708 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.708 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.708 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:49.708 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:38.550476+0000", 2024-09-13T11:25:49.708 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000108782 2024-09-13T11:25:49.708 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.709 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.709 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:25:49.709 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:38.550677+0000", 2024-09-13T11:25:49.709 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00020138099999999999 2024-09-13T11:25:49.709 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.709 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.709 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 12", 2024-09-13T11:25:49.709 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:38.550836+0000", 2024-09-13T11:25:49.709 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000158722 2024-09-13T11:25:49.709 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.709 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.709 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:25:49.709 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:38.553897+0000", 2024-09-13T11:25:49.710 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.003061533 2024-09-13T11:25:49.710 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.710 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.710 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:25:49.710 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:38.841929+0000", 2024-09-13T11:25:49.710 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.28803159 2024-09-13T11:25:49.710 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.710 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.710 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:25:49.710 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:38.842014+0000", 2024-09-13T11:25:49.710 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.5119999999999998e-05 2024-09-13T11:25:49.710 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.710 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.710 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:49.711 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:38.842122+0000", 2024-09-13T11:25:49.711 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00010863599999999999 2024-09-13T11:25:49.711 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:49.711 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:49.711 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:49.711 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:49.711 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:49.711 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-13T11:25:49.713 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_ops_in_flight 2024-09-13T11:25:49.713 INFO:tasks.ceph.osd.4.smithi001.stderr:2024-09-13T11:25:49.711+0000 7ff8e0218640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26310) UID: 0 2024-09-13T11:25:49.813 INFO:tasks.ceph.osd.8.smithi001.stderr:2024-09-13T11:25:49.811+0000 7fb9b48ae640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26312) UID: 0 2024-09-13T11:25:49.914 INFO:tasks.ceph.osd.9.smithi092.stderr:2024-09-13T11:25:49.913+0000 7ff5ab3d0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26348) UID: 0 2024-09-13T11:25:49.918 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-13T11:25:49.918 INFO:teuthology.orchestra.run.smithi001.stdout: "ops": [ 2024-09-13T11:25:49.918 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.918 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:204080 3.2 3:420bacc3:::benchmark_data_smithi177_31116_object63777:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e182)", 2024-09-13T11:25:49.918 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:25:49.917561+0000", 2024-09-13T11:25:49.918 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 0.0001449, 2024-09-13T11:25:49.918 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00016596499999999999, 2024-09-13T11:25:49.918 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:49.918 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:49.919 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-13T11:25:49.919 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:25:49.919 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:25:49.919 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:49.919 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 204080 2024-09-13T11:25:49.919 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.919 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:49.919 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.919 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:49.919 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:49.917561+0000", 2024-09-13T11:25:49.919 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:49.919 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.919 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.920 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:49.920 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:49.917561+0000", 2024-09-13T11:25:49.920 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:49.920 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.920 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.920 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:49.920 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:49.917563+0000", 2024-09-13T11:25:49.920 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.8619999999999999e-06 2024-09-13T11:25:49.920 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.920 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.920 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:49.920 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:49.917572+0000", 2024-09-13T11:25:49.920 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.259e-06 2024-09-13T11:25:49.920 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.921 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.921 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:49.921 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:49.917585+0000", 2024-09-13T11:25:49.921 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2947e-05 2024-09-13T11:25:49.921 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.921 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.921 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:49.921 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:49.917591+0000", 2024-09-13T11:25:49.921 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.8399999999999997e-06 2024-09-13T11:25:49.921 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.921 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.921 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:49.921 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:49.917654+0000", 2024-09-13T11:25:49.922 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.3066999999999998e-05 2024-09-13T11:25:49.922 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:49.922 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:49.922 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:49.922 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.922 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.922 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:204052 3.9 3:91e3f11c:::benchmark_data_smithi177_31116_object63749:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e182)", 2024-09-13T11:25:49.922 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:25:49.897763+0000", 2024-09-13T11:25:49.922 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 0.019943809, 2024-09-13T11:25:49.922 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.020035246, 2024-09-13T11:25:49.922 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:49.922 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:49.922 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "waiting for sub ops", 2024-09-13T11:25:49.922 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:25:49.923 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:25:49.923 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:49.923 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 204052 2024-09-13T11:25:49.923 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.923 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:49.923 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.923 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:49.923 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:49.897763+0000", 2024-09-13T11:25:49.923 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:49.923 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.923 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.923 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:49.923 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:49.897763+0000", 2024-09-13T11:25:49.923 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:49.924 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.924 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.924 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:49.924 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:49.897763+0000", 2024-09-13T11:25:49.924 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.5199999999999996e-07 2024-09-13T11:25:49.924 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.924 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.924 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:49.924 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:49.897766+0000", 2024-09-13T11:25:49.924 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.7439999999999999e-06 2024-09-13T11:25:49.924 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.924 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.924 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:49.924 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:49.897772+0000", 2024-09-13T11:25:49.925 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.7690000000000002e-06 2024-09-13T11:25:49.925 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.925 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.925 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:49.925 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:49.897777+0000", 2024-09-13T11:25:49.925 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.4489999999999998e-06 2024-09-13T11:25:49.925 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.925 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.925 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:49.925 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:49.897816+0000", 2024-09-13T11:25:49.925 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.9326999999999998e-05 2024-09-13T11:25:49.925 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.925 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.925 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:25:49.926 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:49.897887+0000", 2024-09-13T11:25:49.926 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.0483999999999998e-05 2024-09-13T11:25:49.926 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.926 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.926 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 2", 2024-09-13T11:25:49.926 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:49.897967+0000", 2024-09-13T11:25:49.926 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.9797999999999997e-05 2024-09-13T11:25:49.926 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.926 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.926 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:25:49.926 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:49.899851+0000", 2024-09-13T11:25:49.926 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.001884396 2024-09-13T11:25:49.926 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:49.927 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:49.927 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:49.927 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.927 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.927 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:204053 3.25 3:a54e0e16:::benchmark_data_smithi177_31116_object63750:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e182)", 2024-09-13T11:25:49.927 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:25:49.897889+0000", 2024-09-13T11:25:49.927 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 0.019817508000000001, 2024-09-13T11:25:49.927 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.019987004999999999, 2024-09-13T11:25:49.927 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:49.927 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:49.927 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "waiting for sub ops", 2024-09-13T11:25:49.927 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:25:49.927 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:25:49.927 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:49.928 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 204053 2024-09-13T11:25:49.928 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.928 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:49.928 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.928 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:49.928 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:49.897889+0000", 2024-09-13T11:25:49.928 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:49.928 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.928 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.928 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:49.928 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:49.897889+0000", 2024-09-13T11:25:49.928 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:49.928 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.929 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.929 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:49.929 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:49.897890+0000", 2024-09-13T11:25:49.929 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.7100000000000011e-07 2024-09-13T11:25:49.929 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.929 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.929 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:49.929 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:49.897893+0000", 2024-09-13T11:25:49.929 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.4620000000000001e-06 2024-09-13T11:25:49.929 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.929 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.929 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:49.929 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:49.897901+0000", 2024-09-13T11:25:49.929 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.4209999999999998e-06 2024-09-13T11:25:49.930 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.930 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.930 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:49.930 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:49.897907+0000", 2024-09-13T11:25:49.930 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.139e-06 2024-09-13T11:25:49.930 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.930 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.930 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:49.930 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:49.897955+0000", 2024-09-13T11:25:49.930 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.8426e-05 2024-09-13T11:25:49.930 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.930 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.930 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:25:49.930 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:49.898068+0000", 2024-09-13T11:25:49.931 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000112459 2024-09-13T11:25:49.931 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.931 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.931 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 2", 2024-09-13T11:25:49.931 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:49.898160+0000", 2024-09-13T11:25:49.931 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.1978000000000006e-05 2024-09-13T11:25:49.931 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.931 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.931 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:25:49.931 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:49.900101+0000", 2024-09-13T11:25:49.931 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.001941676 2024-09-13T11:25:49.931 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:49.931 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:49.931 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:49.932 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.932 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.932 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:204073 3.1b 3:dc265e25:::benchmark_data_smithi177_31116_object63770:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e182)", 2024-09-13T11:25:49.932 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:25:49.911782+0000", 2024-09-13T11:25:49.932 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 0.0059245840000000001, 2024-09-13T11:25:49.932 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.006169931, 2024-09-13T11:25:49.932 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:49.932 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:49.932 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "waiting for sub ops", 2024-09-13T11:25:49.932 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:25:49.932 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:25:49.932 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:49.932 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 204073 2024-09-13T11:25:49.933 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.933 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:49.933 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.933 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:49.933 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:49.911782+0000", 2024-09-13T11:25:49.933 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:49.933 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.933 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.933 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:49.933 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:49.911782+0000", 2024-09-13T11:25:49.933 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:49.933 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.934 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.934 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:49.934 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:49.911783+0000", 2024-09-13T11:25:49.934 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.6899999999999999e-06 2024-09-13T11:25:49.934 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.934 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.934 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:49.934 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:49.911788+0000", 2024-09-13T11:25:49.934 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.9089999999999999e-06 2024-09-13T11:25:49.934 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.934 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.934 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:49.934 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:49.911802+0000", 2024-09-13T11:25:49.934 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.3359e-05 2024-09-13T11:25:49.935 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.935 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.935 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:49.935 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:49.911811+0000", 2024-09-13T11:25:49.935 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.9900000000000003e-06 2024-09-13T11:25:49.935 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.935 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.935 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:49.935 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:49.911854+0000", 2024-09-13T11:25:49.935 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.3210000000000001e-05 2024-09-13T11:25:49.935 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.935 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.935 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:25:49.935 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:49.911927+0000", 2024-09-13T11:25:49.936 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.3237000000000004e-05 2024-09-13T11:25:49.936 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.936 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.936 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 15", 2024-09-13T11:25:49.936 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:49.911983+0000", 2024-09-13T11:25:49.936 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.5504000000000003e-05 2024-09-13T11:25:49.936 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.936 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.936 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:25:49.936 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:49.913801+0000", 2024-09-13T11:25:49.936 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00181857 2024-09-13T11:25:49.936 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:49.936 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:49.936 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:49.937 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.937 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.937 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4665.0:204079 3.f e182/75 3:f0219fca:::benchmark_data_smithi177_31116_object63776:head v 182'8134, mlcod=182'8132)", 2024-09-13T11:25:49.937 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:25:49.916966+0000", 2024-09-13T11:25:49.937 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 0.00074034999999999997, 2024-09-13T11:25:49.937 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.001061396, 2024-09-13T11:25:49.937 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:49.937 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:49.937 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:25:49.937 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:49.937 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.937 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:49.937 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:49.916966+0000", 2024-09-13T11:25:49.937 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:49.938 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.938 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.938 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:49.938 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:49.916966+0000", 2024-09-13T11:25:49.938 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:49.938 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.938 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.939 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:49.939 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:49.916968+0000", 2024-09-13T11:25:49.939 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.6300000000000001e-06 2024-09-13T11:25:49.939 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.939 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.939 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:49.939 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:49.916979+0000", 2024-09-13T11:25:49.939 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0900999999999999e-05 2024-09-13T11:25:49.939 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.939 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.939 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:49.939 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:49.917012+0000", 2024-09-13T11:25:49.939 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.3895e-05 2024-09-13T11:25:49.939 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.940 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.940 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:49.940 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:49.917020+0000", 2024-09-13T11:25:49.940 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.6159999999999996e-06 2024-09-13T11:25:49.940 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.940 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.940 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:49.940 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:49.917075+0000", 2024-09-13T11:25:49.940 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.4984000000000001e-05 2024-09-13T11:25:49.940 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:49.940 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:49.940 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:25:49.940 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:49.917088+0000", 2024-09-13T11:25:49.941 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.3095000000000001e-05 2024-09-13T11:25:49.941 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:49.941 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:49.941 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:49.941 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:49.941 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-09-13T11:25:49.941 INFO:teuthology.orchestra.run.smithi001.stdout: "num_ops": 5 2024-09-13T11:25:49.941 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-13T11:25:49.942 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_blocked_ops 2024-09-13T11:25:50.016 INFO:tasks.ceph.osd.7.smithi177.stderr:2024-09-13T11:25:50.014+0000 7f897b51d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26575) UID: 0 2024-09-13T11:25:50.115 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-13T11:25:50.115 INFO:teuthology.orchestra.run.smithi001.stdout: "ops": [], 2024-09-13T11:25:50.115 INFO:teuthology.orchestra.run.smithi001.stdout: "complaint_time": 30, 2024-09-13T11:25:50.115 INFO:teuthology.orchestra.run.smithi001.stdout: "num_blocked_ops": 0 2024-09-13T11:25:50.115 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-13T11:25:50.116 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-13T11:25:50.115+0000 7f6a68775640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26307) UID: 0 2024-09-13T11:25:50.131 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T11:25:50.133 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_historic_ops 2024-09-13T11:25:50.218 INFO:tasks.ceph.osd.9.smithi092.stderr:2024-09-13T11:25:50.217+0000 7ff5ab3d0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26348) UID: 0 2024-09-13T11:25:50.313 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-13T11:25:50.314 INFO:teuthology.orchestra.run.smithi001.stdout: "size": 20, 2024-09-13T11:25:50.314 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 600, 2024-09-13T11:25:50.314 INFO:teuthology.orchestra.run.smithi001.stdout: "ops": [ 2024-09-13T11:25:50.314 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:50.314 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop_reply(client.4665.0:167 3.9 e41/30 ondisk, result = 0)", 2024-09-13T11:25:50.314 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:22:47.419696+0000", 2024-09-13T11:25:50.314 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 182.89205760900001, 2024-09-13T11:25:50.314 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.64615778199999996, 2024-09-13T11:25:50.315 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:50.315 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:50.315 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:25:50.315 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:50.315 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:50.315 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:50.315 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:47.419696+0000", 2024-09-13T11:25:50.315 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:50.315 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:50.315 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:50.315 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:50.315 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:47.419696+0000", 2024-09-13T11:25:50.316 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:50.316 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:50.316 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:50.316 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:50.316 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:47.419697+0000", 2024-09-13T11:25:50.316 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.15e-06 2024-09-13T11:25:50.316 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:50.316 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:50.316 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:50.316 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:47.419700+0000", 2024-09-13T11:25:50.316 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.2449999999999998e-06 2024-09-13T11:25:50.317 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:50.320 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:50.320 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:50.320 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:48.065637+0000", 2024-09-13T11:25:50.320 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.64593674300000004 2024-09-13T11:25:50.321 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:50.321 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:50.321 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:50.321 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:48.065648+0000", 2024-09-13T11:25:50.321 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0865e-05 2024-09-13T11:25:50.321 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:50.321 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:50.321 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:50.321 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:48.065720+0000", 2024-09-13T11:25:50.321 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.2020999999999994e-05 2024-09-13T11:25:50.321 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:50.322 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:50.322 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:25:50.322 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:48.065728+0000", 2024-09-13T11:25:50.322 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.7660000000000004e-06 2024-09-13T11:25:50.322 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:50.322 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:50.322 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:50.322 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:48.065853+0000", 2024-09-13T11:25:50.322 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00012499200000000001 2024-09-13T11:25:50.322 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:50.322 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:50.322 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:50.323 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:50.323 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:50.323 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:25383 3.0 3:0a6bb869:::benchmark_data_smithi177_31116_object25382:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e55)", 2024-09-13T11:25:50.323 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:04.447376+0000", 2024-09-13T11:25:50.323 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 165.86437694, 2024-09-13T11:25:50.323 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.21239864, 2024-09-13T11:25:50.323 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:50.324 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:50.324 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:50.324 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:25:50.324 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:25:50.324 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:50.324 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 25383 2024-09-13T11:25:50.324 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:50.324 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:50.324 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:50.324 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:50.324 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.447376+0000", 2024-09-13T11:25:50.324 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:50.325 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:50.325 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:50.325 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:50.325 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.447376+0000", 2024-09-13T11:25:50.325 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:50.325 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:50.325 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:50.325 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:50.325 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.447378+0000", 2024-09-13T11:25:50.325 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.6729999999999999e-06 2024-09-13T11:25:50.325 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:50.326 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:50.326 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:50.326 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.447433+0000", 2024-09-13T11:25:50.326 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.5371000000000002e-05 2024-09-13T11:25:50.326 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:50.326 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:50.327 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:50.327 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.447446+0000", 2024-09-13T11:25:50.327 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.3179999999999999e-05 2024-09-13T11:25:50.327 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:50.327 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:50.327 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:50.327 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.447456+0000", 2024-09-13T11:25:50.327 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0008e-05 2024-09-13T11:25:50.327 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:50.327 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:50.327 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:50.327 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.447514+0000", 2024-09-13T11:25:50.327 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.8023999999999999e-05 2024-09-13T11:25:50.328 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:50.328 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:50.328 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:25:50.328 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.447642+0000", 2024-09-13T11:25:50.328 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000127296 2024-09-13T11:25:50.328 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:50.328 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:50.328 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 15", 2024-09-13T11:25:50.329 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.447767+0000", 2024-09-13T11:25:50.329 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000125547 2024-09-13T11:25:50.329 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:50.329 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:50.329 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:25:50.329 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.449188+0000", 2024-09-13T11:25:50.329 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.001420604 2024-09-13T11:25:50.329 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:50.329 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:50.329 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:25:50.330 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.659668+0000", 2024-09-13T11:25:50.330 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.210479685 2024-09-13T11:25:50.330 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:50.330 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:50.330 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:25:50.330 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.659716+0000", 2024-09-13T11:25:50.330 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.8609999999999997e-05 2024-09-13T11:25:50.330 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:50.330 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:50.330 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:50.331 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.659775+0000", 2024-09-13T11:25:50.331 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.8641999999999999e-05 2024-09-13T11:25:50.331 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:50.331 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:50.331 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:50.331 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:50.331 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:50.331 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4665.0:25408 3.4 e56/40 3:294e23bd:::benchmark_data_smithi177_31116_object25407:head v 56'1645, mlcod=55'1642)", 2024-09-13T11:25:50.332 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:04.494686+0000", 2024-09-13T11:25:50.332 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 165.81706728099999, 2024-09-13T11:25:50.332 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.29725566399999998, 2024-09-13T11:25:50.332 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:50.332 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:50.332 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:50.332 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:50.332 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:50.332 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:50.332 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.494686+0000", 2024-09-13T11:25:50.333 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:50.333 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:50.333 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:50.333 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:50.333 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.494686+0000", 2024-09-13T11:25:50.333 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:50.333 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:50.333 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:50.334 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:50.334 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.494686+0000", 2024-09-13T11:25:50.334 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.2399999999999999e-07 2024-09-13T11:25:50.334 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:50.334 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:50.334 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:50.335 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.494809+0000", 2024-09-13T11:25:50.335 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00012308600000000001 2024-09-13T11:25:50.335 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:50.335 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:50.335 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:50.335 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.789410+0000", 2024-09-13T11:25:50.335 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.29460066899999998 2024-09-13T11:25:50.335 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:50.336 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:50.336 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:50.336 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.789425+0000", 2024-09-13T11:25:50.336 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.4600000000000001e-05 2024-09-13T11:25:50.336 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:50.336 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:50.336 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:50.336 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.789507+0000", 2024-09-13T11:25:50.337 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.2785000000000001e-05 2024-09-13T11:25:50.337 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:50.337 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:50.337 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:25:50.337 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.789525+0000", 2024-09-13T11:25:50.338 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.8117999999999998e-05 2024-09-13T11:25:50.338 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:50.338 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:50.338 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:25:50.338 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.791852+0000", 2024-09-13T11:25:50.338 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0023264459999999998 2024-09-13T11:25:50.338 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:50.339 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:50.339 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:50.339 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.791942+0000", 2024-09-13T11:25:50.339 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.9635999999999994e-05 2024-09-13T11:25:50.340 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:50.340 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:50.340 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:50.340 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:50.340 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:50.340 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4665.0:30815 3.f e59/30 3:fde17cb1:::benchmark_data_smithi177_31116_object30814:head v 59'1919, mlcod=59'1914)", 2024-09-13T11:25:50.341 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:09.494342+0000", 2024-09-13T11:25:50.341 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 160.817411295, 2024-09-13T11:25:50.341 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.22787855700000001, 2024-09-13T11:25:50.341 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:50.341 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:50.341 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:50.341 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:50.342 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:50.342 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:50.342 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.494342+0000", 2024-09-13T11:25:50.342 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:50.342 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:50.342 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:50.342 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:50.342 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.494342+0000", 2024-09-13T11:25:50.342 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:50.342 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:50.342 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:50.342 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:50.343 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.494343+0000", 2024-09-13T11:25:50.343 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.251e-06 2024-09-13T11:25:50.343 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:50.343 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:50.343 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:50.343 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.494387+0000", 2024-09-13T11:25:50.343 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.4276000000000001e-05 2024-09-13T11:25:50.343 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:50.343 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:50.343 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:50.343 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.720454+0000", 2024-09-13T11:25:50.343 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.226066614 2024-09-13T11:25:50.344 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:50.344 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:50.344 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:50.344 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.720462+0000", 2024-09-13T11:25:50.344 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.9240000000000007e-06 2024-09-13T11:25:50.344 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:50.344 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:50.344 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:50.344 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.720537+0000", 2024-09-13T11:25:50.344 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.4967000000000002e-05 2024-09-13T11:25:50.344 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:50.344 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:50.345 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:25:50.345 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.720563+0000", 2024-09-13T11:25:50.345 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.5630999999999999e-05 2024-09-13T11:25:50.345 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:50.345 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:50.345 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:25:50.345 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.722183+0000", 2024-09-13T11:25:50.345 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0016209670000000001 2024-09-13T11:25:50.345 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:50.345 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:50.345 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:50.345 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.722220+0000", 2024-09-13T11:25:50.346 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.6927000000000001e-05 2024-09-13T11:25:50.346 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:50.346 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:50.346 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:50.346 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:50.346 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:50.346 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4665.0:30828 3.f e59/30 3:f3a9c6a9:::benchmark_data_smithi177_31116_object30827:head v 59'1920, mlcod=59'1916)", 2024-09-13T11:25:50.346 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:09.502607+0000", 2024-09-13T11:25:50.346 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 160.80914623800001, 2024-09-13T11:25:50.346 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.26374408300000002, 2024-09-13T11:25:50.346 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:50.346 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:50.347 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:50.347 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:50.347 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:50.347 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:50.347 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.502607+0000", 2024-09-13T11:25:50.347 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:50.348 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:50.348 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:50.348 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:50.348 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.502607+0000", 2024-09-13T11:25:50.348 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:50.348 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:50.349 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:50.349 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:50.349 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.502609+0000", 2024-09-13T11:25:50.349 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.84e-06 2024-09-13T11:25:50.349 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:50.349 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:50.350 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:50.350 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.502663+0000", 2024-09-13T11:25:50.350 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.3971000000000002e-05 2024-09-13T11:25:50.350 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:50.350 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:50.350 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:50.350 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.763912+0000", 2024-09-13T11:25:50.351 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.26124904300000001 2024-09-13T11:25:50.351 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:50.351 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:50.351 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:50.351 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.763920+0000", 2024-09-13T11:25:50.351 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.1149999999999994e-06 2024-09-13T11:25:50.351 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:50.352 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:50.352 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:50.352 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.764024+0000", 2024-09-13T11:25:50.352 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00010380899999999999 2024-09-13T11:25:50.352 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:50.352 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:50.352 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:25:50.353 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.764068+0000", 2024-09-13T11:25:50.353 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.4638999999999997e-05 2024-09-13T11:25:50.353 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:50.353 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:50.353 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:25:50.353 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.766308+0000", 2024-09-13T11:25:50.354 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0022392039999999998 2024-09-13T11:25:50.354 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:50.354 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:50.354 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:50.354 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.766351+0000", 2024-09-13T11:25:50.354 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.3461999999999998e-05 2024-09-13T11:25:50.355 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:50.355 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:50.355 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:50.355 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:50.355 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:50.355 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDPGPush(3.1b 84/78 [PushOp(3:dcddd139:::benchmark_data_smithi177_31116_object10384:head, version: 48'673, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:dcddd139:::benchmark_data_smithi177_31116_object10384:head@48'673, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:25:50.355 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:41.400602+0000", 2024-09-13T11:25:50.356 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 128.91115075299999, 2024-09-13T11:25:50.356 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.43365821999999998, 2024-09-13T11:25:50.356 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:50.356 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:50.356 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:25:50.356 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:50.356 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:50.356 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:50.356 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:41.400602+0000", 2024-09-13T11:25:50.356 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:50.356 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:50.357 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:50.357 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:50.357 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:41.400602+0000", 2024-09-13T11:25:50.357 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:50.357 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:50.357 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:50.357 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:50.357 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:41.400604+0000", 2024-09-13T11:25:50.357 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.99e-06 2024-09-13T11:25:50.357 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:50.357 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:50.357 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:50.357 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:41.400653+0000", 2024-09-13T11:25:50.358 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.8195000000000002e-05 2024-09-13T11:25:50.358 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:50.358 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:50.358 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:50.358 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:41.833180+0000", 2024-09-13T11:25:50.358 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.43252769200000002 2024-09-13T11:25:50.358 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:50.358 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:50.358 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:50.358 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:41.833213+0000", 2024-09-13T11:25:50.358 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.2737999999999997e-05 2024-09-13T11:25:50.358 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:50.358 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:50.359 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:50.359 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:41.833408+0000", 2024-09-13T11:25:50.359 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00019473799999999999 2024-09-13T11:25:50.359 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:50.359 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:50.359 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:25:50.359 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:41.833417+0000", 2024-09-13T11:25:50.359 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.5349999999999993e-06 2024-09-13T11:25:50.359 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:50.359 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:50.359 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:50.359 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:41.834261+0000", 2024-09-13T11:25:50.360 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00084333200000000002 2024-09-13T11:25:50.360 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:50.360 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:50.360 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:50.360 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:50.360 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:50.360 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDPGPushReply(3.20 85/77 [PushReplyOp(3:066982a6:::benchmark_data_smithi177_31116_object16400:head)])", 2024-09-13T11:25:50.360 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:42.671551+0000", 2024-09-13T11:25:50.360 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 127.64020193, 2024-09-13T11:25:50.360 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.28249085499999999, 2024-09-13T11:25:50.360 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:50.360 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:50.361 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-13T11:25:50.361 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:50.361 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:50.361 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:50.361 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.671551+0000", 2024-09-13T11:25:50.361 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:50.361 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:50.361 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:50.361 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:50.361 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.671551+0000", 2024-09-13T11:25:50.361 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:50.362 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:50.362 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:50.362 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:50.362 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.671552+0000", 2024-09-13T11:25:50.362 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.097e-06 2024-09-13T11:25:50.362 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:50.362 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:50.362 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:50.362 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.671556+0000", 2024-09-13T11:25:50.362 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.8519999999999997e-06 2024-09-13T11:25:50.362 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:50.362 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:50.362 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:50.363 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.951769+0000", 2024-09-13T11:25:50.363 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.28021293000000003 2024-09-13T11:25:50.363 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:50.363 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:50.363 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:50.363 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.951780+0000", 2024-09-13T11:25:50.363 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.1059e-05 2024-09-13T11:25:50.363 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:50.363 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:50.363 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:50.363 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.951992+0000", 2024-09-13T11:25:50.363 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000211478 2024-09-13T11:25:50.363 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:50.364 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:50.364 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:50.364 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.954042+0000", 2024-09-13T11:25:50.364 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0020504389999999998 2024-09-13T11:25:50.364 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:50.364 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:50.364 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:50.364 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:50.364 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:50.364 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDPGPush(3.1b 89/78 [PushOp(3:dc4c79e7:::benchmark_data_smithi177_31116_object19857:head, version: 54'1266, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:dc4c79e7:::benchmark_data_smithi177_31116_object19857:head@54'1266, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:25:50.364 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:46.649128+0000", 2024-09-13T11:25:50.364 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 123.662624822, 2024-09-13T11:25:50.364 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.29963603, 2024-09-13T11:25:50.365 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:50.365 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:50.365 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:25:50.365 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:50.365 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:50.365 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:50.365 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.649128+0000", 2024-09-13T11:25:50.365 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:50.365 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:50.365 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:50.365 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:50.365 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.649128+0000", 2024-09-13T11:25:50.366 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:50.366 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:50.366 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:50.366 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:50.366 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.649130+0000", 2024-09-13T11:25:50.366 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2950000000000001e-06 2024-09-13T11:25:50.366 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:50.366 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:50.366 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:50.367 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.649197+0000", 2024-09-13T11:25:50.367 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.7033999999999995e-05 2024-09-13T11:25:50.367 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:50.367 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:50.367 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:50.367 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.947691+0000", 2024-09-13T11:25:50.367 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.29849387300000002 2024-09-13T11:25:50.367 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:50.367 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:50.367 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:50.367 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.947702+0000", 2024-09-13T11:25:50.368 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.1239e-05 2024-09-13T11:25:50.368 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:50.368 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:50.368 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:50.368 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.947799+0000", 2024-09-13T11:25:50.368 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.7283999999999999e-05 2024-09-13T11:25:50.368 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:50.368 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:50.368 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:25:50.368 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.947809+0000", 2024-09-13T11:25:50.368 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.008e-05 2024-09-13T11:25:50.368 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:50.369 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:50.369 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:50.369 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.948764+0000", 2024-09-13T11:25:50.369 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00095522499999999998 2024-09-13T11:25:50.369 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:50.369 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:50.369 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:50.369 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:50.369 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:50.369 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:71927 3.0 3:0197244a:::benchmark_data_smithi177_31116_object71926:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2024-09-13T11:25:50.369 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:54.199022+0000", 2024-09-13T11:25:50.370 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 116.112731466, 2024-09-13T11:25:50.370 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.26819617800000001, 2024-09-13T11:25:50.370 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:50.370 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:50.370 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:50.370 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:25:50.370 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:25:50.371 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:50.371 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 71927 2024-09-13T11:25:50.371 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:50.371 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:50.371 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:50.371 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:50.371 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:54.199022+0000", 2024-09-13T11:25:50.371 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:50.371 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:50.371 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:50.371 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:50.371 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:54.199022+0000", 2024-09-13T11:25:50.371 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:50.372 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:50.372 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:50.372 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:50.372 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:54.199025+0000", 2024-09-13T11:25:50.372 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.2200000000000001e-06 2024-09-13T11:25:50.372 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:50.372 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:50.372 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:50.372 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:54.199076+0000", 2024-09-13T11:25:50.372 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.1184000000000003e-05 2024-09-13T11:25:50.372 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:50.372 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:50.372 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:50.373 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:54.199095+0000", 2024-09-13T11:25:50.373 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.9253999999999999e-05 2024-09-13T11:25:50.373 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:50.373 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:50.373 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:50.373 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:54.199107+0000", 2024-09-13T11:25:50.373 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.1916000000000001e-05 2024-09-13T11:25:50.373 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:50.373 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:50.373 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:50.373 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:54.199196+0000", 2024-09-13T11:25:50.373 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.8645000000000003e-05 2024-09-13T11:25:50.373 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:50.374 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:50.374 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:25:50.374 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:54.199421+0000", 2024-09-13T11:25:50.374 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000224868 2024-09-13T11:25:50.374 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:50.374 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:50.374 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 15", 2024-09-13T11:25:50.374 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:54.199647+0000", 2024-09-13T11:25:50.374 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00022666099999999999 2024-09-13T11:25:50.374 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:50.374 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:50.374 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:25:50.375 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:54.201333+0000", 2024-09-13T11:25:50.375 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.001685761 2024-09-13T11:25:50.375 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:50.375 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:50.375 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:25:50.375 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:54.467097+0000", 2024-09-13T11:25:50.375 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.26576393399999998 2024-09-13T11:25:50.375 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:50.375 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:50.375 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:25:50.375 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:54.467163+0000", 2024-09-13T11:25:50.375 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.6088999999999997e-05 2024-09-13T11:25:50.375 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:50.375 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:50.376 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:50.376 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:54.467218+0000", 2024-09-13T11:25:50.376 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.4645999999999998e-05 2024-09-13T11:25:50.376 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:50.376 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:50.376 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:50.376 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:50.376 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:50.376 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4665.0:98479 3.2 e113/75 3:41084616:::benchmark_data_smithi177_31116_object98478:head v 113'3785, mlcod=113'3782)", 2024-09-13T11:25:50.376 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:24:15.343730+0000", 2024-09-13T11:25:50.376 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 94.968023196000004, 2024-09-13T11:25:50.377 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.196889648, 2024-09-13T11:25:50.377 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:50.377 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:50.377 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:50.377 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:50.377 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:50.377 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:50.377 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.343730+0000", 2024-09-13T11:25:50.377 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:50.377 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:50.377 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:50.377 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:50.377 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.343730+0000", 2024-09-13T11:25:50.378 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:50.378 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:50.378 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:50.378 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:50.378 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.343732+0000", 2024-09-13T11:25:50.378 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.6819999999999999e-06 2024-09-13T11:25:50.378 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:50.378 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:50.378 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:50.379 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.343781+0000", 2024-09-13T11:25:50.379 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.9691999999999998e-05 2024-09-13T11:25:50.379 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:50.379 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:50.379 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:50.379 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.538204+0000", 2024-09-13T11:25:50.379 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.19442271699999999 2024-09-13T11:25:50.379 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:50.379 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:50.379 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:50.379 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.538215+0000", 2024-09-13T11:25:50.379 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0648e-05 2024-09-13T11:25:50.379 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:50.379 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:50.380 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:50.380 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.538342+0000", 2024-09-13T11:25:50.380 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000127448 2024-09-13T11:25:50.380 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:50.380 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:50.380 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:25:50.380 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.538395+0000", 2024-09-13T11:25:50.380 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.2895000000000001e-05 2024-09-13T11:25:50.380 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:50.380 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:50.380 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:25:50.380 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.540544+0000", 2024-09-13T11:25:50.380 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0021487379999999999 2024-09-13T11:25:50.381 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:50.381 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:50.381 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:50.381 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.540620+0000", 2024-09-13T11:25:50.381 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.5828000000000007e-05 2024-09-13T11:25:50.381 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:50.381 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:50.381 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:50.381 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:50.381 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:50.381 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDPGPush(3.25 121/120 [PushOp(3:a5de2229:::benchmark_data_smithi177_31116_object1005:head, version: 43'62, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:a5de2229:::benchmark_data_smithi177_31116_object1005:head@43'62, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:25:50.382 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:24:27.379880+0000", 2024-09-13T11:25:50.382 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 82.931873628000005, 2024-09-13T11:25:50.382 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.72647719499999996, 2024-09-13T11:25:50.382 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:50.382 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:50.382 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:25:50.382 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:50.382 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:50.382 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:50.382 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:27.379880+0000", 2024-09-13T11:25:50.382 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:50.382 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:50.382 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:50.383 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:50.383 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:27.379880+0000", 2024-09-13T11:25:50.383 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:50.383 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:50.383 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:50.383 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:50.383 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:27.379882+0000", 2024-09-13T11:25:50.383 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.1050000000000002e-06 2024-09-13T11:25:50.383 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:50.383 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:50.383 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:50.383 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:27.379976+0000", 2024-09-13T11:25:50.384 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.4468000000000004e-05 2024-09-13T11:25:50.384 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:50.384 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:50.384 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:50.384 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:28.105075+0000", 2024-09-13T11:25:50.384 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.72509911299999996 2024-09-13T11:25:50.384 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:50.384 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:50.384 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:50.384 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:28.105111+0000", 2024-09-13T11:25:50.384 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.5403e-05 2024-09-13T11:25:50.384 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:50.384 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:50.385 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:50.385 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:28.105267+0000", 2024-09-13T11:25:50.385 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00015663600000000001 2024-09-13T11:25:50.385 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:50.385 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:50.385 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:25:50.385 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:28.105291+0000", 2024-09-13T11:25:50.385 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.4164000000000001e-05 2024-09-13T11:25:50.385 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:50.385 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:50.386 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:50.386 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:28.106357+0000", 2024-09-13T11:25:50.386 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.001065306 2024-09-13T11:25:50.386 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:50.386 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:50.386 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:50.386 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:50.386 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:50.386 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:113646 3.20 3:059047a0:::benchmark_data_smithi177_31116_object113645:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124)", 2024-09-13T11:25:50.386 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:24:32.178229+0000", 2024-09-13T11:25:50.386 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 78.133523667999995, 2024-09-13T11:25:50.386 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.23346934699999999, 2024-09-13T11:25:50.387 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:50.387 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:50.387 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:50.387 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:25:50.387 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:25:50.387 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:50.387 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 113646 2024-09-13T11:25:50.387 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:50.387 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:50.387 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:50.387 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:50.387 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:32.178229+0000", 2024-09-13T11:25:50.387 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:50.388 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:50.388 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:50.388 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:50.388 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:32.178229+0000", 2024-09-13T11:25:50.388 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:50.388 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:50.388 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:50.388 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:50.388 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:32.178232+0000", 2024-09-13T11:25:50.388 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.069e-06 2024-09-13T11:25:50.388 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:50.388 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:50.388 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:50.388 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:32.178266+0000", 2024-09-13T11:25:50.389 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.4242999999999997e-05 2024-09-13T11:25:50.389 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:50.389 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:50.389 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:50.389 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:32.178278+0000", 2024-09-13T11:25:50.389 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2558000000000001e-05 2024-09-13T11:25:50.389 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:50.389 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:50.389 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:50.389 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:32.178288+0000", 2024-09-13T11:25:50.389 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.6469999999999993e-06 2024-09-13T11:25:50.389 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:50.389 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:50.390 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:50.390 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:32.178349+0000", 2024-09-13T11:25:50.390 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.0643999999999997e-05 2024-09-13T11:25:50.390 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:50.390 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:50.390 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:25:50.390 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:32.178477+0000", 2024-09-13T11:25:50.390 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000128038 2024-09-13T11:25:50.390 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:50.390 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:50.390 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 0", 2024-09-13T11:25:50.390 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:32.178602+0000", 2024-09-13T11:25:50.391 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00012553600000000001 2024-09-13T11:25:50.391 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:50.391 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:50.391 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:25:50.391 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:32.180157+0000", 2024-09-13T11:25:50.391 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.001554386 2024-09-13T11:25:50.391 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:50.391 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:50.391 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:25:50.391 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:32.411634+0000", 2024-09-13T11:25:50.391 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.231477561 2024-09-13T11:25:50.391 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:50.391 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:50.392 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:25:50.392 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:32.411672+0000", 2024-09-13T11:25:50.392 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.7357999999999997e-05 2024-09-13T11:25:50.392 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:50.392 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:50.392 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:50.392 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:32.411699+0000", 2024-09-13T11:25:50.392 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.7307000000000001e-05 2024-09-13T11:25:50.392 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:50.392 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:50.392 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:50.392 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:50.392 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:50.392 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop_reply(client.4665.0:113646 3.20 e124/77 ondisk, result = 0)", 2024-09-13T11:25:50.393 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:24:32.180524+0000", 2024-09-13T11:25:50.393 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 78.131228887000006, 2024-09-13T11:25:50.393 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.231180894, 2024-09-13T11:25:50.393 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:50.393 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:50.393 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:25:50.393 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:50.393 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:50.393 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:50.393 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:32.180524+0000", 2024-09-13T11:25:50.393 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:50.393 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:50.394 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:50.394 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:50.394 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:32.180524+0000", 2024-09-13T11:25:50.394 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:50.394 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:50.394 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:50.394 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:50.394 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:32.180526+0000", 2024-09-13T11:25:50.394 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.4419999999999999e-06 2024-09-13T11:25:50.394 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:50.394 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:50.395 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:50.395 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:32.180530+0000", 2024-09-13T11:25:50.395 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.7330000000000003e-06 2024-09-13T11:25:50.395 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:50.395 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:50.395 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:50.395 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:32.411549+0000", 2024-09-13T11:25:50.395 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.23101862300000001 2024-09-13T11:25:50.396 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:50.396 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:50.396 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:50.396 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:32.411558+0000", 2024-09-13T11:25:50.396 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.5949999999999999e-06 2024-09-13T11:25:50.396 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:50.397 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:50.397 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:50.397 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:32.411622+0000", 2024-09-13T11:25:50.397 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.4251000000000005e-05 2024-09-13T11:25:50.397 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:50.397 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:50.398 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:25:50.398 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:32.411629+0000", 2024-09-13T11:25:50.398 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.328e-06 2024-09-13T11:25:50.398 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:50.398 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:50.398 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:50.399 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:32.411705+0000", 2024-09-13T11:25:50.399 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.5921999999999996e-05 2024-09-13T11:25:50.399 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:50.399 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:50.399 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:50.399 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:50.400 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:50.400 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop_reply(client.4665.0:129892 3.0 e134/75 ondisk, result = 0)", 2024-09-13T11:25:50.400 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:24:47.136509+0000", 2024-09-13T11:25:50.400 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 63.17524384, 2024-09-13T11:25:50.400 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.211101067, 2024-09-13T11:25:50.400 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:50.400 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:50.400 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:25:50.401 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:50.401 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:50.401 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:50.401 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:47.136509+0000", 2024-09-13T11:25:50.401 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:50.401 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:50.401 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:50.402 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:50.402 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:47.136509+0000", 2024-09-13T11:25:50.402 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:50.402 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:50.402 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:50.402 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:50.403 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:47.136510+0000", 2024-09-13T11:25:50.403 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.17e-06 2024-09-13T11:25:50.403 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:50.403 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:50.403 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:50.403 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:47.136514+0000", 2024-09-13T11:25:50.404 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.5609999999999999e-06 2024-09-13T11:25:50.404 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:50.404 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:50.404 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:50.404 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:47.347445+0000", 2024-09-13T11:25:50.404 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.210931324 2024-09-13T11:25:50.405 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:50.405 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:50.405 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:50.405 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:47.347450+0000", 2024-09-13T11:25:50.405 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.3819999999999997e-06 2024-09-13T11:25:50.405 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:50.405 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:50.405 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:50.405 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:47.347491+0000", 2024-09-13T11:25:50.406 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.1025000000000001e-05 2024-09-13T11:25:50.406 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:50.406 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:50.406 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:25:50.406 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:47.347496+0000", 2024-09-13T11:25:50.406 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.463e-06 2024-09-13T11:25:50.406 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:50.406 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:50.406 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:50.406 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:47.347610+0000", 2024-09-13T11:25:50.406 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00011414200000000001 2024-09-13T11:25:50.406 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:50.406 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:50.407 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:50.407 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:50.407 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:50.407 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:138294 3.9 3:90012dab:::benchmark_data_smithi177_31116_object138293:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e140)", 2024-09-13T11:25:50.407 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:24:54.250266+0000", 2024-09-13T11:25:50.407 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 56.061486954999999, 2024-09-13T11:25:50.407 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.33165556200000001, 2024-09-13T11:25:50.407 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:50.407 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:50.407 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:50.407 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:25:50.407 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:25:50.407 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:50.408 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 138294 2024-09-13T11:25:50.408 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:50.408 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:50.408 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:50.408 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:50.408 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:54.250266+0000", 2024-09-13T11:25:50.408 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:50.408 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:50.408 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:50.408 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:50.408 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:54.250266+0000", 2024-09-13T11:25:50.408 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:50.408 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:50.409 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:50.409 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:50.409 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:54.250269+0000", 2024-09-13T11:25:50.409 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.3159999999999999e-06 2024-09-13T11:25:50.409 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:50.409 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:50.409 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:50.409 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:54.250362+0000", 2024-09-13T11:25:50.409 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.3051000000000001e-05 2024-09-13T11:25:50.409 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:50.409 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:50.409 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:50.409 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:54.250374+0000", 2024-09-13T11:25:50.409 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2327000000000001e-05 2024-09-13T11:25:50.409 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:50.410 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:50.410 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:50.410 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:54.250382+0000", 2024-09-13T11:25:50.410 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.7249999999999997e-06 2024-09-13T11:25:50.410 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:50.410 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:50.410 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:50.410 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:54.250459+0000", 2024-09-13T11:25:50.410 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.7218999999999994e-05 2024-09-13T11:25:50.410 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:50.410 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:50.410 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:25:50.410 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:54.250611+0000", 2024-09-13T11:25:50.411 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00015241000000000001 2024-09-13T11:25:50.411 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:50.411 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:50.411 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 2", 2024-09-13T11:25:50.411 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:54.250777+0000", 2024-09-13T11:25:50.411 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00016575199999999999 2024-09-13T11:25:50.411 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:50.411 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:50.411 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:25:50.411 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:54.252776+0000", 2024-09-13T11:25:50.411 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0019987260000000001 2024-09-13T11:25:50.411 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:50.411 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:50.411 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:25:50.412 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:54.581758+0000", 2024-09-13T11:25:50.412 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.32898232799999999 2024-09-13T11:25:50.412 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:50.412 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:50.412 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:25:50.412 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:54.581854+0000", 2024-09-13T11:25:50.412 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.5468000000000001e-05 2024-09-13T11:25:50.412 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:50.412 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:50.412 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:50.412 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:54.581922+0000", 2024-09-13T11:25:50.412 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.8239999999999997e-05 2024-09-13T11:25:50.412 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:50.412 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:50.413 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:50.413 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:50.413 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:50.413 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop_reply(client.4665.0:140790 3.20 e144/77 ondisk, result = 0)", 2024-09-13T11:25:50.413 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:24:58.385972+0000", 2024-09-13T11:25:50.413 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 51.925781387000001, 2024-09-13T11:25:50.413 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.51829568500000001, 2024-09-13T11:25:50.413 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:50.413 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:50.414 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:25:50.414 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:50.414 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:50.414 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:50.414 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.385972+0000", 2024-09-13T11:25:50.414 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:50.415 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:50.415 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:50.415 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:50.415 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.385972+0000", 2024-09-13T11:25:50.415 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:50.415 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:50.416 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:50.416 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:50.416 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.385974+0000", 2024-09-13T11:25:50.416 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.7400000000000001e-06 2024-09-13T11:25:50.416 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:50.417 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:50.417 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:50.417 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.385979+0000", 2024-09-13T11:25:50.417 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.091e-06 2024-09-13T11:25:50.417 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:50.417 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:50.418 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:50.418 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.904091+0000", 2024-09-13T11:25:50.418 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.51811194900000002 2024-09-13T11:25:50.418 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:50.418 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:50.419 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:50.419 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.904098+0000", 2024-09-13T11:25:50.419 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.0609999999999998e-06 2024-09-13T11:25:50.419 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:50.419 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:50.419 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:50.420 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.904156+0000", 2024-09-13T11:25:50.420 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.7902000000000002e-05 2024-09-13T11:25:50.420 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:50.420 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:50.420 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:25:50.421 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.904163+0000", 2024-09-13T11:25:50.421 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.1430000000000001e-06 2024-09-13T11:25:50.421 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:50.421 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:50.421 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:50.422 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.904267+0000", 2024-09-13T11:25:50.422 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000104799 2024-09-13T11:25:50.422 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:50.422 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:50.422 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:50.423 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:50.423 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:50.423 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:170721 3.9 3:93244b1b:::benchmark_data_smithi177_31116_object30418:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e162)", 2024-09-13T11:25:50.423 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:25:21.721042+0000", 2024-09-13T11:25:50.423 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 28.590711385999999, 2024-09-13T11:25:50.423 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.19807348799999999, 2024-09-13T11:25:50.424 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:50.424 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:50.424 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:50.424 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:25:50.424 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:25:50.425 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:50.425 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 170721 2024-09-13T11:25:50.425 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:50.425 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:50.425 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:50.426 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:50.426 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:21.721042+0000", 2024-09-13T11:25:50.426 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:50.426 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:50.426 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:50.427 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:50.427 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:21.721042+0000", 2024-09-13T11:25:50.427 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:50.427 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:50.427 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:50.428 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:50.428 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:21.721044+0000", 2024-09-13T11:25:50.428 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.781e-06 2024-09-13T11:25:50.428 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:50.428 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:50.429 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:50.429 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:21.721050+0000", 2024-09-13T11:25:50.429 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.1840000000000004e-06 2024-09-13T11:25:50.429 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:50.429 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:50.430 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:50.430 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:21.721065+0000", 2024-09-13T11:25:50.430 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.5073e-05 2024-09-13T11:25:50.430 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:50.430 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:50.430 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:50.431 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:21.721073+0000", 2024-09-13T11:25:50.431 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.6030000000000005e-06 2024-09-13T11:25:50.431 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:50.431 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:50.431 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:50.432 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:21.721140+0000", 2024-09-13T11:25:50.432 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.6979999999999999e-05 2024-09-13T11:25:50.432 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:50.432 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:50.432 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:25:50.433 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:21.721260+0000", 2024-09-13T11:25:50.433 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000119119 2024-09-13T11:25:50.433 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:50.433 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:50.433 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 2", 2024-09-13T11:25:50.433 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:21.721363+0000", 2024-09-13T11:25:50.434 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000103052 2024-09-13T11:25:50.434 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:50.434 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:50.434 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:25:50.434 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:21.723031+0000", 2024-09-13T11:25:50.435 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0016684650000000001 2024-09-13T11:25:50.435 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:50.435 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:50.435 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:25:50.435 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:21.919011+0000", 2024-09-13T11:25:50.435 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.19598026800000001 2024-09-13T11:25:50.436 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:50.436 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:50.436 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:25:50.436 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:21.919070+0000", 2024-09-13T11:25:50.436 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.9110000000000002e-05 2024-09-13T11:25:50.437 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:50.437 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:50.437 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:50.437 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:21.919115+0000", 2024-09-13T11:25:50.437 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.4852999999999999e-05 2024-09-13T11:25:50.437 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:50.438 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:50.438 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:50.438 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:50.438 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:50.438 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4665.0:191082 3.26 e175/140 3:67bd986b:::benchmark_data_smithi177_31116_object50779:head v 175'6183, mlcod=175'6180)", 2024-09-13T11:25:50.439 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:25:38.551017+0000", 2024-09-13T11:25:50.439 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 11.760735967, 2024-09-13T11:25:50.439 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.29063407899999999, 2024-09-13T11:25:50.439 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:50.439 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:50.440 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:50.440 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:50.440 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:50.440 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:50.440 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:38.551017+0000", 2024-09-13T11:25:50.441 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:50.441 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:50.441 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:50.441 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:50.441 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:38.551017+0000", 2024-09-13T11:25:50.441 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:50.442 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:50.442 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:50.442 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:50.442 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:38.551020+0000", 2024-09-13T11:25:50.442 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.9170000000000002e-06 2024-09-13T11:25:50.443 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:50.443 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:50.443 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:50.443 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:38.551033+0000", 2024-09-13T11:25:50.443 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2843000000000001e-05 2024-09-13T11:25:50.443 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:50.444 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:50.444 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:50.444 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:38.838514+0000", 2024-09-13T11:25:50.444 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.28748061600000002 2024-09-13T11:25:50.444 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:50.445 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:50.445 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:50.445 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:38.838532+0000", 2024-09-13T11:25:50.445 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.8012999999999998e-05 2024-09-13T11:25:50.445 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:50.446 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:50.446 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:50.446 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:38.838646+0000", 2024-09-13T11:25:50.446 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000114596 2024-09-13T11:25:50.446 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:50.446 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:50.447 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:25:50.447 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:38.838681+0000", 2024-09-13T11:25:50.447 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.5099999999999999e-05 2024-09-13T11:25:50.447 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:50.447 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:50.448 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:25:50.448 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:38.841593+0000", 2024-09-13T11:25:50.448 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0029117050000000001 2024-09-13T11:25:50.448 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:50.448 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:50.449 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:50.449 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:38.841651+0000", 2024-09-13T11:25:50.449 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.8288999999999997e-05 2024-09-13T11:25:50.449 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:50.449 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:50.450 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:50.450 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:50.450 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:50.450 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4665.0:191198 3.26 e175/140 3:64ff4057:::benchmark_data_smithi177_31116_object50895:head v 175'6185, mlcod=175'6182)", 2024-09-13T11:25:50.450 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:25:38.948739+0000", 2024-09-13T11:25:50.451 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 11.363014506000001, 2024-09-13T11:25:50.451 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.25772718100000003, 2024-09-13T11:25:50.451 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:50.451 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:50.451 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:50.452 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:50.452 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:50.452 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:50.452 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:38.948739+0000", 2024-09-13T11:25:50.452 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:50.453 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:50.453 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:50.453 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:50.453 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:38.948739+0000", 2024-09-13T11:25:50.453 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:50.454 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:50.454 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:50.454 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:50.454 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:38.948743+0000", 2024-09-13T11:25:50.454 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.0600000000000001e-06 2024-09-13T11:25:50.454 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:50.455 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:50.455 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:50.455 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:38.948760+0000", 2024-09-13T11:25:50.455 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.7540000000000001e-05 2024-09-13T11:25:50.455 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:50.456 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:50.456 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:50.456 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:39.204732+0000", 2024-09-13T11:25:50.456 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.25597220199999998 2024-09-13T11:25:50.456 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:50.457 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:50.457 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:50.457 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:39.204742+0000", 2024-09-13T11:25:50.457 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.6579999999999997e-06 2024-09-13T11:25:50.457 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:50.458 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:50.458 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:50.458 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:39.204834+0000", 2024-09-13T11:25:50.458 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.1565999999999997e-05 2024-09-13T11:25:50.458 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:50.458 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:50.459 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:25:50.459 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:39.204854+0000", 2024-09-13T11:25:50.459 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.0809e-05 2024-09-13T11:25:50.459 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:50.459 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:50.460 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:25:50.460 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:39.206423+0000", 2024-09-13T11:25:50.460 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0015685219999999999 2024-09-13T11:25:50.460 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:50.460 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:50.461 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:50.461 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:39.206466+0000", 2024-09-13T11:25:50.461 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.2824000000000002e-05 2024-09-13T11:25:50.461 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:50.461 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:50.461 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:50.461 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:50.461 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:50.461 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:196438 3.20 3:0573f1a4:::benchmark_data_smithi177_31116_object56135:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e178)", 2024-09-13T11:25:50.461 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:25:43.521712+0000", 2024-09-13T11:25:50.461 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 6.7900414070000004, 2024-09-13T11:25:50.461 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.23827029799999999, 2024-09-13T11:25:50.462 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:50.462 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:50.462 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:50.462 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:25:50.462 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:25:50.462 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:50.462 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 196438 2024-09-13T11:25:50.462 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:50.462 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:50.462 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:50.462 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:50.462 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:43.521712+0000", 2024-09-13T11:25:50.462 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:50.462 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:50.463 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:50.463 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:50.463 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:43.521712+0000", 2024-09-13T11:25:50.463 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:50.463 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:50.463 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:50.463 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:50.463 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:43.521714+0000", 2024-09-13T11:25:50.463 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.6970000000000002e-06 2024-09-13T11:25:50.463 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:50.463 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:50.463 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:50.463 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:43.521723+0000", 2024-09-13T11:25:50.464 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.1319999999999994e-06 2024-09-13T11:25:50.464 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:50.464 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:50.464 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:50.464 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:43.521743+0000", 2024-09-13T11:25:50.464 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.0446e-05 2024-09-13T11:25:50.464 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:50.464 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:50.464 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:50.464 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:43.521754+0000", 2024-09-13T11:25:50.464 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.1314e-05 2024-09-13T11:25:50.464 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:50.464 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:50.465 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:50.465 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:43.521839+0000", 2024-09-13T11:25:50.465 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.4951999999999996e-05 2024-09-13T11:25:50.465 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:50.465 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:50.465 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:25:50.465 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:43.522010+0000", 2024-09-13T11:25:50.465 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000171053 2024-09-13T11:25:50.465 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:50.465 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:50.465 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 0", 2024-09-13T11:25:50.465 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:43.522160+0000", 2024-09-13T11:25:50.465 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000149987 2024-09-13T11:25:50.465 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:50.465 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:50.466 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:25:50.466 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:43.524717+0000", 2024-09-13T11:25:50.466 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0025565649999999998 2024-09-13T11:25:50.466 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:50.466 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:50.466 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:25:50.466 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:43.759800+0000", 2024-09-13T11:25:50.466 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.23508320699999999 2024-09-13T11:25:50.466 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:50.466 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:50.466 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:25:50.466 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:43.759917+0000", 2024-09-13T11:25:50.466 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000116605 2024-09-13T11:25:50.467 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:50.467 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:50.467 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:50.467 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:43.759982+0000", 2024-09-13T11:25:50.467 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.5339999999999994e-05 2024-09-13T11:25:50.467 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:50.467 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:50.467 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:50.467 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:50.467 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:50.467 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-13T11:25:50.468 INFO:tasks.ceph.osd.7.smithi177.stderr:2024-09-13T11:25:50.398+0000 7f896ff4a640 -1 osd.7 182 handle_osd_map injecting map crc failure 2024-09-13T11:25:50.469 INFO:tasks.ceph.osd.13.smithi092.stderr:2024-09-13T11:25:50.465+0000 7f3e5afea640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26350) UID: 0 2024-09-13T11:25:50.470 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-13T11:25:50.470 INFO:teuthology.orchestra.run.smithi001.stdout:osd.0: {} 2024-09-13T11:25:50.470 INFO:teuthology.orchestra.run.smithi001.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T11:25:50.471 INFO:teuthology.orchestra.run.smithi001.stdout:osd.1: {} 2024-09-13T11:25:50.471 INFO:teuthology.orchestra.run.smithi001.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T11:25:50.471 INFO:teuthology.orchestra.run.smithi001.stderr:noscrub is set 2024-09-13T11:25:50.471 INFO:teuthology.orchestra.run.smithi001.stdout:osd.2: {} 2024-09-13T11:25:50.472 INFO:teuthology.orchestra.run.smithi001.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T11:25:50.473 INFO:teuthology.orchestra.run.smithi001.stdout:osd.3: {} 2024-09-13T11:25:50.473 INFO:teuthology.orchestra.run.smithi001.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T11:25:50.473 INFO:teuthology.orchestra.run.smithi001.stdout:osd.4: {} 2024-09-13T11:25:50.473 INFO:teuthology.orchestra.run.smithi001.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T11:25:50.479 INFO:teuthology.orchestra.run.smithi001.stdout:osd.5: {} 2024-09-13T11:25:50.479 INFO:teuthology.orchestra.run.smithi001.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T11:25:50.505 INFO:teuthology.orchestra.run.smithi001.stdout:osd.6: {} 2024-09-13T11:25:50.506 INFO:teuthology.orchestra.run.smithi001.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T11:25:50.528 INFO:teuthology.orchestra.run.smithi001.stdout:osd.7: {} 2024-09-13T11:25:50.529 INFO:teuthology.orchestra.run.smithi001.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T11:25:50.556 INFO:teuthology.orchestra.run.smithi001.stdout:osd.8: {} 2024-09-13T11:25:50.556 INFO:teuthology.orchestra.run.smithi001.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-13T11:25:50.569 INFO:tasks.ceph.osd.3.smithi177.stderr:2024-09-13T11:25:50.566+0000 7f10adba4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26574) UID: 0 2024-09-13T11:25:50.582 INFO:teuthology.orchestra.run.smithi001.stdout:osd.9: {} 2024-09-13T11:25:50.583 INFO:teuthology.orchestra.run.smithi001.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-13T11:25:50.608 INFO:teuthology.orchestra.run.smithi001.stdout:osd.10: {} 2024-09-13T11:25:50.608 INFO:teuthology.orchestra.run.smithi001.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-13T11:25:50.629 INFO:teuthology.orchestra.run.smithi092.stdout:ERROR: (22) Invalid argument 2024-09-13T11:25:50.629 INFO:teuthology.orchestra.run.smithi092.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:25:50.634 INFO:teuthology.orchestra.run.smithi001.stdout:osd.11: {} 2024-09-13T11:25:50.634 INFO:teuthology.orchestra.run.smithi001.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-13T11:25:50.648 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-13T11:25:50.659 INFO:teuthology.orchestra.run.smithi001.stdout:osd.12: {} 2024-09-13T11:25:50.659 INFO:teuthology.orchestra.run.smithi001.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-13T11:25:50.684 INFO:teuthology.orchestra.run.smithi001.stdout:osd.13: {} 2024-09-13T11:25:50.684 INFO:teuthology.orchestra.run.smithi001.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-13T11:25:50.697 INFO:tasks.ceph.osd.1.smithi092.stderr:2024-09-13T11:25:50.697+0000 7fe7789c5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33718) UID: 0 2024-09-13T11:25:50.708 INFO:teuthology.orchestra.run.smithi001.stdout:osd.14: {} 2024-09-13T11:25:50.709 INFO:teuthology.orchestra.run.smithi001.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-13T11:25:50.732 INFO:teuthology.orchestra.run.smithi001.stdout:osd.15: {} 2024-09-13T11:25:50.732 INFO:teuthology.orchestra.run.smithi001.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-13T11:25:50.771 INFO:tasks.ceph.osd.11.smithi177.stderr:2024-09-13T11:25:50.770+0000 7f24792c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26580) UID: 0 2024-09-13T11:25:50.784 INFO:teuthology.orchestra.run.smithi092.stdout:ERROR: (22) Invalid argument 2024-09-13T11:25:50.784 INFO:teuthology.orchestra.run.smithi092.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:25:50.801 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-13T11:25:50.872 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-13T11:25:50.871+0000 7f6a68775640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26307) UID: 0 2024-09-13T11:25:50.968 INFO:teuthology.orchestra.run.smithi092.stdout:ERROR: (22) Invalid argument 2024-09-13T11:25:50.968 INFO:teuthology.orchestra.run.smithi092.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:25:50.973 INFO:tasks.ceph.osd.13.smithi092.stderr:2024-09-13T11:25:50.973+0000 7f3e5afea640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26350) UID: 0 2024-09-13T11:25:50.984 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-09-13T11:25:51.074 INFO:tasks.ceph.osd.9.smithi092.stderr:2024-09-13T11:25:51.073+0000 7ff5ab3d0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26348) UID: 0 2024-09-13T11:25:51.151 INFO:teuthology.orchestra.run.smithi092.stdout:ERROR: (22) Invalid argument 2024-09-13T11:25:51.151 INFO:teuthology.orchestra.run.smithi092.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:25:51.166 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-09-13T11:25:51.175 INFO:tasks.ceph.osd.11.smithi177.stderr:2024-09-13T11:25:51.174+0000 7f24792c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26580) UID: 0 2024-09-13T11:25:51.275 INFO:tasks.ceph.osd.4.smithi001.stderr:2024-09-13T11:25:51.275+0000 7ff8e0218640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26310) UID: 0 2024-09-13T11:25:51.353 INFO:teuthology.orchestra.run.smithi092.stdout:ERROR: (22) Invalid argument 2024-09-13T11:25:51.353 INFO:teuthology.orchestra.run.smithi092.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:25:51.370 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-09-13T11:25:51.376 INFO:tasks.ceph.osd.8.smithi001.stderr:2024-09-13T11:25:51.375+0000 7fb9b48ae640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26312) UID: 0 2024-09-13T11:25:51.477 INFO:tasks.ceph.osd.12.smithi001.stderr:2024-09-13T11:25:51.475+0000 7f892c6e6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26314) UID: 0 2024-09-13T11:25:51.515 INFO:teuthology.orchestra.run.smithi092.stdout:ERROR: (22) Invalid argument 2024-09-13T11:25:51.515 INFO:teuthology.orchestra.run.smithi092.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:25:51.532 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_ops_in_flight 2024-09-13T11:25:51.578 INFO:tasks.ceph.osd.15.smithi177.stderr:2024-09-13T11:25:51.574+0000 7f9d6ce77640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26578) UID: 0 2024-09-13T11:25:51.667 INFO:teuthology.orchestra.run.smithi092.stdout:ERROR: (22) Invalid argument 2024-09-13T11:25:51.667 INFO:teuthology.orchestra.run.smithi092.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:25:51.679 INFO:tasks.ceph.osd.13.smithi092.stderr:2024-09-13T11:25:51.677+0000 7f3e5afea640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26350) UID: 0 2024-09-13T11:25:51.683 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_blocked_ops 2024-09-13T11:25:51.780 INFO:tasks.ceph.osd.15.smithi177.stderr:2024-09-13T11:25:51.778+0000 7f9d6ce77640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26578) UID: 0 2024-09-13T11:25:51.871 INFO:teuthology.orchestra.run.smithi092.stdout:ERROR: (22) Invalid argument 2024-09-13T11:25:51.960 INFO:teuthology.orchestra.run.smithi092.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:25:51.961 INFO:tasks.ceph.osd.10.smithi162.stderr:2024-09-13T11:25:51.940+0000 7fb593421640 -1 osd.10 183 handle_osd_map injecting map crc failure 2024-09-13T11:25:51.962 INFO:tasks.ceph.osd.4.smithi001.stderr:2024-09-13T11:25:51.934+0000 7ff8d2c41640 -1 osd.4 183 handle_osd_map injecting map crc failure 2024-09-13T11:25:51.962 INFO:tasks.ceph.osd.4.smithi001.stderr:2024-09-13T11:25:51.958+0000 7ff8e0218640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26310) UID: 0 2024-09-13T11:25:51.963 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_historic_ops 2024-09-13T11:25:52.062 INFO:tasks.ceph.osd.11.smithi177.stderr:2024-09-13T11:25:52.058+0000 7f24792c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26580) UID: 0 2024-09-13T11:25:52.093 INFO:teuthology.orchestra.run.smithi092.stdout:ERROR: (22) Invalid argument 2024-09-13T11:25:52.093 INFO:teuthology.orchestra.run.smithi092.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:25:52.113 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-13T11:25:52.164 INFO:tasks.ceph.osd.7.smithi177.stderr:2024-09-13T11:25:52.162+0000 7f897b51d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26575) UID: 0 2024-09-13T11:25:52.254 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-13T11:25:52.254 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:25:52.265 INFO:tasks.ceph.osd.10.smithi162.stderr:2024-09-13T11:25:52.260+0000 7fb59c9f0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26317) UID: 0 2024-09-13T11:25:52.271 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-13T11:25:52.282 INFO:tasks.ceph.osd.14.smithi162.stderr:2024-09-13T11:25:52.280+0000 7f3dea330640 -1 osd.14 183 handle_osd_map injecting map crc failure 2024-09-13T11:25:52.363 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 15] out_osds: [14] dead_osds: [] live_osds: [0, 4, 8, 12, 5, 9, 13, 2, 6, 10, 3, 7, 11, 15, 1, 14] 2024-09-13T11:25:52.364 INFO:tasks.thrashosds.thrasher:choose_action: min_in 2 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-13T11:25:52.364 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-09-13T11:25:52.364 INFO:tasks.thrashosds.thrasher:Setting osd 0 primary_affinity to 1.000000 2024-09-13T11:25:52.364 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 0 1 2024-09-13T11:25:52.365 INFO:tasks.ceph.osd.3.smithi177.stderr:2024-09-13T11:25:52.362+0000 7f10adba4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26574) UID: 0 2024-09-13T11:25:52.418 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-13T11:25:52.418 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:25:52.435 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-13T11:25:52.474 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-13T11:25:52.485 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-09-13T11:25:52.483+0000 7f97befc9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26315) UID: 0 2024-09-13T11:25:52.562 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-13T11:25:52.563 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:25:52.567 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-09-13T11:25:52.563+0000 7f97befc9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26315) UID: 0 2024-09-13T11:25:52.579 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-13T11:25:52.668 INFO:tasks.ceph.osd.15.smithi177.stderr:2024-09-13T11:25:52.666+0000 7f9d6ce77640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26578) UID: 0 2024-09-13T11:25:52.753 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T11:25:52.766 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-13T11:25:52.767 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:25:52.769 INFO:tasks.ceph.osd.7.smithi177.stderr:2024-09-13T11:25:52.766+0000 7f897b51d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26575) UID: 0 2024-09-13T11:25:52.782 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-13T11:25:52.870 INFO:tasks.ceph.osd.2.smithi162.stderr:2024-09-13T11:25:52.867+0000 7fe92b35a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26313) UID: 0 2024-09-13T11:25:52.927 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-13T11:25:52.927 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:25:52.939 INFO:teuthology.orchestra.run.smithi001.stderr:nodeep-scrub is set 2024-09-13T11:25:52.940 INFO:teuthology.orchestra.run.smithi001.stdout:osd.0: {} 2024-09-13T11:25:52.940 INFO:teuthology.orchestra.run.smithi001.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T11:25:52.943 INFO:teuthology.orchestra.run.smithi001.stderr:set osd.0 primary-affinity to 1 (8655362) 2024-09-13T11:25:52.945 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-13T11:25:52.947 INFO:tasks.ceph.osd.11.smithi177.stderr:2024-09-13T11:25:52.946+0000 7f2469cec640 -1 osd.11 184 handle_osd_map injecting map crc failure 2024-09-13T11:25:52.971 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-13T11:25:52.966+0000 7f6a68775640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26307) UID: 0 2024-09-13T11:25:52.983 INFO:teuthology.orchestra.run.smithi001.stdout:osd.1: {} 2024-09-13T11:25:52.983 INFO:teuthology.orchestra.run.smithi001.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T11:25:53.012 INFO:teuthology.orchestra.run.smithi001.stdout:osd.2: {} 2024-09-13T11:25:53.012 INFO:teuthology.orchestra.run.smithi001.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T11:25:53.040 INFO:teuthology.orchestra.run.smithi001.stdout:osd.3: {} 2024-09-13T11:25:53.040 INFO:teuthology.orchestra.run.smithi001.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T11:25:53.066 INFO:teuthology.orchestra.run.smithi001.stdout:osd.4: {} 2024-09-13T11:25:53.066 INFO:teuthology.orchestra.run.smithi001.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T11:25:53.071 INFO:tasks.ceph.osd.1.smithi092.stderr:2024-09-13T11:25:53.068+0000 7fe7789c5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33718) UID: 0 2024-09-13T11:25:53.097 INFO:teuthology.orchestra.run.smithi001.stdout:osd.5: {} 2024-09-13T11:25:53.097 INFO:teuthology.orchestra.run.smithi001.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T11:25:53.127 INFO:teuthology.orchestra.run.smithi001.stdout:osd.6: {} 2024-09-13T11:25:53.128 INFO:teuthology.orchestra.run.smithi001.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T11:25:53.138 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-13T11:25:53.138 INFO:teuthology.orchestra.run.smithi162.stdout: "size": 20, 2024-09-13T11:25:53.138 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 600, 2024-09-13T11:25:53.138 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [ 2024-09-13T11:25:53.138 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.138 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4257.0:80 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 262144~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-13T11:25:53.139 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:22:12.516065+0000", 2024-09-13T11:25:53.139 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 220.620275351, 2024-09-13T11:25:53.139 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.46461997999999999, 2024-09-13T11:25:53.139 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:25:53.139 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:25:53.139 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:53.139 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:25:53.139 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4257", 2024-09-13T11:25:53.139 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/1488131527", 2024-09-13T11:25:53.139 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 80 2024-09-13T11:25:53.139 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.139 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:25:53.139 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.140 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:25:53.140 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.516065+0000", 2024-09-13T11:25:53.140 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:53.140 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.140 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.140 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:25:53.140 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.516065+0000", 2024-09-13T11:25:53.140 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:53.140 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.140 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.140 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:25:53.140 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.516066+0000", 2024-09-13T11:25:53.140 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0869999999999999e-06 2024-09-13T11:25:53.140 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.141 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.141 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:25:53.141 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.516112+0000", 2024-09-13T11:25:53.141 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.5503000000000001e-05 2024-09-13T11:25:53.141 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.141 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.141 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:25:53.141 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.516128+0000", 2024-09-13T11:25:53.141 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5891000000000001e-05 2024-09-13T11:25:53.141 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.141 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.141 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:25:53.142 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.516138+0000", 2024-09-13T11:25:53.142 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.8120000000000002e-06 2024-09-13T11:25:53.142 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.142 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.142 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:25:53.142 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.520398+0000", 2024-09-13T11:25:53.142 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0042605450000000001 2024-09-13T11:25:53.142 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.142 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.142 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:25:53.142 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.520526+0000", 2024-09-13T11:25:53.142 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000128355 2024-09-13T11:25:53.142 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.142 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.143 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 0", 2024-09-13T11:25:53.143 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.520730+0000", 2024-09-13T11:25:53.143 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00020337799999999999 2024-09-13T11:25:53.143 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.143 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.143 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:25:53.143 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.522417+0000", 2024-09-13T11:25:53.143 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001687184 2024-09-13T11:25:53.143 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.143 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.143 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:25:53.143 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.980408+0000", 2024-09-13T11:25:53.143 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.45799068399999998 2024-09-13T11:25:53.144 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.144 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.144 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:25:53.144 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.980565+0000", 2024-09-13T11:25:53.144 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00015760899999999999 2024-09-13T11:25:53.144 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.144 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.144 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:25:53.144 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.980685+0000", 2024-09-13T11:25:53.144 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000119932 2024-09-13T11:25:53.144 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:53.144 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:25:53.144 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:53.145 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.145 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.145 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop_reply(client.4257.0:80 1.0 e24/23 ondisk, result = 0)", 2024-09-13T11:25:53.145 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:22:12.522938+0000", 2024-09-13T11:25:53.145 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 220.613402318, 2024-09-13T11:25:53.145 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.45776420600000001, 2024-09-13T11:25:53.145 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:25:53.145 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:25:53.145 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:25:53.145 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:25:53.145 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.146 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:25:53.146 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.522938+0000", 2024-09-13T11:25:53.146 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:53.146 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.146 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.146 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:25:53.146 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.522938+0000", 2024-09-13T11:25:53.146 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:53.146 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.146 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.146 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:25:53.146 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.522941+0000", 2024-09-13T11:25:53.146 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.0290000000000001e-06 2024-09-13T11:25:53.146 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.147 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.147 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:25:53.147 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.522950+0000", 2024-09-13T11:25:53.147 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.7499999999999992e-06 2024-09-13T11:25:53.147 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.147 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.147 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:25:53.147 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.980215+0000", 2024-09-13T11:25:53.147 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.45726523800000002 2024-09-13T11:25:53.147 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.147 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.147 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:25:53.147 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.980229+0000", 2024-09-13T11:25:53.148 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3549e-05 2024-09-13T11:25:53.148 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.148 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.148 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:25:53.148 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.980365+0000", 2024-09-13T11:25:53.148 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00013608700000000001 2024-09-13T11:25:53.148 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.148 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.148 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:25:53.148 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.980388+0000", 2024-09-13T11:25:53.148 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.3112000000000002e-05 2024-09-13T11:25:53.148 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.148 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.148 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:25:53.149 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.980702+0000", 2024-09-13T11:25:53.149 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00031444099999999999 2024-09-13T11:25:53.149 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:53.149 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:25:53.149 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:53.149 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.150 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.150 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:25416 3.8 3:1a1fbdd1:::benchmark_data_smithi177_31116_object25415:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e56)", 2024-09-13T11:25:53.150 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:04.512188+0000", 2024-09-13T11:25:53.150 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 168.62415235099999, 2024-09-13T11:25:53.150 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.210208697, 2024-09-13T11:25:53.150 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:25:53.150 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:25:53.151 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:53.151 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:25:53.151 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:25:53.151 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:53.151 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 25416 2024-09-13T11:25:53.151 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.151 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:25:53.151 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.151 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:25:53.151 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.512188+0000", 2024-09-13T11:25:53.151 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:53.151 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.151 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.151 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:25:53.152 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.512188+0000", 2024-09-13T11:25:53.152 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:53.152 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.152 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.152 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:25:53.152 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.512192+0000", 2024-09-13T11:25:53.152 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.2660000000000003e-06 2024-09-13T11:25:53.152 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.152 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.152 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:25:53.152 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.512247+0000", 2024-09-13T11:25:53.152 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.4889000000000003e-05 2024-09-13T11:25:53.152 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.153 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.153 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:25:53.153 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.512279+0000", 2024-09-13T11:25:53.153 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.1146000000000003e-05 2024-09-13T11:25:53.153 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.153 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.153 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:25:53.153 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.512297+0000", 2024-09-13T11:25:53.153 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.825e-05 2024-09-13T11:25:53.153 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.153 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.153 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:25:53.153 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.512377+0000", 2024-09-13T11:25:53.153 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.9875000000000004e-05 2024-09-13T11:25:53.154 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.154 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.154 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:25:53.154 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.512570+0000", 2024-09-13T11:25:53.154 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00019312100000000001 2024-09-13T11:25:53.154 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.154 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.154 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 1", 2024-09-13T11:25:53.154 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.512798+0000", 2024-09-13T11:25:53.154 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00022847899999999999 2024-09-13T11:25:53.154 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.154 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.154 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:25:53.155 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.514975+0000", 2024-09-13T11:25:53.155 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0021769979999999999 2024-09-13T11:25:53.155 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.155 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.155 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:25:53.155 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.722207+0000", 2024-09-13T11:25:53.155 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.20723217099999999 2024-09-13T11:25:53.155 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.155 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.155 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:25:53.155 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.722314+0000", 2024-09-13T11:25:53.155 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00010648400000000001 2024-09-13T11:25:53.156 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.156 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.156 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:25:53.156 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.722397+0000", 2024-09-13T11:25:53.156 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.3017999999999997e-05 2024-09-13T11:25:53.156 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:53.157 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:25:53.157 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:53.157 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.157 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.157 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop_reply(client.4665.0:25416 3.8 e56/40 ondisk, result = 0)", 2024-09-13T11:25:53.157 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:04.515635+0000", 2024-09-13T11:25:53.158 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 168.62070569900001, 2024-09-13T11:25:53.158 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.20677600500000001, 2024-09-13T11:25:53.158 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:25:53.158 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:25:53.158 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:25:53.158 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:25:53.159 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.159 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:25:53.159 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.515635+0000", 2024-09-13T11:25:53.159 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:53.159 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.159 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.159 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:25:53.160 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.515635+0000", 2024-09-13T11:25:53.160 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:53.160 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.160 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.160 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:25:53.160 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.515639+0000", 2024-09-13T11:25:53.161 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.619e-06 2024-09-13T11:25:53.161 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.161 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.162 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:25:53.162 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.515650+0000", 2024-09-13T11:25:53.162 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.111e-05 2024-09-13T11:25:53.162 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.162 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.162 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:25:53.163 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.722004+0000", 2024-09-13T11:25:53.163 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.206354805 2024-09-13T11:25:53.163 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.163 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.163 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:25:53.163 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.722017+0000", 2024-09-13T11:25:53.164 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2840000000000001e-05 2024-09-13T11:25:53.164 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.164 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.164 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:25:53.164 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.722186+0000", 2024-09-13T11:25:53.164 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00016898299999999999 2024-09-13T11:25:53.165 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.165 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.165 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:25:53.165 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.722199+0000", 2024-09-13T11:25:53.165 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.289e-05 2024-09-13T11:25:53.165 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.166 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.166 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:25:53.166 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.722411+0000", 2024-09-13T11:25:53.166 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00021175799999999999 2024-09-13T11:25:53.166 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:53.167 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:25:53.167 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:53.167 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.167 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.167 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:30833 3.8 3:1c203f8d:::benchmark_data_smithi177_31116_object30832:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e59)", 2024-09-13T11:25:53.167 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:09.506656+0000", 2024-09-13T11:25:53.168 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 163.629684801, 2024-09-13T11:25:53.168 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.224862282, 2024-09-13T11:25:53.168 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:25:53.168 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:25:53.168 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:53.168 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:25:53.169 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:25:53.169 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:53.169 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 30833 2024-09-13T11:25:53.169 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.169 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:25:53.169 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.170 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:25:53.170 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.506656+0000", 2024-09-13T11:25:53.170 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:53.170 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.170 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.170 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:25:53.171 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.506656+0000", 2024-09-13T11:25:53.171 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:53.171 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.171 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.171 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:25:53.171 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.506660+0000", 2024-09-13T11:25:53.171 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.6e-06 2024-09-13T11:25:53.172 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.172 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.172 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:25:53.172 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.506717+0000", 2024-09-13T11:25:53.172 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.6131999999999997e-05 2024-09-13T11:25:53.173 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.173 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.173 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:25:53.173 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.506751+0000", 2024-09-13T11:25:53.173 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.4746000000000002e-05 2024-09-13T11:25:53.173 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.174 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.174 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:25:53.174 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.506774+0000", 2024-09-13T11:25:53.174 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.2636000000000001e-05 2024-09-13T11:25:53.175 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.175 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.175 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:25:53.175 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.506878+0000", 2024-09-13T11:25:53.175 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000104057 2024-09-13T11:25:53.175 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.176 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.176 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:25:53.176 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.507052+0000", 2024-09-13T11:25:53.176 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00017402800000000001 2024-09-13T11:25:53.176 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.176 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.177 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 1", 2024-09-13T11:25:53.177 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.507217+0000", 2024-09-13T11:25:53.177 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00016516 2024-09-13T11:25:53.177 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.177 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.177 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:25:53.178 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.508992+0000", 2024-09-13T11:25:53.178 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0017743990000000001 2024-09-13T11:25:53.178 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.178 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.178 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:25:53.178 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.731429+0000", 2024-09-13T11:25:53.179 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.22243718000000001 2024-09-13T11:25:53.179 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.179 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.179 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:25:53.179 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.731479+0000", 2024-09-13T11:25:53.179 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.0146999999999999e-05 2024-09-13T11:25:53.180 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.180 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.180 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:25:53.180 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.731518+0000", 2024-09-13T11:25:53.180 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.9196999999999998e-05 2024-09-13T11:25:53.180 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:53.181 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:25:53.181 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:53.181 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.181 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.181 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop_reply(client.4665.0:30833 3.8 e59/40 ondisk, result = 0)", 2024-09-13T11:25:53.181 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:09.508907+0000", 2024-09-13T11:25:53.182 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 163.62743370699999, 2024-09-13T11:25:53.182 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.222618862, 2024-09-13T11:25:53.182 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:25:53.182 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:25:53.182 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:25:53.182 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:25:53.183 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.183 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:25:53.183 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.508907+0000", 2024-09-13T11:25:53.183 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:53.183 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.183 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.184 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:25:53.184 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.508907+0000", 2024-09-13T11:25:53.184 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:53.184 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.184 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.184 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:25:53.185 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.508910+0000", 2024-09-13T11:25:53.185 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.993e-06 2024-09-13T11:25:53.185 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.185 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.185 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:25:53.186 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.508918+0000", 2024-09-13T11:25:53.186 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.7000000000000008e-06 2024-09-13T11:25:53.186 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.186 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.186 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:25:53.186 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.731341+0000", 2024-09-13T11:25:53.186 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.22242308699999999 2024-09-13T11:25:53.187 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.187 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.187 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:25:53.187 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.731350+0000", 2024-09-13T11:25:53.187 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.3179999999999996e-06 2024-09-13T11:25:53.188 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.188 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.188 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:25:53.188 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.731418+0000", 2024-09-13T11:25:53.188 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.7586e-05 2024-09-13T11:25:53.188 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.189 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.189 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:25:53.189 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.731424+0000", 2024-09-13T11:25:53.192 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.844e-06 2024-09-13T11:25:53.192 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.193 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.193 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:25:53.193 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.731526+0000", 2024-09-13T11:25:53.193 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000101334 2024-09-13T11:25:53.193 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:53.193 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:25:53.193 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:53.193 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.193 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.193 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPush(3.10 76/70 [PushOp(3:0e7d2700:::benchmark_data_smithi177_31116_object17406:head, version: 52'1075, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:0e7d2700:::benchmark_data_smithi177_31116_object17406:head@52'1075, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:25:53.193 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:32.206198+0000", 2024-09-13T11:25:53.194 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 140.93014299800001, 2024-09-13T11:25:53.194 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.25193395000000002, 2024-09-13T11:25:53.194 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:25:53.194 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:25:53.194 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:25:53.194 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:25:53.194 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.194 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:25:53.194 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.206198+0000", 2024-09-13T11:25:53.194 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:53.194 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.194 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.194 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:25:53.194 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.206198+0000", 2024-09-13T11:25:53.195 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:53.195 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.195 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.195 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:25:53.195 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.206199+0000", 2024-09-13T11:25:53.195 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.46e-06 2024-09-13T11:25:53.195 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.195 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.195 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:25:53.195 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.206250+0000", 2024-09-13T11:25:53.195 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.113e-05 2024-09-13T11:25:53.195 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.195 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.196 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:25:53.196 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.456042+0000", 2024-09-13T11:25:53.196 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.24979182999999999 2024-09-13T11:25:53.196 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.196 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.196 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:25:53.196 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.456084+0000", 2024-09-13T11:25:53.196 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.1814000000000003e-05 2024-09-13T11:25:53.196 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.196 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.196 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:25:53.196 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.456342+0000", 2024-09-13T11:25:53.196 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000258404 2024-09-13T11:25:53.197 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.197 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.197 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:25:53.197 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.456367+0000", 2024-09-13T11:25:53.197 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.5099000000000001e-05 2024-09-13T11:25:53.197 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.197 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.197 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:25:53.197 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.458132+0000", 2024-09-13T11:25:53.198 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0017642129999999999 2024-09-13T11:25:53.198 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:53.198 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:25:53.198 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:53.198 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.198 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.198 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPush(3.10 76/70 [PushOp(3:0eee8bcf:::benchmark_data_smithi177_31116_object17589:head, version: 53'1086, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:0eee8bcf:::benchmark_data_smithi177_31116_object17589:head@53'1086, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:25:53.198 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:32.585183+0000", 2024-09-13T11:25:53.198 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 140.551157104, 2024-09-13T11:25:53.198 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.16696202199999999, 2024-09-13T11:25:53.198 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:25:53.198 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:25:53.198 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:25:53.199 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:25:53.199 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.199 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:25:53.199 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.585183+0000", 2024-09-13T11:25:53.199 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:53.199 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.199 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.199 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:25:53.199 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.585183+0000", 2024-09-13T11:25:53.199 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:53.199 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.199 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.199 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:25:53.199 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.585187+0000", 2024-09-13T11:25:53.200 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.8480000000000003e-06 2024-09-13T11:25:53.200 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.200 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.200 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:25:53.200 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.585264+0000", 2024-09-13T11:25:53.200 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.6327000000000003e-05 2024-09-13T11:25:53.200 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.200 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.200 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:25:53.200 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.750459+0000", 2024-09-13T11:25:53.201 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.16519551900000001 2024-09-13T11:25:53.201 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.201 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.201 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:25:53.201 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.750480+0000", 2024-09-13T11:25:53.201 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.0984000000000001e-05 2024-09-13T11:25:53.202 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.202 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.202 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:25:53.202 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.750633+0000", 2024-09-13T11:25:53.202 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000153148 2024-09-13T11:25:53.202 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.202 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.202 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:25:53.202 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.750652+0000", 2024-09-13T11:25:53.202 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.8981e-05 2024-09-13T11:25:53.202 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.202 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.202 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:25:53.202 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.752145+0000", 2024-09-13T11:25:53.203 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001493215 2024-09-13T11:25:53.203 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:53.203 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:25:53.203 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:53.203 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.203 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.203 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:60417 3.10 3:092808b0:::benchmark_data_smithi177_31116_object60416:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e85)", 2024-09-13T11:25:53.203 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:42.173238+0000", 2024-09-13T11:25:53.203 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 130.96310209000001, 2024-09-13T11:25:53.203 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.38965994599999998, 2024-09-13T11:25:53.203 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:25:53.203 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:25:53.203 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:53.203 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:25:53.203 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:25:53.204 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:53.204 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 60417 2024-09-13T11:25:53.204 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.204 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:25:53.204 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.204 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:25:53.204 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.173238+0000", 2024-09-13T11:25:53.204 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:53.204 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.204 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.204 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:25:53.204 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.173238+0000", 2024-09-13T11:25:53.204 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:53.205 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.205 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.205 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:25:53.205 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.173241+0000", 2024-09-13T11:25:53.205 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.6819999999999999e-06 2024-09-13T11:25:53.205 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.205 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.205 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:25:53.205 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.173335+0000", 2024-09-13T11:25:53.205 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.3957999999999997e-05 2024-09-13T11:25:53.205 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.206 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.206 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:25:53.206 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.173358+0000", 2024-09-13T11:25:53.206 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.3023999999999998e-05 2024-09-13T11:25:53.206 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.207 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.207 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:25:53.207 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.173373+0000", 2024-09-13T11:25:53.207 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4867e-05 2024-09-13T11:25:53.207 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.207 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.208 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:25:53.208 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.173466+0000", 2024-09-13T11:25:53.208 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.2542999999999998e-05 2024-09-13T11:25:53.208 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.208 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.209 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:25:53.209 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.173754+0000", 2024-09-13T11:25:53.209 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000288657 2024-09-13T11:25:53.209 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.209 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.209 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 0", 2024-09-13T11:25:53.210 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.174066+0000", 2024-09-13T11:25:53.210 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00031170999999999998 2024-09-13T11:25:53.210 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.210 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.210 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:25:53.210 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.178245+0000", 2024-09-13T11:25:53.211 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00417864 2024-09-13T11:25:53.211 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.211 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.211 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:25:53.211 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.562617+0000", 2024-09-13T11:25:53.211 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.38437253199999999 2024-09-13T11:25:53.211 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.211 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.212 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:25:53.212 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.562782+0000", 2024-09-13T11:25:53.212 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00016499200000000001 2024-09-13T11:25:53.212 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.212 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.212 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:25:53.212 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.562898+0000", 2024-09-13T11:25:53.212 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000116341 2024-09-13T11:25:53.212 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:53.212 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:25:53.212 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:53.212 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.212 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.213 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:83193 3.8 3:10066d83:::benchmark_data_smithi177_31116_object83192:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106)", 2024-09-13T11:25:53.213 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:24:04.283923+0000", 2024-09-13T11:25:53.213 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 108.852417407, 2024-09-13T11:25:53.213 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.64775463499999997, 2024-09-13T11:25:53.213 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:25:53.213 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:25:53.213 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:53.213 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:25:53.213 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:25:53.213 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:53.213 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 83193 2024-09-13T11:25:53.213 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.213 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:25:53.213 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.214 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:25:53.214 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.283923+0000", 2024-09-13T11:25:53.214 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:53.214 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.214 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.214 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:25:53.214 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.283923+0000", 2024-09-13T11:25:53.214 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:53.214 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.214 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.214 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:25:53.214 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.283925+0000", 2024-09-13T11:25:53.214 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.9489999999999999e-06 2024-09-13T11:25:53.214 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.215 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.215 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:25:53.215 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.283960+0000", 2024-09-13T11:25:53.215 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.5063e-05 2024-09-13T11:25:53.215 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.215 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.215 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:25:53.215 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.283974+0000", 2024-09-13T11:25:53.215 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3546e-05 2024-09-13T11:25:53.215 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.215 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.215 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:25:53.215 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.283983+0000", 2024-09-13T11:25:53.215 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.2499999999999995e-06 2024-09-13T11:25:53.216 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.216 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.216 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:25:53.216 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.284061+0000", 2024-09-13T11:25:53.216 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.7946000000000002e-05 2024-09-13T11:25:53.216 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.216 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.216 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:25:53.216 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.284270+0000", 2024-09-13T11:25:53.216 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00020876799999999999 2024-09-13T11:25:53.216 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.216 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.216 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 1", 2024-09-13T11:25:53.216 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.284489+0000", 2024-09-13T11:25:53.217 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00021956800000000001 2024-09-13T11:25:53.217 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.217 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.217 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:25:53.217 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.286456+0000", 2024-09-13T11:25:53.217 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0019665809999999998 2024-09-13T11:25:53.217 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.217 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.217 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:25:53.217 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.931497+0000", 2024-09-13T11:25:53.217 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.64504147700000003 2024-09-13T11:25:53.217 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.218 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.218 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:25:53.218 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.931589+0000", 2024-09-13T11:25:53.218 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.2041999999999997e-05 2024-09-13T11:25:53.218 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.218 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.219 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:25:53.219 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.931678+0000", 2024-09-13T11:25:53.219 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.8444999999999998e-05 2024-09-13T11:25:53.219 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:53.219 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:25:53.219 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:53.219 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.219 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.219 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:115060 3.8 3:11200ba9:::benchmark_data_smithi177_31116_object115059:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e125)", 2024-09-13T11:25:53.219 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:24:33.602178+0000", 2024-09-13T11:25:53.219 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 79.534162413000004, 2024-09-13T11:25:53.219 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.23668186799999999, 2024-09-13T11:25:53.220 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:25:53.220 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:25:53.220 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:53.220 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:25:53.220 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:25:53.220 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:53.220 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 115060 2024-09-13T11:25:53.220 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.220 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:25:53.220 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.220 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:25:53.220 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:33.602178+0000", 2024-09-13T11:25:53.220 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:53.220 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.221 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.221 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:25:53.221 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:33.602178+0000", 2024-09-13T11:25:53.221 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:53.221 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.221 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.221 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:25:53.221 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:33.602181+0000", 2024-09-13T11:25:53.221 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.0369999999999999e-06 2024-09-13T11:25:53.221 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.221 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.221 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:25:53.221 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:33.602217+0000", 2024-09-13T11:25:53.221 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.5837999999999998e-05 2024-09-13T11:25:53.222 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.222 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.222 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:25:53.222 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:33.602235+0000", 2024-09-13T11:25:53.222 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.7571000000000002e-05 2024-09-13T11:25:53.222 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.222 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.222 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:25:53.222 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:33.602245+0000", 2024-09-13T11:25:53.222 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0360000000000001e-05 2024-09-13T11:25:53.222 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.222 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.222 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:25:53.222 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:33.602312+0000", 2024-09-13T11:25:53.223 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.6989000000000005e-05 2024-09-13T11:25:53.223 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.223 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.223 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:25:53.224 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:33.602490+0000", 2024-09-13T11:25:53.224 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000178464 2024-09-13T11:25:53.224 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.224 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.224 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 1", 2024-09-13T11:25:53.224 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:33.602670+0000", 2024-09-13T11:25:53.224 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00018002799999999999 2024-09-13T11:25:53.224 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.224 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.224 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:25:53.225 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:33.604382+0000", 2024-09-13T11:25:53.225 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001711314 2024-09-13T11:25:53.225 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.225 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.225 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:25:53.225 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:33.838738+0000", 2024-09-13T11:25:53.225 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.23435634699999999 2024-09-13T11:25:53.225 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.226 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.226 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:25:53.226 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:33.838815+0000", 2024-09-13T11:25:53.226 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.7381999999999999e-05 2024-09-13T11:25:53.226 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.227 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.227 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:25:53.227 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:33.838860+0000", 2024-09-13T11:25:53.227 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.4537999999999999e-05 2024-09-13T11:25:53.227 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:53.227 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:25:53.227 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:53.227 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.227 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.227 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop_reply(client.4665.0:115060 3.8 e125/117 ondisk, result = 0)", 2024-09-13T11:25:53.228 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:24:33.604404+0000", 2024-09-13T11:25:53.228 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 79.531936154999997, 2024-09-13T11:25:53.228 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.23446235400000001, 2024-09-13T11:25:53.228 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:25:53.228 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:25:53.228 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:25:53.228 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:25:53.228 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.228 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:25:53.229 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:33.604404+0000", 2024-09-13T11:25:53.229 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:53.229 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.229 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.229 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:25:53.229 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:33.604404+0000", 2024-09-13T11:25:53.229 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:53.229 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.229 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.230 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:25:53.230 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:33.604406+0000", 2024-09-13T11:25:53.230 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.418e-06 2024-09-13T11:25:53.230 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.230 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.230 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:25:53.230 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:33.604411+0000", 2024-09-13T11:25:53.230 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.3519999999999997e-06 2024-09-13T11:25:53.230 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.230 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.231 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:25:53.231 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:33.838578+0000", 2024-09-13T11:25:53.231 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.23416656199999999 2024-09-13T11:25:53.231 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.231 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.231 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:25:53.231 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:33.838586+0000", 2024-09-13T11:25:53.231 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.9920000000000007e-06 2024-09-13T11:25:53.231 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.231 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.232 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:25:53.232 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:33.838655+0000", 2024-09-13T11:25:53.232 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.9463999999999998e-05 2024-09-13T11:25:53.232 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.232 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.232 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:25:53.232 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:33.838731+0000", 2024-09-13T11:25:53.232 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.5487000000000004e-05 2024-09-13T11:25:53.232 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.233 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.233 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:25:53.233 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:33.838867+0000", 2024-09-13T11:25:53.233 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000136079 2024-09-13T11:25:53.233 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:53.233 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:25:53.233 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:53.233 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.233 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.233 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPush(3.6 129/122 [PushOp(3:6261f3b6:::benchmark_data_smithi177_31116_object64090:head, version: 90'3486, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:6261f3b6:::benchmark_data_smithi177_31116_object64090:head@90'3486, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:25:53.234 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:24:38.641487+0000", 2024-09-13T11:25:53.234 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 74.494853743999997, 2024-09-13T11:25:53.234 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.209889098, 2024-09-13T11:25:53.234 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:25:53.234 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:25:53.234 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:25:53.234 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:25:53.234 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.234 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:25:53.235 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:38.641487+0000", 2024-09-13T11:25:53.235 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:53.235 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.235 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.235 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:25:53.235 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:38.641487+0000", 2024-09-13T11:25:53.235 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:53.235 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.235 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.235 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:25:53.236 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:38.641489+0000", 2024-09-13T11:25:53.236 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.0980000000000001e-06 2024-09-13T11:25:53.236 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.236 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.236 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:25:53.236 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:38.641560+0000", 2024-09-13T11:25:53.236 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.1404999999999998e-05 2024-09-13T11:25:53.236 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.236 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.237 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:25:53.237 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:38.850498+0000", 2024-09-13T11:25:53.237 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.20893800500000001 2024-09-13T11:25:53.237 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.237 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.237 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:25:53.237 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:38.850510+0000", 2024-09-13T11:25:53.237 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1762e-05 2024-09-13T11:25:53.237 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.237 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.238 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:25:53.238 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:38.850610+0000", 2024-09-13T11:25:53.238 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00010011200000000001 2024-09-13T11:25:53.238 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.238 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.238 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:25:53.238 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:38.850628+0000", 2024-09-13T11:25:53.238 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.7291000000000001e-05 2024-09-13T11:25:53.238 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.238 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.239 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:25:53.239 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:38.851376+0000", 2024-09-13T11:25:53.239 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00074842500000000005 2024-09-13T11:25:53.239 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:53.239 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:25:53.239 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:53.239 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.239 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.239 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.4665.0:126919 3.6 e130/122 3:634ce08e:::benchmark_data_smithi177_31116_object126918:head v 130'5137, mlcod=130'5134)", 2024-09-13T11:25:53.240 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:24:43.780120+0000", 2024-09-13T11:25:53.240 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 69.356220852999996, 2024-09-13T11:25:53.240 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.298086295, 2024-09-13T11:25:53.240 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:25:53.240 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:25:53.240 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:53.240 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:25:53.240 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.240 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:25:53.241 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:43.780120+0000", 2024-09-13T11:25:53.241 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:53.241 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.241 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.241 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:25:53.241 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:43.780120+0000", 2024-09-13T11:25:53.241 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:53.241 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.241 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.242 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:25:53.242 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:43.780123+0000", 2024-09-13T11:25:53.242 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.79e-06 2024-09-13T11:25:53.242 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.242 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.242 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:25:53.242 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:43.780160+0000", 2024-09-13T11:25:53.242 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.7945999999999999e-05 2024-09-13T11:25:53.242 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.242 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.243 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:25:53.243 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:44.075853+0000", 2024-09-13T11:25:53.243 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.29569234999999999 2024-09-13T11:25:53.243 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.243 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.243 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:25:53.243 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:44.075861+0000", 2024-09-13T11:25:53.243 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.9510000000000006e-06 2024-09-13T11:25:53.243 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.244 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.244 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:25:53.244 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:44.075924+0000", 2024-09-13T11:25:53.244 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.2946000000000003e-05 2024-09-13T11:25:53.244 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.244 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.244 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:25:53.244 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:44.075942+0000", 2024-09-13T11:25:53.244 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.8108e-05 2024-09-13T11:25:53.245 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.245 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.245 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:25:53.245 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:44.078164+0000", 2024-09-13T11:25:53.245 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.002222375 2024-09-13T11:25:53.245 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.245 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.245 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:25:53.245 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:44.078206+0000", 2024-09-13T11:25:53.245 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.1829000000000002e-05 2024-09-13T11:25:53.246 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:53.246 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:25:53.246 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:53.246 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.246 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.246 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:137473 3.10 3:094c0efb:::benchmark_data_smithi177_31116_object137472:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e139)", 2024-09-13T11:25:53.246 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:24:53.523319+0000", 2024-09-13T11:25:53.246 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 59.613021814, 2024-09-13T11:25:53.246 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.18297369299999999, 2024-09-13T11:25:53.246 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:25:53.247 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:25:53.247 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:53.247 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:25:53.247 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:25:53.247 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:53.247 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 137473 2024-09-13T11:25:53.247 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.247 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:25:53.247 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.248 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:25:53.248 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:53.523319+0000", 2024-09-13T11:25:53.248 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:53.248 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.248 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.248 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:25:53.248 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:53.523319+0000", 2024-09-13T11:25:53.248 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:53.248 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.248 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.249 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:25:53.249 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:53.523322+0000", 2024-09-13T11:25:53.249 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.2119999999999999e-06 2024-09-13T11:25:53.249 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.249 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.249 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:25:53.249 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:53.523370+0000", 2024-09-13T11:25:53.249 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.7788999999999999e-05 2024-09-13T11:25:53.249 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.250 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.250 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:25:53.250 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:53.523395+0000", 2024-09-13T11:25:53.250 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.5202e-05 2024-09-13T11:25:53.250 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.250 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.250 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:25:53.250 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:53.523409+0000", 2024-09-13T11:25:53.250 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4263999999999999e-05 2024-09-13T11:25:53.250 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.251 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.251 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:25:53.251 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:53.523477+0000", 2024-09-13T11:25:53.251 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.7578999999999998e-05 2024-09-13T11:25:53.251 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.251 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.251 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:25:53.251 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:53.523638+0000", 2024-09-13T11:25:53.251 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00016078300000000001 2024-09-13T11:25:53.251 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.252 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.252 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 0", 2024-09-13T11:25:53.252 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:53.523800+0000", 2024-09-13T11:25:53.252 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000162564 2024-09-13T11:25:53.252 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.252 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.252 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:25:53.252 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:53.526960+0000", 2024-09-13T11:25:53.252 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0031593950000000002 2024-09-13T11:25:53.253 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.253 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.253 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:25:53.253 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:53.706180+0000", 2024-09-13T11:25:53.253 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.17922080600000001 2024-09-13T11:25:53.253 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.253 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.253 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:25:53.253 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:53.706237+0000", 2024-09-13T11:25:53.253 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.6845000000000002e-05 2024-09-13T11:25:53.254 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.254 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.254 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:25:53.254 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:53.706292+0000", 2024-09-13T11:25:53.254 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.5254000000000003e-05 2024-09-13T11:25:53.254 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:53.254 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:25:53.254 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:53.254 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.254 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.255 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop_reply(client.4665.0:137473 3.10 e139/84 ondisk, result = 0)", 2024-09-13T11:25:53.255 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:24:53.527076+0000", 2024-09-13T11:25:53.255 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 59.609264987000003, 2024-09-13T11:25:53.255 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.17922728099999999, 2024-09-13T11:25:53.255 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:25:53.255 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:25:53.255 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:25:53.255 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:25:53.255 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.255 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:25:53.256 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:53.527076+0000", 2024-09-13T11:25:53.256 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:53.256 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.256 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.256 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:25:53.256 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:53.527076+0000", 2024-09-13T11:25:53.256 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:53.256 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.256 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.257 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:25:53.257 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:53.527078+0000", 2024-09-13T11:25:53.257 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.9000000000000002e-06 2024-09-13T11:25:53.257 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.257 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.257 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:25:53.257 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:53.527088+0000", 2024-09-13T11:25:53.257 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.7240000000000004e-06 2024-09-13T11:25:53.257 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.257 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.258 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:25:53.258 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:53.706095+0000", 2024-09-13T11:25:53.258 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.17900633899999999 2024-09-13T11:25:53.258 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.258 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.258 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:25:53.258 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:53.706101+0000", 2024-09-13T11:25:53.258 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.7730000000000002e-06 2024-09-13T11:25:53.258 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.259 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.259 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:25:53.259 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:53.706168+0000", 2024-09-13T11:25:53.259 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.6225999999999999e-05 2024-09-13T11:25:53.259 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.259 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.259 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:25:53.259 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:53.706175+0000", 2024-09-13T11:25:53.259 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.5560000000000002e-06 2024-09-13T11:25:53.260 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.260 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.260 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:25:53.260 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:53.706303+0000", 2024-09-13T11:25:53.260 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000127763 2024-09-13T11:25:53.260 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:53.260 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:25:53.260 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:53.260 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.260 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.260 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:137780 3.8 3:1047f268:::benchmark_data_smithi177_31116_object137779:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e139)", 2024-09-13T11:25:53.260 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:24:53.795212+0000", 2024-09-13T11:25:53.261 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 59.341129068999997, 2024-09-13T11:25:53.261 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.21530981399999999, 2024-09-13T11:25:53.261 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:25:53.261 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:25:53.261 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:53.261 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:25:53.261 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:25:53.261 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:53.262 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 137780 2024-09-13T11:25:53.262 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.262 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:25:53.262 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.262 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:25:53.262 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:53.795212+0000", 2024-09-13T11:25:53.262 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:53.262 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.262 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.262 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:25:53.262 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:53.795212+0000", 2024-09-13T11:25:53.263 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:53.263 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.263 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.263 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:25:53.263 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:53.795214+0000", 2024-09-13T11:25:53.263 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.481e-06 2024-09-13T11:25:53.263 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.263 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.263 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:25:53.263 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:53.795247+0000", 2024-09-13T11:25:53.263 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.3105999999999998e-05 2024-09-13T11:25:53.263 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.263 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.264 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:25:53.264 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:53.795258+0000", 2024-09-13T11:25:53.264 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0872e-05 2024-09-13T11:25:53.264 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.264 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.264 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:25:53.264 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:53.795265+0000", 2024-09-13T11:25:53.264 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.5729999999999996e-06 2024-09-13T11:25:53.264 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.264 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.264 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:25:53.264 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:53.795309+0000", 2024-09-13T11:25:53.264 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.4617999999999999e-05 2024-09-13T11:25:53.265 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.265 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.265 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:25:53.265 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:53.795397+0000", 2024-09-13T11:25:53.265 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.7863999999999998e-05 2024-09-13T11:25:53.265 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.265 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.265 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 1", 2024-09-13T11:25:53.265 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:53.795493+0000", 2024-09-13T11:25:53.265 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.6062000000000003e-05 2024-09-13T11:25:53.265 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.265 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.265 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:25:53.266 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:53.797245+0000", 2024-09-13T11:25:53.266 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0017521559999999999 2024-09-13T11:25:53.266 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.266 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.266 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:25:53.266 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:54.010409+0000", 2024-09-13T11:25:53.266 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.21316385299999999 2024-09-13T11:25:53.266 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.266 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.266 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:25:53.266 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:54.010474+0000", 2024-09-13T11:25:53.266 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.4998000000000003e-05 2024-09-13T11:25:53.266 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.266 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.267 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:25:53.267 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:54.010521+0000", 2024-09-13T11:25:53.267 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.7231000000000002e-05 2024-09-13T11:25:53.267 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:53.267 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:25:53.267 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:53.267 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.267 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.267 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop_reply(client.4665.0:137780 3.8 e139/117 ondisk, result = 0)", 2024-09-13T11:25:53.267 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:24:53.797821+0000", 2024-09-13T11:25:53.267 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 59.338519171000002, 2024-09-13T11:25:53.267 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.21270765599999999, 2024-09-13T11:25:53.267 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:25:53.267 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:25:53.268 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:25:53.268 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:25:53.268 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.268 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:25:53.268 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:53.797821+0000", 2024-09-13T11:25:53.268 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:53.268 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.268 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.268 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:25:53.268 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:53.797821+0000", 2024-09-13T11:25:53.268 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:53.268 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.268 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.269 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:25:53.269 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:53.797822+0000", 2024-09-13T11:25:53.269 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.012e-06 2024-09-13T11:25:53.269 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.269 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.269 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:25:53.269 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:53.797829+0000", 2024-09-13T11:25:53.269 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.8009999999999998e-06 2024-09-13T11:25:53.269 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.269 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.269 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:25:53.269 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:54.010334+0000", 2024-09-13T11:25:53.269 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.212504477 2024-09-13T11:25:53.270 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.270 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.270 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:25:53.270 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:54.010342+0000", 2024-09-13T11:25:53.270 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.6549999999999993e-06 2024-09-13T11:25:53.270 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.270 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.270 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:25:53.270 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:54.010394+0000", 2024-09-13T11:25:53.270 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.1875000000000001e-05 2024-09-13T11:25:53.270 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.270 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.271 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:25:53.271 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:54.010402+0000", 2024-09-13T11:25:53.271 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.6499999999999996e-06 2024-09-13T11:25:53.271 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.271 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.271 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:25:53.271 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:54.010529+0000", 2024-09-13T11:25:53.271 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000127186 2024-09-13T11:25:53.271 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:53.271 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:25:53.271 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:53.271 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.271 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.271 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.4665.0:196211 3.6 e178/122 3:63573deb:::benchmark_data_smithi177_31116_object55908:head v 178'6255, mlcod=178'6252)", 2024-09-13T11:25:53.272 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:25:43.175295+0000", 2024-09-13T11:25:53.272 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 9.9610458410000007, 2024-09-13T11:25:53.272 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.167837029, 2024-09-13T11:25:53.272 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:25:53.272 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:25:53.272 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:53.272 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:25:53.272 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.272 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:25:53.272 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:43.175295+0000", 2024-09-13T11:25:53.272 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:53.272 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.272 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.272 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:25:53.273 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:43.175295+0000", 2024-09-13T11:25:53.273 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:53.273 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.273 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.273 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:25:53.273 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:43.175300+0000", 2024-09-13T11:25:53.273 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.9099999999999996e-06 2024-09-13T11:25:53.273 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.273 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.273 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:25:53.273 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:43.175318+0000", 2024-09-13T11:25:53.273 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.8346e-05 2024-09-13T11:25:53.273 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.273 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.274 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:25:53.274 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:43.340757+0000", 2024-09-13T11:25:53.274 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.165439058 2024-09-13T11:25:53.274 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.274 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.274 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:25:53.274 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:43.340775+0000", 2024-09-13T11:25:53.274 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.7546000000000001e-05 2024-09-13T11:25:53.274 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.274 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.274 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:25:53.274 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:43.340855+0000", 2024-09-13T11:25:53.274 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.0207e-05 2024-09-13T11:25:53.275 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.275 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.275 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:25:53.275 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:43.340879+0000", 2024-09-13T11:25:53.275 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.4494e-05 2024-09-13T11:25:53.275 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.275 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.275 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:25:53.275 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:43.343057+0000", 2024-09-13T11:25:53.275 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0021772380000000002 2024-09-13T11:25:53.275 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.275 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.275 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:25:53.275 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:43.343132+0000", 2024-09-13T11:25:53.276 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.5229999999999996e-05 2024-09-13T11:25:53.276 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:53.276 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:25:53.276 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:53.276 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.276 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.276 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.4665.0:196314 3.6 e178/122 3:62e79b1c:::benchmark_data_smithi177_31116_object56011:head v 178'6256, mlcod=178'6253)", 2024-09-13T11:25:53.276 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:25:43.340693+0000", 2024-09-13T11:25:53.276 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 9.7956475899999997, 2024-09-13T11:25:53.276 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.24109011699999999, 2024-09-13T11:25:53.276 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:25:53.276 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:25:53.276 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:53.277 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:25:53.277 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.277 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:25:53.277 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:43.340693+0000", 2024-09-13T11:25:53.277 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:53.277 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.277 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.277 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:25:53.277 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:43.340693+0000", 2024-09-13T11:25:53.277 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:53.277 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.277 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.277 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:25:53.278 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:43.340704+0000", 2024-09-13T11:25:53.278 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1049e-05 2024-09-13T11:25:53.278 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.278 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.278 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:25:53.278 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:43.340718+0000", 2024-09-13T11:25:53.278 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4214e-05 2024-09-13T11:25:53.278 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.278 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.278 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:25:53.278 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:43.579964+0000", 2024-09-13T11:25:53.278 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.23924590200000001 2024-09-13T11:25:53.278 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.278 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.279 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:25:53.279 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:43.579972+0000", 2024-09-13T11:25:53.279 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.2789999999999998e-06 2024-09-13T11:25:53.279 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.279 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.279 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:25:53.279 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:43.580043+0000", 2024-09-13T11:25:53.279 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.0856000000000001e-05 2024-09-13T11:25:53.279 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.279 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.279 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:25:53.279 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:43.580065+0000", 2024-09-13T11:25:53.279 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.1795999999999999e-05 2024-09-13T11:25:53.280 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.280 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.280 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:25:53.280 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:43.581741+0000", 2024-09-13T11:25:53.280 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0016759190000000001 2024-09-13T11:25:53.280 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.280 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.280 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:25:53.280 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:43.581783+0000", 2024-09-13T11:25:53.280 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.2101999999999998e-05 2024-09-13T11:25:53.280 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:53.280 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:25:53.280 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:53.280 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:53.281 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:25:53.281 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-13T11:25:53.282 INFO:teuthology.orchestra.run.smithi001.stdout:osd.7: {} 2024-09-13T11:25:53.282 INFO:teuthology.orchestra.run.smithi001.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T11:25:53.283 INFO:teuthology.orchestra.run.smithi001.stdout:osd.8: {} 2024-09-13T11:25:53.283 INFO:teuthology.orchestra.run.smithi001.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-13T11:25:53.283 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-09-13T11:25:53.284 INFO:teuthology.orchestra.run.smithi001.stdout:osd.9: {} 2024-09-13T11:25:53.284 INFO:teuthology.orchestra.run.smithi001.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-13T11:25:53.285 INFO:teuthology.orchestra.run.smithi001.stdout:osd.10: {} 2024-09-13T11:25:53.285 INFO:teuthology.orchestra.run.smithi001.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-13T11:25:53.285 INFO:teuthology.orchestra.run.smithi001.stdout:osd.11: {} 2024-09-13T11:25:53.285 INFO:teuthology.orchestra.run.smithi001.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-13T11:25:53.286 INFO:tasks.ceph.osd.8.smithi001.stderr:2024-09-13T11:25:53.278+0000 7fb9b48ae640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26312) UID: 0 2024-09-13T11:25:53.383 INFO:tasks.ceph.osd.14.smithi162.stderr:2024-09-13T11:25:53.379+0000 7f3df5903640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33154) UID: 0 2024-09-13T11:25:53.414 INFO:teuthology.orchestra.run.smithi001.stdout:osd.12: {} 2024-09-13T11:25:53.414 INFO:teuthology.orchestra.run.smithi001.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-13T11:25:53.418 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-13T11:25:53.418 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [ 2024-09-13T11:25:53.418 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.418 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:208908 3.18 3:1c2acd18:::benchmark_data_smithi177_31116_object68605:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e185)", 2024-09-13T11:25:53.418 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:25:53.415524+0000", 2024-09-13T11:25:53.418 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 0.002775469, 2024-09-13T11:25:53.418 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0027946830000000001, 2024-09-13T11:25:53.418 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:25:53.418 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:25:53.418 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "waiting for sub ops", 2024-09-13T11:25:53.418 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:25:53.419 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:25:53.419 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:53.419 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 208908 2024-09-13T11:25:53.419 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.419 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:25:53.419 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.419 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:25:53.419 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:53.415524+0000", 2024-09-13T11:25:53.419 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:53.419 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.419 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.419 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:25:53.419 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:53.415524+0000", 2024-09-13T11:25:53.420 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:53.420 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.420 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.420 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:25:53.420 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:53.415528+0000", 2024-09-13T11:25:53.420 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.163e-06 2024-09-13T11:25:53.420 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.420 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.420 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:25:53.420 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:53.415536+0000", 2024-09-13T11:25:53.420 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.3720000000000005e-06 2024-09-13T11:25:53.420 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.421 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.421 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:25:53.421 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:53.415558+0000", 2024-09-13T11:25:53.421 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.234e-05 2024-09-13T11:25:53.421 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.421 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.421 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:25:53.421 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:53.415569+0000", 2024-09-13T11:25:53.421 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1178e-05 2024-09-13T11:25:53.421 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.421 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.421 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:25:53.421 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:53.415655+0000", 2024-09-13T11:25:53.421 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.5573000000000002e-05 2024-09-13T11:25:53.422 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.422 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.422 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:25:53.422 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:53.415818+0000", 2024-09-13T11:25:53.422 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000162561 2024-09-13T11:25:53.422 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.422 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.422 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 0", 2024-09-13T11:25:53.422 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:53.415943+0000", 2024-09-13T11:25:53.422 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000125127 2024-09-13T11:25:53.422 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:53.422 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:25:53.422 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:53.422 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.423 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.423 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:208753 3.17 3:ec0bf46b:::benchmark_data_smithi177_31116_object68450:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e185)", 2024-09-13T11:25:53.423 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:25:53.304825+0000", 2024-09-13T11:25:53.423 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 0.113474692, 2024-09-13T11:25:53.423 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.113537135, 2024-09-13T11:25:53.423 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:25:53.423 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:25:53.423 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "waiting for sub ops", 2024-09-13T11:25:53.423 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:25:53.423 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:25:53.423 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:53.423 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 208753 2024-09-13T11:25:53.423 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.423 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:25:53.424 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.424 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:25:53.424 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:53.304825+0000", 2024-09-13T11:25:53.424 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:53.424 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.424 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.424 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:25:53.424 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:53.304825+0000", 2024-09-13T11:25:53.424 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:53.424 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.424 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.424 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:25:53.424 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:53.304828+0000", 2024-09-13T11:25:53.424 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.3920000000000001e-06 2024-09-13T11:25:53.425 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.425 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.425 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:25:53.425 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:53.304836+0000", 2024-09-13T11:25:53.425 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.4910000000000005e-06 2024-09-13T11:25:53.425 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.425 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.425 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:25:53.425 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:53.304856+0000", 2024-09-13T11:25:53.425 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.0225999999999999e-05 2024-09-13T11:25:53.425 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.425 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.425 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:25:53.425 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:53.304869+0000", 2024-09-13T11:25:53.426 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2833999999999999e-05 2024-09-13T11:25:53.426 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.426 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.426 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:25:53.426 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:53.304941+0000", 2024-09-13T11:25:53.426 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.1681999999999996e-05 2024-09-13T11:25:53.426 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.426 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.426 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:25:53.426 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:53.305042+0000", 2024-09-13T11:25:53.426 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000101129 2024-09-13T11:25:53.426 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.426 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.426 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 3", 2024-09-13T11:25:53.426 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:53.305130+0000", 2024-09-13T11:25:53.427 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.8146000000000006e-05 2024-09-13T11:25:53.427 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.427 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.427 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:25:53.427 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:53.308168+0000", 2024-09-13T11:25:53.427 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0030375490000000001 2024-09-13T11:25:53.427 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:53.427 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:25:53.427 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:53.427 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:53.427 INFO:teuthology.orchestra.run.smithi162.stdout: ], 2024-09-13T11:25:53.427 INFO:teuthology.orchestra.run.smithi162.stdout: "num_ops": 2 2024-09-13T11:25:53.427 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-13T11:25:53.434 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-09-13T11:25:53.440 INFO:teuthology.orchestra.run.smithi001.stdout:osd.13: {} 2024-09-13T11:25:53.441 INFO:teuthology.orchestra.run.smithi001.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-13T11:25:53.468 INFO:teuthology.orchestra.run.smithi001.stdout:osd.14: {} 2024-09-13T11:25:53.468 INFO:teuthology.orchestra.run.smithi001.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-13T11:25:53.474 INFO:tasks.ceph.osd.14.smithi162.stderr:2024-09-13T11:25:53.471+0000 7f3dea330640 -1 osd.14 184 handle_osd_map injecting map crc failure 2024-09-13T11:25:53.493 INFO:teuthology.orchestra.run.smithi001.stdout:osd.15: {} 2024-09-13T11:25:53.494 INFO:teuthology.orchestra.run.smithi001.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-13T11:25:53.517 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-09-13T11:25:53.515+0000 7f97befc9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26315) UID: 0 2024-09-13T11:25:53.585 INFO:tasks.ceph.osd.13.smithi092.stderr:2024-09-13T11:25:53.584+0000 7f3e5afea640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26350) UID: 0 2024-09-13T11:25:53.615 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-13T11:25:53.615 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [], 2024-09-13T11:25:53.615 INFO:teuthology.orchestra.run.smithi162.stdout: "complaint_time": 30, 2024-09-13T11:25:53.615 INFO:teuthology.orchestra.run.smithi162.stdout: "num_blocked_ops": 0 2024-09-13T11:25:53.615 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-13T11:25:53.631 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-09-13T11:25:53.686 INFO:tasks.ceph.osd.9.smithi092.stderr:2024-09-13T11:25:53.684+0000 7ff5ab3d0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26348) UID: 0 2024-09-13T11:25:53.787 INFO:tasks.ceph.osd.9.smithi092.stderr:2024-09-13T11:25:53.784+0000 7ff5ab3d0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26348) UID: 0 2024-09-13T11:25:53.805 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-13T11:25:53.806 INFO:teuthology.orchestra.run.smithi162.stdout: "size": 20, 2024-09-13T11:25:53.806 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 600, 2024-09-13T11:25:53.806 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [ 2024-09-13T11:25:53.806 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.806 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:45810 3.17 3.9bd13e57 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e69)", 2024-09-13T11:25:53.806 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:21.347960+0000", 2024-09-13T11:25:53.806 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 152.454236417, 2024-09-13T11:25:53.806 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.56814464200000003, 2024-09-13T11:25:53.806 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:25:53.806 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:25:53.806 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-09-13T11:25:53.806 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:25:53.806 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:25:53.807 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:53.807 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 45810 2024-09-13T11:25:53.807 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.807 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:25:53.807 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.807 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:25:53.807 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:21.347960+0000", 2024-09-13T11:25:53.807 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:53.807 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.807 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.807 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:25:53.807 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:21.347960+0000", 2024-09-13T11:25:53.807 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:53.807 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.808 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.808 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:25:53.808 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:21.347965+0000", 2024-09-13T11:25:53.808 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.6720000000000003e-06 2024-09-13T11:25:53.808 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.808 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.808 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:25:53.809 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:21.348019+0000", 2024-09-13T11:25:53.809 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.4391000000000002e-05 2024-09-13T11:25:53.809 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.809 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.809 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:25:53.809 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:21.348052+0000", 2024-09-13T11:25:53.809 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.2246000000000003e-05 2024-09-13T11:25:53.809 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.809 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.809 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:25:53.809 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:21.348072+0000", 2024-09-13T11:25:53.809 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.0622e-05 2024-09-13T11:25:53.810 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.810 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.810 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:25:53.810 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:21.348278+0000", 2024-09-13T11:25:53.810 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000205391 2024-09-13T11:25:53.810 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.810 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.810 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-09-13T11:25:53.810 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:21.348292+0000", 2024-09-13T11:25:53.810 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4759e-05 2024-09-13T11:25:53.810 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.810 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.810 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:25:53.810 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:21.916064+0000", 2024-09-13T11:25:53.811 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.56777128600000004 2024-09-13T11:25:53.811 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.811 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.811 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:25:53.811 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:21.916105+0000", 2024-09-13T11:25:53.811 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.1275e-05 2024-09-13T11:25:53.811 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:53.811 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:25:53.811 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:53.811 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.811 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.811 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:54653 3.23 3:c7a54800:::benchmark_data_smithi177_31116_object54652:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e75)", 2024-09-13T11:25:53.811 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:30.979059+0000", 2024-09-13T11:25:53.812 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 142.82313791799999, 2024-09-13T11:25:53.812 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.97284747100000002, 2024-09-13T11:25:53.812 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:25:53.812 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:25:53.812 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-09-13T11:25:53.812 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:25:53.812 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:25:53.812 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:53.812 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 54653 2024-09-13T11:25:53.812 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.812 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:25:53.812 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.812 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:25:53.813 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.979059+0000", 2024-09-13T11:25:53.813 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:53.813 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.813 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.813 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:25:53.813 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.979059+0000", 2024-09-13T11:25:53.813 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:53.813 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.813 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.813 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:25:53.813 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.979061+0000", 2024-09-13T11:25:53.813 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.4279999999999999e-06 2024-09-13T11:25:53.813 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.814 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.814 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:25:53.814 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.979114+0000", 2024-09-13T11:25:53.814 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.3254000000000002e-05 2024-09-13T11:25:53.814 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.814 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.814 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:25:53.814 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.979157+0000", 2024-09-13T11:25:53.814 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.2018000000000003e-05 2024-09-13T11:25:53.814 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.814 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.814 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:25:53.814 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.979172+0000", 2024-09-13T11:25:53.814 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5636000000000001e-05 2024-09-13T11:25:53.815 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.815 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.815 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:25:53.815 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:31.045109+0000", 2024-09-13T11:25:53.815 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.065936982000000005 2024-09-13T11:25:53.815 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.815 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.815 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-09-13T11:25:53.815 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:31.045121+0000", 2024-09-13T11:25:53.815 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1608e-05 2024-09-13T11:25:53.815 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.816 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.816 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:25:53.816 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:31.951636+0000", 2024-09-13T11:25:53.816 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.90651503899999997 2024-09-13T11:25:53.816 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.816 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.816 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:25:53.816 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:31.951906+0000", 2024-09-13T11:25:53.816 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000270506 2024-09-13T11:25:53.816 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:53.817 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:25:53.817 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:53.817 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.817 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.817 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPush(3.18 76/70 [PushOp(3:1a32c9cd:::benchmark_data_smithi177_31116_object12833:head, version: 49'818, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:1a32c9cd:::benchmark_data_smithi177_31116_object12833:head@49'818, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:25:53.817 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:32.522305+0000", 2024-09-13T11:25:53.817 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 141.27989183400001, 2024-09-13T11:25:53.817 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.40201207100000003, 2024-09-13T11:25:53.818 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:25:53.818 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:25:53.818 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:25:53.818 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:25:53.818 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.818 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:25:53.818 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.522305+0000", 2024-09-13T11:25:53.818 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:53.818 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.818 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.818 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:25:53.818 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.522305+0000", 2024-09-13T11:25:53.818 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:53.818 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.819 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.819 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:25:53.819 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.522309+0000", 2024-09-13T11:25:53.819 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.9829999999999998e-06 2024-09-13T11:25:53.819 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.820 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.820 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:25:53.820 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.522377+0000", 2024-09-13T11:25:53.820 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.8069000000000002e-05 2024-09-13T11:25:53.820 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.820 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.821 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:25:53.821 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.921950+0000", 2024-09-13T11:25:53.821 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.399573438 2024-09-13T11:25:53.821 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.821 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.822 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:25:53.822 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.921983+0000", 2024-09-13T11:25:53.822 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.2858000000000003e-05 2024-09-13T11:25:53.822 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.822 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.822 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:25:53.822 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.922389+0000", 2024-09-13T11:25:53.822 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00040612999999999999 2024-09-13T11:25:53.822 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.822 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.822 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:25:53.823 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.922421+0000", 2024-09-13T11:25:53.823 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.1356999999999998e-05 2024-09-13T11:25:53.823 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.823 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.823 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:25:53.823 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.924317+0000", 2024-09-13T11:25:53.823 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0018962359999999999 2024-09-13T11:25:53.823 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:53.823 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:25:53.823 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:53.823 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.823 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.823 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPush(3.17 76/70 [PushOp(3:ebcbce67:::benchmark_data_smithi177_31116_object18268:head, version: 53'1102, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:ebcbce67:::benchmark_data_smithi177_31116_object18268:head@53'1102, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:25:53.824 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:32.638908+0000", 2024-09-13T11:25:53.824 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 141.163288386, 2024-09-13T11:25:53.824 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.229064031, 2024-09-13T11:25:53.824 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:25:53.824 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:25:53.824 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:25:53.824 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:25:53.824 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.824 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:25:53.824 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.638908+0000", 2024-09-13T11:25:53.824 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:53.824 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.824 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.825 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:25:53.825 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.638908+0000", 2024-09-13T11:25:53.825 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:53.825 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.825 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.825 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:25:53.825 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.638910+0000", 2024-09-13T11:25:53.825 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.7379999999999999e-06 2024-09-13T11:25:53.825 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.825 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.825 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:25:53.825 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.638975+0000", 2024-09-13T11:25:53.826 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.4937000000000005e-05 2024-09-13T11:25:53.826 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.826 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.826 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:25:53.826 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.865860+0000", 2024-09-13T11:25:53.826 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.22688509800000001 2024-09-13T11:25:53.826 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.826 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.826 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:25:53.826 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.865890+0000", 2024-09-13T11:25:53.826 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.0273e-05 2024-09-13T11:25:53.826 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.826 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.827 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:25:53.827 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.866143+0000", 2024-09-13T11:25:53.827 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00025212800000000001 2024-09-13T11:25:53.827 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.827 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.827 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:25:53.827 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.866181+0000", 2024-09-13T11:25:53.827 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.8945000000000001e-05 2024-09-13T11:25:53.827 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.827 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.827 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:25:53.827 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.867972+0000", 2024-09-13T11:25:53.828 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001790912 2024-09-13T11:25:53.828 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:53.828 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:25:53.828 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:53.828 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.828 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.828 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:57435 3.3 3:c0db4777:::benchmark_data_smithi177_31116_object57434:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e80)", 2024-09-13T11:25:53.828 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:37.436034+0000", 2024-09-13T11:25:53.828 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 136.36616292599999, 2024-09-13T11:25:53.828 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.35949142000000001, 2024-09-13T11:25:53.828 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:25:53.828 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:25:53.829 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:53.829 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:25:53.829 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:25:53.829 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:53.829 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 57435 2024-09-13T11:25:53.829 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.829 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:25:53.829 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.829 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:25:53.829 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:37.436034+0000", 2024-09-13T11:25:53.829 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:53.829 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.829 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.830 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:25:53.830 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:37.436034+0000", 2024-09-13T11:25:53.830 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:53.830 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.830 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.830 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:25:53.830 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:37.436037+0000", 2024-09-13T11:25:53.830 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.7829999999999999e-06 2024-09-13T11:25:53.830 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.830 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.831 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:25:53.831 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:37.436090+0000", 2024-09-13T11:25:53.831 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.3665999999999997e-05 2024-09-13T11:25:53.831 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.831 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.831 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:25:53.831 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:37.436112+0000", 2024-09-13T11:25:53.831 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.1556000000000002e-05 2024-09-13T11:25:53.832 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.832 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.832 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:25:53.832 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:37.436122+0000", 2024-09-13T11:25:53.832 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0098000000000001e-05 2024-09-13T11:25:53.833 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.833 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.833 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:25:53.833 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:37.436187+0000", 2024-09-13T11:25:53.833 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.4857999999999995e-05 2024-09-13T11:25:53.833 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.834 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.834 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:25:53.834 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:37.436285+0000", 2024-09-13T11:25:53.834 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.8643999999999999e-05 2024-09-13T11:25:53.834 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.835 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.835 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 7", 2024-09-13T11:25:53.835 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:37.436423+0000", 2024-09-13T11:25:53.835 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00013809299999999999 2024-09-13T11:25:53.835 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.836 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.836 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:25:53.836 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:37.440125+0000", 2024-09-13T11:25:53.836 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0037015239999999999 2024-09-13T11:25:53.836 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.836 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.837 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:25:53.837 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:37.795397+0000", 2024-09-13T11:25:53.837 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.35527234400000002 2024-09-13T11:25:53.837 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.837 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.837 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:25:53.838 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:37.795485+0000", 2024-09-13T11:25:53.838 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.7538000000000001e-05 2024-09-13T11:25:53.838 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.838 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.838 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:25:53.838 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:37.795525+0000", 2024-09-13T11:25:53.839 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.0315999999999998e-05 2024-09-13T11:25:53.839 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:53.839 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:25:53.839 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:53.839 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.839 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.840 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPush(3.1f 80/78 [PushOp(3:fc3ed0c7:::benchmark_data_smithi177_31116_object672:head, version: 43'36, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:fc3ed0c7:::benchmark_data_smithi177_31116_object672:head@43'36, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:25:53.840 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:37.801754+0000", 2024-09-13T11:25:53.840 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 136.000442566, 2024-09-13T11:25:53.840 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.209613519, 2024-09-13T11:25:53.840 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:25:53.840 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:25:53.841 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:25:53.841 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:25:53.841 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.841 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:25:53.841 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:37.801754+0000", 2024-09-13T11:25:53.841 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:53.842 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.842 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.842 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:25:53.842 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:37.801754+0000", 2024-09-13T11:25:53.842 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:53.842 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.842 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.843 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:25:53.843 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:37.801756+0000", 2024-09-13T11:25:53.843 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.804e-06 2024-09-13T11:25:53.843 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.843 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.843 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:25:53.843 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:37.801947+0000", 2024-09-13T11:25:53.844 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000190636 2024-09-13T11:25:53.844 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.844 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.844 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:25:53.844 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:38.010478+0000", 2024-09-13T11:25:53.844 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.20853091100000001 2024-09-13T11:25:53.845 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.845 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.845 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:25:53.845 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:38.010506+0000", 2024-09-13T11:25:53.845 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.8793999999999999e-05 2024-09-13T11:25:53.846 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.846 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.846 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:25:53.846 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:38.010601+0000", 2024-09-13T11:25:53.847 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.4786999999999999e-05 2024-09-13T11:25:53.847 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.847 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.847 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:25:53.847 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:38.010617+0000", 2024-09-13T11:25:53.847 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5790999999999998e-05 2024-09-13T11:25:53.848 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.848 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.848 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:25:53.848 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:38.011368+0000", 2024-09-13T11:25:53.848 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00075079599999999997 2024-09-13T11:25:53.849 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:53.849 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:25:53.849 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:53.849 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.849 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.849 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:60433 3.3 3:c01f7ddc:::benchmark_data_smithi177_31116_object60432:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e85)", 2024-09-13T11:25:53.850 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:42.199382+0000", 2024-09-13T11:25:53.850 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 131.60281503499999, 2024-09-13T11:25:53.850 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.33918965200000001, 2024-09-13T11:25:53.850 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:25:53.850 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:25:53.850 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:53.850 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:25:53.851 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:25:53.851 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:53.851 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 60433 2024-09-13T11:25:53.851 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.851 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:25:53.851 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.851 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:25:53.851 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.199382+0000", 2024-09-13T11:25:53.852 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:53.852 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.852 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.852 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:25:53.852 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.199382+0000", 2024-09-13T11:25:53.852 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:53.852 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.852 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.853 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:25:53.853 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.199386+0000", 2024-09-13T11:25:53.853 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.8759999999999998e-06 2024-09-13T11:25:53.853 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.853 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.853 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:25:53.853 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.199455+0000", 2024-09-13T11:25:53.853 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.9305000000000001e-05 2024-09-13T11:25:53.854 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.854 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.854 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:25:53.854 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.199482+0000", 2024-09-13T11:25:53.854 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.6749000000000001e-05 2024-09-13T11:25:53.854 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.855 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.855 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:25:53.855 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.199503+0000", 2024-09-13T11:25:53.855 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.1137000000000001e-05 2024-09-13T11:25:53.855 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.855 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.855 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:25:53.856 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.199602+0000", 2024-09-13T11:25:53.856 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.8743e-05 2024-09-13T11:25:53.856 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.856 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.856 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:25:53.856 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.199784+0000", 2024-09-13T11:25:53.857 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00018265100000000001 2024-09-13T11:25:53.857 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.857 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.857 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 7", 2024-09-13T11:25:53.857 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.199955+0000", 2024-09-13T11:25:53.857 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00017067100000000001 2024-09-13T11:25:53.858 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.858 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.858 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:25:53.858 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.202075+0000", 2024-09-13T11:25:53.858 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0021206749999999998 2024-09-13T11:25:53.858 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.859 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.859 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:25:53.859 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.538395+0000", 2024-09-13T11:25:53.859 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.33631963599999998 2024-09-13T11:25:53.859 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.860 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.860 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:25:53.860 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.538498+0000", 2024-09-13T11:25:53.860 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00010267099999999999 2024-09-13T11:25:53.860 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.860 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.861 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:25:53.861 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.538571+0000", 2024-09-13T11:25:53.861 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.3538e-05 2024-09-13T11:25:53.861 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:53.861 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:25:53.861 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:53.862 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.862 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.862 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop_reply(client.4665.0:60433 3.3 e85/75 ondisk, result = 0)", 2024-09-13T11:25:53.862 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:42.203186+0000", 2024-09-13T11:25:53.862 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 131.599010857, 2024-09-13T11:25:53.863 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.33540096699999999, 2024-09-13T11:25:53.863 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:25:53.863 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:25:53.863 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:25:53.863 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:25:53.863 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.863 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:25:53.864 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.203186+0000", 2024-09-13T11:25:53.864 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:53.864 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.864 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.864 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:25:53.865 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.203186+0000", 2024-09-13T11:25:53.865 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:53.865 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.865 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.865 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:25:53.866 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.203194+0000", 2024-09-13T11:25:53.866 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.0020000000000006e-06 2024-09-13T11:25:53.866 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.866 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.866 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:25:53.867 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.203202+0000", 2024-09-13T11:25:53.867 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.9829999999999996e-06 2024-09-13T11:25:53.867 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.867 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.867 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:25:53.868 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.538182+0000", 2024-09-13T11:25:53.868 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.33498029899999998 2024-09-13T11:25:53.868 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.868 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.868 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:25:53.869 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.538196+0000", 2024-09-13T11:25:53.869 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3807e-05 2024-09-13T11:25:53.869 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.869 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.869 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:25:53.869 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.538371+0000", 2024-09-13T11:25:53.869 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00017541200000000001 2024-09-13T11:25:53.869 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.869 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.869 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:25:53.869 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.538386+0000", 2024-09-13T11:25:53.870 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4833e-05 2024-09-13T11:25:53.870 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.870 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.870 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:25:53.870 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.538587+0000", 2024-09-13T11:25:53.870 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000200631 2024-09-13T11:25:53.870 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:53.870 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:25:53.871 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:53.871 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.871 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.871 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:69556 3.18 3:1d3dbeba:::benchmark_data_smithi177_31116_object69555:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94)", 2024-09-13T11:25:53.871 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:52.142180+0000", 2024-09-13T11:25:53.871 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 121.660016787, 2024-09-13T11:25:53.871 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.21280995899999999, 2024-09-13T11:25:53.871 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:25:53.871 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:25:53.871 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:53.871 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:25:53.872 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:25:53.872 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:53.872 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 69556 2024-09-13T11:25:53.872 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.872 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:25:53.872 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.872 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:25:53.872 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:52.142180+0000", 2024-09-13T11:25:53.872 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:53.872 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.872 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.873 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:25:53.873 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:52.142180+0000", 2024-09-13T11:25:53.873 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:53.873 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.873 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.873 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:25:53.873 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:52.142181+0000", 2024-09-13T11:25:53.873 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.3699999999999999e-07 2024-09-13T11:25:53.873 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.873 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.874 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:25:53.874 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:52.142212+0000", 2024-09-13T11:25:53.874 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.1334000000000002e-05 2024-09-13T11:25:53.874 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.874 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.875 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:25:53.875 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:52.142219+0000", 2024-09-13T11:25:53.875 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.9990000000000002e-06 2024-09-13T11:25:53.875 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.875 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.875 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:25:53.876 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:52.142224+0000", 2024-09-13T11:25:53.876 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.0259999999999998e-06 2024-09-13T11:25:53.876 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.876 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.876 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:25:53.877 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:52.142276+0000", 2024-09-13T11:25:53.877 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.1436e-05 2024-09-13T11:25:53.877 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.877 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.877 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:25:53.877 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:52.142396+0000", 2024-09-13T11:25:53.878 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00012011 2024-09-13T11:25:53.878 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.878 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.878 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 0", 2024-09-13T11:25:53.878 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:52.142503+0000", 2024-09-13T11:25:53.879 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000107701 2024-09-13T11:25:53.879 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.879 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.879 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:25:53.879 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:52.144566+0000", 2024-09-13T11:25:53.879 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.002062749 2024-09-13T11:25:53.880 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.880 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.880 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:25:53.880 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:52.354839+0000", 2024-09-13T11:25:53.880 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.21027254000000001 2024-09-13T11:25:53.880 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.881 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.881 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:25:53.881 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:52.354932+0000", 2024-09-13T11:25:53.881 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.3510000000000004e-05 2024-09-13T11:25:53.881 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.881 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.882 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:25:53.882 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:52.354990+0000", 2024-09-13T11:25:53.882 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.7617000000000001e-05 2024-09-13T11:25:53.882 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:53.882 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:25:53.882 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:53.882 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.883 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.883 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:83144 3.18 3:1c2a5085:::benchmark_data_smithi177_31116_object83143:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106)", 2024-09-13T11:25:53.883 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:24:04.186395+0000", 2024-09-13T11:25:53.883 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 109.615801738, 2024-09-13T11:25:53.883 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.29655527599999998, 2024-09-13T11:25:53.884 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:25:53.884 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:25:53.884 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:53.884 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:25:53.884 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:25:53.884 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:53.884 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 83144 2024-09-13T11:25:53.885 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.885 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:25:53.885 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.885 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:25:53.885 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.186395+0000", 2024-09-13T11:25:53.885 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:53.885 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.886 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.886 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:25:53.886 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.186395+0000", 2024-09-13T11:25:53.886 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:53.886 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.886 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.887 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:25:53.887 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.186398+0000", 2024-09-13T11:25:53.887 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.5299999999999999e-06 2024-09-13T11:25:53.887 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.887 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.887 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:25:53.888 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.186438+0000", 2024-09-13T11:25:53.888 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.0293000000000002e-05 2024-09-13T11:25:53.888 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.888 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.888 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:25:53.889 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.186450+0000", 2024-09-13T11:25:53.889 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2687000000000001e-05 2024-09-13T11:25:53.889 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.889 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.889 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:25:53.890 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.186456+0000", 2024-09-13T11:25:53.890 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.446e-06 2024-09-13T11:25:53.890 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.890 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.890 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:25:53.890 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.186503+0000", 2024-09-13T11:25:53.890 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.7491000000000003e-05 2024-09-13T11:25:53.890 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.891 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.891 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:25:53.891 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.186618+0000", 2024-09-13T11:25:53.891 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000114999 2024-09-13T11:25:53.891 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.892 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.892 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 0", 2024-09-13T11:25:53.892 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.186712+0000", 2024-09-13T11:25:53.892 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.3805000000000001e-05 2024-09-13T11:25:53.892 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.892 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.893 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:25:53.893 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.189753+0000", 2024-09-13T11:25:53.893 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.003040713 2024-09-13T11:25:53.893 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.893 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.893 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:25:53.893 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.482662+0000", 2024-09-13T11:25:53.893 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.29290945699999998 2024-09-13T11:25:53.893 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.894 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.894 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:25:53.894 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.482831+0000", 2024-09-13T11:25:53.894 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00016900699999999999 2024-09-13T11:25:53.894 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.894 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.894 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:25:53.894 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.482950+0000", 2024-09-13T11:25:53.894 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000118848 2024-09-13T11:25:53.894 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:53.894 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:25:53.894 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:53.895 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.895 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.895 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop_reply(client.4665.0:83144 3.18 e106/91 ondisk, result = 0)", 2024-09-13T11:25:53.895 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:24:04.188413+0000", 2024-09-13T11:25:53.895 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 109.613783728, 2024-09-13T11:25:53.895 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.294557562, 2024-09-13T11:25:53.895 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:25:53.895 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:25:53.895 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:25:53.895 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:25:53.895 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.895 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:25:53.895 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.188413+0000", 2024-09-13T11:25:53.896 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:53.896 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.896 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.896 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:25:53.896 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.188413+0000", 2024-09-13T11:25:53.896 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:53.896 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.896 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.896 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:25:53.896 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.188414+0000", 2024-09-13T11:25:53.896 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4309999999999999e-06 2024-09-13T11:25:53.896 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.896 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.897 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:25:53.897 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.188419+0000", 2024-09-13T11:25:53.897 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.2889999999999999e-06 2024-09-13T11:25:53.897 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.897 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.897 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:25:53.897 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.482442+0000", 2024-09-13T11:25:53.897 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.294023168 2024-09-13T11:25:53.897 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.897 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.897 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:25:53.898 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.482459+0000", 2024-09-13T11:25:53.898 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.7013000000000001e-05 2024-09-13T11:25:53.898 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.898 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.898 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:25:53.898 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.482615+0000", 2024-09-13T11:25:53.898 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00015620400000000001 2024-09-13T11:25:53.898 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.898 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.898 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:25:53.899 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.482644+0000", 2024-09-13T11:25:53.899 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.9093000000000001e-05 2024-09-13T11:25:53.899 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.899 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.899 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:25:53.899 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.482971+0000", 2024-09-13T11:25:53.900 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00032636399999999999 2024-09-13T11:25:53.900 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:53.900 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:25:53.900 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:53.900 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.900 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.901 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPush(3.1f 106/78 [PushOp(3:fd4be819:::benchmark_data_smithi177_31116_object54560:head, version: 74'3410, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:fd4be819:::benchmark_data_smithi177_31116_object54560:head@74'3410, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:25:53.901 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:24:04.397161+0000", 2024-09-13T11:25:53.901 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 109.40503531900001, 2024-09-13T11:25:53.901 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.51611096199999995, 2024-09-13T11:25:53.901 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:25:53.902 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:25:53.902 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:25:53.902 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:25:53.902 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.902 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:25:53.902 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.397161+0000", 2024-09-13T11:25:53.902 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:53.902 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.902 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.903 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:25:53.903 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.397161+0000", 2024-09-13T11:25:53.903 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:53.903 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.903 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.903 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:25:53.904 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.397163+0000", 2024-09-13T11:25:53.904 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4720000000000001e-06 2024-09-13T11:25:53.904 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.904 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.904 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:25:53.904 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.397228+0000", 2024-09-13T11:25:53.904 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.5141999999999994e-05 2024-09-13T11:25:53.904 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.904 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.904 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:25:53.904 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.911583+0000", 2024-09-13T11:25:53.905 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.51435473700000001 2024-09-13T11:25:53.905 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.905 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.905 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:25:53.905 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.911615+0000", 2024-09-13T11:25:53.905 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.2620000000000003e-05 2024-09-13T11:25:53.905 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.905 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.905 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:25:53.905 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.911759+0000", 2024-09-13T11:25:53.905 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00014376300000000001 2024-09-13T11:25:53.905 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.905 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.906 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:25:53.906 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.911779+0000", 2024-09-13T11:25:53.906 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.0003000000000001e-05 2024-09-13T11:25:53.906 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.906 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.906 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:25:53.906 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.913272+0000", 2024-09-13T11:25:53.906 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0014932249999999999 2024-09-13T11:25:53.906 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:53.906 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:25:53.906 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:53.906 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.906 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.907 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:84809 3.17 3:e9e52123:::benchmark_data_smithi177_31116_object84808:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-09-13T11:25:53.907 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:24:05.914353+0000", 2024-09-13T11:25:53.907 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 107.887843301, 2024-09-13T11:25:53.907 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.27210915899999999, 2024-09-13T11:25:53.907 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:25:53.907 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:25:53.907 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:53.907 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:25:53.907 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:25:53.907 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:53.907 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 84809 2024-09-13T11:25:53.907 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.907 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:25:53.907 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.908 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:25:53.908 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.914353+0000", 2024-09-13T11:25:53.908 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:53.908 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.908 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.908 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:25:53.908 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.914353+0000", 2024-09-13T11:25:53.908 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:53.908 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.908 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.908 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:25:53.908 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.914356+0000", 2024-09-13T11:25:53.908 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.8710000000000001e-06 2024-09-13T11:25:53.909 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.909 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.909 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:25:53.909 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.914443+0000", 2024-09-13T11:25:53.909 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.7002000000000005e-05 2024-09-13T11:25:53.909 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.909 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.909 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:25:53.909 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.914461+0000", 2024-09-13T11:25:53.909 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.7691000000000001e-05 2024-09-13T11:25:53.909 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.909 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.909 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:25:53.910 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.914471+0000", 2024-09-13T11:25:53.910 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0447e-05 2024-09-13T11:25:53.910 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.910 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.910 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:25:53.910 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.914550+0000", 2024-09-13T11:25:53.910 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.8683000000000005e-05 2024-09-13T11:25:53.910 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.910 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.910 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:25:53.910 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.914709+0000", 2024-09-13T11:25:53.910 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00015913200000000001 2024-09-13T11:25:53.910 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.910 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.911 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 3", 2024-09-13T11:25:53.911 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.914880+0000", 2024-09-13T11:25:53.911 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00017071499999999999 2024-09-13T11:25:53.911 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.911 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.911 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:25:53.911 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.916701+0000", 2024-09-13T11:25:53.911 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0018205369999999999 2024-09-13T11:25:53.911 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.911 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.911 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:25:53.911 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:06.186401+0000", 2024-09-13T11:25:53.911 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.26970018699999998 2024-09-13T11:25:53.911 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.912 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.912 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:25:53.912 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:06.186434+0000", 2024-09-13T11:25:53.912 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.3448999999999998e-05 2024-09-13T11:25:53.912 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.912 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.912 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:25:53.912 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:06.186463+0000", 2024-09-13T11:25:53.912 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.8445e-05 2024-09-13T11:25:53.912 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:53.912 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:25:53.912 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:53.912 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.912 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.913 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop_reply(client.4665.0:84809 3.17 e107/85 ondisk, result = 0)", 2024-09-13T11:25:53.913 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:24:05.916779+0000", 2024-09-13T11:25:53.913 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 107.88541782999999, 2024-09-13T11:25:53.913 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.26968993499999999, 2024-09-13T11:25:53.913 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:25:53.913 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:25:53.913 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:25:53.913 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:25:53.913 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.913 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:25:53.913 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.916779+0000", 2024-09-13T11:25:53.913 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:53.913 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.913 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.914 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:25:53.914 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.916779+0000", 2024-09-13T11:25:53.914 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:53.914 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.914 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.914 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:25:53.914 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.916780+0000", 2024-09-13T11:25:53.914 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.328e-06 2024-09-13T11:25:53.914 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.914 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.914 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:25:53.914 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.916789+0000", 2024-09-13T11:25:53.914 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.4460000000000001e-06 2024-09-13T11:25:53.915 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.915 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.915 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:25:53.915 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:06.186296+0000", 2024-09-13T11:25:53.915 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.26950707099999999 2024-09-13T11:25:53.915 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.915 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.915 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:25:53.915 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:06.186306+0000", 2024-09-13T11:25:53.915 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0137e-05 2024-09-13T11:25:53.915 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.915 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.915 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:25:53.915 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:06.186388+0000", 2024-09-13T11:25:53.916 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.2149000000000002e-05 2024-09-13T11:25:53.916 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.916 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.916 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:25:53.916 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:06.186397+0000", 2024-09-13T11:25:53.916 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.5210000000000002e-06 2024-09-13T11:25:53.916 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.916 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.916 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:25:53.916 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:06.186469+0000", 2024-09-13T11:25:53.916 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.2282999999999999e-05 2024-09-13T11:25:53.916 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:53.916 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:25:53.917 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:53.917 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.917 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.917 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPush(3.2a 121/120 [PushOp(3:5595108b:::benchmark_data_smithi177_31116_object899:head, version: 43'62, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:5595108b:::benchmark_data_smithi177_31116_object899:head@43'62, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:25:53.917 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:24:27.352355+0000", 2024-09-13T11:25:53.917 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 86.449842024000006, 2024-09-13T11:25:53.917 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.61628666099999996, 2024-09-13T11:25:53.917 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:25:53.917 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:25:53.917 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:25:53.917 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:25:53.917 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.917 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:25:53.918 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:27.352355+0000", 2024-09-13T11:25:53.918 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:53.918 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.918 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.918 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:25:53.918 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:27.352355+0000", 2024-09-13T11:25:53.918 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:53.918 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.918 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.918 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:25:53.918 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:27.352357+0000", 2024-09-13T11:25:53.918 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.328e-06 2024-09-13T11:25:53.918 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.918 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.919 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:25:53.919 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:27.352428+0000", 2024-09-13T11:25:53.919 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.1118999999999995e-05 2024-09-13T11:25:53.919 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.919 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.919 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:25:53.919 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:27.966950+0000", 2024-09-13T11:25:53.919 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.61452202499999997 2024-09-13T11:25:53.919 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.919 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.919 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:25:53.919 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:27.966980+0000", 2024-09-13T11:25:53.919 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.9433000000000001e-05 2024-09-13T11:25:53.920 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.920 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.920 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:25:53.920 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:27.967220+0000", 2024-09-13T11:25:53.920 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00024065499999999999 2024-09-13T11:25:53.920 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.920 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.920 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:25:53.920 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:27.967244+0000", 2024-09-13T11:25:53.920 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.4216e-05 2024-09-13T11:25:53.920 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.920 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.920 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:25:53.921 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:27.968641+0000", 2024-09-13T11:25:53.921 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0013968850000000001 2024-09-13T11:25:53.921 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:53.921 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:25:53.921 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:53.921 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.921 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.921 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:162057 3.1f 3:f8710094:::benchmark_data_smithi177_31116_object21754:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e156)", 2024-09-13T11:25:53.921 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:25:15.442781+0000", 2024-09-13T11:25:53.921 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 38.359415380999998, 2024-09-13T11:25:53.921 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.25670282500000002, 2024-09-13T11:25:53.921 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:25:53.921 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:25:53.922 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:53.922 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:25:53.922 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:25:53.922 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:53.922 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 162057 2024-09-13T11:25:53.922 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.922 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:25:53.922 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.922 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:25:53.922 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:15.442781+0000", 2024-09-13T11:25:53.922 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:53.922 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.922 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.923 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:25:53.923 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:15.442781+0000", 2024-09-13T11:25:53.923 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:53.923 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.923 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.923 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:25:53.923 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:15.442785+0000", 2024-09-13T11:25:53.923 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.4000000000000001e-06 2024-09-13T11:25:53.923 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.923 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.923 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:25:53.923 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:15.442795+0000", 2024-09-13T11:25:53.923 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0193999999999999e-05 2024-09-13T11:25:53.923 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.924 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.924 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:25:53.924 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:15.442820+0000", 2024-09-13T11:25:53.924 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.4683000000000001e-05 2024-09-13T11:25:53.924 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.924 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.924 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:25:53.924 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:15.442840+0000", 2024-09-13T11:25:53.924 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.0596000000000001e-05 2024-09-13T11:25:53.924 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.924 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.924 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:25:53.924 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:15.442952+0000", 2024-09-13T11:25:53.925 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00011215899999999999 2024-09-13T11:25:53.925 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.925 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.925 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:25:53.925 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:15.443143+0000", 2024-09-13T11:25:53.925 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00019058899999999999 2024-09-13T11:25:53.925 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.925 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.925 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 5", 2024-09-13T11:25:53.925 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:15.443327+0000", 2024-09-13T11:25:53.925 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00018364499999999999 2024-09-13T11:25:53.925 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.925 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.926 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:25:53.926 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:15.446727+0000", 2024-09-13T11:25:53.926 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0034000689999999999 2024-09-13T11:25:53.926 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.926 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.926 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:25:53.926 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:15.699356+0000", 2024-09-13T11:25:53.926 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.25262958400000002 2024-09-13T11:25:53.926 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.926 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.926 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:25:53.926 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:15.699428+0000", 2024-09-13T11:25:53.926 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.1810000000000005e-05 2024-09-13T11:25:53.926 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.927 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.927 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:25:53.927 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:15.699484+0000", 2024-09-13T11:25:53.927 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.6096e-05 2024-09-13T11:25:53.927 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:53.927 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:25:53.927 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:53.927 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.927 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.927 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:162073 3.1f 3:fd1a00ae:::benchmark_data_smithi177_31116_object21770:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e156)", 2024-09-13T11:25:53.927 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:25:15.454353+0000", 2024-09-13T11:25:53.927 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 38.347843425999997, 2024-09-13T11:25:53.928 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.27516647, 2024-09-13T11:25:53.928 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:25:53.928 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:25:53.928 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:53.928 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:25:53.928 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:25:53.928 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:53.928 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 162073 2024-09-13T11:25:53.928 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.928 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:25:53.928 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.928 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:25:53.928 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:15.454353+0000", 2024-09-13T11:25:53.929 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:53.929 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.929 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.929 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:25:53.929 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:15.454353+0000", 2024-09-13T11:25:53.929 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:53.929 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.929 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.929 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:25:53.929 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:15.454358+0000", 2024-09-13T11:25:53.929 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.3499999999999999e-06 2024-09-13T11:25:53.929 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.929 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.929 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:25:53.930 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:15.454371+0000", 2024-09-13T11:25:53.930 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3594000000000001e-05 2024-09-13T11:25:53.930 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.930 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.930 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:25:53.930 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:15.454408+0000", 2024-09-13T11:25:53.930 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.6619999999999998e-05 2024-09-13T11:25:53.930 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.930 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.930 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:25:53.930 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:15.454431+0000", 2024-09-13T11:25:53.930 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.3546000000000001e-05 2024-09-13T11:25:53.930 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.931 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.931 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:25:53.931 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:15.454587+0000", 2024-09-13T11:25:53.931 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000155248 2024-09-13T11:25:53.931 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.931 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.931 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:25:53.931 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:15.454897+0000", 2024-09-13T11:25:53.931 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000309955 2024-09-13T11:25:53.931 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.931 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.931 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 5", 2024-09-13T11:25:53.931 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:15.455203+0000", 2024-09-13T11:25:53.932 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00030604999999999999 2024-09-13T11:25:53.932 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.932 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.932 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:25:53.932 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:15.460534+0000", 2024-09-13T11:25:53.932 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0053310199999999997 2024-09-13T11:25:53.932 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.932 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.932 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:25:53.932 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:15.729356+0000", 2024-09-13T11:25:53.932 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.26882267199999998 2024-09-13T11:25:53.932 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.932 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.933 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:25:53.933 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:15.729442+0000", 2024-09-13T11:25:53.933 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.5209000000000004e-05 2024-09-13T11:25:53.933 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.933 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.933 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:25:53.933 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:15.729520+0000", 2024-09-13T11:25:53.933 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.8206000000000003e-05 2024-09-13T11:25:53.933 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:53.933 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:25:53.933 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:53.933 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.933 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.933 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:162102 3.1f 3:fc0cda6c:::benchmark_data_smithi177_31116_object21799:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e156)", 2024-09-13T11:25:53.934 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:25:15.477044+0000", 2024-09-13T11:25:53.934 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 38.325152572, 2024-09-13T11:25:53.934 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.28798895800000002, 2024-09-13T11:25:53.934 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:25:53.934 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:25:53.934 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:53.934 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:25:53.934 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:25:53.934 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:53.934 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 162102 2024-09-13T11:25:53.934 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.935 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:25:53.935 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.935 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:25:53.935 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:15.477044+0000", 2024-09-13T11:25:53.935 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:53.935 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.935 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.935 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:25:53.935 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:15.477044+0000", 2024-09-13T11:25:53.935 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:53.935 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.935 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.935 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:25:53.935 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:15.477048+0000", 2024-09-13T11:25:53.936 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.3569999999999998e-06 2024-09-13T11:25:53.936 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.936 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.936 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:25:53.936 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:15.477058+0000", 2024-09-13T11:25:53.936 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.9899999999999992e-06 2024-09-13T11:25:53.936 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.936 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.936 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:25:53.936 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:15.477087+0000", 2024-09-13T11:25:53.936 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.9026999999999998e-05 2024-09-13T11:25:53.936 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.936 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.937 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:25:53.937 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:15.477101+0000", 2024-09-13T11:25:53.937 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4583000000000001e-05 2024-09-13T11:25:53.937 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.937 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.937 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:25:53.937 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:15.477212+0000", 2024-09-13T11:25:53.937 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00011071200000000001 2024-09-13T11:25:53.937 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.937 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.937 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:25:53.937 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:15.477492+0000", 2024-09-13T11:25:53.937 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00027995400000000001 2024-09-13T11:25:53.937 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.938 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.938 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 5", 2024-09-13T11:25:53.938 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:15.477682+0000", 2024-09-13T11:25:53.938 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000189929 2024-09-13T11:25:53.938 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.938 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.938 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:25:53.938 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:15.481903+0000", 2024-09-13T11:25:53.938 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0042214649999999998 2024-09-13T11:25:53.938 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.938 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.938 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:25:53.938 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:15.764891+0000", 2024-09-13T11:25:53.939 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.28298753599999998 2024-09-13T11:25:53.939 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.939 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.939 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:25:53.939 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:15.764972+0000", 2024-09-13T11:25:53.939 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.1230000000000007e-05 2024-09-13T11:25:53.939 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.939 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.939 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:25:53.939 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:15.765033+0000", 2024-09-13T11:25:53.939 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.1174999999999996e-05 2024-09-13T11:25:53.939 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:53.939 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:25:53.939 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:53.940 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.940 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.940 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop_reply(client.4665.0:162057 3.1f e156/122 ondisk, result = 0)", 2024-09-13T11:25:53.940 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:25:15.479734+0000", 2024-09-13T11:25:53.940 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 38.322462473999998, 2024-09-13T11:25:53.940 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.21975831800000001, 2024-09-13T11:25:53.940 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:25:53.940 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:25:53.940 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:25:53.940 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:25:53.940 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.940 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:25:53.940 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:15.479734+0000", 2024-09-13T11:25:53.940 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:53.941 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.941 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.941 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:25:53.941 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:15.479734+0000", 2024-09-13T11:25:53.941 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:53.941 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.941 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.941 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:25:53.941 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:15.479745+0000", 2024-09-13T11:25:53.941 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.084e-05 2024-09-13T11:25:53.941 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.941 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.941 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:25:53.941 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:15.479757+0000", 2024-09-13T11:25:53.942 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1600000000000001e-05 2024-09-13T11:25:53.942 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.942 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.942 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:25:53.942 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:15.698724+0000", 2024-09-13T11:25:53.942 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.218967044 2024-09-13T11:25:53.942 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.942 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.942 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:25:53.943 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:15.698740+0000", 2024-09-13T11:25:53.943 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.6707000000000001e-05 2024-09-13T11:25:53.943 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.943 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.943 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:25:53.943 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:15.699336+0000", 2024-09-13T11:25:53.943 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00059547500000000004 2024-09-13T11:25:53.943 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.943 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.943 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:25:53.943 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:15.699348+0000", 2024-09-13T11:25:53.943 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1703e-05 2024-09-13T11:25:53.943 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.943 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.944 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:25:53.944 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:15.699493+0000", 2024-09-13T11:25:53.944 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00014494900000000001 2024-09-13T11:25:53.944 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:53.944 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:25:53.944 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:53.944 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.944 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.944 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:191286 3.18 3:1ffc0b26:::benchmark_data_smithi177_31116_object50983:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e175)", 2024-09-13T11:25:53.944 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:25:39.152634+0000", 2024-09-13T11:25:53.944 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 14.649562391, 2024-09-13T11:25:53.944 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.236097576, 2024-09-13T11:25:53.944 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:25:53.945 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:25:53.945 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:53.945 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:25:53.945 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:25:53.945 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:53.945 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 191286 2024-09-13T11:25:53.945 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.945 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:25:53.945 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.945 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:25:53.945 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:39.152634+0000", 2024-09-13T11:25:53.945 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:53.945 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.946 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.946 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:25:53.946 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:39.152634+0000", 2024-09-13T11:25:53.946 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:25:53.946 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.946 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.946 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:25:53.946 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:39.152638+0000", 2024-09-13T11:25:53.946 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.5130000000000001e-06 2024-09-13T11:25:53.946 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.946 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.946 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:25:53.946 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:39.152649+0000", 2024-09-13T11:25:53.946 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0677e-05 2024-09-13T11:25:53.947 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.947 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.947 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:25:53.947 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:39.152678+0000", 2024-09-13T11:25:53.947 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.9436000000000001e-05 2024-09-13T11:25:53.947 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.947 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.947 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:25:53.947 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:39.152693+0000", 2024-09-13T11:25:53.947 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4823e-05 2024-09-13T11:25:53.947 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.947 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.947 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:25:53.947 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:39.152821+0000", 2024-09-13T11:25:53.948 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00012815299999999999 2024-09-13T11:25:53.948 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.948 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.948 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:25:53.948 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:39.153031+0000", 2024-09-13T11:25:53.948 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00021011399999999999 2024-09-13T11:25:53.948 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.948 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.948 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 0", 2024-09-13T11:25:53.948 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:39.153195+0000", 2024-09-13T11:25:53.948 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00016374599999999999 2024-09-13T11:25:53.948 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.948 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.948 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:25:53.949 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:39.157900+0000", 2024-09-13T11:25:53.949 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0047053049999999999 2024-09-13T11:25:53.949 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.949 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.949 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:25:53.949 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:39.388451+0000", 2024-09-13T11:25:53.949 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.230550957 2024-09-13T11:25:53.949 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.949 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.949 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:25:53.949 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:39.388611+0000", 2024-09-13T11:25:53.949 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00016029299999999999 2024-09-13T11:25:53.949 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:25:53.950 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:25:53.950 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:25:53.950 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:39.388732+0000", 2024-09-13T11:25:53.950 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000120559 2024-09-13T11:25:53.950 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:53.950 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:25:53.950 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:53.950 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:25:53.950 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:25:53.950 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-13T11:25:53.952 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-13T11:25:53.952 INFO:tasks.ceph.osd.4.smithi001.stderr:2024-09-13T11:25:53.950+0000 7ff8e0218640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26310) UID: 0 2024-09-13T11:25:54.053 INFO:tasks.ceph.osd.4.smithi001.stderr:2024-09-13T11:25:54.050+0000 7ff8e0218640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26310) UID: 0 2024-09-13T11:25:54.103 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-09-13T11:25:54.103 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [ 2024-09-13T11:25:54.103 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:54.104 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:209653 3.d e185/117 3:b1abd8b0:::benchmark_data_smithi177_31116_object69350:head v 185'6668, mlcod=185'6666)", 2024-09-13T11:25:54.104 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:25:54.102081+0000", 2024-09-13T11:25:54.104 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 0.0019567180000000001, 2024-09-13T11:25:54.104 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001974136, 2024-09-13T11:25:54.104 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:25:54.104 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:25:54.104 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:25:54.104 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:25:54.104 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:54.104 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:25:54.104 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:54.102081+0000", 2024-09-13T11:25:54.105 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:54.105 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:54.105 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:54.105 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:25:54.105 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:54.102081+0000", 2024-09-13T11:25:54.105 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:54.105 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:54.105 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:54.105 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:25:54.105 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:54.102082+0000", 2024-09-13T11:25:54.105 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.553e-06 2024-09-13T11:25:54.106 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:54.106 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:54.106 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:25:54.106 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:54.102089+0000", 2024-09-13T11:25:54.106 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.0929999999999997e-06 2024-09-13T11:25:54.106 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:54.106 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:54.106 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:25:54.106 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:54.102125+0000", 2024-09-13T11:25:54.106 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.561e-05 2024-09-13T11:25:54.106 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:54.106 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:54.107 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:25:54.107 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:54.102132+0000", 2024-09-13T11:25:54.107 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.5970000000000003e-06 2024-09-13T11:25:54.107 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:54.107 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:54.107 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:25:54.107 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:54.102186+0000", 2024-09-13T11:25:54.107 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.3640000000000001e-05 2024-09-13T11:25:54.107 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:54.107 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:54.107 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:25:54.107 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:54.102206+0000", 2024-09-13T11:25:54.108 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.9658999999999999e-05 2024-09-13T11:25:54.108 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:54.108 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:25:54.108 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:54.108 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:54.108 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:54.108 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:209654 3.17 e185/85 3:e8cd2465:::benchmark_data_smithi177_31116_object69351:head v 185'7749, mlcod=185'7747)", 2024-09-13T11:25:54.108 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:25:54.103897+0000", 2024-09-13T11:25:54.108 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 0.00014033900000000001, 2024-09-13T11:25:54.108 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000216574, 2024-09-13T11:25:54.108 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:25:54.108 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:25:54.108 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:25:54.109 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:25:54.109 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:54.109 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:25:54.109 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:54.103897+0000", 2024-09-13T11:25:54.109 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:54.109 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:54.109 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:54.109 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:25:54.109 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:54.103897+0000", 2024-09-13T11:25:54.109 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:54.109 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:54.109 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:54.110 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:25:54.110 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:54.103899+0000", 2024-09-13T11:25:54.110 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.979e-06 2024-09-13T11:25:54.110 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:54.110 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:54.110 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:25:54.110 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:54.103909+0000", 2024-09-13T11:25:54.110 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0061000000000001e-05 2024-09-13T11:25:54.110 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:54.110 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:54.110 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:25:54.110 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:54.103941+0000", 2024-09-13T11:25:54.111 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.1773999999999998e-05 2024-09-13T11:25:54.111 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:54.111 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:54.111 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:25:54.112 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:54.103947+0000", 2024-09-13T11:25:54.112 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.5819999999999998e-06 2024-09-13T11:25:54.112 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:54.112 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:54.112 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:25:54.112 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:54.104004+0000", 2024-09-13T11:25:54.112 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.6150999999999998e-05 2024-09-13T11:25:54.112 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:54.112 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:54.113 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:25:54.113 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:54.104022+0000", 2024-09-13T11:25:54.113 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.8130000000000001e-05 2024-09-13T11:25:54.113 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:54.113 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:25:54.113 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:54.113 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:54.113 INFO:teuthology.orchestra.run.smithi177.stdout: ], 2024-09-13T11:25:54.113 INFO:teuthology.orchestra.run.smithi177.stdout: "num_ops": 2 2024-09-13T11:25:54.113 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-09-13T11:25:54.123 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-13T11:25:54.153 INFO:tasks.ceph.osd.14.smithi162.stderr:2024-09-13T11:25:54.151+0000 7f3df5903640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33154) UID: 0 2024-09-13T11:25:54.254 INFO:tasks.ceph.osd.15.smithi177.stderr:2024-09-13T11:25:54.250+0000 7f9d6ce77640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26578) UID: 0 2024-09-13T11:25:54.269 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-09-13T11:25:54.269 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [], 2024-09-13T11:25:54.269 INFO:teuthology.orchestra.run.smithi177.stdout: "complaint_time": 30, 2024-09-13T11:25:54.269 INFO:teuthology.orchestra.run.smithi177.stdout: "num_blocked_ops": 0 2024-09-13T11:25:54.269 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-09-13T11:25:54.285 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-13T11:25:54.355 INFO:tasks.ceph.osd.4.smithi001.stderr:2024-09-13T11:25:54.354+0000 7ff8e0218640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26310) UID: 0 2024-09-13T11:25:54.456 INFO:tasks.ceph.osd.7.smithi177.stderr:2024-09-13T11:25:54.454+0000 7f897b51d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26575) UID: 0 2024-09-13T11:25:54.457 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-09-13T11:25:54.458 INFO:teuthology.orchestra.run.smithi177.stdout: "size": 20, 2024-09-13T11:25:54.458 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 600, 2024-09-13T11:25:54.458 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [ 2024-09-13T11:25:54.458 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:54.458 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPush(3.17 75/70 [PushOp(3:eda94455:::benchmark_data_smithi177_31116_object13017:head, version: 50'788, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:eda94455:::benchmark_data_smithi177_31116_object13017:head@50'788, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:25:54.458 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:31.038029+0000", 2024-09-13T11:25:54.458 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 143.41900519000001, 2024-09-13T11:25:54.458 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.42064623499999998, 2024-09-13T11:25:54.458 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:25:54.458 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:25:54.458 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:25:54.458 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:25:54.458 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:54.459 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:25:54.459 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.038029+0000", 2024-09-13T11:25:54.459 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:54.459 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:54.459 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:54.459 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:25:54.459 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.038029+0000", 2024-09-13T11:25:54.459 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:54.459 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:54.459 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:54.459 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:25:54.459 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.038030+0000", 2024-09-13T11:25:54.459 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.499e-06 2024-09-13T11:25:54.459 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:54.460 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:54.460 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:25:54.460 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.038088+0000", 2024-09-13T11:25:54.460 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.7771e-05 2024-09-13T11:25:54.460 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:54.460 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:54.460 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:25:54.460 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.457025+0000", 2024-09-13T11:25:54.460 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.41893754900000002 2024-09-13T11:25:54.460 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:54.460 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:54.460 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:25:54.460 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.457073+0000", 2024-09-13T11:25:54.461 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.7450999999999997e-05 2024-09-13T11:25:54.461 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:54.461 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:54.461 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:25:54.461 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.457304+0000", 2024-09-13T11:25:54.461 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000231351 2024-09-13T11:25:54.461 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:54.461 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:54.461 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:25:54.461 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.457331+0000", 2024-09-13T11:25:54.461 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.6372000000000001e-05 2024-09-13T11:25:54.461 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:54.461 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:54.461 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:25:54.462 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.458675+0000", 2024-09-13T11:25:54.462 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001344242 2024-09-13T11:25:54.462 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:54.462 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:25:54.462 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:54.462 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:54.462 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:54.462 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPush(3.17 83/70 [PushOp(3:eb574268:::benchmark_data_smithi177_31116_object40622:head, version: 64'2475, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:eb574268:::benchmark_data_smithi177_31116_object40622:head@64'2475, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:25:54.462 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:40.760042+0000", 2024-09-13T11:25:54.462 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 133.696991473, 2024-09-13T11:25:54.462 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.30544642100000002, 2024-09-13T11:25:54.462 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:25:54.462 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:25:54.463 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:25:54.463 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:25:54.463 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:54.463 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:25:54.463 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:40.760042+0000", 2024-09-13T11:25:54.463 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:54.463 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:54.463 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:54.463 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:25:54.463 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:40.760042+0000", 2024-09-13T11:25:54.463 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:54.463 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:54.463 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:54.463 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:25:54.464 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:40.760044+0000", 2024-09-13T11:25:54.464 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.511e-06 2024-09-13T11:25:54.464 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:54.464 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:54.464 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:25:54.464 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:40.760149+0000", 2024-09-13T11:25:54.464 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000105473 2024-09-13T11:25:54.464 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:54.464 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:54.464 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:25:54.464 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.064465+0000", 2024-09-13T11:25:54.464 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.30431613800000001 2024-09-13T11:25:54.464 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:54.464 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:54.465 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:25:54.465 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.064479+0000", 2024-09-13T11:25:54.465 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3586e-05 2024-09-13T11:25:54.465 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:54.465 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:54.465 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:25:54.465 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.064553+0000", 2024-09-13T11:25:54.465 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.3646000000000006e-05 2024-09-13T11:25:54.465 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:54.465 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:54.465 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:25:54.465 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.064568+0000", 2024-09-13T11:25:54.466 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.52e-05 2024-09-13T11:25:54.466 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:54.466 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:54.466 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:25:54.466 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.065489+0000", 2024-09-13T11:25:54.466 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00092086700000000002 2024-09-13T11:25:54.466 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:54.466 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:25:54.466 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:54.466 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:54.466 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:54.466 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPushReply(3.1a 83/78 [PushReplyOp(3:5c3834bd:::benchmark_data_smithi177_31116_object8669:head)])", 2024-09-13T11:25:54.466 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:40.798236+0000", 2024-09-13T11:25:54.466 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 133.65879804299999, 2024-09-13T11:25:54.466 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.29701520999999997, 2024-09-13T11:25:54.467 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:25:54.467 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:25:54.467 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-13T11:25:54.467 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:25:54.467 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:54.467 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:25:54.467 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:40.798236+0000", 2024-09-13T11:25:54.467 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:54.467 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:54.467 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:54.467 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:25:54.467 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:40.798236+0000", 2024-09-13T11:25:54.467 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:54.467 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:54.468 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:54.468 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:25:54.468 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:40.798237+0000", 2024-09-13T11:25:54.468 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.0500000000000002e-07 2024-09-13T11:25:54.468 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:54.468 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:54.468 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:25:54.468 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:40.798240+0000", 2024-09-13T11:25:54.468 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.4819999999999999e-06 2024-09-13T11:25:54.468 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:54.468 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:54.468 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:25:54.468 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.093360+0000", 2024-09-13T11:25:54.469 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.29512009 2024-09-13T11:25:54.469 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:54.469 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:54.469 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:25:54.469 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.093372+0000", 2024-09-13T11:25:54.469 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.2221e-05 2024-09-13T11:25:54.469 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:54.469 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:54.469 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:25:54.469 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.093482+0000", 2024-09-13T11:25:54.469 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000109571 2024-09-13T11:25:54.469 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:54.469 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:54.469 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:25:54.470 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.095251+0000", 2024-09-13T11:25:54.470 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001768941 2024-09-13T11:25:54.470 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:54.470 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:25:54.470 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:54.470 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:54.470 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:54.470 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPushReply(3.1a 83/78 [PushReplyOp(3:58c7ffc0:::benchmark_data_smithi177_31116_object8738:head)])", 2024-09-13T11:25:54.470 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:40.805941+0000", 2024-09-13T11:25:54.470 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 133.65109278899999, 2024-09-13T11:25:54.470 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.30315678499999998, 2024-09-13T11:25:54.470 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:25:54.471 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:25:54.471 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-13T11:25:54.471 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:25:54.471 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:54.471 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:25:54.471 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:40.805941+0000", 2024-09-13T11:25:54.471 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:54.471 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:54.471 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:54.471 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:25:54.471 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:40.805941+0000", 2024-09-13T11:25:54.471 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:54.471 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:54.471 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:54.472 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:25:54.472 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:40.805942+0000", 2024-09-13T11:25:54.472 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3769999999999999e-06 2024-09-13T11:25:54.472 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:54.472 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:54.472 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:25:54.472 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:40.805947+0000", 2024-09-13T11:25:54.472 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.8980000000000004e-06 2024-09-13T11:25:54.472 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:54.472 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:54.472 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:25:54.472 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.106505+0000", 2024-09-13T11:25:54.472 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.30055777700000003 2024-09-13T11:25:54.473 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:54.473 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:54.473 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:25:54.473 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.106512+0000", 2024-09-13T11:25:54.473 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.3749999999999997e-06 2024-09-13T11:25:54.473 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:54.473 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:54.473 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:25:54.473 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.106566+0000", 2024-09-13T11:25:54.473 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.3103999999999998e-05 2024-09-13T11:25:54.473 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:54.473 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:54.473 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:25:54.474 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.109098+0000", 2024-09-13T11:25:54.474 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0025322539999999998 2024-09-13T11:25:54.474 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:54.474 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:25:54.474 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:54.474 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:54.474 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:54.474 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPushReply(3.1a 83/78 [PushReplyOp(3:58c7ffc0:::benchmark_data_smithi177_31116_object8738:head)])", 2024-09-13T11:25:54.474 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:40.805973+0000", 2024-09-13T11:25:54.474 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 133.65106078599999, 2024-09-13T11:25:54.474 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.28729672099999998, 2024-09-13T11:25:54.474 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:25:54.474 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:25:54.474 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-13T11:25:54.474 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:25:54.475 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:54.475 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:25:54.475 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:40.805973+0000", 2024-09-13T11:25:54.475 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:54.475 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:54.475 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:54.475 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:25:54.475 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:40.805973+0000", 2024-09-13T11:25:54.475 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:54.475 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:54.475 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:54.475 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:25:54.475 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:40.805974+0000", 2024-09-13T11:25:54.475 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.8599999999999998e-07 2024-09-13T11:25:54.476 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:54.476 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:54.476 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:25:54.476 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:40.805976+0000", 2024-09-13T11:25:54.476 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.2809999999999998e-06 2024-09-13T11:25:54.476 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:54.476 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:54.476 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:25:54.476 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.091588+0000", 2024-09-13T11:25:54.476 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.28561173899999998 2024-09-13T11:25:54.476 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:54.476 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:54.476 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:25:54.477 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.091596+0000", 2024-09-13T11:25:54.477 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.0520000000000002e-06 2024-09-13T11:25:54.477 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:54.477 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:54.477 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:25:54.477 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.091907+0000", 2024-09-13T11:25:54.477 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00031179599999999998 2024-09-13T11:25:54.477 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:54.477 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:54.477 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:25:54.478 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.093270+0000", 2024-09-13T11:25:54.478 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0013622669999999999 2024-09-13T11:25:54.478 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:54.478 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:25:54.478 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:54.478 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:54.478 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:54.478 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPushReply(3.1a 84/78 [PushReplyOp(3:581ea97e:::benchmark_data_smithi177_31116_object10255:head)])", 2024-09-13T11:25:54.478 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:41.514228+0000", 2024-09-13T11:25:54.478 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 132.942805298, 2024-09-13T11:25:54.478 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.40695564200000001, 2024-09-13T11:25:54.478 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:25:54.478 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:25:54.479 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-13T11:25:54.479 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:25:54.479 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:54.479 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:25:54.479 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.514228+0000", 2024-09-13T11:25:54.479 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:54.479 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:54.479 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:54.479 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:25:54.480 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.514228+0000", 2024-09-13T11:25:54.480 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:54.480 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:54.480 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:54.480 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:25:54.480 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.514230+0000", 2024-09-13T11:25:54.480 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.499e-06 2024-09-13T11:25:54.480 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:54.480 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:54.480 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:25:54.480 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.514236+0000", 2024-09-13T11:25:54.480 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.5389999999999998e-06 2024-09-13T11:25:54.480 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:54.481 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:54.481 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:25:54.481 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.919787+0000", 2024-09-13T11:25:54.481 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.40555152999999999 2024-09-13T11:25:54.481 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:54.481 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:54.481 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:25:54.481 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.919796+0000", 2024-09-13T11:25:54.481 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.4829999999999999e-06 2024-09-13T11:25:54.481 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:54.481 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:54.481 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:25:54.481 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.920366+0000", 2024-09-13T11:25:54.481 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00057061700000000004 2024-09-13T11:25:54.481 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:54.482 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:54.482 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:25:54.482 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.921184+0000", 2024-09-13T11:25:54.482 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00081797399999999996 2024-09-13T11:25:54.482 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:54.482 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:25:54.482 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:54.482 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:54.482 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:54.482 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:60312 3.14 3:2e846241:::benchmark_data_smithi177_31116_object60311:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e85)", 2024-09-13T11:25:54.482 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:42.065725+0000", 2024-09-13T11:25:54.482 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 132.39130865499999, 2024-09-13T11:25:54.482 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.211323176, 2024-09-13T11:25:54.482 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:25:54.483 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:25:54.483 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:54.483 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:25:54.483 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:25:54.483 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:54.483 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 60312 2024-09-13T11:25:54.483 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:54.483 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:25:54.483 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:54.483 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:25:54.483 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.065725+0000", 2024-09-13T11:25:54.483 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:54.483 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:54.484 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:54.484 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:25:54.484 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.065725+0000", 2024-09-13T11:25:54.484 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:54.484 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:54.484 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:54.484 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:25:54.484 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.065728+0000", 2024-09-13T11:25:54.484 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.3400000000000002e-06 2024-09-13T11:25:54.484 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:54.484 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:54.484 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:25:54.484 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.065759+0000", 2024-09-13T11:25:54.485 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.0074e-05 2024-09-13T11:25:54.485 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:54.485 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:54.485 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:25:54.485 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.065771+0000", 2024-09-13T11:25:54.485 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.2869e-05 2024-09-13T11:25:54.485 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:54.485 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:54.485 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:25:54.485 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.065780+0000", 2024-09-13T11:25:54.485 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.3489999999999992e-06 2024-09-13T11:25:54.485 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:54.485 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:54.485 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:25:54.486 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.065837+0000", 2024-09-13T11:25:54.486 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.7444000000000001e-05 2024-09-13T11:25:54.486 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:54.486 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:54.486 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:25:54.486 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.065966+0000", 2024-09-13T11:25:54.486 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000128973 2024-09-13T11:25:54.486 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:54.486 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:54.486 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 14", 2024-09-13T11:25:54.486 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.066114+0000", 2024-09-13T11:25:54.486 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00014788299999999999 2024-09-13T11:25:54.486 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:54.486 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:54.487 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:25:54.487 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.067604+0000", 2024-09-13T11:25:54.487 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001489637 2024-09-13T11:25:54.487 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:54.487 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:54.487 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:25:54.487 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.276963+0000", 2024-09-13T11:25:54.487 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.20935888499999999 2024-09-13T11:25:54.487 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:54.487 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:54.487 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:25:54.487 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.277003+0000", 2024-09-13T11:25:54.487 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.0241000000000003e-05 2024-09-13T11:25:54.487 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:54.488 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:54.488 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:25:54.488 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.277048+0000", 2024-09-13T11:25:54.488 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.5481e-05 2024-09-13T11:25:54.488 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:54.488 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:25:54.488 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:54.488 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:54.488 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:54.488 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop_reply(client.4665.0:60312 3.14 e85/84 ondisk, result = 0)", 2024-09-13T11:25:54.488 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:42.069120+0000", 2024-09-13T11:25:54.488 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 132.387913535, 2024-09-13T11:25:54.488 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.20804031000000001, 2024-09-13T11:25:54.488 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:25:54.489 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:25:54.489 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:25:54.489 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:25:54.489 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:54.489 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:25:54.489 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.069120+0000", 2024-09-13T11:25:54.489 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:54.489 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:54.489 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:54.489 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:25:54.489 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.069120+0000", 2024-09-13T11:25:54.489 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:54.489 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:54.490 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:54.490 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:25:54.490 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.069123+0000", 2024-09-13T11:25:54.490 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.289e-06 2024-09-13T11:25:54.490 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:54.490 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:54.490 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:25:54.490 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.069128+0000", 2024-09-13T11:25:54.490 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.3530000000000002e-06 2024-09-13T11:25:54.490 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:54.490 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:54.490 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:25:54.490 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.276885+0000", 2024-09-13T11:25:54.490 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.207756989 2024-09-13T11:25:54.490 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:54.491 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:54.491 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:25:54.491 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.276892+0000", 2024-09-13T11:25:54.491 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.922e-06 2024-09-13T11:25:54.491 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:54.491 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:54.491 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:25:54.491 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.276953+0000", 2024-09-13T11:25:54.491 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.0890000000000001e-05 2024-09-13T11:25:54.491 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:54.491 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:54.491 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:25:54.491 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.276959+0000", 2024-09-13T11:25:54.491 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.9179999999999999e-06 2024-09-13T11:25:54.492 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:54.492 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:54.492 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:25:54.492 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.277161+0000", 2024-09-13T11:25:54.492 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00020194899999999999 2024-09-13T11:25:54.492 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:54.492 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:25:54.492 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:54.492 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:54.492 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:54.492 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:60427 3.14 3:2afea229:::benchmark_data_smithi177_31116_object60426:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e85)", 2024-09-13T11:25:54.492 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:42.187231+0000", 2024-09-13T11:25:54.492 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 132.26980242900001, 2024-09-13T11:25:54.493 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.83900596999999999, 2024-09-13T11:25:54.493 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:25:54.493 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:25:54.493 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:54.493 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:25:54.493 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:25:54.493 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:54.493 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 60427 2024-09-13T11:25:54.493 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:54.493 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:25:54.493 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:54.493 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:25:54.493 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.187231+0000", 2024-09-13T11:25:54.493 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:54.494 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:54.494 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:54.494 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:25:54.494 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.187231+0000", 2024-09-13T11:25:54.494 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:54.494 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:54.494 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:54.494 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:25:54.494 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.187235+0000", 2024-09-13T11:25:54.494 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.3579999999999999e-06 2024-09-13T11:25:54.494 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:54.494 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:54.494 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:25:54.494 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.187261+0000", 2024-09-13T11:25:54.495 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.6089e-05 2024-09-13T11:25:54.495 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:54.495 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:54.495 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:25:54.495 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.187275+0000", 2024-09-13T11:25:54.495 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.4653e-05 2024-09-13T11:25:54.495 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:54.495 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:54.495 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:25:54.495 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.187282+0000", 2024-09-13T11:25:54.495 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.8870000000000003e-06 2024-09-13T11:25:54.495 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:54.495 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:54.495 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:25:54.496 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.187336+0000", 2024-09-13T11:25:54.496 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.3875999999999997e-05 2024-09-13T11:25:54.496 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:54.496 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:54.496 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:25:54.496 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.187480+0000", 2024-09-13T11:25:54.496 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00014330899999999999 2024-09-13T11:25:54.496 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:54.496 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:54.496 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 14", 2024-09-13T11:25:54.496 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.187658+0000", 2024-09-13T11:25:54.496 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00017861900000000001 2024-09-13T11:25:54.496 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:54.496 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:54.497 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:25:54.497 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.189353+0000", 2024-09-13T11:25:54.497 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001695144 2024-09-13T11:25:54.497 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:54.497 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:54.497 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:25:54.497 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:43.026081+0000", 2024-09-13T11:25:54.497 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.83672736199999997 2024-09-13T11:25:54.497 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:54.497 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:54.497 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:25:54.497 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:43.026169+0000", 2024-09-13T11:25:54.497 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.8411000000000004e-05 2024-09-13T11:25:54.497 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:54.498 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:54.498 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:25:54.498 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:43.026237+0000", 2024-09-13T11:25:54.498 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.8262000000000005e-05 2024-09-13T11:25:54.498 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:54.498 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:25:54.498 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:54.498 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:54.498 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:54.498 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop_reply(client.4665.0:60427 3.14 e85/84 ondisk, result = 0)", 2024-09-13T11:25:54.498 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:42.190914+0000", 2024-09-13T11:25:54.498 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 132.26611976500001, 2024-09-13T11:25:54.498 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.83534071499999996, 2024-09-13T11:25:54.498 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:25:54.498 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:25:54.499 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:25:54.499 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:25:54.499 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:54.499 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:25:54.499 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.190914+0000", 2024-09-13T11:25:54.499 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:54.499 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:54.499 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:54.499 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:25:54.499 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.190914+0000", 2024-09-13T11:25:54.499 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:54.499 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:54.499 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:54.500 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:25:54.500 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.190916+0000", 2024-09-13T11:25:54.500 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.84e-06 2024-09-13T11:25:54.500 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:54.500 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:54.500 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:25:54.500 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.190921+0000", 2024-09-13T11:25:54.500 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.2039999999999996e-06 2024-09-13T11:25:54.500 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:54.500 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:54.500 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:25:54.500 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:43.025926+0000", 2024-09-13T11:25:54.500 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.83500542 2024-09-13T11:25:54.500 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:54.501 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:54.501 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:25:54.501 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:43.025937+0000", 2024-09-13T11:25:54.501 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0597e-05 2024-09-13T11:25:54.501 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:54.501 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:54.501 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:25:54.501 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:43.026052+0000", 2024-09-13T11:25:54.501 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000114838 2024-09-13T11:25:54.501 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:54.501 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:54.501 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:25:54.501 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:43.026070+0000", 2024-09-13T11:25:54.501 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.8131e-05 2024-09-13T11:25:54.502 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:54.502 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:54.502 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:25:54.502 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:43.026255+0000", 2024-09-13T11:25:54.502 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000184685 2024-09-13T11:25:54.502 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:54.502 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:25:54.502 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:54.502 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:54.502 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:54.502 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPushReply(3.1d 89/78 [PushReplyOp(3:bc1b7cfb:::benchmark_data_smithi177_31116_object22629:head)])", 2024-09-13T11:25:54.502 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:46.663106+0000", 2024-09-13T11:25:54.502 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 127.79392758199999, 2024-09-13T11:25:54.503 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.26645533199999999, 2024-09-13T11:25:54.503 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:25:54.503 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:25:54.503 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-13T11:25:54.503 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:25:54.503 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:54.503 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:25:54.503 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.663106+0000", 2024-09-13T11:25:54.503 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:54.503 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:54.503 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:54.503 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:25:54.503 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.663106+0000", 2024-09-13T11:25:54.504 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:54.504 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:54.504 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:54.504 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:25:54.504 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.663108+0000", 2024-09-13T11:25:54.504 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.0140000000000001e-06 2024-09-13T11:25:54.504 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:54.504 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:54.504 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:25:54.504 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.663116+0000", 2024-09-13T11:25:54.504 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.5440000000000001e-06 2024-09-13T11:25:54.504 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:54.504 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:54.504 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:25:54.505 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.928111+0000", 2024-09-13T11:25:54.505 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.26499566899999999 2024-09-13T11:25:54.505 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:54.505 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:54.505 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:25:54.505 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.928135+0000", 2024-09-13T11:25:54.505 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.4003e-05 2024-09-13T11:25:54.505 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:54.505 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:54.505 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:25:54.505 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.928559+0000", 2024-09-13T11:25:54.505 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000423318 2024-09-13T11:25:54.505 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:54.505 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:54.506 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:25:54.506 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.929562+0000", 2024-09-13T11:25:54.506 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0010027840000000001 2024-09-13T11:25:54.506 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:54.506 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:25:54.506 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:54.506 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:54.506 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:54.506 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "rep_scrubmap(3.14 e84 from shard 14)", 2024-09-13T11:25:54.506 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:48.313615+0000", 2024-09-13T11:25:54.506 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 126.14341878899999, 2024-09-13T11:25:54.506 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.35633836099999999, 2024-09-13T11:25:54.506 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:25:54.506 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:25:54.507 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:25:54.507 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:25:54.507 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:54.507 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:25:54.507 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.313615+0000", 2024-09-13T11:25:54.507 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:54.507 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:54.507 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:54.507 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:25:54.507 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.313615+0000", 2024-09-13T11:25:54.507 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:54.507 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:54.507 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:54.508 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:25:54.508 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.313617+0000", 2024-09-13T11:25:54.508 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.7990000000000001e-06 2024-09-13T11:25:54.508 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:54.508 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:54.508 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:25:54.508 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.313688+0000", 2024-09-13T11:25:54.508 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.1481000000000002e-05 2024-09-13T11:25:54.508 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:54.508 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:54.508 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:25:54.508 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.669785+0000", 2024-09-13T11:25:54.508 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.356096466 2024-09-13T11:25:54.509 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:54.509 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:54.509 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:25:54.509 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.669792+0000", 2024-09-13T11:25:54.509 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.1740000000000003e-06 2024-09-13T11:25:54.509 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:54.509 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:54.509 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:25:54.509 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.669839+0000", 2024-09-13T11:25:54.509 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.7323000000000001e-05 2024-09-13T11:25:54.509 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:54.509 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:54.509 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:25:54.509 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.669847+0000", 2024-09-13T11:25:54.510 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.0469999999999994e-06 2024-09-13T11:25:54.510 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:54.510 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:54.510 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:25:54.510 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.669953+0000", 2024-09-13T11:25:54.510 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00010607099999999999 2024-09-13T11:25:54.510 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:54.510 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:25:54.510 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:54.510 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:54.510 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:54.510 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:110268 3.14 3:2997c4c5:::benchmark_data_smithi177_31116_object110267:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119)", 2024-09-13T11:25:54.510 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:25.359386+0000", 2024-09-13T11:25:54.510 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 89.097647472999995, 2024-09-13T11:25:54.511 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.29072703, 2024-09-13T11:25:54.511 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:25:54.511 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:25:54.511 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:54.511 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:25:54.511 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:25:54.511 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:54.511 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 110268 2024-09-13T11:25:54.511 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:54.511 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:25:54.511 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:54.511 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:25:54.511 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.359386+0000", 2024-09-13T11:25:54.511 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:54.512 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:54.512 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:54.512 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:25:54.512 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.359386+0000", 2024-09-13T11:25:54.512 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:54.512 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:54.512 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:54.512 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:25:54.512 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.359389+0000", 2024-09-13T11:25:54.512 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.8600000000000001e-06 2024-09-13T11:25:54.512 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:54.512 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:54.512 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:25:54.512 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.359424+0000", 2024-09-13T11:25:54.513 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.4755999999999997e-05 2024-09-13T11:25:54.513 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:54.513 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:54.513 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:25:54.513 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.359445+0000", 2024-09-13T11:25:54.513 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.1104999999999999e-05 2024-09-13T11:25:54.513 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:54.513 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:54.513 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:25:54.513 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.359460+0000", 2024-09-13T11:25:54.513 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5396999999999998e-05 2024-09-13T11:25:54.513 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:54.514 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:54.514 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:25:54.514 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.359581+0000", 2024-09-13T11:25:54.515 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000120455 2024-09-13T11:25:54.515 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:54.515 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:54.515 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:25:54.515 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.359827+0000", 2024-09-13T11:25:54.515 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00024571 2024-09-13T11:25:54.515 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:54.515 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:54.515 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 14", 2024-09-13T11:25:54.515 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.360084+0000", 2024-09-13T11:25:54.515 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00025787999999999999 2024-09-13T11:25:54.515 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:54.515 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:54.515 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:25:54.516 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.362243+0000", 2024-09-13T11:25:54.516 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0021581579999999999 2024-09-13T11:25:54.516 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:54.516 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:54.516 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:25:54.516 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.650034+0000", 2024-09-13T11:25:54.516 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.287791662 2024-09-13T11:25:54.516 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:54.516 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:54.516 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:25:54.516 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.650077+0000", 2024-09-13T11:25:54.516 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.2367999999999998e-05 2024-09-13T11:25:54.516 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:54.517 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:54.517 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:25:54.517 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.650113+0000", 2024-09-13T11:25:54.517 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.6678999999999999e-05 2024-09-13T11:25:54.517 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:54.517 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:25:54.517 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:54.517 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:54.517 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:54.517 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:110279 3.14 3:2b17a349:::benchmark_data_smithi177_31116_object110278:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119)", 2024-09-13T11:25:54.517 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:25.645137+0000", 2024-09-13T11:25:54.517 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 88.811896634999997, 2024-09-13T11:25:54.517 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.28172927800000003, 2024-09-13T11:25:54.518 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:25:54.518 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:25:54.518 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:54.518 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:25:54.518 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:25:54.518 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:54.518 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 110279 2024-09-13T11:25:54.518 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:54.518 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:25:54.518 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:54.518 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:25:54.518 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.645137+0000", 2024-09-13T11:25:54.518 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:54.518 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:54.519 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:54.519 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:25:54.519 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.645137+0000", 2024-09-13T11:25:54.519 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:54.519 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:54.519 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:54.519 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:25:54.519 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.645140+0000", 2024-09-13T11:25:54.519 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.0939999999999999e-06 2024-09-13T11:25:54.519 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:54.519 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:54.519 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:25:54.519 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.645184+0000", 2024-09-13T11:25:54.519 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.3412999999999999e-05 2024-09-13T11:25:54.520 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:54.520 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:54.520 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:25:54.520 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.645206+0000", 2024-09-13T11:25:54.520 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.1997e-05 2024-09-13T11:25:54.520 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:54.520 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:54.520 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:25:54.520 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.645216+0000", 2024-09-13T11:25:54.520 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0740999999999999e-05 2024-09-13T11:25:54.520 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:54.520 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:54.520 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:25:54.521 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.645397+0000", 2024-09-13T11:25:54.521 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00018077600000000001 2024-09-13T11:25:54.521 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:54.521 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:54.521 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:25:54.521 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.645638+0000", 2024-09-13T11:25:54.521 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000241091 2024-09-13T11:25:54.521 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:54.521 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:54.521 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 14", 2024-09-13T11:25:54.521 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.645922+0000", 2024-09-13T11:25:54.521 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000283565 2024-09-13T11:25:54.521 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:54.522 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:54.522 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:25:54.522 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.648980+0000", 2024-09-13T11:25:54.522 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00305851 2024-09-13T11:25:54.522 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:54.522 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:54.522 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:25:54.522 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.926702+0000", 2024-09-13T11:25:54.522 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.27772199800000003 2024-09-13T11:25:54.522 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:54.522 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:54.522 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:25:54.522 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.926796+0000", 2024-09-13T11:25:54.522 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.4158000000000002e-05 2024-09-13T11:25:54.523 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:54.523 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:54.523 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:25:54.523 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.926866+0000", 2024-09-13T11:25:54.523 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.9935e-05 2024-09-13T11:25:54.523 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:54.523 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:25:54.523 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:54.523 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:54.523 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:54.523 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:129865 3.14 3:2c872f8e:::benchmark_data_smithi177_31116_object129864:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e134)", 2024-09-13T11:25:54.523 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:47.107186+0000", 2024-09-13T11:25:54.523 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 67.349848077999994, 2024-09-13T11:25:54.523 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.21814059299999999, 2024-09-13T11:25:54.524 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:25:54.524 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:25:54.524 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:54.524 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:25:54.524 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:25:54.524 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:54.524 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 129865 2024-09-13T11:25:54.524 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:54.524 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:25:54.524 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:54.524 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:25:54.524 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:47.107186+0000", 2024-09-13T11:25:54.524 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:54.524 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:54.525 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:54.525 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:25:54.525 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:47.107186+0000", 2024-09-13T11:25:54.525 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:54.525 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:54.525 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:54.525 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:25:54.525 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:47.107187+0000", 2024-09-13T11:25:54.525 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5740000000000001e-06 2024-09-13T11:25:54.525 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:54.525 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:54.525 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:25:54.525 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:47.107207+0000", 2024-09-13T11:25:54.525 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.9304999999999999e-05 2024-09-13T11:25:54.526 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:54.526 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:54.526 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:25:54.526 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:47.107215+0000", 2024-09-13T11:25:54.526 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.7390000000000005e-06 2024-09-13T11:25:54.526 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:54.526 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:54.526 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:25:54.526 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:47.107221+0000", 2024-09-13T11:25:54.526 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.7350000000000001e-06 2024-09-13T11:25:54.526 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:54.526 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:54.526 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:25:54.527 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:47.107278+0000", 2024-09-13T11:25:54.527 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.7209000000000001e-05 2024-09-13T11:25:54.527 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:54.527 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:54.527 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:25:54.527 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:47.107383+0000", 2024-09-13T11:25:54.527 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000104718 2024-09-13T11:25:54.527 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:54.527 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:54.527 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 15", 2024-09-13T11:25:54.527 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:47.107480+0000", 2024-09-13T11:25:54.527 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.7343000000000006e-05 2024-09-13T11:25:54.527 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:54.527 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:54.528 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:25:54.528 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:47.109477+0000", 2024-09-13T11:25:54.528 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0019968490000000002 2024-09-13T11:25:54.528 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:54.528 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:54.528 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:25:54.528 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:47.324147+0000", 2024-09-13T11:25:54.528 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.21467018199999999 2024-09-13T11:25:54.528 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:54.528 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:54.528 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:25:54.528 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:47.324281+0000", 2024-09-13T11:25:54.528 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00013328600000000001 2024-09-13T11:25:54.529 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:54.529 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:54.529 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:25:54.529 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:47.325326+0000", 2024-09-13T11:25:54.529 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0010456529999999999 2024-09-13T11:25:54.529 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:54.529 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:25:54.529 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:54.529 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:54.529 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:54.529 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPush(3.15 140/122 [PushOp(3:ae90e82d:::benchmark_data_smithi177_31116_object53962:head, version: 74'3006, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:ae90e82d:::benchmark_data_smithi177_31116_object53962:head@74'3006, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:25:54.529 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:54.310786+0000", 2024-09-13T11:25:54.529 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 60.146247604999999, 2024-09-13T11:25:54.529 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.229345255, 2024-09-13T11:25:54.530 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:25:54.530 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:25:54.530 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:25:54.530 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:25:54.530 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:54.530 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:25:54.530 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:54.310786+0000", 2024-09-13T11:25:54.530 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:54.530 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:54.530 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:54.530 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:25:54.530 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:54.310786+0000", 2024-09-13T11:25:54.530 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:54.531 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:54.531 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:54.531 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:25:54.531 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:54.310795+0000", 2024-09-13T11:25:54.531 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.1910000000000006e-06 2024-09-13T11:25:54.531 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:54.531 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:54.531 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:25:54.531 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:54.310829+0000", 2024-09-13T11:25:54.531 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.4121000000000001e-05 2024-09-13T11:25:54.531 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:54.531 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:54.531 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:25:54.532 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:54.538419+0000", 2024-09-13T11:25:54.532 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.227589611 2024-09-13T11:25:54.532 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:54.532 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:54.532 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:25:54.532 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:54.538430+0000", 2024-09-13T11:25:54.532 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0456e-05 2024-09-13T11:25:54.532 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:54.532 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:54.532 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:25:54.532 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:54.538497+0000", 2024-09-13T11:25:54.532 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.6948999999999999e-05 2024-09-13T11:25:54.532 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:54.533 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:54.533 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:25:54.533 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:54.538505+0000", 2024-09-13T11:25:54.533 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.2989999999999997e-06 2024-09-13T11:25:54.533 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:54.533 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:54.533 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:25:54.533 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:54.540131+0000", 2024-09-13T11:25:54.533 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001626628 2024-09-13T11:25:54.533 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:54.533 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:25:54.533 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:54.533 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:54.533 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:54.534 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPush(3.15 143/122 [PushOp(3:ae52bb45:::benchmark_data_smithi177_31116_object61893:head, version: 87'3272, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:ae52bb45:::benchmark_data_smithi177_31116_object61893:head@87'3272, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:25:54.534 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:57.633939+0000", 2024-09-13T11:25:54.534 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 56.823094652000002, 2024-09-13T11:25:54.534 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.339405067, 2024-09-13T11:25:54.534 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:25:54.534 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:25:54.534 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:25:54.534 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:25:54.534 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:54.534 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:25:54.534 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:57.633939+0000", 2024-09-13T11:25:54.534 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:54.534 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:54.535 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:54.535 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:25:54.535 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:57.633939+0000", 2024-09-13T11:25:54.535 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:54.535 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:54.535 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:54.535 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:25:54.535 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:57.633941+0000", 2024-09-13T11:25:54.535 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.911e-06 2024-09-13T11:25:54.535 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:54.535 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:54.535 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:25:54.536 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:57.633994+0000", 2024-09-13T11:25:54.536 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.2753999999999997e-05 2024-09-13T11:25:54.536 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:54.536 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:54.536 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:25:54.536 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:57.968403+0000", 2024-09-13T11:25:54.536 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.334409336 2024-09-13T11:25:54.536 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:54.536 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:54.536 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:25:54.536 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:57.968451+0000", 2024-09-13T11:25:54.536 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.7525000000000003e-05 2024-09-13T11:25:54.537 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:54.537 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:54.537 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:25:54.537 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:57.968660+0000", 2024-09-13T11:25:54.537 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000209204 2024-09-13T11:25:54.537 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:54.537 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:54.537 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:25:54.537 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:57.968680+0000", 2024-09-13T11:25:54.537 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.9639999999999998e-05 2024-09-13T11:25:54.537 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:54.537 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:54.537 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:25:54.537 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:57.973344+0000", 2024-09-13T11:25:54.538 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0046646969999999998 2024-09-13T11:25:54.538 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:54.538 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:25:54.538 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:54.538 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:54.538 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:54.538 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:140658 3.17 e144/85 3:ece5a1b3:::benchmark_data_smithi177_31116_object355:head v 144'5654, mlcod=144'5652)", 2024-09-13T11:25:54.538 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:58.247792+0000", 2024-09-13T11:25:54.538 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 56.209242174000003, 2024-09-13T11:25:54.538 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.29148369699999999, 2024-09-13T11:25:54.538 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:25:54.538 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:25:54.538 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:54.538 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:25:54.539 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:54.539 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:25:54.539 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:58.247792+0000", 2024-09-13T11:25:54.539 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:54.539 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:54.539 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:54.539 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:25:54.539 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:58.247792+0000", 2024-09-13T11:25:54.539 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:54.539 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:54.539 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:54.539 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:25:54.539 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:58.247793+0000", 2024-09-13T11:25:54.539 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.347e-06 2024-09-13T11:25:54.540 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:54.540 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:54.540 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:25:54.540 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:58.247799+0000", 2024-09-13T11:25:54.540 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.5239999999999997e-06 2024-09-13T11:25:54.540 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:54.540 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:54.540 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:25:54.540 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:58.535226+0000", 2024-09-13T11:25:54.540 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.28742604900000002 2024-09-13T11:25:54.540 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:54.540 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:54.540 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:25:54.540 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:58.535238+0000", 2024-09-13T11:25:54.541 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.2843000000000001e-05 2024-09-13T11:25:54.541 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:54.541 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:54.541 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:25:54.541 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:58.535393+0000", 2024-09-13T11:25:54.541 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00015481699999999999 2024-09-13T11:25:54.541 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:54.541 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:54.541 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:25:54.541 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:58.535434+0000", 2024-09-13T11:25:54.541 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.1152999999999997e-05 2024-09-13T11:25:54.541 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:54.541 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:54.542 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:25:54.542 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:58.539191+0000", 2024-09-13T11:25:54.542 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0037571520000000001 2024-09-13T11:25:54.542 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:54.542 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:54.542 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:25:54.542 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:58.539275+0000", 2024-09-13T11:25:54.542 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.3812000000000003e-05 2024-09-13T11:25:54.542 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:54.542 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:25:54.542 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:54.542 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:54.542 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:54.542 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPush(3.15 144/122 [PushOp(3:a97ba025:::benchmark_data_smithi177_31116_object64775:head, version: 90'3365, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:a97ba025:::benchmark_data_smithi177_31116_object64775:head@90'3365, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:25:54.543 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:58.825014+0000", 2024-09-13T11:25:54.543 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 55.632019546000002, 2024-09-13T11:25:54.543 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.23868276499999999, 2024-09-13T11:25:54.543 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:25:54.543 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:25:54.543 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:25:54.543 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:25:54.543 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:54.543 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:25:54.543 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:58.825014+0000", 2024-09-13T11:25:54.543 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:54.543 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:54.543 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:54.543 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:25:54.544 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:58.825014+0000", 2024-09-13T11:25:54.544 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:54.544 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:54.544 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:54.544 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:25:54.544 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:58.825017+0000", 2024-09-13T11:25:54.544 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.7630000000000001e-06 2024-09-13T11:25:54.544 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:54.544 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:54.544 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:25:54.544 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:58.825099+0000", 2024-09-13T11:25:54.544 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.2035999999999999e-05 2024-09-13T11:25:54.544 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:54.544 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:54.545 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:25:54.545 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:59.059557+0000", 2024-09-13T11:25:54.545 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.23445790399999999 2024-09-13T11:25:54.545 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:54.545 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:54.545 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:25:54.545 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:59.059609+0000", 2024-09-13T11:25:54.545 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.2550999999999999e-05 2024-09-13T11:25:54.545 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:54.545 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:54.545 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:25:54.545 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:59.060117+0000", 2024-09-13T11:25:54.545 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00050754299999999995 2024-09-13T11:25:54.545 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:54.546 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:54.546 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:25:54.546 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:59.060139+0000", 2024-09-13T11:25:54.546 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.2306e-05 2024-09-13T11:25:54.546 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:54.546 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:54.546 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:25:54.546 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:59.063697+0000", 2024-09-13T11:25:54.546 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0035576620000000001 2024-09-13T11:25:54.546 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:54.546 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:25:54.546 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:54.546 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:54.546 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:54.547 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:147472 3.2d 3:b6aeefb0:::benchmark_data_smithi177_31116_object7169:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e148)", 2024-09-13T11:25:54.547 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:25:04.123331+0000", 2024-09-13T11:25:54.547 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 50.333703284999999, 2024-09-13T11:25:54.547 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.23047060799999999, 2024-09-13T11:25:54.547 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:25:54.547 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:25:54.547 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:54.547 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:25:54.547 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:25:54.547 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:54.547 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 147472 2024-09-13T11:25:54.547 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:54.547 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:25:54.548 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:54.548 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:25:54.548 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:04.123331+0000", 2024-09-13T11:25:54.548 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:54.548 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:54.548 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:54.548 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:25:54.548 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:04.123331+0000", 2024-09-13T11:25:54.548 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:54.548 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:54.548 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:54.548 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:25:54.548 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:04.123332+0000", 2024-09-13T11:25:54.548 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.556e-06 2024-09-13T11:25:54.549 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:54.549 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:54.549 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:25:54.549 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:04.123337+0000", 2024-09-13T11:25:54.549 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.5889999999999996e-06 2024-09-13T11:25:54.549 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:54.549 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:54.549 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:25:54.549 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:04.123344+0000", 2024-09-13T11:25:54.549 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.7889999999999995e-06 2024-09-13T11:25:54.549 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:54.549 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:54.549 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:25:54.550 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:04.123350+0000", 2024-09-13T11:25:54.550 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.3820000000000003e-06 2024-09-13T11:25:54.550 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:54.550 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:54.550 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:25:54.550 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:04.123402+0000", 2024-09-13T11:25:54.550 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.2281999999999999e-05 2024-09-13T11:25:54.550 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:54.550 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:54.550 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for degraded object", 2024-09-13T11:25:54.550 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:04.124030+0000", 2024-09-13T11:25:54.550 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00062812999999999996 2024-09-13T11:25:54.550 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:54.551 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:54.551 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:25:54.551 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:04.137378+0000", 2024-09-13T11:25:54.551 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.013347816 2024-09-13T11:25:54.551 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:54.551 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:54.551 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:25:54.551 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:04.137472+0000", 2024-09-13T11:25:54.551 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.3777999999999996e-05 2024-09-13T11:25:54.551 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:54.551 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:54.551 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 0,5", 2024-09-13T11:25:54.551 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:04.137594+0000", 2024-09-13T11:25:54.551 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00012196099999999999 2024-09-13T11:25:54.551 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:54.552 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:54.552 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:25:54.552 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:04.139184+0000", 2024-09-13T11:25:54.552 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001590508 2024-09-13T11:25:54.552 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:54.552 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:54.552 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:25:54.552 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:04.143627+0000", 2024-09-13T11:25:54.552 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0044427989999999999 2024-09-13T11:25:54.552 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:54.552 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:54.552 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:25:54.552 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:04.352741+0000", 2024-09-13T11:25:54.553 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.209114142 2024-09-13T11:25:54.553 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:54.553 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:54.553 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:25:54.553 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:04.352808+0000", 2024-09-13T11:25:54.553 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.7034999999999998e-05 2024-09-13T11:25:54.553 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:54.553 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:54.553 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:25:54.553 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:04.353801+0000", 2024-09-13T11:25:54.553 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00099284099999999995 2024-09-13T11:25:54.553 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:54.553 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:25:54.553 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:54.554 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:54.554 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:25:54.554 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-09-13T11:25:54.555 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-13T11:25:54.558 INFO:tasks.ceph.osd.1.smithi092.stderr:2024-09-13T11:25:54.556+0000 7fe7789c5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33718) UID: 0 2024-09-13T11:25:54.658 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-13T11:25:54.654+0000 7f6a68775640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26307) UID: 0 2024-09-13T11:25:54.739 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-09-13T11:25:54.739 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [ 2024-09-13T11:25:54.739 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:54.739 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:210322 3.2b e185/119 3:d5358d4d:::benchmark_data_smithi177_31116_object70019:head v 185'6625, mlcod=185'6623)", 2024-09-13T11:25:54.739 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:25:54.738110+0000", 2024-09-13T11:25:54.739 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 0.001474279, 2024-09-13T11:25:54.739 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0014911620000000001, 2024-09-13T11:25:54.739 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:25:54.739 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:25:54.739 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:25:54.740 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:25:54.740 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:54.740 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:25:54.740 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:54.738110+0000", 2024-09-13T11:25:54.740 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:54.740 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:54.740 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:54.740 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:25:54.740 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:54.738110+0000", 2024-09-13T11:25:54.740 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:54.740 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:54.740 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:54.741 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:25:54.741 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:54.738111+0000", 2024-09-13T11:25:54.741 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.342e-06 2024-09-13T11:25:54.741 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:54.741 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:54.741 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:25:54.741 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:54.738117+0000", 2024-09-13T11:25:54.741 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.6219999999999997e-06 2024-09-13T11:25:54.741 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:54.741 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:54.741 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:25:54.741 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:54.738142+0000", 2024-09-13T11:25:54.741 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.5221000000000001e-05 2024-09-13T11:25:54.741 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:54.742 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:54.742 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:25:54.742 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:54.738147+0000", 2024-09-13T11:25:54.742 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.4639999999999997e-06 2024-09-13T11:25:54.742 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:54.742 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:54.742 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:25:54.742 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:54.738209+0000", 2024-09-13T11:25:54.742 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.1543000000000004e-05 2024-09-13T11:25:54.742 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:54.742 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:54.742 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:25:54.742 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:54.738232+0000", 2024-09-13T11:25:54.743 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.3164999999999999e-05 2024-09-13T11:25:54.743 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:54.743 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:25:54.743 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:54.743 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:54.743 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:54.743 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:210323 3.2c e185/144 3:37499ea1:::benchmark_data_smithi177_31116_object70020:head v 185'6759, mlcod=185'6757)", 2024-09-13T11:25:54.743 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:25:54.738279+0000", 2024-09-13T11:25:54.743 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 0.001305241, 2024-09-13T11:25:54.743 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0013555520000000001, 2024-09-13T11:25:54.743 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:25:54.743 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:25:54.743 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:25:54.744 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:25:54.744 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:54.744 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:25:54.744 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:54.738279+0000", 2024-09-13T11:25:54.744 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:54.744 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:54.744 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:54.744 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:25:54.744 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:54.738279+0000", 2024-09-13T11:25:54.744 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:54.744 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:54.744 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:54.744 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:25:54.745 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:54.738280+0000", 2024-09-13T11:25:54.745 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.801e-06 2024-09-13T11:25:54.745 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:54.745 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:54.745 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:25:54.745 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:54.738291+0000", 2024-09-13T11:25:54.745 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0147e-05 2024-09-13T11:25:54.745 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:54.745 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:54.745 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:25:54.745 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:54.738321+0000", 2024-09-13T11:25:54.745 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.0616e-05 2024-09-13T11:25:54.745 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:54.745 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:54.746 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:25:54.746 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:54.738328+0000", 2024-09-13T11:25:54.746 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.9979999999999997e-06 2024-09-13T11:25:54.746 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:54.746 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:54.746 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:25:54.746 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:54.738383+0000", 2024-09-13T11:25:54.746 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.5071999999999997e-05 2024-09-13T11:25:54.746 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:54.746 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:54.746 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:25:54.746 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:54.738398+0000", 2024-09-13T11:25:54.746 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.4815e-05 2024-09-13T11:25:54.747 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:54.747 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:25:54.747 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:54.747 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:54.747 INFO:teuthology.orchestra.run.smithi177.stdout: ], 2024-09-13T11:25:54.747 INFO:teuthology.orchestra.run.smithi177.stdout: "num_ops": 2 2024-09-13T11:25:54.747 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-09-13T11:25:54.760 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-13T11:25:54.761 INFO:tasks.ceph.osd.14.smithi162.stderr:2024-09-13T11:25:54.755+0000 7f3df5903640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33154) UID: 0 2024-09-13T11:25:54.861 INFO:tasks.ceph.osd.12.smithi001.stderr:2024-09-13T11:25:54.858+0000 7f892c6e6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26314) UID: 0 2024-09-13T11:25:54.895 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-09-13T11:25:54.896 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [], 2024-09-13T11:25:54.896 INFO:teuthology.orchestra.run.smithi177.stdout: "complaint_time": 30, 2024-09-13T11:25:54.896 INFO:teuthology.orchestra.run.smithi177.stdout: "num_blocked_ops": 0 2024-09-13T11:25:54.896 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-09-13T11:25:54.913 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-13T11:25:54.963 INFO:tasks.ceph.osd.15.smithi177.stderr:2024-09-13T11:25:54.962+0000 7f9d6ce77640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26578) UID: 0 2024-09-13T11:25:54.969 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-13T11:25:55.053 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-09-13T11:25:55.054 INFO:teuthology.orchestra.run.smithi177.stdout: "size": 20, 2024-09-13T11:25:55.054 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 600, 2024-09-13T11:25:55.054 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [ 2024-09-13T11:25:55.054 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.054 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPushReply(3.13 75/70 [PushReplyOp(3:cfab03d6:::benchmark_data_smithi177_31116_object16599:head)])", 2024-09-13T11:25:55.054 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:31.163480+0000", 2024-09-13T11:25:55.054 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 143.88930531899999, 2024-09-13T11:25:55.054 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.63009029299999997, 2024-09-13T11:25:55.055 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:25:55.055 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:25:55.055 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-13T11:25:55.055 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:25:55.055 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.055 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:25:55.055 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.163480+0000", 2024-09-13T11:25:55.055 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:55.055 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.055 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.055 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:25:55.055 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.163480+0000", 2024-09-13T11:25:55.056 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:55.056 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.056 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.056 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:25:55.056 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.163481+0000", 2024-09-13T11:25:55.056 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0410000000000001e-06 2024-09-13T11:25:55.056 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.056 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.056 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:25:55.056 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.163485+0000", 2024-09-13T11:25:55.056 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.1629999999999998e-06 2024-09-13T11:25:55.056 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.057 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.057 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:25:55.057 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.792166+0000", 2024-09-13T11:25:55.057 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.62868067400000005 2024-09-13T11:25:55.057 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.057 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.057 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:25:55.057 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.792184+0000", 2024-09-13T11:25:55.057 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.7898999999999999e-05 2024-09-13T11:25:55.058 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.058 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.058 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:25:55.058 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.792312+0000", 2024-09-13T11:25:55.058 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00012871000000000001 2024-09-13T11:25:55.058 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.058 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.058 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:25:55.058 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.793570+0000", 2024-09-13T11:25:55.058 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001257806 2024-09-13T11:25:55.058 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:55.058 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:25:55.059 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:55.059 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.059 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.059 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPush(3.12 75/69 [PushOp(3:4a36ca6d:::benchmark_data_smithi177_31116_object31707:head, version: 59'1993, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:4a36ca6d:::benchmark_data_smithi177_31116_object31707:head@59'1993, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:25:55.059 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:31.309623+0000", 2024-09-13T11:25:55.059 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 143.743162364, 2024-09-13T11:25:55.059 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.165459152, 2024-09-13T11:25:55.059 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:25:55.059 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:25:55.059 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:25:55.059 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:25:55.059 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.060 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:25:55.060 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.309623+0000", 2024-09-13T11:25:55.060 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:55.060 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.060 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.060 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:25:55.060 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.309623+0000", 2024-09-13T11:25:55.060 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:55.060 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.060 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.060 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:25:55.060 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.309625+0000", 2024-09-13T11:25:55.061 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.3860000000000001e-06 2024-09-13T11:25:55.061 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.061 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.061 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:25:55.061 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.309680+0000", 2024-09-13T11:25:55.061 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.4388000000000002e-05 2024-09-13T11:25:55.061 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.061 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.061 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:25:55.061 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.473157+0000", 2024-09-13T11:25:55.061 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.163476813 2024-09-13T11:25:55.061 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.061 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.062 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:25:55.062 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.473189+0000", 2024-09-13T11:25:55.062 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.2975000000000002e-05 2024-09-13T11:25:55.062 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.062 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.062 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:25:55.062 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.473395+0000", 2024-09-13T11:25:55.062 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000206009 2024-09-13T11:25:55.062 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.062 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.062 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:25:55.062 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.473424+0000", 2024-09-13T11:25:55.062 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.8255e-05 2024-09-13T11:25:55.063 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.063 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.063 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:25:55.063 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.475082+0000", 2024-09-13T11:25:55.063 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0016583259999999999 2024-09-13T11:25:55.063 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:55.063 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:25:55.063 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:55.063 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.063 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.063 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPushReply(3.13 76/70 [PushReplyOp(3:c9d43a82:::benchmark_data_smithi177_31116_object20026:head)])", 2024-09-13T11:25:55.063 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:32.427709+0000", 2024-09-13T11:25:55.064 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 142.62507580400001, 2024-09-13T11:25:55.064 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.46547316300000002, 2024-09-13T11:25:55.064 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:25:55.064 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:25:55.064 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-13T11:25:55.064 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:25:55.064 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.064 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:25:55.064 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:32.427709+0000", 2024-09-13T11:25:55.064 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:55.064 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.064 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.065 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:25:55.065 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:32.427709+0000", 2024-09-13T11:25:55.065 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:55.065 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.065 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.065 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:25:55.065 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:32.427711+0000", 2024-09-13T11:25:55.065 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5099999999999999e-06 2024-09-13T11:25:55.065 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.065 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.065 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:25:55.065 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:32.427717+0000", 2024-09-13T11:25:55.065 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.1569999999999997e-06 2024-09-13T11:25:55.066 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.066 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.066 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:25:55.066 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:32.892390+0000", 2024-09-13T11:25:55.066 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.46467251100000001 2024-09-13T11:25:55.066 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.066 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.066 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:25:55.066 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:32.892405+0000", 2024-09-13T11:25:55.066 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5296999999999999e-05 2024-09-13T11:25:55.066 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.066 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.066 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:25:55.067 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:32.892503+0000", 2024-09-13T11:25:55.067 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.7820999999999997e-05 2024-09-13T11:25:55.067 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.067 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.067 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:25:55.067 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:32.893183+0000", 2024-09-13T11:25:55.067 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00067986700000000001 2024-09-13T11:25:55.067 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:55.067 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:25:55.067 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:55.067 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.067 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.067 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:57435 3.3 e80/75 3:c0db4777:::benchmark_data_smithi177_31116_object57434:head v 80'3155, mlcod=80'3153)", 2024-09-13T11:25:55.068 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:37.437070+0000", 2024-09-13T11:25:55.068 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 137.61571498699999, 2024-09-13T11:25:55.068 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.35861845999999997, 2024-09-13T11:25:55.068 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:25:55.068 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:25:55.068 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:55.068 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:25:55.068 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.068 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:25:55.068 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:37.437070+0000", 2024-09-13T11:25:55.069 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:55.069 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.069 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.069 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:25:55.069 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:37.437070+0000", 2024-09-13T11:25:55.069 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:55.069 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.069 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.069 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:25:55.069 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:37.437072+0000", 2024-09-13T11:25:55.069 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5549999999999999e-06 2024-09-13T11:25:55.070 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.070 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.070 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:25:55.070 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:37.437141+0000", 2024-09-13T11:25:55.070 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.9494999999999998e-05 2024-09-13T11:25:55.070 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.070 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.070 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:25:55.070 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:37.793400+0000", 2024-09-13T11:25:55.070 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.35625817999999998 2024-09-13T11:25:55.070 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.070 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.070 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:25:55.071 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:37.793406+0000", 2024-09-13T11:25:55.071 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.6830000000000002e-06 2024-09-13T11:25:55.071 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.071 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.071 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:25:55.071 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:37.793472+0000", 2024-09-13T11:25:55.071 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.5671000000000002e-05 2024-09-13T11:25:55.071 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.071 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.071 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:25:55.072 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:37.793492+0000", 2024-09-13T11:25:55.072 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.0051000000000002e-05 2024-09-13T11:25:55.072 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.072 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.072 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:25:55.072 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:37.795626+0000", 2024-09-13T11:25:55.072 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0021344990000000002 2024-09-13T11:25:55.072 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.072 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.072 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:25:55.073 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:37.795689+0000", 2024-09-13T11:25:55.073 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.2325999999999999e-05 2024-09-13T11:25:55.073 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:55.073 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:25:55.073 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:55.073 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.073 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.073 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:57985 3.3 e80/75 3:c330bb03:::benchmark_data_smithi177_31116_object57984:head v 80'3162, mlcod=80'3159)", 2024-09-13T11:25:55.073 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:38.007076+0000", 2024-09-13T11:25:55.073 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 137.04570963500001, 2024-09-13T11:25:55.073 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.18211438199999999, 2024-09-13T11:25:55.073 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:25:55.073 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:25:55.073 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:55.074 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:25:55.074 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.074 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:25:55.074 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:38.007076+0000", 2024-09-13T11:25:55.074 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:55.074 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.074 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.074 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:25:55.074 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:38.007076+0000", 2024-09-13T11:25:55.074 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:55.074 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.074 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.074 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:25:55.074 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:38.007077+0000", 2024-09-13T11:25:55.075 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.6759999999999999e-06 2024-09-13T11:25:55.075 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.075 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.075 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:25:55.075 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:38.007221+0000", 2024-09-13T11:25:55.075 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00014344299999999999 2024-09-13T11:25:55.075 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.075 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.075 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:25:55.075 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:38.186808+0000", 2024-09-13T11:25:55.075 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.17958726699999999 2024-09-13T11:25:55.075 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.075 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.076 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:25:55.076 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:38.186815+0000", 2024-09-13T11:25:55.076 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.0890000000000002e-06 2024-09-13T11:25:55.076 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.076 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.076 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:25:55.076 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:38.186887+0000", 2024-09-13T11:25:55.076 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.2006000000000001e-05 2024-09-13T11:25:55.076 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.076 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.077 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:25:55.077 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:38.186913+0000", 2024-09-13T11:25:55.077 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.5962e-05 2024-09-13T11:25:55.077 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.077 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.077 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:25:55.077 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:38.189123+0000", 2024-09-13T11:25:55.077 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0022099649999999999 2024-09-13T11:25:55.077 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.077 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.077 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:25:55.077 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:38.189190+0000", 2024-09-13T11:25:55.078 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.6973999999999999e-05 2024-09-13T11:25:55.078 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:55.078 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:25:55.078 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:55.078 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.078 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.078 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPushReply(3.23 84/78 [PushReplyOp(3:c7e44616:::benchmark_data_smithi177_31116_object32371:head)])", 2024-09-13T11:25:55.078 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:41.338210+0000", 2024-09-13T11:25:55.078 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 133.714574916, 2024-09-13T11:25:55.078 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.41118492299999998, 2024-09-13T11:25:55.078 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:25:55.078 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:25:55.078 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-13T11:25:55.079 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:25:55.079 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.079 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:25:55.079 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.338210+0000", 2024-09-13T11:25:55.079 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:55.079 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.079 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.079 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:25:55.079 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.338210+0000", 2024-09-13T11:25:55.079 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:55.079 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.079 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.080 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:25:55.080 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.338212+0000", 2024-09-13T11:25:55.080 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.314e-06 2024-09-13T11:25:55.080 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.080 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.080 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:25:55.080 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.338219+0000", 2024-09-13T11:25:55.080 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.6469999999999998e-06 2024-09-13T11:25:55.080 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.080 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.080 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:25:55.080 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.748675+0000", 2024-09-13T11:25:55.081 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.41045567500000002 2024-09-13T11:25:55.081 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.081 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.081 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:25:55.081 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.748684+0000", 2024-09-13T11:25:55.081 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.2150000000000007e-06 2024-09-13T11:25:55.081 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.081 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.081 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:25:55.081 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.748739+0000", 2024-09-13T11:25:55.081 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.4543000000000003e-05 2024-09-13T11:25:55.081 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.082 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.082 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:25:55.082 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.749395+0000", 2024-09-13T11:25:55.082 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00065652899999999999 2024-09-13T11:25:55.082 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:55.082 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:25:55.082 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:55.082 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.082 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.082 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPushReply(3.23 85/78 [PushReplyOp(3:c5c6dce1:::benchmark_data_smithi177_31116_object37293:head)])", 2024-09-13T11:25:55.082 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:42.377862+0000", 2024-09-13T11:25:55.082 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 132.67492345700001, 2024-09-13T11:25:55.082 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.62538808599999995, 2024-09-13T11:25:55.083 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:25:55.083 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:25:55.083 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-13T11:25:55.083 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:25:55.083 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.083 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:25:55.083 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.377862+0000", 2024-09-13T11:25:55.083 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:55.083 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.083 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.083 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:25:55.083 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.377862+0000", 2024-09-13T11:25:55.083 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:55.084 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.084 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.084 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:25:55.084 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.377863+0000", 2024-09-13T11:25:55.084 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.6899999999999996e-07 2024-09-13T11:25:55.084 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.084 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.084 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:25:55.084 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.377866+0000", 2024-09-13T11:25:55.084 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.219e-06 2024-09-13T11:25:55.084 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.084 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.084 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:25:55.085 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:43.001857+0000", 2024-09-13T11:25:55.085 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.62399084299999996 2024-09-13T11:25:55.085 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.085 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.085 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:25:55.085 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:43.001869+0000", 2024-09-13T11:25:55.085 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.2415999999999999e-05 2024-09-13T11:25:55.085 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.085 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.085 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:25:55.085 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:43.002194+0000", 2024-09-13T11:25:55.085 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000324774 2024-09-13T11:25:55.085 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.086 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.086 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:25:55.086 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:43.003250+0000", 2024-09-13T11:25:55.086 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001055865 2024-09-13T11:25:55.086 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:55.086 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:25:55.086 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:55.086 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.086 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.086 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPush(3.1c 89/77 [PushOp(3:38bcbc71:::benchmark_data_smithi177_31116_object31448:head, version: 59'1946, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:38bcbc71:::benchmark_data_smithi177_31116_object31448:head@59'1946, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:25:55.086 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:46.670723+0000", 2024-09-13T11:25:55.086 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 128.382062233, 2024-09-13T11:25:55.086 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.27576065, 2024-09-13T11:25:55.087 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:25:55.087 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:25:55.087 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:25:55.087 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:25:55.087 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.087 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:25:55.087 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.670723+0000", 2024-09-13T11:25:55.087 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:55.087 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.087 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.087 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:25:55.087 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.670723+0000", 2024-09-13T11:25:55.088 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:55.088 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.088 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.088 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:25:55.088 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.670724+0000", 2024-09-13T11:25:55.088 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.4389999999999999e-06 2024-09-13T11:25:55.088 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.088 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.088 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:25:55.088 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.670754+0000", 2024-09-13T11:25:55.088 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.9224999999999999e-05 2024-09-13T11:25:55.088 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.088 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.089 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:25:55.089 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.944369+0000", 2024-09-13T11:25:55.089 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.27361529800000001 2024-09-13T11:25:55.089 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.089 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.089 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:25:55.089 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.944422+0000", 2024-09-13T11:25:55.089 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.3318e-05 2024-09-13T11:25:55.089 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.089 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.089 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:25:55.089 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.944711+0000", 2024-09-13T11:25:55.089 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00028912399999999998 2024-09-13T11:25:55.089 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.090 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.090 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:25:55.090 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.944735+0000", 2024-09-13T11:25:55.090 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.3366999999999998e-05 2024-09-13T11:25:55.090 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.090 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.090 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:25:55.090 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.946484+0000", 2024-09-13T11:25:55.090 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0017488790000000001 2024-09-13T11:25:55.090 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:55.090 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:25:55.090 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:55.091 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.091 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.091 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPush(3.1c 91/77 [PushOp(3:3e18abc2:::benchmark_data_smithi177_31116_object35874:head, version: 62'2235, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:3e18abc2:::benchmark_data_smithi177_31116_object35874:head@62'2235, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:25:55.091 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:48.350968+0000", 2024-09-13T11:25:55.091 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 126.70181692, 2024-09-13T11:25:55.091 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.19257698000000001, 2024-09-13T11:25:55.091 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:25:55.091 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:25:55.091 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:25:55.091 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:25:55.091 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.091 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:25:55.092 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.350968+0000", 2024-09-13T11:25:55.092 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:55.092 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.092 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.092 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:25:55.092 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.350968+0000", 2024-09-13T11:25:55.092 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:55.092 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.092 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.092 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:25:55.092 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.350970+0000", 2024-09-13T11:25:55.092 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.796e-06 2024-09-13T11:25:55.092 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.093 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.093 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:25:55.093 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.351003+0000", 2024-09-13T11:25:55.093 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.2631e-05 2024-09-13T11:25:55.093 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.093 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.093 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:25:55.093 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.542140+0000", 2024-09-13T11:25:55.093 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.191136737 2024-09-13T11:25:55.093 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.093 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.093 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:25:55.093 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.542158+0000", 2024-09-13T11:25:55.093 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.8896999999999999e-05 2024-09-13T11:25:55.094 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.094 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.094 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:25:55.094 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.542263+0000", 2024-09-13T11:25:55.094 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000104158 2024-09-13T11:25:55.094 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.094 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.094 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:25:55.094 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.542279+0000", 2024-09-13T11:25:55.094 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5926999999999998e-05 2024-09-13T11:25:55.094 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.094 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.094 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:25:55.094 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.543545+0000", 2024-09-13T11:25:55.095 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0012668340000000001 2024-09-13T11:25:55.095 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:55.095 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:25:55.095 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:55.095 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.095 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.095 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:83211 3.1c e106/77 3:3ae31a0f:::benchmark_data_smithi177_31116_object83210:head v 106'4362, mlcod=106'4360)", 2024-09-13T11:25:55.095 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:04.345264+0000", 2024-09-13T11:25:55.095 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 110.70752126799999, 2024-09-13T11:25:55.095 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.17036547399999999, 2024-09-13T11:25:55.095 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:25:55.095 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:25:55.095 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:55.095 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:25:55.096 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.096 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:25:55.096 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.345264+0000", 2024-09-13T11:25:55.096 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:55.096 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.096 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.096 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:25:55.096 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.345264+0000", 2024-09-13T11:25:55.096 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:55.096 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.096 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.096 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:25:55.096 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.345267+0000", 2024-09-13T11:25:55.097 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.7590000000000002e-06 2024-09-13T11:25:55.097 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.097 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.097 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:25:55.097 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.345298+0000", 2024-09-13T11:25:55.097 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.1173000000000001e-05 2024-09-13T11:25:55.097 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.097 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.097 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:25:55.097 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.512265+0000", 2024-09-13T11:25:55.097 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.16696657100000001 2024-09-13T11:25:55.097 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.097 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.097 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:25:55.098 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.512279+0000", 2024-09-13T11:25:55.098 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.4953999999999999e-05 2024-09-13T11:25:55.098 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.098 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.098 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:25:55.098 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.512402+0000", 2024-09-13T11:25:55.098 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00012277999999999999 2024-09-13T11:25:55.098 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.098 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.098 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:25:55.098 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.512440+0000", 2024-09-13T11:25:55.098 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.7861999999999998e-05 2024-09-13T11:25:55.098 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.099 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.099 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:25:55.099 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.515515+0000", 2024-09-13T11:25:55.099 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00307472 2024-09-13T11:25:55.099 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.099 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.099 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:25:55.099 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.515629+0000", 2024-09-13T11:25:55.099 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00011465500000000001 2024-09-13T11:25:55.099 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:55.099 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:25:55.099 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:55.099 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.099 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.100 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:83223 3.1c e106/77 3:3bc6018d:::benchmark_data_smithi177_31116_object83222:head v 106'4363, mlcod=106'4360)", 2024-09-13T11:25:55.100 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:04.512139+0000", 2024-09-13T11:25:55.100 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 110.540646122, 2024-09-13T11:25:55.100 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.471923235, 2024-09-13T11:25:55.100 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:25:55.100 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:25:55.100 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:55.100 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:25:55.100 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.100 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:25:55.100 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.512139+0000", 2024-09-13T11:25:55.100 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:55.100 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.101 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.101 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:25:55.101 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.512139+0000", 2024-09-13T11:25:55.101 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:55.101 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.101 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.101 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:25:55.101 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.512143+0000", 2024-09-13T11:25:55.101 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.427e-06 2024-09-13T11:25:55.101 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.101 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.101 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:25:55.101 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.512227+0000", 2024-09-13T11:25:55.101 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.3965e-05 2024-09-13T11:25:55.102 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.102 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.102 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:25:55.102 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.976330+0000", 2024-09-13T11:25:55.102 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.46410342599999999 2024-09-13T11:25:55.102 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.102 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.102 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:25:55.102 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.976350+0000", 2024-09-13T11:25:55.102 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.9533000000000001e-05 2024-09-13T11:25:55.102 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.102 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.102 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:25:55.102 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.976488+0000", 2024-09-13T11:25:55.103 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00013799400000000001 2024-09-13T11:25:55.103 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.103 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.103 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:25:55.103 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.976543+0000", 2024-09-13T11:25:55.103 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.5442000000000002e-05 2024-09-13T11:25:55.103 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.103 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.103 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:25:55.103 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.983227+0000", 2024-09-13T11:25:55.103 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0066842899999999998 2024-09-13T11:25:55.103 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.103 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.103 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:25:55.104 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.984062+0000", 2024-09-13T11:25:55.104 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00083515800000000002 2024-09-13T11:25:55.104 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:55.104 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:25:55.104 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:55.104 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.104 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.104 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:110256 3.13 e119/83 3:cb00d078:::benchmark_data_smithi177_31116_object110255:head v 119'4823, mlcod=119'4821)", 2024-09-13T11:25:55.104 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:25.284530+0000", 2024-09-13T11:25:55.104 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 89.768255662000001, 2024-09-13T11:25:55.104 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.343491771, 2024-09-13T11:25:55.104 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:25:55.104 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:25:55.104 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:55.105 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:25:55.105 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.105 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:25:55.105 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.284530+0000", 2024-09-13T11:25:55.105 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:55.105 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.105 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.105 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:25:55.105 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.284530+0000", 2024-09-13T11:25:55.105 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:55.105 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.105 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.105 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:25:55.106 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.284532+0000", 2024-09-13T11:25:55.106 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.8870000000000001e-06 2024-09-13T11:25:55.106 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.106 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.106 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:25:55.106 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.284558+0000", 2024-09-13T11:25:55.106 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.6899000000000001e-05 2024-09-13T11:25:55.106 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.106 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.107 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:25:55.107 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.626172+0000", 2024-09-13T11:25:55.107 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.34161370800000002 2024-09-13T11:25:55.107 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.107 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.107 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:25:55.107 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.626187+0000", 2024-09-13T11:25:55.107 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.485e-05 2024-09-13T11:25:55.107 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.107 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.107 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:25:55.107 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.626245+0000", 2024-09-13T11:25:55.107 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.7998000000000003e-05 2024-09-13T11:25:55.108 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.108 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.108 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:25:55.108 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.626260+0000", 2024-09-13T11:25:55.108 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5429000000000001e-05 2024-09-13T11:25:55.108 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.108 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.108 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:25:55.108 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.627924+0000", 2024-09-13T11:25:55.109 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0016634060000000001 2024-09-13T11:25:55.109 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.109 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.109 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:25:55.109 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.628021+0000", 2024-09-13T11:25:55.109 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.7594000000000001e-05 2024-09-13T11:25:55.109 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:55.109 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:25:55.109 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:55.109 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.109 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.109 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPush(3.2b 121/119 [PushOp(3:d53a23e8:::benchmark_data_smithi177_31116_object5366:head, version: 45'342, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:d53a23e8:::benchmark_data_smithi177_31116_object5366:head@45'342, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:25:55.109 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:27.268731+0000", 2024-09-13T11:25:55.109 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 87.7840542, 2024-09-13T11:25:55.110 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.72228345000000005, 2024-09-13T11:25:55.110 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:25:55.110 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:25:55.110 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:25:55.110 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:25:55.110 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.110 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:25:55.110 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:27.268731+0000", 2024-09-13T11:25:55.110 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:55.110 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.110 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.110 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:25:55.110 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:27.268731+0000", 2024-09-13T11:25:55.110 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:55.111 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.111 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.111 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:25:55.111 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:27.268733+0000", 2024-09-13T11:25:55.111 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.9740000000000001e-06 2024-09-13T11:25:55.111 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.111 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.111 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:25:55.111 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:27.268778+0000", 2024-09-13T11:25:55.111 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.4679999999999999e-05 2024-09-13T11:25:55.111 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.111 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.111 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:25:55.111 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:27.988594+0000", 2024-09-13T11:25:55.112 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.71981667299999996 2024-09-13T11:25:55.112 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.112 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.112 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:25:55.112 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:27.988652+0000", 2024-09-13T11:25:55.112 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.7640999999999999e-05 2024-09-13T11:25:55.112 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.112 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.112 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:25:55.112 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:27.988889+0000", 2024-09-13T11:25:55.112 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00023706500000000001 2024-09-13T11:25:55.112 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.112 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.113 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:25:55.113 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:27.988933+0000", 2024-09-13T11:25:55.113 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.3865000000000001e-05 2024-09-13T11:25:55.113 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.113 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.113 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:25:55.113 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:27.991015+0000", 2024-09-13T11:25:55.113 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.002081552 2024-09-13T11:25:55.113 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:55.113 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:25:55.113 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:55.113 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.113 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.114 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPush(3.2b 124/119 [PushOp(3:d70bd127:::benchmark_data_smithi177_31116_object32365:head, version: 60'2034, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:d70bd127:::benchmark_data_smithi177_31116_object32365:head@60'2034, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:25:55.114 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:32.037338+0000", 2024-09-13T11:25:55.114 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 83.015446926999999, 2024-09-13T11:25:55.114 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.24358922699999999, 2024-09-13T11:25:55.114 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:25:55.114 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:25:55.114 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:25:55.114 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:25:55.114 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.114 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:25:55.114 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:32.037338+0000", 2024-09-13T11:25:55.114 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:55.114 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.114 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.115 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:25:55.115 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:32.037338+0000", 2024-09-13T11:25:55.115 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:55.115 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.115 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.115 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:25:55.115 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:32.037340+0000", 2024-09-13T11:25:55.115 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3790000000000001e-06 2024-09-13T11:25:55.115 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.115 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.115 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:25:55.115 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:32.037411+0000", 2024-09-13T11:25:55.115 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.1254000000000006e-05 2024-09-13T11:25:55.115 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.116 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.116 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:25:55.116 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:32.277467+0000", 2024-09-13T11:25:55.116 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.24005563199999999 2024-09-13T11:25:55.116 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.117 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.117 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:25:55.117 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:32.277494+0000", 2024-09-13T11:25:55.117 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.7120000000000001e-05 2024-09-13T11:25:55.117 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.117 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.117 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:25:55.117 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:32.279860+0000", 2024-09-13T11:25:55.117 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0023667559999999998 2024-09-13T11:25:55.117 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.117 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.117 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:25:55.118 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:32.279883+0000", 2024-09-13T11:25:55.118 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.2138999999999999e-05 2024-09-13T11:25:55.118 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.118 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.118 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:25:55.118 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:32.280928+0000", 2024-09-13T11:25:55.118 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0010449470000000001 2024-09-13T11:25:55.118 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:55.118 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:25:55.118 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:55.118 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.118 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.118 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPush(3.2b 125/119 [PushOp(3:d4c5ede9:::benchmark_data_smithi177_31116_object33277:head, version: 60'2092, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:d4c5ede9:::benchmark_data_smithi177_31116_object33277:head@60'2092, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:25:55.119 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:32.403929+0000", 2024-09-13T11:25:55.119 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 82.648856476999995, 2024-09-13T11:25:55.119 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.194592864, 2024-09-13T11:25:55.119 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:25:55.119 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:25:55.119 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:25:55.119 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:25:55.119 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.119 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:25:55.119 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:32.403929+0000", 2024-09-13T11:25:55.120 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:55.120 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.120 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.120 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:25:55.120 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:32.403929+0000", 2024-09-13T11:25:55.120 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:55.120 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.120 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.120 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:25:55.120 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:32.403930+0000", 2024-09-13T11:25:55.120 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.6729999999999999e-06 2024-09-13T11:25:55.120 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.120 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.121 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:25:55.121 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:32.403979+0000", 2024-09-13T11:25:55.121 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.8242000000000003e-05 2024-09-13T11:25:55.121 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.121 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.121 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:25:55.121 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:32.597445+0000", 2024-09-13T11:25:55.121 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.19346676500000001 2024-09-13T11:25:55.121 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.121 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.121 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:25:55.121 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:32.597455+0000", 2024-09-13T11:25:55.121 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.2939999999999994e-06 2024-09-13T11:25:55.121 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.122 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.122 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:25:55.122 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:32.597514+0000", 2024-09-13T11:25:55.122 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.9509000000000002e-05 2024-09-13T11:25:55.122 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.122 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.122 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:25:55.122 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:32.597522+0000", 2024-09-13T11:25:55.122 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.0350000000000001e-06 2024-09-13T11:25:55.122 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.122 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.122 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:25:55.122 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:32.598522+0000", 2024-09-13T11:25:55.122 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00099934599999999991 2024-09-13T11:25:55.123 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:55.123 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:25:55.123 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:55.123 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.123 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.123 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPush(3.2b 128/119 [PushOp(3:d6ff590f:::benchmark_data_smithi177_31116_object64581:head, version: 90'3750, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:d6ff590f:::benchmark_data_smithi177_31116_object64581:head@90'3750, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:25:55.123 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:37.982062+0000", 2024-09-13T11:25:55.123 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 77.070723577999999, 2024-09-13T11:25:55.123 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.28899675200000002, 2024-09-13T11:25:55.123 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:25:55.123 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:25:55.123 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:25:55.123 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:25:55.124 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.124 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:25:55.124 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:37.982062+0000", 2024-09-13T11:25:55.124 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:55.124 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.124 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.124 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:25:55.124 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:37.982062+0000", 2024-09-13T11:25:55.124 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:55.124 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.124 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.124 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:25:55.124 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:37.982064+0000", 2024-09-13T11:25:55.125 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.2259999999999999e-06 2024-09-13T11:25:55.125 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.125 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.125 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:25:55.125 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:37.982114+0000", 2024-09-13T11:25:55.125 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.0330000000000001e-05 2024-09-13T11:25:55.125 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.125 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.125 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:25:55.125 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:38.269983+0000", 2024-09-13T11:25:55.125 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.28786869999999998 2024-09-13T11:25:55.125 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.125 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.126 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:25:55.126 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:38.270000+0000", 2024-09-13T11:25:55.126 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.6977999999999999e-05 2024-09-13T11:25:55.126 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.126 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.126 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:25:55.126 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:38.270092+0000", 2024-09-13T11:25:55.126 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.221e-05 2024-09-13T11:25:55.126 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.126 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.126 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:25:55.126 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:38.270103+0000", 2024-09-13T11:25:55.126 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.1253999999999999e-05 2024-09-13T11:25:55.126 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.127 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.127 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:25:55.127 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:38.271058+0000", 2024-09-13T11:25:55.127 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00095505399999999999 2024-09-13T11:25:55.127 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:55.127 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:25:55.127 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:55.127 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.127 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.127 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:150366 3.2c e149/144 3:348937c7:::benchmark_data_smithi177_31116_object10063:head v 149'5822, mlcod=149'5819)", 2024-09-13T11:25:55.127 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:25:06.284607+0000", 2024-09-13T11:25:55.127 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 48.768178288000001, 2024-09-13T11:25:55.127 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.16600746599999999, 2024-09-13T11:25:55.128 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:25:55.128 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:25:55.128 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:55.128 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:25:55.128 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.128 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:25:55.128 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:06.284607+0000", 2024-09-13T11:25:55.128 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:55.128 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.128 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.128 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:25:55.128 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:06.284607+0000", 2024-09-13T11:25:55.129 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:55.129 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.129 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.129 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:25:55.129 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:06.284608+0000", 2024-09-13T11:25:55.129 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.018e-06 2024-09-13T11:25:55.129 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.129 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.129 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:25:55.129 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:06.284612+0000", 2024-09-13T11:25:55.129 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.3239999999999996e-06 2024-09-13T11:25:55.129 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.129 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.129 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:25:55.130 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:06.449402+0000", 2024-09-13T11:25:55.130 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.16478921099999999 2024-09-13T11:25:55.130 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.130 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.130 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:25:55.130 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:06.449408+0000", 2024-09-13T11:25:55.130 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.1990000000000003e-06 2024-09-13T11:25:55.130 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.130 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.130 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:25:55.130 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:06.449467+0000", 2024-09-13T11:25:55.130 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.8792999999999998e-05 2024-09-13T11:25:55.130 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.131 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.131 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:25:55.131 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:06.449490+0000", 2024-09-13T11:25:55.131 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.3895000000000001e-05 2024-09-13T11:25:55.131 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.131 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.131 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:25:55.131 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:06.450581+0000", 2024-09-13T11:25:55.131 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001090653 2024-09-13T11:25:55.131 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.131 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.131 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:25:55.131 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:06.450614+0000", 2024-09-13T11:25:55.132 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.3373e-05 2024-09-13T11:25:55.132 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:55.132 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:25:55.132 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:55.132 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.132 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.132 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:183827 3.2b e171/119 3:d63d3986:::benchmark_data_smithi177_31116_object43524:head v 171'6235, mlcod=171'6232)", 2024-09-13T11:25:55.132 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:25:32.487782+0000", 2024-09-13T11:25:55.132 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 22.565003364999999, 2024-09-13T11:25:55.132 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.21250508400000001, 2024-09-13T11:25:55.132 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:25:55.132 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:25:55.133 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:55.133 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:25:55.133 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.133 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:25:55.133 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:32.487782+0000", 2024-09-13T11:25:55.133 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:55.133 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.133 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.133 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:25:55.133 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:32.487782+0000", 2024-09-13T11:25:55.133 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:55.133 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.133 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.134 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:25:55.134 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:32.487783+0000", 2024-09-13T11:25:55.134 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.9699999999999994e-07 2024-09-13T11:25:55.134 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.134 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.134 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:25:55.134 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:32.487787+0000", 2024-09-13T11:25:55.134 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.5689999999999998e-06 2024-09-13T11:25:55.134 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.134 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.134 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:25:55.134 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:32.699019+0000", 2024-09-13T11:25:55.134 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.21123154499999999 2024-09-13T11:25:55.134 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.135 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.135 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:25:55.135 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:32.699024+0000", 2024-09-13T11:25:55.135 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.6229999999999996e-06 2024-09-13T11:25:55.135 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.135 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.135 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:25:55.135 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:32.699068+0000", 2024-09-13T11:25:55.135 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.4722000000000003e-05 2024-09-13T11:25:55.135 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.135 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.135 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:25:55.136 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:32.699084+0000", 2024-09-13T11:25:55.136 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5282999999999999e-05 2024-09-13T11:25:55.136 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.136 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.136 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:25:55.136 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:32.700246+0000", 2024-09-13T11:25:55.136 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001162735 2024-09-13T11:25:55.136 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.136 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.136 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:25:55.136 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:32.700287+0000", 2024-09-13T11:25:55.136 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.0609999999999999e-05 2024-09-13T11:25:55.136 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:55.137 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:25:55.137 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:55.137 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.137 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.137 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:191665 3.a e176/117 3:53060b28:::benchmark_data_smithi177_31116_object51362:head v 176'6424, mlcod=176'6422)", 2024-09-13T11:25:55.137 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:25:39.657701+0000", 2024-09-13T11:25:55.137 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 15.395083851000001, 2024-09-13T11:25:55.137 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.25090320900000002, 2024-09-13T11:25:55.137 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:25:55.137 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:25:55.137 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:55.137 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:25:55.137 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.138 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:25:55.138 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:39.657701+0000", 2024-09-13T11:25:55.138 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:55.138 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.138 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.138 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:25:55.138 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:39.657701+0000", 2024-09-13T11:25:55.138 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:55.138 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.138 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.138 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:25:55.138 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:39.657704+0000", 2024-09-13T11:25:55.138 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.7650000000000002e-06 2024-09-13T11:25:55.139 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.139 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.139 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:25:55.139 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:39.657714+0000", 2024-09-13T11:25:55.139 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.5719999999999992e-06 2024-09-13T11:25:55.139 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.139 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.139 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:25:55.139 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:39.904490+0000", 2024-09-13T11:25:55.139 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.246775831 2024-09-13T11:25:55.139 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.139 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.139 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:25:55.139 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:39.904503+0000", 2024-09-13T11:25:55.140 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3723000000000001e-05 2024-09-13T11:25:55.140 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.140 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.140 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:25:55.140 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:39.904660+0000", 2024-09-13T11:25:55.140 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00015705499999999999 2024-09-13T11:25:55.140 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.140 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.140 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:25:55.140 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:39.904699+0000", 2024-09-13T11:25:55.140 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.8877000000000001e-05 2024-09-13T11:25:55.140 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.140 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.141 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:25:55.141 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:39.908544+0000", 2024-09-13T11:25:55.141 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0038448739999999999 2024-09-13T11:25:55.141 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.141 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.141 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:25:55.141 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:39.908605+0000", 2024-09-13T11:25:55.141 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.0511999999999999e-05 2024-09-13T11:25:55.141 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:55.141 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:25:55.141 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:55.141 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.141 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.141 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:196349 3.d 3:b2d956a9:::benchmark_data_smithi177_31116_object56046:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e178)", 2024-09-13T11:25:55.142 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:25:43.404118+0000", 2024-09-13T11:25:55.142 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 11.648667758, 2024-09-13T11:25:55.142 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.17729215100000001, 2024-09-13T11:25:55.142 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:25:55.142 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:25:55.142 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:55.142 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:25:55.142 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:25:55.142 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:55.142 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 196349 2024-09-13T11:25:55.142 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.142 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:25:55.142 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.142 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:25:55.143 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:43.404118+0000", 2024-09-13T11:25:55.143 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:55.143 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.143 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.143 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:25:55.143 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:43.404118+0000", 2024-09-13T11:25:55.143 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:55.143 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.143 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.143 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:25:55.143 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:43.404121+0000", 2024-09-13T11:25:55.143 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.743e-06 2024-09-13T11:25:55.143 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.143 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.144 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:25:55.144 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:43.404132+0000", 2024-09-13T11:25:55.144 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0433e-05 2024-09-13T11:25:55.144 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.144 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.144 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:25:55.144 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:43.404159+0000", 2024-09-13T11:25:55.144 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.7064e-05 2024-09-13T11:25:55.144 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.144 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.144 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:25:55.144 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:43.404172+0000", 2024-09-13T11:25:55.144 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3606e-05 2024-09-13T11:25:55.144 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.145 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.145 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:25:55.145 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:43.404312+0000", 2024-09-13T11:25:55.145 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000139262 2024-09-13T11:25:55.145 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.145 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.145 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:25:55.145 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:43.404531+0000", 2024-09-13T11:25:55.145 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00021986 2024-09-13T11:25:55.145 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.145 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.145 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 3", 2024-09-13T11:25:55.145 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:43.404706+0000", 2024-09-13T11:25:55.146 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000174358 2024-09-13T11:25:55.146 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.146 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.146 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:25:55.146 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:43.408021+0000", 2024-09-13T11:25:55.146 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0033152939999999999 2024-09-13T11:25:55.146 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.146 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.146 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:25:55.146 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:43.581239+0000", 2024-09-13T11:25:55.146 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.17321809699999999 2024-09-13T11:25:55.146 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.146 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.147 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:25:55.147 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:43.581331+0000", 2024-09-13T11:25:55.147 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.1877000000000001e-05 2024-09-13T11:25:55.147 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.147 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.147 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:25:55.147 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:43.581410+0000", 2024-09-13T11:25:55.147 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.8557e-05 2024-09-13T11:25:55.147 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:55.147 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:25:55.147 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:55.147 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:55.147 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:25:55.147 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-09-13T11:25:55.149 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-09-13T11:25:55.150 INFO:tasks.ceph.osd.10.smithi162.stderr:2024-09-13T11:25:55.147+0000 7fb59c9f0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26317) UID: 0 2024-09-13T11:25:55.249 INFO:tasks.ceph.osd.12.smithi001.stderr:2024-09-13T11:25:55.246+0000 7f892c6e6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26314) UID: 0 2024-09-13T11:25:55.327 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-09-13T11:25:55.327 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [ 2024-09-13T11:25:55.327 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.327 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:211103 3.1c 3:39edbf94:::benchmark_data_smithi177_31116_object70800:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e185)", 2024-09-13T11:25:55.327 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:25:55.295363+0000", 2024-09-13T11:25:55.327 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 0.032037547, 2024-09-13T11:25:55.327 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.032055647, 2024-09-13T11:25:55.327 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:25:55.327 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:25:55.327 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "waiting for sub ops", 2024-09-13T11:25:55.328 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:25:55.328 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:25:55.328 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:55.328 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 211103 2024-09-13T11:25:55.328 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.328 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:25:55.328 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.328 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:25:55.328 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:55.295363+0000", 2024-09-13T11:25:55.328 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:55.328 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.328 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.328 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:25:55.329 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:55.295363+0000", 2024-09-13T11:25:55.329 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:55.329 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.329 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.329 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:25:55.329 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:55.295364+0000", 2024-09-13T11:25:55.329 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0219999999999999e-06 2024-09-13T11:25:55.329 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.329 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.329 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:25:55.329 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:55.295368+0000", 2024-09-13T11:25:55.329 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.8260000000000003e-06 2024-09-13T11:25:55.329 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.329 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.330 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:25:55.330 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:55.295377+0000", 2024-09-13T11:25:55.330 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.5429999999999994e-06 2024-09-13T11:25:55.330 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.330 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.330 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:25:55.330 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:55.295383+0000", 2024-09-13T11:25:55.330 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.1550000000000004e-06 2024-09-13T11:25:55.330 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.330 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.330 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:25:55.330 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:55.295436+0000", 2024-09-13T11:25:55.330 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.295e-05 2024-09-13T11:25:55.330 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.331 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.331 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:25:55.331 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:55.295552+0000", 2024-09-13T11:25:55.331 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00011621299999999999 2024-09-13T11:25:55.331 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.331 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.331 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 7", 2024-09-13T11:25:55.331 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:55.295646+0000", 2024-09-13T11:25:55.331 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.3617999999999997e-05 2024-09-13T11:25:55.331 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.331 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.331 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:25:55.331 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:55.299184+0000", 2024-09-13T11:25:55.332 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.003538219 2024-09-13T11:25:55.332 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:55.332 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:25:55.332 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:55.332 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.332 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.332 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:211133 3.16 3:6c64c744:::benchmark_data_smithi177_31116_object70830:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e185)", 2024-09-13T11:25:55.332 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:25:55.319208+0000", 2024-09-13T11:25:55.332 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 0.0081927719999999992, 2024-09-13T11:25:55.332 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0082761529999999996, 2024-09-13T11:25:55.332 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:25:55.332 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:25:55.332 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "waiting for sub ops", 2024-09-13T11:25:55.333 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:25:55.333 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:25:55.333 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:55.333 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 211133 2024-09-13T11:25:55.333 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.333 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:25:55.333 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.333 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:25:55.333 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:55.319208+0000", 2024-09-13T11:25:55.333 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:55.333 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.333 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.333 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:25:55.334 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:55.319208+0000", 2024-09-13T11:25:55.334 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:55.334 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.334 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.334 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:25:55.334 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:55.319211+0000", 2024-09-13T11:25:55.334 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.7410000000000001e-06 2024-09-13T11:25:55.334 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.334 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.334 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:25:55.334 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:55.319218+0000", 2024-09-13T11:25:55.334 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.2840000000000001e-06 2024-09-13T11:25:55.334 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.334 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.335 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:25:55.335 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:55.319236+0000", 2024-09-13T11:25:55.335 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.7972e-05 2024-09-13T11:25:55.335 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.335 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.335 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:25:55.335 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:55.319244+0000", 2024-09-13T11:25:55.335 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.3470000000000001e-06 2024-09-13T11:25:55.335 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.335 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.335 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:25:55.335 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:55.319309+0000", 2024-09-13T11:25:55.335 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.5474000000000004e-05 2024-09-13T11:25:55.335 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.336 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.336 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:25:55.336 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:55.319459+0000", 2024-09-13T11:25:55.336 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000149693 2024-09-13T11:25:55.336 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.336 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.336 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 15", 2024-09-13T11:25:55.336 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:55.319572+0000", 2024-09-13T11:25:55.336 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00011283899999999999 2024-09-13T11:25:55.336 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.336 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.336 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:25:55.336 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:55.322652+0000", 2024-09-13T11:25:55.336 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0030806150000000001 2024-09-13T11:25:55.337 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:55.337 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:25:55.337 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:55.337 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.337 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.337 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:211142 3.11 e185/81 3:89a6fa00:::benchmark_data_smithi177_31116_object70839:head v 185'8029, mlcod=185'8027)", 2024-09-13T11:25:55.337 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:25:55.326999+0000", 2024-09-13T11:25:55.337 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 0.00040246900000000003, 2024-09-13T11:25:55.337 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00052543399999999997, 2024-09-13T11:25:55.337 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:25:55.337 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:25:55.337 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:25:55.337 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:25:55.337 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.338 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:25:55.338 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:55.326999+0000", 2024-09-13T11:25:55.338 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:55.338 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.338 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.338 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:25:55.338 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:55.326999+0000", 2024-09-13T11:25:55.338 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:55.338 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.338 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.338 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:25:55.338 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:55.327000+0000", 2024-09-13T11:25:55.338 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.646e-06 2024-09-13T11:25:55.338 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.339 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.339 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:25:55.339 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:55.327007+0000", 2024-09-13T11:25:55.339 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.2479999999999999e-06 2024-09-13T11:25:55.339 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.339 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.339 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:25:55.339 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:55.327037+0000", 2024-09-13T11:25:55.339 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.9091e-05 2024-09-13T11:25:55.339 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.339 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.339 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:25:55.339 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:55.327043+0000", 2024-09-13T11:25:55.339 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.4690000000000002e-06 2024-09-13T11:25:55.340 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.340 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.340 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:25:55.340 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:55.327094+0000", 2024-09-13T11:25:55.340 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.0909000000000003e-05 2024-09-13T11:25:55.340 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.340 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.340 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:25:55.340 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:55.327109+0000", 2024-09-13T11:25:55.340 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.4586000000000001e-05 2024-09-13T11:25:55.340 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:55.340 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:25:55.340 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:55.340 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.341 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.341 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:211143 3.13 3:cdf6a748:::benchmark_data_smithi177_31116_object70840:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e185)", 2024-09-13T11:25:55.341 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:25:55.327273+0000", 2024-09-13T11:25:55.341 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 0.00012797899999999999, 2024-09-13T11:25:55.341 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00028833899999999998, 2024-09-13T11:25:55.341 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:25:55.341 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:25:55.341 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "waiting for sub ops", 2024-09-13T11:25:55.341 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:25:55.341 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:25:55.341 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:55.341 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 211143 2024-09-13T11:25:55.341 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.342 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:25:55.342 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.342 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:25:55.342 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:55.327273+0000", 2024-09-13T11:25:55.342 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:55.342 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.342 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.342 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:25:55.342 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:55.327273+0000", 2024-09-13T11:25:55.342 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:55.342 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.342 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.342 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:25:55.342 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:55.327275+0000", 2024-09-13T11:25:55.342 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.2979999999999999e-06 2024-09-13T11:25:55.343 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.343 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.343 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:25:55.343 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:55.327281+0000", 2024-09-13T11:25:55.343 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.8390000000000003e-06 2024-09-13T11:25:55.343 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.343 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.343 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:25:55.343 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:55.327295+0000", 2024-09-13T11:25:55.343 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3981999999999999e-05 2024-09-13T11:25:55.343 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.343 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.343 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:25:55.343 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:55.327302+0000", 2024-09-13T11:25:55.344 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.0679999999999999e-06 2024-09-13T11:25:55.344 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.344 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.344 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:25:55.344 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:55.327358+0000", 2024-09-13T11:25:55.344 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.6084e-05 2024-09-13T11:25:55.344 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.344 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.344 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:25:55.344 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:55.327470+0000", 2024-09-13T11:25:55.344 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000111505 2024-09-13T11:25:55.344 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.344 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.345 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 7", 2024-09-13T11:25:55.345 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:55.327538+0000", 2024-09-13T11:25:55.345 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.8190999999999998e-05 2024-09-13T11:25:55.345 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:55.345 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:25:55.345 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:55.345 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:55.345 INFO:teuthology.orchestra.run.smithi177.stdout: ], 2024-09-13T11:25:55.345 INFO:teuthology.orchestra.run.smithi177.stdout: "num_ops": 4 2024-09-13T11:25:55.345 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-09-13T11:25:55.346 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-09-13T11:25:55.350 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-13T11:25:55.346+0000 7f6a68775640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26307) UID: 0 2024-09-13T11:25:55.451 INFO:tasks.ceph.osd.4.smithi001.stderr:2024-09-13T11:25:55.446+0000 7ff8e0218640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26310) UID: 0 2024-09-13T11:25:55.480 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-09-13T11:25:55.480 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [], 2024-09-13T11:25:55.480 INFO:teuthology.orchestra.run.smithi177.stdout: "complaint_time": 30, 2024-09-13T11:25:55.480 INFO:teuthology.orchestra.run.smithi177.stdout: "num_blocked_ops": 0 2024-09-13T11:25:55.480 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-09-13T11:25:55.494 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-09-13T11:25:55.515 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T11:25:55.552 INFO:tasks.ceph.osd.11.smithi177.stderr:2024-09-13T11:25:55.550+0000 7f24792c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26580) UID: 0 2024-09-13T11:25:55.625 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-09-13T11:25:55.625 INFO:teuthology.orchestra.run.smithi177.stdout: "size": 20, 2024-09-13T11:25:55.625 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 600, 2024-09-13T11:25:55.626 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [ 2024-09-13T11:25:55.626 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.626 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:79 3.1 e32/30 3:841df5b5:::benchmark_data_smithi177_31116_object78:head v 32'6, mlcod=32'4)", 2024-09-13T11:25:55.626 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:22:27.057292+0000", 2024-09-13T11:25:55.626 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 208.56754635199999, 2024-09-13T11:25:55.626 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.89296786800000005, 2024-09-13T11:25:55.626 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:25:55.626 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:25:55.626 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:55.626 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:25:55.626 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.626 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:25:55.626 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.057292+0000", 2024-09-13T11:25:55.627 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:55.627 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.627 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.627 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:25:55.627 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.057292+0000", 2024-09-13T11:25:55.627 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:55.627 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.627 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.627 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:25:55.627 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.057293+0000", 2024-09-13T11:25:55.627 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.6109999999999999e-06 2024-09-13T11:25:55.627 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.627 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.627 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:25:55.628 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.057330+0000", 2024-09-13T11:25:55.628 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.6721999999999998e-05 2024-09-13T11:25:55.628 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.628 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.628 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:25:55.628 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.948197+0000", 2024-09-13T11:25:55.628 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.890867042 2024-09-13T11:25:55.628 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.628 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.628 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:25:55.628 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.948213+0000", 2024-09-13T11:25:55.628 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5826000000000001e-05 2024-09-13T11:25:55.628 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.629 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.629 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:25:55.629 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.948377+0000", 2024-09-13T11:25:55.629 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00016348600000000001 2024-09-13T11:25:55.629 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.629 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.629 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:25:55.629 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.948447+0000", 2024-09-13T11:25:55.629 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.0891999999999998e-05 2024-09-13T11:25:55.629 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.630 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.630 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:25:55.630 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.950150+0000", 2024-09-13T11:25:55.630 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001702329 2024-09-13T11:25:55.630 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.630 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.630 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:25:55.630 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.950260+0000", 2024-09-13T11:25:55.630 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00010996 2024-09-13T11:25:55.630 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:55.630 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:25:55.630 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:55.630 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.631 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.631 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:30828 3.f 3:f3a9c6a9:::benchmark_data_smithi177_31116_object30827:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e59)", 2024-09-13T11:25:55.631 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:09.503661+0000", 2024-09-13T11:25:55.631 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 166.12117700300001, 2024-09-13T11:25:55.631 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.26477667100000002, 2024-09-13T11:25:55.631 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:25:55.631 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:25:55.631 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:55.631 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:25:55.631 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:25:55.632 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:55.632 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 30828 2024-09-13T11:25:55.632 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.632 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:25:55.632 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.632 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:25:55.632 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.503661+0000", 2024-09-13T11:25:55.632 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:55.632 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.632 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.633 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:25:55.633 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.503661+0000", 2024-09-13T11:25:55.633 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:55.633 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.633 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.633 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:25:55.633 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.503663+0000", 2024-09-13T11:25:55.633 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.968e-06 2024-09-13T11:25:55.633 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.633 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.633 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:25:55.634 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.503687+0000", 2024-09-13T11:25:55.634 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.3983999999999999e-05 2024-09-13T11:25:55.634 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.634 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.634 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:25:55.634 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.503701+0000", 2024-09-13T11:25:55.634 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3416e-05 2024-09-13T11:25:55.634 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.634 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.635 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:25:55.635 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.503708+0000", 2024-09-13T11:25:55.635 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.9530000000000002e-06 2024-09-13T11:25:55.635 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.635 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.635 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:25:55.635 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.503759+0000", 2024-09-13T11:25:55.635 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.1075999999999997e-05 2024-09-13T11:25:55.635 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.635 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.635 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:25:55.636 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.503861+0000", 2024-09-13T11:25:55.636 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000102372 2024-09-13T11:25:55.636 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.636 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.636 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 12", 2024-09-13T11:25:55.636 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.503952+0000", 2024-09-13T11:25:55.636 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.1135000000000001e-05 2024-09-13T11:25:55.636 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.636 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.636 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:25:55.636 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.505340+0000", 2024-09-13T11:25:55.636 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0013877169999999999 2024-09-13T11:25:55.637 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.637 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.637 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:25:55.637 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.768281+0000", 2024-09-13T11:25:55.637 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.26294092400000002 2024-09-13T11:25:55.637 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.637 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.637 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:25:55.637 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.768369+0000", 2024-09-13T11:25:55.637 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.7818999999999994e-05 2024-09-13T11:25:55.637 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.637 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.637 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:25:55.638 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.768438+0000", 2024-09-13T11:25:55.638 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.9307000000000006e-05 2024-09-13T11:25:55.638 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:55.638 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:25:55.638 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:55.638 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.638 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.638 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:45783 3.16 3.17c10a76 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e69)", 2024-09-13T11:25:55.638 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:20.947774+0000", 2024-09-13T11:25:55.638 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 154.67706410100001, 2024-09-13T11:25:55.638 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.96682922400000004, 2024-09-13T11:25:55.639 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:25:55.639 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:25:55.639 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-13T11:25:55.639 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:25:55.639 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:25:55.639 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:55.639 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 45783 2024-09-13T11:25:55.639 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.639 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:25:55.639 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.639 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:25:55.639 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:20.947774+0000", 2024-09-13T11:25:55.639 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:55.640 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.640 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.640 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:25:55.640 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:20.947774+0000", 2024-09-13T11:25:55.640 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:55.640 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.640 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.640 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:25:55.640 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:20.947777+0000", 2024-09-13T11:25:55.640 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.57e-06 2024-09-13T11:25:55.640 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.640 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.641 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:25:55.641 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:20.947802+0000", 2024-09-13T11:25:55.641 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.5528999999999999e-05 2024-09-13T11:25:55.641 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.641 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.641 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:25:55.641 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:20.947813+0000", 2024-09-13T11:25:55.641 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0473e-05 2024-09-13T11:25:55.641 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.641 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.641 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:25:55.641 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:20.947821+0000", 2024-09-13T11:25:55.641 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.3440000000000001e-06 2024-09-13T11:25:55.642 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.642 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.644 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:25:55.645 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:20.947933+0000", 2024-09-13T11:25:55.645 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00011210200000000001 2024-09-13T11:25:55.645 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.645 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.645 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for peered", 2024-09-13T11:25:55.645 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:20.947942+0000", 2024-09-13T11:25:55.645 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.6610000000000006e-06 2024-09-13T11:25:55.645 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.645 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.645 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:25:55.645 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:21.914579+0000", 2024-09-13T11:25:55.645 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.96663672300000003 2024-09-13T11:25:55.645 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.646 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.646 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:25:55.646 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:21.914603+0000", 2024-09-13T11:25:55.646 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.4822000000000001e-05 2024-09-13T11:25:55.646 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:55.646 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:25:55.646 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:55.646 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.646 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.646 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:54625 3.1f 3:f812cdfb:::benchmark_data_smithi177_31116_object54624:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e75)", 2024-09-13T11:25:55.646 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:30.979327+0000", 2024-09-13T11:25:55.646 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 144.645511312, 2024-09-13T11:25:55.646 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.96163670899999998, 2024-09-13T11:25:55.646 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:25:55.647 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:25:55.647 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-13T11:25:55.647 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:25:55.647 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:25:55.647 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:55.647 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 54625 2024-09-13T11:25:55.647 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.647 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:25:55.647 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.647 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:25:55.647 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.979327+0000", 2024-09-13T11:25:55.648 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:55.648 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.648 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.648 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:25:55.648 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.979327+0000", 2024-09-13T11:25:55.648 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:55.648 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.648 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.648 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:25:55.648 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.979329+0000", 2024-09-13T11:25:55.648 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.686e-06 2024-09-13T11:25:55.649 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.649 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.649 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:25:55.649 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.979364+0000", 2024-09-13T11:25:55.649 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.5750000000000002e-05 2024-09-13T11:25:55.649 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.649 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.649 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:25:55.649 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.979408+0000", 2024-09-13T11:25:55.649 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.3795999999999999e-05 2024-09-13T11:25:55.649 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.649 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.650 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:25:55.650 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.979427+0000", 2024-09-13T11:25:55.650 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.9040999999999999e-05 2024-09-13T11:25:55.650 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.650 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.650 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:25:55.650 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.122291+0000", 2024-09-13T11:25:55.650 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.14286373599999999 2024-09-13T11:25:55.650 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.650 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.650 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for peered", 2024-09-13T11:25:55.651 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.122298+0000", 2024-09-13T11:25:55.651 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.3719999999999999e-06 2024-09-13T11:25:55.651 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.651 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.651 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:25:55.651 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.940341+0000", 2024-09-13T11:25:55.651 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.81804249699999998 2024-09-13T11:25:55.651 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.651 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.651 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:25:55.652 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.940964+0000", 2024-09-13T11:25:55.652 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00062283100000000001 2024-09-13T11:25:55.652 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:55.652 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:25:55.652 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:55.652 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.653 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.653 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:54649 3.1c 3:3a457a93:::benchmark_data_smithi177_31116_object54648:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e75)", 2024-09-13T11:25:55.653 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:30.979460+0000", 2024-09-13T11:25:55.653 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 144.64537812200001, 2024-09-13T11:25:55.653 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.95579424000000002, 2024-09-13T11:25:55.653 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:25:55.654 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:25:55.654 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-13T11:25:55.654 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:25:55.654 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:25:55.654 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:55.654 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 54649 2024-09-13T11:25:55.655 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.655 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:25:55.655 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.655 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:25:55.655 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.979460+0000", 2024-09-13T11:25:55.655 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:55.656 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.656 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.656 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:25:55.656 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.979460+0000", 2024-09-13T11:25:55.656 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:55.656 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.657 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.657 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:25:55.657 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.979462+0000", 2024-09-13T11:25:55.657 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5799999999999999e-06 2024-09-13T11:25:55.657 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.657 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.658 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:25:55.658 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.979524+0000", 2024-09-13T11:25:55.658 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.2521999999999995e-05 2024-09-13T11:25:55.658 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.658 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.658 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:25:55.659 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.979561+0000", 2024-09-13T11:25:55.659 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.6349e-05 2024-09-13T11:25:55.659 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.659 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.659 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:25:55.660 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.979571+0000", 2024-09-13T11:25:55.660 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0457e-05 2024-09-13T11:25:55.660 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.660 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.660 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:25:55.660 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.121029+0000", 2024-09-13T11:25:55.661 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.14145766200000001 2024-09-13T11:25:55.661 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.661 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.661 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for peered", 2024-09-13T11:25:55.661 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.121039+0000", 2024-09-13T11:25:55.662 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0084e-05 2024-09-13T11:25:55.662 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.662 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.662 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:25:55.662 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.935157+0000", 2024-09-13T11:25:55.662 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.81411807899999999 2024-09-13T11:25:55.663 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.663 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.663 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:25:55.663 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.935254+0000", 2024-09-13T11:25:55.663 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.7507e-05 2024-09-13T11:25:55.663 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:55.664 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:25:55.664 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:55.664 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.664 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.664 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:54654 3.1c 3:3f0bb31a:::benchmark_data_smithi177_31116_object54653:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e75)", 2024-09-13T11:25:55.665 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:30.979665+0000", 2024-09-13T11:25:55.665 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 144.645172957, 2024-09-13T11:25:55.665 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.95623908099999999, 2024-09-13T11:25:55.665 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:25:55.665 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:25:55.665 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-13T11:25:55.666 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:25:55.666 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:25:55.666 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:55.666 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 54654 2024-09-13T11:25:55.666 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.666 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:25:55.667 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.667 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:25:55.667 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.979665+0000", 2024-09-13T11:25:55.667 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:55.667 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.667 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.668 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:25:55.668 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.979665+0000", 2024-09-13T11:25:55.668 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:55.668 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.668 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.669 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:25:55.669 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.979667+0000", 2024-09-13T11:25:55.669 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3540000000000001e-06 2024-09-13T11:25:55.669 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.669 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.669 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:25:55.670 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.979703+0000", 2024-09-13T11:25:55.670 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.6758000000000003e-05 2024-09-13T11:25:55.670 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.670 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.670 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:25:55.670 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.979714+0000", 2024-09-13T11:25:55.671 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0804e-05 2024-09-13T11:25:55.671 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.671 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.671 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:25:55.671 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.979724+0000", 2024-09-13T11:25:55.672 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0040999999999999e-05 2024-09-13T11:25:55.672 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.672 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.672 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:25:55.672 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.121061+0000", 2024-09-13T11:25:55.672 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.141336713 2024-09-13T11:25:55.673 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.673 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.673 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for peered", 2024-09-13T11:25:55.673 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.121067+0000", 2024-09-13T11:25:55.673 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.4150000000000004e-06 2024-09-13T11:25:55.674 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.674 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.674 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:25:55.674 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.935776+0000", 2024-09-13T11:25:55.674 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.81470863100000002 2024-09-13T11:25:55.674 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.675 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.675 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:25:55.675 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.935904+0000", 2024-09-13T11:25:55.675 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00012836499999999999 2024-09-13T11:25:55.676 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:55.676 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:25:55.676 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:55.676 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.676 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.677 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPushReply(3.11 75/70 [PushReplyOp(3:8df10771:::benchmark_data_smithi177_31116_object24734:head)])", 2024-09-13T11:25:55.677 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:31.149798+0000", 2024-09-13T11:25:55.677 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 144.475039947, 2024-09-13T11:25:55.677 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.32402702500000002, 2024-09-13T11:25:55.677 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:25:55.677 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:25:55.678 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-13T11:25:55.678 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:25:55.678 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.678 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:25:55.678 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.149798+0000", 2024-09-13T11:25:55.678 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:55.678 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.679 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.679 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:25:55.679 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.149798+0000", 2024-09-13T11:25:55.679 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:55.679 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.680 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.680 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:25:55.680 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.149799+0000", 2024-09-13T11:25:55.680 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.5700000000000001e-07 2024-09-13T11:25:55.680 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.680 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.680 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:25:55.681 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.149803+0000", 2024-09-13T11:25:55.681 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.0040000000000001e-06 2024-09-13T11:25:55.681 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.681 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.681 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:25:55.682 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.472172+0000", 2024-09-13T11:25:55.682 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.32236856699999999 2024-09-13T11:25:55.682 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.682 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.682 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:25:55.682 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.472187+0000", 2024-09-13T11:25:55.683 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5016e-05 2024-09-13T11:25:55.683 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.683 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.683 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:25:55.683 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.472302+0000", 2024-09-13T11:25:55.683 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000115121 2024-09-13T11:25:55.683 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.683 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.684 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:25:55.684 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.473825+0000", 2024-09-13T11:25:55.684 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0015234599999999999 2024-09-13T11:25:55.684 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:55.684 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:25:55.684 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:55.684 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.684 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.684 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:54668 3.1c 3:3d2c5e40:::benchmark_data_smithi177_31116_object54667:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e75)", 2024-09-13T11:25:55.684 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:31.245525+0000", 2024-09-13T11:25:55.684 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 144.37931369, 2024-09-13T11:25:55.684 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.69828973000000005, 2024-09-13T11:25:55.684 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:25:55.685 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:25:55.685 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:55.685 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:25:55.685 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:25:55.685 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:55.685 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 54668 2024-09-13T11:25:55.685 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.685 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:25:55.685 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.685 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:25:55.685 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.245525+0000", 2024-09-13T11:25:55.686 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:55.686 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.686 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.686 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:25:55.686 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.245525+0000", 2024-09-13T11:25:55.686 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:55.686 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.686 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.686 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:25:55.686 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.245526+0000", 2024-09-13T11:25:55.686 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.719e-06 2024-09-13T11:25:55.687 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.687 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.687 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:25:55.687 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.245547+0000", 2024-09-13T11:25:55.687 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.1084999999999999e-05 2024-09-13T11:25:55.687 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.687 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.687 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:25:55.687 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.245558+0000", 2024-09-13T11:25:55.687 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0934000000000001e-05 2024-09-13T11:25:55.687 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.687 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.688 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:25:55.688 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.245564+0000", 2024-09-13T11:25:55.688 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.6269999999999997e-06 2024-09-13T11:25:55.688 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.688 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.688 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:25:55.688 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.245620+0000", 2024-09-13T11:25:55.688 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.6289999999999998e-05 2024-09-13T11:25:55.688 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.688 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.688 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for peered", 2024-09-13T11:25:55.688 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.245632+0000", 2024-09-13T11:25:55.688 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.1748e-05 2024-09-13T11:25:55.689 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.689 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.689 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:25:55.689 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.935976+0000", 2024-09-13T11:25:55.689 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.69034455299999997 2024-09-13T11:25:55.689 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.689 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.689 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:25:55.689 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.936302+0000", 2024-09-13T11:25:55.689 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000325397 2024-09-13T11:25:55.689 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.689 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.690 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 13", 2024-09-13T11:25:55.690 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.936459+0000", 2024-09-13T11:25:55.690 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00015754300000000001 2024-09-13T11:25:55.690 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.690 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.690 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:25:55.690 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.938813+0000", 2024-09-13T11:25:55.690 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0023530999999999999 2024-09-13T11:25:55.690 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.690 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.690 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:25:55.690 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.943695+0000", 2024-09-13T11:25:55.690 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.004882506 2024-09-13T11:25:55.691 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.691 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.691 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:25:55.691 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.943767+0000", 2024-09-13T11:25:55.691 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.2454999999999997e-05 2024-09-13T11:25:55.691 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.691 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.691 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:25:55.691 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.943814+0000", 2024-09-13T11:25:55.691 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.6773000000000001e-05 2024-09-13T11:25:55.691 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:55.691 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:25:55.691 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:55.691 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.691 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.692 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPush(3.16 84/70 [PushOp(3:6f7be541:::benchmark_data_smithi177_31116_object43332:head, version: 66'2788, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:6f7be541:::benchmark_data_smithi177_31116_object43332:head@66'2788, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:25:55.692 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:41.207515+0000", 2024-09-13T11:25:55.692 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 134.41732291899999, 2024-09-13T11:25:55.692 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.52643771500000003, 2024-09-13T11:25:55.692 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:25:55.692 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:25:55.692 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:25:55.692 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:25:55.692 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.692 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:25:55.692 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.207515+0000", 2024-09-13T11:25:55.692 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:55.693 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.693 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.693 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:25:55.693 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.207515+0000", 2024-09-13T11:25:55.693 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:55.693 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.693 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.693 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:25:55.693 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.207516+0000", 2024-09-13T11:25:55.693 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.096e-06 2024-09-13T11:25:55.693 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.693 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.694 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:25:55.694 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.207717+0000", 2024-09-13T11:25:55.694 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000200199 2024-09-13T11:25:55.694 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.694 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.694 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:25:55.694 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.732785+0000", 2024-09-13T11:25:55.694 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.52506869599999995 2024-09-13T11:25:55.694 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.694 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.694 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:25:55.694 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.732801+0000", 2024-09-13T11:25:55.694 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5305999999999999e-05 2024-09-13T11:25:55.695 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.695 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.695 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:25:55.695 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.732889+0000", 2024-09-13T11:25:55.695 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.8157000000000003e-05 2024-09-13T11:25:55.695 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.695 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.695 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:25:55.695 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.732901+0000", 2024-09-13T11:25:55.695 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.1866e-05 2024-09-13T11:25:55.695 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.695 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.695 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:25:55.695 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.733953+0000", 2024-09-13T11:25:55.696 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001052395 2024-09-13T11:25:55.696 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:55.696 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:25:55.696 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:55.696 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.696 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.696 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPushReply(3.1f 85/78 [PushReplyOp(3:fd221d21:::benchmark_data_smithi177_31116_object7639:head)])", 2024-09-13T11:25:55.696 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:42.198715+0000", 2024-09-13T11:25:55.696 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 133.42612335000001, 2024-09-13T11:25:55.696 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.48930391099999998, 2024-09-13T11:25:55.696 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:25:55.696 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:25:55.696 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-13T11:25:55.697 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:25:55.697 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.697 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:25:55.697 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.198715+0000", 2024-09-13T11:25:55.697 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:55.697 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.697 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.697 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:25:55.697 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.198715+0000", 2024-09-13T11:25:55.697 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:55.697 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.697 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.697 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:25:55.697 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.198716+0000", 2024-09-13T11:25:55.698 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.3500000000000005e-07 2024-09-13T11:25:55.698 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.698 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.698 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:25:55.698 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.198719+0000", 2024-09-13T11:25:55.698 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.4539999999999999e-06 2024-09-13T11:25:55.698 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.698 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.698 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:25:55.698 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.684732+0000", 2024-09-13T11:25:55.698 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.48601262099999998 2024-09-13T11:25:55.698 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.698 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.698 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:25:55.699 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.684746+0000", 2024-09-13T11:25:55.699 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.4518999999999999e-05 2024-09-13T11:25:55.699 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.699 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.699 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:25:55.699 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.684911+0000", 2024-09-13T11:25:55.699 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000164728 2024-09-13T11:25:55.699 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.699 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.699 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:25:55.699 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.688019+0000", 2024-09-13T11:25:55.699 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0031077539999999999 2024-09-13T11:25:55.699 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:55.699 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:25:55.700 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:55.700 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.700 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.700 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPushReply(3.1c 85/77 [PushReplyOp(3:3a669437:::benchmark_data_smithi177_31116_object25280:head)])", 2024-09-13T11:25:55.700 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:42.352466+0000", 2024-09-13T11:25:55.700 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 133.272372456, 2024-09-13T11:25:55.700 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.36670725300000001, 2024-09-13T11:25:55.700 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:25:55.700 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:25:55.700 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-13T11:25:55.700 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:25:55.700 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.701 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:25:55.701 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.352466+0000", 2024-09-13T11:25:55.701 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:55.701 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.701 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.701 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:25:55.701 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.352466+0000", 2024-09-13T11:25:55.701 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:55.701 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.701 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.701 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:25:55.701 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.352467+0000", 2024-09-13T11:25:55.701 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.1429999999999999e-06 2024-09-13T11:25:55.702 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.702 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.702 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:25:55.702 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.352474+0000", 2024-09-13T11:25:55.702 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.6100000000000002e-06 2024-09-13T11:25:55.702 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.702 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.702 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:25:55.702 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.717011+0000", 2024-09-13T11:25:55.702 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.36453790200000002 2024-09-13T11:25:55.702 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.702 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.702 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:25:55.702 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.717032+0000", 2024-09-13T11:25:55.703 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.0999e-05 2024-09-13T11:25:55.703 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.703 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.703 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:25:55.703 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.717162+0000", 2024-09-13T11:25:55.703 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000129159 2024-09-13T11:25:55.703 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.703 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.703 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:25:55.703 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.719173+0000", 2024-09-13T11:25:55.703 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0020114400000000002 2024-09-13T11:25:55.703 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:55.703 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:25:55.704 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:55.704 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.704 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.704 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPushReply(3.1f 85/78 [PushReplyOp(3:fecb0d6a:::benchmark_data_smithi177_31116_object7644:head)])", 2024-09-13T11:25:55.704 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:42.705351+0000", 2024-09-13T11:25:55.704 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 132.91948761399999, 2024-09-13T11:25:55.704 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.290549681, 2024-09-13T11:25:55.704 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:25:55.704 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:25:55.704 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-13T11:25:55.704 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:25:55.704 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.704 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:25:55.705 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.705351+0000", 2024-09-13T11:25:55.705 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:55.705 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.705 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.705 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:25:55.705 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.705351+0000", 2024-09-13T11:25:55.705 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:55.705 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.705 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.705 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:25:55.705 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.705353+0000", 2024-09-13T11:25:55.706 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.4169999999999999e-06 2024-09-13T11:25:55.706 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.706 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.706 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:25:55.706 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.705365+0000", 2024-09-13T11:25:55.706 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.1894e-05 2024-09-13T11:25:55.707 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.707 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.707 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:25:55.707 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.992081+0000", 2024-09-13T11:25:55.707 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.28671641199999998 2024-09-13T11:25:55.707 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.708 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.708 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:25:55.708 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.992111+0000", 2024-09-13T11:25:55.708 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.9427000000000001e-05 2024-09-13T11:25:55.708 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.708 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.709 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:25:55.709 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.993573+0000", 2024-09-13T11:25:55.709 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0014623590000000001 2024-09-13T11:25:55.709 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.709 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.709 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:25:55.710 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.995900+0000", 2024-09-13T11:25:55.710 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0023271720000000002 2024-09-13T11:25:55.710 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:55.710 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:25:55.710 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:55.710 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.711 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.711 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPushReply(3.21 89/78 [PushReplyOp(3:8534d087:::benchmark_data_smithi177_31116_object5474:head)])", 2024-09-13T11:25:55.711 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:46.632834+0000", 2024-09-13T11:25:55.711 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 128.99200417399999, 2024-09-13T11:25:55.711 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.324805556, 2024-09-13T11:25:55.711 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:25:55.712 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:25:55.712 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-13T11:25:55.712 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:25:55.712 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.712 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:25:55.712 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.632834+0000", 2024-09-13T11:25:55.713 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:55.713 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.713 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.713 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:25:55.713 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.632834+0000", 2024-09-13T11:25:55.713 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:55.713 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.713 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.713 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:25:55.713 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.632835+0000", 2024-09-13T11:25:55.713 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.4270000000000001e-06 2024-09-13T11:25:55.713 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.714 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.714 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:25:55.714 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.632840+0000", 2024-09-13T11:25:55.714 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.7809999999999996e-06 2024-09-13T11:25:55.714 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.714 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.714 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:25:55.714 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.956282+0000", 2024-09-13T11:25:55.714 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.323441639 2024-09-13T11:25:55.714 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.714 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.714 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:25:55.714 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.956293+0000", 2024-09-13T11:25:55.715 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.083e-05 2024-09-13T11:25:55.715 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.715 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.715 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:25:55.715 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.956395+0000", 2024-09-13T11:25:55.715 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00010184200000000001 2024-09-13T11:25:55.715 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.715 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.715 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:25:55.715 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.957640+0000", 2024-09-13T11:25:55.715 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0012450370000000001 2024-09-13T11:25:55.715 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:55.715 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:25:55.716 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:55.716 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.716 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.716 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPushReply(3.1f 89/78 [PushReplyOp(3:ff150976:::benchmark_data_smithi177_31116_object12907:head)])", 2024-09-13T11:25:55.716 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:46.697504+0000", 2024-09-13T11:25:55.716 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 128.927334575, 2024-09-13T11:25:55.716 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.26836818099999998, 2024-09-13T11:25:55.716 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:25:55.716 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:25:55.716 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-13T11:25:55.716 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:25:55.716 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.716 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:25:55.717 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.697504+0000", 2024-09-13T11:25:55.717 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:55.717 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.717 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.717 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:25:55.717 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.697504+0000", 2024-09-13T11:25:55.717 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:55.717 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.717 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.717 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:25:55.717 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.697505+0000", 2024-09-13T11:25:55.717 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0529999999999999e-06 2024-09-13T11:25:55.717 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.717 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.718 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:25:55.718 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.697509+0000", 2024-09-13T11:25:55.718 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.9249999999999997e-06 2024-09-13T11:25:55.718 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.718 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.718 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:25:55.718 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.964086+0000", 2024-09-13T11:25:55.718 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.26657776100000002 2024-09-13T11:25:55.718 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.718 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.718 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:25:55.718 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.964110+0000", 2024-09-13T11:25:55.718 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.3189000000000001e-05 2024-09-13T11:25:55.719 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.719 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.719 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:25:55.719 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.964354+0000", 2024-09-13T11:25:55.719 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00024481399999999999 2024-09-13T11:25:55.719 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.719 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.719 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:25:55.719 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.965872+0000", 2024-09-13T11:25:55.719 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001517439 2024-09-13T11:25:55.719 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:55.719 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:25:55.719 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:55.720 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.720 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.720 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:83223 3.1c 3:3bc6018d:::benchmark_data_smithi177_31116_object83222:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106)", 2024-09-13T11:25:55.720 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:04.511163+0000", 2024-09-13T11:25:55.720 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 111.113674797, 2024-09-13T11:25:55.720 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.47558093200000001, 2024-09-13T11:25:55.720 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:25:55.720 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:25:55.720 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:55.720 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:25:55.720 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:25:55.720 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:55.720 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 83223 2024-09-13T11:25:55.720 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.721 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:25:55.721 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.721 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:25:55.721 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.511163+0000", 2024-09-13T11:25:55.721 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:55.721 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.721 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.721 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:25:55.721 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.511163+0000", 2024-09-13T11:25:55.721 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:55.721 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.721 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.722 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:25:55.722 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.511167+0000", 2024-09-13T11:25:55.722 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.8330000000000004e-06 2024-09-13T11:25:55.722 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.722 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.722 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:25:55.722 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.511219+0000", 2024-09-13T11:25:55.722 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.1771999999999998e-05 2024-09-13T11:25:55.722 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.722 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.722 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:25:55.722 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.511252+0000", 2024-09-13T11:25:55.722 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.3315999999999997e-05 2024-09-13T11:25:55.723 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.723 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.723 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:25:55.723 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.511268+0000", 2024-09-13T11:25:55.723 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5812999999999999e-05 2024-09-13T11:25:55.723 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.723 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.723 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:25:55.723 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.511418+0000", 2024-09-13T11:25:55.723 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000150023 2024-09-13T11:25:55.723 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.723 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.723 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:25:55.724 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.511670+0000", 2024-09-13T11:25:55.724 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00025185499999999999 2024-09-13T11:25:55.724 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.724 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.724 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 7", 2024-09-13T11:25:55.724 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.511905+0000", 2024-09-13T11:25:55.724 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00023493799999999999 2024-09-13T11:25:55.724 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.724 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.724 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:25:55.724 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.514960+0000", 2024-09-13T11:25:55.724 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.003054977 2024-09-13T11:25:55.724 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.724 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.725 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:25:55.725 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.986432+0000", 2024-09-13T11:25:55.725 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.47147209200000001 2024-09-13T11:25:55.725 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.725 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.725 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:25:55.725 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.986635+0000", 2024-09-13T11:25:55.725 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00020309899999999999 2024-09-13T11:25:55.725 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.725 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.725 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:25:55.725 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.986744+0000", 2024-09-13T11:25:55.725 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000109214 2024-09-13T11:25:55.725 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:55.726 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:25:55.726 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:55.726 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.726 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.726 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop_reply(client.4665.0:84803 3.1f e107/78)", 2024-09-13T11:25:55.726 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:05.912627+0000", 2024-09-13T11:25:55.726 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 109.71221088999999, 2024-09-13T11:25:55.726 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.33726566600000002, 2024-09-13T11:25:55.726 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:25:55.726 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:25:55.726 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-13T11:25:55.726 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:25:55.727 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.727 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:25:55.727 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:05.912627+0000", 2024-09-13T11:25:55.727 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:55.727 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.727 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.727 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:25:55.727 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:05.912627+0000", 2024-09-13T11:25:55.727 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:55.727 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.727 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.727 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:25:55.727 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:05.912629+0000", 2024-09-13T11:25:55.728 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.4440000000000001e-06 2024-09-13T11:25:55.728 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.728 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.728 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:25:55.728 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:05.912634+0000", 2024-09-13T11:25:55.728 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.6869999999999999e-06 2024-09-13T11:25:55.728 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.728 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.728 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:25:55.728 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:06.249722+0000", 2024-09-13T11:25:55.728 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.33708766099999998 2024-09-13T11:25:55.728 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.728 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.729 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:25:55.729 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:06.249736+0000", 2024-09-13T11:25:55.729 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3623e-05 2024-09-13T11:25:55.729 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.729 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.729 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:25:55.729 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:06.249878+0000", 2024-09-13T11:25:55.729 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00014264599999999999 2024-09-13T11:25:55.729 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.729 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.729 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:25:55.729 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:06.249893+0000", 2024-09-13T11:25:55.729 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.4605e-05 2024-09-13T11:25:55.730 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:55.730 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:25:55.730 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:55.730 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.730 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.730 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop_reply(client.4665.0:110260 3.f e119/75 ondisk, result = 0)", 2024-09-13T11:25:55.730 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:25.333188+0000", 2024-09-13T11:25:55.730 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 90.291650262999994, 2024-09-13T11:25:55.730 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.29195486199999998, 2024-09-13T11:25:55.730 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:25:55.730 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:25:55.730 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:25:55.730 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:25:55.731 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.731 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:25:55.731 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.333188+0000", 2024-09-13T11:25:55.731 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:55.731 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.731 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.731 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:25:55.731 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.333188+0000", 2024-09-13T11:25:55.731 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:55.731 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.731 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.731 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:25:55.731 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.333190+0000", 2024-09-13T11:25:55.732 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.2120000000000002e-06 2024-09-13T11:25:55.732 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.732 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.732 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:25:55.732 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.333197+0000", 2024-09-13T11:25:55.732 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.7469999999999999e-06 2024-09-13T11:25:55.732 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.732 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.732 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:25:55.732 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.624892+0000", 2024-09-13T11:25:55.732 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.291694913 2024-09-13T11:25:55.732 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.732 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.733 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:25:55.733 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.624907+0000", 2024-09-13T11:25:55.733 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5053e-05 2024-09-13T11:25:55.733 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.733 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.733 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:25:55.733 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.624999+0000", 2024-09-13T11:25:55.733 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.2603999999999996e-05 2024-09-13T11:25:55.733 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.733 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.733 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:25:55.733 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.625011+0000", 2024-09-13T11:25:55.733 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.1253999999999999e-05 2024-09-13T11:25:55.733 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.734 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.734 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:25:55.734 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.625143+0000", 2024-09-13T11:25:55.734 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00013207900000000001 2024-09-13T11:25:55.734 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:55.734 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:25:55.734 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:55.734 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.734 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.734 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:110265 3.1 e119/75 3:8236484d:::benchmark_data_smithi177_31116_object110264:head v 119'3935, mlcod=118'3933)", 2024-09-13T11:25:55.734 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:25.343348+0000", 2024-09-13T11:25:55.734 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 90.281490648000002, 2024-09-13T11:25:55.734 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.60013013500000001, 2024-09-13T11:25:55.735 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:25:55.735 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:25:55.735 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:55.735 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:25:55.735 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.735 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:25:55.735 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.343348+0000", 2024-09-13T11:25:55.735 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:55.735 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.735 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.735 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:25:55.735 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.343348+0000", 2024-09-13T11:25:55.735 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:55.736 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.736 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.736 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:25:55.736 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.343350+0000", 2024-09-13T11:25:55.736 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.576e-06 2024-09-13T11:25:55.736 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.736 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.736 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:25:55.736 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.343443+0000", 2024-09-13T11:25:55.736 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.2989e-05 2024-09-13T11:25:55.736 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.736 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.736 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:25:55.737 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.940863+0000", 2024-09-13T11:25:55.737 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.59742003300000002 2024-09-13T11:25:55.737 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.737 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.737 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:25:55.737 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.940875+0000", 2024-09-13T11:25:55.737 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.2147000000000001e-05 2024-09-13T11:25:55.737 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.737 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.737 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:25:55.737 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.940993+0000", 2024-09-13T11:25:55.737 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000117903 2024-09-13T11:25:55.737 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.737 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.738 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:25:55.738 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.941017+0000", 2024-09-13T11:25:55.738 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.3442999999999999e-05 2024-09-13T11:25:55.738 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.738 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.738 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:25:55.738 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.943347+0000", 2024-09-13T11:25:55.738 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0023299869999999999 2024-09-13T11:25:55.738 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.738 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.738 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:25:55.738 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.943478+0000", 2024-09-13T11:25:55.738 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00013105699999999999 2024-09-13T11:25:55.739 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:55.739 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:25:55.739 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:55.739 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.739 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.739 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop_reply(client.4665.0:129919 3.16 e134/88 ondisk, result = 0)", 2024-09-13T11:25:55.739 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:47.175742+0000", 2024-09-13T11:25:55.739 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 68.449096577999995, 2024-09-13T11:25:55.739 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.27593726299999999, 2024-09-13T11:25:55.739 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:25:55.739 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:25:55.739 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:25:55.739 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:25:55.740 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.740 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:25:55.740 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:47.175742+0000", 2024-09-13T11:25:55.740 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:55.740 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.740 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.740 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:25:55.740 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:47.175742+0000", 2024-09-13T11:25:55.740 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:55.740 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.740 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.740 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:25:55.740 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:47.175744+0000", 2024-09-13T11:25:55.741 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.0030000000000001e-06 2024-09-13T11:25:55.741 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.741 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.741 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:25:55.741 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:47.175751+0000", 2024-09-13T11:25:55.741 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.9779999999999999e-06 2024-09-13T11:25:55.741 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.741 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.741 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:25:55.741 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:47.451490+0000", 2024-09-13T11:25:55.741 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.27573892500000002 2024-09-13T11:25:55.741 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.741 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.741 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:25:55.742 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:47.451496+0000", 2024-09-13T11:25:55.742 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.0079999999999999e-06 2024-09-13T11:25:55.742 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.742 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.742 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:25:55.742 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:47.451560+0000", 2024-09-13T11:25:55.742 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.3955999999999995e-05 2024-09-13T11:25:55.742 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.742 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.742 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:25:55.742 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:47.451567+0000", 2024-09-13T11:25:55.742 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.1980000000000004e-06 2024-09-13T11:25:55.742 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.742 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.743 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:25:55.743 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:47.451679+0000", 2024-09-13T11:25:55.743 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00011219500000000001 2024-09-13T11:25:55.743 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:55.743 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:25:55.743 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:55.743 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.743 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.743 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:209869 3.27 e185/169 3:e6b1fcc5:::benchmark_data_smithi177_31116_object69566:head v 185'6350, mlcod=185'6347)", 2024-09-13T11:25:55.743 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:25:54.301402+0000", 2024-09-13T11:25:55.743 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 1.3234359069999999, 2024-09-13T11:25:55.743 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.30148222499999999, 2024-09-13T11:25:55.743 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:25:55.743 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:25:55.744 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:55.744 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:25:55.744 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.744 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:25:55.744 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:54.301402+0000", 2024-09-13T11:25:55.744 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:55.744 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.744 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.744 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:25:55.744 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:54.301402+0000", 2024-09-13T11:25:55.744 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:55.744 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.744 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.744 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:25:55.745 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:54.301404+0000", 2024-09-13T11:25:55.745 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.1939999999999999e-06 2024-09-13T11:25:55.745 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.745 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.745 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:25:55.745 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:54.301409+0000", 2024-09-13T11:25:55.745 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.0259999999999998e-06 2024-09-13T11:25:55.745 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.745 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.745 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:25:55.745 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:54.601076+0000", 2024-09-13T11:25:55.745 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.29966704799999999 2024-09-13T11:25:55.745 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.745 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.746 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:25:55.746 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:54.601081+0000", 2024-09-13T11:25:55.746 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.7069999999999997e-06 2024-09-13T11:25:55.746 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.746 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.746 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:25:55.746 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:54.601135+0000", 2024-09-13T11:25:55.746 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.3439000000000001e-05 2024-09-13T11:25:55.746 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.746 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.746 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:25:55.746 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:54.601150+0000", 2024-09-13T11:25:55.746 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5443000000000001e-05 2024-09-13T11:25:55.746 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.746 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.747 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:25:55.747 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:54.602854+0000", 2024-09-13T11:25:55.747 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0017035220000000001 2024-09-13T11:25:55.747 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.747 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.747 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:25:55.747 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:54.602885+0000", 2024-09-13T11:25:55.747 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.0846000000000002e-05 2024-09-13T11:25:55.747 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:55.747 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:25:55.747 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:55.747 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:55.747 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:25:55.747 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-09-13T11:25:55.748 INFO:teuthology.orchestra.run.smithi001.stdout:osd.0: {} 2024-09-13T11:25:55.749 INFO:teuthology.orchestra.run.smithi001.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T11:25:55.750 INFO:teuthology.orchestra.run.smithi001.stdout:osd.1: {} 2024-09-13T11:25:55.750 INFO:teuthology.orchestra.run.smithi001.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T11:25:55.750 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_ops_in_flight 2024-09-13T11:25:55.751 INFO:teuthology.orchestra.run.smithi001.stdout:osd.2: {} 2024-09-13T11:25:55.751 INFO:teuthology.orchestra.run.smithi001.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T11:25:55.751 INFO:tasks.ceph.osd.5.smithi092.stderr:2024-09-13T11:25:55.748+0000 7f6f0b705640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26346) UID: 0 2024-09-13T11:25:55.772 INFO:teuthology.orchestra.run.smithi001.stdout:osd.3: {} 2024-09-13T11:25:55.773 INFO:teuthology.orchestra.run.smithi001.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T11:25:55.798 INFO:teuthology.orchestra.run.smithi001.stdout:osd.4: {} 2024-09-13T11:25:55.799 INFO:teuthology.orchestra.run.smithi001.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T11:25:55.824 INFO:teuthology.orchestra.run.smithi001.stdout:osd.5: {} 2024-09-13T11:25:55.824 INFO:teuthology.orchestra.run.smithi001.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T11:25:55.850 INFO:tasks.ceph.osd.15.smithi177.stderr:2024-09-13T11:25:55.846+0000 7f9d6ce77640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26578) UID: 0 2024-09-13T11:25:55.851 INFO:teuthology.orchestra.run.smithi001.stdout:osd.6: {} 2024-09-13T11:25:55.851 INFO:teuthology.orchestra.run.smithi001.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T11:25:55.882 INFO:teuthology.orchestra.run.smithi001.stdout:osd.7: {} 2024-09-13T11:25:55.882 INFO:teuthology.orchestra.run.smithi001.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T11:25:55.910 INFO:teuthology.orchestra.run.smithi001.stdout:osd.8: {} 2024-09-13T11:25:55.910 INFO:teuthology.orchestra.run.smithi001.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-13T11:25:55.935 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-09-13T11:25:55.936 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [ 2024-09-13T11:25:55.936 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.936 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:211968 3.5 3:a3e15c63:::benchmark_data_smithi177_31116_object71665:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e185)", 2024-09-13T11:25:55.936 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:25:55.923145+0000", 2024-09-13T11:25:55.936 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 0.013187083, 2024-09-13T11:25:55.936 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.013207585000000001, 2024-09-13T11:25:55.936 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:25:55.936 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:25:55.936 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:25:55.936 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:25:55.936 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:25:55.937 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:55.937 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 211968 2024-09-13T11:25:55.937 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.937 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:25:55.937 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.937 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:25:55.937 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:55.923145+0000", 2024-09-13T11:25:55.937 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:55.937 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.937 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.937 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:25:55.937 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:55.923145+0000", 2024-09-13T11:25:55.937 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:55.938 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.938 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.938 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:25:55.938 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:55.923146+0000", 2024-09-13T11:25:55.938 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.7099999999999999e-06 2024-09-13T11:25:55.938 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.938 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.938 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:25:55.938 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:55.923151+0000", 2024-09-13T11:25:55.938 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.7879999999999997e-06 2024-09-13T11:25:55.938 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.938 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.938 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:25:55.939 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:55.923162+0000", 2024-09-13T11:25:55.939 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.1175000000000001e-05 2024-09-13T11:25:55.939 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.939 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.939 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:25:55.939 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:55.923170+0000", 2024-09-13T11:25:55.939 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.3549999999999999e-06 2024-09-13T11:25:55.939 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.939 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.939 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:25:55.939 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:55.936196+0000", 2024-09-13T11:25:55.939 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.013026203 2024-09-13T11:25:55.939 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.939 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.940 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:25:55.940 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:55.936310+0000", 2024-09-13T11:25:55.940 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00011411199999999999 2024-09-13T11:25:55.940 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:55.940 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:25:55.940 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:55.940 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.940 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.940 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:211981 3.1d e185/102 3:bb494d19:::benchmark_data_smithi177_31116_object71678:head v 185'8241, mlcod=185'8239)", 2024-09-13T11:25:55.940 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:25:55.934655+0000", 2024-09-13T11:25:55.940 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 0.001677071, 2024-09-13T11:25:55.940 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001767743, 2024-09-13T11:25:55.940 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:25:55.941 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:25:55.941 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:25:55.941 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:25:55.941 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.941 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:25:55.941 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:55.934655+0000", 2024-09-13T11:25:55.941 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:55.941 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.941 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.941 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:25:55.941 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:55.934655+0000", 2024-09-13T11:25:55.941 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:25:55.941 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.941 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.942 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:25:55.942 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:55.934656+0000", 2024-09-13T11:25:55.942 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.187e-06 2024-09-13T11:25:55.942 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.942 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.942 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:25:55.942 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:55.934661+0000", 2024-09-13T11:25:55.942 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.5450000000000003e-06 2024-09-13T11:25:55.942 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.942 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.942 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:25:55.942 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:55.934685+0000", 2024-09-13T11:25:55.942 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.3241e-05 2024-09-13T11:25:55.942 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.943 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.943 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:25:55.943 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:55.934691+0000", 2024-09-13T11:25:55.943 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.0759999999999999e-06 2024-09-13T11:25:55.943 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.943 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.943 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:25:55.943 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:55.936176+0000", 2024-09-13T11:25:55.943 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001485042 2024-09-13T11:25:55.943 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:25:55.943 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:25:55.943 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:25:55.943 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:55.936195+0000", 2024-09-13T11:25:55.943 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.8912000000000001e-05 2024-09-13T11:25:55.944 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:55.944 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:25:55.944 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:55.944 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:25:55.944 INFO:teuthology.orchestra.run.smithi177.stdout: ], 2024-09-13T11:25:55.944 INFO:teuthology.orchestra.run.smithi177.stdout: "num_ops": 2 2024-09-13T11:25:55.944 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-09-13T11:25:55.944 INFO:teuthology.orchestra.run.smithi001.stdout:osd.9: {} 2024-09-13T11:25:55.945 INFO:teuthology.orchestra.run.smithi001.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-13T11:25:55.951 INFO:tasks.ceph.osd.1.smithi092.stderr:2024-09-13T11:25:55.948+0000 7fe7789c5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33718) UID: 0 2024-09-13T11:25:55.955 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_blocked_ops 2024-09-13T11:25:55.965 INFO:teuthology.orchestra.run.smithi001.stderr:noscrub is unset 2024-09-13T11:25:55.971 INFO:teuthology.orchestra.run.smithi001.stdout:osd.10: {} 2024-09-13T11:25:55.972 INFO:teuthology.orchestra.run.smithi001.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-13T11:25:55.998 INFO:teuthology.orchestra.run.smithi001.stdout:osd.11: {} 2024-09-13T11:25:55.998 INFO:teuthology.orchestra.run.smithi001.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-13T11:25:56.036 INFO:teuthology.orchestra.run.smithi001.stdout:osd.12: {} 2024-09-13T11:25:56.036 INFO:teuthology.orchestra.run.smithi001.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-13T11:25:56.054 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-13T11:25:56.050+0000 7f6a68775640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26307) UID: 0 2024-09-13T11:25:56.063 INFO:teuthology.orchestra.run.smithi001.stdout:osd.13: {} 2024-09-13T11:25:56.063 INFO:teuthology.orchestra.run.smithi001.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-13T11:25:56.092 INFO:teuthology.orchestra.run.smithi001.stdout:osd.14: {} 2024-09-13T11:25:56.092 INFO:teuthology.orchestra.run.smithi001.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-13T11:25:56.122 INFO:teuthology.orchestra.run.smithi001.stdout:osd.15: {} 2024-09-13T11:25:56.122 INFO:teuthology.orchestra.run.smithi001.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-13T11:25:56.135 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-09-13T11:25:56.136 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:25:56.154 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_historic_ops 2024-09-13T11:25:56.155 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-09-13T11:25:56.151+0000 7f97befc9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26315) UID: 0 2024-09-13T11:25:56.255 INFO:tasks.ceph.osd.4.smithi001.stderr:2024-09-13T11:25:56.250+0000 7ff8e0218640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26310) UID: 0 2024-09-13T11:25:56.293 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-09-13T11:25:56.294 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:25:56.312 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-13T11:25:56.357 INFO:tasks.ceph.osd.5.smithi092.stderr:2024-09-13T11:25:56.356+0000 7f6f0b705640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26346) UID: 0 2024-09-13T11:25:56.453 INFO:teuthology.orchestra.run.smithi092.stdout:ERROR: (22) Invalid argument 2024-09-13T11:25:56.453 INFO:teuthology.orchestra.run.smithi092.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:25:56.461 INFO:tasks.ceph.osd.7.smithi177.stderr:2024-09-13T11:25:56.454+0000 7f897b51d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26575) UID: 0 2024-09-13T11:25:56.475 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-13T11:25:56.559 INFO:tasks.ceph.osd.7.smithi177.stderr:2024-09-13T11:25:56.554+0000 7f897b51d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26575) UID: 0 2024-09-13T11:25:56.657 INFO:teuthology.orchestra.run.smithi092.stdout:ERROR: (22) Invalid argument 2024-09-13T11:25:56.657 INFO:teuthology.orchestra.run.smithi092.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:25:56.660 INFO:tasks.ceph.osd.4.smithi001.stderr:2024-09-13T11:25:56.658+0000 7ff8e0218640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26310) UID: 0 2024-09-13T11:25:56.674 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-13T11:25:56.760 INFO:tasks.ceph.osd.13.smithi092.stderr:2024-09-13T11:25:56.756+0000 7f3e5afea640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26350) UID: 0 2024-09-13T11:25:56.815 INFO:teuthology.orchestra.run.smithi092.stdout:ERROR: (22) Invalid argument 2024-09-13T11:25:56.815 INFO:teuthology.orchestra.run.smithi092.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:25:56.835 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_ops_in_flight 2024-09-13T11:25:56.861 INFO:tasks.ceph.osd.11.smithi177.stderr:2024-09-13T11:25:56.858+0000 7f24792c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26580) UID: 0 2024-09-13T11:25:56.962 INFO:tasks.ceph.osd.11.smithi177.stderr:2024-09-13T11:25:56.958+0000 7f24792c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26580) UID: 0 2024-09-13T11:25:56.982 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-13T11:25:56.982 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:25:56.997 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_blocked_ops 2024-09-13T11:25:57.062 INFO:tasks.ceph.osd.12.smithi001.stderr:2024-09-13T11:25:57.058+0000 7f892c6e6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26314) UID: 0 2024-09-13T11:25:57.129 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-13T11:25:57.129 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:25:57.146 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_historic_ops 2024-09-13T11:25:57.163 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-13T11:25:57.158+0000 7f6a68775640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26307) UID: 0 2024-09-13T11:25:57.263 INFO:tasks.ceph.osd.12.smithi001.stderr:2024-09-13T11:25:57.262+0000 7f892c6e6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26314) UID: 0 2024-09-13T11:25:57.279 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-13T11:25:57.279 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:25:57.294 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-13T11:25:57.364 INFO:tasks.ceph.osd.12.smithi001.stderr:2024-09-13T11:25:57.362+0000 7f892c6e6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26314) UID: 0 2024-09-13T11:25:57.434 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-13T11:25:57.434 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:25:57.450 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-13T11:25:57.466 INFO:tasks.ceph.osd.13.smithi092.stderr:2024-09-13T11:25:57.464+0000 7f3e5afea640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26350) UID: 0 2024-09-13T11:25:57.567 INFO:tasks.ceph.osd.13.smithi092.stderr:2024-09-13T11:25:57.564+0000 7f3e5afea640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26350) UID: 0 2024-09-13T11:25:57.628 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-13T11:25:57.628 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:25:57.644 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-13T11:25:57.668 INFO:tasks.ceph.osd.15.smithi177.stderr:2024-09-13T11:25:57.666+0000 7f9d6ce77640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26578) UID: 0 2024-09-13T11:25:57.769 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-09-13T11:25:57.767+0000 7f97befc9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26315) UID: 0 2024-09-13T11:25:57.783 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-13T11:25:57.784 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:25:57.800 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-13T11:25:57.870 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-09-13T11:25:57.867+0000 7f97befc9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26315) UID: 0 2024-09-13T11:25:57.934 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-13T11:25:57.934 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:25:57.953 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-13T11:25:57.971 INFO:tasks.ceph.osd.1.smithi092.stderr:2024-09-13T11:25:57.968+0000 7fe7789c5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33718) UID: 0 2024-09-13T11:25:57.979 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 15] out_osds: [14] dead_osds: [] live_osds: [0, 4, 8, 12, 5, 9, 13, 2, 6, 10, 3, 7, 11, 15, 1, 14] 2024-09-13T11:25:57.980 INFO:tasks.thrashosds.thrasher:choose_action: min_in 2 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-13T11:25:57.980 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2024-09-13T11:25:57.980 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump -f json-pretty 2024-09-13T11:25:57.993 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-13T11:25:58.072 INFO:tasks.ceph.osd.7.smithi177.stderr:2024-09-13T11:25:58.070+0000 7f897b51d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26575) UID: 0 2024-09-13T11:25:58.114 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-13T11:25:58.115 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:25:58.136 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-13T11:25:58.141 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T11:25:58.173 INFO:tasks.ceph.osd.5.smithi092.stderr:2024-09-13T11:25:58.172+0000 7f6f0b705640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26346) UID: 0 2024-09-13T11:25:58.274 INFO:tasks.ceph.osd.11.smithi177.stderr:2024-09-13T11:25:58.269+0000 7f24792c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26580) UID: 0 2024-09-13T11:25:58.325 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-13T11:25:58.325 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:25:58.356 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-09-13T11:25:58.403 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-13T11:25:58.398+0000 7f6a68775640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26307) UID: 0 2024-09-13T11:25:58.447 INFO:teuthology.orchestra.run.smithi001.stdout:osd.0: {} 2024-09-13T11:25:58.447 INFO:teuthology.orchestra.run.smithi001.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T11:25:58.477 INFO:tasks.ceph.osd.5.smithi092.stderr:2024-09-13T11:25:58.476+0000 7f6f0b705640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26346) UID: 0 2024-09-13T11:25:58.490 INFO:teuthology.orchestra.run.smithi001.stdout:osd.1: {} 2024-09-13T11:25:58.490 INFO:teuthology.orchestra.run.smithi001.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T11:25:58.524 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-13T11:25:58.524 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:25:58.530 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-13T11:25:58.530 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-13T11:25:58.531 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 186, 2024-09-13T11:25:58.531 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "2fd68ac7-5581-4d90-80a4-30bcf5252a7a", 2024-09-13T11:25:58.531 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-09-13T11:21:57.223153+0000", 2024-09-13T11:25:58.531 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-09-13T11:25:55.955245+0000", 2024-09-13T11:25:58.531 INFO:teuthology.orchestra.run.smithi001.stdout: "last_up_change": "2024-09-13T11:24:55.968636+0000", 2024-09-13T11:25:58.531 INFO:teuthology.orchestra.run.smithi001.stdout: "last_in_change": "2024-09-13T11:24:28.080124+0000", 2024-09-13T11:25:58.531 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": "nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-13T11:25:58.531 INFO:teuthology.orchestra.run.smithi001.stdout: "flags_num": 5804032, 2024-09-13T11:25:58.531 INFO:teuthology.orchestra.run.smithi001.stdout: "flags_set": [ 2024-09-13T11:25:58.531 INFO:teuthology.orchestra.run.smithi001.stdout: "nodeep-scrub", 2024-09-13T11:25:58.531 INFO:teuthology.orchestra.run.smithi001.stdout: "pglog_hardlimit", 2024-09-13T11:25:58.531 INFO:teuthology.orchestra.run.smithi001.stdout: "purged_snapdirs", 2024-09-13T11:25:58.531 INFO:teuthology.orchestra.run.smithi001.stdout: "recovery_deletes", 2024-09-13T11:25:58.532 INFO:teuthology.orchestra.run.smithi001.stdout: "sortbitwise" 2024-09-13T11:25:58.532 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-09-13T11:25:58.532 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_version": 7, 2024-09-13T11:25:58.532 INFO:teuthology.orchestra.run.smithi001.stdout: "full_ratio": 0.94999998807907104, 2024-09-13T11:25:58.532 INFO:teuthology.orchestra.run.smithi001.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-09-13T11:25:58.532 INFO:teuthology.orchestra.run.smithi001.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-13T11:25:58.532 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_snapshot": "", 2024-09-13T11:25:58.532 INFO:teuthology.orchestra.run.smithi001.stdout: "pool_max": 3, 2024-09-13T11:25:58.532 INFO:teuthology.orchestra.run.smithi001.stdout: "max_osd": 16, 2024-09-13T11:25:58.532 INFO:teuthology.orchestra.run.smithi001.stdout: "require_min_compat_client": "luminous", 2024-09-13T11:25:58.532 INFO:teuthology.orchestra.run.smithi001.stdout: "min_compat_client": "luminous", 2024-09-13T11:25:58.532 INFO:teuthology.orchestra.run.smithi001.stdout: "require_osd_release": "squid", 2024-09-13T11:25:58.532 INFO:teuthology.orchestra.run.smithi001.stdout: "allow_crimson": false, 2024-09-13T11:25:58.533 INFO:teuthology.orchestra.run.smithi001.stdout: "pools": [ 2024-09-13T11:25:58.533 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:58.533 INFO:teuthology.orchestra.run.smithi001.stdout: "pool": 1, 2024-09-13T11:25:58.533 INFO:teuthology.orchestra.run.smithi001.stdout: "pool_name": ".mgr", 2024-09-13T11:25:58.533 INFO:teuthology.orchestra.run.smithi001.stdout: "create_time": "2024-09-13T11:22:10.162266+0000", 2024-09-13T11:25:58.533 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-09-13T11:25:58.533 INFO:teuthology.orchestra.run.smithi001.stdout: "flags_names": "hashpspool", 2024-09-13T11:25:58.533 INFO:teuthology.orchestra.run.smithi001.stdout: "type": 1, 2024-09-13T11:25:58.533 INFO:teuthology.orchestra.run.smithi001.stdout: "size": 2, 2024-09-13T11:25:58.533 INFO:teuthology.orchestra.run.smithi001.stdout: "min_size": 2, 2024-09-13T11:25:58.533 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_rule": 0, 2024-09-13T11:25:58.533 INFO:teuthology.orchestra.run.smithi001.stdout: "peering_crush_bucket_count": 0, 2024-09-13T11:25:58.533 INFO:teuthology.orchestra.run.smithi001.stdout: "peering_crush_bucket_target": 0, 2024-09-13T11:25:58.534 INFO:teuthology.orchestra.run.smithi001.stdout: "peering_crush_bucket_barrier": 0, 2024-09-13T11:25:58.534 INFO:teuthology.orchestra.run.smithi001.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-13T11:25:58.534 INFO:teuthology.orchestra.run.smithi001.stdout: "is_stretch_pool": false, 2024-09-13T11:25:58.534 INFO:teuthology.orchestra.run.smithi001.stdout: "object_hash": 2, 2024-09-13T11:25:58.534 INFO:teuthology.orchestra.run.smithi001.stdout: "pg_autoscale_mode": "off", 2024-09-13T11:25:58.534 INFO:teuthology.orchestra.run.smithi001.stdout: "pg_num": 1, 2024-09-13T11:25:58.534 INFO:teuthology.orchestra.run.smithi001.stdout: "pg_placement_num": 1, 2024-09-13T11:25:58.534 INFO:teuthology.orchestra.run.smithi001.stdout: "pg_placement_num_target": 1, 2024-09-13T11:25:58.534 INFO:teuthology.orchestra.run.smithi001.stdout: "pg_num_target": 1, 2024-09-13T11:25:58.534 INFO:teuthology.orchestra.run.smithi001.stdout: "pg_num_pending": 1, 2024-09-13T11:25:58.534 INFO:teuthology.orchestra.run.smithi001.stdout: "last_pg_merge_meta": { 2024-09-13T11:25:58.534 INFO:teuthology.orchestra.run.smithi001.stdout: "source_pgid": "0.0", 2024-09-13T11:25:58.534 INFO:teuthology.orchestra.run.smithi001.stdout: "ready_epoch": 0, 2024-09-13T11:25:58.535 INFO:teuthology.orchestra.run.smithi001.stdout: "last_epoch_started": 0, 2024-09-13T11:25:58.535 INFO:teuthology.orchestra.run.smithi001.stdout: "last_epoch_clean": 0, 2024-09-13T11:25:58.535 INFO:teuthology.orchestra.run.smithi001.stdout: "source_version": "0'0", 2024-09-13T11:25:58.535 INFO:teuthology.orchestra.run.smithi001.stdout: "target_version": "0'0" 2024-09-13T11:25:58.535 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:58.535 INFO:teuthology.orchestra.run.smithi001.stdout: "last_change": "25", 2024-09-13T11:25:58.535 INFO:teuthology.orchestra.run.smithi001.stdout: "last_force_op_resend": "0", 2024-09-13T11:25:58.535 INFO:teuthology.orchestra.run.smithi001.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-13T11:25:58.535 INFO:teuthology.orchestra.run.smithi001.stdout: "last_force_op_resend_preluminous": "0", 2024-09-13T11:25:58.535 INFO:teuthology.orchestra.run.smithi001.stdout: "auid": 0, 2024-09-13T11:25:58.535 INFO:teuthology.orchestra.run.smithi001.stdout: "snap_mode": "selfmanaged", 2024-09-13T11:25:58.535 INFO:teuthology.orchestra.run.smithi001.stdout: "snap_seq": 0, 2024-09-13T11:25:58.535 INFO:teuthology.orchestra.run.smithi001.stdout: "snap_epoch": 0, 2024-09-13T11:25:58.536 INFO:teuthology.orchestra.run.smithi001.stdout: "pool_snaps": [], 2024-09-13T11:25:58.536 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_snaps": "[]", 2024-09-13T11:25:58.536 INFO:teuthology.orchestra.run.smithi001.stdout: "quota_max_bytes": 0, 2024-09-13T11:25:58.536 INFO:teuthology.orchestra.run.smithi001.stdout: "quota_max_objects": 0, 2024-09-13T11:25:58.536 INFO:teuthology.orchestra.run.smithi001.stdout: "tiers": [], 2024-09-13T11:25:58.536 INFO:teuthology.orchestra.run.smithi001.stdout: "tier_of": -1, 2024-09-13T11:25:58.536 INFO:teuthology.orchestra.run.smithi001.stdout: "read_tier": -1, 2024-09-13T11:25:58.536 INFO:teuthology.orchestra.run.smithi001.stdout: "write_tier": -1, 2024-09-13T11:25:58.536 INFO:teuthology.orchestra.run.smithi001.stdout: "cache_mode": "none", 2024-09-13T11:25:58.536 INFO:teuthology.orchestra.run.smithi001.stdout: "target_max_bytes": 0, 2024-09-13T11:25:58.536 INFO:teuthology.orchestra.run.smithi001.stdout: "target_max_objects": 0, 2024-09-13T11:25:58.536 INFO:teuthology.orchestra.run.smithi001.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-13T11:25:58.536 INFO:teuthology.orchestra.run.smithi001.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-13T11:25:58.537 INFO:teuthology.orchestra.run.smithi001.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-13T11:25:58.537 INFO:teuthology.orchestra.run.smithi001.stdout: "cache_min_flush_age": 0, 2024-09-13T11:25:58.537 INFO:teuthology.orchestra.run.smithi001.stdout: "cache_min_evict_age": 0, 2024-09-13T11:25:58.537 INFO:teuthology.orchestra.run.smithi001.stdout: "erasure_code_profile": "", 2024-09-13T11:25:58.537 INFO:teuthology.orchestra.run.smithi001.stdout: "hit_set_params": { 2024-09-13T11:25:58.537 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "none" 2024-09-13T11:25:58.537 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:58.537 INFO:teuthology.orchestra.run.smithi001.stdout: "hit_set_period": 0, 2024-09-13T11:25:58.537 INFO:teuthology.orchestra.run.smithi001.stdout: "hit_set_count": 0, 2024-09-13T11:25:58.537 INFO:teuthology.orchestra.run.smithi001.stdout: "use_gmt_hitset": true, 2024-09-13T11:25:58.537 INFO:teuthology.orchestra.run.smithi001.stdout: "min_read_recency_for_promote": 0, 2024-09-13T11:25:58.537 INFO:teuthology.orchestra.run.smithi001.stdout: "min_write_recency_for_promote": 0, 2024-09-13T11:25:58.538 INFO:teuthology.orchestra.run.smithi001.stdout: "hit_set_grade_decay_rate": 0, 2024-09-13T11:25:58.538 INFO:teuthology.orchestra.run.smithi001.stdout: "hit_set_search_last_n": 0, 2024-09-13T11:25:58.538 INFO:teuthology.orchestra.run.smithi001.stdout: "grade_table": [], 2024-09-13T11:25:58.538 INFO:teuthology.orchestra.run.smithi001.stdout: "stripe_width": 0, 2024-09-13T11:25:58.538 INFO:teuthology.orchestra.run.smithi001.stdout: "expected_num_objects": 0, 2024-09-13T11:25:58.538 INFO:teuthology.orchestra.run.smithi001.stdout: "fast_read": false, 2024-09-13T11:25:58.538 INFO:teuthology.orchestra.run.smithi001.stdout: "options": { 2024-09-13T11:25:58.538 INFO:teuthology.orchestra.run.smithi001.stdout: "pg_num_max": 32, 2024-09-13T11:25:58.538 INFO:teuthology.orchestra.run.smithi001.stdout: "pg_num_min": 1 2024-09-13T11:25:58.538 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:58.538 INFO:teuthology.orchestra.run.smithi001.stdout: "application_metadata": { 2024-09-13T11:25:58.538 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": {} 2024-09-13T11:25:58.538 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:58.538 INFO:teuthology.orchestra.run.smithi001.stdout: "read_balance": { 2024-09-13T11:25:58.539 INFO:teuthology.orchestra.run.smithi001.stdout: "score_type": "Fair distribution", 2024-09-13T11:25:58.539 INFO:teuthology.orchestra.run.smithi001.stdout: "score_acting": 15.380000114440918, 2024-09-13T11:25:58.539 INFO:teuthology.orchestra.run.smithi001.stdout: "score_stable": 15.380000114440918, 2024-09-13T11:25:58.539 INFO:teuthology.orchestra.run.smithi001.stdout: "optimal_score": 0.12999999523162842, 2024-09-13T11:25:58.539 INFO:teuthology.orchestra.run.smithi001.stdout: "raw_score_acting": 2, 2024-09-13T11:25:58.539 INFO:teuthology.orchestra.run.smithi001.stdout: "raw_score_stable": 2, 2024-09-13T11:25:58.539 INFO:teuthology.orchestra.run.smithi001.stdout: "primary_affinity_weighted": 1, 2024-09-13T11:25:58.539 INFO:teuthology.orchestra.run.smithi001.stdout: "average_primary_affinity": 1, 2024-09-13T11:25:58.539 INFO:teuthology.orchestra.run.smithi001.stdout: "average_primary_affinity_weighted": 1 2024-09-13T11:25:58.539 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:58.539 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:58.539 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:58.539 INFO:teuthology.orchestra.run.smithi001.stdout: "pool": 2, 2024-09-13T11:25:58.539 INFO:teuthology.orchestra.run.smithi001.stdout: "pool_name": "rbd", 2024-09-13T11:25:58.540 INFO:teuthology.orchestra.run.smithi001.stdout: "create_time": "2024-09-13T11:22:12.235549+0000", 2024-09-13T11:25:58.540 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 8193, 2024-09-13T11:25:58.540 INFO:teuthology.orchestra.run.smithi001.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-09-13T11:25:58.540 INFO:teuthology.orchestra.run.smithi001.stdout: "type": 1, 2024-09-13T11:25:58.540 INFO:teuthology.orchestra.run.smithi001.stdout: "size": 2, 2024-09-13T11:25:58.540 INFO:teuthology.orchestra.run.smithi001.stdout: "min_size": 2, 2024-09-13T11:25:58.540 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_rule": 0, 2024-09-13T11:25:58.540 INFO:teuthology.orchestra.run.smithi001.stdout: "peering_crush_bucket_count": 0, 2024-09-13T11:25:58.540 INFO:teuthology.orchestra.run.smithi001.stdout: "peering_crush_bucket_target": 0, 2024-09-13T11:25:58.540 INFO:teuthology.orchestra.run.smithi001.stdout: "peering_crush_bucket_barrier": 0, 2024-09-13T11:25:58.540 INFO:teuthology.orchestra.run.smithi001.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-13T11:25:58.540 INFO:teuthology.orchestra.run.smithi001.stdout: "is_stretch_pool": false, 2024-09-13T11:25:58.540 INFO:teuthology.orchestra.run.smithi001.stdout: "object_hash": 2, 2024-09-13T11:25:58.541 INFO:teuthology.orchestra.run.smithi001.stdout: "pg_autoscale_mode": "off", 2024-09-13T11:25:58.541 INFO:teuthology.orchestra.run.smithi001.stdout: "pg_num": 8, 2024-09-13T11:25:58.541 INFO:teuthology.orchestra.run.smithi001.stdout: "pg_placement_num": 8, 2024-09-13T11:25:58.541 INFO:teuthology.orchestra.run.smithi001.stdout: "pg_placement_num_target": 8, 2024-09-13T11:25:58.541 INFO:teuthology.orchestra.run.smithi001.stdout: "pg_num_target": 8, 2024-09-13T11:25:58.541 INFO:teuthology.orchestra.run.smithi001.stdout: "pg_num_pending": 8, 2024-09-13T11:25:58.541 INFO:teuthology.orchestra.run.smithi001.stdout: "last_pg_merge_meta": { 2024-09-13T11:25:58.541 INFO:teuthology.orchestra.run.smithi001.stdout: "source_pgid": "0.0", 2024-09-13T11:25:58.541 INFO:teuthology.orchestra.run.smithi001.stdout: "ready_epoch": 0, 2024-09-13T11:25:58.541 INFO:teuthology.orchestra.run.smithi001.stdout: "last_epoch_started": 0, 2024-09-13T11:25:58.541 INFO:teuthology.orchestra.run.smithi001.stdout: "last_epoch_clean": 0, 2024-09-13T11:25:58.541 INFO:teuthology.orchestra.run.smithi001.stdout: "source_version": "0'0", 2024-09-13T11:25:58.541 INFO:teuthology.orchestra.run.smithi001.stdout: "target_version": "0'0" 2024-09-13T11:25:58.542 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:58.542 INFO:teuthology.orchestra.run.smithi001.stdout: "last_change": "28", 2024-09-13T11:25:58.542 INFO:teuthology.orchestra.run.smithi001.stdout: "last_force_op_resend": "0", 2024-09-13T11:25:58.542 INFO:teuthology.orchestra.run.smithi001.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-13T11:25:58.542 INFO:teuthology.orchestra.run.smithi001.stdout: "last_force_op_resend_preluminous": "0", 2024-09-13T11:25:58.542 INFO:teuthology.orchestra.run.smithi001.stdout: "auid": 0, 2024-09-13T11:25:58.542 INFO:teuthology.orchestra.run.smithi001.stdout: "snap_mode": "selfmanaged", 2024-09-13T11:25:58.542 INFO:teuthology.orchestra.run.smithi001.stdout: "snap_seq": 2, 2024-09-13T11:25:58.542 INFO:teuthology.orchestra.run.smithi001.stdout: "snap_epoch": 28, 2024-09-13T11:25:58.542 INFO:teuthology.orchestra.run.smithi001.stdout: "pool_snaps": [], 2024-09-13T11:25:58.542 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_snaps": "[]", 2024-09-13T11:25:58.542 INFO:teuthology.orchestra.run.smithi001.stdout: "quota_max_bytes": 0, 2024-09-13T11:25:58.542 INFO:teuthology.orchestra.run.smithi001.stdout: "quota_max_objects": 0, 2024-09-13T11:25:58.542 INFO:teuthology.orchestra.run.smithi001.stdout: "tiers": [], 2024-09-13T11:25:58.542 INFO:teuthology.orchestra.run.smithi001.stdout: "tier_of": -1, 2024-09-13T11:25:58.543 INFO:teuthology.orchestra.run.smithi001.stdout: "read_tier": -1, 2024-09-13T11:25:58.543 INFO:teuthology.orchestra.run.smithi001.stdout: "write_tier": -1, 2024-09-13T11:25:58.543 INFO:teuthology.orchestra.run.smithi001.stdout: "cache_mode": "none", 2024-09-13T11:25:58.543 INFO:teuthology.orchestra.run.smithi001.stdout: "target_max_bytes": 0, 2024-09-13T11:25:58.543 INFO:teuthology.orchestra.run.smithi001.stdout: "target_max_objects": 0, 2024-09-13T11:25:58.543 INFO:teuthology.orchestra.run.smithi001.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-13T11:25:58.543 INFO:teuthology.orchestra.run.smithi001.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-13T11:25:58.543 INFO:teuthology.orchestra.run.smithi001.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-13T11:25:58.543 INFO:teuthology.orchestra.run.smithi001.stdout: "cache_min_flush_age": 0, 2024-09-13T11:25:58.543 INFO:teuthology.orchestra.run.smithi001.stdout: "cache_min_evict_age": 0, 2024-09-13T11:25:58.543 INFO:teuthology.orchestra.run.smithi001.stdout: "erasure_code_profile": "", 2024-09-13T11:25:58.543 INFO:teuthology.orchestra.run.smithi001.stdout: "hit_set_params": { 2024-09-13T11:25:58.543 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "none" 2024-09-13T11:25:58.543 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:58.543 INFO:teuthology.orchestra.run.smithi001.stdout: "hit_set_period": 0, 2024-09-13T11:25:58.544 INFO:teuthology.orchestra.run.smithi001.stdout: "hit_set_count": 0, 2024-09-13T11:25:58.544 INFO:teuthology.orchestra.run.smithi001.stdout: "use_gmt_hitset": true, 2024-09-13T11:25:58.544 INFO:teuthology.orchestra.run.smithi001.stdout: "min_read_recency_for_promote": 0, 2024-09-13T11:25:58.544 INFO:teuthology.orchestra.run.smithi001.stdout: "min_write_recency_for_promote": 0, 2024-09-13T11:25:58.544 INFO:teuthology.orchestra.run.smithi001.stdout: "hit_set_grade_decay_rate": 0, 2024-09-13T11:25:58.544 INFO:teuthology.orchestra.run.smithi001.stdout: "hit_set_search_last_n": 0, 2024-09-13T11:25:58.544 INFO:teuthology.orchestra.run.smithi001.stdout: "grade_table": [], 2024-09-13T11:25:58.544 INFO:teuthology.orchestra.run.smithi001.stdout: "stripe_width": 0, 2024-09-13T11:25:58.544 INFO:teuthology.orchestra.run.smithi001.stdout: "expected_num_objects": 0, 2024-09-13T11:25:58.544 INFO:teuthology.orchestra.run.smithi001.stdout: "fast_read": false, 2024-09-13T11:25:58.544 INFO:teuthology.orchestra.run.smithi001.stdout: "options": {}, 2024-09-13T11:25:58.544 INFO:teuthology.orchestra.run.smithi001.stdout: "application_metadata": { 2024-09-13T11:25:58.544 INFO:teuthology.orchestra.run.smithi001.stdout: "rbd": {} 2024-09-13T11:25:58.545 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:58.545 INFO:teuthology.orchestra.run.smithi001.stdout: "read_balance": { 2024-09-13T11:25:58.545 INFO:teuthology.orchestra.run.smithi001.stdout: "score_type": "Fair distribution", 2024-09-13T11:25:58.545 INFO:teuthology.orchestra.run.smithi001.stdout: "score_acting": 3.9700000286102295, 2024-09-13T11:25:58.545 INFO:teuthology.orchestra.run.smithi001.stdout: "score_stable": 3.9700000286102295, 2024-09-13T11:25:58.545 INFO:teuthology.orchestra.run.smithi001.stdout: "optimal_score": 0.62999999523162842, 2024-09-13T11:25:58.545 INFO:teuthology.orchestra.run.smithi001.stdout: "raw_score_acting": 2.5, 2024-09-13T11:25:58.545 INFO:teuthology.orchestra.run.smithi001.stdout: "raw_score_stable": 2.5, 2024-09-13T11:25:58.545 INFO:teuthology.orchestra.run.smithi001.stdout: "primary_affinity_weighted": 1, 2024-09-13T11:25:58.545 INFO:teuthology.orchestra.run.smithi001.stdout: "average_primary_affinity": 1, 2024-09-13T11:25:58.545 INFO:teuthology.orchestra.run.smithi001.stdout: "average_primary_affinity_weighted": 1 2024-09-13T11:25:58.545 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:58.545 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:58.545 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:58.546 INFO:teuthology.orchestra.run.smithi001.stdout: "pool": 3, 2024-09-13T11:25:58.546 INFO:teuthology.orchestra.run.smithi001.stdout: "pool_name": "unique_pool_0", 2024-09-13T11:25:58.546 INFO:teuthology.orchestra.run.smithi001.stdout: "create_time": "2024-09-13T11:22:24.305521+0000", 2024-09-13T11:25:58.546 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-09-13T11:25:58.546 INFO:teuthology.orchestra.run.smithi001.stdout: "flags_names": "hashpspool", 2024-09-13T11:25:58.546 INFO:teuthology.orchestra.run.smithi001.stdout: "type": 1, 2024-09-13T11:25:58.546 INFO:teuthology.orchestra.run.smithi001.stdout: "size": 2, 2024-09-13T11:25:58.546 INFO:teuthology.orchestra.run.smithi001.stdout: "min_size": 2, 2024-09-13T11:25:58.546 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_rule": 0, 2024-09-13T11:25:58.546 INFO:teuthology.orchestra.run.smithi001.stdout: "peering_crush_bucket_count": 0, 2024-09-13T11:25:58.546 INFO:teuthology.orchestra.run.smithi001.stdout: "peering_crush_bucket_target": 0, 2024-09-13T11:25:58.546 INFO:teuthology.orchestra.run.smithi001.stdout: "peering_crush_bucket_barrier": 0, 2024-09-13T11:25:58.547 INFO:teuthology.orchestra.run.smithi001.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-13T11:25:58.547 INFO:teuthology.orchestra.run.smithi001.stdout: "is_stretch_pool": false, 2024-09-13T11:25:58.547 INFO:teuthology.orchestra.run.smithi001.stdout: "object_hash": 2, 2024-09-13T11:25:58.547 INFO:teuthology.orchestra.run.smithi001.stdout: "pg_autoscale_mode": "off", 2024-09-13T11:25:58.547 INFO:teuthology.orchestra.run.smithi001.stdout: "pg_num": 46, 2024-09-13T11:25:58.547 INFO:teuthology.orchestra.run.smithi001.stdout: "pg_placement_num": 46, 2024-09-13T11:25:58.547 INFO:teuthology.orchestra.run.smithi001.stdout: "pg_placement_num_target": 46, 2024-09-13T11:25:58.547 INFO:teuthology.orchestra.run.smithi001.stdout: "pg_num_target": 46, 2024-09-13T11:25:58.547 INFO:teuthology.orchestra.run.smithi001.stdout: "pg_num_pending": 46, 2024-09-13T11:25:58.547 INFO:teuthology.orchestra.run.smithi001.stdout: "last_pg_merge_meta": { 2024-09-13T11:25:58.547 INFO:teuthology.orchestra.run.smithi001.stdout: "source_pgid": "0.0", 2024-09-13T11:25:58.547 INFO:teuthology.orchestra.run.smithi001.stdout: "ready_epoch": 0, 2024-09-13T11:25:58.547 INFO:teuthology.orchestra.run.smithi001.stdout: "last_epoch_started": 0, 2024-09-13T11:25:58.547 INFO:teuthology.orchestra.run.smithi001.stdout: "last_epoch_clean": 0, 2024-09-13T11:25:58.548 INFO:teuthology.orchestra.run.smithi001.stdout: "source_version": "0'0", 2024-09-13T11:25:58.548 INFO:teuthology.orchestra.run.smithi001.stdout: "target_version": "0'0" 2024-09-13T11:25:58.548 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:58.548 INFO:teuthology.orchestra.run.smithi001.stdout: "last_change": "119", 2024-09-13T11:25:58.548 INFO:teuthology.orchestra.run.smithi001.stdout: "last_force_op_resend": "0", 2024-09-13T11:25:58.548 INFO:teuthology.orchestra.run.smithi001.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-13T11:25:58.548 INFO:teuthology.orchestra.run.smithi001.stdout: "last_force_op_resend_preluminous": "117", 2024-09-13T11:25:58.548 INFO:teuthology.orchestra.run.smithi001.stdout: "auid": 0, 2024-09-13T11:25:58.548 INFO:teuthology.orchestra.run.smithi001.stdout: "snap_mode": "selfmanaged", 2024-09-13T11:25:58.548 INFO:teuthology.orchestra.run.smithi001.stdout: "snap_seq": 0, 2024-09-13T11:25:58.548 INFO:teuthology.orchestra.run.smithi001.stdout: "snap_epoch": 0, 2024-09-13T11:25:58.548 INFO:teuthology.orchestra.run.smithi001.stdout: "pool_snaps": [], 2024-09-13T11:25:58.548 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_snaps": "[]", 2024-09-13T11:25:58.549 INFO:teuthology.orchestra.run.smithi001.stdout: "quota_max_bytes": 0, 2024-09-13T11:25:58.549 INFO:teuthology.orchestra.run.smithi001.stdout: "quota_max_objects": 0, 2024-09-13T11:25:58.549 INFO:teuthology.orchestra.run.smithi001.stdout: "tiers": [], 2024-09-13T11:25:58.549 INFO:teuthology.orchestra.run.smithi001.stdout: "tier_of": -1, 2024-09-13T11:25:58.549 INFO:teuthology.orchestra.run.smithi001.stdout: "read_tier": -1, 2024-09-13T11:25:58.549 INFO:teuthology.orchestra.run.smithi001.stdout: "write_tier": -1, 2024-09-13T11:25:58.549 INFO:teuthology.orchestra.run.smithi001.stdout: "cache_mode": "none", 2024-09-13T11:25:58.549 INFO:teuthology.orchestra.run.smithi001.stdout: "target_max_bytes": 0, 2024-09-13T11:25:58.549 INFO:teuthology.orchestra.run.smithi001.stdout: "target_max_objects": 0, 2024-09-13T11:25:58.549 INFO:teuthology.orchestra.run.smithi001.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-13T11:25:58.549 INFO:teuthology.orchestra.run.smithi001.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-13T11:25:58.549 INFO:teuthology.orchestra.run.smithi001.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-13T11:25:58.549 INFO:teuthology.orchestra.run.smithi001.stdout: "cache_min_flush_age": 0, 2024-09-13T11:25:58.549 INFO:teuthology.orchestra.run.smithi001.stdout: "cache_min_evict_age": 0, 2024-09-13T11:25:58.550 INFO:teuthology.orchestra.run.smithi001.stdout: "erasure_code_profile": "", 2024-09-13T11:25:58.550 INFO:teuthology.orchestra.run.smithi001.stdout: "hit_set_params": { 2024-09-13T11:25:58.550 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "none" 2024-09-13T11:25:58.550 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:58.550 INFO:teuthology.orchestra.run.smithi001.stdout: "hit_set_period": 0, 2024-09-13T11:25:58.550 INFO:teuthology.orchestra.run.smithi001.stdout: "hit_set_count": 0, 2024-09-13T11:25:58.550 INFO:teuthology.orchestra.run.smithi001.stdout: "use_gmt_hitset": true, 2024-09-13T11:25:58.550 INFO:teuthology.orchestra.run.smithi001.stdout: "min_read_recency_for_promote": 0, 2024-09-13T11:25:58.550 INFO:teuthology.orchestra.run.smithi001.stdout: "min_write_recency_for_promote": 0, 2024-09-13T11:25:58.550 INFO:teuthology.orchestra.run.smithi001.stdout: "hit_set_grade_decay_rate": 0, 2024-09-13T11:25:58.550 INFO:teuthology.orchestra.run.smithi001.stdout: "hit_set_search_last_n": 0, 2024-09-13T11:25:58.550 INFO:teuthology.orchestra.run.smithi001.stdout: "grade_table": [], 2024-09-13T11:25:58.550 INFO:teuthology.orchestra.run.smithi001.stdout: "stripe_width": 0, 2024-09-13T11:25:58.551 INFO:teuthology.orchestra.run.smithi001.stdout: "expected_num_objects": 0, 2024-09-13T11:25:58.551 INFO:teuthology.orchestra.run.smithi001.stdout: "fast_read": false, 2024-09-13T11:25:58.551 INFO:teuthology.orchestra.run.smithi001.stdout: "options": {}, 2024-09-13T11:25:58.551 INFO:teuthology.orchestra.run.smithi001.stdout: "application_metadata": { 2024-09-13T11:25:58.551 INFO:teuthology.orchestra.run.smithi001.stdout: "rados": {} 2024-09-13T11:25:58.551 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:58.551 INFO:teuthology.orchestra.run.smithi001.stdout: "read_balance": { 2024-09-13T11:25:58.551 INFO:teuthology.orchestra.run.smithi001.stdout: "score_type": "Fair distribution", 2024-09-13T11:25:58.551 INFO:teuthology.orchestra.run.smithi001.stdout: "score_acting": 2.0899999141693115, 2024-09-13T11:25:58.551 INFO:teuthology.orchestra.run.smithi001.stdout: "score_stable": 2.0899999141693115, 2024-09-13T11:25:58.551 INFO:teuthology.orchestra.run.smithi001.stdout: "optimal_score": 0.93999999761581421, 2024-09-13T11:25:58.551 INFO:teuthology.orchestra.run.smithi001.stdout: "raw_score_acting": 1.9600000381469727, 2024-09-13T11:25:58.551 INFO:teuthology.orchestra.run.smithi001.stdout: "raw_score_stable": 1.9600000381469727, 2024-09-13T11:25:58.552 INFO:teuthology.orchestra.run.smithi001.stdout: "primary_affinity_weighted": 1, 2024-09-13T11:25:58.552 INFO:teuthology.orchestra.run.smithi001.stdout: "average_primary_affinity": 1, 2024-09-13T11:25:58.552 INFO:teuthology.orchestra.run.smithi001.stdout: "average_primary_affinity_weighted": 1 2024-09-13T11:25:58.552 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:58.552 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:58.552 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-09-13T11:25:58.552 INFO:teuthology.orchestra.run.smithi001.stdout: "osds": [ 2024-09-13T11:25:58.552 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:58.552 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": 0, 2024-09-13T11:25:58.552 INFO:teuthology.orchestra.run.smithi001.stdout: "uuid": "0fc12b3c-e0de-4546-b935-db77efe7c567", 2024-09-13T11:25:58.552 INFO:teuthology.orchestra.run.smithi001.stdout: "up": 1, 2024-09-13T11:25:58.552 INFO:teuthology.orchestra.run.smithi001.stdout: "in": 1, 2024-09-13T11:25:58.552 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 1, 2024-09-13T11:25:58.553 INFO:teuthology.orchestra.run.smithi001.stdout: "primary_affinity": 1, 2024-09-13T11:25:58.553 INFO:teuthology.orchestra.run.smithi001.stdout: "last_clean_begin": 0, 2024-09-13T11:25:58.553 INFO:teuthology.orchestra.run.smithi001.stdout: "last_clean_end": 0, 2024-09-13T11:25:58.553 INFO:teuthology.orchestra.run.smithi001.stdout: "up_from": 22, 2024-09-13T11:25:58.553 INFO:teuthology.orchestra.run.smithi001.stdout: "up_thru": 175, 2024-09-13T11:25:58.553 INFO:teuthology.orchestra.run.smithi001.stdout: "down_at": 0, 2024-09-13T11:25:58.553 INFO:teuthology.orchestra.run.smithi001.stdout: "lost_at": 0, 2024-09-13T11:25:58.553 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-09-13T11:25:58.553 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:25:58.553 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:58.553 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:25:58.553 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6800", 2024-09-13T11:25:58.553 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 1544776477 2024-09-13T11:25:58.553 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:58.554 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:58.554 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:58.554 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_addrs": { 2024-09-13T11:25:58.554 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:25:58.554 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:58.554 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:25:58.554 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6801", 2024-09-13T11:25:58.554 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 1544776477 2024-09-13T11:25:58.554 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:58.554 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:58.554 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:58.554 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_back_addrs": { 2024-09-13T11:25:58.554 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:25:58.555 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:58.555 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:25:58.555 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6803", 2024-09-13T11:25:58.555 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 1544776477 2024-09-13T11:25:58.555 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:58.555 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:58.555 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:58.555 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_front_addrs": { 2024-09-13T11:25:58.555 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:25:58.555 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:58.555 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:25:58.555 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6802", 2024-09-13T11:25:58.555 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 1544776477 2024-09-13T11:25:58.556 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:58.556 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:58.556 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:58.556 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6800/1544776477", 2024-09-13T11:25:58.556 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_addr": "172.21.15.1:6801/1544776477", 2024-09-13T11:25:58.556 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_back_addr": "172.21.15.1:6803/1544776477", 2024-09-13T11:25:58.556 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_front_addr": "172.21.15.1:6802/1544776477", 2024-09-13T11:25:58.556 INFO:teuthology.orchestra.run.smithi001.stdout: "state": [ 2024-09-13T11:25:58.556 INFO:teuthology.orchestra.run.smithi001.stdout: "exists", 2024-09-13T11:25:58.556 INFO:teuthology.orchestra.run.smithi001.stdout: "up" 2024-09-13T11:25:58.556 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:58.556 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:58.556 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:58.557 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": 1, 2024-09-13T11:25:58.557 INFO:teuthology.orchestra.run.smithi001.stdout: "uuid": "b77fe331-cc92-4adb-b146-af5c380f6d4f", 2024-09-13T11:25:58.557 INFO:teuthology.orchestra.run.smithi001.stdout: "up": 1, 2024-09-13T11:25:58.557 INFO:teuthology.orchestra.run.smithi001.stdout: "in": 1, 2024-09-13T11:25:58.557 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 1, 2024-09-13T11:25:58.557 INFO:teuthology.orchestra.run.smithi001.stdout: "primary_affinity": 1, 2024-09-13T11:25:58.557 INFO:teuthology.orchestra.run.smithi001.stdout: "last_clean_begin": 23, 2024-09-13T11:25:58.557 INFO:teuthology.orchestra.run.smithi001.stdout: "last_clean_end": 29, 2024-09-13T11:25:58.557 INFO:teuthology.orchestra.run.smithi001.stdout: "up_from": 40, 2024-09-13T11:25:58.557 INFO:teuthology.orchestra.run.smithi001.stdout: "up_thru": 171, 2024-09-13T11:25:58.557 INFO:teuthology.orchestra.run.smithi001.stdout: "down_at": 30, 2024-09-13T11:25:58.557 INFO:teuthology.orchestra.run.smithi001.stdout: "lost_at": 0, 2024-09-13T11:25:58.557 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-09-13T11:25:58.558 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:25:58.558 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:58.558 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:25:58.558 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.92:6800", 2024-09-13T11:25:58.558 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 3591430847 2024-09-13T11:25:58.558 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:58.558 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:58.558 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:58.558 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_addrs": { 2024-09-13T11:25:58.558 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:25:58.559 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:58.559 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:25:58.559 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.92:6801", 2024-09-13T11:25:58.559 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 3591430847 2024-09-13T11:25:58.559 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:58.559 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:58.559 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:58.559 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_back_addrs": { 2024-09-13T11:25:58.559 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:25:58.559 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:58.559 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:25:58.559 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.92:6803", 2024-09-13T11:25:58.559 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 3591430847 2024-09-13T11:25:58.559 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:58.560 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:58.560 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:58.560 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_front_addrs": { 2024-09-13T11:25:58.560 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:25:58.560 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:58.560 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:25:58.560 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.92:6802", 2024-09-13T11:25:58.560 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 3591430847 2024-09-13T11:25:58.560 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:58.560 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:58.560 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:58.560 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.92:6800/3591430847", 2024-09-13T11:25:58.560 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_addr": "172.21.15.92:6801/3591430847", 2024-09-13T11:25:58.561 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_back_addr": "172.21.15.92:6803/3591430847", 2024-09-13T11:25:58.561 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_front_addr": "172.21.15.92:6802/3591430847", 2024-09-13T11:25:58.561 INFO:teuthology.orchestra.run.smithi001.stdout: "state": [ 2024-09-13T11:25:58.561 INFO:teuthology.orchestra.run.smithi001.stdout: "exists", 2024-09-13T11:25:58.561 INFO:teuthology.orchestra.run.smithi001.stdout: "up" 2024-09-13T11:25:58.561 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:58.561 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:58.561 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:58.561 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": 2, 2024-09-13T11:25:58.561 INFO:teuthology.orchestra.run.smithi001.stdout: "uuid": "c9d031c8-e885-4229-affa-08e1729d00e8", 2024-09-13T11:25:58.561 INFO:teuthology.orchestra.run.smithi001.stdout: "up": 1, 2024-09-13T11:25:58.561 INFO:teuthology.orchestra.run.smithi001.stdout: "in": 1, 2024-09-13T11:25:58.562 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 1, 2024-09-13T11:25:58.562 INFO:teuthology.orchestra.run.smithi001.stdout: "primary_affinity": 1, 2024-09-13T11:25:58.562 INFO:teuthology.orchestra.run.smithi001.stdout: "last_clean_begin": 0, 2024-09-13T11:25:58.562 INFO:teuthology.orchestra.run.smithi001.stdout: "last_clean_end": 0, 2024-09-13T11:25:58.562 INFO:teuthology.orchestra.run.smithi001.stdout: "up_from": 22, 2024-09-13T11:25:58.562 INFO:teuthology.orchestra.run.smithi001.stdout: "up_thru": 152, 2024-09-13T11:25:58.562 INFO:teuthology.orchestra.run.smithi001.stdout: "down_at": 0, 2024-09-13T11:25:58.562 INFO:teuthology.orchestra.run.smithi001.stdout: "lost_at": 0, 2024-09-13T11:25:58.562 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-09-13T11:25:58.562 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:25:58.562 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:58.562 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:25:58.562 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.162:6812", 2024-09-13T11:25:58.563 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 2712148101 2024-09-13T11:25:58.563 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:58.563 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:58.563 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:58.563 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_addrs": { 2024-09-13T11:25:58.563 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:25:58.563 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:58.563 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:25:58.563 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.162:6813", 2024-09-13T11:25:58.563 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 2712148101 2024-09-13T11:25:58.563 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:58.563 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:58.563 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:58.563 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_back_addrs": { 2024-09-13T11:25:58.564 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:25:58.564 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:58.564 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:25:58.564 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.162:6815", 2024-09-13T11:25:58.564 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 2712148101 2024-09-13T11:25:58.564 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:58.564 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:58.564 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:58.564 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_front_addrs": { 2024-09-13T11:25:58.564 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:25:58.564 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:58.564 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:25:58.564 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.162:6814", 2024-09-13T11:25:58.565 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 2712148101 2024-09-13T11:25:58.565 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:58.565 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:58.565 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:58.565 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.162:6812/2712148101", 2024-09-13T11:25:58.565 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_addr": "172.21.15.162:6813/2712148101", 2024-09-13T11:25:58.565 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_back_addr": "172.21.15.162:6815/2712148101", 2024-09-13T11:25:58.565 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_front_addr": "172.21.15.162:6814/2712148101", 2024-09-13T11:25:58.565 INFO:teuthology.orchestra.run.smithi001.stdout: "state": [ 2024-09-13T11:25:58.565 INFO:teuthology.orchestra.run.smithi001.stdout: "exists", 2024-09-13T11:25:58.565 INFO:teuthology.orchestra.run.smithi001.stdout: "up" 2024-09-13T11:25:58.565 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:58.565 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:58.565 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:58.566 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": 3, 2024-09-13T11:25:58.566 INFO:teuthology.orchestra.run.smithi001.stdout: "uuid": "bf50d71a-6c3a-4fbc-a407-cb81bad14a7b", 2024-09-13T11:25:58.566 INFO:teuthology.orchestra.run.smithi001.stdout: "up": 1, 2024-09-13T11:25:58.566 INFO:teuthology.orchestra.run.smithi001.stdout: "in": 1, 2024-09-13T11:25:58.566 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 1, 2024-09-13T11:25:58.566 INFO:teuthology.orchestra.run.smithi001.stdout: "primary_affinity": 1, 2024-09-13T11:25:58.566 INFO:teuthology.orchestra.run.smithi001.stdout: "last_clean_begin": 0, 2024-09-13T11:25:58.566 INFO:teuthology.orchestra.run.smithi001.stdout: "last_clean_end": 0, 2024-09-13T11:25:58.566 INFO:teuthology.orchestra.run.smithi001.stdout: "up_from": 21, 2024-09-13T11:25:58.566 INFO:teuthology.orchestra.run.smithi001.stdout: "up_thru": 174, 2024-09-13T11:25:58.566 INFO:teuthology.orchestra.run.smithi001.stdout: "down_at": 0, 2024-09-13T11:25:58.566 INFO:teuthology.orchestra.run.smithi001.stdout: "lost_at": 0, 2024-09-13T11:25:58.566 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-09-13T11:25:58.566 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:25:58.566 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:58.567 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:25:58.567 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.177:6801", 2024-09-13T11:25:58.567 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 3367751690 2024-09-13T11:25:58.567 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:58.567 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:58.567 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:58.567 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_addrs": { 2024-09-13T11:25:58.567 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:25:58.567 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:58.567 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:25:58.567 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.177:6802", 2024-09-13T11:25:58.567 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 3367751690 2024-09-13T11:25:58.567 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:58.568 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:58.568 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:58.568 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_back_addrs": { 2024-09-13T11:25:58.568 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:25:58.568 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:58.568 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:25:58.568 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.177:6804", 2024-09-13T11:25:58.568 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 3367751690 2024-09-13T11:25:58.568 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:58.568 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:58.568 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:58.568 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_front_addrs": { 2024-09-13T11:25:58.568 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:25:58.569 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:58.569 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:25:58.569 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.177:6803", 2024-09-13T11:25:58.569 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 3367751690 2024-09-13T11:25:58.569 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:58.569 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:58.569 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:58.569 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.177:6801/3367751690", 2024-09-13T11:25:58.569 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_addr": "172.21.15.177:6802/3367751690", 2024-09-13T11:25:58.569 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_back_addr": "172.21.15.177:6804/3367751690", 2024-09-13T11:25:58.569 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_front_addr": "172.21.15.177:6803/3367751690", 2024-09-13T11:25:58.569 INFO:teuthology.orchestra.run.smithi001.stdout: "state": [ 2024-09-13T11:25:58.569 INFO:teuthology.orchestra.run.smithi001.stdout: "exists", 2024-09-13T11:25:58.570 INFO:teuthology.orchestra.run.smithi001.stdout: "up" 2024-09-13T11:25:58.570 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:58.570 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:58.570 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:58.570 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": 4, 2024-09-13T11:25:58.570 INFO:teuthology.orchestra.run.smithi001.stdout: "uuid": "d1fe3f63-df3f-4a73-a1e2-8d7c8a3f1fa0", 2024-09-13T11:25:58.570 INFO:teuthology.orchestra.run.smithi001.stdout: "up": 1, 2024-09-13T11:25:58.570 INFO:teuthology.orchestra.run.smithi001.stdout: "in": 1, 2024-09-13T11:25:58.570 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 1, 2024-09-13T11:25:58.570 INFO:teuthology.orchestra.run.smithi001.stdout: "primary_affinity": 1, 2024-09-13T11:25:58.570 INFO:teuthology.orchestra.run.smithi001.stdout: "last_clean_begin": 0, 2024-09-13T11:25:58.570 INFO:teuthology.orchestra.run.smithi001.stdout: "last_clean_end": 0, 2024-09-13T11:25:58.570 INFO:teuthology.orchestra.run.smithi001.stdout: "up_from": 22, 2024-09-13T11:25:58.570 INFO:teuthology.orchestra.run.smithi001.stdout: "up_thru": 105, 2024-09-13T11:25:58.571 INFO:teuthology.orchestra.run.smithi001.stdout: "down_at": 0, 2024-09-13T11:25:58.571 INFO:teuthology.orchestra.run.smithi001.stdout: "lost_at": 0, 2024-09-13T11:25:58.571 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-09-13T11:25:58.571 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:25:58.571 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:58.571 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:25:58.571 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6808", 2024-09-13T11:25:58.571 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 3648540926 2024-09-13T11:25:58.571 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:58.571 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:58.571 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:58.571 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_addrs": { 2024-09-13T11:25:58.571 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:25:58.572 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:58.572 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:25:58.572 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6809", 2024-09-13T11:25:58.572 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 3648540926 2024-09-13T11:25:58.572 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:58.572 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:58.572 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:58.572 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_back_addrs": { 2024-09-13T11:25:58.572 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:25:58.572 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:58.572 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:25:58.572 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6811", 2024-09-13T11:25:58.572 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 3648540926 2024-09-13T11:25:58.573 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:58.573 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:58.573 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:58.573 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_front_addrs": { 2024-09-13T11:25:58.573 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:25:58.573 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:58.573 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:25:58.573 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6810", 2024-09-13T11:25:58.573 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 3648540926 2024-09-13T11:25:58.573 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:58.573 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:58.573 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:58.573 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6808/3648540926", 2024-09-13T11:25:58.573 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_addr": "172.21.15.1:6809/3648540926", 2024-09-13T11:25:58.574 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_back_addr": "172.21.15.1:6811/3648540926", 2024-09-13T11:25:58.574 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_front_addr": "172.21.15.1:6810/3648540926", 2024-09-13T11:25:58.574 INFO:teuthology.orchestra.run.smithi001.stdout: "state": [ 2024-09-13T11:25:58.574 INFO:teuthology.orchestra.run.smithi001.stdout: "exists", 2024-09-13T11:25:58.574 INFO:teuthology.orchestra.run.smithi001.stdout: "up" 2024-09-13T11:25:58.574 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:58.574 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:58.574 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:58.574 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": 5, 2024-09-13T11:25:58.574 INFO:teuthology.orchestra.run.smithi001.stdout: "uuid": "e2ead447-2f2c-40c9-82e7-53b8dc41e2af", 2024-09-13T11:25:58.574 INFO:teuthology.orchestra.run.smithi001.stdout: "up": 1, 2024-09-13T11:25:58.574 INFO:teuthology.orchestra.run.smithi001.stdout: "in": 1, 2024-09-13T11:25:58.574 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 1, 2024-09-13T11:25:58.575 INFO:teuthology.orchestra.run.smithi001.stdout: "primary_affinity": 1, 2024-09-13T11:25:58.575 INFO:teuthology.orchestra.run.smithi001.stdout: "last_clean_begin": 0, 2024-09-13T11:25:58.575 INFO:teuthology.orchestra.run.smithi001.stdout: "last_clean_end": 0, 2024-09-13T11:25:58.575 INFO:teuthology.orchestra.run.smithi001.stdout: "up_from": 21, 2024-09-13T11:25:58.575 INFO:teuthology.orchestra.run.smithi001.stdout: "up_thru": 169, 2024-09-13T11:25:58.575 INFO:teuthology.orchestra.run.smithi001.stdout: "down_at": 0, 2024-09-13T11:25:58.575 INFO:teuthology.orchestra.run.smithi001.stdout: "lost_at": 0, 2024-09-13T11:25:58.575 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-09-13T11:25:58.575 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:25:58.575 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:58.575 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:25:58.575 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.92:6804", 2024-09-13T11:25:58.575 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 3399529091 2024-09-13T11:25:58.576 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:58.576 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:58.576 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:58.576 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_addrs": { 2024-09-13T11:25:58.576 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:25:58.576 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:58.576 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:25:58.576 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.92:6805", 2024-09-13T11:25:58.576 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 3399529091 2024-09-13T11:25:58.576 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:58.576 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:58.576 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:58.576 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_back_addrs": { 2024-09-13T11:25:58.577 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:25:58.577 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:58.577 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:25:58.577 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.92:6807", 2024-09-13T11:25:58.577 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 3399529091 2024-09-13T11:25:58.577 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:58.577 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:58.577 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:58.577 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_front_addrs": { 2024-09-13T11:25:58.577 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:25:58.577 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:58.577 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:25:58.577 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.92:6806", 2024-09-13T11:25:58.578 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 3399529091 2024-09-13T11:25:58.578 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:58.578 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:58.578 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:58.578 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.92:6804/3399529091", 2024-09-13T11:25:58.578 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_addr": "172.21.15.92:6805/3399529091", 2024-09-13T11:25:58.578 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_back_addr": "172.21.15.92:6807/3399529091", 2024-09-13T11:25:58.578 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_front_addr": "172.21.15.92:6806/3399529091", 2024-09-13T11:25:58.578 INFO:teuthology.orchestra.run.smithi001.stdout: "state": [ 2024-09-13T11:25:58.578 INFO:teuthology.orchestra.run.smithi001.stdout: "exists", 2024-09-13T11:25:58.578 INFO:teuthology.orchestra.run.smithi001.stdout: "up" 2024-09-13T11:25:58.578 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:58.578 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:58.578 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:58.579 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": 6, 2024-09-13T11:25:58.579 INFO:teuthology.orchestra.run.smithi001.stdout: "uuid": "dbc20439-144e-4ec7-89de-674eb783f263", 2024-09-13T11:25:58.579 INFO:teuthology.orchestra.run.smithi001.stdout: "up": 1, 2024-09-13T11:25:58.579 INFO:teuthology.orchestra.run.smithi001.stdout: "in": 1, 2024-09-13T11:25:58.579 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 1, 2024-09-13T11:25:58.579 INFO:teuthology.orchestra.run.smithi001.stdout: "primary_affinity": 1, 2024-09-13T11:25:58.579 INFO:teuthology.orchestra.run.smithi001.stdout: "last_clean_begin": 0, 2024-09-13T11:25:58.579 INFO:teuthology.orchestra.run.smithi001.stdout: "last_clean_end": 0, 2024-09-13T11:25:58.579 INFO:teuthology.orchestra.run.smithi001.stdout: "up_from": 22, 2024-09-13T11:25:58.579 INFO:teuthology.orchestra.run.smithi001.stdout: "up_thru": 117, 2024-09-13T11:25:58.579 INFO:teuthology.orchestra.run.smithi001.stdout: "down_at": 0, 2024-09-13T11:25:58.579 INFO:teuthology.orchestra.run.smithi001.stdout: "lost_at": 0, 2024-09-13T11:25:58.579 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-09-13T11:25:58.580 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:25:58.580 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:58.580 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:25:58.580 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.162:6808", 2024-09-13T11:25:58.580 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 2287343806 2024-09-13T11:25:58.580 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:58.580 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:58.580 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:58.580 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_addrs": { 2024-09-13T11:25:58.580 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:25:58.580 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:58.580 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:25:58.580 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.162:6809", 2024-09-13T11:25:58.581 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 2287343806 2024-09-13T11:25:58.581 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:58.581 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:58.581 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:58.581 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_back_addrs": { 2024-09-13T11:25:58.581 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:25:58.581 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:58.581 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:25:58.581 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.162:6811", 2024-09-13T11:25:58.581 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 2287343806 2024-09-13T11:25:58.581 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:58.581 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:58.581 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:58.582 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_front_addrs": { 2024-09-13T11:25:58.582 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:25:58.582 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:58.582 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:25:58.582 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.162:6810", 2024-09-13T11:25:58.582 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 2287343806 2024-09-13T11:25:58.582 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:58.582 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:58.582 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:58.582 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.162:6808/2287343806", 2024-09-13T11:25:58.582 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_addr": "172.21.15.162:6809/2287343806", 2024-09-13T11:25:58.582 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_back_addr": "172.21.15.162:6811/2287343806", 2024-09-13T11:25:58.582 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_front_addr": "172.21.15.162:6810/2287343806", 2024-09-13T11:25:58.582 INFO:teuthology.orchestra.run.smithi001.stdout: "state": [ 2024-09-13T11:25:58.583 INFO:teuthology.orchestra.run.smithi001.stdout: "exists", 2024-09-13T11:25:58.583 INFO:teuthology.orchestra.run.smithi001.stdout: "up" 2024-09-13T11:25:58.583 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:58.583 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:58.583 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:58.583 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": 7, 2024-09-13T11:25:58.583 INFO:teuthology.orchestra.run.smithi001.stdout: "uuid": "60aa5206-cace-40f8-8d1a-10ec28d97096", 2024-09-13T11:25:58.583 INFO:teuthology.orchestra.run.smithi001.stdout: "up": 1, 2024-09-13T11:25:58.583 INFO:teuthology.orchestra.run.smithi001.stdout: "in": 1, 2024-09-13T11:25:58.583 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 1, 2024-09-13T11:25:58.583 INFO:teuthology.orchestra.run.smithi001.stdout: "primary_affinity": 1, 2024-09-13T11:25:58.583 INFO:teuthology.orchestra.run.smithi001.stdout: "last_clean_begin": 0, 2024-09-13T11:25:58.583 INFO:teuthology.orchestra.run.smithi001.stdout: "last_clean_end": 0, 2024-09-13T11:25:58.584 INFO:teuthology.orchestra.run.smithi001.stdout: "up_from": 22, 2024-09-13T11:25:58.584 INFO:teuthology.orchestra.run.smithi001.stdout: "up_thru": 122, 2024-09-13T11:25:58.584 INFO:teuthology.orchestra.run.smithi001.stdout: "down_at": 0, 2024-09-13T11:25:58.584 INFO:teuthology.orchestra.run.smithi001.stdout: "lost_at": 0, 2024-09-13T11:25:58.584 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-09-13T11:25:58.584 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:25:58.584 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:58.584 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:25:58.584 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.177:6813", 2024-09-13T11:25:58.584 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 2110809906 2024-09-13T11:25:58.584 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:58.584 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:58.584 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:58.585 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_addrs": { 2024-09-13T11:25:58.585 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:25:58.585 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:58.585 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:25:58.585 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.177:6814", 2024-09-13T11:25:58.585 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 2110809906 2024-09-13T11:25:58.585 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:58.585 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:58.585 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:58.585 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_back_addrs": { 2024-09-13T11:25:58.585 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:25:58.585 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:58.585 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:25:58.585 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.177:6816", 2024-09-13T11:25:58.586 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 2110809906 2024-09-13T11:25:58.586 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:58.586 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:58.586 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:58.586 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_front_addrs": { 2024-09-13T11:25:58.586 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:25:58.586 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:58.586 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:25:58.586 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.177:6815", 2024-09-13T11:25:58.586 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 2110809906 2024-09-13T11:25:58.586 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:58.586 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:58.586 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:58.587 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.177:6813/2110809906", 2024-09-13T11:25:58.587 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_addr": "172.21.15.177:6814/2110809906", 2024-09-13T11:25:58.587 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_back_addr": "172.21.15.177:6816/2110809906", 2024-09-13T11:25:58.587 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_front_addr": "172.21.15.177:6815/2110809906", 2024-09-13T11:25:58.587 INFO:teuthology.orchestra.run.smithi001.stdout: "state": [ 2024-09-13T11:25:58.587 INFO:teuthology.orchestra.run.smithi001.stdout: "exists", 2024-09-13T11:25:58.587 INFO:teuthology.orchestra.run.smithi001.stdout: "up" 2024-09-13T11:25:58.587 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:58.587 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:58.587 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:58.587 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": 8, 2024-09-13T11:25:58.587 INFO:teuthology.orchestra.run.smithi001.stdout: "uuid": "f3d99325-48ff-403c-8662-1d7745f7c97d", 2024-09-13T11:25:58.587 INFO:teuthology.orchestra.run.smithi001.stdout: "up": 1, 2024-09-13T11:25:58.588 INFO:teuthology.orchestra.run.smithi001.stdout: "in": 1, 2024-09-13T11:25:58.588 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 1, 2024-09-13T11:25:58.588 INFO:teuthology.orchestra.run.smithi001.stdout: "primary_affinity": 1, 2024-09-13T11:25:58.588 INFO:teuthology.orchestra.run.smithi001.stdout: "last_clean_begin": 0, 2024-09-13T11:25:58.588 INFO:teuthology.orchestra.run.smithi001.stdout: "last_clean_end": 0, 2024-09-13T11:25:58.588 INFO:teuthology.orchestra.run.smithi001.stdout: "up_from": 22, 2024-09-13T11:25:58.588 INFO:teuthology.orchestra.run.smithi001.stdout: "up_thru": 172, 2024-09-13T11:25:58.588 INFO:teuthology.orchestra.run.smithi001.stdout: "down_at": 0, 2024-09-13T11:25:58.588 INFO:teuthology.orchestra.run.smithi001.stdout: "lost_at": 0, 2024-09-13T11:25:58.588 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-09-13T11:25:58.588 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:25:58.588 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:58.588 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:25:58.589 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6804", 2024-09-13T11:25:58.589 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 2766567910 2024-09-13T11:25:58.589 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:58.589 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:58.589 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:58.589 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_addrs": { 2024-09-13T11:25:58.589 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:25:58.589 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:58.589 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:25:58.589 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6805", 2024-09-13T11:25:58.589 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 2766567910 2024-09-13T11:25:58.589 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:58.589 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:58.590 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:58.590 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_back_addrs": { 2024-09-13T11:25:58.590 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:25:58.590 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:58.590 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:25:58.590 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6807", 2024-09-13T11:25:58.590 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 2766567910 2024-09-13T11:25:58.590 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:58.590 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:58.590 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:58.590 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_front_addrs": { 2024-09-13T11:25:58.590 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:25:58.590 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:58.590 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:25:58.591 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6806", 2024-09-13T11:25:58.591 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 2766567910 2024-09-13T11:25:58.591 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:58.591 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:58.591 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:58.591 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6804/2766567910", 2024-09-13T11:25:58.591 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_addr": "172.21.15.1:6805/2766567910", 2024-09-13T11:25:58.591 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_back_addr": "172.21.15.1:6807/2766567910", 2024-09-13T11:25:58.591 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_front_addr": "172.21.15.1:6806/2766567910", 2024-09-13T11:25:58.591 INFO:teuthology.orchestra.run.smithi001.stdout: "state": [ 2024-09-13T11:25:58.591 INFO:teuthology.orchestra.run.smithi001.stdout: "exists", 2024-09-13T11:25:58.591 INFO:teuthology.orchestra.run.smithi001.stdout: "up" 2024-09-13T11:25:58.591 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:58.591 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:58.592 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:58.592 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": 9, 2024-09-13T11:25:58.592 INFO:teuthology.orchestra.run.smithi001.stdout: "uuid": "88511e24-3675-4551-968c-1a52e7e28f96", 2024-09-13T11:25:58.592 INFO:teuthology.orchestra.run.smithi001.stdout: "up": 1, 2024-09-13T11:25:58.592 INFO:teuthology.orchestra.run.smithi001.stdout: "in": 1, 2024-09-13T11:25:58.592 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 1, 2024-09-13T11:25:58.592 INFO:teuthology.orchestra.run.smithi001.stdout: "primary_affinity": 1, 2024-09-13T11:25:58.592 INFO:teuthology.orchestra.run.smithi001.stdout: "last_clean_begin": 0, 2024-09-13T11:25:58.592 INFO:teuthology.orchestra.run.smithi001.stdout: "last_clean_end": 0, 2024-09-13T11:25:58.592 INFO:teuthology.orchestra.run.smithi001.stdout: "up_from": 23, 2024-09-13T11:25:58.592 INFO:teuthology.orchestra.run.smithi001.stdout: "up_thru": 133, 2024-09-13T11:25:58.592 INFO:teuthology.orchestra.run.smithi001.stdout: "down_at": 0, 2024-09-13T11:25:58.593 INFO:teuthology.orchestra.run.smithi001.stdout: "lost_at": 0, 2024-09-13T11:25:58.593 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-09-13T11:25:58.593 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:25:58.593 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:58.593 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:25:58.593 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.92:6812", 2024-09-13T11:25:58.593 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 2488187511 2024-09-13T11:25:58.593 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:58.593 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:58.593 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:58.593 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_addrs": { 2024-09-13T11:25:58.593 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:25:58.593 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:58.593 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:25:58.594 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.92:6813", 2024-09-13T11:25:58.594 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 2488187511 2024-09-13T11:25:58.594 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:58.594 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:58.594 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:58.594 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_back_addrs": { 2024-09-13T11:25:58.594 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:25:58.594 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:58.594 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:25:58.594 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.92:6815", 2024-09-13T11:25:58.594 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 2488187511 2024-09-13T11:25:58.594 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:58.594 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:58.594 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:58.595 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_front_addrs": { 2024-09-13T11:25:58.595 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:25:58.595 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:58.595 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:25:58.595 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.92:6814", 2024-09-13T11:25:58.595 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 2488187511 2024-09-13T11:25:58.595 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:58.595 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:58.595 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:58.595 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.92:6812/2488187511", 2024-09-13T11:25:58.595 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_addr": "172.21.15.92:6813/2488187511", 2024-09-13T11:25:58.595 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_back_addr": "172.21.15.92:6815/2488187511", 2024-09-13T11:25:58.595 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_front_addr": "172.21.15.92:6814/2488187511", 2024-09-13T11:25:58.596 INFO:teuthology.orchestra.run.smithi001.stdout: "state": [ 2024-09-13T11:25:58.596 INFO:teuthology.orchestra.run.smithi001.stdout: "exists", 2024-09-13T11:25:58.596 INFO:teuthology.orchestra.run.smithi001.stdout: "up" 2024-09-13T11:25:58.596 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:58.596 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:58.596 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:58.596 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": 10, 2024-09-13T11:25:58.596 INFO:teuthology.orchestra.run.smithi001.stdout: "uuid": "acf4c64d-99af-42be-8406-413c86f54670", 2024-09-13T11:25:58.596 INFO:teuthology.orchestra.run.smithi001.stdout: "up": 1, 2024-09-13T11:25:58.596 INFO:teuthology.orchestra.run.smithi001.stdout: "in": 1, 2024-09-13T11:25:58.596 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 1, 2024-09-13T11:25:58.596 INFO:teuthology.orchestra.run.smithi001.stdout: "primary_affinity": 1, 2024-09-13T11:25:58.596 INFO:teuthology.orchestra.run.smithi001.stdout: "last_clean_begin": 0, 2024-09-13T11:25:58.597 INFO:teuthology.orchestra.run.smithi001.stdout: "last_clean_end": 0, 2024-09-13T11:25:58.597 INFO:teuthology.orchestra.run.smithi001.stdout: "up_from": 22, 2024-09-13T11:25:58.597 INFO:teuthology.orchestra.run.smithi001.stdout: "up_thru": 144, 2024-09-13T11:25:58.597 INFO:teuthology.orchestra.run.smithi001.stdout: "down_at": 0, 2024-09-13T11:25:58.597 INFO:teuthology.orchestra.run.smithi001.stdout: "lost_at": 0, 2024-09-13T11:25:58.597 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-09-13T11:25:58.597 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:25:58.597 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:58.597 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:25:58.597 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.162:6804", 2024-09-13T11:25:58.597 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 1450266459 2024-09-13T11:25:58.598 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:58.598 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:58.598 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:58.598 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_addrs": { 2024-09-13T11:25:58.598 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:25:58.598 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:58.598 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:25:58.598 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.162:6805", 2024-09-13T11:25:58.598 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 1450266459 2024-09-13T11:25:58.598 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:58.598 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:58.598 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:58.598 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_back_addrs": { 2024-09-13T11:25:58.599 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:25:58.599 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:58.599 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:25:58.599 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.162:6807", 2024-09-13T11:25:58.599 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 1450266459 2024-09-13T11:25:58.599 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:58.599 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:58.599 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:58.599 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_front_addrs": { 2024-09-13T11:25:58.599 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:25:58.599 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:58.599 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:25:58.599 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.162:6806", 2024-09-13T11:25:58.599 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 1450266459 2024-09-13T11:25:58.600 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:58.600 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:58.600 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:58.600 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.162:6804/1450266459", 2024-09-13T11:25:58.600 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_addr": "172.21.15.162:6805/1450266459", 2024-09-13T11:25:58.600 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_back_addr": "172.21.15.162:6807/1450266459", 2024-09-13T11:25:58.600 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_front_addr": "172.21.15.162:6806/1450266459", 2024-09-13T11:25:58.600 INFO:teuthology.orchestra.run.smithi001.stdout: "state": [ 2024-09-13T11:25:58.600 INFO:teuthology.orchestra.run.smithi001.stdout: "exists", 2024-09-13T11:25:58.600 INFO:teuthology.orchestra.run.smithi001.stdout: "up" 2024-09-13T11:25:58.600 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:58.600 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:58.600 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:58.601 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": 11, 2024-09-13T11:25:58.601 INFO:teuthology.orchestra.run.smithi001.stdout: "uuid": "a4bfdfbe-b586-4080-8e09-6f26428e59b5", 2024-09-13T11:25:58.601 INFO:teuthology.orchestra.run.smithi001.stdout: "up": 1, 2024-09-13T11:25:58.601 INFO:teuthology.orchestra.run.smithi001.stdout: "in": 1, 2024-09-13T11:25:58.601 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 1, 2024-09-13T11:25:58.601 INFO:teuthology.orchestra.run.smithi001.stdout: "primary_affinity": 1, 2024-09-13T11:25:58.601 INFO:teuthology.orchestra.run.smithi001.stdout: "last_clean_begin": 0, 2024-09-13T11:25:58.601 INFO:teuthology.orchestra.run.smithi001.stdout: "last_clean_end": 0, 2024-09-13T11:25:58.601 INFO:teuthology.orchestra.run.smithi001.stdout: "up_from": 22, 2024-09-13T11:25:58.601 INFO:teuthology.orchestra.run.smithi001.stdout: "up_thru": 143, 2024-09-13T11:25:58.601 INFO:teuthology.orchestra.run.smithi001.stdout: "down_at": 0, 2024-09-13T11:25:58.601 INFO:teuthology.orchestra.run.smithi001.stdout: "lost_at": 0, 2024-09-13T11:25:58.601 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-09-13T11:25:58.601 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:25:58.602 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:58.602 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:25:58.602 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.177:6809", 2024-09-13T11:25:58.602 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 3239521954 2024-09-13T11:25:58.602 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:58.602 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:58.602 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:58.602 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_addrs": { 2024-09-13T11:25:58.602 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:25:58.602 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:58.602 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:25:58.602 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.177:6810", 2024-09-13T11:25:58.602 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 3239521954 2024-09-13T11:25:58.603 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:58.603 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:58.603 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:58.603 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_back_addrs": { 2024-09-13T11:25:58.603 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:25:58.603 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:58.603 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:25:58.603 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.177:6812", 2024-09-13T11:25:58.603 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 3239521954 2024-09-13T11:25:58.603 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:58.603 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:58.603 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:58.603 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_front_addrs": { 2024-09-13T11:25:58.604 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:25:58.604 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:58.604 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:25:58.604 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.177:6811", 2024-09-13T11:25:58.604 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 3239521954 2024-09-13T11:25:58.604 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:58.604 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:58.604 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:58.604 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.177:6809/3239521954", 2024-09-13T11:25:58.604 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_addr": "172.21.15.177:6810/3239521954", 2024-09-13T11:25:58.604 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_back_addr": "172.21.15.177:6812/3239521954", 2024-09-13T11:25:58.604 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_front_addr": "172.21.15.177:6811/3239521954", 2024-09-13T11:25:58.604 INFO:teuthology.orchestra.run.smithi001.stdout: "state": [ 2024-09-13T11:25:58.604 INFO:teuthology.orchestra.run.smithi001.stdout: "exists", 2024-09-13T11:25:58.605 INFO:teuthology.orchestra.run.smithi001.stdout: "up" 2024-09-13T11:25:58.605 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:58.605 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:58.605 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:58.605 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": 12, 2024-09-13T11:25:58.605 INFO:teuthology.orchestra.run.smithi001.stdout: "uuid": "dc571fb6-d8b5-465b-bb25-afa7d0c47b5a", 2024-09-13T11:25:58.605 INFO:teuthology.orchestra.run.smithi001.stdout: "up": 1, 2024-09-13T11:25:58.605 INFO:teuthology.orchestra.run.smithi001.stdout: "in": 1, 2024-09-13T11:25:58.605 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 1, 2024-09-13T11:25:58.605 INFO:teuthology.orchestra.run.smithi001.stdout: "primary_affinity": 1, 2024-09-13T11:25:58.605 INFO:teuthology.orchestra.run.smithi001.stdout: "last_clean_begin": 0, 2024-09-13T11:25:58.605 INFO:teuthology.orchestra.run.smithi001.stdout: "last_clean_end": 0, 2024-09-13T11:25:58.605 INFO:teuthology.orchestra.run.smithi001.stdout: "up_from": 22, 2024-09-13T11:25:58.605 INFO:teuthology.orchestra.run.smithi001.stdout: "up_thru": 141, 2024-09-13T11:25:58.606 INFO:teuthology.orchestra.run.smithi001.stdout: "down_at": 0, 2024-09-13T11:25:58.606 INFO:teuthology.orchestra.run.smithi001.stdout: "lost_at": 0, 2024-09-13T11:25:58.606 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-09-13T11:25:58.606 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:25:58.606 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:58.606 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:25:58.606 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6812", 2024-09-13T11:25:58.606 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 4267062742 2024-09-13T11:25:58.606 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:58.606 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:58.606 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:58.606 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_addrs": { 2024-09-13T11:25:58.606 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:25:58.607 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:58.607 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:25:58.607 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6813", 2024-09-13T11:25:58.607 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 4267062742 2024-09-13T11:25:58.607 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:58.607 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:58.607 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:58.607 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_back_addrs": { 2024-09-13T11:25:58.607 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:25:58.607 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:58.607 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:25:58.607 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6815", 2024-09-13T11:25:58.607 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 4267062742 2024-09-13T11:25:58.608 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:58.608 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:58.608 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:58.608 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_front_addrs": { 2024-09-13T11:25:58.608 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:25:58.608 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:58.608 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:25:58.608 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6814", 2024-09-13T11:25:58.608 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 4267062742 2024-09-13T11:25:58.608 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:58.608 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:58.608 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:58.608 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6812/4267062742", 2024-09-13T11:25:58.609 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_addr": "172.21.15.1:6813/4267062742", 2024-09-13T11:25:58.609 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_back_addr": "172.21.15.1:6815/4267062742", 2024-09-13T11:25:58.609 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_front_addr": "172.21.15.1:6814/4267062742", 2024-09-13T11:25:58.609 INFO:teuthology.orchestra.run.smithi001.stdout: "state": [ 2024-09-13T11:25:58.609 INFO:teuthology.orchestra.run.smithi001.stdout: "exists", 2024-09-13T11:25:58.609 INFO:teuthology.orchestra.run.smithi001.stdout: "up" 2024-09-13T11:25:58.609 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:58.609 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:58.609 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:58.609 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": 13, 2024-09-13T11:25:58.609 INFO:teuthology.orchestra.run.smithi001.stdout: "uuid": "5d321d4b-1b26-47fe-bf84-25751b8cde7d", 2024-09-13T11:25:58.609 INFO:teuthology.orchestra.run.smithi001.stdout: "up": 1, 2024-09-13T11:25:58.609 INFO:teuthology.orchestra.run.smithi001.stdout: "in": 1, 2024-09-13T11:25:58.609 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 1, 2024-09-13T11:25:58.610 INFO:teuthology.orchestra.run.smithi001.stdout: "primary_affinity": 1, 2024-09-13T11:25:58.610 INFO:teuthology.orchestra.run.smithi001.stdout: "last_clean_begin": 0, 2024-09-13T11:25:58.610 INFO:teuthology.orchestra.run.smithi001.stdout: "last_clean_end": 0, 2024-09-13T11:25:58.610 INFO:teuthology.orchestra.run.smithi001.stdout: "up_from": 23, 2024-09-13T11:25:58.610 INFO:teuthology.orchestra.run.smithi001.stdout: "up_thru": 140, 2024-09-13T11:25:58.610 INFO:teuthology.orchestra.run.smithi001.stdout: "down_at": 0, 2024-09-13T11:25:58.610 INFO:teuthology.orchestra.run.smithi001.stdout: "lost_at": 0, 2024-09-13T11:25:58.610 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-09-13T11:25:58.610 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:25:58.610 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:58.610 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:25:58.610 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.92:6808", 2024-09-13T11:25:58.610 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 1759585046 2024-09-13T11:25:58.611 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:58.611 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:58.611 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:58.611 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_addrs": { 2024-09-13T11:25:58.611 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:25:58.611 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:58.611 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:25:58.611 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.92:6809", 2024-09-13T11:25:58.611 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 1759585046 2024-09-13T11:25:58.611 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:58.611 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:58.611 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:58.611 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_back_addrs": { 2024-09-13T11:25:58.612 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:25:58.612 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:58.612 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:25:58.612 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.92:6811", 2024-09-13T11:25:58.612 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 1759585046 2024-09-13T11:25:58.612 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:58.612 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:58.612 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:58.612 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_front_addrs": { 2024-09-13T11:25:58.612 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:25:58.612 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:58.612 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:25:58.612 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.92:6810", 2024-09-13T11:25:58.613 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 1759585046 2024-09-13T11:25:58.613 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:58.613 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:58.613 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:58.613 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.92:6808/1759585046", 2024-09-13T11:25:58.613 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_addr": "172.21.15.92:6809/1759585046", 2024-09-13T11:25:58.613 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_back_addr": "172.21.15.92:6811/1759585046", 2024-09-13T11:25:58.613 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_front_addr": "172.21.15.92:6810/1759585046", 2024-09-13T11:25:58.613 INFO:teuthology.orchestra.run.smithi001.stdout: "state": [ 2024-09-13T11:25:58.613 INFO:teuthology.orchestra.run.smithi001.stdout: "exists", 2024-09-13T11:25:58.613 INFO:teuthology.orchestra.run.smithi001.stdout: "up" 2024-09-13T11:25:58.613 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:58.613 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:58.614 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:58.614 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": 14, 2024-09-13T11:25:58.614 INFO:teuthology.orchestra.run.smithi001.stdout: "uuid": "fc3e2499-0c2a-414d-a25e-2628876ae630", 2024-09-13T11:25:58.614 INFO:teuthology.orchestra.run.smithi001.stdout: "up": 1, 2024-09-13T11:25:58.614 INFO:teuthology.orchestra.run.smithi001.stdout: "in": 0, 2024-09-13T11:25:58.614 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-09-13T11:25:58.614 INFO:teuthology.orchestra.run.smithi001.stdout: "primary_affinity": 1, 2024-09-13T11:25:58.614 INFO:teuthology.orchestra.run.smithi001.stdout: "last_clean_begin": 22, 2024-09-13T11:25:58.614 INFO:teuthology.orchestra.run.smithi001.stdout: "last_clean_end": 120, 2024-09-13T11:25:58.614 INFO:teuthology.orchestra.run.smithi001.stdout: "up_from": 142, 2024-09-13T11:25:58.614 INFO:teuthology.orchestra.run.smithi001.stdout: "up_thru": 85, 2024-09-13T11:25:58.614 INFO:teuthology.orchestra.run.smithi001.stdout: "down_at": 121, 2024-09-13T11:25:58.614 INFO:teuthology.orchestra.run.smithi001.stdout: "lost_at": 0, 2024-09-13T11:25:58.615 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-09-13T11:25:58.615 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:25:58.615 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:58.615 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:25:58.615 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.162:6800", 2024-09-13T11:25:58.615 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 3999392664 2024-09-13T11:25:58.615 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:58.615 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:58.615 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:58.615 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_addrs": { 2024-09-13T11:25:58.615 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:25:58.615 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:58.615 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:25:58.616 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.162:6801", 2024-09-13T11:25:58.616 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 3999392664 2024-09-13T11:25:58.616 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:58.616 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:58.616 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:58.616 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_back_addrs": { 2024-09-13T11:25:58.616 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:25:58.616 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:58.616 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:25:58.616 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.162:6803", 2024-09-13T11:25:58.616 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 3999392664 2024-09-13T11:25:58.616 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:58.616 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:58.617 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:58.617 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_front_addrs": { 2024-09-13T11:25:58.617 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:25:58.617 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:58.617 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:25:58.617 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.162:6802", 2024-09-13T11:25:58.617 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 3999392664 2024-09-13T11:25:58.617 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:58.617 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:58.617 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:58.617 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.162:6800/3999392664", 2024-09-13T11:25:58.617 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_addr": "172.21.15.162:6801/3999392664", 2024-09-13T11:25:58.617 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_back_addr": "172.21.15.162:6803/3999392664", 2024-09-13T11:25:58.618 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_front_addr": "172.21.15.162:6802/3999392664", 2024-09-13T11:25:58.618 INFO:teuthology.orchestra.run.smithi001.stdout: "state": [ 2024-09-13T11:25:58.618 INFO:teuthology.orchestra.run.smithi001.stdout: "exists", 2024-09-13T11:25:58.618 INFO:teuthology.orchestra.run.smithi001.stdout: "up" 2024-09-13T11:25:58.618 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:58.618 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:58.618 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:58.618 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": 15, 2024-09-13T11:25:58.618 INFO:teuthology.orchestra.run.smithi001.stdout: "uuid": "d2f69775-edfe-4527-91ec-f7c7e558bec8", 2024-09-13T11:25:58.618 INFO:teuthology.orchestra.run.smithi001.stdout: "up": 1, 2024-09-13T11:25:58.618 INFO:teuthology.orchestra.run.smithi001.stdout: "in": 1, 2024-09-13T11:25:58.618 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 1, 2024-09-13T11:25:58.618 INFO:teuthology.orchestra.run.smithi001.stdout: "primary_affinity": 1, 2024-09-13T11:25:58.618 INFO:teuthology.orchestra.run.smithi001.stdout: "last_clean_begin": 0, 2024-09-13T11:25:58.619 INFO:teuthology.orchestra.run.smithi001.stdout: "last_clean_end": 0, 2024-09-13T11:25:58.619 INFO:teuthology.orchestra.run.smithi001.stdout: "up_from": 22, 2024-09-13T11:25:58.619 INFO:teuthology.orchestra.run.smithi001.stdout: "up_thru": 122, 2024-09-13T11:25:58.619 INFO:teuthology.orchestra.run.smithi001.stdout: "down_at": 0, 2024-09-13T11:25:58.619 INFO:teuthology.orchestra.run.smithi001.stdout: "lost_at": 0, 2024-09-13T11:25:58.619 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-09-13T11:25:58.619 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:25:58.619 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:58.619 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:25:58.619 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.177:6805", 2024-09-13T11:25:58.619 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 1294337172 2024-09-13T11:25:58.619 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:58.619 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:58.619 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:58.620 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_addrs": { 2024-09-13T11:25:58.620 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:25:58.620 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:58.620 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:25:58.620 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.177:6806", 2024-09-13T11:25:58.620 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 1294337172 2024-09-13T11:25:58.620 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:58.620 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:58.620 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:58.620 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_back_addrs": { 2024-09-13T11:25:58.620 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:25:58.620 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:58.620 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:25:58.621 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.177:6808", 2024-09-13T11:25:58.621 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 1294337172 2024-09-13T11:25:58.621 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:58.621 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:58.621 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:58.621 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_front_addrs": { 2024-09-13T11:25:58.621 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:25:58.621 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:58.621 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:25:58.621 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.177:6807", 2024-09-13T11:25:58.621 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 1294337172 2024-09-13T11:25:58.621 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:58.621 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:58.621 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:58.622 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.177:6805/1294337172", 2024-09-13T11:25:58.622 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_addr": "172.21.15.177:6806/1294337172", 2024-09-13T11:25:58.622 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_back_addr": "172.21.15.177:6808/1294337172", 2024-09-13T11:25:58.622 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_front_addr": "172.21.15.177:6807/1294337172", 2024-09-13T11:25:58.622 INFO:teuthology.orchestra.run.smithi001.stdout: "state": [ 2024-09-13T11:25:58.622 INFO:teuthology.orchestra.run.smithi001.stdout: "exists", 2024-09-13T11:25:58.622 INFO:teuthology.orchestra.run.smithi001.stdout: "up" 2024-09-13T11:25:58.622 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:58.622 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:58.622 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-09-13T11:25:58.622 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_xinfo": [ 2024-09-13T11:25:58.622 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:58.622 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": 0, 2024-09-13T11:25:58.622 INFO:teuthology.orchestra.run.smithi001.stdout: "down_stamp": "0.000000", 2024-09-13T11:25:58.623 INFO:teuthology.orchestra.run.smithi001.stdout: "laggy_probability": 0, 2024-09-13T11:25:58.623 INFO:teuthology.orchestra.run.smithi001.stdout: "laggy_interval": 0, 2024-09-13T11:25:58.623 INFO:teuthology.orchestra.run.smithi001.stdout: "features": 4540719139924082687, 2024-09-13T11:25:58.623 INFO:teuthology.orchestra.run.smithi001.stdout: "old_weight": 0, 2024-09-13T11:25:58.623 INFO:teuthology.orchestra.run.smithi001.stdout: "last_purged_snaps_scrub": "2024-09-13T11:24:57.434802+0000", 2024-09-13T11:25:58.623 INFO:teuthology.orchestra.run.smithi001.stdout: "dead_epoch": 0 2024-09-13T11:25:58.623 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:58.623 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:58.623 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": 1, 2024-09-13T11:25:58.623 INFO:teuthology.orchestra.run.smithi001.stdout: "down_stamp": "0.000000", 2024-09-13T11:25:58.624 INFO:teuthology.orchestra.run.smithi001.stdout: "laggy_probability": 0, 2024-09-13T11:25:58.624 INFO:teuthology.orchestra.run.smithi001.stdout: "laggy_interval": 0, 2024-09-13T11:25:58.624 INFO:teuthology.orchestra.run.smithi001.stdout: "features": 4540719139924082687, 2024-09-13T11:25:58.624 INFO:teuthology.orchestra.run.smithi001.stdout: "old_weight": 0, 2024-09-13T11:25:58.624 INFO:teuthology.orchestra.run.smithi001.stdout: "last_purged_snaps_scrub": "2024-09-13T11:25:47.780833+0000", 2024-09-13T11:25:58.624 INFO:teuthology.orchestra.run.smithi001.stdout: "dead_epoch": 30 2024-09-13T11:25:58.624 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:58.624 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:58.624 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": 2, 2024-09-13T11:25:58.624 INFO:teuthology.orchestra.run.smithi001.stdout: "down_stamp": "0.000000", 2024-09-13T11:25:58.624 INFO:teuthology.orchestra.run.smithi001.stdout: "laggy_probability": 0, 2024-09-13T11:25:58.624 INFO:teuthology.orchestra.run.smithi001.stdout: "laggy_interval": 0, 2024-09-13T11:25:58.624 INFO:teuthology.orchestra.run.smithi001.stdout: "features": 4540719139924082687, 2024-09-13T11:25:58.625 INFO:teuthology.orchestra.run.smithi001.stdout: "old_weight": 0, 2024-09-13T11:25:58.625 INFO:teuthology.orchestra.run.smithi001.stdout: "last_purged_snaps_scrub": "2024-09-13T11:24:48.897188+0000", 2024-09-13T11:25:58.625 INFO:teuthology.orchestra.run.smithi001.stdout: "dead_epoch": 0 2024-09-13T11:25:58.625 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:58.625 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:58.625 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": 3, 2024-09-13T11:25:58.625 INFO:teuthology.orchestra.run.smithi001.stdout: "down_stamp": "0.000000", 2024-09-13T11:25:58.625 INFO:teuthology.orchestra.run.smithi001.stdout: "laggy_probability": 0, 2024-09-13T11:25:58.625 INFO:teuthology.orchestra.run.smithi001.stdout: "laggy_interval": 0, 2024-09-13T11:25:58.625 INFO:teuthology.orchestra.run.smithi001.stdout: "features": 4540719139924082687, 2024-09-13T11:25:58.625 INFO:teuthology.orchestra.run.smithi001.stdout: "old_weight": 0, 2024-09-13T11:25:58.625 INFO:teuthology.orchestra.run.smithi001.stdout: "last_purged_snaps_scrub": "2024-09-13T11:24:47.037766+0000", 2024-09-13T11:25:58.625 INFO:teuthology.orchestra.run.smithi001.stdout: "dead_epoch": 0 2024-09-13T11:25:58.626 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:58.626 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:58.626 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": 4, 2024-09-13T11:25:58.626 INFO:teuthology.orchestra.run.smithi001.stdout: "down_stamp": "0.000000", 2024-09-13T11:25:58.626 INFO:teuthology.orchestra.run.smithi001.stdout: "laggy_probability": 0, 2024-09-13T11:25:58.626 INFO:teuthology.orchestra.run.smithi001.stdout: "laggy_interval": 0, 2024-09-13T11:25:58.626 INFO:teuthology.orchestra.run.smithi001.stdout: "features": 4540719139924082687, 2024-09-13T11:25:58.626 INFO:teuthology.orchestra.run.smithi001.stdout: "old_weight": 0, 2024-09-13T11:25:58.626 INFO:teuthology.orchestra.run.smithi001.stdout: "last_purged_snaps_scrub": "2024-09-13T11:24:52.121278+0000", 2024-09-13T11:25:58.626 INFO:teuthology.orchestra.run.smithi001.stdout: "dead_epoch": 0 2024-09-13T11:25:58.626 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:58.626 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:58.626 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": 5, 2024-09-13T11:25:58.627 INFO:teuthology.orchestra.run.smithi001.stdout: "down_stamp": "0.000000", 2024-09-13T11:25:58.627 INFO:teuthology.orchestra.run.smithi001.stdout: "laggy_probability": 0, 2024-09-13T11:25:58.627 INFO:teuthology.orchestra.run.smithi001.stdout: "laggy_interval": 0, 2024-09-13T11:25:58.627 INFO:teuthology.orchestra.run.smithi001.stdout: "features": 4540719139924082687, 2024-09-13T11:25:58.627 INFO:teuthology.orchestra.run.smithi001.stdout: "old_weight": 0, 2024-09-13T11:25:58.627 INFO:teuthology.orchestra.run.smithi001.stdout: "last_purged_snaps_scrub": "2024-09-13T11:24:59.304368+0000", 2024-09-13T11:25:58.627 INFO:teuthology.orchestra.run.smithi001.stdout: "dead_epoch": 0 2024-09-13T11:25:58.627 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:58.627 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:58.627 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": 6, 2024-09-13T11:25:58.627 INFO:teuthology.orchestra.run.smithi001.stdout: "down_stamp": "0.000000", 2024-09-13T11:25:58.627 INFO:teuthology.orchestra.run.smithi001.stdout: "laggy_probability": 0, 2024-09-13T11:25:58.627 INFO:teuthology.orchestra.run.smithi001.stdout: "laggy_interval": 0, 2024-09-13T11:25:58.627 INFO:teuthology.orchestra.run.smithi001.stdout: "features": 4540719139924082687, 2024-09-13T11:25:58.628 INFO:teuthology.orchestra.run.smithi001.stdout: "old_weight": 0, 2024-09-13T11:25:58.628 INFO:teuthology.orchestra.run.smithi001.stdout: "last_purged_snaps_scrub": "2024-09-13T11:24:52.504800+0000", 2024-09-13T11:25:58.628 INFO:teuthology.orchestra.run.smithi001.stdout: "dead_epoch": 0 2024-09-13T11:25:58.628 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:58.628 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:58.628 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": 7, 2024-09-13T11:25:58.628 INFO:teuthology.orchestra.run.smithi001.stdout: "down_stamp": "0.000000", 2024-09-13T11:25:58.628 INFO:teuthology.orchestra.run.smithi001.stdout: "laggy_probability": 0, 2024-09-13T11:25:58.628 INFO:teuthology.orchestra.run.smithi001.stdout: "laggy_interval": 0, 2024-09-13T11:25:58.628 INFO:teuthology.orchestra.run.smithi001.stdout: "features": 4540719139924082687, 2024-09-13T11:25:58.628 INFO:teuthology.orchestra.run.smithi001.stdout: "old_weight": 0, 2024-09-13T11:25:58.628 INFO:teuthology.orchestra.run.smithi001.stdout: "last_purged_snaps_scrub": "2024-09-13T11:24:44.955554+0000", 2024-09-13T11:25:58.628 INFO:teuthology.orchestra.run.smithi001.stdout: "dead_epoch": 0 2024-09-13T11:25:58.629 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:58.629 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:58.629 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": 8, 2024-09-13T11:25:58.629 INFO:teuthology.orchestra.run.smithi001.stdout: "down_stamp": "0.000000", 2024-09-13T11:25:58.629 INFO:teuthology.orchestra.run.smithi001.stdout: "laggy_probability": 0, 2024-09-13T11:25:58.629 INFO:teuthology.orchestra.run.smithi001.stdout: "laggy_interval": 0, 2024-09-13T11:25:58.629 INFO:teuthology.orchestra.run.smithi001.stdout: "features": 4540719139924082687, 2024-09-13T11:25:58.629 INFO:teuthology.orchestra.run.smithi001.stdout: "old_weight": 0, 2024-09-13T11:25:58.629 INFO:teuthology.orchestra.run.smithi001.stdout: "last_purged_snaps_scrub": "2024-09-13T11:24:50.514485+0000", 2024-09-13T11:25:58.629 INFO:teuthology.orchestra.run.smithi001.stdout: "dead_epoch": 0 2024-09-13T11:25:58.629 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:58.630 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:58.630 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": 9, 2024-09-13T11:25:58.630 INFO:teuthology.orchestra.run.smithi001.stdout: "down_stamp": "0.000000", 2024-09-13T11:25:58.630 INFO:teuthology.orchestra.run.smithi001.stdout: "laggy_probability": 0, 2024-09-13T11:25:58.630 INFO:teuthology.orchestra.run.smithi001.stdout: "laggy_interval": 0, 2024-09-13T11:25:58.630 INFO:teuthology.orchestra.run.smithi001.stdout: "features": 4540719139924082687, 2024-09-13T11:25:58.630 INFO:teuthology.orchestra.run.smithi001.stdout: "old_weight": 0, 2024-09-13T11:25:58.630 INFO:teuthology.orchestra.run.smithi001.stdout: "last_purged_snaps_scrub": "2024-09-13T11:25:00.288643+0000", 2024-09-13T11:25:58.630 INFO:teuthology.orchestra.run.smithi001.stdout: "dead_epoch": 0 2024-09-13T11:25:58.630 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:58.630 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:58.631 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": 10, 2024-09-13T11:25:58.631 INFO:teuthology.orchestra.run.smithi001.stdout: "down_stamp": "0.000000", 2024-09-13T11:25:58.631 INFO:teuthology.orchestra.run.smithi001.stdout: "laggy_probability": 0, 2024-09-13T11:25:58.631 INFO:teuthology.orchestra.run.smithi001.stdout: "laggy_interval": 0, 2024-09-13T11:25:58.631 INFO:teuthology.orchestra.run.smithi001.stdout: "features": 4540719139924082687, 2024-09-13T11:25:58.631 INFO:teuthology.orchestra.run.smithi001.stdout: "old_weight": 0, 2024-09-13T11:25:58.631 INFO:teuthology.orchestra.run.smithi001.stdout: "last_purged_snaps_scrub": "2024-09-13T11:24:38.091398+0000", 2024-09-13T11:25:58.631 INFO:teuthology.orchestra.run.smithi001.stdout: "dead_epoch": 0 2024-09-13T11:25:58.631 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:58.631 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:58.631 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": 11, 2024-09-13T11:25:58.631 INFO:teuthology.orchestra.run.smithi001.stdout: "down_stamp": "0.000000", 2024-09-13T11:25:58.631 INFO:teuthology.orchestra.run.smithi001.stdout: "laggy_probability": 0, 2024-09-13T11:25:58.631 INFO:teuthology.orchestra.run.smithi001.stdout: "laggy_interval": 0, 2024-09-13T11:25:58.632 INFO:teuthology.orchestra.run.smithi001.stdout: "features": 4540719139924082687, 2024-09-13T11:25:58.632 INFO:teuthology.orchestra.run.smithi001.stdout: "old_weight": 0, 2024-09-13T11:25:58.632 INFO:teuthology.orchestra.run.smithi001.stdout: "last_purged_snaps_scrub": "2024-09-13T11:25:01.213643+0000", 2024-09-13T11:25:58.632 INFO:teuthology.orchestra.run.smithi001.stdout: "dead_epoch": 0 2024-09-13T11:25:58.632 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:58.632 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:58.632 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": 12, 2024-09-13T11:25:58.632 INFO:teuthology.orchestra.run.smithi001.stdout: "down_stamp": "0.000000", 2024-09-13T11:25:58.632 INFO:teuthology.orchestra.run.smithi001.stdout: "laggy_probability": 0, 2024-09-13T11:25:58.632 INFO:teuthology.orchestra.run.smithi001.stdout: "laggy_interval": 0, 2024-09-13T11:25:58.632 INFO:teuthology.orchestra.run.smithi001.stdout: "features": 4540719139924082687, 2024-09-13T11:25:58.632 INFO:teuthology.orchestra.run.smithi001.stdout: "old_weight": 0, 2024-09-13T11:25:58.632 INFO:teuthology.orchestra.run.smithi001.stdout: "last_purged_snaps_scrub": "2024-09-13T11:25:47.548386+0000", 2024-09-13T11:25:58.632 INFO:teuthology.orchestra.run.smithi001.stdout: "dead_epoch": 0 2024-09-13T11:25:58.633 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:58.633 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:58.633 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": 13, 2024-09-13T11:25:58.633 INFO:teuthology.orchestra.run.smithi001.stdout: "down_stamp": "0.000000", 2024-09-13T11:25:58.633 INFO:teuthology.orchestra.run.smithi001.stdout: "laggy_probability": 0, 2024-09-13T11:25:58.633 INFO:teuthology.orchestra.run.smithi001.stdout: "laggy_interval": 0, 2024-09-13T11:25:58.633 INFO:teuthology.orchestra.run.smithi001.stdout: "features": 4540719139924082687, 2024-09-13T11:25:58.633 INFO:teuthology.orchestra.run.smithi001.stdout: "old_weight": 0, 2024-09-13T11:25:58.633 INFO:teuthology.orchestra.run.smithi001.stdout: "last_purged_snaps_scrub": "2024-09-13T11:25:45.821466+0000", 2024-09-13T11:25:58.633 INFO:teuthology.orchestra.run.smithi001.stdout: "dead_epoch": 0 2024-09-13T11:25:58.633 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:58.633 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:58.633 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": 14, 2024-09-13T11:25:58.633 INFO:teuthology.orchestra.run.smithi001.stdout: "down_stamp": "0.000000", 2024-09-13T11:25:58.634 INFO:teuthology.orchestra.run.smithi001.stdout: "laggy_probability": 0, 2024-09-13T11:25:58.634 INFO:teuthology.orchestra.run.smithi001.stdout: "laggy_interval": 0, 2024-09-13T11:25:58.634 INFO:teuthology.orchestra.run.smithi001.stdout: "features": 4540719139924082687, 2024-09-13T11:25:58.634 INFO:teuthology.orchestra.run.smithi001.stdout: "old_weight": 65536, 2024-09-13T11:25:58.634 INFO:teuthology.orchestra.run.smithi001.stdout: "last_purged_snaps_scrub": "2024-09-13T11:24:55.108407+0000", 2024-09-13T11:25:58.634 INFO:teuthology.orchestra.run.smithi001.stdout: "dead_epoch": 121 2024-09-13T11:25:58.634 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:58.634 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:58.634 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": 15, 2024-09-13T11:25:58.634 INFO:teuthology.orchestra.run.smithi001.stdout: "down_stamp": "0.000000", 2024-09-13T11:25:58.634 INFO:teuthology.orchestra.run.smithi001.stdout: "laggy_probability": 0, 2024-09-13T11:25:58.634 INFO:teuthology.orchestra.run.smithi001.stdout: "laggy_interval": 0, 2024-09-13T11:25:58.634 INFO:teuthology.orchestra.run.smithi001.stdout: "features": 4540719139924082687, 2024-09-13T11:25:58.635 INFO:teuthology.orchestra.run.smithi001.stdout: "old_weight": 0, 2024-09-13T11:25:58.635 INFO:teuthology.orchestra.run.smithi001.stdout: "last_purged_snaps_scrub": "2024-09-13T11:24:33.420656+0000", 2024-09-13T11:25:58.635 INFO:teuthology.orchestra.run.smithi001.stdout: "dead_epoch": 0 2024-09-13T11:25:58.635 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:58.635 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-09-13T11:25:58.635 INFO:teuthology.orchestra.run.smithi001.stdout: "pg_upmap": [ 2024-09-13T11:25:58.635 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:58.635 INFO:teuthology.orchestra.run.smithi001.stdout: "pgid": "2.2", 2024-09-13T11:25:58.635 INFO:teuthology.orchestra.run.smithi001.stdout: "osds": [ 2024-09-13T11:25:58.635 INFO:teuthology.orchestra.run.smithi001.stdout: 2, 2024-09-13T11:25:58.635 INFO:teuthology.orchestra.run.smithi001.stdout: 8 2024-09-13T11:25:58.635 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:58.635 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:58.635 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:58.636 INFO:teuthology.orchestra.run.smithi001.stdout: "pgid": "3.e", 2024-09-13T11:25:58.636 INFO:teuthology.orchestra.run.smithi001.stdout: "osds": [ 2024-09-13T11:25:58.636 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-09-13T11:25:58.636 INFO:teuthology.orchestra.run.smithi001.stdout: 1 2024-09-13T11:25:58.636 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:58.636 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:58.636 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-09-13T11:25:58.636 INFO:teuthology.orchestra.run.smithi001.stdout: "pg_upmap_items": [], 2024-09-13T11:25:58.636 INFO:teuthology.orchestra.run.smithi001.stdout: "pg_upmap_primaries": [], 2024-09-13T11:25:58.636 INFO:teuthology.orchestra.run.smithi001.stdout: "pg_temp": [], 2024-09-13T11:25:58.636 INFO:teuthology.orchestra.run.smithi001.stdout: "primary_temp": [], 2024-09-13T11:25:58.636 INFO:teuthology.orchestra.run.smithi001.stdout: "blocklist": {}, 2024-09-13T11:25:58.636 INFO:teuthology.orchestra.run.smithi001.stdout: "range_blocklist": {}, 2024-09-13T11:25:58.637 INFO:teuthology.orchestra.run.smithi001.stdout: "erasure_code_profiles": { 2024-09-13T11:25:58.637 INFO:teuthology.orchestra.run.smithi001.stdout: "default": { 2024-09-13T11:25:58.637 INFO:teuthology.orchestra.run.smithi001.stdout: "crush-failure-domain": "osd", 2024-09-13T11:25:58.637 INFO:teuthology.orchestra.run.smithi001.stdout: "k": "2", 2024-09-13T11:25:58.637 INFO:teuthology.orchestra.run.smithi001.stdout: "m": "1", 2024-09-13T11:25:58.637 INFO:teuthology.orchestra.run.smithi001.stdout: "plugin": "jerasure", 2024-09-13T11:25:58.637 INFO:teuthology.orchestra.run.smithi001.stdout: "technique": "reed_sol_van" 2024-09-13T11:25:58.637 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:58.637 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:58.637 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_snaps_queue": [], 2024-09-13T11:25:58.637 INFO:teuthology.orchestra.run.smithi001.stdout: "new_removed_snaps": [], 2024-09-13T11:25:58.637 INFO:teuthology.orchestra.run.smithi001.stdout: "new_purged_snaps": [], 2024-09-13T11:25:58.637 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_node_flags": {}, 2024-09-13T11:25:58.638 INFO:teuthology.orchestra.run.smithi001.stdout: "device_class_flags": {}, 2024-09-13T11:25:58.638 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": { 2024-09-13T11:25:58.638 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode_enabled": false, 2024-09-13T11:25:58.638 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_bucket_count": 0, 2024-09-13T11:25:58.638 INFO:teuthology.orchestra.run.smithi001.stdout: "degraded_stretch_mode": 0, 2024-09-13T11:25:58.638 INFO:teuthology.orchestra.run.smithi001.stdout: "recovering_stretch_mode": 0, 2024-09-13T11:25:58.638 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode_bucket": 0 2024-09-13T11:25:58.638 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:58.638 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-13T11:25:58.639 INFO:teuthology.orchestra.run.smithi001.stdout:osd.2: {} 2024-09-13T11:25:58.640 INFO:teuthology.orchestra.run.smithi001.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T11:25:58.640 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-09-13T11:25:58.642 INFO:tasks.thrashosds.thrasher:j is {'epoch': 186, 'fsid': '2fd68ac7-5581-4d90-80a4-30bcf5252a7a', 'created': '2024-09-13T11:21:57.223153+0000', 'modified': '2024-09-13T11:25:55.955245+0000', 'last_up_change': '2024-09-13T11:24:55.968636+0000', 'last_in_change': '2024-09-13T11:24:28.080124+0000', 'flags': 'nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5804032, 'flags_set': ['nodeep-scrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 7, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 3, 'max_osd': 16, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-13T11:22:10.162266+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 15.380000114440918, 'score_stable': 15.380000114440918, 'optimal_score': 0.12999999523162842, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2024-09-13T11:22:12.235549+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '28', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 2, 'snap_epoch': 28, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 3.9700000286102295, 'score_stable': 3.9700000286102295, 'optimal_score': 0.6299999952316284, 'raw_score_acting': 2.5, 'raw_score_stable': 2.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'unique_pool_0', 'create_time': '2024-09-13T11:22:24.305521+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 46, 'pg_placement_num': 46, 'pg_placement_num_target': 46, 'pg_num_target': 46, 'pg_num_pending': 46, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '119', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '117', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.0899999141693115, 'score_stable': 2.0899999141693115, 'optimal_score': 0.9399999976158142, 'raw_score_acting': 1.9600000381469727, 'raw_score_stable': 1.9600000381469727, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}], 'osds': [{'osd': 0, 'uuid': '0fc12b3c-e0de-4546-b935-db77efe7c567', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 22, 'up_thru': 175, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.1:6800', 'nonce': 1544776477}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.1:6801', 'nonce': 1544776477}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.1:6803', 'nonce': 1544776477}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.1:6802', 'nonce': 1544776477}]}, 'public_addr': '172.21.15.1:6800/1544776477', 'cluster_addr': '172.21.15.1:6801/1544776477', 'heartbeat_back_addr': '172.21.15.1:6803/1544776477', 'heartbeat_front_addr': '172.21.15.1:6802/1544776477', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': 'b77fe331-cc92-4adb-b146-af5c380f6d4f', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 23, 'last_clean_end': 29, 'up_from': 40, 'up_thru': 171, 'down_at': 30, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.92:6800', 'nonce': 3591430847}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.92:6801', 'nonce': 3591430847}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.92:6803', 'nonce': 3591430847}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.92:6802', 'nonce': 3591430847}]}, 'public_addr': '172.21.15.92:6800/3591430847', 'cluster_addr': '172.21.15.92:6801/3591430847', 'heartbeat_back_addr': '172.21.15.92:6803/3591430847', 'heartbeat_front_addr': '172.21.15.92:6802/3591430847', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': 'c9d031c8-e885-4229-affa-08e1729d00e8', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 22, 'up_thru': 152, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.162:6812', 'nonce': 2712148101}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.162:6813', 'nonce': 2712148101}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.162:6815', 'nonce': 2712148101}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.162:6814', 'nonce': 2712148101}]}, 'public_addr': '172.21.15.162:6812/2712148101', 'cluster_addr': '172.21.15.162:6813/2712148101', 'heartbeat_back_addr': '172.21.15.162:6815/2712148101', 'heartbeat_front_addr': '172.21.15.162:6814/2712148101', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': 'bf50d71a-6c3a-4fbc-a407-cb81bad14a7b', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 21, 'up_thru': 174, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.177:6801', 'nonce': 3367751690}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.177:6802', 'nonce': 3367751690}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.177:6804', 'nonce': 3367751690}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.177:6803', 'nonce': 3367751690}]}, 'public_addr': '172.21.15.177:6801/3367751690', 'cluster_addr': '172.21.15.177:6802/3367751690', 'heartbeat_back_addr': '172.21.15.177:6804/3367751690', 'heartbeat_front_addr': '172.21.15.177:6803/3367751690', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': 'd1fe3f63-df3f-4a73-a1e2-8d7c8a3f1fa0', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 22, 'up_thru': 105, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.1:6808', 'nonce': 3648540926}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.1:6809', 'nonce': 3648540926}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.1:6811', 'nonce': 3648540926}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.1:6810', 'nonce': 3648540926}]}, 'public_addr': '172.21.15.1:6808/3648540926', 'cluster_addr': '172.21.15.1:6809/3648540926', 'heartbeat_back_addr': '172.21.15.1:6811/3648540926', 'heartbeat_front_addr': '172.21.15.1:6810/3648540926', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': 'e2ead447-2f2c-40c9-82e7-53b8dc41e2af', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 21, 'up_thru': 169, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.92:6804', 'nonce': 3399529091}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.92:6805', 'nonce': 3399529091}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.92:6807', 'nonce': 3399529091}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.92:6806', 'nonce': 3399529091}]}, 'public_addr': '172.21.15.92:6804/3399529091', 'cluster_addr': '172.21.15.92:6805/3399529091', 'heartbeat_back_addr': '172.21.15.92:6807/3399529091', 'heartbeat_front_addr': '172.21.15.92:6806/3399529091', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': 'dbc20439-144e-4ec7-89de-674eb783f263', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 22, 'up_thru': 117, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.162:6808', 'nonce': 2287343806}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.162:6809', 'nonce': 2287343806}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.162:6811', 'nonce': 2287343806}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.162:6810', 'nonce': 2287343806}]}, 'public_addr': '172.21.15.162:6808/2287343806', 'cluster_addr': '172.21.15.162:6809/2287343806', 'heartbeat_back_addr': '172.21.15.162:6811/2287343806', 'heartbeat_front_addr': '172.21.15.162:6810/2287343806', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '60aa5206-cace-40f8-8d1a-10ec28d97096', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 22, 'up_thru': 122, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.177:6813', 'nonce': 2110809906}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.177:6814', 'nonce': 2110809906}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.177:6816', 'nonce': 2110809906}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.177:6815', 'nonce': 2110809906}]}, 'public_addr': '172.21.15.177:6813/2110809906', 'cluster_addr': '172.21.15.177:6814/2110809906', 'heartbeat_back_addr': '172.21.15.177:6816/2110809906', 'heartbeat_front_addr': '172.21.15.177:6815/2110809906', 'state': ['exists', 'up']}, {'osd': 8, 'uuid': 'f3d99325-48ff-403c-8662-1d7745f7c97d', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 22, 'up_thru': 172, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.1:6804', 'nonce': 2766567910}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.1:6805', 'nonce': 2766567910}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.1:6807', 'nonce': 2766567910}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.1:6806', 'nonce': 2766567910}]}, 'public_addr': '172.21.15.1:6804/2766567910', 'cluster_addr': '172.21.15.1:6805/2766567910', 'heartbeat_back_addr': '172.21.15.1:6807/2766567910', 'heartbeat_front_addr': '172.21.15.1:6806/2766567910', 'state': ['exists', 'up']}, {'osd': 9, 'uuid': '88511e24-3675-4551-968c-1a52e7e28f96', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 23, 'up_thru': 133, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.92:6812', 'nonce': 2488187511}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.92:6813', 'nonce': 2488187511}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.92:6815', 'nonce': 2488187511}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.92:6814', 'nonce': 2488187511}]}, 'public_addr': '172.21.15.92:6812/2488187511', 'cluster_addr': '172.21.15.92:6813/2488187511', 'heartbeat_back_addr': '172.21.15.92:6815/2488187511', 'heartbeat_front_addr': '172.21.15.92:6814/2488187511', 'state': ['exists', 'up']}, {'osd': 10, 'uuid': 'acf4c64d-99af-42be-8406-413c86f54670', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 22, 'up_thru': 144, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.162:6804', 'nonce': 1450266459}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.162:6805', 'nonce': 1450266459}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.162:6807', 'nonce': 1450266459}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.162:6806', 'nonce': 1450266459}]}, 'public_addr': '172.21.15.162:6804/1450266459', 'cluster_addr': '172.21.15.162:6805/1450266459', 'heartbeat_back_addr': '172.21.15.162:6807/1450266459', 'heartbeat_front_addr': '172.21.15.162:6806/1450266459', 'state': ['exists', 'up']}, {'osd': 11, 'uuid': 'a4bfdfbe-b586-4080-8e09-6f26428e59b5', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 22, 'up_thru': 143, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.177:6809', 'nonce': 3239521954}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.177:6810', 'nonce': 3239521954}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.177:6812', 'nonce': 3239521954}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.177:6811', 'nonce': 3239521954}]}, 'public_addr': '172.21.15.177:6809/3239521954', 'cluster_addr': '172.21.15.177:6810/3239521954', 'heartbeat_back_addr': '172.21.15.177:6812/3239521954', 'heartbeat_front_addr': '172.21.15.177:6811/3239521954', 'state': ['exists', 'up']}, {'osd': 12, 'uuid': 'dc571fb6-d8b5-465b-bb25-afa7d0c47b5a', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 22, 'up_thru': 141, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.1:6812', 'nonce': 4267062742}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.1:6813', 'nonce': 4267062742}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.1:6815', 'nonce': 4267062742}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.1:6814', 'nonce': 4267062742}]}, 'public_addr': '172.21.15.1:6812/4267062742', 'cluster_addr': '172.21.15.1:6813/4267062742', 'heartbeat_back_addr': '172.21.15.1:6815/4267062742', 'heartbeat_front_addr': '172.21.15.1:6814/4267062742', 'state': ['exists', 'up']}, {'osd': 13, 'uuid': '5d321d4b-1b26-47fe-bf84-25751b8cde7d', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 23, 'up_thru': 140, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.92:6808', 'nonce': 1759585046}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.92:6809', 'nonce': 1759585046}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.92:6811', 'nonce': 1759585046}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.92:6810', 'nonce': 1759585046}]}, 'public_addr': '172.21.15.92:6808/1759585046', 'cluster_addr': '172.21.15.92:6809/1759585046', 'heartbeat_back_addr': '172.21.15.92:6811/1759585046', 'heartbeat_front_addr': '172.21.15.92:6810/1759585046', 'state': ['exists', 'up']}, {'osd': 14, 'uuid': 'fc3e2499-0c2a-414d-a25e-2628876ae630', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 22, 'last_clean_end': 120, 'up_from': 142, 'up_thru': 85, 'down_at': 121, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.162:6800', 'nonce': 3999392664}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.162:6801', 'nonce': 3999392664}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.162:6803', 'nonce': 3999392664}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.162:6802', 'nonce': 3999392664}]}, 'public_addr': '172.21.15.162:6800/3999392664', 'cluster_addr': '172.21.15.162:6801/3999392664', 'heartbeat_back_addr': '172.21.15.162:6803/3999392664', 'heartbeat_front_addr': '172.21.15.162:6802/3999392664', 'state': ['exists', 'up']}, {'osd': 15, 'uuid': 'd2f69775-edfe-4527-91ec-f7c7e558bec8', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 22, 'up_thru': 122, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.177:6805', 'nonce': 1294337172}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.177:6806', 'nonce': 1294337172}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.177:6808', 'nonce': 1294337172}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.177:6807', 'nonce': 1294337172}]}, 'public_addr': '172.21.15.177:6805/1294337172', 'cluster_addr': '172.21.15.177:6806/1294337172', 'heartbeat_back_addr': '172.21.15.177:6808/1294337172', 'heartbeat_front_addr': '172.21.15.177:6807/1294337172', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-13T11:24:57.434802+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-13T11:25:47.780833+0000', 'dead_epoch': 30}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-13T11:24:48.897188+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-13T11:24:47.037766+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-13T11:24:52.121278+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-13T11:24:59.304368+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-13T11:24:52.504800+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-13T11:24:44.955554+0000', 'dead_epoch': 0}, {'osd': 8, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-13T11:24:50.514485+0000', 'dead_epoch': 0}, {'osd': 9, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-13T11:25:00.288643+0000', 'dead_epoch': 0}, {'osd': 10, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-13T11:24:38.091398+0000', 'dead_epoch': 0}, {'osd': 11, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-13T11:25:01.213643+0000', 'dead_epoch': 0}, {'osd': 12, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-13T11:25:47.548386+0000', 'dead_epoch': 0}, {'osd': 13, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-13T11:25:45.821466+0000', 'dead_epoch': 0}, {'osd': 14, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-09-13T11:24:55.108407+0000', 'dead_epoch': 121}, {'osd': 15, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-13T11:24:33.420656+0000', 'dead_epoch': 0}], 'pg_upmap': [{'pgid': '2.2', 'osds': [2, 8]}, {'pgid': '3.e', 'osds': [0, 1]}], 'pg_upmap_items': [], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-09-13T11:25:58.642 INFO:tasks.thrashosds.thrasher:No pg_upmap entries; doing nothing 2024-09-13T11:25:58.643 INFO:teuthology.orchestra.run.smithi001.stdout:osd.3: {} 2024-09-13T11:25:58.643 INFO:teuthology.orchestra.run.smithi001.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T11:25:58.643 INFO:teuthology.orchestra.run.smithi001.stdout:osd.4: {} 2024-09-13T11:25:58.644 INFO:teuthology.orchestra.run.smithi001.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T11:25:58.644 INFO:teuthology.orchestra.run.smithi001.stdout:osd.5: {} 2024-09-13T11:25:58.644 INFO:teuthology.orchestra.run.smithi001.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T11:25:58.644 INFO:tasks.ceph.osd.8.smithi001.stderr:2024-09-13T11:25:58.638+0000 7fb9b48ae640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26312) UID: 0 2024-09-13T11:25:58.646 INFO:teuthology.orchestra.run.smithi001.stdout:osd.6: {} 2024-09-13T11:25:58.646 INFO:teuthology.orchestra.run.smithi001.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T11:25:58.741 INFO:tasks.ceph.osd.10.smithi162.stderr:2024-09-13T11:25:58.739+0000 7fb59c9f0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26317) UID: 0 2024-09-13T11:25:58.774 INFO:teuthology.orchestra.run.smithi001.stdout:osd.7: {} 2024-09-13T11:25:58.774 INFO:teuthology.orchestra.run.smithi001.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T11:25:58.778 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-13T11:25:58.778 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:25:58.796 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-09-13T11:25:58.799 INFO:teuthology.orchestra.run.smithi001.stdout:osd.8: {} 2024-09-13T11:25:58.799 INFO:teuthology.orchestra.run.smithi001.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-13T11:25:58.825 INFO:teuthology.orchestra.run.smithi001.stdout:osd.9: {} 2024-09-13T11:25:58.825 INFO:teuthology.orchestra.run.smithi001.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-13T11:25:58.842 INFO:tasks.ceph.osd.12.smithi001.stderr:2024-09-13T11:25:58.838+0000 7f892c6e6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26314) UID: 0 2024-09-13T11:25:58.852 INFO:teuthology.orchestra.run.smithi001.stdout:osd.10: {} 2024-09-13T11:25:58.852 INFO:teuthology.orchestra.run.smithi001.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-13T11:25:58.877 INFO:teuthology.orchestra.run.smithi001.stdout:osd.11: {} 2024-09-13T11:25:58.877 INFO:teuthology.orchestra.run.smithi001.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-13T11:25:58.908 INFO:teuthology.orchestra.run.smithi001.stdout:osd.12: {} 2024-09-13T11:25:58.908 INFO:teuthology.orchestra.run.smithi001.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-13T11:25:58.926 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-13T11:25:58.926 INFO:teuthology.orchestra.run.smithi001.stdout: "size": 20, 2024-09-13T11:25:58.926 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 600, 2024-09-13T11:25:58.926 INFO:teuthology.orchestra.run.smithi001.stdout: "ops": [ 2024-09-13T11:25:58.926 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:58.926 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:54656 3.1e 3.6a49163e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e75)", 2024-09-13T11:25:58.926 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:30.979993+0000", 2024-09-13T11:25:58.926 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 147.94456505599999, 2024-09-13T11:25:58.927 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.9269557150000001, 2024-09-13T11:25:58.927 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:58.927 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:58.927 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-13T11:25:58.927 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:25:58.927 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:25:58.927 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:58.927 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 54656 2024-09-13T11:25:58.927 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:58.927 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:58.928 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:58.928 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:58.928 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:30.979993+0000", 2024-09-13T11:25:58.928 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:58.928 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:58.928 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:58.928 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:58.928 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:30.979993+0000", 2024-09-13T11:25:58.928 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:58.928 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:58.928 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:58.929 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:58.929 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:30.979997+0000", 2024-09-13T11:25:58.929 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.0319999999999997e-06 2024-09-13T11:25:58.929 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:58.929 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:58.929 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:58.929 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:30.980057+0000", 2024-09-13T11:25:58.929 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.0341999999999999e-05 2024-09-13T11:25:58.929 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:58.929 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:58.930 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:58.930 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:30.980094+0000", 2024-09-13T11:25:58.930 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.6770000000000002e-05 2024-09-13T11:25:58.930 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:58.931 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:58.931 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:58.931 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:30.980113+0000", 2024-09-13T11:25:58.931 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.8791999999999999e-05 2024-09-13T11:25:58.931 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:58.931 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:58.932 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:58.932 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.097773+0000", 2024-09-13T11:25:58.932 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.117660379 2024-09-13T11:25:58.932 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:58.932 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:58.932 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for peered", 2024-09-13T11:25:58.932 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.097784+0000", 2024-09-13T11:25:58.932 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.1025e-05 2024-09-13T11:25:58.932 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:58.932 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:58.933 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:58.933 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.906937+0000", 2024-09-13T11:25:58.933 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.8091523789999999 2024-09-13T11:25:58.933 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:58.933 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:58.933 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:58.933 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.906949+0000", 2024-09-13T11:25:58.933 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.1996e-05 2024-09-13T11:25:58.933 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:58.933 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:58.934 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:58.934 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:58.934 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:58.934 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:54720 3.1e 3.a944303e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-13T11:25:58.934 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:31.973315+0000", 2024-09-13T11:25:58.934 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 146.95124282099999, 2024-09-13T11:25:58.934 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.93366527200000005, 2024-09-13T11:25:58.934 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:58.934 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:58.934 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-13T11:25:58.934 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:25:58.935 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:25:58.935 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:58.935 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 54720 2024-09-13T11:25:58.935 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:58.935 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:58.935 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:58.935 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:58.935 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.973315+0000", 2024-09-13T11:25:58.935 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:58.935 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:58.935 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:58.935 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:58.936 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.973315+0000", 2024-09-13T11:25:58.936 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:58.936 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:58.936 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:58.936 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:58.936 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.973318+0000", 2024-09-13T11:25:58.936 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.5830000000000001e-06 2024-09-13T11:25:58.936 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:58.936 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:58.936 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:58.936 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.973465+0000", 2024-09-13T11:25:58.937 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000146816 2024-09-13T11:25:58.937 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:58.937 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:58.937 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:58.937 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.973478+0000", 2024-09-13T11:25:58.937 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2931e-05 2024-09-13T11:25:58.937 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:58.937 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:58.937 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:58.937 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.973483+0000", 2024-09-13T11:25:58.938 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.8969999999999996e-06 2024-09-13T11:25:58.938 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:58.938 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:58.938 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:58.938 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.973548+0000", 2024-09-13T11:25:58.938 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.4771999999999996e-05 2024-09-13T11:25:58.938 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:58.938 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:58.938 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for peered", 2024-09-13T11:25:58.938 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.973557+0000", 2024-09-13T11:25:58.939 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.0359999999999995e-06 2024-09-13T11:25:58.939 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:58.939 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:58.939 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:58.939 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.906974+0000", 2024-09-13T11:25:58.939 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.93341706000000002 2024-09-13T11:25:58.939 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:58.939 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:58.939 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:58.939 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.906980+0000", 2024-09-13T11:25:58.939 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.1770000000000003e-06 2024-09-13T11:25:58.940 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:58.940 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:58.940 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:58.940 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:58.940 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:58.940 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:54734 3.1e 3.d7f6f7e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-13T11:25:58.940 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:31.983342+0000", 2024-09-13T11:25:58.940 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 146.94121601800001, 2024-09-13T11:25:58.940 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.92367058599999996, 2024-09-13T11:25:58.940 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:58.940 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:58.941 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-13T11:25:58.941 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:25:58.941 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:25:58.941 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:58.941 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 54734 2024-09-13T11:25:58.941 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:58.941 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:58.942 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:58.942 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:58.942 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.983342+0000", 2024-09-13T11:25:58.942 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:58.942 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:58.942 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:58.942 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:58.942 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.983342+0000", 2024-09-13T11:25:58.942 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:58.943 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:58.943 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:58.943 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:58.943 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.983345+0000", 2024-09-13T11:25:58.943 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.6220000000000001e-06 2024-09-13T11:25:58.943 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:58.943 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:58.943 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:58.943 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.983438+0000", 2024-09-13T11:25:58.943 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.3517000000000006e-05 2024-09-13T11:25:58.944 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:58.944 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:58.944 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:58.944 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.983451+0000", 2024-09-13T11:25:58.944 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2527999999999999e-05 2024-09-13T11:25:58.944 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:58.944 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:58.944 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:58.944 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.983458+0000", 2024-09-13T11:25:58.944 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.6380000000000004e-06 2024-09-13T11:25:58.944 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:58.945 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:58.945 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:58.945 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.983521+0000", 2024-09-13T11:25:58.945 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.2729000000000002e-05 2024-09-13T11:25:58.945 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:58.945 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:58.945 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for peered", 2024-09-13T11:25:58.945 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.983548+0000", 2024-09-13T11:25:58.945 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.6537999999999999e-05 2024-09-13T11:25:58.945 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:58.945 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:58.946 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:58.946 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.907005+0000", 2024-09-13T11:25:58.946 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.923456991 2024-09-13T11:25:58.946 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:58.946 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:58.946 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:58.946 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.907013+0000", 2024-09-13T11:25:58.946 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.0229999999999992e-06 2024-09-13T11:25:58.946 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:58.946 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:58.947 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:58.947 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:58.947 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:58.947 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:54760 3.1e 3.8c32c13e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-13T11:25:58.947 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:31.999685+0000", 2024-09-13T11:25:58.947 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 146.92487278600001, 2024-09-13T11:25:58.947 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.90735639599999995, 2024-09-13T11:25:58.947 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:58.947 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:58.947 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-13T11:25:58.947 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:25:58.948 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:25:58.948 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:58.948 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 54760 2024-09-13T11:25:58.948 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:58.948 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:58.948 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:58.948 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:58.949 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.999685+0000", 2024-09-13T11:25:58.949 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:58.949 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:58.949 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:58.949 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:58.949 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.999685+0000", 2024-09-13T11:25:58.949 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:58.949 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:58.949 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:58.949 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:58.950 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.999690+0000", 2024-09-13T11:25:58.950 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.7879999999999997e-06 2024-09-13T11:25:58.950 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:58.950 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:58.950 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:58.950 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.999729+0000", 2024-09-13T11:25:58.950 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.9032000000000002e-05 2024-09-13T11:25:58.950 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:58.950 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:58.950 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:58.950 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.999741+0000", 2024-09-13T11:25:58.951 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2136999999999999e-05 2024-09-13T11:25:58.951 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:58.951 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:58.951 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:58.951 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.999748+0000", 2024-09-13T11:25:58.951 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.9120000000000001e-06 2024-09-13T11:25:58.951 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:58.951 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:58.951 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:58.951 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.999803+0000", 2024-09-13T11:25:58.951 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.5064e-05 2024-09-13T11:25:58.952 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:58.952 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:58.952 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for peered", 2024-09-13T11:25:58.952 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.999815+0000", 2024-09-13T11:25:58.952 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2041e-05 2024-09-13T11:25:58.952 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:58.952 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:58.952 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:58.952 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.907034+0000", 2024-09-13T11:25:58.952 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.90721858200000005 2024-09-13T11:25:58.953 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:58.953 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:58.953 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:58.953 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.907042+0000", 2024-09-13T11:25:58.953 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.8399999999999995e-06 2024-09-13T11:25:58.953 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:58.953 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:58.953 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:58.953 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:58.953 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:58.954 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:54768 3.1e 3.6dad499e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-13T11:25:58.954 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:32.003482+0000", 2024-09-13T11:25:58.954 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 146.92107613600001, 2024-09-13T11:25:58.954 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.90358804699999995, 2024-09-13T11:25:58.954 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:58.954 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:58.954 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-13T11:25:58.954 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:25:58.954 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:25:58.954 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:58.955 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 54768 2024-09-13T11:25:58.955 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:58.955 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:58.955 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:58.955 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:58.955 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.003482+0000", 2024-09-13T11:25:58.955 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:58.955 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:58.955 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:58.955 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:58.956 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.003482+0000", 2024-09-13T11:25:58.956 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:58.956 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:58.956 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:58.956 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:58.956 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.003486+0000", 2024-09-13T11:25:58.956 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.1030000000000004e-06 2024-09-13T11:25:58.956 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:58.956 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:58.956 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:58.957 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.003523+0000", 2024-09-13T11:25:58.957 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.6732e-05 2024-09-13T11:25:58.957 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:58.957 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:58.957 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:58.957 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.003535+0000", 2024-09-13T11:25:58.957 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2520999999999999e-05 2024-09-13T11:25:58.957 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:58.957 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:58.957 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:58.958 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.003541+0000", 2024-09-13T11:25:58.958 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.8690000000000001e-06 2024-09-13T11:25:58.958 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:58.958 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:58.958 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:58.958 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.003593+0000", 2024-09-13T11:25:58.958 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.1786000000000002e-05 2024-09-13T11:25:58.958 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:58.958 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:58.958 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for peered", 2024-09-13T11:25:58.958 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.003602+0000", 2024-09-13T11:25:58.959 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.9679999999999995e-06 2024-09-13T11:25:58.959 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:58.959 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:58.959 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:58.959 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.907063+0000", 2024-09-13T11:25:58.959 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.90346116600000004 2024-09-13T11:25:58.959 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:58.959 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:58.959 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:58.959 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.907070+0000", 2024-09-13T11:25:58.960 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.9020000000000002e-06 2024-09-13T11:25:58.960 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:58.960 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:58.960 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:58.960 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:58.960 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:58.960 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:54785 3.1e 3.d32f1ede (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-13T11:25:58.960 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:32.016238+0000", 2024-09-13T11:25:58.960 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 146.90831992599999, 2024-09-13T11:25:58.960 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.89086243399999998, 2024-09-13T11:25:58.961 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:58.961 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:58.961 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-13T11:25:58.961 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:25:58.961 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:25:58.961 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:58.961 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 54785 2024-09-13T11:25:58.961 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:58.961 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:58.961 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:58.961 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:58.962 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.016238+0000", 2024-09-13T11:25:58.962 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:58.962 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:58.962 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:58.962 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:58.962 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.016238+0000", 2024-09-13T11:25:58.962 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:58.962 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:58.962 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:58.962 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:58.962 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.016241+0000", 2024-09-13T11:25:58.963 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.6469999999999998e-06 2024-09-13T11:25:58.963 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:58.963 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:58.963 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:58.963 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.016271+0000", 2024-09-13T11:25:58.963 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.0505999999999999e-05 2024-09-13T11:25:58.963 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:58.963 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:58.963 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:58.963 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.016287+0000", 2024-09-13T11:25:58.964 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.6073999999999999e-05 2024-09-13T11:25:58.964 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:58.964 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:58.964 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:58.964 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.016295+0000", 2024-09-13T11:25:58.964 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.0590000000000003e-06 2024-09-13T11:25:58.964 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:58.965 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:58.965 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:58.965 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.016373+0000", 2024-09-13T11:25:58.965 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.7634999999999998e-05 2024-09-13T11:25:58.965 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:58.965 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:58.965 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for peered", 2024-09-13T11:25:58.965 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.016383+0000", 2024-09-13T11:25:58.965 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0246e-05 2024-09-13T11:25:58.965 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:58.965 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:58.966 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:58.966 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.907092+0000", 2024-09-13T11:25:58.966 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.89070849200000002 2024-09-13T11:25:58.966 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:58.966 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:58.966 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:58.966 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.907101+0000", 2024-09-13T11:25:58.966 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.7749999999999998e-06 2024-09-13T11:25:58.966 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:58.966 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:58.967 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:58.967 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:58.967 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:58.967 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:54811 3.1e 3.3754367e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-13T11:25:58.967 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:32.045424+0000", 2024-09-13T11:25:58.967 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 146.87913427500001, 2024-09-13T11:25:58.967 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.86170719399999995, 2024-09-13T11:25:58.967 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:58.967 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:58.967 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-13T11:25:58.968 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:25:58.968 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:25:58.968 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:58.968 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 54811 2024-09-13T11:25:58.968 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:58.968 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:58.968 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:58.968 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:58.968 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.045424+0000", 2024-09-13T11:25:58.968 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:58.969 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:58.969 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:58.969 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:58.969 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.045424+0000", 2024-09-13T11:25:58.969 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:58.969 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:58.969 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:58.969 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:58.969 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.045426+0000", 2024-09-13T11:25:58.970 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.0260000000000002e-06 2024-09-13T11:25:58.970 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:58.970 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:58.970 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:58.970 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.045556+0000", 2024-09-13T11:25:58.970 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00013003099999999999 2024-09-13T11:25:58.970 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:58.970 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:58.970 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:58.970 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.045568+0000", 2024-09-13T11:25:58.971 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.1727e-05 2024-09-13T11:25:58.971 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:58.971 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:58.971 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:58.971 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.045576+0000", 2024-09-13T11:25:58.971 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.6459999999999999e-06 2024-09-13T11:25:58.971 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:58.971 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:58.971 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:58.971 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.045650+0000", 2024-09-13T11:25:58.972 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.4000999999999998e-05 2024-09-13T11:25:58.972 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:58.972 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:58.972 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for peered", 2024-09-13T11:25:58.972 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.045658+0000", 2024-09-13T11:25:58.972 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.3869999999999997e-06 2024-09-13T11:25:58.972 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:58.972 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:58.972 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:58.972 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.907123+0000", 2024-09-13T11:25:58.973 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.86146508399999999 2024-09-13T11:25:58.973 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:58.973 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:58.973 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:58.973 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.907131+0000", 2024-09-13T11:25:58.973 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.2919999999999996e-06 2024-09-13T11:25:58.973 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:58.973 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:58.973 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:58.973 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:58.974 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:58.974 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:54817 3.1e 3.36dd2fde (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-13T11:25:58.974 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:32.050108+0000", 2024-09-13T11:25:58.974 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 146.87445038300001, 2024-09-13T11:25:58.974 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.85705549199999997, 2024-09-13T11:25:58.974 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:58.974 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:58.974 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-13T11:25:58.974 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:25:58.974 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:25:58.974 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:58.975 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 54817 2024-09-13T11:25:58.975 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:58.975 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:58.975 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:58.975 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:58.975 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.050108+0000", 2024-09-13T11:25:58.975 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:58.975 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:58.975 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:58.975 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:58.976 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.050108+0000", 2024-09-13T11:25:58.976 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:58.976 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:58.976 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:58.976 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:58.976 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.050110+0000", 2024-09-13T11:25:58.976 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.0209999999999998e-06 2024-09-13T11:25:58.976 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:58.976 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:58.976 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:58.976 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.050166+0000", 2024-09-13T11:25:58.977 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.6026000000000002e-05 2024-09-13T11:25:58.977 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:58.977 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:58.977 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:58.977 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.050180+0000", 2024-09-13T11:25:58.977 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.4307e-05 2024-09-13T11:25:58.977 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:58.977 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:58.977 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:58.978 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.050188+0000", 2024-09-13T11:25:58.978 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.5410000000000003e-06 2024-09-13T11:25:58.978 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:58.978 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:58.978 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:58.978 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.050260+0000", 2024-09-13T11:25:58.978 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.2157999999999996e-05 2024-09-13T11:25:58.978 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:58.978 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:58.978 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for peered", 2024-09-13T11:25:58.979 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.050267+0000", 2024-09-13T11:25:58.979 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.0190000000000001e-06 2024-09-13T11:25:58.979 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:58.979 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:58.979 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:58.979 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.907155+0000", 2024-09-13T11:25:58.979 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.85688824399999997 2024-09-13T11:25:58.979 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:58.979 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:58.979 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:58.979 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.907163+0000", 2024-09-13T11:25:58.979 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.1759999999999993e-06 2024-09-13T11:25:58.980 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:58.980 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:58.980 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:58.980 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:58.980 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:58.980 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:54818 3.1e 3.5cfcc69e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-13T11:25:58.980 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:32.051374+0000", 2024-09-13T11:25:58.980 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 146.87318396699999, 2024-09-13T11:25:58.980 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.85581781000000001, 2024-09-13T11:25:58.980 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:58.981 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:58.981 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-13T11:25:58.981 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:25:58.981 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:25:58.981 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:58.981 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 54818 2024-09-13T11:25:58.981 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:58.981 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:58.981 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:58.981 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:58.982 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.051374+0000", 2024-09-13T11:25:58.982 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:58.982 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:58.982 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:58.982 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:58.982 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.051374+0000", 2024-09-13T11:25:58.982 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:58.982 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:58.982 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:58.982 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:58.982 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.051375+0000", 2024-09-13T11:25:58.983 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.4440000000000001e-06 2024-09-13T11:25:58.983 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:58.983 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:58.983 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:58.983 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.051426+0000", 2024-09-13T11:25:58.983 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.0809000000000001e-05 2024-09-13T11:25:58.983 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:58.983 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:58.983 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:58.983 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.051441+0000", 2024-09-13T11:25:58.983 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.4460999999999999e-05 2024-09-13T11:25:58.984 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:58.984 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:58.984 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:58.984 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.051449+0000", 2024-09-13T11:25:58.984 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.4880000000000007e-06 2024-09-13T11:25:58.984 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:58.984 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:58.984 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:58.984 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.051499+0000", 2024-09-13T11:25:58.984 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.9691999999999998e-05 2024-09-13T11:25:58.984 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:58.985 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:58.985 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for peered", 2024-09-13T11:25:58.985 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.051505+0000", 2024-09-13T11:25:58.985 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.2029999999999997e-06 2024-09-13T11:25:58.985 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:58.985 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:58.985 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:58.985 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.907184+0000", 2024-09-13T11:25:58.985 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.85567906599999999 2024-09-13T11:25:58.985 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:58.986 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:58.986 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:58.986 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.907192+0000", 2024-09-13T11:25:58.986 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.6469999999999998e-06 2024-09-13T11:25:58.986 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:58.986 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:58.986 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:58.986 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:58.986 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:58.986 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:54828 3.1e 3.e920c11e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-13T11:25:58.986 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:32.063344+0000", 2024-09-13T11:25:58.987 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 146.86121353999999, 2024-09-13T11:25:58.987 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.84387899099999997, 2024-09-13T11:25:58.987 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:58.987 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:58.987 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-13T11:25:58.987 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:25:58.987 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:25:58.987 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:58.987 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 54828 2024-09-13T11:25:58.987 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:58.988 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:58.988 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:58.988 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:58.988 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.063344+0000", 2024-09-13T11:25:58.988 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:58.988 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:58.988 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:58.988 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:58.988 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.063344+0000", 2024-09-13T11:25:58.988 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:58.988 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:58.989 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:58.989 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:58.989 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.063347+0000", 2024-09-13T11:25:58.989 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.2299999999999998e-06 2024-09-13T11:25:58.989 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:58.989 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:58.989 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:58.989 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.063384+0000", 2024-09-13T11:25:58.989 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.7290999999999999e-05 2024-09-13T11:25:58.989 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:58.989 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:58.990 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:58.990 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.063397+0000", 2024-09-13T11:25:58.990 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2955999999999999e-05 2024-09-13T11:25:58.990 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:58.990 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:58.990 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:58.990 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.063404+0000", 2024-09-13T11:25:58.990 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.4889999999999998e-06 2024-09-13T11:25:58.990 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:58.990 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:58.991 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:58.991 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.063470+0000", 2024-09-13T11:25:58.991 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.5516000000000001e-05 2024-09-13T11:25:58.991 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:58.991 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:58.991 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for peered", 2024-09-13T11:25:58.991 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.063482+0000", 2024-09-13T11:25:58.991 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2065000000000001e-05 2024-09-13T11:25:58.991 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:58.992 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:58.992 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:58.992 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.907216+0000", 2024-09-13T11:25:58.992 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.84373388900000001 2024-09-13T11:25:58.992 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:58.992 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:58.992 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:58.992 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.907223+0000", 2024-09-13T11:25:58.992 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.5549999999999997e-06 2024-09-13T11:25:58.992 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:58.993 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:58.993 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:58.993 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:58.993 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:58.993 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:54829 3.1e 3.192f62be (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-13T11:25:58.993 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:32.065642+0000", 2024-09-13T11:25:58.993 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 146.858916472, 2024-09-13T11:25:58.993 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.84161483100000001, 2024-09-13T11:25:58.993 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:58.993 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:58.993 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-13T11:25:58.993 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:25:58.994 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:25:58.994 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:58.994 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 54829 2024-09-13T11:25:58.994 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:58.994 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:58.994 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:58.994 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:58.994 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.065642+0000", 2024-09-13T11:25:58.994 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:58.994 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:58.994 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:58.995 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:58.995 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.065642+0000", 2024-09-13T11:25:58.995 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:58.995 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:58.995 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:58.995 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:58.995 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.065644+0000", 2024-09-13T11:25:58.995 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.1780000000000002e-06 2024-09-13T11:25:58.995 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:58.995 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:58.996 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:58.996 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.065695+0000", 2024-09-13T11:25:58.996 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.1632000000000003e-05 2024-09-13T11:25:58.996 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:58.996 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:58.996 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:58.996 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.065711+0000", 2024-09-13T11:25:58.996 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.5522000000000001e-05 2024-09-13T11:25:58.996 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:58.996 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:58.996 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:58.997 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.065719+0000", 2024-09-13T11:25:58.997 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.3480000000000004e-06 2024-09-13T11:25:58.997 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:58.997 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:58.997 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:58.997 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.065785+0000", 2024-09-13T11:25:58.997 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.5627e-05 2024-09-13T11:25:58.997 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:58.997 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:58.998 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for peered", 2024-09-13T11:25:58.998 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.065797+0000", 2024-09-13T11:25:58.998 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.1766000000000001e-05 2024-09-13T11:25:58.998 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:58.998 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:58.998 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:58.998 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.907248+0000", 2024-09-13T11:25:58.998 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.84145150300000005 2024-09-13T11:25:58.998 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:58.998 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:58.998 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:58.999 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.907256+0000", 2024-09-13T11:25:58.999 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.2549999999999997e-06 2024-09-13T11:25:58.999 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:58.999 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:58.999 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:58.999 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:58.999 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:58.999 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:54830 3.1e 3.b31554de (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-13T11:25:58.999 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:32.068259+0000", 2024-09-13T11:25:58.999 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 146.85629860399999, 2024-09-13T11:25:58.999 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.83902489599999996, 2024-09-13T11:25:59.000 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:59.000 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:59.000 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-13T11:25:59.000 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:25:59.000 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:25:59.000 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:59.000 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 54830 2024-09-13T11:25:59.000 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.000 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:59.000 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.001 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:59.001 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.068259+0000", 2024-09-13T11:25:59.001 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:59.001 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.001 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.001 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:59.001 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.068259+0000", 2024-09-13T11:25:59.001 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:59.001 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.001 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.002 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:59.002 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.068262+0000", 2024-09-13T11:25:59.002 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.1799999999999999e-06 2024-09-13T11:25:59.002 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.002 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.002 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:59.002 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.068301+0000", 2024-09-13T11:25:59.002 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.9289999999999998e-05 2024-09-13T11:25:59.002 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.002 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.002 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:59.003 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.068314+0000", 2024-09-13T11:25:59.003 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.3502999999999999e-05 2024-09-13T11:25:59.003 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.003 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.003 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:59.003 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.068324+0000", 2024-09-13T11:25:59.003 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.1630000000000002e-06 2024-09-13T11:25:59.003 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.003 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.003 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:59.004 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.068375+0000", 2024-09-13T11:25:59.004 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.1805000000000003e-05 2024-09-13T11:25:59.004 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.004 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.004 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for peered", 2024-09-13T11:25:59.004 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.068382+0000", 2024-09-13T11:25:59.004 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.224e-06 2024-09-13T11:25:59.004 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.004 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.005 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:59.005 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.907277+0000", 2024-09-13T11:25:59.005 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.83889509500000004 2024-09-13T11:25:59.005 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.005 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.005 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:59.005 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.907284+0000", 2024-09-13T11:25:59.005 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.6359999999999994e-06 2024-09-13T11:25:59.005 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:59.005 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:59.005 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:59.006 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.006 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.006 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:54846 3.1e 3.742e049e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-13T11:25:59.006 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:32.153567+0000", 2024-09-13T11:25:59.006 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 146.77099138, 2024-09-13T11:25:59.006 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.75378642399999995, 2024-09-13T11:25:59.006 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:59.006 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:59.006 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-13T11:25:59.006 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:25:59.007 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:25:59.007 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:59.007 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 54846 2024-09-13T11:25:59.007 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.007 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:59.007 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.007 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:59.007 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.153567+0000", 2024-09-13T11:25:59.007 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:59.007 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.008 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.008 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:59.008 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.153567+0000", 2024-09-13T11:25:59.008 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:59.008 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.008 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.008 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:59.008 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.153569+0000", 2024-09-13T11:25:59.008 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.4660000000000002e-06 2024-09-13T11:25:59.008 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.008 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.009 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:59.009 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.153619+0000", 2024-09-13T11:25:59.009 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.9617000000000003e-05 2024-09-13T11:25:59.009 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.009 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.009 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:59.009 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.153634+0000", 2024-09-13T11:25:59.009 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.4904e-05 2024-09-13T11:25:59.009 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.009 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.010 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:59.010 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.153642+0000", 2024-09-13T11:25:59.010 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.5599999999999994e-06 2024-09-13T11:25:59.010 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.010 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.010 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:59.010 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.153721+0000", 2024-09-13T11:25:59.010 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.8603999999999995e-05 2024-09-13T11:25:59.010 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.010 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.010 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for peered", 2024-09-13T11:25:59.010 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.153730+0000", 2024-09-13T11:25:59.011 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.3840000000000003e-06 2024-09-13T11:25:59.011 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.011 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.011 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:59.011 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.907346+0000", 2024-09-13T11:25:59.011 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.75361563099999995 2024-09-13T11:25:59.011 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.011 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.011 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:59.011 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.907353+0000", 2024-09-13T11:25:59.011 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.2579999999999998e-06 2024-09-13T11:25:59.012 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:59.012 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:59.012 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:59.012 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.012 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.012 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:54888 3.1e 3.43e14ede (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-13T11:25:59.012 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:32.423064+0000", 2024-09-13T11:25:59.012 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 146.50149370899999, 2024-09-13T11:25:59.012 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.48433585299999998, 2024-09-13T11:25:59.013 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:59.013 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:59.013 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-13T11:25:59.013 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:25:59.013 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:25:59.013 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:59.013 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 54888 2024-09-13T11:25:59.013 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.013 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:59.013 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.013 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:59.014 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.423064+0000", 2024-09-13T11:25:59.014 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:59.014 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.014 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.014 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:59.014 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.423064+0000", 2024-09-13T11:25:59.014 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:59.014 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.014 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.014 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:59.014 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.423069+0000", 2024-09-13T11:25:59.015 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.7389999999999999e-06 2024-09-13T11:25:59.015 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.015 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.015 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:59.015 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.423115+0000", 2024-09-13T11:25:59.015 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.5847000000000003e-05 2024-09-13T11:25:59.015 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.015 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.015 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:59.015 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.423126+0000", 2024-09-13T11:25:59.015 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.1418e-05 2024-09-13T11:25:59.016 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.016 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.016 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:59.016 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.423132+0000", 2024-09-13T11:25:59.016 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.57e-06 2024-09-13T11:25:59.016 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.016 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.016 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:59.016 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.423191+0000", 2024-09-13T11:25:59.016 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.9250999999999999e-05 2024-09-13T11:25:59.017 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.017 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.017 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for peered", 2024-09-13T11:25:59.017 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.423198+0000", 2024-09-13T11:25:59.017 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.9179999999999997e-06 2024-09-13T11:25:59.017 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.017 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.017 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:59.017 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.907392+0000", 2024-09-13T11:25:59.017 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.48419437500000001 2024-09-13T11:25:59.017 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.018 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.018 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:59.018 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.907400+0000", 2024-09-13T11:25:59.018 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.7349999999999996e-06 2024-09-13T11:25:59.018 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:59.018 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:59.018 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:59.018 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.018 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.018 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4665.0:59977 3.21 e83/78 3:86b4d8f6:::benchmark_data_smithi177_31116_object59976:head v 83'3136, mlcod=0'0)", 2024-09-13T11:25:59.019 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:40.571147+0000", 2024-09-13T11:25:59.019 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 138.35341126, 2024-09-13T11:25:59.019 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.51907542799999995, 2024-09-13T11:25:59.019 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:59.019 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:59.019 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:59.019 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:59.019 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.019 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:59.019 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:40.571147+0000", 2024-09-13T11:25:59.020 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:59.020 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.020 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.020 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:59.020 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:40.571147+0000", 2024-09-13T11:25:59.020 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:59.020 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.020 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.020 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:59.020 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:40.571151+0000", 2024-09-13T11:25:59.021 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.5650000000000003e-06 2024-09-13T11:25:59.021 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.021 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.021 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:59.021 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:40.571204+0000", 2024-09-13T11:25:59.021 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.2720999999999999e-05 2024-09-13T11:25:59.021 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.021 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.021 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:59.022 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:41.088332+0000", 2024-09-13T11:25:59.022 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.51712822000000003 2024-09-13T11:25:59.022 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.022 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.022 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:59.022 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:41.088341+0000", 2024-09-13T11:25:59.022 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.3100000000000001e-06 2024-09-13T11:25:59.022 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.022 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.023 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:59.023 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:41.088444+0000", 2024-09-13T11:25:59.023 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000103708 2024-09-13T11:25:59.023 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.023 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.023 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:25:59.023 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:41.088475+0000", 2024-09-13T11:25:59.023 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.0497e-05 2024-09-13T11:25:59.023 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.024 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.024 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:25:59.024 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:41.090192+0000", 2024-09-13T11:25:59.024 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.001717288 2024-09-13T11:25:59.024 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.024 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.024 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:59.025 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:41.090222+0000", 2024-09-13T11:25:59.025 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.0119000000000001e-05 2024-09-13T11:25:59.025 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:59.025 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:59.025 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:59.025 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.025 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.025 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:110286 3.26 3.c41c0fa6 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e119)", 2024-09-13T11:25:59.025 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:24:25.654272+0000", 2024-09-13T11:25:59.025 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 93.270285568000006, 2024-09-13T11:25:59.026 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.43337235400000002, 2024-09-13T11:25:59.026 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:59.026 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:59.026 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-13T11:25:59.026 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:25:59.026 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:25:59.026 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:59.026 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 110286 2024-09-13T11:25:59.026 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.026 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:59.027 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.027 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:59.027 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:25.654272+0000", 2024-09-13T11:25:59.027 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:59.027 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.027 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.027 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:59.027 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:25.654272+0000", 2024-09-13T11:25:59.027 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:59.027 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.028 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.028 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:59.028 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:25.654275+0000", 2024-09-13T11:25:59.028 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.3199999999999998e-06 2024-09-13T11:25:59.028 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.028 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.028 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:59.028 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:25.654517+0000", 2024-09-13T11:25:59.028 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00024270199999999999 2024-09-13T11:25:59.028 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.028 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.029 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:59.029 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:25.654550+0000", 2024-09-13T11:25:59.029 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.2549000000000003e-05 2024-09-13T11:25:59.029 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.029 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.029 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:59.029 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:25.654566+0000", 2024-09-13T11:25:59.029 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.5517000000000001e-05 2024-09-13T11:25:59.029 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.029 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.030 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:59.030 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:25.654655+0000", 2024-09-13T11:25:59.030 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.9961000000000001e-05 2024-09-13T11:25:59.030 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.030 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.030 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for peered", 2024-09-13T11:25:59.030 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:25.654665+0000", 2024-09-13T11:25:59.030 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.0520000000000007e-06 2024-09-13T11:25:59.030 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.030 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.031 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:59.031 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:26.087629+0000", 2024-09-13T11:25:59.031 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.43296476900000003 2024-09-13T11:25:59.031 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.031 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.031 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:59.031 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:26.087645+0000", 2024-09-13T11:25:59.031 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.5483999999999999e-05 2024-09-13T11:25:59.031 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:59.031 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:59.032 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:59.032 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.032 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.032 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:177913 3.27 3:e5268e7a:::benchmark_data_smithi177_31116_object37610:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e168)", 2024-09-13T11:25:59.032 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:25:27.607015+0000", 2024-09-13T11:25:59.032 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 31.317543218000001, 2024-09-13T11:25:59.032 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.301660555, 2024-09-13T11:25:59.032 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:59.032 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:59.032 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:59.033 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:25:59.033 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:25:59.033 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:59.033 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 177913 2024-09-13T11:25:59.033 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.033 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:59.033 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.033 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:59.033 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:27.607015+0000", 2024-09-13T11:25:59.033 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:59.034 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.034 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.034 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:59.034 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:27.607015+0000", 2024-09-13T11:25:59.034 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:59.034 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.034 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.034 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:59.034 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:27.607018+0000", 2024-09-13T11:25:59.034 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.7000000000000002e-06 2024-09-13T11:25:59.034 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.035 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.035 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:59.035 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:27.607029+0000", 2024-09-13T11:25:59.035 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0933e-05 2024-09-13T11:25:59.035 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.035 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.035 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:59.035 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:27.607062+0000", 2024-09-13T11:25:59.035 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.2786000000000001e-05 2024-09-13T11:25:59.035 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.035 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.036 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:59.036 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:27.607078+0000", 2024-09-13T11:25:59.036 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.6232999999999999e-05 2024-09-13T11:25:59.036 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.036 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.036 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:59.036 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:27.607204+0000", 2024-09-13T11:25:59.036 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000125458 2024-09-13T11:25:59.036 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.036 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.037 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:25:59.037 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:27.607544+0000", 2024-09-13T11:25:59.037 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00033990600000000001 2024-09-13T11:25:59.037 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.037 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.037 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 5,11", 2024-09-13T11:25:59.037 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:27.607882+0000", 2024-09-13T11:25:59.037 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00033788399999999999 2024-09-13T11:25:59.037 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.037 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.037 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:25:59.038 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:27.610120+0000", 2024-09-13T11:25:59.038 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0022379930000000002 2024-09-13T11:25:59.038 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.038 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.038 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:25:59.038 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:27.613170+0000", 2024-09-13T11:25:59.038 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.003050545 2024-09-13T11:25:59.038 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.038 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.038 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:25:59.039 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:27.908398+0000", 2024-09-13T11:25:59.039 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.29522760599999998 2024-09-13T11:25:59.039 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.039 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.039 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:25:59.039 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:27.908458+0000", 2024-09-13T11:25:59.039 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.9718e-05 2024-09-13T11:25:59.039 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.039 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.039 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:59.039 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:27.908675+0000", 2024-09-13T11:25:59.039 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00021779300000000001 2024-09-13T11:25:59.040 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:59.040 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:59.040 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:59.040 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.040 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.040 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop_reply(client.4665.0:177913 3.27 e168/120 ondisk, result = 0)", 2024-09-13T11:25:59.040 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:25:27.610053+0000", 2024-09-13T11:25:59.040 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 31.314505058999998, 2024-09-13T11:25:59.040 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.29862817200000003, 2024-09-13T11:25:59.040 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:59.040 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:59.041 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:25:59.041 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:59.041 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.041 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:59.041 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:27.610053+0000", 2024-09-13T11:25:59.041 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:59.041 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.041 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.041 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:59.041 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:27.610053+0000", 2024-09-13T11:25:59.041 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:59.042 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.042 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.042 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:59.042 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:27.610056+0000", 2024-09-13T11:25:59.042 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.49e-06 2024-09-13T11:25:59.042 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.042 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.042 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:59.042 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:27.610067+0000", 2024-09-13T11:25:59.042 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0599e-05 2024-09-13T11:25:59.043 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.043 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.043 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:59.043 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:27.908314+0000", 2024-09-13T11:25:59.043 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.29824728700000003 2024-09-13T11:25:59.043 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.043 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.043 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:59.043 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:27.908320+0000", 2024-09-13T11:25:59.043 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.9100000000000002e-06 2024-09-13T11:25:59.043 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.043 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.044 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:59.044 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:27.908378+0000", 2024-09-13T11:25:59.044 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.7575999999999999e-05 2024-09-13T11:25:59.044 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.044 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.044 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:25:59.044 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:27.908390+0000", 2024-09-13T11:25:59.044 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.187e-05 2024-09-13T11:25:59.044 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.044 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.044 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:59.045 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:27.908681+0000", 2024-09-13T11:25:59.045 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00029144000000000002 2024-09-13T11:25:59.045 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:59.045 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:59.045 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:59.045 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.045 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.045 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDPGPushReply(3.27 168/120 [PushReplyOp(3:e535cc18:::benchmark_data_smithi177_31116_object103295:head)])", 2024-09-13T11:25:59.045 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:25:27.615014+0000", 2024-09-13T11:25:59.045 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 31.309543725000001, 2024-09-13T11:25:59.045 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.31810488799999997, 2024-09-13T11:25:59.045 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:59.046 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:59.046 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-13T11:25:59.046 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:59.046 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.046 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:59.046 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:27.615014+0000", 2024-09-13T11:25:59.046 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:59.046 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.046 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.046 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:59.046 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:27.615014+0000", 2024-09-13T11:25:59.047 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:59.047 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.047 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.047 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:59.047 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:27.615017+0000", 2024-09-13T11:25:59.047 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.6199999999999999e-06 2024-09-13T11:25:59.047 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.047 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.047 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:59.047 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:27.615030+0000", 2024-09-13T11:25:59.047 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2833000000000001e-05 2024-09-13T11:25:59.047 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.048 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.048 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:59.048 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:27.932588+0000", 2024-09-13T11:25:59.048 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.31755857700000001 2024-09-13T11:25:59.048 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.048 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.048 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:59.049 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:27.932595+0000", 2024-09-13T11:25:59.049 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.2160000000000003e-06 2024-09-13T11:25:59.049 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.049 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.049 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:59.049 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:27.932658+0000", 2024-09-13T11:25:59.049 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.3618e-05 2024-09-13T11:25:59.049 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.049 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.049 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:59.049 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:27.933119+0000", 2024-09-13T11:25:59.049 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00046102399999999998 2024-09-13T11:25:59.050 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:59.050 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:59.050 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:59.050 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.050 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.050 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:191082 3.26 3:67bd986b:::benchmark_data_smithi177_31116_object50779:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e175)", 2024-09-13T11:25:59.050 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:25:38.550314+0000", 2024-09-13T11:25:59.050 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 20.374244097999998, 2024-09-13T11:25:59.050 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.29180858100000001, 2024-09-13T11:25:59.050 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:59.050 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:59.051 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:59.051 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:25:59.051 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:25:59.051 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:59.051 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 191082 2024-09-13T11:25:59.051 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.051 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:59.051 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.051 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:59.051 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:38.550314+0000", 2024-09-13T11:25:59.051 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:59.052 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.052 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.052 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:59.052 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:38.550314+0000", 2024-09-13T11:25:59.052 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:59.052 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.052 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.052 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:59.052 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:38.550317+0000", 2024-09-13T11:25:59.052 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.14e-06 2024-09-13T11:25:59.052 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.053 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.053 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:59.053 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:38.550327+0000", 2024-09-13T11:25:59.053 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0386e-05 2024-09-13T11:25:59.053 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.053 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.053 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:59.053 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:38.550353+0000", 2024-09-13T11:25:59.053 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.5431000000000001e-05 2024-09-13T11:25:59.053 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.053 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.053 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:59.053 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:38.550367+0000", 2024-09-13T11:25:59.054 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.3859999999999999e-05 2024-09-13T11:25:59.054 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.054 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.054 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:59.054 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:38.550476+0000", 2024-09-13T11:25:59.054 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000108782 2024-09-13T11:25:59.054 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.054 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.054 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:25:59.054 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:38.550677+0000", 2024-09-13T11:25:59.054 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00020138099999999999 2024-09-13T11:25:59.055 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.055 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.055 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 12", 2024-09-13T11:25:59.055 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:38.550836+0000", 2024-09-13T11:25:59.055 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000158722 2024-09-13T11:25:59.055 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.055 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.055 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:25:59.055 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:38.553897+0000", 2024-09-13T11:25:59.055 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.003061533 2024-09-13T11:25:59.055 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.055 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.056 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:25:59.056 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:38.841929+0000", 2024-09-13T11:25:59.056 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.28803159 2024-09-13T11:25:59.056 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.056 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.056 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:25:59.056 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:38.842014+0000", 2024-09-13T11:25:59.056 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.5119999999999998e-05 2024-09-13T11:25:59.056 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.056 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.056 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:59.057 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:38.842122+0000", 2024-09-13T11:25:59.057 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00010863599999999999 2024-09-13T11:25:59.057 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:59.057 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:59.057 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:59.057 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:59.057 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:59.057 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-13T11:25:59.058 INFO:teuthology.orchestra.run.smithi001.stdout:osd.13: {} 2024-09-13T11:25:59.059 INFO:tasks.ceph.osd.5.smithi092.stderr:2024-09-13T11:25:59.056+0000 7f6f0b705640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26346) UID: 0 2024-09-13T11:25:59.059 INFO:teuthology.orchestra.run.smithi001.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-13T11:25:59.060 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_ops_in_flight 2024-09-13T11:25:59.060 INFO:teuthology.orchestra.run.smithi001.stdout:osd.14: {} 2024-09-13T11:25:59.061 INFO:teuthology.orchestra.run.smithi001.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-13T11:25:59.061 INFO:teuthology.orchestra.run.smithi001.stderr:nodeep-scrub is unset 2024-09-13T11:25:59.061 INFO:teuthology.orchestra.run.smithi001.stdout:osd.15: {} 2024-09-13T11:25:59.062 INFO:teuthology.orchestra.run.smithi001.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-13T11:25:59.159 INFO:tasks.ceph.osd.1.smithi092.stderr:2024-09-13T11:25:59.156+0000 7fe7789c5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33718) UID: 0 2024-09-13T11:25:59.247 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-13T11:25:59.247 INFO:teuthology.orchestra.run.smithi001.stdout: "ops": [], 2024-09-13T11:25:59.247 INFO:teuthology.orchestra.run.smithi001.stdout: "num_ops": 0 2024-09-13T11:25:59.248 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-13T11:25:59.261 INFO:tasks.ceph.osd.10.smithi162.stderr:2024-09-13T11:25:59.259+0000 7fb59c9f0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26317) UID: 0 2024-09-13T11:25:59.263 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_blocked_ops 2024-09-13T11:25:59.362 INFO:tasks.ceph.osd.4.smithi001.stderr:2024-09-13T11:25:59.358+0000 7ff8e0218640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26310) UID: 0 2024-09-13T11:25:59.396 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-13T11:25:59.397 INFO:teuthology.orchestra.run.smithi001.stdout: "ops": [], 2024-09-13T11:25:59.397 INFO:teuthology.orchestra.run.smithi001.stdout: "complaint_time": 30, 2024-09-13T11:25:59.397 INFO:teuthology.orchestra.run.smithi001.stdout: "num_blocked_ops": 0 2024-09-13T11:25:59.397 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-13T11:25:59.414 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_historic_ops 2024-09-13T11:25:59.463 INFO:tasks.ceph.osd.5.smithi092.stderr:2024-09-13T11:25:59.460+0000 7f6f0b705640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26346) UID: 0 2024-09-13T11:25:59.564 INFO:tasks.ceph.osd.1.smithi092.stderr:2024-09-13T11:25:59.560+0000 7fe7789c5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33718) UID: 0 2024-09-13T11:25:59.592 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-13T11:25:59.592 INFO:teuthology.orchestra.run.smithi001.stdout: "size": 20, 2024-09-13T11:25:59.592 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 600, 2024-09-13T11:25:59.592 INFO:teuthology.orchestra.run.smithi001.stdout: "ops": [ 2024-09-13T11:25:59.592 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.592 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop_reply(client.4665.0:167 3.9 e41/30 ondisk, result = 0)", 2024-09-13T11:25:59.592 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:22:47.419696+0000", 2024-09-13T11:25:59.592 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 192.17053748999999, 2024-09-13T11:25:59.592 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.64615778199999996, 2024-09-13T11:25:59.592 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:59.592 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:59.593 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:25:59.593 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:59.593 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.593 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:59.593 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:47.419696+0000", 2024-09-13T11:25:59.593 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:59.593 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.593 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.593 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:59.593 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:47.419696+0000", 2024-09-13T11:25:59.593 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:59.593 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.593 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.593 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:59.594 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:47.419697+0000", 2024-09-13T11:25:59.594 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.15e-06 2024-09-13T11:25:59.594 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.594 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.594 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:59.594 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:47.419700+0000", 2024-09-13T11:25:59.594 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.2449999999999998e-06 2024-09-13T11:25:59.594 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.594 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.594 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:59.594 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:48.065637+0000", 2024-09-13T11:25:59.594 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.64593674300000004 2024-09-13T11:25:59.594 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.595 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.595 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:59.595 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:48.065648+0000", 2024-09-13T11:25:59.595 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0865e-05 2024-09-13T11:25:59.595 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.595 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.595 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:59.595 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:48.065720+0000", 2024-09-13T11:25:59.595 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.2020999999999994e-05 2024-09-13T11:25:59.595 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.595 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.595 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:25:59.595 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:48.065728+0000", 2024-09-13T11:25:59.595 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.7660000000000004e-06 2024-09-13T11:25:59.596 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.596 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.596 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:59.596 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:48.065853+0000", 2024-09-13T11:25:59.596 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00012499200000000001 2024-09-13T11:25:59.596 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:59.596 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:59.596 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:59.596 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.596 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.596 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:25383 3.0 3:0a6bb869:::benchmark_data_smithi177_31116_object25382:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e55)", 2024-09-13T11:25:59.596 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:04.447376+0000", 2024-09-13T11:25:59.596 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 175.14285682100001, 2024-09-13T11:25:59.596 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.21239864, 2024-09-13T11:25:59.597 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:59.597 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:59.597 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:59.597 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:25:59.597 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:25:59.597 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:59.597 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 25383 2024-09-13T11:25:59.597 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.597 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:59.597 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.597 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:59.597 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.447376+0000", 2024-09-13T11:25:59.597 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:59.598 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.598 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.598 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:59.598 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.447376+0000", 2024-09-13T11:25:59.598 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:59.598 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.598 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.598 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:59.598 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.447378+0000", 2024-09-13T11:25:59.598 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.6729999999999999e-06 2024-09-13T11:25:59.598 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.598 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.599 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:59.599 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.447433+0000", 2024-09-13T11:25:59.599 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.5371000000000002e-05 2024-09-13T11:25:59.599 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.599 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.599 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:59.599 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.447446+0000", 2024-09-13T11:25:59.599 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.3179999999999999e-05 2024-09-13T11:25:59.599 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.599 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.599 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:59.599 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.447456+0000", 2024-09-13T11:25:59.599 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0008e-05 2024-09-13T11:25:59.599 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.600 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.600 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:59.600 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.447514+0000", 2024-09-13T11:25:59.600 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.8023999999999999e-05 2024-09-13T11:25:59.600 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.600 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.600 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:25:59.600 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.447642+0000", 2024-09-13T11:25:59.600 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000127296 2024-09-13T11:25:59.600 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.600 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.600 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 15", 2024-09-13T11:25:59.600 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.447767+0000", 2024-09-13T11:25:59.601 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000125547 2024-09-13T11:25:59.601 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.601 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.601 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:25:59.601 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.449188+0000", 2024-09-13T11:25:59.601 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.001420604 2024-09-13T11:25:59.601 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.601 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.601 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:25:59.601 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.659668+0000", 2024-09-13T11:25:59.601 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.210479685 2024-09-13T11:25:59.601 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.601 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.601 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:25:59.602 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.659716+0000", 2024-09-13T11:25:59.602 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.8609999999999997e-05 2024-09-13T11:25:59.602 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.602 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.602 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:59.602 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.659775+0000", 2024-09-13T11:25:59.602 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.8641999999999999e-05 2024-09-13T11:25:59.602 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:59.602 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:59.602 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:59.602 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.602 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.602 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4665.0:25408 3.4 e56/40 3:294e23bd:::benchmark_data_smithi177_31116_object25407:head v 56'1645, mlcod=55'1642)", 2024-09-13T11:25:59.602 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:04.494686+0000", 2024-09-13T11:25:59.603 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 175.095547162, 2024-09-13T11:25:59.603 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.29725566399999998, 2024-09-13T11:25:59.603 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:59.603 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:59.603 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:59.603 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:59.603 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.603 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:59.603 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.494686+0000", 2024-09-13T11:25:59.603 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:59.603 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.603 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.603 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:59.604 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.494686+0000", 2024-09-13T11:25:59.604 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:59.604 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.604 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.604 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:59.604 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.494686+0000", 2024-09-13T11:25:59.604 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.2399999999999999e-07 2024-09-13T11:25:59.604 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.604 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.604 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:59.604 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.494809+0000", 2024-09-13T11:25:59.604 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00012308600000000001 2024-09-13T11:25:59.604 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.605 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.605 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:59.605 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.789410+0000", 2024-09-13T11:25:59.605 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.29460066899999998 2024-09-13T11:25:59.605 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.605 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.605 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:59.605 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.789425+0000", 2024-09-13T11:25:59.605 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.4600000000000001e-05 2024-09-13T11:25:59.605 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.605 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.605 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:59.605 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.789507+0000", 2024-09-13T11:25:59.605 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.2785000000000001e-05 2024-09-13T11:25:59.606 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.606 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.606 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:25:59.606 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.789525+0000", 2024-09-13T11:25:59.606 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.8117999999999998e-05 2024-09-13T11:25:59.606 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.606 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.606 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:25:59.606 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.791852+0000", 2024-09-13T11:25:59.606 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0023264459999999998 2024-09-13T11:25:59.606 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.606 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.606 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:59.607 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.791942+0000", 2024-09-13T11:25:59.607 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.9635999999999994e-05 2024-09-13T11:25:59.607 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:59.607 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:59.607 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:59.607 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.607 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.607 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4665.0:30815 3.f e59/30 3:fde17cb1:::benchmark_data_smithi177_31116_object30814:head v 59'1919, mlcod=59'1914)", 2024-09-13T11:25:59.607 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:09.494342+0000", 2024-09-13T11:25:59.607 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 170.09589117600001, 2024-09-13T11:25:59.607 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.22787855700000001, 2024-09-13T11:25:59.607 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:59.607 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:59.607 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:59.608 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:59.608 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.608 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:59.608 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.494342+0000", 2024-09-13T11:25:59.608 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:59.608 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.608 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.608 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:59.608 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.494342+0000", 2024-09-13T11:25:59.608 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:59.608 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.608 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.608 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:59.609 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.494343+0000", 2024-09-13T11:25:59.609 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.251e-06 2024-09-13T11:25:59.609 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.609 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.609 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:59.609 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.494387+0000", 2024-09-13T11:25:59.609 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.4276000000000001e-05 2024-09-13T11:25:59.609 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.609 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.609 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:59.609 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.720454+0000", 2024-09-13T11:25:59.609 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.226066614 2024-09-13T11:25:59.609 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.609 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.610 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:59.610 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.720462+0000", 2024-09-13T11:25:59.610 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.9240000000000007e-06 2024-09-13T11:25:59.610 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.610 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.610 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:59.610 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.720537+0000", 2024-09-13T11:25:59.610 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.4967000000000002e-05 2024-09-13T11:25:59.610 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.610 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.610 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:25:59.610 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.720563+0000", 2024-09-13T11:25:59.610 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.5630999999999999e-05 2024-09-13T11:25:59.610 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.611 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.611 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:25:59.611 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.722183+0000", 2024-09-13T11:25:59.611 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0016209670000000001 2024-09-13T11:25:59.611 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.611 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.611 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:59.611 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.722220+0000", 2024-09-13T11:25:59.611 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.6927000000000001e-05 2024-09-13T11:25:59.611 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:59.611 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:59.611 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:59.611 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.612 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.612 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4665.0:30828 3.f e59/30 3:f3a9c6a9:::benchmark_data_smithi177_31116_object30827:head v 59'1920, mlcod=59'1916)", 2024-09-13T11:25:59.612 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:09.502607+0000", 2024-09-13T11:25:59.612 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 170.08762611899999, 2024-09-13T11:25:59.612 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.26374408300000002, 2024-09-13T11:25:59.612 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:59.612 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:59.612 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:59.612 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:59.612 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.612 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:59.612 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.502607+0000", 2024-09-13T11:25:59.612 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:59.613 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.613 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.613 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:59.613 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.502607+0000", 2024-09-13T11:25:59.613 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:59.613 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.613 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.613 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:59.613 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.502609+0000", 2024-09-13T11:25:59.613 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.84e-06 2024-09-13T11:25:59.613 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.613 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.614 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:59.614 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.502663+0000", 2024-09-13T11:25:59.614 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.3971000000000002e-05 2024-09-13T11:25:59.614 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.614 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.614 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:59.614 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.763912+0000", 2024-09-13T11:25:59.614 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.26124904300000001 2024-09-13T11:25:59.614 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.614 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.614 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:59.614 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.763920+0000", 2024-09-13T11:25:59.614 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.1149999999999994e-06 2024-09-13T11:25:59.615 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.615 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.615 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:59.615 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.764024+0000", 2024-09-13T11:25:59.615 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00010380899999999999 2024-09-13T11:25:59.615 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.615 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.615 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:25:59.615 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.764068+0000", 2024-09-13T11:25:59.615 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.4638999999999997e-05 2024-09-13T11:25:59.615 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.615 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.615 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:25:59.615 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.766308+0000", 2024-09-13T11:25:59.616 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0022392039999999998 2024-09-13T11:25:59.616 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.616 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.616 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:59.616 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.766351+0000", 2024-09-13T11:25:59.616 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.3461999999999998e-05 2024-09-13T11:25:59.616 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:59.616 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:59.616 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:59.616 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.616 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.616 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDPGPush(3.1b 84/78 [PushOp(3:dcddd139:::benchmark_data_smithi177_31116_object10384:head, version: 48'673, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:dcddd139:::benchmark_data_smithi177_31116_object10384:head@48'673, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:25:59.616 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:41.400602+0000", 2024-09-13T11:25:59.617 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 138.189630634, 2024-09-13T11:25:59.617 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.43365821999999998, 2024-09-13T11:25:59.617 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:59.617 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:59.617 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:25:59.617 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:59.617 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.617 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:59.617 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:41.400602+0000", 2024-09-13T11:25:59.617 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:59.617 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.617 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.618 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:59.618 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:41.400602+0000", 2024-09-13T11:25:59.618 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:59.618 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.618 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.618 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:59.618 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:41.400604+0000", 2024-09-13T11:25:59.618 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.99e-06 2024-09-13T11:25:59.618 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.618 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.618 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:59.618 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:41.400653+0000", 2024-09-13T11:25:59.618 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.8195000000000002e-05 2024-09-13T11:25:59.619 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.619 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.619 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:59.619 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:41.833180+0000", 2024-09-13T11:25:59.619 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.43252769200000002 2024-09-13T11:25:59.619 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.619 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.619 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:59.619 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:41.833213+0000", 2024-09-13T11:25:59.619 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.2737999999999997e-05 2024-09-13T11:25:59.619 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.619 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.619 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:59.620 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:41.833408+0000", 2024-09-13T11:25:59.620 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00019473799999999999 2024-09-13T11:25:59.620 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.620 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.620 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:25:59.620 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:41.833417+0000", 2024-09-13T11:25:59.620 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.5349999999999993e-06 2024-09-13T11:25:59.620 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.620 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.620 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:59.620 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:41.834261+0000", 2024-09-13T11:25:59.620 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00084333200000000002 2024-09-13T11:25:59.620 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:59.621 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:59.621 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:59.621 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.621 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.621 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDPGPushReply(3.20 85/77 [PushReplyOp(3:066982a6:::benchmark_data_smithi177_31116_object16400:head)])", 2024-09-13T11:25:59.621 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:42.671551+0000", 2024-09-13T11:25:59.621 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 136.918681811, 2024-09-13T11:25:59.621 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.28249085499999999, 2024-09-13T11:25:59.621 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:59.621 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:59.621 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-13T11:25:59.621 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:59.621 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.621 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:59.622 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.671551+0000", 2024-09-13T11:25:59.622 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:59.622 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.622 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.622 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:59.622 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.671551+0000", 2024-09-13T11:25:59.622 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:59.622 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.622 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.622 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:59.622 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.671552+0000", 2024-09-13T11:25:59.622 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.097e-06 2024-09-13T11:25:59.622 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.623 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.623 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:59.623 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.671556+0000", 2024-09-13T11:25:59.623 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.8519999999999997e-06 2024-09-13T11:25:59.623 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.623 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.623 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:59.623 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.951769+0000", 2024-09-13T11:25:59.623 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.28021293000000003 2024-09-13T11:25:59.623 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.623 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.623 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:59.623 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.951780+0000", 2024-09-13T11:25:59.623 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.1059e-05 2024-09-13T11:25:59.624 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.624 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.624 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:59.624 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.951992+0000", 2024-09-13T11:25:59.624 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000211478 2024-09-13T11:25:59.624 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.624 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.624 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:59.624 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.954042+0000", 2024-09-13T11:25:59.624 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0020504389999999998 2024-09-13T11:25:59.624 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:59.624 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:59.624 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:59.625 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.625 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.625 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDPGPush(3.1b 89/78 [PushOp(3:dc4c79e7:::benchmark_data_smithi177_31116_object19857:head, version: 54'1266, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:dc4c79e7:::benchmark_data_smithi177_31116_object19857:head@54'1266, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:25:59.625 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:46.649128+0000", 2024-09-13T11:25:59.625 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 132.94110470300001, 2024-09-13T11:25:59.625 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.29963603, 2024-09-13T11:25:59.625 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:59.625 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:59.625 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:25:59.625 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:59.625 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.625 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:59.625 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.649128+0000", 2024-09-13T11:25:59.626 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:59.626 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.626 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.626 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:59.626 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.649128+0000", 2024-09-13T11:25:59.626 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:59.626 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.626 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.626 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:59.626 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.649130+0000", 2024-09-13T11:25:59.626 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2950000000000001e-06 2024-09-13T11:25:59.626 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.626 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.627 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:59.627 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.649197+0000", 2024-09-13T11:25:59.627 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.7033999999999995e-05 2024-09-13T11:25:59.627 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.627 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.627 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:59.627 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.947691+0000", 2024-09-13T11:25:59.627 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.29849387300000002 2024-09-13T11:25:59.627 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.627 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.627 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:59.627 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.947702+0000", 2024-09-13T11:25:59.627 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.1239e-05 2024-09-13T11:25:59.627 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.628 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.628 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:59.628 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.947799+0000", 2024-09-13T11:25:59.628 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.7283999999999999e-05 2024-09-13T11:25:59.628 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.628 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.628 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:25:59.628 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.947809+0000", 2024-09-13T11:25:59.628 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.008e-05 2024-09-13T11:25:59.628 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.628 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.628 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:59.628 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.948764+0000", 2024-09-13T11:25:59.629 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00095522499999999998 2024-09-13T11:25:59.629 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:59.629 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:59.629 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:59.629 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.629 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.629 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:71927 3.0 3:0197244a:::benchmark_data_smithi177_31116_object71926:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2024-09-13T11:25:59.629 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:54.199022+0000", 2024-09-13T11:25:59.629 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 125.391211347, 2024-09-13T11:25:59.629 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.26819617800000001, 2024-09-13T11:25:59.629 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:59.629 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:59.629 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:59.629 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:25:59.630 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:25:59.630 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:59.630 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 71927 2024-09-13T11:25:59.630 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.630 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:59.630 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.630 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:59.630 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:54.199022+0000", 2024-09-13T11:25:59.630 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:59.630 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.630 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.630 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:59.631 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:54.199022+0000", 2024-09-13T11:25:59.631 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:59.631 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.631 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.631 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:59.631 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:54.199025+0000", 2024-09-13T11:25:59.631 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.2200000000000001e-06 2024-09-13T11:25:59.631 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.631 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.631 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:59.631 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:54.199076+0000", 2024-09-13T11:25:59.631 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.1184000000000003e-05 2024-09-13T11:25:59.631 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.631 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.632 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:59.632 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:54.199095+0000", 2024-09-13T11:25:59.632 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.9253999999999999e-05 2024-09-13T11:25:59.632 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.632 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.632 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:59.632 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:54.199107+0000", 2024-09-13T11:25:59.632 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.1916000000000001e-05 2024-09-13T11:25:59.632 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.632 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.632 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:59.632 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:54.199196+0000", 2024-09-13T11:25:59.633 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.8645000000000003e-05 2024-09-13T11:25:59.633 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.633 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.633 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:25:59.633 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:54.199421+0000", 2024-09-13T11:25:59.633 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000224868 2024-09-13T11:25:59.633 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.633 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.634 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 15", 2024-09-13T11:25:59.634 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:54.199647+0000", 2024-09-13T11:25:59.634 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00022666099999999999 2024-09-13T11:25:59.634 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.634 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.634 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:25:59.634 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:54.201333+0000", 2024-09-13T11:25:59.634 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.001685761 2024-09-13T11:25:59.634 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.634 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.634 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:25:59.634 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:54.467097+0000", 2024-09-13T11:25:59.635 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.26576393399999998 2024-09-13T11:25:59.635 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.635 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.635 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:25:59.635 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:54.467163+0000", 2024-09-13T11:25:59.635 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.6088999999999997e-05 2024-09-13T11:25:59.635 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.635 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.635 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:59.635 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:54.467218+0000", 2024-09-13T11:25:59.635 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.4645999999999998e-05 2024-09-13T11:25:59.635 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:59.635 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:59.635 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:59.636 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.636 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.636 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDPGPush(3.25 121/120 [PushOp(3:a5de2229:::benchmark_data_smithi177_31116_object1005:head, version: 43'62, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:a5de2229:::benchmark_data_smithi177_31116_object1005:head@43'62, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:25:59.636 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:24:27.379880+0000", 2024-09-13T11:25:59.636 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 92.210353509000001, 2024-09-13T11:25:59.636 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.72647719499999996, 2024-09-13T11:25:59.636 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:59.636 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:59.636 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:25:59.636 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:59.636 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.636 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:59.636 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:27.379880+0000", 2024-09-13T11:25:59.637 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:59.637 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.637 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.637 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:59.637 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:27.379880+0000", 2024-09-13T11:25:59.637 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:59.637 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.637 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.637 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:59.637 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:27.379882+0000", 2024-09-13T11:25:59.637 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.1050000000000002e-06 2024-09-13T11:25:59.637 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.637 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.638 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:59.638 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:27.379976+0000", 2024-09-13T11:25:59.638 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.4468000000000004e-05 2024-09-13T11:25:59.638 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.638 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.638 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:59.638 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:28.105075+0000", 2024-09-13T11:25:59.638 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.72509911299999996 2024-09-13T11:25:59.638 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.638 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.638 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:59.638 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:28.105111+0000", 2024-09-13T11:25:59.638 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.5403e-05 2024-09-13T11:25:59.638 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.639 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.639 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:59.639 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:28.105267+0000", 2024-09-13T11:25:59.639 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00015663600000000001 2024-09-13T11:25:59.639 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.639 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.639 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:25:59.639 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:28.105291+0000", 2024-09-13T11:25:59.639 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.4164000000000001e-05 2024-09-13T11:25:59.639 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.639 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.639 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:59.639 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:28.106357+0000", 2024-09-13T11:25:59.639 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.001065306 2024-09-13T11:25:59.640 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:59.640 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:59.640 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:59.640 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.640 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.640 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:113646 3.20 3:059047a0:::benchmark_data_smithi177_31116_object113645:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124)", 2024-09-13T11:25:59.640 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:24:32.178229+0000", 2024-09-13T11:25:59.640 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 87.412003549000005, 2024-09-13T11:25:59.640 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.23346934699999999, 2024-09-13T11:25:59.640 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:59.640 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:59.640 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:59.640 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:25:59.640 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:25:59.641 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:59.641 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 113646 2024-09-13T11:25:59.641 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.641 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:59.641 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.641 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:59.641 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:32.178229+0000", 2024-09-13T11:25:59.641 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:59.641 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.641 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.641 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:59.641 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:32.178229+0000", 2024-09-13T11:25:59.641 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:59.642 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.642 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.642 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:59.642 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:32.178232+0000", 2024-09-13T11:25:59.642 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.069e-06 2024-09-13T11:25:59.642 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.642 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.642 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:59.642 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:32.178266+0000", 2024-09-13T11:25:59.642 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.4242999999999997e-05 2024-09-13T11:25:59.642 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.642 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.642 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:59.642 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:32.178278+0000", 2024-09-13T11:25:59.643 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2558000000000001e-05 2024-09-13T11:25:59.643 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.643 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.643 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:59.643 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:32.178288+0000", 2024-09-13T11:25:59.643 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.6469999999999993e-06 2024-09-13T11:25:59.643 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.643 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.643 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:59.643 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:32.178349+0000", 2024-09-13T11:25:59.643 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.0643999999999997e-05 2024-09-13T11:25:59.643 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.643 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.644 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:25:59.644 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:32.178477+0000", 2024-09-13T11:25:59.644 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000128038 2024-09-13T11:25:59.644 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.644 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.644 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 0", 2024-09-13T11:25:59.644 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:32.178602+0000", 2024-09-13T11:25:59.644 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00012553600000000001 2024-09-13T11:25:59.644 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.644 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.644 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:25:59.644 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:32.180157+0000", 2024-09-13T11:25:59.644 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.001554386 2024-09-13T11:25:59.645 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.645 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.645 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:25:59.645 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:32.411634+0000", 2024-09-13T11:25:59.645 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.231477561 2024-09-13T11:25:59.645 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.645 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.645 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:25:59.645 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:32.411672+0000", 2024-09-13T11:25:59.645 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.7357999999999997e-05 2024-09-13T11:25:59.645 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.645 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.645 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:59.645 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:32.411699+0000", 2024-09-13T11:25:59.645 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.7307000000000001e-05 2024-09-13T11:25:59.646 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:59.646 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:59.646 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:59.646 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.646 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.646 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop_reply(client.4665.0:113646 3.20 e124/77 ondisk, result = 0)", 2024-09-13T11:25:59.646 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:24:32.180524+0000", 2024-09-13T11:25:59.646 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 87.409708768000002, 2024-09-13T11:25:59.646 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.231180894, 2024-09-13T11:25:59.646 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:59.646 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:59.646 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:25:59.646 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:59.647 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.647 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:59.647 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:32.180524+0000", 2024-09-13T11:25:59.647 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:59.647 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.647 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.647 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:59.647 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:32.180524+0000", 2024-09-13T11:25:59.647 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:59.647 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.647 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.647 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:59.647 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:32.180526+0000", 2024-09-13T11:25:59.647 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.4419999999999999e-06 2024-09-13T11:25:59.648 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.648 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.648 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:59.648 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:32.180530+0000", 2024-09-13T11:25:59.648 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.7330000000000003e-06 2024-09-13T11:25:59.648 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.648 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.648 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:59.648 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:32.411549+0000", 2024-09-13T11:25:59.648 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.23101862300000001 2024-09-13T11:25:59.648 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.648 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.648 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:59.649 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:32.411558+0000", 2024-09-13T11:25:59.649 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.5949999999999999e-06 2024-09-13T11:25:59.649 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.649 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.649 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:59.649 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:32.411622+0000", 2024-09-13T11:25:59.649 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.4251000000000005e-05 2024-09-13T11:25:59.649 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.649 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.649 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:25:59.649 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:32.411629+0000", 2024-09-13T11:25:59.649 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.328e-06 2024-09-13T11:25:59.649 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.650 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.650 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:59.650 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:32.411705+0000", 2024-09-13T11:25:59.650 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.5921999999999996e-05 2024-09-13T11:25:59.650 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:59.650 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:59.650 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:59.650 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.650 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.650 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop_reply(client.4665.0:129892 3.0 e134/75 ondisk, result = 0)", 2024-09-13T11:25:59.650 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:24:47.136509+0000", 2024-09-13T11:25:59.650 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 72.453723721000003, 2024-09-13T11:25:59.650 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.211101067, 2024-09-13T11:25:59.650 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:59.651 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:59.651 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:25:59.651 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:59.651 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.651 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:59.651 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:47.136509+0000", 2024-09-13T11:25:59.651 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:59.651 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.651 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.651 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:59.651 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:47.136509+0000", 2024-09-13T11:25:59.651 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:59.651 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.651 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.652 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:59.652 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:47.136510+0000", 2024-09-13T11:25:59.652 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.17e-06 2024-09-13T11:25:59.652 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.652 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.652 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:59.652 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:47.136514+0000", 2024-09-13T11:25:59.652 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.5609999999999999e-06 2024-09-13T11:25:59.652 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.652 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.652 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:59.652 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:47.347445+0000", 2024-09-13T11:25:59.653 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.210931324 2024-09-13T11:25:59.653 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.653 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.653 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:59.653 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:47.347450+0000", 2024-09-13T11:25:59.653 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.3819999999999997e-06 2024-09-13T11:25:59.653 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.653 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.653 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:59.653 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:47.347491+0000", 2024-09-13T11:25:59.653 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.1025000000000001e-05 2024-09-13T11:25:59.653 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.653 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.653 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:25:59.654 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:47.347496+0000", 2024-09-13T11:25:59.654 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.463e-06 2024-09-13T11:25:59.654 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.654 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.654 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:59.654 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:47.347610+0000", 2024-09-13T11:25:59.654 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00011414200000000001 2024-09-13T11:25:59.654 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:59.654 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:59.654 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:59.654 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.654 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.655 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:138294 3.9 3:90012dab:::benchmark_data_smithi177_31116_object138293:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e140)", 2024-09-13T11:25:59.655 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:24:54.250266+0000", 2024-09-13T11:25:59.655 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 65.339966836000002, 2024-09-13T11:25:59.655 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.33165556200000001, 2024-09-13T11:25:59.655 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:59.655 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:59.655 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:59.655 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:25:59.655 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:25:59.655 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:59.655 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 138294 2024-09-13T11:25:59.655 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.655 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:59.655 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.656 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:59.656 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:54.250266+0000", 2024-09-13T11:25:59.656 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:59.656 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.656 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.656 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:59.656 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:54.250266+0000", 2024-09-13T11:25:59.656 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:59.656 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.656 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.656 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:59.656 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:54.250269+0000", 2024-09-13T11:25:59.656 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.3159999999999999e-06 2024-09-13T11:25:59.657 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.657 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.657 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:59.657 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:54.250362+0000", 2024-09-13T11:25:59.657 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.3051000000000001e-05 2024-09-13T11:25:59.657 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.657 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.657 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:59.657 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:54.250374+0000", 2024-09-13T11:25:59.657 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2327000000000001e-05 2024-09-13T11:25:59.657 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.657 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.657 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:59.657 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:54.250382+0000", 2024-09-13T11:25:59.658 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.7249999999999997e-06 2024-09-13T11:25:59.658 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.658 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.658 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:59.658 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:54.250459+0000", 2024-09-13T11:25:59.658 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.7218999999999994e-05 2024-09-13T11:25:59.658 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.658 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.658 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:25:59.658 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:54.250611+0000", 2024-09-13T11:25:59.658 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00015241000000000001 2024-09-13T11:25:59.658 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.658 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.658 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 2", 2024-09-13T11:25:59.659 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:54.250777+0000", 2024-09-13T11:25:59.659 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00016575199999999999 2024-09-13T11:25:59.659 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.659 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.659 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:25:59.659 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:54.252776+0000", 2024-09-13T11:25:59.659 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0019987260000000001 2024-09-13T11:25:59.659 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.659 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.659 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:25:59.659 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:54.581758+0000", 2024-09-13T11:25:59.659 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.32898232799999999 2024-09-13T11:25:59.659 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.659 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.660 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:25:59.660 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:54.581854+0000", 2024-09-13T11:25:59.660 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.5468000000000001e-05 2024-09-13T11:25:59.660 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.660 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.660 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:59.660 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:54.581922+0000", 2024-09-13T11:25:59.660 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.8239999999999997e-05 2024-09-13T11:25:59.660 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:59.660 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:59.660 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:59.660 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.660 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.660 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop_reply(client.4665.0:140790 3.20 e144/77 ondisk, result = 0)", 2024-09-13T11:25:59.661 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:24:58.385972+0000", 2024-09-13T11:25:59.661 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 61.204261268000003, 2024-09-13T11:25:59.661 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.51829568500000001, 2024-09-13T11:25:59.661 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:59.661 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:59.661 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:25:59.661 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:59.661 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.661 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:59.661 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.385972+0000", 2024-09-13T11:25:59.661 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:59.661 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.661 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.661 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:59.662 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.385972+0000", 2024-09-13T11:25:59.662 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:59.662 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.662 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.662 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:59.662 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.385974+0000", 2024-09-13T11:25:59.662 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.7400000000000001e-06 2024-09-13T11:25:59.662 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.662 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.662 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:59.662 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.385979+0000", 2024-09-13T11:25:59.662 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.091e-06 2024-09-13T11:25:59.662 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.663 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.663 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:59.663 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.904091+0000", 2024-09-13T11:25:59.663 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.51811194900000002 2024-09-13T11:25:59.663 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.663 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.663 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:59.663 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.904098+0000", 2024-09-13T11:25:59.663 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.0609999999999998e-06 2024-09-13T11:25:59.663 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.663 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.663 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:59.663 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.904156+0000", 2024-09-13T11:25:59.664 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.7902000000000002e-05 2024-09-13T11:25:59.664 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.664 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.664 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:25:59.664 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.904163+0000", 2024-09-13T11:25:59.664 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.1430000000000001e-06 2024-09-13T11:25:59.664 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.664 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.664 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:59.664 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.904267+0000", 2024-09-13T11:25:59.664 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000104799 2024-09-13T11:25:59.664 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:59.664 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:59.664 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:59.665 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.665 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.665 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:170721 3.9 3:93244b1b:::benchmark_data_smithi177_31116_object30418:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e162)", 2024-09-13T11:25:59.665 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:25:21.721042+0000", 2024-09-13T11:25:59.665 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 37.869191266999998, 2024-09-13T11:25:59.665 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.19807348799999999, 2024-09-13T11:25:59.665 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:59.665 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:59.665 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:59.665 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:25:59.665 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:25:59.665 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:59.665 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 170721 2024-09-13T11:25:59.666 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.666 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:59.666 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.666 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:59.666 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:21.721042+0000", 2024-09-13T11:25:59.666 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:59.666 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.666 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.666 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:59.666 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:21.721042+0000", 2024-09-13T11:25:59.666 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:59.667 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.667 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.667 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:59.667 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:21.721044+0000", 2024-09-13T11:25:59.667 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.781e-06 2024-09-13T11:25:59.667 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.667 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.667 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:59.667 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:21.721050+0000", 2024-09-13T11:25:59.667 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.1840000000000004e-06 2024-09-13T11:25:59.667 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.667 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.668 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:59.668 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:21.721065+0000", 2024-09-13T11:25:59.668 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.5073e-05 2024-09-13T11:25:59.668 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.668 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.668 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:59.668 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:21.721073+0000", 2024-09-13T11:25:59.668 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.6030000000000005e-06 2024-09-13T11:25:59.668 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.668 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.668 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:59.668 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:21.721140+0000", 2024-09-13T11:25:59.669 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.6979999999999999e-05 2024-09-13T11:25:59.669 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.669 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.669 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:25:59.669 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:21.721260+0000", 2024-09-13T11:25:59.669 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000119119 2024-09-13T11:25:59.669 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.669 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.669 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 2", 2024-09-13T11:25:59.669 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:21.721363+0000", 2024-09-13T11:25:59.669 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000103052 2024-09-13T11:25:59.669 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.669 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.670 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:25:59.670 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:21.723031+0000", 2024-09-13T11:25:59.670 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0016684650000000001 2024-09-13T11:25:59.670 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.670 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.670 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:25:59.670 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:21.919011+0000", 2024-09-13T11:25:59.670 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.19598026800000001 2024-09-13T11:25:59.670 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.670 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.670 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:25:59.670 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:21.919070+0000", 2024-09-13T11:25:59.671 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.9110000000000002e-05 2024-09-13T11:25:59.671 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.671 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.671 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:59.671 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:21.919115+0000", 2024-09-13T11:25:59.671 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.4852999999999999e-05 2024-09-13T11:25:59.671 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:59.671 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:59.671 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:59.671 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.671 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.671 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4665.0:191082 3.26 e175/140 3:67bd986b:::benchmark_data_smithi177_31116_object50779:head v 175'6183, mlcod=175'6180)", 2024-09-13T11:25:59.671 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:25:38.551017+0000", 2024-09-13T11:25:59.672 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 21.039215848000001, 2024-09-13T11:25:59.672 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.29063407899999999, 2024-09-13T11:25:59.672 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:59.672 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:59.672 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:59.672 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:59.672 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.672 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:59.672 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:38.551017+0000", 2024-09-13T11:25:59.672 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:59.672 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.672 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.673 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:59.673 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:38.551017+0000", 2024-09-13T11:25:59.673 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:59.673 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.673 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.673 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:59.673 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:38.551020+0000", 2024-09-13T11:25:59.673 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.9170000000000002e-06 2024-09-13T11:25:59.673 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.673 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.673 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:59.673 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:38.551033+0000", 2024-09-13T11:25:59.673 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2843000000000001e-05 2024-09-13T11:25:59.673 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.674 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.674 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:59.674 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:38.838514+0000", 2024-09-13T11:25:59.674 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.28748061600000002 2024-09-13T11:25:59.674 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.674 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.674 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:59.674 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:38.838532+0000", 2024-09-13T11:25:59.674 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.8012999999999998e-05 2024-09-13T11:25:59.674 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.674 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.674 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:59.674 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:38.838646+0000", 2024-09-13T11:25:59.674 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000114596 2024-09-13T11:25:59.675 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.675 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.675 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:25:59.675 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:38.838681+0000", 2024-09-13T11:25:59.675 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.5099999999999999e-05 2024-09-13T11:25:59.675 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.675 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.675 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:25:59.675 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:38.841593+0000", 2024-09-13T11:25:59.675 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0029117050000000001 2024-09-13T11:25:59.675 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.675 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.675 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:59.676 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:38.841651+0000", 2024-09-13T11:25:59.676 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.8288999999999997e-05 2024-09-13T11:25:59.676 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:59.676 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:59.676 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:59.676 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.676 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.676 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4665.0:191198 3.26 e175/140 3:64ff4057:::benchmark_data_smithi177_31116_object50895:head v 175'6185, mlcod=175'6182)", 2024-09-13T11:25:59.676 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:25:38.948739+0000", 2024-09-13T11:25:59.676 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 20.641494387000002, 2024-09-13T11:25:59.676 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.25772718100000003, 2024-09-13T11:25:59.676 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:59.676 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:59.677 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:59.677 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:59.677 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.677 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:59.677 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:38.948739+0000", 2024-09-13T11:25:59.677 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:59.677 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.677 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.677 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:59.677 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:38.948739+0000", 2024-09-13T11:25:59.677 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:59.677 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.677 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.678 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:59.678 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:38.948743+0000", 2024-09-13T11:25:59.678 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.0600000000000001e-06 2024-09-13T11:25:59.678 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.678 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.678 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:59.678 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:38.948760+0000", 2024-09-13T11:25:59.678 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.7540000000000001e-05 2024-09-13T11:25:59.678 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.678 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.678 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:59.678 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:39.204732+0000", 2024-09-13T11:25:59.678 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.25597220199999998 2024-09-13T11:25:59.679 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.679 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.679 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:59.679 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:39.204742+0000", 2024-09-13T11:25:59.679 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.6579999999999997e-06 2024-09-13T11:25:59.679 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.679 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.679 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:59.679 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:39.204834+0000", 2024-09-13T11:25:59.679 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.1565999999999997e-05 2024-09-13T11:25:59.679 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.679 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.679 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:25:59.680 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:39.204854+0000", 2024-09-13T11:25:59.680 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.0809e-05 2024-09-13T11:25:59.680 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.680 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.680 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:25:59.680 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:39.206423+0000", 2024-09-13T11:25:59.680 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0015685219999999999 2024-09-13T11:25:59.680 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.680 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.680 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:59.680 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:39.206466+0000", 2024-09-13T11:25:59.680 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.2824000000000002e-05 2024-09-13T11:25:59.680 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:59.681 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:59.681 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:59.681 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.681 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.681 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:196438 3.20 3:0573f1a4:::benchmark_data_smithi177_31116_object56135:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e178)", 2024-09-13T11:25:59.681 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:25:43.521712+0000", 2024-09-13T11:25:59.681 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 16.068521287999999, 2024-09-13T11:25:59.681 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.23827029799999999, 2024-09-13T11:25:59.681 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:59.681 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:59.681 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:59.681 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:25:59.682 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:25:59.682 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:59.682 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 196438 2024-09-13T11:25:59.682 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.682 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:59.682 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.682 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:59.682 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:43.521712+0000", 2024-09-13T11:25:59.682 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:59.682 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.682 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.682 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:59.682 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:43.521712+0000", 2024-09-13T11:25:59.683 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:59.683 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.683 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.683 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:59.683 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:43.521714+0000", 2024-09-13T11:25:59.683 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.6970000000000002e-06 2024-09-13T11:25:59.683 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.683 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.683 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:59.683 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:43.521723+0000", 2024-09-13T11:25:59.683 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.1319999999999994e-06 2024-09-13T11:25:59.683 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.684 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.684 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:59.684 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:43.521743+0000", 2024-09-13T11:25:59.684 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.0446e-05 2024-09-13T11:25:59.684 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.684 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.684 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:59.684 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:43.521754+0000", 2024-09-13T11:25:59.684 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.1314e-05 2024-09-13T11:25:59.684 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.684 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.684 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:59.684 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:43.521839+0000", 2024-09-13T11:25:59.685 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.4951999999999996e-05 2024-09-13T11:25:59.685 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.685 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.685 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:25:59.685 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:43.522010+0000", 2024-09-13T11:25:59.685 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000171053 2024-09-13T11:25:59.685 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.685 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.685 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 0", 2024-09-13T11:25:59.685 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:43.522160+0000", 2024-09-13T11:25:59.685 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000149987 2024-09-13T11:25:59.685 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.686 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.686 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:25:59.686 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:43.524717+0000", 2024-09-13T11:25:59.686 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0025565649999999998 2024-09-13T11:25:59.686 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.686 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.686 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:25:59.686 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:43.759800+0000", 2024-09-13T11:25:59.686 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.23508320699999999 2024-09-13T11:25:59.686 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.686 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.687 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:25:59.687 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:43.759917+0000", 2024-09-13T11:25:59.687 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000116605 2024-09-13T11:25:59.687 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.687 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.687 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:59.687 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:43.759982+0000", 2024-09-13T11:25:59.687 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.5339999999999994e-05 2024-09-13T11:25:59.687 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:59.687 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:59.687 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:59.687 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.688 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.688 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:211295 3.2 3:433aa750:::benchmark_data_smithi177_31116_object70992:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e185)", 2024-09-13T11:25:59.688 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:25:55.459639+0000", 2024-09-13T11:25:59.688 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 4.1305940379999999, 2024-09-13T11:25:59.688 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.28751246200000002, 2024-09-13T11:25:59.688 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:25:59.688 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:25:59.688 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:25:59.688 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:25:59.688 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:25:59.688 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:59.688 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 211295 2024-09-13T11:25:59.688 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.689 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:25:59.689 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.689 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:25:59.689 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:55.459639+0000", 2024-09-13T11:25:59.689 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:59.689 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.689 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.689 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:25:59.689 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:55.459639+0000", 2024-09-13T11:25:59.689 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:25:59.689 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.689 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.689 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:25:59.689 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:55.459642+0000", 2024-09-13T11:25:59.690 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.5940000000000001e-06 2024-09-13T11:25:59.690 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.690 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.690 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:25:59.690 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:55.459649+0000", 2024-09-13T11:25:59.690 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.0899999999999999e-06 2024-09-13T11:25:59.690 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.690 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.690 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:25:59.690 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:55.459669+0000", 2024-09-13T11:25:59.690 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.0305999999999998e-05 2024-09-13T11:25:59.690 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.690 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.691 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:25:59.691 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:55.459683+0000", 2024-09-13T11:25:59.691 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.4219999999999999e-05 2024-09-13T11:25:59.691 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.691 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.691 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:25:59.691 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:55.459761+0000", 2024-09-13T11:25:59.691 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.8072999999999996e-05 2024-09-13T11:25:59.691 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.691 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.691 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:25:59.691 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:55.459912+0000", 2024-09-13T11:25:59.691 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000150304 2024-09-13T11:25:59.691 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.692 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.692 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 3", 2024-09-13T11:25:59.692 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:55.460013+0000", 2024-09-13T11:25:59.692 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000101828 2024-09-13T11:25:59.692 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.692 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.692 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:25:59.692 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:55.462204+0000", 2024-09-13T11:25:59.692 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.002191082 2024-09-13T11:25:59.692 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.692 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.692 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:25:59.692 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:55.747086+0000", 2024-09-13T11:25:59.693 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.28488175100000002 2024-09-13T11:25:59.693 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.693 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.693 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:25:59.693 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:55.747120+0000", 2024-09-13T11:25:59.693 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.3448000000000002e-05 2024-09-13T11:25:59.693 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:25:59.693 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:25:59.693 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:25:59.693 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:55.747151+0000", 2024-09-13T11:25:59.693 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.1766e-05 2024-09-13T11:25:59.693 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:59.693 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:59.694 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:59.694 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:25:59.694 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:25:59.694 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-13T11:25:59.695 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-13T11:25:59.696 INFO:tasks.ceph.osd.12.smithi001.stderr:2024-09-13T11:25:59.690+0000 7f892c6e6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26314) UID: 0 2024-09-13T11:25:59.796 INFO:tasks.ceph.osd.5.smithi092.stderr:2024-09-13T11:25:59.792+0000 7f6f0b705640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26346) UID: 0 2024-09-13T11:25:59.841 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-09-13T11:25:59.841 INFO:teuthology.orchestra.run.smithi092.stdout: "ops": [ 2024-09-13T11:25:59.841 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:59.841 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:217026 3.23 3:c50abe18:::benchmark_data_smithi177_31116_object76723:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e187)", 2024-09-13T11:25:59.841 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:25:59.831898+0000", 2024-09-13T11:25:59.841 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 0.0092536059999999993, 2024-09-13T11:25:59.841 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0092789329999999996, 2024-09-13T11:25:59.841 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:25:59.842 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:25:59.842 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "waiting for sub ops", 2024-09-13T11:25:59.842 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:25:59.842 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:25:59.842 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:25:59.842 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 217026 2024-09-13T11:25:59.842 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:59.842 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:25:59.842 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:59.842 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:25:59.842 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:59.831898+0000", 2024-09-13T11:25:59.843 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:59.843 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:59.843 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:59.843 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:25:59.843 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:59.831898+0000", 2024-09-13T11:25:59.843 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:59.843 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:59.843 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:59.843 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:25:59.843 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:59.831900+0000", 2024-09-13T11:25:59.843 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.5880000000000001e-06 2024-09-13T11:25:59.843 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:59.843 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:59.844 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:25:59.844 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:59.831905+0000", 2024-09-13T11:25:59.844 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.728e-06 2024-09-13T11:25:59.844 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:59.844 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:59.844 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:25:59.844 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:59.831921+0000", 2024-09-13T11:25:59.844 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.5356e-05 2024-09-13T11:25:59.844 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:59.844 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:59.844 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:25:59.844 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:59.831930+0000", 2024-09-13T11:25:59.844 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.6760000000000002e-06 2024-09-13T11:25:59.845 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:59.845 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:59.845 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:25:59.845 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:59.831974+0000", 2024-09-13T11:25:59.845 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.3120000000000001e-05 2024-09-13T11:25:59.845 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:59.845 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:59.845 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:25:59.845 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:59.832053+0000", 2024-09-13T11:25:59.845 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.9239000000000005e-05 2024-09-13T11:25:59.845 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:59.845 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:59.845 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 9", 2024-09-13T11:25:59.846 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:59.832113+0000", 2024-09-13T11:25:59.846 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.9697999999999997e-05 2024-09-13T11:25:59.846 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:59.846 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:59.846 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:25:59.846 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:59.833821+0000", 2024-09-13T11:25:59.846 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00170813 2024-09-13T11:25:59.846 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:59.846 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:25:59.846 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:59.846 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:59.846 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:59.847 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop(client.4665.0:217038 3.12 e187/122 3:4ed5843b:::benchmark_data_smithi177_31116_object76735:head v 187'8193, mlcod=187'8191)", 2024-09-13T11:25:59.847 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:25:59.840924+0000", 2024-09-13T11:25:59.847 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 0.00022816199999999999, 2024-09-13T11:25:59.847 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000328065, 2024-09-13T11:25:59.847 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:25:59.847 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:25:59.847 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-09-13T11:25:59.847 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:25:59.847 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:59.847 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:25:59.847 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:59.840924+0000", 2024-09-13T11:25:59.847 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:59.848 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:59.848 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:59.848 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:25:59.848 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:59.840924+0000", 2024-09-13T11:25:59.848 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:25:59.848 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:59.848 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:59.848 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:25:59.848 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:59.840926+0000", 2024-09-13T11:25:59.848 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.99e-06 2024-09-13T11:25:59.848 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:59.848 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:59.848 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:25:59.849 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:59.840936+0000", 2024-09-13T11:25:59.849 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.9399999999999997e-06 2024-09-13T11:25:59.849 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:59.849 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:59.849 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:25:59.849 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:59.840978+0000", 2024-09-13T11:25:59.849 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.2135000000000002e-05 2024-09-13T11:25:59.849 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:59.849 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:59.849 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:25:59.849 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:59.840990+0000", 2024-09-13T11:25:59.849 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.2394e-05 2024-09-13T11:25:59.849 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:59.849 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:59.850 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:25:59.850 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:59.841043+0000", 2024-09-13T11:25:59.850 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.2948000000000002e-05 2024-09-13T11:25:59.850 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:25:59.850 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:25:59.850 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:25:59.850 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:59.841057+0000", 2024-09-13T11:25:59.850 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.3937999999999999e-05 2024-09-13T11:25:59.850 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:59.850 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:25:59.850 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:59.850 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:25:59.850 INFO:teuthology.orchestra.run.smithi092.stdout: ], 2024-09-13T11:25:59.851 INFO:teuthology.orchestra.run.smithi092.stdout: "num_ops": 2 2024-09-13T11:25:59.851 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-09-13T11:25:59.858 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-13T11:25:59.897 INFO:tasks.ceph.osd.8.smithi001.stderr:2024-09-13T11:25:59.894+0000 7fb9b48ae640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26312) UID: 0 2024-09-13T11:25:59.990 INFO:tasks.ceph.osd.12.smithi001.stderr:2024-09-13T11:25:59.986+0000 7f891f10f640 -1 osd.12 187 handle_osd_map injecting map crc failure 2024-09-13T11:25:59.992 INFO:tasks.ceph.osd.4.smithi001.stderr:2024-09-13T11:25:59.990+0000 7ff8d0c3d640 -1 osd.4 187 handle_osd_map injecting map crc failure 2024-09-13T11:26:00.002 INFO:tasks.ceph.osd.4.smithi001.stderr:2024-09-13T11:25:59.994+0000 7ff8e0218640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26310) UID: 0 2024-09-13T11:26:00.023 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-09-13T11:26:00.023 INFO:teuthology.orchestra.run.smithi092.stdout: "ops": [], 2024-09-13T11:26:00.024 INFO:teuthology.orchestra.run.smithi092.stdout: "complaint_time": 30, 2024-09-13T11:26:00.024 INFO:teuthology.orchestra.run.smithi092.stdout: "num_blocked_ops": 0 2024-09-13T11:26:00.024 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-09-13T11:26:00.039 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-13T11:26:00.100 INFO:tasks.ceph.osd.4.smithi001.stderr:2024-09-13T11:26:00.098+0000 7ff8e0218640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26310) UID: 0 2024-09-13T11:26:00.181 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-09-13T11:26:00.181 INFO:teuthology.orchestra.run.smithi092.stdout: "size": 20, 2024-09-13T11:26:00.181 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 600, 2024-09-13T11:26:00.181 INFO:teuthology.orchestra.run.smithi092.stdout: "ops": [ 2024-09-13T11:26:00.181 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.181 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "MOSDPGPush(3.23 83/78 [PushOp(3:c61f6bab:::benchmark_data_smithi177_31116_object29479:head, version: 59'1860, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:c61f6bab:::benchmark_data_smithi177_31116_object29479:head@59'1860, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:26:00.181 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:40.770864+0000", 2024-09-13T11:26:00.181 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 139.40810836099999, 2024-09-13T11:26:00.181 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.31229662800000002, 2024-09-13T11:26:00.182 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:26:00.182 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:26:00.182 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-09-13T11:26:00.182 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:26:00.182 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.182 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:26:00.182 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:40.770864+0000", 2024-09-13T11:26:00.182 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:00.182 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.183 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.183 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:26:00.183 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:40.770864+0000", 2024-09-13T11:26:00.183 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:00.183 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.183 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.183 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:26:00.183 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:40.770865+0000", 2024-09-13T11:26:00.183 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.288e-06 2024-09-13T11:26:00.183 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.183 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.184 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:26:00.184 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:40.770955+0000", 2024-09-13T11:26:00.184 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.9871999999999996e-05 2024-09-13T11:26:00.184 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.184 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.184 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:26:00.184 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:41.082105+0000", 2024-09-13T11:26:00.184 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.31114963400000001 2024-09-13T11:26:00.184 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.184 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.185 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:26:00.185 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:41.082136+0000", 2024-09-13T11:26:00.185 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.0972000000000001e-05 2024-09-13T11:26:00.185 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.185 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.185 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:00.185 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:41.082404+0000", 2024-09-13T11:26:00.185 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00026844800000000002 2024-09-13T11:26:00.185 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.185 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.186 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:26:00.186 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:41.082429+0000", 2024-09-13T11:26:00.186 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.5021e-05 2024-09-13T11:26:00.186 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.186 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.186 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:26:00.186 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:41.083161+0000", 2024-09-13T11:26:00.186 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000731393 2024-09-13T11:26:00.186 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:00.186 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:00.186 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:00.187 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.187 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.187 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:110289 3.29 3.27287969 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e119)", 2024-09-13T11:26:00.187 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:24:25.774634+0000", 2024-09-13T11:26:00.187 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 94.404338285999998, 2024-09-13T11:26:00.187 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.31721633999999999, 2024-09-13T11:26:00.187 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:26:00.187 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:26:00.187 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "reached pg", 2024-09-13T11:26:00.187 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:26:00.187 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:26:00.188 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:00.188 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 110289 2024-09-13T11:26:00.188 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.188 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:26:00.188 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.188 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:26:00.188 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:25.774634+0000", 2024-09-13T11:26:00.188 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:00.188 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.188 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.189 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:26:00.189 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:25.774634+0000", 2024-09-13T11:26:00.189 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:00.189 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.189 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.189 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:26:00.189 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:25.774639+0000", 2024-09-13T11:26:00.189 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.6029999999999998e-06 2024-09-13T11:26:00.189 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.189 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.189 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:26:00.190 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:25.774695+0000", 2024-09-13T11:26:00.190 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.6694999999999998e-05 2024-09-13T11:26:00.190 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.190 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.190 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:26:00.190 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:25.774728+0000", 2024-09-13T11:26:00.190 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.2502000000000002e-05 2024-09-13T11:26:00.190 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.190 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.190 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:26:00.191 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:25.774750+0000", 2024-09-13T11:26:00.191 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.2002999999999999e-05 2024-09-13T11:26:00.191 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.191 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.191 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:00.191 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:25.774942+0000", 2024-09-13T11:26:00.191 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000192265 2024-09-13T11:26:00.191 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.191 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.191 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-09-13T11:26:00.192 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:25.774960+0000", 2024-09-13T11:26:00.192 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.8275999999999999e-05 2024-09-13T11:26:00.192 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.192 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.192 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:00.192 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:26.091821+0000", 2024-09-13T11:26:00.192 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.31686065000000002 2024-09-13T11:26:00.192 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.192 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.193 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:26:00.193 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:26.091850+0000", 2024-09-13T11:26:00.193 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.9346e-05 2024-09-13T11:26:00.193 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:00.193 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:00.193 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:00.193 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.193 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.193 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "MOSDPGPush(3.22 125/122 [PushOp(3:44e4f9f2:::benchmark_data_smithi177_31116_object16663:head, version: 52'1032, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:44e4f9f2:::benchmark_data_smithi177_31116_object16663:head@52'1032, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:26:00.194 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:24:32.743254+0000", 2024-09-13T11:26:00.194 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 87.435718197, 2024-09-13T11:26:00.194 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.215485287, 2024-09-13T11:26:00.194 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:26:00.194 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:26:00.194 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-09-13T11:26:00.194 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:26:00.194 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.194 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:26:00.194 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:32.743254+0000", 2024-09-13T11:26:00.195 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:00.195 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.195 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.195 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:26:00.195 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:32.743254+0000", 2024-09-13T11:26:00.195 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:00.195 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.195 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.195 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:26:00.196 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:32.743255+0000", 2024-09-13T11:26:00.196 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.2500000000000004e-07 2024-09-13T11:26:00.196 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.196 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.196 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:26:00.196 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:32.743310+0000", 2024-09-13T11:26:00.196 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.5442000000000002e-05 2024-09-13T11:26:00.196 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.196 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.197 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:26:00.197 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:32.957754+0000", 2024-09-13T11:26:00.197 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.21444387000000001 2024-09-13T11:26:00.197 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.197 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.197 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:26:00.197 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:32.957765+0000", 2024-09-13T11:26:00.197 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0458999999999999e-05 2024-09-13T11:26:00.197 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.198 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.198 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:00.198 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:32.957841+0000", 2024-09-13T11:26:00.198 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.5993000000000002e-05 2024-09-13T11:26:00.198 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.198 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.198 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:26:00.198 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:32.957852+0000", 2024-09-13T11:26:00.198 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.1453e-05 2024-09-13T11:26:00.199 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.199 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.199 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:26:00.199 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:32.958739+0000", 2024-09-13T11:26:00.199 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00088724500000000005 2024-09-13T11:26:00.199 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:00.199 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:00.199 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:00.199 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.199 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.200 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:138370 3.23 3:c5730cd2:::benchmark_data_smithi177_31116_object138369:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e140)", 2024-09-13T11:26:00.200 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:24:54.415176+0000", 2024-09-13T11:26:00.200 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 65.763796189999994, 2024-09-13T11:26:00.200 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.24549895499999999, 2024-09-13T11:26:00.200 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:26:00.200 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:26:00.200 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:00.200 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:26:00.200 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:26:00.200 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:00.201 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 138370 2024-09-13T11:26:00.201 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.201 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:26:00.201 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.201 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:26:00.201 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:54.415176+0000", 2024-09-13T11:26:00.201 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:00.201 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.201 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.201 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:26:00.202 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:54.415176+0000", 2024-09-13T11:26:00.202 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:00.202 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.202 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.202 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:26:00.202 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:54.415180+0000", 2024-09-13T11:26:00.202 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.1670000000000001e-06 2024-09-13T11:26:00.202 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.202 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.202 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:26:00.203 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:54.415269+0000", 2024-09-13T11:26:00.203 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.8659000000000006e-05 2024-09-13T11:26:00.203 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.203 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.203 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:26:00.203 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:54.415292+0000", 2024-09-13T11:26:00.203 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.3152000000000001e-05 2024-09-13T11:26:00.203 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.203 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.203 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:26:00.204 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:54.415304+0000", 2024-09-13T11:26:00.204 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.1841999999999999e-05 2024-09-13T11:26:00.204 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.204 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.204 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:00.204 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:54.415432+0000", 2024-09-13T11:26:00.204 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00012802000000000001 2024-09-13T11:26:00.204 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.204 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.204 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:26:00.205 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:54.415710+0000", 2024-09-13T11:26:00.205 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00027840299999999998 2024-09-13T11:26:00.205 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.205 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.205 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 9", 2024-09-13T11:26:00.205 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:54.415950+0000", 2024-09-13T11:26:00.205 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00023928299999999999 2024-09-13T11:26:00.205 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.205 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.205 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:26:00.205 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:54.419265+0000", 2024-09-13T11:26:00.206 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0033150440000000001 2024-09-13T11:26:00.206 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.206 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.206 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:00.206 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:54.660448+0000", 2024-09-13T11:26:00.206 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.24118348100000001 2024-09-13T11:26:00.206 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.206 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.206 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:26:00.206 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:54.660571+0000", 2024-09-13T11:26:00.206 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00012332900000000001 2024-09-13T11:26:00.206 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.207 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.207 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:26:00.207 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:54.660675+0000", 2024-09-13T11:26:00.207 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000103575 2024-09-13T11:26:00.207 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:00.207 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:00.207 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:00.207 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.207 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.207 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:140742 3.1e 3:79c694a0:::benchmark_data_smithi177_31116_object439:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e144)", 2024-09-13T11:26:00.208 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:24:58.321262+0000", 2024-09-13T11:26:00.208 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 61.857709773000003, 2024-09-13T11:26:00.208 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.23727013499999999, 2024-09-13T11:26:00.208 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:26:00.208 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:26:00.208 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:00.208 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:26:00.208 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:26:00.208 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:00.208 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 140742 2024-09-13T11:26:00.209 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.209 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:26:00.209 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.209 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:26:00.209 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:58.321262+0000", 2024-09-13T11:26:00.209 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:00.209 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.209 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.209 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:26:00.210 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:58.321262+0000", 2024-09-13T11:26:00.210 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:00.210 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.210 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.210 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:26:00.210 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:58.321264+0000", 2024-09-13T11:26:00.210 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.581e-06 2024-09-13T11:26:00.210 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.210 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.210 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:26:00.210 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:58.321272+0000", 2024-09-13T11:26:00.211 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.2179999999999999e-06 2024-09-13T11:26:00.211 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.211 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.211 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:26:00.211 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:58.321282+0000", 2024-09-13T11:26:00.211 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0173000000000001e-05 2024-09-13T11:26:00.211 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.211 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.211 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:26:00.212 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:58.321290+0000", 2024-09-13T11:26:00.212 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.943e-06 2024-09-13T11:26:00.212 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.212 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.212 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:00.212 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:58.321355+0000", 2024-09-13T11:26:00.212 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.4461000000000005e-05 2024-09-13T11:26:00.212 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.212 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.212 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:26:00.212 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:58.321450+0000", 2024-09-13T11:26:00.213 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.4769e-05 2024-09-13T11:26:00.213 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.213 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.213 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 4", 2024-09-13T11:26:00.213 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:58.321516+0000", 2024-09-13T11:26:00.213 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.5998000000000001e-05 2024-09-13T11:26:00.213 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.213 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.213 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:26:00.213 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:58.324045+0000", 2024-09-13T11:26:00.214 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0025290579999999998 2024-09-13T11:26:00.214 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.214 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.214 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:00.214 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:58.558244+0000", 2024-09-13T11:26:00.214 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.234199455 2024-09-13T11:26:00.214 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.214 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.214 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:26:00.214 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:58.558401+0000", 2024-09-13T11:26:00.215 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00015700800000000001 2024-09-13T11:26:00.215 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.215 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.215 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:26:00.215 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:58.558533+0000", 2024-09-13T11:26:00.215 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000131471 2024-09-13T11:26:00.215 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:00.215 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:00.215 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:00.216 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.216 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.216 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:140796 3.1e 3:7987a652:::benchmark_data_smithi177_31116_object493:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e144)", 2024-09-13T11:26:00.216 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:24:58.552900+0000", 2024-09-13T11:26:00.216 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 61.626072237999999, 2024-09-13T11:26:00.216 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.254397398, 2024-09-13T11:26:00.216 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:26:00.216 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:26:00.216 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:00.216 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:26:00.217 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:26:00.217 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:00.217 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 140796 2024-09-13T11:26:00.217 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.217 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:26:00.217 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.217 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:26:00.217 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:58.552900+0000", 2024-09-13T11:26:00.217 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:00.217 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.218 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.218 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:26:00.218 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:58.552900+0000", 2024-09-13T11:26:00.218 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:00.218 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.218 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.218 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:26:00.218 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:58.552904+0000", 2024-09-13T11:26:00.218 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.76e-06 2024-09-13T11:26:00.218 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.218 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.219 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:26:00.219 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:58.552915+0000", 2024-09-13T11:26:00.219 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0763e-05 2024-09-13T11:26:00.219 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.219 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.219 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:26:00.219 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:58.552943+0000", 2024-09-13T11:26:00.219 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.8469000000000001e-05 2024-09-13T11:26:00.219 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.220 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.220 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:26:00.220 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:58.552959+0000", 2024-09-13T11:26:00.220 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.5549999999999999e-05 2024-09-13T11:26:00.220 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.220 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.220 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:00.220 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:58.553106+0000", 2024-09-13T11:26:00.220 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00014729500000000001 2024-09-13T11:26:00.220 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.221 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.221 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:26:00.221 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:58.553348+0000", 2024-09-13T11:26:00.221 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000242353 2024-09-13T11:26:00.221 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.221 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.221 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 4", 2024-09-13T11:26:00.221 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:58.553556+0000", 2024-09-13T11:26:00.221 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00020765700000000001 2024-09-13T11:26:00.221 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.222 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.222 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:26:00.222 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:58.557140+0000", 2024-09-13T11:26:00.222 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0035844840000000002 2024-09-13T11:26:00.222 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.222 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.222 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:00.222 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:58.807126+0000", 2024-09-13T11:26:00.222 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.24998542000000001 2024-09-13T11:26:00.222 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.222 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.223 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:26:00.223 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:58.807214+0000", 2024-09-13T11:26:00.223 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.8561999999999997e-05 2024-09-13T11:26:00.223 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.223 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.223 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:26:00.223 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:58.807297+0000", 2024-09-13T11:26:00.223 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.3084999999999995e-05 2024-09-13T11:26:00.223 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:00.223 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:00.224 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:00.224 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.224 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.224 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop(client.4665.0:162073 3.1f e156/122 3:fd1a00ae:::benchmark_data_smithi177_31116_object21770:head v 156'6684, mlcod=156'6679)", 2024-09-13T11:26:00.224 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:25:15.455533+0000", 2024-09-13T11:26:00.224 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 44.723438870000003, 2024-09-13T11:26:00.224 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.24315087499999999, 2024-09-13T11:26:00.224 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:26:00.224 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:26:00.224 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:00.224 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:26:00.225 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.225 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:26:00.225 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:15.455533+0000", 2024-09-13T11:26:00.225 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:00.225 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.225 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.225 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:26:00.225 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:15.455533+0000", 2024-09-13T11:26:00.225 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:00.225 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.226 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.226 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:26:00.226 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:15.455535+0000", 2024-09-13T11:26:00.226 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.42e-06 2024-09-13T11:26:00.226 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.226 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.226 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:26:00.226 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:15.455542+0000", 2024-09-13T11:26:00.226 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.9820000000000002e-06 2024-09-13T11:26:00.226 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.226 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.226 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:26:00.227 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:15.696032+0000", 2024-09-13T11:26:00.227 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.240490483 2024-09-13T11:26:00.227 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.227 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.227 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:26:00.227 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:15.696039+0000", 2024-09-13T11:26:00.227 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.832e-06 2024-09-13T11:26:00.227 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.227 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.227 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:00.228 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:15.696098+0000", 2024-09-13T11:26:00.228 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.8901999999999999e-05 2024-09-13T11:26:00.228 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.228 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.228 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:26:00.228 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:15.696115+0000", 2024-09-13T11:26:00.228 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.7479999999999999e-05 2024-09-13T11:26:00.228 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.228 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.228 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:26:00.228 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:15.698637+0000", 2024-09-13T11:26:00.229 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0025215889999999999 2024-09-13T11:26:00.229 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.229 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.229 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:26:00.229 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:15.698684+0000", 2024-09-13T11:26:00.229 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.7187000000000001e-05 2024-09-13T11:26:00.229 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:00.229 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:00.229 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:00.229 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.229 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.230 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop(client.4665.0:162102 3.1f e156/122 3:fc0cda6c:::benchmark_data_smithi177_31116_object21799:head v 156'6685, mlcod=156'6680)", 2024-09-13T11:26:00.230 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:25:15.477994+0000", 2024-09-13T11:26:00.230 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 44.700978069000001, 2024-09-13T11:26:00.230 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.25110996400000002, 2024-09-13T11:26:00.230 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:26:00.230 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:26:00.230 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:00.230 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:26:00.230 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.230 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:26:00.231 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:15.477994+0000", 2024-09-13T11:26:00.231 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:00.231 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.231 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.231 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:26:00.231 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:15.477994+0000", 2024-09-13T11:26:00.231 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:00.231 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.231 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.231 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:26:00.231 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:15.477995+0000", 2024-09-13T11:26:00.232 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0210000000000001e-06 2024-09-13T11:26:00.232 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.232 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.232 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:26:00.232 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:15.478001+0000", 2024-09-13T11:26:00.232 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.0020000000000003e-06 2024-09-13T11:26:00.232 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.232 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.232 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:26:00.232 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:15.726384+0000", 2024-09-13T11:26:00.232 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.248382609 2024-09-13T11:26:00.233 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.233 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.233 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:26:00.233 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:15.726390+0000", 2024-09-13T11:26:00.233 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.1029999999999998e-06 2024-09-13T11:26:00.233 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.233 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.233 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:00.233 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:15.727107+0000", 2024-09-13T11:26:00.233 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000717525 2024-09-13T11:26:00.233 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.234 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.234 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:26:00.234 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:15.727130+0000", 2024-09-13T11:26:00.234 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.2572999999999999e-05 2024-09-13T11:26:00.234 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.234 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.234 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:26:00.234 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:15.729066+0000", 2024-09-13T11:26:00.234 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0019362170000000001 2024-09-13T11:26:00.234 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.234 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.234 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:26:00.235 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:15.729104+0000", 2024-09-13T11:26:00.235 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.7914000000000003e-05 2024-09-13T11:26:00.235 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:00.235 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:00.235 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:00.235 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.235 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.236 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "MOSDPGPush(3.28 168/120 [PushOp(3:14eb8ed9:::benchmark_data_smithi177_31116_object87538:head, version: 110'4071, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:14eb8ed9:::benchmark_data_smithi177_31116_object87538:head@110'4071, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:26:00.236 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:25:27.274802+0000", 2024-09-13T11:26:00.236 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 32.904170520999998, 2024-09-13T11:26:00.236 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.369268239, 2024-09-13T11:26:00.236 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:26:00.236 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:26:00.236 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-09-13T11:26:00.236 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:26:00.236 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.236 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:26:00.237 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:27.274802+0000", 2024-09-13T11:26:00.237 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:00.237 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.237 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.237 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:26:00.237 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:27.274802+0000", 2024-09-13T11:26:00.237 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:00.237 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.237 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.237 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:26:00.238 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:27.274803+0000", 2024-09-13T11:26:00.238 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.7579999999999999e-06 2024-09-13T11:26:00.238 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.238 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.238 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:26:00.238 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:27.274831+0000", 2024-09-13T11:26:00.238 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.7464e-05 2024-09-13T11:26:00.238 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.238 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.238 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:26:00.239 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:27.641355+0000", 2024-09-13T11:26:00.239 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.36652425199999999 2024-09-13T11:26:00.239 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.239 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.239 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:26:00.239 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:27.641385+0000", 2024-09-13T11:26:00.239 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.9433000000000001e-05 2024-09-13T11:26:00.239 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.239 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.239 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:00.240 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:27.641611+0000", 2024-09-13T11:26:00.240 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00022646299999999999 2024-09-13T11:26:00.240 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.240 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.240 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:26:00.240 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:27.641638+0000", 2024-09-13T11:26:00.240 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.6482000000000002e-05 2024-09-13T11:26:00.240 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.240 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.240 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:26:00.240 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:27.644070+0000", 2024-09-13T11:26:00.241 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0024323869999999998 2024-09-13T11:26:00.241 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:00.241 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:00.241 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:00.241 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.241 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.241 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "MOSDPGPush(3.2d 174/120 [PushOp(3:b42ee771:::benchmark_data_smithi177_31116_object109317:head, version: 118'5087, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:b42ee771:::benchmark_data_smithi177_31116_object109317:head@118'5087, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:26:00.241 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:25:36.282270+0000", 2024-09-13T11:26:00.241 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 23.896701904, 2024-09-13T11:26:00.241 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.26768705100000001, 2024-09-13T11:26:00.241 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:26:00.242 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:26:00.242 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-09-13T11:26:00.242 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:26:00.242 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.242 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:26:00.242 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:36.282270+0000", 2024-09-13T11:26:00.242 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:00.242 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.242 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.242 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:26:00.242 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:36.282270+0000", 2024-09-13T11:26:00.243 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:00.243 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.243 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.243 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:26:00.243 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:36.282272+0000", 2024-09-13T11:26:00.243 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.832e-06 2024-09-13T11:26:00.243 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.243 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.243 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:26:00.243 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:36.282337+0000", 2024-09-13T11:26:00.243 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.4834999999999998e-05 2024-09-13T11:26:00.244 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.244 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.244 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:26:00.244 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:36.547263+0000", 2024-09-13T11:26:00.244 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.26492590599999999 2024-09-13T11:26:00.244 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.244 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.244 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:26:00.244 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:36.547288+0000", 2024-09-13T11:26:00.244 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.4661999999999999e-05 2024-09-13T11:26:00.245 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.245 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.245 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:00.245 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:36.547493+0000", 2024-09-13T11:26:00.245 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00020533500000000001 2024-09-13T11:26:00.245 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.245 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.245 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:26:00.245 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:36.547515+0000", 2024-09-13T11:26:00.245 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.2124999999999999e-05 2024-09-13T11:26:00.245 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.246 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.246 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:26:00.246 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:36.549957+0000", 2024-09-13T11:26:00.246 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0024423560000000001 2024-09-13T11:26:00.246 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:00.246 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:00.246 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:00.246 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.246 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.246 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:191021 3.23 3:c4df18dd:::benchmark_data_smithi177_31116_object50718:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e175)", 2024-09-13T11:26:00.247 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:25:38.460411+0000", 2024-09-13T11:26:00.247 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 21.718561095999998, 2024-09-13T11:26:00.247 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.40138001400000001, 2024-09-13T11:26:00.247 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:26:00.247 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:26:00.247 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:00.247 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:26:00.247 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:26:00.247 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:00.247 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 191021 2024-09-13T11:26:00.248 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.248 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:26:00.248 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.248 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:26:00.248 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:38.460411+0000", 2024-09-13T11:26:00.248 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:00.248 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.248 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.248 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:26:00.248 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:38.460411+0000", 2024-09-13T11:26:00.248 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:00.249 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.249 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.249 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:26:00.249 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:38.460413+0000", 2024-09-13T11:26:00.249 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.075e-06 2024-09-13T11:26:00.249 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.249 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.249 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:26:00.249 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:38.460419+0000", 2024-09-13T11:26:00.249 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.6869999999999999e-06 2024-09-13T11:26:00.250 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.250 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.250 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:26:00.250 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:38.460431+0000", 2024-09-13T11:26:00.250 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.2451e-05 2024-09-13T11:26:00.250 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.250 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.250 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:26:00.250 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:38.460440+0000", 2024-09-13T11:26:00.250 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.5140000000000001e-06 2024-09-13T11:26:00.250 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.251 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.251 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:00.251 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:38.460501+0000", 2024-09-13T11:26:00.251 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.0772e-05 2024-09-13T11:26:00.251 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.251 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.251 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:26:00.251 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:38.460632+0000", 2024-09-13T11:26:00.251 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00013175200000000001 2024-09-13T11:26:00.251 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.252 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.252 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 9", 2024-09-13T11:26:00.252 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:38.460731+0000", 2024-09-13T11:26:00.252 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.8647000000000006e-05 2024-09-13T11:26:00.252 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.252 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.252 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:26:00.252 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:38.462322+0000", 2024-09-13T11:26:00.252 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.001590658 2024-09-13T11:26:00.252 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.252 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.253 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:00.253 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:38.861634+0000", 2024-09-13T11:26:00.253 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.39931206499999999 2024-09-13T11:26:00.253 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.253 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.253 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:26:00.253 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:38.861719+0000", 2024-09-13T11:26:00.253 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.4968000000000004e-05 2024-09-13T11:26:00.253 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.253 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.253 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:26:00.254 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:38.861791+0000", 2024-09-13T11:26:00.254 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.2424999999999999e-05 2024-09-13T11:26:00.254 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:00.254 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:00.254 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:00.254 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.254 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.254 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop_reply(client.4665.0:191021 3.23 e175/90 ondisk, result = 0)", 2024-09-13T11:26:00.255 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:25:38.462483+0000", 2024-09-13T11:26:00.255 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 21.716489674000002, 2024-09-13T11:26:00.255 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.39932300900000001, 2024-09-13T11:26:00.255 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:26:00.255 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:26:00.255 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-09-13T11:26:00.255 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:26:00.255 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.255 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:26:00.256 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:38.462483+0000", 2024-09-13T11:26:00.256 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:00.256 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.256 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.256 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:26:00.256 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:38.462483+0000", 2024-09-13T11:26:00.256 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:00.256 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.256 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.256 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:26:00.257 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:38.462484+0000", 2024-09-13T11:26:00.257 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.1990000000000001e-06 2024-09-13T11:26:00.257 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.257 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.257 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:26:00.257 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:38.462488+0000", 2024-09-13T11:26:00.257 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.985e-06 2024-09-13T11:26:00.257 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.257 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.257 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:26:00.258 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:38.861475+0000", 2024-09-13T11:26:00.258 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.39898754800000003 2024-09-13T11:26:00.258 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.258 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.258 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:26:00.258 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:38.861488+0000", 2024-09-13T11:26:00.258 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.3037000000000001e-05 2024-09-13T11:26:00.258 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.258 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.258 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:00.258 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:38.861607+0000", 2024-09-13T11:26:00.259 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00011835 2024-09-13T11:26:00.259 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.259 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.259 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:26:00.259 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:38.861624+0000", 2024-09-13T11:26:00.259 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.7730000000000001e-05 2024-09-13T11:26:00.259 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.259 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.259 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:26:00.260 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:38.861806+0000", 2024-09-13T11:26:00.260 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00018116000000000001 2024-09-13T11:26:00.260 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:00.260 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:00.260 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:00.260 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.260 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.260 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:191079 3.29 3:949485de:::benchmark_data_smithi177_31116_object50776:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e175)", 2024-09-13T11:26:00.260 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:25:38.540455+0000", 2024-09-13T11:26:00.260 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 21.638517418999999, 2024-09-13T11:26:00.261 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.221851678, 2024-09-13T11:26:00.261 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:26:00.261 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:26:00.261 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:00.261 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:26:00.261 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:26:00.261 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:00.261 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 191079 2024-09-13T11:26:00.261 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.261 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:26:00.261 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.262 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:26:00.262 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:38.540455+0000", 2024-09-13T11:26:00.262 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:00.262 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.262 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.262 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:26:00.262 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:38.540455+0000", 2024-09-13T11:26:00.262 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:00.262 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.262 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.263 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:26:00.263 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:38.540456+0000", 2024-09-13T11:26:00.263 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.499e-06 2024-09-13T11:26:00.263 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.263 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.263 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:26:00.263 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:38.540461+0000", 2024-09-13T11:26:00.263 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.1359999999999996e-06 2024-09-13T11:26:00.263 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.263 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.264 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:26:00.264 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:38.540472+0000", 2024-09-13T11:26:00.264 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0225e-05 2024-09-13T11:26:00.264 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.264 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.264 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:26:00.264 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:38.540477+0000", 2024-09-13T11:26:00.264 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.6999999999999996e-06 2024-09-13T11:26:00.264 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.264 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.265 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:00.265 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:38.540527+0000", 2024-09-13T11:26:00.265 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.9611000000000003e-05 2024-09-13T11:26:00.265 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.265 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.265 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:26:00.265 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:38.540619+0000", 2024-09-13T11:26:00.265 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.2180000000000002e-05 2024-09-13T11:26:00.265 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.265 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.266 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 4", 2024-09-13T11:26:00.266 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:38.540689+0000", 2024-09-13T11:26:00.266 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.9511999999999994e-05 2024-09-13T11:26:00.266 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.266 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.266 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:26:00.266 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:38.542339+0000", 2024-09-13T11:26:00.266 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0016505370000000001 2024-09-13T11:26:00.266 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.266 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.266 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:00.267 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:38.762150+0000", 2024-09-13T11:26:00.267 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.219810585 2024-09-13T11:26:00.267 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.267 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.267 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:26:00.267 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:38.762237+0000", 2024-09-13T11:26:00.267 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.7577999999999994e-05 2024-09-13T11:26:00.267 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.267 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.267 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:26:00.268 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:38.762307+0000", 2024-09-13T11:26:00.268 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.9115000000000005e-05 2024-09-13T11:26:00.268 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:00.268 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:00.268 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:00.268 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.268 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.268 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:191081 3.1e 3:7e784823:::benchmark_data_smithi177_31116_object50778:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e175)", 2024-09-13T11:26:00.268 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:25:38.546903+0000", 2024-09-13T11:26:00.268 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 21.632068963000002, 2024-09-13T11:26:00.269 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.22503042600000001, 2024-09-13T11:26:00.269 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:26:00.269 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:26:00.269 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:00.269 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:26:00.269 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:26:00.269 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:00.269 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 191081 2024-09-13T11:26:00.269 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.269 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:26:00.269 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.270 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:26:00.270 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:38.546903+0000", 2024-09-13T11:26:00.270 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:00.270 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.270 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.270 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:26:00.270 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:38.546903+0000", 2024-09-13T11:26:00.270 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:00.270 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.270 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.271 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:26:00.271 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:38.546912+0000", 2024-09-13T11:26:00.271 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.0100000000000001e-06 2024-09-13T11:26:00.271 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.271 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.271 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:26:00.271 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:38.546922+0000", 2024-09-13T11:26:00.271 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.7200000000000001e-06 2024-09-13T11:26:00.271 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.271 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.272 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:26:00.272 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:38.546950+0000", 2024-09-13T11:26:00.272 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.8082e-05 2024-09-13T11:26:00.272 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.272 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.272 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:26:00.272 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:38.546965+0000", 2024-09-13T11:26:00.272 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.4664000000000001e-05 2024-09-13T11:26:00.272 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.272 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.273 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:00.273 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:38.547130+0000", 2024-09-13T11:26:00.273 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000165286 2024-09-13T11:26:00.273 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.273 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.273 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:26:00.273 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:38.547356+0000", 2024-09-13T11:26:00.273 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000225758 2024-09-13T11:26:00.273 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.273 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.274 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 4", 2024-09-13T11:26:00.274 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:38.547540+0000", 2024-09-13T11:26:00.274 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000183936 2024-09-13T11:26:00.274 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.274 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.274 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:26:00.274 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:38.551954+0000", 2024-09-13T11:26:00.274 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0044143710000000003 2024-09-13T11:26:00.274 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.274 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.275 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:00.275 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:38.771785+0000", 2024-09-13T11:26:00.275 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.219830936 2024-09-13T11:26:00.275 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.275 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.275 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:26:00.275 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:38.771868+0000", 2024-09-13T11:26:00.275 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.2707999999999995e-05 2024-09-13T11:26:00.275 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.275 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.276 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:26:00.276 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:38.771934+0000", 2024-09-13T11:26:00.276 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.5955000000000001e-05 2024-09-13T11:26:00.276 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:00.276 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:00.276 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:00.276 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.276 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.276 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:191087 3.1e 3:7a697202:::benchmark_data_smithi177_31116_object50784:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e175)", 2024-09-13T11:26:00.276 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:25:38.561241+0000", 2024-09-13T11:26:00.277 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 21.617730978000001, 2024-09-13T11:26:00.277 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.22700689399999999, 2024-09-13T11:26:00.277 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:26:00.277 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:26:00.277 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:00.277 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:26:00.277 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:26:00.277 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:00.277 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 191087 2024-09-13T11:26:00.277 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.278 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:26:00.278 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.278 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:26:00.278 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:38.561241+0000", 2024-09-13T11:26:00.278 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:00.278 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.278 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.278 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:26:00.278 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:38.561241+0000", 2024-09-13T11:26:00.278 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:00.279 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.279 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.279 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:26:00.279 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:38.561243+0000", 2024-09-13T11:26:00.279 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.7540000000000001e-06 2024-09-13T11:26:00.279 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.279 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.279 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:26:00.279 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:38.561248+0000", 2024-09-13T11:26:00.279 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.3229999999999997e-06 2024-09-13T11:26:00.280 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.280 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.280 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:26:00.280 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:38.561261+0000", 2024-09-13T11:26:00.280 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.2393e-05 2024-09-13T11:26:00.280 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.280 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.280 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:26:00.280 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:38.561267+0000", 2024-09-13T11:26:00.281 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.2060000000000004e-06 2024-09-13T11:26:00.281 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.281 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.281 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:00.281 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:38.561365+0000", 2024-09-13T11:26:00.281 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.8009000000000003e-05 2024-09-13T11:26:00.281 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.281 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.281 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:26:00.281 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:38.561621+0000", 2024-09-13T11:26:00.282 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00025571099999999999 2024-09-13T11:26:00.282 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.282 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.282 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 4", 2024-09-13T11:26:00.282 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:38.561808+0000", 2024-09-13T11:26:00.282 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000187393 2024-09-13T11:26:00.282 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.282 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.282 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:26:00.282 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:38.565928+0000", 2024-09-13T11:26:00.283 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0041195229999999999 2024-09-13T11:26:00.283 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.283 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.283 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:00.283 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:38.788094+0000", 2024-09-13T11:26:00.283 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.22216676799999999 2024-09-13T11:26:00.283 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.283 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.283 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:26:00.283 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:38.788180+0000", 2024-09-13T11:26:00.284 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.6127999999999999e-05 2024-09-13T11:26:00.284 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.284 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.284 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:26:00.284 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:38.788248+0000", 2024-09-13T11:26:00.284 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.7686000000000002e-05 2024-09-13T11:26:00.284 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:00.284 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:00.284 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:00.285 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.285 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.285 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:209500 3.27 3:e6941997:::benchmark_data_smithi177_31116_object69197:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e185)", 2024-09-13T11:26:00.285 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:25:53.962062+0000", 2024-09-13T11:26:00.285 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 6.2169099299999999, 2024-09-13T11:26:00.285 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.26007328699999999, 2024-09-13T11:26:00.285 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:26:00.285 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:26:00.285 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:00.285 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:26:00.286 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:26:00.286 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:00.286 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 209500 2024-09-13T11:26:00.286 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.286 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:26:00.286 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.286 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:26:00.286 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.962062+0000", 2024-09-13T11:26:00.286 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:00.286 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.286 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.287 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:26:00.287 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.962062+0000", 2024-09-13T11:26:00.287 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:00.287 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.287 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.287 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:26:00.287 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.962065+0000", 2024-09-13T11:26:00.287 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.9019999999999999e-06 2024-09-13T11:26:00.287 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.287 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.287 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:26:00.288 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.962074+0000", 2024-09-13T11:26:00.288 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.6030000000000005e-06 2024-09-13T11:26:00.288 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.288 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.288 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:26:00.288 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.962098+0000", 2024-09-13T11:26:00.288 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.4076000000000001e-05 2024-09-13T11:26:00.288 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.288 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.289 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:26:00.289 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.962114+0000", 2024-09-13T11:26:00.289 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.5820000000000001e-05 2024-09-13T11:26:00.289 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.289 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.289 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:00.289 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.962280+0000", 2024-09-13T11:26:00.289 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00016648300000000001 2024-09-13T11:26:00.289 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.289 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.290 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:26:00.290 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.962598+0000", 2024-09-13T11:26:00.290 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00031824000000000002 2024-09-13T11:26:00.290 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.290 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.290 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 11", 2024-09-13T11:26:00.290 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.962906+0000", 2024-09-13T11:26:00.290 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00030721699999999998 2024-09-13T11:26:00.290 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.290 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.291 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:26:00.291 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.965943+0000", 2024-09-13T11:26:00.291 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0030373010000000001 2024-09-13T11:26:00.291 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.291 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.291 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:00.291 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:54.222003+0000", 2024-09-13T11:26:00.291 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.25605972900000001 2024-09-13T11:26:00.291 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.291 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.292 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:26:00.292 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:54.222081+0000", 2024-09-13T11:26:00.292 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.8442000000000006e-05 2024-09-13T11:26:00.292 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.292 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.292 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:26:00.292 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:54.222136+0000", 2024-09-13T11:26:00.292 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.4474000000000001e-05 2024-09-13T11:26:00.292 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:00.292 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:00.293 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:00.293 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.293 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.293 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:209869 3.27 3:e6b1fcc5:::benchmark_data_smithi177_31116_object69566:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e185)", 2024-09-13T11:26:00.293 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:25:54.300263+0000", 2024-09-13T11:26:00.293 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 5.8787087769999999, 2024-09-13T11:26:00.293 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.30266981599999998, 2024-09-13T11:26:00.293 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:26:00.293 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:26:00.293 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:00.293 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:26:00.294 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:26:00.294 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:00.294 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 209869 2024-09-13T11:26:00.294 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.294 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:26:00.294 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.294 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:26:00.294 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:54.300263+0000", 2024-09-13T11:26:00.294 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:00.294 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.295 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.295 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:26:00.295 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:54.300263+0000", 2024-09-13T11:26:00.295 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:00.295 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.295 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.295 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:26:00.295 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:54.300267+0000", 2024-09-13T11:26:00.295 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.2200000000000001e-06 2024-09-13T11:26:00.295 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.296 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.296 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:26:00.296 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:54.300276+0000", 2024-09-13T11:26:00.296 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.3570000000000004e-06 2024-09-13T11:26:00.296 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.296 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.296 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:26:00.296 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:54.300301+0000", 2024-09-13T11:26:00.296 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.5191e-05 2024-09-13T11:26:00.296 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.297 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.297 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:26:00.297 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:54.300321+0000", 2024-09-13T11:26:00.297 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.9967999999999999e-05 2024-09-13T11:26:00.297 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.297 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.297 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:00.297 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:54.300427+0000", 2024-09-13T11:26:00.297 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000105781 2024-09-13T11:26:00.297 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.297 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.298 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:26:00.298 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:54.300590+0000", 2024-09-13T11:26:00.298 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00016341000000000001 2024-09-13T11:26:00.298 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.298 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.298 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 11", 2024-09-13T11:26:00.298 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:54.300724+0000", 2024-09-13T11:26:00.298 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000133323 2024-09-13T11:26:00.299 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.299 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.299 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:26:00.299 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:54.303373+0000", 2024-09-13T11:26:00.299 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0026488050000000002 2024-09-13T11:26:00.299 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.299 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.299 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:00.299 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:54.602650+0000", 2024-09-13T11:26:00.299 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.299277976 2024-09-13T11:26:00.300 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.300 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.300 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:26:00.300 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:54.602795+0000", 2024-09-13T11:26:00.300 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000144127 2024-09-13T11:26:00.300 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.300 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.300 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:26:00.300 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:54.602933+0000", 2024-09-13T11:26:00.300 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000138658 2024-09-13T11:26:00.301 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:00.301 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:00.301 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:00.301 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.301 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.301 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:217059 3.23 3:c741d888:::benchmark_data_smithi177_31116_object76756:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e187)", 2024-09-13T11:26:00.301 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:25:59.856369+0000", 2024-09-13T11:26:00.301 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 0.32260348900000002, 2024-09-13T11:26:00.301 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.22477011399999999, 2024-09-13T11:26:00.301 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:26:00.302 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:26:00.302 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:00.302 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:26:00.302 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:26:00.302 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:00.302 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 217059 2024-09-13T11:26:00.302 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.302 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:26:00.302 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.302 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:26:00.303 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:59.856369+0000", 2024-09-13T11:26:00.303 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:00.303 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.303 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.303 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:26:00.303 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:59.856369+0000", 2024-09-13T11:26:00.303 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:00.303 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.303 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.303 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:26:00.304 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:59.856372+0000", 2024-09-13T11:26:00.304 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.7690000000000001e-06 2024-09-13T11:26:00.304 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.304 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.304 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:26:00.304 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:59.856378+0000", 2024-09-13T11:26:00.304 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.0759999999999999e-06 2024-09-13T11:26:00.304 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.304 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.304 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:26:00.304 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:59.856391+0000", 2024-09-13T11:26:00.304 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.3485999999999999e-05 2024-09-13T11:26:00.305 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.305 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.305 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:26:00.305 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:59.856399+0000", 2024-09-13T11:26:00.305 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.0050000000000004e-06 2024-09-13T11:26:00.305 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.305 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.305 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:00.305 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:59.856460+0000", 2024-09-13T11:26:00.305 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.0469e-05 2024-09-13T11:26:00.305 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.306 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.306 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:26:00.306 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:59.856602+0000", 2024-09-13T11:26:00.306 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000142096 2024-09-13T11:26:00.306 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.306 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.306 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 9", 2024-09-13T11:26:00.306 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:59.856723+0000", 2024-09-13T11:26:00.306 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000121476 2024-09-13T11:26:00.306 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.307 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.307 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:26:00.307 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:59.859050+0000", 2024-09-13T11:26:00.307 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.002326519 2024-09-13T11:26:00.307 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.307 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.307 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:00.307 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:00.081000+0000", 2024-09-13T11:26:00.307 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.22195034599999999 2024-09-13T11:26:00.307 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.307 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.307 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:26:00.308 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:00.081078+0000", 2024-09-13T11:26:00.308 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.7849000000000006e-05 2024-09-13T11:26:00.308 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.308 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.308 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:26:00.308 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:00.081139+0000", 2024-09-13T11:26:00.308 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.1023000000000002e-05 2024-09-13T11:26:00.308 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:00.308 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:00.309 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:00.309 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.309 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.309 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop_reply(client.4665.0:217059 3.23 e187/90 ondisk, result = 0)", 2024-09-13T11:26:00.309 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:25:59.859090+0000", 2024-09-13T11:26:00.309 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 0.31988192300000001, 2024-09-13T11:26:00.309 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.222058691, 2024-09-13T11:26:00.309 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:26:00.309 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:26:00.309 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-09-13T11:26:00.310 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:26:00.310 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.310 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:26:00.310 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:59.859090+0000", 2024-09-13T11:26:00.310 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:00.310 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.310 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.310 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:26:00.310 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:59.859090+0000", 2024-09-13T11:26:00.310 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:00.310 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.311 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.311 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:26:00.311 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:59.859092+0000", 2024-09-13T11:26:00.311 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.375e-06 2024-09-13T11:26:00.311 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.311 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.311 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:26:00.311 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:59.859096+0000", 2024-09-13T11:26:00.311 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.69e-06 2024-09-13T11:26:00.311 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.311 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.312 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:26:00.312 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:00.080900+0000", 2024-09-13T11:26:00.312 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.221803625 2024-09-13T11:26:00.312 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.312 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.312 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:26:00.312 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:00.080908+0000", 2024-09-13T11:26:00.312 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.5610000000000001e-06 2024-09-13T11:26:00.312 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.312 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.313 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:00.313 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:00.080979+0000", 2024-09-13T11:26:00.313 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.1358000000000004e-05 2024-09-13T11:26:00.313 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.313 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.313 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:26:00.313 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:00.080991+0000", 2024-09-13T11:26:00.313 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.1987999999999999e-05 2024-09-13T11:26:00.313 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.313 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.314 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:26:00.314 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:00.081149+0000", 2024-09-13T11:26:00.314 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00015809400000000001 2024-09-13T11:26:00.314 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:00.314 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:00.314 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:00.314 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.314 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.314 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop(client.4665.0:217082 3.22 e187/122 3:449e4b87:::benchmark_data_smithi177_31116_object76779:head v 187'5666, mlcod=187'5664)", 2024-09-13T11:26:00.314 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:25:59.884121+0000", 2024-09-13T11:26:00.314 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 0.29485109700000001, 2024-09-13T11:26:00.315 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.25479380699999998, 2024-09-13T11:26:00.315 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:26:00.315 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:26:00.315 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:00.315 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:26:00.315 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.315 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:26:00.315 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:59.884121+0000", 2024-09-13T11:26:00.315 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:00.315 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.315 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.316 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:26:00.316 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:59.884121+0000", 2024-09-13T11:26:00.316 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:00.316 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.316 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.316 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:26:00.316 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:59.884123+0000", 2024-09-13T11:26:00.316 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.677e-06 2024-09-13T11:26:00.316 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.316 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.316 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:26:00.317 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:59.884129+0000", 2024-09-13T11:26:00.317 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.669e-06 2024-09-13T11:26:00.317 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.317 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.317 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:26:00.317 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:00.137020+0000", 2024-09-13T11:26:00.317 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.25289069199999997 2024-09-13T11:26:00.317 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.317 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.317 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:26:00.317 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:00.137030+0000", 2024-09-13T11:26:00.318 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.8600000000000005e-06 2024-09-13T11:26:00.318 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.318 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.318 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:00.318 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:00.137123+0000", 2024-09-13T11:26:00.318 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.2842000000000003e-05 2024-09-13T11:26:00.318 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.318 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.318 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:26:00.318 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:00.137139+0000", 2024-09-13T11:26:00.319 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.6388e-05 2024-09-13T11:26:00.319 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.319 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.319 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:26:00.319 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:00.138876+0000", 2024-09-13T11:26:00.319 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0017370809999999999 2024-09-13T11:26:00.319 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.319 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.319 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:26:00.319 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:00.138915+0000", 2024-09-13T11:26:00.319 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.8597999999999999e-05 2024-09-13T11:26:00.320 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:00.320 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:00.320 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:00.320 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:00.320 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:00.320 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-09-13T11:26:00.322 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-13T11:26:00.318+0000 7f6a68775640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26307) UID: 0 2024-09-13T11:26:00.322 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-09-13T11:26:00.422 INFO:tasks.ceph.osd.7.smithi177.stderr:2024-09-13T11:26:00.417+0000 7f897b51d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26575) UID: 0 2024-09-13T11:26:00.468 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-09-13T11:26:00.468 INFO:teuthology.orchestra.run.smithi092.stdout: "ops": [ 2024-09-13T11:26:00.468 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.469 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:217641 3.19 3:9cb30276:::benchmark_data_smithi177_31116_object77338:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e188)", 2024-09-13T11:26:00.469 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:26:00.456080+0000", 2024-09-13T11:26:00.469 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 0.01233685, 2024-09-13T11:26:00.469 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.012356948, 2024-09-13T11:26:00.469 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:26:00.469 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:26:00.469 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "waiting for sub ops", 2024-09-13T11:26:00.469 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:26:00.469 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:26:00.469 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:00.469 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 217641 2024-09-13T11:26:00.469 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.469 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:26:00.469 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.470 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:26:00.470 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:00.456080+0000", 2024-09-13T11:26:00.470 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:00.470 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.470 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.470 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:26:00.470 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:00.456080+0000", 2024-09-13T11:26:00.470 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:00.470 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.470 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.470 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:26:00.470 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:00.456082+0000", 2024-09-13T11:26:00.470 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.671e-06 2024-09-13T11:26:00.471 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.471 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.471 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:26:00.471 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:00.456087+0000", 2024-09-13T11:26:00.471 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.2730000000000002e-06 2024-09-13T11:26:00.471 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.471 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.471 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:26:00.471 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:00.456100+0000", 2024-09-13T11:26:00.471 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.2768999999999999e-05 2024-09-13T11:26:00.471 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.471 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.471 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:26:00.471 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:00.456107+0000", 2024-09-13T11:26:00.471 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.9670000000000004e-06 2024-09-13T11:26:00.472 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.472 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.472 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:00.472 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:00.456648+0000", 2024-09-13T11:26:00.472 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00054067500000000001 2024-09-13T11:26:00.472 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.472 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.472 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:26:00.472 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:00.456809+0000", 2024-09-13T11:26:00.472 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000161471 2024-09-13T11:26:00.472 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.472 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.472 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1", 2024-09-13T11:26:00.473 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:00.456890+0000", 2024-09-13T11:26:00.473 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.1420000000000003e-05 2024-09-13T11:26:00.473 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.473 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.473 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:26:00.473 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:00.460304+0000", 2024-09-13T11:26:00.473 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0034140329999999999 2024-09-13T11:26:00.473 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:00.473 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:00.473 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:00.473 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.473 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.474 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:217659 3.24 3:273cc24e:::benchmark_data_smithi177_31116_object77356:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e188)", 2024-09-13T11:26:00.474 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:26:00.466231+0000", 2024-09-13T11:26:00.474 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 0.002186257, 2024-09-13T11:26:00.474 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.002274706, 2024-09-13T11:26:00.474 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:26:00.474 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:26:00.474 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "waiting for sub ops", 2024-09-13T11:26:00.474 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:26:00.474 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:26:00.474 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:00.474 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 217659 2024-09-13T11:26:00.474 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.474 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:26:00.474 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.474 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:26:00.475 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:00.466231+0000", 2024-09-13T11:26:00.475 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:00.475 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.475 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.475 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:26:00.475 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:00.466231+0000", 2024-09-13T11:26:00.475 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:00.475 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.475 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.475 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:26:00.475 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:00.466233+0000", 2024-09-13T11:26:00.475 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.4140000000000001e-06 2024-09-13T11:26:00.475 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.475 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.476 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:26:00.476 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:00.466240+0000", 2024-09-13T11:26:00.476 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.4219999999999997e-06 2024-09-13T11:26:00.476 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.476 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.476 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:26:00.476 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:00.466255+0000", 2024-09-13T11:26:00.476 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.5398000000000001e-05 2024-09-13T11:26:00.476 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.476 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.476 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:26:00.476 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:00.466263+0000", 2024-09-13T11:26:00.476 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.3089999999999996e-06 2024-09-13T11:26:00.477 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.477 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.477 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:00.477 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:00.466338+0000", 2024-09-13T11:26:00.477 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.4398000000000002e-05 2024-09-13T11:26:00.477 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.477 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.477 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:26:00.477 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:00.466445+0000", 2024-09-13T11:26:00.477 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000107667 2024-09-13T11:26:00.477 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.477 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.477 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1", 2024-09-13T11:26:00.477 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:00.466528+0000", 2024-09-13T11:26:00.478 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.2823000000000003e-05 2024-09-13T11:26:00.478 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:00.478 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:00.478 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:00.478 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:00.478 INFO:teuthology.orchestra.run.smithi092.stdout: ], 2024-09-13T11:26:00.478 INFO:teuthology.orchestra.run.smithi092.stdout: "num_ops": 2 2024-09-13T11:26:00.478 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-09-13T11:26:00.484 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-09-13T11:26:00.523 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-13T11:26:00.522+0000 7f6a68775640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26307) UID: 0 2024-09-13T11:26:00.622 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-09-13T11:26:00.623 INFO:teuthology.orchestra.run.smithi092.stdout: "ops": [], 2024-09-13T11:26:00.623 INFO:teuthology.orchestra.run.smithi092.stdout: "complaint_time": 30, 2024-09-13T11:26:00.623 INFO:teuthology.orchestra.run.smithi092.stdout: "num_blocked_ops": 0 2024-09-13T11:26:00.623 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-09-13T11:26:00.624 INFO:tasks.ceph.osd.12.smithi001.stderr:2024-09-13T11:26:00.622+0000 7f892c6e6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26314) UID: 0 2024-09-13T11:26:00.638 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-09-13T11:26:00.725 INFO:tasks.ceph.osd.14.smithi162.stderr:2024-09-13T11:26:00.723+0000 7f3df5903640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33154) UID: 0 2024-09-13T11:26:00.780 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-09-13T11:26:00.780 INFO:teuthology.orchestra.run.smithi092.stdout: "size": 20, 2024-09-13T11:26:00.780 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 600, 2024-09-13T11:26:00.780 INFO:teuthology.orchestra.run.smithi092.stdout: "ops": [ 2024-09-13T11:26:00.781 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.781 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "MOSDPGPush(3.19 76/70 [PushOp(3:9eacf3d0:::benchmark_data_smithi177_31116_object19747:head, version: 54'1236, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:9eacf3d0:::benchmark_data_smithi177_31116_object19747:head@54'1236, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:26:00.781 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:32.704676+0000", 2024-09-13T11:26:00.781 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 148.073547039, 2024-09-13T11:26:00.781 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.23525014899999999, 2024-09-13T11:26:00.781 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:26:00.781 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:26:00.781 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-09-13T11:26:00.781 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:26:00.781 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.781 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:26:00.781 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:32.704676+0000", 2024-09-13T11:26:00.782 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:00.782 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.782 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.782 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:26:00.782 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:32.704676+0000", 2024-09-13T11:26:00.782 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:00.782 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.782 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.782 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:26:00.782 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:32.704677+0000", 2024-09-13T11:26:00.782 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.531e-06 2024-09-13T11:26:00.782 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.782 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.783 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:26:00.783 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:32.704749+0000", 2024-09-13T11:26:00.783 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.1451000000000004e-05 2024-09-13T11:26:00.783 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.783 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.783 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:26:00.783 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:32.938053+0000", 2024-09-13T11:26:00.783 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.233304752 2024-09-13T11:26:00.783 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.783 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.783 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:26:00.783 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:32.938087+0000", 2024-09-13T11:26:00.783 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.4016000000000001e-05 2024-09-13T11:26:00.784 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.784 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.784 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:00.784 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:32.938351+0000", 2024-09-13T11:26:00.784 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00026397500000000001 2024-09-13T11:26:00.784 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.784 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.784 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:26:00.784 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:32.938381+0000", 2024-09-13T11:26:00.784 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.0001999999999998e-05 2024-09-13T11:26:00.784 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.784 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.784 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:26:00.785 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:32.939926+0000", 2024-09-13T11:26:00.785 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0015444219999999999 2024-09-13T11:26:00.785 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:00.785 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:00.785 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:00.785 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.785 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.785 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "MOSDPGPush(3.23 80/78 [PushOp(3:c59ecc6d:::benchmark_data_smithi177_31116_object17005:head, version: 52'1054, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:c59ecc6d:::benchmark_data_smithi177_31116_object17005:head@52'1054, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:26:00.785 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:37.433040+0000", 2024-09-13T11:26:00.785 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 143.34518306300001, 2024-09-13T11:26:00.785 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.26658219100000002, 2024-09-13T11:26:00.785 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:26:00.785 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:26:00.785 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-09-13T11:26:00.786 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:26:00.786 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.786 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:26:00.786 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:37.433040+0000", 2024-09-13T11:26:00.786 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:00.786 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.786 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.786 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:26:00.786 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:37.433040+0000", 2024-09-13T11:26:00.786 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:00.786 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.786 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.786 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:26:00.787 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:37.433042+0000", 2024-09-13T11:26:00.787 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.3999999999999999e-06 2024-09-13T11:26:00.787 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.787 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.787 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:26:00.787 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:37.433090+0000", 2024-09-13T11:26:00.787 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.8065999999999997e-05 2024-09-13T11:26:00.787 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.787 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.787 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:26:00.787 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:37.698797+0000", 2024-09-13T11:26:00.787 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.26570674500000002 2024-09-13T11:26:00.787 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.788 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.788 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:26:00.788 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:37.698827+0000", 2024-09-13T11:26:00.788 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.0505e-05 2024-09-13T11:26:00.788 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.788 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.788 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:00.788 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:37.698941+0000", 2024-09-13T11:26:00.788 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00011403799999999999 2024-09-13T11:26:00.788 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.788 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.788 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:26:00.788 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:37.698956+0000", 2024-09-13T11:26:00.788 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.4765e-05 2024-09-13T11:26:00.789 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.789 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.789 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:26:00.789 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:37.699622+0000", 2024-09-13T11:26:00.789 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00066567199999999997 2024-09-13T11:26:00.789 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:00.789 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:00.789 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:00.789 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.789 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.789 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "MOSDPGPush(3.1a 85/78 [PushOp(3:5dc85525:::benchmark_data_smithi177_31116_object11755:head, version: 49'723, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:5dc85525:::benchmark_data_smithi177_31116_object11755:head@49'723, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:26:00.789 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:42.422061+0000", 2024-09-13T11:26:00.790 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 138.35616160999999, 2024-09-13T11:26:00.790 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.52875617500000005, 2024-09-13T11:26:00.790 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:26:00.790 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:26:00.790 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-09-13T11:26:00.790 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:26:00.790 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.790 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:26:00.790 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.422061+0000", 2024-09-13T11:26:00.790 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:00.790 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.790 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.790 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:26:00.790 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.422061+0000", 2024-09-13T11:26:00.791 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:00.791 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.791 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.791 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:26:00.791 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.422064+0000", 2024-09-13T11:26:00.791 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.5409999999999999e-06 2024-09-13T11:26:00.791 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.791 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.791 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:26:00.791 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.422139+0000", 2024-09-13T11:26:00.791 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.5488999999999995e-05 2024-09-13T11:26:00.791 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.792 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.792 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:26:00.792 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.948781+0000", 2024-09-13T11:26:00.792 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.52664230999999995 2024-09-13T11:26:00.792 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.792 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.792 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:26:00.792 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.948841+0000", 2024-09-13T11:26:00.792 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.9450999999999997e-05 2024-09-13T11:26:00.792 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.792 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.792 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:00.793 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.949035+0000", 2024-09-13T11:26:00.793 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00019380600000000001 2024-09-13T11:26:00.793 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.793 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.793 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:26:00.793 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.949060+0000", 2024-09-13T11:26:00.793 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.5381999999999999e-05 2024-09-13T11:26:00.793 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.793 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.793 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:26:00.793 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.950817+0000", 2024-09-13T11:26:00.793 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0017571959999999999 2024-09-13T11:26:00.793 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:00.794 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:00.794 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:00.794 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.794 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.794 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "MOSDPGPush(3.1a 89/78 [PushOp(3:59270734:::benchmark_data_smithi177_31116_object20039:head, version: 54'1256, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:59270734:::benchmark_data_smithi177_31116_object20039:head@54'1256, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:26:00.794 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:46.607562+0000", 2024-09-13T11:26:00.794 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 134.170660833, 2024-09-13T11:26:00.794 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.32123945300000001, 2024-09-13T11:26:00.794 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:26:00.794 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:26:00.794 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-09-13T11:26:00.794 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:26:00.794 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.795 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:26:00.795 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.607562+0000", 2024-09-13T11:26:00.795 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:00.795 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.795 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.795 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:26:00.795 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.607562+0000", 2024-09-13T11:26:00.795 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:00.795 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.795 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.795 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:26:00.795 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.607564+0000", 2024-09-13T11:26:00.795 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.8610000000000001e-06 2024-09-13T11:26:00.795 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.796 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.796 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:26:00.796 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.607663+0000", 2024-09-13T11:26:00.796 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.9210999999999996e-05 2024-09-13T11:26:00.796 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.796 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.796 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:26:00.796 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.925193+0000", 2024-09-13T11:26:00.796 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.317530586 2024-09-13T11:26:00.796 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.796 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.796 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:26:00.796 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.925250+0000", 2024-09-13T11:26:00.797 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.6925000000000001e-05 2024-09-13T11:26:00.797 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.797 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.797 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:00.797 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.927013+0000", 2024-09-13T11:26:00.797 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.001762853 2024-09-13T11:26:00.797 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.797 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.797 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:26:00.797 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.927067+0000", 2024-09-13T11:26:00.797 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.3601999999999999e-05 2024-09-13T11:26:00.797 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.797 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.798 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:26:00.798 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.928801+0000", 2024-09-13T11:26:00.798 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.001734415 2024-09-13T11:26:00.798 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:00.798 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:00.798 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:00.798 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.798 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.798 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "MOSDPGPush(3.1a 91/78 [PushOp(3:5e5010d2:::benchmark_data_smithi177_31116_object25733:head, version: 58'1609, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:5e5010d2:::benchmark_data_smithi177_31116_object25733:head@58'1609, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:26:00.798 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:48.608485+0000", 2024-09-13T11:26:00.798 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 132.169737715, 2024-09-13T11:26:00.798 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.35035599699999997, 2024-09-13T11:26:00.798 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:26:00.799 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:26:00.799 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-09-13T11:26:00.799 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:26:00.799 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.799 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:26:00.799 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:48.608485+0000", 2024-09-13T11:26:00.799 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:00.799 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.799 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.799 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:26:00.799 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:48.608485+0000", 2024-09-13T11:26:00.799 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:00.799 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.800 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.800 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:26:00.800 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:48.608488+0000", 2024-09-13T11:26:00.800 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.9399999999999998e-06 2024-09-13T11:26:00.800 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.800 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.800 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:26:00.800 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:48.608552+0000", 2024-09-13T11:26:00.800 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.3731000000000004e-05 2024-09-13T11:26:00.800 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.800 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.800 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:26:00.800 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:48.957330+0000", 2024-09-13T11:26:00.800 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.34877806099999997 2024-09-13T11:26:00.801 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.801 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.801 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:26:00.801 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:48.957361+0000", 2024-09-13T11:26:00.801 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.0852000000000002e-05 2024-09-13T11:26:00.801 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.801 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.801 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:00.801 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:48.957544+0000", 2024-09-13T11:26:00.801 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000183537 2024-09-13T11:26:00.801 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.801 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.801 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:26:00.802 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:48.957582+0000", 2024-09-13T11:26:00.802 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.7472999999999998e-05 2024-09-13T11:26:00.802 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.802 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.802 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:26:00.802 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:48.958841+0000", 2024-09-13T11:26:00.802 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0012594030000000001 2024-09-13T11:26:00.802 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:00.802 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:00.802 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:00.802 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.802 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.802 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop(client.4665.0:72923 3.23 e97/90 3:c6230123:::benchmark_data_smithi177_31116_object72922:head v 97'3418, mlcod=97'3414)", 2024-09-13T11:26:00.802 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:55.047734+0000", 2024-09-13T11:26:00.803 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 125.730488198, 2024-09-13T11:26:00.803 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.20075528000000001, 2024-09-13T11:26:00.803 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:26:00.803 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:26:00.803 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:00.803 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:26:00.803 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.803 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:26:00.803 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:55.047734+0000", 2024-09-13T11:26:00.803 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:00.803 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.803 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.804 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:26:00.804 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:55.047734+0000", 2024-09-13T11:26:00.804 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:00.804 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.804 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.804 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:26:00.804 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:55.047736+0000", 2024-09-13T11:26:00.804 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.466e-06 2024-09-13T11:26:00.804 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.804 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.804 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:26:00.804 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:55.047772+0000", 2024-09-13T11:26:00.804 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.6576999999999999e-05 2024-09-13T11:26:00.804 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.805 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.805 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:26:00.805 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:55.246478+0000", 2024-09-13T11:26:00.805 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.19870581300000001 2024-09-13T11:26:00.805 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.805 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.805 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:26:00.805 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:55.246488+0000", 2024-09-13T11:26:00.805 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.5419999999999994e-06 2024-09-13T11:26:00.805 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.805 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.805 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:00.806 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:55.246631+0000", 2024-09-13T11:26:00.806 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00014287300000000001 2024-09-13T11:26:00.806 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.806 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.806 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:26:00.806 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:55.246665+0000", 2024-09-13T11:26:00.806 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.4684999999999997e-05 2024-09-13T11:26:00.806 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.806 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.806 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:26:00.806 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:55.248453+0000", 2024-09-13T11:26:00.806 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0017879459999999999 2024-09-13T11:26:00.806 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.807 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.807 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:26:00.807 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:55.248490+0000", 2024-09-13T11:26:00.807 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.6378000000000003e-05 2024-09-13T11:26:00.807 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:00.807 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:00.807 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:00.807 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.807 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.807 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:79823 3.19 3:9e038f3c:::benchmark_data_smithi177_31116_object79822:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102)", 2024-09-13T11:26:00.807 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:24:00.563188+0000", 2024-09-13T11:26:00.807 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 120.215034573, 2024-09-13T11:26:00.807 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.164156422, 2024-09-13T11:26:00.807 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:26:00.808 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:26:00.808 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:00.808 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:26:00.808 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:26:00.808 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:00.808 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 79823 2024-09-13T11:26:00.808 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.808 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:26:00.808 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.808 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:26:00.808 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:00.563188+0000", 2024-09-13T11:26:00.808 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:00.808 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.809 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.809 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:26:00.809 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:00.563188+0000", 2024-09-13T11:26:00.809 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:00.809 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.809 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.809 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:26:00.809 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:00.563201+0000", 2024-09-13T11:26:00.809 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.3183000000000001e-05 2024-09-13T11:26:00.809 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.809 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.809 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:26:00.809 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:00.563263+0000", 2024-09-13T11:26:00.809 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.1904999999999998e-05 2024-09-13T11:26:00.810 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.810 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.810 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:26:00.810 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:00.563297+0000", 2024-09-13T11:26:00.810 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.3979000000000001e-05 2024-09-13T11:26:00.810 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.810 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.810 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:26:00.810 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:00.563318+0000", 2024-09-13T11:26:00.810 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.0703000000000002e-05 2024-09-13T11:26:00.810 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.810 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.810 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:00.810 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:00.563513+0000", 2024-09-13T11:26:00.811 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00019524200000000001 2024-09-13T11:26:00.811 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.811 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.811 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:26:00.811 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:00.563758+0000", 2024-09-13T11:26:00.811 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000244711 2024-09-13T11:26:00.811 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.811 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.811 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1", 2024-09-13T11:26:00.811 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:00.563978+0000", 2024-09-13T11:26:00.811 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00022011700000000001 2024-09-13T11:26:00.811 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.812 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.812 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:26:00.812 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:00.566893+0000", 2024-09-13T11:26:00.812 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0029147650000000002 2024-09-13T11:26:00.812 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.812 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.812 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:00.812 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:00.727176+0000", 2024-09-13T11:26:00.812 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.16028374100000001 2024-09-13T11:26:00.812 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.812 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.812 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:26:00.812 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:00.727271+0000", 2024-09-13T11:26:00.813 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.4481000000000006e-05 2024-09-13T11:26:00.813 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.813 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.813 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:26:00.813 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:00.727344+0000", 2024-09-13T11:26:00.813 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.3595000000000003e-05 2024-09-13T11:26:00.813 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:00.813 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:00.813 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:00.813 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.813 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.813 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop(client.4665.0:98582 3.1a e113/105 3:59f9cd75:::benchmark_data_smithi177_31116_object98581:head v 113'4743, mlcod=113'4741)", 2024-09-13T11:26:00.813 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:24:15.432027+0000", 2024-09-13T11:26:00.814 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 105.346195583, 2024-09-13T11:26:00.814 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.196879784, 2024-09-13T11:26:00.814 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:26:00.814 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:26:00.814 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:00.814 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:26:00.814 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.814 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:26:00.814 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:15.432027+0000", 2024-09-13T11:26:00.814 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:00.814 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.814 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.814 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:26:00.814 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:15.432027+0000", 2024-09-13T11:26:00.815 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:00.815 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.815 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.815 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:26:00.815 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:15.432029+0000", 2024-09-13T11:26:00.815 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.838e-06 2024-09-13T11:26:00.815 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.815 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.815 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:26:00.815 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:15.432088+0000", 2024-09-13T11:26:00.815 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.8767999999999997e-05 2024-09-13T11:26:00.815 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.815 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.815 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:26:00.816 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:15.624731+0000", 2024-09-13T11:26:00.816 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.192643221 2024-09-13T11:26:00.816 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.816 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.816 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:26:00.816 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:15.624746+0000", 2024-09-13T11:26:00.816 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.4705000000000001e-05 2024-09-13T11:26:00.816 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.816 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.816 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:00.816 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:15.624882+0000", 2024-09-13T11:26:00.816 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00013602300000000001 2024-09-13T11:26:00.816 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.817 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.817 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:26:00.817 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:15.624941+0000", 2024-09-13T11:26:00.817 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.9351000000000001e-05 2024-09-13T11:26:00.817 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.817 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.817 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:26:00.817 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:15.628836+0000", 2024-09-13T11:26:00.817 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0038955109999999999 2024-09-13T11:26:00.817 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.817 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.817 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:26:00.817 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:15.628907+0000", 2024-09-13T11:26:00.818 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.0366999999999999e-05 2024-09-13T11:26:00.818 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:00.818 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:00.818 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:00.818 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.818 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.818 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop(client.4665.0:99105 3.1a e113/105 3:5ac5090c:::benchmark_data_smithi177_31116_object99104:head v 113'4749, mlcod=113'4746)", 2024-09-13T11:26:00.818 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:24:15.963602+0000", 2024-09-13T11:26:00.818 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 104.814620401, 2024-09-13T11:26:00.818 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.23699906300000001, 2024-09-13T11:26:00.818 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:26:00.818 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:26:00.818 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:00.818 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:26:00.819 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.819 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:26:00.819 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:15.963602+0000", 2024-09-13T11:26:00.819 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:00.819 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.819 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.819 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:26:00.819 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:15.963602+0000", 2024-09-13T11:26:00.819 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:00.819 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.819 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.819 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:26:00.819 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:15.963605+0000", 2024-09-13T11:26:00.820 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.0400000000000001e-06 2024-09-13T11:26:00.820 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.820 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.820 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:26:00.820 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:15.963718+0000", 2024-09-13T11:26:00.820 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000112796 2024-09-13T11:26:00.820 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.820 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.820 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:26:00.820 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:16.197994+0000", 2024-09-13T11:26:00.820 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.234276285 2024-09-13T11:26:00.820 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.820 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.821 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:26:00.821 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:16.198004+0000", 2024-09-13T11:26:00.821 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.5559999999999996e-06 2024-09-13T11:26:00.821 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.821 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.821 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:00.821 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:16.198110+0000", 2024-09-13T11:26:00.821 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00010614 2024-09-13T11:26:00.821 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.821 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.821 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:26:00.821 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:16.198152+0000", 2024-09-13T11:26:00.821 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.1899e-05 2024-09-13T11:26:00.822 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.822 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.822 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:26:00.822 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:16.200528+0000", 2024-09-13T11:26:00.822 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00237587 2024-09-13T11:26:00.822 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.822 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.822 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:26:00.822 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:16.200601+0000", 2024-09-13T11:26:00.822 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.3477000000000001e-05 2024-09-13T11:26:00.822 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:00.822 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:00.822 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:00.822 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.822 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.823 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop(client.4665.0:129921 3.1a e134/105 3:5fb9088b:::benchmark_data_smithi177_31116_object129920:head v 134'5758, mlcod=134'5755)", 2024-09-13T11:26:00.823 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:24:47.178021+0000", 2024-09-13T11:26:00.823 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 73.600201394999999, 2024-09-13T11:26:00.823 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.33723278600000001, 2024-09-13T11:26:00.823 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:26:00.823 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:26:00.823 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:00.823 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:26:00.823 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.823 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:26:00.823 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:47.178021+0000", 2024-09-13T11:26:00.823 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:00.823 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.824 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.824 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:26:00.824 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:47.178021+0000", 2024-09-13T11:26:00.824 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:00.824 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.824 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.824 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:26:00.824 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:47.178024+0000", 2024-09-13T11:26:00.824 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.3609999999999999e-06 2024-09-13T11:26:00.824 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.824 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.824 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:26:00.825 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:47.178102+0000", 2024-09-13T11:26:00.825 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.8325000000000006e-05 2024-09-13T11:26:00.825 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.825 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.825 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:26:00.825 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:47.512707+0000", 2024-09-13T11:26:00.825 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.33460516000000001 2024-09-13T11:26:00.825 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.825 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.825 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:26:00.825 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:47.512716+0000", 2024-09-13T11:26:00.825 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.0389999999999993e-06 2024-09-13T11:26:00.825 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.826 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.826 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:00.826 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:47.512788+0000", 2024-09-13T11:26:00.826 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.1840000000000003e-05 2024-09-13T11:26:00.826 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.826 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.826 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:26:00.826 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:47.512809+0000", 2024-09-13T11:26:00.826 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.1005e-05 2024-09-13T11:26:00.826 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.826 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.826 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:26:00.826 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:47.515182+0000", 2024-09-13T11:26:00.826 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0023731939999999999 2024-09-13T11:26:00.827 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.827 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.827 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:26:00.827 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:47.515254+0000", 2024-09-13T11:26:00.827 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.1861999999999997e-05 2024-09-13T11:26:00.827 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:00.827 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:00.827 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:00.827 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.827 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.827 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop(client.4665.0:138370 3.23 e140/90 3:c5730cd2:::benchmark_data_smithi177_31116_object138369:head v 140'4458, mlcod=140'4455)", 2024-09-13T11:26:00.827 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:24:54.416118+0000", 2024-09-13T11:26:00.827 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 66.362104916999996, 2024-09-13T11:26:00.827 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.244024991, 2024-09-13T11:26:00.828 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:26:00.828 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:26:00.828 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:00.828 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:26:00.828 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.828 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:26:00.828 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:54.416118+0000", 2024-09-13T11:26:00.828 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:00.828 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.828 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.828 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:26:00.828 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:54.416118+0000", 2024-09-13T11:26:00.828 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:00.829 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.829 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.829 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:26:00.829 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:54.416120+0000", 2024-09-13T11:26:00.829 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.6280000000000001e-06 2024-09-13T11:26:00.829 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.829 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.829 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:26:00.829 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:54.416161+0000", 2024-09-13T11:26:00.829 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.0877000000000002e-05 2024-09-13T11:26:00.829 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.829 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.829 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:26:00.830 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:54.657128+0000", 2024-09-13T11:26:00.830 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.24096696300000001 2024-09-13T11:26:00.830 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.830 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.830 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:26:00.830 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:54.657142+0000", 2024-09-13T11:26:00.830 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.358e-05 2024-09-13T11:26:00.830 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.830 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.830 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:00.830 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:54.657290+0000", 2024-09-13T11:26:00.830 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00014794400000000001 2024-09-13T11:26:00.830 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.830 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.830 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:26:00.831 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:54.657325+0000", 2024-09-13T11:26:00.831 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.5623000000000001e-05 2024-09-13T11:26:00.831 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.831 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.831 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:26:00.831 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:54.660079+0000", 2024-09-13T11:26:00.831 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0027536489999999999 2024-09-13T11:26:00.831 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.831 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.831 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:26:00.831 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:54.660143+0000", 2024-09-13T11:26:00.831 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.3726999999999995e-05 2024-09-13T11:26:00.831 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:00.831 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:00.832 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:00.832 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.832 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.832 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop(client.4665.0:168782 3.1a e161/105 3:590b82e1:::benchmark_data_smithi177_31116_object28479:head v 161'6914, mlcod=161'6910)", 2024-09-13T11:26:00.832 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:25:20.643397+0000", 2024-09-13T11:26:00.832 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 40.134825665000001, 2024-09-13T11:26:00.832 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.17239671500000001, 2024-09-13T11:26:00.832 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:26:00.832 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:26:00.832 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:00.832 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:26:00.832 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.832 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:26:00.833 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:20.643397+0000", 2024-09-13T11:26:00.833 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:00.833 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.833 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.833 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:26:00.833 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:20.643397+0000", 2024-09-13T11:26:00.833 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:00.833 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.833 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.833 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:26:00.833 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:20.643399+0000", 2024-09-13T11:26:00.833 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.807e-06 2024-09-13T11:26:00.833 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.833 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.834 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:26:00.834 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:20.643410+0000", 2024-09-13T11:26:00.834 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0966e-05 2024-09-13T11:26:00.834 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.834 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.834 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:26:00.834 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:20.811600+0000", 2024-09-13T11:26:00.834 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.16819072400000001 2024-09-13T11:26:00.834 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.834 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.834 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:26:00.834 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:20.811613+0000", 2024-09-13T11:26:00.834 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.2166e-05 2024-09-13T11:26:00.834 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.835 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.835 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:00.835 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:20.811710+0000", 2024-09-13T11:26:00.835 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.7501e-05 2024-09-13T11:26:00.835 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.835 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.835 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:26:00.835 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:20.811727+0000", 2024-09-13T11:26:00.835 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.6645000000000001e-05 2024-09-13T11:26:00.835 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.835 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.835 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:26:00.835 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:20.815721+0000", 2024-09-13T11:26:00.835 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0039940260000000003 2024-09-13T11:26:00.836 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.836 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.836 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:26:00.836 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:20.815794+0000", 2024-09-13T11:26:00.836 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.2879999999999993e-05 2024-09-13T11:26:00.836 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:00.836 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:00.836 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:00.836 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.836 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.836 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop(client.4665.0:174971 3.1a e166/105 3:5a8f64c5:::benchmark_data_smithi177_31116_object34668:head v 166'7101, mlcod=166'7098)", 2024-09-13T11:26:00.836 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:25:25.423119+0000", 2024-09-13T11:26:00.836 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 35.355103477999997, 2024-09-13T11:26:00.836 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.18331898799999999, 2024-09-13T11:26:00.837 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:26:00.837 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:26:00.837 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:00.837 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:26:00.837 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.837 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:26:00.837 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:25.423119+0000", 2024-09-13T11:26:00.837 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:00.837 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.837 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.837 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:26:00.837 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:25.423119+0000", 2024-09-13T11:26:00.837 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:00.837 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.838 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.838 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:26:00.838 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:25.423120+0000", 2024-09-13T11:26:00.838 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.184e-06 2024-09-13T11:26:00.838 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.838 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.838 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:26:00.838 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:25.423127+0000", 2024-09-13T11:26:00.838 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.7139999999999996e-06 2024-09-13T11:26:00.838 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.838 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.838 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:26:00.838 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:25.604412+0000", 2024-09-13T11:26:00.838 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.18128530200000001 2024-09-13T11:26:00.839 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.839 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.839 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:26:00.839 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:25.604418+0000", 2024-09-13T11:26:00.839 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.6099999999999997e-06 2024-09-13T11:26:00.839 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.839 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.839 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:00.839 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:25.604467+0000", 2024-09-13T11:26:00.839 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.8729000000000001e-05 2024-09-13T11:26:00.839 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.839 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.839 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:26:00.839 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:25.604482+0000", 2024-09-13T11:26:00.840 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.5182e-05 2024-09-13T11:26:00.840 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.840 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.840 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:26:00.840 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:25.606402+0000", 2024-09-13T11:26:00.840 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0019203530000000001 2024-09-13T11:26:00.840 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.840 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.840 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:26:00.840 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:25.606438+0000", 2024-09-13T11:26:00.840 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.5914000000000002e-05 2024-09-13T11:26:00.840 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:00.840 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:00.841 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:00.841 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.841 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.841 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:209225 3.24 3:2478ed54:::benchmark_data_smithi177_31116_object68922:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e185)", 2024-09-13T11:26:00.841 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:25:53.632417+0000", 2024-09-13T11:26:00.841 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 7.1458054840000003, 2024-09-13T11:26:00.841 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.219758709, 2024-09-13T11:26:00.841 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:26:00.841 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:26:00.841 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:00.841 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:26:00.841 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:26:00.841 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:00.842 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 209225 2024-09-13T11:26:00.842 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.842 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:26:00.842 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.842 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:26:00.842 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.632417+0000", 2024-09-13T11:26:00.842 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:00.842 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.842 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.842 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:26:00.842 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.632417+0000", 2024-09-13T11:26:00.842 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:00.842 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.842 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.843 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:26:00.843 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.632420+0000", 2024-09-13T11:26:00.843 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.4980000000000001e-06 2024-09-13T11:26:00.843 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.843 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.843 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:26:00.843 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.632427+0000", 2024-09-13T11:26:00.843 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.5120000000000002e-06 2024-09-13T11:26:00.843 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.843 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.843 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:26:00.843 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.632441+0000", 2024-09-13T11:26:00.843 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.4072e-05 2024-09-13T11:26:00.844 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.844 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.844 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:26:00.844 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.632454+0000", 2024-09-13T11:26:00.844 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.2886e-05 2024-09-13T11:26:00.844 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.844 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.844 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:00.844 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.632529+0000", 2024-09-13T11:26:00.844 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.5326000000000004e-05 2024-09-13T11:26:00.844 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.844 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.844 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:26:00.844 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.632642+0000", 2024-09-13T11:26:00.845 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00011249200000000001 2024-09-13T11:26:00.845 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.845 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.845 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1", 2024-09-13T11:26:00.845 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.632712+0000", 2024-09-13T11:26:00.845 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.0376000000000005e-05 2024-09-13T11:26:00.845 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.845 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.845 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:26:00.845 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.635270+0000", 2024-09-13T11:26:00.845 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0025574339999999999 2024-09-13T11:26:00.845 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.845 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.846 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:00.846 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.851915+0000", 2024-09-13T11:26:00.846 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.21664515400000001 2024-09-13T11:26:00.846 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.846 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.846 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:26:00.846 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.852065+0000", 2024-09-13T11:26:00.846 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00014972799999999999 2024-09-13T11:26:00.846 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.846 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.846 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:26:00.846 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.852176+0000", 2024-09-13T11:26:00.846 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00011123100000000001 2024-09-13T11:26:00.846 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:00.847 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:00.847 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:00.847 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.847 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.847 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:209229 3.19 3:9a976dcb:::benchmark_data_smithi177_31116_object68926:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e185)", 2024-09-13T11:26:00.847 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:25:53.633903+0000", 2024-09-13T11:26:00.847 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 7.1443195790000003, 2024-09-13T11:26:00.847 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.25298130200000002, 2024-09-13T11:26:00.847 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:26:00.847 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:26:00.847 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:00.847 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:26:00.847 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:26:00.847 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:00.848 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 209229 2024-09-13T11:26:00.848 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.848 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:26:00.848 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.848 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:26:00.848 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.633903+0000", 2024-09-13T11:26:00.848 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:00.848 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.848 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.848 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:26:00.848 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.633903+0000", 2024-09-13T11:26:00.848 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:00.848 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.849 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.849 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:26:00.849 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.633905+0000", 2024-09-13T11:26:00.849 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.1600000000000001e-06 2024-09-13T11:26:00.849 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.849 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.849 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:26:00.849 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.633917+0000", 2024-09-13T11:26:00.849 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.1881e-05 2024-09-13T11:26:00.849 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.849 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.849 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:26:00.849 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.633931+0000", 2024-09-13T11:26:00.850 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.432e-05 2024-09-13T11:26:00.850 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.850 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.850 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:26:00.850 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.633941+0000", 2024-09-13T11:26:00.850 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.8079999999999999e-06 2024-09-13T11:26:00.850 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.850 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.850 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:00.850 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.634009+0000", 2024-09-13T11:26:00.850 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.7895e-05 2024-09-13T11:26:00.850 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.850 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.851 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:26:00.851 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.634137+0000", 2024-09-13T11:26:00.851 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000127644 2024-09-13T11:26:00.851 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.851 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.851 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1", 2024-09-13T11:26:00.851 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.634245+0000", 2024-09-13T11:26:00.851 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00010776399999999999 2024-09-13T11:26:00.851 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.851 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.851 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:26:00.851 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.637244+0000", 2024-09-13T11:26:00.851 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0029993149999999998 2024-09-13T11:26:00.851 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.851 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.852 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:00.852 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.886675+0000", 2024-09-13T11:26:00.852 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.249431456 2024-09-13T11:26:00.852 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.852 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.852 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:26:00.852 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.886785+0000", 2024-09-13T11:26:00.852 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000109326 2024-09-13T11:26:00.852 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.852 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.852 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:26:00.852 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.886884+0000", 2024-09-13T11:26:00.852 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.9733000000000002e-05 2024-09-13T11:26:00.853 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:00.853 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:00.853 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:00.853 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.853 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.853 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:209252 3.24 3:25ab37ee:::benchmark_data_smithi177_31116_object68949:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e185)", 2024-09-13T11:26:00.853 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:25:53.647942+0000", 2024-09-13T11:26:00.853 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 7.1302805070000002, 2024-09-13T11:26:00.853 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.21571831799999999, 2024-09-13T11:26:00.853 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:26:00.853 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:26:00.853 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:00.853 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:26:00.854 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:26:00.854 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:00.854 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 209252 2024-09-13T11:26:00.854 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.854 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:26:00.854 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.854 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:26:00.854 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.647942+0000", 2024-09-13T11:26:00.854 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:00.854 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.854 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.854 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:26:00.854 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.647942+0000", 2024-09-13T11:26:00.854 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:00.855 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.855 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.855 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:26:00.855 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.647945+0000", 2024-09-13T11:26:00.855 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.717e-06 2024-09-13T11:26:00.855 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.855 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.855 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:26:00.855 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.647951+0000", 2024-09-13T11:26:00.855 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.353e-06 2024-09-13T11:26:00.855 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.855 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.855 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:26:00.855 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.647966+0000", 2024-09-13T11:26:00.856 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.4877e-05 2024-09-13T11:26:00.856 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.856 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.856 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:26:00.856 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.647973+0000", 2024-09-13T11:26:00.856 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.2640000000000002e-06 2024-09-13T11:26:00.856 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.856 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.856 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:00.856 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.648038+0000", 2024-09-13T11:26:00.856 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.4366e-05 2024-09-13T11:26:00.856 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.856 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.857 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:26:00.857 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.648131+0000", 2024-09-13T11:26:00.857 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.3017999999999996e-05 2024-09-13T11:26:00.857 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.857 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.857 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1", 2024-09-13T11:26:00.857 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.648200+0000", 2024-09-13T11:26:00.857 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.9244000000000003e-05 2024-09-13T11:26:00.857 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.857 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.857 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:26:00.857 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.649817+0000", 2024-09-13T11:26:00.857 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0016166920000000001 2024-09-13T11:26:00.857 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.858 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.858 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:00.858 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.863329+0000", 2024-09-13T11:26:00.858 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.21351207699999999 2024-09-13T11:26:00.858 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.858 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.858 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:26:00.858 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.863524+0000", 2024-09-13T11:26:00.858 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00019551 2024-09-13T11:26:00.858 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.858 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.858 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:26:00.858 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.863660+0000", 2024-09-13T11:26:00.859 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0001362 2024-09-13T11:26:00.859 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:00.859 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:00.859 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:00.859 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.859 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.859 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:209264 3.19 3:9fb0ed8f:::benchmark_data_smithi177_31116_object68961:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e185)", 2024-09-13T11:26:00.859 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:25:53.655043+0000", 2024-09-13T11:26:00.859 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 7.1231795939999998, 2024-09-13T11:26:00.859 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.232156005, 2024-09-13T11:26:00.859 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:26:00.859 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:26:00.859 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:00.859 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:26:00.860 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:26:00.860 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:00.860 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 209264 2024-09-13T11:26:00.860 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.860 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:26:00.860 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.860 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:26:00.860 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.655043+0000", 2024-09-13T11:26:00.860 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:00.860 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.860 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.860 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:26:00.860 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.655043+0000", 2024-09-13T11:26:00.861 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:00.861 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.861 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.861 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:26:00.861 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.655045+0000", 2024-09-13T11:26:00.861 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.7090000000000001e-06 2024-09-13T11:26:00.861 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.861 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.861 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:26:00.861 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.655051+0000", 2024-09-13T11:26:00.861 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.3239999999999999e-06 2024-09-13T11:26:00.861 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.861 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.861 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:26:00.862 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.655065+0000", 2024-09-13T11:26:00.862 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.4137e-05 2024-09-13T11:26:00.862 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.862 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.862 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:26:00.862 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.655072+0000", 2024-09-13T11:26:00.862 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.827e-06 2024-09-13T11:26:00.862 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.862 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.862 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:00.862 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.655133+0000", 2024-09-13T11:26:00.862 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.0912000000000002e-05 2024-09-13T11:26:00.862 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.863 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.863 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:26:00.863 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.655228+0000", 2024-09-13T11:26:00.863 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.5440999999999996e-05 2024-09-13T11:26:00.863 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.863 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.863 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1", 2024-09-13T11:26:00.863 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.655296+0000", 2024-09-13T11:26:00.863 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.7316000000000004e-05 2024-09-13T11:26:00.863 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.863 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.863 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:26:00.863 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.658618+0000", 2024-09-13T11:26:00.863 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0033226179999999998 2024-09-13T11:26:00.864 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.864 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.864 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:00.864 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.887043+0000", 2024-09-13T11:26:00.864 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.22842430699999999 2024-09-13T11:26:00.864 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.864 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.864 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:26:00.864 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.887132+0000", 2024-09-13T11:26:00.864 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.9635000000000005e-05 2024-09-13T11:26:00.864 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.864 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.864 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:26:00.864 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.887199+0000", 2024-09-13T11:26:00.865 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.6779000000000006e-05 2024-09-13T11:26:00.865 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:00.865 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:00.865 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:00.865 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.865 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.865 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:209273 3.24 3:24d19ad8:::benchmark_data_smithi177_31116_object68970:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e185)", 2024-09-13T11:26:00.865 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:25:53.660601+0000", 2024-09-13T11:26:00.865 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 7.1176213710000003, 2024-09-13T11:26:00.865 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.203427316, 2024-09-13T11:26:00.865 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:26:00.865 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:26:00.865 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:00.866 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:26:00.866 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:26:00.866 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:00.866 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 209273 2024-09-13T11:26:00.866 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.866 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:26:00.866 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.866 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:26:00.866 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.660601+0000", 2024-09-13T11:26:00.866 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:00.866 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.866 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.866 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:26:00.866 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.660601+0000", 2024-09-13T11:26:00.866 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:00.867 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.867 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.867 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:26:00.867 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.660603+0000", 2024-09-13T11:26:00.867 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.7039999999999999e-06 2024-09-13T11:26:00.867 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.867 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.867 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:26:00.867 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.660609+0000", 2024-09-13T11:26:00.867 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.5910000000000003e-06 2024-09-13T11:26:00.867 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.867 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.867 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:26:00.868 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.660623+0000", 2024-09-13T11:26:00.868 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.4338000000000001e-05 2024-09-13T11:26:00.868 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.868 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.868 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:26:00.868 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.660631+0000", 2024-09-13T11:26:00.868 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.1349999999999992e-06 2024-09-13T11:26:00.868 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.868 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.868 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:00.868 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.660694+0000", 2024-09-13T11:26:00.868 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.2755000000000005e-05 2024-09-13T11:26:00.869 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.869 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.869 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:26:00.869 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.660816+0000", 2024-09-13T11:26:00.869 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000121716 2024-09-13T11:26:00.869 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.869 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.869 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1", 2024-09-13T11:26:00.869 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.660900+0000", 2024-09-13T11:26:00.869 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.4924000000000002e-05 2024-09-13T11:26:00.869 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.869 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.869 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:26:00.870 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.662468+0000", 2024-09-13T11:26:00.870 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0015679279999999999 2024-09-13T11:26:00.870 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.870 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.870 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:00.870 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.863779+0000", 2024-09-13T11:26:00.870 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.201310406 2024-09-13T11:26:00.870 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.870 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.870 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:26:00.870 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.863921+0000", 2024-09-13T11:26:00.870 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00014240100000000001 2024-09-13T11:26:00.870 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.870 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.871 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:26:00.871 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.864029+0000", 2024-09-13T11:26:00.871 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00010741799999999999 2024-09-13T11:26:00.871 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:00.871 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:00.871 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:00.871 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.871 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.871 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:209298 3.19 3:9e1ecea7:::benchmark_data_smithi177_31116_object68995:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e185)", 2024-09-13T11:26:00.871 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:25:53.682311+0000", 2024-09-13T11:26:00.871 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 7.095911149, 2024-09-13T11:26:00.871 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.20511669699999999, 2024-09-13T11:26:00.871 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:26:00.871 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:26:00.872 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:00.872 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:26:00.872 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:26:00.872 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:00.872 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 209298 2024-09-13T11:26:00.872 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.872 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:26:00.872 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.872 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:26:00.872 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.682311+0000", 2024-09-13T11:26:00.872 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:00.872 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.872 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.872 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:26:00.872 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.682311+0000", 2024-09-13T11:26:00.873 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:00.873 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.873 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.873 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:26:00.873 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.682314+0000", 2024-09-13T11:26:00.873 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.588e-06 2024-09-13T11:26:00.873 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.873 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.873 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:26:00.873 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.682320+0000", 2024-09-13T11:26:00.873 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.3949999999999998e-06 2024-09-13T11:26:00.873 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.873 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.873 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:26:00.874 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.682335+0000", 2024-09-13T11:26:00.874 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.4789e-05 2024-09-13T11:26:00.874 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.874 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.874 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:26:00.874 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.682343+0000", 2024-09-13T11:26:00.874 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.3919999999999997e-06 2024-09-13T11:26:00.874 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.874 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.874 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:00.874 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.682456+0000", 2024-09-13T11:26:00.874 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00011341999999999999 2024-09-13T11:26:00.874 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.874 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.875 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:26:00.875 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.682583+0000", 2024-09-13T11:26:00.875 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00012655500000000001 2024-09-13T11:26:00.875 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.875 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.875 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1", 2024-09-13T11:26:00.875 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.682681+0000", 2024-09-13T11:26:00.875 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.8158000000000004e-05 2024-09-13T11:26:00.875 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.875 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.875 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:26:00.875 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.685667+0000", 2024-09-13T11:26:00.875 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0029857540000000002 2024-09-13T11:26:00.875 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.875 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.876 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:00.876 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.887298+0000", 2024-09-13T11:26:00.876 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.201631591 2024-09-13T11:26:00.876 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.876 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.876 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:26:00.876 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.887369+0000", 2024-09-13T11:26:00.876 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.0743999999999999e-05 2024-09-13T11:26:00.876 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.876 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.876 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:26:00.876 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.887428+0000", 2024-09-13T11:26:00.876 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.9311000000000002e-05 2024-09-13T11:26:00.877 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:00.877 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:00.877 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:00.877 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.877 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.877 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:209301 3.24 3:24a6266e:::benchmark_data_smithi177_31116_object68998:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e185)", 2024-09-13T11:26:00.877 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:25:53.685255+0000", 2024-09-13T11:26:00.877 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 7.0929676419999996, 2024-09-13T11:26:00.877 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.17909287500000001, 2024-09-13T11:26:00.877 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:26:00.877 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:26:00.877 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:00.877 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:26:00.878 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:26:00.878 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:00.878 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 209301 2024-09-13T11:26:00.878 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.878 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:26:00.878 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.878 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:26:00.878 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.685255+0000", 2024-09-13T11:26:00.878 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:00.879 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.879 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.879 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:26:00.879 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.685255+0000", 2024-09-13T11:26:00.879 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:00.879 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.879 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.879 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:26:00.879 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.685257+0000", 2024-09-13T11:26:00.879 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.5319999999999999e-06 2024-09-13T11:26:00.879 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.879 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.879 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:26:00.880 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.685262+0000", 2024-09-13T11:26:00.880 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.0119999999999996e-06 2024-09-13T11:26:00.880 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.880 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.880 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:26:00.880 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.685270+0000", 2024-09-13T11:26:00.880 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.8829999999999995e-06 2024-09-13T11:26:00.880 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.880 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.880 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:26:00.880 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.685276+0000", 2024-09-13T11:26:00.880 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.9159999999999998e-06 2024-09-13T11:26:00.880 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.880 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.881 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:00.881 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.685338+0000", 2024-09-13T11:26:00.881 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.1937e-05 2024-09-13T11:26:00.881 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.881 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.881 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:26:00.881 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.685465+0000", 2024-09-13T11:26:00.881 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00012644500000000001 2024-09-13T11:26:00.881 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.881 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.881 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1", 2024-09-13T11:26:00.881 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.685558+0000", 2024-09-13T11:26:00.881 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.3377999999999999e-05 2024-09-13T11:26:00.881 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.882 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.882 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:26:00.882 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.687192+0000", 2024-09-13T11:26:00.882 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0016342959999999999 2024-09-13T11:26:00.882 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.882 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.882 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:00.882 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.864128+0000", 2024-09-13T11:26:00.882 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.176935336 2024-09-13T11:26:00.882 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.882 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.882 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:26:00.882 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.864254+0000", 2024-09-13T11:26:00.882 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00012664100000000001 2024-09-13T11:26:00.883 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:00.883 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:00.883 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:26:00.883 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.864348+0000", 2024-09-13T11:26:00.883 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.3498999999999994e-05 2024-09-13T11:26:00.883 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:00.883 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:00.883 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:00.883 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:00.883 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:00.883 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-09-13T11:26:00.885 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_ops_in_flight 2024-09-13T11:26:00.885 INFO:tasks.ceph.osd.9.smithi092.stderr:2024-09-13T11:26:00.880+0000 7ff5ab3d0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26348) UID: 0 2024-09-13T11:26:00.985 INFO:tasks.ceph.osd.1.smithi092.stderr:2024-09-13T11:26:00.984+0000 7fe7789c5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33718) UID: 0 2024-09-13T11:26:01.064 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-13T11:26:01.064 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T11:26:01.075 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-09-13T11:26:01.075 INFO:teuthology.orchestra.run.smithi092.stdout: "ops": [ 2024-09-13T11:26:01.075 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.075 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:218555 3.b 3:d3b7a36d:::benchmark_data_smithi177_31116_object78252:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e188)", 2024-09-13T11:26:01.075 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:26:01.061572+0000", 2024-09-13T11:26:01.076 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 0.013866998, 2024-09-13T11:26:01.076 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.013887603, 2024-09-13T11:26:01.076 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:26:01.076 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:26:01.076 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "waiting for sub ops", 2024-09-13T11:26:01.076 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:26:01.076 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:26:01.076 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:01.076 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 218555 2024-09-13T11:26:01.076 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.076 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:26:01.076 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.076 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:26:01.077 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:01.061572+0000", 2024-09-13T11:26:01.077 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:01.077 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.077 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.077 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:26:01.077 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:01.061572+0000", 2024-09-13T11:26:01.077 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:01.077 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.077 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.077 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:26:01.077 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:01.061575+0000", 2024-09-13T11:26:01.077 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.3149999999999999e-06 2024-09-13T11:26:01.077 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.078 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.078 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:26:01.078 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:01.061581+0000", 2024-09-13T11:26:01.078 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.4200000000000004e-06 2024-09-13T11:26:01.078 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.078 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.078 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:26:01.078 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:01.061595+0000", 2024-09-13T11:26:01.078 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.3528e-05 2024-09-13T11:26:01.078 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.079 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.079 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:26:01.079 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:01.061603+0000", 2024-09-13T11:26:01.079 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.5089999999999993e-06 2024-09-13T11:26:01.079 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.079 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.079 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:01.079 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:01.061672+0000", 2024-09-13T11:26:01.079 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.8627000000000005e-05 2024-09-13T11:26:01.079 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.079 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.079 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:26:01.080 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:01.061821+0000", 2024-09-13T11:26:01.080 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00014913599999999999 2024-09-13T11:26:01.080 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.080 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.080 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1", 2024-09-13T11:26:01.080 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:01.061938+0000", 2024-09-13T11:26:01.080 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000117082 2024-09-13T11:26:01.080 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.080 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.080 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:26:01.080 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:01.064270+0000", 2024-09-13T11:26:01.080 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0023317260000000001 2024-09-13T11:26:01.080 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:01.080 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:01.081 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:01.081 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.081 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.081 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:218560 3.1 3:80a0c853:::benchmark_data_smithi177_31116_object78257:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e188)", 2024-09-13T11:26:01.081 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:26:01.065292+0000", 2024-09-13T11:26:01.081 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 0.010146954, 2024-09-13T11:26:01.081 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.010214267000000001, 2024-09-13T11:26:01.081 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:26:01.081 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:26:01.081 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "waiting for sub ops", 2024-09-13T11:26:01.081 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:26:01.081 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:26:01.081 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:01.081 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 218560 2024-09-13T11:26:01.081 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.082 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:26:01.082 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.082 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:26:01.082 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:01.065292+0000", 2024-09-13T11:26:01.082 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:01.082 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.082 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.082 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:26:01.082 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:01.065292+0000", 2024-09-13T11:26:01.082 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:01.082 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.082 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.082 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:26:01.082 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:01.065294+0000", 2024-09-13T11:26:01.083 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.995e-06 2024-09-13T11:26:01.083 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.083 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.083 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:26:01.083 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:01.065301+0000", 2024-09-13T11:26:01.083 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.2079999999999997e-06 2024-09-13T11:26:01.083 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.083 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.083 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:26:01.083 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:01.065315+0000", 2024-09-13T11:26:01.084 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.4416000000000001e-05 2024-09-13T11:26:01.084 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.084 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.084 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:26:01.084 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:01.065324+0000", 2024-09-13T11:26:01.084 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.5839999999999995e-06 2024-09-13T11:26:01.084 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.084 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.084 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:01.084 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:01.065388+0000", 2024-09-13T11:26:01.084 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.4676000000000002e-05 2024-09-13T11:26:01.085 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.085 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.085 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:26:01.085 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:01.065534+0000", 2024-09-13T11:26:01.085 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000146105 2024-09-13T11:26:01.085 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.085 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.085 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 11", 2024-09-13T11:26:01.085 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:01.065641+0000", 2024-09-13T11:26:01.085 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000106433 2024-09-13T11:26:01.085 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.085 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.085 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:26:01.085 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:01.067685+0000", 2024-09-13T11:26:01.086 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0020444759999999999 2024-09-13T11:26:01.086 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:01.086 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:01.086 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:01.086 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:01.086 INFO:teuthology.orchestra.run.smithi092.stdout: ], 2024-09-13T11:26:01.086 INFO:teuthology.orchestra.run.smithi092.stdout: "num_ops": 2 2024-09-13T11:26:01.086 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-09-13T11:26:01.087 INFO:tasks.ceph.osd.7.smithi177.stderr:2024-09-13T11:26:01.085+0000 7f897b51d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26575) UID: 0 2024-09-13T11:26:01.101 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_blocked_ops 2024-09-13T11:26:01.193 INFO:tasks.ceph.osd.5.smithi092.stderr:2024-09-13T11:26:01.192+0000 7f6f0b705640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26346) UID: 0 2024-09-13T11:26:01.261 INFO:teuthology.orchestra.run.smithi001.stdout:osd.0: {} 2024-09-13T11:26:01.262 INFO:teuthology.orchestra.run.smithi001.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T11:26:01.278 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-09-13T11:26:01.279 INFO:teuthology.orchestra.run.smithi092.stdout: "ops": [], 2024-09-13T11:26:01.279 INFO:teuthology.orchestra.run.smithi092.stdout: "complaint_time": 30, 2024-09-13T11:26:01.279 INFO:teuthology.orchestra.run.smithi092.stdout: "num_blocked_ops": 0 2024-09-13T11:26:01.279 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-09-13T11:26:01.289 INFO:teuthology.orchestra.run.smithi001.stdout:osd.1: {} 2024-09-13T11:26:01.289 INFO:teuthology.orchestra.run.smithi001.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T11:26:01.289 INFO:tasks.ceph.osd.5.smithi092.stderr:2024-09-13T11:26:01.288+0000 7f6f0b705640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26346) UID: 0 2024-09-13T11:26:01.297 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_historic_ops 2024-09-13T11:26:01.321 INFO:teuthology.orchestra.run.smithi001.stdout:osd.2: {} 2024-09-13T11:26:01.321 INFO:teuthology.orchestra.run.smithi001.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T11:26:01.350 INFO:teuthology.orchestra.run.smithi001.stdout:osd.3: {} 2024-09-13T11:26:01.351 INFO:teuthology.orchestra.run.smithi001.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T11:26:01.387 INFO:teuthology.orchestra.run.smithi001.stdout:osd.4: {} 2024-09-13T11:26:01.388 INFO:teuthology.orchestra.run.smithi001.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T11:26:01.390 INFO:tasks.ceph.osd.1.smithi092.stderr:2024-09-13T11:26:01.388+0000 7fe7789c5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33718) UID: 0 2024-09-13T11:26:01.411 INFO:teuthology.orchestra.run.smithi001.stdout:osd.5: {} 2024-09-13T11:26:01.412 INFO:teuthology.orchestra.run.smithi001.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T11:26:01.444 INFO:teuthology.orchestra.run.smithi001.stdout:osd.6: {} 2024-09-13T11:26:01.444 INFO:teuthology.orchestra.run.smithi001.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T11:26:01.474 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-09-13T11:26:01.475 INFO:teuthology.orchestra.run.smithi092.stdout: "size": 20, 2024-09-13T11:26:01.475 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 600, 2024-09-13T11:26:01.475 INFO:teuthology.orchestra.run.smithi092.stdout: "ops": [ 2024-09-13T11:26:01.475 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.475 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:79 3.1 3:841df5b5:::benchmark_data_smithi177_31116_object78:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-09-13T11:26:01.475 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:27.056083+0000", 2024-09-13T11:26:01.475 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 214.41678633999999, 2024-09-13T11:26:01.475 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.89469730199999997, 2024-09-13T11:26:01.476 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:26:01.476 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:26:01.476 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:01.476 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:26:01.476 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:26:01.476 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:01.476 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 79 2024-09-13T11:26:01.476 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.476 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:26:01.476 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.476 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:26:01.477 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.056083+0000", 2024-09-13T11:26:01.477 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:01.477 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.477 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.477 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:26:01.477 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.056083+0000", 2024-09-13T11:26:01.477 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:01.477 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.477 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.477 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:26:01.477 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.056087+0000", 2024-09-13T11:26:01.477 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.0659999999999997e-06 2024-09-13T11:26:01.477 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.477 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.478 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:26:01.478 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.056148+0000", 2024-09-13T11:26:01.478 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.1376000000000003e-05 2024-09-13T11:26:01.478 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.478 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.478 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:26:01.478 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.056180+0000", 2024-09-13T11:26:01.478 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.2370999999999999e-05 2024-09-13T11:26:01.478 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.478 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.478 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:26:01.478 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.056195+0000", 2024-09-13T11:26:01.478 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.5119e-05 2024-09-13T11:26:01.479 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.479 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.479 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:01.479 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.056304+0000", 2024-09-13T11:26:01.479 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000108365 2024-09-13T11:26:01.479 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.479 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.479 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:26:01.479 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.056478+0000", 2024-09-13T11:26:01.479 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000174542 2024-09-13T11:26:01.479 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.479 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.479 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 11", 2024-09-13T11:26:01.479 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.056690+0000", 2024-09-13T11:26:01.480 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00021167500000000001 2024-09-13T11:26:01.480 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.480 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.480 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:26:01.480 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.058063+0000", 2024-09-13T11:26:01.480 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0013731819999999999 2024-09-13T11:26:01.480 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.480 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.480 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:01.480 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.950477+0000", 2024-09-13T11:26:01.480 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.89241369400000004 2024-09-13T11:26:01.480 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.480 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.481 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:26:01.481 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.950652+0000", 2024-09-13T11:26:01.482 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00017526400000000001 2024-09-13T11:26:01.482 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.482 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.482 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:26:01.482 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.950780+0000", 2024-09-13T11:26:01.482 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00012764800000000001 2024-09-13T11:26:01.482 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:01.482 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:01.482 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:01.483 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.483 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.483 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:7 3.b 3:de78393b:::benchmark_data_smithi177_31116_object6:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T11:26:01.483 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:48.313952+0000", 2024-09-13T11:26:01.483 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 193.15891659100001, 2024-09-13T11:26:01.483 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.86421288600000001, 2024-09-13T11:26:01.484 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:26:01.484 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:26:01.484 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:01.484 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:26:01.484 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:26:01.484 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:01.484 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 7 2024-09-13T11:26:01.484 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.484 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:26:01.484 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.484 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:26:01.484 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.313952+0000", 2024-09-13T11:26:01.485 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:01.485 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.485 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.485 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:26:01.485 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.313952+0000", 2024-09-13T11:26:01.485 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:01.485 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.485 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.485 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:26:01.485 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.313957+0000", 2024-09-13T11:26:01.485 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.3599999999999998e-06 2024-09-13T11:26:01.485 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.485 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.486 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:26:01.486 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314043+0000", 2024-09-13T11:26:01.486 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.6766000000000002e-05 2024-09-13T11:26:01.486 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.486 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.486 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:26:01.486 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314065+0000", 2024-09-13T11:26:01.486 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.1416e-05 2024-09-13T11:26:01.486 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.486 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.486 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:26:01.487 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314080+0000", 2024-09-13T11:26:01.487 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.5519000000000002e-05 2024-09-13T11:26:01.487 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.487 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.487 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:01.487 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315478+0000", 2024-09-13T11:26:01.487 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.001397765 2024-09-13T11:26:01.487 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.487 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.487 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-09-13T11:26:01.487 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315507+0000", 2024-09-13T11:26:01.487 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.9128999999999998e-05 2024-09-13T11:26:01.487 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.487 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.488 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:01.488 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.167978+0000", 2024-09-13T11:26:01.488 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.85247068400000003 2024-09-13T11:26:01.488 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.488 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.488 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:26:01.488 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.168307+0000", 2024-09-13T11:26:01.488 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00032886699999999999 2024-09-13T11:26:01.488 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.488 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.488 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1", 2024-09-13T11:26:01.489 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.168647+0000", 2024-09-13T11:26:01.489 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00033998399999999999 2024-09-13T11:26:01.489 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.489 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.489 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:01.489 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.178011+0000", 2024-09-13T11:26:01.489 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0093639169999999994 2024-09-13T11:26:01.489 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.489 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.489 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:26:01.489 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.178026+0000", 2024-09-13T11:26:01.490 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.5274999999999999e-05 2024-09-13T11:26:01.490 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.490 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.490 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:26:01.490 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.178080+0000", 2024-09-13T11:26:01.490 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.3601999999999999e-05 2024-09-13T11:26:01.490 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.490 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.490 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:26:01.490 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.178165+0000", 2024-09-13T11:26:01.490 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.5601999999999998e-05 2024-09-13T11:26:01.490 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:01.491 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:01.491 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:01.491 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.491 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.491 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:9 3.b 3:dafadff6:::benchmark_data_smithi177_31116_object8:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T11:26:01.491 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:48.314114+0000", 2024-09-13T11:26:01.491 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 193.15875477700001, 2024-09-13T11:26:01.491 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.86613615899999996, 2024-09-13T11:26:01.491 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:26:01.491 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:26:01.491 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:01.491 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:26:01.492 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:26:01.492 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:01.492 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 9 2024-09-13T11:26:01.492 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.492 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:26:01.492 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.492 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:26:01.492 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314114+0000", 2024-09-13T11:26:01.492 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:01.492 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.492 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.492 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:26:01.493 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314114+0000", 2024-09-13T11:26:01.493 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:01.493 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.493 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.493 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:26:01.493 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314117+0000", 2024-09-13T11:26:01.493 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.3860000000000001e-06 2024-09-13T11:26:01.493 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.493 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.493 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:26:01.493 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314215+0000", 2024-09-13T11:26:01.493 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.8900000000000005e-05 2024-09-13T11:26:01.493 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.494 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.494 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:26:01.494 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314229+0000", 2024-09-13T11:26:01.494 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.3444999999999999e-05 2024-09-13T11:26:01.494 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.494 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.494 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:26:01.494 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314239+0000", 2024-09-13T11:26:01.494 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.8470000000000008e-06 2024-09-13T11:26:01.494 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.494 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.494 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:01.494 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315648+0000", 2024-09-13T11:26:01.495 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.001409291 2024-09-13T11:26:01.495 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.495 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.495 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-09-13T11:26:01.495 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315671+0000", 2024-09-13T11:26:01.495 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.3169000000000001e-05 2024-09-13T11:26:01.495 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.495 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.495 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:01.495 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.176036+0000", 2024-09-13T11:26:01.495 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.86036497499999998 2024-09-13T11:26:01.495 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.495 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.496 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:26:01.496 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.176189+0000", 2024-09-13T11:26:01.496 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00015271100000000001 2024-09-13T11:26:01.496 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.496 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.496 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1", 2024-09-13T11:26:01.496 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.176339+0000", 2024-09-13T11:26:01.496 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00015057499999999999 2024-09-13T11:26:01.496 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.496 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.496 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:01.496 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.180148+0000", 2024-09-13T11:26:01.497 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.003808891 2024-09-13T11:26:01.497 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.497 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.497 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:26:01.497 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.180168+0000", 2024-09-13T11:26:01.497 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.9507000000000001e-05 2024-09-13T11:26:01.497 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.497 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.497 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:26:01.497 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.180213+0000", 2024-09-13T11:26:01.497 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.5504999999999999e-05 2024-09-13T11:26:01.497 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.497 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.498 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:26:01.498 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.180250+0000", 2024-09-13T11:26:01.498 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.6956999999999999e-05 2024-09-13T11:26:01.498 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:01.498 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:01.498 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:01.498 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.498 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.498 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:10 3.b 3:dbe028a8:::benchmark_data_smithi177_31116_object9:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T11:26:01.498 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:48.314259+0000", 2024-09-13T11:26:01.498 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 193.15860964699999, 2024-09-13T11:26:01.498 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.86637313599999999, 2024-09-13T11:26:01.499 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:26:01.499 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:26:01.499 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:01.499 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:26:01.499 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:26:01.499 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:01.499 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 10 2024-09-13T11:26:01.499 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.499 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:26:01.499 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.499 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:26:01.499 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314259+0000", 2024-09-13T11:26:01.499 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:01.500 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.500 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.500 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:26:01.500 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314259+0000", 2024-09-13T11:26:01.500 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:01.500 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.500 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.500 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:26:01.500 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314261+0000", 2024-09-13T11:26:01.500 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.1500000000000002e-06 2024-09-13T11:26:01.500 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.500 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.500 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:26:01.501 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314349+0000", 2024-09-13T11:26:01.501 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.8070000000000002e-05 2024-09-13T11:26:01.501 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.501 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.501 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:26:01.501 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314365+0000", 2024-09-13T11:26:01.501 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.5835999999999999e-05 2024-09-13T11:26:01.501 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.501 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.501 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:26:01.501 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314377+0000", 2024-09-13T11:26:01.501 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.1589e-05 2024-09-13T11:26:01.501 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.502 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.502 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:01.502 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315788+0000", 2024-09-13T11:26:01.502 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0014109579999999999 2024-09-13T11:26:01.502 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.502 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.502 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-09-13T11:26:01.502 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315809+0000", 2024-09-13T11:26:01.502 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.0985999999999999e-05 2024-09-13T11:26:01.502 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.502 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.502 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:01.502 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.176822+0000", 2024-09-13T11:26:01.503 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.86101310799999997 2024-09-13T11:26:01.503 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.503 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.503 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:26:01.503 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.177021+0000", 2024-09-13T11:26:01.503 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00019911199999999999 2024-09-13T11:26:01.503 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.503 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.503 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1", 2024-09-13T11:26:01.503 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.177226+0000", 2024-09-13T11:26:01.503 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00020516600000000001 2024-09-13T11:26:01.503 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.503 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.503 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:26:01.504 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.180264+0000", 2024-09-13T11:26:01.504 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0030381739999999998 2024-09-13T11:26:01.504 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.504 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.504 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:01.504 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.180387+0000", 2024-09-13T11:26:01.504 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00012257699999999999 2024-09-13T11:26:01.504 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.504 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.504 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:26:01.504 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.180518+0000", 2024-09-13T11:26:01.504 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00013135800000000001 2024-09-13T11:26:01.504 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.505 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.505 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:26:01.505 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.180632+0000", 2024-09-13T11:26:01.505 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000114052 2024-09-13T11:26:01.505 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:01.505 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:01.505 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:01.505 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.505 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.505 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:39 3.b 3:d91854b3:::benchmark_data_smithi177_31116_object38:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T11:26:01.505 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:48.314393+0000", 2024-09-13T11:26:01.505 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 193.15847610500001, 2024-09-13T11:26:01.505 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.86850121700000005, 2024-09-13T11:26:01.506 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:26:01.506 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:26:01.506 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:01.506 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:26:01.506 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:26:01.506 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:01.507 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 39 2024-09-13T11:26:01.507 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.507 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:26:01.507 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.507 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:26:01.507 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314393+0000", 2024-09-13T11:26:01.507 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:01.507 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.507 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.507 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:26:01.507 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314393+0000", 2024-09-13T11:26:01.507 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:01.507 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.508 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.508 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:26:01.508 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314394+0000", 2024-09-13T11:26:01.508 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.513e-06 2024-09-13T11:26:01.508 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.508 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.508 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:26:01.508 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314463+0000", 2024-09-13T11:26:01.508 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.8261000000000003e-05 2024-09-13T11:26:01.508 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.508 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.508 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:26:01.508 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314477+0000", 2024-09-13T11:26:01.509 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.4619e-05 2024-09-13T11:26:01.509 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.509 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.509 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:26:01.509 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314485+0000", 2024-09-13T11:26:01.509 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.1699999999999997e-06 2024-09-13T11:26:01.509 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.509 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.509 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:01.509 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315917+0000", 2024-09-13T11:26:01.509 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0014316089999999999 2024-09-13T11:26:01.509 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.509 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.510 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-09-13T11:26:01.510 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315933+0000", 2024-09-13T11:26:01.510 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.5999999999999999e-05 2024-09-13T11:26:01.510 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.510 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.510 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:01.510 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.178205+0000", 2024-09-13T11:26:01.510 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.86227247100000004 2024-09-13T11:26:01.510 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.510 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.510 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:26:01.511 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.178316+0000", 2024-09-13T11:26:01.511 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000110788 2024-09-13T11:26:01.511 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.511 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.511 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1", 2024-09-13T11:26:01.511 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.178448+0000", 2024-09-13T11:26:01.511 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000132056 2024-09-13T11:26:01.511 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.511 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.511 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:26:01.511 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.182721+0000", 2024-09-13T11:26:01.511 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0042729780000000002 2024-09-13T11:26:01.511 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.512 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.512 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:01.512 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.182801+0000", 2024-09-13T11:26:01.512 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.0209000000000004e-05 2024-09-13T11:26:01.512 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.512 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.512 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:26:01.512 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.182855+0000", 2024-09-13T11:26:01.512 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.3749000000000003e-05 2024-09-13T11:26:01.512 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.512 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.512 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:26:01.513 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.182894+0000", 2024-09-13T11:26:01.513 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.8794000000000002e-05 2024-09-13T11:26:01.513 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:01.513 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:01.513 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:01.513 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.513 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.513 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:50 3.b 3:d1c144b8:::benchmark_data_smithi177_31116_object49:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T11:26:01.513 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:48.314498+0000", 2024-09-13T11:26:01.513 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 193.15837069, 2024-09-13T11:26:01.513 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.86878673299999998, 2024-09-13T11:26:01.513 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:26:01.514 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:26:01.514 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:01.514 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:26:01.514 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:26:01.514 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:01.514 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 50 2024-09-13T11:26:01.514 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.514 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:26:01.514 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.514 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:26:01.514 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314498+0000", 2024-09-13T11:26:01.514 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:01.514 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.515 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.515 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:26:01.515 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314498+0000", 2024-09-13T11:26:01.515 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:01.515 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.515 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.515 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:26:01.515 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314499+0000", 2024-09-13T11:26:01.515 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.1969999999999999e-06 2024-09-13T11:26:01.515 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.515 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.515 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:26:01.515 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314565+0000", 2024-09-13T11:26:01.516 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.5628000000000003e-05 2024-09-13T11:26:01.516 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.516 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.516 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:26:01.516 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314579+0000", 2024-09-13T11:26:01.516 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.4222e-05 2024-09-13T11:26:01.516 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.516 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.516 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:26:01.516 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314590+0000", 2024-09-13T11:26:01.516 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.1240000000000001e-05 2024-09-13T11:26:01.516 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.517 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.517 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:01.517 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316060+0000", 2024-09-13T11:26:01.517 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.001469515 2024-09-13T11:26:01.517 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.517 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.517 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-09-13T11:26:01.517 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316076+0000", 2024-09-13T11:26:01.517 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.5608999999999999e-05 2024-09-13T11:26:01.517 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.517 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.517 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:01.518 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.178857+0000", 2024-09-13T11:26:01.518 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.86278099900000005 2024-09-13T11:26:01.518 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.518 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.518 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:26:01.518 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.178961+0000", 2024-09-13T11:26:01.518 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000104664 2024-09-13T11:26:01.518 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.518 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.518 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1", 2024-09-13T11:26:01.518 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.179123+0000", 2024-09-13T11:26:01.518 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000161738 2024-09-13T11:26:01.518 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.519 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.519 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:26:01.519 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.182755+0000", 2024-09-13T11:26:01.519 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0036323200000000001 2024-09-13T11:26:01.519 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.519 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.519 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:01.519 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.183023+0000", 2024-09-13T11:26:01.519 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00026749100000000002 2024-09-13T11:26:01.519 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.519 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.519 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:26:01.520 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.183168+0000", 2024-09-13T11:26:01.520 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00014499099999999999 2024-09-13T11:26:01.520 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.520 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.520 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:26:01.520 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.183285+0000", 2024-09-13T11:26:01.520 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000117119 2024-09-13T11:26:01.520 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:01.520 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:01.520 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:01.520 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.520 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.521 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:60 3.b 3:d67218b0:::benchmark_data_smithi177_31116_object59:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T11:26:01.521 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:48.314603+0000", 2024-09-13T11:26:01.521 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 193.15826612699999, 2024-09-13T11:26:01.521 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.87106201100000002, 2024-09-13T11:26:01.521 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:26:01.521 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:26:01.521 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:01.521 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:26:01.521 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:26:01.521 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:01.521 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 60 2024-09-13T11:26:01.521 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.522 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:26:01.522 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.522 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:26:01.522 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314603+0000", 2024-09-13T11:26:01.522 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:01.522 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.522 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.522 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:26:01.522 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314603+0000", 2024-09-13T11:26:01.522 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:01.522 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.522 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.522 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:26:01.523 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314604+0000", 2024-09-13T11:26:01.523 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.2359999999999999e-06 2024-09-13T11:26:01.523 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.523 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.523 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:26:01.523 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314667+0000", 2024-09-13T11:26:01.523 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.3077999999999995e-05 2024-09-13T11:26:01.523 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.523 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.523 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:26:01.523 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314699+0000", 2024-09-13T11:26:01.523 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.1965999999999999e-05 2024-09-13T11:26:01.524 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.524 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.524 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:26:01.524 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314710+0000", 2024-09-13T11:26:01.524 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.1148999999999999e-05 2024-09-13T11:26:01.524 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.524 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.524 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:01.524 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316291+0000", 2024-09-13T11:26:01.524 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0015804440000000001 2024-09-13T11:26:01.524 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.524 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.525 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-09-13T11:26:01.525 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316310+0000", 2024-09-13T11:26:01.525 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.9799000000000001e-05 2024-09-13T11:26:01.525 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.525 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.525 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:01.525 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.179555+0000", 2024-09-13T11:26:01.525 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.86324469500000001 2024-09-13T11:26:01.525 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.525 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.525 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:26:01.525 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.179666+0000", 2024-09-13T11:26:01.525 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000110443 2024-09-13T11:26:01.526 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.526 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.526 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1", 2024-09-13T11:26:01.526 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.179791+0000", 2024-09-13T11:26:01.526 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00012560499999999999 2024-09-13T11:26:01.526 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.526 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.526 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:26:01.526 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.182761+0000", 2024-09-13T11:26:01.526 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0029697019999999998 2024-09-13T11:26:01.526 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.526 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.526 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:01.527 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.185572+0000", 2024-09-13T11:26:01.527 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.002811512 2024-09-13T11:26:01.527 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.527 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.527 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:26:01.527 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.185625+0000", 2024-09-13T11:26:01.527 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.2716000000000001e-05 2024-09-13T11:26:01.527 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.527 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.527 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:26:01.527 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.185665+0000", 2024-09-13T11:26:01.527 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.9666000000000003e-05 2024-09-13T11:26:01.528 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:01.528 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:01.528 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:01.528 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.528 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.528 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:62 3.b 3:d20ae5a8:::benchmark_data_smithi177_31116_object61:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T11:26:01.528 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:48.314723+0000", 2024-09-13T11:26:01.528 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 193.158145585, 2024-09-13T11:26:01.528 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.87338985099999999, 2024-09-13T11:26:01.528 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:26:01.528 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:26:01.529 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:01.529 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:26:01.529 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:26:01.529 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:01.529 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 62 2024-09-13T11:26:01.529 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.529 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:26:01.529 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.529 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:26:01.529 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314723+0000", 2024-09-13T11:26:01.529 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:01.529 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.530 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.530 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:26:01.530 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314723+0000", 2024-09-13T11:26:01.530 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:01.530 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.530 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.530 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:26:01.530 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314725+0000", 2024-09-13T11:26:01.530 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.3069999999999999e-06 2024-09-13T11:26:01.530 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.530 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.530 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:26:01.530 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314797+0000", 2024-09-13T11:26:01.531 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.2652999999999997e-05 2024-09-13T11:26:01.531 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.531 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.531 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:26:01.531 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314809+0000", 2024-09-13T11:26:01.531 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.1496e-05 2024-09-13T11:26:01.531 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.531 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.531 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:26:01.531 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314816+0000", 2024-09-13T11:26:01.531 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.6639999999999998e-06 2024-09-13T11:26:01.531 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.532 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.532 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:01.532 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316424+0000", 2024-09-13T11:26:01.532 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0016077559999999999 2024-09-13T11:26:01.532 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.532 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.532 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-09-13T11:26:01.532 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316441+0000", 2024-09-13T11:26:01.532 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.7305000000000001e-05 2024-09-13T11:26:01.532 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.532 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.533 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:01.533 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.180688+0000", 2024-09-13T11:26:01.533 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.86424692800000003 2024-09-13T11:26:01.533 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.533 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.533 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:26:01.533 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.180807+0000", 2024-09-13T11:26:01.533 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000118977 2024-09-13T11:26:01.533 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.533 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.533 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1", 2024-09-13T11:26:01.533 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.180938+0000", 2024-09-13T11:26:01.533 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00013049700000000001 2024-09-13T11:26:01.534 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.534 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.534 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:26:01.534 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.185680+0000", 2024-09-13T11:26:01.534 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0047417029999999999 2024-09-13T11:26:01.534 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.534 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.534 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:01.534 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.187856+0000", 2024-09-13T11:26:01.534 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0021760550000000001 2024-09-13T11:26:01.534 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.534 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.534 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:26:01.535 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.188007+0000", 2024-09-13T11:26:01.535 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00015185099999999999 2024-09-13T11:26:01.535 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.535 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.535 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:26:01.535 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.188113+0000", 2024-09-13T11:26:01.535 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000105659 2024-09-13T11:26:01.535 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:01.535 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:01.535 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:01.535 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.535 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.535 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:73 3.b 3:d0060d5f:::benchmark_data_smithi177_31116_object72:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T11:26:01.536 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:48.314829+0000", 2024-09-13T11:26:01.536 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 193.15804023600001, 2024-09-13T11:26:01.536 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.87364757199999998, 2024-09-13T11:26:01.536 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:26:01.536 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:26:01.536 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:01.536 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:26:01.536 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:26:01.536 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:01.536 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 73 2024-09-13T11:26:01.536 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.536 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:26:01.536 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.537 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:26:01.537 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314829+0000", 2024-09-13T11:26:01.537 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:01.537 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.537 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.537 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:26:01.537 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314829+0000", 2024-09-13T11:26:01.537 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:01.537 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.537 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.537 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:26:01.537 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314830+0000", 2024-09-13T11:26:01.537 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.243e-06 2024-09-13T11:26:01.538 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.538 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.538 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:26:01.538 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314965+0000", 2024-09-13T11:26:01.538 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00013470800000000001 2024-09-13T11:26:01.538 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.538 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.538 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:26:01.538 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314979+0000", 2024-09-13T11:26:01.538 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.4375999999999999e-05 2024-09-13T11:26:01.538 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.538 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.538 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:26:01.539 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314988+0000", 2024-09-13T11:26:01.539 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.8570000000000001e-06 2024-09-13T11:26:01.539 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.540 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.540 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:01.541 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316545+0000", 2024-09-13T11:26:01.541 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.001556878 2024-09-13T11:26:01.541 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.541 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.541 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-09-13T11:26:01.541 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316556+0000", 2024-09-13T11:26:01.541 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.1453e-05 2024-09-13T11:26:01.541 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.541 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.541 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:01.541 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.182124+0000", 2024-09-13T11:26:01.542 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.86556744200000002 2024-09-13T11:26:01.542 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.542 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.542 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:26:01.542 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.182246+0000", 2024-09-13T11:26:01.542 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00012210500000000001 2024-09-13T11:26:01.542 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.542 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.542 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1", 2024-09-13T11:26:01.542 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.182375+0000", 2024-09-13T11:26:01.542 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00012920999999999999 2024-09-13T11:26:01.542 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.542 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.543 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:26:01.543 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.185684+0000", 2024-09-13T11:26:01.543 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0033093969999999999 2024-09-13T11:26:01.543 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.543 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.543 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:01.543 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.188244+0000", 2024-09-13T11:26:01.543 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0025601550000000002 2024-09-13T11:26:01.543 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.543 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.543 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:26:01.543 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.188379+0000", 2024-09-13T11:26:01.543 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00013463400000000001 2024-09-13T11:26:01.543 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.544 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.544 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:26:01.544 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.188476+0000", 2024-09-13T11:26:01.544 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.7114000000000006e-05 2024-09-13T11:26:01.544 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:01.544 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:01.544 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:01.544 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.544 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.544 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:81 3.b 3:d705fd70:::benchmark_data_smithi177_31116_object80:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T11:26:01.544 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:48.315000+0000", 2024-09-13T11:26:01.544 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 193.15786839899999, 2024-09-13T11:26:01.545 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.87561418300000005, 2024-09-13T11:26:01.545 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:26:01.545 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:26:01.545 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:01.545 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:26:01.545 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:26:01.545 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:01.545 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 81 2024-09-13T11:26:01.545 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.545 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:26:01.545 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.545 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:26:01.545 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315000+0000", 2024-09-13T11:26:01.546 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:01.546 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.546 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.546 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:26:01.546 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315000+0000", 2024-09-13T11:26:01.546 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:01.546 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.546 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.546 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:26:01.546 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315002+0000", 2024-09-13T11:26:01.547 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.3799999999999999e-06 2024-09-13T11:26:01.547 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.547 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.547 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:26:01.547 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315119+0000", 2024-09-13T11:26:01.547 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000116925 2024-09-13T11:26:01.547 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.547 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.547 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:26:01.547 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315133+0000", 2024-09-13T11:26:01.547 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.3971999999999999e-05 2024-09-13T11:26:01.547 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.548 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.548 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:26:01.548 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315142+0000", 2024-09-13T11:26:01.548 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.4399999999999994e-06 2024-09-13T11:26:01.548 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.548 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.548 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:01.548 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316652+0000", 2024-09-13T11:26:01.548 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.001509907 2024-09-13T11:26:01.549 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.549 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.549 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-09-13T11:26:01.549 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316668+0000", 2024-09-13T11:26:01.549 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.5945999999999999e-05 2024-09-13T11:26:01.549 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.549 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.549 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:01.550 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.183405+0000", 2024-09-13T11:26:01.550 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.866737384 2024-09-13T11:26:01.550 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.550 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.550 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:26:01.550 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.183701+0000", 2024-09-13T11:26:01.551 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00029511099999999997 2024-09-13T11:26:01.551 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.551 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.551 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1", 2024-09-13T11:26:01.551 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.184078+0000", 2024-09-13T11:26:01.551 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00037741000000000001 2024-09-13T11:26:01.551 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.551 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.551 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:26:01.551 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.188504+0000", 2024-09-13T11:26:01.551 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0044260280000000003 2024-09-13T11:26:01.551 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.551 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.552 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:01.552 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.190562+0000", 2024-09-13T11:26:01.552 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0020580099999999999 2024-09-13T11:26:01.552 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.552 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.552 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:26:01.552 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.190592+0000", 2024-09-13T11:26:01.552 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.0057999999999999e-05 2024-09-13T11:26:01.552 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.552 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.552 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:26:01.552 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.190615+0000", 2024-09-13T11:26:01.552 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.2612e-05 2024-09-13T11:26:01.553 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:01.553 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:01.553 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:01.553 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.553 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.553 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:87 3.b 3:dd7de0ad:::benchmark_data_smithi177_31116_object86:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T11:26:01.553 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:48.315155+0000", 2024-09-13T11:26:01.553 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 193.157713602, 2024-09-13T11:26:01.553 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.87555916099999997, 2024-09-13T11:26:01.553 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:26:01.553 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:26:01.553 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:01.553 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:26:01.553 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:26:01.554 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:01.554 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 87 2024-09-13T11:26:01.554 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.554 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:26:01.554 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.554 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:26:01.554 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315155+0000", 2024-09-13T11:26:01.554 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:01.554 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.554 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.554 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:26:01.554 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315155+0000", 2024-09-13T11:26:01.554 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:01.554 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.555 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.555 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:26:01.555 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315157+0000", 2024-09-13T11:26:01.555 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.3829999999999999e-06 2024-09-13T11:26:01.555 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.555 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.555 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:26:01.555 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315271+0000", 2024-09-13T11:26:01.555 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000114539 2024-09-13T11:26:01.555 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.555 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.555 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:26:01.556 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315285+0000", 2024-09-13T11:26:01.556 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.4042e-05 2024-09-13T11:26:01.556 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.556 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.556 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:26:01.556 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315295+0000", 2024-09-13T11:26:01.556 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.2739999999999996e-06 2024-09-13T11:26:01.556 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.556 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.556 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:01.556 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316739+0000", 2024-09-13T11:26:01.556 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0014446960000000001 2024-09-13T11:26:01.556 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.557 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.557 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-09-13T11:26:01.557 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316747+0000", 2024-09-13T11:26:01.557 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.0790000000000001e-06 2024-09-13T11:26:01.557 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.557 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.557 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:01.557 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.184922+0000", 2024-09-13T11:26:01.557 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.86817446399999998 2024-09-13T11:26:01.557 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.557 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.557 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:26:01.558 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.185037+0000", 2024-09-13T11:26:01.558 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00011565099999999999 2024-09-13T11:26:01.558 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.558 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.558 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1", 2024-09-13T11:26:01.558 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.185171+0000", 2024-09-13T11:26:01.558 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00013316900000000001 2024-09-13T11:26:01.558 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.558 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.558 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:26:01.558 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.188520+0000", 2024-09-13T11:26:01.558 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0033489150000000001 2024-09-13T11:26:01.558 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.558 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.559 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:01.559 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.190652+0000", 2024-09-13T11:26:01.559 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0021325649999999999 2024-09-13T11:26:01.559 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.559 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.559 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:26:01.559 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.190686+0000", 2024-09-13T11:26:01.559 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.3827e-05 2024-09-13T11:26:01.559 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.559 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.559 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:26:01.559 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.190714+0000", 2024-09-13T11:26:01.559 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.8557000000000001e-05 2024-09-13T11:26:01.560 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:01.560 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:01.560 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:01.560 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.560 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.560 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:99 3.b 3:d011b600:::benchmark_data_smithi177_31116_object98:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T11:26:01.560 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:48.315307+0000", 2024-09-13T11:26:01.560 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 193.157561775, 2024-09-13T11:26:01.560 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.87551770799999995, 2024-09-13T11:26:01.560 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:26:01.560 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:26:01.560 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:01.561 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:26:01.561 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:26:01.561 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:01.561 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 99 2024-09-13T11:26:01.561 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.561 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:26:01.561 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.561 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:26:01.561 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315307+0000", 2024-09-13T11:26:01.561 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:01.561 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.561 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.561 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:26:01.561 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315307+0000", 2024-09-13T11:26:01.562 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:01.562 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.562 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.562 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:26:01.562 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315308+0000", 2024-09-13T11:26:01.562 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.316e-06 2024-09-13T11:26:01.562 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.562 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.562 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:26:01.562 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315453+0000", 2024-09-13T11:26:01.563 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00014473200000000001 2024-09-13T11:26:01.563 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.563 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.563 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:26:01.563 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315471+0000", 2024-09-13T11:26:01.563 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.8019000000000001e-05 2024-09-13T11:26:01.563 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.563 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.563 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:26:01.563 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315489+0000", 2024-09-13T11:26:01.563 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.7861999999999999e-05 2024-09-13T11:26:01.563 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.563 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.563 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:01.564 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316837+0000", 2024-09-13T11:26:01.564 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.001347787 2024-09-13T11:26:01.564 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.564 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.564 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-09-13T11:26:01.564 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316858+0000", 2024-09-13T11:26:01.564 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.1103000000000001e-05 2024-09-13T11:26:01.564 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.564 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.564 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:01.564 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.185713+0000", 2024-09-13T11:26:01.564 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.86885540800000005 2024-09-13T11:26:01.564 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.564 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.565 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:26:01.565 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.185820+0000", 2024-09-13T11:26:01.565 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000107081 2024-09-13T11:26:01.565 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.565 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.565 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1", 2024-09-13T11:26:01.566 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.185948+0000", 2024-09-13T11:26:01.566 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00012775800000000001 2024-09-13T11:26:01.566 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.566 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.566 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:26:01.566 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.190527+0000", 2024-09-13T11:26:01.567 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0045783910000000002 2024-09-13T11:26:01.567 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.567 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.567 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:01.567 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.190759+0000", 2024-09-13T11:26:01.568 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00023266199999999999 2024-09-13T11:26:01.568 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.568 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.568 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:26:01.568 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.190789+0000", 2024-09-13T11:26:01.568 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.9666999999999999e-05 2024-09-13T11:26:01.568 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.569 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.569 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:26:01.569 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.190825+0000", 2024-09-13T11:26:01.569 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.5921999999999999e-05 2024-09-13T11:26:01.569 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:01.570 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:01.570 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:01.570 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.570 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.570 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:108 3.b 3:d5c90217:::benchmark_data_smithi177_31116_object107:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T11:26:01.570 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:48.315513+0000", 2024-09-13T11:26:01.570 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 193.15735606999999, 2024-09-13T11:26:01.571 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.87544772500000001, 2024-09-13T11:26:01.571 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:26:01.571 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:26:01.571 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:01.571 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:26:01.571 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:26:01.572 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:01.572 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 108 2024-09-13T11:26:01.572 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.572 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:26:01.572 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.572 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:26:01.573 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315513+0000", 2024-09-13T11:26:01.573 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:01.573 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.573 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.573 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:26:01.573 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315513+0000", 2024-09-13T11:26:01.573 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:01.573 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.573 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.573 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:26:01.574 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315515+0000", 2024-09-13T11:26:01.574 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.9369999999999998e-06 2024-09-13T11:26:01.574 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.574 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.574 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:26:01.574 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315644+0000", 2024-09-13T11:26:01.574 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000129011 2024-09-13T11:26:01.574 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.574 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.574 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:26:01.574 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315657+0000", 2024-09-13T11:26:01.575 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.3689999999999999e-05 2024-09-13T11:26:01.575 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.575 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.575 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:26:01.575 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315668+0000", 2024-09-13T11:26:01.575 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0589e-05 2024-09-13T11:26:01.575 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.576 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.576 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:01.576 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316925+0000", 2024-09-13T11:26:01.576 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0012568340000000001 2024-09-13T11:26:01.576 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.576 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.576 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-09-13T11:26:01.576 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316932+0000", 2024-09-13T11:26:01.576 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.2230000000000001e-06 2024-09-13T11:26:01.576 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.576 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.576 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:01.576 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.186619+0000", 2024-09-13T11:26:01.577 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.86968679599999998 2024-09-13T11:26:01.577 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.577 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.577 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:26:01.577 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.186914+0000", 2024-09-13T11:26:01.577 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00029528900000000001 2024-09-13T11:26:01.577 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.577 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.577 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1", 2024-09-13T11:26:01.577 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.187217+0000", 2024-09-13T11:26:01.577 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00030232099999999998 2024-09-13T11:26:01.577 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.578 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.578 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:26:01.578 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.190532+0000", 2024-09-13T11:26:01.578 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.003315391 2024-09-13T11:26:01.578 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.578 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.578 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:01.578 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.190875+0000", 2024-09-13T11:26:01.578 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00034326300000000001 2024-09-13T11:26:01.578 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.578 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.578 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:26:01.578 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.190925+0000", 2024-09-13T11:26:01.578 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.9429999999999999e-05 2024-09-13T11:26:01.579 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.579 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.579 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:26:01.579 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.190961+0000", 2024-09-13T11:26:01.579 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.5951000000000002e-05 2024-09-13T11:26:01.579 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:01.579 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:01.579 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:01.579 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.579 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.579 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:172 3.b 3:dbec1eeb:::benchmark_data_smithi177_31116_object171:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T11:26:01.579 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:48.315684+0000", 2024-09-13T11:26:01.579 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 193.15718491300001, 2024-09-13T11:26:01.579 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.87615312999999995, 2024-09-13T11:26:01.580 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:26:01.580 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:26:01.580 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:01.580 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:26:01.580 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:26:01.580 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:01.580 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 172 2024-09-13T11:26:01.580 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.580 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:26:01.580 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.580 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:26:01.580 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315684+0000", 2024-09-13T11:26:01.581 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:01.581 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.581 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.581 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:26:01.581 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315684+0000", 2024-09-13T11:26:01.581 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:01.581 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.581 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.581 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:26:01.581 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315686+0000", 2024-09-13T11:26:01.581 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.886e-06 2024-09-13T11:26:01.582 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.582 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.582 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:26:01.582 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315820+0000", 2024-09-13T11:26:01.582 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00013388500000000001 2024-09-13T11:26:01.582 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.582 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.582 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:26:01.582 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315837+0000", 2024-09-13T11:26:01.582 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.6756e-05 2024-09-13T11:26:01.583 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.583 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.583 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:26:01.583 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315851+0000", 2024-09-13T11:26:01.583 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.4566000000000001e-05 2024-09-13T11:26:01.583 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.583 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.583 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:01.583 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.317021+0000", 2024-09-13T11:26:01.583 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0011701769999999999 2024-09-13T11:26:01.583 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.583 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.583 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-09-13T11:26:01.583 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.317037+0000", 2024-09-13T11:26:01.584 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.609e-05 2024-09-13T11:26:01.584 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.584 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.584 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:01.584 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.188634+0000", 2024-09-13T11:26:01.584 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.87159711900000003 2024-09-13T11:26:01.584 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.584 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.584 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:26:01.584 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.188974+0000", 2024-09-13T11:26:01.584 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00033922299999999999 2024-09-13T11:26:01.584 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.584 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.584 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1", 2024-09-13T11:26:01.585 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.189510+0000", 2024-09-13T11:26:01.585 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00053618999999999997 2024-09-13T11:26:01.585 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.585 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.585 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:01.585 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.191027+0000", 2024-09-13T11:26:01.585 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0015168390000000001 2024-09-13T11:26:01.585 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.585 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.585 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:26:01.585 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.191766+0000", 2024-09-13T11:26:01.585 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000739064 2024-09-13T11:26:01.585 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.586 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.586 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:26:01.586 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.191810+0000", 2024-09-13T11:26:01.587 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.3755999999999999e-05 2024-09-13T11:26:01.588 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.588 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.588 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:26:01.588 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.191837+0000", 2024-09-13T11:26:01.588 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.7579000000000001e-05 2024-09-13T11:26:01.588 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:01.588 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:01.588 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:01.588 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.588 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.588 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:177 3.b 3:de89c779:::benchmark_data_smithi177_31116_object176:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T11:26:01.588 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:48.315898+0000", 2024-09-13T11:26:01.589 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 193.15697059799999, 2024-09-13T11:26:01.589 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.87600712599999997, 2024-09-13T11:26:01.589 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:26:01.589 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:26:01.589 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:01.589 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:26:01.589 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:26:01.589 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:01.589 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 177 2024-09-13T11:26:01.589 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.589 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:26:01.589 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.589 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:26:01.589 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315898+0000", 2024-09-13T11:26:01.590 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:01.590 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.590 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.590 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:26:01.590 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315898+0000", 2024-09-13T11:26:01.590 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:01.590 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.590 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.590 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:26:01.590 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315900+0000", 2024-09-13T11:26:01.590 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.4929999999999999e-06 2024-09-13T11:26:01.590 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.590 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.590 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:26:01.591 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316027+0000", 2024-09-13T11:26:01.591 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000127568 2024-09-13T11:26:01.591 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.591 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.591 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:26:01.591 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316048+0000", 2024-09-13T11:26:01.591 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.0795999999999999e-05 2024-09-13T11:26:01.591 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.591 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.591 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:26:01.591 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316061+0000", 2024-09-13T11:26:01.591 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.3283e-05 2024-09-13T11:26:01.591 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.591 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.592 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:01.592 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.317107+0000", 2024-09-13T11:26:01.592 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0010452949999999999 2024-09-13T11:26:01.592 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.592 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.592 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-09-13T11:26:01.592 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.317114+0000", 2024-09-13T11:26:01.592 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.4569999999999999e-06 2024-09-13T11:26:01.592 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.592 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.592 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:01.592 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.190017+0000", 2024-09-13T11:26:01.593 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.87290258499999995 2024-09-13T11:26:01.593 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.593 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.593 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:26:01.593 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.190138+0000", 2024-09-13T11:26:01.593 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000121418 2024-09-13T11:26:01.593 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.593 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.593 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1", 2024-09-13T11:26:01.593 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.190216+0000", 2024-09-13T11:26:01.593 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.7410999999999995e-05 2024-09-13T11:26:01.593 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.594 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.594 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:01.594 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.191755+0000", 2024-09-13T11:26:01.594 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0015395420000000001 2024-09-13T11:26:01.594 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.594 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.594 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:26:01.594 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.191843+0000", 2024-09-13T11:26:01.594 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.8022999999999994e-05 2024-09-13T11:26:01.594 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.594 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.594 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:26:01.594 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.191876+0000", 2024-09-13T11:26:01.594 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.2619000000000001e-05 2024-09-13T11:26:01.595 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.595 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.595 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:26:01.595 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.191905+0000", 2024-09-13T11:26:01.595 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.9635999999999999e-05 2024-09-13T11:26:01.595 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:01.595 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:01.595 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:01.595 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.595 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.595 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:187 3.b 3:da03c8dd:::benchmark_data_smithi177_31116_object186:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T11:26:01.595 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:48.316083+0000", 2024-09-13T11:26:01.595 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 193.15678564199999, 2024-09-13T11:26:01.596 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.87835499500000003, 2024-09-13T11:26:01.596 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:26:01.596 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:26:01.596 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:01.596 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:26:01.596 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:26:01.596 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:01.596 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 187 2024-09-13T11:26:01.596 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.596 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:26:01.596 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.596 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:26:01.597 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316083+0000", 2024-09-13T11:26:01.597 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:01.597 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.597 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.597 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:26:01.597 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316083+0000", 2024-09-13T11:26:01.597 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:01.597 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.597 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.597 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:26:01.597 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316086+0000", 2024-09-13T11:26:01.598 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.5560000000000002e-06 2024-09-13T11:26:01.598 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.598 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.598 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:26:01.598 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316199+0000", 2024-09-13T11:26:01.598 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000113522 2024-09-13T11:26:01.598 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.598 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.598 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:26:01.598 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316215+0000", 2024-09-13T11:26:01.598 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.5265999999999999e-05 2024-09-13T11:26:01.598 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.598 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.599 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:26:01.599 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316226+0000", 2024-09-13T11:26:01.599 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0967e-05 2024-09-13T11:26:01.599 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.599 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.599 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:01.599 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.317198+0000", 2024-09-13T11:26:01.599 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00097194100000000004 2024-09-13T11:26:01.599 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.599 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.599 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-09-13T11:26:01.599 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.317213+0000", 2024-09-13T11:26:01.599 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.5472999999999999e-05 2024-09-13T11:26:01.599 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.600 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.600 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:01.600 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.191077+0000", 2024-09-13T11:26:01.600 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.87386435200000001 2024-09-13T11:26:01.600 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.600 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.600 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:26:01.600 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.191181+0000", 2024-09-13T11:26:01.600 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000103618 2024-09-13T11:26:01.600 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.600 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.600 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1", 2024-09-13T11:26:01.600 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.191305+0000", 2024-09-13T11:26:01.601 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00012358599999999999 2024-09-13T11:26:01.601 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.601 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.601 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:01.601 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.194279+0000", 2024-09-13T11:26:01.601 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0029746970000000001 2024-09-13T11:26:01.601 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.601 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.601 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:26:01.601 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.194336+0000", 2024-09-13T11:26:01.601 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.7244000000000003e-05 2024-09-13T11:26:01.601 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.601 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.601 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:26:01.602 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.194397+0000", 2024-09-13T11:26:01.602 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.0278000000000001e-05 2024-09-13T11:26:01.602 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.602 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.602 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:26:01.602 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.194438+0000", 2024-09-13T11:26:01.602 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.1495000000000002e-05 2024-09-13T11:26:01.602 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:01.602 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:01.602 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:01.602 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.602 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.602 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:200 3.b 3:d90039f1:::benchmark_data_smithi177_31116_object199:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T11:26:01.603 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:48.316280+0000", 2024-09-13T11:26:01.603 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 193.156589177, 2024-09-13T11:26:01.603 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.87879273000000002, 2024-09-13T11:26:01.603 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:26:01.603 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:26:01.603 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:01.603 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:26:01.603 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:26:01.603 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:01.603 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 200 2024-09-13T11:26:01.603 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.603 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:26:01.603 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.603 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:26:01.604 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316280+0000", 2024-09-13T11:26:01.604 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:01.604 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.604 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.604 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:26:01.604 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316280+0000", 2024-09-13T11:26:01.604 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:01.604 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.604 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.604 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:26:01.604 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316282+0000", 2024-09-13T11:26:01.604 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.497e-06 2024-09-13T11:26:01.604 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.604 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.605 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:26:01.605 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316411+0000", 2024-09-13T11:26:01.605 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00012924799999999999 2024-09-13T11:26:01.605 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.605 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.605 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:26:01.605 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316426+0000", 2024-09-13T11:26:01.605 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.4183e-05 2024-09-13T11:26:01.605 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.605 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.605 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:26:01.605 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316435+0000", 2024-09-13T11:26:01.605 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.0990000000000005e-06 2024-09-13T11:26:01.606 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.606 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.606 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:01.606 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.317250+0000", 2024-09-13T11:26:01.606 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00081501100000000003 2024-09-13T11:26:01.606 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.606 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.606 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-09-13T11:26:01.606 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.317257+0000", 2024-09-13T11:26:01.606 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.2169999999999997e-06 2024-09-13T11:26:01.606 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.606 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.606 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:01.607 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.191936+0000", 2024-09-13T11:26:01.607 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.87467922099999995 2024-09-13T11:26:01.607 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.607 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.607 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:26:01.607 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.192056+0000", 2024-09-13T11:26:01.607 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000119638 2024-09-13T11:26:01.607 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.607 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.607 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1", 2024-09-13T11:26:01.607 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.192184+0000", 2024-09-13T11:26:01.607 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00012813399999999999 2024-09-13T11:26:01.607 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.607 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.608 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:01.608 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.194324+0000", 2024-09-13T11:26:01.608 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0021399420000000001 2024-09-13T11:26:01.608 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.608 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.608 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:26:01.608 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.194802+0000", 2024-09-13T11:26:01.608 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00047798799999999999 2024-09-13T11:26:01.608 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.608 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.608 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:26:01.608 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.194951+0000", 2024-09-13T11:26:01.608 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0001494 2024-09-13T11:26:01.609 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.609 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.609 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:26:01.609 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.195072+0000", 2024-09-13T11:26:01.609 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00012115199999999999 2024-09-13T11:26:01.609 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:01.609 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:01.609 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:01.609 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.609 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.609 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop(client.4665.0:79801 3.5 e102/67 3:a6efdf84:::benchmark_data_smithi177_31116_object79800:head v 102'3851, mlcod=102'3848)", 2024-09-13T11:26:01.609 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:24:00.403754+0000", 2024-09-13T11:26:01.609 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 121.069114825, 2024-09-13T11:26:01.609 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.31749079899999999, 2024-09-13T11:26:01.610 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:26:01.610 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:26:01.610 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:01.610 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:26:01.610 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.610 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:26:01.610 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:00.403754+0000", 2024-09-13T11:26:01.610 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:01.610 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.610 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.610 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:26:01.610 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:00.403754+0000", 2024-09-13T11:26:01.610 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:01.611 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.611 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.611 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:26:01.611 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:00.403758+0000", 2024-09-13T11:26:01.611 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.507e-06 2024-09-13T11:26:01.611 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.611 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.611 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:26:01.611 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:00.403819+0000", 2024-09-13T11:26:01.611 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.1042000000000003e-05 2024-09-13T11:26:01.611 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.611 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.611 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:26:01.611 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:00.717963+0000", 2024-09-13T11:26:01.612 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.314144594 2024-09-13T11:26:01.612 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.612 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.612 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:26:01.612 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:00.717976+0000", 2024-09-13T11:26:01.612 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.3029e-05 2024-09-13T11:26:01.612 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.612 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.612 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:01.612 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:00.718143+0000", 2024-09-13T11:26:01.612 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00016639000000000001 2024-09-13T11:26:01.612 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.612 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.613 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:26:01.613 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:00.718184+0000", 2024-09-13T11:26:01.613 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.1801999999999997e-05 2024-09-13T11:26:01.613 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.613 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.613 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:26:01.613 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:00.721162+0000", 2024-09-13T11:26:01.613 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0029772800000000001 2024-09-13T11:26:01.613 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.613 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.613 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:26:01.613 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:00.721245+0000", 2024-09-13T11:26:01.614 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.3154999999999999e-05 2024-09-13T11:26:01.614 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:01.614 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:01.614 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:01.614 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.614 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.614 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "MOSDPGPushReply(3.26 121/121 [PushReplyOp(3:670b9aab:::benchmark_data_smithi177_31116_object93:head)])", 2024-09-13T11:26:01.614 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:24:27.300318+0000", 2024-09-13T11:26:01.614 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 94.172550956999999, 2024-09-13T11:26:01.614 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.811436728, 2024-09-13T11:26:01.614 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:26:01.614 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:26:01.614 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "reached pg", 2024-09-13T11:26:01.615 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:26:01.615 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.615 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:26:01.615 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:27.300318+0000", 2024-09-13T11:26:01.615 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:01.615 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.615 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.615 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:26:01.615 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:27.300318+0000", 2024-09-13T11:26:01.615 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:01.615 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.615 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.615 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:26:01.616 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:27.300320+0000", 2024-09-13T11:26:01.616 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.778e-06 2024-09-13T11:26:01.616 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.616 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.616 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:26:01.616 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:27.300326+0000", 2024-09-13T11:26:01.616 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.7700000000000004e-06 2024-09-13T11:26:01.616 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.616 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.616 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:26:01.616 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:28.109665+0000", 2024-09-13T11:26:01.616 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.80933853499999997 2024-09-13T11:26:01.616 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.617 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.617 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:26:01.617 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:28.109678+0000", 2024-09-13T11:26:01.617 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.2870999999999999e-05 2024-09-13T11:26:01.617 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.617 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.617 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:01.617 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:28.109800+0000", 2024-09-13T11:26:01.617 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000122328 2024-09-13T11:26:01.617 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.617 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.617 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:26:01.617 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:28.111755+0000", 2024-09-13T11:26:01.618 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0019544459999999999 2024-09-13T11:26:01.618 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:01.618 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:01.618 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:01.618 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.618 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.618 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:184026 3.b 3:d3052347:::benchmark_data_smithi177_31116_object43723:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e171)", 2024-09-13T11:26:01.618 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:25:32.645242+0000", 2024-09-13T11:26:01.618 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 28.827626917, 2024-09-13T11:26:01.618 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.348925019, 2024-09-13T11:26:01.618 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:26:01.618 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:26:01.618 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:01.619 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:26:01.619 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:26:01.619 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:01.619 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 184026 2024-09-13T11:26:01.619 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.619 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:26:01.619 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.619 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:26:01.619 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:32.645242+0000", 2024-09-13T11:26:01.619 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:01.619 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.619 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.619 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:26:01.619 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:32.645242+0000", 2024-09-13T11:26:01.620 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:01.620 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.620 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.620 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:26:01.620 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:32.645244+0000", 2024-09-13T11:26:01.620 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.368e-06 2024-09-13T11:26:01.620 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.620 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.620 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:26:01.620 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:32.645251+0000", 2024-09-13T11:26:01.620 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.2519999999999996e-06 2024-09-13T11:26:01.620 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.620 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.621 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:26:01.621 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:32.645267+0000", 2024-09-13T11:26:01.621 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.6529e-05 2024-09-13T11:26:01.621 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.621 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.621 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:26:01.621 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:32.645278+0000", 2024-09-13T11:26:01.621 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.119e-05 2024-09-13T11:26:01.621 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.621 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.621 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:01.621 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:32.645348+0000", 2024-09-13T11:26:01.621 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.9333999999999997e-05 2024-09-13T11:26:01.622 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.622 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.622 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:26:01.622 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:32.645479+0000", 2024-09-13T11:26:01.622 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00013153199999999999 2024-09-13T11:26:01.622 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.622 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.622 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1", 2024-09-13T11:26:01.622 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:32.645599+0000", 2024-09-13T11:26:01.622 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000119481 2024-09-13T11:26:01.622 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.622 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.622 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:26:01.623 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:32.648042+0000", 2024-09-13T11:26:01.623 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0024437059999999999 2024-09-13T11:26:01.623 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.623 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.623 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:01.623 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:32.994082+0000", 2024-09-13T11:26:01.623 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.34604006399999998 2024-09-13T11:26:01.623 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.623 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.623 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:26:01.623 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:32.994124+0000", 2024-09-13T11:26:01.623 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.1573999999999999e-05 2024-09-13T11:26:01.623 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:01.623 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:01.624 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:26:01.624 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:32.994167+0000", 2024-09-13T11:26:01.624 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.2988999999999998e-05 2024-09-13T11:26:01.624 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:01.624 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:01.624 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:01.624 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:01.624 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:01.624 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-09-13T11:26:01.625 INFO:teuthology.orchestra.run.smithi001.stdout:osd.7: {} 2024-09-13T11:26:01.626 INFO:teuthology.orchestra.run.smithi001.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T11:26:01.626 INFO:teuthology.orchestra.run.smithi001.stdout:osd.8: {} 2024-09-13T11:26:01.627 INFO:teuthology.orchestra.run.smithi001.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-13T11:26:01.627 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-13T11:26:01.628 INFO:teuthology.orchestra.run.smithi001.stdout:osd.9: {} 2024-09-13T11:26:01.628 INFO:teuthology.orchestra.run.smithi001.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-13T11:26:01.628 INFO:teuthology.orchestra.run.smithi001.stdout:osd.10: {} 2024-09-13T11:26:01.628 INFO:teuthology.orchestra.run.smithi001.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-13T11:26:01.629 INFO:teuthology.orchestra.run.smithi001.stdout:osd.11: {} 2024-09-13T11:26:01.629 INFO:teuthology.orchestra.run.smithi001.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-13T11:26:01.629 INFO:teuthology.orchestra.run.smithi001.stdout:osd.12: {} 2024-09-13T11:26:01.630 INFO:teuthology.orchestra.run.smithi001.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-13T11:26:01.630 INFO:tasks.ceph.osd.8.smithi001.stderr:2024-09-13T11:26:01.622+0000 7fb9b48ae640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26312) UID: 0 2024-09-13T11:26:01.635 INFO:teuthology.orchestra.run.smithi001.stdout:osd.13: {} 2024-09-13T11:26:01.635 INFO:teuthology.orchestra.run.smithi001.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-13T11:26:01.660 INFO:teuthology.orchestra.run.smithi001.stdout:osd.14: {} 2024-09-13T11:26:01.660 INFO:teuthology.orchestra.run.smithi001.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-13T11:26:01.687 INFO:teuthology.orchestra.run.smithi001.stdout:osd.15: {} 2024-09-13T11:26:01.687 INFO:teuthology.orchestra.run.smithi001.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-13T11:26:01.726 INFO:tasks.ceph.osd.14.smithi162.stderr:2024-09-13T11:26:01.723+0000 7f3df5903640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33154) UID: 0 2024-09-13T11:26:01.766 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-13T11:26:01.766 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:26:01.781 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-13T11:26:01.827 INFO:tasks.ceph.osd.9.smithi092.stderr:2024-09-13T11:26:01.824+0000 7ff5ab3d0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26348) UID: 0 2024-09-13T11:26:01.928 INFO:tasks.ceph.osd.11.smithi177.stderr:2024-09-13T11:26:01.925+0000 7f24792c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26580) UID: 0 2024-09-13T11:26:01.957 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-13T11:26:01.958 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:26:01.973 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-13T11:26:01.996 INFO:teuthology.orchestra.run.smithi001.stderr:noscrub is set 2024-09-13T11:26:02.001 INFO:tasks.ceph.osd.4.smithi001.stderr:2024-09-13T11:26:01.998+0000 7ff8d0c3d640 -1 osd.4 188 handle_osd_map injecting map crc failure 2024-09-13T11:26:02.029 INFO:tasks.ceph.osd.3.smithi177.stderr:2024-09-13T11:26:02.025+0000 7f10adba4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26574) UID: 0 2024-09-13T11:26:02.056 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-09-13T11:26:02.051+0000 7f97b39f6640 -1 osd.6 188 handle_osd_map injecting map crc failure 2024-09-13T11:26:02.100 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-13T11:26:02.100 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:26:02.117 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-13T11:26:02.130 INFO:tasks.ceph.osd.7.smithi177.stderr:2024-09-13T11:26:02.125+0000 7f897b51d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26575) UID: 0 2024-09-13T11:26:02.231 INFO:tasks.ceph.osd.11.smithi177.stderr:2024-09-13T11:26:02.229+0000 7f24792c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26580) UID: 0 2024-09-13T11:26:02.244 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-13T11:26:02.244 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:26:02.260 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-13T11:26:02.331 INFO:tasks.ceph.osd.7.smithi177.stderr:2024-09-13T11:26:02.329+0000 7f897b51d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26575) UID: 0 2024-09-13T11:26:02.397 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-13T11:26:02.397 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:26:02.413 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-13T11:26:02.457 INFO:tasks.ceph.osd.2.smithi162.stderr:2024-09-13T11:26:02.455+0000 7fe92b35a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26313) UID: 0 2024-09-13T11:26:02.534 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-13T11:26:02.530+0000 7f6a68775640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26307) UID: 0 2024-09-13T11:26:02.546 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-13T11:26:02.546 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:26:02.561 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-09-13T11:26:02.635 INFO:tasks.ceph.osd.8.smithi001.stderr:2024-09-13T11:26:02.630+0000 7fb9b48ae640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26312) UID: 0 2024-09-13T11:26:02.735 INFO:tasks.ceph.osd.5.smithi092.stderr:2024-09-13T11:26:02.732+0000 7f6f0b705640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26346) UID: 0 2024-09-13T11:26:02.744 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-13T11:26:02.744 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:26:02.761 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-09-13T11:26:02.837 INFO:tasks.ceph.osd.9.smithi092.stderr:2024-09-13T11:26:02.836+0000 7ff5ab3d0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26348) UID: 0 2024-09-13T11:26:02.895 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-13T11:26:02.896 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:26:02.913 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-09-13T11:26:02.938 INFO:tasks.ceph.osd.5.smithi092.stderr:2024-09-13T11:26:02.936+0000 7f6f0b705640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26346) UID: 0 2024-09-13T11:26:03.038 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-09-13T11:26:03.035+0000 7f97befc9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26315) UID: 0 2024-09-13T11:26:03.049 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-13T11:26:03.049 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:26:03.056 INFO:tasks.ceph.osd.2.smithi162.stderr:2024-09-13T11:26:03.055+0000 7fe921d8b640 -1 osd.2 189 handle_osd_map injecting map crc failure 2024-09-13T11:26:03.066 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-13T11:26:03.140 INFO:tasks.ceph.osd.2.smithi162.stderr:2024-09-13T11:26:03.135+0000 7fe92b35a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26313) UID: 0 2024-09-13T11:26:03.229 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-09-13T11:26:03.230 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:26:03.241 INFO:tasks.ceph.osd.14.smithi162.stderr:2024-09-13T11:26:03.239+0000 7f3df5903640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33154) UID: 0 2024-09-13T11:26:03.251 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-13T11:26:03.343 INFO:tasks.ceph.osd.2.smithi162.stderr:2024-09-13T11:26:03.339+0000 7fe92b35a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26313) UID: 0 2024-09-13T11:26:03.372 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-09-13T11:26:03.373 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:26:03.386 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-13T11:26:03.444 INFO:tasks.ceph.osd.13.smithi092.stderr:2024-09-13T11:26:03.440+0000 7f3e5afea640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26350) UID: 0 2024-09-13T11:26:03.506 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-09-13T11:26:03.506 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:26:03.521 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-13T11:26:03.545 INFO:tasks.ceph.osd.12.smithi001.stderr:2024-09-13T11:26:03.542+0000 7f892c6e6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26314) UID: 0 2024-09-13T11:26:03.639 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-09-13T11:26:03.639 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:26:03.643 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 15] out_osds: [14] dead_osds: [] live_osds: [0, 4, 8, 12, 5, 9, 13, 2, 6, 10, 3, 7, 11, 15, 1, 14] 2024-09-13T11:26:03.643 INFO:tasks.thrashosds.thrasher:choose_action: min_in 2 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-13T11:26:03.644 INFO:tasks.thrashosds.thrasher:Adding osd 14 2024-09-13T11:26:03.644 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd in 14 2024-09-13T11:26:03.645 INFO:tasks.ceph.osd.5.smithi092.stderr:2024-09-13T11:26:03.644+0000 7f6f0b705640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26346) UID: 0 2024-09-13T11:26:03.653 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-13T11:26:03.707 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T11:26:03.747 INFO:tasks.ceph.osd.9.smithi092.stderr:2024-09-13T11:26:03.744+0000 7ff5ab3d0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26348) UID: 0 2024-09-13T11:26:03.779 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-09-13T11:26:03.779 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:26:03.795 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-13T11:26:03.847 INFO:tasks.ceph.osd.11.smithi177.stderr:2024-09-13T11:26:03.845+0000 7f24792c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26580) UID: 0 2024-09-13T11:26:03.856 INFO:teuthology.orchestra.run.smithi001.stdout:osd.0: {} 2024-09-13T11:26:03.856 INFO:teuthology.orchestra.run.smithi001.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T11:26:03.882 INFO:teuthology.orchestra.run.smithi001.stdout:osd.1: {} 2024-09-13T11:26:03.883 INFO:teuthology.orchestra.run.smithi001.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T11:26:03.908 INFO:teuthology.orchestra.run.smithi001.stdout:osd.2: {} 2024-09-13T11:26:03.908 INFO:teuthology.orchestra.run.smithi001.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T11:26:03.918 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-09-13T11:26:03.918 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:26:03.932 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-09-13T11:26:03.933 INFO:teuthology.orchestra.run.smithi001.stdout:osd.3: {} 2024-09-13T11:26:03.933 INFO:teuthology.orchestra.run.smithi001.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T11:26:03.948 INFO:tasks.ceph.osd.12.smithi001.stderr:2024-09-13T11:26:03.946+0000 7f892c6e6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26314) UID: 0 2024-09-13T11:26:03.955 INFO:teuthology.orchestra.run.smithi001.stdout:osd.4: {} 2024-09-13T11:26:03.955 INFO:teuthology.orchestra.run.smithi001.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T11:26:03.980 INFO:teuthology.orchestra.run.smithi001.stdout:osd.5: {} 2024-09-13T11:26:03.981 INFO:teuthology.orchestra.run.smithi001.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T11:26:03.984 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-09-13T11:26:03.982+0000 7fb2a5596640 -1 mon.a@0(leader).osd e190 definitely_dead 0 2024-09-13T11:26:04.008 INFO:teuthology.orchestra.run.smithi001.stdout:osd.6: {} 2024-09-13T11:26:04.009 INFO:teuthology.orchestra.run.smithi001.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T11:26:04.015 INFO:teuthology.orchestra.run.smithi001.stderr:marked in osd.14. 2024-09-13T11:26:04.024 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-13T11:26:04.027 INFO:tasks.ceph.osd.10.smithi162.stderr:2024-09-13T11:26:04.023+0000 7fb59141d640 -1 osd.10 190 handle_osd_map injecting map crc failure 2024-09-13T11:26:04.036 INFO:teuthology.orchestra.run.smithi001.stdout:osd.7: {} 2024-09-13T11:26:04.036 INFO:teuthology.orchestra.run.smithi001.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T11:26:04.038 INFO:tasks.thrashosds.thrasher:Added osd 14 2024-09-13T11:26:04.059 INFO:teuthology.orchestra.run.smithi001.stdout:osd.8: {} 2024-09-13T11:26:04.059 INFO:teuthology.orchestra.run.smithi001.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-13T11:26:04.059 INFO:tasks.ceph.osd.8.smithi001.stderr:2024-09-13T11:26:04.054+0000 7fb9b48ae640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26312) UID: 0 2024-09-13T11:26:04.084 INFO:teuthology.orchestra.run.smithi001.stdout:osd.9: {} 2024-09-13T11:26:04.084 INFO:teuthology.orchestra.run.smithi001.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-13T11:26:04.096 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-09-13T11:26:04.096 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:26:04.109 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-09-13T11:26:04.110 INFO:teuthology.orchestra.run.smithi001.stdout:osd.10: {} 2024-09-13T11:26:04.110 INFO:teuthology.orchestra.run.smithi001.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-13T11:26:04.137 INFO:teuthology.orchestra.run.smithi001.stdout:osd.11: {} 2024-09-13T11:26:04.137 INFO:teuthology.orchestra.run.smithi001.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-13T11:26:04.149 INFO:tasks.ceph.osd.4.smithi001.stderr:2024-09-13T11:26:04.146+0000 7ff8e0218640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26310) UID: 0 2024-09-13T11:26:04.165 INFO:teuthology.orchestra.run.smithi001.stdout:osd.12: {} 2024-09-13T11:26:04.165 INFO:teuthology.orchestra.run.smithi001.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-13T11:26:04.192 INFO:teuthology.orchestra.run.smithi001.stdout:osd.13: {} 2024-09-13T11:26:04.193 INFO:teuthology.orchestra.run.smithi001.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-13T11:26:04.220 INFO:teuthology.orchestra.run.smithi001.stdout:osd.14: {} 2024-09-13T11:26:04.220 INFO:teuthology.orchestra.run.smithi001.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-13T11:26:04.226 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-09-13T11:26:04.226 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [], 2024-09-13T11:26:04.226 INFO:teuthology.orchestra.run.smithi177.stdout: "complaint_time": 30, 2024-09-13T11:26:04.226 INFO:teuthology.orchestra.run.smithi177.stdout: "num_blocked_ops": 0 2024-09-13T11:26:04.226 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-09-13T11:26:04.240 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-09-13T11:26:04.245 INFO:teuthology.orchestra.run.smithi001.stdout:osd.15: {} 2024-09-13T11:26:04.245 INFO:teuthology.orchestra.run.smithi001.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-13T11:26:04.253 INFO:tasks.ceph.osd.10.smithi162.stderr:2024-09-13T11:26:04.251+0000 7fb59c9f0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26317) UID: 0 2024-09-13T11:26:04.354 INFO:tasks.ceph.osd.2.smithi162.stderr:2024-09-13T11:26:04.351+0000 7fe92b35a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26313) UID: 0 2024-09-13T11:26:04.368 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-09-13T11:26:04.368 INFO:teuthology.orchestra.run.smithi177.stdout: "size": 20, 2024-09-13T11:26:04.368 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 600, 2024-09-13T11:26:04.368 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [ 2024-09-13T11:26:04.369 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.369 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:79 3.1 e32/30 3:841df5b5:::benchmark_data_smithi177_31116_object78:head v 32'6, mlcod=32'4)", 2024-09-13T11:26:04.369 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:22:27.057292+0000", 2024-09-13T11:26:04.369 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 217.30906295899999, 2024-09-13T11:26:04.369 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.89296786800000005, 2024-09-13T11:26:04.369 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:26:04.369 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:26:04.369 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:04.369 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:26:04.369 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.369 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:26:04.369 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.057292+0000", 2024-09-13T11:26:04.369 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:04.370 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.370 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.370 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:26:04.370 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.057292+0000", 2024-09-13T11:26:04.370 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:04.370 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.371 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.371 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:26:04.371 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.057293+0000", 2024-09-13T11:26:04.371 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.6109999999999999e-06 2024-09-13T11:26:04.371 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.371 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.371 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:26:04.371 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.057330+0000", 2024-09-13T11:26:04.371 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.6721999999999998e-05 2024-09-13T11:26:04.372 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.372 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.372 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:26:04.372 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.948197+0000", 2024-09-13T11:26:04.372 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.890867042 2024-09-13T11:26:04.372 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.372 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.372 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:26:04.372 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.948213+0000", 2024-09-13T11:26:04.372 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5826000000000001e-05 2024-09-13T11:26:04.372 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.372 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.373 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:26:04.373 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.948377+0000", 2024-09-13T11:26:04.373 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00016348600000000001 2024-09-13T11:26:04.373 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.373 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.373 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:26:04.373 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.948447+0000", 2024-09-13T11:26:04.373 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.0891999999999998e-05 2024-09-13T11:26:04.373 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.373 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.373 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:26:04.373 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.950150+0000", 2024-09-13T11:26:04.373 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001702329 2024-09-13T11:26:04.373 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.374 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.374 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:26:04.374 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.950260+0000", 2024-09-13T11:26:04.374 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00010996 2024-09-13T11:26:04.374 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:04.374 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:26:04.374 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:04.374 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.374 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.374 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:30828 3.f 3:f3a9c6a9:::benchmark_data_smithi177_31116_object30827:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e59)", 2024-09-13T11:26:04.374 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:09.503661+0000", 2024-09-13T11:26:04.374 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 174.86269361000001, 2024-09-13T11:26:04.374 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.26477667100000002, 2024-09-13T11:26:04.374 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:26:04.375 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:26:04.375 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:04.375 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:26:04.375 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:26:04.375 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:04.375 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 30828 2024-09-13T11:26:04.375 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.375 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:26:04.375 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.375 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:26:04.375 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.503661+0000", 2024-09-13T11:26:04.375 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:04.375 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.376 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.376 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:26:04.376 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.503661+0000", 2024-09-13T11:26:04.376 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:04.376 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.376 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.376 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:26:04.376 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.503663+0000", 2024-09-13T11:26:04.376 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.968e-06 2024-09-13T11:26:04.376 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.376 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.376 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:26:04.376 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.503687+0000", 2024-09-13T11:26:04.376 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.3983999999999999e-05 2024-09-13T11:26:04.377 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.377 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.377 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:26:04.377 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.503701+0000", 2024-09-13T11:26:04.377 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3416e-05 2024-09-13T11:26:04.377 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.377 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.377 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:26:04.377 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.503708+0000", 2024-09-13T11:26:04.377 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.9530000000000002e-06 2024-09-13T11:26:04.377 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.377 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.377 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:26:04.377 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.503759+0000", 2024-09-13T11:26:04.378 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.1075999999999997e-05 2024-09-13T11:26:04.378 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.378 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.378 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:26:04.378 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.503861+0000", 2024-09-13T11:26:04.378 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000102372 2024-09-13T11:26:04.378 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.378 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.378 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 12", 2024-09-13T11:26:04.378 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.503952+0000", 2024-09-13T11:26:04.378 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.1135000000000001e-05 2024-09-13T11:26:04.378 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.379 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.379 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:26:04.379 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.505340+0000", 2024-09-13T11:26:04.379 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0013877169999999999 2024-09-13T11:26:04.379 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.379 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.379 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:04.379 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.768281+0000", 2024-09-13T11:26:04.379 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.26294092400000002 2024-09-13T11:26:04.379 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.379 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.380 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:26:04.380 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.768369+0000", 2024-09-13T11:26:04.380 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.7818999999999994e-05 2024-09-13T11:26:04.380 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.380 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.380 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:26:04.380 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:09.768438+0000", 2024-09-13T11:26:04.380 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.9307000000000006e-05 2024-09-13T11:26:04.380 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:04.380 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:26:04.380 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:04.380 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.380 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.380 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:45783 3.16 3.17c10a76 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e69)", 2024-09-13T11:26:04.381 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:20.947774+0000", 2024-09-13T11:26:04.381 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 163.41858070800001, 2024-09-13T11:26:04.381 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.96682922400000004, 2024-09-13T11:26:04.381 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:26:04.381 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:26:04.381 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-13T11:26:04.381 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:26:04.381 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:26:04.381 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:04.381 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 45783 2024-09-13T11:26:04.381 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.381 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:26:04.381 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.382 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:26:04.382 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:20.947774+0000", 2024-09-13T11:26:04.382 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:04.382 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.382 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.382 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:26:04.382 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:20.947774+0000", 2024-09-13T11:26:04.382 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:04.382 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.382 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.382 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:26:04.382 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:20.947777+0000", 2024-09-13T11:26:04.382 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.57e-06 2024-09-13T11:26:04.383 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.383 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.383 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:26:04.383 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:20.947802+0000", 2024-09-13T11:26:04.383 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.5528999999999999e-05 2024-09-13T11:26:04.383 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.383 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.383 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:26:04.383 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:20.947813+0000", 2024-09-13T11:26:04.383 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0473e-05 2024-09-13T11:26:04.383 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.383 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.383 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:26:04.383 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:20.947821+0000", 2024-09-13T11:26:04.384 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.3440000000000001e-06 2024-09-13T11:26:04.384 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.384 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.384 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:26:04.384 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:20.947933+0000", 2024-09-13T11:26:04.384 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00011210200000000001 2024-09-13T11:26:04.384 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.384 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.384 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for peered", 2024-09-13T11:26:04.384 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:20.947942+0000", 2024-09-13T11:26:04.384 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.6610000000000006e-06 2024-09-13T11:26:04.384 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.384 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.384 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:26:04.385 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:21.914579+0000", 2024-09-13T11:26:04.385 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.96663672300000003 2024-09-13T11:26:04.385 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.385 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.385 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:26:04.385 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:21.914603+0000", 2024-09-13T11:26:04.385 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.4822000000000001e-05 2024-09-13T11:26:04.385 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:04.385 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:26:04.385 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:04.385 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.385 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.385 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:54625 3.1f 3:f812cdfb:::benchmark_data_smithi177_31116_object54624:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e75)", 2024-09-13T11:26:04.386 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:30.979327+0000", 2024-09-13T11:26:04.386 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 153.38702791899999, 2024-09-13T11:26:04.386 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.96163670899999998, 2024-09-13T11:26:04.386 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:26:04.386 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:26:04.386 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-13T11:26:04.386 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:26:04.386 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:26:04.386 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:04.386 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 54625 2024-09-13T11:26:04.386 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.386 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:26:04.386 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.386 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:26:04.387 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.979327+0000", 2024-09-13T11:26:04.387 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:04.387 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.387 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.387 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:26:04.387 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.979327+0000", 2024-09-13T11:26:04.387 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:04.387 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.387 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.387 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:26:04.387 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.979329+0000", 2024-09-13T11:26:04.387 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.686e-06 2024-09-13T11:26:04.387 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.387 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.388 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:26:04.388 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.979364+0000", 2024-09-13T11:26:04.388 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.5750000000000002e-05 2024-09-13T11:26:04.388 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.388 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.388 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:26:04.388 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.979408+0000", 2024-09-13T11:26:04.388 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.3795999999999999e-05 2024-09-13T11:26:04.388 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.388 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.388 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:26:04.388 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.979427+0000", 2024-09-13T11:26:04.388 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.9040999999999999e-05 2024-09-13T11:26:04.389 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.389 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.389 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:26:04.389 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.122291+0000", 2024-09-13T11:26:04.389 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.14286373599999999 2024-09-13T11:26:04.389 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.389 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.389 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for peered", 2024-09-13T11:26:04.389 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.122298+0000", 2024-09-13T11:26:04.389 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.3719999999999999e-06 2024-09-13T11:26:04.389 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.389 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.389 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:26:04.389 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.940341+0000", 2024-09-13T11:26:04.390 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.81804249699999998 2024-09-13T11:26:04.390 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.390 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.390 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:26:04.390 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.940964+0000", 2024-09-13T11:26:04.390 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00062283100000000001 2024-09-13T11:26:04.390 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:04.390 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:26:04.390 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:04.390 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.390 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.390 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:54649 3.1c 3:3a457a93:::benchmark_data_smithi177_31116_object54648:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e75)", 2024-09-13T11:26:04.390 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:30.979460+0000", 2024-09-13T11:26:04.391 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 153.38689472900001, 2024-09-13T11:26:04.391 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.95579424000000002, 2024-09-13T11:26:04.391 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:26:04.391 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:26:04.391 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-13T11:26:04.391 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:26:04.391 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:26:04.391 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:04.391 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 54649 2024-09-13T11:26:04.391 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.391 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:26:04.391 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.391 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:26:04.392 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.979460+0000", 2024-09-13T11:26:04.392 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:04.392 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.392 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.392 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:26:04.392 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.979460+0000", 2024-09-13T11:26:04.392 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:04.392 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.392 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.392 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:26:04.392 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.979462+0000", 2024-09-13T11:26:04.392 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5799999999999999e-06 2024-09-13T11:26:04.392 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.392 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.393 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:26:04.393 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.979524+0000", 2024-09-13T11:26:04.393 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.2521999999999995e-05 2024-09-13T11:26:04.393 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.393 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.393 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:26:04.393 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.979561+0000", 2024-09-13T11:26:04.393 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.6349e-05 2024-09-13T11:26:04.393 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.393 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.393 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:26:04.393 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.979571+0000", 2024-09-13T11:26:04.393 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0457e-05 2024-09-13T11:26:04.393 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.394 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.394 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:26:04.394 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.121029+0000", 2024-09-13T11:26:04.394 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.14145766200000001 2024-09-13T11:26:04.394 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.394 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.394 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for peered", 2024-09-13T11:26:04.394 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.121039+0000", 2024-09-13T11:26:04.394 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0084e-05 2024-09-13T11:26:04.394 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.394 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.394 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:26:04.394 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.935157+0000", 2024-09-13T11:26:04.394 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.81411807899999999 2024-09-13T11:26:04.395 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.395 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.395 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:26:04.395 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.935254+0000", 2024-09-13T11:26:04.395 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.7507e-05 2024-09-13T11:26:04.395 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:04.395 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:26:04.395 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:04.395 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.395 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.395 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:54654 3.1c 3:3f0bb31a:::benchmark_data_smithi177_31116_object54653:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e75)", 2024-09-13T11:26:04.395 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:30.979665+0000", 2024-09-13T11:26:04.395 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 153.38668956399999, 2024-09-13T11:26:04.395 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.95623908099999999, 2024-09-13T11:26:04.396 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:26:04.396 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:26:04.396 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-13T11:26:04.396 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:26:04.396 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:26:04.396 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:04.396 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 54654 2024-09-13T11:26:04.396 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.396 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:26:04.396 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.396 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:26:04.396 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.979665+0000", 2024-09-13T11:26:04.396 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:04.396 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.397 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.397 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:26:04.397 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.979665+0000", 2024-09-13T11:26:04.397 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:04.397 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.397 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.397 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:26:04.397 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.979667+0000", 2024-09-13T11:26:04.397 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3540000000000001e-06 2024-09-13T11:26:04.397 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.397 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.397 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:26:04.397 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.979703+0000", 2024-09-13T11:26:04.398 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.6758000000000003e-05 2024-09-13T11:26:04.398 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.398 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.398 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:26:04.398 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.979714+0000", 2024-09-13T11:26:04.398 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0804e-05 2024-09-13T11:26:04.398 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.398 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.398 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:26:04.398 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.979724+0000", 2024-09-13T11:26:04.398 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0040999999999999e-05 2024-09-13T11:26:04.398 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.399 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.399 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:26:04.399 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.121061+0000", 2024-09-13T11:26:04.399 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.141336713 2024-09-13T11:26:04.399 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.399 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.399 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for peered", 2024-09-13T11:26:04.399 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.121067+0000", 2024-09-13T11:26:04.399 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.4150000000000004e-06 2024-09-13T11:26:04.399 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.399 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.399 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:26:04.399 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.935776+0000", 2024-09-13T11:26:04.400 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.81470863100000002 2024-09-13T11:26:04.400 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.400 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.400 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:26:04.400 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.935904+0000", 2024-09-13T11:26:04.400 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00012836499999999999 2024-09-13T11:26:04.400 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:04.400 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:26:04.400 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:04.400 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.400 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.400 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPushReply(3.11 75/70 [PushReplyOp(3:8df10771:::benchmark_data_smithi177_31116_object24734:head)])", 2024-09-13T11:26:04.400 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:31.149798+0000", 2024-09-13T11:26:04.400 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 153.21655655399999, 2024-09-13T11:26:04.401 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.32402702500000002, 2024-09-13T11:26:04.401 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:26:04.401 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:26:04.401 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-13T11:26:04.401 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:26:04.401 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.401 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:26:04.401 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.149798+0000", 2024-09-13T11:26:04.401 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:04.401 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.401 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.401 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:26:04.401 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.149798+0000", 2024-09-13T11:26:04.401 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:04.402 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.402 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.402 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:26:04.402 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.149799+0000", 2024-09-13T11:26:04.402 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.5700000000000001e-07 2024-09-13T11:26:04.402 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.402 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.402 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:26:04.402 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.149803+0000", 2024-09-13T11:26:04.402 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.0040000000000001e-06 2024-09-13T11:26:04.402 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.402 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.402 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:26:04.402 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.472172+0000", 2024-09-13T11:26:04.403 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.32236856699999999 2024-09-13T11:26:04.403 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.403 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.403 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:26:04.403 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.472187+0000", 2024-09-13T11:26:04.403 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5016e-05 2024-09-13T11:26:04.403 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.403 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.403 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:26:04.403 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.472302+0000", 2024-09-13T11:26:04.403 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000115121 2024-09-13T11:26:04.403 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.403 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.403 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:26:04.404 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.473825+0000", 2024-09-13T11:26:04.404 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0015234599999999999 2024-09-13T11:26:04.404 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:04.404 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:26:04.404 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:04.404 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.404 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.404 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:54668 3.1c 3:3d2c5e40:::benchmark_data_smithi177_31116_object54667:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e75)", 2024-09-13T11:26:04.404 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:31.245525+0000", 2024-09-13T11:26:04.404 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 153.120830297, 2024-09-13T11:26:04.404 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.69828973000000005, 2024-09-13T11:26:04.404 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:26:04.404 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:26:04.404 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:04.405 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:26:04.405 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:26:04.405 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:04.405 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 54668 2024-09-13T11:26:04.405 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.405 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:26:04.405 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.405 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:26:04.405 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.245525+0000", 2024-09-13T11:26:04.405 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:04.405 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.405 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.405 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:26:04.406 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.245525+0000", 2024-09-13T11:26:04.406 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:04.406 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.406 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.406 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:26:04.406 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.245526+0000", 2024-09-13T11:26:04.406 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.719e-06 2024-09-13T11:26:04.406 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.406 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.406 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:26:04.406 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.245547+0000", 2024-09-13T11:26:04.406 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.1084999999999999e-05 2024-09-13T11:26:04.406 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.407 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.407 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:26:04.407 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.245558+0000", 2024-09-13T11:26:04.407 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0934000000000001e-05 2024-09-13T11:26:04.407 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.407 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.407 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:26:04.407 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.245564+0000", 2024-09-13T11:26:04.407 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.6269999999999997e-06 2024-09-13T11:26:04.407 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.407 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.407 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:26:04.407 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.245620+0000", 2024-09-13T11:26:04.408 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.6289999999999998e-05 2024-09-13T11:26:04.408 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.408 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.408 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for peered", 2024-09-13T11:26:04.408 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.245632+0000", 2024-09-13T11:26:04.408 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.1748e-05 2024-09-13T11:26:04.408 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.408 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.408 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:26:04.408 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.935976+0000", 2024-09-13T11:26:04.408 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.69034455299999997 2024-09-13T11:26:04.408 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.408 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.408 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:26:04.409 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.936302+0000", 2024-09-13T11:26:04.409 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000325397 2024-09-13T11:26:04.409 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.409 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.409 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 13", 2024-09-13T11:26:04.409 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.936459+0000", 2024-09-13T11:26:04.409 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00015754300000000001 2024-09-13T11:26:04.409 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.409 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.409 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:26:04.409 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.938813+0000", 2024-09-13T11:26:04.409 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0023530999999999999 2024-09-13T11:26:04.409 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.409 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.410 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:04.410 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.943695+0000", 2024-09-13T11:26:04.410 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.004882506 2024-09-13T11:26:04.410 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.410 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.410 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:26:04.410 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.943767+0000", 2024-09-13T11:26:04.410 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.2454999999999997e-05 2024-09-13T11:26:04.410 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.410 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.410 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:26:04.410 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.943814+0000", 2024-09-13T11:26:04.410 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.6773000000000001e-05 2024-09-13T11:26:04.410 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:04.411 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:26:04.411 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:04.411 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.411 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.411 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPush(3.16 84/70 [PushOp(3:6f7be541:::benchmark_data_smithi177_31116_object43332:head, version: 66'2788, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:6f7be541:::benchmark_data_smithi177_31116_object43332:head@66'2788, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:26:04.411 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:41.207515+0000", 2024-09-13T11:26:04.411 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 143.15883952600001, 2024-09-13T11:26:04.411 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.52643771500000003, 2024-09-13T11:26:04.411 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:26:04.411 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:26:04.411 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:26:04.411 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:26:04.411 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.412 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:26:04.412 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.207515+0000", 2024-09-13T11:26:04.412 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:04.412 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.412 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.412 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:26:04.412 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.207515+0000", 2024-09-13T11:26:04.412 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:04.412 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.412 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.412 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:26:04.412 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.207516+0000", 2024-09-13T11:26:04.412 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.096e-06 2024-09-13T11:26:04.412 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.413 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.413 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:26:04.413 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.207717+0000", 2024-09-13T11:26:04.413 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000200199 2024-09-13T11:26:04.413 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.413 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.413 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:26:04.413 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.732785+0000", 2024-09-13T11:26:04.413 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.52506869599999995 2024-09-13T11:26:04.413 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.413 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.413 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:26:04.413 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.732801+0000", 2024-09-13T11:26:04.414 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5305999999999999e-05 2024-09-13T11:26:04.414 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.414 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.414 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:26:04.414 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.732889+0000", 2024-09-13T11:26:04.414 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.8157000000000003e-05 2024-09-13T11:26:04.414 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.414 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.414 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:26:04.414 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.732901+0000", 2024-09-13T11:26:04.414 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.1866e-05 2024-09-13T11:26:04.414 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.414 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.415 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:26:04.415 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.733953+0000", 2024-09-13T11:26:04.415 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001052395 2024-09-13T11:26:04.415 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:04.415 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:26:04.415 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:04.415 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.415 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.415 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPushReply(3.1f 85/78 [PushReplyOp(3:fd221d21:::benchmark_data_smithi177_31116_object7639:head)])", 2024-09-13T11:26:04.415 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:42.198715+0000", 2024-09-13T11:26:04.415 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 142.16763995700001, 2024-09-13T11:26:04.415 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.48930391099999998, 2024-09-13T11:26:04.415 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:26:04.416 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:26:04.416 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-13T11:26:04.416 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:26:04.416 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.416 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:26:04.416 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.198715+0000", 2024-09-13T11:26:04.416 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:04.416 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.416 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.416 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:26:04.416 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.198715+0000", 2024-09-13T11:26:04.416 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:04.416 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.417 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.417 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:26:04.417 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.198716+0000", 2024-09-13T11:26:04.417 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.3500000000000005e-07 2024-09-13T11:26:04.417 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.417 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.417 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:26:04.417 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.198719+0000", 2024-09-13T11:26:04.417 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.4539999999999999e-06 2024-09-13T11:26:04.417 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.417 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.417 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:26:04.417 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.684732+0000", 2024-09-13T11:26:04.418 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.48601262099999998 2024-09-13T11:26:04.418 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.418 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.418 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:26:04.418 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.684746+0000", 2024-09-13T11:26:04.418 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.4518999999999999e-05 2024-09-13T11:26:04.418 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.418 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.418 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:26:04.418 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.684911+0000", 2024-09-13T11:26:04.418 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000164728 2024-09-13T11:26:04.418 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.418 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.419 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:26:04.419 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.688019+0000", 2024-09-13T11:26:04.419 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0031077539999999999 2024-09-13T11:26:04.419 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:04.419 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:26:04.419 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:04.419 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.419 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.419 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPushReply(3.1c 85/77 [PushReplyOp(3:3a669437:::benchmark_data_smithi177_31116_object25280:head)])", 2024-09-13T11:26:04.419 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:42.352466+0000", 2024-09-13T11:26:04.419 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 142.01388906299999, 2024-09-13T11:26:04.419 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.36670725300000001, 2024-09-13T11:26:04.419 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:26:04.419 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:26:04.420 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-13T11:26:04.420 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:26:04.420 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.420 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:26:04.420 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.352466+0000", 2024-09-13T11:26:04.420 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:04.420 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.420 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.420 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:26:04.420 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.352466+0000", 2024-09-13T11:26:04.420 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:04.420 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.420 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.421 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:26:04.421 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.352467+0000", 2024-09-13T11:26:04.421 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.1429999999999999e-06 2024-09-13T11:26:04.421 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.421 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.421 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:26:04.421 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.352474+0000", 2024-09-13T11:26:04.421 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.6100000000000002e-06 2024-09-13T11:26:04.421 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.421 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.421 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:26:04.421 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.717011+0000", 2024-09-13T11:26:04.421 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.36453790200000002 2024-09-13T11:26:04.421 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.422 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.422 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:26:04.422 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.717032+0000", 2024-09-13T11:26:04.422 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.0999e-05 2024-09-13T11:26:04.422 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.422 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.422 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:26:04.422 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.717162+0000", 2024-09-13T11:26:04.422 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000129159 2024-09-13T11:26:04.422 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.422 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.422 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:26:04.422 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.719173+0000", 2024-09-13T11:26:04.422 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0020114400000000002 2024-09-13T11:26:04.423 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:04.423 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:26:04.423 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:04.423 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.423 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.423 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPushReply(3.1f 85/78 [PushReplyOp(3:fecb0d6a:::benchmark_data_smithi177_31116_object7644:head)])", 2024-09-13T11:26:04.423 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:42.705351+0000", 2024-09-13T11:26:04.423 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 141.66100422100001, 2024-09-13T11:26:04.423 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.290549681, 2024-09-13T11:26:04.423 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:26:04.423 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:26:04.423 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-13T11:26:04.423 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:26:04.424 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.424 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:26:04.424 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.705351+0000", 2024-09-13T11:26:04.424 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:04.424 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.424 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.424 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:26:04.424 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.705351+0000", 2024-09-13T11:26:04.424 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:04.424 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.424 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.424 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:26:04.424 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.705353+0000", 2024-09-13T11:26:04.425 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.4169999999999999e-06 2024-09-13T11:26:04.425 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.425 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.425 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:26:04.425 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.705365+0000", 2024-09-13T11:26:04.425 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.1894e-05 2024-09-13T11:26:04.425 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.425 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.425 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:26:04.425 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.992081+0000", 2024-09-13T11:26:04.425 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.28671641199999998 2024-09-13T11:26:04.425 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.425 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.426 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:26:04.426 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.992111+0000", 2024-09-13T11:26:04.426 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.9427000000000001e-05 2024-09-13T11:26:04.426 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.426 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.426 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:26:04.426 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.993573+0000", 2024-09-13T11:26:04.426 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0014623590000000001 2024-09-13T11:26:04.426 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.426 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.426 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:26:04.426 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.995900+0000", 2024-09-13T11:26:04.426 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0023271720000000002 2024-09-13T11:26:04.427 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:04.427 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:26:04.427 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:04.427 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.427 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.427 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPushReply(3.21 89/78 [PushReplyOp(3:8534d087:::benchmark_data_smithi177_31116_object5474:head)])", 2024-09-13T11:26:04.427 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:46.632834+0000", 2024-09-13T11:26:04.427 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 137.73352078100001, 2024-09-13T11:26:04.427 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.324805556, 2024-09-13T11:26:04.427 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:26:04.427 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:26:04.427 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-13T11:26:04.427 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:26:04.428 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.428 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:26:04.428 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.632834+0000", 2024-09-13T11:26:04.428 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:04.428 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.428 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.428 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:26:04.428 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.632834+0000", 2024-09-13T11:26:04.428 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:04.428 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.428 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.428 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:26:04.428 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.632835+0000", 2024-09-13T11:26:04.428 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.4270000000000001e-06 2024-09-13T11:26:04.429 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.429 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.429 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:26:04.429 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.632840+0000", 2024-09-13T11:26:04.429 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.7809999999999996e-06 2024-09-13T11:26:04.429 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.429 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.429 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:26:04.429 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.956282+0000", 2024-09-13T11:26:04.429 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.323441639 2024-09-13T11:26:04.429 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.429 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.429 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:26:04.430 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.956293+0000", 2024-09-13T11:26:04.430 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.083e-05 2024-09-13T11:26:04.430 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.430 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.430 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:26:04.430 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.956395+0000", 2024-09-13T11:26:04.430 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00010184200000000001 2024-09-13T11:26:04.430 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.430 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.430 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:26:04.430 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.957640+0000", 2024-09-13T11:26:04.430 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0012450370000000001 2024-09-13T11:26:04.430 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:04.430 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:26:04.431 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:04.431 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.431 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.431 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPushReply(3.1f 89/78 [PushReplyOp(3:ff150976:::benchmark_data_smithi177_31116_object12907:head)])", 2024-09-13T11:26:04.431 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:46.697504+0000", 2024-09-13T11:26:04.431 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 137.668851182, 2024-09-13T11:26:04.431 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.26836818099999998, 2024-09-13T11:26:04.431 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:26:04.431 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:26:04.431 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-13T11:26:04.431 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:26:04.431 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.431 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:26:04.432 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.697504+0000", 2024-09-13T11:26:04.432 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:04.432 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.432 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.432 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:26:04.432 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.697504+0000", 2024-09-13T11:26:04.432 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:04.432 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.432 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.432 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:26:04.432 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.697505+0000", 2024-09-13T11:26:04.432 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0529999999999999e-06 2024-09-13T11:26:04.432 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.433 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.433 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:26:04.433 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.697509+0000", 2024-09-13T11:26:04.433 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.9249999999999997e-06 2024-09-13T11:26:04.433 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.433 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.433 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:26:04.433 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.964086+0000", 2024-09-13T11:26:04.433 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.26657776100000002 2024-09-13T11:26:04.433 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.433 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.433 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:26:04.433 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.964110+0000", 2024-09-13T11:26:04.434 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.3189000000000001e-05 2024-09-13T11:26:04.434 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.434 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.434 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:26:04.434 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.964354+0000", 2024-09-13T11:26:04.434 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00024481399999999999 2024-09-13T11:26:04.434 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.434 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.434 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:26:04.434 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.965872+0000", 2024-09-13T11:26:04.434 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001517439 2024-09-13T11:26:04.434 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:04.434 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:26:04.435 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:04.435 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.435 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.435 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:83223 3.1c 3:3bc6018d:::benchmark_data_smithi177_31116_object83222:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106)", 2024-09-13T11:26:04.435 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:04.511163+0000", 2024-09-13T11:26:04.435 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 119.855191404, 2024-09-13T11:26:04.435 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.47558093200000001, 2024-09-13T11:26:04.435 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:26:04.435 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:26:04.435 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:04.435 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:26:04.435 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:26:04.435 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:04.435 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 83223 2024-09-13T11:26:04.436 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.436 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:26:04.436 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.436 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:26:04.436 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.511163+0000", 2024-09-13T11:26:04.436 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:04.436 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.436 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.436 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:26:04.436 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.511163+0000", 2024-09-13T11:26:04.436 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:04.436 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.436 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.437 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:26:04.437 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.511167+0000", 2024-09-13T11:26:04.437 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.8330000000000004e-06 2024-09-13T11:26:04.437 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.437 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.437 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:26:04.437 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.511219+0000", 2024-09-13T11:26:04.437 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.1771999999999998e-05 2024-09-13T11:26:04.437 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.437 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.437 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:26:04.437 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.511252+0000", 2024-09-13T11:26:04.437 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.3315999999999997e-05 2024-09-13T11:26:04.437 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.438 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.438 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:26:04.438 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.511268+0000", 2024-09-13T11:26:04.438 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5812999999999999e-05 2024-09-13T11:26:04.438 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.438 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.438 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:26:04.438 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.511418+0000", 2024-09-13T11:26:04.438 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000150023 2024-09-13T11:26:04.438 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.438 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.438 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:26:04.438 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.511670+0000", 2024-09-13T11:26:04.438 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00025185499999999999 2024-09-13T11:26:04.439 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.439 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.439 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 7", 2024-09-13T11:26:04.439 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.511905+0000", 2024-09-13T11:26:04.439 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00023493799999999999 2024-09-13T11:26:04.439 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.439 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.439 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:26:04.439 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.514960+0000", 2024-09-13T11:26:04.439 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.003054977 2024-09-13T11:26:04.439 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.439 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.439 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:04.439 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.986432+0000", 2024-09-13T11:26:04.440 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.47147209200000001 2024-09-13T11:26:04.440 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.440 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.440 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:26:04.440 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.986635+0000", 2024-09-13T11:26:04.440 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00020309899999999999 2024-09-13T11:26:04.440 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.440 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.440 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:26:04.440 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.986744+0000", 2024-09-13T11:26:04.440 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000109214 2024-09-13T11:26:04.440 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:04.440 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:26:04.441 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:04.441 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.441 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.441 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop_reply(client.4665.0:84803 3.1f e107/78)", 2024-09-13T11:26:04.441 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:05.912627+0000", 2024-09-13T11:26:04.441 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 118.453727497, 2024-09-13T11:26:04.441 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.33726566600000002, 2024-09-13T11:26:04.441 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:26:04.441 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:26:04.441 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-13T11:26:04.441 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:26:04.441 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.441 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:26:04.442 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:05.912627+0000", 2024-09-13T11:26:04.442 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:04.442 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.442 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.442 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:26:04.442 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:05.912627+0000", 2024-09-13T11:26:04.442 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:04.442 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.442 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.442 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:26:04.442 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:05.912629+0000", 2024-09-13T11:26:04.442 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.4440000000000001e-06 2024-09-13T11:26:04.442 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.443 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.443 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:26:04.443 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:05.912634+0000", 2024-09-13T11:26:04.443 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.6869999999999999e-06 2024-09-13T11:26:04.443 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.443 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.443 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:26:04.443 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:06.249722+0000", 2024-09-13T11:26:04.443 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.33708766099999998 2024-09-13T11:26:04.443 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.443 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.443 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:26:04.443 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:06.249736+0000", 2024-09-13T11:26:04.443 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3623e-05 2024-09-13T11:26:04.444 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.444 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.444 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:26:04.444 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:06.249878+0000", 2024-09-13T11:26:04.444 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00014264599999999999 2024-09-13T11:26:04.444 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.444 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.444 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:26:04.444 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:06.249893+0000", 2024-09-13T11:26:04.444 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.4605e-05 2024-09-13T11:26:04.444 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:04.444 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:26:04.444 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:04.444 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.445 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.445 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop_reply(client.4665.0:110260 3.f e119/75 ondisk, result = 0)", 2024-09-13T11:26:04.445 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:25.333188+0000", 2024-09-13T11:26:04.445 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 99.033166870000002, 2024-09-13T11:26:04.445 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.29195486199999998, 2024-09-13T11:26:04.445 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:26:04.445 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:26:04.445 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:26:04.445 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:26:04.445 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.445 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:26:04.445 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.333188+0000", 2024-09-13T11:26:04.445 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:04.446 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.446 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.446 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:26:04.446 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.333188+0000", 2024-09-13T11:26:04.446 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:04.446 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.446 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.446 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:26:04.446 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.333190+0000", 2024-09-13T11:26:04.446 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.2120000000000002e-06 2024-09-13T11:26:04.447 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.447 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.447 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:26:04.447 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.333197+0000", 2024-09-13T11:26:04.447 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.7469999999999999e-06 2024-09-13T11:26:04.447 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.447 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.447 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:26:04.447 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.624892+0000", 2024-09-13T11:26:04.447 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.291694913 2024-09-13T11:26:04.447 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.447 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.447 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:26:04.448 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.624907+0000", 2024-09-13T11:26:04.448 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5053e-05 2024-09-13T11:26:04.448 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.448 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.448 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:26:04.448 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.624999+0000", 2024-09-13T11:26:04.448 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.2603999999999996e-05 2024-09-13T11:26:04.448 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.448 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.448 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:26:04.448 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.625011+0000", 2024-09-13T11:26:04.448 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.1253999999999999e-05 2024-09-13T11:26:04.448 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.449 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.449 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:26:04.449 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.625143+0000", 2024-09-13T11:26:04.449 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00013207900000000001 2024-09-13T11:26:04.449 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:04.449 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:26:04.449 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:04.449 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.449 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.449 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:110265 3.1 e119/75 3:8236484d:::benchmark_data_smithi177_31116_object110264:head v 119'3935, mlcod=118'3933)", 2024-09-13T11:26:04.449 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:25.343348+0000", 2024-09-13T11:26:04.449 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 99.023007254999996, 2024-09-13T11:26:04.449 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.60013013500000001, 2024-09-13T11:26:04.449 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:26:04.450 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:26:04.450 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:04.450 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:26:04.450 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.450 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:26:04.450 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.343348+0000", 2024-09-13T11:26:04.450 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:04.450 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.450 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.450 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:26:04.450 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.343348+0000", 2024-09-13T11:26:04.450 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:04.450 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.451 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.451 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:26:04.451 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.343350+0000", 2024-09-13T11:26:04.451 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.576e-06 2024-09-13T11:26:04.451 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.451 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.451 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:26:04.451 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.343443+0000", 2024-09-13T11:26:04.451 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.2989e-05 2024-09-13T11:26:04.452 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.452 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.452 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:26:04.452 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.940863+0000", 2024-09-13T11:26:04.452 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.59742003300000002 2024-09-13T11:26:04.452 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.452 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.452 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:26:04.452 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.940875+0000", 2024-09-13T11:26:04.453 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.2147000000000001e-05 2024-09-13T11:26:04.453 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.453 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.453 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:26:04.453 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.940993+0000", 2024-09-13T11:26:04.453 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000117903 2024-09-13T11:26:04.453 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.453 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.453 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:26:04.453 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.941017+0000", 2024-09-13T11:26:04.453 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.3442999999999999e-05 2024-09-13T11:26:04.454 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.454 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.454 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:26:04.454 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.943347+0000", 2024-09-13T11:26:04.454 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0023299869999999999 2024-09-13T11:26:04.454 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.454 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.454 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:26:04.454 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.943478+0000", 2024-09-13T11:26:04.454 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00013105699999999999 2024-09-13T11:26:04.454 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:04.454 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:26:04.454 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:04.455 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.455 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.455 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop_reply(client.4665.0:129919 3.16 e134/88 ondisk, result = 0)", 2024-09-13T11:26:04.455 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:47.175742+0000", 2024-09-13T11:26:04.455 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 77.190613185000004, 2024-09-13T11:26:04.455 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.27593726299999999, 2024-09-13T11:26:04.455 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:26:04.455 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:26:04.455 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:26:04.455 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:26:04.455 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.455 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:26:04.455 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:47.175742+0000", 2024-09-13T11:26:04.456 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:04.456 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.456 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.456 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:26:04.456 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:47.175742+0000", 2024-09-13T11:26:04.456 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:04.456 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.456 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.456 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:26:04.456 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:47.175744+0000", 2024-09-13T11:26:04.456 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.0030000000000001e-06 2024-09-13T11:26:04.456 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.456 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.456 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:26:04.457 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:47.175751+0000", 2024-09-13T11:26:04.457 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.9779999999999999e-06 2024-09-13T11:26:04.457 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.457 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.457 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:26:04.457 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:47.451490+0000", 2024-09-13T11:26:04.457 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.27573892500000002 2024-09-13T11:26:04.457 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.458 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.458 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:26:04.458 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:47.451496+0000", 2024-09-13T11:26:04.458 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.0079999999999999e-06 2024-09-13T11:26:04.458 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.458 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.458 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:26:04.458 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:47.451560+0000", 2024-09-13T11:26:04.458 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.3955999999999995e-05 2024-09-13T11:26:04.458 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.458 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.458 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:26:04.458 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:47.451567+0000", 2024-09-13T11:26:04.459 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.1980000000000004e-06 2024-09-13T11:26:04.459 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.459 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.459 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:26:04.459 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:47.451679+0000", 2024-09-13T11:26:04.459 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00011219500000000001 2024-09-13T11:26:04.459 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:04.459 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:26:04.459 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:04.459 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.459 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.459 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:209869 3.27 e185/169 3:e6b1fcc5:::benchmark_data_smithi177_31116_object69566:head v 185'6350, mlcod=185'6347)", 2024-09-13T11:26:04.459 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:25:54.301402+0000", 2024-09-13T11:26:04.459 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 10.064952514, 2024-09-13T11:26:04.460 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.30148222499999999, 2024-09-13T11:26:04.460 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:26:04.460 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:26:04.460 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:04.460 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:26:04.460 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.460 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:26:04.460 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:54.301402+0000", 2024-09-13T11:26:04.460 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:04.460 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.460 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.460 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:26:04.460 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:54.301402+0000", 2024-09-13T11:26:04.460 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:04.461 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.461 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.461 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:26:04.461 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:54.301404+0000", 2024-09-13T11:26:04.461 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.1939999999999999e-06 2024-09-13T11:26:04.461 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.461 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.461 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:26:04.461 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:54.301409+0000", 2024-09-13T11:26:04.461 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.0259999999999998e-06 2024-09-13T11:26:04.461 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.461 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.461 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:26:04.461 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:54.601076+0000", 2024-09-13T11:26:04.462 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.29966704799999999 2024-09-13T11:26:04.462 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.462 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.462 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:26:04.462 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:54.601081+0000", 2024-09-13T11:26:04.462 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.7069999999999997e-06 2024-09-13T11:26:04.462 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.462 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.462 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:26:04.463 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:54.601135+0000", 2024-09-13T11:26:04.463 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.3439000000000001e-05 2024-09-13T11:26:04.463 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.463 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.463 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:26:04.463 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:54.601150+0000", 2024-09-13T11:26:04.463 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5443000000000001e-05 2024-09-13T11:26:04.463 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.463 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.463 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:26:04.463 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:54.602854+0000", 2024-09-13T11:26:04.463 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0017035220000000001 2024-09-13T11:26:04.463 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.463 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.463 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:26:04.464 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:54.602885+0000", 2024-09-13T11:26:04.464 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.0846000000000002e-05 2024-09-13T11:26:04.464 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:04.464 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:26:04.464 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:04.464 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:04.464 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:26:04.464 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-09-13T11:26:04.466 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_ops_in_flight 2024-09-13T11:26:04.466 INFO:tasks.ceph.osd.5.smithi092.stderr:2024-09-13T11:26:04.464+0000 7f6f0b705640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26346) UID: 0 2024-09-13T11:26:04.567 INFO:tasks.ceph.osd.7.smithi177.stderr:2024-09-13T11:26:04.565+0000 7f897b51d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26575) UID: 0 2024-09-13T11:26:04.593 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-09-13T11:26:04.593 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [], 2024-09-13T11:26:04.593 INFO:teuthology.orchestra.run.smithi177.stdout: "num_ops": 0 2024-09-13T11:26:04.593 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-09-13T11:26:04.607 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_blocked_ops 2024-09-13T11:26:04.668 INFO:tasks.ceph.osd.14.smithi162.stderr:2024-09-13T11:26:04.663+0000 7f3df5903640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33154) UID: 0 2024-09-13T11:26:04.769 INFO:tasks.ceph.osd.10.smithi162.stderr:2024-09-13T11:26:04.767+0000 7fb59c9f0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26317) UID: 0 2024-09-13T11:26:04.776 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-09-13T11:26:04.776 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [], 2024-09-13T11:26:04.776 INFO:teuthology.orchestra.run.smithi177.stdout: "complaint_time": 30, 2024-09-13T11:26:04.776 INFO:teuthology.orchestra.run.smithi177.stdout: "num_blocked_ops": 0 2024-09-13T11:26:04.776 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-09-13T11:26:04.790 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_historic_ops 2024-09-13T11:26:04.870 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-09-13T11:26:04.867+0000 7f97befc9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26315) UID: 0 2024-09-13T11:26:04.911 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-09-13T11:26:04.911 INFO:teuthology.orchestra.run.smithi177.stdout: "size": 20, 2024-09-13T11:26:04.911 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 600, 2024-09-13T11:26:04.911 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [ 2024-09-13T11:26:04.911 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.911 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:25383 3.0 e55/30 3:0a6bb869:::benchmark_data_smithi177_31116_object25382:head v 55'1620, mlcod=55'1617)", 2024-09-13T11:26:04.912 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:04.449956+0000", 2024-09-13T11:26:04.912 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 180.45892215399999, 2024-09-13T11:26:04.912 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.21143479600000001, 2024-09-13T11:26:04.912 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:26:04.912 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:26:04.912 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:04.912 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:26:04.912 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.912 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:26:04.912 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.449956+0000", 2024-09-13T11:26:04.913 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:04.913 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.913 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.913 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:26:04.913 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.449956+0000", 2024-09-13T11:26:04.913 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:04.913 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.913 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.913 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:26:04.913 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.449959+0000", 2024-09-13T11:26:04.914 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.393e-06 2024-09-13T11:26:04.914 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.914 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.914 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:26:04.914 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.450014+0000", 2024-09-13T11:26:04.914 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.4200000000000003e-05 2024-09-13T11:26:04.914 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.914 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.914 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:26:04.914 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.658637+0000", 2024-09-13T11:26:04.914 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.20862334799999999 2024-09-13T11:26:04.915 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.915 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.915 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:26:04.915 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.658650+0000", 2024-09-13T11:26:04.915 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3148e-05 2024-09-13T11:26:04.915 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.915 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.915 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:26:04.915 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.658867+0000", 2024-09-13T11:26:04.915 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00021673499999999999 2024-09-13T11:26:04.915 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.916 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.916 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:26:04.916 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.658931+0000", 2024-09-13T11:26:04.916 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.4045000000000001e-05 2024-09-13T11:26:04.916 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.916 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.916 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:26:04.916 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.661317+0000", 2024-09-13T11:26:04.916 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.002385823 2024-09-13T11:26:04.916 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.916 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.917 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:26:04.917 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.661391+0000", 2024-09-13T11:26:04.917 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.4103999999999994e-05 2024-09-13T11:26:04.917 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:04.917 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:26:04.917 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:04.917 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.917 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.917 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPush(3.16 85/70 [PushOp(3:69aac9fd:::benchmark_data_smithi177_31116_object43914:head, version: 66'2837, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:69aac9fd:::benchmark_data_smithi177_31116_object43914:head@66'2837, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:26:04.918 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:42.295995+0000", 2024-09-13T11:26:04.918 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 142.612882834, 2024-09-13T11:26:04.918 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.215089431, 2024-09-13T11:26:04.918 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:26:04.918 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:26:04.918 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:26:04.918 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:26:04.918 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.918 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:26:04.918 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.295995+0000", 2024-09-13T11:26:04.918 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:04.919 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.919 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.919 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:26:04.919 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.295995+0000", 2024-09-13T11:26:04.919 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:04.919 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.919 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.919 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:26:04.919 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.295997+0000", 2024-09-13T11:26:04.919 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3170000000000001e-06 2024-09-13T11:26:04.919 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.920 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.920 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:26:04.920 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.296058+0000", 2024-09-13T11:26:04.920 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.1690000000000001e-05 2024-09-13T11:26:04.920 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.920 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.920 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:26:04.920 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.509606+0000", 2024-09-13T11:26:04.920 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.21354748300000001 2024-09-13T11:26:04.920 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.920 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.921 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:26:04.921 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.509638+0000", 2024-09-13T11:26:04.921 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.1644999999999999e-05 2024-09-13T11:26:04.921 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.921 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.921 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:26:04.921 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.509814+0000", 2024-09-13T11:26:04.921 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00017679499999999999 2024-09-13T11:26:04.921 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.921 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.921 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:26:04.921 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.509840+0000", 2024-09-13T11:26:04.922 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.5871e-05 2024-09-13T11:26:04.922 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.922 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.922 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:26:04.922 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.511085+0000", 2024-09-13T11:26:04.922 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00124463 2024-09-13T11:26:04.922 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:04.922 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:26:04.922 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:04.922 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.923 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.923 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPush(3.1d 91/78 [PushOp(3:be6a0ae4:::benchmark_data_smithi177_31116_object25671:head, version: 58'1569, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:be6a0ae4:::benchmark_data_smithi177_31116_object25671:head@58'1569, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:26:04.923 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:48.481971+0000", 2024-09-13T11:26:04.923 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 136.426907589, 2024-09-13T11:26:04.923 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.47945454199999998, 2024-09-13T11:26:04.923 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:26:04.923 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:26:04.923 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:26:04.923 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:26:04.923 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.923 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:26:04.924 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.481971+0000", 2024-09-13T11:26:04.924 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:04.924 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.924 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.924 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:26:04.924 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.481971+0000", 2024-09-13T11:26:04.924 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:04.924 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.924 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.924 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:26:04.924 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.481972+0000", 2024-09-13T11:26:04.925 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5179999999999999e-06 2024-09-13T11:26:04.925 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.925 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.925 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:26:04.925 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.482000+0000", 2024-09-13T11:26:04.925 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.8249e-05 2024-09-13T11:26:04.925 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.925 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.925 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:26:04.925 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.959926+0000", 2024-09-13T11:26:04.925 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.47792585100000001 2024-09-13T11:26:04.926 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.926 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.926 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:26:04.926 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.959944+0000", 2024-09-13T11:26:04.926 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.7620000000000001e-05 2024-09-13T11:26:04.926 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.926 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.926 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:26:04.926 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.960059+0000", 2024-09-13T11:26:04.926 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000115188 2024-09-13T11:26:04.926 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.927 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.927 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:26:04.927 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.960074+0000", 2024-09-13T11:26:04.927 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.4738e-05 2024-09-13T11:26:04.927 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.927 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.927 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:26:04.927 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.961425+0000", 2024-09-13T11:26:04.927 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0013513780000000001 2024-09-13T11:26:04.927 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:04.928 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:26:04.928 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:04.928 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.928 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.928 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:71884 3.16 e96/88 3:6ca846b2:::benchmark_data_smithi177_31116_object71883:head v 96'3755, mlcod=96'3753)", 2024-09-13T11:26:04.928 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:54.108299+0000", 2024-09-13T11:26:04.928 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 130.80057941600001, 2024-09-13T11:26:04.928 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.20685140099999999, 2024-09-13T11:26:04.928 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:26:04.928 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:26:04.929 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:04.929 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:26:04.929 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.929 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:26:04.929 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.108299+0000", 2024-09-13T11:26:04.929 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:04.929 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.929 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.929 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:26:04.929 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.108299+0000", 2024-09-13T11:26:04.930 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:04.930 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.930 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.930 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:26:04.930 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.108300+0000", 2024-09-13T11:26:04.930 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.173e-06 2024-09-13T11:26:04.930 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.930 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.930 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:26:04.931 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.108323+0000", 2024-09-13T11:26:04.931 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.2591000000000001e-05 2024-09-13T11:26:04.931 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.931 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.931 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:26:04.931 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.312377+0000", 2024-09-13T11:26:04.931 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.20405456599999999 2024-09-13T11:26:04.931 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.931 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.931 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:26:04.931 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.312382+0000", 2024-09-13T11:26:04.931 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.1429999999999997e-06 2024-09-13T11:26:04.932 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.932 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.932 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:26:04.932 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.312480+0000", 2024-09-13T11:26:04.932 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.7742e-05 2024-09-13T11:26:04.932 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.932 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.932 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:26:04.932 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.312521+0000", 2024-09-13T11:26:04.932 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.1369000000000003e-05 2024-09-13T11:26:04.932 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.933 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.933 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:26:04.933 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.315105+0000", 2024-09-13T11:26:04.933 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0025835060000000002 2024-09-13T11:26:04.933 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.933 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.933 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:26:04.933 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.315150+0000", 2024-09-13T11:26:04.933 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.5311e-05 2024-09-13T11:26:04.933 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:04.933 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:26:04.933 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:04.934 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.934 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.934 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:71927 3.0 e97/75 3:0197244a:::benchmark_data_smithi177_31116_object71926:head v 97'3351, mlcod=96'3348)", 2024-09-13T11:26:04.934 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:54.200161+0000", 2024-09-13T11:26:04.934 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 130.70871718800001, 2024-09-13T11:26:04.934 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.26691738999999998, 2024-09-13T11:26:04.934 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:26:04.934 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:26:04.934 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:04.934 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:26:04.934 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.934 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:26:04.935 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.200161+0000", 2024-09-13T11:26:04.935 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:04.935 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.935 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.935 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:26:04.935 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.200161+0000", 2024-09-13T11:26:04.935 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:04.935 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.935 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.935 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:26:04.935 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.200162+0000", 2024-09-13T11:26:04.936 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.78e-07 2024-09-13T11:26:04.936 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.936 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.936 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:26:04.936 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.200200+0000", 2024-09-13T11:26:04.936 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.8121000000000003e-05 2024-09-13T11:26:04.936 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.936 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.936 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:26:04.936 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.465397+0000", 2024-09-13T11:26:04.936 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.26519681099999998 2024-09-13T11:26:04.937 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.937 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.937 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:26:04.937 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.465404+0000", 2024-09-13T11:26:04.937 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.7399999999999998e-06 2024-09-13T11:26:04.937 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.937 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.937 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:26:04.937 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.465456+0000", 2024-09-13T11:26:04.937 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.2626000000000001e-05 2024-09-13T11:26:04.937 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.938 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.938 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:26:04.938 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.465482+0000", 2024-09-13T11:26:04.938 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.6086000000000001e-05 2024-09-13T11:26:04.938 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.938 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.938 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:26:04.938 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.467035+0000", 2024-09-13T11:26:04.938 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0015523729999999999 2024-09-13T11:26:04.938 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.938 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.938 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:26:04.939 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.467078+0000", 2024-09-13T11:26:04.939 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.3754999999999997e-05 2024-09-13T11:26:04.939 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:04.939 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:26:04.939 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:04.939 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.939 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.939 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:79634 3.5 3:a6c32e6f:::benchmark_data_smithi177_31116_object79633:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101)", 2024-09-13T11:26:04.939 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:00.168616+0000", 2024-09-13T11:26:04.939 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 124.7402618, 2024-09-13T11:26:04.939 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.169445552, 2024-09-13T11:26:04.939 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:26:04.940 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:26:04.940 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:04.940 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:26:04.940 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:26:04.940 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:04.940 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 79634 2024-09-13T11:26:04.940 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.940 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:26:04.940 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.940 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:26:04.940 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.168616+0000", 2024-09-13T11:26:04.941 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:04.941 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.941 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.941 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:26:04.941 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.168616+0000", 2024-09-13T11:26:04.941 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:04.941 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.941 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.941 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:26:04.941 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.168619+0000", 2024-09-13T11:26:04.941 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.565e-06 2024-09-13T11:26:04.941 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.941 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.942 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:26:04.942 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.168639+0000", 2024-09-13T11:26:04.942 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.0384999999999999e-05 2024-09-13T11:26:04.942 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.942 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.942 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:26:04.942 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.168652+0000", 2024-09-13T11:26:04.942 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.2779000000000001e-05 2024-09-13T11:26:04.942 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.942 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.942 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:26:04.943 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.168658+0000", 2024-09-13T11:26:04.943 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.2269999999999998e-06 2024-09-13T11:26:04.943 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.943 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.943 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:26:04.943 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.168722+0000", 2024-09-13T11:26:04.943 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.3888999999999998e-05 2024-09-13T11:26:04.943 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.943 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.943 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:26:04.943 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.168863+0000", 2024-09-13T11:26:04.944 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00014050799999999999 2024-09-13T11:26:04.944 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.944 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.944 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 13", 2024-09-13T11:26:04.944 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.169010+0000", 2024-09-13T11:26:04.944 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00014746200000000001 2024-09-13T11:26:04.944 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.944 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.944 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:26:04.944 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.170891+0000", 2024-09-13T11:26:04.945 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001880945 2024-09-13T11:26:04.945 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.945 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.945 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:04.945 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.337961+0000", 2024-09-13T11:26:04.945 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.16706952 2024-09-13T11:26:04.945 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.945 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.945 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:26:04.945 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.338019+0000", 2024-09-13T11:26:04.945 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.7958000000000003e-05 2024-09-13T11:26:04.946 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.946 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.946 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:26:04.946 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.338062+0000", 2024-09-13T11:26:04.946 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.3313999999999999e-05 2024-09-13T11:26:04.946 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:04.946 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:26:04.946 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:04.946 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.946 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.946 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPush(3.1b 102/78 [PushOp(3:df786fc7:::benchmark_data_smithi177_31116_object48840:head, version: 71'3079, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:df786fc7:::benchmark_data_smithi177_31116_object48840:head@71'3079, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:26:04.947 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:00.290536+0000", 2024-09-13T11:26:04.947 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 124.618342426, 2024-09-13T11:26:04.947 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.20017585399999999, 2024-09-13T11:26:04.947 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:26:04.947 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:26:04.947 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:26:04.947 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:26:04.947 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.947 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:26:04.947 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.290536+0000", 2024-09-13T11:26:04.947 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:04.948 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.948 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.948 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:26:04.948 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.290536+0000", 2024-09-13T11:26:04.948 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:04.948 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.948 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.948 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:26:04.948 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.290537+0000", 2024-09-13T11:26:04.948 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.327e-06 2024-09-13T11:26:04.948 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.949 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.949 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:26:04.949 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.290581+0000", 2024-09-13T11:26:04.949 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.3893999999999997e-05 2024-09-13T11:26:04.949 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.949 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.949 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:26:04.949 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.488588+0000", 2024-09-13T11:26:04.949 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.198006708 2024-09-13T11:26:04.949 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.949 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.950 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:26:04.950 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.488607+0000", 2024-09-13T11:26:04.950 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.9162999999999999e-05 2024-09-13T11:26:04.950 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.950 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.950 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:26:04.950 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.488747+0000", 2024-09-13T11:26:04.950 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00014040799999999999 2024-09-13T11:26:04.950 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.950 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.950 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:26:04.951 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.488770+0000", 2024-09-13T11:26:04.951 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.279e-05 2024-09-13T11:26:04.951 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.951 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.951 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:26:04.951 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.490712+0000", 2024-09-13T11:26:04.951 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001941564 2024-09-13T11:26:04.951 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:04.951 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:26:04.951 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:04.951 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.952 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.952 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPush(3.1b 102/78 [PushOp(3:d82bf6ec:::benchmark_data_smithi177_31116_object48847:head, version: 71'3080, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:d82bf6ec:::benchmark_data_smithi177_31116_object48847:head@71'3080, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:26:04.952 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:00.293979+0000", 2024-09-13T11:26:04.952 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 124.614898899, 2024-09-13T11:26:04.952 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.20314832099999999, 2024-09-13T11:26:04.952 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:26:04.952 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:26:04.952 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:26:04.952 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:26:04.952 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.952 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:26:04.952 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.293979+0000", 2024-09-13T11:26:04.953 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:04.953 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.953 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.953 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:26:04.953 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.293979+0000", 2024-09-13T11:26:04.953 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:04.953 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.953 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.953 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:26:04.953 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.293981+0000", 2024-09-13T11:26:04.953 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.4449999999999999e-06 2024-09-13T11:26:04.954 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.954 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.954 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:26:04.954 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.294085+0000", 2024-09-13T11:26:04.954 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000103848 2024-09-13T11:26:04.954 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.954 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.954 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:26:04.954 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.495571+0000", 2024-09-13T11:26:04.954 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.20148640800000001 2024-09-13T11:26:04.954 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.954 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.955 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:26:04.955 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.495609+0000", 2024-09-13T11:26:04.955 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.7904000000000002e-05 2024-09-13T11:26:04.955 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.955 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.955 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:26:04.955 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.495729+0000", 2024-09-13T11:26:04.955 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000120556 2024-09-13T11:26:04.955 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.955 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.955 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:26:04.955 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.495751+0000", 2024-09-13T11:26:04.956 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.1511000000000001e-05 2024-09-13T11:26:04.956 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.956 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.956 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:26:04.956 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.497128+0000", 2024-09-13T11:26:04.956 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001376649 2024-09-13T11:26:04.956 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:04.956 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:26:04.956 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:04.957 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.957 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.957 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:79801 3.5 3:a6efdf84:::benchmark_data_smithi177_31116_object79800:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102)", 2024-09-13T11:26:04.957 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:00.403201+0000", 2024-09-13T11:26:04.957 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 124.505677312, 2024-09-13T11:26:04.957 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.31868626300000003, 2024-09-13T11:26:04.957 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:26:04.957 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:26:04.957 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:04.958 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:26:04.958 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:26:04.958 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:04.958 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 79801 2024-09-13T11:26:04.958 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.958 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:26:04.958 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.958 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:26:04.958 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.403201+0000", 2024-09-13T11:26:04.958 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:04.959 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.959 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.959 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:26:04.959 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.403201+0000", 2024-09-13T11:26:04.959 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:04.959 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.959 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.959 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:26:04.959 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.403204+0000", 2024-09-13T11:26:04.959 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.0819999999999999e-06 2024-09-13T11:26:04.959 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.960 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.960 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:26:04.960 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.403260+0000", 2024-09-13T11:26:04.960 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.5624999999999997e-05 2024-09-13T11:26:04.960 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.960 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.960 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:26:04.960 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.403280+0000", 2024-09-13T11:26:04.960 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.0060000000000001e-05 2024-09-13T11:26:04.961 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.961 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.961 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:26:04.961 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.403291+0000", 2024-09-13T11:26:04.961 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.1335000000000001e-05 2024-09-13T11:26:04.961 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.961 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.961 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:26:04.961 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.403344+0000", 2024-09-13T11:26:04.961 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.3217000000000002e-05 2024-09-13T11:26:04.961 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.962 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.962 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:26:04.962 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.403456+0000", 2024-09-13T11:26:04.962 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000111706 2024-09-13T11:26:04.962 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.962 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.962 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 13", 2024-09-13T11:26:04.962 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.403547+0000", 2024-09-13T11:26:04.962 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.1556000000000002e-05 2024-09-13T11:26:04.962 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.963 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.963 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:26:04.963 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.407436+0000", 2024-09-13T11:26:04.963 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.003888856 2024-09-13T11:26:04.963 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.963 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.963 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:04.963 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.721722+0000", 2024-09-13T11:26:04.963 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.31428562700000001 2024-09-13T11:26:04.963 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.963 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.963 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:26:04.964 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.721814+0000", 2024-09-13T11:26:04.964 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.1564000000000006e-05 2024-09-13T11:26:04.964 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.964 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.964 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:26:04.964 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.721887+0000", 2024-09-13T11:26:04.964 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.3634999999999995e-05 2024-09-13T11:26:04.964 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:04.964 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:26:04.964 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:04.965 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.965 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.965 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:83199 3.1d e106/102 3:be2c6fae:::benchmark_data_smithi177_31116_object83198:head v 106'4244, mlcod=106'4242)", 2024-09-13T11:26:04.965 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:04.298299+0000", 2024-09-13T11:26:04.965 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 120.61057918500001, 2024-09-13T11:26:04.965 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.25001107099999997, 2024-09-13T11:26:04.965 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:26:04.965 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:26:04.966 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:04.966 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:26:04.966 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.966 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:26:04.966 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.298299+0000", 2024-09-13T11:26:04.966 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:04.966 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.966 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.966 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:26:04.967 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.298299+0000", 2024-09-13T11:26:04.967 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:04.967 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.967 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.967 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:26:04.967 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.298301+0000", 2024-09-13T11:26:04.967 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5859999999999999e-06 2024-09-13T11:26:04.967 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.967 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.967 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:26:04.967 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.298368+0000", 2024-09-13T11:26:04.968 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.6928999999999996e-05 2024-09-13T11:26:04.968 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.968 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.968 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:26:04.968 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.545180+0000", 2024-09-13T11:26:04.968 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.24681209700000001 2024-09-13T11:26:04.968 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.968 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.968 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:26:04.968 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.545193+0000", 2024-09-13T11:26:04.968 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3427e-05 2024-09-13T11:26:04.969 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.969 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.969 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:26:04.969 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.545327+0000", 2024-09-13T11:26:04.969 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00013359400000000001 2024-09-13T11:26:04.969 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.969 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.969 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:26:04.969 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.545385+0000", 2024-09-13T11:26:04.969 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.8492000000000002e-05 2024-09-13T11:26:04.969 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.969 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.970 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:26:04.970 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.548225+0000", 2024-09-13T11:26:04.970 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.002839472 2024-09-13T11:26:04.970 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.970 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.970 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:26:04.970 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.548310+0000", 2024-09-13T11:26:04.970 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.5474000000000002e-05 2024-09-13T11:26:04.970 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:04.970 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:26:04.970 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:04.971 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.971 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.971 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:84621 3.1d e107/102 3:becd72aa:::benchmark_data_smithi177_31116_object84620:head v 107'4296, mlcod=107'4293)", 2024-09-13T11:26:04.971 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:05.742238+0000", 2024-09-13T11:26:04.971 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 119.166640085, 2024-09-13T11:26:04.971 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.17888174300000001, 2024-09-13T11:26:04.971 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:26:04.971 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:26:04.971 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:04.971 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:26:04.971 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.972 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:26:04.972 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:05.742238+0000", 2024-09-13T11:26:04.972 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:04.972 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.972 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.972 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:26:04.972 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:05.742238+0000", 2024-09-13T11:26:04.972 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:04.972 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.972 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.973 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:26:04.973 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:05.742240+0000", 2024-09-13T11:26:04.973 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.3159999999999999e-06 2024-09-13T11:26:04.973 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.973 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.973 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:26:04.973 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:05.742315+0000", 2024-09-13T11:26:04.973 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.4122000000000006e-05 2024-09-13T11:26:04.973 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.973 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.974 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:26:04.974 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:05.917757+0000", 2024-09-13T11:26:04.974 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.175442031 2024-09-13T11:26:04.974 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.974 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.974 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:26:04.974 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:05.917764+0000", 2024-09-13T11:26:04.974 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.661e-06 2024-09-13T11:26:04.974 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.974 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.974 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:26:04.975 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:05.917846+0000", 2024-09-13T11:26:04.975 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.2006000000000001e-05 2024-09-13T11:26:04.975 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.975 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.975 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:26:04.975 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:05.917879+0000", 2024-09-13T11:26:04.975 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.2796000000000002e-05 2024-09-13T11:26:04.975 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.975 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.975 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:26:04.975 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:05.921070+0000", 2024-09-13T11:26:04.975 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0031908489999999999 2024-09-13T11:26:04.976 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.976 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.976 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:26:04.976 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:05.921120+0000", 2024-09-13T11:26:04.976 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.9962e-05 2024-09-13T11:26:04.976 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:04.976 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:26:04.976 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:04.976 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.976 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.976 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:107910 3.16 e116/88 3:6cf57aa3:::benchmark_data_smithi177_31116_object107909:head v 116'4911, mlcod=116'4908)", 2024-09-13T11:26:04.977 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:22.516225+0000", 2024-09-13T11:26:04.977 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 102.39265354200001, 2024-09-13T11:26:04.977 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.18319666600000001, 2024-09-13T11:26:04.977 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:26:04.977 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:26:04.977 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:04.977 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:26:04.977 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.977 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:26:04.978 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:22.516225+0000", 2024-09-13T11:26:04.978 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:04.978 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.978 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.978 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:26:04.978 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:22.516225+0000", 2024-09-13T11:26:04.978 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:04.978 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.978 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.979 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:26:04.979 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:22.516226+0000", 2024-09-13T11:26:04.979 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.716e-06 2024-09-13T11:26:04.979 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.979 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.979 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:26:04.979 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:22.516250+0000", 2024-09-13T11:26:04.979 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.3975e-05 2024-09-13T11:26:04.979 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.979 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.980 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:26:04.980 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:22.697686+0000", 2024-09-13T11:26:04.980 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.18143539 2024-09-13T11:26:04.980 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.980 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.980 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:26:04.980 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:22.697692+0000", 2024-09-13T11:26:04.980 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.0769999999999996e-06 2024-09-13T11:26:04.980 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.980 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.980 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:26:04.980 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:22.697744+0000", 2024-09-13T11:26:04.981 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.2525000000000003e-05 2024-09-13T11:26:04.981 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.981 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.981 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:26:04.981 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:22.697757+0000", 2024-09-13T11:26:04.981 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.2819e-05 2024-09-13T11:26:04.981 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.981 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.981 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:26:04.981 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:22.699388+0000", 2024-09-13T11:26:04.981 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0016303999999999999 2024-09-13T11:26:04.981 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.982 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.982 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:26:04.982 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:22.699421+0000", 2024-09-13T11:26:04.982 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.3763999999999997e-05 2024-09-13T11:26:04.982 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:04.982 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:26:04.982 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:04.982 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.982 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.982 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:127682 3.16 e131/88 3:6e6da02e:::benchmark_data_smithi177_31116_object127681:head v 131'5491, mlcod=131'5488)", 2024-09-13T11:26:04.982 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:44.556692+0000", 2024-09-13T11:26:04.982 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 80.352186613000001, 2024-09-13T11:26:04.983 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.18048624299999999, 2024-09-13T11:26:04.983 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:26:04.983 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:26:04.983 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:04.983 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:26:04.983 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.983 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:26:04.983 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:44.556692+0000", 2024-09-13T11:26:04.983 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:04.983 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.983 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.984 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:26:04.984 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:44.556692+0000", 2024-09-13T11:26:04.984 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:04.984 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.984 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.984 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:26:04.984 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:44.556693+0000", 2024-09-13T11:26:04.984 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.8109999999999999e-06 2024-09-13T11:26:04.984 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.984 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.984 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:26:04.985 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:44.556715+0000", 2024-09-13T11:26:04.985 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.1328999999999999e-05 2024-09-13T11:26:04.985 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.985 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.985 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:26:04.985 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:44.734908+0000", 2024-09-13T11:26:04.985 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.178192884 2024-09-13T11:26:04.985 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.985 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.985 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:26:04.985 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:44.734915+0000", 2024-09-13T11:26:04.985 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.9970000000000001e-06 2024-09-13T11:26:04.986 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.986 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.986 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:26:04.986 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:44.734971+0000", 2024-09-13T11:26:04.986 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.6368e-05 2024-09-13T11:26:04.986 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.986 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.986 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:26:04.986 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:44.734990+0000", 2024-09-13T11:26:04.986 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.929e-05 2024-09-13T11:26:04.986 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.986 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.987 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:26:04.987 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:44.737096+0000", 2024-09-13T11:26:04.987 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0021053090000000001 2024-09-13T11:26:04.987 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.987 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.987 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:26:04.987 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:44.737178+0000", 2024-09-13T11:26:04.987 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.2255000000000004e-05 2024-09-13T11:26:04.987 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:04.987 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:26:04.987 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:04.987 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.988 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.988 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:137062 3.5 3:a16e178f:::benchmark_data_smithi177_31116_object137061:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e139)", 2024-09-13T11:26:04.988 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:53.140525+0000", 2024-09-13T11:26:04.988 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 71.768353591999997, 2024-09-13T11:26:04.988 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.20697954599999999, 2024-09-13T11:26:04.988 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:26:04.988 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:26:04.988 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:04.988 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:26:04.988 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:26:04.988 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:04.989 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 137062 2024-09-13T11:26:04.989 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.989 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:26:04.989 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.989 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:26:04.989 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:53.140525+0000", 2024-09-13T11:26:04.989 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:04.989 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.989 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.989 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:26:04.989 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:53.140525+0000", 2024-09-13T11:26:04.990 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:04.990 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.990 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.990 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:26:04.990 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:53.140527+0000", 2024-09-13T11:26:04.990 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.1739999999999999e-06 2024-09-13T11:26:04.990 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.990 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.990 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:26:04.990 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:53.140553+0000", 2024-09-13T11:26:04.990 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.6631e-05 2024-09-13T11:26:04.990 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.991 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.991 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:26:04.991 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:53.140566+0000", 2024-09-13T11:26:04.991 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.2268e-05 2024-09-13T11:26:04.991 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.991 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.991 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:26:04.991 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:53.140574+0000", 2024-09-13T11:26:04.991 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.5669999999999995e-06 2024-09-13T11:26:04.991 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.992 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.992 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:26:04.992 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:53.140622+0000", 2024-09-13T11:26:04.992 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.8143999999999998e-05 2024-09-13T11:26:04.992 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.992 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.992 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:26:04.992 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:53.140759+0000", 2024-09-13T11:26:04.992 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000136499 2024-09-13T11:26:04.992 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.992 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.993 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 13", 2024-09-13T11:26:04.993 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:53.140887+0000", 2024-09-13T11:26:04.993 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00012796300000000001 2024-09-13T11:26:04.993 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.993 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.993 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:26:04.993 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:53.142675+0000", 2024-09-13T11:26:04.993 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001788073 2024-09-13T11:26:04.993 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.993 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.993 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:04.994 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:53.347407+0000", 2024-09-13T11:26:04.994 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.20473206899999999 2024-09-13T11:26:04.994 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.994 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.994 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:26:04.994 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:53.347464+0000", 2024-09-13T11:26:04.994 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.6536000000000002e-05 2024-09-13T11:26:04.994 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.994 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.994 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:26:04.994 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:53.347504+0000", 2024-09-13T11:26:04.995 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.0621999999999998e-05 2024-09-13T11:26:04.995 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:04.995 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:26:04.995 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:04.995 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.995 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.995 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:161601 3.0 e155/75 3:03e8e5c6:::benchmark_data_smithi177_31116_object21298:head v 155'4743, mlcod=155'4741)", 2024-09-13T11:26:04.996 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:25:15.060319+0000", 2024-09-13T11:26:04.996 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 49.848559586, 2024-09-13T11:26:04.996 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.156824982, 2024-09-13T11:26:04.996 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:26:04.996 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:26:04.996 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:04.996 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:26:04.996 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.996 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:26:04.996 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:15.060319+0000", 2024-09-13T11:26:04.997 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:04.997 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.997 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.997 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:26:04.997 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:15.060319+0000", 2024-09-13T11:26:04.997 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:04.997 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.997 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.997 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:26:04.997 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:15.060320+0000", 2024-09-13T11:26:04.997 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.082e-06 2024-09-13T11:26:04.998 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.998 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.998 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:26:04.998 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:15.060325+0000", 2024-09-13T11:26:04.998 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.8110000000000002e-06 2024-09-13T11:26:04.998 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.998 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.998 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:26:04.998 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:15.212041+0000", 2024-09-13T11:26:04.999 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.15171630799999999 2024-09-13T11:26:04.999 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.999 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.999 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:26:04.999 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:15.212048+0000", 2024-09-13T11:26:04.999 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.7499999999999997e-06 2024-09-13T11:26:04.999 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:04.999 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:04.999 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:26:04.999 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:15.212106+0000", 2024-09-13T11:26:04.999 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.8870999999999999e-05 2024-09-13T11:26:04.999 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:05.000 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:05.000 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:26:05.000 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:15.212122+0000", 2024-09-13T11:26:05.000 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5175e-05 2024-09-13T11:26:05.000 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:05.000 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:05.000 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:26:05.000 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:15.217112+0000", 2024-09-13T11:26:05.000 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.004989991 2024-09-13T11:26:05.000 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:05.000 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:05.001 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:26:05.001 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:15.217144+0000", 2024-09-13T11:26:05.001 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.1993999999999999e-05 2024-09-13T11:26:05.001 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:05.001 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:26:05.001 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:05.001 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:05.001 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:05.001 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:176711 3.16 e167/88 3:68b3a0bf:::benchmark_data_smithi177_31116_object36408:head v 167'7062, mlcod=167'7058)", 2024-09-13T11:26:05.001 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:25:26.616425+0000", 2024-09-13T11:26:05.002 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 38.292452955000002, 2024-09-13T11:26:05.002 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.16947335599999999, 2024-09-13T11:26:05.002 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:26:05.002 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:26:05.002 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:05.002 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:26:05.002 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:05.002 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:26:05.002 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:26.616425+0000", 2024-09-13T11:26:05.002 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:05.003 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:05.003 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:05.003 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:26:05.003 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:26.616425+0000", 2024-09-13T11:26:05.003 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:05.003 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:05.003 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:05.003 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:26:05.003 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:26.616426+0000", 2024-09-13T11:26:05.003 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.1209999999999999e-06 2024-09-13T11:26:05.004 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:05.004 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:05.004 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:26:05.004 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:26.616432+0000", 2024-09-13T11:26:05.004 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.186e-06 2024-09-13T11:26:05.004 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:05.004 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:05.004 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:26:05.005 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:26.782875+0000", 2024-09-13T11:26:05.005 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.16644309199999999 2024-09-13T11:26:05.005 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:05.005 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:05.005 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:26:05.005 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:26.782880+0000", 2024-09-13T11:26:05.005 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.4580000000000001e-06 2024-09-13T11:26:05.005 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:05.005 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:05.005 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:26:05.005 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:26.782935+0000", 2024-09-13T11:26:05.006 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.5194e-05 2024-09-13T11:26:05.006 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:05.006 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:05.006 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:26:05.006 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:26.782951+0000", 2024-09-13T11:26:05.006 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5330999999999999e-05 2024-09-13T11:26:05.006 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:05.006 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:05.006 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:26:05.006 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:26.785854+0000", 2024-09-13T11:26:05.006 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0029030739999999998 2024-09-13T11:26:05.007 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:05.007 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:05.007 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:26:05.007 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:26.785899+0000", 2024-09-13T11:26:05.007 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.49e-05 2024-09-13T11:26:05.007 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:05.007 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:26:05.007 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:05.008 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:05.008 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:05.008 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:191058 3.1d e175/102 3:bd6e7029:::benchmark_data_smithi177_31116_object50755:head v 175'7612, mlcod=175'7609)", 2024-09-13T11:26:05.008 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:25:38.513527+0000", 2024-09-13T11:26:05.008 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 26.395350935, 2024-09-13T11:26:05.008 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.386104477, 2024-09-13T11:26:05.008 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:26:05.008 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:26:05.008 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:05.008 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:26:05.009 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:05.009 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:26:05.009 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:38.513527+0000", 2024-09-13T11:26:05.009 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:05.009 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:05.009 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:05.009 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:26:05.009 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:38.513527+0000", 2024-09-13T11:26:05.009 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:05.009 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:05.009 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:05.010 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:26:05.010 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:38.513530+0000", 2024-09-13T11:26:05.010 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.0029999999999999e-06 2024-09-13T11:26:05.010 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:05.010 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:05.010 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:26:05.010 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:38.513542+0000", 2024-09-13T11:26:05.010 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.1477e-05 2024-09-13T11:26:05.010 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:05.011 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:05.011 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:26:05.011 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:38.891617+0000", 2024-09-13T11:26:05.011 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.378075045 2024-09-13T11:26:05.011 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:05.011 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:05.011 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:26:05.011 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:38.891635+0000", 2024-09-13T11:26:05.011 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.7697e-05 2024-09-13T11:26:05.011 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:05.011 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:05.012 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:26:05.012 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:38.891775+0000", 2024-09-13T11:26:05.012 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000140665 2024-09-13T11:26:05.012 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:05.012 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:05.012 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:26:05.012 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:38.891833+0000", 2024-09-13T11:26:05.012 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.7769999999999997e-05 2024-09-13T11:26:05.012 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:05.012 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:05.012 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:26:05.013 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:38.899567+0000", 2024-09-13T11:26:05.013 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0077339970000000003 2024-09-13T11:26:05.013 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:05.013 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:05.013 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:26:05.013 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:38.899632+0000", 2024-09-13T11:26:05.013 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.4822999999999999e-05 2024-09-13T11:26:05.013 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:05.013 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:26:05.013 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:05.014 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:05.014 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:05.014 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:197236 3.5 3:a3127298:::benchmark_data_smithi177_31116_object56933:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e178)", 2024-09-13T11:26:05.014 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:25:44.300667+0000", 2024-09-13T11:26:05.014 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 20.608210843999998, 2024-09-13T11:26:05.014 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.37481205200000001, 2024-09-13T11:26:05.014 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:26:05.014 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:26:05.014 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:05.014 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:26:05.015 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:26:05.015 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:05.015 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 197236 2024-09-13T11:26:05.015 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:05.015 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:26:05.015 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:05.015 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:26:05.015 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:44.300667+0000", 2024-09-13T11:26:05.015 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:05.015 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:05.015 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:05.016 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:26:05.016 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:44.300667+0000", 2024-09-13T11:26:05.016 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:05.016 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:05.016 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:05.016 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:26:05.016 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:44.300670+0000", 2024-09-13T11:26:05.016 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.7259999999999998e-06 2024-09-13T11:26:05.016 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:05.016 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:05.016 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:26:05.016 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:44.300677+0000", 2024-09-13T11:26:05.017 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.1550000000000001e-06 2024-09-13T11:26:05.017 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:05.017 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:05.017 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:26:05.017 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:44.300695+0000", 2024-09-13T11:26:05.017 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.7552999999999999e-05 2024-09-13T11:26:05.017 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:05.017 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:05.017 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:26:05.017 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:44.300704+0000", 2024-09-13T11:26:05.018 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.5710000000000004e-06 2024-09-13T11:26:05.018 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:05.018 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:05.018 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:26:05.018 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:44.300762+0000", 2024-09-13T11:26:05.018 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.7351e-05 2024-09-13T11:26:05.018 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:05.018 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:05.018 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:26:05.018 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:44.300888+0000", 2024-09-13T11:26:05.018 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00012617999999999999 2024-09-13T11:26:05.018 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:05.018 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:05.019 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 13", 2024-09-13T11:26:05.019 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:44.300993+0000", 2024-09-13T11:26:05.019 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00010505600000000001 2024-09-13T11:26:05.019 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:05.019 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:05.019 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:26:05.019 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:44.302936+0000", 2024-09-13T11:26:05.019 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0019428749999999999 2024-09-13T11:26:05.019 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:05.020 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:05.020 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:05.020 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:44.675352+0000", 2024-09-13T11:26:05.020 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.37241603400000001 2024-09-13T11:26:05.020 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:05.020 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:05.020 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:26:05.020 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:44.675412+0000", 2024-09-13T11:26:05.020 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.9679999999999998e-05 2024-09-13T11:26:05.020 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:05.020 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:05.021 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:26:05.021 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:44.675479+0000", 2024-09-13T11:26:05.021 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.7871000000000001e-05 2024-09-13T11:26:05.021 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:05.021 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:26:05.021 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:05.021 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:05.021 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:05.022 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop_reply(client.4665.0:197236 3.5 e178/117 ondisk, result = 0)", 2024-09-13T11:26:05.022 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:25:44.304480+0000", 2024-09-13T11:26:05.022 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 20.604398577000001, 2024-09-13T11:26:05.022 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.37100864099999997, 2024-09-13T11:26:05.022 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:26:05.022 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:26:05.022 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:26:05.022 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:26:05.022 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:05.022 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:26:05.022 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:44.304480+0000", 2024-09-13T11:26:05.023 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:05.023 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:05.023 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:05.023 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:26:05.023 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:44.304480+0000", 2024-09-13T11:26:05.023 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:05.023 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:05.023 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:05.023 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:26:05.023 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:44.304481+0000", 2024-09-13T11:26:05.023 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.7349999999999999e-06 2024-09-13T11:26:05.023 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:05.024 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:05.024 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:26:05.024 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:44.304492+0000", 2024-09-13T11:26:05.024 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0394000000000001e-05 2024-09-13T11:26:05.024 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:05.024 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:05.024 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:26:05.024 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:44.675037+0000", 2024-09-13T11:26:05.024 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.37054519400000002 2024-09-13T11:26:05.024 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:05.024 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:05.024 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:26:05.025 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:44.675045+0000", 2024-09-13T11:26:05.025 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.1370000000000002e-06 2024-09-13T11:26:05.025 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:05.025 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:05.025 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:26:05.025 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:44.675334+0000", 2024-09-13T11:26:05.025 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00028848699999999998 2024-09-13T11:26:05.025 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:05.025 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:05.025 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:26:05.025 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:44.675345+0000", 2024-09-13T11:26:05.025 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.1198e-05 2024-09-13T11:26:05.026 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:05.026 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:05.026 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:26:05.026 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:44.675488+0000", 2024-09-13T11:26:05.026 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00014349600000000001 2024-09-13T11:26:05.026 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:05.026 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:26:05.026 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:05.026 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:05.026 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:05.027 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:217032 3.16 e187/88 3:6d28a27c:::benchmark_data_smithi177_31116_object76729:head v 187'8398, mlcod=187'8395)", 2024-09-13T11:26:05.027 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:25:59.836895+0000", 2024-09-13T11:26:05.027 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 5.0719836919999999, 2024-09-13T11:26:05.027 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.22574986, 2024-09-13T11:26:05.027 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:26:05.027 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:26:05.027 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:05.027 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:26:05.027 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:05.027 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:26:05.027 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:59.836895+0000", 2024-09-13T11:26:05.027 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:05.028 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:05.028 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:05.028 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:26:05.028 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:59.836895+0000", 2024-09-13T11:26:05.028 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:05.028 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:05.028 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:05.028 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:26:05.028 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:59.836896+0000", 2024-09-13T11:26:05.028 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.432e-06 2024-09-13T11:26:05.028 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:05.029 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:05.029 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:26:05.029 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:59.836904+0000", 2024-09-13T11:26:05.029 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.9149999999999996e-06 2024-09-13T11:26:05.029 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:05.029 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:05.029 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:26:05.029 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:00.053571+0000", 2024-09-13T11:26:05.029 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.21666679799999999 2024-09-13T11:26:05.029 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:05.029 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:05.029 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:26:05.030 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:00.053582+0000", 2024-09-13T11:26:05.030 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0927999999999999e-05 2024-09-13T11:26:05.030 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:05.030 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:05.030 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:26:05.030 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:00.053811+0000", 2024-09-13T11:26:05.030 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00022964700000000001 2024-09-13T11:26:05.030 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:05.030 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:05.030 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:26:05.030 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:00.053864+0000", 2024-09-13T11:26:05.031 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.2710000000000002e-05 2024-09-13T11:26:05.031 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:05.031 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:05.031 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:26:05.031 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:00.062547+0000", 2024-09-13T11:26:05.031 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0086832650000000008 2024-09-13T11:26:05.031 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:05.031 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:05.031 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:26:05.031 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:00.062644+0000", 2024-09-13T11:26:05.032 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.7164999999999996e-05 2024-09-13T11:26:05.032 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:05.032 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:26:05.032 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:05.032 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:05.032 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:26:05.032 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-09-13T11:26:05.033 INFO:tasks.ceph.osd.1.smithi092.stderr:2024-09-13T11:26:05.024+0000 7fe76f3f6640 -1 osd.1 191 handle_osd_map injecting map crc failure 2024-09-13T11:26:05.034 INFO:teuthology.orchestra.run.smithi001.stderr:nodeep-scrub is set 2024-09-13T11:26:05.034 INFO:tasks.ceph.osd.5.smithi092.stderr:2024-09-13T11:26:05.032+0000 7f6f0b705640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26346) UID: 0 2024-09-13T11:26:05.035 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-13T11:26:05.035 INFO:tasks.ceph.osd.3.smithi177.stderr:2024-09-13T11:26:05.029+0000 7f10a25d1640 -1 osd.3 191 handle_osd_map injecting map crc failure 2024-09-13T11:26:05.134 INFO:tasks.ceph.osd.12.smithi001.stderr:2024-09-13T11:26:05.130+0000 7f892c6e6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26314) UID: 0 2024-09-13T11:26:05.179 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-09-13T11:26:05.179 INFO:teuthology.orchestra.run.smithi092.stdout: "ops": [], 2024-09-13T11:26:05.179 INFO:teuthology.orchestra.run.smithi092.stdout: "num_ops": 0 2024-09-13T11:26:05.179 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-09-13T11:26:05.194 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-13T11:26:05.235 INFO:tasks.ceph.osd.14.smithi162.stderr:2024-09-13T11:26:05.231+0000 7f3df5903640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33154) UID: 0 2024-09-13T11:26:05.319 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-09-13T11:26:05.319 INFO:teuthology.orchestra.run.smithi092.stdout: "ops": [], 2024-09-13T11:26:05.319 INFO:teuthology.orchestra.run.smithi092.stdout: "complaint_time": 30, 2024-09-13T11:26:05.319 INFO:teuthology.orchestra.run.smithi092.stdout: "num_blocked_ops": 0 2024-09-13T11:26:05.319 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-09-13T11:26:05.336 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-13T11:26:05.337 INFO:tasks.ceph.osd.3.smithi177.stderr:2024-09-13T11:26:05.333+0000 7f10adba4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26574) UID: 0 2024-09-13T11:26:05.438 INFO:tasks.ceph.osd.1.smithi092.stderr:2024-09-13T11:26:05.436+0000 7fe7789c5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33718) UID: 0 2024-09-13T11:26:05.472 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-09-13T11:26:05.472 INFO:teuthology.orchestra.run.smithi092.stdout: "size": 20, 2024-09-13T11:26:05.472 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 600, 2024-09-13T11:26:05.472 INFO:teuthology.orchestra.run.smithi092.stdout: "ops": [ 2024-09-13T11:26:05.472 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:05.473 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:3 3.4 3.125e09e4 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e40)", 2024-09-13T11:26:05.473 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:46.304641+0000", 2024-09-13T11:26:05.473 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 199.16598490300001, 2024-09-13T11:26:05.473 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.374372119, 2024-09-13T11:26:05.473 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:26:05.473 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:26:05.473 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "reached pg", 2024-09-13T11:26:05.473 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:26:05.473 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:26:05.473 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:05.473 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 3 2024-09-13T11:26:05.473 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:05.473 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:26:05.474 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:05.474 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:26:05.474 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.304641+0000", 2024-09-13T11:26:05.474 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:05.474 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:05.474 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:05.474 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:26:05.474 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.304641+0000", 2024-09-13T11:26:05.474 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:05.474 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:05.474 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:05.474 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:26:05.474 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.304644+0000", 2024-09-13T11:26:05.475 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.5130000000000001e-06 2024-09-13T11:26:05.475 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:05.475 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:05.475 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:26:05.475 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305058+0000", 2024-09-13T11:26:05.475 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000414277 2024-09-13T11:26:05.475 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:05.475 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:05.475 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:26:05.475 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305124+0000", 2024-09-13T11:26:05.475 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.5813000000000002e-05 2024-09-13T11:26:05.475 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:05.475 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:05.476 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:26:05.476 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305156+0000", 2024-09-13T11:26:05.476 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.1489000000000003e-05 2024-09-13T11:26:05.476 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:05.476 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:05.476 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:05.476 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.678939+0000", 2024-09-13T11:26:05.476 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.37378339999999999 2024-09-13T11:26:05.476 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:05.476 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:05.476 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:26:05.476 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.679013+0000", 2024-09-13T11:26:05.476 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.3627000000000005e-05 2024-09-13T11:26:05.477 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:05.477 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:05.477 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:05.477 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:05.477 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:05.477 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:29 3.4 3.b7bbfbe4 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-09-13T11:26:05.477 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:46.305215+0000", 2024-09-13T11:26:05.477 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 199.1654106, 2024-09-13T11:26:05.477 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.37387964600000001, 2024-09-13T11:26:05.477 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:26:05.477 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:26:05.477 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "reached pg", 2024-09-13T11:26:05.477 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:26:05.478 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:26:05.478 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:05.479 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 29 2024-09-13T11:26:05.479 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:05.479 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:26:05.480 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:05.480 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:26:05.480 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305215+0000", 2024-09-13T11:26:05.480 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:05.480 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:05.480 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:05.480 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:26:05.480 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305215+0000", 2024-09-13T11:26:05.480 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:05.480 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:05.480 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:05.480 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:26:05.480 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305218+0000", 2024-09-13T11:26:05.481 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.7030000000000002e-06 2024-09-13T11:26:05.481 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:05.481 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:05.481 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:26:05.481 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305366+0000", 2024-09-13T11:26:05.481 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00014825400000000001 2024-09-13T11:26:05.481 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:05.481 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:05.481 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:26:05.481 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305386+0000", 2024-09-13T11:26:05.481 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.9689000000000001e-05 2024-09-13T11:26:05.481 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:05.481 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:05.482 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:26:05.482 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305403+0000", 2024-09-13T11:26:05.482 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.7600000000000001e-05 2024-09-13T11:26:05.482 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:05.482 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:05.482 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:05.482 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.679067+0000", 2024-09-13T11:26:05.482 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.37366339100000001 2024-09-13T11:26:05.482 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:05.482 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:05.482 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:26:05.482 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.679095+0000", 2024-09-13T11:26:05.482 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.8008999999999999e-05 2024-09-13T11:26:05.483 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:05.483 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:05.483 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:05.483 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:05.483 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:05.483 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:30 3.4 3.547ec0f4 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-09-13T11:26:05.483 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:46.305453+0000", 2024-09-13T11:26:05.483 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 199.16517247600001, 2024-09-13T11:26:05.483 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.373764914, 2024-09-13T11:26:05.483 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:26:05.483 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:26:05.483 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "reached pg", 2024-09-13T11:26:05.483 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:26:05.484 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:26:05.484 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:05.484 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 30 2024-09-13T11:26:05.484 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:05.484 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:26:05.484 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:05.484 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:26:05.484 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305453+0000", 2024-09-13T11:26:05.484 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:05.484 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:05.484 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:05.484 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:26:05.484 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305453+0000", 2024-09-13T11:26:05.485 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:05.485 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:05.485 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:05.485 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:26:05.485 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305456+0000", 2024-09-13T11:26:05.485 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.9560000000000002e-06 2024-09-13T11:26:05.485 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:05.485 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:05.485 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:26:05.485 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305597+0000", 2024-09-13T11:26:05.485 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00014122700000000001 2024-09-13T11:26:05.485 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:05.485 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:05.486 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:26:05.486 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305615+0000", 2024-09-13T11:26:05.486 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.8139999999999999e-05 2024-09-13T11:26:05.486 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:05.486 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:05.486 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:26:05.486 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305632+0000", 2024-09-13T11:26:05.486 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.6101999999999999e-05 2024-09-13T11:26:05.486 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:05.486 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:05.486 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:05.486 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.679195+0000", 2024-09-13T11:26:05.486 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.373563486 2024-09-13T11:26:05.487 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:05.487 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:05.487 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:26:05.487 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.679218+0000", 2024-09-13T11:26:05.487 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.3003e-05 2024-09-13T11:26:05.487 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:05.487 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:05.487 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:05.487 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:05.487 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:05.487 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:70 3.4 3.1404a604 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-09-13T11:26:05.487 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:46.305665+0000", 2024-09-13T11:26:05.487 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 199.16496032800001, 2024-09-13T11:26:05.487 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.37365550199999997, 2024-09-13T11:26:05.488 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:26:05.488 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:26:05.488 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "reached pg", 2024-09-13T11:26:05.488 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:26:05.488 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:26:05.488 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:05.488 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 70 2024-09-13T11:26:05.488 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:05.488 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:26:05.488 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:05.488 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:26:05.488 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305665+0000", 2024-09-13T11:26:05.488 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:05.489 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:05.489 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:05.489 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:26:05.489 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305665+0000", 2024-09-13T11:26:05.489 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:05.489 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:05.489 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:05.489 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:26:05.489 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305668+0000", 2024-09-13T11:26:05.489 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.5890000000000001e-06 2024-09-13T11:26:05.489 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:05.489 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:05.489 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:26:05.490 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305791+0000", 2024-09-13T11:26:05.490 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000123545 2024-09-13T11:26:05.490 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:05.490 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:05.490 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:26:05.490 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305808+0000", 2024-09-13T11:26:05.490 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.668e-05 2024-09-13T11:26:05.490 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:05.490 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:05.490 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:26:05.491 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305824+0000", 2024-09-13T11:26:05.491 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.6385e-05 2024-09-13T11:26:05.491 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:05.491 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:05.491 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:05.491 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.679305+0000", 2024-09-13T11:26:05.491 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.37348059900000002 2024-09-13T11:26:05.491 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:05.491 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:05.491 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:26:05.491 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.679321+0000", 2024-09-13T11:26:05.491 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.5704000000000001e-05 2024-09-13T11:26:05.492 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:05.492 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:05.492 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:05.492 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:05.492 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:05.492 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:25408 3.4 3:294e23bd:::benchmark_data_smithi177_31116_object25407:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e56)", 2024-09-13T11:26:05.492 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:04.495455+0000", 2024-09-13T11:26:05.492 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 180.975170676, 2024-09-13T11:26:05.492 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.29897367600000002, 2024-09-13T11:26:05.492 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:26:05.493 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:26:05.493 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:05.493 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:26:05.493 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:26:05.493 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:05.493 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 25408 2024-09-13T11:26:05.493 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:05.493 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:26:05.493 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:05.493 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:26:05.494 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.495455+0000", 2024-09-13T11:26:05.494 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:05.494 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:05.494 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:05.494 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:26:05.494 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.495455+0000", 2024-09-13T11:26:05.494 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:05.494 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:05.494 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:05.495 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:26:05.495 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.495459+0000", 2024-09-13T11:26:05.495 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.5630000000000002e-06 2024-09-13T11:26:05.495 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:05.495 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:05.495 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:26:05.495 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.495545+0000", 2024-09-13T11:26:05.495 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.5704000000000005e-05 2024-09-13T11:26:05.495 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:05.495 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:05.495 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:26:05.495 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.495581+0000", 2024-09-13T11:26:05.495 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.5689000000000003e-05 2024-09-13T11:26:05.495 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:05.496 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:05.496 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:26:05.496 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.495597+0000", 2024-09-13T11:26:05.496 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.6053e-05 2024-09-13T11:26:05.496 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:05.496 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:05.496 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:05.496 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.495721+0000", 2024-09-13T11:26:05.497 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00012379099999999999 2024-09-13T11:26:05.497 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:05.497 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:05.497 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:26:05.497 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.495940+0000", 2024-09-13T11:26:05.497 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00021967699999999999 2024-09-13T11:26:05.497 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:05.497 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:05.497 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 12", 2024-09-13T11:26:05.497 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.496162+0000", 2024-09-13T11:26:05.497 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000221917 2024-09-13T11:26:05.497 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:05.498 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:05.498 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:26:05.498 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.498699+0000", 2024-09-13T11:26:05.498 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0025366690000000001 2024-09-13T11:26:05.498 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:05.498 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:05.498 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:05.498 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.794139+0000", 2024-09-13T11:26:05.498 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.295440025 2024-09-13T11:26:05.498 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:05.498 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:05.498 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:26:05.499 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.794306+0000", 2024-09-13T11:26:05.499 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00016689799999999999 2024-09-13T11:26:05.499 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:05.499 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:05.499 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:26:05.499 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.794429+0000", 2024-09-13T11:26:05.499 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00012269000000000001 2024-09-13T11:26:05.499 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:05.499 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:05.499 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:05.499 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:05.499 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:05.499 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "MOSDPGPushReply(3.14 75/70 [PushReplyOp(3:2f14928a:::benchmark_data_smithi177_31116_object15123:head)])", 2024-09-13T11:26:05.500 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:31.569662+0000", 2024-09-13T11:26:05.500 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 153.90096390400001, 2024-09-13T11:26:05.500 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.35759148299999999, 2024-09-13T11:26:05.500 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:26:05.500 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:26:05.500 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "reached pg", 2024-09-13T11:26:05.500 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:26:05.500 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:05.500 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:26:05.500 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:31.569662+0000", 2024-09-13T11:26:05.500 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:05.500 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:05.501 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:05.501 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:26:05.501 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:31.569662+0000", 2024-09-13T11:26:05.501 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:05.501 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:05.501 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:05.501 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:26:05.501 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:31.569663+0000", 2024-09-13T11:26:05.501 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.333e-06 2024-09-13T11:26:05.501 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:05.501 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:05.501 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:26:05.502 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:31.569667+0000", 2024-09-13T11:26:05.502 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.8929999999999998e-06 2024-09-13T11:26:05.502 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:05.502 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:05.502 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:26:05.502 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:31.923221+0000", 2024-09-13T11:26:05.502 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.35355413699999999 2024-09-13T11:26:05.502 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:05.502 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:05.502 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:26:05.502 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:31.923230+0000", 2024-09-13T11:26:05.502 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.6829999999999997e-06 2024-09-13T11:26:05.503 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:05.503 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:05.503 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:05.503 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:31.923807+0000", 2024-09-13T11:26:05.503 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00057686800000000002 2024-09-13T11:26:05.503 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:05.503 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:05.503 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:26:05.503 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:31.927253+0000", 2024-09-13T11:26:05.503 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.003446569 2024-09-13T11:26:05.503 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:05.503 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:05.503 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:05.503 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:05.504 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:05.504 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "MOSDPGPushReply(3.18 85/70 [PushReplyOp(3:1e410119:::benchmark_data_smithi177_31116_object35700:head)])", 2024-09-13T11:26:05.504 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:42.216614+0000", 2024-09-13T11:26:05.504 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 143.25401156000001, 2024-09-13T11:26:05.504 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.54260090699999997, 2024-09-13T11:26:05.504 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:26:05.504 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:26:05.504 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "reached pg", 2024-09-13T11:26:05.504 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:26:05.504 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:05.504 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:26:05.504 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.216614+0000", 2024-09-13T11:26:05.504 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:05.505 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:05.505 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:05.505 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:26:05.505 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.216614+0000", 2024-09-13T11:26:05.505 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:05.505 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:05.505 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:05.505 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:26:05.505 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.216615+0000", 2024-09-13T11:26:05.505 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.0200000000000001e-07 2024-09-13T11:26:05.505 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:05.505 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:05.505 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:26:05.506 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.216619+0000", 2024-09-13T11:26:05.506 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.805e-06 2024-09-13T11:26:05.506 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:05.506 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:05.506 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:26:05.506 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.757476+0000", 2024-09-13T11:26:05.506 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.54085762000000004 2024-09-13T11:26:05.506 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:05.506 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:05.506 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:26:05.506 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.757498+0000", 2024-09-13T11:26:05.506 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.1936000000000001e-05 2024-09-13T11:26:05.506 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:05.507 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:05.507 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:05.507 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.757678+0000", 2024-09-13T11:26:05.507 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00018006599999999999 2024-09-13T11:26:05.507 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:05.507 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:05.507 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:26:05.507 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.759215+0000", 2024-09-13T11:26:05.507 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0015366780000000001 2024-09-13T11:26:05.507 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:05.507 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:05.507 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:05.507 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:05.508 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:05.508 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "MOSDPGPushReply(3.1b 85/78 [PushReplyOp(3:dabda1cc:::benchmark_data_smithi177_31116_object12988:head)])", 2024-09-13T11:26:05.508 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:42.485171+0000", 2024-09-13T11:26:05.508 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 142.98545465199999, 2024-09-13T11:26:05.508 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.308420427, 2024-09-13T11:26:05.508 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:26:05.508 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:26:05.508 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "reached pg", 2024-09-13T11:26:05.508 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:26:05.508 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:05.508 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:26:05.508 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.485171+0000", 2024-09-13T11:26:05.509 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:05.509 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:05.509 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:05.509 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:26:05.509 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.485171+0000", 2024-09-13T11:26:05.509 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:05.509 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:05.509 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:05.509 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:26:05.509 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.485173+0000", 2024-09-13T11:26:05.509 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.1569999999999998e-06 2024-09-13T11:26:05.509 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:05.509 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:05.509 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:26:05.510 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.485182+0000", 2024-09-13T11:26:05.510 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.6759999999999996e-06 2024-09-13T11:26:05.510 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:05.510 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:05.510 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:26:05.510 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.791336+0000", 2024-09-13T11:26:05.510 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.30615446499999999 2024-09-13T11:26:05.510 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:05.510 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:05.510 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:26:05.510 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.791369+0000", 2024-09-13T11:26:05.510 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.3155999999999999e-05 2024-09-13T11:26:05.510 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:05.511 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:05.511 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:05.511 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.791472+0000", 2024-09-13T11:26:05.511 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00010247900000000001 2024-09-13T11:26:05.511 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:05.511 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:05.511 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:26:05.511 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.793591+0000", 2024-09-13T11:26:05.511 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.002119494 2024-09-13T11:26:05.511 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:05.511 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:05.511 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:05.511 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:05.512 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:05.512 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:64066 3.1b 3:dd84e0ec:::benchmark_data_smithi177_31116_object64065:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89)", 2024-09-13T11:26:05.512 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:46.633608+0000", 2024-09-13T11:26:05.512 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 138.837017213, 2024-09-13T11:26:05.512 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.32564315399999999, 2024-09-13T11:26:05.512 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:26:05.512 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:26:05.512 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:05.512 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:26:05.512 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:26:05.512 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:05.512 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 64066 2024-09-13T11:26:05.512 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:05.512 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:26:05.513 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:05.513 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:26:05.513 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.633608+0000", 2024-09-13T11:26:05.513 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:05.513 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:05.513 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:05.513 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:26:05.513 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.633608+0000", 2024-09-13T11:26:05.513 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:05.513 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:05.513 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:05.513 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:26:05.514 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.633612+0000", 2024-09-13T11:26:05.514 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.856e-06 2024-09-13T11:26:05.514 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:05.514 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:05.514 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:26:05.514 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.633683+0000", 2024-09-13T11:26:05.514 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.1000999999999993e-05 2024-09-13T11:26:05.514 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:05.514 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:05.514 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:26:05.515 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.633711+0000", 2024-09-13T11:26:05.515 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.8204999999999998e-05 2024-09-13T11:26:05.515 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:05.515 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:05.515 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:26:05.515 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.633732+0000", 2024-09-13T11:26:05.515 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.0662999999999999e-05 2024-09-13T11:26:05.515 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:05.515 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:05.515 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:05.515 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.635573+0000", 2024-09-13T11:26:05.515 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.001841365 2024-09-13T11:26:05.515 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:05.516 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:05.516 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:26:05.516 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.635995+0000", 2024-09-13T11:26:05.516 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00042113699999999998 2024-09-13T11:26:05.516 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:05.516 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:05.516 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 12,15", 2024-09-13T11:26:05.516 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.636475+0000", 2024-09-13T11:26:05.516 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00048067699999999999 2024-09-13T11:26:05.516 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:05.516 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:05.516 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:05.516 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.640247+0000", 2024-09-13T11:26:05.517 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0037716450000000001 2024-09-13T11:26:05.517 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:05.517 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:05.517 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:26:05.517 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.649308+0000", 2024-09-13T11:26:05.517 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0090613240000000008 2024-09-13T11:26:05.517 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:05.517 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:05.517 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:05.517 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.956912+0000", 2024-09-13T11:26:05.517 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.30760338500000001 2024-09-13T11:26:05.517 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:05.517 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:05.518 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:26:05.518 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.957156+0000", 2024-09-13T11:26:05.518 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000244031 2024-09-13T11:26:05.518 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:05.518 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:05.518 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:26:05.518 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.959252+0000", 2024-09-13T11:26:05.518 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0020958650000000001 2024-09-13T11:26:05.518 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:05.518 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:05.518 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:05.518 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:05.518 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:05.518 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop_reply(client.4665.0:64066 3.1b e89/78 ondisk, result = 0)", 2024-09-13T11:26:05.519 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:46.651596+0000", 2024-09-13T11:26:05.519 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 138.819029382, 2024-09-13T11:26:05.519 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.30767612599999999, 2024-09-13T11:26:05.519 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:26:05.519 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:26:05.519 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-09-13T11:26:05.519 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:26:05.519 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:05.519 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:26:05.519 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.651596+0000", 2024-09-13T11:26:05.519 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:05.519 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:05.519 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:05.520 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:26:05.520 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.651596+0000", 2024-09-13T11:26:05.520 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:05.520 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:05.520 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:05.520 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:26:05.520 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.651600+0000", 2024-09-13T11:26:05.520 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.6969999999999999e-06 2024-09-13T11:26:05.520 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:05.520 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:05.520 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:26:05.520 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.651611+0000", 2024-09-13T11:26:05.521 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.1516e-05 2024-09-13T11:26:05.521 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:05.521 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:05.521 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:26:05.521 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.948118+0000", 2024-09-13T11:26:05.521 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.29650663500000002 2024-09-13T11:26:05.521 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:05.521 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:05.521 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:26:05.521 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.948137+0000", 2024-09-13T11:26:05.521 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.8529000000000002e-05 2024-09-13T11:26:05.521 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:05.522 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:05.522 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:05.522 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.956870+0000", 2024-09-13T11:26:05.522 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0087336380000000002 2024-09-13T11:26:05.522 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:05.522 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:05.522 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:26:05.522 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.956893+0000", 2024-09-13T11:26:05.522 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.317e-05 2024-09-13T11:26:05.522 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:05.522 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:05.522 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:26:05.522 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.959272+0000", 2024-09-13T11:26:05.523 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0023789409999999999 2024-09-13T11:26:05.523 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:05.523 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:05.523 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:05.523 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:05.523 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:05.523 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "MOSDPGPushReply(3.1b 91/78 [PushReplyOp(3:d9ac6889:::benchmark_data_smithi177_31116_object22254:head)])", 2024-09-13T11:26:05.523 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:48.392729+0000", 2024-09-13T11:26:05.523 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 137.07789646699999, 2024-09-13T11:26:05.523 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.30924993099999998, 2024-09-13T11:26:05.523 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:26:05.523 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:26:05.524 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "reached pg", 2024-09-13T11:26:05.524 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:26:05.524 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:05.524 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:26:05.524 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:48.392729+0000", 2024-09-13T11:26:05.524 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:05.524 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:05.524 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:05.524 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:26:05.524 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:48.392729+0000", 2024-09-13T11:26:05.524 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:05.524 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:05.524 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:05.525 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:26:05.525 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:48.392730+0000", 2024-09-13T11:26:05.525 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.336e-06 2024-09-13T11:26:05.525 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:05.525 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:05.525 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:26:05.525 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:48.392739+0000", 2024-09-13T11:26:05.525 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.8580000000000006e-06 2024-09-13T11:26:05.525 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:05.525 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:05.525 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:26:05.525 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:48.700180+0000", 2024-09-13T11:26:05.525 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.30744035600000003 2024-09-13T11:26:05.526 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:05.526 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:05.526 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:26:05.526 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:48.700197+0000", 2024-09-13T11:26:05.526 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.7649999999999999e-05 2024-09-13T11:26:05.526 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:05.526 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:05.526 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:05.526 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:48.700398+0000", 2024-09-13T11:26:05.526 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000201113 2024-09-13T11:26:05.526 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:05.526 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:05.526 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:26:05.527 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:48.701979+0000", 2024-09-13T11:26:05.527 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.001580618 2024-09-13T11:26:05.527 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:05.527 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:05.527 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:05.527 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:05.527 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:05.527 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop(client.4665.0:83193 3.8 e106/67 3:10066d83:::benchmark_data_smithi177_31116_object83192:head v 106'3925, mlcod=106'3922)", 2024-09-13T11:26:05.527 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:24:04.285068+0000", 2024-09-13T11:26:05.527 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 121.185557712, 2024-09-13T11:26:05.527 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.64647716600000005, 2024-09-13T11:26:05.527 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:26:05.527 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:26:05.528 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:05.528 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:26:05.528 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:05.528 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:26:05.528 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:04.285068+0000", 2024-09-13T11:26:05.528 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:05.528 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:05.528 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:05.528 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:26:05.528 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:04.285068+0000", 2024-09-13T11:26:05.528 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:05.528 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:05.528 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:05.529 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:26:05.529 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:04.285071+0000", 2024-09-13T11:26:05.529 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.729e-06 2024-09-13T11:26:05.529 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:05.529 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:05.529 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:26:05.529 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:04.285215+0000", 2024-09-13T11:26:05.529 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00014410000000000001 2024-09-13T11:26:05.529 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:05.529 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:05.529 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:26:05.529 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:04.928221+0000", 2024-09-13T11:26:05.529 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.64300653299999999 2024-09-13T11:26:05.530 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:05.530 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:05.530 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:26:05.530 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:04.928240+0000", 2024-09-13T11:26:05.530 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.8678999999999999e-05 2024-09-13T11:26:05.530 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:05.530 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:05.530 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:05.530 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:04.928388+0000", 2024-09-13T11:26:05.530 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00014809 2024-09-13T11:26:05.530 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:05.530 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:05.531 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:26:05.531 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:04.928428+0000", 2024-09-13T11:26:05.531 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.9776e-05 2024-09-13T11:26:05.531 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:05.531 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:05.531 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:26:05.531 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:04.931463+0000", 2024-09-13T11:26:05.531 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.003035168 2024-09-13T11:26:05.531 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:05.531 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:05.531 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:26:05.531 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:04.931545+0000", 2024-09-13T11:26:05.531 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.2090999999999997e-05 2024-09-13T11:26:05.531 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:05.532 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:05.532 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:05.532 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:05.533 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:05.533 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop(client.4665.0:83239 3.e e106/75 3:74547f17:::benchmark_data_smithi177_31116_object83238:head v 106'4391, mlcod=106'4388)", 2024-09-13T11:26:05.533 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:24:04.581196+0000", 2024-09-13T11:26:05.533 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 120.889429206, 2024-09-13T11:26:05.533 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.33617203600000001, 2024-09-13T11:26:05.533 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:26:05.533 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:26:05.533 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:05.533 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:26:05.533 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:05.533 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:26:05.533 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:04.581196+0000", 2024-09-13T11:26:05.534 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:05.534 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:05.534 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:05.534 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:26:05.534 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:04.581196+0000", 2024-09-13T11:26:05.534 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:05.534 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:05.534 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:05.534 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:26:05.534 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:04.581200+0000", 2024-09-13T11:26:05.534 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.19e-06 2024-09-13T11:26:05.534 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:05.534 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:05.535 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:26:05.535 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:04.581257+0000", 2024-09-13T11:26:05.535 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.7531999999999997e-05 2024-09-13T11:26:05.535 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:05.535 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:05.535 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:26:05.535 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:04.914011+0000", 2024-09-13T11:26:05.535 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.33275413700000001 2024-09-13T11:26:05.535 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:05.535 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:05.535 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:26:05.535 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:04.914028+0000", 2024-09-13T11:26:05.535 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.7082999999999999e-05 2024-09-13T11:26:05.536 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:05.536 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:05.536 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:05.536 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:04.914151+0000", 2024-09-13T11:26:05.536 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00012283300000000001 2024-09-13T11:26:05.536 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:05.536 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:05.536 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:26:05.536 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:04.914185+0000", 2024-09-13T11:26:05.536 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.4255999999999999e-05 2024-09-13T11:26:05.536 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:05.536 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:05.536 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:26:05.536 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:04.917300+0000", 2024-09-13T11:26:05.537 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0031141599999999999 2024-09-13T11:26:05.537 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:05.537 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:05.537 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:26:05.537 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:04.917368+0000", 2024-09-13T11:26:05.537 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.8844999999999996e-05 2024-09-13T11:26:05.537 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:05.537 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:05.537 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:05.537 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:05.537 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:05.537 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:131687 3.28 3:14f65078:::benchmark_data_smithi177_31116_object131686:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e135)", 2024-09-13T11:26:05.538 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:24:48.925565+0000", 2024-09-13T11:26:05.538 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 76.545060208999999, 2024-09-13T11:26:05.538 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.41421604099999998, 2024-09-13T11:26:05.538 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:26:05.538 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:26:05.538 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:05.538 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:26:05.538 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:26:05.538 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:05.538 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 131687 2024-09-13T11:26:05.538 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:05.538 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:26:05.538 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:05.539 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:26:05.539 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:48.925565+0000", 2024-09-13T11:26:05.539 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:05.539 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:05.539 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:05.539 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:26:05.539 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:48.925565+0000", 2024-09-13T11:26:05.539 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:05.539 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:05.539 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:05.539 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:26:05.539 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:48.925568+0000", 2024-09-13T11:26:05.539 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.441e-06 2024-09-13T11:26:05.539 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:05.540 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:05.540 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:26:05.540 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:48.925617+0000", 2024-09-13T11:26:05.540 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.8739000000000002e-05 2024-09-13T11:26:05.540 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:05.540 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:05.540 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:26:05.540 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:48.925631+0000", 2024-09-13T11:26:05.540 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.4463e-05 2024-09-13T11:26:05.540 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:05.540 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:05.540 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:26:05.540 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:48.925638+0000", 2024-09-13T11:26:05.541 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.2119999999999997e-06 2024-09-13T11:26:05.541 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:05.541 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:05.541 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:05.541 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:48.925703+0000", 2024-09-13T11:26:05.541 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.4794000000000003e-05 2024-09-13T11:26:05.541 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:05.541 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:05.541 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:26:05.541 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:48.925857+0000", 2024-09-13T11:26:05.541 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00015427 2024-09-13T11:26:05.541 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:05.541 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:05.541 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 5,8", 2024-09-13T11:26:05.542 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:48.925995+0000", 2024-09-13T11:26:05.542 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000137365 2024-09-13T11:26:05.542 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:05.542 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:05.542 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:26:05.542 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:48.927261+0000", 2024-09-13T11:26:05.542 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.001266021 2024-09-13T11:26:05.542 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:05.542 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:05.542 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:05.542 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:48.930015+0000", 2024-09-13T11:26:05.542 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0027545130000000001 2024-09-13T11:26:05.542 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:05.543 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:05.543 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:05.543 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:49.338898+0000", 2024-09-13T11:26:05.543 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.40888254299999999 2024-09-13T11:26:05.543 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:05.543 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:05.543 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:26:05.543 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:49.338967+0000", 2024-09-13T11:26:05.543 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.9402999999999999e-05 2024-09-13T11:26:05.543 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:05.543 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:05.543 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:26:05.543 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:49.339781+0000", 2024-09-13T11:26:05.543 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00081427699999999997 2024-09-13T11:26:05.544 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:05.544 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:05.544 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:05.544 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:05.544 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:05.544 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop_reply(client.4665.0:131687 3.28 e135/120 ondisk, result = 0)", 2024-09-13T11:26:05.544 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:24:48.928935+0000", 2024-09-13T11:26:05.544 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 76.541690908000007, 2024-09-13T11:26:05.544 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.41085638499999999, 2024-09-13T11:26:05.544 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:26:05.544 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:26:05.544 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-09-13T11:26:05.544 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:26:05.545 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:05.545 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:26:05.545 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:48.928935+0000", 2024-09-13T11:26:05.545 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:05.545 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:05.545 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:05.545 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:26:05.545 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:48.928935+0000", 2024-09-13T11:26:05.545 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:05.545 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:05.545 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:05.545 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:26:05.545 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:48.928936+0000", 2024-09-13T11:26:05.546 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.519e-06 2024-09-13T11:26:05.546 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:05.546 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:05.546 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:26:05.546 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:48.928941+0000", 2024-09-13T11:26:05.546 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.0710000000000002e-06 2024-09-13T11:26:05.546 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:05.546 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:05.546 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:26:05.546 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:49.338799+0000", 2024-09-13T11:26:05.546 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.40985808499999998 2024-09-13T11:26:05.546 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:05.546 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:05.547 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:26:05.547 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:49.338809+0000", 2024-09-13T11:26:05.547 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.4390000000000006e-06 2024-09-13T11:26:05.547 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:05.547 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:05.547 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:05.547 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:49.338878+0000", 2024-09-13T11:26:05.547 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.9429000000000002e-05 2024-09-13T11:26:05.547 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:05.547 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:05.547 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:26:05.547 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:49.338890+0000", 2024-09-13T11:26:05.547 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.2217e-05 2024-09-13T11:26:05.548 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:05.548 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:05.548 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:26:05.548 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:49.339791+0000", 2024-09-13T11:26:05.548 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00090062499999999995 2024-09-13T11:26:05.548 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:05.548 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:05.548 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:05.548 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:05.548 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:05.548 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "MOSDPGPushReply(3.28 163/120 [PushReplyOp(3:169fe9fd:::benchmark_data_smithi177_31116_object59744:head)])", 2024-09-13T11:26:05.548 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:25:22.263786+0000", 2024-09-13T11:26:05.549 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 43.206839639999998, 2024-09-13T11:26:05.549 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.28927712799999999, 2024-09-13T11:26:05.549 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:26:05.549 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:26:05.549 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "reached pg", 2024-09-13T11:26:05.549 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:26:05.549 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:05.549 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:26:05.549 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:22.263786+0000", 2024-09-13T11:26:05.549 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:05.549 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:05.549 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:05.549 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:26:05.550 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:22.263786+0000", 2024-09-13T11:26:05.550 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:05.550 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:05.550 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:05.550 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:26:05.550 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:22.263788+0000", 2024-09-13T11:26:05.550 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.542e-06 2024-09-13T11:26:05.550 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:05.550 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:05.550 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:26:05.550 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:22.263797+0000", 2024-09-13T11:26:05.550 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.234e-06 2024-09-13T11:26:05.550 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:05.551 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:05.551 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:26:05.551 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:22.552310+0000", 2024-09-13T11:26:05.551 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.28851281499999998 2024-09-13T11:26:05.551 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:05.551 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:05.551 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:26:05.551 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:22.552315+0000", 2024-09-13T11:26:05.551 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.5990000000000001e-06 2024-09-13T11:26:05.551 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:05.551 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:05.551 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:05.551 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:22.552362+0000", 2024-09-13T11:26:05.551 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.7144000000000001e-05 2024-09-13T11:26:05.552 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:05.552 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:05.552 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:26:05.552 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:22.553063+0000", 2024-09-13T11:26:05.552 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00070079400000000003 2024-09-13T11:26:05.552 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:05.552 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:05.552 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:05.552 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:05.552 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:05.552 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:181661 3.4 3:2019f47a:::benchmark_data_smithi177_31116_object41358:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e170)", 2024-09-13T11:26:05.552 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:25:31.131054+0000", 2024-09-13T11:26:05.552 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 34.339571141999997, 2024-09-13T11:26:05.553 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.44761152399999998, 2024-09-13T11:26:05.553 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:26:05.553 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:26:05.553 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:05.553 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:26:05.553 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:26:05.553 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:05.553 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 181661 2024-09-13T11:26:05.553 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:05.553 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:26:05.553 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:05.553 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:26:05.554 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:31.131054+0000", 2024-09-13T11:26:05.554 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:05.554 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:05.554 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:05.554 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:26:05.554 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:31.131054+0000", 2024-09-13T11:26:05.554 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:05.554 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:05.554 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:05.554 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:26:05.554 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:31.131056+0000", 2024-09-13T11:26:05.554 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.8640000000000001e-06 2024-09-13T11:26:05.555 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:05.555 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:05.555 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:26:05.555 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:31.131061+0000", 2024-09-13T11:26:05.555 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.6249999999999998e-06 2024-09-13T11:26:05.555 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:05.555 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:05.555 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:26:05.555 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:31.131069+0000", 2024-09-13T11:26:05.555 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.1049999999999995e-06 2024-09-13T11:26:05.555 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:05.555 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:05.555 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:26:05.555 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:31.131074+0000", 2024-09-13T11:26:05.556 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.344e-06 2024-09-13T11:26:05.556 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:05.556 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:05.556 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:05.556 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:31.131121+0000", 2024-09-13T11:26:05.556 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.6794e-05 2024-09-13T11:26:05.556 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:05.556 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:05.556 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:26:05.556 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:31.131225+0000", 2024-09-13T11:26:05.556 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000103502 2024-09-13T11:26:05.556 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:05.557 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:05.557 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 12", 2024-09-13T11:26:05.557 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:31.131289+0000", 2024-09-13T11:26:05.557 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.4790999999999997e-05 2024-09-13T11:26:05.557 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:05.557 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:05.557 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:26:05.557 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:31.133191+0000", 2024-09-13T11:26:05.557 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0019013859999999999 2024-09-13T11:26:05.557 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:05.557 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:05.557 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:05.557 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:31.578585+0000", 2024-09-13T11:26:05.558 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.44539403 2024-09-13T11:26:05.558 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:05.558 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:05.558 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:26:05.558 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:31.578620+0000", 2024-09-13T11:26:05.558 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.4657999999999999e-05 2024-09-13T11:26:05.558 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:05.558 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:05.558 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:26:05.558 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:31.578666+0000", 2024-09-13T11:26:05.558 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.6424999999999997e-05 2024-09-13T11:26:05.558 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:05.558 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:05.559 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:05.559 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:05.559 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:05.559 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop_reply(client.4665.0:181661 3.4 e170/117 ondisk, result = 0)", 2024-09-13T11:26:05.559 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:25:31.135385+0000", 2024-09-13T11:26:05.559 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 34.335240321999997, 2024-09-13T11:26:05.559 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.44328512599999997, 2024-09-13T11:26:05.559 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:26:05.559 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:26:05.559 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-09-13T11:26:05.559 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:26:05.559 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:05.560 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:26:05.560 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:31.135385+0000", 2024-09-13T11:26:05.560 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:05.560 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:05.560 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:05.560 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:26:05.560 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:31.135385+0000", 2024-09-13T11:26:05.560 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:05.560 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:05.560 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:05.560 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:26:05.560 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:31.135387+0000", 2024-09-13T11:26:05.560 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.217e-06 2024-09-13T11:26:05.560 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:05.561 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:05.561 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:26:05.561 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:31.135391+0000", 2024-09-13T11:26:05.561 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.0820000000000001e-06 2024-09-13T11:26:05.561 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:05.561 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:05.561 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:26:05.561 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:31.578504+0000", 2024-09-13T11:26:05.561 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.44311382100000002 2024-09-13T11:26:05.561 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:05.561 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:05.561 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:26:05.561 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:31.578530+0000", 2024-09-13T11:26:05.562 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.5367e-05 2024-09-13T11:26:05.562 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:05.562 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:05.562 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:05.562 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:31.578576+0000", 2024-09-13T11:26:05.562 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.5840999999999997e-05 2024-09-13T11:26:05.562 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:05.562 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:05.562 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:26:05.562 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:31.578581+0000", 2024-09-13T11:26:05.562 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.5620000000000003e-06 2024-09-13T11:26:05.562 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:05.563 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:05.563 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:26:05.563 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:31.578670+0000", 2024-09-13T11:26:05.563 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.9235999999999997e-05 2024-09-13T11:26:05.563 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:05.563 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:05.563 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:05.563 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:05.563 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:05.563 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop(client.4665.0:184026 3.b e171/117 3:d3052347:::benchmark_data_smithi177_31116_object43723:head v 171'6273, mlcod=171'6269)", 2024-09-13T11:26:05.563 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:25:32.645695+0000", 2024-09-13T11:26:05.563 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 32.824930588000001, 2024-09-13T11:26:05.564 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.34822244200000002, 2024-09-13T11:26:05.564 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:26:05.564 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:26:05.564 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:05.564 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:26:05.564 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:05.564 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:26:05.564 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:32.645695+0000", 2024-09-13T11:26:05.564 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:05.564 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:05.564 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:05.564 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:26:05.564 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:32.645695+0000", 2024-09-13T11:26:05.565 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:05.565 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:05.565 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:05.565 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:26:05.565 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:32.645696+0000", 2024-09-13T11:26:05.565 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.4780000000000001e-06 2024-09-13T11:26:05.565 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:05.565 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:05.565 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:26:05.565 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:32.645704+0000", 2024-09-13T11:26:05.565 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.1980000000000004e-06 2024-09-13T11:26:05.565 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:05.565 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:05.565 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:26:05.566 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:32.992228+0000", 2024-09-13T11:26:05.566 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.346523992 2024-09-13T11:26:05.566 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:05.566 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:05.566 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:26:05.566 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:32.992232+0000", 2024-09-13T11:26:05.566 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.3109999999999999e-06 2024-09-13T11:26:05.566 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:05.566 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:05.566 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:05.566 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:32.992275+0000", 2024-09-13T11:26:05.566 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.2589999999999997e-05 2024-09-13T11:26:05.566 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:05.566 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:05.566 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:26:05.567 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:32.992289+0000", 2024-09-13T11:26:05.567 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.4116e-05 2024-09-13T11:26:05.567 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:05.567 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:05.567 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:26:05.567 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:32.993877+0000", 2024-09-13T11:26:05.567 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0015879259999999999 2024-09-13T11:26:05.567 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:05.567 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:05.567 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:26:05.567 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:32.993917+0000", 2024-09-13T11:26:05.567 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.0831000000000003e-05 2024-09-13T11:26:05.567 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:05.568 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:05.568 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:05.568 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:05.568 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:05.568 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop(client.4665.0:184072 3.b e171/117 3:d319cf93:::benchmark_data_smithi177_31116_object43769:head v 171'6274, mlcod=171'6270)", 2024-09-13T11:26:05.568 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:25:32.701516+0000", 2024-09-13T11:26:05.568 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 32.769109899999997, 2024-09-13T11:26:05.568 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.30463955399999998, 2024-09-13T11:26:05.568 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:26:05.568 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:26:05.568 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:05.568 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:26:05.568 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:05.569 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:26:05.569 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:32.701516+0000", 2024-09-13T11:26:05.569 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:05.569 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:05.569 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:05.569 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:26:05.569 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:32.701516+0000", 2024-09-13T11:26:05.569 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:05.569 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:05.569 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:05.569 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:26:05.569 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:32.701519+0000", 2024-09-13T11:26:05.569 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.0029999999999999e-06 2024-09-13T11:26:05.569 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:05.570 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:05.570 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:26:05.570 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:32.701530+0000", 2024-09-13T11:26:05.570 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.1759999999999999e-05 2024-09-13T11:26:05.570 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:05.570 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:05.570 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:26:05.570 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:33.003551+0000", 2024-09-13T11:26:05.570 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.302020819 2024-09-13T11:26:05.570 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:05.570 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:05.570 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:26:05.570 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:33.003558+0000", 2024-09-13T11:26:05.571 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.2299999999999996e-06 2024-09-13T11:26:05.571 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:05.571 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:05.571 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:05.571 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:33.003630+0000", 2024-09-13T11:26:05.571 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.2125000000000005e-05 2024-09-13T11:26:05.571 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:05.571 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:05.571 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:26:05.571 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:33.003648+0000", 2024-09-13T11:26:05.571 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.8672e-05 2024-09-13T11:26:05.571 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:05.571 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:05.571 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:26:05.572 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:33.006114+0000", 2024-09-13T11:26:05.572 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0024656790000000001 2024-09-13T11:26:05.572 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:05.572 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:05.572 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:26:05.572 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:33.006155+0000", 2024-09-13T11:26:05.572 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.1266000000000001e-05 2024-09-13T11:26:05.572 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:05.572 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:05.572 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:05.572 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:05.572 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:05.572 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-09-13T11:26:05.574 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_ops_in_flight 2024-09-13T11:26:05.574 INFO:tasks.ceph.osd.15.smithi177.stderr:2024-09-13T11:26:05.569+0000 7f9d6ce77640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26578) UID: 0 2024-09-13T11:26:05.674 INFO:tasks.ceph.osd.2.smithi162.stderr:2024-09-13T11:26:05.671+0000 7fe92b35a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26313) UID: 0 2024-09-13T11:26:05.721 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-13T11:26:05.721 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [], 2024-09-13T11:26:05.721 INFO:teuthology.orchestra.run.smithi162.stdout: "num_ops": 0 2024-09-13T11:26:05.721 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-13T11:26:05.737 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_blocked_ops 2024-09-13T11:26:05.775 INFO:tasks.ceph.osd.7.smithi177.stderr:2024-09-13T11:26:05.773+0000 7f897b51d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26575) UID: 0 2024-09-13T11:26:05.876 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-09-13T11:26:05.871+0000 7f97befc9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26315) UID: 0 2024-09-13T11:26:05.912 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-13T11:26:05.913 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [], 2024-09-13T11:26:05.913 INFO:teuthology.orchestra.run.smithi162.stdout: "complaint_time": 30, 2024-09-13T11:26:05.913 INFO:teuthology.orchestra.run.smithi162.stdout: "num_blocked_ops": 0 2024-09-13T11:26:05.913 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-13T11:26:05.928 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_historic_ops 2024-09-13T11:26:05.977 INFO:tasks.ceph.osd.4.smithi001.stderr:2024-09-13T11:26:05.974+0000 7ff8e0218640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26310) UID: 0 2024-09-13T11:26:06.066 INFO:tasks.ceph.osd.15.smithi177.stderr:2024-09-13T11:26:06.061+0000 7f9d618a4640 -1 osd.15 192 handle_osd_map injecting map crc failure 2024-09-13T11:26:06.078 INFO:tasks.ceph.osd.15.smithi177.stderr:2024-09-13T11:26:06.073+0000 7f9d6ce77640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26578) UID: 0 2024-09-13T11:26:06.132 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-13T11:26:06.133 INFO:teuthology.orchestra.run.smithi162.stdout: "size": 20, 2024-09-13T11:26:06.133 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 600, 2024-09-13T11:26:06.133 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [ 2024-09-13T11:26:06.133 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:06.133 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:221526 3.2 3.ea98b382 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e191)", 2024-09-13T11:26:06.133 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:26:04.295546+0000", 2024-09-13T11:26:06.133 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 1.8345276149999998, 2024-09-13T11:26:06.133 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.73989113200000001, 2024-09-13T11:26:06.133 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:26:06.133 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:26:06.133 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-09-13T11:26:06.133 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:26:06.134 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:26:06.134 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:06.134 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 221526 2024-09-13T11:26:06.134 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:06.134 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:26:06.134 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:06.134 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:26:06.134 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:04.295546+0000", 2024-09-13T11:26:06.134 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:06.134 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:06.134 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:06.134 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:26:06.135 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:04.295546+0000", 2024-09-13T11:26:06.135 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:06.135 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:06.135 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:06.135 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:26:06.135 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:04.295551+0000", 2024-09-13T11:26:06.135 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.6940000000000003e-06 2024-09-13T11:26:06.135 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:06.135 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:06.135 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:26:06.135 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:04.295567+0000", 2024-09-13T11:26:06.135 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5896000000000002e-05 2024-09-13T11:26:06.135 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:06.136 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:06.136 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:26:06.136 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:04.295599+0000", 2024-09-13T11:26:06.136 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.243e-05 2024-09-13T11:26:06.136 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:06.136 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:06.136 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:26:06.136 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:04.295622+0000", 2024-09-13T11:26:06.136 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.3149000000000001e-05 2024-09-13T11:26:06.136 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:06.136 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:06.136 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:26:06.136 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:04.295764+0000", 2024-09-13T11:26:06.136 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00014193300000000001 2024-09-13T11:26:06.137 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:06.137 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:06.137 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-09-13T11:26:06.137 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:04.295778+0000", 2024-09-13T11:26:06.137 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3763e-05 2024-09-13T11:26:06.137 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:06.137 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:06.137 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:26:06.137 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:05.035420+0000", 2024-09-13T11:26:06.137 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.73964228300000001 2024-09-13T11:26:06.137 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:06.137 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:06.137 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:26:06.138 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:05.035437+0000", 2024-09-13T11:26:06.138 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.6983999999999999e-05 2024-09-13T11:26:06.138 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:06.138 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:26:06.138 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:06.138 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:06.138 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:06.138 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:221528 3.15 3.b7a45bf5 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e191)", 2024-09-13T11:26:06.138 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:26:04.296174+0000", 2024-09-13T11:26:06.138 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 1.83389997, 2024-09-13T11:26:06.138 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.73909410399999997, 2024-09-13T11:26:06.138 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:26:06.138 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:26:06.139 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-09-13T11:26:06.139 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:26:06.139 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:26:06.139 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:06.139 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 221528 2024-09-13T11:26:06.139 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:06.139 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:26:06.139 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:06.139 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:26:06.139 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:04.296174+0000", 2024-09-13T11:26:06.139 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:06.139 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:06.139 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:06.140 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:26:06.140 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:04.296174+0000", 2024-09-13T11:26:06.140 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:06.140 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:06.140 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:06.140 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:26:06.140 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:04.296188+0000", 2024-09-13T11:26:06.140 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4037e-05 2024-09-13T11:26:06.140 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:06.140 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:06.140 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:26:06.140 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:04.296201+0000", 2024-09-13T11:26:06.140 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3096e-05 2024-09-13T11:26:06.140 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:06.141 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:06.141 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:26:06.141 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:04.296242+0000", 2024-09-13T11:26:06.141 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.1103000000000003e-05 2024-09-13T11:26:06.141 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:06.141 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:06.141 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:26:06.141 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:04.296264+0000", 2024-09-13T11:26:06.141 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.173e-05 2024-09-13T11:26:06.141 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:06.141 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:06.141 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:26:06.141 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:04.296410+0000", 2024-09-13T11:26:06.141 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00014574600000000001 2024-09-13T11:26:06.142 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:06.142 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:06.142 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-09-13T11:26:06.142 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:04.296424+0000", 2024-09-13T11:26:06.142 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3926e-05 2024-09-13T11:26:06.142 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:06.142 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:06.142 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:26:06.142 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:05.035221+0000", 2024-09-13T11:26:06.142 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.73879784999999998 2024-09-13T11:26:06.142 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:06.142 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:06.142 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:26:06.143 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:05.035268+0000", 2024-09-13T11:26:06.143 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.6616000000000002e-05 2024-09-13T11:26:06.143 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:06.143 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:26:06.143 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:06.143 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:06.143 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:06.143 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:221544 3.12 3.ec3fbdf2 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e191)", 2024-09-13T11:26:06.143 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:26:04.384452+0000", 2024-09-13T11:26:06.143 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 1.7456213780000001, 2024-09-13T11:26:06.143 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.65065541999999998, 2024-09-13T11:26:06.143 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:26:06.143 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:26:06.143 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-09-13T11:26:06.144 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:26:06.144 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:26:06.144 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:06.144 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 221544 2024-09-13T11:26:06.144 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:06.144 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:26:06.144 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:06.144 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:26:06.144 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:04.384452+0000", 2024-09-13T11:26:06.144 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:06.144 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:06.144 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:06.144 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:26:06.145 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:04.384452+0000", 2024-09-13T11:26:06.145 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:06.145 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:06.145 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:06.145 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:26:06.145 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:04.384457+0000", 2024-09-13T11:26:06.145 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.6469999999999997e-06 2024-09-13T11:26:06.145 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:06.145 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:06.145 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:26:06.145 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:04.384470+0000", 2024-09-13T11:26:06.145 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3186999999999999e-05 2024-09-13T11:26:06.145 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:06.145 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:06.146 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:26:06.146 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:04.384504+0000", 2024-09-13T11:26:06.146 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.3413e-05 2024-09-13T11:26:06.146 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:06.146 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:06.146 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:26:06.146 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:04.384526+0000", 2024-09-13T11:26:06.146 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.2690000000000001e-05 2024-09-13T11:26:06.146 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:06.146 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:06.146 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:26:06.146 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:04.384653+0000", 2024-09-13T11:26:06.146 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00012697199999999999 2024-09-13T11:26:06.146 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:06.146 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:06.147 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-09-13T11:26:06.147 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:04.384668+0000", 2024-09-13T11:26:06.147 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4936000000000001e-05 2024-09-13T11:26:06.147 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:06.147 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:06.147 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:26:06.147 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:05.035087+0000", 2024-09-13T11:26:06.147 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.65041902900000004 2024-09-13T11:26:06.147 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:06.147 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:06.147 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:26:06.147 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:05.035108+0000", 2024-09-13T11:26:06.147 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.0545999999999999e-05 2024-09-13T11:26:06.148 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:06.148 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:26:06.148 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:06.148 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:06.148 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:06.148 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPush(3.24 192/191 [PushOp(3:25e4c2a1:::benchmark_data_smithi177_31116_object84891:head, version: 108'4000, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:25e4c2a1:::benchmark_data_smithi177_31116_object84891:head@108'4000, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:26:06.148 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:26:05.520406+0000", 2024-09-13T11:26:06.148 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 0.60966794999999996, 2024-09-13T11:26:06.148 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.019666152999999999, 2024-09-13T11:26:06.148 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:26:06.148 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:26:06.148 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:26:06.148 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:26:06.148 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:06.149 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:26:06.149 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:05.520406+0000", 2024-09-13T11:26:06.149 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:06.149 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:06.149 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:06.149 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:26:06.149 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:05.520406+0000", 2024-09-13T11:26:06.149 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:06.149 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:06.149 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:06.149 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:26:06.149 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:05.520410+0000", 2024-09-13T11:26:06.149 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.7000000000000002e-06 2024-09-13T11:26:06.149 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:06.150 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:06.150 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:26:06.150 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:05.520532+0000", 2024-09-13T11:26:06.150 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00012194200000000001 2024-09-13T11:26:06.150 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:06.150 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:06.150 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:26:06.150 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:05.520633+0000", 2024-09-13T11:26:06.150 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000101603 2024-09-13T11:26:06.150 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:06.150 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:06.150 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:26:06.150 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:05.520687+0000", 2024-09-13T11:26:06.150 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.3510000000000001e-05 2024-09-13T11:26:06.151 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:06.151 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:06.151 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:26:06.151 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:05.520967+0000", 2024-09-13T11:26:06.151 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000280294 2024-09-13T11:26:06.151 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:06.151 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:06.151 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:26:06.151 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:05.521006+0000", 2024-09-13T11:26:06.151 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.8962999999999999e-05 2024-09-13T11:26:06.151 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:06.151 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:06.151 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:26:06.152 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:05.540072+0000", 2024-09-13T11:26:06.152 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.019066141000000002 2024-09-13T11:26:06.152 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:06.152 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:26:06.152 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:06.152 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:06.152 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:06.152 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPush(3.6 192/191 [PushOp(3:618796b9:::benchmark_data_smithi177_31116_object82360:head, version: 104'4057, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:618796b9:::benchmark_data_smithi177_31116_object82360:head@104'4057, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:26:06.152 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:26:05.525419+0000", 2024-09-13T11:26:06.152 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 0.60465487399999995, 2024-09-13T11:26:06.152 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.017197657000000002, 2024-09-13T11:26:06.152 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:26:06.152 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:26:06.152 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:26:06.153 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:26:06.153 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:06.153 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:26:06.153 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:05.525419+0000", 2024-09-13T11:26:06.153 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:06.153 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:06.153 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:06.153 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:26:06.153 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:05.525419+0000", 2024-09-13T11:26:06.153 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:06.153 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:06.153 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:06.153 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:26:06.153 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:05.525422+0000", 2024-09-13T11:26:06.154 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.6460000000000002e-06 2024-09-13T11:26:06.154 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:06.154 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:06.154 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:26:06.154 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:05.525509+0000", 2024-09-13T11:26:06.154 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.6889000000000001e-05 2024-09-13T11:26:06.154 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:06.154 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:06.154 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:26:06.154 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:05.539851+0000", 2024-09-13T11:26:06.154 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.014342720999999999 2024-09-13T11:26:06.154 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:06.154 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:06.154 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:26:06.155 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:05.539955+0000", 2024-09-13T11:26:06.155 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00010391900000000001 2024-09-13T11:26:06.155 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:06.155 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:06.155 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:26:06.155 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:05.540176+0000", 2024-09-13T11:26:06.155 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000220662 2024-09-13T11:26:06.155 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:06.155 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:06.155 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:26:06.155 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:05.540197+0000", 2024-09-13T11:26:06.155 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.1246e-05 2024-09-13T11:26:06.155 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:06.155 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:06.155 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:26:06.156 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:05.542617+0000", 2024-09-13T11:26:06.156 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0024195739999999999 2024-09-13T11:26:06.156 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:06.156 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:26:06.156 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:06.156 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:06.156 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:06.156 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPush(3.24 192/191 [PushOp(3:26189364:::benchmark_data_smithi177_31116_object84929:head, version: 108'4001, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:26189364:::benchmark_data_smithi177_31116_object84929:head@108'4001, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:26:06.156 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:26:05.525822+0000", 2024-09-13T11:26:06.156 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 0.60425234100000003, 2024-09-13T11:26:06.156 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.017465531999999999, 2024-09-13T11:26:06.156 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:26:06.156 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:26:06.156 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:26:06.157 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:26:06.157 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:06.157 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:26:06.157 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:05.525822+0000", 2024-09-13T11:26:06.157 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:06.157 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:06.157 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:06.157 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:26:06.157 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:05.525822+0000", 2024-09-13T11:26:06.157 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:06.157 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:06.157 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:06.157 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:26:06.157 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:05.525825+0000", 2024-09-13T11:26:06.158 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.1070000000000001e-06 2024-09-13T11:26:06.158 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:06.158 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:06.158 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:26:06.158 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:05.525890+0000", 2024-09-13T11:26:06.158 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.5326000000000004e-05 2024-09-13T11:26:06.158 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:06.158 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:06.158 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:26:06.158 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:05.526056+0000", 2024-09-13T11:26:06.158 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000166019 2024-09-13T11:26:06.158 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:06.158 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:06.158 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:26:06.159 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:05.526090+0000", 2024-09-13T11:26:06.159 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.3532999999999999e-05 2024-09-13T11:26:06.159 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:06.159 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:06.159 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:26:06.159 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:05.540229+0000", 2024-09-13T11:26:06.159 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.014139598 2024-09-13T11:26:06.159 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:06.159 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:06.159 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:26:06.159 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:05.540266+0000", 2024-09-13T11:26:06.159 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.7370000000000003e-05 2024-09-13T11:26:06.159 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:06.159 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:06.160 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:26:06.160 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:05.543287+0000", 2024-09-13T11:26:06.160 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0030205789999999998 2024-09-13T11:26:06.160 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:06.160 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:26:06.160 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:06.160 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:06.160 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:06.160 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPush(3.22 192/191 [PushOp(3:4638ce03:::benchmark_data_smithi177_31116_object83781:head, version: 107'3547, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:4638ce03:::benchmark_data_smithi177_31116_object83781:head@107'3547, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:26:06.160 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:26:05.632674+0000", 2024-09-13T11:26:06.160 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 0.49739945499999999, 2024-09-13T11:26:06.160 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.013600945999999999, 2024-09-13T11:26:06.161 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:26:06.161 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:26:06.161 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:26:06.161 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:26:06.161 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:06.161 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:26:06.161 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:05.632674+0000", 2024-09-13T11:26:06.161 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:06.161 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:06.161 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:06.161 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:26:06.161 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:05.632674+0000", 2024-09-13T11:26:06.161 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:06.161 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:06.162 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:06.162 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:26:06.162 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:05.632678+0000", 2024-09-13T11:26:06.162 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.0130000000000004e-06 2024-09-13T11:26:06.162 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:06.162 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:06.162 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:26:06.162 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:05.632745+0000", 2024-09-13T11:26:06.162 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.6456000000000002e-05 2024-09-13T11:26:06.162 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:06.162 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:06.162 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:26:06.162 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:05.644910+0000", 2024-09-13T11:26:06.162 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.012164975 2024-09-13T11:26:06.163 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:06.163 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:06.163 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:26:06.163 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:05.644939+0000", 2024-09-13T11:26:06.163 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.9306e-05 2024-09-13T11:26:06.163 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:06.163 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:06.163 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:26:06.163 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:05.645060+0000", 2024-09-13T11:26:06.163 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000120481 2024-09-13T11:26:06.163 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:06.163 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:06.164 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:26:06.164 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:05.645070+0000", 2024-09-13T11:26:06.164 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0049e-05 2024-09-13T11:26:06.164 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:06.164 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:06.164 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:26:06.164 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:05.646275+0000", 2024-09-13T11:26:06.164 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001205666 2024-09-13T11:26:06.164 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:06.164 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:26:06.164 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:06.164 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:06.164 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:06.164 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPush(3.22 192/191 [PushOp(3:4443e8a5:::benchmark_data_smithi177_31116_object84025:head, version: 107'3549, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:4443e8a5:::benchmark_data_smithi177_31116_object84025:head@107'3549, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:26:06.165 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:26:05.655024+0000", 2024-09-13T11:26:06.165 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 0.47504994900000003, 2024-09-13T11:26:06.165 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0126312, 2024-09-13T11:26:06.165 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:26:06.165 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:26:06.165 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:26:06.165 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:26:06.165 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:06.165 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:26:06.165 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:05.655024+0000", 2024-09-13T11:26:06.165 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:06.165 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:06.166 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:06.166 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:26:06.166 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:05.655024+0000", 2024-09-13T11:26:06.166 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:06.166 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:06.166 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:06.166 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:26:06.166 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:05.655026+0000", 2024-09-13T11:26:06.166 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.632e-06 2024-09-13T11:26:06.166 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:06.166 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:06.166 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:26:06.166 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:05.655090+0000", 2024-09-13T11:26:06.166 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.4746000000000007e-05 2024-09-13T11:26:06.167 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:06.167 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:06.167 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:26:06.167 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:05.666366+0000", 2024-09-13T11:26:06.167 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.011275376 2024-09-13T11:26:06.167 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:06.167 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:06.167 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:26:06.167 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:05.666385+0000", 2024-09-13T11:26:06.167 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.8947e-05 2024-09-13T11:26:06.167 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:06.167 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:06.167 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:26:06.167 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:05.666493+0000", 2024-09-13T11:26:06.168 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00010794999999999999 2024-09-13T11:26:06.168 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:06.168 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:06.168 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:26:06.168 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:05.666504+0000", 2024-09-13T11:26:06.168 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1291000000000001e-05 2024-09-13T11:26:06.168 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:06.168 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:06.168 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:26:06.168 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:05.667655+0000", 2024-09-13T11:26:06.168 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0011512580000000001 2024-09-13T11:26:06.168 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:06.168 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:26:06.169 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:06.169 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:06.169 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:06.169 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPush(3.22 192/191 [PushOp(3:46e9d7fc:::benchmark_data_smithi177_31116_object84247:head, version: 107'3553, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:46e9d7fc:::benchmark_data_smithi177_31116_object84247:head@107'3553, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:26:06.169 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:26:05.696499+0000", 2024-09-13T11:26:06.169 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 0.43357493200000002, 2024-09-13T11:26:06.169 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.013217616999999999, 2024-09-13T11:26:06.169 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:26:06.169 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:26:06.169 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:26:06.169 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:26:06.169 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:06.169 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:26:06.170 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:05.696499+0000", 2024-09-13T11:26:06.170 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:06.170 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:06.170 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:06.170 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:26:06.170 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:05.696499+0000", 2024-09-13T11:26:06.170 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:06.170 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:06.170 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:06.170 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:26:06.170 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:05.696501+0000", 2024-09-13T11:26:06.170 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.745e-06 2024-09-13T11:26:06.171 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:06.171 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:06.171 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:26:06.171 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:05.696537+0000", 2024-09-13T11:26:06.171 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.5858999999999997e-05 2024-09-13T11:26:06.171 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:06.171 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:06.171 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:26:06.171 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:05.707501+0000", 2024-09-13T11:26:06.171 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.010964366999999999 2024-09-13T11:26:06.171 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:06.171 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:06.171 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:26:06.171 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:05.707515+0000", 2024-09-13T11:26:06.172 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3951000000000001e-05 2024-09-13T11:26:06.172 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:06.172 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:06.172 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:26:06.172 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:05.707703+0000", 2024-09-13T11:26:06.172 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00018799800000000001 2024-09-13T11:26:06.172 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:06.172 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:06.172 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:26:06.172 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:05.707725+0000", 2024-09-13T11:26:06.172 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.2463000000000002e-05 2024-09-13T11:26:06.172 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:06.172 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:06.173 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:26:06.173 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:05.709717+0000", 2024-09-13T11:26:06.173 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0019912340000000001 2024-09-13T11:26:06.173 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:06.173 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:26:06.173 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:06.173 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:06.173 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:06.173 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPush(3.22 192/191 [PushOp(3:44111ea8:::benchmark_data_smithi177_31116_object84335:head, version: 107'3556, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:44111ea8:::benchmark_data_smithi177_31116_object84335:head@107'3556, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:26:06.173 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:26:05.728293+0000", 2024-09-13T11:26:06.173 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 0.40178053200000002, 2024-09-13T11:26:06.173 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.01397236, 2024-09-13T11:26:06.173 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:26:06.173 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:26:06.173 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:26:06.174 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:26:06.174 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:06.174 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:26:06.174 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:05.728293+0000", 2024-09-13T11:26:06.174 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:06.174 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:06.174 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:06.174 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:26:06.174 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:05.728293+0000", 2024-09-13T11:26:06.174 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:06.174 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:06.174 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:06.174 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:26:06.174 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:05.728297+0000", 2024-09-13T11:26:06.175 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.4800000000000001e-06 2024-09-13T11:26:06.175 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:06.175 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:06.175 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:26:06.175 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:05.728364+0000", 2024-09-13T11:26:06.175 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.7015999999999997e-05 2024-09-13T11:26:06.175 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:06.175 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:06.175 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:26:06.175 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:05.739813+0000", 2024-09-13T11:26:06.175 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.011449241000000001 2024-09-13T11:26:06.175 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:06.175 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:06.175 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:26:06.176 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:05.739861+0000", 2024-09-13T11:26:06.176 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.8299e-05 2024-09-13T11:26:06.176 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:06.176 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:06.176 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:26:06.176 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:05.740052+0000", 2024-09-13T11:26:06.176 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000190929 2024-09-13T11:26:06.176 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:06.176 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:06.176 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:26:06.176 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:05.740073+0000", 2024-09-13T11:26:06.176 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.067e-05 2024-09-13T11:26:06.176 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:06.176 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:06.177 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:26:06.177 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:05.742266+0000", 2024-09-13T11:26:06.177 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.002192725 2024-09-13T11:26:06.177 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:06.177 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:26:06.177 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:06.177 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:06.177 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:06.177 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPush(3.22 192/191 [PushOp(3:44885d03:::benchmark_data_smithi177_31116_object84342:head, version: 107'3557, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:44885d03:::benchmark_data_smithi177_31116_object84342:head@107'3557, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:26:06.177 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:26:05.739690+0000", 2024-09-13T11:26:06.177 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 0.39038380099999997, 2024-09-13T11:26:06.177 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.013351201, 2024-09-13T11:26:06.177 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:26:06.177 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:26:06.178 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:26:06.178 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:26:06.178 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:06.178 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:26:06.178 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:05.739690+0000", 2024-09-13T11:26:06.178 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:06.178 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:06.178 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:06.178 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:26:06.178 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:05.739690+0000", 2024-09-13T11:26:06.178 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:06.178 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:06.178 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:06.178 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:26:06.179 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:05.739694+0000", 2024-09-13T11:26:06.179 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.4800000000000001e-06 2024-09-13T11:26:06.179 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:06.179 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:06.179 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:26:06.179 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:05.739759+0000", 2024-09-13T11:26:06.179 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.5256e-05 2024-09-13T11:26:06.179 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:06.179 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:06.179 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:26:06.179 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:05.750834+0000", 2024-09-13T11:26:06.179 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.011075476000000001 2024-09-13T11:26:06.179 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:06.179 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:06.179 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:26:06.180 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:05.750866+0000", 2024-09-13T11:26:06.180 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.1927000000000001e-05 2024-09-13T11:26:06.180 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:06.180 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:06.180 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:26:06.180 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:05.751042+0000", 2024-09-13T11:26:06.180 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00017555199999999999 2024-09-13T11:26:06.180 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:06.180 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:06.180 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:26:06.180 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:05.751064+0000", 2024-09-13T11:26:06.180 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.2670000000000001e-05 2024-09-13T11:26:06.180 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:06.180 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:06.181 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:26:06.181 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:05.753041+0000", 2024-09-13T11:26:06.181 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0019768400000000001 2024-09-13T11:26:06.181 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:06.181 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:26:06.181 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:06.181 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:06.181 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:06.181 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPush(3.22 192/191 [PushOp(3:452991f0:::benchmark_data_smithi177_31116_object84739:head, version: 107'3567, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:452991f0:::benchmark_data_smithi177_31116_object84739:head@107'3567, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:26:06.181 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:26:05.841979+0000", 2024-09-13T11:26:06.181 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 0.28809439799999997, 2024-09-13T11:26:06.181 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.014940792, 2024-09-13T11:26:06.181 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:26:06.182 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:26:06.182 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:26:06.182 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:26:06.182 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:06.182 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:26:06.182 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:05.841979+0000", 2024-09-13T11:26:06.182 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:06.182 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:06.182 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:06.182 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:26:06.182 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:05.841979+0000", 2024-09-13T11:26:06.182 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:06.182 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:06.182 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:06.183 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:26:06.183 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:05.841981+0000", 2024-09-13T11:26:06.183 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4389999999999999e-06 2024-09-13T11:26:06.183 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:06.183 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:06.183 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:26:06.183 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:05.842016+0000", 2024-09-13T11:26:06.183 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.523e-05 2024-09-13T11:26:06.183 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:06.183 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:06.183 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:26:06.183 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:05.854617+0000", 2024-09-13T11:26:06.183 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.012600963999999999 2024-09-13T11:26:06.183 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:06.184 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:06.184 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:26:06.184 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:05.854668+0000", 2024-09-13T11:26:06.184 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.0680000000000003e-05 2024-09-13T11:26:06.184 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:06.184 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:06.184 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:26:06.184 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:05.854817+0000", 2024-09-13T11:26:06.184 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000149677 2024-09-13T11:26:06.184 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:06.184 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:06.184 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:26:06.184 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:05.854836+0000", 2024-09-13T11:26:06.185 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.8717000000000001e-05 2024-09-13T11:26:06.185 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:06.185 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:06.185 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:26:06.185 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:05.856920+0000", 2024-09-13T11:26:06.185 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0020840849999999998 2024-09-13T11:26:06.185 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:06.185 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:26:06.185 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:06.185 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:06.185 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:06.185 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPush(3.22 192/191 [PushOp(3:44fa24ae:::benchmark_data_smithi177_31116_object84742:head, version: 107'3568, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:44fa24ae:::benchmark_data_smithi177_31116_object84742:head@107'3568, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:26:06.185 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:26:05.854487+0000", 2024-09-13T11:26:06.185 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 0.27558721400000002, 2024-09-13T11:26:06.186 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.013220066000000001, 2024-09-13T11:26:06.186 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:26:06.186 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:26:06.186 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:26:06.186 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:26:06.186 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:06.186 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:26:06.186 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:05.854487+0000", 2024-09-13T11:26:06.186 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:06.186 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:06.186 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:06.186 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:26:06.186 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:05.854487+0000", 2024-09-13T11:26:06.186 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:06.187 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:06.187 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:06.187 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:26:06.187 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:05.854490+0000", 2024-09-13T11:26:06.187 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.3330000000000001e-06 2024-09-13T11:26:06.187 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:06.187 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:06.187 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:26:06.187 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:05.854557+0000", 2024-09-13T11:26:06.187 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.6649999999999994e-05 2024-09-13T11:26:06.187 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:06.187 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:06.187 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:26:06.187 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:05.865989+0000", 2024-09-13T11:26:06.188 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.011431894 2024-09-13T11:26:06.188 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:06.188 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:06.188 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:26:06.188 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:05.866004+0000", 2024-09-13T11:26:06.188 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5342999999999999e-05 2024-09-13T11:26:06.188 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:06.188 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:06.188 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:26:06.188 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:05.866113+0000", 2024-09-13T11:26:06.188 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00010943299999999999 2024-09-13T11:26:06.188 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:06.189 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:06.189 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:26:06.189 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:05.866128+0000", 2024-09-13T11:26:06.189 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4814000000000001e-05 2024-09-13T11:26:06.189 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:06.189 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:06.189 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:26:06.189 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:05.867707+0000", 2024-09-13T11:26:06.189 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001578599 2024-09-13T11:26:06.189 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:06.189 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:26:06.189 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:06.189 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:06.190 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:06.190 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPush(3.22 192/191 [PushOp(3:4643c833:::benchmark_data_smithi177_31116_object84792:head, version: 107'3570, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:4643c833:::benchmark_data_smithi177_31116_object84792:head@107'3570, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:26:06.190 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:26:05.876618+0000", 2024-09-13T11:26:06.190 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 0.25345625399999999, 2024-09-13T11:26:06.190 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.012719509, 2024-09-13T11:26:06.190 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:26:06.190 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:26:06.190 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:26:06.190 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:26:06.190 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:06.190 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:26:06.190 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:05.876618+0000", 2024-09-13T11:26:06.190 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:06.191 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:06.191 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:06.191 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:26:06.191 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:05.876618+0000", 2024-09-13T11:26:06.191 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:06.191 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:06.191 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:06.191 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:26:06.191 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:05.876621+0000", 2024-09-13T11:26:06.191 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.5540000000000002e-06 2024-09-13T11:26:06.191 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:06.191 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:06.191 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:26:06.191 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:05.876684+0000", 2024-09-13T11:26:06.191 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.2855999999999996e-05 2024-09-13T11:26:06.192 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:06.192 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:06.192 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:26:06.192 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:05.887537+0000", 2024-09-13T11:26:06.192 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.010853058 2024-09-13T11:26:06.192 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:06.192 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:06.192 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:26:06.192 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:05.887567+0000", 2024-09-13T11:26:06.192 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.9632999999999999e-05 2024-09-13T11:26:06.192 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:06.192 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:06.192 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:26:06.193 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:05.887689+0000", 2024-09-13T11:26:06.193 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00012274000000000001 2024-09-13T11:26:06.193 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:06.193 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:06.193 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:26:06.193 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:05.887704+0000", 2024-09-13T11:26:06.193 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4843e-05 2024-09-13T11:26:06.193 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:06.193 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:06.193 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:26:06.193 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:05.889337+0000", 2024-09-13T11:26:06.193 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0016328250000000001 2024-09-13T11:26:06.193 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:06.193 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:26:06.194 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:06.194 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:06.194 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:06.194 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPush(3.22 192/191 [PushOp(3:4682cb8f:::benchmark_data_smithi177_31116_object84815:head, version: 107'3571, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:4682cb8f:::benchmark_data_smithi177_31116_object84815:head@107'3571, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:26:06.194 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:26:05.887419+0000", 2024-09-13T11:26:06.194 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 0.242654965, 2024-09-13T11:26:06.194 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.012715189, 2024-09-13T11:26:06.194 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:26:06.194 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:26:06.194 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:26:06.194 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:26:06.194 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:06.194 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:26:06.195 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:05.887419+0000", 2024-09-13T11:26:06.195 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:06.195 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:06.195 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:06.195 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:26:06.195 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:05.887419+0000", 2024-09-13T11:26:06.195 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:06.195 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:06.195 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:06.195 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:26:06.195 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:05.887422+0000", 2024-09-13T11:26:06.195 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.1030000000000002e-06 2024-09-13T11:26:06.195 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:06.195 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:06.196 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:26:06.196 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:05.887483+0000", 2024-09-13T11:26:06.196 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.0977000000000002e-05 2024-09-13T11:26:06.196 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:06.196 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:06.196 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:26:06.196 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:05.898466+0000", 2024-09-13T11:26:06.196 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.010982594 2024-09-13T11:26:06.196 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:06.196 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:06.196 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:26:06.196 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:05.898495+0000", 2024-09-13T11:26:06.196 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.9349e-05 2024-09-13T11:26:06.197 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:06.197 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:06.197 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:26:06.197 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:05.898621+0000", 2024-09-13T11:26:06.197 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000126119 2024-09-13T11:26:06.197 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:06.197 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:06.197 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:26:06.197 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:05.898636+0000", 2024-09-13T11:26:06.197 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5182e-05 2024-09-13T11:26:06.197 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:06.197 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:06.197 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:26:06.197 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:05.900134+0000", 2024-09-13T11:26:06.198 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0014978649999999999 2024-09-13T11:26:06.198 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:06.198 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:26:06.198 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:06.198 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:06.198 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:06.198 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPush(3.22 192/191 [PushOp(3:44cf004e:::benchmark_data_smithi177_31116_object84845:head, version: 108'3573, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:44cf004e:::benchmark_data_smithi177_31116_object84845:head@108'3573, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:26:06.198 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:26:05.909026+0000", 2024-09-13T11:26:06.198 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 0.22104805699999999, 2024-09-13T11:26:06.198 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.012921758, 2024-09-13T11:26:06.198 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:26:06.198 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:26:06.198 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:26:06.198 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:26:06.199 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:06.199 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:26:06.199 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:05.909026+0000", 2024-09-13T11:26:06.199 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:06.199 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:06.199 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:06.199 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:26:06.199 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:05.909026+0000", 2024-09-13T11:26:06.199 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:06.199 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:06.199 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:06.199 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:26:06.199 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:05.909029+0000", 2024-09-13T11:26:06.199 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.7800000000000001e-06 2024-09-13T11:26:06.199 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:06.200 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:06.200 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:26:06.200 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:05.909095+0000", 2024-09-13T11:26:06.200 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.5965999999999998e-05 2024-09-13T11:26:06.200 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:06.200 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:06.200 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:26:06.200 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:05.920118+0000", 2024-09-13T11:26:06.200 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.011023794 2024-09-13T11:26:06.201 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:06.201 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:06.201 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:26:06.201 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:05.920153+0000", 2024-09-13T11:26:06.201 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.4471999999999998e-05 2024-09-13T11:26:06.201 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:06.201 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:06.201 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:26:06.201 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:05.920302+0000", 2024-09-13T11:26:06.201 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000148994 2024-09-13T11:26:06.201 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:06.201 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:06.201 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:26:06.201 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:05.920320+0000", 2024-09-13T11:26:06.202 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.8148e-05 2024-09-13T11:26:06.202 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:06.202 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:06.202 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:26:06.202 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:05.921948+0000", 2024-09-13T11:26:06.202 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0016276039999999999 2024-09-13T11:26:06.202 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:06.202 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:26:06.202 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:06.202 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:06.202 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:06.202 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPush(3.22 192/191 [PushOp(3:45af1fd9:::benchmark_data_smithi177_31116_object85444:head, version: 109'3579, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:45af1fd9:::benchmark_data_smithi177_31116_object85444:head@109'3579, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:26:06.202 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:26:05.971606+0000", 2024-09-13T11:26:06.202 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 0.158467788, 2024-09-13T11:26:06.203 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.014628416, 2024-09-13T11:26:06.203 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:26:06.203 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:26:06.203 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:26:06.203 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:26:06.203 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:06.203 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:26:06.203 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:05.971606+0000", 2024-09-13T11:26:06.203 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:06.203 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:06.203 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:06.203 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:26:06.203 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:05.971606+0000", 2024-09-13T11:26:06.203 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:06.203 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:06.204 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:06.204 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:26:06.204 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:05.971608+0000", 2024-09-13T11:26:06.204 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.824e-06 2024-09-13T11:26:06.204 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:06.204 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:06.204 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:26:06.204 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:05.971645+0000", 2024-09-13T11:26:06.204 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.7258999999999997e-05 2024-09-13T11:26:06.204 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:06.204 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:06.204 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:26:06.204 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:05.984836+0000", 2024-09-13T11:26:06.205 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.013191033 2024-09-13T11:26:06.205 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:06.205 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:06.205 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:26:06.205 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:05.984854+0000", 2024-09-13T11:26:06.205 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.7739000000000001e-05 2024-09-13T11:26:06.205 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:06.205 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:06.205 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:26:06.205 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:05.984961+0000", 2024-09-13T11:26:06.205 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000107333 2024-09-13T11:26:06.205 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:06.205 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:06.205 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:26:06.206 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:05.984973+0000", 2024-09-13T11:26:06.206 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1539000000000001e-05 2024-09-13T11:26:06.206 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:06.206 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:06.206 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:26:06.206 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:05.986234+0000", 2024-09-13T11:26:06.206 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0012616890000000001 2024-09-13T11:26:06.206 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:06.206 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:26:06.206 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:06.206 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:06.206 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:06.206 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPush(3.22 192/191 [PushOp(3:446bd418:::benchmark_data_smithi177_31116_object85719:head, version: 109'3581, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:446bd418:::benchmark_data_smithi177_31116_object85719:head@109'3581, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:26:06.206 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:26:05.995010+0000", 2024-09-13T11:26:06.207 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 0.13506395299999999, 2024-09-13T11:26:06.207 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.053065821999999999, 2024-09-13T11:26:06.207 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:26:06.207 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:26:06.207 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:26:06.207 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:26:06.207 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:06.207 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:26:06.207 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:05.995010+0000", 2024-09-13T11:26:06.207 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:06.207 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:06.207 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:06.207 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:26:06.207 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:05.995010+0000", 2024-09-13T11:26:06.208 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:06.208 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:06.208 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:06.208 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:26:06.208 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:05.995012+0000", 2024-09-13T11:26:06.208 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.204e-06 2024-09-13T11:26:06.208 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:06.208 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:06.208 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:26:06.208 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:05.995056+0000", 2024-09-13T11:26:06.208 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.4162999999999997e-05 2024-09-13T11:26:06.208 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:06.208 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:06.208 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:26:06.209 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:06.046734+0000", 2024-09-13T11:26:06.209 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.051677901999999998 2024-09-13T11:26:06.209 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:06.209 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:06.209 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:26:06.209 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:06.046756+0000", 2024-09-13T11:26:06.209 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.1926999999999999e-05 2024-09-13T11:26:06.209 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:06.209 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:06.209 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:26:06.209 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:06.046864+0000", 2024-09-13T11:26:06.209 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000107939 2024-09-13T11:26:06.209 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:06.209 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:06.210 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:26:06.210 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:06.046877+0000", 2024-09-13T11:26:06.210 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3004e-05 2024-09-13T11:26:06.210 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:06.210 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:06.210 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:26:06.210 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:06.048076+0000", 2024-09-13T11:26:06.210 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0011986830000000001 2024-09-13T11:26:06.210 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:06.210 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:26:06.210 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:06.210 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:06.210 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:06.210 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPush(3.22 192/191 [PushOp(3:45cfe954:::benchmark_data_smithi177_31116_object85795:head, version: 109'3582, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:45cfe954:::benchmark_data_smithi177_31116_object85795:head@109'3582, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:26:06.211 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:26:06.004768+0000", 2024-09-13T11:26:06.211 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 0.125306312, 2024-09-13T11:26:06.211 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.046332168999999999, 2024-09-13T11:26:06.211 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:26:06.211 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:26:06.211 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:26:06.211 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:26:06.211 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:06.211 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:26:06.211 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:06.004768+0000", 2024-09-13T11:26:06.211 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:06.211 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:06.211 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:06.211 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:26:06.212 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:06.004768+0000", 2024-09-13T11:26:06.212 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:06.212 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:06.212 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:06.212 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:26:06.212 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:06.004769+0000", 2024-09-13T11:26:06.212 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.742e-06 2024-09-13T11:26:06.212 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:06.212 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:06.212 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:26:06.212 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:06.004813+0000", 2024-09-13T11:26:06.212 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.3302e-05 2024-09-13T11:26:06.212 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:06.212 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:06.213 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:26:06.213 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:06.049415+0000", 2024-09-13T11:26:06.213 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.044602248999999997 2024-09-13T11:26:06.213 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:06.213 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:06.213 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:26:06.213 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:06.049431+0000", 2024-09-13T11:26:06.213 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.6467999999999999e-05 2024-09-13T11:26:06.213 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:06.213 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:06.213 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:26:06.213 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:06.049512+0000", 2024-09-13T11:26:06.213 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.0927999999999995e-05 2024-09-13T11:26:06.213 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:06.213 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:06.214 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:26:06.214 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:06.049524+0000", 2024-09-13T11:26:06.214 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1657e-05 2024-09-13T11:26:06.214 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:06.214 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:06.214 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:26:06.214 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:06.051100+0000", 2024-09-13T11:26:06.214 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0015758230000000001 2024-09-13T11:26:06.214 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:06.214 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:26:06.214 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:06.214 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:06.214 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:06.214 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPush(3.22 193/191 [PushOp(3:45a02edc:::benchmark_data_smithi177_31116_object85800:head, version: 109'3583, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:45a02edc:::benchmark_data_smithi177_31116_object85800:head@109'3583, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:26:06.215 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:26:06.046632+0000", 2024-09-13T11:26:06.215 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 0.083442014999999994, 2024-09-13T11:26:06.215 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.023750098000000001, 2024-09-13T11:26:06.215 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:26:06.215 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:26:06.215 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:26:06.215 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:26:06.215 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:06.215 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:26:06.215 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:06.046632+0000", 2024-09-13T11:26:06.215 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:06.215 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:06.215 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:06.216 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:26:06.216 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:06.046632+0000", 2024-09-13T11:26:06.216 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:06.216 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:06.216 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:06.216 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:26:06.216 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:06.046636+0000", 2024-09-13T11:26:06.216 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.1609999999999996e-06 2024-09-13T11:26:06.216 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:06.216 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:06.216 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:26:06.216 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:06.046690+0000", 2024-09-13T11:26:06.216 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.3950999999999999e-05 2024-09-13T11:26:06.216 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:06.217 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:06.217 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:26:06.217 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:06.067813+0000", 2024-09-13T11:26:06.217 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.021122835999999999 2024-09-13T11:26:06.217 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:06.217 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:06.217 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:26:06.217 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:06.067841+0000", 2024-09-13T11:26:06.217 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.7804e-05 2024-09-13T11:26:06.217 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:06.217 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:06.217 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:26:06.217 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:06.067978+0000", 2024-09-13T11:26:06.217 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000137751 2024-09-13T11:26:06.218 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:06.218 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:06.218 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:26:06.218 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:06.067997+0000", 2024-09-13T11:26:06.218 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.8352999999999999e-05 2024-09-13T11:26:06.218 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:06.218 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:06.218 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:26:06.218 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:06.070382+0000", 2024-09-13T11:26:06.218 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0023852420000000001 2024-09-13T11:26:06.218 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:06.218 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:26:06.218 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:06.218 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:06.218 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:26:06.219 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-13T11:26:06.220 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-09-13T11:26:06.187+0000 7f97b59fa640 -1 osd.6 192 handle_osd_map injecting map crc failure 2024-09-13T11:26:06.220 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-13T11:26:06.221 INFO:tasks.ceph.osd.12.smithi001.stderr:2024-09-13T11:26:06.218+0000 7f892c6e6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26314) UID: 0 2024-09-13T11:26:06.264 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T11:26:06.321 INFO:tasks.ceph.osd.3.smithi177.stderr:2024-09-13T11:26:06.317+0000 7f10adba4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26574) UID: 0 2024-09-13T11:26:06.405 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-13T11:26:06.405 INFO:teuthology.orchestra.run.smithi001.stdout: "ops": [ 2024-09-13T11:26:06.405 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:06.405 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:221758 3.e 3:718a4a57:::benchmark_data_smithi177_31116_object81455:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e193)", 2024-09-13T11:26:06.405 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:26:06.397975+0000", 2024-09-13T11:26:06.406 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 0.0069120789999999998, 2024-09-13T11:26:06.406 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0069339550000000003, 2024-09-13T11:26:06.406 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:26:06.406 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:26:06.406 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "waiting for sub ops", 2024-09-13T11:26:06.406 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:26:06.406 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:26:06.406 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:06.406 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 221758 2024-09-13T11:26:06.406 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:06.406 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:26:06.406 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:06.406 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:26:06.406 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:06.397975+0000", 2024-09-13T11:26:06.407 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:06.407 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:06.407 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:06.407 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:26:06.407 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:06.397975+0000", 2024-09-13T11:26:06.407 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:06.407 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:06.407 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:06.407 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:26:06.407 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:06.397976+0000", 2024-09-13T11:26:06.407 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.3990000000000001e-06 2024-09-13T11:26:06.407 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:06.407 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:06.407 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:26:06.408 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:06.397981+0000", 2024-09-13T11:26:06.408 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.5589999999999999e-06 2024-09-13T11:26:06.408 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:06.408 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:06.408 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:26:06.408 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:06.397991+0000", 2024-09-13T11:26:06.408 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.9210000000000004e-06 2024-09-13T11:26:06.408 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:06.408 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:06.408 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:26:06.408 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:06.397998+0000", 2024-09-13T11:26:06.408 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.6079999999999998e-06 2024-09-13T11:26:06.408 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:06.409 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:06.409 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:26:06.409 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:06.398059+0000", 2024-09-13T11:26:06.409 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.0758999999999999e-05 2024-09-13T11:26:06.409 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:06.409 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:06.409 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:26:06.409 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:06.398180+0000", 2024-09-13T11:26:06.409 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000120851 2024-09-13T11:26:06.409 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:06.409 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:06.409 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 1", 2024-09-13T11:26:06.409 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:06.398244+0000", 2024-09-13T11:26:06.409 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.3936000000000006e-05 2024-09-13T11:26:06.410 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:06.410 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:06.410 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:06.410 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:06.410 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:06.410 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:221762 3.e 3:774081cc:::benchmark_data_smithi177_31116_object81459:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e193)", 2024-09-13T11:26:06.410 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:26:06.400511+0000", 2024-09-13T11:26:06.410 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 0.0043754149999999997, 2024-09-13T11:26:06.410 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0044780389999999996, 2024-09-13T11:26:06.410 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:26:06.410 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:26:06.410 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "waiting for sub ops", 2024-09-13T11:26:06.410 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:26:06.410 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:26:06.411 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:06.411 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 221762 2024-09-13T11:26:06.411 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:06.411 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:26:06.411 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:06.411 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:26:06.411 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:06.400511+0000", 2024-09-13T11:26:06.411 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:06.411 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:06.411 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:06.411 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:26:06.411 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:06.400511+0000", 2024-09-13T11:26:06.411 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:06.411 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:06.411 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:06.412 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:26:06.412 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:06.400514+0000", 2024-09-13T11:26:06.412 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.5660000000000001e-06 2024-09-13T11:26:06.412 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:06.412 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:06.412 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:26:06.412 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:06.400523+0000", 2024-09-13T11:26:06.412 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.2939999999999994e-06 2024-09-13T11:26:06.412 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:06.412 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:06.412 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:26:06.412 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:06.400540+0000", 2024-09-13T11:26:06.412 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.6589999999999999e-05 2024-09-13T11:26:06.413 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:06.413 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:06.413 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:26:06.413 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:06.400548+0000", 2024-09-13T11:26:06.413 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.8180000000000004e-06 2024-09-13T11:26:06.413 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:06.413 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:06.413 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:26:06.413 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:06.402088+0000", 2024-09-13T11:26:06.413 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.001540748 2024-09-13T11:26:06.413 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:06.413 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:06.413 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:26:06.413 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:06.402244+0000", 2024-09-13T11:26:06.414 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000155651 2024-09-13T11:26:06.414 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:06.414 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:06.414 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 1", 2024-09-13T11:26:06.414 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:06.402355+0000", 2024-09-13T11:26:06.414 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000110376 2024-09-13T11:26:06.414 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:06.414 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:06.414 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:06.414 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:06.414 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:06.414 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop_reply(client.4665.0:221758 3.e e193/75)", 2024-09-13T11:26:06.414 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:26:06.402371+0000", 2024-09-13T11:26:06.414 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 0.0025162069999999999, 2024-09-13T11:26:06.415 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00267636, 2024-09-13T11:26:06.415 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:26:06.415 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:26:06.415 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "queued for pg", 2024-09-13T11:26:06.415 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:26:06.415 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:06.415 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:26:06.415 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:06.402371+0000", 2024-09-13T11:26:06.415 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:06.415 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:06.415 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:06.415 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:26:06.415 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:06.402371+0000", 2024-09-13T11:26:06.415 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:06.415 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:06.416 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:06.416 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:26:06.416 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:06.402372+0000", 2024-09-13T11:26:06.416 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.6169999999999999e-06 2024-09-13T11:26:06.416 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:06.416 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:06.416 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:26:06.416 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:06.402380+0000", 2024-09-13T11:26:06.416 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.8779999999999998e-06 2024-09-13T11:26:06.416 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:06.416 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:06.416 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:26:06.416 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:06.402396+0000", 2024-09-13T11:26:06.416 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.6245000000000001e-05 2024-09-13T11:26:06.417 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:06.417 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:06.417 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:26:06.417 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:06.402403+0000", 2024-09-13T11:26:06.417 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.8460000000000002e-06 2024-09-13T11:26:06.417 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:06.417 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:06.417 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:06.417 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:06.417 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-09-13T11:26:06.417 INFO:teuthology.orchestra.run.smithi001.stdout: "num_ops": 3 2024-09-13T11:26:06.417 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-13T11:26:06.421 INFO:tasks.ceph.osd.1.smithi092.stderr:2024-09-13T11:26:06.420+0000 7fe7789c5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33718) UID: 0 2024-09-13T11:26:06.428 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-13T11:26:06.465 INFO:teuthology.orchestra.run.smithi001.stdout:osd.0: {} 2024-09-13T11:26:06.466 INFO:teuthology.orchestra.run.smithi001.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T11:26:06.491 INFO:teuthology.orchestra.run.smithi001.stdout:osd.1: {} 2024-09-13T11:26:06.492 INFO:teuthology.orchestra.run.smithi001.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T11:26:06.523 INFO:tasks.ceph.osd.14.smithi162.stderr:2024-09-13T11:26:06.519+0000 7f3df5903640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33154) UID: 0 2024-09-13T11:26:06.523 INFO:teuthology.orchestra.run.smithi001.stdout:osd.2: {} 2024-09-13T11:26:06.524 INFO:teuthology.orchestra.run.smithi001.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T11:26:06.547 INFO:teuthology.orchestra.run.smithi001.stdout:osd.3: {} 2024-09-13T11:26:06.548 INFO:teuthology.orchestra.run.smithi001.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T11:26:06.569 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-13T11:26:06.569 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:26:06.573 INFO:teuthology.orchestra.run.smithi001.stdout:osd.4: {} 2024-09-13T11:26:06.574 INFO:teuthology.orchestra.run.smithi001.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T11:26:06.587 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-13T11:26:06.599 INFO:teuthology.orchestra.run.smithi001.stdout:osd.5: {} 2024-09-13T11:26:06.600 INFO:teuthology.orchestra.run.smithi001.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T11:26:06.626 INFO:teuthology.orchestra.run.smithi001.stdout:osd.6: {} 2024-09-13T11:26:06.626 INFO:teuthology.orchestra.run.smithi001.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T11:26:06.626 INFO:tasks.ceph.osd.4.smithi001.stderr:2024-09-13T11:26:06.622+0000 7ff8e0218640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26310) UID: 0 2024-09-13T11:26:06.652 INFO:teuthology.orchestra.run.smithi001.stdout:osd.7: {} 2024-09-13T11:26:06.653 INFO:teuthology.orchestra.run.smithi001.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T11:26:06.678 INFO:teuthology.orchestra.run.smithi001.stdout:osd.8: {} 2024-09-13T11:26:06.679 INFO:teuthology.orchestra.run.smithi001.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-13T11:26:06.703 INFO:teuthology.orchestra.run.smithi001.stdout:osd.9: {} 2024-09-13T11:26:06.704 INFO:teuthology.orchestra.run.smithi001.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-13T11:26:06.725 INFO:tasks.ceph.osd.5.smithi092.stderr:2024-09-13T11:26:06.724+0000 7f6f0b705640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26346) UID: 0 2024-09-13T11:26:06.728 INFO:teuthology.orchestra.run.smithi001.stdout:osd.10: {} 2024-09-13T11:26:06.728 INFO:teuthology.orchestra.run.smithi001.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-13T11:26:06.739 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-13T11:26:06.740 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:26:06.756 INFO:teuthology.orchestra.run.smithi001.stdout:osd.11: {} 2024-09-13T11:26:06.756 INFO:teuthology.orchestra.run.smithi001.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-13T11:26:06.757 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-13T11:26:06.782 INFO:teuthology.orchestra.run.smithi001.stdout:osd.12: {} 2024-09-13T11:26:06.783 INFO:teuthology.orchestra.run.smithi001.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-13T11:26:06.811 INFO:teuthology.orchestra.run.smithi001.stdout:osd.13: {} 2024-09-13T11:26:06.811 INFO:teuthology.orchestra.run.smithi001.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-13T11:26:06.825 INFO:tasks.ceph.osd.15.smithi177.stderr:2024-09-13T11:26:06.821+0000 7f9d6ce77640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26578) UID: 0 2024-09-13T11:26:06.840 INFO:teuthology.orchestra.run.smithi001.stdout:osd.14: {} 2024-09-13T11:26:06.840 INFO:teuthology.orchestra.run.smithi001.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-13T11:26:06.867 INFO:teuthology.orchestra.run.smithi001.stdout:osd.15: {} 2024-09-13T11:26:06.867 INFO:teuthology.orchestra.run.smithi001.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-13T11:26:06.926 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-13T11:26:06.926 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:26:06.926 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-13T11:26:06.922+0000 7f6a68775640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26307) UID: 0 2024-09-13T11:26:06.942 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-13T11:26:06.948 INFO:tasks.ceph.osd.15.smithi177.stderr:2024-09-13T11:26:06.945+0000 7f9d618a4640 -1 osd.15 193 handle_osd_map injecting map crc failure 2024-09-13T11:26:06.953 INFO:tasks.ceph.osd.10.smithi162.stderr:2024-09-13T11:26:06.951+0000 7fb593421640 -1 osd.10 193 handle_osd_map injecting map crc failure 2024-09-13T11:26:07.027 INFO:tasks.ceph.osd.15.smithi177.stderr:2024-09-13T11:26:07.025+0000 7f9d6ce77640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26578) UID: 0 2024-09-13T11:26:07.059 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-13T11:26:07.111 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-13T11:26:07.111 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:26:07.126 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-13T11:26:07.128 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-09-13T11:26:07.123+0000 7f97befc9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26315) UID: 0 2024-09-13T11:26:07.228 INFO:tasks.ceph.osd.8.smithi001.stderr:2024-09-13T11:26:07.226+0000 7fb9b48ae640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26312) UID: 0 2024-09-13T11:26:07.260 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-13T11:26:07.260 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:26:07.277 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-09-13T11:26:07.329 INFO:tasks.ceph.osd.7.smithi177.stderr:2024-09-13T11:26:07.325+0000 7f897b51d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26575) UID: 0 2024-09-13T11:26:07.430 INFO:tasks.ceph.osd.11.smithi177.stderr:2024-09-13T11:26:07.425+0000 7f24792c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26580) UID: 0 2024-09-13T11:26:07.456 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-13T11:26:07.456 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:26:07.479 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-09-13T11:26:07.530 INFO:tasks.ceph.osd.3.smithi177.stderr:2024-09-13T11:26:07.525+0000 7f10adba4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26574) UID: 0 2024-09-13T11:26:07.614 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-13T11:26:07.615 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:26:07.631 INFO:tasks.ceph.osd.1.smithi092.stderr:2024-09-13T11:26:07.628+0000 7fe7789c5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33718) UID: 0 2024-09-13T11:26:07.632 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-09-13T11:26:07.732 INFO:tasks.ceph.osd.4.smithi001.stderr:2024-09-13T11:26:07.730+0000 7ff8e0218640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26310) UID: 0 2024-09-13T11:26:07.764 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-13T11:26:07.765 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:26:07.780 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_ops_in_flight 2024-09-13T11:26:07.834 INFO:tasks.ceph.osd.11.smithi177.stderr:2024-09-13T11:26:07.829+0000 7f24792c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26580) UID: 0 2024-09-13T11:26:07.938 INFO:tasks.ceph.osd.8.smithi001.stderr:2024-09-13T11:26:07.934+0000 7fb9b48ae640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26312) UID: 0 2024-09-13T11:26:07.959 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-13T11:26:07.959 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:26:07.976 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_blocked_ops 2024-09-13T11:26:08.038 INFO:tasks.ceph.osd.5.smithi092.stderr:2024-09-13T11:26:08.036+0000 7f6f0b705640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26346) UID: 0 2024-09-13T11:26:08.055 INFO:teuthology.orchestra.run.smithi001.stderr:noscrub is unset 2024-09-13T11:26:08.139 INFO:tasks.ceph.osd.5.smithi092.stderr:2024-09-13T11:26:08.136+0000 7f6f0b705640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26346) UID: 0 2024-09-13T11:26:08.163 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-13T11:26:08.163 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:26:08.183 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_historic_ops 2024-09-13T11:26:08.239 INFO:tasks.ceph.osd.10.smithi162.stderr:2024-09-13T11:26:08.235+0000 7fb59c9f0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26317) UID: 0 2024-09-13T11:26:08.323 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-13T11:26:08.323 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:26:08.341 INFO:tasks.ceph.osd.13.smithi092.stderr:2024-09-13T11:26:08.340+0000 7f3e5afea640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26350) UID: 0 2024-09-13T11:26:08.342 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-13T11:26:08.441 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-13T11:26:08.438+0000 7f6a68775640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26307) UID: 0 2024-09-13T11:26:08.490 INFO:teuthology.orchestra.run.smithi092.stdout:ERROR: (22) Invalid argument 2024-09-13T11:26:08.490 INFO:teuthology.orchestra.run.smithi092.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:26:08.511 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-13T11:26:08.542 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-13T11:26:08.538+0000 7f6a68775640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26307) UID: 0 2024-09-13T11:26:08.643 INFO:tasks.ceph.osd.11.smithi177.stderr:2024-09-13T11:26:08.641+0000 7f24792c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26580) UID: 0 2024-09-13T11:26:08.654 INFO:teuthology.orchestra.run.smithi092.stdout:ERROR: (22) Invalid argument 2024-09-13T11:26:08.654 INFO:teuthology.orchestra.run.smithi092.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:26:08.673 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-13T11:26:08.744 INFO:tasks.ceph.osd.5.smithi092.stderr:2024-09-13T11:26:08.740+0000 7f6f0b705640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26346) UID: 0 2024-09-13T11:26:08.812 INFO:teuthology.orchestra.run.smithi092.stdout:ERROR: (22) Invalid argument 2024-09-13T11:26:08.813 INFO:teuthology.orchestra.run.smithi092.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:26:08.830 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-09-13T11:26:08.877 INFO:tasks.ceph.osd.9.smithi092.stderr:2024-09-13T11:26:08.876+0000 7ff5ab3d0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26348) UID: 0 2024-09-13T11:26:08.887 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T11:26:08.946 INFO:tasks.ceph.osd.2.smithi162.stderr:2024-09-13T11:26:08.943+0000 7fe92b35a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26313) UID: 0 2024-09-13T11:26:09.014 INFO:teuthology.orchestra.run.smithi092.stdout:ERROR: (22) Invalid argument 2024-09-13T11:26:09.015 INFO:teuthology.orchestra.run.smithi092.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:26:09.032 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-09-13T11:26:09.039 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 15, 14] out_osds: [] dead_osds: [] live_osds: [0, 4, 8, 12, 5, 9, 13, 2, 6, 10, 3, 7, 11, 15, 1, 14] 2024-09-13T11:26:09.039 INFO:tasks.thrashosds.thrasher:choose_action: min_in 2 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-13T11:26:09.039 INFO:tasks.thrashosds.thrasher:fixing pg num pool unique_pool_0 2024-09-13T11:26:09.039 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-13T11:26:09.047 INFO:tasks.ceph.osd.15.smithi177.stderr:2024-09-13T11:26:09.045+0000 7f9d6ce77640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26578) UID: 0 2024-09-13T11:26:09.119 INFO:teuthology.orchestra.run.smithi001.stdout:osd.0: {} 2024-09-13T11:26:09.120 INFO:teuthology.orchestra.run.smithi001.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T11:26:09.145 INFO:teuthology.orchestra.run.smithi001.stdout:osd.1: {} 2024-09-13T11:26:09.145 INFO:teuthology.orchestra.run.smithi001.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T11:26:09.147 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-13T11:26:09.146+0000 7f6a68775640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26307) UID: 0 2024-09-13T11:26:09.173 INFO:teuthology.orchestra.run.smithi092.stdout:ERROR: (22) Invalid argument 2024-09-13T11:26:09.173 INFO:teuthology.orchestra.run.smithi092.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:26:09.176 INFO:teuthology.orchestra.run.smithi001.stdout:osd.2: {} 2024-09-13T11:26:09.176 INFO:teuthology.orchestra.run.smithi001.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T11:26:09.193 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-09-13T11:26:09.211 INFO:teuthology.orchestra.run.smithi001.stdout:osd.3: {} 2024-09-13T11:26:09.211 INFO:teuthology.orchestra.run.smithi001.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T11:26:09.248 INFO:teuthology.orchestra.run.smithi001.stdout:osd.4: {} 2024-09-13T11:26:09.248 INFO:teuthology.orchestra.run.smithi001.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T11:26:09.249 INFO:tasks.ceph.osd.10.smithi162.stderr:2024-09-13T11:26:09.247+0000 7fb59c9f0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26317) UID: 0 2024-09-13T11:26:09.287 INFO:teuthology.orchestra.run.smithi001.stdout:osd.5: {} 2024-09-13T11:26:09.287 INFO:teuthology.orchestra.run.smithi001.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T11:26:09.318 INFO:teuthology.orchestra.run.smithi001.stdout:osd.6: {} 2024-09-13T11:26:09.318 INFO:teuthology.orchestra.run.smithi001.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T11:26:09.336 INFO:teuthology.orchestra.run.smithi092.stdout:ERROR: (22) Invalid argument 2024-09-13T11:26:09.336 INFO:teuthology.orchestra.run.smithi092.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:26:09.349 INFO:tasks.ceph.osd.5.smithi092.stderr:2024-09-13T11:26:09.348+0000 7f6f0b705640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26346) UID: 0 2024-09-13T11:26:09.354 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_ops_in_flight 2024-09-13T11:26:09.361 INFO:teuthology.orchestra.run.smithi001.stdout:osd.7: {} 2024-09-13T11:26:09.361 INFO:teuthology.orchestra.run.smithi001.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T11:26:09.402 INFO:teuthology.orchestra.run.smithi001.stdout:osd.8: {} 2024-09-13T11:26:09.402 INFO:teuthology.orchestra.run.smithi001.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-13T11:26:09.440 INFO:teuthology.orchestra.run.smithi001.stdout:osd.9: {} 2024-09-13T11:26:09.440 INFO:teuthology.orchestra.run.smithi001.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-13T11:26:09.450 INFO:tasks.ceph.osd.13.smithi092.stderr:2024-09-13T11:26:09.448+0000 7f3e5afea640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26350) UID: 0 2024-09-13T11:26:09.472 INFO:teuthology.orchestra.run.smithi001.stdout:osd.10: {} 2024-09-13T11:26:09.472 INFO:teuthology.orchestra.run.smithi001.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-13T11:26:09.486 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-13T11:26:09.489 INFO:teuthology.orchestra.run.smithi001.stderr:dumped all 2024-09-13T11:26:09.501 INFO:teuthology.orchestra.run.smithi001.stdout:osd.11: {} 2024-09-13T11:26:09.501 INFO:teuthology.orchestra.run.smithi001.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-13T11:26:09.513 INFO:teuthology.orchestra.run.smithi001.stdout:{"pg_ready":true,"pg_map":{"version":300,"stamp":"2024-09-13T11:26:08.172405+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":3855090251,"num_objects":58820,"num_object_clones":0,"num_object_copies":117640,"num_objects_missing_on_primary":0,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":7890,"num_objects_unfound":0,"num_objects_dirty":58820,"num_whiteouts":0,"num_read":87,"num_read_kb":74,"num_write":131279,"num_write_kb":3356723,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17476,"num_bytes_recovered":1145241619,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":331378,"ondisk_log_size":331378,"up":110,"acting":110,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":120,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":16,"kb":1509949440,"kb_used":9745740,"kb_used_data":8305420,"kb_used_omap":68,"kb_used_meta":1439995,"kb_avail":1500203700,"statfs":{"total":1546188226560,"available":1536208588800,"internally_reserved":0,"allocated":8504750080,"data_stored":8491322784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":70045,"internal_metadata":1474555491},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-198574080,"num_objects":-3030,"num_object_clones":0,"num_object_copies":-6060,"num_objects_missing_on_primary":0,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":7890,"num_objects_unfound":0,"num_objects_dirty":-3030,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3030,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":390,"num_bytes_recovered":25624576,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.001379"},"pg_stats":[{"pgid":"3.2d","version":"192'6876","reported_seq":12445,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-13T11:26:06.035586+0000","last_change":"2024-09-13T11:25:45.355259+0000","last_active":"2024-09-13T11:26:06.035586+0000","last_peered":"2024-09-13T11:26:06.035586+0000","last_clean":"2024-09-13T11:26:06.035586+0000","last_became_active":"2024-09-13T11:25:39.334408+0000","last_became_peered":"2024-09-13T11:25:39.334408+0000","last_unstale":"2024-09-13T11:26:06.035586+0000","last_undegraded":"2024-09-13T11:26:06.035586+0000","last_fullsized":"2024-09-13T11:26:06.035586+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":6,"last_scrub":"179'6515","last_scrub_stamp":"2024-09-13T11:25:45.355191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:45.355191+0000","objects_scrubbed":1320,"log_size":6876,"log_dups_size":0,"ondisk_log_size":6876,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:27:10.296174+0000","scrub_duration":199,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":62586880,"num_objects":955,"num_object_clones":0,"num_object_copies":1910,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":955,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":365,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"192'6914","reported_seq":12300,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-13T11:26:06.033991+0000","last_change":"2024-09-13T11:24:59.456762+0000","last_active":"2024-09-13T11:26:06.033991+0000","last_peered":"2024-09-13T11:26:06.033991+0000","last_clean":"2024-09-13T11:26:06.033991+0000","last_became_active":"2024-09-13T11:24:59.021895+0000","last_became_peered":"2024-09-13T11:24:59.021895+0000","last_unstale":"2024-09-13T11:26:06.033991+0000","last_undegraded":"2024-09-13T11:26:06.033991+0000","last_fullsized":"2024-09-13T11:26:06.033991+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":6,"last_scrub":"145'5672","last_scrub_stamp":"2024-09-13T11:24:59.456653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:24:59.456653+0000","objects_scrubbed":2192,"log_size":6914,"log_dups_size":0,"ondisk_log_size":6914,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:26:20.877991+0000","scrub_duration":279,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":62259200,"num_objects":950,"num_object_clones":0,"num_object_copies":1900,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":950,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1242,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,7],"acting":[10,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.2b","version":"192'6798","reported_seq":10556,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T11:26:05.096242+0000","last_change":"2024-09-13T11:26:01.126562+0000","last_active":"2024-09-13T11:26:05.096242+0000","last_peered":"2024-09-13T11:26:05.096242+0000","last_clean":"2024-09-13T11:26:05.096242+0000","last_became_active":"2024-09-13T11:24:26.180578+0000","last_became_peered":"2024-09-13T11:24:26.180578+0000","last_unstale":"2024-09-13T11:26:05.096242+0000","last_undegraded":"2024-09-13T11:26:05.096242+0000","last_fullsized":"2024-09-13T11:26:05.096242+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":6,"last_scrub":"188'6745","last_scrub_stamp":"2024-09-13T11:26:01.126423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:26:01.126423+0000","objects_scrubbed":916,"log_size":6798,"log_dups_size":0,"ondisk_log_size":6798,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:27:04.804300+0000","scrub_duration":391,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56098816,"num_objects":856,"num_object_clones":0,"num_object_copies":1712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":856,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1416,"num_write_kb":10624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2a","version":"192'6940","reported_seq":10870,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-13T11:26:06.033964+0000","last_change":"2024-09-13T11:24:46.625117+0000","last_active":"2024-09-13T11:26:06.033964+0000","last_peered":"2024-09-13T11:26:06.033964+0000","last_clean":"2024-09-13T11:26:06.033964+0000","last_became_active":"2024-09-13T11:24:45.320540+0000","last_became_peered":"2024-09-13T11:24:45.320540+0000","last_unstale":"2024-09-13T11:26:06.033964+0000","last_undegraded":"2024-09-13T11:26:06.033964+0000","last_fullsized":"2024-09-13T11:26:06.033964+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":6,"last_scrub":"133'5491","last_scrub_stamp":"2024-09-13T11:24:46.624970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:24:46.624970+0000","objects_scrubbed":2094,"log_size":6940,"log_dups_size":0,"ondisk_log_size":6940,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":65077248,"num_objects":993,"num_object_clones":0,"num_object_copies":1986,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":993,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1451,"num_write_kb":11200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,7],"acting":[10,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.29","version":"192'6632","reported_seq":12377,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-13T11:26:06.035350+0000","last_change":"2024-09-13T11:25:13.333062+0000","last_active":"2024-09-13T11:26:06.035350+0000","last_peered":"2024-09-13T11:26:06.035350+0000","last_clean":"2024-09-13T11:26:06.035350+0000","last_became_active":"2024-09-13T11:25:12.944028+0000","last_became_peered":"2024-09-13T11:25:12.944028+0000","last_unstale":"2024-09-13T11:26:06.035350+0000","last_undegraded":"2024-09-13T11:26:06.035350+0000","last_fullsized":"2024-09-13T11:26:06.035350+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":6,"last_scrub":"154'5579","last_scrub_stamp":"2024-09-13T11:25:13.332959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:13.332959+0000","objects_scrubbed":1937,"log_size":6632,"log_dups_size":0,"ondisk_log_size":6632,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:26:24.089228+0000","scrub_duration":243,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":57802752,"num_objects":882,"num_object_clones":0,"num_object_copies":1764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":882,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1055,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.28","version":"190'6420","reported_seq":11893,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-13T11:26:06.029375+0000","last_change":"2024-09-13T11:25:36.143261+0000","last_active":"2024-09-13T11:26:06.029375+0000","last_peered":"2024-09-13T11:26:06.029375+0000","last_clean":"2024-09-13T11:26:06.029375+0000","last_became_active":"2024-09-13T11:25:34.278640+0000","last_became_peered":"2024-09-13T11:25:34.278640+0000","last_unstale":"2024-09-13T11:26:06.029375+0000","last_undegraded":"2024-09-13T11:26:06.029375+0000","last_fullsized":"2024-09-13T11:26:06.029375+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":6,"last_scrub":"174'5936","last_scrub_stamp":"2024-09-13T11:25:36.143113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:36.143113+0000","objects_scrubbed":1447,"log_size":6420,"log_dups_size":0,"ondisk_log_size":6420,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:27:02.442082+0000","scrub_duration":1147,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":61865984,"num_objects":944,"num_object_clones":0,"num_object_copies":1888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":944,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":503,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5],"acting":[8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.27","version":"192'6532","reported_seq":12123,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-13T11:26:06.036074+0000","last_change":"2024-09-13T11:25:37.344464+0000","last_active":"2024-09-13T11:26:06.036074+0000","last_peered":"2024-09-13T11:26:06.036074+0000","last_clean":"2024-09-13T11:26:06.036074+0000","last_became_active":"2024-09-13T11:25:30.257911+0000","last_became_peered":"2024-09-13T11:25:30.257911+0000","last_unstale":"2024-09-13T11:26:06.036074+0000","last_undegraded":"2024-09-13T11:26:06.036074+0000","last_fullsized":"2024-09-13T11:26:06.036074+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":6,"last_scrub":"174'6026","last_scrub_stamp":"2024-09-13T11:25:37.344335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:37.344335+0000","objects_scrubbed":1463,"log_size":6532,"log_dups_size":0,"ondisk_log_size":6532,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:26:49.139373+0000","scrub_duration":195,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":62652416,"num_objects":956,"num_object_clones":0,"num_object_copies":1912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":507,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,11],"acting":[5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.26","version":"192'6663","reported_seq":12288,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-13T11:26:06.029695+0000","last_change":"2024-09-13T11:25:01.697082+0000","last_active":"2024-09-13T11:26:06.029695+0000","last_peered":"2024-09-13T11:26:06.029695+0000","last_clean":"2024-09-13T11:26:06.029695+0000","last_became_active":"2024-09-13T11:24:54.976919+0000","last_became_peered":"2024-09-13T11:24:54.976919+0000","last_unstale":"2024-09-13T11:26:06.029695+0000","last_undegraded":"2024-09-13T11:26:06.029695+0000","last_fullsized":"2024-09-13T11:26:06.029695+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":6,"last_scrub":"147'5433","last_scrub_stamp":"2024-09-13T11:25:01.696886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:01.696886+0000","objects_scrubbed":2158,"log_size":6663,"log_dups_size":0,"ondisk_log_size":6663,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:26:22.070621+0000","scrub_duration":403,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":60293120,"num_objects":920,"num_object_clones":0,"num_object_copies":1840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":920,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1238,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,12],"acting":[8,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.25","version":"191'6540","reported_seq":12324,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-13T11:26:06.067532+0000","last_change":"2024-09-13T11:24:57.073162+0000","last_active":"2024-09-13T11:26:06.067532+0000","last_peered":"2024-09-13T11:26:06.067532+0000","last_clean":"2024-09-13T11:26:06.067532+0000","last_became_active":"2024-09-13T11:24:55.999019+0000","last_became_peered":"2024-09-13T11:24:55.999019+0000","last_unstale":"2024-09-13T11:26:06.067532+0000","last_undegraded":"2024-09-13T11:26:06.067532+0000","last_fullsized":"2024-09-13T11:26:06.067532+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":6,"last_scrub":"142'5214","last_scrub_stamp":"2024-09-13T11:24:57.073061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:24:57.073061+0000","objects_scrubbed":2249,"log_size":6540,"log_dups_size":0,"ondisk_log_size":6540,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:26:12.348519+0000","scrub_duration":571,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":60686336,"num_objects":926,"num_object_clones":0,"num_object_copies":1852,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":926,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1329,"num_write_kb":192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,2],"acting":[12,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[]},{"pgid":"3.24","version":"194'6561","reported_seq":10804,"reported_epoch":194,"state":"active+recovering","last_fresh":"2024-09-13T11:26:06.965392+0000","last_change":"2024-09-13T11:26:05.265265+0000","last_active":"2024-09-13T11:26:06.965392+0000","last_peered":"2024-09-13T11:26:06.965392+0000","last_clean":"2024-09-13T11:26:03.399501+0000","last_became_active":"2024-09-13T11:26:05.240552+0000","last_became_peered":"2024-09-13T11:26:05.240552+0000","last_unstale":"2024-09-13T11:26:06.965392+0000","last_undegraded":"2024-09-13T11:26:06.965392+0000","last_fullsized":"2024-09-13T11:26:06.965392+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":6,"last_scrub":"188'6512","last_scrub_stamp":"2024-09-13T11:26:01.118438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:26:01.118438+0000","objects_scrubbed":972,"log_size":6561,"log_dups_size":0,"ondisk_log_size":6561,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":183,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":60489728,"num_objects":923,"num_object_clones":0,"num_object_copies":1846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":806,"num_objects_unfound":0,"num_objects_dirty":923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1469,"num_write_kb":9280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":125,"num_bytes_recovered":8192000,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,14],"acting":[9,14],"avail_no_missing":["9","1"],"object_location_counts":[{"shards":"1,9","objects":806},{"shards":"9,14","objects":117}],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.23","version":"190'5813","reported_seq":8751,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-13T11:26:06.035427+0000","last_change":"2024-09-13T11:25:56.586649+0000","last_active":"2024-09-13T11:26:06.035427+0000","last_peered":"2024-09-13T11:26:06.035427+0000","last_clean":"2024-09-13T11:26:06.035427+0000","last_became_active":"2024-09-13T11:23:47.926677+0000","last_became_peered":"2024-09-13T11:23:47.926677+0000","last_unstale":"2024-09-13T11:26:06.035427+0000","last_undegraded":"2024-09-13T11:26:06.035427+0000","last_fullsized":"2024-09-13T11:26:06.035427+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":6,"last_scrub":"186'5649","last_scrub_stamp":"2024-09-13T11:25:56.586425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:56.586425+0000","objects_scrubbed":1098,"log_size":5813,"log_dups_size":0,"ondisk_log_size":5813,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:27:16.699680+0000","scrub_duration":135,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":61145088,"num_objects":933,"num_object_clones":0,"num_object_copies":1866,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":933,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2536,"num_write_kb":78080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9],"acting":[5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.22","version":"192'5717","reported_seq":11543,"reported_epoch":193,"state":"active+recovering","last_fresh":"2024-09-13T11:26:06.074622+0000","last_change":"2024-09-13T11:26:05.254491+0000","last_active":"2024-09-13T11:26:06.074622+0000","last_peered":"2024-09-13T11:26:06.074622+0000","last_clean":"2024-09-13T11:26:03.171705+0000","last_became_active":"2024-09-13T11:26:05.226450+0000","last_became_peered":"2024-09-13T11:26:05.226450+0000","last_unstale":"2024-09-13T11:26:06.074622+0000","last_undegraded":"2024-09-13T11:26:06.074622+0000","last_fullsized":"2024-09-13T11:26:06.074622+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":6,"last_scrub":"186'5611","last_scrub_stamp":"2024-09-13T11:25:57.123569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:57.123569+0000","objects_scrubbed":1032,"log_size":5717,"log_dups_size":0,"ondisk_log_size":5717,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":147,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":60686336,"num_objects":926,"num_object_clones":0,"num_object_copies":1852,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":844,"num_objects_unfound":0,"num_objects_dirty":926,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2473,"num_write_kb":75712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1852,"num_bytes_recovered":121372672,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,14],"acting":[0,14],"avail_no_missing":["0","5"],"object_location_counts":[{"shards":"0,5","objects":844},{"shards":"0,14","objects":82}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.21","version":"192'5578","reported_seq":8518,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-13T11:26:06.035296+0000","last_change":"2024-09-13T11:25:49.226484+0000","last_active":"2024-09-13T11:26:06.035296+0000","last_peered":"2024-09-13T11:26:06.035296+0000","last_clean":"2024-09-13T11:26:06.035296+0000","last_became_active":"2024-09-13T11:24:01.250308+0000","last_became_peered":"2024-09-13T11:24:01.250308+0000","last_unstale":"2024-09-13T11:26:06.035296+0000","last_undegraded":"2024-09-13T11:26:06.035296+0000","last_fullsized":"2024-09-13T11:26:06.035296+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":6,"last_scrub":"182'5306","last_scrub_stamp":"2024-09-13T11:25:49.226272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:49.226272+0000","objects_scrubbed":1176,"log_size":5578,"log_dups_size":0,"ondisk_log_size":5578,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:26:50.150514+0000","scrub_duration":151,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":59113472,"num_objects":902,"num_object_clones":0,"num_object_copies":1804,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":902,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1911,"num_write_kb":44224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8],"acting":[0,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.20","version":"192'5769","reported_seq":7999,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-13T11:26:06.067241+0000","last_change":"2024-09-13T11:25:35.773483+0000","last_active":"2024-09-13T11:26:06.067241+0000","last_peered":"2024-09-13T11:26:06.067241+0000","last_clean":"2024-09-13T11:26:06.067241+0000","last_became_active":"2024-09-13T11:23:33.954946+0000","last_became_peered":"2024-09-13T11:23:33.954946+0000","last_unstale":"2024-09-13T11:26:06.067241+0000","last_undegraded":"2024-09-13T11:26:06.067241+0000","last_fullsized":"2024-09-13T11:26:06.067241+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":6,"last_scrub":"174'5240","last_scrub_stamp":"2024-09-13T11:25:35.773409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:35.773409+0000","objects_scrubbed":1424,"log_size":5769,"log_dups_size":0,"ondisk_log_size":5769,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:26:53.605828+0000","scrub_duration":215,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":58523648,"num_objects":893,"num_object_clones":0,"num_object_copies":1786,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":893,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2504,"num_write_kb":74752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,0],"acting":[12,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[]},{"pgid":"3.1f","version":"190'8468","reported_seq":20472,"reported_epoch":193,"state":"activating+remapped","last_fresh":"2024-09-13T11:26:06.049693+0000","last_change":"2024-09-13T11:26:06.049693+0000","last_active":"2024-09-13T11:26:04.037686+0000","last_peered":"2024-09-13T11:26:03.341211+0000","last_clean":"2024-09-13T11:26:03.341211+0000","last_became_active":"2024-09-13T11:24:29.290326+0000","last_became_peered":"2024-09-13T11:24:29.290326+0000","last_unstale":"2024-09-13T11:26:06.049693+0000","last_undegraded":"2024-09-13T11:26:06.049693+0000","last_fullsized":"2024-09-13T11:26:06.049693+0000","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":5,"last_scrub":"161'6865","last_scrub_stamp":"2024-09-13T11:25:20.773641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:20.773641+0000","objects_scrubbed":3425,"log_size":8468,"log_dups_size":0,"ondisk_log_size":8468,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":535,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":119078912,"num_objects":1817,"num_object_clones":0,"num_object_copies":3634,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1817,"num_objects_unfound":0,"num_objects_dirty":1817,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4048,"num_write_kb":102848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3306,"num_bytes_recovered":216662016,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,14],"acting":[10,5],"avail_no_missing":["10","5"],"object_location_counts":[{"shards":"5,10","objects":1817}],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1e","version":"192'8754","reported_seq":19326,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-13T11:26:06.035722+0000","last_change":"2024-09-13T11:24:08.338485+0000","last_active":"2024-09-13T11:26:06.035722+0000","last_peered":"2024-09-13T11:26:06.035722+0000","last_clean":"2024-09-13T11:26:06.035722+0000","last_became_active":"2024-09-13T11:24:01.261448+0000","last_became_peered":"2024-09-13T11:24:01.261448+0000","last_unstale":"2024-09-13T11:26:06.035722+0000","last_undegraded":"2024-09-13T11:26:06.035722+0000","last_fullsized":"2024-09-13T11:26:06.035722+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":5,"last_scrub":"110'4537","last_scrub_stamp":"2024-09-13T11:24:08.338354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:24:08.338354+0000","objects_scrubbed":2760,"log_size":8754,"log_dups_size":0,"ondisk_log_size":8754,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:26:07.469609+0000","scrub_duration":367,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":121503744,"num_objects":1854,"num_object_clones":0,"num_object_copies":3708,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1854,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4226,"num_write_kb":106240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"192'8529","reported_seq":14062,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T11:26:05.538168+0000","last_change":"2024-09-13T11:25:46.627072+0000","last_active":"2024-09-13T11:26:05.538168+0000","last_peered":"2024-09-13T11:26:05.538168+0000","last_clean":"2024-09-13T11:26:05.538168+0000","last_became_active":"2024-09-13T11:24:01.258225+0000","last_became_peered":"2024-09-13T11:24:01.258225+0000","last_unstale":"2024-09-13T11:26:05.538168+0000","last_undegraded":"2024-09-13T11:26:05.538168+0000","last_fullsized":"2024-09-13T11:26:05.538168+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":5,"last_scrub":"180'7891","last_scrub_stamp":"2024-09-13T11:25:46.626952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:46.626952+0000","objects_scrubbed":2517,"log_size":8529,"log_dups_size":0,"ondisk_log_size":8529,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:27:08.450527+0000","scrub_duration":875,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":122486808,"num_objects":1870,"num_object_clones":0,"num_object_copies":3740,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1870,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":3963,"num_write_kb":94593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,15],"acting":[2,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1c","version":"194'8749","reported_seq":12642,"reported_epoch":194,"state":"active+clean","last_fresh":"2024-09-13T11:26:07.048193+0000","last_change":"2024-09-13T11:25:46.456795+0000","last_active":"2024-09-13T11:26:07.048193+0000","last_peered":"2024-09-13T11:26:07.048193+0000","last_clean":"2024-09-13T11:26:07.048193+0000","last_became_active":"2024-09-13T11:23:33.989492+0000","last_became_peered":"2024-09-13T11:23:33.989492+0000","last_unstale":"2024-09-13T11:26:07.048193+0000","last_undegraded":"2024-09-13T11:26:07.048193+0000","last_fullsized":"2024-09-13T11:26:07.048193+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":5,"last_scrub":"179'8037","last_scrub_stamp":"2024-09-13T11:25:46.456644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:46.456644+0000","objects_scrubbed":2528,"log_size":8749,"log_dups_size":0,"ondisk_log_size":8749,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:27:11.089632+0000","scrub_duration":451,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":118620160,"num_objects":1810,"num_object_clones":0,"num_object_copies":3620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1810,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4474,"num_write_kb":119680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7],"acting":[11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1b","version":"192'8673","reported_seq":14384,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-13T11:26:06.067392+0000","last_change":"2024-09-13T11:25:49.890080+0000","last_active":"2024-09-13T11:26:06.067392+0000","last_peered":"2024-09-13T11:26:06.067392+0000","last_clean":"2024-09-13T11:26:06.067392+0000","last_became_active":"2024-09-13T11:24:04.950821+0000","last_became_peered":"2024-09-13T11:24:04.950821+0000","last_unstale":"2024-09-13T11:26:06.067392+0000","last_undegraded":"2024-09-13T11:26:06.067392+0000","last_fullsized":"2024-09-13T11:26:06.067392+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":5,"last_scrub":"182'8139","last_scrub_stamp":"2024-09-13T11:25:49.889914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:49.889914+0000","objects_scrubbed":2406,"log_size":8673,"log_dups_size":0,"ondisk_log_size":8673,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:26:54.473895+0000","scrub_duration":519,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":122093568,"num_objects":1863,"num_object_clones":0,"num_object_copies":3726,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1863,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4029,"num_write_kb":94656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,15],"acting":[12,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[]},{"pgid":"3.1a","version":"192'8452","reported_seq":13825,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T11:26:05.135528+0000","last_change":"2024-09-13T11:25:46.321420+0000","last_active":"2024-09-13T11:26:05.135528+0000","last_peered":"2024-09-13T11:26:05.135528+0000","last_clean":"2024-09-13T11:26:05.135528+0000","last_became_active":"2024-09-13T11:24:03.935185+0000","last_became_peered":"2024-09-13T11:24:03.935185+0000","last_unstale":"2024-09-13T11:26:05.135528+0000","last_undegraded":"2024-09-13T11:26:05.135528+0000","last_fullsized":"2024-09-13T11:26:05.135528+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":5,"last_scrub":"179'7826","last_scrub_stamp":"2024-09-13T11:25:46.321284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:46.321284+0000","objects_scrubbed":2526,"log_size":8452,"log_dups_size":0,"ondisk_log_size":8452,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:27:02.429140+0000","scrub_duration":935,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":123731968,"num_objects":1888,"num_object_clones":0,"num_object_copies":3776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1888,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4030,"num_write_kb":104448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9],"acting":[4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.19","version":"194'8180","reported_seq":12709,"reported_epoch":194,"state":"active+clean","last_fresh":"2024-09-13T11:26:07.051016+0000","last_change":"2024-09-13T11:25:10.784789+0000","last_active":"2024-09-13T11:26:07.051016+0000","last_peered":"2024-09-13T11:26:07.051016+0000","last_clean":"2024-09-13T11:26:07.051016+0000","last_became_active":"2024-09-13T11:23:41.924941+0000","last_became_peered":"2024-09-13T11:23:41.924941+0000","last_unstale":"2024-09-13T11:26:07.051016+0000","last_undegraded":"2024-09-13T11:26:07.051016+0000","last_fullsized":"2024-09-13T11:26:07.051016+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":5,"last_scrub":"151'6156","last_scrub_stamp":"2024-09-13T11:25:10.784716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:10.784716+0000","objects_scrubbed":3781,"log_size":8180,"log_dups_size":0,"ondisk_log_size":8180,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:26:30.390310+0000","scrub_duration":1131,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":113967104,"num_objects":1739,"num_object_clones":0,"num_object_copies":3478,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1739,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4778,"num_write_kb":145216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1],"acting":[9,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.18","version":"192'8321","reported_seq":13147,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-13T11:26:06.033969+0000","last_change":"2024-09-13T11:25:37.928304+0000","last_active":"2024-09-13T11:26:06.033969+0000","last_peered":"2024-09-13T11:26:06.033969+0000","last_clean":"2024-09-13T11:26:06.033969+0000","last_became_active":"2024-09-13T11:23:48.926014+0000","last_became_peered":"2024-09-13T11:23:48.926014+0000","last_unstale":"2024-09-13T11:26:06.033969+0000","last_undegraded":"2024-09-13T11:26:06.033969+0000","last_fullsized":"2024-09-13T11:26:06.033969+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":5,"last_scrub":"174'7306","last_scrub_stamp":"2024-09-13T11:25:37.928052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:37.928052+0000","objects_scrubbed":2815,"log_size":8321,"log_dups_size":0,"ondisk_log_size":8321,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:26:56.053559+0000","scrub_duration":631,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117047296,"num_objects":1786,"num_object_clones":0,"num_object_copies":3572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1786,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3954,"num_write_kb":86528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0],"acting":[10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.17","version":"192'8150","reported_seq":12411,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-13T11:26:06.034020+0000","last_change":"2024-09-13T11:25:12.658562+0000","last_active":"2024-09-13T11:26:06.034020+0000","last_peered":"2024-09-13T11:26:06.034020+0000","last_clean":"2024-09-13T11:26:06.034020+0000","last_became_active":"2024-09-13T11:23:42.935601+0000","last_became_peered":"2024-09-13T11:23:42.935601+0000","last_unstale":"2024-09-13T11:26:06.034020+0000","last_undegraded":"2024-09-13T11:26:06.034020+0000","last_fullsized":"2024-09-13T11:26:06.034020+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":5,"last_scrub":"153'6197","last_scrub_stamp":"2024-09-13T11:25:12.658464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:12.658464+0000","objects_scrubbed":3743,"log_size":8150,"log_dups_size":0,"ondisk_log_size":8150,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:26:25.625277+0000","scrub_duration":467,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117243904,"num_objects":1789,"num_object_clones":0,"num_object_copies":3578,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1789,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4799,"num_write_kb":146688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3],"acting":[10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"194'8575","reported_seq":13472,"reported_epoch":194,"state":"active+clean","last_fresh":"2024-09-13T11:26:07.104950+0000","last_change":"2024-09-13T11:25:01.903487+0000","last_active":"2024-09-13T11:26:07.104950+0000","last_peered":"2024-09-13T11:26:07.104950+0000","last_clean":"2024-09-13T11:26:07.104950+0000","last_became_active":"2024-09-13T11:23:46.105237+0000","last_became_peered":"2024-09-13T11:23:46.105237+0000","last_unstale":"2024-09-13T11:26:07.104950+0000","last_undegraded":"2024-09-13T11:26:07.104950+0000","last_fullsized":"2024-09-13T11:26:07.104950+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":5,"last_scrub":"147'6035","last_scrub_stamp":"2024-09-13T11:25:01.903395+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:01.903395+0000","objects_scrubbed":4359,"log_size":8575,"log_dups_size":0,"ondisk_log_size":8575,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:26:16.436377+0000","scrub_duration":743,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":118226944,"num_objects":1804,"num_object_clones":0,"num_object_copies":3608,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1804,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5100,"num_write_kb":155200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,15],"acting":[11,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.15","version":"190'8145","reported_seq":19554,"reported_epoch":193,"state":"activating+remapped","last_fresh":"2024-09-13T11:26:06.057217+0000","last_change":"2024-09-13T11:26:06.057217+0000","last_active":"2024-09-13T11:26:05.032627+0000","last_peered":"2024-09-13T11:26:03.255044+0000","last_clean":"2024-09-13T11:26:03.255044+0000","last_became_active":"2024-09-13T11:24:29.226615+0000","last_became_peered":"2024-09-13T11:24:29.226615+0000","last_unstale":"2024-09-13T11:26:06.057217+0000","last_undegraded":"2024-09-13T11:26:06.057217+0000","last_fullsized":"2024-09-13T11:26:06.057217+0000","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":5,"last_scrub":"181'7535","last_scrub_stamp":"2024-09-13T11:25:48.075568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:48.075568+0000","objects_scrubbed":2413,"log_size":8145,"log_dups_size":0,"ondisk_log_size":8145,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":991,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":116654080,"num_objects":1780,"num_object_clones":0,"num_object_copies":3560,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1780,"num_objects_unfound":0,"num_objects_dirty":1780,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4762,"num_write_kb":144320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3385,"num_bytes_recovered":221839360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,0],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,3","objects":1780}],"blocked_by":[],"up_primary":14,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.14","version":"190'8320","reported_seq":19659,"reported_epoch":192,"state":"remapped+peering","last_fresh":"2024-09-13T11:26:05.042149+0000","last_change":"2024-09-13T11:26:05.042149+0000","last_active":"2024-09-13T11:26:04.030468+0000","last_peered":"2024-09-13T11:26:03.284559+0000","last_clean":"2024-09-13T11:26:03.284559+0000","last_became_active":"2024-09-13T11:24:29.253106+0000","last_became_peered":"2024-09-13T11:24:29.253106+0000","last_unstale":"2024-09-13T11:26:05.042149+0000","last_undegraded":"2024-09-13T11:26:05.042149+0000","last_fullsized":"2024-09-13T11:26:05.042149+0000","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":5,"last_scrub":"174'7321","last_scrub_stamp":"2024-09-13T11:25:38.085667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:38.085667+0000","objects_scrubbed":2938,"log_size":8320,"log_dups_size":0,"ondisk_log_size":8320,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":431,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":126615552,"num_objects":1932,"num_object_clones":0,"num_object_copies":3864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1932,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4959,"num_write_kb":158592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3437,"num_bytes_recovered":225247232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,14],"acting":[3,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":366,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-13T11:26:06.073307+0000","last_change":"2024-09-13T11:25:46.222900+0000","last_active":"2024-09-13T11:26:06.073307+0000","last_peered":"2024-09-13T11:26:06.073307+0000","last_clean":"2024-09-13T11:26:06.073307+0000","last_became_active":"2024-09-13T11:22:14.192897+0000","last_became_peered":"2024-09-13T11:22:14.192897+0000","last_unstale":"2024-09-13T11:26:06.073307+0000","last_undegraded":"2024-09-13T11:26:06.073307+0000","last_fullsized":"2024-09-13T11:26:06.073307+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:25:46.222806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:25:46.222806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:27:02.988562+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00084655899999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,10],"acting":[15,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"3.6","version":"194'6669","reported_seq":10956,"reported_epoch":194,"state":"active+recovering","last_fresh":"2024-09-13T11:26:07.101951+0000","last_change":"2024-09-13T11:26:05.828206+0000","last_active":"2024-09-13T11:26:07.101951+0000","last_peered":"2024-09-13T11:26:07.101951+0000","last_clean":"2024-09-13T11:26:03.120046+0000","last_became_active":"2024-09-13T11:26:05.246522+0000","last_became_peered":"2024-09-13T11:26:05.246522+0000","last_unstale":"2024-09-13T11:26:07.101951+0000","last_undegraded":"2024-09-13T11:26:07.101951+0000","last_fullsized":"2024-09-13T11:26:07.101951+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"136'5216","last_scrub_stamp":"2024-09-13T11:24:49.416706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:24:49.416706+0000","objects_scrubbed":2083,"log_size":6669,"log_dups_size":0,"ondisk_log_size":6669,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":247,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":60489728,"num_objects":923,"num_object_clones":0,"num_object_copies":1846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":761,"num_objects_unfound":0,"num_objects_dirty":923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1454,"num_write_kb":9408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":182,"num_bytes_recovered":11927552,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,14],"acting":[13,14],"avail_no_missing":["13","6"],"object_location_counts":[{"shards":"6,13","objects":761},{"shards":"13,14","objects":162}],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":367,"reported_epoch":194,"state":"active+clean","last_fresh":"2024-09-13T11:26:06.948730+0000","last_change":"2024-09-13T11:24:59.556253+0000","last_active":"2024-09-13T11:26:06.948730+0000","last_peered":"2024-09-13T11:26:06.948730+0000","last_clean":"2024-09-13T11:26:06.948730+0000","last_became_active":"2024-09-13T11:22:14.211923+0000","last_became_peered":"2024-09-13T11:22:14.211923+0000","last_unstale":"2024-09-13T11:26:06.948730+0000","last_undegraded":"2024-09-13T11:26:06.948730+0000","last_fullsized":"2024-09-13T11:26:06.948730+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:24:59.556180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:24:59.556180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011689529999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0],"acting":[9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"194'6460","reported_seq":6890,"reported_epoch":194,"state":"active+clean","last_fresh":"2024-09-13T11:26:06.978121+0000","last_change":"2024-09-13T11:25:59.279849+0000","last_active":"2024-09-13T11:26:06.978121+0000","last_peered":"2024-09-13T11:26:06.978121+0000","last_clean":"2024-09-13T11:26:06.978121+0000","last_became_active":"2024-09-13T11:24:24.092975+0000","last_became_peered":"2024-09-13T11:24:24.092975+0000","last_unstale":"2024-09-13T11:26:06.978121+0000","last_undegraded":"2024-09-13T11:26:06.978121+0000","last_fullsized":"2024-09-13T11:26:06.978121+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"187'6375","last_scrub_stamp":"2024-09-13T11:25:59.279677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:59.279677+0000","objects_scrubbed":1033,"log_size":6460,"log_dups_size":0,"ondisk_log_size":6460,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:27:02.732059+0000","scrub_duration":139,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":62128128,"num_objects":948,"num_object_clones":0,"num_object_copies":1896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":948,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1642,"num_write_kb":26368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,8],"acting":[13,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":366,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-13T11:26:06.029476+0000","last_change":"2024-09-13T11:25:33.957283+0000","last_active":"2024-09-13T11:26:06.029476+0000","last_peered":"2024-09-13T11:26:06.029476+0000","last_clean":"2024-09-13T11:26:06.029476+0000","last_became_active":"2024-09-13T11:22:14.581458+0000","last_became_peered":"2024-09-13T11:22:14.581458+0000","last_unstale":"2024-09-13T11:26:06.029476+0000","last_undegraded":"2024-09-13T11:26:06.029476+0000","last_fullsized":"2024-09-13T11:26:06.029476+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:25:33.957179+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:25:33.957179+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:26:52.126360+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.001026047,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5],"acting":[8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"191'6522","reported_seq":6923,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T11:26:05.039559+0000","last_change":"2024-09-13T11:25:48.001902+0000","last_active":"2024-09-13T11:26:05.039559+0000","last_peered":"2024-09-13T11:26:05.039559+0000","last_clean":"2024-09-13T11:26:05.039559+0000","last_became_active":"2024-09-13T11:24:24.083077+0000","last_became_peered":"2024-09-13T11:24:24.083077+0000","last_unstale":"2024-09-13T11:26:05.039559+0000","last_undegraded":"2024-09-13T11:26:05.039559+0000","last_fullsized":"2024-09-13T11:26:05.039559+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"181'6222","last_scrub_stamp":"2024-09-13T11:25:48.001778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:48.001778+0000","objects_scrubbed":1224,"log_size":6522,"log_dups_size":0,"ondisk_log_size":6522,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:27:03.363579+0000","scrub_duration":171,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":60555264,"num_objects":924,"num_object_clones":0,"num_object_copies":1848,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":924,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1788,"num_write_kb":32704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,12],"acting":[1,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":352,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T11:26:05.038646+0000","last_change":"2024-09-13T11:25:45.851854+0000","last_active":"2024-09-13T11:26:05.038646+0000","last_peered":"2024-09-13T11:26:05.038646+0000","last_clean":"2024-09-13T11:26:05.038646+0000","last_became_active":"2024-09-13T11:22:47.719636+0000","last_became_peered":"2024-09-13T11:22:47.719636+0000","last_unstale":"2024-09-13T11:26:05.038646+0000","last_undegraded":"2024-09-13T11:26:05.038646+0000","last_fullsized":"2024-09-13T11:26:05.038646+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:25:45.851784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:25:45.851784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:26:50.817831+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00087476000000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,9],"acting":[1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"191'6459","reported_seq":6881,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-13T11:26:06.073374+0000","last_change":"2024-09-13T11:25:49.468719+0000","last_active":"2024-09-13T11:26:06.073374+0000","last_peered":"2024-09-13T11:26:06.073374+0000","last_clean":"2024-09-13T11:26:06.073374+0000","last_became_active":"2024-09-13T11:24:24.093492+0000","last_became_peered":"2024-09-13T11:24:24.093492+0000","last_unstale":"2024-09-13T11:26:06.073374+0000","last_undegraded":"2024-09-13T11:26:06.073374+0000","last_fullsized":"2024-09-13T11:26:06.073374+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"182'6171","last_scrub_stamp":"2024-09-13T11:25:49.468616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:49.468616+0000","objects_scrubbed":1202,"log_size":6459,"log_dups_size":0,"ondisk_log_size":6459,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:27:16.215875+0000","scrub_duration":175,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":59899904,"num_objects":914,"num_object_clones":0,"num_object_copies":1828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":914,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1722,"num_write_kb":32384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,13],"acting":[15,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"3.3","version":"192'5629","reported_seq":6051,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-13T11:26:06.033960+0000","last_change":"2024-09-13T11:25:08.394545+0000","last_active":"2024-09-13T11:26:06.033960+0000","last_peered":"2024-09-13T11:26:06.033960+0000","last_clean":"2024-09-13T11:26:06.033960+0000","last_became_active":"2024-09-13T11:23:31.930448+0000","last_became_peered":"2024-09-13T11:23:31.930448+0000","last_unstale":"2024-09-13T11:26:06.033960+0000","last_undegraded":"2024-09-13T11:26:06.033960+0000","last_fullsized":"2024-09-13T11:26:06.033960+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"151'4587","last_scrub_stamp":"2024-09-13T11:25:08.394441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:08.394441+0000","objects_scrubbed":1942,"log_size":5629,"log_dups_size":0,"ondisk_log_size":5629,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:26:27.641900+0000","scrub_duration":219,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":58785792,"num_objects":897,"num_object_clones":0,"num_object_copies":1794,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":897,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2507,"num_write_kb":80896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,7],"acting":[10,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"28'2","reported_seq":294,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T11:26:05.098986+0000","last_change":"2024-09-13T11:25:13.869564+0000","last_active":"2024-09-13T11:26:05.098986+0000","last_peered":"2024-09-13T11:26:05.098986+0000","last_clean":"2024-09-13T11:26:05.098986+0000","last_became_active":"2024-09-13T11:23:12.546348+0000","last_became_peered":"2024-09-13T11:23:12.546348+0000","last_unstale":"2024-09-13T11:26:05.098986+0000","last_undegraded":"2024-09-13T11:26:05.098986+0000","last_fullsized":"2024-09-13T11:26:05.098986+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2","last_scrub_stamp":"2024-09-13T11:25:13.869413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:25:13.869413+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:26:38.761718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0025826730000000002,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8],"acting":[2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"190'5668","reported_seq":6088,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-13T11:26:06.067280+0000","last_change":"2024-09-13T11:25:09.842875+0000","last_active":"2024-09-13T11:26:06.067280+0000","last_peered":"2024-09-13T11:26:06.067280+0000","last_clean":"2024-09-13T11:26:06.067280+0000","last_became_active":"2024-09-13T11:23:31.921388+0000","last_became_peered":"2024-09-13T11:23:31.921388+0000","last_unstale":"2024-09-13T11:26:06.067280+0000","last_undegraded":"2024-09-13T11:26:06.067280+0000","last_fullsized":"2024-09-13T11:26:06.067280+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"151'4644","last_scrub_stamp":"2024-09-13T11:25:09.842728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:09.842728+0000","objects_scrubbed":1944,"log_size":5668,"log_dups_size":0,"ondisk_log_size":5668,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:26:25.553642+0000","scrub_duration":395,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":60293120,"num_objects":920,"num_object_clones":0,"num_object_copies":1840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":920,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2562,"num_write_kb":84032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,15],"acting":[12,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2,"reported_epoch":191,"state":"peering","last_fresh":"2024-09-13T11:26:03.010832+0000","last_change":"2024-09-13T11:26:05.014044+0000","last_active":"2024-09-13T11:26:05.014044+0000","last_peered":"2024-09-13T11:26:03.010832+0000","last_clean":"2024-09-13T11:26:03.010832+0000","last_became_active":"2024-09-13T11:24:29.096575+0000","last_became_peered":"2024-09-13T11:24:29.096575+0000","last_unstale":"2024-09-13T11:26:03.010832+0000","last_undegraded":"2024-09-13T11:26:03.010832+0000","last_fullsized":"2024-09-13T11:26:03.010832+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:25:34.822831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:25:34.822831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00081225399999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,1],"acting":[14,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"3.1","version":"193'5663","reported_seq":6088,"reported_epoch":194,"state":"active+clean","last_fresh":"2024-09-13T11:26:06.948310+0000","last_change":"2024-09-13T11:25:00.861990+0000","last_active":"2024-09-13T11:26:06.948310+0000","last_peered":"2024-09-13T11:26:06.948310+0000","last_clean":"2024-09-13T11:26:06.948310+0000","last_became_active":"2024-09-13T11:23:31.926528+0000","last_became_peered":"2024-09-13T11:23:31.926528+0000","last_unstale":"2024-09-13T11:26:06.948310+0000","last_undegraded":"2024-09-13T11:26:06.948310+0000","last_fullsized":"2024-09-13T11:26:06.948310+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"146'4447","last_scrub_stamp":"2024-09-13T11:25:00.861834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:00.861834+0000","objects_scrubbed":2126,"log_size":5663,"log_dups_size":0,"ondisk_log_size":5663,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:26:15.411903+0000","scrub_duration":787,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":59441152,"num_objects":907,"num_object_clones":0,"num_object_copies":1814,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":907,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2593,"num_write_kb":85056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,11],"acting":[13,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":364,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T11:26:05.041230+0000","last_change":"2024-09-13T11:25:56.557174+0000","last_active":"2024-09-13T11:26:05.041230+0000","last_peered":"2024-09-13T11:26:05.041230+0000","last_clean":"2024-09-13T11:26:05.041230+0000","last_became_active":"2024-09-13T11:22:14.211435+0000","last_became_peered":"2024-09-13T11:22:14.211435+0000","last_unstale":"2024-09-13T11:26:05.041230+0000","last_undegraded":"2024-09-13T11:26:05.041230+0000","last_fullsized":"2024-09-13T11:26:05.041230+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:25:56.557113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:25:56.557113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:27:18.356524+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0010010360000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2","version":"190'5730","reported_seq":9610,"reported_epoch":192,"state":"remapped+peering","last_fresh":"2024-09-13T11:26:05.042426+0000","last_change":"2024-09-13T11:26:05.042426+0000","last_active":"2024-09-13T11:26:05.042426+0000","last_peered":"2024-09-13T11:26:03.273688+0000","last_clean":"2024-09-13T11:26:03.273688+0000","last_became_active":"2024-09-13T11:24:29.175407+0000","last_became_peered":"2024-09-13T11:24:29.175407+0000","last_unstale":"2024-09-13T11:26:05.042426+0000","last_undegraded":"2024-09-13T11:26:05.042426+0000","last_fullsized":"2024-09-13T11:26:05.042426+0000","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"164'4994","last_scrub_stamp":"2024-09-13T11:25:23.765047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:23.765047+0000","objects_scrubbed":1689,"log_size":5730,"log_dups_size":0,"ondisk_log_size":5730,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":283,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":62259200,"num_objects":950,"num_object_clones":0,"num_object_copies":1900,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":950,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2673,"num_write_kb":89024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1731,"num_bytes_recovered":113442816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,12],"acting":[3,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"26'1","reported_seq":2,"reported_epoch":191,"state":"peering","last_fresh":"2024-09-13T11:24:29.090041+0000","last_change":"2024-09-13T11:24:27.108657+0000","last_active":"2024-09-13T11:24:27.108657+0000","last_peered":"2024-09-13T11:22:14.146608+0000","last_clean":"2024-09-13T11:22:13.113563+0000","last_became_active":"2024-09-13T11:22:14.146507+0000","last_became_peered":"2024-09-13T11:22:14.146507+0000","last_unstale":"2024-09-13T11:24:29.090041+0000","last_undegraded":"2024-09-13T11:24:29.090041+0000","last_fullsized":"2024-09-13T11:24:29.090041+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"26'1","last_scrub_stamp":"2024-09-13T11:25:23.034247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:25:23.034247+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,15],"acting":[14,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"1.0","version":"26'112","reported_seq":498,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T11:26:05.068374+0000","last_change":"2024-09-13T11:25:23.642366+0000","last_active":"2024-09-13T11:26:05.068374+0000","last_peered":"2024-09-13T11:26:05.068374+0000","last_clean":"2024-09-13T11:26:05.068374+0000","last_became_active":"2024-09-13T11:22:12.129721+0000","last_became_peered":"2024-09-13T11:22:12.129721+0000","last_unstale":"2024-09-13T11:26:05.068374+0000","last_undegraded":"2024-09-13T11:26:05.068374+0000","last_fullsized":"2024-09-13T11:26:05.068374+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"26'112","last_scrub_stamp":"2024-09-13T11:25:23.642204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:11.098388+0000","last_clean_scrub_stamp":"2024-09-13T11:25:23.642204+0000","objects_scrubbed":2,"log_size":112,"log_dups_size":0,"ondisk_log_size":112,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:26:42.628319+0000","scrub_duration":67,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.8","version":"192'6496","reported_seq":6914,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T11:26:05.067919+0000","last_change":"2024-09-13T11:26:00.418168+0000","last_active":"2024-09-13T11:26:05.067919+0000","last_peered":"2024-09-13T11:26:05.067919+0000","last_clean":"2024-09-13T11:26:05.067919+0000","last_became_active":"2024-09-13T11:24:24.096220+0000","last_became_peered":"2024-09-13T11:24:24.096220+0000","last_unstale":"2024-09-13T11:26:05.067919+0000","last_undegraded":"2024-09-13T11:26:05.067919+0000","last_fullsized":"2024-09-13T11:26:05.067919+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"188'6425","last_scrub_stamp":"2024-09-13T11:26:00.417942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:26:00.417942+0000","objects_scrubbed":989,"log_size":6496,"log_dups_size":0,"ondisk_log_size":6496,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:27:27.750354+0000","scrub_duration":199,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":60162048,"num_objects":918,"num_object_clones":0,"num_object_copies":1836,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":918,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1748,"num_write_kb":31232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.9","version":"192'6527","reported_seq":6929,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-13T11:26:06.067368+0000","last_change":"2024-09-13T11:24:24.928964+0000","last_active":"2024-09-13T11:26:06.067368+0000","last_peered":"2024-09-13T11:26:06.067368+0000","last_clean":"2024-09-13T11:26:06.067368+0000","last_became_active":"2024-09-13T11:24:24.082504+0000","last_became_peered":"2024-09-13T11:24:24.082504+0000","last_unstale":"2024-09-13T11:26:06.067368+0000","last_undegraded":"2024-09-13T11:26:06.067368+0000","last_fullsized":"2024-09-13T11:26:06.067368+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'4799","last_scrub_stamp":"2024-09-13T11:24:24.928724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:24:24.928724+0000","objects_scrubbed":1705,"log_size":6527,"log_dups_size":0,"ondisk_log_size":6527,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:26:08.328055+0000","scrub_duration":255,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":61997056,"num_objects":946,"num_object_clones":0,"num_object_copies":1892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":946,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1731,"num_write_kb":31104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,2],"acting":[12,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[]},{"pgid":"3.a","version":"191'6909","reported_seq":7331,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T11:26:05.043736+0000","last_change":"2024-09-13T11:25:59.695285+0000","last_active":"2024-09-13T11:26:05.043736+0000","last_peered":"2024-09-13T11:26:05.043736+0000","last_clean":"2024-09-13T11:26:05.043736+0000","last_became_active":"2024-09-13T11:24:24.089253+0000","last_became_peered":"2024-09-13T11:24:24.089253+0000","last_unstale":"2024-09-13T11:26:05.043736+0000","last_undegraded":"2024-09-13T11:26:05.043736+0000","last_fullsized":"2024-09-13T11:26:05.043736+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"187'6833","last_scrub_stamp":"2024-09-13T11:25:59.695136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:59.695136+0000","objects_scrubbed":973,"log_size":6909,"log_dups_size":0,"ondisk_log_size":6909,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:27:24.613862+0000","scrub_duration":131,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":58720256,"num_objects":896,"num_object_clones":0,"num_object_copies":1792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":896,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1755,"num_write_kb":27968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.b","version":"194'6868","reported_seq":7307,"reported_epoch":194,"state":"active+clean","last_fresh":"2024-09-13T11:26:06.948372+0000","last_change":"2024-09-13T11:25:58.292249+0000","last_active":"2024-09-13T11:26:06.948372+0000","last_peered":"2024-09-13T11:26:06.948372+0000","last_clean":"2024-09-13T11:26:06.948372+0000","last_became_active":"2024-09-13T11:24:24.085684+0000","last_became_peered":"2024-09-13T11:24:24.085684+0000","last_unstale":"2024-09-13T11:26:06.948372+0000","last_undegraded":"2024-09-13T11:26:06.948372+0000","last_fullsized":"2024-09-13T11:26:06.948372+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"186'6759","last_scrub_stamp":"2024-09-13T11:25:58.292153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:58.292153+0000","objects_scrubbed":1030,"log_size":6868,"log_dups_size":0,"ondisk_log_size":6868,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:27:05.520730+0000","scrub_duration":119,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":60293120,"num_objects":920,"num_object_clones":0,"num_object_copies":1840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":920,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1756,"num_write_kb":30656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,1],"acting":[13,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"3.c","version":"194'6888","reported_seq":7318,"reported_epoch":194,"state":"active+clean","last_fresh":"2024-09-13T11:26:07.021630+0000","last_change":"2024-09-13T11:25:36.279464+0000","last_active":"2024-09-13T11:26:07.021630+0000","last_peered":"2024-09-13T11:26:07.021630+0000","last_clean":"2024-09-13T11:26:07.021630+0000","last_became_active":"2024-09-13T11:24:24.090847+0000","last_became_peered":"2024-09-13T11:24:24.090847+0000","last_unstale":"2024-09-13T11:26:07.021630+0000","last_undegraded":"2024-09-13T11:26:07.021630+0000","last_fullsized":"2024-09-13T11:26:07.021630+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"174'6334","last_scrub_stamp":"2024-09-13T11:25:36.279390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:36.279390+0000","objects_scrubbed":1456,"log_size":6888,"log_dups_size":0,"ondisk_log_size":6888,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:27:04.013051+0000","scrub_duration":235,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":59047936,"num_objects":901,"num_object_clones":0,"num_object_copies":1802,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":901,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1739,"num_write_kb":29440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,13],"acting":[11,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.d","version":"192'6859","reported_seq":7283,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-13T11:26:06.033288+0000","last_change":"2024-09-13T11:25:57.320482+0000","last_active":"2024-09-13T11:26:06.033288+0000","last_peered":"2024-09-13T11:26:06.033288+0000","last_clean":"2024-09-13T11:26:06.033288+0000","last_became_active":"2024-09-13T11:24:24.087224+0000","last_became_peered":"2024-09-13T11:24:24.087224+0000","last_unstale":"2024-09-13T11:26:06.033288+0000","last_undegraded":"2024-09-13T11:26:06.033288+0000","last_fullsized":"2024-09-13T11:26:06.033288+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"186'6735","last_scrub_stamp":"2024-09-13T11:25:57.320378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:57.320378+0000","objects_scrubbed":1064,"log_size":6859,"log_dups_size":0,"ondisk_log_size":6859,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:27:26.258353+0000","scrub_duration":211,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":61538304,"num_objects":939,"num_object_clones":0,"num_object_copies":1878,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":939,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1752,"num_write_kb":30912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.e","version":"192'8712","reported_seq":14058,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-13T11:26:06.035319+0000","last_change":"2024-09-13T11:25:46.438251+0000","last_active":"2024-09-13T11:26:06.035319+0000","last_peered":"2024-09-13T11:26:06.035319+0000","last_clean":"2024-09-13T11:26:06.035319+0000","last_became_active":"2024-09-13T11:23:31.925869+0000","last_became_peered":"2024-09-13T11:23:31.925869+0000","last_unstale":"2024-09-13T11:26:06.035319+0000","last_undegraded":"2024-09-13T11:26:06.035319+0000","last_fullsized":"2024-09-13T11:26:06.035319+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"180'8032","last_scrub_stamp":"2024-09-13T11:25:46.438107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:46.438107+0000","objects_scrubbed":2456,"log_size":8712,"log_dups_size":0,"ondisk_log_size":8712,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:26:50.686322+0000","scrub_duration":327,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":116129792,"num_objects":1772,"num_object_clones":0,"num_object_copies":3544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1772,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5186,"num_write_kb":165888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"194'8676","reported_seq":9102,"reported_epoch":194,"state":"active+clean","last_fresh":"2024-09-13T11:26:07.049229+0000","last_change":"2024-09-13T11:25:00.289940+0000","last_active":"2024-09-13T11:26:07.049229+0000","last_peered":"2024-09-13T11:26:07.049229+0000","last_clean":"2024-09-13T11:26:07.049229+0000","last_became_active":"2024-09-13T11:23:31.935239+0000","last_became_peered":"2024-09-13T11:23:31.935239+0000","last_unstale":"2024-09-13T11:26:07.049229+0000","last_undegraded":"2024-09-13T11:26:07.049229+0000","last_fullsized":"2024-09-13T11:26:07.049229+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"146'6154","last_scrub_stamp":"2024-09-13T11:25:00.289673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:00.289673+0000","objects_scrubbed":4394,"log_size":8676,"log_dups_size":0,"ondisk_log_size":8676,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:26:24.274856+0000","scrub_duration":1187,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":121241600,"num_objects":1850,"num_object_clones":0,"num_object_copies":3700,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1850,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5256,"num_write_kb":170560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,12],"acting":[11,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.10","version":"192'8413","reported_seq":12988,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T11:26:05.067911+0000","last_change":"2024-09-13T11:25:10.161564+0000","last_active":"2024-09-13T11:26:05.067911+0000","last_peered":"2024-09-13T11:26:05.067911+0000","last_clean":"2024-09-13T11:26:05.067911+0000","last_became_active":"2024-09-13T11:23:41.931183+0000","last_became_peered":"2024-09-13T11:23:41.931183+0000","last_unstale":"2024-09-13T11:26:05.067911+0000","last_undegraded":"2024-09-13T11:26:05.067911+0000","last_fullsized":"2024-09-13T11:26:05.067911+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":5,"last_scrub":"151'6298","last_scrub_stamp":"2024-09-13T11:25:10.161464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:10.161464+0000","objects_scrubbed":3964,"log_size":8413,"log_dups_size":0,"ondisk_log_size":8413,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:26:33.015478+0000","scrub_duration":607,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":120193024,"num_objects":1834,"num_object_clones":0,"num_object_copies":3668,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1834,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4947,"num_write_kb":152000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.11","version":"192'8349","reported_seq":11697,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T11:26:05.586792+0000","last_change":"2024-09-13T11:25:24.287326+0000","last_active":"2024-09-13T11:26:05.586792+0000","last_peered":"2024-09-13T11:26:05.586792+0000","last_clean":"2024-09-13T11:26:05.586792+0000","last_became_active":"2024-09-13T11:23:39.063172+0000","last_became_peered":"2024-09-13T11:23:39.063172+0000","last_unstale":"2024-09-13T11:26:05.586792+0000","last_undegraded":"2024-09-13T11:26:05.586792+0000","last_fullsized":"2024-09-13T11:26:05.586792+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":5,"last_scrub":"165'6821","last_scrub_stamp":"2024-09-13T11:25:24.287082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:24.287082+0000","objects_scrubbed":3406,"log_size":8349,"log_dups_size":0,"ondisk_log_size":8349,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:26:49.405332+0000","scrub_duration":947,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":122224640,"num_objects":1865,"num_object_clones":0,"num_object_copies":3730,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1865,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5008,"num_write_kb":155584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11],"acting":[4,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.12","version":"190'8325","reported_seq":18532,"reported_epoch":193,"state":"activating+remapped","last_fresh":"2024-09-13T11:26:06.062216+0000","last_change":"2024-09-13T11:26:06.062216+0000","last_active":"2024-09-13T11:26:05.041207+0000","last_peered":"2024-09-13T11:26:03.227632+0000","last_clean":"2024-09-13T11:26:03.227632+0000","last_became_active":"2024-09-13T11:24:29.268229+0000","last_became_peered":"2024-09-13T11:24:29.268229+0000","last_unstale":"2024-09-13T11:26:06.062216+0000","last_undegraded":"2024-09-13T11:26:06.062216+0000","last_fullsized":"2024-09-13T11:26:06.062216+0000","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":5,"last_scrub":"186'8089","last_scrub_stamp":"2024-09-13T11:25:57.085837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:57.085837+0000","objects_scrubbed":2130,"log_size":8325,"log_dups_size":0,"ondisk_log_size":8325,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1023,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":123338752,"num_objects":1882,"num_object_clones":0,"num_object_copies":3764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1882,"num_objects_unfound":0,"num_objects_dirty":1882,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5216,"num_write_kb":171712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3457,"num_bytes_recovered":226557952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,7],"acting":[5,7],"avail_no_missing":["5","7"],"object_location_counts":[{"shards":"5,7","objects":1882}],"blocked_by":[],"up_primary":14,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"194'8372","reported_seq":11658,"reported_epoch":194,"state":"active+clean","last_fresh":"2024-09-13T11:26:07.099085+0000","last_change":"2024-09-13T11:25:01.153044+0000","last_active":"2024-09-13T11:26:07.099085+0000","last_peered":"2024-09-13T11:26:07.099085+0000","last_clean":"2024-09-13T11:26:07.099085+0000","last_became_active":"2024-09-13T11:23:41.082433+0000","last_became_peered":"2024-09-13T11:23:41.082433+0000","last_unstale":"2024-09-13T11:26:07.099085+0000","last_undegraded":"2024-09-13T11:26:07.099085+0000","last_fullsized":"2024-09-13T11:26:07.099085+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":5,"last_scrub":"147'5921","last_scrub_stamp":"2024-09-13T11:25:01.152971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:01.152971+0000","objects_scrubbed":4311,"log_size":8372,"log_dups_size":0,"ondisk_log_size":8372,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1011,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":119209984,"num_objects":1819,"num_object_clones":0,"num_object_copies":3638,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1819,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4976,"num_write_kb":154048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7],"acting":[11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":46,"stat_sum":{"num_bytes":3854499864,"num_objects":58816,"num_object_clones":0,"num_object_copies":117632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":7890,"num_objects_unfound":0,"num_objects_dirty":58816,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":131132,"num_write_kb":3354049,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17475,"num_bytes_recovered":1145241600,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8456904704,"data_stored":8456896560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":43282,"internal_metadata":0},"log_size":331263,"ondisk_log_size":331263,"up":92,"acting":92,"num_store_stats":16},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":11},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":112,"ondisk_log_size":112,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":13,"up_from":23,"seq":98784247859,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":466360,"kb_used_data":358824,"kb_used_omap":1,"kb_used_meta":107518,"kb_avail":93905480,"statfs":{"total":96636764160,"available":96159211520,"internally_reserved":0,"allocated":367435776,"data_stored":366562389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":110098885},"hb_peers":[0,1,2,4,6,8,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":23,"seq":98784247858,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":414528,"kb_used_data":351408,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":93957312,"statfs":{"total":96636764160,"available":96212287488,"internally_reserved":0,"allocated":359841792,"data_stored":359020310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":64616896},"hb_peers":[0,1,3,8,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":40,"seq":171798691881,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":673156,"kb_used_data":556468,"kb_used_omap":4,"kb_used_meta":116667,"kb_avail":93698684,"statfs":{"total":96636764160,"available":95947452416,"internally_reserved":0,"allocated":569823232,"data_stored":568986435,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4433,"internal_metadata":119467695},"hb_peers":[0,2,4,5,6,7,8,9,11,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":21,"seq":90194313266,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":687780,"kb_used_data":576084,"kb_used_omap":1,"kb_used_meta":111678,"kb_avail":93684060,"statfs":{"total":96636764160,"available":95932477440,"internally_reserved":0,"allocated":589910016,"data_stored":589047234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":114358727},"hb_peers":[0,2,4,5,7,9,10,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":21,"seq":90194313269,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":841508,"kb_used_data":715924,"kb_used_omap":1,"kb_used_meta":125566,"kb_avail":93530332,"statfs":{"total":96636764160,"available":95775059968,"internally_reserved":0,"allocated":733106176,"data_stored":732247012,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":128580039},"hb_peers":[0,2,4,6,7,8,9,11,12,13],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":22,"seq":94489280561,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":808620,"kb_used_data":696924,"kb_used_omap":1,"kb_used_meta":111678,"kb_avail":93563220,"statfs":{"total":96636764160,"available":95808737280,"internally_reserved":0,"allocated":713650176,"data_stored":712844693,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":114358725},"hb_peers":[1,3,4,5,7,8,10,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":22,"seq":94489280562,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":336468,"kb_used_data":265028,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":94035372,"statfs":{"total":96636764160,"available":96292220928,"internally_reserved":0,"allocated":271388672,"data_stored":270549209,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73136581},"hb_peers":[0,1,3,4,5,7,8,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":15,"up_from":22,"seq":94489280562,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":707168,"kb_used_data":599632,"kb_used_omap":1,"kb_used_meta":107518,"kb_avail":93664672,"statfs":{"total":96636764160,"available":95912624128,"internally_reserved":0,"allocated":614023168,"data_stored":613229351,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":110098886},"hb_peers":[0,4,5,6,7,8,10,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":22,"seq":94489280563,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":484772,"kb_used_data":417492,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":93887068,"statfs":{"total":96636764160,"available":96140357632,"internally_reserved":0,"allocated":427511808,"data_stored":426657734,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[3,5,6,7,9,10,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":22,"seq":94489280562,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":336960,"kb_used_data":287728,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94034880,"statfs":{"total":96636764160,"available":96291717120,"internally_reserved":0,"allocated":294633472,"data_stored":293810544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50395590},"hb_peers":[0,1,3,5,7,8,10,11,12,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":22,"seq":94489280562,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":873088,"kb_used_data":769712,"kb_used_omap":1,"kb_used_meta":103358,"kb_avail":93498752,"statfs":{"total":96636764160,"available":95742722048,"internally_reserved":0,"allocated":788185088,"data_stored":787365586,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":105839050},"hb_peers":[0,3,5,6,8,9,10,11,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":22,"seq":94489280562,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":342020,"kb_used_data":274740,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94029820,"statfs":{"total":96636764160,"available":96286535680,"internally_reserved":0,"allocated":281333760,"data_stored":280451411,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[1,5,6,7,9,10,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":22,"seq":94489280561,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":628484,"kb_used_data":529268,"kb_used_omap":1,"kb_used_meta":99198,"kb_avail":93743356,"statfs":{"total":96636764160,"available":95993196544,"internally_reserved":0,"allocated":541970432,"data_stored":541143401,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":101579200},"hb_peers":[0,1,3,5,7,9,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":11,"up_from":22,"seq":94489280563,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":864820,"kb_used_data":765604,"kb_used_omap":1,"kb_used_meta":99198,"kb_avail":93507020,"statfs":{"total":96636764160,"available":95751188480,"internally_reserved":0,"allocated":783978496,"data_stored":783112660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":101579204},"hb_peers":[0,1,2,4,7,8,10,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":12,"up_from":22,"seq":94489280562,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":764196,"kb_used_data":656660,"kb_used_omap":1,"kb_used_meta":107518,"kb_avail":93607644,"statfs":{"total":96636764160,"available":95854227456,"internally_reserved":0,"allocated":672419840,"data_stored":671562689,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":110098885},"hb_peers":[0,1,2,3,4,5,10,11,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":14,"up_from":142,"seq":609885356047,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":515812,"kb_used_data":483924,"kb_used_omap":42,"kb_used_meta":31829,"kb_avail":93856028,"statfs":{"total":96636764160,"available":96108572672,"internally_reserved":0,"allocated":495538176,"data_stored":494732126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":43282,"internal_metadata":32593646},"hb_peers":[0,1,2,3,4,7,9,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":710934528,"data_stored":710934528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":567410688,"data_stored":567410688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":269225984,"data_stored":269221912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":587726848,"data_stored":587726848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":425263104,"data_stored":425263104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":730791936,"data_stored":730791936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":289800192,"data_stored":289800192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":785973248,"data_stored":785973248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":279117824,"data_stored":279117824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":357040128,"data_stored":357040128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":539492352,"data_stored":539492352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":778567680,"data_stored":778567680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":669843456,"data_stored":669843456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":360579072,"data_stored":360579072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":493420544,"data_stored":493420544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":43282,"internal_metadata":0},{"poolid":3,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":611717120,"data_stored":611713048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T11:26:09.516 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool set unique_pool_0 pgp_num 46 2024-09-13T11:26:09.528 INFO:teuthology.orchestra.run.smithi001.stdout:osd.12: {} 2024-09-13T11:26:09.529 INFO:teuthology.orchestra.run.smithi001.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-13T11:26:09.535 INFO:teuthology.orchestra.run.smithi092.stdout:ERROR: (22) Invalid argument 2024-09-13T11:26:09.535 INFO:teuthology.orchestra.run.smithi092.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:26:09.551 INFO:tasks.ceph.osd.15.smithi177.stderr:2024-09-13T11:26:09.549+0000 7f9d6ce77640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26578) UID: 0 2024-09-13T11:26:09.554 INFO:teuthology.orchestra.run.smithi001.stdout:osd.13: {} 2024-09-13T11:26:09.554 INFO:teuthology.orchestra.run.smithi001.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-13T11:26:09.556 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_blocked_ops 2024-09-13T11:26:09.579 INFO:teuthology.orchestra.run.smithi001.stdout:osd.14: {} 2024-09-13T11:26:09.579 INFO:teuthology.orchestra.run.smithi001.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-13T11:26:09.613 INFO:teuthology.orchestra.run.smithi001.stdout:osd.15: {} 2024-09-13T11:26:09.613 INFO:teuthology.orchestra.run.smithi001.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-13T11:26:09.653 INFO:tasks.ceph.osd.15.smithi177.stderr:2024-09-13T11:26:09.649+0000 7f9d6ce77640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26578) UID: 0 2024-09-13T11:26:09.745 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-09-13T11:26:09.745 INFO:teuthology.orchestra.run.smithi092.stdout: "ops": [], 2024-09-13T11:26:09.745 INFO:teuthology.orchestra.run.smithi092.stdout: "complaint_time": 30, 2024-09-13T11:26:09.746 INFO:teuthology.orchestra.run.smithi092.stdout: "num_blocked_ops": 0 2024-09-13T11:26:09.746 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-09-13T11:26:09.754 INFO:tasks.ceph.osd.15.smithi177.stderr:2024-09-13T11:26:09.749+0000 7f9d6ce77640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26578) UID: 0 2024-09-13T11:26:09.765 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_historic_ops 2024-09-13T11:26:09.854 INFO:tasks.ceph.osd.9.smithi092.stderr:2024-09-13T11:26:09.852+0000 7ff5ab3d0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26348) UID: 0 2024-09-13T11:26:09.916 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-09-13T11:26:09.916 INFO:teuthology.orchestra.run.smithi092.stdout: "size": 20, 2024-09-13T11:26:09.916 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 600, 2024-09-13T11:26:09.916 INFO:teuthology.orchestra.run.smithi092.stdout: "ops": [ 2024-09-13T11:26:09.916 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:09.916 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:79 3.1 3:841df5b5:::benchmark_data_smithi177_31116_object78:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-09-13T11:26:09.916 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:27.056083+0000", 2024-09-13T11:26:09.916 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 222.85851538099999, 2024-09-13T11:26:09.917 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.89469730199999997, 2024-09-13T11:26:09.917 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:26:09.917 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:26:09.917 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:09.917 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:26:09.917 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:26:09.917 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:09.917 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 79 2024-09-13T11:26:09.917 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:09.917 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:26:09.917 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:09.917 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:26:09.917 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.056083+0000", 2024-09-13T11:26:09.917 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:09.918 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:09.918 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:09.918 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:26:09.918 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.056083+0000", 2024-09-13T11:26:09.918 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:09.918 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:09.918 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:09.918 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:26:09.918 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.056087+0000", 2024-09-13T11:26:09.918 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.0659999999999997e-06 2024-09-13T11:26:09.918 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:09.918 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:09.918 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:26:09.918 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.056148+0000", 2024-09-13T11:26:09.919 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.1376000000000003e-05 2024-09-13T11:26:09.919 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:09.919 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:09.919 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:26:09.919 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.056180+0000", 2024-09-13T11:26:09.919 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.2370999999999999e-05 2024-09-13T11:26:09.919 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:09.919 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:09.919 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:26:09.919 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.056195+0000", 2024-09-13T11:26:09.919 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.5119e-05 2024-09-13T11:26:09.919 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:09.919 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:09.919 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:09.920 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.056304+0000", 2024-09-13T11:26:09.920 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000108365 2024-09-13T11:26:09.920 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:09.920 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:09.920 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:26:09.920 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.056478+0000", 2024-09-13T11:26:09.920 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000174542 2024-09-13T11:26:09.920 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:09.920 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:09.921 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 11", 2024-09-13T11:26:09.921 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.056690+0000", 2024-09-13T11:26:09.921 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00021167500000000001 2024-09-13T11:26:09.921 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:09.921 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:09.921 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:26:09.921 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.058063+0000", 2024-09-13T11:26:09.921 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0013731819999999999 2024-09-13T11:26:09.921 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:09.922 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:09.922 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:09.922 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.950477+0000", 2024-09-13T11:26:09.922 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.89241369400000004 2024-09-13T11:26:09.922 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:09.922 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:09.922 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:26:09.922 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.950652+0000", 2024-09-13T11:26:09.922 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00017526400000000001 2024-09-13T11:26:09.922 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:09.922 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:09.922 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:26:09.922 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.950780+0000", 2024-09-13T11:26:09.923 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00012764800000000001 2024-09-13T11:26:09.923 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:09.923 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:09.923 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:09.923 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:09.923 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:09.923 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:7 3.b 3:de78393b:::benchmark_data_smithi177_31116_object6:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T11:26:09.923 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:48.313952+0000", 2024-09-13T11:26:09.923 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 201.60064563200001, 2024-09-13T11:26:09.924 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.86421288600000001, 2024-09-13T11:26:09.924 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:26:09.924 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:26:09.924 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:09.924 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:26:09.924 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:26:09.924 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:09.924 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 7 2024-09-13T11:26:09.924 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:09.924 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:26:09.924 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:09.924 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:26:09.925 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.313952+0000", 2024-09-13T11:26:09.925 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:09.925 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:09.925 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:09.925 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:26:09.925 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.313952+0000", 2024-09-13T11:26:09.925 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:09.925 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:09.925 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:09.926 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:26:09.926 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.313957+0000", 2024-09-13T11:26:09.926 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.3599999999999998e-06 2024-09-13T11:26:09.926 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:09.926 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:09.926 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:26:09.926 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314043+0000", 2024-09-13T11:26:09.926 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.6766000000000002e-05 2024-09-13T11:26:09.926 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:09.926 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:09.926 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:26:09.926 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314065+0000", 2024-09-13T11:26:09.927 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.1416e-05 2024-09-13T11:26:09.927 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:09.927 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:09.927 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:26:09.927 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314080+0000", 2024-09-13T11:26:09.927 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.5519000000000002e-05 2024-09-13T11:26:09.927 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:09.927 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:09.927 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:09.927 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315478+0000", 2024-09-13T11:26:09.927 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.001397765 2024-09-13T11:26:09.927 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:09.927 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:09.927 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-09-13T11:26:09.928 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315507+0000", 2024-09-13T11:26:09.928 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.9128999999999998e-05 2024-09-13T11:26:09.928 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:09.928 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:09.928 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:09.928 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.167978+0000", 2024-09-13T11:26:09.928 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.85247068400000003 2024-09-13T11:26:09.928 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:09.928 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:09.928 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:26:09.928 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.168307+0000", 2024-09-13T11:26:09.928 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00032886699999999999 2024-09-13T11:26:09.928 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:09.928 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:09.928 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1", 2024-09-13T11:26:09.929 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.168647+0000", 2024-09-13T11:26:09.929 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00033998399999999999 2024-09-13T11:26:09.929 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:09.929 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:09.929 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:09.929 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.178011+0000", 2024-09-13T11:26:09.929 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0093639169999999994 2024-09-13T11:26:09.929 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:09.929 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:09.929 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:26:09.929 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.178026+0000", 2024-09-13T11:26:09.929 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.5274999999999999e-05 2024-09-13T11:26:09.929 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:09.929 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:09.930 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:26:09.930 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.178080+0000", 2024-09-13T11:26:09.930 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.3601999999999999e-05 2024-09-13T11:26:09.930 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:09.930 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:09.930 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:26:09.930 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.178165+0000", 2024-09-13T11:26:09.930 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.5601999999999998e-05 2024-09-13T11:26:09.930 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:09.930 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:09.930 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:09.930 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:09.930 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:09.930 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:9 3.b 3:dafadff6:::benchmark_data_smithi177_31116_object8:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T11:26:09.931 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:48.314114+0000", 2024-09-13T11:26:09.931 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 201.60048381799999, 2024-09-13T11:26:09.931 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.86613615899999996, 2024-09-13T11:26:09.931 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:26:09.931 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:26:09.931 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:09.931 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:26:09.931 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:26:09.931 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:09.931 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 9 2024-09-13T11:26:09.931 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:09.931 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:26:09.931 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:09.931 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:26:09.932 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314114+0000", 2024-09-13T11:26:09.932 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:09.932 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:09.932 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:09.932 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:26:09.932 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314114+0000", 2024-09-13T11:26:09.932 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:09.932 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:09.932 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:09.932 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:26:09.932 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314117+0000", 2024-09-13T11:26:09.932 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.3860000000000001e-06 2024-09-13T11:26:09.932 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:09.932 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:09.933 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:26:09.933 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314215+0000", 2024-09-13T11:26:09.933 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.8900000000000005e-05 2024-09-13T11:26:09.933 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:09.933 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:09.933 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:26:09.933 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314229+0000", 2024-09-13T11:26:09.933 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.3444999999999999e-05 2024-09-13T11:26:09.933 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:09.933 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:09.933 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:26:09.934 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314239+0000", 2024-09-13T11:26:09.934 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.8470000000000008e-06 2024-09-13T11:26:09.934 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:09.934 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:09.934 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:09.934 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315648+0000", 2024-09-13T11:26:09.934 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.001409291 2024-09-13T11:26:09.934 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:09.934 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:09.934 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-09-13T11:26:09.934 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315671+0000", 2024-09-13T11:26:09.934 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.3169000000000001e-05 2024-09-13T11:26:09.934 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:09.935 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:09.935 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:09.935 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.176036+0000", 2024-09-13T11:26:09.935 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.86036497499999998 2024-09-13T11:26:09.935 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:09.935 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:09.935 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:26:09.935 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.176189+0000", 2024-09-13T11:26:09.935 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00015271100000000001 2024-09-13T11:26:09.935 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:09.935 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:09.935 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1", 2024-09-13T11:26:09.935 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.176339+0000", 2024-09-13T11:26:09.936 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00015057499999999999 2024-09-13T11:26:09.936 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:09.936 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:09.936 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:09.936 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.180148+0000", 2024-09-13T11:26:09.936 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.003808891 2024-09-13T11:26:09.936 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:09.936 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:09.936 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:26:09.936 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.180168+0000", 2024-09-13T11:26:09.936 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.9507000000000001e-05 2024-09-13T11:26:09.936 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:09.936 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:09.936 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:26:09.937 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.180213+0000", 2024-09-13T11:26:09.937 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.5504999999999999e-05 2024-09-13T11:26:09.937 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:09.937 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:09.937 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:26:09.937 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.180250+0000", 2024-09-13T11:26:09.937 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.6956999999999999e-05 2024-09-13T11:26:09.937 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:09.937 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:09.937 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:09.937 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:09.937 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:09.937 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:10 3.b 3:dbe028a8:::benchmark_data_smithi177_31116_object9:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T11:26:09.937 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:48.314259+0000", 2024-09-13T11:26:09.938 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 201.60033868799999, 2024-09-13T11:26:09.938 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.86637313599999999, 2024-09-13T11:26:09.938 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:26:09.938 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:26:09.938 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:09.938 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:26:09.938 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:26:09.938 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:09.938 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 10 2024-09-13T11:26:09.938 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:09.938 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:26:09.938 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:09.938 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:26:09.938 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314259+0000", 2024-09-13T11:26:09.939 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:09.939 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:09.939 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:09.939 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:26:09.939 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314259+0000", 2024-09-13T11:26:09.939 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:09.939 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:09.939 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:09.939 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:26:09.939 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314261+0000", 2024-09-13T11:26:09.939 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.1500000000000002e-06 2024-09-13T11:26:09.939 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:09.939 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:09.940 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:26:09.940 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314349+0000", 2024-09-13T11:26:09.940 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.8070000000000002e-05 2024-09-13T11:26:09.940 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:09.940 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:09.940 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:26:09.940 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314365+0000", 2024-09-13T11:26:09.940 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.5835999999999999e-05 2024-09-13T11:26:09.940 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:09.940 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:09.940 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:26:09.940 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314377+0000", 2024-09-13T11:26:09.940 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.1589e-05 2024-09-13T11:26:09.940 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:09.941 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:09.941 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:09.941 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315788+0000", 2024-09-13T11:26:09.941 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0014109579999999999 2024-09-13T11:26:09.941 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:09.941 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:09.941 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-09-13T11:26:09.941 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315809+0000", 2024-09-13T11:26:09.941 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.0985999999999999e-05 2024-09-13T11:26:09.941 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:09.941 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:09.941 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:09.941 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.176822+0000", 2024-09-13T11:26:09.941 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.86101310799999997 2024-09-13T11:26:09.942 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:09.942 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:09.942 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:26:09.942 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.177021+0000", 2024-09-13T11:26:09.942 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00019911199999999999 2024-09-13T11:26:09.942 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:09.942 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:09.942 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1", 2024-09-13T11:26:09.942 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.177226+0000", 2024-09-13T11:26:09.942 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00020516600000000001 2024-09-13T11:26:09.942 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:09.942 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:09.942 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:26:09.943 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.180264+0000", 2024-09-13T11:26:09.943 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0030381739999999998 2024-09-13T11:26:09.943 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:09.943 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:09.943 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:09.943 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.180387+0000", 2024-09-13T11:26:09.943 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00012257699999999999 2024-09-13T11:26:09.943 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:09.943 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:09.943 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:26:09.943 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.180518+0000", 2024-09-13T11:26:09.943 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00013135800000000001 2024-09-13T11:26:09.943 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:09.943 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:09.943 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:26:09.944 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.180632+0000", 2024-09-13T11:26:09.944 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000114052 2024-09-13T11:26:09.944 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:09.944 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:09.944 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:09.944 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:09.944 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:09.944 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:39 3.b 3:d91854b3:::benchmark_data_smithi177_31116_object38:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T11:26:09.944 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:48.314393+0000", 2024-09-13T11:26:09.944 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 201.60020514600001, 2024-09-13T11:26:09.944 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.86850121700000005, 2024-09-13T11:26:09.944 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:26:09.944 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:26:09.944 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:09.945 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:26:09.945 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:26:09.945 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:09.945 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 39 2024-09-13T11:26:09.945 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:09.945 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:26:09.945 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:09.945 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:26:09.945 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314393+0000", 2024-09-13T11:26:09.945 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:09.945 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:09.945 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:09.945 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:26:09.946 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314393+0000", 2024-09-13T11:26:09.946 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:09.946 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:09.946 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:09.946 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:26:09.946 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314394+0000", 2024-09-13T11:26:09.946 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.513e-06 2024-09-13T11:26:09.946 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:09.946 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:09.946 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:26:09.946 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314463+0000", 2024-09-13T11:26:09.946 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.8261000000000003e-05 2024-09-13T11:26:09.946 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:09.946 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:09.947 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:26:09.947 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314477+0000", 2024-09-13T11:26:09.947 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.4619e-05 2024-09-13T11:26:09.947 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:09.947 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:09.947 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:26:09.947 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314485+0000", 2024-09-13T11:26:09.947 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.1699999999999997e-06 2024-09-13T11:26:09.947 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:09.947 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:09.947 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:09.947 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315917+0000", 2024-09-13T11:26:09.947 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0014316089999999999 2024-09-13T11:26:09.947 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:09.947 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:09.948 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-09-13T11:26:09.948 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315933+0000", 2024-09-13T11:26:09.948 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.5999999999999999e-05 2024-09-13T11:26:09.948 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:09.948 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:09.948 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:09.948 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.178205+0000", 2024-09-13T11:26:09.948 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.86227247100000004 2024-09-13T11:26:09.948 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:09.948 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:09.948 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:26:09.948 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.178316+0000", 2024-09-13T11:26:09.949 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000110788 2024-09-13T11:26:09.949 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:09.949 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:09.949 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1", 2024-09-13T11:26:09.949 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.178448+0000", 2024-09-13T11:26:09.949 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000132056 2024-09-13T11:26:09.949 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:09.949 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:09.949 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:26:09.949 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.182721+0000", 2024-09-13T11:26:09.949 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0042729780000000002 2024-09-13T11:26:09.949 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:09.949 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:09.950 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:09.950 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.182801+0000", 2024-09-13T11:26:09.950 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.0209000000000004e-05 2024-09-13T11:26:09.950 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:09.950 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:09.950 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:26:09.950 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.182855+0000", 2024-09-13T11:26:09.950 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.3749000000000003e-05 2024-09-13T11:26:09.950 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:09.950 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:09.950 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:26:09.950 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.182894+0000", 2024-09-13T11:26:09.950 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.8794000000000002e-05 2024-09-13T11:26:09.951 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:09.951 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:09.951 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:09.951 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:09.951 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:09.951 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:50 3.b 3:d1c144b8:::benchmark_data_smithi177_31116_object49:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T11:26:09.951 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:48.314498+0000", 2024-09-13T11:26:09.951 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 201.600099731, 2024-09-13T11:26:09.951 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.86878673299999998, 2024-09-13T11:26:09.951 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:26:09.951 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:26:09.951 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:09.951 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:26:09.952 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:26:09.952 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:09.952 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 50 2024-09-13T11:26:09.952 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:09.952 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:26:09.952 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:09.952 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:26:09.952 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314498+0000", 2024-09-13T11:26:09.952 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:09.952 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:09.952 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:09.952 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:26:09.952 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314498+0000", 2024-09-13T11:26:09.952 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:09.953 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:09.953 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:09.953 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:26:09.953 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314499+0000", 2024-09-13T11:26:09.953 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.1969999999999999e-06 2024-09-13T11:26:09.953 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:09.953 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:09.953 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:26:09.953 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314565+0000", 2024-09-13T11:26:09.953 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.5628000000000003e-05 2024-09-13T11:26:09.953 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:09.953 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:09.953 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:26:09.954 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314579+0000", 2024-09-13T11:26:09.954 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.4222e-05 2024-09-13T11:26:09.954 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:09.954 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:09.954 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:26:09.954 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314590+0000", 2024-09-13T11:26:09.954 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.1240000000000001e-05 2024-09-13T11:26:09.954 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:09.955 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:09.956 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:09.956 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316060+0000", 2024-09-13T11:26:09.956 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.001469515 2024-09-13T11:26:09.956 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:09.956 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:09.956 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-09-13T11:26:09.956 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316076+0000", 2024-09-13T11:26:09.956 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.5608999999999999e-05 2024-09-13T11:26:09.956 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:09.956 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:09.956 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:09.956 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.178857+0000", 2024-09-13T11:26:09.956 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.86278099900000005 2024-09-13T11:26:09.957 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:09.957 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:09.957 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:26:09.957 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.178961+0000", 2024-09-13T11:26:09.957 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000104664 2024-09-13T11:26:09.957 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:09.957 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:09.957 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1", 2024-09-13T11:26:09.957 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.179123+0000", 2024-09-13T11:26:09.957 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000161738 2024-09-13T11:26:09.957 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:09.958 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:09.958 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:26:09.958 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.182755+0000", 2024-09-13T11:26:09.958 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0036323200000000001 2024-09-13T11:26:09.958 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:09.958 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:09.958 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:09.958 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.183023+0000", 2024-09-13T11:26:09.958 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00026749100000000002 2024-09-13T11:26:09.958 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:09.958 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:09.958 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:26:09.958 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.183168+0000", 2024-09-13T11:26:09.958 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00014499099999999999 2024-09-13T11:26:09.959 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:09.959 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:09.959 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:26:09.959 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.183285+0000", 2024-09-13T11:26:09.959 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000117119 2024-09-13T11:26:09.959 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:09.959 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:09.959 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:09.959 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:09.959 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:09.959 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:60 3.b 3:d67218b0:::benchmark_data_smithi177_31116_object59:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T11:26:09.959 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:48.314603+0000", 2024-09-13T11:26:09.959 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 201.59999516799999, 2024-09-13T11:26:09.959 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.87106201100000002, 2024-09-13T11:26:09.960 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:26:09.960 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:26:09.960 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:09.960 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:26:09.960 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:26:09.960 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:09.960 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 60 2024-09-13T11:26:09.960 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:09.960 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:26:09.960 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:09.960 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:26:09.960 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314603+0000", 2024-09-13T11:26:09.960 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:09.961 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:09.961 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:09.961 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:26:09.961 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314603+0000", 2024-09-13T11:26:09.961 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:09.961 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:09.961 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:09.961 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:26:09.961 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314604+0000", 2024-09-13T11:26:09.961 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.2359999999999999e-06 2024-09-13T11:26:09.961 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:09.961 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:09.961 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:26:09.962 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314667+0000", 2024-09-13T11:26:09.962 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.3077999999999995e-05 2024-09-13T11:26:09.962 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:09.962 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:09.962 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:26:09.962 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314699+0000", 2024-09-13T11:26:09.962 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.1965999999999999e-05 2024-09-13T11:26:09.962 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:09.962 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:09.962 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:26:09.962 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314710+0000", 2024-09-13T11:26:09.962 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.1148999999999999e-05 2024-09-13T11:26:09.962 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:09.962 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:09.963 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:09.963 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316291+0000", 2024-09-13T11:26:09.963 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0015804440000000001 2024-09-13T11:26:09.963 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:09.963 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:09.963 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-09-13T11:26:09.963 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316310+0000", 2024-09-13T11:26:09.963 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.9799000000000001e-05 2024-09-13T11:26:09.963 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:09.963 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:09.963 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:09.963 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.179555+0000", 2024-09-13T11:26:09.963 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.86324469500000001 2024-09-13T11:26:09.963 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:09.964 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:09.964 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:26:09.964 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.179666+0000", 2024-09-13T11:26:09.964 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000110443 2024-09-13T11:26:09.964 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:09.964 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:09.964 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1", 2024-09-13T11:26:09.964 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.179791+0000", 2024-09-13T11:26:09.964 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00012560499999999999 2024-09-13T11:26:09.964 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:09.964 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:09.964 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:26:09.964 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.182761+0000", 2024-09-13T11:26:09.964 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0029697019999999998 2024-09-13T11:26:09.965 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:09.965 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:09.965 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:09.965 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.185572+0000", 2024-09-13T11:26:09.965 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.002811512 2024-09-13T11:26:09.965 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:09.965 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:09.965 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:26:09.965 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.185625+0000", 2024-09-13T11:26:09.965 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.2716000000000001e-05 2024-09-13T11:26:09.965 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:09.965 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:09.965 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:26:09.966 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.185665+0000", 2024-09-13T11:26:09.966 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.9666000000000003e-05 2024-09-13T11:26:09.966 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:09.966 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:09.966 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:09.966 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:09.966 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:09.966 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:62 3.b 3:d20ae5a8:::benchmark_data_smithi177_31116_object61:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T11:26:09.966 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:48.314723+0000", 2024-09-13T11:26:09.966 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 201.599874626, 2024-09-13T11:26:09.966 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.87338985099999999, 2024-09-13T11:26:09.966 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:26:09.966 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:26:09.966 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:09.967 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:26:09.967 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:26:09.967 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:09.967 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 62 2024-09-13T11:26:09.967 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:09.967 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:26:09.967 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:09.967 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:26:09.967 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314723+0000", 2024-09-13T11:26:09.967 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:09.967 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:09.967 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:09.967 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:26:09.967 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314723+0000", 2024-09-13T11:26:09.968 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:09.968 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:09.968 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:09.968 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:26:09.968 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314725+0000", 2024-09-13T11:26:09.968 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.3069999999999999e-06 2024-09-13T11:26:09.968 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:09.968 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:09.968 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:26:09.968 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314797+0000", 2024-09-13T11:26:09.968 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.2652999999999997e-05 2024-09-13T11:26:09.968 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:09.969 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:09.969 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:26:09.969 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314809+0000", 2024-09-13T11:26:09.969 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.1496e-05 2024-09-13T11:26:09.969 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:09.969 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:09.969 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:26:09.969 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314816+0000", 2024-09-13T11:26:09.969 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.6639999999999998e-06 2024-09-13T11:26:09.969 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:09.969 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:09.969 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:09.969 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316424+0000", 2024-09-13T11:26:09.969 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0016077559999999999 2024-09-13T11:26:09.969 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:09.970 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:09.970 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-09-13T11:26:09.970 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316441+0000", 2024-09-13T11:26:09.970 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.7305000000000001e-05 2024-09-13T11:26:09.970 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:09.970 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:09.970 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:09.970 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.180688+0000", 2024-09-13T11:26:09.970 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.86424692800000003 2024-09-13T11:26:09.970 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:09.970 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:09.970 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:26:09.971 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.180807+0000", 2024-09-13T11:26:09.971 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000118977 2024-09-13T11:26:09.971 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:09.971 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:09.971 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1", 2024-09-13T11:26:09.971 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.180938+0000", 2024-09-13T11:26:09.971 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00013049700000000001 2024-09-13T11:26:09.971 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:09.971 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:09.971 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:26:09.971 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.185680+0000", 2024-09-13T11:26:09.971 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0047417029999999999 2024-09-13T11:26:09.971 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:09.972 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:09.972 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:09.972 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.187856+0000", 2024-09-13T11:26:09.972 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0021760550000000001 2024-09-13T11:26:09.972 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:09.972 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:09.972 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:26:09.972 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.188007+0000", 2024-09-13T11:26:09.972 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00015185099999999999 2024-09-13T11:26:09.972 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:09.972 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:09.972 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:26:09.973 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.188113+0000", 2024-09-13T11:26:09.973 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000105659 2024-09-13T11:26:09.973 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:09.973 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:09.973 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:09.973 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:09.973 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:09.973 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:73 3.b 3:d0060d5f:::benchmark_data_smithi177_31116_object72:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T11:26:09.973 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:48.314829+0000", 2024-09-13T11:26:09.973 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 201.59976927700001, 2024-09-13T11:26:09.973 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.87364757199999998, 2024-09-13T11:26:09.973 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:26:09.973 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:26:09.973 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:09.974 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:26:09.974 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:26:09.974 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:09.974 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 73 2024-09-13T11:26:09.974 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:09.974 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:26:09.974 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:09.974 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:26:09.974 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314829+0000", 2024-09-13T11:26:09.974 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:09.974 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:09.974 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:09.974 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:26:09.974 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314829+0000", 2024-09-13T11:26:09.975 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:09.975 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:09.975 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:09.975 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:26:09.975 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314830+0000", 2024-09-13T11:26:09.975 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.243e-06 2024-09-13T11:26:09.975 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:09.975 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:09.975 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:26:09.975 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314965+0000", 2024-09-13T11:26:09.975 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00013470800000000001 2024-09-13T11:26:09.975 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:09.976 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:09.976 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:26:09.976 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314979+0000", 2024-09-13T11:26:09.976 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.4375999999999999e-05 2024-09-13T11:26:09.976 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:09.976 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:09.976 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:26:09.976 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314988+0000", 2024-09-13T11:26:09.976 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.8570000000000001e-06 2024-09-13T11:26:09.976 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:09.976 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:09.976 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:09.976 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316545+0000", 2024-09-13T11:26:09.977 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.001556878 2024-09-13T11:26:09.977 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:09.977 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:09.977 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-09-13T11:26:09.977 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316556+0000", 2024-09-13T11:26:09.977 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.1453e-05 2024-09-13T11:26:09.977 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:09.977 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:09.977 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:09.977 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.182124+0000", 2024-09-13T11:26:09.977 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.86556744200000002 2024-09-13T11:26:09.977 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:09.978 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:09.978 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:26:09.978 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.182246+0000", 2024-09-13T11:26:09.978 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00012210500000000001 2024-09-13T11:26:09.978 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:09.978 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:09.978 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1", 2024-09-13T11:26:09.978 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.182375+0000", 2024-09-13T11:26:09.978 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00012920999999999999 2024-09-13T11:26:09.978 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:09.978 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:09.978 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:26:09.978 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.185684+0000", 2024-09-13T11:26:09.978 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0033093969999999999 2024-09-13T11:26:09.979 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:09.979 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:09.979 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:09.979 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.188244+0000", 2024-09-13T11:26:09.979 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0025601550000000002 2024-09-13T11:26:09.979 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:09.979 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:09.979 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:26:09.979 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.188379+0000", 2024-09-13T11:26:09.979 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00013463400000000001 2024-09-13T11:26:09.979 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:09.979 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:09.979 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:26:09.979 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.188476+0000", 2024-09-13T11:26:09.980 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.7114000000000006e-05 2024-09-13T11:26:09.980 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:09.980 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:09.980 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:09.980 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:09.980 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:09.980 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:81 3.b 3:d705fd70:::benchmark_data_smithi177_31116_object80:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T11:26:09.980 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:48.315000+0000", 2024-09-13T11:26:09.980 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 201.59959744, 2024-09-13T11:26:09.980 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.87561418300000005, 2024-09-13T11:26:09.980 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:26:09.980 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:26:09.980 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:09.981 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:26:09.981 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:26:09.981 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:09.981 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 81 2024-09-13T11:26:09.981 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:09.981 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:26:09.981 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:09.981 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:26:09.981 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315000+0000", 2024-09-13T11:26:09.981 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:09.981 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:09.981 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:09.981 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:26:09.982 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315000+0000", 2024-09-13T11:26:09.982 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:09.982 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:09.982 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:09.982 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:26:09.982 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315002+0000", 2024-09-13T11:26:09.982 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.3799999999999999e-06 2024-09-13T11:26:09.982 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:09.982 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:09.982 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:26:09.982 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315119+0000", 2024-09-13T11:26:09.982 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000116925 2024-09-13T11:26:09.982 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:09.983 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:09.983 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:26:09.983 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315133+0000", 2024-09-13T11:26:09.983 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.3971999999999999e-05 2024-09-13T11:26:09.983 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:09.983 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:09.983 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:26:09.983 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315142+0000", 2024-09-13T11:26:09.983 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.4399999999999994e-06 2024-09-13T11:26:09.983 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:09.983 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:09.983 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:09.984 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316652+0000", 2024-09-13T11:26:09.984 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.001509907 2024-09-13T11:26:09.984 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:09.984 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:09.984 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-09-13T11:26:09.984 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316668+0000", 2024-09-13T11:26:09.984 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.5945999999999999e-05 2024-09-13T11:26:09.984 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:09.984 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:09.984 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:09.984 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.183405+0000", 2024-09-13T11:26:09.984 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.866737384 2024-09-13T11:26:09.985 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:09.985 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:09.985 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:26:09.985 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.183701+0000", 2024-09-13T11:26:09.985 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00029511099999999997 2024-09-13T11:26:09.985 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:09.985 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:09.985 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1", 2024-09-13T11:26:09.985 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.184078+0000", 2024-09-13T11:26:09.985 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00037741000000000001 2024-09-13T11:26:09.985 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:09.985 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:09.985 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:26:09.985 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.188504+0000", 2024-09-13T11:26:09.985 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0044260280000000003 2024-09-13T11:26:09.986 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:09.986 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:09.986 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:09.986 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.190562+0000", 2024-09-13T11:26:09.986 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0020580099999999999 2024-09-13T11:26:09.986 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:09.986 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:09.986 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:26:09.986 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.190592+0000", 2024-09-13T11:26:09.986 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.0057999999999999e-05 2024-09-13T11:26:09.986 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:09.986 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:09.986 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:26:09.986 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.190615+0000", 2024-09-13T11:26:09.987 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.2612e-05 2024-09-13T11:26:09.987 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:09.987 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:09.987 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:09.987 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:09.987 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:09.987 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:87 3.b 3:dd7de0ad:::benchmark_data_smithi177_31116_object86:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T11:26:09.987 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:48.315155+0000", 2024-09-13T11:26:09.987 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 201.599442643, 2024-09-13T11:26:09.987 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.87555916099999997, 2024-09-13T11:26:09.987 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:26:09.987 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:26:09.987 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:09.987 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:26:09.988 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:26:09.988 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:09.988 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 87 2024-09-13T11:26:09.988 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:09.988 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:26:09.988 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:09.988 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:26:09.988 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315155+0000", 2024-09-13T11:26:09.988 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:09.988 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:09.988 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:09.988 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:26:09.988 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315155+0000", 2024-09-13T11:26:09.989 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:09.989 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:09.989 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:09.989 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:26:09.989 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315157+0000", 2024-09-13T11:26:09.989 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.3829999999999999e-06 2024-09-13T11:26:09.989 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:09.989 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:09.989 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:26:09.989 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315271+0000", 2024-09-13T11:26:09.989 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000114539 2024-09-13T11:26:09.989 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:09.989 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:09.990 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:26:09.990 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315285+0000", 2024-09-13T11:26:09.990 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.4042e-05 2024-09-13T11:26:09.990 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:09.990 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:09.990 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:26:09.990 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315295+0000", 2024-09-13T11:26:09.990 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.2739999999999996e-06 2024-09-13T11:26:09.990 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:09.990 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:09.990 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:09.990 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316739+0000", 2024-09-13T11:26:09.990 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0014446960000000001 2024-09-13T11:26:09.991 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:09.991 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:09.991 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-09-13T11:26:09.991 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316747+0000", 2024-09-13T11:26:09.991 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.0790000000000001e-06 2024-09-13T11:26:09.991 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:09.991 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:09.991 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:09.991 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.184922+0000", 2024-09-13T11:26:09.991 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.86817446399999998 2024-09-13T11:26:09.991 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:09.991 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:09.991 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:26:09.991 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.185037+0000", 2024-09-13T11:26:09.991 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00011565099999999999 2024-09-13T11:26:09.992 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:09.992 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:09.992 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1", 2024-09-13T11:26:09.992 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.185171+0000", 2024-09-13T11:26:09.992 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00013316900000000001 2024-09-13T11:26:09.992 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:09.992 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:09.992 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:26:09.992 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.188520+0000", 2024-09-13T11:26:09.992 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0033489150000000001 2024-09-13T11:26:09.992 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:09.992 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:09.993 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:09.993 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.190652+0000", 2024-09-13T11:26:09.993 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0021325649999999999 2024-09-13T11:26:09.993 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:09.993 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:09.993 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:26:09.993 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.190686+0000", 2024-09-13T11:26:09.993 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.3827e-05 2024-09-13T11:26:09.993 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:09.993 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:09.993 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:26:09.993 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.190714+0000", 2024-09-13T11:26:09.993 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.8557000000000001e-05 2024-09-13T11:26:09.993 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:09.994 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:09.994 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:09.994 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:09.994 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:09.994 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:99 3.b 3:d011b600:::benchmark_data_smithi177_31116_object98:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T11:26:09.994 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:48.315307+0000", 2024-09-13T11:26:09.994 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 201.59929081600001, 2024-09-13T11:26:09.994 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.87551770799999995, 2024-09-13T11:26:09.994 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:26:09.994 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:26:09.994 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:09.994 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:26:09.994 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:26:09.994 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:09.995 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 99 2024-09-13T11:26:09.995 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:09.995 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:26:09.995 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:09.995 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:26:09.995 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315307+0000", 2024-09-13T11:26:09.995 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:09.995 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:09.995 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:09.995 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:26:09.995 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315307+0000", 2024-09-13T11:26:09.995 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:09.995 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:09.995 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:09.996 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:26:09.996 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315308+0000", 2024-09-13T11:26:09.996 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.316e-06 2024-09-13T11:26:09.996 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:09.996 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:09.996 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:26:09.996 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315453+0000", 2024-09-13T11:26:09.996 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00014473200000000001 2024-09-13T11:26:09.996 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:09.996 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:09.996 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:26:09.996 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315471+0000", 2024-09-13T11:26:09.996 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.8019000000000001e-05 2024-09-13T11:26:09.997 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:09.997 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:09.997 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:26:09.997 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315489+0000", 2024-09-13T11:26:09.997 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.7861999999999999e-05 2024-09-13T11:26:09.997 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:09.997 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:09.997 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:09.997 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316837+0000", 2024-09-13T11:26:09.997 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.001347787 2024-09-13T11:26:09.997 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:09.997 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:09.997 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-09-13T11:26:09.998 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316858+0000", 2024-09-13T11:26:09.998 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.1103000000000001e-05 2024-09-13T11:26:09.998 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:09.998 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:09.998 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:09.998 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.185713+0000", 2024-09-13T11:26:09.998 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.86885540800000005 2024-09-13T11:26:09.998 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:09.998 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:09.998 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:26:09.998 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.185820+0000", 2024-09-13T11:26:09.999 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000107081 2024-09-13T11:26:09.999 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:09.999 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:09.999 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1", 2024-09-13T11:26:09.999 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.185948+0000", 2024-09-13T11:26:09.999 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00012775800000000001 2024-09-13T11:26:09.999 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:09.999 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:09.999 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:26:09.999 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.190527+0000", 2024-09-13T11:26:09.999 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0045783910000000002 2024-09-13T11:26:09.999 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:09.999 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:09.999 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:10.000 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.190759+0000", 2024-09-13T11:26:10.000 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00023266199999999999 2024-09-13T11:26:10.000 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:10.000 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:10.000 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:26:10.000 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.190789+0000", 2024-09-13T11:26:10.000 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.9666999999999999e-05 2024-09-13T11:26:10.000 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:10.000 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:10.000 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:26:10.000 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.190825+0000", 2024-09-13T11:26:10.000 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.5921999999999999e-05 2024-09-13T11:26:10.000 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:10.000 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:10.001 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:10.001 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:10.001 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:10.001 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:108 3.b 3:d5c90217:::benchmark_data_smithi177_31116_object107:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T11:26:10.001 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:48.315513+0000", 2024-09-13T11:26:10.001 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 201.59908511099999, 2024-09-13T11:26:10.001 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.87544772500000001, 2024-09-13T11:26:10.001 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:26:10.001 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:26:10.001 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:10.001 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:26:10.001 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:26:10.001 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:10.001 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 108 2024-09-13T11:26:10.002 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:10.002 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:26:10.002 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:10.002 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:26:10.002 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315513+0000", 2024-09-13T11:26:10.002 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:10.002 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:10.002 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:10.002 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:26:10.002 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315513+0000", 2024-09-13T11:26:10.002 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:10.002 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:10.002 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:10.002 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:26:10.002 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315515+0000", 2024-09-13T11:26:10.003 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.9369999999999998e-06 2024-09-13T11:26:10.003 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:10.003 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:10.003 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:26:10.003 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315644+0000", 2024-09-13T11:26:10.003 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000129011 2024-09-13T11:26:10.003 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:10.003 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:10.003 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:26:10.003 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315657+0000", 2024-09-13T11:26:10.003 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.3689999999999999e-05 2024-09-13T11:26:10.003 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:10.003 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:10.003 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:26:10.004 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315668+0000", 2024-09-13T11:26:10.004 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0589e-05 2024-09-13T11:26:10.004 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:10.004 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:10.004 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:10.004 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316925+0000", 2024-09-13T11:26:10.004 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0012568340000000001 2024-09-13T11:26:10.004 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:10.004 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:10.004 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-09-13T11:26:10.004 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316932+0000", 2024-09-13T11:26:10.004 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.2230000000000001e-06 2024-09-13T11:26:10.005 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:10.005 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:10.005 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:10.005 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.186619+0000", 2024-09-13T11:26:10.005 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.86968679599999998 2024-09-13T11:26:10.005 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:10.005 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:10.005 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:26:10.005 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.186914+0000", 2024-09-13T11:26:10.005 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00029528900000000001 2024-09-13T11:26:10.005 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:10.005 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:10.005 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1", 2024-09-13T11:26:10.006 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.187217+0000", 2024-09-13T11:26:10.006 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00030232099999999998 2024-09-13T11:26:10.006 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:10.006 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:10.006 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:26:10.006 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.190532+0000", 2024-09-13T11:26:10.006 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.003315391 2024-09-13T11:26:10.006 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:10.006 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:10.006 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:10.006 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.190875+0000", 2024-09-13T11:26:10.006 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00034326300000000001 2024-09-13T11:26:10.006 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:10.006 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:10.007 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:26:10.007 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.190925+0000", 2024-09-13T11:26:10.007 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.9429999999999999e-05 2024-09-13T11:26:10.007 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:10.007 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:10.007 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:26:10.007 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.190961+0000", 2024-09-13T11:26:10.007 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.5951000000000002e-05 2024-09-13T11:26:10.007 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:10.007 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:10.007 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:10.007 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:10.007 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:10.007 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:172 3.b 3:dbec1eeb:::benchmark_data_smithi177_31116_object171:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T11:26:10.007 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:48.315684+0000", 2024-09-13T11:26:10.008 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 201.59891395400001, 2024-09-13T11:26:10.008 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.87615312999999995, 2024-09-13T11:26:10.008 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:26:10.008 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:26:10.008 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:10.008 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:26:10.008 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:26:10.008 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:10.008 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 172 2024-09-13T11:26:10.008 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:10.008 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:26:10.008 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:10.008 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:26:10.009 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315684+0000", 2024-09-13T11:26:10.009 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:10.009 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:10.009 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:10.009 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:26:10.009 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315684+0000", 2024-09-13T11:26:10.009 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:10.009 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:10.009 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:10.009 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:26:10.009 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315686+0000", 2024-09-13T11:26:10.009 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.886e-06 2024-09-13T11:26:10.009 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:10.009 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:10.010 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:26:10.010 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315820+0000", 2024-09-13T11:26:10.010 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00013388500000000001 2024-09-13T11:26:10.010 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:10.010 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:10.010 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:26:10.010 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315837+0000", 2024-09-13T11:26:10.010 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.6756e-05 2024-09-13T11:26:10.010 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:10.010 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:10.010 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:26:10.010 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315851+0000", 2024-09-13T11:26:10.010 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.4566000000000001e-05 2024-09-13T11:26:10.010 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:10.011 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:10.011 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:10.011 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.317021+0000", 2024-09-13T11:26:10.011 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0011701769999999999 2024-09-13T11:26:10.011 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:10.011 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:10.011 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-09-13T11:26:10.011 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.317037+0000", 2024-09-13T11:26:10.011 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.609e-05 2024-09-13T11:26:10.011 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:10.011 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:10.011 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:10.011 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.188634+0000", 2024-09-13T11:26:10.011 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.87159711900000003 2024-09-13T11:26:10.011 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:10.012 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:10.012 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:26:10.012 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.188974+0000", 2024-09-13T11:26:10.012 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00033922299999999999 2024-09-13T11:26:10.012 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:10.012 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:10.012 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1", 2024-09-13T11:26:10.012 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.189510+0000", 2024-09-13T11:26:10.012 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00053618999999999997 2024-09-13T11:26:10.012 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:10.012 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:10.012 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:10.012 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.191027+0000", 2024-09-13T11:26:10.012 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0015168390000000001 2024-09-13T11:26:10.013 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:10.013 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:10.013 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:26:10.013 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.191766+0000", 2024-09-13T11:26:10.013 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000739064 2024-09-13T11:26:10.013 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:10.013 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:10.013 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:26:10.013 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.191810+0000", 2024-09-13T11:26:10.013 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.3755999999999999e-05 2024-09-13T11:26:10.013 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:10.013 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:10.013 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:26:10.013 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.191837+0000", 2024-09-13T11:26:10.013 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.7579000000000001e-05 2024-09-13T11:26:10.014 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:10.014 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:10.014 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:10.014 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:10.014 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:10.014 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:177 3.b 3:de89c779:::benchmark_data_smithi177_31116_object176:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T11:26:10.014 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:48.315898+0000", 2024-09-13T11:26:10.014 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 201.59869963899999, 2024-09-13T11:26:10.014 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.87600712599999997, 2024-09-13T11:26:10.014 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:26:10.014 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:26:10.014 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:10.014 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:26:10.014 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:26:10.015 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:10.015 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 177 2024-09-13T11:26:10.015 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:10.015 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:26:10.015 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:10.015 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:26:10.015 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315898+0000", 2024-09-13T11:26:10.015 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:10.015 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:10.015 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:10.015 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:26:10.015 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315898+0000", 2024-09-13T11:26:10.015 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:10.015 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:10.016 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:10.016 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:26:10.016 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315900+0000", 2024-09-13T11:26:10.016 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.4929999999999999e-06 2024-09-13T11:26:10.016 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:10.016 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:10.016 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:26:10.016 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316027+0000", 2024-09-13T11:26:10.016 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000127568 2024-09-13T11:26:10.016 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:10.016 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:10.016 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:26:10.016 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316048+0000", 2024-09-13T11:26:10.017 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.0795999999999999e-05 2024-09-13T11:26:10.017 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:10.017 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:10.017 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:26:10.017 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316061+0000", 2024-09-13T11:26:10.017 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.3283e-05 2024-09-13T11:26:10.017 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:10.017 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:10.017 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:10.017 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.317107+0000", 2024-09-13T11:26:10.017 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0010452949999999999 2024-09-13T11:26:10.017 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:10.017 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:10.017 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-09-13T11:26:10.017 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.317114+0000", 2024-09-13T11:26:10.018 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.4569999999999999e-06 2024-09-13T11:26:10.018 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:10.018 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:10.018 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:10.018 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.190017+0000", 2024-09-13T11:26:10.018 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.87290258499999995 2024-09-13T11:26:10.018 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:10.018 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:10.018 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:26:10.018 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.190138+0000", 2024-09-13T11:26:10.018 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000121418 2024-09-13T11:26:10.018 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:10.018 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:10.019 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1", 2024-09-13T11:26:10.019 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.190216+0000", 2024-09-13T11:26:10.019 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.7410999999999995e-05 2024-09-13T11:26:10.019 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:10.019 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:10.019 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:10.019 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.191755+0000", 2024-09-13T11:26:10.019 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0015395420000000001 2024-09-13T11:26:10.019 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:10.019 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:10.019 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:26:10.019 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.191843+0000", 2024-09-13T11:26:10.019 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.8022999999999994e-05 2024-09-13T11:26:10.019 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:10.020 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:10.020 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:26:10.020 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.191876+0000", 2024-09-13T11:26:10.020 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.2619000000000001e-05 2024-09-13T11:26:10.020 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:10.020 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:10.020 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:26:10.020 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.191905+0000", 2024-09-13T11:26:10.020 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.9635999999999999e-05 2024-09-13T11:26:10.020 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:10.021 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:10.021 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:10.021 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:10.021 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:10.021 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:187 3.b 3:da03c8dd:::benchmark_data_smithi177_31116_object186:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T11:26:10.021 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:48.316083+0000", 2024-09-13T11:26:10.021 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 201.59851468299999, 2024-09-13T11:26:10.021 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.87835499500000003, 2024-09-13T11:26:10.021 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:26:10.021 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:26:10.021 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:10.021 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:26:10.021 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:26:10.022 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:10.022 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 187 2024-09-13T11:26:10.022 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:10.022 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:26:10.022 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:10.022 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:26:10.022 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316083+0000", 2024-09-13T11:26:10.022 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:10.022 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:10.022 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:10.023 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:26:10.023 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316083+0000", 2024-09-13T11:26:10.023 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:10.023 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:10.023 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:10.023 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:26:10.023 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316086+0000", 2024-09-13T11:26:10.023 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.5560000000000002e-06 2024-09-13T11:26:10.023 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:10.023 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:10.023 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:26:10.023 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316199+0000", 2024-09-13T11:26:10.023 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000113522 2024-09-13T11:26:10.023 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:10.023 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:10.024 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:26:10.024 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316215+0000", 2024-09-13T11:26:10.024 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.5265999999999999e-05 2024-09-13T11:26:10.024 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:10.024 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:10.024 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:26:10.024 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316226+0000", 2024-09-13T11:26:10.024 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0967e-05 2024-09-13T11:26:10.024 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:10.024 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:10.024 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:10.024 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.317198+0000", 2024-09-13T11:26:10.024 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00097194100000000004 2024-09-13T11:26:10.025 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:10.025 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:10.025 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-09-13T11:26:10.025 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.317213+0000", 2024-09-13T11:26:10.025 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.5472999999999999e-05 2024-09-13T11:26:10.025 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:10.025 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:10.025 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:10.025 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.191077+0000", 2024-09-13T11:26:10.025 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.87386435200000001 2024-09-13T11:26:10.025 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:10.025 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:10.025 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:26:10.025 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.191181+0000", 2024-09-13T11:26:10.026 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000103618 2024-09-13T11:26:10.026 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:10.026 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:10.026 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1", 2024-09-13T11:26:10.026 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.191305+0000", 2024-09-13T11:26:10.026 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00012358599999999999 2024-09-13T11:26:10.026 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:10.026 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:10.026 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:10.026 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.194279+0000", 2024-09-13T11:26:10.026 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0029746970000000001 2024-09-13T11:26:10.026 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:10.026 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:10.026 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:26:10.026 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.194336+0000", 2024-09-13T11:26:10.027 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.7244000000000003e-05 2024-09-13T11:26:10.027 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:10.027 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:10.027 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:26:10.027 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.194397+0000", 2024-09-13T11:26:10.027 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.0278000000000001e-05 2024-09-13T11:26:10.027 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:10.027 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:10.027 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:26:10.027 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.194438+0000", 2024-09-13T11:26:10.027 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.1495000000000002e-05 2024-09-13T11:26:10.027 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:10.027 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:10.027 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:10.028 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:10.028 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:10.028 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:200 3.b 3:d90039f1:::benchmark_data_smithi177_31116_object199:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T11:26:10.028 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:48.316280+0000", 2024-09-13T11:26:10.028 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 201.598318218, 2024-09-13T11:26:10.028 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.87879273000000002, 2024-09-13T11:26:10.028 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:26:10.028 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:26:10.028 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:10.028 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:26:10.028 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:26:10.028 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:10.028 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 200 2024-09-13T11:26:10.028 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:10.029 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:26:10.029 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:10.029 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:26:10.029 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316280+0000", 2024-09-13T11:26:10.029 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:10.029 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:10.029 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:10.029 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:26:10.029 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316280+0000", 2024-09-13T11:26:10.029 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:10.029 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:10.029 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:10.029 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:26:10.029 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316282+0000", 2024-09-13T11:26:10.030 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.497e-06 2024-09-13T11:26:10.030 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:10.030 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:10.030 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:26:10.030 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316411+0000", 2024-09-13T11:26:10.030 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00012924799999999999 2024-09-13T11:26:10.030 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:10.030 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:10.030 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:26:10.030 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316426+0000", 2024-09-13T11:26:10.030 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.4183e-05 2024-09-13T11:26:10.030 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:10.030 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:10.030 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:26:10.031 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316435+0000", 2024-09-13T11:26:10.031 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.0990000000000005e-06 2024-09-13T11:26:10.031 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:10.031 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:10.031 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:10.031 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.317250+0000", 2024-09-13T11:26:10.031 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00081501100000000003 2024-09-13T11:26:10.031 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:10.031 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:10.031 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-09-13T11:26:10.031 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.317257+0000", 2024-09-13T11:26:10.031 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.2169999999999997e-06 2024-09-13T11:26:10.031 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:10.031 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:10.032 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:10.032 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.191936+0000", 2024-09-13T11:26:10.032 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.87467922099999995 2024-09-13T11:26:10.032 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:10.032 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:10.032 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:26:10.032 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.192056+0000", 2024-09-13T11:26:10.032 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000119638 2024-09-13T11:26:10.032 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:10.032 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:10.032 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1", 2024-09-13T11:26:10.032 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.192184+0000", 2024-09-13T11:26:10.032 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00012813399999999999 2024-09-13T11:26:10.032 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:10.033 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:10.033 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:10.033 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.194324+0000", 2024-09-13T11:26:10.033 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0021399420000000001 2024-09-13T11:26:10.033 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:10.033 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:10.033 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:26:10.033 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.194802+0000", 2024-09-13T11:26:10.033 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00047798799999999999 2024-09-13T11:26:10.033 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:10.033 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:10.033 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:26:10.033 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.194951+0000", 2024-09-13T11:26:10.033 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0001494 2024-09-13T11:26:10.034 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:10.034 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:10.034 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:26:10.034 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.195072+0000", 2024-09-13T11:26:10.034 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00012115199999999999 2024-09-13T11:26:10.034 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:10.034 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:10.034 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:10.034 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:10.034 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:10.034 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop(client.4665.0:79801 3.5 e102/67 3:a6efdf84:::benchmark_data_smithi177_31116_object79800:head v 102'3851, mlcod=102'3848)", 2024-09-13T11:26:10.034 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:24:00.403754+0000", 2024-09-13T11:26:10.034 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 129.51084386599999, 2024-09-13T11:26:10.034 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.31749079899999999, 2024-09-13T11:26:10.034 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:26:10.035 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:26:10.035 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:10.035 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:26:10.035 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:10.035 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:26:10.035 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:00.403754+0000", 2024-09-13T11:26:10.035 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:10.035 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:10.035 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:10.035 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:26:10.035 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:00.403754+0000", 2024-09-13T11:26:10.035 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:10.035 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:10.035 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:10.036 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:26:10.036 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:00.403758+0000", 2024-09-13T11:26:10.036 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.507e-06 2024-09-13T11:26:10.036 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:10.036 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:10.036 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:26:10.036 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:00.403819+0000", 2024-09-13T11:26:10.036 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.1042000000000003e-05 2024-09-13T11:26:10.036 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:10.036 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:10.036 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:26:10.036 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:00.717963+0000", 2024-09-13T11:26:10.036 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.314144594 2024-09-13T11:26:10.036 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:10.037 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:10.037 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:26:10.037 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:00.717976+0000", 2024-09-13T11:26:10.037 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.3029e-05 2024-09-13T11:26:10.037 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:10.037 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:10.037 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:10.037 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:00.718143+0000", 2024-09-13T11:26:10.037 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00016639000000000001 2024-09-13T11:26:10.037 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:10.037 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:10.037 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:26:10.037 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:00.718184+0000", 2024-09-13T11:26:10.037 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.1801999999999997e-05 2024-09-13T11:26:10.038 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:10.038 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:10.038 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:26:10.038 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:00.721162+0000", 2024-09-13T11:26:10.038 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0029772800000000001 2024-09-13T11:26:10.038 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:10.038 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:10.038 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:26:10.038 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:00.721245+0000", 2024-09-13T11:26:10.038 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.3154999999999999e-05 2024-09-13T11:26:10.038 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:10.038 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:10.038 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:10.038 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:10.038 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:10.039 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "MOSDPGPushReply(3.26 121/121 [PushReplyOp(3:670b9aab:::benchmark_data_smithi177_31116_object93:head)])", 2024-09-13T11:26:10.039 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:24:27.300318+0000", 2024-09-13T11:26:10.039 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 102.614279998, 2024-09-13T11:26:10.039 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.811436728, 2024-09-13T11:26:10.039 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:26:10.039 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:26:10.039 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "reached pg", 2024-09-13T11:26:10.039 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:26:10.039 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:10.039 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:26:10.039 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:27.300318+0000", 2024-09-13T11:26:10.039 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:10.039 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:10.039 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:10.040 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:26:10.040 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:27.300318+0000", 2024-09-13T11:26:10.040 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:10.040 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:10.040 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:10.040 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:26:10.040 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:27.300320+0000", 2024-09-13T11:26:10.040 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.778e-06 2024-09-13T11:26:10.040 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:10.040 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:10.040 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:26:10.040 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:27.300326+0000", 2024-09-13T11:26:10.040 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.7700000000000004e-06 2024-09-13T11:26:10.040 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:10.041 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:10.041 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:26:10.041 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:28.109665+0000", 2024-09-13T11:26:10.041 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.80933853499999997 2024-09-13T11:26:10.041 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:10.041 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:10.041 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:26:10.041 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:28.109678+0000", 2024-09-13T11:26:10.041 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.2870999999999999e-05 2024-09-13T11:26:10.041 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:10.041 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:10.041 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:10.041 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:28.109800+0000", 2024-09-13T11:26:10.041 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000122328 2024-09-13T11:26:10.042 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:10.042 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:10.042 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:26:10.042 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:28.111755+0000", 2024-09-13T11:26:10.042 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0019544459999999999 2024-09-13T11:26:10.042 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:10.042 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:10.042 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:10.042 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:10.042 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:10.042 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:184026 3.b 3:d3052347:::benchmark_data_smithi177_31116_object43723:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e171)", 2024-09-13T11:26:10.042 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:25:32.645242+0000", 2024-09-13T11:26:10.042 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 37.269355957999998, 2024-09-13T11:26:10.042 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.348925019, 2024-09-13T11:26:10.043 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:26:10.043 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:26:10.043 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:10.043 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:26:10.043 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:26:10.043 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:10.043 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 184026 2024-09-13T11:26:10.043 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:10.043 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:26:10.043 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:10.043 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:26:10.043 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:32.645242+0000", 2024-09-13T11:26:10.043 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:10.043 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:10.043 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:10.044 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:26:10.044 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:32.645242+0000", 2024-09-13T11:26:10.044 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:10.044 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:10.044 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:10.044 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:26:10.044 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:32.645244+0000", 2024-09-13T11:26:10.044 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.368e-06 2024-09-13T11:26:10.044 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:10.044 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:10.044 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:26:10.044 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:32.645251+0000", 2024-09-13T11:26:10.044 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.2519999999999996e-06 2024-09-13T11:26:10.045 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:10.045 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:10.045 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:26:10.045 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:32.645267+0000", 2024-09-13T11:26:10.045 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.6529e-05 2024-09-13T11:26:10.045 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:10.045 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:10.045 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:26:10.045 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:32.645278+0000", 2024-09-13T11:26:10.045 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.119e-05 2024-09-13T11:26:10.045 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:10.045 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:10.045 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:10.045 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:32.645348+0000", 2024-09-13T11:26:10.046 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.9333999999999997e-05 2024-09-13T11:26:10.046 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:10.046 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:10.046 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:26:10.046 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:32.645479+0000", 2024-09-13T11:26:10.046 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00013153199999999999 2024-09-13T11:26:10.046 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:10.046 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:10.046 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1", 2024-09-13T11:26:10.046 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:32.645599+0000", 2024-09-13T11:26:10.046 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000119481 2024-09-13T11:26:10.046 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:10.046 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:10.046 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:26:10.046 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:32.648042+0000", 2024-09-13T11:26:10.047 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0024437059999999999 2024-09-13T11:26:10.047 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:10.047 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:10.047 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:10.047 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:32.994082+0000", 2024-09-13T11:26:10.047 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.34604006399999998 2024-09-13T11:26:10.047 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:10.047 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:10.047 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:26:10.047 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:32.994124+0000", 2024-09-13T11:26:10.047 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.1573999999999999e-05 2024-09-13T11:26:10.047 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:10.047 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:10.047 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:26:10.048 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:32.994167+0000", 2024-09-13T11:26:10.048 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.2988999999999998e-05 2024-09-13T11:26:10.048 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:10.048 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:10.048 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:10.048 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:10.048 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:10.048 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-09-13T11:26:10.049 INFO:tasks.ceph.osd.11.smithi177.stderr:2024-09-13T11:26:10.045+0000 7f24792c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26580) UID: 0 2024-09-13T11:26:10.050 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-13T11:26:10.081 INFO:teuthology.orchestra.run.smithi001.stderr:set pool 3 pgp_num to 46 2024-09-13T11:26:10.082 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-13T11:26:10.083 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-09-13T11:26:10.079+0000 7f97b59fa640 -1 osd.6 195 handle_osd_map injecting map crc failure 2024-09-13T11:26:10.150 INFO:tasks.ceph.osd.14.smithi162.stderr:2024-09-13T11:26:10.147+0000 7f3df5903640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33154) UID: 0 2024-09-13T11:26:10.196 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-13T11:26:10.197 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [], 2024-09-13T11:26:10.197 INFO:teuthology.orchestra.run.smithi162.stdout: "num_ops": 0 2024-09-13T11:26:10.197 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-13T11:26:10.217 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-13T11:26:10.251 INFO:tasks.ceph.osd.4.smithi001.stderr:2024-09-13T11:26:10.250+0000 7ff8e0218640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26310) UID: 0 2024-09-13T11:26:10.352 INFO:tasks.ceph.osd.15.smithi177.stderr:2024-09-13T11:26:10.349+0000 7f9d6ce77640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26578) UID: 0 2024-09-13T11:26:10.397 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-13T11:26:10.397 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [], 2024-09-13T11:26:10.397 INFO:teuthology.orchestra.run.smithi162.stdout: "complaint_time": 30, 2024-09-13T11:26:10.397 INFO:teuthology.orchestra.run.smithi162.stdout: "num_blocked_ops": 0 2024-09-13T11:26:10.397 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-13T11:26:10.415 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-13T11:26:10.461 INFO:tasks.ceph.osd.14.smithi162.stderr:2024-09-13T11:26:10.459+0000 7f3df5903640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33154) UID: 0 2024-09-13T11:26:10.554 INFO:tasks.ceph.osd.4.smithi001.stderr:2024-09-13T11:26:10.550+0000 7ff8e0218640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26310) UID: 0 2024-09-13T11:26:10.556 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-13T11:26:10.556 INFO:teuthology.orchestra.run.smithi162.stdout: "size": 20, 2024-09-13T11:26:10.556 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 600, 2024-09-13T11:26:10.556 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [ 2024-09-13T11:26:10.556 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:10.556 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.4665.0:171 3.9 e41/30 3:9e1c7eeb:::benchmark_data_smithi177_31116_object170:head v 41'13, mlcod=41'9)", 2024-09-13T11:26:10.556 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:22:47.418602+0000", 2024-09-13T11:26:10.556 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 203.13619853700001, 2024-09-13T11:26:10.556 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.67235367300000004, 2024-09-13T11:26:10.556 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:26:10.556 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:26:10.557 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:10.557 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:26:10.557 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:10.557 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:26:10.557 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:47.418602+0000", 2024-09-13T11:26:10.557 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:10.557 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:10.557 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:10.557 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:26:10.557 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:47.418602+0000", 2024-09-13T11:26:10.557 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:10.557 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:10.557 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:10.557 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:26:10.558 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:47.418605+0000", 2024-09-13T11:26:10.558 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.2200000000000001e-06 2024-09-13T11:26:10.558 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:10.558 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:10.558 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:26:10.558 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:47.418668+0000", 2024-09-13T11:26:10.558 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.2921000000000003e-05 2024-09-13T11:26:10.558 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:10.558 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:10.558 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:26:10.558 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:48.089380+0000", 2024-09-13T11:26:10.558 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.67071191100000005 2024-09-13T11:26:10.558 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:10.559 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:10.559 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:26:10.559 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:48.089423+0000", 2024-09-13T11:26:10.559 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.2916e-05 2024-09-13T11:26:10.559 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:10.559 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:10.559 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:26:10.559 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:48.089533+0000", 2024-09-13T11:26:10.559 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000109996 2024-09-13T11:26:10.559 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:10.559 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:10.559 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:26:10.559 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:48.089570+0000", 2024-09-13T11:26:10.560 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.7314999999999998e-05 2024-09-13T11:26:10.560 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:10.560 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:10.560 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:26:10.560 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:48.090852+0000", 2024-09-13T11:26:10.560 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001281641 2024-09-13T11:26:10.560 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:10.560 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:10.560 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:26:10.560 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:48.090956+0000", 2024-09-13T11:26:10.560 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00010375299999999999 2024-09-13T11:26:10.560 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:10.560 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:26:10.560 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:10.561 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:10.561 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:10.561 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPushReply(3.19 74/70 [PushReplyOp(3:9edf939c:::benchmark_data_smithi177_31116_object17207:head)])", 2024-09-13T11:26:10.561 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:30.897971+0000", 2024-09-13T11:26:10.561 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 159.65682968600001, 2024-09-13T11:26:10.561 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.40616071100000001, 2024-09-13T11:26:10.561 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:26:10.561 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:26:10.562 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-09-13T11:26:10.562 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:26:10.562 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:10.562 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:26:10.562 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.897971+0000", 2024-09-13T11:26:10.562 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:10.562 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:10.562 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:10.562 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:26:10.562 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.897971+0000", 2024-09-13T11:26:10.562 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:10.562 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:10.562 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:10.563 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:26:10.563 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.897972+0000", 2024-09-13T11:26:10.563 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1429999999999999e-06 2024-09-13T11:26:10.563 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:10.563 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:10.563 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:26:10.563 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.897977+0000", 2024-09-13T11:26:10.563 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.5199999999999999e-06 2024-09-13T11:26:10.563 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:10.563 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:10.563 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:26:10.563 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:31.302500+0000", 2024-09-13T11:26:10.563 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.40452312400000001 2024-09-13T11:26:10.563 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:10.564 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:10.564 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:26:10.564 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:31.302513+0000", 2024-09-13T11:26:10.564 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2753e-05 2024-09-13T11:26:10.564 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:10.564 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:10.564 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:26:10.564 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:31.302645+0000", 2024-09-13T11:26:10.564 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00013292100000000001 2024-09-13T11:26:10.564 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:10.564 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:10.564 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:26:10.564 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:31.304132+0000", 2024-09-13T11:26:10.565 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0014862499999999999 2024-09-13T11:26:10.565 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:10.565 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:26:10.565 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:10.565 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:10.565 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:10.565 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPushReply(3.19 75/70 [PushReplyOp(3:9d1489b1:::benchmark_data_smithi177_31116_object18698:head)])", 2024-09-13T11:26:10.565 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:31.697212+0000", 2024-09-13T11:26:10.565 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 158.857588685, 2024-09-13T11:26:10.565 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.22850633200000001, 2024-09-13T11:26:10.565 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:26:10.565 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:26:10.565 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-09-13T11:26:10.565 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:26:10.566 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:10.566 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:26:10.566 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:31.697212+0000", 2024-09-13T11:26:10.566 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:10.566 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:10.566 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:10.566 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:26:10.566 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:31.697212+0000", 2024-09-13T11:26:10.566 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:10.566 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:10.566 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:10.566 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:26:10.566 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:31.697214+0000", 2024-09-13T11:26:10.567 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.7439999999999999e-06 2024-09-13T11:26:10.567 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:10.567 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:10.567 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:26:10.567 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:31.697224+0000", 2024-09-13T11:26:10.567 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0173000000000001e-05 2024-09-13T11:26:10.567 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:10.567 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:10.567 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:26:10.567 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:31.923931+0000", 2024-09-13T11:26:10.567 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.226706926 2024-09-13T11:26:10.567 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:10.567 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:10.568 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:26:10.568 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:31.923950+0000", 2024-09-13T11:26:10.568 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.8899e-05 2024-09-13T11:26:10.568 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:10.568 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:10.568 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:26:10.568 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:31.924449+0000", 2024-09-13T11:26:10.568 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00049907800000000002 2024-09-13T11:26:10.568 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:10.568 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:10.568 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:26:10.568 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:31.925718+0000", 2024-09-13T11:26:10.568 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001269512 2024-09-13T11:26:10.568 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:10.569 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:26:10.569 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:10.569 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:10.569 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:10.569 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPushReply(3.19 76/70 [PushReplyOp(3:9faf6d5f:::benchmark_data_smithi177_31116_object18954:head)])", 2024-09-13T11:26:10.569 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:32.042874+0000", 2024-09-13T11:26:10.569 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 158.511926816, 2024-09-13T11:26:10.569 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.40212404299999999, 2024-09-13T11:26:10.569 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:26:10.569 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:26:10.569 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-09-13T11:26:10.569 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:26:10.569 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:10.570 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:26:10.570 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.042874+0000", 2024-09-13T11:26:10.570 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:10.570 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:10.570 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:10.570 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:26:10.570 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.042874+0000", 2024-09-13T11:26:10.570 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:10.570 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:10.570 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:10.570 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:26:10.570 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.042875+0000", 2024-09-13T11:26:10.570 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.432e-06 2024-09-13T11:26:10.570 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:10.571 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:10.571 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:26:10.571 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.042886+0000", 2024-09-13T11:26:10.571 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0265000000000001e-05 2024-09-13T11:26:10.571 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:10.571 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:10.571 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:26:10.571 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.443455+0000", 2024-09-13T11:26:10.571 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.40056951400000002 2024-09-13T11:26:10.571 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:10.572 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:10.572 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:26:10.572 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.443480+0000", 2024-09-13T11:26:10.572 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.4929000000000001e-05 2024-09-13T11:26:10.572 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:10.572 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:10.572 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:26:10.572 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.443666+0000", 2024-09-13T11:26:10.572 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000186446 2024-09-13T11:26:10.572 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:10.572 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:10.572 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:26:10.573 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.444998+0000", 2024-09-13T11:26:10.573 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0013314570000000001 2024-09-13T11:26:10.573 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:10.573 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:26:10.573 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:10.573 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:10.573 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:10.573 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPush(3.1d 85/78 [PushOp(3:bc4b9336:::benchmark_data_smithi177_31116_object11944:head, version: 49'734, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:bc4b9336:::benchmark_data_smithi177_31116_object11944:head@49'734, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:26:10.574 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:42.225966+0000", 2024-09-13T11:26:10.574 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 148.32883460299999, 2024-09-13T11:26:10.574 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.23879947800000001, 2024-09-13T11:26:10.574 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:26:10.574 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:26:10.574 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:26:10.574 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:26:10.574 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:10.574 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:26:10.574 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.225966+0000", 2024-09-13T11:26:10.574 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:10.574 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:10.574 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:10.574 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:26:10.575 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.225966+0000", 2024-09-13T11:26:10.575 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:10.575 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:10.575 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:10.575 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:26:10.575 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.225969+0000", 2024-09-13T11:26:10.575 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.0400000000000001e-06 2024-09-13T11:26:10.575 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:10.575 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:10.575 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:26:10.575 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.226036+0000", 2024-09-13T11:26:10.575 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.6978999999999997e-05 2024-09-13T11:26:10.575 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:10.576 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:10.576 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:26:10.576 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.463110+0000", 2024-09-13T11:26:10.576 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.23707400000000001 2024-09-13T11:26:10.576 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:10.576 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:10.576 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:26:10.576 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.463140+0000", 2024-09-13T11:26:10.576 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.9575999999999999e-05 2024-09-13T11:26:10.576 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:10.576 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:10.576 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:26:10.576 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.463326+0000", 2024-09-13T11:26:10.577 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000186298 2024-09-13T11:26:10.577 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:10.577 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:10.577 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:26:10.577 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.463351+0000", 2024-09-13T11:26:10.577 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.4862999999999999e-05 2024-09-13T11:26:10.577 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:10.577 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:10.577 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:26:10.577 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.464766+0000", 2024-09-13T11:26:10.577 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001414722 2024-09-13T11:26:10.577 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:10.577 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:26:10.577 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:10.578 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:10.578 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:10.578 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPush(3.1d 89/78 [PushOp(3:bc1b7cfb:::benchmark_data_smithi177_31116_object22629:head, version: 55'1387, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:bc1b7cfb:::benchmark_data_smithi177_31116_object22629:head@55'1387, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:26:10.578 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:46.660855+0000", 2024-09-13T11:26:10.578 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 143.893946107, 2024-09-13T11:26:10.578 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.26622785599999998, 2024-09-13T11:26:10.578 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:26:10.578 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:26:10.578 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:26:10.578 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:26:10.578 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:10.578 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:26:10.578 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:46.660855+0000", 2024-09-13T11:26:10.579 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:10.579 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:10.579 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:10.579 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:26:10.579 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:46.660855+0000", 2024-09-13T11:26:10.579 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:10.579 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:10.579 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:10.579 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:26:10.579 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:46.660857+0000", 2024-09-13T11:26:10.580 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.6560000000000001e-06 2024-09-13T11:26:10.580 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:10.580 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:10.580 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:26:10.580 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:46.660931+0000", 2024-09-13T11:26:10.580 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.4041999999999993e-05 2024-09-13T11:26:10.580 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:10.580 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:10.580 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:26:10.580 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:46.924654+0000", 2024-09-13T11:26:10.580 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.26372239800000002 2024-09-13T11:26:10.580 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:10.580 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:10.581 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:26:10.581 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:46.924729+0000", 2024-09-13T11:26:10.581 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.5061999999999994e-05 2024-09-13T11:26:10.581 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:10.581 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:10.581 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:26:10.581 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:46.924986+0000", 2024-09-13T11:26:10.581 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00025762000000000002 2024-09-13T11:26:10.581 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:10.581 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:10.582 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:26:10.582 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:46.925013+0000", 2024-09-13T11:26:10.582 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.6179000000000001e-05 2024-09-13T11:26:10.582 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:10.582 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:10.582 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:26:10.582 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:46.927082+0000", 2024-09-13T11:26:10.582 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.002069899 2024-09-13T11:26:10.582 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:10.582 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:26:10.582 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:10.582 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:10.582 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:10.583 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPush(3.1d 90/78 [PushOp(3:bc0e7228:::benchmark_data_smithi177_31116_object24733:head, version: 55'1512, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:bc0e7228:::benchmark_data_smithi177_31116_object24733:head@55'1512, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:26:10.583 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:47.529126+0000", 2024-09-13T11:26:10.583 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 143.02567513299999, 2024-09-13T11:26:10.583 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.21343989499999999, 2024-09-13T11:26:10.583 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:26:10.583 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:26:10.583 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:26:10.583 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:26:10.583 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:10.583 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:26:10.583 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:47.529126+0000", 2024-09-13T11:26:10.583 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:10.583 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:10.583 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:10.584 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:26:10.584 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:47.529126+0000", 2024-09-13T11:26:10.584 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:10.584 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:10.584 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:10.584 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:26:10.584 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:47.529127+0000", 2024-09-13T11:26:10.584 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.252e-06 2024-09-13T11:26:10.584 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:10.584 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:10.584 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:26:10.584 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:47.529185+0000", 2024-09-13T11:26:10.584 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.8224999999999999e-05 2024-09-13T11:26:10.585 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:10.585 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:10.585 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:26:10.585 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:47.740216+0000", 2024-09-13T11:26:10.585 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.21103060800000001 2024-09-13T11:26:10.585 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:10.585 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:10.585 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:26:10.585 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:47.740248+0000", 2024-09-13T11:26:10.585 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.2821999999999998e-05 2024-09-13T11:26:10.585 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:10.585 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:10.585 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:26:10.586 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:47.740504+0000", 2024-09-13T11:26:10.586 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00025518399999999999 2024-09-13T11:26:10.586 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:10.586 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:10.586 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:26:10.586 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:47.740544+0000", 2024-09-13T11:26:10.586 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.0438999999999997e-05 2024-09-13T11:26:10.586 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:10.586 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:10.586 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:26:10.587 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:47.742565+0000", 2024-09-13T11:26:10.587 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0020213649999999998 2024-09-13T11:26:10.587 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:10.587 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:26:10.587 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:10.587 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:10.587 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:10.587 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:83199 3.1d 3:be2c6fae:::benchmark_data_smithi177_31116_object83198:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106)", 2024-09-13T11:26:10.587 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:24:04.297306+0000", 2024-09-13T11:26:10.587 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 126.257495181, 2024-09-13T11:26:10.587 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.25114587999999999, 2024-09-13T11:26:10.587 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:26:10.587 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:26:10.588 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:10.588 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:26:10.588 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:26:10.588 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:10.588 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 83199 2024-09-13T11:26:10.588 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:10.588 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:26:10.588 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:10.588 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:26:10.588 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.297306+0000", 2024-09-13T11:26:10.588 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:10.588 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:10.588 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:10.588 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:26:10.589 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.297306+0000", 2024-09-13T11:26:10.589 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:10.589 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:10.589 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:10.589 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:26:10.589 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.297308+0000", 2024-09-13T11:26:10.589 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.4150000000000002e-06 2024-09-13T11:26:10.589 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:10.589 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:10.589 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:26:10.589 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.297352+0000", 2024-09-13T11:26:10.589 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.4431000000000002e-05 2024-09-13T11:26:10.589 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:10.589 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:10.590 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:26:10.590 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.297369+0000", 2024-09-13T11:26:10.590 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.6685e-05 2024-09-13T11:26:10.590 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:10.590 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:10.590 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:26:10.590 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.297376+0000", 2024-09-13T11:26:10.590 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.7299999999999999e-06 2024-09-13T11:26:10.590 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:10.590 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:10.590 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:26:10.590 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.297427+0000", 2024-09-13T11:26:10.590 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.1461000000000001e-05 2024-09-13T11:26:10.590 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:10.591 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:10.591 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:26:10.591 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.297528+0000", 2024-09-13T11:26:10.591 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00010028599999999999 2024-09-13T11:26:10.591 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:10.591 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:10.591 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 15", 2024-09-13T11:26:10.591 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.297685+0000", 2024-09-13T11:26:10.591 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00015760500000000001 2024-09-13T11:26:10.591 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:10.591 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:10.591 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:26:10.591 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.300779+0000", 2024-09-13T11:26:10.591 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0030936700000000002 2024-09-13T11:26:10.592 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:10.592 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:10.592 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:10.592 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.548279+0000", 2024-09-13T11:26:10.592 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.24750059999999999 2024-09-13T11:26:10.592 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:10.592 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:10.592 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:26:10.593 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.548373+0000", 2024-09-13T11:26:10.593 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.3451999999999999e-05 2024-09-13T11:26:10.593 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:10.593 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:10.593 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:26:10.593 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.548451+0000", 2024-09-13T11:26:10.593 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.8545000000000001e-05 2024-09-13T11:26:10.593 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:10.593 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:26:10.593 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:10.593 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:10.593 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:10.593 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:84621 3.1d 3:becd72aa:::benchmark_data_smithi177_31116_object84620:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-09-13T11:26:10.593 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:24:05.741008+0000", 2024-09-13T11:26:10.594 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 124.81379318099999, 2024-09-13T11:26:10.594 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.17994101100000001, 2024-09-13T11:26:10.594 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:26:10.594 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:26:10.594 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:10.594 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:26:10.594 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:26:10.594 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:10.594 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 84621 2024-09-13T11:26:10.594 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:10.594 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:26:10.594 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:10.594 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:26:10.595 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.741008+0000", 2024-09-13T11:26:10.595 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:10.595 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:10.595 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:10.595 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:26:10.595 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.741008+0000", 2024-09-13T11:26:10.595 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:10.595 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:10.595 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:10.595 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:26:10.595 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.741010+0000", 2024-09-13T11:26:10.595 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.898e-06 2024-09-13T11:26:10.595 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:10.595 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:10.596 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:26:10.596 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.741061+0000", 2024-09-13T11:26:10.596 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.0898e-05 2024-09-13T11:26:10.596 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:10.596 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:10.596 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:26:10.596 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.741085+0000", 2024-09-13T11:26:10.596 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.3558e-05 2024-09-13T11:26:10.596 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:10.596 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:10.596 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:26:10.596 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.741093+0000", 2024-09-13T11:26:10.596 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.3890000000000005e-06 2024-09-13T11:26:10.596 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:10.597 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:10.597 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:26:10.597 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.741178+0000", 2024-09-13T11:26:10.597 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.4792999999999999e-05 2024-09-13T11:26:10.597 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:10.597 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:10.597 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:26:10.597 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.741386+0000", 2024-09-13T11:26:10.597 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000207505 2024-09-13T11:26:10.597 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:10.597 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:10.597 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 15", 2024-09-13T11:26:10.597 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.741597+0000", 2024-09-13T11:26:10.598 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00021158200000000001 2024-09-13T11:26:10.598 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:10.598 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:10.598 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:26:10.598 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.743392+0000", 2024-09-13T11:26:10.598 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00179518 2024-09-13T11:26:10.598 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:10.598 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:10.598 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:10.598 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.920866+0000", 2024-09-13T11:26:10.598 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.17747373699999999 2024-09-13T11:26:10.598 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:10.598 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:10.598 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:26:10.599 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.920913+0000", 2024-09-13T11:26:10.599 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.6451e-05 2024-09-13T11:26:10.599 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:10.599 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:10.599 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:26:10.599 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.920949+0000", 2024-09-13T11:26:10.599 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.6019999999999997e-05 2024-09-13T11:26:10.599 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:10.599 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:26:10.599 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:10.599 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:10.599 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:10.599 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPush(3.25 124/120 [PushOp(3:a6d4ee3f:::benchmark_data_smithi177_31116_object14712:head, version: 51'872, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:a6d4ee3f:::benchmark_data_smithi177_31116_object14712:head@51'872, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:26:10.599 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:24:31.996253+0000", 2024-09-13T11:26:10.600 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 98.558547408999999, 2024-09-13T11:26:10.600 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.17694863499999999, 2024-09-13T11:26:10.600 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:26:10.600 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:26:10.600 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:26:10.600 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:26:10.600 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:10.600 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:26:10.600 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:31.996253+0000", 2024-09-13T11:26:10.600 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:10.600 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:10.600 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:10.600 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:26:10.600 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:31.996253+0000", 2024-09-13T11:26:10.601 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:10.601 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:10.601 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:10.601 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:26:10.601 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:31.996257+0000", 2024-09-13T11:26:10.601 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.2629999999999999e-06 2024-09-13T11:26:10.601 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:10.601 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:10.601 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:26:10.601 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:31.996359+0000", 2024-09-13T11:26:10.601 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000102456 2024-09-13T11:26:10.601 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:10.601 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:10.601 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:26:10.602 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:32.172313+0000", 2024-09-13T11:26:10.602 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.17595388000000001 2024-09-13T11:26:10.602 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:10.602 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:10.602 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:26:10.602 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:32.172326+0000", 2024-09-13T11:26:10.602 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2632999999999999e-05 2024-09-13T11:26:10.603 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:10.603 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:10.603 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:26:10.603 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:32.172423+0000", 2024-09-13T11:26:10.603 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.7084999999999996e-05 2024-09-13T11:26:10.603 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:10.603 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:10.603 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:26:10.603 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:32.172438+0000", 2024-09-13T11:26:10.603 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5775999999999999e-05 2024-09-13T11:26:10.603 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:10.603 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:10.604 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:26:10.604 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:32.173202+0000", 2024-09-13T11:26:10.604 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00076354199999999998 2024-09-13T11:26:10.604 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:10.604 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:26:10.604 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:10.604 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:10.604 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:10.604 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:126039 3.29 3:96100181:::benchmark_data_smithi177_31116_object126038:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e130)", 2024-09-13T11:26:10.604 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:24:43.012785+0000", 2024-09-13T11:26:10.604 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 87.542016059000005, 2024-09-13T11:26:10.604 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.26685311900000003, 2024-09-13T11:26:10.605 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:26:10.605 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:26:10.605 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:10.605 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:26:10.605 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:26:10.605 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:10.605 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 126039 2024-09-13T11:26:10.605 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:10.605 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:26:10.605 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:10.605 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:26:10.605 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:43.012785+0000", 2024-09-13T11:26:10.605 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:10.606 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:10.606 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:10.606 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:26:10.606 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:43.012785+0000", 2024-09-13T11:26:10.606 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:10.606 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:10.606 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:10.606 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:26:10.606 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:43.012787+0000", 2024-09-13T11:26:10.608 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.875e-06 2024-09-13T11:26:10.608 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:10.608 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:10.608 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:26:10.608 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:43.012826+0000", 2024-09-13T11:26:10.608 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.9483999999999997e-05 2024-09-13T11:26:10.609 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:10.609 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:10.609 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:26:10.609 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:43.012839+0000", 2024-09-13T11:26:10.609 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3405e-05 2024-09-13T11:26:10.609 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:10.609 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:10.609 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:26:10.609 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:43.012845+0000", 2024-09-13T11:26:10.609 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.9100000000000002e-06 2024-09-13T11:26:10.609 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:10.609 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:10.609 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:26:10.609 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:43.012903+0000", 2024-09-13T11:26:10.610 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.8131000000000003e-05 2024-09-13T11:26:10.610 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:10.610 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:10.610 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:26:10.610 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:43.013066+0000", 2024-09-13T11:26:10.610 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000162058 2024-09-13T11:26:10.610 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:10.610 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:10.610 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 4,5", 2024-09-13T11:26:10.610 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:43.013235+0000", 2024-09-13T11:26:10.610 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00016929600000000001 2024-09-13T11:26:10.610 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:10.610 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:10.611 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:10.611 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:43.017063+0000", 2024-09-13T11:26:10.611 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0038283890000000002 2024-09-13T11:26:10.611 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:10.611 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:10.611 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:26:10.611 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:43.017086+0000", 2024-09-13T11:26:10.611 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.2934000000000001e-05 2024-09-13T11:26:10.611 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:10.611 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:10.611 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:10.611 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:43.278804+0000", 2024-09-13T11:26:10.611 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.26171751799999998 2024-09-13T11:26:10.611 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:10.612 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:10.612 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:26:10.612 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:43.278862+0000", 2024-09-13T11:26:10.612 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.8431999999999999e-05 2024-09-13T11:26:10.612 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:10.612 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:10.612 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:26:10.612 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:43.279638+0000", 2024-09-13T11:26:10.612 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00077568699999999995 2024-09-13T11:26:10.612 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:10.612 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:26:10.612 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:10.612 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:10.613 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:10.613 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop_reply(client.4665.0:126039 3.29 e130/120 ondisk, result = 0)", 2024-09-13T11:26:10.613 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:24:43.021691+0000", 2024-09-13T11:26:10.613 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 87.533110132999994, 2024-09-13T11:26:10.613 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.25795579699999999, 2024-09-13T11:26:10.613 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:26:10.613 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:26:10.613 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:26:10.613 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:26:10.613 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:10.613 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:26:10.613 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:43.021691+0000", 2024-09-13T11:26:10.613 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:10.613 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:10.614 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:10.614 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:26:10.614 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:43.021691+0000", 2024-09-13T11:26:10.614 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:10.614 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:10.614 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:10.614 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:26:10.614 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:43.021692+0000", 2024-09-13T11:26:10.614 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3880000000000001e-06 2024-09-13T11:26:10.614 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:10.614 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:10.614 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:26:10.614 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:43.021700+0000", 2024-09-13T11:26:10.615 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.9310000000000008e-06 2024-09-13T11:26:10.615 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:10.615 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:10.615 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:26:10.615 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:43.278719+0000", 2024-09-13T11:26:10.615 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.25701909299999998 2024-09-13T11:26:10.615 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:10.615 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:10.615 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:26:10.615 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:43.278728+0000", 2024-09-13T11:26:10.616 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.5000000000000005e-06 2024-09-13T11:26:10.616 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:10.616 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:10.616 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:26:10.616 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:43.278786+0000", 2024-09-13T11:26:10.616 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.7640999999999999e-05 2024-09-13T11:26:10.616 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:10.616 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:10.616 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:26:10.616 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:43.278796+0000", 2024-09-13T11:26:10.616 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.9199999999999999e-06 2024-09-13T11:26:10.616 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:10.617 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:10.617 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:26:10.617 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:43.279646+0000", 2024-09-13T11:26:10.617 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00085032399999999996 2024-09-13T11:26:10.617 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:10.617 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:26:10.617 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:10.617 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:10.617 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:10.617 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPushReply(3.29 134/120 [PushReplyOp(3:95648f22:::benchmark_data_smithi177_31116_object7446:head)])", 2024-09-13T11:26:10.617 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:24:47.110443+0000", 2024-09-13T11:26:10.618 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 83.444358012999999, 2024-09-13T11:26:10.618 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.171602386, 2024-09-13T11:26:10.618 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:26:10.618 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:26:10.618 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-09-13T11:26:10.618 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:26:10.618 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:10.618 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:26:10.618 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:47.110443+0000", 2024-09-13T11:26:10.618 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:10.618 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:10.618 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:10.618 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:26:10.619 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:47.110443+0000", 2024-09-13T11:26:10.619 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:10.619 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:10.619 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:10.619 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:26:10.619 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:47.110444+0000", 2024-09-13T11:26:10.619 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.192e-06 2024-09-13T11:26:10.619 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:10.619 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:10.619 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:26:10.619 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:47.110449+0000", 2024-09-13T11:26:10.619 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.0869999999999997e-06 2024-09-13T11:26:10.619 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:10.620 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:10.620 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:26:10.620 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:47.280321+0000", 2024-09-13T11:26:10.620 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.169872157 2024-09-13T11:26:10.620 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:10.620 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:10.620 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:26:10.620 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:47.280337+0000", 2024-09-13T11:26:10.620 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5656000000000001e-05 2024-09-13T11:26:10.620 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:10.620 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:10.620 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:26:10.620 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:47.280414+0000", 2024-09-13T11:26:10.621 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.7083999999999996e-05 2024-09-13T11:26:10.621 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:10.621 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:10.621 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:26:10.621 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:47.282045+0000", 2024-09-13T11:26:10.621 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0016312099999999999 2024-09-13T11:26:10.621 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:10.621 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:26:10.621 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:10.621 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:10.621 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:10.621 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.4665.0:138294 3.9 e140/117 3:90012dab:::benchmark_data_smithi177_31116_object138293:head v 140'5246, mlcod=140'5243)", 2024-09-13T11:26:10.621 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:24:54.252227+0000", 2024-09-13T11:26:10.621 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 76.302573644999995, 2024-09-13T11:26:10.622 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.32919836400000002, 2024-09-13T11:26:10.622 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:26:10.622 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:26:10.622 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:10.622 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:26:10.622 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:10.622 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:26:10.622 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:54.252227+0000", 2024-09-13T11:26:10.622 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:10.622 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:10.622 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:10.622 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:26:10.622 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:54.252227+0000", 2024-09-13T11:26:10.623 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:10.623 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:10.623 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:10.623 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:26:10.623 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:54.252228+0000", 2024-09-13T11:26:10.623 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.8899999999999998e-07 2024-09-13T11:26:10.623 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:10.623 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:10.623 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:26:10.623 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:54.252276+0000", 2024-09-13T11:26:10.623 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.8137999999999999e-05 2024-09-13T11:26:10.623 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:10.623 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:10.624 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:26:10.624 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:54.578216+0000", 2024-09-13T11:26:10.624 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.32594014399999999 2024-09-13T11:26:10.624 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:10.624 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:10.624 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:26:10.624 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:54.578235+0000", 2024-09-13T11:26:10.624 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.8943000000000001e-05 2024-09-13T11:26:10.624 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:10.624 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:10.624 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:26:10.624 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:54.578396+0000", 2024-09-13T11:26:10.624 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000160922 2024-09-13T11:26:10.624 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:10.625 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:10.625 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:26:10.625 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:54.578457+0000", 2024-09-13T11:26:10.625 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.0665999999999998e-05 2024-09-13T11:26:10.625 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:10.625 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:10.625 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:26:10.625 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:54.581339+0000", 2024-09-13T11:26:10.625 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0028826839999999999 2024-09-13T11:26:10.625 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:10.625 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:10.625 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:26:10.625 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:54.581425+0000", 2024-09-13T11:26:10.625 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.6077999999999998e-05 2024-09-13T11:26:10.626 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:10.626 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:26:10.626 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:10.626 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:10.626 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:10.626 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPushReply(3.29 143/120 [PushReplyOp(3:973d2047:::benchmark_data_smithi177_31116_object49749:head)])", 2024-09-13T11:26:10.626 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:24:57.387106+0000", 2024-09-13T11:26:10.626 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 73.167694318000002, 2024-09-13T11:26:10.626 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.64813637000000002, 2024-09-13T11:26:10.626 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:26:10.626 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:26:10.626 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-09-13T11:26:10.626 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:26:10.627 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:10.627 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:26:10.627 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:57.387106+0000", 2024-09-13T11:26:10.627 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:10.627 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:10.627 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:10.627 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:26:10.627 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:57.387106+0000", 2024-09-13T11:26:10.627 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:10.627 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:10.627 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:10.627 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:26:10.627 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:57.387116+0000", 2024-09-13T11:26:10.627 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.1530000000000003e-06 2024-09-13T11:26:10.628 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:10.628 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:10.628 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:26:10.628 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:57.387127+0000", 2024-09-13T11:26:10.628 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1197e-05 2024-09-13T11:26:10.628 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:10.628 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:10.628 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:26:10.628 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:58.034264+0000", 2024-09-13T11:26:10.628 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.64713691799999995 2024-09-13T11:26:10.628 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:10.628 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:10.628 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:26:10.628 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:58.034273+0000", 2024-09-13T11:26:10.629 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.0929999999999992e-06 2024-09-13T11:26:10.629 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:10.629 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:10.629 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:26:10.629 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:58.034321+0000", 2024-09-13T11:26:10.629 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.8396999999999998e-05 2024-09-13T11:26:10.629 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:10.629 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:10.629 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:26:10.629 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:58.035243+0000", 2024-09-13T11:26:10.629 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00092161200000000004 2024-09-13T11:26:10.629 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:10.629 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:26:10.630 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:10.630 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:10.630 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:10.630 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPushReply(3.29 144/120 [PushReplyOp(3:94a6fc3a:::benchmark_data_smithi177_31116_object50245:head)])", 2024-09-13T11:26:10.630 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:24:58.320977+0000", 2024-09-13T11:26:10.630 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 72.233823735000001, 2024-09-13T11:26:10.630 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.31110569599999999, 2024-09-13T11:26:10.630 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:26:10.630 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:26:10.630 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-09-13T11:26:10.630 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:26:10.630 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:10.630 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:26:10.631 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:58.320977+0000", 2024-09-13T11:26:10.631 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:10.631 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:10.631 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:10.631 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:26:10.631 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:58.320977+0000", 2024-09-13T11:26:10.631 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:10.631 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:10.631 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:10.631 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:26:10.631 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:58.320978+0000", 2024-09-13T11:26:10.631 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.8700000000000004e-07 2024-09-13T11:26:10.631 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:10.631 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:10.632 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:26:10.632 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:58.320985+0000", 2024-09-13T11:26:10.632 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.742e-06 2024-09-13T11:26:10.632 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:10.632 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:10.632 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:26:10.632 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:58.631147+0000", 2024-09-13T11:26:10.632 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.31016212900000001 2024-09-13T11:26:10.632 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:10.632 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:10.632 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:26:10.632 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:58.631159+0000", 2024-09-13T11:26:10.632 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1946999999999999e-05 2024-09-13T11:26:10.632 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:10.633 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:10.633 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:26:10.633 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:58.631257+0000", 2024-09-13T11:26:10.633 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.8096000000000004e-05 2024-09-13T11:26:10.633 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:10.633 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:10.633 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:26:10.633 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:58.632083+0000", 2024-09-13T11:26:10.633 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00082589499999999999 2024-09-13T11:26:10.633 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:10.633 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:26:10.633 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:10.633 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:10.634 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:10.634 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:147568 3.1d 3:b82a2ed2:::benchmark_data_smithi177_31116_object7265:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e148)", 2024-09-13T11:26:10.634 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:25:04.213894+0000", 2024-09-13T11:26:10.634 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 66.340906963999998, 2024-09-13T11:26:10.634 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.16897441999999999, 2024-09-13T11:26:10.634 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:26:10.634 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:26:10.634 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:10.634 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:26:10.634 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:26:10.634 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:10.634 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 147568 2024-09-13T11:26:10.634 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:10.634 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:26:10.635 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:10.635 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:26:10.635 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:04.213894+0000", 2024-09-13T11:26:10.635 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:10.635 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:10.635 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:10.635 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:26:10.635 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:04.213894+0000", 2024-09-13T11:26:10.635 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:10.635 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:10.635 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:10.635 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:26:10.635 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:04.213898+0000", 2024-09-13T11:26:10.635 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.9049999999999999e-06 2024-09-13T11:26:10.636 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:10.636 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:10.636 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:26:10.636 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:04.213908+0000", 2024-09-13T11:26:10.636 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0069e-05 2024-09-13T11:26:10.636 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:10.636 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:10.636 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:26:10.636 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:04.213931+0000", 2024-09-13T11:26:10.636 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.3538999999999999e-05 2024-09-13T11:26:10.636 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:10.636 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:10.636 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:26:10.637 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:04.213941+0000", 2024-09-13T11:26:10.637 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0073e-05 2024-09-13T11:26:10.637 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:10.637 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:10.637 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:26:10.637 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:04.214039+0000", 2024-09-13T11:26:10.637 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.7666999999999998e-05 2024-09-13T11:26:10.637 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:10.637 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:10.637 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:26:10.637 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:04.214226+0000", 2024-09-13T11:26:10.637 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000187166 2024-09-13T11:26:10.637 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:10.638 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:10.638 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 15", 2024-09-13T11:26:10.638 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:04.214365+0000", 2024-09-13T11:26:10.638 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000139016 2024-09-13T11:26:10.638 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:10.638 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:10.638 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:26:10.638 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:04.217864+0000", 2024-09-13T11:26:10.638 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.003499177 2024-09-13T11:26:10.638 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:10.638 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:10.638 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:10.638 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:04.382775+0000", 2024-09-13T11:26:10.638 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.16491091499999999 2024-09-13T11:26:10.639 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:10.639 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:10.639 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:26:10.639 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:04.382823+0000", 2024-09-13T11:26:10.639 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.8232000000000002e-05 2024-09-13T11:26:10.639 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:10.639 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:10.639 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:26:10.639 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:04.382868+0000", 2024-09-13T11:26:10.639 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.4660999999999998e-05 2024-09-13T11:26:10.639 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:10.639 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:26:10.639 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:10.639 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:10.640 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:10.640 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.4665.0:170721 3.9 e162/117 3:93244b1b:::benchmark_data_smithi177_31116_object30418:head v 162'5743, mlcod=162'5740)", 2024-09-13T11:26:10.640 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:25:21.721723+0000", 2024-09-13T11:26:10.640 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 48.833078020000002, 2024-09-13T11:26:10.640 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.19722754300000001, 2024-09-13T11:26:10.640 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:26:10.640 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:26:10.640 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:10.640 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:26:10.640 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:10.640 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:26:10.640 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:21.721723+0000", 2024-09-13T11:26:10.640 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:10.640 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:10.641 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:10.641 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:26:10.641 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:21.721723+0000", 2024-09-13T11:26:10.641 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:10.641 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:10.641 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:10.641 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:26:10.641 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:21.721726+0000", 2024-09-13T11:26:10.641 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.1930000000000002e-06 2024-09-13T11:26:10.641 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:10.641 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:10.641 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:26:10.641 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:21.721741+0000", 2024-09-13T11:26:10.642 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5299999999999999e-05 2024-09-13T11:26:10.642 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:10.642 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:10.642 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:26:10.642 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:21.915450+0000", 2024-09-13T11:26:10.642 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.193708519 2024-09-13T11:26:10.642 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:10.642 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:10.642 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:26:10.642 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:21.915462+0000", 2024-09-13T11:26:10.642 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2414e-05 2024-09-13T11:26:10.642 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:10.642 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:10.642 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:26:10.643 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:21.915603+0000", 2024-09-13T11:26:10.643 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00014116500000000001 2024-09-13T11:26:10.643 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:10.643 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:10.643 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:26:10.643 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:21.915668+0000", 2024-09-13T11:26:10.643 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.4488999999999999e-05 2024-09-13T11:26:10.643 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:10.643 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:10.643 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:26:10.643 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:21.918850+0000", 2024-09-13T11:26:10.643 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0031825310000000002 2024-09-13T11:26:10.643 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:10.643 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:10.644 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:26:10.644 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:21.918950+0000", 2024-09-13T11:26:10.644 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.9932000000000005e-05 2024-09-13T11:26:10.644 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:10.644 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:26:10.644 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:10.644 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:10.644 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:10.644 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:191058 3.1d 3:bd6e7029:::benchmark_data_smithi177_31116_object50755:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e175)", 2024-09-13T11:26:10.645 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:25:38.512403+0000", 2024-09-13T11:26:10.645 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 32.042397475000001, 2024-09-13T11:26:10.645 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.38726333200000002, 2024-09-13T11:26:10.645 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:26:10.645 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:26:10.645 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:10.645 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:26:10.645 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:26:10.645 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:10.645 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 191058 2024-09-13T11:26:10.645 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:10.645 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:26:10.645 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:10.646 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:26:10.646 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:38.512403+0000", 2024-09-13T11:26:10.646 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:10.646 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:10.646 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:10.646 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:26:10.646 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:38.512403+0000", 2024-09-13T11:26:10.646 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:10.646 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:10.646 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:10.646 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:26:10.646 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:38.512406+0000", 2024-09-13T11:26:10.646 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.4059999999999999e-06 2024-09-13T11:26:10.647 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:10.647 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:10.647 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:26:10.647 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:38.512412+0000", 2024-09-13T11:26:10.647 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.8170000000000001e-06 2024-09-13T11:26:10.647 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:10.647 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:10.647 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:26:10.647 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:38.512430+0000", 2024-09-13T11:26:10.647 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.7397999999999998e-05 2024-09-13T11:26:10.647 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:10.647 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:10.647 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:26:10.648 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:38.512438+0000", 2024-09-13T11:26:10.648 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.3459999999999994e-06 2024-09-13T11:26:10.648 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:10.648 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:10.648 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:26:10.648 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:38.512526+0000", 2024-09-13T11:26:10.648 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.8146999999999995e-05 2024-09-13T11:26:10.648 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:10.648 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:10.648 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:26:10.648 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:38.512722+0000", 2024-09-13T11:26:10.648 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00019587599999999999 2024-09-13T11:26:10.648 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:10.649 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:10.649 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 15", 2024-09-13T11:26:10.649 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:38.512882+0000", 2024-09-13T11:26:10.649 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00015957300000000001 2024-09-13T11:26:10.649 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:10.649 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:10.649 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:26:10.649 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:38.517854+0000", 2024-09-13T11:26:10.649 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0049725500000000001 2024-09-13T11:26:10.649 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:10.649 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:10.649 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:10.649 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:38.899562+0000", 2024-09-13T11:26:10.649 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.38170719800000003 2024-09-13T11:26:10.650 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:10.650 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:10.650 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:26:10.650 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:38.899622+0000", 2024-09-13T11:26:10.650 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.0396999999999998e-05 2024-09-13T11:26:10.650 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:10.650 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:10.650 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:26:10.650 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:38.899667+0000", 2024-09-13T11:26:10.650 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.4623999999999998e-05 2024-09-13T11:26:10.650 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:10.650 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:26:10.650 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:10.651 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:10.651 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:10.651 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.4665.0:191547 3.25 e176/141 3:a7ac3e5b:::benchmark_data_smithi177_31116_object51244:head v 176'6020, mlcod=175'6018)", 2024-09-13T11:26:10.651 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:25:39.496290+0000", 2024-09-13T11:26:10.651 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 31.058510913999999, 2024-09-13T11:26:10.651 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.179206057, 2024-09-13T11:26:10.651 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:26:10.651 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:26:10.651 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:10.651 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:26:10.651 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:10.651 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:26:10.651 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:39.496290+0000", 2024-09-13T11:26:10.651 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:10.652 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:10.652 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:10.652 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:26:10.652 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:39.496290+0000", 2024-09-13T11:26:10.652 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:10.652 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:10.652 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:10.652 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:26:10.652 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:39.496293+0000", 2024-09-13T11:26:10.652 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.8499999999999998e-06 2024-09-13T11:26:10.652 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:10.652 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:10.652 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:26:10.653 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:39.496305+0000", 2024-09-13T11:26:10.653 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2286999999999999e-05 2024-09-13T11:26:10.653 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:10.653 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:10.653 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:26:10.653 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:39.670614+0000", 2024-09-13T11:26:10.653 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.17430884699999999 2024-09-13T11:26:10.653 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:10.653 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:10.653 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:26:10.653 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:39.670625+0000", 2024-09-13T11:26:10.653 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1233000000000001e-05 2024-09-13T11:26:10.653 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:10.653 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:10.654 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:26:10.654 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:39.670722+0000", 2024-09-13T11:26:10.654 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.6885999999999994e-05 2024-09-13T11:26:10.654 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:10.654 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:10.654 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:26:10.654 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:39.670756+0000", 2024-09-13T11:26:10.654 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.4125999999999998e-05 2024-09-13T11:26:10.654 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:10.654 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:10.654 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:26:10.654 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:39.675411+0000", 2024-09-13T11:26:10.654 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0046546160000000003 2024-09-13T11:26:10.654 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:10.655 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:10.655 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:26:10.655 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:39.675496+0000", 2024-09-13T11:26:10.655 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.5211999999999997e-05 2024-09-13T11:26:10.655 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:10.655 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:26:10.655 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:10.655 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:10.655 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:26:10.655 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-13T11:26:10.657 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-13T11:26:10.654+0000 7f6a68775640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26307) UID: 0 2024-09-13T11:26:10.657 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-13T11:26:10.758 INFO:tasks.ceph.osd.3.smithi177.stderr:2024-09-13T11:26:10.753+0000 7f10adba4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26574) UID: 0 2024-09-13T11:26:10.848 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-13T11:26:10.848 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [ 2024-09-13T11:26:10.848 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:10.848 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:227145 3.10 3:0ca9ec7d:::benchmark_data_smithi177_31116_object86842:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e196)", 2024-09-13T11:26:10.848 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:26:10.829912+0000", 2024-09-13T11:26:10.848 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 0.018359718000000001, 2024-09-13T11:26:10.848 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.018382955999999999, 2024-09-13T11:26:10.848 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:26:10.848 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:26:10.848 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "waiting for sub ops", 2024-09-13T11:26:10.849 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:26:10.849 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:26:10.849 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:10.849 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 227145 2024-09-13T11:26:10.849 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:10.849 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:26:10.849 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:10.849 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:26:10.849 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:10.829912+0000", 2024-09-13T11:26:10.849 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:10.849 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:10.849 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:10.849 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:26:10.850 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:10.829912+0000", 2024-09-13T11:26:10.850 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:10.850 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:10.850 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:10.850 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:26:10.850 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:10.829915+0000", 2024-09-13T11:26:10.850 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.2780000000000002e-06 2024-09-13T11:26:10.850 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:10.850 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:10.850 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:26:10.850 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:10.829924+0000", 2024-09-13T11:26:10.850 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.5869999999999999e-06 2024-09-13T11:26:10.850 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:10.850 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:10.851 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:26:10.851 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:10.829945+0000", 2024-09-13T11:26:10.851 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.0746000000000001e-05 2024-09-13T11:26:10.851 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:10.851 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:10.851 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:26:10.851 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:10.829957+0000", 2024-09-13T11:26:10.851 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2038999999999999e-05 2024-09-13T11:26:10.851 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:10.851 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:10.851 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:26:10.851 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:10.830022+0000", 2024-09-13T11:26:10.851 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.5234999999999995e-05 2024-09-13T11:26:10.851 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:10.852 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:10.852 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:26:10.852 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:10.830129+0000", 2024-09-13T11:26:10.852 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00010641099999999999 2024-09-13T11:26:10.852 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:10.852 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:10.852 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 0", 2024-09-13T11:26:10.852 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:10.830201+0000", 2024-09-13T11:26:10.852 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.2514000000000004e-05 2024-09-13T11:26:10.852 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:10.852 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:10.852 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:26:10.852 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:10.833201+0000", 2024-09-13T11:26:10.852 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0029998160000000002 2024-09-13T11:26:10.853 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:10.853 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:26:10.853 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:10.853 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:10.853 INFO:teuthology.orchestra.run.smithi162.stdout: ], 2024-09-13T11:26:10.853 INFO:teuthology.orchestra.run.smithi162.stdout: "num_ops": 1 2024-09-13T11:26:10.853 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-13T11:26:10.858 INFO:tasks.ceph.osd.14.smithi162.stderr:2024-09-13T11:26:10.855+0000 7f3df5903640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33154) UID: 0 2024-09-13T11:26:10.867 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-13T11:26:10.959 INFO:tasks.ceph.osd.14.smithi162.stderr:2024-09-13T11:26:10.955+0000 7f3df5903640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33154) UID: 0 2024-09-13T11:26:11.042 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-13T11:26:11.042 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [], 2024-09-13T11:26:11.042 INFO:teuthology.orchestra.run.smithi162.stdout: "complaint_time": 30, 2024-09-13T11:26:11.043 INFO:teuthology.orchestra.run.smithi162.stdout: "num_blocked_ops": 0 2024-09-13T11:26:11.043 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-13T11:26:11.058 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-13T11:26:11.060 INFO:tasks.ceph.osd.1.smithi092.stderr:2024-09-13T11:26:11.056+0000 7fe7789c5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33718) UID: 0 2024-09-13T11:26:11.086 INFO:teuthology.orchestra.run.smithi001.stderr:nodeep-scrub is unset 2024-09-13T11:26:11.086 INFO:tasks.ceph.osd.11.smithi177.stderr:2024-09-13T11:26:11.081+0000 7f246bcf0640 -1 osd.11 196 handle_osd_map injecting map crc failure 2024-09-13T11:26:11.096 INFO:tasks.ceph.osd.10.smithi162.stderr:2024-09-13T11:26:11.091+0000 7fb593421640 -1 osd.10 196 handle_osd_map injecting map crc failure 2024-09-13T11:26:11.105 INFO:tasks.ceph.osd.3.smithi177.stderr:2024-09-13T11:26:11.101+0000 7f10a25d1640 -1 osd.3 196 handle_osd_map injecting map crc failure 2024-09-13T11:26:11.161 INFO:tasks.ceph.osd.10.smithi162.stderr:2024-09-13T11:26:11.159+0000 7fb59c9f0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26317) UID: 0 2024-09-13T11:26:11.227 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-13T11:26:11.227 INFO:teuthology.orchestra.run.smithi162.stdout: "size": 20, 2024-09-13T11:26:11.227 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 600, 2024-09-13T11:26:11.227 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [ 2024-09-13T11:26:11.228 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.228 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4257.0:80 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 262144~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-13T11:26:11.228 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:22:12.516065+0000", 2024-09-13T11:26:11.228 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 238.70881244200001, 2024-09-13T11:26:11.228 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.46461997999999999, 2024-09-13T11:26:11.228 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:26:11.228 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:26:11.228 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:11.228 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:26:11.228 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4257", 2024-09-13T11:26:11.228 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/1488131527", 2024-09-13T11:26:11.228 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 80 2024-09-13T11:26:11.228 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.229 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:26:11.229 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.229 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:26:11.229 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.516065+0000", 2024-09-13T11:26:11.229 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:11.229 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.229 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.229 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:26:11.229 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.516065+0000", 2024-09-13T11:26:11.229 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:11.229 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.229 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.229 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:26:11.230 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.516066+0000", 2024-09-13T11:26:11.230 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0869999999999999e-06 2024-09-13T11:26:11.230 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.230 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.230 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:26:11.230 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.516112+0000", 2024-09-13T11:26:11.230 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.5503000000000001e-05 2024-09-13T11:26:11.230 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.230 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.230 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:26:11.230 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.516128+0000", 2024-09-13T11:26:11.230 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5891000000000001e-05 2024-09-13T11:26:11.230 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.230 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.231 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:26:11.231 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.516138+0000", 2024-09-13T11:26:11.231 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.8120000000000002e-06 2024-09-13T11:26:11.231 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.231 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.231 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:26:11.231 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.520398+0000", 2024-09-13T11:26:11.231 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0042605450000000001 2024-09-13T11:26:11.231 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.231 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.231 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:26:11.231 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.520526+0000", 2024-09-13T11:26:11.231 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000128355 2024-09-13T11:26:11.232 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.232 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.232 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 0", 2024-09-13T11:26:11.232 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.520730+0000", 2024-09-13T11:26:11.232 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00020337799999999999 2024-09-13T11:26:11.232 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.232 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.232 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:26:11.232 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.522417+0000", 2024-09-13T11:26:11.232 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001687184 2024-09-13T11:26:11.232 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.232 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.232 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:11.233 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.980408+0000", 2024-09-13T11:26:11.233 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.45799068399999998 2024-09-13T11:26:11.233 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.233 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.233 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:26:11.233 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.980565+0000", 2024-09-13T11:26:11.233 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00015760899999999999 2024-09-13T11:26:11.233 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.233 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.233 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:26:11.233 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.980685+0000", 2024-09-13T11:26:11.233 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000119932 2024-09-13T11:26:11.233 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:11.234 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:26:11.234 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:11.234 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.234 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.234 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop_reply(client.4257.0:80 1.0 e24/23 ondisk, result = 0)", 2024-09-13T11:26:11.234 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:22:12.522938+0000", 2024-09-13T11:26:11.234 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 238.701939409, 2024-09-13T11:26:11.234 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.45776420600000001, 2024-09-13T11:26:11.234 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:26:11.234 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:26:11.234 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:26:11.234 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:26:11.234 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.235 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:26:11.235 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.522938+0000", 2024-09-13T11:26:11.235 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:11.235 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.235 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.235 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:26:11.235 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.522938+0000", 2024-09-13T11:26:11.235 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:11.235 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.235 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.235 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:26:11.235 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.522941+0000", 2024-09-13T11:26:11.235 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.0290000000000001e-06 2024-09-13T11:26:11.235 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.236 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.236 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:26:11.236 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.522950+0000", 2024-09-13T11:26:11.236 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.7499999999999992e-06 2024-09-13T11:26:11.236 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.236 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.236 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:26:11.236 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.980215+0000", 2024-09-13T11:26:11.236 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.45726523800000002 2024-09-13T11:26:11.236 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.236 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.236 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:26:11.236 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.980229+0000", 2024-09-13T11:26:11.237 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3549e-05 2024-09-13T11:26:11.237 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.237 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.237 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:26:11.237 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.980365+0000", 2024-09-13T11:26:11.237 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00013608700000000001 2024-09-13T11:26:11.237 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.237 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.237 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:26:11.237 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.980388+0000", 2024-09-13T11:26:11.237 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.3112000000000002e-05 2024-09-13T11:26:11.237 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.237 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.238 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:26:11.238 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.980702+0000", 2024-09-13T11:26:11.238 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00031444099999999999 2024-09-13T11:26:11.238 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:11.238 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:26:11.238 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:11.238 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.238 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.238 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:25416 3.8 3:1a1fbdd1:::benchmark_data_smithi177_31116_object25415:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e56)", 2024-09-13T11:26:11.238 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:04.512188+0000", 2024-09-13T11:26:11.238 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 186.712689442, 2024-09-13T11:26:11.238 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.210208697, 2024-09-13T11:26:11.238 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:26:11.238 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:26:11.239 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:11.239 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:26:11.239 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:26:11.239 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:11.239 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 25416 2024-09-13T11:26:11.239 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.239 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:26:11.239 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.239 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:26:11.239 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.512188+0000", 2024-09-13T11:26:11.239 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:11.239 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.239 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.239 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:26:11.240 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.512188+0000", 2024-09-13T11:26:11.240 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:11.240 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.240 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.240 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:26:11.240 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.512192+0000", 2024-09-13T11:26:11.240 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.2660000000000003e-06 2024-09-13T11:26:11.240 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.240 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.240 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:26:11.240 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.512247+0000", 2024-09-13T11:26:11.240 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.4889000000000003e-05 2024-09-13T11:26:11.240 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.241 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.241 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:26:11.241 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.512279+0000", 2024-09-13T11:26:11.241 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.1146000000000003e-05 2024-09-13T11:26:11.241 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.241 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.241 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:26:11.241 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.512297+0000", 2024-09-13T11:26:11.241 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.825e-05 2024-09-13T11:26:11.241 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.241 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.242 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:26:11.242 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.512377+0000", 2024-09-13T11:26:11.242 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.9875000000000004e-05 2024-09-13T11:26:11.242 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.242 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.242 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:26:11.242 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.512570+0000", 2024-09-13T11:26:11.242 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00019312100000000001 2024-09-13T11:26:11.242 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.243 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.243 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 1", 2024-09-13T11:26:11.243 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.512798+0000", 2024-09-13T11:26:11.243 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00022847899999999999 2024-09-13T11:26:11.243 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.243 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.243 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:26:11.243 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.514975+0000", 2024-09-13T11:26:11.243 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0021769979999999999 2024-09-13T11:26:11.243 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.243 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.243 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:11.243 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.722207+0000", 2024-09-13T11:26:11.244 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.20723217099999999 2024-09-13T11:26:11.244 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.244 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.244 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:26:11.244 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.722314+0000", 2024-09-13T11:26:11.244 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00010648400000000001 2024-09-13T11:26:11.244 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.244 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.244 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:26:11.244 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.722397+0000", 2024-09-13T11:26:11.244 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.3017999999999997e-05 2024-09-13T11:26:11.244 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:11.244 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:26:11.244 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:11.245 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.245 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.245 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop_reply(client.4665.0:25416 3.8 e56/40 ondisk, result = 0)", 2024-09-13T11:26:11.245 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:04.515635+0000", 2024-09-13T11:26:11.245 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 186.70924278999999, 2024-09-13T11:26:11.245 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.20677600500000001, 2024-09-13T11:26:11.245 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:26:11.245 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:26:11.245 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:26:11.245 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:26:11.245 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.245 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:26:11.245 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.515635+0000", 2024-09-13T11:26:11.246 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:11.246 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.246 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.246 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:26:11.246 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.515635+0000", 2024-09-13T11:26:11.246 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:11.246 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.246 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.246 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:26:11.246 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.515639+0000", 2024-09-13T11:26:11.246 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.619e-06 2024-09-13T11:26:11.246 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.246 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.246 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:26:11.247 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.515650+0000", 2024-09-13T11:26:11.247 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.111e-05 2024-09-13T11:26:11.247 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.247 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.247 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:26:11.247 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.722004+0000", 2024-09-13T11:26:11.247 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.206354805 2024-09-13T11:26:11.247 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.247 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.247 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:26:11.247 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.722017+0000", 2024-09-13T11:26:11.247 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2840000000000001e-05 2024-09-13T11:26:11.247 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.247 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.248 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:26:11.248 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.722186+0000", 2024-09-13T11:26:11.248 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00016898299999999999 2024-09-13T11:26:11.248 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.248 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.248 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:26:11.248 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.722199+0000", 2024-09-13T11:26:11.248 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.289e-05 2024-09-13T11:26:11.248 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.248 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.248 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:26:11.248 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.722411+0000", 2024-09-13T11:26:11.248 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00021175799999999999 2024-09-13T11:26:11.249 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:11.249 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:26:11.249 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:11.249 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.249 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.249 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:30833 3.8 3:1c203f8d:::benchmark_data_smithi177_31116_object30832:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e59)", 2024-09-13T11:26:11.249 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:09.506656+0000", 2024-09-13T11:26:11.249 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 181.718221892, 2024-09-13T11:26:11.249 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.224862282, 2024-09-13T11:26:11.249 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:26:11.249 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:26:11.249 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:11.249 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:26:11.250 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:26:11.250 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:11.250 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 30833 2024-09-13T11:26:11.250 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.250 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:26:11.250 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.250 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:26:11.250 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.506656+0000", 2024-09-13T11:26:11.250 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:11.250 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.250 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.250 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:26:11.250 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.506656+0000", 2024-09-13T11:26:11.251 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:11.251 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.251 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.251 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:26:11.251 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.506660+0000", 2024-09-13T11:26:11.251 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.6e-06 2024-09-13T11:26:11.251 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.251 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.251 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:26:11.251 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.506717+0000", 2024-09-13T11:26:11.252 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.6131999999999997e-05 2024-09-13T11:26:11.252 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.252 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.252 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:26:11.252 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.506751+0000", 2024-09-13T11:26:11.252 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.4746000000000002e-05 2024-09-13T11:26:11.252 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.252 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.252 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:26:11.252 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.506774+0000", 2024-09-13T11:26:11.252 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.2636000000000001e-05 2024-09-13T11:26:11.252 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.253 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.253 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:26:11.253 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.506878+0000", 2024-09-13T11:26:11.253 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000104057 2024-09-13T11:26:11.253 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.253 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.253 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:26:11.253 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.507052+0000", 2024-09-13T11:26:11.253 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00017402800000000001 2024-09-13T11:26:11.253 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.253 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.253 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 1", 2024-09-13T11:26:11.254 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.507217+0000", 2024-09-13T11:26:11.254 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00016516 2024-09-13T11:26:11.254 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.254 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.254 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:26:11.254 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.508992+0000", 2024-09-13T11:26:11.254 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0017743990000000001 2024-09-13T11:26:11.254 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.254 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.254 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:11.254 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.731429+0000", 2024-09-13T11:26:11.254 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.22243718000000001 2024-09-13T11:26:11.254 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.255 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.255 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:26:11.255 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.731479+0000", 2024-09-13T11:26:11.255 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.0146999999999999e-05 2024-09-13T11:26:11.255 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.255 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.255 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:26:11.255 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.731518+0000", 2024-09-13T11:26:11.255 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.9196999999999998e-05 2024-09-13T11:26:11.255 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:11.255 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:26:11.255 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:11.255 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.255 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.256 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop_reply(client.4665.0:30833 3.8 e59/40 ondisk, result = 0)", 2024-09-13T11:26:11.256 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:09.508907+0000", 2024-09-13T11:26:11.256 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 181.715970798, 2024-09-13T11:26:11.256 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.222618862, 2024-09-13T11:26:11.256 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:26:11.256 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:26:11.256 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:26:11.256 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:26:11.256 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.256 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:26:11.256 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.508907+0000", 2024-09-13T11:26:11.256 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:11.256 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.257 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.257 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:26:11.257 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.508907+0000", 2024-09-13T11:26:11.257 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:11.257 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.257 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.257 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:26:11.257 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.508910+0000", 2024-09-13T11:26:11.257 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.993e-06 2024-09-13T11:26:11.257 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.257 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.257 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:26:11.257 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.508918+0000", 2024-09-13T11:26:11.258 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.7000000000000008e-06 2024-09-13T11:26:11.258 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.258 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.258 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:26:11.258 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.731341+0000", 2024-09-13T11:26:11.258 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.22242308699999999 2024-09-13T11:26:11.258 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.258 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.258 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:26:11.258 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.731350+0000", 2024-09-13T11:26:11.258 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.3179999999999996e-06 2024-09-13T11:26:11.258 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.259 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.259 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:26:11.259 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.731418+0000", 2024-09-13T11:26:11.259 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.7586e-05 2024-09-13T11:26:11.259 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.259 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.259 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:26:11.259 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.731424+0000", 2024-09-13T11:26:11.259 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.844e-06 2024-09-13T11:26:11.259 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.259 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.259 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:26:11.259 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.731526+0000", 2024-09-13T11:26:11.260 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000101334 2024-09-13T11:26:11.260 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:11.260 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:26:11.260 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:11.260 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.260 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.260 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPush(3.10 76/70 [PushOp(3:0e7d2700:::benchmark_data_smithi177_31116_object17406:head, version: 52'1075, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:0e7d2700:::benchmark_data_smithi177_31116_object17406:head@52'1075, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:26:11.260 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:32.206198+0000", 2024-09-13T11:26:11.260 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 159.01868008899999, 2024-09-13T11:26:11.260 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.25193395000000002, 2024-09-13T11:26:11.260 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:26:11.260 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:26:11.260 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:26:11.261 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:26:11.261 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.261 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:26:11.261 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.206198+0000", 2024-09-13T11:26:11.261 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:11.261 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.261 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.261 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:26:11.261 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.206198+0000", 2024-09-13T11:26:11.261 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:11.261 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.261 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.262 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:26:11.262 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.206199+0000", 2024-09-13T11:26:11.262 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.46e-06 2024-09-13T11:26:11.262 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.262 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.262 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:26:11.262 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.206250+0000", 2024-09-13T11:26:11.262 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.113e-05 2024-09-13T11:26:11.262 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.262 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.262 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:26:11.262 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.456042+0000", 2024-09-13T11:26:11.262 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.24979182999999999 2024-09-13T11:26:11.263 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.263 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.263 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:26:11.263 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.456084+0000", 2024-09-13T11:26:11.263 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.1814000000000003e-05 2024-09-13T11:26:11.263 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.263 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.263 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:26:11.263 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.456342+0000", 2024-09-13T11:26:11.263 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000258404 2024-09-13T11:26:11.263 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.263 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.263 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:26:11.264 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.456367+0000", 2024-09-13T11:26:11.264 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.5099000000000001e-05 2024-09-13T11:26:11.264 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.264 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.264 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:26:11.264 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.458132+0000", 2024-09-13T11:26:11.264 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0017642129999999999 2024-09-13T11:26:11.264 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:11.264 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:26:11.264 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:11.264 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.264 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.265 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPush(3.10 76/70 [PushOp(3:0eee8bcf:::benchmark_data_smithi177_31116_object17589:head, version: 53'1086, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:0eee8bcf:::benchmark_data_smithi177_31116_object17589:head@53'1086, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:26:11.265 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:32.585183+0000", 2024-09-13T11:26:11.265 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 158.639694195, 2024-09-13T11:26:11.265 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.16696202199999999, 2024-09-13T11:26:11.265 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:26:11.265 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:26:11.265 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:26:11.265 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:26:11.265 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.265 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:26:11.265 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.585183+0000", 2024-09-13T11:26:11.265 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:11.265 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.265 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.266 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:26:11.266 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.585183+0000", 2024-09-13T11:26:11.266 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:11.266 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.266 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.266 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:26:11.266 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.585187+0000", 2024-09-13T11:26:11.266 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.8480000000000003e-06 2024-09-13T11:26:11.266 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.266 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.266 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:26:11.266 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.585264+0000", 2024-09-13T11:26:11.266 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.6327000000000003e-05 2024-09-13T11:26:11.266 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.267 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.267 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:26:11.267 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.750459+0000", 2024-09-13T11:26:11.267 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.16519551900000001 2024-09-13T11:26:11.267 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.267 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.267 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:26:11.267 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.750480+0000", 2024-09-13T11:26:11.267 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.0984000000000001e-05 2024-09-13T11:26:11.267 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.267 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.267 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:26:11.267 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.750633+0000", 2024-09-13T11:26:11.267 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000153148 2024-09-13T11:26:11.268 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.268 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.268 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:26:11.268 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.750652+0000", 2024-09-13T11:26:11.268 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.8981e-05 2024-09-13T11:26:11.268 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.268 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.268 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:26:11.268 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.752145+0000", 2024-09-13T11:26:11.268 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001493215 2024-09-13T11:26:11.268 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:11.268 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:26:11.268 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:11.269 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.269 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.269 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:60417 3.10 3:092808b0:::benchmark_data_smithi177_31116_object60416:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e85)", 2024-09-13T11:26:11.269 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:42.173238+0000", 2024-09-13T11:26:11.269 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 149.05163918100001, 2024-09-13T11:26:11.269 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.38965994599999998, 2024-09-13T11:26:11.269 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:26:11.269 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:26:11.269 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:11.269 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:26:11.269 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:26:11.269 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:11.269 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 60417 2024-09-13T11:26:11.269 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.270 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:26:11.270 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.270 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:26:11.270 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.173238+0000", 2024-09-13T11:26:11.270 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:11.270 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.270 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.270 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:26:11.270 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.173238+0000", 2024-09-13T11:26:11.270 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:11.270 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.270 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.270 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:26:11.271 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.173241+0000", 2024-09-13T11:26:11.271 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.6819999999999999e-06 2024-09-13T11:26:11.271 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.271 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.271 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:26:11.271 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.173335+0000", 2024-09-13T11:26:11.271 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.3957999999999997e-05 2024-09-13T11:26:11.271 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.271 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.271 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:26:11.271 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.173358+0000", 2024-09-13T11:26:11.271 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.3023999999999998e-05 2024-09-13T11:26:11.271 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.271 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.272 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:26:11.272 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.173373+0000", 2024-09-13T11:26:11.272 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4867e-05 2024-09-13T11:26:11.272 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.272 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.272 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:26:11.272 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.173466+0000", 2024-09-13T11:26:11.272 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.2542999999999998e-05 2024-09-13T11:26:11.272 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.272 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.272 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:26:11.272 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.173754+0000", 2024-09-13T11:26:11.272 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000288657 2024-09-13T11:26:11.273 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.273 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.273 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 0", 2024-09-13T11:26:11.273 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.174066+0000", 2024-09-13T11:26:11.273 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00031170999999999998 2024-09-13T11:26:11.273 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.273 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.273 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:26:11.273 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.178245+0000", 2024-09-13T11:26:11.273 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00417864 2024-09-13T11:26:11.273 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.273 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.273 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:11.273 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.562617+0000", 2024-09-13T11:26:11.274 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.38437253199999999 2024-09-13T11:26:11.274 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.274 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.274 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:26:11.274 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.562782+0000", 2024-09-13T11:26:11.274 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00016499200000000001 2024-09-13T11:26:11.274 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.274 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.274 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:26:11.274 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.562898+0000", 2024-09-13T11:26:11.274 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000116341 2024-09-13T11:26:11.274 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:11.274 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:26:11.275 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:11.275 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.275 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.275 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:83193 3.8 3:10066d83:::benchmark_data_smithi177_31116_object83192:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106)", 2024-09-13T11:26:11.275 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:24:04.283923+0000", 2024-09-13T11:26:11.275 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 126.940954498, 2024-09-13T11:26:11.275 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.64775463499999997, 2024-09-13T11:26:11.275 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:26:11.275 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:26:11.275 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:11.275 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:26:11.275 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:26:11.275 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:11.276 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 83193 2024-09-13T11:26:11.276 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.276 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:26:11.276 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.276 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:26:11.276 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.283923+0000", 2024-09-13T11:26:11.276 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:11.276 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.276 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.276 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:26:11.276 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.283923+0000", 2024-09-13T11:26:11.276 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:11.277 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.277 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.277 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:26:11.277 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.283925+0000", 2024-09-13T11:26:11.277 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.9489999999999999e-06 2024-09-13T11:26:11.277 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.277 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.277 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:26:11.278 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.283960+0000", 2024-09-13T11:26:11.278 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.5063e-05 2024-09-13T11:26:11.278 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.278 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.278 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:26:11.278 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.283974+0000", 2024-09-13T11:26:11.278 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3546e-05 2024-09-13T11:26:11.278 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.278 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.278 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:26:11.278 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.283983+0000", 2024-09-13T11:26:11.278 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.2499999999999995e-06 2024-09-13T11:26:11.278 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.279 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.279 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:26:11.279 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.284061+0000", 2024-09-13T11:26:11.279 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.7946000000000002e-05 2024-09-13T11:26:11.279 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.279 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.279 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:26:11.279 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.284270+0000", 2024-09-13T11:26:11.279 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00020876799999999999 2024-09-13T11:26:11.279 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.279 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.279 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 1", 2024-09-13T11:26:11.279 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.284489+0000", 2024-09-13T11:26:11.279 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00021956800000000001 2024-09-13T11:26:11.280 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.280 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.280 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:26:11.280 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.286456+0000", 2024-09-13T11:26:11.280 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0019665809999999998 2024-09-13T11:26:11.280 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.280 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.280 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:11.280 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.931497+0000", 2024-09-13T11:26:11.280 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.64504147700000003 2024-09-13T11:26:11.280 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.280 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.280 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:26:11.281 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.931589+0000", 2024-09-13T11:26:11.281 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.2041999999999997e-05 2024-09-13T11:26:11.281 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.281 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.281 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:26:11.281 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.931678+0000", 2024-09-13T11:26:11.281 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.8444999999999998e-05 2024-09-13T11:26:11.281 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:11.281 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:26:11.281 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:11.281 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.281 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.281 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:115060 3.8 3:11200ba9:::benchmark_data_smithi177_31116_object115059:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e125)", 2024-09-13T11:26:11.282 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:24:33.602178+0000", 2024-09-13T11:26:11.282 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 97.622699503999996, 2024-09-13T11:26:11.282 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.23668186799999999, 2024-09-13T11:26:11.282 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:26:11.282 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:26:11.282 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:11.282 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:26:11.282 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:26:11.282 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:11.282 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 115060 2024-09-13T11:26:11.282 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.282 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:26:11.282 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.282 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:26:11.283 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:33.602178+0000", 2024-09-13T11:26:11.283 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:11.283 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.283 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.283 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:26:11.283 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:33.602178+0000", 2024-09-13T11:26:11.283 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:11.283 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.283 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.283 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:26:11.283 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:33.602181+0000", 2024-09-13T11:26:11.283 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.0369999999999999e-06 2024-09-13T11:26:11.283 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.283 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.284 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:26:11.284 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:33.602217+0000", 2024-09-13T11:26:11.284 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.5837999999999998e-05 2024-09-13T11:26:11.284 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.284 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.284 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:26:11.284 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:33.602235+0000", 2024-09-13T11:26:11.284 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.7571000000000002e-05 2024-09-13T11:26:11.284 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.284 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.285 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:26:11.285 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:33.602245+0000", 2024-09-13T11:26:11.285 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0360000000000001e-05 2024-09-13T11:26:11.285 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.285 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.285 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:26:11.285 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:33.602312+0000", 2024-09-13T11:26:11.285 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.6989000000000005e-05 2024-09-13T11:26:11.285 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.285 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.285 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:26:11.285 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:33.602490+0000", 2024-09-13T11:26:11.286 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000178464 2024-09-13T11:26:11.286 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.286 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.286 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 1", 2024-09-13T11:26:11.286 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:33.602670+0000", 2024-09-13T11:26:11.286 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00018002799999999999 2024-09-13T11:26:11.286 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.286 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.286 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:26:11.286 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:33.604382+0000", 2024-09-13T11:26:11.286 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001711314 2024-09-13T11:26:11.286 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.286 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.287 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:11.287 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:33.838738+0000", 2024-09-13T11:26:11.287 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.23435634699999999 2024-09-13T11:26:11.287 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.287 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.287 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:26:11.287 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:33.838815+0000", 2024-09-13T11:26:11.287 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.7381999999999999e-05 2024-09-13T11:26:11.287 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.287 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.287 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:26:11.287 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:33.838860+0000", 2024-09-13T11:26:11.287 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.4537999999999999e-05 2024-09-13T11:26:11.288 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:11.288 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:26:11.288 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:11.288 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.288 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.288 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop_reply(client.4665.0:115060 3.8 e125/117 ondisk, result = 0)", 2024-09-13T11:26:11.288 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:24:33.604404+0000", 2024-09-13T11:26:11.288 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 97.620473246000003, 2024-09-13T11:26:11.288 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.23446235400000001, 2024-09-13T11:26:11.288 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:26:11.288 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:26:11.288 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:26:11.288 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:26:11.289 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.289 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:26:11.289 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:33.604404+0000", 2024-09-13T11:26:11.289 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:11.289 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.289 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.289 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:26:11.289 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:33.604404+0000", 2024-09-13T11:26:11.289 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:11.289 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.289 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.289 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:26:11.289 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:33.604406+0000", 2024-09-13T11:26:11.290 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.418e-06 2024-09-13T11:26:11.290 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.290 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.290 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:26:11.290 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:33.604411+0000", 2024-09-13T11:26:11.290 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.3519999999999997e-06 2024-09-13T11:26:11.290 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.290 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.290 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:26:11.290 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:33.838578+0000", 2024-09-13T11:26:11.290 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.23416656199999999 2024-09-13T11:26:11.290 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.290 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.290 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:26:11.291 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:33.838586+0000", 2024-09-13T11:26:11.291 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.9920000000000007e-06 2024-09-13T11:26:11.291 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.291 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.291 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:26:11.291 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:33.838655+0000", 2024-09-13T11:26:11.291 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.9463999999999998e-05 2024-09-13T11:26:11.291 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.291 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.291 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:26:11.291 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:33.838731+0000", 2024-09-13T11:26:11.291 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.5487000000000004e-05 2024-09-13T11:26:11.291 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.292 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.292 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:26:11.292 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:33.838867+0000", 2024-09-13T11:26:11.292 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000136079 2024-09-13T11:26:11.292 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:11.292 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:26:11.292 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:11.292 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.292 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.292 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPush(3.6 129/122 [PushOp(3:6261f3b6:::benchmark_data_smithi177_31116_object64090:head, version: 90'3486, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:6261f3b6:::benchmark_data_smithi177_31116_object64090:head@90'3486, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:26:11.292 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:24:38.641487+0000", 2024-09-13T11:26:11.292 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 92.583390835000003, 2024-09-13T11:26:11.293 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.209889098, 2024-09-13T11:26:11.293 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:26:11.293 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:26:11.293 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:26:11.293 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:26:11.293 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.293 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:26:11.293 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:38.641487+0000", 2024-09-13T11:26:11.293 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:11.293 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.293 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.293 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:26:11.293 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:38.641487+0000", 2024-09-13T11:26:11.294 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:11.294 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.294 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.294 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:26:11.294 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:38.641489+0000", 2024-09-13T11:26:11.294 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.0980000000000001e-06 2024-09-13T11:26:11.294 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.294 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.294 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:26:11.294 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:38.641560+0000", 2024-09-13T11:26:11.294 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.1404999999999998e-05 2024-09-13T11:26:11.294 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.294 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.294 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:26:11.295 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:38.850498+0000", 2024-09-13T11:26:11.295 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.20893800500000001 2024-09-13T11:26:11.295 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.295 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.295 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:26:11.295 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:38.850510+0000", 2024-09-13T11:26:11.295 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1762e-05 2024-09-13T11:26:11.295 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.295 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.295 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:26:11.295 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:38.850610+0000", 2024-09-13T11:26:11.295 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00010011200000000001 2024-09-13T11:26:11.295 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.296 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.296 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:26:11.296 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:38.850628+0000", 2024-09-13T11:26:11.296 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.7291000000000001e-05 2024-09-13T11:26:11.296 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.296 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.296 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:26:11.296 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:38.851376+0000", 2024-09-13T11:26:11.296 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00074842500000000005 2024-09-13T11:26:11.296 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:11.296 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:26:11.296 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:11.296 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.297 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.297 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.4665.0:126919 3.6 e130/122 3:634ce08e:::benchmark_data_smithi177_31116_object126918:head v 130'5137, mlcod=130'5134)", 2024-09-13T11:26:11.297 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:24:43.780120+0000", 2024-09-13T11:26:11.297 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 87.444757944000003, 2024-09-13T11:26:11.297 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.298086295, 2024-09-13T11:26:11.297 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:26:11.297 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:26:11.297 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:11.297 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:26:11.297 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.297 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:26:11.297 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:43.780120+0000", 2024-09-13T11:26:11.297 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:11.298 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.298 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.298 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:26:11.298 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:43.780120+0000", 2024-09-13T11:26:11.298 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:11.298 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.298 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.298 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:26:11.298 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:43.780123+0000", 2024-09-13T11:26:11.298 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.79e-06 2024-09-13T11:26:11.298 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.298 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.298 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:26:11.299 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:43.780160+0000", 2024-09-13T11:26:11.299 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.7945999999999999e-05 2024-09-13T11:26:11.299 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.299 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.299 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:26:11.299 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:44.075853+0000", 2024-09-13T11:26:11.299 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.29569234999999999 2024-09-13T11:26:11.299 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.299 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.299 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:26:11.299 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:44.075861+0000", 2024-09-13T11:26:11.299 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.9510000000000006e-06 2024-09-13T11:26:11.299 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.299 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.300 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:26:11.300 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:44.075924+0000", 2024-09-13T11:26:11.300 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.2946000000000003e-05 2024-09-13T11:26:11.300 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.300 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.300 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:26:11.300 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:44.075942+0000", 2024-09-13T11:26:11.300 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.8108e-05 2024-09-13T11:26:11.300 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.300 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.300 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:26:11.300 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:44.078164+0000", 2024-09-13T11:26:11.301 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.002222375 2024-09-13T11:26:11.301 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.301 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.301 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:26:11.301 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:44.078206+0000", 2024-09-13T11:26:11.301 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.1829000000000002e-05 2024-09-13T11:26:11.301 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:11.301 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:26:11.301 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:11.301 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.301 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.301 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:137473 3.10 3:094c0efb:::benchmark_data_smithi177_31116_object137472:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e139)", 2024-09-13T11:26:11.301 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:24:53.523319+0000", 2024-09-13T11:26:11.301 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 77.701558904999999, 2024-09-13T11:26:11.302 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.18297369299999999, 2024-09-13T11:26:11.302 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:26:11.302 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:26:11.302 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:11.302 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:26:11.302 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:26:11.302 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:11.302 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 137473 2024-09-13T11:26:11.302 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.302 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:26:11.302 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.302 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:26:11.302 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:53.523319+0000", 2024-09-13T11:26:11.302 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:11.303 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.303 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.303 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:26:11.303 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:53.523319+0000", 2024-09-13T11:26:11.303 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:11.303 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.303 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.303 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:26:11.303 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:53.523322+0000", 2024-09-13T11:26:11.303 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.2119999999999999e-06 2024-09-13T11:26:11.303 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.303 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.303 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:26:11.303 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:53.523370+0000", 2024-09-13T11:26:11.303 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.7788999999999999e-05 2024-09-13T11:26:11.304 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.304 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.304 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:26:11.304 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:53.523395+0000", 2024-09-13T11:26:11.304 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.5202e-05 2024-09-13T11:26:11.304 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.304 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.304 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:26:11.304 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:53.523409+0000", 2024-09-13T11:26:11.304 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4263999999999999e-05 2024-09-13T11:26:11.304 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.304 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.304 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:26:11.305 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:53.523477+0000", 2024-09-13T11:26:11.305 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.7578999999999998e-05 2024-09-13T11:26:11.305 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.305 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.305 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:26:11.305 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:53.523638+0000", 2024-09-13T11:26:11.305 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00016078300000000001 2024-09-13T11:26:11.305 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.305 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.305 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 0", 2024-09-13T11:26:11.305 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:53.523800+0000", 2024-09-13T11:26:11.305 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000162564 2024-09-13T11:26:11.305 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.305 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.306 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:26:11.306 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:53.526960+0000", 2024-09-13T11:26:11.306 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0031593950000000002 2024-09-13T11:26:11.306 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.306 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.306 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:11.306 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:53.706180+0000", 2024-09-13T11:26:11.306 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.17922080600000001 2024-09-13T11:26:11.306 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.306 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.306 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:26:11.306 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:53.706237+0000", 2024-09-13T11:26:11.306 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.6845000000000002e-05 2024-09-13T11:26:11.307 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.307 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.307 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:26:11.307 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:53.706292+0000", 2024-09-13T11:26:11.307 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.5254000000000003e-05 2024-09-13T11:26:11.307 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:11.307 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:26:11.307 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:11.307 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.307 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.307 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop_reply(client.4665.0:137473 3.10 e139/84 ondisk, result = 0)", 2024-09-13T11:26:11.307 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:24:53.527076+0000", 2024-09-13T11:26:11.307 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 77.697802077999995, 2024-09-13T11:26:11.308 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.17922728099999999, 2024-09-13T11:26:11.308 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:26:11.308 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:26:11.308 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:26:11.308 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:26:11.308 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.308 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:26:11.308 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:53.527076+0000", 2024-09-13T11:26:11.308 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:11.308 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.308 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.308 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:26:11.308 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:53.527076+0000", 2024-09-13T11:26:11.309 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:11.309 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.309 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.309 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:26:11.309 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:53.527078+0000", 2024-09-13T11:26:11.309 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.9000000000000002e-06 2024-09-13T11:26:11.309 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.309 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.309 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:26:11.309 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:53.527088+0000", 2024-09-13T11:26:11.309 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.7240000000000004e-06 2024-09-13T11:26:11.309 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.309 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.310 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:26:11.310 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:53.706095+0000", 2024-09-13T11:26:11.310 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.17900633899999999 2024-09-13T11:26:11.310 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.310 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.310 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:26:11.310 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:53.706101+0000", 2024-09-13T11:26:11.310 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.7730000000000002e-06 2024-09-13T11:26:11.310 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.310 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.310 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:26:11.310 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:53.706168+0000", 2024-09-13T11:26:11.311 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.6225999999999999e-05 2024-09-13T11:26:11.311 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.311 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.311 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:26:11.311 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:53.706175+0000", 2024-09-13T11:26:11.311 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.5560000000000002e-06 2024-09-13T11:26:11.311 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.311 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.311 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:26:11.311 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:53.706303+0000", 2024-09-13T11:26:11.311 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000127763 2024-09-13T11:26:11.312 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:11.312 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:26:11.312 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:11.312 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.312 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.312 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:137780 3.8 3:1047f268:::benchmark_data_smithi177_31116_object137779:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e139)", 2024-09-13T11:26:11.312 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:24:53.795212+0000", 2024-09-13T11:26:11.312 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 77.429666159999996, 2024-09-13T11:26:11.312 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.21530981399999999, 2024-09-13T11:26:11.312 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:26:11.312 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:26:11.312 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:11.312 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:26:11.313 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:26:11.313 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:11.313 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 137780 2024-09-13T11:26:11.313 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.313 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:26:11.313 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.313 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:26:11.313 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:53.795212+0000", 2024-09-13T11:26:11.313 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:11.313 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.313 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.313 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:26:11.313 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:53.795212+0000", 2024-09-13T11:26:11.314 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:11.314 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.314 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.314 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:26:11.314 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:53.795214+0000", 2024-09-13T11:26:11.314 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.481e-06 2024-09-13T11:26:11.314 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.314 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.314 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:26:11.314 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:53.795247+0000", 2024-09-13T11:26:11.314 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.3105999999999998e-05 2024-09-13T11:26:11.314 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.314 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.315 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:26:11.315 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:53.795258+0000", 2024-09-13T11:26:11.315 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0872e-05 2024-09-13T11:26:11.315 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.315 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.315 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:26:11.315 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:53.795265+0000", 2024-09-13T11:26:11.315 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.5729999999999996e-06 2024-09-13T11:26:11.315 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.315 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.315 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:26:11.315 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:53.795309+0000", 2024-09-13T11:26:11.315 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.4617999999999999e-05 2024-09-13T11:26:11.316 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.316 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.316 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:26:11.316 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:53.795397+0000", 2024-09-13T11:26:11.316 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.7863999999999998e-05 2024-09-13T11:26:11.316 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.316 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.316 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 1", 2024-09-13T11:26:11.316 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:53.795493+0000", 2024-09-13T11:26:11.316 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.6062000000000003e-05 2024-09-13T11:26:11.316 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.317 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.317 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:26:11.317 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:53.797245+0000", 2024-09-13T11:26:11.317 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0017521559999999999 2024-09-13T11:26:11.317 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.317 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.317 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:11.317 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:54.010409+0000", 2024-09-13T11:26:11.317 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.21316385299999999 2024-09-13T11:26:11.317 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.317 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.317 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:26:11.317 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:54.010474+0000", 2024-09-13T11:26:11.318 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.4998000000000003e-05 2024-09-13T11:26:11.318 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.318 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.318 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:26:11.318 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:54.010521+0000", 2024-09-13T11:26:11.318 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.7231000000000002e-05 2024-09-13T11:26:11.318 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:11.318 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:26:11.318 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:11.318 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.318 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.318 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop_reply(client.4665.0:137780 3.8 e139/117 ondisk, result = 0)", 2024-09-13T11:26:11.318 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:24:53.797821+0000", 2024-09-13T11:26:11.319 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 77.427056261999994, 2024-09-13T11:26:11.319 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.21270765599999999, 2024-09-13T11:26:11.319 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:26:11.319 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:26:11.319 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:26:11.319 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:26:11.319 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.319 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:26:11.319 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:53.797821+0000", 2024-09-13T11:26:11.319 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:11.319 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.319 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.319 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:26:11.319 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:53.797821+0000", 2024-09-13T11:26:11.320 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:11.320 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.320 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.320 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:26:11.320 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:53.797822+0000", 2024-09-13T11:26:11.320 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.012e-06 2024-09-13T11:26:11.320 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.320 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.320 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:26:11.320 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:53.797829+0000", 2024-09-13T11:26:11.320 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.8009999999999998e-06 2024-09-13T11:26:11.320 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.321 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.321 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:26:11.321 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:54.010334+0000", 2024-09-13T11:26:11.321 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.212504477 2024-09-13T11:26:11.321 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.321 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.321 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:26:11.321 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:54.010342+0000", 2024-09-13T11:26:11.321 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.6549999999999993e-06 2024-09-13T11:26:11.321 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.321 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.321 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:26:11.321 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:54.010394+0000", 2024-09-13T11:26:11.322 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.1875000000000001e-05 2024-09-13T11:26:11.322 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.322 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.322 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:26:11.322 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:54.010402+0000", 2024-09-13T11:26:11.322 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.6499999999999996e-06 2024-09-13T11:26:11.322 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.322 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.322 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:26:11.322 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:54.010529+0000", 2024-09-13T11:26:11.322 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000127186 2024-09-13T11:26:11.322 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:11.322 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:26:11.323 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:11.323 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.323 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.323 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.4665.0:196211 3.6 e178/122 3:63573deb:::benchmark_data_smithi177_31116_object55908:head v 178'6255, mlcod=178'6252)", 2024-09-13T11:26:11.323 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:25:43.175295+0000", 2024-09-13T11:26:11.323 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 28.049582932, 2024-09-13T11:26:11.323 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.167837029, 2024-09-13T11:26:11.323 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:26:11.323 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:26:11.323 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:11.323 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:26:11.323 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.323 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:26:11.324 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:43.175295+0000", 2024-09-13T11:26:11.324 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:11.324 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.324 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.324 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:26:11.324 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:43.175295+0000", 2024-09-13T11:26:11.324 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:11.324 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.324 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.324 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:26:11.324 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:43.175300+0000", 2024-09-13T11:26:11.324 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.9099999999999996e-06 2024-09-13T11:26:11.324 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.325 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.325 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:26:11.325 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:43.175318+0000", 2024-09-13T11:26:11.325 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.8346e-05 2024-09-13T11:26:11.325 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.325 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.325 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:26:11.325 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:43.340757+0000", 2024-09-13T11:26:11.325 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.165439058 2024-09-13T11:26:11.325 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.325 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.325 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:26:11.325 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:43.340775+0000", 2024-09-13T11:26:11.326 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.7546000000000001e-05 2024-09-13T11:26:11.326 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.326 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.326 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:26:11.326 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:43.340855+0000", 2024-09-13T11:26:11.326 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.0207e-05 2024-09-13T11:26:11.326 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.326 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.326 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:26:11.326 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:43.340879+0000", 2024-09-13T11:26:11.326 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.4494e-05 2024-09-13T11:26:11.326 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.326 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.327 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:26:11.327 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:43.343057+0000", 2024-09-13T11:26:11.327 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0021772380000000002 2024-09-13T11:26:11.327 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.327 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.327 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:26:11.327 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:43.343132+0000", 2024-09-13T11:26:11.327 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.5229999999999996e-05 2024-09-13T11:26:11.327 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:11.327 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:26:11.327 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:11.327 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.327 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.328 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.4665.0:196314 3.6 e178/122 3:62e79b1c:::benchmark_data_smithi177_31116_object56011:head v 178'6256, mlcod=178'6253)", 2024-09-13T11:26:11.328 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:25:43.340693+0000", 2024-09-13T11:26:11.328 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 27.884184681000001, 2024-09-13T11:26:11.328 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.24109011699999999, 2024-09-13T11:26:11.328 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:26:11.328 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:26:11.328 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:11.328 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:26:11.328 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.328 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:26:11.328 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:43.340693+0000", 2024-09-13T11:26:11.328 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:11.328 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.329 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.329 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:26:11.329 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:43.340693+0000", 2024-09-13T11:26:11.329 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:11.329 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.329 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.329 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:26:11.329 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:43.340704+0000", 2024-09-13T11:26:11.329 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1049e-05 2024-09-13T11:26:11.329 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.329 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.329 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:26:11.329 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:43.340718+0000", 2024-09-13T11:26:11.330 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4214e-05 2024-09-13T11:26:11.330 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.330 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.330 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:26:11.330 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:43.579964+0000", 2024-09-13T11:26:11.330 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.23924590200000001 2024-09-13T11:26:11.330 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.330 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.330 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:26:11.330 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:43.579972+0000", 2024-09-13T11:26:11.330 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.2789999999999998e-06 2024-09-13T11:26:11.330 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.330 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.331 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:26:11.331 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:43.580043+0000", 2024-09-13T11:26:11.331 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.0856000000000001e-05 2024-09-13T11:26:11.331 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.331 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.331 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:26:11.331 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:43.580065+0000", 2024-09-13T11:26:11.331 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.1795999999999999e-05 2024-09-13T11:26:11.331 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.331 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.331 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:26:11.331 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:43.581741+0000", 2024-09-13T11:26:11.331 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0016759190000000001 2024-09-13T11:26:11.331 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.332 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.332 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:26:11.332 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:43.581783+0000", 2024-09-13T11:26:11.332 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.2101999999999998e-05 2024-09-13T11:26:11.332 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:11.332 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:26:11.332 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:11.332 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:11.332 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:26:11.332 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-13T11:26:11.334 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-09-13T11:26:11.334 INFO:tasks.ceph.osd.12.smithi001.stderr:2024-09-13T11:26:11.330+0000 7f892c6e6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26314) UID: 0 2024-09-13T11:26:11.434 INFO:tasks.ceph.osd.10.smithi162.stderr:2024-09-13T11:26:11.431+0000 7fb59c9f0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26317) UID: 0 2024-09-13T11:26:11.515 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-13T11:26:11.515 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [], 2024-09-13T11:26:11.515 INFO:teuthology.orchestra.run.smithi162.stdout: "num_ops": 0 2024-09-13T11:26:11.516 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-13T11:26:11.531 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-09-13T11:26:11.536 INFO:tasks.ceph.osd.12.smithi001.stderr:2024-09-13T11:26:11.534+0000 7f892c6e6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26314) UID: 0 2024-09-13T11:26:11.636 INFO:tasks.ceph.osd.8.smithi001.stderr:2024-09-13T11:26:11.634+0000 7fb9b48ae640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26312) UID: 0 2024-09-13T11:26:11.638 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T11:26:11.723 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-13T11:26:11.723 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [], 2024-09-13T11:26:11.723 INFO:teuthology.orchestra.run.smithi162.stdout: "complaint_time": 30, 2024-09-13T11:26:11.723 INFO:teuthology.orchestra.run.smithi162.stdout: "num_blocked_ops": 0 2024-09-13T11:26:11.724 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-13T11:26:11.737 INFO:tasks.ceph.osd.12.smithi001.stderr:2024-09-13T11:26:11.734+0000 7f892c6e6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26314) UID: 0 2024-09-13T11:26:11.741 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-09-13T11:26:11.824 INFO:teuthology.orchestra.run.smithi001.stdout:osd.0: {} 2024-09-13T11:26:11.825 INFO:teuthology.orchestra.run.smithi001.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T11:26:11.840 INFO:tasks.ceph.osd.9.smithi092.stderr:2024-09-13T11:26:11.836+0000 7ff5ab3d0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26348) UID: 0 2024-09-13T11:26:11.855 INFO:teuthology.orchestra.run.smithi001.stdout:osd.1: {} 2024-09-13T11:26:11.855 INFO:teuthology.orchestra.run.smithi001.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T11:26:11.883 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-13T11:26:11.883 INFO:teuthology.orchestra.run.smithi162.stdout: "size": 20, 2024-09-13T11:26:11.883 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 600, 2024-09-13T11:26:11.883 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [ 2024-09-13T11:26:11.883 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.883 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:45810 3.17 3.9bd13e57 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e69)", 2024-09-13T11:26:11.884 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:21.347960+0000", 2024-09-13T11:26:11.884 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 170.53427657500001, 2024-09-13T11:26:11.884 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.56814464200000003, 2024-09-13T11:26:11.884 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:26:11.884 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:26:11.884 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-09-13T11:26:11.884 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:26:11.884 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:26:11.884 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:11.884 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 45810 2024-09-13T11:26:11.884 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.884 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:26:11.885 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.885 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:26:11.885 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:21.347960+0000", 2024-09-13T11:26:11.885 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:11.885 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.885 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.885 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:26:11.885 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:21.347960+0000", 2024-09-13T11:26:11.885 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:11.885 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.885 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.885 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:26:11.885 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:21.347965+0000", 2024-09-13T11:26:11.886 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.6720000000000003e-06 2024-09-13T11:26:11.886 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.886 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.886 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:26:11.886 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:21.348019+0000", 2024-09-13T11:26:11.886 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.4391000000000002e-05 2024-09-13T11:26:11.886 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.886 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.886 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:26:11.886 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:21.348052+0000", 2024-09-13T11:26:11.886 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.2246000000000003e-05 2024-09-13T11:26:11.886 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.886 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.887 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:26:11.887 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:21.348072+0000", 2024-09-13T11:26:11.887 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.0622e-05 2024-09-13T11:26:11.887 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.887 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.887 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:26:11.887 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:21.348278+0000", 2024-09-13T11:26:11.887 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000205391 2024-09-13T11:26:11.887 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.887 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.887 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-09-13T11:26:11.887 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:21.348292+0000", 2024-09-13T11:26:11.887 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4759e-05 2024-09-13T11:26:11.887 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.888 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.888 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:26:11.888 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:21.916064+0000", 2024-09-13T11:26:11.888 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.56777128600000004 2024-09-13T11:26:11.888 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.888 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.888 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:26:11.888 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:21.916105+0000", 2024-09-13T11:26:11.888 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.1275e-05 2024-09-13T11:26:11.888 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:11.888 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:26:11.888 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:11.889 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.889 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.889 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:54653 3.23 3:c7a54800:::benchmark_data_smithi177_31116_object54652:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e75)", 2024-09-13T11:26:11.889 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:30.979059+0000", 2024-09-13T11:26:11.889 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 160.90317807599999, 2024-09-13T11:26:11.889 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.97284747100000002, 2024-09-13T11:26:11.889 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:26:11.889 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:26:11.889 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-09-13T11:26:11.889 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:26:11.889 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:26:11.889 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:11.889 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 54653 2024-09-13T11:26:11.890 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.890 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:26:11.890 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.890 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:26:11.890 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.979059+0000", 2024-09-13T11:26:11.890 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:11.890 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.890 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.890 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:26:11.890 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.979059+0000", 2024-09-13T11:26:11.890 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:11.890 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.890 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.891 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:26:11.891 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.979061+0000", 2024-09-13T11:26:11.891 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.4279999999999999e-06 2024-09-13T11:26:11.891 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.891 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.891 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:26:11.891 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.979114+0000", 2024-09-13T11:26:11.891 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.3254000000000002e-05 2024-09-13T11:26:11.891 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.891 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.891 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:26:11.891 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.979157+0000", 2024-09-13T11:26:11.891 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.2018000000000003e-05 2024-09-13T11:26:11.892 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.892 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.892 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:26:11.892 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.979172+0000", 2024-09-13T11:26:11.892 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5636000000000001e-05 2024-09-13T11:26:11.892 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.892 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.892 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:26:11.892 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:31.045109+0000", 2024-09-13T11:26:11.892 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.065936982000000005 2024-09-13T11:26:11.892 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.892 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.892 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-09-13T11:26:11.893 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:31.045121+0000", 2024-09-13T11:26:11.893 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1608e-05 2024-09-13T11:26:11.893 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.893 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.893 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:26:11.893 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:31.951636+0000", 2024-09-13T11:26:11.893 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.90651503899999997 2024-09-13T11:26:11.893 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.893 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.893 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:26:11.893 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:31.951906+0000", 2024-09-13T11:26:11.893 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000270506 2024-09-13T11:26:11.894 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:11.894 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:26:11.894 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:11.894 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.894 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.894 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPush(3.18 76/70 [PushOp(3:1a32c9cd:::benchmark_data_smithi177_31116_object12833:head, version: 49'818, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:1a32c9cd:::benchmark_data_smithi177_31116_object12833:head@49'818, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:26:11.894 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:32.522305+0000", 2024-09-13T11:26:11.894 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 159.35993199200001, 2024-09-13T11:26:11.894 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.40201207100000003, 2024-09-13T11:26:11.894 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:26:11.894 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:26:11.894 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:26:11.894 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:26:11.894 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.895 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:26:11.895 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.522305+0000", 2024-09-13T11:26:11.895 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:11.895 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.895 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.895 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:26:11.895 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.522305+0000", 2024-09-13T11:26:11.895 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:11.895 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.895 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.895 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:26:11.895 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.522309+0000", 2024-09-13T11:26:11.895 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.9829999999999998e-06 2024-09-13T11:26:11.895 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.896 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.896 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:26:11.896 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.522377+0000", 2024-09-13T11:26:11.896 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.8069000000000002e-05 2024-09-13T11:26:11.896 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.896 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.896 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:26:11.896 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.921950+0000", 2024-09-13T11:26:11.896 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.399573438 2024-09-13T11:26:11.896 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.896 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.896 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:26:11.896 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.921983+0000", 2024-09-13T11:26:11.897 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.2858000000000003e-05 2024-09-13T11:26:11.897 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.897 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.897 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:26:11.897 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.922389+0000", 2024-09-13T11:26:11.897 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00040612999999999999 2024-09-13T11:26:11.897 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.897 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.897 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:26:11.897 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.922421+0000", 2024-09-13T11:26:11.897 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.1356999999999998e-05 2024-09-13T11:26:11.897 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.898 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.898 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:26:11.898 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.924317+0000", 2024-09-13T11:26:11.898 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0018962359999999999 2024-09-13T11:26:11.898 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:11.898 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:26:11.898 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:11.898 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.898 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.898 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPush(3.17 76/70 [PushOp(3:ebcbce67:::benchmark_data_smithi177_31116_object18268:head, version: 53'1102, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:ebcbce67:::benchmark_data_smithi177_31116_object18268:head@53'1102, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:26:11.898 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:32.638908+0000", 2024-09-13T11:26:11.898 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 159.24332854400001, 2024-09-13T11:26:11.898 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.229064031, 2024-09-13T11:26:11.899 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:26:11.899 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:26:11.899 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:26:11.899 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:26:11.899 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.899 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:26:11.899 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.638908+0000", 2024-09-13T11:26:11.899 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:11.899 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.899 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.899 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:26:11.899 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.638908+0000", 2024-09-13T11:26:11.900 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:11.900 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.900 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.900 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:26:11.900 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.638910+0000", 2024-09-13T11:26:11.900 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.7379999999999999e-06 2024-09-13T11:26:11.900 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.900 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.900 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:26:11.900 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.638975+0000", 2024-09-13T11:26:11.900 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.4937000000000005e-05 2024-09-13T11:26:11.900 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.900 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.901 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:26:11.901 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.865860+0000", 2024-09-13T11:26:11.901 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.22688509800000001 2024-09-13T11:26:11.901 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.901 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.901 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:26:11.901 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.865890+0000", 2024-09-13T11:26:11.901 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.0273e-05 2024-09-13T11:26:11.901 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.901 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.901 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:26:11.901 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.866143+0000", 2024-09-13T11:26:11.902 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00025212800000000001 2024-09-13T11:26:11.902 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.902 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.902 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:26:11.902 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.866181+0000", 2024-09-13T11:26:11.902 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.8945000000000001e-05 2024-09-13T11:26:11.902 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.902 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.902 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:26:11.902 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.867972+0000", 2024-09-13T11:26:11.902 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001790912 2024-09-13T11:26:11.902 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:11.902 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:26:11.902 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:11.903 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.903 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.903 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:57435 3.3 3:c0db4777:::benchmark_data_smithi177_31116_object57434:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e80)", 2024-09-13T11:26:11.903 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:37.436034+0000", 2024-09-13T11:26:11.903 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 154.44620308399999, 2024-09-13T11:26:11.903 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.35949142000000001, 2024-09-13T11:26:11.903 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:26:11.903 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:26:11.903 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:11.903 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:26:11.903 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:26:11.903 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:11.903 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 57435 2024-09-13T11:26:11.904 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.904 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:26:11.904 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.904 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:26:11.904 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:37.436034+0000", 2024-09-13T11:26:11.904 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:11.904 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.904 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.904 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:26:11.904 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:37.436034+0000", 2024-09-13T11:26:11.904 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:11.904 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.904 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.905 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:26:11.905 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:37.436037+0000", 2024-09-13T11:26:11.905 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.7829999999999999e-06 2024-09-13T11:26:11.905 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.905 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.905 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:26:11.905 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:37.436090+0000", 2024-09-13T11:26:11.905 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.3665999999999997e-05 2024-09-13T11:26:11.905 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.905 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.905 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:26:11.905 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:37.436112+0000", 2024-09-13T11:26:11.905 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.1556000000000002e-05 2024-09-13T11:26:11.906 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.906 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.906 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:26:11.906 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:37.436122+0000", 2024-09-13T11:26:11.906 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0098000000000001e-05 2024-09-13T11:26:11.906 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.906 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.906 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:26:11.906 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:37.436187+0000", 2024-09-13T11:26:11.906 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.4857999999999995e-05 2024-09-13T11:26:11.906 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.906 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.906 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:26:11.907 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:37.436285+0000", 2024-09-13T11:26:11.907 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.8643999999999999e-05 2024-09-13T11:26:11.907 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.907 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.907 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 7", 2024-09-13T11:26:11.907 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:37.436423+0000", 2024-09-13T11:26:11.907 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00013809299999999999 2024-09-13T11:26:11.907 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.907 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.907 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:26:11.907 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:37.440125+0000", 2024-09-13T11:26:11.907 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0037015239999999999 2024-09-13T11:26:11.907 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.908 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.908 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:11.908 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:37.795397+0000", 2024-09-13T11:26:11.908 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.35527234400000002 2024-09-13T11:26:11.908 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.908 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.908 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:26:11.908 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:37.795485+0000", 2024-09-13T11:26:11.908 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.7538000000000001e-05 2024-09-13T11:26:11.908 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.908 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.908 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:26:11.908 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:37.795525+0000", 2024-09-13T11:26:11.909 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.0315999999999998e-05 2024-09-13T11:26:11.909 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:11.909 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:26:11.909 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:11.909 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.909 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.909 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:60433 3.3 3:c01f7ddc:::benchmark_data_smithi177_31116_object60432:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e85)", 2024-09-13T11:26:11.909 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:42.199382+0000", 2024-09-13T11:26:11.909 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 149.68285519299999, 2024-09-13T11:26:11.909 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.33918965200000001, 2024-09-13T11:26:11.909 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:26:11.909 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:26:11.909 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:11.910 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:26:11.910 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:26:11.910 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:11.910 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 60433 2024-09-13T11:26:11.910 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.910 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:26:11.910 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.910 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:26:11.910 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.199382+0000", 2024-09-13T11:26:11.910 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:11.910 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.910 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.910 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:26:11.910 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.199382+0000", 2024-09-13T11:26:11.911 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:11.911 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.911 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.911 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:26:11.911 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.199386+0000", 2024-09-13T11:26:11.911 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.8759999999999998e-06 2024-09-13T11:26:11.911 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.911 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.911 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:26:11.911 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.199455+0000", 2024-09-13T11:26:11.911 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.9305000000000001e-05 2024-09-13T11:26:11.911 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.911 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.912 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:26:11.912 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.199482+0000", 2024-09-13T11:26:11.912 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.6749000000000001e-05 2024-09-13T11:26:11.912 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.912 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.912 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:26:11.912 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.199503+0000", 2024-09-13T11:26:11.912 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.1137000000000001e-05 2024-09-13T11:26:11.912 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.912 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.912 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:26:11.912 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.199602+0000", 2024-09-13T11:26:11.913 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.8743e-05 2024-09-13T11:26:11.913 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.913 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.913 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:26:11.913 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.199784+0000", 2024-09-13T11:26:11.913 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00018265100000000001 2024-09-13T11:26:11.913 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.913 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.913 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 7", 2024-09-13T11:26:11.913 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.199955+0000", 2024-09-13T11:26:11.913 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00017067100000000001 2024-09-13T11:26:11.913 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.913 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.914 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:26:11.914 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.202075+0000", 2024-09-13T11:26:11.914 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0021206749999999998 2024-09-13T11:26:11.914 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.914 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.914 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:11.914 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.538395+0000", 2024-09-13T11:26:11.914 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.33631963599999998 2024-09-13T11:26:11.914 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.914 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.914 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:26:11.914 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.538498+0000", 2024-09-13T11:26:11.914 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00010267099999999999 2024-09-13T11:26:11.914 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.915 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.915 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:26:11.915 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.538571+0000", 2024-09-13T11:26:11.915 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.3538e-05 2024-09-13T11:26:11.915 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:11.915 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:26:11.915 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:11.915 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.915 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.915 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop_reply(client.4665.0:60433 3.3 e85/75 ondisk, result = 0)", 2024-09-13T11:26:11.915 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:42.203186+0000", 2024-09-13T11:26:11.915 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 149.679051015, 2024-09-13T11:26:11.916 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.33540096699999999, 2024-09-13T11:26:11.916 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:26:11.916 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:26:11.916 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:26:11.916 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:26:11.916 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.916 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:26:11.916 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.203186+0000", 2024-09-13T11:26:11.916 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:11.916 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.916 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.916 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:26:11.916 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.203186+0000", 2024-09-13T11:26:11.917 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:11.917 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.917 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.917 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:26:11.917 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.203194+0000", 2024-09-13T11:26:11.917 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.0020000000000006e-06 2024-09-13T11:26:11.917 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.917 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.917 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:26:11.917 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.203202+0000", 2024-09-13T11:26:11.917 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.9829999999999996e-06 2024-09-13T11:26:11.917 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.917 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.918 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:26:11.918 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.538182+0000", 2024-09-13T11:26:11.918 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.33498029899999998 2024-09-13T11:26:11.918 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.918 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.918 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:26:11.918 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.538196+0000", 2024-09-13T11:26:11.918 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3807e-05 2024-09-13T11:26:11.918 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.918 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.918 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:26:11.918 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.538371+0000", 2024-09-13T11:26:11.918 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00017541200000000001 2024-09-13T11:26:11.919 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.919 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.919 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:26:11.919 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.538386+0000", 2024-09-13T11:26:11.919 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4833e-05 2024-09-13T11:26:11.919 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.919 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.919 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:26:11.919 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.538587+0000", 2024-09-13T11:26:11.919 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000200631 2024-09-13T11:26:11.919 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:11.919 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:26:11.919 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:11.919 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.920 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.920 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:83144 3.18 3:1c2a5085:::benchmark_data_smithi177_31116_object83143:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106)", 2024-09-13T11:26:11.920 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:24:04.186395+0000", 2024-09-13T11:26:11.920 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 127.695841896, 2024-09-13T11:26:11.920 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.29655527599999998, 2024-09-13T11:26:11.920 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:26:11.920 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:26:11.920 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:11.920 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:26:11.920 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:26:11.920 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:11.920 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 83144 2024-09-13T11:26:11.921 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.921 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:26:11.921 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.921 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:26:11.921 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.186395+0000", 2024-09-13T11:26:11.921 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:11.921 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.921 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.921 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:26:11.921 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.186395+0000", 2024-09-13T11:26:11.921 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:11.921 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.921 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.921 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:26:11.922 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.186398+0000", 2024-09-13T11:26:11.922 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.5299999999999999e-06 2024-09-13T11:26:11.922 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.922 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.922 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:26:11.922 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.186438+0000", 2024-09-13T11:26:11.922 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.0293000000000002e-05 2024-09-13T11:26:11.922 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.922 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.922 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:26:11.922 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.186450+0000", 2024-09-13T11:26:11.922 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2687000000000001e-05 2024-09-13T11:26:11.922 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.922 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.923 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:26:11.923 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.186456+0000", 2024-09-13T11:26:11.923 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.446e-06 2024-09-13T11:26:11.923 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.923 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.923 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:26:11.923 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.186503+0000", 2024-09-13T11:26:11.923 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.7491000000000003e-05 2024-09-13T11:26:11.923 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.923 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.923 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:26:11.923 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.186618+0000", 2024-09-13T11:26:11.923 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000114999 2024-09-13T11:26:11.924 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.924 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.924 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 0", 2024-09-13T11:26:11.924 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.186712+0000", 2024-09-13T11:26:11.924 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.3805000000000001e-05 2024-09-13T11:26:11.924 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.924 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.924 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:26:11.924 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.189753+0000", 2024-09-13T11:26:11.924 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.003040713 2024-09-13T11:26:11.924 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.924 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.924 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:11.925 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.482662+0000", 2024-09-13T11:26:11.925 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.29290945699999998 2024-09-13T11:26:11.925 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.925 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.925 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:26:11.925 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.482831+0000", 2024-09-13T11:26:11.925 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00016900699999999999 2024-09-13T11:26:11.925 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.925 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.925 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:26:11.925 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.482950+0000", 2024-09-13T11:26:11.925 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000118848 2024-09-13T11:26:11.925 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:11.926 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:26:11.926 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:11.926 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.926 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.926 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop_reply(client.4665.0:83144 3.18 e106/91 ondisk, result = 0)", 2024-09-13T11:26:11.926 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:24:04.188413+0000", 2024-09-13T11:26:11.926 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 127.693823886, 2024-09-13T11:26:11.926 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.294557562, 2024-09-13T11:26:11.926 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:26:11.926 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:26:11.926 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:26:11.926 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:26:11.926 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.926 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:26:11.927 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.188413+0000", 2024-09-13T11:26:11.927 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:11.927 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.927 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.927 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:26:11.927 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.188413+0000", 2024-09-13T11:26:11.927 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:11.927 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.927 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.927 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:26:11.927 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.188414+0000", 2024-09-13T11:26:11.927 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4309999999999999e-06 2024-09-13T11:26:11.927 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.928 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.928 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:26:11.928 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.188419+0000", 2024-09-13T11:26:11.928 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.2889999999999999e-06 2024-09-13T11:26:11.928 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.928 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.928 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:26:11.928 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.482442+0000", 2024-09-13T11:26:11.928 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.294023168 2024-09-13T11:26:11.928 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.928 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.928 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:26:11.928 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.482459+0000", 2024-09-13T11:26:11.928 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.7013000000000001e-05 2024-09-13T11:26:11.929 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.929 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.929 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:26:11.929 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.482615+0000", 2024-09-13T11:26:11.929 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00015620400000000001 2024-09-13T11:26:11.929 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.929 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.929 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:26:11.929 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.482644+0000", 2024-09-13T11:26:11.929 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.9093000000000001e-05 2024-09-13T11:26:11.929 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.929 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.929 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:26:11.929 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.482971+0000", 2024-09-13T11:26:11.930 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00032636399999999999 2024-09-13T11:26:11.930 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:11.930 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:26:11.930 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:11.930 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.930 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.930 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPush(3.1f 106/78 [PushOp(3:fd4be819:::benchmark_data_smithi177_31116_object54560:head, version: 74'3410, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:fd4be819:::benchmark_data_smithi177_31116_object54560:head@74'3410, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:26:11.930 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:24:04.397161+0000", 2024-09-13T11:26:11.930 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 127.485075477, 2024-09-13T11:26:11.930 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.51611096199999995, 2024-09-13T11:26:11.930 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:26:11.930 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:26:11.930 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:26:11.931 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:26:11.931 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.931 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:26:11.931 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.397161+0000", 2024-09-13T11:26:11.931 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:11.931 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.931 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.931 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:26:11.931 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.397161+0000", 2024-09-13T11:26:11.931 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:11.931 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.931 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.931 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:26:11.932 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.397163+0000", 2024-09-13T11:26:11.932 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4720000000000001e-06 2024-09-13T11:26:11.932 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.932 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.932 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:26:11.932 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.397228+0000", 2024-09-13T11:26:11.932 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.5141999999999994e-05 2024-09-13T11:26:11.932 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.932 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.932 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:26:11.932 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.911583+0000", 2024-09-13T11:26:11.932 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.51435473700000001 2024-09-13T11:26:11.932 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.933 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.933 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:26:11.933 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.911615+0000", 2024-09-13T11:26:11.933 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.2620000000000003e-05 2024-09-13T11:26:11.933 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.933 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.933 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:26:11.933 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.911759+0000", 2024-09-13T11:26:11.933 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00014376300000000001 2024-09-13T11:26:11.933 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.933 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.933 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:26:11.933 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.911779+0000", 2024-09-13T11:26:11.934 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.0003000000000001e-05 2024-09-13T11:26:11.934 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.934 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.934 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:26:11.934 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.913272+0000", 2024-09-13T11:26:11.934 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0014932249999999999 2024-09-13T11:26:11.934 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:11.934 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:26:11.934 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:11.934 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.934 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.934 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:84809 3.17 3:e9e52123:::benchmark_data_smithi177_31116_object84808:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-09-13T11:26:11.934 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:24:05.914353+0000", 2024-09-13T11:26:11.935 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 125.96788345900001, 2024-09-13T11:26:11.935 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.27210915899999999, 2024-09-13T11:26:11.935 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:26:11.935 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:26:11.935 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:11.935 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:26:11.935 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:26:11.935 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:11.935 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 84809 2024-09-13T11:26:11.935 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.935 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:26:11.935 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.935 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:26:11.936 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.914353+0000", 2024-09-13T11:26:11.936 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:11.936 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.936 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.936 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:26:11.936 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.914353+0000", 2024-09-13T11:26:11.936 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:11.936 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.936 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.936 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:26:11.936 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.914356+0000", 2024-09-13T11:26:11.936 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.8710000000000001e-06 2024-09-13T11:26:11.937 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.937 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.937 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:26:11.937 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.914443+0000", 2024-09-13T11:26:11.937 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.7002000000000005e-05 2024-09-13T11:26:11.937 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.937 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.937 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:26:11.937 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.914461+0000", 2024-09-13T11:26:11.937 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.7691000000000001e-05 2024-09-13T11:26:11.937 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.937 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.937 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:26:11.938 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.914471+0000", 2024-09-13T11:26:11.938 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0447e-05 2024-09-13T11:26:11.938 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.938 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.938 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:26:11.938 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.914550+0000", 2024-09-13T11:26:11.938 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.8683000000000005e-05 2024-09-13T11:26:11.938 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.938 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.938 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:26:11.938 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.914709+0000", 2024-09-13T11:26:11.938 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00015913200000000001 2024-09-13T11:26:11.938 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.938 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.939 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 3", 2024-09-13T11:26:11.939 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.914880+0000", 2024-09-13T11:26:11.939 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00017071499999999999 2024-09-13T11:26:11.939 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.939 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.939 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:26:11.939 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.916701+0000", 2024-09-13T11:26:11.939 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0018205369999999999 2024-09-13T11:26:11.939 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.939 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.939 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:11.939 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:06.186401+0000", 2024-09-13T11:26:11.940 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.26970018699999998 2024-09-13T11:26:11.940 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.940 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.940 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:26:11.940 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:06.186434+0000", 2024-09-13T11:26:11.940 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.3448999999999998e-05 2024-09-13T11:26:11.940 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.940 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.940 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:26:11.940 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:06.186463+0000", 2024-09-13T11:26:11.940 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.8445e-05 2024-09-13T11:26:11.940 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:11.940 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:26:11.941 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:11.941 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.941 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.941 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop_reply(client.4665.0:84809 3.17 e107/85 ondisk, result = 0)", 2024-09-13T11:26:11.941 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:24:05.916779+0000", 2024-09-13T11:26:11.941 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 125.965457988, 2024-09-13T11:26:11.941 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.26968993499999999, 2024-09-13T11:26:11.941 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:26:11.941 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:26:11.941 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:26:11.941 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:26:11.941 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.941 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:26:11.941 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.916779+0000", 2024-09-13T11:26:11.942 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:11.942 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.942 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.942 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:26:11.942 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.916779+0000", 2024-09-13T11:26:11.942 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:11.942 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.942 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.942 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:26:11.942 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.916780+0000", 2024-09-13T11:26:11.942 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.328e-06 2024-09-13T11:26:11.942 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.942 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.942 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:26:11.943 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.916789+0000", 2024-09-13T11:26:11.943 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.4460000000000001e-06 2024-09-13T11:26:11.943 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.943 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.943 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:26:11.943 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:06.186296+0000", 2024-09-13T11:26:11.943 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.26950707099999999 2024-09-13T11:26:11.943 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.943 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.943 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:26:11.943 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:06.186306+0000", 2024-09-13T11:26:11.943 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0137e-05 2024-09-13T11:26:11.943 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.943 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.944 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:26:11.944 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:06.186388+0000", 2024-09-13T11:26:11.944 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.2149000000000002e-05 2024-09-13T11:26:11.944 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.944 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.944 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:26:11.944 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:06.186397+0000", 2024-09-13T11:26:11.944 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.5210000000000002e-06 2024-09-13T11:26:11.944 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.945 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.945 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:26:11.945 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:06.186469+0000", 2024-09-13T11:26:11.945 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.2282999999999999e-05 2024-09-13T11:26:11.945 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:11.945 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:26:11.945 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:11.945 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.945 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.945 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPush(3.2a 121/120 [PushOp(3:5595108b:::benchmark_data_smithi177_31116_object899:head, version: 43'62, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:5595108b:::benchmark_data_smithi177_31116_object899:head@43'62, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:26:11.945 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:24:27.352355+0000", 2024-09-13T11:26:11.945 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 104.52988218199999, 2024-09-13T11:26:11.945 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.61628666099999996, 2024-09-13T11:26:11.946 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:26:11.946 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:26:11.946 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:26:11.946 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:26:11.946 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.946 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:26:11.946 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:27.352355+0000", 2024-09-13T11:26:11.946 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:11.946 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.946 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.946 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:26:11.946 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:27.352355+0000", 2024-09-13T11:26:11.946 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:11.947 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.947 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.947 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:26:11.947 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:27.352357+0000", 2024-09-13T11:26:11.947 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.328e-06 2024-09-13T11:26:11.947 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.947 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.947 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:26:11.947 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:27.352428+0000", 2024-09-13T11:26:11.947 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.1118999999999995e-05 2024-09-13T11:26:11.947 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.947 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.947 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:26:11.948 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:27.966950+0000", 2024-09-13T11:26:11.948 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.61452202499999997 2024-09-13T11:26:11.948 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.948 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.948 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:26:11.948 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:27.966980+0000", 2024-09-13T11:26:11.948 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.9433000000000001e-05 2024-09-13T11:26:11.948 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.948 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.948 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:26:11.948 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:27.967220+0000", 2024-09-13T11:26:11.948 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00024065499999999999 2024-09-13T11:26:11.948 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.949 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.949 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:26:11.950 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:27.967244+0000", 2024-09-13T11:26:11.950 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.4216e-05 2024-09-13T11:26:11.950 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.950 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.950 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:26:11.950 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:27.968641+0000", 2024-09-13T11:26:11.950 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0013968850000000001 2024-09-13T11:26:11.950 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:11.950 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:26:11.950 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:11.950 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.950 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.951 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:162057 3.1f 3:f8710094:::benchmark_data_smithi177_31116_object21754:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e156)", 2024-09-13T11:26:11.951 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:25:15.442781+0000", 2024-09-13T11:26:11.951 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 56.439455539000001, 2024-09-13T11:26:11.951 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.25670282500000002, 2024-09-13T11:26:11.951 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:26:11.951 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:26:11.951 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:11.951 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:26:11.951 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:26:11.951 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:11.951 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 162057 2024-09-13T11:26:11.951 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.951 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:26:11.952 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.952 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:26:11.952 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:15.442781+0000", 2024-09-13T11:26:11.952 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:11.952 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.952 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.952 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:26:11.952 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:15.442781+0000", 2024-09-13T11:26:11.952 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:11.952 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.952 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.952 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:26:11.952 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:15.442785+0000", 2024-09-13T11:26:11.953 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.4000000000000001e-06 2024-09-13T11:26:11.953 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.953 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.953 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:26:11.953 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:15.442795+0000", 2024-09-13T11:26:11.953 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0193999999999999e-05 2024-09-13T11:26:11.953 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.953 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.953 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:26:11.953 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:15.442820+0000", 2024-09-13T11:26:11.953 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.4683000000000001e-05 2024-09-13T11:26:11.953 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.953 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.954 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:26:11.954 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:15.442840+0000", 2024-09-13T11:26:11.954 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.0596000000000001e-05 2024-09-13T11:26:11.954 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.954 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.954 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:26:11.954 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:15.442952+0000", 2024-09-13T11:26:11.954 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00011215899999999999 2024-09-13T11:26:11.954 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.954 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.954 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:26:11.954 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:15.443143+0000", 2024-09-13T11:26:11.955 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00019058899999999999 2024-09-13T11:26:11.955 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.955 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.955 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 5", 2024-09-13T11:26:11.955 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:15.443327+0000", 2024-09-13T11:26:11.955 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00018364499999999999 2024-09-13T11:26:11.955 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.955 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.955 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:26:11.955 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:15.446727+0000", 2024-09-13T11:26:11.955 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0034000689999999999 2024-09-13T11:26:11.955 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.955 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.955 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:11.956 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:15.699356+0000", 2024-09-13T11:26:11.956 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.25262958400000002 2024-09-13T11:26:11.956 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.956 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.956 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:26:11.956 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:15.699428+0000", 2024-09-13T11:26:11.956 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.1810000000000005e-05 2024-09-13T11:26:11.956 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.956 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.956 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:26:11.956 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:15.699484+0000", 2024-09-13T11:26:11.956 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.6096e-05 2024-09-13T11:26:11.956 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:11.957 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:26:11.957 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:11.957 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.957 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.957 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:162073 3.1f 3:fd1a00ae:::benchmark_data_smithi177_31116_object21770:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e156)", 2024-09-13T11:26:11.957 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:25:15.454353+0000", 2024-09-13T11:26:11.957 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 56.427883584, 2024-09-13T11:26:11.957 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.27516647, 2024-09-13T11:26:11.957 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:26:11.957 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:26:11.957 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:11.957 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:26:11.957 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:26:11.957 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:11.958 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 162073 2024-09-13T11:26:11.958 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.958 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:26:11.958 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.958 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:26:11.958 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:15.454353+0000", 2024-09-13T11:26:11.958 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:11.958 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.958 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.958 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:26:11.958 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:15.454353+0000", 2024-09-13T11:26:11.958 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:11.958 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.958 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.959 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:26:11.959 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:15.454358+0000", 2024-09-13T11:26:11.959 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.3499999999999999e-06 2024-09-13T11:26:11.959 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.959 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.959 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:26:11.959 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:15.454371+0000", 2024-09-13T11:26:11.959 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3594000000000001e-05 2024-09-13T11:26:11.959 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.959 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.959 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:26:11.959 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:15.454408+0000", 2024-09-13T11:26:11.960 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.6619999999999998e-05 2024-09-13T11:26:11.960 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.960 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.960 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:26:11.960 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:15.454431+0000", 2024-09-13T11:26:11.960 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.3546000000000001e-05 2024-09-13T11:26:11.960 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.960 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.960 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:26:11.960 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:15.454587+0000", 2024-09-13T11:26:11.960 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000155248 2024-09-13T11:26:11.960 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.960 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.961 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:26:11.961 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:15.454897+0000", 2024-09-13T11:26:11.961 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000309955 2024-09-13T11:26:11.961 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.961 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.961 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 5", 2024-09-13T11:26:11.961 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:15.455203+0000", 2024-09-13T11:26:11.961 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00030604999999999999 2024-09-13T11:26:11.961 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.961 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.961 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:26:11.961 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:15.460534+0000", 2024-09-13T11:26:11.961 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0053310199999999997 2024-09-13T11:26:11.961 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.962 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.962 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:11.962 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:15.729356+0000", 2024-09-13T11:26:11.962 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.26882267199999998 2024-09-13T11:26:11.962 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.962 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.962 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:26:11.962 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:15.729442+0000", 2024-09-13T11:26:11.962 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.5209000000000004e-05 2024-09-13T11:26:11.962 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.962 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.962 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:26:11.962 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:15.729520+0000", 2024-09-13T11:26:11.962 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.8206000000000003e-05 2024-09-13T11:26:11.963 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:11.963 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:26:11.963 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:11.963 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.963 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.963 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:162102 3.1f 3:fc0cda6c:::benchmark_data_smithi177_31116_object21799:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e156)", 2024-09-13T11:26:11.963 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:25:15.477044+0000", 2024-09-13T11:26:11.963 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 56.405192730000003, 2024-09-13T11:26:11.963 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.28798895800000002, 2024-09-13T11:26:11.963 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:26:11.963 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:26:11.963 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:11.963 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:26:11.963 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:26:11.964 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:11.964 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 162102 2024-09-13T11:26:11.964 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.964 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:26:11.964 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.964 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:26:11.964 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:15.477044+0000", 2024-09-13T11:26:11.964 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:11.964 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.964 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.964 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:26:11.964 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:15.477044+0000", 2024-09-13T11:26:11.964 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:11.965 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.965 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.965 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:26:11.965 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:15.477048+0000", 2024-09-13T11:26:11.965 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.3569999999999998e-06 2024-09-13T11:26:11.965 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.965 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.965 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:26:11.965 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:15.477058+0000", 2024-09-13T11:26:11.965 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.9899999999999992e-06 2024-09-13T11:26:11.965 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.965 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.965 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:26:11.965 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:15.477087+0000", 2024-09-13T11:26:11.966 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.9026999999999998e-05 2024-09-13T11:26:11.966 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.966 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.966 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:26:11.966 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:15.477101+0000", 2024-09-13T11:26:11.966 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4583000000000001e-05 2024-09-13T11:26:11.966 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.966 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.966 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:26:11.966 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:15.477212+0000", 2024-09-13T11:26:11.966 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00011071200000000001 2024-09-13T11:26:11.966 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.966 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.967 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:26:11.967 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:15.477492+0000", 2024-09-13T11:26:11.967 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00027995400000000001 2024-09-13T11:26:11.967 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.967 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.967 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 5", 2024-09-13T11:26:11.967 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:15.477682+0000", 2024-09-13T11:26:11.967 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000189929 2024-09-13T11:26:11.967 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.967 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.967 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:26:11.967 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:15.481903+0000", 2024-09-13T11:26:11.967 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0042214649999999998 2024-09-13T11:26:11.967 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.968 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.968 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:11.968 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:15.764891+0000", 2024-09-13T11:26:11.968 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.28298753599999998 2024-09-13T11:26:11.968 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.968 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.968 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:26:11.968 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:15.764972+0000", 2024-09-13T11:26:11.968 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.1230000000000007e-05 2024-09-13T11:26:11.968 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.968 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.968 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:26:11.968 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:15.765033+0000", 2024-09-13T11:26:11.969 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.1174999999999996e-05 2024-09-13T11:26:11.969 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:11.969 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:26:11.969 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:11.969 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.969 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.969 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:191286 3.18 3:1ffc0b26:::benchmark_data_smithi177_31116_object50983:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e175)", 2024-09-13T11:26:11.969 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:25:39.152634+0000", 2024-09-13T11:26:11.969 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 32.729602548999999, 2024-09-13T11:26:11.969 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.236097576, 2024-09-13T11:26:11.969 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:26:11.969 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:26:11.969 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:11.969 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:26:11.970 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:26:11.970 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:11.970 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 191286 2024-09-13T11:26:11.970 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.970 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:26:11.970 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.970 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:26:11.970 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:39.152634+0000", 2024-09-13T11:26:11.970 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:11.970 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.970 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.970 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:26:11.970 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:39.152634+0000", 2024-09-13T11:26:11.971 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:11.971 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.971 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.971 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:26:11.971 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:39.152638+0000", 2024-09-13T11:26:11.971 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.5130000000000001e-06 2024-09-13T11:26:11.971 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.971 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.971 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:26:11.971 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:39.152649+0000", 2024-09-13T11:26:11.971 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0677e-05 2024-09-13T11:26:11.971 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.971 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.971 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:26:11.972 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:39.152678+0000", 2024-09-13T11:26:11.972 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.9436000000000001e-05 2024-09-13T11:26:11.972 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.972 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.972 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:26:11.972 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:39.152693+0000", 2024-09-13T11:26:11.972 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4823e-05 2024-09-13T11:26:11.972 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.972 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.972 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:26:11.972 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:39.152821+0000", 2024-09-13T11:26:11.972 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00012815299999999999 2024-09-13T11:26:11.972 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.973 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.973 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:26:11.973 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:39.153031+0000", 2024-09-13T11:26:11.973 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00021011399999999999 2024-09-13T11:26:11.973 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.973 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.973 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 0", 2024-09-13T11:26:11.973 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:39.153195+0000", 2024-09-13T11:26:11.973 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00016374599999999999 2024-09-13T11:26:11.973 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.973 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.973 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:26:11.973 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:39.157900+0000", 2024-09-13T11:26:11.974 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0047053049999999999 2024-09-13T11:26:11.974 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.974 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.974 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:11.974 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:39.388451+0000", 2024-09-13T11:26:11.974 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.230550957 2024-09-13T11:26:11.974 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.974 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.974 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:26:11.974 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:39.388611+0000", 2024-09-13T11:26:11.974 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00016029299999999999 2024-09-13T11:26:11.974 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.974 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.974 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:26:11.975 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:39.388732+0000", 2024-09-13T11:26:11.975 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000120559 2024-09-13T11:26:11.975 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:11.975 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:26:11.975 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:11.975 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.975 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.975 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:221532 3.17 3:ec6fb681:::benchmark_data_smithi177_31116_object81229:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e191)", 2024-09-13T11:26:11.975 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:26:04.303286+0000", 2024-09-13T11:26:11.975 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 7.5789509830000004, 2024-09-13T11:26:11.975 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.79092058600000004, 2024-09-13T11:26:11.975 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:26:11.975 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:26:11.976 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:11.976 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:26:11.976 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:26:11.976 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:11.976 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 221532 2024-09-13T11:26:11.976 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.976 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:26:11.976 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.976 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:26:11.976 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:04.303286+0000", 2024-09-13T11:26:11.976 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:11.976 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.976 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.976 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:26:11.977 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:04.303286+0000", 2024-09-13T11:26:11.977 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:11.977 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.977 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.977 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:26:11.977 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:04.303290+0000", 2024-09-13T11:26:11.977 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.9600000000000002e-06 2024-09-13T11:26:11.977 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.977 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.977 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:26:11.977 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:04.303302+0000", 2024-09-13T11:26:11.977 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2126e-05 2024-09-13T11:26:11.977 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.978 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.978 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:26:11.978 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:04.303332+0000", 2024-09-13T11:26:11.978 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.0015999999999999e-05 2024-09-13T11:26:11.978 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.978 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.978 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:26:11.978 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:04.303347+0000", 2024-09-13T11:26:11.978 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5324000000000001e-05 2024-09-13T11:26:11.978 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.978 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.978 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:26:11.978 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:04.303484+0000", 2024-09-13T11:26:11.979 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00013702199999999999 2024-09-13T11:26:11.979 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.979 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.979 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:26:11.979 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:04.303700+0000", 2024-09-13T11:26:11.979 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00021570500000000001 2024-09-13T11:26:11.979 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.979 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.979 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 3", 2024-09-13T11:26:11.979 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:04.303868+0000", 2024-09-13T11:26:11.980 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00016751799999999999 2024-09-13T11:26:11.980 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.980 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.980 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:26:11.980 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:04.309489+0000", 2024-09-13T11:26:11.980 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.005621455 2024-09-13T11:26:11.980 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.980 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.980 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:11.980 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:05.094014+0000", 2024-09-13T11:26:11.980 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.78452480499999999 2024-09-13T11:26:11.980 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.980 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.981 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:26:11.981 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:05.094132+0000", 2024-09-13T11:26:11.981 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000118093 2024-09-13T11:26:11.981 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.981 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.981 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:26:11.981 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:05.094206+0000", 2024-09-13T11:26:11.981 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.4561999999999995e-05 2024-09-13T11:26:11.981 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:11.981 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:26:11.981 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:11.981 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.982 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.982 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop_reply(client.4665.0:221532 3.17 e191/85 ondisk, result = 0)", 2024-09-13T11:26:11.982 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:26:04.309386+0000", 2024-09-13T11:26:11.982 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 7.5728506800000002, 2024-09-13T11:26:11.982 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.78483394299999998, 2024-09-13T11:26:11.982 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:26:11.982 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:26:11.982 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:26:11.982 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:26:11.982 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.982 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:26:11.982 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:04.309386+0000", 2024-09-13T11:26:11.982 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:11.982 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.983 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.983 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:26:11.983 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:04.309386+0000", 2024-09-13T11:26:11.983 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:11.983 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.983 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.983 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:26:11.983 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:04.309390+0000", 2024-09-13T11:26:11.983 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.5700000000000001e-06 2024-09-13T11:26:11.983 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.983 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.983 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:26:11.983 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:04.309403+0000", 2024-09-13T11:26:11.983 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3567000000000001e-05 2024-09-13T11:26:11.984 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.984 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.984 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:26:11.984 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:05.093783+0000", 2024-09-13T11:26:11.984 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.78437944699999995 2024-09-13T11:26:11.984 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.984 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.984 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:26:11.984 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:05.093802+0000", 2024-09-13T11:26:11.984 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.8816000000000001e-05 2024-09-13T11:26:11.984 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.984 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.984 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:26:11.985 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:05.093968+0000", 2024-09-13T11:26:11.985 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000166572 2024-09-13T11:26:11.985 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.985 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.985 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:26:11.985 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:05.093999+0000", 2024-09-13T11:26:11.985 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.0352999999999999e-05 2024-09-13T11:26:11.985 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.985 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.985 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:26:11.985 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:05.094220+0000", 2024-09-13T11:26:11.985 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000221618 2024-09-13T11:26:11.985 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:11.985 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:26:11.986 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:11.986 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.986 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.986 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:227356 3.18 3:1c24ab12:::benchmark_data_smithi177_31116_object87053:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e196)", 2024-09-13T11:26:11.986 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:26:10.982466+0000", 2024-09-13T11:26:11.986 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 0.89977057000000005, 2024-09-13T11:26:11.986 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.23166883999999999, 2024-09-13T11:26:11.986 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:26:11.986 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:26:11.986 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:11.986 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:26:11.986 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:26:11.986 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:11.986 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 227356 2024-09-13T11:26:11.987 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.987 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:26:11.987 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.987 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:26:11.987 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:10.982466+0000", 2024-09-13T11:26:11.987 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:11.987 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.987 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.987 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:26:11.987 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:10.982466+0000", 2024-09-13T11:26:11.987 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:11.987 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.987 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.987 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:26:11.988 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:10.982469+0000", 2024-09-13T11:26:11.988 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.2500000000000001e-06 2024-09-13T11:26:11.988 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.988 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.988 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:26:11.988 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:10.982479+0000", 2024-09-13T11:26:11.988 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0644e-05 2024-09-13T11:26:11.988 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.988 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.988 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:26:11.988 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:10.982494+0000", 2024-09-13T11:26:11.988 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4634000000000001e-05 2024-09-13T11:26:11.988 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.988 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.989 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:26:11.989 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:10.982502+0000", 2024-09-13T11:26:11.989 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.2120000000000003e-06 2024-09-13T11:26:11.989 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.989 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.989 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:26:11.989 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:10.982567+0000", 2024-09-13T11:26:11.989 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.5168999999999999e-05 2024-09-13T11:26:11.989 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.989 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.989 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:26:11.989 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:10.982715+0000", 2024-09-13T11:26:11.989 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00014747499999999999 2024-09-13T11:26:11.989 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.990 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.990 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 0", 2024-09-13T11:26:11.990 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:10.982839+0000", 2024-09-13T11:26:11.990 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00012430000000000001 2024-09-13T11:26:11.990 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.990 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.990 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:26:11.990 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:10.989491+0000", 2024-09-13T11:26:11.990 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0066519989999999996 2024-09-13T11:26:11.990 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.990 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.990 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:11.990 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:11.213989+0000", 2024-09-13T11:26:11.990 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.22449838999999999 2024-09-13T11:26:11.991 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.991 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.991 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:26:11.991 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:11.214049+0000", 2024-09-13T11:26:11.991 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.9431000000000001e-05 2024-09-13T11:26:11.991 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:11.991 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:11.991 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:26:11.991 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:11.214135+0000", 2024-09-13T11:26:11.991 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.6335999999999995e-05 2024-09-13T11:26:11.991 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:11.991 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:26:11.991 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:11.991 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:11.992 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:26:11.992 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-13T11:26:11.993 INFO:teuthology.orchestra.run.smithi001.stdout:osd.2: {} 2024-09-13T11:26:11.993 INFO:teuthology.orchestra.run.smithi001.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T11:26:11.994 INFO:teuthology.orchestra.run.smithi001.stdout:osd.3: {} 2024-09-13T11:26:11.994 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-13T11:26:11.995 INFO:teuthology.orchestra.run.smithi001.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T11:26:11.995 INFO:teuthology.orchestra.run.smithi001.stdout:osd.4: {} 2024-09-13T11:26:11.995 INFO:teuthology.orchestra.run.smithi001.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T11:26:11.995 INFO:tasks.ceph.osd.12.smithi001.stderr:2024-09-13T11:26:11.942+0000 7f891f10f640 -1 osd.12 197 handle_osd_map injecting map crc failure 2024-09-13T11:26:11.996 INFO:teuthology.orchestra.run.smithi001.stdout:osd.5: {} 2024-09-13T11:26:11.996 INFO:teuthology.orchestra.run.smithi001.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T11:26:11.996 INFO:tasks.ceph.osd.8.smithi001.stderr:2024-09-13T11:26:11.990+0000 7fb9b48ae640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26312) UID: 0 2024-09-13T11:26:12.002 INFO:teuthology.orchestra.run.smithi001.stdout:osd.6: {} 2024-09-13T11:26:12.003 INFO:teuthology.orchestra.run.smithi001.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T11:26:12.094 INFO:tasks.ceph.osd.11.smithi177.stderr:2024-09-13T11:26:12.089+0000 7f24792c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26580) UID: 0 2024-09-13T11:26:12.132 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-09-13T11:26:12.132 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:26:12.150 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-13T11:26:12.199 INFO:tasks.ceph.osd.15.smithi177.stderr:2024-09-13T11:26:12.197+0000 7f9d6ce77640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26578) UID: 0 2024-09-13T11:26:12.296 INFO:tasks.ceph.osd.2.smithi162.stderr:2024-09-13T11:26:12.291+0000 7fe92b35a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26313) UID: 0 2024-09-13T11:26:12.328 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-09-13T11:26:12.328 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:26:12.343 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-13T11:26:12.397 INFO:tasks.ceph.osd.14.smithi162.stderr:2024-09-13T11:26:12.395+0000 7f3df5903640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33154) UID: 0 2024-09-13T11:26:12.464 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-09-13T11:26:12.464 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:26:12.479 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-13T11:26:12.498 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-09-13T11:26:12.495+0000 7f97befc9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26315) UID: 0 2024-09-13T11:26:12.600 INFO:tasks.ceph.osd.10.smithi162.stderr:2024-09-13T11:26:12.595+0000 7fb59c9f0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26317) UID: 0 2024-09-13T11:26:12.608 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-09-13T11:26:12.608 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [ 2024-09-13T11:26:12.608 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:12.608 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:228893 3.1c e198/77 3:3ae6b532:::benchmark_data_smithi177_31116_object88590:head v 198'8945, mlcod=198'8943)", 2024-09-13T11:26:12.608 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:26:12.606556+0000", 2024-09-13T11:26:12.608 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 0.0020348419999999998, 2024-09-13T11:26:12.608 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0020678699999999999, 2024-09-13T11:26:12.608 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:26:12.608 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:26:12.608 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:26:12.608 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:26:12.609 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:12.609 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:26:12.609 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:12.606556+0000", 2024-09-13T11:26:12.609 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:12.609 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:12.609 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:12.609 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:26:12.609 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:12.606556+0000", 2024-09-13T11:26:12.609 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:12.609 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:12.609 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:12.609 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:26:12.609 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:12.606557+0000", 2024-09-13T11:26:12.610 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.699e-06 2024-09-13T11:26:12.610 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:12.610 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:12.610 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:26:12.610 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:12.606563+0000", 2024-09-13T11:26:12.610 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.8069999999999996e-06 2024-09-13T11:26:12.610 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:12.610 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:12.610 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:26:12.610 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:12.606579+0000", 2024-09-13T11:26:12.610 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.6291999999999999e-05 2024-09-13T11:26:12.610 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:12.610 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:12.611 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:26:12.611 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:12.606586+0000", 2024-09-13T11:26:12.611 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.7789999999999998e-06 2024-09-13T11:26:12.611 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:12.611 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:12.611 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:26:12.611 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:12.606636+0000", 2024-09-13T11:26:12.611 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.0179000000000002e-05 2024-09-13T11:26:12.611 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:12.611 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:12.611 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:26:12.611 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:12.606655+0000", 2024-09-13T11:26:12.611 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.8221000000000001e-05 2024-09-13T11:26:12.612 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:12.612 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:26:12.612 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:12.612 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:12.612 INFO:teuthology.orchestra.run.smithi177.stdout: ], 2024-09-13T11:26:12.612 INFO:teuthology.orchestra.run.smithi177.stdout: "num_ops": 1 2024-09-13T11:26:12.612 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-09-13T11:26:12.627 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-13T11:26:12.701 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-13T11:26:12.698+0000 7f6a68775640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26307) UID: 0 2024-09-13T11:26:12.751 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-09-13T11:26:12.752 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [], 2024-09-13T11:26:12.752 INFO:teuthology.orchestra.run.smithi177.stdout: "complaint_time": 30, 2024-09-13T11:26:12.752 INFO:teuthology.orchestra.run.smithi177.stdout: "num_blocked_ops": 0 2024-09-13T11:26:12.752 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-09-13T11:26:12.767 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-13T11:26:12.802 INFO:tasks.ceph.osd.4.smithi001.stderr:2024-09-13T11:26:12.798+0000 7ff8e0218640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26310) UID: 0 2024-09-13T11:26:12.838 INFO:teuthology.orchestra.run.smithi001.stdout:osd.7: {} 2024-09-13T11:26:12.839 INFO:teuthology.orchestra.run.smithi001.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T11:26:12.867 INFO:teuthology.orchestra.run.smithi001.stdout:osd.8: {} 2024-09-13T11:26:12.867 INFO:teuthology.orchestra.run.smithi001.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-13T11:26:12.894 INFO:teuthology.orchestra.run.smithi001.stdout:osd.9: {} 2024-09-13T11:26:12.894 INFO:teuthology.orchestra.run.smithi001.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-13T11:26:12.902 INFO:tasks.ceph.osd.7.smithi177.stderr:2024-09-13T11:26:12.901+0000 7f897b51d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26575) UID: 0 2024-09-13T11:26:12.903 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-09-13T11:26:12.903 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:26:12.921 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-09-13T11:26:12.928 INFO:teuthology.orchestra.run.smithi001.stdout:osd.10: {} 2024-09-13T11:26:12.929 INFO:teuthology.orchestra.run.smithi001.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-13T11:26:12.954 INFO:teuthology.orchestra.run.smithi001.stdout:osd.11: {} 2024-09-13T11:26:12.955 INFO:teuthology.orchestra.run.smithi001.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-13T11:26:12.979 INFO:teuthology.orchestra.run.smithi001.stdout:osd.12: {} 2024-09-13T11:26:12.980 INFO:teuthology.orchestra.run.smithi001.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-13T11:26:13.003 INFO:tasks.ceph.osd.8.smithi001.stderr:2024-09-13T11:26:12.998+0000 7fb9b48ae640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26312) UID: 0 2024-09-13T11:26:13.007 INFO:teuthology.orchestra.run.smithi001.stdout:osd.13: {} 2024-09-13T11:26:13.007 INFO:teuthology.orchestra.run.smithi001.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-13T11:26:13.030 INFO:teuthology.orchestra.run.smithi001.stdout:osd.14: {} 2024-09-13T11:26:13.030 INFO:teuthology.orchestra.run.smithi001.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-13T11:26:13.054 INFO:teuthology.orchestra.run.smithi001.stdout:osd.15: {} 2024-09-13T11:26:13.054 INFO:teuthology.orchestra.run.smithi001.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-13T11:26:13.093 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-09-13T11:26:13.093 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:26:13.104 INFO:tasks.ceph.osd.13.smithi092.stderr:2024-09-13T11:26:13.100+0000 7f3e5afea640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26350) UID: 0 2024-09-13T11:26:13.109 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-13T11:26:13.111 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-09-13T11:26:13.205 INFO:tasks.ceph.osd.15.smithi177.stderr:2024-09-13T11:26:13.201+0000 7f9d6ce77640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26578) UID: 0 2024-09-13T11:26:13.244 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-09-13T11:26:13.244 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:26:13.260 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-09-13T11:26:13.308 INFO:tasks.ceph.osd.5.smithi092.stderr:2024-09-13T11:26:13.304+0000 7f6f0b705640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26346) UID: 0 2024-09-13T11:26:13.407 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-09-13T11:26:13.403+0000 7f97befc9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26315) UID: 0 2024-09-13T11:26:13.456 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-09-13T11:26:13.456 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:26:13.477 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_ops_in_flight 2024-09-13T11:26:13.508 INFO:tasks.ceph.osd.13.smithi092.stderr:2024-09-13T11:26:13.504+0000 7f3e5afea640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26350) UID: 0 2024-09-13T11:26:13.609 INFO:tasks.ceph.osd.12.smithi001.stderr:2024-09-13T11:26:13.606+0000 7f892c6e6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26314) UID: 0 2024-09-13T11:26:13.621 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-09-13T11:26:13.621 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:26:13.639 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_blocked_ops 2024-09-13T11:26:13.709 INFO:tasks.ceph.osd.3.smithi177.stderr:2024-09-13T11:26:13.705+0000 7f10adba4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26574) UID: 0 2024-09-13T11:26:13.772 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-09-13T11:26:13.772 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:26:13.789 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_historic_ops 2024-09-13T11:26:13.811 INFO:tasks.ceph.osd.9.smithi092.stderr:2024-09-13T11:26:13.808+0000 7ff5ab3d0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26348) UID: 0 2024-09-13T11:26:13.911 INFO:tasks.ceph.osd.12.smithi001.stderr:2024-09-13T11:26:13.906+0000 7f892c6e6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26314) UID: 0 2024-09-13T11:26:13.965 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-09-13T11:26:13.965 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:26:13.986 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-13T11:26:14.012 INFO:tasks.ceph.osd.10.smithi162.stderr:2024-09-13T11:26:14.007+0000 7fb59c9f0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26317) UID: 0 2024-09-13T11:26:14.101 INFO:teuthology.orchestra.run.smithi001.stderr:noscrub is set 2024-09-13T11:26:14.105 INFO:tasks.ceph.osd.9.smithi092.stderr:2024-09-13T11:26:14.104+0000 7ff59ddf9640 -1 osd.9 198 handle_osd_map injecting map crc failure 2024-09-13T11:26:14.113 INFO:tasks.ceph.osd.3.smithi177.stderr:2024-09-13T11:26:14.109+0000 7f10adba4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26574) UID: 0 2024-09-13T11:26:14.124 INFO:teuthology.orchestra.run.smithi092.stdout:ERROR: (22) Invalid argument 2024-09-13T11:26:14.124 INFO:teuthology.orchestra.run.smithi092.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:26:14.139 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-13T11:26:14.214 INFO:tasks.ceph.osd.10.smithi162.stderr:2024-09-13T11:26:14.211+0000 7fb59c9f0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26317) UID: 0 2024-09-13T11:26:14.280 INFO:teuthology.orchestra.run.smithi092.stdout:ERROR: (22) Invalid argument 2024-09-13T11:26:14.281 INFO:teuthology.orchestra.run.smithi092.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:26:14.298 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-13T11:26:14.315 INFO:tasks.ceph.osd.15.smithi177.stderr:2024-09-13T11:26:14.313+0000 7f9d6ce77640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26578) UID: 0 2024-09-13T11:26:14.416 INFO:tasks.ceph.osd.3.smithi177.stderr:2024-09-13T11:26:14.413+0000 7f10adba4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26574) UID: 0 2024-09-13T11:26:14.438 INFO:teuthology.orchestra.run.smithi092.stdout:ERROR: (22) Invalid argument 2024-09-13T11:26:14.438 INFO:teuthology.orchestra.run.smithi092.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:26:14.453 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_ops_in_flight 2024-09-13T11:26:14.517 INFO:tasks.ceph.osd.8.smithi001.stderr:2024-09-13T11:26:14.514+0000 7fb9b48ae640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26312) UID: 0 2024-09-13T11:26:14.600 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-13T11:26:14.601 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:26:14.618 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_blocked_ops 2024-09-13T11:26:14.618 INFO:tasks.ceph.osd.14.smithi162.stderr:2024-09-13T11:26:14.615+0000 7f3df5903640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33154) UID: 0 2024-09-13T11:26:14.718 INFO:tasks.ceph.osd.14.smithi162.stderr:2024-09-13T11:26:14.715+0000 7f3df5903640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33154) UID: 0 2024-09-13T11:26:14.807 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-13T11:26:14.808 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:26:14.819 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-13T11:26:14.814+0000 7f6a68775640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26307) UID: 0 2024-09-13T11:26:14.826 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_historic_ops 2024-09-13T11:26:14.921 INFO:tasks.ceph.osd.2.smithi162.stderr:2024-09-13T11:26:14.915+0000 7fe92b35a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26313) UID: 0 2024-09-13T11:26:15.006 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-13T11:26:15.006 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:26:15.022 INFO:tasks.ceph.osd.11.smithi177.stderr:2024-09-13T11:26:15.017+0000 7f24792c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26580) UID: 0 2024-09-13T11:26:15.024 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-13T11:26:15.073 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T11:26:15.106 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 15, 14] out_osds: [] dead_osds: [] live_osds: [0, 4, 8, 12, 5, 9, 13, 2, 6, 10, 3, 7, 11, 15, 1, 14] 2024-09-13T11:26:15.106 INFO:tasks.thrashosds.thrasher:choose_action: min_in 2 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-13T11:26:15.107 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2024-09-13T11:26:15.107 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump -f json-pretty 2024-09-13T11:26:15.116 INFO:tasks.ceph.osd.5.smithi092.stderr:2024-09-13T11:26:15.112+0000 7f6f02136640 -1 osd.5 199 handle_osd_map injecting map crc failure 2024-09-13T11:26:15.117 INFO:tasks.ceph.osd.13.smithi092.stderr:2024-09-13T11:26:15.116+0000 7f3e4da13640 -1 osd.13 199 handle_osd_map injecting map crc failure 2024-09-13T11:26:15.164 INFO:tasks.ceph.osd.8.smithi001.stderr:2024-09-13T11:26:15.162+0000 7fb9b48ae640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26312) UID: 0 2024-09-13T11:26:15.193 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-13T11:26:15.193 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:26:15.221 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-13T11:26:15.223 INFO:tasks.ceph.osd.9.smithi092.stderr:2024-09-13T11:26:15.220+0000 7ff5ab3d0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26348) UID: 0 2024-09-13T11:26:15.279 INFO:teuthology.orchestra.run.smithi001.stdout:osd.0: {} 2024-09-13T11:26:15.280 INFO:teuthology.orchestra.run.smithi001.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T11:26:15.306 INFO:teuthology.orchestra.run.smithi001.stdout:osd.1: {} 2024-09-13T11:26:15.307 INFO:teuthology.orchestra.run.smithi001.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T11:26:15.324 INFO:tasks.ceph.osd.7.smithi177.stderr:2024-09-13T11:26:15.321+0000 7f897b51d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26575) UID: 0 2024-09-13T11:26:15.339 INFO:teuthology.orchestra.run.smithi001.stdout:osd.2: {} 2024-09-13T11:26:15.339 INFO:teuthology.orchestra.run.smithi001.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T11:26:15.382 INFO:teuthology.orchestra.run.smithi001.stdout:osd.3: {} 2024-09-13T11:26:15.382 INFO:teuthology.orchestra.run.smithi001.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T11:26:15.417 INFO:teuthology.orchestra.run.smithi001.stdout:osd.4: {} 2024-09-13T11:26:15.417 INFO:teuthology.orchestra.run.smithi001.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T11:26:15.425 INFO:tasks.ceph.osd.14.smithi162.stderr:2024-09-13T11:26:15.423+0000 7f3df5903640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33154) UID: 0 2024-09-13T11:26:15.427 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-13T11:26:15.427 INFO:teuthology.orchestra.run.smithi001.stdout: "ops": [], 2024-09-13T11:26:15.428 INFO:teuthology.orchestra.run.smithi001.stdout: "complaint_time": 30, 2024-09-13T11:26:15.428 INFO:teuthology.orchestra.run.smithi001.stdout: "num_blocked_ops": 0 2024-09-13T11:26:15.428 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-13T11:26:15.449 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-13T11:26:15.450 INFO:teuthology.orchestra.run.smithi001.stdout:osd.5: {} 2024-09-13T11:26:15.450 INFO:teuthology.orchestra.run.smithi001.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T11:26:15.474 INFO:teuthology.orchestra.run.smithi001.stdout:osd.6: {} 2024-09-13T11:26:15.475 INFO:teuthology.orchestra.run.smithi001.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T11:26:15.515 INFO:teuthology.orchestra.run.smithi001.stdout:osd.7: {} 2024-09-13T11:26:15.515 INFO:teuthology.orchestra.run.smithi001.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T11:26:15.527 INFO:tasks.ceph.osd.5.smithi092.stderr:2024-09-13T11:26:15.524+0000 7f6f0b705640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26346) UID: 0 2024-09-13T11:26:15.553 INFO:teuthology.orchestra.run.smithi001.stdout:osd.8: {} 2024-09-13T11:26:15.553 INFO:teuthology.orchestra.run.smithi001.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-13T11:26:15.581 INFO:teuthology.orchestra.run.smithi001.stdout:osd.9: {} 2024-09-13T11:26:15.581 INFO:teuthology.orchestra.run.smithi001.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-13T11:26:15.602 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-13T11:26:15.603 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-13T11:26:15.603 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 200, 2024-09-13T11:26:15.603 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "2fd68ac7-5581-4d90-80a4-30bcf5252a7a", 2024-09-13T11:26:15.603 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-09-13T11:21:57.223153+0000", 2024-09-13T11:26:15.603 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-09-13T11:26:15.101027+0000", 2024-09-13T11:26:15.603 INFO:teuthology.orchestra.run.smithi001.stdout: "last_up_change": "2024-09-13T11:24:55.968636+0000", 2024-09-13T11:26:15.603 INFO:teuthology.orchestra.run.smithi001.stdout: "last_in_change": "2024-09-13T11:26:04.007363+0000", 2024-09-13T11:26:15.603 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": "noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-13T11:26:15.603 INFO:teuthology.orchestra.run.smithi001.stdout: "flags_num": 5801984, 2024-09-13T11:26:15.603 INFO:teuthology.orchestra.run.smithi001.stdout: "flags_set": [ 2024-09-13T11:26:15.603 INFO:teuthology.orchestra.run.smithi001.stdout: "noscrub", 2024-09-13T11:26:15.603 INFO:teuthology.orchestra.run.smithi001.stdout: "pglog_hardlimit", 2024-09-13T11:26:15.603 INFO:teuthology.orchestra.run.smithi001.stdout: "purged_snapdirs", 2024-09-13T11:26:15.604 INFO:teuthology.orchestra.run.smithi001.stdout: "recovery_deletes", 2024-09-13T11:26:15.604 INFO:teuthology.orchestra.run.smithi001.stdout: "sortbitwise" 2024-09-13T11:26:15.604 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-09-13T11:26:15.604 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_version": 7, 2024-09-13T11:26:15.604 INFO:teuthology.orchestra.run.smithi001.stdout: "full_ratio": 0.94999998807907104, 2024-09-13T11:26:15.604 INFO:teuthology.orchestra.run.smithi001.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-09-13T11:26:15.604 INFO:teuthology.orchestra.run.smithi001.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-13T11:26:15.604 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_snapshot": "", 2024-09-13T11:26:15.604 INFO:teuthology.orchestra.run.smithi001.stdout: "pool_max": 3, 2024-09-13T11:26:15.604 INFO:teuthology.orchestra.run.smithi001.stdout: "max_osd": 16, 2024-09-13T11:26:15.604 INFO:teuthology.orchestra.run.smithi001.stdout: "require_min_compat_client": "luminous", 2024-09-13T11:26:15.604 INFO:teuthology.orchestra.run.smithi001.stdout: "min_compat_client": "luminous", 2024-09-13T11:26:15.604 INFO:teuthology.orchestra.run.smithi001.stdout: "require_osd_release": "squid", 2024-09-13T11:26:15.605 INFO:teuthology.orchestra.run.smithi001.stdout: "allow_crimson": false, 2024-09-13T11:26:15.605 INFO:teuthology.orchestra.run.smithi001.stdout: "pools": [ 2024-09-13T11:26:15.605 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.605 INFO:teuthology.orchestra.run.smithi001.stdout: "pool": 1, 2024-09-13T11:26:15.605 INFO:teuthology.orchestra.run.smithi001.stdout: "pool_name": ".mgr", 2024-09-13T11:26:15.605 INFO:teuthology.orchestra.run.smithi001.stdout: "create_time": "2024-09-13T11:22:10.162266+0000", 2024-09-13T11:26:15.605 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-09-13T11:26:15.605 INFO:teuthology.orchestra.run.smithi001.stdout: "flags_names": "hashpspool", 2024-09-13T11:26:15.605 INFO:teuthology.orchestra.run.smithi001.stdout: "type": 1, 2024-09-13T11:26:15.605 INFO:teuthology.orchestra.run.smithi001.stdout: "size": 2, 2024-09-13T11:26:15.605 INFO:teuthology.orchestra.run.smithi001.stdout: "min_size": 2, 2024-09-13T11:26:15.606 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_rule": 0, 2024-09-13T11:26:15.606 INFO:teuthology.orchestra.run.smithi001.stdout: "peering_crush_bucket_count": 0, 2024-09-13T11:26:15.606 INFO:teuthology.orchestra.run.smithi001.stdout: "peering_crush_bucket_target": 0, 2024-09-13T11:26:15.606 INFO:teuthology.orchestra.run.smithi001.stdout: "peering_crush_bucket_barrier": 0, 2024-09-13T11:26:15.606 INFO:teuthology.orchestra.run.smithi001.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-13T11:26:15.606 INFO:teuthology.orchestra.run.smithi001.stdout: "is_stretch_pool": false, 2024-09-13T11:26:15.606 INFO:teuthology.orchestra.run.smithi001.stdout: "object_hash": 2, 2024-09-13T11:26:15.606 INFO:teuthology.orchestra.run.smithi001.stdout: "pg_autoscale_mode": "off", 2024-09-13T11:26:15.606 INFO:teuthology.orchestra.run.smithi001.stdout: "pg_num": 1, 2024-09-13T11:26:15.607 INFO:teuthology.orchestra.run.smithi001.stdout: "pg_placement_num": 1, 2024-09-13T11:26:15.607 INFO:teuthology.orchestra.run.smithi001.stdout: "pg_placement_num_target": 1, 2024-09-13T11:26:15.607 INFO:teuthology.orchestra.run.smithi001.stdout: "pg_num_target": 1, 2024-09-13T11:26:15.607 INFO:teuthology.orchestra.run.smithi001.stdout: "pg_num_pending": 1, 2024-09-13T11:26:15.607 INFO:teuthology.orchestra.run.smithi001.stdout: "last_pg_merge_meta": { 2024-09-13T11:26:15.607 INFO:teuthology.orchestra.run.smithi001.stdout: "source_pgid": "0.0", 2024-09-13T11:26:15.607 INFO:teuthology.orchestra.run.smithi001.stdout: "ready_epoch": 0, 2024-09-13T11:26:15.607 INFO:teuthology.orchestra.run.smithi001.stdout: "last_epoch_started": 0, 2024-09-13T11:26:15.607 INFO:teuthology.orchestra.run.smithi001.stdout: "last_epoch_clean": 0, 2024-09-13T11:26:15.607 INFO:teuthology.orchestra.run.smithi001.stdout: "source_version": "0'0", 2024-09-13T11:26:15.607 INFO:teuthology.orchestra.run.smithi001.stdout: "target_version": "0'0" 2024-09-13T11:26:15.608 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.608 INFO:teuthology.orchestra.run.smithi001.stdout: "last_change": "25", 2024-09-13T11:26:15.608 INFO:teuthology.orchestra.run.smithi001.stdout: "last_force_op_resend": "0", 2024-09-13T11:26:15.608 INFO:teuthology.orchestra.run.smithi001.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-13T11:26:15.608 INFO:teuthology.orchestra.run.smithi001.stdout: "last_force_op_resend_preluminous": "0", 2024-09-13T11:26:15.608 INFO:teuthology.orchestra.run.smithi001.stdout: "auid": 0, 2024-09-13T11:26:15.608 INFO:teuthology.orchestra.run.smithi001.stdout: "snap_mode": "selfmanaged", 2024-09-13T11:26:15.608 INFO:teuthology.orchestra.run.smithi001.stdout: "snap_seq": 0, 2024-09-13T11:26:15.608 INFO:teuthology.orchestra.run.smithi001.stdout: "snap_epoch": 0, 2024-09-13T11:26:15.608 INFO:teuthology.orchestra.run.smithi001.stdout: "pool_snaps": [], 2024-09-13T11:26:15.608 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_snaps": "[]", 2024-09-13T11:26:15.608 INFO:teuthology.orchestra.run.smithi001.stdout: "quota_max_bytes": 0, 2024-09-13T11:26:15.608 INFO:teuthology.orchestra.run.smithi001.stdout: "quota_max_objects": 0, 2024-09-13T11:26:15.609 INFO:teuthology.orchestra.run.smithi001.stdout: "tiers": [], 2024-09-13T11:26:15.609 INFO:teuthology.orchestra.run.smithi001.stdout: "tier_of": -1, 2024-09-13T11:26:15.609 INFO:teuthology.orchestra.run.smithi001.stdout: "read_tier": -1, 2024-09-13T11:26:15.609 INFO:teuthology.orchestra.run.smithi001.stdout: "write_tier": -1, 2024-09-13T11:26:15.609 INFO:teuthology.orchestra.run.smithi001.stdout: "cache_mode": "none", 2024-09-13T11:26:15.609 INFO:teuthology.orchestra.run.smithi001.stdout: "target_max_bytes": 0, 2024-09-13T11:26:15.609 INFO:teuthology.orchestra.run.smithi001.stdout: "target_max_objects": 0, 2024-09-13T11:26:15.609 INFO:teuthology.orchestra.run.smithi001.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-13T11:26:15.609 INFO:teuthology.orchestra.run.smithi001.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-13T11:26:15.609 INFO:teuthology.orchestra.run.smithi001.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-13T11:26:15.609 INFO:teuthology.orchestra.run.smithi001.stdout: "cache_min_flush_age": 0, 2024-09-13T11:26:15.609 INFO:teuthology.orchestra.run.smithi001.stdout: "cache_min_evict_age": 0, 2024-09-13T11:26:15.610 INFO:teuthology.orchestra.run.smithi001.stdout: "erasure_code_profile": "", 2024-09-13T11:26:15.610 INFO:teuthology.orchestra.run.smithi001.stdout: "hit_set_params": { 2024-09-13T11:26:15.610 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "none" 2024-09-13T11:26:15.610 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.610 INFO:teuthology.orchestra.run.smithi001.stdout: "hit_set_period": 0, 2024-09-13T11:26:15.610 INFO:teuthology.orchestra.run.smithi001.stdout: "hit_set_count": 0, 2024-09-13T11:26:15.610 INFO:teuthology.orchestra.run.smithi001.stdout: "use_gmt_hitset": true, 2024-09-13T11:26:15.610 INFO:teuthology.orchestra.run.smithi001.stdout: "min_read_recency_for_promote": 0, 2024-09-13T11:26:15.610 INFO:teuthology.orchestra.run.smithi001.stdout: "min_write_recency_for_promote": 0, 2024-09-13T11:26:15.610 INFO:teuthology.orchestra.run.smithi001.stdout: "hit_set_grade_decay_rate": 0, 2024-09-13T11:26:15.610 INFO:teuthology.orchestra.run.smithi001.stdout: "hit_set_search_last_n": 0, 2024-09-13T11:26:15.610 INFO:teuthology.orchestra.run.smithi001.stdout: "grade_table": [], 2024-09-13T11:26:15.610 INFO:teuthology.orchestra.run.smithi001.stdout: "stripe_width": 0, 2024-09-13T11:26:15.611 INFO:teuthology.orchestra.run.smithi001.stdout: "expected_num_objects": 0, 2024-09-13T11:26:15.611 INFO:teuthology.orchestra.run.smithi001.stdout: "fast_read": false, 2024-09-13T11:26:15.611 INFO:teuthology.orchestra.run.smithi001.stdout: "options": { 2024-09-13T11:26:15.611 INFO:teuthology.orchestra.run.smithi001.stdout: "pg_num_max": 32, 2024-09-13T11:26:15.611 INFO:teuthology.orchestra.run.smithi001.stdout: "pg_num_min": 1 2024-09-13T11:26:15.611 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.611 INFO:teuthology.orchestra.run.smithi001.stdout: "application_metadata": { 2024-09-13T11:26:15.611 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": {} 2024-09-13T11:26:15.611 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.611 INFO:teuthology.orchestra.run.smithi001.stdout: "read_balance": { 2024-09-13T11:26:15.611 INFO:teuthology.orchestra.run.smithi001.stdout: "score_type": "Fair distribution", 2024-09-13T11:26:15.611 INFO:teuthology.orchestra.run.smithi001.stdout: "score_acting": 15.380000114440918, 2024-09-13T11:26:15.611 INFO:teuthology.orchestra.run.smithi001.stdout: "score_stable": 15.380000114440918, 2024-09-13T11:26:15.611 INFO:teuthology.orchestra.run.smithi001.stdout: "optimal_score": 0.12999999523162842, 2024-09-13T11:26:15.612 INFO:teuthology.orchestra.run.smithi001.stdout: "raw_score_acting": 2, 2024-09-13T11:26:15.612 INFO:teuthology.orchestra.run.smithi001.stdout: "raw_score_stable": 2, 2024-09-13T11:26:15.612 INFO:teuthology.orchestra.run.smithi001.stdout: "primary_affinity_weighted": 1, 2024-09-13T11:26:15.612 INFO:teuthology.orchestra.run.smithi001.stdout: "average_primary_affinity": 1, 2024-09-13T11:26:15.612 INFO:teuthology.orchestra.run.smithi001.stdout: "average_primary_affinity_weighted": 1 2024-09-13T11:26:15.612 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:15.612 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.612 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.612 INFO:teuthology.orchestra.run.smithi001.stdout: "pool": 2, 2024-09-13T11:26:15.612 INFO:teuthology.orchestra.run.smithi001.stdout: "pool_name": "rbd", 2024-09-13T11:26:15.612 INFO:teuthology.orchestra.run.smithi001.stdout: "create_time": "2024-09-13T11:22:12.235549+0000", 2024-09-13T11:26:15.612 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 8193, 2024-09-13T11:26:15.612 INFO:teuthology.orchestra.run.smithi001.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-09-13T11:26:15.613 INFO:teuthology.orchestra.run.smithi001.stdout: "type": 1, 2024-09-13T11:26:15.613 INFO:teuthology.orchestra.run.smithi001.stdout: "size": 2, 2024-09-13T11:26:15.613 INFO:teuthology.orchestra.run.smithi001.stdout: "min_size": 2, 2024-09-13T11:26:15.613 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_rule": 0, 2024-09-13T11:26:15.613 INFO:teuthology.orchestra.run.smithi001.stdout: "peering_crush_bucket_count": 0, 2024-09-13T11:26:15.613 INFO:teuthology.orchestra.run.smithi001.stdout: "peering_crush_bucket_target": 0, 2024-09-13T11:26:15.613 INFO:teuthology.orchestra.run.smithi001.stdout: "peering_crush_bucket_barrier": 0, 2024-09-13T11:26:15.613 INFO:teuthology.orchestra.run.smithi001.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-13T11:26:15.613 INFO:teuthology.orchestra.run.smithi001.stdout: "is_stretch_pool": false, 2024-09-13T11:26:15.614 INFO:teuthology.orchestra.run.smithi001.stdout: "object_hash": 2, 2024-09-13T11:26:15.614 INFO:teuthology.orchestra.run.smithi001.stdout: "pg_autoscale_mode": "off", 2024-09-13T11:26:15.614 INFO:teuthology.orchestra.run.smithi001.stdout: "pg_num": 8, 2024-09-13T11:26:15.614 INFO:teuthology.orchestra.run.smithi001.stdout: "pg_placement_num": 8, 2024-09-13T11:26:15.614 INFO:teuthology.orchestra.run.smithi001.stdout: "pg_placement_num_target": 8, 2024-09-13T11:26:15.614 INFO:teuthology.orchestra.run.smithi001.stdout: "pg_num_target": 8, 2024-09-13T11:26:15.614 INFO:teuthology.orchestra.run.smithi001.stdout: "pg_num_pending": 8, 2024-09-13T11:26:15.614 INFO:teuthology.orchestra.run.smithi001.stdout: "last_pg_merge_meta": { 2024-09-13T11:26:15.614 INFO:teuthology.orchestra.run.smithi001.stdout: "source_pgid": "0.0", 2024-09-13T11:26:15.614 INFO:teuthology.orchestra.run.smithi001.stdout: "ready_epoch": 0, 2024-09-13T11:26:15.614 INFO:teuthology.orchestra.run.smithi001.stdout: "last_epoch_started": 0, 2024-09-13T11:26:15.614 INFO:teuthology.orchestra.run.smithi001.stdout: "last_epoch_clean": 0, 2024-09-13T11:26:15.614 INFO:teuthology.orchestra.run.smithi001.stdout: "source_version": "0'0", 2024-09-13T11:26:15.614 INFO:teuthology.orchestra.run.smithi001.stdout: "target_version": "0'0" 2024-09-13T11:26:15.615 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.615 INFO:teuthology.orchestra.run.smithi001.stdout: "last_change": "28", 2024-09-13T11:26:15.615 INFO:teuthology.orchestra.run.smithi001.stdout: "last_force_op_resend": "0", 2024-09-13T11:26:15.615 INFO:teuthology.orchestra.run.smithi001.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-13T11:26:15.615 INFO:teuthology.orchestra.run.smithi001.stdout: "last_force_op_resend_preluminous": "0", 2024-09-13T11:26:15.615 INFO:teuthology.orchestra.run.smithi001.stdout: "auid": 0, 2024-09-13T11:26:15.615 INFO:teuthology.orchestra.run.smithi001.stdout: "snap_mode": "selfmanaged", 2024-09-13T11:26:15.615 INFO:teuthology.orchestra.run.smithi001.stdout: "snap_seq": 2, 2024-09-13T11:26:15.615 INFO:teuthology.orchestra.run.smithi001.stdout: "snap_epoch": 28, 2024-09-13T11:26:15.615 INFO:teuthology.orchestra.run.smithi001.stdout: "pool_snaps": [], 2024-09-13T11:26:15.615 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_snaps": "[]", 2024-09-13T11:26:15.615 INFO:teuthology.orchestra.run.smithi001.stdout: "quota_max_bytes": 0, 2024-09-13T11:26:15.615 INFO:teuthology.orchestra.run.smithi001.stdout: "quota_max_objects": 0, 2024-09-13T11:26:15.616 INFO:teuthology.orchestra.run.smithi001.stdout: "tiers": [], 2024-09-13T11:26:15.616 INFO:teuthology.orchestra.run.smithi001.stdout: "tier_of": -1, 2024-09-13T11:26:15.616 INFO:teuthology.orchestra.run.smithi001.stdout: "read_tier": -1, 2024-09-13T11:26:15.616 INFO:teuthology.orchestra.run.smithi001.stdout: "write_tier": -1, 2024-09-13T11:26:15.616 INFO:teuthology.orchestra.run.smithi001.stdout: "cache_mode": "none", 2024-09-13T11:26:15.616 INFO:teuthology.orchestra.run.smithi001.stdout: "target_max_bytes": 0, 2024-09-13T11:26:15.616 INFO:teuthology.orchestra.run.smithi001.stdout: "target_max_objects": 0, 2024-09-13T11:26:15.616 INFO:teuthology.orchestra.run.smithi001.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-13T11:26:15.616 INFO:teuthology.orchestra.run.smithi001.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-13T11:26:15.616 INFO:teuthology.orchestra.run.smithi001.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-13T11:26:15.616 INFO:teuthology.orchestra.run.smithi001.stdout: "cache_min_flush_age": 0, 2024-09-13T11:26:15.616 INFO:teuthology.orchestra.run.smithi001.stdout: "cache_min_evict_age": 0, 2024-09-13T11:26:15.616 INFO:teuthology.orchestra.run.smithi001.stdout: "erasure_code_profile": "", 2024-09-13T11:26:15.616 INFO:teuthology.orchestra.run.smithi001.stdout: "hit_set_params": { 2024-09-13T11:26:15.617 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "none" 2024-09-13T11:26:15.617 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.617 INFO:teuthology.orchestra.run.smithi001.stdout: "hit_set_period": 0, 2024-09-13T11:26:15.617 INFO:teuthology.orchestra.run.smithi001.stdout: "hit_set_count": 0, 2024-09-13T11:26:15.617 INFO:teuthology.orchestra.run.smithi001.stdout: "use_gmt_hitset": true, 2024-09-13T11:26:15.617 INFO:teuthology.orchestra.run.smithi001.stdout: "min_read_recency_for_promote": 0, 2024-09-13T11:26:15.617 INFO:teuthology.orchestra.run.smithi001.stdout: "min_write_recency_for_promote": 0, 2024-09-13T11:26:15.617 INFO:teuthology.orchestra.run.smithi001.stdout: "hit_set_grade_decay_rate": 0, 2024-09-13T11:26:15.617 INFO:teuthology.orchestra.run.smithi001.stdout: "hit_set_search_last_n": 0, 2024-09-13T11:26:15.617 INFO:teuthology.orchestra.run.smithi001.stdout: "grade_table": [], 2024-09-13T11:26:15.617 INFO:teuthology.orchestra.run.smithi001.stdout: "stripe_width": 0, 2024-09-13T11:26:15.617 INFO:teuthology.orchestra.run.smithi001.stdout: "expected_num_objects": 0, 2024-09-13T11:26:15.618 INFO:teuthology.orchestra.run.smithi001.stdout: "fast_read": false, 2024-09-13T11:26:15.618 INFO:teuthology.orchestra.run.smithi001.stdout: "options": {}, 2024-09-13T11:26:15.618 INFO:teuthology.orchestra.run.smithi001.stdout: "application_metadata": { 2024-09-13T11:26:15.618 INFO:teuthology.orchestra.run.smithi001.stdout: "rbd": {} 2024-09-13T11:26:15.618 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.618 INFO:teuthology.orchestra.run.smithi001.stdout: "read_balance": { 2024-09-13T11:26:15.618 INFO:teuthology.orchestra.run.smithi001.stdout: "score_type": "Fair distribution", 2024-09-13T11:26:15.618 INFO:teuthology.orchestra.run.smithi001.stdout: "score_acting": 3.9700000286102295, 2024-09-13T11:26:15.618 INFO:teuthology.orchestra.run.smithi001.stdout: "score_stable": 3.9700000286102295, 2024-09-13T11:26:15.618 INFO:teuthology.orchestra.run.smithi001.stdout: "optimal_score": 0.62999999523162842, 2024-09-13T11:26:15.618 INFO:teuthology.orchestra.run.smithi001.stdout: "raw_score_acting": 2.5, 2024-09-13T11:26:15.619 INFO:teuthology.orchestra.run.smithi001.stdout: "raw_score_stable": 2.5, 2024-09-13T11:26:15.619 INFO:teuthology.orchestra.run.smithi001.stdout: "primary_affinity_weighted": 1, 2024-09-13T11:26:15.619 INFO:teuthology.orchestra.run.smithi001.stdout: "average_primary_affinity": 1, 2024-09-13T11:26:15.619 INFO:teuthology.orchestra.run.smithi001.stdout: "average_primary_affinity_weighted": 1 2024-09-13T11:26:15.619 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:15.619 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.619 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.619 INFO:teuthology.orchestra.run.smithi001.stdout: "pool": 3, 2024-09-13T11:26:15.619 INFO:teuthology.orchestra.run.smithi001.stdout: "pool_name": "unique_pool_0", 2024-09-13T11:26:15.619 INFO:teuthology.orchestra.run.smithi001.stdout: "create_time": "2024-09-13T11:22:24.305521+0000", 2024-09-13T11:26:15.620 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-09-13T11:26:15.620 INFO:teuthology.orchestra.run.smithi001.stdout: "flags_names": "hashpspool", 2024-09-13T11:26:15.620 INFO:teuthology.orchestra.run.smithi001.stdout: "type": 1, 2024-09-13T11:26:15.620 INFO:teuthology.orchestra.run.smithi001.stdout: "size": 2, 2024-09-13T11:26:15.620 INFO:teuthology.orchestra.run.smithi001.stdout: "min_size": 2, 2024-09-13T11:26:15.620 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_rule": 0, 2024-09-13T11:26:15.620 INFO:teuthology.orchestra.run.smithi001.stdout: "peering_crush_bucket_count": 0, 2024-09-13T11:26:15.620 INFO:teuthology.orchestra.run.smithi001.stdout: "peering_crush_bucket_target": 0, 2024-09-13T11:26:15.620 INFO:teuthology.orchestra.run.smithi001.stdout: "peering_crush_bucket_barrier": 0, 2024-09-13T11:26:15.620 INFO:teuthology.orchestra.run.smithi001.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-13T11:26:15.620 INFO:teuthology.orchestra.run.smithi001.stdout: "is_stretch_pool": false, 2024-09-13T11:26:15.620 INFO:teuthology.orchestra.run.smithi001.stdout: "object_hash": 2, 2024-09-13T11:26:15.620 INFO:teuthology.orchestra.run.smithi001.stdout: "pg_autoscale_mode": "off", 2024-09-13T11:26:15.621 INFO:teuthology.orchestra.run.smithi001.stdout: "pg_num": 46, 2024-09-13T11:26:15.621 INFO:teuthology.orchestra.run.smithi001.stdout: "pg_placement_num": 46, 2024-09-13T11:26:15.621 INFO:teuthology.orchestra.run.smithi001.stdout: "pg_placement_num_target": 46, 2024-09-13T11:26:15.621 INFO:teuthology.orchestra.run.smithi001.stdout: "pg_num_target": 46, 2024-09-13T11:26:15.621 INFO:teuthology.orchestra.run.smithi001.stdout: "pg_num_pending": 46, 2024-09-13T11:26:15.621 INFO:teuthology.orchestra.run.smithi001.stdout: "last_pg_merge_meta": { 2024-09-13T11:26:15.621 INFO:teuthology.orchestra.run.smithi001.stdout: "source_pgid": "0.0", 2024-09-13T11:26:15.621 INFO:teuthology.orchestra.run.smithi001.stdout: "ready_epoch": 0, 2024-09-13T11:26:15.621 INFO:teuthology.orchestra.run.smithi001.stdout: "last_epoch_started": 0, 2024-09-13T11:26:15.621 INFO:teuthology.orchestra.run.smithi001.stdout: "last_epoch_clean": 0, 2024-09-13T11:26:15.621 INFO:teuthology.orchestra.run.smithi001.stdout: "source_version": "0'0", 2024-09-13T11:26:15.621 INFO:teuthology.orchestra.run.smithi001.stdout: "target_version": "0'0" 2024-09-13T11:26:15.622 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.622 INFO:teuthology.orchestra.run.smithi001.stdout: "last_change": "196", 2024-09-13T11:26:15.622 INFO:teuthology.orchestra.run.smithi001.stdout: "last_force_op_resend": "0", 2024-09-13T11:26:15.622 INFO:teuthology.orchestra.run.smithi001.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-13T11:26:15.622 INFO:teuthology.orchestra.run.smithi001.stdout: "last_force_op_resend_preluminous": "117", 2024-09-13T11:26:15.622 INFO:teuthology.orchestra.run.smithi001.stdout: "auid": 0, 2024-09-13T11:26:15.622 INFO:teuthology.orchestra.run.smithi001.stdout: "snap_mode": "selfmanaged", 2024-09-13T11:26:15.622 INFO:teuthology.orchestra.run.smithi001.stdout: "snap_seq": 0, 2024-09-13T11:26:15.622 INFO:teuthology.orchestra.run.smithi001.stdout: "snap_epoch": 0, 2024-09-13T11:26:15.622 INFO:teuthology.orchestra.run.smithi001.stdout: "pool_snaps": [], 2024-09-13T11:26:15.622 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_snaps": "[]", 2024-09-13T11:26:15.622 INFO:teuthology.orchestra.run.smithi001.stdout: "quota_max_bytes": 0, 2024-09-13T11:26:15.623 INFO:teuthology.orchestra.run.smithi001.stdout: "quota_max_objects": 0, 2024-09-13T11:26:15.623 INFO:teuthology.orchestra.run.smithi001.stdout: "tiers": [], 2024-09-13T11:26:15.623 INFO:teuthology.orchestra.run.smithi001.stdout: "tier_of": -1, 2024-09-13T11:26:15.623 INFO:teuthology.orchestra.run.smithi001.stdout: "read_tier": -1, 2024-09-13T11:26:15.623 INFO:teuthology.orchestra.run.smithi001.stdout: "write_tier": -1, 2024-09-13T11:26:15.623 INFO:teuthology.orchestra.run.smithi001.stdout: "cache_mode": "none", 2024-09-13T11:26:15.623 INFO:teuthology.orchestra.run.smithi001.stdout: "target_max_bytes": 0, 2024-09-13T11:26:15.623 INFO:teuthology.orchestra.run.smithi001.stdout: "target_max_objects": 0, 2024-09-13T11:26:15.623 INFO:teuthology.orchestra.run.smithi001.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-13T11:26:15.623 INFO:teuthology.orchestra.run.smithi001.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-13T11:26:15.623 INFO:teuthology.orchestra.run.smithi001.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-13T11:26:15.624 INFO:teuthology.orchestra.run.smithi001.stdout: "cache_min_flush_age": 0, 2024-09-13T11:26:15.624 INFO:teuthology.orchestra.run.smithi001.stdout: "cache_min_evict_age": 0, 2024-09-13T11:26:15.624 INFO:teuthology.orchestra.run.smithi001.stdout: "erasure_code_profile": "", 2024-09-13T11:26:15.624 INFO:teuthology.orchestra.run.smithi001.stdout: "hit_set_params": { 2024-09-13T11:26:15.624 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "none" 2024-09-13T11:26:15.624 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.624 INFO:teuthology.orchestra.run.smithi001.stdout: "hit_set_period": 0, 2024-09-13T11:26:15.624 INFO:teuthology.orchestra.run.smithi001.stdout: "hit_set_count": 0, 2024-09-13T11:26:15.624 INFO:teuthology.orchestra.run.smithi001.stdout: "use_gmt_hitset": true, 2024-09-13T11:26:15.624 INFO:teuthology.orchestra.run.smithi001.stdout: "min_read_recency_for_promote": 0, 2024-09-13T11:26:15.624 INFO:teuthology.orchestra.run.smithi001.stdout: "min_write_recency_for_promote": 0, 2024-09-13T11:26:15.624 INFO:teuthology.orchestra.run.smithi001.stdout: "hit_set_grade_decay_rate": 0, 2024-09-13T11:26:15.624 INFO:teuthology.orchestra.run.smithi001.stdout: "hit_set_search_last_n": 0, 2024-09-13T11:26:15.624 INFO:teuthology.orchestra.run.smithi001.stdout: "grade_table": [], 2024-09-13T11:26:15.625 INFO:teuthology.orchestra.run.smithi001.stdout: "stripe_width": 0, 2024-09-13T11:26:15.625 INFO:teuthology.orchestra.run.smithi001.stdout: "expected_num_objects": 0, 2024-09-13T11:26:15.625 INFO:teuthology.orchestra.run.smithi001.stdout: "fast_read": false, 2024-09-13T11:26:15.625 INFO:teuthology.orchestra.run.smithi001.stdout: "options": {}, 2024-09-13T11:26:15.625 INFO:teuthology.orchestra.run.smithi001.stdout: "application_metadata": { 2024-09-13T11:26:15.625 INFO:teuthology.orchestra.run.smithi001.stdout: "rados": {} 2024-09-13T11:26:15.625 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.625 INFO:teuthology.orchestra.run.smithi001.stdout: "read_balance": { 2024-09-13T11:26:15.625 INFO:teuthology.orchestra.run.smithi001.stdout: "score_type": "Fair distribution", 2024-09-13T11:26:15.625 INFO:teuthology.orchestra.run.smithi001.stdout: "score_acting": 2.0899999141693115, 2024-09-13T11:26:15.625 INFO:teuthology.orchestra.run.smithi001.stdout: "score_stable": 2.0899999141693115, 2024-09-13T11:26:15.626 INFO:teuthology.orchestra.run.smithi001.stdout: "optimal_score": 1, 2024-09-13T11:26:15.626 INFO:teuthology.orchestra.run.smithi001.stdout: "raw_score_acting": 2.0899999141693115, 2024-09-13T11:26:15.626 INFO:teuthology.orchestra.run.smithi001.stdout: "raw_score_stable": 2.0899999141693115, 2024-09-13T11:26:15.626 INFO:teuthology.orchestra.run.smithi001.stdout: "primary_affinity_weighted": 1, 2024-09-13T11:26:15.626 INFO:teuthology.orchestra.run.smithi001.stdout: "average_primary_affinity": 1, 2024-09-13T11:26:15.626 INFO:teuthology.orchestra.run.smithi001.stdout: "average_primary_affinity_weighted": 1 2024-09-13T11:26:15.626 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:15.626 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:15.626 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-09-13T11:26:15.626 INFO:teuthology.orchestra.run.smithi001.stdout: "osds": [ 2024-09-13T11:26:15.626 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.626 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": 0, 2024-09-13T11:26:15.626 INFO:teuthology.orchestra.run.smithi001.stdout: "uuid": "0fc12b3c-e0de-4546-b935-db77efe7c567", 2024-09-13T11:26:15.626 INFO:teuthology.orchestra.run.smithi001.stdout: "up": 1, 2024-09-13T11:26:15.627 INFO:teuthology.orchestra.run.smithi001.stdout: "in": 1, 2024-09-13T11:26:15.627 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 1, 2024-09-13T11:26:15.627 INFO:teuthology.orchestra.run.smithi001.stdout: "primary_affinity": 1, 2024-09-13T11:26:15.627 INFO:teuthology.orchestra.run.smithi001.stdout: "last_clean_begin": 0, 2024-09-13T11:26:15.627 INFO:teuthology.orchestra.run.smithi001.stdout: "last_clean_end": 0, 2024-09-13T11:26:15.627 INFO:teuthology.orchestra.run.smithi001.stdout: "up_from": 22, 2024-09-13T11:26:15.627 INFO:teuthology.orchestra.run.smithi001.stdout: "up_thru": 192, 2024-09-13T11:26:15.627 INFO:teuthology.orchestra.run.smithi001.stdout: "down_at": 0, 2024-09-13T11:26:15.627 INFO:teuthology.orchestra.run.smithi001.stdout: "lost_at": 0, 2024-09-13T11:26:15.627 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-09-13T11:26:15.627 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:26:15.627 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.627 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:26:15.627 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6800", 2024-09-13T11:26:15.628 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 1544776477 2024-09-13T11:26:15.628 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:15.628 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:15.628 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.628 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_addrs": { 2024-09-13T11:26:15.628 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:26:15.628 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.628 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:26:15.628 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6801", 2024-09-13T11:26:15.628 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 1544776477 2024-09-13T11:26:15.628 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:15.628 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:15.628 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.629 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_back_addrs": { 2024-09-13T11:26:15.629 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:26:15.629 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.629 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:26:15.629 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6803", 2024-09-13T11:26:15.629 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 1544776477 2024-09-13T11:26:15.629 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:15.629 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:15.629 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.629 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_front_addrs": { 2024-09-13T11:26:15.629 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:26:15.629 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.629 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:26:15.629 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6802", 2024-09-13T11:26:15.630 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 1544776477 2024-09-13T11:26:15.630 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:15.630 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:15.630 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.630 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6800/1544776477", 2024-09-13T11:26:15.630 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_addr": "172.21.15.1:6801/1544776477", 2024-09-13T11:26:15.630 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_back_addr": "172.21.15.1:6803/1544776477", 2024-09-13T11:26:15.630 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_front_addr": "172.21.15.1:6802/1544776477", 2024-09-13T11:26:15.630 INFO:teuthology.orchestra.run.smithi001.stdout: "state": [ 2024-09-13T11:26:15.630 INFO:teuthology.orchestra.run.smithi001.stdout: "exists", 2024-09-13T11:26:15.630 INFO:teuthology.orchestra.run.smithi001.stdout: "up" 2024-09-13T11:26:15.630 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:15.630 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.630 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.631 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": 1, 2024-09-13T11:26:15.631 INFO:teuthology.orchestra.run.smithi001.stdout: "uuid": "b77fe331-cc92-4adb-b146-af5c380f6d4f", 2024-09-13T11:26:15.631 INFO:teuthology.orchestra.run.smithi001.stdout: "up": 1, 2024-09-13T11:26:15.631 INFO:teuthology.orchestra.run.smithi001.stdout: "in": 1, 2024-09-13T11:26:15.631 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 1, 2024-09-13T11:26:15.631 INFO:teuthology.orchestra.run.smithi001.stdout: "primary_affinity": 1, 2024-09-13T11:26:15.631 INFO:teuthology.orchestra.run.smithi001.stdout: "last_clean_begin": 23, 2024-09-13T11:26:15.631 INFO:teuthology.orchestra.run.smithi001.stdout: "last_clean_end": 29, 2024-09-13T11:26:15.631 INFO:teuthology.orchestra.run.smithi001.stdout: "up_from": 40, 2024-09-13T11:26:15.631 INFO:teuthology.orchestra.run.smithi001.stdout: "up_thru": 171, 2024-09-13T11:26:15.631 INFO:teuthology.orchestra.run.smithi001.stdout: "down_at": 30, 2024-09-13T11:26:15.632 INFO:teuthology.orchestra.run.smithi001.stdout: "lost_at": 0, 2024-09-13T11:26:15.632 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-09-13T11:26:15.632 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:26:15.632 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.632 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:26:15.632 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.92:6800", 2024-09-13T11:26:15.632 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 3591430847 2024-09-13T11:26:15.632 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:15.632 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:15.632 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.632 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_addrs": { 2024-09-13T11:26:15.632 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:26:15.632 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.633 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:26:15.633 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.92:6801", 2024-09-13T11:26:15.633 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 3591430847 2024-09-13T11:26:15.633 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:15.633 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:15.633 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.633 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_back_addrs": { 2024-09-13T11:26:15.633 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:26:15.633 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.633 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:26:15.634 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.92:6803", 2024-09-13T11:26:15.634 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 3591430847 2024-09-13T11:26:15.634 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:15.634 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:15.634 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.634 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_front_addrs": { 2024-09-13T11:26:15.634 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:26:15.634 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.634 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:26:15.635 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.92:6802", 2024-09-13T11:26:15.635 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 3591430847 2024-09-13T11:26:15.635 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:15.635 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:15.635 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.635 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.92:6800/3591430847", 2024-09-13T11:26:15.635 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_addr": "172.21.15.92:6801/3591430847", 2024-09-13T11:26:15.635 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_back_addr": "172.21.15.92:6803/3591430847", 2024-09-13T11:26:15.635 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_front_addr": "172.21.15.92:6802/3591430847", 2024-09-13T11:26:15.635 INFO:teuthology.orchestra.run.smithi001.stdout: "state": [ 2024-09-13T11:26:15.635 INFO:teuthology.orchestra.run.smithi001.stdout: "exists", 2024-09-13T11:26:15.635 INFO:teuthology.orchestra.run.smithi001.stdout: "up" 2024-09-13T11:26:15.635 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:15.635 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.636 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.636 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": 2, 2024-09-13T11:26:15.636 INFO:teuthology.orchestra.run.smithi001.stdout: "uuid": "c9d031c8-e885-4229-affa-08e1729d00e8", 2024-09-13T11:26:15.636 INFO:teuthology.orchestra.run.smithi001.stdout: "up": 1, 2024-09-13T11:26:15.636 INFO:teuthology.orchestra.run.smithi001.stdout: "in": 1, 2024-09-13T11:26:15.636 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 1, 2024-09-13T11:26:15.636 INFO:teuthology.orchestra.run.smithi001.stdout: "primary_affinity": 1, 2024-09-13T11:26:15.636 INFO:teuthology.orchestra.run.smithi001.stdout: "last_clean_begin": 0, 2024-09-13T11:26:15.636 INFO:teuthology.orchestra.run.smithi001.stdout: "last_clean_end": 0, 2024-09-13T11:26:15.636 INFO:teuthology.orchestra.run.smithi001.stdout: "up_from": 22, 2024-09-13T11:26:15.636 INFO:teuthology.orchestra.run.smithi001.stdout: "up_thru": 152, 2024-09-13T11:26:15.636 INFO:teuthology.orchestra.run.smithi001.stdout: "down_at": 0, 2024-09-13T11:26:15.636 INFO:teuthology.orchestra.run.smithi001.stdout: "lost_at": 0, 2024-09-13T11:26:15.637 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-09-13T11:26:15.637 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:26:15.637 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.637 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:26:15.637 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.162:6812", 2024-09-13T11:26:15.637 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 2712148101 2024-09-13T11:26:15.637 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:15.637 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:15.637 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.638 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_addrs": { 2024-09-13T11:26:15.638 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:26:15.638 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.638 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:26:15.638 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.162:6813", 2024-09-13T11:26:15.638 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 2712148101 2024-09-13T11:26:15.638 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:15.638 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:15.638 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.638 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_back_addrs": { 2024-09-13T11:26:15.639 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:26:15.639 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.639 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:26:15.639 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.162:6815", 2024-09-13T11:26:15.639 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 2712148101 2024-09-13T11:26:15.639 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:15.639 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:15.639 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.639 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_front_addrs": { 2024-09-13T11:26:15.639 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:26:15.639 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.639 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:26:15.639 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.162:6814", 2024-09-13T11:26:15.640 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 2712148101 2024-09-13T11:26:15.640 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:15.640 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:15.640 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.640 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.162:6812/2712148101", 2024-09-13T11:26:15.640 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_addr": "172.21.15.162:6813/2712148101", 2024-09-13T11:26:15.640 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_back_addr": "172.21.15.162:6815/2712148101", 2024-09-13T11:26:15.640 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_front_addr": "172.21.15.162:6814/2712148101", 2024-09-13T11:26:15.640 INFO:teuthology.orchestra.run.smithi001.stdout: "state": [ 2024-09-13T11:26:15.640 INFO:teuthology.orchestra.run.smithi001.stdout: "exists", 2024-09-13T11:26:15.640 INFO:teuthology.orchestra.run.smithi001.stdout: "up" 2024-09-13T11:26:15.640 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:15.641 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.641 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.641 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": 3, 2024-09-13T11:26:15.641 INFO:teuthology.orchestra.run.smithi001.stdout: "uuid": "bf50d71a-6c3a-4fbc-a407-cb81bad14a7b", 2024-09-13T11:26:15.641 INFO:teuthology.orchestra.run.smithi001.stdout: "up": 1, 2024-09-13T11:26:15.641 INFO:teuthology.orchestra.run.smithi001.stdout: "in": 1, 2024-09-13T11:26:15.641 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 1, 2024-09-13T11:26:15.641 INFO:teuthology.orchestra.run.smithi001.stdout: "primary_affinity": 1, 2024-09-13T11:26:15.641 INFO:teuthology.orchestra.run.smithi001.stdout: "last_clean_begin": 0, 2024-09-13T11:26:15.641 INFO:teuthology.orchestra.run.smithi001.stdout: "last_clean_end": 0, 2024-09-13T11:26:15.641 INFO:teuthology.orchestra.run.smithi001.stdout: "up_from": 21, 2024-09-13T11:26:15.641 INFO:teuthology.orchestra.run.smithi001.stdout: "up_thru": 192, 2024-09-13T11:26:15.641 INFO:teuthology.orchestra.run.smithi001.stdout: "down_at": 0, 2024-09-13T11:26:15.642 INFO:teuthology.orchestra.run.smithi001.stdout: "lost_at": 0, 2024-09-13T11:26:15.642 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-09-13T11:26:15.642 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:26:15.642 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.642 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:26:15.642 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.177:6801", 2024-09-13T11:26:15.642 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 3367751690 2024-09-13T11:26:15.642 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:15.642 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:15.642 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.642 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_addrs": { 2024-09-13T11:26:15.642 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:26:15.642 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.642 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:26:15.643 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.177:6802", 2024-09-13T11:26:15.643 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 3367751690 2024-09-13T11:26:15.643 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:15.643 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:15.643 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.643 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_back_addrs": { 2024-09-13T11:26:15.643 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:26:15.643 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.643 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:26:15.643 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.177:6804", 2024-09-13T11:26:15.643 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 3367751690 2024-09-13T11:26:15.643 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:15.643 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:15.643 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.644 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_front_addrs": { 2024-09-13T11:26:15.644 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:26:15.644 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.644 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:26:15.644 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.177:6803", 2024-09-13T11:26:15.644 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 3367751690 2024-09-13T11:26:15.644 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:15.644 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:15.644 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.644 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.177:6801/3367751690", 2024-09-13T11:26:15.644 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_addr": "172.21.15.177:6802/3367751690", 2024-09-13T11:26:15.644 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_back_addr": "172.21.15.177:6804/3367751690", 2024-09-13T11:26:15.645 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_front_addr": "172.21.15.177:6803/3367751690", 2024-09-13T11:26:15.645 INFO:teuthology.orchestra.run.smithi001.stdout: "state": [ 2024-09-13T11:26:15.645 INFO:teuthology.orchestra.run.smithi001.stdout: "exists", 2024-09-13T11:26:15.645 INFO:teuthology.orchestra.run.smithi001.stdout: "up" 2024-09-13T11:26:15.645 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:15.645 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.645 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.645 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": 4, 2024-09-13T11:26:15.645 INFO:teuthology.orchestra.run.smithi001.stdout: "uuid": "d1fe3f63-df3f-4a73-a1e2-8d7c8a3f1fa0", 2024-09-13T11:26:15.645 INFO:teuthology.orchestra.run.smithi001.stdout: "up": 1, 2024-09-13T11:26:15.645 INFO:teuthology.orchestra.run.smithi001.stdout: "in": 1, 2024-09-13T11:26:15.645 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 1, 2024-09-13T11:26:15.645 INFO:teuthology.orchestra.run.smithi001.stdout: "primary_affinity": 1, 2024-09-13T11:26:15.645 INFO:teuthology.orchestra.run.smithi001.stdout: "last_clean_begin": 0, 2024-09-13T11:26:15.646 INFO:teuthology.orchestra.run.smithi001.stdout: "last_clean_end": 0, 2024-09-13T11:26:15.646 INFO:teuthology.orchestra.run.smithi001.stdout: "up_from": 22, 2024-09-13T11:26:15.646 INFO:teuthology.orchestra.run.smithi001.stdout: "up_thru": 105, 2024-09-13T11:26:15.646 INFO:teuthology.orchestra.run.smithi001.stdout: "down_at": 0, 2024-09-13T11:26:15.646 INFO:teuthology.orchestra.run.smithi001.stdout: "lost_at": 0, 2024-09-13T11:26:15.646 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-09-13T11:26:15.646 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:26:15.646 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.646 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:26:15.646 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6808", 2024-09-13T11:26:15.646 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 3648540926 2024-09-13T11:26:15.646 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:15.646 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:15.647 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.647 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_addrs": { 2024-09-13T11:26:15.647 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:26:15.647 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.647 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:26:15.647 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6809", 2024-09-13T11:26:15.647 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 3648540926 2024-09-13T11:26:15.647 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:15.647 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:15.647 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.647 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_back_addrs": { 2024-09-13T11:26:15.647 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:26:15.647 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.648 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:26:15.648 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6811", 2024-09-13T11:26:15.648 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 3648540926 2024-09-13T11:26:15.648 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:15.648 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:15.648 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.648 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_front_addrs": { 2024-09-13T11:26:15.648 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:26:15.648 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.648 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:26:15.648 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6810", 2024-09-13T11:26:15.648 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 3648540926 2024-09-13T11:26:15.648 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:15.649 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:15.649 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.649 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6808/3648540926", 2024-09-13T11:26:15.649 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_addr": "172.21.15.1:6809/3648540926", 2024-09-13T11:26:15.649 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_back_addr": "172.21.15.1:6811/3648540926", 2024-09-13T11:26:15.649 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_front_addr": "172.21.15.1:6810/3648540926", 2024-09-13T11:26:15.649 INFO:teuthology.orchestra.run.smithi001.stdout: "state": [ 2024-09-13T11:26:15.649 INFO:teuthology.orchestra.run.smithi001.stdout: "exists", 2024-09-13T11:26:15.649 INFO:teuthology.orchestra.run.smithi001.stdout: "up" 2024-09-13T11:26:15.650 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:15.650 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.650 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.650 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": 5, 2024-09-13T11:26:15.650 INFO:teuthology.orchestra.run.smithi001.stdout: "uuid": "e2ead447-2f2c-40c9-82e7-53b8dc41e2af", 2024-09-13T11:26:15.650 INFO:teuthology.orchestra.run.smithi001.stdout: "up": 1, 2024-09-13T11:26:15.650 INFO:teuthology.orchestra.run.smithi001.stdout: "in": 1, 2024-09-13T11:26:15.650 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 1, 2024-09-13T11:26:15.650 INFO:teuthology.orchestra.run.smithi001.stdout: "primary_affinity": 1, 2024-09-13T11:26:15.650 INFO:teuthology.orchestra.run.smithi001.stdout: "last_clean_begin": 0, 2024-09-13T11:26:15.650 INFO:teuthology.orchestra.run.smithi001.stdout: "last_clean_end": 0, 2024-09-13T11:26:15.650 INFO:teuthology.orchestra.run.smithi001.stdout: "up_from": 21, 2024-09-13T11:26:15.651 INFO:teuthology.orchestra.run.smithi001.stdout: "up_thru": 192, 2024-09-13T11:26:15.651 INFO:teuthology.orchestra.run.smithi001.stdout: "down_at": 0, 2024-09-13T11:26:15.651 INFO:teuthology.orchestra.run.smithi001.stdout: "lost_at": 0, 2024-09-13T11:26:15.651 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-09-13T11:26:15.651 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:26:15.651 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.651 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:26:15.651 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.92:6804", 2024-09-13T11:26:15.651 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 3399529091 2024-09-13T11:26:15.651 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:15.651 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:15.651 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.651 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_addrs": { 2024-09-13T11:26:15.651 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:26:15.652 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.652 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:26:15.652 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.92:6805", 2024-09-13T11:26:15.652 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 3399529091 2024-09-13T11:26:15.652 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:15.652 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:15.652 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.652 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_back_addrs": { 2024-09-13T11:26:15.652 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:26:15.652 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.652 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:26:15.652 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.92:6807", 2024-09-13T11:26:15.652 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 3399529091 2024-09-13T11:26:15.652 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:15.653 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:15.653 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.653 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_front_addrs": { 2024-09-13T11:26:15.653 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:26:15.653 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.653 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:26:15.653 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.92:6806", 2024-09-13T11:26:15.653 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 3399529091 2024-09-13T11:26:15.653 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:15.653 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:15.653 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.653 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.92:6804/3399529091", 2024-09-13T11:26:15.654 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_addr": "172.21.15.92:6805/3399529091", 2024-09-13T11:26:15.654 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_back_addr": "172.21.15.92:6807/3399529091", 2024-09-13T11:26:15.654 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_front_addr": "172.21.15.92:6806/3399529091", 2024-09-13T11:26:15.654 INFO:teuthology.orchestra.run.smithi001.stdout: "state": [ 2024-09-13T11:26:15.654 INFO:teuthology.orchestra.run.smithi001.stdout: "exists", 2024-09-13T11:26:15.654 INFO:teuthology.orchestra.run.smithi001.stdout: "up" 2024-09-13T11:26:15.654 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:15.654 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.654 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.654 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": 6, 2024-09-13T11:26:15.654 INFO:teuthology.orchestra.run.smithi001.stdout: "uuid": "dbc20439-144e-4ec7-89de-674eb783f263", 2024-09-13T11:26:15.654 INFO:teuthology.orchestra.run.smithi001.stdout: "up": 1, 2024-09-13T11:26:15.654 INFO:teuthology.orchestra.run.smithi001.stdout: "in": 1, 2024-09-13T11:26:15.655 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 1, 2024-09-13T11:26:15.655 INFO:teuthology.orchestra.run.smithi001.stdout: "primary_affinity": 1, 2024-09-13T11:26:15.655 INFO:teuthology.orchestra.run.smithi001.stdout: "last_clean_begin": 0, 2024-09-13T11:26:15.655 INFO:teuthology.orchestra.run.smithi001.stdout: "last_clean_end": 0, 2024-09-13T11:26:15.655 INFO:teuthology.orchestra.run.smithi001.stdout: "up_from": 22, 2024-09-13T11:26:15.655 INFO:teuthology.orchestra.run.smithi001.stdout: "up_thru": 117, 2024-09-13T11:26:15.655 INFO:teuthology.orchestra.run.smithi001.stdout: "down_at": 0, 2024-09-13T11:26:15.655 INFO:teuthology.orchestra.run.smithi001.stdout: "lost_at": 0, 2024-09-13T11:26:15.655 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-09-13T11:26:15.655 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:26:15.656 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.656 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:26:15.656 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.162:6808", 2024-09-13T11:26:15.656 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 2287343806 2024-09-13T11:26:15.656 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:15.656 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:15.656 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.656 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_addrs": { 2024-09-13T11:26:15.656 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:26:15.656 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.656 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:26:15.656 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.162:6809", 2024-09-13T11:26:15.657 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 2287343806 2024-09-13T11:26:15.657 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:15.657 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:15.657 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.657 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_back_addrs": { 2024-09-13T11:26:15.657 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:26:15.657 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.657 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:26:15.657 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.162:6811", 2024-09-13T11:26:15.657 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 2287343806 2024-09-13T11:26:15.657 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:15.657 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:15.657 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.658 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_front_addrs": { 2024-09-13T11:26:15.658 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:26:15.658 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.658 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:26:15.658 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.162:6810", 2024-09-13T11:26:15.658 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 2287343806 2024-09-13T11:26:15.658 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:15.658 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:15.658 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.658 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.162:6808/2287343806", 2024-09-13T11:26:15.658 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_addr": "172.21.15.162:6809/2287343806", 2024-09-13T11:26:15.658 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_back_addr": "172.21.15.162:6811/2287343806", 2024-09-13T11:26:15.658 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_front_addr": "172.21.15.162:6810/2287343806", 2024-09-13T11:26:15.658 INFO:teuthology.orchestra.run.smithi001.stdout: "state": [ 2024-09-13T11:26:15.659 INFO:teuthology.orchestra.run.smithi001.stdout: "exists", 2024-09-13T11:26:15.659 INFO:teuthology.orchestra.run.smithi001.stdout: "up" 2024-09-13T11:26:15.659 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:15.659 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.659 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.659 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": 7, 2024-09-13T11:26:15.659 INFO:teuthology.orchestra.run.smithi001.stdout: "uuid": "60aa5206-cace-40f8-8d1a-10ec28d97096", 2024-09-13T11:26:15.659 INFO:teuthology.orchestra.run.smithi001.stdout: "up": 1, 2024-09-13T11:26:15.659 INFO:teuthology.orchestra.run.smithi001.stdout: "in": 1, 2024-09-13T11:26:15.659 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 1, 2024-09-13T11:26:15.659 INFO:teuthology.orchestra.run.smithi001.stdout: "primary_affinity": 1, 2024-09-13T11:26:15.659 INFO:teuthology.orchestra.run.smithi001.stdout: "last_clean_begin": 0, 2024-09-13T11:26:15.659 INFO:teuthology.orchestra.run.smithi001.stdout: "last_clean_end": 0, 2024-09-13T11:26:15.659 INFO:teuthology.orchestra.run.smithi001.stdout: "up_from": 22, 2024-09-13T11:26:15.660 INFO:teuthology.orchestra.run.smithi001.stdout: "up_thru": 122, 2024-09-13T11:26:15.660 INFO:teuthology.orchestra.run.smithi001.stdout: "down_at": 0, 2024-09-13T11:26:15.660 INFO:teuthology.orchestra.run.smithi001.stdout: "lost_at": 0, 2024-09-13T11:26:15.660 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-09-13T11:26:15.660 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:26:15.660 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.660 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:26:15.660 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.177:6813", 2024-09-13T11:26:15.660 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 2110809906 2024-09-13T11:26:15.660 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:15.660 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:15.660 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.661 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_addrs": { 2024-09-13T11:26:15.661 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:26:15.661 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.661 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:26:15.661 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.177:6814", 2024-09-13T11:26:15.661 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 2110809906 2024-09-13T11:26:15.661 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:15.661 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:15.661 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.661 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_back_addrs": { 2024-09-13T11:26:15.661 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:26:15.661 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.662 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:26:15.662 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.177:6816", 2024-09-13T11:26:15.662 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 2110809906 2024-09-13T11:26:15.662 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:15.662 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:15.662 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.662 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_front_addrs": { 2024-09-13T11:26:15.662 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:26:15.662 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.662 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:26:15.662 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.177:6815", 2024-09-13T11:26:15.662 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 2110809906 2024-09-13T11:26:15.662 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:15.663 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:15.663 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.663 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.177:6813/2110809906", 2024-09-13T11:26:15.663 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_addr": "172.21.15.177:6814/2110809906", 2024-09-13T11:26:15.663 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_back_addr": "172.21.15.177:6816/2110809906", 2024-09-13T11:26:15.663 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_front_addr": "172.21.15.177:6815/2110809906", 2024-09-13T11:26:15.663 INFO:teuthology.orchestra.run.smithi001.stdout: "state": [ 2024-09-13T11:26:15.663 INFO:teuthology.orchestra.run.smithi001.stdout: "exists", 2024-09-13T11:26:15.663 INFO:teuthology.orchestra.run.smithi001.stdout: "up" 2024-09-13T11:26:15.663 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:15.663 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.663 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.664 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": 8, 2024-09-13T11:26:15.664 INFO:teuthology.orchestra.run.smithi001.stdout: "uuid": "f3d99325-48ff-403c-8662-1d7745f7c97d", 2024-09-13T11:26:15.664 INFO:teuthology.orchestra.run.smithi001.stdout: "up": 1, 2024-09-13T11:26:15.664 INFO:teuthology.orchestra.run.smithi001.stdout: "in": 1, 2024-09-13T11:26:15.664 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 1, 2024-09-13T11:26:15.664 INFO:teuthology.orchestra.run.smithi001.stdout: "primary_affinity": 1, 2024-09-13T11:26:15.664 INFO:teuthology.orchestra.run.smithi001.stdout: "last_clean_begin": 0, 2024-09-13T11:26:15.664 INFO:teuthology.orchestra.run.smithi001.stdout: "last_clean_end": 0, 2024-09-13T11:26:15.664 INFO:teuthology.orchestra.run.smithi001.stdout: "up_from": 22, 2024-09-13T11:26:15.664 INFO:teuthology.orchestra.run.smithi001.stdout: "up_thru": 172, 2024-09-13T11:26:15.664 INFO:teuthology.orchestra.run.smithi001.stdout: "down_at": 0, 2024-09-13T11:26:15.664 INFO:teuthology.orchestra.run.smithi001.stdout: "lost_at": 0, 2024-09-13T11:26:15.664 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-09-13T11:26:15.665 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:26:15.665 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.665 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:26:15.665 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6804", 2024-09-13T11:26:15.665 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 2766567910 2024-09-13T11:26:15.665 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:15.665 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:15.665 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.665 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_addrs": { 2024-09-13T11:26:15.665 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:26:15.665 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.665 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:26:15.665 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6805", 2024-09-13T11:26:15.666 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 2766567910 2024-09-13T11:26:15.666 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:15.666 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:15.666 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.666 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_back_addrs": { 2024-09-13T11:26:15.666 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:26:15.666 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.666 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:26:15.666 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6807", 2024-09-13T11:26:15.666 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 2766567910 2024-09-13T11:26:15.666 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:15.666 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:15.667 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.667 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_front_addrs": { 2024-09-13T11:26:15.667 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:26:15.667 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.667 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:26:15.667 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6806", 2024-09-13T11:26:15.667 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 2766567910 2024-09-13T11:26:15.667 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:15.667 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:15.667 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.667 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6804/2766567910", 2024-09-13T11:26:15.668 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_addr": "172.21.15.1:6805/2766567910", 2024-09-13T11:26:15.668 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_back_addr": "172.21.15.1:6807/2766567910", 2024-09-13T11:26:15.668 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_front_addr": "172.21.15.1:6806/2766567910", 2024-09-13T11:26:15.668 INFO:teuthology.orchestra.run.smithi001.stdout: "state": [ 2024-09-13T11:26:15.668 INFO:teuthology.orchestra.run.smithi001.stdout: "exists", 2024-09-13T11:26:15.668 INFO:teuthology.orchestra.run.smithi001.stdout: "up" 2024-09-13T11:26:15.668 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:15.668 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.668 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.668 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": 9, 2024-09-13T11:26:15.668 INFO:teuthology.orchestra.run.smithi001.stdout: "uuid": "88511e24-3675-4551-968c-1a52e7e28f96", 2024-09-13T11:26:15.668 INFO:teuthology.orchestra.run.smithi001.stdout: "up": 1, 2024-09-13T11:26:15.668 INFO:teuthology.orchestra.run.smithi001.stdout: "in": 1, 2024-09-13T11:26:15.669 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 1, 2024-09-13T11:26:15.669 INFO:teuthology.orchestra.run.smithi001.stdout: "primary_affinity": 1, 2024-09-13T11:26:15.669 INFO:teuthology.orchestra.run.smithi001.stdout: "last_clean_begin": 0, 2024-09-13T11:26:15.669 INFO:teuthology.orchestra.run.smithi001.stdout: "last_clean_end": 0, 2024-09-13T11:26:15.669 INFO:teuthology.orchestra.run.smithi001.stdout: "up_from": 23, 2024-09-13T11:26:15.669 INFO:teuthology.orchestra.run.smithi001.stdout: "up_thru": 191, 2024-09-13T11:26:15.669 INFO:teuthology.orchestra.run.smithi001.stdout: "down_at": 0, 2024-09-13T11:26:15.669 INFO:teuthology.orchestra.run.smithi001.stdout: "lost_at": 0, 2024-09-13T11:26:15.669 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-09-13T11:26:15.669 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:26:15.669 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.669 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:26:15.670 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.92:6812", 2024-09-13T11:26:15.670 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 2488187511 2024-09-13T11:26:15.670 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:15.670 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:15.670 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.670 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_addrs": { 2024-09-13T11:26:15.670 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:26:15.670 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.670 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:26:15.670 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.92:6813", 2024-09-13T11:26:15.670 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 2488187511 2024-09-13T11:26:15.670 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:15.670 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:15.671 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.671 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_back_addrs": { 2024-09-13T11:26:15.671 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:26:15.671 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.671 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:26:15.671 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.92:6815", 2024-09-13T11:26:15.671 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 2488187511 2024-09-13T11:26:15.671 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:15.671 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:15.671 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.671 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_front_addrs": { 2024-09-13T11:26:15.671 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:26:15.671 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.671 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:26:15.671 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.92:6814", 2024-09-13T11:26:15.672 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 2488187511 2024-09-13T11:26:15.672 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:15.672 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:15.672 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.672 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.92:6812/2488187511", 2024-09-13T11:26:15.672 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_addr": "172.21.15.92:6813/2488187511", 2024-09-13T11:26:15.672 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_back_addr": "172.21.15.92:6815/2488187511", 2024-09-13T11:26:15.672 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_front_addr": "172.21.15.92:6814/2488187511", 2024-09-13T11:26:15.672 INFO:teuthology.orchestra.run.smithi001.stdout: "state": [ 2024-09-13T11:26:15.672 INFO:teuthology.orchestra.run.smithi001.stdout: "exists", 2024-09-13T11:26:15.672 INFO:teuthology.orchestra.run.smithi001.stdout: "up" 2024-09-13T11:26:15.672 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:15.672 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.673 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.673 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": 10, 2024-09-13T11:26:15.673 INFO:teuthology.orchestra.run.smithi001.stdout: "uuid": "acf4c64d-99af-42be-8406-413c86f54670", 2024-09-13T11:26:15.673 INFO:teuthology.orchestra.run.smithi001.stdout: "up": 1, 2024-09-13T11:26:15.673 INFO:teuthology.orchestra.run.smithi001.stdout: "in": 1, 2024-09-13T11:26:15.673 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 1, 2024-09-13T11:26:15.673 INFO:teuthology.orchestra.run.smithi001.stdout: "primary_affinity": 1, 2024-09-13T11:26:15.673 INFO:teuthology.orchestra.run.smithi001.stdout: "last_clean_begin": 0, 2024-09-13T11:26:15.673 INFO:teuthology.orchestra.run.smithi001.stdout: "last_clean_end": 0, 2024-09-13T11:26:15.673 INFO:teuthology.orchestra.run.smithi001.stdout: "up_from": 22, 2024-09-13T11:26:15.673 INFO:teuthology.orchestra.run.smithi001.stdout: "up_thru": 192, 2024-09-13T11:26:15.673 INFO:teuthology.orchestra.run.smithi001.stdout: "down_at": 0, 2024-09-13T11:26:15.673 INFO:teuthology.orchestra.run.smithi001.stdout: "lost_at": 0, 2024-09-13T11:26:15.674 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-09-13T11:26:15.674 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:26:15.674 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.674 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:26:15.674 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.162:6804", 2024-09-13T11:26:15.674 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 1450266459 2024-09-13T11:26:15.674 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:15.674 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:15.674 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.674 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_addrs": { 2024-09-13T11:26:15.674 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:26:15.674 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.674 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:26:15.675 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.162:6805", 2024-09-13T11:26:15.675 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 1450266459 2024-09-13T11:26:15.675 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:15.675 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:15.675 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.675 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_back_addrs": { 2024-09-13T11:26:15.675 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:26:15.675 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.675 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:26:15.675 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.162:6807", 2024-09-13T11:26:15.675 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 1450266459 2024-09-13T11:26:15.675 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:15.675 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:15.676 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.676 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_front_addrs": { 2024-09-13T11:26:15.676 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:26:15.676 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.676 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:26:15.676 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.162:6806", 2024-09-13T11:26:15.676 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 1450266459 2024-09-13T11:26:15.676 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:15.676 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:15.676 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.676 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.162:6804/1450266459", 2024-09-13T11:26:15.676 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_addr": "172.21.15.162:6805/1450266459", 2024-09-13T11:26:15.676 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_back_addr": "172.21.15.162:6807/1450266459", 2024-09-13T11:26:15.677 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_front_addr": "172.21.15.162:6806/1450266459", 2024-09-13T11:26:15.677 INFO:teuthology.orchestra.run.smithi001.stdout: "state": [ 2024-09-13T11:26:15.677 INFO:teuthology.orchestra.run.smithi001.stdout: "exists", 2024-09-13T11:26:15.677 INFO:teuthology.orchestra.run.smithi001.stdout: "up" 2024-09-13T11:26:15.677 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:15.677 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.677 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.677 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": 11, 2024-09-13T11:26:15.677 INFO:teuthology.orchestra.run.smithi001.stdout: "uuid": "a4bfdfbe-b586-4080-8e09-6f26428e59b5", 2024-09-13T11:26:15.677 INFO:teuthology.orchestra.run.smithi001.stdout: "up": 1, 2024-09-13T11:26:15.677 INFO:teuthology.orchestra.run.smithi001.stdout: "in": 1, 2024-09-13T11:26:15.677 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 1, 2024-09-13T11:26:15.677 INFO:teuthology.orchestra.run.smithi001.stdout: "primary_affinity": 1, 2024-09-13T11:26:15.678 INFO:teuthology.orchestra.run.smithi001.stdout: "last_clean_begin": 0, 2024-09-13T11:26:15.678 INFO:teuthology.orchestra.run.smithi001.stdout: "last_clean_end": 0, 2024-09-13T11:26:15.678 INFO:teuthology.orchestra.run.smithi001.stdout: "up_from": 22, 2024-09-13T11:26:15.678 INFO:teuthology.orchestra.run.smithi001.stdout: "up_thru": 143, 2024-09-13T11:26:15.678 INFO:teuthology.orchestra.run.smithi001.stdout: "down_at": 0, 2024-09-13T11:26:15.678 INFO:teuthology.orchestra.run.smithi001.stdout: "lost_at": 0, 2024-09-13T11:26:15.678 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-09-13T11:26:15.678 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:26:15.678 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.678 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:26:15.678 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.177:6809", 2024-09-13T11:26:15.678 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 3239521954 2024-09-13T11:26:15.678 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:15.679 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:15.679 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.679 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_addrs": { 2024-09-13T11:26:15.679 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:26:15.679 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.679 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:26:15.679 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.177:6810", 2024-09-13T11:26:15.679 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 3239521954 2024-09-13T11:26:15.679 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:15.679 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:15.679 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.679 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_back_addrs": { 2024-09-13T11:26:15.679 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:26:15.680 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.680 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:26:15.680 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.177:6812", 2024-09-13T11:26:15.680 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 3239521954 2024-09-13T11:26:15.680 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:15.680 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:15.680 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.680 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_front_addrs": { 2024-09-13T11:26:15.680 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:26:15.680 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.680 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:26:15.680 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.177:6811", 2024-09-13T11:26:15.680 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 3239521954 2024-09-13T11:26:15.681 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:15.681 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:15.681 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.681 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.177:6809/3239521954", 2024-09-13T11:26:15.681 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_addr": "172.21.15.177:6810/3239521954", 2024-09-13T11:26:15.681 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_back_addr": "172.21.15.177:6812/3239521954", 2024-09-13T11:26:15.681 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_front_addr": "172.21.15.177:6811/3239521954", 2024-09-13T11:26:15.681 INFO:teuthology.orchestra.run.smithi001.stdout: "state": [ 2024-09-13T11:26:15.681 INFO:teuthology.orchestra.run.smithi001.stdout: "exists", 2024-09-13T11:26:15.681 INFO:teuthology.orchestra.run.smithi001.stdout: "up" 2024-09-13T11:26:15.681 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:15.681 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.681 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.682 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": 12, 2024-09-13T11:26:15.682 INFO:teuthology.orchestra.run.smithi001.stdout: "uuid": "dc571fb6-d8b5-465b-bb25-afa7d0c47b5a", 2024-09-13T11:26:15.682 INFO:teuthology.orchestra.run.smithi001.stdout: "up": 1, 2024-09-13T11:26:15.682 INFO:teuthology.orchestra.run.smithi001.stdout: "in": 1, 2024-09-13T11:26:15.682 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 1, 2024-09-13T11:26:15.682 INFO:teuthology.orchestra.run.smithi001.stdout: "primary_affinity": 1, 2024-09-13T11:26:15.682 INFO:teuthology.orchestra.run.smithi001.stdout: "last_clean_begin": 0, 2024-09-13T11:26:15.682 INFO:teuthology.orchestra.run.smithi001.stdout: "last_clean_end": 0, 2024-09-13T11:26:15.682 INFO:teuthology.orchestra.run.smithi001.stdout: "up_from": 22, 2024-09-13T11:26:15.682 INFO:teuthology.orchestra.run.smithi001.stdout: "up_thru": 141, 2024-09-13T11:26:15.682 INFO:teuthology.orchestra.run.smithi001.stdout: "down_at": 0, 2024-09-13T11:26:15.682 INFO:teuthology.orchestra.run.smithi001.stdout: "lost_at": 0, 2024-09-13T11:26:15.682 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-09-13T11:26:15.682 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:26:15.683 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.683 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:26:15.683 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6812", 2024-09-13T11:26:15.683 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 4267062742 2024-09-13T11:26:15.683 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:15.683 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:15.683 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.683 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_addrs": { 2024-09-13T11:26:15.683 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:26:15.683 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.683 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:26:15.683 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6813", 2024-09-13T11:26:15.683 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 4267062742 2024-09-13T11:26:15.683 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:15.684 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:15.684 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.684 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_back_addrs": { 2024-09-13T11:26:15.684 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:26:15.684 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.684 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:26:15.684 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6815", 2024-09-13T11:26:15.684 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 4267062742 2024-09-13T11:26:15.684 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:15.684 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:15.684 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.684 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_front_addrs": { 2024-09-13T11:26:15.684 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:26:15.684 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.685 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:26:15.685 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6814", 2024-09-13T11:26:15.685 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 4267062742 2024-09-13T11:26:15.685 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:15.685 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:15.685 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.685 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6812/4267062742", 2024-09-13T11:26:15.685 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_addr": "172.21.15.1:6813/4267062742", 2024-09-13T11:26:15.685 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_back_addr": "172.21.15.1:6815/4267062742", 2024-09-13T11:26:15.685 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_front_addr": "172.21.15.1:6814/4267062742", 2024-09-13T11:26:15.685 INFO:teuthology.orchestra.run.smithi001.stdout: "state": [ 2024-09-13T11:26:15.685 INFO:teuthology.orchestra.run.smithi001.stdout: "exists", 2024-09-13T11:26:15.685 INFO:teuthology.orchestra.run.smithi001.stdout: "up" 2024-09-13T11:26:15.686 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:15.686 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.686 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.686 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": 13, 2024-09-13T11:26:15.686 INFO:teuthology.orchestra.run.smithi001.stdout: "uuid": "5d321d4b-1b26-47fe-bf84-25751b8cde7d", 2024-09-13T11:26:15.686 INFO:teuthology.orchestra.run.smithi001.stdout: "up": 1, 2024-09-13T11:26:15.686 INFO:teuthology.orchestra.run.smithi001.stdout: "in": 1, 2024-09-13T11:26:15.686 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 1, 2024-09-13T11:26:15.686 INFO:teuthology.orchestra.run.smithi001.stdout: "primary_affinity": 1, 2024-09-13T11:26:15.686 INFO:teuthology.orchestra.run.smithi001.stdout: "last_clean_begin": 0, 2024-09-13T11:26:15.686 INFO:teuthology.orchestra.run.smithi001.stdout: "last_clean_end": 0, 2024-09-13T11:26:15.686 INFO:teuthology.orchestra.run.smithi001.stdout: "up_from": 23, 2024-09-13T11:26:15.686 INFO:teuthology.orchestra.run.smithi001.stdout: "up_thru": 191, 2024-09-13T11:26:15.687 INFO:teuthology.orchestra.run.smithi001.stdout: "down_at": 0, 2024-09-13T11:26:15.687 INFO:teuthology.orchestra.run.smithi001.stdout: "lost_at": 0, 2024-09-13T11:26:15.687 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-09-13T11:26:15.687 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:26:15.687 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.687 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:26:15.687 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.92:6808", 2024-09-13T11:26:15.687 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 1759585046 2024-09-13T11:26:15.687 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:15.687 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:15.687 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.687 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_addrs": { 2024-09-13T11:26:15.687 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:26:15.688 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.688 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:26:15.688 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.92:6809", 2024-09-13T11:26:15.688 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 1759585046 2024-09-13T11:26:15.688 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:15.688 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:15.688 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.688 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_back_addrs": { 2024-09-13T11:26:15.688 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:26:15.688 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.688 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:26:15.688 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.92:6811", 2024-09-13T11:26:15.688 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 1759585046 2024-09-13T11:26:15.688 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:15.689 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:15.689 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.689 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_front_addrs": { 2024-09-13T11:26:15.689 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:26:15.689 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.689 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:26:15.689 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.92:6810", 2024-09-13T11:26:15.689 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 1759585046 2024-09-13T11:26:15.689 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:15.689 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:15.689 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.689 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.92:6808/1759585046", 2024-09-13T11:26:15.689 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_addr": "172.21.15.92:6809/1759585046", 2024-09-13T11:26:15.690 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_back_addr": "172.21.15.92:6811/1759585046", 2024-09-13T11:26:15.690 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_front_addr": "172.21.15.92:6810/1759585046", 2024-09-13T11:26:15.690 INFO:teuthology.orchestra.run.smithi001.stdout: "state": [ 2024-09-13T11:26:15.690 INFO:teuthology.orchestra.run.smithi001.stdout: "exists", 2024-09-13T11:26:15.690 INFO:teuthology.orchestra.run.smithi001.stdout: "up" 2024-09-13T11:26:15.690 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:15.690 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.690 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.690 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": 14, 2024-09-13T11:26:15.690 INFO:teuthology.orchestra.run.smithi001.stdout: "uuid": "fc3e2499-0c2a-414d-a25e-2628876ae630", 2024-09-13T11:26:15.690 INFO:teuthology.orchestra.run.smithi001.stdout: "up": 1, 2024-09-13T11:26:15.690 INFO:teuthology.orchestra.run.smithi001.stdout: "in": 1, 2024-09-13T11:26:15.691 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 1, 2024-09-13T11:26:15.691 INFO:teuthology.orchestra.run.smithi001.stdout: "primary_affinity": 1, 2024-09-13T11:26:15.691 INFO:teuthology.orchestra.run.smithi001.stdout: "last_clean_begin": 22, 2024-09-13T11:26:15.691 INFO:teuthology.orchestra.run.smithi001.stdout: "last_clean_end": 120, 2024-09-13T11:26:15.691 INFO:teuthology.orchestra.run.smithi001.stdout: "up_from": 142, 2024-09-13T11:26:15.691 INFO:teuthology.orchestra.run.smithi001.stdout: "up_thru": 191, 2024-09-13T11:26:15.691 INFO:teuthology.orchestra.run.smithi001.stdout: "down_at": 121, 2024-09-13T11:26:15.691 INFO:teuthology.orchestra.run.smithi001.stdout: "lost_at": 0, 2024-09-13T11:26:15.691 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-09-13T11:26:15.691 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:26:15.691 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.691 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:26:15.691 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.162:6800", 2024-09-13T11:26:15.692 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 3999392664 2024-09-13T11:26:15.692 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:15.692 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:15.692 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.692 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_addrs": { 2024-09-13T11:26:15.692 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:26:15.692 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.692 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:26:15.692 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.162:6801", 2024-09-13T11:26:15.692 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 3999392664 2024-09-13T11:26:15.692 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:15.692 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:15.693 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.693 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_back_addrs": { 2024-09-13T11:26:15.693 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:26:15.693 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.693 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:26:15.693 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.162:6803", 2024-09-13T11:26:15.693 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 3999392664 2024-09-13T11:26:15.693 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:15.693 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:15.693 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.693 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_front_addrs": { 2024-09-13T11:26:15.693 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:26:15.693 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.694 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:26:15.694 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.162:6802", 2024-09-13T11:26:15.694 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 3999392664 2024-09-13T11:26:15.694 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:15.694 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:15.694 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.694 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.162:6800/3999392664", 2024-09-13T11:26:15.694 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_addr": "172.21.15.162:6801/3999392664", 2024-09-13T11:26:15.694 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_back_addr": "172.21.15.162:6803/3999392664", 2024-09-13T11:26:15.694 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_front_addr": "172.21.15.162:6802/3999392664", 2024-09-13T11:26:15.694 INFO:teuthology.orchestra.run.smithi001.stdout: "state": [ 2024-09-13T11:26:15.694 INFO:teuthology.orchestra.run.smithi001.stdout: "exists", 2024-09-13T11:26:15.694 INFO:teuthology.orchestra.run.smithi001.stdout: "up" 2024-09-13T11:26:15.695 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:15.695 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.695 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.695 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": 15, 2024-09-13T11:26:15.695 INFO:teuthology.orchestra.run.smithi001.stdout: "uuid": "d2f69775-edfe-4527-91ec-f7c7e558bec8", 2024-09-13T11:26:15.695 INFO:teuthology.orchestra.run.smithi001.stdout: "up": 1, 2024-09-13T11:26:15.695 INFO:teuthology.orchestra.run.smithi001.stdout: "in": 1, 2024-09-13T11:26:15.695 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 1, 2024-09-13T11:26:15.695 INFO:teuthology.orchestra.run.smithi001.stdout: "primary_affinity": 1, 2024-09-13T11:26:15.695 INFO:teuthology.orchestra.run.smithi001.stdout: "last_clean_begin": 0, 2024-09-13T11:26:15.695 INFO:teuthology.orchestra.run.smithi001.stdout: "last_clean_end": 0, 2024-09-13T11:26:15.695 INFO:teuthology.orchestra.run.smithi001.stdout: "up_from": 22, 2024-09-13T11:26:15.695 INFO:teuthology.orchestra.run.smithi001.stdout: "up_thru": 122, 2024-09-13T11:26:15.695 INFO:teuthology.orchestra.run.smithi001.stdout: "down_at": 0, 2024-09-13T11:26:15.696 INFO:teuthology.orchestra.run.smithi001.stdout: "lost_at": 0, 2024-09-13T11:26:15.696 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-09-13T11:26:15.696 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:26:15.696 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.696 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:26:15.696 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.177:6805", 2024-09-13T11:26:15.696 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 1294337172 2024-09-13T11:26:15.696 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:15.696 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:15.696 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.696 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_addrs": { 2024-09-13T11:26:15.696 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:26:15.696 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.696 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:26:15.697 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.177:6806", 2024-09-13T11:26:15.697 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 1294337172 2024-09-13T11:26:15.697 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:15.697 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:15.697 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.697 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_back_addrs": { 2024-09-13T11:26:15.697 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:26:15.697 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.697 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:26:15.697 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.177:6808", 2024-09-13T11:26:15.697 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 1294337172 2024-09-13T11:26:15.697 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:15.697 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:15.697 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.698 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_front_addrs": { 2024-09-13T11:26:15.698 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:26:15.698 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.698 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:26:15.698 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.177:6807", 2024-09-13T11:26:15.698 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 1294337172 2024-09-13T11:26:15.698 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:15.698 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:15.698 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.698 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.177:6805/1294337172", 2024-09-13T11:26:15.698 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_addr": "172.21.15.177:6806/1294337172", 2024-09-13T11:26:15.698 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_back_addr": "172.21.15.177:6808/1294337172", 2024-09-13T11:26:15.698 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_front_addr": "172.21.15.177:6807/1294337172", 2024-09-13T11:26:15.698 INFO:teuthology.orchestra.run.smithi001.stdout: "state": [ 2024-09-13T11:26:15.699 INFO:teuthology.orchestra.run.smithi001.stdout: "exists", 2024-09-13T11:26:15.699 INFO:teuthology.orchestra.run.smithi001.stdout: "up" 2024-09-13T11:26:15.699 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:15.699 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:15.699 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-09-13T11:26:15.699 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_xinfo": [ 2024-09-13T11:26:15.699 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.699 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": 0, 2024-09-13T11:26:15.699 INFO:teuthology.orchestra.run.smithi001.stdout: "down_stamp": "0.000000", 2024-09-13T11:26:15.699 INFO:teuthology.orchestra.run.smithi001.stdout: "laggy_probability": 0, 2024-09-13T11:26:15.699 INFO:teuthology.orchestra.run.smithi001.stdout: "laggy_interval": 0, 2024-09-13T11:26:15.699 INFO:teuthology.orchestra.run.smithi001.stdout: "features": 4540719139924082687, 2024-09-13T11:26:15.699 INFO:teuthology.orchestra.run.smithi001.stdout: "old_weight": 0, 2024-09-13T11:26:15.699 INFO:teuthology.orchestra.run.smithi001.stdout: "last_purged_snaps_scrub": "2024-09-13T11:24:57.434802+0000", 2024-09-13T11:26:15.700 INFO:teuthology.orchestra.run.smithi001.stdout: "dead_epoch": 0 2024-09-13T11:26:15.700 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.700 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.700 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": 1, 2024-09-13T11:26:15.700 INFO:teuthology.orchestra.run.smithi001.stdout: "down_stamp": "0.000000", 2024-09-13T11:26:15.700 INFO:teuthology.orchestra.run.smithi001.stdout: "laggy_probability": 0, 2024-09-13T11:26:15.700 INFO:teuthology.orchestra.run.smithi001.stdout: "laggy_interval": 0, 2024-09-13T11:26:15.700 INFO:teuthology.orchestra.run.smithi001.stdout: "features": 4540719139924082687, 2024-09-13T11:26:15.700 INFO:teuthology.orchestra.run.smithi001.stdout: "old_weight": 0, 2024-09-13T11:26:15.700 INFO:teuthology.orchestra.run.smithi001.stdout: "last_purged_snaps_scrub": "2024-09-13T11:25:47.780833+0000", 2024-09-13T11:26:15.700 INFO:teuthology.orchestra.run.smithi001.stdout: "dead_epoch": 30 2024-09-13T11:26:15.700 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.700 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.700 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": 2, 2024-09-13T11:26:15.701 INFO:teuthology.orchestra.run.smithi001.stdout: "down_stamp": "0.000000", 2024-09-13T11:26:15.701 INFO:teuthology.orchestra.run.smithi001.stdout: "laggy_probability": 0, 2024-09-13T11:26:15.701 INFO:teuthology.orchestra.run.smithi001.stdout: "laggy_interval": 0, 2024-09-13T11:26:15.701 INFO:teuthology.orchestra.run.smithi001.stdout: "features": 4540719139924082687, 2024-09-13T11:26:15.701 INFO:teuthology.orchestra.run.smithi001.stdout: "old_weight": 0, 2024-09-13T11:26:15.701 INFO:teuthology.orchestra.run.smithi001.stdout: "last_purged_snaps_scrub": "2024-09-13T11:24:48.897188+0000", 2024-09-13T11:26:15.701 INFO:teuthology.orchestra.run.smithi001.stdout: "dead_epoch": 0 2024-09-13T11:26:15.701 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.701 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.701 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": 3, 2024-09-13T11:26:15.701 INFO:teuthology.orchestra.run.smithi001.stdout: "down_stamp": "0.000000", 2024-09-13T11:26:15.701 INFO:teuthology.orchestra.run.smithi001.stdout: "laggy_probability": 0, 2024-09-13T11:26:15.701 INFO:teuthology.orchestra.run.smithi001.stdout: "laggy_interval": 0, 2024-09-13T11:26:15.702 INFO:teuthology.orchestra.run.smithi001.stdout: "features": 4540719139924082687, 2024-09-13T11:26:15.702 INFO:teuthology.orchestra.run.smithi001.stdout: "old_weight": 0, 2024-09-13T11:26:15.702 INFO:teuthology.orchestra.run.smithi001.stdout: "last_purged_snaps_scrub": "2024-09-13T11:24:47.037766+0000", 2024-09-13T11:26:15.702 INFO:teuthology.orchestra.run.smithi001.stdout: "dead_epoch": 0 2024-09-13T11:26:15.702 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.702 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.702 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": 4, 2024-09-13T11:26:15.702 INFO:teuthology.orchestra.run.smithi001.stdout: "down_stamp": "0.000000", 2024-09-13T11:26:15.702 INFO:teuthology.orchestra.run.smithi001.stdout: "laggy_probability": 0, 2024-09-13T11:26:15.702 INFO:teuthology.orchestra.run.smithi001.stdout: "laggy_interval": 0, 2024-09-13T11:26:15.702 INFO:teuthology.orchestra.run.smithi001.stdout: "features": 4540719139924082687, 2024-09-13T11:26:15.702 INFO:teuthology.orchestra.run.smithi001.stdout: "old_weight": 0, 2024-09-13T11:26:15.702 INFO:teuthology.orchestra.run.smithi001.stdout: "last_purged_snaps_scrub": "2024-09-13T11:25:59.330894+0000", 2024-09-13T11:26:15.702 INFO:teuthology.orchestra.run.smithi001.stdout: "dead_epoch": 0 2024-09-13T11:26:15.703 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.703 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.703 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": 5, 2024-09-13T11:26:15.703 INFO:teuthology.orchestra.run.smithi001.stdout: "down_stamp": "0.000000", 2024-09-13T11:26:15.703 INFO:teuthology.orchestra.run.smithi001.stdout: "laggy_probability": 0, 2024-09-13T11:26:15.703 INFO:teuthology.orchestra.run.smithi001.stdout: "laggy_interval": 0, 2024-09-13T11:26:15.703 INFO:teuthology.orchestra.run.smithi001.stdout: "features": 4540719139924082687, 2024-09-13T11:26:15.703 INFO:teuthology.orchestra.run.smithi001.stdout: "old_weight": 0, 2024-09-13T11:26:15.703 INFO:teuthology.orchestra.run.smithi001.stdout: "last_purged_snaps_scrub": "2024-09-13T11:24:59.304368+0000", 2024-09-13T11:26:15.703 INFO:teuthology.orchestra.run.smithi001.stdout: "dead_epoch": 0 2024-09-13T11:26:15.703 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.703 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.703 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": 6, 2024-09-13T11:26:15.703 INFO:teuthology.orchestra.run.smithi001.stdout: "down_stamp": "0.000000", 2024-09-13T11:26:15.704 INFO:teuthology.orchestra.run.smithi001.stdout: "laggy_probability": 0, 2024-09-13T11:26:15.704 INFO:teuthology.orchestra.run.smithi001.stdout: "laggy_interval": 0, 2024-09-13T11:26:15.704 INFO:teuthology.orchestra.run.smithi001.stdout: "features": 4540719139924082687, 2024-09-13T11:26:15.704 INFO:teuthology.orchestra.run.smithi001.stdout: "old_weight": 0, 2024-09-13T11:26:15.704 INFO:teuthology.orchestra.run.smithi001.stdout: "last_purged_snaps_scrub": "2024-09-13T11:24:52.504800+0000", 2024-09-13T11:26:15.704 INFO:teuthology.orchestra.run.smithi001.stdout: "dead_epoch": 0 2024-09-13T11:26:15.704 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.704 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.704 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": 7, 2024-09-13T11:26:15.704 INFO:teuthology.orchestra.run.smithi001.stdout: "down_stamp": "0.000000", 2024-09-13T11:26:15.704 INFO:teuthology.orchestra.run.smithi001.stdout: "laggy_probability": 0, 2024-09-13T11:26:15.704 INFO:teuthology.orchestra.run.smithi001.stdout: "laggy_interval": 0, 2024-09-13T11:26:15.704 INFO:teuthology.orchestra.run.smithi001.stdout: "features": 4540719139924082687, 2024-09-13T11:26:15.705 INFO:teuthology.orchestra.run.smithi001.stdout: "old_weight": 0, 2024-09-13T11:26:15.705 INFO:teuthology.orchestra.run.smithi001.stdout: "last_purged_snaps_scrub": "2024-09-13T11:26:01.857996+0000", 2024-09-13T11:26:15.705 INFO:teuthology.orchestra.run.smithi001.stdout: "dead_epoch": 0 2024-09-13T11:26:15.705 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.705 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.705 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": 8, 2024-09-13T11:26:15.705 INFO:teuthology.orchestra.run.smithi001.stdout: "down_stamp": "0.000000", 2024-09-13T11:26:15.705 INFO:teuthology.orchestra.run.smithi001.stdout: "laggy_probability": 0, 2024-09-13T11:26:15.705 INFO:teuthology.orchestra.run.smithi001.stdout: "laggy_interval": 0, 2024-09-13T11:26:15.705 INFO:teuthology.orchestra.run.smithi001.stdout: "features": 4540719139924082687, 2024-09-13T11:26:15.705 INFO:teuthology.orchestra.run.smithi001.stdout: "old_weight": 0, 2024-09-13T11:26:15.705 INFO:teuthology.orchestra.run.smithi001.stdout: "last_purged_snaps_scrub": "2024-09-13T11:26:14.379046+0000", 2024-09-13T11:26:15.705 INFO:teuthology.orchestra.run.smithi001.stdout: "dead_epoch": 0 2024-09-13T11:26:15.705 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.706 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.706 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": 9, 2024-09-13T11:26:15.706 INFO:teuthology.orchestra.run.smithi001.stdout: "down_stamp": "0.000000", 2024-09-13T11:26:15.706 INFO:teuthology.orchestra.run.smithi001.stdout: "laggy_probability": 0, 2024-09-13T11:26:15.706 INFO:teuthology.orchestra.run.smithi001.stdout: "laggy_interval": 0, 2024-09-13T11:26:15.706 INFO:teuthology.orchestra.run.smithi001.stdout: "features": 4540719139924082687, 2024-09-13T11:26:15.706 INFO:teuthology.orchestra.run.smithi001.stdout: "old_weight": 0, 2024-09-13T11:26:15.706 INFO:teuthology.orchestra.run.smithi001.stdout: "last_purged_snaps_scrub": "2024-09-13T11:26:09.278966+0000", 2024-09-13T11:26:15.706 INFO:teuthology.orchestra.run.smithi001.stdout: "dead_epoch": 0 2024-09-13T11:26:15.706 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.706 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.706 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": 10, 2024-09-13T11:26:15.706 INFO:teuthology.orchestra.run.smithi001.stdout: "down_stamp": "0.000000", 2024-09-13T11:26:15.707 INFO:teuthology.orchestra.run.smithi001.stdout: "laggy_probability": 0, 2024-09-13T11:26:15.707 INFO:teuthology.orchestra.run.smithi001.stdout: "laggy_interval": 0, 2024-09-13T11:26:15.707 INFO:teuthology.orchestra.run.smithi001.stdout: "features": 4540719139924082687, 2024-09-13T11:26:15.707 INFO:teuthology.orchestra.run.smithi001.stdout: "old_weight": 0, 2024-09-13T11:26:15.707 INFO:teuthology.orchestra.run.smithi001.stdout: "last_purged_snaps_scrub": "2024-09-13T11:26:02.999347+0000", 2024-09-13T11:26:15.707 INFO:teuthology.orchestra.run.smithi001.stdout: "dead_epoch": 0 2024-09-13T11:26:15.707 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.707 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.707 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": 11, 2024-09-13T11:26:15.707 INFO:teuthology.orchestra.run.smithi001.stdout: "down_stamp": "0.000000", 2024-09-13T11:26:15.707 INFO:teuthology.orchestra.run.smithi001.stdout: "laggy_probability": 0, 2024-09-13T11:26:15.707 INFO:teuthology.orchestra.run.smithi001.stdout: "laggy_interval": 0, 2024-09-13T11:26:15.707 INFO:teuthology.orchestra.run.smithi001.stdout: "features": 4540719139924082687, 2024-09-13T11:26:15.707 INFO:teuthology.orchestra.run.smithi001.stdout: "old_weight": 0, 2024-09-13T11:26:15.708 INFO:teuthology.orchestra.run.smithi001.stdout: "last_purged_snaps_scrub": "2024-09-13T11:25:01.213643+0000", 2024-09-13T11:26:15.708 INFO:teuthology.orchestra.run.smithi001.stdout: "dead_epoch": 0 2024-09-13T11:26:15.708 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.708 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.708 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": 12, 2024-09-13T11:26:15.708 INFO:teuthology.orchestra.run.smithi001.stdout: "down_stamp": "0.000000", 2024-09-13T11:26:15.708 INFO:teuthology.orchestra.run.smithi001.stdout: "laggy_probability": 0, 2024-09-13T11:26:15.708 INFO:teuthology.orchestra.run.smithi001.stdout: "laggy_interval": 0, 2024-09-13T11:26:15.708 INFO:teuthology.orchestra.run.smithi001.stdout: "features": 4540719139924082687, 2024-09-13T11:26:15.708 INFO:teuthology.orchestra.run.smithi001.stdout: "old_weight": 0, 2024-09-13T11:26:15.708 INFO:teuthology.orchestra.run.smithi001.stdout: "last_purged_snaps_scrub": "2024-09-13T11:25:47.548386+0000", 2024-09-13T11:26:15.708 INFO:teuthology.orchestra.run.smithi001.stdout: "dead_epoch": 0 2024-09-13T11:26:15.708 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.708 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.709 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": 13, 2024-09-13T11:26:15.709 INFO:teuthology.orchestra.run.smithi001.stdout: "down_stamp": "0.000000", 2024-09-13T11:26:15.709 INFO:teuthology.orchestra.run.smithi001.stdout: "laggy_probability": 0, 2024-09-13T11:26:15.709 INFO:teuthology.orchestra.run.smithi001.stdout: "laggy_interval": 0, 2024-09-13T11:26:15.709 INFO:teuthology.orchestra.run.smithi001.stdout: "features": 4540719139924082687, 2024-09-13T11:26:15.709 INFO:teuthology.orchestra.run.smithi001.stdout: "old_weight": 0, 2024-09-13T11:26:15.709 INFO:teuthology.orchestra.run.smithi001.stdout: "last_purged_snaps_scrub": "2024-09-13T11:25:45.821466+0000", 2024-09-13T11:26:15.709 INFO:teuthology.orchestra.run.smithi001.stdout: "dead_epoch": 0 2024-09-13T11:26:15.709 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.709 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.709 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": 14, 2024-09-13T11:26:15.709 INFO:teuthology.orchestra.run.smithi001.stdout: "down_stamp": "0.000000", 2024-09-13T11:26:15.709 INFO:teuthology.orchestra.run.smithi001.stdout: "laggy_probability": 0, 2024-09-13T11:26:15.709 INFO:teuthology.orchestra.run.smithi001.stdout: "laggy_interval": 0, 2024-09-13T11:26:15.710 INFO:teuthology.orchestra.run.smithi001.stdout: "features": 4540719139924082687, 2024-09-13T11:26:15.710 INFO:teuthology.orchestra.run.smithi001.stdout: "old_weight": 0, 2024-09-13T11:26:15.710 INFO:teuthology.orchestra.run.smithi001.stdout: "last_purged_snaps_scrub": "2024-09-13T11:26:03.927526+0000", 2024-09-13T11:26:15.710 INFO:teuthology.orchestra.run.smithi001.stdout: "dead_epoch": 121 2024-09-13T11:26:15.710 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.710 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.710 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": 15, 2024-09-13T11:26:15.710 INFO:teuthology.orchestra.run.smithi001.stdout: "down_stamp": "0.000000", 2024-09-13T11:26:15.710 INFO:teuthology.orchestra.run.smithi001.stdout: "laggy_probability": 0, 2024-09-13T11:26:15.710 INFO:teuthology.orchestra.run.smithi001.stdout: "laggy_interval": 0, 2024-09-13T11:26:15.710 INFO:teuthology.orchestra.run.smithi001.stdout: "features": 4540719139924082687, 2024-09-13T11:26:15.710 INFO:teuthology.orchestra.run.smithi001.stdout: "old_weight": 0, 2024-09-13T11:26:15.710 INFO:teuthology.orchestra.run.smithi001.stdout: "last_purged_snaps_scrub": "2024-09-13T11:25:57.987079+0000", 2024-09-13T11:26:15.710 INFO:teuthology.orchestra.run.smithi001.stdout: "dead_epoch": 0 2024-09-13T11:26:15.711 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:15.711 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-09-13T11:26:15.711 INFO:teuthology.orchestra.run.smithi001.stdout: "pg_upmap": [ 2024-09-13T11:26:15.711 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.711 INFO:teuthology.orchestra.run.smithi001.stdout: "pgid": "2.2", 2024-09-13T11:26:15.711 INFO:teuthology.orchestra.run.smithi001.stdout: "osds": [ 2024-09-13T11:26:15.711 INFO:teuthology.orchestra.run.smithi001.stdout: 2, 2024-09-13T11:26:15.711 INFO:teuthology.orchestra.run.smithi001.stdout: 8 2024-09-13T11:26:15.711 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:15.711 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.711 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.711 INFO:teuthology.orchestra.run.smithi001.stdout: "pgid": "3.e", 2024-09-13T11:26:15.711 INFO:teuthology.orchestra.run.smithi001.stdout: "osds": [ 2024-09-13T11:26:15.711 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-09-13T11:26:15.712 INFO:teuthology.orchestra.run.smithi001.stdout: 1 2024-09-13T11:26:15.712 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:15.712 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:15.712 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-09-13T11:26:15.712 INFO:teuthology.orchestra.run.smithi001.stdout: "pg_upmap_items": [], 2024-09-13T11:26:15.712 INFO:teuthology.orchestra.run.smithi001.stdout: "pg_upmap_primaries": [], 2024-09-13T11:26:15.712 INFO:teuthology.orchestra.run.smithi001.stdout: "pg_temp": [ 2024-09-13T11:26:15.712 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.712 INFO:teuthology.orchestra.run.smithi001.stdout: "pgid": "3.2", 2024-09-13T11:26:15.712 INFO:teuthology.orchestra.run.smithi001.stdout: "osds": [ 2024-09-13T11:26:15.712 INFO:teuthology.orchestra.run.smithi001.stdout: 3, 2024-09-13T11:26:15.712 INFO:teuthology.orchestra.run.smithi001.stdout: 12 2024-09-13T11:26:15.712 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:15.713 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.713 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.713 INFO:teuthology.orchestra.run.smithi001.stdout: "pgid": "3.12", 2024-09-13T11:26:15.713 INFO:teuthology.orchestra.run.smithi001.stdout: "osds": [ 2024-09-13T11:26:15.713 INFO:teuthology.orchestra.run.smithi001.stdout: 5, 2024-09-13T11:26:15.713 INFO:teuthology.orchestra.run.smithi001.stdout: 7 2024-09-13T11:26:15.713 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:15.713 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.713 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.713 INFO:teuthology.orchestra.run.smithi001.stdout: "pgid": "3.14", 2024-09-13T11:26:15.713 INFO:teuthology.orchestra.run.smithi001.stdout: "osds": [ 2024-09-13T11:26:15.713 INFO:teuthology.orchestra.run.smithi001.stdout: 3, 2024-09-13T11:26:15.713 INFO:teuthology.orchestra.run.smithi001.stdout: 15 2024-09-13T11:26:15.714 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:15.714 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.714 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.714 INFO:teuthology.orchestra.run.smithi001.stdout: "pgid": "3.15", 2024-09-13T11:26:15.714 INFO:teuthology.orchestra.run.smithi001.stdout: "osds": [ 2024-09-13T11:26:15.714 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-09-13T11:26:15.714 INFO:teuthology.orchestra.run.smithi001.stdout: 3 2024-09-13T11:26:15.714 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:15.714 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.714 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.714 INFO:teuthology.orchestra.run.smithi001.stdout: "pgid": "3.1f", 2024-09-13T11:26:15.714 INFO:teuthology.orchestra.run.smithi001.stdout: "osds": [ 2024-09-13T11:26:15.716 INFO:teuthology.orchestra.run.smithi001.stdout: 10, 2024-09-13T11:26:15.716 INFO:teuthology.orchestra.run.smithi001.stdout: 5 2024-09-13T11:26:15.716 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:15.716 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:15.716 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-09-13T11:26:15.716 INFO:teuthology.orchestra.run.smithi001.stdout: "primary_temp": [], 2024-09-13T11:26:15.716 INFO:teuthology.orchestra.run.smithi001.stdout: "blocklist": {}, 2024-09-13T11:26:15.716 INFO:teuthology.orchestra.run.smithi001.stdout: "range_blocklist": {}, 2024-09-13T11:26:15.716 INFO:teuthology.orchestra.run.smithi001.stdout: "erasure_code_profiles": { 2024-09-13T11:26:15.717 INFO:teuthology.orchestra.run.smithi001.stdout: "default": { 2024-09-13T11:26:15.717 INFO:teuthology.orchestra.run.smithi001.stdout: "crush-failure-domain": "osd", 2024-09-13T11:26:15.717 INFO:teuthology.orchestra.run.smithi001.stdout: "k": "2", 2024-09-13T11:26:15.717 INFO:teuthology.orchestra.run.smithi001.stdout: "m": "1", 2024-09-13T11:26:15.717 INFO:teuthology.orchestra.run.smithi001.stdout: "plugin": "jerasure", 2024-09-13T11:26:15.717 INFO:teuthology.orchestra.run.smithi001.stdout: "technique": "reed_sol_van" 2024-09-13T11:26:15.717 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:15.717 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.717 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_snaps_queue": [], 2024-09-13T11:26:15.717 INFO:teuthology.orchestra.run.smithi001.stdout: "new_removed_snaps": [], 2024-09-13T11:26:15.717 INFO:teuthology.orchestra.run.smithi001.stdout: "new_purged_snaps": [], 2024-09-13T11:26:15.717 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_node_flags": {}, 2024-09-13T11:26:15.717 INFO:teuthology.orchestra.run.smithi001.stdout: "device_class_flags": {}, 2024-09-13T11:26:15.717 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": { 2024-09-13T11:26:15.718 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode_enabled": false, 2024-09-13T11:26:15.718 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_bucket_count": 0, 2024-09-13T11:26:15.718 INFO:teuthology.orchestra.run.smithi001.stdout: "degraded_stretch_mode": 0, 2024-09-13T11:26:15.718 INFO:teuthology.orchestra.run.smithi001.stdout: "recovering_stretch_mode": 0, 2024-09-13T11:26:15.718 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode_bucket": 0 2024-09-13T11:26:15.718 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:15.718 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-13T11:26:15.719 INFO:teuthology.orchestra.run.smithi001.stdout:osd.10: {} 2024-09-13T11:26:15.719 INFO:teuthology.orchestra.run.smithi001.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-13T11:26:15.721 INFO:tasks.thrashosds.thrasher:j is {'epoch': 200, 'fsid': '2fd68ac7-5581-4d90-80a4-30bcf5252a7a', 'created': '2024-09-13T11:21:57.223153+0000', 'modified': '2024-09-13T11:26:15.101027+0000', 'last_up_change': '2024-09-13T11:24:55.968636+0000', 'last_in_change': '2024-09-13T11:26:04.007363+0000', 'flags': 'noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5801984, 'flags_set': ['noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 7, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 3, 'max_osd': 16, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-13T11:22:10.162266+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 15.380000114440918, 'score_stable': 15.380000114440918, 'optimal_score': 0.12999999523162842, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2024-09-13T11:22:12.235549+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '28', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 2, 'snap_epoch': 28, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 3.9700000286102295, 'score_stable': 3.9700000286102295, 'optimal_score': 0.6299999952316284, 'raw_score_acting': 2.5, 'raw_score_stable': 2.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'unique_pool_0', 'create_time': '2024-09-13T11:22:24.305521+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 46, 'pg_placement_num': 46, 'pg_placement_num_target': 46, 'pg_num_target': 46, 'pg_num_pending': 46, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '196', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '117', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.0899999141693115, 'score_stable': 2.0899999141693115, 'optimal_score': 1, 'raw_score_acting': 2.0899999141693115, 'raw_score_stable': 2.0899999141693115, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}], 'osds': [{'osd': 0, 'uuid': '0fc12b3c-e0de-4546-b935-db77efe7c567', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 22, 'up_thru': 192, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.1:6800', 'nonce': 1544776477}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.1:6801', 'nonce': 1544776477}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.1:6803', 'nonce': 1544776477}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.1:6802', 'nonce': 1544776477}]}, 'public_addr': '172.21.15.1:6800/1544776477', 'cluster_addr': '172.21.15.1:6801/1544776477', 'heartbeat_back_addr': '172.21.15.1:6803/1544776477', 'heartbeat_front_addr': '172.21.15.1:6802/1544776477', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': 'b77fe331-cc92-4adb-b146-af5c380f6d4f', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 23, 'last_clean_end': 29, 'up_from': 40, 'up_thru': 171, 'down_at': 30, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.92:6800', 'nonce': 3591430847}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.92:6801', 'nonce': 3591430847}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.92:6803', 'nonce': 3591430847}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.92:6802', 'nonce': 3591430847}]}, 'public_addr': '172.21.15.92:6800/3591430847', 'cluster_addr': '172.21.15.92:6801/3591430847', 'heartbeat_back_addr': '172.21.15.92:6803/3591430847', 'heartbeat_front_addr': '172.21.15.92:6802/3591430847', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': 'c9d031c8-e885-4229-affa-08e1729d00e8', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 22, 'up_thru': 152, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.162:6812', 'nonce': 2712148101}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.162:6813', 'nonce': 2712148101}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.162:6815', 'nonce': 2712148101}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.162:6814', 'nonce': 2712148101}]}, 'public_addr': '172.21.15.162:6812/2712148101', 'cluster_addr': '172.21.15.162:6813/2712148101', 'heartbeat_back_addr': '172.21.15.162:6815/2712148101', 'heartbeat_front_addr': '172.21.15.162:6814/2712148101', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': 'bf50d71a-6c3a-4fbc-a407-cb81bad14a7b', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 21, 'up_thru': 192, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.177:6801', 'nonce': 3367751690}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.177:6802', 'nonce': 3367751690}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.177:6804', 'nonce': 3367751690}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.177:6803', 'nonce': 3367751690}]}, 'public_addr': '172.21.15.177:6801/3367751690', 'cluster_addr': '172.21.15.177:6802/3367751690', 'heartbeat_back_addr': '172.21.15.177:6804/3367751690', 'heartbeat_front_addr': '172.21.15.177:6803/3367751690', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': 'd1fe3f63-df3f-4a73-a1e2-8d7c8a3f1fa0', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 22, 'up_thru': 105, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.1:6808', 'nonce': 3648540926}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.1:6809', 'nonce': 3648540926}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.1:6811', 'nonce': 3648540926}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.1:6810', 'nonce': 3648540926}]}, 'public_addr': '172.21.15.1:6808/3648540926', 'cluster_addr': '172.21.15.1:6809/3648540926', 'heartbeat_back_addr': '172.21.15.1:6811/3648540926', 'heartbeat_front_addr': '172.21.15.1:6810/3648540926', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': 'e2ead447-2f2c-40c9-82e7-53b8dc41e2af', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 21, 'up_thru': 192, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.92:6804', 'nonce': 3399529091}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.92:6805', 'nonce': 3399529091}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.92:6807', 'nonce': 3399529091}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.92:6806', 'nonce': 3399529091}]}, 'public_addr': '172.21.15.92:6804/3399529091', 'cluster_addr': '172.21.15.92:6805/3399529091', 'heartbeat_back_addr': '172.21.15.92:6807/3399529091', 'heartbeat_front_addr': '172.21.15.92:6806/3399529091', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': 'dbc20439-144e-4ec7-89de-674eb783f263', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 22, 'up_thru': 117, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.162:6808', 'nonce': 2287343806}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.162:6809', 'nonce': 2287343806}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.162:6811', 'nonce': 2287343806}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.162:6810', 'nonce': 2287343806}]}, 'public_addr': '172.21.15.162:6808/2287343806', 'cluster_addr': '172.21.15.162:6809/2287343806', 'heartbeat_back_addr': '172.21.15.162:6811/2287343806', 'heartbeat_front_addr': '172.21.15.162:6810/2287343806', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '60aa5206-cace-40f8-8d1a-10ec28d97096', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 22, 'up_thru': 122, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.177:6813', 'nonce': 2110809906}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.177:6814', 'nonce': 2110809906}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.177:6816', 'nonce': 2110809906}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.177:6815', 'nonce': 2110809906}]}, 'public_addr': '172.21.15.177:6813/2110809906', 'cluster_addr': '172.21.15.177:6814/2110809906', 'heartbeat_back_addr': '172.21.15.177:6816/2110809906', 'heartbeat_front_addr': '172.21.15.177:6815/2110809906', 'state': ['exists', 'up']}, {'osd': 8, 'uuid': 'f3d99325-48ff-403c-8662-1d7745f7c97d', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 22, 'up_thru': 172, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.1:6804', 'nonce': 2766567910}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.1:6805', 'nonce': 2766567910}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.1:6807', 'nonce': 2766567910}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.1:6806', 'nonce': 2766567910}]}, 'public_addr': '172.21.15.1:6804/2766567910', 'cluster_addr': '172.21.15.1:6805/2766567910', 'heartbeat_back_addr': '172.21.15.1:6807/2766567910', 'heartbeat_front_addr': '172.21.15.1:6806/2766567910', 'state': ['exists', 'up']}, {'osd': 9, 'uuid': '88511e24-3675-4551-968c-1a52e7e28f96', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 23, 'up_thru': 191, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.92:6812', 'nonce': 2488187511}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.92:6813', 'nonce': 2488187511}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.92:6815', 'nonce': 2488187511}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.92:6814', 'nonce': 2488187511}]}, 'public_addr': '172.21.15.92:6812/2488187511', 'cluster_addr': '172.21.15.92:6813/2488187511', 'heartbeat_back_addr': '172.21.15.92:6815/2488187511', 'heartbeat_front_addr': '172.21.15.92:6814/2488187511', 'state': ['exists', 'up']}, {'osd': 10, 'uuid': 'acf4c64d-99af-42be-8406-413c86f54670', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 22, 'up_thru': 192, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.162:6804', 'nonce': 1450266459}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.162:6805', 'nonce': 1450266459}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.162:6807', 'nonce': 1450266459}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.162:6806', 'nonce': 1450266459}]}, 'public_addr': '172.21.15.162:6804/1450266459', 'cluster_addr': '172.21.15.162:6805/1450266459', 'heartbeat_back_addr': '172.21.15.162:6807/1450266459', 'heartbeat_front_addr': '172.21.15.162:6806/1450266459', 'state': ['exists', 'up']}, {'osd': 11, 'uuid': 'a4bfdfbe-b586-4080-8e09-6f26428e59b5', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 22, 'up_thru': 143, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.177:6809', 'nonce': 3239521954}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.177:6810', 'nonce': 3239521954}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.177:6812', 'nonce': 3239521954}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.177:6811', 'nonce': 3239521954}]}, 'public_addr': '172.21.15.177:6809/3239521954', 'cluster_addr': '172.21.15.177:6810/3239521954', 'heartbeat_back_addr': '172.21.15.177:6812/3239521954', 'heartbeat_front_addr': '172.21.15.177:6811/3239521954', 'state': ['exists', 'up']}, {'osd': 12, 'uuid': 'dc571fb6-d8b5-465b-bb25-afa7d0c47b5a', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 22, 'up_thru': 141, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.1:6812', 'nonce': 4267062742}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.1:6813', 'nonce': 4267062742}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.1:6815', 'nonce': 4267062742}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.1:6814', 'nonce': 4267062742}]}, 'public_addr': '172.21.15.1:6812/4267062742', 'cluster_addr': '172.21.15.1:6813/4267062742', 'heartbeat_back_addr': '172.21.15.1:6815/4267062742', 'heartbeat_front_addr': '172.21.15.1:6814/4267062742', 'state': ['exists', 'up']}, {'osd': 13, 'uuid': '5d321d4b-1b26-47fe-bf84-25751b8cde7d', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 23, 'up_thru': 191, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.92:6808', 'nonce': 1759585046}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.92:6809', 'nonce': 1759585046}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.92:6811', 'nonce': 1759585046}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.92:6810', 'nonce': 1759585046}]}, 'public_addr': '172.21.15.92:6808/1759585046', 'cluster_addr': '172.21.15.92:6809/1759585046', 'heartbeat_back_addr': '172.21.15.92:6811/1759585046', 'heartbeat_front_addr': '172.21.15.92:6810/1759585046', 'state': ['exists', 'up']}, {'osd': 14, 'uuid': 'fc3e2499-0c2a-414d-a25e-2628876ae630', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 22, 'last_clean_end': 120, 'up_from': 142, 'up_thru': 191, 'down_at': 121, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.162:6800', 'nonce': 3999392664}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.162:6801', 'nonce': 3999392664}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.162:6803', 'nonce': 3999392664}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.162:6802', 'nonce': 3999392664}]}, 'public_addr': '172.21.15.162:6800/3999392664', 'cluster_addr': '172.21.15.162:6801/3999392664', 'heartbeat_back_addr': '172.21.15.162:6803/3999392664', 'heartbeat_front_addr': '172.21.15.162:6802/3999392664', 'state': ['exists', 'up']}, {'osd': 15, 'uuid': 'd2f69775-edfe-4527-91ec-f7c7e558bec8', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 22, 'up_thru': 122, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.177:6805', 'nonce': 1294337172}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.177:6806', 'nonce': 1294337172}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.177:6808', 'nonce': 1294337172}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.177:6807', 'nonce': 1294337172}]}, 'public_addr': '172.21.15.177:6805/1294337172', 'cluster_addr': '172.21.15.177:6806/1294337172', 'heartbeat_back_addr': '172.21.15.177:6808/1294337172', 'heartbeat_front_addr': '172.21.15.177:6807/1294337172', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-13T11:24:57.434802+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-13T11:25:47.780833+0000', 'dead_epoch': 30}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-13T11:24:48.897188+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-13T11:24:47.037766+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-13T11:25:59.330894+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-13T11:24:59.304368+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-13T11:24:52.504800+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-13T11:26:01.857996+0000', 'dead_epoch': 0}, {'osd': 8, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-13T11:26:14.379046+0000', 'dead_epoch': 0}, {'osd': 9, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-13T11:26:09.278966+0000', 'dead_epoch': 0}, {'osd': 10, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-13T11:26:02.999347+0000', 'dead_epoch': 0}, {'osd': 11, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-13T11:25:01.213643+0000', 'dead_epoch': 0}, {'osd': 12, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-13T11:25:47.548386+0000', 'dead_epoch': 0}, {'osd': 13, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-13T11:25:45.821466+0000', 'dead_epoch': 0}, {'osd': 14, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-13T11:26:03.927526+0000', 'dead_epoch': 121}, {'osd': 15, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-13T11:25:57.987079+0000', 'dead_epoch': 0}], 'pg_upmap': [{'pgid': '2.2', 'osds': [2, 8]}, {'pgid': '3.e', 'osds': [0, 1]}], 'pg_upmap_items': [], 'pg_upmap_primaries': [], 'pg_temp': [{'pgid': '3.2', 'osds': [3, 12]}, {'pgid': '3.12', 'osds': [5, 7]}, {'pgid': '3.14', 'osds': [3, 15]}, {'pgid': '3.15', 'osds': [0, 3]}, {'pgid': '3.1f', 'osds': [10, 5]}], 'primary_temp': [], 'blocklist': {}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-09-13T11:26:15.722 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-13T11:26:15.722 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-13T11:26:15.722 INFO:teuthology.orchestra.run.smithi001.stdout: "size": 20, 2024-09-13T11:26:15.722 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 600, 2024-09-13T11:26:15.722 INFO:teuthology.orchestra.run.smithi001.stdout: "ops": [ 2024-09-13T11:26:15.722 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.723 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDPGPushReply(3.1e 84/78 [PushReplyOp(3:79ee54b9:::benchmark_data_smithi177_31116_object11701:head)])", 2024-09-13T11:26:15.723 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:41.407335+0000", 2024-09-13T11:26:15.723 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 154.23194163299999, 2024-09-13T11:26:15.723 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.51622350400000006, 2024-09-13T11:26:15.723 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:26:15.723 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:26:15.723 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-13T11:26:15.723 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:26:15.723 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.723 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:26:15.723 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:41.407335+0000", 2024-09-13T11:26:15.723 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:15.723 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.723 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.724 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:26:15.724 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:41.407335+0000", 2024-09-13T11:26:15.724 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:15.724 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.724 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.724 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:26:15.724 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:41.407337+0000", 2024-09-13T11:26:15.724 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.505e-06 2024-09-13T11:26:15.724 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.724 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.724 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:26:15.724 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:41.407343+0000", 2024-09-13T11:26:15.724 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.9089999999999997e-06 2024-09-13T11:26:15.725 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.725 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.725 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:26:15.725 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:41.921977+0000", 2024-09-13T11:26:15.725 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.51463463499999995 2024-09-13T11:26:15.725 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.725 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.725 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:26:15.725 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:41.921988+0000", 2024-09-13T11:26:15.725 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.092e-05 2024-09-13T11:26:15.725 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.725 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.725 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:26:15.725 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:41.922088+0000", 2024-09-13T11:26:15.726 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.9528e-05 2024-09-13T11:26:15.726 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.726 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.726 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:26:15.726 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:41.923559+0000", 2024-09-13T11:26:15.726 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0014710070000000001 2024-09-13T11:26:15.726 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:15.726 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:15.726 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:15.726 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.726 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.726 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4665.0:60417 3.10 e85/84 3:092808b0:::benchmark_data_smithi177_31116_object60416:head v 85'3295, mlcod=85'3292)", 2024-09-13T11:26:15.726 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:42.174488+0000", 2024-09-13T11:26:15.727 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 153.46478907900001, 2024-09-13T11:26:15.727 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.387908896, 2024-09-13T11:26:15.727 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:26:15.727 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:26:15.727 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:15.727 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:26:15.727 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.727 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:26:15.727 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.174488+0000", 2024-09-13T11:26:15.727 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:15.727 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.727 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.727 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:26:15.727 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.174488+0000", 2024-09-13T11:26:15.728 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:15.728 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.728 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.728 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:26:15.728 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.174490+0000", 2024-09-13T11:26:15.728 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.9099999999999999e-06 2024-09-13T11:26:15.728 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.728 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.728 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:26:15.728 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.174527+0000", 2024-09-13T11:26:15.728 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.7543000000000003e-05 2024-09-13T11:26:15.728 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.729 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.729 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:26:15.729 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.561093+0000", 2024-09-13T11:26:15.729 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.38656578400000002 2024-09-13T11:26:15.729 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.729 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.729 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:26:15.729 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.561101+0000", 2024-09-13T11:26:15.729 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.6090000000000003e-06 2024-09-13T11:26:15.729 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.729 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.729 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:26:15.729 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.561143+0000", 2024-09-13T11:26:15.729 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.2435000000000003e-05 2024-09-13T11:26:15.730 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.730 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.730 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:26:15.730 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.561164+0000", 2024-09-13T11:26:15.730 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.1276000000000001e-05 2024-09-13T11:26:15.730 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.730 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.730 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:26:15.730 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.562360+0000", 2024-09-13T11:26:15.730 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00119596 2024-09-13T11:26:15.730 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.730 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.730 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:26:15.731 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.562397+0000", 2024-09-13T11:26:15.731 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.6378999999999998e-05 2024-09-13T11:26:15.731 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:15.731 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:15.731 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:15.731 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.731 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.731 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDPGPush(3.22 85/78 [PushOp(3:456e8765:::benchmark_data_smithi177_31116_object16212:head, version: 52'1006, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:456e8765:::benchmark_data_smithi177_31116_object16212:head@52'1006, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:26:15.731 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:42.673067+0000", 2024-09-13T11:26:15.731 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 152.966209937, 2024-09-13T11:26:15.731 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.27935889899999999, 2024-09-13T11:26:15.731 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:26:15.732 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:26:15.732 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:26:15.732 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:26:15.732 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.732 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:26:15.732 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.673067+0000", 2024-09-13T11:26:15.732 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:15.732 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.732 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.732 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:26:15.733 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.673067+0000", 2024-09-13T11:26:15.733 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:15.733 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.733 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.733 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:26:15.733 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.673068+0000", 2024-09-13T11:26:15.733 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.9699999999999994e-07 2024-09-13T11:26:15.733 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.733 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.733 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:26:15.733 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.673090+0000", 2024-09-13T11:26:15.733 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.2416999999999999e-05 2024-09-13T11:26:15.733 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.734 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.734 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:26:15.734 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.951363+0000", 2024-09-13T11:26:15.734 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.27827291199999998 2024-09-13T11:26:15.734 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.734 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.734 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:26:15.734 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.951386+0000", 2024-09-13T11:26:15.734 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.2226999999999999e-05 2024-09-13T11:26:15.734 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.734 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.734 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:26:15.735 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.951581+0000", 2024-09-13T11:26:15.735 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000195713 2024-09-13T11:26:15.735 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.735 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.735 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:26:15.735 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.951605+0000", 2024-09-13T11:26:15.735 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.3767000000000001e-05 2024-09-13T11:26:15.735 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.735 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.735 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:26:15.736 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.952426+0000", 2024-09-13T11:26:15.736 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00082086599999999996 2024-09-13T11:26:15.736 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:15.736 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:15.736 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:15.736 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.736 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.736 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "replica_scrub(pg: 3.20,from:0'0,to:89'3254,epoch:89/77,start:3:04000000::::0,end:3:045ba805:::benchmark_data_smithi177_31116_object43251:0,chunky:1,deep:0,version:9,allow_preemption:1,priority=5)", 2024-09-13T11:26:15.736 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:46.641364+0000", 2024-09-13T11:26:15.736 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 148.997913256, 2024-09-13T11:26:15.736 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.27141393000000003, 2024-09-13T11:26:15.736 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:26:15.736 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:26:15.737 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:26:15.737 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:26:15.737 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.737 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:26:15.737 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.641364+0000", 2024-09-13T11:26:15.737 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:15.737 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.737 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.737 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:26:15.737 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.641364+0000", 2024-09-13T11:26:15.737 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:15.737 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.737 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.738 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:26:15.738 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.641365+0000", 2024-09-13T11:26:15.738 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.266e-06 2024-09-13T11:26:15.738 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.738 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.738 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:26:15.738 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.641370+0000", 2024-09-13T11:26:15.738 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.8539999999999996e-06 2024-09-13T11:26:15.738 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.738 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.738 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:26:15.738 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.912491+0000", 2024-09-13T11:26:15.738 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.27112103100000001 2024-09-13T11:26:15.739 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.739 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.739 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:26:15.739 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.912520+0000", 2024-09-13T11:26:15.739 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.9379000000000001e-05 2024-09-13T11:26:15.739 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.739 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.739 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:26:15.739 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.912679+0000", 2024-09-13T11:26:15.739 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00015915800000000001 2024-09-13T11:26:15.739 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.739 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.739 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:26:15.740 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.912714+0000", 2024-09-13T11:26:15.740 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.4687999999999997e-05 2024-09-13T11:26:15.740 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.740 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.740 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:26:15.740 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.912778+0000", 2024-09-13T11:26:15.740 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.3553999999999995e-05 2024-09-13T11:26:15.740 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:15.740 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:15.740 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:15.740 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.740 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.740 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDPGPush(3.18 89/70 [PushOp(3:1ac6c361:::benchmark_data_smithi177_31116_object44972:head, version: 68'2767, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:1ac6c361:::benchmark_data_smithi177_31116_object44972:head@68'2767, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:26:15.741 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:46.671018+0000", 2024-09-13T11:26:15.741 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 148.96825912400001, 2024-09-13T11:26:15.741 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.271282931, 2024-09-13T11:26:15.741 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:26:15.741 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:26:15.741 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:26:15.741 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:26:15.741 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.741 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:26:15.741 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.671018+0000", 2024-09-13T11:26:15.741 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:15.741 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.741 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.742 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:26:15.742 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.671018+0000", 2024-09-13T11:26:15.742 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:15.742 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.742 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.742 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:26:15.742 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.671020+0000", 2024-09-13T11:26:15.742 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.018e-06 2024-09-13T11:26:15.742 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.742 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.742 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:26:15.742 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.671097+0000", 2024-09-13T11:26:15.742 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.7218999999999994e-05 2024-09-13T11:26:15.743 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.743 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.743 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:26:15.743 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.939773+0000", 2024-09-13T11:26:15.743 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.26867550200000001 2024-09-13T11:26:15.743 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.743 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.743 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:26:15.743 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.939791+0000", 2024-09-13T11:26:15.743 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.8185999999999998e-05 2024-09-13T11:26:15.743 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.743 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.743 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:26:15.743 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.939926+0000", 2024-09-13T11:26:15.744 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000135291 2024-09-13T11:26:15.744 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.744 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.744 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:26:15.744 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.939938+0000", 2024-09-13T11:26:15.744 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2393e-05 2024-09-13T11:26:15.744 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.744 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.744 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:26:15.744 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.942301+0000", 2024-09-13T11:26:15.744 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.002362322 2024-09-13T11:26:15.744 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:15.744 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:15.744 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:15.744 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.745 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.745 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "replica_scrub(pg: 3.20,from:0'0,to:90'3264,epoch:90/77,start:3:0650aac8:::benchmark_data_smithi177_31116_object1734:0,end:3:06af01e3:::benchmark_data_smithi177_31116_object31572:0,chunky:1,deep:0,version:9,allow_preemption:1,priority=5)", 2024-09-13T11:26:15.745 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:47.380615+0000", 2024-09-13T11:26:15.745 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 148.25866170099999, 2024-09-13T11:26:15.745 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.51951398400000004, 2024-09-13T11:26:15.745 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:26:15.745 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:26:15.745 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:26:15.745 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:26:15.745 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.745 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:26:15.745 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:47.380615+0000", 2024-09-13T11:26:15.745 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:15.746 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.746 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.746 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:26:15.746 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:47.380615+0000", 2024-09-13T11:26:15.746 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:15.746 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.746 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.746 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:26:15.746 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:47.380617+0000", 2024-09-13T11:26:15.746 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.671e-06 2024-09-13T11:26:15.746 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.746 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.746 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:26:15.747 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:47.380623+0000", 2024-09-13T11:26:15.747 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.1190000000000002e-06 2024-09-13T11:26:15.747 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.747 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.747 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:26:15.747 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:47.900050+0000", 2024-09-13T11:26:15.747 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.51942737400000005 2024-09-13T11:26:15.747 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.747 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.747 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:26:15.747 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:47.900057+0000", 2024-09-13T11:26:15.747 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.4359999999999999e-06 2024-09-13T11:26:15.748 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.748 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.748 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:26:15.748 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:47.900101+0000", 2024-09-13T11:26:15.748 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.4675000000000002e-05 2024-09-13T11:26:15.748 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.748 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.748 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:26:15.748 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:47.900110+0000", 2024-09-13T11:26:15.748 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.0569999999999993e-06 2024-09-13T11:26:15.748 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.748 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.748 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:26:15.748 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:47.900129+0000", 2024-09-13T11:26:15.749 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.9652000000000001e-05 2024-09-13T11:26:15.749 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:15.749 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:15.749 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:15.749 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.749 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.749 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDPGPush(3.21 91/78 [PushOp(3:8492321f:::benchmark_data_smithi177_31116_object13767:head, version: 50'840, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:8492321f:::benchmark_data_smithi177_31116_object13767:head@50'840, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:26:15.749 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:48.319251+0000", 2024-09-13T11:26:15.749 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 147.32002617200001, 2024-09-13T11:26:15.749 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.60903802200000001, 2024-09-13T11:26:15.749 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:26:15.749 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:26:15.750 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:26:15.750 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:26:15.750 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.750 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:26:15.750 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:48.319251+0000", 2024-09-13T11:26:15.750 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:15.750 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.750 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.750 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:26:15.750 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:48.319251+0000", 2024-09-13T11:26:15.750 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:15.750 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.750 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.751 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:26:15.751 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:48.319252+0000", 2024-09-13T11:26:15.751 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2640000000000001e-06 2024-09-13T11:26:15.751 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.751 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.751 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:26:15.751 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:48.319363+0000", 2024-09-13T11:26:15.751 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000110559 2024-09-13T11:26:15.751 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.751 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.751 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:26:15.751 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:48.927115+0000", 2024-09-13T11:26:15.751 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.60775277800000005 2024-09-13T11:26:15.751 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.751 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.752 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:26:15.752 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:48.927144+0000", 2024-09-13T11:26:15.752 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.8861e-05 2024-09-13T11:26:15.752 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.752 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.752 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:26:15.752 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:48.927318+0000", 2024-09-13T11:26:15.752 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00017399000000000001 2024-09-13T11:26:15.752 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.752 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.752 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:26:15.752 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:48.927340+0000", 2024-09-13T11:26:15.752 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.1433999999999998e-05 2024-09-13T11:26:15.752 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.753 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.753 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:26:15.753 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:48.928289+0000", 2024-09-13T11:26:15.753 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00094913599999999995 2024-09-13T11:26:15.753 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:15.753 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:15.753 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:15.753 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.753 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.753 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:83239 3.e 3:74547f17:::benchmark_data_smithi177_31116_object83238:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106)", 2024-09-13T11:26:15.753 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:24:04.580085+0000", 2024-09-13T11:26:15.753 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 131.059192091, 2024-09-13T11:26:15.754 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.33785762600000002, 2024-09-13T11:26:15.754 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:26:15.754 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:26:15.754 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:15.754 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:26:15.754 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:26:15.754 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:15.754 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 83239 2024-09-13T11:26:15.754 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.754 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:26:15.754 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.754 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:26:15.754 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:04.580085+0000", 2024-09-13T11:26:15.754 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:15.755 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.755 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.755 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:26:15.755 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:04.580085+0000", 2024-09-13T11:26:15.755 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:15.755 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.755 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.755 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:26:15.755 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:04.580089+0000", 2024-09-13T11:26:15.755 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.053e-06 2024-09-13T11:26:15.755 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.755 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.755 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:26:15.755 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:04.580166+0000", 2024-09-13T11:26:15.755 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.7102999999999997e-05 2024-09-13T11:26:15.756 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.756 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.756 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:26:15.756 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:04.580196+0000", 2024-09-13T11:26:15.756 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.0196e-05 2024-09-13T11:26:15.756 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.756 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.756 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:26:15.756 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:04.580212+0000", 2024-09-13T11:26:15.756 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.535e-05 2024-09-13T11:26:15.756 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.756 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.756 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:26:15.756 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:04.580336+0000", 2024-09-13T11:26:15.757 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00012431899999999999 2024-09-13T11:26:15.757 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.757 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.757 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:26:15.757 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:04.580575+0000", 2024-09-13T11:26:15.757 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00023924100000000001 2024-09-13T11:26:15.757 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.757 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.757 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 1", 2024-09-13T11:26:15.757 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:04.580798+0000", 2024-09-13T11:26:15.757 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00022301499999999999 2024-09-13T11:26:15.757 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.757 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.758 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:26:15.758 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:04.583404+0000", 2024-09-13T11:26:15.758 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0026056120000000002 2024-09-13T11:26:15.758 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.758 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.758 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:15.758 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:04.917768+0000", 2024-09-13T11:26:15.758 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.33436387499999998 2024-09-13T11:26:15.758 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.758 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.758 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:26:15.758 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:04.917859+0000", 2024-09-13T11:26:15.758 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.1732999999999997e-05 2024-09-13T11:26:15.759 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.759 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.759 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:26:15.759 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:04.917942+0000", 2024-09-13T11:26:15.759 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.3128999999999996e-05 2024-09-13T11:26:15.759 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:15.759 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:15.759 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:15.759 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.759 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.759 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:110271 3.2d 3.3c2681ad (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e119)", 2024-09-13T11:26:15.759 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:24:25.403006+0000", 2024-09-13T11:26:15.759 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 110.23627060699999, 2024-09-13T11:26:15.760 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.68768755999999998, 2024-09-13T11:26:15.760 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:26:15.760 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:26:15.760 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-13T11:26:15.760 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:26:15.760 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:26:15.760 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:15.760 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 110271 2024-09-13T11:26:15.760 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.760 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:26:15.760 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.760 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:26:15.760 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:25.403006+0000", 2024-09-13T11:26:15.761 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:15.761 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.761 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.761 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:26:15.761 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:25.403006+0000", 2024-09-13T11:26:15.761 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:15.761 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.761 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.761 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:26:15.761 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:25.403011+0000", 2024-09-13T11:26:15.761 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.5199999999999999e-06 2024-09-13T11:26:15.761 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.761 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.761 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:26:15.762 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:25.403065+0000", 2024-09-13T11:26:15.762 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.4478999999999998e-05 2024-09-13T11:26:15.762 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.762 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.762 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:26:15.762 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:25.403097+0000", 2024-09-13T11:26:15.762 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.1636e-05 2024-09-13T11:26:15.762 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.762 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.762 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:26:15.762 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:25.403121+0000", 2024-09-13T11:26:15.762 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.3924e-05 2024-09-13T11:26:15.762 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.763 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.763 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:26:15.763 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:25.403216+0000", 2024-09-13T11:26:15.763 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.5557999999999995e-05 2024-09-13T11:26:15.763 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.763 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.763 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for peered", 2024-09-13T11:26:15.763 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:25.403234+0000", 2024-09-13T11:26:15.763 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.7104000000000001e-05 2024-09-13T11:26:15.763 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.763 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.763 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:26:15.763 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:26.090684+0000", 2024-09-13T11:26:15.764 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.68745000700000003 2024-09-13T11:26:15.764 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.764 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.764 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:26:15.764 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:26.090694+0000", 2024-09-13T11:26:15.764 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0332e-05 2024-09-13T11:26:15.764 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:15.764 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:15.764 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:15.764 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.764 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.764 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4665.0:129901 3.20 e134/77 3:067f5dfc:::benchmark_data_smithi177_31116_object129900:head v 134'4294, mlcod=134'4291)", 2024-09-13T11:26:15.764 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:24:47.145530+0000", 2024-09-13T11:26:15.764 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 88.493746532000003, 2024-09-13T11:26:15.765 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.333688349, 2024-09-13T11:26:15.765 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:26:15.765 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:26:15.765 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:15.765 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:26:15.765 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.765 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:26:15.765 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:47.145530+0000", 2024-09-13T11:26:15.765 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:15.765 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.765 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.765 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:26:15.766 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:47.145530+0000", 2024-09-13T11:26:15.766 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:15.766 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.766 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.766 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:26:15.766 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:47.145532+0000", 2024-09-13T11:26:15.766 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.7999999999999999e-06 2024-09-13T11:26:15.766 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.766 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.766 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:26:15.766 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:47.145557+0000", 2024-09-13T11:26:15.766 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.4678999999999999e-05 2024-09-13T11:26:15.766 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.766 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.767 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:26:15.767 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:47.477539+0000", 2024-09-13T11:26:15.767 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.33198164099999999 2024-09-13T11:26:15.767 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.767 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.767 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:26:15.767 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:47.477544+0000", 2024-09-13T11:26:15.767 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.9059999999999999e-06 2024-09-13T11:26:15.767 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.767 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.767 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:26:15.767 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:47.477597+0000", 2024-09-13T11:26:15.767 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.295e-05 2024-09-13T11:26:15.767 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.767 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.768 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:26:15.768 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:47.477619+0000", 2024-09-13T11:26:15.768 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.1835999999999999e-05 2024-09-13T11:26:15.768 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.768 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.768 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:26:15.768 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:47.479175+0000", 2024-09-13T11:26:15.768 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0015560470000000001 2024-09-13T11:26:15.768 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.768 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.768 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:26:15.768 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:47.479219+0000", 2024-09-13T11:26:15.768 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.3489999999999999e-05 2024-09-13T11:26:15.768 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:15.769 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:15.769 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:15.769 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.769 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.769 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:140840 3.21 3:87ae3db6:::benchmark_data_smithi177_31116_object537:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e144)", 2024-09-13T11:26:15.769 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:24:58.810939+0000", 2024-09-13T11:26:15.769 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 76.828337418999993, 2024-09-13T11:26:15.769 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.25628309300000002, 2024-09-13T11:26:15.769 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:26:15.769 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:26:15.769 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:15.769 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:26:15.769 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:26:15.769 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:15.770 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 140840 2024-09-13T11:26:15.770 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.770 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:26:15.770 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.770 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:26:15.770 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.810939+0000", 2024-09-13T11:26:15.770 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:15.770 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.770 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.770 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:26:15.770 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.810939+0000", 2024-09-13T11:26:15.770 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:15.770 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.771 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.771 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:26:15.771 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.810942+0000", 2024-09-13T11:26:15.771 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.6259999999999999e-06 2024-09-13T11:26:15.771 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.771 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.771 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:26:15.771 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.810950+0000", 2024-09-13T11:26:15.771 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.9500000000000001e-06 2024-09-13T11:26:15.771 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.771 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.771 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:26:15.771 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.810970+0000", 2024-09-13T11:26:15.771 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.9539999999999999e-05 2024-09-13T11:26:15.771 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.772 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.772 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:26:15.772 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.810978+0000", 2024-09-13T11:26:15.772 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.8680000000000005e-06 2024-09-13T11:26:15.772 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.772 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.772 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:26:15.772 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.811060+0000", 2024-09-13T11:26:15.772 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.1068000000000004e-05 2024-09-13T11:26:15.772 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.772 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.772 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:26:15.772 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.811174+0000", 2024-09-13T11:26:15.772 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000114385 2024-09-13T11:26:15.773 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.773 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.773 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 8", 2024-09-13T11:26:15.773 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.811255+0000", 2024-09-13T11:26:15.773 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.1298000000000007e-05 2024-09-13T11:26:15.773 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.773 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.773 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:26:15.773 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.812869+0000", 2024-09-13T11:26:15.773 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.001614234 2024-09-13T11:26:15.773 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.773 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.773 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:15.773 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:59.067155+0000", 2024-09-13T11:26:15.773 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.25428578299999999 2024-09-13T11:26:15.774 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.774 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.774 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:26:15.774 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:59.067192+0000", 2024-09-13T11:26:15.774 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.659e-05 2024-09-13T11:26:15.774 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.774 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.774 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:26:15.774 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:59.067223+0000", 2024-09-13T11:26:15.774 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.0750999999999997e-05 2024-09-13T11:26:15.774 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:15.774 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:15.774 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:15.774 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.774 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.775 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop_reply(client.4665.0:188185 3.22 e174/122 ondisk, result = 0)", 2024-09-13T11:26:15.775 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:25:36.235023+0000", 2024-09-13T11:26:15.775 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 39.404253636, 2024-09-13T11:26:15.775 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.23136179600000001, 2024-09-13T11:26:15.775 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:26:15.775 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:26:15.775 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:26:15.775 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:26:15.775 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.775 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:26:15.775 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:36.235023+0000", 2024-09-13T11:26:15.775 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:15.775 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.775 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.775 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:26:15.776 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:36.235023+0000", 2024-09-13T11:26:15.776 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:15.776 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.776 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.776 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:26:15.776 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:36.235024+0000", 2024-09-13T11:26:15.776 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.212e-06 2024-09-13T11:26:15.776 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.776 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.776 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:26:15.776 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:36.235029+0000", 2024-09-13T11:26:15.776 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.4839999999999998e-06 2024-09-13T11:26:15.776 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.776 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.777 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:26:15.777 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:36.466056+0000", 2024-09-13T11:26:15.777 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.23102687 2024-09-13T11:26:15.777 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.777 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.777 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:26:15.777 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:36.466061+0000", 2024-09-13T11:26:15.777 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.8629999999999999e-06 2024-09-13T11:26:15.777 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.777 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.777 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:26:15.777 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:36.466133+0000", 2024-09-13T11:26:15.777 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.1999e-05 2024-09-13T11:26:15.777 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.778 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.778 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:26:15.778 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:36.466148+0000", 2024-09-13T11:26:15.778 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.4800000000000001e-05 2024-09-13T11:26:15.778 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.778 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.778 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:26:15.778 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:36.466385+0000", 2024-09-13T11:26:15.778 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00023756800000000001 2024-09-13T11:26:15.778 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:15.778 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:15.778 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:15.778 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.778 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.779 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4665.0:191286 3.18 e175/91 3:1ffc0b26:::benchmark_data_smithi177_31116_object50983:head v 175'7339, mlcod=175'7337)", 2024-09-13T11:26:15.779 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:25:39.153114+0000", 2024-09-13T11:26:15.779 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 36.486162806999999, 2024-09-13T11:26:15.779 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.23200507400000001, 2024-09-13T11:26:15.779 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:26:15.779 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:26:15.779 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:15.779 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:26:15.779 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.779 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:26:15.779 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:39.153114+0000", 2024-09-13T11:26:15.779 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:15.779 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.779 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.780 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:26:15.780 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:39.153114+0000", 2024-09-13T11:26:15.780 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:15.780 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.780 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.780 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:26:15.780 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:39.153117+0000", 2024-09-13T11:26:15.780 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.9170000000000002e-06 2024-09-13T11:26:15.780 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.780 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.780 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:26:15.780 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:39.153130+0000", 2024-09-13T11:26:15.780 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2493e-05 2024-09-13T11:26:15.780 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.781 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.781 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:26:15.781 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:39.381951+0000", 2024-09-13T11:26:15.781 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.22882103000000001 2024-09-13T11:26:15.781 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.781 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.781 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:26:15.781 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:39.381956+0000", 2024-09-13T11:26:15.781 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.1359999999999996e-06 2024-09-13T11:26:15.781 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.781 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.781 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:26:15.782 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:39.382030+0000", 2024-09-13T11:26:15.782 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.4270999999999994e-05 2024-09-13T11:26:15.782 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.782 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.782 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:26:15.782 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:39.382054+0000", 2024-09-13T11:26:15.782 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.4142999999999999e-05 2024-09-13T11:26:15.782 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.782 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.782 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:26:15.782 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:39.385086+0000", 2024-09-13T11:26:15.782 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00303163 2024-09-13T11:26:15.782 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.782 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.783 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:26:15.783 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:39.385119+0000", 2024-09-13T11:26:15.783 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.3454000000000002e-05 2024-09-13T11:26:15.783 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:15.783 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:15.783 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:15.783 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.783 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.783 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4665.0:196438 3.20 e178/77 3:0573f1a4:::benchmark_data_smithi177_31116_object56135:head v 178'5356, mlcod=178'5353)", 2024-09-13T11:26:15.783 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:25:43.522272+0000", 2024-09-13T11:26:15.783 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 32.117004717, 2024-09-13T11:26:15.783 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.23723141, 2024-09-13T11:26:15.783 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:26:15.784 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:26:15.784 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:15.784 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:26:15.784 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.784 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:26:15.784 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:43.522272+0000", 2024-09-13T11:26:15.784 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:15.784 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.784 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.784 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:26:15.784 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:43.522272+0000", 2024-09-13T11:26:15.784 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:15.784 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.784 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.785 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:26:15.785 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:43.522274+0000", 2024-09-13T11:26:15.785 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.573e-06 2024-09-13T11:26:15.785 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.785 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.785 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:26:15.785 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:43.522281+0000", 2024-09-13T11:26:15.785 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.5490000000000001e-06 2024-09-13T11:26:15.785 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.785 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.785 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:26:15.785 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:43.756836+0000", 2024-09-13T11:26:15.785 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.23455496000000001 2024-09-13T11:26:15.785 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.786 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.786 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:26:15.786 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:43.756843+0000", 2024-09-13T11:26:15.786 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.4620000000000001e-06 2024-09-13T11:26:15.786 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.786 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.786 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:26:15.786 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:43.756914+0000", 2024-09-13T11:26:15.786 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.1276e-05 2024-09-13T11:26:15.786 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.786 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.786 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:26:15.786 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:43.756938+0000", 2024-09-13T11:26:15.786 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.4281999999999999e-05 2024-09-13T11:26:15.787 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.787 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.787 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:26:15.787 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:43.759408+0000", 2024-09-13T11:26:15.787 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0024699449999999999 2024-09-13T11:26:15.787 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.787 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.787 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:26:15.787 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:43.759504+0000", 2024-09-13T11:26:15.787 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.5363000000000002e-05 2024-09-13T11:26:15.787 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:15.787 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:15.787 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:15.787 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.788 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.788 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:201234 3.21 3:85402570:::benchmark_data_smithi177_31116_object60931:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e181)", 2024-09-13T11:26:15.788 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:25:47.835618+0000", 2024-09-13T11:26:15.788 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 27.803659253999999, 2024-09-13T11:26:15.788 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.26153554899999998, 2024-09-13T11:26:15.788 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:26:15.788 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:26:15.788 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:15.788 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:26:15.788 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:26:15.788 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:15.788 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 201234 2024-09-13T11:26:15.788 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.789 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:26:15.789 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.789 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:26:15.789 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:47.835618+0000", 2024-09-13T11:26:15.789 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:15.789 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.789 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.789 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:26:15.789 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:47.835618+0000", 2024-09-13T11:26:15.789 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:15.789 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.789 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.789 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:26:15.789 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:47.835619+0000", 2024-09-13T11:26:15.790 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.638e-06 2024-09-13T11:26:15.790 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.790 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.790 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:26:15.790 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:47.835627+0000", 2024-09-13T11:26:15.790 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.0150000000000003e-06 2024-09-13T11:26:15.790 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.790 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.790 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:26:15.790 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:47.835639+0000", 2024-09-13T11:26:15.790 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.1939e-05 2024-09-13T11:26:15.790 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.790 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.790 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:26:15.791 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:47.835644+0000", 2024-09-13T11:26:15.791 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.006e-06 2024-09-13T11:26:15.791 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.791 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.791 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:26:15.791 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:47.835696+0000", 2024-09-13T11:26:15.791 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.1668000000000001e-05 2024-09-13T11:26:15.791 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.791 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.791 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:26:15.791 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:47.835775+0000", 2024-09-13T11:26:15.791 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.8737000000000002e-05 2024-09-13T11:26:15.791 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.791 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.792 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 8", 2024-09-13T11:26:15.792 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:47.835857+0000", 2024-09-13T11:26:15.792 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.2508000000000004e-05 2024-09-13T11:26:15.792 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.792 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.792 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:26:15.792 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:47.837366+0000", 2024-09-13T11:26:15.792 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.001509292 2024-09-13T11:26:15.792 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.792 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.792 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:15.792 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:48.097051+0000", 2024-09-13T11:26:15.792 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.25968466699999998 2024-09-13T11:26:15.793 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.793 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.793 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:26:15.793 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:48.097109+0000", 2024-09-13T11:26:15.793 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.7778000000000001e-05 2024-09-13T11:26:15.793 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.793 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.793 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:26:15.793 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:48.097153+0000", 2024-09-13T11:26:15.793 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.4301000000000002e-05 2024-09-13T11:26:15.793 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:15.793 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:15.793 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:15.794 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.794 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.794 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:217082 3.22 3:449e4b87:::benchmark_data_smithi177_31116_object76779:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e187)", 2024-09-13T11:26:15.794 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:25:59.883418+0000", 2024-09-13T11:26:15.794 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 15.755859257000001, 2024-09-13T11:26:15.794 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.25557793200000001, 2024-09-13T11:26:15.794 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:26:15.794 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:26:15.794 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:15.794 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:26:15.794 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:26:15.794 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:15.794 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 217082 2024-09-13T11:26:15.795 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.795 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:26:15.795 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.795 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:26:15.795 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:59.883418+0000", 2024-09-13T11:26:15.795 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:15.795 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.795 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.795 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:26:15.795 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:59.883418+0000", 2024-09-13T11:26:15.795 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:15.795 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.795 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.795 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:26:15.795 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:59.883419+0000", 2024-09-13T11:26:15.796 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.235e-06 2024-09-13T11:26:15.796 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.796 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.796 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:26:15.796 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:59.883423+0000", 2024-09-13T11:26:15.796 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.7280000000000001e-06 2024-09-13T11:26:15.796 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.796 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.796 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:26:15.796 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:59.883433+0000", 2024-09-13T11:26:15.796 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0498e-05 2024-09-13T11:26:15.796 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.796 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.796 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:26:15.797 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:59.883439+0000", 2024-09-13T11:26:15.797 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.5890000000000002e-06 2024-09-13T11:26:15.797 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.797 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.797 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:26:15.797 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:59.883483+0000", 2024-09-13T11:26:15.797 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.4710999999999999e-05 2024-09-13T11:26:15.797 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.797 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.797 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:26:15.797 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:59.883558+0000", 2024-09-13T11:26:15.797 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.4308999999999996e-05 2024-09-13T11:26:15.797 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.798 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.798 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 5", 2024-09-13T11:26:15.798 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:59.883615+0000", 2024-09-13T11:26:15.798 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.7745999999999999e-05 2024-09-13T11:26:15.798 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.798 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.798 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:26:15.798 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:59.885044+0000", 2024-09-13T11:26:15.798 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0014282940000000001 2024-09-13T11:26:15.798 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.798 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.798 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:15.798 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:00.138864+0000", 2024-09-13T11:26:15.798 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.253819923 2024-09-13T11:26:15.799 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.799 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.799 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:26:15.799 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:00.138936+0000", 2024-09-13T11:26:15.799 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.2268999999999995e-05 2024-09-13T11:26:15.799 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.799 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.799 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:26:15.799 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:00.138996+0000", 2024-09-13T11:26:15.799 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.9630000000000003e-05 2024-09-13T11:26:15.799 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:15.799 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:15.799 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:15.800 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.800 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.800 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:221463 3.15 3:aaa4e83d:::benchmark_data_smithi177_31116_object81160:head [delete] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e193)", 2024-09-13T11:26:15.800 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:26:06.057961+0000", 2024-09-13T11:26:15.800 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 9.5813157419999992, 2024-09-13T11:26:15.800 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.24029746699999999, 2024-09-13T11:26:15.800 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:26:15.800 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:26:15.800 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:15.800 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:26:15.800 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:26:15.800 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:15.800 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 221463 2024-09-13T11:26:15.800 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.800 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:26:15.801 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.801 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:26:15.801 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:06.057961+0000", 2024-09-13T11:26:15.801 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:15.801 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.801 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.801 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:26:15.801 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:06.057961+0000", 2024-09-13T11:26:15.801 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:15.801 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.801 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.801 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:26:15.801 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:06.057963+0000", 2024-09-13T11:26:15.802 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.61e-06 2024-09-13T11:26:15.802 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.802 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.802 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:26:15.802 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:06.057970+0000", 2024-09-13T11:26:15.802 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.9999999999999999e-06 2024-09-13T11:26:15.802 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.802 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.802 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:26:15.802 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:06.058003+0000", 2024-09-13T11:26:15.802 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.2733e-05 2024-09-13T11:26:15.802 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.802 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.802 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:26:15.802 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:06.058013+0000", 2024-09-13T11:26:15.803 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0349999999999999e-05 2024-09-13T11:26:15.803 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.803 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.803 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:26:15.803 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:06.058333+0000", 2024-09-13T11:26:15.803 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000319947 2024-09-13T11:26:15.803 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.803 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.803 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for peered", 2024-09-13T11:26:15.803 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:06.058346+0000", 2024-09-13T11:26:15.803 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.3657e-05 2024-09-13T11:26:15.803 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.803 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.803 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:26:15.803 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:06.277695+0000", 2024-09-13T11:26:15.804 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.219348129 2024-09-13T11:26:15.804 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.804 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.804 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:26:15.804 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:06.277819+0000", 2024-09-13T11:26:15.804 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00012432599999999999 2024-09-13T11:26:15.804 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.804 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.804 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 3,14", 2024-09-13T11:26:15.804 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:06.277928+0000", 2024-09-13T11:26:15.804 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00010860499999999999 2024-09-13T11:26:15.804 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.804 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.804 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:15.804 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:06.286021+0000", 2024-09-13T11:26:15.805 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0080932020000000007 2024-09-13T11:26:15.805 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.805 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.805 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:26:15.805 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:06.291903+0000", 2024-09-13T11:26:15.805 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0058824630000000001 2024-09-13T11:26:15.805 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.805 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.805 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:15.805 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:06.298142+0000", 2024-09-13T11:26:15.805 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.006239301 2024-09-13T11:26:15.805 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.805 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.805 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:26:15.806 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:06.298189+0000", 2024-09-13T11:26:15.806 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.6115999999999997e-05 2024-09-13T11:26:15.806 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.806 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.806 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:26:15.806 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:06.298259+0000", 2024-09-13T11:26:15.806 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.0028000000000001e-05 2024-09-13T11:26:15.806 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:15.806 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:15.806 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:15.806 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.806 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.806 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:221475 3.15 3:abeb12df:::benchmark_data_smithi177_31116_object81172:head [delete] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e193)", 2024-09-13T11:26:15.806 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:26:06.058032+0000", 2024-09-13T11:26:15.806 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 9.5812447089999999, 2024-09-13T11:26:15.807 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.24037931400000001, 2024-09-13T11:26:15.807 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:26:15.807 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:26:15.807 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:15.807 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:26:15.807 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:26:15.807 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:15.807 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 221475 2024-09-13T11:26:15.807 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.807 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:26:15.807 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.807 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:26:15.807 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:06.058032+0000", 2024-09-13T11:26:15.807 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:15.807 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.808 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.808 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:26:15.808 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:06.058032+0000", 2024-09-13T11:26:15.808 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:15.808 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.808 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.808 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:26:15.808 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:06.058033+0000", 2024-09-13T11:26:15.808 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.136e-06 2024-09-13T11:26:15.808 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.808 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.808 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:26:15.808 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:06.058038+0000", 2024-09-13T11:26:15.808 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.9169999999999997e-06 2024-09-13T11:26:15.808 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.809 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.809 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:26:15.809 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:06.058048+0000", 2024-09-13T11:26:15.809 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.3170000000000008e-06 2024-09-13T11:26:15.809 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.809 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.809 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:26:15.809 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:06.058059+0000", 2024-09-13T11:26:15.809 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.1479e-05 2024-09-13T11:26:15.809 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.809 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.809 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:26:15.809 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:06.058444+0000", 2024-09-13T11:26:15.809 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00038485700000000001 2024-09-13T11:26:15.810 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.810 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.810 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for peered", 2024-09-13T11:26:15.810 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:06.058471+0000", 2024-09-13T11:26:15.810 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.7172999999999999e-05 2024-09-13T11:26:15.810 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.810 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.810 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:26:15.810 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:06.282046+0000", 2024-09-13T11:26:15.810 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.22357485099999999 2024-09-13T11:26:15.810 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.810 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.810 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:26:15.810 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:06.282204+0000", 2024-09-13T11:26:15.811 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000158414 2024-09-13T11:26:15.811 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.811 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.811 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 3,14", 2024-09-13T11:26:15.811 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:06.282377+0000", 2024-09-13T11:26:15.811 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00017289900000000001 2024-09-13T11:26:15.811 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.811 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.811 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:15.811 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:06.295990+0000", 2024-09-13T11:26:15.811 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.013612406000000001 2024-09-13T11:26:15.811 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.811 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.811 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:26:15.812 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:06.296380+0000", 2024-09-13T11:26:15.812 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00039020599999999998 2024-09-13T11:26:15.812 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.812 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.812 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:15.812 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:06.298303+0000", 2024-09-13T11:26:15.812 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.001923258 2024-09-13T11:26:15.812 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.812 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.812 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:26:15.812 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:06.298343+0000", 2024-09-13T11:26:15.812 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.9783999999999997e-05 2024-09-13T11:26:15.812 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.812 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.813 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:26:15.813 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:06.298412+0000", 2024-09-13T11:26:15.813 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.8616999999999997e-05 2024-09-13T11:26:15.813 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:15.813 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:15.813 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:15.813 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.813 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.813 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:221503 3.15 3:ab531595:::benchmark_data_smithi177_31116_object81200:head [delete] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e193)", 2024-09-13T11:26:15.813 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:26:06.058069+0000", 2024-09-13T11:26:15.813 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 9.5812082000000007, 2024-09-13T11:26:15.813 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.24966487400000001, 2024-09-13T11:26:15.813 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:26:15.813 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:26:15.814 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:15.814 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:26:15.814 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:26:15.814 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:15.814 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 221503 2024-09-13T11:26:15.814 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.814 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:26:15.814 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.814 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:26:15.814 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:06.058069+0000", 2024-09-13T11:26:15.814 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:15.814 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.814 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.814 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:26:15.815 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:06.058069+0000", 2024-09-13T11:26:15.815 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:15.815 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.815 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.815 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:26:15.815 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:06.058070+0000", 2024-09-13T11:26:15.815 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.7000000000000003e-07 2024-09-13T11:26:15.815 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.815 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.815 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:26:15.815 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:06.058074+0000", 2024-09-13T11:26:15.815 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.5070000000000002e-06 2024-09-13T11:26:15.815 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.816 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.816 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:26:15.816 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:06.058089+0000", 2024-09-13T11:26:15.816 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.4593000000000001e-05 2024-09-13T11:26:15.816 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.816 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.816 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:26:15.816 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:06.058102+0000", 2024-09-13T11:26:15.816 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.3349999999999999e-05 2024-09-13T11:26:15.816 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.816 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.816 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:26:15.816 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:06.058550+0000", 2024-09-13T11:26:15.816 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00044800299999999999 2024-09-13T11:26:15.816 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.817 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.817 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for peered", 2024-09-13T11:26:15.817 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:06.058560+0000", 2024-09-13T11:26:15.817 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.5200000000000003e-06 2024-09-13T11:26:15.817 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.817 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.817 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:26:15.817 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:06.286141+0000", 2024-09-13T11:26:15.817 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.22758104200000001 2024-09-13T11:26:15.817 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.817 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.817 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:26:15.817 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:06.286335+0000", 2024-09-13T11:26:15.817 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00019429700000000001 2024-09-13T11:26:15.818 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.818 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.818 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 3,14", 2024-09-13T11:26:15.818 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:06.286518+0000", 2024-09-13T11:26:15.818 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00018350500000000001 2024-09-13T11:26:15.818 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.818 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.818 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:15.818 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:06.296043+0000", 2024-09-13T11:26:15.818 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0095243600000000008 2024-09-13T11:26:15.818 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.818 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.818 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:26:15.819 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:06.296384+0000", 2024-09-13T11:26:15.819 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00034132100000000001 2024-09-13T11:26:15.819 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.819 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.819 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:15.819 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:06.307532+0000", 2024-09-13T11:26:15.819 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.011148025000000001 2024-09-13T11:26:15.819 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.819 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.819 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:26:15.819 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:06.307611+0000", 2024-09-13T11:26:15.819 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.8465000000000002e-05 2024-09-13T11:26:15.819 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.820 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.820 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:26:15.820 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:06.307734+0000", 2024-09-13T11:26:15.820 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00012301599999999999 2024-09-13T11:26:15.820 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:15.820 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:15.820 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:15.820 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.820 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.820 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:221528 3.15 3:afda25ed:::benchmark_data_smithi177_31116_object81225:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e193)", 2024-09-13T11:26:15.820 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:26:06.058152+0000", 2024-09-13T11:26:15.820 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 9.5811253430000001, 2024-09-13T11:26:15.820 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.25716789899999998, 2024-09-13T11:26:15.820 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:26:15.821 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:26:15.821 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:15.821 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:26:15.821 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:26:15.821 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:15.821 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 221528 2024-09-13T11:26:15.821 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.821 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:26:15.821 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.821 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:26:15.821 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:06.058152+0000", 2024-09-13T11:26:15.821 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:15.821 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.822 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.822 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:26:15.822 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:06.058152+0000", 2024-09-13T11:26:15.822 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:15.822 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.822 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.822 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:26:15.822 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:06.058153+0000", 2024-09-13T11:26:15.822 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.773e-06 2024-09-13T11:26:15.822 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.822 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.822 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:26:15.822 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:06.058161+0000", 2024-09-13T11:26:15.822 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.9160000000000001e-06 2024-09-13T11:26:15.823 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.823 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.823 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:26:15.823 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:06.058186+0000", 2024-09-13T11:26:15.823 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.5037000000000001e-05 2024-09-13T11:26:15.823 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.823 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.823 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:26:15.823 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:06.058202+0000", 2024-09-13T11:26:15.823 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.5709000000000002e-05 2024-09-13T11:26:15.823 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.823 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.823 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:26:15.823 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:06.058640+0000", 2024-09-13T11:26:15.824 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00043771400000000002 2024-09-13T11:26:15.824 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.824 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.824 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for peered", 2024-09-13T11:26:15.824 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:06.058664+0000", 2024-09-13T11:26:15.824 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.385e-05 2024-09-13T11:26:15.824 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.824 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.824 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:26:15.824 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:06.291953+0000", 2024-09-13T11:26:15.824 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.23328986800000001 2024-09-13T11:26:15.824 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.824 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.824 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:26:15.825 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:06.292123+0000", 2024-09-13T11:26:15.825 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00016977600000000001 2024-09-13T11:26:15.825 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.825 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.825 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 3,14", 2024-09-13T11:26:15.825 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:06.292276+0000", 2024-09-13T11:26:15.825 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000153196 2024-09-13T11:26:15.825 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.825 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.825 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:26:15.825 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:06.296387+0000", 2024-09-13T11:26:15.825 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0041105459999999996 2024-09-13T11:26:15.825 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.826 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.826 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:15.826 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:06.307779+0000", 2024-09-13T11:26:15.826 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.011391712 2024-09-13T11:26:15.826 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.826 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.826 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:15.826 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:06.315151+0000", 2024-09-13T11:26:15.826 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0073723169999999998 2024-09-13T11:26:15.826 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.826 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.826 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:26:15.826 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:06.315219+0000", 2024-09-13T11:26:15.827 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.7984999999999994e-05 2024-09-13T11:26:15.827 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:15.827 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:15.827 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:26:15.827 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:06.315319+0000", 2024-09-13T11:26:15.827 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0001005 2024-09-13T11:26:15.827 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:15.827 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:15.827 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:15.827 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:15.827 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:15.827 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-13T11:26:15.828 INFO:teuthology.orchestra.run.smithi001.stdout:osd.11: {} 2024-09-13T11:26:15.828 INFO:teuthology.orchestra.run.smithi001.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-13T11:26:15.829 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-13T11:26:15.829 INFO:teuthology.orchestra.run.smithi001.stdout:osd.12: {} 2024-09-13T11:26:15.830 INFO:teuthology.orchestra.run.smithi001.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-13T11:26:15.830 INFO:teuthology.orchestra.run.smithi001.stdout:osd.13: {} 2024-09-13T11:26:15.831 INFO:teuthology.orchestra.run.smithi001.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-13T11:26:15.831 INFO:tasks.ceph.osd.11.smithi177.stderr:2024-09-13T11:26:15.717+0000 7f24792c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26580) UID: 0 2024-09-13T11:26:15.832 INFO:tasks.ceph.osd.15.smithi177.stderr:2024-09-13T11:26:15.829+0000 7f9d6ce77640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26578) UID: 0 2024-09-13T11:26:15.832 INFO:teuthology.orchestra.run.smithi001.stdout:osd.14: {} 2024-09-13T11:26:15.832 INFO:teuthology.orchestra.run.smithi001.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-13T11:26:15.832 INFO:teuthology.orchestra.run.smithi001.stdout:osd.15: {} 2024-09-13T11:26:15.833 INFO:teuthology.orchestra.run.smithi001.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-13T11:26:15.932 INFO:tasks.ceph.osd.12.smithi001.stderr:2024-09-13T11:26:15.930+0000 7f892c6e6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26314) UID: 0 2024-09-13T11:26:16.018 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-13T11:26:16.018 INFO:teuthology.orchestra.run.smithi001.stdout: "ops": [ 2024-09-13T11:26:16.018 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.018 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:232903 3.1a 3:582ac9d4:::benchmark_data_smithi177_31116_object92600:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e200)", 2024-09-13T11:26:16.018 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:26:16.012836+0000", 2024-09-13T11:26:16.018 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 0.0051150240000000001, 2024-09-13T11:26:16.018 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0051410689999999998, 2024-09-13T11:26:16.018 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:26:16.019 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:26:16.019 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "waiting for sub ops", 2024-09-13T11:26:16.019 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:26:16.019 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:26:16.019 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:16.019 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 232903 2024-09-13T11:26:16.019 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.019 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:26:16.019 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.019 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:26:16.019 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:16.012836+0000", 2024-09-13T11:26:16.019 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:16.019 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.020 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.020 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:26:16.020 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:16.012836+0000", 2024-09-13T11:26:16.020 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:16.020 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.020 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.020 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:26:16.020 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:16.012839+0000", 2024-09-13T11:26:16.020 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.8490000000000002e-06 2024-09-13T11:26:16.020 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.020 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.020 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:26:16.020 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:16.012846+0000", 2024-09-13T11:26:16.020 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.4560000000000003e-06 2024-09-13T11:26:16.021 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.021 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.021 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:26:16.021 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:16.012864+0000", 2024-09-13T11:26:16.021 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.7554000000000002e-05 2024-09-13T11:26:16.021 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.021 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.021 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:26:16.021 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:16.012873+0000", 2024-09-13T11:26:16.021 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.6010000000000001e-06 2024-09-13T11:26:16.021 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.021 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.021 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:26:16.022 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:16.012927+0000", 2024-09-13T11:26:16.022 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.4199000000000001e-05 2024-09-13T11:26:16.022 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.022 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.022 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:26:16.022 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:16.013019+0000", 2024-09-13T11:26:16.022 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.1126999999999997e-05 2024-09-13T11:26:16.022 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.022 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.022 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 9", 2024-09-13T11:26:16.022 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:16.013103+0000", 2024-09-13T11:26:16.022 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.4289999999999994e-05 2024-09-13T11:26:16.022 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.023 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.023 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:26:16.023 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:16.016337+0000", 2024-09-13T11:26:16.023 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0032338390000000001 2024-09-13T11:26:16.023 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:16.023 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:16.023 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:16.023 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:16.023 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-09-13T11:26:16.023 INFO:teuthology.orchestra.run.smithi001.stdout: "num_ops": 1 2024-09-13T11:26:16.023 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-13T11:26:16.036 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-13T11:26:16.036 INFO:tasks.ceph.osd.1.smithi092.stderr:2024-09-13T11:26:16.032+0000 7fe7789c5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33718) UID: 0 2024-09-13T11:26:16.128 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-13T11:26:16.134 INFO:tasks.ceph.osd.2.smithi162.stderr:2024-09-13T11:26:16.131+0000 7fe92b35a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26313) UID: 0 2024-09-13T11:26:16.221 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-13T11:26:16.222 INFO:teuthology.orchestra.run.smithi001.stdout: "ops": [], 2024-09-13T11:26:16.222 INFO:teuthology.orchestra.run.smithi001.stdout: "complaint_time": 30, 2024-09-13T11:26:16.222 INFO:teuthology.orchestra.run.smithi001.stdout: "num_blocked_ops": 0 2024-09-13T11:26:16.222 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-13T11:26:16.235 INFO:tasks.ceph.osd.9.smithi092.stderr:2024-09-13T11:26:16.232+0000 7ff5ab3d0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26348) UID: 0 2024-09-13T11:26:16.238 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-13T11:26:16.265 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-13T11:26:16.265 INFO:teuthology.orchestra.run.smithi001.stderr:dumped all 2024-09-13T11:26:16.294 INFO:teuthology.orchestra.run.smithi001.stdout:{"pg_ready":true,"pg_map":{"version":309,"stamp":"2024-09-13T11:26:16.174384+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":3203334731,"num_objects":48875,"num_object_clones":0,"num_object_copies":97750,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":8874,"num_objects_unfound":0,"num_objects_dirty":48875,"num_whiteouts":0,"num_read":87,"num_read_kb":74,"num_write":141224,"num_write_kb":3356723,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19355,"num_bytes_recovered":1260912659,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":341323,"ondisk_log_size":341323,"up":110,"acting":110,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":124,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":16,"kb":1509949440,"kb_used":8649352,"kb_used_data":7171208,"kb_used_omap":68,"kb_used_meta":1477499,"kb_avail":1501300088,"statfs":{"total":1546188226560,"available":1537331290112,"internally_reserved":0,"allocated":7343316992,"data_stored":7329427998,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":70045,"internal_metadata":1512959587},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-525598720,"num_objects":-8020,"num_object_clones":0,"num_object_copies":-16040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-2844,"num_objects_unfound":0,"num_objects_dirty":-8020,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8020,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1877,"num_bytes_recovered":115671040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.081834"},"pg_stats":[{"pgid":"3.2d","version":"200'7061","reported_seq":12645,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-13T11:26:15.947981+0000","last_change":"2024-09-13T11:25:45.355259+0000","last_active":"2024-09-13T11:26:15.947981+0000","last_peered":"2024-09-13T11:26:15.947981+0000","last_clean":"2024-09-13T11:26:15.947981+0000","last_became_active":"2024-09-13T11:25:39.334408+0000","last_became_peered":"2024-09-13T11:25:39.334408+0000","last_unstale":"2024-09-13T11:26:15.947981+0000","last_undegraded":"2024-09-13T11:26:15.947981+0000","last_fullsized":"2024-09-13T11:26:15.947981+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":6,"last_scrub":"179'6515","last_scrub_stamp":"2024-09-13T11:25:45.355191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:45.355191+0000","objects_scrubbed":1320,"log_size":7061,"log_dups_size":0,"ondisk_log_size":7061,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:27:05.293671+0000","scrub_duration":199,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50462720,"num_objects":770,"num_object_clones":0,"num_object_copies":1540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":770,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":550,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"200'7093","reported_seq":12494,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-13T11:26:16.059105+0000","last_change":"2024-09-13T11:24:59.456762+0000","last_active":"2024-09-13T11:26:16.059105+0000","last_peered":"2024-09-13T11:26:16.059105+0000","last_clean":"2024-09-13T11:26:16.059105+0000","last_became_active":"2024-09-13T11:24:59.021895+0000","last_became_peered":"2024-09-13T11:24:59.021895+0000","last_unstale":"2024-09-13T11:26:16.059105+0000","last_undegraded":"2024-09-13T11:26:16.059105+0000","last_fullsized":"2024-09-13T11:26:16.059105+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":6,"last_scrub":"145'5672","last_scrub_stamp":"2024-09-13T11:24:59.456653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:24:59.456653+0000","objects_scrubbed":2192,"log_size":7093,"log_dups_size":0,"ondisk_log_size":7093,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:26:22.622756+0000","scrub_duration":279,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50528256,"num_objects":771,"num_object_clones":0,"num_object_copies":1542,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":771,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1421,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,7],"acting":[10,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.2b","version":"200'6954","reported_seq":10729,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-13T11:26:15.263640+0000","last_change":"2024-09-13T11:26:01.126562+0000","last_active":"2024-09-13T11:26:15.263640+0000","last_peered":"2024-09-13T11:26:15.263640+0000","last_clean":"2024-09-13T11:26:15.263640+0000","last_became_active":"2024-09-13T11:24:26.180578+0000","last_became_peered":"2024-09-13T11:24:26.180578+0000","last_unstale":"2024-09-13T11:26:15.263640+0000","last_undegraded":"2024-09-13T11:26:15.263640+0000","last_fullsized":"2024-09-13T11:26:15.263640+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":6,"last_scrub":"188'6745","last_scrub_stamp":"2024-09-13T11:26:01.126423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:26:01.126423+0000","objects_scrubbed":916,"log_size":6954,"log_dups_size":0,"ondisk_log_size":6954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:27:08.681933+0000","scrub_duration":391,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45875200,"num_objects":700,"num_object_clones":0,"num_object_copies":1400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":700,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1572,"num_write_kb":10624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2a","version":"200'7138","reported_seq":11099,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-13T11:26:16.042004+0000","last_change":"2024-09-13T11:26:09.477886+0000","last_active":"2024-09-13T11:26:16.042004+0000","last_peered":"2024-09-13T11:26:16.042004+0000","last_clean":"2024-09-13T11:26:16.042004+0000","last_became_active":"2024-09-13T11:24:45.320540+0000","last_became_peered":"2024-09-13T11:24:45.320540+0000","last_unstale":"2024-09-13T11:26:16.042004+0000","last_undegraded":"2024-09-13T11:26:16.042004+0000","last_fullsized":"2024-09-13T11:26:16.042004+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":6,"last_scrub":"195'7004","last_scrub_stamp":"2024-09-13T11:26:09.477793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:26:09.477793+0000","objects_scrubbed":929,"log_size":7138,"log_dups_size":0,"ondisk_log_size":7138,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:27:33.724690+0000","scrub_duration":115,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52101120,"num_objects":795,"num_object_clones":0,"num_object_copies":1590,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":795,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1649,"num_write_kb":11200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,7],"acting":[10,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.29","version":"200'6801","reported_seq":12561,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-13T11:26:15.911113+0000","last_change":"2024-09-13T11:25:13.333062+0000","last_active":"2024-09-13T11:26:15.911113+0000","last_peered":"2024-09-13T11:26:15.911113+0000","last_clean":"2024-09-13T11:26:15.911113+0000","last_became_active":"2024-09-13T11:25:12.944028+0000","last_became_peered":"2024-09-13T11:25:12.944028+0000","last_unstale":"2024-09-13T11:26:15.911113+0000","last_undegraded":"2024-09-13T11:26:15.911113+0000","last_fullsized":"2024-09-13T11:26:15.911113+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":6,"last_scrub":"154'5579","last_scrub_stamp":"2024-09-13T11:25:13.332959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:13.332959+0000","objects_scrubbed":1937,"log_size":6801,"log_dups_size":0,"ondisk_log_size":6801,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:26:23.751009+0000","scrub_duration":243,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46727168,"num_objects":713,"num_object_clones":0,"num_object_copies":1426,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":713,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1224,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.28","version":"200'6609","reported_seq":12097,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-13T11:26:16.087204+0000","last_change":"2024-09-13T11:25:36.143261+0000","last_active":"2024-09-13T11:26:16.087204+0000","last_peered":"2024-09-13T11:26:16.087204+0000","last_clean":"2024-09-13T11:26:16.087204+0000","last_became_active":"2024-09-13T11:25:34.278640+0000","last_became_peered":"2024-09-13T11:25:34.278640+0000","last_unstale":"2024-09-13T11:26:16.087204+0000","last_undegraded":"2024-09-13T11:26:16.087204+0000","last_fullsized":"2024-09-13T11:26:16.087204+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":6,"last_scrub":"174'5936","last_scrub_stamp":"2024-09-13T11:25:36.143113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:36.143113+0000","objects_scrubbed":1447,"log_size":6609,"log_dups_size":0,"ondisk_log_size":6609,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:26:43.135020+0000","scrub_duration":1147,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49479680,"num_objects":755,"num_object_clones":0,"num_object_copies":1510,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":755,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":692,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5],"acting":[8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.27","version":"200'6701","reported_seq":12307,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-13T11:26:16.041405+0000","last_change":"2024-09-13T11:25:37.344464+0000","last_active":"2024-09-13T11:26:16.041405+0000","last_peered":"2024-09-13T11:26:16.041405+0000","last_clean":"2024-09-13T11:26:16.041405+0000","last_became_active":"2024-09-13T11:25:30.257911+0000","last_became_peered":"2024-09-13T11:25:30.257911+0000","last_unstale":"2024-09-13T11:26:16.041405+0000","last_undegraded":"2024-09-13T11:26:16.041405+0000","last_fullsized":"2024-09-13T11:26:16.041405+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":6,"last_scrub":"174'6026","last_scrub_stamp":"2024-09-13T11:25:37.344335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:37.344335+0000","objects_scrubbed":1463,"log_size":6701,"log_dups_size":0,"ondisk_log_size":6701,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:27:06.776518+0000","scrub_duration":195,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51576832,"num_objects":787,"num_object_clones":0,"num_object_copies":1574,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":787,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":676,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,11],"acting":[5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.26","version":"200'6833","reported_seq":12489,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-13T11:26:16.097216+0000","last_change":"2024-09-13T11:26:11.113925+0000","last_active":"2024-09-13T11:26:16.097216+0000","last_peered":"2024-09-13T11:26:16.097216+0000","last_clean":"2024-09-13T11:26:16.097216+0000","last_became_active":"2024-09-13T11:24:54.976919+0000","last_became_peered":"2024-09-13T11:24:54.976919+0000","last_unstale":"2024-09-13T11:26:16.097216+0000","last_undegraded":"2024-09-13T11:26:16.097216+0000","last_fullsized":"2024-09-13T11:26:16.097216+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":6,"last_scrub":"196'6743","last_scrub_stamp":"2024-09-13T11:26:11.113811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:26:11.113811+0000","objects_scrubbed":840,"log_size":6833,"log_dups_size":0,"ondisk_log_size":6833,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:27:35.746195+0000","scrub_duration":103,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49152000,"num_objects":750,"num_object_clones":0,"num_object_copies":1500,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":750,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1408,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,12],"acting":[8,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.25","version":"200'6727","reported_seq":12526,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-13T11:26:16.075504+0000","last_change":"2024-09-13T11:24:57.073162+0000","last_active":"2024-09-13T11:26:16.075504+0000","last_peered":"2024-09-13T11:26:16.075504+0000","last_clean":"2024-09-13T11:26:16.075504+0000","last_became_active":"2024-09-13T11:24:55.999019+0000","last_became_peered":"2024-09-13T11:24:55.999019+0000","last_unstale":"2024-09-13T11:26:16.075504+0000","last_undegraded":"2024-09-13T11:26:16.075504+0000","last_fullsized":"2024-09-13T11:26:16.075504+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":6,"last_scrub":"142'5214","last_scrub_stamp":"2024-09-13T11:24:57.073061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:24:57.073061+0000","objects_scrubbed":2249,"log_size":6727,"log_dups_size":0,"ondisk_log_size":6727,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":571,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48431104,"num_objects":739,"num_object_clones":0,"num_object_copies":1478,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":739,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1516,"num_write_kb":192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,2],"acting":[12,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[]},{"pgid":"3.24","version":"198'6663","reported_seq":11727,"reported_epoch":198,"state":"active+recovering","last_fresh":"2024-09-13T11:26:12.036993+0000","last_change":"2024-09-13T11:26:05.265265+0000","last_active":"2024-09-13T11:26:12.036993+0000","last_peered":"2024-09-13T11:26:12.036993+0000","last_clean":"2024-09-13T11:26:03.399501+0000","last_became_active":"2024-09-13T11:26:05.240552+0000","last_became_peered":"2024-09-13T11:26:05.240552+0000","last_unstale":"2024-09-13T11:26:12.036993+0000","last_undegraded":"2024-09-13T11:26:12.036993+0000","last_fullsized":"2024-09-13T11:26:12.036993+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":6,"last_scrub":"188'6512","last_scrub_stamp":"2024-09-13T11:26:01.118438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:26:01.118438+0000","objects_scrubbed":972,"log_size":6663,"log_dups_size":0,"ondisk_log_size":6663,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":183,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53805056,"num_objects":821,"num_object_clones":0,"num_object_copies":1642,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":399,"num_objects_unfound":0,"num_objects_dirty":821,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1571,"num_write_kb":9280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":532,"num_bytes_recovered":34865152,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,14],"acting":[9,14],"avail_no_missing":["9","1"],"object_location_counts":[{"shards":"1,9","objects":399},{"shards":"9,14","objects":422}],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.23","version":"200'5980","reported_seq":8933,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-13T11:26:15.880024+0000","last_change":"2024-09-13T11:25:56.586649+0000","last_active":"2024-09-13T11:26:15.880024+0000","last_peered":"2024-09-13T11:26:15.880024+0000","last_clean":"2024-09-13T11:26:15.880024+0000","last_became_active":"2024-09-13T11:23:47.926677+0000","last_became_peered":"2024-09-13T11:23:47.926677+0000","last_unstale":"2024-09-13T11:26:15.880024+0000","last_undegraded":"2024-09-13T11:26:15.880024+0000","last_fullsized":"2024-09-13T11:26:15.880024+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":6,"last_scrub":"186'5649","last_scrub_stamp":"2024-09-13T11:25:56.586425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:56.586425+0000","objects_scrubbed":1098,"log_size":5980,"log_dups_size":0,"ondisk_log_size":5980,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:27:16.538921+0000","scrub_duration":135,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50200576,"num_objects":766,"num_object_clones":0,"num_object_copies":1532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":766,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2703,"num_write_kb":78080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9],"acting":[5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.22","version":"200'5924","reported_seq":13676,"reported_epoch":200,"state":"active+recovering","last_fresh":"2024-09-13T11:26:16.123429+0000","last_change":"2024-09-13T11:26:05.254491+0000","last_active":"2024-09-13T11:26:16.123429+0000","last_peered":"2024-09-13T11:26:16.123429+0000","last_clean":"2024-09-13T11:26:03.171705+0000","last_became_active":"2024-09-13T11:26:05.226450+0000","last_became_peered":"2024-09-13T11:26:05.226450+0000","last_unstale":"2024-09-13T11:26:16.123429+0000","last_undegraded":"2024-09-13T11:26:16.123429+0000","last_fullsized":"2024-09-13T11:26:16.123429+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":6,"last_scrub":"186'5611","last_scrub_stamp":"2024-09-13T11:25:57.123569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:57.123569+0000","objects_scrubbed":1032,"log_size":5924,"log_dups_size":0,"ondisk_log_size":5924,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":147,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47120384,"num_objects":719,"num_object_clones":0,"num_object_copies":1438,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":719,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2680,"num_write_kb":75712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2808,"num_bytes_recovered":176685056,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,14],"acting":[0,14],"avail_no_missing":["0","5"],"object_location_counts":[{"shards":"0,14","objects":719}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.21","version":"200'5768","reported_seq":8723,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-13T11:26:16.105786+0000","last_change":"2024-09-13T11:25:49.226484+0000","last_active":"2024-09-13T11:26:16.105786+0000","last_peered":"2024-09-13T11:26:16.105786+0000","last_clean":"2024-09-13T11:26:16.105786+0000","last_became_active":"2024-09-13T11:24:01.250308+0000","last_became_peered":"2024-09-13T11:24:01.250308+0000","last_unstale":"2024-09-13T11:26:16.105786+0000","last_undegraded":"2024-09-13T11:26:16.105786+0000","last_fullsized":"2024-09-13T11:26:16.105786+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":6,"last_scrub":"182'5306","last_scrub_stamp":"2024-09-13T11:25:49.226272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:49.226272+0000","objects_scrubbed":1176,"log_size":5768,"log_dups_size":0,"ondisk_log_size":5768,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:27:04.742252+0000","scrub_duration":151,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46661632,"num_objects":712,"num_object_clones":0,"num_object_copies":1424,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":712,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2101,"num_write_kb":44224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8],"acting":[0,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.20","version":"200'5952","reported_seq":8197,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-13T11:26:16.087922+0000","last_change":"2024-09-13T11:25:35.773483+0000","last_active":"2024-09-13T11:26:16.087922+0000","last_peered":"2024-09-13T11:26:16.087922+0000","last_clean":"2024-09-13T11:26:16.087922+0000","last_became_active":"2024-09-13T11:23:33.954946+0000","last_became_peered":"2024-09-13T11:23:33.954946+0000","last_unstale":"2024-09-13T11:26:16.087922+0000","last_undegraded":"2024-09-13T11:26:16.087922+0000","last_fullsized":"2024-09-13T11:26:16.087922+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":6,"last_scrub":"174'5240","last_scrub_stamp":"2024-09-13T11:25:35.773409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:35.773409+0000","objects_scrubbed":1424,"log_size":5952,"log_dups_size":0,"ondisk_log_size":5952,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:26:59.046210+0000","scrub_duration":215,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46530560,"num_objects":710,"num_object_clones":0,"num_object_copies":1420,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":710,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2687,"num_write_kb":74752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,0],"acting":[12,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[]},{"pgid":"3.1f","version":"200'8829","reported_seq":20854,"reported_epoch":200,"state":"active+remapped+backfill_toofull","last_fresh":"2024-09-13T11:26:16.134124+0000","last_change":"2024-09-13T11:26:06.223779+0000","last_active":"2024-09-13T11:26:16.134124+0000","last_peered":"2024-09-13T11:26:16.134124+0000","last_clean":"2024-09-13T11:26:03.341211+0000","last_became_active":"2024-09-13T11:26:06.196771+0000","last_became_peered":"2024-09-13T11:26:06.196771+0000","last_unstale":"2024-09-13T11:26:16.134124+0000","last_undegraded":"2024-09-13T11:26:16.134124+0000","last_fullsized":"2024-09-13T11:26:16.134124+0000","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":5,"last_scrub":"161'6865","last_scrub_stamp":"2024-09-13T11:25:20.773641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:20.773641+0000","objects_scrubbed":3425,"log_size":8829,"log_dups_size":0,"ondisk_log_size":8829,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":535,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":95420416,"num_objects":1456,"num_object_clones":0,"num_object_copies":2912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1456,"num_objects_unfound":0,"num_objects_dirty":1456,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4409,"num_write_kb":102848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3306,"num_bytes_recovered":216662016,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,14],"acting":[10,5],"avail_no_missing":["10","5"],"object_location_counts":[{"shards":"5,10","objects":1456}],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1e","version":"200'9117","reported_seq":19704,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-13T11:26:16.017273+0000","last_change":"2024-09-13T11:24:08.338485+0000","last_active":"2024-09-13T11:26:16.017273+0000","last_peered":"2024-09-13T11:26:16.017273+0000","last_clean":"2024-09-13T11:26:16.017273+0000","last_became_active":"2024-09-13T11:24:01.261448+0000","last_became_peered":"2024-09-13T11:24:01.261448+0000","last_unstale":"2024-09-13T11:26:16.017273+0000","last_undegraded":"2024-09-13T11:26:16.017273+0000","last_fullsized":"2024-09-13T11:26:16.017273+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":5,"last_scrub":"110'4537","last_scrub_stamp":"2024-09-13T11:24:08.338354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:24:08.338354+0000","objects_scrubbed":2760,"log_size":9117,"log_dups_size":0,"ondisk_log_size":9117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":367,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":97714176,"num_objects":1491,"num_object_clones":0,"num_object_copies":2982,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1491,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4589,"num_write_kb":106240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"200'8891","reported_seq":14441,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-13T11:26:16.007274+0000","last_change":"2024-09-13T11:25:46.627072+0000","last_active":"2024-09-13T11:26:16.007274+0000","last_peered":"2024-09-13T11:26:16.007274+0000","last_clean":"2024-09-13T11:26:16.007274+0000","last_became_active":"2024-09-13T11:24:01.258225+0000","last_became_peered":"2024-09-13T11:24:01.258225+0000","last_unstale":"2024-09-13T11:26:16.007274+0000","last_undegraded":"2024-09-13T11:26:16.007274+0000","last_fullsized":"2024-09-13T11:26:16.007274+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":5,"last_scrub":"180'7891","last_scrub_stamp":"2024-09-13T11:25:46.626952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:46.626952+0000","objects_scrubbed":2517,"log_size":8891,"log_dups_size":0,"ondisk_log_size":8891,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:26:47.953682+0000","scrub_duration":875,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":98762776,"num_objects":1508,"num_object_clones":0,"num_object_copies":3016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1508,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4325,"num_write_kb":94593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,15],"acting":[2,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1c","version":"198'8935","reported_seq":12837,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-13T11:26:12.091765+0000","last_change":"2024-09-13T11:25:46.456795+0000","last_active":"2024-09-13T11:26:12.091765+0000","last_peered":"2024-09-13T11:26:12.091765+0000","last_clean":"2024-09-13T11:26:12.091765+0000","last_became_active":"2024-09-13T11:23:33.989492+0000","last_became_peered":"2024-09-13T11:23:33.989492+0000","last_unstale":"2024-09-13T11:26:12.091765+0000","last_undegraded":"2024-09-13T11:26:12.091765+0000","last_fullsized":"2024-09-13T11:26:12.091765+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":5,"last_scrub":"179'8037","last_scrub_stamp":"2024-09-13T11:25:46.456644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:46.456644+0000","objects_scrubbed":2528,"log_size":8935,"log_dups_size":0,"ondisk_log_size":8935,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:26:47.370789+0000","scrub_duration":451,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":106430464,"num_objects":1624,"num_object_clones":0,"num_object_copies":3248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1624,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4660,"num_write_kb":119680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7],"acting":[11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1b","version":"200'9042","reported_seq":14767,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-13T11:26:16.073392+0000","last_change":"2024-09-13T11:25:49.890080+0000","last_active":"2024-09-13T11:26:16.073392+0000","last_peered":"2024-09-13T11:26:16.073392+0000","last_clean":"2024-09-13T11:26:16.073392+0000","last_became_active":"2024-09-13T11:24:04.950821+0000","last_became_peered":"2024-09-13T11:24:04.950821+0000","last_unstale":"2024-09-13T11:26:16.073392+0000","last_undegraded":"2024-09-13T11:26:16.073392+0000","last_fullsized":"2024-09-13T11:26:16.073392+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":5,"last_scrub":"182'8139","last_scrub_stamp":"2024-09-13T11:25:49.889914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:49.889914+0000","objects_scrubbed":2406,"log_size":9042,"log_dups_size":0,"ondisk_log_size":9042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:27:03.098951+0000","scrub_duration":519,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":97910784,"num_objects":1494,"num_object_clones":0,"num_object_copies":2988,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1494,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4398,"num_write_kb":94656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,15],"acting":[12,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[]},{"pgid":"3.1a","version":"200'8822","reported_seq":14212,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-13T11:26:16.038437+0000","last_change":"2024-09-13T11:25:46.321420+0000","last_active":"2024-09-13T11:26:16.038437+0000","last_peered":"2024-09-13T11:26:16.038437+0000","last_clean":"2024-09-13T11:26:16.038437+0000","last_became_active":"2024-09-13T11:24:03.935185+0000","last_became_peered":"2024-09-13T11:24:03.935185+0000","last_unstale":"2024-09-13T11:26:16.038437+0000","last_undegraded":"2024-09-13T11:26:16.038437+0000","last_fullsized":"2024-09-13T11:26:16.038437+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":5,"last_scrub":"179'7826","last_scrub_stamp":"2024-09-13T11:25:46.321284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:46.321284+0000","objects_scrubbed":2526,"log_size":8822,"log_dups_size":0,"ondisk_log_size":8822,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:27:04.297171+0000","scrub_duration":935,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":99483648,"num_objects":1518,"num_object_clones":0,"num_object_copies":3036,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1518,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4400,"num_write_kb":104448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9],"acting":[4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.19","version":"198'8357","reported_seq":12896,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-13T11:26:11.981502+0000","last_change":"2024-09-13T11:25:10.784789+0000","last_active":"2024-09-13T11:26:11.981502+0000","last_peered":"2024-09-13T11:26:11.981502+0000","last_clean":"2024-09-13T11:26:11.981502+0000","last_became_active":"2024-09-13T11:23:41.924941+0000","last_became_peered":"2024-09-13T11:23:41.924941+0000","last_unstale":"2024-09-13T11:26:11.981502+0000","last_undegraded":"2024-09-13T11:26:11.981502+0000","last_fullsized":"2024-09-13T11:26:11.981502+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":5,"last_scrub":"151'6156","last_scrub_stamp":"2024-09-13T11:25:10.784716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:10.784716+0000","objects_scrubbed":3781,"log_size":8357,"log_dups_size":0,"ondisk_log_size":8357,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:26:22.643281+0000","scrub_duration":1131,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":102367232,"num_objects":1562,"num_object_clones":0,"num_object_copies":3124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1562,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4955,"num_write_kb":145216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1],"acting":[9,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.18","version":"200'8669","reported_seq":13510,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-13T11:26:15.930504+0000","last_change":"2024-09-13T11:25:37.928304+0000","last_active":"2024-09-13T11:26:15.930504+0000","last_peered":"2024-09-13T11:26:15.930504+0000","last_clean":"2024-09-13T11:26:15.930504+0000","last_became_active":"2024-09-13T11:23:48.926014+0000","last_became_peered":"2024-09-13T11:23:48.926014+0000","last_unstale":"2024-09-13T11:26:15.930504+0000","last_undegraded":"2024-09-13T11:26:15.930504+0000","last_fullsized":"2024-09-13T11:26:15.930504+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":5,"last_scrub":"174'7306","last_scrub_stamp":"2024-09-13T11:25:37.928052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:37.928052+0000","objects_scrubbed":2815,"log_size":8669,"log_dups_size":0,"ondisk_log_size":8669,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:27:06.240226+0000","scrub_duration":631,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":94240768,"num_objects":1438,"num_object_clones":0,"num_object_copies":2876,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1438,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4302,"num_write_kb":86528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0],"acting":[10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.17","version":"200'8454","reported_seq":12730,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-13T11:26:16.070218+0000","last_change":"2024-09-13T11:25:12.658562+0000","last_active":"2024-09-13T11:26:16.070218+0000","last_peered":"2024-09-13T11:26:16.070218+0000","last_clean":"2024-09-13T11:26:16.070218+0000","last_became_active":"2024-09-13T11:23:42.935601+0000","last_became_peered":"2024-09-13T11:23:42.935601+0000","last_unstale":"2024-09-13T11:26:16.070218+0000","last_undegraded":"2024-09-13T11:26:16.070218+0000","last_fullsized":"2024-09-13T11:26:16.070218+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":5,"last_scrub":"153'6197","last_scrub_stamp":"2024-09-13T11:25:12.658464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:12.658464+0000","objects_scrubbed":3743,"log_size":8454,"log_dups_size":0,"ondisk_log_size":8454,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:26:17.463516+0000","scrub_duration":467,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":97320960,"num_objects":1485,"num_object_clones":0,"num_object_copies":2970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1485,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5103,"num_write_kb":146688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3],"acting":[10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"198'8794","reported_seq":13702,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-13T11:26:12.082138+0000","last_change":"2024-09-13T11:25:01.903487+0000","last_active":"2024-09-13T11:26:12.082138+0000","last_peered":"2024-09-13T11:26:12.082138+0000","last_clean":"2024-09-13T11:26:12.082138+0000","last_became_active":"2024-09-13T11:23:46.105237+0000","last_became_peered":"2024-09-13T11:23:46.105237+0000","last_unstale":"2024-09-13T11:26:12.082138+0000","last_undegraded":"2024-09-13T11:26:12.082138+0000","last_fullsized":"2024-09-13T11:26:12.082138+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":5,"last_scrub":"147'6035","last_scrub_stamp":"2024-09-13T11:25:01.903395+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:01.903395+0000","objects_scrubbed":4359,"log_size":8794,"log_dups_size":0,"ondisk_log_size":8794,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:26:19.652551+0000","scrub_duration":743,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":103874560,"num_objects":1585,"num_object_clones":0,"num_object_copies":3170,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1585,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5319,"num_write_kb":155200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,15],"acting":[11,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.15","version":"200'8509","reported_seq":19937,"reported_epoch":200,"state":"active+remapped+backfill_wait","last_fresh":"2024-09-13T11:26:16.099956+0000","last_change":"2024-09-13T11:26:06.276983+0000","last_active":"2024-09-13T11:26:16.099956+0000","last_peered":"2024-09-13T11:26:16.099956+0000","last_clean":"2024-09-13T11:26:03.255044+0000","last_became_active":"2024-09-13T11:26:06.250801+0000","last_became_peered":"2024-09-13T11:26:06.250801+0000","last_unstale":"2024-09-13T11:26:16.099956+0000","last_undegraded":"2024-09-13T11:26:16.099956+0000","last_fullsized":"2024-09-13T11:26:16.099956+0000","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":5,"last_scrub":"181'7535","last_scrub_stamp":"2024-09-13T11:25:48.075568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:48.075568+0000","objects_scrubbed":2413,"log_size":8509,"log_dups_size":0,"ondisk_log_size":8509,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":991,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92798976,"num_objects":1416,"num_object_clones":0,"num_object_copies":2832,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1416,"num_objects_unfound":0,"num_objects_dirty":1416,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5126,"num_write_kb":144320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3385,"num_bytes_recovered":221839360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,0],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,3","objects":1416}],"blocked_by":[],"up_primary":14,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.14","version":"200'8671","reported_seq":20032,"reported_epoch":200,"state":"active+remapped+backfill_wait","last_fresh":"2024-09-13T11:26:15.116698+0000","last_change":"2024-09-13T11:26:06.325969+0000","last_active":"2024-09-13T11:26:15.116698+0000","last_peered":"2024-09-13T11:26:15.116698+0000","last_clean":"2024-09-13T11:26:03.284559+0000","last_became_active":"2024-09-13T11:26:06.252189+0000","last_became_peered":"2024-09-13T11:26:06.252189+0000","last_unstale":"2024-09-13T11:26:15.116698+0000","last_undegraded":"2024-09-13T11:26:15.116698+0000","last_fullsized":"2024-09-13T11:26:15.116698+0000","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":5,"last_scrub":"174'7321","last_scrub_stamp":"2024-09-13T11:25:38.085667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:38.085667+0000","objects_scrubbed":2938,"log_size":8671,"log_dups_size":0,"ondisk_log_size":8671,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":431,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":103612416,"num_objects":1581,"num_object_clones":0,"num_object_copies":3162,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1581,"num_objects_unfound":0,"num_objects_dirty":1581,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5310,"num_write_kb":158592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3437,"num_bytes_recovered":225247232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,14],"acting":[3,15],"avail_no_missing":["3","15"],"object_location_counts":[{"shards":"3,15","objects":1581}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":380,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-13T11:26:15.124541+0000","last_change":"2024-09-13T11:25:46.222900+0000","last_active":"2024-09-13T11:26:15.124541+0000","last_peered":"2024-09-13T11:26:15.124541+0000","last_clean":"2024-09-13T11:26:15.124541+0000","last_became_active":"2024-09-13T11:22:14.192897+0000","last_became_peered":"2024-09-13T11:22:14.192897+0000","last_unstale":"2024-09-13T11:26:15.124541+0000","last_undegraded":"2024-09-13T11:26:15.124541+0000","last_fullsized":"2024-09-13T11:26:15.124541+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:25:46.222806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:25:46.222806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:27:02.988562+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00084655899999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,10],"acting":[15,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"3.6","version":"198'6766","reported_seq":12089,"reported_epoch":198,"state":"active+recovering","last_fresh":"2024-09-13T11:26:12.083326+0000","last_change":"2024-09-13T11:26:05.828206+0000","last_active":"2024-09-13T11:26:12.083326+0000","last_peered":"2024-09-13T11:26:12.083326+0000","last_clean":"2024-09-13T11:26:03.120046+0000","last_became_active":"2024-09-13T11:26:05.246522+0000","last_became_peered":"2024-09-13T11:26:05.246522+0000","last_unstale":"2024-09-13T11:26:12.083326+0000","last_undegraded":"2024-09-13T11:26:12.083326+0000","last_fullsized":"2024-09-13T11:26:12.083326+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"136'5216","last_scrub_stamp":"2024-09-13T11:24:49.416706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:24:49.416706+0000","objects_scrubbed":2083,"log_size":6766,"log_dups_size":0,"ondisk_log_size":6766,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":247,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":54132736,"num_objects":826,"num_object_clones":0,"num_object_copies":1652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":247,"num_objects_unfound":0,"num_objects_dirty":826,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1551,"num_write_kb":9408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":696,"num_bytes_recovered":45613056,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,14],"acting":[13,14],"avail_no_missing":["13","6"],"object_location_counts":[{"shards":"6,13","objects":247},{"shards":"13,14","objects":579}],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":375,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-13T11:26:11.960213+0000","last_change":"2024-09-13T11:24:59.556253+0000","last_active":"2024-09-13T11:26:11.960213+0000","last_peered":"2024-09-13T11:26:11.960213+0000","last_clean":"2024-09-13T11:26:11.960213+0000","last_became_active":"2024-09-13T11:22:14.211923+0000","last_became_peered":"2024-09-13T11:22:14.211923+0000","last_unstale":"2024-09-13T11:26:11.960213+0000","last_undegraded":"2024-09-13T11:26:11.960213+0000","last_fullsized":"2024-09-13T11:26:11.960213+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:24:59.556180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:24:59.556180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011689529999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0],"acting":[9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"198'6562","reported_seq":7002,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-13T11:26:12.025634+0000","last_change":"2024-09-13T11:25:59.279849+0000","last_active":"2024-09-13T11:26:12.025634+0000","last_peered":"2024-09-13T11:26:12.025634+0000","last_clean":"2024-09-13T11:26:12.025634+0000","last_became_active":"2024-09-13T11:24:24.092975+0000","last_became_peered":"2024-09-13T11:24:24.092975+0000","last_unstale":"2024-09-13T11:26:12.025634+0000","last_undegraded":"2024-09-13T11:26:12.025634+0000","last_fullsized":"2024-09-13T11:26:12.025634+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"187'6375","last_scrub_stamp":"2024-09-13T11:25:59.279677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:59.279677+0000","objects_scrubbed":1033,"log_size":6562,"log_dups_size":0,"ondisk_log_size":6562,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:27:27.660554+0000","scrub_duration":139,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":55443456,"num_objects":846,"num_object_clones":0,"num_object_copies":1692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":846,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1744,"num_write_kb":26368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,8],"acting":[13,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":380,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-13T11:26:15.148050+0000","last_change":"2024-09-13T11:25:33.957283+0000","last_active":"2024-09-13T11:26:15.148050+0000","last_peered":"2024-09-13T11:26:15.148050+0000","last_clean":"2024-09-13T11:26:15.148050+0000","last_became_active":"2024-09-13T11:22:14.581458+0000","last_became_peered":"2024-09-13T11:22:14.581458+0000","last_unstale":"2024-09-13T11:26:15.148050+0000","last_undegraded":"2024-09-13T11:26:15.148050+0000","last_fullsized":"2024-09-13T11:26:15.148050+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:25:33.957179+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:25:33.957179+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:26:52.126360+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.001026047,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5],"acting":[8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"200'6670","reported_seq":7088,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-13T11:26:15.350585+0000","last_change":"2024-09-13T11:25:48.001902+0000","last_active":"2024-09-13T11:26:15.350585+0000","last_peered":"2024-09-13T11:26:15.350585+0000","last_clean":"2024-09-13T11:26:15.350585+0000","last_became_active":"2024-09-13T11:24:24.083077+0000","last_became_peered":"2024-09-13T11:24:24.083077+0000","last_unstale":"2024-09-13T11:26:15.350585+0000","last_undegraded":"2024-09-13T11:26:15.350585+0000","last_fullsized":"2024-09-13T11:26:15.350585+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"181'6222","last_scrub_stamp":"2024-09-13T11:25:48.001778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:48.001778+0000","objects_scrubbed":1224,"log_size":6670,"log_dups_size":0,"ondisk_log_size":6670,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:27:16.908122+0000","scrub_duration":171,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50855936,"num_objects":776,"num_object_clones":0,"num_object_copies":1552,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":776,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1936,"num_write_kb":32704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,12],"acting":[1,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":368,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-13T11:26:15.127347+0000","last_change":"2024-09-13T11:25:45.851854+0000","last_active":"2024-09-13T11:26:15.127347+0000","last_peered":"2024-09-13T11:26:15.127347+0000","last_clean":"2024-09-13T11:26:15.127347+0000","last_became_active":"2024-09-13T11:22:47.719636+0000","last_became_peered":"2024-09-13T11:22:47.719636+0000","last_unstale":"2024-09-13T11:26:15.127347+0000","last_undegraded":"2024-09-13T11:26:15.127347+0000","last_fullsized":"2024-09-13T11:26:15.127347+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:25:45.851784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:25:45.851784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:26:50.817831+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00087476000000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,9],"acting":[1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"200'6618","reported_seq":7055,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-13T11:26:16.069874+0000","last_change":"2024-09-13T11:25:49.468719+0000","last_active":"2024-09-13T11:26:16.069874+0000","last_peered":"2024-09-13T11:26:16.069874+0000","last_clean":"2024-09-13T11:26:16.069874+0000","last_became_active":"2024-09-13T11:24:24.093492+0000","last_became_peered":"2024-09-13T11:24:24.093492+0000","last_unstale":"2024-09-13T11:26:16.069874+0000","last_undegraded":"2024-09-13T11:26:16.069874+0000","last_fullsized":"2024-09-13T11:26:16.069874+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"182'6171","last_scrub_stamp":"2024-09-13T11:25:49.468616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:49.468616+0000","objects_scrubbed":1202,"log_size":6618,"log_dups_size":0,"ondisk_log_size":6618,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:27:09.039829+0000","scrub_duration":175,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49479680,"num_objects":755,"num_object_clones":0,"num_object_copies":1510,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":755,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1881,"num_write_kb":32384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,13],"acting":[15,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"3.3","version":"200'5792","reported_seq":6229,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-13T11:26:15.968588+0000","last_change":"2024-09-13T11:25:08.394545+0000","last_active":"2024-09-13T11:26:15.968588+0000","last_peered":"2024-09-13T11:26:15.968588+0000","last_clean":"2024-09-13T11:26:15.968588+0000","last_became_active":"2024-09-13T11:23:31.930448+0000","last_became_peered":"2024-09-13T11:23:31.930448+0000","last_unstale":"2024-09-13T11:26:15.968588+0000","last_undegraded":"2024-09-13T11:26:15.968588+0000","last_fullsized":"2024-09-13T11:26:15.968588+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"151'4587","last_scrub_stamp":"2024-09-13T11:25:08.394441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:08.394441+0000","objects_scrubbed":1942,"log_size":5792,"log_dups_size":0,"ondisk_log_size":5792,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:26:24.755803+0000","scrub_duration":219,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48103424,"num_objects":734,"num_object_clones":0,"num_object_copies":1468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":734,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2670,"num_write_kb":80896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,7],"acting":[10,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"28'2","reported_seq":310,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-13T11:26:15.119332+0000","last_change":"2024-09-13T11:25:13.869564+0000","last_active":"2024-09-13T11:26:15.119332+0000","last_peered":"2024-09-13T11:26:15.119332+0000","last_clean":"2024-09-13T11:26:15.119332+0000","last_became_active":"2024-09-13T11:23:12.546348+0000","last_became_peered":"2024-09-13T11:23:12.546348+0000","last_unstale":"2024-09-13T11:26:15.119332+0000","last_undegraded":"2024-09-13T11:26:15.119332+0000","last_fullsized":"2024-09-13T11:26:15.119332+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2","last_scrub_stamp":"2024-09-13T11:25:13.869413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:25:13.869413+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:26:38.761718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0025826730000000002,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8],"acting":[2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"200'5849","reported_seq":6284,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-13T11:26:16.033518+0000","last_change":"2024-09-13T11:25:09.842875+0000","last_active":"2024-09-13T11:26:16.033518+0000","last_peered":"2024-09-13T11:26:16.033518+0000","last_clean":"2024-09-13T11:26:16.033518+0000","last_became_active":"2024-09-13T11:23:31.921388+0000","last_became_peered":"2024-09-13T11:23:31.921388+0000","last_unstale":"2024-09-13T11:26:16.033518+0000","last_undegraded":"2024-09-13T11:26:16.033518+0000","last_fullsized":"2024-09-13T11:26:16.033518+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"151'4644","last_scrub_stamp":"2024-09-13T11:25:09.842728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:09.842728+0000","objects_scrubbed":1944,"log_size":5849,"log_dups_size":0,"ondisk_log_size":5849,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":395,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48431104,"num_objects":739,"num_object_clones":0,"num_object_copies":1478,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":739,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2743,"num_write_kb":84032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,15],"acting":[12,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":25,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-09-13T11:26:14.113813+0000","last_change":"2024-09-13T11:26:05.047291+0000","last_active":"2024-09-13T11:26:14.113813+0000","last_peered":"2024-09-13T11:26:14.113813+0000","last_clean":"2024-09-13T11:26:14.113813+0000","last_became_active":"2024-09-13T11:26:05.046717+0000","last_became_peered":"2024-09-13T11:26:05.046717+0000","last_unstale":"2024-09-13T11:26:14.113813+0000","last_undegraded":"2024-09-13T11:26:14.113813+0000","last_fullsized":"2024-09-13T11:26:14.113813+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:25:34.822831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:25:34.822831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:26:38.326557+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00081225399999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,1],"acting":[14,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"3.1","version":"198'5758","reported_seq":6192,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-13T11:26:12.098014+0000","last_change":"2024-09-13T11:25:00.861990+0000","last_active":"2024-09-13T11:26:12.098014+0000","last_peered":"2024-09-13T11:26:12.098014+0000","last_clean":"2024-09-13T11:26:12.098014+0000","last_became_active":"2024-09-13T11:23:31.926528+0000","last_became_peered":"2024-09-13T11:23:31.926528+0000","last_unstale":"2024-09-13T11:26:12.098014+0000","last_undegraded":"2024-09-13T11:26:12.098014+0000","last_fullsized":"2024-09-13T11:26:12.098014+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"146'4447","last_scrub_stamp":"2024-09-13T11:25:00.861834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:00.861834+0000","objects_scrubbed":2126,"log_size":5758,"log_dups_size":0,"ondisk_log_size":5758,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":787,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53215232,"num_objects":812,"num_object_clones":0,"num_object_copies":1624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":812,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2688,"num_write_kb":85056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,11],"acting":[13,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":381,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-13T11:26:15.116311+0000","last_change":"2024-09-13T11:25:56.557174+0000","last_active":"2024-09-13T11:26:15.116311+0000","last_peered":"2024-09-13T11:26:15.116311+0000","last_clean":"2024-09-13T11:26:15.116311+0000","last_became_active":"2024-09-13T11:22:14.211435+0000","last_became_peered":"2024-09-13T11:22:14.211435+0000","last_unstale":"2024-09-13T11:26:15.116311+0000","last_undegraded":"2024-09-13T11:26:15.116311+0000","last_fullsized":"2024-09-13T11:26:15.116311+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:25:56.557113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:25:56.557113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:27:18.356524+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0010010360000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2","version":"200'5905","reported_seq":9808,"reported_epoch":200,"state":"active+recovery_wait+remapped","last_fresh":"2024-09-13T11:26:15.117945+0000","last_change":"2024-09-13T11:26:06.240222+0000","last_active":"2024-09-13T11:26:15.117945+0000","last_peered":"2024-09-13T11:26:15.117945+0000","last_clean":"2024-09-13T11:26:03.273688+0000","last_became_active":"2024-09-13T11:26:06.214764+0000","last_became_peered":"2024-09-13T11:26:06.214764+0000","last_unstale":"2024-09-13T11:26:15.117945+0000","last_undegraded":"2024-09-13T11:26:15.117945+0000","last_fullsized":"2024-09-13T11:26:15.117945+0000","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"164'4994","last_scrub_stamp":"2024-09-13T11:25:23.765047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:23.765047+0000","objects_scrubbed":1689,"log_size":5905,"log_dups_size":0,"ondisk_log_size":5905,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":283,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50790400,"num_objects":775,"num_object_clones":0,"num_object_copies":1550,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2231,"num_objects_unfound":0,"num_objects_dirty":775,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2848,"num_write_kb":89024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1732,"num_bytes_recovered":113442816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,12],"acting":[3,12],"avail_no_missing":["3","12"],"object_location_counts":[{"shards":"3,12","objects":-174}],"blocked_by":[],"up_primary":14,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"26'1","reported_seq":29,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-09-13T11:26:14.113814+0000","last_change":"2024-09-13T11:26:05.046376+0000","last_active":"2024-09-13T11:26:14.113814+0000","last_peered":"2024-09-13T11:26:14.113814+0000","last_clean":"2024-09-13T11:26:14.113814+0000","last_became_active":"2024-09-13T11:26:05.041032+0000","last_became_peered":"2024-09-13T11:26:05.041032+0000","last_unstale":"2024-09-13T11:26:14.113814+0000","last_undegraded":"2024-09-13T11:26:14.113814+0000","last_fullsized":"2024-09-13T11:26:14.113814+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"26'1","last_scrub_stamp":"2024-09-13T11:25:23.034247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:25:23.034247+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:26:23.675301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,15],"acting":[14,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"1.0","version":"26'112","reported_seq":515,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-13T11:26:15.176980+0000","last_change":"2024-09-13T11:25:23.642366+0000","last_active":"2024-09-13T11:26:15.176980+0000","last_peered":"2024-09-13T11:26:15.176980+0000","last_clean":"2024-09-13T11:26:15.176980+0000","last_became_active":"2024-09-13T11:22:12.129721+0000","last_became_peered":"2024-09-13T11:22:12.129721+0000","last_unstale":"2024-09-13T11:26:15.176980+0000","last_undegraded":"2024-09-13T11:26:15.176980+0000","last_fullsized":"2024-09-13T11:26:15.176980+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"26'112","last_scrub_stamp":"2024-09-13T11:25:23.642204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:11.098388+0000","last_clean_scrub_stamp":"2024-09-13T11:25:23.642204+0000","objects_scrubbed":2,"log_size":112,"log_dups_size":0,"ondisk_log_size":112,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:26:42.628319+0000","scrub_duration":67,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.8","version":"200'6690","reported_seq":7125,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-13T11:26:15.974251+0000","last_change":"2024-09-13T11:26:00.418168+0000","last_active":"2024-09-13T11:26:15.974251+0000","last_peered":"2024-09-13T11:26:15.974251+0000","last_clean":"2024-09-13T11:26:15.974251+0000","last_became_active":"2024-09-13T11:24:24.096220+0000","last_became_peered":"2024-09-13T11:24:24.096220+0000","last_unstale":"2024-09-13T11:26:15.974251+0000","last_undegraded":"2024-09-13T11:26:15.974251+0000","last_fullsized":"2024-09-13T11:26:15.974251+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"188'6425","last_scrub_stamp":"2024-09-13T11:26:00.417942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:26:00.417942+0000","objects_scrubbed":989,"log_size":6690,"log_dups_size":0,"ondisk_log_size":6690,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:27:07.107188+0000","scrub_duration":199,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47448064,"num_objects":724,"num_object_clones":0,"num_object_copies":1448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":724,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1942,"num_write_kb":31232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.9","version":"200'6713","reported_seq":7130,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-13T11:26:15.992959+0000","last_change":"2024-09-13T11:24:24.928964+0000","last_active":"2024-09-13T11:26:15.992959+0000","last_peered":"2024-09-13T11:26:15.992959+0000","last_clean":"2024-09-13T11:26:15.992959+0000","last_became_active":"2024-09-13T11:24:24.082504+0000","last_became_peered":"2024-09-13T11:24:24.082504+0000","last_unstale":"2024-09-13T11:26:15.992959+0000","last_undegraded":"2024-09-13T11:26:15.992959+0000","last_fullsized":"2024-09-13T11:26:15.992959+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'4799","last_scrub_stamp":"2024-09-13T11:24:24.928724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:24:24.928724+0000","objects_scrubbed":1705,"log_size":6713,"log_dups_size":0,"ondisk_log_size":6713,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":255,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49807360,"num_objects":760,"num_object_clones":0,"num_object_copies":1520,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":760,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1917,"num_write_kb":31104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,2],"acting":[12,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[]},{"pgid":"3.a","version":"199'7065","reported_seq":7504,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-13T11:26:15.118538+0000","last_change":"2024-09-13T11:25:59.695285+0000","last_active":"2024-09-13T11:26:15.118538+0000","last_peered":"2024-09-13T11:26:15.118538+0000","last_clean":"2024-09-13T11:26:15.118538+0000","last_became_active":"2024-09-13T11:24:24.089253+0000","last_became_peered":"2024-09-13T11:24:24.089253+0000","last_unstale":"2024-09-13T11:26:15.118538+0000","last_undegraded":"2024-09-13T11:26:15.118538+0000","last_fullsized":"2024-09-13T11:26:15.118538+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"187'6833","last_scrub_stamp":"2024-09-13T11:25:59.695136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:59.695136+0000","objects_scrubbed":973,"log_size":7065,"log_dups_size":0,"ondisk_log_size":7065,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:27:23.532121+0000","scrub_duration":131,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48496640,"num_objects":740,"num_object_clones":0,"num_object_copies":1480,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":740,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1911,"num_write_kb":27968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.b","version":"198'6975","reported_seq":7423,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-13T11:26:12.045575+0000","last_change":"2024-09-13T11:25:58.292249+0000","last_active":"2024-09-13T11:26:12.045575+0000","last_peered":"2024-09-13T11:26:12.045575+0000","last_clean":"2024-09-13T11:26:12.045575+0000","last_became_active":"2024-09-13T11:24:24.085684+0000","last_became_peered":"2024-09-13T11:24:24.085684+0000","last_unstale":"2024-09-13T11:26:12.045575+0000","last_undegraded":"2024-09-13T11:26:12.045575+0000","last_fullsized":"2024-09-13T11:26:12.045575+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"186'6759","last_scrub_stamp":"2024-09-13T11:25:58.292153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:58.292153+0000","objects_scrubbed":1030,"log_size":6975,"log_dups_size":0,"ondisk_log_size":6975,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:27:02.082370+0000","scrub_duration":119,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53280768,"num_objects":813,"num_object_clones":0,"num_object_copies":1626,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":813,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1863,"num_write_kb":30656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,1],"acting":[13,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"3.c","version":"198'6980","reported_seq":7419,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-13T11:26:12.001830+0000","last_change":"2024-09-13T11:25:36.279464+0000","last_active":"2024-09-13T11:26:12.001830+0000","last_peered":"2024-09-13T11:26:12.001830+0000","last_clean":"2024-09-13T11:26:12.001830+0000","last_became_active":"2024-09-13T11:24:24.090847+0000","last_became_peered":"2024-09-13T11:24:24.090847+0000","last_unstale":"2024-09-13T11:26:12.001830+0000","last_undegraded":"2024-09-13T11:26:12.001830+0000","last_fullsized":"2024-09-13T11:26:12.001830+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"174'6334","last_scrub_stamp":"2024-09-13T11:25:36.279390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:36.279390+0000","objects_scrubbed":1456,"log_size":6980,"log_dups_size":0,"ondisk_log_size":6980,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:26:57.115518+0000","scrub_duration":235,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53018624,"num_objects":809,"num_object_clones":0,"num_object_copies":1618,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":809,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1831,"num_write_kb":29440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,13],"acting":[11,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.d","version":"196'6962","reported_seq":7395,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-09-13T11:26:11.093788+0000","last_change":"2024-09-13T11:25:57.320482+0000","last_active":"2024-09-13T11:26:11.093788+0000","last_peered":"2024-09-13T11:26:11.093788+0000","last_clean":"2024-09-13T11:26:11.093788+0000","last_became_active":"2024-09-13T11:24:24.087224+0000","last_became_peered":"2024-09-13T11:24:24.087224+0000","last_unstale":"2024-09-13T11:26:11.093788+0000","last_undegraded":"2024-09-13T11:26:11.093788+0000","last_fullsized":"2024-09-13T11:26:11.093788+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"186'6735","last_scrub_stamp":"2024-09-13T11:25:57.320378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:57.320378+0000","objects_scrubbed":1064,"log_size":6962,"log_dups_size":0,"ondisk_log_size":6962,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:27:00.291410+0000","scrub_duration":211,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":54788096,"num_objects":836,"num_object_clones":0,"num_object_copies":1672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":836,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1855,"num_write_kb":30912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.e","version":"200'9035","reported_seq":14396,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-13T11:26:16.065562+0000","last_change":"2024-09-13T11:25:46.438251+0000","last_active":"2024-09-13T11:26:16.065562+0000","last_peered":"2024-09-13T11:26:16.065562+0000","last_clean":"2024-09-13T11:26:16.065562+0000","last_became_active":"2024-09-13T11:23:31.925869+0000","last_became_peered":"2024-09-13T11:23:31.925869+0000","last_unstale":"2024-09-13T11:26:16.065562+0000","last_undegraded":"2024-09-13T11:26:16.065562+0000","last_fullsized":"2024-09-13T11:26:16.065562+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"180'8032","last_scrub_stamp":"2024-09-13T11:25:46.438107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:46.438107+0000","objects_scrubbed":2456,"log_size":9035,"log_dups_size":0,"ondisk_log_size":9035,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:26:55.276011+0000","scrub_duration":327,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":94961664,"num_objects":1449,"num_object_clones":0,"num_object_copies":2898,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1449,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5509,"num_write_kb":165888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"198'8878","reported_seq":9314,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-13T11:26:12.095667+0000","last_change":"2024-09-13T11:25:00.289940+0000","last_active":"2024-09-13T11:26:12.095667+0000","last_peered":"2024-09-13T11:26:12.095667+0000","last_clean":"2024-09-13T11:26:12.095667+0000","last_became_active":"2024-09-13T11:23:31.935239+0000","last_became_peered":"2024-09-13T11:23:31.935239+0000","last_unstale":"2024-09-13T11:26:12.095667+0000","last_undegraded":"2024-09-13T11:26:12.095667+0000","last_fullsized":"2024-09-13T11:26:12.095667+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"146'6154","last_scrub_stamp":"2024-09-13T11:25:00.289673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:00.289673+0000","objects_scrubbed":4394,"log_size":8878,"log_dups_size":0,"ondisk_log_size":8878,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:26:25.833168+0000","scrub_duration":1187,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":108003328,"num_objects":1648,"num_object_clones":0,"num_object_copies":3296,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1648,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5458,"num_write_kb":170560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,12],"acting":[11,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.10","version":"200'8802","reported_seq":13394,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-13T11:26:16.048947+0000","last_change":"2024-09-13T11:25:10.161564+0000","last_active":"2024-09-13T11:26:16.048947+0000","last_peered":"2024-09-13T11:26:16.048947+0000","last_clean":"2024-09-13T11:26:16.048947+0000","last_became_active":"2024-09-13T11:23:41.931183+0000","last_became_peered":"2024-09-13T11:23:41.931183+0000","last_unstale":"2024-09-13T11:26:16.048947+0000","last_undegraded":"2024-09-13T11:26:16.048947+0000","last_fullsized":"2024-09-13T11:26:16.048947+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":5,"last_scrub":"151'6298","last_scrub_stamp":"2024-09-13T11:25:10.161464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:10.161464+0000","objects_scrubbed":3964,"log_size":8802,"log_dups_size":0,"ondisk_log_size":8802,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:26:21.692888+0000","scrub_duration":607,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":94699520,"num_objects":1445,"num_object_clones":0,"num_object_copies":2890,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1445,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5336,"num_write_kb":152000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.11","version":"200'8681","reported_seq":12046,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-13T11:26:15.980336+0000","last_change":"2024-09-13T11:25:24.287326+0000","last_active":"2024-09-13T11:26:15.980336+0000","last_peered":"2024-09-13T11:26:15.980336+0000","last_clean":"2024-09-13T11:26:15.980336+0000","last_became_active":"2024-09-13T11:23:39.063172+0000","last_became_peered":"2024-09-13T11:23:39.063172+0000","last_unstale":"2024-09-13T11:26:15.980336+0000","last_undegraded":"2024-09-13T11:26:15.980336+0000","last_fullsized":"2024-09-13T11:26:15.980336+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":5,"last_scrub":"165'6821","last_scrub_stamp":"2024-09-13T11:25:24.287082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:24.287082+0000","objects_scrubbed":3406,"log_size":8681,"log_dups_size":0,"ondisk_log_size":8681,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:26:53.631356+0000","scrub_duration":947,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":100466688,"num_objects":1533,"num_object_clones":0,"num_object_copies":3066,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1533,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5340,"num_write_kb":155584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11],"acting":[4,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.12","version":"200'8663","reported_seq":18890,"reported_epoch":200,"state":"active+remapped+backfill_wait","last_fresh":"2024-09-13T11:26:16.035952+0000","last_change":"2024-09-13T11:26:06.274632+0000","last_active":"2024-09-13T11:26:16.035952+0000","last_peered":"2024-09-13T11:26:16.035952+0000","last_clean":"2024-09-13T11:26:03.227632+0000","last_became_active":"2024-09-13T11:26:06.225212+0000","last_became_peered":"2024-09-13T11:26:06.225212+0000","last_unstale":"2024-09-13T11:26:16.035952+0000","last_undegraded":"2024-09-13T11:26:16.035952+0000","last_fullsized":"2024-09-13T11:26:16.035952+0000","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":5,"last_scrub":"186'8089","last_scrub_stamp":"2024-09-13T11:25:57.085837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:57.085837+0000","objects_scrubbed":2130,"log_size":8663,"log_dups_size":0,"ondisk_log_size":8663,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1023,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":101187584,"num_objects":1544,"num_object_clones":0,"num_object_copies":3088,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1544,"num_objects_unfound":0,"num_objects_dirty":1544,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5554,"num_write_kb":171712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3457,"num_bytes_recovered":226557952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,7],"acting":[5,7],"avail_no_missing":["5","7"],"object_location_counts":[{"shards":"5,7","objects":1544}],"blocked_by":[],"up_primary":14,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"198'8550","reported_seq":11845,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-13T11:26:12.023775+0000","last_change":"2024-09-13T11:25:01.153044+0000","last_active":"2024-09-13T11:26:12.023775+0000","last_peered":"2024-09-13T11:26:12.023775+0000","last_clean":"2024-09-13T11:26:12.023775+0000","last_became_active":"2024-09-13T11:23:41.082433+0000","last_became_peered":"2024-09-13T11:23:41.082433+0000","last_unstale":"2024-09-13T11:26:12.023775+0000","last_undegraded":"2024-09-13T11:26:12.023775+0000","last_fullsized":"2024-09-13T11:26:12.023775+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":5,"last_scrub":"147'5921","last_scrub_stamp":"2024-09-13T11:25:01.152971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:01.152971+0000","objects_scrubbed":4311,"log_size":8550,"log_dups_size":0,"ondisk_log_size":8550,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:26:29.889906+0000","scrub_duration":1011,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":107544576,"num_objects":1641,"num_object_clones":0,"num_object_copies":3282,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1641,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5154,"num_write_kb":154048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7],"acting":[11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":46,"stat_sum":{"num_bytes":3202744344,"num_objects":48871,"num_object_clones":0,"num_object_copies":97742,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":8874,"num_objects_unfound":0,"num_objects_dirty":48871,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":141077,"num_write_kb":3354049,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19353,"num_bytes_recovered":1260912640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":7240425472,"data_stored":7240417328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":43282,"internal_metadata":0},"log_size":341208,"ondisk_log_size":341208,"up":92,"acting":92,"num_store_stats":16},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":11},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":112,"ondisk_log_size":112,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":13,"up_from":23,"seq":98784247860,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":435876,"kb_used_data":324180,"kb_used_omap":1,"kb_used_meta":111678,"kb_avail":93935964,"statfs":{"total":96636764160,"available":96190427136,"internally_reserved":0,"allocated":331960320,"data_stored":331072087,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":114358725},"hb_peers":[0,1,2,4,6,8,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":23,"seq":98784247859,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":382968,"kb_used_data":315688,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":93988872,"statfs":{"total":96636764160,"available":96244604928,"internally_reserved":0,"allocated":323264512,"data_stored":322423412,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":68876736},"hb_peers":[0,1,3,8,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":40,"seq":171798691883,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":563988,"kb_used_data":459716,"kb_used_omap":4,"kb_used_meta":104187,"kb_avail":93807852,"statfs":{"total":96636764160,"available":96059240448,"internally_reserved":0,"allocated":470749184,"data_stored":469890009,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4433,"internal_metadata":106688175},"hb_peers":[0,2,4,5,6,7,8,9,11,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":21,"seq":90194313268,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":569664,"kb_used_data":470384,"kb_used_omap":1,"kb_used_meta":99198,"kb_avail":93802176,"statfs":{"total":96636764160,"available":96053428224,"internally_reserved":0,"allocated":481673216,"data_stored":480776725,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":101579207},"hb_peers":[0,2,4,5,7,10,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":21,"seq":90194313271,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":716604,"kb_used_data":599276,"kb_used_omap":1,"kb_used_meta":117246,"kb_avail":93655236,"statfs":{"total":96636764160,"available":95902961664,"internally_reserved":0,"allocated":613658624,"data_stored":612769410,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":120060359},"hb_peers":[0,2,4,6,7,9,11,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":22,"seq":94489280563,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":663628,"kb_used_data":560188,"kb_used_omap":1,"kb_used_meta":103358,"kb_avail":93708212,"statfs":{"total":96636764160,"available":95957209088,"internally_reserved":0,"allocated":573632512,"data_stored":572789325,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":105839045},"hb_peers":[1,3,4,5,7,8,10,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":22,"seq":94489280564,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":276212,"kb_used_data":200612,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94095628,"statfs":{"total":96636764160,"available":96353923072,"internally_reserved":0,"allocated":205426688,"data_stored":204549521,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,1,3,4,5,7,8,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":15,"up_from":22,"seq":94489280564,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":589940,"kb_used_data":490724,"kb_used_omap":1,"kb_used_meta":99198,"kb_avail":93781900,"statfs":{"total":96636764160,"available":96032665600,"internally_reserved":0,"allocated":502501376,"data_stored":501681379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":101579206},"hb_peers":[0,4,5,6,7,8,10,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":22,"seq":94489280565,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":413764,"kb_used_data":342260,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":93958076,"statfs":{"total":96636764160,"available":96213069824,"internally_reserved":0,"allocated":350474240,"data_stored":349582462,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73136581},"hb_peers":[3,5,6,7,9,10,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":22,"seq":94489280564,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":300700,"kb_used_data":229260,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":94071140,"statfs":{"total":96636764160,"available":96328847360,"internally_reserved":0,"allocated":234762240,"data_stored":233905428,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":73136582},"hb_peers":[0,1,3,5,7,8,10,11,12,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":22,"seq":94489280563,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":798768,"kb_used_data":691232,"kb_used_omap":1,"kb_used_meta":107518,"kb_avail":93573072,"statfs":{"total":96636764160,"available":95818825728,"internally_reserved":0,"allocated":707821568,"data_stored":706983368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":110098890},"hb_peers":[0,3,5,6,8,9,10,11,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":22,"seq":94489280564,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":281112,"kb_used_data":209672,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":94090728,"statfs":{"total":96636764160,"available":96348905472,"internally_reserved":0,"allocated":214704128,"data_stored":213788815,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73136581},"hb_peers":[1,5,6,7,9,10,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":22,"seq":94489280564,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":545180,"kb_used_data":437644,"kb_used_omap":1,"kb_used_meta":107518,"kb_avail":93826660,"statfs":{"total":96636764160,"available":96078499840,"internally_reserved":0,"allocated":448147456,"data_stored":447290425,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":110098880},"hb_peers":[0,1,3,5,7,9,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":11,"up_from":22,"seq":94489280564,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":791588,"kb_used_data":688212,"kb_used_omap":1,"kb_used_meta":103358,"kb_avail":93580252,"statfs":{"total":96636764160,"available":95826178048,"internally_reserved":0,"allocated":704729088,"data_stored":703844554,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":105839044},"hb_peers":[0,1,2,4,7,8,10,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":12,"up_from":22,"seq":94489280564,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":655872,"kb_used_data":538608,"kb_used_omap":1,"kb_used_meta":117246,"kb_avail":93715968,"statfs":{"total":96636764160,"available":95965151232,"internally_reserved":0,"allocated":551534592,"data_stored":550643589,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":120060357},"hb_peers":[0,1,2,3,4,5,10,11,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":14,"up_from":142,"seq":609885356049,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":663488,"kb_used_data":613552,"kb_used_omap":42,"kb_used_meta":49877,"kb_avail":93708352,"statfs":{"total":96636764160,"available":95957352448,"internally_reserved":0,"allocated":628277248,"data_stored":627437489,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":43282,"internal_metadata":51074798},"hb_peers":[0,1,3,5,7,8,9,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":567476224,"data_stored":567476224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":462487552,"data_stored":462487552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":201461760,"data_stored":201457688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":472645632,"data_stored":472645632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":344326144,"data_stored":344326144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":604176384,"data_stored":604176384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":231931904,"data_stored":231931904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":705626112,"data_stored":705626112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":211550208,"data_stored":211550208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":320798720,"data_stored":320798720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":437649408,"data_stored":437649408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":697565184,"data_stored":697565184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":538771456,"data_stored":538771456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":322633728,"data_stored":322633728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":630194176,"data_stored":630194176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":43282,"internal_metadata":0},{"poolid":3,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":491130880,"data_stored":491126808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T11:26:16.297 INFO:tasks.thrashosds.thrasher:Setting 3.22 to [14, 12] 2024-09-13T11:26:16.297 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap', '3.22', '14', '12'] 2024-09-13T11:26:16.297 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pg-upmap 3.22 14 12 2024-09-13T11:26:16.336 INFO:tasks.ceph.osd.12.smithi001.stderr:2024-09-13T11:26:16.334+0000 7f892c6e6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26314) UID: 0 2024-09-13T11:26:16.401 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-13T11:26:16.402 INFO:teuthology.orchestra.run.smithi001.stdout: "size": 20, 2024-09-13T11:26:16.402 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 600, 2024-09-13T11:26:16.402 INFO:teuthology.orchestra.run.smithi001.stdout: "ops": [ 2024-09-13T11:26:16.402 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.402 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDPGPush(3.1e 89/78 [PushOp(3:7a07f38c:::benchmark_data_smithi177_31116_object20966:head, version: 54'1402, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:7a07f38c:::benchmark_data_smithi177_31116_object20966:head@54'1402, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:26:16.402 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:46.645251+0000", 2024-09-13T11:26:16.402 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 149.75355353800001, 2024-09-13T11:26:16.402 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.17670883800000001, 2024-09-13T11:26:16.402 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:26:16.402 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:26:16.402 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:26:16.403 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:26:16.403 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.403 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:26:16.403 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.645251+0000", 2024-09-13T11:26:16.403 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:16.403 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.403 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.403 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:26:16.403 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.645251+0000", 2024-09-13T11:26:16.403 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:16.403 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.403 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.403 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:26:16.404 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.645253+0000", 2024-09-13T11:26:16.404 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.1399999999999998e-06 2024-09-13T11:26:16.404 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.404 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.404 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:26:16.404 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.645295+0000", 2024-09-13T11:26:16.404 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.2323000000000001e-05 2024-09-13T11:26:16.404 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.404 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.404 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:26:16.404 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.820731+0000", 2024-09-13T11:26:16.404 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.17543607999999999 2024-09-13T11:26:16.404 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.405 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.405 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:26:16.405 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.820750+0000", 2024-09-13T11:26:16.405 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.8793000000000001e-05 2024-09-13T11:26:16.405 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.405 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.405 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:26:16.405 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.820914+0000", 2024-09-13T11:26:16.405 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000164062 2024-09-13T11:26:16.405 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.405 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.405 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:26:16.405 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.820939+0000", 2024-09-13T11:26:16.406 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.4672e-05 2024-09-13T11:26:16.406 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.406 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.406 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:26:16.406 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.821960+0000", 2024-09-13T11:26:16.406 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.001020768 2024-09-13T11:26:16.406 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:16.406 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:16.406 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:16.406 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.406 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.406 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDPGPush(3.1a 102/78 [PushOp(3:5df50766:::benchmark_data_smithi177_31116_object50543:head, version: 72'3152, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:5df50766:::benchmark_data_smithi177_31116_object50543:head@72'3152, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:26:16.406 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:24:00.499311+0000", 2024-09-13T11:26:16.406 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 135.89949345299999, 2024-09-13T11:26:16.407 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.26146664800000002, 2024-09-13T11:26:16.407 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:26:16.407 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:26:16.407 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:26:16.407 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:26:16.407 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.407 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:26:16.407 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:00.499311+0000", 2024-09-13T11:26:16.407 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:16.407 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.407 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.407 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:26:16.407 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:00.499311+0000", 2024-09-13T11:26:16.407 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:16.408 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.408 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.408 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:26:16.408 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:00.499320+0000", 2024-09-13T11:26:16.408 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.6400000000000003e-06 2024-09-13T11:26:16.408 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.408 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.408 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:26:16.408 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:00.499384+0000", 2024-09-13T11:26:16.408 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.4554999999999994e-05 2024-09-13T11:26:16.408 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.408 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.408 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:26:16.409 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:00.759713+0000", 2024-09-13T11:26:16.409 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.260328742 2024-09-13T11:26:16.409 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.409 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.409 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:26:16.409 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:00.759742+0000", 2024-09-13T11:26:16.409 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.9298e-05 2024-09-13T11:26:16.409 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.409 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.409 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:26:16.409 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:00.759871+0000", 2024-09-13T11:26:16.409 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000128992 2024-09-13T11:26:16.409 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.409 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.410 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:26:16.410 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:00.759884+0000", 2024-09-13T11:26:16.410 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2687999999999999e-05 2024-09-13T11:26:16.410 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.410 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.410 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:26:16.410 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:00.760778+0000", 2024-09-13T11:26:16.410 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00089373299999999996 2024-09-13T11:26:16.410 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:16.410 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:16.410 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:16.410 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.410 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.410 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4665.0:82769 3.1e e105/102 3:7a4e88b5:::benchmark_data_smithi177_31116_object82768:head v 105'4381, mlcod=105'4379)", 2024-09-13T11:26:16.411 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:24:03.745047+0000", 2024-09-13T11:26:16.411 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 132.65375722799999, 2024-09-13T11:26:16.411 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.213806145, 2024-09-13T11:26:16.411 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:26:16.411 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:26:16.411 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:16.411 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:26:16.411 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.411 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:26:16.411 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:03.745047+0000", 2024-09-13T11:26:16.411 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:16.411 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.411 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.411 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:26:16.412 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:03.745047+0000", 2024-09-13T11:26:16.412 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:16.412 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.412 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.412 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:26:16.412 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:03.745049+0000", 2024-09-13T11:26:16.412 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.5710000000000001e-06 2024-09-13T11:26:16.412 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.412 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.412 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:26:16.412 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:03.745107+0000", 2024-09-13T11:26:16.412 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.8799999999999999e-05 2024-09-13T11:26:16.412 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.413 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.413 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:26:16.413 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:03.955145+0000", 2024-09-13T11:26:16.413 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.210037735 2024-09-13T11:26:16.413 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.413 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.413 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:26:16.413 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:03.955163+0000", 2024-09-13T11:26:16.413 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.7520000000000002e-05 2024-09-13T11:26:16.413 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.413 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.413 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:26:16.413 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:03.955473+0000", 2024-09-13T11:26:16.413 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00031014399999999999 2024-09-13T11:26:16.414 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.414 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.414 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:26:16.414 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:03.955538+0000", 2024-09-13T11:26:16.414 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.5059000000000002e-05 2024-09-13T11:26:16.414 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.414 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.414 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:26:16.414 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:03.958753+0000", 2024-09-13T11:26:16.414 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0032152769999999999 2024-09-13T11:26:16.414 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.414 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.414 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:26:16.415 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:03.958853+0000", 2024-09-13T11:26:16.415 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000100039 2024-09-13T11:26:16.415 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:16.415 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:16.415 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:16.415 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.415 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.415 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:98582 3.1a 3:59f9cd75:::benchmark_data_smithi177_31116_object98581:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113)", 2024-09-13T11:26:16.415 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:24:15.431226+0000", 2024-09-13T11:26:16.415 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 120.96757875900001, 2024-09-13T11:26:16.415 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.19820755800000001, 2024-09-13T11:26:16.415 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:26:16.415 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:26:16.416 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:16.416 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:26:16.416 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:26:16.416 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:16.416 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 98582 2024-09-13T11:26:16.416 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.416 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:26:16.416 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.416 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:26:16.416 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.431226+0000", 2024-09-13T11:26:16.416 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:16.416 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.416 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.417 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:26:16.417 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.431226+0000", 2024-09-13T11:26:16.417 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:16.417 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.417 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.417 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:26:16.417 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.431229+0000", 2024-09-13T11:26:16.417 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.7100000000000001e-06 2024-09-13T11:26:16.417 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.417 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.417 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:26:16.417 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.431334+0000", 2024-09-13T11:26:16.417 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00010506900000000001 2024-09-13T11:26:16.418 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.418 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.418 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:26:16.418 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.431364+0000", 2024-09-13T11:26:16.418 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.9607e-05 2024-09-13T11:26:16.418 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.418 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.418 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:26:16.418 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.431388+0000", 2024-09-13T11:26:16.418 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.3808999999999999e-05 2024-09-13T11:26:16.418 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.418 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.418 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:26:16.418 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.431483+0000", 2024-09-13T11:26:16.419 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.5402999999999994e-05 2024-09-13T11:26:16.419 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.419 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.419 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:26:16.419 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.431642+0000", 2024-09-13T11:26:16.419 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000158412 2024-09-13T11:26:16.419 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.419 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.419 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 9", 2024-09-13T11:26:16.419 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.431796+0000", 2024-09-13T11:26:16.419 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00015467899999999999 2024-09-13T11:26:16.419 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.419 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.420 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:26:16.420 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.433731+0000", 2024-09-13T11:26:16.420 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.001934945 2024-09-13T11:26:16.420 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.420 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.420 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:16.420 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.629245+0000", 2024-09-13T11:26:16.420 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.19551390199999999 2024-09-13T11:26:16.420 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.420 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.420 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:26:16.420 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.629346+0000", 2024-09-13T11:26:16.420 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000100869 2024-09-13T11:26:16.420 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.421 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.421 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:26:16.421 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.629433+0000", 2024-09-13T11:26:16.421 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.7152999999999997e-05 2024-09-13T11:26:16.421 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:16.421 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:16.421 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:16.421 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.421 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.421 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:99105 3.1a 3:5ac5090c:::benchmark_data_smithi177_31116_object99104:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113)", 2024-09-13T11:26:16.421 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:24:15.962466+0000", 2024-09-13T11:26:16.421 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 120.43633847, 2024-09-13T11:26:16.421 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.23832849, 2024-09-13T11:26:16.422 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:26:16.422 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:26:16.422 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:16.422 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:26:16.422 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:26:16.422 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:16.422 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 99105 2024-09-13T11:26:16.422 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.422 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:26:16.422 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.422 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:26:16.422 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.962466+0000", 2024-09-13T11:26:16.422 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:16.422 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.423 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.423 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:26:16.423 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.962466+0000", 2024-09-13T11:26:16.423 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:16.423 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.423 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.423 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:26:16.423 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.962471+0000", 2024-09-13T11:26:16.423 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.9570000000000001e-06 2024-09-13T11:26:16.423 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.423 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.423 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:26:16.423 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.962547+0000", 2024-09-13T11:26:16.423 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.5947999999999999e-05 2024-09-13T11:26:16.424 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.424 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.424 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:26:16.424 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.962576+0000", 2024-09-13T11:26:16.424 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.9547999999999999e-05 2024-09-13T11:26:16.424 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.424 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.424 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:26:16.424 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.962594+0000", 2024-09-13T11:26:16.424 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.8023e-05 2024-09-13T11:26:16.424 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.424 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.424 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:26:16.425 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.962770+0000", 2024-09-13T11:26:16.425 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00017551099999999999 2024-09-13T11:26:16.425 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.425 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.425 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:26:16.425 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.963114+0000", 2024-09-13T11:26:16.425 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00034440399999999998 2024-09-13T11:26:16.425 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.425 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.425 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 9", 2024-09-13T11:26:16.425 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.963351+0000", 2024-09-13T11:26:16.425 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000237209 2024-09-13T11:26:16.425 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.425 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.426 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:26:16.426 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.966345+0000", 2024-09-13T11:26:16.426 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0029933519999999999 2024-09-13T11:26:16.426 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.426 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.426 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:16.426 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:16.200724+0000", 2024-09-13T11:26:16.426 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.23437934599999999 2024-09-13T11:26:16.426 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.426 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.426 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:26:16.426 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:16.200761+0000", 2024-09-13T11:26:16.426 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.6406999999999999e-05 2024-09-13T11:26:16.426 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.427 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.427 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:26:16.427 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:16.200794+0000", 2024-09-13T11:26:16.427 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.3785000000000003e-05 2024-09-13T11:26:16.427 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:16.427 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:16.427 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:16.427 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.427 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.427 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4665.0:104331 3.1e e115/102 3:79890035:::benchmark_data_smithi177_31116_object104330:head v 115'5083, mlcod=115'5080)", 2024-09-13T11:26:16.427 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:24:20.242656+0000", 2024-09-13T11:26:16.427 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 116.156148378, 2024-09-13T11:26:16.427 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.16813846700000001, 2024-09-13T11:26:16.428 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:26:16.428 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:26:16.428 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:16.428 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:26:16.428 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.428 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:26:16.428 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:20.242656+0000", 2024-09-13T11:26:16.428 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:16.428 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.428 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.428 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:26:16.428 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:20.242656+0000", 2024-09-13T11:26:16.428 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:16.429 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.429 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.429 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:26:16.429 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:20.242657+0000", 2024-09-13T11:26:16.429 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.4750000000000001e-06 2024-09-13T11:26:16.429 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.429 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.429 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:26:16.429 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:20.242699+0000", 2024-09-13T11:26:16.429 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.193e-05 2024-09-13T11:26:16.429 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.429 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.429 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:26:16.429 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:20.408772+0000", 2024-09-13T11:26:16.430 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.16607291199999999 2024-09-13T11:26:16.430 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.430 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.430 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:26:16.430 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:20.408779+0000", 2024-09-13T11:26:16.430 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.3600000000000001e-06 2024-09-13T11:26:16.430 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.430 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.430 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:26:16.430 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:20.408840+0000", 2024-09-13T11:26:16.430 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.1834999999999993e-05 2024-09-13T11:26:16.430 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.430 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.431 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:26:16.431 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:20.408865+0000", 2024-09-13T11:26:16.431 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.4294000000000002e-05 2024-09-13T11:26:16.431 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.431 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.431 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:26:16.431 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:20.410751+0000", 2024-09-13T11:26:16.431 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.001885811 2024-09-13T11:26:16.431 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.431 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.431 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:26:16.431 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:20.410794+0000", 2024-09-13T11:26:16.431 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.3850000000000002e-05 2024-09-13T11:26:16.431 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:16.432 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:16.432 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:16.432 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.432 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.432 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:129921 3.1a 3:5fb9088b:::benchmark_data_smithi177_31116_object129920:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e134)", 2024-09-13T11:26:16.432 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:24:47.177556+0000", 2024-09-13T11:26:16.432 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 89.221248191000001, 2024-09-13T11:26:16.432 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.33804469999999998, 2024-09-13T11:26:16.432 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:26:16.432 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:26:16.432 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:16.432 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:26:16.432 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:26:16.433 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:16.433 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 129921 2024-09-13T11:26:16.433 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.433 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:26:16.433 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.433 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:26:16.433 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:47.177556+0000", 2024-09-13T11:26:16.433 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:16.433 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.433 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.433 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:26:16.433 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:47.177556+0000", 2024-09-13T11:26:16.433 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:16.433 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.434 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.434 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:26:16.434 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:47.177559+0000", 2024-09-13T11:26:16.434 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.0790000000000001e-06 2024-09-13T11:26:16.434 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.434 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.434 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:26:16.434 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:47.177589+0000", 2024-09-13T11:26:16.434 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.0173999999999999e-05 2024-09-13T11:26:16.434 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.434 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.434 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:26:16.434 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:47.177606+0000", 2024-09-13T11:26:16.435 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.6475999999999999e-05 2024-09-13T11:26:16.435 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.435 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.435 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:26:16.435 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:47.177615+0000", 2024-09-13T11:26:16.435 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.5729999999999997e-06 2024-09-13T11:26:16.435 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.435 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.435 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:26:16.435 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:47.177680+0000", 2024-09-13T11:26:16.435 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.4349000000000003e-05 2024-09-13T11:26:16.435 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.435 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.435 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:26:16.435 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:47.177794+0000", 2024-09-13T11:26:16.436 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000114119 2024-09-13T11:26:16.436 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.436 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.436 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 9", 2024-09-13T11:26:16.436 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:47.177895+0000", 2024-09-13T11:26:16.436 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000101078 2024-09-13T11:26:16.436 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.436 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.436 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:26:16.436 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:47.179634+0000", 2024-09-13T11:26:16.436 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.001738606 2024-09-13T11:26:16.436 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.436 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.437 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:16.437 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:47.515495+0000", 2024-09-13T11:26:16.437 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.33586143000000002 2024-09-13T11:26:16.437 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.437 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.437 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:26:16.437 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:47.515551+0000", 2024-09-13T11:26:16.437 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.6048999999999998e-05 2024-09-13T11:26:16.437 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.437 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.437 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:26:16.437 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:47.515601+0000", 2024-09-13T11:26:16.437 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.9767e-05 2024-09-13T11:26:16.437 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:16.438 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:16.438 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:16.438 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.438 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.438 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4665.0:140742 3.1e e144/102 3:79c694a0:::benchmark_data_smithi177_31116_object439:head v 144'6204, mlcod=144'6200)", 2024-09-13T11:26:16.438 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:24:58.321534+0000", 2024-09-13T11:26:16.438 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 78.077270666999993, 2024-09-13T11:26:16.438 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.23629921500000001, 2024-09-13T11:26:16.438 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:26:16.438 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:26:16.438 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:16.438 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:26:16.438 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.438 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:26:16.439 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.321534+0000", 2024-09-13T11:26:16.439 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:16.439 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.439 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.439 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:26:16.439 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.321534+0000", 2024-09-13T11:26:16.439 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:16.439 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.439 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.439 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:26:16.439 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.321535+0000", 2024-09-13T11:26:16.439 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.159e-06 2024-09-13T11:26:16.439 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.439 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.440 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:26:16.440 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.321545+0000", 2024-09-13T11:26:16.440 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.9219999999999992e-06 2024-09-13T11:26:16.440 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.440 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.440 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:26:16.440 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.553719+0000", 2024-09-13T11:26:16.440 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.23217401700000001 2024-09-13T11:26:16.440 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.440 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.440 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:26:16.440 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.553734+0000", 2024-09-13T11:26:16.440 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.5344000000000001e-05 2024-09-13T11:26:16.441 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.441 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.441 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:26:16.441 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.553858+0000", 2024-09-13T11:26:16.441 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00012357500000000001 2024-09-13T11:26:16.441 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.441 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.441 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:26:16.441 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.553903+0000", 2024-09-13T11:26:16.441 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.5374999999999998e-05 2024-09-13T11:26:16.441 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.441 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.441 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:26:16.441 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.557719+0000", 2024-09-13T11:26:16.442 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0038158670000000001 2024-09-13T11:26:16.442 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.442 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.442 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:26:16.442 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.557833+0000", 2024-09-13T11:26:16.442 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000113956 2024-09-13T11:26:16.442 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:16.442 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:16.442 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:16.442 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.442 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.442 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4665.0:140796 3.1e e144/102 3:7987a652:::benchmark_data_smithi177_31116_object493:head v 144'6205, mlcod=144'6202)", 2024-09-13T11:26:16.442 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:24:58.553657+0000", 2024-09-13T11:26:16.443 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 77.845147690999994, 2024-09-13T11:26:16.443 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.25308323100000002, 2024-09-13T11:26:16.443 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:26:16.443 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:26:16.443 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:16.443 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:26:16.443 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.443 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:26:16.443 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.553657+0000", 2024-09-13T11:26:16.443 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:16.443 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.443 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.443 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:26:16.443 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.553657+0000", 2024-09-13T11:26:16.444 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:16.444 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.444 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.444 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:26:16.444 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.553660+0000", 2024-09-13T11:26:16.444 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.7519999999999998e-06 2024-09-13T11:26:16.444 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.444 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.444 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:26:16.444 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.553678+0000", 2024-09-13T11:26:16.444 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.7156e-05 2024-09-13T11:26:16.444 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.444 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.445 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:26:16.445 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.804342+0000", 2024-09-13T11:26:16.445 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.25066436199999997 2024-09-13T11:26:16.445 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.445 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.445 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:26:16.445 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.804353+0000", 2024-09-13T11:26:16.445 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.1147e-05 2024-09-13T11:26:16.445 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.445 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.445 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:26:16.445 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.804433+0000", 2024-09-13T11:26:16.445 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.9586999999999995e-05 2024-09-13T11:26:16.445 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.446 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.446 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:26:16.446 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.804457+0000", 2024-09-13T11:26:16.446 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.4108e-05 2024-09-13T11:26:16.446 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.446 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.446 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:26:16.446 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.806684+0000", 2024-09-13T11:26:16.446 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0022271309999999998 2024-09-13T11:26:16.446 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.446 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.446 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:26:16.446 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.806740+0000", 2024-09-13T11:26:16.447 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.5988e-05 2024-09-13T11:26:16.447 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:16.447 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:16.447 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:16.447 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.447 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.447 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:149955 3.11 3:8fa29838:::benchmark_data_smithi177_31116_object9652:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e149)", 2024-09-13T11:26:16.447 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:25:05.905492+0000", 2024-09-13T11:26:16.447 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 70.493312317000004, 2024-09-13T11:26:16.447 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.204181955, 2024-09-13T11:26:16.447 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:26:16.447 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:26:16.447 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:16.447 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:26:16.448 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:26:16.448 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:16.448 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 149955 2024-09-13T11:26:16.448 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.448 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:26:16.448 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.448 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:26:16.448 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:05.905492+0000", 2024-09-13T11:26:16.448 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:16.448 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.448 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.448 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:26:16.448 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:05.905492+0000", 2024-09-13T11:26:16.449 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:16.449 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.449 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.449 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:26:16.449 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:05.905494+0000", 2024-09-13T11:26:16.449 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.9369999999999998e-06 2024-09-13T11:26:16.449 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.449 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.449 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:26:16.449 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:05.905500+0000", 2024-09-13T11:26:16.449 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.7039999999999999e-06 2024-09-13T11:26:16.449 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.449 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.449 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:26:16.450 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:05.905512+0000", 2024-09-13T11:26:16.450 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.257e-05 2024-09-13T11:26:16.450 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.450 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.450 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:26:16.450 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:05.905519+0000", 2024-09-13T11:26:16.450 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.5420000000000002e-06 2024-09-13T11:26:16.450 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.450 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.450 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:26:16.450 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:05.905584+0000", 2024-09-13T11:26:16.450 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.4819000000000004e-05 2024-09-13T11:26:16.450 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.450 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.451 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:26:16.451 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:05.905696+0000", 2024-09-13T11:26:16.451 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00011248599999999999 2024-09-13T11:26:16.451 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.451 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.451 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 11", 2024-09-13T11:26:16.451 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:05.905759+0000", 2024-09-13T11:26:16.451 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.2451000000000002e-05 2024-09-13T11:26:16.451 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.451 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.451 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:26:16.451 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:05.907584+0000", 2024-09-13T11:26:16.451 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0018250110000000001 2024-09-13T11:26:16.451 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.452 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.452 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:16.452 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:06.109588+0000", 2024-09-13T11:26:16.452 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.20200470300000001 2024-09-13T11:26:16.452 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.452 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.452 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:26:16.452 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:06.109637+0000", 2024-09-13T11:26:16.452 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.8439000000000002e-05 2024-09-13T11:26:16.452 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.452 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.452 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:26:16.452 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:06.109674+0000", 2024-09-13T11:26:16.453 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.7292999999999997e-05 2024-09-13T11:26:16.453 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:16.453 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:16.453 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:16.453 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.453 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.453 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:150414 3.11 3:8f72eed4:::benchmark_data_smithi177_31116_object10111:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e149)", 2024-09-13T11:26:16.453 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:25:06.348925+0000", 2024-09-13T11:26:16.453 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 70.049879204999996, 2024-09-13T11:26:16.453 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.17250813300000001, 2024-09-13T11:26:16.453 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:26:16.453 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:26:16.453 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:16.454 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:26:16.454 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:26:16.454 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:16.454 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 150414 2024-09-13T11:26:16.454 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.454 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:26:16.454 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.454 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:26:16.454 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:06.348925+0000", 2024-09-13T11:26:16.454 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:16.454 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.454 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.454 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:26:16.455 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:06.348925+0000", 2024-09-13T11:26:16.455 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:16.455 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.455 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.455 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:26:16.455 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:06.348927+0000", 2024-09-13T11:26:16.455 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.7430000000000001e-06 2024-09-13T11:26:16.455 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.455 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.455 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:26:16.455 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:06.348931+0000", 2024-09-13T11:26:16.455 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.5210000000000004e-06 2024-09-13T11:26:16.455 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.455 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.456 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:26:16.456 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:06.348943+0000", 2024-09-13T11:26:16.456 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.1168e-05 2024-09-13T11:26:16.456 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.456 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.456 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:26:16.456 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:06.348950+0000", 2024-09-13T11:26:16.456 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.1430000000000001e-06 2024-09-13T11:26:16.456 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.456 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.456 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:26:16.456 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:06.349032+0000", 2024-09-13T11:26:16.456 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.2517999999999998e-05 2024-09-13T11:26:16.457 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.457 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.457 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:26:16.457 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:06.349179+0000", 2024-09-13T11:26:16.457 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000146935 2024-09-13T11:26:16.457 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.457 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.457 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 11", 2024-09-13T11:26:16.457 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:06.349290+0000", 2024-09-13T11:26:16.457 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000110927 2024-09-13T11:26:16.457 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.457 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.457 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:26:16.458 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:06.351686+0000", 2024-09-13T11:26:16.458 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0023963560000000001 2024-09-13T11:26:16.458 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.458 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.458 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:16.458 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:06.521374+0000", 2024-09-13T11:26:16.458 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.169687216 2024-09-13T11:26:16.458 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.458 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.458 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:26:16.458 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:06.521406+0000", 2024-09-13T11:26:16.458 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.1841000000000002e-05 2024-09-13T11:26:16.458 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.458 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.459 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:26:16.459 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:06.521433+0000", 2024-09-13T11:26:16.459 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.7764999999999999e-05 2024-09-13T11:26:16.459 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:16.459 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:16.459 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:16.459 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.459 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.459 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:168772 3.1a 3:5be647e9:::benchmark_data_smithi177_31116_object28469:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e161)", 2024-09-13T11:26:16.459 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:25:20.636095+0000", 2024-09-13T11:26:16.459 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 55.762709796000003, 2024-09-13T11:26:16.459 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.17968553000000001, 2024-09-13T11:26:16.459 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:26:16.459 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:26:16.460 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:16.460 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:26:16.460 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:26:16.460 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:16.460 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 168772 2024-09-13T11:26:16.460 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.460 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:26:16.460 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.460 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:26:16.460 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:20.636095+0000", 2024-09-13T11:26:16.460 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:16.460 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.460 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.461 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:26:16.461 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:20.636095+0000", 2024-09-13T11:26:16.461 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:16.461 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.461 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.461 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:26:16.461 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:20.636096+0000", 2024-09-13T11:26:16.461 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.852e-06 2024-09-13T11:26:16.461 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.461 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.461 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:26:16.461 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:20.636101+0000", 2024-09-13T11:26:16.461 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.0329999999999999e-06 2024-09-13T11:26:16.461 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.462 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.462 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:26:16.462 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:20.636114+0000", 2024-09-13T11:26:16.462 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2816999999999999e-05 2024-09-13T11:26:16.462 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.462 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.462 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:26:16.462 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:20.636120+0000", 2024-09-13T11:26:16.462 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.7429999999999999e-06 2024-09-13T11:26:16.462 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.462 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.462 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:26:16.462 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:20.636194+0000", 2024-09-13T11:26:16.463 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.3520000000000001e-05 2024-09-13T11:26:16.463 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.463 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.463 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:26:16.463 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:20.636369+0000", 2024-09-13T11:26:16.463 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00017552 2024-09-13T11:26:16.463 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.463 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.463 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 9", 2024-09-13T11:26:16.463 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:20.636519+0000", 2024-09-13T11:26:16.463 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000149949 2024-09-13T11:26:16.463 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.463 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.463 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:26:16.464 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:20.638735+0000", 2024-09-13T11:26:16.464 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0022156509999999999 2024-09-13T11:26:16.464 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.464 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.464 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:16.464 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:20.815699+0000", 2024-09-13T11:26:16.464 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.176964343 2024-09-13T11:26:16.464 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.464 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.464 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:26:16.464 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:20.815748+0000", 2024-09-13T11:26:16.464 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.9518000000000003e-05 2024-09-13T11:26:16.464 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.465 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.465 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:26:16.465 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:20.815780+0000", 2024-09-13T11:26:16.465 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.1584000000000001e-05 2024-09-13T11:26:16.465 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:16.465 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:16.465 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:16.465 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.465 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.465 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:168782 3.1a 3:590b82e1:::benchmark_data_smithi177_31116_object28479:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e161)", 2024-09-13T11:26:16.465 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:25:20.641567+0000", 2024-09-13T11:26:16.465 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 55.757236964000001, 2024-09-13T11:26:16.465 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.19100501, 2024-09-13T11:26:16.465 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:26:16.466 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:26:16.466 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:16.466 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:26:16.466 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:26:16.466 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:16.466 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 168782 2024-09-13T11:26:16.466 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.466 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:26:16.466 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.466 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:26:16.466 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:20.641567+0000", 2024-09-13T11:26:16.466 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:16.466 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.467 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.467 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:26:16.467 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:20.641567+0000", 2024-09-13T11:26:16.467 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:16.467 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.467 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.467 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:26:16.467 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:20.641569+0000", 2024-09-13T11:26:16.467 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.869e-06 2024-09-13T11:26:16.467 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.467 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.467 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:26:16.467 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:20.641574+0000", 2024-09-13T11:26:16.467 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.1680000000000003e-06 2024-09-13T11:26:16.468 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.468 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.468 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:26:16.468 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:20.641587+0000", 2024-09-13T11:26:16.468 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2858e-05 2024-09-13T11:26:16.468 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.468 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.468 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:26:16.468 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:20.641595+0000", 2024-09-13T11:26:16.468 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.6149999999999999e-06 2024-09-13T11:26:16.468 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.468 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.468 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:26:16.469 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:20.641655+0000", 2024-09-13T11:26:16.469 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.9673999999999998e-05 2024-09-13T11:26:16.469 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.469 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.469 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:26:16.469 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:20.641748+0000", 2024-09-13T11:26:16.469 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.3919999999999995e-05 2024-09-13T11:26:16.469 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.469 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.469 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 9", 2024-09-13T11:26:16.469 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:20.642990+0000", 2024-09-13T11:26:16.469 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0012415709999999999 2024-09-13T11:26:16.469 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.469 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.470 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:26:16.470 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:20.645016+0000", 2024-09-13T11:26:16.470 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.002025448 2024-09-13T11:26:16.470 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.470 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.470 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:16.470 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:20.832409+0000", 2024-09-13T11:26:16.470 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.18739319600000001 2024-09-13T11:26:16.470 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.470 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.470 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:26:16.470 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:20.832525+0000", 2024-09-13T11:26:16.470 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000116273 2024-09-13T11:26:16.470 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.471 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.471 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:26:16.471 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:20.832572+0000", 2024-09-13T11:26:16.471 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.7417999999999999e-05 2024-09-13T11:26:16.471 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:16.471 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:16.471 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:16.471 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.471 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.471 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop_reply(client.4665.0:168772 3.1a e161/105 ondisk, result = 0)", 2024-09-13T11:26:16.471 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:25:20.645700+0000", 2024-09-13T11:26:16.471 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 55.753104782999998, 2024-09-13T11:26:16.471 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.17008606200000001, 2024-09-13T11:26:16.472 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:26:16.472 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:26:16.472 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:26:16.472 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:26:16.472 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.472 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:26:16.472 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:20.645700+0000", 2024-09-13T11:26:16.472 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:16.472 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.472 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.472 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:26:16.472 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:20.645700+0000", 2024-09-13T11:26:16.472 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:16.473 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.473 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.473 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:26:16.473 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:20.645701+0000", 2024-09-13T11:26:16.473 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.474e-06 2024-09-13T11:26:16.473 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.473 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.473 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:26:16.473 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:20.645714+0000", 2024-09-13T11:26:16.473 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2931999999999999e-05 2024-09-13T11:26:16.473 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.473 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.473 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:26:16.473 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:20.815603+0000", 2024-09-13T11:26:16.474 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.16988872199999999 2024-09-13T11:26:16.474 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.474 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.474 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:26:16.474 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:20.815610+0000", 2024-09-13T11:26:16.474 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.2139999999999999e-06 2024-09-13T11:26:16.474 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.474 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.474 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:26:16.474 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:20.815689+0000", 2024-09-13T11:26:16.474 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.9302999999999996e-05 2024-09-13T11:26:16.474 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.474 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.475 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:26:16.475 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:20.815695+0000", 2024-09-13T11:26:16.475 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.818e-06 2024-09-13T11:26:16.475 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.475 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.475 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:26:16.475 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:20.815786+0000", 2024-09-13T11:26:16.475 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.0599000000000005e-05 2024-09-13T11:26:16.475 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:16.475 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:16.475 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:16.475 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.475 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.475 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:174971 3.1a 3:5a8f64c5:::benchmark_data_smithi177_31116_object34668:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e166)", 2024-09-13T11:26:16.476 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:25:25.422480+0000", 2024-09-13T11:26:16.476 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 50.976324353000003, 2024-09-13T11:26:16.476 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.18399536399999999, 2024-09-13T11:26:16.476 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:26:16.476 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:26:16.476 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:16.476 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:26:16.476 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:26:16.476 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:16.476 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 174971 2024-09-13T11:26:16.476 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.476 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:26:16.476 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.477 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:26:16.477 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:25.422480+0000", 2024-09-13T11:26:16.477 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:16.477 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.477 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.477 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:26:16.477 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:25.422480+0000", 2024-09-13T11:26:16.477 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:16.477 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.477 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.477 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:26:16.477 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:25.422483+0000", 2024-09-13T11:26:16.477 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.4529999999999998e-06 2024-09-13T11:26:16.477 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.478 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.478 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:26:16.478 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:25.422492+0000", 2024-09-13T11:26:16.478 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.8820000000000007e-06 2024-09-13T11:26:16.478 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.478 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.478 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:26:16.478 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:25.422515+0000", 2024-09-13T11:26:16.478 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.2756e-05 2024-09-13T11:26:16.478 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.478 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.478 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:26:16.478 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:25.422526+0000", 2024-09-13T11:26:16.478 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0684e-05 2024-09-13T11:26:16.479 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.479 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.479 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:26:16.479 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:25.422590+0000", 2024-09-13T11:26:16.479 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.4163000000000002e-05 2024-09-13T11:26:16.479 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.479 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.479 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:26:16.479 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:25.422675+0000", 2024-09-13T11:26:16.479 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.5311999999999999e-05 2024-09-13T11:26:16.479 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.479 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.479 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 9", 2024-09-13T11:26:16.480 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:25.422745+0000", 2024-09-13T11:26:16.480 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.0217999999999998e-05 2024-09-13T11:26:16.480 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.480 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.480 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:26:16.480 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:25.424910+0000", 2024-09-13T11:26:16.480 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0021641500000000001 2024-09-13T11:26:16.480 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.480 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.480 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:16.480 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:25.606366+0000", 2024-09-13T11:26:16.480 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.181456267 2024-09-13T11:26:16.480 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.480 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.481 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:26:16.481 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:25.606428+0000", 2024-09-13T11:26:16.481 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.2312999999999998e-05 2024-09-13T11:26:16.481 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.481 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.481 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:26:16.481 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:25.606475+0000", 2024-09-13T11:26:16.481 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.7166000000000002e-05 2024-09-13T11:26:16.481 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:16.481 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:16.481 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:16.481 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.481 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.482 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4665.0:191081 3.1e e175/102 3:7e784823:::benchmark_data_smithi177_31116_object50778:head v 175'7812, mlcod=175'7809)", 2024-09-13T11:26:16.482 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:25:38.547539+0000", 2024-09-13T11:26:16.482 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 37.851265067, 2024-09-13T11:26:16.482 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.213922061, 2024-09-13T11:26:16.482 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:26:16.482 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:26:16.482 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:16.482 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:26:16.482 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.482 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:26:16.482 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:38.547539+0000", 2024-09-13T11:26:16.482 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:16.482 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.482 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.483 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:26:16.483 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:38.547539+0000", 2024-09-13T11:26:16.483 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:16.483 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.483 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.483 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:26:16.483 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:38.547543+0000", 2024-09-13T11:26:16.483 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.3629999999999998e-06 2024-09-13T11:26:16.483 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.483 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.483 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:26:16.483 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:38.547558+0000", 2024-09-13T11:26:16.483 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.5043e-05 2024-09-13T11:26:16.483 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.484 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.484 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:26:16.484 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:38.756778+0000", 2024-09-13T11:26:16.484 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.209219936 2024-09-13T11:26:16.484 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.484 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.484 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:26:16.484 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:38.756790+0000", 2024-09-13T11:26:16.484 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2775999999999999e-05 2024-09-13T11:26:16.484 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.484 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.484 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:26:16.484 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:38.756934+0000", 2024-09-13T11:26:16.484 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00014401600000000001 2024-09-13T11:26:16.485 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.485 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.485 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:26:16.485 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:38.757001+0000", 2024-09-13T11:26:16.485 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.6716000000000003e-05 2024-09-13T11:26:16.485 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.485 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.485 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:26:16.485 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:38.761376+0000", 2024-09-13T11:26:16.485 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0043747320000000001 2024-09-13T11:26:16.485 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.485 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.485 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:26:16.486 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:38.761461+0000", 2024-09-13T11:26:16.486 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.5478999999999999e-05 2024-09-13T11:26:16.486 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:16.486 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:16.486 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:16.486 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.486 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.486 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4665.0:191087 3.1e e175/102 3:7a697202:::benchmark_data_smithi177_31116_object50784:head v 175'7813, mlcod=175'7809)", 2024-09-13T11:26:16.486 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:25:38.561827+0000", 2024-09-13T11:26:16.486 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 37.836976857000003, 2024-09-13T11:26:16.486 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.209395781, 2024-09-13T11:26:16.486 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:26:16.486 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:26:16.487 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:16.487 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:26:16.487 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.487 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:26:16.487 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:38.561827+0000", 2024-09-13T11:26:16.487 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:16.487 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.487 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.487 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:26:16.487 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:38.561827+0000", 2024-09-13T11:26:16.487 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:16.487 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.487 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.487 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:26:16.488 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:38.561837+0000", 2024-09-13T11:26:16.488 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.2329999999999995e-06 2024-09-13T11:26:16.488 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.488 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.488 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:26:16.488 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:38.561849+0000", 2024-09-13T11:26:16.488 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2593e-05 2024-09-13T11:26:16.488 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.488 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.488 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:26:16.488 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:38.766804+0000", 2024-09-13T11:26:16.488 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.20495488000000001 2024-09-13T11:26:16.488 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.489 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.489 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:26:16.489 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:38.766823+0000", 2024-09-13T11:26:16.489 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.9143e-05 2024-09-13T11:26:16.489 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.489 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.489 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:26:16.489 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:38.766973+0000", 2024-09-13T11:26:16.489 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000149226 2024-09-13T11:26:16.489 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.489 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.489 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:26:16.489 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:38.767014+0000", 2024-09-13T11:26:16.489 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.1183000000000002e-05 2024-09-13T11:26:16.490 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.490 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.490 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:26:16.490 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:38.771145+0000", 2024-09-13T11:26:16.490 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0041315270000000003 2024-09-13T11:26:16.490 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.490 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.490 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:26:16.490 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:38.771223+0000", 2024-09-13T11:26:16.490 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.7996000000000003e-05 2024-09-13T11:26:16.490 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:16.490 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:16.490 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:16.490 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.491 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.491 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:191672 3.11 3:8b81ea05:::benchmark_data_smithi177_31116_object51369:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e176)", 2024-09-13T11:26:16.491 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:25:39.675914+0000", 2024-09-13T11:26:16.491 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 36.722890812000003, 2024-09-13T11:26:16.491 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.165205352, 2024-09-13T11:26:16.491 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:26:16.491 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:26:16.491 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:16.491 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:26:16.491 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:26:16.491 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:16.491 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 191672 2024-09-13T11:26:16.491 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.491 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:26:16.492 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.492 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:26:16.492 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:39.675914+0000", 2024-09-13T11:26:16.492 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:16.492 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.492 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.492 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:26:16.492 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:39.675914+0000", 2024-09-13T11:26:16.492 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:16.492 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.492 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.492 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:26:16.492 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:39.675916+0000", 2024-09-13T11:26:16.493 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.8059999999999999e-06 2024-09-13T11:26:16.493 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.493 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.493 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:26:16.493 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:39.675923+0000", 2024-09-13T11:26:16.493 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.3489999999999997e-06 2024-09-13T11:26:16.493 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.493 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.493 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:26:16.493 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:39.675940+0000", 2024-09-13T11:26:16.493 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.7489000000000001e-05 2024-09-13T11:26:16.493 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.493 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.494 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:26:16.494 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:39.675948+0000", 2024-09-13T11:26:16.494 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.0690000000000002e-06 2024-09-13T11:26:16.494 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.494 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.494 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:26:16.494 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:39.676017+0000", 2024-09-13T11:26:16.494 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.8714000000000007e-05 2024-09-13T11:26:16.494 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.494 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.494 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:26:16.494 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:39.676142+0000", 2024-09-13T11:26:16.494 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000125535 2024-09-13T11:26:16.494 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.495 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.495 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 11", 2024-09-13T11:26:16.495 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:39.676236+0000", 2024-09-13T11:26:16.495 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.3249000000000001e-05 2024-09-13T11:26:16.495 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.495 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.495 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:26:16.495 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:39.679269+0000", 2024-09-13T11:26:16.495 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0030331780000000001 2024-09-13T11:26:16.495 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.495 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.495 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:16.495 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:39.841008+0000", 2024-09-13T11:26:16.495 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.16173915799999999 2024-09-13T11:26:16.496 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.496 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.496 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:26:16.496 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:39.841068+0000", 2024-09-13T11:26:16.496 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.9579e-05 2024-09-13T11:26:16.496 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.496 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.496 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:26:16.496 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:39.841119+0000", 2024-09-13T11:26:16.496 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.1226e-05 2024-09-13T11:26:16.496 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:16.496 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:16.496 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:16.497 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.497 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.497 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:202145 3.1a 3:5fe5b9e3:::benchmark_data_smithi177_31116_object61842:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e182)", 2024-09-13T11:26:16.497 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:25:48.598830+0000", 2024-09-13T11:26:16.497 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 27.799974208999998, 2024-09-13T11:26:16.497 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.16940577600000001, 2024-09-13T11:26:16.497 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:26:16.497 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:26:16.497 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:16.497 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:26:16.497 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:26:16.497 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:16.497 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 202145 2024-09-13T11:26:16.497 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.498 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:26:16.498 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.498 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:26:16.498 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:48.598830+0000", 2024-09-13T11:26:16.498 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:16.498 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.498 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.498 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:26:16.498 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:48.598830+0000", 2024-09-13T11:26:16.498 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:16.498 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.498 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.498 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:26:16.499 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:48.598833+0000", 2024-09-13T11:26:16.499 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.79e-06 2024-09-13T11:26:16.499 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.499 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.499 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:26:16.499 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:48.598840+0000", 2024-09-13T11:26:16.499 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.5100000000000001e-06 2024-09-13T11:26:16.499 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.499 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.499 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:26:16.499 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:48.598858+0000", 2024-09-13T11:26:16.499 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.7294000000000001e-05 2024-09-13T11:26:16.499 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.499 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.500 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:26:16.500 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:48.598865+0000", 2024-09-13T11:26:16.500 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.7219999999999999e-06 2024-09-13T11:26:16.500 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.500 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.500 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:26:16.500 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:48.598931+0000", 2024-09-13T11:26:16.500 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.5734000000000005e-05 2024-09-13T11:26:16.500 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.500 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.500 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:26:16.500 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:48.599037+0000", 2024-09-13T11:26:16.500 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000105806 2024-09-13T11:26:16.500 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.501 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.501 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 9", 2024-09-13T11:26:16.501 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:48.599110+0000", 2024-09-13T11:26:16.501 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.3464e-05 2024-09-13T11:26:16.501 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.501 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.501 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:26:16.501 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:48.601703+0000", 2024-09-13T11:26:16.501 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0025923729999999998 2024-09-13T11:26:16.501 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.501 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.501 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:16.501 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:48.768165+0000", 2024-09-13T11:26:16.502 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.16646239400000001 2024-09-13T11:26:16.502 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.502 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.502 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:26:16.502 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:48.768198+0000", 2024-09-13T11:26:16.502 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.2568999999999999e-05 2024-09-13T11:26:16.502 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.502 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.502 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:26:16.502 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:48.768236+0000", 2024-09-13T11:26:16.502 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.8120000000000001e-05 2024-09-13T11:26:16.502 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:16.502 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:16.502 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:16.503 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.503 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.503 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop_reply(client.4665.0:202145 3.1a e182/105 ondisk, result = 0)", 2024-09-13T11:26:16.503 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:25:48.602776+0000", 2024-09-13T11:26:16.503 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 27.796027851000002, 2024-09-13T11:26:16.503 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.16546329100000001, 2024-09-13T11:26:16.503 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:26:16.503 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:26:16.503 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:26:16.503 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:26:16.503 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.503 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:26:16.503 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:48.602776+0000", 2024-09-13T11:26:16.503 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:16.504 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.504 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.504 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:26:16.504 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:48.602776+0000", 2024-09-13T11:26:16.504 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:16.504 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.504 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.504 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:26:16.504 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:48.602778+0000", 2024-09-13T11:26:16.504 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.195e-06 2024-09-13T11:26:16.504 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.504 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.504 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:26:16.505 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:48.602785+0000", 2024-09-13T11:26:16.505 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.4869999999999996e-06 2024-09-13T11:26:16.505 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.505 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.505 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:26:16.505 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:48.768090+0000", 2024-09-13T11:26:16.505 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.165304597 2024-09-13T11:26:16.505 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.505 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.505 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:26:16.505 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:48.768099+0000", 2024-09-13T11:26:16.505 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.3449999999999995e-06 2024-09-13T11:26:16.505 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.505 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.506 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:26:16.506 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:48.768157+0000", 2024-09-13T11:26:16.506 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.7479999999999999e-05 2024-09-13T11:26:16.506 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.506 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.506 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:26:16.506 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:48.768162+0000", 2024-09-13T11:26:16.506 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.3890000000000004e-06 2024-09-13T11:26:16.506 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.506 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.506 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:26:16.506 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:48.768240+0000", 2024-09-13T11:26:16.506 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.7798000000000003e-05 2024-09-13T11:26:16.506 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:16.507 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:16.507 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:16.507 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:16.507 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:16.507 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-13T11:26:16.508 INFO:tasks.ceph.osd.10.smithi162.stderr:2024-09-13T11:26:16.503+0000 7fb59c9f0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26317) UID: 0 2024-09-13T11:26:16.509 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-09-13T11:26:16.609 INFO:tasks.ceph.osd.10.smithi162.stderr:2024-09-13T11:26:16.607+0000 7fb59c9f0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26317) UID: 0 2024-09-13T11:26:16.651 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-13T11:26:16.651 INFO:teuthology.orchestra.run.smithi001.stdout: "ops": [ 2024-09-13T11:26:16.651 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.651 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:233663 3.26 3:67af7a71:::benchmark_data_smithi177_31116_object93360:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e200)", 2024-09-13T11:26:16.651 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:26:16.613407+0000", 2024-09-13T11:26:16.651 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 0.037478279000000003, 2024-09-13T11:26:16.651 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.037500804999999998, 2024-09-13T11:26:16.651 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:26:16.651 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:26:16.651 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "waiting for sub ops", 2024-09-13T11:26:16.652 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:26:16.652 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:26:16.652 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:16.652 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 233663 2024-09-13T11:26:16.652 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.652 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:26:16.652 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.652 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:26:16.652 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:16.613407+0000", 2024-09-13T11:26:16.652 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:16.652 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.652 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.652 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:26:16.652 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:16.613407+0000", 2024-09-13T11:26:16.653 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:16.653 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.653 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.653 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:26:16.653 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:16.613409+0000", 2024-09-13T11:26:16.653 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.8220000000000001e-06 2024-09-13T11:26:16.653 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.653 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.653 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:26:16.653 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:16.613414+0000", 2024-09-13T11:26:16.653 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.8049999999999997e-06 2024-09-13T11:26:16.653 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.653 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.653 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:26:16.654 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:16.613426+0000", 2024-09-13T11:26:16.654 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2319e-05 2024-09-13T11:26:16.654 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.654 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.654 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:26:16.654 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:16.613432+0000", 2024-09-13T11:26:16.654 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.604e-06 2024-09-13T11:26:16.654 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.654 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.654 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:26:16.654 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:16.613490+0000", 2024-09-13T11:26:16.654 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.7742999999999999e-05 2024-09-13T11:26:16.654 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.654 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.654 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:26:16.655 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:16.613614+0000", 2024-09-13T11:26:16.655 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00012410899999999999 2024-09-13T11:26:16.655 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.655 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.655 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 12", 2024-09-13T11:26:16.655 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:16.613672+0000", 2024-09-13T11:26:16.655 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.8053999999999997e-05 2024-09-13T11:26:16.655 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.655 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.655 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:26:16.655 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:16.615571+0000", 2024-09-13T11:26:16.655 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0018996830000000001 2024-09-13T11:26:16.655 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:16.655 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:16.655 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:16.655 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:16.656 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-09-13T11:26:16.656 INFO:teuthology.orchestra.run.smithi001.stdout: "num_ops": 1 2024-09-13T11:26:16.656 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-13T11:26:16.666 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-09-13T11:26:16.710 INFO:tasks.ceph.osd.11.smithi177.stderr:2024-09-13T11:26:16.709+0000 7f24792c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26580) UID: 0 2024-09-13T11:26:16.782 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-13T11:26:16.782 INFO:teuthology.orchestra.run.smithi001.stdout: "ops": [], 2024-09-13T11:26:16.782 INFO:teuthology.orchestra.run.smithi001.stdout: "complaint_time": 30, 2024-09-13T11:26:16.782 INFO:teuthology.orchestra.run.smithi001.stdout: "num_blocked_ops": 0 2024-09-13T11:26:16.783 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-13T11:26:16.797 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-09-13T11:26:16.811 INFO:tasks.ceph.osd.7.smithi177.stderr:2024-09-13T11:26:16.809+0000 7f897b51d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26575) UID: 0 2024-09-13T11:26:16.912 INFO:tasks.ceph.osd.1.smithi092.stderr:2024-09-13T11:26:16.908+0000 7fe7789c5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33718) UID: 0 2024-09-13T11:26:16.932 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-13T11:26:16.933 INFO:teuthology.orchestra.run.smithi001.stdout: "size": 20, 2024-09-13T11:26:16.933 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 600, 2024-09-13T11:26:16.933 INFO:teuthology.orchestra.run.smithi001.stdout: "ops": [ 2024-09-13T11:26:16.933 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.933 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:54656 3.1e 3.6a49163e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e75)", 2024-09-13T11:26:16.933 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:30.979993+0000", 2024-09-13T11:26:16.933 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 165.949659454, 2024-09-13T11:26:16.933 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.9269557150000001, 2024-09-13T11:26:16.933 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:26:16.933 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:26:16.933 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-13T11:26:16.933 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:26:16.933 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:26:16.934 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:16.934 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 54656 2024-09-13T11:26:16.934 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.934 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:26:16.934 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.934 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:26:16.934 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:30.979993+0000", 2024-09-13T11:26:16.934 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:16.934 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.934 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.934 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:26:16.934 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:30.979993+0000", 2024-09-13T11:26:16.934 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:16.935 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.935 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.935 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:26:16.935 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:30.979997+0000", 2024-09-13T11:26:16.935 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.0319999999999997e-06 2024-09-13T11:26:16.935 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.935 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.935 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:26:16.935 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:30.980057+0000", 2024-09-13T11:26:16.935 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.0341999999999999e-05 2024-09-13T11:26:16.935 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.935 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.935 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:26:16.936 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:30.980094+0000", 2024-09-13T11:26:16.936 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.6770000000000002e-05 2024-09-13T11:26:16.936 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.936 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.936 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:26:16.936 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:30.980113+0000", 2024-09-13T11:26:16.936 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.8791999999999999e-05 2024-09-13T11:26:16.936 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.936 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.936 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:26:16.936 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.097773+0000", 2024-09-13T11:26:16.936 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.117660379 2024-09-13T11:26:16.936 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.936 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.937 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for peered", 2024-09-13T11:26:16.937 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.097784+0000", 2024-09-13T11:26:16.937 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.1025e-05 2024-09-13T11:26:16.937 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.937 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.937 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:26:16.937 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.906937+0000", 2024-09-13T11:26:16.937 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.8091523789999999 2024-09-13T11:26:16.937 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.937 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.937 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:26:16.937 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.906949+0000", 2024-09-13T11:26:16.937 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.1996e-05 2024-09-13T11:26:16.937 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:16.938 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:16.938 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:16.938 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.938 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.938 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:54720 3.1e 3.a944303e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-13T11:26:16.938 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:31.973315+0000", 2024-09-13T11:26:16.938 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 164.95633721900001, 2024-09-13T11:26:16.938 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.93366527200000005, 2024-09-13T11:26:16.938 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:26:16.938 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:26:16.938 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-13T11:26:16.938 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:26:16.938 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:26:16.938 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:16.939 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 54720 2024-09-13T11:26:16.939 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.939 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:26:16.939 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.939 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:26:16.939 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.973315+0000", 2024-09-13T11:26:16.939 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:16.939 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.940 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.940 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:26:16.940 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.973315+0000", 2024-09-13T11:26:16.940 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:16.940 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.940 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.940 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:26:16.940 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.973318+0000", 2024-09-13T11:26:16.940 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.5830000000000001e-06 2024-09-13T11:26:16.940 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.940 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.940 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:26:16.940 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.973465+0000", 2024-09-13T11:26:16.941 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000146816 2024-09-13T11:26:16.941 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.941 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.941 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:26:16.941 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.973478+0000", 2024-09-13T11:26:16.941 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2931e-05 2024-09-13T11:26:16.941 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.941 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.941 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:26:16.941 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.973483+0000", 2024-09-13T11:26:16.941 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.8969999999999996e-06 2024-09-13T11:26:16.941 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.941 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.941 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:26:16.942 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.973548+0000", 2024-09-13T11:26:16.942 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.4771999999999996e-05 2024-09-13T11:26:16.942 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.942 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.942 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for peered", 2024-09-13T11:26:16.942 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.973557+0000", 2024-09-13T11:26:16.942 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.0359999999999995e-06 2024-09-13T11:26:16.942 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.942 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.942 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:26:16.942 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.906974+0000", 2024-09-13T11:26:16.942 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.93341706000000002 2024-09-13T11:26:16.942 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.943 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.943 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:26:16.943 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.906980+0000", 2024-09-13T11:26:16.943 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.1770000000000003e-06 2024-09-13T11:26:16.943 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:16.943 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:16.943 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:16.943 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.943 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.943 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:54734 3.1e 3.d7f6f7e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-13T11:26:16.943 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:31.983342+0000", 2024-09-13T11:26:16.943 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 164.94631041599999, 2024-09-13T11:26:16.943 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.92367058599999996, 2024-09-13T11:26:16.944 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:26:16.944 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:26:16.944 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-13T11:26:16.944 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:26:16.944 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:26:16.944 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:16.944 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 54734 2024-09-13T11:26:16.944 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.944 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:26:16.944 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.944 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:26:16.944 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.983342+0000", 2024-09-13T11:26:16.944 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:16.945 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.945 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.945 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:26:16.945 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.983342+0000", 2024-09-13T11:26:16.945 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:16.945 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.945 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.945 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:26:16.945 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.983345+0000", 2024-09-13T11:26:16.945 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.6220000000000001e-06 2024-09-13T11:26:16.945 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.945 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.945 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:26:16.946 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.983438+0000", 2024-09-13T11:26:16.946 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.3517000000000006e-05 2024-09-13T11:26:16.946 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.946 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.946 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:26:16.946 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.983451+0000", 2024-09-13T11:26:16.946 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2527999999999999e-05 2024-09-13T11:26:16.946 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.946 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.946 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:26:16.946 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.983458+0000", 2024-09-13T11:26:16.946 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.6380000000000004e-06 2024-09-13T11:26:16.946 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.946 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.947 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:26:16.947 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.983521+0000", 2024-09-13T11:26:16.947 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.2729000000000002e-05 2024-09-13T11:26:16.947 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.947 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.947 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for peered", 2024-09-13T11:26:16.947 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.983548+0000", 2024-09-13T11:26:16.947 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.6537999999999999e-05 2024-09-13T11:26:16.947 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.947 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.947 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:26:16.947 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.907005+0000", 2024-09-13T11:26:16.947 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.923456991 2024-09-13T11:26:16.947 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.948 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.948 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:26:16.948 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.907013+0000", 2024-09-13T11:26:16.948 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.0229999999999992e-06 2024-09-13T11:26:16.948 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:16.948 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:16.948 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:16.948 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.948 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.948 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:54760 3.1e 3.8c32c13e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-13T11:26:16.948 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:31.999685+0000", 2024-09-13T11:26:16.948 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 164.92996718399999, 2024-09-13T11:26:16.948 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.90735639599999995, 2024-09-13T11:26:16.949 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:26:16.949 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:26:16.949 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-13T11:26:16.949 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:26:16.949 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:26:16.949 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:16.949 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 54760 2024-09-13T11:26:16.949 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.949 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:26:16.949 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.949 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:26:16.949 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.999685+0000", 2024-09-13T11:26:16.949 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:16.950 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.950 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.950 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:26:16.950 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.999685+0000", 2024-09-13T11:26:16.950 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:16.950 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.950 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.950 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:26:16.950 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.999690+0000", 2024-09-13T11:26:16.950 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.7879999999999997e-06 2024-09-13T11:26:16.950 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.950 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.950 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:26:16.951 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.999729+0000", 2024-09-13T11:26:16.951 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.9032000000000002e-05 2024-09-13T11:26:16.951 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.951 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.951 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:26:16.951 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.999741+0000", 2024-09-13T11:26:16.951 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2136999999999999e-05 2024-09-13T11:26:16.951 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.951 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.951 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:26:16.951 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.999748+0000", 2024-09-13T11:26:16.951 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.9120000000000001e-06 2024-09-13T11:26:16.951 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.951 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.952 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:26:16.952 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.999803+0000", 2024-09-13T11:26:16.952 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.5064e-05 2024-09-13T11:26:16.952 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.952 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.952 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for peered", 2024-09-13T11:26:16.952 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.999815+0000", 2024-09-13T11:26:16.952 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2041e-05 2024-09-13T11:26:16.952 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.952 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.952 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:26:16.952 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.907034+0000", 2024-09-13T11:26:16.952 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.90721858200000005 2024-09-13T11:26:16.953 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.953 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.953 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:26:16.953 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.907042+0000", 2024-09-13T11:26:16.953 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.8399999999999995e-06 2024-09-13T11:26:16.953 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:16.953 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:16.953 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:16.953 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.953 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.953 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:54768 3.1e 3.6dad499e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-13T11:26:16.953 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:32.003482+0000", 2024-09-13T11:26:16.953 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 164.92617053399999, 2024-09-13T11:26:16.953 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.90358804699999995, 2024-09-13T11:26:16.954 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:26:16.954 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:26:16.954 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-13T11:26:16.954 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:26:16.954 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:26:16.954 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:16.954 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 54768 2024-09-13T11:26:16.954 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.954 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:26:16.954 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.954 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:26:16.954 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.003482+0000", 2024-09-13T11:26:16.954 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:16.954 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.955 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.955 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:26:16.955 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.003482+0000", 2024-09-13T11:26:16.955 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:16.955 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.955 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.955 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:26:16.955 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.003486+0000", 2024-09-13T11:26:16.955 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.1030000000000004e-06 2024-09-13T11:26:16.955 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.955 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.955 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:26:16.955 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.003523+0000", 2024-09-13T11:26:16.956 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.6732e-05 2024-09-13T11:26:16.956 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.956 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.956 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:26:16.956 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.003535+0000", 2024-09-13T11:26:16.956 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2520999999999999e-05 2024-09-13T11:26:16.956 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.956 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.956 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:26:16.956 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.003541+0000", 2024-09-13T11:26:16.956 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.8690000000000001e-06 2024-09-13T11:26:16.956 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.956 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.957 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:26:16.957 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.003593+0000", 2024-09-13T11:26:16.957 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.1786000000000002e-05 2024-09-13T11:26:16.957 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.957 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.957 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for peered", 2024-09-13T11:26:16.957 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.003602+0000", 2024-09-13T11:26:16.957 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.9679999999999995e-06 2024-09-13T11:26:16.957 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.957 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.957 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:26:16.957 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.907063+0000", 2024-09-13T11:26:16.957 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.90346116600000004 2024-09-13T11:26:16.957 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.958 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.958 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:26:16.958 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.907070+0000", 2024-09-13T11:26:16.958 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.9020000000000002e-06 2024-09-13T11:26:16.958 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:16.958 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:16.958 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:16.958 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.958 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.958 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:54785 3.1e 3.d32f1ede (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-13T11:26:16.958 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:32.016238+0000", 2024-09-13T11:26:16.958 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 164.913414324, 2024-09-13T11:26:16.958 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.89086243399999998, 2024-09-13T11:26:16.959 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:26:16.959 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:26:16.959 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-13T11:26:16.959 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:26:16.959 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:26:16.959 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:16.959 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 54785 2024-09-13T11:26:16.959 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.959 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:26:16.959 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.959 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:26:16.960 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.016238+0000", 2024-09-13T11:26:16.960 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:16.960 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.960 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.960 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:26:16.960 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.016238+0000", 2024-09-13T11:26:16.960 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:16.960 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.960 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.960 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:26:16.960 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.016241+0000", 2024-09-13T11:26:16.960 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.6469999999999998e-06 2024-09-13T11:26:16.960 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.960 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.961 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:26:16.961 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.016271+0000", 2024-09-13T11:26:16.961 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.0505999999999999e-05 2024-09-13T11:26:16.961 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.961 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.961 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:26:16.961 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.016287+0000", 2024-09-13T11:26:16.961 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.6073999999999999e-05 2024-09-13T11:26:16.961 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.961 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.961 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:26:16.961 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.016295+0000", 2024-09-13T11:26:16.961 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.0590000000000003e-06 2024-09-13T11:26:16.962 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.962 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.962 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:26:16.962 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.016373+0000", 2024-09-13T11:26:16.962 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.7634999999999998e-05 2024-09-13T11:26:16.962 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.962 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.962 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for peered", 2024-09-13T11:26:16.962 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.016383+0000", 2024-09-13T11:26:16.962 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0246e-05 2024-09-13T11:26:16.962 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.962 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.962 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:26:16.962 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.907092+0000", 2024-09-13T11:26:16.963 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.89070849200000002 2024-09-13T11:26:16.963 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.963 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.963 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:26:16.963 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.907101+0000", 2024-09-13T11:26:16.963 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.7749999999999998e-06 2024-09-13T11:26:16.963 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:16.963 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:16.963 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:16.963 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.963 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.963 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:54811 3.1e 3.3754367e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-13T11:26:16.963 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:32.045424+0000", 2024-09-13T11:26:16.964 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 164.884228673, 2024-09-13T11:26:16.964 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.86170719399999995, 2024-09-13T11:26:16.964 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:26:16.964 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:26:16.964 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-13T11:26:16.964 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:26:16.964 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:26:16.964 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:16.964 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 54811 2024-09-13T11:26:16.964 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.964 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:26:16.964 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.964 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:26:16.964 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.045424+0000", 2024-09-13T11:26:16.965 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:16.965 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.965 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.965 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:26:16.965 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.045424+0000", 2024-09-13T11:26:16.965 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:16.965 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.965 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.965 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:26:16.965 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.045426+0000", 2024-09-13T11:26:16.965 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.0260000000000002e-06 2024-09-13T11:26:16.965 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.965 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.965 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:26:16.966 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.045556+0000", 2024-09-13T11:26:16.966 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00013003099999999999 2024-09-13T11:26:16.966 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.966 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.966 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:26:16.966 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.045568+0000", 2024-09-13T11:26:16.966 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.1727e-05 2024-09-13T11:26:16.966 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.966 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.966 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:26:16.966 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.045576+0000", 2024-09-13T11:26:16.966 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.6459999999999999e-06 2024-09-13T11:26:16.966 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.966 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.967 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:26:16.967 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.045650+0000", 2024-09-13T11:26:16.967 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.4000999999999998e-05 2024-09-13T11:26:16.967 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.967 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.967 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for peered", 2024-09-13T11:26:16.967 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.045658+0000", 2024-09-13T11:26:16.967 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.3869999999999997e-06 2024-09-13T11:26:16.967 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.967 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.967 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:26:16.967 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.907123+0000", 2024-09-13T11:26:16.967 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.86146508399999999 2024-09-13T11:26:16.967 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.968 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.968 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:26:16.968 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.907131+0000", 2024-09-13T11:26:16.968 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.2919999999999996e-06 2024-09-13T11:26:16.968 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:16.968 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:16.968 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:16.968 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.968 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.968 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:54817 3.1e 3.36dd2fde (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-13T11:26:16.968 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:32.050108+0000", 2024-09-13T11:26:16.968 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 164.87954478099999, 2024-09-13T11:26:16.968 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.85705549199999997, 2024-09-13T11:26:16.969 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:26:16.969 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:26:16.969 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-13T11:26:16.969 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:26:16.969 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:26:16.969 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:16.969 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 54817 2024-09-13T11:26:16.969 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.969 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:26:16.969 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.969 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:26:16.969 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.050108+0000", 2024-09-13T11:26:16.969 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:16.969 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.970 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.970 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:26:16.970 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.050108+0000", 2024-09-13T11:26:16.970 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:16.970 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.970 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.970 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:26:16.970 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.050110+0000", 2024-09-13T11:26:16.970 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.0209999999999998e-06 2024-09-13T11:26:16.970 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.970 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.970 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:26:16.970 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.050166+0000", 2024-09-13T11:26:16.970 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.6026000000000002e-05 2024-09-13T11:26:16.971 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.971 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.971 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:26:16.971 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.050180+0000", 2024-09-13T11:26:16.971 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.4307e-05 2024-09-13T11:26:16.971 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.971 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.971 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:26:16.971 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.050188+0000", 2024-09-13T11:26:16.971 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.5410000000000003e-06 2024-09-13T11:26:16.971 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.971 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.971 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:26:16.971 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.050260+0000", 2024-09-13T11:26:16.972 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.2157999999999996e-05 2024-09-13T11:26:16.972 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.972 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.972 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for peered", 2024-09-13T11:26:16.972 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.050267+0000", 2024-09-13T11:26:16.972 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.0190000000000001e-06 2024-09-13T11:26:16.972 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.972 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.972 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:26:16.972 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.907155+0000", 2024-09-13T11:26:16.972 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.85688824399999997 2024-09-13T11:26:16.972 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.972 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.973 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:26:16.973 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.907163+0000", 2024-09-13T11:26:16.973 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.1759999999999993e-06 2024-09-13T11:26:16.973 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:16.973 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:16.973 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:16.973 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.973 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.973 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:54818 3.1e 3.5cfcc69e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-13T11:26:16.973 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:32.051374+0000", 2024-09-13T11:26:16.973 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 164.878278365, 2024-09-13T11:26:16.973 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.85581781000000001, 2024-09-13T11:26:16.973 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:26:16.974 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:26:16.974 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-13T11:26:16.974 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:26:16.974 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:26:16.974 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:16.974 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 54818 2024-09-13T11:26:16.974 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.974 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:26:16.974 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.974 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:26:16.974 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.051374+0000", 2024-09-13T11:26:16.974 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:16.974 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.974 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.975 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:26:16.975 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.051374+0000", 2024-09-13T11:26:16.975 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:16.975 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.975 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.975 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:26:16.975 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.051375+0000", 2024-09-13T11:26:16.975 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.4440000000000001e-06 2024-09-13T11:26:16.975 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.975 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.975 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:26:16.975 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.051426+0000", 2024-09-13T11:26:16.975 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.0809000000000001e-05 2024-09-13T11:26:16.976 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.976 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.976 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:26:16.976 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.051441+0000", 2024-09-13T11:26:16.976 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.4460999999999999e-05 2024-09-13T11:26:16.976 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.976 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.976 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:26:16.976 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.051449+0000", 2024-09-13T11:26:16.976 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.4880000000000007e-06 2024-09-13T11:26:16.976 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.976 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.977 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:26:16.977 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.051499+0000", 2024-09-13T11:26:16.977 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.9691999999999998e-05 2024-09-13T11:26:16.977 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.977 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.977 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for peered", 2024-09-13T11:26:16.977 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.051505+0000", 2024-09-13T11:26:16.977 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.2029999999999997e-06 2024-09-13T11:26:16.977 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.977 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.977 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:26:16.977 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.907184+0000", 2024-09-13T11:26:16.977 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.85567906599999999 2024-09-13T11:26:16.978 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.978 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.978 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:26:16.978 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.907192+0000", 2024-09-13T11:26:16.978 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.6469999999999998e-06 2024-09-13T11:26:16.978 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:16.978 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:16.978 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:16.978 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.978 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.978 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:54828 3.1e 3.e920c11e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-13T11:26:16.978 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:32.063344+0000", 2024-09-13T11:26:16.978 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 164.86630793800001, 2024-09-13T11:26:16.979 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.84387899099999997, 2024-09-13T11:26:16.979 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:26:16.979 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:26:16.979 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-13T11:26:16.979 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:26:16.979 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:26:16.979 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:16.979 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 54828 2024-09-13T11:26:16.979 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.979 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:26:16.979 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.979 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:26:16.979 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.063344+0000", 2024-09-13T11:26:16.979 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:16.980 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.980 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.980 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:26:16.980 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.063344+0000", 2024-09-13T11:26:16.980 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:16.980 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.980 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.980 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:26:16.980 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.063347+0000", 2024-09-13T11:26:16.980 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.2299999999999998e-06 2024-09-13T11:26:16.980 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.980 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.980 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:26:16.981 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.063384+0000", 2024-09-13T11:26:16.981 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.7290999999999999e-05 2024-09-13T11:26:16.981 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.981 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.981 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:26:16.981 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.063397+0000", 2024-09-13T11:26:16.981 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2955999999999999e-05 2024-09-13T11:26:16.981 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.981 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.981 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:26:16.981 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.063404+0000", 2024-09-13T11:26:16.981 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.4889999999999998e-06 2024-09-13T11:26:16.981 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.981 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.982 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:26:16.982 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.063470+0000", 2024-09-13T11:26:16.982 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.5516000000000001e-05 2024-09-13T11:26:16.982 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.982 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.982 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for peered", 2024-09-13T11:26:16.982 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.063482+0000", 2024-09-13T11:26:16.982 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2065000000000001e-05 2024-09-13T11:26:16.982 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.982 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.982 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:26:16.982 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.907216+0000", 2024-09-13T11:26:16.982 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.84373388900000001 2024-09-13T11:26:16.983 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.983 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.983 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:26:16.983 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.907223+0000", 2024-09-13T11:26:16.983 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.5549999999999997e-06 2024-09-13T11:26:16.983 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:16.983 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:16.983 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:16.983 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.983 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.983 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:54829 3.1e 3.192f62be (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-13T11:26:16.983 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:32.065642+0000", 2024-09-13T11:26:16.983 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 164.86401086999999, 2024-09-13T11:26:16.984 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.84161483100000001, 2024-09-13T11:26:16.984 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:26:16.984 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:26:16.984 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-13T11:26:16.984 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:26:16.984 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:26:16.984 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:16.984 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 54829 2024-09-13T11:26:16.984 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.984 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:26:16.984 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.984 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:26:16.985 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.065642+0000", 2024-09-13T11:26:16.985 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:16.985 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.985 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.985 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:26:16.985 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.065642+0000", 2024-09-13T11:26:16.985 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:16.985 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.985 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.985 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:26:16.985 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.065644+0000", 2024-09-13T11:26:16.985 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.1780000000000002e-06 2024-09-13T11:26:16.985 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.986 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.986 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:26:16.986 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.065695+0000", 2024-09-13T11:26:16.986 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.1632000000000003e-05 2024-09-13T11:26:16.986 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.986 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.986 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:26:16.986 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.065711+0000", 2024-09-13T11:26:16.986 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.5522000000000001e-05 2024-09-13T11:26:16.986 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.986 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.986 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:26:16.986 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.065719+0000", 2024-09-13T11:26:16.987 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.3480000000000004e-06 2024-09-13T11:26:16.987 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.987 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.987 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:26:16.987 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.065785+0000", 2024-09-13T11:26:16.987 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.5627e-05 2024-09-13T11:26:16.987 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.987 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.987 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for peered", 2024-09-13T11:26:16.987 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.065797+0000", 2024-09-13T11:26:16.987 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.1766000000000001e-05 2024-09-13T11:26:16.987 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.987 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.987 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:26:16.988 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.907248+0000", 2024-09-13T11:26:16.988 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.84145150300000005 2024-09-13T11:26:16.988 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.988 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.988 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:26:16.988 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.907256+0000", 2024-09-13T11:26:16.988 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.2549999999999997e-06 2024-09-13T11:26:16.988 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:16.988 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:16.988 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:16.988 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.988 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.989 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:54830 3.1e 3.b31554de (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-13T11:26:16.989 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:32.068259+0000", 2024-09-13T11:26:16.989 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 164.861393002, 2024-09-13T11:26:16.989 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.83902489599999996, 2024-09-13T11:26:16.989 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:26:16.989 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:26:16.989 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-13T11:26:16.989 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:26:16.989 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:26:16.989 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:16.989 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 54830 2024-09-13T11:26:16.989 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.989 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:26:16.989 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.990 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:26:16.990 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.068259+0000", 2024-09-13T11:26:16.990 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:16.990 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.990 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.990 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:26:16.990 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.068259+0000", 2024-09-13T11:26:16.990 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:16.990 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.990 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.990 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:26:16.990 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.068262+0000", 2024-09-13T11:26:16.990 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.1799999999999999e-06 2024-09-13T11:26:16.990 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.991 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.991 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:26:16.991 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.068301+0000", 2024-09-13T11:26:16.991 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.9289999999999998e-05 2024-09-13T11:26:16.991 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.991 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.991 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:26:16.991 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.068314+0000", 2024-09-13T11:26:16.991 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.3502999999999999e-05 2024-09-13T11:26:16.991 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.991 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.991 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:26:16.991 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.068324+0000", 2024-09-13T11:26:16.992 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.1630000000000002e-06 2024-09-13T11:26:16.992 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.992 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.992 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:26:16.992 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.068375+0000", 2024-09-13T11:26:16.992 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.1805000000000003e-05 2024-09-13T11:26:16.992 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.992 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.992 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for peered", 2024-09-13T11:26:16.992 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.068382+0000", 2024-09-13T11:26:16.992 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.224e-06 2024-09-13T11:26:16.992 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.992 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.993 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:26:16.993 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.907277+0000", 2024-09-13T11:26:16.993 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.83889509500000004 2024-09-13T11:26:16.993 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.993 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.993 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:26:16.993 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.907284+0000", 2024-09-13T11:26:16.993 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.6359999999999994e-06 2024-09-13T11:26:16.993 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:16.993 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:16.993 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:16.993 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.993 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.994 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:54846 3.1e 3.742e049e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-13T11:26:16.994 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:32.153567+0000", 2024-09-13T11:26:16.994 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 164.77608577800001, 2024-09-13T11:26:16.994 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.75378642399999995, 2024-09-13T11:26:16.994 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:26:16.994 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:26:16.994 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-13T11:26:16.994 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:26:16.994 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:26:16.994 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:16.994 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 54846 2024-09-13T11:26:16.994 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.994 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:26:16.995 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.995 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:26:16.995 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.153567+0000", 2024-09-13T11:26:16.995 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:16.995 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.995 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.995 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:26:16.995 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.153567+0000", 2024-09-13T11:26:16.995 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:16.995 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.995 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.995 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:26:16.995 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.153569+0000", 2024-09-13T11:26:16.996 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.4660000000000002e-06 2024-09-13T11:26:16.996 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.996 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.996 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:26:16.996 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.153619+0000", 2024-09-13T11:26:16.996 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.9617000000000003e-05 2024-09-13T11:26:16.996 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.996 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.996 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:26:16.996 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.153634+0000", 2024-09-13T11:26:16.996 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.4904e-05 2024-09-13T11:26:16.996 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.996 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.997 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:26:16.997 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.153642+0000", 2024-09-13T11:26:16.997 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.5599999999999994e-06 2024-09-13T11:26:16.997 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.997 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.997 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:26:16.997 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.153721+0000", 2024-09-13T11:26:16.997 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.8603999999999995e-05 2024-09-13T11:26:16.997 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.997 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.997 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for peered", 2024-09-13T11:26:16.997 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.153730+0000", 2024-09-13T11:26:16.997 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.3840000000000003e-06 2024-09-13T11:26:16.998 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.998 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.998 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:26:16.998 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.907346+0000", 2024-09-13T11:26:16.998 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.75361563099999995 2024-09-13T11:26:16.998 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.998 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.998 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:26:16.998 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.907353+0000", 2024-09-13T11:26:16.998 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.2579999999999998e-06 2024-09-13T11:26:16.998 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:16.998 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:16.998 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:16.999 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:16.999 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:16.999 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:54888 3.1e 3.43e14ede (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-13T11:26:16.999 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:32.423064+0000", 2024-09-13T11:26:16.999 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 164.506588107, 2024-09-13T11:26:16.999 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.48433585299999998, 2024-09-13T11:26:16.999 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:26:16.999 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:26:16.999 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-13T11:26:16.999 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:26:16.999 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:26:16.999 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:16.999 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 54888 2024-09-13T11:26:17.000 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.000 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:26:17.000 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.000 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:26:17.000 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.423064+0000", 2024-09-13T11:26:17.000 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:17.000 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.000 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.000 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:26:17.000 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.423064+0000", 2024-09-13T11:26:17.000 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:17.000 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.001 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.001 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:26:17.001 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.423069+0000", 2024-09-13T11:26:17.001 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.7389999999999999e-06 2024-09-13T11:26:17.001 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.001 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.001 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:26:17.001 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.423115+0000", 2024-09-13T11:26:17.001 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.5847000000000003e-05 2024-09-13T11:26:17.001 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.001 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.001 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:26:17.001 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.423126+0000", 2024-09-13T11:26:17.001 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.1418e-05 2024-09-13T11:26:17.002 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.002 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.002 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:26:17.002 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.423132+0000", 2024-09-13T11:26:17.002 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.57e-06 2024-09-13T11:26:17.002 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.002 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.002 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:26:17.002 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.423191+0000", 2024-09-13T11:26:17.002 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.9250999999999999e-05 2024-09-13T11:26:17.002 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.002 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.002 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for peered", 2024-09-13T11:26:17.003 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.423198+0000", 2024-09-13T11:26:17.003 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.9179999999999997e-06 2024-09-13T11:26:17.003 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.003 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.003 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:26:17.003 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.907392+0000", 2024-09-13T11:26:17.003 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.48419437500000001 2024-09-13T11:26:17.003 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.003 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.003 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:26:17.003 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.907400+0000", 2024-09-13T11:26:17.003 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.7349999999999996e-06 2024-09-13T11:26:17.003 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:17.003 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:17.004 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:17.004 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.004 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.004 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4665.0:59977 3.21 e83/78 3:86b4d8f6:::benchmark_data_smithi177_31116_object59976:head v 83'3136, mlcod=0'0)", 2024-09-13T11:26:17.004 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:40.571147+0000", 2024-09-13T11:26:17.004 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 156.35850565800001, 2024-09-13T11:26:17.004 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.51907542799999995, 2024-09-13T11:26:17.004 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:26:17.004 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:26:17.004 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:17.004 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:26:17.004 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.004 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:26:17.005 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:40.571147+0000", 2024-09-13T11:26:17.005 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:17.005 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.005 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.005 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:26:17.005 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:40.571147+0000", 2024-09-13T11:26:17.005 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:17.005 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.005 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.005 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:26:17.005 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:40.571151+0000", 2024-09-13T11:26:17.005 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.5650000000000003e-06 2024-09-13T11:26:17.005 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.006 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.006 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:26:17.006 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:40.571204+0000", 2024-09-13T11:26:17.006 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.2720999999999999e-05 2024-09-13T11:26:17.006 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.006 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.006 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:26:17.006 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:41.088332+0000", 2024-09-13T11:26:17.006 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.51712822000000003 2024-09-13T11:26:17.006 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.006 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.006 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:26:17.006 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:41.088341+0000", 2024-09-13T11:26:17.006 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.3100000000000001e-06 2024-09-13T11:26:17.007 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.007 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.007 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:26:17.007 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:41.088444+0000", 2024-09-13T11:26:17.007 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000103708 2024-09-13T11:26:17.007 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.007 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.007 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:26:17.007 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:41.088475+0000", 2024-09-13T11:26:17.007 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.0497e-05 2024-09-13T11:26:17.007 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.007 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.007 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:26:17.007 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:41.090192+0000", 2024-09-13T11:26:17.007 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.001717288 2024-09-13T11:26:17.008 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.008 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.008 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:26:17.008 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:41.090222+0000", 2024-09-13T11:26:17.008 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.0119000000000001e-05 2024-09-13T11:26:17.008 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:17.008 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:17.008 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:17.008 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.008 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.008 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:110286 3.26 3.c41c0fa6 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e119)", 2024-09-13T11:26:17.008 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:24:25.654272+0000", 2024-09-13T11:26:17.008 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 111.275379966, 2024-09-13T11:26:17.008 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.43337235400000002, 2024-09-13T11:26:17.008 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:26:17.009 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:26:17.009 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-13T11:26:17.009 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:26:17.009 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:26:17.009 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:17.009 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 110286 2024-09-13T11:26:17.009 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.009 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:26:17.009 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.009 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:26:17.009 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:25.654272+0000", 2024-09-13T11:26:17.009 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:17.009 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.009 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.010 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:26:17.010 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:25.654272+0000", 2024-09-13T11:26:17.010 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:17.010 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.010 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.010 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:26:17.010 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:25.654275+0000", 2024-09-13T11:26:17.010 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.3199999999999998e-06 2024-09-13T11:26:17.010 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.010 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.010 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:26:17.010 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:25.654517+0000", 2024-09-13T11:26:17.010 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00024270199999999999 2024-09-13T11:26:17.010 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.010 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.010 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:26:17.011 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:25.654550+0000", 2024-09-13T11:26:17.011 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.2549000000000003e-05 2024-09-13T11:26:17.011 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.011 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.011 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:26:17.011 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:25.654566+0000", 2024-09-13T11:26:17.011 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.5517000000000001e-05 2024-09-13T11:26:17.011 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.011 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.011 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:26:17.011 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:25.654655+0000", 2024-09-13T11:26:17.011 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.9961000000000001e-05 2024-09-13T11:26:17.011 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.011 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.011 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for peered", 2024-09-13T11:26:17.012 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:25.654665+0000", 2024-09-13T11:26:17.012 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.0520000000000007e-06 2024-09-13T11:26:17.012 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.012 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.012 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:26:17.012 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:26.087629+0000", 2024-09-13T11:26:17.012 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.43296476900000003 2024-09-13T11:26:17.012 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.012 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.012 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:26:17.012 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:26.087645+0000", 2024-09-13T11:26:17.012 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.5483999999999999e-05 2024-09-13T11:26:17.012 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:17.013 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:17.013 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:17.013 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.013 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.013 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:177913 3.27 3:e5268e7a:::benchmark_data_smithi177_31116_object37610:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e168)", 2024-09-13T11:26:17.013 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:25:27.607015+0000", 2024-09-13T11:26:17.013 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 49.322637616000002, 2024-09-13T11:26:17.013 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.301660555, 2024-09-13T11:26:17.013 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:26:17.013 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:26:17.013 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:17.013 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:26:17.013 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:26:17.013 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:17.013 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 177913 2024-09-13T11:26:17.014 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.014 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:26:17.014 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.014 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:26:17.014 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:27.607015+0000", 2024-09-13T11:26:17.014 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:17.014 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.014 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.014 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:26:17.014 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:27.607015+0000", 2024-09-13T11:26:17.014 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:17.014 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.014 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.014 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:26:17.014 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:27.607018+0000", 2024-09-13T11:26:17.014 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.7000000000000002e-06 2024-09-13T11:26:17.015 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.015 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.015 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:26:17.015 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:27.607029+0000", 2024-09-13T11:26:17.015 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0933e-05 2024-09-13T11:26:17.015 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.015 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.015 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:26:17.015 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:27.607062+0000", 2024-09-13T11:26:17.015 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.2786000000000001e-05 2024-09-13T11:26:17.015 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.015 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.015 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:26:17.015 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:27.607078+0000", 2024-09-13T11:26:17.015 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.6232999999999999e-05 2024-09-13T11:26:17.016 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.016 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.016 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:26:17.016 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:27.607204+0000", 2024-09-13T11:26:17.016 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000125458 2024-09-13T11:26:17.016 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.016 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.016 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:26:17.016 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:27.607544+0000", 2024-09-13T11:26:17.016 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00033990600000000001 2024-09-13T11:26:17.016 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.016 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.016 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 5,11", 2024-09-13T11:26:17.016 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:27.607882+0000", 2024-09-13T11:26:17.017 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00033788399999999999 2024-09-13T11:26:17.017 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.017 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.017 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:26:17.017 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:27.610120+0000", 2024-09-13T11:26:17.017 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0022379930000000002 2024-09-13T11:26:17.017 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.017 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.017 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:17.017 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:27.613170+0000", 2024-09-13T11:26:17.017 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.003050545 2024-09-13T11:26:17.017 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.017 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.017 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:17.018 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:27.908398+0000", 2024-09-13T11:26:17.018 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.29522760599999998 2024-09-13T11:26:17.018 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.018 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.018 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:26:17.018 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:27.908458+0000", 2024-09-13T11:26:17.018 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.9718e-05 2024-09-13T11:26:17.018 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.018 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.018 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:26:17.018 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:27.908675+0000", 2024-09-13T11:26:17.018 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00021779300000000001 2024-09-13T11:26:17.018 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:17.018 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:17.019 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:17.019 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.019 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.019 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop_reply(client.4665.0:177913 3.27 e168/120 ondisk, result = 0)", 2024-09-13T11:26:17.019 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:25:27.610053+0000", 2024-09-13T11:26:17.019 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 49.319599457000002, 2024-09-13T11:26:17.019 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.29862817200000003, 2024-09-13T11:26:17.019 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:26:17.019 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:26:17.019 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:26:17.019 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:26:17.019 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.019 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:26:17.020 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:27.610053+0000", 2024-09-13T11:26:17.020 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:17.020 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.020 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.020 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:26:17.020 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:27.610053+0000", 2024-09-13T11:26:17.020 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:17.020 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.020 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.020 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:26:17.020 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:27.610056+0000", 2024-09-13T11:26:17.020 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.49e-06 2024-09-13T11:26:17.020 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.021 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.021 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:26:17.021 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:27.610067+0000", 2024-09-13T11:26:17.021 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0599e-05 2024-09-13T11:26:17.021 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.021 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.021 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:26:17.021 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:27.908314+0000", 2024-09-13T11:26:17.021 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.29824728700000003 2024-09-13T11:26:17.021 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.021 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.021 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:26:17.021 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:27.908320+0000", 2024-09-13T11:26:17.022 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.9100000000000002e-06 2024-09-13T11:26:17.022 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.022 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.022 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:26:17.022 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:27.908378+0000", 2024-09-13T11:26:17.022 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.7575999999999999e-05 2024-09-13T11:26:17.022 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.022 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.022 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:26:17.022 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:27.908390+0000", 2024-09-13T11:26:17.022 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.187e-05 2024-09-13T11:26:17.022 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.022 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.022 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:26:17.023 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:27.908681+0000", 2024-09-13T11:26:17.023 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00029144000000000002 2024-09-13T11:26:17.023 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:17.023 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:17.023 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:17.023 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.023 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.023 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDPGPushReply(3.27 168/120 [PushReplyOp(3:e535cc18:::benchmark_data_smithi177_31116_object103295:head)])", 2024-09-13T11:26:17.023 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:25:27.615014+0000", 2024-09-13T11:26:17.023 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 49.314638123000002, 2024-09-13T11:26:17.023 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.31810488799999997, 2024-09-13T11:26:17.023 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:26:17.023 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:26:17.024 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-13T11:26:17.024 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:26:17.024 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.024 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:26:17.024 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:27.615014+0000", 2024-09-13T11:26:17.024 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:17.024 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.024 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.024 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:26:17.024 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:27.615014+0000", 2024-09-13T11:26:17.024 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:17.024 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.024 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.024 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:26:17.025 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:27.615017+0000", 2024-09-13T11:26:17.025 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.6199999999999999e-06 2024-09-13T11:26:17.025 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.025 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.025 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:26:17.025 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:27.615030+0000", 2024-09-13T11:26:17.025 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2833000000000001e-05 2024-09-13T11:26:17.025 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.025 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.025 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:26:17.025 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:27.932588+0000", 2024-09-13T11:26:17.025 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.31755857700000001 2024-09-13T11:26:17.025 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.025 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.026 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:26:17.026 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:27.932595+0000", 2024-09-13T11:26:17.026 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.2160000000000003e-06 2024-09-13T11:26:17.026 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.026 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.026 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:26:17.026 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:27.932658+0000", 2024-09-13T11:26:17.026 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.3618e-05 2024-09-13T11:26:17.026 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.026 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.026 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:26:17.026 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:27.933119+0000", 2024-09-13T11:26:17.026 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00046102399999999998 2024-09-13T11:26:17.026 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:17.027 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:17.027 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:17.027 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.027 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.027 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:191082 3.26 3:67bd986b:::benchmark_data_smithi177_31116_object50779:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e175)", 2024-09-13T11:26:17.027 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:25:38.550314+0000", 2024-09-13T11:26:17.027 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 38.379338496000003, 2024-09-13T11:26:17.027 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.29180858100000001, 2024-09-13T11:26:17.027 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:26:17.027 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:26:17.027 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:17.027 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:26:17.027 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:26:17.027 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:17.028 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 191082 2024-09-13T11:26:17.028 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.028 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:26:17.028 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.028 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:26:17.028 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:38.550314+0000", 2024-09-13T11:26:17.028 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:17.028 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.028 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.028 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:26:17.028 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:38.550314+0000", 2024-09-13T11:26:17.028 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:17.028 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.028 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.028 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:26:17.029 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:38.550317+0000", 2024-09-13T11:26:17.029 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.14e-06 2024-09-13T11:26:17.029 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.029 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.029 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:26:17.029 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:38.550327+0000", 2024-09-13T11:26:17.029 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0386e-05 2024-09-13T11:26:17.029 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.029 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.029 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:26:17.029 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:38.550353+0000", 2024-09-13T11:26:17.029 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.5431000000000001e-05 2024-09-13T11:26:17.029 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.029 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.030 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:26:17.030 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:38.550367+0000", 2024-09-13T11:26:17.030 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.3859999999999999e-05 2024-09-13T11:26:17.030 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.030 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.030 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:26:17.030 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:38.550476+0000", 2024-09-13T11:26:17.030 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000108782 2024-09-13T11:26:17.030 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.030 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.030 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:26:17.030 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:38.550677+0000", 2024-09-13T11:26:17.030 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00020138099999999999 2024-09-13T11:26:17.030 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.031 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.031 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 12", 2024-09-13T11:26:17.031 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:38.550836+0000", 2024-09-13T11:26:17.031 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000158722 2024-09-13T11:26:17.031 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.031 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.031 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:26:17.031 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:38.553897+0000", 2024-09-13T11:26:17.031 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.003061533 2024-09-13T11:26:17.031 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.031 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.031 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:17.031 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:38.841929+0000", 2024-09-13T11:26:17.032 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.28803159 2024-09-13T11:26:17.032 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.032 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.032 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:26:17.032 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:38.842014+0000", 2024-09-13T11:26:17.032 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.5119999999999998e-05 2024-09-13T11:26:17.032 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.032 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.032 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:26:17.032 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:38.842122+0000", 2024-09-13T11:26:17.032 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00010863599999999999 2024-09-13T11:26:17.032 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:17.032 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:17.032 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:17.033 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:17.033 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:17.033 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-13T11:26:17.034 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_ops_in_flight 2024-09-13T11:26:17.035 INFO:tasks.ceph.osd.13.smithi092.stderr:2024-09-13T11:26:17.032+0000 7f3e5afea640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26350) UID: 0 2024-09-13T11:26:17.121 INFO:teuthology.orchestra.run.smithi001.stderr:nodeep-scrub is set 2024-09-13T11:26:17.121 INFO:teuthology.orchestra.run.smithi001.stderr:set 3.22 pg_upmap mapping to [14,12] 2024-09-13T11:26:17.127 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-13T11:26:17.126+0000 7f6a5d1a2640 -1 osd.0 200 handle_osd_map injecting map crc failure 2024-09-13T11:26:17.130 INFO:tasks.ceph.osd.15.smithi177.stderr:2024-09-13T11:26:17.129+0000 7f9d638a8640 -1 osd.15 200 handle_osd_map injecting map crc failure 2024-09-13T11:26:17.134 INFO:tasks.ceph.osd.14.smithi162.stderr:2024-09-13T11:26:17.131+0000 7f3df5903640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33154) UID: 0 2024-09-13T11:26:17.179 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-13T11:26:17.180 INFO:teuthology.orchestra.run.smithi001.stdout: "ops": [ 2024-09-13T11:26:17.180 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.180 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:234128 3.9 3:91313bf5:::benchmark_data_smithi177_31116_object93825:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e201)", 2024-09-13T11:26:17.180 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:26:17.158428+0000", 2024-09-13T11:26:17.180 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 0.021260459999999998, 2024-09-13T11:26:17.180 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.021286915, 2024-09-13T11:26:17.180 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:26:17.180 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:26:17.180 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "waiting for sub ops", 2024-09-13T11:26:17.180 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:26:17.180 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:26:17.181 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:17.181 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 234128 2024-09-13T11:26:17.181 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.181 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:26:17.181 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.181 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:26:17.181 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:17.158428+0000", 2024-09-13T11:26:17.181 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:17.181 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.181 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.181 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:26:17.181 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:17.158428+0000", 2024-09-13T11:26:17.181 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:17.182 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.182 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.182 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:26:17.182 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:17.158430+0000", 2024-09-13T11:26:17.182 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.4589999999999999e-06 2024-09-13T11:26:17.182 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.182 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.182 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:26:17.182 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:17.158434+0000", 2024-09-13T11:26:17.182 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.3660000000000002e-06 2024-09-13T11:26:17.182 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.182 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.182 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:26:17.183 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:17.158442+0000", 2024-09-13T11:26:17.183 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.354e-06 2024-09-13T11:26:17.183 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.183 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.183 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:26:17.183 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:17.158448+0000", 2024-09-13T11:26:17.183 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.5759999999999996e-06 2024-09-13T11:26:17.183 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.183 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.183 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:26:17.183 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:17.158503+0000", 2024-09-13T11:26:17.183 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.4936999999999999e-05 2024-09-13T11:26:17.183 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.184 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.184 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:26:17.184 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:17.158637+0000", 2024-09-13T11:26:17.184 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00013380999999999999 2024-09-13T11:26:17.184 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.184 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.184 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 2", 2024-09-13T11:26:17.184 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:17.158748+0000", 2024-09-13T11:26:17.184 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00011127499999999999 2024-09-13T11:26:17.184 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.184 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.184 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:26:17.184 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:17.160755+0000", 2024-09-13T11:26:17.184 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0020068780000000001 2024-09-13T11:26:17.185 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:17.185 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:17.185 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:17.185 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.185 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.185 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:234149 3.0 3:01c32108:::benchmark_data_smithi177_31116_object93846:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e201)", 2024-09-13T11:26:17.185 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:26:17.177709+0000", 2024-09-13T11:26:17.185 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 0.001979953, 2024-09-13T11:26:17.185 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0020787599999999998, 2024-09-13T11:26:17.185 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:26:17.185 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:26:17.185 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "waiting for sub ops", 2024-09-13T11:26:17.185 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:26:17.186 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:26:17.186 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:17.186 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 234149 2024-09-13T11:26:17.186 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.186 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:26:17.186 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.186 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:26:17.186 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:17.177709+0000", 2024-09-13T11:26:17.186 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:17.186 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.186 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.186 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:26:17.186 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:17.177709+0000", 2024-09-13T11:26:17.187 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:17.187 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.187 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.187 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:26:17.187 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:17.177710+0000", 2024-09-13T11:26:17.187 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.5650000000000001e-06 2024-09-13T11:26:17.187 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.187 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.187 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:26:17.187 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:17.177715+0000", 2024-09-13T11:26:17.187 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.0119999999999996e-06 2024-09-13T11:26:17.187 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.187 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.187 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:26:17.187 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:17.177726+0000", 2024-09-13T11:26:17.187 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0557000000000001e-05 2024-09-13T11:26:17.188 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.188 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.188 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:26:17.188 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:17.177731+0000", 2024-09-13T11:26:17.188 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.5489999999999997e-06 2024-09-13T11:26:17.188 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.188 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.188 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:26:17.188 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:17.177782+0000", 2024-09-13T11:26:17.188 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.0893000000000002e-05 2024-09-13T11:26:17.188 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.188 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.188 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:26:17.189 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:17.177868+0000", 2024-09-13T11:26:17.189 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.5338000000000002e-05 2024-09-13T11:26:17.189 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.189 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.189 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 15", 2024-09-13T11:26:17.189 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:17.177932+0000", 2024-09-13T11:26:17.189 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.4739999999999993e-05 2024-09-13T11:26:17.189 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:17.189 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:17.189 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:17.189 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:17.189 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-09-13T11:26:17.189 INFO:teuthology.orchestra.run.smithi001.stdout: "num_ops": 2 2024-09-13T11:26:17.189 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-13T11:26:17.197 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_blocked_ops 2024-09-13T11:26:17.235 INFO:tasks.ceph.osd.1.smithi092.stderr:2024-09-13T11:26:17.232+0000 7fe7789c5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33718) UID: 0 2024-09-13T11:26:17.336 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-09-13T11:26:17.335+0000 7f97befc9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26315) UID: 0 2024-09-13T11:26:17.358 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-13T11:26:17.358 INFO:teuthology.orchestra.run.smithi001.stdout: "ops": [], 2024-09-13T11:26:17.358 INFO:teuthology.orchestra.run.smithi001.stdout: "complaint_time": 30, 2024-09-13T11:26:17.358 INFO:teuthology.orchestra.run.smithi001.stdout: "num_blocked_ops": 0 2024-09-13T11:26:17.358 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-13T11:26:17.373 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_historic_ops 2024-09-13T11:26:17.437 INFO:tasks.ceph.osd.15.smithi177.stderr:2024-09-13T11:26:17.433+0000 7f9d6ce77640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26578) UID: 0 2024-09-13T11:26:17.495 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-13T11:26:17.496 INFO:teuthology.orchestra.run.smithi001.stdout: "size": 20, 2024-09-13T11:26:17.496 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 600, 2024-09-13T11:26:17.496 INFO:teuthology.orchestra.run.smithi001.stdout: "ops": [ 2024-09-13T11:26:17.496 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.496 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop_reply(client.4665.0:167 3.9 e41/30 ondisk, result = 0)", 2024-09-13T11:26:17.496 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:22:47.419696+0000", 2024-09-13T11:26:17.496 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 210.073962851, 2024-09-13T11:26:17.496 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.64615778199999996, 2024-09-13T11:26:17.496 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:26:17.496 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:26:17.496 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:26:17.497 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:26:17.497 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.497 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:26:17.497 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:47.419696+0000", 2024-09-13T11:26:17.497 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:17.497 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.497 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.497 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:26:17.497 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:47.419696+0000", 2024-09-13T11:26:17.497 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:17.498 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.498 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.498 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:26:17.498 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:47.419697+0000", 2024-09-13T11:26:17.498 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.15e-06 2024-09-13T11:26:17.498 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.498 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.498 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:26:17.498 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:47.419700+0000", 2024-09-13T11:26:17.498 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.2449999999999998e-06 2024-09-13T11:26:17.498 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.498 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.499 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:26:17.499 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:48.065637+0000", 2024-09-13T11:26:17.499 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.64593674300000004 2024-09-13T11:26:17.499 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.499 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.499 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:26:17.499 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:48.065648+0000", 2024-09-13T11:26:17.499 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0865e-05 2024-09-13T11:26:17.499 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.499 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.499 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:26:17.499 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:48.065720+0000", 2024-09-13T11:26:17.499 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.2020999999999994e-05 2024-09-13T11:26:17.500 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.500 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.500 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:26:17.500 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:48.065728+0000", 2024-09-13T11:26:17.500 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.7660000000000004e-06 2024-09-13T11:26:17.500 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.500 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.500 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:26:17.500 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:48.065853+0000", 2024-09-13T11:26:17.500 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00012499200000000001 2024-09-13T11:26:17.500 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:17.500 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:17.500 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:17.501 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.501 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.501 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:25383 3.0 3:0a6bb869:::benchmark_data_smithi177_31116_object25382:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e55)", 2024-09-13T11:26:17.501 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:04.447376+0000", 2024-09-13T11:26:17.501 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 193.046282182, 2024-09-13T11:26:17.501 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.21239864, 2024-09-13T11:26:17.501 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:26:17.501 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:26:17.501 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:17.501 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:26:17.501 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:26:17.501 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:17.501 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 25383 2024-09-13T11:26:17.502 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.502 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:26:17.502 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.502 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:26:17.502 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.447376+0000", 2024-09-13T11:26:17.502 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:17.502 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.502 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.502 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:26:17.502 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.447376+0000", 2024-09-13T11:26:17.502 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:17.502 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.502 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.503 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:26:17.503 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.447378+0000", 2024-09-13T11:26:17.503 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.6729999999999999e-06 2024-09-13T11:26:17.503 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.503 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.503 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:26:17.503 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.447433+0000", 2024-09-13T11:26:17.503 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.5371000000000002e-05 2024-09-13T11:26:17.503 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.503 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.503 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:26:17.503 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.447446+0000", 2024-09-13T11:26:17.503 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.3179999999999999e-05 2024-09-13T11:26:17.503 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.503 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.504 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:26:17.504 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.447456+0000", 2024-09-13T11:26:17.504 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0008e-05 2024-09-13T11:26:17.504 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.504 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.504 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:26:17.504 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.447514+0000", 2024-09-13T11:26:17.504 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.8023999999999999e-05 2024-09-13T11:26:17.504 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.504 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.504 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:26:17.504 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.447642+0000", 2024-09-13T11:26:17.505 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000127296 2024-09-13T11:26:17.505 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.505 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.505 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 15", 2024-09-13T11:26:17.505 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.447767+0000", 2024-09-13T11:26:17.505 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000125547 2024-09-13T11:26:17.505 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.505 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.505 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:26:17.505 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.449188+0000", 2024-09-13T11:26:17.505 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.001420604 2024-09-13T11:26:17.505 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.505 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.505 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:17.506 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.659668+0000", 2024-09-13T11:26:17.506 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.210479685 2024-09-13T11:26:17.506 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.506 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.506 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:26:17.506 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.659716+0000", 2024-09-13T11:26:17.506 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.8609999999999997e-05 2024-09-13T11:26:17.506 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.506 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.506 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:26:17.506 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.659775+0000", 2024-09-13T11:26:17.506 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.8641999999999999e-05 2024-09-13T11:26:17.506 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:17.506 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:17.507 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:17.507 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.507 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.507 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4665.0:25408 3.4 e56/40 3:294e23bd:::benchmark_data_smithi177_31116_object25407:head v 56'1645, mlcod=55'1642)", 2024-09-13T11:26:17.507 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:04.494686+0000", 2024-09-13T11:26:17.507 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 192.99897252299999, 2024-09-13T11:26:17.507 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.29725566399999998, 2024-09-13T11:26:17.507 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:26:17.507 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:26:17.507 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:17.507 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:26:17.507 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.507 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:26:17.507 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.494686+0000", 2024-09-13T11:26:17.508 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:17.508 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.508 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.508 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:26:17.508 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.494686+0000", 2024-09-13T11:26:17.508 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:17.508 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.508 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.508 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:26:17.508 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.494686+0000", 2024-09-13T11:26:17.508 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.2399999999999999e-07 2024-09-13T11:26:17.508 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.509 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.509 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:26:17.509 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.494809+0000", 2024-09-13T11:26:17.509 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00012308600000000001 2024-09-13T11:26:17.509 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.510 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.510 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:26:17.510 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.789410+0000", 2024-09-13T11:26:17.510 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.29460066899999998 2024-09-13T11:26:17.510 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.510 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.510 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:26:17.510 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.789425+0000", 2024-09-13T11:26:17.510 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.4600000000000001e-05 2024-09-13T11:26:17.510 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.510 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.510 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:26:17.510 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.789507+0000", 2024-09-13T11:26:17.511 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.2785000000000001e-05 2024-09-13T11:26:17.511 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.511 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.511 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:26:17.511 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.789525+0000", 2024-09-13T11:26:17.511 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.8117999999999998e-05 2024-09-13T11:26:17.511 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.511 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.511 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:26:17.511 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.791852+0000", 2024-09-13T11:26:17.511 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0023264459999999998 2024-09-13T11:26:17.511 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.511 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.512 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:26:17.512 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.791942+0000", 2024-09-13T11:26:17.512 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.9635999999999994e-05 2024-09-13T11:26:17.512 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:17.512 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:17.512 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:17.512 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.512 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.512 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4665.0:30815 3.f e59/30 3:fde17cb1:::benchmark_data_smithi177_31116_object30814:head v 59'1919, mlcod=59'1914)", 2024-09-13T11:26:17.512 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:09.494342+0000", 2024-09-13T11:26:17.512 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 187.999316537, 2024-09-13T11:26:17.512 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.22787855700000001, 2024-09-13T11:26:17.512 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:26:17.513 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:26:17.513 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:17.513 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:26:17.513 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.513 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:26:17.513 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.494342+0000", 2024-09-13T11:26:17.513 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:17.513 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.513 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.513 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:26:17.513 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.494342+0000", 2024-09-13T11:26:17.513 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:17.513 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.513 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.514 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:26:17.514 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.494343+0000", 2024-09-13T11:26:17.514 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.251e-06 2024-09-13T11:26:17.514 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.514 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.514 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:26:17.514 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.494387+0000", 2024-09-13T11:26:17.514 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.4276000000000001e-05 2024-09-13T11:26:17.514 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.514 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.514 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:26:17.514 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.720454+0000", 2024-09-13T11:26:17.514 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.226066614 2024-09-13T11:26:17.514 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.514 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.515 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:26:17.515 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.720462+0000", 2024-09-13T11:26:17.515 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.9240000000000007e-06 2024-09-13T11:26:17.515 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.515 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.515 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:26:17.515 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.720537+0000", 2024-09-13T11:26:17.515 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.4967000000000002e-05 2024-09-13T11:26:17.515 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.515 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.515 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:26:17.515 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.720563+0000", 2024-09-13T11:26:17.515 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.5630999999999999e-05 2024-09-13T11:26:17.515 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.515 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.516 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:26:17.516 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.722183+0000", 2024-09-13T11:26:17.516 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0016209670000000001 2024-09-13T11:26:17.516 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.516 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.516 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:26:17.516 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.722220+0000", 2024-09-13T11:26:17.516 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.6927000000000001e-05 2024-09-13T11:26:17.516 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:17.516 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:17.516 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:17.516 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.516 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.517 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4665.0:30828 3.f e59/30 3:f3a9c6a9:::benchmark_data_smithi177_31116_object30827:head v 59'1920, mlcod=59'1916)", 2024-09-13T11:26:17.517 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:09.502607+0000", 2024-09-13T11:26:17.517 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 187.99105148000001, 2024-09-13T11:26:17.517 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.26374408300000002, 2024-09-13T11:26:17.517 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:26:17.517 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:26:17.517 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:17.517 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:26:17.517 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.517 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:26:17.517 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.502607+0000", 2024-09-13T11:26:17.517 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:17.517 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.517 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.518 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:26:17.518 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.502607+0000", 2024-09-13T11:26:17.518 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:17.518 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.518 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.518 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:26:17.518 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.502609+0000", 2024-09-13T11:26:17.518 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.84e-06 2024-09-13T11:26:17.518 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.518 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.518 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:26:17.518 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.502663+0000", 2024-09-13T11:26:17.518 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.3971000000000002e-05 2024-09-13T11:26:17.518 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.518 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.519 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:26:17.519 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.763912+0000", 2024-09-13T11:26:17.519 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.26124904300000001 2024-09-13T11:26:17.519 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.519 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.519 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:26:17.519 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.763920+0000", 2024-09-13T11:26:17.519 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.1149999999999994e-06 2024-09-13T11:26:17.519 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.519 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.519 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:26:17.519 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.764024+0000", 2024-09-13T11:26:17.519 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00010380899999999999 2024-09-13T11:26:17.519 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.520 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.520 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:26:17.520 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.764068+0000", 2024-09-13T11:26:17.520 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.4638999999999997e-05 2024-09-13T11:26:17.520 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.520 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.520 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:26:17.520 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.766308+0000", 2024-09-13T11:26:17.520 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0022392039999999998 2024-09-13T11:26:17.520 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.520 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.520 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:26:17.520 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.766351+0000", 2024-09-13T11:26:17.520 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.3461999999999998e-05 2024-09-13T11:26:17.521 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:17.521 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:17.521 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:17.521 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.521 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.521 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDPGPush(3.1b 84/78 [PushOp(3:dcddd139:::benchmark_data_smithi177_31116_object10384:head, version: 48'673, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:dcddd139:::benchmark_data_smithi177_31116_object10384:head@48'673, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:26:17.521 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:41.400602+0000", 2024-09-13T11:26:17.521 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 156.09305599499999, 2024-09-13T11:26:17.521 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.43365821999999998, 2024-09-13T11:26:17.521 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:26:17.521 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:26:17.521 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:26:17.521 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:26:17.521 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.522 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:26:17.522 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:41.400602+0000", 2024-09-13T11:26:17.522 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:17.522 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.522 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.522 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:26:17.522 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:41.400602+0000", 2024-09-13T11:26:17.522 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:17.522 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.522 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.522 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:26:17.522 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:41.400604+0000", 2024-09-13T11:26:17.522 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.99e-06 2024-09-13T11:26:17.522 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.522 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.523 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:26:17.523 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:41.400653+0000", 2024-09-13T11:26:17.523 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.8195000000000002e-05 2024-09-13T11:26:17.523 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.523 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.523 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:26:17.523 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:41.833180+0000", 2024-09-13T11:26:17.523 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.43252769200000002 2024-09-13T11:26:17.523 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.523 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.523 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:26:17.523 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:41.833213+0000", 2024-09-13T11:26:17.523 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.2737999999999997e-05 2024-09-13T11:26:17.523 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.524 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.524 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:26:17.524 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:41.833408+0000", 2024-09-13T11:26:17.524 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00019473799999999999 2024-09-13T11:26:17.524 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.524 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.524 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:26:17.524 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:41.833417+0000", 2024-09-13T11:26:17.524 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.5349999999999993e-06 2024-09-13T11:26:17.524 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.524 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.524 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:26:17.524 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:41.834261+0000", 2024-09-13T11:26:17.524 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00084333200000000002 2024-09-13T11:26:17.525 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:17.525 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:17.525 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:17.525 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.525 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.525 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDPGPushReply(3.20 85/77 [PushReplyOp(3:066982a6:::benchmark_data_smithi177_31116_object16400:head)])", 2024-09-13T11:26:17.525 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:42.671551+0000", 2024-09-13T11:26:17.525 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 154.82210717199999, 2024-09-13T11:26:17.525 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.28249085499999999, 2024-09-13T11:26:17.525 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:26:17.525 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:26:17.525 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-13T11:26:17.525 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:26:17.525 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.526 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:26:17.526 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.671551+0000", 2024-09-13T11:26:17.526 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:17.526 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.526 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.526 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:26:17.526 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.671551+0000", 2024-09-13T11:26:17.526 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:17.526 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.526 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.526 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:26:17.526 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.671552+0000", 2024-09-13T11:26:17.526 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.097e-06 2024-09-13T11:26:17.526 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.526 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.527 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:26:17.527 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.671556+0000", 2024-09-13T11:26:17.527 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.8519999999999997e-06 2024-09-13T11:26:17.527 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.527 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.527 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:26:17.527 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.951769+0000", 2024-09-13T11:26:17.527 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.28021293000000003 2024-09-13T11:26:17.527 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.527 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.527 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:26:17.527 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.951780+0000", 2024-09-13T11:26:17.527 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.1059e-05 2024-09-13T11:26:17.528 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.528 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.528 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:26:17.528 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.951992+0000", 2024-09-13T11:26:17.528 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000211478 2024-09-13T11:26:17.528 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.528 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.528 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:26:17.528 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.954042+0000", 2024-09-13T11:26:17.528 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0020504389999999998 2024-09-13T11:26:17.528 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:17.528 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:17.528 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:17.528 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.528 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.529 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDPGPush(3.1b 89/78 [PushOp(3:dc4c79e7:::benchmark_data_smithi177_31116_object19857:head, version: 54'1266, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:dc4c79e7:::benchmark_data_smithi177_31116_object19857:head@54'1266, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:26:17.529 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:46.649128+0000", 2024-09-13T11:26:17.529 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 150.844530064, 2024-09-13T11:26:17.529 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.29963603, 2024-09-13T11:26:17.529 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:26:17.529 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:26:17.529 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:26:17.529 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:26:17.529 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.529 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:26:17.529 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.649128+0000", 2024-09-13T11:26:17.529 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:17.529 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.530 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.530 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:26:17.530 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.649128+0000", 2024-09-13T11:26:17.530 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:17.530 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.530 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.530 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:26:17.530 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.649130+0000", 2024-09-13T11:26:17.530 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2950000000000001e-06 2024-09-13T11:26:17.530 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.530 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.530 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:26:17.530 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.649197+0000", 2024-09-13T11:26:17.530 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.7033999999999995e-05 2024-09-13T11:26:17.530 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.531 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.531 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:26:17.531 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.947691+0000", 2024-09-13T11:26:17.531 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.29849387300000002 2024-09-13T11:26:17.531 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.531 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.531 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:26:17.531 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.947702+0000", 2024-09-13T11:26:17.531 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.1239e-05 2024-09-13T11:26:17.531 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.531 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.531 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:26:17.531 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.947799+0000", 2024-09-13T11:26:17.531 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.7283999999999999e-05 2024-09-13T11:26:17.532 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.532 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.532 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:26:17.532 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.947809+0000", 2024-09-13T11:26:17.532 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.008e-05 2024-09-13T11:26:17.532 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.532 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.532 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:26:17.532 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.948764+0000", 2024-09-13T11:26:17.532 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00095522499999999998 2024-09-13T11:26:17.532 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:17.532 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:17.532 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:17.532 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.533 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.533 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:71927 3.0 3:0197244a:::benchmark_data_smithi177_31116_object71926:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2024-09-13T11:26:17.533 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:54.199022+0000", 2024-09-13T11:26:17.533 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 143.29463670800001, 2024-09-13T11:26:17.533 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.26819617800000001, 2024-09-13T11:26:17.533 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:26:17.533 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:26:17.533 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:17.533 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:26:17.533 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:26:17.533 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:17.533 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 71927 2024-09-13T11:26:17.533 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.533 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:26:17.534 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.534 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:26:17.534 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:54.199022+0000", 2024-09-13T11:26:17.534 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:17.534 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.534 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.534 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:26:17.534 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:54.199022+0000", 2024-09-13T11:26:17.534 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:17.535 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.535 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.535 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:26:17.535 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:54.199025+0000", 2024-09-13T11:26:17.535 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.2200000000000001e-06 2024-09-13T11:26:17.535 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.535 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.535 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:26:17.535 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:54.199076+0000", 2024-09-13T11:26:17.535 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.1184000000000003e-05 2024-09-13T11:26:17.535 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.535 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.535 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:26:17.535 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:54.199095+0000", 2024-09-13T11:26:17.535 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.9253999999999999e-05 2024-09-13T11:26:17.536 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.536 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.536 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:26:17.536 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:54.199107+0000", 2024-09-13T11:26:17.536 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.1916000000000001e-05 2024-09-13T11:26:17.536 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.536 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.536 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:26:17.536 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:54.199196+0000", 2024-09-13T11:26:17.536 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.8645000000000003e-05 2024-09-13T11:26:17.536 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.536 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.536 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:26:17.537 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:54.199421+0000", 2024-09-13T11:26:17.537 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000224868 2024-09-13T11:26:17.537 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.537 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.537 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 15", 2024-09-13T11:26:17.537 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:54.199647+0000", 2024-09-13T11:26:17.537 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00022666099999999999 2024-09-13T11:26:17.537 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.537 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.537 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:26:17.537 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:54.201333+0000", 2024-09-13T11:26:17.537 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.001685761 2024-09-13T11:26:17.537 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.538 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.538 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:17.538 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:54.467097+0000", 2024-09-13T11:26:17.538 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.26576393399999998 2024-09-13T11:26:17.538 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.538 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.538 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:26:17.538 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:54.467163+0000", 2024-09-13T11:26:17.538 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.6088999999999997e-05 2024-09-13T11:26:17.538 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.538 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.538 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:26:17.538 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:54.467218+0000", 2024-09-13T11:26:17.538 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.4645999999999998e-05 2024-09-13T11:26:17.539 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:17.539 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:17.539 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:17.539 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.539 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.539 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDPGPush(3.25 121/120 [PushOp(3:a5de2229:::benchmark_data_smithi177_31116_object1005:head, version: 43'62, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:a5de2229:::benchmark_data_smithi177_31116_object1005:head@43'62, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:26:17.539 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:24:27.379880+0000", 2024-09-13T11:26:17.539 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 110.11377887, 2024-09-13T11:26:17.539 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.72647719499999996, 2024-09-13T11:26:17.539 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:26:17.539 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:26:17.539 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:26:17.539 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:26:17.540 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.540 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:26:17.540 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:27.379880+0000", 2024-09-13T11:26:17.540 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:17.540 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.540 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.540 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:26:17.540 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:27.379880+0000", 2024-09-13T11:26:17.540 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:17.540 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.540 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.540 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:26:17.540 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:27.379882+0000", 2024-09-13T11:26:17.541 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.1050000000000002e-06 2024-09-13T11:26:17.541 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.541 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.541 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:26:17.541 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:27.379976+0000", 2024-09-13T11:26:17.541 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.4468000000000004e-05 2024-09-13T11:26:17.541 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.541 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.541 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:26:17.541 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:28.105075+0000", 2024-09-13T11:26:17.541 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.72509911299999996 2024-09-13T11:26:17.541 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.541 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.542 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:26:17.542 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:28.105111+0000", 2024-09-13T11:26:17.542 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.5403e-05 2024-09-13T11:26:17.542 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.542 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.542 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:26:17.542 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:28.105267+0000", 2024-09-13T11:26:17.542 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00015663600000000001 2024-09-13T11:26:17.542 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.542 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.542 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:26:17.542 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:28.105291+0000", 2024-09-13T11:26:17.542 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.4164000000000001e-05 2024-09-13T11:26:17.543 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.543 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.543 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:26:17.543 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:28.106357+0000", 2024-09-13T11:26:17.543 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.001065306 2024-09-13T11:26:17.543 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:17.543 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:17.543 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:17.543 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.543 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.543 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:113646 3.20 3:059047a0:::benchmark_data_smithi177_31116_object113645:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124)", 2024-09-13T11:26:17.543 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:24:32.178229+0000", 2024-09-13T11:26:17.543 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 105.31542890999999, 2024-09-13T11:26:17.544 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.23346934699999999, 2024-09-13T11:26:17.544 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:26:17.544 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:26:17.544 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:17.544 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:26:17.544 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:26:17.544 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:17.544 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 113646 2024-09-13T11:26:17.544 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.544 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:26:17.544 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.544 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:26:17.545 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:32.178229+0000", 2024-09-13T11:26:17.545 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:17.545 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.545 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.545 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:26:17.545 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:32.178229+0000", 2024-09-13T11:26:17.545 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:17.545 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.545 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.545 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:26:17.545 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:32.178232+0000", 2024-09-13T11:26:17.545 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.069e-06 2024-09-13T11:26:17.545 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.546 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.546 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:26:17.546 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:32.178266+0000", 2024-09-13T11:26:17.546 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.4242999999999997e-05 2024-09-13T11:26:17.546 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.546 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.546 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:26:17.546 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:32.178278+0000", 2024-09-13T11:26:17.546 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2558000000000001e-05 2024-09-13T11:26:17.546 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.546 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.546 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:26:17.546 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:32.178288+0000", 2024-09-13T11:26:17.546 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.6469999999999993e-06 2024-09-13T11:26:17.547 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.547 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.547 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:26:17.547 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:32.178349+0000", 2024-09-13T11:26:17.547 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.0643999999999997e-05 2024-09-13T11:26:17.547 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.547 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.547 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:26:17.547 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:32.178477+0000", 2024-09-13T11:26:17.547 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000128038 2024-09-13T11:26:17.547 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.547 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.547 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 0", 2024-09-13T11:26:17.547 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:32.178602+0000", 2024-09-13T11:26:17.548 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00012553600000000001 2024-09-13T11:26:17.548 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.548 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.548 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:26:17.548 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:32.180157+0000", 2024-09-13T11:26:17.548 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.001554386 2024-09-13T11:26:17.548 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.548 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.548 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:17.548 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:32.411634+0000", 2024-09-13T11:26:17.548 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.231477561 2024-09-13T11:26:17.548 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.548 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.548 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:26:17.549 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:32.411672+0000", 2024-09-13T11:26:17.549 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.7357999999999997e-05 2024-09-13T11:26:17.549 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.549 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.549 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:26:17.549 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:32.411699+0000", 2024-09-13T11:26:17.549 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.7307000000000001e-05 2024-09-13T11:26:17.549 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:17.549 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:17.549 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:17.549 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.550 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.550 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop_reply(client.4665.0:113646 3.20 e124/77 ondisk, result = 0)", 2024-09-13T11:26:17.550 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:24:32.180524+0000", 2024-09-13T11:26:17.550 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 105.31313412900001, 2024-09-13T11:26:17.550 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.231180894, 2024-09-13T11:26:17.550 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:26:17.550 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:26:17.550 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:26:17.550 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:26:17.550 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.550 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:26:17.551 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:32.180524+0000", 2024-09-13T11:26:17.551 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:17.551 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.551 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.551 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:26:17.551 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:32.180524+0000", 2024-09-13T11:26:17.551 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:17.551 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.551 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.551 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:26:17.551 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:32.180526+0000", 2024-09-13T11:26:17.551 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.4419999999999999e-06 2024-09-13T11:26:17.552 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.552 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.552 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:26:17.552 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:32.180530+0000", 2024-09-13T11:26:17.552 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.7330000000000003e-06 2024-09-13T11:26:17.552 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.552 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.552 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:26:17.552 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:32.411549+0000", 2024-09-13T11:26:17.552 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.23101862300000001 2024-09-13T11:26:17.552 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.552 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.552 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:26:17.553 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:32.411558+0000", 2024-09-13T11:26:17.553 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.5949999999999999e-06 2024-09-13T11:26:17.553 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.553 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.553 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:26:17.553 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:32.411622+0000", 2024-09-13T11:26:17.553 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.4251000000000005e-05 2024-09-13T11:26:17.553 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.553 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.553 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:26:17.553 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:32.411629+0000", 2024-09-13T11:26:17.553 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.328e-06 2024-09-13T11:26:17.553 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.554 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.554 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:26:17.554 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:32.411705+0000", 2024-09-13T11:26:17.554 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.5921999999999996e-05 2024-09-13T11:26:17.554 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:17.554 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:17.554 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:17.554 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.554 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.556 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:138294 3.9 3:90012dab:::benchmark_data_smithi177_31116_object138293:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e140)", 2024-09-13T11:26:17.556 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:24:54.250266+0000", 2024-09-13T11:26:17.556 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 83.243392197000006, 2024-09-13T11:26:17.556 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.33165556200000001, 2024-09-13T11:26:17.556 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:26:17.556 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:26:17.556 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:17.556 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:26:17.557 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:26:17.557 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:17.557 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 138294 2024-09-13T11:26:17.557 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.557 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:26:17.557 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.557 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:26:17.557 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:54.250266+0000", 2024-09-13T11:26:17.557 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:17.557 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.557 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.557 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:26:17.557 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:54.250266+0000", 2024-09-13T11:26:17.558 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:17.558 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.558 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.558 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:26:17.558 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:54.250269+0000", 2024-09-13T11:26:17.558 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.3159999999999999e-06 2024-09-13T11:26:17.558 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.558 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.558 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:26:17.558 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:54.250362+0000", 2024-09-13T11:26:17.558 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.3051000000000001e-05 2024-09-13T11:26:17.558 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.558 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.559 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:26:17.559 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:54.250374+0000", 2024-09-13T11:26:17.559 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2327000000000001e-05 2024-09-13T11:26:17.559 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.559 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.559 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:26:17.559 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:54.250382+0000", 2024-09-13T11:26:17.559 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.7249999999999997e-06 2024-09-13T11:26:17.559 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.559 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.559 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:26:17.559 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:54.250459+0000", 2024-09-13T11:26:17.559 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.7218999999999994e-05 2024-09-13T11:26:17.559 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.560 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.560 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:26:17.560 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:54.250611+0000", 2024-09-13T11:26:17.560 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00015241000000000001 2024-09-13T11:26:17.560 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.560 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.560 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 2", 2024-09-13T11:26:17.560 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:54.250777+0000", 2024-09-13T11:26:17.560 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00016575199999999999 2024-09-13T11:26:17.560 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.560 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.560 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:26:17.560 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:54.252776+0000", 2024-09-13T11:26:17.560 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0019987260000000001 2024-09-13T11:26:17.561 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.561 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.561 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:17.561 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:54.581758+0000", 2024-09-13T11:26:17.561 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.32898232799999999 2024-09-13T11:26:17.561 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.561 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.561 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:26:17.561 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:54.581854+0000", 2024-09-13T11:26:17.561 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.5468000000000001e-05 2024-09-13T11:26:17.561 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.561 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.561 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:26:17.561 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:54.581922+0000", 2024-09-13T11:26:17.562 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.8239999999999997e-05 2024-09-13T11:26:17.562 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:17.562 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:17.562 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:17.562 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.562 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.562 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop_reply(client.4665.0:140790 3.20 e144/77 ondisk, result = 0)", 2024-09-13T11:26:17.562 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:24:58.385972+0000", 2024-09-13T11:26:17.562 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 79.107686629, 2024-09-13T11:26:17.562 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.51829568500000001, 2024-09-13T11:26:17.562 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:26:17.562 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:26:17.562 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:26:17.562 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:26:17.563 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.563 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:26:17.563 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.385972+0000", 2024-09-13T11:26:17.563 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:17.563 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.563 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.563 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:26:17.563 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.385972+0000", 2024-09-13T11:26:17.563 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:17.563 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.563 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.563 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:26:17.563 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.385974+0000", 2024-09-13T11:26:17.563 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.7400000000000001e-06 2024-09-13T11:26:17.564 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.564 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.564 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:26:17.564 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.385979+0000", 2024-09-13T11:26:17.564 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.091e-06 2024-09-13T11:26:17.564 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.564 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.564 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:26:17.564 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.904091+0000", 2024-09-13T11:26:17.564 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.51811194900000002 2024-09-13T11:26:17.564 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.564 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.564 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:26:17.564 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.904098+0000", 2024-09-13T11:26:17.565 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.0609999999999998e-06 2024-09-13T11:26:17.565 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.565 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.565 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:26:17.565 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.904156+0000", 2024-09-13T11:26:17.565 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.7902000000000002e-05 2024-09-13T11:26:17.565 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.565 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.565 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:26:17.565 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.904163+0000", 2024-09-13T11:26:17.565 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.1430000000000001e-06 2024-09-13T11:26:17.565 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.565 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.565 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:26:17.566 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.904267+0000", 2024-09-13T11:26:17.566 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000104799 2024-09-13T11:26:17.566 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:17.566 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:17.566 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:17.566 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.566 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.566 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4665.0:191082 3.26 e175/140 3:67bd986b:::benchmark_data_smithi177_31116_object50779:head v 175'6183, mlcod=175'6180)", 2024-09-13T11:26:17.566 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:25:38.551017+0000", 2024-09-13T11:26:17.566 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 38.942641209000001, 2024-09-13T11:26:17.566 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.29063407899999999, 2024-09-13T11:26:17.566 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:26:17.566 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:26:17.566 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:17.567 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:26:17.567 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.567 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:26:17.567 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:38.551017+0000", 2024-09-13T11:26:17.567 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:17.567 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.567 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.567 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:26:17.567 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:38.551017+0000", 2024-09-13T11:26:17.567 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:17.567 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.567 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.567 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:26:17.568 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:38.551020+0000", 2024-09-13T11:26:17.568 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.9170000000000002e-06 2024-09-13T11:26:17.568 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.568 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.568 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:26:17.568 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:38.551033+0000", 2024-09-13T11:26:17.568 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2843000000000001e-05 2024-09-13T11:26:17.568 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.568 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.568 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:26:17.568 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:38.838514+0000", 2024-09-13T11:26:17.568 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.28748061600000002 2024-09-13T11:26:17.568 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.568 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.569 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:26:17.569 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:38.838532+0000", 2024-09-13T11:26:17.569 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.8012999999999998e-05 2024-09-13T11:26:17.569 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.569 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.569 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:26:17.569 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:38.838646+0000", 2024-09-13T11:26:17.569 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000114596 2024-09-13T11:26:17.569 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.569 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.569 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:26:17.569 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:38.838681+0000", 2024-09-13T11:26:17.569 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.5099999999999999e-05 2024-09-13T11:26:17.570 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.570 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.570 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:26:17.570 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:38.841593+0000", 2024-09-13T11:26:17.570 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0029117050000000001 2024-09-13T11:26:17.570 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.570 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.570 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:26:17.570 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:38.841651+0000", 2024-09-13T11:26:17.570 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.8288999999999997e-05 2024-09-13T11:26:17.570 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:17.570 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:17.570 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:17.570 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.570 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.571 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4665.0:191198 3.26 e175/140 3:64ff4057:::benchmark_data_smithi177_31116_object50895:head v 175'6185, mlcod=175'6182)", 2024-09-13T11:26:17.571 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:25:38.948739+0000", 2024-09-13T11:26:17.571 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 38.544919747999998, 2024-09-13T11:26:17.571 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.25772718100000003, 2024-09-13T11:26:17.571 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:26:17.571 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:26:17.571 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:17.571 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:26:17.571 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.571 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:26:17.571 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:38.948739+0000", 2024-09-13T11:26:17.571 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:17.571 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.572 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.572 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:26:17.572 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:38.948739+0000", 2024-09-13T11:26:17.572 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:17.572 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.572 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.572 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:26:17.572 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:38.948743+0000", 2024-09-13T11:26:17.572 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.0600000000000001e-06 2024-09-13T11:26:17.572 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.572 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.572 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:26:17.572 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:38.948760+0000", 2024-09-13T11:26:17.572 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.7540000000000001e-05 2024-09-13T11:26:17.573 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.573 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.573 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:26:17.573 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:39.204732+0000", 2024-09-13T11:26:17.573 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.25597220199999998 2024-09-13T11:26:17.573 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.573 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.573 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:26:17.573 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:39.204742+0000", 2024-09-13T11:26:17.573 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.6579999999999997e-06 2024-09-13T11:26:17.573 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.573 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.573 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:26:17.573 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:39.204834+0000", 2024-09-13T11:26:17.574 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.1565999999999997e-05 2024-09-13T11:26:17.574 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.574 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.574 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:26:17.574 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:39.204854+0000", 2024-09-13T11:26:17.574 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.0809e-05 2024-09-13T11:26:17.574 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.574 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.574 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:26:17.574 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:39.206423+0000", 2024-09-13T11:26:17.574 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0015685219999999999 2024-09-13T11:26:17.574 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.574 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.574 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:26:17.575 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:39.206466+0000", 2024-09-13T11:26:17.575 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.2824000000000002e-05 2024-09-13T11:26:17.575 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:17.575 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:17.575 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:17.575 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.575 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.575 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:196438 3.20 3:0573f1a4:::benchmark_data_smithi177_31116_object56135:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e178)", 2024-09-13T11:26:17.575 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:25:43.521712+0000", 2024-09-13T11:26:17.575 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 33.971946649000003, 2024-09-13T11:26:17.575 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.23827029799999999, 2024-09-13T11:26:17.575 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:26:17.575 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:26:17.576 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:17.576 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:26:17.576 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:26:17.576 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:17.576 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 196438 2024-09-13T11:26:17.576 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.576 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:26:17.576 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.576 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:26:17.576 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:43.521712+0000", 2024-09-13T11:26:17.576 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:17.576 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.576 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.576 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:26:17.577 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:43.521712+0000", 2024-09-13T11:26:17.577 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:17.577 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.577 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.577 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:26:17.577 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:43.521714+0000", 2024-09-13T11:26:17.577 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.6970000000000002e-06 2024-09-13T11:26:17.577 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.577 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.577 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:26:17.577 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:43.521723+0000", 2024-09-13T11:26:17.577 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.1319999999999994e-06 2024-09-13T11:26:17.577 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.578 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.578 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:26:17.578 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:43.521743+0000", 2024-09-13T11:26:17.578 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.0446e-05 2024-09-13T11:26:17.578 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.578 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.578 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:26:17.578 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:43.521754+0000", 2024-09-13T11:26:17.578 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.1314e-05 2024-09-13T11:26:17.578 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.578 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.578 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:26:17.578 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:43.521839+0000", 2024-09-13T11:26:17.578 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.4951999999999996e-05 2024-09-13T11:26:17.579 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.579 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.579 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:26:17.579 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:43.522010+0000", 2024-09-13T11:26:17.579 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000171053 2024-09-13T11:26:17.579 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.579 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.579 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 0", 2024-09-13T11:26:17.579 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:43.522160+0000", 2024-09-13T11:26:17.579 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000149987 2024-09-13T11:26:17.579 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.579 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.579 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:26:17.579 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:43.524717+0000", 2024-09-13T11:26:17.579 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0025565649999999998 2024-09-13T11:26:17.580 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.580 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.580 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:17.580 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:43.759800+0000", 2024-09-13T11:26:17.580 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.23508320699999999 2024-09-13T11:26:17.580 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.580 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.580 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:26:17.580 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:43.759917+0000", 2024-09-13T11:26:17.580 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000116605 2024-09-13T11:26:17.580 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.580 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.580 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:26:17.580 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:43.759982+0000", 2024-09-13T11:26:17.581 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.5339999999999994e-05 2024-09-13T11:26:17.581 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:17.581 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:17.581 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:17.581 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.581 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.581 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:211295 3.2 3:433aa750:::benchmark_data_smithi177_31116_object70992:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e185)", 2024-09-13T11:26:17.581 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:25:55.459639+0000", 2024-09-13T11:26:17.581 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 22.034019399000002, 2024-09-13T11:26:17.581 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.28751246200000002, 2024-09-13T11:26:17.581 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:26:17.581 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:26:17.581 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:17.582 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:26:17.582 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:26:17.582 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:17.582 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 211295 2024-09-13T11:26:17.582 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.582 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:26:17.582 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.582 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:26:17.582 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:55.459639+0000", 2024-09-13T11:26:17.582 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:17.582 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.582 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.582 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:26:17.582 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:55.459639+0000", 2024-09-13T11:26:17.583 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:17.583 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.583 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.583 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:26:17.583 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:55.459642+0000", 2024-09-13T11:26:17.583 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.5940000000000001e-06 2024-09-13T11:26:17.583 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.583 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.583 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:26:17.583 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:55.459649+0000", 2024-09-13T11:26:17.583 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.0899999999999999e-06 2024-09-13T11:26:17.583 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.583 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.584 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:26:17.584 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:55.459669+0000", 2024-09-13T11:26:17.584 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.0305999999999998e-05 2024-09-13T11:26:17.584 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.584 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.584 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:26:17.584 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:55.459683+0000", 2024-09-13T11:26:17.584 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.4219999999999999e-05 2024-09-13T11:26:17.584 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.584 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.584 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:26:17.584 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:55.459761+0000", 2024-09-13T11:26:17.584 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.8072999999999996e-05 2024-09-13T11:26:17.584 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.585 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.585 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:26:17.585 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:55.459912+0000", 2024-09-13T11:26:17.585 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000150304 2024-09-13T11:26:17.585 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.585 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.585 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 3", 2024-09-13T11:26:17.585 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:55.460013+0000", 2024-09-13T11:26:17.585 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000101828 2024-09-13T11:26:17.585 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.585 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.585 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:26:17.585 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:55.462204+0000", 2024-09-13T11:26:17.585 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.002191082 2024-09-13T11:26:17.586 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.586 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.586 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:17.586 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:55.747086+0000", 2024-09-13T11:26:17.586 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.28488175100000002 2024-09-13T11:26:17.586 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.586 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.586 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:26:17.586 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:55.747120+0000", 2024-09-13T11:26:17.586 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.3448000000000002e-05 2024-09-13T11:26:17.586 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.586 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.586 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:26:17.586 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:55.747151+0000", 2024-09-13T11:26:17.586 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.1766e-05 2024-09-13T11:26:17.587 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:17.587 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:17.587 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:17.587 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.587 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.587 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop_reply(client.4665.0:221502 3.20 e190/77 ondisk, result = 0)", 2024-09-13T11:26:17.587 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:26:03.873964+0000", 2024-09-13T11:26:17.587 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 13.619694844, 2024-09-13T11:26:17.587 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.48664245099999998, 2024-09-13T11:26:17.587 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:26:17.587 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:26:17.587 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:26:17.587 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:26:17.587 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.588 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:26:17.588 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:03.873964+0000", 2024-09-13T11:26:17.588 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:17.588 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.588 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.588 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:26:17.588 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:03.873964+0000", 2024-09-13T11:26:17.588 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:17.588 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.588 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.588 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:26:17.588 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:03.873965+0000", 2024-09-13T11:26:17.588 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.801e-06 2024-09-13T11:26:17.588 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.589 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.589 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:26:17.589 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:03.873972+0000", 2024-09-13T11:26:17.589 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.3029999999999996e-06 2024-09-13T11:26:17.589 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.589 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.589 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:26:17.589 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:04.360178+0000", 2024-09-13T11:26:17.589 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.48620634400000001 2024-09-13T11:26:17.589 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.589 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.589 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:26:17.589 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:04.360183+0000", 2024-09-13T11:26:17.589 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.2669999999999997e-06 2024-09-13T11:26:17.590 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.590 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.590 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:26:17.590 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:04.360368+0000", 2024-09-13T11:26:17.590 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00018470100000000001 2024-09-13T11:26:17.590 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.590 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.590 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:26:17.590 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:04.360387+0000", 2024-09-13T11:26:17.590 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.874e-05 2024-09-13T11:26:17.590 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.590 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.590 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:26:17.591 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:04.360606+0000", 2024-09-13T11:26:17.591 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00021929499999999999 2024-09-13T11:26:17.591 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:17.591 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:17.591 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:17.591 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.591 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.591 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4665.0:226607 3.4 e196/117 3:218872c8:::benchmark_data_smithi177_31116_object86304:head v 196'6594, mlcod=195'6591)", 2024-09-13T11:26:17.591 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:26:10.278888+0000", 2024-09-13T11:26:17.591 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 7.2147700930000003, 2024-09-13T11:26:17.591 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.25848660899999998, 2024-09-13T11:26:17.591 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:26:17.591 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:26:17.591 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:17.591 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:26:17.592 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.592 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:26:17.592 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:10.278888+0000", 2024-09-13T11:26:17.592 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:17.592 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.592 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.592 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:26:17.592 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:10.278888+0000", 2024-09-13T11:26:17.592 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:17.592 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.592 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.592 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:26:17.592 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:10.278890+0000", 2024-09-13T11:26:17.592 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.392e-06 2024-09-13T11:26:17.593 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.593 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.593 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:26:17.593 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:10.278896+0000", 2024-09-13T11:26:17.593 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.562e-06 2024-09-13T11:26:17.593 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.593 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.593 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:26:17.593 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:10.535554+0000", 2024-09-13T11:26:17.593 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.256657723 2024-09-13T11:26:17.593 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.593 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.593 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:26:17.593 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:10.535559+0000", 2024-09-13T11:26:17.594 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.4720000000000003e-06 2024-09-13T11:26:17.594 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.594 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.594 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:26:17.594 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:10.535621+0000", 2024-09-13T11:26:17.594 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.2012000000000002e-05 2024-09-13T11:26:17.594 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.594 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.594 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:26:17.594 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:10.535644+0000", 2024-09-13T11:26:17.594 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.2911000000000001e-05 2024-09-13T11:26:17.594 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.594 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.594 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:26:17.595 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:10.537316+0000", 2024-09-13T11:26:17.595 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.001671692 2024-09-13T11:26:17.595 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:17.595 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:17.595 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:26:17.595 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:10.537375+0000", 2024-09-13T11:26:17.595 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.8845000000000003e-05 2024-09-13T11:26:17.595 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:17.595 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:17.595 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:17.595 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:17.595 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:17.595 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-13T11:26:17.597 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-13T11:26:17.597 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-13T11:26:17.594+0000 7f6a68775640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26307) UID: 0 2024-09-13T11:26:17.698 INFO:tasks.ceph.osd.4.smithi001.stderr:2024-09-13T11:26:17.694+0000 7ff8e0218640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26310) UID: 0 2024-09-13T11:26:17.743 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-09-13T11:26:17.743 INFO:teuthology.orchestra.run.smithi092.stdout: "ops": [], 2024-09-13T11:26:17.743 INFO:teuthology.orchestra.run.smithi092.stdout: "num_ops": 0 2024-09-13T11:26:17.744 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-09-13T11:26:17.761 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-13T11:26:17.798 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-13T11:26:17.794+0000 7f6a68775640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26307) UID: 0 2024-09-13T11:26:17.834 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T11:26:17.899 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-09-13T11:26:17.900 INFO:teuthology.orchestra.run.smithi092.stdout: "ops": [], 2024-09-13T11:26:17.900 INFO:teuthology.orchestra.run.smithi092.stdout: "complaint_time": 30, 2024-09-13T11:26:17.900 INFO:teuthology.orchestra.run.smithi092.stdout: "num_blocked_ops": 0 2024-09-13T11:26:17.900 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-09-13T11:26:17.900 INFO:tasks.ceph.osd.8.smithi001.stderr:2024-09-13T11:26:17.898+0000 7fb9b48ae640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26312) UID: 0 2024-09-13T11:26:17.915 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-13T11:26:18.001 INFO:tasks.ceph.osd.13.smithi092.stderr:2024-09-13T11:26:18.000+0000 7f3e5afea640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26350) UID: 0 2024-09-13T11:26:18.007 INFO:teuthology.orchestra.run.smithi001.stdout:osd.0: {} 2024-09-13T11:26:18.008 INFO:teuthology.orchestra.run.smithi001.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T11:26:18.030 INFO:teuthology.orchestra.run.smithi001.stdout:osd.1: {} 2024-09-13T11:26:18.030 INFO:teuthology.orchestra.run.smithi001.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T11:26:18.046 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-09-13T11:26:18.047 INFO:teuthology.orchestra.run.smithi092.stdout: "size": 20, 2024-09-13T11:26:18.047 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 600, 2024-09-13T11:26:18.047 INFO:teuthology.orchestra.run.smithi092.stdout: "ops": [ 2024-09-13T11:26:18.047 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:18.047 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "MOSDPGPush(3.23 83/78 [PushOp(3:c61f6bab:::benchmark_data_smithi177_31116_object29479:head, version: 59'1860, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:c61f6bab:::benchmark_data_smithi177_31116_object29479:head@59'1860, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:26:18.047 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:40.770864+0000", 2024-09-13T11:26:18.047 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 157.273190701, 2024-09-13T11:26:18.047 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.31229662800000002, 2024-09-13T11:26:18.047 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:26:18.047 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:26:18.047 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-09-13T11:26:18.047 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:26:18.047 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:18.048 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:26:18.048 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:40.770864+0000", 2024-09-13T11:26:18.048 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:18.048 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:18.048 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:18.048 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:26:18.048 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:40.770864+0000", 2024-09-13T11:26:18.048 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:18.048 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:18.048 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:18.048 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:26:18.048 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:40.770865+0000", 2024-09-13T11:26:18.048 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.288e-06 2024-09-13T11:26:18.049 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:18.049 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:18.049 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:26:18.049 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:40.770955+0000", 2024-09-13T11:26:18.049 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.9871999999999996e-05 2024-09-13T11:26:18.049 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:18.049 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:18.049 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:26:18.049 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:41.082105+0000", 2024-09-13T11:26:18.049 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.31114963400000001 2024-09-13T11:26:18.049 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:18.049 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:18.049 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:26:18.050 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:41.082136+0000", 2024-09-13T11:26:18.050 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.0972000000000001e-05 2024-09-13T11:26:18.050 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:18.050 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:18.050 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:18.050 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:41.082404+0000", 2024-09-13T11:26:18.050 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00026844800000000002 2024-09-13T11:26:18.050 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:18.050 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:18.050 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:26:18.050 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:41.082429+0000", 2024-09-13T11:26:18.050 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.5021e-05 2024-09-13T11:26:18.050 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:18.051 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:18.051 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:26:18.051 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:41.083161+0000", 2024-09-13T11:26:18.051 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000731393 2024-09-13T11:26:18.051 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:18.051 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:18.051 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:18.051 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:18.051 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:18.051 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:110289 3.29 3.27287969 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e119)", 2024-09-13T11:26:18.051 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:24:25.774634+0000", 2024-09-13T11:26:18.051 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 112.269420626, 2024-09-13T11:26:18.051 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.31721633999999999, 2024-09-13T11:26:18.051 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:26:18.051 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:26:18.052 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "reached pg", 2024-09-13T11:26:18.052 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:26:18.052 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:26:18.052 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:18.052 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 110289 2024-09-13T11:26:18.052 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:18.052 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:26:18.052 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:18.052 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:26:18.052 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:25.774634+0000", 2024-09-13T11:26:18.052 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:18.052 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:18.053 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:18.053 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:26:18.053 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:25.774634+0000", 2024-09-13T11:26:18.053 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:18.053 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:18.053 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:18.053 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:26:18.053 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:25.774639+0000", 2024-09-13T11:26:18.053 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.6029999999999998e-06 2024-09-13T11:26:18.053 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:18.053 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:18.053 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:26:18.053 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:25.774695+0000", 2024-09-13T11:26:18.054 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.6694999999999998e-05 2024-09-13T11:26:18.054 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:18.054 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:18.054 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:26:18.054 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:25.774728+0000", 2024-09-13T11:26:18.054 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.2502000000000002e-05 2024-09-13T11:26:18.054 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:18.054 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:18.054 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:26:18.054 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:25.774750+0000", 2024-09-13T11:26:18.054 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.2002999999999999e-05 2024-09-13T11:26:18.054 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:18.054 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:18.055 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:18.055 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:25.774942+0000", 2024-09-13T11:26:18.055 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000192265 2024-09-13T11:26:18.055 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:18.055 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:18.055 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-09-13T11:26:18.055 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:25.774960+0000", 2024-09-13T11:26:18.055 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.8275999999999999e-05 2024-09-13T11:26:18.055 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:18.055 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:18.055 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:18.056 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:26.091821+0000", 2024-09-13T11:26:18.056 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.31686065000000002 2024-09-13T11:26:18.056 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:18.056 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:18.056 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:26:18.056 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:26.091850+0000", 2024-09-13T11:26:18.056 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.9346e-05 2024-09-13T11:26:18.056 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:18.056 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:18.056 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:18.056 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:18.056 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:18.057 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:138370 3.23 3:c5730cd2:::benchmark_data_smithi177_31116_object138369:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e140)", 2024-09-13T11:26:18.057 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:24:54.415176+0000", 2024-09-13T11:26:18.057 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 83.628878529999994, 2024-09-13T11:26:18.057 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.24549895499999999, 2024-09-13T11:26:18.057 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:26:18.057 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:26:18.057 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:18.057 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:26:18.057 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:26:18.057 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:18.057 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 138370 2024-09-13T11:26:18.057 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:18.058 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:26:18.058 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:18.058 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:26:18.058 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:54.415176+0000", 2024-09-13T11:26:18.058 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:18.058 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:18.058 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:18.058 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:26:18.058 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:54.415176+0000", 2024-09-13T11:26:18.058 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:18.058 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:18.058 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:18.058 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:26:18.059 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:54.415180+0000", 2024-09-13T11:26:18.059 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.1670000000000001e-06 2024-09-13T11:26:18.059 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:18.059 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:18.059 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:26:18.059 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:54.415269+0000", 2024-09-13T11:26:18.059 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.8659000000000006e-05 2024-09-13T11:26:18.059 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:18.059 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:18.059 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:26:18.059 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:54.415292+0000", 2024-09-13T11:26:18.059 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.3152000000000001e-05 2024-09-13T11:26:18.060 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:18.060 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:18.060 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:26:18.060 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:54.415304+0000", 2024-09-13T11:26:18.060 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.1841999999999999e-05 2024-09-13T11:26:18.060 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:18.060 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:18.060 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:18.060 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:54.415432+0000", 2024-09-13T11:26:18.060 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00012802000000000001 2024-09-13T11:26:18.060 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:18.060 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:18.060 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:26:18.060 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:54.415710+0000", 2024-09-13T11:26:18.061 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00027840299999999998 2024-09-13T11:26:18.061 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:18.061 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:18.061 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 9", 2024-09-13T11:26:18.061 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:54.415950+0000", 2024-09-13T11:26:18.061 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00023928299999999999 2024-09-13T11:26:18.061 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:18.061 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:18.061 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:26:18.061 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:54.419265+0000", 2024-09-13T11:26:18.061 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0033150440000000001 2024-09-13T11:26:18.061 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:18.061 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:18.062 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:18.062 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:54.660448+0000", 2024-09-13T11:26:18.062 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.24118348100000001 2024-09-13T11:26:18.062 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:18.062 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:18.062 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:26:18.062 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:54.660571+0000", 2024-09-13T11:26:18.062 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00012332900000000001 2024-09-13T11:26:18.062 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:18.062 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:18.062 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:26:18.062 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:54.660675+0000", 2024-09-13T11:26:18.062 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000103575 2024-09-13T11:26:18.062 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:18.063 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:18.063 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:18.063 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:18.063 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:18.063 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:140742 3.1e 3:79c694a0:::benchmark_data_smithi177_31116_object439:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e144)", 2024-09-13T11:26:18.063 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:24:58.321262+0000", 2024-09-13T11:26:18.063 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 79.722792112999997, 2024-09-13T11:26:18.063 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.23727013499999999, 2024-09-13T11:26:18.063 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:26:18.063 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:26:18.063 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:18.063 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:26:18.064 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:26:18.064 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:18.064 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 140742 2024-09-13T11:26:18.064 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:18.064 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:26:18.064 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:18.064 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:26:18.064 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:58.321262+0000", 2024-09-13T11:26:18.064 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:18.064 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:18.064 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:18.064 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:26:18.064 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:58.321262+0000", 2024-09-13T11:26:18.065 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:18.065 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:18.065 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:18.065 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:26:18.065 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:58.321264+0000", 2024-09-13T11:26:18.065 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.581e-06 2024-09-13T11:26:18.065 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:18.065 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:18.065 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:26:18.065 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:58.321272+0000", 2024-09-13T11:26:18.065 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.2179999999999999e-06 2024-09-13T11:26:18.065 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:18.066 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:18.066 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:26:18.066 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:58.321282+0000", 2024-09-13T11:26:18.066 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0173000000000001e-05 2024-09-13T11:26:18.066 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:18.066 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:18.066 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:26:18.066 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:58.321290+0000", 2024-09-13T11:26:18.066 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.943e-06 2024-09-13T11:26:18.066 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:18.066 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:18.066 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:18.066 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:58.321355+0000", 2024-09-13T11:26:18.067 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.4461000000000005e-05 2024-09-13T11:26:18.067 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:18.067 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:18.067 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:26:18.067 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:58.321450+0000", 2024-09-13T11:26:18.067 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.4769e-05 2024-09-13T11:26:18.067 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:18.067 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:18.067 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 4", 2024-09-13T11:26:18.067 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:58.321516+0000", 2024-09-13T11:26:18.067 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.5998000000000001e-05 2024-09-13T11:26:18.067 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:18.067 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:18.067 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:26:18.068 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:58.324045+0000", 2024-09-13T11:26:18.068 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0025290579999999998 2024-09-13T11:26:18.068 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:18.068 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:18.068 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:18.068 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:58.558244+0000", 2024-09-13T11:26:18.068 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.234199455 2024-09-13T11:26:18.068 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:18.068 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:18.068 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:26:18.068 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:58.558401+0000", 2024-09-13T11:26:18.068 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00015700800000000001 2024-09-13T11:26:18.069 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:18.069 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:18.069 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:26:18.069 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:58.558533+0000", 2024-09-13T11:26:18.069 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000131471 2024-09-13T11:26:18.069 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:18.069 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:18.069 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:18.069 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:18.069 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:18.069 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:140796 3.1e 3:7987a652:::benchmark_data_smithi177_31116_object493:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e144)", 2024-09-13T11:26:18.069 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:24:58.552900+0000", 2024-09-13T11:26:18.069 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 79.491154578000007, 2024-09-13T11:26:18.070 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.254397398, 2024-09-13T11:26:18.070 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:26:18.070 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:26:18.070 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:18.070 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:26:18.070 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:26:18.070 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:18.070 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 140796 2024-09-13T11:26:18.070 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:18.070 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:26:18.070 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:18.070 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:26:18.070 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:58.552900+0000", 2024-09-13T11:26:18.070 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:18.071 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:18.071 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:18.071 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:26:18.071 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:58.552900+0000", 2024-09-13T11:26:18.071 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:18.071 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:18.071 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:18.071 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:26:18.071 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:58.552904+0000", 2024-09-13T11:26:18.071 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.76e-06 2024-09-13T11:26:18.071 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:18.071 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:18.071 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:26:18.071 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:58.552915+0000", 2024-09-13T11:26:18.072 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0763e-05 2024-09-13T11:26:18.072 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:18.072 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:18.072 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:26:18.072 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:58.552943+0000", 2024-09-13T11:26:18.072 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.8469000000000001e-05 2024-09-13T11:26:18.072 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:18.072 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:18.072 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:26:18.072 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:58.552959+0000", 2024-09-13T11:26:18.072 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.5549999999999999e-05 2024-09-13T11:26:18.072 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:18.072 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:18.073 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:18.073 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:58.553106+0000", 2024-09-13T11:26:18.073 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00014729500000000001 2024-09-13T11:26:18.073 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:18.073 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:18.073 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:26:18.073 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:58.553348+0000", 2024-09-13T11:26:18.073 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000242353 2024-09-13T11:26:18.073 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:18.073 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:18.073 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 4", 2024-09-13T11:26:18.073 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:58.553556+0000", 2024-09-13T11:26:18.074 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00020765700000000001 2024-09-13T11:26:18.074 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:18.074 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:18.074 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:26:18.074 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:58.557140+0000", 2024-09-13T11:26:18.074 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0035844840000000002 2024-09-13T11:26:18.074 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:18.074 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:18.074 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:18.074 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:58.807126+0000", 2024-09-13T11:26:18.074 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.24998542000000001 2024-09-13T11:26:18.074 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:18.074 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:18.075 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:26:18.075 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:58.807214+0000", 2024-09-13T11:26:18.075 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.8561999999999997e-05 2024-09-13T11:26:18.075 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:18.075 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:18.075 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:26:18.075 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:58.807297+0000", 2024-09-13T11:26:18.075 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.3084999999999995e-05 2024-09-13T11:26:18.075 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:18.075 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:18.075 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:18.075 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:18.075 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:18.075 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop(client.4665.0:162073 3.1f e156/122 3:fd1a00ae:::benchmark_data_smithi177_31116_object21770:head v 156'6684, mlcod=156'6679)", 2024-09-13T11:26:18.076 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:25:15.455533+0000", 2024-09-13T11:26:18.076 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 62.588521210000003, 2024-09-13T11:26:18.076 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.24315087499999999, 2024-09-13T11:26:18.076 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:26:18.076 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:26:18.076 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:18.076 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:26:18.076 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:18.076 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:26:18.076 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:15.455533+0000", 2024-09-13T11:26:18.076 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:18.076 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:18.076 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:18.076 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:26:18.077 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:15.455533+0000", 2024-09-13T11:26:18.077 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:18.077 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:18.077 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:18.077 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:26:18.077 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:15.455535+0000", 2024-09-13T11:26:18.077 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.42e-06 2024-09-13T11:26:18.077 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:18.077 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:18.077 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:26:18.077 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:15.455542+0000", 2024-09-13T11:26:18.077 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.9820000000000002e-06 2024-09-13T11:26:18.077 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:18.078 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:18.078 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:26:18.078 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:15.696032+0000", 2024-09-13T11:26:18.078 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.240490483 2024-09-13T11:26:18.078 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:18.078 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:18.078 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:26:18.078 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:15.696039+0000", 2024-09-13T11:26:18.078 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.832e-06 2024-09-13T11:26:18.078 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:18.078 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:18.078 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:18.078 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:15.696098+0000", 2024-09-13T11:26:18.079 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.8901999999999999e-05 2024-09-13T11:26:18.079 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:18.079 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:18.079 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:26:18.079 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:15.696115+0000", 2024-09-13T11:26:18.079 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.7479999999999999e-05 2024-09-13T11:26:18.079 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:18.079 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:18.079 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:26:18.079 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:15.698637+0000", 2024-09-13T11:26:18.079 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0025215889999999999 2024-09-13T11:26:18.079 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:18.079 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:18.079 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:26:18.080 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:15.698684+0000", 2024-09-13T11:26:18.080 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.7187000000000001e-05 2024-09-13T11:26:18.080 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:18.080 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:18.080 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:18.080 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:18.080 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:18.080 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop(client.4665.0:162102 3.1f e156/122 3:fc0cda6c:::benchmark_data_smithi177_31116_object21799:head v 156'6685, mlcod=156'6680)", 2024-09-13T11:26:18.080 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:25:15.477994+0000", 2024-09-13T11:26:18.080 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 62.566060409000002, 2024-09-13T11:26:18.080 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.25110996400000002, 2024-09-13T11:26:18.080 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:26:18.081 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:26:18.081 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:18.081 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:26:18.081 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:18.081 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:26:18.081 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:15.477994+0000", 2024-09-13T11:26:18.081 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:18.081 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:18.081 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:18.081 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:26:18.081 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:15.477994+0000", 2024-09-13T11:26:18.081 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:18.081 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:18.081 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:18.082 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:26:18.082 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:15.477995+0000", 2024-09-13T11:26:18.082 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0210000000000001e-06 2024-09-13T11:26:18.082 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:18.082 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:18.082 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:26:18.082 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:15.478001+0000", 2024-09-13T11:26:18.082 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.0020000000000003e-06 2024-09-13T11:26:18.082 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:18.082 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:18.082 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:26:18.082 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:15.726384+0000", 2024-09-13T11:26:18.082 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.248382609 2024-09-13T11:26:18.082 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:18.083 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:18.083 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:26:18.083 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:15.726390+0000", 2024-09-13T11:26:18.083 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.1029999999999998e-06 2024-09-13T11:26:18.083 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:18.083 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:18.083 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:18.083 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:15.727107+0000", 2024-09-13T11:26:18.083 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000717525 2024-09-13T11:26:18.083 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:18.083 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:18.083 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:26:18.083 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:15.727130+0000", 2024-09-13T11:26:18.084 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.2572999999999999e-05 2024-09-13T11:26:18.084 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:18.084 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:18.084 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:26:18.084 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:15.729066+0000", 2024-09-13T11:26:18.084 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0019362170000000001 2024-09-13T11:26:18.084 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:18.084 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:18.084 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:26:18.084 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:15.729104+0000", 2024-09-13T11:26:18.084 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.7914000000000003e-05 2024-09-13T11:26:18.084 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:18.084 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:18.084 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:18.085 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:18.085 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:18.085 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "MOSDPGPush(3.28 168/120 [PushOp(3:14eb8ed9:::benchmark_data_smithi177_31116_object87538:head, version: 110'4071, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:14eb8ed9:::benchmark_data_smithi177_31116_object87538:head@110'4071, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:26:18.085 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:25:27.274802+0000", 2024-09-13T11:26:18.085 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 50.769252860999998, 2024-09-13T11:26:18.085 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.369268239, 2024-09-13T11:26:18.085 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:26:18.085 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:26:18.085 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-09-13T11:26:18.085 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:26:18.085 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:18.085 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:26:18.085 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:27.274802+0000", 2024-09-13T11:26:18.085 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:18.086 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:18.086 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:18.086 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:26:18.086 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:27.274802+0000", 2024-09-13T11:26:18.086 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:18.086 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:18.086 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:18.086 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:26:18.086 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:27.274803+0000", 2024-09-13T11:26:18.086 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.7579999999999999e-06 2024-09-13T11:26:18.086 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:18.086 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:18.086 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:26:18.086 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:27.274831+0000", 2024-09-13T11:26:18.087 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.7464e-05 2024-09-13T11:26:18.087 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:18.087 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:18.087 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:26:18.087 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:27.641355+0000", 2024-09-13T11:26:18.087 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.36652425199999999 2024-09-13T11:26:18.087 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:18.087 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:18.087 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:26:18.087 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:27.641385+0000", 2024-09-13T11:26:18.087 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.9433000000000001e-05 2024-09-13T11:26:18.087 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:18.087 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:18.087 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:18.088 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:27.641611+0000", 2024-09-13T11:26:18.088 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00022646299999999999 2024-09-13T11:26:18.088 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:18.088 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:18.088 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:26:18.088 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:27.641638+0000", 2024-09-13T11:26:18.088 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.6482000000000002e-05 2024-09-13T11:26:18.088 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:18.088 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:18.088 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:26:18.088 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:27.644070+0000", 2024-09-13T11:26:18.088 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0024323869999999998 2024-09-13T11:26:18.088 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:18.088 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:18.089 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:18.089 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:18.089 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:18.089 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "MOSDPGPush(3.2d 174/120 [PushOp(3:b42ee771:::benchmark_data_smithi177_31116_object109317:head, version: 118'5087, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:b42ee771:::benchmark_data_smithi177_31116_object109317:head@118'5087, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:26:18.089 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:25:36.282270+0000", 2024-09-13T11:26:18.089 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 41.761784243999998, 2024-09-13T11:26:18.089 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.26768705100000001, 2024-09-13T11:26:18.089 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:26:18.089 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:26:18.089 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-09-13T11:26:18.089 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:26:18.089 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:18.089 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:26:18.090 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:36.282270+0000", 2024-09-13T11:26:18.090 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:18.090 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:18.090 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:18.090 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:26:18.090 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:36.282270+0000", 2024-09-13T11:26:18.090 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:18.090 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:18.090 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:18.090 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:26:18.090 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:36.282272+0000", 2024-09-13T11:26:18.090 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.832e-06 2024-09-13T11:26:18.090 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:18.090 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:18.091 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:26:18.091 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:36.282337+0000", 2024-09-13T11:26:18.091 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.4834999999999998e-05 2024-09-13T11:26:18.091 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:18.091 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:18.091 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:26:18.091 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:36.547263+0000", 2024-09-13T11:26:18.091 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.26492590599999999 2024-09-13T11:26:18.091 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:18.091 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:18.091 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:26:18.091 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:36.547288+0000", 2024-09-13T11:26:18.091 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.4661999999999999e-05 2024-09-13T11:26:18.092 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:18.092 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:18.092 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:18.092 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:36.547493+0000", 2024-09-13T11:26:18.092 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00020533500000000001 2024-09-13T11:26:18.092 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:18.092 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:18.092 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:26:18.092 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:36.547515+0000", 2024-09-13T11:26:18.092 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.2124999999999999e-05 2024-09-13T11:26:18.092 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:18.092 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:18.092 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:26:18.092 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:36.549957+0000", 2024-09-13T11:26:18.093 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0024423560000000001 2024-09-13T11:26:18.093 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:18.093 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:18.093 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:18.093 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:18.093 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:18.093 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:191021 3.23 3:c4df18dd:::benchmark_data_smithi177_31116_object50718:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e175)", 2024-09-13T11:26:18.093 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:25:38.460411+0000", 2024-09-13T11:26:18.093 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 39.583643436000003, 2024-09-13T11:26:18.093 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.40138001400000001, 2024-09-13T11:26:18.093 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:26:18.093 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:26:18.093 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:18.094 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:26:18.094 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:26:18.094 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:18.094 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 191021 2024-09-13T11:26:18.094 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:18.094 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:26:18.094 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:18.094 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:26:18.094 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:38.460411+0000", 2024-09-13T11:26:18.094 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:18.094 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:18.094 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:18.094 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:26:18.095 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:38.460411+0000", 2024-09-13T11:26:18.095 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:18.095 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:18.095 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:18.095 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:26:18.095 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:38.460413+0000", 2024-09-13T11:26:18.095 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.075e-06 2024-09-13T11:26:18.095 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:18.095 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:18.095 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:26:18.095 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:38.460419+0000", 2024-09-13T11:26:18.095 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.6869999999999999e-06 2024-09-13T11:26:18.096 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:18.096 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:18.096 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:26:18.096 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:38.460431+0000", 2024-09-13T11:26:18.096 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.2451e-05 2024-09-13T11:26:18.096 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:18.096 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:18.096 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:26:18.096 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:38.460440+0000", 2024-09-13T11:26:18.096 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.5140000000000001e-06 2024-09-13T11:26:18.096 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:18.097 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:18.097 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:18.097 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:38.460501+0000", 2024-09-13T11:26:18.097 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.0772e-05 2024-09-13T11:26:18.097 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:18.097 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:18.097 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:26:18.097 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:38.460632+0000", 2024-09-13T11:26:18.097 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00013175200000000001 2024-09-13T11:26:18.097 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:18.097 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:18.097 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 9", 2024-09-13T11:26:18.097 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:38.460731+0000", 2024-09-13T11:26:18.098 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.8647000000000006e-05 2024-09-13T11:26:18.098 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:18.098 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:18.098 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:26:18.098 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:38.462322+0000", 2024-09-13T11:26:18.098 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.001590658 2024-09-13T11:26:18.098 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:18.098 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:18.098 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:18.098 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:38.861634+0000", 2024-09-13T11:26:18.098 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.39931206499999999 2024-09-13T11:26:18.098 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:18.098 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:18.098 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:26:18.099 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:38.861719+0000", 2024-09-13T11:26:18.099 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.4968000000000004e-05 2024-09-13T11:26:18.099 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:18.099 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:18.099 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:26:18.099 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:38.861791+0000", 2024-09-13T11:26:18.099 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.2424999999999999e-05 2024-09-13T11:26:18.099 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:18.099 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:18.099 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:18.099 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:18.099 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:18.099 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop_reply(client.4665.0:191021 3.23 e175/90 ondisk, result = 0)", 2024-09-13T11:26:18.099 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:25:38.462483+0000", 2024-09-13T11:26:18.099 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 39.581572014000002, 2024-09-13T11:26:18.100 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.39932300900000001, 2024-09-13T11:26:18.100 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:26:18.100 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:26:18.100 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-09-13T11:26:18.100 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:26:18.100 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:18.100 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:26:18.100 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:38.462483+0000", 2024-09-13T11:26:18.100 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:18.100 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:18.100 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:18.100 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:26:18.100 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:38.462483+0000", 2024-09-13T11:26:18.100 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:18.101 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:18.101 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:18.101 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:26:18.101 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:38.462484+0000", 2024-09-13T11:26:18.101 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.1990000000000001e-06 2024-09-13T11:26:18.101 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:18.101 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:18.101 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:26:18.101 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:38.462488+0000", 2024-09-13T11:26:18.101 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.985e-06 2024-09-13T11:26:18.101 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:18.101 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:18.102 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:26:18.102 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:38.861475+0000", 2024-09-13T11:26:18.102 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.39898754800000003 2024-09-13T11:26:18.102 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:18.102 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:18.102 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:26:18.102 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:38.861488+0000", 2024-09-13T11:26:18.102 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.3037000000000001e-05 2024-09-13T11:26:18.102 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:18.102 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:18.102 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:18.102 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:38.861607+0000", 2024-09-13T11:26:18.102 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00011835 2024-09-13T11:26:18.102 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:18.103 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:18.103 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:26:18.103 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:38.861624+0000", 2024-09-13T11:26:18.103 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.7730000000000001e-05 2024-09-13T11:26:18.103 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:18.103 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:18.103 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:26:18.103 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:38.861806+0000", 2024-09-13T11:26:18.103 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00018116000000000001 2024-09-13T11:26:18.103 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:18.103 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:18.103 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:18.103 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:18.104 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:18.104 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:209500 3.27 3:e6941997:::benchmark_data_smithi177_31116_object69197:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e185)", 2024-09-13T11:26:18.104 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:25:53.962062+0000", 2024-09-13T11:26:18.104 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 24.081992270000001, 2024-09-13T11:26:18.104 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.26007328699999999, 2024-09-13T11:26:18.104 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:26:18.104 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:26:18.104 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:18.104 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:26:18.104 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:26:18.104 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:18.104 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 209500 2024-09-13T11:26:18.104 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:18.105 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:26:18.105 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:18.105 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:26:18.105 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.962062+0000", 2024-09-13T11:26:18.105 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:18.105 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:18.105 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:18.105 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:26:18.105 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.962062+0000", 2024-09-13T11:26:18.105 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:18.105 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:18.105 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:18.105 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:26:18.106 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.962065+0000", 2024-09-13T11:26:18.106 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.9019999999999999e-06 2024-09-13T11:26:18.106 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:18.106 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:18.106 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:26:18.106 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.962074+0000", 2024-09-13T11:26:18.106 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.6030000000000005e-06 2024-09-13T11:26:18.106 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:18.106 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:18.106 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:26:18.106 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.962098+0000", 2024-09-13T11:26:18.106 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.4076000000000001e-05 2024-09-13T11:26:18.106 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:18.106 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:18.107 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:26:18.107 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.962114+0000", 2024-09-13T11:26:18.107 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.5820000000000001e-05 2024-09-13T11:26:18.107 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:18.107 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:18.107 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:18.107 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.962280+0000", 2024-09-13T11:26:18.107 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00016648300000000001 2024-09-13T11:26:18.107 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:18.107 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:18.107 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:26:18.107 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.962598+0000", 2024-09-13T11:26:18.107 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00031824000000000002 2024-09-13T11:26:18.107 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:18.108 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:18.108 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 11", 2024-09-13T11:26:18.108 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.962906+0000", 2024-09-13T11:26:18.108 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00030721699999999998 2024-09-13T11:26:18.108 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:18.108 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:18.108 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:26:18.108 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.965943+0000", 2024-09-13T11:26:18.108 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0030373010000000001 2024-09-13T11:26:18.108 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:18.108 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:18.108 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:18.108 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:54.222003+0000", 2024-09-13T11:26:18.108 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.25605972900000001 2024-09-13T11:26:18.109 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:18.109 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:18.109 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:26:18.109 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:54.222081+0000", 2024-09-13T11:26:18.109 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.8442000000000006e-05 2024-09-13T11:26:18.109 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:18.109 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:18.109 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:26:18.109 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:54.222136+0000", 2024-09-13T11:26:18.109 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.4474000000000001e-05 2024-09-13T11:26:18.109 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:18.109 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:18.110 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:18.110 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:18.110 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:18.110 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:209869 3.27 3:e6b1fcc5:::benchmark_data_smithi177_31116_object69566:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e185)", 2024-09-13T11:26:18.110 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:25:54.300263+0000", 2024-09-13T11:26:18.110 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 23.743791117000001, 2024-09-13T11:26:18.110 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.30266981599999998, 2024-09-13T11:26:18.110 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:26:18.110 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:26:18.110 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:18.110 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:26:18.110 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:26:18.110 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:18.111 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 209869 2024-09-13T11:26:18.111 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:18.111 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:26:18.111 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:18.111 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:26:18.111 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:54.300263+0000", 2024-09-13T11:26:18.111 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:18.111 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:18.111 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:18.111 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:26:18.111 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:54.300263+0000", 2024-09-13T11:26:18.111 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:18.111 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:18.112 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:18.112 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:26:18.112 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:54.300267+0000", 2024-09-13T11:26:18.112 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.2200000000000001e-06 2024-09-13T11:26:18.112 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:18.112 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:18.112 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:26:18.112 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:54.300276+0000", 2024-09-13T11:26:18.112 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.3570000000000004e-06 2024-09-13T11:26:18.112 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:18.112 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:18.112 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:26:18.112 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:54.300301+0000", 2024-09-13T11:26:18.112 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.5191e-05 2024-09-13T11:26:18.113 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:18.113 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:18.113 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:26:18.113 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:54.300321+0000", 2024-09-13T11:26:18.113 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.9967999999999999e-05 2024-09-13T11:26:18.113 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:18.113 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:18.113 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:18.113 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:54.300427+0000", 2024-09-13T11:26:18.113 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000105781 2024-09-13T11:26:18.113 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:18.113 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:18.113 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:26:18.113 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:54.300590+0000", 2024-09-13T11:26:18.114 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00016341000000000001 2024-09-13T11:26:18.114 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:18.114 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:18.114 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 11", 2024-09-13T11:26:18.114 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:54.300724+0000", 2024-09-13T11:26:18.114 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000133323 2024-09-13T11:26:18.114 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:18.114 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:18.114 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:26:18.114 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:54.303373+0000", 2024-09-13T11:26:18.114 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0026488050000000002 2024-09-13T11:26:18.114 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:18.114 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:18.115 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:18.115 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:54.602650+0000", 2024-09-13T11:26:18.115 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.299277976 2024-09-13T11:26:18.115 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:18.115 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:18.115 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:26:18.115 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:54.602795+0000", 2024-09-13T11:26:18.115 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000144127 2024-09-13T11:26:18.115 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:18.115 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:18.115 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:26:18.115 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:54.602933+0000", 2024-09-13T11:26:18.115 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000138658 2024-09-13T11:26:18.115 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:18.116 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:18.116 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:18.116 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:18.116 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:18.116 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop(client.4665.0:217082 3.22 e187/122 3:449e4b87:::benchmark_data_smithi177_31116_object76779:head v 187'5666, mlcod=187'5664)", 2024-09-13T11:26:18.116 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:25:59.884121+0000", 2024-09-13T11:26:18.116 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 18.159933436999999, 2024-09-13T11:26:18.116 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.25479380699999998, 2024-09-13T11:26:18.116 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:26:18.116 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:26:18.116 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:18.116 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:26:18.116 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:18.117 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:26:18.117 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:59.884121+0000", 2024-09-13T11:26:18.117 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:18.117 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:18.117 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:18.117 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:26:18.117 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:59.884121+0000", 2024-09-13T11:26:18.117 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:18.117 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:18.117 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:18.117 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:26:18.117 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:59.884123+0000", 2024-09-13T11:26:18.117 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.677e-06 2024-09-13T11:26:18.118 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:18.118 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:18.118 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:26:18.118 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:59.884129+0000", 2024-09-13T11:26:18.118 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.669e-06 2024-09-13T11:26:18.118 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:18.118 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:18.118 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:26:18.118 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:00.137020+0000", 2024-09-13T11:26:18.118 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.25289069199999997 2024-09-13T11:26:18.118 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:18.118 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:18.118 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:26:18.118 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:00.137030+0000", 2024-09-13T11:26:18.118 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.8600000000000005e-06 2024-09-13T11:26:18.118 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:18.119 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:18.119 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:18.119 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:00.137123+0000", 2024-09-13T11:26:18.119 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.2842000000000003e-05 2024-09-13T11:26:18.119 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:18.119 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:18.119 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:26:18.119 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:00.137139+0000", 2024-09-13T11:26:18.119 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.6388e-05 2024-09-13T11:26:18.119 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:18.119 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:18.119 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:26:18.120 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:00.138876+0000", 2024-09-13T11:26:18.120 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0017370809999999999 2024-09-13T11:26:18.120 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:18.120 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:18.120 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:26:18.120 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:00.138915+0000", 2024-09-13T11:26:18.120 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.8597999999999999e-05 2024-09-13T11:26:18.120 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:18.120 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:18.120 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:18.120 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:18.120 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:18.120 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:221550 3.12 3:4b76b0c0:::benchmark_data_smithi177_31116_object81247:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e193)", 2024-09-13T11:26:18.121 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:26:06.064537+0000", 2024-09-13T11:26:18.121 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 11.979517713, 2024-09-13T11:26:18.121 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.229264668, 2024-09-13T11:26:18.121 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:26:18.121 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:26:18.121 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:18.121 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:26:18.121 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:26:18.121 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:18.121 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 221550 2024-09-13T11:26:18.121 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:18.121 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:26:18.121 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:18.121 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:26:18.122 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:06.064537+0000", 2024-09-13T11:26:18.122 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:18.122 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:18.122 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:18.122 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:26:18.122 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:06.064537+0000", 2024-09-13T11:26:18.122 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:18.122 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:18.122 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:18.122 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:26:18.122 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:06.064539+0000", 2024-09-13T11:26:18.122 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.753e-06 2024-09-13T11:26:18.123 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:18.123 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:18.123 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:26:18.123 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:06.064545+0000", 2024-09-13T11:26:18.123 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.6669999999999999e-06 2024-09-13T11:26:18.123 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:18.123 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:18.123 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:26:18.123 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:06.064613+0000", 2024-09-13T11:26:18.123 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.8199000000000002e-05 2024-09-13T11:26:18.123 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:18.123 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:18.123 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:26:18.123 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:06.064628+0000", 2024-09-13T11:26:18.123 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.455e-05 2024-09-13T11:26:18.124 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:18.124 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:18.124 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:18.124 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:06.064722+0000", 2024-09-13T11:26:18.124 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.4044999999999998e-05 2024-09-13T11:26:18.124 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:18.124 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:18.124 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-09-13T11:26:18.124 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:06.064747+0000", 2024-09-13T11:26:18.124 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.4949000000000001e-05 2024-09-13T11:26:18.124 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:18.124 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:18.124 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:18.125 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:06.275169+0000", 2024-09-13T11:26:18.125 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.21042197700000001 2024-09-13T11:26:18.125 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:18.125 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:18.125 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:26:18.125 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:06.275558+0000", 2024-09-13T11:26:18.125 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000388775 2024-09-13T11:26:18.125 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:18.125 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:18.125 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 7,14", 2024-09-13T11:26:18.125 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:06.275931+0000", 2024-09-13T11:26:18.125 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00037323499999999997 2024-09-13T11:26:18.125 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:18.125 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:18.126 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:26:18.126 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:06.288727+0000", 2024-09-13T11:26:18.126 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0127963 2024-09-13T11:26:18.126 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:18.126 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:18.126 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:18.126 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:06.288823+0000", 2024-09-13T11:26:18.126 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.5513000000000005e-05 2024-09-13T11:26:18.126 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:18.126 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:18.126 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:18.126 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:06.293498+0000", 2024-09-13T11:26:18.126 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0046750419999999999 2024-09-13T11:26:18.126 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:18.127 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:18.127 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:26:18.127 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:06.293600+0000", 2024-09-13T11:26:18.127 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000102376 2024-09-13T11:26:18.127 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:18.127 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:18.127 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:26:18.127 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:06.293802+0000", 2024-09-13T11:26:18.127 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000201287 2024-09-13T11:26:18.127 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:18.127 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:18.127 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:18.127 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:18.127 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:18.128 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:221567 3.12 3:4823d14b:::benchmark_data_smithi177_31116_object81264:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e193)", 2024-09-13T11:26:18.128 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:26:06.064686+0000", 2024-09-13T11:26:18.128 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 11.979368879000001, 2024-09-13T11:26:18.128 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.28539763299999998, 2024-09-13T11:26:18.128 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:26:18.128 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:26:18.128 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:18.128 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:26:18.128 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:26:18.128 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:18.128 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 221567 2024-09-13T11:26:18.128 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:18.129 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:26:18.129 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:18.129 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:26:18.129 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:06.064686+0000", 2024-09-13T11:26:18.129 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:18.129 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:18.129 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:18.129 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:26:18.129 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:06.064686+0000", 2024-09-13T11:26:18.129 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:18.129 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:18.129 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:18.129 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:26:18.129 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:06.064687+0000", 2024-09-13T11:26:18.129 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.6300000000000001e-06 2024-09-13T11:26:18.130 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:18.130 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:18.130 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:26:18.130 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:06.064693+0000", 2024-09-13T11:26:18.130 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.7339999999999996e-06 2024-09-13T11:26:18.130 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:18.130 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:18.130 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:26:18.130 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:06.064704+0000", 2024-09-13T11:26:18.130 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.1253000000000001e-05 2024-09-13T11:26:18.130 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:18.130 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:18.130 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:26:18.130 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:06.064717+0000", 2024-09-13T11:26:18.130 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.2293000000000001e-05 2024-09-13T11:26:18.131 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:18.131 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:18.131 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:18.131 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:06.064866+0000", 2024-09-13T11:26:18.131 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00014957499999999999 2024-09-13T11:26:18.131 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:18.131 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:18.131 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-09-13T11:26:18.131 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:06.064893+0000", 2024-09-13T11:26:18.131 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.6614e-05 2024-09-13T11:26:18.131 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:18.131 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:18.131 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:18.131 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:06.282646+0000", 2024-09-13T11:26:18.132 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.21775271800000001 2024-09-13T11:26:18.132 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:18.132 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:18.132 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:26:18.132 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:06.283003+0000", 2024-09-13T11:26:18.132 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00035780799999999999 2024-09-13T11:26:18.132 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:18.132 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:18.132 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 7,14", 2024-09-13T11:26:18.132 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:06.283345+0000", 2024-09-13T11:26:18.132 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00034164200000000002 2024-09-13T11:26:18.132 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:18.132 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:18.133 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:26:18.133 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:06.288935+0000", 2024-09-13T11:26:18.133 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0055902429999999999 2024-09-13T11:26:18.133 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:18.133 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:18.133 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:18.133 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:06.289795+0000", 2024-09-13T11:26:18.133 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00085952200000000004 2024-09-13T11:26:18.133 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:18.133 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:18.133 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:18.133 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:06.349852+0000", 2024-09-13T11:26:18.133 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.060057352000000001 2024-09-13T11:26:18.133 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:18.134 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:18.134 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:26:18.134 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:06.349922+0000", 2024-09-13T11:26:18.134 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.0272999999999996e-05 2024-09-13T11:26:18.134 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:18.134 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:18.134 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:26:18.134 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:06.350083+0000", 2024-09-13T11:26:18.134 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00016097600000000001 2024-09-13T11:26:18.134 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:18.134 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:18.134 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:18.134 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:18.134 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:18.134 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:227066 3.23 3:c67f2f28:::benchmark_data_smithi177_31116_object86763:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e196)", 2024-09-13T11:26:18.135 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:26:10.753624+0000", 2024-09-13T11:26:18.135 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 7.2904308059999998, 2024-09-13T11:26:18.135 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.28670347299999999, 2024-09-13T11:26:18.135 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:26:18.135 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:26:18.135 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:18.135 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:26:18.135 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:26:18.135 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:18.135 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 227066 2024-09-13T11:26:18.135 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:18.135 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:26:18.135 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:18.135 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:26:18.135 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:10.753624+0000", 2024-09-13T11:26:18.136 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:18.136 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:18.136 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:18.136 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:26:18.136 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:10.753624+0000", 2024-09-13T11:26:18.136 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:18.136 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:18.136 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:18.136 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:26:18.136 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:10.753627+0000", 2024-09-13T11:26:18.136 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.8030000000000001e-06 2024-09-13T11:26:18.136 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:18.136 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:18.137 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:26:18.137 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:10.753634+0000", 2024-09-13T11:26:18.137 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.9659999999999999e-06 2024-09-13T11:26:18.137 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:18.137 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:18.137 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:26:18.137 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:10.753650+0000", 2024-09-13T11:26:18.137 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.6167999999999998e-05 2024-09-13T11:26:18.137 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:18.137 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:18.137 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:26:18.137 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:10.753657+0000", 2024-09-13T11:26:18.137 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.7600000000000002e-06 2024-09-13T11:26:18.137 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:18.138 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:18.138 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:18.138 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:10.753741+0000", 2024-09-13T11:26:18.138 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.3523999999999995e-05 2024-09-13T11:26:18.138 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:18.138 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:18.138 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:26:18.138 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:10.753878+0000", 2024-09-13T11:26:18.138 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000137207 2024-09-13T11:26:18.138 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:18.138 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:18.138 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 9", 2024-09-13T11:26:18.138 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:10.754027+0000", 2024-09-13T11:26:18.138 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00014856599999999999 2024-09-13T11:26:18.138 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:18.138 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:18.139 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:26:18.139 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:10.756313+0000", 2024-09-13T11:26:18.139 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.002286045 2024-09-13T11:26:18.139 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:18.139 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:18.139 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:18.139 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:11.040266+0000", 2024-09-13T11:26:18.139 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.283953434 2024-09-13T11:26:18.139 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:18.139 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:18.139 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:26:18.139 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:11.040300+0000", 2024-09-13T11:26:18.139 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.4227000000000003e-05 2024-09-13T11:26:18.140 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:18.140 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:18.140 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:26:18.140 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:11.040327+0000", 2024-09-13T11:26:18.140 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.6772999999999999e-05 2024-09-13T11:26:18.140 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:18.140 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:18.140 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:18.140 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:18.140 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:18.140 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop_reply(client.4665.0:227066 3.23 e196/90 ondisk, result = 0)", 2024-09-13T11:26:18.140 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:26:10.756369+0000", 2024-09-13T11:26:18.140 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 7.287685432, 2024-09-13T11:26:18.140 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.28396190599999999, 2024-09-13T11:26:18.141 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:26:18.141 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:26:18.141 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-09-13T11:26:18.141 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:26:18.141 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:18.141 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:26:18.141 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:10.756369+0000", 2024-09-13T11:26:18.141 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:18.141 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:18.141 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:18.141 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:26:18.141 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:10.756369+0000", 2024-09-13T11:26:18.141 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:18.141 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:18.141 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:18.141 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:26:18.142 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:10.756370+0000", 2024-09-13T11:26:18.142 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.164e-06 2024-09-13T11:26:18.142 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:18.142 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:18.142 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:26:18.142 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:10.756374+0000", 2024-09-13T11:26:18.142 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.0659999999999997e-06 2024-09-13T11:26:18.142 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:18.142 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:18.142 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:26:18.142 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:11.040177+0000", 2024-09-13T11:26:18.142 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.28380288599999998 2024-09-13T11:26:18.142 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:18.142 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:18.143 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:26:18.143 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:11.040184+0000", 2024-09-13T11:26:18.143 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.7229999999999998e-06 2024-09-13T11:26:18.143 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:18.143 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:18.143 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:18.143 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:11.040253+0000", 2024-09-13T11:26:18.143 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.8997000000000004e-05 2024-09-13T11:26:18.143 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:18.143 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:18.143 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:26:18.144 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:11.040261+0000", 2024-09-13T11:26:18.144 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.3590000000000008e-06 2024-09-13T11:26:18.144 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:18.144 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:18.144 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:26:18.144 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:11.040331+0000", 2024-09-13T11:26:18.144 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.9710999999999997e-05 2024-09-13T11:26:18.144 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:18.144 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:18.144 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:18.144 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:18.144 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:18.145 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:227420 3.23 3:c4add80a:::benchmark_data_smithi177_31116_object87117:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e196)", 2024-09-13T11:26:18.145 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:26:11.037912+0000", 2024-09-13T11:26:18.145 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 7.0061427859999998, 2024-09-13T11:26:18.145 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.33121650499999999, 2024-09-13T11:26:18.145 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:26:18.145 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:26:18.145 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:18.145 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:26:18.145 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:26:18.145 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:18.145 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 227420 2024-09-13T11:26:18.146 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:18.146 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:26:18.146 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:18.146 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:26:18.146 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:11.037912+0000", 2024-09-13T11:26:18.146 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:18.146 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:18.146 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:18.146 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:26:18.146 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:11.037912+0000", 2024-09-13T11:26:18.146 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:18.146 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:18.147 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:18.147 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:26:18.147 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:11.037913+0000", 2024-09-13T11:26:18.147 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.606e-06 2024-09-13T11:26:18.147 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:18.147 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:18.147 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:26:18.147 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:11.037918+0000", 2024-09-13T11:26:18.147 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.0509999999999996e-06 2024-09-13T11:26:18.147 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:18.147 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:18.147 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:26:18.148 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:11.037930+0000", 2024-09-13T11:26:18.148 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.1466e-05 2024-09-13T11:26:18.148 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:18.148 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:18.148 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:26:18.148 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:11.037936+0000", 2024-09-13T11:26:18.148 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.5259999999999999e-06 2024-09-13T11:26:18.148 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:18.148 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:18.148 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:18.148 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:11.037994+0000", 2024-09-13T11:26:18.148 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.7776999999999999e-05 2024-09-13T11:26:18.149 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:18.149 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:18.149 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:26:18.149 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:11.038129+0000", 2024-09-13T11:26:18.149 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00013500299999999999 2024-09-13T11:26:18.149 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:18.149 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:18.149 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 9", 2024-09-13T11:26:18.149 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:11.038208+0000", 2024-09-13T11:26:18.149 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.9016000000000004e-05 2024-09-13T11:26:18.149 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:18.149 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:18.150 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:26:18.150 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:11.041461+0000", 2024-09-13T11:26:18.150 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00325246 2024-09-13T11:26:18.150 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:18.150 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:18.150 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:18.150 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:11.369025+0000", 2024-09-13T11:26:18.150 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.32756474800000002 2024-09-13T11:26:18.150 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:18.150 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:18.150 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:26:18.150 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:11.369086+0000", 2024-09-13T11:26:18.151 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.0140999999999999e-05 2024-09-13T11:26:18.151 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:18.151 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:18.151 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:26:18.151 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:11.369128+0000", 2024-09-13T11:26:18.151 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.2710999999999998e-05 2024-09-13T11:26:18.151 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:18.151 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:18.151 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:18.151 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:18.151 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:18.151 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop_reply(client.4665.0:227420 3.23 e196/90 ondisk, result = 0)", 2024-09-13T11:26:18.151 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:26:11.040152+0000", 2024-09-13T11:26:18.151 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 7.0039029480000004, 2024-09-13T11:26:18.152 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.328983201, 2024-09-13T11:26:18.152 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:26:18.152 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:26:18.152 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-09-13T11:26:18.152 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:26:18.152 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:18.152 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:26:18.152 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:11.040152+0000", 2024-09-13T11:26:18.152 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:18.152 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:18.152 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:18.152 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:26:18.152 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:11.040152+0000", 2024-09-13T11:26:18.152 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:18.153 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:18.153 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:18.153 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:26:18.153 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:11.040157+0000", 2024-09-13T11:26:18.153 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.1309999999999997e-06 2024-09-13T11:26:18.153 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:18.153 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:18.153 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:26:18.153 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:11.040162+0000", 2024-09-13T11:26:18.153 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.8720000000000001e-06 2024-09-13T11:26:18.153 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:18.153 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:18.153 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:26:18.153 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:11.368952+0000", 2024-09-13T11:26:18.154 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.32879022400000002 2024-09-13T11:26:18.154 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:18.154 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:18.154 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:26:18.154 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:11.368960+0000", 2024-09-13T11:26:18.154 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.8460000000000008e-06 2024-09-13T11:26:18.154 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:18.154 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:18.154 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:18.154 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:11.369013+0000", 2024-09-13T11:26:18.154 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.3201000000000001e-05 2024-09-13T11:26:18.154 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:18.154 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:18.154 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:26:18.155 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:11.369020+0000", 2024-09-13T11:26:18.155 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.3050000000000004e-06 2024-09-13T11:26:18.155 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:18.155 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:18.155 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:26:18.155 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:11.369135+0000", 2024-09-13T11:26:18.155 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000114622 2024-09-13T11:26:18.155 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:18.155 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:18.155 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:18.155 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:18.155 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:18.155 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-09-13T11:26:18.156 INFO:teuthology.orchestra.run.smithi001.stdout:osd.2: {} 2024-09-13T11:26:18.157 INFO:teuthology.orchestra.run.smithi001.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T11:26:18.158 INFO:teuthology.orchestra.run.smithi001.stdout:osd.3: {} 2024-09-13T11:26:18.158 INFO:teuthology.orchestra.run.smithi001.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T11:26:18.158 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-09-13T11:26:18.159 INFO:teuthology.orchestra.run.smithi001.stdout:osd.4: {} 2024-09-13T11:26:18.159 INFO:teuthology.orchestra.run.smithi001.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T11:26:18.159 INFO:teuthology.orchestra.run.smithi001.stdout:osd.5: {} 2024-09-13T11:26:18.159 INFO:teuthology.orchestra.run.smithi001.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T11:26:18.160 INFO:tasks.ceph.osd.4.smithi001.stderr:2024-09-13T11:26:18.150+0000 7ff8d2c41640 -1 osd.4 201 handle_osd_map injecting map crc failure 2024-09-13T11:26:18.160 INFO:tasks.ceph.osd.7.smithi177.stderr:2024-09-13T11:26:18.153+0000 7f897b51d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26575) UID: 0 2024-09-13T11:26:18.185 INFO:teuthology.orchestra.run.smithi001.stdout:osd.6: {} 2024-09-13T11:26:18.186 INFO:teuthology.orchestra.run.smithi001.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T11:26:18.219 INFO:teuthology.orchestra.run.smithi001.stdout:osd.7: {} 2024-09-13T11:26:18.219 INFO:teuthology.orchestra.run.smithi001.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T11:26:18.248 INFO:teuthology.orchestra.run.smithi001.stdout:osd.8: {} 2024-09-13T11:26:18.249 INFO:teuthology.orchestra.run.smithi001.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-13T11:26:18.257 INFO:tasks.ceph.osd.9.smithi092.stderr:2024-09-13T11:26:18.256+0000 7ff5ab3d0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26348) UID: 0 2024-09-13T11:26:18.278 INFO:teuthology.orchestra.run.smithi001.stdout:osd.9: {} 2024-09-13T11:26:18.278 INFO:teuthology.orchestra.run.smithi001.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-13T11:26:18.313 INFO:teuthology.orchestra.run.smithi001.stdout:osd.10: {} 2024-09-13T11:26:18.314 INFO:teuthology.orchestra.run.smithi001.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-13T11:26:18.339 INFO:teuthology.orchestra.run.smithi001.stdout:osd.11: {} 2024-09-13T11:26:18.339 INFO:teuthology.orchestra.run.smithi001.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-13T11:26:18.352 INFO:teuthology.orchestra.run.smithi092.stdout:ERROR: (22) Invalid argument 2024-09-13T11:26:18.352 INFO:teuthology.orchestra.run.smithi092.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:26:18.357 INFO:tasks.ceph.osd.5.smithi092.stderr:2024-09-13T11:26:18.356+0000 7f6f0b705640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26346) UID: 0 2024-09-13T11:26:18.372 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-09-13T11:26:18.373 INFO:teuthology.orchestra.run.smithi001.stdout:osd.12: {} 2024-09-13T11:26:18.373 INFO:teuthology.orchestra.run.smithi001.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-13T11:26:18.401 INFO:teuthology.orchestra.run.smithi001.stdout:osd.13: {} 2024-09-13T11:26:18.401 INFO:teuthology.orchestra.run.smithi001.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-13T11:26:18.433 INFO:teuthology.orchestra.run.smithi001.stdout:osd.14: {} 2024-09-13T11:26:18.433 INFO:teuthology.orchestra.run.smithi001.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-13T11:26:18.441 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-09-13T11:26:18.439+0000 7f97b59fa640 -1 osd.6 201 handle_osd_map injecting map crc failure 2024-09-13T11:26:18.459 INFO:tasks.ceph.osd.14.smithi162.stderr:2024-09-13T11:26:18.455+0000 7f3df5903640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33154) UID: 0 2024-09-13T11:26:18.472 INFO:teuthology.orchestra.run.smithi001.stdout:osd.15: {} 2024-09-13T11:26:18.472 INFO:teuthology.orchestra.run.smithi001.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-13T11:26:18.554 INFO:teuthology.orchestra.run.smithi092.stdout:ERROR: (22) Invalid argument 2024-09-13T11:26:18.554 INFO:teuthology.orchestra.run.smithi092.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:26:18.560 INFO:tasks.ceph.osd.12.smithi001.stderr:2024-09-13T11:26:18.558+0000 7f892c6e6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26314) UID: 0 2024-09-13T11:26:18.572 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-09-13T11:26:18.661 INFO:tasks.ceph.osd.12.smithi001.stderr:2024-09-13T11:26:18.658+0000 7f892c6e6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26314) UID: 0 2024-09-13T11:26:18.707 INFO:teuthology.orchestra.run.smithi092.stdout:ERROR: (22) Invalid argument 2024-09-13T11:26:18.707 INFO:teuthology.orchestra.run.smithi092.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:26:18.723 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_ops_in_flight 2024-09-13T11:26:18.762 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-09-13T11:26:18.759+0000 7f97befc9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26315) UID: 0 2024-09-13T11:26:18.862 INFO:teuthology.orchestra.run.smithi092.stdout:ERROR: (22) Invalid argument 2024-09-13T11:26:18.863 INFO:teuthology.orchestra.run.smithi092.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:26:18.863 INFO:tasks.ceph.osd.12.smithi001.stderr:2024-09-13T11:26:18.858+0000 7f892c6e6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26314) UID: 0 2024-09-13T11:26:18.878 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_blocked_ops 2024-09-13T11:26:18.964 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-13T11:26:18.962+0000 7f6a68775640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26307) UID: 0 2024-09-13T11:26:19.003 INFO:teuthology.orchestra.run.smithi092.stdout:ERROR: (22) Invalid argument 2024-09-13T11:26:19.003 INFO:teuthology.orchestra.run.smithi092.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:26:19.017 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_historic_ops 2024-09-13T11:26:19.065 INFO:tasks.ceph.osd.3.smithi177.stderr:2024-09-13T11:26:19.061+0000 7f10adba4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26574) UID: 0 2024-09-13T11:26:19.143 INFO:tasks.ceph.osd.1.smithi092.stderr:2024-09-13T11:26:19.136+0000 7fe76f3f6640 -1 osd.1 202 handle_osd_map injecting map crc failure 2024-09-13T11:26:19.143 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-13T11:26:19.142+0000 7f6a5d1a2640 -1 osd.0 202 handle_osd_map injecting map crc failure 2024-09-13T11:26:19.147 INFO:teuthology.orchestra.run.smithi092.stdout:ERROR: (22) Invalid argument 2024-09-13T11:26:19.147 INFO:teuthology.orchestra.run.smithi092.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:26:19.155 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-13T11:26:19.162 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-13T11:26:19.196 INFO:tasks.ceph.osd.11.smithi177.stderr:2024-09-13T11:26:19.193+0000 7f2469cec640 -1 osd.11 202 handle_osd_map injecting map crc failure 2024-09-13T11:26:19.204 INFO:tasks.ceph.osd.12.smithi001.stderr:2024-09-13T11:26:19.202+0000 7f892c6e6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26314) UID: 0 2024-09-13T11:26:19.267 INFO:tasks.ceph.osd.14.smithi162.stderr:2024-09-13T11:26:19.263+0000 7f3df5903640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33154) UID: 0 2024-09-13T11:26:19.308 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-13T11:26:19.308 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:26:19.324 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-13T11:26:19.368 INFO:tasks.ceph.osd.5.smithi092.stderr:2024-09-13T11:26:19.364+0000 7f6f0b705640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26346) UID: 0 2024-09-13T11:26:19.469 INFO:tasks.ceph.osd.4.smithi001.stderr:2024-09-13T11:26:19.466+0000 7ff8e0218640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26310) UID: 0 2024-09-13T11:26:19.516 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-13T11:26:19.516 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:26:19.534 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-13T11:26:19.570 INFO:tasks.ceph.osd.5.smithi092.stderr:2024-09-13T11:26:19.568+0000 7f6f0b705640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26346) UID: 0 2024-09-13T11:26:19.671 INFO:tasks.ceph.osd.13.smithi092.stderr:2024-09-13T11:26:19.668+0000 7f3e5afea640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26350) UID: 0 2024-09-13T11:26:19.689 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-13T11:26:19.689 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:26:19.707 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-13T11:26:19.774 INFO:tasks.ceph.osd.10.smithi162.stderr:2024-09-13T11:26:19.771+0000 7fb59c9f0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26317) UID: 0 2024-09-13T11:26:19.853 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-13T11:26:19.853 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:26:19.873 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-13T11:26:19.874 INFO:tasks.ceph.osd.3.smithi177.stderr:2024-09-13T11:26:19.869+0000 7f10adba4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26574) UID: 0 2024-09-13T11:26:19.974 INFO:tasks.ceph.osd.8.smithi001.stderr:2024-09-13T11:26:19.970+0000 7fb9b48ae640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26312) UID: 0 2024-09-13T11:26:20.065 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-13T11:26:20.065 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:26:20.075 INFO:tasks.ceph.osd.8.smithi001.stderr:2024-09-13T11:26:20.070+0000 7fb9b48ae640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26312) UID: 0 2024-09-13T11:26:20.087 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-13T11:26:20.139 INFO:teuthology.orchestra.run.smithi001.stderr:noscrub is unset 2024-09-13T11:26:20.176 INFO:tasks.ceph.osd.14.smithi162.stderr:2024-09-13T11:26:20.171+0000 7f3df5903640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33154) UID: 0 2024-09-13T11:26:20.237 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-13T11:26:20.237 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:26:20.257 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-09-13T11:26:20.276 INFO:tasks.ceph.osd.1.smithi092.stderr:2024-09-13T11:26:20.272+0000 7fe7789c5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33718) UID: 0 2024-09-13T11:26:20.377 INFO:tasks.ceph.osd.15.smithi177.stderr:2024-09-13T11:26:20.373+0000 7f9d6ce77640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26578) UID: 0 2024-09-13T11:26:20.440 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-13T11:26:20.440 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:26:20.458 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-09-13T11:26:20.478 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-13T11:26:20.474+0000 7f6a68775640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26307) UID: 0 2024-09-13T11:26:20.500 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T11:26:20.579 INFO:tasks.ceph.osd.1.smithi092.stderr:2024-09-13T11:26:20.576+0000 7fe7789c5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33718) UID: 0 2024-09-13T11:26:20.590 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-13T11:26:20.590 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:26:20.610 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-09-13T11:26:20.668 INFO:teuthology.orchestra.run.smithi001.stdout:osd.0: {} 2024-09-13T11:26:20.669 INFO:teuthology.orchestra.run.smithi001.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T11:26:20.680 INFO:tasks.ceph.osd.1.smithi092.stderr:2024-09-13T11:26:20.676+0000 7fe7789c5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33718) UID: 0 2024-09-13T11:26:20.703 INFO:teuthology.orchestra.run.smithi001.stdout:osd.1: {} 2024-09-13T11:26:20.703 INFO:teuthology.orchestra.run.smithi001.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T11:26:20.728 INFO:teuthology.orchestra.run.smithi001.stdout:osd.2: {} 2024-09-13T11:26:20.728 INFO:teuthology.orchestra.run.smithi001.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T11:26:20.742 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-13T11:26:20.742 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:26:20.754 INFO:teuthology.orchestra.run.smithi001.stdout:osd.3: {} 2024-09-13T11:26:20.754 INFO:teuthology.orchestra.run.smithi001.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T11:26:20.758 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-13T11:26:20.777 INFO:teuthology.orchestra.run.smithi001.stdout:osd.4: {} 2024-09-13T11:26:20.777 INFO:teuthology.orchestra.run.smithi001.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T11:26:20.780 INFO:tasks.ceph.osd.1.smithi092.stderr:2024-09-13T11:26:20.780+0000 7fe7789c5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33718) UID: 0 2024-09-13T11:26:20.803 INFO:teuthology.orchestra.run.smithi001.stdout:osd.5: {} 2024-09-13T11:26:20.804 INFO:teuthology.orchestra.run.smithi001.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T11:26:20.830 INFO:teuthology.orchestra.run.smithi001.stdout:osd.6: {} 2024-09-13T11:26:20.831 INFO:teuthology.orchestra.run.smithi001.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T11:26:20.856 INFO:teuthology.orchestra.run.smithi001.stdout:osd.7: {} 2024-09-13T11:26:20.856 INFO:teuthology.orchestra.run.smithi001.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T11:26:20.881 INFO:teuthology.orchestra.run.smithi001.stdout:osd.8: {} 2024-09-13T11:26:20.881 INFO:teuthology.orchestra.run.smithi001.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-13T11:26:20.881 INFO:tasks.ceph.osd.14.smithi162.stderr:2024-09-13T11:26:20.879+0000 7f3df5903640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33154) UID: 0 2024-09-13T11:26:20.895 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-09-13T11:26:20.895 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [ 2024-09-13T11:26:20.895 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:20.895 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:237309 3.2 3:4089983f:::benchmark_data_smithi177_31116_object97006:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e204)", 2024-09-13T11:26:20.895 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:26:20.891983+0000", 2024-09-13T11:26:20.895 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 0.0045791770000000002, 2024-09-13T11:26:20.896 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0045969590000000003, 2024-09-13T11:26:20.896 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:26:20.896 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:26:20.896 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "waiting for sub ops", 2024-09-13T11:26:20.896 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:26:20.896 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:26:20.896 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:20.896 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 237309 2024-09-13T11:26:20.896 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:20.896 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:26:20.896 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:20.896 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:26:20.896 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:20.891983+0000", 2024-09-13T11:26:20.896 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:20.897 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:20.897 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:20.897 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:26:20.897 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:20.891983+0000", 2024-09-13T11:26:20.897 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:20.897 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:20.897 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:20.897 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:26:20.897 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:20.891986+0000", 2024-09-13T11:26:20.897 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.1909999999999999e-06 2024-09-13T11:26:20.897 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:20.897 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:20.897 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:26:20.898 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:20.891991+0000", 2024-09-13T11:26:20.898 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.057e-06 2024-09-13T11:26:20.898 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:20.898 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:20.898 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:26:20.898 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:20.892000+0000", 2024-09-13T11:26:20.898 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.5030000000000003e-06 2024-09-13T11:26:20.898 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:20.898 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:20.898 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:26:20.898 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:20.892005+0000", 2024-09-13T11:26:20.898 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.3120000000000001e-06 2024-09-13T11:26:20.898 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:20.898 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:20.898 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:26:20.899 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:20.892076+0000", 2024-09-13T11:26:20.899 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.0083e-05 2024-09-13T11:26:20.899 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:20.899 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:20.899 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:26:20.899 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:20.892187+0000", 2024-09-13T11:26:20.899 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000111284 2024-09-13T11:26:20.899 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:20.899 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:20.899 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 12,14", 2024-09-13T11:26:20.899 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:20.892287+0000", 2024-09-13T11:26:20.899 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000100388 2024-09-13T11:26:20.899 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:20.899 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:20.900 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:26:20.900 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:20.893805+0000", 2024-09-13T11:26:20.900 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0015179080000000001 2024-09-13T11:26:20.900 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:20.900 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:26:20.900 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:20.900 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:20.900 INFO:teuthology.orchestra.run.smithi177.stdout: ], 2024-09-13T11:26:20.900 INFO:teuthology.orchestra.run.smithi177.stdout: "num_ops": 1 2024-09-13T11:26:20.900 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-09-13T11:26:20.906 INFO:teuthology.orchestra.run.smithi001.stdout:osd.9: {} 2024-09-13T11:26:20.906 INFO:teuthology.orchestra.run.smithi001.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-13T11:26:20.911 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-13T11:26:20.933 INFO:teuthology.orchestra.run.smithi001.stdout:osd.10: {} 2024-09-13T11:26:20.933 INFO:teuthology.orchestra.run.smithi001.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-13T11:26:20.959 INFO:teuthology.orchestra.run.smithi001.stdout:osd.11: {} 2024-09-13T11:26:20.960 INFO:teuthology.orchestra.run.smithi001.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-13T11:26:20.981 INFO:tasks.ceph.osd.14.smithi162.stderr:2024-09-13T11:26:20.979+0000 7f3df5903640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33154) UID: 0 2024-09-13T11:26:20.986 INFO:teuthology.orchestra.run.smithi001.stdout:osd.12: {} 2024-09-13T11:26:20.986 INFO:teuthology.orchestra.run.smithi001.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-13T11:26:21.012 INFO:teuthology.orchestra.run.smithi001.stdout:osd.13: {} 2024-09-13T11:26:21.013 INFO:teuthology.orchestra.run.smithi001.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-13T11:26:21.040 INFO:teuthology.orchestra.run.smithi001.stdout:osd.14: {} 2024-09-13T11:26:21.040 INFO:teuthology.orchestra.run.smithi001.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-13T11:26:21.047 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-09-13T11:26:21.047 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [], 2024-09-13T11:26:21.048 INFO:teuthology.orchestra.run.smithi177.stdout: "complaint_time": 30, 2024-09-13T11:26:21.048 INFO:teuthology.orchestra.run.smithi177.stdout: "num_blocked_ops": 0 2024-09-13T11:26:21.048 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-09-13T11:26:21.064 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-13T11:26:21.065 INFO:teuthology.orchestra.run.smithi001.stdout:osd.15: {} 2024-09-13T11:26:21.065 INFO:teuthology.orchestra.run.smithi001.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-13T11:26:21.082 INFO:tasks.ceph.osd.14.smithi162.stderr:2024-09-13T11:26:21.079+0000 7f3df5903640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33154) UID: 0 2024-09-13T11:26:21.183 INFO:tasks.ceph.osd.4.smithi001.stderr:2024-09-13T11:26:21.178+0000 7ff8e0218640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26310) UID: 0 2024-09-13T11:26:21.199 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-09-13T11:26:21.199 INFO:teuthology.orchestra.run.smithi177.stdout: "size": 20, 2024-09-13T11:26:21.199 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 600, 2024-09-13T11:26:21.199 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [ 2024-09-13T11:26:21.199 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.199 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPush(3.17 75/70 [PushOp(3:eda94455:::benchmark_data_smithi177_31116_object13017:head, version: 50'788, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:eda94455:::benchmark_data_smithi177_31116_object13017:head@50'788, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:26:21.199 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:31.038029+0000", 2024-09-13T11:26:21.199 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 170.15997294300001, 2024-09-13T11:26:21.199 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.42064623499999998, 2024-09-13T11:26:21.199 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:26:21.199 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:26:21.200 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:26:21.200 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:26:21.200 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.200 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:26:21.200 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.038029+0000", 2024-09-13T11:26:21.200 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:21.200 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.200 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.200 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:26:21.200 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.038029+0000", 2024-09-13T11:26:21.200 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:21.200 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.200 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.201 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:26:21.201 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.038030+0000", 2024-09-13T11:26:21.201 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.499e-06 2024-09-13T11:26:21.201 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.201 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.201 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:26:21.201 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.038088+0000", 2024-09-13T11:26:21.201 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.7771e-05 2024-09-13T11:26:21.201 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.201 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.201 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:26:21.201 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.457025+0000", 2024-09-13T11:26:21.201 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.41893754900000002 2024-09-13T11:26:21.201 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.202 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.202 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:26:21.202 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.457073+0000", 2024-09-13T11:26:21.202 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.7450999999999997e-05 2024-09-13T11:26:21.202 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.202 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.202 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:26:21.202 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.457304+0000", 2024-09-13T11:26:21.202 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000231351 2024-09-13T11:26:21.203 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.203 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.203 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:26:21.203 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.457331+0000", 2024-09-13T11:26:21.203 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.6372000000000001e-05 2024-09-13T11:26:21.203 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.203 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.203 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:26:21.203 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.458675+0000", 2024-09-13T11:26:21.203 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001344242 2024-09-13T11:26:21.203 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:21.204 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:26:21.204 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:21.204 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.204 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.204 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPush(3.17 83/70 [PushOp(3:eb574268:::benchmark_data_smithi177_31116_object40622:head, version: 64'2475, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:eb574268:::benchmark_data_smithi177_31116_object40622:head@64'2475, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:26:21.204 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:40.760042+0000", 2024-09-13T11:26:21.204 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 160.437959226, 2024-09-13T11:26:21.204 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.30544642100000002, 2024-09-13T11:26:21.204 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:26:21.204 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:26:21.204 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:26:21.204 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:26:21.205 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.205 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:26:21.205 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:40.760042+0000", 2024-09-13T11:26:21.205 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:21.205 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.205 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.205 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:26:21.205 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:40.760042+0000", 2024-09-13T11:26:21.205 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:21.205 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.205 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.206 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:26:21.206 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:40.760044+0000", 2024-09-13T11:26:21.206 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.511e-06 2024-09-13T11:26:21.206 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.206 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.206 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:26:21.206 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:40.760149+0000", 2024-09-13T11:26:21.207 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000105473 2024-09-13T11:26:21.207 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.207 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.207 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:26:21.207 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.064465+0000", 2024-09-13T11:26:21.207 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.30431613800000001 2024-09-13T11:26:21.207 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.207 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.207 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:26:21.208 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.064479+0000", 2024-09-13T11:26:21.208 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3586e-05 2024-09-13T11:26:21.208 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.208 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.208 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:26:21.208 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.064553+0000", 2024-09-13T11:26:21.208 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.3646000000000006e-05 2024-09-13T11:26:21.208 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.208 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.208 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:26:21.208 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.064568+0000", 2024-09-13T11:26:21.209 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.52e-05 2024-09-13T11:26:21.209 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.209 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.209 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:26:21.209 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.065489+0000", 2024-09-13T11:26:21.209 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00092086700000000002 2024-09-13T11:26:21.209 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:21.209 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:26:21.209 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:21.209 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.209 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.209 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPushReply(3.1a 83/78 [PushReplyOp(3:5c3834bd:::benchmark_data_smithi177_31116_object8669:head)])", 2024-09-13T11:26:21.210 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:40.798236+0000", 2024-09-13T11:26:21.210 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 160.399765796, 2024-09-13T11:26:21.210 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.29701520999999997, 2024-09-13T11:26:21.210 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:26:21.210 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:26:21.210 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-13T11:26:21.210 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:26:21.210 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.210 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:26:21.210 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:40.798236+0000", 2024-09-13T11:26:21.210 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:21.211 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.211 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.211 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:26:21.211 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:40.798236+0000", 2024-09-13T11:26:21.211 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:21.211 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.211 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.211 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:26:21.211 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:40.798237+0000", 2024-09-13T11:26:21.211 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.0500000000000002e-07 2024-09-13T11:26:21.211 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.212 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.212 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:26:21.212 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:40.798240+0000", 2024-09-13T11:26:21.212 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.4819999999999999e-06 2024-09-13T11:26:21.212 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.212 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.212 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:26:21.212 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.093360+0000", 2024-09-13T11:26:21.212 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.29512009 2024-09-13T11:26:21.212 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.212 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.212 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:26:21.212 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.093372+0000", 2024-09-13T11:26:21.213 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.2221e-05 2024-09-13T11:26:21.213 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.213 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.213 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:26:21.213 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.093482+0000", 2024-09-13T11:26:21.213 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000109571 2024-09-13T11:26:21.213 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.213 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.213 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:26:21.213 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.095251+0000", 2024-09-13T11:26:21.213 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001768941 2024-09-13T11:26:21.214 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:21.214 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:26:21.214 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:21.214 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.214 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.214 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPushReply(3.1a 83/78 [PushReplyOp(3:58c7ffc0:::benchmark_data_smithi177_31116_object8738:head)])", 2024-09-13T11:26:21.214 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:40.805941+0000", 2024-09-13T11:26:21.214 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 160.392060542, 2024-09-13T11:26:21.214 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.30315678499999998, 2024-09-13T11:26:21.214 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:26:21.214 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:26:21.214 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-13T11:26:21.214 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:26:21.215 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.215 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:26:21.215 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:40.805941+0000", 2024-09-13T11:26:21.215 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:21.215 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.215 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.215 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:26:21.215 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:40.805941+0000", 2024-09-13T11:26:21.215 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:21.215 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.215 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.215 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:26:21.215 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:40.805942+0000", 2024-09-13T11:26:21.216 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3769999999999999e-06 2024-09-13T11:26:21.216 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.216 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.216 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:26:21.216 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:40.805947+0000", 2024-09-13T11:26:21.216 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.8980000000000004e-06 2024-09-13T11:26:21.216 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.217 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.217 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:26:21.217 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.106505+0000", 2024-09-13T11:26:21.217 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.30055777700000003 2024-09-13T11:26:21.217 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.217 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.218 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:26:21.218 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.106512+0000", 2024-09-13T11:26:21.218 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.3749999999999997e-06 2024-09-13T11:26:21.218 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.218 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.218 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:26:21.219 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.106566+0000", 2024-09-13T11:26:21.219 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.3103999999999998e-05 2024-09-13T11:26:21.219 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.219 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.219 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:26:21.219 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.109098+0000", 2024-09-13T11:26:21.219 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0025322539999999998 2024-09-13T11:26:21.219 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:21.219 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:26:21.220 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:21.220 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.220 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.220 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPushReply(3.1a 83/78 [PushReplyOp(3:58c7ffc0:::benchmark_data_smithi177_31116_object8738:head)])", 2024-09-13T11:26:21.220 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:40.805973+0000", 2024-09-13T11:26:21.220 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 160.39202853899999, 2024-09-13T11:26:21.220 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.28729672099999998, 2024-09-13T11:26:21.220 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:26:21.220 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:26:21.221 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-13T11:26:21.221 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:26:21.221 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.221 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:26:21.221 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:40.805973+0000", 2024-09-13T11:26:21.221 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:21.221 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.221 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.222 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:26:21.222 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:40.805973+0000", 2024-09-13T11:26:21.222 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:21.222 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.222 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.222 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:26:21.222 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:40.805974+0000", 2024-09-13T11:26:21.222 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.8599999999999998e-07 2024-09-13T11:26:21.222 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.222 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.222 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:26:21.222 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:40.805976+0000", 2024-09-13T11:26:21.222 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.2809999999999998e-06 2024-09-13T11:26:21.222 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.223 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.223 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:26:21.223 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.091588+0000", 2024-09-13T11:26:21.223 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.28561173899999998 2024-09-13T11:26:21.223 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.223 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.223 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:26:21.223 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.091596+0000", 2024-09-13T11:26:21.223 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.0520000000000002e-06 2024-09-13T11:26:21.223 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.223 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.223 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:26:21.223 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.091907+0000", 2024-09-13T11:26:21.223 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00031179599999999998 2024-09-13T11:26:21.223 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.224 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.224 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:26:21.224 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.093270+0000", 2024-09-13T11:26:21.224 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0013622669999999999 2024-09-13T11:26:21.224 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:21.224 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:26:21.224 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:21.224 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.224 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.224 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPushReply(3.1a 84/78 [PushReplyOp(3:581ea97e:::benchmark_data_smithi177_31116_object10255:head)])", 2024-09-13T11:26:21.224 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:41.514228+0000", 2024-09-13T11:26:21.225 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 159.683773051, 2024-09-13T11:26:21.225 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.40695564200000001, 2024-09-13T11:26:21.225 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:26:21.225 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:26:21.225 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-13T11:26:21.225 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:26:21.225 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.225 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:26:21.225 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.514228+0000", 2024-09-13T11:26:21.225 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:21.225 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.225 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.225 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:26:21.226 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.514228+0000", 2024-09-13T11:26:21.226 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:21.226 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.226 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.226 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:26:21.226 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.514230+0000", 2024-09-13T11:26:21.226 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.499e-06 2024-09-13T11:26:21.226 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.227 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.227 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:26:21.227 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.514236+0000", 2024-09-13T11:26:21.227 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.5389999999999998e-06 2024-09-13T11:26:21.227 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.227 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.227 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:26:21.228 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.919787+0000", 2024-09-13T11:26:21.228 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.40555152999999999 2024-09-13T11:26:21.228 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.228 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.228 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:26:21.228 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.919796+0000", 2024-09-13T11:26:21.228 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.4829999999999999e-06 2024-09-13T11:26:21.229 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.229 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.229 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:26:21.229 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.920366+0000", 2024-09-13T11:26:21.229 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00057061700000000004 2024-09-13T11:26:21.229 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.229 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.229 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:26:21.229 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.921184+0000", 2024-09-13T11:26:21.229 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00081797399999999996 2024-09-13T11:26:21.229 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:21.229 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:26:21.229 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:21.229 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.230 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.230 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:60427 3.14 3:2afea229:::benchmark_data_smithi177_31116_object60426:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e85)", 2024-09-13T11:26:21.230 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:42.187231+0000", 2024-09-13T11:26:21.230 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 159.01077018199999, 2024-09-13T11:26:21.230 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.83900596999999999, 2024-09-13T11:26:21.230 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:26:21.230 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:26:21.230 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:21.230 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:26:21.230 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:26:21.230 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:21.231 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 60427 2024-09-13T11:26:21.231 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.231 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:26:21.231 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.231 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:26:21.231 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.187231+0000", 2024-09-13T11:26:21.231 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:21.231 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.231 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.231 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:26:21.231 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.187231+0000", 2024-09-13T11:26:21.231 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:21.231 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.231 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.232 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:26:21.232 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.187235+0000", 2024-09-13T11:26:21.232 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.3579999999999999e-06 2024-09-13T11:26:21.232 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.232 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.232 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:26:21.232 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.187261+0000", 2024-09-13T11:26:21.232 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.6089e-05 2024-09-13T11:26:21.232 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.232 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.232 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:26:21.232 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.187275+0000", 2024-09-13T11:26:21.232 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.4653e-05 2024-09-13T11:26:21.233 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.233 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.233 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:26:21.233 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.187282+0000", 2024-09-13T11:26:21.233 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.8870000000000003e-06 2024-09-13T11:26:21.233 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.233 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.233 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:26:21.233 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.187336+0000", 2024-09-13T11:26:21.233 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.3875999999999997e-05 2024-09-13T11:26:21.234 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.234 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.234 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:26:21.234 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.187480+0000", 2024-09-13T11:26:21.234 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00014330899999999999 2024-09-13T11:26:21.234 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.234 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.234 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 14", 2024-09-13T11:26:21.234 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.187658+0000", 2024-09-13T11:26:21.234 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00017861900000000001 2024-09-13T11:26:21.234 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.234 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.234 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:26:21.235 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.189353+0000", 2024-09-13T11:26:21.235 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001695144 2024-09-13T11:26:21.235 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.235 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.235 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:21.235 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:43.026081+0000", 2024-09-13T11:26:21.235 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.83672736199999997 2024-09-13T11:26:21.235 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.235 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.235 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:26:21.236 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:43.026169+0000", 2024-09-13T11:26:21.236 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.8411000000000004e-05 2024-09-13T11:26:21.236 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.236 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.236 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:26:21.237 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:43.026237+0000", 2024-09-13T11:26:21.237 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.8262000000000005e-05 2024-09-13T11:26:21.237 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:21.237 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:26:21.237 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:21.240 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.240 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.240 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop_reply(client.4665.0:60427 3.14 e85/84 ondisk, result = 0)", 2024-09-13T11:26:21.241 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:42.190914+0000", 2024-09-13T11:26:21.241 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 159.00708751799999, 2024-09-13T11:26:21.241 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.83534071499999996, 2024-09-13T11:26:21.241 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:26:21.241 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:26:21.241 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:26:21.241 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:26:21.241 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.241 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:26:21.241 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.190914+0000", 2024-09-13T11:26:21.241 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:21.242 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.242 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.242 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:26:21.242 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.190914+0000", 2024-09-13T11:26:21.242 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:21.242 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.242 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.242 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:26:21.242 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.190916+0000", 2024-09-13T11:26:21.242 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.84e-06 2024-09-13T11:26:21.242 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.242 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.243 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:26:21.243 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.190921+0000", 2024-09-13T11:26:21.243 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.2039999999999996e-06 2024-09-13T11:26:21.243 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.243 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.243 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:26:21.243 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:43.025926+0000", 2024-09-13T11:26:21.243 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.83500542 2024-09-13T11:26:21.243 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.243 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.243 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:26:21.243 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:43.025937+0000", 2024-09-13T11:26:21.243 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0597e-05 2024-09-13T11:26:21.244 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.244 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.244 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:26:21.244 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:43.026052+0000", 2024-09-13T11:26:21.244 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000114838 2024-09-13T11:26:21.244 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.244 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.244 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:26:21.244 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:43.026070+0000", 2024-09-13T11:26:21.244 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.8131e-05 2024-09-13T11:26:21.244 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.244 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.244 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:26:21.245 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:43.026255+0000", 2024-09-13T11:26:21.245 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000184685 2024-09-13T11:26:21.245 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:21.245 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:26:21.245 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:21.245 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.245 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.245 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPushReply(3.1d 89/78 [PushReplyOp(3:bc1b7cfb:::benchmark_data_smithi177_31116_object22629:head)])", 2024-09-13T11:26:21.245 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:46.663106+0000", 2024-09-13T11:26:21.245 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 154.53489533499999, 2024-09-13T11:26:21.245 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.26645533199999999, 2024-09-13T11:26:21.245 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:26:21.245 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:26:21.246 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-13T11:26:21.246 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:26:21.246 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.246 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:26:21.246 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.663106+0000", 2024-09-13T11:26:21.246 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:21.246 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.246 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.246 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:26:21.246 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.663106+0000", 2024-09-13T11:26:21.246 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:21.247 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.247 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.247 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:26:21.247 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.663108+0000", 2024-09-13T11:26:21.247 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.0140000000000001e-06 2024-09-13T11:26:21.247 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.247 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.248 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:26:21.248 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.663116+0000", 2024-09-13T11:26:21.248 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.5440000000000001e-06 2024-09-13T11:26:21.248 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.248 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.248 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:26:21.248 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.928111+0000", 2024-09-13T11:26:21.248 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.26499566899999999 2024-09-13T11:26:21.248 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.248 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.249 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:26:21.249 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.928135+0000", 2024-09-13T11:26:21.249 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.4003e-05 2024-09-13T11:26:21.249 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.249 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.249 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:26:21.249 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.928559+0000", 2024-09-13T11:26:21.249 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000423318 2024-09-13T11:26:21.249 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.249 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.249 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:26:21.250 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.929562+0000", 2024-09-13T11:26:21.250 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0010027840000000001 2024-09-13T11:26:21.250 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:21.250 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:26:21.250 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:21.250 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.250 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.250 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "rep_scrubmap(3.14 e84 from shard 14)", 2024-09-13T11:26:21.250 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:48.313615+0000", 2024-09-13T11:26:21.250 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 152.88438654199999, 2024-09-13T11:26:21.250 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.35633836099999999, 2024-09-13T11:26:21.250 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:26:21.250 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:26:21.251 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:26:21.251 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:26:21.251 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.251 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:26:21.251 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.313615+0000", 2024-09-13T11:26:21.251 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:21.251 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.251 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.252 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:26:21.252 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.313615+0000", 2024-09-13T11:26:21.252 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:21.252 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.252 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.252 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:26:21.252 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.313617+0000", 2024-09-13T11:26:21.252 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.7990000000000001e-06 2024-09-13T11:26:21.252 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.253 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.253 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:26:21.253 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.313688+0000", 2024-09-13T11:26:21.253 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.1481000000000002e-05 2024-09-13T11:26:21.253 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.253 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.253 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:26:21.253 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.669785+0000", 2024-09-13T11:26:21.253 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.356096466 2024-09-13T11:26:21.253 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.253 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.253 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:26:21.253 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.669792+0000", 2024-09-13T11:26:21.253 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.1740000000000003e-06 2024-09-13T11:26:21.254 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.254 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.254 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:26:21.254 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.669839+0000", 2024-09-13T11:26:21.254 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.7323000000000001e-05 2024-09-13T11:26:21.254 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.254 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.254 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:26:21.255 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.669847+0000", 2024-09-13T11:26:21.255 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.0469999999999994e-06 2024-09-13T11:26:21.255 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.255 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.255 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:26:21.255 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.669953+0000", 2024-09-13T11:26:21.255 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00010607099999999999 2024-09-13T11:26:21.255 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:21.255 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:26:21.256 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:21.256 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.256 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.256 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:110268 3.14 3:2997c4c5:::benchmark_data_smithi177_31116_object110267:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119)", 2024-09-13T11:26:21.256 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:25.359386+0000", 2024-09-13T11:26:21.256 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 115.838615226, 2024-09-13T11:26:21.256 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.29072703, 2024-09-13T11:26:21.256 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:26:21.256 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:26:21.257 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:21.257 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:26:21.257 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:26:21.257 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:21.257 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 110268 2024-09-13T11:26:21.257 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.257 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:26:21.257 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.257 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:26:21.257 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.359386+0000", 2024-09-13T11:26:21.257 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:21.257 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.257 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.258 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:26:21.258 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.359386+0000", 2024-09-13T11:26:21.258 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:21.258 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.258 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.258 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:26:21.258 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.359389+0000", 2024-09-13T11:26:21.258 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.8600000000000001e-06 2024-09-13T11:26:21.258 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.259 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.259 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:26:21.259 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.359424+0000", 2024-09-13T11:26:21.259 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.4755999999999997e-05 2024-09-13T11:26:21.259 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.259 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.259 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:26:21.259 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.359445+0000", 2024-09-13T11:26:21.259 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.1104999999999999e-05 2024-09-13T11:26:21.259 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.260 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.260 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:26:21.260 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.359460+0000", 2024-09-13T11:26:21.260 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5396999999999998e-05 2024-09-13T11:26:21.260 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.260 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.260 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:26:21.260 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.359581+0000", 2024-09-13T11:26:21.260 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000120455 2024-09-13T11:26:21.260 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.261 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.261 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:26:21.261 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.359827+0000", 2024-09-13T11:26:21.261 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00024571 2024-09-13T11:26:21.261 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.261 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.261 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 14", 2024-09-13T11:26:21.261 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.360084+0000", 2024-09-13T11:26:21.262 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00025787999999999999 2024-09-13T11:26:21.262 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.262 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.262 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:26:21.262 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.362243+0000", 2024-09-13T11:26:21.262 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0021581579999999999 2024-09-13T11:26:21.262 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.262 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.262 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:21.262 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.650034+0000", 2024-09-13T11:26:21.262 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.287791662 2024-09-13T11:26:21.262 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.262 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.263 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:26:21.263 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.650077+0000", 2024-09-13T11:26:21.263 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.2367999999999998e-05 2024-09-13T11:26:21.263 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.263 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.263 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:26:21.263 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.650113+0000", 2024-09-13T11:26:21.263 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.6678999999999999e-05 2024-09-13T11:26:21.263 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:21.263 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:26:21.263 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:21.263 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.263 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.263 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:110279 3.14 3:2b17a349:::benchmark_data_smithi177_31116_object110278:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119)", 2024-09-13T11:26:21.264 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:25.645137+0000", 2024-09-13T11:26:21.264 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 115.552864388, 2024-09-13T11:26:21.264 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.28172927800000003, 2024-09-13T11:26:21.264 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:26:21.264 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:26:21.264 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:21.264 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:26:21.264 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:26:21.264 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:21.264 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 110279 2024-09-13T11:26:21.264 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.264 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:26:21.265 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.265 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:26:21.265 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.645137+0000", 2024-09-13T11:26:21.265 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:21.265 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.265 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.265 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:26:21.265 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.645137+0000", 2024-09-13T11:26:21.265 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:21.265 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.265 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.265 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:26:21.265 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.645140+0000", 2024-09-13T11:26:21.266 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.0939999999999999e-06 2024-09-13T11:26:21.266 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.266 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.266 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:26:21.266 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.645184+0000", 2024-09-13T11:26:21.266 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.3412999999999999e-05 2024-09-13T11:26:21.266 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.266 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.266 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:26:21.266 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.645206+0000", 2024-09-13T11:26:21.266 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.1997e-05 2024-09-13T11:26:21.266 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.266 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.266 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:26:21.266 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.645216+0000", 2024-09-13T11:26:21.267 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0740999999999999e-05 2024-09-13T11:26:21.267 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.267 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.267 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:26:21.267 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.645397+0000", 2024-09-13T11:26:21.267 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00018077600000000001 2024-09-13T11:26:21.267 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.267 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.267 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:26:21.267 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.645638+0000", 2024-09-13T11:26:21.267 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000241091 2024-09-13T11:26:21.267 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.267 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.267 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 14", 2024-09-13T11:26:21.268 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.645922+0000", 2024-09-13T11:26:21.268 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000283565 2024-09-13T11:26:21.268 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.268 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.268 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:26:21.268 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.648980+0000", 2024-09-13T11:26:21.268 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00305851 2024-09-13T11:26:21.268 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.268 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.268 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:21.269 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.926702+0000", 2024-09-13T11:26:21.269 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.27772199800000003 2024-09-13T11:26:21.269 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.269 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.269 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:26:21.269 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.926796+0000", 2024-09-13T11:26:21.269 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.4158000000000002e-05 2024-09-13T11:26:21.269 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.269 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.269 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:26:21.270 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.926866+0000", 2024-09-13T11:26:21.270 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.9935e-05 2024-09-13T11:26:21.270 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:21.270 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:26:21.270 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:21.270 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.271 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.271 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPush(3.15 143/122 [PushOp(3:ae52bb45:::benchmark_data_smithi177_31116_object61893:head, version: 87'3272, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:ae52bb45:::benchmark_data_smithi177_31116_object61893:head@87'3272, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:26:21.271 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:57.633939+0000", 2024-09-13T11:26:21.271 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 83.564062405000001, 2024-09-13T11:26:21.271 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.339405067, 2024-09-13T11:26:21.271 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:26:21.271 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:26:21.271 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:26:21.271 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:26:21.271 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.271 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:26:21.271 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:57.633939+0000", 2024-09-13T11:26:21.272 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:21.272 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.272 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.272 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:26:21.272 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:57.633939+0000", 2024-09-13T11:26:21.272 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:21.272 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.272 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.272 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:26:21.272 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:57.633941+0000", 2024-09-13T11:26:21.272 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.911e-06 2024-09-13T11:26:21.272 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.272 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.273 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:26:21.273 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:57.633994+0000", 2024-09-13T11:26:21.273 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.2753999999999997e-05 2024-09-13T11:26:21.273 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.273 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.273 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:26:21.273 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:57.968403+0000", 2024-09-13T11:26:21.273 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.334409336 2024-09-13T11:26:21.273 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.273 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.273 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:26:21.273 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:57.968451+0000", 2024-09-13T11:26:21.273 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.7525000000000003e-05 2024-09-13T11:26:21.273 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.274 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.274 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:26:21.274 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:57.968660+0000", 2024-09-13T11:26:21.274 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000209204 2024-09-13T11:26:21.274 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.274 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.274 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:26:21.274 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:57.968680+0000", 2024-09-13T11:26:21.274 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.9639999999999998e-05 2024-09-13T11:26:21.274 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.274 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.274 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:26:21.274 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:57.973344+0000", 2024-09-13T11:26:21.275 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0046646969999999998 2024-09-13T11:26:21.275 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:21.275 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:26:21.275 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:21.275 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.275 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.275 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:140658 3.17 e144/85 3:ece5a1b3:::benchmark_data_smithi177_31116_object355:head v 144'5654, mlcod=144'5652)", 2024-09-13T11:26:21.276 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:58.247792+0000", 2024-09-13T11:26:21.276 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 82.950209927000003, 2024-09-13T11:26:21.276 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.29148369699999999, 2024-09-13T11:26:21.276 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:26:21.276 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:26:21.276 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:21.276 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:26:21.276 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.276 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:26:21.276 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:58.247792+0000", 2024-09-13T11:26:21.276 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:21.276 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.276 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.277 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:26:21.277 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:58.247792+0000", 2024-09-13T11:26:21.277 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:21.277 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.277 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.277 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:26:21.277 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:58.247793+0000", 2024-09-13T11:26:21.277 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.347e-06 2024-09-13T11:26:21.277 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.277 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.277 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:26:21.277 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:58.247799+0000", 2024-09-13T11:26:21.277 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.5239999999999997e-06 2024-09-13T11:26:21.277 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.278 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.278 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:26:21.278 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:58.535226+0000", 2024-09-13T11:26:21.278 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.28742604900000002 2024-09-13T11:26:21.278 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.278 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.278 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:26:21.278 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:58.535238+0000", 2024-09-13T11:26:21.278 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.2843000000000001e-05 2024-09-13T11:26:21.278 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.278 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.278 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:26:21.278 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:58.535393+0000", 2024-09-13T11:26:21.278 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00015481699999999999 2024-09-13T11:26:21.279 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.279 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.279 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:26:21.279 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:58.535434+0000", 2024-09-13T11:26:21.279 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.1152999999999997e-05 2024-09-13T11:26:21.279 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.279 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.279 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:26:21.280 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:58.539191+0000", 2024-09-13T11:26:21.280 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0037571520000000001 2024-09-13T11:26:21.280 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.280 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.280 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:26:21.280 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:58.539275+0000", 2024-09-13T11:26:21.280 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.3812000000000003e-05 2024-09-13T11:26:21.280 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:21.280 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:26:21.280 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:21.281 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.281 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.281 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPush(3.15 144/122 [PushOp(3:a97ba025:::benchmark_data_smithi177_31116_object64775:head, version: 90'3365, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:a97ba025:::benchmark_data_smithi177_31116_object64775:head@90'3365, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:26:21.281 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:58.825014+0000", 2024-09-13T11:26:21.281 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 82.372987299000002, 2024-09-13T11:26:21.281 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.23868276499999999, 2024-09-13T11:26:21.281 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:26:21.281 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:26:21.281 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:26:21.281 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:26:21.282 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.282 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:26:21.282 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:58.825014+0000", 2024-09-13T11:26:21.282 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:21.282 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.282 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.282 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:26:21.282 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:58.825014+0000", 2024-09-13T11:26:21.282 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:21.282 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.282 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.282 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:26:21.282 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:58.825017+0000", 2024-09-13T11:26:21.282 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.7630000000000001e-06 2024-09-13T11:26:21.283 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.283 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.283 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:26:21.283 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:58.825099+0000", 2024-09-13T11:26:21.283 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.2035999999999999e-05 2024-09-13T11:26:21.283 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.283 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.283 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:26:21.283 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:59.059557+0000", 2024-09-13T11:26:21.283 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.23445790399999999 2024-09-13T11:26:21.283 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.283 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.283 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:26:21.283 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:59.059609+0000", 2024-09-13T11:26:21.284 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.2550999999999999e-05 2024-09-13T11:26:21.284 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.284 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.284 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:26:21.284 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:59.060117+0000", 2024-09-13T11:26:21.284 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00050754299999999995 2024-09-13T11:26:21.284 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.284 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.284 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:26:21.284 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:59.060139+0000", 2024-09-13T11:26:21.284 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.2306e-05 2024-09-13T11:26:21.284 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.284 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.284 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:26:21.285 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:59.063697+0000", 2024-09-13T11:26:21.285 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0035576620000000001 2024-09-13T11:26:21.285 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:21.285 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:26:21.285 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:21.285 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.285 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.285 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:211295 3.2 e185/122 3:433aa750:::benchmark_data_smithi177_31116_object70992:head v 185'5563, mlcod=185'5560)", 2024-09-13T11:26:21.285 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:25:55.461013+0000", 2024-09-13T11:26:21.285 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 25.736988240999999, 2024-09-13T11:26:21.285 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.28679000700000001, 2024-09-13T11:26:21.285 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:26:21.285 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:26:21.286 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:21.286 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:26:21.286 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.286 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:26:21.286 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:55.461013+0000", 2024-09-13T11:26:21.286 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:21.286 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.286 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.286 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:26:21.286 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:55.461013+0000", 2024-09-13T11:26:21.286 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:21.286 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.286 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.286 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:26:21.287 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:55.461015+0000", 2024-09-13T11:26:21.287 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5600000000000001e-06 2024-09-13T11:26:21.287 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.287 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.287 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:26:21.287 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:55.461024+0000", 2024-09-13T11:26:21.287 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.1640000000000007e-06 2024-09-13T11:26:21.287 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.287 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.287 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:26:21.287 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:55.746330+0000", 2024-09-13T11:26:21.287 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.28530630000000001 2024-09-13T11:26:21.287 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.288 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.288 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:26:21.288 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:55.746335+0000", 2024-09-13T11:26:21.288 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.8119999999999998e-06 2024-09-13T11:26:21.288 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.288 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.288 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:26:21.288 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:55.746377+0000", 2024-09-13T11:26:21.288 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.1947999999999999e-05 2024-09-13T11:26:21.288 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.288 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.288 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:26:21.288 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:55.746389+0000", 2024-09-13T11:26:21.289 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.2237e-05 2024-09-13T11:26:21.289 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.289 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.289 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:26:21.289 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:55.747767+0000", 2024-09-13T11:26:21.289 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0013774779999999999 2024-09-13T11:26:21.289 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.289 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.289 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:26:21.289 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:55.747803+0000", 2024-09-13T11:26:21.289 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.6507999999999997e-05 2024-09-13T11:26:21.289 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:21.289 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:26:21.290 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:21.290 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.290 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.290 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:221537 3.14 3.d5a7b194 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e191)", 2024-09-13T11:26:21.290 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:26:04.320591+0000", 2024-09-13T11:26:21.290 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 16.877410383000001, 2024-09-13T11:26:21.290 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.72342120499999996, 2024-09-13T11:26:21.290 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:26:21.290 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:26:21.290 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-13T11:26:21.290 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:26:21.290 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:26:21.290 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:21.291 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 221537 2024-09-13T11:26:21.291 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.291 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:26:21.291 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.291 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:26:21.291 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:04.320591+0000", 2024-09-13T11:26:21.291 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:21.291 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.291 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.291 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:26:21.291 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:04.320591+0000", 2024-09-13T11:26:21.291 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:21.291 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.291 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.292 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:26:21.292 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:04.320595+0000", 2024-09-13T11:26:21.292 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.6500000000000002e-06 2024-09-13T11:26:21.292 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.292 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.292 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:26:21.292 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:04.320604+0000", 2024-09-13T11:26:21.292 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.5400000000000001e-06 2024-09-13T11:26:21.292 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.292 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.292 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:26:21.292 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:04.320630+0000", 2024-09-13T11:26:21.292 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.5340000000000001e-05 2024-09-13T11:26:21.293 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.293 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.293 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:26:21.293 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:04.320644+0000", 2024-09-13T11:26:21.293 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.4008e-05 2024-09-13T11:26:21.293 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.293 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.293 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:26:21.293 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:04.320708+0000", 2024-09-13T11:26:21.293 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.4268000000000002e-05 2024-09-13T11:26:21.293 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.293 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.293 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for peered", 2024-09-13T11:26:21.293 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:04.320716+0000", 2024-09-13T11:26:21.294 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.2139999999999996e-06 2024-09-13T11:26:21.294 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.294 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.294 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:26:21.294 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:05.043955+0000", 2024-09-13T11:26:21.294 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.72323873599999999 2024-09-13T11:26:21.294 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.294 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.294 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:26:21.294 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:05.044012+0000", 2024-09-13T11:26:21.294 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.7448999999999998e-05 2024-09-13T11:26:21.294 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:21.294 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:26:21.294 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:21.295 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.295 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.295 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:221537 3.14 3:298de5ab:::benchmark_data_smithi177_31116_object81234:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e193)", 2024-09-13T11:26:21.295 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:26:06.066780+0000", 2024-09-13T11:26:21.295 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 15.131221708, 2024-09-13T11:26:21.295 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.26485703900000002, 2024-09-13T11:26:21.295 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:26:21.295 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:26:21.295 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:21.295 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:26:21.295 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:26:21.295 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:21.295 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 221537 2024-09-13T11:26:21.296 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.296 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:26:21.296 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.296 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:26:21.296 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:06.066780+0000", 2024-09-13T11:26:21.296 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:21.296 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.296 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.296 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:26:21.296 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:06.066780+0000", 2024-09-13T11:26:21.296 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:21.296 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.296 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.296 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:26:21.297 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:06.066784+0000", 2024-09-13T11:26:21.297 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.8e-06 2024-09-13T11:26:21.297 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.297 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.297 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:26:21.297 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:06.066796+0000", 2024-09-13T11:26:21.297 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.2317000000000001e-05 2024-09-13T11:26:21.297 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.297 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.297 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:26:21.297 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:06.066834+0000", 2024-09-13T11:26:21.297 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.765e-05 2024-09-13T11:26:21.297 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.297 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.298 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:26:21.298 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:06.066855+0000", 2024-09-13T11:26:21.298 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.1379e-05 2024-09-13T11:26:21.298 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.298 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.298 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:26:21.298 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:06.067121+0000", 2024-09-13T11:26:21.298 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00026621700000000002 2024-09-13T11:26:21.298 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.298 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.298 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for peered", 2024-09-13T11:26:21.298 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:06.067165+0000", 2024-09-13T11:26:21.298 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.3442999999999997e-05 2024-09-13T11:26:21.298 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.299 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.299 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:26:21.299 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:06.318533+0000", 2024-09-13T11:26:21.299 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.251368486 2024-09-13T11:26:21.299 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.299 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.299 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:26:21.299 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:06.318943+0000", 2024-09-13T11:26:21.299 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00040997899999999999 2024-09-13T11:26:21.299 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.299 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.299 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 14,15", 2024-09-13T11:26:21.299 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:06.319277+0000", 2024-09-13T11:26:21.299 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00033357900000000003 2024-09-13T11:26:21.299 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.300 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.300 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:21.300 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:06.326462+0000", 2024-09-13T11:26:21.300 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0071850350000000002 2024-09-13T11:26:21.300 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.300 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.300 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:21.300 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:06.331274+0000", 2024-09-13T11:26:21.300 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0048118079999999999 2024-09-13T11:26:21.300 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.300 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.300 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:26:21.300 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:06.331312+0000", 2024-09-13T11:26:21.300 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.8917e-05 2024-09-13T11:26:21.301 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.301 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.301 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:26:21.301 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:06.331436+0000", 2024-09-13T11:26:21.301 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00012390799999999999 2024-09-13T11:26:21.301 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.301 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.301 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:26:21.301 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:06.331637+0000", 2024-09-13T11:26:21.301 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000200521 2024-09-13T11:26:21.301 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:21.301 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:26:21.301 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:21.301 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.302 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.302 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:221570 3.14 3:289328c0:::benchmark_data_smithi177_31116_object81267:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e193)", 2024-09-13T11:26:21.302 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:26:06.066892+0000", 2024-09-13T11:26:21.302 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 15.131109876, 2024-09-13T11:26:21.302 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.27338219400000002, 2024-09-13T11:26:21.302 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:26:21.302 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:26:21.302 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:21.302 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:26:21.302 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:26:21.302 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:21.302 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 221570 2024-09-13T11:26:21.302 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.302 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:26:21.303 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.303 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:26:21.303 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:06.066892+0000", 2024-09-13T11:26:21.303 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:21.303 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.303 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.303 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:26:21.303 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:06.066892+0000", 2024-09-13T11:26:21.303 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:21.303 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.303 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.303 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:26:21.303 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:06.066894+0000", 2024-09-13T11:26:21.303 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.0600000000000002e-06 2024-09-13T11:26:21.304 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.304 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.304 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:26:21.304 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:06.066902+0000", 2024-09-13T11:26:21.304 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.1799999999999996e-06 2024-09-13T11:26:21.304 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.304 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.304 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:26:21.304 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:06.066919+0000", 2024-09-13T11:26:21.304 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.6807e-05 2024-09-13T11:26:21.304 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.304 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.304 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:26:21.304 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:06.066940+0000", 2024-09-13T11:26:21.305 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.1189999999999999e-05 2024-09-13T11:26:21.305 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.305 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.305 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:26:21.305 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:06.067283+0000", 2024-09-13T11:26:21.305 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000342846 2024-09-13T11:26:21.305 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.305 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.305 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for peered", 2024-09-13T11:26:21.305 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:06.067306+0000", 2024-09-13T11:26:21.305 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.3292e-05 2024-09-13T11:26:21.305 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.305 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.305 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:26:21.306 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:06.326644+0000", 2024-09-13T11:26:21.306 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.25933820600000002 2024-09-13T11:26:21.306 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.306 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.306 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:26:21.306 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:06.326791+0000", 2024-09-13T11:26:21.306 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00014677399999999999 2024-09-13T11:26:21.306 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.306 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.306 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 14,15", 2024-09-13T11:26:21.306 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:06.326916+0000", 2024-09-13T11:26:21.306 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00012494700000000001 2024-09-13T11:26:21.306 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.306 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.307 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:21.307 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:06.331718+0000", 2024-09-13T11:26:21.307 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0048017779999999996 2024-09-13T11:26:21.307 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.307 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.307 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:26:21.307 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:06.337102+0000", 2024-09-13T11:26:21.307 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0053843099999999998 2024-09-13T11:26:21.307 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.307 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.307 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:21.307 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:06.340124+0000", 2024-09-13T11:26:21.307 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.003022158 2024-09-13T11:26:21.307 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.308 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.308 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:26:21.308 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:06.340179+0000", 2024-09-13T11:26:21.308 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.4863e-05 2024-09-13T11:26:21.308 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.308 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.308 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:26:21.308 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:06.340274+0000", 2024-09-13T11:26:21.308 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.4783000000000004e-05 2024-09-13T11:26:21.308 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:21.308 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:26:21.308 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:21.308 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.309 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.309 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:221576 3.14 3:2b549bdf:::benchmark_data_smithi177_31116_object81273:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e193)", 2024-09-13T11:26:21.309 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:26:06.066961+0000", 2024-09-13T11:26:21.309 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 15.131040083, 2024-09-13T11:26:21.309 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.27360902999999998, 2024-09-13T11:26:21.309 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:26:21.309 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:26:21.309 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:21.309 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:26:21.309 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:26:21.309 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:21.309 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 221576 2024-09-13T11:26:21.309 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.309 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:26:21.310 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.310 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:26:21.310 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:06.066961+0000", 2024-09-13T11:26:21.310 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:21.310 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.310 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.310 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:26:21.310 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:06.066961+0000", 2024-09-13T11:26:21.310 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:21.310 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.310 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.310 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:26:21.310 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:06.066964+0000", 2024-09-13T11:26:21.310 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.0700000000000001e-06 2024-09-13T11:26:21.311 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.311 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.311 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:26:21.311 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:06.066971+0000", 2024-09-13T11:26:21.311 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.046e-06 2024-09-13T11:26:21.311 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.311 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.311 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:26:21.311 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:06.066986+0000", 2024-09-13T11:26:21.311 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.4977000000000001e-05 2024-09-13T11:26:21.311 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.311 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.311 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:26:21.311 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:06.066999+0000", 2024-09-13T11:26:21.311 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3879999999999999e-05 2024-09-13T11:26:21.312 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.312 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.312 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:26:21.312 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:06.067368+0000", 2024-09-13T11:26:21.312 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00036871199999999999 2024-09-13T11:26:21.312 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.312 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.312 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for peered", 2024-09-13T11:26:21.312 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:06.067380+0000", 2024-09-13T11:26:21.312 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.1732999999999999e-05 2024-09-13T11:26:21.312 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.312 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.312 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:26:21.313 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:06.331786+0000", 2024-09-13T11:26:21.313 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.26440598300000001 2024-09-13T11:26:21.313 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.313 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.313 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:26:21.313 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:06.332023+0000", 2024-09-13T11:26:21.313 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00023757300000000001 2024-09-13T11:26:21.313 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.313 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.313 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 14,15", 2024-09-13T11:26:21.313 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:06.332273+0000", 2024-09-13T11:26:21.313 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00024968899999999999 2024-09-13T11:26:21.313 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.313 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.314 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:21.314 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:06.340349+0000", 2024-09-13T11:26:21.314 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.008076214 2024-09-13T11:26:21.314 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.314 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.314 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:21.314 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:06.340432+0000", 2024-09-13T11:26:21.314 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.2671999999999997e-05 2024-09-13T11:26:21.314 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.314 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.314 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:26:21.314 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:06.340446+0000", 2024-09-13T11:26:21.314 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3577000000000001e-05 2024-09-13T11:26:21.314 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.315 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.315 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:26:21.315 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:06.340489+0000", 2024-09-13T11:26:21.315 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.3636e-05 2024-09-13T11:26:21.315 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.315 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.315 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:26:21.315 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:06.340570+0000", 2024-09-13T11:26:21.315 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.1267999999999995e-05 2024-09-13T11:26:21.315 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:21.315 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:26:21.315 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:21.315 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:21.315 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:26:21.316 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-09-13T11:26:21.317 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-13T11:26:21.318 INFO:tasks.ceph.osd.12.smithi001.stderr:2024-09-13T11:26:21.314+0000 7f892c6e6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26314) UID: 0 2024-09-13T11:26:21.418 INFO:tasks.ceph.osd.9.smithi092.stderr:2024-09-13T11:26:21.416+0000 7ff5ab3d0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26348) UID: 0 2024-09-13T11:26:21.445 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-09-13T11:26:21.446 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [ 2024-09-13T11:26:21.446 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.446 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:237797 3.13 e204/83 3:ccabdbf4:::benchmark_data_smithi177_31116_object97494:head v 204'8830, mlcod=204'8828)", 2024-09-13T11:26:21.446 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:26:21.436266+0000", 2024-09-13T11:26:21.446 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 0.010554944, 2024-09-13T11:26:21.446 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.010572867999999999, 2024-09-13T11:26:21.446 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:26:21.446 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:26:21.446 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:26:21.446 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:26:21.446 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.446 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:26:21.446 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:21.436266+0000", 2024-09-13T11:26:21.447 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:21.447 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.447 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.447 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:26:21.447 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:21.436266+0000", 2024-09-13T11:26:21.447 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:21.447 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.447 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.447 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:26:21.447 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:21.436268+0000", 2024-09-13T11:26:21.447 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.7829999999999999e-06 2024-09-13T11:26:21.448 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.448 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.448 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:26:21.448 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:21.436275+0000", 2024-09-13T11:26:21.448 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.2969999999999998e-06 2024-09-13T11:26:21.448 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.448 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.448 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:26:21.448 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:21.444338+0000", 2024-09-13T11:26:21.448 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.008062952 2024-09-13T11:26:21.448 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.448 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.449 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:26:21.449 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:21.444343+0000", 2024-09-13T11:26:21.449 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.3929999999999999e-06 2024-09-13T11:26:21.449 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.449 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.449 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:26:21.449 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:21.444395+0000", 2024-09-13T11:26:21.449 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.1761000000000001e-05 2024-09-13T11:26:21.449 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.449 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.449 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:26:21.449 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:21.444417+0000", 2024-09-13T11:26:21.449 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.1713999999999999e-05 2024-09-13T11:26:21.450 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:21.450 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:26:21.450 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:21.450 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:21.450 INFO:teuthology.orchestra.run.smithi177.stdout: ], 2024-09-13T11:26:21.450 INFO:teuthology.orchestra.run.smithi177.stdout: "num_ops": 1 2024-09-13T11:26:21.450 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-09-13T11:26:21.460 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-13T11:26:21.519 INFO:tasks.ceph.osd.1.smithi092.stderr:2024-09-13T11:26:21.516+0000 7fe7789c5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33718) UID: 0 2024-09-13T11:26:21.585 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-09-13T11:26:21.585 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [], 2024-09-13T11:26:21.585 INFO:teuthology.orchestra.run.smithi177.stdout: "complaint_time": 30, 2024-09-13T11:26:21.585 INFO:teuthology.orchestra.run.smithi177.stdout: "num_blocked_ops": 0 2024-09-13T11:26:21.585 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-09-13T11:26:21.601 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-13T11:26:21.621 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-09-13T11:26:21.619+0000 7f97befc9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26315) UID: 0 2024-09-13T11:26:21.722 INFO:tasks.ceph.osd.8.smithi001.stderr:2024-09-13T11:26:21.718+0000 7fb9b48ae640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26312) UID: 0 2024-09-13T11:26:21.747 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-09-13T11:26:21.747 INFO:teuthology.orchestra.run.smithi177.stdout: "size": 20, 2024-09-13T11:26:21.747 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 600, 2024-09-13T11:26:21.747 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [ 2024-09-13T11:26:21.747 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.747 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPushReply(3.13 75/70 [PushReplyOp(3:cfab03d6:::benchmark_data_smithi177_31116_object16599:head)])", 2024-09-13T11:26:21.747 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:31.163480+0000", 2024-09-13T11:26:21.748 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 170.58293362800001, 2024-09-13T11:26:21.748 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.63009029299999997, 2024-09-13T11:26:21.748 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:26:21.748 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:26:21.748 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-13T11:26:21.748 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:26:21.748 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.748 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:26:21.748 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.163480+0000", 2024-09-13T11:26:21.748 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:21.748 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.748 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.748 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:26:21.749 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.163480+0000", 2024-09-13T11:26:21.749 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:21.749 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.749 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.749 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:26:21.749 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.163481+0000", 2024-09-13T11:26:21.749 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0410000000000001e-06 2024-09-13T11:26:21.749 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.749 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.749 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:26:21.749 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.163485+0000", 2024-09-13T11:26:21.749 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.1629999999999998e-06 2024-09-13T11:26:21.749 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.749 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.749 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:26:21.750 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.792166+0000", 2024-09-13T11:26:21.750 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.62868067400000005 2024-09-13T11:26:21.750 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.750 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.750 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:26:21.750 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.792184+0000", 2024-09-13T11:26:21.750 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.7898999999999999e-05 2024-09-13T11:26:21.750 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.750 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.750 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:26:21.751 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.792312+0000", 2024-09-13T11:26:21.751 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00012871000000000001 2024-09-13T11:26:21.751 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.751 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.751 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:26:21.751 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.793570+0000", 2024-09-13T11:26:21.751 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001257806 2024-09-13T11:26:21.751 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:21.751 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:26:21.751 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:21.751 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.751 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.751 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPushReply(3.13 76/70 [PushReplyOp(3:c9d43a82:::benchmark_data_smithi177_31116_object20026:head)])", 2024-09-13T11:26:21.751 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:32.427709+0000", 2024-09-13T11:26:21.752 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 169.318704113, 2024-09-13T11:26:21.752 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.46547316300000002, 2024-09-13T11:26:21.752 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:26:21.752 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:26:21.752 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-13T11:26:21.752 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:26:21.752 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.752 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:26:21.752 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:32.427709+0000", 2024-09-13T11:26:21.752 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:21.752 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.752 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.753 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:26:21.753 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:32.427709+0000", 2024-09-13T11:26:21.753 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:21.753 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.753 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.753 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:26:21.753 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:32.427711+0000", 2024-09-13T11:26:21.753 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5099999999999999e-06 2024-09-13T11:26:21.753 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.753 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.753 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:26:21.753 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:32.427717+0000", 2024-09-13T11:26:21.754 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.1569999999999997e-06 2024-09-13T11:26:21.754 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.754 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.754 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:26:21.754 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:32.892390+0000", 2024-09-13T11:26:21.754 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.46467251100000001 2024-09-13T11:26:21.754 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.754 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.754 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:26:21.754 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:32.892405+0000", 2024-09-13T11:26:21.754 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5296999999999999e-05 2024-09-13T11:26:21.754 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.754 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.754 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:26:21.755 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:32.892503+0000", 2024-09-13T11:26:21.755 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.7820999999999997e-05 2024-09-13T11:26:21.755 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.755 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.755 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:26:21.755 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:32.893183+0000", 2024-09-13T11:26:21.755 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00067986700000000001 2024-09-13T11:26:21.755 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:21.755 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:26:21.755 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:21.755 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.755 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.755 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:57435 3.3 e80/75 3:c0db4777:::benchmark_data_smithi177_31116_object57434:head v 80'3155, mlcod=80'3153)", 2024-09-13T11:26:21.755 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:37.437070+0000", 2024-09-13T11:26:21.755 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 164.30934329600001, 2024-09-13T11:26:21.756 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.35861845999999997, 2024-09-13T11:26:21.756 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:26:21.756 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:26:21.756 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:21.756 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:26:21.756 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.756 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:26:21.756 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:37.437070+0000", 2024-09-13T11:26:21.756 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:21.756 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.756 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.756 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:26:21.756 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:37.437070+0000", 2024-09-13T11:26:21.757 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:21.757 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.757 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.757 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:26:21.757 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:37.437072+0000", 2024-09-13T11:26:21.757 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5549999999999999e-06 2024-09-13T11:26:21.757 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.757 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.757 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:26:21.757 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:37.437141+0000", 2024-09-13T11:26:21.757 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.9494999999999998e-05 2024-09-13T11:26:21.757 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.758 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.758 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:26:21.758 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:37.793400+0000", 2024-09-13T11:26:21.758 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.35625817999999998 2024-09-13T11:26:21.758 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.758 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.758 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:26:21.758 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:37.793406+0000", 2024-09-13T11:26:21.758 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.6830000000000002e-06 2024-09-13T11:26:21.758 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.759 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.759 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:26:21.759 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:37.793472+0000", 2024-09-13T11:26:21.759 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.5671000000000002e-05 2024-09-13T11:26:21.759 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.759 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.759 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:26:21.759 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:37.793492+0000", 2024-09-13T11:26:21.759 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.0051000000000002e-05 2024-09-13T11:26:21.759 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.759 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.760 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:26:21.760 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:37.795626+0000", 2024-09-13T11:26:21.760 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0021344990000000002 2024-09-13T11:26:21.760 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.760 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.760 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:26:21.760 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:37.795689+0000", 2024-09-13T11:26:21.760 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.2325999999999999e-05 2024-09-13T11:26:21.760 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:21.760 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:26:21.760 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:21.760 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.760 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.760 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:57985 3.3 e80/75 3:c330bb03:::benchmark_data_smithi177_31116_object57984:head v 80'3162, mlcod=80'3159)", 2024-09-13T11:26:21.760 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:38.007076+0000", 2024-09-13T11:26:21.761 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 163.739337944, 2024-09-13T11:26:21.761 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.18211438199999999, 2024-09-13T11:26:21.761 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:26:21.761 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:26:21.761 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:21.761 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:26:21.761 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.761 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:26:21.761 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:38.007076+0000", 2024-09-13T11:26:21.761 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:21.761 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.761 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.761 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:26:21.761 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:38.007076+0000", 2024-09-13T11:26:21.762 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:21.762 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.762 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.762 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:26:21.762 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:38.007077+0000", 2024-09-13T11:26:21.762 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.6759999999999999e-06 2024-09-13T11:26:21.762 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.762 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.762 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:26:21.762 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:38.007221+0000", 2024-09-13T11:26:21.762 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00014344299999999999 2024-09-13T11:26:21.762 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.762 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.762 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:26:21.763 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:38.186808+0000", 2024-09-13T11:26:21.763 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.17958726699999999 2024-09-13T11:26:21.763 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.763 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.763 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:26:21.763 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:38.186815+0000", 2024-09-13T11:26:21.763 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.0890000000000002e-06 2024-09-13T11:26:21.763 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.763 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.763 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:26:21.763 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:38.186887+0000", 2024-09-13T11:26:21.763 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.2006000000000001e-05 2024-09-13T11:26:21.763 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.764 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.764 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:26:21.764 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:38.186913+0000", 2024-09-13T11:26:21.764 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.5962e-05 2024-09-13T11:26:21.764 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.764 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.764 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:26:21.764 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:38.189123+0000", 2024-09-13T11:26:21.764 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0022099649999999999 2024-09-13T11:26:21.764 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.764 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.764 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:26:21.764 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:38.189190+0000", 2024-09-13T11:26:21.764 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.6973999999999999e-05 2024-09-13T11:26:21.765 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:21.765 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:26:21.765 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:21.765 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.765 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.765 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPushReply(3.23 84/78 [PushReplyOp(3:c7e44616:::benchmark_data_smithi177_31116_object32371:head)])", 2024-09-13T11:26:21.765 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:41.338210+0000", 2024-09-13T11:26:21.765 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 160.40820322499999, 2024-09-13T11:26:21.765 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.41118492299999998, 2024-09-13T11:26:21.765 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:26:21.765 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:26:21.765 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-13T11:26:21.765 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:26:21.765 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.766 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:26:21.766 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.338210+0000", 2024-09-13T11:26:21.766 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:21.766 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.766 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.766 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:26:21.766 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.338210+0000", 2024-09-13T11:26:21.766 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:21.766 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.766 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.766 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:26:21.766 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.338212+0000", 2024-09-13T11:26:21.766 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.314e-06 2024-09-13T11:26:21.766 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.766 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.767 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:26:21.767 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.338219+0000", 2024-09-13T11:26:21.767 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.6469999999999998e-06 2024-09-13T11:26:21.767 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.767 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.767 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:26:21.767 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.748675+0000", 2024-09-13T11:26:21.767 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.41045567500000002 2024-09-13T11:26:21.767 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.767 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.767 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:26:21.767 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.748684+0000", 2024-09-13T11:26:21.767 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.2150000000000007e-06 2024-09-13T11:26:21.767 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.768 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.768 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:26:21.768 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.748739+0000", 2024-09-13T11:26:21.768 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.4543000000000003e-05 2024-09-13T11:26:21.768 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.768 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.768 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:26:21.768 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.749395+0000", 2024-09-13T11:26:21.768 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00065652899999999999 2024-09-13T11:26:21.768 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:21.768 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:26:21.768 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:21.768 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.768 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.769 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPushReply(3.23 85/78 [PushReplyOp(3:c5c6dce1:::benchmark_data_smithi177_31116_object37293:head)])", 2024-09-13T11:26:21.769 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:42.377862+0000", 2024-09-13T11:26:21.769 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 159.368551766, 2024-09-13T11:26:21.769 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.62538808599999995, 2024-09-13T11:26:21.769 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:26:21.769 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:26:21.769 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-13T11:26:21.769 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:26:21.769 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.769 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:26:21.769 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.377862+0000", 2024-09-13T11:26:21.769 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:21.769 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.769 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.769 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:26:21.770 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.377862+0000", 2024-09-13T11:26:21.770 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:21.770 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.770 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.770 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:26:21.770 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.377863+0000", 2024-09-13T11:26:21.770 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.6899999999999996e-07 2024-09-13T11:26:21.770 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.770 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.770 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:26:21.770 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.377866+0000", 2024-09-13T11:26:21.770 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.219e-06 2024-09-13T11:26:21.771 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.771 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.771 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:26:21.771 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:43.001857+0000", 2024-09-13T11:26:21.771 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.62399084299999996 2024-09-13T11:26:21.771 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.771 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.771 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:26:21.771 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:43.001869+0000", 2024-09-13T11:26:21.771 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.2415999999999999e-05 2024-09-13T11:26:21.771 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.771 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.771 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:26:21.771 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:43.002194+0000", 2024-09-13T11:26:21.772 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000324774 2024-09-13T11:26:21.772 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.772 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.772 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:26:21.772 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:43.003250+0000", 2024-09-13T11:26:21.772 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001055865 2024-09-13T11:26:21.772 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:21.772 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:26:21.772 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:21.772 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.773 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.773 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPush(3.1c 89/77 [PushOp(3:38bcbc71:::benchmark_data_smithi177_31116_object31448:head, version: 59'1946, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:38bcbc71:::benchmark_data_smithi177_31116_object31448:head@59'1946, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:26:21.773 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:46.670723+0000", 2024-09-13T11:26:21.773 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 155.07569054199999, 2024-09-13T11:26:21.773 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.27576065, 2024-09-13T11:26:21.773 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:26:21.773 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:26:21.773 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:26:21.773 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:26:21.773 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.773 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:26:21.773 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.670723+0000", 2024-09-13T11:26:21.773 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:21.773 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.773 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.774 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:26:21.774 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.670723+0000", 2024-09-13T11:26:21.774 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:21.774 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.774 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.774 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:26:21.774 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.670724+0000", 2024-09-13T11:26:21.774 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.4389999999999999e-06 2024-09-13T11:26:21.774 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.774 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.774 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:26:21.774 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.670754+0000", 2024-09-13T11:26:21.775 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.9224999999999999e-05 2024-09-13T11:26:21.775 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.775 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.775 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:26:21.775 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.944369+0000", 2024-09-13T11:26:21.775 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.27361529800000001 2024-09-13T11:26:21.775 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.775 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.775 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:26:21.775 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.944422+0000", 2024-09-13T11:26:21.775 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.3318e-05 2024-09-13T11:26:21.775 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.775 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.775 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:26:21.776 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.944711+0000", 2024-09-13T11:26:21.776 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00028912399999999998 2024-09-13T11:26:21.776 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.776 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.776 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:26:21.776 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.944735+0000", 2024-09-13T11:26:21.776 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.3366999999999998e-05 2024-09-13T11:26:21.777 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.777 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.777 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:26:21.777 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.946484+0000", 2024-09-13T11:26:21.777 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0017488790000000001 2024-09-13T11:26:21.777 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:21.777 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:26:21.777 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:21.778 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.778 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.778 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPush(3.1c 91/77 [PushOp(3:3e18abc2:::benchmark_data_smithi177_31116_object35874:head, version: 62'2235, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:3e18abc2:::benchmark_data_smithi177_31116_object35874:head@62'2235, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:26:21.778 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:48.350968+0000", 2024-09-13T11:26:21.778 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 153.39544522899999, 2024-09-13T11:26:21.778 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.19257698000000001, 2024-09-13T11:26:21.778 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:26:21.778 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:26:21.778 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:26:21.778 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:26:21.778 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.778 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:26:21.778 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.350968+0000", 2024-09-13T11:26:21.778 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:21.779 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.779 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.779 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:26:21.779 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.350968+0000", 2024-09-13T11:26:21.779 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:21.779 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.779 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.779 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:26:21.779 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.350970+0000", 2024-09-13T11:26:21.779 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.796e-06 2024-09-13T11:26:21.779 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.779 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.779 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:26:21.779 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.351003+0000", 2024-09-13T11:26:21.779 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.2631e-05 2024-09-13T11:26:21.780 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.780 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.780 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:26:21.780 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.542140+0000", 2024-09-13T11:26:21.780 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.191136737 2024-09-13T11:26:21.780 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.780 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.780 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:26:21.780 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.542158+0000", 2024-09-13T11:26:21.780 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.8896999999999999e-05 2024-09-13T11:26:21.780 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.780 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.780 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:26:21.780 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.542263+0000", 2024-09-13T11:26:21.781 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000104158 2024-09-13T11:26:21.781 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.781 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.781 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:26:21.781 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.542279+0000", 2024-09-13T11:26:21.781 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5926999999999998e-05 2024-09-13T11:26:21.781 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.781 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.781 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:26:21.781 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.543545+0000", 2024-09-13T11:26:21.781 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0012668340000000001 2024-09-13T11:26:21.781 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:21.781 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:26:21.781 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:21.782 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.782 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.782 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:83211 3.1c e106/77 3:3ae31a0f:::benchmark_data_smithi177_31116_object83210:head v 106'4362, mlcod=106'4360)", 2024-09-13T11:26:21.782 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:04.345264+0000", 2024-09-13T11:26:21.782 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 137.40114957700001, 2024-09-13T11:26:21.782 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.17036547399999999, 2024-09-13T11:26:21.783 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:26:21.783 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:26:21.783 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:21.783 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:26:21.783 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.783 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:26:21.783 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.345264+0000", 2024-09-13T11:26:21.783 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:21.783 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.783 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.783 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:26:21.783 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.345264+0000", 2024-09-13T11:26:21.783 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:21.784 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.784 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.784 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:26:21.784 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.345267+0000", 2024-09-13T11:26:21.784 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.7590000000000002e-06 2024-09-13T11:26:21.784 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.784 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.784 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:26:21.784 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.345298+0000", 2024-09-13T11:26:21.784 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.1173000000000001e-05 2024-09-13T11:26:21.784 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.784 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.784 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:26:21.784 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.512265+0000", 2024-09-13T11:26:21.785 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.16696657100000001 2024-09-13T11:26:21.785 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.785 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.785 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:26:21.785 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.512279+0000", 2024-09-13T11:26:21.785 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.4953999999999999e-05 2024-09-13T11:26:21.785 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.785 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.785 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:26:21.785 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.512402+0000", 2024-09-13T11:26:21.785 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00012277999999999999 2024-09-13T11:26:21.785 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.785 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.785 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:26:21.785 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.512440+0000", 2024-09-13T11:26:21.786 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.7861999999999998e-05 2024-09-13T11:26:21.786 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.786 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.786 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:26:21.786 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.515515+0000", 2024-09-13T11:26:21.786 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00307472 2024-09-13T11:26:21.786 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.786 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.786 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:26:21.786 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.515629+0000", 2024-09-13T11:26:21.786 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00011465500000000001 2024-09-13T11:26:21.786 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:21.786 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:26:21.786 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:21.787 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.787 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.787 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:83223 3.1c e106/77 3:3bc6018d:::benchmark_data_smithi177_31116_object83222:head v 106'4363, mlcod=106'4360)", 2024-09-13T11:26:21.787 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:04.512139+0000", 2024-09-13T11:26:21.787 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 137.23427443099999, 2024-09-13T11:26:21.787 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.471923235, 2024-09-13T11:26:21.787 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:26:21.787 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:26:21.787 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:21.787 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:26:21.787 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.787 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:26:21.787 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.512139+0000", 2024-09-13T11:26:21.787 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:21.787 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.788 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.788 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:26:21.788 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.512139+0000", 2024-09-13T11:26:21.788 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:21.788 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.788 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.788 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:26:21.788 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.512143+0000", 2024-09-13T11:26:21.788 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.427e-06 2024-09-13T11:26:21.788 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.788 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.788 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:26:21.788 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.512227+0000", 2024-09-13T11:26:21.788 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.3965e-05 2024-09-13T11:26:21.789 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.789 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.789 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:26:21.789 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.976330+0000", 2024-09-13T11:26:21.789 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.46410342599999999 2024-09-13T11:26:21.789 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.789 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.789 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:26:21.789 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.976350+0000", 2024-09-13T11:26:21.789 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.9533000000000001e-05 2024-09-13T11:26:21.789 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.789 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.789 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:26:21.789 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.976488+0000", 2024-09-13T11:26:21.789 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00013799400000000001 2024-09-13T11:26:21.790 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.790 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.790 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:26:21.790 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.976543+0000", 2024-09-13T11:26:21.790 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.5442000000000002e-05 2024-09-13T11:26:21.790 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.790 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.790 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:26:21.790 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.983227+0000", 2024-09-13T11:26:21.790 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0066842899999999998 2024-09-13T11:26:21.790 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.790 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.790 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:26:21.790 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.984062+0000", 2024-09-13T11:26:21.791 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00083515800000000002 2024-09-13T11:26:21.791 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:21.791 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:26:21.791 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:21.791 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.791 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.791 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:110256 3.13 e119/83 3:cb00d078:::benchmark_data_smithi177_31116_object110255:head v 119'4823, mlcod=119'4821)", 2024-09-13T11:26:21.791 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:25.284530+0000", 2024-09-13T11:26:21.791 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 116.46188397100001, 2024-09-13T11:26:21.791 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.343491771, 2024-09-13T11:26:21.791 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:26:21.791 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:26:21.791 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:21.791 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:26:21.792 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.792 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:26:21.792 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.284530+0000", 2024-09-13T11:26:21.792 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:21.792 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.792 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.792 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:26:21.792 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.284530+0000", 2024-09-13T11:26:21.792 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:21.792 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.792 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.792 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:26:21.792 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.284532+0000", 2024-09-13T11:26:21.792 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.8870000000000001e-06 2024-09-13T11:26:21.792 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.793 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.793 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:26:21.793 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.284558+0000", 2024-09-13T11:26:21.793 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.6899000000000001e-05 2024-09-13T11:26:21.793 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.793 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.793 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:26:21.793 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.626172+0000", 2024-09-13T11:26:21.793 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.34161370800000002 2024-09-13T11:26:21.793 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.793 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.793 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:26:21.793 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.626187+0000", 2024-09-13T11:26:21.793 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.485e-05 2024-09-13T11:26:21.794 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.794 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.794 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:26:21.794 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.626245+0000", 2024-09-13T11:26:21.794 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.7998000000000003e-05 2024-09-13T11:26:21.794 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.794 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.794 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:26:21.794 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.626260+0000", 2024-09-13T11:26:21.794 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5429000000000001e-05 2024-09-13T11:26:21.794 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.794 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.794 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:26:21.794 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.627924+0000", 2024-09-13T11:26:21.794 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0016634060000000001 2024-09-13T11:26:21.795 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.795 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.795 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:26:21.795 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.628021+0000", 2024-09-13T11:26:21.795 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.7594000000000001e-05 2024-09-13T11:26:21.795 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:21.795 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:26:21.795 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:21.795 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.795 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.795 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPush(3.2b 121/119 [PushOp(3:d53a23e8:::benchmark_data_smithi177_31116_object5366:head, version: 45'342, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:d53a23e8:::benchmark_data_smithi177_31116_object5366:head@45'342, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:26:21.795 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:27.268731+0000", 2024-09-13T11:26:21.795 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 114.477682509, 2024-09-13T11:26:21.795 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.72228345000000005, 2024-09-13T11:26:21.796 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:26:21.796 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:26:21.796 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:26:21.796 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:26:21.796 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.796 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:26:21.796 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:27.268731+0000", 2024-09-13T11:26:21.796 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:21.796 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.796 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.796 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:26:21.796 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:27.268731+0000", 2024-09-13T11:26:21.796 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:21.796 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.796 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.797 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:26:21.797 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:27.268733+0000", 2024-09-13T11:26:21.797 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.9740000000000001e-06 2024-09-13T11:26:21.797 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.797 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.797 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:26:21.797 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:27.268778+0000", 2024-09-13T11:26:21.797 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.4679999999999999e-05 2024-09-13T11:26:21.797 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.797 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.797 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:26:21.797 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:27.988594+0000", 2024-09-13T11:26:21.797 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.71981667299999996 2024-09-13T11:26:21.797 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.798 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.798 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:26:21.798 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:27.988652+0000", 2024-09-13T11:26:21.798 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.7640999999999999e-05 2024-09-13T11:26:21.798 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.798 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.798 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:26:21.798 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:27.988889+0000", 2024-09-13T11:26:21.798 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00023706500000000001 2024-09-13T11:26:21.798 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.798 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.798 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:26:21.798 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:27.988933+0000", 2024-09-13T11:26:21.798 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.3865000000000001e-05 2024-09-13T11:26:21.798 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.799 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.799 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:26:21.799 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:27.991015+0000", 2024-09-13T11:26:21.799 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.002081552 2024-09-13T11:26:21.799 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:21.799 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:26:21.799 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:21.799 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.799 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.799 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPush(3.2b 124/119 [PushOp(3:d70bd127:::benchmark_data_smithi177_31116_object32365:head, version: 60'2034, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:d70bd127:::benchmark_data_smithi177_31116_object32365:head@60'2034, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:26:21.799 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:32.037338+0000", 2024-09-13T11:26:21.799 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 109.709075236, 2024-09-13T11:26:21.799 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.24358922699999999, 2024-09-13T11:26:21.799 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:26:21.800 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:26:21.800 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:26:21.800 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:26:21.800 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.800 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:26:21.800 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:32.037338+0000", 2024-09-13T11:26:21.800 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:21.800 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.800 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.800 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:26:21.800 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:32.037338+0000", 2024-09-13T11:26:21.800 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:21.800 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.800 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.801 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:26:21.801 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:32.037340+0000", 2024-09-13T11:26:21.801 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3790000000000001e-06 2024-09-13T11:26:21.801 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.801 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.801 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:26:21.801 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:32.037411+0000", 2024-09-13T11:26:21.801 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.1254000000000006e-05 2024-09-13T11:26:21.801 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.801 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.801 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:26:21.801 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:32.277467+0000", 2024-09-13T11:26:21.801 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.24005563199999999 2024-09-13T11:26:21.801 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.802 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.802 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:26:21.802 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:32.277494+0000", 2024-09-13T11:26:21.802 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.7120000000000001e-05 2024-09-13T11:26:21.802 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.802 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.802 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:26:21.802 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:32.279860+0000", 2024-09-13T11:26:21.802 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0023667559999999998 2024-09-13T11:26:21.802 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.802 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.802 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:26:21.802 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:32.279883+0000", 2024-09-13T11:26:21.802 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.2138999999999999e-05 2024-09-13T11:26:21.802 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.803 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.803 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:26:21.803 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:32.280928+0000", 2024-09-13T11:26:21.803 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0010449470000000001 2024-09-13T11:26:21.803 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:21.803 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:26:21.803 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:21.803 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.803 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.803 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPush(3.2b 125/119 [PushOp(3:d4c5ede9:::benchmark_data_smithi177_31116_object33277:head, version: 60'2092, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:d4c5ede9:::benchmark_data_smithi177_31116_object33277:head@60'2092, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:26:21.803 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:32.403929+0000", 2024-09-13T11:26:21.803 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 109.342484786, 2024-09-13T11:26:21.803 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.194592864, 2024-09-13T11:26:21.804 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:26:21.804 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:26:21.804 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:26:21.804 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:26:21.804 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.804 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:26:21.804 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:32.403929+0000", 2024-09-13T11:26:21.804 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:21.804 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.804 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.804 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:26:21.804 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:32.403929+0000", 2024-09-13T11:26:21.804 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:21.804 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.805 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.805 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:26:21.805 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:32.403930+0000", 2024-09-13T11:26:21.805 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.6729999999999999e-06 2024-09-13T11:26:21.805 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.805 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.805 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:26:21.805 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:32.403979+0000", 2024-09-13T11:26:21.805 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.8242000000000003e-05 2024-09-13T11:26:21.805 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.805 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.805 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:26:21.805 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:32.597445+0000", 2024-09-13T11:26:21.805 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.19346676500000001 2024-09-13T11:26:21.805 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.806 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.806 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:26:21.806 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:32.597455+0000", 2024-09-13T11:26:21.806 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.2939999999999994e-06 2024-09-13T11:26:21.806 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.806 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.806 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:26:21.806 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:32.597514+0000", 2024-09-13T11:26:21.806 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.9509000000000002e-05 2024-09-13T11:26:21.806 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.806 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.806 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:26:21.806 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:32.597522+0000", 2024-09-13T11:26:21.806 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.0350000000000001e-06 2024-09-13T11:26:21.807 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.807 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.807 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:26:21.807 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:32.598522+0000", 2024-09-13T11:26:21.807 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00099934599999999991 2024-09-13T11:26:21.807 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:21.807 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:26:21.807 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:21.807 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.807 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.807 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPush(3.2b 128/119 [PushOp(3:d6ff590f:::benchmark_data_smithi177_31116_object64581:head, version: 90'3750, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:d6ff590f:::benchmark_data_smithi177_31116_object64581:head@90'3750, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:26:21.807 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:37.982062+0000", 2024-09-13T11:26:21.807 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 103.764351887, 2024-09-13T11:26:21.807 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.28899675200000002, 2024-09-13T11:26:21.808 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:26:21.808 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:26:21.808 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:26:21.808 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:26:21.808 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.808 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:26:21.808 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:37.982062+0000", 2024-09-13T11:26:21.808 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:21.808 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.808 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.808 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:26:21.808 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:37.982062+0000", 2024-09-13T11:26:21.808 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:21.808 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.809 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.809 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:26:21.809 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:37.982064+0000", 2024-09-13T11:26:21.809 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.2259999999999999e-06 2024-09-13T11:26:21.809 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.809 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.809 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:26:21.809 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:37.982114+0000", 2024-09-13T11:26:21.809 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.0330000000000001e-05 2024-09-13T11:26:21.809 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.809 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.809 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:26:21.809 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:38.269983+0000", 2024-09-13T11:26:21.809 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.28786869999999998 2024-09-13T11:26:21.809 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.810 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.810 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:26:21.810 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:38.270000+0000", 2024-09-13T11:26:21.810 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.6977999999999999e-05 2024-09-13T11:26:21.810 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.810 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.810 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:26:21.810 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:38.270092+0000", 2024-09-13T11:26:21.810 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.221e-05 2024-09-13T11:26:21.810 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.810 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.810 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:26:21.810 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:38.270103+0000", 2024-09-13T11:26:21.810 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.1253999999999999e-05 2024-09-13T11:26:21.811 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.811 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.811 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:26:21.811 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:38.271058+0000", 2024-09-13T11:26:21.811 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00095505399999999999 2024-09-13T11:26:21.811 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:21.811 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:26:21.811 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:21.811 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.811 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.811 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:150366 3.2c e149/144 3:348937c7:::benchmark_data_smithi177_31116_object10063:head v 149'5822, mlcod=149'5819)", 2024-09-13T11:26:21.811 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:25:06.284607+0000", 2024-09-13T11:26:21.811 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 75.461806597000006, 2024-09-13T11:26:21.811 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.16600746599999999, 2024-09-13T11:26:21.811 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:26:21.812 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:26:21.812 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:21.812 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:26:21.812 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.812 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:26:21.812 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:06.284607+0000", 2024-09-13T11:26:21.812 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:21.812 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.812 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.812 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:26:21.812 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:06.284607+0000", 2024-09-13T11:26:21.812 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:21.812 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.812 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.813 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:26:21.813 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:06.284608+0000", 2024-09-13T11:26:21.813 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.018e-06 2024-09-13T11:26:21.813 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.813 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.813 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:26:21.813 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:06.284612+0000", 2024-09-13T11:26:21.813 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.3239999999999996e-06 2024-09-13T11:26:21.813 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.813 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.813 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:26:21.813 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:06.449402+0000", 2024-09-13T11:26:21.813 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.16478921099999999 2024-09-13T11:26:21.813 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.814 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.814 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:26:21.814 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:06.449408+0000", 2024-09-13T11:26:21.814 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.1990000000000003e-06 2024-09-13T11:26:21.814 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.814 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.814 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:26:21.814 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:06.449467+0000", 2024-09-13T11:26:21.814 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.8792999999999998e-05 2024-09-13T11:26:21.814 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.814 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.814 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:26:21.814 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:06.449490+0000", 2024-09-13T11:26:21.814 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.3895000000000001e-05 2024-09-13T11:26:21.814 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.815 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.815 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:26:21.815 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:06.450581+0000", 2024-09-13T11:26:21.815 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001090653 2024-09-13T11:26:21.815 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.815 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.815 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:26:21.815 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:06.450614+0000", 2024-09-13T11:26:21.815 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.3373e-05 2024-09-13T11:26:21.815 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:21.815 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:26:21.815 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:21.815 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.815 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.816 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:183827 3.2b e171/119 3:d63d3986:::benchmark_data_smithi177_31116_object43524:head v 171'6235, mlcod=171'6232)", 2024-09-13T11:26:21.816 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:25:32.487782+0000", 2024-09-13T11:26:21.816 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 49.258631674, 2024-09-13T11:26:21.816 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.21250508400000001, 2024-09-13T11:26:21.816 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:26:21.816 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:26:21.816 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:21.816 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:26:21.816 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.816 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:26:21.816 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:32.487782+0000", 2024-09-13T11:26:21.816 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:21.816 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.816 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.817 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:26:21.817 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:32.487782+0000", 2024-09-13T11:26:21.817 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:21.817 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.817 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.817 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:26:21.817 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:32.487783+0000", 2024-09-13T11:26:21.817 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.9699999999999994e-07 2024-09-13T11:26:21.817 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.817 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.817 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:26:21.817 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:32.487787+0000", 2024-09-13T11:26:21.817 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.5689999999999998e-06 2024-09-13T11:26:21.817 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.817 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.818 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:26:21.818 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:32.699019+0000", 2024-09-13T11:26:21.818 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.21123154499999999 2024-09-13T11:26:21.818 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.818 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.818 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:26:21.818 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:32.699024+0000", 2024-09-13T11:26:21.818 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.6229999999999996e-06 2024-09-13T11:26:21.818 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.818 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.819 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:26:21.819 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:32.699068+0000", 2024-09-13T11:26:21.819 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.4722000000000003e-05 2024-09-13T11:26:21.819 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.819 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.819 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:26:21.819 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:32.699084+0000", 2024-09-13T11:26:21.819 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5282999999999999e-05 2024-09-13T11:26:21.819 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.819 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.819 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:26:21.819 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:32.700246+0000", 2024-09-13T11:26:21.819 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001162735 2024-09-13T11:26:21.819 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.820 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.820 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:26:21.820 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:32.700287+0000", 2024-09-13T11:26:21.820 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.0609999999999999e-05 2024-09-13T11:26:21.820 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:21.820 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:26:21.820 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:21.820 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.820 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.820 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:191665 3.a e176/117 3:53060b28:::benchmark_data_smithi177_31116_object51362:head v 176'6424, mlcod=176'6422)", 2024-09-13T11:26:21.820 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:25:39.657701+0000", 2024-09-13T11:26:21.821 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 42.08871216, 2024-09-13T11:26:21.821 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.25090320900000002, 2024-09-13T11:26:21.821 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:26:21.821 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:26:21.821 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:21.821 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:26:21.821 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.821 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:26:21.821 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:39.657701+0000", 2024-09-13T11:26:21.821 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:21.821 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.821 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.821 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:26:21.821 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:39.657701+0000", 2024-09-13T11:26:21.822 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:21.822 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.822 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.822 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:26:21.822 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:39.657704+0000", 2024-09-13T11:26:21.822 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.7650000000000002e-06 2024-09-13T11:26:21.822 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.822 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.822 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:26:21.822 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:39.657714+0000", 2024-09-13T11:26:21.822 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.5719999999999992e-06 2024-09-13T11:26:21.822 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.822 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.822 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:26:21.823 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:39.904490+0000", 2024-09-13T11:26:21.823 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.246775831 2024-09-13T11:26:21.823 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.823 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.823 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:26:21.823 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:39.904503+0000", 2024-09-13T11:26:21.823 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3723000000000001e-05 2024-09-13T11:26:21.823 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.823 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.823 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:26:21.823 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:39.904660+0000", 2024-09-13T11:26:21.823 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00015705499999999999 2024-09-13T11:26:21.823 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.823 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.823 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:26:21.824 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:39.904699+0000", 2024-09-13T11:26:21.824 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.8877000000000001e-05 2024-09-13T11:26:21.824 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.824 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.824 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:26:21.824 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:39.908544+0000", 2024-09-13T11:26:21.824 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0038448739999999999 2024-09-13T11:26:21.824 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.824 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.824 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:26:21.824 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:39.908605+0000", 2024-09-13T11:26:21.824 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.0511999999999999e-05 2024-09-13T11:26:21.824 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:21.824 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:26:21.825 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:21.825 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.825 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.825 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:196349 3.d 3:b2d956a9:::benchmark_data_smithi177_31116_object56046:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e178)", 2024-09-13T11:26:21.825 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:25:43.404118+0000", 2024-09-13T11:26:21.825 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 38.342296066999999, 2024-09-13T11:26:21.825 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.17729215100000001, 2024-09-13T11:26:21.825 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:26:21.825 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:26:21.825 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:21.825 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:26:21.825 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:26:21.825 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:21.825 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 196349 2024-09-13T11:26:21.825 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.826 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:26:21.826 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.826 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:26:21.826 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:43.404118+0000", 2024-09-13T11:26:21.826 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:21.826 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.826 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.826 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:26:21.826 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:43.404118+0000", 2024-09-13T11:26:21.826 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:21.826 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.826 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.826 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:26:21.826 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:43.404121+0000", 2024-09-13T11:26:21.827 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.743e-06 2024-09-13T11:26:21.827 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.827 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.827 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:26:21.827 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:43.404132+0000", 2024-09-13T11:26:21.827 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0433e-05 2024-09-13T11:26:21.827 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.827 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.827 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:26:21.827 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:43.404159+0000", 2024-09-13T11:26:21.827 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.7064e-05 2024-09-13T11:26:21.827 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.827 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.827 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:26:21.827 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:43.404172+0000", 2024-09-13T11:26:21.828 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3606e-05 2024-09-13T11:26:21.828 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.828 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.828 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:26:21.828 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:43.404312+0000", 2024-09-13T11:26:21.828 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000139262 2024-09-13T11:26:21.828 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.828 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.828 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:26:21.828 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:43.404531+0000", 2024-09-13T11:26:21.828 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00021986 2024-09-13T11:26:21.828 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.828 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.828 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 3", 2024-09-13T11:26:21.829 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:43.404706+0000", 2024-09-13T11:26:21.829 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000174358 2024-09-13T11:26:21.829 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.829 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.829 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:26:21.829 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:43.408021+0000", 2024-09-13T11:26:21.829 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0033152939999999999 2024-09-13T11:26:21.829 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.829 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.829 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:21.829 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:43.581239+0000", 2024-09-13T11:26:21.829 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.17321809699999999 2024-09-13T11:26:21.829 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.829 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.829 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:26:21.830 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:43.581331+0000", 2024-09-13T11:26:21.830 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.1877000000000001e-05 2024-09-13T11:26:21.830 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.830 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.830 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:26:21.830 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:43.581410+0000", 2024-09-13T11:26:21.830 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.8557e-05 2024-09-13T11:26:21.830 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:21.830 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:26:21.830 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:21.830 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.830 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.830 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:227427 3.a e196/117 3:52db129e:::benchmark_data_smithi177_31116_object87124:head v 196'6993, mlcod=196'6991)", 2024-09-13T11:26:21.830 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:26:11.046359+0000", 2024-09-13T11:26:21.831 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 10.700054594999999, 2024-09-13T11:26:21.831 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.19900411000000001, 2024-09-13T11:26:21.831 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:26:21.831 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:26:21.831 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:21.831 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:26:21.831 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.831 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:26:21.831 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:11.046359+0000", 2024-09-13T11:26:21.831 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:21.831 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.831 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.831 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:26:21.831 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:11.046359+0000", 2024-09-13T11:26:21.831 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:21.832 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.832 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.832 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:26:21.832 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:11.046360+0000", 2024-09-13T11:26:21.832 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.215e-06 2024-09-13T11:26:21.832 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.832 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.832 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:26:21.832 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:11.046365+0000", 2024-09-13T11:26:21.832 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.7060000000000003e-06 2024-09-13T11:26:21.832 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.832 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.832 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:26:21.832 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:11.239636+0000", 2024-09-13T11:26:21.833 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.193270687 2024-09-13T11:26:21.833 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.833 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.833 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:26:21.833 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:11.239651+0000", 2024-09-13T11:26:21.833 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5067e-05 2024-09-13T11:26:21.833 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.833 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.833 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:26:21.833 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:11.239970+0000", 2024-09-13T11:26:21.833 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00031894300000000003 2024-09-13T11:26:21.833 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.833 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.833 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:26:21.833 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:11.240027+0000", 2024-09-13T11:26:21.834 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.7419e-05 2024-09-13T11:26:21.834 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.834 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.834 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:26:21.834 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:11.243833+0000", 2024-09-13T11:26:21.834 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0038059629999999999 2024-09-13T11:26:21.834 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.834 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.834 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:26:21.834 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:11.245363+0000", 2024-09-13T11:26:21.834 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0015301100000000001 2024-09-13T11:26:21.834 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:21.834 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:26:21.834 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:21.834 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:21.835 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:26:21.835 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-09-13T11:26:21.836 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-09-13T11:26:21.837 INFO:tasks.ceph.osd.10.smithi162.stderr:2024-09-13T11:26:21.831+0000 7fb59c9f0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26317) UID: 0 2024-09-13T11:26:21.937 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-09-13T11:26:21.935+0000 7f97befc9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26315) UID: 0 2024-09-13T11:26:21.967 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-09-13T11:26:21.967 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [ 2024-09-13T11:26:21.967 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.967 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:238208 3.16 3:6fea8e50:::benchmark_data_smithi177_31116_object97905:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e204)", 2024-09-13T11:26:21.967 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:26:21.891424+0000", 2024-09-13T11:26:21.967 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 0.076535646999999998, 2024-09-13T11:26:21.968 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.076562923000000005, 2024-09-13T11:26:21.968 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:26:21.968 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:26:21.968 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "waiting for sub ops", 2024-09-13T11:26:21.968 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:26:21.968 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:26:21.968 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:21.968 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 238208 2024-09-13T11:26:21.968 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.968 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:26:21.969 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.969 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:26:21.969 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:21.891424+0000", 2024-09-13T11:26:21.969 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:21.969 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.969 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.969 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:26:21.969 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:21.891424+0000", 2024-09-13T11:26:21.969 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:21.969 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.969 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.969 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:26:21.969 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:21.891427+0000", 2024-09-13T11:26:21.969 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.0730000000000001e-06 2024-09-13T11:26:21.970 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.970 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.970 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:26:21.970 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:21.891436+0000", 2024-09-13T11:26:21.970 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.4639999999999995e-06 2024-09-13T11:26:21.970 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.971 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.971 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:26:21.971 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:21.891457+0000", 2024-09-13T11:26:21.971 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.0225999999999999e-05 2024-09-13T11:26:21.971 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.971 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.971 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:26:21.971 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:21.891468+0000", 2024-09-13T11:26:21.971 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.1457999999999999e-05 2024-09-13T11:26:21.971 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.971 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.971 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:26:21.972 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:21.891533+0000", 2024-09-13T11:26:21.972 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.4443999999999995e-05 2024-09-13T11:26:21.972 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.972 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.972 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:26:21.972 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:21.891646+0000", 2024-09-13T11:26:21.972 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000113097 2024-09-13T11:26:21.972 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.972 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.972 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 15", 2024-09-13T11:26:21.972 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:21.891724+0000", 2024-09-13T11:26:21.972 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.7954000000000006e-05 2024-09-13T11:26:21.972 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.972 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.973 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:26:21.973 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:21.894806+0000", 2024-09-13T11:26:21.973 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0030826109999999999 2024-09-13T11:26:21.973 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:21.973 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:26:21.973 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:21.973 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.973 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.973 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:238221 3.16 3:6bae43a6:::benchmark_data_smithi177_31116_object97918:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e204)", 2024-09-13T11:26:21.973 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:26:21.909248+0000", 2024-09-13T11:26:21.973 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 0.058711251999999998, 2024-09-13T11:26:21.973 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.058817900999999999, 2024-09-13T11:26:21.973 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:26:21.974 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:26:21.974 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "waiting for sub ops", 2024-09-13T11:26:21.974 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:26:21.974 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:26:21.974 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:21.974 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 238221 2024-09-13T11:26:21.974 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.974 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:26:21.974 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.974 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:26:21.974 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:21.909248+0000", 2024-09-13T11:26:21.974 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:21.974 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.974 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.975 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:26:21.975 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:21.909248+0000", 2024-09-13T11:26:21.975 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:21.975 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.975 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.975 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:26:21.975 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:21.909251+0000", 2024-09-13T11:26:21.975 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.5440000000000001e-06 2024-09-13T11:26:21.975 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.975 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.975 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:26:21.975 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:21.909259+0000", 2024-09-13T11:26:21.975 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.8299999999999996e-06 2024-09-13T11:26:21.975 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.976 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.976 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:26:21.976 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:21.909278+0000", 2024-09-13T11:26:21.976 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.9589000000000002e-05 2024-09-13T11:26:21.976 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.976 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.976 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:26:21.976 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:21.909288+0000", 2024-09-13T11:26:21.976 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.6870000000000006e-06 2024-09-13T11:26:21.976 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.976 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.976 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:26:21.976 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:21.909363+0000", 2024-09-13T11:26:21.976 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.5137999999999998e-05 2024-09-13T11:26:21.977 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.977 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.977 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:26:21.977 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:21.909496+0000", 2024-09-13T11:26:21.977 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000133239 2024-09-13T11:26:21.977 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.977 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.977 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 15", 2024-09-13T11:26:21.977 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:21.909605+0000", 2024-09-13T11:26:21.977 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000108727 2024-09-13T11:26:21.977 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.977 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.977 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:26:21.977 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:21.913397+0000", 2024-09-13T11:26:21.978 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0037923380000000001 2024-09-13T11:26:21.978 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:21.978 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:26:21.978 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:21.978 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.978 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.978 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:238264 3.13 3:cad4f538:::benchmark_data_smithi177_31116_object97961:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e204)", 2024-09-13T11:26:21.978 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:26:21.952020+0000", 2024-09-13T11:26:21.978 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 0.015939070999999999, 2024-09-13T11:26:21.978 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.016164914999999998, 2024-09-13T11:26:21.978 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:26:21.978 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:26:21.978 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "waiting for sub ops", 2024-09-13T11:26:21.978 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:26:21.979 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:26:21.979 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:21.979 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 238264 2024-09-13T11:26:21.979 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.979 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:26:21.979 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.979 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:26:21.979 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:21.952020+0000", 2024-09-13T11:26:21.979 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:21.979 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.979 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.979 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:26:21.979 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:21.952020+0000", 2024-09-13T11:26:21.979 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:21.980 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.980 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.980 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:26:21.980 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:21.952023+0000", 2024-09-13T11:26:21.980 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.2400000000000002e-06 2024-09-13T11:26:21.980 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.980 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.980 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:26:21.980 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:21.952030+0000", 2024-09-13T11:26:21.980 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.6559999999999992e-06 2024-09-13T11:26:21.980 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.980 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.980 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:26:21.980 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:21.952047+0000", 2024-09-13T11:26:21.981 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.6251000000000001e-05 2024-09-13T11:26:21.981 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.981 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.981 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:26:21.981 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:21.952057+0000", 2024-09-13T11:26:21.981 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.9839999999999996e-06 2024-09-13T11:26:21.981 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.981 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.981 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:26:21.981 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:21.952134+0000", 2024-09-13T11:26:21.981 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.7207999999999997e-05 2024-09-13T11:26:21.982 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.982 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.982 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:26:21.982 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:21.952266+0000", 2024-09-13T11:26:21.982 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00013211599999999999 2024-09-13T11:26:21.982 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.982 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.982 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 7", 2024-09-13T11:26:21.982 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:21.952367+0000", 2024-09-13T11:26:21.982 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000101531 2024-09-13T11:26:21.982 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:21.982 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:21.982 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:26:21.982 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:21.955909+0000", 2024-09-13T11:26:21.983 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0035415540000000001 2024-09-13T11:26:21.983 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:21.983 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:26:21.983 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:21.983 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:21.983 INFO:teuthology.orchestra.run.smithi177.stdout: ], 2024-09-13T11:26:21.983 INFO:teuthology.orchestra.run.smithi177.stdout: "num_ops": 3 2024-09-13T11:26:21.983 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-09-13T11:26:21.984 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-09-13T11:26:22.038 INFO:tasks.ceph.osd.2.smithi162.stderr:2024-09-13T11:26:22.035+0000 7fe92b35a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26313) UID: 0 2024-09-13T11:26:22.121 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-09-13T11:26:22.121 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [], 2024-09-13T11:26:22.121 INFO:teuthology.orchestra.run.smithi177.stdout: "complaint_time": 30, 2024-09-13T11:26:22.121 INFO:teuthology.orchestra.run.smithi177.stdout: "num_blocked_ops": 0 2024-09-13T11:26:22.121 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-09-13T11:26:22.140 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-09-13T11:26:22.140 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-13T11:26:22.138+0000 7f6a68775640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26307) UID: 0 2024-09-13T11:26:22.149 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 15, 14] out_osds: [] dead_osds: [] live_osds: [0, 4, 8, 12, 5, 9, 13, 2, 6, 10, 3, 7, 11, 15, 1, 14] 2024-09-13T11:26:22.149 INFO:tasks.thrashosds.thrasher:choose_action: min_in 2 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-13T11:26:22.150 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-09-13T11:26:22.150 INFO:tasks.thrashosds.thrasher:Setting osd 15 primary_affinity to 0.000000 2024-09-13T11:26:22.150 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 15 0 2024-09-13T11:26:22.163 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-13T11:26:22.240 INFO:tasks.ceph.osd.11.smithi177.stderr:2024-09-13T11:26:22.237+0000 7f24792c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26580) UID: 0 2024-09-13T11:26:22.277 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-09-13T11:26:22.278 INFO:teuthology.orchestra.run.smithi177.stdout: "size": 20, 2024-09-13T11:26:22.278 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 600, 2024-09-13T11:26:22.278 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [ 2024-09-13T11:26:22.278 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.278 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:79 3.1 e32/30 3:841df5b5:::benchmark_data_smithi177_31116_object78:head v 32'6, mlcod=32'4)", 2024-09-13T11:26:22.278 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:22:27.057292+0000", 2024-09-13T11:26:22.278 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 235.22018318600001, 2024-09-13T11:26:22.278 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.89296786800000005, 2024-09-13T11:26:22.278 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:26:22.278 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:26:22.278 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:22.278 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:26:22.278 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.279 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:26:22.279 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.057292+0000", 2024-09-13T11:26:22.279 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:22.279 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.279 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.279 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:26:22.279 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.057292+0000", 2024-09-13T11:26:22.279 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:22.279 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.279 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.279 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:26:22.279 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.057293+0000", 2024-09-13T11:26:22.280 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.6109999999999999e-06 2024-09-13T11:26:22.280 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.280 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.280 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:26:22.280 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.057330+0000", 2024-09-13T11:26:22.280 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.6721999999999998e-05 2024-09-13T11:26:22.280 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.280 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.280 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:26:22.280 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.948197+0000", 2024-09-13T11:26:22.280 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.890867042 2024-09-13T11:26:22.280 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.280 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.281 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:26:22.281 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.948213+0000", 2024-09-13T11:26:22.281 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5826000000000001e-05 2024-09-13T11:26:22.281 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.281 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.281 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:26:22.281 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.948377+0000", 2024-09-13T11:26:22.281 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00016348600000000001 2024-09-13T11:26:22.281 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.281 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.281 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:26:22.281 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.948447+0000", 2024-09-13T11:26:22.281 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.0891999999999998e-05 2024-09-13T11:26:22.281 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.282 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.282 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:26:22.282 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.950150+0000", 2024-09-13T11:26:22.282 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001702329 2024-09-13T11:26:22.282 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.282 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.282 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:26:22.282 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.950260+0000", 2024-09-13T11:26:22.282 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00010996 2024-09-13T11:26:22.282 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:22.282 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:26:22.282 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:22.283 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.283 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.283 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:45783 3.16 3.17c10a76 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e69)", 2024-09-13T11:26:22.283 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:20.947774+0000", 2024-09-13T11:26:22.283 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 181.32970093500001, 2024-09-13T11:26:22.283 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.96682922400000004, 2024-09-13T11:26:22.283 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:26:22.283 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:26:22.283 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-13T11:26:22.283 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:26:22.283 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:26:22.283 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:22.283 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 45783 2024-09-13T11:26:22.283 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.284 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:26:22.284 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.284 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:26:22.284 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:20.947774+0000", 2024-09-13T11:26:22.284 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:22.284 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.284 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.284 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:26:22.284 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:20.947774+0000", 2024-09-13T11:26:22.284 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:22.284 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.284 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.284 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:26:22.285 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:20.947777+0000", 2024-09-13T11:26:22.285 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.57e-06 2024-09-13T11:26:22.285 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.285 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.285 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:26:22.285 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:20.947802+0000", 2024-09-13T11:26:22.285 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.5528999999999999e-05 2024-09-13T11:26:22.285 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.285 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.285 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:26:22.285 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:20.947813+0000", 2024-09-13T11:26:22.285 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0473e-05 2024-09-13T11:26:22.285 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.286 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.286 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:26:22.286 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:20.947821+0000", 2024-09-13T11:26:22.286 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.3440000000000001e-06 2024-09-13T11:26:22.286 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.286 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.286 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:26:22.286 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:20.947933+0000", 2024-09-13T11:26:22.286 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00011210200000000001 2024-09-13T11:26:22.286 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.286 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.286 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for peered", 2024-09-13T11:26:22.287 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:20.947942+0000", 2024-09-13T11:26:22.287 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.6610000000000006e-06 2024-09-13T11:26:22.287 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.287 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.287 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:26:22.287 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:21.914579+0000", 2024-09-13T11:26:22.287 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.96663672300000003 2024-09-13T11:26:22.287 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.287 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.287 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:26:22.287 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:21.914603+0000", 2024-09-13T11:26:22.287 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.4822000000000001e-05 2024-09-13T11:26:22.287 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:22.287 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:26:22.288 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:22.288 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.288 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.288 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:54625 3.1f 3:f812cdfb:::benchmark_data_smithi177_31116_object54624:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e75)", 2024-09-13T11:26:22.288 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:30.979327+0000", 2024-09-13T11:26:22.288 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 171.29814814599999, 2024-09-13T11:26:22.288 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.96163670899999998, 2024-09-13T11:26:22.288 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:26:22.288 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:26:22.288 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-13T11:26:22.288 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:26:22.288 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:26:22.288 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:22.288 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 54625 2024-09-13T11:26:22.289 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.289 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:26:22.289 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.289 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:26:22.289 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.979327+0000", 2024-09-13T11:26:22.289 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:22.289 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.289 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.289 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:26:22.289 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.979327+0000", 2024-09-13T11:26:22.289 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:22.289 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.289 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.289 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:26:22.290 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.979329+0000", 2024-09-13T11:26:22.290 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.686e-06 2024-09-13T11:26:22.290 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.290 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.290 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:26:22.290 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.979364+0000", 2024-09-13T11:26:22.290 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.5750000000000002e-05 2024-09-13T11:26:22.290 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.290 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.290 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:26:22.290 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.979408+0000", 2024-09-13T11:26:22.290 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.3795999999999999e-05 2024-09-13T11:26:22.290 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.290 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.290 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:26:22.291 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.979427+0000", 2024-09-13T11:26:22.291 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.9040999999999999e-05 2024-09-13T11:26:22.291 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.291 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.291 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:26:22.291 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.122291+0000", 2024-09-13T11:26:22.291 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.14286373599999999 2024-09-13T11:26:22.291 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.291 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.291 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for peered", 2024-09-13T11:26:22.291 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.122298+0000", 2024-09-13T11:26:22.291 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.3719999999999999e-06 2024-09-13T11:26:22.291 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.291 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.292 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:26:22.292 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.940341+0000", 2024-09-13T11:26:22.292 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.81804249699999998 2024-09-13T11:26:22.292 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.292 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.292 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:26:22.292 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.940964+0000", 2024-09-13T11:26:22.292 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00062283100000000001 2024-09-13T11:26:22.292 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:22.292 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:26:22.292 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:22.292 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.292 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.293 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:54649 3.1c 3:3a457a93:::benchmark_data_smithi177_31116_object54648:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e75)", 2024-09-13T11:26:22.293 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:30.979460+0000", 2024-09-13T11:26:22.293 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 171.298014956, 2024-09-13T11:26:22.293 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.95579424000000002, 2024-09-13T11:26:22.293 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:26:22.293 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:26:22.293 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-13T11:26:22.293 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:26:22.293 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:26:22.293 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:22.293 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 54649 2024-09-13T11:26:22.293 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.293 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:26:22.293 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.294 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:26:22.294 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.979460+0000", 2024-09-13T11:26:22.294 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:22.294 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.294 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.294 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:26:22.294 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.979460+0000", 2024-09-13T11:26:22.294 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:22.294 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.294 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.294 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:26:22.294 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.979462+0000", 2024-09-13T11:26:22.294 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5799999999999999e-06 2024-09-13T11:26:22.294 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.295 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.295 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:26:22.295 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.979524+0000", 2024-09-13T11:26:22.295 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.2521999999999995e-05 2024-09-13T11:26:22.295 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.295 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.295 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:26:22.295 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.979561+0000", 2024-09-13T11:26:22.295 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.6349e-05 2024-09-13T11:26:22.295 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.295 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.295 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:26:22.295 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.979571+0000", 2024-09-13T11:26:22.295 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0457e-05 2024-09-13T11:26:22.296 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.296 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.296 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:26:22.296 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.121029+0000", 2024-09-13T11:26:22.296 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.14145766200000001 2024-09-13T11:26:22.296 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.296 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.296 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for peered", 2024-09-13T11:26:22.296 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.121039+0000", 2024-09-13T11:26:22.296 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0084e-05 2024-09-13T11:26:22.296 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.296 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.296 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:26:22.296 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.935157+0000", 2024-09-13T11:26:22.297 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.81411807899999999 2024-09-13T11:26:22.297 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.297 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.297 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:26:22.297 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.935254+0000", 2024-09-13T11:26:22.297 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.7507e-05 2024-09-13T11:26:22.297 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:22.297 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:26:22.297 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:22.297 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.297 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.297 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:54654 3.1c 3:3f0bb31a:::benchmark_data_smithi177_31116_object54653:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e75)", 2024-09-13T11:26:22.297 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:30.979665+0000", 2024-09-13T11:26:22.297 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 171.29780979099999, 2024-09-13T11:26:22.298 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.95623908099999999, 2024-09-13T11:26:22.298 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:26:22.298 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:26:22.298 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-13T11:26:22.298 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:26:22.298 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:26:22.298 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:22.298 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 54654 2024-09-13T11:26:22.298 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.298 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:26:22.298 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.298 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:26:22.298 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.979665+0000", 2024-09-13T11:26:22.298 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:22.298 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.299 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.299 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:26:22.299 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.979665+0000", 2024-09-13T11:26:22.299 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:22.299 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.299 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.299 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:26:22.299 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.979667+0000", 2024-09-13T11:26:22.299 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3540000000000001e-06 2024-09-13T11:26:22.299 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.299 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.299 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:26:22.299 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.979703+0000", 2024-09-13T11:26:22.299 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.6758000000000003e-05 2024-09-13T11:26:22.300 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.300 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.300 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:26:22.300 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.979714+0000", 2024-09-13T11:26:22.300 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0804e-05 2024-09-13T11:26:22.300 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.300 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.300 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:26:22.300 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.979724+0000", 2024-09-13T11:26:22.300 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0040999999999999e-05 2024-09-13T11:26:22.300 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.300 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.300 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:26:22.300 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.121061+0000", 2024-09-13T11:26:22.301 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.141336713 2024-09-13T11:26:22.301 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.301 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.301 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for peered", 2024-09-13T11:26:22.301 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.121067+0000", 2024-09-13T11:26:22.301 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.4150000000000004e-06 2024-09-13T11:26:22.301 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.301 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.301 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:26:22.301 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.935776+0000", 2024-09-13T11:26:22.301 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.81470863100000002 2024-09-13T11:26:22.301 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.301 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.302 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:26:22.302 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.935904+0000", 2024-09-13T11:26:22.302 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00012836499999999999 2024-09-13T11:26:22.302 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:22.302 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:26:22.302 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:22.302 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.302 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.302 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPushReply(3.11 75/70 [PushReplyOp(3:8df10771:::benchmark_data_smithi177_31116_object24734:head)])", 2024-09-13T11:26:22.302 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:31.149798+0000", 2024-09-13T11:26:22.302 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 171.12767678099999, 2024-09-13T11:26:22.302 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.32402702500000002, 2024-09-13T11:26:22.302 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:26:22.302 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:26:22.302 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-13T11:26:22.303 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:26:22.303 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.303 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:26:22.303 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.149798+0000", 2024-09-13T11:26:22.303 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:22.303 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.303 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.303 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:26:22.303 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.149798+0000", 2024-09-13T11:26:22.303 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:22.303 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.303 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.303 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:26:22.303 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.149799+0000", 2024-09-13T11:26:22.304 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.5700000000000001e-07 2024-09-13T11:26:22.304 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.304 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.304 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:26:22.304 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.149803+0000", 2024-09-13T11:26:22.304 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.0040000000000001e-06 2024-09-13T11:26:22.304 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.304 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.304 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:26:22.304 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.472172+0000", 2024-09-13T11:26:22.304 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.32236856699999999 2024-09-13T11:26:22.304 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.304 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.304 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:26:22.304 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.472187+0000", 2024-09-13T11:26:22.305 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5016e-05 2024-09-13T11:26:22.305 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.305 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.305 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:26:22.305 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.472302+0000", 2024-09-13T11:26:22.305 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000115121 2024-09-13T11:26:22.305 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.305 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.305 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:26:22.305 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.473825+0000", 2024-09-13T11:26:22.305 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0015234599999999999 2024-09-13T11:26:22.305 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:22.305 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:26:22.305 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:22.306 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.306 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.306 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:54668 3.1c 3:3d2c5e40:::benchmark_data_smithi177_31116_object54667:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e75)", 2024-09-13T11:26:22.306 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:31.245525+0000", 2024-09-13T11:26:22.306 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 171.031950524, 2024-09-13T11:26:22.306 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.69828973000000005, 2024-09-13T11:26:22.306 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:26:22.306 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:26:22.306 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:22.306 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:26:22.306 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:26:22.306 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:22.306 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 54668 2024-09-13T11:26:22.306 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.307 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:26:22.307 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.307 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:26:22.307 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.245525+0000", 2024-09-13T11:26:22.307 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:22.307 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.307 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.307 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:26:22.307 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.245525+0000", 2024-09-13T11:26:22.307 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:22.307 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.307 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.307 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:26:22.307 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.245526+0000", 2024-09-13T11:26:22.308 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.719e-06 2024-09-13T11:26:22.308 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.308 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.308 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:26:22.308 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.245547+0000", 2024-09-13T11:26:22.308 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.1084999999999999e-05 2024-09-13T11:26:22.308 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.308 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.308 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:26:22.308 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.245558+0000", 2024-09-13T11:26:22.308 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0934000000000001e-05 2024-09-13T11:26:22.308 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.308 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.308 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:26:22.309 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.245564+0000", 2024-09-13T11:26:22.309 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.6269999999999997e-06 2024-09-13T11:26:22.309 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.309 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.309 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:26:22.309 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.245620+0000", 2024-09-13T11:26:22.309 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.6289999999999998e-05 2024-09-13T11:26:22.309 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.309 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.309 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for peered", 2024-09-13T11:26:22.309 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.245632+0000", 2024-09-13T11:26:22.309 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.1748e-05 2024-09-13T11:26:22.309 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.309 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.310 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:26:22.310 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.935976+0000", 2024-09-13T11:26:22.310 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.69034455299999997 2024-09-13T11:26:22.310 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.310 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.310 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:26:22.310 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.936302+0000", 2024-09-13T11:26:22.310 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000325397 2024-09-13T11:26:22.310 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.310 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.310 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 13", 2024-09-13T11:26:22.311 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.936459+0000", 2024-09-13T11:26:22.311 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00015754300000000001 2024-09-13T11:26:22.311 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.311 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.311 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:26:22.311 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.938813+0000", 2024-09-13T11:26:22.311 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0023530999999999999 2024-09-13T11:26:22.311 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.311 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.311 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:22.311 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.943695+0000", 2024-09-13T11:26:22.311 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.004882506 2024-09-13T11:26:22.311 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.311 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.312 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:26:22.312 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.943767+0000", 2024-09-13T11:26:22.312 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.2454999999999997e-05 2024-09-13T11:26:22.312 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.312 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.312 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:26:22.312 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.943814+0000", 2024-09-13T11:26:22.312 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.6773000000000001e-05 2024-09-13T11:26:22.312 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:22.312 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:26:22.312 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:22.312 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.312 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.313 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPush(3.16 84/70 [PushOp(3:6f7be541:::benchmark_data_smithi177_31116_object43332:head, version: 66'2788, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:6f7be541:::benchmark_data_smithi177_31116_object43332:head@66'2788, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:26:22.313 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:41.207515+0000", 2024-09-13T11:26:22.313 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 161.06995975300001, 2024-09-13T11:26:22.313 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.52643771500000003, 2024-09-13T11:26:22.313 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:26:22.313 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:26:22.313 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:26:22.313 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:26:22.313 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.313 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:26:22.313 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.207515+0000", 2024-09-13T11:26:22.313 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:22.313 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.313 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.314 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:26:22.314 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.207515+0000", 2024-09-13T11:26:22.314 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:22.314 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.314 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.314 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:26:22.314 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.207516+0000", 2024-09-13T11:26:22.314 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.096e-06 2024-09-13T11:26:22.314 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.314 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.314 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:26:22.314 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.207717+0000", 2024-09-13T11:26:22.314 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000200199 2024-09-13T11:26:22.314 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.314 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.315 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:26:22.315 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.732785+0000", 2024-09-13T11:26:22.315 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.52506869599999995 2024-09-13T11:26:22.315 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.315 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.315 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:26:22.315 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.732801+0000", 2024-09-13T11:26:22.315 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5305999999999999e-05 2024-09-13T11:26:22.315 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.315 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.315 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:26:22.315 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.732889+0000", 2024-09-13T11:26:22.316 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.8157000000000003e-05 2024-09-13T11:26:22.316 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.316 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.316 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:26:22.316 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.732901+0000", 2024-09-13T11:26:22.316 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.1866e-05 2024-09-13T11:26:22.316 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.316 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.316 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:26:22.316 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.733953+0000", 2024-09-13T11:26:22.316 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001052395 2024-09-13T11:26:22.316 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:22.317 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:26:22.317 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:22.317 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.317 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.317 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPushReply(3.1f 85/78 [PushReplyOp(3:fd221d21:::benchmark_data_smithi177_31116_object7639:head)])", 2024-09-13T11:26:22.317 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:42.198715+0000", 2024-09-13T11:26:22.317 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 160.078760184, 2024-09-13T11:26:22.317 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.48930391099999998, 2024-09-13T11:26:22.317 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:26:22.317 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:26:22.317 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-13T11:26:22.317 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:26:22.317 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.317 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:26:22.318 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.198715+0000", 2024-09-13T11:26:22.318 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:22.318 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.318 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.318 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:26:22.318 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.198715+0000", 2024-09-13T11:26:22.318 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:22.318 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.318 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.318 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:26:22.318 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.198716+0000", 2024-09-13T11:26:22.318 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.3500000000000005e-07 2024-09-13T11:26:22.318 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.318 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.318 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:26:22.319 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.198719+0000", 2024-09-13T11:26:22.319 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.4539999999999999e-06 2024-09-13T11:26:22.319 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.319 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.319 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:26:22.319 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.684732+0000", 2024-09-13T11:26:22.319 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.48601262099999998 2024-09-13T11:26:22.319 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.319 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.319 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:26:22.319 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.684746+0000", 2024-09-13T11:26:22.319 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.4518999999999999e-05 2024-09-13T11:26:22.319 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.320 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.320 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:26:22.320 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.684911+0000", 2024-09-13T11:26:22.320 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000164728 2024-09-13T11:26:22.320 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.320 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.320 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:26:22.320 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.688019+0000", 2024-09-13T11:26:22.320 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0031077539999999999 2024-09-13T11:26:22.320 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:22.320 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:26:22.320 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:22.320 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.320 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.321 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPushReply(3.1c 85/77 [PushReplyOp(3:3a669437:::benchmark_data_smithi177_31116_object25280:head)])", 2024-09-13T11:26:22.321 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:42.352466+0000", 2024-09-13T11:26:22.321 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 159.92500928999999, 2024-09-13T11:26:22.321 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.36670725300000001, 2024-09-13T11:26:22.321 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:26:22.321 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:26:22.321 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-13T11:26:22.321 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:26:22.321 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.321 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:26:22.321 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.352466+0000", 2024-09-13T11:26:22.321 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:22.321 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.321 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.322 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:26:22.322 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.352466+0000", 2024-09-13T11:26:22.322 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:22.322 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.322 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.322 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:26:22.322 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.352467+0000", 2024-09-13T11:26:22.322 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.1429999999999999e-06 2024-09-13T11:26:22.322 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.322 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.322 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:26:22.322 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.352474+0000", 2024-09-13T11:26:22.322 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.6100000000000002e-06 2024-09-13T11:26:22.322 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.323 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.323 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:26:22.323 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.717011+0000", 2024-09-13T11:26:22.323 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.36453790200000002 2024-09-13T11:26:22.323 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.323 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.323 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:26:22.323 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.717032+0000", 2024-09-13T11:26:22.323 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.0999e-05 2024-09-13T11:26:22.323 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.323 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.323 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:26:22.323 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.717162+0000", 2024-09-13T11:26:22.323 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000129159 2024-09-13T11:26:22.324 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.324 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.324 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:26:22.324 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.719173+0000", 2024-09-13T11:26:22.324 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0020114400000000002 2024-09-13T11:26:22.324 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:22.324 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:26:22.324 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:22.324 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.324 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.324 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPushReply(3.1f 85/78 [PushReplyOp(3:fecb0d6a:::benchmark_data_smithi177_31116_object7644:head)])", 2024-09-13T11:26:22.324 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:42.705351+0000", 2024-09-13T11:26:22.324 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 159.57212444800001, 2024-09-13T11:26:22.324 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.290549681, 2024-09-13T11:26:22.325 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:26:22.325 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:26:22.325 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-13T11:26:22.325 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:26:22.325 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.325 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:26:22.325 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.705351+0000", 2024-09-13T11:26:22.325 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:22.325 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.325 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.325 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:26:22.325 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.705351+0000", 2024-09-13T11:26:22.325 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:22.325 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.326 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.326 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:26:22.326 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.705353+0000", 2024-09-13T11:26:22.326 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.4169999999999999e-06 2024-09-13T11:26:22.326 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.326 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.326 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:26:22.326 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.705365+0000", 2024-09-13T11:26:22.326 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.1894e-05 2024-09-13T11:26:22.326 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.326 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.326 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:26:22.326 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.992081+0000", 2024-09-13T11:26:22.326 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.28671641199999998 2024-09-13T11:26:22.326 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.327 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.327 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:26:22.327 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.992111+0000", 2024-09-13T11:26:22.327 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.9427000000000001e-05 2024-09-13T11:26:22.327 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.327 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.327 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:26:22.327 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.993573+0000", 2024-09-13T11:26:22.327 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0014623590000000001 2024-09-13T11:26:22.327 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.327 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.327 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:26:22.327 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.995900+0000", 2024-09-13T11:26:22.327 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0023271720000000002 2024-09-13T11:26:22.328 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:22.328 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:26:22.328 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:22.328 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.328 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.328 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPushReply(3.21 89/78 [PushReplyOp(3:8534d087:::benchmark_data_smithi177_31116_object5474:head)])", 2024-09-13T11:26:22.328 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:46.632834+0000", 2024-09-13T11:26:22.328 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 155.64464100800001, 2024-09-13T11:26:22.328 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.324805556, 2024-09-13T11:26:22.328 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:26:22.328 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:26:22.328 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-13T11:26:22.328 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:26:22.328 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.329 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:26:22.329 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.632834+0000", 2024-09-13T11:26:22.329 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:22.329 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.329 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.329 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:26:22.329 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.632834+0000", 2024-09-13T11:26:22.329 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:22.329 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.329 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.329 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:26:22.329 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.632835+0000", 2024-09-13T11:26:22.329 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.4270000000000001e-06 2024-09-13T11:26:22.329 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.330 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.330 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:26:22.330 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.632840+0000", 2024-09-13T11:26:22.330 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.7809999999999996e-06 2024-09-13T11:26:22.330 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.330 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.330 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:26:22.330 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.956282+0000", 2024-09-13T11:26:22.330 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.323441639 2024-09-13T11:26:22.330 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.330 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.330 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:26:22.330 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.956293+0000", 2024-09-13T11:26:22.330 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.083e-05 2024-09-13T11:26:22.330 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.331 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.331 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:26:22.331 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.956395+0000", 2024-09-13T11:26:22.331 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00010184200000000001 2024-09-13T11:26:22.331 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.331 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.331 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:26:22.331 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.957640+0000", 2024-09-13T11:26:22.331 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0012450370000000001 2024-09-13T11:26:22.331 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:22.331 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:26:22.331 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:22.331 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.331 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.332 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPushReply(3.1f 89/78 [PushReplyOp(3:ff150976:::benchmark_data_smithi177_31116_object12907:head)])", 2024-09-13T11:26:22.332 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:46.697504+0000", 2024-09-13T11:26:22.332 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 155.579971409, 2024-09-13T11:26:22.332 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.26836818099999998, 2024-09-13T11:26:22.332 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:26:22.332 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:26:22.332 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-13T11:26:22.332 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:26:22.332 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.332 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:26:22.332 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.697504+0000", 2024-09-13T11:26:22.332 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:22.332 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.332 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.333 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:26:22.333 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.697504+0000", 2024-09-13T11:26:22.333 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:22.333 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.333 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.333 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:26:22.333 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.697505+0000", 2024-09-13T11:26:22.333 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0529999999999999e-06 2024-09-13T11:26:22.333 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.333 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.333 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:26:22.333 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.697509+0000", 2024-09-13T11:26:22.333 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.9249999999999997e-06 2024-09-13T11:26:22.333 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.334 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.334 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:26:22.334 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.964086+0000", 2024-09-13T11:26:22.334 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.26657776100000002 2024-09-13T11:26:22.334 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.334 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.334 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:26:22.334 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.964110+0000", 2024-09-13T11:26:22.334 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.3189000000000001e-05 2024-09-13T11:26:22.334 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.334 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.334 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:26:22.334 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.964354+0000", 2024-09-13T11:26:22.334 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00024481399999999999 2024-09-13T11:26:22.335 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.335 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.335 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:26:22.335 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.965872+0000", 2024-09-13T11:26:22.335 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001517439 2024-09-13T11:26:22.335 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:22.335 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:26:22.335 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:22.335 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.335 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.335 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:83223 3.1c 3:3bc6018d:::benchmark_data_smithi177_31116_object83222:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106)", 2024-09-13T11:26:22.335 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:04.511163+0000", 2024-09-13T11:26:22.335 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 137.76631163100001, 2024-09-13T11:26:22.335 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.47558093200000001, 2024-09-13T11:26:22.336 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:26:22.336 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:26:22.336 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:22.336 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:26:22.336 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:26:22.336 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:22.336 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 83223 2024-09-13T11:26:22.336 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.336 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:26:22.336 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.336 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:26:22.336 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.511163+0000", 2024-09-13T11:26:22.336 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:22.336 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.337 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.337 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:26:22.337 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.511163+0000", 2024-09-13T11:26:22.337 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:22.337 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.337 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.337 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:26:22.337 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.511167+0000", 2024-09-13T11:26:22.337 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.8330000000000004e-06 2024-09-13T11:26:22.337 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.337 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.337 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:26:22.337 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.511219+0000", 2024-09-13T11:26:22.337 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.1771999999999998e-05 2024-09-13T11:26:22.338 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.338 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.338 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:26:22.338 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.511252+0000", 2024-09-13T11:26:22.338 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.3315999999999997e-05 2024-09-13T11:26:22.338 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.338 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.338 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:26:22.338 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.511268+0000", 2024-09-13T11:26:22.338 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5812999999999999e-05 2024-09-13T11:26:22.338 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.338 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.338 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:26:22.338 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.511418+0000", 2024-09-13T11:26:22.339 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000150023 2024-09-13T11:26:22.339 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.339 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.339 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:26:22.339 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.511670+0000", 2024-09-13T11:26:22.339 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00025185499999999999 2024-09-13T11:26:22.339 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.339 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.339 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 7", 2024-09-13T11:26:22.339 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.511905+0000", 2024-09-13T11:26:22.339 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00023493799999999999 2024-09-13T11:26:22.339 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.339 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.339 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:26:22.340 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.514960+0000", 2024-09-13T11:26:22.340 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.003054977 2024-09-13T11:26:22.340 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.340 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.340 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:22.340 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.986432+0000", 2024-09-13T11:26:22.340 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.47147209200000001 2024-09-13T11:26:22.340 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.340 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.340 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:26:22.340 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.986635+0000", 2024-09-13T11:26:22.340 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00020309899999999999 2024-09-13T11:26:22.340 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.340 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.341 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:26:22.341 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.986744+0000", 2024-09-13T11:26:22.341 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000109214 2024-09-13T11:26:22.341 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:22.341 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:26:22.341 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:22.341 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.341 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.341 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop_reply(client.4665.0:84803 3.1f e107/78)", 2024-09-13T11:26:22.341 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:05.912627+0000", 2024-09-13T11:26:22.341 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 136.36484772399999, 2024-09-13T11:26:22.341 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.33726566600000002, 2024-09-13T11:26:22.341 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:26:22.341 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:26:22.342 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-13T11:26:22.342 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:26:22.342 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.342 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:26:22.342 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:05.912627+0000", 2024-09-13T11:26:22.342 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:22.342 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.342 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.342 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:26:22.342 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:05.912627+0000", 2024-09-13T11:26:22.342 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:22.343 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.343 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.343 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:26:22.343 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:05.912629+0000", 2024-09-13T11:26:22.343 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.4440000000000001e-06 2024-09-13T11:26:22.343 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.343 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.343 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:26:22.343 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:05.912634+0000", 2024-09-13T11:26:22.343 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.6869999999999999e-06 2024-09-13T11:26:22.343 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.343 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.343 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:26:22.343 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:06.249722+0000", 2024-09-13T11:26:22.344 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.33708766099999998 2024-09-13T11:26:22.344 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.344 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.344 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:26:22.344 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:06.249736+0000", 2024-09-13T11:26:22.344 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3623e-05 2024-09-13T11:26:22.344 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.344 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.344 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:26:22.344 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:06.249878+0000", 2024-09-13T11:26:22.344 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00014264599999999999 2024-09-13T11:26:22.344 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.344 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.345 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:26:22.345 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:06.249893+0000", 2024-09-13T11:26:22.345 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.4605e-05 2024-09-13T11:26:22.345 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:22.345 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:26:22.345 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:22.345 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.345 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.345 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop_reply(client.4665.0:110260 3.f e119/75 ondisk, result = 0)", 2024-09-13T11:26:22.345 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:25.333188+0000", 2024-09-13T11:26:22.345 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 116.944287097, 2024-09-13T11:26:22.345 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.29195486199999998, 2024-09-13T11:26:22.345 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:26:22.345 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:26:22.346 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:26:22.346 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:26:22.346 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.346 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:26:22.346 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.333188+0000", 2024-09-13T11:26:22.346 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:22.346 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.346 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.346 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:26:22.346 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.333188+0000", 2024-09-13T11:26:22.346 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:22.346 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.346 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.346 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:26:22.346 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.333190+0000", 2024-09-13T11:26:22.347 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.2120000000000002e-06 2024-09-13T11:26:22.347 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.347 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.347 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:26:22.347 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.333197+0000", 2024-09-13T11:26:22.347 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.7469999999999999e-06 2024-09-13T11:26:22.347 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.347 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.347 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:26:22.347 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.624892+0000", 2024-09-13T11:26:22.347 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.291694913 2024-09-13T11:26:22.347 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.347 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.348 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:26:22.348 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.624907+0000", 2024-09-13T11:26:22.348 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5053e-05 2024-09-13T11:26:22.348 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.348 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.348 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:26:22.348 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.624999+0000", 2024-09-13T11:26:22.348 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.2603999999999996e-05 2024-09-13T11:26:22.348 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.348 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.348 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:26:22.348 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.625011+0000", 2024-09-13T11:26:22.348 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.1253999999999999e-05 2024-09-13T11:26:22.349 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.349 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.349 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:26:22.349 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.625143+0000", 2024-09-13T11:26:22.349 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00013207900000000001 2024-09-13T11:26:22.349 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:22.349 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:26:22.349 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:22.349 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.349 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.349 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:110265 3.1 e119/75 3:8236484d:::benchmark_data_smithi177_31116_object110264:head v 119'3935, mlcod=118'3933)", 2024-09-13T11:26:22.349 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:25.343348+0000", 2024-09-13T11:26:22.349 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 116.93412748199999, 2024-09-13T11:26:22.349 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.60013013500000001, 2024-09-13T11:26:22.349 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:26:22.350 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:26:22.350 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:22.350 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:26:22.350 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.350 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:26:22.350 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.343348+0000", 2024-09-13T11:26:22.350 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:22.350 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.350 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.350 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:26:22.350 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.343348+0000", 2024-09-13T11:26:22.350 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:22.350 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.350 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.351 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:26:22.351 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.343350+0000", 2024-09-13T11:26:22.351 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.576e-06 2024-09-13T11:26:22.351 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.351 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.351 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:26:22.351 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.343443+0000", 2024-09-13T11:26:22.351 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.2989e-05 2024-09-13T11:26:22.351 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.351 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.351 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:26:22.351 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.940863+0000", 2024-09-13T11:26:22.351 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.59742003300000002 2024-09-13T11:26:22.351 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.352 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.352 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:26:22.352 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.940875+0000", 2024-09-13T11:26:22.352 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.2147000000000001e-05 2024-09-13T11:26:22.352 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.352 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.352 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:26:22.352 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.940993+0000", 2024-09-13T11:26:22.352 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000117903 2024-09-13T11:26:22.352 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.352 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.352 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:26:22.352 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.941017+0000", 2024-09-13T11:26:22.353 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.3442999999999999e-05 2024-09-13T11:26:22.353 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.353 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.353 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:26:22.353 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.943347+0000", 2024-09-13T11:26:22.353 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0023299869999999999 2024-09-13T11:26:22.353 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.353 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.353 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:26:22.353 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.943478+0000", 2024-09-13T11:26:22.353 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00013105699999999999 2024-09-13T11:26:22.353 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:22.353 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:26:22.353 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:22.354 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.354 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.354 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop_reply(client.4665.0:129919 3.16 e134/88 ondisk, result = 0)", 2024-09-13T11:26:22.354 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:47.175742+0000", 2024-09-13T11:26:22.354 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 95.101733412000002, 2024-09-13T11:26:22.354 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.27593726299999999, 2024-09-13T11:26:22.354 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:26:22.354 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:26:22.354 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:26:22.354 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:26:22.354 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.354 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:26:22.354 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:47.175742+0000", 2024-09-13T11:26:22.354 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:22.355 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.355 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.355 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:26:22.355 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:47.175742+0000", 2024-09-13T11:26:22.355 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:22.355 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.355 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.355 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:26:22.355 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:47.175744+0000", 2024-09-13T11:26:22.355 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.0030000000000001e-06 2024-09-13T11:26:22.355 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.355 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.355 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:26:22.355 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:47.175751+0000", 2024-09-13T11:26:22.356 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.9779999999999999e-06 2024-09-13T11:26:22.356 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.356 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.356 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:26:22.356 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:47.451490+0000", 2024-09-13T11:26:22.356 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.27573892500000002 2024-09-13T11:26:22.356 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.356 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.356 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:26:22.356 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:47.451496+0000", 2024-09-13T11:26:22.356 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.0079999999999999e-06 2024-09-13T11:26:22.356 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.356 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.357 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:26:22.357 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:47.451560+0000", 2024-09-13T11:26:22.357 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.3955999999999995e-05 2024-09-13T11:26:22.357 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.357 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.357 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:26:22.357 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:47.451567+0000", 2024-09-13T11:26:22.357 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.1980000000000004e-06 2024-09-13T11:26:22.357 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.357 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.357 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:26:22.357 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:47.451679+0000", 2024-09-13T11:26:22.357 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00011219500000000001 2024-09-13T11:26:22.357 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:22.358 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:26:22.358 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:22.358 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.358 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.358 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:209869 3.27 e185/169 3:e6b1fcc5:::benchmark_data_smithi177_31116_object69566:head v 185'6350, mlcod=185'6347)", 2024-09-13T11:26:22.358 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:25:54.301402+0000", 2024-09-13T11:26:22.358 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 27.976072740999999, 2024-09-13T11:26:22.358 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.30148222499999999, 2024-09-13T11:26:22.358 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:26:22.358 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:26:22.358 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:22.358 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:26:22.358 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.358 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:26:22.359 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:54.301402+0000", 2024-09-13T11:26:22.359 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:22.359 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.359 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.359 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:26:22.359 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:54.301402+0000", 2024-09-13T11:26:22.359 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:22.359 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.359 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.359 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:26:22.359 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:54.301404+0000", 2024-09-13T11:26:22.359 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.1939999999999999e-06 2024-09-13T11:26:22.359 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.359 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.360 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:26:22.360 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:54.301409+0000", 2024-09-13T11:26:22.360 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.0259999999999998e-06 2024-09-13T11:26:22.360 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.360 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.360 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:26:22.360 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:54.601076+0000", 2024-09-13T11:26:22.360 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.29966704799999999 2024-09-13T11:26:22.360 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.360 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.360 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:26:22.360 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:54.601081+0000", 2024-09-13T11:26:22.360 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.7069999999999997e-06 2024-09-13T11:26:22.361 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.361 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.361 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:26:22.361 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:54.601135+0000", 2024-09-13T11:26:22.361 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.3439000000000001e-05 2024-09-13T11:26:22.361 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.361 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.361 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:26:22.361 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:54.601150+0000", 2024-09-13T11:26:22.361 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5443000000000001e-05 2024-09-13T11:26:22.361 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.361 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.361 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:26:22.361 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:54.602854+0000", 2024-09-13T11:26:22.362 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0017035220000000001 2024-09-13T11:26:22.362 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.362 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.362 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:26:22.362 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:54.602885+0000", 2024-09-13T11:26:22.362 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.0846000000000002e-05 2024-09-13T11:26:22.362 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:22.362 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:26:22.362 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:22.362 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.362 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.362 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:234746 3.1 e201/75 3:83449ff7:::benchmark_data_smithi177_31116_object94443:head v 201'5848, mlcod=201'5845)", 2024-09-13T11:26:22.362 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:26:17.887870+0000", 2024-09-13T11:26:22.363 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 4.3896051839999997, 2024-09-13T11:26:22.363 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.29704435699999998, 2024-09-13T11:26:22.363 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:26:22.363 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:26:22.363 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:22.363 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:26:22.363 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.363 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:26:22.363 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:17.887870+0000", 2024-09-13T11:26:22.363 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:22.363 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.363 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.363 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:26:22.363 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:17.887870+0000", 2024-09-13T11:26:22.364 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:22.364 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.364 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.364 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:26:22.364 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:17.887871+0000", 2024-09-13T11:26:22.364 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.4750000000000001e-06 2024-09-13T11:26:22.364 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.364 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.364 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:26:22.364 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:17.887879+0000", 2024-09-13T11:26:22.364 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.2439999999999996e-06 2024-09-13T11:26:22.364 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.364 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.364 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:26:22.365 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:18.180816+0000", 2024-09-13T11:26:22.365 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.29293715799999998 2024-09-13T11:26:22.365 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.365 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.365 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:26:22.365 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:18.180828+0000", 2024-09-13T11:26:22.365 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.2713e-05 2024-09-13T11:26:22.365 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.365 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.365 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:26:22.365 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:18.181317+0000", 2024-09-13T11:26:22.365 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00048859800000000003 2024-09-13T11:26:22.365 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.366 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.366 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:26:22.366 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:18.181360+0000", 2024-09-13T11:26:22.366 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.3359000000000003e-05 2024-09-13T11:26:22.366 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.366 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.366 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:26:22.366 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:18.184822+0000", 2024-09-13T11:26:22.366 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0034615879999999998 2024-09-13T11:26:22.366 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.366 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.366 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:26:22.366 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:18.184914+0000", 2024-09-13T11:26:22.366 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.2221999999999999e-05 2024-09-13T11:26:22.367 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:22.367 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:26:22.367 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:22.367 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:22.367 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:26:22.367 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-09-13T11:26:22.368 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_ops_in_flight 2024-09-13T11:26:22.369 INFO:tasks.ceph.osd.5.smithi092.stderr:2024-09-13T11:26:22.364+0000 7f6f0b705640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26346) UID: 0 2024-09-13T11:26:22.469 INFO:tasks.ceph.osd.2.smithi162.stderr:2024-09-13T11:26:22.467+0000 7fe92b35a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26313) UID: 0 2024-09-13T11:26:22.550 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-09-13T11:26:22.566 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [], 2024-09-13T11:26:22.567 INFO:teuthology.orchestra.run.smithi177.stdout: "num_ops": 0 2024-09-13T11:26:22.567 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-09-13T11:26:22.569 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_blocked_ops 2024-09-13T11:26:22.570 INFO:tasks.ceph.osd.4.smithi001.stderr:2024-09-13T11:26:22.566+0000 7ff8e0218640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26310) UID: 0 2024-09-13T11:26:22.671 INFO:tasks.ceph.osd.2.smithi162.stderr:2024-09-13T11:26:22.667+0000 7fe92b35a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26313) UID: 0 2024-09-13T11:26:22.725 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-09-13T11:26:22.725 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [], 2024-09-13T11:26:22.725 INFO:teuthology.orchestra.run.smithi177.stdout: "complaint_time": 30, 2024-09-13T11:26:22.725 INFO:teuthology.orchestra.run.smithi177.stdout: "num_blocked_ops": 0 2024-09-13T11:26:22.725 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-09-13T11:26:22.746 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_historic_ops 2024-09-13T11:26:22.772 INFO:tasks.ceph.osd.10.smithi162.stderr:2024-09-13T11:26:22.767+0000 7fb59c9f0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26317) UID: 0 2024-09-13T11:26:22.873 INFO:tasks.ceph.osd.7.smithi177.stderr:2024-09-13T11:26:22.869+0000 7f897b51d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26575) UID: 0 2024-09-13T11:26:22.912 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-09-13T11:26:22.912 INFO:teuthology.orchestra.run.smithi177.stdout: "size": 20, 2024-09-13T11:26:22.912 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 600, 2024-09-13T11:26:22.912 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [ 2024-09-13T11:26:22.912 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.912 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:25383 3.0 e55/30 3:0a6bb869:::benchmark_data_smithi177_31116_object25382:head v 55'1620, mlcod=55'1617)", 2024-09-13T11:26:22.912 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:04.449956+0000", 2024-09-13T11:26:22.913 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 198.461657264, 2024-09-13T11:26:22.913 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.21143479600000001, 2024-09-13T11:26:22.913 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:26:22.913 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:26:22.913 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:22.913 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:26:22.913 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.913 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:26:22.913 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.449956+0000", 2024-09-13T11:26:22.913 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:22.913 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.913 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.913 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:26:22.913 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.449956+0000", 2024-09-13T11:26:22.914 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:22.914 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.914 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.914 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:26:22.914 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.449959+0000", 2024-09-13T11:26:22.914 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.393e-06 2024-09-13T11:26:22.914 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.914 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.914 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:26:22.914 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.450014+0000", 2024-09-13T11:26:22.914 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.4200000000000003e-05 2024-09-13T11:26:22.914 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.914 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.914 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:26:22.914 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.658637+0000", 2024-09-13T11:26:22.915 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.20862334799999999 2024-09-13T11:26:22.915 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.915 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.915 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:26:22.915 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.658650+0000", 2024-09-13T11:26:22.915 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3148e-05 2024-09-13T11:26:22.915 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.915 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.915 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:26:22.915 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.658867+0000", 2024-09-13T11:26:22.915 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00021673499999999999 2024-09-13T11:26:22.915 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.915 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.916 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:26:22.916 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.658931+0000", 2024-09-13T11:26:22.916 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.4045000000000001e-05 2024-09-13T11:26:22.916 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.916 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.916 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:26:22.916 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.661317+0000", 2024-09-13T11:26:22.916 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.002385823 2024-09-13T11:26:22.916 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.916 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.916 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:26:22.916 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.661391+0000", 2024-09-13T11:26:22.916 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.4103999999999994e-05 2024-09-13T11:26:22.916 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:22.917 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:26:22.917 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:22.917 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.917 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.917 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPush(3.16 85/70 [PushOp(3:69aac9fd:::benchmark_data_smithi177_31116_object43914:head, version: 66'2837, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:69aac9fd:::benchmark_data_smithi177_31116_object43914:head@66'2837, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:26:22.917 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:42.295995+0000", 2024-09-13T11:26:22.917 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 160.61561794400001, 2024-09-13T11:26:22.917 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.215089431, 2024-09-13T11:26:22.917 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:26:22.917 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:26:22.917 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:26:22.917 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:26:22.917 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.918 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:26:22.918 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.295995+0000", 2024-09-13T11:26:22.918 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:22.918 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.918 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.918 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:26:22.918 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.295995+0000", 2024-09-13T11:26:22.918 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:22.918 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.918 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.918 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:26:22.918 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.295997+0000", 2024-09-13T11:26:22.918 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3170000000000001e-06 2024-09-13T11:26:22.918 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.919 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.919 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:26:22.919 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.296058+0000", 2024-09-13T11:26:22.919 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.1690000000000001e-05 2024-09-13T11:26:22.919 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.919 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.919 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:26:22.919 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.509606+0000", 2024-09-13T11:26:22.919 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.21354748300000001 2024-09-13T11:26:22.919 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.919 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.919 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:26:22.919 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.509638+0000", 2024-09-13T11:26:22.919 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.1644999999999999e-05 2024-09-13T11:26:22.920 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.920 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.920 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:26:22.920 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.509814+0000", 2024-09-13T11:26:22.920 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00017679499999999999 2024-09-13T11:26:22.920 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.920 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.920 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:26:22.920 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.509840+0000", 2024-09-13T11:26:22.920 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.5871e-05 2024-09-13T11:26:22.920 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.920 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.920 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:26:22.921 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.511085+0000", 2024-09-13T11:26:22.921 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00124463 2024-09-13T11:26:22.921 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:22.921 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:26:22.921 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:22.921 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.921 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.921 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPush(3.1d 91/78 [PushOp(3:be6a0ae4:::benchmark_data_smithi177_31116_object25671:head, version: 58'1569, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:be6a0ae4:::benchmark_data_smithi177_31116_object25671:head@58'1569, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:26:22.921 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:48.481971+0000", 2024-09-13T11:26:22.921 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 154.429642699, 2024-09-13T11:26:22.921 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.47945454199999998, 2024-09-13T11:26:22.921 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:26:22.921 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:26:22.921 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:26:22.922 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:26:22.922 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.922 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:26:22.922 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.481971+0000", 2024-09-13T11:26:22.922 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:22.922 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.922 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.922 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:26:22.922 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.481971+0000", 2024-09-13T11:26:22.922 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:22.922 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.922 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.922 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:26:22.922 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.481972+0000", 2024-09-13T11:26:22.923 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5179999999999999e-06 2024-09-13T11:26:22.923 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.923 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.923 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:26:22.923 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.482000+0000", 2024-09-13T11:26:22.923 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.8249e-05 2024-09-13T11:26:22.923 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.923 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.923 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:26:22.923 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.959926+0000", 2024-09-13T11:26:22.923 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.47792585100000001 2024-09-13T11:26:22.923 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.923 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.923 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:26:22.923 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.959944+0000", 2024-09-13T11:26:22.924 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.7620000000000001e-05 2024-09-13T11:26:22.924 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.924 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.924 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:26:22.924 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.960059+0000", 2024-09-13T11:26:22.924 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000115188 2024-09-13T11:26:22.924 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.924 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.924 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:26:22.924 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.960074+0000", 2024-09-13T11:26:22.924 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.4738e-05 2024-09-13T11:26:22.924 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.924 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.925 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:26:22.925 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.961425+0000", 2024-09-13T11:26:22.925 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0013513780000000001 2024-09-13T11:26:22.925 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:22.925 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:26:22.925 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:22.925 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.925 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.925 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:71884 3.16 e96/88 3:6ca846b2:::benchmark_data_smithi177_31116_object71883:head v 96'3755, mlcod=96'3753)", 2024-09-13T11:26:22.925 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:54.108299+0000", 2024-09-13T11:26:22.925 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 148.80331452600001, 2024-09-13T11:26:22.925 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.20685140099999999, 2024-09-13T11:26:22.925 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:26:22.925 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:26:22.926 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:22.926 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:26:22.926 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.926 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:26:22.926 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.108299+0000", 2024-09-13T11:26:22.926 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:22.926 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.926 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.926 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:26:22.926 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.108299+0000", 2024-09-13T11:26:22.926 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:22.926 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.926 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.926 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:26:22.927 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.108300+0000", 2024-09-13T11:26:22.927 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.173e-06 2024-09-13T11:26:22.927 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.927 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.927 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:26:22.927 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.108323+0000", 2024-09-13T11:26:22.927 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.2591000000000001e-05 2024-09-13T11:26:22.927 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.927 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.927 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:26:22.927 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.312377+0000", 2024-09-13T11:26:22.927 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.20405456599999999 2024-09-13T11:26:22.927 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.927 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.928 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:26:22.928 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.312382+0000", 2024-09-13T11:26:22.928 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.1429999999999997e-06 2024-09-13T11:26:22.928 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.928 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.928 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:26:22.928 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.312480+0000", 2024-09-13T11:26:22.928 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.7742e-05 2024-09-13T11:26:22.928 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.928 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.928 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:26:22.928 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.312521+0000", 2024-09-13T11:26:22.928 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.1369000000000003e-05 2024-09-13T11:26:22.928 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.929 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.929 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:26:22.929 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.315105+0000", 2024-09-13T11:26:22.929 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0025835060000000002 2024-09-13T11:26:22.929 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.929 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.929 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:26:22.929 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.315150+0000", 2024-09-13T11:26:22.929 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.5311e-05 2024-09-13T11:26:22.929 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:22.929 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:26:22.929 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:22.929 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.929 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.929 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:71927 3.0 e97/75 3:0197244a:::benchmark_data_smithi177_31116_object71926:head v 97'3351, mlcod=96'3348)", 2024-09-13T11:26:22.930 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:54.200161+0000", 2024-09-13T11:26:22.930 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 148.71145229800001, 2024-09-13T11:26:22.930 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.26691738999999998, 2024-09-13T11:26:22.930 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:26:22.930 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:26:22.930 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:22.930 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:26:22.930 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.930 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:26:22.930 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.200161+0000", 2024-09-13T11:26:22.930 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:22.930 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.930 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.930 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:26:22.931 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.200161+0000", 2024-09-13T11:26:22.931 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:22.931 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.931 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.931 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:26:22.931 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.200162+0000", 2024-09-13T11:26:22.931 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.78e-07 2024-09-13T11:26:22.931 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.931 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.931 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:26:22.931 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.200200+0000", 2024-09-13T11:26:22.931 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.8121000000000003e-05 2024-09-13T11:26:22.931 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.931 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.932 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:26:22.932 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.465397+0000", 2024-09-13T11:26:22.932 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.26519681099999998 2024-09-13T11:26:22.932 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.932 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.932 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:26:22.932 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.465404+0000", 2024-09-13T11:26:22.932 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.7399999999999998e-06 2024-09-13T11:26:22.932 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.932 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.932 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:26:22.932 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.465456+0000", 2024-09-13T11:26:22.932 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.2626000000000001e-05 2024-09-13T11:26:22.932 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.933 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.933 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:26:22.933 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.465482+0000", 2024-09-13T11:26:22.933 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.6086000000000001e-05 2024-09-13T11:26:22.933 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.933 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.933 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:26:22.933 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.467035+0000", 2024-09-13T11:26:22.933 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0015523729999999999 2024-09-13T11:26:22.933 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.933 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.933 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:26:22.933 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.467078+0000", 2024-09-13T11:26:22.933 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.3754999999999997e-05 2024-09-13T11:26:22.933 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:22.934 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:26:22.934 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:22.934 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.934 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.934 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPush(3.1b 102/78 [PushOp(3:df786fc7:::benchmark_data_smithi177_31116_object48840:head, version: 71'3079, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:df786fc7:::benchmark_data_smithi177_31116_object48840:head@71'3079, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:26:22.934 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:00.290536+0000", 2024-09-13T11:26:22.934 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 142.621077536, 2024-09-13T11:26:22.934 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.20017585399999999, 2024-09-13T11:26:22.934 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:26:22.934 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:26:22.934 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:26:22.934 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:26:22.934 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.934 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:26:22.935 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.290536+0000", 2024-09-13T11:26:22.935 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:22.935 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.935 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.935 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:26:22.935 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.290536+0000", 2024-09-13T11:26:22.935 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:22.935 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.935 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.935 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:26:22.935 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.290537+0000", 2024-09-13T11:26:22.935 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.327e-06 2024-09-13T11:26:22.935 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.935 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.936 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:26:22.936 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.290581+0000", 2024-09-13T11:26:22.936 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.3893999999999997e-05 2024-09-13T11:26:22.936 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.936 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.936 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:26:22.936 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.488588+0000", 2024-09-13T11:26:22.936 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.198006708 2024-09-13T11:26:22.936 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.936 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.936 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:26:22.936 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.488607+0000", 2024-09-13T11:26:22.936 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.9162999999999999e-05 2024-09-13T11:26:22.937 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.937 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.937 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:26:22.937 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.488747+0000", 2024-09-13T11:26:22.937 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00014040799999999999 2024-09-13T11:26:22.937 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.937 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.937 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:26:22.937 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.488770+0000", 2024-09-13T11:26:22.937 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.279e-05 2024-09-13T11:26:22.937 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.938 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.938 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:26:22.938 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.490712+0000", 2024-09-13T11:26:22.938 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001941564 2024-09-13T11:26:22.938 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:22.938 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:26:22.938 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:22.938 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.938 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.938 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPush(3.1b 102/78 [PushOp(3:d82bf6ec:::benchmark_data_smithi177_31116_object48847:head, version: 71'3080, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:d82bf6ec:::benchmark_data_smithi177_31116_object48847:head@71'3080, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:26:22.938 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:00.293979+0000", 2024-09-13T11:26:22.938 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 142.617634009, 2024-09-13T11:26:22.938 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.20314832099999999, 2024-09-13T11:26:22.938 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:26:22.939 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:26:22.939 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:26:22.939 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:26:22.939 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.939 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:26:22.939 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.293979+0000", 2024-09-13T11:26:22.939 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:22.939 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.939 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.939 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:26:22.939 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.293979+0000", 2024-09-13T11:26:22.939 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:22.939 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.940 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.940 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:26:22.940 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.293981+0000", 2024-09-13T11:26:22.940 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.4449999999999999e-06 2024-09-13T11:26:22.940 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.940 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.940 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:26:22.940 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.294085+0000", 2024-09-13T11:26:22.940 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000103848 2024-09-13T11:26:22.940 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.940 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.940 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:26:22.940 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.495571+0000", 2024-09-13T11:26:22.941 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.20148640800000001 2024-09-13T11:26:22.941 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.941 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.941 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:26:22.941 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.495609+0000", 2024-09-13T11:26:22.941 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.7904000000000002e-05 2024-09-13T11:26:22.941 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.941 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.941 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:26:22.941 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.495729+0000", 2024-09-13T11:26:22.941 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000120556 2024-09-13T11:26:22.941 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.941 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.941 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:26:22.942 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.495751+0000", 2024-09-13T11:26:22.942 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.1511000000000001e-05 2024-09-13T11:26:22.942 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.942 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.942 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:26:22.942 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.497128+0000", 2024-09-13T11:26:22.942 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001376649 2024-09-13T11:26:22.942 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:22.942 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:26:22.942 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:22.942 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.942 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.942 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:79801 3.5 3:a6efdf84:::benchmark_data_smithi177_31116_object79800:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102)", 2024-09-13T11:26:22.942 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:00.403201+0000", 2024-09-13T11:26:22.943 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 142.50841242199999, 2024-09-13T11:26:22.943 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.31868626300000003, 2024-09-13T11:26:22.943 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:26:22.943 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:26:22.943 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:22.943 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:26:22.943 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:26:22.943 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:22.943 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 79801 2024-09-13T11:26:22.943 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.943 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:26:22.943 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.943 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:26:22.943 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.403201+0000", 2024-09-13T11:26:22.944 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:22.944 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.944 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.944 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:26:22.944 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.403201+0000", 2024-09-13T11:26:22.944 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:22.944 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.944 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.944 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:26:22.944 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.403204+0000", 2024-09-13T11:26:22.945 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.0819999999999999e-06 2024-09-13T11:26:22.945 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.945 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.945 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:26:22.945 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.403260+0000", 2024-09-13T11:26:22.945 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.5624999999999997e-05 2024-09-13T11:26:22.945 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.945 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.945 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:26:22.945 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.403280+0000", 2024-09-13T11:26:22.945 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.0060000000000001e-05 2024-09-13T11:26:22.945 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.945 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.946 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:26:22.946 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.403291+0000", 2024-09-13T11:26:22.946 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.1335000000000001e-05 2024-09-13T11:26:22.946 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.946 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.946 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:26:22.946 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.403344+0000", 2024-09-13T11:26:22.946 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.3217000000000002e-05 2024-09-13T11:26:22.946 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.946 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.946 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:26:22.946 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.403456+0000", 2024-09-13T11:26:22.946 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000111706 2024-09-13T11:26:22.946 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.947 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.947 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 13", 2024-09-13T11:26:22.947 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.403547+0000", 2024-09-13T11:26:22.947 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.1556000000000002e-05 2024-09-13T11:26:22.947 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.947 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.947 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:26:22.947 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.407436+0000", 2024-09-13T11:26:22.947 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.003888856 2024-09-13T11:26:22.947 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.947 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.947 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:22.947 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.721722+0000", 2024-09-13T11:26:22.947 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.31428562700000001 2024-09-13T11:26:22.948 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.948 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.948 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:26:22.948 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.721814+0000", 2024-09-13T11:26:22.948 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.1564000000000006e-05 2024-09-13T11:26:22.948 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.948 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.948 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:26:22.948 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.721887+0000", 2024-09-13T11:26:22.948 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.3634999999999995e-05 2024-09-13T11:26:22.948 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:22.948 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:26:22.948 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:22.949 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.949 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.949 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:83199 3.1d e106/102 3:be2c6fae:::benchmark_data_smithi177_31116_object83198:head v 106'4244, mlcod=106'4242)", 2024-09-13T11:26:22.949 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:04.298299+0000", 2024-09-13T11:26:22.949 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 138.61331429500001, 2024-09-13T11:26:22.949 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.25001107099999997, 2024-09-13T11:26:22.949 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:26:22.949 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:26:22.949 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:22.949 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:26:22.949 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.949 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:26:22.949 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.298299+0000", 2024-09-13T11:26:22.949 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:22.950 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.950 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.950 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:26:22.950 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.298299+0000", 2024-09-13T11:26:22.950 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:22.950 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.950 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.950 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:26:22.950 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.298301+0000", 2024-09-13T11:26:22.950 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5859999999999999e-06 2024-09-13T11:26:22.950 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.950 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.950 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:26:22.950 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.298368+0000", 2024-09-13T11:26:22.950 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.6928999999999996e-05 2024-09-13T11:26:22.951 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.951 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.951 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:26:22.951 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.545180+0000", 2024-09-13T11:26:22.951 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.24681209700000001 2024-09-13T11:26:22.951 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.951 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.951 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:26:22.951 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.545193+0000", 2024-09-13T11:26:22.951 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3427e-05 2024-09-13T11:26:22.951 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.951 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.951 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:26:22.951 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.545327+0000", 2024-09-13T11:26:22.952 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00013359400000000001 2024-09-13T11:26:22.952 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.952 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.952 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:26:22.952 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.545385+0000", 2024-09-13T11:26:22.952 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.8492000000000002e-05 2024-09-13T11:26:22.952 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.952 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.952 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:26:22.952 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.548225+0000", 2024-09-13T11:26:22.952 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.002839472 2024-09-13T11:26:22.952 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.952 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.952 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:26:22.953 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.548310+0000", 2024-09-13T11:26:22.953 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.5474000000000002e-05 2024-09-13T11:26:22.953 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:22.953 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:26:22.953 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:22.953 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.953 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.953 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:107910 3.16 e116/88 3:6cf57aa3:::benchmark_data_smithi177_31116_object107909:head v 116'4911, mlcod=116'4908)", 2024-09-13T11:26:22.953 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:22.516225+0000", 2024-09-13T11:26:22.953 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 120.39538865199999, 2024-09-13T11:26:22.953 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.18319666600000001, 2024-09-13T11:26:22.953 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:26:22.953 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:26:22.954 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:22.954 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:26:22.954 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.954 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:26:22.954 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:22.516225+0000", 2024-09-13T11:26:22.954 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:22.954 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.954 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.954 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:26:22.954 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:22.516225+0000", 2024-09-13T11:26:22.955 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:22.955 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.955 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.955 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:26:22.955 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:22.516226+0000", 2024-09-13T11:26:22.955 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.716e-06 2024-09-13T11:26:22.955 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.955 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.955 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:26:22.955 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:22.516250+0000", 2024-09-13T11:26:22.955 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.3975e-05 2024-09-13T11:26:22.955 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.955 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.956 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:26:22.956 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:22.697686+0000", 2024-09-13T11:26:22.956 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.18143539 2024-09-13T11:26:22.956 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.956 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.956 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:26:22.956 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:22.697692+0000", 2024-09-13T11:26:22.956 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.0769999999999996e-06 2024-09-13T11:26:22.956 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.956 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.956 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:26:22.956 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:22.697744+0000", 2024-09-13T11:26:22.956 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.2525000000000003e-05 2024-09-13T11:26:22.956 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.957 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.957 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:26:22.957 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:22.697757+0000", 2024-09-13T11:26:22.957 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.2819e-05 2024-09-13T11:26:22.957 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.957 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.957 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:26:22.957 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:22.699388+0000", 2024-09-13T11:26:22.957 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0016303999999999999 2024-09-13T11:26:22.957 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.957 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.957 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:26:22.958 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:22.699421+0000", 2024-09-13T11:26:22.958 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.3763999999999997e-05 2024-09-13T11:26:22.958 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:22.958 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:26:22.958 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:22.958 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.958 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.958 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:127682 3.16 e131/88 3:6e6da02e:::benchmark_data_smithi177_31116_object127681:head v 131'5491, mlcod=131'5488)", 2024-09-13T11:26:22.958 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:44.556692+0000", 2024-09-13T11:26:22.958 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 98.354921723000004, 2024-09-13T11:26:22.958 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.18048624299999999, 2024-09-13T11:26:22.958 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:26:22.958 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:26:22.958 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:22.959 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:26:22.959 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.959 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:26:22.959 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:44.556692+0000", 2024-09-13T11:26:22.959 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:22.959 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.959 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.959 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:26:22.959 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:44.556692+0000", 2024-09-13T11:26:22.959 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:22.959 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.959 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.959 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:26:22.959 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:44.556693+0000", 2024-09-13T11:26:22.960 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.8109999999999999e-06 2024-09-13T11:26:22.960 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.960 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.960 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:26:22.960 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:44.556715+0000", 2024-09-13T11:26:22.960 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.1328999999999999e-05 2024-09-13T11:26:22.960 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.960 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.960 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:26:22.960 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:44.734908+0000", 2024-09-13T11:26:22.960 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.178192884 2024-09-13T11:26:22.960 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.960 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.960 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:26:22.961 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:44.734915+0000", 2024-09-13T11:26:22.961 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.9970000000000001e-06 2024-09-13T11:26:22.961 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.961 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.961 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:26:22.961 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:44.734971+0000", 2024-09-13T11:26:22.961 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.6368e-05 2024-09-13T11:26:22.961 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.961 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.961 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:26:22.961 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:44.734990+0000", 2024-09-13T11:26:22.961 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.929e-05 2024-09-13T11:26:22.961 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.961 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.962 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:26:22.962 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:44.737096+0000", 2024-09-13T11:26:22.962 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0021053090000000001 2024-09-13T11:26:22.962 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.962 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.962 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:26:22.962 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:44.737178+0000", 2024-09-13T11:26:22.962 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.2255000000000004e-05 2024-09-13T11:26:22.962 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:22.962 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:26:22.962 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:22.962 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.962 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.962 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:137062 3.5 3:a16e178f:::benchmark_data_smithi177_31116_object137061:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e139)", 2024-09-13T11:26:22.963 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:53.140525+0000", 2024-09-13T11:26:22.963 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 89.771088702, 2024-09-13T11:26:22.963 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.20697954599999999, 2024-09-13T11:26:22.963 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:26:22.963 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:26:22.963 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:22.963 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:26:22.963 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:26:22.963 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:22.963 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 137062 2024-09-13T11:26:22.963 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.963 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:26:22.963 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.963 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:26:22.963 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:53.140525+0000", 2024-09-13T11:26:22.964 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:22.964 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.964 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.964 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:26:22.964 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:53.140525+0000", 2024-09-13T11:26:22.964 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:22.964 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.964 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.964 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:26:22.964 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:53.140527+0000", 2024-09-13T11:26:22.964 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.1739999999999999e-06 2024-09-13T11:26:22.964 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.964 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.965 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:26:22.965 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:53.140553+0000", 2024-09-13T11:26:22.965 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.6631e-05 2024-09-13T11:26:22.965 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.965 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.965 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:26:22.965 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:53.140566+0000", 2024-09-13T11:26:22.965 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.2268e-05 2024-09-13T11:26:22.965 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.965 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.965 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:26:22.965 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:53.140574+0000", 2024-09-13T11:26:22.965 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.5669999999999995e-06 2024-09-13T11:26:22.965 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.966 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.966 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:26:22.966 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:53.140622+0000", 2024-09-13T11:26:22.966 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.8143999999999998e-05 2024-09-13T11:26:22.966 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.966 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.966 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:26:22.966 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:53.140759+0000", 2024-09-13T11:26:22.966 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000136499 2024-09-13T11:26:22.966 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.966 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.966 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 13", 2024-09-13T11:26:22.966 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:53.140887+0000", 2024-09-13T11:26:22.966 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00012796300000000001 2024-09-13T11:26:22.967 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.967 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.967 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:26:22.967 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:53.142675+0000", 2024-09-13T11:26:22.967 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001788073 2024-09-13T11:26:22.967 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.967 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.967 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:22.967 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:53.347407+0000", 2024-09-13T11:26:22.967 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.20473206899999999 2024-09-13T11:26:22.967 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.967 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.967 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:26:22.967 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:53.347464+0000", 2024-09-13T11:26:22.968 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.6536000000000002e-05 2024-09-13T11:26:22.968 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.968 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.968 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:26:22.968 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:53.347504+0000", 2024-09-13T11:26:22.968 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.0621999999999998e-05 2024-09-13T11:26:22.968 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:22.968 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:26:22.968 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:22.968 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.968 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.968 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:191058 3.1d e175/102 3:bd6e7029:::benchmark_data_smithi177_31116_object50755:head v 175'7612, mlcod=175'7609)", 2024-09-13T11:26:22.968 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:25:38.513527+0000", 2024-09-13T11:26:22.968 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 44.398086044999999, 2024-09-13T11:26:22.969 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.386104477, 2024-09-13T11:26:22.969 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:26:22.969 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:26:22.969 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:22.969 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:26:22.969 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.969 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:26:22.969 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:38.513527+0000", 2024-09-13T11:26:22.969 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:22.969 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.969 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.969 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:26:22.969 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:38.513527+0000", 2024-09-13T11:26:22.970 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:22.970 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.970 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.970 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:26:22.970 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:38.513530+0000", 2024-09-13T11:26:22.970 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.0029999999999999e-06 2024-09-13T11:26:22.970 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.970 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.970 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:26:22.970 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:38.513542+0000", 2024-09-13T11:26:22.970 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.1477e-05 2024-09-13T11:26:22.970 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.970 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.970 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:26:22.971 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:38.891617+0000", 2024-09-13T11:26:22.971 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.378075045 2024-09-13T11:26:22.971 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.971 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.971 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:26:22.971 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:38.891635+0000", 2024-09-13T11:26:22.971 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.7697e-05 2024-09-13T11:26:22.971 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.971 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.971 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:26:22.971 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:38.891775+0000", 2024-09-13T11:26:22.971 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000140665 2024-09-13T11:26:22.971 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.971 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.972 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:26:22.972 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:38.891833+0000", 2024-09-13T11:26:22.972 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.7769999999999997e-05 2024-09-13T11:26:22.972 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.972 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.972 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:26:22.972 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:38.899567+0000", 2024-09-13T11:26:22.972 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0077339970000000003 2024-09-13T11:26:22.972 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.972 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.972 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:26:22.972 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:38.899632+0000", 2024-09-13T11:26:22.972 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.4822999999999999e-05 2024-09-13T11:26:22.973 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:22.973 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:26:22.973 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:22.973 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.973 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.973 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:197236 3.5 3:a3127298:::benchmark_data_smithi177_31116_object56933:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e178)", 2024-09-13T11:26:22.973 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:25:44.300667+0000", 2024-09-13T11:26:22.973 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 38.610945954000002, 2024-09-13T11:26:22.973 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.37481205200000001, 2024-09-13T11:26:22.973 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:26:22.973 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:26:22.973 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:22.973 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:26:22.973 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:26:22.974 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:22.974 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 197236 2024-09-13T11:26:22.974 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.974 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:26:22.974 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.974 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:26:22.974 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:44.300667+0000", 2024-09-13T11:26:22.974 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:22.974 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.974 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.974 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:26:22.974 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:44.300667+0000", 2024-09-13T11:26:22.974 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:22.974 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.975 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.975 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:26:22.975 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:44.300670+0000", 2024-09-13T11:26:22.975 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.7259999999999998e-06 2024-09-13T11:26:22.975 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.975 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.975 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:26:22.975 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:44.300677+0000", 2024-09-13T11:26:22.975 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.1550000000000001e-06 2024-09-13T11:26:22.975 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.975 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.975 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:26:22.975 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:44.300695+0000", 2024-09-13T11:26:22.975 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.7552999999999999e-05 2024-09-13T11:26:22.976 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.976 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.976 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:26:22.976 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:44.300704+0000", 2024-09-13T11:26:22.976 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.5710000000000004e-06 2024-09-13T11:26:22.976 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.976 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.976 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:26:22.976 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:44.300762+0000", 2024-09-13T11:26:22.976 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.7351e-05 2024-09-13T11:26:22.976 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.976 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.976 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:26:22.976 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:44.300888+0000", 2024-09-13T11:26:22.976 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00012617999999999999 2024-09-13T11:26:22.977 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.977 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.977 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 13", 2024-09-13T11:26:22.977 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:44.300993+0000", 2024-09-13T11:26:22.977 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00010505600000000001 2024-09-13T11:26:22.977 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.977 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.977 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:26:22.977 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:44.302936+0000", 2024-09-13T11:26:22.977 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0019428749999999999 2024-09-13T11:26:22.977 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.977 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.977 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:22.977 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:44.675352+0000", 2024-09-13T11:26:22.978 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.37241603400000001 2024-09-13T11:26:22.978 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.978 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.978 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:26:22.978 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:44.675412+0000", 2024-09-13T11:26:22.978 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.9679999999999998e-05 2024-09-13T11:26:22.978 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.978 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.978 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:26:22.978 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:44.675479+0000", 2024-09-13T11:26:22.978 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.7871000000000001e-05 2024-09-13T11:26:22.978 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:22.978 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:26:22.978 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:22.978 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.979 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.979 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop_reply(client.4665.0:197236 3.5 e178/117 ondisk, result = 0)", 2024-09-13T11:26:22.979 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:25:44.304480+0000", 2024-09-13T11:26:22.979 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 38.607133687000001, 2024-09-13T11:26:22.979 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.37100864099999997, 2024-09-13T11:26:22.979 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:26:22.979 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:26:22.979 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:26:22.979 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:26:22.979 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.979 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:26:22.979 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:44.304480+0000", 2024-09-13T11:26:22.979 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:22.980 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.980 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.980 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:26:22.980 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:44.304480+0000", 2024-09-13T11:26:22.980 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:22.980 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.980 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.980 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:26:22.980 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:44.304481+0000", 2024-09-13T11:26:22.980 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.7349999999999999e-06 2024-09-13T11:26:22.980 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.980 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.980 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:26:22.980 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:44.304492+0000", 2024-09-13T11:26:22.981 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0394000000000001e-05 2024-09-13T11:26:22.981 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.981 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.981 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:26:22.981 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:44.675037+0000", 2024-09-13T11:26:22.981 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.37054519400000002 2024-09-13T11:26:22.981 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.981 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.981 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:26:22.981 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:44.675045+0000", 2024-09-13T11:26:22.981 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.1370000000000002e-06 2024-09-13T11:26:22.981 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.981 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.981 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:26:22.982 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:44.675334+0000", 2024-09-13T11:26:22.982 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00028848699999999998 2024-09-13T11:26:22.982 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.982 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.982 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:26:22.982 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:44.675345+0000", 2024-09-13T11:26:22.982 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.1198e-05 2024-09-13T11:26:22.982 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.982 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.982 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:26:22.982 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:44.675488+0000", 2024-09-13T11:26:22.982 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00014349600000000001 2024-09-13T11:26:22.982 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:22.982 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:26:22.983 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:22.983 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.983 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.983 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:217032 3.16 e187/88 3:6d28a27c:::benchmark_data_smithi177_31116_object76729:head v 187'8398, mlcod=187'8395)", 2024-09-13T11:26:22.983 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:25:59.836895+0000", 2024-09-13T11:26:22.983 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 23.074718802, 2024-09-13T11:26:22.983 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.22574986, 2024-09-13T11:26:22.983 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:26:22.983 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:26:22.983 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:22.983 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:26:22.983 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.984 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:26:22.984 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:59.836895+0000", 2024-09-13T11:26:22.984 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:22.984 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.984 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.984 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:26:22.984 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:59.836895+0000", 2024-09-13T11:26:22.984 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:22.984 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.984 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.984 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:26:22.984 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:59.836896+0000", 2024-09-13T11:26:22.984 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.432e-06 2024-09-13T11:26:22.984 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.985 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.985 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:26:22.985 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:59.836904+0000", 2024-09-13T11:26:22.985 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.9149999999999996e-06 2024-09-13T11:26:22.985 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.985 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.985 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:26:22.985 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:00.053571+0000", 2024-09-13T11:26:22.985 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.21666679799999999 2024-09-13T11:26:22.985 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.985 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.985 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:26:22.985 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:00.053582+0000", 2024-09-13T11:26:22.986 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0927999999999999e-05 2024-09-13T11:26:22.986 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.986 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.986 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:26:22.986 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:00.053811+0000", 2024-09-13T11:26:22.986 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00022964700000000001 2024-09-13T11:26:22.986 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.986 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.986 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:26:22.986 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:00.053864+0000", 2024-09-13T11:26:22.986 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.2710000000000002e-05 2024-09-13T11:26:22.986 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.986 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.986 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:26:22.987 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:00.062547+0000", 2024-09-13T11:26:22.987 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0086832650000000008 2024-09-13T11:26:22.987 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.987 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.987 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:26:22.987 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:00.062644+0000", 2024-09-13T11:26:22.987 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.7164999999999996e-05 2024-09-13T11:26:22.987 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:22.987 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:26:22.987 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:22.987 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.987 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.987 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:227461 3.5 3:a2578202:::benchmark_data_smithi177_31116_object87158:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e197)", 2024-09-13T11:26:22.988 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:26:11.215213+0000", 2024-09-13T11:26:22.988 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 11.696400262000001, 2024-09-13T11:26:22.988 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.18373250799999999, 2024-09-13T11:26:22.988 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:26:22.988 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:26:22.988 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:22.988 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:26:22.988 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:26:22.988 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:22.988 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 227461 2024-09-13T11:26:22.989 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.989 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:26:22.989 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.989 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:26:22.989 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:11.215213+0000", 2024-09-13T11:26:22.989 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:22.989 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.989 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.989 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:26:22.989 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:11.215213+0000", 2024-09-13T11:26:22.989 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:22.989 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.989 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.989 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:26:22.990 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:11.215216+0000", 2024-09-13T11:26:22.990 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.1370000000000002e-06 2024-09-13T11:26:22.990 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.990 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.990 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:26:22.990 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:11.215230+0000", 2024-09-13T11:26:22.990 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3976e-05 2024-09-13T11:26:22.990 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.990 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.990 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:26:22.990 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:11.215257+0000", 2024-09-13T11:26:22.990 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.6849999999999999e-05 2024-09-13T11:26:22.990 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.990 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.991 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:26:22.991 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:11.215271+0000", 2024-09-13T11:26:22.991 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3723000000000001e-05 2024-09-13T11:26:22.991 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.991 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.991 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:26:22.991 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:11.215401+0000", 2024-09-13T11:26:22.991 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00013043899999999999 2024-09-13T11:26:22.991 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.991 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.991 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:26:22.991 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:11.215710+0000", 2024-09-13T11:26:22.991 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00030838600000000001 2024-09-13T11:26:22.991 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.992 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.992 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 13", 2024-09-13T11:26:22.992 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:11.215863+0000", 2024-09-13T11:26:22.992 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00015319800000000001 2024-09-13T11:26:22.992 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.992 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.992 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:26:22.992 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:11.219895+0000", 2024-09-13T11:26:22.992 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0040320809999999999 2024-09-13T11:26:22.992 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.992 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.992 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:22.992 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:11.398827+0000", 2024-09-13T11:26:22.992 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.17893184000000001 2024-09-13T11:26:22.993 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.993 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.993 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:26:22.993 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:11.398895+0000", 2024-09-13T11:26:22.993 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.8046000000000005e-05 2024-09-13T11:26:22.993 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.993 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.993 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:26:22.993 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:11.398946+0000", 2024-09-13T11:26:22.993 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.0831999999999997e-05 2024-09-13T11:26:22.993 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:22.993 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:26:22.993 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:22.993 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.994 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.994 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:228538 3.5 3:a2ca23fe:::benchmark_data_smithi177_31116_object88235:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e198)", 2024-09-13T11:26:22.994 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:26:12.109379+0000", 2024-09-13T11:26:22.994 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 10.802234819000001, 2024-09-13T11:26:22.994 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.19925963699999999, 2024-09-13T11:26:22.994 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:26:22.994 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:26:22.994 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:22.994 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:26:22.994 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:26:22.994 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:22.994 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 228538 2024-09-13T11:26:22.994 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.994 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:26:22.995 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.995 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:26:22.995 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:12.109379+0000", 2024-09-13T11:26:22.995 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:22.995 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.995 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.995 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:26:22.995 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:12.109379+0000", 2024-09-13T11:26:22.995 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:22.995 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.995 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.995 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:26:22.995 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:12.109380+0000", 2024-09-13T11:26:22.995 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.9769999999999999e-06 2024-09-13T11:26:22.996 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.996 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.996 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:26:22.996 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:12.109387+0000", 2024-09-13T11:26:22.996 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.071e-06 2024-09-13T11:26:22.996 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.996 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.996 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:26:22.996 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:12.109401+0000", 2024-09-13T11:26:22.996 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.4487e-05 2024-09-13T11:26:22.996 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.996 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.996 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:26:22.996 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:12.109410+0000", 2024-09-13T11:26:22.997 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.4970000000000001e-06 2024-09-13T11:26:22.997 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.997 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.997 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:26:22.997 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:12.109467+0000", 2024-09-13T11:26:22.997 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.7080999999999998e-05 2024-09-13T11:26:22.997 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.997 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.997 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:26:22.997 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:12.109610+0000", 2024-09-13T11:26:22.997 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00014334699999999999 2024-09-13T11:26:22.997 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.997 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.998 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 13", 2024-09-13T11:26:22.998 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:12.109732+0000", 2024-09-13T11:26:22.998 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000122201 2024-09-13T11:26:22.998 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.998 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.998 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:26:22.998 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:12.113915+0000", 2024-09-13T11:26:22.998 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0041823820000000001 2024-09-13T11:26:22.998 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.998 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.998 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:22.998 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:12.308575+0000", 2024-09-13T11:26:22.998 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.194660687 2024-09-13T11:26:22.999 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.999 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.999 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:26:22.999 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:12.308609+0000", 2024-09-13T11:26:22.999 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.4103000000000002e-05 2024-09-13T11:26:22.999 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.999 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:22.999 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:26:22.999 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:12.308638+0000", 2024-09-13T11:26:22.999 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.8804000000000001e-05 2024-09-13T11:26:22.999 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:22.999 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:26:22.999 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:22.999 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:22.999 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:23.000 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:234789 3.5 3:a0df9e27:::benchmark_data_smithi177_31116_object94486:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e201)", 2024-09-13T11:26:23.000 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:26:17.919815+0000", 2024-09-13T11:26:23.000 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 4.9917986570000004, 2024-09-13T11:26:23.000 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.34227359400000001, 2024-09-13T11:26:23.000 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:26:23.000 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:26:23.000 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:23.000 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:26:23.000 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:26:23.000 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:23.000 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 234789 2024-09-13T11:26:23.000 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:23.000 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:26:23.000 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:23.000 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:26:23.001 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:17.919815+0000", 2024-09-13T11:26:23.001 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:23.001 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:23.001 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:23.001 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:26:23.001 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:17.919815+0000", 2024-09-13T11:26:23.001 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:23.001 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:23.001 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:23.001 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:26:23.001 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:17.919817+0000", 2024-09-13T11:26:23.001 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.6369999999999999e-06 2024-09-13T11:26:23.001 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:23.001 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:23.002 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:26:23.002 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:17.919824+0000", 2024-09-13T11:26:23.002 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.9290000000000001e-06 2024-09-13T11:26:23.002 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:23.002 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:23.002 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:26:23.002 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:17.919839+0000", 2024-09-13T11:26:23.002 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.4763000000000001e-05 2024-09-13T11:26:23.002 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:23.002 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:23.002 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:26:23.002 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:17.919846+0000", 2024-09-13T11:26:23.002 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.1829999999999997e-06 2024-09-13T11:26:23.002 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:23.003 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:23.003 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:26:23.003 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:17.919894+0000", 2024-09-13T11:26:23.003 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.7475999999999997e-05 2024-09-13T11:26:23.003 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:23.003 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:23.003 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:26:23.003 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:17.919996+0000", 2024-09-13T11:26:23.003 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00010262199999999999 2024-09-13T11:26:23.003 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:23.003 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:23.003 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 13", 2024-09-13T11:26:23.003 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:17.920092+0000", 2024-09-13T11:26:23.003 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.5546999999999998e-05 2024-09-13T11:26:23.004 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:23.004 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:23.004 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:26:23.004 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:17.921865+0000", 2024-09-13T11:26:23.004 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00177271 2024-09-13T11:26:23.004 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:23.004 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:23.004 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:23.004 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:18.261907+0000", 2024-09-13T11:26:23.004 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.34004223300000003 2024-09-13T11:26:23.004 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:23.004 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:23.004 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:26:23.005 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:18.262002+0000", 2024-09-13T11:26:23.005 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.4745000000000002e-05 2024-09-13T11:26:23.005 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:23.005 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:23.005 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:26:23.005 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:18.262088+0000", 2024-09-13T11:26:23.005 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.6749000000000006e-05 2024-09-13T11:26:23.005 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:23.005 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:26:23.005 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:23.005 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:23.005 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:23.005 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop_reply(client.4665.0:234789 3.5 e201/117 ondisk, result = 0)", 2024-09-13T11:26:23.005 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:26:17.922332+0000", 2024-09-13T11:26:23.005 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 4.9892813189999998, 2024-09-13T11:26:23.006 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.33978349299999999, 2024-09-13T11:26:23.006 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:26:23.006 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:26:23.006 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:26:23.006 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:26:23.006 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:23.006 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:26:23.006 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:17.922332+0000", 2024-09-13T11:26:23.006 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:23.006 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:23.006 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:23.006 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:26:23.006 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:17.922332+0000", 2024-09-13T11:26:23.006 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:23.007 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:23.007 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:23.007 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:26:23.007 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:17.922333+0000", 2024-09-13T11:26:23.007 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.2929999999999999e-06 2024-09-13T11:26:23.007 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:23.007 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:23.007 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:26:23.007 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:17.922337+0000", 2024-09-13T11:26:23.007 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.7469999999999999e-06 2024-09-13T11:26:23.007 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:23.007 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:23.007 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:26:23.007 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:18.261064+0000", 2024-09-13T11:26:23.007 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.33872685499999999 2024-09-13T11:26:23.008 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:23.008 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:23.008 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:26:23.008 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:18.261084+0000", 2024-09-13T11:26:23.008 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.9893000000000001e-05 2024-09-13T11:26:23.008 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:23.008 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:23.008 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:26:23.008 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:18.261880+0000", 2024-09-13T11:26:23.008 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00079579399999999995 2024-09-13T11:26:23.008 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:23.008 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:23.008 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:26:23.008 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:18.261897+0000", 2024-09-13T11:26:23.009 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.7730000000000001e-05 2024-09-13T11:26:23.009 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:23.009 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:23.009 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:26:23.009 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:18.262116+0000", 2024-09-13T11:26:23.009 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00021818099999999999 2024-09-13T11:26:23.009 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:23.009 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:26:23.009 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:23.009 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:23.009 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:26:23.009 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-09-13T11:26:23.011 INFO:tasks.ceph.osd.5.smithi092.stderr:2024-09-13T11:26:22.960+0000 7f6f02136640 -1 osd.5 204 handle_osd_map injecting map crc failure 2024-09-13T11:26:23.011 INFO:teuthology.orchestra.run.smithi001.stderr:set osd.15 primary-affinity to 0 (802) 2024-09-13T11:26:23.011 INFO:tasks.ceph.osd.14.smithi162.stderr:2024-09-13T11:26:23.007+0000 7f3df5903640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33154) UID: 0 2024-09-13T11:26:23.011 INFO:teuthology.orchestra.run.smithi001.stderr:nodeep-scrub is unset 2024-09-13T11:26:23.012 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-13T11:26:22.966+0000 7f6a5f1a6640 -1 osd.0 204 handle_osd_map injecting map crc failure 2024-09-13T11:26:23.013 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-13T11:26:23.085 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T11:26:23.111 INFO:tasks.ceph.osd.2.smithi162.stderr:2024-09-13T11:26:23.107+0000 7fe92b35a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26313) UID: 0 2024-09-13T11:26:23.155 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-09-13T11:26:23.155 INFO:teuthology.orchestra.run.smithi092.stdout: "ops": [ 2024-09-13T11:26:23.155 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:23.156 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:239983 3.4 3:23770a6a:::benchmark_data_smithi177_31116_object99680:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e205)", 2024-09-13T11:26:23.156 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:26:23.143084+0000", 2024-09-13T11:26:23.156 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 0.012947113999999999, 2024-09-13T11:26:23.156 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.012973432999999999, 2024-09-13T11:26:23.156 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:26:23.156 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:26:23.156 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "waiting for sub ops", 2024-09-13T11:26:23.156 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:26:23.156 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:26:23.156 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:23.156 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 239983 2024-09-13T11:26:23.157 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:23.157 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:26:23.157 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:23.157 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:26:23.157 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:23.143084+0000", 2024-09-13T11:26:23.157 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:23.157 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:23.157 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:23.157 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:26:23.157 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:23.143084+0000", 2024-09-13T11:26:23.157 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:23.157 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:23.157 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:23.158 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:26:23.158 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:23.143085+0000", 2024-09-13T11:26:23.158 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.342e-06 2024-09-13T11:26:23.158 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:23.158 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:23.158 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:26:23.158 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:23.143089+0000", 2024-09-13T11:26:23.158 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.5229999999999997e-06 2024-09-13T11:26:23.158 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:23.158 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:23.158 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:26:23.158 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:23.143098+0000", 2024-09-13T11:26:23.158 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.8540000000000003e-06 2024-09-13T11:26:23.158 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:23.159 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:23.159 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:26:23.159 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:23.143105+0000", 2024-09-13T11:26:23.159 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.5339999999999996e-06 2024-09-13T11:26:23.159 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:23.159 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:23.159 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:23.159 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:23.143169+0000", 2024-09-13T11:26:23.159 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.4315999999999999e-05 2024-09-13T11:26:23.159 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:23.159 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:23.159 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:26:23.159 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:23.143312+0000", 2024-09-13T11:26:23.159 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000142912 2024-09-13T11:26:23.160 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:23.160 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:23.160 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 12", 2024-09-13T11:26:23.160 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:23.143428+0000", 2024-09-13T11:26:23.160 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00011618899999999999 2024-09-13T11:26:23.160 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:23.160 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:23.160 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:26:23.160 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:23.145921+0000", 2024-09-13T11:26:23.160 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0024925709999999998 2024-09-13T11:26:23.160 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:23.160 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:23.160 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:23.161 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:23.161 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:23.161 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop(client.4665.0:239998 3.8 e205/117 3:10c88482:::benchmark_data_smithi177_31116_object99695:head v 205'6791, mlcod=205'6789)", 2024-09-13T11:26:23.161 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:26:23.153745+0000", 2024-09-13T11:26:23.161 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 0.002285624, 2024-09-13T11:26:23.161 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0023836819999999998, 2024-09-13T11:26:23.161 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:26:23.161 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:26:23.161 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-09-13T11:26:23.161 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:26:23.161 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:23.161 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:26:23.161 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:23.153745+0000", 2024-09-13T11:26:23.162 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:23.162 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:23.162 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:23.162 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:26:23.162 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:23.153745+0000", 2024-09-13T11:26:23.162 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:23.162 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:23.162 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:23.162 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:26:23.162 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:23.153747+0000", 2024-09-13T11:26:23.162 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.5710000000000001e-06 2024-09-13T11:26:23.163 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:23.163 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:23.163 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:26:23.163 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:23.153754+0000", 2024-09-13T11:26:23.163 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.3409999999999997e-06 2024-09-13T11:26:23.163 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:23.163 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:23.163 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:26:23.163 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:23.153785+0000", 2024-09-13T11:26:23.163 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.1133000000000002e-05 2024-09-13T11:26:23.163 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:23.163 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:23.164 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:26:23.164 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:23.153794+0000", 2024-09-13T11:26:23.164 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.7190000000000007e-06 2024-09-13T11:26:23.164 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:23.164 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:23.164 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:23.164 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:23.153856+0000", 2024-09-13T11:26:23.164 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.2559999999999997e-05 2024-09-13T11:26:23.164 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:23.164 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:23.164 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:26:23.164 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:23.153879+0000", 2024-09-13T11:26:23.164 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.2452000000000001e-05 2024-09-13T11:26:23.164 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:23.165 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:23.165 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:23.165 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:23.165 INFO:teuthology.orchestra.run.smithi092.stdout: ], 2024-09-13T11:26:23.165 INFO:teuthology.orchestra.run.smithi092.stdout: "num_ops": 2 2024-09-13T11:26:23.165 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-09-13T11:26:23.174 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-13T11:26:23.212 INFO:tasks.ceph.osd.10.smithi162.stderr:2024-09-13T11:26:23.207+0000 7fb59c9f0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26317) UID: 0 2024-09-13T11:26:23.260 INFO:teuthology.orchestra.run.smithi001.stdout:osd.0: {} 2024-09-13T11:26:23.260 INFO:teuthology.orchestra.run.smithi001.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T11:26:23.284 INFO:teuthology.orchestra.run.smithi001.stdout:osd.1: {} 2024-09-13T11:26:23.284 INFO:teuthology.orchestra.run.smithi001.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T11:26:23.301 INFO:teuthology.orchestra.run.smithi092.stdout:ERROR: (22) Invalid argument 2024-09-13T11:26:23.301 INFO:teuthology.orchestra.run.smithi092.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:26:23.308 INFO:teuthology.orchestra.run.smithi001.stdout:osd.2: {} 2024-09-13T11:26:23.308 INFO:teuthology.orchestra.run.smithi001.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T11:26:23.312 INFO:tasks.ceph.osd.14.smithi162.stderr:2024-09-13T11:26:23.307+0000 7f3df5903640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33154) UID: 0 2024-09-13T11:26:23.316 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-13T11:26:23.330 INFO:teuthology.orchestra.run.smithi001.stdout:osd.3: {} 2024-09-13T11:26:23.331 INFO:teuthology.orchestra.run.smithi001.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T11:26:23.355 INFO:teuthology.orchestra.run.smithi001.stdout:osd.4: {} 2024-09-13T11:26:23.355 INFO:teuthology.orchestra.run.smithi001.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T11:26:23.378 INFO:teuthology.orchestra.run.smithi001.stdout:osd.5: {} 2024-09-13T11:26:23.378 INFO:teuthology.orchestra.run.smithi001.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T11:26:23.405 INFO:teuthology.orchestra.run.smithi001.stdout:osd.6: {} 2024-09-13T11:26:23.405 INFO:teuthology.orchestra.run.smithi001.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T11:26:23.413 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-13T11:26:23.410+0000 7f6a68775640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26307) UID: 0 2024-09-13T11:26:23.428 INFO:teuthology.orchestra.run.smithi001.stdout:osd.7: {} 2024-09-13T11:26:23.428 INFO:teuthology.orchestra.run.smithi001.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T11:26:23.445 INFO:teuthology.orchestra.run.smithi092.stdout:ERROR: (22) Invalid argument 2024-09-13T11:26:23.445 INFO:teuthology.orchestra.run.smithi092.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:26:23.450 INFO:teuthology.orchestra.run.smithi001.stdout:osd.8: {} 2024-09-13T11:26:23.450 INFO:teuthology.orchestra.run.smithi001.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-13T11:26:23.461 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_ops_in_flight 2024-09-13T11:26:23.473 INFO:teuthology.orchestra.run.smithi001.stdout:osd.9: {} 2024-09-13T11:26:23.474 INFO:teuthology.orchestra.run.smithi001.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-13T11:26:23.505 INFO:teuthology.orchestra.run.smithi001.stdout:osd.10: {} 2024-09-13T11:26:23.514 INFO:tasks.ceph.osd.7.smithi177.stderr:2024-09-13T11:26:23.513+0000 7f897b51d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26575) UID: 0 2024-09-13T11:26:23.528 INFO:teuthology.orchestra.run.smithi001.stdout:osd.11: {} 2024-09-13T11:26:23.528 INFO:teuthology.orchestra.run.smithi001.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-13T11:26:23.548 INFO:teuthology.orchestra.run.smithi001.stdout:osd.12: {} 2024-09-13T11:26:23.549 INFO:teuthology.orchestra.run.smithi001.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-13T11:26:23.577 INFO:teuthology.orchestra.run.smithi001.stdout:osd.13: {} 2024-09-13T11:26:23.577 INFO:teuthology.orchestra.run.smithi001.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-13T11:26:23.590 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-13T11:26:23.591 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [], 2024-09-13T11:26:23.591 INFO:teuthology.orchestra.run.smithi162.stdout: "num_ops": 0 2024-09-13T11:26:23.591 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-13T11:26:23.602 INFO:teuthology.orchestra.run.smithi001.stdout:osd.14: {} 2024-09-13T11:26:23.602 INFO:teuthology.orchestra.run.smithi001.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-13T11:26:23.606 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_blocked_ops 2024-09-13T11:26:23.615 INFO:tasks.ceph.osd.13.smithi092.stderr:2024-09-13T11:26:23.612+0000 7f3e5afea640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26350) UID: 0 2024-09-13T11:26:23.628 INFO:teuthology.orchestra.run.smithi001.stdout:osd.15: {} 2024-09-13T11:26:23.628 INFO:teuthology.orchestra.run.smithi001.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-13T11:26:23.716 INFO:tasks.ceph.osd.8.smithi001.stderr:2024-09-13T11:26:23.714+0000 7fb9b48ae640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26312) UID: 0 2024-09-13T11:26:23.738 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-13T11:26:23.738 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:26:23.754 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_historic_ops 2024-09-13T11:26:23.817 INFO:tasks.ceph.osd.4.smithi001.stderr:2024-09-13T11:26:23.814+0000 7ff8e0218640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26310) UID: 0 2024-09-13T11:26:23.889 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-13T11:26:23.889 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:26:23.905 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-13T11:26:23.952 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-13T11:26:23.950+0000 7f6a68775640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26307) UID: 0 2024-09-13T11:26:23.970 INFO:tasks.ceph.osd.7.smithi177.stderr:2024-09-13T11:26:23.969+0000 7f896ff4a640 -1 osd.7 205 handle_osd_map injecting map crc failure 2024-09-13T11:26:24.011 INFO:tasks.ceph.osd.5.smithi092.stderr:2024-09-13T11:26:24.008+0000 7f6f02136640 -1 osd.5 205 handle_osd_map injecting map crc failure 2024-09-13T11:26:24.018 INFO:tasks.ceph.osd.5.smithi092.stderr:2024-09-13T11:26:24.016+0000 7f6f0b705640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26346) UID: 0 2024-09-13T11:26:24.043 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-13T11:26:24.043 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:26:24.062 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-13T11:26:24.119 INFO:tasks.ceph.osd.1.smithi092.stderr:2024-09-13T11:26:24.116+0000 7fe7789c5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33718) UID: 0 2024-09-13T11:26:24.220 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-09-13T11:26:24.215+0000 7f97befc9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26315) UID: 0 2024-09-13T11:26:24.265 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-13T11:26:24.265 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:26:24.288 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-13T11:26:24.321 INFO:tasks.ceph.osd.15.smithi177.stderr:2024-09-13T11:26:24.317+0000 7f9d6ce77640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26578) UID: 0 2024-09-13T11:26:24.421 INFO:tasks.ceph.osd.4.smithi001.stderr:2024-09-13T11:26:24.418+0000 7ff8e0218640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26310) UID: 0 2024-09-13T11:26:24.426 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-13T11:26:24.426 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:26:24.444 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-13T11:26:24.535 INFO:tasks.ceph.osd.4.smithi001.stderr:2024-09-13T11:26:24.534+0000 7ff8e0218640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26310) UID: 0 2024-09-13T11:26:24.614 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-13T11:26:24.614 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:26:24.623 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-09-13T11:26:24.619+0000 7f97befc9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26315) UID: 0 2024-09-13T11:26:24.627 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-13T11:26:24.723 INFO:tasks.ceph.osd.10.smithi162.stderr:2024-09-13T11:26:24.719+0000 7fb59c9f0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26317) UID: 0 2024-09-13T11:26:24.806 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-13T11:26:24.807 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:26:24.824 INFO:tasks.ceph.osd.5.smithi092.stderr:2024-09-13T11:26:24.820+0000 7f6f0b705640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26346) UID: 0 2024-09-13T11:26:24.826 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-13T11:26:24.925 INFO:tasks.ceph.osd.5.smithi092.stderr:2024-09-13T11:26:24.924+0000 7f6f0b705640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26346) UID: 0 2024-09-13T11:26:24.977 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-13T11:26:24.977 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:26:24.994 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-09-13T11:26:25.014 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-13T11:26:25.027 INFO:tasks.ceph.osd.9.smithi092.stderr:2024-09-13T11:26:25.024+0000 7ff5ab3d0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26348) UID: 0 2024-09-13T11:26:25.127 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-09-13T11:26:25.123+0000 7f97befc9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26315) UID: 0 2024-09-13T11:26:25.149 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-13T11:26:25.149 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:26:25.173 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-09-13T11:26:25.229 INFO:tasks.ceph.osd.3.smithi177.stderr:2024-09-13T11:26:25.225+0000 7f10adba4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26574) UID: 0 2024-09-13T11:26:25.329 INFO:tasks.ceph.osd.1.smithi092.stderr:2024-09-13T11:26:25.328+0000 7fe7789c5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33718) UID: 0 2024-09-13T11:26:25.351 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-13T11:26:25.352 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:26:25.370 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-09-13T11:26:25.430 INFO:tasks.ceph.osd.7.smithi177.stderr:2024-09-13T11:26:25.425+0000 7f897b51d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26575) UID: 0 2024-09-13T11:26:25.508 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-13T11:26:25.509 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:26:25.524 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_ops_in_flight 2024-09-13T11:26:25.533 INFO:tasks.ceph.osd.15.smithi177.stderr:2024-09-13T11:26:25.529+0000 7f9d6ce77640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26578) UID: 0 2024-09-13T11:26:25.634 INFO:tasks.ceph.osd.13.smithi092.stderr:2024-09-13T11:26:25.632+0000 7f3e5afea640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26350) UID: 0 2024-09-13T11:26:25.647 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T11:26:25.656 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-13T11:26:25.656 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:26:25.676 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_blocked_ops 2024-09-13T11:26:25.763 INFO:tasks.ceph.osd.12.smithi001.stderr:2024-09-13T11:26:25.762+0000 7f892c6e6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26314) UID: 0 2024-09-13T11:26:25.812 INFO:teuthology.orchestra.run.smithi001.stdout:osd.0: {} 2024-09-13T11:26:25.812 INFO:teuthology.orchestra.run.smithi001.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T11:26:25.835 INFO:tasks.ceph.osd.9.smithi092.stderr:2024-09-13T11:26:25.832+0000 7ff5ab3d0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26348) UID: 0 2024-09-13T11:26:25.837 INFO:teuthology.orchestra.run.smithi001.stdout:osd.1: {} 2024-09-13T11:26:25.838 INFO:teuthology.orchestra.run.smithi001.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T11:26:25.848 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-13T11:26:25.849 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:26:25.864 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_historic_ops 2024-09-13T11:26:25.864 INFO:teuthology.orchestra.run.smithi001.stdout:osd.2: {} 2024-09-13T11:26:25.864 INFO:teuthology.orchestra.run.smithi001.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T11:26:25.889 INFO:teuthology.orchestra.run.smithi001.stdout:osd.3: {} 2024-09-13T11:26:25.889 INFO:teuthology.orchestra.run.smithi001.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T11:26:25.910 INFO:teuthology.orchestra.run.smithi001.stdout:osd.4: {} 2024-09-13T11:26:25.910 INFO:teuthology.orchestra.run.smithi001.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T11:26:25.936 INFO:tasks.ceph.osd.9.smithi092.stderr:2024-09-13T11:26:25.936+0000 7ff5ab3d0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26348) UID: 0 2024-09-13T11:26:25.951 INFO:teuthology.orchestra.run.smithi001.stdout:osd.5: {} 2024-09-13T11:26:25.952 INFO:teuthology.orchestra.run.smithi001.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T11:26:25.983 INFO:teuthology.orchestra.run.smithi001.stdout:osd.6: {} 2024-09-13T11:26:25.984 INFO:teuthology.orchestra.run.smithi001.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T11:26:26.014 INFO:teuthology.orchestra.run.smithi001.stdout:osd.7: {} 2024-09-13T11:26:26.014 INFO:teuthology.orchestra.run.smithi001.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T11:26:26.037 INFO:tasks.ceph.osd.1.smithi092.stderr:2024-09-13T11:26:26.036+0000 7fe7789c5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33718) UID: 0 2024-09-13T11:26:26.048 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-13T11:26:26.048 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:26:26.051 INFO:teuthology.orchestra.run.smithi001.stdout:osd.8: {} 2024-09-13T11:26:26.051 INFO:teuthology.orchestra.run.smithi001.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-13T11:26:26.068 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-13T11:26:26.081 INFO:teuthology.orchestra.run.smithi001.stdout:osd.9: {} 2024-09-13T11:26:26.082 INFO:teuthology.orchestra.run.smithi001.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-13T11:26:26.111 INFO:teuthology.orchestra.run.smithi001.stdout:osd.10: {} 2024-09-13T11:26:26.111 INFO:teuthology.orchestra.run.smithi001.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-13T11:26:26.138 INFO:tasks.ceph.osd.5.smithi092.stderr:2024-09-13T11:26:26.136+0000 7f6f0b705640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26346) UID: 0 2024-09-13T11:26:26.140 INFO:teuthology.orchestra.run.smithi001.stdout:osd.11: {} 2024-09-13T11:26:26.140 INFO:teuthology.orchestra.run.smithi001.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-13T11:26:26.166 INFO:teuthology.orchestra.run.smithi001.stdout:osd.12: {} 2024-09-13T11:26:26.167 INFO:teuthology.orchestra.run.smithi001.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-13T11:26:26.170 INFO:teuthology.orchestra.run.smithi001.stderr:noscrub is set 2024-09-13T11:26:26.174 INFO:tasks.ceph.osd.5.smithi092.stderr:2024-09-13T11:26:26.172+0000 7f6f00132640 -1 osd.5 206 handle_osd_map injecting map crc failure 2024-09-13T11:26:26.180 INFO:tasks.ceph.osd.9.smithi092.stderr:2024-09-13T11:26:26.176+0000 7ff59bdf5640 -1 osd.9 206 handle_osd_map injecting map crc failure 2024-09-13T11:26:26.203 INFO:teuthology.orchestra.run.smithi001.stdout:osd.13: {} 2024-09-13T11:26:26.203 INFO:teuthology.orchestra.run.smithi001.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-13T11:26:26.218 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-09-13T11:26:26.218 INFO:teuthology.orchestra.run.smithi092.stdout: "ops": [ 2024-09-13T11:26:26.218 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:26.218 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:243058 3.27 3:e6c84232:::benchmark_data_smithi177_31116_object102755:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e207)", 2024-09-13T11:26:26.218 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:26:26.208917+0000", 2024-09-13T11:26:26.218 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 0.0096984740000000003, 2024-09-13T11:26:26.218 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0097195619999999993, 2024-09-13T11:26:26.218 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:26:26.218 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:26:26.219 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "waiting for sub ops", 2024-09-13T11:26:26.219 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:26:26.219 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:26:26.219 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:26.219 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 243058 2024-09-13T11:26:26.219 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:26.219 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:26:26.219 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:26.219 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:26:26.219 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:26.208917+0000", 2024-09-13T11:26:26.219 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:26.219 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:26.219 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:26.219 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:26:26.220 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:26.208917+0000", 2024-09-13T11:26:26.220 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:26.220 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:26.220 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:26.220 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:26:26.220 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:26.208919+0000", 2024-09-13T11:26:26.220 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.762e-06 2024-09-13T11:26:26.220 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:26.220 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:26.220 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:26:26.220 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:26.208924+0000", 2024-09-13T11:26:26.220 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.3569999999999997e-06 2024-09-13T11:26:26.220 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:26.220 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:26.221 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:26:26.221 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:26.208937+0000", 2024-09-13T11:26:26.221 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.2807e-05 2024-09-13T11:26:26.221 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:26.221 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:26.221 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:26:26.221 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:26.208944+0000", 2024-09-13T11:26:26.221 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.007e-06 2024-09-13T11:26:26.221 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:26.221 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:26.221 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:26.221 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:26.209056+0000", 2024-09-13T11:26:26.222 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00011196099999999999 2024-09-13T11:26:26.222 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:26.222 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:26.222 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:26:26.222 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:26.209190+0000", 2024-09-13T11:26:26.222 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000133874 2024-09-13T11:26:26.222 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:26.222 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:26.222 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 11", 2024-09-13T11:26:26.222 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:26.209306+0000", 2024-09-13T11:26:26.222 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000116124 2024-09-13T11:26:26.222 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:26.222 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:26.222 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:26:26.223 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:26.213317+0000", 2024-09-13T11:26:26.223 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0040108499999999998 2024-09-13T11:26:26.223 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:26.223 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:26.223 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:26.223 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:26.223 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:26.223 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:243061 3.27 3:e78e51db:::benchmark_data_smithi177_31116_object102758:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e207)", 2024-09-13T11:26:26.223 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:26:26.210821+0000", 2024-09-13T11:26:26.223 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 0.0077946430000000004, 2024-09-13T11:26:26.223 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0078628870000000007, 2024-09-13T11:26:26.223 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:26:26.223 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:26:26.223 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "waiting for sub ops", 2024-09-13T11:26:26.224 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:26:26.224 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:26:26.224 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:26.224 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 243061 2024-09-13T11:26:26.224 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:26.224 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:26:26.224 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:26.224 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:26:26.224 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:26.210821+0000", 2024-09-13T11:26:26.224 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:26.224 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:26.224 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:26.224 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:26:26.224 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:26.210821+0000", 2024-09-13T11:26:26.225 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:26.225 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:26.225 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:26.225 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:26:26.225 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:26.210822+0000", 2024-09-13T11:26:26.225 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.7209999999999999e-06 2024-09-13T11:26:26.225 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:26.225 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:26.225 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:26:26.225 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:26.210828+0000", 2024-09-13T11:26:26.225 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.4469999999999997e-06 2024-09-13T11:26:26.225 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:26.225 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:26.225 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:26:26.225 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:26.210841+0000", 2024-09-13T11:26:26.226 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.2996999999999999e-05 2024-09-13T11:26:26.226 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:26.226 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:26.226 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:26:26.226 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:26.210849+0000", 2024-09-13T11:26:26.226 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.9820000000000008e-06 2024-09-13T11:26:26.226 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:26.226 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:26.226 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:26.226 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:26.211384+0000", 2024-09-13T11:26:26.226 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00053510999999999999 2024-09-13T11:26:26.226 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:26.226 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:26.226 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:26:26.227 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:26.211509+0000", 2024-09-13T11:26:26.227 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000125108 2024-09-13T11:26:26.227 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:26.227 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:26.227 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 11", 2024-09-13T11:26:26.227 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:26.211611+0000", 2024-09-13T11:26:26.227 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000101592 2024-09-13T11:26:26.227 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:26.227 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:26.227 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:26:26.227 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:26.213708+0000", 2024-09-13T11:26:26.227 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0020978220000000001 2024-09-13T11:26:26.227 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:26.227 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:26.228 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:26.228 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:26.228 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:26.228 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:243015 3.1e 3:7db8d1c5:::benchmark_data_smithi177_31116_object102712:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e207)", 2024-09-13T11:26:26.228 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:26:26.178391+0000", 2024-09-13T11:26:26.228 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 0.040224479, 2024-09-13T11:26:26.228 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.040332352000000002, 2024-09-13T11:26:26.228 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:26:26.228 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:26:26.229 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "waiting for sub ops", 2024-09-13T11:26:26.229 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:26:26.229 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:26:26.229 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:26.229 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 243015 2024-09-13T11:26:26.229 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:26.229 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:26:26.229 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:26.229 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:26:26.229 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:26.178391+0000", 2024-09-13T11:26:26.229 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:26.229 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:26.229 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:26.229 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:26:26.230 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:26.178391+0000", 2024-09-13T11:26:26.230 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:26.230 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:26.230 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:26.230 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:26:26.230 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:26.178394+0000", 2024-09-13T11:26:26.230 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.728e-06 2024-09-13T11:26:26.230 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:26.230 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:26.230 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:26:26.231 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:26.178400+0000", 2024-09-13T11:26:26.231 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.8549999999999996e-06 2024-09-13T11:26:26.231 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:26.231 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:26.231 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:26:26.231 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:26.178415+0000", 2024-09-13T11:26:26.231 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.4955e-05 2024-09-13T11:26:26.231 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:26.231 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:26.231 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:26:26.231 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:26.178426+0000", 2024-09-13T11:26:26.231 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.075e-05 2024-09-13T11:26:26.231 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:26.231 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:26.232 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:26.232 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:26.178481+0000", 2024-09-13T11:26:26.232 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.4455e-05 2024-09-13T11:26:26.232 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:26.232 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:26.232 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op must wait for map", 2024-09-13T11:26:26.232 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:26.178485+0000", 2024-09-13T11:26:26.232 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.5050000000000001e-06 2024-09-13T11:26:26.232 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:26.232 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:26.232 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:26.232 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:26.179783+0000", 2024-09-13T11:26:26.232 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0012974620000000001 2024-09-13T11:26:26.232 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:26.233 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:26.233 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:26:26.233 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:26.179904+0000", 2024-09-13T11:26:26.233 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000120928 2024-09-13T11:26:26.233 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:26.233 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:26.233 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 4", 2024-09-13T11:26:26.233 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:26.179971+0000", 2024-09-13T11:26:26.233 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.7121999999999999e-05 2024-09-13T11:26:26.233 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:26.233 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:26.233 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:26:26.233 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:26.189204+0000", 2024-09-13T11:26:26.233 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0092334960000000008 2024-09-13T11:26:26.234 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:26.234 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:26.234 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:26.234 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:26.234 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:26.234 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:243015 3.1e 3:7db8d1c5:::benchmark_data_smithi177_31116_object102712:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e207)", 2024-09-13T11:26:26.234 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:26:26.191983+0000", 2024-09-13T11:26:26.234 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 0.026632051, 2024-09-13T11:26:26.234 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.026786099000000001, 2024-09-13T11:26:26.234 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:26:26.234 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:26:26.234 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "waiting for ondisk", 2024-09-13T11:26:26.234 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:26:26.234 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:26:26.235 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:26.235 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 243015 2024-09-13T11:26:26.235 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:26.235 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:26:26.235 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:26.235 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:26:26.235 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:26.191983+0000", 2024-09-13T11:26:26.235 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:26.235 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:26.235 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:26.235 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:26:26.235 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:26.191983+0000", 2024-09-13T11:26:26.235 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:26.235 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:26.236 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:26.236 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:26:26.236 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:26.191986+0000", 2024-09-13T11:26:26.236 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.4710000000000001e-06 2024-09-13T11:26:26.236 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:26.236 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:26.236 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:26:26.236 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:26.191995+0000", 2024-09-13T11:26:26.236 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.149e-06 2024-09-13T11:26:26.236 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:26.236 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:26.236 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:26:26.236 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:26.192011+0000", 2024-09-13T11:26:26.236 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.6226999999999999e-05 2024-09-13T11:26:26.236 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:26.237 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:26.237 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:26:26.237 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:26.192020+0000", 2024-09-13T11:26:26.237 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.1069999999999994e-06 2024-09-13T11:26:26.237 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:26.237 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:26.237 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:26.237 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:26.193608+0000", 2024-09-13T11:26:26.237 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0015877829999999999 2024-09-13T11:26:26.237 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:26.237 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:26.237 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for ondisk", 2024-09-13T11:26:26.237 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:26.193686+0000", 2024-09-13T11:26:26.237 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.7545000000000004e-05 2024-09-13T11:26:26.238 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:26.238 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:26.238 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:26.238 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:26.238 INFO:teuthology.orchestra.run.smithi092.stdout: ], 2024-09-13T11:26:26.238 INFO:teuthology.orchestra.run.smithi092.stdout: "num_ops": 4 2024-09-13T11:26:26.238 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-09-13T11:26:26.239 INFO:teuthology.orchestra.run.smithi001.stdout:osd.14: {} 2024-09-13T11:26:26.239 INFO:teuthology.orchestra.run.smithi001.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-13T11:26:26.240 INFO:tasks.ceph.osd.12.smithi001.stderr:2024-09-13T11:26:26.234+0000 7f892c6e6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26314) UID: 0 2024-09-13T11:26:26.240 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-13T11:26:26.252 INFO:teuthology.orchestra.run.smithi001.stdout:osd.15: {} 2024-09-13T11:26:26.253 INFO:teuthology.orchestra.run.smithi001.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-13T11:26:26.340 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-09-13T11:26:26.335+0000 7f97befc9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26315) UID: 0 2024-09-13T11:26:26.419 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-09-13T11:26:26.419 INFO:teuthology.orchestra.run.smithi092.stdout: "ops": [], 2024-09-13T11:26:26.419 INFO:teuthology.orchestra.run.smithi092.stdout: "complaint_time": 30, 2024-09-13T11:26:26.419 INFO:teuthology.orchestra.run.smithi092.stdout: "num_blocked_ops": 0 2024-09-13T11:26:26.419 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-09-13T11:26:26.437 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-13T11:26:26.440 INFO:tasks.ceph.osd.3.smithi177.stderr:2024-09-13T11:26:26.437+0000 7f10adba4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26574) UID: 0 2024-09-13T11:26:26.541 INFO:tasks.ceph.osd.13.smithi092.stderr:2024-09-13T11:26:26.540+0000 7f3e5afea640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26350) UID: 0 2024-09-13T11:26:26.575 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-09-13T11:26:26.575 INFO:teuthology.orchestra.run.smithi092.stdout: "size": 20, 2024-09-13T11:26:26.575 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 600, 2024-09-13T11:26:26.575 INFO:teuthology.orchestra.run.smithi092.stdout: "ops": [ 2024-09-13T11:26:26.575 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:26.575 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "MOSDPGPush(3.23 83/78 [PushOp(3:c61f6bab:::benchmark_data_smithi177_31116_object29479:head, version: 59'1860, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:c61f6bab:::benchmark_data_smithi177_31116_object29479:head@59'1860, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:26:26.575 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:40.770864+0000", 2024-09-13T11:26:26.575 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 165.80264672199999, 2024-09-13T11:26:26.575 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.31229662800000002, 2024-09-13T11:26:26.575 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:26:26.575 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:26:26.576 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-09-13T11:26:26.576 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:26:26.576 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:26.576 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:26:26.576 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:40.770864+0000", 2024-09-13T11:26:26.576 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:26.576 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:26.576 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:26.576 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:26:26.576 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:40.770864+0000", 2024-09-13T11:26:26.576 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:26.576 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:26.576 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:26.577 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:26:26.577 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:40.770865+0000", 2024-09-13T11:26:26.577 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.288e-06 2024-09-13T11:26:26.577 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:26.577 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:26.577 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:26:26.577 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:40.770955+0000", 2024-09-13T11:26:26.577 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.9871999999999996e-05 2024-09-13T11:26:26.577 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:26.577 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:26.577 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:26:26.577 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:41.082105+0000", 2024-09-13T11:26:26.577 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.31114963400000001 2024-09-13T11:26:26.577 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:26.578 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:26.578 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:26:26.578 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:41.082136+0000", 2024-09-13T11:26:26.578 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.0972000000000001e-05 2024-09-13T11:26:26.578 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:26.578 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:26.578 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:26.578 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:41.082404+0000", 2024-09-13T11:26:26.578 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00026844800000000002 2024-09-13T11:26:26.578 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:26.578 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:26.578 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:26:26.578 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:41.082429+0000", 2024-09-13T11:26:26.578 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.5021e-05 2024-09-13T11:26:26.579 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:26.579 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:26.579 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:26:26.579 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:41.083161+0000", 2024-09-13T11:26:26.579 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000731393 2024-09-13T11:26:26.579 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:26.579 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:26.579 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:26.579 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:26.579 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:26.579 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:110289 3.29 3.27287969 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e119)", 2024-09-13T11:26:26.579 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:24:25.774634+0000", 2024-09-13T11:26:26.579 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 120.798876647, 2024-09-13T11:26:26.580 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.31721633999999999, 2024-09-13T11:26:26.580 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:26:26.580 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:26:26.580 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "reached pg", 2024-09-13T11:26:26.580 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:26:26.580 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:26:26.580 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:26.580 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 110289 2024-09-13T11:26:26.580 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:26.580 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:26:26.580 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:26.580 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:26:26.580 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:25.774634+0000", 2024-09-13T11:26:26.580 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:26.581 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:26.581 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:26.581 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:26:26.581 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:25.774634+0000", 2024-09-13T11:26:26.581 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:26.581 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:26.581 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:26.581 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:26:26.581 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:25.774639+0000", 2024-09-13T11:26:26.581 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.6029999999999998e-06 2024-09-13T11:26:26.581 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:26.581 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:26.581 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:26:26.582 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:25.774695+0000", 2024-09-13T11:26:26.582 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.6694999999999998e-05 2024-09-13T11:26:26.582 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:26.582 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:26.582 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:26:26.582 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:25.774728+0000", 2024-09-13T11:26:26.582 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.2502000000000002e-05 2024-09-13T11:26:26.582 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:26.582 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:26.582 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:26:26.582 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:25.774750+0000", 2024-09-13T11:26:26.582 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.2002999999999999e-05 2024-09-13T11:26:26.582 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:26.583 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:26.583 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:26.583 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:25.774942+0000", 2024-09-13T11:26:26.583 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000192265 2024-09-13T11:26:26.583 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:26.583 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:26.583 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-09-13T11:26:26.583 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:25.774960+0000", 2024-09-13T11:26:26.583 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.8275999999999999e-05 2024-09-13T11:26:26.583 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:26.583 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:26.583 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:26.583 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:26.091821+0000", 2024-09-13T11:26:26.583 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.31686065000000002 2024-09-13T11:26:26.584 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:26.584 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:26.584 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:26:26.584 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:26.091850+0000", 2024-09-13T11:26:26.584 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.9346e-05 2024-09-13T11:26:26.584 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:26.584 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:26.584 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:26.584 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:26.584 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:26.584 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:138370 3.23 3:c5730cd2:::benchmark_data_smithi177_31116_object138369:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e140)", 2024-09-13T11:26:26.584 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:24:54.415176+0000", 2024-09-13T11:26:26.584 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 92.158334550999996, 2024-09-13T11:26:26.584 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.24549895499999999, 2024-09-13T11:26:26.585 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:26:26.585 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:26:26.585 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:26.585 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:26:26.585 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:26:26.585 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:26.585 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 138370 2024-09-13T11:26:26.585 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:26.585 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:26:26.585 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:26.585 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:26:26.585 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:54.415176+0000", 2024-09-13T11:26:26.585 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:26.585 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:26.586 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:26.586 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:26:26.586 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:54.415176+0000", 2024-09-13T11:26:26.586 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:26.586 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:26.586 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:26.586 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:26:26.586 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:54.415180+0000", 2024-09-13T11:26:26.586 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.1670000000000001e-06 2024-09-13T11:26:26.586 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:26.586 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:26.586 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:26:26.586 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:54.415269+0000", 2024-09-13T11:26:26.586 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.8659000000000006e-05 2024-09-13T11:26:26.587 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:26.587 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:26.587 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:26:26.587 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:54.415292+0000", 2024-09-13T11:26:26.587 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.3152000000000001e-05 2024-09-13T11:26:26.587 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:26.587 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:26.587 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:26:26.587 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:54.415304+0000", 2024-09-13T11:26:26.587 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.1841999999999999e-05 2024-09-13T11:26:26.587 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:26.587 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:26.587 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:26.588 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:54.415432+0000", 2024-09-13T11:26:26.588 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00012802000000000001 2024-09-13T11:26:26.588 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:26.588 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:26.588 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:26:26.588 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:54.415710+0000", 2024-09-13T11:26:26.588 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00027840299999999998 2024-09-13T11:26:26.588 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:26.588 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:26.588 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 9", 2024-09-13T11:26:26.588 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:54.415950+0000", 2024-09-13T11:26:26.588 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00023928299999999999 2024-09-13T11:26:26.588 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:26.589 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:26.589 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:26:26.589 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:54.419265+0000", 2024-09-13T11:26:26.589 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0033150440000000001 2024-09-13T11:26:26.589 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:26.589 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:26.589 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:26.589 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:54.660448+0000", 2024-09-13T11:26:26.589 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.24118348100000001 2024-09-13T11:26:26.589 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:26.589 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:26.589 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:26:26.589 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:54.660571+0000", 2024-09-13T11:26:26.589 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00012332900000000001 2024-09-13T11:26:26.590 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:26.590 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:26.590 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:26:26.590 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:54.660675+0000", 2024-09-13T11:26:26.590 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000103575 2024-09-13T11:26:26.590 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:26.590 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:26.590 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:26.590 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:26.590 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:26.590 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:140742 3.1e 3:79c694a0:::benchmark_data_smithi177_31116_object439:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e144)", 2024-09-13T11:26:26.590 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:24:58.321262+0000", 2024-09-13T11:26:26.590 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 88.252248133999998, 2024-09-13T11:26:26.590 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.23727013499999999, 2024-09-13T11:26:26.590 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:26:26.591 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:26:26.591 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:26.591 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:26:26.591 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:26:26.591 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:26.591 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 140742 2024-09-13T11:26:26.591 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:26.591 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:26:26.591 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:26.591 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:26:26.591 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:58.321262+0000", 2024-09-13T11:26:26.591 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:26.591 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:26.591 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:26.592 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:26:26.592 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:58.321262+0000", 2024-09-13T11:26:26.592 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:26.592 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:26.592 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:26.592 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:26:26.592 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:58.321264+0000", 2024-09-13T11:26:26.592 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.581e-06 2024-09-13T11:26:26.592 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:26.592 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:26.592 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:26:26.592 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:58.321272+0000", 2024-09-13T11:26:26.592 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.2179999999999999e-06 2024-09-13T11:26:26.592 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:26.593 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:26.593 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:26:26.593 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:58.321282+0000", 2024-09-13T11:26:26.593 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0173000000000001e-05 2024-09-13T11:26:26.593 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:26.593 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:26.593 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:26:26.593 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:58.321290+0000", 2024-09-13T11:26:26.593 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.943e-06 2024-09-13T11:26:26.593 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:26.593 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:26.593 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:26.593 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:58.321355+0000", 2024-09-13T11:26:26.593 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.4461000000000005e-05 2024-09-13T11:26:26.594 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:26.594 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:26.594 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:26:26.594 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:58.321450+0000", 2024-09-13T11:26:26.594 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.4769e-05 2024-09-13T11:26:26.594 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:26.594 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:26.594 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 4", 2024-09-13T11:26:26.594 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:58.321516+0000", 2024-09-13T11:26:26.594 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.5998000000000001e-05 2024-09-13T11:26:26.594 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:26.594 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:26.594 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:26:26.594 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:58.324045+0000", 2024-09-13T11:26:26.595 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0025290579999999998 2024-09-13T11:26:26.595 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:26.595 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:26.595 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:26.595 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:58.558244+0000", 2024-09-13T11:26:26.595 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.234199455 2024-09-13T11:26:26.595 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:26.595 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:26.595 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:26:26.595 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:58.558401+0000", 2024-09-13T11:26:26.595 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00015700800000000001 2024-09-13T11:26:26.595 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:26.595 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:26.595 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:26:26.596 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:58.558533+0000", 2024-09-13T11:26:26.596 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000131471 2024-09-13T11:26:26.596 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:26.596 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:26.596 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:26.596 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:26.596 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:26.596 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:140796 3.1e 3:7987a652:::benchmark_data_smithi177_31116_object493:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e144)", 2024-09-13T11:26:26.596 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:24:58.552900+0000", 2024-09-13T11:26:26.596 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 88.020610598999994, 2024-09-13T11:26:26.596 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.254397398, 2024-09-13T11:26:26.596 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:26:26.596 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:26:26.596 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:26.597 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:26:26.597 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:26:26.597 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:26.597 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 140796 2024-09-13T11:26:26.597 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:26.597 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:26:26.597 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:26.597 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:26:26.597 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:58.552900+0000", 2024-09-13T11:26:26.597 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:26.597 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:26.597 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:26.597 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:26:26.597 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:58.552900+0000", 2024-09-13T11:26:26.598 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:26.598 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:26.598 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:26.598 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:26:26.598 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:58.552904+0000", 2024-09-13T11:26:26.598 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.76e-06 2024-09-13T11:26:26.598 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:26.598 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:26.598 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:26:26.598 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:58.552915+0000", 2024-09-13T11:26:26.598 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0763e-05 2024-09-13T11:26:26.598 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:26.598 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:26.598 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:26:26.598 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:58.552943+0000", 2024-09-13T11:26:26.599 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.8469000000000001e-05 2024-09-13T11:26:26.599 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:26.599 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:26.599 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:26:26.599 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:58.552959+0000", 2024-09-13T11:26:26.599 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.5549999999999999e-05 2024-09-13T11:26:26.599 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:26.599 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:26.599 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:26.599 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:58.553106+0000", 2024-09-13T11:26:26.599 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00014729500000000001 2024-09-13T11:26:26.599 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:26.599 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:26.599 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:26:26.600 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:58.553348+0000", 2024-09-13T11:26:26.600 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000242353 2024-09-13T11:26:26.600 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:26.600 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:26.600 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 4", 2024-09-13T11:26:26.600 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:58.553556+0000", 2024-09-13T11:26:26.600 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00020765700000000001 2024-09-13T11:26:26.600 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:26.600 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:26.600 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:26:26.600 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:58.557140+0000", 2024-09-13T11:26:26.600 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0035844840000000002 2024-09-13T11:26:26.600 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:26.601 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:26.601 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:26.601 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:58.807126+0000", 2024-09-13T11:26:26.601 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.24998542000000001 2024-09-13T11:26:26.601 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:26.601 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:26.601 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:26:26.601 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:58.807214+0000", 2024-09-13T11:26:26.601 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.8561999999999997e-05 2024-09-13T11:26:26.601 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:26.601 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:26.601 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:26:26.601 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:58.807297+0000", 2024-09-13T11:26:26.601 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.3084999999999995e-05 2024-09-13T11:26:26.601 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:26.602 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:26.602 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:26.602 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:26.602 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:26.602 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop(client.4665.0:162073 3.1f e156/122 3:fd1a00ae:::benchmark_data_smithi177_31116_object21770:head v 156'6684, mlcod=156'6679)", 2024-09-13T11:26:26.602 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:25:15.455533+0000", 2024-09-13T11:26:26.602 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 71.117977230999998, 2024-09-13T11:26:26.602 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.24315087499999999, 2024-09-13T11:26:26.602 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:26:26.602 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:26:26.603 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:26.603 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:26:26.603 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:26.603 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:26:26.603 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:15.455533+0000", 2024-09-13T11:26:26.603 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:26.603 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:26.603 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:26.603 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:26:26.603 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:15.455533+0000", 2024-09-13T11:26:26.603 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:26.603 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:26.603 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:26.603 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:26:26.604 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:15.455535+0000", 2024-09-13T11:26:26.604 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.42e-06 2024-09-13T11:26:26.604 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:26.604 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:26.604 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:26:26.604 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:15.455542+0000", 2024-09-13T11:26:26.604 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.9820000000000002e-06 2024-09-13T11:26:26.604 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:26.604 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:26.604 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:26:26.604 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:15.696032+0000", 2024-09-13T11:26:26.604 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.240490483 2024-09-13T11:26:26.604 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:26.605 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:26.605 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:26:26.605 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:15.696039+0000", 2024-09-13T11:26:26.605 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.832e-06 2024-09-13T11:26:26.605 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:26.605 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:26.605 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:26.605 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:15.696098+0000", 2024-09-13T11:26:26.605 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.8901999999999999e-05 2024-09-13T11:26:26.605 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:26.605 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:26.605 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:26:26.605 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:15.696115+0000", 2024-09-13T11:26:26.606 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.7479999999999999e-05 2024-09-13T11:26:26.606 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:26.606 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:26.606 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:26:26.606 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:15.698637+0000", 2024-09-13T11:26:26.606 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0025215889999999999 2024-09-13T11:26:26.606 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:26.606 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:26.606 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:26:26.606 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:15.698684+0000", 2024-09-13T11:26:26.606 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.7187000000000001e-05 2024-09-13T11:26:26.606 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:26.606 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:26.606 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:26.607 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:26.607 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:26.607 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop(client.4665.0:162102 3.1f e156/122 3:fc0cda6c:::benchmark_data_smithi177_31116_object21799:head v 156'6685, mlcod=156'6680)", 2024-09-13T11:26:26.607 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:25:15.477994+0000", 2024-09-13T11:26:26.607 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 71.095516430000004, 2024-09-13T11:26:26.607 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.25110996400000002, 2024-09-13T11:26:26.607 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:26:26.607 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:26:26.607 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:26.607 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:26:26.607 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:26.607 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:26:26.607 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:15.477994+0000", 2024-09-13T11:26:26.608 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:26.608 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:26.608 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:26.608 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:26:26.608 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:15.477994+0000", 2024-09-13T11:26:26.608 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:26.608 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:26.608 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:26.608 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:26:26.608 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:15.477995+0000", 2024-09-13T11:26:26.608 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0210000000000001e-06 2024-09-13T11:26:26.608 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:26.608 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:26.609 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:26:26.609 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:15.478001+0000", 2024-09-13T11:26:26.609 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.0020000000000003e-06 2024-09-13T11:26:26.609 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:26.609 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:26.609 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:26:26.609 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:15.726384+0000", 2024-09-13T11:26:26.609 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.248382609 2024-09-13T11:26:26.609 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:26.609 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:26.609 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:26:26.609 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:15.726390+0000", 2024-09-13T11:26:26.609 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.1029999999999998e-06 2024-09-13T11:26:26.610 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:26.610 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:26.610 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:26.610 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:15.727107+0000", 2024-09-13T11:26:26.610 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000717525 2024-09-13T11:26:26.610 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:26.610 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:26.610 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:26:26.610 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:15.727130+0000", 2024-09-13T11:26:26.610 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.2572999999999999e-05 2024-09-13T11:26:26.610 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:26.610 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:26.610 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:26:26.610 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:15.729066+0000", 2024-09-13T11:26:26.611 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0019362170000000001 2024-09-13T11:26:26.611 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:26.611 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:26.611 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:26:26.611 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:15.729104+0000", 2024-09-13T11:26:26.611 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.7914000000000003e-05 2024-09-13T11:26:26.611 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:26.611 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:26.611 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:26.611 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:26.611 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:26.611 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "MOSDPGPush(3.28 168/120 [PushOp(3:14eb8ed9:::benchmark_data_smithi177_31116_object87538:head, version: 110'4071, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:14eb8ed9:::benchmark_data_smithi177_31116_object87538:head@110'4071, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:26:26.611 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:25:27.274802+0000", 2024-09-13T11:26:26.611 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 59.298708882, 2024-09-13T11:26:26.612 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.369268239, 2024-09-13T11:26:26.612 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:26:26.612 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:26:26.612 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-09-13T11:26:26.612 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:26:26.612 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:26.612 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:26:26.612 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:27.274802+0000", 2024-09-13T11:26:26.612 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:26.612 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:26.612 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:26.612 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:26:26.612 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:27.274802+0000", 2024-09-13T11:26:26.613 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:26.613 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:26.613 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:26.613 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:26:26.613 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:27.274803+0000", 2024-09-13T11:26:26.613 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.7579999999999999e-06 2024-09-13T11:26:26.613 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:26.613 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:26.613 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:26:26.613 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:27.274831+0000", 2024-09-13T11:26:26.613 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.7464e-05 2024-09-13T11:26:26.613 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:26.613 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:26.613 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:26:26.614 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:27.641355+0000", 2024-09-13T11:26:26.614 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.36652425199999999 2024-09-13T11:26:26.614 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:26.614 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:26.614 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:26:26.614 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:27.641385+0000", 2024-09-13T11:26:26.614 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.9433000000000001e-05 2024-09-13T11:26:26.614 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:26.614 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:26.614 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:26.614 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:27.641611+0000", 2024-09-13T11:26:26.614 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00022646299999999999 2024-09-13T11:26:26.614 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:26.614 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:26.614 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:26:26.615 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:27.641638+0000", 2024-09-13T11:26:26.615 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.6482000000000002e-05 2024-09-13T11:26:26.615 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:26.615 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:26.615 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:26:26.615 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:27.644070+0000", 2024-09-13T11:26:26.615 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0024323869999999998 2024-09-13T11:26:26.615 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:26.615 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:26.615 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:26.615 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:26.615 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:26.615 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "MOSDPGPush(3.2d 174/120 [PushOp(3:b42ee771:::benchmark_data_smithi177_31116_object109317:head, version: 118'5087, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:b42ee771:::benchmark_data_smithi177_31116_object109317:head@118'5087, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:26:26.615 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:25:36.282270+0000", 2024-09-13T11:26:26.616 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 50.291240264999999, 2024-09-13T11:26:26.616 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.26768705100000001, 2024-09-13T11:26:26.616 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:26:26.616 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:26:26.616 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-09-13T11:26:26.616 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:26:26.616 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:26.616 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:26:26.616 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:36.282270+0000", 2024-09-13T11:26:26.616 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:26.616 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:26.616 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:26.616 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:26:26.617 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:36.282270+0000", 2024-09-13T11:26:26.617 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:26.617 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:26.617 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:26.617 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:26:26.617 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:36.282272+0000", 2024-09-13T11:26:26.617 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.832e-06 2024-09-13T11:26:26.617 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:26.617 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:26.617 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:26:26.617 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:36.282337+0000", 2024-09-13T11:26:26.617 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.4834999999999998e-05 2024-09-13T11:26:26.617 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:26.617 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:26.618 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:26:26.618 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:36.547263+0000", 2024-09-13T11:26:26.618 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.26492590599999999 2024-09-13T11:26:26.618 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:26.618 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:26.618 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:26:26.618 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:36.547288+0000", 2024-09-13T11:26:26.618 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.4661999999999999e-05 2024-09-13T11:26:26.618 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:26.618 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:26.618 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:26.618 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:36.547493+0000", 2024-09-13T11:26:26.618 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00020533500000000001 2024-09-13T11:26:26.618 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:26.619 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:26.619 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:26:26.619 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:36.547515+0000", 2024-09-13T11:26:26.619 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.2124999999999999e-05 2024-09-13T11:26:26.619 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:26.619 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:26.619 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:26:26.619 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:36.549957+0000", 2024-09-13T11:26:26.619 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0024423560000000001 2024-09-13T11:26:26.619 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:26.619 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:26.619 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:26.619 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:26.619 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:26.620 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:191021 3.23 3:c4df18dd:::benchmark_data_smithi177_31116_object50718:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e175)", 2024-09-13T11:26:26.620 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:25:38.460411+0000", 2024-09-13T11:26:26.620 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 48.113099456999997, 2024-09-13T11:26:26.620 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.40138001400000001, 2024-09-13T11:26:26.620 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:26:26.620 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:26:26.620 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:26.620 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:26:26.620 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:26:26.620 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:26.620 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 191021 2024-09-13T11:26:26.620 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:26.620 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:26:26.621 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:26.621 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:26:26.621 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:38.460411+0000", 2024-09-13T11:26:26.621 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:26.621 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:26.621 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:26.621 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:26:26.621 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:38.460411+0000", 2024-09-13T11:26:26.621 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:26.621 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:26.621 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:26.621 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:26:26.621 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:38.460413+0000", 2024-09-13T11:26:26.621 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.075e-06 2024-09-13T11:26:26.622 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:26.622 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:26.622 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:26:26.622 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:38.460419+0000", 2024-09-13T11:26:26.622 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.6869999999999999e-06 2024-09-13T11:26:26.622 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:26.622 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:26.622 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:26:26.622 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:38.460431+0000", 2024-09-13T11:26:26.622 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.2451e-05 2024-09-13T11:26:26.622 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:26.622 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:26.622 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:26:26.622 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:38.460440+0000", 2024-09-13T11:26:26.623 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.5140000000000001e-06 2024-09-13T11:26:26.623 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:26.623 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:26.623 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:26.623 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:38.460501+0000", 2024-09-13T11:26:26.623 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.0772e-05 2024-09-13T11:26:26.623 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:26.623 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:26.623 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:26:26.623 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:38.460632+0000", 2024-09-13T11:26:26.623 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00013175200000000001 2024-09-13T11:26:26.623 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:26.623 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:26.623 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 9", 2024-09-13T11:26:26.624 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:38.460731+0000", 2024-09-13T11:26:26.624 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.8647000000000006e-05 2024-09-13T11:26:26.624 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:26.624 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:26.624 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:26:26.624 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:38.462322+0000", 2024-09-13T11:26:26.624 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.001590658 2024-09-13T11:26:26.624 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:26.624 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:26.624 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:26.624 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:38.861634+0000", 2024-09-13T11:26:26.624 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.39931206499999999 2024-09-13T11:26:26.624 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:26.624 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:26.625 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:26:26.625 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:38.861719+0000", 2024-09-13T11:26:26.625 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.4968000000000004e-05 2024-09-13T11:26:26.625 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:26.625 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:26.625 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:26:26.625 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:38.861791+0000", 2024-09-13T11:26:26.625 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.2424999999999999e-05 2024-09-13T11:26:26.625 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:26.625 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:26.625 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:26.625 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:26.625 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:26.625 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop_reply(client.4665.0:191021 3.23 e175/90 ondisk, result = 0)", 2024-09-13T11:26:26.626 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:25:38.462483+0000", 2024-09-13T11:26:26.626 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 48.111028034999997, 2024-09-13T11:26:26.626 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.39932300900000001, 2024-09-13T11:26:26.626 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:26:26.626 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:26:26.626 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-09-13T11:26:26.626 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:26:26.626 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:26.626 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:26:26.626 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:38.462483+0000", 2024-09-13T11:26:26.626 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:26.626 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:26.626 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:26.627 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:26:26.627 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:38.462483+0000", 2024-09-13T11:26:26.627 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:26.627 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:26.627 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:26.627 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:26:26.627 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:38.462484+0000", 2024-09-13T11:26:26.627 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.1990000000000001e-06 2024-09-13T11:26:26.627 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:26.627 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:26.627 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:26:26.627 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:38.462488+0000", 2024-09-13T11:26:26.627 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.985e-06 2024-09-13T11:26:26.627 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:26.628 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:26.628 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:26:26.628 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:38.861475+0000", 2024-09-13T11:26:26.628 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.39898754800000003 2024-09-13T11:26:26.628 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:26.628 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:26.628 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:26:26.628 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:38.861488+0000", 2024-09-13T11:26:26.628 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.3037000000000001e-05 2024-09-13T11:26:26.628 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:26.628 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:26.628 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:26.628 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:38.861607+0000", 2024-09-13T11:26:26.629 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00011835 2024-09-13T11:26:26.629 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:26.629 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:26.629 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:26:26.629 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:38.861624+0000", 2024-09-13T11:26:26.629 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.7730000000000001e-05 2024-09-13T11:26:26.629 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:26.629 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:26.629 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:26:26.629 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:38.861806+0000", 2024-09-13T11:26:26.629 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00018116000000000001 2024-09-13T11:26:26.629 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:26.629 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:26.629 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:26.630 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:26.630 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:26.630 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:209500 3.27 3:e6941997:::benchmark_data_smithi177_31116_object69197:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e185)", 2024-09-13T11:26:26.630 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:25:53.962062+0000", 2024-09-13T11:26:26.630 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 32.611448291000002, 2024-09-13T11:26:26.630 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.26007328699999999, 2024-09-13T11:26:26.630 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:26:26.630 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:26:26.630 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:26.630 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:26:26.630 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:26:26.630 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:26.630 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 209500 2024-09-13T11:26:26.630 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:26.631 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:26:26.631 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:26.631 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:26:26.631 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.962062+0000", 2024-09-13T11:26:26.631 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:26.631 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:26.631 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:26.631 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:26:26.631 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.962062+0000", 2024-09-13T11:26:26.631 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:26.631 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:26.631 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:26.631 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:26:26.631 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.962065+0000", 2024-09-13T11:26:26.632 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.9019999999999999e-06 2024-09-13T11:26:26.632 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:26.632 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:26.632 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:26:26.632 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.962074+0000", 2024-09-13T11:26:26.632 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.6030000000000005e-06 2024-09-13T11:26:26.632 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:26.632 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:26.632 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:26:26.632 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.962098+0000", 2024-09-13T11:26:26.632 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.4076000000000001e-05 2024-09-13T11:26:26.632 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:26.632 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:26.632 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:26:26.633 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.962114+0000", 2024-09-13T11:26:26.633 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.5820000000000001e-05 2024-09-13T11:26:26.633 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:26.633 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:26.633 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:26.633 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.962280+0000", 2024-09-13T11:26:26.633 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00016648300000000001 2024-09-13T11:26:26.633 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:26.633 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:26.633 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:26:26.633 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.962598+0000", 2024-09-13T11:26:26.633 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00031824000000000002 2024-09-13T11:26:26.633 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:26.633 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:26.634 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 11", 2024-09-13T11:26:26.634 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.962906+0000", 2024-09-13T11:26:26.634 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00030721699999999998 2024-09-13T11:26:26.634 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:26.634 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:26.634 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:26:26.634 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.965943+0000", 2024-09-13T11:26:26.634 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0030373010000000001 2024-09-13T11:26:26.634 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:26.634 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:26.634 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:26.634 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:54.222003+0000", 2024-09-13T11:26:26.634 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.25605972900000001 2024-09-13T11:26:26.634 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:26.635 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:26.635 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:26:26.635 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:54.222081+0000", 2024-09-13T11:26:26.635 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.8442000000000006e-05 2024-09-13T11:26:26.635 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:26.635 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:26.635 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:26:26.635 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:54.222136+0000", 2024-09-13T11:26:26.635 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.4474000000000001e-05 2024-09-13T11:26:26.635 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:26.635 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:26.635 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:26.635 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:26.635 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:26.636 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:209869 3.27 3:e6b1fcc5:::benchmark_data_smithi177_31116_object69566:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e185)", 2024-09-13T11:26:26.636 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:25:54.300263+0000", 2024-09-13T11:26:26.636 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 32.273247138000002, 2024-09-13T11:26:26.636 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.30266981599999998, 2024-09-13T11:26:26.636 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:26:26.636 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:26:26.636 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:26.636 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:26:26.636 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:26:26.636 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:26.636 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 209869 2024-09-13T11:26:26.636 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:26.636 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:26:26.636 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:26.637 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:26:26.637 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:54.300263+0000", 2024-09-13T11:26:26.637 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:26.637 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:26.637 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:26.637 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:26:26.637 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:54.300263+0000", 2024-09-13T11:26:26.637 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:26.637 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:26.637 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:26.637 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:26:26.637 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:54.300267+0000", 2024-09-13T11:26:26.637 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.2200000000000001e-06 2024-09-13T11:26:26.637 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:26.638 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:26.638 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:26:26.638 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:54.300276+0000", 2024-09-13T11:26:26.638 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.3570000000000004e-06 2024-09-13T11:26:26.638 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:26.638 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:26.638 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:26:26.638 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:54.300301+0000", 2024-09-13T11:26:26.638 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.5191e-05 2024-09-13T11:26:26.638 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:26.638 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:26.638 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:26:26.638 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:54.300321+0000", 2024-09-13T11:26:26.638 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.9967999999999999e-05 2024-09-13T11:26:26.639 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:26.639 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:26.639 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:26.639 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:54.300427+0000", 2024-09-13T11:26:26.639 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000105781 2024-09-13T11:26:26.639 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:26.639 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:26.639 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:26:26.639 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:54.300590+0000", 2024-09-13T11:26:26.639 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00016341000000000001 2024-09-13T11:26:26.639 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:26.639 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:26.639 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 11", 2024-09-13T11:26:26.639 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:54.300724+0000", 2024-09-13T11:26:26.640 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000133323 2024-09-13T11:26:26.640 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:26.640 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:26.640 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:26:26.640 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:54.303373+0000", 2024-09-13T11:26:26.640 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0026488050000000002 2024-09-13T11:26:26.640 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:26.640 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:26.640 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:26.640 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:54.602650+0000", 2024-09-13T11:26:26.640 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.299277976 2024-09-13T11:26:26.640 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:26.640 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:26.640 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:26:26.641 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:54.602795+0000", 2024-09-13T11:26:26.641 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000144127 2024-09-13T11:26:26.641 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:26.641 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:26.641 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:26:26.641 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:54.602933+0000", 2024-09-13T11:26:26.641 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000138658 2024-09-13T11:26:26.641 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:26.641 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:26.641 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:26.641 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:26.641 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:26.641 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop(client.4665.0:217082 3.22 e187/122 3:449e4b87:::benchmark_data_smithi177_31116_object76779:head v 187'5666, mlcod=187'5664)", 2024-09-13T11:26:26.641 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:25:59.884121+0000", 2024-09-13T11:26:26.641 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 26.689389458000001, 2024-09-13T11:26:26.642 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.25479380699999998, 2024-09-13T11:26:26.642 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:26:26.642 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:26:26.642 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:26.642 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:26:26.642 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:26.642 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:26:26.642 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:59.884121+0000", 2024-09-13T11:26:26.642 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:26.642 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:26.642 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:26.642 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:26:26.642 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:59.884121+0000", 2024-09-13T11:26:26.642 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:26.643 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:26.643 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:26.643 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:26:26.643 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:59.884123+0000", 2024-09-13T11:26:26.643 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.677e-06 2024-09-13T11:26:26.643 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:26.643 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:26.643 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:26:26.643 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:59.884129+0000", 2024-09-13T11:26:26.643 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.669e-06 2024-09-13T11:26:26.643 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:26.643 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:26.643 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:26:26.643 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:00.137020+0000", 2024-09-13T11:26:26.644 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.25289069199999997 2024-09-13T11:26:26.644 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:26.644 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:26.644 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:26:26.644 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:00.137030+0000", 2024-09-13T11:26:26.644 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.8600000000000005e-06 2024-09-13T11:26:26.644 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:26.644 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:26.644 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:26.644 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:00.137123+0000", 2024-09-13T11:26:26.644 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.2842000000000003e-05 2024-09-13T11:26:26.644 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:26.644 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:26.645 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:26:26.645 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:00.137139+0000", 2024-09-13T11:26:26.645 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.6388e-05 2024-09-13T11:26:26.645 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:26.645 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:26.645 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:26:26.645 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:00.138876+0000", 2024-09-13T11:26:26.645 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0017370809999999999 2024-09-13T11:26:26.645 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:26.645 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:26.645 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:26:26.645 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:00.138915+0000", 2024-09-13T11:26:26.645 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.8597999999999999e-05 2024-09-13T11:26:26.645 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:26.646 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:26.646 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:26.646 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:26.646 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:26.646 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:221550 3.12 3:4b76b0c0:::benchmark_data_smithi177_31116_object81247:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e193)", 2024-09-13T11:26:26.646 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:26:06.064537+0000", 2024-09-13T11:26:26.646 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 20.508973734000001, 2024-09-13T11:26:26.646 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.229264668, 2024-09-13T11:26:26.646 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:26:26.646 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:26:26.646 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:26.646 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:26:26.646 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:26:26.646 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:26.647 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 221550 2024-09-13T11:26:26.647 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:26.647 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:26:26.647 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:26.647 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:26:26.647 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:06.064537+0000", 2024-09-13T11:26:26.647 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:26.647 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:26.647 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:26.647 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:26:26.647 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:06.064537+0000", 2024-09-13T11:26:26.647 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:26.647 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:26.647 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:26.648 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:26:26.648 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:06.064539+0000", 2024-09-13T11:26:26.648 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.753e-06 2024-09-13T11:26:26.648 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:26.648 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:26.648 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:26:26.648 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:06.064545+0000", 2024-09-13T11:26:26.648 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.6669999999999999e-06 2024-09-13T11:26:26.648 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:26.648 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:26.648 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:26:26.648 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:06.064613+0000", 2024-09-13T11:26:26.648 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.8199000000000002e-05 2024-09-13T11:26:26.649 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:26.649 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:26.649 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:26:26.649 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:06.064628+0000", 2024-09-13T11:26:26.649 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.455e-05 2024-09-13T11:26:26.649 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:26.649 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:26.649 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:26.649 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:06.064722+0000", 2024-09-13T11:26:26.649 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.4044999999999998e-05 2024-09-13T11:26:26.649 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:26.649 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:26.649 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-09-13T11:26:26.649 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:06.064747+0000", 2024-09-13T11:26:26.650 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.4949000000000001e-05 2024-09-13T11:26:26.650 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:26.650 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:26.650 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:26.650 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:06.275169+0000", 2024-09-13T11:26:26.650 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.21042197700000001 2024-09-13T11:26:26.650 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:26.650 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:26.650 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:26:26.650 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:06.275558+0000", 2024-09-13T11:26:26.650 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000388775 2024-09-13T11:26:26.650 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:26.650 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:26.650 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 7,14", 2024-09-13T11:26:26.651 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:06.275931+0000", 2024-09-13T11:26:26.651 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00037323499999999997 2024-09-13T11:26:26.651 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:26.651 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:26.651 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:26:26.651 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:06.288727+0000", 2024-09-13T11:26:26.651 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0127963 2024-09-13T11:26:26.651 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:26.651 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:26.651 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:26.651 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:06.288823+0000", 2024-09-13T11:26:26.651 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.5513000000000005e-05 2024-09-13T11:26:26.651 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:26.651 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:26.651 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:26.652 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:06.293498+0000", 2024-09-13T11:26:26.652 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0046750419999999999 2024-09-13T11:26:26.652 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:26.652 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:26.652 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:26:26.652 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:06.293600+0000", 2024-09-13T11:26:26.652 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000102376 2024-09-13T11:26:26.652 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:26.652 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:26.652 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:26:26.652 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:06.293802+0000", 2024-09-13T11:26:26.652 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000201287 2024-09-13T11:26:26.652 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:26.652 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:26.653 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:26.653 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:26.653 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:26.653 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:221567 3.12 3:4823d14b:::benchmark_data_smithi177_31116_object81264:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e193)", 2024-09-13T11:26:26.653 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:26:06.064686+0000", 2024-09-13T11:26:26.653 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 20.5088249, 2024-09-13T11:26:26.653 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.28539763299999998, 2024-09-13T11:26:26.653 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:26:26.653 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:26:26.653 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:26.653 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:26:26.653 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:26:26.653 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:26.653 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 221567 2024-09-13T11:26:26.654 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:26.654 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:26:26.654 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:26.654 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:26:26.654 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:06.064686+0000", 2024-09-13T11:26:26.654 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:26.654 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:26.654 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:26.654 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:26:26.654 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:06.064686+0000", 2024-09-13T11:26:26.654 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:26.654 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:26.655 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:26.655 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:26:26.655 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:06.064687+0000", 2024-09-13T11:26:26.655 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.6300000000000001e-06 2024-09-13T11:26:26.655 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:26.655 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:26.655 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:26:26.655 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:06.064693+0000", 2024-09-13T11:26:26.655 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.7339999999999996e-06 2024-09-13T11:26:26.655 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:26.655 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:26.655 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:26:26.655 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:06.064704+0000", 2024-09-13T11:26:26.655 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.1253000000000001e-05 2024-09-13T11:26:26.655 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:26.656 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:26.656 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:26:26.656 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:06.064717+0000", 2024-09-13T11:26:26.656 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.2293000000000001e-05 2024-09-13T11:26:26.656 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:26.656 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:26.656 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:26.656 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:06.064866+0000", 2024-09-13T11:26:26.656 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00014957499999999999 2024-09-13T11:26:26.656 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:26.656 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:26.656 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-09-13T11:26:26.656 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:06.064893+0000", 2024-09-13T11:26:26.657 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.6614e-05 2024-09-13T11:26:26.657 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:26.657 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:26.657 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:26.657 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:06.282646+0000", 2024-09-13T11:26:26.657 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.21775271800000001 2024-09-13T11:26:26.657 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:26.657 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:26.657 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:26:26.657 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:06.283003+0000", 2024-09-13T11:26:26.657 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00035780799999999999 2024-09-13T11:26:26.657 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:26.657 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:26.657 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 7,14", 2024-09-13T11:26:26.658 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:06.283345+0000", 2024-09-13T11:26:26.658 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00034164200000000002 2024-09-13T11:26:26.658 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:26.658 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:26.658 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:26:26.658 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:06.288935+0000", 2024-09-13T11:26:26.658 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0055902429999999999 2024-09-13T11:26:26.658 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:26.658 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:26.658 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:26.658 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:06.289795+0000", 2024-09-13T11:26:26.658 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00085952200000000004 2024-09-13T11:26:26.658 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:26.658 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:26.659 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:26.659 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:06.349852+0000", 2024-09-13T11:26:26.659 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.060057352000000001 2024-09-13T11:26:26.659 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:26.659 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:26.659 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:26:26.659 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:06.349922+0000", 2024-09-13T11:26:26.659 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.0272999999999996e-05 2024-09-13T11:26:26.659 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:26.659 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:26.659 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:26:26.659 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:06.350083+0000", 2024-09-13T11:26:26.659 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00016097600000000001 2024-09-13T11:26:26.659 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:26.660 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:26.660 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:26.660 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:26.660 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:26.660 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:227066 3.23 3:c67f2f28:::benchmark_data_smithi177_31116_object86763:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e196)", 2024-09-13T11:26:26.660 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:26:10.753624+0000", 2024-09-13T11:26:26.660 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 15.819886826999999, 2024-09-13T11:26:26.660 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.28670347299999999, 2024-09-13T11:26:26.660 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:26:26.660 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:26:26.660 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:26.660 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:26:26.660 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:26:26.661 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:26.661 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 227066 2024-09-13T11:26:26.661 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:26.661 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:26:26.661 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:26.661 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:26:26.661 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:10.753624+0000", 2024-09-13T11:26:26.661 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:26.661 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:26.661 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:26.661 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:26:26.661 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:10.753624+0000", 2024-09-13T11:26:26.661 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:26.661 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:26.662 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:26.662 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:26:26.662 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:10.753627+0000", 2024-09-13T11:26:26.662 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.8030000000000001e-06 2024-09-13T11:26:26.662 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:26.662 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:26.662 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:26:26.662 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:10.753634+0000", 2024-09-13T11:26:26.662 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.9659999999999999e-06 2024-09-13T11:26:26.662 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:26.662 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:26.662 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:26:26.662 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:10.753650+0000", 2024-09-13T11:26:26.662 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.6167999999999998e-05 2024-09-13T11:26:26.663 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:26.663 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:26.663 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:26:26.663 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:10.753657+0000", 2024-09-13T11:26:26.663 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.7600000000000002e-06 2024-09-13T11:26:26.663 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:26.663 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:26.663 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:26.663 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:10.753741+0000", 2024-09-13T11:26:26.663 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.3523999999999995e-05 2024-09-13T11:26:26.663 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:26.663 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:26.663 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:26:26.663 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:10.753878+0000", 2024-09-13T11:26:26.664 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000137207 2024-09-13T11:26:26.664 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:26.664 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:26.664 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 9", 2024-09-13T11:26:26.664 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:10.754027+0000", 2024-09-13T11:26:26.664 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00014856599999999999 2024-09-13T11:26:26.664 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:26.664 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:26.664 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:26:26.664 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:10.756313+0000", 2024-09-13T11:26:26.664 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.002286045 2024-09-13T11:26:26.664 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:26.664 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:26.664 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:26.665 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:11.040266+0000", 2024-09-13T11:26:26.665 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.283953434 2024-09-13T11:26:26.665 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:26.665 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:26.665 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:26:26.665 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:11.040300+0000", 2024-09-13T11:26:26.665 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.4227000000000003e-05 2024-09-13T11:26:26.665 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:26.665 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:26.665 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:26:26.665 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:11.040327+0000", 2024-09-13T11:26:26.665 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.6772999999999999e-05 2024-09-13T11:26:26.665 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:26.665 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:26.666 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:26.666 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:26.666 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:26.666 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop_reply(client.4665.0:227066 3.23 e196/90 ondisk, result = 0)", 2024-09-13T11:26:26.666 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:26:10.756369+0000", 2024-09-13T11:26:26.666 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 15.817141453, 2024-09-13T11:26:26.666 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.28396190599999999, 2024-09-13T11:26:26.666 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:26:26.666 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:26:26.666 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-09-13T11:26:26.666 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:26:26.666 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:26.666 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:26:26.666 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:10.756369+0000", 2024-09-13T11:26:26.667 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:26.667 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:26.667 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:26.667 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:26:26.667 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:10.756369+0000", 2024-09-13T11:26:26.667 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:26.667 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:26.667 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:26.667 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:26:26.667 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:10.756370+0000", 2024-09-13T11:26:26.667 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.164e-06 2024-09-13T11:26:26.667 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:26.667 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:26.667 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:26:26.668 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:10.756374+0000", 2024-09-13T11:26:26.668 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.0659999999999997e-06 2024-09-13T11:26:26.668 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:26.668 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:26.668 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:26:26.668 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:11.040177+0000", 2024-09-13T11:26:26.668 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.28380288599999998 2024-09-13T11:26:26.668 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:26.668 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:26.668 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:26:26.668 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:11.040184+0000", 2024-09-13T11:26:26.668 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.7229999999999998e-06 2024-09-13T11:26:26.668 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:26.668 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:26.669 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:26.669 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:11.040253+0000", 2024-09-13T11:26:26.669 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.8997000000000004e-05 2024-09-13T11:26:26.669 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:26.669 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:26.669 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:26:26.669 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:11.040261+0000", 2024-09-13T11:26:26.669 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.3590000000000008e-06 2024-09-13T11:26:26.669 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:26.669 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:26.669 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:26:26.669 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:11.040331+0000", 2024-09-13T11:26:26.669 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.9710999999999997e-05 2024-09-13T11:26:26.669 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:26.670 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:26.670 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:26.670 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:26.670 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:26.670 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:227420 3.23 3:c4add80a:::benchmark_data_smithi177_31116_object87117:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e196)", 2024-09-13T11:26:26.670 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:26:11.037912+0000", 2024-09-13T11:26:26.670 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 15.535598807, 2024-09-13T11:26:26.670 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.33121650499999999, 2024-09-13T11:26:26.670 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:26:26.670 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:26:26.670 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:26.670 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:26:26.670 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:26:26.670 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:26.671 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 227420 2024-09-13T11:26:26.671 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:26.671 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:26:26.671 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:26.671 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:26:26.671 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:11.037912+0000", 2024-09-13T11:26:26.671 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:26.671 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:26.671 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:26.671 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:26:26.671 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:11.037912+0000", 2024-09-13T11:26:26.671 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:26.671 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:26.671 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:26.672 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:26:26.672 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:11.037913+0000", 2024-09-13T11:26:26.672 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.606e-06 2024-09-13T11:26:26.672 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:26.672 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:26.672 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:26:26.672 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:11.037918+0000", 2024-09-13T11:26:26.672 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.0509999999999996e-06 2024-09-13T11:26:26.672 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:26.672 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:26.672 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:26:26.672 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:11.037930+0000", 2024-09-13T11:26:26.672 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.1466e-05 2024-09-13T11:26:26.673 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:26.673 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:26.673 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:26:26.673 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:11.037936+0000", 2024-09-13T11:26:26.673 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.5259999999999999e-06 2024-09-13T11:26:26.673 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:26.673 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:26.673 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:26.673 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:11.037994+0000", 2024-09-13T11:26:26.673 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.7776999999999999e-05 2024-09-13T11:26:26.673 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:26.673 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:26.673 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:26:26.673 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:11.038129+0000", 2024-09-13T11:26:26.674 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00013500299999999999 2024-09-13T11:26:26.674 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:26.674 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:26.674 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 9", 2024-09-13T11:26:26.674 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:11.038208+0000", 2024-09-13T11:26:26.674 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.9016000000000004e-05 2024-09-13T11:26:26.674 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:26.674 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:26.674 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:26:26.674 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:11.041461+0000", 2024-09-13T11:26:26.674 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00325246 2024-09-13T11:26:26.674 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:26.674 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:26.674 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:26.675 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:11.369025+0000", 2024-09-13T11:26:26.675 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.32756474800000002 2024-09-13T11:26:26.675 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:26.675 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:26.675 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:26:26.675 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:11.369086+0000", 2024-09-13T11:26:26.675 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.0140999999999999e-05 2024-09-13T11:26:26.675 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:26.675 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:26.675 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:26:26.675 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:11.369128+0000", 2024-09-13T11:26:26.675 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.2710999999999998e-05 2024-09-13T11:26:26.675 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:26.675 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:26.676 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:26.676 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:26.676 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:26.676 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop_reply(client.4665.0:227420 3.23 e196/90 ondisk, result = 0)", 2024-09-13T11:26:26.676 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:26:11.040152+0000", 2024-09-13T11:26:26.676 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 15.533358969, 2024-09-13T11:26:26.676 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.328983201, 2024-09-13T11:26:26.676 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:26:26.676 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:26:26.676 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-09-13T11:26:26.676 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:26:26.676 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:26.676 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:26:26.677 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:11.040152+0000", 2024-09-13T11:26:26.677 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:26.677 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:26.677 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:26.677 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:26:26.677 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:11.040152+0000", 2024-09-13T11:26:26.677 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:26.677 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:26.677 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:26.677 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:26:26.677 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:11.040157+0000", 2024-09-13T11:26:26.677 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.1309999999999997e-06 2024-09-13T11:26:26.677 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:26.677 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:26.678 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:26:26.678 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:11.040162+0000", 2024-09-13T11:26:26.678 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.8720000000000001e-06 2024-09-13T11:26:26.678 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:26.678 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:26.678 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:26:26.678 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:11.368952+0000", 2024-09-13T11:26:26.678 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.32879022400000002 2024-09-13T11:26:26.678 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:26.678 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:26.678 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:26:26.678 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:11.368960+0000", 2024-09-13T11:26:26.678 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.8460000000000008e-06 2024-09-13T11:26:26.678 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:26.679 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:26.679 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:26.679 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:11.369013+0000", 2024-09-13T11:26:26.679 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.3201000000000001e-05 2024-09-13T11:26:26.679 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:26.679 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:26.679 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:26:26.679 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:11.369020+0000", 2024-09-13T11:26:26.679 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.3050000000000004e-06 2024-09-13T11:26:26.679 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:26.679 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:26.679 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:26:26.679 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:11.369135+0000", 2024-09-13T11:26:26.679 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000114622 2024-09-13T11:26:26.680 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:26.680 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:26.680 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:26.680 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:26.680 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:26.680 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-09-13T11:26:26.681 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-09-13T11:26:26.682 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-09-13T11:26:26.679+0000 7f97befc9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26315) UID: 0 2024-09-13T11:26:26.782 INFO:tasks.ceph.osd.11.smithi177.stderr:2024-09-13T11:26:26.777+0000 7f24792c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26580) UID: 0 2024-09-13T11:26:26.856 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-09-13T11:26:26.856 INFO:teuthology.orchestra.run.smithi092.stdout: "ops": [], 2024-09-13T11:26:26.856 INFO:teuthology.orchestra.run.smithi092.stdout: "num_ops": 0 2024-09-13T11:26:26.856 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-09-13T11:26:26.872 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-09-13T11:26:26.882 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-09-13T11:26:26.879+0000 7f97befc9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26315) UID: 0 2024-09-13T11:26:26.955 INFO:tasks.ceph.osd.8.smithi001.stderr:2024-09-13T11:26:26.950+0000 7fb9ab2df640 -1 osd.8 207 handle_osd_map injecting map crc failure 2024-09-13T11:26:26.958 INFO:tasks.ceph.osd.15.smithi177.stderr:2024-09-13T11:26:26.953+0000 7f9d618a4640 -1 osd.15 207 handle_osd_map injecting map crc failure 2024-09-13T11:26:26.983 INFO:tasks.ceph.osd.4.smithi001.stderr:2024-09-13T11:26:26.982+0000 7ff8e0218640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26310) UID: 0 2024-09-13T11:26:27.012 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-09-13T11:26:27.013 INFO:teuthology.orchestra.run.smithi092.stdout: "ops": [], 2024-09-13T11:26:27.013 INFO:teuthology.orchestra.run.smithi092.stdout: "complaint_time": 30, 2024-09-13T11:26:27.013 INFO:teuthology.orchestra.run.smithi092.stdout: "num_blocked_ops": 0 2024-09-13T11:26:27.013 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-09-13T11:26:27.029 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-09-13T11:26:27.084 INFO:tasks.ceph.osd.14.smithi162.stderr:2024-09-13T11:26:27.079+0000 7f3df5903640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33154) UID: 0 2024-09-13T11:26:27.175 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-09-13T11:26:27.176 INFO:teuthology.orchestra.run.smithi092.stdout: "size": 20, 2024-09-13T11:26:27.176 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 600, 2024-09-13T11:26:27.176 INFO:teuthology.orchestra.run.smithi092.stdout: "ops": [ 2024-09-13T11:26:27.176 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.176 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "MOSDPGPush(3.19 76/70 [PushOp(3:9eacf3d0:::benchmark_data_smithi177_31116_object19747:head, version: 54'1236, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:9eacf3d0:::benchmark_data_smithi177_31116_object19747:head@54'1236, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:26:27.176 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:32.704676+0000", 2024-09-13T11:26:27.176 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 174.46988677600001, 2024-09-13T11:26:27.176 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.23525014899999999, 2024-09-13T11:26:27.176 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:26:27.176 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:26:27.176 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-09-13T11:26:27.176 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:26:27.177 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.177 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:26:27.177 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:32.704676+0000", 2024-09-13T11:26:27.177 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:27.177 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.177 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.177 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:26:27.177 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:32.704676+0000", 2024-09-13T11:26:27.177 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:27.177 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.177 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.177 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:26:27.177 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:32.704677+0000", 2024-09-13T11:26:27.177 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.531e-06 2024-09-13T11:26:27.178 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.178 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.178 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:26:27.178 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:32.704749+0000", 2024-09-13T11:26:27.178 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.1451000000000004e-05 2024-09-13T11:26:27.178 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.178 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.178 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:26:27.178 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:32.938053+0000", 2024-09-13T11:26:27.178 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.233304752 2024-09-13T11:26:27.178 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.178 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.178 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:26:27.178 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:32.938087+0000", 2024-09-13T11:26:27.179 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.4016000000000001e-05 2024-09-13T11:26:27.179 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.179 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.179 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:27.179 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:32.938351+0000", 2024-09-13T11:26:27.179 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00026397500000000001 2024-09-13T11:26:27.179 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.179 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.179 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:26:27.179 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:32.938381+0000", 2024-09-13T11:26:27.179 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.0001999999999998e-05 2024-09-13T11:26:27.180 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.180 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.180 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:26:27.180 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:32.939926+0000", 2024-09-13T11:26:27.180 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0015444219999999999 2024-09-13T11:26:27.180 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:27.180 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:27.180 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:27.180 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.180 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.180 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "MOSDPGPush(3.23 80/78 [PushOp(3:c59ecc6d:::benchmark_data_smithi177_31116_object17005:head, version: 52'1054, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:c59ecc6d:::benchmark_data_smithi177_31116_object17005:head@52'1054, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:26:27.180 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:37.433040+0000", 2024-09-13T11:26:27.180 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 169.74152280000001, 2024-09-13T11:26:27.181 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.26658219100000002, 2024-09-13T11:26:27.181 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:26:27.181 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:26:27.181 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-09-13T11:26:27.181 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:26:27.181 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.181 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:26:27.181 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:37.433040+0000", 2024-09-13T11:26:27.181 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:27.181 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.181 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.181 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:26:27.181 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:37.433040+0000", 2024-09-13T11:26:27.181 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:27.181 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.182 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.182 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:26:27.182 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:37.433042+0000", 2024-09-13T11:26:27.182 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.3999999999999999e-06 2024-09-13T11:26:27.182 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.182 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.182 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:26:27.182 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:37.433090+0000", 2024-09-13T11:26:27.182 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.8065999999999997e-05 2024-09-13T11:26:27.182 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.182 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.182 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:26:27.182 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:37.698797+0000", 2024-09-13T11:26:27.182 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.26570674500000002 2024-09-13T11:26:27.183 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.183 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.183 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:26:27.183 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:37.698827+0000", 2024-09-13T11:26:27.183 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.0505e-05 2024-09-13T11:26:27.183 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.183 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.183 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:27.183 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:37.698941+0000", 2024-09-13T11:26:27.183 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00011403799999999999 2024-09-13T11:26:27.183 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.183 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.183 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:26:27.183 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:37.698956+0000", 2024-09-13T11:26:27.184 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.4765e-05 2024-09-13T11:26:27.184 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.184 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.184 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:26:27.184 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:37.699622+0000", 2024-09-13T11:26:27.184 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00066567199999999997 2024-09-13T11:26:27.184 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:27.184 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:27.184 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:27.184 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.184 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.184 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "MOSDPGPush(3.1a 85/78 [PushOp(3:5dc85525:::benchmark_data_smithi177_31116_object11755:head, version: 49'723, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:5dc85525:::benchmark_data_smithi177_31116_object11755:head@49'723, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:26:27.184 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:42.422061+0000", 2024-09-13T11:26:27.184 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 164.75250134699999, 2024-09-13T11:26:27.185 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.52875617500000005, 2024-09-13T11:26:27.185 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:26:27.185 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:26:27.185 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-09-13T11:26:27.185 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:26:27.185 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.185 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:26:27.185 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.422061+0000", 2024-09-13T11:26:27.185 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:27.185 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.185 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.185 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:26:27.185 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.422061+0000", 2024-09-13T11:26:27.185 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:27.185 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.186 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.186 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:26:27.186 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.422064+0000", 2024-09-13T11:26:27.186 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.5409999999999999e-06 2024-09-13T11:26:27.186 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.186 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.186 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:26:27.186 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.422139+0000", 2024-09-13T11:26:27.186 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.5488999999999995e-05 2024-09-13T11:26:27.186 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.186 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.186 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:26:27.186 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.948781+0000", 2024-09-13T11:26:27.186 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.52664230999999995 2024-09-13T11:26:27.186 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.187 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.187 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:26:27.187 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.948841+0000", 2024-09-13T11:26:27.187 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.9450999999999997e-05 2024-09-13T11:26:27.187 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.187 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.187 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:27.187 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.949035+0000", 2024-09-13T11:26:27.187 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00019380600000000001 2024-09-13T11:26:27.187 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.187 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.187 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:26:27.187 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.949060+0000", 2024-09-13T11:26:27.188 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.5381999999999999e-05 2024-09-13T11:26:27.188 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.188 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.188 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:26:27.188 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.950817+0000", 2024-09-13T11:26:27.188 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0017571959999999999 2024-09-13T11:26:27.188 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:27.188 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:27.188 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:27.188 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.188 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.188 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "MOSDPGPush(3.1a 89/78 [PushOp(3:59270734:::benchmark_data_smithi177_31116_object20039:head, version: 54'1256, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:59270734:::benchmark_data_smithi177_31116_object20039:head@54'1256, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:26:27.188 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:46.607562+0000", 2024-09-13T11:26:27.188 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 160.56700057, 2024-09-13T11:26:27.189 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.32123945300000001, 2024-09-13T11:26:27.189 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:26:27.189 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:26:27.189 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-09-13T11:26:27.189 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:26:27.189 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.189 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:26:27.189 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.607562+0000", 2024-09-13T11:26:27.189 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:27.189 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.189 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.189 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:26:27.189 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.607562+0000", 2024-09-13T11:26:27.189 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:27.190 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.190 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.190 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:26:27.190 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.607564+0000", 2024-09-13T11:26:27.190 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.8610000000000001e-06 2024-09-13T11:26:27.190 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.190 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.190 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:26:27.190 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.607663+0000", 2024-09-13T11:26:27.190 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.9210999999999996e-05 2024-09-13T11:26:27.190 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.190 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.190 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:26:27.190 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.925193+0000", 2024-09-13T11:26:27.190 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.317530586 2024-09-13T11:26:27.191 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.191 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.191 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:26:27.191 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.925250+0000", 2024-09-13T11:26:27.191 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.6925000000000001e-05 2024-09-13T11:26:27.191 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.191 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.191 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:27.191 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.927013+0000", 2024-09-13T11:26:27.191 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.001762853 2024-09-13T11:26:27.191 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.191 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.191 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:26:27.191 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.927067+0000", 2024-09-13T11:26:27.192 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.3601999999999999e-05 2024-09-13T11:26:27.192 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.192 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.192 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:26:27.192 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.928801+0000", 2024-09-13T11:26:27.192 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.001734415 2024-09-13T11:26:27.192 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:27.192 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:27.192 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:27.192 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.192 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.192 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "MOSDPGPush(3.1a 91/78 [PushOp(3:5e5010d2:::benchmark_data_smithi177_31116_object25733:head, version: 58'1609, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:5e5010d2:::benchmark_data_smithi177_31116_object25733:head@58'1609, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:26:27.192 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:48.608485+0000", 2024-09-13T11:26:27.192 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 158.566077452, 2024-09-13T11:26:27.193 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.35035599699999997, 2024-09-13T11:26:27.193 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:26:27.193 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:26:27.193 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-09-13T11:26:27.193 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:26:27.193 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.193 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:26:27.193 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:48.608485+0000", 2024-09-13T11:26:27.193 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:27.193 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.193 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.193 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:26:27.193 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:48.608485+0000", 2024-09-13T11:26:27.193 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:27.194 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.194 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.194 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:26:27.194 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:48.608488+0000", 2024-09-13T11:26:27.194 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.9399999999999998e-06 2024-09-13T11:26:27.194 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.194 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.194 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:26:27.194 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:48.608552+0000", 2024-09-13T11:26:27.194 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.3731000000000004e-05 2024-09-13T11:26:27.194 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.194 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.194 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:26:27.194 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:48.957330+0000", 2024-09-13T11:26:27.195 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.34877806099999997 2024-09-13T11:26:27.195 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.195 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.195 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:26:27.195 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:48.957361+0000", 2024-09-13T11:26:27.195 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.0852000000000002e-05 2024-09-13T11:26:27.195 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.195 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.195 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:27.195 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:48.957544+0000", 2024-09-13T11:26:27.195 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000183537 2024-09-13T11:26:27.195 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.195 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.195 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:26:27.195 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:48.957582+0000", 2024-09-13T11:26:27.196 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.7472999999999998e-05 2024-09-13T11:26:27.196 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.196 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.196 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:26:27.196 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:48.958841+0000", 2024-09-13T11:26:27.196 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0012594030000000001 2024-09-13T11:26:27.196 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:27.196 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:27.196 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:27.196 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.196 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.196 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop(client.4665.0:72923 3.23 e97/90 3:c6230123:::benchmark_data_smithi177_31116_object72922:head v 97'3418, mlcod=97'3414)", 2024-09-13T11:26:27.196 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:55.047734+0000", 2024-09-13T11:26:27.196 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 152.12682793499999, 2024-09-13T11:26:27.197 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.20075528000000001, 2024-09-13T11:26:27.197 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:26:27.197 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:26:27.197 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:27.197 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:26:27.197 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.197 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:26:27.197 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:55.047734+0000", 2024-09-13T11:26:27.197 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:27.197 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.197 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.197 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:26:27.197 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:55.047734+0000", 2024-09-13T11:26:27.197 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:27.198 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.198 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.198 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:26:27.198 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:55.047736+0000", 2024-09-13T11:26:27.198 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.466e-06 2024-09-13T11:26:27.198 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.198 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.198 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:26:27.198 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:55.047772+0000", 2024-09-13T11:26:27.198 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.6576999999999999e-05 2024-09-13T11:26:27.198 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.198 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.198 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:26:27.198 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:55.246478+0000", 2024-09-13T11:26:27.199 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.19870581300000001 2024-09-13T11:26:27.199 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.199 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.199 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:26:27.199 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:55.246488+0000", 2024-09-13T11:26:27.199 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.5419999999999994e-06 2024-09-13T11:26:27.199 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.199 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.199 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:27.199 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:55.246631+0000", 2024-09-13T11:26:27.199 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00014287300000000001 2024-09-13T11:26:27.199 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.199 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.199 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:26:27.200 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:55.246665+0000", 2024-09-13T11:26:27.200 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.4684999999999997e-05 2024-09-13T11:26:27.200 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.200 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.200 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:26:27.200 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:55.248453+0000", 2024-09-13T11:26:27.200 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0017879459999999999 2024-09-13T11:26:27.200 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.200 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.200 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:26:27.200 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:55.248490+0000", 2024-09-13T11:26:27.200 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.6378000000000003e-05 2024-09-13T11:26:27.200 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:27.200 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:27.201 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:27.201 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.201 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.201 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop(client.4665.0:98582 3.1a e113/105 3:59f9cd75:::benchmark_data_smithi177_31116_object98581:head v 113'4743, mlcod=113'4741)", 2024-09-13T11:26:27.201 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:24:15.432027+0000", 2024-09-13T11:26:27.201 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 131.74253532, 2024-09-13T11:26:27.201 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.196879784, 2024-09-13T11:26:27.201 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:26:27.201 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:26:27.201 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:27.201 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:26:27.201 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.201 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:26:27.201 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:15.432027+0000", 2024-09-13T11:26:27.202 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:27.202 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.202 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.202 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:26:27.202 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:15.432027+0000", 2024-09-13T11:26:27.202 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:27.202 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.202 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.202 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:26:27.202 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:15.432029+0000", 2024-09-13T11:26:27.202 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.838e-06 2024-09-13T11:26:27.202 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.202 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.202 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:26:27.203 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:15.432088+0000", 2024-09-13T11:26:27.203 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.8767999999999997e-05 2024-09-13T11:26:27.203 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.203 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.203 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:26:27.203 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:15.624731+0000", 2024-09-13T11:26:27.203 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.192643221 2024-09-13T11:26:27.203 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.203 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.203 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:26:27.203 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:15.624746+0000", 2024-09-13T11:26:27.203 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.4705000000000001e-05 2024-09-13T11:26:27.203 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.203 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.204 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:27.204 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:15.624882+0000", 2024-09-13T11:26:27.204 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00013602300000000001 2024-09-13T11:26:27.204 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.204 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.204 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:26:27.204 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:15.624941+0000", 2024-09-13T11:26:27.204 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.9351000000000001e-05 2024-09-13T11:26:27.204 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.204 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.205 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:26:27.205 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:15.628836+0000", 2024-09-13T11:26:27.205 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0038955109999999999 2024-09-13T11:26:27.205 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.205 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.205 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:26:27.205 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:15.628907+0000", 2024-09-13T11:26:27.205 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.0366999999999999e-05 2024-09-13T11:26:27.205 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:27.205 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:27.205 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:27.205 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.205 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.205 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop(client.4665.0:99105 3.1a e113/105 3:5ac5090c:::benchmark_data_smithi177_31116_object99104:head v 113'4749, mlcod=113'4746)", 2024-09-13T11:26:27.206 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:24:15.963602+0000", 2024-09-13T11:26:27.206 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 131.21096013799999, 2024-09-13T11:26:27.206 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.23699906300000001, 2024-09-13T11:26:27.206 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:26:27.206 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:26:27.206 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:27.206 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:26:27.206 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.206 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:26:27.206 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:15.963602+0000", 2024-09-13T11:26:27.206 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:27.206 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.206 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.206 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:26:27.206 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:15.963602+0000", 2024-09-13T11:26:27.207 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:27.207 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.207 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.207 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:26:27.207 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:15.963605+0000", 2024-09-13T11:26:27.207 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.0400000000000001e-06 2024-09-13T11:26:27.207 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.207 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.207 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:26:27.207 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:15.963718+0000", 2024-09-13T11:26:27.207 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000112796 2024-09-13T11:26:27.207 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.207 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.207 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:26:27.208 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:16.197994+0000", 2024-09-13T11:26:27.208 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.234276285 2024-09-13T11:26:27.208 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.208 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.208 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:26:27.208 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:16.198004+0000", 2024-09-13T11:26:27.208 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.5559999999999996e-06 2024-09-13T11:26:27.208 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.208 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.208 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:27.208 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:16.198110+0000", 2024-09-13T11:26:27.208 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00010614 2024-09-13T11:26:27.208 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.208 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.208 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:26:27.209 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:16.198152+0000", 2024-09-13T11:26:27.209 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.1899e-05 2024-09-13T11:26:27.209 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.209 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.209 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:26:27.209 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:16.200528+0000", 2024-09-13T11:26:27.209 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00237587 2024-09-13T11:26:27.209 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.209 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.209 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:26:27.209 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:16.200601+0000", 2024-09-13T11:26:27.209 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.3477000000000001e-05 2024-09-13T11:26:27.209 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:27.209 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:27.210 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:27.210 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.210 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.210 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop(client.4665.0:129921 3.1a e134/105 3:5fb9088b:::benchmark_data_smithi177_31116_object129920:head v 134'5758, mlcod=134'5755)", 2024-09-13T11:26:27.210 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:24:47.178021+0000", 2024-09-13T11:26:27.210 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 99.996541132000004, 2024-09-13T11:26:27.210 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.33723278600000001, 2024-09-13T11:26:27.210 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:26:27.210 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:26:27.210 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:27.210 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:26:27.210 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.210 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:26:27.210 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:47.178021+0000", 2024-09-13T11:26:27.210 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:27.211 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.211 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.211 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:26:27.211 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:47.178021+0000", 2024-09-13T11:26:27.211 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:27.211 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.211 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.211 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:26:27.211 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:47.178024+0000", 2024-09-13T11:26:27.211 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.3609999999999999e-06 2024-09-13T11:26:27.211 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.211 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.211 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:26:27.211 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:47.178102+0000", 2024-09-13T11:26:27.211 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.8325000000000006e-05 2024-09-13T11:26:27.212 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.212 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.212 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:26:27.212 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:47.512707+0000", 2024-09-13T11:26:27.212 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.33460516000000001 2024-09-13T11:26:27.212 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.212 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.212 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:26:27.212 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:47.512716+0000", 2024-09-13T11:26:27.212 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.0389999999999993e-06 2024-09-13T11:26:27.212 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.212 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.212 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:27.212 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:47.512788+0000", 2024-09-13T11:26:27.213 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.1840000000000003e-05 2024-09-13T11:26:27.213 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.213 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.213 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:26:27.213 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:47.512809+0000", 2024-09-13T11:26:27.213 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.1005e-05 2024-09-13T11:26:27.213 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.213 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.213 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:26:27.213 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:47.515182+0000", 2024-09-13T11:26:27.213 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0023731939999999999 2024-09-13T11:26:27.213 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.213 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.213 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:26:27.214 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:47.515254+0000", 2024-09-13T11:26:27.214 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.1861999999999997e-05 2024-09-13T11:26:27.214 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:27.214 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:27.214 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:27.214 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.214 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.214 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop(client.4665.0:138370 3.23 e140/90 3:c5730cd2:::benchmark_data_smithi177_31116_object138369:head v 140'4458, mlcod=140'4455)", 2024-09-13T11:26:27.214 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:24:54.416118+0000", 2024-09-13T11:26:27.214 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 92.758444654000002, 2024-09-13T11:26:27.214 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.244024991, 2024-09-13T11:26:27.214 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:26:27.214 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:26:27.214 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:27.215 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:26:27.215 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.215 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:26:27.215 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:54.416118+0000", 2024-09-13T11:26:27.215 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:27.215 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.215 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.215 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:26:27.215 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:54.416118+0000", 2024-09-13T11:26:27.215 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:27.215 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.215 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.215 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:26:27.215 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:54.416120+0000", 2024-09-13T11:26:27.215 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.6280000000000001e-06 2024-09-13T11:26:27.216 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.216 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.216 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:26:27.216 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:54.416161+0000", 2024-09-13T11:26:27.216 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.0877000000000002e-05 2024-09-13T11:26:27.216 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.216 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.216 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:26:27.216 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:54.657128+0000", 2024-09-13T11:26:27.216 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.24096696300000001 2024-09-13T11:26:27.216 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.216 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.216 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:26:27.216 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:54.657142+0000", 2024-09-13T11:26:27.217 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.358e-05 2024-09-13T11:26:27.217 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.217 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.217 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:27.217 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:54.657290+0000", 2024-09-13T11:26:27.217 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00014794400000000001 2024-09-13T11:26:27.217 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.217 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.217 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:26:27.217 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:54.657325+0000", 2024-09-13T11:26:27.217 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.5623000000000001e-05 2024-09-13T11:26:27.217 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.217 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.217 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:26:27.218 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:54.660079+0000", 2024-09-13T11:26:27.218 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0027536489999999999 2024-09-13T11:26:27.218 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.218 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.218 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:26:27.218 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:54.660143+0000", 2024-09-13T11:26:27.218 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.3726999999999995e-05 2024-09-13T11:26:27.218 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:27.218 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:27.218 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:27.218 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.218 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.218 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:209225 3.24 3:2478ed54:::benchmark_data_smithi177_31116_object68922:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e185)", 2024-09-13T11:26:27.218 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:25:53.632417+0000", 2024-09-13T11:26:27.219 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 33.542145220999998, 2024-09-13T11:26:27.219 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.219758709, 2024-09-13T11:26:27.219 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:26:27.219 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:26:27.219 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:27.219 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:26:27.219 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:26:27.219 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:27.219 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 209225 2024-09-13T11:26:27.219 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.219 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:26:27.219 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.219 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:26:27.219 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.632417+0000", 2024-09-13T11:26:27.219 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:27.220 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.220 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.220 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:26:27.220 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.632417+0000", 2024-09-13T11:26:27.220 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:27.220 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.220 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.220 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:26:27.220 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.632420+0000", 2024-09-13T11:26:27.220 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.4980000000000001e-06 2024-09-13T11:26:27.220 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.220 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.220 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:26:27.220 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.632427+0000", 2024-09-13T11:26:27.221 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.5120000000000002e-06 2024-09-13T11:26:27.221 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.221 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.221 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:26:27.221 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.632441+0000", 2024-09-13T11:26:27.221 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.4072e-05 2024-09-13T11:26:27.221 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.221 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.221 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:26:27.221 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.632454+0000", 2024-09-13T11:26:27.221 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.2886e-05 2024-09-13T11:26:27.221 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.221 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.221 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:27.221 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.632529+0000", 2024-09-13T11:26:27.222 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.5326000000000004e-05 2024-09-13T11:26:27.222 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.222 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.222 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:26:27.222 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.632642+0000", 2024-09-13T11:26:27.222 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00011249200000000001 2024-09-13T11:26:27.222 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.222 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.222 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1", 2024-09-13T11:26:27.222 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.632712+0000", 2024-09-13T11:26:27.222 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.0376000000000005e-05 2024-09-13T11:26:27.222 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.222 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.222 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:26:27.223 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.635270+0000", 2024-09-13T11:26:27.223 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0025574339999999999 2024-09-13T11:26:27.223 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.223 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.223 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:27.223 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.851915+0000", 2024-09-13T11:26:27.223 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.21664515400000001 2024-09-13T11:26:27.223 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.223 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.223 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:26:27.223 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.852065+0000", 2024-09-13T11:26:27.223 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00014972799999999999 2024-09-13T11:26:27.223 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.223 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.223 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:26:27.224 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.852176+0000", 2024-09-13T11:26:27.224 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00011123100000000001 2024-09-13T11:26:27.224 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:27.224 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:27.224 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:27.224 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.224 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.224 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:209229 3.19 3:9a976dcb:::benchmark_data_smithi177_31116_object68926:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e185)", 2024-09-13T11:26:27.224 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:25:53.633903+0000", 2024-09-13T11:26:27.224 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 33.540659316000003, 2024-09-13T11:26:27.224 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.25298130200000002, 2024-09-13T11:26:27.224 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:26:27.224 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:26:27.224 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:27.225 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:26:27.225 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:26:27.225 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:27.225 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 209229 2024-09-13T11:26:27.225 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.225 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:26:27.225 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.225 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:26:27.225 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.633903+0000", 2024-09-13T11:26:27.225 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:27.225 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.225 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.225 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:26:27.225 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.633903+0000", 2024-09-13T11:26:27.226 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:27.226 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.226 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.226 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:26:27.226 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.633905+0000", 2024-09-13T11:26:27.226 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.1600000000000001e-06 2024-09-13T11:26:27.226 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.226 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.226 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:26:27.226 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.633917+0000", 2024-09-13T11:26:27.226 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.1881e-05 2024-09-13T11:26:27.226 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.226 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.226 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:26:27.227 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.633931+0000", 2024-09-13T11:26:27.227 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.432e-05 2024-09-13T11:26:27.227 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.227 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.227 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:26:27.227 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.633941+0000", 2024-09-13T11:26:27.227 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.8079999999999999e-06 2024-09-13T11:26:27.227 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.227 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.227 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:27.227 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.634009+0000", 2024-09-13T11:26:27.227 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.7895e-05 2024-09-13T11:26:27.227 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.227 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.228 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:26:27.228 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.634137+0000", 2024-09-13T11:26:27.228 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000127644 2024-09-13T11:26:27.228 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.228 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.228 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1", 2024-09-13T11:26:27.228 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.634245+0000", 2024-09-13T11:26:27.228 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00010776399999999999 2024-09-13T11:26:27.228 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.228 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.228 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:26:27.228 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.637244+0000", 2024-09-13T11:26:27.228 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0029993149999999998 2024-09-13T11:26:27.228 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.229 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.229 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:27.229 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.886675+0000", 2024-09-13T11:26:27.229 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.249431456 2024-09-13T11:26:27.229 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.229 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.229 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:26:27.229 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.886785+0000", 2024-09-13T11:26:27.229 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000109326 2024-09-13T11:26:27.229 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.229 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.229 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:26:27.229 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.886884+0000", 2024-09-13T11:26:27.230 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.9733000000000002e-05 2024-09-13T11:26:27.230 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:27.230 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:27.230 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:27.230 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.230 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.230 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:209252 3.24 3:25ab37ee:::benchmark_data_smithi177_31116_object68949:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e185)", 2024-09-13T11:26:27.230 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:25:53.647942+0000", 2024-09-13T11:26:27.230 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 33.526620244, 2024-09-13T11:26:27.230 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.21571831799999999, 2024-09-13T11:26:27.230 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:26:27.230 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:26:27.230 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:27.230 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:26:27.231 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:26:27.231 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:27.231 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 209252 2024-09-13T11:26:27.231 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.231 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:26:27.231 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.231 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:26:27.231 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.647942+0000", 2024-09-13T11:26:27.231 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:27.231 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.231 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.231 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:26:27.231 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.647942+0000", 2024-09-13T11:26:27.231 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:27.232 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.232 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.232 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:26:27.232 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.647945+0000", 2024-09-13T11:26:27.232 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.717e-06 2024-09-13T11:26:27.232 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.232 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.232 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:26:27.232 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.647951+0000", 2024-09-13T11:26:27.232 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.353e-06 2024-09-13T11:26:27.232 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.232 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.232 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:26:27.233 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.647966+0000", 2024-09-13T11:26:27.233 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.4877e-05 2024-09-13T11:26:27.233 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.233 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.233 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:26:27.233 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.647973+0000", 2024-09-13T11:26:27.233 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.2640000000000002e-06 2024-09-13T11:26:27.233 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.233 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.233 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:27.233 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.648038+0000", 2024-09-13T11:26:27.233 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.4366e-05 2024-09-13T11:26:27.233 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.233 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.233 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:26:27.234 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.648131+0000", 2024-09-13T11:26:27.234 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.3017999999999996e-05 2024-09-13T11:26:27.234 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.234 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.234 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1", 2024-09-13T11:26:27.234 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.648200+0000", 2024-09-13T11:26:27.234 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.9244000000000003e-05 2024-09-13T11:26:27.234 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.234 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.234 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:26:27.234 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.649817+0000", 2024-09-13T11:26:27.234 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0016166920000000001 2024-09-13T11:26:27.234 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.234 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.235 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:27.235 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.863329+0000", 2024-09-13T11:26:27.235 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.21351207699999999 2024-09-13T11:26:27.235 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.235 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.235 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:26:27.235 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.863524+0000", 2024-09-13T11:26:27.235 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00019551 2024-09-13T11:26:27.235 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.235 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.235 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:26:27.235 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.863660+0000", 2024-09-13T11:26:27.235 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0001362 2024-09-13T11:26:27.235 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:27.236 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:27.236 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:27.236 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.236 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.236 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:209264 3.19 3:9fb0ed8f:::benchmark_data_smithi177_31116_object68961:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e185)", 2024-09-13T11:26:27.236 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:25:53.655043+0000", 2024-09-13T11:26:27.236 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 33.519519330999998, 2024-09-13T11:26:27.236 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.232156005, 2024-09-13T11:26:27.236 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:26:27.236 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:26:27.236 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:27.236 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:26:27.236 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:26:27.236 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:27.237 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 209264 2024-09-13T11:26:27.237 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.237 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:26:27.237 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.237 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:26:27.237 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.655043+0000", 2024-09-13T11:26:27.237 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:27.237 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.237 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.237 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:26:27.237 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.655043+0000", 2024-09-13T11:26:27.237 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:27.237 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.237 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.238 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:26:27.238 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.655045+0000", 2024-09-13T11:26:27.238 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.7090000000000001e-06 2024-09-13T11:26:27.238 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.238 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.238 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:26:27.238 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.655051+0000", 2024-09-13T11:26:27.238 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.3239999999999999e-06 2024-09-13T11:26:27.238 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.238 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.238 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:26:27.238 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.655065+0000", 2024-09-13T11:26:27.238 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.4137e-05 2024-09-13T11:26:27.238 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.238 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.239 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:26:27.239 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.655072+0000", 2024-09-13T11:26:27.239 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.827e-06 2024-09-13T11:26:27.239 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.239 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.239 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:27.239 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.655133+0000", 2024-09-13T11:26:27.239 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.0912000000000002e-05 2024-09-13T11:26:27.239 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.239 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.239 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:26:27.239 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.655228+0000", 2024-09-13T11:26:27.239 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.5440999999999996e-05 2024-09-13T11:26:27.239 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.239 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.240 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1", 2024-09-13T11:26:27.240 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.655296+0000", 2024-09-13T11:26:27.240 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.7316000000000004e-05 2024-09-13T11:26:27.240 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.240 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.240 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:26:27.240 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.658618+0000", 2024-09-13T11:26:27.240 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0033226179999999998 2024-09-13T11:26:27.240 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.240 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.240 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:27.240 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.887043+0000", 2024-09-13T11:26:27.240 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.22842430699999999 2024-09-13T11:26:27.241 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.241 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.241 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:26:27.241 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.887132+0000", 2024-09-13T11:26:27.241 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.9635000000000005e-05 2024-09-13T11:26:27.241 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.241 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.241 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:26:27.241 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.887199+0000", 2024-09-13T11:26:27.241 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.6779000000000006e-05 2024-09-13T11:26:27.241 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:27.241 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:27.241 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:27.241 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.241 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.242 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:209273 3.24 3:24d19ad8:::benchmark_data_smithi177_31116_object68970:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e185)", 2024-09-13T11:26:27.242 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:25:53.660601+0000", 2024-09-13T11:26:27.242 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 33.513961107999997, 2024-09-13T11:26:27.242 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.203427316, 2024-09-13T11:26:27.242 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:26:27.242 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:26:27.242 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:27.242 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:26:27.242 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:26:27.242 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:27.242 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 209273 2024-09-13T11:26:27.242 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.242 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:26:27.242 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.243 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:26:27.243 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.660601+0000", 2024-09-13T11:26:27.243 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:27.243 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.243 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.243 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:26:27.243 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.660601+0000", 2024-09-13T11:26:27.243 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:27.243 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.243 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.243 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:26:27.243 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.660603+0000", 2024-09-13T11:26:27.243 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.7039999999999999e-06 2024-09-13T11:26:27.243 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.243 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.244 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:26:27.244 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.660609+0000", 2024-09-13T11:26:27.244 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.5910000000000003e-06 2024-09-13T11:26:27.244 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.244 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.244 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:26:27.244 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.660623+0000", 2024-09-13T11:26:27.244 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.4338000000000001e-05 2024-09-13T11:26:27.244 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.244 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.244 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:26:27.244 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.660631+0000", 2024-09-13T11:26:27.244 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.1349999999999992e-06 2024-09-13T11:26:27.244 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.245 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.245 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:27.245 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.660694+0000", 2024-09-13T11:26:27.245 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.2755000000000005e-05 2024-09-13T11:26:27.245 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.245 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.245 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:26:27.245 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.660816+0000", 2024-09-13T11:26:27.245 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000121716 2024-09-13T11:26:27.245 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.245 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.245 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1", 2024-09-13T11:26:27.245 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.660900+0000", 2024-09-13T11:26:27.245 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.4924000000000002e-05 2024-09-13T11:26:27.246 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.246 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.246 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:26:27.246 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.662468+0000", 2024-09-13T11:26:27.246 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0015679279999999999 2024-09-13T11:26:27.246 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.246 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.246 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:27.246 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.863779+0000", 2024-09-13T11:26:27.246 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.201310406 2024-09-13T11:26:27.246 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.246 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.246 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:26:27.246 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.863921+0000", 2024-09-13T11:26:27.247 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00014240100000000001 2024-09-13T11:26:27.247 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.247 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.247 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:26:27.247 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.864029+0000", 2024-09-13T11:26:27.247 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00010741799999999999 2024-09-13T11:26:27.247 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:27.247 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:27.247 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:27.247 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.247 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.247 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:209298 3.19 3:9e1ecea7:::benchmark_data_smithi177_31116_object68995:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e185)", 2024-09-13T11:26:27.247 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:25:53.682311+0000", 2024-09-13T11:26:27.247 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 33.492250886000001, 2024-09-13T11:26:27.248 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.20511669699999999, 2024-09-13T11:26:27.248 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:26:27.248 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:26:27.248 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:27.248 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:26:27.248 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:26:27.248 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:27.248 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 209298 2024-09-13T11:26:27.248 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.248 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:26:27.248 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.248 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:26:27.248 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.682311+0000", 2024-09-13T11:26:27.248 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:27.248 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.249 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.249 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:26:27.249 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.682311+0000", 2024-09-13T11:26:27.249 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:27.249 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.249 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.249 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:26:27.249 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.682314+0000", 2024-09-13T11:26:27.249 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.588e-06 2024-09-13T11:26:27.249 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.249 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.249 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:26:27.249 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.682320+0000", 2024-09-13T11:26:27.249 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.3949999999999998e-06 2024-09-13T11:26:27.250 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.250 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.250 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:26:27.250 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.682335+0000", 2024-09-13T11:26:27.250 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.4789e-05 2024-09-13T11:26:27.250 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.250 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.250 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:26:27.250 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.682343+0000", 2024-09-13T11:26:27.250 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.3919999999999997e-06 2024-09-13T11:26:27.250 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.250 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.250 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:27.250 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.682456+0000", 2024-09-13T11:26:27.251 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00011341999999999999 2024-09-13T11:26:27.251 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.251 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.251 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:26:27.251 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.682583+0000", 2024-09-13T11:26:27.251 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00012655500000000001 2024-09-13T11:26:27.251 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.251 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.251 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1", 2024-09-13T11:26:27.251 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.682681+0000", 2024-09-13T11:26:27.251 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.8158000000000004e-05 2024-09-13T11:26:27.251 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.251 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.251 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:26:27.252 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.685667+0000", 2024-09-13T11:26:27.252 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0029857540000000002 2024-09-13T11:26:27.252 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.252 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.252 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:27.252 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.887298+0000", 2024-09-13T11:26:27.252 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.201631591 2024-09-13T11:26:27.252 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.252 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.252 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:26:27.252 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.887369+0000", 2024-09-13T11:26:27.252 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.0743999999999999e-05 2024-09-13T11:26:27.252 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.252 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.252 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:26:27.253 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.887428+0000", 2024-09-13T11:26:27.253 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.9311000000000002e-05 2024-09-13T11:26:27.253 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:27.253 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:27.253 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:27.253 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.253 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.253 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:221512 3.24 3:25f85d9e:::benchmark_data_smithi177_31116_object81209:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e192)", 2024-09-13T11:26:27.253 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:26:05.060948+0000", 2024-09-13T11:26:27.253 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 22.113614764000001, 2024-09-13T11:26:27.253 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.21371807700000001, 2024-09-13T11:26:27.253 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:26:27.253 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:26:27.254 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:27.254 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:26:27.254 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:26:27.254 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:27.254 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 221512 2024-09-13T11:26:27.254 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.254 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:26:27.254 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.254 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:26:27.254 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:05.060948+0000", 2024-09-13T11:26:27.254 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:27.254 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.254 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.254 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:26:27.255 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:05.060948+0000", 2024-09-13T11:26:27.255 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:27.255 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.255 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.255 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:26:27.255 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:05.060950+0000", 2024-09-13T11:26:27.255 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.6469999999999998e-06 2024-09-13T11:26:27.255 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.255 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.255 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:26:27.255 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:05.060959+0000", 2024-09-13T11:26:27.255 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.2460000000000003e-06 2024-09-13T11:26:27.255 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.255 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.256 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:26:27.256 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:05.060977+0000", 2024-09-13T11:26:27.256 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.8779000000000001e-05 2024-09-13T11:26:27.256 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.256 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.256 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:26:27.256 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:05.060993+0000", 2024-09-13T11:26:27.256 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.5464999999999998e-05 2024-09-13T11:26:27.256 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.256 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.256 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:27.256 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:05.062155+0000", 2024-09-13T11:26:27.256 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.001162724 2024-09-13T11:26:27.256 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.257 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.257 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-09-13T11:26:27.257 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:05.062181+0000", 2024-09-13T11:26:27.257 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.5157e-05 2024-09-13T11:26:27.257 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.257 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.257 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:27.257 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:05.257730+0000", 2024-09-13T11:26:27.257 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.19554980299999999 2024-09-13T11:26:27.257 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.257 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.257 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for degraded object", 2024-09-13T11:26:27.257 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:05.258342+0000", 2024-09-13T11:26:27.257 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00061164900000000003 2024-09-13T11:26:27.258 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.258 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.258 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:27.258 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:05.266413+0000", 2024-09-13T11:26:27.258 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0080705780000000001 2024-09-13T11:26:27.258 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.258 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.258 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:26:27.258 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:05.266509+0000", 2024-09-13T11:26:27.258 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.6644999999999994e-05 2024-09-13T11:26:27.258 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.258 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.258 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 14", 2024-09-13T11:26:27.259 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:05.266646+0000", 2024-09-13T11:26:27.259 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00013685499999999999 2024-09-13T11:26:27.259 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.259 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.259 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:27.259 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:05.273976+0000", 2024-09-13T11:26:27.259 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0073293719999999998 2024-09-13T11:26:27.259 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.259 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.259 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:26:27.259 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:05.273999+0000", 2024-09-13T11:26:27.259 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.3252e-05 2024-09-13T11:26:27.259 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.259 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.259 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:26:27.260 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:05.274075+0000", 2024-09-13T11:26:27.260 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.6415999999999995e-05 2024-09-13T11:26:27.260 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.260 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.260 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:26:27.260 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:05.274666+0000", 2024-09-13T11:26:27.260 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00059048900000000003 2024-09-13T11:26:27.260 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:27.260 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:27.260 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:27.260 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.260 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.260 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "MOSDPGPushReply(3.24 192/191 [PushReplyOp(3:26a69d79:::benchmark_data_smithi177_31116_object86105:head)])", 2024-09-13T11:26:27.260 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:26:05.749387+0000", 2024-09-13T11:26:27.261 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 21.425174941000002, 2024-09-13T11:26:27.261 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.34518297100000001, 2024-09-13T11:26:27.261 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:26:27.261 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:26:27.261 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "reached pg", 2024-09-13T11:26:27.261 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:26:27.261 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.261 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:26:27.261 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:05.749387+0000", 2024-09-13T11:26:27.261 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:27.261 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.261 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.261 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:26:27.261 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:05.749387+0000", 2024-09-13T11:26:27.262 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:27.262 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.262 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.262 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:26:27.262 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:05.749391+0000", 2024-09-13T11:26:27.262 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.54e-06 2024-09-13T11:26:27.262 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.262 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.262 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:26:27.262 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:05.749403+0000", 2024-09-13T11:26:27.262 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.1573000000000001e-05 2024-09-13T11:26:27.262 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.262 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.262 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:26:27.263 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:06.092306+0000", 2024-09-13T11:26:27.263 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.34290318600000003 2024-09-13T11:26:27.263 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.263 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.263 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:26:27.263 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:06.092334+0000", 2024-09-13T11:26:27.263 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.7993000000000001e-05 2024-09-13T11:26:27.263 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.263 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.263 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:27.263 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:06.092512+0000", 2024-09-13T11:26:27.263 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000178658 2024-09-13T11:26:27.263 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.263 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.264 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:26:27.264 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:06.094570+0000", 2024-09-13T11:26:27.264 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0020580210000000002 2024-09-13T11:26:27.264 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:27.264 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:27.264 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:27.264 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.264 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.264 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:228454 3.24 3:25beae58:::benchmark_data_smithi177_31116_object88151:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e198)", 2024-09-13T11:26:27.264 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:26:12.032335+0000", 2024-09-13T11:26:27.264 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 15.142227403, 2024-09-13T11:26:27.264 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.206015589, 2024-09-13T11:26:27.264 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:26:27.264 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:26:27.265 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:27.265 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:26:27.265 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:26:27.265 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:27.265 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 228454 2024-09-13T11:26:27.265 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.265 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:26:27.265 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.265 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:26:27.265 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:12.032335+0000", 2024-09-13T11:26:27.265 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:27.265 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.265 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.265 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:26:27.266 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:12.032335+0000", 2024-09-13T11:26:27.266 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:27.266 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.266 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.266 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:26:27.266 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:12.032337+0000", 2024-09-13T11:26:27.266 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.2290000000000002e-06 2024-09-13T11:26:27.266 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.266 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.266 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:26:27.266 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:12.032342+0000", 2024-09-13T11:26:27.266 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.0200000000000002e-06 2024-09-13T11:26:27.266 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.266 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.266 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:26:27.267 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:12.032353+0000", 2024-09-13T11:26:27.267 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.1168e-05 2024-09-13T11:26:27.267 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.267 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.267 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:26:27.267 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:12.032360+0000", 2024-09-13T11:26:27.267 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.7209999999999997e-06 2024-09-13T11:26:27.267 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.267 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.267 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:27.267 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:12.032430+0000", 2024-09-13T11:26:27.267 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.9651000000000001e-05 2024-09-13T11:26:27.267 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.267 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.268 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:26:27.268 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:12.032600+0000", 2024-09-13T11:26:27.268 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00017053000000000001 2024-09-13T11:26:27.268 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.268 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.268 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 14", 2024-09-13T11:26:27.268 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:12.032732+0000", 2024-09-13T11:26:27.268 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00013139099999999999 2024-09-13T11:26:27.268 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.268 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.268 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:26:27.268 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:12.034663+0000", 2024-09-13T11:26:27.268 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.001931012 2024-09-13T11:26:27.268 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.269 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.269 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:27.269 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:12.237696+0000", 2024-09-13T11:26:27.269 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.203033767 2024-09-13T11:26:27.269 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.269 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.269 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:26:27.269 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:12.237753+0000", 2024-09-13T11:26:27.269 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.6146000000000001e-05 2024-09-13T11:26:27.269 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.269 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.269 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:26:27.269 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:12.238351+0000", 2024-09-13T11:26:27.269 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00059795400000000004 2024-09-13T11:26:27.270 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:27.270 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:27.270 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:27.270 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.270 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.270 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "MOSDPGPushReply(3.24 200/191 [PushReplyOp(3:243cc29e:::benchmark_data_smithi177_31116_object132309:head)])", 2024-09-13T11:26:27.270 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:26:16.643152+0000", 2024-09-13T11:26:27.270 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 10.531409977999999, 2024-09-13T11:26:27.270 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.222406362, 2024-09-13T11:26:27.270 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:26:27.270 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:26:27.270 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "reached pg", 2024-09-13T11:26:27.270 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:26:27.270 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.270 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:26:27.271 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:16.643152+0000", 2024-09-13T11:26:27.271 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:27.271 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.271 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.271 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:26:27.271 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:16.643152+0000", 2024-09-13T11:26:27.271 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:27.271 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.271 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.271 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:26:27.271 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:16.643154+0000", 2024-09-13T11:26:27.271 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.0080000000000001e-06 2024-09-13T11:26:27.271 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.271 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.272 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:26:27.272 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:16.643166+0000", 2024-09-13T11:26:27.272 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.1739999999999999e-05 2024-09-13T11:26:27.272 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.272 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.272 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:26:27.272 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:16.864486+0000", 2024-09-13T11:26:27.272 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.221319708 2024-09-13T11:26:27.272 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.272 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.272 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:26:27.272 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:16.864501+0000", 2024-09-13T11:26:27.272 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.5226e-05 2024-09-13T11:26:27.272 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.273 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.273 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:27.273 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:16.864629+0000", 2024-09-13T11:26:27.273 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00012821499999999999 2024-09-13T11:26:27.273 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.273 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.273 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:26:27.273 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:16.865559+0000", 2024-09-13T11:26:27.273 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00092946499999999998 2024-09-13T11:26:27.273 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:27.273 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:27.273 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:27.273 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:27.273 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:27.273 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-09-13T11:26:27.275 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_ops_in_flight 2024-09-13T11:26:27.275 INFO:tasks.ceph.osd.1.smithi092.stderr:2024-09-13T11:26:27.272+0000 7fe7789c5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33718) UID: 0 2024-09-13T11:26:27.376 INFO:tasks.ceph.osd.1.smithi092.stderr:2024-09-13T11:26:27.376+0000 7fe7789c5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33718) UID: 0 2024-09-13T11:26:27.448 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-09-13T11:26:27.448 INFO:teuthology.orchestra.run.smithi092.stdout: "ops": [ 2024-09-13T11:26:27.448 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.449 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:244665 3.1 3:82012079:::benchmark_data_smithi177_31116_object104362:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e208)", 2024-09-13T11:26:27.449 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:26:27.448608+0000", 2024-09-13T11:26:27.449 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 0.00020640599999999999, 2024-09-13T11:26:27.449 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00022433199999999999, 2024-09-13T11:26:27.449 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:26:27.449 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:26:27.449 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-09-13T11:26:27.449 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:26:27.449 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:26:27.449 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:27.449 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 244665 2024-09-13T11:26:27.449 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.449 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:26:27.449 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.450 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:26:27.450 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:27.448608+0000", 2024-09-13T11:26:27.450 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:27.450 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.450 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.450 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:26:27.450 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:27.448608+0000", 2024-09-13T11:26:27.450 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:27.450 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.450 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.450 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:26:27.450 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:27.448611+0000", 2024-09-13T11:26:27.450 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.2639999999999998e-06 2024-09-13T11:26:27.450 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.451 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.451 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:26:27.451 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:27.448616+0000", 2024-09-13T11:26:27.451 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.6529999999999999e-06 2024-09-13T11:26:27.451 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.451 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.451 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:26:27.451 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:27.448630+0000", 2024-09-13T11:26:27.451 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.3453e-05 2024-09-13T11:26:27.451 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.451 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.451 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:26:27.451 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:27.448636+0000", 2024-09-13T11:26:27.451 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.2639999999999997e-06 2024-09-13T11:26:27.452 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.452 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.452 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:27.452 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:27.448693+0000", 2024-09-13T11:26:27.452 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.7037999999999998e-05 2024-09-13T11:26:27.452 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.452 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.452 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:26:27.452 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:27.448836+0000", 2024-09-13T11:26:27.452 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00014356699999999999 2024-09-13T11:26:27.452 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:27.452 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:27.452 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:27.452 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:27.453 INFO:teuthology.orchestra.run.smithi092.stdout: ], 2024-09-13T11:26:27.453 INFO:teuthology.orchestra.run.smithi092.stdout: "num_ops": 1 2024-09-13T11:26:27.453 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-09-13T11:26:27.464 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_blocked_ops 2024-09-13T11:26:27.505 INFO:tasks.ceph.osd.1.smithi092.stderr:2024-09-13T11:26:27.504+0000 7fe7789c5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33718) UID: 0 2024-09-13T11:26:27.581 INFO:tasks.ceph.osd.2.smithi162.stderr:2024-09-13T11:26:27.575+0000 7fe92b35a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26313) UID: 0 2024-09-13T11:26:27.629 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-09-13T11:26:27.629 INFO:teuthology.orchestra.run.smithi092.stdout: "ops": [], 2024-09-13T11:26:27.630 INFO:teuthology.orchestra.run.smithi092.stdout: "complaint_time": 30, 2024-09-13T11:26:27.630 INFO:teuthology.orchestra.run.smithi092.stdout: "num_blocked_ops": 0 2024-09-13T11:26:27.630 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-09-13T11:26:27.646 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_historic_ops 2024-09-13T11:26:27.692 INFO:tasks.ceph.osd.1.smithi092.stderr:2024-09-13T11:26:27.692+0000 7fe7789c5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33718) UID: 0 2024-09-13T11:26:27.780 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-09-13T11:26:27.780 INFO:teuthology.orchestra.run.smithi092.stdout: "size": 20, 2024-09-13T11:26:27.781 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 600, 2024-09-13T11:26:27.781 INFO:teuthology.orchestra.run.smithi092.stdout: "ops": [ 2024-09-13T11:26:27.781 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.781 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:79 3.1 3:841df5b5:::benchmark_data_smithi177_31116_object78:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-09-13T11:26:27.781 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:27.056083+0000", 2024-09-13T11:26:27.781 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 240.72058797, 2024-09-13T11:26:27.781 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.89469730199999997, 2024-09-13T11:26:27.781 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:26:27.781 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:26:27.781 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:27.781 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:26:27.781 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:26:27.781 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:27.782 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 79 2024-09-13T11:26:27.784 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.784 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:26:27.784 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.784 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:26:27.784 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.056083+0000", 2024-09-13T11:26:27.784 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:27.784 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.784 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.784 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:26:27.784 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.056083+0000", 2024-09-13T11:26:27.785 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:27.785 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.785 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.785 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:26:27.785 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.056087+0000", 2024-09-13T11:26:27.785 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.0659999999999997e-06 2024-09-13T11:26:27.785 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.785 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.785 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:26:27.785 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.056148+0000", 2024-09-13T11:26:27.785 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.1376000000000003e-05 2024-09-13T11:26:27.785 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.785 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.785 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:26:27.786 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.056180+0000", 2024-09-13T11:26:27.786 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.2370999999999999e-05 2024-09-13T11:26:27.786 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.786 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.786 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:26:27.786 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.056195+0000", 2024-09-13T11:26:27.786 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.5119e-05 2024-09-13T11:26:27.786 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.786 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.786 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:27.786 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.056304+0000", 2024-09-13T11:26:27.786 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000108365 2024-09-13T11:26:27.786 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.786 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.786 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:26:27.787 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.056478+0000", 2024-09-13T11:26:27.787 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000174542 2024-09-13T11:26:27.787 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.787 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.787 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 11", 2024-09-13T11:26:27.787 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.056690+0000", 2024-09-13T11:26:27.787 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00021167500000000001 2024-09-13T11:26:27.787 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.787 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.787 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:26:27.787 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.058063+0000", 2024-09-13T11:26:27.787 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0013731819999999999 2024-09-13T11:26:27.787 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.787 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.788 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:27.788 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.950477+0000", 2024-09-13T11:26:27.788 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.89241369400000004 2024-09-13T11:26:27.788 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.788 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.788 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:26:27.788 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.950652+0000", 2024-09-13T11:26:27.788 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00017526400000000001 2024-09-13T11:26:27.788 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.788 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.788 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:26:27.788 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.950780+0000", 2024-09-13T11:26:27.788 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00012764800000000001 2024-09-13T11:26:27.789 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:27.789 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:27.789 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:27.789 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.789 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.789 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:7 3.b 3:de78393b:::benchmark_data_smithi177_31116_object6:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T11:26:27.789 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:48.313952+0000", 2024-09-13T11:26:27.789 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 219.46271822099999, 2024-09-13T11:26:27.789 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.86421288600000001, 2024-09-13T11:26:27.789 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:26:27.789 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:26:27.789 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:27.789 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:26:27.789 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:26:27.790 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:27.790 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 7 2024-09-13T11:26:27.790 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.790 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:26:27.790 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.790 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:26:27.790 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.313952+0000", 2024-09-13T11:26:27.790 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:27.790 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.790 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.790 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:26:27.790 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.313952+0000", 2024-09-13T11:26:27.790 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:27.790 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.790 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.791 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:26:27.791 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.313957+0000", 2024-09-13T11:26:27.791 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.3599999999999998e-06 2024-09-13T11:26:27.791 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.791 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.791 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:26:27.791 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314043+0000", 2024-09-13T11:26:27.791 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.6766000000000002e-05 2024-09-13T11:26:27.791 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.791 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.791 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:26:27.791 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314065+0000", 2024-09-13T11:26:27.791 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.1416e-05 2024-09-13T11:26:27.791 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.792 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.792 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:26:27.792 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314080+0000", 2024-09-13T11:26:27.792 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.5519000000000002e-05 2024-09-13T11:26:27.792 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.792 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.792 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:27.792 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315478+0000", 2024-09-13T11:26:27.792 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.001397765 2024-09-13T11:26:27.792 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.792 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.792 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-09-13T11:26:27.793 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315507+0000", 2024-09-13T11:26:27.793 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.9128999999999998e-05 2024-09-13T11:26:27.793 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.793 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.793 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:27.793 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.167978+0000", 2024-09-13T11:26:27.793 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.85247068400000003 2024-09-13T11:26:27.793 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.794 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.794 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:26:27.794 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.168307+0000", 2024-09-13T11:26:27.794 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00032886699999999999 2024-09-13T11:26:27.794 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.794 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.795 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1", 2024-09-13T11:26:27.795 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.168647+0000", 2024-09-13T11:26:27.795 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00033998399999999999 2024-09-13T11:26:27.795 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.795 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.795 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:27.795 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.178011+0000", 2024-09-13T11:26:27.795 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0093639169999999994 2024-09-13T11:26:27.795 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.795 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.795 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:26:27.795 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.178026+0000", 2024-09-13T11:26:27.795 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.5274999999999999e-05 2024-09-13T11:26:27.795 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.796 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.796 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:26:27.796 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.178080+0000", 2024-09-13T11:26:27.796 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.3601999999999999e-05 2024-09-13T11:26:27.796 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.796 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.796 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:26:27.796 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.178165+0000", 2024-09-13T11:26:27.796 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.5601999999999998e-05 2024-09-13T11:26:27.796 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:27.796 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:27.796 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:27.796 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.796 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.797 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:9 3.b 3:dafadff6:::benchmark_data_smithi177_31116_object8:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T11:26:27.797 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:48.314114+0000", 2024-09-13T11:26:27.797 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 219.46255640699999, 2024-09-13T11:26:27.797 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.86613615899999996, 2024-09-13T11:26:27.797 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:26:27.797 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:26:27.797 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:27.797 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:26:27.797 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:26:27.797 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:27.797 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 9 2024-09-13T11:26:27.797 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.797 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:26:27.798 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.798 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:26:27.798 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314114+0000", 2024-09-13T11:26:27.798 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:27.798 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.798 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.798 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:26:27.798 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314114+0000", 2024-09-13T11:26:27.798 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:27.798 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.798 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.798 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:26:27.799 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314117+0000", 2024-09-13T11:26:27.799 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.3860000000000001e-06 2024-09-13T11:26:27.799 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.799 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.799 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:26:27.799 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314215+0000", 2024-09-13T11:26:27.799 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.8900000000000005e-05 2024-09-13T11:26:27.799 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.799 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.799 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:26:27.799 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314229+0000", 2024-09-13T11:26:27.800 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.3444999999999999e-05 2024-09-13T11:26:27.800 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.800 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.800 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:26:27.800 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314239+0000", 2024-09-13T11:26:27.800 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.8470000000000008e-06 2024-09-13T11:26:27.800 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.800 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.800 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:27.800 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315648+0000", 2024-09-13T11:26:27.800 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.001409291 2024-09-13T11:26:27.800 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.800 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.801 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-09-13T11:26:27.801 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315671+0000", 2024-09-13T11:26:27.801 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.3169000000000001e-05 2024-09-13T11:26:27.801 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.801 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.801 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:27.801 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.176036+0000", 2024-09-13T11:26:27.801 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.86036497499999998 2024-09-13T11:26:27.801 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.801 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.801 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:26:27.801 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.176189+0000", 2024-09-13T11:26:27.801 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00015271100000000001 2024-09-13T11:26:27.801 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.802 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.802 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1", 2024-09-13T11:26:27.802 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.176339+0000", 2024-09-13T11:26:27.802 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00015057499999999999 2024-09-13T11:26:27.802 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.802 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.802 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:27.802 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.180148+0000", 2024-09-13T11:26:27.802 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.003808891 2024-09-13T11:26:27.802 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.802 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.802 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:26:27.802 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.180168+0000", 2024-09-13T11:26:27.803 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.9507000000000001e-05 2024-09-13T11:26:27.803 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.803 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.803 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:26:27.803 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.180213+0000", 2024-09-13T11:26:27.803 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.5504999999999999e-05 2024-09-13T11:26:27.803 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.803 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.803 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:26:27.803 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.180250+0000", 2024-09-13T11:26:27.803 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.6956999999999999e-05 2024-09-13T11:26:27.803 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:27.804 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:27.804 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:27.804 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.804 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.804 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:10 3.b 3:dbe028a8:::benchmark_data_smithi177_31116_object9:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T11:26:27.804 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:48.314259+0000", 2024-09-13T11:26:27.804 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 219.462411277, 2024-09-13T11:26:27.804 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.86637313599999999, 2024-09-13T11:26:27.804 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:26:27.804 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:26:27.804 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:27.805 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:26:27.805 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:26:27.805 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:27.805 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 10 2024-09-13T11:26:27.805 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.805 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:26:27.805 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.805 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:26:27.805 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314259+0000", 2024-09-13T11:26:27.805 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:27.805 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.805 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.806 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:26:27.806 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314259+0000", 2024-09-13T11:26:27.806 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:27.806 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.806 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.806 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:26:27.806 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314261+0000", 2024-09-13T11:26:27.806 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.1500000000000002e-06 2024-09-13T11:26:27.806 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.806 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.806 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:26:27.807 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314349+0000", 2024-09-13T11:26:27.807 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.8070000000000002e-05 2024-09-13T11:26:27.807 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.807 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.807 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:26:27.807 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314365+0000", 2024-09-13T11:26:27.807 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.5835999999999999e-05 2024-09-13T11:26:27.807 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.807 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.807 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:26:27.807 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314377+0000", 2024-09-13T11:26:27.807 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.1589e-05 2024-09-13T11:26:27.807 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.808 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.808 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:27.808 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315788+0000", 2024-09-13T11:26:27.808 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0014109579999999999 2024-09-13T11:26:27.808 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.808 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.808 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-09-13T11:26:27.808 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315809+0000", 2024-09-13T11:26:27.808 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.0985999999999999e-05 2024-09-13T11:26:27.808 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.808 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.808 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:27.809 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.176822+0000", 2024-09-13T11:26:27.809 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.86101310799999997 2024-09-13T11:26:27.809 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.809 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.809 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:26:27.809 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.177021+0000", 2024-09-13T11:26:27.809 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00019911199999999999 2024-09-13T11:26:27.809 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.809 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.809 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1", 2024-09-13T11:26:27.809 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.177226+0000", 2024-09-13T11:26:27.809 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00020516600000000001 2024-09-13T11:26:27.810 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.810 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.810 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:26:27.810 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.180264+0000", 2024-09-13T11:26:27.810 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0030381739999999998 2024-09-13T11:26:27.811 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.811 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.811 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:27.811 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.180387+0000", 2024-09-13T11:26:27.811 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00012257699999999999 2024-09-13T11:26:27.811 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.811 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.811 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:26:27.811 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.180518+0000", 2024-09-13T11:26:27.811 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00013135800000000001 2024-09-13T11:26:27.811 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.811 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.811 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:26:27.812 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.180632+0000", 2024-09-13T11:26:27.812 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000114052 2024-09-13T11:26:27.812 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:27.812 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:27.812 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:27.812 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.812 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.812 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:39 3.b 3:d91854b3:::benchmark_data_smithi177_31116_object38:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T11:26:27.812 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:48.314393+0000", 2024-09-13T11:26:27.813 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 219.46227773499999, 2024-09-13T11:26:27.813 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.86850121700000005, 2024-09-13T11:26:27.813 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:26:27.813 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:26:27.813 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:27.813 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:26:27.813 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:26:27.813 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:27.813 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 39 2024-09-13T11:26:27.813 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.813 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:26:27.813 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.813 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:26:27.814 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314393+0000", 2024-09-13T11:26:27.814 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:27.814 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.814 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.814 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:26:27.814 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314393+0000", 2024-09-13T11:26:27.814 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:27.814 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.814 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.814 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:26:27.814 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314394+0000", 2024-09-13T11:26:27.814 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.513e-06 2024-09-13T11:26:27.814 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.815 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.815 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:26:27.815 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314463+0000", 2024-09-13T11:26:27.815 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.8261000000000003e-05 2024-09-13T11:26:27.815 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.815 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.816 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:26:27.816 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314477+0000", 2024-09-13T11:26:27.816 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.4619e-05 2024-09-13T11:26:27.816 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.816 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.816 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:26:27.816 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314485+0000", 2024-09-13T11:26:27.816 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.1699999999999997e-06 2024-09-13T11:26:27.816 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.816 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.816 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:27.816 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315917+0000", 2024-09-13T11:26:27.816 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0014316089999999999 2024-09-13T11:26:27.817 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.817 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.817 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-09-13T11:26:27.817 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315933+0000", 2024-09-13T11:26:27.817 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.5999999999999999e-05 2024-09-13T11:26:27.817 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.817 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.817 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:27.817 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.178205+0000", 2024-09-13T11:26:27.817 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.86227247100000004 2024-09-13T11:26:27.817 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.817 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.817 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:26:27.817 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.178316+0000", 2024-09-13T11:26:27.818 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000110788 2024-09-13T11:26:27.818 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.818 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.818 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1", 2024-09-13T11:26:27.818 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.178448+0000", 2024-09-13T11:26:27.818 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000132056 2024-09-13T11:26:27.818 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.818 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.818 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:26:27.818 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.182721+0000", 2024-09-13T11:26:27.818 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0042729780000000002 2024-09-13T11:26:27.818 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.818 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.818 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:27.818 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.182801+0000", 2024-09-13T11:26:27.819 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.0209000000000004e-05 2024-09-13T11:26:27.819 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.819 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.819 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:26:27.819 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.182855+0000", 2024-09-13T11:26:27.819 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.3749000000000003e-05 2024-09-13T11:26:27.819 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.819 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.819 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:26:27.819 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.182894+0000", 2024-09-13T11:26:27.819 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.8794000000000002e-05 2024-09-13T11:26:27.819 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:27.819 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:27.819 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:27.820 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.820 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.820 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:50 3.b 3:d1c144b8:::benchmark_data_smithi177_31116_object49:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T11:26:27.820 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:48.314498+0000", 2024-09-13T11:26:27.820 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 219.46217232000001, 2024-09-13T11:26:27.820 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.86878673299999998, 2024-09-13T11:26:27.820 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:26:27.820 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:26:27.820 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:27.820 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:26:27.820 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:26:27.820 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:27.820 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 50 2024-09-13T11:26:27.820 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.821 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:26:27.821 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.821 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:26:27.821 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314498+0000", 2024-09-13T11:26:27.821 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:27.821 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.821 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.821 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:26:27.821 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314498+0000", 2024-09-13T11:26:27.821 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:27.821 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.821 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.821 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:26:27.821 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314499+0000", 2024-09-13T11:26:27.822 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.1969999999999999e-06 2024-09-13T11:26:27.822 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.822 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.822 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:26:27.822 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314565+0000", 2024-09-13T11:26:27.822 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.5628000000000003e-05 2024-09-13T11:26:27.822 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.822 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.822 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:26:27.822 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314579+0000", 2024-09-13T11:26:27.822 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.4222e-05 2024-09-13T11:26:27.822 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.822 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.822 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:26:27.823 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314590+0000", 2024-09-13T11:26:27.823 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.1240000000000001e-05 2024-09-13T11:26:27.823 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.823 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.823 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:27.823 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316060+0000", 2024-09-13T11:26:27.823 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.001469515 2024-09-13T11:26:27.823 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.823 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.823 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-09-13T11:26:27.823 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316076+0000", 2024-09-13T11:26:27.823 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.5608999999999999e-05 2024-09-13T11:26:27.824 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.824 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.824 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:27.824 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.178857+0000", 2024-09-13T11:26:27.824 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.86278099900000005 2024-09-13T11:26:27.824 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.824 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.824 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:26:27.824 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.178961+0000", 2024-09-13T11:26:27.824 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000104664 2024-09-13T11:26:27.825 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.825 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.825 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1", 2024-09-13T11:26:27.825 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.179123+0000", 2024-09-13T11:26:27.825 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000161738 2024-09-13T11:26:27.825 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.825 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.825 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:26:27.825 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.182755+0000", 2024-09-13T11:26:27.825 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0036323200000000001 2024-09-13T11:26:27.825 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.825 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.825 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:27.825 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.183023+0000", 2024-09-13T11:26:27.826 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00026749100000000002 2024-09-13T11:26:27.826 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.826 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.826 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:26:27.826 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.183168+0000", 2024-09-13T11:26:27.826 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00014499099999999999 2024-09-13T11:26:27.826 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.826 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.826 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:26:27.826 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.183285+0000", 2024-09-13T11:26:27.826 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000117119 2024-09-13T11:26:27.826 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:27.826 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:27.826 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:27.826 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.827 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.827 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:60 3.b 3:d67218b0:::benchmark_data_smithi177_31116_object59:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T11:26:27.827 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:48.314603+0000", 2024-09-13T11:26:27.827 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 219.462067757, 2024-09-13T11:26:27.827 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.87106201100000002, 2024-09-13T11:26:27.827 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:26:27.827 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:26:27.827 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:27.827 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:26:27.827 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:26:27.827 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:27.828 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 60 2024-09-13T11:26:27.828 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.828 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:26:27.828 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.828 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:26:27.828 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314603+0000", 2024-09-13T11:26:27.828 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:27.829 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.829 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.829 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:26:27.829 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314603+0000", 2024-09-13T11:26:27.829 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:27.829 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.829 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.829 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:26:27.829 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314604+0000", 2024-09-13T11:26:27.829 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.2359999999999999e-06 2024-09-13T11:26:27.829 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.829 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.829 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:26:27.829 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314667+0000", 2024-09-13T11:26:27.830 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.3077999999999995e-05 2024-09-13T11:26:27.830 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.830 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.830 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:26:27.830 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314699+0000", 2024-09-13T11:26:27.830 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.1965999999999999e-05 2024-09-13T11:26:27.830 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.830 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.830 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:26:27.830 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314710+0000", 2024-09-13T11:26:27.830 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.1148999999999999e-05 2024-09-13T11:26:27.830 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.831 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.831 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:27.831 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316291+0000", 2024-09-13T11:26:27.831 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0015804440000000001 2024-09-13T11:26:27.831 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.831 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.831 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-09-13T11:26:27.831 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316310+0000", 2024-09-13T11:26:27.831 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.9799000000000001e-05 2024-09-13T11:26:27.831 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.831 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.831 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:27.831 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.179555+0000", 2024-09-13T11:26:27.831 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.86324469500000001 2024-09-13T11:26:27.832 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.832 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.832 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:26:27.832 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.179666+0000", 2024-09-13T11:26:27.832 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000110443 2024-09-13T11:26:27.832 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.832 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.832 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1", 2024-09-13T11:26:27.832 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.179791+0000", 2024-09-13T11:26:27.832 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00012560499999999999 2024-09-13T11:26:27.832 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.832 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.832 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:26:27.832 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.182761+0000", 2024-09-13T11:26:27.833 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0029697019999999998 2024-09-13T11:26:27.833 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.833 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.833 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:27.833 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.185572+0000", 2024-09-13T11:26:27.833 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.002811512 2024-09-13T11:26:27.833 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.833 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.833 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:26:27.833 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.185625+0000", 2024-09-13T11:26:27.833 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.2716000000000001e-05 2024-09-13T11:26:27.833 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.833 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.834 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:26:27.834 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.185665+0000", 2024-09-13T11:26:27.834 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.9666000000000003e-05 2024-09-13T11:26:27.834 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:27.834 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:27.834 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:27.834 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.834 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.834 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:62 3.b 3:d20ae5a8:::benchmark_data_smithi177_31116_object61:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T11:26:27.834 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:48.314723+0000", 2024-09-13T11:26:27.834 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 219.46194721500001, 2024-09-13T11:26:27.834 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.87338985099999999, 2024-09-13T11:26:27.834 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:26:27.834 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:26:27.834 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:27.835 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:26:27.835 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:26:27.835 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:27.835 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 62 2024-09-13T11:26:27.835 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.835 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:26:27.835 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.835 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:26:27.835 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314723+0000", 2024-09-13T11:26:27.835 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:27.835 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.835 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.835 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:26:27.835 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314723+0000", 2024-09-13T11:26:27.836 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:27.836 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.836 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.836 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:26:27.836 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314725+0000", 2024-09-13T11:26:27.836 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.3069999999999999e-06 2024-09-13T11:26:27.836 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.836 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.836 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:26:27.836 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314797+0000", 2024-09-13T11:26:27.836 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.2652999999999997e-05 2024-09-13T11:26:27.836 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.836 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.837 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:26:27.837 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314809+0000", 2024-09-13T11:26:27.837 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.1496e-05 2024-09-13T11:26:27.837 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.837 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.837 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:26:27.837 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314816+0000", 2024-09-13T11:26:27.837 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.6639999999999998e-06 2024-09-13T11:26:27.837 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.837 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.837 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:27.838 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316424+0000", 2024-09-13T11:26:27.838 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0016077559999999999 2024-09-13T11:26:27.838 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.838 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.838 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-09-13T11:26:27.838 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316441+0000", 2024-09-13T11:26:27.838 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.7305000000000001e-05 2024-09-13T11:26:27.838 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.838 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.838 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:27.838 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.180688+0000", 2024-09-13T11:26:27.838 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.86424692800000003 2024-09-13T11:26:27.838 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.838 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.838 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:26:27.839 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.180807+0000", 2024-09-13T11:26:27.839 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000118977 2024-09-13T11:26:27.839 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.839 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.839 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1", 2024-09-13T11:26:27.839 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.180938+0000", 2024-09-13T11:26:27.839 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00013049700000000001 2024-09-13T11:26:27.839 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.839 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.839 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:26:27.839 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.185680+0000", 2024-09-13T11:26:27.839 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0047417029999999999 2024-09-13T11:26:27.839 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.839 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.840 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:27.840 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.187856+0000", 2024-09-13T11:26:27.840 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0021760550000000001 2024-09-13T11:26:27.840 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.840 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.840 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:26:27.840 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.188007+0000", 2024-09-13T11:26:27.840 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00015185099999999999 2024-09-13T11:26:27.840 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.840 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.840 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:26:27.840 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.188113+0000", 2024-09-13T11:26:27.840 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000105659 2024-09-13T11:26:27.841 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:27.841 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:27.841 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:27.841 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.841 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.841 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:73 3.b 3:d0060d5f:::benchmark_data_smithi177_31116_object72:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T11:26:27.841 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:48.314829+0000", 2024-09-13T11:26:27.841 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 219.46184186599999, 2024-09-13T11:26:27.841 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.87364757199999998, 2024-09-13T11:26:27.841 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:26:27.841 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:26:27.841 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:27.841 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:26:27.841 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:26:27.842 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:27.842 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 73 2024-09-13T11:26:27.842 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.842 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:26:27.842 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.842 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:26:27.842 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314829+0000", 2024-09-13T11:26:27.842 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:27.842 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.842 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.842 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:26:27.842 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314829+0000", 2024-09-13T11:26:27.842 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:27.843 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.843 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.843 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:26:27.843 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314830+0000", 2024-09-13T11:26:27.843 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.243e-06 2024-09-13T11:26:27.843 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.843 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.843 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:26:27.843 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314965+0000", 2024-09-13T11:26:27.843 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00013470800000000001 2024-09-13T11:26:27.843 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.843 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.843 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:26:27.843 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314979+0000", 2024-09-13T11:26:27.843 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.4375999999999999e-05 2024-09-13T11:26:27.844 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.844 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.844 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:26:27.844 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314988+0000", 2024-09-13T11:26:27.844 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.8570000000000001e-06 2024-09-13T11:26:27.844 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.844 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.844 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:27.844 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316545+0000", 2024-09-13T11:26:27.844 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.001556878 2024-09-13T11:26:27.844 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.844 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.844 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-09-13T11:26:27.844 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316556+0000", 2024-09-13T11:26:27.845 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.1453e-05 2024-09-13T11:26:27.845 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.845 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.845 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:27.845 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.182124+0000", 2024-09-13T11:26:27.845 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.86556744200000002 2024-09-13T11:26:27.845 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.845 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.845 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:26:27.845 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.182246+0000", 2024-09-13T11:26:27.845 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00012210500000000001 2024-09-13T11:26:27.845 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.845 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.845 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1", 2024-09-13T11:26:27.846 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.182375+0000", 2024-09-13T11:26:27.846 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00012920999999999999 2024-09-13T11:26:27.846 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.846 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.846 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:26:27.846 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.185684+0000", 2024-09-13T11:26:27.846 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0033093969999999999 2024-09-13T11:26:27.846 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.846 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.846 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:27.846 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.188244+0000", 2024-09-13T11:26:27.846 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0025601550000000002 2024-09-13T11:26:27.846 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.846 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.846 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:26:27.847 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.188379+0000", 2024-09-13T11:26:27.847 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00013463400000000001 2024-09-13T11:26:27.847 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.847 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.847 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:26:27.847 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.188476+0000", 2024-09-13T11:26:27.847 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.7114000000000006e-05 2024-09-13T11:26:27.847 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:27.847 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:27.847 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:27.847 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.847 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.847 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:81 3.b 3:d705fd70:::benchmark_data_smithi177_31116_object80:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T11:26:27.847 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:48.315000+0000", 2024-09-13T11:26:27.848 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 219.461670029, 2024-09-13T11:26:27.848 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.87561418300000005, 2024-09-13T11:26:27.848 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:26:27.848 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:26:27.848 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:27.848 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:26:27.848 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:26:27.848 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:27.848 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 81 2024-09-13T11:26:27.848 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.848 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:26:27.848 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.848 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:26:27.848 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315000+0000", 2024-09-13T11:26:27.849 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:27.849 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.849 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.849 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:26:27.849 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315000+0000", 2024-09-13T11:26:27.849 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:27.849 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.849 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.849 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:26:27.849 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315002+0000", 2024-09-13T11:26:27.849 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.3799999999999999e-06 2024-09-13T11:26:27.849 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.849 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.849 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:26:27.849 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315119+0000", 2024-09-13T11:26:27.850 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000116925 2024-09-13T11:26:27.850 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.850 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.850 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:26:27.850 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315133+0000", 2024-09-13T11:26:27.850 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.3971999999999999e-05 2024-09-13T11:26:27.850 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.850 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.850 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:26:27.850 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315142+0000", 2024-09-13T11:26:27.850 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.4399999999999994e-06 2024-09-13T11:26:27.850 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.850 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.850 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:27.851 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316652+0000", 2024-09-13T11:26:27.851 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.001509907 2024-09-13T11:26:27.851 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.851 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.851 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-09-13T11:26:27.851 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316668+0000", 2024-09-13T11:26:27.851 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.5945999999999999e-05 2024-09-13T11:26:27.851 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.851 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.851 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:27.851 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.183405+0000", 2024-09-13T11:26:27.851 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.866737384 2024-09-13T11:26:27.851 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.851 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.851 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:26:27.852 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.183701+0000", 2024-09-13T11:26:27.852 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00029511099999999997 2024-09-13T11:26:27.852 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.852 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.852 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1", 2024-09-13T11:26:27.852 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.184078+0000", 2024-09-13T11:26:27.852 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00037741000000000001 2024-09-13T11:26:27.852 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.852 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.852 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:26:27.852 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.188504+0000", 2024-09-13T11:26:27.852 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0044260280000000003 2024-09-13T11:26:27.852 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.852 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.853 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:27.853 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.190562+0000", 2024-09-13T11:26:27.853 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0020580099999999999 2024-09-13T11:26:27.853 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.853 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.853 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:26:27.853 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.190592+0000", 2024-09-13T11:26:27.853 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.0057999999999999e-05 2024-09-13T11:26:27.853 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.853 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.853 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:26:27.853 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.190615+0000", 2024-09-13T11:26:27.853 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.2612e-05 2024-09-13T11:26:27.853 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:27.854 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:27.854 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:27.854 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.854 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.854 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:87 3.b 3:dd7de0ad:::benchmark_data_smithi177_31116_object86:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T11:26:27.854 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:48.315155+0000", 2024-09-13T11:26:27.854 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 219.46151523200001, 2024-09-13T11:26:27.854 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.87555916099999997, 2024-09-13T11:26:27.854 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:26:27.854 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:26:27.854 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:27.854 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:26:27.854 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:26:27.854 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:27.854 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 87 2024-09-13T11:26:27.855 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.855 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:26:27.855 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.855 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:26:27.855 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315155+0000", 2024-09-13T11:26:27.855 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:27.855 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.855 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.855 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:26:27.855 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315155+0000", 2024-09-13T11:26:27.855 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:27.855 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.855 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.855 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:26:27.856 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315157+0000", 2024-09-13T11:26:27.856 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.3829999999999999e-06 2024-09-13T11:26:27.856 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.856 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.856 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:26:27.856 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315271+0000", 2024-09-13T11:26:27.856 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000114539 2024-09-13T11:26:27.856 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.856 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.856 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:26:27.856 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315285+0000", 2024-09-13T11:26:27.856 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.4042e-05 2024-09-13T11:26:27.856 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.856 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.857 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:26:27.857 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315295+0000", 2024-09-13T11:26:27.857 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.2739999999999996e-06 2024-09-13T11:26:27.857 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.857 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.857 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:27.857 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316739+0000", 2024-09-13T11:26:27.857 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0014446960000000001 2024-09-13T11:26:27.857 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.857 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.857 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-09-13T11:26:27.857 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316747+0000", 2024-09-13T11:26:27.857 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.0790000000000001e-06 2024-09-13T11:26:27.857 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.858 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.858 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:27.858 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.184922+0000", 2024-09-13T11:26:27.858 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.86817446399999998 2024-09-13T11:26:27.858 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.858 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.858 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:26:27.858 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.185037+0000", 2024-09-13T11:26:27.858 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00011565099999999999 2024-09-13T11:26:27.858 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.858 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.858 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1", 2024-09-13T11:26:27.858 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.185171+0000", 2024-09-13T11:26:27.858 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00013316900000000001 2024-09-13T11:26:27.859 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.859 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.859 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:26:27.859 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.188520+0000", 2024-09-13T11:26:27.859 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0033489150000000001 2024-09-13T11:26:27.859 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.859 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.859 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:27.859 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.190652+0000", 2024-09-13T11:26:27.859 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0021325649999999999 2024-09-13T11:26:27.859 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.859 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.859 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:26:27.859 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.190686+0000", 2024-09-13T11:26:27.860 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.3827e-05 2024-09-13T11:26:27.860 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.860 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.860 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:26:27.860 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.190714+0000", 2024-09-13T11:26:27.860 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.8557000000000001e-05 2024-09-13T11:26:27.860 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:27.860 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:27.860 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:27.860 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.860 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.860 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:99 3.b 3:d011b600:::benchmark_data_smithi177_31116_object98:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T11:26:27.860 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:48.315307+0000", 2024-09-13T11:26:27.860 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 219.46136340499999, 2024-09-13T11:26:27.861 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.87551770799999995, 2024-09-13T11:26:27.861 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:26:27.861 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:26:27.861 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:27.861 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:26:27.861 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:26:27.861 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:27.861 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 99 2024-09-13T11:26:27.861 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.861 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:26:27.861 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.861 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:26:27.861 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315307+0000", 2024-09-13T11:26:27.861 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:27.862 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.862 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.862 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:26:27.862 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315307+0000", 2024-09-13T11:26:27.862 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:27.862 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.862 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.862 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:26:27.862 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315308+0000", 2024-09-13T11:26:27.862 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.316e-06 2024-09-13T11:26:27.862 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.862 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.862 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:26:27.862 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315453+0000", 2024-09-13T11:26:27.862 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00014473200000000001 2024-09-13T11:26:27.863 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.863 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.863 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:26:27.863 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315471+0000", 2024-09-13T11:26:27.863 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.8019000000000001e-05 2024-09-13T11:26:27.863 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.863 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.863 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:26:27.863 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315489+0000", 2024-09-13T11:26:27.863 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.7861999999999999e-05 2024-09-13T11:26:27.863 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.863 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.863 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:27.863 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316837+0000", 2024-09-13T11:26:27.864 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.001347787 2024-09-13T11:26:27.864 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.864 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.864 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-09-13T11:26:27.864 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316858+0000", 2024-09-13T11:26:27.864 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.1103000000000001e-05 2024-09-13T11:26:27.864 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.864 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.864 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:27.864 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.185713+0000", 2024-09-13T11:26:27.864 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.86885540800000005 2024-09-13T11:26:27.864 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.864 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.864 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:26:27.865 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.185820+0000", 2024-09-13T11:26:27.865 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000107081 2024-09-13T11:26:27.865 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.865 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.865 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1", 2024-09-13T11:26:27.865 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.185948+0000", 2024-09-13T11:26:27.865 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00012775800000000001 2024-09-13T11:26:27.865 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.865 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.865 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:26:27.865 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.190527+0000", 2024-09-13T11:26:27.865 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0045783910000000002 2024-09-13T11:26:27.865 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.865 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.865 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:27.866 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.190759+0000", 2024-09-13T11:26:27.866 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00023266199999999999 2024-09-13T11:26:27.866 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.866 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.866 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:26:27.866 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.190789+0000", 2024-09-13T11:26:27.866 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.9666999999999999e-05 2024-09-13T11:26:27.866 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.866 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.866 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:26:27.866 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.190825+0000", 2024-09-13T11:26:27.866 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.5921999999999999e-05 2024-09-13T11:26:27.866 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:27.866 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:27.866 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:27.867 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.867 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.867 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:108 3.b 3:d5c90217:::benchmark_data_smithi177_31116_object107:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T11:26:27.867 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:48.315513+0000", 2024-09-13T11:26:27.867 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 219.4611577, 2024-09-13T11:26:27.867 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.87544772500000001, 2024-09-13T11:26:27.867 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:26:27.867 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:26:27.867 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:27.867 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:26:27.867 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:26:27.868 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:27.868 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 108 2024-09-13T11:26:27.868 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.868 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:26:27.868 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.868 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:26:27.868 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315513+0000", 2024-09-13T11:26:27.868 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:27.868 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.868 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.868 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:26:27.868 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315513+0000", 2024-09-13T11:26:27.868 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:27.868 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.868 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.869 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:26:27.869 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315515+0000", 2024-09-13T11:26:27.869 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.9369999999999998e-06 2024-09-13T11:26:27.869 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.869 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.869 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:26:27.869 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315644+0000", 2024-09-13T11:26:27.869 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000129011 2024-09-13T11:26:27.869 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.869 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.869 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:26:27.869 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315657+0000", 2024-09-13T11:26:27.869 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.3689999999999999e-05 2024-09-13T11:26:27.870 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.870 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.870 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:26:27.870 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315668+0000", 2024-09-13T11:26:27.870 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0589e-05 2024-09-13T11:26:27.870 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.870 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.870 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:27.870 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316925+0000", 2024-09-13T11:26:27.870 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0012568340000000001 2024-09-13T11:26:27.870 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.870 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.870 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-09-13T11:26:27.870 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316932+0000", 2024-09-13T11:26:27.871 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.2230000000000001e-06 2024-09-13T11:26:27.871 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.871 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.871 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:27.871 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.186619+0000", 2024-09-13T11:26:27.871 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.86968679599999998 2024-09-13T11:26:27.871 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.871 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.871 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:26:27.871 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.186914+0000", 2024-09-13T11:26:27.871 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00029528900000000001 2024-09-13T11:26:27.871 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.871 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.871 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1", 2024-09-13T11:26:27.871 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.187217+0000", 2024-09-13T11:26:27.872 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00030232099999999998 2024-09-13T11:26:27.872 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.872 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.872 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:26:27.872 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.190532+0000", 2024-09-13T11:26:27.872 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.003315391 2024-09-13T11:26:27.872 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.872 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.872 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:27.872 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.190875+0000", 2024-09-13T11:26:27.872 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00034326300000000001 2024-09-13T11:26:27.872 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.872 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.872 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:26:27.873 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.190925+0000", 2024-09-13T11:26:27.873 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.9429999999999999e-05 2024-09-13T11:26:27.873 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.873 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.873 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:26:27.873 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.190961+0000", 2024-09-13T11:26:27.873 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.5951000000000002e-05 2024-09-13T11:26:27.873 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:27.873 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:27.873 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:27.873 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.873 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.873 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:172 3.b 3:dbec1eeb:::benchmark_data_smithi177_31116_object171:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T11:26:27.873 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:48.315684+0000", 2024-09-13T11:26:27.874 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 219.46098654299999, 2024-09-13T11:26:27.874 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.87615312999999995, 2024-09-13T11:26:27.874 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:26:27.874 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:26:27.874 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:27.874 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:26:27.874 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:26:27.874 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:27.874 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 172 2024-09-13T11:26:27.874 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.874 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:26:27.874 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.874 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:26:27.874 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315684+0000", 2024-09-13T11:26:27.875 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:27.875 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.875 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.875 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:26:27.875 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315684+0000", 2024-09-13T11:26:27.875 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:27.875 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.875 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.875 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:26:27.875 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315686+0000", 2024-09-13T11:26:27.875 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.886e-06 2024-09-13T11:26:27.875 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.875 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.875 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:26:27.876 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315820+0000", 2024-09-13T11:26:27.876 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00013388500000000001 2024-09-13T11:26:27.876 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.876 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.876 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:26:27.876 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315837+0000", 2024-09-13T11:26:27.876 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.6756e-05 2024-09-13T11:26:27.876 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.876 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.876 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:26:27.876 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315851+0000", 2024-09-13T11:26:27.876 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.4566000000000001e-05 2024-09-13T11:26:27.876 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.876 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.877 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:27.877 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.317021+0000", 2024-09-13T11:26:27.877 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0011701769999999999 2024-09-13T11:26:27.877 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.877 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.877 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-09-13T11:26:27.877 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.317037+0000", 2024-09-13T11:26:27.877 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.609e-05 2024-09-13T11:26:27.877 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.877 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.877 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:27.877 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.188634+0000", 2024-09-13T11:26:27.877 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.87159711900000003 2024-09-13T11:26:27.877 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.878 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.878 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:26:27.878 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.188974+0000", 2024-09-13T11:26:27.878 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00033922299999999999 2024-09-13T11:26:27.878 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.878 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.878 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1", 2024-09-13T11:26:27.878 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.189510+0000", 2024-09-13T11:26:27.878 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00053618999999999997 2024-09-13T11:26:27.878 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.878 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.878 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:27.878 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.191027+0000", 2024-09-13T11:26:27.878 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0015168390000000001 2024-09-13T11:26:27.878 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.879 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.879 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:26:27.879 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.191766+0000", 2024-09-13T11:26:27.879 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000739064 2024-09-13T11:26:27.879 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.879 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.879 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:26:27.879 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.191810+0000", 2024-09-13T11:26:27.879 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.3755999999999999e-05 2024-09-13T11:26:27.879 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.879 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.879 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:26:27.879 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.191837+0000", 2024-09-13T11:26:27.880 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.7579000000000001e-05 2024-09-13T11:26:27.880 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:27.880 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:27.880 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:27.880 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.880 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.880 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:177 3.b 3:de89c779:::benchmark_data_smithi177_31116_object176:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T11:26:27.880 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:48.315898+0000", 2024-09-13T11:26:27.880 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 219.460772228, 2024-09-13T11:26:27.880 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.87600712599999997, 2024-09-13T11:26:27.880 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:26:27.880 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:26:27.880 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:27.880 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:26:27.880 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:26:27.881 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:27.881 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 177 2024-09-13T11:26:27.881 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.881 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:26:27.881 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.881 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:26:27.881 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315898+0000", 2024-09-13T11:26:27.881 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:27.881 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.881 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.881 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:26:27.881 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315898+0000", 2024-09-13T11:26:27.881 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:27.881 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.881 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.882 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:26:27.882 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315900+0000", 2024-09-13T11:26:27.882 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.4929999999999999e-06 2024-09-13T11:26:27.882 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.882 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.882 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:26:27.882 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316027+0000", 2024-09-13T11:26:27.882 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000127568 2024-09-13T11:26:27.882 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.882 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.882 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:26:27.882 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316048+0000", 2024-09-13T11:26:27.882 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.0795999999999999e-05 2024-09-13T11:26:27.882 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.883 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.883 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:26:27.883 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316061+0000", 2024-09-13T11:26:27.883 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.3283e-05 2024-09-13T11:26:27.883 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.883 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.883 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:27.883 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.317107+0000", 2024-09-13T11:26:27.883 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0010452949999999999 2024-09-13T11:26:27.883 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.883 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.883 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-09-13T11:26:27.883 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.317114+0000", 2024-09-13T11:26:27.883 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.4569999999999999e-06 2024-09-13T11:26:27.884 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.884 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.884 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:27.884 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.190017+0000", 2024-09-13T11:26:27.884 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.87290258499999995 2024-09-13T11:26:27.884 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.884 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.884 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:26:27.884 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.190138+0000", 2024-09-13T11:26:27.884 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000121418 2024-09-13T11:26:27.884 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.884 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.884 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1", 2024-09-13T11:26:27.884 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.190216+0000", 2024-09-13T11:26:27.885 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.7410999999999995e-05 2024-09-13T11:26:27.885 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.885 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.885 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:27.885 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.191755+0000", 2024-09-13T11:26:27.885 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0015395420000000001 2024-09-13T11:26:27.885 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.885 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.885 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:26:27.885 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.191843+0000", 2024-09-13T11:26:27.885 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.8022999999999994e-05 2024-09-13T11:26:27.885 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.885 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.885 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:26:27.886 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.191876+0000", 2024-09-13T11:26:27.886 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.2619000000000001e-05 2024-09-13T11:26:27.886 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.886 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.886 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:26:27.886 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.191905+0000", 2024-09-13T11:26:27.886 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.9635999999999999e-05 2024-09-13T11:26:27.886 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:27.886 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:27.886 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:27.886 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.886 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.886 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:187 3.b 3:da03c8dd:::benchmark_data_smithi177_31116_object186:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T11:26:27.886 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:48.316083+0000", 2024-09-13T11:26:27.887 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 219.460587272, 2024-09-13T11:26:27.887 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.87835499500000003, 2024-09-13T11:26:27.887 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:26:27.887 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:26:27.887 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:27.887 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:26:27.887 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:26:27.887 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:27.887 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 187 2024-09-13T11:26:27.887 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.887 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:26:27.887 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.887 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:26:27.887 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316083+0000", 2024-09-13T11:26:27.887 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:27.888 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.888 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.888 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:26:27.888 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316083+0000", 2024-09-13T11:26:27.888 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:27.888 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.888 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.888 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:26:27.888 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316086+0000", 2024-09-13T11:26:27.888 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.5560000000000002e-06 2024-09-13T11:26:27.888 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.888 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.888 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:26:27.888 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316199+0000", 2024-09-13T11:26:27.889 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000113522 2024-09-13T11:26:27.889 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.889 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.889 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:26:27.889 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316215+0000", 2024-09-13T11:26:27.889 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.5265999999999999e-05 2024-09-13T11:26:27.889 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.889 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.889 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:26:27.889 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316226+0000", 2024-09-13T11:26:27.889 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0967e-05 2024-09-13T11:26:27.889 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.889 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.889 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:27.890 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.317198+0000", 2024-09-13T11:26:27.890 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00097194100000000004 2024-09-13T11:26:27.890 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.890 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.890 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-09-13T11:26:27.890 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.317213+0000", 2024-09-13T11:26:27.890 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.5472999999999999e-05 2024-09-13T11:26:27.890 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.890 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.890 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:27.890 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.191077+0000", 2024-09-13T11:26:27.890 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.87386435200000001 2024-09-13T11:26:27.890 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.890 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.891 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:26:27.891 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.191181+0000", 2024-09-13T11:26:27.891 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000103618 2024-09-13T11:26:27.891 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.891 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.891 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1", 2024-09-13T11:26:27.891 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.191305+0000", 2024-09-13T11:26:27.891 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00012358599999999999 2024-09-13T11:26:27.891 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.891 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.891 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:27.891 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.194279+0000", 2024-09-13T11:26:27.891 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0029746970000000001 2024-09-13T11:26:27.891 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.891 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.892 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:26:27.892 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.194336+0000", 2024-09-13T11:26:27.892 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.7244000000000003e-05 2024-09-13T11:26:27.892 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.892 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.892 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:26:27.892 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.194397+0000", 2024-09-13T11:26:27.892 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.0278000000000001e-05 2024-09-13T11:26:27.892 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.892 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.892 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:26:27.892 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.194438+0000", 2024-09-13T11:26:27.892 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.1495000000000002e-05 2024-09-13T11:26:27.892 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:27.893 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:27.893 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:27.893 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.893 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.893 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:200 3.b 3:d90039f1:::benchmark_data_smithi177_31116_object199:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T11:26:27.893 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:48.316280+0000", 2024-09-13T11:26:27.893 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 219.46039080700001, 2024-09-13T11:26:27.893 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.87879273000000002, 2024-09-13T11:26:27.893 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:26:27.893 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:26:27.893 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:27.893 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:26:27.893 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:26:27.893 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:27.893 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 200 2024-09-13T11:26:27.894 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.894 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:26:27.894 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.894 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:26:27.894 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316280+0000", 2024-09-13T11:26:27.894 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:27.894 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.894 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.894 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:26:27.894 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316280+0000", 2024-09-13T11:26:27.894 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:27.894 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.894 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.894 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:26:27.895 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316282+0000", 2024-09-13T11:26:27.895 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.497e-06 2024-09-13T11:26:27.895 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.895 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.895 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:26:27.895 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316411+0000", 2024-09-13T11:26:27.895 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00012924799999999999 2024-09-13T11:26:27.895 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.895 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.895 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:26:27.895 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316426+0000", 2024-09-13T11:26:27.895 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.4183e-05 2024-09-13T11:26:27.895 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.895 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.896 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:26:27.896 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316435+0000", 2024-09-13T11:26:27.896 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.0990000000000005e-06 2024-09-13T11:26:27.896 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.896 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.896 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:27.896 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.317250+0000", 2024-09-13T11:26:27.896 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00081501100000000003 2024-09-13T11:26:27.896 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.896 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.896 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-09-13T11:26:27.896 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.317257+0000", 2024-09-13T11:26:27.896 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.2169999999999997e-06 2024-09-13T11:26:27.897 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.897 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.897 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:27.897 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.191936+0000", 2024-09-13T11:26:27.897 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.87467922099999995 2024-09-13T11:26:27.897 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.897 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.897 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:26:27.897 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.192056+0000", 2024-09-13T11:26:27.897 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000119638 2024-09-13T11:26:27.897 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.897 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.897 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1", 2024-09-13T11:26:27.897 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.192184+0000", 2024-09-13T11:26:27.898 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00012813399999999999 2024-09-13T11:26:27.898 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.898 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.898 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:27.898 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.194324+0000", 2024-09-13T11:26:27.898 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0021399420000000001 2024-09-13T11:26:27.898 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.898 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.898 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:26:27.898 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.194802+0000", 2024-09-13T11:26:27.898 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00047798799999999999 2024-09-13T11:26:27.898 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.898 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.898 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:26:27.898 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.194951+0000", 2024-09-13T11:26:27.899 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0001494 2024-09-13T11:26:27.899 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.899 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.899 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:26:27.899 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.195072+0000", 2024-09-13T11:26:27.899 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00012115199999999999 2024-09-13T11:26:27.899 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:27.899 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:27.899 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:27.899 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.899 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.899 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop(client.4665.0:79801 3.5 e102/67 3:a6efdf84:::benchmark_data_smithi177_31116_object79800:head v 102'3851, mlcod=102'3848)", 2024-09-13T11:26:27.899 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:24:00.403754+0000", 2024-09-13T11:26:27.899 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 147.372916455, 2024-09-13T11:26:27.900 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.31749079899999999, 2024-09-13T11:26:27.900 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:26:27.900 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:26:27.900 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:27.900 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:26:27.900 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.900 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:26:27.900 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:00.403754+0000", 2024-09-13T11:26:27.900 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:27.900 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.900 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.900 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:26:27.900 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:00.403754+0000", 2024-09-13T11:26:27.901 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:27.901 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.901 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.901 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:26:27.901 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:00.403758+0000", 2024-09-13T11:26:27.901 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.507e-06 2024-09-13T11:26:27.901 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.901 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.901 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:26:27.901 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:00.403819+0000", 2024-09-13T11:26:27.901 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.1042000000000003e-05 2024-09-13T11:26:27.901 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.901 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.901 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:26:27.902 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:00.717963+0000", 2024-09-13T11:26:27.902 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.314144594 2024-09-13T11:26:27.902 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.902 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.902 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:26:27.902 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:00.717976+0000", 2024-09-13T11:26:27.902 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.3029e-05 2024-09-13T11:26:27.902 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.902 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.902 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:27.902 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:00.718143+0000", 2024-09-13T11:26:27.902 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00016639000000000001 2024-09-13T11:26:27.902 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.902 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.903 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:26:27.903 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:00.718184+0000", 2024-09-13T11:26:27.903 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.1801999999999997e-05 2024-09-13T11:26:27.903 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.903 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.903 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:26:27.903 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:00.721162+0000", 2024-09-13T11:26:27.903 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0029772800000000001 2024-09-13T11:26:27.903 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.903 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.903 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:26:27.903 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:00.721245+0000", 2024-09-13T11:26:27.903 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.3154999999999999e-05 2024-09-13T11:26:27.903 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:27.903 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:27.904 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:27.904 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.904 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.904 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "MOSDPGPushReply(3.26 121/121 [PushReplyOp(3:670b9aab:::benchmark_data_smithi177_31116_object93:head)])", 2024-09-13T11:26:27.904 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:24:27.300318+0000", 2024-09-13T11:26:27.904 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 120.47635258699999, 2024-09-13T11:26:27.904 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.811436728, 2024-09-13T11:26:27.904 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:26:27.904 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:26:27.904 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "reached pg", 2024-09-13T11:26:27.904 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:26:27.904 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.904 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:26:27.905 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:27.300318+0000", 2024-09-13T11:26:27.905 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:27.905 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.905 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.905 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:26:27.905 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:27.300318+0000", 2024-09-13T11:26:27.905 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:27.905 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.905 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.905 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:26:27.905 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:27.300320+0000", 2024-09-13T11:26:27.905 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.778e-06 2024-09-13T11:26:27.905 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.905 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.906 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:26:27.906 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:27.300326+0000", 2024-09-13T11:26:27.906 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.7700000000000004e-06 2024-09-13T11:26:27.906 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.906 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.906 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:26:27.906 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:28.109665+0000", 2024-09-13T11:26:27.906 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.80933853499999997 2024-09-13T11:26:27.906 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.906 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.906 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:26:27.906 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:28.109678+0000", 2024-09-13T11:26:27.906 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.2870999999999999e-05 2024-09-13T11:26:27.906 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.906 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.907 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:27.907 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:28.109800+0000", 2024-09-13T11:26:27.907 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000122328 2024-09-13T11:26:27.907 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.907 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.907 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:26:27.907 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:28.111755+0000", 2024-09-13T11:26:27.907 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0019544459999999999 2024-09-13T11:26:27.907 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:27.907 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:27.907 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:27.907 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.907 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.907 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:184026 3.b 3:d3052347:::benchmark_data_smithi177_31116_object43723:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e171)", 2024-09-13T11:26:27.908 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:25:32.645242+0000", 2024-09-13T11:26:27.908 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 55.131428546999999, 2024-09-13T11:26:27.908 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.348925019, 2024-09-13T11:26:27.908 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:26:27.908 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:26:27.908 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:27.908 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:26:27.908 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:26:27.908 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:27.908 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 184026 2024-09-13T11:26:27.908 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.908 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:26:27.908 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.908 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:26:27.909 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:32.645242+0000", 2024-09-13T11:26:27.909 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:27.909 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.909 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.909 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:26:27.909 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:32.645242+0000", 2024-09-13T11:26:27.909 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:27.909 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.909 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.909 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:26:27.909 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:32.645244+0000", 2024-09-13T11:26:27.909 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.368e-06 2024-09-13T11:26:27.909 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.909 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.910 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:26:27.910 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:32.645251+0000", 2024-09-13T11:26:27.910 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.2519999999999996e-06 2024-09-13T11:26:27.910 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.910 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.910 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:26:27.910 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:32.645267+0000", 2024-09-13T11:26:27.910 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.6529e-05 2024-09-13T11:26:27.910 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.910 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.910 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:26:27.910 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:32.645278+0000", 2024-09-13T11:26:27.910 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.119e-05 2024-09-13T11:26:27.910 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.911 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.911 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:27.911 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:32.645348+0000", 2024-09-13T11:26:27.911 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.9333999999999997e-05 2024-09-13T11:26:27.911 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.911 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.911 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:26:27.911 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:32.645479+0000", 2024-09-13T11:26:27.911 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00013153199999999999 2024-09-13T11:26:27.911 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.911 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.911 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1", 2024-09-13T11:26:27.911 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:32.645599+0000", 2024-09-13T11:26:27.911 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000119481 2024-09-13T11:26:27.911 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.912 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.912 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:26:27.912 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:32.648042+0000", 2024-09-13T11:26:27.912 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0024437059999999999 2024-09-13T11:26:27.912 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.912 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.912 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:27.912 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:32.994082+0000", 2024-09-13T11:26:27.912 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.34604006399999998 2024-09-13T11:26:27.912 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.912 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.912 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:26:27.912 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:32.994124+0000", 2024-09-13T11:26:27.913 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.1573999999999999e-05 2024-09-13T11:26:27.913 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:27.913 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:27.913 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:26:27.913 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:32.994167+0000", 2024-09-13T11:26:27.913 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.2988999999999998e-05 2024-09-13T11:26:27.913 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:27.913 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:27.913 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:27.913 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:27.913 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:27.913 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-09-13T11:26:27.915 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-13T11:26:27.915 INFO:tasks.ceph.osd.12.smithi001.stderr:2024-09-13T11:26:27.910+0000 7f892c6e6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26314) UID: 0 2024-09-13T11:26:28.014 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 15, 14] out_osds: [] dead_osds: [] live_osds: [0, 4, 8, 12, 5, 9, 13, 2, 6, 10, 3, 7, 11, 15, 1, 14] 2024-09-13T11:26:28.015 INFO:tasks.thrashosds.thrasher:choose_action: min_in 2 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-13T11:26:28.015 INFO:tasks.thrashosds.thrasher:inject_pause on osd.12 2024-09-13T11:26:28.015 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-13T11:26:28.015 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-13T11:26:28.015 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok config set filestore_inject_stall 3 2024-09-13T11:26:28.017 INFO:tasks.ceph.osd.2.smithi162.stderr:2024-09-13T11:26:28.015+0000 7fe92b35a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26313) UID: 0 2024-09-13T11:26:28.065 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-13T11:26:28.065 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [ 2024-09-13T11:26:28.065 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:28.065 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:245361 3.1d 3:bddcee77:::benchmark_data_smithi177_31116_object105058:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e208)", 2024-09-13T11:26:28.065 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:26:28.058415+0000", 2024-09-13T11:26:28.065 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 0.0068848290000000003, 2024-09-13T11:26:28.065 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0069043569999999999, 2024-09-13T11:26:28.065 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:26:28.065 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:26:28.065 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "waiting for sub ops", 2024-09-13T11:26:28.066 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:26:28.066 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:26:28.066 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:28.066 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 245361 2024-09-13T11:26:28.066 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:28.066 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:26:28.066 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:28.066 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:26:28.066 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:28.058415+0000", 2024-09-13T11:26:28.066 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:28.066 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:28.066 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:28.066 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:26:28.066 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:28.058415+0000", 2024-09-13T11:26:28.067 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:28.067 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:28.067 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:28.067 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:26:28.067 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:28.058417+0000", 2024-09-13T11:26:28.067 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.3649999999999998e-06 2024-09-13T11:26:28.067 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:28.067 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:28.067 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:26:28.067 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:28.058423+0000", 2024-09-13T11:26:28.067 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.3949999999999998e-06 2024-09-13T11:26:28.067 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:28.067 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:28.068 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:26:28.068 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:28.058440+0000", 2024-09-13T11:26:28.068 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.7178999999999999e-05 2024-09-13T11:26:28.068 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:28.068 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:28.068 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:26:28.068 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:28.058449+0000", 2024-09-13T11:26:28.068 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.4209999999999995e-06 2024-09-13T11:26:28.068 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:28.068 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:28.068 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:26:28.068 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:28.058502+0000", 2024-09-13T11:26:28.068 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.3350999999999998e-05 2024-09-13T11:26:28.069 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:28.069 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:28.069 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:26:28.069 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:28.058625+0000", 2024-09-13T11:26:28.069 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00012295399999999999 2024-09-13T11:26:28.069 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:28.069 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:28.069 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 15", 2024-09-13T11:26:28.069 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:28.058729+0000", 2024-09-13T11:26:28.069 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000103819 2024-09-13T11:26:28.069 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:28.069 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:28.069 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:26:28.069 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:28.063568+0000", 2024-09-13T11:26:28.070 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0048391019999999996 2024-09-13T11:26:28.070 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:28.070 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:26:28.070 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:28.070 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:28.070 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:28.070 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.4665.0:245362 3.9 e208/117 3:939a36fe:::benchmark_data_smithi177_31116_object105059:head v 208'6920, mlcod=208'6918)", 2024-09-13T11:26:28.070 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:26:28.060447+0000", 2024-09-13T11:26:28.070 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 0.0048521049999999998, 2024-09-13T11:26:28.070 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0049159959999999997, 2024-09-13T11:26:28.070 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:26:28.070 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:26:28.070 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:26:28.070 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:26:28.071 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:28.071 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:26:28.071 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:28.060447+0000", 2024-09-13T11:26:28.071 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:28.071 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:28.071 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:28.071 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:26:28.071 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:28.060447+0000", 2024-09-13T11:26:28.071 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:28.071 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:28.071 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:28.071 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:26:28.071 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:28.060449+0000", 2024-09-13T11:26:28.072 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.544e-06 2024-09-13T11:26:28.072 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:28.072 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:28.072 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:26:28.072 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:28.060455+0000", 2024-09-13T11:26:28.072 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.7089999999999999e-06 2024-09-13T11:26:28.072 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:28.072 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:28.072 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:26:28.072 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:28.060478+0000", 2024-09-13T11:26:28.072 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.3377e-05 2024-09-13T11:26:28.072 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:28.072 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:28.072 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:26:28.073 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:28.060483+0000", 2024-09-13T11:26:28.073 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.5389999999999998e-06 2024-09-13T11:26:28.073 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:28.073 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:28.073 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:26:28.073 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:28.060537+0000", 2024-09-13T11:26:28.073 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.3091999999999999e-05 2024-09-13T11:26:28.073 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:28.073 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:28.073 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:26:28.073 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:28.060558+0000", 2024-09-13T11:26:28.073 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.1005999999999998e-05 2024-09-13T11:26:28.073 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:28.073 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:26:28.074 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:28.074 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:28.074 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:28.074 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.4665.0:245365 3.25 e208/141 3:a499ca70:::benchmark_data_smithi177_31116_object105062:head v 208'6922, mlcod=208'6920)", 2024-09-13T11:26:28.074 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:26:28.060733+0000", 2024-09-13T11:26:28.074 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 0.0045666700000000001, 2024-09-13T11:26:28.074 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0046592910000000003, 2024-09-13T11:26:28.074 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:26:28.074 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:26:28.074 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:26:28.074 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:26:28.074 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:28.074 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:26:28.074 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:28.060733+0000", 2024-09-13T11:26:28.075 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:28.075 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:28.075 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:28.075 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:26:28.075 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:28.060733+0000", 2024-09-13T11:26:28.075 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:28.075 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:28.075 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:28.075 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:26:28.075 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:28.060733+0000", 2024-09-13T11:26:28.075 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.7199999999999998e-07 2024-09-13T11:26:28.075 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:28.075 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:28.075 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:26:28.076 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:28.060737+0000", 2024-09-13T11:26:28.076 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.8120000000000001e-06 2024-09-13T11:26:28.076 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:28.076 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:28.076 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:26:28.076 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:28.060755+0000", 2024-09-13T11:26:28.076 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.7929000000000001e-05 2024-09-13T11:26:28.076 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:28.076 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:28.076 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:26:28.076 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:28.060761+0000", 2024-09-13T11:26:28.076 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.524e-06 2024-09-13T11:26:28.076 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:28.076 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:28.077 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:26:28.077 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:28.060812+0000", 2024-09-13T11:26:28.077 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.1406999999999998e-05 2024-09-13T11:26:28.077 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:28.077 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:28.077 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:26:28.077 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:28.060835+0000", 2024-09-13T11:26:28.077 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.2904e-05 2024-09-13T11:26:28.077 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:28.077 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:26:28.077 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:28.077 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:28.077 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:28.077 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:245369 3.1d 3:b9e156c7:::benchmark_data_smithi177_31116_object105066:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e208)", 2024-09-13T11:26:28.078 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:26:28.064004+0000", 2024-09-13T11:26:28.078 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 0.001295551, 2024-09-13T11:26:28.078 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0014161969999999999, 2024-09-13T11:26:28.078 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:26:28.078 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:26:28.078 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "waiting for sub ops", 2024-09-13T11:26:28.078 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:26:28.078 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:26:28.078 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:28.078 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 245369 2024-09-13T11:26:28.078 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:28.078 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:26:28.078 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:28.079 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:26:28.079 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:28.064004+0000", 2024-09-13T11:26:28.079 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:28.079 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:28.079 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:28.079 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:26:28.079 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:28.064004+0000", 2024-09-13T11:26:28.079 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:28.079 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:28.079 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:28.079 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:26:28.079 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:28.064005+0000", 2024-09-13T11:26:28.079 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.291e-06 2024-09-13T11:26:28.079 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:28.080 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:28.080 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:26:28.080 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:28.064009+0000", 2024-09-13T11:26:28.080 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.681e-06 2024-09-13T11:26:28.080 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:28.080 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:28.080 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:26:28.080 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:28.064017+0000", 2024-09-13T11:26:28.080 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.3040000000000005e-06 2024-09-13T11:26:28.080 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:28.080 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:28.080 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:26:28.080 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:28.064021+0000", 2024-09-13T11:26:28.080 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.3279999999999999e-06 2024-09-13T11:26:28.081 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:28.081 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:28.081 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:26:28.081 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:28.064066+0000", 2024-09-13T11:26:28.081 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.4487999999999998e-05 2024-09-13T11:26:28.081 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:28.081 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:28.081 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:26:28.081 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:28.064189+0000", 2024-09-13T11:26:28.081 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0001227 2024-09-13T11:26:28.081 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:28.081 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:28.081 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 15", 2024-09-13T11:26:28.082 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:28.064290+0000", 2024-09-13T11:26:28.082 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000101705 2024-09-13T11:26:28.082 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:28.082 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:26:28.082 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:28.082 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:28.082 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:28.082 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop_reply(client.4665.0:245361 3.1d e208/102)", 2024-09-13T11:26:28.082 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:26:28.064119+0000", 2024-09-13T11:26:28.082 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 0.001180581, 2024-09-13T11:26:28.082 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001333531, 2024-09-13T11:26:28.082 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:26:28.082 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:26:28.082 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "queued for pg", 2024-09-13T11:26:28.082 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:26:28.083 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:28.083 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:26:28.083 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:28.064119+0000", 2024-09-13T11:26:28.083 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:28.083 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:28.083 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:28.083 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:26:28.083 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:28.064119+0000", 2024-09-13T11:26:28.083 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:28.083 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:28.083 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:28.083 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:26:28.083 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:28.064119+0000", 2024-09-13T11:26:28.083 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.2799999999999996e-07 2024-09-13T11:26:28.084 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:28.084 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:28.084 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:26:28.084 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:28.064121+0000", 2024-09-13T11:26:28.084 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.1430000000000001e-06 2024-09-13T11:26:28.084 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:28.084 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:28.084 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:26:28.084 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:28.064138+0000", 2024-09-13T11:26:28.084 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.6317999999999999e-05 2024-09-13T11:26:28.084 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:28.084 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:28.084 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:26:28.085 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:28.064141+0000", 2024-09-13T11:26:28.085 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.4759999999999998e-06 2024-09-13T11:26:28.085 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:28.085 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:26:28.085 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:28.085 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:28.085 INFO:teuthology.orchestra.run.smithi162.stdout: ], 2024-09-13T11:26:28.085 INFO:teuthology.orchestra.run.smithi162.stdout: "num_ops": 5 2024-09-13T11:26:28.085 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-13T11:26:28.086 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-13T11:26:28.129 INFO:tasks.ceph.osd.2.smithi162.stderr:2024-09-13T11:26:28.127+0000 7fe92b35a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26313) UID: 0 2024-09-13T11:26:28.171 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-13T11:26:28.171 INFO:teuthology.orchestra.run.smithi001.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-09-13T11:26:28.171 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-13T11:26:28.197 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-13T11:26:28.199 INFO:tasks.ceph.osd.5.smithi092.stderr:2024-09-13T11:26:28.196+0000 7f6f02136640 -1 osd.5 208 handle_osd_map injecting map crc failure 2024-09-13T11:26:28.204 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-13T11:26:28.202+0000 7f6a5d1a2640 -1 osd.0 208 handle_osd_map injecting map crc failure 2024-09-13T11:26:28.218 INFO:tasks.ceph.osd.2.smithi162.stderr:2024-09-13T11:26:28.215+0000 7fe92b35a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26313) UID: 0 2024-09-13T11:26:28.271 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T11:26:28.275 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-13T11:26:28.275 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [], 2024-09-13T11:26:28.275 INFO:teuthology.orchestra.run.smithi162.stdout: "complaint_time": 30, 2024-09-13T11:26:28.275 INFO:teuthology.orchestra.run.smithi162.stdout: "num_blocked_ops": 0 2024-09-13T11:26:28.275 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-13T11:26:28.294 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-13T11:26:28.319 INFO:tasks.ceph.osd.7.smithi177.stderr:2024-09-13T11:26:28.317+0000 7f897b51d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26575) UID: 0 2024-09-13T11:26:28.420 INFO:tasks.ceph.osd.9.smithi092.stderr:2024-09-13T11:26:28.420+0000 7ff5ab3d0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26348) UID: 0 2024-09-13T11:26:28.471 INFO:teuthology.orchestra.run.smithi001.stdout:osd.0: {} 2024-09-13T11:26:28.471 INFO:teuthology.orchestra.run.smithi001.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T11:26:28.484 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-13T11:26:28.484 INFO:teuthology.orchestra.run.smithi162.stdout: "size": 20, 2024-09-13T11:26:28.484 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 600, 2024-09-13T11:26:28.484 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [ 2024-09-13T11:26:28.484 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:28.484 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.4665.0:171 3.9 e41/30 3:9e1c7eeb:::benchmark_data_smithi177_31116_object170:head v 41'13, mlcod=41'9)", 2024-09-13T11:26:28.484 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:22:47.418602+0000", 2024-09-13T11:26:28.484 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 221.06440645500001, 2024-09-13T11:26:28.484 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.67235367300000004, 2024-09-13T11:26:28.484 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:26:28.485 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:26:28.485 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:28.485 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:26:28.485 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:28.485 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:26:28.485 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:47.418602+0000", 2024-09-13T11:26:28.485 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:28.485 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:28.485 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:28.485 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:26:28.485 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:47.418602+0000", 2024-09-13T11:26:28.485 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:28.485 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:28.486 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:28.486 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:26:28.486 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:47.418605+0000", 2024-09-13T11:26:28.486 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.2200000000000001e-06 2024-09-13T11:26:28.486 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:28.486 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:28.486 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:26:28.486 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:47.418668+0000", 2024-09-13T11:26:28.486 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.2921000000000003e-05 2024-09-13T11:26:28.486 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:28.487 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:28.487 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:26:28.487 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:48.089380+0000", 2024-09-13T11:26:28.487 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.67071191100000005 2024-09-13T11:26:28.487 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:28.487 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:28.487 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:26:28.487 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:48.089423+0000", 2024-09-13T11:26:28.487 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.2916e-05 2024-09-13T11:26:28.487 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:28.487 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:28.487 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:26:28.487 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:48.089533+0000", 2024-09-13T11:26:28.488 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000109996 2024-09-13T11:26:28.488 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:28.488 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:28.488 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:26:28.488 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:48.089570+0000", 2024-09-13T11:26:28.488 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.7314999999999998e-05 2024-09-13T11:26:28.488 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:28.488 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:28.488 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:26:28.488 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:48.090852+0000", 2024-09-13T11:26:28.488 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001281641 2024-09-13T11:26:28.488 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:28.488 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:28.489 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:26:28.489 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:48.090956+0000", 2024-09-13T11:26:28.489 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00010375299999999999 2024-09-13T11:26:28.489 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:28.489 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:26:28.489 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:28.489 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:28.489 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:28.489 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPushReply(3.19 74/70 [PushReplyOp(3:9edf939c:::benchmark_data_smithi177_31116_object17207:head)])", 2024-09-13T11:26:28.489 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:30.897971+0000", 2024-09-13T11:26:28.489 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 177.58503760400001, 2024-09-13T11:26:28.489 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.40616071100000001, 2024-09-13T11:26:28.489 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:26:28.489 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:26:28.490 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-09-13T11:26:28.490 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:26:28.490 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:28.490 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:26:28.490 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.897971+0000", 2024-09-13T11:26:28.490 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:28.490 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:28.490 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:28.490 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:26:28.490 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.897971+0000", 2024-09-13T11:26:28.490 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:28.490 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:28.490 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:28.491 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:26:28.491 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.897972+0000", 2024-09-13T11:26:28.491 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1429999999999999e-06 2024-09-13T11:26:28.491 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:28.491 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:28.491 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:26:28.491 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.897977+0000", 2024-09-13T11:26:28.491 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.5199999999999999e-06 2024-09-13T11:26:28.491 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:28.491 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:28.491 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:26:28.491 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:31.302500+0000", 2024-09-13T11:26:28.491 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.40452312400000001 2024-09-13T11:26:28.491 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:28.492 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:28.492 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:26:28.492 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:31.302513+0000", 2024-09-13T11:26:28.492 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2753e-05 2024-09-13T11:26:28.492 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:28.492 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:28.492 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:26:28.492 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:31.302645+0000", 2024-09-13T11:26:28.492 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00013292100000000001 2024-09-13T11:26:28.492 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:28.492 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:28.492 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:26:28.492 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:31.304132+0000", 2024-09-13T11:26:28.493 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0014862499999999999 2024-09-13T11:26:28.493 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:28.493 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:26:28.493 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:28.493 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:28.493 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:28.493 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPushReply(3.19 75/70 [PushReplyOp(3:9d1489b1:::benchmark_data_smithi177_31116_object18698:head)])", 2024-09-13T11:26:28.493 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:31.697212+0000", 2024-09-13T11:26:28.493 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 176.78579660299999, 2024-09-13T11:26:28.493 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.22850633200000001, 2024-09-13T11:26:28.493 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:26:28.493 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:26:28.493 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-09-13T11:26:28.493 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:26:28.493 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:28.494 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:26:28.494 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:31.697212+0000", 2024-09-13T11:26:28.494 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:28.494 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:28.494 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:28.494 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:26:28.494 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:31.697212+0000", 2024-09-13T11:26:28.494 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:28.494 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:28.494 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:28.494 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:26:28.494 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:31.697214+0000", 2024-09-13T11:26:28.494 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.7439999999999999e-06 2024-09-13T11:26:28.495 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:28.495 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:28.495 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:26:28.495 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:31.697224+0000", 2024-09-13T11:26:28.495 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0173000000000001e-05 2024-09-13T11:26:28.495 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:28.495 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:28.495 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:26:28.495 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:31.923931+0000", 2024-09-13T11:26:28.495 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.226706926 2024-09-13T11:26:28.495 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:28.495 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:28.495 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:26:28.495 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:31.923950+0000", 2024-09-13T11:26:28.495 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.8899e-05 2024-09-13T11:26:28.496 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:28.496 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:28.496 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:26:28.496 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:31.924449+0000", 2024-09-13T11:26:28.496 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00049907800000000002 2024-09-13T11:26:28.496 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:28.496 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:28.496 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:26:28.496 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:31.925718+0000", 2024-09-13T11:26:28.496 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001269512 2024-09-13T11:26:28.496 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:28.496 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:26:28.496 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:28.497 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:28.497 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:28.497 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPushReply(3.19 76/70 [PushReplyOp(3:9faf6d5f:::benchmark_data_smithi177_31116_object18954:head)])", 2024-09-13T11:26:28.497 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:32.042874+0000", 2024-09-13T11:26:28.497 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 176.440134734, 2024-09-13T11:26:28.497 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.40212404299999999, 2024-09-13T11:26:28.497 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:26:28.497 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:26:28.497 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-09-13T11:26:28.497 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:26:28.497 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:28.497 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:26:28.497 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.042874+0000", 2024-09-13T11:26:28.497 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:28.498 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:28.498 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:28.498 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:26:28.498 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.042874+0000", 2024-09-13T11:26:28.498 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:28.498 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:28.498 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:28.498 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:26:28.498 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.042875+0000", 2024-09-13T11:26:28.498 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.432e-06 2024-09-13T11:26:28.498 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:28.498 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:28.498 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:26:28.498 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.042886+0000", 2024-09-13T11:26:28.499 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0265000000000001e-05 2024-09-13T11:26:28.499 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:28.499 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:28.499 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:26:28.499 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.443455+0000", 2024-09-13T11:26:28.499 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.40056951400000002 2024-09-13T11:26:28.499 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:28.499 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:28.499 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:26:28.499 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.443480+0000", 2024-09-13T11:26:28.499 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.4929000000000001e-05 2024-09-13T11:26:28.499 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:28.499 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:28.500 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:26:28.500 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.443666+0000", 2024-09-13T11:26:28.500 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000186446 2024-09-13T11:26:28.500 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:28.500 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:28.500 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:26:28.500 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.444998+0000", 2024-09-13T11:26:28.500 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0013314570000000001 2024-09-13T11:26:28.500 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:28.500 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:26:28.500 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:28.500 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:28.500 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:28.501 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPush(3.1d 85/78 [PushOp(3:bc4b9336:::benchmark_data_smithi177_31116_object11944:head, version: 49'734, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:bc4b9336:::benchmark_data_smithi177_31116_object11944:head@49'734, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:26:28.501 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:42.225966+0000", 2024-09-13T11:26:28.501 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 166.25704252099999, 2024-09-13T11:26:28.501 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.23879947800000001, 2024-09-13T11:26:28.501 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:26:28.501 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:26:28.501 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:26:28.501 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:26:28.501 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:28.501 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:26:28.501 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.225966+0000", 2024-09-13T11:26:28.501 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:28.501 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:28.501 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:28.502 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:26:28.502 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.225966+0000", 2024-09-13T11:26:28.502 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:28.502 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:28.502 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:28.502 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:26:28.502 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.225969+0000", 2024-09-13T11:26:28.502 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.0400000000000001e-06 2024-09-13T11:26:28.502 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:28.502 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:28.502 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:26:28.502 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.226036+0000", 2024-09-13T11:26:28.502 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.6978999999999997e-05 2024-09-13T11:26:28.503 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:28.503 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:28.503 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:26:28.503 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.463110+0000", 2024-09-13T11:26:28.503 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.23707400000000001 2024-09-13T11:26:28.503 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:28.503 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:28.503 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:26:28.503 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.463140+0000", 2024-09-13T11:26:28.503 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.9575999999999999e-05 2024-09-13T11:26:28.503 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:28.503 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:28.503 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:26:28.503 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.463326+0000", 2024-09-13T11:26:28.504 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000186298 2024-09-13T11:26:28.504 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:28.504 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:28.504 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:26:28.504 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.463351+0000", 2024-09-13T11:26:28.504 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.4862999999999999e-05 2024-09-13T11:26:28.504 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:28.504 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:28.505 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:26:28.505 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.464766+0000", 2024-09-13T11:26:28.505 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001414722 2024-09-13T11:26:28.505 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:28.505 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:26:28.505 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:28.505 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:28.505 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:28.505 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPush(3.1d 89/78 [PushOp(3:bc1b7cfb:::benchmark_data_smithi177_31116_object22629:head, version: 55'1387, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:bc1b7cfb:::benchmark_data_smithi177_31116_object22629:head@55'1387, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:26:28.505 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:46.660855+0000", 2024-09-13T11:26:28.505 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 161.822154025, 2024-09-13T11:26:28.505 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.26622785599999998, 2024-09-13T11:26:28.505 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:26:28.506 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:26:28.506 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:26:28.506 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:26:28.506 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:28.506 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:26:28.506 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:46.660855+0000", 2024-09-13T11:26:28.506 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:28.506 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:28.506 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:28.506 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:26:28.506 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:46.660855+0000", 2024-09-13T11:26:28.506 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:28.506 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:28.506 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:28.507 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:26:28.507 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:46.660857+0000", 2024-09-13T11:26:28.507 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.6560000000000001e-06 2024-09-13T11:26:28.507 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:28.507 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:28.507 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:26:28.507 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:46.660931+0000", 2024-09-13T11:26:28.507 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.4041999999999993e-05 2024-09-13T11:26:28.507 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:28.507 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:28.507 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:26:28.507 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:46.924654+0000", 2024-09-13T11:26:28.507 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.26372239800000002 2024-09-13T11:26:28.508 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:28.508 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:28.508 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:26:28.508 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:46.924729+0000", 2024-09-13T11:26:28.508 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.5061999999999994e-05 2024-09-13T11:26:28.508 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:28.508 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:28.508 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:26:28.508 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:46.924986+0000", 2024-09-13T11:26:28.508 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00025762000000000002 2024-09-13T11:26:28.508 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:28.508 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:28.508 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:26:28.509 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:46.925013+0000", 2024-09-13T11:26:28.509 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.6179000000000001e-05 2024-09-13T11:26:28.509 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:28.509 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:28.509 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:26:28.509 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:46.927082+0000", 2024-09-13T11:26:28.509 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.002069899 2024-09-13T11:26:28.509 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:28.509 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:26:28.509 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:28.509 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:28.509 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:28.509 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPush(3.1d 90/78 [PushOp(3:bc0e7228:::benchmark_data_smithi177_31116_object24733:head, version: 55'1512, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:bc0e7228:::benchmark_data_smithi177_31116_object24733:head@55'1512, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:26:28.509 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:47.529126+0000", 2024-09-13T11:26:28.510 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 160.95388305099999, 2024-09-13T11:26:28.510 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.21343989499999999, 2024-09-13T11:26:28.510 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:26:28.510 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:26:28.510 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:26:28.510 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:26:28.510 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:28.510 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:26:28.510 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:47.529126+0000", 2024-09-13T11:26:28.510 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:28.510 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:28.510 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:28.510 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:26:28.510 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:47.529126+0000", 2024-09-13T11:26:28.511 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:28.511 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:28.511 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:28.511 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:26:28.511 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:47.529127+0000", 2024-09-13T11:26:28.511 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.252e-06 2024-09-13T11:26:28.511 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:28.511 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:28.511 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:26:28.511 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:47.529185+0000", 2024-09-13T11:26:28.511 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.8224999999999999e-05 2024-09-13T11:26:28.511 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:28.511 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:28.512 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:26:28.512 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:47.740216+0000", 2024-09-13T11:26:28.512 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.21103060800000001 2024-09-13T11:26:28.512 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:28.512 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:28.512 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:26:28.512 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:47.740248+0000", 2024-09-13T11:26:28.512 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.2821999999999998e-05 2024-09-13T11:26:28.512 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:28.512 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:28.512 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:26:28.512 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:47.740504+0000", 2024-09-13T11:26:28.512 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00025518399999999999 2024-09-13T11:26:28.513 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:28.513 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:28.513 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:26:28.513 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:47.740544+0000", 2024-09-13T11:26:28.513 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.0438999999999997e-05 2024-09-13T11:26:28.513 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:28.513 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:28.513 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:26:28.513 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:47.742565+0000", 2024-09-13T11:26:28.514 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0020213649999999998 2024-09-13T11:26:28.514 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:28.514 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:26:28.514 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:28.514 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:28.514 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:28.514 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:83199 3.1d 3:be2c6fae:::benchmark_data_smithi177_31116_object83198:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106)", 2024-09-13T11:26:28.514 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:24:04.297306+0000", 2024-09-13T11:26:28.514 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 144.18570309899999, 2024-09-13T11:26:28.514 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.25114587999999999, 2024-09-13T11:26:28.514 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:26:28.514 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:26:28.515 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:28.515 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:26:28.515 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:26:28.515 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:28.515 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 83199 2024-09-13T11:26:28.515 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:28.515 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:26:28.515 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:28.515 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:26:28.515 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.297306+0000", 2024-09-13T11:26:28.515 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:28.515 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:28.516 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:28.516 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:26:28.516 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.297306+0000", 2024-09-13T11:26:28.516 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:28.516 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:28.516 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:28.516 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:26:28.516 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.297308+0000", 2024-09-13T11:26:28.516 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.4150000000000002e-06 2024-09-13T11:26:28.516 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:28.516 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:28.517 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:26:28.517 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.297352+0000", 2024-09-13T11:26:28.517 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.4431000000000002e-05 2024-09-13T11:26:28.517 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:28.517 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:28.517 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:26:28.517 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.297369+0000", 2024-09-13T11:26:28.517 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.6685e-05 2024-09-13T11:26:28.517 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:28.517 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:28.517 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:26:28.517 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.297376+0000", 2024-09-13T11:26:28.518 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.7299999999999999e-06 2024-09-13T11:26:28.518 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:28.518 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:28.518 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:26:28.518 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.297427+0000", 2024-09-13T11:26:28.518 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.1461000000000001e-05 2024-09-13T11:26:28.518 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:28.518 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:28.518 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:26:28.518 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.297528+0000", 2024-09-13T11:26:28.518 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00010028599999999999 2024-09-13T11:26:28.519 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:28.519 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:28.519 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 15", 2024-09-13T11:26:28.519 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.297685+0000", 2024-09-13T11:26:28.519 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00015760500000000001 2024-09-13T11:26:28.519 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:28.519 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:28.519 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:26:28.519 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.300779+0000", 2024-09-13T11:26:28.519 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0030936700000000002 2024-09-13T11:26:28.519 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:28.519 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:28.520 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:28.520 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.548279+0000", 2024-09-13T11:26:28.520 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.24750059999999999 2024-09-13T11:26:28.520 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:28.520 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:28.520 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:26:28.520 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.548373+0000", 2024-09-13T11:26:28.520 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.3451999999999999e-05 2024-09-13T11:26:28.520 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:28.520 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:28.520 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:26:28.520 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.548451+0000", 2024-09-13T11:26:28.521 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.8545000000000001e-05 2024-09-13T11:26:28.521 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:28.521 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:26:28.521 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:28.521 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:28.521 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:28.521 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:84621 3.1d 3:becd72aa:::benchmark_data_smithi177_31116_object84620:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-09-13T11:26:28.521 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:24:05.741008+0000", 2024-09-13T11:26:28.521 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 142.74200109899999, 2024-09-13T11:26:28.521 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.17994101100000001, 2024-09-13T11:26:28.521 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:26:28.521 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:26:28.521 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:28.522 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:26:28.522 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:26:28.522 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:28.522 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 84621 2024-09-13T11:26:28.522 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:28.522 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:26:28.522 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:28.522 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:26:28.522 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.741008+0000", 2024-09-13T11:26:28.522 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:28.522 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:28.522 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:28.523 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:26:28.523 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.741008+0000", 2024-09-13T11:26:28.523 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:28.523 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:28.523 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:28.523 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:26:28.523 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.741010+0000", 2024-09-13T11:26:28.523 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.898e-06 2024-09-13T11:26:28.523 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:28.523 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:28.523 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:26:28.523 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.741061+0000", 2024-09-13T11:26:28.523 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.0898e-05 2024-09-13T11:26:28.524 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:28.524 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:28.524 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:26:28.524 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.741085+0000", 2024-09-13T11:26:28.524 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.3558e-05 2024-09-13T11:26:28.524 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:28.524 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:28.524 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:26:28.524 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.741093+0000", 2024-09-13T11:26:28.524 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.3890000000000005e-06 2024-09-13T11:26:28.524 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:28.525 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:28.525 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:26:28.525 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.741178+0000", 2024-09-13T11:26:28.525 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.4792999999999999e-05 2024-09-13T11:26:28.525 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:28.525 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:28.525 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:26:28.525 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.741386+0000", 2024-09-13T11:26:28.525 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000207505 2024-09-13T11:26:28.525 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:28.525 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:28.525 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 15", 2024-09-13T11:26:28.526 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.741597+0000", 2024-09-13T11:26:28.526 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00021158200000000001 2024-09-13T11:26:28.526 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:28.526 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:28.526 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:26:28.526 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.743392+0000", 2024-09-13T11:26:28.526 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00179518 2024-09-13T11:26:28.526 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:28.526 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:28.526 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:28.526 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.920866+0000", 2024-09-13T11:26:28.526 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.17747373699999999 2024-09-13T11:26:28.526 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:28.527 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:28.527 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:26:28.527 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.920913+0000", 2024-09-13T11:26:28.527 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.6451e-05 2024-09-13T11:26:28.527 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:28.527 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:28.527 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:26:28.527 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.920949+0000", 2024-09-13T11:26:28.527 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.6019999999999997e-05 2024-09-13T11:26:28.527 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:28.527 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:26:28.527 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:28.527 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:28.528 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:28.528 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPush(3.25 124/120 [PushOp(3:a6d4ee3f:::benchmark_data_smithi177_31116_object14712:head, version: 51'872, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:a6d4ee3f:::benchmark_data_smithi177_31116_object14712:head@51'872, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:26:28.528 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:24:31.996253+0000", 2024-09-13T11:26:28.528 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 116.486755327, 2024-09-13T11:26:28.528 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.17694863499999999, 2024-09-13T11:26:28.528 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:26:28.528 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:26:28.528 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:26:28.528 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:26:28.528 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:28.528 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:26:28.528 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:31.996253+0000", 2024-09-13T11:26:28.529 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:28.529 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:28.529 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:28.529 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:26:28.529 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:31.996253+0000", 2024-09-13T11:26:28.529 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:28.529 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:28.529 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:28.529 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:26:28.529 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:31.996257+0000", 2024-09-13T11:26:28.529 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.2629999999999999e-06 2024-09-13T11:26:28.529 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:28.529 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:28.530 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:26:28.530 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:31.996359+0000", 2024-09-13T11:26:28.530 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000102456 2024-09-13T11:26:28.530 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:28.530 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:28.530 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:26:28.530 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:32.172313+0000", 2024-09-13T11:26:28.530 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.17595388000000001 2024-09-13T11:26:28.530 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:28.530 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:28.530 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:26:28.530 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:32.172326+0000", 2024-09-13T11:26:28.530 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2632999999999999e-05 2024-09-13T11:26:28.531 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:28.531 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:28.531 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:26:28.531 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:32.172423+0000", 2024-09-13T11:26:28.531 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.7084999999999996e-05 2024-09-13T11:26:28.531 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:28.531 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:28.531 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:26:28.531 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:32.172438+0000", 2024-09-13T11:26:28.531 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5775999999999999e-05 2024-09-13T11:26:28.531 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:28.531 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:28.531 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:26:28.532 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:32.173202+0000", 2024-09-13T11:26:28.532 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00076354199999999998 2024-09-13T11:26:28.532 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:28.532 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:26:28.532 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:28.532 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:28.532 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:28.532 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:126039 3.29 3:96100181:::benchmark_data_smithi177_31116_object126038:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e130)", 2024-09-13T11:26:28.532 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:24:43.012785+0000", 2024-09-13T11:26:28.532 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 105.470223977, 2024-09-13T11:26:28.532 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.26685311900000003, 2024-09-13T11:26:28.532 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:26:28.533 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:26:28.533 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:28.533 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:26:28.533 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:26:28.533 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:28.533 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 126039 2024-09-13T11:26:28.533 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:28.533 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:26:28.533 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:28.533 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:26:28.533 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:43.012785+0000", 2024-09-13T11:26:28.533 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:28.534 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:28.534 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:28.534 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:26:28.534 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:43.012785+0000", 2024-09-13T11:26:28.534 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:28.534 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:28.534 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:28.534 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:26:28.534 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:43.012787+0000", 2024-09-13T11:26:28.534 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.875e-06 2024-09-13T11:26:28.534 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:28.534 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:28.534 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:26:28.535 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:43.012826+0000", 2024-09-13T11:26:28.535 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.9483999999999997e-05 2024-09-13T11:26:28.535 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:28.535 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:28.535 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:26:28.535 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:43.012839+0000", 2024-09-13T11:26:28.535 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3405e-05 2024-09-13T11:26:28.535 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:28.535 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:28.535 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:26:28.535 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:43.012845+0000", 2024-09-13T11:26:28.536 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.9100000000000002e-06 2024-09-13T11:26:28.536 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:28.536 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:28.536 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:26:28.536 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:43.012903+0000", 2024-09-13T11:26:28.536 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.8131000000000003e-05 2024-09-13T11:26:28.536 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:28.536 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:28.536 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:26:28.536 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:43.013066+0000", 2024-09-13T11:26:28.536 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000162058 2024-09-13T11:26:28.537 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:28.537 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:28.537 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 4,5", 2024-09-13T11:26:28.537 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:43.013235+0000", 2024-09-13T11:26:28.537 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00016929600000000001 2024-09-13T11:26:28.537 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:28.537 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:28.537 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:28.537 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:43.017063+0000", 2024-09-13T11:26:28.537 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0038283890000000002 2024-09-13T11:26:28.537 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:28.537 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:28.538 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:26:28.538 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:43.017086+0000", 2024-09-13T11:26:28.538 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.2934000000000001e-05 2024-09-13T11:26:28.538 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:28.538 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:28.538 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:28.538 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:43.278804+0000", 2024-09-13T11:26:28.538 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.26171751799999998 2024-09-13T11:26:28.538 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:28.538 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:28.538 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:26:28.538 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:43.278862+0000", 2024-09-13T11:26:28.538 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.8431999999999999e-05 2024-09-13T11:26:28.539 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:28.539 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:28.539 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:26:28.539 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:43.279638+0000", 2024-09-13T11:26:28.539 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00077568699999999995 2024-09-13T11:26:28.539 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:28.539 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:26:28.539 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:28.539 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:28.540 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:28.540 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop_reply(client.4665.0:126039 3.29 e130/120 ondisk, result = 0)", 2024-09-13T11:26:28.540 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:24:43.021691+0000", 2024-09-13T11:26:28.540 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 105.46131805100001, 2024-09-13T11:26:28.540 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.25795579699999999, 2024-09-13T11:26:28.540 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:26:28.540 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:26:28.540 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:26:28.540 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:26:28.540 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:28.540 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:26:28.541 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:43.021691+0000", 2024-09-13T11:26:28.541 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:28.541 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:28.541 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:28.541 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:26:28.541 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:43.021691+0000", 2024-09-13T11:26:28.541 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:28.541 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:28.541 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:28.541 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:26:28.541 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:43.021692+0000", 2024-09-13T11:26:28.541 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3880000000000001e-06 2024-09-13T11:26:28.542 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:28.542 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:28.542 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:26:28.542 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:43.021700+0000", 2024-09-13T11:26:28.542 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.9310000000000008e-06 2024-09-13T11:26:28.542 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:28.542 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:28.542 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:26:28.542 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:43.278719+0000", 2024-09-13T11:26:28.542 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.25701909299999998 2024-09-13T11:26:28.542 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:28.542 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:28.543 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:26:28.543 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:43.278728+0000", 2024-09-13T11:26:28.543 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.5000000000000005e-06 2024-09-13T11:26:28.543 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:28.543 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:28.543 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:26:28.543 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:43.278786+0000", 2024-09-13T11:26:28.543 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.7640999999999999e-05 2024-09-13T11:26:28.543 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:28.543 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:28.543 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:26:28.543 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:43.278796+0000", 2024-09-13T11:26:28.543 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.9199999999999999e-06 2024-09-13T11:26:28.544 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:28.544 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:28.544 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:26:28.544 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:43.279646+0000", 2024-09-13T11:26:28.544 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00085032399999999996 2024-09-13T11:26:28.544 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:28.544 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:26:28.544 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:28.544 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:28.544 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:28.544 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPushReply(3.29 134/120 [PushReplyOp(3:95648f22:::benchmark_data_smithi177_31116_object7446:head)])", 2024-09-13T11:26:28.544 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:24:47.110443+0000", 2024-09-13T11:26:28.545 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 101.372565931, 2024-09-13T11:26:28.545 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.171602386, 2024-09-13T11:26:28.545 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:26:28.545 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:26:28.545 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-09-13T11:26:28.545 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:26:28.545 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:28.545 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:26:28.545 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:47.110443+0000", 2024-09-13T11:26:28.545 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:28.545 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:28.545 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:28.546 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:26:28.546 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:47.110443+0000", 2024-09-13T11:26:28.546 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:28.546 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:28.546 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:28.546 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:26:28.546 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:47.110444+0000", 2024-09-13T11:26:28.546 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.192e-06 2024-09-13T11:26:28.546 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:28.546 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:28.546 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:26:28.546 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:47.110449+0000", 2024-09-13T11:26:28.546 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.0869999999999997e-06 2024-09-13T11:26:28.547 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:28.547 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:28.547 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:26:28.547 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:47.280321+0000", 2024-09-13T11:26:28.547 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.169872157 2024-09-13T11:26:28.547 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:28.547 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:28.547 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:26:28.547 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:47.280337+0000", 2024-09-13T11:26:28.547 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5656000000000001e-05 2024-09-13T11:26:28.547 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:28.547 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:28.548 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:26:28.548 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:47.280414+0000", 2024-09-13T11:26:28.548 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.7083999999999996e-05 2024-09-13T11:26:28.548 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:28.548 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:28.548 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:26:28.548 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:47.282045+0000", 2024-09-13T11:26:28.548 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0016312099999999999 2024-09-13T11:26:28.548 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:28.548 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:26:28.548 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:28.548 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:28.549 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:28.549 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.4665.0:138294 3.9 e140/117 3:90012dab:::benchmark_data_smithi177_31116_object138293:head v 140'5246, mlcod=140'5243)", 2024-09-13T11:26:28.549 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:24:54.252227+0000", 2024-09-13T11:26:28.549 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 94.230781562999994, 2024-09-13T11:26:28.549 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.32919836400000002, 2024-09-13T11:26:28.549 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:26:28.549 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:26:28.549 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:28.549 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:26:28.549 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:28.549 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:26:28.549 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:54.252227+0000", 2024-09-13T11:26:28.550 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:28.550 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:28.550 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:28.550 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:26:28.550 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:54.252227+0000", 2024-09-13T11:26:28.550 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:28.550 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:28.550 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:28.550 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:26:28.550 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:54.252228+0000", 2024-09-13T11:26:28.550 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.8899999999999998e-07 2024-09-13T11:26:28.550 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:28.551 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:28.551 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:26:28.551 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:54.252276+0000", 2024-09-13T11:26:28.551 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.8137999999999999e-05 2024-09-13T11:26:28.551 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:28.551 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:28.551 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:26:28.551 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:54.578216+0000", 2024-09-13T11:26:28.551 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.32594014399999999 2024-09-13T11:26:28.551 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:28.551 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:28.551 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:26:28.551 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:54.578235+0000", 2024-09-13T11:26:28.552 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.8943000000000001e-05 2024-09-13T11:26:28.552 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:28.552 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:28.552 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:26:28.552 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:54.578396+0000", 2024-09-13T11:26:28.552 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000160922 2024-09-13T11:26:28.552 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:28.552 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:28.552 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:26:28.552 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:54.578457+0000", 2024-09-13T11:26:28.552 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.0665999999999998e-05 2024-09-13T11:26:28.553 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:28.553 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:28.553 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:26:28.553 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:54.581339+0000", 2024-09-13T11:26:28.553 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0028826839999999999 2024-09-13T11:26:28.553 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:28.553 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:28.553 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:26:28.553 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:54.581425+0000", 2024-09-13T11:26:28.553 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.6077999999999998e-05 2024-09-13T11:26:28.553 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:28.553 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:26:28.554 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:28.554 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:28.554 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:28.554 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPushReply(3.29 143/120 [PushReplyOp(3:973d2047:::benchmark_data_smithi177_31116_object49749:head)])", 2024-09-13T11:26:28.554 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:24:57.387106+0000", 2024-09-13T11:26:28.554 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 91.095902236000001, 2024-09-13T11:26:28.554 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.64813637000000002, 2024-09-13T11:26:28.554 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:26:28.554 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:26:28.554 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-09-13T11:26:28.554 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:26:28.554 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:28.555 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:26:28.555 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:57.387106+0000", 2024-09-13T11:26:28.555 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:28.555 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:28.555 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:28.555 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:26:28.555 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:57.387106+0000", 2024-09-13T11:26:28.555 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:28.555 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:28.555 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:28.555 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:26:28.555 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:57.387116+0000", 2024-09-13T11:26:28.555 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.1530000000000003e-06 2024-09-13T11:26:28.556 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:28.556 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:28.556 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:26:28.556 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:57.387127+0000", 2024-09-13T11:26:28.556 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1197e-05 2024-09-13T11:26:28.556 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:28.556 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:28.556 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:26:28.556 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:58.034264+0000", 2024-09-13T11:26:28.556 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.64713691799999995 2024-09-13T11:26:28.556 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:28.556 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:28.557 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:26:28.557 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:58.034273+0000", 2024-09-13T11:26:28.557 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.0929999999999992e-06 2024-09-13T11:26:28.557 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:28.557 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:28.557 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:26:28.557 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:58.034321+0000", 2024-09-13T11:26:28.557 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.8396999999999998e-05 2024-09-13T11:26:28.557 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:28.557 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:28.557 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:26:28.557 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:58.035243+0000", 2024-09-13T11:26:28.557 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00092161200000000004 2024-09-13T11:26:28.558 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:28.558 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:26:28.558 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:28.558 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:28.558 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:28.558 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPushReply(3.29 144/120 [PushReplyOp(3:94a6fc3a:::benchmark_data_smithi177_31116_object50245:head)])", 2024-09-13T11:26:28.558 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:24:58.320977+0000", 2024-09-13T11:26:28.558 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 90.162031653, 2024-09-13T11:26:28.558 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.31110569599999999, 2024-09-13T11:26:28.558 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:26:28.558 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:26:28.558 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-09-13T11:26:28.559 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:26:28.559 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:28.559 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:26:28.559 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:58.320977+0000", 2024-09-13T11:26:28.559 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:28.559 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:28.559 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:28.559 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:26:28.559 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:58.320977+0000", 2024-09-13T11:26:28.559 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:28.559 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:28.559 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:28.559 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:26:28.559 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:58.320978+0000", 2024-09-13T11:26:28.560 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.8700000000000004e-07 2024-09-13T11:26:28.560 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:28.560 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:28.560 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:26:28.560 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:58.320985+0000", 2024-09-13T11:26:28.560 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.742e-06 2024-09-13T11:26:28.560 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:28.560 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:28.560 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:26:28.560 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:58.631147+0000", 2024-09-13T11:26:28.560 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.31016212900000001 2024-09-13T11:26:28.560 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:28.561 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:28.561 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:26:28.561 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:58.631159+0000", 2024-09-13T11:26:28.561 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1946999999999999e-05 2024-09-13T11:26:28.561 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:28.561 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:28.561 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:26:28.561 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:58.631257+0000", 2024-09-13T11:26:28.561 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.8096000000000004e-05 2024-09-13T11:26:28.561 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:28.561 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:28.561 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:26:28.561 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:58.632083+0000", 2024-09-13T11:26:28.561 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00082589499999999999 2024-09-13T11:26:28.562 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:28.562 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:26:28.562 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:28.562 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:28.562 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:28.562 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:147568 3.1d 3:b82a2ed2:::benchmark_data_smithi177_31116_object7265:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e148)", 2024-09-13T11:26:28.562 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:25:04.213894+0000", 2024-09-13T11:26:28.562 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 84.269114881999997, 2024-09-13T11:26:28.562 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.16897441999999999, 2024-09-13T11:26:28.562 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:26:28.562 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:26:28.563 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:28.563 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:26:28.563 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:26:28.563 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:28.563 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 147568 2024-09-13T11:26:28.563 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:28.563 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:26:28.563 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:28.563 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:26:28.563 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:04.213894+0000", 2024-09-13T11:26:28.563 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:28.564 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:28.564 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:28.564 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:26:28.564 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:04.213894+0000", 2024-09-13T11:26:28.564 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:28.564 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:28.564 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:28.564 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:26:28.564 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:04.213898+0000", 2024-09-13T11:26:28.564 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.9049999999999999e-06 2024-09-13T11:26:28.564 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:28.564 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:28.564 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:26:28.564 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:04.213908+0000", 2024-09-13T11:26:28.565 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0069e-05 2024-09-13T11:26:28.565 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:28.565 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:28.565 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:26:28.565 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:04.213931+0000", 2024-09-13T11:26:28.565 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.3538999999999999e-05 2024-09-13T11:26:28.565 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:28.565 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:28.565 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:26:28.565 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:04.213941+0000", 2024-09-13T11:26:28.565 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0073e-05 2024-09-13T11:26:28.566 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:28.566 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:28.566 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:26:28.566 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:04.214039+0000", 2024-09-13T11:26:28.566 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.7666999999999998e-05 2024-09-13T11:26:28.566 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:28.566 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:28.566 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:26:28.566 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:04.214226+0000", 2024-09-13T11:26:28.566 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000187166 2024-09-13T11:26:28.566 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:28.566 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:28.567 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 15", 2024-09-13T11:26:28.567 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:04.214365+0000", 2024-09-13T11:26:28.567 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000139016 2024-09-13T11:26:28.567 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:28.567 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:28.567 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:26:28.567 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:04.217864+0000", 2024-09-13T11:26:28.567 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.003499177 2024-09-13T11:26:28.567 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:28.567 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:28.567 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:28.567 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:04.382775+0000", 2024-09-13T11:26:28.567 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.16491091499999999 2024-09-13T11:26:28.568 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:28.568 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:28.568 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:26:28.568 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:04.382823+0000", 2024-09-13T11:26:28.568 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.8232000000000002e-05 2024-09-13T11:26:28.568 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:28.568 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:28.568 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:26:28.568 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:04.382868+0000", 2024-09-13T11:26:28.568 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.4660999999999998e-05 2024-09-13T11:26:28.568 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:28.568 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:26:28.568 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:28.569 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:28.569 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:28.569 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.4665.0:170721 3.9 e162/117 3:93244b1b:::benchmark_data_smithi177_31116_object30418:head v 162'5743, mlcod=162'5740)", 2024-09-13T11:26:28.569 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:25:21.721723+0000", 2024-09-13T11:26:28.569 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 66.761285938, 2024-09-13T11:26:28.569 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.19722754300000001, 2024-09-13T11:26:28.569 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:26:28.569 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:26:28.569 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:28.569 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:26:28.569 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:28.569 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:26:28.570 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:21.721723+0000", 2024-09-13T11:26:28.570 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:28.570 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:28.570 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:28.570 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:26:28.570 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:21.721723+0000", 2024-09-13T11:26:28.570 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:28.570 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:28.570 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:28.570 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:26:28.570 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:21.721726+0000", 2024-09-13T11:26:28.570 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.1930000000000002e-06 2024-09-13T11:26:28.570 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:28.571 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:28.571 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:26:28.571 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:21.721741+0000", 2024-09-13T11:26:28.571 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5299999999999999e-05 2024-09-13T11:26:28.571 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:28.571 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:28.571 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:26:28.571 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:21.915450+0000", 2024-09-13T11:26:28.571 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.193708519 2024-09-13T11:26:28.571 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:28.571 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:28.571 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:26:28.571 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:21.915462+0000", 2024-09-13T11:26:28.571 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2414e-05 2024-09-13T11:26:28.572 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:28.572 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:28.572 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:26:28.572 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:21.915603+0000", 2024-09-13T11:26:28.572 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00014116500000000001 2024-09-13T11:26:28.572 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:28.572 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:28.572 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:26:28.572 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:21.915668+0000", 2024-09-13T11:26:28.572 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.4488999999999999e-05 2024-09-13T11:26:28.572 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:28.572 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:28.572 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:26:28.572 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:21.918850+0000", 2024-09-13T11:26:28.573 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0031825310000000002 2024-09-13T11:26:28.573 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:28.573 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:28.573 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:26:28.573 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:21.918950+0000", 2024-09-13T11:26:28.573 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.9932000000000005e-05 2024-09-13T11:26:28.573 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:28.573 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:26:28.573 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:28.573 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:28.573 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:28.573 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:191058 3.1d 3:bd6e7029:::benchmark_data_smithi177_31116_object50755:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e175)", 2024-09-13T11:26:28.574 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:25:38.512403+0000", 2024-09-13T11:26:28.574 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 49.970605393, 2024-09-13T11:26:28.574 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.38726333200000002, 2024-09-13T11:26:28.574 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:26:28.574 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:26:28.574 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:28.574 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:26:28.574 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:26:28.574 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:28.574 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 191058 2024-09-13T11:26:28.574 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:28.574 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:26:28.574 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:28.574 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:26:28.575 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:38.512403+0000", 2024-09-13T11:26:28.575 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:28.575 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:28.575 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:28.575 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:26:28.575 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:38.512403+0000", 2024-09-13T11:26:28.575 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:28.575 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:28.575 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:28.575 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:26:28.575 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:38.512406+0000", 2024-09-13T11:26:28.575 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.4059999999999999e-06 2024-09-13T11:26:28.575 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:28.576 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:28.576 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:26:28.576 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:38.512412+0000", 2024-09-13T11:26:28.576 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.8170000000000001e-06 2024-09-13T11:26:28.576 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:28.576 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:28.576 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:26:28.576 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:38.512430+0000", 2024-09-13T11:26:28.576 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.7397999999999998e-05 2024-09-13T11:26:28.576 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:28.576 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:28.576 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:26:28.576 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:38.512438+0000", 2024-09-13T11:26:28.577 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.3459999999999994e-06 2024-09-13T11:26:28.577 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:28.577 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:28.577 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:26:28.577 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:38.512526+0000", 2024-09-13T11:26:28.577 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.8146999999999995e-05 2024-09-13T11:26:28.577 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:28.577 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:28.577 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:26:28.577 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:38.512722+0000", 2024-09-13T11:26:28.577 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00019587599999999999 2024-09-13T11:26:28.577 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:28.577 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:28.578 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 15", 2024-09-13T11:26:28.578 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:38.512882+0000", 2024-09-13T11:26:28.578 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00015957300000000001 2024-09-13T11:26:28.578 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:28.578 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:28.578 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:26:28.578 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:38.517854+0000", 2024-09-13T11:26:28.578 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0049725500000000001 2024-09-13T11:26:28.578 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:28.578 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:28.578 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:28.578 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:38.899562+0000", 2024-09-13T11:26:28.578 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.38170719800000003 2024-09-13T11:26:28.578 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:28.579 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:28.579 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:26:28.579 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:38.899622+0000", 2024-09-13T11:26:28.579 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.0396999999999998e-05 2024-09-13T11:26:28.579 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:28.579 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:28.579 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:26:28.579 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:38.899667+0000", 2024-09-13T11:26:28.579 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.4623999999999998e-05 2024-09-13T11:26:28.579 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:28.579 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:26:28.579 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:28.579 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:28.580 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:28.580 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.4665.0:191547 3.25 e176/141 3:a7ac3e5b:::benchmark_data_smithi177_31116_object51244:head v 176'6020, mlcod=175'6018)", 2024-09-13T11:26:28.580 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:25:39.496290+0000", 2024-09-13T11:26:28.580 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 48.986718832000001, 2024-09-13T11:26:28.580 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.179206057, 2024-09-13T11:26:28.580 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:26:28.580 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:26:28.580 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:28.580 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:26:28.580 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:28.580 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:26:28.580 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:39.496290+0000", 2024-09-13T11:26:28.580 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:28.581 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:28.581 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:28.581 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:26:28.581 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:39.496290+0000", 2024-09-13T11:26:28.581 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:28.581 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:28.581 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:28.581 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:26:28.581 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:39.496293+0000", 2024-09-13T11:26:28.581 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.8499999999999998e-06 2024-09-13T11:26:28.581 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:28.581 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:28.581 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:26:28.582 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:39.496305+0000", 2024-09-13T11:26:28.582 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2286999999999999e-05 2024-09-13T11:26:28.582 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:28.582 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:28.582 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:26:28.582 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:39.670614+0000", 2024-09-13T11:26:28.582 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.17430884699999999 2024-09-13T11:26:28.582 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:28.582 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:28.582 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:26:28.582 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:39.670625+0000", 2024-09-13T11:26:28.582 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1233000000000001e-05 2024-09-13T11:26:28.583 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:28.583 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:28.583 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:26:28.583 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:39.670722+0000", 2024-09-13T11:26:28.583 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.6885999999999994e-05 2024-09-13T11:26:28.583 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:28.583 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:28.583 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:26:28.583 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:39.670756+0000", 2024-09-13T11:26:28.583 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.4125999999999998e-05 2024-09-13T11:26:28.583 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:28.583 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:28.583 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:26:28.583 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:39.675411+0000", 2024-09-13T11:26:28.583 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0046546160000000003 2024-09-13T11:26:28.584 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:28.584 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:28.584 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:26:28.584 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:39.675496+0000", 2024-09-13T11:26:28.584 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.5211999999999997e-05 2024-09-13T11:26:28.584 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:28.584 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:26:28.584 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:28.585 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:28.585 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:26:28.585 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-13T11:26:28.586 INFO:teuthology.orchestra.run.smithi001.stdout:osd.1: {} 2024-09-13T11:26:28.587 INFO:tasks.ceph.osd.11.smithi177.stderr:2024-09-13T11:26:28.581+0000 7f24792c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26580) UID: 0 2024-09-13T11:26:28.587 INFO:teuthology.orchestra.run.smithi001.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T11:26:28.587 INFO:teuthology.orchestra.run.smithi001.stdout:osd.2: {} 2024-09-13T11:26:28.588 INFO:teuthology.orchestra.run.smithi001.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T11:26:28.588 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-13T11:26:28.588 INFO:teuthology.orchestra.run.smithi001.stdout:osd.3: {} 2024-09-13T11:26:28.589 INFO:teuthology.orchestra.run.smithi001.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T11:26:28.589 INFO:teuthology.orchestra.run.smithi001.stdout:osd.4: {} 2024-09-13T11:26:28.589 INFO:teuthology.orchestra.run.smithi001.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T11:26:28.608 INFO:teuthology.orchestra.run.smithi001.stdout:osd.5: {} 2024-09-13T11:26:28.608 INFO:teuthology.orchestra.run.smithi001.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T11:26:28.635 INFO:teuthology.orchestra.run.smithi001.stdout:osd.6: {} 2024-09-13T11:26:28.636 INFO:teuthology.orchestra.run.smithi001.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T11:26:28.663 INFO:teuthology.orchestra.run.smithi001.stdout:osd.7: {} 2024-09-13T11:26:28.663 INFO:teuthology.orchestra.run.smithi001.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T11:26:28.687 INFO:tasks.ceph.osd.4.smithi001.stderr:2024-09-13T11:26:28.682+0000 7ff8e0218640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26310) UID: 0 2024-09-13T11:26:28.691 INFO:teuthology.orchestra.run.smithi001.stdout:osd.8: {} 2024-09-13T11:26:28.691 INFO:teuthology.orchestra.run.smithi001.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-13T11:26:28.720 INFO:teuthology.orchestra.run.smithi001.stdout:osd.9: {} 2024-09-13T11:26:28.721 INFO:teuthology.orchestra.run.smithi001.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-13T11:26:28.730 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-13T11:26:28.730 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:26:28.749 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-13T11:26:28.750 INFO:teuthology.orchestra.run.smithi001.stdout:osd.10: {} 2024-09-13T11:26:28.750 INFO:teuthology.orchestra.run.smithi001.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-13T11:26:28.777 INFO:teuthology.orchestra.run.smithi001.stdout:osd.11: {} 2024-09-13T11:26:28.777 INFO:teuthology.orchestra.run.smithi001.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-13T11:26:28.789 INFO:tasks.ceph.osd.11.smithi177.stderr:2024-09-13T11:26:28.785+0000 7f24792c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26580) UID: 0 2024-09-13T11:26:28.805 INFO:teuthology.orchestra.run.smithi001.stdout:osd.12: {} 2024-09-13T11:26:28.805 INFO:teuthology.orchestra.run.smithi001.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-13T11:26:28.829 INFO:teuthology.orchestra.run.smithi001.stdout:osd.13: {} 2024-09-13T11:26:28.829 INFO:teuthology.orchestra.run.smithi001.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-13T11:26:28.858 INFO:teuthology.orchestra.run.smithi001.stdout:osd.14: {} 2024-09-13T11:26:28.858 INFO:teuthology.orchestra.run.smithi001.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-13T11:26:28.883 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-13T11:26:28.883 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:26:28.884 INFO:teuthology.orchestra.run.smithi001.stdout:osd.15: {} 2024-09-13T11:26:28.884 INFO:teuthology.orchestra.run.smithi001.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-13T11:26:28.888 INFO:tasks.ceph.osd.8.smithi001.stderr:2024-09-13T11:26:28.886+0000 7fb9b48ae640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26312) UID: 0 2024-09-13T11:26:28.901 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-13T11:26:28.989 INFO:tasks.ceph.osd.15.smithi177.stderr:2024-09-13T11:26:28.985+0000 7f9d6ce77640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26578) UID: 0 2024-09-13T11:26:29.038 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-13T11:26:29.038 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:26:29.054 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-09-13T11:26:29.089 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-13T11:26:29.086+0000 7f6a68775640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26307) UID: 0 2024-09-13T11:26:29.183 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-13T11:26:29.184 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:26:29.190 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-09-13T11:26:29.187+0000 7f97befc9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26315) UID: 0 2024-09-13T11:26:29.198 INFO:teuthology.orchestra.run.smithi001.stderr:nodeep-scrub is set 2024-09-13T11:26:29.202 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-09-13T11:26:29.291 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-09-13T11:26:29.287+0000 7f97befc9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26315) UID: 0 2024-09-13T11:26:29.387 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-13T11:26:29.387 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:26:29.392 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-09-13T11:26:29.391+0000 7f97befc9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26315) UID: 0 2024-09-13T11:26:29.403 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-09-13T11:26:29.493 INFO:tasks.ceph.osd.11.smithi177.stderr:2024-09-13T11:26:29.489+0000 7f24792c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26580) UID: 0 2024-09-13T11:26:29.574 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-13T11:26:29.574 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:26:29.590 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-13T11:26:29.634 INFO:tasks.ceph.osd.11.smithi177.stderr:2024-09-13T11:26:29.633+0000 7f24792c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26580) UID: 0 2024-09-13T11:26:29.694 INFO:tasks.ceph.osd.4.smithi001.stderr:2024-09-13T11:26:29.690+0000 7ff8e0218640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26310) UID: 0 2024-09-13T11:26:29.739 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-09-13T11:26:29.739 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:26:29.757 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-13T11:26:29.796 INFO:tasks.ceph.osd.10.smithi162.stderr:2024-09-13T11:26:29.791+0000 7fb59c9f0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26317) UID: 0 2024-09-13T11:26:29.897 INFO:tasks.ceph.osd.5.smithi092.stderr:2024-09-13T11:26:29.896+0000 7f6f0b705640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26346) UID: 0 2024-09-13T11:26:29.947 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-09-13T11:26:29.947 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:26:29.965 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-13T11:26:29.997 INFO:tasks.ceph.osd.4.smithi001.stderr:2024-09-13T11:26:29.994+0000 7ff8e0218640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26310) UID: 0 2024-09-13T11:26:30.098 INFO:tasks.ceph.osd.7.smithi177.stderr:2024-09-13T11:26:30.097+0000 7f897b51d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26575) UID: 0 2024-09-13T11:26:30.117 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-09-13T11:26:30.117 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:26:30.137 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-13T11:26:30.199 INFO:tasks.ceph.osd.5.smithi092.stderr:2024-09-13T11:26:30.196+0000 7f6f0b705640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26346) UID: 0 2024-09-13T11:26:30.300 INFO:tasks.ceph.osd.10.smithi162.stderr:2024-09-13T11:26:30.295+0000 7fb59c9f0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26317) UID: 0 2024-09-13T11:26:30.316 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-09-13T11:26:30.316 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:26:30.334 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-13T11:26:30.401 INFO:tasks.ceph.osd.9.smithi092.stderr:2024-09-13T11:26:30.400+0000 7ff5ab3d0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26348) UID: 0 2024-09-13T11:26:30.473 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-09-13T11:26:30.473 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:26:30.492 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-13T11:26:30.502 INFO:tasks.ceph.osd.14.smithi162.stderr:2024-09-13T11:26:30.499+0000 7f3df5903640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33154) UID: 0 2024-09-13T11:26:30.602 INFO:tasks.ceph.osd.4.smithi001.stderr:2024-09-13T11:26:30.598+0000 7ff8e0218640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26310) UID: 0 2024-09-13T11:26:30.617 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-09-13T11:26:30.618 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:26:30.637 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-09-13T11:26:30.703 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-09-13T11:26:30.699+0000 7f97befc9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26315) UID: 0 2024-09-13T11:26:30.771 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-09-13T11:26:30.771 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:26:30.787 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-09-13T11:26:30.805 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-13T11:26:30.802+0000 7f6a68775640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26307) UID: 0 2024-09-13T11:26:30.904 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T11:26:30.906 INFO:tasks.ceph.osd.4.smithi001.stderr:2024-09-13T11:26:30.902+0000 7ff8e0218640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26310) UID: 0 2024-09-13T11:26:30.912 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-09-13T11:26:30.912 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:26:30.927 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-09-13T11:26:31.007 INFO:tasks.ceph.osd.9.smithi092.stderr:2024-09-13T11:26:31.004+0000 7ff5ab3d0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26348) UID: 0 2024-09-13T11:26:31.045 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-09-13T11:26:31.046 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:26:31.062 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_ops_in_flight 2024-09-13T11:26:31.075 INFO:teuthology.orchestra.run.smithi001.stdout:osd.0: {} 2024-09-13T11:26:31.075 INFO:teuthology.orchestra.run.smithi001.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T11:26:31.106 INFO:teuthology.orchestra.run.smithi001.stdout:osd.1: {} 2024-09-13T11:26:31.106 INFO:teuthology.orchestra.run.smithi001.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T11:26:31.107 INFO:tasks.ceph.osd.15.smithi177.stderr:2024-09-13T11:26:31.105+0000 7f9d6ce77640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26578) UID: 0 2024-09-13T11:26:31.134 INFO:teuthology.orchestra.run.smithi001.stdout:osd.2: {} 2024-09-13T11:26:31.135 INFO:teuthology.orchestra.run.smithi001.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T11:26:31.162 INFO:teuthology.orchestra.run.smithi001.stdout:osd.3: {} 2024-09-13T11:26:31.162 INFO:teuthology.orchestra.run.smithi001.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T11:26:31.191 INFO:teuthology.orchestra.run.smithi001.stdout:osd.4: {} 2024-09-13T11:26:31.191 INFO:teuthology.orchestra.run.smithi001.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T11:26:31.201 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-09-13T11:26:31.201 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:26:31.208 INFO:tasks.ceph.osd.13.smithi092.stderr:2024-09-13T11:26:31.204+0000 7f3e5afea640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26350) UID: 0 2024-09-13T11:26:31.217 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_blocked_ops 2024-09-13T11:26:31.218 INFO:teuthology.orchestra.run.smithi001.stdout:osd.5: {} 2024-09-13T11:26:31.218 INFO:teuthology.orchestra.run.smithi001.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T11:26:31.233 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-13T11:26:31.251 INFO:teuthology.orchestra.run.smithi001.stdout:osd.6: {} 2024-09-13T11:26:31.251 INFO:teuthology.orchestra.run.smithi001.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T11:26:31.279 INFO:teuthology.orchestra.run.smithi001.stdout:osd.7: {} 2024-09-13T11:26:31.279 INFO:teuthology.orchestra.run.smithi001.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T11:26:31.304 INFO:teuthology.orchestra.run.smithi001.stdout:osd.8: {} 2024-09-13T11:26:31.304 INFO:teuthology.orchestra.run.smithi001.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-13T11:26:31.308 INFO:tasks.ceph.osd.8.smithi001.stderr:2024-09-13T11:26:31.306+0000 7fb9b48ae640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26312) UID: 0 2024-09-13T11:26:31.335 INFO:teuthology.orchestra.run.smithi001.stdout:osd.9: {} 2024-09-13T11:26:31.335 INFO:teuthology.orchestra.run.smithi001.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-13T11:26:31.374 INFO:teuthology.orchestra.run.smithi001.stdout:osd.10: {} 2024-09-13T11:26:31.374 INFO:teuthology.orchestra.run.smithi001.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-13T11:26:31.406 INFO:teuthology.orchestra.run.smithi001.stdout:osd.11: {} 2024-09-13T11:26:31.407 INFO:teuthology.orchestra.run.smithi001.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-13T11:26:31.409 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-09-13T11:26:31.409 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:26:31.411 INFO:tasks.ceph.osd.4.smithi001.stderr:2024-09-13T11:26:31.406+0000 7ff8e0218640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26310) UID: 0 2024-09-13T11:26:31.437 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_historic_ops 2024-09-13T11:26:31.444 INFO:teuthology.orchestra.run.smithi001.stdout:osd.12: {} 2024-09-13T11:26:31.444 INFO:teuthology.orchestra.run.smithi001.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-13T11:26:31.476 INFO:teuthology.orchestra.run.smithi001.stdout:osd.13: {} 2024-09-13T11:26:31.477 INFO:teuthology.orchestra.run.smithi001.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-13T11:26:31.511 INFO:tasks.ceph.osd.8.smithi001.stderr:2024-09-13T11:26:31.506+0000 7fb9b48ae640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26312) UID: 0 2024-09-13T11:26:31.512 INFO:teuthology.orchestra.run.smithi001.stdout:osd.14: {} 2024-09-13T11:26:31.513 INFO:teuthology.orchestra.run.smithi001.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-13T11:26:31.550 INFO:teuthology.orchestra.run.smithi001.stdout:osd.15: {} 2024-09-13T11:26:31.550 INFO:teuthology.orchestra.run.smithi001.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-13T11:26:31.594 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-09-13T11:26:31.595 INFO:teuthology.orchestra.run.smithi177.stdout: "size": 20, 2024-09-13T11:26:31.595 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 600, 2024-09-13T11:26:31.595 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [ 2024-09-13T11:26:31.595 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:31.595 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:25383 3.0 e55/30 3:0a6bb869:::benchmark_data_smithi177_31116_object25382:head v 55'1620, mlcod=55'1617)", 2024-09-13T11:26:31.595 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:04.449956+0000", 2024-09-13T11:26:31.595 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 207.14424189299999, 2024-09-13T11:26:31.595 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.21143479600000001, 2024-09-13T11:26:31.595 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:26:31.595 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:26:31.595 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:31.595 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:26:31.595 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:31.596 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:26:31.596 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.449956+0000", 2024-09-13T11:26:31.596 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:31.596 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:31.596 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:31.596 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:26:31.596 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.449956+0000", 2024-09-13T11:26:31.596 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:31.596 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:31.596 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:31.596 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:26:31.596 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.449959+0000", 2024-09-13T11:26:31.596 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.393e-06 2024-09-13T11:26:31.596 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:31.597 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:31.597 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:26:31.597 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.450014+0000", 2024-09-13T11:26:31.597 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.4200000000000003e-05 2024-09-13T11:26:31.597 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:31.597 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:31.597 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:26:31.597 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.658637+0000", 2024-09-13T11:26:31.597 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.20862334799999999 2024-09-13T11:26:31.597 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:31.597 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:31.597 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:26:31.597 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.658650+0000", 2024-09-13T11:26:31.598 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3148e-05 2024-09-13T11:26:31.598 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:31.598 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:31.598 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:26:31.598 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.658867+0000", 2024-09-13T11:26:31.598 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00021673499999999999 2024-09-13T11:26:31.598 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:31.598 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:31.598 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:26:31.598 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.658931+0000", 2024-09-13T11:26:31.598 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.4045000000000001e-05 2024-09-13T11:26:31.598 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:31.598 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:31.598 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:26:31.599 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.661317+0000", 2024-09-13T11:26:31.599 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.002385823 2024-09-13T11:26:31.599 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:31.599 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:31.599 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:26:31.599 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.661391+0000", 2024-09-13T11:26:31.599 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.4103999999999994e-05 2024-09-13T11:26:31.599 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:31.599 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:26:31.599 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:31.599 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:31.599 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:31.599 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPush(3.16 85/70 [PushOp(3:69aac9fd:::benchmark_data_smithi177_31116_object43914:head, version: 66'2837, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:69aac9fd:::benchmark_data_smithi177_31116_object43914:head@66'2837, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:26:31.599 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:42.295995+0000", 2024-09-13T11:26:31.600 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 169.298202573, 2024-09-13T11:26:31.600 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.215089431, 2024-09-13T11:26:31.600 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:26:31.600 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:26:31.600 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:26:31.600 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:26:31.600 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:31.600 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:26:31.600 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.295995+0000", 2024-09-13T11:26:31.600 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:31.600 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:31.600 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:31.600 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:26:31.600 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.295995+0000", 2024-09-13T11:26:31.601 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:31.601 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:31.601 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:31.601 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:26:31.601 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.295997+0000", 2024-09-13T11:26:31.601 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3170000000000001e-06 2024-09-13T11:26:31.601 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:31.601 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:31.601 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:26:31.601 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.296058+0000", 2024-09-13T11:26:31.601 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.1690000000000001e-05 2024-09-13T11:26:31.601 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:31.601 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:31.601 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:26:31.602 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.509606+0000", 2024-09-13T11:26:31.602 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.21354748300000001 2024-09-13T11:26:31.602 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:31.602 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:31.602 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:26:31.602 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.509638+0000", 2024-09-13T11:26:31.602 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.1644999999999999e-05 2024-09-13T11:26:31.602 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:31.602 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:31.602 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:26:31.602 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.509814+0000", 2024-09-13T11:26:31.602 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00017679499999999999 2024-09-13T11:26:31.602 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:31.602 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:31.603 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:26:31.603 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.509840+0000", 2024-09-13T11:26:31.603 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.5871e-05 2024-09-13T11:26:31.603 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:31.603 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:31.603 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:26:31.603 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.511085+0000", 2024-09-13T11:26:31.603 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00124463 2024-09-13T11:26:31.603 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:31.603 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:26:31.603 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:31.603 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:31.603 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:31.603 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPush(3.1d 91/78 [PushOp(3:be6a0ae4:::benchmark_data_smithi177_31116_object25671:head, version: 58'1569, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:be6a0ae4:::benchmark_data_smithi177_31116_object25671:head@58'1569, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:26:31.603 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:48.481971+0000", 2024-09-13T11:26:31.604 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 163.11222732799999, 2024-09-13T11:26:31.604 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.47945454199999998, 2024-09-13T11:26:31.604 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:26:31.604 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:26:31.604 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:26:31.604 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:26:31.604 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:31.604 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:26:31.604 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.481971+0000", 2024-09-13T11:26:31.604 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:31.604 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:31.604 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:31.604 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:26:31.604 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.481971+0000", 2024-09-13T11:26:31.605 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:31.605 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:31.605 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:31.605 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:26:31.605 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.481972+0000", 2024-09-13T11:26:31.605 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5179999999999999e-06 2024-09-13T11:26:31.605 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:31.605 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:31.605 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:26:31.605 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.482000+0000", 2024-09-13T11:26:31.605 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.8249e-05 2024-09-13T11:26:31.605 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:31.605 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:31.605 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:26:31.606 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.959926+0000", 2024-09-13T11:26:31.606 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.47792585100000001 2024-09-13T11:26:31.606 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:31.606 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:31.606 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:26:31.606 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.959944+0000", 2024-09-13T11:26:31.606 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.7620000000000001e-05 2024-09-13T11:26:31.606 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:31.606 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:31.606 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:26:31.606 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.960059+0000", 2024-09-13T11:26:31.606 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000115188 2024-09-13T11:26:31.606 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:31.606 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:31.607 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:26:31.607 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.960074+0000", 2024-09-13T11:26:31.607 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.4738e-05 2024-09-13T11:26:31.607 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:31.607 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:31.607 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:26:31.607 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.961425+0000", 2024-09-13T11:26:31.607 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0013513780000000001 2024-09-13T11:26:31.607 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:31.607 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:26:31.607 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:31.607 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:31.607 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:31.607 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:71884 3.16 e96/88 3:6ca846b2:::benchmark_data_smithi177_31116_object71883:head v 96'3755, mlcod=96'3753)", 2024-09-13T11:26:31.608 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:54.108299+0000", 2024-09-13T11:26:31.608 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 157.485899155, 2024-09-13T11:26:31.608 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.20685140099999999, 2024-09-13T11:26:31.608 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:26:31.608 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:26:31.608 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:31.608 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:26:31.608 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:31.608 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:26:31.608 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.108299+0000", 2024-09-13T11:26:31.608 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:31.608 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:31.608 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:31.608 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:26:31.609 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.108299+0000", 2024-09-13T11:26:31.609 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:31.609 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:31.609 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:31.609 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:26:31.609 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.108300+0000", 2024-09-13T11:26:31.609 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.173e-06 2024-09-13T11:26:31.609 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:31.609 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:31.609 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:26:31.609 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.108323+0000", 2024-09-13T11:26:31.609 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.2591000000000001e-05 2024-09-13T11:26:31.609 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:31.609 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:31.610 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:26:31.610 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.312377+0000", 2024-09-13T11:26:31.610 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.20405456599999999 2024-09-13T11:26:31.610 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:31.610 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:31.610 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:26:31.610 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.312382+0000", 2024-09-13T11:26:31.610 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.1429999999999997e-06 2024-09-13T11:26:31.610 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:31.610 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:31.610 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:26:31.610 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.312480+0000", 2024-09-13T11:26:31.610 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.7742e-05 2024-09-13T11:26:31.610 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:31.611 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:31.611 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:26:31.611 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.312521+0000", 2024-09-13T11:26:31.611 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.1369000000000003e-05 2024-09-13T11:26:31.611 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:31.611 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:31.611 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:26:31.611 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.315105+0000", 2024-09-13T11:26:31.611 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0025835060000000002 2024-09-13T11:26:31.611 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:31.611 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:31.611 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:26:31.611 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.315150+0000", 2024-09-13T11:26:31.612 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.5311e-05 2024-09-13T11:26:31.612 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:31.612 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:26:31.612 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:31.612 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:31.612 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:31.612 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:71927 3.0 e97/75 3:0197244a:::benchmark_data_smithi177_31116_object71926:head v 97'3351, mlcod=96'3348)", 2024-09-13T11:26:31.612 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:54.200161+0000", 2024-09-13T11:26:31.612 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 157.394036927, 2024-09-13T11:26:31.612 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.26691738999999998, 2024-09-13T11:26:31.612 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:26:31.612 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:26:31.612 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:31.612 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:26:31.613 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:31.613 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:26:31.613 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.200161+0000", 2024-09-13T11:26:31.613 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:31.613 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:31.613 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:31.613 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:26:31.613 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.200161+0000", 2024-09-13T11:26:31.613 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:31.613 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:31.613 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:31.613 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:26:31.613 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.200162+0000", 2024-09-13T11:26:31.614 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.78e-07 2024-09-13T11:26:31.614 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:31.614 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:31.614 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:26:31.614 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.200200+0000", 2024-09-13T11:26:31.614 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.8121000000000003e-05 2024-09-13T11:26:31.614 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:31.614 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:31.614 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:26:31.614 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.465397+0000", 2024-09-13T11:26:31.614 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.26519681099999998 2024-09-13T11:26:31.614 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:31.614 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:31.614 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:26:31.615 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.465404+0000", 2024-09-13T11:26:31.615 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.7399999999999998e-06 2024-09-13T11:26:31.615 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:31.615 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:31.615 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:26:31.615 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.465456+0000", 2024-09-13T11:26:31.615 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.2626000000000001e-05 2024-09-13T11:26:31.615 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:31.615 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:31.615 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:26:31.615 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.465482+0000", 2024-09-13T11:26:31.615 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.6086000000000001e-05 2024-09-13T11:26:31.615 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:31.615 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:31.616 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:26:31.616 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.467035+0000", 2024-09-13T11:26:31.616 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0015523729999999999 2024-09-13T11:26:31.616 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:31.616 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:31.616 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:26:31.616 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.467078+0000", 2024-09-13T11:26:31.616 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.3754999999999997e-05 2024-09-13T11:26:31.616 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:31.616 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:26:31.616 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:31.616 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:31.616 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:31.616 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPush(3.1b 102/78 [PushOp(3:df786fc7:::benchmark_data_smithi177_31116_object48840:head, version: 71'3079, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:df786fc7:::benchmark_data_smithi177_31116_object48840:head@71'3079, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:26:31.617 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:00.290536+0000", 2024-09-13T11:26:31.617 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 151.30366216499999, 2024-09-13T11:26:31.617 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.20017585399999999, 2024-09-13T11:26:31.617 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:26:31.617 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:26:31.617 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:26:31.617 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:26:31.617 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:31.617 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:26:31.617 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.290536+0000", 2024-09-13T11:26:31.617 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:31.617 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:31.617 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:31.617 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:26:31.618 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.290536+0000", 2024-09-13T11:26:31.618 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:31.618 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:31.618 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:31.618 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:26:31.618 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.290537+0000", 2024-09-13T11:26:31.618 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.327e-06 2024-09-13T11:26:31.618 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:31.618 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:31.618 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:26:31.618 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.290581+0000", 2024-09-13T11:26:31.618 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.3893999999999997e-05 2024-09-13T11:26:31.618 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:31.618 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:31.619 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:26:31.619 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.488588+0000", 2024-09-13T11:26:31.619 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.198006708 2024-09-13T11:26:31.619 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:31.619 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:31.619 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:26:31.619 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.488607+0000", 2024-09-13T11:26:31.619 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.9162999999999999e-05 2024-09-13T11:26:31.619 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:31.619 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:31.619 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:26:31.619 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.488747+0000", 2024-09-13T11:26:31.619 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00014040799999999999 2024-09-13T11:26:31.619 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:31.620 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:31.620 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:26:31.620 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.488770+0000", 2024-09-13T11:26:31.620 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.279e-05 2024-09-13T11:26:31.620 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:31.620 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:31.620 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:26:31.620 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.490712+0000", 2024-09-13T11:26:31.620 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001941564 2024-09-13T11:26:31.620 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:31.620 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:26:31.620 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:31.620 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:31.621 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:31.621 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPush(3.1b 102/78 [PushOp(3:d82bf6ec:::benchmark_data_smithi177_31116_object48847:head, version: 71'3080, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:d82bf6ec:::benchmark_data_smithi177_31116_object48847:head@71'3080, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:26:31.621 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:00.293979+0000", 2024-09-13T11:26:31.621 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 151.30021863799999, 2024-09-13T11:26:31.621 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.20314832099999999, 2024-09-13T11:26:31.621 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:26:31.621 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:26:31.621 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:26:31.621 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:26:31.621 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:31.621 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:26:31.621 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.293979+0000", 2024-09-13T11:26:31.621 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:31.621 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:31.622 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:31.622 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:26:31.622 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.293979+0000", 2024-09-13T11:26:31.622 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:31.622 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:31.622 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:31.622 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:26:31.622 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.293981+0000", 2024-09-13T11:26:31.622 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.4449999999999999e-06 2024-09-13T11:26:31.622 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:31.622 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:31.622 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:26:31.622 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.294085+0000", 2024-09-13T11:26:31.623 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000103848 2024-09-13T11:26:31.623 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:31.623 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:31.623 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:26:31.623 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.495571+0000", 2024-09-13T11:26:31.623 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.20148640800000001 2024-09-13T11:26:31.623 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:31.623 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:31.623 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:26:31.623 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.495609+0000", 2024-09-13T11:26:31.623 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.7904000000000002e-05 2024-09-13T11:26:31.623 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:31.623 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:31.623 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:26:31.624 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.495729+0000", 2024-09-13T11:26:31.624 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000120556 2024-09-13T11:26:31.624 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:31.624 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:31.624 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:26:31.624 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.495751+0000", 2024-09-13T11:26:31.624 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.1511000000000001e-05 2024-09-13T11:26:31.624 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:31.624 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:31.624 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:26:31.624 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.497128+0000", 2024-09-13T11:26:31.624 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001376649 2024-09-13T11:26:31.624 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:31.624 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:26:31.625 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:31.625 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:31.625 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:31.625 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:79801 3.5 3:a6efdf84:::benchmark_data_smithi177_31116_object79800:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102)", 2024-09-13T11:26:31.625 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:00.403201+0000", 2024-09-13T11:26:31.625 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 151.19099705100001, 2024-09-13T11:26:31.625 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.31868626300000003, 2024-09-13T11:26:31.625 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:26:31.625 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:26:31.625 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:31.625 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:26:31.625 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:26:31.625 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:31.626 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 79801 2024-09-13T11:26:31.626 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:31.626 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:26:31.626 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:31.626 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:26:31.626 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.403201+0000", 2024-09-13T11:26:31.626 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:31.626 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:31.626 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:31.626 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:26:31.626 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.403201+0000", 2024-09-13T11:26:31.626 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:31.626 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:31.626 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:31.627 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:26:31.627 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.403204+0000", 2024-09-13T11:26:31.627 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.0819999999999999e-06 2024-09-13T11:26:31.627 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:31.627 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:31.627 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:26:31.627 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.403260+0000", 2024-09-13T11:26:31.627 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.5624999999999997e-05 2024-09-13T11:26:31.627 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:31.627 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:31.627 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:26:31.627 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.403280+0000", 2024-09-13T11:26:31.627 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.0060000000000001e-05 2024-09-13T11:26:31.627 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:31.627 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:31.628 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:26:31.628 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.403291+0000", 2024-09-13T11:26:31.628 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.1335000000000001e-05 2024-09-13T11:26:31.628 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:31.628 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:31.628 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:26:31.628 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.403344+0000", 2024-09-13T11:26:31.628 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.3217000000000002e-05 2024-09-13T11:26:31.628 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:31.628 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:31.628 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:26:31.628 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.403456+0000", 2024-09-13T11:26:31.628 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000111706 2024-09-13T11:26:31.628 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:31.629 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:31.629 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 13", 2024-09-13T11:26:31.629 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.403547+0000", 2024-09-13T11:26:31.629 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.1556000000000002e-05 2024-09-13T11:26:31.629 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:31.629 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:31.629 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:26:31.629 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.407436+0000", 2024-09-13T11:26:31.629 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.003888856 2024-09-13T11:26:31.629 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:31.629 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:31.629 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:31.629 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.721722+0000", 2024-09-13T11:26:31.629 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.31428562700000001 2024-09-13T11:26:31.630 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:31.630 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:31.630 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:26:31.630 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.721814+0000", 2024-09-13T11:26:31.630 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.1564000000000006e-05 2024-09-13T11:26:31.630 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:31.630 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:31.630 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:26:31.630 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.721887+0000", 2024-09-13T11:26:31.630 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.3634999999999995e-05 2024-09-13T11:26:31.630 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:31.630 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:26:31.630 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:31.630 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:31.631 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:31.631 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:83199 3.1d e106/102 3:be2c6fae:::benchmark_data_smithi177_31116_object83198:head v 106'4244, mlcod=106'4242)", 2024-09-13T11:26:31.631 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:04.298299+0000", 2024-09-13T11:26:31.631 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 147.295898924, 2024-09-13T11:26:31.631 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.25001107099999997, 2024-09-13T11:26:31.631 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:26:31.631 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:26:31.631 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:31.631 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:26:31.631 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:31.631 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:26:31.631 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.298299+0000", 2024-09-13T11:26:31.631 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:31.631 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:31.631 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:31.632 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:26:31.632 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.298299+0000", 2024-09-13T11:26:31.632 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:31.632 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:31.632 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:31.632 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:26:31.632 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.298301+0000", 2024-09-13T11:26:31.632 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5859999999999999e-06 2024-09-13T11:26:31.632 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:31.632 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:31.632 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:26:31.632 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.298368+0000", 2024-09-13T11:26:31.632 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.6928999999999996e-05 2024-09-13T11:26:31.632 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:31.633 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:31.633 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:26:31.633 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.545180+0000", 2024-09-13T11:26:31.633 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.24681209700000001 2024-09-13T11:26:31.633 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:31.633 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:31.633 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:26:31.633 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.545193+0000", 2024-09-13T11:26:31.633 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3427e-05 2024-09-13T11:26:31.633 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:31.633 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:31.633 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:26:31.633 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.545327+0000", 2024-09-13T11:26:31.634 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00013359400000000001 2024-09-13T11:26:31.634 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:31.634 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:31.634 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:26:31.634 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.545385+0000", 2024-09-13T11:26:31.634 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.8492000000000002e-05 2024-09-13T11:26:31.634 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:31.634 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:31.634 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:26:31.634 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.548225+0000", 2024-09-13T11:26:31.634 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.002839472 2024-09-13T11:26:31.634 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:31.634 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:31.634 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:26:31.634 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.548310+0000", 2024-09-13T11:26:31.635 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.5474000000000002e-05 2024-09-13T11:26:31.635 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:31.635 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:26:31.635 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:31.635 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:31.635 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:31.635 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:107910 3.16 e116/88 3:6cf57aa3:::benchmark_data_smithi177_31116_object107909:head v 116'4911, mlcod=116'4908)", 2024-09-13T11:26:31.635 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:22.516225+0000", 2024-09-13T11:26:31.635 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 129.077973281, 2024-09-13T11:26:31.635 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.18319666600000001, 2024-09-13T11:26:31.635 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:26:31.635 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:26:31.635 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:31.636 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:26:31.636 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:31.636 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:26:31.636 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:22.516225+0000", 2024-09-13T11:26:31.636 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:31.636 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:31.636 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:31.636 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:26:31.636 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:22.516225+0000", 2024-09-13T11:26:31.636 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:31.636 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:31.636 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:31.636 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:26:31.636 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:22.516226+0000", 2024-09-13T11:26:31.637 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.716e-06 2024-09-13T11:26:31.637 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:31.637 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:31.637 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:26:31.637 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:22.516250+0000", 2024-09-13T11:26:31.637 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.3975e-05 2024-09-13T11:26:31.637 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:31.637 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:31.637 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:26:31.637 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:22.697686+0000", 2024-09-13T11:26:31.637 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.18143539 2024-09-13T11:26:31.637 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:31.637 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:31.637 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:26:31.638 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:22.697692+0000", 2024-09-13T11:26:31.638 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.0769999999999996e-06 2024-09-13T11:26:31.638 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:31.638 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:31.638 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:26:31.638 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:22.697744+0000", 2024-09-13T11:26:31.638 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.2525000000000003e-05 2024-09-13T11:26:31.638 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:31.638 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:31.638 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:26:31.638 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:22.697757+0000", 2024-09-13T11:26:31.638 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.2819e-05 2024-09-13T11:26:31.638 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:31.638 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:31.639 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:26:31.639 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:22.699388+0000", 2024-09-13T11:26:31.639 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0016303999999999999 2024-09-13T11:26:31.639 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:31.639 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:31.639 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:26:31.639 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:22.699421+0000", 2024-09-13T11:26:31.639 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.3763999999999997e-05 2024-09-13T11:26:31.639 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:31.639 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:26:31.639 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:31.639 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:31.639 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:31.639 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:127682 3.16 e131/88 3:6e6da02e:::benchmark_data_smithi177_31116_object127681:head v 131'5491, mlcod=131'5488)", 2024-09-13T11:26:31.640 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:44.556692+0000", 2024-09-13T11:26:31.640 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 107.03750635199999, 2024-09-13T11:26:31.640 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.18048624299999999, 2024-09-13T11:26:31.640 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:26:31.640 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:26:31.640 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:31.640 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:26:31.640 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:31.640 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:26:31.640 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:44.556692+0000", 2024-09-13T11:26:31.640 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:31.640 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:31.640 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:31.640 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:26:31.641 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:44.556692+0000", 2024-09-13T11:26:31.641 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:31.641 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:31.641 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:31.641 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:26:31.641 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:44.556693+0000", 2024-09-13T11:26:31.641 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.8109999999999999e-06 2024-09-13T11:26:31.641 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:31.641 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:31.641 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:26:31.641 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:44.556715+0000", 2024-09-13T11:26:31.641 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.1328999999999999e-05 2024-09-13T11:26:31.641 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:31.641 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:31.642 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:26:31.642 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:44.734908+0000", 2024-09-13T11:26:31.642 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.178192884 2024-09-13T11:26:31.642 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:31.642 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:31.642 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:26:31.642 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:44.734915+0000", 2024-09-13T11:26:31.642 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.9970000000000001e-06 2024-09-13T11:26:31.642 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:31.642 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:31.642 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:26:31.642 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:44.734971+0000", 2024-09-13T11:26:31.642 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.6368e-05 2024-09-13T11:26:31.642 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:31.643 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:31.643 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:26:31.643 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:44.734990+0000", 2024-09-13T11:26:31.643 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.929e-05 2024-09-13T11:26:31.643 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:31.643 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:31.643 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:26:31.643 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:44.737096+0000", 2024-09-13T11:26:31.643 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0021053090000000001 2024-09-13T11:26:31.643 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:31.643 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:31.643 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:26:31.643 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:44.737178+0000", 2024-09-13T11:26:31.644 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.2255000000000004e-05 2024-09-13T11:26:31.644 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:31.644 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:26:31.644 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:31.644 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:31.644 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:31.644 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:137062 3.5 3:a16e178f:::benchmark_data_smithi177_31116_object137061:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e139)", 2024-09-13T11:26:31.644 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:53.140525+0000", 2024-09-13T11:26:31.644 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 98.453673331000005, 2024-09-13T11:26:31.644 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.20697954599999999, 2024-09-13T11:26:31.644 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:26:31.644 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:26:31.644 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:31.644 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:26:31.645 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:26:31.645 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:31.645 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 137062 2024-09-13T11:26:31.645 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:31.645 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:26:31.645 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:31.645 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:26:31.645 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:53.140525+0000", 2024-09-13T11:26:31.645 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:31.645 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:31.645 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:31.645 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:26:31.645 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:53.140525+0000", 2024-09-13T11:26:31.645 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:31.646 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:31.646 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:31.646 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:26:31.646 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:53.140527+0000", 2024-09-13T11:26:31.646 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.1739999999999999e-06 2024-09-13T11:26:31.646 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:31.646 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:31.646 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:26:31.646 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:53.140553+0000", 2024-09-13T11:26:31.646 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.6631e-05 2024-09-13T11:26:31.646 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:31.646 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:31.646 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:26:31.646 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:53.140566+0000", 2024-09-13T11:26:31.647 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.2268e-05 2024-09-13T11:26:31.647 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:31.647 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:31.647 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:26:31.647 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:53.140574+0000", 2024-09-13T11:26:31.647 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.5669999999999995e-06 2024-09-13T11:26:31.647 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:31.647 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:31.647 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:26:31.647 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:53.140622+0000", 2024-09-13T11:26:31.647 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.8143999999999998e-05 2024-09-13T11:26:31.647 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:31.647 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:31.647 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:26:31.648 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:53.140759+0000", 2024-09-13T11:26:31.648 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000136499 2024-09-13T11:26:31.648 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:31.648 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:31.648 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 13", 2024-09-13T11:26:31.648 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:53.140887+0000", 2024-09-13T11:26:31.648 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00012796300000000001 2024-09-13T11:26:31.648 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:31.648 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:31.648 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:26:31.648 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:53.142675+0000", 2024-09-13T11:26:31.648 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001788073 2024-09-13T11:26:31.648 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:31.649 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:31.649 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:31.649 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:53.347407+0000", 2024-09-13T11:26:31.649 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.20473206899999999 2024-09-13T11:26:31.649 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:31.649 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:31.649 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:26:31.649 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:53.347464+0000", 2024-09-13T11:26:31.649 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.6536000000000002e-05 2024-09-13T11:26:31.649 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:31.649 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:31.649 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:26:31.649 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:53.347504+0000", 2024-09-13T11:26:31.650 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.0621999999999998e-05 2024-09-13T11:26:31.650 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:31.650 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:26:31.650 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:31.650 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:31.650 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:31.650 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:191058 3.1d e175/102 3:bd6e7029:::benchmark_data_smithi177_31116_object50755:head v 175'7612, mlcod=175'7609)", 2024-09-13T11:26:31.650 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:25:38.513527+0000", 2024-09-13T11:26:31.650 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 53.080670673999997, 2024-09-13T11:26:31.650 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.386104477, 2024-09-13T11:26:31.650 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:26:31.650 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:26:31.650 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:31.650 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:26:31.650 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:31.651 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:26:31.651 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:38.513527+0000", 2024-09-13T11:26:31.651 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:31.651 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:31.651 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:31.651 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:26:31.651 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:38.513527+0000", 2024-09-13T11:26:31.651 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:31.651 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:31.651 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:31.651 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:26:31.651 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:38.513530+0000", 2024-09-13T11:26:31.651 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.0029999999999999e-06 2024-09-13T11:26:31.651 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:31.652 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:31.652 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:26:31.652 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:38.513542+0000", 2024-09-13T11:26:31.652 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.1477e-05 2024-09-13T11:26:31.652 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:31.652 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:31.652 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:26:31.652 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:38.891617+0000", 2024-09-13T11:26:31.652 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.378075045 2024-09-13T11:26:31.652 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:31.652 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:31.652 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:26:31.652 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:38.891635+0000", 2024-09-13T11:26:31.653 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.7697e-05 2024-09-13T11:26:31.653 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:31.653 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:31.653 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:26:31.653 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:38.891775+0000", 2024-09-13T11:26:31.653 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000140665 2024-09-13T11:26:31.653 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:31.653 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:31.653 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:26:31.653 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:38.891833+0000", 2024-09-13T11:26:31.653 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.7769999999999997e-05 2024-09-13T11:26:31.653 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:31.653 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:31.653 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:26:31.653 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:38.899567+0000", 2024-09-13T11:26:31.654 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0077339970000000003 2024-09-13T11:26:31.654 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:31.654 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:31.654 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:26:31.654 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:38.899632+0000", 2024-09-13T11:26:31.654 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.4822999999999999e-05 2024-09-13T11:26:31.654 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:31.654 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:26:31.654 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:31.654 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:31.654 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:31.654 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:197236 3.5 3:a3127298:::benchmark_data_smithi177_31116_object56933:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e178)", 2024-09-13T11:26:31.654 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:25:44.300667+0000", 2024-09-13T11:26:31.655 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 47.293530582999999, 2024-09-13T11:26:31.655 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.37481205200000001, 2024-09-13T11:26:31.655 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:26:31.655 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:26:31.655 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:31.655 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:26:31.655 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:26:31.655 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:31.655 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 197236 2024-09-13T11:26:31.655 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:31.655 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:26:31.655 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:31.655 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:26:31.656 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:44.300667+0000", 2024-09-13T11:26:31.656 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:31.656 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:31.656 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:31.656 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:26:31.656 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:44.300667+0000", 2024-09-13T11:26:31.656 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:31.656 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:31.656 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:31.656 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:26:31.656 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:44.300670+0000", 2024-09-13T11:26:31.656 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.7259999999999998e-06 2024-09-13T11:26:31.656 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:31.656 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:31.657 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:26:31.657 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:44.300677+0000", 2024-09-13T11:26:31.657 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.1550000000000001e-06 2024-09-13T11:26:31.657 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:31.657 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:31.657 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:26:31.657 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:44.300695+0000", 2024-09-13T11:26:31.657 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.7552999999999999e-05 2024-09-13T11:26:31.657 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:31.657 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:31.657 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:26:31.657 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:44.300704+0000", 2024-09-13T11:26:31.657 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.5710000000000004e-06 2024-09-13T11:26:31.657 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:31.658 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:31.658 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:26:31.658 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:44.300762+0000", 2024-09-13T11:26:31.658 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.7351e-05 2024-09-13T11:26:31.658 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:31.658 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:31.658 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:26:31.658 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:44.300888+0000", 2024-09-13T11:26:31.658 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00012617999999999999 2024-09-13T11:26:31.658 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:31.658 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:31.658 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 13", 2024-09-13T11:26:31.658 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:44.300993+0000", 2024-09-13T11:26:31.659 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00010505600000000001 2024-09-13T11:26:31.659 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:31.659 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:31.659 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:26:31.659 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:44.302936+0000", 2024-09-13T11:26:31.659 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0019428749999999999 2024-09-13T11:26:31.659 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:31.659 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:31.659 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:31.659 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:44.675352+0000", 2024-09-13T11:26:31.659 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.37241603400000001 2024-09-13T11:26:31.659 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:31.659 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:31.659 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:26:31.659 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:44.675412+0000", 2024-09-13T11:26:31.660 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.9679999999999998e-05 2024-09-13T11:26:31.660 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:31.660 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:31.660 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:26:31.660 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:44.675479+0000", 2024-09-13T11:26:31.660 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.7871000000000001e-05 2024-09-13T11:26:31.660 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:31.660 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:26:31.660 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:31.660 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:31.660 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:31.660 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop_reply(client.4665.0:197236 3.5 e178/117 ondisk, result = 0)", 2024-09-13T11:26:31.660 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:25:44.304480+0000", 2024-09-13T11:26:31.661 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 47.289718315999998, 2024-09-13T11:26:31.661 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.37100864099999997, 2024-09-13T11:26:31.661 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:26:31.661 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:26:31.661 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:26:31.661 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:26:31.661 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:31.661 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:26:31.661 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:44.304480+0000", 2024-09-13T11:26:31.661 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:31.661 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:31.661 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:31.661 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:26:31.662 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:44.304480+0000", 2024-09-13T11:26:31.662 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:31.662 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:31.662 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:31.662 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:26:31.662 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:44.304481+0000", 2024-09-13T11:26:31.662 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.7349999999999999e-06 2024-09-13T11:26:31.662 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:31.662 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:31.662 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:26:31.662 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:44.304492+0000", 2024-09-13T11:26:31.662 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0394000000000001e-05 2024-09-13T11:26:31.662 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:31.662 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:31.662 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:26:31.663 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:44.675037+0000", 2024-09-13T11:26:31.663 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.37054519400000002 2024-09-13T11:26:31.663 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:31.663 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:31.663 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:26:31.663 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:44.675045+0000", 2024-09-13T11:26:31.663 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.1370000000000002e-06 2024-09-13T11:26:31.663 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:31.663 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:31.663 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:26:31.663 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:44.675334+0000", 2024-09-13T11:26:31.663 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00028848699999999998 2024-09-13T11:26:31.663 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:31.663 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:31.663 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:26:31.664 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:44.675345+0000", 2024-09-13T11:26:31.664 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.1198e-05 2024-09-13T11:26:31.664 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:31.664 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:31.664 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:26:31.664 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:44.675488+0000", 2024-09-13T11:26:31.664 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00014349600000000001 2024-09-13T11:26:31.664 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:31.664 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:26:31.664 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:31.664 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:31.664 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:31.664 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:217032 3.16 e187/88 3:6d28a27c:::benchmark_data_smithi177_31116_object76729:head v 187'8398, mlcod=187'8395)", 2024-09-13T11:26:31.664 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:25:59.836895+0000", 2024-09-13T11:26:31.665 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 31.757303431, 2024-09-13T11:26:31.665 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.22574986, 2024-09-13T11:26:31.665 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:26:31.665 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:26:31.665 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:31.665 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:26:31.665 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:31.665 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:26:31.665 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:59.836895+0000", 2024-09-13T11:26:31.665 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:31.665 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:31.665 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:31.665 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:26:31.665 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:59.836895+0000", 2024-09-13T11:26:31.666 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:31.666 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:31.666 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:31.666 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:26:31.666 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:59.836896+0000", 2024-09-13T11:26:31.666 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.432e-06 2024-09-13T11:26:31.666 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:31.666 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:31.666 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:26:31.666 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:59.836904+0000", 2024-09-13T11:26:31.666 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.9149999999999996e-06 2024-09-13T11:26:31.666 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:31.666 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:31.666 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:26:31.667 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:00.053571+0000", 2024-09-13T11:26:31.667 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.21666679799999999 2024-09-13T11:26:31.667 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:31.667 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:31.667 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:26:31.667 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:00.053582+0000", 2024-09-13T11:26:31.667 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0927999999999999e-05 2024-09-13T11:26:31.667 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:31.667 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:31.667 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:26:31.667 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:00.053811+0000", 2024-09-13T11:26:31.667 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00022964700000000001 2024-09-13T11:26:31.667 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:31.667 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:31.668 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:26:31.668 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:00.053864+0000", 2024-09-13T11:26:31.668 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.2710000000000002e-05 2024-09-13T11:26:31.668 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:31.668 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:31.668 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:26:31.668 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:00.062547+0000", 2024-09-13T11:26:31.668 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0086832650000000008 2024-09-13T11:26:31.668 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:31.668 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:31.668 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:26:31.668 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:00.062644+0000", 2024-09-13T11:26:31.669 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.7164999999999996e-05 2024-09-13T11:26:31.669 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:31.669 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:26:31.669 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:31.669 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:31.669 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:31.669 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:227461 3.5 3:a2578202:::benchmark_data_smithi177_31116_object87158:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e197)", 2024-09-13T11:26:31.669 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:26:11.215213+0000", 2024-09-13T11:26:31.669 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 20.378984891000002, 2024-09-13T11:26:31.669 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.18373250799999999, 2024-09-13T11:26:31.669 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:26:31.669 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:26:31.669 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:31.669 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:26:31.670 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:26:31.670 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:31.670 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 227461 2024-09-13T11:26:31.670 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:31.670 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:26:31.670 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:31.670 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:26:31.670 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:11.215213+0000", 2024-09-13T11:26:31.670 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:31.670 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:31.670 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:31.670 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:26:31.670 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:11.215213+0000", 2024-09-13T11:26:31.670 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:31.671 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:31.671 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:31.671 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:26:31.671 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:11.215216+0000", 2024-09-13T11:26:31.671 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.1370000000000002e-06 2024-09-13T11:26:31.671 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:31.671 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:31.671 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:26:31.671 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:11.215230+0000", 2024-09-13T11:26:31.671 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3976e-05 2024-09-13T11:26:31.671 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:31.671 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:31.671 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:26:31.671 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:11.215257+0000", 2024-09-13T11:26:31.672 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.6849999999999999e-05 2024-09-13T11:26:31.672 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:31.672 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:31.672 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:26:31.672 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:11.215271+0000", 2024-09-13T11:26:31.672 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3723000000000001e-05 2024-09-13T11:26:31.672 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:31.672 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:31.672 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:26:31.672 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:11.215401+0000", 2024-09-13T11:26:31.672 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00013043899999999999 2024-09-13T11:26:31.672 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:31.672 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:31.672 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:26:31.673 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:11.215710+0000", 2024-09-13T11:26:31.673 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00030838600000000001 2024-09-13T11:26:31.673 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:31.673 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:31.673 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 13", 2024-09-13T11:26:31.673 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:11.215863+0000", 2024-09-13T11:26:31.673 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00015319800000000001 2024-09-13T11:26:31.673 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:31.673 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:31.673 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:26:31.673 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:11.219895+0000", 2024-09-13T11:26:31.673 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0040320809999999999 2024-09-13T11:26:31.673 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:31.673 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:31.673 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:31.674 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:11.398827+0000", 2024-09-13T11:26:31.674 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.17893184000000001 2024-09-13T11:26:31.674 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:31.674 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:31.674 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:26:31.674 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:11.398895+0000", 2024-09-13T11:26:31.674 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.8046000000000005e-05 2024-09-13T11:26:31.674 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:31.674 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:31.674 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:26:31.674 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:11.398946+0000", 2024-09-13T11:26:31.674 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.0831999999999997e-05 2024-09-13T11:26:31.674 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:31.674 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:26:31.675 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:31.675 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:31.675 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:31.675 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:228538 3.5 3:a2ca23fe:::benchmark_data_smithi177_31116_object88235:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e198)", 2024-09-13T11:26:31.675 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:26:12.109379+0000", 2024-09-13T11:26:31.675 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 19.484819448, 2024-09-13T11:26:31.675 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.19925963699999999, 2024-09-13T11:26:31.675 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:26:31.675 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:26:31.675 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:31.675 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:26:31.675 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:26:31.675 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:31.675 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 228538 2024-09-13T11:26:31.676 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:31.676 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:26:31.676 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:31.676 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:26:31.676 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:12.109379+0000", 2024-09-13T11:26:31.676 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:31.676 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:31.676 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:31.676 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:26:31.676 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:12.109379+0000", 2024-09-13T11:26:31.676 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:31.676 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:31.676 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:31.676 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:26:31.677 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:12.109380+0000", 2024-09-13T11:26:31.677 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.9769999999999999e-06 2024-09-13T11:26:31.677 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:31.677 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:31.677 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:26:31.677 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:12.109387+0000", 2024-09-13T11:26:31.677 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.071e-06 2024-09-13T11:26:31.677 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:31.677 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:31.677 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:26:31.677 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:12.109401+0000", 2024-09-13T11:26:31.677 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.4487e-05 2024-09-13T11:26:31.677 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:31.677 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:31.678 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:26:31.678 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:12.109410+0000", 2024-09-13T11:26:31.678 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.4970000000000001e-06 2024-09-13T11:26:31.678 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:31.678 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:31.678 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:26:31.678 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:12.109467+0000", 2024-09-13T11:26:31.678 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.7080999999999998e-05 2024-09-13T11:26:31.678 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:31.678 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:31.678 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:26:31.678 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:12.109610+0000", 2024-09-13T11:26:31.678 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00014334699999999999 2024-09-13T11:26:31.678 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:31.679 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:31.679 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 13", 2024-09-13T11:26:31.679 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:12.109732+0000", 2024-09-13T11:26:31.679 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000122201 2024-09-13T11:26:31.679 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:31.679 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:31.679 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:26:31.679 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:12.113915+0000", 2024-09-13T11:26:31.679 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0041823820000000001 2024-09-13T11:26:31.679 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:31.679 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:31.679 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:31.679 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:12.308575+0000", 2024-09-13T11:26:31.679 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.194660687 2024-09-13T11:26:31.679 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:31.680 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:31.680 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:26:31.680 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:12.308609+0000", 2024-09-13T11:26:31.680 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.4103000000000002e-05 2024-09-13T11:26:31.680 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:31.680 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:31.680 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:26:31.680 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:12.308638+0000", 2024-09-13T11:26:31.680 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.8804000000000001e-05 2024-09-13T11:26:31.680 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:31.680 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:26:31.680 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:31.680 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:31.680 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:31.681 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:234789 3.5 3:a0df9e27:::benchmark_data_smithi177_31116_object94486:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e201)", 2024-09-13T11:26:31.681 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:26:17.919815+0000", 2024-09-13T11:26:31.681 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 13.674383285999999, 2024-09-13T11:26:31.681 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.34227359400000001, 2024-09-13T11:26:31.681 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:26:31.681 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:26:31.681 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:31.681 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:26:31.681 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:26:31.681 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:31.681 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 234789 2024-09-13T11:26:31.681 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:31.681 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:26:31.681 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:31.682 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:26:31.682 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:17.919815+0000", 2024-09-13T11:26:31.682 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:31.682 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:31.682 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:31.682 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:26:31.682 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:17.919815+0000", 2024-09-13T11:26:31.682 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:31.682 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:31.682 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:31.682 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:26:31.682 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:17.919817+0000", 2024-09-13T11:26:31.682 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.6369999999999999e-06 2024-09-13T11:26:31.682 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:31.682 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:31.683 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:26:31.683 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:17.919824+0000", 2024-09-13T11:26:31.683 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.9290000000000001e-06 2024-09-13T11:26:31.683 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:31.683 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:31.683 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:26:31.683 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:17.919839+0000", 2024-09-13T11:26:31.683 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.4763000000000001e-05 2024-09-13T11:26:31.683 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:31.683 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:31.683 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:26:31.683 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:17.919846+0000", 2024-09-13T11:26:31.683 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.1829999999999997e-06 2024-09-13T11:26:31.684 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:31.684 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:31.684 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:26:31.684 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:17.919894+0000", 2024-09-13T11:26:31.684 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.7475999999999997e-05 2024-09-13T11:26:31.684 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:31.684 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:31.684 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:26:31.684 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:17.919996+0000", 2024-09-13T11:26:31.684 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00010262199999999999 2024-09-13T11:26:31.684 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:31.684 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:31.684 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 13", 2024-09-13T11:26:31.684 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:17.920092+0000", 2024-09-13T11:26:31.685 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.5546999999999998e-05 2024-09-13T11:26:31.685 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:31.685 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:31.685 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:26:31.685 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:17.921865+0000", 2024-09-13T11:26:31.685 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00177271 2024-09-13T11:26:31.685 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:31.685 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:31.685 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:31.685 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:18.261907+0000", 2024-09-13T11:26:31.685 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.34004223300000003 2024-09-13T11:26:31.685 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:31.685 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:31.685 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:26:31.686 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:18.262002+0000", 2024-09-13T11:26:31.686 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.4745000000000002e-05 2024-09-13T11:26:31.686 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:31.686 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:31.686 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:26:31.686 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:18.262088+0000", 2024-09-13T11:26:31.686 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.6749000000000006e-05 2024-09-13T11:26:31.686 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:31.686 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:26:31.686 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:31.686 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:31.686 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:31.686 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop_reply(client.4665.0:234789 3.5 e201/117 ondisk, result = 0)", 2024-09-13T11:26:31.686 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:26:17.922332+0000", 2024-09-13T11:26:31.687 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 13.671865948000001, 2024-09-13T11:26:31.687 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.33978349299999999, 2024-09-13T11:26:31.687 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:26:31.687 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:26:31.687 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:26:31.687 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:26:31.687 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:31.687 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:26:31.687 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:17.922332+0000", 2024-09-13T11:26:31.687 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:31.687 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:31.687 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:31.687 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:26:31.687 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:17.922332+0000", 2024-09-13T11:26:31.688 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:31.688 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:31.688 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:31.688 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:26:31.688 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:17.922333+0000", 2024-09-13T11:26:31.688 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.2929999999999999e-06 2024-09-13T11:26:31.688 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:31.688 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:31.688 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:26:31.688 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:17.922337+0000", 2024-09-13T11:26:31.688 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.7469999999999999e-06 2024-09-13T11:26:31.688 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:31.688 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:31.688 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:26:31.689 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:18.261064+0000", 2024-09-13T11:26:31.689 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.33872685499999999 2024-09-13T11:26:31.689 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:31.689 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:31.689 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:26:31.689 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:18.261084+0000", 2024-09-13T11:26:31.689 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.9893000000000001e-05 2024-09-13T11:26:31.689 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:31.689 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:31.689 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:26:31.689 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:18.261880+0000", 2024-09-13T11:26:31.689 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00079579399999999995 2024-09-13T11:26:31.689 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:31.689 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:31.689 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:26:31.690 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:18.261897+0000", 2024-09-13T11:26:31.690 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.7730000000000001e-05 2024-09-13T11:26:31.690 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:31.690 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:31.690 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:26:31.690 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:18.262116+0000", 2024-09-13T11:26:31.690 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00021818099999999999 2024-09-13T11:26:31.690 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:31.690 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:26:31.690 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:31.690 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:31.690 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:26:31.690 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-09-13T11:26:31.692 INFO:tasks.ceph.osd.9.smithi092.stderr:2024-09-13T11:26:31.688+0000 7ff5ab3d0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26348) UID: 0 2024-09-13T11:26:31.693 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-13T11:26:31.793 INFO:tasks.ceph.osd.2.smithi162.stderr:2024-09-13T11:26:31.791+0000 7fe92b35a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26313) UID: 0 2024-09-13T11:26:31.830 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-09-13T11:26:31.830 INFO:teuthology.orchestra.run.smithi092.stdout: "ops": [ 2024-09-13T11:26:31.830 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:31.830 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop(client.4665.0:249867 3.e e210/75 3:74268d86:::benchmark_data_smithi177_31116_object109564:head v 210'9594, mlcod=210'9592)", 2024-09-13T11:26:31.830 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:26:31.829950+0000", 2024-09-13T11:26:31.831 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 0.00061061600000000002, 2024-09-13T11:26:31.831 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00062702500000000002, 2024-09-13T11:26:31.831 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:26:31.831 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:26:31.831 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-09-13T11:26:31.831 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:26:31.831 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:31.831 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:26:31.831 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:31.829950+0000", 2024-09-13T11:26:31.831 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:31.831 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:31.831 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:31.832 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:26:31.832 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:31.829950+0000", 2024-09-13T11:26:31.832 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:31.832 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:31.832 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:31.832 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:26:31.832 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:31.829951+0000", 2024-09-13T11:26:31.832 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.2780000000000001e-06 2024-09-13T11:26:31.832 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:31.832 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:31.832 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:26:31.832 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:31.829957+0000", 2024-09-13T11:26:31.832 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.7180000000000001e-06 2024-09-13T11:26:31.832 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:31.833 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:31.833 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:26:31.833 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:31.829979+0000", 2024-09-13T11:26:31.833 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.1758000000000001e-05 2024-09-13T11:26:31.833 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:31.833 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:31.833 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:26:31.833 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:31.829985+0000", 2024-09-13T11:26:31.833 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.2879999999999998e-06 2024-09-13T11:26:31.833 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:31.833 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:31.833 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:31.834 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:31.830033+0000", 2024-09-13T11:26:31.834 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.7905999999999998e-05 2024-09-13T11:26:31.834 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:31.834 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:31.834 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:26:31.834 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:31.830049+0000", 2024-09-13T11:26:31.834 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.5625999999999999e-05 2024-09-13T11:26:31.834 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:31.834 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:31.834 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:31.834 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:31.834 INFO:teuthology.orchestra.run.smithi092.stdout: ], 2024-09-13T11:26:31.834 INFO:teuthology.orchestra.run.smithi092.stdout: "num_ops": 1 2024-09-13T11:26:31.834 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-09-13T11:26:31.846 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-13T11:26:31.896 INFO:tasks.ceph.osd.5.smithi092.stderr:2024-09-13T11:26:31.896+0000 7f6f0b705640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26346) UID: 0 2024-09-13T11:26:31.981 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-09-13T11:26:31.981 INFO:teuthology.orchestra.run.smithi092.stdout: "ops": [], 2024-09-13T11:26:31.982 INFO:teuthology.orchestra.run.smithi092.stdout: "complaint_time": 30, 2024-09-13T11:26:31.982 INFO:teuthology.orchestra.run.smithi092.stdout: "num_blocked_ops": 0 2024-09-13T11:26:31.982 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-09-13T11:26:31.995 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-13T11:26:31.996 INFO:tasks.ceph.osd.8.smithi001.stderr:2024-09-13T11:26:31.990+0000 7fb9b48ae640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26312) UID: 0 2024-09-13T11:26:32.096 INFO:tasks.ceph.osd.7.smithi177.stderr:2024-09-13T11:26:32.093+0000 7f897b51d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26575) UID: 0 2024-09-13T11:26:32.179 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-09-13T11:26:32.179 INFO:teuthology.orchestra.run.smithi092.stdout: "size": 20, 2024-09-13T11:26:32.179 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 600, 2024-09-13T11:26:32.179 INFO:teuthology.orchestra.run.smithi092.stdout: "ops": [ 2024-09-13T11:26:32.180 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:32.180 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:3 3.4 3.125e09e4 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e40)", 2024-09-13T11:26:32.180 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:46.304641+0000", 2024-09-13T11:26:32.180 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 225.87370296399999, 2024-09-13T11:26:32.180 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.374372119, 2024-09-13T11:26:32.180 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:26:32.180 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:26:32.180 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "reached pg", 2024-09-13T11:26:32.180 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:26:32.180 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:26:32.180 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:32.180 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 3 2024-09-13T11:26:32.180 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:32.181 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:26:32.181 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:32.181 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:26:32.181 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.304641+0000", 2024-09-13T11:26:32.181 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:32.181 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:32.181 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:32.181 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:26:32.181 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.304641+0000", 2024-09-13T11:26:32.181 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:32.181 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:32.181 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:32.181 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:26:32.182 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.304644+0000", 2024-09-13T11:26:32.182 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.5130000000000001e-06 2024-09-13T11:26:32.182 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:32.182 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:32.182 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:26:32.182 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305058+0000", 2024-09-13T11:26:32.182 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000414277 2024-09-13T11:26:32.182 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:32.182 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:32.182 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:26:32.182 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305124+0000", 2024-09-13T11:26:32.182 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.5813000000000002e-05 2024-09-13T11:26:32.182 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:32.183 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:32.183 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:26:32.183 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305156+0000", 2024-09-13T11:26:32.183 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.1489000000000003e-05 2024-09-13T11:26:32.183 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:32.183 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:32.183 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:32.183 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.678939+0000", 2024-09-13T11:26:32.183 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.37378339999999999 2024-09-13T11:26:32.183 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:32.183 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:32.183 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:26:32.183 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.679013+0000", 2024-09-13T11:26:32.184 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.3627000000000005e-05 2024-09-13T11:26:32.184 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:32.184 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:32.184 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:32.184 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:32.184 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:32.184 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:29 3.4 3.b7bbfbe4 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-09-13T11:26:32.184 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:46.305215+0000", 2024-09-13T11:26:32.184 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 225.87312866100001, 2024-09-13T11:26:32.184 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.37387964600000001, 2024-09-13T11:26:32.184 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:26:32.184 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:26:32.185 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "reached pg", 2024-09-13T11:26:32.185 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:26:32.185 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:26:32.185 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:32.185 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 29 2024-09-13T11:26:32.185 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:32.185 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:26:32.185 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:32.185 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:26:32.185 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305215+0000", 2024-09-13T11:26:32.185 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:32.185 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:32.186 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:32.186 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:26:32.186 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305215+0000", 2024-09-13T11:26:32.186 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:32.186 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:32.186 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:32.186 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:26:32.186 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305218+0000", 2024-09-13T11:26:32.186 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.7030000000000002e-06 2024-09-13T11:26:32.186 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:32.186 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:32.186 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:26:32.186 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305366+0000", 2024-09-13T11:26:32.187 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00014825400000000001 2024-09-13T11:26:32.187 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:32.187 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:32.187 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:26:32.187 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305386+0000", 2024-09-13T11:26:32.187 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.9689000000000001e-05 2024-09-13T11:26:32.187 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:32.187 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:32.187 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:26:32.187 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305403+0000", 2024-09-13T11:26:32.187 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.7600000000000001e-05 2024-09-13T11:26:32.187 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:32.188 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:32.188 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:32.188 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.679067+0000", 2024-09-13T11:26:32.188 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.37366339100000001 2024-09-13T11:26:32.188 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:32.188 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:32.188 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:26:32.188 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.679095+0000", 2024-09-13T11:26:32.188 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.8008999999999999e-05 2024-09-13T11:26:32.188 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:32.188 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:32.188 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:32.189 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:32.189 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:32.189 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:30 3.4 3.547ec0f4 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-09-13T11:26:32.189 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:46.305453+0000", 2024-09-13T11:26:32.189 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 225.87289053699999, 2024-09-13T11:26:32.189 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.373764914, 2024-09-13T11:26:32.189 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:26:32.189 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:26:32.189 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "reached pg", 2024-09-13T11:26:32.189 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:26:32.189 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:26:32.189 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:32.190 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 30 2024-09-13T11:26:32.190 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:32.190 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:26:32.190 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:32.190 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:26:32.190 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305453+0000", 2024-09-13T11:26:32.190 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:32.190 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:32.190 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:32.190 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:26:32.190 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305453+0000", 2024-09-13T11:26:32.190 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:32.190 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:32.191 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:32.191 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:26:32.191 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305456+0000", 2024-09-13T11:26:32.191 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.9560000000000002e-06 2024-09-13T11:26:32.191 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:32.191 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:32.191 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:26:32.191 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305597+0000", 2024-09-13T11:26:32.191 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00014122700000000001 2024-09-13T11:26:32.191 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:32.191 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:32.192 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:26:32.192 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305615+0000", 2024-09-13T11:26:32.192 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.8139999999999999e-05 2024-09-13T11:26:32.192 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:32.192 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:32.192 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:26:32.192 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305632+0000", 2024-09-13T11:26:32.192 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.6101999999999999e-05 2024-09-13T11:26:32.192 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:32.192 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:32.192 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:32.192 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.679195+0000", 2024-09-13T11:26:32.193 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.373563486 2024-09-13T11:26:32.193 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:32.193 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:32.193 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:26:32.193 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.679218+0000", 2024-09-13T11:26:32.193 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.3003e-05 2024-09-13T11:26:32.193 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:32.193 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:32.193 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:32.193 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:32.193 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:32.193 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:70 3.4 3.1404a604 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-09-13T11:26:32.193 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:46.305665+0000", 2024-09-13T11:26:32.194 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 225.87267838899999, 2024-09-13T11:26:32.194 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.37365550199999997, 2024-09-13T11:26:32.194 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:26:32.194 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:26:32.194 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "reached pg", 2024-09-13T11:26:32.194 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:26:32.194 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:26:32.194 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:32.194 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 70 2024-09-13T11:26:32.194 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:32.194 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:26:32.194 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:32.194 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:26:32.195 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305665+0000", 2024-09-13T11:26:32.195 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:32.195 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:32.195 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:32.195 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:26:32.195 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305665+0000", 2024-09-13T11:26:32.195 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:32.195 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:32.195 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:32.195 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:26:32.195 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305668+0000", 2024-09-13T11:26:32.195 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.5890000000000001e-06 2024-09-13T11:26:32.196 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:32.196 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:32.196 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:26:32.196 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305791+0000", 2024-09-13T11:26:32.196 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000123545 2024-09-13T11:26:32.196 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:32.196 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:32.196 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:26:32.196 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305808+0000", 2024-09-13T11:26:32.196 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.668e-05 2024-09-13T11:26:32.196 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:32.196 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:32.196 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:26:32.197 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305824+0000", 2024-09-13T11:26:32.197 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.6385e-05 2024-09-13T11:26:32.197 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:32.197 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:32.197 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:32.197 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.679305+0000", 2024-09-13T11:26:32.197 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.37348059900000002 2024-09-13T11:26:32.197 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:32.197 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:32.197 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:26:32.197 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.679321+0000", 2024-09-13T11:26:32.197 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.5704000000000001e-05 2024-09-13T11:26:32.198 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:32.198 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:32.198 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:32.198 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:32.198 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:32.198 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:25408 3.4 3:294e23bd:::benchmark_data_smithi177_31116_object25407:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e56)", 2024-09-13T11:26:32.198 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:04.495455+0000", 2024-09-13T11:26:32.198 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 207.68288873700001, 2024-09-13T11:26:32.198 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.29897367600000002, 2024-09-13T11:26:32.198 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:26:32.198 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:26:32.198 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:32.198 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:26:32.198 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:26:32.199 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:32.199 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 25408 2024-09-13T11:26:32.199 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:32.199 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:26:32.199 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:32.199 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:26:32.199 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.495455+0000", 2024-09-13T11:26:32.199 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:32.199 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:32.199 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:32.199 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:26:32.199 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.495455+0000", 2024-09-13T11:26:32.200 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:32.200 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:32.200 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:32.200 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:26:32.200 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.495459+0000", 2024-09-13T11:26:32.200 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.5630000000000002e-06 2024-09-13T11:26:32.200 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:32.200 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:32.200 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:26:32.200 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.495545+0000", 2024-09-13T11:26:32.200 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.5704000000000005e-05 2024-09-13T11:26:32.200 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:32.200 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:32.201 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:26:32.201 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.495581+0000", 2024-09-13T11:26:32.201 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.5689000000000003e-05 2024-09-13T11:26:32.201 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:32.201 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:32.201 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:26:32.201 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.495597+0000", 2024-09-13T11:26:32.201 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.6053e-05 2024-09-13T11:26:32.201 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:32.201 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:32.201 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:32.202 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.495721+0000", 2024-09-13T11:26:32.202 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00012379099999999999 2024-09-13T11:26:32.202 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:32.202 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:32.202 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:26:32.202 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.495940+0000", 2024-09-13T11:26:32.202 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00021967699999999999 2024-09-13T11:26:32.202 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:32.202 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:32.202 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 12", 2024-09-13T11:26:32.202 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.496162+0000", 2024-09-13T11:26:32.202 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000221917 2024-09-13T11:26:32.202 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:32.202 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:32.203 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:26:32.203 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.498699+0000", 2024-09-13T11:26:32.203 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0025366690000000001 2024-09-13T11:26:32.203 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:32.203 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:32.203 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:32.203 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.794139+0000", 2024-09-13T11:26:32.203 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.295440025 2024-09-13T11:26:32.203 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:32.203 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:32.204 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:26:32.204 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.794306+0000", 2024-09-13T11:26:32.204 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00016689799999999999 2024-09-13T11:26:32.204 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:32.204 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:32.204 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:26:32.204 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.794429+0000", 2024-09-13T11:26:32.205 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00012269000000000001 2024-09-13T11:26:32.205 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:32.205 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:32.205 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:32.205 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:32.205 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:32.205 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "MOSDPGPushReply(3.14 75/70 [PushReplyOp(3:2f14928a:::benchmark_data_smithi177_31116_object15123:head)])", 2024-09-13T11:26:32.205 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:31.569662+0000", 2024-09-13T11:26:32.205 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 180.60868196499999, 2024-09-13T11:26:32.205 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.35759148299999999, 2024-09-13T11:26:32.205 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:26:32.205 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:26:32.205 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "reached pg", 2024-09-13T11:26:32.206 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:26:32.206 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:32.206 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:26:32.206 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:31.569662+0000", 2024-09-13T11:26:32.206 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:32.206 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:32.206 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:32.206 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:26:32.206 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:31.569662+0000", 2024-09-13T11:26:32.206 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:32.206 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:32.206 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:32.207 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:26:32.207 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:31.569663+0000", 2024-09-13T11:26:32.207 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.333e-06 2024-09-13T11:26:32.207 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:32.207 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:32.207 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:26:32.207 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:31.569667+0000", 2024-09-13T11:26:32.207 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.8929999999999998e-06 2024-09-13T11:26:32.207 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:32.207 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:32.207 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:26:32.207 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:31.923221+0000", 2024-09-13T11:26:32.208 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.35355413699999999 2024-09-13T11:26:32.208 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:32.208 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:32.208 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:26:32.208 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:31.923230+0000", 2024-09-13T11:26:32.208 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.6829999999999997e-06 2024-09-13T11:26:32.208 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:32.208 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:32.208 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:32.208 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:31.923807+0000", 2024-09-13T11:26:32.208 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00057686800000000002 2024-09-13T11:26:32.208 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:32.209 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:32.209 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:26:32.209 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:31.927253+0000", 2024-09-13T11:26:32.209 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.003446569 2024-09-13T11:26:32.209 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:32.209 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:32.209 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:32.209 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:32.209 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:32.209 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "MOSDPGPushReply(3.18 85/70 [PushReplyOp(3:1e410119:::benchmark_data_smithi177_31116_object35700:head)])", 2024-09-13T11:26:32.209 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:42.216614+0000", 2024-09-13T11:26:32.209 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 169.96172962099999, 2024-09-13T11:26:32.210 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.54260090699999997, 2024-09-13T11:26:32.210 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:26:32.210 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:26:32.210 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "reached pg", 2024-09-13T11:26:32.210 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:26:32.210 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:32.210 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:26:32.210 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.216614+0000", 2024-09-13T11:26:32.210 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:32.210 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:32.210 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:32.210 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:26:32.210 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.216614+0000", 2024-09-13T11:26:32.211 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:32.211 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:32.211 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:32.211 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:26:32.211 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.216615+0000", 2024-09-13T11:26:32.211 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.0200000000000001e-07 2024-09-13T11:26:32.211 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:32.211 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:32.211 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:26:32.211 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.216619+0000", 2024-09-13T11:26:32.211 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.805e-06 2024-09-13T11:26:32.211 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:32.212 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:32.212 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:26:32.212 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.757476+0000", 2024-09-13T11:26:32.212 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.54085762000000004 2024-09-13T11:26:32.212 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:32.212 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:32.212 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:26:32.212 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.757498+0000", 2024-09-13T11:26:32.212 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.1936000000000001e-05 2024-09-13T11:26:32.212 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:32.212 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:32.212 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:32.213 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.757678+0000", 2024-09-13T11:26:32.213 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00018006599999999999 2024-09-13T11:26:32.213 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:32.213 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:32.213 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:26:32.213 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.759215+0000", 2024-09-13T11:26:32.213 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0015366780000000001 2024-09-13T11:26:32.213 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:32.213 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:32.213 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:32.213 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:32.213 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:32.213 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "MOSDPGPushReply(3.1b 85/78 [PushReplyOp(3:dabda1cc:::benchmark_data_smithi177_31116_object12988:head)])", 2024-09-13T11:26:32.214 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:42.485171+0000", 2024-09-13T11:26:32.214 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 169.693172713, 2024-09-13T11:26:32.214 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.308420427, 2024-09-13T11:26:32.214 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:26:32.214 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:26:32.214 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "reached pg", 2024-09-13T11:26:32.214 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:26:32.214 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:32.214 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:26:32.214 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.485171+0000", 2024-09-13T11:26:32.214 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:32.214 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:32.214 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:32.215 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:26:32.215 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.485171+0000", 2024-09-13T11:26:32.215 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:32.215 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:32.215 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:32.215 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:26:32.215 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.485173+0000", 2024-09-13T11:26:32.215 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.1569999999999998e-06 2024-09-13T11:26:32.215 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:32.215 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:32.215 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:26:32.215 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.485182+0000", 2024-09-13T11:26:32.216 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.6759999999999996e-06 2024-09-13T11:26:32.216 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:32.216 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:32.216 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:26:32.216 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.791336+0000", 2024-09-13T11:26:32.216 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.30615446499999999 2024-09-13T11:26:32.216 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:32.216 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:32.216 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:26:32.216 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.791369+0000", 2024-09-13T11:26:32.216 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.3155999999999999e-05 2024-09-13T11:26:32.216 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:32.217 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:32.217 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:32.217 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.791472+0000", 2024-09-13T11:26:32.217 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00010247900000000001 2024-09-13T11:26:32.217 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:32.217 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:32.217 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:26:32.217 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.793591+0000", 2024-09-13T11:26:32.217 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.002119494 2024-09-13T11:26:32.217 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:32.217 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:32.217 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:32.217 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:32.218 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:32.218 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:64066 3.1b 3:dd84e0ec:::benchmark_data_smithi177_31116_object64065:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89)", 2024-09-13T11:26:32.218 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:46.633608+0000", 2024-09-13T11:26:32.218 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 165.544735274, 2024-09-13T11:26:32.218 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.32564315399999999, 2024-09-13T11:26:32.218 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:26:32.218 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:26:32.218 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:32.218 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:26:32.218 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:26:32.218 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:32.218 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 64066 2024-09-13T11:26:32.218 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:32.219 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:26:32.219 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:32.219 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:26:32.219 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.633608+0000", 2024-09-13T11:26:32.219 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:32.219 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:32.219 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:32.219 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:26:32.219 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.633608+0000", 2024-09-13T11:26:32.219 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:32.219 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:32.219 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:32.220 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:26:32.220 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.633612+0000", 2024-09-13T11:26:32.220 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.856e-06 2024-09-13T11:26:32.220 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:32.220 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:32.220 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:26:32.220 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.633683+0000", 2024-09-13T11:26:32.220 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.1000999999999993e-05 2024-09-13T11:26:32.220 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:32.220 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:32.220 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:26:32.220 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.633711+0000", 2024-09-13T11:26:32.220 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.8204999999999998e-05 2024-09-13T11:26:32.221 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:32.221 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:32.221 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:26:32.221 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.633732+0000", 2024-09-13T11:26:32.221 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.0662999999999999e-05 2024-09-13T11:26:32.221 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:32.221 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:32.221 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:32.221 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.635573+0000", 2024-09-13T11:26:32.221 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.001841365 2024-09-13T11:26:32.221 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:32.221 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:32.221 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:26:32.222 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.635995+0000", 2024-09-13T11:26:32.222 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00042113699999999998 2024-09-13T11:26:32.222 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:32.222 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:32.222 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 12,15", 2024-09-13T11:26:32.222 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.636475+0000", 2024-09-13T11:26:32.222 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00048067699999999999 2024-09-13T11:26:32.222 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:32.222 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:32.222 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:32.222 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.640247+0000", 2024-09-13T11:26:32.222 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0037716450000000001 2024-09-13T11:26:32.223 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:32.223 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:32.223 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:26:32.223 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.649308+0000", 2024-09-13T11:26:32.223 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0090613240000000008 2024-09-13T11:26:32.223 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:32.223 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:32.223 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:32.223 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.956912+0000", 2024-09-13T11:26:32.223 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.30760338500000001 2024-09-13T11:26:32.223 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:32.223 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:32.223 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:26:32.224 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.957156+0000", 2024-09-13T11:26:32.224 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000244031 2024-09-13T11:26:32.224 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:32.224 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:32.224 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:26:32.224 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.959252+0000", 2024-09-13T11:26:32.224 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0020958650000000001 2024-09-13T11:26:32.224 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:32.224 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:32.224 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:32.224 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:32.224 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:32.224 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop_reply(client.4665.0:64066 3.1b e89/78 ondisk, result = 0)", 2024-09-13T11:26:32.225 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:46.651596+0000", 2024-09-13T11:26:32.225 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 165.526747443, 2024-09-13T11:26:32.225 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.30767612599999999, 2024-09-13T11:26:32.225 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:26:32.225 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:26:32.225 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-09-13T11:26:32.225 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:26:32.225 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:32.225 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:26:32.225 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.651596+0000", 2024-09-13T11:26:32.225 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:32.225 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:32.225 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:32.225 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:26:32.226 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.651596+0000", 2024-09-13T11:26:32.226 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:32.226 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:32.226 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:32.226 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:26:32.226 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.651600+0000", 2024-09-13T11:26:32.226 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.6969999999999999e-06 2024-09-13T11:26:32.226 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:32.226 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:32.226 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:26:32.226 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.651611+0000", 2024-09-13T11:26:32.226 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.1516e-05 2024-09-13T11:26:32.226 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:32.226 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:32.227 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:26:32.227 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.948118+0000", 2024-09-13T11:26:32.227 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.29650663500000002 2024-09-13T11:26:32.227 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:32.227 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:32.227 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:26:32.227 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.948137+0000", 2024-09-13T11:26:32.227 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.8529000000000002e-05 2024-09-13T11:26:32.227 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:32.227 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:32.227 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:32.227 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.956870+0000", 2024-09-13T11:26:32.227 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0087336380000000002 2024-09-13T11:26:32.228 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:32.228 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:32.228 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:26:32.228 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.956893+0000", 2024-09-13T11:26:32.228 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.317e-05 2024-09-13T11:26:32.228 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:32.228 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:32.228 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:26:32.228 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.959272+0000", 2024-09-13T11:26:32.228 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0023789409999999999 2024-09-13T11:26:32.228 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:32.228 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:32.229 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:32.229 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:32.229 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:32.229 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "MOSDPGPushReply(3.1b 91/78 [PushReplyOp(3:d9ac6889:::benchmark_data_smithi177_31116_object22254:head)])", 2024-09-13T11:26:32.229 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:48.392729+0000", 2024-09-13T11:26:32.229 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 163.785614528, 2024-09-13T11:26:32.229 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.30924993099999998, 2024-09-13T11:26:32.229 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:26:32.229 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:26:32.229 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "reached pg", 2024-09-13T11:26:32.229 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:26:32.229 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:32.230 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:26:32.230 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:48.392729+0000", 2024-09-13T11:26:32.230 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:32.230 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:32.230 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:32.230 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:26:32.230 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:48.392729+0000", 2024-09-13T11:26:32.230 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:32.230 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:32.230 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:32.230 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:26:32.230 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:48.392730+0000", 2024-09-13T11:26:32.231 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.336e-06 2024-09-13T11:26:32.231 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:32.231 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:32.231 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:26:32.231 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:48.392739+0000", 2024-09-13T11:26:32.231 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.8580000000000006e-06 2024-09-13T11:26:32.231 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:32.231 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:32.231 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:26:32.231 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:48.700180+0000", 2024-09-13T11:26:32.231 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.30744035600000003 2024-09-13T11:26:32.231 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:32.232 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:32.232 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:26:32.232 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:48.700197+0000", 2024-09-13T11:26:32.232 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.7649999999999999e-05 2024-09-13T11:26:32.232 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:32.232 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:32.232 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:32.232 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:48.700398+0000", 2024-09-13T11:26:32.232 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000201113 2024-09-13T11:26:32.232 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:32.232 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:32.232 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:26:32.233 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:48.701979+0000", 2024-09-13T11:26:32.233 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.001580618 2024-09-13T11:26:32.233 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:32.233 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:32.233 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:32.233 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:32.233 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:32.233 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop(client.4665.0:83193 3.8 e106/67 3:10066d83:::benchmark_data_smithi177_31116_object83192:head v 106'3925, mlcod=106'3922)", 2024-09-13T11:26:32.233 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:24:04.285068+0000", 2024-09-13T11:26:32.233 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 147.893275773, 2024-09-13T11:26:32.233 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.64647716600000005, 2024-09-13T11:26:32.233 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:26:32.234 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:26:32.234 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:32.234 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:26:32.234 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:32.234 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:26:32.234 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:04.285068+0000", 2024-09-13T11:26:32.234 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:32.234 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:32.234 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:32.234 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:26:32.234 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:04.285068+0000", 2024-09-13T11:26:32.234 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:32.234 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:32.235 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:32.235 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:26:32.235 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:04.285071+0000", 2024-09-13T11:26:32.235 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.729e-06 2024-09-13T11:26:32.235 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:32.235 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:32.235 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:26:32.235 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:04.285215+0000", 2024-09-13T11:26:32.235 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00014410000000000001 2024-09-13T11:26:32.235 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:32.235 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:32.235 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:26:32.235 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:04.928221+0000", 2024-09-13T11:26:32.235 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.64300653299999999 2024-09-13T11:26:32.236 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:32.236 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:32.236 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:26:32.236 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:04.928240+0000", 2024-09-13T11:26:32.236 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.8678999999999999e-05 2024-09-13T11:26:32.236 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:32.236 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:32.236 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:32.236 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:04.928388+0000", 2024-09-13T11:26:32.236 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00014809 2024-09-13T11:26:32.236 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:32.236 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:32.236 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:26:32.236 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:04.928428+0000", 2024-09-13T11:26:32.237 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.9776e-05 2024-09-13T11:26:32.237 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:32.237 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:32.237 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:26:32.237 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:04.931463+0000", 2024-09-13T11:26:32.237 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.003035168 2024-09-13T11:26:32.237 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:32.237 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:32.237 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:26:32.237 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:04.931545+0000", 2024-09-13T11:26:32.237 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.2090999999999997e-05 2024-09-13T11:26:32.237 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:32.238 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:32.238 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:32.238 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:32.238 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:32.238 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop(client.4665.0:83239 3.e e106/75 3:74547f17:::benchmark_data_smithi177_31116_object83238:head v 106'4391, mlcod=106'4388)", 2024-09-13T11:26:32.238 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:24:04.581196+0000", 2024-09-13T11:26:32.238 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 147.597147267, 2024-09-13T11:26:32.238 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.33617203600000001, 2024-09-13T11:26:32.238 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:26:32.238 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:26:32.238 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:32.238 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:26:32.239 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:32.239 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:26:32.239 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:04.581196+0000", 2024-09-13T11:26:32.239 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:32.239 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:32.239 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:32.239 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:26:32.239 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:04.581196+0000", 2024-09-13T11:26:32.239 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:32.239 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:32.239 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:32.239 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:26:32.239 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:04.581200+0000", 2024-09-13T11:26:32.239 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.19e-06 2024-09-13T11:26:32.240 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:32.240 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:32.240 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:26:32.240 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:04.581257+0000", 2024-09-13T11:26:32.240 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.7531999999999997e-05 2024-09-13T11:26:32.240 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:32.240 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:32.240 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:26:32.240 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:04.914011+0000", 2024-09-13T11:26:32.240 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.33275413700000001 2024-09-13T11:26:32.240 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:32.240 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:32.241 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:26:32.241 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:04.914028+0000", 2024-09-13T11:26:32.241 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.7082999999999999e-05 2024-09-13T11:26:32.241 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:32.241 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:32.241 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:32.241 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:04.914151+0000", 2024-09-13T11:26:32.241 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00012283300000000001 2024-09-13T11:26:32.241 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:32.241 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:32.241 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:26:32.241 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:04.914185+0000", 2024-09-13T11:26:32.241 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.4255999999999999e-05 2024-09-13T11:26:32.242 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:32.242 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:32.242 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:26:32.242 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:04.917300+0000", 2024-09-13T11:26:32.242 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0031141599999999999 2024-09-13T11:26:32.242 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:32.242 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:32.242 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:26:32.242 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:04.917368+0000", 2024-09-13T11:26:32.242 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.8844999999999996e-05 2024-09-13T11:26:32.242 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:32.243 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:32.243 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:32.243 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:32.243 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:32.243 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:131687 3.28 3:14f65078:::benchmark_data_smithi177_31116_object131686:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e135)", 2024-09-13T11:26:32.243 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:24:48.925565+0000", 2024-09-13T11:26:32.243 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 103.25277826999999, 2024-09-13T11:26:32.243 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.41421604099999998, 2024-09-13T11:26:32.243 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:26:32.243 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:26:32.243 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:32.243 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:26:32.243 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:26:32.244 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:32.244 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 131687 2024-09-13T11:26:32.244 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:32.244 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:26:32.244 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:32.244 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:26:32.244 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:48.925565+0000", 2024-09-13T11:26:32.244 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:32.244 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:32.244 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:32.244 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:26:32.245 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:48.925565+0000", 2024-09-13T11:26:32.245 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:32.245 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:32.245 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:32.245 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:26:32.245 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:48.925568+0000", 2024-09-13T11:26:32.245 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.441e-06 2024-09-13T11:26:32.245 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:32.245 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:32.245 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:26:32.245 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:48.925617+0000", 2024-09-13T11:26:32.245 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.8739000000000002e-05 2024-09-13T11:26:32.246 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:32.246 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:32.246 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:26:32.246 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:48.925631+0000", 2024-09-13T11:26:32.246 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.4463e-05 2024-09-13T11:26:32.246 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:32.246 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:32.246 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:26:32.246 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:48.925638+0000", 2024-09-13T11:26:32.246 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.2119999999999997e-06 2024-09-13T11:26:32.246 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:32.246 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:32.246 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:32.247 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:48.925703+0000", 2024-09-13T11:26:32.247 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.4794000000000003e-05 2024-09-13T11:26:32.247 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:32.247 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:32.247 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:26:32.247 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:48.925857+0000", 2024-09-13T11:26:32.247 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00015427 2024-09-13T11:26:32.247 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:32.247 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:32.247 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 5,8", 2024-09-13T11:26:32.247 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:48.925995+0000", 2024-09-13T11:26:32.247 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000137365 2024-09-13T11:26:32.247 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:32.248 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:32.248 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:26:32.248 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:48.927261+0000", 2024-09-13T11:26:32.248 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.001266021 2024-09-13T11:26:32.248 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:32.248 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:32.248 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:32.248 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:48.930015+0000", 2024-09-13T11:26:32.248 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0027545130000000001 2024-09-13T11:26:32.248 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:32.248 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:32.248 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:32.249 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:49.338898+0000", 2024-09-13T11:26:32.249 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.40888254299999999 2024-09-13T11:26:32.249 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:32.249 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:32.249 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:26:32.249 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:49.338967+0000", 2024-09-13T11:26:32.249 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.9402999999999999e-05 2024-09-13T11:26:32.249 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:32.249 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:32.249 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:26:32.249 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:49.339781+0000", 2024-09-13T11:26:32.249 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00081427699999999997 2024-09-13T11:26:32.249 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:32.249 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:32.250 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:32.250 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:32.250 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:32.250 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop_reply(client.4665.0:131687 3.28 e135/120 ondisk, result = 0)", 2024-09-13T11:26:32.250 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:24:48.928935+0000", 2024-09-13T11:26:32.250 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 103.249408969, 2024-09-13T11:26:32.250 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.41085638499999999, 2024-09-13T11:26:32.250 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:26:32.250 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:26:32.250 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-09-13T11:26:32.250 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:26:32.250 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:32.250 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:26:32.251 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:48.928935+0000", 2024-09-13T11:26:32.251 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:32.251 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:32.251 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:32.251 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:26:32.251 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:48.928935+0000", 2024-09-13T11:26:32.251 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:32.251 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:32.251 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:32.251 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:26:32.251 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:48.928936+0000", 2024-09-13T11:26:32.251 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.519e-06 2024-09-13T11:26:32.251 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:32.252 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:32.252 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:26:32.252 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:48.928941+0000", 2024-09-13T11:26:32.252 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.0710000000000002e-06 2024-09-13T11:26:32.252 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:32.252 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:32.252 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:26:32.252 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:49.338799+0000", 2024-09-13T11:26:32.252 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.40985808499999998 2024-09-13T11:26:32.252 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:32.252 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:32.253 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:26:32.253 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:49.338809+0000", 2024-09-13T11:26:32.253 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.4390000000000006e-06 2024-09-13T11:26:32.253 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:32.253 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:32.253 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:32.253 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:49.338878+0000", 2024-09-13T11:26:32.253 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.9429000000000002e-05 2024-09-13T11:26:32.254 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:32.254 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:32.254 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:26:32.254 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:49.338890+0000", 2024-09-13T11:26:32.254 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.2217e-05 2024-09-13T11:26:32.254 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:32.254 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:32.254 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:26:32.254 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:49.339791+0000", 2024-09-13T11:26:32.254 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00090062499999999995 2024-09-13T11:26:32.254 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:32.254 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:32.254 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:32.255 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:32.255 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:32.255 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "MOSDPGPushReply(3.28 163/120 [PushReplyOp(3:169fe9fd:::benchmark_data_smithi177_31116_object59744:head)])", 2024-09-13T11:26:32.255 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:25:22.263786+0000", 2024-09-13T11:26:32.255 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 69.914557701000007, 2024-09-13T11:26:32.255 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.28927712799999999, 2024-09-13T11:26:32.255 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:26:32.255 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:26:32.255 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "reached pg", 2024-09-13T11:26:32.255 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:26:32.255 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:32.255 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:26:32.255 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:22.263786+0000", 2024-09-13T11:26:32.255 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:32.256 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:32.256 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:32.256 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:26:32.256 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:22.263786+0000", 2024-09-13T11:26:32.256 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:32.256 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:32.256 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:32.256 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:26:32.256 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:22.263788+0000", 2024-09-13T11:26:32.256 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.542e-06 2024-09-13T11:26:32.256 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:32.256 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:32.256 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:26:32.257 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:22.263797+0000", 2024-09-13T11:26:32.257 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.234e-06 2024-09-13T11:26:32.257 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:32.257 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:32.257 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:26:32.257 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:22.552310+0000", 2024-09-13T11:26:32.257 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.28851281499999998 2024-09-13T11:26:32.257 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:32.257 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:32.257 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:26:32.257 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:22.552315+0000", 2024-09-13T11:26:32.257 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.5990000000000001e-06 2024-09-13T11:26:32.258 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:32.258 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:32.258 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:32.258 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:22.552362+0000", 2024-09-13T11:26:32.258 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.7144000000000001e-05 2024-09-13T11:26:32.258 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:32.258 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:32.258 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:26:32.258 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:22.553063+0000", 2024-09-13T11:26:32.258 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00070079400000000003 2024-09-13T11:26:32.258 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:32.259 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:32.259 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:32.259 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:32.259 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:32.259 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:181661 3.4 3:2019f47a:::benchmark_data_smithi177_31116_object41358:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e170)", 2024-09-13T11:26:32.259 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:25:31.131054+0000", 2024-09-13T11:26:32.259 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 61.047289202999998, 2024-09-13T11:26:32.259 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.44761152399999998, 2024-09-13T11:26:32.259 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:26:32.259 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:26:32.259 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:32.259 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:26:32.259 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:26:32.260 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:32.260 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 181661 2024-09-13T11:26:32.260 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:32.260 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:26:32.260 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:32.260 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:26:32.260 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:31.131054+0000", 2024-09-13T11:26:32.260 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:32.260 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:32.260 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:32.260 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:26:32.260 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:31.131054+0000", 2024-09-13T11:26:32.260 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:32.261 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:32.261 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:32.261 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:26:32.261 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:31.131056+0000", 2024-09-13T11:26:32.261 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.8640000000000001e-06 2024-09-13T11:26:32.261 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:32.261 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:32.261 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:26:32.261 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:31.131061+0000", 2024-09-13T11:26:32.261 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.6249999999999998e-06 2024-09-13T11:26:32.261 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:32.261 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:32.262 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:26:32.262 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:31.131069+0000", 2024-09-13T11:26:32.262 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.1049999999999995e-06 2024-09-13T11:26:32.262 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:32.262 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:32.262 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:26:32.262 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:31.131074+0000", 2024-09-13T11:26:32.262 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.344e-06 2024-09-13T11:26:32.262 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:32.262 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:32.262 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:32.262 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:31.131121+0000", 2024-09-13T11:26:32.262 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.6794e-05 2024-09-13T11:26:32.263 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:32.263 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:32.263 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:26:32.263 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:31.131225+0000", 2024-09-13T11:26:32.263 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000103502 2024-09-13T11:26:32.263 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:32.263 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:32.263 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 12", 2024-09-13T11:26:32.263 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:31.131289+0000", 2024-09-13T11:26:32.263 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.4790999999999997e-05 2024-09-13T11:26:32.263 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:32.263 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:32.263 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:26:32.264 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:31.133191+0000", 2024-09-13T11:26:32.264 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0019013859999999999 2024-09-13T11:26:32.264 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:32.264 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:32.264 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:32.264 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:31.578585+0000", 2024-09-13T11:26:32.264 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.44539403 2024-09-13T11:26:32.264 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:32.264 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:32.264 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:26:32.264 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:31.578620+0000", 2024-09-13T11:26:32.264 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.4657999999999999e-05 2024-09-13T11:26:32.265 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:32.265 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:32.265 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:26:32.265 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:31.578666+0000", 2024-09-13T11:26:32.265 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.6424999999999997e-05 2024-09-13T11:26:32.265 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:32.265 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:32.265 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:32.265 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:32.265 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:32.265 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop_reply(client.4665.0:181661 3.4 e170/117 ondisk, result = 0)", 2024-09-13T11:26:32.265 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:25:31.135385+0000", 2024-09-13T11:26:32.265 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 61.042958382999998, 2024-09-13T11:26:32.266 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.44328512599999997, 2024-09-13T11:26:32.266 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:26:32.266 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:26:32.266 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-09-13T11:26:32.266 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:26:32.266 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:32.266 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:26:32.266 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:31.135385+0000", 2024-09-13T11:26:32.266 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:32.266 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:32.266 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:32.266 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:26:32.266 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:31.135385+0000", 2024-09-13T11:26:32.267 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:32.267 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:32.267 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:32.267 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:26:32.267 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:31.135387+0000", 2024-09-13T11:26:32.267 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.217e-06 2024-09-13T11:26:32.267 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:32.267 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:32.267 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:26:32.267 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:31.135391+0000", 2024-09-13T11:26:32.267 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.0820000000000001e-06 2024-09-13T11:26:32.267 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:32.267 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:32.268 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:26:32.268 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:31.578504+0000", 2024-09-13T11:26:32.268 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.44311382100000002 2024-09-13T11:26:32.268 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:32.268 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:32.268 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:26:32.268 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:31.578530+0000", 2024-09-13T11:26:32.268 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.5367e-05 2024-09-13T11:26:32.268 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:32.268 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:32.268 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:32.268 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:31.578576+0000", 2024-09-13T11:26:32.268 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.5840999999999997e-05 2024-09-13T11:26:32.268 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:32.269 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:32.269 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:26:32.269 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:31.578581+0000", 2024-09-13T11:26:32.269 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.5620000000000003e-06 2024-09-13T11:26:32.269 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:32.269 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:32.269 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:26:32.269 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:31.578670+0000", 2024-09-13T11:26:32.269 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.9235999999999997e-05 2024-09-13T11:26:32.269 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:32.269 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:32.269 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:32.269 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:32.270 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:32.270 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop(client.4665.0:184026 3.b e171/117 3:d3052347:::benchmark_data_smithi177_31116_object43723:head v 171'6273, mlcod=171'6269)", 2024-09-13T11:26:32.270 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:25:32.645695+0000", 2024-09-13T11:26:32.270 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 59.532648649000002, 2024-09-13T11:26:32.270 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.34822244200000002, 2024-09-13T11:26:32.270 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:26:32.270 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:26:32.270 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:32.270 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:26:32.270 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:32.270 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:26:32.270 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:32.645695+0000", 2024-09-13T11:26:32.270 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:32.271 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:32.271 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:32.271 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:26:32.271 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:32.645695+0000", 2024-09-13T11:26:32.271 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:32.271 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:32.271 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:32.271 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:26:32.271 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:32.645696+0000", 2024-09-13T11:26:32.271 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.4780000000000001e-06 2024-09-13T11:26:32.271 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:32.271 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:32.271 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:26:32.271 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:32.645704+0000", 2024-09-13T11:26:32.272 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.1980000000000004e-06 2024-09-13T11:26:32.272 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:32.272 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:32.272 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:26:32.272 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:32.992228+0000", 2024-09-13T11:26:32.272 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.346523992 2024-09-13T11:26:32.272 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:32.272 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:32.272 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:26:32.272 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:32.992232+0000", 2024-09-13T11:26:32.272 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.3109999999999999e-06 2024-09-13T11:26:32.272 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:32.272 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:32.273 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:32.273 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:32.992275+0000", 2024-09-13T11:26:32.273 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.2589999999999997e-05 2024-09-13T11:26:32.273 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:32.273 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:32.273 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:26:32.273 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:32.992289+0000", 2024-09-13T11:26:32.273 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.4116e-05 2024-09-13T11:26:32.273 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:32.273 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:32.273 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:26:32.273 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:32.993877+0000", 2024-09-13T11:26:32.273 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0015879259999999999 2024-09-13T11:26:32.274 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:32.274 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:32.274 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:26:32.274 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:32.993917+0000", 2024-09-13T11:26:32.274 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.0831000000000003e-05 2024-09-13T11:26:32.274 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:32.274 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:32.274 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:32.274 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:32.274 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:32.274 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop(client.4665.0:184072 3.b e171/117 3:d319cf93:::benchmark_data_smithi177_31116_object43769:head v 171'6274, mlcod=171'6270)", 2024-09-13T11:26:32.274 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:25:32.701516+0000", 2024-09-13T11:26:32.275 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 59.476827960999998, 2024-09-13T11:26:32.275 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.30463955399999998, 2024-09-13T11:26:32.275 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:26:32.275 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:26:32.275 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:32.275 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:26:32.275 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:32.275 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:26:32.275 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:32.701516+0000", 2024-09-13T11:26:32.275 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:32.275 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:32.275 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:32.275 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:26:32.276 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:32.701516+0000", 2024-09-13T11:26:32.276 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:32.276 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:32.276 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:32.276 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:26:32.276 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:32.701519+0000", 2024-09-13T11:26:32.276 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.0029999999999999e-06 2024-09-13T11:26:32.276 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:32.276 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:32.276 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:26:32.276 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:32.701530+0000", 2024-09-13T11:26:32.276 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.1759999999999999e-05 2024-09-13T11:26:32.276 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:32.277 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:32.277 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:26:32.277 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:33.003551+0000", 2024-09-13T11:26:32.277 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.302020819 2024-09-13T11:26:32.277 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:32.277 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:32.277 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:26:32.277 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:33.003558+0000", 2024-09-13T11:26:32.277 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.2299999999999996e-06 2024-09-13T11:26:32.277 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:32.277 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:32.277 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:32.278 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:33.003630+0000", 2024-09-13T11:26:32.278 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.2125000000000005e-05 2024-09-13T11:26:32.278 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:32.278 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:32.278 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:26:32.278 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:33.003648+0000", 2024-09-13T11:26:32.278 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.8672e-05 2024-09-13T11:26:32.278 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:32.278 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:32.278 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:26:32.278 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:33.006114+0000", 2024-09-13T11:26:32.278 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0024656790000000001 2024-09-13T11:26:32.278 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:32.279 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:32.279 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:26:32.279 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:33.006155+0000", 2024-09-13T11:26:32.279 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.1266000000000001e-05 2024-09-13T11:26:32.279 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:32.279 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:32.279 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:32.279 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:32.279 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:32.279 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-09-13T11:26:32.280 INFO:tasks.ceph.osd.11.smithi177.stderr:2024-09-13T11:26:32.221+0000 7f2469cec640 -1 osd.11 210 handle_osd_map injecting map crc failure 2024-09-13T11:26:32.281 INFO:teuthology.orchestra.run.smithi001.stderr:noscrub is unset 2024-09-13T11:26:32.284 INFO:tasks.ceph.osd.12.smithi001.stderr:2024-09-13T11:26:32.214+0000 7f891d10b640 -1 osd.12 210 handle_osd_map injecting map crc failure 2024-09-13T11:26:32.286 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_ops_in_flight 2024-09-13T11:26:32.286 INFO:tasks.ceph.osd.1.smithi092.stderr:2024-09-13T11:26:32.236+0000 7fe76f3f6640 -1 osd.1 210 handle_osd_map injecting map crc failure 2024-09-13T11:26:32.286 INFO:tasks.ceph.osd.5.smithi092.stderr:2024-09-13T11:26:32.280+0000 7f6f0b705640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26346) UID: 0 2024-09-13T11:26:32.381 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-13T11:26:32.378+0000 7f6a68775640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26307) UID: 0 2024-09-13T11:26:32.425 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-13T11:26:32.425 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [], 2024-09-13T11:26:32.425 INFO:teuthology.orchestra.run.smithi162.stdout: "num_ops": 0 2024-09-13T11:26:32.425 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-13T11:26:32.442 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_blocked_ops 2024-09-13T11:26:32.482 INFO:tasks.ceph.osd.13.smithi092.stderr:2024-09-13T11:26:32.480+0000 7f3e5afea640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26350) UID: 0 2024-09-13T11:26:32.574 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-13T11:26:32.575 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [], 2024-09-13T11:26:32.575 INFO:teuthology.orchestra.run.smithi162.stdout: "complaint_time": 30, 2024-09-13T11:26:32.575 INFO:teuthology.orchestra.run.smithi162.stdout: "num_blocked_ops": 0 2024-09-13T11:26:32.575 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-13T11:26:32.583 INFO:tasks.ceph.osd.5.smithi092.stderr:2024-09-13T11:26:32.580+0000 7f6f0b705640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26346) UID: 0 2024-09-13T11:26:32.589 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_historic_ops 2024-09-13T11:26:32.685 INFO:tasks.ceph.osd.8.smithi001.stderr:2024-09-13T11:26:32.682+0000 7fb9b48ae640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26312) UID: 0 2024-09-13T11:26:32.722 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-13T11:26:32.722 INFO:teuthology.orchestra.run.smithi162.stdout: "size": 20, 2024-09-13T11:26:32.722 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 600, 2024-09-13T11:26:32.722 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [ 2024-09-13T11:26:32.722 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:32.722 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:221526 3.2 3.ea98b382 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e191)", 2024-09-13T11:26:32.722 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:26:04.295546+0000", 2024-09-13T11:26:32.722 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 28.425457284, 2024-09-13T11:26:32.722 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.73989113200000001, 2024-09-13T11:26:32.722 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:26:32.723 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:26:32.723 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-09-13T11:26:32.723 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:26:32.723 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:26:32.723 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:32.723 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 221526 2024-09-13T11:26:32.723 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:32.723 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:26:32.723 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:32.723 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:26:32.723 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:04.295546+0000", 2024-09-13T11:26:32.723 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:32.723 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:32.723 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:32.724 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:26:32.724 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:04.295546+0000", 2024-09-13T11:26:32.724 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:32.724 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:32.724 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:32.724 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:26:32.724 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:04.295551+0000", 2024-09-13T11:26:32.724 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.6940000000000003e-06 2024-09-13T11:26:32.724 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:32.724 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:32.724 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:26:32.724 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:04.295567+0000", 2024-09-13T11:26:32.725 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5896000000000002e-05 2024-09-13T11:26:32.725 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:32.725 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:32.725 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:26:32.725 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:04.295599+0000", 2024-09-13T11:26:32.725 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.243e-05 2024-09-13T11:26:32.725 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:32.725 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:32.725 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:26:32.725 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:04.295622+0000", 2024-09-13T11:26:32.725 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.3149000000000001e-05 2024-09-13T11:26:32.725 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:32.725 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:32.726 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:26:32.726 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:04.295764+0000", 2024-09-13T11:26:32.726 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00014193300000000001 2024-09-13T11:26:32.726 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:32.726 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:32.726 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-09-13T11:26:32.726 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:04.295778+0000", 2024-09-13T11:26:32.726 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3763e-05 2024-09-13T11:26:32.726 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:32.726 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:32.726 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:26:32.726 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:05.035420+0000", 2024-09-13T11:26:32.726 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.73964228300000001 2024-09-13T11:26:32.726 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:32.727 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:32.727 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:26:32.727 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:05.035437+0000", 2024-09-13T11:26:32.727 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.6983999999999999e-05 2024-09-13T11:26:32.727 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:32.727 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:26:32.727 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:32.727 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:32.727 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:32.727 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:221528 3.15 3.b7a45bf5 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e191)", 2024-09-13T11:26:32.727 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:26:04.296174+0000", 2024-09-13T11:26:32.727 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 28.424829638999999, 2024-09-13T11:26:32.727 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.73909410399999997, 2024-09-13T11:26:32.728 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:26:32.728 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:26:32.728 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-09-13T11:26:32.728 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:26:32.728 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:26:32.728 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:32.728 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 221528 2024-09-13T11:26:32.728 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:32.728 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:26:32.728 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:32.728 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:26:32.728 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:04.296174+0000", 2024-09-13T11:26:32.728 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:32.728 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:32.729 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:32.729 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:26:32.729 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:04.296174+0000", 2024-09-13T11:26:32.729 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:32.729 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:32.729 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:32.729 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:26:32.729 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:04.296188+0000", 2024-09-13T11:26:32.729 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4037e-05 2024-09-13T11:26:32.729 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:32.729 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:32.729 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:26:32.729 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:04.296201+0000", 2024-09-13T11:26:32.729 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3096e-05 2024-09-13T11:26:32.730 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:32.730 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:32.730 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:26:32.730 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:04.296242+0000", 2024-09-13T11:26:32.730 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.1103000000000003e-05 2024-09-13T11:26:32.730 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:32.730 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:32.730 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:26:32.730 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:04.296264+0000", 2024-09-13T11:26:32.730 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.173e-05 2024-09-13T11:26:32.730 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:32.730 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:32.730 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:26:32.730 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:04.296410+0000", 2024-09-13T11:26:32.731 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00014574600000000001 2024-09-13T11:26:32.731 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:32.731 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:32.731 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-09-13T11:26:32.731 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:04.296424+0000", 2024-09-13T11:26:32.731 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3926e-05 2024-09-13T11:26:32.731 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:32.731 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:32.731 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:26:32.731 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:05.035221+0000", 2024-09-13T11:26:32.731 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.73879784999999998 2024-09-13T11:26:32.731 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:32.731 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:32.731 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:26:32.731 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:05.035268+0000", 2024-09-13T11:26:32.732 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.6616000000000002e-05 2024-09-13T11:26:32.732 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:32.732 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:26:32.732 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:32.732 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:32.732 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:32.732 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:221544 3.12 3.ec3fbdf2 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e191)", 2024-09-13T11:26:32.732 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:26:04.384452+0000", 2024-09-13T11:26:32.732 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 28.336551047, 2024-09-13T11:26:32.732 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.65065541999999998, 2024-09-13T11:26:32.732 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:26:32.732 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:26:32.732 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-09-13T11:26:32.732 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:26:32.733 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:26:32.733 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:32.733 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 221544 2024-09-13T11:26:32.733 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:32.733 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:26:32.733 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:32.733 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:26:32.733 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:04.384452+0000", 2024-09-13T11:26:32.733 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:32.733 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:32.733 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:32.733 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:26:32.733 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:04.384452+0000", 2024-09-13T11:26:32.733 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:32.733 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:32.734 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:32.734 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:26:32.734 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:04.384457+0000", 2024-09-13T11:26:32.734 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.6469999999999997e-06 2024-09-13T11:26:32.734 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:32.734 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:32.734 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:26:32.734 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:04.384470+0000", 2024-09-13T11:26:32.734 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3186999999999999e-05 2024-09-13T11:26:32.734 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:32.734 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:32.734 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:26:32.734 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:04.384504+0000", 2024-09-13T11:26:32.734 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.3413e-05 2024-09-13T11:26:32.735 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:32.735 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:32.735 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:26:32.735 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:04.384526+0000", 2024-09-13T11:26:32.735 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.2690000000000001e-05 2024-09-13T11:26:32.735 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:32.735 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:32.735 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:26:32.735 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:04.384653+0000", 2024-09-13T11:26:32.735 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00012697199999999999 2024-09-13T11:26:32.735 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:32.735 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:32.735 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-09-13T11:26:32.735 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:04.384668+0000", 2024-09-13T11:26:32.735 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4936000000000001e-05 2024-09-13T11:26:32.736 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:32.736 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:32.736 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:26:32.736 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:05.035087+0000", 2024-09-13T11:26:32.736 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.65041902900000004 2024-09-13T11:26:32.736 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:32.736 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:32.736 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:26:32.736 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:05.035108+0000", 2024-09-13T11:26:32.736 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.0545999999999999e-05 2024-09-13T11:26:32.736 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:32.736 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:26:32.736 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:32.737 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:32.737 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:32.737 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPush(3.22 193/191 [PushOp(3:446c4776:::benchmark_data_smithi177_31116_object85951:head, version: 109'3584, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:446c4776:::benchmark_data_smithi177_31116_object85951:head@109'3584, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:26:32.737 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:26:06.067726+0000", 2024-09-13T11:26:32.737 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 26.653277348, 2024-09-13T11:26:32.737 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.123514185, 2024-09-13T11:26:32.737 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:26:32.737 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:26:32.737 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:26:32.737 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:26:32.737 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:32.737 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:26:32.737 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:06.067726+0000", 2024-09-13T11:26:32.737 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:32.738 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:32.738 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:32.738 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:26:32.738 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:06.067726+0000", 2024-09-13T11:26:32.738 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:32.738 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:32.738 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:32.738 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:26:32.738 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:06.067728+0000", 2024-09-13T11:26:32.738 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.962e-06 2024-09-13T11:26:32.738 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:32.738 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:32.738 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:26:32.738 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:06.067774+0000", 2024-09-13T11:26:32.739 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.5691e-05 2024-09-13T11:26:32.739 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:32.739 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:32.739 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:26:32.739 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:06.091244+0000", 2024-09-13T11:26:32.739 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.023470560000000001 2024-09-13T11:26:32.739 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:32.739 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:32.739 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:26:32.739 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:06.091276+0000", 2024-09-13T11:26:32.739 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.1587000000000001e-05 2024-09-13T11:26:32.739 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:32.739 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:32.739 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:26:32.739 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:06.189504+0000", 2024-09-13T11:26:32.740 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.098228111000000007 2024-09-13T11:26:32.740 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:32.740 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:32.740 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:26:32.740 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:06.189515+0000", 2024-09-13T11:26:32.740 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1063000000000001e-05 2024-09-13T11:26:32.740 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:32.740 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:32.740 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:26:32.740 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:06.191240+0000", 2024-09-13T11:26:32.740 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0017252109999999999 2024-09-13T11:26:32.740 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:32.740 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:26:32.740 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:32.741 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:32.741 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:32.741 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPush(3.22 193/191 [PushOp(3:442d7aa4:::benchmark_data_smithi177_31116_object86008:head, version: 109'3585, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:442d7aa4:::benchmark_data_smithi177_31116_object86008:head@109'3585, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:26:32.741 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:26:06.088007+0000", 2024-09-13T11:26:32.741 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 26.632996122000002, 2024-09-13T11:26:32.741 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.135647079, 2024-09-13T11:26:32.741 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:26:32.741 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:26:32.741 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:26:32.741 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:26:32.741 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:32.741 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:26:32.741 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:06.088007+0000", 2024-09-13T11:26:32.741 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:32.742 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:32.742 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:32.742 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:26:32.742 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:06.088007+0000", 2024-09-13T11:26:32.742 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:32.742 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:32.742 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:32.742 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:26:32.742 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:06.088009+0000", 2024-09-13T11:26:32.742 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.525e-06 2024-09-13T11:26:32.742 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:32.742 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:32.742 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:26:32.742 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:06.088071+0000", 2024-09-13T11:26:32.743 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.2539999999999994e-05 2024-09-13T11:26:32.743 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:32.743 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:32.743 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:26:32.743 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:06.221398+0000", 2024-09-13T11:26:32.743 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.133326469 2024-09-13T11:26:32.743 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:32.743 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:32.743 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:26:32.743 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:06.221441+0000", 2024-09-13T11:26:32.743 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.2611999999999998e-05 2024-09-13T11:26:32.743 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:32.743 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:32.744 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:26:32.744 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:06.221571+0000", 2024-09-13T11:26:32.744 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00013056999999999999 2024-09-13T11:26:32.744 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:32.744 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:32.744 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:26:32.744 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:06.221581+0000", 2024-09-13T11:26:32.744 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0263e-05 2024-09-13T11:26:32.744 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:32.744 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:32.744 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:26:32.744 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:06.223654+0000", 2024-09-13T11:26:32.744 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0020731 2024-09-13T11:26:32.745 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:32.745 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:26:32.745 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:32.745 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:32.745 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:32.745 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPush(3.24 198/191 [PushOp(3:259c1bd3:::benchmark_data_smithi177_31116_object114151:head, version: 125'4839, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:259c1bd3:::benchmark_data_smithi177_31116_object114151:head@125'4839, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:26:32.745 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:26:12.041809+0000", 2024-09-13T11:26:32.745 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 20.679194398, 2024-09-13T11:26:32.745 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.195123457, 2024-09-13T11:26:32.745 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:26:32.745 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:26:32.745 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:26:32.745 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:26:32.745 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:32.746 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:26:32.746 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:12.041809+0000", 2024-09-13T11:26:32.746 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:32.746 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:32.746 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:32.746 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:26:32.746 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:12.041809+0000", 2024-09-13T11:26:32.746 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:32.746 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:32.746 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:32.746 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:26:32.746 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:12.041811+0000", 2024-09-13T11:26:32.746 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.1469999999999999e-06 2024-09-13T11:26:32.746 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:32.746 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:32.747 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:26:32.747 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:12.041875+0000", 2024-09-13T11:26:32.747 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.3785e-05 2024-09-13T11:26:32.747 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:32.747 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:32.747 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:26:32.747 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:12.235597+0000", 2024-09-13T11:26:32.747 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.193721853 2024-09-13T11:26:32.747 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:32.747 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:32.747 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:26:32.747 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:12.235610+0000", 2024-09-13T11:26:32.747 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.31e-05 2024-09-13T11:26:32.748 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:32.748 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:32.748 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:26:32.748 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:12.235733+0000", 2024-09-13T11:26:32.748 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00012339600000000001 2024-09-13T11:26:32.748 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:32.748 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:32.748 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:26:32.748 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:12.235744+0000", 2024-09-13T11:26:32.748 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0665999999999999e-05 2024-09-13T11:26:32.748 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:32.748 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:32.748 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:26:32.749 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:12.236933+0000", 2024-09-13T11:26:32.749 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00118851 2024-09-13T11:26:32.749 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:32.749 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:26:32.749 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:32.749 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:32.749 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:32.749 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.4665.0:233710 3.1f e200/192 3:fe3927a3:::benchmark_data_smithi177_31116_object93407:head v 200'8853, mlcod=200'8850)", 2024-09-13T11:26:32.749 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:26:16.659700+0000", 2024-09-13T11:26:32.749 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 16.061303403, 2024-09-13T11:26:32.749 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.129674331, 2024-09-13T11:26:32.749 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:26:32.749 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:26:32.749 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:32.750 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:26:32.750 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:32.750 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:26:32.750 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:16.659700+0000", 2024-09-13T11:26:32.750 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:32.750 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:32.750 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:32.750 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:26:32.750 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:16.659700+0000", 2024-09-13T11:26:32.750 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:32.750 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:32.750 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:32.751 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:26:32.751 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:16.659703+0000", 2024-09-13T11:26:32.751 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.6469999999999998e-06 2024-09-13T11:26:32.751 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:32.751 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:32.751 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:26:32.751 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:16.659715+0000", 2024-09-13T11:26:32.751 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2012e-05 2024-09-13T11:26:32.751 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:32.751 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:32.751 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:26:32.751 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:16.784281+0000", 2024-09-13T11:26:32.751 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.12456668 2024-09-13T11:26:32.751 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:32.752 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:32.752 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:26:32.752 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:16.784296+0000", 2024-09-13T11:26:32.752 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4063000000000001e-05 2024-09-13T11:26:32.752 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:32.752 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:32.752 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:26:32.752 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:16.784444+0000", 2024-09-13T11:26:32.752 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000148622 2024-09-13T11:26:32.752 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:32.752 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:32.752 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:26:32.752 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:16.784509+0000", 2024-09-13T11:26:32.752 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.5097000000000004e-05 2024-09-13T11:26:32.753 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:32.753 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:32.753 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:26:32.753 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:16.789279+0000", 2024-09-13T11:26:32.753 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0047700040000000004 2024-09-13T11:26:32.753 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:32.753 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:32.753 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:26:32.753 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:16.789374+0000", 2024-09-13T11:26:32.753 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.5205999999999996e-05 2024-09-13T11:26:32.753 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:32.753 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:26:32.753 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:32.753 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:32.754 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:32.754 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.4665.0:233786 3.1f e200/192 3:fbb85971:::benchmark_data_smithi177_31116_object93483:head v 200'8854, mlcod=200'8851)", 2024-09-13T11:26:32.754 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:26:16.784220+0000", 2024-09-13T11:26:32.754 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 15.936783869999999, 2024-09-13T11:26:32.754 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.138243172, 2024-09-13T11:26:32.754 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:26:32.754 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:26:32.754 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:32.754 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:26:32.754 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:32.754 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:26:32.754 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:16.784220+0000", 2024-09-13T11:26:32.754 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:32.754 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:32.755 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:32.755 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:26:32.755 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:16.784220+0000", 2024-09-13T11:26:32.755 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:32.755 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:32.755 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:32.755 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:26:32.755 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:16.784224+0000", 2024-09-13T11:26:32.755 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.1860000000000002e-06 2024-09-13T11:26:32.755 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:32.755 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:32.755 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:26:32.755 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:16.784240+0000", 2024-09-13T11:26:32.755 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5707000000000001e-05 2024-09-13T11:26:32.756 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:32.756 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:32.756 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:26:32.756 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:16.917820+0000", 2024-09-13T11:26:32.756 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.13358021 2024-09-13T11:26:32.756 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:32.756 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:32.756 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:26:32.756 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:16.917838+0000", 2024-09-13T11:26:32.756 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.7853999999999999e-05 2024-09-13T11:26:32.756 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:32.756 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:32.756 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:26:32.756 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:16.917982+0000", 2024-09-13T11:26:32.757 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00014408499999999999 2024-09-13T11:26:32.757 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:32.757 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:32.757 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:26:32.757 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:16.918024+0000", 2024-09-13T11:26:32.757 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.2345999999999997e-05 2024-09-13T11:26:32.757 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:32.757 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:32.757 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:26:32.757 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:16.922345+0000", 2024-09-13T11:26:32.757 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0043210480000000001 2024-09-13T11:26:32.757 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:32.757 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:32.757 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:26:32.758 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:16.922463+0000", 2024-09-13T11:26:32.758 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000117736 2024-09-13T11:26:32.758 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:32.758 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:26:32.758 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:32.758 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:32.758 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:32.758 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGRecoveryDelete(3.6 e200,191 3:619d604e:::benchmark_data_smithi177_31116_object7745:head,148'5485)", 2024-09-13T11:26:32.758 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:26:16.997534+0000", 2024-09-13T11:26:32.758 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 15.72346997, 2024-09-13T11:26:32.758 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.13337686300000001, 2024-09-13T11:26:32.758 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:26:32.758 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:26:32.758 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:26:32.758 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:26:32.759 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:32.759 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:26:32.759 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:16.997534+0000", 2024-09-13T11:26:32.759 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:32.759 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:32.759 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:32.759 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:26:32.759 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:16.997534+0000", 2024-09-13T11:26:32.759 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:32.759 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:32.759 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:32.759 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:26:32.759 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:16.997535+0000", 2024-09-13T11:26:32.759 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.7370000000000001e-06 2024-09-13T11:26:32.760 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:32.760 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:32.760 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:26:32.760 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:16.997542+0000", 2024-09-13T11:26:32.760 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.742e-06 2024-09-13T11:26:32.760 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:32.760 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:32.760 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:26:32.760 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:17.130708+0000", 2024-09-13T11:26:32.760 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.13316600300000001 2024-09-13T11:26:32.760 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:32.760 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:32.760 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:26:32.760 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:17.130717+0000", 2024-09-13T11:26:32.761 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.3589999999999997e-06 2024-09-13T11:26:32.761 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:32.761 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:32.761 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:26:32.761 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:17.130802+0000", 2024-09-13T11:26:32.761 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.4122999999999994e-05 2024-09-13T11:26:32.761 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:32.761 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:32.761 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:26:32.761 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:17.130813+0000", 2024-09-13T11:26:32.761 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1131999999999999e-05 2024-09-13T11:26:32.761 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:32.761 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:32.761 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:26:32.762 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:17.130910+0000", 2024-09-13T11:26:32.762 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.7767000000000001e-05 2024-09-13T11:26:32.762 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:32.762 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:26:32.762 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:32.762 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:32.762 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:32.762 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:234164 3.22 3.d472cf62 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e201)", 2024-09-13T11:26:32.762 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:26:17.191424+0000", 2024-09-13T11:26:32.762 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 15.529579928, 2024-09-13T11:26:32.762 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.95096563000000001, 2024-09-13T11:26:32.762 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:26:32.762 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:26:32.763 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-09-13T11:26:32.763 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:26:32.763 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:26:32.763 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:32.763 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 234164 2024-09-13T11:26:32.763 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:32.763 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:26:32.763 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:32.763 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:26:32.763 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:17.191424+0000", 2024-09-13T11:26:32.763 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:32.763 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:32.763 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:32.763 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:26:32.763 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:17.191424+0000", 2024-09-13T11:26:32.764 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:32.764 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:32.764 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:32.764 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:26:32.764 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:17.191426+0000", 2024-09-13T11:26:32.764 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.3269999999999999e-06 2024-09-13T11:26:32.765 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:32.765 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:32.765 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:26:32.765 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:17.191434+0000", 2024-09-13T11:26:32.765 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.5490000000000006e-06 2024-09-13T11:26:32.765 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:32.765 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:32.765 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:26:32.765 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:17.191450+0000", 2024-09-13T11:26:32.765 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.596e-05 2024-09-13T11:26:32.765 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:32.765 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:32.765 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:26:32.766 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:17.191458+0000", 2024-09-13T11:26:32.766 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.6070000000000002e-06 2024-09-13T11:26:32.766 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:32.766 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:32.766 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:26:32.766 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:17.191518+0000", 2024-09-13T11:26:32.766 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.9488999999999999e-05 2024-09-13T11:26:32.766 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:32.766 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:32.766 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-09-13T11:26:32.766 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:17.191526+0000", 2024-09-13T11:26:32.766 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.8049999999999996e-06 2024-09-13T11:26:32.766 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:32.766 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:32.767 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:26:32.767 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:18.142341+0000", 2024-09-13T11:26:32.767 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.95081438299999999 2024-09-13T11:26:32.767 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:32.767 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:32.767 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:26:32.767 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:18.142389+0000", 2024-09-13T11:26:32.767 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.8510000000000001e-05 2024-09-13T11:26:32.767 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:32.767 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:26:32.767 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:32.768 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:32.768 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:32.768 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:234214 3.22 3.3ebf73a2 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e201)", 2024-09-13T11:26:32.768 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:26:17.239598+0000", 2024-09-13T11:26:32.768 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 15.481405657, 2024-09-13T11:26:32.768 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.90310166300000005, 2024-09-13T11:26:32.769 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:26:32.769 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:26:32.769 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-09-13T11:26:32.769 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:26:32.769 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:26:32.769 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:32.769 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 234214 2024-09-13T11:26:32.769 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:32.769 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:26:32.769 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:32.769 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:26:32.770 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:17.239598+0000", 2024-09-13T11:26:32.770 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:32.770 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:32.770 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:32.770 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:26:32.770 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:17.239598+0000", 2024-09-13T11:26:32.770 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:32.770 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:32.770 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:32.770 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:26:32.770 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:17.239601+0000", 2024-09-13T11:26:32.771 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.021e-06 2024-09-13T11:26:32.771 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:32.771 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:32.771 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:26:32.771 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:17.239609+0000", 2024-09-13T11:26:32.771 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.9459999999999998e-06 2024-09-13T11:26:32.771 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:32.771 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:32.771 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:26:32.771 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:17.239628+0000", 2024-09-13T11:26:32.771 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.9029e-05 2024-09-13T11:26:32.771 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:32.771 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:32.772 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:26:32.772 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:17.239637+0000", 2024-09-13T11:26:32.772 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.7530000000000007e-06 2024-09-13T11:26:32.772 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:32.772 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:32.772 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:26:32.772 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:17.239716+0000", 2024-09-13T11:26:32.772 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.9789000000000005e-05 2024-09-13T11:26:32.772 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:32.772 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:32.772 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-09-13T11:26:32.773 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:17.239729+0000", 2024-09-13T11:26:32.773 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2215000000000001e-05 2024-09-13T11:26:32.773 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:32.773 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:32.773 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:26:32.773 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:18.142672+0000", 2024-09-13T11:26:32.773 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.90294309299999997 2024-09-13T11:26:32.773 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:32.773 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:32.773 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:26:32.773 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:18.142700+0000", 2024-09-13T11:26:32.773 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.7817000000000002e-05 2024-09-13T11:26:32.774 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:32.774 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:26:32.774 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:32.774 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:32.774 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:32.774 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:234350 3.22 3.6c9f6f62 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e201)", 2024-09-13T11:26:32.774 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:26:17.412568+0000", 2024-09-13T11:26:32.774 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 15.308435857999999, 2024-09-13T11:26:32.774 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.73023408300000003, 2024-09-13T11:26:32.774 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:26:32.774 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:26:32.774 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-09-13T11:26:32.775 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:26:32.775 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:26:32.775 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:32.775 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 234350 2024-09-13T11:26:32.775 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:32.775 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:26:32.775 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:32.775 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:26:32.775 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:17.412568+0000", 2024-09-13T11:26:32.775 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:32.775 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:32.775 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:32.776 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:26:32.776 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:17.412568+0000", 2024-09-13T11:26:32.776 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:32.776 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:32.776 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:32.776 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:26:32.776 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:17.412571+0000", 2024-09-13T11:26:32.776 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.044e-06 2024-09-13T11:26:32.776 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:32.776 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:32.776 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:26:32.777 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:17.412579+0000", 2024-09-13T11:26:32.777 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.7639999999999994e-06 2024-09-13T11:26:32.777 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:32.777 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:32.777 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:26:32.777 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:17.412595+0000", 2024-09-13T11:26:32.777 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5637999999999998e-05 2024-09-13T11:26:32.777 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:32.777 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:32.777 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:26:32.777 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:17.412606+0000", 2024-09-13T11:26:32.777 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1175999999999999e-05 2024-09-13T11:26:32.777 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:32.778 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:32.778 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:26:32.778 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:17.412690+0000", 2024-09-13T11:26:32.778 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.3486000000000007e-05 2024-09-13T11:26:32.778 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:32.778 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:32.778 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-09-13T11:26:32.778 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:17.412703+0000", 2024-09-13T11:26:32.778 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3196e-05 2024-09-13T11:26:32.778 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:32.778 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:32.778 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:26:32.779 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:18.142767+0000", 2024-09-13T11:26:32.779 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.73006412600000004 2024-09-13T11:26:32.779 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:32.779 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:32.779 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:26:32.779 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:18.142802+0000", 2024-09-13T11:26:32.779 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.4653000000000002e-05 2024-09-13T11:26:32.779 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:32.779 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:26:32.779 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:32.779 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:32.779 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:32.780 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGRecoveryDelete(3.6 e201,191 3:60767a7b:::benchmark_data_smithi177_31116_object14927:head,151'5598)", 2024-09-13T11:26:32.780 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:26:17.890539+0000", 2024-09-13T11:26:32.780 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 14.830464636, 2024-09-13T11:26:32.780 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.29203489500000002, 2024-09-13T11:26:32.780 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:26:32.780 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:26:32.780 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:26:32.780 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:26:32.780 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:32.780 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:26:32.780 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:17.890539+0000", 2024-09-13T11:26:32.780 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:32.781 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:32.781 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:32.781 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:26:32.781 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:17.890539+0000", 2024-09-13T11:26:32.781 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:32.781 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:32.781 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:32.781 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:26:32.781 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:17.890541+0000", 2024-09-13T11:26:32.781 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.795e-06 2024-09-13T11:26:32.781 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:32.781 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:32.781 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:26:32.781 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:17.890547+0000", 2024-09-13T11:26:32.782 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.3770000000000001e-06 2024-09-13T11:26:32.782 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:32.782 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:32.782 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:26:32.782 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:18.182175+0000", 2024-09-13T11:26:32.782 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.29162787000000001 2024-09-13T11:26:32.782 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:32.782 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:32.782 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:26:32.782 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:18.182190+0000", 2024-09-13T11:26:32.782 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4976e-05 2024-09-13T11:26:32.782 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:32.782 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:32.782 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:26:32.783 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:18.182397+0000", 2024-09-13T11:26:32.783 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00020751499999999999 2024-09-13T11:26:32.783 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:32.783 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:32.783 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:26:32.783 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:18.182425+0000", 2024-09-13T11:26:32.783 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.7076999999999998e-05 2024-09-13T11:26:32.783 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:32.783 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:32.783 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:26:32.783 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:18.182574+0000", 2024-09-13T11:26:32.783 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000149285 2024-09-13T11:26:32.783 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:32.783 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:26:32.784 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:32.784 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:32.784 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:32.784 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:234778 3.22 3.36a6aba2 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e201)", 2024-09-13T11:26:32.784 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:26:17.909436+0000", 2024-09-13T11:26:32.784 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 14.811567653999999, 2024-09-13T11:26:32.784 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.233461425, 2024-09-13T11:26:32.784 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:26:32.784 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:26:32.784 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-09-13T11:26:32.784 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:26:32.784 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:26:32.784 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:32.785 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 234778 2024-09-13T11:26:32.785 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:32.785 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:26:32.785 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:32.785 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:26:32.785 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:17.909436+0000", 2024-09-13T11:26:32.785 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:32.785 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:32.785 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:32.785 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:26:32.785 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:17.909436+0000", 2024-09-13T11:26:32.785 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:32.785 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:32.786 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:32.786 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:26:32.786 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:17.909439+0000", 2024-09-13T11:26:32.786 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.717e-06 2024-09-13T11:26:32.786 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:32.786 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:32.786 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:26:32.786 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:17.909446+0000", 2024-09-13T11:26:32.786 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.1980000000000004e-06 2024-09-13T11:26:32.786 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:32.786 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:32.786 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:26:32.786 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:17.909462+0000", 2024-09-13T11:26:32.787 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5809999999999999e-05 2024-09-13T11:26:32.787 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:32.787 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:32.787 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:26:32.787 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:17.909471+0000", 2024-09-13T11:26:32.787 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.3100000000000006e-06 2024-09-13T11:26:32.787 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:32.787 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:32.787 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:26:32.787 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:17.909554+0000", 2024-09-13T11:26:32.787 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.2917000000000006e-05 2024-09-13T11:26:32.787 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:32.787 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:32.788 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-09-13T11:26:32.788 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:17.909569+0000", 2024-09-13T11:26:32.788 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5201000000000001e-05 2024-09-13T11:26:32.788 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:32.788 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:32.788 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:26:32.788 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:18.142863+0000", 2024-09-13T11:26:32.788 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.23329387900000001 2024-09-13T11:26:32.788 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:32.788 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:32.788 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:26:32.788 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:18.142897+0000", 2024-09-13T11:26:32.788 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.4393000000000001e-05 2024-09-13T11:26:32.789 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:32.789 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:26:32.789 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:32.789 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:32.789 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:32.789 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:234793 3.22 3.215b2da2 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e201)", 2024-09-13T11:26:32.789 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:26:17.923001+0000", 2024-09-13T11:26:32.789 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 14.798002268999999, 2024-09-13T11:26:32.789 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.220029066, 2024-09-13T11:26:32.789 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:26:32.789 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:26:32.789 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-09-13T11:26:32.789 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:26:32.790 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:26:32.790 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:32.790 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 234793 2024-09-13T11:26:32.790 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:32.790 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:26:32.790 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:32.790 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:26:32.790 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:17.923001+0000", 2024-09-13T11:26:32.790 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:32.790 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:32.790 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:32.790 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:26:32.790 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:17.923001+0000", 2024-09-13T11:26:32.790 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:32.791 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:32.791 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:32.791 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:26:32.791 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:17.923004+0000", 2024-09-13T11:26:32.791 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.4619999999999999e-06 2024-09-13T11:26:32.791 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:32.791 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:32.791 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:26:32.791 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:17.923010+0000", 2024-09-13T11:26:32.791 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.759e-06 2024-09-13T11:26:32.791 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:32.791 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:32.791 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:26:32.792 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:17.923022+0000", 2024-09-13T11:26:32.792 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1361e-05 2024-09-13T11:26:32.792 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:32.792 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:32.792 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:26:32.792 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:17.923029+0000", 2024-09-13T11:26:32.792 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.7349999999999999e-06 2024-09-13T11:26:32.792 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:32.792 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:32.792 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:26:32.792 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:17.923080+0000", 2024-09-13T11:26:32.792 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.1535e-05 2024-09-13T11:26:32.792 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:32.792 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:32.793 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-09-13T11:26:32.793 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:17.923087+0000", 2024-09-13T11:26:32.793 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.1389999999999998e-06 2024-09-13T11:26:32.793 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:32.793 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:32.793 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:26:32.793 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:18.142999+0000", 2024-09-13T11:26:32.793 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.21991205799999999 2024-09-13T11:26:32.793 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:32.793 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:32.793 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:26:32.793 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:18.143030+0000", 2024-09-13T11:26:32.793 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.1016999999999998e-05 2024-09-13T11:26:32.794 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:32.794 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:26:32.794 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:32.794 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:32.794 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:32.794 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPush(3.2 201/192 [PushOp(3:409d6ad2:::benchmark_data_smithi177_31116_object97365:head, version: 113'3767, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:409d6ad2:::benchmark_data_smithi177_31116_object97365:head@113'3767, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:26:32.794 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:26:17.942627+0000", 2024-09-13T11:26:32.794 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 14.778376982999999, 2024-09-13T11:26:32.794 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.25342641300000002, 2024-09-13T11:26:32.794 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:26:32.794 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:26:32.794 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:26:32.794 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:26:32.795 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:32.795 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:26:32.795 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:17.942627+0000", 2024-09-13T11:26:32.795 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:32.795 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:32.795 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:32.795 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:26:32.795 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:17.942627+0000", 2024-09-13T11:26:32.795 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:32.795 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:32.795 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:32.795 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:26:32.795 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:17.942628+0000", 2024-09-13T11:26:32.796 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3179999999999999e-06 2024-09-13T11:26:32.796 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:32.796 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:32.796 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:26:32.796 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:17.942665+0000", 2024-09-13T11:26:32.796 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.6748999999999997e-05 2024-09-13T11:26:32.796 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:32.796 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:32.796 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:26:32.796 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:18.192821+0000", 2024-09-13T11:26:32.796 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.25015650900000003 2024-09-13T11:26:32.796 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:32.797 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:32.797 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:26:32.797 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:18.192853+0000", 2024-09-13T11:26:32.797 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.1579999999999999e-05 2024-09-13T11:26:32.797 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:32.797 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:32.797 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:26:32.797 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:18.193147+0000", 2024-09-13T11:26:32.797 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000294424 2024-09-13T11:26:32.797 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:32.797 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:32.797 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:26:32.797 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:18.193191+0000", 2024-09-13T11:26:32.798 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.3463e-05 2024-09-13T11:26:32.798 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:32.798 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:32.798 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:26:32.798 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:18.196053+0000", 2024-09-13T11:26:32.798 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0028623699999999999 2024-09-13T11:26:32.798 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:32.798 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:26:32.798 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:32.798 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:32.798 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:32.798 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGRecoveryDelete(3.6 e204,191 3:60665223:::benchmark_data_smithi177_31116_object38983:head,168'5973)", 2024-09-13T11:26:32.798 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:26:21.915203+0000", 2024-09-13T11:26:32.799 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 10.80580078, 2024-09-13T11:26:32.799 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.144080137, 2024-09-13T11:26:32.799 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:26:32.799 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:26:32.799 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:26:32.799 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:26:32.799 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:32.799 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:26:32.799 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:21.915203+0000", 2024-09-13T11:26:32.799 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:32.799 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:32.799 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:32.800 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:26:32.800 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:21.915203+0000", 2024-09-13T11:26:32.800 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:32.800 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:32.800 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:32.800 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:26:32.800 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:21.915205+0000", 2024-09-13T11:26:32.800 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.9360000000000002e-06 2024-09-13T11:26:32.800 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:32.800 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:32.800 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:26:32.800 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:21.915215+0000", 2024-09-13T11:26:32.800 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0641e-05 2024-09-13T11:26:32.801 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:32.801 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:32.801 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:26:32.801 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:22.059132+0000", 2024-09-13T11:26:32.801 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.14391625399999999 2024-09-13T11:26:32.801 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:32.801 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:32.801 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:26:32.801 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:22.059141+0000", 2024-09-13T11:26:32.801 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.3300000000000005e-06 2024-09-13T11:26:32.801 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:32.801 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:32.801 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:26:32.802 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:22.059207+0000", 2024-09-13T11:26:32.802 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.5778000000000006e-05 2024-09-13T11:26:32.802 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:32.802 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:32.802 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:26:32.802 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:22.059214+0000", 2024-09-13T11:26:32.802 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.2760000000000003e-06 2024-09-13T11:26:32.802 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:32.802 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:32.802 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:26:32.802 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:22.059283+0000", 2024-09-13T11:26:32.803 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.8922000000000002e-05 2024-09-13T11:26:32.803 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:32.803 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:26:32.803 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:32.803 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:32.803 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:32.803 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGRecoveryDelete(3.6 e205,191 3:60b10143:::benchmark_data_smithi177_31116_object41933:head,170'6043)", 2024-09-13T11:26:32.803 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:26:23.191752+0000", 2024-09-13T11:26:32.803 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 9.5292510789999998, 2024-09-13T11:26:32.803 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.12325507600000001, 2024-09-13T11:26:32.803 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:26:32.803 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:26:32.804 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:26:32.804 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:26:32.804 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:32.804 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:26:32.804 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:23.191752+0000", 2024-09-13T11:26:32.804 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:32.804 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:32.804 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:32.804 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:26:32.804 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:23.191752+0000", 2024-09-13T11:26:32.804 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:32.804 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:32.804 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:32.805 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:26:32.805 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:23.191754+0000", 2024-09-13T11:26:32.805 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4079999999999999e-06 2024-09-13T11:26:32.805 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:32.805 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:32.805 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:26:32.805 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:23.191758+0000", 2024-09-13T11:26:32.805 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.3830000000000002e-06 2024-09-13T11:26:32.805 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:32.805 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:32.805 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:26:32.805 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:23.314647+0000", 2024-09-13T11:26:32.805 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.122888368 2024-09-13T11:26:32.806 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:32.806 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:32.806 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:26:32.806 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:23.314656+0000", 2024-09-13T11:26:32.806 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.1370000000000008e-06 2024-09-13T11:26:32.806 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:32.806 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:32.806 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:26:32.806 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:23.314803+0000", 2024-09-13T11:26:32.806 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000147472 2024-09-13T11:26:32.806 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:32.806 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:32.806 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:26:32.807 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:23.314834+0000", 2024-09-13T11:26:32.807 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.0809999999999998e-05 2024-09-13T11:26:32.807 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:32.807 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:32.807 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:26:32.807 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:23.315008+0000", 2024-09-13T11:26:32.807 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00017349800000000001 2024-09-13T11:26:32.807 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:32.807 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:26:32.807 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:32.807 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:32.807 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:32.808 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPush(3.2 205/192 [PushOp(3:42b70fdc:::benchmark_data_smithi177_31116_object116542:head, version: 126'4065, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:42b70fdc:::benchmark_data_smithi177_31116_object116542:head@126'4065, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:26:32.808 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:26:23.424423+0000", 2024-09-13T11:26:32.808 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 9.2965801120000009, 2024-09-13T11:26:32.808 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.178307523, 2024-09-13T11:26:32.808 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:26:32.808 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:26:32.808 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:26:32.808 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:26:32.808 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:32.808 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:26:32.808 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:23.424423+0000", 2024-09-13T11:26:32.808 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:32.808 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:32.809 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:32.809 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:26:32.809 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:23.424423+0000", 2024-09-13T11:26:32.809 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:32.809 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:32.809 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:32.809 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:26:32.809 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:23.424432+0000", 2024-09-13T11:26:32.809 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.4260000000000003e-06 2024-09-13T11:26:32.809 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:32.809 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:32.809 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:26:32.809 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:23.424502+0000", 2024-09-13T11:26:32.810 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.0476999999999996e-05 2024-09-13T11:26:32.810 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:32.810 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:32.810 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:26:32.810 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:23.599928+0000", 2024-09-13T11:26:32.810 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.17542566600000001 2024-09-13T11:26:32.810 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:32.810 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:32.810 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:26:32.810 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:23.599953+0000", 2024-09-13T11:26:32.810 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.5389999999999999e-05 2024-09-13T11:26:32.810 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:32.810 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:32.810 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:26:32.811 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:23.600144+0000", 2024-09-13T11:26:32.811 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00019051800000000001 2024-09-13T11:26:32.811 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:32.811 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:32.811 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:26:32.811 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:23.600164+0000", 2024-09-13T11:26:32.811 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.0230000000000001e-05 2024-09-13T11:26:32.811 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:32.811 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:32.811 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:26:32.811 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:23.602731+0000", 2024-09-13T11:26:32.811 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.002566816 2024-09-13T11:26:32.811 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:32.811 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:26:32.812 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:32.812 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:32.812 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:32.812 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.4665.0:250605 3.1f e211/192 3:fa3fa2ea:::benchmark_data_smithi177_31116_object110302:head v 211'9333, mlcod=211'9330)", 2024-09-13T11:26:32.812 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:26:32.483026+0000", 2024-09-13T11:26:32.812 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 0.23797775099999999, 2024-09-13T11:26:32.812 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.124620835, 2024-09-13T11:26:32.812 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:26:32.812 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:26:32.813 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:32.813 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:26:32.813 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:32.813 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:26:32.813 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:32.483026+0000", 2024-09-13T11:26:32.813 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:32.813 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:32.813 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:32.813 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:26:32.813 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:32.483026+0000", 2024-09-13T11:26:32.813 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:32.813 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:32.813 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:32.814 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:26:32.814 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:32.483027+0000", 2024-09-13T11:26:32.814 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.437e-06 2024-09-13T11:26:32.814 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:32.814 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:32.814 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:26:32.814 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:32.483034+0000", 2024-09-13T11:26:32.814 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.0740000000000004e-06 2024-09-13T11:26:32.814 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:32.814 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:32.814 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:26:32.814 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:32.602543+0000", 2024-09-13T11:26:32.814 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.11950895 2024-09-13T11:26:32.815 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:32.815 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:32.815 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:26:32.815 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:32.602551+0000", 2024-09-13T11:26:32.815 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.0260000000000007e-06 2024-09-13T11:26:32.815 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:32.815 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:32.815 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:26:32.815 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:32.602617+0000", 2024-09-13T11:26:32.815 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.5815000000000006e-05 2024-09-13T11:26:32.815 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:32.815 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:32.815 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:26:32.815 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:32.602644+0000", 2024-09-13T11:26:32.816 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.6480999999999999e-05 2024-09-13T11:26:32.816 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:32.816 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:32.816 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:26:32.816 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:32.607600+0000", 2024-09-13T11:26:32.816 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0049561279999999997 2024-09-13T11:26:32.816 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:32.816 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:32.816 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:26:32.816 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:32.607647+0000", 2024-09-13T11:26:32.816 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.6924e-05 2024-09-13T11:26:32.816 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:32.816 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:26:32.817 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:32.817 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:32.817 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:26:32.817 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-13T11:26:32.818 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-13T11:26:32.819 INFO:tasks.ceph.osd.8.smithi001.stderr:2024-09-13T11:26:32.814+0000 7fb9b48ae640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26312) UID: 0 2024-09-13T11:26:32.919 INFO:tasks.ceph.osd.1.smithi092.stderr:2024-09-13T11:26:32.916+0000 7fe7789c5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33718) UID: 0 2024-09-13T11:26:32.976 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-13T11:26:32.976 INFO:teuthology.orchestra.run.smithi001.stdout: "ops": [ 2024-09-13T11:26:32.976 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:32.976 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:251069 3.e 3:777a8da7:::benchmark_data_smithi177_31116_object110766:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e211)", 2024-09-13T11:26:32.976 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:26:32.972317+0000", 2024-09-13T11:26:32.976 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 0.0034281329999999999, 2024-09-13T11:26:32.976 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0034559730000000002, 2024-09-13T11:26:32.977 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:26:32.977 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:26:32.977 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "waiting for sub ops", 2024-09-13T11:26:32.977 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:26:32.977 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:26:32.977 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:32.977 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 251069 2024-09-13T11:26:32.977 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:32.977 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:26:32.977 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:32.977 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:26:32.977 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:32.972317+0000", 2024-09-13T11:26:32.977 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:32.978 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:32.978 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:32.978 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:26:32.978 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:32.972317+0000", 2024-09-13T11:26:32.978 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:32.978 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:32.978 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:32.978 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:26:32.978 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:32.972320+0000", 2024-09-13T11:26:32.978 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.0710000000000002e-06 2024-09-13T11:26:32.978 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:32.978 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:32.978 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:26:32.979 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:32.972325+0000", 2024-09-13T11:26:32.979 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.5620000000000003e-06 2024-09-13T11:26:32.979 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:32.979 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:32.979 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:26:32.979 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:32.972342+0000", 2024-09-13T11:26:32.979 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.6889999999999999e-05 2024-09-13T11:26:32.979 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:32.979 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:32.979 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:26:32.979 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:32.972351+0000", 2024-09-13T11:26:32.979 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.3870000000000001e-06 2024-09-13T11:26:32.979 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:32.979 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:32.980 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:26:32.980 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:32.972420+0000", 2024-09-13T11:26:32.980 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.8149000000000001e-05 2024-09-13T11:26:32.980 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:32.980 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:32.980 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:26:32.980 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:32.972580+0000", 2024-09-13T11:26:32.980 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00016004800000000001 2024-09-13T11:26:32.980 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:32.980 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:32.980 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 1", 2024-09-13T11:26:32.980 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:32.972708+0000", 2024-09-13T11:26:32.980 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000127988 2024-09-13T11:26:32.980 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:32.981 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:32.981 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:32.981 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:32.981 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:32.981 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4665.0:251070 3.20 e211/77 3:06f655d6:::benchmark_data_smithi177_31116_object110767:head v 211'6224, mlcod=211'6222)", 2024-09-13T11:26:32.981 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:26:32.972978+0000", 2024-09-13T11:26:32.981 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 0.0027674140000000002, 2024-09-13T11:26:32.981 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0028893510000000001, 2024-09-13T11:26:32.981 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:26:32.981 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:26:32.981 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:26:32.981 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:26:32.981 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:32.982 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:26:32.982 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:32.972978+0000", 2024-09-13T11:26:32.982 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:32.982 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:32.982 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:32.982 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:26:32.982 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:32.972978+0000", 2024-09-13T11:26:32.982 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:32.982 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:32.982 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:32.982 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:26:32.982 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:32.972979+0000", 2024-09-13T11:26:32.982 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0729999999999999e-06 2024-09-13T11:26:32.983 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:32.983 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:32.983 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:26:32.983 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:32.972985+0000", 2024-09-13T11:26:32.983 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.879e-06 2024-09-13T11:26:32.983 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:32.983 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:32.983 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:26:32.983 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:32.973002+0000", 2024-09-13T11:26:32.983 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.6384000000000001e-05 2024-09-13T11:26:32.983 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:32.983 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:32.983 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:26:32.983 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:32.973007+0000", 2024-09-13T11:26:32.983 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.5459999999999999e-06 2024-09-13T11:26:32.984 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:32.984 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:32.984 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:26:32.984 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:32.973060+0000", 2024-09-13T11:26:32.984 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.3062999999999997e-05 2024-09-13T11:26:32.984 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:32.984 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:32.984 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:26:32.984 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:32.973076+0000", 2024-09-13T11:26:32.984 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.541e-05 2024-09-13T11:26:32.984 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:32.984 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:32.984 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:32.985 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:32.985 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:32.985 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDPGPushReply(3.15 211/192 [PushReplyOp(3:a9788e37:::benchmark_data_smithi177_31116_object111516:head)])", 2024-09-13T11:26:32.985 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:26:32.975592+0000", 2024-09-13T11:26:32.985 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 0.000153902, 2024-09-13T11:26:32.985 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00030654800000000002, 2024-09-13T11:26:32.985 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:26:32.985 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:26:32.985 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-13T11:26:32.985 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:26:32.985 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:32.985 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:26:32.985 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:32.975592+0000", 2024-09-13T11:26:32.986 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:32.986 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:32.986 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:32.986 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:26:32.986 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:32.975592+0000", 2024-09-13T11:26:32.986 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:32.986 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:32.986 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:32.986 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:26:32.986 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:32.975592+0000", 2024-09-13T11:26:32.986 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.8799999999999998e-07 2024-09-13T11:26:32.986 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:32.986 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:32.986 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:26:32.987 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:32.975596+0000", 2024-09-13T11:26:32.987 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.7469999999999999e-06 2024-09-13T11:26:32.987 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:32.987 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:32.987 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:26:32.987 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:32.975659+0000", 2024-09-13T11:26:32.987 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.3079999999999999e-05 2024-09-13T11:26:32.987 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:32.987 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:32.987 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:26:32.987 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:32.975666+0000", 2024-09-13T11:26:32.987 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.8770000000000004e-06 2024-09-13T11:26:32.987 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:32.987 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:32.987 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:26:32.988 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:32.975742+0000", 2024-09-13T11:26:32.988 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.6369e-05 2024-09-13T11:26:32.988 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:32.988 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:32.988 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:32.988 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:32.988 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:32.988 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:251040 3.15 3:ae371e0d:::benchmark_data_smithi177_31116_object110737:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e211)", 2024-09-13T11:26:32.988 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:26:32.946025+0000", 2024-09-13T11:26:32.988 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 0.029720612, 2024-09-13T11:26:32.988 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.029917997000000002, 2024-09-13T11:26:32.988 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:26:32.988 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:26:32.988 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "waiting for sub ops", 2024-09-13T11:26:32.989 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:26:32.989 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:26:32.989 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:32.989 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 251040 2024-09-13T11:26:32.989 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:32.989 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:26:32.989 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:32.989 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:26:32.989 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:32.946025+0000", 2024-09-13T11:26:32.989 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:32.989 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:32.989 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:32.989 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:26:32.990 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:32.946025+0000", 2024-09-13T11:26:32.990 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:32.990 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:32.990 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:32.990 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:26:32.990 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:32.946028+0000", 2024-09-13T11:26:32.990 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.7180000000000001e-06 2024-09-13T11:26:32.990 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:32.990 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:32.990 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:26:32.990 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:32.946034+0000", 2024-09-13T11:26:32.990 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.6270000000000002e-06 2024-09-13T11:26:32.990 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:32.990 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:32.991 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:26:32.991 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:32.946046+0000", 2024-09-13T11:26:32.991 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.1629e-05 2024-09-13T11:26:32.991 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:32.991 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:32.991 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:26:32.991 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:32.946052+0000", 2024-09-13T11:26:32.991 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.6319999999999996e-06 2024-09-13T11:26:32.991 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:32.991 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:32.991 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:26:32.991 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:32.946278+0000", 2024-09-13T11:26:32.991 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000225968 2024-09-13T11:26:32.991 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:32.992 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:32.992 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:26:32.992 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:32.946490+0000", 2024-09-13T11:26:32.992 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000212418 2024-09-13T11:26:32.992 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:32.992 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:32.992 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 3,14", 2024-09-13T11:26:32.992 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:32.946687+0000", 2024-09-13T11:26:32.992 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000196715 2024-09-13T11:26:32.992 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:32.992 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:32.992 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:26:32.992 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:32.950432+0000", 2024-09-13T11:26:32.992 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0037448939999999999 2024-09-13T11:26:32.993 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:32.993 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:32.993 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:32.993 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:32.953932+0000", 2024-09-13T11:26:32.993 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.003500076 2024-09-13T11:26:32.993 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:32.993 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:32.993 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:32.993 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:32.993 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-09-13T11:26:32.993 INFO:teuthology.orchestra.run.smithi001.stdout: "num_ops": 4 2024-09-13T11:26:32.993 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-13T11:26:32.995 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-13T11:26:33.044 INFO:tasks.ceph.osd.8.smithi001.stderr:2024-09-13T11:26:33.042+0000 7fb9b48ae640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26312) UID: 0 2024-09-13T11:26:33.120 INFO:tasks.ceph.osd.7.smithi177.stderr:2024-09-13T11:26:33.117+0000 7f897b51d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26575) UID: 0 2024-09-13T11:26:33.142 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-13T11:26:33.143 INFO:teuthology.orchestra.run.smithi001.stdout: "ops": [], 2024-09-13T11:26:33.143 INFO:teuthology.orchestra.run.smithi001.stdout: "complaint_time": 30, 2024-09-13T11:26:33.143 INFO:teuthology.orchestra.run.smithi001.stdout: "num_blocked_ops": 0 2024-09-13T11:26:33.143 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-13T11:26:33.164 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-13T11:26:33.191 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 15, 14] out_osds: [] dead_osds: [] live_osds: [0, 4, 8, 12, 5, 9, 13, 2, 6, 10, 3, 7, 11, 15, 1, 14] 2024-09-13T11:26:33.192 INFO:tasks.thrashosds.thrasher:choose_action: min_in 2 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-13T11:26:33.192 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2024-09-13T11:26:33.192 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump -f json-pretty 2024-09-13T11:26:33.221 INFO:tasks.ceph.osd.10.smithi162.stderr:2024-09-13T11:26:33.219+0000 7fb59c9f0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26317) UID: 0 2024-09-13T11:26:33.322 INFO:tasks.ceph.osd.13.smithi092.stderr:2024-09-13T11:26:33.320+0000 7f3e5afea640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26350) UID: 0 2024-09-13T11:26:33.372 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-13T11:26:33.372 INFO:teuthology.orchestra.run.smithi001.stdout: "size": 20, 2024-09-13T11:26:33.372 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 600, 2024-09-13T11:26:33.372 INFO:teuthology.orchestra.run.smithi001.stdout: "ops": [ 2024-09-13T11:26:33.372 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.372 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDPGPushReply(3.1e 84/78 [PushReplyOp(3:79ee54b9:::benchmark_data_smithi177_31116_object11701:head)])", 2024-09-13T11:26:33.372 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:41.407335+0000", 2024-09-13T11:26:33.373 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 171.96211030399999, 2024-09-13T11:26:33.373 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.51622350400000006, 2024-09-13T11:26:33.373 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:26:33.373 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:26:33.373 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-13T11:26:33.373 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:26:33.373 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.373 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:26:33.373 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:41.407335+0000", 2024-09-13T11:26:33.373 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:33.373 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.373 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.374 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:26:33.374 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:41.407335+0000", 2024-09-13T11:26:33.374 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:33.374 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.374 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.374 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:26:33.374 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:41.407337+0000", 2024-09-13T11:26:33.374 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.505e-06 2024-09-13T11:26:33.374 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.374 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.374 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:26:33.374 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:41.407343+0000", 2024-09-13T11:26:33.374 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.9089999999999997e-06 2024-09-13T11:26:33.375 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.375 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.375 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:26:33.375 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:41.921977+0000", 2024-09-13T11:26:33.375 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.51463463499999995 2024-09-13T11:26:33.375 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.375 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.375 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:26:33.375 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:41.921988+0000", 2024-09-13T11:26:33.375 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.092e-05 2024-09-13T11:26:33.375 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.375 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.375 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:26:33.375 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:41.922088+0000", 2024-09-13T11:26:33.376 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.9528e-05 2024-09-13T11:26:33.376 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.376 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.376 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:26:33.376 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:41.923559+0000", 2024-09-13T11:26:33.376 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0014710070000000001 2024-09-13T11:26:33.376 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:33.376 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:33.376 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:33.376 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.376 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.376 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4665.0:60417 3.10 e85/84 3:092808b0:::benchmark_data_smithi177_31116_object60416:head v 85'3295, mlcod=85'3292)", 2024-09-13T11:26:33.376 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:42.174488+0000", 2024-09-13T11:26:33.376 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 171.19495774999999, 2024-09-13T11:26:33.377 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.387908896, 2024-09-13T11:26:33.377 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:26:33.377 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:26:33.377 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:33.377 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:26:33.377 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.377 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:26:33.377 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.174488+0000", 2024-09-13T11:26:33.377 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:33.377 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.377 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.377 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:26:33.377 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.174488+0000", 2024-09-13T11:26:33.378 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:33.378 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.378 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.378 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:26:33.378 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.174490+0000", 2024-09-13T11:26:33.378 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.9099999999999999e-06 2024-09-13T11:26:33.378 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.378 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.378 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:26:33.378 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.174527+0000", 2024-09-13T11:26:33.378 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.7543000000000003e-05 2024-09-13T11:26:33.378 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.378 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.378 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:26:33.379 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.561093+0000", 2024-09-13T11:26:33.379 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.38656578400000002 2024-09-13T11:26:33.379 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.379 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.379 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:26:33.379 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.561101+0000", 2024-09-13T11:26:33.379 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.6090000000000003e-06 2024-09-13T11:26:33.379 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.379 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.379 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:26:33.379 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.561143+0000", 2024-09-13T11:26:33.379 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.2435000000000003e-05 2024-09-13T11:26:33.379 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.379 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.380 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:26:33.380 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.561164+0000", 2024-09-13T11:26:33.380 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.1276000000000001e-05 2024-09-13T11:26:33.380 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.380 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.380 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:26:33.380 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.562360+0000", 2024-09-13T11:26:33.380 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00119596 2024-09-13T11:26:33.380 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.380 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.380 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:26:33.380 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.562397+0000", 2024-09-13T11:26:33.380 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.6378999999999998e-05 2024-09-13T11:26:33.380 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:33.381 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:33.381 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:33.381 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.381 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.381 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDPGPush(3.22 85/78 [PushOp(3:456e8765:::benchmark_data_smithi177_31116_object16212:head, version: 52'1006, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:456e8765:::benchmark_data_smithi177_31116_object16212:head@52'1006, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:26:33.381 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:42.673067+0000", 2024-09-13T11:26:33.381 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 170.696378608, 2024-09-13T11:26:33.381 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.27935889899999999, 2024-09-13T11:26:33.381 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:26:33.381 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:26:33.381 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:26:33.381 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:26:33.381 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.381 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:26:33.382 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.673067+0000", 2024-09-13T11:26:33.382 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:33.382 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.382 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.382 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:26:33.382 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.673067+0000", 2024-09-13T11:26:33.382 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:33.382 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.382 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.382 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:26:33.382 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.673068+0000", 2024-09-13T11:26:33.382 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.9699999999999994e-07 2024-09-13T11:26:33.382 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.383 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.383 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:26:33.383 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.673090+0000", 2024-09-13T11:26:33.383 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.2416999999999999e-05 2024-09-13T11:26:33.383 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.383 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.383 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:26:33.383 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.951363+0000", 2024-09-13T11:26:33.383 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.27827291199999998 2024-09-13T11:26:33.383 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.383 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.383 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:26:33.383 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.951386+0000", 2024-09-13T11:26:33.384 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.2226999999999999e-05 2024-09-13T11:26:33.384 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.384 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.384 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:26:33.384 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.951581+0000", 2024-09-13T11:26:33.384 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000195713 2024-09-13T11:26:33.384 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.384 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.384 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:26:33.384 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.951605+0000", 2024-09-13T11:26:33.384 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.3767000000000001e-05 2024-09-13T11:26:33.384 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.385 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.385 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:26:33.385 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.952426+0000", 2024-09-13T11:26:33.385 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00082086599999999996 2024-09-13T11:26:33.385 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:33.385 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:33.385 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:33.385 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.385 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.385 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "replica_scrub(pg: 3.20,from:0'0,to:89'3254,epoch:89/77,start:3:04000000::::0,end:3:045ba805:::benchmark_data_smithi177_31116_object43251:0,chunky:1,deep:0,version:9,allow_preemption:1,priority=5)", 2024-09-13T11:26:33.385 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:46.641364+0000", 2024-09-13T11:26:33.385 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 166.72808192700001, 2024-09-13T11:26:33.385 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.27141393000000003, 2024-09-13T11:26:33.386 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:26:33.386 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:26:33.386 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:26:33.386 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:26:33.386 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.386 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:26:33.386 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.641364+0000", 2024-09-13T11:26:33.386 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:33.386 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.386 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.386 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:26:33.386 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.641364+0000", 2024-09-13T11:26:33.386 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:33.387 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.387 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.387 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:26:33.387 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.641365+0000", 2024-09-13T11:26:33.387 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.266e-06 2024-09-13T11:26:33.387 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.387 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.387 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:26:33.387 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.641370+0000", 2024-09-13T11:26:33.387 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.8539999999999996e-06 2024-09-13T11:26:33.387 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.387 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.387 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:26:33.387 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.912491+0000", 2024-09-13T11:26:33.388 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.27112103100000001 2024-09-13T11:26:33.388 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.388 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.388 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:26:33.388 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.912520+0000", 2024-09-13T11:26:33.388 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.9379000000000001e-05 2024-09-13T11:26:33.388 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.388 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.388 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:26:33.388 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.912679+0000", 2024-09-13T11:26:33.388 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00015915800000000001 2024-09-13T11:26:33.388 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.388 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.388 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:26:33.389 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.912714+0000", 2024-09-13T11:26:33.389 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.4687999999999997e-05 2024-09-13T11:26:33.389 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.389 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.389 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:26:33.389 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.912778+0000", 2024-09-13T11:26:33.389 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.3553999999999995e-05 2024-09-13T11:26:33.389 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:33.389 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:33.389 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:33.389 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.389 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.389 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDPGPush(3.18 89/70 [PushOp(3:1ac6c361:::benchmark_data_smithi177_31116_object44972:head, version: 68'2767, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:1ac6c361:::benchmark_data_smithi177_31116_object44972:head@68'2767, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:26:33.390 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:46.671018+0000", 2024-09-13T11:26:33.390 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 166.69842779499999, 2024-09-13T11:26:33.390 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.271282931, 2024-09-13T11:26:33.390 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:26:33.390 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:26:33.390 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:26:33.390 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:26:33.390 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.390 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:26:33.390 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.671018+0000", 2024-09-13T11:26:33.390 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:33.390 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.390 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.390 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:26:33.391 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.671018+0000", 2024-09-13T11:26:33.391 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:33.391 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.391 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.391 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:26:33.391 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.671020+0000", 2024-09-13T11:26:33.391 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.018e-06 2024-09-13T11:26:33.391 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.391 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.391 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:26:33.391 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.671097+0000", 2024-09-13T11:26:33.391 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.7218999999999994e-05 2024-09-13T11:26:33.391 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.391 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.391 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:26:33.392 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.939773+0000", 2024-09-13T11:26:33.392 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.26867550200000001 2024-09-13T11:26:33.392 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.392 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.392 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:26:33.392 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.939791+0000", 2024-09-13T11:26:33.392 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.8185999999999998e-05 2024-09-13T11:26:33.392 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.392 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.392 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:26:33.392 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.939926+0000", 2024-09-13T11:26:33.392 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000135291 2024-09-13T11:26:33.392 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.392 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.393 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:26:33.393 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.939938+0000", 2024-09-13T11:26:33.393 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2393e-05 2024-09-13T11:26:33.393 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.393 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.393 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:26:33.393 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.942301+0000", 2024-09-13T11:26:33.393 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.002362322 2024-09-13T11:26:33.393 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:33.393 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:33.393 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:33.393 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.393 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.393 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "replica_scrub(pg: 3.20,from:0'0,to:90'3264,epoch:90/77,start:3:0650aac8:::benchmark_data_smithi177_31116_object1734:0,end:3:06af01e3:::benchmark_data_smithi177_31116_object31572:0,chunky:1,deep:0,version:9,allow_preemption:1,priority=5)", 2024-09-13T11:26:33.394 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:47.380615+0000", 2024-09-13T11:26:33.394 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 165.988830372, 2024-09-13T11:26:33.394 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.51951398400000004, 2024-09-13T11:26:33.394 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:26:33.394 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:26:33.394 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:26:33.394 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:26:33.394 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.394 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:26:33.394 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:47.380615+0000", 2024-09-13T11:26:33.394 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:33.394 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.394 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.395 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:26:33.395 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:47.380615+0000", 2024-09-13T11:26:33.395 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:33.395 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.395 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.395 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:26:33.395 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:47.380617+0000", 2024-09-13T11:26:33.395 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.671e-06 2024-09-13T11:26:33.395 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.395 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.395 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:26:33.395 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:47.380623+0000", 2024-09-13T11:26:33.395 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.1190000000000002e-06 2024-09-13T11:26:33.395 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.396 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.396 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:26:33.396 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:47.900050+0000", 2024-09-13T11:26:33.396 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.51942737400000005 2024-09-13T11:26:33.396 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.396 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.396 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:26:33.396 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:47.900057+0000", 2024-09-13T11:26:33.396 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.4359999999999999e-06 2024-09-13T11:26:33.396 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.396 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.396 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:26:33.396 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:47.900101+0000", 2024-09-13T11:26:33.396 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.4675000000000002e-05 2024-09-13T11:26:33.397 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.397 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.397 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:26:33.397 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:47.900110+0000", 2024-09-13T11:26:33.397 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.0569999999999993e-06 2024-09-13T11:26:33.397 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.397 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.397 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:26:33.397 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:47.900129+0000", 2024-09-13T11:26:33.397 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.9652000000000001e-05 2024-09-13T11:26:33.397 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:33.397 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:33.397 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:33.398 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.398 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.398 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDPGPush(3.21 91/78 [PushOp(3:8492321f:::benchmark_data_smithi177_31116_object13767:head, version: 50'840, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:8492321f:::benchmark_data_smithi177_31116_object13767:head@50'840, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:26:33.398 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:48.319251+0000", 2024-09-13T11:26:33.398 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 165.05019484300001, 2024-09-13T11:26:33.398 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.60903802200000001, 2024-09-13T11:26:33.398 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:26:33.398 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:26:33.398 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:26:33.398 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:26:33.398 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.398 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:26:33.398 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:48.319251+0000", 2024-09-13T11:26:33.398 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:33.399 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.399 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.399 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:26:33.399 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:48.319251+0000", 2024-09-13T11:26:33.399 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:33.399 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.399 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.399 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:26:33.399 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:48.319252+0000", 2024-09-13T11:26:33.399 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2640000000000001e-06 2024-09-13T11:26:33.399 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.399 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.399 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:26:33.399 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:48.319363+0000", 2024-09-13T11:26:33.400 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000110559 2024-09-13T11:26:33.400 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.400 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.400 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:26:33.400 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:48.927115+0000", 2024-09-13T11:26:33.400 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.60775277800000005 2024-09-13T11:26:33.400 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.400 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.400 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:26:33.400 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:48.927144+0000", 2024-09-13T11:26:33.400 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.8861e-05 2024-09-13T11:26:33.400 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.401 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.401 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:26:33.401 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:48.927318+0000", 2024-09-13T11:26:33.401 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00017399000000000001 2024-09-13T11:26:33.401 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.401 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.401 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:26:33.401 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:48.927340+0000", 2024-09-13T11:26:33.401 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.1433999999999998e-05 2024-09-13T11:26:33.401 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.401 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.401 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:26:33.401 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:48.928289+0000", 2024-09-13T11:26:33.402 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00094913599999999995 2024-09-13T11:26:33.402 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:33.402 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:33.402 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:33.402 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.402 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.402 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:83239 3.e 3:74547f17:::benchmark_data_smithi177_31116_object83238:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106)", 2024-09-13T11:26:33.402 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:24:04.580085+0000", 2024-09-13T11:26:33.402 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 148.789360762, 2024-09-13T11:26:33.402 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.33785762600000002, 2024-09-13T11:26:33.402 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:26:33.402 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:26:33.402 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:33.403 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:26:33.403 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:26:33.403 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:33.403 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 83239 2024-09-13T11:26:33.403 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.403 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:26:33.403 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.403 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:26:33.403 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:04.580085+0000", 2024-09-13T11:26:33.403 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:33.403 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.403 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.403 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:26:33.403 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:04.580085+0000", 2024-09-13T11:26:33.404 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:33.404 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.404 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.404 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:26:33.404 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:04.580089+0000", 2024-09-13T11:26:33.404 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.053e-06 2024-09-13T11:26:33.404 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.404 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.404 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:26:33.404 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:04.580166+0000", 2024-09-13T11:26:33.404 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.7102999999999997e-05 2024-09-13T11:26:33.404 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.404 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.405 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:26:33.405 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:04.580196+0000", 2024-09-13T11:26:33.405 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.0196e-05 2024-09-13T11:26:33.405 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.405 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.405 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:26:33.405 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:04.580212+0000", 2024-09-13T11:26:33.405 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.535e-05 2024-09-13T11:26:33.405 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.406 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.406 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:26:33.406 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:04.580336+0000", 2024-09-13T11:26:33.406 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00012431899999999999 2024-09-13T11:26:33.406 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.406 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.406 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:26:33.406 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:04.580575+0000", 2024-09-13T11:26:33.406 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00023924100000000001 2024-09-13T11:26:33.406 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.406 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.406 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 1", 2024-09-13T11:26:33.406 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:04.580798+0000", 2024-09-13T11:26:33.407 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00022301499999999999 2024-09-13T11:26:33.407 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.407 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.407 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:26:33.407 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:04.583404+0000", 2024-09-13T11:26:33.407 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0026056120000000002 2024-09-13T11:26:33.407 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.407 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.407 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:33.407 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:04.917768+0000", 2024-09-13T11:26:33.407 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.33436387499999998 2024-09-13T11:26:33.407 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.407 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.408 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:26:33.408 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:04.917859+0000", 2024-09-13T11:26:33.408 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.1732999999999997e-05 2024-09-13T11:26:33.408 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.408 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.408 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:26:33.408 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:04.917942+0000", 2024-09-13T11:26:33.408 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.3128999999999996e-05 2024-09-13T11:26:33.408 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:33.408 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:33.408 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:33.408 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.409 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.409 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:110271 3.2d 3.3c2681ad (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e119)", 2024-09-13T11:26:33.409 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:24:25.403006+0000", 2024-09-13T11:26:33.409 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 127.966439278, 2024-09-13T11:26:33.409 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.68768755999999998, 2024-09-13T11:26:33.409 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:26:33.409 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:26:33.409 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-13T11:26:33.409 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:26:33.409 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:26:33.409 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:33.409 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 110271 2024-09-13T11:26:33.409 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.410 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:26:33.410 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.410 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:26:33.410 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:25.403006+0000", 2024-09-13T11:26:33.410 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:33.410 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.410 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.410 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:26:33.410 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:25.403006+0000", 2024-09-13T11:26:33.410 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:33.410 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.410 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.410 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:26:33.411 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:25.403011+0000", 2024-09-13T11:26:33.411 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.5199999999999999e-06 2024-09-13T11:26:33.411 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.411 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.411 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:26:33.411 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:25.403065+0000", 2024-09-13T11:26:33.411 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.4478999999999998e-05 2024-09-13T11:26:33.411 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.411 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.411 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:26:33.411 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:25.403097+0000", 2024-09-13T11:26:33.411 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.1636e-05 2024-09-13T11:26:33.411 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.411 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.412 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:26:33.412 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:25.403121+0000", 2024-09-13T11:26:33.412 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.3924e-05 2024-09-13T11:26:33.412 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.412 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.412 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:26:33.412 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:25.403216+0000", 2024-09-13T11:26:33.412 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.5557999999999995e-05 2024-09-13T11:26:33.412 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.412 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.412 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for peered", 2024-09-13T11:26:33.412 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:25.403234+0000", 2024-09-13T11:26:33.412 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.7104000000000001e-05 2024-09-13T11:26:33.412 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.413 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.413 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:26:33.413 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:26.090684+0000", 2024-09-13T11:26:33.413 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.68745000700000003 2024-09-13T11:26:33.413 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.413 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.413 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:26:33.413 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:26.090694+0000", 2024-09-13T11:26:33.413 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0332e-05 2024-09-13T11:26:33.413 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:33.413 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:33.413 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:33.414 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.414 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.414 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4665.0:129901 3.20 e134/77 3:067f5dfc:::benchmark_data_smithi177_31116_object129900:head v 134'4294, mlcod=134'4291)", 2024-09-13T11:26:33.414 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:24:47.145530+0000", 2024-09-13T11:26:33.414 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 106.223915203, 2024-09-13T11:26:33.414 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.333688349, 2024-09-13T11:26:33.414 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:26:33.414 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:26:33.414 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:33.414 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:26:33.414 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.414 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:26:33.414 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:47.145530+0000", 2024-09-13T11:26:33.415 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:33.415 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.415 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.415 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:26:33.415 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:47.145530+0000", 2024-09-13T11:26:33.415 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:33.415 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.415 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.415 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:26:33.415 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:47.145532+0000", 2024-09-13T11:26:33.415 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.7999999999999999e-06 2024-09-13T11:26:33.415 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.415 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.415 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:26:33.416 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:47.145557+0000", 2024-09-13T11:26:33.416 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.4678999999999999e-05 2024-09-13T11:26:33.416 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.416 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.416 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:26:33.416 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:47.477539+0000", 2024-09-13T11:26:33.416 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.33198164099999999 2024-09-13T11:26:33.416 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.416 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.416 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:26:33.416 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:47.477544+0000", 2024-09-13T11:26:33.416 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.9059999999999999e-06 2024-09-13T11:26:33.416 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.417 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.417 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:26:33.417 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:47.477597+0000", 2024-09-13T11:26:33.417 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.295e-05 2024-09-13T11:26:33.417 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.417 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.417 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:26:33.417 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:47.477619+0000", 2024-09-13T11:26:33.417 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.1835999999999999e-05 2024-09-13T11:26:33.417 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.417 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.417 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:26:33.417 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:47.479175+0000", 2024-09-13T11:26:33.418 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0015560470000000001 2024-09-13T11:26:33.418 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.418 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.418 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:26:33.418 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:47.479219+0000", 2024-09-13T11:26:33.418 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.3489999999999999e-05 2024-09-13T11:26:33.418 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:33.418 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:33.418 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:33.418 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.418 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.418 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:140840 3.21 3:87ae3db6:::benchmark_data_smithi177_31116_object537:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e144)", 2024-09-13T11:26:33.418 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:24:58.810939+0000", 2024-09-13T11:26:33.419 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 94.558506089999995, 2024-09-13T11:26:33.419 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.25628309300000002, 2024-09-13T11:26:33.419 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:26:33.419 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:26:33.419 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:33.419 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:26:33.419 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:26:33.419 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:33.419 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 140840 2024-09-13T11:26:33.419 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.419 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:26:33.419 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.419 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:26:33.420 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.810939+0000", 2024-09-13T11:26:33.420 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:33.420 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.420 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.420 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:26:33.420 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.810939+0000", 2024-09-13T11:26:33.420 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:33.420 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.420 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.420 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:26:33.420 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.810942+0000", 2024-09-13T11:26:33.420 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.6259999999999999e-06 2024-09-13T11:26:33.420 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.421 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.421 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:26:33.421 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.810950+0000", 2024-09-13T11:26:33.421 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.9500000000000001e-06 2024-09-13T11:26:33.421 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.421 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.421 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:26:33.421 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.810970+0000", 2024-09-13T11:26:33.421 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.9539999999999999e-05 2024-09-13T11:26:33.421 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.421 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.421 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:26:33.421 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.810978+0000", 2024-09-13T11:26:33.421 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.8680000000000005e-06 2024-09-13T11:26:33.422 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.422 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.422 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:26:33.422 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.811060+0000", 2024-09-13T11:26:33.422 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.1068000000000004e-05 2024-09-13T11:26:33.422 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.422 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.422 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:26:33.422 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.811174+0000", 2024-09-13T11:26:33.422 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000114385 2024-09-13T11:26:33.422 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.422 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.422 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 8", 2024-09-13T11:26:33.423 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.811255+0000", 2024-09-13T11:26:33.423 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.1298000000000007e-05 2024-09-13T11:26:33.423 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.423 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.423 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:26:33.423 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.812869+0000", 2024-09-13T11:26:33.423 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.001614234 2024-09-13T11:26:33.423 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.423 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.423 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:33.423 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:59.067155+0000", 2024-09-13T11:26:33.423 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.25428578299999999 2024-09-13T11:26:33.424 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.424 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.424 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:26:33.424 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:59.067192+0000", 2024-09-13T11:26:33.424 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.659e-05 2024-09-13T11:26:33.424 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.424 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.424 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:26:33.424 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:59.067223+0000", 2024-09-13T11:26:33.424 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.0750999999999997e-05 2024-09-13T11:26:33.424 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:33.424 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:33.425 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:33.425 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.425 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.425 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4665.0:191286 3.18 e175/91 3:1ffc0b26:::benchmark_data_smithi177_31116_object50983:head v 175'7339, mlcod=175'7337)", 2024-09-13T11:26:33.425 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:25:39.153114+0000", 2024-09-13T11:26:33.425 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 54.216331478000001, 2024-09-13T11:26:33.425 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.23200507400000001, 2024-09-13T11:26:33.425 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:26:33.425 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:26:33.425 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:33.425 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:26:33.425 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.425 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:26:33.426 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:39.153114+0000", 2024-09-13T11:26:33.426 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:33.426 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.426 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.426 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:26:33.426 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:39.153114+0000", 2024-09-13T11:26:33.426 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:33.426 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.426 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.426 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:26:33.426 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:39.153117+0000", 2024-09-13T11:26:33.426 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.9170000000000002e-06 2024-09-13T11:26:33.426 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.426 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.427 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:26:33.427 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:39.153130+0000", 2024-09-13T11:26:33.427 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2493e-05 2024-09-13T11:26:33.427 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.427 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.427 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:26:33.427 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:39.381951+0000", 2024-09-13T11:26:33.427 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.22882103000000001 2024-09-13T11:26:33.427 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.427 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.427 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:26:33.427 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:39.381956+0000", 2024-09-13T11:26:33.427 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.1359999999999996e-06 2024-09-13T11:26:33.428 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.428 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.428 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:26:33.428 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:39.382030+0000", 2024-09-13T11:26:33.428 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.4270999999999994e-05 2024-09-13T11:26:33.428 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.428 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.428 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:26:33.428 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:39.382054+0000", 2024-09-13T11:26:33.428 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.4142999999999999e-05 2024-09-13T11:26:33.428 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.428 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.428 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:26:33.429 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:39.385086+0000", 2024-09-13T11:26:33.429 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00303163 2024-09-13T11:26:33.429 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.429 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.429 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:26:33.429 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:39.385119+0000", 2024-09-13T11:26:33.429 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.3454000000000002e-05 2024-09-13T11:26:33.429 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:33.429 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:33.429 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:33.429 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.429 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.429 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4665.0:196438 3.20 e178/77 3:0573f1a4:::benchmark_data_smithi177_31116_object56135:head v 178'5356, mlcod=178'5353)", 2024-09-13T11:26:33.429 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:25:43.522272+0000", 2024-09-13T11:26:33.430 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 49.847173388000002, 2024-09-13T11:26:33.430 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.23723141, 2024-09-13T11:26:33.430 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:26:33.430 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:26:33.430 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:33.430 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:26:33.430 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.430 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:26:33.430 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:43.522272+0000", 2024-09-13T11:26:33.430 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:33.430 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.430 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.430 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:26:33.431 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:43.522272+0000", 2024-09-13T11:26:33.431 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:33.431 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.431 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.431 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:26:33.431 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:43.522274+0000", 2024-09-13T11:26:33.431 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.573e-06 2024-09-13T11:26:33.431 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.431 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.431 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:26:33.431 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:43.522281+0000", 2024-09-13T11:26:33.431 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.5490000000000001e-06 2024-09-13T11:26:33.431 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.431 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.432 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:26:33.432 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:43.756836+0000", 2024-09-13T11:26:33.432 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.23455496000000001 2024-09-13T11:26:33.432 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.432 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.432 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:26:33.432 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:43.756843+0000", 2024-09-13T11:26:33.432 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.4620000000000001e-06 2024-09-13T11:26:33.432 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.432 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.432 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:26:33.432 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:43.756914+0000", 2024-09-13T11:26:33.432 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.1276e-05 2024-09-13T11:26:33.432 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.433 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.433 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:26:33.433 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:43.756938+0000", 2024-09-13T11:26:33.433 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.4281999999999999e-05 2024-09-13T11:26:33.433 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.433 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.433 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:26:33.433 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:43.759408+0000", 2024-09-13T11:26:33.433 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0024699449999999999 2024-09-13T11:26:33.433 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.433 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.433 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:26:33.433 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:43.759504+0000", 2024-09-13T11:26:33.434 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.5363000000000002e-05 2024-09-13T11:26:33.434 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:33.434 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:33.434 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:33.434 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.434 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.434 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:201234 3.21 3:85402570:::benchmark_data_smithi177_31116_object60931:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e181)", 2024-09-13T11:26:33.434 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:25:47.835618+0000", 2024-09-13T11:26:33.434 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 45.533827924999997, 2024-09-13T11:26:33.434 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.26153554899999998, 2024-09-13T11:26:33.434 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:26:33.434 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:26:33.434 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:33.434 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:26:33.435 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:26:33.435 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:33.435 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 201234 2024-09-13T11:26:33.435 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.435 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:26:33.435 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.435 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:26:33.435 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:47.835618+0000", 2024-09-13T11:26:33.435 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:33.435 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.435 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.435 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:26:33.435 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:47.835618+0000", 2024-09-13T11:26:33.435 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:33.436 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.436 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.436 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:26:33.436 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:47.835619+0000", 2024-09-13T11:26:33.436 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.638e-06 2024-09-13T11:26:33.436 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.436 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.436 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:26:33.436 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:47.835627+0000", 2024-09-13T11:26:33.436 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.0150000000000003e-06 2024-09-13T11:26:33.436 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.436 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.436 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:26:33.436 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:47.835639+0000", 2024-09-13T11:26:33.437 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.1939e-05 2024-09-13T11:26:33.437 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.437 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.437 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:26:33.437 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:47.835644+0000", 2024-09-13T11:26:33.437 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.006e-06 2024-09-13T11:26:33.437 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.437 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.437 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:26:33.437 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:47.835696+0000", 2024-09-13T11:26:33.437 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.1668000000000001e-05 2024-09-13T11:26:33.437 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.437 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.438 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:26:33.438 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:47.835775+0000", 2024-09-13T11:26:33.438 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.8737000000000002e-05 2024-09-13T11:26:33.438 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.438 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.438 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 8", 2024-09-13T11:26:33.438 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:47.835857+0000", 2024-09-13T11:26:33.438 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.2508000000000004e-05 2024-09-13T11:26:33.438 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.438 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.438 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:26:33.438 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:47.837366+0000", 2024-09-13T11:26:33.438 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.001509292 2024-09-13T11:26:33.439 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.439 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.439 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:33.439 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:48.097051+0000", 2024-09-13T11:26:33.439 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.25968466699999998 2024-09-13T11:26:33.439 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.439 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.439 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:26:33.439 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:48.097109+0000", 2024-09-13T11:26:33.439 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.7778000000000001e-05 2024-09-13T11:26:33.439 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.439 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.439 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:26:33.440 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:48.097153+0000", 2024-09-13T11:26:33.440 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.4301000000000002e-05 2024-09-13T11:26:33.440 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:33.440 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:33.440 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:33.440 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.440 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.440 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:217082 3.22 3:449e4b87:::benchmark_data_smithi177_31116_object76779:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e187)", 2024-09-13T11:26:33.440 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:25:59.883418+0000", 2024-09-13T11:26:33.440 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 33.486027927999999, 2024-09-13T11:26:33.440 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.25557793200000001, 2024-09-13T11:26:33.440 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:26:33.440 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:26:33.441 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:33.441 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:26:33.441 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:26:33.441 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:33.441 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 217082 2024-09-13T11:26:33.441 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.441 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:26:33.441 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.441 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:26:33.441 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:59.883418+0000", 2024-09-13T11:26:33.441 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:33.441 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.441 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.441 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:26:33.442 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:59.883418+0000", 2024-09-13T11:26:33.442 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:33.442 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.442 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.442 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:26:33.442 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:59.883419+0000", 2024-09-13T11:26:33.442 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.235e-06 2024-09-13T11:26:33.442 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.442 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.442 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:26:33.442 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:59.883423+0000", 2024-09-13T11:26:33.442 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.7280000000000001e-06 2024-09-13T11:26:33.442 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.442 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.443 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:26:33.443 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:59.883433+0000", 2024-09-13T11:26:33.443 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0498e-05 2024-09-13T11:26:33.443 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.443 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.443 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:26:33.443 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:59.883439+0000", 2024-09-13T11:26:33.443 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.5890000000000002e-06 2024-09-13T11:26:33.443 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.443 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.443 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:26:33.443 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:59.883483+0000", 2024-09-13T11:26:33.443 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.4710999999999999e-05 2024-09-13T11:26:33.443 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.444 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.444 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:26:33.444 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:59.883558+0000", 2024-09-13T11:26:33.444 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.4308999999999996e-05 2024-09-13T11:26:33.444 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.444 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.444 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 5", 2024-09-13T11:26:33.444 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:59.883615+0000", 2024-09-13T11:26:33.444 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.7745999999999999e-05 2024-09-13T11:26:33.444 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.444 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.444 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:26:33.444 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:59.885044+0000", 2024-09-13T11:26:33.445 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0014282940000000001 2024-09-13T11:26:33.445 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.445 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.445 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:33.445 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:00.138864+0000", 2024-09-13T11:26:33.445 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.253819923 2024-09-13T11:26:33.445 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.445 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.445 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:26:33.445 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:00.138936+0000", 2024-09-13T11:26:33.445 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.2268999999999995e-05 2024-09-13T11:26:33.445 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.445 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.446 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:26:33.446 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:00.138996+0000", 2024-09-13T11:26:33.446 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.9630000000000003e-05 2024-09-13T11:26:33.446 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:33.446 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:33.446 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:33.446 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.446 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.446 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:221463 3.15 3:aaa4e83d:::benchmark_data_smithi177_31116_object81160:head [delete] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e193)", 2024-09-13T11:26:33.446 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:26:06.057961+0000", 2024-09-13T11:26:33.446 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 27.311484412999999, 2024-09-13T11:26:33.446 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.24029746699999999, 2024-09-13T11:26:33.446 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:26:33.447 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:26:33.447 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:33.447 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:26:33.447 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:26:33.447 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:33.447 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 221463 2024-09-13T11:26:33.447 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.447 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:26:33.447 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.447 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:26:33.447 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:06.057961+0000", 2024-09-13T11:26:33.447 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:33.448 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.448 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.448 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:26:33.448 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:06.057961+0000", 2024-09-13T11:26:33.448 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:33.448 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.448 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.448 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:26:33.448 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:06.057963+0000", 2024-09-13T11:26:33.448 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.61e-06 2024-09-13T11:26:33.448 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.448 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.448 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:26:33.449 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:06.057970+0000", 2024-09-13T11:26:33.449 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.9999999999999999e-06 2024-09-13T11:26:33.449 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.449 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.449 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:26:33.449 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:06.058003+0000", 2024-09-13T11:26:33.449 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.2733e-05 2024-09-13T11:26:33.449 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.449 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.449 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:26:33.449 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:06.058013+0000", 2024-09-13T11:26:33.449 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0349999999999999e-05 2024-09-13T11:26:33.449 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.450 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.450 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:26:33.450 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:06.058333+0000", 2024-09-13T11:26:33.450 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000319947 2024-09-13T11:26:33.450 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.450 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.450 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for peered", 2024-09-13T11:26:33.450 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:06.058346+0000", 2024-09-13T11:26:33.450 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.3657e-05 2024-09-13T11:26:33.450 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.450 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.450 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:26:33.450 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:06.277695+0000", 2024-09-13T11:26:33.451 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.219348129 2024-09-13T11:26:33.451 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.451 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.451 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:26:33.451 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:06.277819+0000", 2024-09-13T11:26:33.451 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00012432599999999999 2024-09-13T11:26:33.451 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.451 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.451 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 3,14", 2024-09-13T11:26:33.451 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:06.277928+0000", 2024-09-13T11:26:33.451 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00010860499999999999 2024-09-13T11:26:33.451 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.451 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.451 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:33.452 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:06.286021+0000", 2024-09-13T11:26:33.452 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0080932020000000007 2024-09-13T11:26:33.452 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.452 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.452 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:26:33.452 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:06.291903+0000", 2024-09-13T11:26:33.452 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0058824630000000001 2024-09-13T11:26:33.452 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.452 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.452 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:33.452 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:06.298142+0000", 2024-09-13T11:26:33.452 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.006239301 2024-09-13T11:26:33.452 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.453 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.453 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:26:33.453 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:06.298189+0000", 2024-09-13T11:26:33.453 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.6115999999999997e-05 2024-09-13T11:26:33.453 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.453 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.453 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:26:33.453 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:06.298259+0000", 2024-09-13T11:26:33.453 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.0028000000000001e-05 2024-09-13T11:26:33.453 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:33.453 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:33.453 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:33.453 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.454 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.454 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:221475 3.15 3:abeb12df:::benchmark_data_smithi177_31116_object81172:head [delete] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e193)", 2024-09-13T11:26:33.454 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:26:06.058032+0000", 2024-09-13T11:26:33.454 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 27.311413380000001, 2024-09-13T11:26:33.454 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.24037931400000001, 2024-09-13T11:26:33.454 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:26:33.454 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:26:33.454 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:33.454 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:26:33.454 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:26:33.454 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:33.454 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 221475 2024-09-13T11:26:33.455 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.455 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:26:33.455 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.455 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:26:33.455 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:06.058032+0000", 2024-09-13T11:26:33.455 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:33.455 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.455 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.455 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:26:33.455 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:06.058032+0000", 2024-09-13T11:26:33.455 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:33.455 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.455 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.456 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:26:33.456 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:06.058033+0000", 2024-09-13T11:26:33.456 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.136e-06 2024-09-13T11:26:33.456 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.456 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.456 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:26:33.456 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:06.058038+0000", 2024-09-13T11:26:33.456 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.9169999999999997e-06 2024-09-13T11:26:33.456 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.456 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.456 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:26:33.456 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:06.058048+0000", 2024-09-13T11:26:33.456 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.3170000000000008e-06 2024-09-13T11:26:33.457 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.457 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.457 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:26:33.457 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:06.058059+0000", 2024-09-13T11:26:33.457 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.1479e-05 2024-09-13T11:26:33.457 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.457 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.457 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:26:33.457 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:06.058444+0000", 2024-09-13T11:26:33.457 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00038485700000000001 2024-09-13T11:26:33.457 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.457 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.457 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for peered", 2024-09-13T11:26:33.458 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:06.058471+0000", 2024-09-13T11:26:33.458 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.7172999999999999e-05 2024-09-13T11:26:33.458 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.458 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.458 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:26:33.458 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:06.282046+0000", 2024-09-13T11:26:33.458 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.22357485099999999 2024-09-13T11:26:33.458 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.458 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.458 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:26:33.458 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:06.282204+0000", 2024-09-13T11:26:33.458 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000158414 2024-09-13T11:26:33.459 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.459 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.459 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 3,14", 2024-09-13T11:26:33.459 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:06.282377+0000", 2024-09-13T11:26:33.459 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00017289900000000001 2024-09-13T11:26:33.459 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.459 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.459 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:33.459 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:06.295990+0000", 2024-09-13T11:26:33.459 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.013612406000000001 2024-09-13T11:26:33.459 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.459 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.460 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:26:33.460 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:06.296380+0000", 2024-09-13T11:26:33.460 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00039020599999999998 2024-09-13T11:26:33.460 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.460 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.460 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:33.460 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:06.298303+0000", 2024-09-13T11:26:33.460 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.001923258 2024-09-13T11:26:33.460 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.460 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.460 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:26:33.460 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:06.298343+0000", 2024-09-13T11:26:33.460 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.9783999999999997e-05 2024-09-13T11:26:33.460 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.461 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.461 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:26:33.461 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:06.298412+0000", 2024-09-13T11:26:33.461 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.8616999999999997e-05 2024-09-13T11:26:33.461 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:33.461 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:33.461 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:33.461 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.461 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.461 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:221503 3.15 3:ab531595:::benchmark_data_smithi177_31116_object81200:head [delete] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e193)", 2024-09-13T11:26:33.461 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:26:06.058069+0000", 2024-09-13T11:26:33.461 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 27.311376871, 2024-09-13T11:26:33.461 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.24966487400000001, 2024-09-13T11:26:33.462 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:26:33.462 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:26:33.462 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:33.462 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:26:33.462 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:26:33.462 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:33.462 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 221503 2024-09-13T11:26:33.462 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.462 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:26:33.462 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.462 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:26:33.462 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:06.058069+0000", 2024-09-13T11:26:33.462 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:33.462 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.463 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.463 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:26:33.463 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:06.058069+0000", 2024-09-13T11:26:33.463 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:33.463 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.463 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.463 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:26:33.463 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:06.058070+0000", 2024-09-13T11:26:33.463 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.7000000000000003e-07 2024-09-13T11:26:33.463 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.463 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.463 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:26:33.464 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:06.058074+0000", 2024-09-13T11:26:33.464 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.5070000000000002e-06 2024-09-13T11:26:33.464 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.464 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.464 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:26:33.464 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:06.058089+0000", 2024-09-13T11:26:33.464 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.4593000000000001e-05 2024-09-13T11:26:33.464 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.464 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.464 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:26:33.464 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:06.058102+0000", 2024-09-13T11:26:33.464 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.3349999999999999e-05 2024-09-13T11:26:33.464 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.465 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.465 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:26:33.465 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:06.058550+0000", 2024-09-13T11:26:33.465 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00044800299999999999 2024-09-13T11:26:33.465 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.465 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.465 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for peered", 2024-09-13T11:26:33.465 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:06.058560+0000", 2024-09-13T11:26:33.465 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.5200000000000003e-06 2024-09-13T11:26:33.465 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.465 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.466 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:26:33.466 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:06.286141+0000", 2024-09-13T11:26:33.466 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.22758104200000001 2024-09-13T11:26:33.466 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.466 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.466 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:26:33.466 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:06.286335+0000", 2024-09-13T11:26:33.466 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00019429700000000001 2024-09-13T11:26:33.466 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.466 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.466 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 3,14", 2024-09-13T11:26:33.466 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:06.286518+0000", 2024-09-13T11:26:33.466 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00018350500000000001 2024-09-13T11:26:33.466 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.467 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.467 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:33.467 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:06.296043+0000", 2024-09-13T11:26:33.467 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0095243600000000008 2024-09-13T11:26:33.467 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.467 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.467 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:26:33.467 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:06.296384+0000", 2024-09-13T11:26:33.467 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00034132100000000001 2024-09-13T11:26:33.467 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.467 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.467 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:33.467 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:06.307532+0000", 2024-09-13T11:26:33.467 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.011148025000000001 2024-09-13T11:26:33.468 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.468 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.468 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:26:33.468 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:06.307611+0000", 2024-09-13T11:26:33.468 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.8465000000000002e-05 2024-09-13T11:26:33.468 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.468 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.468 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:26:33.468 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:06.307734+0000", 2024-09-13T11:26:33.468 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00012301599999999999 2024-09-13T11:26:33.468 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:33.468 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:33.468 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:33.469 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.469 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.469 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:221528 3.15 3:afda25ed:::benchmark_data_smithi177_31116_object81225:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e193)", 2024-09-13T11:26:33.469 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:26:06.058152+0000", 2024-09-13T11:26:33.469 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 27.311294014000001, 2024-09-13T11:26:33.469 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.25716789899999998, 2024-09-13T11:26:33.469 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:26:33.469 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:26:33.469 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:33.469 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:26:33.469 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:26:33.469 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:33.469 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 221528 2024-09-13T11:26:33.470 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.470 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:26:33.470 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.470 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:26:33.470 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:06.058152+0000", 2024-09-13T11:26:33.470 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:33.470 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.470 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.470 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:26:33.470 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:06.058152+0000", 2024-09-13T11:26:33.470 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:33.470 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.470 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.470 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:26:33.470 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:06.058153+0000", 2024-09-13T11:26:33.471 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.773e-06 2024-09-13T11:26:33.471 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.471 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.471 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:26:33.471 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:06.058161+0000", 2024-09-13T11:26:33.471 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.9160000000000001e-06 2024-09-13T11:26:33.471 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.471 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.471 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:26:33.471 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:06.058186+0000", 2024-09-13T11:26:33.471 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.5037000000000001e-05 2024-09-13T11:26:33.471 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.471 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.472 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:26:33.472 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:06.058202+0000", 2024-09-13T11:26:33.472 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.5709000000000002e-05 2024-09-13T11:26:33.472 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.472 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.472 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:26:33.472 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:06.058640+0000", 2024-09-13T11:26:33.472 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00043771400000000002 2024-09-13T11:26:33.472 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.472 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.472 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for peered", 2024-09-13T11:26:33.472 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:06.058664+0000", 2024-09-13T11:26:33.472 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.385e-05 2024-09-13T11:26:33.473 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.473 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.473 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:26:33.473 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:06.291953+0000", 2024-09-13T11:26:33.473 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.23328986800000001 2024-09-13T11:26:33.473 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.473 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.473 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:26:33.473 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:06.292123+0000", 2024-09-13T11:26:33.473 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00016977600000000001 2024-09-13T11:26:33.473 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.473 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.473 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 3,14", 2024-09-13T11:26:33.474 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:06.292276+0000", 2024-09-13T11:26:33.474 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000153196 2024-09-13T11:26:33.474 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.474 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.474 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:26:33.474 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:06.296387+0000", 2024-09-13T11:26:33.474 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0041105459999999996 2024-09-13T11:26:33.474 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.474 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.474 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:33.474 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:06.307779+0000", 2024-09-13T11:26:33.474 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.011391712 2024-09-13T11:26:33.475 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.475 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.475 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:33.475 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:06.315151+0000", 2024-09-13T11:26:33.475 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0073723169999999998 2024-09-13T11:26:33.475 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.475 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.475 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:26:33.475 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:06.315219+0000", 2024-09-13T11:26:33.475 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.7984999999999994e-05 2024-09-13T11:26:33.475 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.475 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.475 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:26:33.475 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:06.315319+0000", 2024-09-13T11:26:33.475 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0001005 2024-09-13T11:26:33.476 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:33.476 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:33.476 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:33.476 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.476 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.476 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:243211 3.22 3:445e4152:::benchmark_data_smithi177_31116_object102908:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e207)", 2024-09-13T11:26:33.476 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:26:26.318899+0000", 2024-09-13T11:26:33.476 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 7.0505469439999997, 2024-09-13T11:26:33.476 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2078296390000001, 2024-09-13T11:26:33.476 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:26:33.476 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:26:33.476 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-13T11:26:33.476 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:26:33.477 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:26:33.477 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:33.477 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 243211 2024-09-13T11:26:33.477 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.477 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:26:33.477 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.477 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:26:33.477 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:26.318899+0000", 2024-09-13T11:26:33.477 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:33.477 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.477 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.477 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:26:33.477 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:26.318899+0000", 2024-09-13T11:26:33.477 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:33.478 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.478 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.478 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:26:33.478 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:26.318899+0000", 2024-09-13T11:26:33.478 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.7199999999999999e-07 2024-09-13T11:26:33.478 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.478 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.478 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:26:33.478 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:26.318902+0000", 2024-09-13T11:26:33.478 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.616e-06 2024-09-13T11:26:33.478 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.478 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.478 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:26:33.479 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:26.318906+0000", 2024-09-13T11:26:33.479 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.2440000000000004e-06 2024-09-13T11:26:33.479 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.479 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.479 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:26:33.479 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:26.318909+0000", 2024-09-13T11:26:33.479 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.9830000000000001e-06 2024-09-13T11:26:33.479 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.479 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.479 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:26:33.479 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:26.319101+0000", 2024-09-13T11:26:33.479 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00019198800000000001 2024-09-13T11:26:33.479 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.479 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.480 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for missing object", 2024-09-13T11:26:33.480 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:26.319166+0000", 2024-09-13T11:26:33.480 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.4906999999999994e-05 2024-09-13T11:26:33.480 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.480 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.480 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:26:33.480 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:27.526716+0000", 2024-09-13T11:26:33.480 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.207550323 2024-09-13T11:26:33.480 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.480 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.480 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:26:33.480 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:27.526728+0000", 2024-09-13T11:26:33.480 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2006e-05 2024-09-13T11:26:33.480 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:33.481 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:33.481 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:33.481 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:33.481 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:33.481 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-13T11:26:33.483 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-13T11:26:33.483 INFO:tasks.ceph.osd.12.smithi001.stderr:2024-09-13T11:26:33.478+0000 7f892c6e6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26314) UID: 0 2024-09-13T11:26:33.583 INFO:tasks.ceph.osd.11.smithi177.stderr:2024-09-13T11:26:33.581+0000 7f24792c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26580) UID: 0 2024-09-13T11:26:33.649 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-13T11:26:33.649 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-13T11:26:33.649 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 211, 2024-09-13T11:26:33.649 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "2fd68ac7-5581-4d90-80a4-30bcf5252a7a", 2024-09-13T11:26:33.649 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-09-13T11:21:57.223153+0000", 2024-09-13T11:26:33.649 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-09-13T11:26:32.212257+0000", 2024-09-13T11:26:33.649 INFO:teuthology.orchestra.run.smithi001.stdout: "last_up_change": "2024-09-13T11:24:55.968636+0000", 2024-09-13T11:26:33.650 INFO:teuthology.orchestra.run.smithi001.stdout: "last_in_change": "2024-09-13T11:26:04.007363+0000", 2024-09-13T11:26:33.650 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": "nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-13T11:26:33.650 INFO:teuthology.orchestra.run.smithi001.stdout: "flags_num": 5804032, 2024-09-13T11:26:33.650 INFO:teuthology.orchestra.run.smithi001.stdout: "flags_set": [ 2024-09-13T11:26:33.650 INFO:teuthology.orchestra.run.smithi001.stdout: "nodeep-scrub", 2024-09-13T11:26:33.650 INFO:teuthology.orchestra.run.smithi001.stdout: "pglog_hardlimit", 2024-09-13T11:26:33.650 INFO:teuthology.orchestra.run.smithi001.stdout: "purged_snapdirs", 2024-09-13T11:26:33.650 INFO:teuthology.orchestra.run.smithi001.stdout: "recovery_deletes", 2024-09-13T11:26:33.650 INFO:teuthology.orchestra.run.smithi001.stdout: "sortbitwise" 2024-09-13T11:26:33.650 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-09-13T11:26:33.650 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_version": 7, 2024-09-13T11:26:33.650 INFO:teuthology.orchestra.run.smithi001.stdout: "full_ratio": 0.94999998807907104, 2024-09-13T11:26:33.650 INFO:teuthology.orchestra.run.smithi001.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-09-13T11:26:33.651 INFO:teuthology.orchestra.run.smithi001.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-13T11:26:33.651 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_snapshot": "", 2024-09-13T11:26:33.651 INFO:teuthology.orchestra.run.smithi001.stdout: "pool_max": 3, 2024-09-13T11:26:33.651 INFO:teuthology.orchestra.run.smithi001.stdout: "max_osd": 16, 2024-09-13T11:26:33.651 INFO:teuthology.orchestra.run.smithi001.stdout: "require_min_compat_client": "luminous", 2024-09-13T11:26:33.651 INFO:teuthology.orchestra.run.smithi001.stdout: "min_compat_client": "luminous", 2024-09-13T11:26:33.651 INFO:teuthology.orchestra.run.smithi001.stdout: "require_osd_release": "squid", 2024-09-13T11:26:33.651 INFO:teuthology.orchestra.run.smithi001.stdout: "allow_crimson": false, 2024-09-13T11:26:33.651 INFO:teuthology.orchestra.run.smithi001.stdout: "pools": [ 2024-09-13T11:26:33.651 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.651 INFO:teuthology.orchestra.run.smithi001.stdout: "pool": 1, 2024-09-13T11:26:33.651 INFO:teuthology.orchestra.run.smithi001.stdout: "pool_name": ".mgr", 2024-09-13T11:26:33.651 INFO:teuthology.orchestra.run.smithi001.stdout: "create_time": "2024-09-13T11:22:10.162266+0000", 2024-09-13T11:26:33.652 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-09-13T11:26:33.652 INFO:teuthology.orchestra.run.smithi001.stdout: "flags_names": "hashpspool", 2024-09-13T11:26:33.652 INFO:teuthology.orchestra.run.smithi001.stdout: "type": 1, 2024-09-13T11:26:33.652 INFO:teuthology.orchestra.run.smithi001.stdout: "size": 2, 2024-09-13T11:26:33.652 INFO:teuthology.orchestra.run.smithi001.stdout: "min_size": 2, 2024-09-13T11:26:33.652 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_rule": 0, 2024-09-13T11:26:33.652 INFO:teuthology.orchestra.run.smithi001.stdout: "peering_crush_bucket_count": 0, 2024-09-13T11:26:33.652 INFO:teuthology.orchestra.run.smithi001.stdout: "peering_crush_bucket_target": 0, 2024-09-13T11:26:33.652 INFO:teuthology.orchestra.run.smithi001.stdout: "peering_crush_bucket_barrier": 0, 2024-09-13T11:26:33.652 INFO:teuthology.orchestra.run.smithi001.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-13T11:26:33.652 INFO:teuthology.orchestra.run.smithi001.stdout: "is_stretch_pool": false, 2024-09-13T11:26:33.652 INFO:teuthology.orchestra.run.smithi001.stdout: "object_hash": 2, 2024-09-13T11:26:33.652 INFO:teuthology.orchestra.run.smithi001.stdout: "pg_autoscale_mode": "off", 2024-09-13T11:26:33.652 INFO:teuthology.orchestra.run.smithi001.stdout: "pg_num": 1, 2024-09-13T11:26:33.653 INFO:teuthology.orchestra.run.smithi001.stdout: "pg_placement_num": 1, 2024-09-13T11:26:33.653 INFO:teuthology.orchestra.run.smithi001.stdout: "pg_placement_num_target": 1, 2024-09-13T11:26:33.653 INFO:teuthology.orchestra.run.smithi001.stdout: "pg_num_target": 1, 2024-09-13T11:26:33.653 INFO:teuthology.orchestra.run.smithi001.stdout: "pg_num_pending": 1, 2024-09-13T11:26:33.653 INFO:teuthology.orchestra.run.smithi001.stdout: "last_pg_merge_meta": { 2024-09-13T11:26:33.653 INFO:teuthology.orchestra.run.smithi001.stdout: "source_pgid": "0.0", 2024-09-13T11:26:33.653 INFO:teuthology.orchestra.run.smithi001.stdout: "ready_epoch": 0, 2024-09-13T11:26:33.653 INFO:teuthology.orchestra.run.smithi001.stdout: "last_epoch_started": 0, 2024-09-13T11:26:33.653 INFO:teuthology.orchestra.run.smithi001.stdout: "last_epoch_clean": 0, 2024-09-13T11:26:33.653 INFO:teuthology.orchestra.run.smithi001.stdout: "source_version": "0'0", 2024-09-13T11:26:33.653 INFO:teuthology.orchestra.run.smithi001.stdout: "target_version": "0'0" 2024-09-13T11:26:33.653 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.653 INFO:teuthology.orchestra.run.smithi001.stdout: "last_change": "25", 2024-09-13T11:26:33.654 INFO:teuthology.orchestra.run.smithi001.stdout: "last_force_op_resend": "0", 2024-09-13T11:26:33.654 INFO:teuthology.orchestra.run.smithi001.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-13T11:26:33.654 INFO:teuthology.orchestra.run.smithi001.stdout: "last_force_op_resend_preluminous": "0", 2024-09-13T11:26:33.654 INFO:teuthology.orchestra.run.smithi001.stdout: "auid": 0, 2024-09-13T11:26:33.654 INFO:teuthology.orchestra.run.smithi001.stdout: "snap_mode": "selfmanaged", 2024-09-13T11:26:33.654 INFO:teuthology.orchestra.run.smithi001.stdout: "snap_seq": 0, 2024-09-13T11:26:33.654 INFO:teuthology.orchestra.run.smithi001.stdout: "snap_epoch": 0, 2024-09-13T11:26:33.654 INFO:teuthology.orchestra.run.smithi001.stdout: "pool_snaps": [], 2024-09-13T11:26:33.654 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_snaps": "[]", 2024-09-13T11:26:33.654 INFO:teuthology.orchestra.run.smithi001.stdout: "quota_max_bytes": 0, 2024-09-13T11:26:33.654 INFO:teuthology.orchestra.run.smithi001.stdout: "quota_max_objects": 0, 2024-09-13T11:26:33.654 INFO:teuthology.orchestra.run.smithi001.stdout: "tiers": [], 2024-09-13T11:26:33.655 INFO:teuthology.orchestra.run.smithi001.stdout: "tier_of": -1, 2024-09-13T11:26:33.655 INFO:teuthology.orchestra.run.smithi001.stdout: "read_tier": -1, 2024-09-13T11:26:33.655 INFO:teuthology.orchestra.run.smithi001.stdout: "write_tier": -1, 2024-09-13T11:26:33.655 INFO:teuthology.orchestra.run.smithi001.stdout: "cache_mode": "none", 2024-09-13T11:26:33.655 INFO:teuthology.orchestra.run.smithi001.stdout: "target_max_bytes": 0, 2024-09-13T11:26:33.655 INFO:teuthology.orchestra.run.smithi001.stdout: "target_max_objects": 0, 2024-09-13T11:26:33.655 INFO:teuthology.orchestra.run.smithi001.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-13T11:26:33.655 INFO:teuthology.orchestra.run.smithi001.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-13T11:26:33.655 INFO:teuthology.orchestra.run.smithi001.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-13T11:26:33.655 INFO:teuthology.orchestra.run.smithi001.stdout: "cache_min_flush_age": 0, 2024-09-13T11:26:33.655 INFO:teuthology.orchestra.run.smithi001.stdout: "cache_min_evict_age": 0, 2024-09-13T11:26:33.655 INFO:teuthology.orchestra.run.smithi001.stdout: "erasure_code_profile": "", 2024-09-13T11:26:33.656 INFO:teuthology.orchestra.run.smithi001.stdout: "hit_set_params": { 2024-09-13T11:26:33.656 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "none" 2024-09-13T11:26:33.656 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.656 INFO:teuthology.orchestra.run.smithi001.stdout: "hit_set_period": 0, 2024-09-13T11:26:33.656 INFO:teuthology.orchestra.run.smithi001.stdout: "hit_set_count": 0, 2024-09-13T11:26:33.656 INFO:teuthology.orchestra.run.smithi001.stdout: "use_gmt_hitset": true, 2024-09-13T11:26:33.656 INFO:teuthology.orchestra.run.smithi001.stdout: "min_read_recency_for_promote": 0, 2024-09-13T11:26:33.656 INFO:teuthology.orchestra.run.smithi001.stdout: "min_write_recency_for_promote": 0, 2024-09-13T11:26:33.656 INFO:teuthology.orchestra.run.smithi001.stdout: "hit_set_grade_decay_rate": 0, 2024-09-13T11:26:33.656 INFO:teuthology.orchestra.run.smithi001.stdout: "hit_set_search_last_n": 0, 2024-09-13T11:26:33.656 INFO:teuthology.orchestra.run.smithi001.stdout: "grade_table": [], 2024-09-13T11:26:33.656 INFO:teuthology.orchestra.run.smithi001.stdout: "stripe_width": 0, 2024-09-13T11:26:33.656 INFO:teuthology.orchestra.run.smithi001.stdout: "expected_num_objects": 0, 2024-09-13T11:26:33.657 INFO:teuthology.orchestra.run.smithi001.stdout: "fast_read": false, 2024-09-13T11:26:33.657 INFO:teuthology.orchestra.run.smithi001.stdout: "options": { 2024-09-13T11:26:33.657 INFO:teuthology.orchestra.run.smithi001.stdout: "pg_num_max": 32, 2024-09-13T11:26:33.657 INFO:teuthology.orchestra.run.smithi001.stdout: "pg_num_min": 1 2024-09-13T11:26:33.657 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.657 INFO:teuthology.orchestra.run.smithi001.stdout: "application_metadata": { 2024-09-13T11:26:33.657 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": {} 2024-09-13T11:26:33.657 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.657 INFO:teuthology.orchestra.run.smithi001.stdout: "read_balance": { 2024-09-13T11:26:33.657 INFO:teuthology.orchestra.run.smithi001.stdout: "score_type": "Fair distribution", 2024-09-13T11:26:33.657 INFO:teuthology.orchestra.run.smithi001.stdout: "score_acting": 15.380000114440918, 2024-09-13T11:26:33.657 INFO:teuthology.orchestra.run.smithi001.stdout: "score_stable": 15.380000114440918, 2024-09-13T11:26:33.657 INFO:teuthology.orchestra.run.smithi001.stdout: "optimal_score": 0.12999999523162842, 2024-09-13T11:26:33.658 INFO:teuthology.orchestra.run.smithi001.stdout: "raw_score_acting": 2, 2024-09-13T11:26:33.658 INFO:teuthology.orchestra.run.smithi001.stdout: "raw_score_stable": 2, 2024-09-13T11:26:33.658 INFO:teuthology.orchestra.run.smithi001.stdout: "primary_affinity_weighted": 0.9375, 2024-09-13T11:26:33.658 INFO:teuthology.orchestra.run.smithi001.stdout: "average_primary_affinity": 1, 2024-09-13T11:26:33.658 INFO:teuthology.orchestra.run.smithi001.stdout: "average_primary_affinity_weighted": 0.93999999761581421 2024-09-13T11:26:33.658 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:33.658 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.658 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.658 INFO:teuthology.orchestra.run.smithi001.stdout: "pool": 2, 2024-09-13T11:26:33.658 INFO:teuthology.orchestra.run.smithi001.stdout: "pool_name": "rbd", 2024-09-13T11:26:33.658 INFO:teuthology.orchestra.run.smithi001.stdout: "create_time": "2024-09-13T11:22:12.235549+0000", 2024-09-13T11:26:33.658 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 8193, 2024-09-13T11:26:33.659 INFO:teuthology.orchestra.run.smithi001.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-09-13T11:26:33.659 INFO:teuthology.orchestra.run.smithi001.stdout: "type": 1, 2024-09-13T11:26:33.659 INFO:teuthology.orchestra.run.smithi001.stdout: "size": 2, 2024-09-13T11:26:33.659 INFO:teuthology.orchestra.run.smithi001.stdout: "min_size": 2, 2024-09-13T11:26:33.659 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_rule": 0, 2024-09-13T11:26:33.659 INFO:teuthology.orchestra.run.smithi001.stdout: "peering_crush_bucket_count": 0, 2024-09-13T11:26:33.659 INFO:teuthology.orchestra.run.smithi001.stdout: "peering_crush_bucket_target": 0, 2024-09-13T11:26:33.659 INFO:teuthology.orchestra.run.smithi001.stdout: "peering_crush_bucket_barrier": 0, 2024-09-13T11:26:33.659 INFO:teuthology.orchestra.run.smithi001.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-13T11:26:33.659 INFO:teuthology.orchestra.run.smithi001.stdout: "is_stretch_pool": false, 2024-09-13T11:26:33.659 INFO:teuthology.orchestra.run.smithi001.stdout: "object_hash": 2, 2024-09-13T11:26:33.659 INFO:teuthology.orchestra.run.smithi001.stdout: "pg_autoscale_mode": "off", 2024-09-13T11:26:33.659 INFO:teuthology.orchestra.run.smithi001.stdout: "pg_num": 8, 2024-09-13T11:26:33.660 INFO:teuthology.orchestra.run.smithi001.stdout: "pg_placement_num": 8, 2024-09-13T11:26:33.660 INFO:teuthology.orchestra.run.smithi001.stdout: "pg_placement_num_target": 8, 2024-09-13T11:26:33.660 INFO:teuthology.orchestra.run.smithi001.stdout: "pg_num_target": 8, 2024-09-13T11:26:33.660 INFO:teuthology.orchestra.run.smithi001.stdout: "pg_num_pending": 8, 2024-09-13T11:26:33.660 INFO:teuthology.orchestra.run.smithi001.stdout: "last_pg_merge_meta": { 2024-09-13T11:26:33.660 INFO:teuthology.orchestra.run.smithi001.stdout: "source_pgid": "0.0", 2024-09-13T11:26:33.660 INFO:teuthology.orchestra.run.smithi001.stdout: "ready_epoch": 0, 2024-09-13T11:26:33.660 INFO:teuthology.orchestra.run.smithi001.stdout: "last_epoch_started": 0, 2024-09-13T11:26:33.660 INFO:teuthology.orchestra.run.smithi001.stdout: "last_epoch_clean": 0, 2024-09-13T11:26:33.660 INFO:teuthology.orchestra.run.smithi001.stdout: "source_version": "0'0", 2024-09-13T11:26:33.660 INFO:teuthology.orchestra.run.smithi001.stdout: "target_version": "0'0" 2024-09-13T11:26:33.660 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.660 INFO:teuthology.orchestra.run.smithi001.stdout: "last_change": "28", 2024-09-13T11:26:33.661 INFO:teuthology.orchestra.run.smithi001.stdout: "last_force_op_resend": "0", 2024-09-13T11:26:33.661 INFO:teuthology.orchestra.run.smithi001.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-13T11:26:33.661 INFO:teuthology.orchestra.run.smithi001.stdout: "last_force_op_resend_preluminous": "0", 2024-09-13T11:26:33.661 INFO:teuthology.orchestra.run.smithi001.stdout: "auid": 0, 2024-09-13T11:26:33.661 INFO:teuthology.orchestra.run.smithi001.stdout: "snap_mode": "selfmanaged", 2024-09-13T11:26:33.661 INFO:teuthology.orchestra.run.smithi001.stdout: "snap_seq": 2, 2024-09-13T11:26:33.661 INFO:teuthology.orchestra.run.smithi001.stdout: "snap_epoch": 28, 2024-09-13T11:26:33.661 INFO:teuthology.orchestra.run.smithi001.stdout: "pool_snaps": [], 2024-09-13T11:26:33.661 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_snaps": "[]", 2024-09-13T11:26:33.661 INFO:teuthology.orchestra.run.smithi001.stdout: "quota_max_bytes": 0, 2024-09-13T11:26:33.661 INFO:teuthology.orchestra.run.smithi001.stdout: "quota_max_objects": 0, 2024-09-13T11:26:33.662 INFO:teuthology.orchestra.run.smithi001.stdout: "tiers": [], 2024-09-13T11:26:33.662 INFO:teuthology.orchestra.run.smithi001.stdout: "tier_of": -1, 2024-09-13T11:26:33.662 INFO:teuthology.orchestra.run.smithi001.stdout: "read_tier": -1, 2024-09-13T11:26:33.662 INFO:teuthology.orchestra.run.smithi001.stdout: "write_tier": -1, 2024-09-13T11:26:33.662 INFO:teuthology.orchestra.run.smithi001.stdout: "cache_mode": "none", 2024-09-13T11:26:33.662 INFO:teuthology.orchestra.run.smithi001.stdout: "target_max_bytes": 0, 2024-09-13T11:26:33.662 INFO:teuthology.orchestra.run.smithi001.stdout: "target_max_objects": 0, 2024-09-13T11:26:33.662 INFO:teuthology.orchestra.run.smithi001.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-13T11:26:33.662 INFO:teuthology.orchestra.run.smithi001.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-13T11:26:33.662 INFO:teuthology.orchestra.run.smithi001.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-13T11:26:33.662 INFO:teuthology.orchestra.run.smithi001.stdout: "cache_min_flush_age": 0, 2024-09-13T11:26:33.662 INFO:teuthology.orchestra.run.smithi001.stdout: "cache_min_evict_age": 0, 2024-09-13T11:26:33.662 INFO:teuthology.orchestra.run.smithi001.stdout: "erasure_code_profile": "", 2024-09-13T11:26:33.663 INFO:teuthology.orchestra.run.smithi001.stdout: "hit_set_params": { 2024-09-13T11:26:33.663 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "none" 2024-09-13T11:26:33.663 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.663 INFO:teuthology.orchestra.run.smithi001.stdout: "hit_set_period": 0, 2024-09-13T11:26:33.663 INFO:teuthology.orchestra.run.smithi001.stdout: "hit_set_count": 0, 2024-09-13T11:26:33.663 INFO:teuthology.orchestra.run.smithi001.stdout: "use_gmt_hitset": true, 2024-09-13T11:26:33.663 INFO:teuthology.orchestra.run.smithi001.stdout: "min_read_recency_for_promote": 0, 2024-09-13T11:26:33.663 INFO:teuthology.orchestra.run.smithi001.stdout: "min_write_recency_for_promote": 0, 2024-09-13T11:26:33.663 INFO:teuthology.orchestra.run.smithi001.stdout: "hit_set_grade_decay_rate": 0, 2024-09-13T11:26:33.663 INFO:teuthology.orchestra.run.smithi001.stdout: "hit_set_search_last_n": 0, 2024-09-13T11:26:33.663 INFO:teuthology.orchestra.run.smithi001.stdout: "grade_table": [], 2024-09-13T11:26:33.663 INFO:teuthology.orchestra.run.smithi001.stdout: "stripe_width": 0, 2024-09-13T11:26:33.663 INFO:teuthology.orchestra.run.smithi001.stdout: "expected_num_objects": 0, 2024-09-13T11:26:33.664 INFO:teuthology.orchestra.run.smithi001.stdout: "fast_read": false, 2024-09-13T11:26:33.664 INFO:teuthology.orchestra.run.smithi001.stdout: "options": {}, 2024-09-13T11:26:33.664 INFO:teuthology.orchestra.run.smithi001.stdout: "application_metadata": { 2024-09-13T11:26:33.664 INFO:teuthology.orchestra.run.smithi001.stdout: "rbd": {} 2024-09-13T11:26:33.664 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.664 INFO:teuthology.orchestra.run.smithi001.stdout: "read_balance": { 2024-09-13T11:26:33.664 INFO:teuthology.orchestra.run.smithi001.stdout: "score_type": "Fair distribution", 2024-09-13T11:26:33.664 INFO:teuthology.orchestra.run.smithi001.stdout: "score_acting": 3.7300000190734863, 2024-09-13T11:26:33.664 INFO:teuthology.orchestra.run.smithi001.stdout: "score_stable": 3.7300000190734863, 2024-09-13T11:26:33.664 INFO:teuthology.orchestra.run.smithi001.stdout: "optimal_score": 0.67000001668930054, 2024-09-13T11:26:33.664 INFO:teuthology.orchestra.run.smithi001.stdout: "raw_score_acting": 2.5, 2024-09-13T11:26:33.664 INFO:teuthology.orchestra.run.smithi001.stdout: "raw_score_stable": 2.5, 2024-09-13T11:26:33.664 INFO:teuthology.orchestra.run.smithi001.stdout: "primary_affinity_weighted": 0.9375, 2024-09-13T11:26:33.665 INFO:teuthology.orchestra.run.smithi001.stdout: "average_primary_affinity": 1, 2024-09-13T11:26:33.665 INFO:teuthology.orchestra.run.smithi001.stdout: "average_primary_affinity_weighted": 0.93999999761581421 2024-09-13T11:26:33.665 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:33.665 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.665 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.665 INFO:teuthology.orchestra.run.smithi001.stdout: "pool": 3, 2024-09-13T11:26:33.665 INFO:teuthology.orchestra.run.smithi001.stdout: "pool_name": "unique_pool_0", 2024-09-13T11:26:33.665 INFO:teuthology.orchestra.run.smithi001.stdout: "create_time": "2024-09-13T11:22:24.305521+0000", 2024-09-13T11:26:33.665 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-09-13T11:26:33.665 INFO:teuthology.orchestra.run.smithi001.stdout: "flags_names": "hashpspool", 2024-09-13T11:26:33.665 INFO:teuthology.orchestra.run.smithi001.stdout: "type": 1, 2024-09-13T11:26:33.665 INFO:teuthology.orchestra.run.smithi001.stdout: "size": 2, 2024-09-13T11:26:33.665 INFO:teuthology.orchestra.run.smithi001.stdout: "min_size": 2, 2024-09-13T11:26:33.666 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_rule": 0, 2024-09-13T11:26:33.666 INFO:teuthology.orchestra.run.smithi001.stdout: "peering_crush_bucket_count": 0, 2024-09-13T11:26:33.666 INFO:teuthology.orchestra.run.smithi001.stdout: "peering_crush_bucket_target": 0, 2024-09-13T11:26:33.666 INFO:teuthology.orchestra.run.smithi001.stdout: "peering_crush_bucket_barrier": 0, 2024-09-13T11:26:33.666 INFO:teuthology.orchestra.run.smithi001.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-13T11:26:33.666 INFO:teuthology.orchestra.run.smithi001.stdout: "is_stretch_pool": false, 2024-09-13T11:26:33.666 INFO:teuthology.orchestra.run.smithi001.stdout: "object_hash": 2, 2024-09-13T11:26:33.666 INFO:teuthology.orchestra.run.smithi001.stdout: "pg_autoscale_mode": "off", 2024-09-13T11:26:33.666 INFO:teuthology.orchestra.run.smithi001.stdout: "pg_num": 46, 2024-09-13T11:26:33.666 INFO:teuthology.orchestra.run.smithi001.stdout: "pg_placement_num": 46, 2024-09-13T11:26:33.666 INFO:teuthology.orchestra.run.smithi001.stdout: "pg_placement_num_target": 46, 2024-09-13T11:26:33.666 INFO:teuthology.orchestra.run.smithi001.stdout: "pg_num_target": 46, 2024-09-13T11:26:33.666 INFO:teuthology.orchestra.run.smithi001.stdout: "pg_num_pending": 46, 2024-09-13T11:26:33.666 INFO:teuthology.orchestra.run.smithi001.stdout: "last_pg_merge_meta": { 2024-09-13T11:26:33.667 INFO:teuthology.orchestra.run.smithi001.stdout: "source_pgid": "0.0", 2024-09-13T11:26:33.667 INFO:teuthology.orchestra.run.smithi001.stdout: "ready_epoch": 0, 2024-09-13T11:26:33.667 INFO:teuthology.orchestra.run.smithi001.stdout: "last_epoch_started": 0, 2024-09-13T11:26:33.667 INFO:teuthology.orchestra.run.smithi001.stdout: "last_epoch_clean": 0, 2024-09-13T11:26:33.667 INFO:teuthology.orchestra.run.smithi001.stdout: "source_version": "0'0", 2024-09-13T11:26:33.667 INFO:teuthology.orchestra.run.smithi001.stdout: "target_version": "0'0" 2024-09-13T11:26:33.667 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.667 INFO:teuthology.orchestra.run.smithi001.stdout: "last_change": "196", 2024-09-13T11:26:33.667 INFO:teuthology.orchestra.run.smithi001.stdout: "last_force_op_resend": "0", 2024-09-13T11:26:33.667 INFO:teuthology.orchestra.run.smithi001.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-13T11:26:33.667 INFO:teuthology.orchestra.run.smithi001.stdout: "last_force_op_resend_preluminous": "117", 2024-09-13T11:26:33.667 INFO:teuthology.orchestra.run.smithi001.stdout: "auid": 0, 2024-09-13T11:26:33.667 INFO:teuthology.orchestra.run.smithi001.stdout: "snap_mode": "selfmanaged", 2024-09-13T11:26:33.668 INFO:teuthology.orchestra.run.smithi001.stdout: "snap_seq": 0, 2024-09-13T11:26:33.668 INFO:teuthology.orchestra.run.smithi001.stdout: "snap_epoch": 0, 2024-09-13T11:26:33.668 INFO:teuthology.orchestra.run.smithi001.stdout: "pool_snaps": [], 2024-09-13T11:26:33.668 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_snaps": "[]", 2024-09-13T11:26:33.668 INFO:teuthology.orchestra.run.smithi001.stdout: "quota_max_bytes": 0, 2024-09-13T11:26:33.668 INFO:teuthology.orchestra.run.smithi001.stdout: "quota_max_objects": 0, 2024-09-13T11:26:33.668 INFO:teuthology.orchestra.run.smithi001.stdout: "tiers": [], 2024-09-13T11:26:33.668 INFO:teuthology.orchestra.run.smithi001.stdout: "tier_of": -1, 2024-09-13T11:26:33.668 INFO:teuthology.orchestra.run.smithi001.stdout: "read_tier": -1, 2024-09-13T11:26:33.668 INFO:teuthology.orchestra.run.smithi001.stdout: "write_tier": -1, 2024-09-13T11:26:33.668 INFO:teuthology.orchestra.run.smithi001.stdout: "cache_mode": "none", 2024-09-13T11:26:33.668 INFO:teuthology.orchestra.run.smithi001.stdout: "target_max_bytes": 0, 2024-09-13T11:26:33.668 INFO:teuthology.orchestra.run.smithi001.stdout: "target_max_objects": 0, 2024-09-13T11:26:33.669 INFO:teuthology.orchestra.run.smithi001.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-13T11:26:33.669 INFO:teuthology.orchestra.run.smithi001.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-13T11:26:33.669 INFO:teuthology.orchestra.run.smithi001.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-13T11:26:33.669 INFO:teuthology.orchestra.run.smithi001.stdout: "cache_min_flush_age": 0, 2024-09-13T11:26:33.669 INFO:teuthology.orchestra.run.smithi001.stdout: "cache_min_evict_age": 0, 2024-09-13T11:26:33.669 INFO:teuthology.orchestra.run.smithi001.stdout: "erasure_code_profile": "", 2024-09-13T11:26:33.669 INFO:teuthology.orchestra.run.smithi001.stdout: "hit_set_params": { 2024-09-13T11:26:33.669 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "none" 2024-09-13T11:26:33.669 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.669 INFO:teuthology.orchestra.run.smithi001.stdout: "hit_set_period": 0, 2024-09-13T11:26:33.669 INFO:teuthology.orchestra.run.smithi001.stdout: "hit_set_count": 0, 2024-09-13T11:26:33.669 INFO:teuthology.orchestra.run.smithi001.stdout: "use_gmt_hitset": true, 2024-09-13T11:26:33.669 INFO:teuthology.orchestra.run.smithi001.stdout: "min_read_recency_for_promote": 0, 2024-09-13T11:26:33.669 INFO:teuthology.orchestra.run.smithi001.stdout: "min_write_recency_for_promote": 0, 2024-09-13T11:26:33.670 INFO:teuthology.orchestra.run.smithi001.stdout: "hit_set_grade_decay_rate": 0, 2024-09-13T11:26:33.670 INFO:teuthology.orchestra.run.smithi001.stdout: "hit_set_search_last_n": 0, 2024-09-13T11:26:33.670 INFO:teuthology.orchestra.run.smithi001.stdout: "grade_table": [], 2024-09-13T11:26:33.670 INFO:teuthology.orchestra.run.smithi001.stdout: "stripe_width": 0, 2024-09-13T11:26:33.670 INFO:teuthology.orchestra.run.smithi001.stdout: "expected_num_objects": 0, 2024-09-13T11:26:33.670 INFO:teuthology.orchestra.run.smithi001.stdout: "fast_read": false, 2024-09-13T11:26:33.670 INFO:teuthology.orchestra.run.smithi001.stdout: "options": {}, 2024-09-13T11:26:33.670 INFO:teuthology.orchestra.run.smithi001.stdout: "application_metadata": { 2024-09-13T11:26:33.670 INFO:teuthology.orchestra.run.smithi001.stdout: "rados": {} 2024-09-13T11:26:33.670 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.670 INFO:teuthology.orchestra.run.smithi001.stdout: "read_balance": { 2024-09-13T11:26:33.670 INFO:teuthology.orchestra.run.smithi001.stdout: "score_type": "Fair distribution", 2024-09-13T11:26:33.670 INFO:teuthology.orchestra.run.smithi001.stdout: "score_acting": 1.9500000476837158, 2024-09-13T11:26:33.671 INFO:teuthology.orchestra.run.smithi001.stdout: "score_stable": 1.9500000476837158, 2024-09-13T11:26:33.671 INFO:teuthology.orchestra.run.smithi001.stdout: "optimal_score": 1.0700000524520874, 2024-09-13T11:26:33.671 INFO:teuthology.orchestra.run.smithi001.stdout: "raw_score_acting": 2.0899999141693115, 2024-09-13T11:26:33.671 INFO:teuthology.orchestra.run.smithi001.stdout: "raw_score_stable": 2.0899999141693115, 2024-09-13T11:26:33.671 INFO:teuthology.orchestra.run.smithi001.stdout: "primary_affinity_weighted": 0.9375, 2024-09-13T11:26:33.671 INFO:teuthology.orchestra.run.smithi001.stdout: "average_primary_affinity": 1, 2024-09-13T11:26:33.671 INFO:teuthology.orchestra.run.smithi001.stdout: "average_primary_affinity_weighted": 0.93999999761581421 2024-09-13T11:26:33.671 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:33.671 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:33.671 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-09-13T11:26:33.671 INFO:teuthology.orchestra.run.smithi001.stdout: "osds": [ 2024-09-13T11:26:33.671 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.671 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": 0, 2024-09-13T11:26:33.672 INFO:teuthology.orchestra.run.smithi001.stdout: "uuid": "0fc12b3c-e0de-4546-b935-db77efe7c567", 2024-09-13T11:26:33.672 INFO:teuthology.orchestra.run.smithi001.stdout: "up": 1, 2024-09-13T11:26:33.672 INFO:teuthology.orchestra.run.smithi001.stdout: "in": 1, 2024-09-13T11:26:33.672 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 1, 2024-09-13T11:26:33.672 INFO:teuthology.orchestra.run.smithi001.stdout: "primary_affinity": 1, 2024-09-13T11:26:33.672 INFO:teuthology.orchestra.run.smithi001.stdout: "last_clean_begin": 0, 2024-09-13T11:26:33.672 INFO:teuthology.orchestra.run.smithi001.stdout: "last_clean_end": 0, 2024-09-13T11:26:33.672 INFO:teuthology.orchestra.run.smithi001.stdout: "up_from": 22, 2024-09-13T11:26:33.672 INFO:teuthology.orchestra.run.smithi001.stdout: "up_thru": 202, 2024-09-13T11:26:33.672 INFO:teuthology.orchestra.run.smithi001.stdout: "down_at": 0, 2024-09-13T11:26:33.672 INFO:teuthology.orchestra.run.smithi001.stdout: "lost_at": 0, 2024-09-13T11:26:33.672 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-09-13T11:26:33.672 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:26:33.672 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.673 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:26:33.673 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6800", 2024-09-13T11:26:33.673 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 1544776477 2024-09-13T11:26:33.673 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:33.673 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:33.673 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.673 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_addrs": { 2024-09-13T11:26:33.673 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:26:33.673 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.673 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:26:33.673 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6801", 2024-09-13T11:26:33.673 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 1544776477 2024-09-13T11:26:33.674 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:33.674 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:33.674 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.674 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_back_addrs": { 2024-09-13T11:26:33.674 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:26:33.674 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.674 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:26:33.674 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6803", 2024-09-13T11:26:33.674 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 1544776477 2024-09-13T11:26:33.674 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:33.674 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:33.674 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.674 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_front_addrs": { 2024-09-13T11:26:33.674 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:26:33.675 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.675 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:26:33.675 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6802", 2024-09-13T11:26:33.675 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 1544776477 2024-09-13T11:26:33.675 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:33.675 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:33.675 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.675 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6800/1544776477", 2024-09-13T11:26:33.675 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_addr": "172.21.15.1:6801/1544776477", 2024-09-13T11:26:33.675 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_back_addr": "172.21.15.1:6803/1544776477", 2024-09-13T11:26:33.675 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_front_addr": "172.21.15.1:6802/1544776477", 2024-09-13T11:26:33.675 INFO:teuthology.orchestra.run.smithi001.stdout: "state": [ 2024-09-13T11:26:33.675 INFO:teuthology.orchestra.run.smithi001.stdout: "exists", 2024-09-13T11:26:33.675 INFO:teuthology.orchestra.run.smithi001.stdout: "up" 2024-09-13T11:26:33.676 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:33.676 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.676 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.676 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": 1, 2024-09-13T11:26:33.676 INFO:teuthology.orchestra.run.smithi001.stdout: "uuid": "b77fe331-cc92-4adb-b146-af5c380f6d4f", 2024-09-13T11:26:33.676 INFO:teuthology.orchestra.run.smithi001.stdout: "up": 1, 2024-09-13T11:26:33.676 INFO:teuthology.orchestra.run.smithi001.stdout: "in": 1, 2024-09-13T11:26:33.676 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 1, 2024-09-13T11:26:33.676 INFO:teuthology.orchestra.run.smithi001.stdout: "primary_affinity": 1, 2024-09-13T11:26:33.676 INFO:teuthology.orchestra.run.smithi001.stdout: "last_clean_begin": 23, 2024-09-13T11:26:33.676 INFO:teuthology.orchestra.run.smithi001.stdout: "last_clean_end": 29, 2024-09-13T11:26:33.676 INFO:teuthology.orchestra.run.smithi001.stdout: "up_from": 40, 2024-09-13T11:26:33.676 INFO:teuthology.orchestra.run.smithi001.stdout: "up_thru": 171, 2024-09-13T11:26:33.677 INFO:teuthology.orchestra.run.smithi001.stdout: "down_at": 30, 2024-09-13T11:26:33.677 INFO:teuthology.orchestra.run.smithi001.stdout: "lost_at": 0, 2024-09-13T11:26:33.677 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-09-13T11:26:33.677 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:26:33.677 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.677 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:26:33.677 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.92:6800", 2024-09-13T11:26:33.677 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 3591430847 2024-09-13T11:26:33.677 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:33.677 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:33.677 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.677 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_addrs": { 2024-09-13T11:26:33.677 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:26:33.678 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.678 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:26:33.678 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.92:6801", 2024-09-13T11:26:33.678 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 3591430847 2024-09-13T11:26:33.678 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:33.678 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:33.678 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.678 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_back_addrs": { 2024-09-13T11:26:33.678 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:26:33.678 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.678 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:26:33.678 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.92:6803", 2024-09-13T11:26:33.678 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 3591430847 2024-09-13T11:26:33.678 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:33.679 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:33.679 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.679 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_front_addrs": { 2024-09-13T11:26:33.679 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:26:33.679 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.679 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:26:33.679 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.92:6802", 2024-09-13T11:26:33.679 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 3591430847 2024-09-13T11:26:33.679 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:33.679 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:33.679 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.679 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.92:6800/3591430847", 2024-09-13T11:26:33.679 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_addr": "172.21.15.92:6801/3591430847", 2024-09-13T11:26:33.680 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_back_addr": "172.21.15.92:6803/3591430847", 2024-09-13T11:26:33.680 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_front_addr": "172.21.15.92:6802/3591430847", 2024-09-13T11:26:33.680 INFO:teuthology.orchestra.run.smithi001.stdout: "state": [ 2024-09-13T11:26:33.680 INFO:teuthology.orchestra.run.smithi001.stdout: "exists", 2024-09-13T11:26:33.680 INFO:teuthology.orchestra.run.smithi001.stdout: "up" 2024-09-13T11:26:33.680 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:33.680 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.680 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.680 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": 2, 2024-09-13T11:26:33.680 INFO:teuthology.orchestra.run.smithi001.stdout: "uuid": "c9d031c8-e885-4229-affa-08e1729d00e8", 2024-09-13T11:26:33.680 INFO:teuthology.orchestra.run.smithi001.stdout: "up": 1, 2024-09-13T11:26:33.680 INFO:teuthology.orchestra.run.smithi001.stdout: "in": 1, 2024-09-13T11:26:33.680 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 1, 2024-09-13T11:26:33.681 INFO:teuthology.orchestra.run.smithi001.stdout: "primary_affinity": 1, 2024-09-13T11:26:33.681 INFO:teuthology.orchestra.run.smithi001.stdout: "last_clean_begin": 0, 2024-09-13T11:26:33.681 INFO:teuthology.orchestra.run.smithi001.stdout: "last_clean_end": 0, 2024-09-13T11:26:33.681 INFO:teuthology.orchestra.run.smithi001.stdout: "up_from": 22, 2024-09-13T11:26:33.681 INFO:teuthology.orchestra.run.smithi001.stdout: "up_thru": 152, 2024-09-13T11:26:33.681 INFO:teuthology.orchestra.run.smithi001.stdout: "down_at": 0, 2024-09-13T11:26:33.681 INFO:teuthology.orchestra.run.smithi001.stdout: "lost_at": 0, 2024-09-13T11:26:33.681 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-09-13T11:26:33.681 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:26:33.681 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.681 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:26:33.681 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.162:6812", 2024-09-13T11:26:33.681 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 2712148101 2024-09-13T11:26:33.682 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:33.682 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:33.682 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.682 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_addrs": { 2024-09-13T11:26:33.682 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:26:33.682 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.682 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:26:33.682 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.162:6813", 2024-09-13T11:26:33.682 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 2712148101 2024-09-13T11:26:33.682 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:33.682 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:33.682 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.682 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_back_addrs": { 2024-09-13T11:26:33.682 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:26:33.683 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.683 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:26:33.683 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.162:6815", 2024-09-13T11:26:33.683 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 2712148101 2024-09-13T11:26:33.683 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:33.683 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:33.683 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.683 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_front_addrs": { 2024-09-13T11:26:33.683 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:26:33.683 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.683 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:26:33.683 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.162:6814", 2024-09-13T11:26:33.683 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 2712148101 2024-09-13T11:26:33.683 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:33.684 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:33.684 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.684 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.162:6812/2712148101", 2024-09-13T11:26:33.684 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_addr": "172.21.15.162:6813/2712148101", 2024-09-13T11:26:33.684 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_back_addr": "172.21.15.162:6815/2712148101", 2024-09-13T11:26:33.684 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_front_addr": "172.21.15.162:6814/2712148101", 2024-09-13T11:26:33.684 INFO:teuthology.orchestra.run.smithi001.stdout: "state": [ 2024-09-13T11:26:33.684 INFO:teuthology.orchestra.run.smithi001.stdout: "exists", 2024-09-13T11:26:33.684 INFO:teuthology.orchestra.run.smithi001.stdout: "up" 2024-09-13T11:26:33.684 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:33.684 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.684 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.684 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": 3, 2024-09-13T11:26:33.685 INFO:teuthology.orchestra.run.smithi001.stdout: "uuid": "bf50d71a-6c3a-4fbc-a407-cb81bad14a7b", 2024-09-13T11:26:33.685 INFO:teuthology.orchestra.run.smithi001.stdout: "up": 1, 2024-09-13T11:26:33.685 INFO:teuthology.orchestra.run.smithi001.stdout: "in": 1, 2024-09-13T11:26:33.685 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 1, 2024-09-13T11:26:33.685 INFO:teuthology.orchestra.run.smithi001.stdout: "primary_affinity": 1, 2024-09-13T11:26:33.685 INFO:teuthology.orchestra.run.smithi001.stdout: "last_clean_begin": 0, 2024-09-13T11:26:33.685 INFO:teuthology.orchestra.run.smithi001.stdout: "last_clean_end": 0, 2024-09-13T11:26:33.685 INFO:teuthology.orchestra.run.smithi001.stdout: "up_from": 21, 2024-09-13T11:26:33.685 INFO:teuthology.orchestra.run.smithi001.stdout: "up_thru": 192, 2024-09-13T11:26:33.685 INFO:teuthology.orchestra.run.smithi001.stdout: "down_at": 0, 2024-09-13T11:26:33.685 INFO:teuthology.orchestra.run.smithi001.stdout: "lost_at": 0, 2024-09-13T11:26:33.685 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-09-13T11:26:33.685 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:26:33.685 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.686 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:26:33.686 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.177:6801", 2024-09-13T11:26:33.686 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 3367751690 2024-09-13T11:26:33.686 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:33.686 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:33.686 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.686 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_addrs": { 2024-09-13T11:26:33.686 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:26:33.686 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.686 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:26:33.686 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.177:6802", 2024-09-13T11:26:33.686 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 3367751690 2024-09-13T11:26:33.686 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:33.687 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:33.687 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.687 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_back_addrs": { 2024-09-13T11:26:33.687 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:26:33.687 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.687 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:26:33.687 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.177:6804", 2024-09-13T11:26:33.687 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 3367751690 2024-09-13T11:26:33.687 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:33.687 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:33.687 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.687 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_front_addrs": { 2024-09-13T11:26:33.687 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:26:33.687 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.688 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:26:33.688 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.177:6803", 2024-09-13T11:26:33.688 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 3367751690 2024-09-13T11:26:33.688 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:33.688 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:33.688 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.688 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.177:6801/3367751690", 2024-09-13T11:26:33.688 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_addr": "172.21.15.177:6802/3367751690", 2024-09-13T11:26:33.688 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_back_addr": "172.21.15.177:6804/3367751690", 2024-09-13T11:26:33.688 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_front_addr": "172.21.15.177:6803/3367751690", 2024-09-13T11:26:33.688 INFO:teuthology.orchestra.run.smithi001.stdout: "state": [ 2024-09-13T11:26:33.688 INFO:teuthology.orchestra.run.smithi001.stdout: "exists", 2024-09-13T11:26:33.688 INFO:teuthology.orchestra.run.smithi001.stdout: "up" 2024-09-13T11:26:33.689 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:33.689 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.689 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.689 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": 4, 2024-09-13T11:26:33.689 INFO:teuthology.orchestra.run.smithi001.stdout: "uuid": "d1fe3f63-df3f-4a73-a1e2-8d7c8a3f1fa0", 2024-09-13T11:26:33.689 INFO:teuthology.orchestra.run.smithi001.stdout: "up": 1, 2024-09-13T11:26:33.689 INFO:teuthology.orchestra.run.smithi001.stdout: "in": 1, 2024-09-13T11:26:33.689 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 1, 2024-09-13T11:26:33.689 INFO:teuthology.orchestra.run.smithi001.stdout: "primary_affinity": 1, 2024-09-13T11:26:33.689 INFO:teuthology.orchestra.run.smithi001.stdout: "last_clean_begin": 0, 2024-09-13T11:26:33.689 INFO:teuthology.orchestra.run.smithi001.stdout: "last_clean_end": 0, 2024-09-13T11:26:33.689 INFO:teuthology.orchestra.run.smithi001.stdout: "up_from": 22, 2024-09-13T11:26:33.689 INFO:teuthology.orchestra.run.smithi001.stdout: "up_thru": 105, 2024-09-13T11:26:33.689 INFO:teuthology.orchestra.run.smithi001.stdout: "down_at": 0, 2024-09-13T11:26:33.690 INFO:teuthology.orchestra.run.smithi001.stdout: "lost_at": 0, 2024-09-13T11:26:33.690 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-09-13T11:26:33.690 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:26:33.690 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.690 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:26:33.690 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6808", 2024-09-13T11:26:33.690 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 3648540926 2024-09-13T11:26:33.690 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:33.690 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:33.690 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.690 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_addrs": { 2024-09-13T11:26:33.690 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:26:33.690 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.690 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:26:33.691 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6809", 2024-09-13T11:26:33.691 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 3648540926 2024-09-13T11:26:33.691 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:33.691 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:33.691 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.691 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_back_addrs": { 2024-09-13T11:26:33.691 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:26:33.691 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.691 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:26:33.691 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6811", 2024-09-13T11:26:33.691 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 3648540926 2024-09-13T11:26:33.691 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:33.691 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:33.692 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.692 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_front_addrs": { 2024-09-13T11:26:33.692 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:26:33.692 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.692 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:26:33.692 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6810", 2024-09-13T11:26:33.692 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 3648540926 2024-09-13T11:26:33.692 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:33.692 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:33.692 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.692 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6808/3648540926", 2024-09-13T11:26:33.692 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_addr": "172.21.15.1:6809/3648540926", 2024-09-13T11:26:33.692 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_back_addr": "172.21.15.1:6811/3648540926", 2024-09-13T11:26:33.692 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_front_addr": "172.21.15.1:6810/3648540926", 2024-09-13T11:26:33.693 INFO:teuthology.orchestra.run.smithi001.stdout: "state": [ 2024-09-13T11:26:33.693 INFO:teuthology.orchestra.run.smithi001.stdout: "exists", 2024-09-13T11:26:33.693 INFO:teuthology.orchestra.run.smithi001.stdout: "up" 2024-09-13T11:26:33.693 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:33.693 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.693 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.693 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": 5, 2024-09-13T11:26:33.693 INFO:teuthology.orchestra.run.smithi001.stdout: "uuid": "e2ead447-2f2c-40c9-82e7-53b8dc41e2af", 2024-09-13T11:26:33.693 INFO:teuthology.orchestra.run.smithi001.stdout: "up": 1, 2024-09-13T11:26:33.693 INFO:teuthology.orchestra.run.smithi001.stdout: "in": 1, 2024-09-13T11:26:33.693 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 1, 2024-09-13T11:26:33.693 INFO:teuthology.orchestra.run.smithi001.stdout: "primary_affinity": 1, 2024-09-13T11:26:33.693 INFO:teuthology.orchestra.run.smithi001.stdout: "last_clean_begin": 0, 2024-09-13T11:26:33.694 INFO:teuthology.orchestra.run.smithi001.stdout: "last_clean_end": 0, 2024-09-13T11:26:33.694 INFO:teuthology.orchestra.run.smithi001.stdout: "up_from": 21, 2024-09-13T11:26:33.694 INFO:teuthology.orchestra.run.smithi001.stdout: "up_thru": 192, 2024-09-13T11:26:33.694 INFO:teuthology.orchestra.run.smithi001.stdout: "down_at": 0, 2024-09-13T11:26:33.694 INFO:teuthology.orchestra.run.smithi001.stdout: "lost_at": 0, 2024-09-13T11:26:33.694 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-09-13T11:26:33.694 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:26:33.694 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.694 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:26:33.694 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.92:6804", 2024-09-13T11:26:33.694 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 3399529091 2024-09-13T11:26:33.694 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:33.694 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:33.695 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.695 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_addrs": { 2024-09-13T11:26:33.695 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:26:33.695 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.695 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:26:33.695 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.92:6805", 2024-09-13T11:26:33.695 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 3399529091 2024-09-13T11:26:33.695 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:33.695 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:33.695 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.695 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_back_addrs": { 2024-09-13T11:26:33.695 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:26:33.695 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.695 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:26:33.696 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.92:6807", 2024-09-13T11:26:33.696 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 3399529091 2024-09-13T11:26:33.696 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:33.696 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:33.696 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.696 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_front_addrs": { 2024-09-13T11:26:33.696 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:26:33.696 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.696 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:26:33.696 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.92:6806", 2024-09-13T11:26:33.696 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 3399529091 2024-09-13T11:26:33.696 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:33.696 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:33.697 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.697 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.92:6804/3399529091", 2024-09-13T11:26:33.697 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_addr": "172.21.15.92:6805/3399529091", 2024-09-13T11:26:33.697 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_back_addr": "172.21.15.92:6807/3399529091", 2024-09-13T11:26:33.697 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_front_addr": "172.21.15.92:6806/3399529091", 2024-09-13T11:26:33.697 INFO:teuthology.orchestra.run.smithi001.stdout: "state": [ 2024-09-13T11:26:33.697 INFO:teuthology.orchestra.run.smithi001.stdout: "exists", 2024-09-13T11:26:33.697 INFO:teuthology.orchestra.run.smithi001.stdout: "up" 2024-09-13T11:26:33.697 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:33.697 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.697 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.697 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": 6, 2024-09-13T11:26:33.697 INFO:teuthology.orchestra.run.smithi001.stdout: "uuid": "dbc20439-144e-4ec7-89de-674eb783f263", 2024-09-13T11:26:33.698 INFO:teuthology.orchestra.run.smithi001.stdout: "up": 1, 2024-09-13T11:26:33.698 INFO:teuthology.orchestra.run.smithi001.stdout: "in": 1, 2024-09-13T11:26:33.698 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 1, 2024-09-13T11:26:33.698 INFO:teuthology.orchestra.run.smithi001.stdout: "primary_affinity": 1, 2024-09-13T11:26:33.698 INFO:teuthology.orchestra.run.smithi001.stdout: "last_clean_begin": 0, 2024-09-13T11:26:33.698 INFO:teuthology.orchestra.run.smithi001.stdout: "last_clean_end": 0, 2024-09-13T11:26:33.698 INFO:teuthology.orchestra.run.smithi001.stdout: "up_from": 22, 2024-09-13T11:26:33.698 INFO:teuthology.orchestra.run.smithi001.stdout: "up_thru": 117, 2024-09-13T11:26:33.698 INFO:teuthology.orchestra.run.smithi001.stdout: "down_at": 0, 2024-09-13T11:26:33.698 INFO:teuthology.orchestra.run.smithi001.stdout: "lost_at": 0, 2024-09-13T11:26:33.698 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-09-13T11:26:33.698 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:26:33.698 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.698 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:26:33.699 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.162:6808", 2024-09-13T11:26:33.699 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 2287343806 2024-09-13T11:26:33.699 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:33.699 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:33.699 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.699 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_addrs": { 2024-09-13T11:26:33.699 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:26:33.699 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.699 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:26:33.699 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.162:6809", 2024-09-13T11:26:33.699 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 2287343806 2024-09-13T11:26:33.699 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:33.699 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:33.699 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.700 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_back_addrs": { 2024-09-13T11:26:33.700 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:26:33.700 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.700 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:26:33.700 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.162:6811", 2024-09-13T11:26:33.700 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 2287343806 2024-09-13T11:26:33.700 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:33.700 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:33.700 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.700 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_front_addrs": { 2024-09-13T11:26:33.700 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:26:33.700 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.700 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:26:33.701 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.162:6810", 2024-09-13T11:26:33.701 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 2287343806 2024-09-13T11:26:33.701 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:33.701 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:33.701 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.701 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.162:6808/2287343806", 2024-09-13T11:26:33.701 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_addr": "172.21.15.162:6809/2287343806", 2024-09-13T11:26:33.701 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_back_addr": "172.21.15.162:6811/2287343806", 2024-09-13T11:26:33.701 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_front_addr": "172.21.15.162:6810/2287343806", 2024-09-13T11:26:33.701 INFO:teuthology.orchestra.run.smithi001.stdout: "state": [ 2024-09-13T11:26:33.701 INFO:teuthology.orchestra.run.smithi001.stdout: "exists", 2024-09-13T11:26:33.701 INFO:teuthology.orchestra.run.smithi001.stdout: "up" 2024-09-13T11:26:33.701 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:33.701 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.702 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.702 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": 7, 2024-09-13T11:26:33.702 INFO:teuthology.orchestra.run.smithi001.stdout: "uuid": "60aa5206-cace-40f8-8d1a-10ec28d97096", 2024-09-13T11:26:33.702 INFO:teuthology.orchestra.run.smithi001.stdout: "up": 1, 2024-09-13T11:26:33.702 INFO:teuthology.orchestra.run.smithi001.stdout: "in": 1, 2024-09-13T11:26:33.702 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 1, 2024-09-13T11:26:33.702 INFO:teuthology.orchestra.run.smithi001.stdout: "primary_affinity": 1, 2024-09-13T11:26:33.702 INFO:teuthology.orchestra.run.smithi001.stdout: "last_clean_begin": 0, 2024-09-13T11:26:33.702 INFO:teuthology.orchestra.run.smithi001.stdout: "last_clean_end": 0, 2024-09-13T11:26:33.702 INFO:teuthology.orchestra.run.smithi001.stdout: "up_from": 22, 2024-09-13T11:26:33.703 INFO:teuthology.orchestra.run.smithi001.stdout: "up_thru": 122, 2024-09-13T11:26:33.703 INFO:teuthology.orchestra.run.smithi001.stdout: "down_at": 0, 2024-09-13T11:26:33.704 INFO:teuthology.orchestra.run.smithi001.stdout: "lost_at": 0, 2024-09-13T11:26:33.704 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-09-13T11:26:33.704 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:26:33.704 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.704 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:26:33.704 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.177:6813", 2024-09-13T11:26:33.704 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 2110809906 2024-09-13T11:26:33.704 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:33.704 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:33.704 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.704 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_addrs": { 2024-09-13T11:26:33.704 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:26:33.705 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.705 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:26:33.705 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.177:6814", 2024-09-13T11:26:33.705 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 2110809906 2024-09-13T11:26:33.705 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:33.705 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:33.705 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.705 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_back_addrs": { 2024-09-13T11:26:33.705 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:26:33.705 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.705 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:26:33.705 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.177:6816", 2024-09-13T11:26:33.705 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 2110809906 2024-09-13T11:26:33.705 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:33.706 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:33.706 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.706 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_front_addrs": { 2024-09-13T11:26:33.706 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:26:33.706 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.706 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:26:33.706 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.177:6815", 2024-09-13T11:26:33.706 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 2110809906 2024-09-13T11:26:33.706 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:33.707 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:33.707 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.707 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.177:6813/2110809906", 2024-09-13T11:26:33.707 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_addr": "172.21.15.177:6814/2110809906", 2024-09-13T11:26:33.707 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_back_addr": "172.21.15.177:6816/2110809906", 2024-09-13T11:26:33.707 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_front_addr": "172.21.15.177:6815/2110809906", 2024-09-13T11:26:33.707 INFO:teuthology.orchestra.run.smithi001.stdout: "state": [ 2024-09-13T11:26:33.707 INFO:teuthology.orchestra.run.smithi001.stdout: "exists", 2024-09-13T11:26:33.707 INFO:teuthology.orchestra.run.smithi001.stdout: "up" 2024-09-13T11:26:33.707 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:33.707 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.707 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.707 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": 8, 2024-09-13T11:26:33.708 INFO:teuthology.orchestra.run.smithi001.stdout: "uuid": "f3d99325-48ff-403c-8662-1d7745f7c97d", 2024-09-13T11:26:33.708 INFO:teuthology.orchestra.run.smithi001.stdout: "up": 1, 2024-09-13T11:26:33.708 INFO:teuthology.orchestra.run.smithi001.stdout: "in": 1, 2024-09-13T11:26:33.708 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 1, 2024-09-13T11:26:33.708 INFO:teuthology.orchestra.run.smithi001.stdout: "primary_affinity": 1, 2024-09-13T11:26:33.708 INFO:teuthology.orchestra.run.smithi001.stdout: "last_clean_begin": 0, 2024-09-13T11:26:33.708 INFO:teuthology.orchestra.run.smithi001.stdout: "last_clean_end": 0, 2024-09-13T11:26:33.708 INFO:teuthology.orchestra.run.smithi001.stdout: "up_from": 22, 2024-09-13T11:26:33.708 INFO:teuthology.orchestra.run.smithi001.stdout: "up_thru": 172, 2024-09-13T11:26:33.708 INFO:teuthology.orchestra.run.smithi001.stdout: "down_at": 0, 2024-09-13T11:26:33.708 INFO:teuthology.orchestra.run.smithi001.stdout: "lost_at": 0, 2024-09-13T11:26:33.708 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-09-13T11:26:33.708 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:26:33.709 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.709 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:26:33.709 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6804", 2024-09-13T11:26:33.709 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 2766567910 2024-09-13T11:26:33.709 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:33.709 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:33.709 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.709 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_addrs": { 2024-09-13T11:26:33.709 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:26:33.709 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.709 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:26:33.709 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6805", 2024-09-13T11:26:33.709 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 2766567910 2024-09-13T11:26:33.709 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:33.710 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:33.710 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.710 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_back_addrs": { 2024-09-13T11:26:33.710 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:26:33.710 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.710 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:26:33.710 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6807", 2024-09-13T11:26:33.710 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 2766567910 2024-09-13T11:26:33.710 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:33.710 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:33.710 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.710 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_front_addrs": { 2024-09-13T11:26:33.711 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:26:33.711 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.711 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:26:33.711 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6806", 2024-09-13T11:26:33.711 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 2766567910 2024-09-13T11:26:33.711 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:33.711 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:33.711 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.711 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6804/2766567910", 2024-09-13T11:26:33.711 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_addr": "172.21.15.1:6805/2766567910", 2024-09-13T11:26:33.711 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_back_addr": "172.21.15.1:6807/2766567910", 2024-09-13T11:26:33.711 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_front_addr": "172.21.15.1:6806/2766567910", 2024-09-13T11:26:33.711 INFO:teuthology.orchestra.run.smithi001.stdout: "state": [ 2024-09-13T11:26:33.711 INFO:teuthology.orchestra.run.smithi001.stdout: "exists", 2024-09-13T11:26:33.712 INFO:teuthology.orchestra.run.smithi001.stdout: "up" 2024-09-13T11:26:33.712 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:33.712 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.712 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.712 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": 9, 2024-09-13T11:26:33.712 INFO:teuthology.orchestra.run.smithi001.stdout: "uuid": "88511e24-3675-4551-968c-1a52e7e28f96", 2024-09-13T11:26:33.712 INFO:teuthology.orchestra.run.smithi001.stdout: "up": 1, 2024-09-13T11:26:33.712 INFO:teuthology.orchestra.run.smithi001.stdout: "in": 1, 2024-09-13T11:26:33.712 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 1, 2024-09-13T11:26:33.712 INFO:teuthology.orchestra.run.smithi001.stdout: "primary_affinity": 1, 2024-09-13T11:26:33.712 INFO:teuthology.orchestra.run.smithi001.stdout: "last_clean_begin": 0, 2024-09-13T11:26:33.712 INFO:teuthology.orchestra.run.smithi001.stdout: "last_clean_end": 0, 2024-09-13T11:26:33.712 INFO:teuthology.orchestra.run.smithi001.stdout: "up_from": 23, 2024-09-13T11:26:33.713 INFO:teuthology.orchestra.run.smithi001.stdout: "up_thru": 191, 2024-09-13T11:26:33.713 INFO:teuthology.orchestra.run.smithi001.stdout: "down_at": 0, 2024-09-13T11:26:33.713 INFO:teuthology.orchestra.run.smithi001.stdout: "lost_at": 0, 2024-09-13T11:26:33.713 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-09-13T11:26:33.713 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:26:33.713 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.713 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:26:33.713 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.92:6812", 2024-09-13T11:26:33.713 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 2488187511 2024-09-13T11:26:33.716 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:33.716 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:33.716 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.716 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_addrs": { 2024-09-13T11:26:33.716 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:26:33.716 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.716 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:26:33.716 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.92:6813", 2024-09-13T11:26:33.717 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 2488187511 2024-09-13T11:26:33.717 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:33.717 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:33.717 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.717 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_back_addrs": { 2024-09-13T11:26:33.717 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:26:33.717 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.717 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:26:33.717 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.92:6815", 2024-09-13T11:26:33.717 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 2488187511 2024-09-13T11:26:33.717 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:33.717 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:33.717 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.718 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_front_addrs": { 2024-09-13T11:26:33.718 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:26:33.718 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.718 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:26:33.718 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.92:6814", 2024-09-13T11:26:33.718 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 2488187511 2024-09-13T11:26:33.718 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:33.718 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:33.718 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.718 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.92:6812/2488187511", 2024-09-13T11:26:33.718 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_addr": "172.21.15.92:6813/2488187511", 2024-09-13T11:26:33.718 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_back_addr": "172.21.15.92:6815/2488187511", 2024-09-13T11:26:33.718 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_front_addr": "172.21.15.92:6814/2488187511", 2024-09-13T11:26:33.718 INFO:teuthology.orchestra.run.smithi001.stdout: "state": [ 2024-09-13T11:26:33.719 INFO:teuthology.orchestra.run.smithi001.stdout: "exists", 2024-09-13T11:26:33.719 INFO:teuthology.orchestra.run.smithi001.stdout: "up" 2024-09-13T11:26:33.719 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:33.719 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.719 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.719 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": 10, 2024-09-13T11:26:33.719 INFO:teuthology.orchestra.run.smithi001.stdout: "uuid": "acf4c64d-99af-42be-8406-413c86f54670", 2024-09-13T11:26:33.719 INFO:teuthology.orchestra.run.smithi001.stdout: "up": 1, 2024-09-13T11:26:33.719 INFO:teuthology.orchestra.run.smithi001.stdout: "in": 1, 2024-09-13T11:26:33.719 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 1, 2024-09-13T11:26:33.719 INFO:teuthology.orchestra.run.smithi001.stdout: "primary_affinity": 1, 2024-09-13T11:26:33.719 INFO:teuthology.orchestra.run.smithi001.stdout: "last_clean_begin": 0, 2024-09-13T11:26:33.719 INFO:teuthology.orchestra.run.smithi001.stdout: "last_clean_end": 0, 2024-09-13T11:26:33.720 INFO:teuthology.orchestra.run.smithi001.stdout: "up_from": 22, 2024-09-13T11:26:33.720 INFO:teuthology.orchestra.run.smithi001.stdout: "up_thru": 205, 2024-09-13T11:26:33.720 INFO:teuthology.orchestra.run.smithi001.stdout: "down_at": 0, 2024-09-13T11:26:33.720 INFO:teuthology.orchestra.run.smithi001.stdout: "lost_at": 0, 2024-09-13T11:26:33.720 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-09-13T11:26:33.720 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:26:33.720 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.720 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:26:33.720 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.162:6804", 2024-09-13T11:26:33.720 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 1450266459 2024-09-13T11:26:33.720 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:33.720 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:33.720 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.721 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_addrs": { 2024-09-13T11:26:33.721 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:26:33.721 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.721 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:26:33.721 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.162:6805", 2024-09-13T11:26:33.721 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 1450266459 2024-09-13T11:26:33.721 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:33.721 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:33.721 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.721 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_back_addrs": { 2024-09-13T11:26:33.721 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:26:33.721 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.721 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:26:33.721 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.162:6807", 2024-09-13T11:26:33.722 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 1450266459 2024-09-13T11:26:33.722 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:33.722 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:33.722 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.722 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_front_addrs": { 2024-09-13T11:26:33.722 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:26:33.722 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.722 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:26:33.722 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.162:6806", 2024-09-13T11:26:33.722 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 1450266459 2024-09-13T11:26:33.722 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:33.722 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:33.723 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.723 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.162:6804/1450266459", 2024-09-13T11:26:33.723 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_addr": "172.21.15.162:6805/1450266459", 2024-09-13T11:26:33.723 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_back_addr": "172.21.15.162:6807/1450266459", 2024-09-13T11:26:33.723 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_front_addr": "172.21.15.162:6806/1450266459", 2024-09-13T11:26:33.723 INFO:teuthology.orchestra.run.smithi001.stdout: "state": [ 2024-09-13T11:26:33.723 INFO:teuthology.orchestra.run.smithi001.stdout: "exists", 2024-09-13T11:26:33.723 INFO:teuthology.orchestra.run.smithi001.stdout: "up" 2024-09-13T11:26:33.723 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:33.723 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.723 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.723 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": 11, 2024-09-13T11:26:33.723 INFO:teuthology.orchestra.run.smithi001.stdout: "uuid": "a4bfdfbe-b586-4080-8e09-6f26428e59b5", 2024-09-13T11:26:33.723 INFO:teuthology.orchestra.run.smithi001.stdout: "up": 1, 2024-09-13T11:26:33.724 INFO:teuthology.orchestra.run.smithi001.stdout: "in": 1, 2024-09-13T11:26:33.724 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 1, 2024-09-13T11:26:33.724 INFO:teuthology.orchestra.run.smithi001.stdout: "primary_affinity": 1, 2024-09-13T11:26:33.724 INFO:teuthology.orchestra.run.smithi001.stdout: "last_clean_begin": 0, 2024-09-13T11:26:33.724 INFO:teuthology.orchestra.run.smithi001.stdout: "last_clean_end": 0, 2024-09-13T11:26:33.724 INFO:teuthology.orchestra.run.smithi001.stdout: "up_from": 22, 2024-09-13T11:26:33.724 INFO:teuthology.orchestra.run.smithi001.stdout: "up_thru": 143, 2024-09-13T11:26:33.724 INFO:teuthology.orchestra.run.smithi001.stdout: "down_at": 0, 2024-09-13T11:26:33.724 INFO:teuthology.orchestra.run.smithi001.stdout: "lost_at": 0, 2024-09-13T11:26:33.724 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-09-13T11:26:33.724 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:26:33.724 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.724 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:26:33.724 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.177:6809", 2024-09-13T11:26:33.725 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 3239521954 2024-09-13T11:26:33.725 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:33.725 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:33.725 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.725 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_addrs": { 2024-09-13T11:26:33.725 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:26:33.725 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.725 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:26:33.725 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.177:6810", 2024-09-13T11:26:33.725 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 3239521954 2024-09-13T11:26:33.725 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:33.725 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:33.725 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.725 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_back_addrs": { 2024-09-13T11:26:33.726 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:26:33.726 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.726 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:26:33.726 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.177:6812", 2024-09-13T11:26:33.726 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 3239521954 2024-09-13T11:26:33.726 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:33.726 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:33.726 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.726 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_front_addrs": { 2024-09-13T11:26:33.726 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:26:33.726 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.726 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:26:33.726 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.177:6811", 2024-09-13T11:26:33.727 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 3239521954 2024-09-13T11:26:33.727 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:33.727 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:33.727 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.727 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.177:6809/3239521954", 2024-09-13T11:26:33.727 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_addr": "172.21.15.177:6810/3239521954", 2024-09-13T11:26:33.727 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_back_addr": "172.21.15.177:6812/3239521954", 2024-09-13T11:26:33.727 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_front_addr": "172.21.15.177:6811/3239521954", 2024-09-13T11:26:33.727 INFO:teuthology.orchestra.run.smithi001.stdout: "state": [ 2024-09-13T11:26:33.727 INFO:teuthology.orchestra.run.smithi001.stdout: "exists", 2024-09-13T11:26:33.727 INFO:teuthology.orchestra.run.smithi001.stdout: "up" 2024-09-13T11:26:33.727 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:33.727 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.727 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.728 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": 12, 2024-09-13T11:26:33.728 INFO:teuthology.orchestra.run.smithi001.stdout: "uuid": "dc571fb6-d8b5-465b-bb25-afa7d0c47b5a", 2024-09-13T11:26:33.728 INFO:teuthology.orchestra.run.smithi001.stdout: "up": 1, 2024-09-13T11:26:33.728 INFO:teuthology.orchestra.run.smithi001.stdout: "in": 1, 2024-09-13T11:26:33.728 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 1, 2024-09-13T11:26:33.728 INFO:teuthology.orchestra.run.smithi001.stdout: "primary_affinity": 1, 2024-09-13T11:26:33.728 INFO:teuthology.orchestra.run.smithi001.stdout: "last_clean_begin": 0, 2024-09-13T11:26:33.728 INFO:teuthology.orchestra.run.smithi001.stdout: "last_clean_end": 0, 2024-09-13T11:26:33.728 INFO:teuthology.orchestra.run.smithi001.stdout: "up_from": 22, 2024-09-13T11:26:33.728 INFO:teuthology.orchestra.run.smithi001.stdout: "up_thru": 141, 2024-09-13T11:26:33.728 INFO:teuthology.orchestra.run.smithi001.stdout: "down_at": 0, 2024-09-13T11:26:33.728 INFO:teuthology.orchestra.run.smithi001.stdout: "lost_at": 0, 2024-09-13T11:26:33.728 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-09-13T11:26:33.729 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:26:33.729 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.729 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:26:33.729 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6812", 2024-09-13T11:26:33.729 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 4267062742 2024-09-13T11:26:33.729 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:33.729 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:33.729 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.729 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_addrs": { 2024-09-13T11:26:33.729 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:26:33.729 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.729 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:26:33.729 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6813", 2024-09-13T11:26:33.729 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 4267062742 2024-09-13T11:26:33.730 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:33.730 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:33.730 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.730 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_back_addrs": { 2024-09-13T11:26:33.730 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:26:33.730 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.730 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:26:33.730 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6815", 2024-09-13T11:26:33.730 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 4267062742 2024-09-13T11:26:33.730 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:33.730 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:33.730 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.730 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_front_addrs": { 2024-09-13T11:26:33.731 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:26:33.731 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.731 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:26:33.731 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6814", 2024-09-13T11:26:33.731 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 4267062742 2024-09-13T11:26:33.731 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:33.731 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:33.731 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.731 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6812/4267062742", 2024-09-13T11:26:33.731 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_addr": "172.21.15.1:6813/4267062742", 2024-09-13T11:26:33.731 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_back_addr": "172.21.15.1:6815/4267062742", 2024-09-13T11:26:33.731 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_front_addr": "172.21.15.1:6814/4267062742", 2024-09-13T11:26:33.731 INFO:teuthology.orchestra.run.smithi001.stdout: "state": [ 2024-09-13T11:26:33.731 INFO:teuthology.orchestra.run.smithi001.stdout: "exists", 2024-09-13T11:26:33.732 INFO:teuthology.orchestra.run.smithi001.stdout: "up" 2024-09-13T11:26:33.732 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:33.732 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.732 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.732 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": 13, 2024-09-13T11:26:33.732 INFO:teuthology.orchestra.run.smithi001.stdout: "uuid": "5d321d4b-1b26-47fe-bf84-25751b8cde7d", 2024-09-13T11:26:33.732 INFO:teuthology.orchestra.run.smithi001.stdout: "up": 1, 2024-09-13T11:26:33.732 INFO:teuthology.orchestra.run.smithi001.stdout: "in": 1, 2024-09-13T11:26:33.732 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 1, 2024-09-13T11:26:33.732 INFO:teuthology.orchestra.run.smithi001.stdout: "primary_affinity": 1, 2024-09-13T11:26:33.732 INFO:teuthology.orchestra.run.smithi001.stdout: "last_clean_begin": 0, 2024-09-13T11:26:33.732 INFO:teuthology.orchestra.run.smithi001.stdout: "last_clean_end": 0, 2024-09-13T11:26:33.732 INFO:teuthology.orchestra.run.smithi001.stdout: "up_from": 23, 2024-09-13T11:26:33.732 INFO:teuthology.orchestra.run.smithi001.stdout: "up_thru": 205, 2024-09-13T11:26:33.733 INFO:teuthology.orchestra.run.smithi001.stdout: "down_at": 0, 2024-09-13T11:26:33.733 INFO:teuthology.orchestra.run.smithi001.stdout: "lost_at": 0, 2024-09-13T11:26:33.733 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-09-13T11:26:33.733 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:26:33.733 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.733 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:26:33.733 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.92:6808", 2024-09-13T11:26:33.733 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 1759585046 2024-09-13T11:26:33.733 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:33.733 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:33.733 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.733 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_addrs": { 2024-09-13T11:26:33.733 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:26:33.733 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.734 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:26:33.734 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.92:6809", 2024-09-13T11:26:33.734 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 1759585046 2024-09-13T11:26:33.734 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:33.734 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:33.734 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.734 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_back_addrs": { 2024-09-13T11:26:33.734 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:26:33.734 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.734 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:26:33.734 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.92:6811", 2024-09-13T11:26:33.734 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 1759585046 2024-09-13T11:26:33.734 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:33.735 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:33.735 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.735 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_front_addrs": { 2024-09-13T11:26:33.735 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:26:33.735 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.735 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:26:33.735 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.92:6810", 2024-09-13T11:26:33.735 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 1759585046 2024-09-13T11:26:33.735 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:33.735 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:33.735 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.735 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.92:6808/1759585046", 2024-09-13T11:26:33.735 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_addr": "172.21.15.92:6809/1759585046", 2024-09-13T11:26:33.735 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_back_addr": "172.21.15.92:6811/1759585046", 2024-09-13T11:26:33.736 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_front_addr": "172.21.15.92:6810/1759585046", 2024-09-13T11:26:33.736 INFO:teuthology.orchestra.run.smithi001.stdout: "state": [ 2024-09-13T11:26:33.736 INFO:teuthology.orchestra.run.smithi001.stdout: "exists", 2024-09-13T11:26:33.736 INFO:teuthology.orchestra.run.smithi001.stdout: "up" 2024-09-13T11:26:33.736 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:33.736 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.736 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.736 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": 14, 2024-09-13T11:26:33.736 INFO:teuthology.orchestra.run.smithi001.stdout: "uuid": "fc3e2499-0c2a-414d-a25e-2628876ae630", 2024-09-13T11:26:33.736 INFO:teuthology.orchestra.run.smithi001.stdout: "up": 1, 2024-09-13T11:26:33.736 INFO:teuthology.orchestra.run.smithi001.stdout: "in": 1, 2024-09-13T11:26:33.736 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 1, 2024-09-13T11:26:33.737 INFO:teuthology.orchestra.run.smithi001.stdout: "primary_affinity": 1, 2024-09-13T11:26:33.737 INFO:teuthology.orchestra.run.smithi001.stdout: "last_clean_begin": 22, 2024-09-13T11:26:33.737 INFO:teuthology.orchestra.run.smithi001.stdout: "last_clean_end": 120, 2024-09-13T11:26:33.737 INFO:teuthology.orchestra.run.smithi001.stdout: "up_from": 142, 2024-09-13T11:26:33.737 INFO:teuthology.orchestra.run.smithi001.stdout: "up_thru": 201, 2024-09-13T11:26:33.737 INFO:teuthology.orchestra.run.smithi001.stdout: "down_at": 121, 2024-09-13T11:26:33.737 INFO:teuthology.orchestra.run.smithi001.stdout: "lost_at": 0, 2024-09-13T11:26:33.737 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-09-13T11:26:33.737 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:26:33.737 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.737 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:26:33.737 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.162:6800", 2024-09-13T11:26:33.737 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 3999392664 2024-09-13T11:26:33.737 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:33.738 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:33.738 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.738 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_addrs": { 2024-09-13T11:26:33.738 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:26:33.738 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.738 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:26:33.738 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.162:6801", 2024-09-13T11:26:33.738 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 3999392664 2024-09-13T11:26:33.738 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:33.738 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:33.738 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.738 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_back_addrs": { 2024-09-13T11:26:33.738 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:26:33.739 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.739 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:26:33.739 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.162:6803", 2024-09-13T11:26:33.739 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 3999392664 2024-09-13T11:26:33.739 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:33.739 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:33.739 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.739 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_front_addrs": { 2024-09-13T11:26:33.739 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:26:33.739 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.739 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:26:33.739 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.162:6802", 2024-09-13T11:26:33.739 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 3999392664 2024-09-13T11:26:33.739 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:33.740 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:33.740 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.740 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.162:6800/3999392664", 2024-09-13T11:26:33.740 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_addr": "172.21.15.162:6801/3999392664", 2024-09-13T11:26:33.740 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_back_addr": "172.21.15.162:6803/3999392664", 2024-09-13T11:26:33.740 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_front_addr": "172.21.15.162:6802/3999392664", 2024-09-13T11:26:33.740 INFO:teuthology.orchestra.run.smithi001.stdout: "state": [ 2024-09-13T11:26:33.740 INFO:teuthology.orchestra.run.smithi001.stdout: "exists", 2024-09-13T11:26:33.740 INFO:teuthology.orchestra.run.smithi001.stdout: "up" 2024-09-13T11:26:33.740 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:33.740 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.740 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.740 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": 15, 2024-09-13T11:26:33.741 INFO:teuthology.orchestra.run.smithi001.stdout: "uuid": "d2f69775-edfe-4527-91ec-f7c7e558bec8", 2024-09-13T11:26:33.741 INFO:teuthology.orchestra.run.smithi001.stdout: "up": 1, 2024-09-13T11:26:33.741 INFO:teuthology.orchestra.run.smithi001.stdout: "in": 1, 2024-09-13T11:26:33.741 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 1, 2024-09-13T11:26:33.741 INFO:teuthology.orchestra.run.smithi001.stdout: "primary_affinity": 0, 2024-09-13T11:26:33.741 INFO:teuthology.orchestra.run.smithi001.stdout: "last_clean_begin": 0, 2024-09-13T11:26:33.741 INFO:teuthology.orchestra.run.smithi001.stdout: "last_clean_end": 0, 2024-09-13T11:26:33.741 INFO:teuthology.orchestra.run.smithi001.stdout: "up_from": 22, 2024-09-13T11:26:33.741 INFO:teuthology.orchestra.run.smithi001.stdout: "up_thru": 122, 2024-09-13T11:26:33.741 INFO:teuthology.orchestra.run.smithi001.stdout: "down_at": 0, 2024-09-13T11:26:33.741 INFO:teuthology.orchestra.run.smithi001.stdout: "lost_at": 0, 2024-09-13T11:26:33.741 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-09-13T11:26:33.741 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:26:33.742 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.742 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:26:33.742 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.177:6805", 2024-09-13T11:26:33.742 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 1294337172 2024-09-13T11:26:33.742 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:33.742 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:33.742 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.742 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_addrs": { 2024-09-13T11:26:33.742 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:26:33.742 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.742 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:26:33.742 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.177:6806", 2024-09-13T11:26:33.742 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 1294337172 2024-09-13T11:26:33.742 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:33.742 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:33.743 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.743 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_back_addrs": { 2024-09-13T11:26:33.743 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:26:33.743 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.743 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:26:33.743 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.177:6808", 2024-09-13T11:26:33.743 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 1294337172 2024-09-13T11:26:33.743 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:33.743 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:33.743 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.743 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_front_addrs": { 2024-09-13T11:26:33.743 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:26:33.743 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.743 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:26:33.744 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.177:6807", 2024-09-13T11:26:33.744 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 1294337172 2024-09-13T11:26:33.744 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:33.744 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:33.744 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.744 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.177:6805/1294337172", 2024-09-13T11:26:33.744 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_addr": "172.21.15.177:6806/1294337172", 2024-09-13T11:26:33.744 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_back_addr": "172.21.15.177:6808/1294337172", 2024-09-13T11:26:33.744 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_front_addr": "172.21.15.177:6807/1294337172", 2024-09-13T11:26:33.744 INFO:teuthology.orchestra.run.smithi001.stdout: "state": [ 2024-09-13T11:26:33.744 INFO:teuthology.orchestra.run.smithi001.stdout: "exists", 2024-09-13T11:26:33.744 INFO:teuthology.orchestra.run.smithi001.stdout: "up" 2024-09-13T11:26:33.744 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:33.745 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:33.745 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-09-13T11:26:33.745 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_xinfo": [ 2024-09-13T11:26:33.745 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.745 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": 0, 2024-09-13T11:26:33.745 INFO:teuthology.orchestra.run.smithi001.stdout: "down_stamp": "0.000000", 2024-09-13T11:26:33.745 INFO:teuthology.orchestra.run.smithi001.stdout: "laggy_probability": 0, 2024-09-13T11:26:33.745 INFO:teuthology.orchestra.run.smithi001.stdout: "laggy_interval": 0, 2024-09-13T11:26:33.745 INFO:teuthology.orchestra.run.smithi001.stdout: "features": 4540719139924082687, 2024-09-13T11:26:33.745 INFO:teuthology.orchestra.run.smithi001.stdout: "old_weight": 0, 2024-09-13T11:26:33.745 INFO:teuthology.orchestra.run.smithi001.stdout: "last_purged_snaps_scrub": "2024-09-13T11:26:19.725882+0000", 2024-09-13T11:26:33.745 INFO:teuthology.orchestra.run.smithi001.stdout: "dead_epoch": 0 2024-09-13T11:26:33.745 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.745 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.746 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": 1, 2024-09-13T11:26:33.746 INFO:teuthology.orchestra.run.smithi001.stdout: "down_stamp": "0.000000", 2024-09-13T11:26:33.746 INFO:teuthology.orchestra.run.smithi001.stdout: "laggy_probability": 0, 2024-09-13T11:26:33.746 INFO:teuthology.orchestra.run.smithi001.stdout: "laggy_interval": 0, 2024-09-13T11:26:33.746 INFO:teuthology.orchestra.run.smithi001.stdout: "features": 4540719139924082687, 2024-09-13T11:26:33.746 INFO:teuthology.orchestra.run.smithi001.stdout: "old_weight": 0, 2024-09-13T11:26:33.746 INFO:teuthology.orchestra.run.smithi001.stdout: "last_purged_snaps_scrub": "2024-09-13T11:25:47.780833+0000", 2024-09-13T11:26:33.746 INFO:teuthology.orchestra.run.smithi001.stdout: "dead_epoch": 30 2024-09-13T11:26:33.746 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.746 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.746 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": 2, 2024-09-13T11:26:33.746 INFO:teuthology.orchestra.run.smithi001.stdout: "down_stamp": "0.000000", 2024-09-13T11:26:33.746 INFO:teuthology.orchestra.run.smithi001.stdout: "laggy_probability": 0, 2024-09-13T11:26:33.746 INFO:teuthology.orchestra.run.smithi001.stdout: "laggy_interval": 0, 2024-09-13T11:26:33.747 INFO:teuthology.orchestra.run.smithi001.stdout: "features": 4540719139924082687, 2024-09-13T11:26:33.747 INFO:teuthology.orchestra.run.smithi001.stdout: "old_weight": 0, 2024-09-13T11:26:33.747 INFO:teuthology.orchestra.run.smithi001.stdout: "last_purged_snaps_scrub": "2024-09-13T11:26:18.925695+0000", 2024-09-13T11:26:33.747 INFO:teuthology.orchestra.run.smithi001.stdout: "dead_epoch": 0 2024-09-13T11:26:33.747 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.747 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.747 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": 3, 2024-09-13T11:26:33.747 INFO:teuthology.orchestra.run.smithi001.stdout: "down_stamp": "0.000000", 2024-09-13T11:26:33.747 INFO:teuthology.orchestra.run.smithi001.stdout: "laggy_probability": 0, 2024-09-13T11:26:33.747 INFO:teuthology.orchestra.run.smithi001.stdout: "laggy_interval": 0, 2024-09-13T11:26:33.747 INFO:teuthology.orchestra.run.smithi001.stdout: "features": 4540719139924082687, 2024-09-13T11:26:33.747 INFO:teuthology.orchestra.run.smithi001.stdout: "old_weight": 0, 2024-09-13T11:26:33.747 INFO:teuthology.orchestra.run.smithi001.stdout: "last_purged_snaps_scrub": "2024-09-13T11:26:17.200385+0000", 2024-09-13T11:26:33.747 INFO:teuthology.orchestra.run.smithi001.stdout: "dead_epoch": 0 2024-09-13T11:26:33.748 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.748 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.748 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": 4, 2024-09-13T11:26:33.748 INFO:teuthology.orchestra.run.smithi001.stdout: "down_stamp": "0.000000", 2024-09-13T11:26:33.748 INFO:teuthology.orchestra.run.smithi001.stdout: "laggy_probability": 0, 2024-09-13T11:26:33.748 INFO:teuthology.orchestra.run.smithi001.stdout: "laggy_interval": 0, 2024-09-13T11:26:33.748 INFO:teuthology.orchestra.run.smithi001.stdout: "features": 4540719139924082687, 2024-09-13T11:26:33.748 INFO:teuthology.orchestra.run.smithi001.stdout: "old_weight": 0, 2024-09-13T11:26:33.748 INFO:teuthology.orchestra.run.smithi001.stdout: "last_purged_snaps_scrub": "2024-09-13T11:25:59.330894+0000", 2024-09-13T11:26:33.748 INFO:teuthology.orchestra.run.smithi001.stdout: "dead_epoch": 0 2024-09-13T11:26:33.748 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.748 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.749 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": 5, 2024-09-13T11:26:33.749 INFO:teuthology.orchestra.run.smithi001.stdout: "down_stamp": "0.000000", 2024-09-13T11:26:33.749 INFO:teuthology.orchestra.run.smithi001.stdout: "laggy_probability": 0, 2024-09-13T11:26:33.749 INFO:teuthology.orchestra.run.smithi001.stdout: "laggy_interval": 0, 2024-09-13T11:26:33.749 INFO:teuthology.orchestra.run.smithi001.stdout: "features": 4540719139924082687, 2024-09-13T11:26:33.749 INFO:teuthology.orchestra.run.smithi001.stdout: "old_weight": 0, 2024-09-13T11:26:33.749 INFO:teuthology.orchestra.run.smithi001.stdout: "last_purged_snaps_scrub": "2024-09-13T11:26:21.996515+0000", 2024-09-13T11:26:33.749 INFO:teuthology.orchestra.run.smithi001.stdout: "dead_epoch": 0 2024-09-13T11:26:33.749 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.749 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.749 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": 6, 2024-09-13T11:26:33.749 INFO:teuthology.orchestra.run.smithi001.stdout: "down_stamp": "0.000000", 2024-09-13T11:26:33.749 INFO:teuthology.orchestra.run.smithi001.stdout: "laggy_probability": 0, 2024-09-13T11:26:33.750 INFO:teuthology.orchestra.run.smithi001.stdout: "laggy_interval": 0, 2024-09-13T11:26:33.750 INFO:teuthology.orchestra.run.smithi001.stdout: "features": 4540719139924082687, 2024-09-13T11:26:33.750 INFO:teuthology.orchestra.run.smithi001.stdout: "old_weight": 0, 2024-09-13T11:26:33.750 INFO:teuthology.orchestra.run.smithi001.stdout: "last_purged_snaps_scrub": "2024-09-13T11:26:16.679806+0000", 2024-09-13T11:26:33.750 INFO:teuthology.orchestra.run.smithi001.stdout: "dead_epoch": 0 2024-09-13T11:26:33.750 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.750 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.750 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": 7, 2024-09-13T11:26:33.750 INFO:teuthology.orchestra.run.smithi001.stdout: "down_stamp": "0.000000", 2024-09-13T11:26:33.750 INFO:teuthology.orchestra.run.smithi001.stdout: "laggy_probability": 0, 2024-09-13T11:26:33.750 INFO:teuthology.orchestra.run.smithi001.stdout: "laggy_interval": 0, 2024-09-13T11:26:33.750 INFO:teuthology.orchestra.run.smithi001.stdout: "features": 4540719139924082687, 2024-09-13T11:26:33.750 INFO:teuthology.orchestra.run.smithi001.stdout: "old_weight": 0, 2024-09-13T11:26:33.750 INFO:teuthology.orchestra.run.smithi001.stdout: "last_purged_snaps_scrub": "2024-09-13T11:26:01.857996+0000", 2024-09-13T11:26:33.751 INFO:teuthology.orchestra.run.smithi001.stdout: "dead_epoch": 0 2024-09-13T11:26:33.751 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.751 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.751 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": 8, 2024-09-13T11:26:33.751 INFO:teuthology.orchestra.run.smithi001.stdout: "down_stamp": "0.000000", 2024-09-13T11:26:33.751 INFO:teuthology.orchestra.run.smithi001.stdout: "laggy_probability": 0, 2024-09-13T11:26:33.751 INFO:teuthology.orchestra.run.smithi001.stdout: "laggy_interval": 0, 2024-09-13T11:26:33.751 INFO:teuthology.orchestra.run.smithi001.stdout: "features": 4540719139924082687, 2024-09-13T11:26:33.751 INFO:teuthology.orchestra.run.smithi001.stdout: "old_weight": 0, 2024-09-13T11:26:33.751 INFO:teuthology.orchestra.run.smithi001.stdout: "last_purged_snaps_scrub": "2024-09-13T11:26:14.379046+0000", 2024-09-13T11:26:33.751 INFO:teuthology.orchestra.run.smithi001.stdout: "dead_epoch": 0 2024-09-13T11:26:33.751 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.751 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.751 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": 9, 2024-09-13T11:26:33.752 INFO:teuthology.orchestra.run.smithi001.stdout: "down_stamp": "0.000000", 2024-09-13T11:26:33.752 INFO:teuthology.orchestra.run.smithi001.stdout: "laggy_probability": 0, 2024-09-13T11:26:33.752 INFO:teuthology.orchestra.run.smithi001.stdout: "laggy_interval": 0, 2024-09-13T11:26:33.752 INFO:teuthology.orchestra.run.smithi001.stdout: "features": 4540719139924082687, 2024-09-13T11:26:33.752 INFO:teuthology.orchestra.run.smithi001.stdout: "old_weight": 0, 2024-09-13T11:26:33.752 INFO:teuthology.orchestra.run.smithi001.stdout: "last_purged_snaps_scrub": "2024-09-13T11:26:09.278966+0000", 2024-09-13T11:26:33.752 INFO:teuthology.orchestra.run.smithi001.stdout: "dead_epoch": 0 2024-09-13T11:26:33.752 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.752 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.752 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": 10, 2024-09-13T11:26:33.752 INFO:teuthology.orchestra.run.smithi001.stdout: "down_stamp": "0.000000", 2024-09-13T11:26:33.752 INFO:teuthology.orchestra.run.smithi001.stdout: "laggy_probability": 0, 2024-09-13T11:26:33.752 INFO:teuthology.orchestra.run.smithi001.stdout: "laggy_interval": 0, 2024-09-13T11:26:33.753 INFO:teuthology.orchestra.run.smithi001.stdout: "features": 4540719139924082687, 2024-09-13T11:26:33.753 INFO:teuthology.orchestra.run.smithi001.stdout: "old_weight": 0, 2024-09-13T11:26:33.753 INFO:teuthology.orchestra.run.smithi001.stdout: "last_purged_snaps_scrub": "2024-09-13T11:26:02.999347+0000", 2024-09-13T11:26:33.753 INFO:teuthology.orchestra.run.smithi001.stdout: "dead_epoch": 0 2024-09-13T11:26:33.753 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.753 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.753 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": 11, 2024-09-13T11:26:33.753 INFO:teuthology.orchestra.run.smithi001.stdout: "down_stamp": "0.000000", 2024-09-13T11:26:33.753 INFO:teuthology.orchestra.run.smithi001.stdout: "laggy_probability": 0, 2024-09-13T11:26:33.753 INFO:teuthology.orchestra.run.smithi001.stdout: "laggy_interval": 0, 2024-09-13T11:26:33.753 INFO:teuthology.orchestra.run.smithi001.stdout: "features": 4540719139924082687, 2024-09-13T11:26:33.753 INFO:teuthology.orchestra.run.smithi001.stdout: "old_weight": 0, 2024-09-13T11:26:33.753 INFO:teuthology.orchestra.run.smithi001.stdout: "last_purged_snaps_scrub": "2024-09-13T11:26:27.208176+0000", 2024-09-13T11:26:33.754 INFO:teuthology.orchestra.run.smithi001.stdout: "dead_epoch": 0 2024-09-13T11:26:33.754 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.754 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.754 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": 12, 2024-09-13T11:26:33.754 INFO:teuthology.orchestra.run.smithi001.stdout: "down_stamp": "0.000000", 2024-09-13T11:26:33.754 INFO:teuthology.orchestra.run.smithi001.stdout: "laggy_probability": 0, 2024-09-13T11:26:33.754 INFO:teuthology.orchestra.run.smithi001.stdout: "laggy_interval": 0, 2024-09-13T11:26:33.754 INFO:teuthology.orchestra.run.smithi001.stdout: "features": 4540719139924082687, 2024-09-13T11:26:33.754 INFO:teuthology.orchestra.run.smithi001.stdout: "old_weight": 0, 2024-09-13T11:26:33.754 INFO:teuthology.orchestra.run.smithi001.stdout: "last_purged_snaps_scrub": "2024-09-13T11:25:47.548386+0000", 2024-09-13T11:26:33.754 INFO:teuthology.orchestra.run.smithi001.stdout: "dead_epoch": 0 2024-09-13T11:26:33.754 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.754 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.754 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": 13, 2024-09-13T11:26:33.755 INFO:teuthology.orchestra.run.smithi001.stdout: "down_stamp": "0.000000", 2024-09-13T11:26:33.755 INFO:teuthology.orchestra.run.smithi001.stdout: "laggy_probability": 0, 2024-09-13T11:26:33.755 INFO:teuthology.orchestra.run.smithi001.stdout: "laggy_interval": 0, 2024-09-13T11:26:33.755 INFO:teuthology.orchestra.run.smithi001.stdout: "features": 4540719139924082687, 2024-09-13T11:26:33.755 INFO:teuthology.orchestra.run.smithi001.stdout: "old_weight": 0, 2024-09-13T11:26:33.755 INFO:teuthology.orchestra.run.smithi001.stdout: "last_purged_snaps_scrub": "2024-09-13T11:25:45.821466+0000", 2024-09-13T11:26:33.755 INFO:teuthology.orchestra.run.smithi001.stdout: "dead_epoch": 0 2024-09-13T11:26:33.755 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.755 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.755 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": 14, 2024-09-13T11:26:33.755 INFO:teuthology.orchestra.run.smithi001.stdout: "down_stamp": "0.000000", 2024-09-13T11:26:33.755 INFO:teuthology.orchestra.run.smithi001.stdout: "laggy_probability": 0, 2024-09-13T11:26:33.755 INFO:teuthology.orchestra.run.smithi001.stdout: "laggy_interval": 0, 2024-09-13T11:26:33.755 INFO:teuthology.orchestra.run.smithi001.stdout: "features": 4540719139924082687, 2024-09-13T11:26:33.756 INFO:teuthology.orchestra.run.smithi001.stdout: "old_weight": 0, 2024-09-13T11:26:33.756 INFO:teuthology.orchestra.run.smithi001.stdout: "last_purged_snaps_scrub": "2024-09-13T11:26:03.927526+0000", 2024-09-13T11:26:33.756 INFO:teuthology.orchestra.run.smithi001.stdout: "dead_epoch": 121 2024-09-13T11:26:33.756 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.756 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.756 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": 15, 2024-09-13T11:26:33.756 INFO:teuthology.orchestra.run.smithi001.stdout: "down_stamp": "0.000000", 2024-09-13T11:26:33.756 INFO:teuthology.orchestra.run.smithi001.stdout: "laggy_probability": 0, 2024-09-13T11:26:33.756 INFO:teuthology.orchestra.run.smithi001.stdout: "laggy_interval": 0, 2024-09-13T11:26:33.756 INFO:teuthology.orchestra.run.smithi001.stdout: "features": 4540719139924082687, 2024-09-13T11:26:33.756 INFO:teuthology.orchestra.run.smithi001.stdout: "old_weight": 0, 2024-09-13T11:26:33.756 INFO:teuthology.orchestra.run.smithi001.stdout: "last_purged_snaps_scrub": "2024-09-13T11:25:57.987079+0000", 2024-09-13T11:26:33.756 INFO:teuthology.orchestra.run.smithi001.stdout: "dead_epoch": 0 2024-09-13T11:26:33.756 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:33.757 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-09-13T11:26:33.757 INFO:teuthology.orchestra.run.smithi001.stdout: "pg_upmap": [ 2024-09-13T11:26:33.757 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.757 INFO:teuthology.orchestra.run.smithi001.stdout: "pgid": "2.2", 2024-09-13T11:26:33.757 INFO:teuthology.orchestra.run.smithi001.stdout: "osds": [ 2024-09-13T11:26:33.757 INFO:teuthology.orchestra.run.smithi001.stdout: 2, 2024-09-13T11:26:33.757 INFO:teuthology.orchestra.run.smithi001.stdout: 8 2024-09-13T11:26:33.757 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:33.757 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.757 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.757 INFO:teuthology.orchestra.run.smithi001.stdout: "pgid": "3.e", 2024-09-13T11:26:33.757 INFO:teuthology.orchestra.run.smithi001.stdout: "osds": [ 2024-09-13T11:26:33.757 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-09-13T11:26:33.758 INFO:teuthology.orchestra.run.smithi001.stdout: 1 2024-09-13T11:26:33.758 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:33.758 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.758 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.758 INFO:teuthology.orchestra.run.smithi001.stdout: "pgid": "3.22", 2024-09-13T11:26:33.758 INFO:teuthology.orchestra.run.smithi001.stdout: "osds": [ 2024-09-13T11:26:33.758 INFO:teuthology.orchestra.run.smithi001.stdout: 14, 2024-09-13T11:26:33.758 INFO:teuthology.orchestra.run.smithi001.stdout: 12 2024-09-13T11:26:33.758 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:33.758 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:33.758 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-09-13T11:26:33.758 INFO:teuthology.orchestra.run.smithi001.stdout: "pg_upmap_items": [], 2024-09-13T11:26:33.758 INFO:teuthology.orchestra.run.smithi001.stdout: "pg_upmap_primaries": [], 2024-09-13T11:26:33.758 INFO:teuthology.orchestra.run.smithi001.stdout: "pg_temp": [ 2024-09-13T11:26:33.759 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.759 INFO:teuthology.orchestra.run.smithi001.stdout: "pgid": "3.2", 2024-09-13T11:26:33.759 INFO:teuthology.orchestra.run.smithi001.stdout: "osds": [ 2024-09-13T11:26:33.759 INFO:teuthology.orchestra.run.smithi001.stdout: 3, 2024-09-13T11:26:33.759 INFO:teuthology.orchestra.run.smithi001.stdout: 12 2024-09-13T11:26:33.759 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:33.759 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.759 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.759 INFO:teuthology.orchestra.run.smithi001.stdout: "pgid": "3.12", 2024-09-13T11:26:33.759 INFO:teuthology.orchestra.run.smithi001.stdout: "osds": [ 2024-09-13T11:26:33.759 INFO:teuthology.orchestra.run.smithi001.stdout: 5, 2024-09-13T11:26:33.759 INFO:teuthology.orchestra.run.smithi001.stdout: 7 2024-09-13T11:26:33.759 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:33.759 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.760 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.760 INFO:teuthology.orchestra.run.smithi001.stdout: "pgid": "3.14", 2024-09-13T11:26:33.760 INFO:teuthology.orchestra.run.smithi001.stdout: "osds": [ 2024-09-13T11:26:33.760 INFO:teuthology.orchestra.run.smithi001.stdout: 3, 2024-09-13T11:26:33.760 INFO:teuthology.orchestra.run.smithi001.stdout: 15 2024-09-13T11:26:33.760 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:33.760 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.760 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.760 INFO:teuthology.orchestra.run.smithi001.stdout: "pgid": "3.15", 2024-09-13T11:26:33.760 INFO:teuthology.orchestra.run.smithi001.stdout: "osds": [ 2024-09-13T11:26:33.760 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-09-13T11:26:33.760 INFO:teuthology.orchestra.run.smithi001.stdout: 3 2024-09-13T11:26:33.760 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:33.761 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.761 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.761 INFO:teuthology.orchestra.run.smithi001.stdout: "pgid": "3.1f", 2024-09-13T11:26:33.761 INFO:teuthology.orchestra.run.smithi001.stdout: "osds": [ 2024-09-13T11:26:33.761 INFO:teuthology.orchestra.run.smithi001.stdout: 10, 2024-09-13T11:26:33.761 INFO:teuthology.orchestra.run.smithi001.stdout: 5 2024-09-13T11:26:33.761 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:33.761 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.761 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.761 INFO:teuthology.orchestra.run.smithi001.stdout: "pgid": "3.22", 2024-09-13T11:26:33.761 INFO:teuthology.orchestra.run.smithi001.stdout: "osds": [ 2024-09-13T11:26:33.761 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-09-13T11:26:33.761 INFO:teuthology.orchestra.run.smithi001.stdout: 14 2024-09-13T11:26:33.762 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:33.762 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:33.762 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-09-13T11:26:33.762 INFO:teuthology.orchestra.run.smithi001.stdout: "primary_temp": [], 2024-09-13T11:26:33.762 INFO:teuthology.orchestra.run.smithi001.stdout: "blocklist": {}, 2024-09-13T11:26:33.762 INFO:teuthology.orchestra.run.smithi001.stdout: "range_blocklist": {}, 2024-09-13T11:26:33.762 INFO:teuthology.orchestra.run.smithi001.stdout: "erasure_code_profiles": { 2024-09-13T11:26:33.762 INFO:teuthology.orchestra.run.smithi001.stdout: "default": { 2024-09-13T11:26:33.762 INFO:teuthology.orchestra.run.smithi001.stdout: "crush-failure-domain": "osd", 2024-09-13T11:26:33.762 INFO:teuthology.orchestra.run.smithi001.stdout: "k": "2", 2024-09-13T11:26:33.762 INFO:teuthology.orchestra.run.smithi001.stdout: "m": "1", 2024-09-13T11:26:33.762 INFO:teuthology.orchestra.run.smithi001.stdout: "plugin": "jerasure", 2024-09-13T11:26:33.762 INFO:teuthology.orchestra.run.smithi001.stdout: "technique": "reed_sol_van" 2024-09-13T11:26:33.763 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:33.763 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.763 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_snaps_queue": [], 2024-09-13T11:26:33.763 INFO:teuthology.orchestra.run.smithi001.stdout: "new_removed_snaps": [], 2024-09-13T11:26:33.763 INFO:teuthology.orchestra.run.smithi001.stdout: "new_purged_snaps": [], 2024-09-13T11:26:33.763 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_node_flags": {}, 2024-09-13T11:26:33.763 INFO:teuthology.orchestra.run.smithi001.stdout: "device_class_flags": {}, 2024-09-13T11:26:33.763 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": { 2024-09-13T11:26:33.763 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode_enabled": false, 2024-09-13T11:26:33.763 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_bucket_count": 0, 2024-09-13T11:26:33.763 INFO:teuthology.orchestra.run.smithi001.stdout: "degraded_stretch_mode": 0, 2024-09-13T11:26:33.763 INFO:teuthology.orchestra.run.smithi001.stdout: "recovering_stretch_mode": 0, 2024-09-13T11:26:33.763 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode_bucket": 0 2024-09-13T11:26:33.764 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:33.764 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-13T11:26:33.764 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T11:26:33.767 INFO:tasks.thrashosds.thrasher:j is {'epoch': 211, 'fsid': '2fd68ac7-5581-4d90-80a4-30bcf5252a7a', 'created': '2024-09-13T11:21:57.223153+0000', 'modified': '2024-09-13T11:26:32.212257+0000', 'last_up_change': '2024-09-13T11:24:55.968636+0000', 'last_in_change': '2024-09-13T11:26:04.007363+0000', 'flags': 'nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5804032, 'flags_set': ['nodeep-scrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 7, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 3, 'max_osd': 16, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-13T11:22:10.162266+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 15.380000114440918, 'score_stable': 15.380000114440918, 'optimal_score': 0.12999999523162842, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.9375, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.9399999976158142}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2024-09-13T11:22:12.235549+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '28', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 2, 'snap_epoch': 28, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 3.7300000190734863, 'score_stable': 3.7300000190734863, 'optimal_score': 0.6700000166893005, 'raw_score_acting': 2.5, 'raw_score_stable': 2.5, 'primary_affinity_weighted': 0.9375, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.9399999976158142}}, {'pool': 3, 'pool_name': 'unique_pool_0', 'create_time': '2024-09-13T11:22:24.305521+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 46, 'pg_placement_num': 46, 'pg_placement_num_target': 46, 'pg_num_target': 46, 'pg_num_pending': 46, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '196', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '117', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.9500000476837158, 'score_stable': 1.9500000476837158, 'optimal_score': 1.0700000524520874, 'raw_score_acting': 2.0899999141693115, 'raw_score_stable': 2.0899999141693115, 'primary_affinity_weighted': 0.9375, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.9399999976158142}}], 'osds': [{'osd': 0, 'uuid': '0fc12b3c-e0de-4546-b935-db77efe7c567', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 22, 'up_thru': 202, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.1:6800', 'nonce': 1544776477}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.1:6801', 'nonce': 1544776477}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.1:6803', 'nonce': 1544776477}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.1:6802', 'nonce': 1544776477}]}, 'public_addr': '172.21.15.1:6800/1544776477', 'cluster_addr': '172.21.15.1:6801/1544776477', 'heartbeat_back_addr': '172.21.15.1:6803/1544776477', 'heartbeat_front_addr': '172.21.15.1:6802/1544776477', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': 'b77fe331-cc92-4adb-b146-af5c380f6d4f', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 23, 'last_clean_end': 29, 'up_from': 40, 'up_thru': 171, 'down_at': 30, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.92:6800', 'nonce': 3591430847}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.92:6801', 'nonce': 3591430847}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.92:6803', 'nonce': 3591430847}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.92:6802', 'nonce': 3591430847}]}, 'public_addr': '172.21.15.92:6800/3591430847', 'cluster_addr': '172.21.15.92:6801/3591430847', 'heartbeat_back_addr': '172.21.15.92:6803/3591430847', 'heartbeat_front_addr': '172.21.15.92:6802/3591430847', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': 'c9d031c8-e885-4229-affa-08e1729d00e8', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 22, 'up_thru': 152, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.162:6812', 'nonce': 2712148101}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.162:6813', 'nonce': 2712148101}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.162:6815', 'nonce': 2712148101}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.162:6814', 'nonce': 2712148101}]}, 'public_addr': '172.21.15.162:6812/2712148101', 'cluster_addr': '172.21.15.162:6813/2712148101', 'heartbeat_back_addr': '172.21.15.162:6815/2712148101', 'heartbeat_front_addr': '172.21.15.162:6814/2712148101', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': 'bf50d71a-6c3a-4fbc-a407-cb81bad14a7b', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 21, 'up_thru': 192, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.177:6801', 'nonce': 3367751690}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.177:6802', 'nonce': 3367751690}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.177:6804', 'nonce': 3367751690}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.177:6803', 'nonce': 3367751690}]}, 'public_addr': '172.21.15.177:6801/3367751690', 'cluster_addr': '172.21.15.177:6802/3367751690', 'heartbeat_back_addr': '172.21.15.177:6804/3367751690', 'heartbeat_front_addr': '172.21.15.177:6803/3367751690', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': 'd1fe3f63-df3f-4a73-a1e2-8d7c8a3f1fa0', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 22, 'up_thru': 105, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.1:6808', 'nonce': 3648540926}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.1:6809', 'nonce': 3648540926}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.1:6811', 'nonce': 3648540926}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.1:6810', 'nonce': 3648540926}]}, 'public_addr': '172.21.15.1:6808/3648540926', 'cluster_addr': '172.21.15.1:6809/3648540926', 'heartbeat_back_addr': '172.21.15.1:6811/3648540926', 'heartbeat_front_addr': '172.21.15.1:6810/3648540926', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': 'e2ead447-2f2c-40c9-82e7-53b8dc41e2af', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 21, 'up_thru': 192, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.92:6804', 'nonce': 3399529091}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.92:6805', 'nonce': 3399529091}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.92:6807', 'nonce': 3399529091}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.92:6806', 'nonce': 3399529091}]}, 'public_addr': '172.21.15.92:6804/3399529091', 'cluster_addr': '172.21.15.92:6805/3399529091', 'heartbeat_back_addr': '172.21.15.92:6807/3399529091', 'heartbeat_front_addr': '172.21.15.92:6806/3399529091', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': 'dbc20439-144e-4ec7-89de-674eb783f263', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 22, 'up_thru': 117, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.162:6808', 'nonce': 2287343806}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.162:6809', 'nonce': 2287343806}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.162:6811', 'nonce': 2287343806}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.162:6810', 'nonce': 2287343806}]}, 'public_addr': '172.21.15.162:6808/2287343806', 'cluster_addr': '172.21.15.162:6809/2287343806', 'heartbeat_back_addr': '172.21.15.162:6811/2287343806', 'heartbeat_front_addr': '172.21.15.162:6810/2287343806', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '60aa5206-cace-40f8-8d1a-10ec28d97096', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 22, 'up_thru': 122, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.177:6813', 'nonce': 2110809906}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.177:6814', 'nonce': 2110809906}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.177:6816', 'nonce': 2110809906}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.177:6815', 'nonce': 2110809906}]}, 'public_addr': '172.21.15.177:6813/2110809906', 'cluster_addr': '172.21.15.177:6814/2110809906', 'heartbeat_back_addr': '172.21.15.177:6816/2110809906', 'heartbeat_front_addr': '172.21.15.177:6815/2110809906', 'state': ['exists', 'up']}, {'osd': 8, 'uuid': 'f3d99325-48ff-403c-8662-1d7745f7c97d', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 22, 'up_thru': 172, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.1:6804', 'nonce': 2766567910}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.1:6805', 'nonce': 2766567910}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.1:6807', 'nonce': 2766567910}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.1:6806', 'nonce': 2766567910}]}, 'public_addr': '172.21.15.1:6804/2766567910', 'cluster_addr': '172.21.15.1:6805/2766567910', 'heartbeat_back_addr': '172.21.15.1:6807/2766567910', 'heartbeat_front_addr': '172.21.15.1:6806/2766567910', 'state': ['exists', 'up']}, {'osd': 9, 'uuid': '88511e24-3675-4551-968c-1a52e7e28f96', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 23, 'up_thru': 191, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.92:6812', 'nonce': 2488187511}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.92:6813', 'nonce': 2488187511}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.92:6815', 'nonce': 2488187511}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.92:6814', 'nonce': 2488187511}]}, 'public_addr': '172.21.15.92:6812/2488187511', 'cluster_addr': '172.21.15.92:6813/2488187511', 'heartbeat_back_addr': '172.21.15.92:6815/2488187511', 'heartbeat_front_addr': '172.21.15.92:6814/2488187511', 'state': ['exists', 'up']}, {'osd': 10, 'uuid': 'acf4c64d-99af-42be-8406-413c86f54670', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 22, 'up_thru': 205, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.162:6804', 'nonce': 1450266459}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.162:6805', 'nonce': 1450266459}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.162:6807', 'nonce': 1450266459}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.162:6806', 'nonce': 1450266459}]}, 'public_addr': '172.21.15.162:6804/1450266459', 'cluster_addr': '172.21.15.162:6805/1450266459', 'heartbeat_back_addr': '172.21.15.162:6807/1450266459', 'heartbeat_front_addr': '172.21.15.162:6806/1450266459', 'state': ['exists', 'up']}, {'osd': 11, 'uuid': 'a4bfdfbe-b586-4080-8e09-6f26428e59b5', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 22, 'up_thru': 143, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.177:6809', 'nonce': 3239521954}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.177:6810', 'nonce': 3239521954}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.177:6812', 'nonce': 3239521954}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.177:6811', 'nonce': 3239521954}]}, 'public_addr': '172.21.15.177:6809/3239521954', 'cluster_addr': '172.21.15.177:6810/3239521954', 'heartbeat_back_addr': '172.21.15.177:6812/3239521954', 'heartbeat_front_addr': '172.21.15.177:6811/3239521954', 'state': ['exists', 'up']}, {'osd': 12, 'uuid': 'dc571fb6-d8b5-465b-bb25-afa7d0c47b5a', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 22, 'up_thru': 141, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.1:6812', 'nonce': 4267062742}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.1:6813', 'nonce': 4267062742}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.1:6815', 'nonce': 4267062742}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.1:6814', 'nonce': 4267062742}]}, 'public_addr': '172.21.15.1:6812/4267062742', 'cluster_addr': '172.21.15.1:6813/4267062742', 'heartbeat_back_addr': '172.21.15.1:6815/4267062742', 'heartbeat_front_addr': '172.21.15.1:6814/4267062742', 'state': ['exists', 'up']}, {'osd': 13, 'uuid': '5d321d4b-1b26-47fe-bf84-25751b8cde7d', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 23, 'up_thru': 205, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.92:6808', 'nonce': 1759585046}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.92:6809', 'nonce': 1759585046}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.92:6811', 'nonce': 1759585046}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.92:6810', 'nonce': 1759585046}]}, 'public_addr': '172.21.15.92:6808/1759585046', 'cluster_addr': '172.21.15.92:6809/1759585046', 'heartbeat_back_addr': '172.21.15.92:6811/1759585046', 'heartbeat_front_addr': '172.21.15.92:6810/1759585046', 'state': ['exists', 'up']}, {'osd': 14, 'uuid': 'fc3e2499-0c2a-414d-a25e-2628876ae630', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 22, 'last_clean_end': 120, 'up_from': 142, 'up_thru': 201, 'down_at': 121, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.162:6800', 'nonce': 3999392664}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.162:6801', 'nonce': 3999392664}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.162:6803', 'nonce': 3999392664}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.162:6802', 'nonce': 3999392664}]}, 'public_addr': '172.21.15.162:6800/3999392664', 'cluster_addr': '172.21.15.162:6801/3999392664', 'heartbeat_back_addr': '172.21.15.162:6803/3999392664', 'heartbeat_front_addr': '172.21.15.162:6802/3999392664', 'state': ['exists', 'up']}, {'osd': 15, 'uuid': 'd2f69775-edfe-4527-91ec-f7c7e558bec8', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 22, 'up_thru': 122, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.177:6805', 'nonce': 1294337172}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.177:6806', 'nonce': 1294337172}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.177:6808', 'nonce': 1294337172}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.177:6807', 'nonce': 1294337172}]}, 'public_addr': '172.21.15.177:6805/1294337172', 'cluster_addr': '172.21.15.177:6806/1294337172', 'heartbeat_back_addr': '172.21.15.177:6808/1294337172', 'heartbeat_front_addr': '172.21.15.177:6807/1294337172', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-13T11:26:19.725882+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-13T11:25:47.780833+0000', 'dead_epoch': 30}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-13T11:26:18.925695+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-13T11:26:17.200385+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-13T11:25:59.330894+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-13T11:26:21.996515+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-13T11:26:16.679806+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-13T11:26:01.857996+0000', 'dead_epoch': 0}, {'osd': 8, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-13T11:26:14.379046+0000', 'dead_epoch': 0}, {'osd': 9, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-13T11:26:09.278966+0000', 'dead_epoch': 0}, {'osd': 10, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-13T11:26:02.999347+0000', 'dead_epoch': 0}, {'osd': 11, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-13T11:26:27.208176+0000', 'dead_epoch': 0}, {'osd': 12, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-13T11:25:47.548386+0000', 'dead_epoch': 0}, {'osd': 13, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-13T11:25:45.821466+0000', 'dead_epoch': 0}, {'osd': 14, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-13T11:26:03.927526+0000', 'dead_epoch': 121}, {'osd': 15, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-13T11:25:57.987079+0000', 'dead_epoch': 0}], 'pg_upmap': [{'pgid': '2.2', 'osds': [2, 8]}, {'pgid': '3.e', 'osds': [0, 1]}, {'pgid': '3.22', 'osds': [14, 12]}], 'pg_upmap_items': [], 'pg_upmap_primaries': [], 'pg_temp': [{'pgid': '3.2', 'osds': [3, 12]}, {'pgid': '3.12', 'osds': [5, 7]}, {'pgid': '3.14', 'osds': [3, 15]}, {'pgid': '3.15', 'osds': [0, 3]}, {'pgid': '3.1f', 'osds': [10, 5]}, {'pgid': '3.22', 'osds': [0, 14]}], 'primary_temp': [], 'blocklist': {}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-09-13T11:26:33.767 INFO:tasks.thrashosds.thrasher:Clearing pg_upmap on 2.2 2024-09-13T11:26:33.767 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd rm-pg-upmap 2.2 2024-09-13T11:26:33.768 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-13T11:26:33.768 INFO:teuthology.orchestra.run.smithi001.stdout: "ops": [ 2024-09-13T11:26:33.768 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.768 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:251927 3.1a 3:5e9da227:::benchmark_data_smithi177_31116_object111624:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e211)", 2024-09-13T11:26:33.768 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:26:33.674402+0000", 2024-09-13T11:26:33.768 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 0.00052791000000000003, 2024-09-13T11:26:33.768 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00055221000000000003, 2024-09-13T11:26:33.768 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:26:33.768 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:26:33.768 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "waiting for sub ops", 2024-09-13T11:26:33.769 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:26:33.769 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:26:33.769 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:33.769 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 251927 2024-09-13T11:26:33.769 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.769 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:26:33.769 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.769 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:26:33.769 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:33.674402+0000", 2024-09-13T11:26:33.769 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:33.769 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.769 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.769 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:26:33.769 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:33.674402+0000", 2024-09-13T11:26:33.770 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:33.770 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.770 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.770 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:26:33.770 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:33.674404+0000", 2024-09-13T11:26:33.770 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.007e-06 2024-09-13T11:26:33.770 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.770 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.770 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:26:33.770 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:33.674410+0000", 2024-09-13T11:26:33.770 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.7250000000000002e-06 2024-09-13T11:26:33.770 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.770 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.770 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:26:33.771 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:33.674427+0000", 2024-09-13T11:26:33.771 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.6769000000000002e-05 2024-09-13T11:26:33.771 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.771 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.771 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:26:33.771 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:33.674436+0000", 2024-09-13T11:26:33.771 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.833e-06 2024-09-13T11:26:33.771 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.771 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.771 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:26:33.771 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:33.674494+0000", 2024-09-13T11:26:33.771 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.8777999999999999e-05 2024-09-13T11:26:33.771 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.771 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.771 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:26:33.772 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:33.674595+0000", 2024-09-13T11:26:33.772 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00010049599999999999 2024-09-13T11:26:33.772 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:33.772 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:33.772 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 9", 2024-09-13T11:26:33.772 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:33.674697+0000", 2024-09-13T11:26:33.772 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000101758 2024-09-13T11:26:33.772 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:33.772 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:33.772 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:33.772 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:33.772 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-09-13T11:26:33.772 INFO:teuthology.orchestra.run.smithi001.stdout: "num_ops": 1 2024-09-13T11:26:33.773 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-13T11:26:33.773 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-13T11:26:33.773 INFO:tasks.ceph.osd.12.smithi001.stderr:2024-09-13T11:26:33.762+0000 7f892c6e6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26314) UID: 0 2024-09-13T11:26:33.866 INFO:tasks.ceph.osd.14.smithi162.stderr:2024-09-13T11:26:33.863+0000 7f3df5903640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33154) UID: 0 2024-09-13T11:26:33.956 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-13T11:26:33.956 INFO:teuthology.orchestra.run.smithi001.stdout: "ops": [], 2024-09-13T11:26:33.957 INFO:teuthology.orchestra.run.smithi001.stdout: "complaint_time": 30, 2024-09-13T11:26:33.957 INFO:teuthology.orchestra.run.smithi001.stdout: "num_blocked_ops": 0 2024-09-13T11:26:33.957 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-13T11:26:33.967 INFO:tasks.ceph.osd.1.smithi092.stderr:2024-09-13T11:26:33.964+0000 7fe7789c5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33718) UID: 0 2024-09-13T11:26:33.977 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-13T11:26:34.028 INFO:teuthology.orchestra.run.smithi001.stdout:osd.0: {} 2024-09-13T11:26:34.028 INFO:teuthology.orchestra.run.smithi001.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T11:26:34.068 INFO:teuthology.orchestra.run.smithi001.stdout:osd.1: {} 2024-09-13T11:26:34.068 INFO:tasks.ceph.osd.1.smithi092.stderr:2024-09-13T11:26:34.064+0000 7fe7789c5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33718) UID: 0 2024-09-13T11:26:34.068 INFO:teuthology.orchestra.run.smithi001.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T11:26:34.113 INFO:teuthology.orchestra.run.smithi001.stdout:osd.2: {} 2024-09-13T11:26:34.113 INFO:teuthology.orchestra.run.smithi001.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T11:26:34.146 INFO:teuthology.orchestra.run.smithi001.stdout:osd.3: {} 2024-09-13T11:26:34.147 INFO:teuthology.orchestra.run.smithi001.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T11:26:34.158 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-13T11:26:34.158 INFO:teuthology.orchestra.run.smithi001.stdout: "size": 20, 2024-09-13T11:26:34.158 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 600, 2024-09-13T11:26:34.158 INFO:teuthology.orchestra.run.smithi001.stdout: "ops": [ 2024-09-13T11:26:34.158 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:34.158 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDPGPush(3.1e 89/78 [PushOp(3:7a07f38c:::benchmark_data_smithi177_31116_object20966:head, version: 54'1402, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:7a07f38c:::benchmark_data_smithi177_31116_object20966:head@54'1402, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:26:34.158 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:46.645251+0000", 2024-09-13T11:26:34.158 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 167.51093867899999, 2024-09-13T11:26:34.158 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.17670883800000001, 2024-09-13T11:26:34.158 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:26:34.158 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:26:34.158 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:26:34.159 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:26:34.159 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:34.159 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:26:34.159 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.645251+0000", 2024-09-13T11:26:34.159 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:34.159 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:34.159 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:34.159 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:26:34.159 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.645251+0000", 2024-09-13T11:26:34.159 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:34.159 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:34.159 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:34.160 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:26:34.160 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.645253+0000", 2024-09-13T11:26:34.160 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.1399999999999998e-06 2024-09-13T11:26:34.160 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:34.160 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:34.160 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:26:34.160 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.645295+0000", 2024-09-13T11:26:34.160 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.2323000000000001e-05 2024-09-13T11:26:34.160 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:34.160 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:34.160 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:26:34.160 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.820731+0000", 2024-09-13T11:26:34.160 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.17543607999999999 2024-09-13T11:26:34.161 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:34.161 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:34.161 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:26:34.161 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.820750+0000", 2024-09-13T11:26:34.161 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.8793000000000001e-05 2024-09-13T11:26:34.161 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:34.161 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:34.161 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:26:34.161 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.820914+0000", 2024-09-13T11:26:34.161 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000164062 2024-09-13T11:26:34.161 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:34.162 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:34.162 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:26:34.162 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.820939+0000", 2024-09-13T11:26:34.162 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.4672e-05 2024-09-13T11:26:34.162 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:34.162 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:34.162 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:26:34.162 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.821960+0000", 2024-09-13T11:26:34.162 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.001020768 2024-09-13T11:26:34.162 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:34.162 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:34.162 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:34.163 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:34.163 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:34.163 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDPGPush(3.1a 102/78 [PushOp(3:5df50766:::benchmark_data_smithi177_31116_object50543:head, version: 72'3152, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:5df50766:::benchmark_data_smithi177_31116_object50543:head@72'3152, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:26:34.163 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:24:00.499311+0000", 2024-09-13T11:26:34.163 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 153.65687859400001, 2024-09-13T11:26:34.163 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.26146664800000002, 2024-09-13T11:26:34.163 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:26:34.163 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:26:34.163 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:26:34.163 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:26:34.163 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:34.163 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:26:34.163 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:00.499311+0000", 2024-09-13T11:26:34.164 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:34.164 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:34.164 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:34.164 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:26:34.164 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:00.499311+0000", 2024-09-13T11:26:34.164 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:34.164 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:34.164 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:34.164 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:26:34.164 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:00.499320+0000", 2024-09-13T11:26:34.164 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.6400000000000003e-06 2024-09-13T11:26:34.164 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:34.164 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:34.164 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:26:34.165 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:00.499384+0000", 2024-09-13T11:26:34.165 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.4554999999999994e-05 2024-09-13T11:26:34.165 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:34.165 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:34.165 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:26:34.165 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:00.759713+0000", 2024-09-13T11:26:34.165 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.260328742 2024-09-13T11:26:34.165 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:34.165 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:34.165 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:26:34.165 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:00.759742+0000", 2024-09-13T11:26:34.165 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.9298e-05 2024-09-13T11:26:34.165 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:34.165 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:34.166 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:26:34.166 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:00.759871+0000", 2024-09-13T11:26:34.166 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000128992 2024-09-13T11:26:34.166 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:34.166 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:34.166 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:26:34.166 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:00.759884+0000", 2024-09-13T11:26:34.166 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2687999999999999e-05 2024-09-13T11:26:34.166 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:34.166 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:34.166 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:26:34.166 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:00.760778+0000", 2024-09-13T11:26:34.166 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00089373299999999996 2024-09-13T11:26:34.166 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:34.167 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:34.167 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:34.167 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:34.167 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:34.167 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4665.0:82769 3.1e e105/102 3:7a4e88b5:::benchmark_data_smithi177_31116_object82768:head v 105'4381, mlcod=105'4379)", 2024-09-13T11:26:34.167 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:24:03.745047+0000", 2024-09-13T11:26:34.167 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 150.411142369, 2024-09-13T11:26:34.167 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.213806145, 2024-09-13T11:26:34.167 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:26:34.167 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:26:34.167 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:34.167 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:26:34.167 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:34.168 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:26:34.168 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:03.745047+0000", 2024-09-13T11:26:34.168 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:34.168 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:34.168 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:34.168 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:26:34.168 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:03.745047+0000", 2024-09-13T11:26:34.168 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:34.168 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:34.168 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:34.168 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:26:34.168 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:03.745049+0000", 2024-09-13T11:26:34.168 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.5710000000000001e-06 2024-09-13T11:26:34.168 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:34.169 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:34.169 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:26:34.169 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:03.745107+0000", 2024-09-13T11:26:34.169 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.8799999999999999e-05 2024-09-13T11:26:34.169 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:34.169 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:34.169 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:26:34.169 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:03.955145+0000", 2024-09-13T11:26:34.169 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.210037735 2024-09-13T11:26:34.169 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:34.169 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:34.169 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:26:34.169 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:03.955163+0000", 2024-09-13T11:26:34.170 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.7520000000000002e-05 2024-09-13T11:26:34.170 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:34.170 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:34.170 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:26:34.170 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:03.955473+0000", 2024-09-13T11:26:34.170 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00031014399999999999 2024-09-13T11:26:34.170 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:34.170 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:34.170 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:26:34.170 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:03.955538+0000", 2024-09-13T11:26:34.170 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.5059000000000002e-05 2024-09-13T11:26:34.170 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:34.170 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:34.171 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:26:34.171 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:03.958753+0000", 2024-09-13T11:26:34.171 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0032152769999999999 2024-09-13T11:26:34.171 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:34.171 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:34.171 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:26:34.171 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:03.958853+0000", 2024-09-13T11:26:34.171 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000100039 2024-09-13T11:26:34.171 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:34.171 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:34.171 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:34.171 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:34.171 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:34.171 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:98582 3.1a 3:59f9cd75:::benchmark_data_smithi177_31116_object98581:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113)", 2024-09-13T11:26:34.172 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:24:15.431226+0000", 2024-09-13T11:26:34.172 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 138.72496390000001, 2024-09-13T11:26:34.172 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.19820755800000001, 2024-09-13T11:26:34.172 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:26:34.172 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:26:34.172 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:34.172 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:26:34.172 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:26:34.172 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:34.172 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 98582 2024-09-13T11:26:34.172 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:34.172 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:26:34.172 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:34.173 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:26:34.173 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.431226+0000", 2024-09-13T11:26:34.173 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:34.173 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:34.173 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:34.173 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:26:34.173 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.431226+0000", 2024-09-13T11:26:34.173 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:34.173 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:34.173 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:34.173 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:26:34.173 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.431229+0000", 2024-09-13T11:26:34.174 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.7100000000000001e-06 2024-09-13T11:26:34.174 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:34.174 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:34.174 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:26:34.174 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.431334+0000", 2024-09-13T11:26:34.174 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00010506900000000001 2024-09-13T11:26:34.174 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:34.174 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:34.174 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:26:34.174 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.431364+0000", 2024-09-13T11:26:34.174 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.9607e-05 2024-09-13T11:26:34.174 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:34.174 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:34.174 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:26:34.175 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.431388+0000", 2024-09-13T11:26:34.175 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.3808999999999999e-05 2024-09-13T11:26:34.175 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:34.175 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:34.175 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:26:34.175 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.431483+0000", 2024-09-13T11:26:34.175 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.5402999999999994e-05 2024-09-13T11:26:34.175 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:34.175 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:34.175 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:26:34.175 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.431642+0000", 2024-09-13T11:26:34.175 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000158412 2024-09-13T11:26:34.175 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:34.176 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:34.176 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 9", 2024-09-13T11:26:34.176 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.431796+0000", 2024-09-13T11:26:34.176 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00015467899999999999 2024-09-13T11:26:34.176 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:34.176 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:34.176 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:26:34.176 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.433731+0000", 2024-09-13T11:26:34.176 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.001934945 2024-09-13T11:26:34.176 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:34.176 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:34.176 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:34.176 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.629245+0000", 2024-09-13T11:26:34.176 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.19551390199999999 2024-09-13T11:26:34.177 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:34.177 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:34.177 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:26:34.177 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.629346+0000", 2024-09-13T11:26:34.177 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000100869 2024-09-13T11:26:34.177 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:34.177 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:34.177 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:26:34.177 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.629433+0000", 2024-09-13T11:26:34.177 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.7152999999999997e-05 2024-09-13T11:26:34.177 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:34.177 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:34.177 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:34.178 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:34.178 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:34.178 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:99105 3.1a 3:5ac5090c:::benchmark_data_smithi177_31116_object99104:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113)", 2024-09-13T11:26:34.178 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:24:15.962466+0000", 2024-09-13T11:26:34.178 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 138.193723611, 2024-09-13T11:26:34.178 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.23832849, 2024-09-13T11:26:34.178 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:26:34.178 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:26:34.178 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:34.178 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:26:34.178 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:26:34.178 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:34.178 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 99105 2024-09-13T11:26:34.178 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:34.179 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:26:34.179 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:34.179 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:26:34.179 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.962466+0000", 2024-09-13T11:26:34.179 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:34.179 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:34.179 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:34.179 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:26:34.179 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.962466+0000", 2024-09-13T11:26:34.179 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:34.179 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:34.179 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:34.179 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:26:34.179 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.962471+0000", 2024-09-13T11:26:34.180 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.9570000000000001e-06 2024-09-13T11:26:34.180 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:34.180 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:34.180 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:26:34.180 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.962547+0000", 2024-09-13T11:26:34.180 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.5947999999999999e-05 2024-09-13T11:26:34.180 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:34.180 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:34.180 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:26:34.180 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.962576+0000", 2024-09-13T11:26:34.180 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.9547999999999999e-05 2024-09-13T11:26:34.180 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:34.180 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:34.180 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:26:34.181 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.962594+0000", 2024-09-13T11:26:34.181 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.8023e-05 2024-09-13T11:26:34.181 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:34.181 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:34.181 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:26:34.181 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.962770+0000", 2024-09-13T11:26:34.181 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00017551099999999999 2024-09-13T11:26:34.181 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:34.181 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:34.181 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:26:34.181 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.963114+0000", 2024-09-13T11:26:34.181 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00034440399999999998 2024-09-13T11:26:34.181 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:34.181 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:34.182 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 9", 2024-09-13T11:26:34.182 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.963351+0000", 2024-09-13T11:26:34.182 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000237209 2024-09-13T11:26:34.182 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:34.182 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:34.182 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:26:34.182 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.966345+0000", 2024-09-13T11:26:34.182 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0029933519999999999 2024-09-13T11:26:34.182 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:34.182 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:34.182 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:34.182 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:16.200724+0000", 2024-09-13T11:26:34.182 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.23437934599999999 2024-09-13T11:26:34.182 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:34.183 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:34.183 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:26:34.183 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:16.200761+0000", 2024-09-13T11:26:34.183 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.6406999999999999e-05 2024-09-13T11:26:34.183 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:34.183 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:34.183 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:26:34.183 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:16.200794+0000", 2024-09-13T11:26:34.183 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.3785000000000003e-05 2024-09-13T11:26:34.183 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:34.183 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:34.183 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:34.183 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:34.183 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:34.184 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:129921 3.1a 3:5fb9088b:::benchmark_data_smithi177_31116_object129920:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e134)", 2024-09-13T11:26:34.184 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:24:47.177556+0000", 2024-09-13T11:26:34.184 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 106.978633332, 2024-09-13T11:26:34.184 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.33804469999999998, 2024-09-13T11:26:34.184 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:26:34.184 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:26:34.184 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:34.184 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:26:34.184 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:26:34.184 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:34.184 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 129921 2024-09-13T11:26:34.184 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:34.184 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:26:34.184 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:34.185 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:26:34.185 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:47.177556+0000", 2024-09-13T11:26:34.185 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:34.185 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:34.185 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:34.185 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:26:34.185 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:47.177556+0000", 2024-09-13T11:26:34.185 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:34.185 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:34.185 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:34.185 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:26:34.185 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:47.177559+0000", 2024-09-13T11:26:34.185 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.0790000000000001e-06 2024-09-13T11:26:34.186 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:34.186 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:34.186 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:26:34.186 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:47.177589+0000", 2024-09-13T11:26:34.186 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.0173999999999999e-05 2024-09-13T11:26:34.186 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:34.186 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:34.186 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:26:34.186 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:47.177606+0000", 2024-09-13T11:26:34.186 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.6475999999999999e-05 2024-09-13T11:26:34.186 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:34.186 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:34.186 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:26:34.186 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:47.177615+0000", 2024-09-13T11:26:34.187 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.5729999999999997e-06 2024-09-13T11:26:34.187 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:34.187 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:34.187 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:26:34.187 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:47.177680+0000", 2024-09-13T11:26:34.187 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.4349000000000003e-05 2024-09-13T11:26:34.187 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:34.187 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:34.187 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:26:34.187 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:47.177794+0000", 2024-09-13T11:26:34.187 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000114119 2024-09-13T11:26:34.187 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:34.187 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:34.187 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 9", 2024-09-13T11:26:34.188 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:47.177895+0000", 2024-09-13T11:26:34.188 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000101078 2024-09-13T11:26:34.188 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:34.188 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:34.188 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:26:34.188 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:47.179634+0000", 2024-09-13T11:26:34.188 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.001738606 2024-09-13T11:26:34.188 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:34.188 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:34.188 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:34.188 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:47.515495+0000", 2024-09-13T11:26:34.188 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.33586143000000002 2024-09-13T11:26:34.188 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:34.189 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:34.189 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:26:34.189 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:47.515551+0000", 2024-09-13T11:26:34.189 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.6048999999999998e-05 2024-09-13T11:26:34.189 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:34.189 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:34.189 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:26:34.189 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:47.515601+0000", 2024-09-13T11:26:34.189 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.9767e-05 2024-09-13T11:26:34.189 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:34.189 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:34.189 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:34.189 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:34.190 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:34.190 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4665.0:140742 3.1e e144/102 3:79c694a0:::benchmark_data_smithi177_31116_object439:head v 144'6204, mlcod=144'6200)", 2024-09-13T11:26:34.190 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:24:58.321534+0000", 2024-09-13T11:26:34.190 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 95.834655807999994, 2024-09-13T11:26:34.190 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.23629921500000001, 2024-09-13T11:26:34.190 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:26:34.190 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:26:34.190 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:34.190 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:26:34.190 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:34.190 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:26:34.190 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.321534+0000", 2024-09-13T11:26:34.190 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:34.190 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:34.191 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:34.191 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:26:34.191 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.321534+0000", 2024-09-13T11:26:34.191 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:34.191 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:34.191 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:34.191 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:26:34.191 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.321535+0000", 2024-09-13T11:26:34.191 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.159e-06 2024-09-13T11:26:34.191 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:34.191 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:34.191 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:26:34.191 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.321545+0000", 2024-09-13T11:26:34.191 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.9219999999999992e-06 2024-09-13T11:26:34.192 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:34.192 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:34.192 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:26:34.192 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.553719+0000", 2024-09-13T11:26:34.192 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.23217401700000001 2024-09-13T11:26:34.192 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:34.192 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:34.192 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:26:34.192 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.553734+0000", 2024-09-13T11:26:34.192 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.5344000000000001e-05 2024-09-13T11:26:34.192 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:34.192 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:34.192 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:26:34.193 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.553858+0000", 2024-09-13T11:26:34.193 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00012357500000000001 2024-09-13T11:26:34.193 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:34.193 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:34.193 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:26:34.193 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.553903+0000", 2024-09-13T11:26:34.193 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.5374999999999998e-05 2024-09-13T11:26:34.193 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:34.193 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:34.193 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:26:34.193 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.557719+0000", 2024-09-13T11:26:34.193 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0038158670000000001 2024-09-13T11:26:34.193 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:34.193 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:34.194 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:26:34.194 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.557833+0000", 2024-09-13T11:26:34.194 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000113956 2024-09-13T11:26:34.194 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:34.194 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:34.194 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:34.194 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:34.194 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:34.194 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4665.0:140796 3.1e e144/102 3:7987a652:::benchmark_data_smithi177_31116_object493:head v 144'6205, mlcod=144'6202)", 2024-09-13T11:26:34.194 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:24:58.553657+0000", 2024-09-13T11:26:34.194 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 95.602532831999994, 2024-09-13T11:26:34.194 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.25308323100000002, 2024-09-13T11:26:34.194 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:26:34.194 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:26:34.195 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:34.195 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:26:34.195 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:34.195 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:26:34.195 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.553657+0000", 2024-09-13T11:26:34.195 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:34.195 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:34.195 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:34.195 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:26:34.195 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.553657+0000", 2024-09-13T11:26:34.195 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:34.195 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:34.195 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:34.195 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:26:34.196 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.553660+0000", 2024-09-13T11:26:34.196 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.7519999999999998e-06 2024-09-13T11:26:34.196 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:34.196 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:34.196 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:26:34.196 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.553678+0000", 2024-09-13T11:26:34.196 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.7156e-05 2024-09-13T11:26:34.196 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:34.196 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:34.196 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:26:34.196 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.804342+0000", 2024-09-13T11:26:34.196 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.25066436199999997 2024-09-13T11:26:34.196 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:34.196 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:34.197 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:26:34.197 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.804353+0000", 2024-09-13T11:26:34.197 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.1147e-05 2024-09-13T11:26:34.197 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:34.197 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:34.197 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:26:34.197 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.804433+0000", 2024-09-13T11:26:34.197 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.9586999999999995e-05 2024-09-13T11:26:34.197 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:34.197 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:34.197 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:26:34.197 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.804457+0000", 2024-09-13T11:26:34.197 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.4108e-05 2024-09-13T11:26:34.197 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:34.198 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:34.198 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:26:34.198 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.806684+0000", 2024-09-13T11:26:34.198 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0022271309999999998 2024-09-13T11:26:34.198 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:34.198 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:34.198 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:26:34.198 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.806740+0000", 2024-09-13T11:26:34.198 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.5988e-05 2024-09-13T11:26:34.198 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:34.198 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:34.198 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:34.198 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:34.199 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:34.199 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:149955 3.11 3:8fa29838:::benchmark_data_smithi177_31116_object9652:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e149)", 2024-09-13T11:26:34.199 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:25:05.905492+0000", 2024-09-13T11:26:34.199 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 88.250697458000005, 2024-09-13T11:26:34.199 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.204181955, 2024-09-13T11:26:34.199 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:26:34.199 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:26:34.199 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:34.199 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:26:34.199 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:26:34.199 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:34.199 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 149955 2024-09-13T11:26:34.199 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:34.200 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:26:34.200 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:34.200 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:26:34.200 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:05.905492+0000", 2024-09-13T11:26:34.200 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:34.200 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:34.200 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:34.200 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:26:34.200 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:05.905492+0000", 2024-09-13T11:26:34.200 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:34.200 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:34.200 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:34.200 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:26:34.201 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:05.905494+0000", 2024-09-13T11:26:34.201 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.9369999999999998e-06 2024-09-13T11:26:34.201 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:34.201 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:34.201 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:26:34.201 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:05.905500+0000", 2024-09-13T11:26:34.201 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.7039999999999999e-06 2024-09-13T11:26:34.201 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:34.201 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:34.201 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:26:34.201 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:05.905512+0000", 2024-09-13T11:26:34.201 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.257e-05 2024-09-13T11:26:34.201 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:34.202 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:34.202 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:26:34.202 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:05.905519+0000", 2024-09-13T11:26:34.202 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.5420000000000002e-06 2024-09-13T11:26:34.202 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:34.202 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:34.202 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:26:34.202 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:05.905584+0000", 2024-09-13T11:26:34.202 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.4819000000000004e-05 2024-09-13T11:26:34.202 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:34.202 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:34.202 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:26:34.202 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:05.905696+0000", 2024-09-13T11:26:34.202 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00011248599999999999 2024-09-13T11:26:34.203 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:34.203 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:34.203 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 11", 2024-09-13T11:26:34.203 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:05.905759+0000", 2024-09-13T11:26:34.203 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.2451000000000002e-05 2024-09-13T11:26:34.203 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:34.203 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:34.203 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:26:34.203 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:05.907584+0000", 2024-09-13T11:26:34.203 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0018250110000000001 2024-09-13T11:26:34.203 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:34.203 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:34.203 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:34.203 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:06.109588+0000", 2024-09-13T11:26:34.204 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.20200470300000001 2024-09-13T11:26:34.204 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:34.204 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:34.204 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:26:34.204 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:06.109637+0000", 2024-09-13T11:26:34.204 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.8439000000000002e-05 2024-09-13T11:26:34.204 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:34.204 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:34.204 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:26:34.204 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:06.109674+0000", 2024-09-13T11:26:34.204 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.7292999999999997e-05 2024-09-13T11:26:34.204 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:34.205 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:34.205 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:34.205 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:34.205 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:34.205 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:150414 3.11 3:8f72eed4:::benchmark_data_smithi177_31116_object10111:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e149)", 2024-09-13T11:26:34.205 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:25:06.348925+0000", 2024-09-13T11:26:34.205 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 87.807264345999997, 2024-09-13T11:26:34.205 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.17250813300000001, 2024-09-13T11:26:34.205 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:26:34.205 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:26:34.205 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:34.205 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:26:34.205 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:26:34.205 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:34.206 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 150414 2024-09-13T11:26:34.206 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:34.206 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:26:34.206 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:34.206 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:26:34.206 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:06.348925+0000", 2024-09-13T11:26:34.206 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:34.206 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:34.206 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:34.206 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:26:34.206 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:06.348925+0000", 2024-09-13T11:26:34.206 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:34.206 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:34.206 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:34.206 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:26:34.207 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:06.348927+0000", 2024-09-13T11:26:34.207 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.7430000000000001e-06 2024-09-13T11:26:34.207 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:34.207 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:34.207 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:26:34.207 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:06.348931+0000", 2024-09-13T11:26:34.207 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.5210000000000004e-06 2024-09-13T11:26:34.207 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:34.207 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:34.207 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:26:34.207 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:06.348943+0000", 2024-09-13T11:26:34.207 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.1168e-05 2024-09-13T11:26:34.207 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:34.208 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:34.208 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:26:34.208 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:06.348950+0000", 2024-09-13T11:26:34.208 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.1430000000000001e-06 2024-09-13T11:26:34.208 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:34.208 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:34.208 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:26:34.208 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:06.349032+0000", 2024-09-13T11:26:34.208 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.2517999999999998e-05 2024-09-13T11:26:34.209 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:34.209 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:34.209 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:26:34.209 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:06.349179+0000", 2024-09-13T11:26:34.209 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000146935 2024-09-13T11:26:34.209 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:34.209 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:34.209 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 11", 2024-09-13T11:26:34.209 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:06.349290+0000", 2024-09-13T11:26:34.209 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000110927 2024-09-13T11:26:34.209 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:34.209 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:34.209 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:26:34.209 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:06.351686+0000", 2024-09-13T11:26:34.210 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0023963560000000001 2024-09-13T11:26:34.210 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:34.210 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:34.210 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:34.210 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:06.521374+0000", 2024-09-13T11:26:34.210 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.169687216 2024-09-13T11:26:34.210 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:34.210 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:34.210 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:26:34.210 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:06.521406+0000", 2024-09-13T11:26:34.210 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.1841000000000002e-05 2024-09-13T11:26:34.210 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:34.210 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:34.210 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:26:34.211 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:06.521433+0000", 2024-09-13T11:26:34.211 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.7764999999999999e-05 2024-09-13T11:26:34.211 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:34.211 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:34.211 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:34.211 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:34.211 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:34.211 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:168772 3.1a 3:5be647e9:::benchmark_data_smithi177_31116_object28469:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e161)", 2024-09-13T11:26:34.211 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:25:20.636095+0000", 2024-09-13T11:26:34.211 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 73.520094936999996, 2024-09-13T11:26:34.211 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.17968553000000001, 2024-09-13T11:26:34.211 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:26:34.211 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:26:34.211 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:34.212 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:26:34.212 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:26:34.212 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:34.212 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 168772 2024-09-13T11:26:34.212 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:34.212 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:26:34.212 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:34.212 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:26:34.212 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:20.636095+0000", 2024-09-13T11:26:34.212 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:34.212 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:34.212 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:34.212 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:26:34.213 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:20.636095+0000", 2024-09-13T11:26:34.213 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:34.213 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:34.213 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:34.213 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:26:34.213 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:20.636096+0000", 2024-09-13T11:26:34.213 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.852e-06 2024-09-13T11:26:34.213 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:34.213 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:34.213 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:26:34.213 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:20.636101+0000", 2024-09-13T11:26:34.213 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.0329999999999999e-06 2024-09-13T11:26:34.213 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:34.214 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:34.214 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:26:34.214 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:20.636114+0000", 2024-09-13T11:26:34.214 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2816999999999999e-05 2024-09-13T11:26:34.214 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:34.214 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:34.214 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:26:34.214 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:20.636120+0000", 2024-09-13T11:26:34.214 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.7429999999999999e-06 2024-09-13T11:26:34.214 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:34.214 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:34.214 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:26:34.214 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:20.636194+0000", 2024-09-13T11:26:34.214 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.3520000000000001e-05 2024-09-13T11:26:34.215 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:34.215 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:34.215 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:26:34.215 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:20.636369+0000", 2024-09-13T11:26:34.215 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00017552 2024-09-13T11:26:34.215 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:34.215 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:34.215 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 9", 2024-09-13T11:26:34.215 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:20.636519+0000", 2024-09-13T11:26:34.215 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000149949 2024-09-13T11:26:34.215 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:34.215 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:34.215 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:26:34.215 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:20.638735+0000", 2024-09-13T11:26:34.216 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0022156509999999999 2024-09-13T11:26:34.216 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:34.216 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:34.216 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:34.216 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:20.815699+0000", 2024-09-13T11:26:34.216 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.176964343 2024-09-13T11:26:34.216 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:34.216 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:34.216 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:26:34.216 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:20.815748+0000", 2024-09-13T11:26:34.216 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.9518000000000003e-05 2024-09-13T11:26:34.216 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:34.216 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:34.217 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:26:34.217 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:20.815780+0000", 2024-09-13T11:26:34.217 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.1584000000000001e-05 2024-09-13T11:26:34.217 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:34.217 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:34.217 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:34.217 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:34.217 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:34.217 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:168782 3.1a 3:590b82e1:::benchmark_data_smithi177_31116_object28479:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e161)", 2024-09-13T11:26:34.217 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:25:20.641567+0000", 2024-09-13T11:26:34.217 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 73.514622105000001, 2024-09-13T11:26:34.217 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.19100501, 2024-09-13T11:26:34.217 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:26:34.217 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:26:34.218 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:34.218 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:26:34.218 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:26:34.218 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:34.218 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 168782 2024-09-13T11:26:34.218 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:34.218 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:26:34.218 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:34.218 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:26:34.218 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:20.641567+0000", 2024-09-13T11:26:34.218 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:34.218 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:34.218 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:34.218 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:26:34.218 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:20.641567+0000", 2024-09-13T11:26:34.219 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:34.219 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:34.219 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:34.219 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:26:34.219 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:20.641569+0000", 2024-09-13T11:26:34.219 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.869e-06 2024-09-13T11:26:34.219 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:34.219 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:34.219 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:26:34.219 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:20.641574+0000", 2024-09-13T11:26:34.219 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.1680000000000003e-06 2024-09-13T11:26:34.219 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:34.219 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:34.219 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:26:34.220 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:20.641587+0000", 2024-09-13T11:26:34.220 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2858e-05 2024-09-13T11:26:34.220 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:34.220 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:34.220 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:26:34.220 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:20.641595+0000", 2024-09-13T11:26:34.220 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.6149999999999999e-06 2024-09-13T11:26:34.220 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:34.220 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:34.220 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:26:34.220 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:20.641655+0000", 2024-09-13T11:26:34.220 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.9673999999999998e-05 2024-09-13T11:26:34.220 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:34.220 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:34.221 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:26:34.221 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:20.641748+0000", 2024-09-13T11:26:34.221 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.3919999999999995e-05 2024-09-13T11:26:34.221 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:34.221 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:34.221 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 9", 2024-09-13T11:26:34.221 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:20.642990+0000", 2024-09-13T11:26:34.221 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0012415709999999999 2024-09-13T11:26:34.221 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:34.221 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:34.221 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:26:34.221 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:20.645016+0000", 2024-09-13T11:26:34.221 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.002025448 2024-09-13T11:26:34.222 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:34.222 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:34.222 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:34.222 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:20.832409+0000", 2024-09-13T11:26:34.222 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.18739319600000001 2024-09-13T11:26:34.222 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:34.222 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:34.222 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:26:34.222 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:20.832525+0000", 2024-09-13T11:26:34.222 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000116273 2024-09-13T11:26:34.222 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:34.222 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:34.222 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:26:34.222 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:20.832572+0000", 2024-09-13T11:26:34.223 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.7417999999999999e-05 2024-09-13T11:26:34.223 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:34.223 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:34.223 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:34.223 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:34.223 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:34.223 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:174971 3.1a 3:5a8f64c5:::benchmark_data_smithi177_31116_object34668:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e166)", 2024-09-13T11:26:34.223 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:25:25.422480+0000", 2024-09-13T11:26:34.223 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 68.733709493999996, 2024-09-13T11:26:34.223 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.18399536399999999, 2024-09-13T11:26:34.223 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:26:34.223 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:26:34.223 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:34.223 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:26:34.224 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:26:34.224 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:34.224 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 174971 2024-09-13T11:26:34.224 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:34.224 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:26:34.224 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:34.224 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:26:34.224 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:25.422480+0000", 2024-09-13T11:26:34.224 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:34.224 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:34.224 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:34.224 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:26:34.224 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:25.422480+0000", 2024-09-13T11:26:34.225 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:34.225 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:34.225 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:34.225 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:26:34.225 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:25.422483+0000", 2024-09-13T11:26:34.225 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.4529999999999998e-06 2024-09-13T11:26:34.225 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:34.225 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:34.225 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:26:34.225 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:25.422492+0000", 2024-09-13T11:26:34.225 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.8820000000000007e-06 2024-09-13T11:26:34.225 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:34.225 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:34.225 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:26:34.226 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:25.422515+0000", 2024-09-13T11:26:34.226 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.2756e-05 2024-09-13T11:26:34.226 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:34.226 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:34.226 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:26:34.226 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:25.422526+0000", 2024-09-13T11:26:34.226 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0684e-05 2024-09-13T11:26:34.226 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:34.226 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:34.226 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:26:34.226 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:25.422590+0000", 2024-09-13T11:26:34.226 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.4163000000000002e-05 2024-09-13T11:26:34.226 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:34.226 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:34.227 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:26:34.227 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:25.422675+0000", 2024-09-13T11:26:34.227 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.5311999999999999e-05 2024-09-13T11:26:34.227 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:34.227 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:34.227 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 9", 2024-09-13T11:26:34.227 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:25.422745+0000", 2024-09-13T11:26:34.227 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.0217999999999998e-05 2024-09-13T11:26:34.227 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:34.227 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:34.227 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:26:34.227 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:25.424910+0000", 2024-09-13T11:26:34.227 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0021641500000000001 2024-09-13T11:26:34.227 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:34.227 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:34.228 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:34.228 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:25.606366+0000", 2024-09-13T11:26:34.228 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.181456267 2024-09-13T11:26:34.228 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:34.228 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:34.228 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:26:34.228 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:25.606428+0000", 2024-09-13T11:26:34.228 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.2312999999999998e-05 2024-09-13T11:26:34.228 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:34.228 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:34.229 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:26:34.229 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:25.606475+0000", 2024-09-13T11:26:34.229 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.7166000000000002e-05 2024-09-13T11:26:34.229 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:34.229 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:34.229 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:34.229 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:34.229 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:34.229 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4665.0:191081 3.1e e175/102 3:7e784823:::benchmark_data_smithi177_31116_object50778:head v 175'7812, mlcod=175'7809)", 2024-09-13T11:26:34.229 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:25:38.547539+0000", 2024-09-13T11:26:34.230 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 55.608650208, 2024-09-13T11:26:34.230 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.213922061, 2024-09-13T11:26:34.230 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:26:34.230 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:26:34.230 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:34.230 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:26:34.230 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:34.230 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:26:34.230 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:38.547539+0000", 2024-09-13T11:26:34.230 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:34.230 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:34.230 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:34.230 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:26:34.230 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:38.547539+0000", 2024-09-13T11:26:34.231 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:34.231 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:34.231 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:34.231 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:26:34.231 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:38.547543+0000", 2024-09-13T11:26:34.231 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.3629999999999998e-06 2024-09-13T11:26:34.231 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:34.231 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:34.231 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:26:34.231 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:38.547558+0000", 2024-09-13T11:26:34.231 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.5043e-05 2024-09-13T11:26:34.231 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:34.231 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:34.231 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:26:34.232 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:38.756778+0000", 2024-09-13T11:26:34.232 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.209219936 2024-09-13T11:26:34.232 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:34.232 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:34.232 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:26:34.232 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:38.756790+0000", 2024-09-13T11:26:34.232 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2775999999999999e-05 2024-09-13T11:26:34.232 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:34.232 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:34.232 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:26:34.232 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:38.756934+0000", 2024-09-13T11:26:34.232 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00014401600000000001 2024-09-13T11:26:34.233 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:34.233 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:34.233 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:26:34.233 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:38.757001+0000", 2024-09-13T11:26:34.233 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.6716000000000003e-05 2024-09-13T11:26:34.233 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:34.233 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:34.233 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:26:34.233 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:38.761376+0000", 2024-09-13T11:26:34.233 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0043747320000000001 2024-09-13T11:26:34.233 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:34.233 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:34.233 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:26:34.233 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:38.761461+0000", 2024-09-13T11:26:34.234 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.5478999999999999e-05 2024-09-13T11:26:34.234 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:34.234 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:34.234 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:34.234 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:34.234 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:34.234 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4665.0:191087 3.1e e175/102 3:7a697202:::benchmark_data_smithi177_31116_object50784:head v 175'7813, mlcod=175'7809)", 2024-09-13T11:26:34.234 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:25:38.561827+0000", 2024-09-13T11:26:34.234 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 55.594361997999997, 2024-09-13T11:26:34.234 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.209395781, 2024-09-13T11:26:34.234 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:26:34.234 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:26:34.234 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:34.234 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:26:34.235 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:34.235 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:26:34.235 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:38.561827+0000", 2024-09-13T11:26:34.235 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:34.235 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:34.235 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:34.235 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:26:34.235 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:38.561827+0000", 2024-09-13T11:26:34.235 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:34.235 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:34.235 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:34.235 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:26:34.235 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:38.561837+0000", 2024-09-13T11:26:34.236 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.2329999999999995e-06 2024-09-13T11:26:34.236 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:34.236 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:34.236 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:26:34.236 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:38.561849+0000", 2024-09-13T11:26:34.236 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2593e-05 2024-09-13T11:26:34.236 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:34.236 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:34.236 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:26:34.236 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:38.766804+0000", 2024-09-13T11:26:34.236 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.20495488000000001 2024-09-13T11:26:34.236 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:34.236 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:34.237 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:26:34.237 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:38.766823+0000", 2024-09-13T11:26:34.237 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.9143e-05 2024-09-13T11:26:34.237 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:34.237 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:34.237 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:26:34.237 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:38.766973+0000", 2024-09-13T11:26:34.237 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000149226 2024-09-13T11:26:34.237 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:34.237 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:34.237 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:26:34.237 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:38.767014+0000", 2024-09-13T11:26:34.237 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.1183000000000002e-05 2024-09-13T11:26:34.237 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:34.238 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:34.238 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:26:34.238 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:38.771145+0000", 2024-09-13T11:26:34.238 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0041315270000000003 2024-09-13T11:26:34.238 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:34.238 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:34.238 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:26:34.238 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:38.771223+0000", 2024-09-13T11:26:34.238 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.7996000000000003e-05 2024-09-13T11:26:34.238 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:34.238 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:34.238 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:34.238 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:34.239 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:34.239 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:233802 3.11 3:8e27a54a:::benchmark_data_smithi177_31116_object93499:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e200)", 2024-09-13T11:26:34.239 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:26:16.812014+0000", 2024-09-13T11:26:34.239 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 17.34417573, 2024-09-13T11:26:34.239 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.34299280599999998, 2024-09-13T11:26:34.239 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:26:34.239 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:26:34.239 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:34.239 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:26:34.239 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:26:34.239 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:34.239 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 233802 2024-09-13T11:26:34.239 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:34.240 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:26:34.240 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:34.240 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:26:34.240 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:16.812014+0000", 2024-09-13T11:26:34.240 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:34.240 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:34.240 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:34.240 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:26:34.240 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:16.812014+0000", 2024-09-13T11:26:34.240 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:34.240 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:34.240 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:34.241 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:26:34.241 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:16.812015+0000", 2024-09-13T11:26:34.241 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.19e-06 2024-09-13T11:26:34.241 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:34.241 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:34.241 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:26:34.241 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:16.812019+0000", 2024-09-13T11:26:34.241 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.8720000000000004e-06 2024-09-13T11:26:34.241 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:34.241 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:34.241 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:26:34.241 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:16.812025+0000", 2024-09-13T11:26:34.241 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.3339999999999998e-06 2024-09-13T11:26:34.241 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:34.242 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:34.242 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:26:34.242 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:16.812029+0000", 2024-09-13T11:26:34.242 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.2989999999999998e-06 2024-09-13T11:26:34.242 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:34.242 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:34.242 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:26:34.242 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:16.812160+0000", 2024-09-13T11:26:34.242 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000130288 2024-09-13T11:26:34.242 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:34.242 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:34.242 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:26:34.242 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:16.812298+0000", 2024-09-13T11:26:34.243 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00013822500000000001 2024-09-13T11:26:34.243 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:34.243 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:34.243 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 11", 2024-09-13T11:26:34.243 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:16.812412+0000", 2024-09-13T11:26:34.243 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000113947 2024-09-13T11:26:34.243 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:34.243 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:34.243 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:26:34.243 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:16.815915+0000", 2024-09-13T11:26:34.243 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0035030259999999998 2024-09-13T11:26:34.243 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:34.243 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:34.243 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:34.244 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:17.154942+0000", 2024-09-13T11:26:34.244 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.33902705700000002 2024-09-13T11:26:34.244 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:34.244 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:34.244 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:26:34.244 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:17.154978+0000", 2024-09-13T11:26:34.244 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.5549999999999997e-05 2024-09-13T11:26:34.244 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:34.244 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:34.244 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:26:34.244 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:17.155007+0000", 2024-09-13T11:26:34.244 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.9017999999999999e-05 2024-09-13T11:26:34.244 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:34.244 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:34.245 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:34.245 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:34.245 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:34.245 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop_reply(client.4665.0:233802 3.11 e200/81 ondisk, result = 0)", 2024-09-13T11:26:34.245 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:26:16.815681+0000", 2024-09-13T11:26:34.245 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 17.340508231000001, 2024-09-13T11:26:34.245 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.33932907800000001, 2024-09-13T11:26:34.245 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:26:34.245 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:26:34.245 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:26:34.245 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:26:34.245 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:34.245 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:26:34.246 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:16.815681+0000", 2024-09-13T11:26:34.246 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:34.246 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:34.246 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:34.246 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:26:34.246 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:16.815681+0000", 2024-09-13T11:26:34.246 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:34.246 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:34.246 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:34.246 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:26:34.246 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:16.815682+0000", 2024-09-13T11:26:34.246 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.1230000000000001e-06 2024-09-13T11:26:34.246 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:34.247 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:34.247 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:26:34.247 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:16.815689+0000", 2024-09-13T11:26:34.247 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.5599999999999999e-06 2024-09-13T11:26:34.247 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:34.247 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:34.247 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:26:34.247 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:17.154882+0000", 2024-09-13T11:26:34.247 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.33919308599999998 2024-09-13T11:26:34.247 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:34.247 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:34.247 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:26:34.247 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:17.154890+0000", 2024-09-13T11:26:34.247 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.6149999999999999e-06 2024-09-13T11:26:34.248 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:34.248 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:34.248 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:26:34.248 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:17.154931+0000", 2024-09-13T11:26:34.248 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.1057999999999999e-05 2024-09-13T11:26:34.248 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:34.248 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:34.248 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:26:34.248 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:17.154937+0000", 2024-09-13T11:26:34.248 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.6719999999999998e-06 2024-09-13T11:26:34.248 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:34.248 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:34.249 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:26:34.249 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:17.155010+0000", 2024-09-13T11:26:34.249 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.2964000000000001e-05 2024-09-13T11:26:34.249 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:34.249 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:34.249 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:34.249 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:34.249 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:34.249 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:234291 3.11 3:8c4e5be8:::benchmark_data_smithi177_31116_object93988:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e201)", 2024-09-13T11:26:34.249 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:26:17.320509+0000", 2024-09-13T11:26:34.249 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 16.835680905, 2024-09-13T11:26:34.249 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.231498433, 2024-09-13T11:26:34.249 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:26:34.249 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:26:34.250 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:34.250 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:26:34.250 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:26:34.250 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:34.250 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 234291 2024-09-13T11:26:34.250 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:34.250 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:26:34.250 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:34.250 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:26:34.250 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:17.320509+0000", 2024-09-13T11:26:34.250 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:34.250 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:34.250 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:34.250 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:26:34.251 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:17.320509+0000", 2024-09-13T11:26:34.251 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:34.251 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:34.251 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:34.251 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:26:34.251 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:17.320510+0000", 2024-09-13T11:26:34.251 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.1999999999999999e-06 2024-09-13T11:26:34.251 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:34.251 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:34.251 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:26:34.251 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:17.320514+0000", 2024-09-13T11:26:34.251 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.8139999999999998e-06 2024-09-13T11:26:34.251 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:34.251 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:34.252 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:26:34.252 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:17.320523+0000", 2024-09-13T11:26:34.252 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.7589999999999992e-06 2024-09-13T11:26:34.252 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:34.252 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:34.252 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:26:34.252 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:17.320529+0000", 2024-09-13T11:26:34.252 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.2499999999999997e-06 2024-09-13T11:26:34.252 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:34.252 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:34.252 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:26:34.252 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:17.320580+0000", 2024-09-13T11:26:34.252 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.0980999999999999e-05 2024-09-13T11:26:34.253 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:34.253 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:34.253 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:26:34.253 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:17.320704+0000", 2024-09-13T11:26:34.253 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00012416599999999999 2024-09-13T11:26:34.253 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:34.253 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:34.253 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 11", 2024-09-13T11:26:34.253 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:17.320812+0000", 2024-09-13T11:26:34.253 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000108383 2024-09-13T11:26:34.253 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:34.253 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:34.253 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:26:34.254 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:17.323672+0000", 2024-09-13T11:26:34.254 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0028601329999999999 2024-09-13T11:26:34.254 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:34.254 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:34.254 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:34.254 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:17.551809+0000", 2024-09-13T11:26:34.254 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.22813635400000001 2024-09-13T11:26:34.254 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:34.254 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:34.254 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:26:34.254 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:17.551920+0000", 2024-09-13T11:26:34.254 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000111805 2024-09-13T11:26:34.254 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:34.254 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:34.255 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:26:34.255 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:17.552007+0000", 2024-09-13T11:26:34.255 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.6588000000000005e-05 2024-09-13T11:26:34.255 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:34.255 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:34.255 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:34.255 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:34.255 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:34.255 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop_reply(client.4665.0:234291 3.11 e201/81 ondisk, result = 0)", 2024-09-13T11:26:34.255 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:26:17.324111+0000", 2024-09-13T11:26:34.255 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 16.832078450000001, 2024-09-13T11:26:34.255 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.22790548199999999, 2024-09-13T11:26:34.256 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:26:34.256 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:26:34.256 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:26:34.256 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:26:34.256 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:34.256 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:26:34.256 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:17.324111+0000", 2024-09-13T11:26:34.256 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:34.256 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:34.256 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:34.256 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:26:34.256 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:17.324111+0000", 2024-09-13T11:26:34.256 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:34.256 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:34.257 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:34.257 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:26:34.257 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:17.324112+0000", 2024-09-13T11:26:34.257 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.313e-06 2024-09-13T11:26:34.257 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:34.257 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:34.257 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:26:34.257 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:17.324117+0000", 2024-09-13T11:26:34.257 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.5870000000000003e-06 2024-09-13T11:26:34.257 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:34.257 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:34.257 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:26:34.257 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:17.551656+0000", 2024-09-13T11:26:34.258 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.22753944200000001 2024-09-13T11:26:34.258 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:34.258 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:34.258 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:26:34.258 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:17.551669+0000", 2024-09-13T11:26:34.258 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2233e-05 2024-09-13T11:26:34.258 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:34.258 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:34.258 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:26:34.258 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:17.551780+0000", 2024-09-13T11:26:34.258 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000111471 2024-09-13T11:26:34.258 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:34.258 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:34.259 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:26:34.259 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:17.551798+0000", 2024-09-13T11:26:34.259 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.7887e-05 2024-09-13T11:26:34.259 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:34.259 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:34.259 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:26:34.259 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:17.552017+0000", 2024-09-13T11:26:34.259 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00021854900000000001 2024-09-13T11:26:34.259 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:34.259 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:34.259 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:34.259 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:34.259 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:34.259 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:243814 3.1a 3:5aaae69c:::benchmark_data_smithi177_31116_object103511:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e207)", 2024-09-13T11:26:34.259 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:26:26.761070+0000", 2024-09-13T11:26:34.260 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 7.3951193990000004, 2024-09-13T11:26:34.260 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.222488768, 2024-09-13T11:26:34.260 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:26:34.260 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:26:34.260 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:34.260 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:26:34.260 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:26:34.260 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:34.260 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 243814 2024-09-13T11:26:34.260 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:34.260 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:26:34.260 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:34.260 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:26:34.261 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:26.761070+0000", 2024-09-13T11:26:34.261 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:34.261 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:34.261 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:34.261 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:26:34.261 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:26.761070+0000", 2024-09-13T11:26:34.261 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:34.261 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:34.261 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:34.261 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:26:34.261 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:26.761073+0000", 2024-09-13T11:26:34.261 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.9119999999999998e-06 2024-09-13T11:26:34.261 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:34.262 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:34.262 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:26:34.262 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:26.761082+0000", 2024-09-13T11:26:34.262 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.7960000000000001e-06 2024-09-13T11:26:34.262 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:34.262 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:34.262 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:26:34.262 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:26.761105+0000", 2024-09-13T11:26:34.262 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.2976000000000002e-05 2024-09-13T11:26:34.262 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:34.262 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:34.262 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:26:34.262 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:26.761117+0000", 2024-09-13T11:26:34.262 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2323999999999999e-05 2024-09-13T11:26:34.263 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:34.263 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:34.263 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:26:34.263 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:26.761207+0000", 2024-09-13T11:26:34.263 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.9534e-05 2024-09-13T11:26:34.263 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:34.263 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:34.263 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:26:34.263 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:26.761326+0000", 2024-09-13T11:26:34.263 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000119478 2024-09-13T11:26:34.263 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:34.263 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:34.263 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 9", 2024-09-13T11:26:34.264 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:26.761422+0000", 2024-09-13T11:26:34.264 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.577e-05 2024-09-13T11:26:34.264 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:34.264 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:34.264 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:26:34.264 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:26.765182+0000", 2024-09-13T11:26:34.264 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0037604280000000001 2024-09-13T11:26:34.264 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:34.264 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:34.265 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:34.265 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:26.983388+0000", 2024-09-13T11:26:34.265 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.21820598199999999 2024-09-13T11:26:34.265 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:34.265 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:34.265 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:26:34.265 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:26.983485+0000", 2024-09-13T11:26:34.265 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.7072999999999997e-05 2024-09-13T11:26:34.265 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:34.265 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:34.265 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:26:34.265 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:26.983559+0000", 2024-09-13T11:26:34.265 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.3495e-05 2024-09-13T11:26:34.265 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:34.266 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:34.266 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:34.266 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:34.266 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:34.266 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-13T11:26:34.268 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-09-13T11:26:34.268 INFO:teuthology.orchestra.run.smithi001.stdout:osd.4: {} 2024-09-13T11:26:34.268 INFO:teuthology.orchestra.run.smithi001.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T11:26:34.269 INFO:teuthology.orchestra.run.smithi001.stdout:osd.5: {} 2024-09-13T11:26:34.269 INFO:teuthology.orchestra.run.smithi001.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T11:26:34.269 INFO:teuthology.orchestra.run.smithi001.stdout:osd.6: {} 2024-09-13T11:26:34.269 INFO:teuthology.orchestra.run.smithi001.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T11:26:34.270 INFO:teuthology.orchestra.run.smithi001.stdout:osd.7: {} 2024-09-13T11:26:34.270 INFO:teuthology.orchestra.run.smithi001.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T11:26:34.271 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-09-13T11:26:34.263+0000 7f97befc9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26315) UID: 0 2024-09-13T11:26:34.287 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-13T11:26:34.291 INFO:teuthology.orchestra.run.smithi001.stdout:osd.8: {} 2024-09-13T11:26:34.292 INFO:teuthology.orchestra.run.smithi001.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-13T11:26:34.332 INFO:teuthology.orchestra.run.smithi001.stdout:osd.9: {} 2024-09-13T11:26:34.334 INFO:teuthology.orchestra.run.smithi001.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-13T11:26:34.368 INFO:teuthology.orchestra.run.smithi001.stdout:osd.10: {} 2024-09-13T11:26:34.368 INFO:teuthology.orchestra.run.smithi001.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-13T11:26:34.368 INFO:tasks.ceph.osd.12.smithi001.stderr:2024-09-13T11:26:34.366+0000 7f892c6e6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26314) UID: 0 2024-09-13T11:26:34.396 INFO:teuthology.orchestra.run.smithi001.stdout:osd.11: {} 2024-09-13T11:26:34.397 INFO:teuthology.orchestra.run.smithi001.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-13T11:26:34.417 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-13T11:26:34.417 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:26:34.436 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-09-13T11:26:34.439 INFO:teuthology.orchestra.run.smithi001.stdout:osd.12: {} 2024-09-13T11:26:34.440 INFO:teuthology.orchestra.run.smithi001.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-13T11:26:34.468 INFO:tasks.ceph.osd.13.smithi092.stderr:2024-09-13T11:26:34.468+0000 7f3e5afea640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26350) UID: 0 2024-09-13T11:26:34.473 INFO:teuthology.orchestra.run.smithi001.stdout:osd.13: {} 2024-09-13T11:26:34.473 INFO:teuthology.orchestra.run.smithi001.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-13T11:26:34.501 INFO:teuthology.orchestra.run.smithi001.stdout:osd.14: {} 2024-09-13T11:26:34.501 INFO:teuthology.orchestra.run.smithi001.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-13T11:26:34.530 INFO:teuthology.orchestra.run.smithi001.stdout:osd.15: {} 2024-09-13T11:26:34.530 INFO:teuthology.orchestra.run.smithi001.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-13T11:26:34.569 INFO:tasks.ceph.osd.13.smithi092.stderr:2024-09-13T11:26:34.568+0000 7f3e5afea640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26350) UID: 0 2024-09-13T11:26:34.596 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-13T11:26:34.596 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:26:34.611 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-09-13T11:26:34.707 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-13T11:26:34.706+0000 7f6a68775640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26307) UID: 0 2024-09-13T11:26:34.771 INFO:tasks.ceph.osd.8.smithi001.stderr:2024-09-13T11:26:34.766+0000 7fb9b48ae640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26312) UID: 0 2024-09-13T11:26:34.800 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-13T11:26:34.800 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:26:34.820 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_ops_in_flight 2024-09-13T11:26:34.872 INFO:tasks.ceph.osd.13.smithi092.stderr:2024-09-13T11:26:34.868+0000 7f3e5afea640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26350) UID: 0 2024-09-13T11:26:34.973 INFO:tasks.ceph.osd.14.smithi162.stderr:2024-09-13T11:26:34.971+0000 7f3df5903640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33154) UID: 0 2024-09-13T11:26:34.996 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-13T11:26:34.996 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:26:35.015 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_blocked_ops 2024-09-13T11:26:35.074 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-13T11:26:35.070+0000 7f6a68775640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26307) UID: 0 2024-09-13T11:26:35.152 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-13T11:26:35.153 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:26:35.168 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_historic_ops 2024-09-13T11:26:35.174 INFO:tasks.ceph.osd.3.smithi177.stderr:2024-09-13T11:26:35.173+0000 7f10adba4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26574) UID: 0 2024-09-13T11:26:35.240 INFO:teuthology.orchestra.run.smithi001.stderr:nodeep-scrub is unset 2024-09-13T11:26:35.241 INFO:teuthology.orchestra.run.smithi001.stderr:clear 2.2 pg_upmap mapping 2024-09-13T11:26:35.264 INFO:tasks.ceph.osd.12.smithi001.stderr:2024-09-13T11:26:35.250+0000 7f891d10b640 -1 osd.12 211 handle_osd_map injecting map crc failure 2024-09-13T11:26:35.276 INFO:tasks.ceph.osd.13.smithi092.stderr:2024-09-13T11:26:35.272+0000 7f3e5afea640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26350) UID: 0 2024-09-13T11:26:35.370 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-13T11:26:35.370 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:26:35.376 INFO:tasks.ceph.osd.7.smithi177.stderr:2024-09-13T11:26:35.373+0000 7f897b51d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26575) UID: 0 2024-09-13T11:26:35.390 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-13T11:26:35.477 INFO:tasks.ceph.osd.9.smithi092.stderr:2024-09-13T11:26:35.476+0000 7ff5ab3d0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26348) UID: 0 2024-09-13T11:26:35.532 INFO:teuthology.orchestra.run.smithi092.stdout:ERROR: (22) Invalid argument 2024-09-13T11:26:35.532 INFO:teuthology.orchestra.run.smithi092.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:26:35.550 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-13T11:26:35.578 INFO:tasks.ceph.osd.7.smithi177.stderr:2024-09-13T11:26:35.573+0000 7f897b51d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26575) UID: 0 2024-09-13T11:26:35.679 INFO:tasks.ceph.osd.15.smithi177.stderr:2024-09-13T11:26:35.677+0000 7f9d6ce77640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26578) UID: 0 2024-09-13T11:26:35.728 INFO:teuthology.orchestra.run.smithi092.stdout:ERROR: (22) Invalid argument 2024-09-13T11:26:35.728 INFO:teuthology.orchestra.run.smithi092.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:26:35.745 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-13T11:26:35.780 INFO:tasks.ceph.osd.11.smithi177.stderr:2024-09-13T11:26:35.777+0000 7f24792c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26580) UID: 0 2024-09-13T11:26:35.881 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-13T11:26:35.878+0000 7f6a68775640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26307) UID: 0 2024-09-13T11:26:35.882 INFO:teuthology.orchestra.run.smithi092.stdout:ERROR: (22) Invalid argument 2024-09-13T11:26:35.882 INFO:teuthology.orchestra.run.smithi092.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:26:35.899 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-09-13T11:26:35.982 INFO:tasks.ceph.osd.10.smithi162.stderr:2024-09-13T11:26:35.979+0000 7fb59c9f0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26317) UID: 0 2024-09-13T11:26:36.037 INFO:teuthology.orchestra.run.smithi092.stdout:ERROR: (22) Invalid argument 2024-09-13T11:26:36.037 INFO:teuthology.orchestra.run.smithi092.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:26:36.054 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-09-13T11:26:36.105 INFO:tasks.ceph.osd.5.smithi092.stderr:2024-09-13T11:26:36.103+0000 7f6f0b705640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26346) UID: 0 2024-09-13T11:26:36.184 INFO:tasks.ceph.osd.15.smithi177.stderr:2024-09-13T11:26:36.181+0000 7f9d6ce77640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26578) UID: 0 2024-09-13T11:26:36.186 INFO:teuthology.orchestra.run.smithi092.stdout:ERROR: (22) Invalid argument 2024-09-13T11:26:36.187 INFO:teuthology.orchestra.run.smithi092.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:26:36.206 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-09-13T11:26:36.261 INFO:tasks.ceph.osd.9.smithi092.stderr:2024-09-13T11:26:36.259+0000 7ff59ddf9640 -1 osd.9 212 handle_osd_map injecting map crc failure 2024-09-13T11:26:36.264 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-09-13T11:26:36.263+0000 7f97b59fa640 -1 osd.6 212 handle_osd_map injecting map crc failure 2024-09-13T11:26:36.285 INFO:tasks.ceph.osd.9.smithi092.stderr:2024-09-13T11:26:36.283+0000 7ff5ab3d0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26348) UID: 0 2024-09-13T11:26:36.386 INFO:teuthology.orchestra.run.smithi092.stdout:ERROR: (22) Invalid argument 2024-09-13T11:26:36.387 INFO:teuthology.orchestra.run.smithi092.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:26:36.387 INFO:tasks.ceph.osd.14.smithi162.stderr:2024-09-13T11:26:36.383+0000 7f3df5903640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33154) UID: 0 2024-09-13T11:26:36.405 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_ops_in_flight 2024-09-13T11:26:36.496 INFO:tasks.ceph.osd.1.smithi092.stderr:2024-09-13T11:26:36.495+0000 7fe7789c5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33718) UID: 0 2024-09-13T11:26:36.560 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T11:26:36.587 INFO:teuthology.orchestra.run.smithi092.stdout:ERROR: (22) Invalid argument 2024-09-13T11:26:36.587 INFO:teuthology.orchestra.run.smithi092.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:26:36.588 INFO:tasks.ceph.osd.15.smithi177.stderr:2024-09-13T11:26:36.585+0000 7f9d6ce77640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26578) UID: 0 2024-09-13T11:26:36.605 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_blocked_ops 2024-09-13T11:26:36.693 INFO:tasks.ceph.osd.5.smithi092.stderr:2024-09-13T11:26:36.691+0000 7f6f0b705640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26346) UID: 0 2024-09-13T11:26:36.733 INFO:teuthology.orchestra.run.smithi001.stdout:osd.0: {} 2024-09-13T11:26:36.733 INFO:teuthology.orchestra.run.smithi001.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T11:26:36.761 INFO:teuthology.orchestra.run.smithi001.stdout:osd.1: {} 2024-09-13T11:26:36.761 INFO:teuthology.orchestra.run.smithi001.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T11:26:36.786 INFO:teuthology.orchestra.run.smithi001.stdout:osd.2: {} 2024-09-13T11:26:36.787 INFO:teuthology.orchestra.run.smithi001.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T11:26:36.789 INFO:tasks.ceph.osd.13.smithi092.stderr:2024-09-13T11:26:36.787+0000 7f3e5afea640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26350) UID: 0 2024-09-13T11:26:36.795 INFO:teuthology.orchestra.run.smithi092.stdout:ERROR: (22) Invalid argument 2024-09-13T11:26:36.795 INFO:teuthology.orchestra.run.smithi092.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:26:36.812 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_historic_ops 2024-09-13T11:26:36.813 INFO:teuthology.orchestra.run.smithi001.stdout:osd.3: {} 2024-09-13T11:26:36.813 INFO:teuthology.orchestra.run.smithi001.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T11:26:36.845 INFO:teuthology.orchestra.run.smithi001.stdout:osd.4: {} 2024-09-13T11:26:36.845 INFO:teuthology.orchestra.run.smithi001.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T11:26:36.872 INFO:teuthology.orchestra.run.smithi001.stdout:osd.5: {} 2024-09-13T11:26:36.873 INFO:teuthology.orchestra.run.smithi001.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T11:26:36.890 INFO:tasks.ceph.osd.15.smithi177.stderr:2024-09-13T11:26:36.889+0000 7f9d6ce77640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26578) UID: 0 2024-09-13T11:26:36.905 INFO:teuthology.orchestra.run.smithi001.stdout:osd.6: {} 2024-09-13T11:26:36.906 INFO:teuthology.orchestra.run.smithi001.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T11:26:36.932 INFO:teuthology.orchestra.run.smithi001.stdout:osd.7: {} 2024-09-13T11:26:36.932 INFO:teuthology.orchestra.run.smithi001.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T11:26:36.959 INFO:teuthology.orchestra.run.smithi001.stdout:osd.8: {} 2024-09-13T11:26:36.959 INFO:teuthology.orchestra.run.smithi001.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-13T11:26:36.992 INFO:teuthology.orchestra.run.smithi092.stdout:ERROR: (22) Invalid argument 2024-09-13T11:26:36.993 INFO:teuthology.orchestra.run.smithi092.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:26:37.004 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-13T11:26:37.029 INFO:teuthology.orchestra.run.smithi001.stdout:osd.9: {} 2024-09-13T11:26:37.029 INFO:teuthology.orchestra.run.smithi001.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-13T11:26:37.041 INFO:tasks.ceph.osd.9.smithi092.stderr:2024-09-13T11:26:36.991+0000 7ff5ab3d0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26348) UID: 0 2024-09-13T11:26:37.054 INFO:teuthology.orchestra.run.smithi001.stdout:osd.10: {} 2024-09-13T11:26:37.054 INFO:teuthology.orchestra.run.smithi001.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-13T11:26:37.077 INFO:teuthology.orchestra.run.smithi001.stdout:osd.11: {} 2024-09-13T11:26:37.077 INFO:teuthology.orchestra.run.smithi001.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-13T11:26:37.094 INFO:tasks.ceph.osd.1.smithi092.stderr:2024-09-13T11:26:37.091+0000 7fe7789c5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33718) UID: 0 2024-09-13T11:26:37.104 INFO:teuthology.orchestra.run.smithi001.stdout:osd.12: {} 2024-09-13T11:26:37.104 INFO:teuthology.orchestra.run.smithi001.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-13T11:26:37.131 INFO:teuthology.orchestra.run.smithi001.stdout:osd.13: {} 2024-09-13T11:26:37.131 INFO:teuthology.orchestra.run.smithi001.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-13T11:26:37.150 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-13T11:26:37.150 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [], 2024-09-13T11:26:37.150 INFO:teuthology.orchestra.run.smithi162.stdout: "num_ops": 0 2024-09-13T11:26:37.150 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-13T11:26:37.161 INFO:teuthology.orchestra.run.smithi001.stdout:osd.14: {} 2024-09-13T11:26:37.169 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-13T11:26:37.191 INFO:teuthology.orchestra.run.smithi001.stdout:osd.15: {} 2024-09-13T11:26:37.191 INFO:teuthology.orchestra.run.smithi001.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-13T11:26:37.217 INFO:tasks.ceph.osd.2.smithi162.stderr:2024-09-13T11:26:37.215+0000 7fe92b35a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26313) UID: 0 2024-09-13T11:26:37.291 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-13T11:26:37.298 INFO:tasks.ceph.osd.7.smithi177.stderr:2024-09-13T11:26:37.297+0000 7f897b51d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26575) UID: 0 2024-09-13T11:26:37.315 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-13T11:26:37.315 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [], 2024-09-13T11:26:37.315 INFO:teuthology.orchestra.run.smithi162.stdout: "complaint_time": 30, 2024-09-13T11:26:37.315 INFO:teuthology.orchestra.run.smithi162.stdout: "num_blocked_ops": 0 2024-09-13T11:26:37.315 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-13T11:26:37.333 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-13T11:26:37.425 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-09-13T11:26:37.423+0000 7f97befc9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26315) UID: 0 2024-09-13T11:26:37.502 INFO:tasks.ceph.osd.7.smithi177.stderr:2024-09-13T11:26:37.497+0000 7f897b51d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26575) UID: 0 2024-09-13T11:26:37.516 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-13T11:26:37.516 INFO:teuthology.orchestra.run.smithi162.stdout: "size": 20, 2024-09-13T11:26:37.516 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 600, 2024-09-13T11:26:37.516 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [ 2024-09-13T11:26:37.517 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:37.517 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.4665.0:171 3.9 e41/30 3:9e1c7eeb:::benchmark_data_smithi177_31116_object170:head v 41'13, mlcod=41'9)", 2024-09-13T11:26:37.517 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:22:47.418602+0000", 2024-09-13T11:26:37.517 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 230.096293417, 2024-09-13T11:26:37.517 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.67235367300000004, 2024-09-13T11:26:37.517 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:26:37.517 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:26:37.517 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:37.517 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:26:37.517 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:37.518 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:26:37.518 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:47.418602+0000", 2024-09-13T11:26:37.518 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:37.518 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:37.518 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:37.518 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:26:37.518 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:47.418602+0000", 2024-09-13T11:26:37.518 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:37.518 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:37.518 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:37.518 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:26:37.518 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:47.418605+0000", 2024-09-13T11:26:37.518 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.2200000000000001e-06 2024-09-13T11:26:37.519 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:37.519 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:37.519 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:26:37.519 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:47.418668+0000", 2024-09-13T11:26:37.519 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.2921000000000003e-05 2024-09-13T11:26:37.519 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:37.519 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:37.519 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:26:37.519 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:48.089380+0000", 2024-09-13T11:26:37.519 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.67071191100000005 2024-09-13T11:26:37.519 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:37.519 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:37.519 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:26:37.519 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:48.089423+0000", 2024-09-13T11:26:37.520 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.2916e-05 2024-09-13T11:26:37.520 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:37.520 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:37.520 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:26:37.520 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:48.089533+0000", 2024-09-13T11:26:37.520 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000109996 2024-09-13T11:26:37.520 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:37.520 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:37.520 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:26:37.520 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:48.089570+0000", 2024-09-13T11:26:37.520 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.7314999999999998e-05 2024-09-13T11:26:37.520 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:37.520 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:37.521 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:26:37.521 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:48.090852+0000", 2024-09-13T11:26:37.521 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001281641 2024-09-13T11:26:37.521 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:37.521 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:37.521 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:26:37.521 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:48.090956+0000", 2024-09-13T11:26:37.521 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00010375299999999999 2024-09-13T11:26:37.521 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:37.521 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:26:37.521 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:37.521 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:37.522 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:37.522 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPushReply(3.19 74/70 [PushReplyOp(3:9edf939c:::benchmark_data_smithi177_31116_object17207:head)])", 2024-09-13T11:26:37.522 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:30.897971+0000", 2024-09-13T11:26:37.522 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 186.61692456599999, 2024-09-13T11:26:37.522 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.40616071100000001, 2024-09-13T11:26:37.522 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:26:37.522 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:26:37.522 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-09-13T11:26:37.522 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:26:37.522 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:37.522 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:26:37.523 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.897971+0000", 2024-09-13T11:26:37.523 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:37.523 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:37.523 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:37.523 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:26:37.523 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.897971+0000", 2024-09-13T11:26:37.523 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:37.523 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:37.524 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:37.524 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:26:37.524 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.897972+0000", 2024-09-13T11:26:37.524 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1429999999999999e-06 2024-09-13T11:26:37.524 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:37.524 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:37.524 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:26:37.524 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.897977+0000", 2024-09-13T11:26:37.524 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.5199999999999999e-06 2024-09-13T11:26:37.524 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:37.524 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:37.524 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:26:37.524 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:31.302500+0000", 2024-09-13T11:26:37.525 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.40452312400000001 2024-09-13T11:26:37.525 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:37.525 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:37.525 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:26:37.525 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:31.302513+0000", 2024-09-13T11:26:37.525 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2753e-05 2024-09-13T11:26:37.525 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:37.525 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:37.525 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:26:37.525 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:31.302645+0000", 2024-09-13T11:26:37.525 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00013292100000000001 2024-09-13T11:26:37.525 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:37.525 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:37.526 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:26:37.526 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:31.304132+0000", 2024-09-13T11:26:37.526 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0014862499999999999 2024-09-13T11:26:37.526 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:37.526 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:26:37.526 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:37.526 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:37.526 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:37.526 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPushReply(3.19 75/70 [PushReplyOp(3:9d1489b1:::benchmark_data_smithi177_31116_object18698:head)])", 2024-09-13T11:26:37.526 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:31.697212+0000", 2024-09-13T11:26:37.526 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 185.81768356500001, 2024-09-13T11:26:37.526 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.22850633200000001, 2024-09-13T11:26:37.526 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:26:37.526 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:26:37.527 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-09-13T11:26:37.527 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:26:37.527 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:37.527 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:26:37.527 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:31.697212+0000", 2024-09-13T11:26:37.527 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:37.527 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:37.527 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:37.527 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:26:37.527 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:31.697212+0000", 2024-09-13T11:26:37.527 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:37.528 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:37.528 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:37.528 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:26:37.528 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:31.697214+0000", 2024-09-13T11:26:37.528 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.7439999999999999e-06 2024-09-13T11:26:37.528 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:37.528 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:37.528 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:26:37.529 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:31.697224+0000", 2024-09-13T11:26:37.529 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0173000000000001e-05 2024-09-13T11:26:37.529 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:37.529 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:37.529 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:26:37.529 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:31.923931+0000", 2024-09-13T11:26:37.529 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.226706926 2024-09-13T11:26:37.529 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:37.529 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:37.529 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:26:37.529 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:31.923950+0000", 2024-09-13T11:26:37.530 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.8899e-05 2024-09-13T11:26:37.530 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:37.530 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:37.530 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:26:37.530 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:31.924449+0000", 2024-09-13T11:26:37.530 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00049907800000000002 2024-09-13T11:26:37.530 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:37.530 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:37.530 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:26:37.530 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:31.925718+0000", 2024-09-13T11:26:37.531 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001269512 2024-09-13T11:26:37.531 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:37.531 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:26:37.531 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:37.531 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:37.531 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:37.531 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPushReply(3.19 76/70 [PushReplyOp(3:9faf6d5f:::benchmark_data_smithi177_31116_object18954:head)])", 2024-09-13T11:26:37.531 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:32.042874+0000", 2024-09-13T11:26:37.531 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 185.47202169600001, 2024-09-13T11:26:37.531 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.40212404299999999, 2024-09-13T11:26:37.531 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:26:37.532 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:26:37.532 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-09-13T11:26:37.532 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:26:37.532 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:37.532 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:26:37.532 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.042874+0000", 2024-09-13T11:26:37.532 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:37.532 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:37.532 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:37.532 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:26:37.533 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.042874+0000", 2024-09-13T11:26:37.533 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:37.533 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:37.533 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:37.533 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:26:37.533 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.042875+0000", 2024-09-13T11:26:37.533 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.432e-06 2024-09-13T11:26:37.533 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:37.533 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:37.533 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:26:37.534 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.042886+0000", 2024-09-13T11:26:37.534 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0265000000000001e-05 2024-09-13T11:26:37.534 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:37.534 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:37.534 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:26:37.534 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.443455+0000", 2024-09-13T11:26:37.534 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.40056951400000002 2024-09-13T11:26:37.534 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:37.534 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:37.534 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:26:37.534 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.443480+0000", 2024-09-13T11:26:37.535 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.4929000000000001e-05 2024-09-13T11:26:37.535 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:37.535 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:37.535 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:26:37.535 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.443666+0000", 2024-09-13T11:26:37.535 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000186446 2024-09-13T11:26:37.535 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:37.535 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:37.535 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:26:37.535 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.444998+0000", 2024-09-13T11:26:37.535 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0013314570000000001 2024-09-13T11:26:37.536 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:37.536 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:26:37.536 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:37.536 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:37.536 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:37.536 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPush(3.1d 85/78 [PushOp(3:bc4b9336:::benchmark_data_smithi177_31116_object11944:head, version: 49'734, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:bc4b9336:::benchmark_data_smithi177_31116_object11944:head@49'734, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:26:37.536 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:42.225966+0000", 2024-09-13T11:26:37.536 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 175.288929483, 2024-09-13T11:26:37.536 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.23879947800000001, 2024-09-13T11:26:37.536 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:26:37.536 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:26:37.537 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:26:37.537 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:26:37.537 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:37.537 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:26:37.537 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.225966+0000", 2024-09-13T11:26:37.537 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:37.537 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:37.537 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:37.537 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:26:37.537 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.225966+0000", 2024-09-13T11:26:37.537 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:37.538 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:37.538 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:37.538 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:26:37.538 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.225969+0000", 2024-09-13T11:26:37.538 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.0400000000000001e-06 2024-09-13T11:26:37.538 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:37.538 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:37.538 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:26:37.538 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.226036+0000", 2024-09-13T11:26:37.538 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.6978999999999997e-05 2024-09-13T11:26:37.538 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:37.539 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:37.539 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:26:37.539 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.463110+0000", 2024-09-13T11:26:37.539 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.23707400000000001 2024-09-13T11:26:37.539 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:37.539 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:37.539 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:26:37.539 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.463140+0000", 2024-09-13T11:26:37.539 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.9575999999999999e-05 2024-09-13T11:26:37.539 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:37.539 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:37.539 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:26:37.539 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.463326+0000", 2024-09-13T11:26:37.540 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000186298 2024-09-13T11:26:37.540 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:37.540 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:37.540 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:26:37.540 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.463351+0000", 2024-09-13T11:26:37.540 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.4862999999999999e-05 2024-09-13T11:26:37.540 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:37.540 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:37.540 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:26:37.540 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.464766+0000", 2024-09-13T11:26:37.540 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001414722 2024-09-13T11:26:37.541 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:37.541 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:26:37.541 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:37.541 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:37.541 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:37.541 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPush(3.1d 89/78 [PushOp(3:bc1b7cfb:::benchmark_data_smithi177_31116_object22629:head, version: 55'1387, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:bc1b7cfb:::benchmark_data_smithi177_31116_object22629:head@55'1387, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:26:37.541 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:46.660855+0000", 2024-09-13T11:26:37.541 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 170.85404098699999, 2024-09-13T11:26:37.541 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.26622785599999998, 2024-09-13T11:26:37.541 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:26:37.541 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:26:37.541 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:26:37.541 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:26:37.542 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:37.542 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:26:37.542 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:46.660855+0000", 2024-09-13T11:26:37.542 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:37.542 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:37.542 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:37.542 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:26:37.542 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:46.660855+0000", 2024-09-13T11:26:37.542 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:37.542 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:37.542 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:37.542 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:26:37.542 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:46.660857+0000", 2024-09-13T11:26:37.542 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.6560000000000001e-06 2024-09-13T11:26:37.542 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:37.543 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:37.543 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:26:37.543 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:46.660931+0000", 2024-09-13T11:26:37.543 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.4041999999999993e-05 2024-09-13T11:26:37.543 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:37.543 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:37.543 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:26:37.543 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:46.924654+0000", 2024-09-13T11:26:37.543 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.26372239800000002 2024-09-13T11:26:37.543 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:37.543 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:37.543 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:26:37.543 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:46.924729+0000", 2024-09-13T11:26:37.544 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.5061999999999994e-05 2024-09-13T11:26:37.544 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:37.544 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:37.544 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:26:37.544 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:46.924986+0000", 2024-09-13T11:26:37.544 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00025762000000000002 2024-09-13T11:26:37.544 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:37.544 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:37.544 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:26:37.544 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:46.925013+0000", 2024-09-13T11:26:37.544 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.6179000000000001e-05 2024-09-13T11:26:37.544 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:37.545 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:37.545 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:26:37.545 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:46.927082+0000", 2024-09-13T11:26:37.545 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.002069899 2024-09-13T11:26:37.545 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:37.545 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:26:37.545 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:37.545 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:37.545 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:37.545 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPush(3.1d 90/78 [PushOp(3:bc0e7228:::benchmark_data_smithi177_31116_object24733:head, version: 55'1512, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:bc0e7228:::benchmark_data_smithi177_31116_object24733:head@55'1512, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:26:37.545 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:47.529126+0000", 2024-09-13T11:26:37.545 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 169.98577001300001, 2024-09-13T11:26:37.546 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.21343989499999999, 2024-09-13T11:26:37.546 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:26:37.546 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:26:37.546 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:26:37.546 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:26:37.546 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:37.546 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:26:37.546 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:47.529126+0000", 2024-09-13T11:26:37.546 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:37.546 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:37.546 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:37.546 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:26:37.547 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:47.529126+0000", 2024-09-13T11:26:37.547 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:37.547 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:37.547 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:37.547 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:26:37.547 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:47.529127+0000", 2024-09-13T11:26:37.547 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.252e-06 2024-09-13T11:26:37.547 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:37.547 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:37.547 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:26:37.547 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:47.529185+0000", 2024-09-13T11:26:37.548 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.8224999999999999e-05 2024-09-13T11:26:37.548 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:37.548 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:37.548 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:26:37.548 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:47.740216+0000", 2024-09-13T11:26:37.548 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.21103060800000001 2024-09-13T11:26:37.548 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:37.548 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:37.548 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:26:37.548 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:47.740248+0000", 2024-09-13T11:26:37.548 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.2821999999999998e-05 2024-09-13T11:26:37.548 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:37.549 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:37.549 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:26:37.549 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:47.740504+0000", 2024-09-13T11:26:37.549 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00025518399999999999 2024-09-13T11:26:37.549 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:37.549 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:37.549 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:26:37.549 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:47.740544+0000", 2024-09-13T11:26:37.549 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.0438999999999997e-05 2024-09-13T11:26:37.549 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:37.549 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:37.550 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:26:37.550 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:47.742565+0000", 2024-09-13T11:26:37.550 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0020213649999999998 2024-09-13T11:26:37.550 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:37.550 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:26:37.550 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:37.550 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:37.550 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:37.550 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:83199 3.1d 3:be2c6fae:::benchmark_data_smithi177_31116_object83198:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106)", 2024-09-13T11:26:37.550 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:24:04.297306+0000", 2024-09-13T11:26:37.550 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 153.21759006100001, 2024-09-13T11:26:37.550 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.25114587999999999, 2024-09-13T11:26:37.551 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:26:37.551 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:26:37.551 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:37.551 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:26:37.551 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:26:37.551 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:37.551 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 83199 2024-09-13T11:26:37.551 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:37.551 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:26:37.551 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:37.551 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:26:37.552 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.297306+0000", 2024-09-13T11:26:37.552 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:37.552 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:37.552 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:37.552 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:26:37.552 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.297306+0000", 2024-09-13T11:26:37.552 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:37.552 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:37.552 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:37.552 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:26:37.552 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.297308+0000", 2024-09-13T11:26:37.552 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.4150000000000002e-06 2024-09-13T11:26:37.553 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:37.553 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:37.553 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:26:37.553 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.297352+0000", 2024-09-13T11:26:37.553 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.4431000000000002e-05 2024-09-13T11:26:37.553 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:37.553 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:37.553 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:26:37.553 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.297369+0000", 2024-09-13T11:26:37.553 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.6685e-05 2024-09-13T11:26:37.553 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:37.554 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:37.554 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:26:37.554 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.297376+0000", 2024-09-13T11:26:37.554 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.7299999999999999e-06 2024-09-13T11:26:37.554 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:37.554 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:37.554 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:26:37.554 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.297427+0000", 2024-09-13T11:26:37.554 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.1461000000000001e-05 2024-09-13T11:26:37.554 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:37.554 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:37.554 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:26:37.554 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.297528+0000", 2024-09-13T11:26:37.555 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00010028599999999999 2024-09-13T11:26:37.555 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:37.555 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:37.555 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 15", 2024-09-13T11:26:37.555 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.297685+0000", 2024-09-13T11:26:37.555 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00015760500000000001 2024-09-13T11:26:37.555 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:37.555 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:37.555 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:26:37.555 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.300779+0000", 2024-09-13T11:26:37.555 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0030936700000000002 2024-09-13T11:26:37.555 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:37.555 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:37.556 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:37.556 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.548279+0000", 2024-09-13T11:26:37.556 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.24750059999999999 2024-09-13T11:26:37.556 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:37.556 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:37.556 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:26:37.556 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.548373+0000", 2024-09-13T11:26:37.556 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.3451999999999999e-05 2024-09-13T11:26:37.556 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:37.556 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:37.557 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:26:37.557 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.548451+0000", 2024-09-13T11:26:37.557 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.8545000000000001e-05 2024-09-13T11:26:37.557 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:37.557 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:26:37.557 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:37.557 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:37.557 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:37.557 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:84621 3.1d 3:becd72aa:::benchmark_data_smithi177_31116_object84620:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-09-13T11:26:37.557 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:24:05.741008+0000", 2024-09-13T11:26:37.557 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 151.77388806100001, 2024-09-13T11:26:37.557 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.17994101100000001, 2024-09-13T11:26:37.557 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:26:37.557 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:26:37.558 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:37.558 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:26:37.558 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:26:37.558 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:37.558 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 84621 2024-09-13T11:26:37.558 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:37.558 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:26:37.558 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:37.558 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:26:37.558 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.741008+0000", 2024-09-13T11:26:37.558 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:37.558 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:37.558 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:37.559 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:26:37.559 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.741008+0000", 2024-09-13T11:26:37.559 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:37.559 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:37.559 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:37.559 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:26:37.559 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.741010+0000", 2024-09-13T11:26:37.559 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.898e-06 2024-09-13T11:26:37.559 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:37.559 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:37.559 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:26:37.559 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.741061+0000", 2024-09-13T11:26:37.560 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.0898e-05 2024-09-13T11:26:37.560 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:37.560 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:37.560 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:26:37.560 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.741085+0000", 2024-09-13T11:26:37.560 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.3558e-05 2024-09-13T11:26:37.560 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:37.560 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:37.560 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:26:37.560 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.741093+0000", 2024-09-13T11:26:37.560 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.3890000000000005e-06 2024-09-13T11:26:37.560 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:37.560 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:37.561 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:26:37.561 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.741178+0000", 2024-09-13T11:26:37.561 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.4792999999999999e-05 2024-09-13T11:26:37.561 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:37.561 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:37.561 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:26:37.561 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.741386+0000", 2024-09-13T11:26:37.561 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000207505 2024-09-13T11:26:37.561 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:37.561 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:37.562 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 15", 2024-09-13T11:26:37.562 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.741597+0000", 2024-09-13T11:26:37.562 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00021158200000000001 2024-09-13T11:26:37.562 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:37.562 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:37.562 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:26:37.562 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.743392+0000", 2024-09-13T11:26:37.562 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00179518 2024-09-13T11:26:37.562 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:37.562 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:37.562 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:37.562 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.920866+0000", 2024-09-13T11:26:37.563 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.17747373699999999 2024-09-13T11:26:37.563 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:37.563 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:37.563 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:26:37.563 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.920913+0000", 2024-09-13T11:26:37.563 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.6451e-05 2024-09-13T11:26:37.563 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:37.563 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:37.563 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:26:37.563 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.920949+0000", 2024-09-13T11:26:37.563 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.6019999999999997e-05 2024-09-13T11:26:37.564 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:37.564 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:26:37.564 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:37.564 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:37.564 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:37.564 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPush(3.25 124/120 [PushOp(3:a6d4ee3f:::benchmark_data_smithi177_31116_object14712:head, version: 51'872, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:a6d4ee3f:::benchmark_data_smithi177_31116_object14712:head@51'872, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:26:37.564 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:24:31.996253+0000", 2024-09-13T11:26:37.564 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 125.518642289, 2024-09-13T11:26:37.564 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.17694863499999999, 2024-09-13T11:26:37.564 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:26:37.565 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:26:37.565 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:26:37.565 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:26:37.565 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:37.565 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:26:37.565 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:31.996253+0000", 2024-09-13T11:26:37.565 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:37.565 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:37.565 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:37.565 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:26:37.565 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:31.996253+0000", 2024-09-13T11:26:37.565 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:37.565 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:37.565 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:37.566 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:26:37.566 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:31.996257+0000", 2024-09-13T11:26:37.566 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.2629999999999999e-06 2024-09-13T11:26:37.566 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:37.566 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:37.566 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:26:37.566 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:31.996359+0000", 2024-09-13T11:26:37.566 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000102456 2024-09-13T11:26:37.566 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:37.566 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:37.566 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:26:37.566 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:32.172313+0000", 2024-09-13T11:26:37.566 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.17595388000000001 2024-09-13T11:26:37.567 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:37.567 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:37.567 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:26:37.567 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:32.172326+0000", 2024-09-13T11:26:37.567 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2632999999999999e-05 2024-09-13T11:26:37.567 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:37.567 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:37.567 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:26:37.567 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:32.172423+0000", 2024-09-13T11:26:37.567 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.7084999999999996e-05 2024-09-13T11:26:37.567 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:37.567 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:37.567 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:26:37.568 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:32.172438+0000", 2024-09-13T11:26:37.568 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5775999999999999e-05 2024-09-13T11:26:37.568 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:37.568 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:37.568 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:26:37.568 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:32.173202+0000", 2024-09-13T11:26:37.568 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00076354199999999998 2024-09-13T11:26:37.568 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:37.568 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:26:37.568 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:37.568 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:37.568 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:37.569 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:126039 3.29 3:96100181:::benchmark_data_smithi177_31116_object126038:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e130)", 2024-09-13T11:26:37.569 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:24:43.012785+0000", 2024-09-13T11:26:37.569 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 114.502110939, 2024-09-13T11:26:37.569 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.26685311900000003, 2024-09-13T11:26:37.569 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:26:37.569 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:26:37.569 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:37.569 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:26:37.569 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:26:37.569 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:37.569 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 126039 2024-09-13T11:26:37.569 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:37.570 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:26:37.570 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:37.570 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:26:37.570 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:43.012785+0000", 2024-09-13T11:26:37.570 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:37.570 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:37.570 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:37.570 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:26:37.570 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:43.012785+0000", 2024-09-13T11:26:37.570 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:37.570 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:37.570 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:37.571 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:26:37.571 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:43.012787+0000", 2024-09-13T11:26:37.571 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.875e-06 2024-09-13T11:26:37.571 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:37.571 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:37.571 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:26:37.571 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:43.012826+0000", 2024-09-13T11:26:37.571 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.9483999999999997e-05 2024-09-13T11:26:37.571 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:37.571 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:37.571 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:26:37.571 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:43.012839+0000", 2024-09-13T11:26:37.572 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3405e-05 2024-09-13T11:26:37.572 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:37.572 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:37.572 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:26:37.572 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:43.012845+0000", 2024-09-13T11:26:37.572 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.9100000000000002e-06 2024-09-13T11:26:37.572 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:37.572 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:37.572 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:26:37.572 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:43.012903+0000", 2024-09-13T11:26:37.572 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.8131000000000003e-05 2024-09-13T11:26:37.572 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:37.573 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:37.573 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:26:37.573 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:43.013066+0000", 2024-09-13T11:26:37.573 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000162058 2024-09-13T11:26:37.573 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:37.573 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:37.573 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 4,5", 2024-09-13T11:26:37.573 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:43.013235+0000", 2024-09-13T11:26:37.573 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00016929600000000001 2024-09-13T11:26:37.573 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:37.573 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:37.573 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:37.574 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:43.017063+0000", 2024-09-13T11:26:37.574 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0038283890000000002 2024-09-13T11:26:37.574 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:37.574 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:37.574 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:26:37.574 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:43.017086+0000", 2024-09-13T11:26:37.574 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.2934000000000001e-05 2024-09-13T11:26:37.574 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:37.574 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:37.574 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:37.574 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:43.278804+0000", 2024-09-13T11:26:37.575 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.26171751799999998 2024-09-13T11:26:37.575 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:37.575 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:37.575 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:26:37.575 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:43.278862+0000", 2024-09-13T11:26:37.575 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.8431999999999999e-05 2024-09-13T11:26:37.575 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:37.575 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:37.575 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:26:37.575 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:43.279638+0000", 2024-09-13T11:26:37.575 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00077568699999999995 2024-09-13T11:26:37.575 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:37.575 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:26:37.575 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:37.576 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:37.576 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:37.576 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop_reply(client.4665.0:126039 3.29 e130/120 ondisk, result = 0)", 2024-09-13T11:26:37.576 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:24:43.021691+0000", 2024-09-13T11:26:37.576 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 114.49320501299999, 2024-09-13T11:26:37.576 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.25795579699999999, 2024-09-13T11:26:37.576 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:26:37.576 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:26:37.576 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:26:37.576 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:26:37.576 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:37.576 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:26:37.576 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:43.021691+0000", 2024-09-13T11:26:37.577 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:37.577 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:37.577 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:37.577 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:26:37.577 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:43.021691+0000", 2024-09-13T11:26:37.577 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:37.577 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:37.577 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:37.577 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:26:37.577 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:43.021692+0000", 2024-09-13T11:26:37.577 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3880000000000001e-06 2024-09-13T11:26:37.577 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:37.578 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:37.578 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:26:37.578 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:43.021700+0000", 2024-09-13T11:26:37.578 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.9310000000000008e-06 2024-09-13T11:26:37.578 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:37.578 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:37.578 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:26:37.578 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:43.278719+0000", 2024-09-13T11:26:37.578 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.25701909299999998 2024-09-13T11:26:37.578 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:37.578 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:37.578 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:26:37.578 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:43.278728+0000", 2024-09-13T11:26:37.579 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.5000000000000005e-06 2024-09-13T11:26:37.579 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:37.579 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:37.579 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:26:37.579 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:43.278786+0000", 2024-09-13T11:26:37.579 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.7640999999999999e-05 2024-09-13T11:26:37.579 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:37.579 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:37.579 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:26:37.579 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:43.278796+0000", 2024-09-13T11:26:37.579 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.9199999999999999e-06 2024-09-13T11:26:37.579 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:37.580 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:37.580 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:26:37.580 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:43.279646+0000", 2024-09-13T11:26:37.580 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00085032399999999996 2024-09-13T11:26:37.580 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:37.580 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:26:37.580 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:37.580 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:37.580 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:37.580 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPushReply(3.29 134/120 [PushReplyOp(3:95648f22:::benchmark_data_smithi177_31116_object7446:head)])", 2024-09-13T11:26:37.580 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:24:47.110443+0000", 2024-09-13T11:26:37.580 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 110.404452893, 2024-09-13T11:26:37.580 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.171602386, 2024-09-13T11:26:37.581 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:26:37.581 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:26:37.581 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-09-13T11:26:37.581 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:26:37.581 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:37.581 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:26:37.581 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:47.110443+0000", 2024-09-13T11:26:37.581 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:37.581 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:37.581 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:37.581 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:26:37.581 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:47.110443+0000", 2024-09-13T11:26:37.582 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:37.582 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:37.582 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:37.582 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:26:37.582 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:47.110444+0000", 2024-09-13T11:26:37.582 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.192e-06 2024-09-13T11:26:37.582 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:37.582 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:37.582 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:26:37.582 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:47.110449+0000", 2024-09-13T11:26:37.582 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.0869999999999997e-06 2024-09-13T11:26:37.582 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:37.582 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:37.583 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:26:37.583 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:47.280321+0000", 2024-09-13T11:26:37.583 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.169872157 2024-09-13T11:26:37.583 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:37.583 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:37.583 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:26:37.583 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:47.280337+0000", 2024-09-13T11:26:37.583 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5656000000000001e-05 2024-09-13T11:26:37.583 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:37.583 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:37.583 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:26:37.583 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:47.280414+0000", 2024-09-13T11:26:37.583 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.7083999999999996e-05 2024-09-13T11:26:37.584 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:37.584 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:37.584 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:26:37.584 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:47.282045+0000", 2024-09-13T11:26:37.584 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0016312099999999999 2024-09-13T11:26:37.584 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:37.584 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:26:37.584 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:37.584 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:37.584 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:37.584 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.4665.0:138294 3.9 e140/117 3:90012dab:::benchmark_data_smithi177_31116_object138293:head v 140'5246, mlcod=140'5243)", 2024-09-13T11:26:37.584 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:24:54.252227+0000", 2024-09-13T11:26:37.584 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 103.262668525, 2024-09-13T11:26:37.584 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.32919836400000002, 2024-09-13T11:26:37.585 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:26:37.585 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:26:37.585 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:37.585 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:26:37.585 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:37.585 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:26:37.585 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:54.252227+0000", 2024-09-13T11:26:37.585 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:37.585 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:37.585 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:37.585 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:26:37.585 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:54.252227+0000", 2024-09-13T11:26:37.585 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:37.585 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:37.586 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:37.586 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:26:37.586 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:54.252228+0000", 2024-09-13T11:26:37.586 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.8899999999999998e-07 2024-09-13T11:26:37.586 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:37.586 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:37.586 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:26:37.586 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:54.252276+0000", 2024-09-13T11:26:37.586 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.8137999999999999e-05 2024-09-13T11:26:37.586 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:37.586 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:37.586 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:26:37.586 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:54.578216+0000", 2024-09-13T11:26:37.587 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.32594014399999999 2024-09-13T11:26:37.587 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:37.587 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:37.587 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:26:37.587 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:54.578235+0000", 2024-09-13T11:26:37.587 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.8943000000000001e-05 2024-09-13T11:26:37.587 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:37.587 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:37.587 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:26:37.587 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:54.578396+0000", 2024-09-13T11:26:37.587 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000160922 2024-09-13T11:26:37.587 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:37.587 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:37.587 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:26:37.588 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:54.578457+0000", 2024-09-13T11:26:37.588 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.0665999999999998e-05 2024-09-13T11:26:37.588 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:37.588 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:37.588 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:26:37.588 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:54.581339+0000", 2024-09-13T11:26:37.588 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0028826839999999999 2024-09-13T11:26:37.588 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:37.588 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:37.588 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:26:37.588 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:54.581425+0000", 2024-09-13T11:26:37.588 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.6077999999999998e-05 2024-09-13T11:26:37.588 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:37.589 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:26:37.589 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:37.589 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:37.589 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:37.589 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPushReply(3.29 143/120 [PushReplyOp(3:973d2047:::benchmark_data_smithi177_31116_object49749:head)])", 2024-09-13T11:26:37.589 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:24:57.387106+0000", 2024-09-13T11:26:37.589 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 100.127789198, 2024-09-13T11:26:37.589 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.64813637000000002, 2024-09-13T11:26:37.589 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:26:37.589 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:26:37.589 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-09-13T11:26:37.589 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:26:37.590 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:37.590 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:26:37.590 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:57.387106+0000", 2024-09-13T11:26:37.590 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:37.590 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:37.590 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:37.590 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:26:37.590 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:57.387106+0000", 2024-09-13T11:26:37.590 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:37.590 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:37.590 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:37.590 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:26:37.590 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:57.387116+0000", 2024-09-13T11:26:37.591 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.1530000000000003e-06 2024-09-13T11:26:37.591 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:37.591 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:37.591 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:26:37.591 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:57.387127+0000", 2024-09-13T11:26:37.591 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1197e-05 2024-09-13T11:26:37.591 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:37.591 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:37.591 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:26:37.591 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:58.034264+0000", 2024-09-13T11:26:37.591 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.64713691799999995 2024-09-13T11:26:37.591 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:37.591 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:37.592 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:26:37.592 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:58.034273+0000", 2024-09-13T11:26:37.592 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.0929999999999992e-06 2024-09-13T11:26:37.592 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:37.592 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:37.592 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:26:37.592 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:58.034321+0000", 2024-09-13T11:26:37.592 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.8396999999999998e-05 2024-09-13T11:26:37.592 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:37.592 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:37.592 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:26:37.592 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:58.035243+0000", 2024-09-13T11:26:37.592 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00092161200000000004 2024-09-13T11:26:37.593 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:37.593 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:26:37.593 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:37.593 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:37.593 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:37.593 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPushReply(3.29 144/120 [PushReplyOp(3:94a6fc3a:::benchmark_data_smithi177_31116_object50245:head)])", 2024-09-13T11:26:37.593 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:24:58.320977+0000", 2024-09-13T11:26:37.593 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 99.193918615000001, 2024-09-13T11:26:37.593 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.31110569599999999, 2024-09-13T11:26:37.593 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:26:37.593 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:26:37.593 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-09-13T11:26:37.593 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:26:37.593 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:37.594 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:26:37.594 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:58.320977+0000", 2024-09-13T11:26:37.594 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:37.594 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:37.594 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:37.594 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:26:37.594 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:58.320977+0000", 2024-09-13T11:26:37.594 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:37.594 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:37.594 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:37.594 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:26:37.594 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:58.320978+0000", 2024-09-13T11:26:37.595 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.8700000000000004e-07 2024-09-13T11:26:37.595 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:37.595 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:37.595 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:26:37.595 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:58.320985+0000", 2024-09-13T11:26:37.595 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.742e-06 2024-09-13T11:26:37.595 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:37.595 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:37.595 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:26:37.595 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:58.631147+0000", 2024-09-13T11:26:37.595 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.31016212900000001 2024-09-13T11:26:37.595 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:37.596 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:37.596 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:26:37.596 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:58.631159+0000", 2024-09-13T11:26:37.596 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1946999999999999e-05 2024-09-13T11:26:37.596 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:37.596 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:37.596 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:26:37.596 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:58.631257+0000", 2024-09-13T11:26:37.596 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.8096000000000004e-05 2024-09-13T11:26:37.596 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:37.596 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:37.596 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:26:37.597 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:58.632083+0000", 2024-09-13T11:26:37.597 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00082589499999999999 2024-09-13T11:26:37.597 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:37.597 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:26:37.597 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:37.597 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:37.597 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:37.597 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.4665.0:170721 3.9 e162/117 3:93244b1b:::benchmark_data_smithi177_31116_object30418:head v 162'5743, mlcod=162'5740)", 2024-09-13T11:26:37.597 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:25:21.721723+0000", 2024-09-13T11:26:37.597 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 75.793172900000002, 2024-09-13T11:26:37.597 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.19722754300000001, 2024-09-13T11:26:37.597 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:26:37.598 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:26:37.598 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:37.598 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:26:37.598 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:37.598 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:26:37.598 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:21.721723+0000", 2024-09-13T11:26:37.598 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:37.598 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:37.598 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:37.599 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:26:37.599 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:21.721723+0000", 2024-09-13T11:26:37.599 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:37.599 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:37.599 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:37.599 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:26:37.599 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:21.721726+0000", 2024-09-13T11:26:37.599 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.1930000000000002e-06 2024-09-13T11:26:37.599 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:37.599 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:37.600 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:26:37.600 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:21.721741+0000", 2024-09-13T11:26:37.600 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5299999999999999e-05 2024-09-13T11:26:37.600 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:37.600 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:37.600 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:26:37.600 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:21.915450+0000", 2024-09-13T11:26:37.600 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.193708519 2024-09-13T11:26:37.600 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:37.600 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:37.600 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:26:37.601 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:21.915462+0000", 2024-09-13T11:26:37.601 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2414e-05 2024-09-13T11:26:37.601 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:37.601 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:37.601 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:26:37.601 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:21.915603+0000", 2024-09-13T11:26:37.601 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00014116500000000001 2024-09-13T11:26:37.601 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:37.601 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:37.601 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:26:37.601 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:21.915668+0000", 2024-09-13T11:26:37.601 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.4488999999999999e-05 2024-09-13T11:26:37.601 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:37.602 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:37.602 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:26:37.602 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:21.918850+0000", 2024-09-13T11:26:37.602 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0031825310000000002 2024-09-13T11:26:37.602 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:37.602 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:37.602 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:26:37.602 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:21.918950+0000", 2024-09-13T11:26:37.602 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.9932000000000005e-05 2024-09-13T11:26:37.602 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:37.602 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:26:37.602 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:37.602 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:37.602 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:37.604 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:191058 3.1d 3:bd6e7029:::benchmark_data_smithi177_31116_object50755:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e175)", 2024-09-13T11:26:37.604 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:25:38.512403+0000", 2024-09-13T11:26:37.604 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 59.002492355000001, 2024-09-13T11:26:37.604 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.38726333200000002, 2024-09-13T11:26:37.604 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:26:37.605 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:26:37.605 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:37.605 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:26:37.605 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:26:37.605 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:37.605 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 191058 2024-09-13T11:26:37.605 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:37.605 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:26:37.605 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:37.605 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:26:37.605 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:38.512403+0000", 2024-09-13T11:26:37.605 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:37.606 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:37.606 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:37.606 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:26:37.606 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:38.512403+0000", 2024-09-13T11:26:37.606 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:37.606 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:37.606 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:37.606 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:26:37.606 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:38.512406+0000", 2024-09-13T11:26:37.606 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.4059999999999999e-06 2024-09-13T11:26:37.606 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:37.606 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:37.606 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:26:37.606 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:38.512412+0000", 2024-09-13T11:26:37.606 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.8170000000000001e-06 2024-09-13T11:26:37.607 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:37.607 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:37.607 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:26:37.607 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:38.512430+0000", 2024-09-13T11:26:37.607 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.7397999999999998e-05 2024-09-13T11:26:37.607 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:37.607 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:37.607 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:26:37.607 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:38.512438+0000", 2024-09-13T11:26:37.607 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.3459999999999994e-06 2024-09-13T11:26:37.607 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:37.607 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:37.607 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:26:37.608 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:38.512526+0000", 2024-09-13T11:26:37.608 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.8146999999999995e-05 2024-09-13T11:26:37.608 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:37.608 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:37.608 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:26:37.608 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:38.512722+0000", 2024-09-13T11:26:37.608 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00019587599999999999 2024-09-13T11:26:37.608 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:37.608 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:37.608 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 15", 2024-09-13T11:26:37.608 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:38.512882+0000", 2024-09-13T11:26:37.608 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00015957300000000001 2024-09-13T11:26:37.608 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:37.608 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:37.609 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:26:37.609 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:38.517854+0000", 2024-09-13T11:26:37.609 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0049725500000000001 2024-09-13T11:26:37.609 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:37.609 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:37.609 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:37.609 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:38.899562+0000", 2024-09-13T11:26:37.609 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.38170719800000003 2024-09-13T11:26:37.609 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:37.609 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:37.609 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:26:37.609 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:38.899622+0000", 2024-09-13T11:26:37.609 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.0396999999999998e-05 2024-09-13T11:26:37.610 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:37.610 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:37.610 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:26:37.610 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:38.899667+0000", 2024-09-13T11:26:37.610 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.4623999999999998e-05 2024-09-13T11:26:37.610 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:37.610 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:26:37.610 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:37.610 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:37.610 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:37.610 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.4665.0:191547 3.25 e176/141 3:a7ac3e5b:::benchmark_data_smithi177_31116_object51244:head v 176'6020, mlcod=175'6018)", 2024-09-13T11:26:37.610 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:25:39.496290+0000", 2024-09-13T11:26:37.610 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 58.018605794000003, 2024-09-13T11:26:37.611 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.179206057, 2024-09-13T11:26:37.611 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:26:37.611 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:26:37.611 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:37.611 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:26:37.611 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:37.611 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:26:37.611 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:39.496290+0000", 2024-09-13T11:26:37.611 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:37.611 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:37.611 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:37.611 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:26:37.611 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:39.496290+0000", 2024-09-13T11:26:37.611 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:37.612 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:37.612 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:37.612 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:26:37.612 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:39.496293+0000", 2024-09-13T11:26:37.612 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.8499999999999998e-06 2024-09-13T11:26:37.612 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:37.612 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:37.612 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:26:37.612 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:39.496305+0000", 2024-09-13T11:26:37.612 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2286999999999999e-05 2024-09-13T11:26:37.612 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:37.612 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:37.612 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:26:37.613 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:39.670614+0000", 2024-09-13T11:26:37.613 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.17430884699999999 2024-09-13T11:26:37.613 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:37.613 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:37.613 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:26:37.613 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:39.670625+0000", 2024-09-13T11:26:37.613 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1233000000000001e-05 2024-09-13T11:26:37.613 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:37.613 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:37.613 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:26:37.613 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:39.670722+0000", 2024-09-13T11:26:37.613 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.6885999999999994e-05 2024-09-13T11:26:37.613 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:37.614 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:37.614 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:26:37.614 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:39.670756+0000", 2024-09-13T11:26:37.614 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.4125999999999998e-05 2024-09-13T11:26:37.614 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:37.614 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:37.614 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:26:37.614 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:39.675411+0000", 2024-09-13T11:26:37.614 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0046546160000000003 2024-09-13T11:26:37.614 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:37.614 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:37.615 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:26:37.615 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:39.675496+0000", 2024-09-13T11:26:37.615 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.5211999999999997e-05 2024-09-13T11:26:37.615 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:37.615 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:26:37.615 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:37.615 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:37.615 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:37.615 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.4665.0:250598 3.9 e211/117 3:929b3711:::benchmark_data_smithi177_31116_object110295:head v 211'6996, mlcod=211'6992)", 2024-09-13T11:26:37.615 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:26:32.474084+0000", 2024-09-13T11:26:37.615 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 5.0408117649999999, 2024-09-13T11:26:37.615 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.24130304, 2024-09-13T11:26:37.615 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:26:37.615 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:26:37.616 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:37.616 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:26:37.616 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:37.616 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:26:37.616 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:32.474084+0000", 2024-09-13T11:26:37.616 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:37.616 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:37.616 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:37.616 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:26:37.616 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:32.474084+0000", 2024-09-13T11:26:37.616 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:37.616 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:37.616 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:37.616 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:26:37.617 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:32.474085+0000", 2024-09-13T11:26:37.617 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.2699999999999998e-07 2024-09-13T11:26:37.617 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:37.617 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:37.617 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:26:37.617 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:32.474090+0000", 2024-09-13T11:26:37.617 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.8339999999999998e-06 2024-09-13T11:26:37.617 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:37.617 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:37.617 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:26:37.617 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:32.713078+0000", 2024-09-13T11:26:37.617 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.23898883900000001 2024-09-13T11:26:37.617 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:37.617 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:37.618 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:26:37.618 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:32.713088+0000", 2024-09-13T11:26:37.618 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.5170000000000005e-06 2024-09-13T11:26:37.618 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:37.618 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:37.618 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:26:37.618 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:32.713150+0000", 2024-09-13T11:26:37.618 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.1715000000000001e-05 2024-09-13T11:26:37.618 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:37.618 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:37.618 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:26:37.618 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:32.713165+0000", 2024-09-13T11:26:37.618 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5804999999999999e-05 2024-09-13T11:26:37.618 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:37.619 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:37.619 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:26:37.619 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:32.715350+0000", 2024-09-13T11:26:37.619 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0021848979999999998 2024-09-13T11:26:37.619 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:37.619 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:37.619 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:26:37.619 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:32.715387+0000", 2024-09-13T11:26:37.619 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.6504999999999997e-05 2024-09-13T11:26:37.619 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:37.619 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:26:37.619 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:37.619 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:37.619 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:26:37.619 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-13T11:26:37.621 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-13T11:26:37.622 INFO:tasks.ceph.osd.5.smithi092.stderr:2024-09-13T11:26:37.619+0000 7f6f0b705640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26346) UID: 0 2024-09-13T11:26:37.721 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-13T11:26:37.718+0000 7f6a68775640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26307) UID: 0 2024-09-13T11:26:37.809 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-13T11:26:37.809 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [], 2024-09-13T11:26:37.809 INFO:teuthology.orchestra.run.smithi162.stdout: "num_ops": 0 2024-09-13T11:26:37.809 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-13T11:26:37.822 INFO:tasks.ceph.osd.14.smithi162.stderr:2024-09-13T11:26:37.819+0000 7f3df5903640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33154) UID: 0 2024-09-13T11:26:37.825 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-13T11:26:37.923 INFO:tasks.ceph.osd.13.smithi092.stderr:2024-09-13T11:26:37.919+0000 7f3e5afea640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26350) UID: 0 2024-09-13T11:26:38.009 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-13T11:26:38.010 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [], 2024-09-13T11:26:38.010 INFO:teuthology.orchestra.run.smithi162.stdout: "complaint_time": 30, 2024-09-13T11:26:38.010 INFO:teuthology.orchestra.run.smithi162.stdout: "num_blocked_ops": 0 2024-09-13T11:26:38.010 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-13T11:26:38.024 INFO:tasks.ceph.osd.10.smithi162.stderr:2024-09-13T11:26:38.023+0000 7fb59c9f0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26317) UID: 0 2024-09-13T11:26:38.027 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-13T11:26:38.126 INFO:tasks.ceph.osd.7.smithi177.stderr:2024-09-13T11:26:38.121+0000 7f897b51d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26575) UID: 0 2024-09-13T11:26:38.210 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-13T11:26:38.210 INFO:teuthology.orchestra.run.smithi162.stdout: "size": 20, 2024-09-13T11:26:38.211 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 600, 2024-09-13T11:26:38.211 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [ 2024-09-13T11:26:38.211 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.211 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4257.0:80 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 262144~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-13T11:26:38.211 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:22:12.516065+0000", 2024-09-13T11:26:38.211 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 265.69315386400001, 2024-09-13T11:26:38.211 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.46461997999999999, 2024-09-13T11:26:38.211 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:26:38.211 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:26:38.211 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:38.211 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:26:38.211 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4257", 2024-09-13T11:26:38.211 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/1488131527", 2024-09-13T11:26:38.212 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 80 2024-09-13T11:26:38.212 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.212 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:26:38.212 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.212 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:26:38.213 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.516065+0000", 2024-09-13T11:26:38.213 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:38.213 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.213 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.213 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:26:38.213 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.516065+0000", 2024-09-13T11:26:38.213 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:38.214 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.214 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.214 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:26:38.214 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.516066+0000", 2024-09-13T11:26:38.214 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0869999999999999e-06 2024-09-13T11:26:38.214 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.214 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.214 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:26:38.215 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.516112+0000", 2024-09-13T11:26:38.215 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.5503000000000001e-05 2024-09-13T11:26:38.215 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.215 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.215 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:26:38.215 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.516128+0000", 2024-09-13T11:26:38.215 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5891000000000001e-05 2024-09-13T11:26:38.215 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.215 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.216 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:26:38.216 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.516138+0000", 2024-09-13T11:26:38.216 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.8120000000000002e-06 2024-09-13T11:26:38.216 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.216 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.216 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:26:38.216 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.520398+0000", 2024-09-13T11:26:38.216 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0042605450000000001 2024-09-13T11:26:38.216 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.216 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.216 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:26:38.217 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.520526+0000", 2024-09-13T11:26:38.217 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000128355 2024-09-13T11:26:38.217 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.217 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.217 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 0", 2024-09-13T11:26:38.218 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.520730+0000", 2024-09-13T11:26:38.218 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00020337799999999999 2024-09-13T11:26:38.218 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.218 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.218 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:26:38.218 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.522417+0000", 2024-09-13T11:26:38.219 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001687184 2024-09-13T11:26:38.220 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.220 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.220 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:38.220 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.980408+0000", 2024-09-13T11:26:38.220 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.45799068399999998 2024-09-13T11:26:38.220 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.221 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.221 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:26:38.221 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.980565+0000", 2024-09-13T11:26:38.221 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00015760899999999999 2024-09-13T11:26:38.221 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.221 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.221 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:26:38.221 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.980685+0000", 2024-09-13T11:26:38.221 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000119932 2024-09-13T11:26:38.221 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:38.221 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:26:38.221 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:38.221 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.221 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.222 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop_reply(client.4257.0:80 1.0 e24/23 ondisk, result = 0)", 2024-09-13T11:26:38.222 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:22:12.522938+0000", 2024-09-13T11:26:38.222 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 265.68628083099998, 2024-09-13T11:26:38.222 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.45776420600000001, 2024-09-13T11:26:38.222 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:26:38.222 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:26:38.222 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:26:38.222 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:26:38.222 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.222 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:26:38.222 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.522938+0000", 2024-09-13T11:26:38.222 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:38.222 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.223 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.223 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:26:38.223 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.522938+0000", 2024-09-13T11:26:38.223 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:38.223 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.223 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.223 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:26:38.223 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.522941+0000", 2024-09-13T11:26:38.223 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.0290000000000001e-06 2024-09-13T11:26:38.223 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.223 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.223 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:26:38.223 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.522950+0000", 2024-09-13T11:26:38.223 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.7499999999999992e-06 2024-09-13T11:26:38.223 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.224 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.224 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:26:38.224 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.980215+0000", 2024-09-13T11:26:38.224 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.45726523800000002 2024-09-13T11:26:38.224 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.224 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.224 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:26:38.224 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.980229+0000", 2024-09-13T11:26:38.224 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3549e-05 2024-09-13T11:26:38.224 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.224 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.224 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:26:38.224 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.980365+0000", 2024-09-13T11:26:38.224 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00013608700000000001 2024-09-13T11:26:38.225 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.225 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.225 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:26:38.225 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.980388+0000", 2024-09-13T11:26:38.225 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.3112000000000002e-05 2024-09-13T11:26:38.225 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.225 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.225 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:26:38.225 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.980702+0000", 2024-09-13T11:26:38.225 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00031444099999999999 2024-09-13T11:26:38.225 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:38.225 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:26:38.225 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:38.226 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.226 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.226 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:25416 3.8 3:1a1fbdd1:::benchmark_data_smithi177_31116_object25415:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e56)", 2024-09-13T11:26:38.226 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:04.512188+0000", 2024-09-13T11:26:38.226 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 213.697030864, 2024-09-13T11:26:38.226 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.210208697, 2024-09-13T11:26:38.226 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:26:38.226 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:26:38.226 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:38.226 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:26:38.226 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:26:38.226 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:38.226 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 25416 2024-09-13T11:26:38.226 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.227 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:26:38.227 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.227 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:26:38.227 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.512188+0000", 2024-09-13T11:26:38.227 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:38.227 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.227 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.227 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:26:38.227 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.512188+0000", 2024-09-13T11:26:38.227 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:38.227 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.227 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.227 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:26:38.227 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.512192+0000", 2024-09-13T11:26:38.228 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.2660000000000003e-06 2024-09-13T11:26:38.228 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.228 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.228 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:26:38.228 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.512247+0000", 2024-09-13T11:26:38.228 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.4889000000000003e-05 2024-09-13T11:26:38.228 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.228 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.228 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:26:38.228 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.512279+0000", 2024-09-13T11:26:38.228 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.1146000000000003e-05 2024-09-13T11:26:38.228 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.228 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.228 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:26:38.229 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.512297+0000", 2024-09-13T11:26:38.229 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.825e-05 2024-09-13T11:26:38.229 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.229 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.229 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:26:38.229 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.512377+0000", 2024-09-13T11:26:38.229 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.9875000000000004e-05 2024-09-13T11:26:38.229 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.229 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.229 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:26:38.229 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.512570+0000", 2024-09-13T11:26:38.229 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00019312100000000001 2024-09-13T11:26:38.229 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.230 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.230 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 1", 2024-09-13T11:26:38.230 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.512798+0000", 2024-09-13T11:26:38.230 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00022847899999999999 2024-09-13T11:26:38.230 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.230 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.230 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:26:38.230 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.514975+0000", 2024-09-13T11:26:38.230 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0021769979999999999 2024-09-13T11:26:38.230 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.230 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.230 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:38.230 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.722207+0000", 2024-09-13T11:26:38.230 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.20723217099999999 2024-09-13T11:26:38.231 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.231 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.231 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:26:38.231 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.722314+0000", 2024-09-13T11:26:38.231 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00010648400000000001 2024-09-13T11:26:38.231 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.231 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.231 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:26:38.231 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.722397+0000", 2024-09-13T11:26:38.231 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.3017999999999997e-05 2024-09-13T11:26:38.231 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:38.231 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:26:38.231 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:38.231 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.232 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.232 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop_reply(client.4665.0:25416 3.8 e56/40 ondisk, result = 0)", 2024-09-13T11:26:38.232 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:04.515635+0000", 2024-09-13T11:26:38.232 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 213.69358421199999, 2024-09-13T11:26:38.232 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.20677600500000001, 2024-09-13T11:26:38.232 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:26:38.232 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:26:38.232 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:26:38.232 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:26:38.232 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.232 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:26:38.232 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.515635+0000", 2024-09-13T11:26:38.232 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:38.232 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.233 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.233 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:26:38.233 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.515635+0000", 2024-09-13T11:26:38.233 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:38.233 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.233 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.233 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:26:38.233 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.515639+0000", 2024-09-13T11:26:38.233 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.619e-06 2024-09-13T11:26:38.233 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.233 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.233 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:26:38.233 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.515650+0000", 2024-09-13T11:26:38.234 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.111e-05 2024-09-13T11:26:38.234 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.234 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.234 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:26:38.234 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.722004+0000", 2024-09-13T11:26:38.234 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.206354805 2024-09-13T11:26:38.234 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.234 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.234 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:26:38.234 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.722017+0000", 2024-09-13T11:26:38.234 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2840000000000001e-05 2024-09-13T11:26:38.234 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.234 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.234 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:26:38.235 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.722186+0000", 2024-09-13T11:26:38.235 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00016898299999999999 2024-09-13T11:26:38.235 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.235 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.235 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:26:38.235 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.722199+0000", 2024-09-13T11:26:38.235 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.289e-05 2024-09-13T11:26:38.235 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.236 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.236 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:26:38.236 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.722411+0000", 2024-09-13T11:26:38.236 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00021175799999999999 2024-09-13T11:26:38.236 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:38.236 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:26:38.236 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:38.236 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.237 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.237 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:30833 3.8 3:1c203f8d:::benchmark_data_smithi177_31116_object30832:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e59)", 2024-09-13T11:26:38.237 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:09.506656+0000", 2024-09-13T11:26:38.237 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 208.702563314, 2024-09-13T11:26:38.237 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.224862282, 2024-09-13T11:26:38.237 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:26:38.237 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:26:38.237 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:38.237 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:26:38.237 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:26:38.237 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:38.237 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 30833 2024-09-13T11:26:38.237 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.237 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:26:38.238 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.238 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:26:38.238 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.506656+0000", 2024-09-13T11:26:38.238 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:38.238 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.238 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.238 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:26:38.238 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.506656+0000", 2024-09-13T11:26:38.238 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:38.238 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.238 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.238 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:26:38.238 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.506660+0000", 2024-09-13T11:26:38.239 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.6e-06 2024-09-13T11:26:38.239 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.239 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.239 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:26:38.239 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.506717+0000", 2024-09-13T11:26:38.239 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.6131999999999997e-05 2024-09-13T11:26:38.239 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.239 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.239 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:26:38.239 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.506751+0000", 2024-09-13T11:26:38.239 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.4746000000000002e-05 2024-09-13T11:26:38.239 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.239 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.239 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:26:38.240 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.506774+0000", 2024-09-13T11:26:38.240 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.2636000000000001e-05 2024-09-13T11:26:38.240 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.240 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.240 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:26:38.240 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.506878+0000", 2024-09-13T11:26:38.240 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000104057 2024-09-13T11:26:38.240 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.240 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.240 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:26:38.240 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.507052+0000", 2024-09-13T11:26:38.240 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00017402800000000001 2024-09-13T11:26:38.240 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.240 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.241 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 1", 2024-09-13T11:26:38.241 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.507217+0000", 2024-09-13T11:26:38.241 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00016516 2024-09-13T11:26:38.241 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.241 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.241 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:26:38.241 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.508992+0000", 2024-09-13T11:26:38.241 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0017743990000000001 2024-09-13T11:26:38.241 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.241 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.241 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:38.241 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.731429+0000", 2024-09-13T11:26:38.241 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.22243718000000001 2024-09-13T11:26:38.241 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.242 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.242 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:26:38.242 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.731479+0000", 2024-09-13T11:26:38.242 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.0146999999999999e-05 2024-09-13T11:26:38.242 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.242 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.242 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:26:38.242 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.731518+0000", 2024-09-13T11:26:38.242 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.9196999999999998e-05 2024-09-13T11:26:38.242 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:38.242 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:26:38.242 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:38.242 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.242 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.243 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop_reply(client.4665.0:30833 3.8 e59/40 ondisk, result = 0)", 2024-09-13T11:26:38.243 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:09.508907+0000", 2024-09-13T11:26:38.243 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 208.70031222, 2024-09-13T11:26:38.243 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.222618862, 2024-09-13T11:26:38.243 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:26:38.243 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:26:38.244 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:26:38.244 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:26:38.244 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.244 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:26:38.244 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.508907+0000", 2024-09-13T11:26:38.244 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:38.244 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.244 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.244 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:26:38.244 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.508907+0000", 2024-09-13T11:26:38.244 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:38.244 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.244 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.245 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:26:38.245 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.508910+0000", 2024-09-13T11:26:38.245 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.993e-06 2024-09-13T11:26:38.245 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.245 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.245 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:26:38.245 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.508918+0000", 2024-09-13T11:26:38.245 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.7000000000000008e-06 2024-09-13T11:26:38.245 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.245 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.245 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:26:38.245 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.731341+0000", 2024-09-13T11:26:38.245 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.22242308699999999 2024-09-13T11:26:38.246 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.246 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.246 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:26:38.246 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.731350+0000", 2024-09-13T11:26:38.246 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.3179999999999996e-06 2024-09-13T11:26:38.246 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.246 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.246 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:26:38.246 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.731418+0000", 2024-09-13T11:26:38.246 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.7586e-05 2024-09-13T11:26:38.246 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.246 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.246 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:26:38.246 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.731424+0000", 2024-09-13T11:26:38.247 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.844e-06 2024-09-13T11:26:38.247 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.247 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.247 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:26:38.247 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.731526+0000", 2024-09-13T11:26:38.247 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000101334 2024-09-13T11:26:38.247 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:38.247 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:26:38.247 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:38.247 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.247 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.247 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPush(3.10 76/70 [PushOp(3:0e7d2700:::benchmark_data_smithi177_31116_object17406:head, version: 52'1075, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:0e7d2700:::benchmark_data_smithi177_31116_object17406:head@52'1075, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:26:38.247 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:32.206198+0000", 2024-09-13T11:26:38.248 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 186.00302151099999, 2024-09-13T11:26:38.248 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.25193395000000002, 2024-09-13T11:26:38.248 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:26:38.248 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:26:38.248 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:26:38.248 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:26:38.248 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.248 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:26:38.248 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.206198+0000", 2024-09-13T11:26:38.248 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:38.248 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.249 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.249 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:26:38.249 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.206198+0000", 2024-09-13T11:26:38.249 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:38.249 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.249 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.249 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:26:38.249 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.206199+0000", 2024-09-13T11:26:38.249 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.46e-06 2024-09-13T11:26:38.249 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.249 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.249 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:26:38.249 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.206250+0000", 2024-09-13T11:26:38.250 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.113e-05 2024-09-13T11:26:38.250 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.250 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.250 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:26:38.250 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.456042+0000", 2024-09-13T11:26:38.250 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.24979182999999999 2024-09-13T11:26:38.250 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.250 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.250 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:26:38.250 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.456084+0000", 2024-09-13T11:26:38.250 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.1814000000000003e-05 2024-09-13T11:26:38.250 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.250 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.251 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:26:38.251 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.456342+0000", 2024-09-13T11:26:38.251 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000258404 2024-09-13T11:26:38.251 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.251 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.251 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:26:38.251 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.456367+0000", 2024-09-13T11:26:38.251 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.5099000000000001e-05 2024-09-13T11:26:38.251 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.251 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.251 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:26:38.251 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.458132+0000", 2024-09-13T11:26:38.251 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0017642129999999999 2024-09-13T11:26:38.251 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:38.251 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:26:38.252 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:38.252 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.252 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.252 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:60417 3.10 3:092808b0:::benchmark_data_smithi177_31116_object60416:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e85)", 2024-09-13T11:26:38.252 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:42.173238+0000", 2024-09-13T11:26:38.252 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 176.03598060300001, 2024-09-13T11:26:38.252 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.38965994599999998, 2024-09-13T11:26:38.252 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:26:38.252 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:26:38.252 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:38.252 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:26:38.252 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:26:38.252 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:38.253 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 60417 2024-09-13T11:26:38.253 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.253 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:26:38.253 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.253 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:26:38.253 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.173238+0000", 2024-09-13T11:26:38.253 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:38.253 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.253 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.253 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:26:38.253 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.173238+0000", 2024-09-13T11:26:38.253 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:38.253 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.253 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.254 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:26:38.254 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.173241+0000", 2024-09-13T11:26:38.254 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.6819999999999999e-06 2024-09-13T11:26:38.254 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.254 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.254 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:26:38.254 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.173335+0000", 2024-09-13T11:26:38.254 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.3957999999999997e-05 2024-09-13T11:26:38.254 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.254 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.254 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:26:38.254 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.173358+0000", 2024-09-13T11:26:38.254 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.3023999999999998e-05 2024-09-13T11:26:38.254 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.255 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.255 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:26:38.255 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.173373+0000", 2024-09-13T11:26:38.255 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4867e-05 2024-09-13T11:26:38.255 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.255 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.255 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:26:38.255 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.173466+0000", 2024-09-13T11:26:38.255 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.2542999999999998e-05 2024-09-13T11:26:38.255 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.255 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.255 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:26:38.255 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.173754+0000", 2024-09-13T11:26:38.255 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000288657 2024-09-13T11:26:38.256 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.256 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.256 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 0", 2024-09-13T11:26:38.256 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.174066+0000", 2024-09-13T11:26:38.256 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00031170999999999998 2024-09-13T11:26:38.256 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.256 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.256 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:26:38.257 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.178245+0000", 2024-09-13T11:26:38.257 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00417864 2024-09-13T11:26:38.257 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.257 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.257 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:38.257 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.562617+0000", 2024-09-13T11:26:38.257 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.38437253199999999 2024-09-13T11:26:38.257 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.257 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.258 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:26:38.258 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.562782+0000", 2024-09-13T11:26:38.258 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00016499200000000001 2024-09-13T11:26:38.258 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.258 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.258 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:26:38.258 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.562898+0000", 2024-09-13T11:26:38.258 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000116341 2024-09-13T11:26:38.258 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:38.258 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:26:38.258 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:38.258 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.259 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.259 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:83193 3.8 3:10066d83:::benchmark_data_smithi177_31116_object83192:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106)", 2024-09-13T11:26:38.259 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:24:04.283923+0000", 2024-09-13T11:26:38.259 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 153.92529592, 2024-09-13T11:26:38.259 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.64775463499999997, 2024-09-13T11:26:38.259 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:26:38.260 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:26:38.260 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:38.260 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:26:38.260 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:26:38.260 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:38.261 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 83193 2024-09-13T11:26:38.261 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.261 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:26:38.261 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.261 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:26:38.261 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.283923+0000", 2024-09-13T11:26:38.262 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:38.262 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.262 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.262 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:26:38.262 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.283923+0000", 2024-09-13T11:26:38.263 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:38.263 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.263 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.263 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:26:38.263 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.283925+0000", 2024-09-13T11:26:38.264 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.9489999999999999e-06 2024-09-13T11:26:38.264 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.264 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.264 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:26:38.264 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.283960+0000", 2024-09-13T11:26:38.265 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.5063e-05 2024-09-13T11:26:38.265 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.265 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.265 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:26:38.265 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.283974+0000", 2024-09-13T11:26:38.265 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3546e-05 2024-09-13T11:26:38.266 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.266 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.266 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:26:38.266 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.283983+0000", 2024-09-13T11:26:38.266 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.2499999999999995e-06 2024-09-13T11:26:38.267 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.267 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.267 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:26:38.267 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.284061+0000", 2024-09-13T11:26:38.267 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.7946000000000002e-05 2024-09-13T11:26:38.267 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.268 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.268 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:26:38.268 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.284270+0000", 2024-09-13T11:26:38.268 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00020876799999999999 2024-09-13T11:26:38.269 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.269 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.269 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 1", 2024-09-13T11:26:38.269 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.284489+0000", 2024-09-13T11:26:38.269 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00021956800000000001 2024-09-13T11:26:38.269 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.270 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.270 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:26:38.270 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.286456+0000", 2024-09-13T11:26:38.271 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0019665809999999998 2024-09-13T11:26:38.271 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.271 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.271 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:38.271 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.931497+0000", 2024-09-13T11:26:38.271 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.64504147700000003 2024-09-13T11:26:38.272 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.272 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.272 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:26:38.272 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.931589+0000", 2024-09-13T11:26:38.272 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.2041999999999997e-05 2024-09-13T11:26:38.273 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.273 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.273 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:26:38.273 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.931678+0000", 2024-09-13T11:26:38.273 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.8444999999999998e-05 2024-09-13T11:26:38.274 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:38.274 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:26:38.274 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:38.274 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.274 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.275 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:115060 3.8 3:11200ba9:::benchmark_data_smithi177_31116_object115059:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e125)", 2024-09-13T11:26:38.275 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:24:33.602178+0000", 2024-09-13T11:26:38.275 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 124.607040926, 2024-09-13T11:26:38.275 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.23668186799999999, 2024-09-13T11:26:38.275 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:26:38.275 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:26:38.276 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:38.276 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:26:38.276 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:26:38.276 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:38.276 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 115060 2024-09-13T11:26:38.277 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.277 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:26:38.277 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.277 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:26:38.277 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:33.602178+0000", 2024-09-13T11:26:38.277 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:38.278 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.278 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.278 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:26:38.278 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:33.602178+0000", 2024-09-13T11:26:38.278 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:38.279 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.279 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.279 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:26:38.279 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:33.602181+0000", 2024-09-13T11:26:38.279 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.0369999999999999e-06 2024-09-13T11:26:38.279 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.280 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.280 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:26:38.280 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:33.602217+0000", 2024-09-13T11:26:38.280 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.5837999999999998e-05 2024-09-13T11:26:38.280 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.281 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.281 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:26:38.281 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:33.602235+0000", 2024-09-13T11:26:38.281 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.7571000000000002e-05 2024-09-13T11:26:38.281 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.282 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.282 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:26:38.282 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:33.602245+0000", 2024-09-13T11:26:38.282 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0360000000000001e-05 2024-09-13T11:26:38.282 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.282 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.283 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:26:38.283 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:33.602312+0000", 2024-09-13T11:26:38.283 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.6989000000000005e-05 2024-09-13T11:26:38.283 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.283 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.284 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:26:38.284 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:33.602490+0000", 2024-09-13T11:26:38.284 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000178464 2024-09-13T11:26:38.284 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.284 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.285 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 1", 2024-09-13T11:26:38.285 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:33.602670+0000", 2024-09-13T11:26:38.285 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00018002799999999999 2024-09-13T11:26:38.285 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.285 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.285 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:26:38.286 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:33.604382+0000", 2024-09-13T11:26:38.286 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001711314 2024-09-13T11:26:38.286 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.286 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.286 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:38.287 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:33.838738+0000", 2024-09-13T11:26:38.287 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.23435634699999999 2024-09-13T11:26:38.287 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.287 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.287 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:26:38.288 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:33.838815+0000", 2024-09-13T11:26:38.288 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.7381999999999999e-05 2024-09-13T11:26:38.288 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.288 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.288 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:26:38.289 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:33.838860+0000", 2024-09-13T11:26:38.289 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.4537999999999999e-05 2024-09-13T11:26:38.289 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:38.289 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:26:38.289 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:38.289 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.290 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.290 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop_reply(client.4665.0:115060 3.8 e125/117 ondisk, result = 0)", 2024-09-13T11:26:38.290 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:24:33.604404+0000", 2024-09-13T11:26:38.290 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 124.604814668, 2024-09-13T11:26:38.290 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.23446235400000001, 2024-09-13T11:26:38.291 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:26:38.291 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:26:38.291 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:26:38.291 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:26:38.291 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.292 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:26:38.292 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:33.604404+0000", 2024-09-13T11:26:38.292 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:38.292 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.292 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.293 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:26:38.293 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:33.604404+0000", 2024-09-13T11:26:38.293 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:38.293 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.293 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.294 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:26:38.294 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:33.604406+0000", 2024-09-13T11:26:38.294 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.418e-06 2024-09-13T11:26:38.294 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.294 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.294 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:26:38.295 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:33.604411+0000", 2024-09-13T11:26:38.295 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.3519999999999997e-06 2024-09-13T11:26:38.295 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.295 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.295 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:26:38.295 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:33.838578+0000", 2024-09-13T11:26:38.296 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.23416656199999999 2024-09-13T11:26:38.296 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.296 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.296 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:26:38.296 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:33.838586+0000", 2024-09-13T11:26:38.297 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.9920000000000007e-06 2024-09-13T11:26:38.297 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.297 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.297 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:26:38.297 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:33.838655+0000", 2024-09-13T11:26:38.298 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.9463999999999998e-05 2024-09-13T11:26:38.298 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.298 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.298 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:26:38.299 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:33.838731+0000", 2024-09-13T11:26:38.299 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.5487000000000004e-05 2024-09-13T11:26:38.299 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.299 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.299 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:26:38.300 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:33.838867+0000", 2024-09-13T11:26:38.300 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000136079 2024-09-13T11:26:38.300 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:38.300 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:26:38.300 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:38.301 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.301 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.301 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPush(3.6 129/122 [PushOp(3:6261f3b6:::benchmark_data_smithi177_31116_object64090:head, version: 90'3486, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:6261f3b6:::benchmark_data_smithi177_31116_object64090:head@90'3486, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:26:38.301 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:24:38.641487+0000", 2024-09-13T11:26:38.302 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 119.567732257, 2024-09-13T11:26:38.302 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.209889098, 2024-09-13T11:26:38.302 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:26:38.303 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:26:38.303 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:26:38.303 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:26:38.303 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.303 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:26:38.304 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:38.641487+0000", 2024-09-13T11:26:38.304 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:38.304 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.304 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.305 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:26:38.305 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:38.641487+0000", 2024-09-13T11:26:38.305 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:38.305 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.305 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.306 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:26:38.306 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:38.641489+0000", 2024-09-13T11:26:38.306 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.0980000000000001e-06 2024-09-13T11:26:38.306 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.307 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.307 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:26:38.308 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:38.641560+0000", 2024-09-13T11:26:38.308 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.1404999999999998e-05 2024-09-13T11:26:38.308 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.308 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.308 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:26:38.309 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:38.850498+0000", 2024-09-13T11:26:38.309 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.20893800500000001 2024-09-13T11:26:38.309 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.309 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.310 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:26:38.310 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:38.850510+0000", 2024-09-13T11:26:38.310 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1762e-05 2024-09-13T11:26:38.310 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.310 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.311 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:26:38.311 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:38.850610+0000", 2024-09-13T11:26:38.311 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00010011200000000001 2024-09-13T11:26:38.311 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.311 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.312 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:26:38.312 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:38.850628+0000", 2024-09-13T11:26:38.312 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.7291000000000001e-05 2024-09-13T11:26:38.312 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.312 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.313 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:26:38.313 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:38.851376+0000", 2024-09-13T11:26:38.313 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00074842500000000005 2024-09-13T11:26:38.314 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:38.314 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:26:38.314 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:38.314 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.314 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.314 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.4665.0:126919 3.6 e130/122 3:634ce08e:::benchmark_data_smithi177_31116_object126918:head v 130'5137, mlcod=130'5134)", 2024-09-13T11:26:38.314 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:24:43.780120+0000", 2024-09-13T11:26:38.314 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 114.429099366, 2024-09-13T11:26:38.314 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.298086295, 2024-09-13T11:26:38.314 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:26:38.314 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:26:38.314 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:38.314 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:26:38.315 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.315 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:26:38.315 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:43.780120+0000", 2024-09-13T11:26:38.315 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:38.315 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.315 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.315 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:26:38.315 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:43.780120+0000", 2024-09-13T11:26:38.315 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:38.315 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.315 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.315 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:26:38.315 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:43.780123+0000", 2024-09-13T11:26:38.316 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.79e-06 2024-09-13T11:26:38.316 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.316 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.316 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:26:38.316 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:43.780160+0000", 2024-09-13T11:26:38.316 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.7945999999999999e-05 2024-09-13T11:26:38.316 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.316 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.316 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:26:38.316 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:44.075853+0000", 2024-09-13T11:26:38.316 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.29569234999999999 2024-09-13T11:26:38.316 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.316 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.317 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:26:38.317 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:44.075861+0000", 2024-09-13T11:26:38.317 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.9510000000000006e-06 2024-09-13T11:26:38.317 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.317 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.317 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:26:38.317 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:44.075924+0000", 2024-09-13T11:26:38.317 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.2946000000000003e-05 2024-09-13T11:26:38.317 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.317 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.317 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:26:38.318 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:44.075942+0000", 2024-09-13T11:26:38.318 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.8108e-05 2024-09-13T11:26:38.318 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.318 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.318 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:26:38.318 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:44.078164+0000", 2024-09-13T11:26:38.318 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.002222375 2024-09-13T11:26:38.318 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.318 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.318 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:26:38.318 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:44.078206+0000", 2024-09-13T11:26:38.318 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.1829000000000002e-05 2024-09-13T11:26:38.318 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:38.318 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:26:38.319 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:38.319 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.319 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.319 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:137473 3.10 3:094c0efb:::benchmark_data_smithi177_31116_object137472:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e139)", 2024-09-13T11:26:38.319 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:24:53.523319+0000", 2024-09-13T11:26:38.319 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 104.685900327, 2024-09-13T11:26:38.319 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.18297369299999999, 2024-09-13T11:26:38.319 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:26:38.319 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:26:38.319 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:38.319 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:26:38.319 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:26:38.319 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:38.320 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 137473 2024-09-13T11:26:38.320 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.320 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:26:38.320 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.320 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:26:38.320 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:53.523319+0000", 2024-09-13T11:26:38.320 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:38.320 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.320 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.320 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:26:38.320 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:53.523319+0000", 2024-09-13T11:26:38.320 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:38.321 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.321 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.321 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:26:38.321 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:53.523322+0000", 2024-09-13T11:26:38.321 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.2119999999999999e-06 2024-09-13T11:26:38.321 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.321 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.321 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:26:38.321 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:53.523370+0000", 2024-09-13T11:26:38.321 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.7788999999999999e-05 2024-09-13T11:26:38.321 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.321 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.322 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:26:38.322 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:53.523395+0000", 2024-09-13T11:26:38.322 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.5202e-05 2024-09-13T11:26:38.322 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.322 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.322 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:26:38.322 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:53.523409+0000", 2024-09-13T11:26:38.322 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4263999999999999e-05 2024-09-13T11:26:38.322 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.322 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.322 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:26:38.322 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:53.523477+0000", 2024-09-13T11:26:38.323 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.7578999999999998e-05 2024-09-13T11:26:38.323 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.323 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.323 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:26:38.323 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:53.523638+0000", 2024-09-13T11:26:38.323 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00016078300000000001 2024-09-13T11:26:38.323 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.323 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.323 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 0", 2024-09-13T11:26:38.323 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:53.523800+0000", 2024-09-13T11:26:38.323 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000162564 2024-09-13T11:26:38.324 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.324 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.324 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:26:38.324 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:53.526960+0000", 2024-09-13T11:26:38.324 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0031593950000000002 2024-09-13T11:26:38.324 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.324 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.324 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:38.324 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:53.706180+0000", 2024-09-13T11:26:38.324 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.17922080600000001 2024-09-13T11:26:38.324 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.324 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.324 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:26:38.325 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:53.706237+0000", 2024-09-13T11:26:38.325 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.6845000000000002e-05 2024-09-13T11:26:38.325 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.325 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.325 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:26:38.325 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:53.706292+0000", 2024-09-13T11:26:38.325 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.5254000000000003e-05 2024-09-13T11:26:38.325 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:38.325 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:26:38.325 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:38.325 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.325 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.325 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop_reply(client.4665.0:137473 3.10 e139/84 ondisk, result = 0)", 2024-09-13T11:26:38.325 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:24:53.527076+0000", 2024-09-13T11:26:38.326 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 104.6821435, 2024-09-13T11:26:38.326 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.17922728099999999, 2024-09-13T11:26:38.326 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:26:38.326 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:26:38.326 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:26:38.326 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:26:38.326 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.326 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:26:38.326 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:53.527076+0000", 2024-09-13T11:26:38.326 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:38.326 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.326 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.326 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:26:38.327 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:53.527076+0000", 2024-09-13T11:26:38.327 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:38.327 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.327 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.327 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:26:38.327 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:53.527078+0000", 2024-09-13T11:26:38.327 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.9000000000000002e-06 2024-09-13T11:26:38.327 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.327 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.327 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:26:38.327 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:53.527088+0000", 2024-09-13T11:26:38.327 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.7240000000000004e-06 2024-09-13T11:26:38.327 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.328 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.328 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:26:38.328 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:53.706095+0000", 2024-09-13T11:26:38.328 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.17900633899999999 2024-09-13T11:26:38.328 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.328 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.328 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:26:38.328 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:53.706101+0000", 2024-09-13T11:26:38.328 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.7730000000000002e-06 2024-09-13T11:26:38.328 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.328 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.328 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:26:38.328 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:53.706168+0000", 2024-09-13T11:26:38.329 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.6225999999999999e-05 2024-09-13T11:26:38.329 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.329 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.329 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:26:38.329 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:53.706175+0000", 2024-09-13T11:26:38.329 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.5560000000000002e-06 2024-09-13T11:26:38.329 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.329 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.329 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:26:38.329 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:53.706303+0000", 2024-09-13T11:26:38.329 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000127763 2024-09-13T11:26:38.329 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:38.329 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:26:38.329 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:38.330 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.330 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.330 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:137780 3.8 3:1047f268:::benchmark_data_smithi177_31116_object137779:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e139)", 2024-09-13T11:26:38.330 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:24:53.795212+0000", 2024-09-13T11:26:38.330 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 104.414007582, 2024-09-13T11:26:38.330 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.21530981399999999, 2024-09-13T11:26:38.330 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:26:38.330 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:26:38.330 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:38.330 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:26:38.330 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:26:38.330 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:38.330 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 137780 2024-09-13T11:26:38.331 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.331 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:26:38.331 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.331 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:26:38.331 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:53.795212+0000", 2024-09-13T11:26:38.331 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:38.331 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.331 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.331 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:26:38.331 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:53.795212+0000", 2024-09-13T11:26:38.331 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:38.331 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.331 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.332 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:26:38.332 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:53.795214+0000", 2024-09-13T11:26:38.332 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.481e-06 2024-09-13T11:26:38.332 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.332 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.332 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:26:38.332 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:53.795247+0000", 2024-09-13T11:26:38.332 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.3105999999999998e-05 2024-09-13T11:26:38.332 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.332 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.332 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:26:38.332 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:53.795258+0000", 2024-09-13T11:26:38.332 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0872e-05 2024-09-13T11:26:38.333 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.333 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.333 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:26:38.333 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:53.795265+0000", 2024-09-13T11:26:38.333 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.5729999999999996e-06 2024-09-13T11:26:38.333 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.333 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.333 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:26:38.333 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:53.795309+0000", 2024-09-13T11:26:38.333 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.4617999999999999e-05 2024-09-13T11:26:38.333 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.333 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.333 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:26:38.333 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:53.795397+0000", 2024-09-13T11:26:38.334 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.7863999999999998e-05 2024-09-13T11:26:38.334 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.334 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.334 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 1", 2024-09-13T11:26:38.334 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:53.795493+0000", 2024-09-13T11:26:38.334 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.6062000000000003e-05 2024-09-13T11:26:38.334 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.334 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.334 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:26:38.334 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:53.797245+0000", 2024-09-13T11:26:38.334 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0017521559999999999 2024-09-13T11:26:38.334 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.334 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.335 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:38.335 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:54.010409+0000", 2024-09-13T11:26:38.335 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.21316385299999999 2024-09-13T11:26:38.335 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.335 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.335 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:26:38.335 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:54.010474+0000", 2024-09-13T11:26:38.335 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.4998000000000003e-05 2024-09-13T11:26:38.335 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.335 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.335 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:26:38.335 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:54.010521+0000", 2024-09-13T11:26:38.335 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.7231000000000002e-05 2024-09-13T11:26:38.335 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:38.336 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:26:38.336 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:38.336 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.336 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.336 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop_reply(client.4665.0:137780 3.8 e139/117 ondisk, result = 0)", 2024-09-13T11:26:38.336 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:24:53.797821+0000", 2024-09-13T11:26:38.336 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 104.41139768399999, 2024-09-13T11:26:38.336 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.21270765599999999, 2024-09-13T11:26:38.336 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:26:38.336 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:26:38.336 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:26:38.336 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:26:38.336 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.337 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:26:38.337 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:53.797821+0000", 2024-09-13T11:26:38.337 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:38.337 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.337 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.337 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:26:38.337 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:53.797821+0000", 2024-09-13T11:26:38.337 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:38.337 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.337 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.337 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:26:38.337 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:53.797822+0000", 2024-09-13T11:26:38.338 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.012e-06 2024-09-13T11:26:38.338 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.338 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.338 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:26:38.338 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:53.797829+0000", 2024-09-13T11:26:38.338 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.8009999999999998e-06 2024-09-13T11:26:38.338 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.338 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.338 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:26:38.338 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:54.010334+0000", 2024-09-13T11:26:38.338 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.212504477 2024-09-13T11:26:38.338 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.338 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.338 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:26:38.339 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:54.010342+0000", 2024-09-13T11:26:38.339 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.6549999999999993e-06 2024-09-13T11:26:38.339 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.339 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.339 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:26:38.339 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:54.010394+0000", 2024-09-13T11:26:38.339 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.1875000000000001e-05 2024-09-13T11:26:38.339 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.339 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.339 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:26:38.339 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:54.010402+0000", 2024-09-13T11:26:38.339 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.6499999999999996e-06 2024-09-13T11:26:38.339 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.340 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.340 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:26:38.340 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:54.010529+0000", 2024-09-13T11:26:38.340 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000127186 2024-09-13T11:26:38.340 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:38.340 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:26:38.340 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:38.340 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.340 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.340 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.4665.0:196314 3.6 e178/122 3:62e79b1c:::benchmark_data_smithi177_31116_object56011:head v 178'6256, mlcod=178'6253)", 2024-09-13T11:26:38.340 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:25:43.340693+0000", 2024-09-13T11:26:38.340 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 54.868526103000001, 2024-09-13T11:26:38.340 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.24109011699999999, 2024-09-13T11:26:38.340 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:26:38.341 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:26:38.341 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:38.341 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:26:38.341 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.341 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:26:38.341 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:43.340693+0000", 2024-09-13T11:26:38.341 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:38.341 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.341 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.341 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:26:38.341 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:43.340693+0000", 2024-09-13T11:26:38.341 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:38.342 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.342 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.342 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:26:38.342 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:43.340704+0000", 2024-09-13T11:26:38.342 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1049e-05 2024-09-13T11:26:38.342 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.342 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.342 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:26:38.342 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:43.340718+0000", 2024-09-13T11:26:38.342 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4214e-05 2024-09-13T11:26:38.342 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.342 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.343 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:26:38.343 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:43.579964+0000", 2024-09-13T11:26:38.343 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.23924590200000001 2024-09-13T11:26:38.343 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.343 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.343 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:26:38.343 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:43.579972+0000", 2024-09-13T11:26:38.343 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.2789999999999998e-06 2024-09-13T11:26:38.343 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.343 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.343 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:26:38.343 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:43.580043+0000", 2024-09-13T11:26:38.343 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.0856000000000001e-05 2024-09-13T11:26:38.344 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.344 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.344 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:26:38.344 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:43.580065+0000", 2024-09-13T11:26:38.344 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.1795999999999999e-05 2024-09-13T11:26:38.344 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.344 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.344 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:26:38.344 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:43.581741+0000", 2024-09-13T11:26:38.344 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0016759190000000001 2024-09-13T11:26:38.344 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.344 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.344 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:26:38.345 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:43.581783+0000", 2024-09-13T11:26:38.345 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.2101999999999998e-05 2024-09-13T11:26:38.345 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:38.345 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:26:38.345 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:38.345 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.345 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.345 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:227446 3.10 3:0a4bd1eb:::benchmark_data_smithi177_31116_object87143:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e196)", 2024-09-13T11:26:38.345 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:26:11.077279+0000", 2024-09-13T11:26:38.345 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 27.131940334999999, 2024-09-13T11:26:38.345 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.18995021400000001, 2024-09-13T11:26:38.345 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:26:38.345 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:26:38.346 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:38.346 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:26:38.346 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:26:38.346 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:38.346 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 227446 2024-09-13T11:26:38.346 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.346 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:26:38.346 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.346 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:26:38.346 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:11.077279+0000", 2024-09-13T11:26:38.346 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:38.347 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.347 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.347 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:26:38.347 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:11.077279+0000", 2024-09-13T11:26:38.347 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:38.347 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.347 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.347 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:26:38.347 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:11.077281+0000", 2024-09-13T11:26:38.347 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.3429999999999998e-06 2024-09-13T11:26:38.347 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.347 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.347 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:26:38.347 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:11.077287+0000", 2024-09-13T11:26:38.348 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.353e-06 2024-09-13T11:26:38.348 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.348 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.348 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:26:38.348 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:11.077303+0000", 2024-09-13T11:26:38.348 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5523e-05 2024-09-13T11:26:38.348 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.348 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.348 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:26:38.348 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:11.077312+0000", 2024-09-13T11:26:38.348 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.6049999999999998e-06 2024-09-13T11:26:38.348 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.348 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.349 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:26:38.349 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:11.077359+0000", 2024-09-13T11:26:38.349 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.7388000000000001e-05 2024-09-13T11:26:38.349 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.349 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.349 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:26:38.349 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:11.077448+0000", 2024-09-13T11:26:38.349 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.9431999999999993e-05 2024-09-13T11:26:38.349 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.349 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.349 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 0", 2024-09-13T11:26:38.349 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:11.077513+0000", 2024-09-13T11:26:38.350 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.4543999999999997e-05 2024-09-13T11:26:38.350 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.350 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.350 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:26:38.350 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:11.080314+0000", 2024-09-13T11:26:38.350 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.002801441 2024-09-13T11:26:38.350 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.350 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.350 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:38.350 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:11.267144+0000", 2024-09-13T11:26:38.350 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.18682944700000001 2024-09-13T11:26:38.350 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.350 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.351 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:26:38.351 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:11.267191+0000", 2024-09-13T11:26:38.351 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.7227e-05 2024-09-13T11:26:38.351 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.351 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.351 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:26:38.351 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:11.267229+0000", 2024-09-13T11:26:38.351 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.7910999999999997e-05 2024-09-13T11:26:38.351 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:38.351 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:26:38.351 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:38.351 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.351 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.351 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop_reply(client.4665.0:227446 3.10 e196/84 ondisk, result = 0)", 2024-09-13T11:26:38.352 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:26:11.079968+0000", 2024-09-13T11:26:38.352 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 27.129251540999999, 2024-09-13T11:26:38.352 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.18726515199999999, 2024-09-13T11:26:38.352 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:26:38.352 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:26:38.352 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:26:38.352 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:26:38.352 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.352 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:26:38.352 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:11.079968+0000", 2024-09-13T11:26:38.352 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:38.352 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.352 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.353 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:26:38.353 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:11.079968+0000", 2024-09-13T11:26:38.353 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:38.353 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.353 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.353 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:26:38.353 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:11.079969+0000", 2024-09-13T11:26:38.353 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.42e-06 2024-09-13T11:26:38.353 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.353 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.353 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:26:38.353 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:11.079974+0000", 2024-09-13T11:26:38.354 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.3739999999999997e-06 2024-09-13T11:26:38.354 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.354 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.354 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:26:38.354 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:11.267071+0000", 2024-09-13T11:26:38.354 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.18709714699999999 2024-09-13T11:26:38.354 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.354 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.354 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:26:38.354 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:11.267080+0000", 2024-09-13T11:26:38.354 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.8589999999999994e-06 2024-09-13T11:26:38.354 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.354 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.354 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:26:38.355 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:11.267134+0000", 2024-09-13T11:26:38.355 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.4131000000000001e-05 2024-09-13T11:26:38.355 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.355 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.355 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:26:38.355 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:11.267140+0000", 2024-09-13T11:26:38.355 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.9499999999999998e-06 2024-09-13T11:26:38.355 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.355 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.355 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:26:38.355 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:11.267233+0000", 2024-09-13T11:26:38.355 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.2270999999999998e-05 2024-09-13T11:26:38.355 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:38.356 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:26:38.356 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:38.356 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:38.356 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:26:38.356 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-13T11:26:38.357 INFO:tasks.ceph.osd.13.smithi092.stderr:2024-09-13T11:26:38.243+0000 7f3e4da13640 -1 osd.13 214 handle_osd_map injecting map crc failure 2024-09-13T11:26:38.358 INFO:tasks.ceph.osd.15.smithi177.stderr:2024-09-13T11:26:38.353+0000 7f9d6ce77640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26578) UID: 0 2024-09-13T11:26:38.358 INFO:teuthology.orchestra.run.smithi001.stderr:noscrub is set 2024-09-13T11:26:38.359 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-09-13T11:26:38.458 INFO:tasks.ceph.osd.7.smithi177.stderr:2024-09-13T11:26:38.457+0000 7f897b51d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26575) UID: 0 2024-09-13T11:26:38.499 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-13T11:26:38.499 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [ 2024-09-13T11:26:38.499 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.499 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:257559 3.2a 3:56d3ab0c:::benchmark_data_smithi177_31116_object117256:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e215)", 2024-09-13T11:26:38.499 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:26:38.460017+0000", 2024-09-13T11:26:38.499 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 0.039713574000000001, 2024-09-13T11:26:38.499 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.039733031000000002, 2024-09-13T11:26:38.499 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:26:38.499 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:26:38.499 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "waiting for sub ops", 2024-09-13T11:26:38.500 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:26:38.500 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:26:38.500 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:38.500 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 257559 2024-09-13T11:26:38.500 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.500 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:26:38.500 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.500 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:26:38.500 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:38.460017+0000", 2024-09-13T11:26:38.500 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:38.500 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.500 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.501 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:26:38.501 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:38.460017+0000", 2024-09-13T11:26:38.501 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:38.501 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.501 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.501 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:26:38.501 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:38.460018+0000", 2024-09-13T11:26:38.501 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.7300000000000004e-07 2024-09-13T11:26:38.501 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.501 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.501 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:26:38.501 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:38.460022+0000", 2024-09-13T11:26:38.501 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.3280000000000002e-06 2024-09-13T11:26:38.501 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.502 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.502 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:26:38.502 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:38.460029+0000", 2024-09-13T11:26:38.502 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.2989999999999999e-06 2024-09-13T11:26:38.502 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.502 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.502 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:26:38.502 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:38.460033+0000", 2024-09-13T11:26:38.502 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.9099999999999998e-06 2024-09-13T11:26:38.502 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.502 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.502 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:26:38.502 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:38.460072+0000", 2024-09-13T11:26:38.502 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.9113000000000003e-05 2024-09-13T11:26:38.503 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.503 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.503 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:26:38.503 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:38.460169+0000", 2024-09-13T11:26:38.503 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.6939000000000001e-05 2024-09-13T11:26:38.503 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.503 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.503 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 7", 2024-09-13T11:26:38.503 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:38.460231+0000", 2024-09-13T11:26:38.503 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.2203000000000001e-05 2024-09-13T11:26:38.503 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.503 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.503 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:26:38.504 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:38.464405+0000", 2024-09-13T11:26:38.504 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0041738679999999998 2024-09-13T11:26:38.504 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:38.504 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:26:38.504 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:38.504 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.504 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.504 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:257596 3.18 3:1b738c13:::benchmark_data_smithi177_31116_object117293:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e215)", 2024-09-13T11:26:38.504 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:26:38.492865+0000", 2024-09-13T11:26:38.504 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 0.0068657559999999998, 2024-09-13T11:26:38.504 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0069312419999999998, 2024-09-13T11:26:38.504 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:26:38.504 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:26:38.505 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "waiting for sub ops", 2024-09-13T11:26:38.505 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:26:38.505 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:26:38.505 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:38.505 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 257596 2024-09-13T11:26:38.505 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.505 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:26:38.505 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.505 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:26:38.505 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:38.492865+0000", 2024-09-13T11:26:38.505 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:38.505 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.505 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.505 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:26:38.506 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:38.492865+0000", 2024-09-13T11:26:38.506 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:38.506 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.506 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.506 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:26:38.506 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:38.492867+0000", 2024-09-13T11:26:38.506 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4640000000000001e-06 2024-09-13T11:26:38.506 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.506 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.506 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:26:38.506 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:38.492871+0000", 2024-09-13T11:26:38.506 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.2540000000000003e-06 2024-09-13T11:26:38.506 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.506 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.507 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:26:38.507 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:38.492881+0000", 2024-09-13T11:26:38.507 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.7750000000000004e-06 2024-09-13T11:26:38.507 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.507 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.507 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:26:38.507 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:38.492889+0000", 2024-09-13T11:26:38.507 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.2369999999999992e-06 2024-09-13T11:26:38.507 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.507 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.507 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:26:38.507 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:38.492947+0000", 2024-09-13T11:26:38.507 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.8459999999999999e-05 2024-09-13T11:26:38.507 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.508 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.508 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:26:38.508 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:38.493082+0000", 2024-09-13T11:26:38.508 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000134628 2024-09-13T11:26:38.508 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.508 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.508 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 0", 2024-09-13T11:26:38.508 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:38.493195+0000", 2024-09-13T11:26:38.508 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000112487 2024-09-13T11:26:38.508 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.508 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.508 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:26:38.508 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:38.495982+0000", 2024-09-13T11:26:38.508 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0027877499999999999 2024-09-13T11:26:38.509 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:38.509 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:26:38.509 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:38.509 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.509 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.509 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:257601 3.3 3:c1fc4bef:::benchmark_data_smithi177_31116_object117298:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e215)", 2024-09-13T11:26:38.509 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:26:38.499139+0000", 2024-09-13T11:26:38.509 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 0.00059207200000000002, 2024-09-13T11:26:38.509 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00069914399999999996, 2024-09-13T11:26:38.509 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:26:38.509 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:26:38.509 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "waiting for sub ops", 2024-09-13T11:26:38.509 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:26:38.509 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:26:38.509 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:38.510 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 257601 2024-09-13T11:26:38.510 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.510 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:26:38.510 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.510 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:26:38.510 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:38.499139+0000", 2024-09-13T11:26:38.510 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:38.510 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.510 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.510 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:26:38.510 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:38.499139+0000", 2024-09-13T11:26:38.510 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:38.510 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.510 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.511 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:26:38.511 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:38.499140+0000", 2024-09-13T11:26:38.511 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.7699999999999992e-07 2024-09-13T11:26:38.511 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.511 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.511 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:26:38.511 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:38.499143+0000", 2024-09-13T11:26:38.511 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.1650000000000002e-06 2024-09-13T11:26:38.511 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.511 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.511 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:26:38.511 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:38.499151+0000", 2024-09-13T11:26:38.511 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.3410000000000003e-06 2024-09-13T11:26:38.511 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.512 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.512 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:26:38.512 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:38.499159+0000", 2024-09-13T11:26:38.512 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.187e-06 2024-09-13T11:26:38.512 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.512 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.512 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:26:38.512 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:38.499201+0000", 2024-09-13T11:26:38.512 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.2589000000000002e-05 2024-09-13T11:26:38.512 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.512 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.512 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:26:38.512 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:38.499284+0000", 2024-09-13T11:26:38.512 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.2546000000000006e-05 2024-09-13T11:26:38.513 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.513 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.513 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 7", 2024-09-13T11:26:38.513 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:38.499347+0000", 2024-09-13T11:26:38.513 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.3693000000000002e-05 2024-09-13T11:26:38.513 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:38.513 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:26:38.513 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:38.513 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:38.513 INFO:teuthology.orchestra.run.smithi162.stdout: ], 2024-09-13T11:26:38.513 INFO:teuthology.orchestra.run.smithi162.stdout: "num_ops": 3 2024-09-13T11:26:38.513 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-13T11:26:38.516 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-09-13T11:26:38.559 INFO:tasks.ceph.osd.13.smithi092.stderr:2024-09-13T11:26:38.555+0000 7f3e5afea640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26350) UID: 0 2024-09-13T11:26:38.653 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-13T11:26:38.653 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [], 2024-09-13T11:26:38.653 INFO:teuthology.orchestra.run.smithi162.stdout: "complaint_time": 30, 2024-09-13T11:26:38.653 INFO:teuthology.orchestra.run.smithi162.stdout: "num_blocked_ops": 0 2024-09-13T11:26:38.653 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-13T11:26:38.660 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-09-13T11:26:38.659+0000 7f97befc9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26315) UID: 0 2024-09-13T11:26:38.670 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-09-13T11:26:38.762 INFO:tasks.ceph.osd.5.smithi092.stderr:2024-09-13T11:26:38.759+0000 7f6f0b705640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26346) UID: 0 2024-09-13T11:26:38.856 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-13T11:26:38.856 INFO:teuthology.orchestra.run.smithi162.stdout: "size": 20, 2024-09-13T11:26:38.856 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 600, 2024-09-13T11:26:38.856 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [ 2024-09-13T11:26:38.856 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.856 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:45810 3.17 3.9bd13e57 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e69)", 2024-09-13T11:26:38.856 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:21.347960+0000", 2024-09-13T11:26:38.856 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 197.50659826099999, 2024-09-13T11:26:38.856 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.56814464200000003, 2024-09-13T11:26:38.856 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:26:38.856 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:26:38.857 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-09-13T11:26:38.857 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:26:38.857 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:26:38.857 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:38.857 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 45810 2024-09-13T11:26:38.857 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.857 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:26:38.857 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.857 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:26:38.857 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:21.347960+0000", 2024-09-13T11:26:38.857 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:38.857 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.857 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.857 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:26:38.858 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:21.347960+0000", 2024-09-13T11:26:38.858 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:38.858 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.858 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.858 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:26:38.858 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:21.347965+0000", 2024-09-13T11:26:38.858 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.6720000000000003e-06 2024-09-13T11:26:38.858 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.858 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.858 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:26:38.858 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:21.348019+0000", 2024-09-13T11:26:38.858 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.4391000000000002e-05 2024-09-13T11:26:38.858 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.859 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.859 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:26:38.859 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:21.348052+0000", 2024-09-13T11:26:38.859 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.2246000000000003e-05 2024-09-13T11:26:38.859 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.859 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.859 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:26:38.859 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:21.348072+0000", 2024-09-13T11:26:38.859 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.0622e-05 2024-09-13T11:26:38.859 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.859 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.859 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:26:38.859 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:21.348278+0000", 2024-09-13T11:26:38.859 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000205391 2024-09-13T11:26:38.860 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.860 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.860 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-09-13T11:26:38.860 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:21.348292+0000", 2024-09-13T11:26:38.860 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4759e-05 2024-09-13T11:26:38.860 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.860 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.860 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:26:38.860 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:21.916064+0000", 2024-09-13T11:26:38.860 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.56777128600000004 2024-09-13T11:26:38.860 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.860 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.860 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:26:38.861 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:21.916105+0000", 2024-09-13T11:26:38.861 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.1275e-05 2024-09-13T11:26:38.861 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:38.861 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:26:38.861 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:38.861 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.861 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.861 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:54653 3.23 3:c7a54800:::benchmark_data_smithi177_31116_object54652:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e75)", 2024-09-13T11:26:38.861 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:30.979059+0000", 2024-09-13T11:26:38.861 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 187.875499762, 2024-09-13T11:26:38.861 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.97284747100000002, 2024-09-13T11:26:38.861 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:26:38.861 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:26:38.861 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-09-13T11:26:38.862 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:26:38.862 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:26:38.862 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:38.862 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 54653 2024-09-13T11:26:38.862 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.862 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:26:38.862 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.862 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:26:38.862 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.979059+0000", 2024-09-13T11:26:38.862 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:38.862 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.862 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.862 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:26:38.863 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.979059+0000", 2024-09-13T11:26:38.863 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:38.863 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.863 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.863 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:26:38.863 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.979061+0000", 2024-09-13T11:26:38.863 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.4279999999999999e-06 2024-09-13T11:26:38.863 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.863 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.863 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:26:38.863 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.979114+0000", 2024-09-13T11:26:38.863 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.3254000000000002e-05 2024-09-13T11:26:38.863 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.863 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.864 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:26:38.864 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.979157+0000", 2024-09-13T11:26:38.864 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.2018000000000003e-05 2024-09-13T11:26:38.864 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.864 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.864 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:26:38.864 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.979172+0000", 2024-09-13T11:26:38.864 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5636000000000001e-05 2024-09-13T11:26:38.864 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.864 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.864 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:26:38.864 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:31.045109+0000", 2024-09-13T11:26:38.864 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.065936982000000005 2024-09-13T11:26:38.864 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.865 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.865 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-09-13T11:26:38.865 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:31.045121+0000", 2024-09-13T11:26:38.865 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1608e-05 2024-09-13T11:26:38.865 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.865 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.865 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:26:38.865 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:31.951636+0000", 2024-09-13T11:26:38.865 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.90651503899999997 2024-09-13T11:26:38.865 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.865 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.865 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:26:38.865 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:31.951906+0000", 2024-09-13T11:26:38.865 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000270506 2024-09-13T11:26:38.866 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:38.866 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:26:38.866 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:38.866 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.866 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.866 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPush(3.18 76/70 [PushOp(3:1a32c9cd:::benchmark_data_smithi177_31116_object12833:head, version: 49'818, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:1a32c9cd:::benchmark_data_smithi177_31116_object12833:head@49'818, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:26:38.866 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:32.522305+0000", 2024-09-13T11:26:38.866 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 186.332253678, 2024-09-13T11:26:38.866 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.40201207100000003, 2024-09-13T11:26:38.866 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:26:38.866 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:26:38.866 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:26:38.866 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:26:38.866 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.867 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:26:38.867 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.522305+0000", 2024-09-13T11:26:38.867 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:38.867 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.867 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.867 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:26:38.867 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.522305+0000", 2024-09-13T11:26:38.867 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:38.867 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.867 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.867 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:26:38.867 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.522309+0000", 2024-09-13T11:26:38.867 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.9829999999999998e-06 2024-09-13T11:26:38.867 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.868 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.868 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:26:38.868 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.522377+0000", 2024-09-13T11:26:38.868 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.8069000000000002e-05 2024-09-13T11:26:38.868 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.868 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.868 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:26:38.868 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.921950+0000", 2024-09-13T11:26:38.868 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.399573438 2024-09-13T11:26:38.868 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.868 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.868 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:26:38.868 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.921983+0000", 2024-09-13T11:26:38.869 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.2858000000000003e-05 2024-09-13T11:26:38.869 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.869 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.869 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:26:38.869 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.922389+0000", 2024-09-13T11:26:38.869 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00040612999999999999 2024-09-13T11:26:38.869 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.869 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.869 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:26:38.869 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.922421+0000", 2024-09-13T11:26:38.869 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.1356999999999998e-05 2024-09-13T11:26:38.869 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.869 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.869 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:26:38.870 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.924317+0000", 2024-09-13T11:26:38.870 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0018962359999999999 2024-09-13T11:26:38.870 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:38.870 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:26:38.870 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:38.870 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.870 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.870 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPush(3.17 76/70 [PushOp(3:ebcbce67:::benchmark_data_smithi177_31116_object18268:head, version: 53'1102, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:ebcbce67:::benchmark_data_smithi177_31116_object18268:head@53'1102, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:26:38.870 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:32.638908+0000", 2024-09-13T11:26:38.870 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 186.21565022999999, 2024-09-13T11:26:38.870 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.229064031, 2024-09-13T11:26:38.870 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:26:38.870 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:26:38.871 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:26:38.871 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:26:38.871 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.871 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:26:38.871 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.638908+0000", 2024-09-13T11:26:38.871 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:38.871 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.871 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.871 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:26:38.871 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.638908+0000", 2024-09-13T11:26:38.871 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:38.871 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.871 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.871 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:26:38.872 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.638910+0000", 2024-09-13T11:26:38.872 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.7379999999999999e-06 2024-09-13T11:26:38.872 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.872 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.872 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:26:38.872 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.638975+0000", 2024-09-13T11:26:38.872 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.4937000000000005e-05 2024-09-13T11:26:38.872 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.872 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.872 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:26:38.872 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.865860+0000", 2024-09-13T11:26:38.872 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.22688509800000001 2024-09-13T11:26:38.872 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.872 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.873 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:26:38.873 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.865890+0000", 2024-09-13T11:26:38.873 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.0273e-05 2024-09-13T11:26:38.873 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.873 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.873 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:26:38.873 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.866143+0000", 2024-09-13T11:26:38.873 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00025212800000000001 2024-09-13T11:26:38.873 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.873 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.873 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:26:38.873 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.866181+0000", 2024-09-13T11:26:38.873 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.8945000000000001e-05 2024-09-13T11:26:38.873 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.874 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.874 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:26:38.874 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.867972+0000", 2024-09-13T11:26:38.874 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001790912 2024-09-13T11:26:38.874 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:38.874 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:26:38.874 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:38.874 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.874 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.874 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:57435 3.3 3:c0db4777:::benchmark_data_smithi177_31116_object57434:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e80)", 2024-09-13T11:26:38.874 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:37.436034+0000", 2024-09-13T11:26:38.874 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 181.41852477, 2024-09-13T11:26:38.874 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.35949142000000001, 2024-09-13T11:26:38.874 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:26:38.875 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:26:38.875 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:38.875 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:26:38.875 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:26:38.875 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:38.875 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 57435 2024-09-13T11:26:38.875 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.875 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:26:38.875 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.875 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:26:38.875 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:37.436034+0000", 2024-09-13T11:26:38.875 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:38.875 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.875 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.876 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:26:38.876 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:37.436034+0000", 2024-09-13T11:26:38.876 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:38.876 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.876 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.876 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:26:38.876 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:37.436037+0000", 2024-09-13T11:26:38.876 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.7829999999999999e-06 2024-09-13T11:26:38.876 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.876 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.876 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:26:38.876 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:37.436090+0000", 2024-09-13T11:26:38.876 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.3665999999999997e-05 2024-09-13T11:26:38.876 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.877 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.877 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:26:38.877 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:37.436112+0000", 2024-09-13T11:26:38.877 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.1556000000000002e-05 2024-09-13T11:26:38.877 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.877 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.877 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:26:38.877 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:37.436122+0000", 2024-09-13T11:26:38.877 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0098000000000001e-05 2024-09-13T11:26:38.877 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.877 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.877 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:26:38.877 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:37.436187+0000", 2024-09-13T11:26:38.877 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.4857999999999995e-05 2024-09-13T11:26:38.878 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.878 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.878 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:26:38.878 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:37.436285+0000", 2024-09-13T11:26:38.878 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.8643999999999999e-05 2024-09-13T11:26:38.878 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.878 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.878 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 7", 2024-09-13T11:26:38.878 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:37.436423+0000", 2024-09-13T11:26:38.878 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00013809299999999999 2024-09-13T11:26:38.878 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.878 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.878 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:26:38.878 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:37.440125+0000", 2024-09-13T11:26:38.879 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0037015239999999999 2024-09-13T11:26:38.879 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.879 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.879 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:38.879 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:37.795397+0000", 2024-09-13T11:26:38.879 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.35527234400000002 2024-09-13T11:26:38.879 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.879 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.879 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:26:38.879 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:37.795485+0000", 2024-09-13T11:26:38.880 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.7538000000000001e-05 2024-09-13T11:26:38.880 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.880 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.880 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:26:38.880 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:37.795525+0000", 2024-09-13T11:26:38.880 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.0315999999999998e-05 2024-09-13T11:26:38.880 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:38.880 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:26:38.880 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:38.880 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.880 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.880 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:60433 3.3 3:c01f7ddc:::benchmark_data_smithi177_31116_object60432:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e85)", 2024-09-13T11:26:38.880 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:42.199382+0000", 2024-09-13T11:26:38.881 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 176.65517687900001, 2024-09-13T11:26:38.881 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.33918965200000001, 2024-09-13T11:26:38.881 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:26:38.881 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:26:38.881 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:38.881 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:26:38.881 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:26:38.881 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:38.881 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 60433 2024-09-13T11:26:38.881 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.881 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:26:38.881 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.881 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:26:38.882 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.199382+0000", 2024-09-13T11:26:38.882 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:38.882 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.882 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.882 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:26:38.882 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.199382+0000", 2024-09-13T11:26:38.882 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:38.882 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.882 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.882 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:26:38.882 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.199386+0000", 2024-09-13T11:26:38.882 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.8759999999999998e-06 2024-09-13T11:26:38.882 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.882 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.883 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:26:38.883 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.199455+0000", 2024-09-13T11:26:38.883 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.9305000000000001e-05 2024-09-13T11:26:38.883 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.883 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.883 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:26:38.883 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.199482+0000", 2024-09-13T11:26:38.883 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.6749000000000001e-05 2024-09-13T11:26:38.883 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.883 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.883 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:26:38.883 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.199503+0000", 2024-09-13T11:26:38.883 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.1137000000000001e-05 2024-09-13T11:26:38.883 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.884 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.884 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:26:38.884 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.199602+0000", 2024-09-13T11:26:38.884 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.8743e-05 2024-09-13T11:26:38.884 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.884 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.884 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:26:38.884 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.199784+0000", 2024-09-13T11:26:38.884 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00018265100000000001 2024-09-13T11:26:38.884 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.884 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.884 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 7", 2024-09-13T11:26:38.884 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.199955+0000", 2024-09-13T11:26:38.884 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00017067100000000001 2024-09-13T11:26:38.885 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.885 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.885 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:26:38.885 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.202075+0000", 2024-09-13T11:26:38.885 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0021206749999999998 2024-09-13T11:26:38.885 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.885 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.885 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:38.885 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.538395+0000", 2024-09-13T11:26:38.885 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.33631963599999998 2024-09-13T11:26:38.885 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.885 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.885 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:26:38.885 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.538498+0000", 2024-09-13T11:26:38.886 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00010267099999999999 2024-09-13T11:26:38.886 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.886 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.886 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:26:38.886 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.538571+0000", 2024-09-13T11:26:38.886 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.3538e-05 2024-09-13T11:26:38.886 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:38.886 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:26:38.886 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:38.886 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.886 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.886 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop_reply(client.4665.0:60433 3.3 e85/75 ondisk, result = 0)", 2024-09-13T11:26:38.886 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:42.203186+0000", 2024-09-13T11:26:38.886 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 176.65137270100001, 2024-09-13T11:26:38.887 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.33540096699999999, 2024-09-13T11:26:38.887 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:26:38.887 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:26:38.887 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:26:38.887 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:26:38.887 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.887 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:26:38.887 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.203186+0000", 2024-09-13T11:26:38.887 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:38.887 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.887 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.887 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:26:38.887 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.203186+0000", 2024-09-13T11:26:38.887 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:38.888 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.888 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.888 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:26:38.888 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.203194+0000", 2024-09-13T11:26:38.888 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.0020000000000006e-06 2024-09-13T11:26:38.888 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.888 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.888 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:26:38.888 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.203202+0000", 2024-09-13T11:26:38.888 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.9829999999999996e-06 2024-09-13T11:26:38.888 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.888 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.888 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:26:38.888 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.538182+0000", 2024-09-13T11:26:38.889 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.33498029899999998 2024-09-13T11:26:38.889 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.889 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.889 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:26:38.889 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.538196+0000", 2024-09-13T11:26:38.889 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3807e-05 2024-09-13T11:26:38.889 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.889 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.889 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:26:38.889 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.538371+0000", 2024-09-13T11:26:38.889 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00017541200000000001 2024-09-13T11:26:38.889 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.889 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.889 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:26:38.890 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.538386+0000", 2024-09-13T11:26:38.890 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4833e-05 2024-09-13T11:26:38.890 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.890 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.890 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:26:38.890 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.538587+0000", 2024-09-13T11:26:38.890 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000200631 2024-09-13T11:26:38.890 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:38.890 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:26:38.890 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:38.890 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.890 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.890 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:83144 3.18 3:1c2a5085:::benchmark_data_smithi177_31116_object83143:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106)", 2024-09-13T11:26:38.890 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:24:04.186395+0000", 2024-09-13T11:26:38.891 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 154.66816358200001, 2024-09-13T11:26:38.891 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.29655527599999998, 2024-09-13T11:26:38.891 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:26:38.891 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:26:38.891 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:38.891 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:26:38.891 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:26:38.891 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:38.891 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 83144 2024-09-13T11:26:38.891 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.891 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:26:38.891 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.891 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:26:38.891 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.186395+0000", 2024-09-13T11:26:38.892 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:38.892 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.892 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.892 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:26:38.892 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.186395+0000", 2024-09-13T11:26:38.892 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:38.892 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.892 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.892 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:26:38.892 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.186398+0000", 2024-09-13T11:26:38.892 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.5299999999999999e-06 2024-09-13T11:26:38.892 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.892 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.893 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:26:38.893 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.186438+0000", 2024-09-13T11:26:38.893 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.0293000000000002e-05 2024-09-13T11:26:38.893 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.893 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.893 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:26:38.893 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.186450+0000", 2024-09-13T11:26:38.893 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2687000000000001e-05 2024-09-13T11:26:38.893 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.893 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.893 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:26:38.893 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.186456+0000", 2024-09-13T11:26:38.893 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.446e-06 2024-09-13T11:26:38.893 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.894 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.894 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:26:38.894 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.186503+0000", 2024-09-13T11:26:38.894 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.7491000000000003e-05 2024-09-13T11:26:38.894 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.894 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.894 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:26:38.894 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.186618+0000", 2024-09-13T11:26:38.894 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000114999 2024-09-13T11:26:38.894 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.894 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.894 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 0", 2024-09-13T11:26:38.894 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.186712+0000", 2024-09-13T11:26:38.894 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.3805000000000001e-05 2024-09-13T11:26:38.895 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.895 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.895 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:26:38.895 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.189753+0000", 2024-09-13T11:26:38.895 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.003040713 2024-09-13T11:26:38.895 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.895 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.895 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:38.895 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.482662+0000", 2024-09-13T11:26:38.895 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.29290945699999998 2024-09-13T11:26:38.895 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.895 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.895 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:26:38.895 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.482831+0000", 2024-09-13T11:26:38.895 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00016900699999999999 2024-09-13T11:26:38.896 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.896 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.896 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:26:38.896 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.482950+0000", 2024-09-13T11:26:38.896 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000118848 2024-09-13T11:26:38.896 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:38.896 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:26:38.896 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:38.896 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.896 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.896 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop_reply(client.4665.0:83144 3.18 e106/91 ondisk, result = 0)", 2024-09-13T11:26:38.896 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:24:04.188413+0000", 2024-09-13T11:26:38.896 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 154.666145572, 2024-09-13T11:26:38.897 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.294557562, 2024-09-13T11:26:38.897 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:26:38.897 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:26:38.897 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:26:38.897 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:26:38.897 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.897 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:26:38.897 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.188413+0000", 2024-09-13T11:26:38.897 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:38.897 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.897 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.897 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:26:38.897 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.188413+0000", 2024-09-13T11:26:38.897 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:38.898 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.898 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.898 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:26:38.898 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.188414+0000", 2024-09-13T11:26:38.898 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4309999999999999e-06 2024-09-13T11:26:38.898 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.898 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.898 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:26:38.898 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.188419+0000", 2024-09-13T11:26:38.898 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.2889999999999999e-06 2024-09-13T11:26:38.898 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.898 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.898 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:26:38.898 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.482442+0000", 2024-09-13T11:26:38.899 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.294023168 2024-09-13T11:26:38.899 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.899 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.899 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:26:38.899 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.482459+0000", 2024-09-13T11:26:38.899 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.7013000000000001e-05 2024-09-13T11:26:38.899 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.899 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.899 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:26:38.899 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.482615+0000", 2024-09-13T11:26:38.899 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00015620400000000001 2024-09-13T11:26:38.899 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.899 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.900 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:26:38.900 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.482644+0000", 2024-09-13T11:26:38.900 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.9093000000000001e-05 2024-09-13T11:26:38.900 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.900 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.900 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:26:38.900 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.482971+0000", 2024-09-13T11:26:38.900 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00032636399999999999 2024-09-13T11:26:38.900 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:38.900 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:26:38.900 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:38.900 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.900 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.900 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPush(3.1f 106/78 [PushOp(3:fd4be819:::benchmark_data_smithi177_31116_object54560:head, version: 74'3410, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:fd4be819:::benchmark_data_smithi177_31116_object54560:head@74'3410, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:26:38.901 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:24:04.397161+0000", 2024-09-13T11:26:38.901 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 154.457397163, 2024-09-13T11:26:38.901 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.51611096199999995, 2024-09-13T11:26:38.901 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:26:38.901 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:26:38.901 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:26:38.901 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:26:38.901 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.901 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:26:38.901 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.397161+0000", 2024-09-13T11:26:38.901 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:38.901 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.901 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.902 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:26:38.902 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.397161+0000", 2024-09-13T11:26:38.902 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:38.902 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.902 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.902 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:26:38.902 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.397163+0000", 2024-09-13T11:26:38.902 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4720000000000001e-06 2024-09-13T11:26:38.902 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.902 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.902 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:26:38.902 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.397228+0000", 2024-09-13T11:26:38.902 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.5141999999999994e-05 2024-09-13T11:26:38.902 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.903 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.903 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:26:38.903 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.911583+0000", 2024-09-13T11:26:38.903 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.51435473700000001 2024-09-13T11:26:38.903 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.903 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.903 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:26:38.903 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.911615+0000", 2024-09-13T11:26:38.903 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.2620000000000003e-05 2024-09-13T11:26:38.903 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.903 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.903 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:26:38.903 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.911759+0000", 2024-09-13T11:26:38.904 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00014376300000000001 2024-09-13T11:26:38.904 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.904 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.904 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:26:38.904 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.911779+0000", 2024-09-13T11:26:38.904 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.0003000000000001e-05 2024-09-13T11:26:38.904 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.904 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.904 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:26:38.904 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.913272+0000", 2024-09-13T11:26:38.904 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0014932249999999999 2024-09-13T11:26:38.904 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:38.904 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:26:38.904 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:38.905 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.905 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.905 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:84809 3.17 3:e9e52123:::benchmark_data_smithi177_31116_object84808:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-09-13T11:26:38.905 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:24:05.914353+0000", 2024-09-13T11:26:38.905 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 152.94020514499999, 2024-09-13T11:26:38.905 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.27210915899999999, 2024-09-13T11:26:38.905 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:26:38.905 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:26:38.905 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:38.905 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:26:38.905 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:26:38.905 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:38.905 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 84809 2024-09-13T11:26:38.905 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.906 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:26:38.906 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.906 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:26:38.906 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.914353+0000", 2024-09-13T11:26:38.906 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:38.906 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.906 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.906 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:26:38.906 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.914353+0000", 2024-09-13T11:26:38.906 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:38.906 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.906 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.906 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:26:38.906 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.914356+0000", 2024-09-13T11:26:38.907 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.8710000000000001e-06 2024-09-13T11:26:38.907 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.907 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.907 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:26:38.907 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.914443+0000", 2024-09-13T11:26:38.907 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.7002000000000005e-05 2024-09-13T11:26:38.907 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.907 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.907 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:26:38.907 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.914461+0000", 2024-09-13T11:26:38.907 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.7691000000000001e-05 2024-09-13T11:26:38.907 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.907 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.907 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:26:38.908 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.914471+0000", 2024-09-13T11:26:38.908 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0447e-05 2024-09-13T11:26:38.908 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.908 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.908 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:26:38.908 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.914550+0000", 2024-09-13T11:26:38.908 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.8683000000000005e-05 2024-09-13T11:26:38.908 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.908 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.908 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:26:38.908 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.914709+0000", 2024-09-13T11:26:38.908 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00015913200000000001 2024-09-13T11:26:38.908 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.908 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.909 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 3", 2024-09-13T11:26:38.909 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.914880+0000", 2024-09-13T11:26:38.909 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00017071499999999999 2024-09-13T11:26:38.909 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.909 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.909 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:26:38.909 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.916701+0000", 2024-09-13T11:26:38.909 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0018205369999999999 2024-09-13T11:26:38.909 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.909 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.909 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:38.909 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:06.186401+0000", 2024-09-13T11:26:38.909 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.26970018699999998 2024-09-13T11:26:38.910 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.910 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.910 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:26:38.910 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:06.186434+0000", 2024-09-13T11:26:38.910 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.3448999999999998e-05 2024-09-13T11:26:38.910 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.910 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.910 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:26:38.910 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:06.186463+0000", 2024-09-13T11:26:38.910 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.8445e-05 2024-09-13T11:26:38.910 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:38.910 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:26:38.910 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:38.910 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.911 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.911 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop_reply(client.4665.0:84809 3.17 e107/85 ondisk, result = 0)", 2024-09-13T11:26:38.911 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:24:05.916779+0000", 2024-09-13T11:26:38.911 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 152.93777967400001, 2024-09-13T11:26:38.911 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.26968993499999999, 2024-09-13T11:26:38.911 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:26:38.911 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:26:38.911 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:26:38.911 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:26:38.911 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.911 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:26:38.911 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.916779+0000", 2024-09-13T11:26:38.911 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:38.911 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.912 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.912 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:26:38.912 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.916779+0000", 2024-09-13T11:26:38.912 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:38.912 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.912 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.912 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:26:38.912 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.916780+0000", 2024-09-13T11:26:38.912 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.328e-06 2024-09-13T11:26:38.912 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.912 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.912 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:26:38.912 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.916789+0000", 2024-09-13T11:26:38.913 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.4460000000000001e-06 2024-09-13T11:26:38.913 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.913 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.913 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:26:38.913 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:06.186296+0000", 2024-09-13T11:26:38.913 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.26950707099999999 2024-09-13T11:26:38.913 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.913 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.913 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:26:38.913 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:06.186306+0000", 2024-09-13T11:26:38.913 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0137e-05 2024-09-13T11:26:38.913 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.913 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.913 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:26:38.914 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:06.186388+0000", 2024-09-13T11:26:38.914 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.2149000000000002e-05 2024-09-13T11:26:38.914 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.914 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.914 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:26:38.914 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:06.186397+0000", 2024-09-13T11:26:38.914 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.5210000000000002e-06 2024-09-13T11:26:38.914 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.914 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.914 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:26:38.914 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:06.186469+0000", 2024-09-13T11:26:38.914 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.2282999999999999e-05 2024-09-13T11:26:38.914 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:38.914 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:26:38.915 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:38.915 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.915 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.915 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPush(3.2a 121/120 [PushOp(3:5595108b:::benchmark_data_smithi177_31116_object899:head, version: 43'62, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:5595108b:::benchmark_data_smithi177_31116_object899:head@43'62, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:26:38.915 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:24:27.352355+0000", 2024-09-13T11:26:38.915 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 131.50220386800001, 2024-09-13T11:26:38.915 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.61628666099999996, 2024-09-13T11:26:38.915 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:26:38.915 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:26:38.915 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:26:38.915 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:26:38.915 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.915 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:26:38.915 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:27.352355+0000", 2024-09-13T11:26:38.916 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:38.916 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.916 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.916 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:26:38.916 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:27.352355+0000", 2024-09-13T11:26:38.916 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:38.916 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.916 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.916 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:26:38.916 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:27.352357+0000", 2024-09-13T11:26:38.916 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.328e-06 2024-09-13T11:26:38.916 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.916 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.916 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:26:38.917 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:27.352428+0000", 2024-09-13T11:26:38.917 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.1118999999999995e-05 2024-09-13T11:26:38.917 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.917 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.917 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:26:38.917 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:27.966950+0000", 2024-09-13T11:26:38.917 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.61452202499999997 2024-09-13T11:26:38.917 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.917 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.917 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:26:38.917 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:27.966980+0000", 2024-09-13T11:26:38.917 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.9433000000000001e-05 2024-09-13T11:26:38.917 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.917 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.918 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:26:38.918 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:27.967220+0000", 2024-09-13T11:26:38.918 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00024065499999999999 2024-09-13T11:26:38.918 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.918 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.918 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:26:38.918 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:27.967244+0000", 2024-09-13T11:26:38.918 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.4216e-05 2024-09-13T11:26:38.918 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.918 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.918 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:26:38.918 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:27.968641+0000", 2024-09-13T11:26:38.918 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0013968850000000001 2024-09-13T11:26:38.918 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:38.919 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:26:38.919 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:38.919 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.919 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.919 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:162057 3.1f 3:f8710094:::benchmark_data_smithi177_31116_object21754:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e156)", 2024-09-13T11:26:38.919 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:25:15.442781+0000", 2024-09-13T11:26:38.919 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 83.411777224999994, 2024-09-13T11:26:38.919 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.25670282500000002, 2024-09-13T11:26:38.919 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:26:38.919 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:26:38.919 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:38.919 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:26:38.919 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:26:38.919 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:38.919 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 162057 2024-09-13T11:26:38.920 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.920 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:26:38.920 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.920 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:26:38.920 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:15.442781+0000", 2024-09-13T11:26:38.920 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:38.920 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.920 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.920 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:26:38.920 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:15.442781+0000", 2024-09-13T11:26:38.920 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:38.920 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.920 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.920 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:26:38.921 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:15.442785+0000", 2024-09-13T11:26:38.921 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.4000000000000001e-06 2024-09-13T11:26:38.921 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.921 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.921 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:26:38.921 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:15.442795+0000", 2024-09-13T11:26:38.921 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0193999999999999e-05 2024-09-13T11:26:38.921 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.921 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.921 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:26:38.921 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:15.442820+0000", 2024-09-13T11:26:38.922 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.4683000000000001e-05 2024-09-13T11:26:38.922 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.922 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.922 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:26:38.922 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:15.442840+0000", 2024-09-13T11:26:38.922 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.0596000000000001e-05 2024-09-13T11:26:38.922 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.922 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.922 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:26:38.922 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:15.442952+0000", 2024-09-13T11:26:38.922 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00011215899999999999 2024-09-13T11:26:38.922 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.923 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.923 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:26:38.923 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:15.443143+0000", 2024-09-13T11:26:38.923 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00019058899999999999 2024-09-13T11:26:38.923 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.923 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.923 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 5", 2024-09-13T11:26:38.923 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:15.443327+0000", 2024-09-13T11:26:38.923 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00018364499999999999 2024-09-13T11:26:38.923 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.923 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.924 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:26:38.924 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:15.446727+0000", 2024-09-13T11:26:38.924 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0034000689999999999 2024-09-13T11:26:38.924 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.924 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.924 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:38.924 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:15.699356+0000", 2024-09-13T11:26:38.924 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.25262958400000002 2024-09-13T11:26:38.924 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.924 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.924 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:26:38.924 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:15.699428+0000", 2024-09-13T11:26:38.924 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.1810000000000005e-05 2024-09-13T11:26:38.924 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.925 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.925 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:26:38.925 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:15.699484+0000", 2024-09-13T11:26:38.925 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.6096e-05 2024-09-13T11:26:38.925 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:38.925 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:26:38.925 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:38.925 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.925 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.925 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:162073 3.1f 3:fd1a00ae:::benchmark_data_smithi177_31116_object21770:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e156)", 2024-09-13T11:26:38.925 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:25:15.454353+0000", 2024-09-13T11:26:38.925 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 83.400205270000001, 2024-09-13T11:26:38.925 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.27516647, 2024-09-13T11:26:38.925 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:26:38.926 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:26:38.926 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:38.926 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:26:38.926 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:26:38.926 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:38.926 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 162073 2024-09-13T11:26:38.926 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.926 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:26:38.926 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.926 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:26:38.926 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:15.454353+0000", 2024-09-13T11:26:38.926 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:38.926 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.927 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.927 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:26:38.927 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:15.454353+0000", 2024-09-13T11:26:38.927 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:38.927 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.927 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.927 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:26:38.927 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:15.454358+0000", 2024-09-13T11:26:38.927 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.3499999999999999e-06 2024-09-13T11:26:38.927 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.927 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.927 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:26:38.927 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:15.454371+0000", 2024-09-13T11:26:38.927 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3594000000000001e-05 2024-09-13T11:26:38.928 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.928 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.928 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:26:38.928 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:15.454408+0000", 2024-09-13T11:26:38.928 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.6619999999999998e-05 2024-09-13T11:26:38.928 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.928 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.928 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:26:38.928 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:15.454431+0000", 2024-09-13T11:26:38.928 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.3546000000000001e-05 2024-09-13T11:26:38.928 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.928 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.928 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:26:38.928 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:15.454587+0000", 2024-09-13T11:26:38.929 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000155248 2024-09-13T11:26:38.929 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.929 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.929 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:26:38.929 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:15.454897+0000", 2024-09-13T11:26:38.929 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000309955 2024-09-13T11:26:38.929 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.929 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.929 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 5", 2024-09-13T11:26:38.929 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:15.455203+0000", 2024-09-13T11:26:38.929 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00030604999999999999 2024-09-13T11:26:38.929 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.929 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.929 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:26:38.930 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:15.460534+0000", 2024-09-13T11:26:38.930 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0053310199999999997 2024-09-13T11:26:38.930 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.930 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.930 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:38.930 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:15.729356+0000", 2024-09-13T11:26:38.930 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.26882267199999998 2024-09-13T11:26:38.930 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.930 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.930 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:26:38.930 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:15.729442+0000", 2024-09-13T11:26:38.930 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.5209000000000004e-05 2024-09-13T11:26:38.930 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.931 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.931 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:26:38.931 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:15.729520+0000", 2024-09-13T11:26:38.931 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.8206000000000003e-05 2024-09-13T11:26:38.931 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:38.931 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:26:38.931 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:38.931 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.931 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.931 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:162102 3.1f 3:fc0cda6c:::benchmark_data_smithi177_31116_object21799:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e156)", 2024-09-13T11:26:38.931 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:25:15.477044+0000", 2024-09-13T11:26:38.931 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 83.377514415999997, 2024-09-13T11:26:38.931 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.28798895800000002, 2024-09-13T11:26:38.931 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:26:38.932 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:26:38.932 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:38.932 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:26:38.932 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:26:38.932 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:38.932 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 162102 2024-09-13T11:26:38.932 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.932 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:26:38.932 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.932 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:26:38.932 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:15.477044+0000", 2024-09-13T11:26:38.932 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:38.932 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.932 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.933 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:26:38.933 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:15.477044+0000", 2024-09-13T11:26:38.933 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:38.933 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.933 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.933 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:26:38.933 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:15.477048+0000", 2024-09-13T11:26:38.933 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.3569999999999998e-06 2024-09-13T11:26:38.933 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.933 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.933 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:26:38.933 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:15.477058+0000", 2024-09-13T11:26:38.933 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.9899999999999992e-06 2024-09-13T11:26:38.934 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.934 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.934 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:26:38.934 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:15.477087+0000", 2024-09-13T11:26:38.934 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.9026999999999998e-05 2024-09-13T11:26:38.934 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.934 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.934 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:26:38.934 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:15.477101+0000", 2024-09-13T11:26:38.934 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4583000000000001e-05 2024-09-13T11:26:38.934 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.934 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.934 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:26:38.934 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:15.477212+0000", 2024-09-13T11:26:38.935 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00011071200000000001 2024-09-13T11:26:38.935 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.935 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.935 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:26:38.935 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:15.477492+0000", 2024-09-13T11:26:38.935 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00027995400000000001 2024-09-13T11:26:38.935 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.935 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.935 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 5", 2024-09-13T11:26:38.935 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:15.477682+0000", 2024-09-13T11:26:38.935 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000189929 2024-09-13T11:26:38.935 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.935 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.935 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:26:38.936 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:15.481903+0000", 2024-09-13T11:26:38.936 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0042214649999999998 2024-09-13T11:26:38.936 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.936 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.936 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:38.936 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:15.764891+0000", 2024-09-13T11:26:38.936 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.28298753599999998 2024-09-13T11:26:38.936 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.936 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.936 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:26:38.936 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:15.764972+0000", 2024-09-13T11:26:38.936 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.1230000000000007e-05 2024-09-13T11:26:38.936 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.936 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.937 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:26:38.937 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:15.765033+0000", 2024-09-13T11:26:38.937 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.1174999999999996e-05 2024-09-13T11:26:38.937 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:38.937 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:26:38.937 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:38.937 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.937 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.937 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:191286 3.18 3:1ffc0b26:::benchmark_data_smithi177_31116_object50983:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e175)", 2024-09-13T11:26:38.937 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:25:39.152634+0000", 2024-09-13T11:26:38.937 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 59.701924235, 2024-09-13T11:26:38.937 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.236097576, 2024-09-13T11:26:38.937 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:26:38.937 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:26:38.938 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:38.938 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:26:38.938 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:26:38.938 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:38.938 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 191286 2024-09-13T11:26:38.938 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.938 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:26:38.938 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.938 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:26:38.938 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:39.152634+0000", 2024-09-13T11:26:38.938 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:38.938 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.938 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.938 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:26:38.939 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:39.152634+0000", 2024-09-13T11:26:38.939 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:38.939 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.939 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.939 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:26:38.939 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:39.152638+0000", 2024-09-13T11:26:38.939 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.5130000000000001e-06 2024-09-13T11:26:38.939 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.939 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.939 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:26:38.939 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:39.152649+0000", 2024-09-13T11:26:38.939 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0677e-05 2024-09-13T11:26:38.939 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.940 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.940 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:26:38.940 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:39.152678+0000", 2024-09-13T11:26:38.940 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.9436000000000001e-05 2024-09-13T11:26:38.940 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.940 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.940 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:26:38.940 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:39.152693+0000", 2024-09-13T11:26:38.940 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4823e-05 2024-09-13T11:26:38.940 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.940 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.940 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:26:38.940 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:39.152821+0000", 2024-09-13T11:26:38.940 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00012815299999999999 2024-09-13T11:26:38.941 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.941 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.941 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:26:38.941 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:39.153031+0000", 2024-09-13T11:26:38.941 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00021011399999999999 2024-09-13T11:26:38.941 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.941 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.941 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 0", 2024-09-13T11:26:38.941 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:39.153195+0000", 2024-09-13T11:26:38.941 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00016374599999999999 2024-09-13T11:26:38.941 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.941 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.941 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:26:38.941 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:39.157900+0000", 2024-09-13T11:26:38.942 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0047053049999999999 2024-09-13T11:26:38.942 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.942 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.942 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:38.942 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:39.388451+0000", 2024-09-13T11:26:38.942 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.230550957 2024-09-13T11:26:38.942 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.942 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.942 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:26:38.942 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:39.388611+0000", 2024-09-13T11:26:38.942 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00016029299999999999 2024-09-13T11:26:38.942 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.942 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.942 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:26:38.943 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:39.388732+0000", 2024-09-13T11:26:38.943 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000120559 2024-09-13T11:26:38.943 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:38.943 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:26:38.943 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:38.943 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.943 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.943 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:221532 3.17 3:ec6fb681:::benchmark_data_smithi177_31116_object81229:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e191)", 2024-09-13T11:26:38.943 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:26:04.303286+0000", 2024-09-13T11:26:38.943 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 34.551272668999999, 2024-09-13T11:26:38.943 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.79092058600000004, 2024-09-13T11:26:38.943 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:26:38.943 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:26:38.943 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:38.944 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:26:38.944 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:26:38.944 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:38.944 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 221532 2024-09-13T11:26:38.944 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.944 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:26:38.944 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.944 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:26:38.944 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:04.303286+0000", 2024-09-13T11:26:38.944 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:38.944 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.944 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.944 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:26:38.944 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:04.303286+0000", 2024-09-13T11:26:38.945 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:38.945 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.945 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.945 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:26:38.945 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:04.303290+0000", 2024-09-13T11:26:38.945 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.9600000000000002e-06 2024-09-13T11:26:38.945 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.945 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.945 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:26:38.945 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:04.303302+0000", 2024-09-13T11:26:38.945 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2126e-05 2024-09-13T11:26:38.945 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.945 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.946 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:26:38.946 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:04.303332+0000", 2024-09-13T11:26:38.946 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.0015999999999999e-05 2024-09-13T11:26:38.946 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.946 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.946 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:26:38.946 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:04.303347+0000", 2024-09-13T11:26:38.946 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5324000000000001e-05 2024-09-13T11:26:38.946 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.946 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.946 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:26:38.946 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:04.303484+0000", 2024-09-13T11:26:38.946 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00013702199999999999 2024-09-13T11:26:38.946 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.946 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.947 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:26:38.947 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:04.303700+0000", 2024-09-13T11:26:38.947 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00021570500000000001 2024-09-13T11:26:38.947 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.947 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.947 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 3", 2024-09-13T11:26:38.947 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:04.303868+0000", 2024-09-13T11:26:38.947 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00016751799999999999 2024-09-13T11:26:38.947 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.947 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.947 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:26:38.947 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:04.309489+0000", 2024-09-13T11:26:38.947 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.005621455 2024-09-13T11:26:38.947 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.947 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.948 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:38.948 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:05.094014+0000", 2024-09-13T11:26:38.948 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.78452480499999999 2024-09-13T11:26:38.948 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.948 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.948 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:26:38.948 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:05.094132+0000", 2024-09-13T11:26:38.948 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000118093 2024-09-13T11:26:38.948 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.948 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.948 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:26:38.948 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:05.094206+0000", 2024-09-13T11:26:38.948 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.4561999999999995e-05 2024-09-13T11:26:38.949 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:38.949 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:26:38.949 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:38.949 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.949 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.949 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop_reply(client.4665.0:221532 3.17 e191/85 ondisk, result = 0)", 2024-09-13T11:26:38.949 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:26:04.309386+0000", 2024-09-13T11:26:38.949 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 34.545172366000003, 2024-09-13T11:26:38.949 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.78483394299999998, 2024-09-13T11:26:38.949 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:26:38.949 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:26:38.949 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:26:38.949 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:26:38.949 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.950 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:26:38.950 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:04.309386+0000", 2024-09-13T11:26:38.950 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:38.950 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.950 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.950 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:26:38.950 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:04.309386+0000", 2024-09-13T11:26:38.950 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:38.950 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.950 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.950 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:26:38.950 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:04.309390+0000", 2024-09-13T11:26:38.950 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.5700000000000001e-06 2024-09-13T11:26:38.950 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.951 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.951 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:26:38.951 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:04.309403+0000", 2024-09-13T11:26:38.951 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3567000000000001e-05 2024-09-13T11:26:38.951 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.951 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.951 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:26:38.951 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:05.093783+0000", 2024-09-13T11:26:38.951 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.78437944699999995 2024-09-13T11:26:38.951 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.951 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.951 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:26:38.951 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:05.093802+0000", 2024-09-13T11:26:38.952 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.8816000000000001e-05 2024-09-13T11:26:38.952 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.952 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.952 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:26:38.952 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:05.093968+0000", 2024-09-13T11:26:38.952 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000166572 2024-09-13T11:26:38.952 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.952 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.952 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:26:38.952 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:05.093999+0000", 2024-09-13T11:26:38.952 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.0352999999999999e-05 2024-09-13T11:26:38.952 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.952 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.952 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:26:38.953 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:05.094220+0000", 2024-09-13T11:26:38.953 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000221618 2024-09-13T11:26:38.953 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:38.953 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:26:38.953 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:38.953 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.953 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.953 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:227356 3.18 3:1c24ab12:::benchmark_data_smithi177_31116_object87053:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e196)", 2024-09-13T11:26:38.953 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:26:10.982466+0000", 2024-09-13T11:26:38.953 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 27.872092255999998, 2024-09-13T11:26:38.953 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.23166883999999999, 2024-09-13T11:26:38.953 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:26:38.953 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:26:38.953 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:38.953 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:26:38.954 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:26:38.954 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:38.954 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 227356 2024-09-13T11:26:38.954 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.954 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:26:38.954 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.954 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:26:38.954 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:10.982466+0000", 2024-09-13T11:26:38.954 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:38.954 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.954 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.954 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:26:38.954 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:10.982466+0000", 2024-09-13T11:26:38.954 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:38.955 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.955 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.955 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:26:38.955 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:10.982469+0000", 2024-09-13T11:26:38.955 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.2500000000000001e-06 2024-09-13T11:26:38.955 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.955 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.955 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:26:38.955 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:10.982479+0000", 2024-09-13T11:26:38.955 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0644e-05 2024-09-13T11:26:38.955 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.955 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.955 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:26:38.955 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:10.982494+0000", 2024-09-13T11:26:38.956 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4634000000000001e-05 2024-09-13T11:26:38.956 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.956 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.956 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:26:38.956 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:10.982502+0000", 2024-09-13T11:26:38.956 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.2120000000000003e-06 2024-09-13T11:26:38.956 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.956 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.956 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:26:38.956 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:10.982567+0000", 2024-09-13T11:26:38.956 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.5168999999999999e-05 2024-09-13T11:26:38.956 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.956 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.956 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:26:38.957 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:10.982715+0000", 2024-09-13T11:26:38.957 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00014747499999999999 2024-09-13T11:26:38.957 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.957 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.957 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 0", 2024-09-13T11:26:38.957 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:10.982839+0000", 2024-09-13T11:26:38.957 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00012430000000000001 2024-09-13T11:26:38.957 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.957 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.957 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:26:38.957 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:10.989491+0000", 2024-09-13T11:26:38.957 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0066519989999999996 2024-09-13T11:26:38.957 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.957 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.958 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:38.958 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:11.213989+0000", 2024-09-13T11:26:38.958 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.22449838999999999 2024-09-13T11:26:38.958 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.958 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.958 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:26:38.958 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:11.214049+0000", 2024-09-13T11:26:38.958 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.9431000000000001e-05 2024-09-13T11:26:38.958 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:38.958 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:38.958 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:26:38.958 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:11.214135+0000", 2024-09-13T11:26:38.958 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.6335999999999995e-05 2024-09-13T11:26:38.958 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:38.958 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:26:38.959 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:38.959 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:38.959 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:26:38.959 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-13T11:26:38.960 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-13T11:26:38.961 INFO:tasks.ceph.osd.9.smithi092.stderr:2024-09-13T11:26:38.955+0000 7ff5ab3d0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26348) UID: 0 2024-09-13T11:26:39.061 INFO:tasks.ceph.osd.9.smithi092.stderr:2024-09-13T11:26:39.059+0000 7ff5ab3d0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26348) UID: 0 2024-09-13T11:26:39.112 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-09-13T11:26:39.112 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [ 2024-09-13T11:26:39.113 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:39.113 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:258156 3.14 3:2913f7a1:::benchmark_data_smithi177_31116_object117853:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e215)", 2024-09-13T11:26:39.113 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:26:39.112425+0000", 2024-09-13T11:26:39.113 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 0.0013286280000000001, 2024-09-13T11:26:39.113 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0013539509999999999, 2024-09-13T11:26:39.113 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:26:39.113 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:26:39.113 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "waiting for sub ops", 2024-09-13T11:26:39.113 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:26:39.113 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:26:39.113 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:39.113 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 258156 2024-09-13T11:26:39.113 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:39.114 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:26:39.114 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:39.114 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:26:39.114 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:39.112425+0000", 2024-09-13T11:26:39.114 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:39.114 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:39.114 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:39.114 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:26:39.114 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:39.112425+0000", 2024-09-13T11:26:39.114 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:39.114 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:39.114 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:39.114 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:26:39.114 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:39.112426+0000", 2024-09-13T11:26:39.115 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.7969999999999999e-06 2024-09-13T11:26:39.115 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:39.115 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:39.115 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:26:39.115 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:39.112431+0000", 2024-09-13T11:26:39.115 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.6369999999999998e-06 2024-09-13T11:26:39.115 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:39.115 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:39.115 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:26:39.115 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:39.112444+0000", 2024-09-13T11:26:39.115 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.2867999999999999e-05 2024-09-13T11:26:39.115 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:39.115 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:39.116 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:26:39.116 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:39.112450+0000", 2024-09-13T11:26:39.116 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.925e-06 2024-09-13T11:26:39.116 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:39.116 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:39.116 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:26:39.116 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:39.112509+0000", 2024-09-13T11:26:39.116 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.9308999999999997e-05 2024-09-13T11:26:39.116 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:39.116 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:39.116 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:26:39.116 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:39.112614+0000", 2024-09-13T11:26:39.116 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000104597 2024-09-13T11:26:39.116 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:39.117 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:39.117 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 14,15", 2024-09-13T11:26:39.117 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:39.112686+0000", 2024-09-13T11:26:39.117 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.2619000000000004e-05 2024-09-13T11:26:39.117 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:39.117 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:26:39.117 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:39.117 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:39.117 INFO:teuthology.orchestra.run.smithi177.stdout: ], 2024-09-13T11:26:39.117 INFO:teuthology.orchestra.run.smithi177.stdout: "num_ops": 1 2024-09-13T11:26:39.117 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-09-13T11:26:39.130 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-13T11:26:39.161 INFO:tasks.ceph.osd.14.smithi162.stderr:2024-09-13T11:26:39.159+0000 7f3df5903640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33154) UID: 0 2024-09-13T11:26:39.213 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T11:26:39.253 INFO:tasks.ceph.osd.15.smithi177.stderr:2024-09-13T11:26:39.249+0000 7f9d618a4640 -1 osd.15 215 handle_osd_map injecting map crc failure 2024-09-13T11:26:39.258 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-09-13T11:26:39.255+0000 7f97b59fa640 -1 osd.6 215 handle_osd_map injecting map crc failure 2024-09-13T11:26:39.263 INFO:tasks.ceph.osd.3.smithi177.stderr:2024-09-13T11:26:39.257+0000 7f10adba4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26574) UID: 0 2024-09-13T11:26:39.284 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-09-13T11:26:39.284 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [], 2024-09-13T11:26:39.284 INFO:teuthology.orchestra.run.smithi177.stdout: "complaint_time": 30, 2024-09-13T11:26:39.284 INFO:teuthology.orchestra.run.smithi177.stdout: "num_blocked_ops": 0 2024-09-13T11:26:39.284 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-09-13T11:26:39.302 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-13T11:26:39.363 INFO:tasks.ceph.osd.10.smithi162.stderr:2024-09-13T11:26:39.359+0000 7fb59c9f0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26317) UID: 0 2024-09-13T11:26:39.385 INFO:teuthology.orchestra.run.smithi001.stdout:osd.0: {} 2024-09-13T11:26:39.385 INFO:teuthology.orchestra.run.smithi001.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T11:26:39.408 INFO:teuthology.orchestra.run.smithi001.stdout:osd.1: {} 2024-09-13T11:26:39.408 INFO:teuthology.orchestra.run.smithi001.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T11:26:39.430 INFO:teuthology.orchestra.run.smithi001.stdout:osd.2: {} 2024-09-13T11:26:39.430 INFO:teuthology.orchestra.run.smithi001.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T11:26:39.453 INFO:teuthology.orchestra.run.smithi001.stdout:osd.3: {} 2024-09-13T11:26:39.453 INFO:teuthology.orchestra.run.smithi001.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T11:26:39.464 INFO:tasks.ceph.osd.11.smithi177.stderr:2024-09-13T11:26:39.461+0000 7f24792c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26580) UID: 0 2024-09-13T11:26:39.475 INFO:teuthology.orchestra.run.smithi001.stdout:osd.4: {} 2024-09-13T11:26:39.476 INFO:teuthology.orchestra.run.smithi001.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T11:26:39.483 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-09-13T11:26:39.483 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:26:39.498 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-13T11:26:39.499 INFO:teuthology.orchestra.run.smithi001.stdout:osd.5: {} 2024-09-13T11:26:39.499 INFO:teuthology.orchestra.run.smithi001.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T11:26:39.522 INFO:teuthology.orchestra.run.smithi001.stdout:osd.6: {} 2024-09-13T11:26:39.522 INFO:teuthology.orchestra.run.smithi001.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T11:26:39.544 INFO:teuthology.orchestra.run.smithi001.stdout:osd.7: {} 2024-09-13T11:26:39.544 INFO:teuthology.orchestra.run.smithi001.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T11:26:39.565 INFO:tasks.ceph.osd.14.smithi162.stderr:2024-09-13T11:26:39.563+0000 7f3df5903640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33154) UID: 0 2024-09-13T11:26:39.567 INFO:teuthology.orchestra.run.smithi001.stdout:osd.8: {} 2024-09-13T11:26:39.567 INFO:teuthology.orchestra.run.smithi001.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-13T11:26:39.592 INFO:teuthology.orchestra.run.smithi001.stdout:osd.9: {} 2024-09-13T11:26:39.593 INFO:teuthology.orchestra.run.smithi001.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-13T11:26:39.616 INFO:teuthology.orchestra.run.smithi001.stdout:osd.10: {} 2024-09-13T11:26:39.616 INFO:teuthology.orchestra.run.smithi001.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-13T11:26:39.641 INFO:teuthology.orchestra.run.smithi001.stdout:osd.11: {} 2024-09-13T11:26:39.641 INFO:teuthology.orchestra.run.smithi001.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-13T11:26:39.660 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-09-13T11:26:39.660 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:26:39.665 INFO:teuthology.orchestra.run.smithi001.stdout:osd.12: {} 2024-09-13T11:26:39.665 INFO:teuthology.orchestra.run.smithi001.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-13T11:26:39.666 INFO:tasks.ceph.osd.13.smithi092.stderr:2024-09-13T11:26:39.663+0000 7f3e5afea640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26350) UID: 0 2024-09-13T11:26:39.673 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-13T11:26:39.767 INFO:tasks.ceph.osd.7.smithi177.stderr:2024-09-13T11:26:39.765+0000 7f897b51d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26575) UID: 0 2024-09-13T11:26:39.798 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-09-13T11:26:39.798 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:26:39.812 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-13T11:26:39.820 INFO:teuthology.orchestra.run.smithi001.stdout:osd.13: {} 2024-09-13T11:26:39.821 INFO:teuthology.orchestra.run.smithi001.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-13T11:26:39.856 INFO:teuthology.orchestra.run.smithi001.stdout:osd.14: {} 2024-09-13T11:26:39.856 INFO:teuthology.orchestra.run.smithi001.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-13T11:26:39.867 INFO:tasks.ceph.osd.12.smithi001.stderr:2024-09-13T11:26:39.866+0000 7f892c6e6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26314) UID: 0 2024-09-13T11:26:39.899 INFO:teuthology.orchestra.run.smithi001.stdout:osd.15: {} 2024-09-13T11:26:39.899 INFO:teuthology.orchestra.run.smithi001.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-13T11:26:39.969 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-09-13T11:26:39.967+0000 7f97befc9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26315) UID: 0 2024-09-13T11:26:39.971 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-09-13T11:26:39.971 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:26:39.984 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-09-13T11:26:40.070 INFO:tasks.ceph.osd.7.smithi177.stderr:2024-09-13T11:26:40.065+0000 7f897b51d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26575) UID: 0 2024-09-13T11:26:40.092 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-09-13T11:26:40.092 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:26:40.105 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-09-13T11:26:40.171 INFO:tasks.ceph.osd.12.smithi001.stderr:2024-09-13T11:26:40.170+0000 7f892c6e6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26314) UID: 0 2024-09-13T11:26:40.265 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-09-13T11:26:40.265 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:26:40.272 INFO:tasks.ceph.osd.9.smithi092.stderr:2024-09-13T11:26:40.267+0000 7ff5ab3d0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26348) UID: 0 2024-09-13T11:26:40.279 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-09-13T11:26:40.290 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 15, 14] out_osds: [] dead_osds: [] live_osds: [0, 4, 8, 12, 5, 9, 13, 2, 6, 10, 3, 7, 11, 15, 1, 14] 2024-09-13T11:26:40.290 INFO:tasks.thrashosds.thrasher:choose_action: min_in 2 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-13T11:26:40.290 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-09-13T11:26:40.290 INFO:tasks.thrashosds.thrasher:Setting osd 14 primary_affinity to 0.826624 2024-09-13T11:26:40.290 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 14 0.8266239794061151 2024-09-13T11:26:40.360 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-13T11:26:40.413 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-13T11:26:40.410+0000 7f6a68775640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26307) UID: 0 2024-09-13T11:26:40.417 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-09-13T11:26:40.418 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:26:40.440 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_ops_in_flight 2024-09-13T11:26:40.474 INFO:tasks.ceph.osd.5.smithi092.stderr:2024-09-13T11:26:40.471+0000 7f6f0b705640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26346) UID: 0 2024-09-13T11:26:40.575 INFO:tasks.ceph.osd.14.smithi162.stderr:2024-09-13T11:26:40.571+0000 7f3df5903640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33154) UID: 0 2024-09-13T11:26:40.599 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-09-13T11:26:40.599 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:26:40.621 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_blocked_ops 2024-09-13T11:26:40.676 INFO:tasks.ceph.osd.11.smithi177.stderr:2024-09-13T11:26:40.673+0000 7f24792c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26580) UID: 0 2024-09-13T11:26:40.761 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-09-13T11:26:40.762 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:26:40.777 INFO:tasks.ceph.osd.15.smithi177.stderr:2024-09-13T11:26:40.773+0000 7f9d6ce77640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26578) UID: 0 2024-09-13T11:26:40.778 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_historic_ops 2024-09-13T11:26:40.878 INFO:tasks.ceph.osd.10.smithi162.stderr:2024-09-13T11:26:40.875+0000 7fb59c9f0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26317) UID: 0 2024-09-13T11:26:40.950 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-09-13T11:26:40.950 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:26:40.968 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-13T11:26:40.979 INFO:tasks.ceph.osd.11.smithi177.stderr:2024-09-13T11:26:40.977+0000 7f24792c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26580) UID: 0 2024-09-13T11:26:41.080 INFO:tasks.ceph.osd.14.smithi162.stderr:2024-09-13T11:26:41.075+0000 7f3df5903640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33154) UID: 0 2024-09-13T11:26:41.120 INFO:teuthology.orchestra.run.smithi092.stdout:ERROR: (22) Invalid argument 2024-09-13T11:26:41.120 INFO:teuthology.orchestra.run.smithi092.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:26:41.136 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-13T11:26:41.180 INFO:tasks.ceph.osd.8.smithi001.stderr:2024-09-13T11:26:41.178+0000 7fb9b48ae640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26312) UID: 0 2024-09-13T11:26:41.259 INFO:teuthology.orchestra.run.smithi001.stderr:nodeep-scrub is set 2024-09-13T11:26:41.267 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-13T11:26:41.262+0000 7f6a5d1a2640 -1 osd.0 217 handle_osd_map injecting map crc failure 2024-09-13T11:26:41.267 INFO:teuthology.orchestra.run.smithi001.stderr:set osd.14 primary-affinity to 0.826624 (8541732) 2024-09-13T11:26:41.279 INFO:teuthology.orchestra.run.smithi092.stdout:ERROR: (22) Invalid argument 2024-09-13T11:26:41.279 INFO:teuthology.orchestra.run.smithi092.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:26:41.281 INFO:tasks.ceph.osd.8.smithi001.stderr:2024-09-13T11:26:41.278+0000 7fb9b48ae640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26312) UID: 0 2024-09-13T11:26:41.295 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-13T11:26:41.381 INFO:tasks.ceph.osd.5.smithi092.stderr:2024-09-13T11:26:41.379+0000 7f6f0b705640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26346) UID: 0 2024-09-13T11:26:41.423 INFO:teuthology.orchestra.run.smithi092.stdout:ERROR: (22) Invalid argument 2024-09-13T11:26:41.424 INFO:teuthology.orchestra.run.smithi092.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:26:41.440 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_ops_in_flight 2024-09-13T11:26:41.483 INFO:tasks.ceph.osd.15.smithi177.stderr:2024-09-13T11:26:41.481+0000 7f9d6ce77640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26578) UID: 0 2024-09-13T11:26:41.584 INFO:tasks.ceph.osd.15.smithi177.stderr:2024-09-13T11:26:41.581+0000 7f9d6ce77640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26578) UID: 0 2024-09-13T11:26:41.584 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-13T11:26:41.584 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:26:41.603 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_blocked_ops 2024-09-13T11:26:41.685 INFO:tasks.ceph.osd.2.smithi162.stderr:2024-09-13T11:26:41.683+0000 7fe92b35a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26313) UID: 0 2024-09-13T11:26:41.740 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-13T11:26:41.740 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:26:41.761 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_historic_ops 2024-09-13T11:26:41.809 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-09-13T11:26:41.807+0000 7f97befc9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26315) UID: 0 2024-09-13T11:26:41.886 INFO:tasks.ceph.osd.2.smithi162.stderr:2024-09-13T11:26:41.883+0000 7fe92b35a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26313) UID: 0 2024-09-13T11:26:41.927 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T11:26:41.941 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-13T11:26:41.941 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:26:41.957 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-13T11:26:41.958 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-09-13T11:26:41.955+0000 7f97b59fa640 -1 osd.6 218 handle_osd_map injecting map crc failure 2024-09-13T11:26:41.962 INFO:tasks.ceph.osd.3.smithi177.stderr:2024-09-13T11:26:41.961+0000 7f10a45d5640 -1 osd.3 218 handle_osd_map injecting map crc failure 2024-09-13T11:26:41.987 INFO:tasks.ceph.osd.11.smithi177.stderr:2024-09-13T11:26:41.985+0000 7f24792c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26580) UID: 0 2024-09-13T11:26:42.088 INFO:tasks.ceph.osd.11.smithi177.stderr:2024-09-13T11:26:42.085+0000 7f24792c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26580) UID: 0 2024-09-13T11:26:42.124 INFO:teuthology.orchestra.run.smithi001.stdout:osd.0: {} 2024-09-13T11:26:42.125 INFO:teuthology.orchestra.run.smithi001.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T11:26:42.128 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-13T11:26:42.128 INFO:teuthology.orchestra.run.smithi001.stdout: "ops": [], 2024-09-13T11:26:42.128 INFO:teuthology.orchestra.run.smithi001.stdout: "num_ops": 0 2024-09-13T11:26:42.128 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-13T11:26:42.143 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-13T11:26:42.152 INFO:teuthology.orchestra.run.smithi001.stdout:osd.1: {} 2024-09-13T11:26:42.152 INFO:teuthology.orchestra.run.smithi001.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T11:26:42.179 INFO:teuthology.orchestra.run.smithi001.stdout:osd.2: {} 2024-09-13T11:26:42.179 INFO:teuthology.orchestra.run.smithi001.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T11:26:42.189 INFO:tasks.ceph.osd.2.smithi162.stderr:2024-09-13T11:26:42.187+0000 7fe92b35a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26313) UID: 0 2024-09-13T11:26:42.202 INFO:teuthology.orchestra.run.smithi001.stdout:osd.3: {} 2024-09-13T11:26:42.203 INFO:teuthology.orchestra.run.smithi001.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T11:26:42.241 INFO:teuthology.orchestra.run.smithi001.stdout:osd.4: {} 2024-09-13T11:26:42.242 INFO:teuthology.orchestra.run.smithi001.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T11:26:42.281 INFO:teuthology.orchestra.run.smithi001.stdout:osd.5: {} 2024-09-13T11:26:42.281 INFO:teuthology.orchestra.run.smithi001.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T11:26:42.289 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-13T11:26:42.286+0000 7f6a68775640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26307) UID: 0 2024-09-13T11:26:42.293 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-13T11:26:42.293 INFO:teuthology.orchestra.run.smithi001.stdout: "ops": [], 2024-09-13T11:26:42.294 INFO:teuthology.orchestra.run.smithi001.stdout: "complaint_time": 30, 2024-09-13T11:26:42.294 INFO:teuthology.orchestra.run.smithi001.stdout: "num_blocked_ops": 0 2024-09-13T11:26:42.294 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-13T11:26:42.315 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-13T11:26:42.315 INFO:teuthology.orchestra.run.smithi001.stdout:osd.6: {} 2024-09-13T11:26:42.315 INFO:teuthology.orchestra.run.smithi001.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T11:26:42.335 INFO:teuthology.orchestra.run.smithi001.stdout:osd.7: {} 2024-09-13T11:26:42.335 INFO:teuthology.orchestra.run.smithi001.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T11:26:42.358 INFO:teuthology.orchestra.run.smithi001.stdout:osd.8: {} 2024-09-13T11:26:42.358 INFO:teuthology.orchestra.run.smithi001.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-13T11:26:42.384 INFO:teuthology.orchestra.run.smithi001.stdout:osd.9: {} 2024-09-13T11:26:42.384 INFO:teuthology.orchestra.run.smithi001.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-13T11:26:42.391 INFO:tasks.ceph.osd.3.smithi177.stderr:2024-09-13T11:26:42.389+0000 7f10adba4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26574) UID: 0 2024-09-13T11:26:42.410 INFO:teuthology.orchestra.run.smithi001.stdout:osd.10: {} 2024-09-13T11:26:42.410 INFO:teuthology.orchestra.run.smithi001.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-13T11:26:42.433 INFO:teuthology.orchestra.run.smithi001.stdout:osd.11: {} 2024-09-13T11:26:42.433 INFO:teuthology.orchestra.run.smithi001.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-13T11:26:42.457 INFO:teuthology.orchestra.run.smithi001.stdout:osd.12: {} 2024-09-13T11:26:42.458 INFO:teuthology.orchestra.run.smithi001.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-13T11:26:42.467 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-13T11:26:42.467 INFO:teuthology.orchestra.run.smithi001.stdout: "size": 20, 2024-09-13T11:26:42.467 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 600, 2024-09-13T11:26:42.467 INFO:teuthology.orchestra.run.smithi001.stdout: "ops": [ 2024-09-13T11:26:42.467 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:42.467 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDPGPushReply(3.1e 84/78 [PushReplyOp(3:79ee54b9:::benchmark_data_smithi177_31116_object11701:head)])", 2024-09-13T11:26:42.468 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:41.407335+0000", 2024-09-13T11:26:42.468 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 181.05798512999999, 2024-09-13T11:26:42.468 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.51622350400000006, 2024-09-13T11:26:42.468 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:26:42.468 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:26:42.468 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-13T11:26:42.468 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:26:42.468 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:42.468 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:26:42.468 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:41.407335+0000", 2024-09-13T11:26:42.468 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:42.468 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:42.468 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:42.469 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:26:42.469 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:41.407335+0000", 2024-09-13T11:26:42.469 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:42.469 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:42.469 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:42.469 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:26:42.469 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:41.407337+0000", 2024-09-13T11:26:42.469 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.505e-06 2024-09-13T11:26:42.469 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:42.469 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:42.469 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:26:42.469 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:41.407343+0000", 2024-09-13T11:26:42.470 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.9089999999999997e-06 2024-09-13T11:26:42.470 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:42.470 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:42.470 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:26:42.470 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:41.921977+0000", 2024-09-13T11:26:42.470 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.51463463499999995 2024-09-13T11:26:42.470 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:42.470 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:42.470 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:26:42.470 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:41.921988+0000", 2024-09-13T11:26:42.470 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.092e-05 2024-09-13T11:26:42.470 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:42.470 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:42.470 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:26:42.471 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:41.922088+0000", 2024-09-13T11:26:42.471 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.9528e-05 2024-09-13T11:26:42.471 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:42.471 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:42.471 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:26:42.471 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:41.923559+0000", 2024-09-13T11:26:42.471 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0014710070000000001 2024-09-13T11:26:42.471 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:42.471 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:42.471 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:42.471 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:42.471 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:42.472 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4665.0:60417 3.10 e85/84 3:092808b0:::benchmark_data_smithi177_31116_object60416:head v 85'3295, mlcod=85'3292)", 2024-09-13T11:26:42.472 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:42.174488+0000", 2024-09-13T11:26:42.472 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 180.29083257600001, 2024-09-13T11:26:42.472 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.387908896, 2024-09-13T11:26:42.472 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:26:42.472 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:26:42.472 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:42.472 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:26:42.472 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:42.472 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:26:42.472 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.174488+0000", 2024-09-13T11:26:42.472 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:42.472 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:42.472 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:42.473 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:26:42.473 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.174488+0000", 2024-09-13T11:26:42.473 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:42.473 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:42.473 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:42.473 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:26:42.473 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.174490+0000", 2024-09-13T11:26:42.473 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.9099999999999999e-06 2024-09-13T11:26:42.473 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:42.473 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:42.473 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:26:42.473 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.174527+0000", 2024-09-13T11:26:42.473 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.7543000000000003e-05 2024-09-13T11:26:42.474 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:42.474 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:42.474 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:26:42.474 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.561093+0000", 2024-09-13T11:26:42.474 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.38656578400000002 2024-09-13T11:26:42.474 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:42.474 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:42.474 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:26:42.474 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.561101+0000", 2024-09-13T11:26:42.474 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.6090000000000003e-06 2024-09-13T11:26:42.474 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:42.474 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:42.474 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:26:42.474 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.561143+0000", 2024-09-13T11:26:42.475 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.2435000000000003e-05 2024-09-13T11:26:42.475 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:42.475 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:42.475 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:26:42.475 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.561164+0000", 2024-09-13T11:26:42.475 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.1276000000000001e-05 2024-09-13T11:26:42.475 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:42.475 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:42.475 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:26:42.475 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.562360+0000", 2024-09-13T11:26:42.475 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00119596 2024-09-13T11:26:42.475 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:42.475 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:42.475 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:26:42.475 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.562397+0000", 2024-09-13T11:26:42.476 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.6378999999999998e-05 2024-09-13T11:26:42.476 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:42.476 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:42.476 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:42.476 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:42.476 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:42.476 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDPGPush(3.22 85/78 [PushOp(3:456e8765:::benchmark_data_smithi177_31116_object16212:head, version: 52'1006, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:456e8765:::benchmark_data_smithi177_31116_object16212:head@52'1006, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:26:42.476 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:42.673067+0000", 2024-09-13T11:26:42.476 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 179.792253434, 2024-09-13T11:26:42.476 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.27935889899999999, 2024-09-13T11:26:42.476 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:26:42.476 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:26:42.476 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:26:42.477 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:26:42.477 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:42.477 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:26:42.477 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.673067+0000", 2024-09-13T11:26:42.477 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:42.477 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:42.477 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:42.477 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:26:42.477 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.673067+0000", 2024-09-13T11:26:42.477 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:42.477 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:42.477 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:42.477 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:26:42.478 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.673068+0000", 2024-09-13T11:26:42.478 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.9699999999999994e-07 2024-09-13T11:26:42.478 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:42.478 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:42.478 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:26:42.478 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.673090+0000", 2024-09-13T11:26:42.478 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.2416999999999999e-05 2024-09-13T11:26:42.478 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:42.478 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:42.478 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:26:42.478 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.951363+0000", 2024-09-13T11:26:42.478 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.27827291199999998 2024-09-13T11:26:42.478 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:42.478 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:42.479 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:26:42.479 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.951386+0000", 2024-09-13T11:26:42.479 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.2226999999999999e-05 2024-09-13T11:26:42.479 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:42.479 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:42.479 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:26:42.479 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.951581+0000", 2024-09-13T11:26:42.479 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000195713 2024-09-13T11:26:42.479 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:42.479 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:42.479 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:26:42.479 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.951605+0000", 2024-09-13T11:26:42.479 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.3767000000000001e-05 2024-09-13T11:26:42.479 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:42.480 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:42.480 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:26:42.480 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.952426+0000", 2024-09-13T11:26:42.480 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00082086599999999996 2024-09-13T11:26:42.480 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:42.480 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:42.480 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:42.480 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:42.480 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:42.480 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "replica_scrub(pg: 3.20,from:0'0,to:89'3254,epoch:89/77,start:3:04000000::::0,end:3:045ba805:::benchmark_data_smithi177_31116_object43251:0,chunky:1,deep:0,version:9,allow_preemption:1,priority=5)", 2024-09-13T11:26:42.480 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:46.641364+0000", 2024-09-13T11:26:42.480 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 175.823956753, 2024-09-13T11:26:42.480 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.27141393000000003, 2024-09-13T11:26:42.480 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:26:42.481 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:26:42.481 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:26:42.481 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:26:42.481 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:42.481 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:26:42.481 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.641364+0000", 2024-09-13T11:26:42.481 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:42.481 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:42.481 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:42.481 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:26:42.481 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.641364+0000", 2024-09-13T11:26:42.481 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:42.481 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:42.482 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:42.482 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:26:42.482 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.641365+0000", 2024-09-13T11:26:42.482 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.266e-06 2024-09-13T11:26:42.482 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:42.482 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:42.482 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:26:42.482 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.641370+0000", 2024-09-13T11:26:42.482 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.8539999999999996e-06 2024-09-13T11:26:42.482 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:42.482 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:42.482 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:26:42.482 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.912491+0000", 2024-09-13T11:26:42.483 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.27112103100000001 2024-09-13T11:26:42.483 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:42.483 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:42.483 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:26:42.483 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.912520+0000", 2024-09-13T11:26:42.483 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.9379000000000001e-05 2024-09-13T11:26:42.483 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:42.483 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:42.483 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:26:42.483 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.912679+0000", 2024-09-13T11:26:42.483 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00015915800000000001 2024-09-13T11:26:42.483 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:42.483 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:42.483 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:26:42.483 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.912714+0000", 2024-09-13T11:26:42.484 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.4687999999999997e-05 2024-09-13T11:26:42.484 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:42.484 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:42.484 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:26:42.484 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.912778+0000", 2024-09-13T11:26:42.484 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.3553999999999995e-05 2024-09-13T11:26:42.484 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:42.484 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:42.484 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:42.484 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:42.484 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:42.484 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDPGPush(3.18 89/70 [PushOp(3:1ac6c361:::benchmark_data_smithi177_31116_object44972:head, version: 68'2767, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:1ac6c361:::benchmark_data_smithi177_31116_object44972:head@68'2767, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:26:42.485 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:46.671018+0000", 2024-09-13T11:26:42.485 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 175.79430262100001, 2024-09-13T11:26:42.485 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.271282931, 2024-09-13T11:26:42.485 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:26:42.485 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:26:42.485 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:26:42.485 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:26:42.485 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:42.485 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:26:42.485 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.671018+0000", 2024-09-13T11:26:42.485 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:42.485 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:42.485 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:42.486 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:26:42.486 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.671018+0000", 2024-09-13T11:26:42.486 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:42.486 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:42.486 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:42.486 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:26:42.486 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.671020+0000", 2024-09-13T11:26:42.486 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.018e-06 2024-09-13T11:26:42.486 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:42.486 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:42.486 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:26:42.486 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.671097+0000", 2024-09-13T11:26:42.487 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.7218999999999994e-05 2024-09-13T11:26:42.487 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:42.487 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:42.487 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:26:42.487 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.939773+0000", 2024-09-13T11:26:42.487 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.26867550200000001 2024-09-13T11:26:42.487 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:42.487 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:42.487 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:26:42.487 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.939791+0000", 2024-09-13T11:26:42.487 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.8185999999999998e-05 2024-09-13T11:26:42.487 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:42.487 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:42.487 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:26:42.488 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.939926+0000", 2024-09-13T11:26:42.488 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000135291 2024-09-13T11:26:42.488 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:42.488 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:42.488 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:26:42.488 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.939938+0000", 2024-09-13T11:26:42.488 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2393e-05 2024-09-13T11:26:42.488 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:42.488 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:42.488 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:26:42.488 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.942301+0000", 2024-09-13T11:26:42.488 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.002362322 2024-09-13T11:26:42.488 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:42.488 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:42.489 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:42.489 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:42.489 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:42.489 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "replica_scrub(pg: 3.20,from:0'0,to:90'3264,epoch:90/77,start:3:0650aac8:::benchmark_data_smithi177_31116_object1734:0,end:3:06af01e3:::benchmark_data_smithi177_31116_object31572:0,chunky:1,deep:0,version:9,allow_preemption:1,priority=5)", 2024-09-13T11:26:42.489 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:47.380615+0000", 2024-09-13T11:26:42.489 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 175.08470519799999, 2024-09-13T11:26:42.489 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.51951398400000004, 2024-09-13T11:26:42.489 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:26:42.489 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:26:42.489 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:26:42.489 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:26:42.489 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:42.489 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:26:42.490 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:47.380615+0000", 2024-09-13T11:26:42.490 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:42.490 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:42.490 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:42.490 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:26:42.490 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:47.380615+0000", 2024-09-13T11:26:42.490 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:42.490 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:42.490 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:42.490 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:26:42.490 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:47.380617+0000", 2024-09-13T11:26:42.490 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.671e-06 2024-09-13T11:26:42.491 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:42.491 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:42.491 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:26:42.491 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:47.380623+0000", 2024-09-13T11:26:42.491 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.1190000000000002e-06 2024-09-13T11:26:42.491 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:42.491 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:42.491 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:26:42.491 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:47.900050+0000", 2024-09-13T11:26:42.491 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.51942737400000005 2024-09-13T11:26:42.491 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:42.491 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:42.491 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:26:42.492 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:47.900057+0000", 2024-09-13T11:26:42.492 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.4359999999999999e-06 2024-09-13T11:26:42.492 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:42.492 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:42.492 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:26:42.492 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:47.900101+0000", 2024-09-13T11:26:42.492 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.4675000000000002e-05 2024-09-13T11:26:42.492 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:42.492 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:42.492 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:26:42.492 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:47.900110+0000", 2024-09-13T11:26:42.492 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.0569999999999993e-06 2024-09-13T11:26:42.493 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:42.493 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:42.493 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:26:42.493 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:47.900129+0000", 2024-09-13T11:26:42.493 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.9652000000000001e-05 2024-09-13T11:26:42.493 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:42.493 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:42.493 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:42.493 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:42.493 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:42.493 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDPGPush(3.21 91/78 [PushOp(3:8492321f:::benchmark_data_smithi177_31116_object13767:head, version: 50'840, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:8492321f:::benchmark_data_smithi177_31116_object13767:head@50'840, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:26:42.493 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:48.319251+0000", 2024-09-13T11:26:42.494 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 174.14606966900001, 2024-09-13T11:26:42.494 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.60903802200000001, 2024-09-13T11:26:42.494 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:26:42.494 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:26:42.494 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:26:42.494 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:26:42.494 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:42.494 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:26:42.494 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:48.319251+0000", 2024-09-13T11:26:42.494 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:42.494 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:42.494 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:42.494 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:26:42.494 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:48.319251+0000", 2024-09-13T11:26:42.495 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:42.495 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:42.495 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:42.495 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:26:42.495 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:48.319252+0000", 2024-09-13T11:26:42.495 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2640000000000001e-06 2024-09-13T11:26:42.495 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:42.495 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:42.495 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:26:42.495 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:48.319363+0000", 2024-09-13T11:26:42.495 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000110559 2024-09-13T11:26:42.495 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:42.495 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:42.496 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:26:42.496 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:48.927115+0000", 2024-09-13T11:26:42.496 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.60775277800000005 2024-09-13T11:26:42.496 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:42.496 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:42.496 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:26:42.496 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:48.927144+0000", 2024-09-13T11:26:42.496 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.8861e-05 2024-09-13T11:26:42.496 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:42.496 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:42.496 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:26:42.496 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:48.927318+0000", 2024-09-13T11:26:42.496 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00017399000000000001 2024-09-13T11:26:42.497 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:42.497 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:42.497 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:26:42.497 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:48.927340+0000", 2024-09-13T11:26:42.497 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.1433999999999998e-05 2024-09-13T11:26:42.497 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:42.497 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:42.497 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:26:42.497 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:48.928289+0000", 2024-09-13T11:26:42.497 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00094913599999999995 2024-09-13T11:26:42.497 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:42.497 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:42.497 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:42.498 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:42.498 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:42.498 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:83239 3.e 3:74547f17:::benchmark_data_smithi177_31116_object83238:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106)", 2024-09-13T11:26:42.498 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:24:04.580085+0000", 2024-09-13T11:26:42.498 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 157.885235588, 2024-09-13T11:26:42.498 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.33785762600000002, 2024-09-13T11:26:42.498 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:26:42.498 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:26:42.498 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:42.498 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:26:42.498 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:26:42.498 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:42.498 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 83239 2024-09-13T11:26:42.498 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:42.499 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:26:42.499 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:42.499 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:26:42.499 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:04.580085+0000", 2024-09-13T11:26:42.499 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:42.499 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:42.499 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:42.499 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:26:42.499 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:04.580085+0000", 2024-09-13T11:26:42.499 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:42.499 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:42.499 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:42.500 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:26:42.500 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:04.580089+0000", 2024-09-13T11:26:42.500 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.053e-06 2024-09-13T11:26:42.500 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:42.500 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:42.500 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:26:42.500 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:04.580166+0000", 2024-09-13T11:26:42.500 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.7102999999999997e-05 2024-09-13T11:26:42.500 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:42.500 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:42.500 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:26:42.500 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:04.580196+0000", 2024-09-13T11:26:42.501 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.0196e-05 2024-09-13T11:26:42.501 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:42.501 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:42.501 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:26:42.501 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:04.580212+0000", 2024-09-13T11:26:42.501 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.535e-05 2024-09-13T11:26:42.501 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:42.501 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:42.501 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:26:42.501 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:04.580336+0000", 2024-09-13T11:26:42.501 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00012431899999999999 2024-09-13T11:26:42.501 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:42.501 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:42.502 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:26:42.502 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:04.580575+0000", 2024-09-13T11:26:42.502 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00023924100000000001 2024-09-13T11:26:42.502 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:42.502 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:42.502 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 1", 2024-09-13T11:26:42.502 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:04.580798+0000", 2024-09-13T11:26:42.502 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00022301499999999999 2024-09-13T11:26:42.502 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:42.502 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:42.502 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:26:42.502 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:04.583404+0000", 2024-09-13T11:26:42.503 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0026056120000000002 2024-09-13T11:26:42.503 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:42.503 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:42.503 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:42.503 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:04.917768+0000", 2024-09-13T11:26:42.503 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.33436387499999998 2024-09-13T11:26:42.503 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:42.503 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:42.503 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:26:42.503 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:04.917859+0000", 2024-09-13T11:26:42.503 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.1732999999999997e-05 2024-09-13T11:26:42.503 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:42.504 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:42.504 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:26:42.504 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:04.917942+0000", 2024-09-13T11:26:42.504 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.3128999999999996e-05 2024-09-13T11:26:42.504 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:42.504 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:42.504 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:42.504 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:42.504 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:42.504 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:110271 3.2d 3.3c2681ad (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e119)", 2024-09-13T11:26:42.504 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:24:25.403006+0000", 2024-09-13T11:26:42.504 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 137.062314104, 2024-09-13T11:26:42.504 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.68768755999999998, 2024-09-13T11:26:42.504 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:26:42.505 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:26:42.505 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-13T11:26:42.505 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:26:42.505 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:26:42.505 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:42.505 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 110271 2024-09-13T11:26:42.505 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:42.505 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:26:42.505 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:42.505 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:26:42.505 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:25.403006+0000", 2024-09-13T11:26:42.505 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:42.505 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:42.505 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:42.505 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:26:42.506 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:25.403006+0000", 2024-09-13T11:26:42.506 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:42.506 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:42.506 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:42.506 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:26:42.506 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:25.403011+0000", 2024-09-13T11:26:42.506 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.5199999999999999e-06 2024-09-13T11:26:42.506 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:42.506 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:42.506 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:26:42.506 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:25.403065+0000", 2024-09-13T11:26:42.506 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.4478999999999998e-05 2024-09-13T11:26:42.507 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:42.507 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:42.507 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:26:42.507 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:25.403097+0000", 2024-09-13T11:26:42.507 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.1636e-05 2024-09-13T11:26:42.507 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:42.507 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:42.507 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:26:42.507 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:25.403121+0000", 2024-09-13T11:26:42.507 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.3924e-05 2024-09-13T11:26:42.507 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:42.507 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:42.507 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:26:42.508 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:25.403216+0000", 2024-09-13T11:26:42.508 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.5557999999999995e-05 2024-09-13T11:26:42.508 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:42.508 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:42.508 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for peered", 2024-09-13T11:26:42.508 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:25.403234+0000", 2024-09-13T11:26:42.508 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.7104000000000001e-05 2024-09-13T11:26:42.508 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:42.508 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:42.508 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:26:42.508 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:26.090684+0000", 2024-09-13T11:26:42.508 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.68745000700000003 2024-09-13T11:26:42.509 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:42.509 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:42.509 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:26:42.509 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:26.090694+0000", 2024-09-13T11:26:42.509 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0332e-05 2024-09-13T11:26:42.509 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:42.509 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:42.509 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:42.509 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:42.509 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:42.509 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4665.0:129901 3.20 e134/77 3:067f5dfc:::benchmark_data_smithi177_31116_object129900:head v 134'4294, mlcod=134'4291)", 2024-09-13T11:26:42.509 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:24:47.145530+0000", 2024-09-13T11:26:42.509 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 115.319790029, 2024-09-13T11:26:42.509 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.333688349, 2024-09-13T11:26:42.510 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:26:42.510 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:26:42.510 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:42.510 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:26:42.510 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:42.510 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:26:42.510 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:47.145530+0000", 2024-09-13T11:26:42.510 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:42.510 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:42.510 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:42.510 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:26:42.510 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:47.145530+0000", 2024-09-13T11:26:42.510 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:42.511 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:42.511 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:42.511 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:26:42.511 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:47.145532+0000", 2024-09-13T11:26:42.511 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.7999999999999999e-06 2024-09-13T11:26:42.511 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:42.511 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:42.511 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:26:42.511 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:47.145557+0000", 2024-09-13T11:26:42.511 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.4678999999999999e-05 2024-09-13T11:26:42.511 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:42.511 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:42.512 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:26:42.512 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:47.477539+0000", 2024-09-13T11:26:42.512 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.33198164099999999 2024-09-13T11:26:42.512 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:42.512 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:42.512 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:26:42.512 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:47.477544+0000", 2024-09-13T11:26:42.512 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.9059999999999999e-06 2024-09-13T11:26:42.512 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:42.512 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:42.512 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:26:42.512 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:47.477597+0000", 2024-09-13T11:26:42.512 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.295e-05 2024-09-13T11:26:42.513 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:42.513 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:42.513 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:26:42.513 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:47.477619+0000", 2024-09-13T11:26:42.513 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.1835999999999999e-05 2024-09-13T11:26:42.513 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:42.513 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:42.513 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:26:42.513 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:47.479175+0000", 2024-09-13T11:26:42.513 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0015560470000000001 2024-09-13T11:26:42.513 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:42.513 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:42.513 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:26:42.514 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:47.479219+0000", 2024-09-13T11:26:42.514 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.3489999999999999e-05 2024-09-13T11:26:42.514 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:42.514 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:42.514 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:42.514 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:42.514 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:42.514 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:140840 3.21 3:87ae3db6:::benchmark_data_smithi177_31116_object537:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e144)", 2024-09-13T11:26:42.514 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:24:58.810939+0000", 2024-09-13T11:26:42.514 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 103.65438091599999, 2024-09-13T11:26:42.514 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.25628309300000002, 2024-09-13T11:26:42.514 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:26:42.514 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:26:42.514 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:42.515 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:26:42.515 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:26:42.515 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:42.515 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 140840 2024-09-13T11:26:42.515 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:42.515 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:26:42.515 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:42.515 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:26:42.515 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.810939+0000", 2024-09-13T11:26:42.515 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:42.515 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:42.516 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:42.516 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:26:42.516 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.810939+0000", 2024-09-13T11:26:42.516 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:42.516 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:42.516 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:42.516 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:26:42.516 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.810942+0000", 2024-09-13T11:26:42.516 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.6259999999999999e-06 2024-09-13T11:26:42.516 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:42.516 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:42.516 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:26:42.517 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.810950+0000", 2024-09-13T11:26:42.517 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.9500000000000001e-06 2024-09-13T11:26:42.517 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:42.517 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:42.517 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:26:42.517 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.810970+0000", 2024-09-13T11:26:42.517 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.9539999999999999e-05 2024-09-13T11:26:42.517 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:42.517 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:42.517 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:26:42.517 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.810978+0000", 2024-09-13T11:26:42.517 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.8680000000000005e-06 2024-09-13T11:26:42.518 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:42.518 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:42.518 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:26:42.518 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.811060+0000", 2024-09-13T11:26:42.518 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.1068000000000004e-05 2024-09-13T11:26:42.518 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:42.518 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:42.518 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:26:42.518 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.811174+0000", 2024-09-13T11:26:42.518 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000114385 2024-09-13T11:26:42.518 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:42.518 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:42.518 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 8", 2024-09-13T11:26:42.518 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.811255+0000", 2024-09-13T11:26:42.519 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.1298000000000007e-05 2024-09-13T11:26:42.519 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:42.519 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:42.519 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:26:42.519 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.812869+0000", 2024-09-13T11:26:42.519 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.001614234 2024-09-13T11:26:42.519 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:42.519 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:42.519 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:42.519 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:59.067155+0000", 2024-09-13T11:26:42.519 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.25428578299999999 2024-09-13T11:26:42.519 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:42.519 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:42.519 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:26:42.519 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:59.067192+0000", 2024-09-13T11:26:42.520 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.659e-05 2024-09-13T11:26:42.520 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:42.520 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:42.520 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:26:42.520 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:59.067223+0000", 2024-09-13T11:26:42.520 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.0750999999999997e-05 2024-09-13T11:26:42.520 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:42.520 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:42.520 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:42.520 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:42.520 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:42.520 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4665.0:191286 3.18 e175/91 3:1ffc0b26:::benchmark_data_smithi177_31116_object50983:head v 175'7339, mlcod=175'7337)", 2024-09-13T11:26:42.520 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:25:39.153114+0000", 2024-09-13T11:26:42.520 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 63.312206304, 2024-09-13T11:26:42.520 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.23200507400000001, 2024-09-13T11:26:42.521 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:26:42.521 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:26:42.521 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:42.521 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:26:42.521 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:42.521 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:26:42.521 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:39.153114+0000", 2024-09-13T11:26:42.521 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:42.521 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:42.521 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:42.521 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:26:42.521 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:39.153114+0000", 2024-09-13T11:26:42.521 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:42.521 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:42.522 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:42.522 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:26:42.522 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:39.153117+0000", 2024-09-13T11:26:42.522 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.9170000000000002e-06 2024-09-13T11:26:42.522 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:42.522 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:42.522 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:26:42.522 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:39.153130+0000", 2024-09-13T11:26:42.522 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2493e-05 2024-09-13T11:26:42.522 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:42.522 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:42.522 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:26:42.522 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:39.381951+0000", 2024-09-13T11:26:42.523 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.22882103000000001 2024-09-13T11:26:42.523 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:42.523 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:42.523 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:26:42.523 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:39.381956+0000", 2024-09-13T11:26:42.523 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.1359999999999996e-06 2024-09-13T11:26:42.523 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:42.523 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:42.523 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:26:42.523 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:39.382030+0000", 2024-09-13T11:26:42.523 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.4270999999999994e-05 2024-09-13T11:26:42.523 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:42.524 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:42.524 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:26:42.524 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:39.382054+0000", 2024-09-13T11:26:42.524 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.4142999999999999e-05 2024-09-13T11:26:42.524 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:42.524 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:42.524 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:26:42.524 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:39.385086+0000", 2024-09-13T11:26:42.524 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00303163 2024-09-13T11:26:42.524 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:42.524 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:42.524 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:26:42.525 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:39.385119+0000", 2024-09-13T11:26:42.525 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.3454000000000002e-05 2024-09-13T11:26:42.525 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:42.525 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:42.525 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:42.525 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:42.525 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:42.525 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4665.0:196438 3.20 e178/77 3:0573f1a4:::benchmark_data_smithi177_31116_object56135:head v 178'5356, mlcod=178'5353)", 2024-09-13T11:26:42.525 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:25:43.522272+0000", 2024-09-13T11:26:42.525 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 58.943048214000001, 2024-09-13T11:26:42.525 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.23723141, 2024-09-13T11:26:42.525 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:26:42.526 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:26:42.526 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:42.526 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:26:42.526 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:42.526 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:26:42.526 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:43.522272+0000", 2024-09-13T11:26:42.526 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:42.526 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:42.526 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:42.526 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:26:42.527 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:43.522272+0000", 2024-09-13T11:26:42.527 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:42.527 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:42.527 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:42.527 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:26:42.527 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:43.522274+0000", 2024-09-13T11:26:42.527 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.573e-06 2024-09-13T11:26:42.527 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:42.527 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:42.527 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:26:42.528 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:43.522281+0000", 2024-09-13T11:26:42.528 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.5490000000000001e-06 2024-09-13T11:26:42.528 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:42.528 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:42.528 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:26:42.528 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:43.756836+0000", 2024-09-13T11:26:42.528 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.23455496000000001 2024-09-13T11:26:42.528 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:42.528 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:42.528 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:26:42.528 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:43.756843+0000", 2024-09-13T11:26:42.528 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.4620000000000001e-06 2024-09-13T11:26:42.528 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:42.528 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:42.529 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:26:42.529 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:43.756914+0000", 2024-09-13T11:26:42.529 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.1276e-05 2024-09-13T11:26:42.529 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:42.529 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:42.529 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:26:42.529 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:43.756938+0000", 2024-09-13T11:26:42.529 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.4281999999999999e-05 2024-09-13T11:26:42.529 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:42.529 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:42.529 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:26:42.529 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:43.759408+0000", 2024-09-13T11:26:42.529 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0024699449999999999 2024-09-13T11:26:42.529 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:42.530 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:42.530 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:26:42.530 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:43.759504+0000", 2024-09-13T11:26:42.530 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.5363000000000002e-05 2024-09-13T11:26:42.530 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:42.530 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:42.530 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:42.530 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:42.530 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:42.530 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:201234 3.21 3:85402570:::benchmark_data_smithi177_31116_object60931:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e181)", 2024-09-13T11:26:42.530 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:25:47.835618+0000", 2024-09-13T11:26:42.530 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 54.629702750999996, 2024-09-13T11:26:42.530 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.26153554899999998, 2024-09-13T11:26:42.530 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:26:42.531 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:26:42.531 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:42.531 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:26:42.531 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:26:42.531 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:42.531 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 201234 2024-09-13T11:26:42.531 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:42.531 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:26:42.531 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:42.531 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:26:42.531 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:47.835618+0000", 2024-09-13T11:26:42.531 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:42.531 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:42.531 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:42.532 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:26:42.532 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:47.835618+0000", 2024-09-13T11:26:42.532 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:42.532 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:42.532 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:42.532 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:26:42.532 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:47.835619+0000", 2024-09-13T11:26:42.532 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.638e-06 2024-09-13T11:26:42.532 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:42.532 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:42.532 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:26:42.532 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:47.835627+0000", 2024-09-13T11:26:42.532 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.0150000000000003e-06 2024-09-13T11:26:42.532 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:42.533 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:42.533 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:26:42.533 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:47.835639+0000", 2024-09-13T11:26:42.533 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.1939e-05 2024-09-13T11:26:42.533 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:42.533 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:42.533 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:26:42.533 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:47.835644+0000", 2024-09-13T11:26:42.533 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.006e-06 2024-09-13T11:26:42.533 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:42.533 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:42.533 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:26:42.533 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:47.835696+0000", 2024-09-13T11:26:42.533 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.1668000000000001e-05 2024-09-13T11:26:42.534 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:42.534 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:42.534 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:26:42.534 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:47.835775+0000", 2024-09-13T11:26:42.534 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.8737000000000002e-05 2024-09-13T11:26:42.534 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:42.534 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:42.534 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 8", 2024-09-13T11:26:42.534 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:47.835857+0000", 2024-09-13T11:26:42.534 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.2508000000000004e-05 2024-09-13T11:26:42.534 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:42.534 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:42.534 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:26:42.534 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:47.837366+0000", 2024-09-13T11:26:42.534 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.001509292 2024-09-13T11:26:42.535 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:42.535 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:42.535 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:42.535 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:48.097051+0000", 2024-09-13T11:26:42.535 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.25968466699999998 2024-09-13T11:26:42.535 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:42.535 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:42.535 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:26:42.535 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:48.097109+0000", 2024-09-13T11:26:42.535 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.7778000000000001e-05 2024-09-13T11:26:42.535 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:42.535 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:42.535 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:26:42.535 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:48.097153+0000", 2024-09-13T11:26:42.536 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.4301000000000002e-05 2024-09-13T11:26:42.536 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:42.536 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:42.536 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:42.536 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:42.536 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:42.536 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:217082 3.22 3:449e4b87:::benchmark_data_smithi177_31116_object76779:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e187)", 2024-09-13T11:26:42.536 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:25:59.883418+0000", 2024-09-13T11:26:42.536 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 42.581902753999998, 2024-09-13T11:26:42.536 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.25557793200000001, 2024-09-13T11:26:42.536 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:26:42.536 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:26:42.536 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:42.536 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:26:42.536 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:26:42.537 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:42.537 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 217082 2024-09-13T11:26:42.537 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:42.537 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:26:42.537 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:42.537 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:26:42.537 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:59.883418+0000", 2024-09-13T11:26:42.537 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:42.537 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:42.537 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:42.537 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:26:42.537 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:59.883418+0000", 2024-09-13T11:26:42.537 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:42.537 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:42.538 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:42.538 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:26:42.538 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:59.883419+0000", 2024-09-13T11:26:42.538 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.235e-06 2024-09-13T11:26:42.538 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:42.538 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:42.538 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:26:42.538 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:59.883423+0000", 2024-09-13T11:26:42.538 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.7280000000000001e-06 2024-09-13T11:26:42.538 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:42.538 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:42.538 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:26:42.538 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:59.883433+0000", 2024-09-13T11:26:42.538 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0498e-05 2024-09-13T11:26:42.539 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:42.539 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:42.539 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:26:42.539 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:59.883439+0000", 2024-09-13T11:26:42.539 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.5890000000000002e-06 2024-09-13T11:26:42.539 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:42.539 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:42.539 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:26:42.539 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:59.883483+0000", 2024-09-13T11:26:42.539 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.4710999999999999e-05 2024-09-13T11:26:42.539 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:42.539 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:42.539 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:26:42.539 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:59.883558+0000", 2024-09-13T11:26:42.540 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.4308999999999996e-05 2024-09-13T11:26:42.540 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:42.540 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:42.540 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 5", 2024-09-13T11:26:42.540 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:59.883615+0000", 2024-09-13T11:26:42.540 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.7745999999999999e-05 2024-09-13T11:26:42.540 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:42.540 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:42.540 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:26:42.540 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:59.885044+0000", 2024-09-13T11:26:42.540 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0014282940000000001 2024-09-13T11:26:42.540 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:42.540 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:42.540 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:42.541 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:00.138864+0000", 2024-09-13T11:26:42.541 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.253819923 2024-09-13T11:26:42.541 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:42.541 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:42.541 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:26:42.541 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:00.138936+0000", 2024-09-13T11:26:42.541 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.2268999999999995e-05 2024-09-13T11:26:42.541 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:42.541 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:42.541 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:26:42.541 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:00.138996+0000", 2024-09-13T11:26:42.541 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.9630000000000003e-05 2024-09-13T11:26:42.541 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:42.541 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:42.541 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:42.542 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:42.542 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:42.542 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:221463 3.15 3:aaa4e83d:::benchmark_data_smithi177_31116_object81160:head [delete] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e193)", 2024-09-13T11:26:42.542 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:26:06.057961+0000", 2024-09-13T11:26:42.542 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 36.407359239000002, 2024-09-13T11:26:42.542 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.24029746699999999, 2024-09-13T11:26:42.542 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:26:42.542 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:26:42.542 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:42.542 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:26:42.542 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:26:42.542 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:42.542 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 221463 2024-09-13T11:26:42.542 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:42.543 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:26:42.543 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:42.543 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:26:42.543 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:06.057961+0000", 2024-09-13T11:26:42.543 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:42.543 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:42.543 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:42.543 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:26:42.543 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:06.057961+0000", 2024-09-13T11:26:42.543 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:42.543 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:42.543 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:42.543 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:26:42.543 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:06.057963+0000", 2024-09-13T11:26:42.544 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.61e-06 2024-09-13T11:26:42.544 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:42.544 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:42.544 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:26:42.544 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:06.057970+0000", 2024-09-13T11:26:42.544 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.9999999999999999e-06 2024-09-13T11:26:42.544 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:42.544 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:42.544 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:26:42.544 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:06.058003+0000", 2024-09-13T11:26:42.544 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.2733e-05 2024-09-13T11:26:42.544 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:42.544 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:42.544 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:26:42.545 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:06.058013+0000", 2024-09-13T11:26:42.545 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0349999999999999e-05 2024-09-13T11:26:42.545 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:42.545 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:42.545 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:26:42.545 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:06.058333+0000", 2024-09-13T11:26:42.545 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000319947 2024-09-13T11:26:42.545 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:42.545 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:42.545 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for peered", 2024-09-13T11:26:42.545 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:06.058346+0000", 2024-09-13T11:26:42.545 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.3657e-05 2024-09-13T11:26:42.545 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:42.545 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:42.546 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:26:42.546 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:06.277695+0000", 2024-09-13T11:26:42.546 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.219348129 2024-09-13T11:26:42.546 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:42.546 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:42.546 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:26:42.546 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:06.277819+0000", 2024-09-13T11:26:42.546 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00012432599999999999 2024-09-13T11:26:42.546 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:42.546 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:42.546 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 3,14", 2024-09-13T11:26:42.546 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:06.277928+0000", 2024-09-13T11:26:42.546 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00010860499999999999 2024-09-13T11:26:42.546 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:42.546 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:42.547 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:42.547 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:06.286021+0000", 2024-09-13T11:26:42.547 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0080932020000000007 2024-09-13T11:26:42.547 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:42.547 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:42.547 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:26:42.547 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:06.291903+0000", 2024-09-13T11:26:42.547 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0058824630000000001 2024-09-13T11:26:42.547 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:42.547 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:42.547 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:42.547 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:06.298142+0000", 2024-09-13T11:26:42.547 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.006239301 2024-09-13T11:26:42.547 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:42.548 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:42.548 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:26:42.548 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:06.298189+0000", 2024-09-13T11:26:42.548 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.6115999999999997e-05 2024-09-13T11:26:42.548 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:42.548 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:42.548 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:26:42.548 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:06.298259+0000", 2024-09-13T11:26:42.548 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.0028000000000001e-05 2024-09-13T11:26:42.548 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:42.548 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:42.548 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:42.548 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:42.548 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:42.549 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:221475 3.15 3:abeb12df:::benchmark_data_smithi177_31116_object81172:head [delete] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e193)", 2024-09-13T11:26:42.549 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:26:06.058032+0000", 2024-09-13T11:26:42.549 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 36.407288205999997, 2024-09-13T11:26:42.549 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.24037931400000001, 2024-09-13T11:26:42.549 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:26:42.549 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:26:42.549 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:42.549 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:26:42.549 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:26:42.549 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:42.549 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 221475 2024-09-13T11:26:42.549 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:42.549 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:26:42.549 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:42.549 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:26:42.550 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:06.058032+0000", 2024-09-13T11:26:42.550 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:42.550 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:42.550 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:42.550 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:26:42.550 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:06.058032+0000", 2024-09-13T11:26:42.550 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:42.550 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:42.550 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:42.550 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:26:42.550 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:06.058033+0000", 2024-09-13T11:26:42.550 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.136e-06 2024-09-13T11:26:42.550 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:42.550 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:42.551 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:26:42.551 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:06.058038+0000", 2024-09-13T11:26:42.551 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.9169999999999997e-06 2024-09-13T11:26:42.551 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:42.551 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:42.551 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:26:42.551 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:06.058048+0000", 2024-09-13T11:26:42.551 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.3170000000000008e-06 2024-09-13T11:26:42.551 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:42.551 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:42.551 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:26:42.551 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:06.058059+0000", 2024-09-13T11:26:42.551 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.1479e-05 2024-09-13T11:26:42.551 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:42.552 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:42.552 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:26:42.552 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:06.058444+0000", 2024-09-13T11:26:42.552 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00038485700000000001 2024-09-13T11:26:42.552 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:42.552 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:42.552 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for peered", 2024-09-13T11:26:42.552 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:06.058471+0000", 2024-09-13T11:26:42.552 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.7172999999999999e-05 2024-09-13T11:26:42.552 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:42.552 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:42.552 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:26:42.552 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:06.282046+0000", 2024-09-13T11:26:42.552 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.22357485099999999 2024-09-13T11:26:42.553 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:42.553 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:42.553 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:26:42.553 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:06.282204+0000", 2024-09-13T11:26:42.553 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000158414 2024-09-13T11:26:42.553 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:42.553 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:42.553 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 3,14", 2024-09-13T11:26:42.553 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:06.282377+0000", 2024-09-13T11:26:42.553 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00017289900000000001 2024-09-13T11:26:42.553 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:42.553 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:42.553 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:42.553 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:06.295990+0000", 2024-09-13T11:26:42.554 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.013612406000000001 2024-09-13T11:26:42.554 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:42.554 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:42.554 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:26:42.554 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:06.296380+0000", 2024-09-13T11:26:42.554 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00039020599999999998 2024-09-13T11:26:42.554 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:42.554 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:42.554 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:42.554 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:06.298303+0000", 2024-09-13T11:26:42.554 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.001923258 2024-09-13T11:26:42.554 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:42.554 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:42.554 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:26:42.555 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:06.298343+0000", 2024-09-13T11:26:42.555 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.9783999999999997e-05 2024-09-13T11:26:42.555 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:42.555 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:42.555 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:26:42.555 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:06.298412+0000", 2024-09-13T11:26:42.555 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.8616999999999997e-05 2024-09-13T11:26:42.555 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:42.555 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:42.555 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:42.555 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:42.555 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:42.555 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:221503 3.15 3:ab531595:::benchmark_data_smithi177_31116_object81200:head [delete] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e193)", 2024-09-13T11:26:42.555 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:26:06.058069+0000", 2024-09-13T11:26:42.555 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 36.407251697, 2024-09-13T11:26:42.556 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.24966487400000001, 2024-09-13T11:26:42.556 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:26:42.556 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:26:42.556 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:42.556 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:26:42.556 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:26:42.556 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:42.556 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 221503 2024-09-13T11:26:42.556 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:42.556 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:26:42.556 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:42.556 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:26:42.556 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:06.058069+0000", 2024-09-13T11:26:42.557 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:42.557 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:42.557 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:42.557 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:26:42.557 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:06.058069+0000", 2024-09-13T11:26:42.557 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:42.557 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:42.557 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:42.557 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:26:42.557 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:06.058070+0000", 2024-09-13T11:26:42.557 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.7000000000000003e-07 2024-09-13T11:26:42.557 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:42.557 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:42.557 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:26:42.558 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:06.058074+0000", 2024-09-13T11:26:42.558 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.5070000000000002e-06 2024-09-13T11:26:42.558 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:42.558 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:42.558 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:26:42.558 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:06.058089+0000", 2024-09-13T11:26:42.558 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.4593000000000001e-05 2024-09-13T11:26:42.558 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:42.558 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:42.558 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:26:42.558 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:06.058102+0000", 2024-09-13T11:26:42.558 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.3349999999999999e-05 2024-09-13T11:26:42.558 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:42.558 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:42.558 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:26:42.559 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:06.058550+0000", 2024-09-13T11:26:42.559 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00044800299999999999 2024-09-13T11:26:42.559 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:42.559 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:42.559 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for peered", 2024-09-13T11:26:42.559 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:06.058560+0000", 2024-09-13T11:26:42.559 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.5200000000000003e-06 2024-09-13T11:26:42.559 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:42.559 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:42.559 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:26:42.559 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:06.286141+0000", 2024-09-13T11:26:42.559 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.22758104200000001 2024-09-13T11:26:42.559 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:42.559 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:42.560 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:26:42.560 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:06.286335+0000", 2024-09-13T11:26:42.560 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00019429700000000001 2024-09-13T11:26:42.560 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:42.560 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:42.560 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 3,14", 2024-09-13T11:26:42.560 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:06.286518+0000", 2024-09-13T11:26:42.560 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00018350500000000001 2024-09-13T11:26:42.560 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:42.560 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:42.560 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:42.560 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:06.296043+0000", 2024-09-13T11:26:42.560 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0095243600000000008 2024-09-13T11:26:42.560 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:42.561 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:42.561 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:26:42.561 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:06.296384+0000", 2024-09-13T11:26:42.561 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00034132100000000001 2024-09-13T11:26:42.561 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:42.561 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:42.561 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:42.561 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:06.307532+0000", 2024-09-13T11:26:42.561 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.011148025000000001 2024-09-13T11:26:42.561 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:42.561 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:42.561 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:26:42.561 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:06.307611+0000", 2024-09-13T11:26:42.561 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.8465000000000002e-05 2024-09-13T11:26:42.562 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:42.562 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:42.562 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:26:42.562 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:06.307734+0000", 2024-09-13T11:26:42.562 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00012301599999999999 2024-09-13T11:26:42.562 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:42.562 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:42.562 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:42.562 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:42.562 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:42.562 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:221528 3.15 3:afda25ed:::benchmark_data_smithi177_31116_object81225:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e193)", 2024-09-13T11:26:42.562 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:26:06.058152+0000", 2024-09-13T11:26:42.562 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 36.407168839999997, 2024-09-13T11:26:42.562 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.25716789899999998, 2024-09-13T11:26:42.562 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:26:42.563 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:26:42.563 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:42.563 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:26:42.563 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:26:42.563 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:42.563 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 221528 2024-09-13T11:26:42.563 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:42.563 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:26:42.563 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:42.563 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:26:42.563 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:06.058152+0000", 2024-09-13T11:26:42.563 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:42.563 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:42.563 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:42.563 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:26:42.564 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:06.058152+0000", 2024-09-13T11:26:42.564 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:42.564 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:42.564 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:42.564 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:26:42.564 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:06.058153+0000", 2024-09-13T11:26:42.564 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.773e-06 2024-09-13T11:26:42.564 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:42.564 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:42.564 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:26:42.564 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:06.058161+0000", 2024-09-13T11:26:42.564 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.9160000000000001e-06 2024-09-13T11:26:42.564 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:42.565 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:42.565 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:26:42.565 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:06.058186+0000", 2024-09-13T11:26:42.565 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.5037000000000001e-05 2024-09-13T11:26:42.565 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:42.565 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:42.565 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:26:42.565 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:06.058202+0000", 2024-09-13T11:26:42.565 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.5709000000000002e-05 2024-09-13T11:26:42.565 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:42.565 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:42.565 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:26:42.565 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:06.058640+0000", 2024-09-13T11:26:42.565 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00043771400000000002 2024-09-13T11:26:42.566 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:42.566 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:42.566 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for peered", 2024-09-13T11:26:42.566 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:06.058664+0000", 2024-09-13T11:26:42.566 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.385e-05 2024-09-13T11:26:42.566 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:42.566 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:42.566 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:26:42.566 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:06.291953+0000", 2024-09-13T11:26:42.566 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.23328986800000001 2024-09-13T11:26:42.566 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:42.566 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:42.566 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:26:42.566 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:06.292123+0000", 2024-09-13T11:26:42.566 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00016977600000000001 2024-09-13T11:26:42.567 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:42.567 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:42.567 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 3,14", 2024-09-13T11:26:42.567 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:06.292276+0000", 2024-09-13T11:26:42.567 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000153196 2024-09-13T11:26:42.567 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:42.567 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:42.567 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:26:42.567 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:06.296387+0000", 2024-09-13T11:26:42.567 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0041105459999999996 2024-09-13T11:26:42.567 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:42.567 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:42.567 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:42.567 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:06.307779+0000", 2024-09-13T11:26:42.568 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.011391712 2024-09-13T11:26:42.568 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:42.568 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:42.568 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:42.568 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:06.315151+0000", 2024-09-13T11:26:42.568 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0073723169999999998 2024-09-13T11:26:42.568 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:42.568 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:42.568 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:26:42.568 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:06.315219+0000", 2024-09-13T11:26:42.568 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.7984999999999994e-05 2024-09-13T11:26:42.568 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:42.568 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:42.569 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:26:42.569 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:06.315319+0000", 2024-09-13T11:26:42.569 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0001005 2024-09-13T11:26:42.569 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:42.569 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:42.569 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:42.569 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:42.569 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:42.569 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:243211 3.22 3:445e4152:::benchmark_data_smithi177_31116_object102908:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e207)", 2024-09-13T11:26:42.569 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:26:26.318899+0000", 2024-09-13T11:26:42.569 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 16.14642177, 2024-09-13T11:26:42.569 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2078296390000001, 2024-09-13T11:26:42.569 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:26:42.569 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:26:42.569 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-13T11:26:42.570 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:26:42.570 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:26:42.570 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:42.570 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 243211 2024-09-13T11:26:42.570 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:42.570 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:26:42.570 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:42.570 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:26:42.570 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:26.318899+0000", 2024-09-13T11:26:42.570 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:42.570 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:42.570 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:42.570 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:26:42.570 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:26.318899+0000", 2024-09-13T11:26:42.571 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:42.571 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:42.571 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:42.571 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:26:42.571 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:26.318899+0000", 2024-09-13T11:26:42.571 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.7199999999999999e-07 2024-09-13T11:26:42.571 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:42.571 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:42.571 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:26:42.571 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:26.318902+0000", 2024-09-13T11:26:42.571 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.616e-06 2024-09-13T11:26:42.571 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:42.571 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:42.571 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:26:42.572 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:26.318906+0000", 2024-09-13T11:26:42.572 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.2440000000000004e-06 2024-09-13T11:26:42.572 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:42.572 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:42.572 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:26:42.572 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:26.318909+0000", 2024-09-13T11:26:42.572 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.9830000000000001e-06 2024-09-13T11:26:42.572 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:42.572 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:42.572 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:26:42.572 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:26.319101+0000", 2024-09-13T11:26:42.572 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00019198800000000001 2024-09-13T11:26:42.572 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:42.572 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:42.573 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for missing object", 2024-09-13T11:26:42.573 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:26.319166+0000", 2024-09-13T11:26:42.573 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.4906999999999994e-05 2024-09-13T11:26:42.573 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:42.573 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:42.573 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:26:42.573 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:27.526716+0000", 2024-09-13T11:26:42.573 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.207550323 2024-09-13T11:26:42.573 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:42.573 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:42.573 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:26:42.573 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:27.526728+0000", 2024-09-13T11:26:42.573 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2006e-05 2024-09-13T11:26:42.573 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:42.574 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:42.574 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:42.574 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:42.574 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:42.574 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-13T11:26:42.576 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-13T11:26:42.576 INFO:teuthology.orchestra.run.smithi001.stdout:osd.13: {} 2024-09-13T11:26:42.576 INFO:teuthology.orchestra.run.smithi001.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-13T11:26:42.576 INFO:teuthology.orchestra.run.smithi001.stdout:osd.14: {} 2024-09-13T11:26:42.577 INFO:teuthology.orchestra.run.smithi001.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-13T11:26:42.577 INFO:teuthology.orchestra.run.smithi001.stdout:osd.15: {} 2024-09-13T11:26:42.578 INFO:teuthology.orchestra.run.smithi001.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-13T11:26:42.578 INFO:tasks.ceph.osd.10.smithi162.stderr:2024-09-13T11:26:42.571+0000 7fb59c9f0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26317) UID: 0 2024-09-13T11:26:42.676 INFO:tasks.ceph.osd.10.smithi162.stderr:2024-09-13T11:26:42.675+0000 7fb59c9f0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26317) UID: 0 2024-09-13T11:26:42.745 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-13T11:26:42.745 INFO:teuthology.orchestra.run.smithi001.stdout: "ops": [ 2024-09-13T11:26:42.745 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:42.746 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:260396 3.1a 3:5c32470c:::benchmark_data_smithi177_31116_object120093:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e219)", 2024-09-13T11:26:42.746 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:26:42.693257+0000", 2024-09-13T11:26:42.746 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 0.051961607999999999, 2024-09-13T11:26:42.746 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.052019451000000001, 2024-09-13T11:26:42.746 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:26:42.746 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:26:42.746 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "waiting for sub ops", 2024-09-13T11:26:42.746 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:26:42.746 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:26:42.746 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:42.746 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 260396 2024-09-13T11:26:42.746 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:42.746 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:26:42.747 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:42.747 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:26:42.747 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:42.693257+0000", 2024-09-13T11:26:42.747 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:42.747 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:42.747 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:42.747 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:26:42.747 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:42.693257+0000", 2024-09-13T11:26:42.747 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:42.748 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:42.748 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:42.748 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:26:42.748 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:42.693260+0000", 2024-09-13T11:26:42.748 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.2959999999999999e-06 2024-09-13T11:26:42.748 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:42.748 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:42.748 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:26:42.748 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:42.693270+0000", 2024-09-13T11:26:42.748 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.7429999999999997e-06 2024-09-13T11:26:42.748 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:42.748 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:42.749 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:26:42.749 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:42.693298+0000", 2024-09-13T11:26:42.749 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.7483000000000001e-05 2024-09-13T11:26:42.749 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:42.749 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:42.749 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:26:42.749 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:42.693312+0000", 2024-09-13T11:26:42.749 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.4127e-05 2024-09-13T11:26:42.749 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:42.749 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:42.749 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:26:42.749 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:42.693449+0000", 2024-09-13T11:26:42.749 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000137496 2024-09-13T11:26:42.750 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:42.750 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:42.750 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:26:42.750 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:42.693674+0000", 2024-09-13T11:26:42.750 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000224448 2024-09-13T11:26:42.750 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:42.750 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:42.750 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 9", 2024-09-13T11:26:42.750 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:42.693874+0000", 2024-09-13T11:26:42.750 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000200349 2024-09-13T11:26:42.750 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:42.750 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:42.751 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:26:42.751 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:42.698884+0000", 2024-09-13T11:26:42.751 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0050097600000000003 2024-09-13T11:26:42.751 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:42.751 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:42.751 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:42.751 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:42.751 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-09-13T11:26:42.751 INFO:teuthology.orchestra.run.smithi001.stdout: "num_ops": 1 2024-09-13T11:26:42.751 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-13T11:26:42.765 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-13T11:26:42.777 INFO:tasks.ceph.osd.13.smithi092.stderr:2024-09-13T11:26:42.775+0000 7f3e5afea640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26350) UID: 0 2024-09-13T11:26:42.878 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-09-13T11:26:42.875+0000 7f97befc9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26315) UID: 0 2024-09-13T11:26:42.909 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-13T11:26:42.909 INFO:teuthology.orchestra.run.smithi001.stdout: "ops": [], 2024-09-13T11:26:42.909 INFO:teuthology.orchestra.run.smithi001.stdout: "complaint_time": 30, 2024-09-13T11:26:42.909 INFO:teuthology.orchestra.run.smithi001.stdout: "num_blocked_ops": 0 2024-09-13T11:26:42.909 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-13T11:26:42.925 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-13T11:26:42.979 INFO:tasks.ceph.osd.13.smithi092.stderr:2024-09-13T11:26:42.975+0000 7f3e5afea640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26350) UID: 0 2024-09-13T11:26:43.010 INFO:tasks.ceph.osd.7.smithi177.stderr:2024-09-13T11:26:43.005+0000 7f896ff4a640 -1 osd.7 219 handle_osd_map injecting map crc failure 2024-09-13T11:26:43.077 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-13T11:26:43.077 INFO:teuthology.orchestra.run.smithi001.stdout: "size": 20, 2024-09-13T11:26:43.077 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 600, 2024-09-13T11:26:43.078 INFO:teuthology.orchestra.run.smithi001.stdout: "ops": [ 2024-09-13T11:26:43.078 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.078 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDPGPush(3.1a 102/78 [PushOp(3:5df50766:::benchmark_data_smithi177_31116_object50543:head, version: 72'3152, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:5df50766:::benchmark_data_smithi177_31116_object50543:head@72'3152, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:26:43.078 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:24:00.499311+0000", 2024-09-13T11:26:43.078 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 162.57515577999999, 2024-09-13T11:26:43.078 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.26146664800000002, 2024-09-13T11:26:43.078 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:26:43.078 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:26:43.078 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:26:43.078 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:26:43.078 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.078 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:26:43.078 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:00.499311+0000", 2024-09-13T11:26:43.079 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:43.079 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.079 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.079 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:26:43.079 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:00.499311+0000", 2024-09-13T11:26:43.079 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:43.079 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.079 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.079 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:26:43.079 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:00.499320+0000", 2024-09-13T11:26:43.079 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.6400000000000003e-06 2024-09-13T11:26:43.079 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.079 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.079 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:26:43.080 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:00.499384+0000", 2024-09-13T11:26:43.080 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.4554999999999994e-05 2024-09-13T11:26:43.080 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.080 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.080 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:26:43.080 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:00.759713+0000", 2024-09-13T11:26:43.080 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.260328742 2024-09-13T11:26:43.080 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.080 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.080 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:26:43.080 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:00.759742+0000", 2024-09-13T11:26:43.080 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.9298e-05 2024-09-13T11:26:43.080 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.081 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.081 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:26:43.081 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:00.759871+0000", 2024-09-13T11:26:43.081 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000128992 2024-09-13T11:26:43.081 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.081 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.081 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:26:43.081 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:00.759884+0000", 2024-09-13T11:26:43.081 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2687999999999999e-05 2024-09-13T11:26:43.081 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.081 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.081 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:26:43.081 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:00.760778+0000", 2024-09-13T11:26:43.081 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00089373299999999996 2024-09-13T11:26:43.082 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:43.082 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:43.082 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:43.082 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.082 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.082 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4665.0:82769 3.1e e105/102 3:7a4e88b5:::benchmark_data_smithi177_31116_object82768:head v 105'4381, mlcod=105'4379)", 2024-09-13T11:26:43.082 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:24:03.745047+0000", 2024-09-13T11:26:43.082 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 159.32941955499999, 2024-09-13T11:26:43.082 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.213806145, 2024-09-13T11:26:43.082 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:26:43.082 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:26:43.082 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:43.082 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:26:43.082 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.083 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:26:43.083 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:03.745047+0000", 2024-09-13T11:26:43.083 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:43.083 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.083 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.083 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:26:43.083 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:03.745047+0000", 2024-09-13T11:26:43.083 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:43.083 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.083 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.083 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:26:43.083 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:03.745049+0000", 2024-09-13T11:26:43.083 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.5710000000000001e-06 2024-09-13T11:26:43.083 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.084 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.084 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:26:43.084 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:03.745107+0000", 2024-09-13T11:26:43.084 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.8799999999999999e-05 2024-09-13T11:26:43.084 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.084 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.084 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:26:43.084 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:03.955145+0000", 2024-09-13T11:26:43.084 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.210037735 2024-09-13T11:26:43.084 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.084 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.084 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:26:43.084 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:03.955163+0000", 2024-09-13T11:26:43.085 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.7520000000000002e-05 2024-09-13T11:26:43.085 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.085 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.085 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:26:43.085 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:03.955473+0000", 2024-09-13T11:26:43.085 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00031014399999999999 2024-09-13T11:26:43.085 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.085 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.085 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:26:43.085 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:03.955538+0000", 2024-09-13T11:26:43.085 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.5059000000000002e-05 2024-09-13T11:26:43.085 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.085 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.085 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:26:43.086 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:03.958753+0000", 2024-09-13T11:26:43.086 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0032152769999999999 2024-09-13T11:26:43.086 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.086 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.086 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:26:43.086 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:03.958853+0000", 2024-09-13T11:26:43.086 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000100039 2024-09-13T11:26:43.086 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:43.086 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:43.086 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:43.086 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.086 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.086 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:98582 3.1a 3:59f9cd75:::benchmark_data_smithi177_31116_object98581:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113)", 2024-09-13T11:26:43.086 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:24:15.431226+0000", 2024-09-13T11:26:43.087 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 147.64324108599999, 2024-09-13T11:26:43.087 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.19820755800000001, 2024-09-13T11:26:43.087 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:26:43.087 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:26:43.087 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:43.087 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:26:43.087 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:26:43.087 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:43.087 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 98582 2024-09-13T11:26:43.087 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.087 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:26:43.087 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.087 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:26:43.087 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.431226+0000", 2024-09-13T11:26:43.088 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:43.088 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.088 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.088 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:26:43.088 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.431226+0000", 2024-09-13T11:26:43.088 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:43.088 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.088 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.088 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:26:43.088 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.431229+0000", 2024-09-13T11:26:43.088 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.7100000000000001e-06 2024-09-13T11:26:43.088 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.089 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.089 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:26:43.089 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.431334+0000", 2024-09-13T11:26:43.089 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00010506900000000001 2024-09-13T11:26:43.089 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.089 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.089 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:26:43.089 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.431364+0000", 2024-09-13T11:26:43.089 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.9607e-05 2024-09-13T11:26:43.089 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.089 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.089 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:26:43.089 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.431388+0000", 2024-09-13T11:26:43.089 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.3808999999999999e-05 2024-09-13T11:26:43.090 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.090 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.090 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:26:43.090 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.431483+0000", 2024-09-13T11:26:43.090 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.5402999999999994e-05 2024-09-13T11:26:43.090 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.090 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.090 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:26:43.090 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.431642+0000", 2024-09-13T11:26:43.090 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000158412 2024-09-13T11:26:43.091 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.091 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.091 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 9", 2024-09-13T11:26:43.091 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.431796+0000", 2024-09-13T11:26:43.091 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00015467899999999999 2024-09-13T11:26:43.091 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.091 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.091 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:26:43.091 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.433731+0000", 2024-09-13T11:26:43.091 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.001934945 2024-09-13T11:26:43.091 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.091 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.091 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:43.091 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.629245+0000", 2024-09-13T11:26:43.092 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.19551390199999999 2024-09-13T11:26:43.092 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.092 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.092 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:26:43.092 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.629346+0000", 2024-09-13T11:26:43.092 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000100869 2024-09-13T11:26:43.092 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.092 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.092 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:26:43.092 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.629433+0000", 2024-09-13T11:26:43.092 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.7152999999999997e-05 2024-09-13T11:26:43.092 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:43.092 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:43.092 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:43.093 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.093 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.093 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:99105 3.1a 3:5ac5090c:::benchmark_data_smithi177_31116_object99104:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113)", 2024-09-13T11:26:43.093 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:24:15.962466+0000", 2024-09-13T11:26:43.093 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 147.11200079700001, 2024-09-13T11:26:43.093 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.23832849, 2024-09-13T11:26:43.093 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:26:43.093 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:26:43.093 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:43.093 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:26:43.093 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:26:43.093 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:43.093 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 99105 2024-09-13T11:26:43.093 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.094 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:26:43.094 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.094 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:26:43.094 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.962466+0000", 2024-09-13T11:26:43.094 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:43.094 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.094 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.094 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:26:43.094 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.962466+0000", 2024-09-13T11:26:43.094 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:43.094 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.094 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.094 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:26:43.095 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.962471+0000", 2024-09-13T11:26:43.095 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.9570000000000001e-06 2024-09-13T11:26:43.095 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.095 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.095 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:26:43.095 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.962547+0000", 2024-09-13T11:26:43.095 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.5947999999999999e-05 2024-09-13T11:26:43.095 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.095 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.095 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:26:43.095 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.962576+0000", 2024-09-13T11:26:43.095 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.9547999999999999e-05 2024-09-13T11:26:43.095 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.095 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.096 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:26:43.096 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.962594+0000", 2024-09-13T11:26:43.096 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.8023e-05 2024-09-13T11:26:43.096 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.096 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.096 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:26:43.096 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.962770+0000", 2024-09-13T11:26:43.096 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00017551099999999999 2024-09-13T11:26:43.096 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.096 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.096 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:26:43.096 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.963114+0000", 2024-09-13T11:26:43.096 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00034440399999999998 2024-09-13T11:26:43.096 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.097 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.097 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 9", 2024-09-13T11:26:43.097 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.963351+0000", 2024-09-13T11:26:43.097 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000237209 2024-09-13T11:26:43.097 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.097 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.097 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:26:43.097 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.966345+0000", 2024-09-13T11:26:43.097 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0029933519999999999 2024-09-13T11:26:43.097 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.097 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.097 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:43.097 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:16.200724+0000", 2024-09-13T11:26:43.098 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.23437934599999999 2024-09-13T11:26:43.098 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.098 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.098 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:26:43.098 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:16.200761+0000", 2024-09-13T11:26:43.098 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.6406999999999999e-05 2024-09-13T11:26:43.098 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.098 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.098 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:26:43.098 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:16.200794+0000", 2024-09-13T11:26:43.098 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.3785000000000003e-05 2024-09-13T11:26:43.098 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:43.098 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:43.098 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:43.099 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.099 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.099 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:129921 3.1a 3:5fb9088b:::benchmark_data_smithi177_31116_object129920:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e134)", 2024-09-13T11:26:43.099 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:24:47.177556+0000", 2024-09-13T11:26:43.099 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 115.896910518, 2024-09-13T11:26:43.099 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.33804469999999998, 2024-09-13T11:26:43.099 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:26:43.099 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:26:43.099 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:43.099 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:26:43.099 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:26:43.099 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:43.099 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 129921 2024-09-13T11:26:43.099 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.099 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:26:43.100 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.100 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:26:43.100 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:47.177556+0000", 2024-09-13T11:26:43.100 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:43.100 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.100 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.100 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:26:43.100 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:47.177556+0000", 2024-09-13T11:26:43.100 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:43.100 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.100 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.100 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:26:43.100 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:47.177559+0000", 2024-09-13T11:26:43.100 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.0790000000000001e-06 2024-09-13T11:26:43.101 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.101 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.101 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:26:43.101 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:47.177589+0000", 2024-09-13T11:26:43.101 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.0173999999999999e-05 2024-09-13T11:26:43.101 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.101 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.101 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:26:43.101 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:47.177606+0000", 2024-09-13T11:26:43.101 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.6475999999999999e-05 2024-09-13T11:26:43.101 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.101 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.101 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:26:43.101 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:47.177615+0000", 2024-09-13T11:26:43.102 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.5729999999999997e-06 2024-09-13T11:26:43.102 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.102 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.102 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:26:43.102 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:47.177680+0000", 2024-09-13T11:26:43.102 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.4349000000000003e-05 2024-09-13T11:26:43.102 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.102 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.102 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:26:43.102 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:47.177794+0000", 2024-09-13T11:26:43.102 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000114119 2024-09-13T11:26:43.102 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.102 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.102 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 9", 2024-09-13T11:26:43.103 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:47.177895+0000", 2024-09-13T11:26:43.103 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000101078 2024-09-13T11:26:43.103 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.103 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.103 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:26:43.103 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:47.179634+0000", 2024-09-13T11:26:43.103 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.001738606 2024-09-13T11:26:43.103 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.103 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.103 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:43.103 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:47.515495+0000", 2024-09-13T11:26:43.103 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.33586143000000002 2024-09-13T11:26:43.103 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.103 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.104 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:26:43.104 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:47.515551+0000", 2024-09-13T11:26:43.104 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.6048999999999998e-05 2024-09-13T11:26:43.104 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.104 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.104 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:26:43.104 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:47.515601+0000", 2024-09-13T11:26:43.104 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.9767e-05 2024-09-13T11:26:43.104 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:43.104 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:43.104 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:43.104 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.104 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.104 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4665.0:140742 3.1e e144/102 3:79c694a0:::benchmark_data_smithi177_31116_object439:head v 144'6204, mlcod=144'6200)", 2024-09-13T11:26:43.104 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:24:58.321534+0000", 2024-09-13T11:26:43.105 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 104.75293299400001, 2024-09-13T11:26:43.105 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.23629921500000001, 2024-09-13T11:26:43.105 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:26:43.105 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:26:43.105 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:43.105 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:26:43.105 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.105 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:26:43.105 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.321534+0000", 2024-09-13T11:26:43.105 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:43.105 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.105 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.105 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:26:43.106 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.321534+0000", 2024-09-13T11:26:43.106 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:43.106 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.106 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.106 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:26:43.106 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.321535+0000", 2024-09-13T11:26:43.106 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.159e-06 2024-09-13T11:26:43.106 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.106 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.106 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:26:43.106 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.321545+0000", 2024-09-13T11:26:43.106 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.9219999999999992e-06 2024-09-13T11:26:43.106 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.106 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.106 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:26:43.107 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.553719+0000", 2024-09-13T11:26:43.107 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.23217401700000001 2024-09-13T11:26:43.107 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.107 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.107 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:26:43.107 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.553734+0000", 2024-09-13T11:26:43.107 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.5344000000000001e-05 2024-09-13T11:26:43.107 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.107 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.107 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:26:43.107 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.553858+0000", 2024-09-13T11:26:43.107 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00012357500000000001 2024-09-13T11:26:43.107 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.107 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.108 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:26:43.108 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.553903+0000", 2024-09-13T11:26:43.108 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.5374999999999998e-05 2024-09-13T11:26:43.108 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.108 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.108 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:26:43.108 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.557719+0000", 2024-09-13T11:26:43.108 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0038158670000000001 2024-09-13T11:26:43.108 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.108 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.108 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:26:43.108 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.557833+0000", 2024-09-13T11:26:43.108 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000113956 2024-09-13T11:26:43.109 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:43.109 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:43.109 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:43.109 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.109 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.109 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4665.0:140796 3.1e e144/102 3:7987a652:::benchmark_data_smithi177_31116_object493:head v 144'6205, mlcod=144'6202)", 2024-09-13T11:26:43.109 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:24:58.553657+0000", 2024-09-13T11:26:43.109 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 104.52081001800001, 2024-09-13T11:26:43.109 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.25308323100000002, 2024-09-13T11:26:43.109 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:26:43.109 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:26:43.109 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:43.109 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:26:43.109 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.110 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:26:43.110 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.553657+0000", 2024-09-13T11:26:43.110 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:43.110 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.110 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.110 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:26:43.110 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.553657+0000", 2024-09-13T11:26:43.110 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:43.110 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.110 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.110 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:26:43.110 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.553660+0000", 2024-09-13T11:26:43.110 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.7519999999999998e-06 2024-09-13T11:26:43.110 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.110 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.111 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:26:43.111 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.553678+0000", 2024-09-13T11:26:43.111 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.7156e-05 2024-09-13T11:26:43.111 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.111 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.111 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:26:43.111 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.804342+0000", 2024-09-13T11:26:43.111 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.25066436199999997 2024-09-13T11:26:43.111 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.111 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.111 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:26:43.111 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.804353+0000", 2024-09-13T11:26:43.111 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.1147e-05 2024-09-13T11:26:43.111 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.111 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.112 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:26:43.112 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.804433+0000", 2024-09-13T11:26:43.112 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.9586999999999995e-05 2024-09-13T11:26:43.112 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.112 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.112 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:26:43.112 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.804457+0000", 2024-09-13T11:26:43.112 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.4108e-05 2024-09-13T11:26:43.112 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.112 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.112 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:26:43.112 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.806684+0000", 2024-09-13T11:26:43.112 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0022271309999999998 2024-09-13T11:26:43.112 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.113 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.113 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:26:43.113 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.806740+0000", 2024-09-13T11:26:43.113 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.5988e-05 2024-09-13T11:26:43.113 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:43.113 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:43.113 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:43.113 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.113 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.113 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:149955 3.11 3:8fa29838:::benchmark_data_smithi177_31116_object9652:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e149)", 2024-09-13T11:26:43.113 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:25:05.905492+0000", 2024-09-13T11:26:43.113 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 97.168974644000002, 2024-09-13T11:26:43.113 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.204181955, 2024-09-13T11:26:43.113 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:26:43.114 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:26:43.114 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:43.114 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:26:43.114 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:26:43.114 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:43.114 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 149955 2024-09-13T11:26:43.114 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.114 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:26:43.114 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.114 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:26:43.114 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:05.905492+0000", 2024-09-13T11:26:43.114 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:43.114 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.114 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.115 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:26:43.115 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:05.905492+0000", 2024-09-13T11:26:43.115 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:43.115 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.115 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.115 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:26:43.115 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:05.905494+0000", 2024-09-13T11:26:43.115 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.9369999999999998e-06 2024-09-13T11:26:43.115 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.115 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.115 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:26:43.115 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:05.905500+0000", 2024-09-13T11:26:43.115 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.7039999999999999e-06 2024-09-13T11:26:43.115 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.116 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.116 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:26:43.116 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:05.905512+0000", 2024-09-13T11:26:43.116 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.257e-05 2024-09-13T11:26:43.116 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.116 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.116 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:26:43.116 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:05.905519+0000", 2024-09-13T11:26:43.116 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.5420000000000002e-06 2024-09-13T11:26:43.116 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.116 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.116 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:26:43.116 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:05.905584+0000", 2024-09-13T11:26:43.116 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.4819000000000004e-05 2024-09-13T11:26:43.117 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.117 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.117 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:26:43.117 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:05.905696+0000", 2024-09-13T11:26:43.117 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00011248599999999999 2024-09-13T11:26:43.117 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.117 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.117 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 11", 2024-09-13T11:26:43.117 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:05.905759+0000", 2024-09-13T11:26:43.117 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.2451000000000002e-05 2024-09-13T11:26:43.117 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.117 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.117 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:26:43.117 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:05.907584+0000", 2024-09-13T11:26:43.118 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0018250110000000001 2024-09-13T11:26:43.118 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.118 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.118 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:43.118 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:06.109588+0000", 2024-09-13T11:26:43.118 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.20200470300000001 2024-09-13T11:26:43.118 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.118 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.118 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:26:43.118 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:06.109637+0000", 2024-09-13T11:26:43.118 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.8439000000000002e-05 2024-09-13T11:26:43.118 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.118 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.118 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:26:43.119 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:06.109674+0000", 2024-09-13T11:26:43.119 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.7292999999999997e-05 2024-09-13T11:26:43.119 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:43.119 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:43.119 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:43.119 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.119 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.119 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4665.0:191081 3.1e e175/102 3:7e784823:::benchmark_data_smithi177_31116_object50778:head v 175'7812, mlcod=175'7809)", 2024-09-13T11:26:43.119 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:25:38.547539+0000", 2024-09-13T11:26:43.119 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 64.526927393999998, 2024-09-13T11:26:43.119 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.213922061, 2024-09-13T11:26:43.119 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:26:43.119 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:26:43.119 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:43.120 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:26:43.120 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.120 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:26:43.120 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:38.547539+0000", 2024-09-13T11:26:43.120 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:43.120 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.120 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.120 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:26:43.120 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:38.547539+0000", 2024-09-13T11:26:43.120 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:43.120 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.120 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.120 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:26:43.121 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:38.547543+0000", 2024-09-13T11:26:43.121 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.3629999999999998e-06 2024-09-13T11:26:43.121 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.121 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.121 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:26:43.121 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:38.547558+0000", 2024-09-13T11:26:43.121 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.5043e-05 2024-09-13T11:26:43.121 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.121 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.121 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:26:43.121 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:38.756778+0000", 2024-09-13T11:26:43.121 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.209219936 2024-09-13T11:26:43.121 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.121 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.121 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:26:43.122 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:38.756790+0000", 2024-09-13T11:26:43.122 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2775999999999999e-05 2024-09-13T11:26:43.122 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.122 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.122 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:26:43.122 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:38.756934+0000", 2024-09-13T11:26:43.122 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00014401600000000001 2024-09-13T11:26:43.122 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.122 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.122 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:26:43.122 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:38.757001+0000", 2024-09-13T11:26:43.122 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.6716000000000003e-05 2024-09-13T11:26:43.122 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.122 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.123 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:26:43.123 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:38.761376+0000", 2024-09-13T11:26:43.123 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0043747320000000001 2024-09-13T11:26:43.123 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.123 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.123 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:26:43.123 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:38.761461+0000", 2024-09-13T11:26:43.123 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.5478999999999999e-05 2024-09-13T11:26:43.123 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:43.123 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:43.123 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:43.123 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.123 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.123 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4665.0:191087 3.1e e175/102 3:7a697202:::benchmark_data_smithi177_31116_object50784:head v 175'7813, mlcod=175'7809)", 2024-09-13T11:26:43.124 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:25:38.561827+0000", 2024-09-13T11:26:43.124 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 64.512639183999994, 2024-09-13T11:26:43.124 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.209395781, 2024-09-13T11:26:43.124 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:26:43.124 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:26:43.124 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:43.124 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:26:43.124 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.124 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:26:43.124 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:38.561827+0000", 2024-09-13T11:26:43.124 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:43.124 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.124 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.125 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:26:43.125 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:38.561827+0000", 2024-09-13T11:26:43.125 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:43.125 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.125 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.125 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:26:43.125 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:38.561837+0000", 2024-09-13T11:26:43.125 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.2329999999999995e-06 2024-09-13T11:26:43.125 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.125 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.125 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:26:43.125 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:38.561849+0000", 2024-09-13T11:26:43.125 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2593e-05 2024-09-13T11:26:43.125 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.126 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.126 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:26:43.126 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:38.766804+0000", 2024-09-13T11:26:43.126 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.20495488000000001 2024-09-13T11:26:43.126 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.126 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.126 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:26:43.126 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:38.766823+0000", 2024-09-13T11:26:43.126 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.9143e-05 2024-09-13T11:26:43.126 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.126 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.126 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:26:43.126 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:38.766973+0000", 2024-09-13T11:26:43.126 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000149226 2024-09-13T11:26:43.126 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.127 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.127 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:26:43.127 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:38.767014+0000", 2024-09-13T11:26:43.127 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.1183000000000002e-05 2024-09-13T11:26:43.127 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.127 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.127 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:26:43.127 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:38.771145+0000", 2024-09-13T11:26:43.127 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0041315270000000003 2024-09-13T11:26:43.127 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.127 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.127 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:26:43.127 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:38.771223+0000", 2024-09-13T11:26:43.127 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.7996000000000003e-05 2024-09-13T11:26:43.128 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:43.128 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:43.128 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:43.128 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.128 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.128 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:233802 3.11 3:8e27a54a:::benchmark_data_smithi177_31116_object93499:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e200)", 2024-09-13T11:26:43.128 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:26:16.812014+0000", 2024-09-13T11:26:43.128 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 26.262452916000001, 2024-09-13T11:26:43.128 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.34299280599999998, 2024-09-13T11:26:43.128 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:26:43.128 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:26:43.128 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:43.128 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:26:43.129 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:26:43.129 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:43.129 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 233802 2024-09-13T11:26:43.129 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.129 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:26:43.129 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.129 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:26:43.129 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:16.812014+0000", 2024-09-13T11:26:43.129 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:43.129 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.129 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.129 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:26:43.129 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:16.812014+0000", 2024-09-13T11:26:43.130 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:43.130 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.130 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.130 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:26:43.130 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:16.812015+0000", 2024-09-13T11:26:43.130 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.19e-06 2024-09-13T11:26:43.130 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.130 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.130 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:26:43.130 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:16.812019+0000", 2024-09-13T11:26:43.130 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.8720000000000004e-06 2024-09-13T11:26:43.130 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.130 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.130 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:26:43.131 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:16.812025+0000", 2024-09-13T11:26:43.131 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.3339999999999998e-06 2024-09-13T11:26:43.131 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.131 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.131 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:26:43.131 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:16.812029+0000", 2024-09-13T11:26:43.131 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.2989999999999998e-06 2024-09-13T11:26:43.131 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.131 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.131 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:26:43.131 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:16.812160+0000", 2024-09-13T11:26:43.131 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000130288 2024-09-13T11:26:43.131 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.131 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.131 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:26:43.132 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:16.812298+0000", 2024-09-13T11:26:43.132 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00013822500000000001 2024-09-13T11:26:43.132 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.132 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.132 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 11", 2024-09-13T11:26:43.132 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:16.812412+0000", 2024-09-13T11:26:43.132 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000113947 2024-09-13T11:26:43.132 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.132 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.132 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:26:43.132 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:16.815915+0000", 2024-09-13T11:26:43.132 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0035030259999999998 2024-09-13T11:26:43.132 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.133 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.133 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:43.133 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:17.154942+0000", 2024-09-13T11:26:43.133 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.33902705700000002 2024-09-13T11:26:43.133 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.133 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.133 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:26:43.133 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:17.154978+0000", 2024-09-13T11:26:43.133 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.5549999999999997e-05 2024-09-13T11:26:43.133 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.133 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.133 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:26:43.133 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:17.155007+0000", 2024-09-13T11:26:43.133 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.9017999999999999e-05 2024-09-13T11:26:43.134 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:43.134 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:43.134 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:43.134 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.134 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.134 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop_reply(client.4665.0:233802 3.11 e200/81 ondisk, result = 0)", 2024-09-13T11:26:43.134 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:26:16.815681+0000", 2024-09-13T11:26:43.134 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 26.258785416999999, 2024-09-13T11:26:43.134 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.33932907800000001, 2024-09-13T11:26:43.134 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:26:43.134 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:26:43.134 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:26:43.134 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:26:43.135 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.135 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:26:43.135 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:16.815681+0000", 2024-09-13T11:26:43.135 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:43.135 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.135 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.135 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:26:43.135 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:16.815681+0000", 2024-09-13T11:26:43.135 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:43.135 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.135 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.135 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:26:43.135 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:16.815682+0000", 2024-09-13T11:26:43.135 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.1230000000000001e-06 2024-09-13T11:26:43.135 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.136 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.136 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:26:43.136 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:16.815689+0000", 2024-09-13T11:26:43.136 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.5599999999999999e-06 2024-09-13T11:26:43.136 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.136 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.136 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:26:43.136 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:17.154882+0000", 2024-09-13T11:26:43.136 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.33919308599999998 2024-09-13T11:26:43.136 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.136 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.136 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:26:43.136 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:17.154890+0000", 2024-09-13T11:26:43.136 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.6149999999999999e-06 2024-09-13T11:26:43.137 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.137 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.137 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:26:43.137 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:17.154931+0000", 2024-09-13T11:26:43.137 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.1057999999999999e-05 2024-09-13T11:26:43.137 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.137 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.137 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:26:43.137 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:17.154937+0000", 2024-09-13T11:26:43.137 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.6719999999999998e-06 2024-09-13T11:26:43.137 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.137 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.137 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:26:43.138 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:17.155010+0000", 2024-09-13T11:26:43.138 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.2964000000000001e-05 2024-09-13T11:26:43.138 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:43.138 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:43.138 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:43.138 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.138 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.138 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:234291 3.11 3:8c4e5be8:::benchmark_data_smithi177_31116_object93988:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e201)", 2024-09-13T11:26:43.138 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:26:17.320509+0000", 2024-09-13T11:26:43.138 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 25.753958091000001, 2024-09-13T11:26:43.138 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.231498433, 2024-09-13T11:26:43.138 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:26:43.138 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:26:43.138 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:43.139 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:26:43.139 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:26:43.139 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:43.139 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 234291 2024-09-13T11:26:43.139 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.139 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:26:43.139 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.139 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:26:43.139 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:17.320509+0000", 2024-09-13T11:26:43.139 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:43.139 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.139 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.139 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:26:43.139 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:17.320509+0000", 2024-09-13T11:26:43.140 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:43.140 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.140 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.140 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:26:43.140 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:17.320510+0000", 2024-09-13T11:26:43.140 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.1999999999999999e-06 2024-09-13T11:26:43.140 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.140 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.140 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:26:43.140 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:17.320514+0000", 2024-09-13T11:26:43.140 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.8139999999999998e-06 2024-09-13T11:26:43.140 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.140 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.141 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:26:43.141 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:17.320523+0000", 2024-09-13T11:26:43.141 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.7589999999999992e-06 2024-09-13T11:26:43.141 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.141 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.141 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:26:43.141 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:17.320529+0000", 2024-09-13T11:26:43.141 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.2499999999999997e-06 2024-09-13T11:26:43.141 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.141 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.141 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:26:43.141 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:17.320580+0000", 2024-09-13T11:26:43.141 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.0980999999999999e-05 2024-09-13T11:26:43.141 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.142 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.142 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:26:43.142 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:17.320704+0000", 2024-09-13T11:26:43.142 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00012416599999999999 2024-09-13T11:26:43.142 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.142 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.142 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 11", 2024-09-13T11:26:43.142 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:17.320812+0000", 2024-09-13T11:26:43.142 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000108383 2024-09-13T11:26:43.142 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.142 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.142 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:26:43.142 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:17.323672+0000", 2024-09-13T11:26:43.142 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0028601329999999999 2024-09-13T11:26:43.143 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.143 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.143 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:43.143 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:17.551809+0000", 2024-09-13T11:26:43.143 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.22813635400000001 2024-09-13T11:26:43.143 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.143 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.143 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:26:43.143 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:17.551920+0000", 2024-09-13T11:26:43.143 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000111805 2024-09-13T11:26:43.143 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.143 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.143 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:26:43.144 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:17.552007+0000", 2024-09-13T11:26:43.144 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.6588000000000005e-05 2024-09-13T11:26:43.144 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:43.144 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:43.144 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:43.144 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.144 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.144 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop_reply(client.4665.0:234291 3.11 e201/81 ondisk, result = 0)", 2024-09-13T11:26:43.144 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:26:17.324111+0000", 2024-09-13T11:26:43.144 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 25.750355635999998, 2024-09-13T11:26:43.144 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.22790548199999999, 2024-09-13T11:26:43.144 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:26:43.144 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:26:43.144 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:26:43.145 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:26:43.145 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.145 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:26:43.145 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:17.324111+0000", 2024-09-13T11:26:43.145 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:43.145 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.145 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.145 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:26:43.145 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:17.324111+0000", 2024-09-13T11:26:43.145 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:43.145 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.145 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.145 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:26:43.145 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:17.324112+0000", 2024-09-13T11:26:43.146 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.313e-06 2024-09-13T11:26:43.146 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.146 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.146 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:26:43.146 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:17.324117+0000", 2024-09-13T11:26:43.146 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.5870000000000003e-06 2024-09-13T11:26:43.146 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.146 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.146 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:26:43.146 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:17.551656+0000", 2024-09-13T11:26:43.146 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.22753944200000001 2024-09-13T11:26:43.146 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.146 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.147 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:26:43.147 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:17.551669+0000", 2024-09-13T11:26:43.147 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2233e-05 2024-09-13T11:26:43.147 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.147 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.147 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:26:43.147 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:17.551780+0000", 2024-09-13T11:26:43.147 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000111471 2024-09-13T11:26:43.147 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.147 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.147 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:26:43.147 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:17.551798+0000", 2024-09-13T11:26:43.147 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.7887e-05 2024-09-13T11:26:43.147 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.148 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.148 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:26:43.148 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:17.552017+0000", 2024-09-13T11:26:43.148 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00021854900000000001 2024-09-13T11:26:43.148 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:43.148 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:43.148 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:43.148 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.148 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.148 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:243814 3.1a 3:5aaae69c:::benchmark_data_smithi177_31116_object103511:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e207)", 2024-09-13T11:26:43.148 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:26:26.761070+0000", 2024-09-13T11:26:43.148 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 16.313396585, 2024-09-13T11:26:43.148 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.222488768, 2024-09-13T11:26:43.148 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:26:43.149 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:26:43.149 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:43.149 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:26:43.149 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:26:43.149 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:43.149 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 243814 2024-09-13T11:26:43.149 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.149 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:26:43.149 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.149 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:26:43.149 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:26.761070+0000", 2024-09-13T11:26:43.149 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:43.149 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.149 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.150 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:26:43.150 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:26.761070+0000", 2024-09-13T11:26:43.150 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:43.150 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.150 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.150 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:26:43.150 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:26.761073+0000", 2024-09-13T11:26:43.150 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.9119999999999998e-06 2024-09-13T11:26:43.150 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.150 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.150 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:26:43.150 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:26.761082+0000", 2024-09-13T11:26:43.150 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.7960000000000001e-06 2024-09-13T11:26:43.150 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.151 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.151 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:26:43.151 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:26.761105+0000", 2024-09-13T11:26:43.151 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.2976000000000002e-05 2024-09-13T11:26:43.151 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.151 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.151 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:26:43.151 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:26.761117+0000", 2024-09-13T11:26:43.151 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2323999999999999e-05 2024-09-13T11:26:43.151 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.151 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.151 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:26:43.151 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:26.761207+0000", 2024-09-13T11:26:43.151 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.9534e-05 2024-09-13T11:26:43.152 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.152 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.152 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:26:43.152 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:26.761326+0000", 2024-09-13T11:26:43.152 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000119478 2024-09-13T11:26:43.152 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.152 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.152 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 9", 2024-09-13T11:26:43.152 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:26.761422+0000", 2024-09-13T11:26:43.152 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.577e-05 2024-09-13T11:26:43.152 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.152 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.152 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:26:43.152 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:26.765182+0000", 2024-09-13T11:26:43.153 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0037604280000000001 2024-09-13T11:26:43.153 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.153 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.153 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:43.153 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:26.983388+0000", 2024-09-13T11:26:43.153 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.21820598199999999 2024-09-13T11:26:43.153 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.153 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.153 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:26:43.153 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:26.983485+0000", 2024-09-13T11:26:43.153 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.7072999999999997e-05 2024-09-13T11:26:43.153 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.153 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.153 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:26:43.154 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:26.983559+0000", 2024-09-13T11:26:43.154 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.3495e-05 2024-09-13T11:26:43.154 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:43.154 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:43.154 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:43.154 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.154 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.154 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:256318 3.1a 3:58998337:::benchmark_data_smithi177_31116_object116015:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e214)", 2024-09-13T11:26:43.154 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:26:37.418760+0000", 2024-09-13T11:26:43.154 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 5.6557070469999999, 2024-09-13T11:26:43.154 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.19965370800000001, 2024-09-13T11:26:43.154 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:26:43.154 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:26:43.154 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:43.155 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:26:43.155 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:26:43.155 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:43.155 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 256318 2024-09-13T11:26:43.155 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.155 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:26:43.155 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.155 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:26:43.155 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:37.418760+0000", 2024-09-13T11:26:43.155 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:43.155 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.155 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.155 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:26:43.155 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:37.418760+0000", 2024-09-13T11:26:43.156 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:43.156 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.156 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.156 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:26:43.156 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:37.418763+0000", 2024-09-13T11:26:43.156 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.2770000000000001e-06 2024-09-13T11:26:43.156 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.156 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.156 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:26:43.156 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:37.418771+0000", 2024-09-13T11:26:43.156 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.4209999999999995e-06 2024-09-13T11:26:43.156 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.156 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.156 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:26:43.157 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:37.418790+0000", 2024-09-13T11:26:43.157 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.8898000000000001e-05 2024-09-13T11:26:43.157 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.157 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.157 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:26:43.157 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:37.418800+0000", 2024-09-13T11:26:43.157 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.3819999999999993e-06 2024-09-13T11:26:43.157 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.157 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.157 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:26:43.157 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:37.418868+0000", 2024-09-13T11:26:43.157 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.7972000000000006e-05 2024-09-13T11:26:43.157 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.157 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.158 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:26:43.158 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:37.418974+0000", 2024-09-13T11:26:43.158 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000106036 2024-09-13T11:26:43.158 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.158 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.158 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 9", 2024-09-13T11:26:43.158 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:37.419044+0000", 2024-09-13T11:26:43.158 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.0516e-05 2024-09-13T11:26:43.158 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.158 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.158 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:26:43.158 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:37.421773+0000", 2024-09-13T11:26:43.158 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.002729319 2024-09-13T11:26:43.158 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.159 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.159 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:43.159 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:37.618309+0000", 2024-09-13T11:26:43.159 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.196535667 2024-09-13T11:26:43.159 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.159 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.159 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:26:43.159 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:37.618369+0000", 2024-09-13T11:26:43.159 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.0340999999999997e-05 2024-09-13T11:26:43.159 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.159 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.159 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:26:43.159 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:37.618413+0000", 2024-09-13T11:26:43.159 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.3878999999999998e-05 2024-09-13T11:26:43.160 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:43.160 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:43.160 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:43.160 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.160 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.160 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop_reply(client.4665.0:256318 3.1a e214/105 ondisk, result = 0)", 2024-09-13T11:26:43.160 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:26:37.424396+0000", 2024-09-13T11:26:43.160 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 5.6500704510000004, 2024-09-13T11:26:43.160 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.19402288300000001, 2024-09-13T11:26:43.160 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:26:43.160 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:26:43.160 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:26:43.160 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:26:43.160 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.161 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:26:43.161 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:37.424396+0000", 2024-09-13T11:26:43.161 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:43.161 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.161 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.161 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:26:43.161 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:37.424396+0000", 2024-09-13T11:26:43.161 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:43.161 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.161 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.161 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:26:43.161 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:37.424398+0000", 2024-09-13T11:26:43.161 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.5889999999999999e-06 2024-09-13T11:26:43.162 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.162 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.162 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:26:43.162 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:37.424404+0000", 2024-09-13T11:26:43.162 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.8459999999999996e-06 2024-09-13T11:26:43.162 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.162 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.162 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:26:43.162 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:37.618236+0000", 2024-09-13T11:26:43.162 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.19383228699999999 2024-09-13T11:26:43.162 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.162 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.162 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:26:43.162 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:37.618241+0000", 2024-09-13T11:26:43.163 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.0139999999999998e-06 2024-09-13T11:26:43.163 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.163 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.163 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:26:43.163 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:37.618293+0000", 2024-09-13T11:26:43.163 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.2380999999999999e-05 2024-09-13T11:26:43.163 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.163 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.163 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:26:43.163 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:37.618303+0000", 2024-09-13T11:26:43.163 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.5699999999999999e-06 2024-09-13T11:26:43.163 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.163 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.163 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:26:43.164 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:37.618419+0000", 2024-09-13T11:26:43.164 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000116196 2024-09-13T11:26:43.164 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:43.164 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:43.164 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:43.164 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.164 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.164 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop_reply(client.4665.0:260146 3.1a e219/105 ondisk, result = 0)", 2024-09-13T11:26:43.164 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:26:42.138512+0000", 2024-09-13T11:26:43.164 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 0.93595473600000001, 2024-09-13T11:26:43.164 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.240341576, 2024-09-13T11:26:43.164 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:26:43.164 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:26:43.164 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:26:43.165 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:26:43.165 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.165 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:26:43.165 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:42.138512+0000", 2024-09-13T11:26:43.165 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:43.165 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.165 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.165 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:26:43.165 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:42.138512+0000", 2024-09-13T11:26:43.165 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:43.165 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.165 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.165 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:26:43.165 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:42.138515+0000", 2024-09-13T11:26:43.166 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.9560000000000002e-06 2024-09-13T11:26:43.166 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.166 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.166 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:26:43.166 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:42.138524+0000", 2024-09-13T11:26:43.166 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.0350000000000007e-06 2024-09-13T11:26:43.166 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.166 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.166 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:26:43.166 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:42.378633+0000", 2024-09-13T11:26:43.166 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.24010957299999999 2024-09-13T11:26:43.166 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.166 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.167 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:26:43.167 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:42.378642+0000", 2024-09-13T11:26:43.167 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.7390000000000005e-06 2024-09-13T11:26:43.167 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.167 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.167 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:26:43.167 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:42.378709+0000", 2024-09-13T11:26:43.167 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.6586000000000002e-05 2024-09-13T11:26:43.167 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.167 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.167 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:26:43.167 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:42.378718+0000", 2024-09-13T11:26:43.167 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.5899999999999997e-06 2024-09-13T11:26:43.167 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.168 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.168 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:26:43.168 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:42.378854+0000", 2024-09-13T11:26:43.168 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00013509700000000001 2024-09-13T11:26:43.168 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:43.168 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:43.168 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:43.168 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.168 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.168 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:260356 3.1a 3:5c5cb7ed:::benchmark_data_smithi177_31116_object120053:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e219)", 2024-09-13T11:26:43.168 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:26:42.487175+0000", 2024-09-13T11:26:43.168 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 0.58729129599999996, 2024-09-13T11:26:43.168 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.20567476500000001, 2024-09-13T11:26:43.168 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:26:43.169 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:26:43.169 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:43.169 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:26:43.169 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:26:43.169 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:43.169 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 260356 2024-09-13T11:26:43.169 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.169 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:26:43.169 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.169 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:26:43.169 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:42.487175+0000", 2024-09-13T11:26:43.169 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:43.169 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.169 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.170 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:26:43.170 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:42.487175+0000", 2024-09-13T11:26:43.170 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:43.170 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.170 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.170 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:26:43.170 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:42.487180+0000", 2024-09-13T11:26:43.170 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.7570000000000004e-06 2024-09-13T11:26:43.170 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.170 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.170 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:26:43.170 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:42.487195+0000", 2024-09-13T11:26:43.170 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.5016e-05 2024-09-13T11:26:43.170 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.171 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.171 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:26:43.171 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:42.487227+0000", 2024-09-13T11:26:43.171 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.1909999999999998e-05 2024-09-13T11:26:43.171 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.171 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.171 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:26:43.171 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:42.487252+0000", 2024-09-13T11:26:43.171 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.4902999999999999e-05 2024-09-13T11:26:43.171 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.171 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.171 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:26:43.171 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:42.487451+0000", 2024-09-13T11:26:43.171 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000198839 2024-09-13T11:26:43.172 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.172 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.172 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:26:43.172 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:42.487696+0000", 2024-09-13T11:26:43.172 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00024567399999999998 2024-09-13T11:26:43.172 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.172 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.172 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 9", 2024-09-13T11:26:43.172 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:42.487919+0000", 2024-09-13T11:26:43.172 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00022273099999999999 2024-09-13T11:26:43.172 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.172 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.172 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:26:43.172 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:42.491000+0000", 2024-09-13T11:26:43.173 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0030812769999999999 2024-09-13T11:26:43.173 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.173 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.173 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:43.173 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:42.692668+0000", 2024-09-13T11:26:43.173 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.20166735199999999 2024-09-13T11:26:43.173 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.173 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.173 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:26:43.173 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:42.692765+0000", 2024-09-13T11:26:43.173 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.7607000000000002e-05 2024-09-13T11:26:43.173 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.173 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.173 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:26:43.174 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:42.692850+0000", 2024-09-13T11:26:43.174 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.4698999999999997e-05 2024-09-13T11:26:43.174 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:43.174 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:43.174 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:43.174 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.174 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.174 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop_reply(client.4665.0:260356 3.1a e219/105 ondisk, result = 0)", 2024-09-13T11:26:43.174 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:26:42.494201+0000", 2024-09-13T11:26:43.174 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 0.58026613100000002, 2024-09-13T11:26:43.175 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.198661267, 2024-09-13T11:26:43.175 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:26:43.175 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:26:43.175 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:26:43.175 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:26:43.175 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.175 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:26:43.175 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:42.494201+0000", 2024-09-13T11:26:43.175 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:43.175 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.175 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.175 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:26:43.175 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:42.494201+0000", 2024-09-13T11:26:43.175 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:43.176 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.176 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.176 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:26:43.176 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:42.494202+0000", 2024-09-13T11:26:43.176 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.852e-06 2024-09-13T11:26:43.176 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.176 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.176 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:26:43.176 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:42.494209+0000", 2024-09-13T11:26:43.176 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.7440000000000001e-06 2024-09-13T11:26:43.176 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.176 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.176 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:26:43.176 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:42.692508+0000", 2024-09-13T11:26:43.176 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.198299117 2024-09-13T11:26:43.177 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.177 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.177 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:26:43.177 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:42.692522+0000", 2024-09-13T11:26:43.177 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.3672999999999999e-05 2024-09-13T11:26:43.177 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.177 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.177 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:26:43.177 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:42.692641+0000", 2024-09-13T11:26:43.177 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000118682 2024-09-13T11:26:43.177 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.177 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.177 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:26:43.177 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:42.692657+0000", 2024-09-13T11:26:43.178 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.6103000000000002e-05 2024-09-13T11:26:43.178 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.178 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.178 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:26:43.178 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:42.692862+0000", 2024-09-13T11:26:43.178 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00020509599999999999 2024-09-13T11:26:43.178 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:43.178 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:43.178 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:43.178 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:43.178 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:43.178 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-13T11:26:43.180 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-09-13T11:26:43.180 INFO:tasks.ceph.osd.1.smithi092.stderr:2024-09-13T11:26:43.175+0000 7fe7789c5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33718) UID: 0 2024-09-13T11:26:43.280 INFO:tasks.ceph.osd.14.smithi162.stderr:2024-09-13T11:26:43.279+0000 7f3df5903640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33154) UID: 0 2024-09-13T11:26:43.303 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-13T11:26:43.346 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-13T11:26:43.346 INFO:teuthology.orchestra.run.smithi001.stdout: "ops": [], 2024-09-13T11:26:43.346 INFO:teuthology.orchestra.run.smithi001.stdout: "num_ops": 0 2024-09-13T11:26:43.347 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-13T11:26:43.368 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-09-13T11:26:43.381 INFO:tasks.ceph.osd.3.smithi177.stderr:2024-09-13T11:26:43.377+0000 7f10adba4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26574) UID: 0 2024-09-13T11:26:43.482 INFO:tasks.ceph.osd.13.smithi092.stderr:2024-09-13T11:26:43.479+0000 7f3e5afea640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26350) UID: 0 2024-09-13T11:26:43.560 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-13T11:26:43.561 INFO:teuthology.orchestra.run.smithi001.stdout: "ops": [], 2024-09-13T11:26:43.561 INFO:teuthology.orchestra.run.smithi001.stdout: "complaint_time": 30, 2024-09-13T11:26:43.561 INFO:teuthology.orchestra.run.smithi001.stdout: "num_blocked_ops": 0 2024-09-13T11:26:43.561 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-13T11:26:43.576 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-09-13T11:26:43.582 INFO:tasks.ceph.osd.11.smithi177.stderr:2024-09-13T11:26:43.581+0000 7f24792c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26580) UID: 0 2024-09-13T11:26:43.683 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-09-13T11:26:43.679+0000 7f97befc9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26315) UID: 0 2024-09-13T11:26:43.728 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-13T11:26:43.728 INFO:teuthology.orchestra.run.smithi001.stdout: "size": 20, 2024-09-13T11:26:43.728 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 600, 2024-09-13T11:26:43.728 INFO:teuthology.orchestra.run.smithi001.stdout: "ops": [ 2024-09-13T11:26:43.729 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.729 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:54656 3.1e 3.6a49163e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e75)", 2024-09-13T11:26:43.729 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:30.979993+0000", 2024-09-13T11:26:43.729 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 192.74658845799999, 2024-09-13T11:26:43.729 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.9269557150000001, 2024-09-13T11:26:43.729 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:26:43.729 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:26:43.729 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-13T11:26:43.729 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:26:43.729 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:26:43.729 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:43.729 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 54656 2024-09-13T11:26:43.729 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.729 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:26:43.730 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.730 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:26:43.730 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:30.979993+0000", 2024-09-13T11:26:43.730 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:43.730 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.730 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.730 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:26:43.730 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:30.979993+0000", 2024-09-13T11:26:43.730 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:43.730 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.730 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.730 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:26:43.731 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:30.979997+0000", 2024-09-13T11:26:43.731 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.0319999999999997e-06 2024-09-13T11:26:43.731 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.731 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.731 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:26:43.731 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:30.980057+0000", 2024-09-13T11:26:43.731 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.0341999999999999e-05 2024-09-13T11:26:43.731 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.731 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.731 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:26:43.731 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:30.980094+0000", 2024-09-13T11:26:43.731 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.6770000000000002e-05 2024-09-13T11:26:43.732 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.732 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.732 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:26:43.732 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:30.980113+0000", 2024-09-13T11:26:43.732 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.8791999999999999e-05 2024-09-13T11:26:43.732 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.732 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.732 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:26:43.732 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.097773+0000", 2024-09-13T11:26:43.732 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.117660379 2024-09-13T11:26:43.732 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.732 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.732 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for peered", 2024-09-13T11:26:43.733 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.097784+0000", 2024-09-13T11:26:43.733 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.1025e-05 2024-09-13T11:26:43.733 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.733 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.733 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:26:43.733 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.906937+0000", 2024-09-13T11:26:43.733 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.8091523789999999 2024-09-13T11:26:43.733 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.733 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.733 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:26:43.733 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.906949+0000", 2024-09-13T11:26:43.733 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.1996e-05 2024-09-13T11:26:43.733 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:43.733 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:43.734 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:43.734 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.734 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.734 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:54720 3.1e 3.a944303e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-13T11:26:43.734 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:31.973315+0000", 2024-09-13T11:26:43.734 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 191.753266223, 2024-09-13T11:26:43.734 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.93366527200000005, 2024-09-13T11:26:43.734 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:26:43.734 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:26:43.734 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-13T11:26:43.734 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:26:43.734 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:26:43.735 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:43.735 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 54720 2024-09-13T11:26:43.735 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.735 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:26:43.735 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.735 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:26:43.735 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.973315+0000", 2024-09-13T11:26:43.735 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:43.735 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.735 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.735 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:26:43.735 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.973315+0000", 2024-09-13T11:26:43.735 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:43.735 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.736 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.736 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:26:43.736 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.973318+0000", 2024-09-13T11:26:43.736 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.5830000000000001e-06 2024-09-13T11:26:43.736 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.736 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.736 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:26:43.736 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.973465+0000", 2024-09-13T11:26:43.736 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000146816 2024-09-13T11:26:43.736 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.736 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.736 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:26:43.736 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.973478+0000", 2024-09-13T11:26:43.736 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2931e-05 2024-09-13T11:26:43.737 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.737 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.737 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:26:43.737 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.973483+0000", 2024-09-13T11:26:43.737 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.8969999999999996e-06 2024-09-13T11:26:43.737 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.737 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.737 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:26:43.737 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.973548+0000", 2024-09-13T11:26:43.737 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.4771999999999996e-05 2024-09-13T11:26:43.737 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.737 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.737 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for peered", 2024-09-13T11:26:43.737 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.973557+0000", 2024-09-13T11:26:43.738 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.0359999999999995e-06 2024-09-13T11:26:43.738 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.738 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.738 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:26:43.738 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.906974+0000", 2024-09-13T11:26:43.738 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.93341706000000002 2024-09-13T11:26:43.738 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.738 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.738 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:26:43.738 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.906980+0000", 2024-09-13T11:26:43.738 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.1770000000000003e-06 2024-09-13T11:26:43.738 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:43.738 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:43.738 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:43.739 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.739 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.739 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:54734 3.1e 3.d7f6f7e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-13T11:26:43.739 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:31.983342+0000", 2024-09-13T11:26:43.739 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 191.74323942000001, 2024-09-13T11:26:43.739 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.92367058599999996, 2024-09-13T11:26:43.739 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:26:43.739 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:26:43.739 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-13T11:26:43.739 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:26:43.739 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:26:43.739 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:43.739 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 54734 2024-09-13T11:26:43.740 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.740 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:26:43.740 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.740 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:26:43.740 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.983342+0000", 2024-09-13T11:26:43.740 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:43.740 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.740 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.740 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:26:43.740 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.983342+0000", 2024-09-13T11:26:43.740 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:43.740 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.740 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.741 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:26:43.744 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.983345+0000", 2024-09-13T11:26:43.745 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.6220000000000001e-06 2024-09-13T11:26:43.745 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.745 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.745 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:26:43.745 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.983438+0000", 2024-09-13T11:26:43.745 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.3517000000000006e-05 2024-09-13T11:26:43.745 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.745 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.745 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:26:43.745 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.983451+0000", 2024-09-13T11:26:43.745 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2527999999999999e-05 2024-09-13T11:26:43.745 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.746 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.746 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:26:43.746 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.983458+0000", 2024-09-13T11:26:43.746 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.6380000000000004e-06 2024-09-13T11:26:43.746 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.746 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.746 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:26:43.746 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.983521+0000", 2024-09-13T11:26:43.746 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.2729000000000002e-05 2024-09-13T11:26:43.746 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.746 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.746 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for peered", 2024-09-13T11:26:43.746 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.983548+0000", 2024-09-13T11:26:43.747 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.6537999999999999e-05 2024-09-13T11:26:43.747 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.747 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.747 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:26:43.747 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.907005+0000", 2024-09-13T11:26:43.747 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.923456991 2024-09-13T11:26:43.747 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.747 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.747 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:26:43.747 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.907013+0000", 2024-09-13T11:26:43.747 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.0229999999999992e-06 2024-09-13T11:26:43.747 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:43.747 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:43.748 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:43.748 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.748 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.748 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:54760 3.1e 3.8c32c13e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-13T11:26:43.748 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:31.999685+0000", 2024-09-13T11:26:43.748 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 191.72689618800001, 2024-09-13T11:26:43.748 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.90735639599999995, 2024-09-13T11:26:43.748 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:26:43.748 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:26:43.748 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-13T11:26:43.748 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:26:43.748 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:26:43.748 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:43.749 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 54760 2024-09-13T11:26:43.749 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.749 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:26:43.749 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.749 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:26:43.749 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.999685+0000", 2024-09-13T11:26:43.749 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:43.749 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.749 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.749 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:26:43.749 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.999685+0000", 2024-09-13T11:26:43.749 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:43.749 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.750 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.750 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:26:43.750 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.999690+0000", 2024-09-13T11:26:43.750 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.7879999999999997e-06 2024-09-13T11:26:43.750 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.750 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.750 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:26:43.750 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.999729+0000", 2024-09-13T11:26:43.750 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.9032000000000002e-05 2024-09-13T11:26:43.750 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.750 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.750 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:26:43.750 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.999741+0000", 2024-09-13T11:26:43.750 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2136999999999999e-05 2024-09-13T11:26:43.751 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.751 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.751 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:26:43.751 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.999748+0000", 2024-09-13T11:26:43.751 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.9120000000000001e-06 2024-09-13T11:26:43.751 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.751 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.751 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:26:43.751 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.999803+0000", 2024-09-13T11:26:43.751 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.5064e-05 2024-09-13T11:26:43.751 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.751 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.751 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for peered", 2024-09-13T11:26:43.751 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.999815+0000", 2024-09-13T11:26:43.752 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2041e-05 2024-09-13T11:26:43.752 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.752 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.752 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:26:43.752 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.907034+0000", 2024-09-13T11:26:43.752 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.90721858200000005 2024-09-13T11:26:43.752 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.752 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.752 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:26:43.752 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.907042+0000", 2024-09-13T11:26:43.752 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.8399999999999995e-06 2024-09-13T11:26:43.752 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:43.752 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:43.752 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:43.753 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.753 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.753 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:54768 3.1e 3.6dad499e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-13T11:26:43.753 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:32.003482+0000", 2024-09-13T11:26:43.753 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 191.72309953800001, 2024-09-13T11:26:43.753 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.90358804699999995, 2024-09-13T11:26:43.753 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:26:43.753 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:26:43.753 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-13T11:26:43.753 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:26:43.753 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:26:43.753 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:43.753 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 54768 2024-09-13T11:26:43.753 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.754 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:26:43.754 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.754 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:26:43.754 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.003482+0000", 2024-09-13T11:26:43.754 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:43.754 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.754 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.754 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:26:43.754 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.003482+0000", 2024-09-13T11:26:43.754 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:43.754 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.754 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.754 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:26:43.754 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.003486+0000", 2024-09-13T11:26:43.755 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.1030000000000004e-06 2024-09-13T11:26:43.755 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.755 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.755 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:26:43.755 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.003523+0000", 2024-09-13T11:26:43.755 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.6732e-05 2024-09-13T11:26:43.755 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.755 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.755 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:26:43.755 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.003535+0000", 2024-09-13T11:26:43.755 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2520999999999999e-05 2024-09-13T11:26:43.755 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.755 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.755 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:26:43.755 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.003541+0000", 2024-09-13T11:26:43.756 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.8690000000000001e-06 2024-09-13T11:26:43.756 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.756 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.756 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:26:43.756 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.003593+0000", 2024-09-13T11:26:43.756 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.1786000000000002e-05 2024-09-13T11:26:43.756 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.756 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.756 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for peered", 2024-09-13T11:26:43.756 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.003602+0000", 2024-09-13T11:26:43.756 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.9679999999999995e-06 2024-09-13T11:26:43.756 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.756 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.757 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:26:43.757 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.907063+0000", 2024-09-13T11:26:43.757 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.90346116600000004 2024-09-13T11:26:43.757 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.757 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.757 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:26:43.757 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.907070+0000", 2024-09-13T11:26:43.757 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.9020000000000002e-06 2024-09-13T11:26:43.757 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:43.757 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:43.757 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:43.757 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.757 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.758 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:54785 3.1e 3.d32f1ede (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-13T11:26:43.758 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:32.016238+0000", 2024-09-13T11:26:43.758 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 191.71034332799999, 2024-09-13T11:26:43.758 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.89086243399999998, 2024-09-13T11:26:43.758 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:26:43.758 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:26:43.758 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-13T11:26:43.758 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:26:43.758 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:26:43.758 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:43.758 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 54785 2024-09-13T11:26:43.758 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.758 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:26:43.758 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.759 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:26:43.759 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.016238+0000", 2024-09-13T11:26:43.759 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:43.759 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.759 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.759 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:26:43.759 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.016238+0000", 2024-09-13T11:26:43.759 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:43.759 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.759 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.759 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:26:43.759 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.016241+0000", 2024-09-13T11:26:43.759 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.6469999999999998e-06 2024-09-13T11:26:43.759 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.759 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.760 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:26:43.760 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.016271+0000", 2024-09-13T11:26:43.760 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.0505999999999999e-05 2024-09-13T11:26:43.760 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.760 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.760 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:26:43.760 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.016287+0000", 2024-09-13T11:26:43.760 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.6073999999999999e-05 2024-09-13T11:26:43.760 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.760 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.760 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:26:43.760 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.016295+0000", 2024-09-13T11:26:43.760 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.0590000000000003e-06 2024-09-13T11:26:43.761 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.761 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.761 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:26:43.761 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.016373+0000", 2024-09-13T11:26:43.761 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.7634999999999998e-05 2024-09-13T11:26:43.761 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.761 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.761 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for peered", 2024-09-13T11:26:43.761 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.016383+0000", 2024-09-13T11:26:43.761 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0246e-05 2024-09-13T11:26:43.761 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.761 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.762 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:26:43.762 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.907092+0000", 2024-09-13T11:26:43.762 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.89070849200000002 2024-09-13T11:26:43.762 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.762 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.762 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:26:43.762 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.907101+0000", 2024-09-13T11:26:43.762 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.7749999999999998e-06 2024-09-13T11:26:43.762 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:43.762 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:43.762 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:43.762 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.762 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.762 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:54811 3.1e 3.3754367e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-13T11:26:43.763 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:32.045424+0000", 2024-09-13T11:26:43.763 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 191.68115767699999, 2024-09-13T11:26:43.763 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.86170719399999995, 2024-09-13T11:26:43.763 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:26:43.763 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:26:43.763 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-13T11:26:43.763 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:26:43.763 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:26:43.763 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:43.763 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 54811 2024-09-13T11:26:43.763 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.763 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:26:43.763 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.763 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:26:43.763 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.045424+0000", 2024-09-13T11:26:43.764 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:43.764 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.764 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.764 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:26:43.764 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.045424+0000", 2024-09-13T11:26:43.764 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:43.764 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.764 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.764 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:26:43.764 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.045426+0000", 2024-09-13T11:26:43.764 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.0260000000000002e-06 2024-09-13T11:26:43.764 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.764 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.764 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:26:43.764 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.045556+0000", 2024-09-13T11:26:43.765 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00013003099999999999 2024-09-13T11:26:43.765 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.765 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.765 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:26:43.765 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.045568+0000", 2024-09-13T11:26:43.765 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.1727e-05 2024-09-13T11:26:43.765 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.765 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.765 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:26:43.765 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.045576+0000", 2024-09-13T11:26:43.765 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.6459999999999999e-06 2024-09-13T11:26:43.765 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.765 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.765 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:26:43.766 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.045650+0000", 2024-09-13T11:26:43.766 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.4000999999999998e-05 2024-09-13T11:26:43.766 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.766 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.766 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for peered", 2024-09-13T11:26:43.766 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.045658+0000", 2024-09-13T11:26:43.766 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.3869999999999997e-06 2024-09-13T11:26:43.766 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.766 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.766 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:26:43.766 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.907123+0000", 2024-09-13T11:26:43.766 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.86146508399999999 2024-09-13T11:26:43.766 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.767 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.767 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:26:43.767 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.907131+0000", 2024-09-13T11:26:43.767 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.2919999999999996e-06 2024-09-13T11:26:43.767 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:43.767 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:43.767 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:43.767 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.767 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.767 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:54817 3.1e 3.36dd2fde (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-13T11:26:43.767 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:32.050108+0000", 2024-09-13T11:26:43.767 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 191.67647378500001, 2024-09-13T11:26:43.767 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.85705549199999997, 2024-09-13T11:26:43.768 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:26:43.768 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:26:43.768 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-13T11:26:43.768 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:26:43.768 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:26:43.768 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:43.768 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 54817 2024-09-13T11:26:43.768 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.768 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:26:43.768 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.768 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:26:43.768 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.050108+0000", 2024-09-13T11:26:43.768 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:43.768 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.769 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.769 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:26:43.769 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.050108+0000", 2024-09-13T11:26:43.769 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:43.769 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.769 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.769 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:26:43.769 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.050110+0000", 2024-09-13T11:26:43.769 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.0209999999999998e-06 2024-09-13T11:26:43.769 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.769 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.769 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:26:43.769 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.050166+0000", 2024-09-13T11:26:43.769 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.6026000000000002e-05 2024-09-13T11:26:43.770 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.770 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.770 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:26:43.770 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.050180+0000", 2024-09-13T11:26:43.770 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.4307e-05 2024-09-13T11:26:43.770 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.770 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.770 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:26:43.770 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.050188+0000", 2024-09-13T11:26:43.770 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.5410000000000003e-06 2024-09-13T11:26:43.770 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.770 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.770 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:26:43.770 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.050260+0000", 2024-09-13T11:26:43.771 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.2157999999999996e-05 2024-09-13T11:26:43.771 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.771 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.771 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for peered", 2024-09-13T11:26:43.771 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.050267+0000", 2024-09-13T11:26:43.771 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.0190000000000001e-06 2024-09-13T11:26:43.771 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.771 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.771 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:26:43.771 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.907155+0000", 2024-09-13T11:26:43.771 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.85688824399999997 2024-09-13T11:26:43.771 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.771 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.771 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:26:43.771 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.907163+0000", 2024-09-13T11:26:43.772 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.1759999999999993e-06 2024-09-13T11:26:43.772 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:43.772 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:43.772 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:43.772 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.772 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.772 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:54818 3.1e 3.5cfcc69e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-13T11:26:43.772 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:32.051374+0000", 2024-09-13T11:26:43.772 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 191.67520736899999, 2024-09-13T11:26:43.772 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.85581781000000001, 2024-09-13T11:26:43.772 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:26:43.772 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:26:43.772 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-13T11:26:43.772 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:26:43.773 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:26:43.773 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:43.773 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 54818 2024-09-13T11:26:43.773 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.773 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:26:43.773 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.773 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:26:43.773 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.051374+0000", 2024-09-13T11:26:43.773 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:43.773 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.773 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.773 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:26:43.773 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.051374+0000", 2024-09-13T11:26:43.773 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:43.774 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.774 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.774 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:26:43.774 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.051375+0000", 2024-09-13T11:26:43.774 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.4440000000000001e-06 2024-09-13T11:26:43.774 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.774 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.774 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:26:43.774 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.051426+0000", 2024-09-13T11:26:43.774 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.0809000000000001e-05 2024-09-13T11:26:43.774 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.774 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.774 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:26:43.775 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.051441+0000", 2024-09-13T11:26:43.775 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.4460999999999999e-05 2024-09-13T11:26:43.775 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.775 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.775 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:26:43.775 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.051449+0000", 2024-09-13T11:26:43.775 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.4880000000000007e-06 2024-09-13T11:26:43.775 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.775 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.775 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:26:43.775 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.051499+0000", 2024-09-13T11:26:43.775 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.9691999999999998e-05 2024-09-13T11:26:43.775 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.775 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.776 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for peered", 2024-09-13T11:26:43.776 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.051505+0000", 2024-09-13T11:26:43.776 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.2029999999999997e-06 2024-09-13T11:26:43.776 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.776 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.776 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:26:43.776 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.907184+0000", 2024-09-13T11:26:43.776 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.85567906599999999 2024-09-13T11:26:43.776 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.776 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.776 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:26:43.776 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.907192+0000", 2024-09-13T11:26:43.776 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.6469999999999998e-06 2024-09-13T11:26:43.776 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:43.777 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:43.777 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:43.777 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.777 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.777 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:54828 3.1e 3.e920c11e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-13T11:26:43.777 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:32.063344+0000", 2024-09-13T11:26:43.777 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 191.663236942, 2024-09-13T11:26:43.777 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.84387899099999997, 2024-09-13T11:26:43.777 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:26:43.777 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:26:43.777 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-13T11:26:43.777 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:26:43.777 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:26:43.777 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:43.778 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 54828 2024-09-13T11:26:43.778 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.778 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:26:43.778 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.778 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:26:43.778 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.063344+0000", 2024-09-13T11:26:43.778 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:43.778 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.778 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.778 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:26:43.778 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.063344+0000", 2024-09-13T11:26:43.778 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:43.778 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.779 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.779 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:26:43.779 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.063347+0000", 2024-09-13T11:26:43.779 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.2299999999999998e-06 2024-09-13T11:26:43.779 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.779 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.779 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:26:43.779 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.063384+0000", 2024-09-13T11:26:43.779 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.7290999999999999e-05 2024-09-13T11:26:43.779 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.779 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.779 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:26:43.779 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.063397+0000", 2024-09-13T11:26:43.779 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2955999999999999e-05 2024-09-13T11:26:43.779 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.780 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.780 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:26:43.780 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.063404+0000", 2024-09-13T11:26:43.780 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.4889999999999998e-06 2024-09-13T11:26:43.780 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.780 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.780 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:26:43.780 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.063470+0000", 2024-09-13T11:26:43.780 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.5516000000000001e-05 2024-09-13T11:26:43.780 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.780 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.780 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for peered", 2024-09-13T11:26:43.780 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.063482+0000", 2024-09-13T11:26:43.781 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2065000000000001e-05 2024-09-13T11:26:43.781 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.781 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.781 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:26:43.781 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.907216+0000", 2024-09-13T11:26:43.781 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.84373388900000001 2024-09-13T11:26:43.781 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.781 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.781 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:26:43.781 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.907223+0000", 2024-09-13T11:26:43.781 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.5549999999999997e-06 2024-09-13T11:26:43.781 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:43.781 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:43.782 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:43.782 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.782 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.782 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:54829 3.1e 3.192f62be (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-13T11:26:43.782 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:32.065642+0000", 2024-09-13T11:26:43.782 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 191.66093987400001, 2024-09-13T11:26:43.782 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.84161483100000001, 2024-09-13T11:26:43.782 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:26:43.782 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:26:43.782 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-13T11:26:43.782 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:26:43.782 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:26:43.782 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:43.782 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 54829 2024-09-13T11:26:43.783 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.783 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:26:43.783 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.783 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:26:43.783 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.065642+0000", 2024-09-13T11:26:43.783 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:43.783 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.783 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.783 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:26:43.783 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.065642+0000", 2024-09-13T11:26:43.783 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:43.783 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.783 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.783 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:26:43.783 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.065644+0000", 2024-09-13T11:26:43.784 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.1780000000000002e-06 2024-09-13T11:26:43.784 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.784 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.784 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:26:43.784 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.065695+0000", 2024-09-13T11:26:43.784 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.1632000000000003e-05 2024-09-13T11:26:43.784 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.784 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.784 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:26:43.784 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.065711+0000", 2024-09-13T11:26:43.784 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.5522000000000001e-05 2024-09-13T11:26:43.784 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.784 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.784 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:26:43.785 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.065719+0000", 2024-09-13T11:26:43.785 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.3480000000000004e-06 2024-09-13T11:26:43.785 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.785 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.785 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:26:43.785 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.065785+0000", 2024-09-13T11:26:43.785 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.5627e-05 2024-09-13T11:26:43.785 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.785 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.785 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for peered", 2024-09-13T11:26:43.785 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.065797+0000", 2024-09-13T11:26:43.785 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.1766000000000001e-05 2024-09-13T11:26:43.785 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.785 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.786 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:26:43.786 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.907248+0000", 2024-09-13T11:26:43.786 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.84145150300000005 2024-09-13T11:26:43.786 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.786 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.786 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:26:43.786 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.907256+0000", 2024-09-13T11:26:43.786 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.2549999999999997e-06 2024-09-13T11:26:43.786 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:43.786 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:43.786 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:43.786 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.786 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.787 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:54830 3.1e 3.b31554de (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-13T11:26:43.787 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:32.068259+0000", 2024-09-13T11:26:43.787 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 191.65832200599999, 2024-09-13T11:26:43.787 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.83902489599999996, 2024-09-13T11:26:43.787 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:26:43.787 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:26:43.787 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-13T11:26:43.787 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:26:43.787 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:26:43.787 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:43.787 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 54830 2024-09-13T11:26:43.787 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.787 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:26:43.787 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.787 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:26:43.787 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.068259+0000", 2024-09-13T11:26:43.788 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:43.788 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.788 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.788 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:26:43.788 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.068259+0000", 2024-09-13T11:26:43.788 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:43.788 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.788 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.788 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:26:43.788 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.068262+0000", 2024-09-13T11:26:43.788 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.1799999999999999e-06 2024-09-13T11:26:43.788 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.788 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.789 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:26:43.789 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.068301+0000", 2024-09-13T11:26:43.789 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.9289999999999998e-05 2024-09-13T11:26:43.789 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.789 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.789 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:26:43.789 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.068314+0000", 2024-09-13T11:26:43.789 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.3502999999999999e-05 2024-09-13T11:26:43.789 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.789 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.789 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:26:43.789 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.068324+0000", 2024-09-13T11:26:43.789 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.1630000000000002e-06 2024-09-13T11:26:43.789 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.790 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.790 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:26:43.790 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.068375+0000", 2024-09-13T11:26:43.790 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.1805000000000003e-05 2024-09-13T11:26:43.790 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.790 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.790 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for peered", 2024-09-13T11:26:43.790 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.068382+0000", 2024-09-13T11:26:43.790 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.224e-06 2024-09-13T11:26:43.790 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.790 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.790 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:26:43.790 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.907277+0000", 2024-09-13T11:26:43.790 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.83889509500000004 2024-09-13T11:26:43.790 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.791 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.791 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:26:43.791 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.907284+0000", 2024-09-13T11:26:43.791 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.6359999999999994e-06 2024-09-13T11:26:43.791 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:43.791 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:43.791 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:43.791 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.791 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.791 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:54846 3.1e 3.742e049e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-13T11:26:43.791 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:32.153567+0000", 2024-09-13T11:26:43.791 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 191.573014782, 2024-09-13T11:26:43.791 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.75378642399999995, 2024-09-13T11:26:43.791 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:26:43.791 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:26:43.792 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-13T11:26:43.792 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:26:43.792 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:26:43.792 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:43.792 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 54846 2024-09-13T11:26:43.792 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.792 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:26:43.792 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.792 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:26:43.792 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.153567+0000", 2024-09-13T11:26:43.792 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:43.792 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.792 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.792 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:26:43.792 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.153567+0000", 2024-09-13T11:26:43.793 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:43.793 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.793 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.793 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:26:43.793 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.153569+0000", 2024-09-13T11:26:43.793 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.4660000000000002e-06 2024-09-13T11:26:43.793 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.793 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.793 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:26:43.793 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.153619+0000", 2024-09-13T11:26:43.793 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.9617000000000003e-05 2024-09-13T11:26:43.793 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.793 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.793 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:26:43.794 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.153634+0000", 2024-09-13T11:26:43.794 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.4904e-05 2024-09-13T11:26:43.794 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.794 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.794 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:26:43.794 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.153642+0000", 2024-09-13T11:26:43.794 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.5599999999999994e-06 2024-09-13T11:26:43.794 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.794 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.794 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:26:43.794 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.153721+0000", 2024-09-13T11:26:43.794 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.8603999999999995e-05 2024-09-13T11:26:43.794 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.795 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.795 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for peered", 2024-09-13T11:26:43.795 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.153730+0000", 2024-09-13T11:26:43.795 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.3840000000000003e-06 2024-09-13T11:26:43.795 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.795 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.795 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:26:43.795 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.907346+0000", 2024-09-13T11:26:43.795 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.75361563099999995 2024-09-13T11:26:43.795 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.795 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.795 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:26:43.795 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.907353+0000", 2024-09-13T11:26:43.795 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.2579999999999998e-06 2024-09-13T11:26:43.795 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:43.796 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:43.796 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:43.796 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.796 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.796 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:54888 3.1e 3.43e14ede (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-13T11:26:43.796 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:32.423064+0000", 2024-09-13T11:26:43.796 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 191.30351711099999, 2024-09-13T11:26:43.796 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.48433585299999998, 2024-09-13T11:26:43.796 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:26:43.796 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:26:43.796 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-13T11:26:43.796 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:26:43.796 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:26:43.797 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:43.797 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 54888 2024-09-13T11:26:43.797 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.797 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:26:43.797 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.797 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:26:43.797 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.423064+0000", 2024-09-13T11:26:43.797 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:43.797 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.797 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.797 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:26:43.797 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.423064+0000", 2024-09-13T11:26:43.797 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:43.797 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.798 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.798 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:26:43.798 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.423069+0000", 2024-09-13T11:26:43.798 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.7389999999999999e-06 2024-09-13T11:26:43.798 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.798 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.798 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:26:43.798 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.423115+0000", 2024-09-13T11:26:43.798 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.5847000000000003e-05 2024-09-13T11:26:43.798 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.799 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.799 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:26:43.799 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.423126+0000", 2024-09-13T11:26:43.799 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.1418e-05 2024-09-13T11:26:43.799 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.799 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.799 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:26:43.799 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.423132+0000", 2024-09-13T11:26:43.799 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.57e-06 2024-09-13T11:26:43.799 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.799 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.799 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:26:43.799 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.423191+0000", 2024-09-13T11:26:43.800 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.9250999999999999e-05 2024-09-13T11:26:43.800 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.800 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.800 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for peered", 2024-09-13T11:26:43.800 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.423198+0000", 2024-09-13T11:26:43.800 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.9179999999999997e-06 2024-09-13T11:26:43.800 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.800 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.800 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:26:43.800 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.907392+0000", 2024-09-13T11:26:43.800 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.48419437500000001 2024-09-13T11:26:43.800 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.800 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.800 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:26:43.801 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.907400+0000", 2024-09-13T11:26:43.801 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.7349999999999996e-06 2024-09-13T11:26:43.801 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:43.801 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:43.801 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:43.801 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.801 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.801 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4665.0:59977 3.21 e83/78 3:86b4d8f6:::benchmark_data_smithi177_31116_object59976:head v 83'3136, mlcod=0'0)", 2024-09-13T11:26:43.801 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:40.571147+0000", 2024-09-13T11:26:43.801 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 183.155434662, 2024-09-13T11:26:43.801 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.51907542799999995, 2024-09-13T11:26:43.801 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:26:43.801 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:26:43.801 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:43.802 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:26:43.802 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.802 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:26:43.802 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:40.571147+0000", 2024-09-13T11:26:43.802 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:43.802 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.802 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.802 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:26:43.802 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:40.571147+0000", 2024-09-13T11:26:43.802 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:43.802 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.802 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.802 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:26:43.802 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:40.571151+0000", 2024-09-13T11:26:43.803 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.5650000000000003e-06 2024-09-13T11:26:43.803 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.803 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.803 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:26:43.803 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:40.571204+0000", 2024-09-13T11:26:43.803 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.2720999999999999e-05 2024-09-13T11:26:43.803 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.803 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.803 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:26:43.803 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:41.088332+0000", 2024-09-13T11:26:43.803 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.51712822000000003 2024-09-13T11:26:43.803 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.803 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.803 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:26:43.804 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:41.088341+0000", 2024-09-13T11:26:43.804 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.3100000000000001e-06 2024-09-13T11:26:43.804 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.804 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.804 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:26:43.804 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:41.088444+0000", 2024-09-13T11:26:43.804 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000103708 2024-09-13T11:26:43.804 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.804 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.804 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:26:43.804 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:41.088475+0000", 2024-09-13T11:26:43.804 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.0497e-05 2024-09-13T11:26:43.804 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.804 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.805 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:26:43.805 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:41.090192+0000", 2024-09-13T11:26:43.805 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.001717288 2024-09-13T11:26:43.805 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.805 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.805 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:26:43.805 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:41.090222+0000", 2024-09-13T11:26:43.805 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.0119000000000001e-05 2024-09-13T11:26:43.805 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:43.805 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:43.805 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:43.805 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.805 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.806 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:110286 3.26 3.c41c0fa6 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e119)", 2024-09-13T11:26:43.806 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:24:25.654272+0000", 2024-09-13T11:26:43.806 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 138.07230896999999, 2024-09-13T11:26:43.806 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.43337235400000002, 2024-09-13T11:26:43.806 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:26:43.806 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:26:43.806 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-13T11:26:43.806 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:26:43.806 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:26:43.806 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:43.806 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 110286 2024-09-13T11:26:43.806 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.806 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:26:43.806 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.807 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:26:43.807 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:25.654272+0000", 2024-09-13T11:26:43.807 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:43.807 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.807 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.807 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:26:43.807 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:25.654272+0000", 2024-09-13T11:26:43.807 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:43.807 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.807 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.807 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:26:43.807 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:25.654275+0000", 2024-09-13T11:26:43.807 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.3199999999999998e-06 2024-09-13T11:26:43.807 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.808 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.808 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:26:43.808 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:25.654517+0000", 2024-09-13T11:26:43.808 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00024270199999999999 2024-09-13T11:26:43.808 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.808 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.808 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:26:43.808 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:25.654550+0000", 2024-09-13T11:26:43.808 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.2549000000000003e-05 2024-09-13T11:26:43.808 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.808 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.808 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:26:43.808 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:25.654566+0000", 2024-09-13T11:26:43.808 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.5517000000000001e-05 2024-09-13T11:26:43.809 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.809 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.809 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:26:43.809 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:25.654655+0000", 2024-09-13T11:26:43.809 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.9961000000000001e-05 2024-09-13T11:26:43.809 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.809 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.809 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for peered", 2024-09-13T11:26:43.809 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:25.654665+0000", 2024-09-13T11:26:43.809 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.0520000000000007e-06 2024-09-13T11:26:43.809 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.809 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.809 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:26:43.810 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:26.087629+0000", 2024-09-13T11:26:43.810 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.43296476900000003 2024-09-13T11:26:43.810 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.810 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.810 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:26:43.810 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:26.087645+0000", 2024-09-13T11:26:43.810 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.5483999999999999e-05 2024-09-13T11:26:43.810 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:43.810 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:43.810 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:43.810 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.810 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.810 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:177913 3.27 3:e5268e7a:::benchmark_data_smithi177_31116_object37610:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e168)", 2024-09-13T11:26:43.810 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:25:27.607015+0000", 2024-09-13T11:26:43.810 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 76.119566620000001, 2024-09-13T11:26:43.811 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.301660555, 2024-09-13T11:26:43.811 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:26:43.811 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:26:43.811 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:43.811 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:26:43.811 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:26:43.811 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:43.811 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 177913 2024-09-13T11:26:43.811 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.811 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:26:43.811 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.811 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:26:43.811 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:27.607015+0000", 2024-09-13T11:26:43.811 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:43.812 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.812 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.812 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:26:43.812 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:27.607015+0000", 2024-09-13T11:26:43.812 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:43.812 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.812 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.812 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:26:43.812 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:27.607018+0000", 2024-09-13T11:26:43.812 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.7000000000000002e-06 2024-09-13T11:26:43.812 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.812 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.812 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:26:43.812 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:27.607029+0000", 2024-09-13T11:26:43.813 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0933e-05 2024-09-13T11:26:43.813 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.813 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.813 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:26:43.813 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:27.607062+0000", 2024-09-13T11:26:43.813 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.2786000000000001e-05 2024-09-13T11:26:43.813 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.813 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.813 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:26:43.813 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:27.607078+0000", 2024-09-13T11:26:43.813 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.6232999999999999e-05 2024-09-13T11:26:43.813 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.813 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.814 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:26:43.814 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:27.607204+0000", 2024-09-13T11:26:43.814 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000125458 2024-09-13T11:26:43.814 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.814 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.814 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:26:43.814 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:27.607544+0000", 2024-09-13T11:26:43.814 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00033990600000000001 2024-09-13T11:26:43.814 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.814 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.814 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 5,11", 2024-09-13T11:26:43.814 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:27.607882+0000", 2024-09-13T11:26:43.814 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00033788399999999999 2024-09-13T11:26:43.815 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.815 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.815 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:26:43.815 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:27.610120+0000", 2024-09-13T11:26:43.815 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0022379930000000002 2024-09-13T11:26:43.815 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.815 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.815 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:43.815 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:27.613170+0000", 2024-09-13T11:26:43.815 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.003050545 2024-09-13T11:26:43.815 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.815 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.815 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:43.815 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:27.908398+0000", 2024-09-13T11:26:43.816 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.29522760599999998 2024-09-13T11:26:43.816 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.816 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.816 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:26:43.816 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:27.908458+0000", 2024-09-13T11:26:43.816 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.9718e-05 2024-09-13T11:26:43.816 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.816 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.816 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:26:43.816 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:27.908675+0000", 2024-09-13T11:26:43.816 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00021779300000000001 2024-09-13T11:26:43.816 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:43.816 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:43.816 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:43.816 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.817 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.817 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop_reply(client.4665.0:177913 3.27 e168/120 ondisk, result = 0)", 2024-09-13T11:26:43.817 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:25:27.610053+0000", 2024-09-13T11:26:43.817 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 76.116528461000001, 2024-09-13T11:26:43.817 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.29862817200000003, 2024-09-13T11:26:43.817 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:26:43.817 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:26:43.817 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:26:43.817 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:26:43.817 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.817 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:26:43.817 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:27.610053+0000", 2024-09-13T11:26:43.817 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:43.818 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.818 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.818 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:26:43.818 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:27.610053+0000", 2024-09-13T11:26:43.818 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:43.818 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.818 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.818 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:26:43.818 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:27.610056+0000", 2024-09-13T11:26:43.818 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.49e-06 2024-09-13T11:26:43.818 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.818 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.818 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:26:43.818 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:27.610067+0000", 2024-09-13T11:26:43.819 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0599e-05 2024-09-13T11:26:43.819 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.819 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.819 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:26:43.819 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:27.908314+0000", 2024-09-13T11:26:43.819 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.29824728700000003 2024-09-13T11:26:43.819 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.819 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.819 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:26:43.819 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:27.908320+0000", 2024-09-13T11:26:43.819 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.9100000000000002e-06 2024-09-13T11:26:43.819 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.819 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.819 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:26:43.819 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:27.908378+0000", 2024-09-13T11:26:43.820 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.7575999999999999e-05 2024-09-13T11:26:43.820 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.820 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.820 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:26:43.820 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:27.908390+0000", 2024-09-13T11:26:43.820 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.187e-05 2024-09-13T11:26:43.820 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.820 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.820 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:26:43.820 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:27.908681+0000", 2024-09-13T11:26:43.820 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00029144000000000002 2024-09-13T11:26:43.820 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:43.820 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:43.820 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:43.821 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.821 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.821 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDPGPushReply(3.27 168/120 [PushReplyOp(3:e535cc18:::benchmark_data_smithi177_31116_object103295:head)])", 2024-09-13T11:26:43.821 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:25:27.615014+0000", 2024-09-13T11:26:43.821 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 76.111567127000001, 2024-09-13T11:26:43.821 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.31810488799999997, 2024-09-13T11:26:43.821 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:26:43.821 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:26:43.821 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-13T11:26:43.821 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:26:43.821 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.821 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:26:43.821 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:27.615014+0000", 2024-09-13T11:26:43.821 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:43.821 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.822 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.822 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:26:43.822 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:27.615014+0000", 2024-09-13T11:26:43.822 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:43.822 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.822 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.822 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:26:43.822 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:27.615017+0000", 2024-09-13T11:26:43.822 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.6199999999999999e-06 2024-09-13T11:26:43.822 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.822 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.822 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:26:43.822 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:27.615030+0000", 2024-09-13T11:26:43.823 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2833000000000001e-05 2024-09-13T11:26:43.823 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.823 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.823 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:26:43.823 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:27.932588+0000", 2024-09-13T11:26:43.823 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.31755857700000001 2024-09-13T11:26:43.823 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.823 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.823 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:26:43.823 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:27.932595+0000", 2024-09-13T11:26:43.823 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.2160000000000003e-06 2024-09-13T11:26:43.823 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.823 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.824 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:26:43.824 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:27.932658+0000", 2024-09-13T11:26:43.824 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.3618e-05 2024-09-13T11:26:43.824 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.824 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.824 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:26:43.824 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:27.933119+0000", 2024-09-13T11:26:43.824 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00046102399999999998 2024-09-13T11:26:43.824 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:43.824 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:43.824 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:43.824 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.824 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.824 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:233663 3.26 3:67af7a71:::benchmark_data_smithi177_31116_object93360:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e200)", 2024-09-13T11:26:43.824 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:26:16.613407+0000", 2024-09-13T11:26:43.824 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 27.113174146999999, 2024-09-13T11:26:43.825 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.32259047600000001, 2024-09-13T11:26:43.825 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:26:43.825 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:26:43.825 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:43.825 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:26:43.825 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:26:43.825 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:43.825 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 233663 2024-09-13T11:26:43.825 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.825 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:26:43.825 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.825 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:26:43.825 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:16.613407+0000", 2024-09-13T11:26:43.825 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:43.826 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.826 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.826 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:26:43.826 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:16.613407+0000", 2024-09-13T11:26:43.826 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:43.826 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.826 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.826 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:26:43.826 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:16.613409+0000", 2024-09-13T11:26:43.826 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.8220000000000001e-06 2024-09-13T11:26:43.826 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.826 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.826 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:26:43.826 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:16.613414+0000", 2024-09-13T11:26:43.827 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.8049999999999997e-06 2024-09-13T11:26:43.827 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.827 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.827 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:26:43.827 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:16.613426+0000", 2024-09-13T11:26:43.827 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2319e-05 2024-09-13T11:26:43.827 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.827 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.827 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:26:43.827 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:16.613432+0000", 2024-09-13T11:26:43.827 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.604e-06 2024-09-13T11:26:43.827 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.827 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.827 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:26:43.828 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:16.613490+0000", 2024-09-13T11:26:43.828 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.7742999999999999e-05 2024-09-13T11:26:43.828 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.828 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.828 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:26:43.828 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:16.613614+0000", 2024-09-13T11:26:43.828 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00012410899999999999 2024-09-13T11:26:43.828 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.828 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.828 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 12", 2024-09-13T11:26:43.828 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:16.613672+0000", 2024-09-13T11:26:43.828 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.8053999999999997e-05 2024-09-13T11:26:43.828 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.828 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.829 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:26:43.829 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:16.615571+0000", 2024-09-13T11:26:43.829 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0018996830000000001 2024-09-13T11:26:43.829 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.829 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.829 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:43.829 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:16.935742+0000", 2024-09-13T11:26:43.829 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.32017025599999999 2024-09-13T11:26:43.829 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.829 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.829 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:26:43.829 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:16.935888+0000", 2024-09-13T11:26:43.829 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000146716 2024-09-13T11:26:43.829 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.830 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.830 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:26:43.830 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:16.935998+0000", 2024-09-13T11:26:43.830 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00010936499999999999 2024-09-13T11:26:43.830 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:43.830 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:43.830 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:43.830 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:43.830 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:43.830 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-13T11:26:43.831 INFO:tasks.ceph.osd.10.smithi162.stderr:2024-09-13T11:26:43.827+0000 7fb59c9f0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26317) UID: 0 2024-09-13T11:26:43.832 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_ops_in_flight 2024-09-13T11:26:43.933 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-13T11:26:43.930+0000 7f6a68775640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26307) UID: 0 2024-09-13T11:26:43.979 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-13T11:26:43.979 INFO:teuthology.orchestra.run.smithi001.stdout: "ops": [ 2024-09-13T11:26:43.979 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.979 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:261687 3.25 3:a765ee71:::benchmark_data_smithi177_31116_object121384:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e220)", 2024-09-13T11:26:43.979 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:26:43.972912+0000", 2024-09-13T11:26:43.979 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 0.0060100140000000002, 2024-09-13T11:26:43.979 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0060371189999999996, 2024-09-13T11:26:43.980 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:26:43.980 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:26:43.980 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "waiting for sub ops", 2024-09-13T11:26:43.980 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:26:43.980 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:26:43.980 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:43.980 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 261687 2024-09-13T11:26:43.980 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.980 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:26:43.980 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.980 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:26:43.980 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:43.972912+0000", 2024-09-13T11:26:43.980 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:43.981 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.981 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.981 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:26:43.981 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:43.972912+0000", 2024-09-13T11:26:43.981 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:43.981 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.981 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.981 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:26:43.981 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:43.972913+0000", 2024-09-13T11:26:43.981 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.9800000000000002e-07 2024-09-13T11:26:43.981 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.981 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.981 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:26:43.981 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:43.972917+0000", 2024-09-13T11:26:43.982 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.5159999999999999e-06 2024-09-13T11:26:43.982 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.982 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.982 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:26:43.982 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:43.972926+0000", 2024-09-13T11:26:43.982 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.1570000000000006e-06 2024-09-13T11:26:43.982 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.982 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.982 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:26:43.982 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:43.972931+0000", 2024-09-13T11:26:43.982 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.2800000000000003e-06 2024-09-13T11:26:43.982 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.982 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.982 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:26:43.983 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:43.972988+0000", 2024-09-13T11:26:43.983 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.6507e-05 2024-09-13T11:26:43.983 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.983 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.983 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:26:43.983 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:43.973075+0000", 2024-09-13T11:26:43.983 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.7826999999999998e-05 2024-09-13T11:26:43.983 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.983 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.983 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 2", 2024-09-13T11:26:43.983 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:43.973140+0000", 2024-09-13T11:26:43.983 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.4338000000000006e-05 2024-09-13T11:26:43.983 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.983 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.984 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:26:43.984 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:43.974780+0000", 2024-09-13T11:26:43.984 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.001639986 2024-09-13T11:26:43.984 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:43.984 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:43.984 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:43.984 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.984 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.984 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:261693 3.9 3:90555f77:::benchmark_data_smithi177_31116_object121390:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e220)", 2024-09-13T11:26:43.984 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:26:43.977070+0000", 2024-09-13T11:26:43.984 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 0.0018516209999999999, 2024-09-13T11:26:43.984 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0019585840000000002, 2024-09-13T11:26:43.984 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:26:43.985 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:26:43.985 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "waiting for sub ops", 2024-09-13T11:26:43.985 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:26:43.985 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:26:43.985 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:43.985 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 261693 2024-09-13T11:26:43.985 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.985 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:26:43.985 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.985 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:26:43.985 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:43.977070+0000", 2024-09-13T11:26:43.985 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:43.985 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.986 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.986 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:26:43.986 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:43.977070+0000", 2024-09-13T11:26:43.986 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:43.986 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.986 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.986 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:26:43.986 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:43.977072+0000", 2024-09-13T11:26:43.986 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2440000000000001e-06 2024-09-13T11:26:43.986 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.986 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.986 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:26:43.986 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:43.977076+0000", 2024-09-13T11:26:43.987 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.4630000000000003e-06 2024-09-13T11:26:43.987 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.987 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.987 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:26:43.987 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:43.977086+0000", 2024-09-13T11:26:43.987 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.6120000000000005e-06 2024-09-13T11:26:43.987 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.987 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.987 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:26:43.987 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:43.977094+0000", 2024-09-13T11:26:43.987 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.3869999999999995e-06 2024-09-13T11:26:43.987 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.987 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.987 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:26:43.988 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:43.977321+0000", 2024-09-13T11:26:43.988 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000226412 2024-09-13T11:26:43.988 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.988 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.988 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:26:43.988 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:43.977453+0000", 2024-09-13T11:26:43.988 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00013239099999999999 2024-09-13T11:26:43.988 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.988 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.988 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 2", 2024-09-13T11:26:43.988 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:43.977561+0000", 2024-09-13T11:26:43.988 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000108001 2024-09-13T11:26:43.988 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:43.988 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:43.989 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:43.989 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.989 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.989 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:261694 3.20 3:05670da8:::benchmark_data_smithi177_31116_object121391:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e220)", 2024-09-13T11:26:43.989 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:26:43.977240+0000", 2024-09-13T11:26:43.989 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 0.0016818650000000001, 2024-09-13T11:26:43.989 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00185626, 2024-09-13T11:26:43.989 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:26:43.989 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:26:43.989 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "waiting for sub ops", 2024-09-13T11:26:43.989 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:26:43.989 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:26:43.989 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:43.989 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 261694 2024-09-13T11:26:43.990 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.990 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:26:43.990 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.990 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:26:43.990 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:43.977240+0000", 2024-09-13T11:26:43.990 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:43.990 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.990 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.990 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:26:43.990 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:43.977240+0000", 2024-09-13T11:26:43.990 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:43.990 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.990 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.990 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:26:43.991 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:43.977241+0000", 2024-09-13T11:26:43.991 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.238e-06 2024-09-13T11:26:43.991 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.991 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.991 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:26:43.991 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:43.977246+0000", 2024-09-13T11:26:43.991 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.1060000000000002e-06 2024-09-13T11:26:43.991 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.991 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.991 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:26:43.991 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:43.977255+0000", 2024-09-13T11:26:43.991 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.9910000000000008e-06 2024-09-13T11:26:43.991 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.991 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.991 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:26:43.992 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:43.977262+0000", 2024-09-13T11:26:43.992 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.3259999999999998e-06 2024-09-13T11:26:43.992 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.992 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.992 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:26:43.992 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:43.977324+0000", 2024-09-13T11:26:43.992 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.2508000000000005e-05 2024-09-13T11:26:43.992 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.992 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.992 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:26:43.992 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:43.977455+0000", 2024-09-13T11:26:43.992 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00013099699999999999 2024-09-13T11:26:43.992 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:43.992 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:43.993 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 0", 2024-09-13T11:26:43.993 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:43.977563+0000", 2024-09-13T11:26:43.993 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000107744 2024-09-13T11:26:43.993 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:43.993 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:43.993 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:43.993 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:43.993 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-09-13T11:26:43.993 INFO:teuthology.orchestra.run.smithi001.stdout: "num_ops": 3 2024-09-13T11:26:43.993 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-13T11:26:43.999 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_blocked_ops 2024-09-13T11:26:44.034 INFO:tasks.ceph.osd.1.smithi092.stderr:2024-09-13T11:26:44.031+0000 7fe7789c5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33718) UID: 0 2024-09-13T11:26:44.134 INFO:tasks.ceph.osd.11.smithi177.stderr:2024-09-13T11:26:44.133+0000 7f24792c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26580) UID: 0 2024-09-13T11:26:44.188 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-13T11:26:44.188 INFO:teuthology.orchestra.run.smithi001.stdout: "ops": [], 2024-09-13T11:26:44.188 INFO:teuthology.orchestra.run.smithi001.stdout: "complaint_time": 30, 2024-09-13T11:26:44.189 INFO:teuthology.orchestra.run.smithi001.stdout: "num_blocked_ops": 0 2024-09-13T11:26:44.189 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-13T11:26:44.208 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_historic_ops 2024-09-13T11:26:44.235 INFO:tasks.ceph.osd.10.smithi162.stderr:2024-09-13T11:26:44.231+0000 7fb59c9f0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26317) UID: 0 2024-09-13T11:26:44.280 INFO:teuthology.orchestra.run.smithi001.stderr:noscrub is unset 2024-09-13T11:26:44.282 INFO:tasks.ceph.osd.8.smithi001.stderr:2024-09-13T11:26:44.278+0000 7fb9ab2df640 -1 osd.8 220 handle_osd_map injecting map crc failure 2024-09-13T11:26:44.336 INFO:tasks.ceph.osd.13.smithi092.stderr:2024-09-13T11:26:44.331+0000 7f3e5afea640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26350) UID: 0 2024-09-13T11:26:44.349 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-13T11:26:44.350 INFO:teuthology.orchestra.run.smithi001.stdout: "size": 20, 2024-09-13T11:26:44.350 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 600, 2024-09-13T11:26:44.350 INFO:teuthology.orchestra.run.smithi001.stdout: "ops": [ 2024-09-13T11:26:44.350 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:44.350 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop_reply(client.4665.0:167 3.9 e41/30 ondisk, result = 0)", 2024-09-13T11:26:44.350 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:22:47.419696+0000", 2024-09-13T11:26:44.350 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 236.92842946600001, 2024-09-13T11:26:44.350 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.64615778199999996, 2024-09-13T11:26:44.350 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:26:44.350 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:26:44.350 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:26:44.350 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:26:44.350 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:44.351 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:26:44.351 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:47.419696+0000", 2024-09-13T11:26:44.351 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:44.351 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:44.351 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:44.351 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:26:44.351 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:47.419696+0000", 2024-09-13T11:26:44.351 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:44.351 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:44.351 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:44.351 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:26:44.351 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:47.419697+0000", 2024-09-13T11:26:44.351 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.15e-06 2024-09-13T11:26:44.351 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:44.352 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:44.352 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:26:44.352 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:47.419700+0000", 2024-09-13T11:26:44.352 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.2449999999999998e-06 2024-09-13T11:26:44.352 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:44.352 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:44.352 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:26:44.352 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:48.065637+0000", 2024-09-13T11:26:44.352 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.64593674300000004 2024-09-13T11:26:44.352 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:44.352 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:44.352 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:26:44.352 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:48.065648+0000", 2024-09-13T11:26:44.353 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0865e-05 2024-09-13T11:26:44.353 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:44.353 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:44.353 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:26:44.353 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:48.065720+0000", 2024-09-13T11:26:44.353 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.2020999999999994e-05 2024-09-13T11:26:44.353 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:44.353 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:44.353 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:26:44.353 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:48.065728+0000", 2024-09-13T11:26:44.353 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.7660000000000004e-06 2024-09-13T11:26:44.354 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:44.354 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:44.354 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:26:44.354 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:48.065853+0000", 2024-09-13T11:26:44.354 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00012499200000000001 2024-09-13T11:26:44.354 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:44.354 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:44.354 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:44.354 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:44.354 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:44.354 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4665.0:25408 3.4 e56/40 3:294e23bd:::benchmark_data_smithi177_31116_object25407:head v 56'1645, mlcod=55'1642)", 2024-09-13T11:26:44.354 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:04.494686+0000", 2024-09-13T11:26:44.354 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 219.853439138, 2024-09-13T11:26:44.354 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.29725566399999998, 2024-09-13T11:26:44.355 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:26:44.355 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:26:44.355 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:44.355 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:26:44.355 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:44.355 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:26:44.355 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.494686+0000", 2024-09-13T11:26:44.355 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:44.355 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:44.355 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:44.355 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:26:44.355 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.494686+0000", 2024-09-13T11:26:44.355 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:44.355 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:44.355 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:44.356 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:26:44.356 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.494686+0000", 2024-09-13T11:26:44.356 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.2399999999999999e-07 2024-09-13T11:26:44.356 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:44.356 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:44.356 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:26:44.356 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.494809+0000", 2024-09-13T11:26:44.356 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00012308600000000001 2024-09-13T11:26:44.356 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:44.356 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:44.356 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:26:44.356 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.789410+0000", 2024-09-13T11:26:44.356 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.29460066899999998 2024-09-13T11:26:44.357 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:44.357 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:44.357 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:26:44.357 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.789425+0000", 2024-09-13T11:26:44.357 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.4600000000000001e-05 2024-09-13T11:26:44.357 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:44.357 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:44.357 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:26:44.357 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.789507+0000", 2024-09-13T11:26:44.357 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.2785000000000001e-05 2024-09-13T11:26:44.357 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:44.357 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:44.357 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:26:44.357 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.789525+0000", 2024-09-13T11:26:44.357 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.8117999999999998e-05 2024-09-13T11:26:44.358 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:44.358 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:44.358 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:26:44.358 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.791852+0000", 2024-09-13T11:26:44.358 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0023264459999999998 2024-09-13T11:26:44.358 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:44.358 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:44.358 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:26:44.358 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.791942+0000", 2024-09-13T11:26:44.358 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.9635999999999994e-05 2024-09-13T11:26:44.358 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:44.358 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:44.360 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:44.360 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:44.360 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:44.360 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4665.0:30828 3.f e59/30 3:f3a9c6a9:::benchmark_data_smithi177_31116_object30827:head v 59'1920, mlcod=59'1916)", 2024-09-13T11:26:44.360 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:09.502607+0000", 2024-09-13T11:26:44.360 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 214.84551809499999, 2024-09-13T11:26:44.361 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.26374408300000002, 2024-09-13T11:26:44.361 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:26:44.361 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:26:44.361 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:44.361 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:26:44.361 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:44.361 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:26:44.361 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.502607+0000", 2024-09-13T11:26:44.361 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:44.361 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:44.361 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:44.361 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:26:44.361 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.502607+0000", 2024-09-13T11:26:44.361 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:44.362 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:44.362 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:44.362 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:26:44.362 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.502609+0000", 2024-09-13T11:26:44.362 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.84e-06 2024-09-13T11:26:44.362 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:44.362 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:44.362 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:26:44.362 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.502663+0000", 2024-09-13T11:26:44.362 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.3971000000000002e-05 2024-09-13T11:26:44.362 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:44.362 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:44.362 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:26:44.362 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.763912+0000", 2024-09-13T11:26:44.362 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.26124904300000001 2024-09-13T11:26:44.362 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:44.363 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:44.363 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:26:44.363 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.763920+0000", 2024-09-13T11:26:44.363 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.1149999999999994e-06 2024-09-13T11:26:44.363 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:44.363 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:44.363 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:26:44.363 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.764024+0000", 2024-09-13T11:26:44.363 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00010380899999999999 2024-09-13T11:26:44.363 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:44.363 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:44.363 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:26:44.363 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.764068+0000", 2024-09-13T11:26:44.363 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.4638999999999997e-05 2024-09-13T11:26:44.364 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:44.364 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:44.364 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:26:44.364 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.766308+0000", 2024-09-13T11:26:44.364 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0022392039999999998 2024-09-13T11:26:44.364 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:44.364 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:44.364 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:26:44.364 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.766351+0000", 2024-09-13T11:26:44.364 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.3461999999999998e-05 2024-09-13T11:26:44.364 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:44.364 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:44.364 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:44.364 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:44.364 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:44.365 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDPGPush(3.1b 84/78 [PushOp(3:dcddd139:::benchmark_data_smithi177_31116_object10384:head, version: 48'673, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:dcddd139:::benchmark_data_smithi177_31116_object10384:head@48'673, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:26:44.365 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:41.400602+0000", 2024-09-13T11:26:44.365 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 182.94752260999999, 2024-09-13T11:26:44.365 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.43365821999999998, 2024-09-13T11:26:44.365 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:26:44.365 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:26:44.365 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:26:44.365 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:26:44.366 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:44.366 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:26:44.366 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:41.400602+0000", 2024-09-13T11:26:44.366 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:44.366 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:44.366 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:44.366 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:26:44.366 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:41.400602+0000", 2024-09-13T11:26:44.366 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:44.366 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:44.366 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:44.366 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:26:44.366 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:41.400604+0000", 2024-09-13T11:26:44.366 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.99e-06 2024-09-13T11:26:44.367 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:44.367 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:44.367 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:26:44.367 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:41.400653+0000", 2024-09-13T11:26:44.367 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.8195000000000002e-05 2024-09-13T11:26:44.367 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:44.367 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:44.367 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:26:44.367 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:41.833180+0000", 2024-09-13T11:26:44.367 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.43252769200000002 2024-09-13T11:26:44.367 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:44.367 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:44.367 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:26:44.367 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:41.833213+0000", 2024-09-13T11:26:44.368 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.2737999999999997e-05 2024-09-13T11:26:44.368 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:44.368 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:44.368 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:26:44.368 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:41.833408+0000", 2024-09-13T11:26:44.368 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00019473799999999999 2024-09-13T11:26:44.368 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:44.368 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:44.368 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:26:44.368 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:41.833417+0000", 2024-09-13T11:26:44.368 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.5349999999999993e-06 2024-09-13T11:26:44.368 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:44.368 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:44.369 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:26:44.369 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:41.834261+0000", 2024-09-13T11:26:44.369 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00084333200000000002 2024-09-13T11:26:44.369 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:44.369 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:44.369 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:44.369 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:44.369 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:44.369 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDPGPushReply(3.20 85/77 [PushReplyOp(3:066982a6:::benchmark_data_smithi177_31116_object16400:head)])", 2024-09-13T11:26:44.369 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:42.671551+0000", 2024-09-13T11:26:44.369 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 181.676573787, 2024-09-13T11:26:44.369 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.28249085499999999, 2024-09-13T11:26:44.369 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:26:44.370 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:26:44.370 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-13T11:26:44.370 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:26:44.370 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:44.370 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:26:44.370 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.671551+0000", 2024-09-13T11:26:44.370 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:44.370 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:44.370 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:44.370 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:26:44.370 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.671551+0000", 2024-09-13T11:26:44.370 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:44.370 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:44.370 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:44.371 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:26:44.371 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.671552+0000", 2024-09-13T11:26:44.371 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.097e-06 2024-09-13T11:26:44.371 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:44.371 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:44.371 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:26:44.371 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.671556+0000", 2024-09-13T11:26:44.371 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.8519999999999997e-06 2024-09-13T11:26:44.371 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:44.371 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:44.371 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:26:44.371 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.951769+0000", 2024-09-13T11:26:44.371 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.28021293000000003 2024-09-13T11:26:44.372 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:44.372 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:44.372 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:26:44.372 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.951780+0000", 2024-09-13T11:26:44.372 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.1059e-05 2024-09-13T11:26:44.372 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:44.372 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:44.372 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:26:44.372 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.951992+0000", 2024-09-13T11:26:44.372 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000211478 2024-09-13T11:26:44.372 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:44.372 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:44.373 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:26:44.373 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.954042+0000", 2024-09-13T11:26:44.373 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0020504389999999998 2024-09-13T11:26:44.373 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:44.373 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:44.373 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:44.373 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:44.373 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:44.373 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDPGPush(3.1b 89/78 [PushOp(3:dc4c79e7:::benchmark_data_smithi177_31116_object19857:head, version: 54'1266, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:dc4c79e7:::benchmark_data_smithi177_31116_object19857:head@54'1266, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:26:44.373 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:46.649128+0000", 2024-09-13T11:26:44.374 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 177.698996679, 2024-09-13T11:26:44.374 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.29963603, 2024-09-13T11:26:44.374 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:26:44.374 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:26:44.374 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:26:44.374 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:26:44.374 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:44.374 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:26:44.374 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.649128+0000", 2024-09-13T11:26:44.374 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:44.374 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:44.374 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:44.374 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:26:44.374 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.649128+0000", 2024-09-13T11:26:44.375 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:44.375 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:44.375 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:44.375 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:26:44.375 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.649130+0000", 2024-09-13T11:26:44.375 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2950000000000001e-06 2024-09-13T11:26:44.375 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:44.375 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:44.375 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:26:44.375 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.649197+0000", 2024-09-13T11:26:44.375 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.7033999999999995e-05 2024-09-13T11:26:44.375 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:44.375 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:44.375 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:26:44.376 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.947691+0000", 2024-09-13T11:26:44.376 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.29849387300000002 2024-09-13T11:26:44.376 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:44.376 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:44.376 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:26:44.376 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.947702+0000", 2024-09-13T11:26:44.376 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.1239e-05 2024-09-13T11:26:44.376 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:44.376 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:44.376 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:26:44.376 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.947799+0000", 2024-09-13T11:26:44.376 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.7283999999999999e-05 2024-09-13T11:26:44.376 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:44.377 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:44.377 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:26:44.377 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.947809+0000", 2024-09-13T11:26:44.377 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.008e-05 2024-09-13T11:26:44.377 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:44.377 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:44.377 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:26:44.377 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.948764+0000", 2024-09-13T11:26:44.377 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00095522499999999998 2024-09-13T11:26:44.377 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:44.377 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:44.377 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:44.377 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:44.377 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:44.378 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:71927 3.0 3:0197244a:::benchmark_data_smithi177_31116_object71926:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2024-09-13T11:26:44.378 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:54.199022+0000", 2024-09-13T11:26:44.378 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 170.14910332299999, 2024-09-13T11:26:44.378 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.26819617800000001, 2024-09-13T11:26:44.378 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:26:44.378 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:26:44.378 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:44.378 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:26:44.378 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:26:44.378 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:44.378 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 71927 2024-09-13T11:26:44.378 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:44.379 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:26:44.379 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:44.379 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:26:44.379 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:54.199022+0000", 2024-09-13T11:26:44.379 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:44.379 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:44.379 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:44.379 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:26:44.379 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:54.199022+0000", 2024-09-13T11:26:44.379 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:44.380 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:44.380 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:44.380 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:26:44.380 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:54.199025+0000", 2024-09-13T11:26:44.380 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.2200000000000001e-06 2024-09-13T11:26:44.380 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:44.380 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:44.380 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:26:44.380 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:54.199076+0000", 2024-09-13T11:26:44.380 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.1184000000000003e-05 2024-09-13T11:26:44.380 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:44.380 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:44.380 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:26:44.380 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:54.199095+0000", 2024-09-13T11:26:44.381 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.9253999999999999e-05 2024-09-13T11:26:44.381 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:44.381 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:44.381 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:26:44.381 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:54.199107+0000", 2024-09-13T11:26:44.381 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.1916000000000001e-05 2024-09-13T11:26:44.381 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:44.381 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:44.381 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:26:44.381 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:54.199196+0000", 2024-09-13T11:26:44.381 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.8645000000000003e-05 2024-09-13T11:26:44.381 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:44.381 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:44.382 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:26:44.382 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:54.199421+0000", 2024-09-13T11:26:44.382 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000224868 2024-09-13T11:26:44.382 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:44.382 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:44.382 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 15", 2024-09-13T11:26:44.382 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:54.199647+0000", 2024-09-13T11:26:44.382 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00022666099999999999 2024-09-13T11:26:44.382 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:44.382 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:44.382 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:26:44.382 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:54.201333+0000", 2024-09-13T11:26:44.382 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.001685761 2024-09-13T11:26:44.382 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:44.383 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:44.383 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:44.383 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:54.467097+0000", 2024-09-13T11:26:44.383 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.26576393399999998 2024-09-13T11:26:44.383 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:44.383 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:44.383 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:26:44.383 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:54.467163+0000", 2024-09-13T11:26:44.383 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.6088999999999997e-05 2024-09-13T11:26:44.383 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:44.383 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:44.383 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:26:44.383 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:54.467218+0000", 2024-09-13T11:26:44.383 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.4645999999999998e-05 2024-09-13T11:26:44.384 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:44.384 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:44.384 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:44.384 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:44.384 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:44.384 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDPGPush(3.25 121/120 [PushOp(3:a5de2229:::benchmark_data_smithi177_31116_object1005:head, version: 43'62, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:a5de2229:::benchmark_data_smithi177_31116_object1005:head@43'62, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:26:44.384 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:24:27.379880+0000", 2024-09-13T11:26:44.384 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 136.96824548500001, 2024-09-13T11:26:44.384 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.72647719499999996, 2024-09-13T11:26:44.384 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:26:44.384 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:26:44.384 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:26:44.384 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:26:44.385 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:44.385 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:26:44.385 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:27.379880+0000", 2024-09-13T11:26:44.385 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:44.385 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:44.385 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:44.385 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:26:44.385 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:27.379880+0000", 2024-09-13T11:26:44.385 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:44.385 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:44.385 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:44.385 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:26:44.385 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:27.379882+0000", 2024-09-13T11:26:44.385 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.1050000000000002e-06 2024-09-13T11:26:44.386 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:44.386 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:44.386 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:26:44.386 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:27.379976+0000", 2024-09-13T11:26:44.386 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.4468000000000004e-05 2024-09-13T11:26:44.386 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:44.386 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:44.386 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:26:44.386 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:28.105075+0000", 2024-09-13T11:26:44.386 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.72509911299999996 2024-09-13T11:26:44.386 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:44.386 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:44.386 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:26:44.386 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:28.105111+0000", 2024-09-13T11:26:44.387 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.5403e-05 2024-09-13T11:26:44.387 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:44.387 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:44.387 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:26:44.387 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:28.105267+0000", 2024-09-13T11:26:44.387 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00015663600000000001 2024-09-13T11:26:44.387 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:44.387 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:44.387 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:26:44.387 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:28.105291+0000", 2024-09-13T11:26:44.387 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.4164000000000001e-05 2024-09-13T11:26:44.387 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:44.387 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:44.387 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:26:44.388 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:28.106357+0000", 2024-09-13T11:26:44.388 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.001065306 2024-09-13T11:26:44.388 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:44.388 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:44.388 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:44.388 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:44.388 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:44.388 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:138294 3.9 3:90012dab:::benchmark_data_smithi177_31116_object138293:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e140)", 2024-09-13T11:26:44.388 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:24:54.250266+0000", 2024-09-13T11:26:44.388 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 110.097858812, 2024-09-13T11:26:44.389 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.33165556200000001, 2024-09-13T11:26:44.389 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:26:44.389 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:26:44.389 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:44.389 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:26:44.389 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:26:44.389 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:44.389 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 138294 2024-09-13T11:26:44.389 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:44.389 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:26:44.389 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:44.389 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:26:44.389 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:54.250266+0000", 2024-09-13T11:26:44.389 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:44.390 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:44.390 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:44.390 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:26:44.390 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:54.250266+0000", 2024-09-13T11:26:44.390 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:44.390 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:44.390 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:44.390 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:26:44.390 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:54.250269+0000", 2024-09-13T11:26:44.390 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.3159999999999999e-06 2024-09-13T11:26:44.390 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:44.390 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:44.390 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:26:44.390 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:54.250362+0000", 2024-09-13T11:26:44.391 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.3051000000000001e-05 2024-09-13T11:26:44.391 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:44.391 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:44.391 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:26:44.391 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:54.250374+0000", 2024-09-13T11:26:44.391 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2327000000000001e-05 2024-09-13T11:26:44.391 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:44.391 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:44.391 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:26:44.391 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:54.250382+0000", 2024-09-13T11:26:44.391 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.7249999999999997e-06 2024-09-13T11:26:44.391 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:44.391 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:44.391 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:26:44.392 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:54.250459+0000", 2024-09-13T11:26:44.392 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.7218999999999994e-05 2024-09-13T11:26:44.392 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:44.392 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:44.392 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:26:44.392 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:54.250611+0000", 2024-09-13T11:26:44.392 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00015241000000000001 2024-09-13T11:26:44.392 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:44.392 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:44.392 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 2", 2024-09-13T11:26:44.392 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:54.250777+0000", 2024-09-13T11:26:44.392 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00016575199999999999 2024-09-13T11:26:44.392 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:44.393 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:44.393 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:26:44.393 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:54.252776+0000", 2024-09-13T11:26:44.393 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0019987260000000001 2024-09-13T11:26:44.393 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:44.393 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:44.393 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:44.393 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:54.581758+0000", 2024-09-13T11:26:44.393 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.32898232799999999 2024-09-13T11:26:44.393 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:44.393 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:44.393 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:26:44.393 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:54.581854+0000", 2024-09-13T11:26:44.393 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.5468000000000001e-05 2024-09-13T11:26:44.394 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:44.394 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:44.394 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:26:44.394 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:54.581922+0000", 2024-09-13T11:26:44.394 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.8239999999999997e-05 2024-09-13T11:26:44.394 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:44.394 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:44.394 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:44.394 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:44.394 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:44.394 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop_reply(client.4665.0:140790 3.20 e144/77 ondisk, result = 0)", 2024-09-13T11:26:44.394 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:24:58.385972+0000", 2024-09-13T11:26:44.394 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 105.96215324400001, 2024-09-13T11:26:44.395 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.51829568500000001, 2024-09-13T11:26:44.395 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:26:44.395 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:26:44.395 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:26:44.395 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:26:44.395 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:44.395 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:26:44.395 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.385972+0000", 2024-09-13T11:26:44.395 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:44.395 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:44.395 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:44.395 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:26:44.395 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.385972+0000", 2024-09-13T11:26:44.395 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:44.396 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:44.396 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:44.396 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:26:44.396 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.385974+0000", 2024-09-13T11:26:44.396 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.7400000000000001e-06 2024-09-13T11:26:44.396 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:44.396 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:44.396 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:26:44.396 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.385979+0000", 2024-09-13T11:26:44.396 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.091e-06 2024-09-13T11:26:44.396 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:44.396 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:44.396 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:26:44.397 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.904091+0000", 2024-09-13T11:26:44.397 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.51811194900000002 2024-09-13T11:26:44.397 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:44.397 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:44.397 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:26:44.397 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.904098+0000", 2024-09-13T11:26:44.397 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.0609999999999998e-06 2024-09-13T11:26:44.397 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:44.397 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:44.397 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:26:44.397 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.904156+0000", 2024-09-13T11:26:44.397 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.7902000000000002e-05 2024-09-13T11:26:44.397 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:44.398 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:44.398 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:26:44.398 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.904163+0000", 2024-09-13T11:26:44.398 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.1430000000000001e-06 2024-09-13T11:26:44.398 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:44.398 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:44.398 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:26:44.398 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.904267+0000", 2024-09-13T11:26:44.398 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000104799 2024-09-13T11:26:44.398 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:44.398 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:44.398 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:44.398 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:44.399 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:44.399 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4665.0:191082 3.26 e175/140 3:67bd986b:::benchmark_data_smithi177_31116_object50779:head v 175'6183, mlcod=175'6180)", 2024-09-13T11:26:44.399 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:25:38.551017+0000", 2024-09-13T11:26:44.399 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 65.797107823999994, 2024-09-13T11:26:44.399 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.29063407899999999, 2024-09-13T11:26:44.399 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:26:44.399 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:26:44.399 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:44.399 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:26:44.399 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:44.399 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:26:44.400 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:38.551017+0000", 2024-09-13T11:26:44.400 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:44.400 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:44.400 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:44.400 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:26:44.400 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:38.551017+0000", 2024-09-13T11:26:44.400 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:44.400 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:44.400 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:44.400 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:26:44.400 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:38.551020+0000", 2024-09-13T11:26:44.400 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.9170000000000002e-06 2024-09-13T11:26:44.400 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:44.401 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:44.401 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:26:44.401 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:38.551033+0000", 2024-09-13T11:26:44.401 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2843000000000001e-05 2024-09-13T11:26:44.401 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:44.401 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:44.401 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:26:44.401 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:38.838514+0000", 2024-09-13T11:26:44.401 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.28748061600000002 2024-09-13T11:26:44.401 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:44.401 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:44.401 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:26:44.401 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:38.838532+0000", 2024-09-13T11:26:44.401 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.8012999999999998e-05 2024-09-13T11:26:44.402 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:44.402 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:44.402 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:26:44.402 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:38.838646+0000", 2024-09-13T11:26:44.402 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000114596 2024-09-13T11:26:44.402 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:44.402 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:44.402 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:26:44.402 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:38.838681+0000", 2024-09-13T11:26:44.402 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.5099999999999999e-05 2024-09-13T11:26:44.402 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:44.402 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:44.402 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:26:44.402 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:38.841593+0000", 2024-09-13T11:26:44.403 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0029117050000000001 2024-09-13T11:26:44.403 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:44.403 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:44.403 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:26:44.403 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:38.841651+0000", 2024-09-13T11:26:44.403 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.8288999999999997e-05 2024-09-13T11:26:44.403 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:44.403 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:44.403 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:44.403 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:44.403 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:44.403 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4665.0:191198 3.26 e175/140 3:64ff4057:::benchmark_data_smithi177_31116_object50895:head v 175'6185, mlcod=175'6182)", 2024-09-13T11:26:44.403 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:25:38.948739+0000", 2024-09-13T11:26:44.403 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 65.399386363000005, 2024-09-13T11:26:44.404 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.25772718100000003, 2024-09-13T11:26:44.404 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:26:44.404 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:26:44.404 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:44.404 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:26:44.404 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:44.404 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:26:44.404 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:38.948739+0000", 2024-09-13T11:26:44.404 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:44.404 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:44.404 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:44.404 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:26:44.404 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:38.948739+0000", 2024-09-13T11:26:44.404 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:44.405 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:44.405 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:44.405 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:26:44.405 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:38.948743+0000", 2024-09-13T11:26:44.405 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.0600000000000001e-06 2024-09-13T11:26:44.405 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:44.405 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:44.405 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:26:44.405 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:38.948760+0000", 2024-09-13T11:26:44.405 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.7540000000000001e-05 2024-09-13T11:26:44.405 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:44.405 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:44.405 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:26:44.405 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:39.204732+0000", 2024-09-13T11:26:44.406 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.25597220199999998 2024-09-13T11:26:44.406 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:44.406 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:44.406 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:26:44.406 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:39.204742+0000", 2024-09-13T11:26:44.406 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.6579999999999997e-06 2024-09-13T11:26:44.406 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:44.406 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:44.406 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:26:44.406 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:39.204834+0000", 2024-09-13T11:26:44.406 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.1565999999999997e-05 2024-09-13T11:26:44.406 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:44.406 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:44.406 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:26:44.407 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:39.204854+0000", 2024-09-13T11:26:44.407 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.0809e-05 2024-09-13T11:26:44.407 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:44.407 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:44.407 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:26:44.407 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:39.206423+0000", 2024-09-13T11:26:44.407 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0015685219999999999 2024-09-13T11:26:44.407 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:44.407 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:44.407 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:26:44.407 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:39.206466+0000", 2024-09-13T11:26:44.407 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.2824000000000002e-05 2024-09-13T11:26:44.407 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:44.408 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:44.408 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:44.408 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:44.408 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:44.408 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:196438 3.20 3:0573f1a4:::benchmark_data_smithi177_31116_object56135:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e178)", 2024-09-13T11:26:44.408 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:25:43.521712+0000", 2024-09-13T11:26:44.408 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 60.826413264000003, 2024-09-13T11:26:44.408 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.23827029799999999, 2024-09-13T11:26:44.408 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:26:44.408 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:26:44.408 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:44.408 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:26:44.408 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:26:44.408 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:44.409 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 196438 2024-09-13T11:26:44.409 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:44.409 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:26:44.409 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:44.409 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:26:44.409 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:43.521712+0000", 2024-09-13T11:26:44.409 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:44.409 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:44.409 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:44.409 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:26:44.409 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:43.521712+0000", 2024-09-13T11:26:44.409 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:44.409 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:44.409 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:44.410 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:26:44.410 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:43.521714+0000", 2024-09-13T11:26:44.410 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.6970000000000002e-06 2024-09-13T11:26:44.410 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:44.410 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:44.410 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:26:44.410 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:43.521723+0000", 2024-09-13T11:26:44.410 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.1319999999999994e-06 2024-09-13T11:26:44.410 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:44.410 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:44.410 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:26:44.410 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:43.521743+0000", 2024-09-13T11:26:44.410 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.0446e-05 2024-09-13T11:26:44.410 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:44.411 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:44.411 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:26:44.411 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:43.521754+0000", 2024-09-13T11:26:44.411 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.1314e-05 2024-09-13T11:26:44.411 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:44.411 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:44.411 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:26:44.411 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:43.521839+0000", 2024-09-13T11:26:44.411 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.4951999999999996e-05 2024-09-13T11:26:44.411 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:44.411 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:44.411 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:26:44.411 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:43.522010+0000", 2024-09-13T11:26:44.411 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000171053 2024-09-13T11:26:44.412 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:44.412 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:44.412 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 0", 2024-09-13T11:26:44.412 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:43.522160+0000", 2024-09-13T11:26:44.412 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000149987 2024-09-13T11:26:44.412 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:44.412 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:44.412 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:26:44.412 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:43.524717+0000", 2024-09-13T11:26:44.412 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0025565649999999998 2024-09-13T11:26:44.412 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:44.412 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:44.412 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:44.413 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:43.759800+0000", 2024-09-13T11:26:44.413 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.23508320699999999 2024-09-13T11:26:44.413 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:44.413 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:44.413 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:26:44.413 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:43.759917+0000", 2024-09-13T11:26:44.413 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000116605 2024-09-13T11:26:44.413 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:44.413 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:44.413 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:26:44.413 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:43.759982+0000", 2024-09-13T11:26:44.413 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.5339999999999994e-05 2024-09-13T11:26:44.413 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:44.413 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:44.414 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:44.414 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:44.414 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:44.414 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:211295 3.2 3:433aa750:::benchmark_data_smithi177_31116_object70992:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e185)", 2024-09-13T11:26:44.414 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:25:55.459639+0000", 2024-09-13T11:26:44.414 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 48.888486014000001, 2024-09-13T11:26:44.414 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.28751246200000002, 2024-09-13T11:26:44.414 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:26:44.414 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:26:44.414 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:44.414 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:26:44.414 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:26:44.414 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:44.414 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 211295 2024-09-13T11:26:44.415 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:44.415 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:26:44.415 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:44.415 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:26:44.415 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:55.459639+0000", 2024-09-13T11:26:44.415 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:44.415 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:44.415 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:44.415 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:26:44.415 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:55.459639+0000", 2024-09-13T11:26:44.415 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:44.415 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:44.415 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:44.415 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:26:44.416 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:55.459642+0000", 2024-09-13T11:26:44.416 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.5940000000000001e-06 2024-09-13T11:26:44.416 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:44.416 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:44.416 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:26:44.416 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:55.459649+0000", 2024-09-13T11:26:44.416 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.0899999999999999e-06 2024-09-13T11:26:44.416 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:44.416 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:44.416 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:26:44.416 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:55.459669+0000", 2024-09-13T11:26:44.416 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.0305999999999998e-05 2024-09-13T11:26:44.416 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:44.416 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:44.417 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:26:44.417 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:55.459683+0000", 2024-09-13T11:26:44.417 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.4219999999999999e-05 2024-09-13T11:26:44.417 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:44.417 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:44.417 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:26:44.417 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:55.459761+0000", 2024-09-13T11:26:44.417 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.8072999999999996e-05 2024-09-13T11:26:44.417 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:44.417 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:44.417 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:26:44.417 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:55.459912+0000", 2024-09-13T11:26:44.417 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000150304 2024-09-13T11:26:44.417 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:44.418 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:44.418 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 3", 2024-09-13T11:26:44.418 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:55.460013+0000", 2024-09-13T11:26:44.418 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000101828 2024-09-13T11:26:44.418 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:44.418 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:44.418 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:26:44.418 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:55.462204+0000", 2024-09-13T11:26:44.418 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.002191082 2024-09-13T11:26:44.418 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:44.418 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:44.418 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:44.418 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:55.747086+0000", 2024-09-13T11:26:44.418 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.28488175100000002 2024-09-13T11:26:44.419 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:44.419 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:44.419 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:26:44.419 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:55.747120+0000", 2024-09-13T11:26:44.419 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.3448000000000002e-05 2024-09-13T11:26:44.419 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:44.419 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:44.419 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:26:44.419 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:55.747151+0000", 2024-09-13T11:26:44.419 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.1766e-05 2024-09-13T11:26:44.419 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:44.419 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:44.419 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:44.419 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:44.420 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:44.420 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop_reply(client.4665.0:221502 3.20 e190/77 ondisk, result = 0)", 2024-09-13T11:26:44.420 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:26:03.873964+0000", 2024-09-13T11:26:44.420 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 40.474161459000001, 2024-09-13T11:26:44.420 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.48664245099999998, 2024-09-13T11:26:44.420 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:26:44.420 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:26:44.420 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:26:44.420 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:26:44.420 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:44.420 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:26:44.420 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:03.873964+0000", 2024-09-13T11:26:44.420 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:44.421 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:44.421 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:44.421 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:26:44.421 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:03.873964+0000", 2024-09-13T11:26:44.421 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:44.421 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:44.421 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:44.421 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:26:44.421 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:03.873965+0000", 2024-09-13T11:26:44.421 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.801e-06 2024-09-13T11:26:44.421 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:44.421 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:44.421 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:26:44.422 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:03.873972+0000", 2024-09-13T11:26:44.422 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.3029999999999996e-06 2024-09-13T11:26:44.422 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:44.422 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:44.422 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:26:44.422 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:04.360178+0000", 2024-09-13T11:26:44.422 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.48620634400000001 2024-09-13T11:26:44.422 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:44.422 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:44.422 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:26:44.422 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:04.360183+0000", 2024-09-13T11:26:44.422 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.2669999999999997e-06 2024-09-13T11:26:44.422 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:44.422 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:44.423 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:26:44.423 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:04.360368+0000", 2024-09-13T11:26:44.423 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00018470100000000001 2024-09-13T11:26:44.423 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:44.423 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:44.423 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:26:44.423 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:04.360387+0000", 2024-09-13T11:26:44.423 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.874e-05 2024-09-13T11:26:44.423 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:44.423 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:44.423 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:26:44.423 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:04.360606+0000", 2024-09-13T11:26:44.423 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00021929499999999999 2024-09-13T11:26:44.423 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:44.424 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:44.424 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:44.424 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:44.424 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:44.424 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4665.0:226607 3.4 e196/117 3:218872c8:::benchmark_data_smithi177_31116_object86304:head v 196'6594, mlcod=195'6591)", 2024-09-13T11:26:44.424 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:26:10.278888+0000", 2024-09-13T11:26:44.424 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 34.069236707999998, 2024-09-13T11:26:44.424 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.25848660899999998, 2024-09-13T11:26:44.424 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:26:44.424 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:26:44.424 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:44.424 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:26:44.424 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:44.424 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:26:44.425 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:10.278888+0000", 2024-09-13T11:26:44.425 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:44.425 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:44.425 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:44.425 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:26:44.425 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:10.278888+0000", 2024-09-13T11:26:44.425 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:44.425 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:44.425 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:44.425 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:26:44.425 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:10.278890+0000", 2024-09-13T11:26:44.425 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.392e-06 2024-09-13T11:26:44.425 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:44.425 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:44.426 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:26:44.426 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:10.278896+0000", 2024-09-13T11:26:44.426 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.562e-06 2024-09-13T11:26:44.426 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:44.426 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:44.426 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:26:44.426 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:10.535554+0000", 2024-09-13T11:26:44.426 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.256657723 2024-09-13T11:26:44.426 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:44.426 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:44.426 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:26:44.426 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:10.535559+0000", 2024-09-13T11:26:44.426 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.4720000000000003e-06 2024-09-13T11:26:44.426 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:44.427 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:44.427 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:26:44.427 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:10.535621+0000", 2024-09-13T11:26:44.427 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.2012000000000002e-05 2024-09-13T11:26:44.427 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:44.427 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:44.427 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:26:44.427 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:10.535644+0000", 2024-09-13T11:26:44.427 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.2911000000000001e-05 2024-09-13T11:26:44.427 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:44.427 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:44.427 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:26:44.427 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:10.537316+0000", 2024-09-13T11:26:44.427 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.001671692 2024-09-13T11:26:44.428 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:44.428 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:44.428 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:26:44.428 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:10.537375+0000", 2024-09-13T11:26:44.428 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.8845000000000003e-05 2024-09-13T11:26:44.428 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:44.428 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:44.428 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:44.428 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:44.428 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:44.428 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:250598 3.9 3:929b3711:::benchmark_data_smithi177_31116_object110295:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e211)", 2024-09-13T11:26:44.428 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:26:32.473105+0000", 2024-09-13T11:26:44.428 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 11.875019912999999, 2024-09-13T11:26:44.429 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.242028942, 2024-09-13T11:26:44.429 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:26:44.429 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:26:44.429 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:44.429 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:26:44.429 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:26:44.429 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:44.429 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 250598 2024-09-13T11:26:44.429 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:44.429 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:26:44.429 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:44.429 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:26:44.429 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:32.473105+0000", 2024-09-13T11:26:44.429 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:44.430 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:44.430 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:44.430 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:26:44.430 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:32.473105+0000", 2024-09-13T11:26:44.430 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:44.430 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:44.430 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:44.430 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:26:44.430 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:32.473106+0000", 2024-09-13T11:26:44.430 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.291e-06 2024-09-13T11:26:44.430 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:44.430 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:44.430 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:26:44.430 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:32.473111+0000", 2024-09-13T11:26:44.431 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.3309999999999997e-06 2024-09-13T11:26:44.431 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:44.431 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:44.431 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:26:44.431 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:32.473122+0000", 2024-09-13T11:26:44.431 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.1345000000000001e-05 2024-09-13T11:26:44.431 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:44.431 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:44.431 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:26:44.431 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:32.473130+0000", 2024-09-13T11:26:44.431 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.8800000000000008e-06 2024-09-13T11:26:44.431 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:44.431 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:44.432 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:26:44.432 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:32.473165+0000", 2024-09-13T11:26:44.432 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.5445999999999999e-05 2024-09-13T11:26:44.432 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:44.432 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:44.432 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:26:44.432 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:32.473237+0000", 2024-09-13T11:26:44.432 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.1969000000000002e-05 2024-09-13T11:26:44.432 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:44.432 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:44.432 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 2", 2024-09-13T11:26:44.432 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:32.473297+0000", 2024-09-13T11:26:44.432 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.9759999999999997e-05 2024-09-13T11:26:44.432 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:44.433 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:44.433 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:26:44.433 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:32.474364+0000", 2024-09-13T11:26:44.433 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.001066521 2024-09-13T11:26:44.433 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:44.433 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:44.433 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:44.433 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:32.714989+0000", 2024-09-13T11:26:44.433 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.240625476 2024-09-13T11:26:44.433 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:44.433 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:44.433 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:26:44.433 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:32.715069+0000", 2024-09-13T11:26:44.433 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.9505000000000005e-05 2024-09-13T11:26:44.434 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:44.434 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:44.434 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:26:44.434 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:32.715134+0000", 2024-09-13T11:26:44.434 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.5418000000000003e-05 2024-09-13T11:26:44.434 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:44.434 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:44.434 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:44.434 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:44.434 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:44.434 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDPGRecoveryDelete(3.22 e219,218 3:44c57693:::benchmark_data_smithi177_31116_object58636:head,179'5345)", 2024-09-13T11:26:44.434 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:26:42.502989+0000", 2024-09-13T11:26:44.434 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 1.845136055, 2024-09-13T11:26:44.434 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.244429074, 2024-09-13T11:26:44.435 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:26:44.435 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:26:44.435 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:26:44.435 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:26:44.435 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:44.435 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:26:44.435 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:42.502989+0000", 2024-09-13T11:26:44.435 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:44.435 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:44.435 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:44.435 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:26:44.435 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:42.502989+0000", 2024-09-13T11:26:44.435 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:44.435 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:44.436 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:44.436 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:26:44.436 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:42.502991+0000", 2024-09-13T11:26:44.436 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.578e-06 2024-09-13T11:26:44.436 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:44.436 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:44.436 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:26:44.436 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:42.502997+0000", 2024-09-13T11:26:44.436 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.1190000000000002e-06 2024-09-13T11:26:44.436 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:44.436 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:44.436 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:26:44.436 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:42.746992+0000", 2024-09-13T11:26:44.437 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.24399506000000001 2024-09-13T11:26:44.437 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:44.437 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:44.437 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:26:44.437 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:42.747009+0000", 2024-09-13T11:26:44.437 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.6877000000000001e-05 2024-09-13T11:26:44.437 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:44.437 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:44.437 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:26:44.437 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:42.747184+0000", 2024-09-13T11:26:44.437 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00017574100000000001 2024-09-13T11:26:44.437 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:44.437 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:44.437 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:26:44.438 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:42.747215+0000", 2024-09-13T11:26:44.438 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.074e-05 2024-09-13T11:26:44.438 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:44.438 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:44.438 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:26:44.438 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:42.747418+0000", 2024-09-13T11:26:44.438 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00020295900000000001 2024-09-13T11:26:44.438 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:44.438 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:44.438 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:44.438 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:44.438 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:44.438 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:260386 3.9 3:93b945ad:::benchmark_data_smithi177_31116_object120083:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e219)", 2024-09-13T11:26:44.438 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:26:42.629421+0000", 2024-09-13T11:26:44.439 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 1.7187039450000001, 2024-09-13T11:26:44.439 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.27837949899999997, 2024-09-13T11:26:44.439 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:26:44.439 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:26:44.439 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:44.439 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:26:44.439 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:26:44.439 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:44.439 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 260386 2024-09-13T11:26:44.439 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:44.439 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:26:44.439 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:44.439 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:26:44.439 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:42.629421+0000", 2024-09-13T11:26:44.440 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:44.440 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:44.440 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:44.440 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:26:44.440 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:42.629421+0000", 2024-09-13T11:26:44.440 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:44.440 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:44.440 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:44.440 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:26:44.440 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:42.629426+0000", 2024-09-13T11:26:44.440 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.6630000000000001e-06 2024-09-13T11:26:44.440 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:44.440 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:44.441 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:26:44.441 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:42.629438+0000", 2024-09-13T11:26:44.441 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2500000000000001e-05 2024-09-13T11:26:44.441 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:44.441 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:44.441 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:26:44.441 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:42.629471+0000", 2024-09-13T11:26:44.441 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.3179999999999997e-05 2024-09-13T11:26:44.441 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:44.441 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:44.441 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:26:44.441 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:42.629488+0000", 2024-09-13T11:26:44.441 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.6946e-05 2024-09-13T11:26:44.441 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:44.442 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:44.442 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:26:44.442 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:42.629628+0000", 2024-09-13T11:26:44.442 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000139369 2024-09-13T11:26:44.442 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:44.442 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:44.442 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:26:44.442 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:42.629892+0000", 2024-09-13T11:26:44.442 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00026411499999999999 2024-09-13T11:26:44.442 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:44.442 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:44.442 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 2", 2024-09-13T11:26:44.442 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:42.630094+0000", 2024-09-13T11:26:44.443 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00020248799999999999 2024-09-13T11:26:44.443 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:44.443 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:44.443 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:26:44.443 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:42.633648+0000", 2024-09-13T11:26:44.443 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.003553956 2024-09-13T11:26:44.443 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:44.443 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:44.443 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:44.443 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:42.907584+0000", 2024-09-13T11:26:44.443 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.27393531500000001 2024-09-13T11:26:44.443 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:44.443 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:44.443 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:26:44.444 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:42.907708+0000", 2024-09-13T11:26:44.444 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00012468200000000001 2024-09-13T11:26:44.444 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:44.444 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:44.444 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:26:44.444 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:42.907801+0000", 2024-09-13T11:26:44.444 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.2285000000000002e-05 2024-09-13T11:26:44.444 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:44.444 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:44.444 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:44.444 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:44.444 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:44.444 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop_reply(client.4665.0:260386 3.9 e219/117 ondisk, result = 0)", 2024-09-13T11:26:44.445 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:26:42.634218+0000", 2024-09-13T11:26:44.445 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 1.713906726, 2024-09-13T11:26:44.445 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.27359229099999999, 2024-09-13T11:26:44.445 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:26:44.445 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:26:44.445 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:26:44.445 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:26:44.445 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:44.445 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:26:44.445 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:42.634218+0000", 2024-09-13T11:26:44.445 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:44.445 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:44.445 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:44.445 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:26:44.446 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:42.634218+0000", 2024-09-13T11:26:44.446 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:26:44.446 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:44.446 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:44.446 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:26:44.446 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:42.634221+0000", 2024-09-13T11:26:44.446 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.7800000000000001e-06 2024-09-13T11:26:44.446 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:44.446 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:44.446 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:26:44.446 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:42.634230+0000", 2024-09-13T11:26:44.446 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.0699999999999996e-06 2024-09-13T11:26:44.446 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:44.446 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:44.447 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:26:44.447 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:42.907405+0000", 2024-09-13T11:26:44.447 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.27317502199999999 2024-09-13T11:26:44.447 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:44.447 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:44.447 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:26:44.447 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:42.907423+0000", 2024-09-13T11:26:44.447 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.7932999999999999e-05 2024-09-13T11:26:44.447 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:44.447 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:44.447 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:26:44.447 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:42.907554+0000", 2024-09-13T11:26:44.447 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000130708 2024-09-13T11:26:44.447 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:44.448 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:44.448 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:26:44.448 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:42.907571+0000", 2024-09-13T11:26:44.448 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.6722999999999999e-05 2024-09-13T11:26:44.448 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:44.448 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:44.448 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:26:44.448 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:42.907811+0000", 2024-09-13T11:26:44.448 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000240055 2024-09-13T11:26:44.448 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:44.448 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:44.448 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:44.448 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:44.448 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:44.449 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-13T11:26:44.450 INFO:tasks.ceph.osd.2.smithi162.stderr:2024-09-13T11:26:44.447+0000 7fe92b35a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26313) UID: 0 2024-09-13T11:26:44.451 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-13T11:26:44.551 INFO:tasks.ceph.osd.1.smithi092.stderr:2024-09-13T11:26:44.547+0000 7fe7789c5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33718) UID: 0 2024-09-13T11:26:44.579 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T11:26:44.596 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-09-13T11:26:44.596 INFO:teuthology.orchestra.run.smithi092.stdout: "ops": [ 2024-09-13T11:26:44.596 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:44.596 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:262467 3.12 3:4ac734bc:::benchmark_data_smithi177_31116_object122164:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e221)", 2024-09-13T11:26:44.596 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:26:44.537730+0000", 2024-09-13T11:26:44.596 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 0.058195019000000001, 2024-09-13T11:26:44.596 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.058223324, 2024-09-13T11:26:44.596 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:26:44.596 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:26:44.596 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "waiting for sub ops", 2024-09-13T11:26:44.596 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:26:44.597 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:26:44.597 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:44.597 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 262467 2024-09-13T11:26:44.597 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:44.597 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:26:44.597 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:44.597 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:26:44.597 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:44.537730+0000", 2024-09-13T11:26:44.597 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:44.597 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:44.597 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:44.597 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:26:44.597 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:44.537730+0000", 2024-09-13T11:26:44.597 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:44.598 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:44.598 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:44.598 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:26:44.598 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:44.537731+0000", 2024-09-13T11:26:44.598 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.471e-06 2024-09-13T11:26:44.598 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:44.598 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:44.598 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:26:44.598 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:44.537736+0000", 2024-09-13T11:26:44.598 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.5299999999999998e-06 2024-09-13T11:26:44.598 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:44.598 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:44.598 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:26:44.599 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:44.537743+0000", 2024-09-13T11:26:44.599 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.9360000000000002e-06 2024-09-13T11:26:44.599 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:44.599 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:44.599 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:26:44.599 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:44.537747+0000", 2024-09-13T11:26:44.599 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.4830000000000001e-06 2024-09-13T11:26:44.599 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:44.599 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:44.599 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:44.599 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:44.537799+0000", 2024-09-13T11:26:44.599 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.1650999999999997e-05 2024-09-13T11:26:44.599 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:44.599 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:44.600 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:26:44.600 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:44.537901+0000", 2024-09-13T11:26:44.600 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000102301 2024-09-13T11:26:44.600 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:44.600 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:44.600 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 7,14", 2024-09-13T11:26:44.600 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:44.537988+0000", 2024-09-13T11:26:44.600 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.7015999999999995e-05 2024-09-13T11:26:44.600 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:44.600 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:44.600 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:26:44.600 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:44.543280+0000", 2024-09-13T11:26:44.600 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0052919070000000002 2024-09-13T11:26:44.601 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:44.601 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:44.601 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:44.601 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:44.544003+0000", 2024-09-13T11:26:44.601 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00072262599999999998 2024-09-13T11:26:44.601 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:44.601 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:44.601 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:44.601 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:44.601 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:44.601 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:262470 3.12 3:4edf1e22:::benchmark_data_smithi177_31116_object122167:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e221)", 2024-09-13T11:26:44.601 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:26:44.539219+0000", 2024-09-13T11:26:44.601 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 0.056706091, 2024-09-13T11:26:44.601 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.056830287, 2024-09-13T11:26:44.602 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:26:44.602 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:26:44.602 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "waiting for sub ops", 2024-09-13T11:26:44.602 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:26:44.602 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:26:44.602 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:44.602 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 262470 2024-09-13T11:26:44.602 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:44.602 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:26:44.602 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:44.602 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:26:44.602 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:44.539219+0000", 2024-09-13T11:26:44.602 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:44.602 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:44.603 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:44.603 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:26:44.603 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:44.539219+0000", 2024-09-13T11:26:44.603 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:44.603 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:44.603 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:44.603 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:26:44.603 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:44.539220+0000", 2024-09-13T11:26:44.603 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.3319999999999999e-06 2024-09-13T11:26:44.603 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:44.603 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:44.603 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:26:44.603 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:44.539225+0000", 2024-09-13T11:26:44.604 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.8110000000000002e-06 2024-09-13T11:26:44.604 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:44.604 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:44.604 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:26:44.604 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:44.539236+0000", 2024-09-13T11:26:44.604 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.1201e-05 2024-09-13T11:26:44.604 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:44.604 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:44.604 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:26:44.605 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:44.539243+0000", 2024-09-13T11:26:44.605 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.2459999999999997e-06 2024-09-13T11:26:44.605 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:44.605 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:44.605 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:44.606 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:44.540628+0000", 2024-09-13T11:26:44.606 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0013848910000000001 2024-09-13T11:26:44.606 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:44.606 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:44.606 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:26:44.606 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:44.540726+0000", 2024-09-13T11:26:44.606 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.7374000000000007e-05 2024-09-13T11:26:44.606 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:44.606 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:44.606 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 7,14", 2024-09-13T11:26:44.606 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:44.540798+0000", 2024-09-13T11:26:44.606 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.2923999999999995e-05 2024-09-13T11:26:44.606 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:44.606 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:44.607 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:26:44.607 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:44.543798+0000", 2024-09-13T11:26:44.607 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0029999169999999999 2024-09-13T11:26:44.607 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:44.607 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:44.607 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:44.607 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:44.544190+0000", 2024-09-13T11:26:44.607 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00039135199999999998 2024-09-13T11:26:44.607 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:44.607 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:44.607 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:44.607 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:44.607 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:44.608 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:262554 3.27 3:e7d5421b:::benchmark_data_smithi177_31116_object122251:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e221)", 2024-09-13T11:26:44.608 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:26:44.591842+0000", 2024-09-13T11:26:44.608 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 0.0040827759999999998, 2024-09-13T11:26:44.608 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0043018700000000002, 2024-09-13T11:26:44.608 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:26:44.608 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:26:44.608 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "waiting for sub ops", 2024-09-13T11:26:44.608 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:26:44.608 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:26:44.608 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:44.608 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 262554 2024-09-13T11:26:44.608 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:44.608 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:26:44.609 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:44.609 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:26:44.609 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:44.591842+0000", 2024-09-13T11:26:44.609 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:44.609 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:44.609 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:44.609 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:26:44.609 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:44.591842+0000", 2024-09-13T11:26:44.609 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:44.609 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:44.609 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:44.609 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:26:44.609 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:44.591845+0000", 2024-09-13T11:26:44.610 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.5849999999999998e-06 2024-09-13T11:26:44.610 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:44.610 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:44.610 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:26:44.610 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:44.591849+0000", 2024-09-13T11:26:44.610 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.3769999999999998e-06 2024-09-13T11:26:44.610 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:44.610 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:44.610 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:26:44.610 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:44.591860+0000", 2024-09-13T11:26:44.610 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0693000000000001e-05 2024-09-13T11:26:44.610 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:44.610 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:44.610 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:26:44.610 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:44.591865+0000", 2024-09-13T11:26:44.611 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.541e-06 2024-09-13T11:26:44.611 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:44.611 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:44.611 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:44.611 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:44.591906+0000", 2024-09-13T11:26:44.611 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.0386000000000003e-05 2024-09-13T11:26:44.611 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:44.611 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:44.611 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:26:44.611 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:44.591985+0000", 2024-09-13T11:26:44.611 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.8974000000000007e-05 2024-09-13T11:26:44.611 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:44.611 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:44.612 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 11", 2024-09-13T11:26:44.612 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:44.592059+0000", 2024-09-13T11:26:44.612 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.3974999999999996e-05 2024-09-13T11:26:44.612 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:44.612 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:44.612 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:26:44.612 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:44.596013+0000", 2024-09-13T11:26:44.612 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0039541250000000002 2024-09-13T11:26:44.612 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:44.612 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:44.612 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:44.612 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:44.613 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:44.613 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:262557 3.12 3:493f1692:::benchmark_data_smithi177_31116_object122254:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e221)", 2024-09-13T11:26:44.613 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:26:44.593924+0000", 2024-09-13T11:26:44.613 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 0.0020007559999999998, 2024-09-13T11:26:44.613 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0022943880000000001, 2024-09-13T11:26:44.613 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:26:44.613 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:26:44.613 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "waiting for sub ops", 2024-09-13T11:26:44.613 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:26:44.613 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:26:44.613 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:44.613 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 262557 2024-09-13T11:26:44.613 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:44.613 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:26:44.614 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:44.614 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:26:44.614 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:44.593924+0000", 2024-09-13T11:26:44.614 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:44.614 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:44.614 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:44.614 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:26:44.614 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:44.593924+0000", 2024-09-13T11:26:44.614 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:44.614 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:44.614 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:44.614 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:26:44.614 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:44.593946+0000", 2024-09-13T11:26:44.615 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.2164999999999999e-05 2024-09-13T11:26:44.615 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:44.615 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:44.615 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:26:44.615 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:44.593954+0000", 2024-09-13T11:26:44.615 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.7919999999999993e-06 2024-09-13T11:26:44.615 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:44.615 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:44.615 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:26:44.615 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:44.593965+0000", 2024-09-13T11:26:44.615 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0769e-05 2024-09-13T11:26:44.615 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:44.615 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:44.615 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:26:44.616 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:44.593974+0000", 2024-09-13T11:26:44.616 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.5429999999999999e-06 2024-09-13T11:26:44.616 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:44.616 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:44.616 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:44.616 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:44.594018+0000", 2024-09-13T11:26:44.616 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.3930000000000001e-05 2024-09-13T11:26:44.616 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:44.616 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:44.616 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:26:44.616 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:44.594159+0000", 2024-09-13T11:26:44.616 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00014040400000000001 2024-09-13T11:26:44.617 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:44.617 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:44.617 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 7,14", 2024-09-13T11:26:44.617 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:44.594257+0000", 2024-09-13T11:26:44.617 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.8748999999999999e-05 2024-09-13T11:26:44.617 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:44.617 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:44.617 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:44.617 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:44.617 INFO:teuthology.orchestra.run.smithi092.stdout: ], 2024-09-13T11:26:44.617 INFO:teuthology.orchestra.run.smithi092.stdout: "num_ops": 4 2024-09-13T11:26:44.617 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-09-13T11:26:44.618 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-13T11:26:44.652 INFO:tasks.ceph.osd.4.smithi001.stderr:2024-09-13T11:26:44.650+0000 7ff8e0218640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26310) UID: 0 2024-09-13T11:26:44.753 INFO:tasks.ceph.osd.12.smithi001.stderr:2024-09-13T11:26:44.750+0000 7f892c6e6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26314) UID: 0 2024-09-13T11:26:44.755 INFO:teuthology.orchestra.run.smithi001.stdout:osd.0: {} 2024-09-13T11:26:44.755 INFO:teuthology.orchestra.run.smithi001.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T11:26:44.782 INFO:teuthology.orchestra.run.smithi001.stdout:osd.1: {} 2024-09-13T11:26:44.782 INFO:teuthology.orchestra.run.smithi001.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T11:26:44.792 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-09-13T11:26:44.792 INFO:teuthology.orchestra.run.smithi092.stdout: "ops": [], 2024-09-13T11:26:44.792 INFO:teuthology.orchestra.run.smithi092.stdout: "complaint_time": 30, 2024-09-13T11:26:44.793 INFO:teuthology.orchestra.run.smithi092.stdout: "num_blocked_ops": 0 2024-09-13T11:26:44.793 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-09-13T11:26:44.809 INFO:teuthology.orchestra.run.smithi001.stdout:osd.2: {} 2024-09-13T11:26:44.809 INFO:teuthology.orchestra.run.smithi001.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T11:26:44.811 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-13T11:26:44.841 INFO:teuthology.orchestra.run.smithi001.stdout:osd.3: {} 2024-09-13T11:26:44.842 INFO:teuthology.orchestra.run.smithi001.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T11:26:44.854 INFO:tasks.ceph.osd.14.smithi162.stderr:2024-09-13T11:26:44.851+0000 7f3df5903640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33154) UID: 0 2024-09-13T11:26:44.872 INFO:teuthology.orchestra.run.smithi001.stdout:osd.4: {} 2024-09-13T11:26:44.872 INFO:teuthology.orchestra.run.smithi001.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T11:26:44.896 INFO:teuthology.orchestra.run.smithi001.stdout:osd.5: {} 2024-09-13T11:26:44.896 INFO:teuthology.orchestra.run.smithi001.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T11:26:44.946 INFO:teuthology.orchestra.run.smithi092.stdout:ERROR: (22) Invalid argument 2024-09-13T11:26:44.947 INFO:teuthology.orchestra.run.smithi092.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:26:44.955 INFO:tasks.ceph.osd.14.smithi162.stderr:2024-09-13T11:26:44.951+0000 7f3df5903640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33154) UID: 0 2024-09-13T11:26:44.967 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-09-13T11:26:45.020 INFO:teuthology.orchestra.run.smithi001.stdout:osd.6: {} 2024-09-13T11:26:45.020 INFO:teuthology.orchestra.run.smithi001.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T11:26:45.043 INFO:teuthology.orchestra.run.smithi001.stdout:osd.7: {} 2024-09-13T11:26:45.043 INFO:teuthology.orchestra.run.smithi001.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T11:26:45.056 INFO:tasks.ceph.osd.12.smithi001.stderr:2024-09-13T11:26:45.054+0000 7f892c6e6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26314) UID: 0 2024-09-13T11:26:45.068 INFO:teuthology.orchestra.run.smithi001.stdout:osd.8: {} 2024-09-13T11:26:45.068 INFO:teuthology.orchestra.run.smithi001.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-13T11:26:45.096 INFO:teuthology.orchestra.run.smithi001.stdout:osd.9: {} 2024-09-13T11:26:45.096 INFO:teuthology.orchestra.run.smithi001.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-13T11:26:45.097 INFO:teuthology.orchestra.run.smithi092.stdout:ERROR: (22) Invalid argument 2024-09-13T11:26:45.097 INFO:teuthology.orchestra.run.smithi092.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:26:45.115 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-09-13T11:26:45.121 INFO:teuthology.orchestra.run.smithi001.stdout:osd.10: {} 2024-09-13T11:26:45.121 INFO:teuthology.orchestra.run.smithi001.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-13T11:26:45.144 INFO:teuthology.orchestra.run.smithi001.stdout:osd.11: {} 2024-09-13T11:26:45.144 INFO:teuthology.orchestra.run.smithi001.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-13T11:26:45.156 INFO:tasks.ceph.osd.10.smithi162.stderr:2024-09-13T11:26:45.155+0000 7fb59c9f0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26317) UID: 0 2024-09-13T11:26:45.168 INFO:teuthology.orchestra.run.smithi001.stdout:osd.12: {} 2024-09-13T11:26:45.168 INFO:teuthology.orchestra.run.smithi001.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-13T11:26:45.191 INFO:teuthology.orchestra.run.smithi001.stdout:osd.13: {} 2024-09-13T11:26:45.192 INFO:teuthology.orchestra.run.smithi001.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-13T11:26:45.223 INFO:teuthology.orchestra.run.smithi001.stdout:osd.14: {} 2024-09-13T11:26:45.224 INFO:teuthology.orchestra.run.smithi001.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-13T11:26:45.251 INFO:teuthology.orchestra.run.smithi001.stdout:osd.15: {} 2024-09-13T11:26:45.251 INFO:teuthology.orchestra.run.smithi001.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-13T11:26:45.255 INFO:teuthology.orchestra.run.smithi092.stdout:ERROR: (22) Invalid argument 2024-09-13T11:26:45.255 INFO:teuthology.orchestra.run.smithi092.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:26:45.257 INFO:tasks.ceph.osd.14.smithi162.stderr:2024-09-13T11:26:45.255+0000 7f3df5903640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33154) UID: 0 2024-09-13T11:26:45.270 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-09-13T11:26:45.358 INFO:tasks.ceph.osd.12.smithi001.stderr:2024-09-13T11:26:45.354+0000 7f892c6e6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26314) UID: 0 2024-09-13T11:26:45.401 INFO:teuthology.orchestra.run.smithi092.stdout:ERROR: (22) Invalid argument 2024-09-13T11:26:45.401 INFO:teuthology.orchestra.run.smithi092.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:26:45.417 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_ops_in_flight 2024-09-13T11:26:45.465 INFO:tasks.ceph.osd.9.smithi092.stderr:2024-09-13T11:26:45.463+0000 7ff5ab3d0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26348) UID: 0 2024-09-13T11:26:45.560 INFO:tasks.ceph.osd.2.smithi162.stderr:2024-09-13T11:26:45.559+0000 7fe92b35a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26313) UID: 0 2024-09-13T11:26:45.561 INFO:teuthology.orchestra.run.smithi092.stdout:ERROR: (22) Invalid argument 2024-09-13T11:26:45.561 INFO:teuthology.orchestra.run.smithi092.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:26:45.582 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_blocked_ops 2024-09-13T11:26:45.661 INFO:tasks.ceph.osd.8.smithi001.stderr:2024-09-13T11:26:45.658+0000 7fb9b48ae640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26312) UID: 0 2024-09-13T11:26:45.749 INFO:teuthology.orchestra.run.smithi092.stdout:ERROR: (22) Invalid argument 2024-09-13T11:26:45.750 INFO:teuthology.orchestra.run.smithi092.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:26:45.762 INFO:tasks.ceph.osd.10.smithi162.stderr:2024-09-13T11:26:45.759+0000 7fb59c9f0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26317) UID: 0 2024-09-13T11:26:45.766 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_historic_ops 2024-09-13T11:26:45.868 INFO:tasks.ceph.osd.3.smithi177.stderr:2024-09-13T11:26:45.865+0000 7f10adba4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26574) UID: 0 2024-09-13T11:26:45.888 INFO:teuthology.orchestra.run.smithi092.stdout:ERROR: (22) Invalid argument 2024-09-13T11:26:45.888 INFO:teuthology.orchestra.run.smithi092.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:26:45.902 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-13T11:26:45.964 INFO:tasks.ceph.osd.4.smithi001.stderr:2024-09-13T11:26:45.962+0000 7ff8e0218640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26310) UID: 0 2024-09-13T11:26:46.054 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-13T11:26:46.055 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:26:46.066 INFO:tasks.ceph.osd.10.smithi162.stderr:2024-09-13T11:26:46.063+0000 7fb59c9f0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26317) UID: 0 2024-09-13T11:26:46.076 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-13T11:26:46.166 INFO:tasks.ceph.osd.11.smithi177.stderr:2024-09-13T11:26:46.165+0000 7f24792c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26580) UID: 0 2024-09-13T11:26:46.267 INFO:tasks.ceph.osd.8.smithi001.stderr:2024-09-13T11:26:46.266+0000 7fb9b48ae640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26312) UID: 0 2024-09-13T11:26:46.268 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-13T11:26:46.268 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:26:46.288 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-13T11:26:46.304 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 15, 14] out_osds: [] dead_osds: [] live_osds: [0, 4, 8, 12, 5, 9, 13, 2, 6, 10, 3, 7, 11, 15, 1, 14] 2024-09-13T11:26:46.304 INFO:tasks.thrashosds.thrasher:choose_action: min_in 2 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-13T11:26:46.305 INFO:tasks.thrashosds.thrasher:Growing pool unique_pool_0 2024-09-13T11:26:46.305 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-13T11:26:46.316 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-13T11:26:46.368 INFO:tasks.ceph.osd.3.smithi177.stderr:2024-09-13T11:26:46.365+0000 7f10adba4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26574) UID: 0 2024-09-13T11:26:46.436 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-13T11:26:46.436 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:26:46.454 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-13T11:26:46.469 INFO:tasks.ceph.osd.12.smithi001.stderr:2024-09-13T11:26:46.466+0000 7f892c6e6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26314) UID: 0 2024-09-13T11:26:46.569 INFO:tasks.ceph.osd.7.smithi177.stderr:2024-09-13T11:26:46.565+0000 7f897b51d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26575) UID: 0 2024-09-13T11:26:46.604 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-13T11:26:46.604 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:26:46.624 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-13T11:26:46.671 INFO:tasks.ceph.osd.12.smithi001.stderr:2024-09-13T11:26:46.666+0000 7f892c6e6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26314) UID: 0 2024-09-13T11:26:46.689 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-13T11:26:46.690 INFO:teuthology.orchestra.run.smithi001.stderr:dumped all 2024-09-13T11:26:46.719 INFO:teuthology.orchestra.run.smithi001.stdout:{"pg_ready":true,"pg_map":{"version":345,"stamp":"2024-09-13T11:26:46.181630+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1143734859,"num_objects":17448,"num_object_clones":0,"num_object_copies":34896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":1069,"num_objects_unfound":0,"num_objects_dirty":17448,"num_whiteouts":0,"num_read":87,"num_read_kb":74,"num_write":172651,"num_write_kb":3356723,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27596,"num_bytes_recovered":1469710374,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":372750,"ondisk_log_size":372750,"up":110,"acting":110,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":119,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":16,"kb":1509949440,"kb_used":4555424,"kb_used_data":2934624,"kb_used_omap":68,"kb_used_meta":1620475,"kb_avail":1505394016,"statfs":{"total":1546188226560,"available":1541523472384,"internally_reserved":0,"allocated":3005054976,"data_stored":2998792844,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":70045,"internal_metadata":1659367011},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,4,3,4],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-312147968,"num_objects":-4763,"num_object_clones":0,"num_object_copies":-9526,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":-1675,"num_objects_unfound":0,"num_objects_dirty":-4763,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4763,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1228,"num_bytes_recovered":50593792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.919939"},"pg_stats":[{"pgid":"3.2d","version":"221'7555","reported_seq":13181,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-09-13T11:26:46.116512+0000","last_change":"2024-09-13T11:25:45.355259+0000","last_active":"2024-09-13T11:26:46.116512+0000","last_peered":"2024-09-13T11:26:46.116512+0000","last_clean":"2024-09-13T11:26:46.116512+0000","last_became_active":"2024-09-13T11:25:39.334408+0000","last_became_peered":"2024-09-13T11:25:39.334408+0000","last_unstale":"2024-09-13T11:26:46.116512+0000","last_undegraded":"2024-09-13T11:26:46.116512+0000","last_fullsized":"2024-09-13T11:26:46.116512+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":6,"last_scrub":"179'6515","last_scrub_stamp":"2024-09-13T11:25:45.355191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:45.355191+0000","objects_scrubbed":1320,"log_size":7555,"log_dups_size":0,"ondisk_log_size":7555,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:27:05.293671+0000","scrub_duration":199,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18087936,"num_objects":276,"num_object_clones":0,"num_object_copies":552,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":276,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1044,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"221'7615","reported_seq":13070,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-09-13T11:26:46.049975+0000","last_change":"2024-09-13T11:26:34.433494+0000","last_active":"2024-09-13T11:26:46.049975+0000","last_peered":"2024-09-13T11:26:46.049975+0000","last_clean":"2024-09-13T11:26:46.049975+0000","last_became_active":"2024-09-13T11:24:59.021895+0000","last_became_peered":"2024-09-13T11:24:59.021895+0000","last_unstale":"2024-09-13T11:26:46.049975+0000","last_undegraded":"2024-09-13T11:26:46.049975+0000","last_fullsized":"2024-09-13T11:26:46.049975+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":6,"last_scrub":"211'7422","last_scrub_stamp":"2024-09-13T11:26:34.433371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:26:34.433371+0000","objects_scrubbed":442,"log_size":7615,"log_dups_size":0,"ondisk_log_size":7615,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:28:02.515615+0000","scrub_duration":63,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16318464,"num_objects":249,"num_object_clones":0,"num_object_copies":498,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":249,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1943,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,7],"acting":[10,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.2b","version":"221'7412","reported_seq":11230,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-09-13T11:26:45.298216+0000","last_change":"2024-09-13T11:26:01.126562+0000","last_active":"2024-09-13T11:26:45.298216+0000","last_peered":"2024-09-13T11:26:45.298216+0000","last_clean":"2024-09-13T11:26:45.298216+0000","last_became_active":"2024-09-13T11:24:26.180578+0000","last_became_peered":"2024-09-13T11:24:26.180578+0000","last_unstale":"2024-09-13T11:26:45.298216+0000","last_undegraded":"2024-09-13T11:26:45.298216+0000","last_fullsized":"2024-09-13T11:26:45.298216+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":6,"last_scrub":"188'6745","last_scrub_stamp":"2024-09-13T11:26:01.126423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:26:01.126423+0000","objects_scrubbed":916,"log_size":7412,"log_dups_size":0,"ondisk_log_size":7412,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:27:08.681933+0000","scrub_duration":391,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15859712,"num_objects":242,"num_object_clones":0,"num_object_copies":484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":242,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2030,"num_write_kb":10624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2a","version":"221'7675","reported_seq":11680,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-09-13T11:26:46.016933+0000","last_change":"2024-09-13T11:26:09.477886+0000","last_active":"2024-09-13T11:26:46.016933+0000","last_peered":"2024-09-13T11:26:46.016933+0000","last_clean":"2024-09-13T11:26:46.016933+0000","last_became_active":"2024-09-13T11:24:45.320540+0000","last_became_peered":"2024-09-13T11:24:45.320540+0000","last_unstale":"2024-09-13T11:26:46.016933+0000","last_undegraded":"2024-09-13T11:26:46.016933+0000","last_fullsized":"2024-09-13T11:26:46.016933+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":6,"last_scrub":"195'7004","last_scrub_stamp":"2024-09-13T11:26:09.477793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:26:09.477793+0000","objects_scrubbed":929,"log_size":7675,"log_dups_size":0,"ondisk_log_size":7675,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:27:33.724690+0000","scrub_duration":115,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16908288,"num_objects":258,"num_object_clones":0,"num_object_copies":516,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":258,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2186,"num_write_kb":11200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,7],"acting":[10,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.29","version":"221'7268","reported_seq":13071,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-09-13T11:26:45.787823+0000","last_change":"2024-09-13T11:25:13.333062+0000","last_active":"2024-09-13T11:26:45.787823+0000","last_peered":"2024-09-13T11:26:45.787823+0000","last_clean":"2024-09-13T11:26:45.787823+0000","last_became_active":"2024-09-13T11:25:12.944028+0000","last_became_peered":"2024-09-13T11:25:12.944028+0000","last_unstale":"2024-09-13T11:26:45.787823+0000","last_undegraded":"2024-09-13T11:26:45.787823+0000","last_fullsized":"2024-09-13T11:26:45.787823+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":6,"last_scrub":"154'5579","last_scrub_stamp":"2024-09-13T11:25:13.332959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:13.332959+0000","objects_scrubbed":1937,"log_size":7268,"log_dups_size":0,"ondisk_log_size":7268,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":243,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16121856,"num_objects":246,"num_object_clones":0,"num_object_copies":492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":246,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1691,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.28","version":"221'7138","reported_seq":12668,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-09-13T11:26:46.107934+0000","last_change":"2024-09-13T11:25:36.143261+0000","last_active":"2024-09-13T11:26:46.107934+0000","last_peered":"2024-09-13T11:26:46.107934+0000","last_clean":"2024-09-13T11:26:46.107934+0000","last_became_active":"2024-09-13T11:25:34.278640+0000","last_became_peered":"2024-09-13T11:25:34.278640+0000","last_unstale":"2024-09-13T11:26:46.107934+0000","last_undegraded":"2024-09-13T11:26:46.107934+0000","last_fullsized":"2024-09-13T11:26:46.107934+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":6,"last_scrub":"174'5936","last_scrub_stamp":"2024-09-13T11:25:36.143113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:36.143113+0000","objects_scrubbed":1447,"log_size":7138,"log_dups_size":0,"ondisk_log_size":7138,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:26:48.290984+0000","scrub_duration":1147,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14811136,"num_objects":226,"num_object_clones":0,"num_object_copies":452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":226,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1221,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5],"acting":[8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.27","version":"221'7216","reported_seq":12865,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-09-13T11:26:45.974358+0000","last_change":"2024-09-13T11:25:37.344464+0000","last_active":"2024-09-13T11:26:45.974358+0000","last_peered":"2024-09-13T11:26:45.974358+0000","last_clean":"2024-09-13T11:26:45.974358+0000","last_became_active":"2024-09-13T11:25:30.257911+0000","last_became_peered":"2024-09-13T11:25:30.257911+0000","last_unstale":"2024-09-13T11:26:45.974358+0000","last_undegraded":"2024-09-13T11:26:45.974358+0000","last_fullsized":"2024-09-13T11:26:45.974358+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":6,"last_scrub":"174'6026","last_scrub_stamp":"2024-09-13T11:25:37.344335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:37.344335+0000","objects_scrubbed":1463,"log_size":7216,"log_dups_size":0,"ondisk_log_size":7216,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:27:06.776518+0000","scrub_duration":195,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17825792,"num_objects":272,"num_object_clones":0,"num_object_copies":544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":272,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1191,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,11],"acting":[5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.26","version":"221'7328","reported_seq":13026,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-09-13T11:26:46.065976+0000","last_change":"2024-09-13T11:26:11.113925+0000","last_active":"2024-09-13T11:26:46.065976+0000","last_peered":"2024-09-13T11:26:46.065976+0000","last_clean":"2024-09-13T11:26:46.065976+0000","last_became_active":"2024-09-13T11:24:54.976919+0000","last_became_peered":"2024-09-13T11:24:54.976919+0000","last_unstale":"2024-09-13T11:26:46.065976+0000","last_undegraded":"2024-09-13T11:26:46.065976+0000","last_fullsized":"2024-09-13T11:26:46.065976+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":6,"last_scrub":"196'6743","last_scrub_stamp":"2024-09-13T11:26:11.113811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:26:11.113811+0000","objects_scrubbed":840,"log_size":7328,"log_dups_size":0,"ondisk_log_size":7328,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:27:35.746195+0000","scrub_duration":103,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16711680,"num_objects":255,"num_object_clones":0,"num_object_copies":510,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":255,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1903,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,12],"acting":[8,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.25","version":"221'7226","reported_seq":13067,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-09-13T11:26:46.096366+0000","last_change":"2024-09-13T11:24:57.073162+0000","last_active":"2024-09-13T11:26:46.096366+0000","last_peered":"2024-09-13T11:26:46.096366+0000","last_clean":"2024-09-13T11:26:46.096366+0000","last_became_active":"2024-09-13T11:24:55.999019+0000","last_became_peered":"2024-09-13T11:24:55.999019+0000","last_unstale":"2024-09-13T11:26:46.096366+0000","last_undegraded":"2024-09-13T11:26:46.096366+0000","last_fullsized":"2024-09-13T11:26:46.096366+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":6,"last_scrub":"142'5214","last_scrub_stamp":"2024-09-13T11:24:57.073061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:24:57.073061+0000","objects_scrubbed":2249,"log_size":7226,"log_dups_size":0,"ondisk_log_size":7226,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":571,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15728640,"num_objects":240,"num_object_clones":0,"num_object_copies":480,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":240,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2015,"num_write_kb":192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,2],"acting":[12,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[]},{"pgid":"3.24","version":"218'7187","reported_seq":15728,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-13T11:26:41.967402+0000","last_change":"2024-09-13T11:26:30.786438+0000","last_active":"2024-09-13T11:26:41.967402+0000","last_peered":"2024-09-13T11:26:41.967402+0000","last_clean":"2024-09-13T11:26:41.967402+0000","last_became_active":"2024-09-13T11:26:05.240552+0000","last_became_peered":"2024-09-13T11:26:05.240552+0000","last_unstale":"2024-09-13T11:26:41.967402+0000","last_undegraded":"2024-09-13T11:26:41.967402+0000","last_fullsized":"2024-09-13T11:26:41.967402+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":6,"last_scrub":"188'6512","last_scrub_stamp":"2024-09-13T11:26:01.118438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:26:01.118438+0000","objects_scrubbed":972,"log_size":7187,"log_dups_size":0,"ondisk_log_size":7187,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:27:14.053206+0000","scrub_duration":183,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19464192,"num_objects":297,"num_object_clones":0,"num_object_copies":594,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":297,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2095,"num_write_kb":9280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2247,"num_bytes_recovered":61014016,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,14],"acting":[9,14],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.23","version":"221'6495","reported_seq":9490,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-09-13T11:26:45.918652+0000","last_change":"2024-09-13T11:25:56.586649+0000","last_active":"2024-09-13T11:26:45.918652+0000","last_peered":"2024-09-13T11:26:45.918652+0000","last_clean":"2024-09-13T11:26:45.918652+0000","last_became_active":"2024-09-13T11:23:47.926677+0000","last_became_peered":"2024-09-13T11:23:47.926677+0000","last_unstale":"2024-09-13T11:26:45.918652+0000","last_undegraded":"2024-09-13T11:26:45.918652+0000","last_fullsized":"2024-09-13T11:26:45.918652+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":6,"last_scrub":"186'5649","last_scrub_stamp":"2024-09-13T11:25:56.586425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:56.586425+0000","objects_scrubbed":1098,"log_size":6495,"log_dups_size":0,"ondisk_log_size":6495,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:27:16.538921+0000","scrub_duration":135,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16449536,"num_objects":251,"num_object_clones":0,"num_object_copies":502,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":251,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3218,"num_write_kb":78080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9],"acting":[5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.22","version":"221'6403","reported_seq":17018,"reported_epoch":221,"state":"active+recovering+degraded+remapped","last_fresh":"2024-09-13T11:26:46.104881+0000","last_change":"2024-09-13T11:26:46.116197+0000","last_active":"2024-09-13T11:26:46.104881+0000","last_peered":"2024-09-13T11:26:46.104881+0000","last_clean":"2024-09-13T11:26:03.171705+0000","last_became_active":"2024-09-13T11:26:42.027697+0000","last_became_peered":"2024-09-13T11:26:42.027697+0000","last_unstale":"2024-09-13T11:26:46.104881+0000","last_undegraded":"2024-09-13T11:26:46.104881+0000","last_fullsized":"2024-09-13T11:26:46.104881+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":6,"last_scrub":"186'5611","last_scrub_stamp":"2024-09-13T11:25:57.123569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:57.123569+0000","objects_scrubbed":1032,"log_size":6403,"log_dups_size":0,"ondisk_log_size":6403,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":147,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15728640,"num_objects":240,"num_object_clones":0,"num_object_copies":480,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":321,"num_objects_unfound":0,"num_objects_dirty":240,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3159,"num_write_kb":75712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3774,"num_bytes_recovered":176685056,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,14],"acting":[0,14],"avail_no_missing":["0","5"],"object_location_counts":[{"shards":"0,5,14","objects":240},{"shards":"0,14","objects":-82},{"shards":"14","objects":1}],"blocked_by":[],"up_primary":12,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.21","version":"221'6245","reported_seq":9242,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-09-13T11:26:46.055684+0000","last_change":"2024-09-13T11:25:49.226484+0000","last_active":"2024-09-13T11:26:46.055684+0000","last_peered":"2024-09-13T11:26:46.055684+0000","last_clean":"2024-09-13T11:26:46.055684+0000","last_became_active":"2024-09-13T11:24:01.250308+0000","last_became_peered":"2024-09-13T11:24:01.250308+0000","last_unstale":"2024-09-13T11:26:46.055684+0000","last_undegraded":"2024-09-13T11:26:46.055684+0000","last_fullsized":"2024-09-13T11:26:46.055684+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":6,"last_scrub":"182'5306","last_scrub_stamp":"2024-09-13T11:25:49.226272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:49.226272+0000","objects_scrubbed":1176,"log_size":6245,"log_dups_size":0,"ondisk_log_size":6245,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:27:04.742252+0000","scrub_duration":151,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15400960,"num_objects":235,"num_object_clones":0,"num_object_copies":470,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":235,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2578,"num_write_kb":44224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8],"acting":[0,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.20","version":"221'6434","reported_seq":8721,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-09-13T11:26:46.054177+0000","last_change":"2024-09-13T11:25:35.773483+0000","last_active":"2024-09-13T11:26:46.054177+0000","last_peered":"2024-09-13T11:26:46.054177+0000","last_clean":"2024-09-13T11:26:46.054177+0000","last_became_active":"2024-09-13T11:23:33.954946+0000","last_became_peered":"2024-09-13T11:23:33.954946+0000","last_unstale":"2024-09-13T11:26:46.054177+0000","last_undegraded":"2024-09-13T11:26:46.054177+0000","last_fullsized":"2024-09-13T11:26:46.054177+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":6,"last_scrub":"174'5240","last_scrub_stamp":"2024-09-13T11:25:35.773409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:35.773409+0000","objects_scrubbed":1424,"log_size":6434,"log_dups_size":0,"ondisk_log_size":6434,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:26:59.046210+0000","scrub_duration":215,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14942208,"num_objects":228,"num_object_clones":0,"num_object_copies":456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":228,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3169,"num_write_kb":74752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,0],"acting":[12,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[]},{"pgid":"3.1f","version":"221'9791","reported_seq":22606,"reported_epoch":221,"state":"active+remapped+backfilling","last_fresh":"2024-09-13T11:26:46.145633+0000","last_change":"2024-09-13T11:26:41.820985+0000","last_active":"2024-09-13T11:26:46.145633+0000","last_peered":"2024-09-13T11:26:46.145633+0000","last_clean":"2024-09-13T11:26:03.341211+0000","last_became_active":"2024-09-13T11:26:06.196771+0000","last_became_peered":"2024-09-13T11:26:06.196771+0000","last_unstale":"2024-09-13T11:26:46.145633+0000","last_undegraded":"2024-09-13T11:26:46.145633+0000","last_fullsized":"2024-09-13T11:26:46.145633+0000","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":5,"last_scrub":"161'6865","last_scrub_stamp":"2024-09-13T11:25:20.773641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:20.773641+0000","objects_scrubbed":3425,"log_size":9791,"log_dups_size":0,"ondisk_log_size":9791,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":535,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32374784,"num_objects":494,"num_object_clones":0,"num_object_copies":988,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":173,"num_objects_unfound":0,"num_objects_dirty":494,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5371,"num_write_kb":102848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3678,"num_bytes_recovered":241041408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,14],"acting":[10,5],"avail_no_missing":["10","5"],"object_location_counts":[{"shards":"5,10","objects":494}],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1e","version":"221'10100","reported_seq":20728,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-09-13T11:26:46.075165+0000","last_change":"2024-09-13T11:24:08.338485+0000","last_active":"2024-09-13T11:26:46.075165+0000","last_peered":"2024-09-13T11:26:46.075165+0000","last_clean":"2024-09-13T11:26:46.075165+0000","last_became_active":"2024-09-13T11:24:01.261448+0000","last_became_peered":"2024-09-13T11:24:01.261448+0000","last_unstale":"2024-09-13T11:26:46.075165+0000","last_undegraded":"2024-09-13T11:26:46.075165+0000","last_fullsized":"2024-09-13T11:26:46.075165+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":5,"last_scrub":"110'4537","last_scrub_stamp":"2024-09-13T11:24:08.338354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:24:08.338354+0000","objects_scrubbed":2760,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":367,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33292288,"num_objects":508,"num_object_clones":0,"num_object_copies":1016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":508,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5572,"num_write_kb":106240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"221'9886","reported_seq":15478,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-09-13T11:26:45.976805+0000","last_change":"2024-09-13T11:25:46.627072+0000","last_active":"2024-09-13T11:26:45.976805+0000","last_peered":"2024-09-13T11:26:45.976805+0000","last_clean":"2024-09-13T11:26:45.976805+0000","last_became_active":"2024-09-13T11:24:01.258225+0000","last_became_peered":"2024-09-13T11:24:01.258225+0000","last_unstale":"2024-09-13T11:26:45.976805+0000","last_undegraded":"2024-09-13T11:26:45.976805+0000","last_fullsized":"2024-09-13T11:26:45.976805+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":5,"last_scrub":"180'7891","last_scrub_stamp":"2024-09-13T11:25:46.626952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:46.626952+0000","objects_scrubbed":2517,"log_size":9886,"log_dups_size":0,"ondisk_log_size":9886,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:26:47.953682+0000","scrub_duration":875,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33554456,"num_objects":513,"num_object_clones":0,"num_object_copies":1026,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":513,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5320,"num_write_kb":94593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,15],"acting":[2,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1c","version":"219'9902","reported_seq":13846,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-13T11:26:42.115609+0000","last_change":"2024-09-13T11:25:46.456795+0000","last_active":"2024-09-13T11:26:42.115609+0000","last_peered":"2024-09-13T11:26:42.115609+0000","last_clean":"2024-09-13T11:26:42.115609+0000","last_became_active":"2024-09-13T11:23:33.989492+0000","last_became_peered":"2024-09-13T11:23:33.989492+0000","last_unstale":"2024-09-13T11:26:42.115609+0000","last_undegraded":"2024-09-13T11:26:42.115609+0000","last_fullsized":"2024-09-13T11:26:42.115609+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":5,"last_scrub":"179'8037","last_scrub_stamp":"2024-09-13T11:25:46.456644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:46.456644+0000","objects_scrubbed":2528,"log_size":9902,"log_dups_size":0,"ondisk_log_size":9902,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:26:47.370789+0000","scrub_duration":451,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43057152,"num_objects":657,"num_object_clones":0,"num_object_copies":1314,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":657,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5627,"num_write_kb":119680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7],"acting":[11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1b","version":"221'10027","reported_seq":15795,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-09-13T11:26:46.127506+0000","last_change":"2024-09-13T11:25:49.890080+0000","last_active":"2024-09-13T11:26:46.127506+0000","last_peered":"2024-09-13T11:26:46.127506+0000","last_clean":"2024-09-13T11:26:46.127506+0000","last_became_active":"2024-09-13T11:24:04.950821+0000","last_became_peered":"2024-09-13T11:24:04.950821+0000","last_unstale":"2024-09-13T11:26:46.127506+0000","last_undegraded":"2024-09-13T11:26:46.127506+0000","last_fullsized":"2024-09-13T11:26:46.127506+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":5,"last_scrub":"182'8139","last_scrub_stamp":"2024-09-13T11:25:49.889914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:49.889914+0000","objects_scrubbed":2406,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:27:03.098951+0000","scrub_duration":519,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33357824,"num_objects":509,"num_object_clones":0,"num_object_copies":1018,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":509,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5383,"num_write_kb":94656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,15],"acting":[12,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[]},{"pgid":"3.1a","version":"221'9846","reported_seq":15278,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-09-13T11:26:46.013166+0000","last_change":"2024-09-13T11:25:46.321420+0000","last_active":"2024-09-13T11:26:46.013166+0000","last_peered":"2024-09-13T11:26:46.013166+0000","last_clean":"2024-09-13T11:26:46.013166+0000","last_became_active":"2024-09-13T11:24:03.935185+0000","last_became_peered":"2024-09-13T11:24:03.935185+0000","last_unstale":"2024-09-13T11:26:46.013166+0000","last_undegraded":"2024-09-13T11:26:46.013166+0000","last_fullsized":"2024-09-13T11:26:46.013166+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":5,"last_scrub":"179'7826","last_scrub_stamp":"2024-09-13T11:25:46.321284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:46.321284+0000","objects_scrubbed":2526,"log_size":9846,"log_dups_size":0,"ondisk_log_size":9846,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:27:04.297171+0000","scrub_duration":935,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32374784,"num_objects":494,"num_object_clones":0,"num_object_copies":988,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":494,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5424,"num_write_kb":104448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9],"acting":[4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.19","version":"219'9345","reported_seq":13942,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-13T11:26:41.976958+0000","last_change":"2024-09-13T11:26:34.674723+0000","last_active":"2024-09-13T11:26:41.976958+0000","last_peered":"2024-09-13T11:26:41.976958+0000","last_clean":"2024-09-13T11:26:41.976958+0000","last_became_active":"2024-09-13T11:23:41.924941+0000","last_became_peered":"2024-09-13T11:23:41.924941+0000","last_unstale":"2024-09-13T11:26:41.976958+0000","last_undegraded":"2024-09-13T11:26:41.976958+0000","last_fullsized":"2024-09-13T11:26:41.976958+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":5,"last_scrub":"211'9143","last_scrub_stamp":"2024-09-13T11:26:34.674563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:26:34.674563+0000","objects_scrubbed":778,"log_size":9345,"log_dups_size":0,"ondisk_log_size":9345,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:27:49.343999+0000","scrub_duration":647,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37617664,"num_objects":574,"num_object_clones":0,"num_object_copies":1148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":574,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5943,"num_write_kb":145216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1],"acting":[9,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.18","version":"221'9640","reported_seq":14523,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-09-13T11:26:46.141816+0000","last_change":"2024-09-13T11:25:37.928304+0000","last_active":"2024-09-13T11:26:46.141816+0000","last_peered":"2024-09-13T11:26:46.141816+0000","last_clean":"2024-09-13T11:26:46.141816+0000","last_became_active":"2024-09-13T11:23:48.926014+0000","last_became_peered":"2024-09-13T11:23:48.926014+0000","last_unstale":"2024-09-13T11:26:46.141816+0000","last_undegraded":"2024-09-13T11:26:46.141816+0000","last_fullsized":"2024-09-13T11:26:46.141816+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":5,"last_scrub":"174'7306","last_scrub_stamp":"2024-09-13T11:25:37.928052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:37.928052+0000","objects_scrubbed":2815,"log_size":9640,"log_dups_size":0,"ondisk_log_size":9640,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:27:06.240226+0000","scrub_duration":631,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30605312,"num_objects":467,"num_object_clones":0,"num_object_copies":934,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":467,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5273,"num_write_kb":86528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0],"acting":[10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.17","version":"221'9432","reported_seq":13772,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-09-13T11:26:46.110230+0000","last_change":"2024-09-13T11:26:25.299859+0000","last_active":"2024-09-13T11:26:46.110230+0000","last_peered":"2024-09-13T11:26:46.110230+0000","last_clean":"2024-09-13T11:26:46.110230+0000","last_became_active":"2024-09-13T11:23:42.935601+0000","last_became_peered":"2024-09-13T11:23:42.935601+0000","last_unstale":"2024-09-13T11:26:46.110230+0000","last_undegraded":"2024-09-13T11:26:46.110230+0000","last_fullsized":"2024-09-13T11:26:46.110230+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":5,"last_scrub":"206'8719","last_scrub_stamp":"2024-09-13T11:26:25.299674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:26:25.299674+0000","objects_scrubbed":1237,"log_size":9432,"log_dups_size":0,"ondisk_log_size":9432,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:27:47.419088+0000","scrub_duration":919,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33226752,"num_objects":507,"num_object_clones":0,"num_object_copies":1014,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":507,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6081,"num_write_kb":146688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3],"acting":[10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"219'9746","reported_seq":14717,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-13T11:26:41.999048+0000","last_change":"2024-09-13T11:26:21.348438+0000","last_active":"2024-09-13T11:26:41.999048+0000","last_peered":"2024-09-13T11:26:41.999048+0000","last_clean":"2024-09-13T11:26:41.999048+0000","last_became_active":"2024-09-13T11:23:46.105237+0000","last_became_peered":"2024-09-13T11:23:46.105237+0000","last_unstale":"2024-09-13T11:26:41.999048+0000","last_undegraded":"2024-09-13T11:26:41.999048+0000","last_fullsized":"2024-09-13T11:26:41.999048+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":5,"last_scrub":"204'9074","last_scrub_stamp":"2024-09-13T11:26:21.348185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:26:21.348185+0000","objects_scrubbed":1320,"log_size":9746,"log_dups_size":0,"ondisk_log_size":9746,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:27:51.286981+0000","scrub_duration":479,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41484288,"num_objects":633,"num_object_clones":0,"num_object_copies":1266,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":633,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6271,"num_write_kb":155200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,15],"acting":[11,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.15","version":"221'9388","reported_seq":22471,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-09-13T11:26:44.925315+0000","last_change":"2024-09-13T11:26:40.336723+0000","last_active":"2024-09-13T11:26:44.925315+0000","last_peered":"2024-09-13T11:26:44.925315+0000","last_clean":"2024-09-13T11:26:44.925315+0000","last_became_active":"2024-09-13T11:26:40.273725+0000","last_became_peered":"2024-09-13T11:26:40.273725+0000","last_unstale":"2024-09-13T11:26:44.925315+0000","last_undegraded":"2024-09-13T11:26:44.925315+0000","last_fullsized":"2024-09-13T11:26:44.925315+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":5,"last_scrub":"181'7535","last_scrub_stamp":"2024-09-13T11:25:48.075568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:48.075568+0000","objects_scrubbed":2413,"log_size":9388,"log_dups_size":0,"ondisk_log_size":9388,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:26:55.650037+0000","scrub_duration":991,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35192832,"num_objects":537,"num_object_clones":0,"num_object_copies":1074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":537,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6005,"num_write_kb":144320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4187,"num_bytes_recovered":274399232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,0],"acting":[14,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"3.14","version":"221'9701","reported_seq":21767,"reported_epoch":221,"state":"active+remapped+backfilling","last_fresh":"2024-09-13T11:26:45.127240+0000","last_change":"2024-09-13T11:26:41.280163+0000","last_active":"2024-09-13T11:26:45.127240+0000","last_peered":"2024-09-13T11:26:45.127240+0000","last_clean":"2024-09-13T11:26:03.284559+0000","last_became_active":"2024-09-13T11:26:06.252189+0000","last_became_peered":"2024-09-13T11:26:06.252189+0000","last_unstale":"2024-09-13T11:26:45.127240+0000","last_undegraded":"2024-09-13T11:26:45.127240+0000","last_fullsized":"2024-09-13T11:26:45.127240+0000","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":5,"last_scrub":"174'7321","last_scrub_stamp":"2024-09-13T11:25:38.085667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:38.085667+0000","objects_scrubbed":2938,"log_size":9701,"log_dups_size":0,"ondisk_log_size":9701,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":431,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36110336,"num_objects":551,"num_object_clones":0,"num_object_copies":1102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":256,"num_objects_unfound":0,"num_objects_dirty":551,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6340,"num_write_kb":158592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3768,"num_bytes_recovered":246939648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,14],"acting":[3,15],"avail_no_missing":["3","15"],"object_location_counts":[{"shards":"3,15","objects":551}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":42,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-09-13T11:26:44.286826+0000","last_change":"2024-09-13T11:26:23.991137+0000","last_active":"2024-09-13T11:26:44.286826+0000","last_peered":"2024-09-13T11:26:44.286826+0000","last_clean":"2024-09-13T11:26:44.286826+0000","last_became_active":"2024-09-13T11:26:23.990749+0000","last_became_peered":"2024-09-13T11:26:23.990749+0000","last_unstale":"2024-09-13T11:26:44.286826+0000","last_undegraded":"2024-09-13T11:26:44.286826+0000","last_fullsized":"2024-09-13T11:26:44.286826+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:25:46.222806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:25:46.222806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:27:14.224983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,15],"acting":[10,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.6","version":"219'7261","reported_seq":15710,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-13T11:26:42.013230+0000","last_change":"2024-09-13T11:26:32.392786+0000","last_active":"2024-09-13T11:26:42.013230+0000","last_peered":"2024-09-13T11:26:42.013230+0000","last_clean":"2024-09-13T11:26:42.013230+0000","last_became_active":"2024-09-13T11:26:05.246522+0000","last_became_peered":"2024-09-13T11:26:05.246522+0000","last_unstale":"2024-09-13T11:26:42.013230+0000","last_undegraded":"2024-09-13T11:26:42.013230+0000","last_fullsized":"2024-09-13T11:26:42.013230+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"211'7110","last_scrub_stamp":"2024-09-13T11:26:32.392666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:26:32.392666+0000","objects_scrubbed":482,"log_size":7261,"log_dups_size":0,"ondisk_log_size":7261,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:27:43.930891+0000","scrub_duration":59,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21692416,"num_objects":331,"num_object_clones":0,"num_object_copies":662,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":331,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2046,"num_write_kb":9408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2230,"num_bytes_recovered":61800448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,14],"acting":[13,14],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":426,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-13T11:26:41.967528+0000","last_change":"2024-09-13T11:26:33.048163+0000","last_active":"2024-09-13T11:26:41.967528+0000","last_peered":"2024-09-13T11:26:41.967528+0000","last_clean":"2024-09-13T11:26:41.967528+0000","last_became_active":"2024-09-13T11:22:14.211923+0000","last_became_peered":"2024-09-13T11:22:14.211923+0000","last_unstale":"2024-09-13T11:26:41.967528+0000","last_undegraded":"2024-09-13T11:26:41.967528+0000","last_fullsized":"2024-09-13T11:26:41.967528+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:26:33.048033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:26:33.048033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:27:40.864163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011689529999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0],"acting":[9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"219'7077","reported_seq":7559,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-13T11:26:42.007214+0000","last_change":"2024-09-13T11:25:59.279849+0000","last_active":"2024-09-13T11:26:42.007214+0000","last_peered":"2024-09-13T11:26:42.007214+0000","last_clean":"2024-09-13T11:26:42.007214+0000","last_became_active":"2024-09-13T11:24:24.092975+0000","last_became_peered":"2024-09-13T11:24:24.092975+0000","last_unstale":"2024-09-13T11:26:42.007214+0000","last_undegraded":"2024-09-13T11:26:42.007214+0000","last_fullsized":"2024-09-13T11:26:42.007214+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"187'6375","last_scrub_stamp":"2024-09-13T11:25:59.279677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:59.279677+0000","objects_scrubbed":1033,"log_size":7077,"log_dups_size":0,"ondisk_log_size":7077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:27:27.660554+0000","scrub_duration":139,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21692416,"num_objects":331,"num_object_clones":0,"num_object_copies":662,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":331,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2259,"num_write_kb":26368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,8],"acting":[13,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":423,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-09-13T11:26:44.285138+0000","last_change":"2024-09-13T11:25:33.957283+0000","last_active":"2024-09-13T11:26:44.285138+0000","last_peered":"2024-09-13T11:26:44.285138+0000","last_clean":"2024-09-13T11:26:44.285138+0000","last_became_active":"2024-09-13T11:22:14.581458+0000","last_became_peered":"2024-09-13T11:22:14.581458+0000","last_unstale":"2024-09-13T11:26:44.285138+0000","last_undegraded":"2024-09-13T11:26:44.285138+0000","last_fullsized":"2024-09-13T11:26:44.285138+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:25:33.957179+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:25:33.957179+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:26:52.126360+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.001026047,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5],"acting":[8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"221'7165","reported_seq":7625,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-09-13T11:26:45.339244+0000","last_change":"2024-09-13T11:25:48.001902+0000","last_active":"2024-09-13T11:26:45.339244+0000","last_peered":"2024-09-13T11:26:45.339244+0000","last_clean":"2024-09-13T11:26:45.339244+0000","last_became_active":"2024-09-13T11:24:24.083077+0000","last_became_peered":"2024-09-13T11:24:24.083077+0000","last_unstale":"2024-09-13T11:26:45.339244+0000","last_undegraded":"2024-09-13T11:26:45.339244+0000","last_fullsized":"2024-09-13T11:26:45.339244+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"181'6222","last_scrub_stamp":"2024-09-13T11:25:48.001778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:48.001778+0000","objects_scrubbed":1224,"log_size":7165,"log_dups_size":0,"ondisk_log_size":7165,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:27:16.908122+0000","scrub_duration":171,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18415616,"num_objects":281,"num_object_clones":0,"num_object_copies":562,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":281,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2431,"num_write_kb":32704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,12],"acting":[1,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":410,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-09-13T11:26:44.286751+0000","last_change":"2024-09-13T11:25:45.851854+0000","last_active":"2024-09-13T11:26:44.286751+0000","last_peered":"2024-09-13T11:26:44.286751+0000","last_clean":"2024-09-13T11:26:44.286751+0000","last_became_active":"2024-09-13T11:22:47.719636+0000","last_became_peered":"2024-09-13T11:22:47.719636+0000","last_unstale":"2024-09-13T11:26:44.286751+0000","last_undegraded":"2024-09-13T11:26:44.286751+0000","last_fullsized":"2024-09-13T11:26:44.286751+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:25:45.851784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:25:45.851784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:26:50.817831+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00087476000000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,9],"acting":[1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"219'7043","reported_seq":7524,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-13T11:26:42.091763+0000","last_change":"2024-09-13T11:26:24.022070+0000","last_active":"2024-09-13T11:26:42.091763+0000","last_peered":"2024-09-13T11:26:42.091763+0000","last_clean":"2024-09-13T11:26:42.091763+0000","last_became_active":"2024-09-13T11:26:23.988370+0000","last_became_peered":"2024-09-13T11:26:23.988370+0000","last_unstale":"2024-09-13T11:26:42.091763+0000","last_undegraded":"2024-09-13T11:26:42.091763+0000","last_fullsized":"2024-09-13T11:26:42.091763+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"182'6171","last_scrub_stamp":"2024-09-13T11:25:49.468616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:49.468616+0000","objects_scrubbed":1202,"log_size":7043,"log_dups_size":0,"ondisk_log_size":7043,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:27:04.449499+0000","scrub_duration":175,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21626880,"num_objects":330,"num_object_clones":0,"num_object_copies":660,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":330,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2306,"num_write_kb":32384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,15],"acting":[13,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"3.3","version":"221'6294","reported_seq":6774,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-09-13T11:26:46.059247+0000","last_change":"2024-09-13T11:25:08.394545+0000","last_active":"2024-09-13T11:26:46.059247+0000","last_peered":"2024-09-13T11:26:46.059247+0000","last_clean":"2024-09-13T11:26:46.059247+0000","last_became_active":"2024-09-13T11:23:31.930448+0000","last_became_peered":"2024-09-13T11:23:31.930448+0000","last_unstale":"2024-09-13T11:26:46.059247+0000","last_undegraded":"2024-09-13T11:26:46.059247+0000","last_fullsized":"2024-09-13T11:26:46.059247+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"151'4587","last_scrub_stamp":"2024-09-13T11:25:08.394441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:08.394441+0000","objects_scrubbed":1942,"log_size":6294,"log_dups_size":0,"ondisk_log_size":6294,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":219,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15204352,"num_objects":232,"num_object_clones":0,"num_object_copies":464,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":232,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3172,"num_write_kb":80896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,7],"acting":[10,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"28'2","reported_seq":33,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-09-13T11:26:44.294170+0000","last_change":"2024-09-13T11:26:36.271659+0000","last_active":"2024-09-13T11:26:44.294170+0000","last_peered":"2024-09-13T11:26:44.294170+0000","last_clean":"2024-09-13T11:26:44.294170+0000","last_became_active":"2024-09-13T11:26:36.267081+0000","last_became_peered":"2024-09-13T11:26:36.267081+0000","last_unstale":"2024-09-13T11:26:44.294170+0000","last_undegraded":"2024-09-13T11:26:44.294170+0000","last_fullsized":"2024-09-13T11:26:44.294170+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2","last_scrub_stamp":"2024-09-13T11:25:13.869413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:25:13.869413+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0025826730000000002,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9],"acting":[3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.0","version":"221'6341","reported_seq":6819,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-09-13T11:26:46.054373+0000","last_change":"2024-09-13T11:25:09.842875+0000","last_active":"2024-09-13T11:26:46.054373+0000","last_peered":"2024-09-13T11:26:46.054373+0000","last_clean":"2024-09-13T11:26:46.054373+0000","last_became_active":"2024-09-13T11:23:31.921388+0000","last_became_peered":"2024-09-13T11:23:31.921388+0000","last_unstale":"2024-09-13T11:26:46.054373+0000","last_undegraded":"2024-09-13T11:26:46.054373+0000","last_fullsized":"2024-09-13T11:26:46.054373+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"151'4644","last_scrub_stamp":"2024-09-13T11:25:09.842728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:09.842728+0000","objects_scrubbed":1944,"log_size":6341,"log_dups_size":0,"ondisk_log_size":6341,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":395,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16187392,"num_objects":247,"num_object_clones":0,"num_object_copies":494,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":247,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3235,"num_write_kb":84032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,15],"acting":[12,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":69,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-09-13T11:26:44.289414+0000","last_change":"2024-09-13T11:26:05.047291+0000","last_active":"2024-09-13T11:26:44.289414+0000","last_peered":"2024-09-13T11:26:44.289414+0000","last_clean":"2024-09-13T11:26:44.289414+0000","last_became_active":"2024-09-13T11:26:05.046717+0000","last_became_peered":"2024-09-13T11:26:05.046717+0000","last_unstale":"2024-09-13T11:26:44.289414+0000","last_undegraded":"2024-09-13T11:26:44.289414+0000","last_fullsized":"2024-09-13T11:26:44.289414+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:25:34.822831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:25:34.822831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00081225399999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,1],"acting":[14,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"3.1","version":"219'6242","reported_seq":6731,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-13T11:26:42.080971+0000","last_change":"2024-09-13T11:26:33.441490+0000","last_active":"2024-09-13T11:26:42.080971+0000","last_peered":"2024-09-13T11:26:42.080971+0000","last_clean":"2024-09-13T11:26:42.080971+0000","last_became_active":"2024-09-13T11:23:31.926528+0000","last_became_peered":"2024-09-13T11:23:31.926528+0000","last_unstale":"2024-09-13T11:26:42.080971+0000","last_undegraded":"2024-09-13T11:26:42.080971+0000","last_fullsized":"2024-09-13T11:26:42.080971+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"211'6118","last_scrub_stamp":"2024-09-13T11:26:33.441303+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:26:33.441303+0000","objects_scrubbed":453,"log_size":6242,"log_dups_size":0,"ondisk_log_size":6242,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:27:59.191924+0000","scrub_duration":71,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21495808,"num_objects":328,"num_object_clones":0,"num_object_copies":656,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":328,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3172,"num_write_kb":85056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,11],"acting":[13,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":423,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-09-13T11:26:44.294159+0000","last_change":"2024-09-13T11:25:56.557174+0000","last_active":"2024-09-13T11:26:44.294159+0000","last_peered":"2024-09-13T11:26:44.294159+0000","last_clean":"2024-09-13T11:26:44.294159+0000","last_became_active":"2024-09-13T11:22:14.211435+0000","last_became_peered":"2024-09-13T11:22:14.211435+0000","last_unstale":"2024-09-13T11:26:44.294159+0000","last_undegraded":"2024-09-13T11:26:44.294159+0000","last_fullsized":"2024-09-13T11:26:44.294159+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:25:56.557113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:25:56.557113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:27:18.356524+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0010010360000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2","version":"221'6414","reported_seq":14829,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-09-13T11:26:45.000338+0000","last_change":"2024-09-13T11:26:43.033025+0000","last_active":"2024-09-13T11:26:45.000338+0000","last_peered":"2024-09-13T11:26:45.000338+0000","last_clean":"2024-09-13T11:26:45.000338+0000","last_became_active":"2024-09-13T11:26:42.985163+0000","last_became_peered":"2024-09-13T11:26:42.985163+0000","last_unstale":"2024-09-13T11:26:45.000338+0000","last_undegraded":"2024-09-13T11:26:45.000338+0000","last_fullsized":"2024-09-13T11:26:45.000338+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"164'4994","last_scrub_stamp":"2024-09-13T11:25:23.765047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:23.765047+0000","objects_scrubbed":1689,"log_size":6414,"log_dups_size":0,"ondisk_log_size":6414,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:26:48.396517+0000","scrub_duration":283,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17432576,"num_objects":266,"num_object_clones":0,"num_object_copies":532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":266,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3357,"num_write_kb":89024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3963,"num_bytes_recovered":162332672,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,12],"acting":[14,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"2.3","version":"26'1","reported_seq":73,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-09-13T11:26:44.289434+0000","last_change":"2024-09-13T11:26:05.046376+0000","last_active":"2024-09-13T11:26:44.289434+0000","last_peered":"2024-09-13T11:26:44.289434+0000","last_clean":"2024-09-13T11:26:44.289434+0000","last_became_active":"2024-09-13T11:26:05.041032+0000","last_became_peered":"2024-09-13T11:26:05.041032+0000","last_unstale":"2024-09-13T11:26:44.289434+0000","last_undegraded":"2024-09-13T11:26:44.289434+0000","last_fullsized":"2024-09-13T11:26:44.289434+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"26'1","last_scrub_stamp":"2024-09-13T11:25:23.034247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:25:23.034247+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,15],"acting":[14,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"1.0","version":"26'112","reported_seq":557,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-09-13T11:26:44.295356+0000","last_change":"2024-09-13T11:25:23.642366+0000","last_active":"2024-09-13T11:26:44.295356+0000","last_peered":"2024-09-13T11:26:44.295356+0000","last_clean":"2024-09-13T11:26:44.295356+0000","last_became_active":"2024-09-13T11:22:12.129721+0000","last_became_peered":"2024-09-13T11:22:12.129721+0000","last_unstale":"2024-09-13T11:26:44.295356+0000","last_undegraded":"2024-09-13T11:26:44.295356+0000","last_fullsized":"2024-09-13T11:26:44.295356+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"26'112","last_scrub_stamp":"2024-09-13T11:25:23.642204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:11.098388+0000","last_clean_scrub_stamp":"2024-09-13T11:25:23.642204+0000","objects_scrubbed":2,"log_size":112,"log_dups_size":0,"ondisk_log_size":112,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":67,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.8","version":"221'7158","reported_seq":7636,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-09-13T11:26:45.995647+0000","last_change":"2024-09-13T11:26:00.418168+0000","last_active":"2024-09-13T11:26:45.995647+0000","last_peered":"2024-09-13T11:26:45.995647+0000","last_clean":"2024-09-13T11:26:45.995647+0000","last_became_active":"2024-09-13T11:24:24.096220+0000","last_became_peered":"2024-09-13T11:24:24.096220+0000","last_unstale":"2024-09-13T11:26:45.995647+0000","last_undegraded":"2024-09-13T11:26:45.995647+0000","last_fullsized":"2024-09-13T11:26:45.995647+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"188'6425","last_scrub_stamp":"2024-09-13T11:26:00.417942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:26:00.417942+0000","objects_scrubbed":989,"log_size":7158,"log_dups_size":0,"ondisk_log_size":7158,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:27:07.107188+0000","scrub_duration":199,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777216,"num_objects":256,"num_object_clones":0,"num_object_copies":512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":256,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2410,"num_write_kb":31232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.9","version":"221'7226","reported_seq":7686,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-09-13T11:26:46.129853+0000","last_change":"2024-09-13T11:24:24.928964+0000","last_active":"2024-09-13T11:26:46.129853+0000","last_peered":"2024-09-13T11:26:46.129853+0000","last_clean":"2024-09-13T11:26:46.129853+0000","last_became_active":"2024-09-13T11:24:24.082504+0000","last_became_peered":"2024-09-13T11:24:24.082504+0000","last_unstale":"2024-09-13T11:26:46.129853+0000","last_undegraded":"2024-09-13T11:26:46.129853+0000","last_fullsized":"2024-09-13T11:26:46.129853+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'4799","last_scrub_stamp":"2024-09-13T11:24:24.928724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:24:24.928724+0000","objects_scrubbed":1705,"log_size":7226,"log_dups_size":0,"ondisk_log_size":7226,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":255,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16187392,"num_objects":247,"num_object_clones":0,"num_object_copies":494,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":247,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2430,"num_write_kb":31104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,2],"acting":[12,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[]},{"pgid":"3.a","version":"221'7561","reported_seq":8043,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-09-13T11:26:45.093137+0000","last_change":"2024-09-13T11:25:59.695285+0000","last_active":"2024-09-13T11:26:45.093137+0000","last_peered":"2024-09-13T11:26:45.093137+0000","last_clean":"2024-09-13T11:26:45.093137+0000","last_became_active":"2024-09-13T11:24:24.089253+0000","last_became_peered":"2024-09-13T11:24:24.089253+0000","last_unstale":"2024-09-13T11:26:45.093137+0000","last_undegraded":"2024-09-13T11:26:45.093137+0000","last_fullsized":"2024-09-13T11:26:45.093137+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"187'6833","last_scrub_stamp":"2024-09-13T11:25:59.695136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:59.695136+0000","objects_scrubbed":973,"log_size":7561,"log_dups_size":0,"ondisk_log_size":7561,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:27:23.532121+0000","scrub_duration":131,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15990784,"num_objects":244,"num_object_clones":0,"num_object_copies":488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":244,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2407,"num_write_kb":27968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.b","version":"219'7455","reported_seq":7944,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-13T11:26:41.966316+0000","last_change":"2024-09-13T11:25:58.292249+0000","last_active":"2024-09-13T11:26:41.966316+0000","last_peered":"2024-09-13T11:26:41.966316+0000","last_clean":"2024-09-13T11:26:41.966316+0000","last_became_active":"2024-09-13T11:24:24.085684+0000","last_became_peered":"2024-09-13T11:24:24.085684+0000","last_unstale":"2024-09-13T11:26:41.966316+0000","last_undegraded":"2024-09-13T11:26:41.966316+0000","last_fullsized":"2024-09-13T11:26:41.966316+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"186'6759","last_scrub_stamp":"2024-09-13T11:25:58.292153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:58.292153+0000","objects_scrubbed":1030,"log_size":7455,"log_dups_size":0,"ondisk_log_size":7455,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:27:02.082370+0000","scrub_duration":119,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21823488,"num_objects":333,"num_object_clones":0,"num_object_copies":666,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":333,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2343,"num_write_kb":30656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,1],"acting":[13,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"3.c","version":"219'7479","reported_seq":7960,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-13T11:26:41.992712+0000","last_change":"2024-09-13T11:25:36.279464+0000","last_active":"2024-09-13T11:26:41.992712+0000","last_peered":"2024-09-13T11:26:41.992712+0000","last_clean":"2024-09-13T11:26:41.992712+0000","last_became_active":"2024-09-13T11:24:24.090847+0000","last_became_peered":"2024-09-13T11:24:24.090847+0000","last_unstale":"2024-09-13T11:26:41.992712+0000","last_undegraded":"2024-09-13T11:26:41.992712+0000","last_fullsized":"2024-09-13T11:26:41.992712+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"174'6334","last_scrub_stamp":"2024-09-13T11:25:36.279390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:36.279390+0000","objects_scrubbed":1456,"log_size":7479,"log_dups_size":0,"ondisk_log_size":7479,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:26:57.115518+0000","scrub_duration":235,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20316160,"num_objects":310,"num_object_clones":0,"num_object_copies":620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":310,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2330,"num_write_kb":29440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,13],"acting":[11,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.d","version":"217'7447","reported_seq":7920,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-09-13T11:26:41.111946+0000","last_change":"2024-09-13T11:25:57.320482+0000","last_active":"2024-09-13T11:26:41.111946+0000","last_peered":"2024-09-13T11:26:41.111946+0000","last_clean":"2024-09-13T11:26:41.111946+0000","last_became_active":"2024-09-13T11:24:24.087224+0000","last_became_peered":"2024-09-13T11:24:24.087224+0000","last_unstale":"2024-09-13T11:26:41.111946+0000","last_undegraded":"2024-09-13T11:26:41.111946+0000","last_fullsized":"2024-09-13T11:26:41.111946+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"186'6735","last_scrub_stamp":"2024-09-13T11:25:57.320378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:57.320378+0000","objects_scrubbed":1064,"log_size":7447,"log_dups_size":0,"ondisk_log_size":7447,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:27:00.291410+0000","scrub_duration":211,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23003136,"num_objects":351,"num_object_clones":0,"num_object_copies":702,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":351,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2340,"num_write_kb":30912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.e","version":"221'10020","reported_seq":15423,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-09-13T11:26:46.010936+0000","last_change":"2024-09-13T11:25:46.438251+0000","last_active":"2024-09-13T11:26:46.010936+0000","last_peered":"2024-09-13T11:26:46.010936+0000","last_clean":"2024-09-13T11:26:46.010936+0000","last_became_active":"2024-09-13T11:23:31.925869+0000","last_became_peered":"2024-09-13T11:23:31.925869+0000","last_unstale":"2024-09-13T11:26:46.010936+0000","last_undegraded":"2024-09-13T11:26:46.010936+0000","last_fullsized":"2024-09-13T11:26:46.010936+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"180'8032","last_scrub_stamp":"2024-09-13T11:25:46.438107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:46.438107+0000","objects_scrubbed":2456,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:26:55.276011+0000","scrub_duration":327,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30408704,"num_objects":464,"num_object_clones":0,"num_object_copies":928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":464,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6494,"num_write_kb":165888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"218'9847","reported_seq":10353,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-13T11:26:41.992666+0000","last_change":"2024-09-13T11:26:34.941012+0000","last_active":"2024-09-13T11:26:41.992666+0000","last_peered":"2024-09-13T11:26:41.992666+0000","last_clean":"2024-09-13T11:26:41.992666+0000","last_became_active":"2024-09-13T11:23:31.935239+0000","last_became_peered":"2024-09-13T11:23:31.935239+0000","last_unstale":"2024-09-13T11:26:41.992666+0000","last_undegraded":"2024-09-13T11:26:41.992666+0000","last_fullsized":"2024-09-13T11:26:41.992666+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"211'9653","last_scrub_stamp":"2024-09-13T11:26:34.940878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:26:34.940878+0000","objects_scrubbed":877,"log_size":9847,"log_dups_size":0,"ondisk_log_size":9847,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:28:02.803036+0000","scrub_duration":119,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44498944,"num_objects":679,"num_object_clones":0,"num_object_copies":1358,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":679,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6427,"num_write_kb":170560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,12],"acting":[11,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.10","version":"221'9762","reported_seq":14417,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-09-13T11:26:46.081804+0000","last_change":"2024-09-13T11:26:22.296956+0000","last_active":"2024-09-13T11:26:46.081804+0000","last_peered":"2024-09-13T11:26:46.081804+0000","last_clean":"2024-09-13T11:26:46.081804+0000","last_became_active":"2024-09-13T11:23:41.931183+0000","last_became_peered":"2024-09-13T11:23:41.931183+0000","last_unstale":"2024-09-13T11:26:46.081804+0000","last_undegraded":"2024-09-13T11:26:46.081804+0000","last_fullsized":"2024-09-13T11:26:46.081804+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":5,"last_scrub":"204'8963","last_scrub_stamp":"2024-09-13T11:26:22.296795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:26:22.296795+0000","objects_scrubbed":1288,"log_size":9762,"log_dups_size":0,"ondisk_log_size":9762,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:27:38.588405+0000","scrub_duration":195,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31784960,"num_objects":485,"num_object_clones":0,"num_object_copies":970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":485,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6296,"num_write_kb":152000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.11","version":"221'9644","reported_seq":13051,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-09-13T11:26:46.052488+0000","last_change":"2024-09-13T11:25:24.287326+0000","last_active":"2024-09-13T11:26:46.052488+0000","last_peered":"2024-09-13T11:26:46.052488+0000","last_clean":"2024-09-13T11:26:46.052488+0000","last_became_active":"2024-09-13T11:23:39.063172+0000","last_became_peered":"2024-09-13T11:23:39.063172+0000","last_unstale":"2024-09-13T11:26:46.052488+0000","last_undegraded":"2024-09-13T11:26:46.052488+0000","last_fullsized":"2024-09-13T11:26:46.052488+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":5,"last_scrub":"165'6821","last_scrub_stamp":"2024-09-13T11:25:24.287082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:24.287082+0000","objects_scrubbed":3406,"log_size":9644,"log_dups_size":0,"ondisk_log_size":9644,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:26:53.631356+0000","scrub_duration":947,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37355520,"num_objects":570,"num_object_clones":0,"num_object_copies":1140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":570,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6303,"num_write_kb":155584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11],"acting":[4,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.12","version":"221'9655","reported_seq":20505,"reported_epoch":221,"state":"active+remapped+backfill_wait","last_fresh":"2024-09-13T11:26:46.074211+0000","last_change":"2024-09-13T11:26:42.081220+0000","last_active":"2024-09-13T11:26:46.074211+0000","last_peered":"2024-09-13T11:26:46.074211+0000","last_clean":"2024-09-13T11:26:03.227632+0000","last_became_active":"2024-09-13T11:26:06.225212+0000","last_became_peered":"2024-09-13T11:26:06.225212+0000","last_unstale":"2024-09-13T11:26:46.074211+0000","last_undegraded":"2024-09-13T11:26:46.074211+0000","last_fullsized":"2024-09-13T11:26:46.074211+0000","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":5,"last_scrub":"186'8089","last_scrub_stamp":"2024-09-13T11:25:57.085837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:57.085837+0000","objects_scrubbed":2130,"log_size":9655,"log_dups_size":0,"ondisk_log_size":9655,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1023,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36175872,"num_objects":552,"num_object_clones":0,"num_object_copies":1104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":319,"num_objects_unfound":0,"num_objects_dirty":552,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6546,"num_write_kb":171712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3746,"num_bytes_recovered":245497856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,7],"acting":[5,7],"avail_no_missing":["5","7"],"object_location_counts":[{"shards":"5,7","objects":552}],"blocked_by":[],"up_primary":14,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"219'9543","reported_seq":12896,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-13T11:26:42.046178+0000","last_change":"2024-09-13T11:26:36.941311+0000","last_active":"2024-09-13T11:26:42.046178+0000","last_peered":"2024-09-13T11:26:42.046178+0000","last_clean":"2024-09-13T11:26:42.046178+0000","last_became_active":"2024-09-13T11:23:41.082433+0000","last_became_peered":"2024-09-13T11:23:41.082433+0000","last_unstale":"2024-09-13T11:26:42.046178+0000","last_undegraded":"2024-09-13T11:26:42.046178+0000","last_fullsized":"2024-09-13T11:26:42.046178+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":5,"last_scrub":"213'9393","last_scrub_stamp":"2024-09-13T11:26:36.941193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:26:36.941193+0000","objects_scrubbed":798,"log_size":9543,"log_dups_size":0,"ondisk_log_size":9543,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:27:53.428452+0000","scrub_duration":119,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42467328,"num_objects":648,"num_object_clones":0,"num_object_copies":1296,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":648,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6147,"num_write_kb":154048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7],"acting":[11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":46,"stat_sum":{"num_bytes":1143144472,"num_objects":17444,"num_object_clones":0,"num_object_copies":34888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":1069,"num_objects_unfound":0,"num_objects_dirty":17444,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":172504,"num_write_kb":3354049,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27593,"num_bytes_recovered":1469710336,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2855477248,"data_stored":2855469104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":43282,"internal_metadata":0},"log_size":372635,"ondisk_log_size":372635,"up":92,"acting":92,"num_store_stats":16},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":11},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":112,"ondisk_log_size":112,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":13,"up_from":23,"seq":98784247866,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":236824,"kb_used_data":129288,"kb_used_omap":1,"kb_used_meta":107518,"kb_avail":94135016,"statfs":{"total":96636764160,"available":96394256384,"internally_reserved":0,"allocated":132390912,"data_stored":131988805,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":110098885},"hb_peers":[0,1,2,4,6,8,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":23,"seq":98784247865,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":185320,"kb_used_data":122200,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94186520,"statfs":{"total":96636764160,"available":96446996480,"internally_reserved":0,"allocated":125132800,"data_stored":124768173,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":64616896},"hb_peers":[0,1,3,8,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":40,"seq":171798691889,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":278344,"kb_used_data":174136,"kb_used_omap":4,"kb_used_meta":104187,"kb_avail":94093496,"statfs":{"total":96636764160,"available":96351739904,"internally_reserved":0,"allocated":178315264,"data_stored":177948489,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4433,"internal_metadata":106688175},"hb_peers":[0,2,4,6,7,8,9,11,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":21,"seq":90194313274,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":283424,"kb_used_data":166160,"kb_used_omap":1,"kb_used_meta":117246,"kb_avail":94088416,"statfs":{"total":96636764160,"available":96346537984,"internally_reserved":0,"allocated":170147840,"data_stored":169751227,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":120060359},"hb_peers":[0,2,4,7,9,10,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":21,"seq":90194313277,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":370080,"kb_used_data":248656,"kb_used_omap":1,"kb_used_meta":121406,"kb_avail":94001760,"statfs":{"total":96636764160,"available":96257802240,"internally_reserved":0,"allocated":254623744,"data_stored":254233993,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":124320199},"hb_peers":[0,2,4,6,7,9,11,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":22,"seq":94489280569,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":336112,"kb_used_data":192480,"kb_used_omap":1,"kb_used_meta":143614,"kb_avail":94035728,"statfs":{"total":96636764160,"available":96292585472,"internally_reserved":0,"allocated":197099520,"data_stored":196694679,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":147061189},"hb_peers":[1,3,5,7,8,10,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,4],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":22,"seq":94489280570,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":133360,"kb_used_data":66080,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94238480,"statfs":{"total":96636764160,"available":96500203520,"internally_reserved":0,"allocated":67665920,"data_stored":67260584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,1,3,4,5,7,8,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":15,"up_from":22,"seq":94489280570,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":279708,"kb_used_data":176332,"kb_used_omap":1,"kb_used_meta":103358,"kb_avail":94092132,"statfs":{"total":96636764160,"available":96350343168,"internally_reserved":0,"allocated":180563968,"data_stored":180177982,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":105839046},"hb_peers":[0,4,5,6,7,8,10,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":22,"seq":94489280571,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":188260,"kb_used_data":120980,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94183580,"statfs":{"total":96636764160,"available":96443985920,"internally_reserved":0,"allocated":123883520,"data_stored":123489739,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[3,5,6,7,9,10,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":22,"seq":94489280570,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":151468,"kb_used_data":75868,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94220372,"statfs":{"total":96636764160,"available":96481660928,"internally_reserved":0,"allocated":77688832,"data_stored":77287961,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":77396422},"hb_peers":[0,1,3,5,7,8,10,11,12,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":22,"seq":94489280570,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":415076,"kb_used_data":267284,"kb_used_omap":1,"kb_used_meta":147774,"kb_avail":93956764,"statfs":{"total":96636764160,"available":96211726336,"internally_reserved":0,"allocated":273698816,"data_stored":273331540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":151321034},"hb_peers":[0,3,5,6,8,9,10,11,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":22,"seq":94489280570,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":131380,"kb_used_data":68260,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94240460,"statfs":{"total":96636764160,"available":96502231040,"internally_reserved":0,"allocated":69898240,"data_stored":69489468,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[1,5,6,7,9,10,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":22,"seq":94489280570,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":261040,"kb_used_data":153504,"kb_used_omap":1,"kb_used_meta":107518,"kb_avail":94110800,"statfs":{"total":96636764160,"available":96369459200,"internally_reserved":0,"allocated":157188096,"data_stored":156783032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":110098880},"hb_peers":[0,1,3,5,7,9,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":11,"up_from":22,"seq":94489280570,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":390492,"kb_used_data":278796,"kb_used_omap":1,"kb_used_meta":111678,"kb_avail":93981348,"statfs":{"total":96636764160,"available":96236900352,"internally_reserved":0,"allocated":285487104,"data_stored":285081213,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":114358724},"hb_peers":[0,1,2,4,7,8,10,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":12,"up_from":22,"seq":94489280570,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":334512,"kb_used_data":186720,"kb_used_omap":1,"kb_used_meta":147774,"kb_avail":94037328,"statfs":{"total":96636764160,"available":96294223872,"internally_reserved":0,"allocated":191201280,"data_stored":190796434,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":151321029},"hb_peers":[0,1,2,3,4,5,10,11,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":14,"up_from":142,"seq":609885356055,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":580024,"kb_used_data":507880,"kb_used_omap":42,"kb_used_meta":72085,"kb_avail":93791816,"statfs":{"total":96636764160,"available":96042819584,"internally_reserved":0,"allocated":520069120,"data_stored":519709525,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":43282,"internal_metadata":73815790},"hb_peers":[0,1,3,5,7,8,9,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":189136896,"data_stored":189136896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":164167680,"data_stored":164167680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65474560,"data_stored":65470488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":157941760,"data_stored":157941760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":119078912,"data_stored":119078912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":246611968,"data_stored":246611968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":75038720,"data_stored":75038720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":262012928,"data_stored":262012928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":64159744,"data_stored":64159744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":120061952,"data_stored":120061952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":144637952,"data_stored":144637952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":282460160,"data_stored":282460160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":181862400,"data_stored":181862400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":128647168,"data_stored":128647168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":486604800,"data_stored":486604800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":43282,"internal_metadata":0},{"poolid":3,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":167579648,"data_stored":167575576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T11:26:46.722 INFO:tasks.ceph.ceph_manager.ceph:increase pool size by 10 2024-09-13T11:26:46.723 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool set unique_pool_0 pg_num 56 2024-09-13T11:26:46.765 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-13T11:26:46.765 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:26:46.772 INFO:tasks.ceph.osd.7.smithi177.stderr:2024-09-13T11:26:46.769+0000 7f897b51d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26575) UID: 0 2024-09-13T11:26:46.785 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-13T11:26:46.872 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-09-13T11:26:46.871+0000 7f97befc9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26315) UID: 0 2024-09-13T11:26:46.918 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-13T11:26:46.918 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:26:46.935 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-09-13T11:26:46.957 INFO:teuthology.orchestra.run.smithi001.stderr:nodeep-scrub is unset 2024-09-13T11:26:46.959 INFO:tasks.ceph.osd.5.smithi092.stderr:2024-09-13T11:26:46.955+0000 7f6f02136640 -1 osd.5 221 handle_osd_map injecting map crc failure 2024-09-13T11:26:46.965 INFO:tasks.ceph.osd.7.smithi177.stderr:2024-09-13T11:26:46.961+0000 7f896ff4a640 -1 osd.7 221 handle_osd_map injecting map crc failure 2024-09-13T11:26:46.973 INFO:tasks.ceph.osd.7.smithi177.stderr:2024-09-13T11:26:46.969+0000 7f897b51d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26575) UID: 0 2024-09-13T11:26:47.074 INFO:tasks.ceph.osd.2.smithi162.stderr:2024-09-13T11:26:47.071+0000 7fe92b35a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26313) UID: 0 2024-09-13T11:26:47.114 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-13T11:26:47.114 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:26:47.134 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-09-13T11:26:47.175 INFO:tasks.ceph.osd.1.smithi092.stderr:2024-09-13T11:26:47.171+0000 7fe7789c5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33718) UID: 0 2024-09-13T11:26:47.276 INFO:tasks.ceph.osd.7.smithi177.stderr:2024-09-13T11:26:47.273+0000 7f897b51d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26575) UID: 0 2024-09-13T11:26:47.280 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T11:26:47.312 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-13T11:26:47.312 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:26:47.328 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-09-13T11:26:47.378 INFO:tasks.ceph.osd.12.smithi001.stderr:2024-09-13T11:26:47.374+0000 7f892c6e6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26314) UID: 0 2024-09-13T11:26:47.444 INFO:teuthology.orchestra.run.smithi001.stdout:osd.0: {} 2024-09-13T11:26:47.445 INFO:teuthology.orchestra.run.smithi001.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T11:26:47.455 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-13T11:26:47.455 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:26:47.470 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-13T11:26:47.471 INFO:teuthology.orchestra.run.smithi001.stdout:osd.1: {} 2024-09-13T11:26:47.471 INFO:teuthology.orchestra.run.smithi001.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T11:26:47.499 INFO:teuthology.orchestra.run.smithi001.stdout:osd.2: {} 2024-09-13T11:26:47.499 INFO:teuthology.orchestra.run.smithi001.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T11:26:47.514 INFO:tasks.ceph.osd.7.smithi177.stderr:2024-09-13T11:26:47.513+0000 7f897b51d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26575) UID: 0 2024-09-13T11:26:47.527 INFO:teuthology.orchestra.run.smithi001.stdout:osd.3: {} 2024-09-13T11:26:47.527 INFO:teuthology.orchestra.run.smithi001.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T11:26:47.554 INFO:teuthology.orchestra.run.smithi001.stdout:osd.4: {} 2024-09-13T11:26:47.555 INFO:teuthology.orchestra.run.smithi001.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T11:26:47.578 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-13T11:26:47.574+0000 7f6a68775640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26307) UID: 0 2024-09-13T11:26:47.583 INFO:teuthology.orchestra.run.smithi001.stdout:osd.5: {} 2024-09-13T11:26:47.583 INFO:teuthology.orchestra.run.smithi001.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T11:26:47.619 INFO:teuthology.orchestra.run.smithi001.stdout:osd.6: {} 2024-09-13T11:26:47.619 INFO:teuthology.orchestra.run.smithi001.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T11:26:47.625 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-09-13T11:26:47.625 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [ 2024-09-13T11:26:47.625 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:47.625 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:266345 3.a 3:5340032a:::benchmark_data_smithi177_31116_object126042:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e222)", 2024-09-13T11:26:47.625 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:26:47.620577+0000", 2024-09-13T11:26:47.625 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 0.0060482929999999997, 2024-09-13T11:26:47.625 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0060688699999999996, 2024-09-13T11:26:47.625 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:26:47.626 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:26:47.626 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "waiting for sub ops", 2024-09-13T11:26:47.626 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:26:47.626 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:26:47.626 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:47.626 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 266345 2024-09-13T11:26:47.626 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:47.626 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:26:47.626 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:47.626 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:26:47.626 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:47.620577+0000", 2024-09-13T11:26:47.626 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:47.626 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:47.626 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:47.627 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:26:47.627 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:47.620577+0000", 2024-09-13T11:26:47.627 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:47.627 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:47.627 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:47.627 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:26:47.627 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:47.620579+0000", 2024-09-13T11:26:47.627 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.663e-06 2024-09-13T11:26:47.627 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:47.627 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:47.627 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:26:47.627 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:47.620584+0000", 2024-09-13T11:26:47.627 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.2789999999999998e-06 2024-09-13T11:26:47.627 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:47.628 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:47.628 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:26:47.628 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:47.620592+0000", 2024-09-13T11:26:47.628 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.2060000000000007e-06 2024-09-13T11:26:47.628 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:47.628 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:47.628 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:26:47.628 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:47.620599+0000", 2024-09-13T11:26:47.628 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.0249999999999997e-06 2024-09-13T11:26:47.628 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:47.628 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:47.628 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:26:47.628 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:47.620654+0000", 2024-09-13T11:26:47.628 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.4514e-05 2024-09-13T11:26:47.629 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:47.629 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:47.629 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:26:47.629 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:47.620787+0000", 2024-09-13T11:26:47.629 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00013271999999999999 2024-09-13T11:26:47.629 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:47.629 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:47.629 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 7", 2024-09-13T11:26:47.629 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:47.620898+0000", 2024-09-13T11:26:47.629 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000111957 2024-09-13T11:26:47.629 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:47.629 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:47.629 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:47.630 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:47.625968+0000", 2024-09-13T11:26:47.630 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0050696079999999998 2024-09-13T11:26:47.630 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:47.630 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:26:47.630 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:47.630 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:47.630 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:47.630 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:266339 3.17 e222/85 3:efcf06c8:::benchmark_data_smithi177_31116_object126036:head v 222'9486, mlcod=222'9484)", 2024-09-13T11:26:47.630 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:26:47.621085+0000", 2024-09-13T11:26:47.630 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 0.00553997, 2024-09-13T11:26:47.630 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0056087419999999999, 2024-09-13T11:26:47.630 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:26:47.630 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:26:47.630 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:26:47.631 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:26:47.631 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:47.631 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:26:47.631 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:47.621085+0000", 2024-09-13T11:26:47.631 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:47.631 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:47.631 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:47.631 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:26:47.631 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:47.621085+0000", 2024-09-13T11:26:47.631 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:47.631 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:47.631 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:47.631 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:26:47.631 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:47.621087+0000", 2024-09-13T11:26:47.632 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0640000000000001e-06 2024-09-13T11:26:47.632 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:47.632 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:47.632 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:26:47.632 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:47.621093+0000", 2024-09-13T11:26:47.632 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.5780000000000004e-06 2024-09-13T11:26:47.632 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:47.632 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:47.632 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:26:47.632 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:47.621123+0000", 2024-09-13T11:26:47.632 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.0091e-05 2024-09-13T11:26:47.632 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:47.632 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:47.633 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:26:47.633 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:47.621130+0000", 2024-09-13T11:26:47.633 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.6939999999999998e-06 2024-09-13T11:26:47.633 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:47.633 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:47.633 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:26:47.633 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:47.621215+0000", 2024-09-13T11:26:47.633 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.4812e-05 2024-09-13T11:26:47.633 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:47.633 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:47.633 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:26:47.633 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:47.621233+0000", 2024-09-13T11:26:47.633 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.8406e-05 2024-09-13T11:26:47.633 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:47.634 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:26:47.634 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:47.634 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:47.634 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:47.634 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:266351 3.17 e222/85 3:ecb80428:::benchmark_data_smithi177_31116_object126048:head v 222'9487, mlcod=222'9484)", 2024-09-13T11:26:47.634 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:26:47.624651+0000", 2024-09-13T11:26:47.634 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 0.0019748449999999998, 2024-09-13T11:26:47.634 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0020739669999999999, 2024-09-13T11:26:47.634 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:26:47.634 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:26:47.634 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:26:47.634 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:26:47.634 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:47.634 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:26:47.635 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:47.624651+0000", 2024-09-13T11:26:47.635 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:47.635 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:47.635 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:47.635 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:26:47.635 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:47.624651+0000", 2024-09-13T11:26:47.635 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:47.635 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:47.635 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:47.635 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:26:47.636 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:47.624653+0000", 2024-09-13T11:26:47.636 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.9360000000000002e-06 2024-09-13T11:26:47.636 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:47.636 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:47.636 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:26:47.636 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:47.624661+0000", 2024-09-13T11:26:47.636 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.0970000000000006e-06 2024-09-13T11:26:47.636 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:47.636 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:47.637 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:26:47.637 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:47.624693+0000", 2024-09-13T11:26:47.637 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.2761e-05 2024-09-13T11:26:47.637 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:47.637 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:47.637 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:26:47.637 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:47.624701+0000", 2024-09-13T11:26:47.637 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.6820000000000003e-06 2024-09-13T11:26:47.637 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:47.637 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:47.638 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:26:47.638 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:47.625181+0000", 2024-09-13T11:26:47.638 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000480285 2024-09-13T11:26:47.638 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:47.638 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:47.638 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:26:47.638 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:47.625201+0000", 2024-09-13T11:26:47.638 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.0064999999999998e-05 2024-09-13T11:26:47.638 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:47.638 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:26:47.638 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:47.638 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:47.638 INFO:teuthology.orchestra.run.smithi177.stdout: ], 2024-09-13T11:26:47.638 INFO:teuthology.orchestra.run.smithi177.stdout: "num_ops": 3 2024-09-13T11:26:47.639 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-09-13T11:26:47.651 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-13T11:26:47.651 INFO:teuthology.orchestra.run.smithi001.stdout:osd.7: {} 2024-09-13T11:26:47.651 INFO:teuthology.orchestra.run.smithi001.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T11:26:47.677 INFO:teuthology.orchestra.run.smithi001.stdout:osd.8: {} 2024-09-13T11:26:47.677 INFO:teuthology.orchestra.run.smithi001.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-13T11:26:47.679 INFO:tasks.ceph.osd.12.smithi001.stderr:2024-09-13T11:26:47.678+0000 7f892c6e6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26314) UID: 0 2024-09-13T11:26:47.706 INFO:teuthology.orchestra.run.smithi001.stdout:osd.9: {} 2024-09-13T11:26:47.706 INFO:teuthology.orchestra.run.smithi001.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-13T11:26:47.733 INFO:teuthology.orchestra.run.smithi001.stdout:osd.10: {} 2024-09-13T11:26:47.734 INFO:teuthology.orchestra.run.smithi001.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-13T11:26:47.758 INFO:teuthology.orchestra.run.smithi001.stdout:osd.11: {} 2024-09-13T11:26:47.758 INFO:teuthology.orchestra.run.smithi001.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-13T11:26:47.780 INFO:tasks.ceph.osd.12.smithi001.stderr:2024-09-13T11:26:47.778+0000 7f892c6e6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26314) UID: 0 2024-09-13T11:26:47.781 INFO:teuthology.orchestra.run.smithi001.stdout:osd.12: {} 2024-09-13T11:26:47.782 INFO:teuthology.orchestra.run.smithi001.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-13T11:26:47.807 INFO:teuthology.orchestra.run.smithi001.stdout:osd.13: {} 2024-09-13T11:26:47.807 INFO:teuthology.orchestra.run.smithi001.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-13T11:26:47.830 INFO:teuthology.orchestra.run.smithi001.stdout:osd.14: {} 2024-09-13T11:26:47.830 INFO:teuthology.orchestra.run.smithi001.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-13T11:26:47.852 INFO:teuthology.orchestra.run.smithi001.stdout:osd.15: {} 2024-09-13T11:26:47.852 INFO:teuthology.orchestra.run.smithi001.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-13T11:26:47.855 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-09-13T11:26:47.855 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [], 2024-09-13T11:26:47.856 INFO:teuthology.orchestra.run.smithi177.stdout: "complaint_time": 30, 2024-09-13T11:26:47.856 INFO:teuthology.orchestra.run.smithi177.stdout: "num_blocked_ops": 0 2024-09-13T11:26:47.856 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-09-13T11:26:47.870 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-13T11:26:47.907 INFO:tasks.ceph.osd.4.smithi001.stderr:2024-09-13T11:26:47.906+0000 7ff8e0218640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26310) UID: 0 2024-09-13T11:26:47.959 INFO:teuthology.orchestra.run.smithi001.stderr:set pool 3 pg_num to 56 2024-09-13T11:26:47.960 INFO:tasks.ceph.osd.3.smithi177.stderr:2024-09-13T11:26:47.957+0000 7f10a45d5640 -1 osd.3 222 handle_osd_map injecting map crc failure 2024-09-13T11:26:47.965 INFO:tasks.ceph.osd.12.smithi001.stderr:2024-09-13T11:26:47.962+0000 7f891f10f640 -1 osd.12 222 handle_osd_map injecting map crc failure 2024-09-13T11:26:47.969 INFO:tasks.ceph.osd.14.smithi162.stderr:2024-09-13T11:26:47.967+0000 7f3dea330640 -1 osd.14 222 handle_osd_map injecting map crc failure 2024-09-13T11:26:47.971 INFO:tasks.ceph.osd.7.smithi177.stderr:2024-09-13T11:26:47.969+0000 7f896ff4a640 -1 osd.7 222 handle_osd_map injecting map crc failure 2024-09-13T11:26:47.972 INFO:tasks.ceph.osd.14.smithi162.stderr:2024-09-13T11:26:47.967+0000 7f3dea330640 -1 osd.14 222 handle_osd_map injecting map crc failure 2024-09-13T11:26:47.999 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-09-13T11:26:47.999 INFO:teuthology.orchestra.run.smithi177.stdout: "size": 20, 2024-09-13T11:26:47.999 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 600, 2024-09-13T11:26:47.999 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [ 2024-09-13T11:26:47.999 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:47.999 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPush(3.17 75/70 [PushOp(3:eda94455:::benchmark_data_smithi177_31116_object13017:head, version: 50'788, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:eda94455:::benchmark_data_smithi177_31116_object13017:head@50'788, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:26:48.000 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:31.038029+0000", 2024-09-13T11:26:48.000 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 196.96053769900001, 2024-09-13T11:26:48.000 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.42064623499999998, 2024-09-13T11:26:48.000 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:26:48.000 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:26:48.000 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:26:48.000 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:26:48.000 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.000 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:26:48.000 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.038029+0000", 2024-09-13T11:26:48.000 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:48.000 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.001 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.001 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:26:48.001 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.038029+0000", 2024-09-13T11:26:48.001 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:48.001 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.001 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.001 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:26:48.001 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.038030+0000", 2024-09-13T11:26:48.001 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.499e-06 2024-09-13T11:26:48.001 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.001 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.001 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:26:48.001 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.038088+0000", 2024-09-13T11:26:48.002 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.7771e-05 2024-09-13T11:26:48.002 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.002 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.002 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:26:48.002 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.457025+0000", 2024-09-13T11:26:48.002 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.41893754900000002 2024-09-13T11:26:48.002 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.002 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.002 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:26:48.002 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.457073+0000", 2024-09-13T11:26:48.002 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.7450999999999997e-05 2024-09-13T11:26:48.002 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.002 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.003 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:26:48.003 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.457304+0000", 2024-09-13T11:26:48.003 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000231351 2024-09-13T11:26:48.003 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.003 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.003 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:26:48.003 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.457331+0000", 2024-09-13T11:26:48.003 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.6372000000000001e-05 2024-09-13T11:26:48.003 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.003 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.003 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:26:48.003 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.458675+0000", 2024-09-13T11:26:48.003 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001344242 2024-09-13T11:26:48.004 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:48.004 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:26:48.004 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:48.004 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.004 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.004 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPush(3.17 83/70 [PushOp(3:eb574268:::benchmark_data_smithi177_31116_object40622:head, version: 64'2475, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:eb574268:::benchmark_data_smithi177_31116_object40622:head@64'2475, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:26:48.004 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:40.760042+0000", 2024-09-13T11:26:48.004 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 187.238523982, 2024-09-13T11:26:48.004 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.30544642100000002, 2024-09-13T11:26:48.004 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:26:48.004 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:26:48.004 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:26:48.004 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:26:48.005 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.005 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:26:48.005 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:40.760042+0000", 2024-09-13T11:26:48.005 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:48.005 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.005 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.005 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:26:48.005 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:40.760042+0000", 2024-09-13T11:26:48.005 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:48.005 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.005 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.005 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:26:48.005 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:40.760044+0000", 2024-09-13T11:26:48.006 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.511e-06 2024-09-13T11:26:48.006 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.006 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.006 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:26:48.006 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:40.760149+0000", 2024-09-13T11:26:48.006 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000105473 2024-09-13T11:26:48.006 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.006 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.006 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:26:48.006 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.064465+0000", 2024-09-13T11:26:48.006 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.30431613800000001 2024-09-13T11:26:48.006 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.006 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.006 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:26:48.007 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.064479+0000", 2024-09-13T11:26:48.007 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3586e-05 2024-09-13T11:26:48.007 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.007 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.007 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:26:48.007 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.064553+0000", 2024-09-13T11:26:48.007 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.3646000000000006e-05 2024-09-13T11:26:48.007 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.007 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.007 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:26:48.007 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.064568+0000", 2024-09-13T11:26:48.007 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.52e-05 2024-09-13T11:26:48.007 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.008 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.008 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:26:48.008 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.065489+0000", 2024-09-13T11:26:48.008 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00092086700000000002 2024-09-13T11:26:48.008 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:48.008 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:26:48.008 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:48.008 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.008 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.008 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPushReply(3.1a 83/78 [PushReplyOp(3:5c3834bd:::benchmark_data_smithi177_31116_object8669:head)])", 2024-09-13T11:26:48.008 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:40.798236+0000", 2024-09-13T11:26:48.008 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 187.200330552, 2024-09-13T11:26:48.008 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.29701520999999997, 2024-09-13T11:26:48.009 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:26:48.009 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:26:48.009 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-13T11:26:48.009 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:26:48.009 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.009 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:26:48.009 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:40.798236+0000", 2024-09-13T11:26:48.009 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:48.009 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.009 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.009 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:26:48.009 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:40.798236+0000", 2024-09-13T11:26:48.009 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:48.009 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.010 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.010 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:26:48.010 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:40.798237+0000", 2024-09-13T11:26:48.010 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.0500000000000002e-07 2024-09-13T11:26:48.010 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.010 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.010 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:26:48.010 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:40.798240+0000", 2024-09-13T11:26:48.010 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.4819999999999999e-06 2024-09-13T11:26:48.010 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.010 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.010 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:26:48.010 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.093360+0000", 2024-09-13T11:26:48.010 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.29512009 2024-09-13T11:26:48.011 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.011 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.011 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:26:48.011 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.093372+0000", 2024-09-13T11:26:48.011 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.2221e-05 2024-09-13T11:26:48.011 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.011 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.011 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:26:48.011 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.093482+0000", 2024-09-13T11:26:48.011 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000109571 2024-09-13T11:26:48.011 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.011 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.011 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:26:48.011 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.095251+0000", 2024-09-13T11:26:48.012 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001768941 2024-09-13T11:26:48.012 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:48.012 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:26:48.012 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:48.012 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.012 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.012 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPushReply(3.1a 83/78 [PushReplyOp(3:58c7ffc0:::benchmark_data_smithi177_31116_object8738:head)])", 2024-09-13T11:26:48.012 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:40.805941+0000", 2024-09-13T11:26:48.012 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 187.192625298, 2024-09-13T11:26:48.012 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.30315678499999998, 2024-09-13T11:26:48.012 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:26:48.012 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:26:48.013 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-13T11:26:48.013 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:26:48.013 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.013 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:26:48.013 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:40.805941+0000", 2024-09-13T11:26:48.013 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:48.013 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.013 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.013 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:26:48.013 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:40.805941+0000", 2024-09-13T11:26:48.013 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:48.013 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.013 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.014 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:26:48.014 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:40.805942+0000", 2024-09-13T11:26:48.014 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3769999999999999e-06 2024-09-13T11:26:48.014 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.014 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.014 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:26:48.014 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:40.805947+0000", 2024-09-13T11:26:48.014 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.8980000000000004e-06 2024-09-13T11:26:48.014 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.014 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.014 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:26:48.014 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.106505+0000", 2024-09-13T11:26:48.014 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.30055777700000003 2024-09-13T11:26:48.014 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.015 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.015 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:26:48.015 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.106512+0000", 2024-09-13T11:26:48.015 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.3749999999999997e-06 2024-09-13T11:26:48.015 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.015 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.015 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:26:48.015 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.106566+0000", 2024-09-13T11:26:48.015 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.3103999999999998e-05 2024-09-13T11:26:48.015 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.015 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.015 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:26:48.015 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.109098+0000", 2024-09-13T11:26:48.016 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0025322539999999998 2024-09-13T11:26:48.016 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:48.016 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:26:48.016 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:48.016 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.016 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.016 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPushReply(3.1a 83/78 [PushReplyOp(3:58c7ffc0:::benchmark_data_smithi177_31116_object8738:head)])", 2024-09-13T11:26:48.016 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:40.805973+0000", 2024-09-13T11:26:48.016 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 187.19259329499999, 2024-09-13T11:26:48.016 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.28729672099999998, 2024-09-13T11:26:48.016 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:26:48.016 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:26:48.016 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-13T11:26:48.017 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:26:48.017 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.017 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:26:48.017 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:40.805973+0000", 2024-09-13T11:26:48.017 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:48.017 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.017 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.017 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:26:48.017 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:40.805973+0000", 2024-09-13T11:26:48.017 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:48.017 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.017 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.017 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:26:48.017 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:40.805974+0000", 2024-09-13T11:26:48.018 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.8599999999999998e-07 2024-09-13T11:26:48.018 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.018 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.018 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:26:48.018 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:40.805976+0000", 2024-09-13T11:26:48.018 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.2809999999999998e-06 2024-09-13T11:26:48.018 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.018 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.018 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:26:48.018 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.091588+0000", 2024-09-13T11:26:48.018 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.28561173899999998 2024-09-13T11:26:48.018 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.019 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.019 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:26:48.019 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.091596+0000", 2024-09-13T11:26:48.019 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.0520000000000002e-06 2024-09-13T11:26:48.019 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.019 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.019 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:26:48.019 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.091907+0000", 2024-09-13T11:26:48.019 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00031179599999999998 2024-09-13T11:26:48.019 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.019 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.019 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:26:48.019 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.093270+0000", 2024-09-13T11:26:48.020 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0013622669999999999 2024-09-13T11:26:48.020 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:48.020 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:26:48.020 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:48.020 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.020 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.020 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPushReply(3.1a 84/78 [PushReplyOp(3:581ea97e:::benchmark_data_smithi177_31116_object10255:head)])", 2024-09-13T11:26:48.020 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:41.514228+0000", 2024-09-13T11:26:48.020 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 186.484337807, 2024-09-13T11:26:48.020 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.40695564200000001, 2024-09-13T11:26:48.020 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:26:48.020 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:26:48.020 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-13T11:26:48.021 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:26:48.021 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.021 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:26:48.021 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.514228+0000", 2024-09-13T11:26:48.021 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:48.021 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.021 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.021 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:26:48.021 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.514228+0000", 2024-09-13T11:26:48.021 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:48.021 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.021 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.021 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:26:48.021 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.514230+0000", 2024-09-13T11:26:48.022 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.499e-06 2024-09-13T11:26:48.022 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.022 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.022 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:26:48.022 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.514236+0000", 2024-09-13T11:26:48.022 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.5389999999999998e-06 2024-09-13T11:26:48.022 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.022 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.022 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:26:48.022 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.919787+0000", 2024-09-13T11:26:48.022 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.40555152999999999 2024-09-13T11:26:48.022 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.022 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.023 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:26:48.023 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.919796+0000", 2024-09-13T11:26:48.023 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.4829999999999999e-06 2024-09-13T11:26:48.023 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.023 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.023 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:26:48.023 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.920366+0000", 2024-09-13T11:26:48.023 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00057061700000000004 2024-09-13T11:26:48.023 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.023 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.023 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:26:48.023 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.921184+0000", 2024-09-13T11:26:48.023 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00081797399999999996 2024-09-13T11:26:48.023 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:48.024 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:26:48.024 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:48.024 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.024 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.024 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:60427 3.14 3:2afea229:::benchmark_data_smithi177_31116_object60426:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e85)", 2024-09-13T11:26:48.024 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:42.187231+0000", 2024-09-13T11:26:48.024 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 185.81133493799999, 2024-09-13T11:26:48.024 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.83900596999999999, 2024-09-13T11:26:48.024 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:26:48.024 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:26:48.024 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:48.024 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:26:48.024 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:26:48.025 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:48.025 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 60427 2024-09-13T11:26:48.025 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.025 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:26:48.025 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.025 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:26:48.025 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.187231+0000", 2024-09-13T11:26:48.025 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:48.025 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.025 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.025 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:26:48.025 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.187231+0000", 2024-09-13T11:26:48.025 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:48.026 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.026 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.026 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:26:48.026 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.187235+0000", 2024-09-13T11:26:48.026 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.3579999999999999e-06 2024-09-13T11:26:48.026 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.026 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.026 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:26:48.026 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.187261+0000", 2024-09-13T11:26:48.026 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.6089e-05 2024-09-13T11:26:48.026 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.026 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.026 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:26:48.027 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.187275+0000", 2024-09-13T11:26:48.027 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.4653e-05 2024-09-13T11:26:48.027 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.027 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.027 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:26:48.027 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.187282+0000", 2024-09-13T11:26:48.027 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.8870000000000003e-06 2024-09-13T11:26:48.027 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.027 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.027 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:26:48.027 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.187336+0000", 2024-09-13T11:26:48.027 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.3875999999999997e-05 2024-09-13T11:26:48.027 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.028 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.028 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:26:48.028 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.187480+0000", 2024-09-13T11:26:48.028 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00014330899999999999 2024-09-13T11:26:48.028 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.028 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.028 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 14", 2024-09-13T11:26:48.028 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.187658+0000", 2024-09-13T11:26:48.028 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00017861900000000001 2024-09-13T11:26:48.028 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.028 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.028 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:26:48.028 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.189353+0000", 2024-09-13T11:26:48.028 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001695144 2024-09-13T11:26:48.029 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.029 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.029 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:48.029 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:43.026081+0000", 2024-09-13T11:26:48.029 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.83672736199999997 2024-09-13T11:26:48.029 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.029 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.029 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:26:48.029 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:43.026169+0000", 2024-09-13T11:26:48.029 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.8411000000000004e-05 2024-09-13T11:26:48.029 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.029 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.030 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:26:48.030 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:43.026237+0000", 2024-09-13T11:26:48.030 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.8262000000000005e-05 2024-09-13T11:26:48.030 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:48.030 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:26:48.030 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:48.030 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.030 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.030 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop_reply(client.4665.0:60427 3.14 e85/84 ondisk, result = 0)", 2024-09-13T11:26:48.030 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:42.190914+0000", 2024-09-13T11:26:48.030 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 185.80765227399999, 2024-09-13T11:26:48.030 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.83534071499999996, 2024-09-13T11:26:48.030 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:26:48.030 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:26:48.031 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:26:48.031 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:26:48.031 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.031 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:26:48.031 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.190914+0000", 2024-09-13T11:26:48.031 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:48.031 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.031 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.031 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:26:48.031 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.190914+0000", 2024-09-13T11:26:48.031 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:48.031 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.031 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.032 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:26:48.032 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.190916+0000", 2024-09-13T11:26:48.032 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.84e-06 2024-09-13T11:26:48.032 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.032 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.032 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:26:48.032 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.190921+0000", 2024-09-13T11:26:48.032 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.2039999999999996e-06 2024-09-13T11:26:48.032 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.032 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.032 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:26:48.032 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:43.025926+0000", 2024-09-13T11:26:48.032 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.83500542 2024-09-13T11:26:48.033 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.033 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.033 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:26:48.033 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:43.025937+0000", 2024-09-13T11:26:48.033 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0597e-05 2024-09-13T11:26:48.033 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.033 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.033 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:26:48.033 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:43.026052+0000", 2024-09-13T11:26:48.033 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000114838 2024-09-13T11:26:48.033 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.033 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.034 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:26:48.034 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:43.026070+0000", 2024-09-13T11:26:48.034 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.8131e-05 2024-09-13T11:26:48.034 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.034 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.034 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:26:48.034 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:43.026255+0000", 2024-09-13T11:26:48.034 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000184685 2024-09-13T11:26:48.034 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:48.034 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:26:48.034 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:48.034 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.034 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.035 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPushReply(3.1d 89/78 [PushReplyOp(3:bc1b7cfb:::benchmark_data_smithi177_31116_object22629:head)])", 2024-09-13T11:26:48.035 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:46.663106+0000", 2024-09-13T11:26:48.035 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 181.33546009099999, 2024-09-13T11:26:48.035 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.26645533199999999, 2024-09-13T11:26:48.035 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:26:48.035 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:26:48.035 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-13T11:26:48.035 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:26:48.035 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.035 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:26:48.035 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.663106+0000", 2024-09-13T11:26:48.035 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:48.035 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.035 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.036 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:26:48.036 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.663106+0000", 2024-09-13T11:26:48.036 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:48.036 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.036 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.036 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:26:48.036 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.663108+0000", 2024-09-13T11:26:48.036 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.0140000000000001e-06 2024-09-13T11:26:48.036 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.036 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.036 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:26:48.036 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.663116+0000", 2024-09-13T11:26:48.037 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.5440000000000001e-06 2024-09-13T11:26:48.037 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.037 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.037 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:26:48.037 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.928111+0000", 2024-09-13T11:26:48.037 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.26499566899999999 2024-09-13T11:26:48.037 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.037 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.037 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:26:48.037 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.928135+0000", 2024-09-13T11:26:48.037 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.4003e-05 2024-09-13T11:26:48.037 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.037 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.038 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:26:48.038 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.928559+0000", 2024-09-13T11:26:48.038 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000423318 2024-09-13T11:26:48.038 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.038 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.038 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:26:48.038 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.929562+0000", 2024-09-13T11:26:48.038 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0010027840000000001 2024-09-13T11:26:48.038 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:48.038 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:26:48.038 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:48.038 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.038 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.038 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "rep_scrubmap(3.14 e84 from shard 14)", 2024-09-13T11:26:48.039 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:48.313615+0000", 2024-09-13T11:26:48.039 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 179.68495129799999, 2024-09-13T11:26:48.039 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.35633836099999999, 2024-09-13T11:26:48.039 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:26:48.039 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:26:48.039 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:26:48.039 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:26:48.039 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.039 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:26:48.039 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.313615+0000", 2024-09-13T11:26:48.039 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:48.039 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.039 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.039 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:26:48.040 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.313615+0000", 2024-09-13T11:26:48.040 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:48.040 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.040 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.040 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:26:48.040 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.313617+0000", 2024-09-13T11:26:48.040 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.7990000000000001e-06 2024-09-13T11:26:48.040 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.040 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.040 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:26:48.040 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.313688+0000", 2024-09-13T11:26:48.040 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.1481000000000002e-05 2024-09-13T11:26:48.040 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.041 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.041 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:26:48.041 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.669785+0000", 2024-09-13T11:26:48.041 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.356096466 2024-09-13T11:26:48.041 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.041 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.041 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:26:48.041 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.669792+0000", 2024-09-13T11:26:48.041 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.1740000000000003e-06 2024-09-13T11:26:48.041 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.041 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.041 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:26:48.041 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.669839+0000", 2024-09-13T11:26:48.041 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.7323000000000001e-05 2024-09-13T11:26:48.042 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.042 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.042 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:26:48.042 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.669847+0000", 2024-09-13T11:26:48.042 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.0469999999999994e-06 2024-09-13T11:26:48.042 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.042 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.042 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:26:48.042 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.669953+0000", 2024-09-13T11:26:48.042 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00010607099999999999 2024-09-13T11:26:48.042 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:48.042 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:26:48.042 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:48.042 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.043 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.043 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:110268 3.14 3:2997c4c5:::benchmark_data_smithi177_31116_object110267:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119)", 2024-09-13T11:26:48.043 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:25.359386+0000", 2024-09-13T11:26:48.043 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 142.639179982, 2024-09-13T11:26:48.043 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.29072703, 2024-09-13T11:26:48.043 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:26:48.043 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:26:48.043 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:48.043 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:26:48.043 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:26:48.043 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:48.043 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 110268 2024-09-13T11:26:48.043 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.044 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:26:48.044 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.044 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:26:48.044 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.359386+0000", 2024-09-13T11:26:48.044 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:48.044 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.044 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.044 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:26:48.044 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.359386+0000", 2024-09-13T11:26:48.044 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:48.044 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.044 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.044 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:26:48.045 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.359389+0000", 2024-09-13T11:26:48.045 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.8600000000000001e-06 2024-09-13T11:26:48.045 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.045 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.045 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:26:48.045 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.359424+0000", 2024-09-13T11:26:48.045 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.4755999999999997e-05 2024-09-13T11:26:48.045 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.045 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.045 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:26:48.045 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.359445+0000", 2024-09-13T11:26:48.045 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.1104999999999999e-05 2024-09-13T11:26:48.045 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.045 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.046 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:26:48.046 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.359460+0000", 2024-09-13T11:26:48.046 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5396999999999998e-05 2024-09-13T11:26:48.046 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.046 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.046 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:26:48.046 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.359581+0000", 2024-09-13T11:26:48.046 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000120455 2024-09-13T11:26:48.046 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.046 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.046 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:26:48.046 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.359827+0000", 2024-09-13T11:26:48.046 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00024571 2024-09-13T11:26:48.046 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.047 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.047 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 14", 2024-09-13T11:26:48.047 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.360084+0000", 2024-09-13T11:26:48.047 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00025787999999999999 2024-09-13T11:26:48.047 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.047 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.047 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:26:48.047 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.362243+0000", 2024-09-13T11:26:48.047 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0021581579999999999 2024-09-13T11:26:48.047 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.047 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.047 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:48.047 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.650034+0000", 2024-09-13T11:26:48.047 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.287791662 2024-09-13T11:26:48.048 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.048 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.048 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:26:48.048 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.650077+0000", 2024-09-13T11:26:48.048 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.2367999999999998e-05 2024-09-13T11:26:48.048 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.048 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.048 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:26:48.048 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.650113+0000", 2024-09-13T11:26:48.048 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.6678999999999999e-05 2024-09-13T11:26:48.048 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:48.048 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:26:48.048 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:48.049 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.049 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.049 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:110279 3.14 3:2b17a349:::benchmark_data_smithi177_31116_object110278:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119)", 2024-09-13T11:26:48.049 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:25.645137+0000", 2024-09-13T11:26:48.049 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 142.35342914399999, 2024-09-13T11:26:48.049 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.28172927800000003, 2024-09-13T11:26:48.049 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:26:48.049 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:26:48.049 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:48.049 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:26:48.049 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:26:48.049 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:48.049 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 110279 2024-09-13T11:26:48.049 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.049 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:26:48.050 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.050 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:26:48.050 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.645137+0000", 2024-09-13T11:26:48.050 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:48.050 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.050 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.050 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:26:48.050 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.645137+0000", 2024-09-13T11:26:48.050 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:48.050 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.050 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.050 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:26:48.050 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.645140+0000", 2024-09-13T11:26:48.050 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.0939999999999999e-06 2024-09-13T11:26:48.050 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.051 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.051 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:26:48.051 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.645184+0000", 2024-09-13T11:26:48.051 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.3412999999999999e-05 2024-09-13T11:26:48.051 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.051 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.051 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:26:48.051 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.645206+0000", 2024-09-13T11:26:48.051 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.1997e-05 2024-09-13T11:26:48.051 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.051 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.051 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:26:48.051 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.645216+0000", 2024-09-13T11:26:48.052 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0740999999999999e-05 2024-09-13T11:26:48.052 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.052 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.052 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:26:48.052 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.645397+0000", 2024-09-13T11:26:48.052 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00018077600000000001 2024-09-13T11:26:48.052 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.052 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.052 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:26:48.052 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.645638+0000", 2024-09-13T11:26:48.052 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000241091 2024-09-13T11:26:48.052 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.052 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.053 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 14", 2024-09-13T11:26:48.053 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.645922+0000", 2024-09-13T11:26:48.053 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000283565 2024-09-13T11:26:48.053 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.053 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.053 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:26:48.053 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.648980+0000", 2024-09-13T11:26:48.053 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00305851 2024-09-13T11:26:48.053 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.053 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.053 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:48.053 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.926702+0000", 2024-09-13T11:26:48.053 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.27772199800000003 2024-09-13T11:26:48.053 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.054 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.054 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:26:48.054 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.926796+0000", 2024-09-13T11:26:48.054 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.4158000000000002e-05 2024-09-13T11:26:48.054 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.054 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.054 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:26:48.054 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.926866+0000", 2024-09-13T11:26:48.054 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.9935e-05 2024-09-13T11:26:48.054 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:48.054 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:26:48.054 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:48.054 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.054 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.055 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPush(3.15 143/122 [PushOp(3:ae52bb45:::benchmark_data_smithi177_31116_object61893:head, version: 87'3272, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:ae52bb45:::benchmark_data_smithi177_31116_object61893:head@87'3272, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:26:48.055 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:57.633939+0000", 2024-09-13T11:26:48.055 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 110.364627161, 2024-09-13T11:26:48.055 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.339405067, 2024-09-13T11:26:48.055 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:26:48.055 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:26:48.055 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:26:48.055 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:26:48.055 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.055 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:26:48.055 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:57.633939+0000", 2024-09-13T11:26:48.055 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:48.055 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.055 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.056 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:26:48.056 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:57.633939+0000", 2024-09-13T11:26:48.056 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:48.056 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.056 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.056 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:26:48.056 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:57.633941+0000", 2024-09-13T11:26:48.056 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.911e-06 2024-09-13T11:26:48.056 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.056 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.056 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:26:48.056 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:57.633994+0000", 2024-09-13T11:26:48.056 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.2753999999999997e-05 2024-09-13T11:26:48.057 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.057 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.057 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:26:48.057 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:57.968403+0000", 2024-09-13T11:26:48.057 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.334409336 2024-09-13T11:26:48.057 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.057 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.057 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:26:48.057 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:57.968451+0000", 2024-09-13T11:26:48.057 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.7525000000000003e-05 2024-09-13T11:26:48.057 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.057 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.057 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:26:48.057 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:57.968660+0000", 2024-09-13T11:26:48.058 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000209204 2024-09-13T11:26:48.058 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.058 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.058 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:26:48.058 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:57.968680+0000", 2024-09-13T11:26:48.058 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.9639999999999998e-05 2024-09-13T11:26:48.058 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.058 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.058 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:26:48.058 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:57.973344+0000", 2024-09-13T11:26:48.058 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0046646969999999998 2024-09-13T11:26:48.058 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:48.058 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:26:48.058 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:48.059 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.059 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.059 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:140658 3.17 e144/85 3:ece5a1b3:::benchmark_data_smithi177_31116_object355:head v 144'5654, mlcod=144'5652)", 2024-09-13T11:26:48.059 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:58.247792+0000", 2024-09-13T11:26:48.059 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 109.750774683, 2024-09-13T11:26:48.059 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.29148369699999999, 2024-09-13T11:26:48.059 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:26:48.059 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:26:48.059 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:48.059 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:26:48.059 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.059 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:26:48.059 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:58.247792+0000", 2024-09-13T11:26:48.059 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:48.060 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.060 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.060 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:26:48.060 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:58.247792+0000", 2024-09-13T11:26:48.060 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:48.060 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.060 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.060 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:26:48.060 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:58.247793+0000", 2024-09-13T11:26:48.060 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.347e-06 2024-09-13T11:26:48.060 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.060 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.060 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:26:48.061 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:58.247799+0000", 2024-09-13T11:26:48.061 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.5239999999999997e-06 2024-09-13T11:26:48.061 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.061 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.061 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:26:48.061 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:58.535226+0000", 2024-09-13T11:26:48.061 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.28742604900000002 2024-09-13T11:26:48.061 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.061 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.061 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:26:48.061 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:58.535238+0000", 2024-09-13T11:26:48.061 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.2843000000000001e-05 2024-09-13T11:26:48.061 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.061 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.062 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:26:48.062 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:58.535393+0000", 2024-09-13T11:26:48.062 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00015481699999999999 2024-09-13T11:26:48.062 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.062 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.062 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:26:48.062 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:58.535434+0000", 2024-09-13T11:26:48.062 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.1152999999999997e-05 2024-09-13T11:26:48.062 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.062 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.062 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:26:48.062 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:58.539191+0000", 2024-09-13T11:26:48.062 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0037571520000000001 2024-09-13T11:26:48.062 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.062 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.063 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:26:48.063 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:58.539275+0000", 2024-09-13T11:26:48.063 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.3812000000000003e-05 2024-09-13T11:26:48.063 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:48.063 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:26:48.063 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:48.063 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.063 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.063 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPush(3.15 144/122 [PushOp(3:a97ba025:::benchmark_data_smithi177_31116_object64775:head, version: 90'3365, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:a97ba025:::benchmark_data_smithi177_31116_object64775:head@90'3365, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:26:48.063 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:58.825014+0000", 2024-09-13T11:26:48.063 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 109.173552055, 2024-09-13T11:26:48.063 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.23868276499999999, 2024-09-13T11:26:48.063 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:26:48.064 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:26:48.064 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:26:48.064 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:26:48.064 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.064 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:26:48.064 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:58.825014+0000", 2024-09-13T11:26:48.064 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:48.064 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.064 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.064 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:26:48.064 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:58.825014+0000", 2024-09-13T11:26:48.064 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:48.064 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.064 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.065 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:26:48.065 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:58.825017+0000", 2024-09-13T11:26:48.065 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.7630000000000001e-06 2024-09-13T11:26:48.065 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.065 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.065 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:26:48.065 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:58.825099+0000", 2024-09-13T11:26:48.065 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.2035999999999999e-05 2024-09-13T11:26:48.065 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.065 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.065 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:26:48.065 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:59.059557+0000", 2024-09-13T11:26:48.065 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.23445790399999999 2024-09-13T11:26:48.065 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.066 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.066 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:26:48.066 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:59.059609+0000", 2024-09-13T11:26:48.066 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.2550999999999999e-05 2024-09-13T11:26:48.066 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.066 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.066 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:26:48.066 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:59.060117+0000", 2024-09-13T11:26:48.066 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00050754299999999995 2024-09-13T11:26:48.066 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.066 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.066 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:26:48.066 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:59.060139+0000", 2024-09-13T11:26:48.066 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.2306e-05 2024-09-13T11:26:48.067 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.067 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.067 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:26:48.067 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:59.063697+0000", 2024-09-13T11:26:48.067 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0035576620000000001 2024-09-13T11:26:48.067 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:48.067 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:26:48.067 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:48.067 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.067 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.067 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:211295 3.2 e185/122 3:433aa750:::benchmark_data_smithi177_31116_object70992:head v 185'5563, mlcod=185'5560)", 2024-09-13T11:26:48.067 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:25:55.461013+0000", 2024-09-13T11:26:48.067 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 52.537552996999999, 2024-09-13T11:26:48.068 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.28679000700000001, 2024-09-13T11:26:48.068 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:26:48.068 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:26:48.068 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:48.068 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:26:48.068 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.068 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:26:48.068 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:55.461013+0000", 2024-09-13T11:26:48.068 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:48.068 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.068 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.068 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:26:48.068 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:55.461013+0000", 2024-09-13T11:26:48.068 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:48.069 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.069 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.069 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:26:48.069 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:55.461015+0000", 2024-09-13T11:26:48.069 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5600000000000001e-06 2024-09-13T11:26:48.069 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.069 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.069 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:26:48.069 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:55.461024+0000", 2024-09-13T11:26:48.069 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.1640000000000007e-06 2024-09-13T11:26:48.069 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.069 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.069 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:26:48.070 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:55.746330+0000", 2024-09-13T11:26:48.070 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.28530630000000001 2024-09-13T11:26:48.070 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.070 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.070 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:26:48.070 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:55.746335+0000", 2024-09-13T11:26:48.070 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.8119999999999998e-06 2024-09-13T11:26:48.070 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.070 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.070 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:26:48.070 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:55.746377+0000", 2024-09-13T11:26:48.070 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.1947999999999999e-05 2024-09-13T11:26:48.070 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.070 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.071 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:26:48.071 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:55.746389+0000", 2024-09-13T11:26:48.071 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.2237e-05 2024-09-13T11:26:48.071 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.071 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.071 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:26:48.071 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:55.747767+0000", 2024-09-13T11:26:48.071 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0013774779999999999 2024-09-13T11:26:48.071 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.071 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.071 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:26:48.071 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:55.747803+0000", 2024-09-13T11:26:48.071 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.6507999999999997e-05 2024-09-13T11:26:48.072 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:48.072 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:26:48.072 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:48.072 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.072 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.072 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:221537 3.14 3.d5a7b194 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e191)", 2024-09-13T11:26:48.072 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:26:04.320591+0000", 2024-09-13T11:26:48.072 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 43.677975138999997, 2024-09-13T11:26:48.072 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.72342120499999996, 2024-09-13T11:26:48.072 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:26:48.072 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:26:48.072 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-13T11:26:48.072 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:26:48.073 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:26:48.073 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:48.073 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 221537 2024-09-13T11:26:48.073 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.073 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:26:48.073 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.073 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:26:48.073 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:04.320591+0000", 2024-09-13T11:26:48.073 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:48.073 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.073 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.073 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:26:48.073 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:04.320591+0000", 2024-09-13T11:26:48.074 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:48.074 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.074 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.074 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:26:48.074 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:04.320595+0000", 2024-09-13T11:26:48.074 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.6500000000000002e-06 2024-09-13T11:26:48.074 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.074 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.074 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:26:48.074 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:04.320604+0000", 2024-09-13T11:26:48.074 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.5400000000000001e-06 2024-09-13T11:26:48.074 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.074 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.074 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:26:48.075 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:04.320630+0000", 2024-09-13T11:26:48.075 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.5340000000000001e-05 2024-09-13T11:26:48.075 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.075 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.075 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:26:48.075 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:04.320644+0000", 2024-09-13T11:26:48.075 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.4008e-05 2024-09-13T11:26:48.075 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.075 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.075 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:26:48.075 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:04.320708+0000", 2024-09-13T11:26:48.075 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.4268000000000002e-05 2024-09-13T11:26:48.075 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.076 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.076 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for peered", 2024-09-13T11:26:48.076 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:04.320716+0000", 2024-09-13T11:26:48.076 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.2139999999999996e-06 2024-09-13T11:26:48.076 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.076 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.076 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:26:48.076 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:05.043955+0000", 2024-09-13T11:26:48.076 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.72323873599999999 2024-09-13T11:26:48.076 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.076 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.076 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:26:48.076 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:05.044012+0000", 2024-09-13T11:26:48.076 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.7448999999999998e-05 2024-09-13T11:26:48.076 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:48.077 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:26:48.077 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:48.077 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.077 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.077 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:221537 3.14 3:298de5ab:::benchmark_data_smithi177_31116_object81234:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e193)", 2024-09-13T11:26:48.077 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:26:06.066780+0000", 2024-09-13T11:26:48.077 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 41.931786463999998, 2024-09-13T11:26:48.077 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.26485703900000002, 2024-09-13T11:26:48.077 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:26:48.077 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:26:48.077 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:48.077 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:26:48.077 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:26:48.077 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:48.077 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 221537 2024-09-13T11:26:48.078 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.078 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:26:48.078 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.078 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:26:48.078 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:06.066780+0000", 2024-09-13T11:26:48.078 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:48.078 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.078 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.078 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:26:48.078 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:06.066780+0000", 2024-09-13T11:26:48.078 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:48.078 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.078 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.078 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:26:48.079 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:06.066784+0000", 2024-09-13T11:26:48.079 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.8e-06 2024-09-13T11:26:48.079 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.079 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.079 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:26:48.079 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:06.066796+0000", 2024-09-13T11:26:48.079 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.2317000000000001e-05 2024-09-13T11:26:48.079 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.079 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.079 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:26:48.079 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:06.066834+0000", 2024-09-13T11:26:48.079 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.765e-05 2024-09-13T11:26:48.079 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.079 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.080 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:26:48.080 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:06.066855+0000", 2024-09-13T11:26:48.080 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.1379e-05 2024-09-13T11:26:48.080 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.080 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.080 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:26:48.080 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:06.067121+0000", 2024-09-13T11:26:48.080 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00026621700000000002 2024-09-13T11:26:48.080 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.080 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.080 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for peered", 2024-09-13T11:26:48.080 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:06.067165+0000", 2024-09-13T11:26:48.080 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.3442999999999997e-05 2024-09-13T11:26:48.080 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.081 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.081 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:26:48.081 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:06.318533+0000", 2024-09-13T11:26:48.081 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.251368486 2024-09-13T11:26:48.081 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.081 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.081 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:26:48.081 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:06.318943+0000", 2024-09-13T11:26:48.081 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00040997899999999999 2024-09-13T11:26:48.081 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.081 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.081 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 14,15", 2024-09-13T11:26:48.081 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:06.319277+0000", 2024-09-13T11:26:48.081 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00033357900000000003 2024-09-13T11:26:48.081 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.082 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.082 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:48.082 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:06.326462+0000", 2024-09-13T11:26:48.082 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0071850350000000002 2024-09-13T11:26:48.082 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.082 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.082 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:48.082 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:06.331274+0000", 2024-09-13T11:26:48.082 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0048118079999999999 2024-09-13T11:26:48.082 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.082 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.082 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:26:48.082 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:06.331312+0000", 2024-09-13T11:26:48.082 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.8917e-05 2024-09-13T11:26:48.083 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.083 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.083 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:26:48.083 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:06.331436+0000", 2024-09-13T11:26:48.083 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00012390799999999999 2024-09-13T11:26:48.083 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.083 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.083 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:26:48.083 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:06.331637+0000", 2024-09-13T11:26:48.083 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000200521 2024-09-13T11:26:48.083 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:48.083 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:26:48.083 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:48.083 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.083 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.084 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:221570 3.14 3:289328c0:::benchmark_data_smithi177_31116_object81267:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e193)", 2024-09-13T11:26:48.084 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:26:06.066892+0000", 2024-09-13T11:26:48.084 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 41.931674631999996, 2024-09-13T11:26:48.084 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.27338219400000002, 2024-09-13T11:26:48.084 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:26:48.084 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:26:48.084 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:48.084 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:26:48.084 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:26:48.084 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:48.084 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 221570 2024-09-13T11:26:48.084 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.084 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:26:48.084 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.085 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:26:48.085 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:06.066892+0000", 2024-09-13T11:26:48.085 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:48.085 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.085 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.085 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:26:48.085 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:06.066892+0000", 2024-09-13T11:26:48.085 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:48.085 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.085 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.085 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:26:48.085 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:06.066894+0000", 2024-09-13T11:26:48.085 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.0600000000000002e-06 2024-09-13T11:26:48.086 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.086 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.086 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:26:48.086 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:06.066902+0000", 2024-09-13T11:26:48.086 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.1799999999999996e-06 2024-09-13T11:26:48.086 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.086 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.086 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:26:48.086 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:06.066919+0000", 2024-09-13T11:26:48.086 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.6807e-05 2024-09-13T11:26:48.086 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.086 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.086 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:26:48.086 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:06.066940+0000", 2024-09-13T11:26:48.087 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.1189999999999999e-05 2024-09-13T11:26:48.087 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.087 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.087 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:26:48.087 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:06.067283+0000", 2024-09-13T11:26:48.087 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000342846 2024-09-13T11:26:48.087 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.087 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.087 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for peered", 2024-09-13T11:26:48.087 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:06.067306+0000", 2024-09-13T11:26:48.087 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.3292e-05 2024-09-13T11:26:48.087 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.087 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.087 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:26:48.088 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:06.326644+0000", 2024-09-13T11:26:48.088 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.25933820600000002 2024-09-13T11:26:48.088 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.088 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.088 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:26:48.088 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:06.326791+0000", 2024-09-13T11:26:48.088 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00014677399999999999 2024-09-13T11:26:48.088 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.088 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.088 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 14,15", 2024-09-13T11:26:48.088 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:06.326916+0000", 2024-09-13T11:26:48.088 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00012494700000000001 2024-09-13T11:26:48.088 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.088 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.088 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:48.089 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:06.331718+0000", 2024-09-13T11:26:48.089 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0048017779999999996 2024-09-13T11:26:48.089 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.089 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.089 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:26:48.089 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:06.337102+0000", 2024-09-13T11:26:48.089 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0053843099999999998 2024-09-13T11:26:48.089 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.089 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.089 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:48.089 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:06.340124+0000", 2024-09-13T11:26:48.089 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.003022158 2024-09-13T11:26:48.089 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.089 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.090 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:26:48.090 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:06.340179+0000", 2024-09-13T11:26:48.090 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.4863e-05 2024-09-13T11:26:48.090 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.090 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.090 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:26:48.090 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:06.340274+0000", 2024-09-13T11:26:48.090 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.4783000000000004e-05 2024-09-13T11:26:48.090 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:48.090 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:26:48.090 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:48.090 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.090 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.090 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:221576 3.14 3:2b549bdf:::benchmark_data_smithi177_31116_object81273:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e193)", 2024-09-13T11:26:48.090 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:26:06.066961+0000", 2024-09-13T11:26:48.091 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 41.931604839000002, 2024-09-13T11:26:48.091 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.27360902999999998, 2024-09-13T11:26:48.091 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:26:48.091 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:26:48.091 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:48.091 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:26:48.091 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:26:48.091 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:48.091 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 221576 2024-09-13T11:26:48.091 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.091 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:26:48.091 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.091 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:26:48.091 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:06.066961+0000", 2024-09-13T11:26:48.091 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:48.092 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.092 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.092 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:26:48.092 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:06.066961+0000", 2024-09-13T11:26:48.092 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:48.092 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.092 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.092 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:26:48.092 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:06.066964+0000", 2024-09-13T11:26:48.092 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.0700000000000001e-06 2024-09-13T11:26:48.092 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.092 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.092 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:26:48.092 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:06.066971+0000", 2024-09-13T11:26:48.093 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.046e-06 2024-09-13T11:26:48.093 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.093 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.093 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:26:48.093 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:06.066986+0000", 2024-09-13T11:26:48.093 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.4977000000000001e-05 2024-09-13T11:26:48.093 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.093 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.093 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:26:48.093 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:06.066999+0000", 2024-09-13T11:26:48.093 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3879999999999999e-05 2024-09-13T11:26:48.093 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.093 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.093 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:26:48.093 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:06.067368+0000", 2024-09-13T11:26:48.094 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00036871199999999999 2024-09-13T11:26:48.094 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.094 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.094 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for peered", 2024-09-13T11:26:48.094 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:06.067380+0000", 2024-09-13T11:26:48.094 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.1732999999999999e-05 2024-09-13T11:26:48.094 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.094 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.094 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:26:48.094 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:06.331786+0000", 2024-09-13T11:26:48.094 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.26440598300000001 2024-09-13T11:26:48.094 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.094 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.094 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:26:48.095 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:06.332023+0000", 2024-09-13T11:26:48.095 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00023757300000000001 2024-09-13T11:26:48.095 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.095 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.095 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 14,15", 2024-09-13T11:26:48.095 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:06.332273+0000", 2024-09-13T11:26:48.095 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00024968899999999999 2024-09-13T11:26:48.095 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.095 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.095 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:48.095 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:06.340349+0000", 2024-09-13T11:26:48.095 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.008076214 2024-09-13T11:26:48.095 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.095 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.096 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:48.096 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:06.340432+0000", 2024-09-13T11:26:48.096 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.2671999999999997e-05 2024-09-13T11:26:48.096 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.096 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.096 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:26:48.096 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:06.340446+0000", 2024-09-13T11:26:48.096 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3577000000000001e-05 2024-09-13T11:26:48.096 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.096 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.096 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:26:48.096 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:06.340489+0000", 2024-09-13T11:26:48.096 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.3636e-05 2024-09-13T11:26:48.096 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.097 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.097 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:26:48.097 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:06.340570+0000", 2024-09-13T11:26:48.097 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.1267999999999995e-05 2024-09-13T11:26:48.097 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:48.097 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:26:48.097 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:48.097 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:48.097 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:26:48.097 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-09-13T11:26:48.098 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-13T11:26:48.022+0000 7f6a68775640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26307) UID: 0 2024-09-13T11:26:48.099 INFO:tasks.ceph.osd.4.smithi001.stderr:2024-09-13T11:26:48.094+0000 7ff8e0218640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26310) UID: 0 2024-09-13T11:26:48.099 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-13T11:26:48.200 INFO:tasks.ceph.osd.9.smithi092.stderr:2024-09-13T11:26:48.195+0000 7ff5ab3d0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26348) UID: 0 2024-09-13T11:26:48.226 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-09-13T11:26:48.226 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [], 2024-09-13T11:26:48.226 INFO:teuthology.orchestra.run.smithi177.stdout: "num_ops": 0 2024-09-13T11:26:48.226 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-09-13T11:26:48.241 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-13T11:26:48.300 INFO:tasks.ceph.osd.4.smithi001.stderr:2024-09-13T11:26:48.298+0000 7ff8e0218640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26310) UID: 0 2024-09-13T11:26:48.357 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-09-13T11:26:48.357 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [], 2024-09-13T11:26:48.357 INFO:teuthology.orchestra.run.smithi177.stdout: "complaint_time": 30, 2024-09-13T11:26:48.357 INFO:teuthology.orchestra.run.smithi177.stdout: "num_blocked_ops": 0 2024-09-13T11:26:48.357 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-09-13T11:26:48.370 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-13T11:26:48.414 INFO:tasks.ceph.osd.11.smithi177.stderr:2024-09-13T11:26:48.413+0000 7f24792c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26580) UID: 0 2024-09-13T11:26:48.488 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-09-13T11:26:48.489 INFO:teuthology.orchestra.run.smithi177.stdout: "size": 20, 2024-09-13T11:26:48.489 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 600, 2024-09-13T11:26:48.489 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [ 2024-09-13T11:26:48.489 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.489 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPushReply(3.13 75/70 [PushReplyOp(3:cfab03d6:::benchmark_data_smithi177_31116_object16599:head)])", 2024-09-13T11:26:48.489 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:31.163480+0000", 2024-09-13T11:26:48.489 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 197.32366587300001, 2024-09-13T11:26:48.489 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.63009029299999997, 2024-09-13T11:26:48.489 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:26:48.489 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:26:48.489 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-13T11:26:48.489 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:26:48.489 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.490 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:26:48.490 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.163480+0000", 2024-09-13T11:26:48.490 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:48.490 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.490 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.490 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:26:48.490 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.163480+0000", 2024-09-13T11:26:48.490 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:48.490 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.490 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.490 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:26:48.490 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.163481+0000", 2024-09-13T11:26:48.490 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0410000000000001e-06 2024-09-13T11:26:48.491 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.491 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.491 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:26:48.491 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.163485+0000", 2024-09-13T11:26:48.491 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.1629999999999998e-06 2024-09-13T11:26:48.491 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.491 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.491 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:26:48.491 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.792166+0000", 2024-09-13T11:26:48.491 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.62868067400000005 2024-09-13T11:26:48.491 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.491 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.491 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:26:48.492 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.792184+0000", 2024-09-13T11:26:48.492 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.7898999999999999e-05 2024-09-13T11:26:48.492 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.492 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.492 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:26:48.492 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.792312+0000", 2024-09-13T11:26:48.492 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00012871000000000001 2024-09-13T11:26:48.492 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.492 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.492 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:26:48.492 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.793570+0000", 2024-09-13T11:26:48.492 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001257806 2024-09-13T11:26:48.492 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:48.493 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:26:48.493 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:48.493 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.493 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.493 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPushReply(3.13 76/70 [PushReplyOp(3:c9d43a82:::benchmark_data_smithi177_31116_object20026:head)])", 2024-09-13T11:26:48.493 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:32.427709+0000", 2024-09-13T11:26:48.493 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 196.059436358, 2024-09-13T11:26:48.493 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.46547316300000002, 2024-09-13T11:26:48.493 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:26:48.493 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:26:48.493 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-13T11:26:48.493 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:26:48.493 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.494 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:26:48.494 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:32.427709+0000", 2024-09-13T11:26:48.494 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:48.494 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.494 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.494 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:26:48.494 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:32.427709+0000", 2024-09-13T11:26:48.494 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:48.494 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.494 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.494 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:26:48.494 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:32.427711+0000", 2024-09-13T11:26:48.495 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5099999999999999e-06 2024-09-13T11:26:48.495 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.495 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.495 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:26:48.495 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:32.427717+0000", 2024-09-13T11:26:48.495 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.1569999999999997e-06 2024-09-13T11:26:48.495 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.495 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.495 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:26:48.495 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:32.892390+0000", 2024-09-13T11:26:48.495 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.46467251100000001 2024-09-13T11:26:48.495 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.495 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.496 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:26:48.496 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:32.892405+0000", 2024-09-13T11:26:48.496 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5296999999999999e-05 2024-09-13T11:26:48.496 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.496 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.496 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:26:48.496 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:32.892503+0000", 2024-09-13T11:26:48.496 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.7820999999999997e-05 2024-09-13T11:26:48.496 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.496 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.496 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:26:48.496 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:32.893183+0000", 2024-09-13T11:26:48.496 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00067986700000000001 2024-09-13T11:26:48.497 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:48.497 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:26:48.497 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:48.497 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.497 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.497 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:57435 3.3 e80/75 3:c0db4777:::benchmark_data_smithi177_31116_object57434:head v 80'3155, mlcod=80'3153)", 2024-09-13T11:26:48.497 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:37.437070+0000", 2024-09-13T11:26:48.497 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 191.05007554100001, 2024-09-13T11:26:48.497 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.35861845999999997, 2024-09-13T11:26:48.497 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:26:48.497 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:26:48.497 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:48.498 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:26:48.498 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.498 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:26:48.498 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:37.437070+0000", 2024-09-13T11:26:48.498 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:48.498 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.498 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.498 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:26:48.498 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:37.437070+0000", 2024-09-13T11:26:48.498 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:48.498 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.498 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.498 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:26:48.498 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:37.437072+0000", 2024-09-13T11:26:48.499 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5549999999999999e-06 2024-09-13T11:26:48.499 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.499 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.499 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:26:48.499 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:37.437141+0000", 2024-09-13T11:26:48.499 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.9494999999999998e-05 2024-09-13T11:26:48.499 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.499 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.499 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:26:48.499 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:37.793400+0000", 2024-09-13T11:26:48.499 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.35625817999999998 2024-09-13T11:26:48.499 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.499 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.499 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:26:48.500 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:37.793406+0000", 2024-09-13T11:26:48.500 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.6830000000000002e-06 2024-09-13T11:26:48.500 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.500 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.500 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:26:48.500 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:37.793472+0000", 2024-09-13T11:26:48.500 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.5671000000000002e-05 2024-09-13T11:26:48.500 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.500 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.500 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:26:48.500 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:37.793492+0000", 2024-09-13T11:26:48.500 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.0051000000000002e-05 2024-09-13T11:26:48.500 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.500 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.501 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:26:48.501 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:37.795626+0000", 2024-09-13T11:26:48.501 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0021344990000000002 2024-09-13T11:26:48.501 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.501 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.501 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:26:48.501 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:37.795689+0000", 2024-09-13T11:26:48.501 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.2325999999999999e-05 2024-09-13T11:26:48.501 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:48.501 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:26:48.501 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:48.501 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.501 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.501 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:57985 3.3 e80/75 3:c330bb03:::benchmark_data_smithi177_31116_object57984:head v 80'3162, mlcod=80'3159)", 2024-09-13T11:26:48.502 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:38.007076+0000", 2024-09-13T11:26:48.502 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 190.480070189, 2024-09-13T11:26:48.502 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.18211438199999999, 2024-09-13T11:26:48.502 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:26:48.502 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:26:48.502 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:48.502 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:26:48.502 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.502 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:26:48.502 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:38.007076+0000", 2024-09-13T11:26:48.502 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:48.502 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.502 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.502 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:26:48.503 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:38.007076+0000", 2024-09-13T11:26:48.503 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:48.503 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.503 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.503 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:26:48.503 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:38.007077+0000", 2024-09-13T11:26:48.503 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.6759999999999999e-06 2024-09-13T11:26:48.503 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.503 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.503 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:26:48.503 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:38.007221+0000", 2024-09-13T11:26:48.503 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00014344299999999999 2024-09-13T11:26:48.503 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.503 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.504 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:26:48.504 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:38.186808+0000", 2024-09-13T11:26:48.504 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.17958726699999999 2024-09-13T11:26:48.504 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.504 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.504 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:26:48.504 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:38.186815+0000", 2024-09-13T11:26:48.504 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.0890000000000002e-06 2024-09-13T11:26:48.504 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.504 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.504 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:26:48.504 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:38.186887+0000", 2024-09-13T11:26:48.504 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.2006000000000001e-05 2024-09-13T11:26:48.505 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.505 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.505 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:26:48.505 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:38.186913+0000", 2024-09-13T11:26:48.505 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.5962e-05 2024-09-13T11:26:48.505 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.505 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.505 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:26:48.505 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:38.189123+0000", 2024-09-13T11:26:48.505 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0022099649999999999 2024-09-13T11:26:48.505 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.505 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.505 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:26:48.505 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:38.189190+0000", 2024-09-13T11:26:48.506 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.6973999999999999e-05 2024-09-13T11:26:48.506 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:48.506 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:26:48.506 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:48.506 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.506 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.506 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPushReply(3.23 84/78 [PushReplyOp(3:c7e44616:::benchmark_data_smithi177_31116_object32371:head)])", 2024-09-13T11:26:48.506 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:41.338210+0000", 2024-09-13T11:26:48.506 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 187.14893547, 2024-09-13T11:26:48.506 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.41118492299999998, 2024-09-13T11:26:48.506 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:26:48.506 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:26:48.507 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-13T11:26:48.507 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:26:48.507 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.507 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:26:48.507 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.338210+0000", 2024-09-13T11:26:48.507 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:48.507 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.507 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.507 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:26:48.507 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.338210+0000", 2024-09-13T11:26:48.507 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:48.507 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.507 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.508 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:26:48.508 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.338212+0000", 2024-09-13T11:26:48.508 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.314e-06 2024-09-13T11:26:48.508 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.508 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.508 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:26:48.508 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.338219+0000", 2024-09-13T11:26:48.508 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.6469999999999998e-06 2024-09-13T11:26:48.508 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.508 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.508 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:26:48.508 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.748675+0000", 2024-09-13T11:26:48.508 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.41045567500000002 2024-09-13T11:26:48.508 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.509 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.509 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:26:48.509 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.748684+0000", 2024-09-13T11:26:48.509 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.2150000000000007e-06 2024-09-13T11:26:48.509 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.509 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.509 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:26:48.509 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.748739+0000", 2024-09-13T11:26:48.509 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.4543000000000003e-05 2024-09-13T11:26:48.509 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.509 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.509 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:26:48.509 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.749395+0000", 2024-09-13T11:26:48.510 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00065652899999999999 2024-09-13T11:26:48.510 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:48.510 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:26:48.510 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:48.510 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.510 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.510 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPushReply(3.23 85/78 [PushReplyOp(3:c5c6dce1:::benchmark_data_smithi177_31116_object37293:head)])", 2024-09-13T11:26:48.510 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:42.377862+0000", 2024-09-13T11:26:48.510 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 186.109284011, 2024-09-13T11:26:48.510 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.62538808599999995, 2024-09-13T11:26:48.510 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:26:48.510 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:26:48.510 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-13T11:26:48.510 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:26:48.511 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.511 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:26:48.511 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.377862+0000", 2024-09-13T11:26:48.511 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:48.511 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.511 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.511 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:26:48.511 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.377862+0000", 2024-09-13T11:26:48.511 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:48.511 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.511 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.511 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:26:48.511 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.377863+0000", 2024-09-13T11:26:48.512 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.6899999999999996e-07 2024-09-13T11:26:48.512 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.512 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.512 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:26:48.512 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.377866+0000", 2024-09-13T11:26:48.512 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.219e-06 2024-09-13T11:26:48.512 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.512 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.512 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:26:48.512 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:43.001857+0000", 2024-09-13T11:26:48.512 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.62399084299999996 2024-09-13T11:26:48.512 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.512 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.513 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:26:48.513 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:43.001869+0000", 2024-09-13T11:26:48.513 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.2415999999999999e-05 2024-09-13T11:26:48.513 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.513 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.513 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:26:48.513 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:43.002194+0000", 2024-09-13T11:26:48.513 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000324774 2024-09-13T11:26:48.513 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.513 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.513 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:26:48.513 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:43.003250+0000", 2024-09-13T11:26:48.513 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001055865 2024-09-13T11:26:48.513 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:48.514 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:26:48.514 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:48.514 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.514 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.514 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPush(3.1c 89/77 [PushOp(3:38bcbc71:::benchmark_data_smithi177_31116_object31448:head, version: 59'1946, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:38bcbc71:::benchmark_data_smithi177_31116_object31448:head@59'1946, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:26:48.514 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:46.670723+0000", 2024-09-13T11:26:48.514 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 181.81642278699999, 2024-09-13T11:26:48.514 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.27576065, 2024-09-13T11:26:48.514 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:26:48.514 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:26:48.514 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:26:48.514 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:26:48.514 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.515 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:26:48.515 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.670723+0000", 2024-09-13T11:26:48.515 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:48.515 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.515 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.515 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:26:48.515 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.670723+0000", 2024-09-13T11:26:48.515 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:48.515 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.515 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.515 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:26:48.515 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.670724+0000", 2024-09-13T11:26:48.515 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.4389999999999999e-06 2024-09-13T11:26:48.515 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.516 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.516 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:26:48.516 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.670754+0000", 2024-09-13T11:26:48.516 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.9224999999999999e-05 2024-09-13T11:26:48.516 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.516 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.516 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:26:48.516 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.944369+0000", 2024-09-13T11:26:48.516 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.27361529800000001 2024-09-13T11:26:48.516 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.516 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.516 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:26:48.516 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.944422+0000", 2024-09-13T11:26:48.517 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.3318e-05 2024-09-13T11:26:48.517 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.517 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.517 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:26:48.517 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.944711+0000", 2024-09-13T11:26:48.517 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00028912399999999998 2024-09-13T11:26:48.517 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.517 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.517 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:26:48.517 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.944735+0000", 2024-09-13T11:26:48.517 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.3366999999999998e-05 2024-09-13T11:26:48.517 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.517 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.518 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:26:48.518 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.946484+0000", 2024-09-13T11:26:48.518 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0017488790000000001 2024-09-13T11:26:48.518 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:48.518 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:26:48.518 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:48.518 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.518 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.518 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPush(3.1c 91/77 [PushOp(3:3e18abc2:::benchmark_data_smithi177_31116_object35874:head, version: 62'2235, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:3e18abc2:::benchmark_data_smithi177_31116_object35874:head@62'2235, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:26:48.518 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:48.350968+0000", 2024-09-13T11:26:48.518 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 180.13617747399999, 2024-09-13T11:26:48.518 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.19257698000000001, 2024-09-13T11:26:48.518 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:26:48.519 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:26:48.519 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:26:48.519 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:26:48.519 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.519 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:26:48.519 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.350968+0000", 2024-09-13T11:26:48.519 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:48.519 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.519 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.519 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:26:48.519 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.350968+0000", 2024-09-13T11:26:48.519 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:48.519 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.519 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.520 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:26:48.520 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.350970+0000", 2024-09-13T11:26:48.520 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.796e-06 2024-09-13T11:26:48.520 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.520 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.520 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:26:48.520 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.351003+0000", 2024-09-13T11:26:48.520 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.2631e-05 2024-09-13T11:26:48.520 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.520 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.520 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:26:48.520 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.542140+0000", 2024-09-13T11:26:48.520 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.191136737 2024-09-13T11:26:48.521 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.521 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.521 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:26:48.521 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.542158+0000", 2024-09-13T11:26:48.521 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.8896999999999999e-05 2024-09-13T11:26:48.521 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.521 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.521 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:26:48.521 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.542263+0000", 2024-09-13T11:26:48.521 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000104158 2024-09-13T11:26:48.521 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.521 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.521 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:26:48.522 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.542279+0000", 2024-09-13T11:26:48.522 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5926999999999998e-05 2024-09-13T11:26:48.522 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.522 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.522 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:26:48.522 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.543545+0000", 2024-09-13T11:26:48.522 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0012668340000000001 2024-09-13T11:26:48.522 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:48.522 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:26:48.522 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:48.522 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.522 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.522 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:83211 3.1c e106/77 3:3ae31a0f:::benchmark_data_smithi177_31116_object83210:head v 106'4362, mlcod=106'4360)", 2024-09-13T11:26:48.522 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:04.345264+0000", 2024-09-13T11:26:48.523 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 164.14188182199999, 2024-09-13T11:26:48.523 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.17036547399999999, 2024-09-13T11:26:48.523 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:26:48.523 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:26:48.523 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:48.523 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:26:48.523 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.523 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:26:48.523 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.345264+0000", 2024-09-13T11:26:48.523 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:48.523 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.523 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.523 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:26:48.524 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.345264+0000", 2024-09-13T11:26:48.524 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:48.524 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.524 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.524 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:26:48.524 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.345267+0000", 2024-09-13T11:26:48.524 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.7590000000000002e-06 2024-09-13T11:26:48.524 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.524 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.524 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:26:48.524 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.345298+0000", 2024-09-13T11:26:48.524 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.1173000000000001e-05 2024-09-13T11:26:48.524 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.524 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.525 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:26:48.525 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.512265+0000", 2024-09-13T11:26:48.525 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.16696657100000001 2024-09-13T11:26:48.525 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.525 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.525 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:26:48.525 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.512279+0000", 2024-09-13T11:26:48.525 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.4953999999999999e-05 2024-09-13T11:26:48.525 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.525 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.525 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:26:48.525 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.512402+0000", 2024-09-13T11:26:48.525 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00012277999999999999 2024-09-13T11:26:48.525 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.526 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.526 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:26:48.526 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.512440+0000", 2024-09-13T11:26:48.526 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.7861999999999998e-05 2024-09-13T11:26:48.526 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.526 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.526 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:26:48.526 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.515515+0000", 2024-09-13T11:26:48.526 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00307472 2024-09-13T11:26:48.526 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.526 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.526 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:26:48.526 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.515629+0000", 2024-09-13T11:26:48.526 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00011465500000000001 2024-09-13T11:26:48.527 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:48.527 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:26:48.527 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:48.527 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.527 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.527 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:83223 3.1c e106/77 3:3bc6018d:::benchmark_data_smithi177_31116_object83222:head v 106'4363, mlcod=106'4360)", 2024-09-13T11:26:48.527 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:04.512139+0000", 2024-09-13T11:26:48.527 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 163.97500667599999, 2024-09-13T11:26:48.527 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.471923235, 2024-09-13T11:26:48.527 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:26:48.527 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:26:48.527 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:48.527 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:26:48.528 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.528 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:26:48.528 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.512139+0000", 2024-09-13T11:26:48.528 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:48.528 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.528 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.528 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:26:48.528 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.512139+0000", 2024-09-13T11:26:48.528 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:48.528 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.528 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.528 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:26:48.528 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.512143+0000", 2024-09-13T11:26:48.528 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.427e-06 2024-09-13T11:26:48.529 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.529 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.529 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:26:48.529 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.512227+0000", 2024-09-13T11:26:48.529 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.3965e-05 2024-09-13T11:26:48.529 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.529 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.529 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:26:48.529 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.976330+0000", 2024-09-13T11:26:48.529 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.46410342599999999 2024-09-13T11:26:48.529 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.529 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.529 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:26:48.530 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.976350+0000", 2024-09-13T11:26:48.530 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.9533000000000001e-05 2024-09-13T11:26:48.530 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.530 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.530 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:26:48.530 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.976488+0000", 2024-09-13T11:26:48.530 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00013799400000000001 2024-09-13T11:26:48.530 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.530 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.530 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:26:48.530 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.976543+0000", 2024-09-13T11:26:48.530 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.5442000000000002e-05 2024-09-13T11:26:48.530 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.530 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.531 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:26:48.531 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.983227+0000", 2024-09-13T11:26:48.531 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0066842899999999998 2024-09-13T11:26:48.531 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.531 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.531 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:26:48.531 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.984062+0000", 2024-09-13T11:26:48.531 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00083515800000000002 2024-09-13T11:26:48.531 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:48.531 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:26:48.531 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:48.531 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.532 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.532 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:110256 3.13 e119/83 3:cb00d078:::benchmark_data_smithi177_31116_object110255:head v 119'4823, mlcod=119'4821)", 2024-09-13T11:26:48.532 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:25.284530+0000", 2024-09-13T11:26:48.532 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 143.202616216, 2024-09-13T11:26:48.532 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.343491771, 2024-09-13T11:26:48.532 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:26:48.532 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:26:48.532 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:48.532 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:26:48.532 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.532 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:26:48.532 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.284530+0000", 2024-09-13T11:26:48.532 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:48.533 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.533 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.533 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:26:48.533 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.284530+0000", 2024-09-13T11:26:48.533 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:48.533 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.533 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.533 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:26:48.533 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.284532+0000", 2024-09-13T11:26:48.533 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.8870000000000001e-06 2024-09-13T11:26:48.533 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.533 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.533 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:26:48.534 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.284558+0000", 2024-09-13T11:26:48.534 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.6899000000000001e-05 2024-09-13T11:26:48.534 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.534 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.534 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:26:48.534 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.626172+0000", 2024-09-13T11:26:48.534 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.34161370800000002 2024-09-13T11:26:48.534 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.534 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.534 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:26:48.534 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.626187+0000", 2024-09-13T11:26:48.534 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.485e-05 2024-09-13T11:26:48.534 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.534 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.535 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:26:48.535 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.626245+0000", 2024-09-13T11:26:48.535 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.7998000000000003e-05 2024-09-13T11:26:48.535 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.535 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.535 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:26:48.535 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.626260+0000", 2024-09-13T11:26:48.535 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5429000000000001e-05 2024-09-13T11:26:48.535 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.535 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.535 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:26:48.535 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.627924+0000", 2024-09-13T11:26:48.535 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0016634060000000001 2024-09-13T11:26:48.535 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.536 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.536 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:26:48.536 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.628021+0000", 2024-09-13T11:26:48.536 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.7594000000000001e-05 2024-09-13T11:26:48.536 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:48.536 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:26:48.536 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:48.536 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.536 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.536 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPush(3.2b 121/119 [PushOp(3:d53a23e8:::benchmark_data_smithi177_31116_object5366:head, version: 45'342, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:d53a23e8:::benchmark_data_smithi177_31116_object5366:head@45'342, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:26:48.536 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:27.268731+0000", 2024-09-13T11:26:48.536 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 141.21841475400001, 2024-09-13T11:26:48.536 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.72228345000000005, 2024-09-13T11:26:48.537 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:26:48.537 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:26:48.537 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:26:48.537 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:26:48.537 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.537 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:26:48.537 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:27.268731+0000", 2024-09-13T11:26:48.537 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:48.537 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.537 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.537 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:26:48.537 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:27.268731+0000", 2024-09-13T11:26:48.538 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:48.538 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.538 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.538 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:26:48.538 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:27.268733+0000", 2024-09-13T11:26:48.538 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.9740000000000001e-06 2024-09-13T11:26:48.538 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.538 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.538 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:26:48.538 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:27.268778+0000", 2024-09-13T11:26:48.538 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.4679999999999999e-05 2024-09-13T11:26:48.538 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.538 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.539 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:26:48.539 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:27.988594+0000", 2024-09-13T11:26:48.539 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.71981667299999996 2024-09-13T11:26:48.539 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.539 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.539 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:26:48.539 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:27.988652+0000", 2024-09-13T11:26:48.539 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.7640999999999999e-05 2024-09-13T11:26:48.539 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.539 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.539 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:26:48.539 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:27.988889+0000", 2024-09-13T11:26:48.539 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00023706500000000001 2024-09-13T11:26:48.540 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.540 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.540 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:26:48.540 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:27.988933+0000", 2024-09-13T11:26:48.540 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.3865000000000001e-05 2024-09-13T11:26:48.540 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.540 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.540 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:26:48.540 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:27.991015+0000", 2024-09-13T11:26:48.540 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.002081552 2024-09-13T11:26:48.540 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:48.540 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:26:48.540 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:48.540 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.541 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.541 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPush(3.2b 124/119 [PushOp(3:d70bd127:::benchmark_data_smithi177_31116_object32365:head, version: 60'2034, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:d70bd127:::benchmark_data_smithi177_31116_object32365:head@60'2034, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:26:48.541 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:32.037338+0000", 2024-09-13T11:26:48.541 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 136.44980748099999, 2024-09-13T11:26:48.541 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.24358922699999999, 2024-09-13T11:26:48.541 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:26:48.541 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:26:48.541 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:26:48.541 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:26:48.541 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.541 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:26:48.541 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:32.037338+0000", 2024-09-13T11:26:48.542 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:48.542 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.542 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.542 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:26:48.542 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:32.037338+0000", 2024-09-13T11:26:48.542 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:48.542 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.542 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.542 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:26:48.542 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:32.037340+0000", 2024-09-13T11:26:48.542 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3790000000000001e-06 2024-09-13T11:26:48.542 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.542 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.542 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:26:48.543 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:32.037411+0000", 2024-09-13T11:26:48.543 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.1254000000000006e-05 2024-09-13T11:26:48.543 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.543 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.543 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:26:48.543 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:32.277467+0000", 2024-09-13T11:26:48.543 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.24005563199999999 2024-09-13T11:26:48.543 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.543 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.543 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:26:48.543 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:32.277494+0000", 2024-09-13T11:26:48.543 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.7120000000000001e-05 2024-09-13T11:26:48.543 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.544 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.544 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:26:48.544 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:32.279860+0000", 2024-09-13T11:26:48.544 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0023667559999999998 2024-09-13T11:26:48.544 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.544 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.544 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:26:48.544 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:32.279883+0000", 2024-09-13T11:26:48.544 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.2138999999999999e-05 2024-09-13T11:26:48.544 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.544 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.544 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:26:48.544 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:32.280928+0000", 2024-09-13T11:26:48.544 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0010449470000000001 2024-09-13T11:26:48.545 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:48.545 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:26:48.545 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:48.545 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.545 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.545 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPush(3.2b 125/119 [PushOp(3:d4c5ede9:::benchmark_data_smithi177_31116_object33277:head, version: 60'2092, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:d4c5ede9:::benchmark_data_smithi177_31116_object33277:head@60'2092, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:26:48.545 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:32.403929+0000", 2024-09-13T11:26:48.545 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 136.083217031, 2024-09-13T11:26:48.545 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.194592864, 2024-09-13T11:26:48.545 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:26:48.545 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:26:48.545 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:26:48.545 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:26:48.546 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.546 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:26:48.546 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:32.403929+0000", 2024-09-13T11:26:48.546 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:48.546 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.546 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.546 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:26:48.546 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:32.403929+0000", 2024-09-13T11:26:48.546 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:48.546 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.546 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.546 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:26:48.546 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:32.403930+0000", 2024-09-13T11:26:48.547 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.6729999999999999e-06 2024-09-13T11:26:48.547 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.547 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.547 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:26:48.547 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:32.403979+0000", 2024-09-13T11:26:48.547 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.8242000000000003e-05 2024-09-13T11:26:48.547 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.547 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.547 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:26:48.547 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:32.597445+0000", 2024-09-13T11:26:48.547 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.19346676500000001 2024-09-13T11:26:48.547 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.547 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.547 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:26:48.548 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:32.597455+0000", 2024-09-13T11:26:48.548 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.2939999999999994e-06 2024-09-13T11:26:48.548 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.548 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.548 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:26:48.548 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:32.597514+0000", 2024-09-13T11:26:48.548 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.9509000000000002e-05 2024-09-13T11:26:48.548 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.548 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.548 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:26:48.548 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:32.597522+0000", 2024-09-13T11:26:48.548 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.0350000000000001e-06 2024-09-13T11:26:48.548 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.549 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.549 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:26:48.549 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:32.598522+0000", 2024-09-13T11:26:48.549 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00099934599999999991 2024-09-13T11:26:48.549 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:48.549 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:26:48.549 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:48.549 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.549 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.549 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPush(3.2b 128/119 [PushOp(3:d6ff590f:::benchmark_data_smithi177_31116_object64581:head, version: 90'3750, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:d6ff590f:::benchmark_data_smithi177_31116_object64581:head@90'3750, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:26:48.549 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:37.982062+0000", 2024-09-13T11:26:48.549 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 130.50508413200001, 2024-09-13T11:26:48.549 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.28899675200000002, 2024-09-13T11:26:48.550 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:26:48.550 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:26:48.550 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:26:48.550 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:26:48.550 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.550 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:26:48.550 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:37.982062+0000", 2024-09-13T11:26:48.550 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:48.550 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.550 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.550 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:26:48.550 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:37.982062+0000", 2024-09-13T11:26:48.550 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:48.551 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.551 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.551 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:26:48.551 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:37.982064+0000", 2024-09-13T11:26:48.551 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.2259999999999999e-06 2024-09-13T11:26:48.551 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.551 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.551 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:26:48.551 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:37.982114+0000", 2024-09-13T11:26:48.551 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.0330000000000001e-05 2024-09-13T11:26:48.551 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.551 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.551 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:26:48.551 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:38.269983+0000", 2024-09-13T11:26:48.552 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.28786869999999998 2024-09-13T11:26:48.552 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.552 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.552 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:26:48.552 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:38.270000+0000", 2024-09-13T11:26:48.552 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.6977999999999999e-05 2024-09-13T11:26:48.552 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.552 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.552 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:26:48.552 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:38.270092+0000", 2024-09-13T11:26:48.552 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.221e-05 2024-09-13T11:26:48.552 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.552 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.553 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:26:48.553 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:38.270103+0000", 2024-09-13T11:26:48.553 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.1253999999999999e-05 2024-09-13T11:26:48.553 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.553 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.553 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:26:48.553 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:38.271058+0000", 2024-09-13T11:26:48.553 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00095505399999999999 2024-09-13T11:26:48.553 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:48.553 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:26:48.553 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:48.553 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.553 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.553 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:183827 3.2b e171/119 3:d63d3986:::benchmark_data_smithi177_31116_object43524:head v 171'6235, mlcod=171'6232)", 2024-09-13T11:26:48.554 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:25:32.487782+0000", 2024-09-13T11:26:48.554 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 75.999363919000004, 2024-09-13T11:26:48.554 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.21250508400000001, 2024-09-13T11:26:48.554 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:26:48.554 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:26:48.554 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:48.554 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:26:48.554 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.554 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:26:48.554 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:32.487782+0000", 2024-09-13T11:26:48.554 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:48.554 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.554 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.554 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:26:48.555 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:32.487782+0000", 2024-09-13T11:26:48.555 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:48.555 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.555 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.555 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:26:48.555 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:32.487783+0000", 2024-09-13T11:26:48.555 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.9699999999999994e-07 2024-09-13T11:26:48.555 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.555 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.555 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:26:48.555 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:32.487787+0000", 2024-09-13T11:26:48.555 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.5689999999999998e-06 2024-09-13T11:26:48.555 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.555 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.556 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:26:48.556 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:32.699019+0000", 2024-09-13T11:26:48.556 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.21123154499999999 2024-09-13T11:26:48.556 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.556 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.556 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:26:48.556 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:32.699024+0000", 2024-09-13T11:26:48.556 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.6229999999999996e-06 2024-09-13T11:26:48.556 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.556 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.556 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:26:48.556 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:32.699068+0000", 2024-09-13T11:26:48.556 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.4722000000000003e-05 2024-09-13T11:26:48.557 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.557 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.557 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:26:48.557 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:32.699084+0000", 2024-09-13T11:26:48.557 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5282999999999999e-05 2024-09-13T11:26:48.557 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.557 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.557 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:26:48.557 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:32.700246+0000", 2024-09-13T11:26:48.557 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001162735 2024-09-13T11:26:48.557 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.557 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.557 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:26:48.558 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:32.700287+0000", 2024-09-13T11:26:48.558 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.0609999999999999e-05 2024-09-13T11:26:48.558 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:48.558 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:26:48.558 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:48.558 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.558 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.558 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:191665 3.a e176/117 3:53060b28:::benchmark_data_smithi177_31116_object51362:head v 176'6424, mlcod=176'6422)", 2024-09-13T11:26:48.558 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:25:39.657701+0000", 2024-09-13T11:26:48.558 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 68.829444405000004, 2024-09-13T11:26:48.558 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.25090320900000002, 2024-09-13T11:26:48.558 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:26:48.558 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:26:48.558 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:48.559 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:26:48.559 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.559 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:26:48.559 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:39.657701+0000", 2024-09-13T11:26:48.559 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:48.559 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.559 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.559 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:26:48.559 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:39.657701+0000", 2024-09-13T11:26:48.559 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:48.559 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.559 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.559 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:26:48.560 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:39.657704+0000", 2024-09-13T11:26:48.560 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.7650000000000002e-06 2024-09-13T11:26:48.560 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.560 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.560 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:26:48.560 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:39.657714+0000", 2024-09-13T11:26:48.560 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.5719999999999992e-06 2024-09-13T11:26:48.560 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.560 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.560 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:26:48.560 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:39.904490+0000", 2024-09-13T11:26:48.560 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.246775831 2024-09-13T11:26:48.560 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.561 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.561 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:26:48.561 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:39.904503+0000", 2024-09-13T11:26:48.561 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3723000000000001e-05 2024-09-13T11:26:48.561 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.561 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.561 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:26:48.561 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:39.904660+0000", 2024-09-13T11:26:48.561 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00015705499999999999 2024-09-13T11:26:48.561 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.561 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.561 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:26:48.562 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:39.904699+0000", 2024-09-13T11:26:48.562 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.8877000000000001e-05 2024-09-13T11:26:48.562 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.562 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.562 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:26:48.562 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:39.908544+0000", 2024-09-13T11:26:48.562 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0038448739999999999 2024-09-13T11:26:48.562 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.563 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.563 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:26:48.563 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:39.908605+0000", 2024-09-13T11:26:48.563 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.0511999999999999e-05 2024-09-13T11:26:48.563 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:48.563 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:26:48.563 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:48.563 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.563 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.563 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:196349 3.d 3:b2d956a9:::benchmark_data_smithi177_31116_object56046:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e178)", 2024-09-13T11:26:48.563 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:25:43.404118+0000", 2024-09-13T11:26:48.563 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 65.083028311999996, 2024-09-13T11:26:48.564 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.17729215100000001, 2024-09-13T11:26:48.564 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:26:48.564 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:26:48.564 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:48.564 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:26:48.564 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:26:48.564 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:48.564 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 196349 2024-09-13T11:26:48.564 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.564 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:26:48.564 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.564 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:26:48.564 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:43.404118+0000", 2024-09-13T11:26:48.564 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:48.565 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.565 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.565 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:26:48.565 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:43.404118+0000", 2024-09-13T11:26:48.565 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:48.565 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.565 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.565 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:26:48.565 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:43.404121+0000", 2024-09-13T11:26:48.565 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.743e-06 2024-09-13T11:26:48.565 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.565 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.565 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:26:48.565 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:43.404132+0000", 2024-09-13T11:26:48.566 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0433e-05 2024-09-13T11:26:48.566 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.566 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.566 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:26:48.566 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:43.404159+0000", 2024-09-13T11:26:48.566 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.7064e-05 2024-09-13T11:26:48.566 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.566 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.566 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:26:48.566 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:43.404172+0000", 2024-09-13T11:26:48.566 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3606e-05 2024-09-13T11:26:48.566 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.566 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.566 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:26:48.567 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:43.404312+0000", 2024-09-13T11:26:48.567 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000139262 2024-09-13T11:26:48.567 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.567 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.567 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:26:48.567 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:43.404531+0000", 2024-09-13T11:26:48.567 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00021986 2024-09-13T11:26:48.567 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.567 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.567 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 3", 2024-09-13T11:26:48.567 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:43.404706+0000", 2024-09-13T11:26:48.567 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000174358 2024-09-13T11:26:48.567 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.567 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.568 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:26:48.568 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:43.408021+0000", 2024-09-13T11:26:48.568 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0033152939999999999 2024-09-13T11:26:48.568 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.568 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.568 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:48.568 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:43.581239+0000", 2024-09-13T11:26:48.568 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.17321809699999999 2024-09-13T11:26:48.568 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.568 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.568 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:26:48.568 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:43.581331+0000", 2024-09-13T11:26:48.568 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.1877000000000001e-05 2024-09-13T11:26:48.568 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.569 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.569 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:26:48.569 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:43.581410+0000", 2024-09-13T11:26:48.569 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.8557e-05 2024-09-13T11:26:48.569 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:48.569 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:26:48.569 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:48.569 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.569 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.569 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:227427 3.a e196/117 3:52db129e:::benchmark_data_smithi177_31116_object87124:head v 196'6993, mlcod=196'6991)", 2024-09-13T11:26:48.569 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:26:11.046359+0000", 2024-09-13T11:26:48.569 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 37.440786840000001, 2024-09-13T11:26:48.569 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.19900411000000001, 2024-09-13T11:26:48.569 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:26:48.570 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:26:48.570 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:48.570 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:26:48.570 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.570 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:26:48.570 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:11.046359+0000", 2024-09-13T11:26:48.570 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:48.570 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.570 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.570 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:26:48.570 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:11.046359+0000", 2024-09-13T11:26:48.570 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:48.570 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.571 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.571 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:26:48.571 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:11.046360+0000", 2024-09-13T11:26:48.571 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.215e-06 2024-09-13T11:26:48.571 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.571 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.571 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:26:48.571 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:11.046365+0000", 2024-09-13T11:26:48.571 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.7060000000000003e-06 2024-09-13T11:26:48.571 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.571 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.571 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:26:48.571 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:11.239636+0000", 2024-09-13T11:26:48.572 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.193270687 2024-09-13T11:26:48.572 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.572 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.572 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:26:48.572 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:11.239651+0000", 2024-09-13T11:26:48.572 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5067e-05 2024-09-13T11:26:48.572 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.572 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.572 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:26:48.572 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:11.239970+0000", 2024-09-13T11:26:48.572 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00031894300000000003 2024-09-13T11:26:48.572 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.572 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.573 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:26:48.573 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:11.240027+0000", 2024-09-13T11:26:48.573 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.7419e-05 2024-09-13T11:26:48.573 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.573 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.573 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:26:48.573 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:11.243833+0000", 2024-09-13T11:26:48.573 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0038059629999999999 2024-09-13T11:26:48.573 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.573 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.573 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:26:48.573 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:11.245363+0000", 2024-09-13T11:26:48.573 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0015301100000000001 2024-09-13T11:26:48.573 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:48.574 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:26:48.574 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:48.574 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.574 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.574 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:266690 3.2a e223/131 3:5440fefb:::benchmark_data_smithi177_31116_object126387:head v 223'7712, mlcod=222'7708)", 2024-09-13T11:26:48.574 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:26:48.043275+0000", 2024-09-13T11:26:48.574 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 0.44387062500000002, 2024-09-13T11:26:48.574 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.18750093900000001, 2024-09-13T11:26:48.574 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:26:48.574 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:26:48.574 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:48.574 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:26:48.574 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.575 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:26:48.575 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:48.043275+0000", 2024-09-13T11:26:48.575 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:48.575 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.575 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.575 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:26:48.575 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:48.043275+0000", 2024-09-13T11:26:48.575 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:48.575 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.575 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.575 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:26:48.575 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:48.043277+0000", 2024-09-13T11:26:48.575 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3120000000000001e-06 2024-09-13T11:26:48.576 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.576 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.576 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:26:48.576 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:48.043282+0000", 2024-09-13T11:26:48.576 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.0989999999999998e-06 2024-09-13T11:26:48.576 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.576 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.576 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:26:48.576 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:48.228541+0000", 2024-09-13T11:26:48.576 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.18525983200000001 2024-09-13T11:26:48.576 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.576 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.576 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:26:48.576 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:48.228548+0000", 2024-09-13T11:26:48.577 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.2740000000000004e-06 2024-09-13T11:26:48.577 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.577 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.577 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:26:48.577 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:48.228596+0000", 2024-09-13T11:26:48.577 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.8467999999999997e-05 2024-09-13T11:26:48.577 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.577 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.577 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:26:48.577 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:48.228612+0000", 2024-09-13T11:26:48.577 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.6189e-05 2024-09-13T11:26:48.577 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.577 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.578 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:26:48.578 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:48.230713+0000", 2024-09-13T11:26:48.578 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0021005780000000001 2024-09-13T11:26:48.578 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:48.578 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:48.578 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:26:48.578 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:48.230776+0000", 2024-09-13T11:26:48.578 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.3187000000000003e-05 2024-09-13T11:26:48.578 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:48.578 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:26:48.578 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:48.578 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:48.578 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:26:48.578 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-09-13T11:26:48.580 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-09-13T11:26:48.581 INFO:tasks.ceph.osd.12.smithi001.stderr:2024-09-13T11:26:48.578+0000 7f892c6e6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26314) UID: 0 2024-09-13T11:26:48.680 INFO:tasks.ceph.osd.5.smithi092.stderr:2024-09-13T11:26:48.675+0000 7f6f0b705640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26346) UID: 0 2024-09-13T11:26:48.761 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-09-13T11:26:48.761 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [], 2024-09-13T11:26:48.761 INFO:teuthology.orchestra.run.smithi177.stdout: "num_ops": 0 2024-09-13T11:26:48.761 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-09-13T11:26:48.776 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-09-13T11:26:48.781 INFO:tasks.ceph.osd.2.smithi162.stderr:2024-09-13T11:26:48.779+0000 7fe92b35a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26313) UID: 0 2024-09-13T11:26:48.882 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-09-13T11:26:48.879+0000 7f97befc9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26315) UID: 0 2024-09-13T11:26:48.894 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-09-13T11:26:48.894 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [], 2024-09-13T11:26:48.894 INFO:teuthology.orchestra.run.smithi177.stdout: "complaint_time": 30, 2024-09-13T11:26:48.894 INFO:teuthology.orchestra.run.smithi177.stdout: "num_blocked_ops": 0 2024-09-13T11:26:48.894 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-09-13T11:26:48.907 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-09-13T11:26:48.970 INFO:tasks.ceph.osd.11.smithi177.stderr:2024-09-13T11:26:48.969+0000 7f246bcf0640 -1 osd.11 223 handle_osd_map injecting map crc failure 2024-09-13T11:26:48.970 INFO:tasks.ceph.osd.3.smithi177.stderr:2024-09-13T11:26:48.969+0000 7f10a45d5640 -1 osd.3 223 handle_osd_map injecting map crc failure 2024-09-13T11:26:48.983 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-13T11:26:48.983 INFO:tasks.ceph.osd.7.smithi177.stderr:2024-09-13T11:26:48.981+0000 7f897b51d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26575) UID: 0 2024-09-13T11:26:49.074 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-09-13T11:26:49.074 INFO:teuthology.orchestra.run.smithi177.stdout: "size": 20, 2024-09-13T11:26:49.075 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 600, 2024-09-13T11:26:49.075 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [ 2024-09-13T11:26:49.075 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.075 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:79 3.1 e32/30 3:841df5b5:::benchmark_data_smithi177_31116_object78:head v 32'6, mlcod=32'4)", 2024-09-13T11:26:49.075 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:22:27.057292+0000", 2024-09-13T11:26:49.075 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 262.01646283299999, 2024-09-13T11:26:49.075 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.89296786800000005, 2024-09-13T11:26:49.075 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:26:49.075 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:26:49.075 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:49.075 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:26:49.075 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.075 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:26:49.075 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.057292+0000", 2024-09-13T11:26:49.076 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:49.076 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.076 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.076 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:26:49.076 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.057292+0000", 2024-09-13T11:26:49.076 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:49.076 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.076 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.076 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:26:49.076 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.057293+0000", 2024-09-13T11:26:49.076 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.6109999999999999e-06 2024-09-13T11:26:49.076 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.076 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.077 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:26:49.077 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.057330+0000", 2024-09-13T11:26:49.077 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.6721999999999998e-05 2024-09-13T11:26:49.077 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.077 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.077 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:26:49.077 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.948197+0000", 2024-09-13T11:26:49.077 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.890867042 2024-09-13T11:26:49.077 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.077 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.077 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:26:49.077 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.948213+0000", 2024-09-13T11:26:49.077 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5826000000000001e-05 2024-09-13T11:26:49.078 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.078 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.078 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:26:49.078 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.948377+0000", 2024-09-13T11:26:49.078 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00016348600000000001 2024-09-13T11:26:49.078 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.078 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.078 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:26:49.078 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.948447+0000", 2024-09-13T11:26:49.078 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.0891999999999998e-05 2024-09-13T11:26:49.078 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.078 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.078 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:26:49.078 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.950150+0000", 2024-09-13T11:26:49.079 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001702329 2024-09-13T11:26:49.079 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.079 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.079 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:26:49.079 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.950260+0000", 2024-09-13T11:26:49.079 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00010996 2024-09-13T11:26:49.079 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:49.079 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:26:49.079 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:49.079 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.079 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.079 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:45783 3.16 3.17c10a76 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e69)", 2024-09-13T11:26:49.079 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:20.947774+0000", 2024-09-13T11:26:49.079 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 208.12598058200001, 2024-09-13T11:26:49.080 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.96682922400000004, 2024-09-13T11:26:49.080 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:26:49.080 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:26:49.080 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-13T11:26:49.080 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:26:49.080 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:26:49.080 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:49.080 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 45783 2024-09-13T11:26:49.080 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.080 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:26:49.080 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.080 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:26:49.080 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:20.947774+0000", 2024-09-13T11:26:49.081 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:49.081 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.081 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.081 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:26:49.081 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:20.947774+0000", 2024-09-13T11:26:49.081 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:49.081 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.081 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.081 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:26:49.081 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:20.947777+0000", 2024-09-13T11:26:49.081 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.57e-06 2024-09-13T11:26:49.081 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.081 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.081 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:26:49.081 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:20.947802+0000", 2024-09-13T11:26:49.082 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.5528999999999999e-05 2024-09-13T11:26:49.082 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.082 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.082 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:26:49.082 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:20.947813+0000", 2024-09-13T11:26:49.082 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0473e-05 2024-09-13T11:26:49.082 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.082 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.082 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:26:49.082 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:20.947821+0000", 2024-09-13T11:26:49.082 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.3440000000000001e-06 2024-09-13T11:26:49.082 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.082 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.082 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:26:49.083 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:20.947933+0000", 2024-09-13T11:26:49.083 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00011210200000000001 2024-09-13T11:26:49.083 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.083 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.083 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for peered", 2024-09-13T11:26:49.083 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:20.947942+0000", 2024-09-13T11:26:49.083 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.6610000000000006e-06 2024-09-13T11:26:49.083 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.083 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.083 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:26:49.083 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:21.914579+0000", 2024-09-13T11:26:49.083 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.96663672300000003 2024-09-13T11:26:49.083 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.083 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.084 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:26:49.084 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:21.914603+0000", 2024-09-13T11:26:49.084 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.4822000000000001e-05 2024-09-13T11:26:49.084 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:49.084 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:26:49.084 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:49.084 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.084 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.084 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:54625 3.1f 3:f812cdfb:::benchmark_data_smithi177_31116_object54624:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e75)", 2024-09-13T11:26:49.084 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:30.979327+0000", 2024-09-13T11:26:49.084 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 198.09442779299999, 2024-09-13T11:26:49.084 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.96163670899999998, 2024-09-13T11:26:49.084 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:26:49.084 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:26:49.085 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-13T11:26:49.085 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:26:49.085 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:26:49.085 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:49.085 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 54625 2024-09-13T11:26:49.085 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.085 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:26:49.085 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.085 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:26:49.085 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.979327+0000", 2024-09-13T11:26:49.085 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:49.085 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.085 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.085 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:26:49.086 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.979327+0000", 2024-09-13T11:26:49.086 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:49.086 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.086 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.086 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:26:49.086 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.979329+0000", 2024-09-13T11:26:49.086 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.686e-06 2024-09-13T11:26:49.086 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.086 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.086 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:26:49.086 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.979364+0000", 2024-09-13T11:26:49.086 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.5750000000000002e-05 2024-09-13T11:26:49.086 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.086 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.087 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:26:49.087 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.979408+0000", 2024-09-13T11:26:49.087 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.3795999999999999e-05 2024-09-13T11:26:49.087 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.087 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.087 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:26:49.087 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.979427+0000", 2024-09-13T11:26:49.087 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.9040999999999999e-05 2024-09-13T11:26:49.087 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.087 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.087 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:26:49.087 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.122291+0000", 2024-09-13T11:26:49.087 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.14286373599999999 2024-09-13T11:26:49.087 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.087 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.088 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for peered", 2024-09-13T11:26:49.088 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.122298+0000", 2024-09-13T11:26:49.088 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.3719999999999999e-06 2024-09-13T11:26:49.088 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.088 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.088 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:26:49.088 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.940341+0000", 2024-09-13T11:26:49.088 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.81804249699999998 2024-09-13T11:26:49.088 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.088 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.088 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:26:49.088 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.940964+0000", 2024-09-13T11:26:49.088 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00062283100000000001 2024-09-13T11:26:49.088 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:49.088 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:26:49.089 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:49.089 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.089 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.089 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:54649 3.1c 3:3a457a93:::benchmark_data_smithi177_31116_object54648:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e75)", 2024-09-13T11:26:49.089 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:30.979460+0000", 2024-09-13T11:26:49.089 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 198.09429460300001, 2024-09-13T11:26:49.089 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.95579424000000002, 2024-09-13T11:26:49.089 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:26:49.089 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:26:49.089 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-13T11:26:49.089 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:26:49.089 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:26:49.089 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:49.089 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 54649 2024-09-13T11:26:49.090 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.090 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:26:49.090 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.090 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:26:49.090 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.979460+0000", 2024-09-13T11:26:49.090 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:49.090 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.090 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.090 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:26:49.090 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.979460+0000", 2024-09-13T11:26:49.090 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:49.090 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.090 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.090 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:26:49.090 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.979462+0000", 2024-09-13T11:26:49.091 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5799999999999999e-06 2024-09-13T11:26:49.091 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.091 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.091 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:26:49.091 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.979524+0000", 2024-09-13T11:26:49.091 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.2521999999999995e-05 2024-09-13T11:26:49.091 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.091 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.091 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:26:49.091 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.979561+0000", 2024-09-13T11:26:49.091 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.6349e-05 2024-09-13T11:26:49.091 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.091 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.091 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:26:49.092 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.979571+0000", 2024-09-13T11:26:49.092 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0457e-05 2024-09-13T11:26:49.092 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.092 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.092 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:26:49.092 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.121029+0000", 2024-09-13T11:26:49.092 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.14145766200000001 2024-09-13T11:26:49.092 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.092 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.092 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for peered", 2024-09-13T11:26:49.092 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.121039+0000", 2024-09-13T11:26:49.092 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0084e-05 2024-09-13T11:26:49.092 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.092 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.092 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:26:49.093 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.935157+0000", 2024-09-13T11:26:49.093 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.81411807899999999 2024-09-13T11:26:49.093 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.093 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.093 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:26:49.093 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.935254+0000", 2024-09-13T11:26:49.093 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.7507e-05 2024-09-13T11:26:49.093 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:49.093 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:26:49.093 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:49.093 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.093 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.093 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:54654 3.1c 3:3f0bb31a:::benchmark_data_smithi177_31116_object54653:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e75)", 2024-09-13T11:26:49.093 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:30.979665+0000", 2024-09-13T11:26:49.093 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 198.094089438, 2024-09-13T11:26:49.094 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.95623908099999999, 2024-09-13T11:26:49.094 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:26:49.094 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:26:49.094 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-13T11:26:49.094 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:26:49.094 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:26:49.094 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:49.094 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 54654 2024-09-13T11:26:49.094 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.094 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:26:49.094 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.094 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:26:49.094 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.979665+0000", 2024-09-13T11:26:49.094 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:49.094 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.095 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.095 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:26:49.095 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.979665+0000", 2024-09-13T11:26:49.095 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:49.095 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.095 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.095 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:26:49.095 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.979667+0000", 2024-09-13T11:26:49.095 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3540000000000001e-06 2024-09-13T11:26:49.095 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.095 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.095 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:26:49.095 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.979703+0000", 2024-09-13T11:26:49.096 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.6758000000000003e-05 2024-09-13T11:26:49.096 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.096 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.096 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:26:49.096 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.979714+0000", 2024-09-13T11:26:49.096 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0804e-05 2024-09-13T11:26:49.096 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.096 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.096 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:26:49.096 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.979724+0000", 2024-09-13T11:26:49.096 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0040999999999999e-05 2024-09-13T11:26:49.096 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.096 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.096 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:26:49.097 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.121061+0000", 2024-09-13T11:26:49.097 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.141336713 2024-09-13T11:26:49.097 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.097 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.097 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for peered", 2024-09-13T11:26:49.097 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.121067+0000", 2024-09-13T11:26:49.097 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.4150000000000004e-06 2024-09-13T11:26:49.097 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.097 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.097 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:26:49.097 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.935776+0000", 2024-09-13T11:26:49.097 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.81470863100000002 2024-09-13T11:26:49.097 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.098 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.098 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:26:49.098 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.935904+0000", 2024-09-13T11:26:49.098 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00012836499999999999 2024-09-13T11:26:49.098 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:49.098 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:26:49.098 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:49.098 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.098 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.098 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPushReply(3.11 75/70 [PushReplyOp(3:8df10771:::benchmark_data_smithi177_31116_object24734:head)])", 2024-09-13T11:26:49.098 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:31.149798+0000", 2024-09-13T11:26:49.098 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 197.923956428, 2024-09-13T11:26:49.098 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.32402702500000002, 2024-09-13T11:26:49.098 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:26:49.098 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:26:49.099 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-13T11:26:49.099 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:26:49.099 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.099 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:26:49.099 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.149798+0000", 2024-09-13T11:26:49.099 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:49.099 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.099 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.099 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:26:49.099 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.149798+0000", 2024-09-13T11:26:49.099 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:49.099 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.099 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.099 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:26:49.099 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.149799+0000", 2024-09-13T11:26:49.100 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.5700000000000001e-07 2024-09-13T11:26:49.100 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.100 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.100 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:26:49.100 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.149803+0000", 2024-09-13T11:26:49.100 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.0040000000000001e-06 2024-09-13T11:26:49.100 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.100 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.100 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:26:49.100 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.472172+0000", 2024-09-13T11:26:49.100 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.32236856699999999 2024-09-13T11:26:49.100 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.100 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.100 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:26:49.101 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.472187+0000", 2024-09-13T11:26:49.101 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5016e-05 2024-09-13T11:26:49.101 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.101 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.101 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:26:49.101 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.472302+0000", 2024-09-13T11:26:49.101 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000115121 2024-09-13T11:26:49.101 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.101 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.101 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:26:49.101 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.473825+0000", 2024-09-13T11:26:49.101 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0015234599999999999 2024-09-13T11:26:49.101 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:49.101 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:26:49.101 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:49.102 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.102 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.102 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:54668 3.1c 3:3d2c5e40:::benchmark_data_smithi177_31116_object54667:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e75)", 2024-09-13T11:26:49.102 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:31.245525+0000", 2024-09-13T11:26:49.102 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 197.828230171, 2024-09-13T11:26:49.102 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.69828973000000005, 2024-09-13T11:26:49.102 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:26:49.102 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:26:49.102 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:49.102 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:26:49.102 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:26:49.102 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:49.102 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 54668 2024-09-13T11:26:49.102 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.103 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:26:49.103 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.103 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:26:49.103 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.245525+0000", 2024-09-13T11:26:49.103 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:49.103 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.103 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.103 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:26:49.103 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.245525+0000", 2024-09-13T11:26:49.103 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:49.103 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.103 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.103 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:26:49.103 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.245526+0000", 2024-09-13T11:26:49.104 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.719e-06 2024-09-13T11:26:49.104 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.104 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.104 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:26:49.104 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.245547+0000", 2024-09-13T11:26:49.104 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.1084999999999999e-05 2024-09-13T11:26:49.104 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.104 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.104 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:26:49.104 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.245558+0000", 2024-09-13T11:26:49.104 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0934000000000001e-05 2024-09-13T11:26:49.104 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.104 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.105 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:26:49.105 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.245564+0000", 2024-09-13T11:26:49.105 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.6269999999999997e-06 2024-09-13T11:26:49.105 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.105 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.105 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:26:49.105 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.245620+0000", 2024-09-13T11:26:49.105 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.6289999999999998e-05 2024-09-13T11:26:49.105 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.105 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.105 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for peered", 2024-09-13T11:26:49.105 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.245632+0000", 2024-09-13T11:26:49.105 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.1748e-05 2024-09-13T11:26:49.105 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.106 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.106 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:26:49.106 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.935976+0000", 2024-09-13T11:26:49.106 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.69034455299999997 2024-09-13T11:26:49.106 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.106 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.106 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:26:49.106 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.936302+0000", 2024-09-13T11:26:49.106 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000325397 2024-09-13T11:26:49.106 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.106 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.106 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 13", 2024-09-13T11:26:49.106 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.936459+0000", 2024-09-13T11:26:49.107 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00015754300000000001 2024-09-13T11:26:49.107 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.107 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.107 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:26:49.107 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.938813+0000", 2024-09-13T11:26:49.107 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0023530999999999999 2024-09-13T11:26:49.107 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.107 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.107 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:49.107 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.943695+0000", 2024-09-13T11:26:49.107 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.004882506 2024-09-13T11:26:49.107 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.107 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.107 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:26:49.108 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.943767+0000", 2024-09-13T11:26:49.108 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.2454999999999997e-05 2024-09-13T11:26:49.108 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.108 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.108 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:26:49.108 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.943814+0000", 2024-09-13T11:26:49.108 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.6773000000000001e-05 2024-09-13T11:26:49.108 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:49.108 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:26:49.108 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:49.108 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.108 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.108 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPush(3.16 84/70 [PushOp(3:6f7be541:::benchmark_data_smithi177_31116_object43332:head, version: 66'2788, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:6f7be541:::benchmark_data_smithi177_31116_object43332:head@66'2788, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:26:49.108 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:41.207515+0000", 2024-09-13T11:26:49.109 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 187.86623940000001, 2024-09-13T11:26:49.109 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.52643771500000003, 2024-09-13T11:26:49.109 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:26:49.109 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:26:49.109 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:26:49.109 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:26:49.109 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.109 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:26:49.109 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.207515+0000", 2024-09-13T11:26:49.109 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:49.109 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.109 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.109 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:26:49.110 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.207515+0000", 2024-09-13T11:26:49.110 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:49.110 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.110 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.110 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:26:49.110 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.207516+0000", 2024-09-13T11:26:49.110 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.096e-06 2024-09-13T11:26:49.110 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.110 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.110 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:26:49.110 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.207717+0000", 2024-09-13T11:26:49.110 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000200199 2024-09-13T11:26:49.110 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.110 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.111 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:26:49.111 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.732785+0000", 2024-09-13T11:26:49.111 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.52506869599999995 2024-09-13T11:26:49.111 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.111 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.111 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:26:49.111 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.732801+0000", 2024-09-13T11:26:49.111 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5305999999999999e-05 2024-09-13T11:26:49.111 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.111 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.111 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:26:49.111 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.732889+0000", 2024-09-13T11:26:49.111 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.8157000000000003e-05 2024-09-13T11:26:49.112 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.112 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.112 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:26:49.112 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.732901+0000", 2024-09-13T11:26:49.112 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.1866e-05 2024-09-13T11:26:49.112 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.112 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.112 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:26:49.112 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.733953+0000", 2024-09-13T11:26:49.112 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001052395 2024-09-13T11:26:49.112 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:49.112 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:26:49.112 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:49.112 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.112 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.113 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPushReply(3.1f 85/78 [PushReplyOp(3:fd221d21:::benchmark_data_smithi177_31116_object7639:head)])", 2024-09-13T11:26:49.113 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:42.198715+0000", 2024-09-13T11:26:49.113 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 186.87503983100001, 2024-09-13T11:26:49.113 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.48930391099999998, 2024-09-13T11:26:49.113 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:26:49.113 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:26:49.113 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-13T11:26:49.113 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:26:49.113 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.113 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:26:49.113 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.198715+0000", 2024-09-13T11:26:49.113 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:49.113 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.113 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.114 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:26:49.114 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.198715+0000", 2024-09-13T11:26:49.114 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:49.114 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.114 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.114 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:26:49.114 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.198716+0000", 2024-09-13T11:26:49.114 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.3500000000000005e-07 2024-09-13T11:26:49.114 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.114 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.114 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:26:49.114 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.198719+0000", 2024-09-13T11:26:49.114 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.4539999999999999e-06 2024-09-13T11:26:49.114 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.114 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.115 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:26:49.115 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.684732+0000", 2024-09-13T11:26:49.115 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.48601262099999998 2024-09-13T11:26:49.115 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.115 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.115 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:26:49.115 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.684746+0000", 2024-09-13T11:26:49.115 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.4518999999999999e-05 2024-09-13T11:26:49.115 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.115 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.115 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:26:49.115 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.684911+0000", 2024-09-13T11:26:49.115 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000164728 2024-09-13T11:26:49.115 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.116 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.116 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:26:49.116 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.688019+0000", 2024-09-13T11:26:49.116 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0031077539999999999 2024-09-13T11:26:49.116 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:49.116 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:26:49.116 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:49.116 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.116 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.116 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPushReply(3.1c 85/77 [PushReplyOp(3:3a669437:::benchmark_data_smithi177_31116_object25280:head)])", 2024-09-13T11:26:49.116 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:42.352466+0000", 2024-09-13T11:26:49.116 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 186.721288937, 2024-09-13T11:26:49.116 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.36670725300000001, 2024-09-13T11:26:49.117 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:26:49.117 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:26:49.117 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-13T11:26:49.117 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:26:49.117 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.117 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:26:49.117 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.352466+0000", 2024-09-13T11:26:49.117 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:49.117 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.117 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.117 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:26:49.117 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.352466+0000", 2024-09-13T11:26:49.117 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:49.117 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.118 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.118 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:26:49.118 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.352467+0000", 2024-09-13T11:26:49.118 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.1429999999999999e-06 2024-09-13T11:26:49.118 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.118 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.118 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:26:49.118 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.352474+0000", 2024-09-13T11:26:49.118 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.6100000000000002e-06 2024-09-13T11:26:49.118 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.118 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.118 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:26:49.118 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.717011+0000", 2024-09-13T11:26:49.118 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.36453790200000002 2024-09-13T11:26:49.119 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.119 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.119 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:26:49.119 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.717032+0000", 2024-09-13T11:26:49.119 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.0999e-05 2024-09-13T11:26:49.119 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.119 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.119 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:26:49.119 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.717162+0000", 2024-09-13T11:26:49.119 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000129159 2024-09-13T11:26:49.119 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.119 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.119 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:26:49.119 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.719173+0000", 2024-09-13T11:26:49.120 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0020114400000000002 2024-09-13T11:26:49.120 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:49.120 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:26:49.120 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:49.120 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.120 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.120 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPushReply(3.1f 85/78 [PushReplyOp(3:fecb0d6a:::benchmark_data_smithi177_31116_object7644:head)])", 2024-09-13T11:26:49.120 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:42.705351+0000", 2024-09-13T11:26:49.120 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 186.36840409499999, 2024-09-13T11:26:49.120 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.290549681, 2024-09-13T11:26:49.120 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:26:49.120 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:26:49.120 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-13T11:26:49.121 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:26:49.121 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.121 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:26:49.121 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.705351+0000", 2024-09-13T11:26:49.121 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:49.121 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.121 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.121 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:26:49.121 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.705351+0000", 2024-09-13T11:26:49.121 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:49.121 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.121 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.121 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:26:49.122 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.705353+0000", 2024-09-13T11:26:49.122 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.4169999999999999e-06 2024-09-13T11:26:49.122 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.122 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.122 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:26:49.122 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.705365+0000", 2024-09-13T11:26:49.122 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.1894e-05 2024-09-13T11:26:49.122 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.122 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.122 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:26:49.122 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.992081+0000", 2024-09-13T11:26:49.122 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.28671641199999998 2024-09-13T11:26:49.122 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.122 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.123 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:26:49.123 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.992111+0000", 2024-09-13T11:26:49.123 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.9427000000000001e-05 2024-09-13T11:26:49.123 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.123 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.123 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:26:49.123 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.993573+0000", 2024-09-13T11:26:49.123 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0014623590000000001 2024-09-13T11:26:49.123 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.123 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.123 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:26:49.123 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.995900+0000", 2024-09-13T11:26:49.123 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0023271720000000002 2024-09-13T11:26:49.123 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:49.124 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:26:49.124 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:49.124 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.124 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.124 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPushReply(3.21 89/78 [PushReplyOp(3:8534d087:::benchmark_data_smithi177_31116_object5474:head)])", 2024-09-13T11:26:49.124 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:46.632834+0000", 2024-09-13T11:26:49.124 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 182.44092065500001, 2024-09-13T11:26:49.124 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.324805556, 2024-09-13T11:26:49.124 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:26:49.124 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:26:49.124 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-13T11:26:49.124 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:26:49.124 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.124 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:26:49.125 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.632834+0000", 2024-09-13T11:26:49.125 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:49.125 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.125 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.125 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:26:49.125 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.632834+0000", 2024-09-13T11:26:49.125 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:49.125 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.125 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.125 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:26:49.125 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.632835+0000", 2024-09-13T11:26:49.125 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.4270000000000001e-06 2024-09-13T11:26:49.125 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.125 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.125 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:26:49.126 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.632840+0000", 2024-09-13T11:26:49.126 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.7809999999999996e-06 2024-09-13T11:26:49.126 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.126 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.126 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:26:49.126 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.956282+0000", 2024-09-13T11:26:49.126 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.323441639 2024-09-13T11:26:49.126 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.126 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.126 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:26:49.126 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.956293+0000", 2024-09-13T11:26:49.126 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.083e-05 2024-09-13T11:26:49.126 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.127 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.127 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:26:49.127 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.956395+0000", 2024-09-13T11:26:49.127 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00010184200000000001 2024-09-13T11:26:49.127 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.127 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.127 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:26:49.127 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.957640+0000", 2024-09-13T11:26:49.127 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0012450370000000001 2024-09-13T11:26:49.127 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:49.127 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:26:49.127 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:49.127 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.127 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.127 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:83223 3.1c 3:3bc6018d:::benchmark_data_smithi177_31116_object83222:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106)", 2024-09-13T11:26:49.128 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:04.511163+0000", 2024-09-13T11:26:49.128 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 164.56259127800001, 2024-09-13T11:26:49.128 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.47558093200000001, 2024-09-13T11:26:49.128 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:26:49.128 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:26:49.128 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:49.128 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:26:49.128 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:26:49.128 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:49.128 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 83223 2024-09-13T11:26:49.128 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.128 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:26:49.128 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.128 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:26:49.128 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.511163+0000", 2024-09-13T11:26:49.129 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:49.129 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.129 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.129 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:26:49.129 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.511163+0000", 2024-09-13T11:26:49.129 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:49.129 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.129 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.129 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:26:49.129 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.511167+0000", 2024-09-13T11:26:49.129 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.8330000000000004e-06 2024-09-13T11:26:49.129 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.129 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.129 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:26:49.130 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.511219+0000", 2024-09-13T11:26:49.130 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.1771999999999998e-05 2024-09-13T11:26:49.130 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.130 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.130 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:26:49.130 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.511252+0000", 2024-09-13T11:26:49.130 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.3315999999999997e-05 2024-09-13T11:26:49.130 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.130 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.130 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:26:49.130 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.511268+0000", 2024-09-13T11:26:49.130 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5812999999999999e-05 2024-09-13T11:26:49.130 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.130 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.131 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:26:49.131 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.511418+0000", 2024-09-13T11:26:49.131 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000150023 2024-09-13T11:26:49.131 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.131 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.131 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:26:49.131 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.511670+0000", 2024-09-13T11:26:49.131 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00025185499999999999 2024-09-13T11:26:49.131 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.131 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.131 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 7", 2024-09-13T11:26:49.131 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.511905+0000", 2024-09-13T11:26:49.131 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00023493799999999999 2024-09-13T11:26:49.131 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.131 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.132 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:26:49.132 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.514960+0000", 2024-09-13T11:26:49.132 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.003054977 2024-09-13T11:26:49.132 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.132 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.132 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:49.132 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.986432+0000", 2024-09-13T11:26:49.132 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.47147209200000001 2024-09-13T11:26:49.132 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.132 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.132 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:26:49.132 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.986635+0000", 2024-09-13T11:26:49.132 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00020309899999999999 2024-09-13T11:26:49.133 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.133 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.133 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:26:49.133 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.986744+0000", 2024-09-13T11:26:49.133 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000109214 2024-09-13T11:26:49.133 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:49.133 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:26:49.133 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:49.133 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.133 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.133 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop_reply(client.4665.0:84803 3.1f e107/78)", 2024-09-13T11:26:49.133 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:05.912627+0000", 2024-09-13T11:26:49.133 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 163.16112737099999, 2024-09-13T11:26:49.133 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.33726566600000002, 2024-09-13T11:26:49.134 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:26:49.134 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:26:49.134 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-13T11:26:49.134 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:26:49.134 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.134 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:26:49.134 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:05.912627+0000", 2024-09-13T11:26:49.134 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:49.134 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.134 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.134 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:26:49.134 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:05.912627+0000", 2024-09-13T11:26:49.134 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:49.134 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.134 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.135 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:26:49.135 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:05.912629+0000", 2024-09-13T11:26:49.135 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.4440000000000001e-06 2024-09-13T11:26:49.135 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.135 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.135 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:26:49.135 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:05.912634+0000", 2024-09-13T11:26:49.135 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.6869999999999999e-06 2024-09-13T11:26:49.135 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.135 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.135 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:26:49.135 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:06.249722+0000", 2024-09-13T11:26:49.135 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.33708766099999998 2024-09-13T11:26:49.136 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.136 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.136 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:26:49.136 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:06.249736+0000", 2024-09-13T11:26:49.136 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3623e-05 2024-09-13T11:26:49.136 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.136 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.136 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:26:49.136 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:06.249878+0000", 2024-09-13T11:26:49.136 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00014264599999999999 2024-09-13T11:26:49.136 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.136 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.136 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:26:49.136 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:06.249893+0000", 2024-09-13T11:26:49.137 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.4605e-05 2024-09-13T11:26:49.137 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:49.137 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:26:49.137 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:49.137 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.137 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.137 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop_reply(client.4665.0:110260 3.f e119/75 ondisk, result = 0)", 2024-09-13T11:26:49.137 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:25.333188+0000", 2024-09-13T11:26:49.137 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 143.74056674400001, 2024-09-13T11:26:49.137 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.29195486199999998, 2024-09-13T11:26:49.137 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:26:49.137 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:26:49.137 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:26:49.138 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:26:49.138 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.138 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:26:49.138 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.333188+0000", 2024-09-13T11:26:49.138 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:49.138 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.138 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.138 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:26:49.138 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.333188+0000", 2024-09-13T11:26:49.138 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:49.138 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.138 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.138 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:26:49.138 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.333190+0000", 2024-09-13T11:26:49.139 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.2120000000000002e-06 2024-09-13T11:26:49.139 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.139 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.139 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:26:49.139 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.333197+0000", 2024-09-13T11:26:49.139 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.7469999999999999e-06 2024-09-13T11:26:49.139 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.139 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.139 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:26:49.139 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.624892+0000", 2024-09-13T11:26:49.139 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.291694913 2024-09-13T11:26:49.139 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.139 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.139 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:26:49.139 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.624907+0000", 2024-09-13T11:26:49.140 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5053e-05 2024-09-13T11:26:49.140 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.140 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.140 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:26:49.140 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.624999+0000", 2024-09-13T11:26:49.140 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.2603999999999996e-05 2024-09-13T11:26:49.140 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.140 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.140 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:26:49.140 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.625011+0000", 2024-09-13T11:26:49.140 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.1253999999999999e-05 2024-09-13T11:26:49.140 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.140 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.141 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:26:49.141 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.625143+0000", 2024-09-13T11:26:49.141 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00013207900000000001 2024-09-13T11:26:49.141 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:49.141 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:26:49.141 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:49.141 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.141 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.141 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:110265 3.1 e119/75 3:8236484d:::benchmark_data_smithi177_31116_object110264:head v 119'3935, mlcod=118'3933)", 2024-09-13T11:26:49.141 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:25.343348+0000", 2024-09-13T11:26:49.141 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 143.73040712900001, 2024-09-13T11:26:49.141 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.60013013500000001, 2024-09-13T11:26:49.141 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:26:49.141 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:26:49.142 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:49.142 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:26:49.142 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.142 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:26:49.142 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.343348+0000", 2024-09-13T11:26:49.142 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:49.142 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.142 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.142 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:26:49.142 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.343348+0000", 2024-09-13T11:26:49.142 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:49.142 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.142 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.142 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:26:49.142 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.343350+0000", 2024-09-13T11:26:49.143 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.576e-06 2024-09-13T11:26:49.143 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.143 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.143 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:26:49.143 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.343443+0000", 2024-09-13T11:26:49.143 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.2989e-05 2024-09-13T11:26:49.143 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.143 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.143 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:26:49.143 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.940863+0000", 2024-09-13T11:26:49.143 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.59742003300000002 2024-09-13T11:26:49.143 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.143 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.144 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:26:49.144 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.940875+0000", 2024-09-13T11:26:49.144 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.2147000000000001e-05 2024-09-13T11:26:49.144 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.144 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.144 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:26:49.144 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.940993+0000", 2024-09-13T11:26:49.144 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000117903 2024-09-13T11:26:49.144 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.144 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.144 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:26:49.144 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.941017+0000", 2024-09-13T11:26:49.144 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.3442999999999999e-05 2024-09-13T11:26:49.145 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.145 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.145 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:26:49.145 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.943347+0000", 2024-09-13T11:26:49.145 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0023299869999999999 2024-09-13T11:26:49.145 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.145 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.145 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:26:49.145 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.943478+0000", 2024-09-13T11:26:49.145 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00013105699999999999 2024-09-13T11:26:49.145 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:49.145 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:26:49.145 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:49.145 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.146 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.146 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop_reply(client.4665.0:129919 3.16 e134/88 ondisk, result = 0)", 2024-09-13T11:26:49.146 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:47.175742+0000", 2024-09-13T11:26:49.146 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 121.89801305899999, 2024-09-13T11:26:49.146 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.27593726299999999, 2024-09-13T11:26:49.146 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:26:49.146 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:26:49.146 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:26:49.146 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:26:49.146 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.146 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:26:49.146 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:47.175742+0000", 2024-09-13T11:26:49.146 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:49.146 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.147 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.147 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:26:49.147 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:47.175742+0000", 2024-09-13T11:26:49.147 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:49.147 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.147 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.147 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:26:49.147 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:47.175744+0000", 2024-09-13T11:26:49.147 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.0030000000000001e-06 2024-09-13T11:26:49.147 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.147 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.147 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:26:49.147 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:47.175751+0000", 2024-09-13T11:26:49.147 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.9779999999999999e-06 2024-09-13T11:26:49.147 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.148 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.148 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:26:49.148 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:47.451490+0000", 2024-09-13T11:26:49.148 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.27573892500000002 2024-09-13T11:26:49.148 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.148 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.148 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:26:49.148 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:47.451496+0000", 2024-09-13T11:26:49.148 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.0079999999999999e-06 2024-09-13T11:26:49.148 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.148 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.148 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:26:49.148 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:47.451560+0000", 2024-09-13T11:26:49.148 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.3955999999999995e-05 2024-09-13T11:26:49.149 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.149 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.149 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:26:49.149 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:47.451567+0000", 2024-09-13T11:26:49.149 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.1980000000000004e-06 2024-09-13T11:26:49.149 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.149 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.149 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:26:49.149 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:47.451679+0000", 2024-09-13T11:26:49.149 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00011219500000000001 2024-09-13T11:26:49.149 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:49.149 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:26:49.149 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:49.150 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.150 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.150 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:209869 3.27 e185/169 3:e6b1fcc5:::benchmark_data_smithi177_31116_object69566:head v 185'6350, mlcod=185'6347)", 2024-09-13T11:26:49.150 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:25:54.301402+0000", 2024-09-13T11:26:49.150 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 54.772352388000002, 2024-09-13T11:26:49.150 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.30148222499999999, 2024-09-13T11:26:49.150 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:26:49.150 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:26:49.150 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:49.150 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:26:49.150 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.150 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:26:49.150 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:54.301402+0000", 2024-09-13T11:26:49.150 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:49.151 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.151 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.151 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:26:49.151 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:54.301402+0000", 2024-09-13T11:26:49.151 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:49.151 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.151 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.151 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:26:49.151 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:54.301404+0000", 2024-09-13T11:26:49.151 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.1939999999999999e-06 2024-09-13T11:26:49.151 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.151 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.151 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:26:49.151 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:54.301409+0000", 2024-09-13T11:26:49.151 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.0259999999999998e-06 2024-09-13T11:26:49.152 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.152 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.152 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:26:49.152 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:54.601076+0000", 2024-09-13T11:26:49.152 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.29966704799999999 2024-09-13T11:26:49.152 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.152 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.152 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:26:49.152 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:54.601081+0000", 2024-09-13T11:26:49.152 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.7069999999999997e-06 2024-09-13T11:26:49.152 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.152 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.152 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:26:49.152 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:54.601135+0000", 2024-09-13T11:26:49.153 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.3439000000000001e-05 2024-09-13T11:26:49.153 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.153 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.153 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:26:49.153 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:54.601150+0000", 2024-09-13T11:26:49.153 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5443000000000001e-05 2024-09-13T11:26:49.153 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.153 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.153 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:26:49.153 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:54.602854+0000", 2024-09-13T11:26:49.153 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0017035220000000001 2024-09-13T11:26:49.153 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.153 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.153 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:26:49.153 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:54.602885+0000", 2024-09-13T11:26:49.154 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.0846000000000002e-05 2024-09-13T11:26:49.154 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:49.154 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:26:49.154 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:49.154 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.154 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.154 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:234746 3.1 e201/75 3:83449ff7:::benchmark_data_smithi177_31116_object94443:head v 201'5848, mlcod=201'5845)", 2024-09-13T11:26:49.154 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:26:17.887870+0000", 2024-09-13T11:26:49.154 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 31.185884830999999, 2024-09-13T11:26:49.154 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.29704435699999998, 2024-09-13T11:26:49.154 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:26:49.154 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:26:49.154 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:49.154 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:26:49.155 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.155 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:26:49.155 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:17.887870+0000", 2024-09-13T11:26:49.155 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:49.155 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.155 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.155 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:26:49.155 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:17.887870+0000", 2024-09-13T11:26:49.155 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:49.155 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.155 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.155 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:26:49.155 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:17.887871+0000", 2024-09-13T11:26:49.155 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.4750000000000001e-06 2024-09-13T11:26:49.156 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.156 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.156 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:26:49.156 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:17.887879+0000", 2024-09-13T11:26:49.156 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.2439999999999996e-06 2024-09-13T11:26:49.156 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.156 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.156 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:26:49.156 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:18.180816+0000", 2024-09-13T11:26:49.156 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.29293715799999998 2024-09-13T11:26:49.156 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.156 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.156 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:26:49.156 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:18.180828+0000", 2024-09-13T11:26:49.157 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.2713e-05 2024-09-13T11:26:49.157 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.157 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.157 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:26:49.157 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:18.181317+0000", 2024-09-13T11:26:49.157 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00048859800000000003 2024-09-13T11:26:49.157 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.157 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.157 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:26:49.157 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:18.181360+0000", 2024-09-13T11:26:49.157 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.3359000000000003e-05 2024-09-13T11:26:49.157 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.157 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.157 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:26:49.158 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:18.184822+0000", 2024-09-13T11:26:49.158 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0034615879999999998 2024-09-13T11:26:49.158 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.158 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.158 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:26:49.158 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:18.184914+0000", 2024-09-13T11:26:49.158 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.2221999999999999e-05 2024-09-13T11:26:49.158 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:49.158 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:26:49.158 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:49.158 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.158 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.158 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:260347 3.16 3:6d3745c1:::benchmark_data_smithi177_31116_object120044:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e219)", 2024-09-13T11:26:49.158 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:26:42.470302+0000", 2024-09-13T11:26:49.159 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 6.603452538, 2024-09-13T11:26:49.159 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.43228830200000001, 2024-09-13T11:26:49.159 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:26:49.159 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:26:49.159 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:49.159 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:26:49.159 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:26:49.159 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:49.159 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 260347 2024-09-13T11:26:49.159 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.159 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:26:49.159 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.159 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:26:49.159 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:42.470302+0000", 2024-09-13T11:26:49.159 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:49.160 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.160 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.160 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:26:49.160 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:42.470302+0000", 2024-09-13T11:26:49.160 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:49.160 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.160 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.160 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:26:49.160 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:42.470306+0000", 2024-09-13T11:26:49.160 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.2899999999999996e-06 2024-09-13T11:26:49.160 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.160 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.160 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:26:49.160 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:42.470318+0000", 2024-09-13T11:26:49.161 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.1426e-05 2024-09-13T11:26:49.161 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.161 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.161 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:26:49.161 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:42.470361+0000", 2024-09-13T11:26:49.161 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.35e-05 2024-09-13T11:26:49.161 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.161 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.161 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:26:49.161 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:42.470371+0000", 2024-09-13T11:26:49.161 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.1740000000000006e-06 2024-09-13T11:26:49.161 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.161 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.161 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:26:49.162 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:42.470474+0000", 2024-09-13T11:26:49.162 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000103225 2024-09-13T11:26:49.162 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.162 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.162 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:26:49.162 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:42.470735+0000", 2024-09-13T11:26:49.162 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00026167399999999999 2024-09-13T11:26:49.162 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.162 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.162 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 15", 2024-09-13T11:26:49.162 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:42.470939+0000", 2024-09-13T11:26:49.162 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00020340000000000001 2024-09-13T11:26:49.162 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.162 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.162 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:26:49.163 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:42.477467+0000", 2024-09-13T11:26:49.163 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0065277319999999996 2024-09-13T11:26:49.163 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.163 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.163 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:49.163 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:42.902321+0000", 2024-09-13T11:26:49.163 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.42485422099999998 2024-09-13T11:26:49.163 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.163 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.163 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:26:49.163 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:42.902468+0000", 2024-09-13T11:26:49.163 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000147591 2024-09-13T11:26:49.163 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.163 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.164 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:26:49.164 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:42.902590+0000", 2024-09-13T11:26:49.164 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000122069 2024-09-13T11:26:49.164 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:49.164 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:26:49.164 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:49.164 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:49.164 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:26:49.164 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-09-13T11:26:49.166 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_ops_in_flight 2024-09-13T11:26:49.167 INFO:tasks.ceph.osd.7.smithi177.stderr:2024-09-13T11:26:49.165+0000 7f897b51d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26575) UID: 0 2024-09-13T11:26:49.266 INFO:tasks.ceph.osd.4.smithi001.stderr:2024-09-13T11:26:49.262+0000 7ff8e0218640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26310) UID: 0 2024-09-13T11:26:49.362 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-09-13T11:26:49.363 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [ 2024-09-13T11:26:49.363 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.363 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:266887 3.5 e224/205 3:a2e9b064:::benchmark_data_smithi177_31116_object126584:head v 224'7150, mlcod=224'7148)", 2024-09-13T11:26:49.363 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:26:49.363980+0000", 2024-09-13T11:26:49.363 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 0.00018996599999999999, 2024-09-13T11:26:49.363 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00020614099999999999, 2024-09-13T11:26:49.363 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:26:49.363 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:26:49.363 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:26:49.363 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:26:49.363 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.363 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:26:49.363 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:49.363980+0000", 2024-09-13T11:26:49.364 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:49.364 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.364 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.364 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:26:49.364 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:49.363980+0000", 2024-09-13T11:26:49.364 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:49.364 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.364 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.364 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:26:49.364 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:49.363981+0000", 2024-09-13T11:26:49.364 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.1990000000000001e-06 2024-09-13T11:26:49.364 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.364 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.365 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:26:49.365 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:49.363988+0000", 2024-09-13T11:26:49.365 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.3910000000000003e-06 2024-09-13T11:26:49.365 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.365 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.365 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:26:49.365 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:49.364009+0000", 2024-09-13T11:26:49.365 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.0638000000000001e-05 2024-09-13T11:26:49.365 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.365 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.365 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:26:49.366 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:49.364016+0000", 2024-09-13T11:26:49.366 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.875e-06 2024-09-13T11:26:49.366 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.366 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.366 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:26:49.366 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:49.364068+0000", 2024-09-13T11:26:49.366 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.1814999999999998e-05 2024-09-13T11:26:49.366 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.366 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.366 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:26:49.366 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:49.364102+0000", 2024-09-13T11:26:49.366 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.3467000000000003e-05 2024-09-13T11:26:49.366 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:49.367 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:26:49.367 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:49.367 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:49.367 INFO:teuthology.orchestra.run.smithi177.stdout: ], 2024-09-13T11:26:49.367 INFO:teuthology.orchestra.run.smithi177.stdout: "num_ops": 1 2024-09-13T11:26:49.367 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-09-13T11:26:49.368 INFO:tasks.ceph.osd.15.smithi177.stderr:2024-09-13T11:26:49.365+0000 7f9d6ce77640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26578) UID: 0 2024-09-13T11:26:49.376 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_blocked_ops 2024-09-13T11:26:49.468 INFO:tasks.ceph.osd.15.smithi177.stderr:2024-09-13T11:26:49.465+0000 7f9d6ce77640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26578) UID: 0 2024-09-13T11:26:49.540 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-09-13T11:26:49.540 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [], 2024-09-13T11:26:49.540 INFO:teuthology.orchestra.run.smithi177.stdout: "complaint_time": 30, 2024-09-13T11:26:49.540 INFO:teuthology.orchestra.run.smithi177.stdout: "num_blocked_ops": 0 2024-09-13T11:26:49.540 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-09-13T11:26:49.554 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_historic_ops 2024-09-13T11:26:49.570 INFO:tasks.ceph.osd.10.smithi162.stderr:2024-09-13T11:26:49.567+0000 7fb59c9f0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26317) UID: 0 2024-09-13T11:26:49.671 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-09-13T11:26:49.667+0000 7f97befc9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26315) UID: 0 2024-09-13T11:26:49.729 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-09-13T11:26:49.729 INFO:teuthology.orchestra.run.smithi177.stdout: "size": 20, 2024-09-13T11:26:49.730 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 600, 2024-09-13T11:26:49.730 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [ 2024-09-13T11:26:49.730 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.730 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:25383 3.0 e55/30 3:0a6bb869:::benchmark_data_smithi177_31116_object25382:head v 55'1620, mlcod=55'1617)", 2024-09-13T11:26:49.730 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:04.449956+0000", 2024-09-13T11:26:49.730 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 225.27856227800001, 2024-09-13T11:26:49.730 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.21143479600000001, 2024-09-13T11:26:49.730 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:26:49.730 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:26:49.730 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:49.730 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:26:49.730 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.730 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:26:49.731 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.449956+0000", 2024-09-13T11:26:49.731 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:49.731 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.731 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.731 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:26:49.731 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.449956+0000", 2024-09-13T11:26:49.731 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:49.731 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.731 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.731 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:26:49.731 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.449959+0000", 2024-09-13T11:26:49.731 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.393e-06 2024-09-13T11:26:49.731 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.731 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.731 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:26:49.732 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.450014+0000", 2024-09-13T11:26:49.732 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.4200000000000003e-05 2024-09-13T11:26:49.732 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.732 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.732 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:26:49.732 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.658637+0000", 2024-09-13T11:26:49.732 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.20862334799999999 2024-09-13T11:26:49.732 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.732 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.732 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:26:49.732 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.658650+0000", 2024-09-13T11:26:49.732 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3148e-05 2024-09-13T11:26:49.732 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.733 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.733 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:26:49.733 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.658867+0000", 2024-09-13T11:26:49.733 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00021673499999999999 2024-09-13T11:26:49.733 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.733 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.733 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:26:49.733 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.658931+0000", 2024-09-13T11:26:49.733 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.4045000000000001e-05 2024-09-13T11:26:49.733 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.733 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.733 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:26:49.733 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.661317+0000", 2024-09-13T11:26:49.733 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.002385823 2024-09-13T11:26:49.733 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.734 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.734 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:26:49.734 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.661391+0000", 2024-09-13T11:26:49.734 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.4103999999999994e-05 2024-09-13T11:26:49.734 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:49.734 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:26:49.734 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:49.734 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.734 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.734 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPush(3.16 85/70 [PushOp(3:69aac9fd:::benchmark_data_smithi177_31116_object43914:head, version: 66'2837, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:69aac9fd:::benchmark_data_smithi177_31116_object43914:head@66'2837, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:26:49.734 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:42.295995+0000", 2024-09-13T11:26:49.734 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 187.43252295799999, 2024-09-13T11:26:49.734 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.215089431, 2024-09-13T11:26:49.735 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:26:49.735 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:26:49.735 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:26:49.735 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:26:49.735 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.735 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:26:49.735 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.295995+0000", 2024-09-13T11:26:49.735 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:49.735 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.735 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.735 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:26:49.735 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.295995+0000", 2024-09-13T11:26:49.735 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:49.736 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.736 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.736 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:26:49.736 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.295997+0000", 2024-09-13T11:26:49.736 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3170000000000001e-06 2024-09-13T11:26:49.736 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.736 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.736 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:26:49.736 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.296058+0000", 2024-09-13T11:26:49.736 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.1690000000000001e-05 2024-09-13T11:26:49.736 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.736 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.736 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:26:49.737 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.509606+0000", 2024-09-13T11:26:49.737 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.21354748300000001 2024-09-13T11:26:49.737 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.737 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.737 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:26:49.737 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.509638+0000", 2024-09-13T11:26:49.737 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.1644999999999999e-05 2024-09-13T11:26:49.737 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.737 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.737 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:26:49.737 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.509814+0000", 2024-09-13T11:26:49.737 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00017679499999999999 2024-09-13T11:26:49.737 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.737 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.737 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:26:49.738 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.509840+0000", 2024-09-13T11:26:49.738 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.5871e-05 2024-09-13T11:26:49.738 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.738 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.738 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:26:49.738 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.511085+0000", 2024-09-13T11:26:49.738 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00124463 2024-09-13T11:26:49.738 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:49.738 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:26:49.738 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:49.738 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.738 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.738 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPush(3.1d 91/78 [PushOp(3:be6a0ae4:::benchmark_data_smithi177_31116_object25671:head, version: 58'1569, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:be6a0ae4:::benchmark_data_smithi177_31116_object25671:head@58'1569, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:26:49.739 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:48.481971+0000", 2024-09-13T11:26:49.739 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 181.24654771300001, 2024-09-13T11:26:49.739 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.47945454199999998, 2024-09-13T11:26:49.739 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:26:49.739 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:26:49.739 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:26:49.739 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:26:49.739 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.739 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:26:49.739 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.481971+0000", 2024-09-13T11:26:49.739 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:49.739 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.739 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.739 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:26:49.740 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.481971+0000", 2024-09-13T11:26:49.740 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:49.740 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.740 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.740 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:26:49.740 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.481972+0000", 2024-09-13T11:26:49.740 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5179999999999999e-06 2024-09-13T11:26:49.740 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.740 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.740 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:26:49.740 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.482000+0000", 2024-09-13T11:26:49.740 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.8249e-05 2024-09-13T11:26:49.740 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.741 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.741 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:26:49.741 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.959926+0000", 2024-09-13T11:26:49.741 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.47792585100000001 2024-09-13T11:26:49.741 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.741 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.741 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:26:49.741 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.959944+0000", 2024-09-13T11:26:49.741 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.7620000000000001e-05 2024-09-13T11:26:49.741 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.741 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.741 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:26:49.741 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.960059+0000", 2024-09-13T11:26:49.742 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000115188 2024-09-13T11:26:49.742 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.742 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.742 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:26:49.742 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.960074+0000", 2024-09-13T11:26:49.742 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.4738e-05 2024-09-13T11:26:49.742 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.742 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.742 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:26:49.742 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.961425+0000", 2024-09-13T11:26:49.742 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0013513780000000001 2024-09-13T11:26:49.742 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:49.742 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:26:49.742 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:49.743 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.743 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.743 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:71884 3.16 e96/88 3:6ca846b2:::benchmark_data_smithi177_31116_object71883:head v 96'3755, mlcod=96'3753)", 2024-09-13T11:26:49.743 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:54.108299+0000", 2024-09-13T11:26:49.743 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 175.62021953999999, 2024-09-13T11:26:49.743 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.20685140099999999, 2024-09-13T11:26:49.743 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:26:49.743 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:26:49.743 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:49.743 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:26:49.743 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.743 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:26:49.743 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.108299+0000", 2024-09-13T11:26:49.743 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:49.744 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.744 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.744 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:26:49.744 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.108299+0000", 2024-09-13T11:26:49.744 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:49.744 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.744 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.744 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:26:49.744 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.108300+0000", 2024-09-13T11:26:49.744 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.173e-06 2024-09-13T11:26:49.744 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.744 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.744 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:26:49.744 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.108323+0000", 2024-09-13T11:26:49.745 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.2591000000000001e-05 2024-09-13T11:26:49.745 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.745 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.745 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:26:49.745 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.312377+0000", 2024-09-13T11:26:49.745 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.20405456599999999 2024-09-13T11:26:49.745 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.745 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.745 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:26:49.745 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.312382+0000", 2024-09-13T11:26:49.745 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.1429999999999997e-06 2024-09-13T11:26:49.745 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.746 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.746 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:26:49.746 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.312480+0000", 2024-09-13T11:26:49.746 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.7742e-05 2024-09-13T11:26:49.746 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.746 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.746 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:26:49.746 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.312521+0000", 2024-09-13T11:26:49.746 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.1369000000000003e-05 2024-09-13T11:26:49.746 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.746 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.746 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:26:49.746 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.315105+0000", 2024-09-13T11:26:49.746 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0025835060000000002 2024-09-13T11:26:49.747 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.747 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.747 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:26:49.747 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.315150+0000", 2024-09-13T11:26:49.747 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.5311e-05 2024-09-13T11:26:49.747 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:49.747 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:26:49.747 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:49.747 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.747 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.747 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:71927 3.0 e97/75 3:0197244a:::benchmark_data_smithi177_31116_object71926:head v 97'3351, mlcod=96'3348)", 2024-09-13T11:26:49.747 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:54.200161+0000", 2024-09-13T11:26:49.747 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 175.528357312, 2024-09-13T11:26:49.747 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.26691738999999998, 2024-09-13T11:26:49.748 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:26:49.748 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:26:49.748 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:49.748 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:26:49.748 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.748 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:26:49.748 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.200161+0000", 2024-09-13T11:26:49.748 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:49.748 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.748 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.748 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:26:49.748 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.200161+0000", 2024-09-13T11:26:49.748 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:49.748 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.749 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.749 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:26:49.749 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.200162+0000", 2024-09-13T11:26:49.749 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.78e-07 2024-09-13T11:26:49.749 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.749 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.749 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:26:49.749 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.200200+0000", 2024-09-13T11:26:49.749 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.8121000000000003e-05 2024-09-13T11:26:49.749 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.749 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.749 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:26:49.750 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.465397+0000", 2024-09-13T11:26:49.750 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.26519681099999998 2024-09-13T11:26:49.750 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.750 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.750 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:26:49.750 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.465404+0000", 2024-09-13T11:26:49.750 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.7399999999999998e-06 2024-09-13T11:26:49.750 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.750 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.750 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:26:49.750 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.465456+0000", 2024-09-13T11:26:49.750 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.2626000000000001e-05 2024-09-13T11:26:49.750 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.750 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.750 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:26:49.751 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.465482+0000", 2024-09-13T11:26:49.751 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.6086000000000001e-05 2024-09-13T11:26:49.751 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.751 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.751 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:26:49.751 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.467035+0000", 2024-09-13T11:26:49.751 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0015523729999999999 2024-09-13T11:26:49.751 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.751 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.751 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:26:49.751 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.467078+0000", 2024-09-13T11:26:49.751 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.3754999999999997e-05 2024-09-13T11:26:49.751 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:49.752 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:26:49.752 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:49.752 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.752 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.752 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPush(3.1b 102/78 [PushOp(3:df786fc7:::benchmark_data_smithi177_31116_object48840:head, version: 71'3079, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:df786fc7:::benchmark_data_smithi177_31116_object48840:head@71'3079, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:26:49.752 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:00.290536+0000", 2024-09-13T11:26:49.752 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 169.43798254999999, 2024-09-13T11:26:49.752 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.20017585399999999, 2024-09-13T11:26:49.752 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:26:49.752 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:26:49.752 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:26:49.752 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:26:49.752 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.752 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:26:49.753 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.290536+0000", 2024-09-13T11:26:49.753 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:49.753 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.753 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.753 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:26:49.753 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.290536+0000", 2024-09-13T11:26:49.753 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:49.753 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.753 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.753 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:26:49.753 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.290537+0000", 2024-09-13T11:26:49.753 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.327e-06 2024-09-13T11:26:49.754 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.754 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.754 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:26:49.754 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.290581+0000", 2024-09-13T11:26:49.754 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.3893999999999997e-05 2024-09-13T11:26:49.754 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.754 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.754 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:26:49.754 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.488588+0000", 2024-09-13T11:26:49.754 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.198006708 2024-09-13T11:26:49.754 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.754 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.754 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:26:49.754 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.488607+0000", 2024-09-13T11:26:49.755 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.9162999999999999e-05 2024-09-13T11:26:49.755 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.755 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.755 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:26:49.755 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.488747+0000", 2024-09-13T11:26:49.755 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00014040799999999999 2024-09-13T11:26:49.755 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.755 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.755 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:26:49.755 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.488770+0000", 2024-09-13T11:26:49.755 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.279e-05 2024-09-13T11:26:49.755 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.755 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.756 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:26:49.756 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.490712+0000", 2024-09-13T11:26:49.756 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001941564 2024-09-13T11:26:49.756 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:49.756 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:26:49.756 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:49.756 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.756 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.756 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPush(3.1b 102/78 [PushOp(3:d82bf6ec:::benchmark_data_smithi177_31116_object48847:head, version: 71'3080, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:d82bf6ec:::benchmark_data_smithi177_31116_object48847:head@71'3080, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:26:49.756 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:00.293979+0000", 2024-09-13T11:26:49.756 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 169.43453902300001, 2024-09-13T11:26:49.756 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.20314832099999999, 2024-09-13T11:26:49.756 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:26:49.756 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:26:49.757 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:26:49.757 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:26:49.757 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.757 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:26:49.757 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.293979+0000", 2024-09-13T11:26:49.757 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:49.757 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.757 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.757 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:26:49.757 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.293979+0000", 2024-09-13T11:26:49.757 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:49.757 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.757 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.757 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:26:49.758 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.293981+0000", 2024-09-13T11:26:49.758 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.4449999999999999e-06 2024-09-13T11:26:49.758 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.758 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.758 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:26:49.758 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.294085+0000", 2024-09-13T11:26:49.758 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000103848 2024-09-13T11:26:49.758 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.758 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.758 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:26:49.758 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.495571+0000", 2024-09-13T11:26:49.758 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.20148640800000001 2024-09-13T11:26:49.758 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.759 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.759 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:26:49.759 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.495609+0000", 2024-09-13T11:26:49.759 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.7904000000000002e-05 2024-09-13T11:26:49.759 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.759 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.759 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:26:49.759 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.495729+0000", 2024-09-13T11:26:49.759 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000120556 2024-09-13T11:26:49.759 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.759 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.759 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:26:49.759 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.495751+0000", 2024-09-13T11:26:49.759 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.1511000000000001e-05 2024-09-13T11:26:49.760 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.760 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.760 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:26:49.760 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.497128+0000", 2024-09-13T11:26:49.760 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001376649 2024-09-13T11:26:49.760 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:49.760 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:26:49.760 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:49.760 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.760 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.760 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:79801 3.5 3:a6efdf84:::benchmark_data_smithi177_31116_object79800:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102)", 2024-09-13T11:26:49.760 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:00.403201+0000", 2024-09-13T11:26:49.760 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 169.32531743600001, 2024-09-13T11:26:49.760 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.31868626300000003, 2024-09-13T11:26:49.761 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:26:49.761 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:26:49.761 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:49.761 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:26:49.761 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:26:49.761 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:49.761 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 79801 2024-09-13T11:26:49.761 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.761 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:26:49.761 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.761 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:26:49.761 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.403201+0000", 2024-09-13T11:26:49.761 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:49.762 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.762 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.762 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:26:49.762 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.403201+0000", 2024-09-13T11:26:49.762 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:49.762 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.762 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.762 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:26:49.762 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.403204+0000", 2024-09-13T11:26:49.762 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.0819999999999999e-06 2024-09-13T11:26:49.762 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.762 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.762 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:26:49.762 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.403260+0000", 2024-09-13T11:26:49.763 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.5624999999999997e-05 2024-09-13T11:26:49.763 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.763 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.763 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:26:49.763 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.403280+0000", 2024-09-13T11:26:49.763 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.0060000000000001e-05 2024-09-13T11:26:49.763 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.763 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.763 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:26:49.763 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.403291+0000", 2024-09-13T11:26:49.763 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.1335000000000001e-05 2024-09-13T11:26:49.763 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.763 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.763 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:26:49.764 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.403344+0000", 2024-09-13T11:26:49.764 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.3217000000000002e-05 2024-09-13T11:26:49.764 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.764 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.764 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:26:49.764 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.403456+0000", 2024-09-13T11:26:49.764 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000111706 2024-09-13T11:26:49.764 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.764 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.764 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 13", 2024-09-13T11:26:49.764 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.403547+0000", 2024-09-13T11:26:49.764 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.1556000000000002e-05 2024-09-13T11:26:49.764 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.764 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.765 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:26:49.765 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.407436+0000", 2024-09-13T11:26:49.765 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.003888856 2024-09-13T11:26:49.765 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.765 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.765 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:49.765 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.721722+0000", 2024-09-13T11:26:49.765 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.31428562700000001 2024-09-13T11:26:49.765 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.765 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.765 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:26:49.765 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.721814+0000", 2024-09-13T11:26:49.765 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.1564000000000006e-05 2024-09-13T11:26:49.765 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.766 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.766 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:26:49.766 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.721887+0000", 2024-09-13T11:26:49.766 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.3634999999999995e-05 2024-09-13T11:26:49.766 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:49.766 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:26:49.766 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:49.766 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.766 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.766 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:83199 3.1d e106/102 3:be2c6fae:::benchmark_data_smithi177_31116_object83198:head v 106'4244, mlcod=106'4242)", 2024-09-13T11:26:49.766 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:04.298299+0000", 2024-09-13T11:26:49.766 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 165.43021930899999, 2024-09-13T11:26:49.766 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.25001107099999997, 2024-09-13T11:26:49.766 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:26:49.767 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:26:49.767 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:49.767 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:26:49.767 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.767 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:26:49.767 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.298299+0000", 2024-09-13T11:26:49.767 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:49.767 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.767 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.767 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:26:49.767 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.298299+0000", 2024-09-13T11:26:49.767 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:49.767 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.767 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.768 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:26:49.768 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.298301+0000", 2024-09-13T11:26:49.768 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5859999999999999e-06 2024-09-13T11:26:49.768 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.768 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.768 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:26:49.768 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.298368+0000", 2024-09-13T11:26:49.768 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.6928999999999996e-05 2024-09-13T11:26:49.768 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.768 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.768 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:26:49.768 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.545180+0000", 2024-09-13T11:26:49.768 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.24681209700000001 2024-09-13T11:26:49.768 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.769 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.769 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:26:49.769 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.545193+0000", 2024-09-13T11:26:49.769 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3427e-05 2024-09-13T11:26:49.769 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.769 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.769 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:26:49.769 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.545327+0000", 2024-09-13T11:26:49.769 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00013359400000000001 2024-09-13T11:26:49.769 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.769 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.769 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:26:49.770 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.545385+0000", 2024-09-13T11:26:49.770 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.8492000000000002e-05 2024-09-13T11:26:49.770 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.770 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.770 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:26:49.770 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.548225+0000", 2024-09-13T11:26:49.770 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.002839472 2024-09-13T11:26:49.770 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.770 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.770 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:26:49.770 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.548310+0000", 2024-09-13T11:26:49.770 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.5474000000000002e-05 2024-09-13T11:26:49.770 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:49.770 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:26:49.771 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:49.771 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.771 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.771 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:137062 3.5 3:a16e178f:::benchmark_data_smithi177_31116_object137061:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e139)", 2024-09-13T11:26:49.771 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:53.140525+0000", 2024-09-13T11:26:49.771 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 116.587993716, 2024-09-13T11:26:49.771 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.20697954599999999, 2024-09-13T11:26:49.771 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:26:49.771 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:26:49.771 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:49.771 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:26:49.771 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:26:49.771 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:49.771 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 137062 2024-09-13T11:26:49.772 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.772 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:26:49.772 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.772 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:26:49.772 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:53.140525+0000", 2024-09-13T11:26:49.772 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:49.772 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.772 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.772 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:26:49.772 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:53.140525+0000", 2024-09-13T11:26:49.772 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:49.772 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.772 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.772 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:26:49.772 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:53.140527+0000", 2024-09-13T11:26:49.773 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.1739999999999999e-06 2024-09-13T11:26:49.773 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.773 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.773 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:26:49.773 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:53.140553+0000", 2024-09-13T11:26:49.773 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.6631e-05 2024-09-13T11:26:49.773 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.773 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.773 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:26:49.773 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:53.140566+0000", 2024-09-13T11:26:49.773 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.2268e-05 2024-09-13T11:26:49.773 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.773 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.774 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:26:49.774 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:53.140574+0000", 2024-09-13T11:26:49.774 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.5669999999999995e-06 2024-09-13T11:26:49.774 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.774 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.774 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:26:49.774 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:53.140622+0000", 2024-09-13T11:26:49.774 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.8143999999999998e-05 2024-09-13T11:26:49.774 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.774 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.774 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:26:49.774 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:53.140759+0000", 2024-09-13T11:26:49.774 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000136499 2024-09-13T11:26:49.774 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.774 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.775 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 13", 2024-09-13T11:26:49.775 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:53.140887+0000", 2024-09-13T11:26:49.775 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00012796300000000001 2024-09-13T11:26:49.775 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.775 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.775 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:26:49.775 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:53.142675+0000", 2024-09-13T11:26:49.775 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001788073 2024-09-13T11:26:49.775 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.775 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.775 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:49.775 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:53.347407+0000", 2024-09-13T11:26:49.775 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.20473206899999999 2024-09-13T11:26:49.775 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.775 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.776 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:26:49.776 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:53.347464+0000", 2024-09-13T11:26:49.776 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.6536000000000002e-05 2024-09-13T11:26:49.776 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.776 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.776 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:26:49.776 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:53.347504+0000", 2024-09-13T11:26:49.776 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.0621999999999998e-05 2024-09-13T11:26:49.776 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:49.776 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:26:49.776 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:49.776 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.776 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.776 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:191058 3.1d e175/102 3:bd6e7029:::benchmark_data_smithi177_31116_object50755:head v 175'7612, mlcod=175'7609)", 2024-09-13T11:26:49.777 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:25:38.513527+0000", 2024-09-13T11:26:49.777 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 71.214991058999999, 2024-09-13T11:26:49.777 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.386104477, 2024-09-13T11:26:49.777 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:26:49.777 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:26:49.777 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:49.777 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:26:49.777 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.777 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:26:49.777 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:38.513527+0000", 2024-09-13T11:26:49.777 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:49.777 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.777 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.777 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:26:49.778 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:38.513527+0000", 2024-09-13T11:26:49.778 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:49.778 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.778 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.778 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:26:49.778 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:38.513530+0000", 2024-09-13T11:26:49.778 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.0029999999999999e-06 2024-09-13T11:26:49.778 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.778 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.778 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:26:49.778 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:38.513542+0000", 2024-09-13T11:26:49.778 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.1477e-05 2024-09-13T11:26:49.778 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.778 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.779 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:26:49.779 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:38.891617+0000", 2024-09-13T11:26:49.779 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.378075045 2024-09-13T11:26:49.779 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.779 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.779 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:26:49.779 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:38.891635+0000", 2024-09-13T11:26:49.779 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.7697e-05 2024-09-13T11:26:49.779 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.779 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.779 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:26:49.779 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:38.891775+0000", 2024-09-13T11:26:49.779 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000140665 2024-09-13T11:26:49.779 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.780 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.780 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:26:49.780 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:38.891833+0000", 2024-09-13T11:26:49.780 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.7769999999999997e-05 2024-09-13T11:26:49.780 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.780 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.780 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:26:49.780 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:38.899567+0000", 2024-09-13T11:26:49.780 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0077339970000000003 2024-09-13T11:26:49.780 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.780 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.780 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:26:49.780 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:38.899632+0000", 2024-09-13T11:26:49.780 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.4822999999999999e-05 2024-09-13T11:26:49.780 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:49.781 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:26:49.781 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:49.781 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.781 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.781 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:197236 3.5 3:a3127298:::benchmark_data_smithi177_31116_object56933:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e178)", 2024-09-13T11:26:49.781 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:25:44.300667+0000", 2024-09-13T11:26:49.781 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 65.427850968000001, 2024-09-13T11:26:49.781 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.37481205200000001, 2024-09-13T11:26:49.781 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:26:49.781 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:26:49.781 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:49.781 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:26:49.781 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:26:49.781 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:49.781 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 197236 2024-09-13T11:26:49.782 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.782 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:26:49.782 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.782 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:26:49.782 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:44.300667+0000", 2024-09-13T11:26:49.782 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:49.782 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.782 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.782 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:26:49.782 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:44.300667+0000", 2024-09-13T11:26:49.782 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:49.782 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.782 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.782 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:26:49.783 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:44.300670+0000", 2024-09-13T11:26:49.783 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.7259999999999998e-06 2024-09-13T11:26:49.783 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.783 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.783 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:26:49.783 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:44.300677+0000", 2024-09-13T11:26:49.783 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.1550000000000001e-06 2024-09-13T11:26:49.783 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.783 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.783 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:26:49.783 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:44.300695+0000", 2024-09-13T11:26:49.783 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.7552999999999999e-05 2024-09-13T11:26:49.783 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.784 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.784 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:26:49.784 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:44.300704+0000", 2024-09-13T11:26:49.784 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.5710000000000004e-06 2024-09-13T11:26:49.784 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.784 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.784 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:26:49.784 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:44.300762+0000", 2024-09-13T11:26:49.784 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.7351e-05 2024-09-13T11:26:49.784 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.784 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.784 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:26:49.784 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:44.300888+0000", 2024-09-13T11:26:49.785 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00012617999999999999 2024-09-13T11:26:49.785 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.785 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.785 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 13", 2024-09-13T11:26:49.785 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:44.300993+0000", 2024-09-13T11:26:49.785 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00010505600000000001 2024-09-13T11:26:49.785 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.785 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.785 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:26:49.785 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:44.302936+0000", 2024-09-13T11:26:49.785 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0019428749999999999 2024-09-13T11:26:49.785 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.785 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.786 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:49.786 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:44.675352+0000", 2024-09-13T11:26:49.786 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.37241603400000001 2024-09-13T11:26:49.786 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.786 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.786 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:26:49.786 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:44.675412+0000", 2024-09-13T11:26:49.786 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.9679999999999998e-05 2024-09-13T11:26:49.786 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.786 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.786 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:26:49.786 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:44.675479+0000", 2024-09-13T11:26:49.786 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.7871000000000001e-05 2024-09-13T11:26:49.786 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:49.786 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:26:49.787 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:49.787 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.787 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.787 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop_reply(client.4665.0:197236 3.5 e178/117 ondisk, result = 0)", 2024-09-13T11:26:49.787 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:25:44.304480+0000", 2024-09-13T11:26:49.787 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 65.424038701000001, 2024-09-13T11:26:49.787 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.37100864099999997, 2024-09-13T11:26:49.787 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:26:49.787 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:26:49.787 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:26:49.787 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:26:49.787 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.788 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:26:49.788 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:44.304480+0000", 2024-09-13T11:26:49.788 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:49.788 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.788 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.788 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:26:49.788 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:44.304480+0000", 2024-09-13T11:26:49.788 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:49.788 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.788 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.788 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:26:49.789 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:44.304481+0000", 2024-09-13T11:26:49.789 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.7349999999999999e-06 2024-09-13T11:26:49.789 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.789 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.789 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:26:49.789 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:44.304492+0000", 2024-09-13T11:26:49.789 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0394000000000001e-05 2024-09-13T11:26:49.789 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.789 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.789 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:26:49.789 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:44.675037+0000", 2024-09-13T11:26:49.789 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.37054519400000002 2024-09-13T11:26:49.789 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.790 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.790 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:26:49.790 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:44.675045+0000", 2024-09-13T11:26:49.790 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.1370000000000002e-06 2024-09-13T11:26:49.790 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.790 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.790 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:26:49.790 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:44.675334+0000", 2024-09-13T11:26:49.790 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00028848699999999998 2024-09-13T11:26:49.790 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.790 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.790 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:26:49.790 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:44.675345+0000", 2024-09-13T11:26:49.790 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.1198e-05 2024-09-13T11:26:49.791 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.791 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.791 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:26:49.791 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:44.675488+0000", 2024-09-13T11:26:49.791 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00014349600000000001 2024-09-13T11:26:49.791 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:49.791 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:26:49.791 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:49.791 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.791 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.791 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:217032 3.16 e187/88 3:6d28a27c:::benchmark_data_smithi177_31116_object76729:head v 187'8398, mlcod=187'8395)", 2024-09-13T11:26:49.791 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:25:59.836895+0000", 2024-09-13T11:26:49.791 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 49.891623815999999, 2024-09-13T11:26:49.791 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.22574986, 2024-09-13T11:26:49.792 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:26:49.792 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:26:49.792 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:49.792 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:26:49.792 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.792 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:26:49.792 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:59.836895+0000", 2024-09-13T11:26:49.792 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:49.792 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.792 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.792 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:26:49.792 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:59.836895+0000", 2024-09-13T11:26:49.792 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:49.792 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.792 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.793 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:26:49.793 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:59.836896+0000", 2024-09-13T11:26:49.793 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.432e-06 2024-09-13T11:26:49.793 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.793 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.793 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:26:49.793 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:59.836904+0000", 2024-09-13T11:26:49.793 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.9149999999999996e-06 2024-09-13T11:26:49.793 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.793 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.793 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:26:49.793 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:00.053571+0000", 2024-09-13T11:26:49.793 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.21666679799999999 2024-09-13T11:26:49.793 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.794 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.794 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:26:49.794 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:00.053582+0000", 2024-09-13T11:26:49.794 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0927999999999999e-05 2024-09-13T11:26:49.794 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.794 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.794 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:26:49.794 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:00.053811+0000", 2024-09-13T11:26:49.794 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00022964700000000001 2024-09-13T11:26:49.794 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.794 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.794 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:26:49.794 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:00.053864+0000", 2024-09-13T11:26:49.794 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.2710000000000002e-05 2024-09-13T11:26:49.794 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.795 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.795 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:26:49.795 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:00.062547+0000", 2024-09-13T11:26:49.795 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0086832650000000008 2024-09-13T11:26:49.795 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.795 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.795 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:26:49.795 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:00.062644+0000", 2024-09-13T11:26:49.795 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.7164999999999996e-05 2024-09-13T11:26:49.795 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:49.795 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:26:49.795 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:49.795 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.795 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.796 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:234789 3.5 3:a0df9e27:::benchmark_data_smithi177_31116_object94486:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e201)", 2024-09-13T11:26:49.796 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:26:17.919815+0000", 2024-09-13T11:26:49.796 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 31.808703671, 2024-09-13T11:26:49.796 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.34227359400000001, 2024-09-13T11:26:49.796 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:26:49.796 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:26:49.796 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:49.796 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:26:49.796 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:26:49.796 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:49.796 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 234789 2024-09-13T11:26:49.796 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.796 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:26:49.796 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.797 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:26:49.797 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:17.919815+0000", 2024-09-13T11:26:49.797 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:49.797 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.797 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.797 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:26:49.797 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:17.919815+0000", 2024-09-13T11:26:49.797 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:49.797 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.797 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.797 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:26:49.797 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:17.919817+0000", 2024-09-13T11:26:49.798 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.6369999999999999e-06 2024-09-13T11:26:49.798 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.798 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.798 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:26:49.798 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:17.919824+0000", 2024-09-13T11:26:49.798 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.9290000000000001e-06 2024-09-13T11:26:49.798 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.798 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.798 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:26:49.798 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:17.919839+0000", 2024-09-13T11:26:49.798 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.4763000000000001e-05 2024-09-13T11:26:49.798 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.798 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.799 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:26:49.799 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:17.919846+0000", 2024-09-13T11:26:49.799 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.1829999999999997e-06 2024-09-13T11:26:49.799 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.799 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.799 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:26:49.799 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:17.919894+0000", 2024-09-13T11:26:49.799 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.7475999999999997e-05 2024-09-13T11:26:49.799 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.799 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.799 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:26:49.799 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:17.919996+0000", 2024-09-13T11:26:49.799 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00010262199999999999 2024-09-13T11:26:49.799 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.800 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.800 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 13", 2024-09-13T11:26:49.800 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:17.920092+0000", 2024-09-13T11:26:49.800 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.5546999999999998e-05 2024-09-13T11:26:49.800 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.800 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.800 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:26:49.800 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:17.921865+0000", 2024-09-13T11:26:49.800 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00177271 2024-09-13T11:26:49.800 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.800 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.800 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:49.800 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:18.261907+0000", 2024-09-13T11:26:49.800 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.34004223300000003 2024-09-13T11:26:49.801 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.801 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.801 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:26:49.801 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:18.262002+0000", 2024-09-13T11:26:49.801 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.4745000000000002e-05 2024-09-13T11:26:49.801 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.801 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.801 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:26:49.801 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:18.262088+0000", 2024-09-13T11:26:49.801 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.6749000000000006e-05 2024-09-13T11:26:49.801 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:49.801 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:26:49.801 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:49.801 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.802 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.802 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop_reply(client.4665.0:234789 3.5 e201/117 ondisk, result = 0)", 2024-09-13T11:26:49.802 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:26:17.922332+0000", 2024-09-13T11:26:49.802 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 31.806186332999999, 2024-09-13T11:26:49.802 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.33978349299999999, 2024-09-13T11:26:49.802 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:26:49.802 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:26:49.802 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:26:49.802 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:26:49.802 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.802 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:26:49.802 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:17.922332+0000", 2024-09-13T11:26:49.802 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:49.803 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.803 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.803 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:26:49.803 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:17.922332+0000", 2024-09-13T11:26:49.803 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:49.803 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.803 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.803 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:26:49.803 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:17.922333+0000", 2024-09-13T11:26:49.803 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.2929999999999999e-06 2024-09-13T11:26:49.803 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.803 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.803 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:26:49.803 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:17.922337+0000", 2024-09-13T11:26:49.804 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.7469999999999999e-06 2024-09-13T11:26:49.804 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.804 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.804 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:26:49.804 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:18.261064+0000", 2024-09-13T11:26:49.804 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.33872685499999999 2024-09-13T11:26:49.804 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.804 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.804 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:26:49.804 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:18.261084+0000", 2024-09-13T11:26:49.804 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.9893000000000001e-05 2024-09-13T11:26:49.804 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.805 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.805 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:26:49.805 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:18.261880+0000", 2024-09-13T11:26:49.805 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00079579399999999995 2024-09-13T11:26:49.805 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.805 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.805 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:26:49.805 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:18.261897+0000", 2024-09-13T11:26:49.805 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.7730000000000001e-05 2024-09-13T11:26:49.805 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.805 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.805 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:26:49.805 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:18.262116+0000", 2024-09-13T11:26:49.806 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00021818099999999999 2024-09-13T11:26:49.806 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:49.806 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:26:49.806 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:49.806 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.806 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.806 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:260347 3.16 e219/88 3:6d3745c1:::benchmark_data_smithi177_31116_object120044:head v 219'9755, mlcod=219'9752)", 2024-09-13T11:26:49.806 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:26:42.471118+0000", 2024-09-13T11:26:49.806 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 7.2574007390000004, 2024-09-13T11:26:49.806 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.43076131200000001, 2024-09-13T11:26:49.806 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:26:49.806 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:26:49.806 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:49.807 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:26:49.807 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.807 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:26:49.807 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:42.471118+0000", 2024-09-13T11:26:49.807 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:49.807 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.807 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.807 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:26:49.807 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:42.471118+0000", 2024-09-13T11:26:49.807 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:49.807 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.807 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.807 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:26:49.807 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:42.471122+0000", 2024-09-13T11:26:49.807 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.233e-06 2024-09-13T11:26:49.808 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.808 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.808 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:26:49.808 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:42.471139+0000", 2024-09-13T11:26:49.808 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.7442999999999999e-05 2024-09-13T11:26:49.808 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.808 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.808 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:26:49.808 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:42.893124+0000", 2024-09-13T11:26:49.808 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.42198470300000002 2024-09-13T11:26:49.808 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.808 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.808 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:26:49.809 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:42.893137+0000", 2024-09-13T11:26:49.809 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3233e-05 2024-09-13T11:26:49.809 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.809 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.809 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:26:49.809 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:42.893272+0000", 2024-09-13T11:26:49.809 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00013433499999999999 2024-09-13T11:26:49.809 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.809 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.809 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:26:49.809 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:42.893335+0000", 2024-09-13T11:26:49.809 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.3373000000000005e-05 2024-09-13T11:26:49.809 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.809 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.810 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:26:49.810 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:42.901776+0000", 2024-09-13T11:26:49.810 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0084411899999999995 2024-09-13T11:26:49.810 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.810 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.810 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:26:49.810 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:42.901879+0000", 2024-09-13T11:26:49.810 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000102802 2024-09-13T11:26:49.810 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:49.810 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:26:49.810 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:49.810 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.810 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.810 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:266456 3.5 e222/205 3:a058f943:::benchmark_data_smithi177_31116_object126153:head v 222'7143, mlcod=222'7139)", 2024-09-13T11:26:49.811 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:26:47.702034+0000", 2024-09-13T11:26:49.811 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 2.0264838840000001, 2024-09-13T11:26:49.811 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.20573660499999999, 2024-09-13T11:26:49.811 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:26:49.811 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:26:49.811 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:49.811 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:26:49.811 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.811 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:26:49.811 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:47.702034+0000", 2024-09-13T11:26:49.811 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:49.811 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.811 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.811 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:26:49.812 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:47.702034+0000", 2024-09-13T11:26:49.812 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:49.812 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.812 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.812 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:26:49.812 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:47.702036+0000", 2024-09-13T11:26:49.812 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.265e-06 2024-09-13T11:26:49.812 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.812 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.812 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:26:49.812 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:47.702043+0000", 2024-09-13T11:26:49.812 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.9820000000000002e-06 2024-09-13T11:26:49.812 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.812 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.813 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:26:49.813 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:47.905566+0000", 2024-09-13T11:26:49.813 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.20352285000000001 2024-09-13T11:26:49.813 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.813 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.813 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:26:49.813 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:47.905575+0000", 2024-09-13T11:26:49.813 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.5570000000000001e-06 2024-09-13T11:26:49.813 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.813 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.813 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:26:49.813 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:47.905657+0000", 2024-09-13T11:26:49.813 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.2088000000000004e-05 2024-09-13T11:26:49.813 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.814 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.814 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:26:49.814 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:47.905676+0000", 2024-09-13T11:26:49.814 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.8561999999999999e-05 2024-09-13T11:26:49.814 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.814 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.814 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:26:49.814 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:47.907724+0000", 2024-09-13T11:26:49.814 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0020485239999999999 2024-09-13T11:26:49.814 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.814 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.814 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:26:49.814 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:47.907771+0000", 2024-09-13T11:26:49.814 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.6777000000000003e-05 2024-09-13T11:26:49.814 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:49.815 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:26:49.815 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:49.815 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.815 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.815 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:266464 3.5 e222/205 3:a2b966a1:::benchmark_data_smithi177_31116_object126161:head v 222'7144, mlcod=222'7140)", 2024-09-13T11:26:49.815 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:26:47.709996+0000", 2024-09-13T11:26:49.815 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 2.018522795, 2024-09-13T11:26:49.815 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.209382814, 2024-09-13T11:26:49.815 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:26:49.815 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:26:49.815 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:49.815 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:26:49.815 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.816 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:26:49.816 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:47.709996+0000", 2024-09-13T11:26:49.816 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:49.816 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.816 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.816 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:26:49.816 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:47.709996+0000", 2024-09-13T11:26:49.816 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:49.816 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.816 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.816 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:26:49.816 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:47.709997+0000", 2024-09-13T11:26:49.816 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.652e-06 2024-09-13T11:26:49.816 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.817 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.817 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:26:49.817 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:47.710006+0000", 2024-09-13T11:26:49.817 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.3480000000000004e-06 2024-09-13T11:26:49.817 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.817 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.817 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:26:49.817 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:47.917127+0000", 2024-09-13T11:26:49.817 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.20712151400000001 2024-09-13T11:26:49.817 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.817 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.817 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:26:49.817 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:47.917136+0000", 2024-09-13T11:26:49.817 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.3500000000000003e-06 2024-09-13T11:26:49.818 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.818 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.818 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:26:49.818 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:47.917204+0000", 2024-09-13T11:26:49.818 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.7793000000000006e-05 2024-09-13T11:26:49.818 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.818 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.818 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:26:49.818 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:47.917230+0000", 2024-09-13T11:26:49.818 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.5721e-05 2024-09-13T11:26:49.818 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.818 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.818 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:26:49.818 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:47.919340+0000", 2024-09-13T11:26:49.819 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0021102550000000001 2024-09-13T11:26:49.819 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.819 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.819 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:26:49.819 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:47.919378+0000", 2024-09-13T11:26:49.819 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.8180999999999999e-05 2024-09-13T11:26:49.819 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:49.819 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:26:49.819 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:49.819 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.819 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.819 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:266535 3.5 e222/205 3:a07a94a7:::benchmark_data_smithi177_31116_object126232:head v 222'7145, mlcod=222'7141)", 2024-09-13T11:26:49.819 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:26:47.759387+0000", 2024-09-13T11:26:49.819 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 1.969131578, 2024-09-13T11:26:49.820 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.23350773499999999, 2024-09-13T11:26:49.820 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:26:49.820 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:26:49.820 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:49.820 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:26:49.820 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.820 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:26:49.820 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:47.759387+0000", 2024-09-13T11:26:49.820 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:49.820 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.820 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.820 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:26:49.820 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:47.759387+0000", 2024-09-13T11:26:49.821 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:26:49.821 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.821 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.821 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:26:49.821 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:47.759389+0000", 2024-09-13T11:26:49.821 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.069e-06 2024-09-13T11:26:49.821 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.821 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.821 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:26:49.821 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:47.759397+0000", 2024-09-13T11:26:49.821 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.7780000000000007e-06 2024-09-13T11:26:49.821 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.821 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.821 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:26:49.822 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:47.990513+0000", 2024-09-13T11:26:49.822 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.23111675800000001 2024-09-13T11:26:49.822 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.822 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.822 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:26:49.822 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:47.990523+0000", 2024-09-13T11:26:49.822 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.3600000000000002e-06 2024-09-13T11:26:49.822 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.822 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.822 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:26:49.822 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:47.990649+0000", 2024-09-13T11:26:49.822 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00012615700000000001 2024-09-13T11:26:49.822 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.822 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.823 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:26:49.823 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:47.990671+0000", 2024-09-13T11:26:49.823 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.1934999999999999e-05 2024-09-13T11:26:49.823 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.823 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.823 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:26:49.823 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:47.992852+0000", 2024-09-13T11:26:49.823 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0021815060000000002 2024-09-13T11:26:49.823 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:26:49.823 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:26:49.823 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:26:49.823 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:47.992895+0000", 2024-09-13T11:26:49.823 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.2172000000000002e-05 2024-09-13T11:26:49.823 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:49.823 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:26:49.824 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:49.824 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:26:49.824 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:26:49.824 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-09-13T11:26:49.825 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-13T11:26:49.826 INFO:tasks.ceph.osd.1.smithi092.stderr:2024-09-13T11:26:49.823+0000 7fe7789c5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33718) UID: 0 2024-09-13T11:26:49.867 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T11:26:49.926 INFO:tasks.ceph.osd.5.smithi092.stderr:2024-09-13T11:26:49.923+0000 7f6f0b705640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26346) UID: 0 2024-09-13T11:26:49.955 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-09-13T11:26:49.955 INFO:teuthology.orchestra.run.smithi092.stdout: "ops": [], 2024-09-13T11:26:49.955 INFO:teuthology.orchestra.run.smithi092.stdout: "num_ops": 0 2024-09-13T11:26:49.956 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-09-13T11:26:49.970 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-13T11:26:49.971 INFO:teuthology.orchestra.run.smithi001.stderr:noscrub is set 2024-09-13T11:26:49.973 INFO:tasks.ceph.osd.5.smithi092.stderr:2024-09-13T11:26:49.971+0000 7f6f02136640 -1 osd.5 224 handle_osd_map injecting map crc failure 2024-09-13T11:26:50.027 INFO:tasks.ceph.osd.2.smithi162.stderr:2024-09-13T11:26:50.023+0000 7fe92b35a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26313) UID: 0 2024-09-13T11:26:50.084 INFO:teuthology.orchestra.run.smithi001.stdout:osd.0: {} 2024-09-13T11:26:50.084 INFO:teuthology.orchestra.run.smithi001.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T11:26:50.123 INFO:teuthology.orchestra.run.smithi001.stdout:osd.1: {} 2024-09-13T11:26:50.123 INFO:teuthology.orchestra.run.smithi001.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T11:26:50.123 INFO:teuthology.orchestra.run.smithi092.stdout:ERROR: (22) Invalid argument 2024-09-13T11:26:50.123 INFO:teuthology.orchestra.run.smithi092.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:26:50.127 INFO:tasks.ceph.osd.3.smithi177.stderr:2024-09-13T11:26:50.125+0000 7f10adba4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26574) UID: 0 2024-09-13T11:26:50.140 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-13T11:26:50.153 INFO:teuthology.orchestra.run.smithi001.stdout:osd.2: {} 2024-09-13T11:26:50.154 INFO:teuthology.orchestra.run.smithi001.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T11:26:50.183 INFO:teuthology.orchestra.run.smithi001.stdout:osd.3: {} 2024-09-13T11:26:50.183 INFO:teuthology.orchestra.run.smithi001.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T11:26:50.226 INFO:teuthology.orchestra.run.smithi001.stdout:osd.4: {} 2024-09-13T11:26:50.226 INFO:teuthology.orchestra.run.smithi001.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T11:26:50.228 INFO:tasks.ceph.osd.14.smithi162.stderr:2024-09-13T11:26:50.227+0000 7f3df5903640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33154) UID: 0 2024-09-13T11:26:50.257 INFO:teuthology.orchestra.run.smithi001.stdout:osd.5: {} 2024-09-13T11:26:50.257 INFO:teuthology.orchestra.run.smithi001.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T11:26:50.273 INFO:teuthology.orchestra.run.smithi092.stdout:ERROR: (22) Invalid argument 2024-09-13T11:26:50.273 INFO:teuthology.orchestra.run.smithi092.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:26:50.287 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_ops_in_flight 2024-09-13T11:26:50.299 INFO:teuthology.orchestra.run.smithi001.stdout:osd.6: {} 2024-09-13T11:26:50.299 INFO:teuthology.orchestra.run.smithi001.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T11:26:50.329 INFO:tasks.ceph.osd.8.smithi001.stderr:2024-09-13T11:26:50.326+0000 7fb9b48ae640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26312) UID: 0 2024-09-13T11:26:50.330 INFO:teuthology.orchestra.run.smithi001.stdout:osd.7: {} 2024-09-13T11:26:50.330 INFO:teuthology.orchestra.run.smithi001.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T11:26:50.354 INFO:teuthology.orchestra.run.smithi001.stdout:osd.8: {} 2024-09-13T11:26:50.354 INFO:teuthology.orchestra.run.smithi001.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-13T11:26:50.381 INFO:teuthology.orchestra.run.smithi001.stdout:osd.9: {} 2024-09-13T11:26:50.381 INFO:teuthology.orchestra.run.smithi001.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-13T11:26:50.414 INFO:teuthology.orchestra.run.smithi001.stdout:osd.10: {} 2024-09-13T11:26:50.414 INFO:teuthology.orchestra.run.smithi001.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-13T11:26:50.415 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-13T11:26:50.415 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [ 2024-09-13T11:26:50.415 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:50.415 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:266944 3.32 3.9cf99832 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e225)", 2024-09-13T11:26:50.415 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:26:50.068348+0000", 2024-09-13T11:26:50.415 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 0.34733394099999998, 2024-09-13T11:26:50.415 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.34737141700000002, 2024-09-13T11:26:50.415 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:26:50.416 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:26:50.416 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "waiting for peered", 2024-09-13T11:26:50.416 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:26:50.416 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:26:50.416 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:50.416 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 266944 2024-09-13T11:26:50.416 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:50.416 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:26:50.416 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:50.416 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:26:50.416 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:50.068348+0000", 2024-09-13T11:26:50.416 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:50.416 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:50.416 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:50.417 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:26:50.417 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:50.068348+0000", 2024-09-13T11:26:50.417 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:50.417 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:50.417 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:50.417 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:26:50.417 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:50.068350+0000", 2024-09-13T11:26:50.417 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.035e-06 2024-09-13T11:26:50.417 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:50.417 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:50.417 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:26:50.417 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:50.068355+0000", 2024-09-13T11:26:50.417 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.5739999999999997e-06 2024-09-13T11:26:50.418 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:50.418 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:50.418 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:26:50.418 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:50.068365+0000", 2024-09-13T11:26:50.418 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0755e-05 2024-09-13T11:26:50.418 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:50.418 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:50.418 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:26:50.418 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:50.068371+0000", 2024-09-13T11:26:50.418 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.5679999999999999e-06 2024-09-13T11:26:50.418 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:50.418 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:50.418 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:26:50.418 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:50.068448+0000", 2024-09-13T11:26:50.419 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.7202999999999999e-05 2024-09-13T11:26:50.419 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:50.419 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:50.419 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-09-13T11:26:50.419 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:50.068455+0000", 2024-09-13T11:26:50.419 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.409e-06 2024-09-13T11:26:50.419 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:50.419 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:26:50.419 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:50.419 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:50.419 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:50.419 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:267198 3.32 3.1d75b6f2 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e225)", 2024-09-13T11:26:50.419 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:26:50.254351+0000", 2024-09-13T11:26:50.419 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 0.16133113600000001, 2024-09-13T11:26:50.420 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.161501374, 2024-09-13T11:26:50.420 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:26:50.420 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:26:50.420 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "waiting for peered", 2024-09-13T11:26:50.420 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:26:50.420 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:26:50.420 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:50.420 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 267198 2024-09-13T11:26:50.420 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:50.420 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:26:50.420 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:50.420 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:26:50.420 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:50.254351+0000", 2024-09-13T11:26:50.421 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:50.421 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:50.421 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:50.421 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:26:50.421 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:50.254351+0000", 2024-09-13T11:26:50.421 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:50.421 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:50.421 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:50.421 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:26:50.421 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:50.254354+0000", 2024-09-13T11:26:50.421 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.4170000000000001e-06 2024-09-13T11:26:50.421 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:50.421 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:50.422 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:26:50.422 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:50.254364+0000", 2024-09-13T11:26:50.422 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.3929999999999997e-06 2024-09-13T11:26:50.422 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:50.422 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:50.422 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:26:50.422 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:50.254398+0000", 2024-09-13T11:26:50.422 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.4486000000000002e-05 2024-09-13T11:26:50.422 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:50.422 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:50.422 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:26:50.422 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:50.254416+0000", 2024-09-13T11:26:50.422 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.8070000000000001e-05 2024-09-13T11:26:50.423 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:50.423 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:50.423 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:26:50.423 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:50.254559+0000", 2024-09-13T11:26:50.423 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000142261 2024-09-13T11:26:50.423 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:50.423 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:50.423 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-09-13T11:26:50.423 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:50.254582+0000", 2024-09-13T11:26:50.423 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.3394e-05 2024-09-13T11:26:50.423 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:50.423 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:26:50.423 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:50.424 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:50.424 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:50.424 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:267211 3.32 3.28de3332 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e225)", 2024-09-13T11:26:50.424 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:26:50.275463+0000", 2024-09-13T11:26:50.424 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 0.14021866199999999, 2024-09-13T11:26:50.424 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.14048459299999999, 2024-09-13T11:26:50.424 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:26:50.424 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:26:50.424 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "waiting for peered", 2024-09-13T11:26:50.424 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:26:50.424 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:26:50.424 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:50.424 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 267211 2024-09-13T11:26:50.425 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:50.425 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:26:50.425 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:50.425 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:26:50.425 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:50.275463+0000", 2024-09-13T11:26:50.425 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:50.425 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:50.425 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:50.425 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:26:50.425 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:50.275463+0000", 2024-09-13T11:26:50.425 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:50.425 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:50.425 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:50.425 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:26:50.426 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:50.275474+0000", 2024-09-13T11:26:50.426 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0217000000000001e-05 2024-09-13T11:26:50.426 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:50.426 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:50.426 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:26:50.426 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:50.275482+0000", 2024-09-13T11:26:50.426 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.191e-06 2024-09-13T11:26:50.426 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:50.426 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:50.426 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:26:50.426 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:50.275501+0000", 2024-09-13T11:26:50.426 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.9329000000000001e-05 2024-09-13T11:26:50.426 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:50.426 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:50.427 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:26:50.427 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:50.275514+0000", 2024-09-13T11:26:50.427 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3108000000000001e-05 2024-09-13T11:26:50.427 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:50.427 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:50.427 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:26:50.427 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:50.275579+0000", 2024-09-13T11:26:50.427 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.5228000000000006e-05 2024-09-13T11:26:50.427 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:50.427 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:50.427 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-09-13T11:26:50.427 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:50.275585+0000", 2024-09-13T11:26:50.427 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.3990000000000003e-06 2024-09-13T11:26:50.428 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:50.428 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:26:50.428 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:50.428 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:50.428 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:50.428 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:266995 3.32 3.b5aadb72 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e225)", 2024-09-13T11:26:50.428 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:26:50.097578+0000", 2024-09-13T11:26:50.428 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 0.31810358300000002, 2024-09-13T11:26:50.428 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.31846087299999998, 2024-09-13T11:26:50.428 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:26:50.428 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:26:50.428 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "waiting for peered", 2024-09-13T11:26:50.428 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:26:50.428 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:26:50.429 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:50.429 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 266995 2024-09-13T11:26:50.429 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:50.429 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:26:50.429 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:50.429 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:26:50.429 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:50.097578+0000", 2024-09-13T11:26:50.429 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:50.429 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:50.429 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:50.429 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:26:50.429 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:50.097578+0000", 2024-09-13T11:26:50.429 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:50.430 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:50.430 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:50.430 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:26:50.430 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:50.097580+0000", 2024-09-13T11:26:50.430 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4759999999999999e-06 2024-09-13T11:26:50.430 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:50.430 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:50.430 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:26:50.430 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:50.097588+0000", 2024-09-13T11:26:50.430 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.1750000000000005e-06 2024-09-13T11:26:50.430 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:50.430 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:50.430 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:26:50.430 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:50.097598+0000", 2024-09-13T11:26:50.431 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0098000000000001e-05 2024-09-13T11:26:50.431 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:50.431 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:50.431 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:26:50.431 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:50.097603+0000", 2024-09-13T11:26:50.431 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.1240000000000004e-06 2024-09-13T11:26:50.431 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:50.431 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:50.431 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:26:50.431 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:50.097661+0000", 2024-09-13T11:26:50.431 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.7729999999999998e-05 2024-09-13T11:26:50.431 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:50.431 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:50.431 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-09-13T11:26:50.431 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:50.097667+0000", 2024-09-13T11:26:50.432 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.6570000000000002e-06 2024-09-13T11:26:50.432 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:50.432 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:26:50.432 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:50.432 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:50.432 INFO:teuthology.orchestra.run.smithi162.stdout: ], 2024-09-13T11:26:50.432 INFO:teuthology.orchestra.run.smithi162.stdout: "num_ops": 4 2024-09-13T11:26:50.432 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-13T11:26:50.433 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_blocked_ops 2024-09-13T11:26:50.434 INFO:tasks.ceph.osd.13.smithi092.stderr:2024-09-13T11:26:50.431+0000 7f3e5afea640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26350) UID: 0 2024-09-13T11:26:50.442 INFO:teuthology.orchestra.run.smithi001.stdout:osd.11: {} 2024-09-13T11:26:50.442 INFO:teuthology.orchestra.run.smithi001.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-13T11:26:50.465 INFO:teuthology.orchestra.run.smithi001.stdout:osd.12: {} 2024-09-13T11:26:50.466 INFO:teuthology.orchestra.run.smithi001.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-13T11:26:50.490 INFO:teuthology.orchestra.run.smithi001.stdout:osd.13: {} 2024-09-13T11:26:50.490 INFO:teuthology.orchestra.run.smithi001.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-13T11:26:50.512 INFO:teuthology.orchestra.run.smithi001.stdout:osd.14: {} 2024-09-13T11:26:50.512 INFO:teuthology.orchestra.run.smithi001.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-13T11:26:50.534 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-13T11:26:50.530+0000 7f6a68775640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26307) UID: 0 2024-09-13T11:26:50.539 INFO:teuthology.orchestra.run.smithi001.stdout:osd.15: {} 2024-09-13T11:26:50.540 INFO:teuthology.orchestra.run.smithi001.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-13T11:26:50.554 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-13T11:26:50.554 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:26:50.569 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_historic_ops 2024-09-13T11:26:50.635 INFO:tasks.ceph.osd.7.smithi177.stderr:2024-09-13T11:26:50.633+0000 7f897b51d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26575) UID: 0 2024-09-13T11:26:50.693 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-13T11:26:50.693 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:26:50.708 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-13T11:26:50.736 INFO:tasks.ceph.osd.5.smithi092.stderr:2024-09-13T11:26:50.731+0000 7f6f0b705640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26346) UID: 0 2024-09-13T11:26:50.837 INFO:tasks.ceph.osd.4.smithi001.stderr:2024-09-13T11:26:50.834+0000 7ff8e0218640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26310) UID: 0 2024-09-13T11:26:50.848 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-13T11:26:50.848 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:26:50.864 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-13T11:26:50.955 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-13T11:26:50.954+0000 7f6a68775640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26307) UID: 0 2024-09-13T11:26:50.986 INFO:tasks.ceph.osd.7.smithi177.stderr:2024-09-13T11:26:50.985+0000 7f896ff4a640 -1 osd.7 225 handle_osd_map injecting map crc failure 2024-09-13T11:26:50.988 INFO:tasks.ceph.osd.10.smithi162.stderr:2024-09-13T11:26:50.987+0000 7fb59141d640 -1 osd.10 225 handle_osd_map injecting map crc failure 2024-09-13T11:26:51.040 INFO:tasks.ceph.osd.1.smithi092.stderr:2024-09-13T11:26:51.035+0000 7fe7789c5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33718) UID: 0 2024-09-13T11:26:51.048 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-13T11:26:51.048 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:26:51.067 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-13T11:26:51.141 INFO:tasks.ceph.osd.15.smithi177.stderr:2024-09-13T11:26:51.137+0000 7f9d6ce77640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26578) UID: 0 2024-09-13T11:26:51.242 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-09-13T11:26:51.239+0000 7f97befc9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26315) UID: 0 2024-09-13T11:26:51.246 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-13T11:26:51.246 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:26:51.263 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-13T11:26:51.344 INFO:tasks.ceph.osd.2.smithi162.stderr:2024-09-13T11:26:51.339+0000 7fe92b35a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26313) UID: 0 2024-09-13T11:26:51.383 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-13T11:26:51.384 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:26:51.398 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-13T11:26:51.444 INFO:tasks.ceph.osd.8.smithi001.stderr:2024-09-13T11:26:51.442+0000 7fb9b48ae640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26312) UID: 0 2024-09-13T11:26:51.522 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-13T11:26:51.523 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:26:51.539 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-13T11:26:51.545 INFO:tasks.ceph.osd.11.smithi177.stderr:2024-09-13T11:26:51.541+0000 7f24792c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26580) UID: 0 2024-09-13T11:26:51.647 INFO:tasks.ceph.osd.9.smithi092.stderr:2024-09-13T11:26:51.643+0000 7ff5ab3d0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26348) UID: 0 2024-09-13T11:26:51.723 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-13T11:26:51.723 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:26:51.740 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-09-13T11:26:51.748 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-09-13T11:26:51.743+0000 7f97befc9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26315) UID: 0 2024-09-13T11:26:51.848 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-09-13T11:26:51.847+0000 7f97befc9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26315) UID: 0 2024-09-13T11:26:51.871 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-13T11:26:51.871 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:26:51.890 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-09-13T11:26:51.949 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-13T11:26:51.946+0000 7f6a68775640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26307) UID: 0 2024-09-13T11:26:51.978 INFO:tasks.ceph.osd.13.smithi092.stderr:2024-09-13T11:26:51.975+0000 7f3e4ba0f640 -1 osd.13 226 handle_osd_map injecting map crc failure 2024-09-13T11:26:52.000 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-13T11:26:52.042 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-13T11:26:52.042 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:26:52.050 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-13T11:26:52.046+0000 7f6a68775640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26307) UID: 0 2024-09-13T11:26:52.063 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-09-13T11:26:52.151 INFO:tasks.ceph.osd.3.smithi177.stderr:2024-09-13T11:26:52.149+0000 7f10adba4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26574) UID: 0 2024-09-13T11:26:52.230 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-13T11:26:52.230 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:26:52.245 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_ops_in_flight 2024-09-13T11:26:52.292 INFO:tasks.ceph.osd.4.smithi001.stderr:2024-09-13T11:26:52.290+0000 7ff8e0218640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26310) UID: 0 2024-09-13T11:26:52.352 INFO:tasks.ceph.osd.5.smithi092.stderr:2024-09-13T11:26:52.351+0000 7f6f0b705640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26346) UID: 0 2024-09-13T11:26:52.365 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-13T11:26:52.365 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:26:52.379 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_blocked_ops 2024-09-13T11:26:52.453 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-09-13T11:26:52.451+0000 7f97befc9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26315) UID: 0 2024-09-13T11:26:52.553 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-13T11:26:52.553 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:26:52.554 INFO:tasks.ceph.osd.11.smithi177.stderr:2024-09-13T11:26:52.553+0000 7f24792c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26580) UID: 0 2024-09-13T11:26:52.558 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T11:26:52.569 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_historic_ops 2024-09-13T11:26:52.654 INFO:tasks.ceph.osd.9.smithi092.stderr:2024-09-13T11:26:52.651+0000 7ff5ab3d0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26348) UID: 0 2024-09-13T11:26:52.715 INFO:teuthology.orchestra.run.smithi001.stdout:osd.0: {} 2024-09-13T11:26:52.715 INFO:teuthology.orchestra.run.smithi001.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T11:26:52.736 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-13T11:26:52.736 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:26:52.740 INFO:teuthology.orchestra.run.smithi001.stdout:osd.1: {} 2024-09-13T11:26:52.740 INFO:teuthology.orchestra.run.smithi001.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T11:26:52.750 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-13T11:26:52.765 INFO:teuthology.orchestra.run.smithi001.stdout:osd.2: {} 2024-09-13T11:26:52.766 INFO:teuthology.orchestra.run.smithi001.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T11:26:52.790 INFO:teuthology.orchestra.run.smithi001.stdout:osd.3: {} 2024-09-13T11:26:52.791 INFO:teuthology.orchestra.run.smithi001.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T11:26:52.797 INFO:tasks.ceph.osd.5.smithi092.stderr:2024-09-13T11:26:52.795+0000 7f6f0b705640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26346) UID: 0 2024-09-13T11:26:52.816 INFO:teuthology.orchestra.run.smithi001.stdout:osd.4: {} 2024-09-13T11:26:52.816 INFO:teuthology.orchestra.run.smithi001.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T11:26:52.844 INFO:teuthology.orchestra.run.smithi001.stdout:osd.5: {} 2024-09-13T11:26:52.844 INFO:teuthology.orchestra.run.smithi001.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T11:26:52.856 INFO:tasks.ceph.osd.9.smithi092.stderr:2024-09-13T11:26:52.855+0000 7ff5ab3d0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26348) UID: 0 2024-09-13T11:26:52.875 INFO:teuthology.orchestra.run.smithi001.stdout:osd.6: {} 2024-09-13T11:26:52.875 INFO:teuthology.orchestra.run.smithi001.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T11:26:52.883 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-09-13T11:26:52.883 INFO:teuthology.orchestra.run.smithi092.stdout: "ops": [], 2024-09-13T11:26:52.883 INFO:teuthology.orchestra.run.smithi092.stdout: "num_ops": 0 2024-09-13T11:26:52.883 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-09-13T11:26:52.897 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-13T11:26:52.906 INFO:teuthology.orchestra.run.smithi001.stdout:osd.7: {} 2024-09-13T11:26:52.906 INFO:teuthology.orchestra.run.smithi001.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T11:26:52.939 INFO:teuthology.orchestra.run.smithi001.stdout:osd.8: {} 2024-09-13T11:26:52.939 INFO:teuthology.orchestra.run.smithi001.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-13T11:26:52.957 INFO:tasks.ceph.osd.8.smithi001.stderr:2024-09-13T11:26:52.954+0000 7fb9b48ae640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26312) UID: 0 2024-09-13T11:26:52.966 INFO:teuthology.orchestra.run.smithi001.stderr:nodeep-scrub is set 2024-09-13T11:26:52.970 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-13T11:26:52.966+0000 7f6a5d1a2640 -1 osd.0 227 handle_osd_map injecting map crc failure 2024-09-13T11:26:52.974 INFO:teuthology.orchestra.run.smithi001.stdout:osd.9: {} 2024-09-13T11:26:52.975 INFO:teuthology.orchestra.run.smithi001.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-13T11:26:52.980 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 15, 14] out_osds: [] dead_osds: [] live_osds: [0, 4, 8, 12, 5, 9, 13, 2, 6, 10, 3, 7, 11, 15, 1, 14] 2024-09-13T11:26:52.980 INFO:tasks.thrashosds.thrasher:choose_action: min_in 2 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-13T11:26:52.981 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2024-09-13T11:26:52.981 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump -f json-pretty 2024-09-13T11:26:53.029 INFO:teuthology.orchestra.run.smithi001.stdout:osd.10: {} 2024-09-13T11:26:53.029 INFO:teuthology.orchestra.run.smithi001.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-13T11:26:53.045 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-09-13T11:26:53.043+0000 7f97b39f6640 -1 osd.6 227 handle_osd_map injecting map crc failure 2024-09-13T11:26:53.057 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-13T11:26:53.054+0000 7f6a68775640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26307) UID: 0 2024-09-13T11:26:53.073 INFO:teuthology.orchestra.run.smithi001.stdout:osd.11: {} 2024-09-13T11:26:53.073 INFO:teuthology.orchestra.run.smithi001.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-13T11:26:53.076 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-09-13T11:26:53.076 INFO:teuthology.orchestra.run.smithi092.stdout: "ops": [], 2024-09-13T11:26:53.076 INFO:teuthology.orchestra.run.smithi092.stdout: "complaint_time": 30, 2024-09-13T11:26:53.076 INFO:teuthology.orchestra.run.smithi092.stdout: "num_blocked_ops": 0 2024-09-13T11:26:53.076 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-09-13T11:26:53.094 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-13T11:26:53.161 INFO:tasks.ceph.osd.1.smithi092.stderr:2024-09-13T11:26:53.159+0000 7fe7789c5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33718) UID: 0 2024-09-13T11:26:53.174 INFO:teuthology.orchestra.run.smithi001.stdout:osd.12: {} 2024-09-13T11:26:53.174 INFO:teuthology.orchestra.run.smithi001.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-13T11:26:53.219 INFO:teuthology.orchestra.run.smithi001.stdout:osd.13: {} 2024-09-13T11:26:53.219 INFO:teuthology.orchestra.run.smithi001.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-13T11:26:53.220 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-09-13T11:26:53.220 INFO:teuthology.orchestra.run.smithi092.stdout: "size": 20, 2024-09-13T11:26:53.220 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 600, 2024-09-13T11:26:53.220 INFO:teuthology.orchestra.run.smithi092.stdout: "ops": [ 2024-09-13T11:26:53.221 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:53.221 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "MOSDPGPush(3.23 83/78 [PushOp(3:c61f6bab:::benchmark_data_smithi177_31116_object29479:head, version: 59'1860, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:c61f6bab:::benchmark_data_smithi177_31116_object29479:head@59'1860, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:26:53.221 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:40.770864+0000", 2024-09-13T11:26:53.221 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 192.446685664, 2024-09-13T11:26:53.221 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.31229662800000002, 2024-09-13T11:26:53.221 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:26:53.221 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:26:53.221 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-09-13T11:26:53.221 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:26:53.221 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:53.221 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:26:53.221 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:40.770864+0000", 2024-09-13T11:26:53.221 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:53.222 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:53.222 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:53.222 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:26:53.222 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:40.770864+0000", 2024-09-13T11:26:53.222 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:53.222 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:53.222 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:53.222 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:26:53.222 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:40.770865+0000", 2024-09-13T11:26:53.222 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.288e-06 2024-09-13T11:26:53.222 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:53.222 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:53.222 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:26:53.222 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:40.770955+0000", 2024-09-13T11:26:53.223 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.9871999999999996e-05 2024-09-13T11:26:53.223 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:53.223 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:53.223 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:26:53.223 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:41.082105+0000", 2024-09-13T11:26:53.223 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.31114963400000001 2024-09-13T11:26:53.223 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:53.223 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:53.223 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:26:53.223 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:41.082136+0000", 2024-09-13T11:26:53.223 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.0972000000000001e-05 2024-09-13T11:26:53.223 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:53.223 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:53.223 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:53.224 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:41.082404+0000", 2024-09-13T11:26:53.224 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00026844800000000002 2024-09-13T11:26:53.224 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:53.224 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:53.224 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:26:53.224 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:41.082429+0000", 2024-09-13T11:26:53.224 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.5021e-05 2024-09-13T11:26:53.224 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:53.224 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:53.224 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:26:53.224 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:41.083161+0000", 2024-09-13T11:26:53.224 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000731393 2024-09-13T11:26:53.224 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:53.225 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:53.225 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:53.225 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:53.225 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:53.225 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:110289 3.29 3.27287969 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e119)", 2024-09-13T11:26:53.225 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:24:25.774634+0000", 2024-09-13T11:26:53.225 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 147.44291558899999, 2024-09-13T11:26:53.225 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.31721633999999999, 2024-09-13T11:26:53.225 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:26:53.225 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:26:53.225 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "reached pg", 2024-09-13T11:26:53.225 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:26:53.225 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:26:53.226 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:53.226 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 110289 2024-09-13T11:26:53.226 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:53.226 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:26:53.226 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:53.226 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:26:53.226 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:25.774634+0000", 2024-09-13T11:26:53.226 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:53.226 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:53.226 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:53.226 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:26:53.226 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:25.774634+0000", 2024-09-13T11:26:53.226 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:53.227 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:53.227 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:53.227 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:26:53.227 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:25.774639+0000", 2024-09-13T11:26:53.227 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.6029999999999998e-06 2024-09-13T11:26:53.227 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:53.227 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:53.227 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:26:53.227 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:25.774695+0000", 2024-09-13T11:26:53.227 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.6694999999999998e-05 2024-09-13T11:26:53.227 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:53.227 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:53.227 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:26:53.228 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:25.774728+0000", 2024-09-13T11:26:53.228 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.2502000000000002e-05 2024-09-13T11:26:53.228 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:53.228 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:53.228 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:26:53.228 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:25.774750+0000", 2024-09-13T11:26:53.228 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.2002999999999999e-05 2024-09-13T11:26:53.228 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:53.228 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:53.228 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:53.228 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:25.774942+0000", 2024-09-13T11:26:53.228 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000192265 2024-09-13T11:26:53.228 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:53.229 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:53.229 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-09-13T11:26:53.229 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:25.774960+0000", 2024-09-13T11:26:53.229 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.8275999999999999e-05 2024-09-13T11:26:53.229 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:53.229 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:53.229 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:53.229 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:26.091821+0000", 2024-09-13T11:26:53.229 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.31686065000000002 2024-09-13T11:26:53.229 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:53.229 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:53.229 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:26:53.229 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:26.091850+0000", 2024-09-13T11:26:53.230 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.9346e-05 2024-09-13T11:26:53.230 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:53.230 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:53.230 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:53.230 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:53.230 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:53.230 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:138370 3.23 3:c5730cd2:::benchmark_data_smithi177_31116_object138369:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e140)", 2024-09-13T11:26:53.230 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:24:54.415176+0000", 2024-09-13T11:26:53.230 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 118.802373493, 2024-09-13T11:26:53.230 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.24549895499999999, 2024-09-13T11:26:53.230 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:26:53.230 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:26:53.230 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:53.231 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:26:53.231 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:26:53.231 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:53.231 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 138370 2024-09-13T11:26:53.231 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:53.231 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:26:53.231 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:53.231 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:26:53.231 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:54.415176+0000", 2024-09-13T11:26:53.231 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:53.231 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:53.231 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:53.231 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:26:53.231 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:54.415176+0000", 2024-09-13T11:26:53.232 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:53.232 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:53.232 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:53.232 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:26:53.232 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:54.415180+0000", 2024-09-13T11:26:53.232 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.1670000000000001e-06 2024-09-13T11:26:53.232 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:53.232 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:53.232 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:26:53.232 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:54.415269+0000", 2024-09-13T11:26:53.232 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.8659000000000006e-05 2024-09-13T11:26:53.233 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:53.233 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:53.233 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:26:53.233 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:54.415292+0000", 2024-09-13T11:26:53.233 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.3152000000000001e-05 2024-09-13T11:26:53.233 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:53.233 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:53.233 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:26:53.233 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:54.415304+0000", 2024-09-13T11:26:53.233 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.1841999999999999e-05 2024-09-13T11:26:53.233 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:53.233 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:53.234 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:53.234 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:54.415432+0000", 2024-09-13T11:26:53.234 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00012802000000000001 2024-09-13T11:26:53.234 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:53.234 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:53.234 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:26:53.234 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:54.415710+0000", 2024-09-13T11:26:53.234 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00027840299999999998 2024-09-13T11:26:53.234 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:53.234 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:53.234 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 9", 2024-09-13T11:26:53.234 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:54.415950+0000", 2024-09-13T11:26:53.234 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00023928299999999999 2024-09-13T11:26:53.235 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:53.235 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:53.235 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:26:53.235 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:54.419265+0000", 2024-09-13T11:26:53.235 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0033150440000000001 2024-09-13T11:26:53.235 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:53.235 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:53.235 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:53.235 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:54.660448+0000", 2024-09-13T11:26:53.235 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.24118348100000001 2024-09-13T11:26:53.235 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:53.235 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:53.235 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:26:53.235 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:54.660571+0000", 2024-09-13T11:26:53.236 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00012332900000000001 2024-09-13T11:26:53.236 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:53.236 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:53.236 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:26:53.236 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:54.660675+0000", 2024-09-13T11:26:53.236 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000103575 2024-09-13T11:26:53.236 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:53.236 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:53.236 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:53.236 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:53.236 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:53.236 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:140796 3.1e 3:7987a652:::benchmark_data_smithi177_31116_object493:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e144)", 2024-09-13T11:26:53.236 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:24:58.552900+0000", 2024-09-13T11:26:53.237 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 114.664649541, 2024-09-13T11:26:53.237 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.254397398, 2024-09-13T11:26:53.237 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:26:53.237 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:26:53.237 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:53.237 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:26:53.237 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:26:53.237 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:53.237 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 140796 2024-09-13T11:26:53.237 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:53.237 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:26:53.237 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:53.237 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:26:53.237 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:58.552900+0000", 2024-09-13T11:26:53.238 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:53.238 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:53.238 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:53.238 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:26:53.238 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:58.552900+0000", 2024-09-13T11:26:53.238 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:53.238 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:53.238 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:53.238 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:26:53.238 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:58.552904+0000", 2024-09-13T11:26:53.238 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.76e-06 2024-09-13T11:26:53.238 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:53.238 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:53.239 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:26:53.239 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:58.552915+0000", 2024-09-13T11:26:53.239 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0763e-05 2024-09-13T11:26:53.239 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:53.239 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:53.239 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:26:53.239 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:58.552943+0000", 2024-09-13T11:26:53.239 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.8469000000000001e-05 2024-09-13T11:26:53.239 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:53.239 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:53.239 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:26:53.239 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:58.552959+0000", 2024-09-13T11:26:53.239 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.5549999999999999e-05 2024-09-13T11:26:53.239 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:53.240 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:53.240 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:53.240 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:58.553106+0000", 2024-09-13T11:26:53.240 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00014729500000000001 2024-09-13T11:26:53.240 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:53.240 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:53.240 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:26:53.240 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:58.553348+0000", 2024-09-13T11:26:53.240 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000242353 2024-09-13T11:26:53.240 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:53.240 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:53.240 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 4", 2024-09-13T11:26:53.240 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:58.553556+0000", 2024-09-13T11:26:53.241 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00020765700000000001 2024-09-13T11:26:53.241 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:53.241 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:53.241 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:26:53.241 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:58.557140+0000", 2024-09-13T11:26:53.241 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0035844840000000002 2024-09-13T11:26:53.241 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:53.241 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:53.241 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:53.241 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:58.807126+0000", 2024-09-13T11:26:53.241 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.24998542000000001 2024-09-13T11:26:53.241 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:53.241 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:53.242 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:26:53.242 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:58.807214+0000", 2024-09-13T11:26:53.242 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.8561999999999997e-05 2024-09-13T11:26:53.242 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:53.242 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:53.242 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:26:53.242 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:58.807297+0000", 2024-09-13T11:26:53.242 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.3084999999999995e-05 2024-09-13T11:26:53.242 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:53.242 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:53.242 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:53.242 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:53.242 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:53.242 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop(client.4665.0:162073 3.1f e156/122 3:fd1a00ae:::benchmark_data_smithi177_31116_object21770:head v 156'6684, mlcod=156'6679)", 2024-09-13T11:26:53.243 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:25:15.455533+0000", 2024-09-13T11:26:53.243 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 97.762016173000006, 2024-09-13T11:26:53.243 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.24315087499999999, 2024-09-13T11:26:53.243 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:26:53.243 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:26:53.243 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:53.243 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:26:53.243 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:53.243 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:26:53.243 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:15.455533+0000", 2024-09-13T11:26:53.243 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:53.243 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:53.243 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:53.243 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:26:53.244 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:15.455533+0000", 2024-09-13T11:26:53.244 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:53.244 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:53.244 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:53.244 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:26:53.244 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:15.455535+0000", 2024-09-13T11:26:53.244 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.42e-06 2024-09-13T11:26:53.244 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:53.244 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:53.244 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:26:53.244 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:15.455542+0000", 2024-09-13T11:26:53.244 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.9820000000000002e-06 2024-09-13T11:26:53.244 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:53.245 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:53.245 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:26:53.245 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:15.696032+0000", 2024-09-13T11:26:53.245 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.240490483 2024-09-13T11:26:53.245 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:53.245 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:53.245 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:26:53.245 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:15.696039+0000", 2024-09-13T11:26:53.245 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.832e-06 2024-09-13T11:26:53.245 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:53.245 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:53.245 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:53.245 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:15.696098+0000", 2024-09-13T11:26:53.245 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.8901999999999999e-05 2024-09-13T11:26:53.246 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:53.246 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:53.246 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:26:53.246 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:15.696115+0000", 2024-09-13T11:26:53.246 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.7479999999999999e-05 2024-09-13T11:26:53.246 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:53.246 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:53.246 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:26:53.246 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:15.698637+0000", 2024-09-13T11:26:53.246 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0025215889999999999 2024-09-13T11:26:53.246 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:53.246 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:53.246 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:26:53.246 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:15.698684+0000", 2024-09-13T11:26:53.246 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.7187000000000001e-05 2024-09-13T11:26:53.247 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:53.247 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:53.299 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:53.299 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:53.299 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:53.299 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop(client.4665.0:162102 3.1f e156/122 3:fc0cda6c:::benchmark_data_smithi177_31116_object21799:head v 156'6685, mlcod=156'6680)", 2024-09-13T11:26:53.299 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:25:15.477994+0000", 2024-09-13T11:26:53.299 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 97.739555371999998, 2024-09-13T11:26:53.299 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.25110996400000002, 2024-09-13T11:26:53.299 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:26:53.299 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:26:53.299 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:53.300 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:26:53.300 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:53.300 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:26:53.300 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:15.477994+0000", 2024-09-13T11:26:53.300 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:53.300 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:53.300 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:53.300 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:26:53.300 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:15.477994+0000", 2024-09-13T11:26:53.300 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:53.300 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:53.300 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:53.301 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:26:53.301 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:15.477995+0000", 2024-09-13T11:26:53.301 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0210000000000001e-06 2024-09-13T11:26:53.301 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:53.301 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:53.301 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:26:53.301 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:15.478001+0000", 2024-09-13T11:26:53.301 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.0020000000000003e-06 2024-09-13T11:26:53.301 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:53.301 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:53.301 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:26:53.301 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:15.726384+0000", 2024-09-13T11:26:53.301 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.248382609 2024-09-13T11:26:53.301 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:53.302 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:53.302 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:26:53.302 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:15.726390+0000", 2024-09-13T11:26:53.302 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.1029999999999998e-06 2024-09-13T11:26:53.302 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:53.302 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:53.302 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:53.302 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:15.727107+0000", 2024-09-13T11:26:53.302 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000717525 2024-09-13T11:26:53.302 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:53.302 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:53.302 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:26:53.302 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:15.727130+0000", 2024-09-13T11:26:53.303 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.2572999999999999e-05 2024-09-13T11:26:53.303 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:53.303 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:53.303 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:26:53.303 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:15.729066+0000", 2024-09-13T11:26:53.303 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0019362170000000001 2024-09-13T11:26:53.303 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:53.303 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:53.303 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:26:53.303 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:15.729104+0000", 2024-09-13T11:26:53.303 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.7914000000000003e-05 2024-09-13T11:26:53.303 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:53.303 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:53.303 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:53.303 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:53.304 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:53.304 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "MOSDPGPush(3.28 168/120 [PushOp(3:14eb8ed9:::benchmark_data_smithi177_31116_object87538:head, version: 110'4071, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:14eb8ed9:::benchmark_data_smithi177_31116_object87538:head@110'4071, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:26:53.304 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:25:27.274802+0000", 2024-09-13T11:26:53.304 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 85.942747823999994, 2024-09-13T11:26:53.304 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.369268239, 2024-09-13T11:26:53.304 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:26:53.304 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:26:53.304 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-09-13T11:26:53.304 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:26:53.304 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:53.304 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:26:53.304 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:27.274802+0000", 2024-09-13T11:26:53.305 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:53.305 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:53.305 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:53.305 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:26:53.305 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:27.274802+0000", 2024-09-13T11:26:53.305 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:53.305 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:53.305 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:53.305 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:26:53.305 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:27.274803+0000", 2024-09-13T11:26:53.305 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.7579999999999999e-06 2024-09-13T11:26:53.305 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:53.306 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:53.306 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:26:53.306 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:27.274831+0000", 2024-09-13T11:26:53.306 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.7464e-05 2024-09-13T11:26:53.306 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:53.306 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:53.306 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:26:53.306 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:27.641355+0000", 2024-09-13T11:26:53.306 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.36652425199999999 2024-09-13T11:26:53.306 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:53.306 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:53.306 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:26:53.306 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:27.641385+0000", 2024-09-13T11:26:53.306 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.9433000000000001e-05 2024-09-13T11:26:53.307 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:53.307 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:53.307 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:53.307 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:27.641611+0000", 2024-09-13T11:26:53.307 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00022646299999999999 2024-09-13T11:26:53.307 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:53.307 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:53.307 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:26:53.307 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:27.641638+0000", 2024-09-13T11:26:53.307 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.6482000000000002e-05 2024-09-13T11:26:53.307 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:53.307 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:53.307 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:26:53.307 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:27.644070+0000", 2024-09-13T11:26:53.308 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0024323869999999998 2024-09-13T11:26:53.308 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:53.308 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:53.308 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:53.308 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:53.308 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:53.308 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "MOSDPGPush(3.2d 174/120 [PushOp(3:b42ee771:::benchmark_data_smithi177_31116_object109317:head, version: 118'5087, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:b42ee771:::benchmark_data_smithi177_31116_object109317:head@118'5087, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:26:53.308 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:25:36.282270+0000", 2024-09-13T11:26:53.308 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 76.935279206999994, 2024-09-13T11:26:53.308 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.26768705100000001, 2024-09-13T11:26:53.308 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:26:53.308 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:26:53.308 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-09-13T11:26:53.309 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:26:53.309 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:53.309 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:26:53.309 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:36.282270+0000", 2024-09-13T11:26:53.309 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:53.309 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:53.309 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:53.309 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:26:53.309 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:36.282270+0000", 2024-09-13T11:26:53.309 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:53.309 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:53.309 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:53.310 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:26:53.310 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:36.282272+0000", 2024-09-13T11:26:53.310 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.832e-06 2024-09-13T11:26:53.310 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:53.310 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:53.310 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:26:53.310 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:36.282337+0000", 2024-09-13T11:26:53.310 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.4834999999999998e-05 2024-09-13T11:26:53.310 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:53.310 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:53.310 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:26:53.310 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:36.547263+0000", 2024-09-13T11:26:53.310 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.26492590599999999 2024-09-13T11:26:53.310 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:53.311 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:53.311 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:26:53.311 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:36.547288+0000", 2024-09-13T11:26:53.311 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.4661999999999999e-05 2024-09-13T11:26:53.311 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:53.311 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:53.311 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:53.311 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:36.547493+0000", 2024-09-13T11:26:53.311 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00020533500000000001 2024-09-13T11:26:53.311 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:53.311 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:53.311 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:26:53.311 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:36.547515+0000", 2024-09-13T11:26:53.312 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.2124999999999999e-05 2024-09-13T11:26:53.312 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:53.312 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:53.312 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:26:53.312 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:36.549957+0000", 2024-09-13T11:26:53.312 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0024423560000000001 2024-09-13T11:26:53.312 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:53.312 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:53.312 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:53.312 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:53.312 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:53.312 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:191021 3.23 3:c4df18dd:::benchmark_data_smithi177_31116_object50718:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e175)", 2024-09-13T11:26:53.312 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:25:38.460411+0000", 2024-09-13T11:26:53.313 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 74.757138398999999, 2024-09-13T11:26:53.313 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.40138001400000001, 2024-09-13T11:26:53.313 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:26:53.313 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:26:53.313 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:53.313 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:26:53.313 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:26:53.313 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:53.313 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 191021 2024-09-13T11:26:53.313 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:53.313 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:26:53.313 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:53.313 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:26:53.314 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:38.460411+0000", 2024-09-13T11:26:53.314 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:53.314 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:53.314 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:53.314 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:26:53.314 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:38.460411+0000", 2024-09-13T11:26:53.314 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:53.314 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:53.314 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:53.314 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:26:53.314 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:38.460413+0000", 2024-09-13T11:26:53.314 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.075e-06 2024-09-13T11:26:53.314 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:53.314 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:53.315 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:26:53.315 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:38.460419+0000", 2024-09-13T11:26:53.315 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.6869999999999999e-06 2024-09-13T11:26:53.315 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:53.315 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:53.315 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:26:53.315 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:38.460431+0000", 2024-09-13T11:26:53.315 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.2451e-05 2024-09-13T11:26:53.315 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:53.315 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:53.315 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:26:53.315 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:38.460440+0000", 2024-09-13T11:26:53.315 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.5140000000000001e-06 2024-09-13T11:26:53.316 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:53.316 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:53.316 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:53.316 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:38.460501+0000", 2024-09-13T11:26:53.316 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.0772e-05 2024-09-13T11:26:53.316 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:53.316 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:53.316 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:26:53.316 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:38.460632+0000", 2024-09-13T11:26:53.316 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00013175200000000001 2024-09-13T11:26:53.316 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:53.316 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:53.316 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 9", 2024-09-13T11:26:53.317 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:38.460731+0000", 2024-09-13T11:26:53.317 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.8647000000000006e-05 2024-09-13T11:26:53.317 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:53.317 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:53.317 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:26:53.317 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:38.462322+0000", 2024-09-13T11:26:53.317 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.001590658 2024-09-13T11:26:53.317 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:53.317 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:53.317 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:53.317 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:38.861634+0000", 2024-09-13T11:26:53.317 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.39931206499999999 2024-09-13T11:26:53.317 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:53.318 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:53.318 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:26:53.318 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:38.861719+0000", 2024-09-13T11:26:53.318 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.4968000000000004e-05 2024-09-13T11:26:53.318 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:53.318 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:53.318 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:26:53.318 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:38.861791+0000", 2024-09-13T11:26:53.318 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.2424999999999999e-05 2024-09-13T11:26:53.318 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:53.318 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:53.318 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:53.318 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:53.318 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:53.319 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop_reply(client.4665.0:191021 3.23 e175/90 ondisk, result = 0)", 2024-09-13T11:26:53.319 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:25:38.462483+0000", 2024-09-13T11:26:53.319 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 74.755066976999998, 2024-09-13T11:26:53.319 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.39932300900000001, 2024-09-13T11:26:53.319 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:26:53.319 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:26:53.319 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-09-13T11:26:53.319 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:26:53.319 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:53.319 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:26:53.319 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:38.462483+0000", 2024-09-13T11:26:53.319 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:53.319 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:53.320 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:53.320 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:26:53.320 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:38.462483+0000", 2024-09-13T11:26:53.320 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:53.320 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:53.320 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:53.320 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:26:53.320 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:38.462484+0000", 2024-09-13T11:26:53.320 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.1990000000000001e-06 2024-09-13T11:26:53.320 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:53.320 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:53.320 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:26:53.320 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:38.462488+0000", 2024-09-13T11:26:53.320 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.985e-06 2024-09-13T11:26:53.321 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:53.321 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:53.321 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:26:53.321 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:38.861475+0000", 2024-09-13T11:26:53.321 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.39898754800000003 2024-09-13T11:26:53.321 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:53.321 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:53.321 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:26:53.321 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:38.861488+0000", 2024-09-13T11:26:53.321 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.3037000000000001e-05 2024-09-13T11:26:53.321 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:53.321 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:53.321 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:53.322 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:38.861607+0000", 2024-09-13T11:26:53.322 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00011835 2024-09-13T11:26:53.322 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:53.322 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:53.322 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:26:53.322 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:38.861624+0000", 2024-09-13T11:26:53.322 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.7730000000000001e-05 2024-09-13T11:26:53.322 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:53.322 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:53.322 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:26:53.322 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:38.861806+0000", 2024-09-13T11:26:53.322 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00018116000000000001 2024-09-13T11:26:53.322 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:53.323 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:53.323 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:53.323 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:53.323 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:53.323 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:209500 3.27 3:e6941997:::benchmark_data_smithi177_31116_object69197:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e185)", 2024-09-13T11:26:53.323 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:25:53.962062+0000", 2024-09-13T11:26:53.323 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 59.255487232999997, 2024-09-13T11:26:53.323 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.26007328699999999, 2024-09-13T11:26:53.323 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:26:53.323 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:26:53.323 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:53.323 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:26:53.323 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:26:53.323 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:53.324 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 209500 2024-09-13T11:26:53.324 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:53.324 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:26:53.324 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:53.324 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:26:53.324 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.962062+0000", 2024-09-13T11:26:53.324 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:53.324 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:53.324 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:53.324 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:26:53.324 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.962062+0000", 2024-09-13T11:26:53.324 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:53.324 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:53.325 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:53.325 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:26:53.325 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.962065+0000", 2024-09-13T11:26:53.325 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.9019999999999999e-06 2024-09-13T11:26:53.325 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:53.325 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:53.325 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:26:53.325 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.962074+0000", 2024-09-13T11:26:53.325 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.6030000000000005e-06 2024-09-13T11:26:53.325 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:53.325 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:53.325 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:26:53.325 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.962098+0000", 2024-09-13T11:26:53.326 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.4076000000000001e-05 2024-09-13T11:26:53.326 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:53.326 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:53.326 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:26:53.326 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.962114+0000", 2024-09-13T11:26:53.326 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.5820000000000001e-05 2024-09-13T11:26:53.326 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:53.326 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:53.326 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:53.326 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.962280+0000", 2024-09-13T11:26:53.326 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00016648300000000001 2024-09-13T11:26:53.326 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:53.326 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:53.326 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:26:53.327 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.962598+0000", 2024-09-13T11:26:53.327 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00031824000000000002 2024-09-13T11:26:53.327 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:53.327 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:53.327 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 11", 2024-09-13T11:26:53.327 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.962906+0000", 2024-09-13T11:26:53.327 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00030721699999999998 2024-09-13T11:26:53.327 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:53.327 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:53.327 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:26:53.327 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.965943+0000", 2024-09-13T11:26:53.327 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0030373010000000001 2024-09-13T11:26:53.327 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:53.328 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:53.328 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:53.328 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:54.222003+0000", 2024-09-13T11:26:53.328 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.25605972900000001 2024-09-13T11:26:53.328 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:53.328 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:53.328 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:26:53.328 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:54.222081+0000", 2024-09-13T11:26:53.328 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.8442000000000006e-05 2024-09-13T11:26:53.328 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:53.328 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:53.328 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:26:53.328 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:54.222136+0000", 2024-09-13T11:26:53.329 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.4474000000000001e-05 2024-09-13T11:26:53.329 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:53.329 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:53.329 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:53.329 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:53.329 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:53.329 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:209869 3.27 3:e6b1fcc5:::benchmark_data_smithi177_31116_object69566:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e185)", 2024-09-13T11:26:53.329 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:25:54.300263+0000", 2024-09-13T11:26:53.329 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 58.917286079999997, 2024-09-13T11:26:53.329 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.30266981599999998, 2024-09-13T11:26:53.329 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:26:53.329 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:26:53.329 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:53.330 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:26:53.330 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:26:53.330 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:53.330 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 209869 2024-09-13T11:26:53.330 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:53.330 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:26:53.330 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:53.330 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:26:53.330 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:54.300263+0000", 2024-09-13T11:26:53.330 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:53.330 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:53.330 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:53.330 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:26:53.330 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:54.300263+0000", 2024-09-13T11:26:53.331 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:53.331 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:53.331 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:53.331 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:26:53.331 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:54.300267+0000", 2024-09-13T11:26:53.331 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.2200000000000001e-06 2024-09-13T11:26:53.331 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:53.331 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:53.331 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:26:53.331 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:54.300276+0000", 2024-09-13T11:26:53.331 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.3570000000000004e-06 2024-09-13T11:26:53.331 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:53.331 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:53.332 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:26:53.332 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:54.300301+0000", 2024-09-13T11:26:53.332 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.5191e-05 2024-09-13T11:26:53.332 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:53.332 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:53.332 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:26:53.332 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:54.300321+0000", 2024-09-13T11:26:53.332 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.9967999999999999e-05 2024-09-13T11:26:53.332 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:53.332 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:53.332 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:53.332 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:54.300427+0000", 2024-09-13T11:26:53.332 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000105781 2024-09-13T11:26:53.333 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:53.333 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:53.333 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:26:53.333 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:54.300590+0000", 2024-09-13T11:26:53.333 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00016341000000000001 2024-09-13T11:26:53.333 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:53.333 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:53.333 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 11", 2024-09-13T11:26:53.333 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:54.300724+0000", 2024-09-13T11:26:53.334 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000133323 2024-09-13T11:26:53.334 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:53.334 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:53.334 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:26:53.334 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:54.303373+0000", 2024-09-13T11:26:53.334 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0026488050000000002 2024-09-13T11:26:53.334 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:53.334 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:53.334 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:53.334 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:54.602650+0000", 2024-09-13T11:26:53.334 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.299277976 2024-09-13T11:26:53.334 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:53.334 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:53.335 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:26:53.335 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:54.602795+0000", 2024-09-13T11:26:53.335 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000144127 2024-09-13T11:26:53.335 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:53.335 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:53.335 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:26:53.335 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:54.602933+0000", 2024-09-13T11:26:53.335 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000138658 2024-09-13T11:26:53.335 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:53.335 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:53.335 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:53.335 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:53.335 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:53.336 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop(client.4665.0:217082 3.22 e187/122 3:449e4b87:::benchmark_data_smithi177_31116_object76779:head v 187'5666, mlcod=187'5664)", 2024-09-13T11:26:53.336 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:25:59.884121+0000", 2024-09-13T11:26:53.336 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 53.333428400000003, 2024-09-13T11:26:53.336 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.25479380699999998, 2024-09-13T11:26:53.336 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:26:53.336 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:26:53.336 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:53.336 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:26:53.336 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:53.336 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:26:53.336 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:59.884121+0000", 2024-09-13T11:26:53.336 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:53.336 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:53.337 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:53.337 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:26:53.337 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:59.884121+0000", 2024-09-13T11:26:53.337 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:53.337 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:53.337 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:53.337 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:26:53.337 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:59.884123+0000", 2024-09-13T11:26:53.337 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.677e-06 2024-09-13T11:26:53.337 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:53.337 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:53.337 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:26:53.337 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:59.884129+0000", 2024-09-13T11:26:53.337 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.669e-06 2024-09-13T11:26:53.338 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:53.338 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:53.338 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:26:53.338 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:00.137020+0000", 2024-09-13T11:26:53.338 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.25289069199999997 2024-09-13T11:26:53.338 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:53.338 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:53.338 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:26:53.338 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:00.137030+0000", 2024-09-13T11:26:53.338 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.8600000000000005e-06 2024-09-13T11:26:53.338 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:53.338 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:53.338 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:53.338 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:00.137123+0000", 2024-09-13T11:26:53.339 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.2842000000000003e-05 2024-09-13T11:26:53.339 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:53.339 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:53.339 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:26:53.339 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:00.137139+0000", 2024-09-13T11:26:53.339 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.6388e-05 2024-09-13T11:26:53.339 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:53.339 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:53.339 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:26:53.339 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:00.138876+0000", 2024-09-13T11:26:53.339 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0017370809999999999 2024-09-13T11:26:53.339 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:53.339 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:53.339 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:26:53.339 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:00.138915+0000", 2024-09-13T11:26:53.340 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.8597999999999999e-05 2024-09-13T11:26:53.340 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:53.340 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:53.340 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:53.340 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:53.340 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:53.340 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:221567 3.12 3:4823d14b:::benchmark_data_smithi177_31116_object81264:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e193)", 2024-09-13T11:26:53.340 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:26:06.064686+0000", 2024-09-13T11:26:53.340 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 47.152863842000002, 2024-09-13T11:26:53.340 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.28539763299999998, 2024-09-13T11:26:53.340 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:26:53.340 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:26:53.340 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:53.341 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:26:53.341 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:26:53.341 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:53.341 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 221567 2024-09-13T11:26:53.341 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:53.341 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:26:53.341 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:53.341 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:26:53.341 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:06.064686+0000", 2024-09-13T11:26:53.341 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:53.341 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:53.341 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:53.341 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:26:53.341 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:06.064686+0000", 2024-09-13T11:26:53.342 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:53.342 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:53.342 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:53.342 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:26:53.342 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:06.064687+0000", 2024-09-13T11:26:53.342 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.6300000000000001e-06 2024-09-13T11:26:53.342 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:53.342 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:53.342 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:26:53.342 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:06.064693+0000", 2024-09-13T11:26:53.342 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.7339999999999996e-06 2024-09-13T11:26:53.342 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:53.342 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:53.343 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:26:53.343 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:06.064704+0000", 2024-09-13T11:26:53.343 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.1253000000000001e-05 2024-09-13T11:26:53.343 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:53.343 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:53.343 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:26:53.343 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:06.064717+0000", 2024-09-13T11:26:53.343 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.2293000000000001e-05 2024-09-13T11:26:53.343 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:53.343 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:53.343 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:53.343 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:06.064866+0000", 2024-09-13T11:26:53.343 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00014957499999999999 2024-09-13T11:26:53.344 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:53.344 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:53.344 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-09-13T11:26:53.344 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:06.064893+0000", 2024-09-13T11:26:53.344 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.6614e-05 2024-09-13T11:26:53.344 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:53.344 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:53.344 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:53.344 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:06.282646+0000", 2024-09-13T11:26:53.344 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.21775271800000001 2024-09-13T11:26:53.344 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:53.344 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:53.344 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:26:53.344 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:06.283003+0000", 2024-09-13T11:26:53.345 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00035780799999999999 2024-09-13T11:26:53.345 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:53.345 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:53.345 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 7,14", 2024-09-13T11:26:53.345 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:06.283345+0000", 2024-09-13T11:26:53.345 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00034164200000000002 2024-09-13T11:26:53.345 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:53.345 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:53.345 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:26:53.345 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:06.288935+0000", 2024-09-13T11:26:53.345 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0055902429999999999 2024-09-13T11:26:53.345 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:53.345 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:53.345 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:53.346 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:06.289795+0000", 2024-09-13T11:26:53.346 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00085952200000000004 2024-09-13T11:26:53.346 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:53.346 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:53.346 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:53.346 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:06.349852+0000", 2024-09-13T11:26:53.346 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.060057352000000001 2024-09-13T11:26:53.346 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:53.346 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:53.346 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:26:53.346 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:06.349922+0000", 2024-09-13T11:26:53.346 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.0272999999999996e-05 2024-09-13T11:26:53.346 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:53.346 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:53.347 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:26:53.347 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:06.350083+0000", 2024-09-13T11:26:53.347 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00016097600000000001 2024-09-13T11:26:53.347 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:53.347 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:53.347 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:53.347 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:53.347 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:53.347 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:227066 3.23 3:c67f2f28:::benchmark_data_smithi177_31116_object86763:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e196)", 2024-09-13T11:26:53.347 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:26:10.753624+0000", 2024-09-13T11:26:53.347 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 42.463925768999999, 2024-09-13T11:26:53.347 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.28670347299999999, 2024-09-13T11:26:53.347 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:26:53.348 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:26:53.348 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:53.348 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:26:53.348 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:26:53.348 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:53.348 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 227066 2024-09-13T11:26:53.348 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:53.348 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:26:53.348 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:53.348 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:26:53.348 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:10.753624+0000", 2024-09-13T11:26:53.348 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:53.349 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:53.349 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:53.349 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:26:53.349 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:10.753624+0000", 2024-09-13T11:26:53.349 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:53.349 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:53.349 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:53.349 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:26:53.349 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:10.753627+0000", 2024-09-13T11:26:53.349 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.8030000000000001e-06 2024-09-13T11:26:53.349 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:53.349 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:53.349 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:26:53.350 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:10.753634+0000", 2024-09-13T11:26:53.350 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.9659999999999999e-06 2024-09-13T11:26:53.350 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:53.350 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:53.350 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:26:53.350 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:10.753650+0000", 2024-09-13T11:26:53.350 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.6167999999999998e-05 2024-09-13T11:26:53.350 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:53.350 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:53.350 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:26:53.350 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:10.753657+0000", 2024-09-13T11:26:53.350 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.7600000000000002e-06 2024-09-13T11:26:53.350 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:53.351 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:53.351 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:53.351 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:10.753741+0000", 2024-09-13T11:26:53.351 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.3523999999999995e-05 2024-09-13T11:26:53.351 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:53.351 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:53.351 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:26:53.351 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:10.753878+0000", 2024-09-13T11:26:53.351 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000137207 2024-09-13T11:26:53.351 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:53.351 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:53.351 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 9", 2024-09-13T11:26:53.351 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:10.754027+0000", 2024-09-13T11:26:53.352 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00014856599999999999 2024-09-13T11:26:53.352 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:53.352 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:53.352 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:26:53.352 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:10.756313+0000", 2024-09-13T11:26:53.352 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.002286045 2024-09-13T11:26:53.352 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:53.352 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:53.352 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:53.352 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:11.040266+0000", 2024-09-13T11:26:53.352 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.283953434 2024-09-13T11:26:53.352 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:53.352 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:53.352 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:26:53.353 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:11.040300+0000", 2024-09-13T11:26:53.353 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.4227000000000003e-05 2024-09-13T11:26:53.353 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:53.353 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:53.353 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:26:53.353 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:11.040327+0000", 2024-09-13T11:26:53.353 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.6772999999999999e-05 2024-09-13T11:26:53.353 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:53.353 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:53.353 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:53.353 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:53.353 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:53.354 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop_reply(client.4665.0:227066 3.23 e196/90 ondisk, result = 0)", 2024-09-13T11:26:53.354 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:26:10.756369+0000", 2024-09-13T11:26:53.354 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 42.461180395, 2024-09-13T11:26:53.354 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.28396190599999999, 2024-09-13T11:26:53.354 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:26:53.354 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:26:53.354 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-09-13T11:26:53.354 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:26:53.354 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:53.354 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:26:53.354 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:10.756369+0000", 2024-09-13T11:26:53.354 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:53.354 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:53.355 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:53.355 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:26:53.355 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:10.756369+0000", 2024-09-13T11:26:53.355 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:53.355 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:53.355 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:53.355 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:26:53.355 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:10.756370+0000", 2024-09-13T11:26:53.355 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.164e-06 2024-09-13T11:26:53.355 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:53.355 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:53.355 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:26:53.355 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:10.756374+0000", 2024-09-13T11:26:53.355 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.0659999999999997e-06 2024-09-13T11:26:53.356 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:53.356 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:53.356 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:26:53.356 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:11.040177+0000", 2024-09-13T11:26:53.356 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.28380288599999998 2024-09-13T11:26:53.356 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:53.356 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:53.356 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:26:53.356 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:11.040184+0000", 2024-09-13T11:26:53.356 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.7229999999999998e-06 2024-09-13T11:26:53.356 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:53.356 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:53.356 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:53.356 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:11.040253+0000", 2024-09-13T11:26:53.357 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.8997000000000004e-05 2024-09-13T11:26:53.357 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:53.357 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:53.357 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:26:53.357 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:11.040261+0000", 2024-09-13T11:26:53.357 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.3590000000000008e-06 2024-09-13T11:26:53.357 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:53.357 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:53.357 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:26:53.357 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:11.040331+0000", 2024-09-13T11:26:53.357 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.9710999999999997e-05 2024-09-13T11:26:53.357 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:53.358 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:53.358 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:53.358 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:53.358 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:53.358 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:227420 3.23 3:c4add80a:::benchmark_data_smithi177_31116_object87117:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e196)", 2024-09-13T11:26:53.358 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:26:11.037912+0000", 2024-09-13T11:26:53.358 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 42.179637749000001, 2024-09-13T11:26:53.358 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.33121650499999999, 2024-09-13T11:26:53.358 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:26:53.358 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:26:53.358 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:53.358 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:26:53.358 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:26:53.358 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:53.359 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 227420 2024-09-13T11:26:53.359 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:53.359 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:26:53.359 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:53.359 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:26:53.359 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:11.037912+0000", 2024-09-13T11:26:53.359 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:53.359 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:53.359 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:53.359 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:26:53.359 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:11.037912+0000", 2024-09-13T11:26:53.359 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:53.359 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:53.359 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:53.360 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:26:53.360 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:11.037913+0000", 2024-09-13T11:26:53.360 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.606e-06 2024-09-13T11:26:53.360 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:53.360 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:53.360 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:26:53.360 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:11.037918+0000", 2024-09-13T11:26:53.360 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.0509999999999996e-06 2024-09-13T11:26:53.360 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:53.360 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:53.360 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:26:53.360 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:11.037930+0000", 2024-09-13T11:26:53.360 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.1466e-05 2024-09-13T11:26:53.360 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:53.361 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:53.361 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:26:53.361 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:11.037936+0000", 2024-09-13T11:26:53.361 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.5259999999999999e-06 2024-09-13T11:26:53.361 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:53.361 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:53.361 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:53.361 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:11.037994+0000", 2024-09-13T11:26:53.361 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.7776999999999999e-05 2024-09-13T11:26:53.361 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:53.361 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:53.361 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:26:53.362 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:11.038129+0000", 2024-09-13T11:26:53.362 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00013500299999999999 2024-09-13T11:26:53.362 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:53.362 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:53.362 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 9", 2024-09-13T11:26:53.362 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:11.038208+0000", 2024-09-13T11:26:53.362 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.9016000000000004e-05 2024-09-13T11:26:53.362 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:53.362 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:53.362 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:26:53.362 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:11.041461+0000", 2024-09-13T11:26:53.362 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00325246 2024-09-13T11:26:53.362 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:53.363 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:53.363 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:53.363 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:11.369025+0000", 2024-09-13T11:26:53.363 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.32756474800000002 2024-09-13T11:26:53.363 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:53.363 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:53.363 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:26:53.363 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:11.369086+0000", 2024-09-13T11:26:53.363 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.0140999999999999e-05 2024-09-13T11:26:53.363 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:53.363 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:53.363 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:26:53.363 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:11.369128+0000", 2024-09-13T11:26:53.364 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.2710999999999998e-05 2024-09-13T11:26:53.364 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:53.364 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:53.364 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:53.364 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:53.364 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:53.364 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop_reply(client.4665.0:227420 3.23 e196/90 ondisk, result = 0)", 2024-09-13T11:26:53.364 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:26:11.040152+0000", 2024-09-13T11:26:53.364 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 42.177397911, 2024-09-13T11:26:53.364 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.328983201, 2024-09-13T11:26:53.364 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:26:53.364 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:26:53.364 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-09-13T11:26:53.365 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:26:53.365 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:53.365 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:26:53.365 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:11.040152+0000", 2024-09-13T11:26:53.365 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:53.365 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:53.365 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:53.365 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:26:53.365 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:11.040152+0000", 2024-09-13T11:26:53.365 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:53.365 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:53.365 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:53.365 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:26:53.365 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:11.040157+0000", 2024-09-13T11:26:53.366 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.1309999999999997e-06 2024-09-13T11:26:53.366 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:53.366 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:53.366 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:26:53.366 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:11.040162+0000", 2024-09-13T11:26:53.366 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.8720000000000001e-06 2024-09-13T11:26:53.366 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:53.366 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:53.366 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:26:53.366 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:11.368952+0000", 2024-09-13T11:26:53.366 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.32879022400000002 2024-09-13T11:26:53.366 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:53.366 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:53.366 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:26:53.367 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:11.368960+0000", 2024-09-13T11:26:53.367 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.8460000000000008e-06 2024-09-13T11:26:53.367 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:53.367 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:53.367 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:53.367 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:11.369013+0000", 2024-09-13T11:26:53.367 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.3201000000000001e-05 2024-09-13T11:26:53.367 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:53.367 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:53.367 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:26:53.367 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:11.369020+0000", 2024-09-13T11:26:53.367 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.3050000000000004e-06 2024-09-13T11:26:53.367 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:53.367 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:53.368 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:26:53.368 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:11.369135+0000", 2024-09-13T11:26:53.368 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000114622 2024-09-13T11:26:53.368 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:53.368 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:53.368 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:53.368 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:53.368 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:53.368 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:244694 3.27 3:e45f60ea:::benchmark_data_smithi177_31116_object104391:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e208)", 2024-09-13T11:26:53.368 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:26:27.471915+0000", 2024-09-13T11:26:53.368 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 25.745634257999999, 2024-09-13T11:26:53.368 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.28616412299999999, 2024-09-13T11:26:53.368 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:26:53.368 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:26:53.369 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:53.369 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:26:53.369 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:26:53.369 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:53.369 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 244694 2024-09-13T11:26:53.369 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:53.369 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:26:53.369 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:53.369 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:26:53.369 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:27.471915+0000", 2024-09-13T11:26:53.369 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:53.369 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:53.370 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:53.370 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:26:53.370 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:27.471915+0000", 2024-09-13T11:26:53.370 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:53.370 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:53.370 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:53.370 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:26:53.370 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:27.471916+0000", 2024-09-13T11:26:53.370 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.8100000000000001e-07 2024-09-13T11:26:53.370 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:53.370 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:53.370 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:26:53.370 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:27.471919+0000", 2024-09-13T11:26:53.370 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.2179999999999999e-06 2024-09-13T11:26:53.371 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:53.371 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:53.371 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:26:53.371 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:27.471928+0000", 2024-09-13T11:26:53.371 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.4279999999999996e-06 2024-09-13T11:26:53.371 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:53.371 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:53.371 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:26:53.371 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:27.471934+0000", 2024-09-13T11:26:53.371 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.3180000000000003e-06 2024-09-13T11:26:53.371 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:53.371 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:53.371 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:53.372 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:27.471983+0000", 2024-09-13T11:26:53.372 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.8393000000000002e-05 2024-09-13T11:26:53.372 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:53.372 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:53.372 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:26:53.372 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:27.472085+0000", 2024-09-13T11:26:53.372 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000102723 2024-09-13T11:26:53.372 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:53.372 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:53.372 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 11", 2024-09-13T11:26:53.372 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:27.472218+0000", 2024-09-13T11:26:53.372 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00013276600000000001 2024-09-13T11:26:53.372 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:53.372 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:53.373 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:26:53.373 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:27.474061+0000", 2024-09-13T11:26:53.373 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.001842797 2024-09-13T11:26:53.373 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:53.373 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:53.373 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:53.373 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:27.757978+0000", 2024-09-13T11:26:53.373 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.28391740999999998 2024-09-13T11:26:53.373 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:53.373 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:53.373 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:26:53.373 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:27.758033+0000", 2024-09-13T11:26:53.373 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.4997000000000002e-05 2024-09-13T11:26:53.373 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:53.374 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:53.374 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:26:53.374 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:27.758079+0000", 2024-09-13T11:26:53.374 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.6091999999999998e-05 2024-09-13T11:26:53.374 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:53.374 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:53.374 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:53.374 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:53.374 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:53.374 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop_reply(client.4665.0:244694 3.27 e208/169 ondisk, result = 0)", 2024-09-13T11:26:53.374 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:26:27.474690+0000", 2024-09-13T11:26:53.374 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 25.742859331999998, 2024-09-13T11:26:53.374 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.28339594400000001, 2024-09-13T11:26:53.374 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:26:53.375 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:26:53.375 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-09-13T11:26:53.375 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:26:53.375 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:53.375 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:26:53.375 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:27.474690+0000", 2024-09-13T11:26:53.375 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:53.375 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:53.375 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:53.375 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:26:53.375 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:27.474690+0000", 2024-09-13T11:26:53.375 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:53.376 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:53.376 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:53.376 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:26:53.376 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:27.474692+0000", 2024-09-13T11:26:53.376 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.615e-06 2024-09-13T11:26:53.376 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:53.376 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:53.376 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:26:53.376 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:27.474696+0000", 2024-09-13T11:26:53.376 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.233e-06 2024-09-13T11:26:53.376 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:53.376 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:53.376 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:26:53.376 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:27.757907+0000", 2024-09-13T11:26:53.377 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.28321128000000001 2024-09-13T11:26:53.377 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:53.377 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:53.377 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:26:53.377 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:27.757916+0000", 2024-09-13T11:26:53.377 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.1890000000000007e-06 2024-09-13T11:26:53.377 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:53.377 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:53.377 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:53.377 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:27.757964+0000", 2024-09-13T11:26:53.377 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.8609999999999997e-05 2024-09-13T11:26:53.377 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:53.377 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:53.378 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:26:53.378 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:27.757972+0000", 2024-09-13T11:26:53.378 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.8709999999999997e-06 2024-09-13T11:26:53.378 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:53.378 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:53.378 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:26:53.378 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:27.758086+0000", 2024-09-13T11:26:53.378 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000114146 2024-09-13T11:26:53.378 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:53.378 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:53.378 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:53.378 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:53.378 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:53.378 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-09-13T11:26:53.379 INFO:teuthology.orchestra.run.smithi001.stdout:osd.14: {} 2024-09-13T11:26:53.380 INFO:teuthology.orchestra.run.smithi001.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-13T11:26:53.381 INFO:teuthology.orchestra.run.smithi001.stdout:osd.15: {} 2024-09-13T11:26:53.381 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-09-13T11:26:53.382 INFO:teuthology.orchestra.run.smithi001.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-13T11:26:53.383 INFO:tasks.ceph.osd.8.smithi001.stderr:2024-09-13T11:26:53.378+0000 7fb9b48ae640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26312) UID: 0 2024-09-13T11:26:53.483 INFO:tasks.ceph.osd.15.smithi177.stderr:2024-09-13T11:26:53.481+0000 7f9d6ce77640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26578) UID: 0 2024-09-13T11:26:53.536 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-13T11:26:53.536 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-13T11:26:53.536 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 228, 2024-09-13T11:26:53.536 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "2fd68ac7-5581-4d90-80a4-30bcf5252a7a", 2024-09-13T11:26:53.536 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-09-13T11:21:57.223153+0000", 2024-09-13T11:26:53.536 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-09-13T11:26:52.959392+0000", 2024-09-13T11:26:53.536 INFO:teuthology.orchestra.run.smithi001.stdout: "last_up_change": "2024-09-13T11:24:55.968636+0000", 2024-09-13T11:26:53.537 INFO:teuthology.orchestra.run.smithi001.stdout: "last_in_change": "2024-09-13T11:26:04.007363+0000", 2024-09-13T11:26:53.537 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": "noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-13T11:26:53.537 INFO:teuthology.orchestra.run.smithi001.stdout: "flags_num": 5806080, 2024-09-13T11:26:53.537 INFO:teuthology.orchestra.run.smithi001.stdout: "flags_set": [ 2024-09-13T11:26:53.537 INFO:teuthology.orchestra.run.smithi001.stdout: "nodeep-scrub", 2024-09-13T11:26:53.537 INFO:teuthology.orchestra.run.smithi001.stdout: "noscrub", 2024-09-13T11:26:53.537 INFO:teuthology.orchestra.run.smithi001.stdout: "pglog_hardlimit", 2024-09-13T11:26:53.537 INFO:teuthology.orchestra.run.smithi001.stdout: "purged_snapdirs", 2024-09-13T11:26:53.537 INFO:teuthology.orchestra.run.smithi001.stdout: "recovery_deletes", 2024-09-13T11:26:53.537 INFO:teuthology.orchestra.run.smithi001.stdout: "sortbitwise" 2024-09-13T11:26:53.538 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-09-13T11:26:53.538 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_version": 7, 2024-09-13T11:26:53.538 INFO:teuthology.orchestra.run.smithi001.stdout: "full_ratio": 0.94999998807907104, 2024-09-13T11:26:53.538 INFO:teuthology.orchestra.run.smithi001.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-09-13T11:26:53.538 INFO:teuthology.orchestra.run.smithi001.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-13T11:26:53.538 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_snapshot": "", 2024-09-13T11:26:53.538 INFO:teuthology.orchestra.run.smithi001.stdout: "pool_max": 3, 2024-09-13T11:26:53.538 INFO:teuthology.orchestra.run.smithi001.stdout: "max_osd": 16, 2024-09-13T11:26:53.538 INFO:teuthology.orchestra.run.smithi001.stdout: "require_min_compat_client": "luminous", 2024-09-13T11:26:53.538 INFO:teuthology.orchestra.run.smithi001.stdout: "min_compat_client": "luminous", 2024-09-13T11:26:53.538 INFO:teuthology.orchestra.run.smithi001.stdout: "require_osd_release": "squid", 2024-09-13T11:26:53.538 INFO:teuthology.orchestra.run.smithi001.stdout: "allow_crimson": false, 2024-09-13T11:26:53.538 INFO:teuthology.orchestra.run.smithi001.stdout: "pools": [ 2024-09-13T11:26:53.538 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:53.539 INFO:teuthology.orchestra.run.smithi001.stdout: "pool": 1, 2024-09-13T11:26:53.539 INFO:teuthology.orchestra.run.smithi001.stdout: "pool_name": ".mgr", 2024-09-13T11:26:53.539 INFO:teuthology.orchestra.run.smithi001.stdout: "create_time": "2024-09-13T11:22:10.162266+0000", 2024-09-13T11:26:53.539 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-09-13T11:26:53.539 INFO:teuthology.orchestra.run.smithi001.stdout: "flags_names": "hashpspool", 2024-09-13T11:26:53.539 INFO:teuthology.orchestra.run.smithi001.stdout: "type": 1, 2024-09-13T11:26:53.539 INFO:teuthology.orchestra.run.smithi001.stdout: "size": 2, 2024-09-13T11:26:53.539 INFO:teuthology.orchestra.run.smithi001.stdout: "min_size": 2, 2024-09-13T11:26:53.539 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_rule": 0, 2024-09-13T11:26:53.539 INFO:teuthology.orchestra.run.smithi001.stdout: "peering_crush_bucket_count": 0, 2024-09-13T11:26:53.539 INFO:teuthology.orchestra.run.smithi001.stdout: "peering_crush_bucket_target": 0, 2024-09-13T11:26:53.539 INFO:teuthology.orchestra.run.smithi001.stdout: "peering_crush_bucket_barrier": 0, 2024-09-13T11:26:53.540 INFO:teuthology.orchestra.run.smithi001.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-13T11:26:53.540 INFO:teuthology.orchestra.run.smithi001.stdout: "is_stretch_pool": false, 2024-09-13T11:26:53.540 INFO:teuthology.orchestra.run.smithi001.stdout: "object_hash": 2, 2024-09-13T11:26:53.540 INFO:teuthology.orchestra.run.smithi001.stdout: "pg_autoscale_mode": "off", 2024-09-13T11:26:53.540 INFO:teuthology.orchestra.run.smithi001.stdout: "pg_num": 1, 2024-09-13T11:26:53.540 INFO:teuthology.orchestra.run.smithi001.stdout: "pg_placement_num": 1, 2024-09-13T11:26:53.540 INFO:teuthology.orchestra.run.smithi001.stdout: "pg_placement_num_target": 1, 2024-09-13T11:26:53.540 INFO:teuthology.orchestra.run.smithi001.stdout: "pg_num_target": 1, 2024-09-13T11:26:53.540 INFO:teuthology.orchestra.run.smithi001.stdout: "pg_num_pending": 1, 2024-09-13T11:26:53.540 INFO:teuthology.orchestra.run.smithi001.stdout: "last_pg_merge_meta": { 2024-09-13T11:26:53.540 INFO:teuthology.orchestra.run.smithi001.stdout: "source_pgid": "0.0", 2024-09-13T11:26:53.540 INFO:teuthology.orchestra.run.smithi001.stdout: "ready_epoch": 0, 2024-09-13T11:26:53.541 INFO:teuthology.orchestra.run.smithi001.stdout: "last_epoch_started": 0, 2024-09-13T11:26:53.541 INFO:teuthology.orchestra.run.smithi001.stdout: "last_epoch_clean": 0, 2024-09-13T11:26:53.541 INFO:teuthology.orchestra.run.smithi001.stdout: "source_version": "0'0", 2024-09-13T11:26:53.541 INFO:teuthology.orchestra.run.smithi001.stdout: "target_version": "0'0" 2024-09-13T11:26:53.541 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:53.541 INFO:teuthology.orchestra.run.smithi001.stdout: "last_change": "25", 2024-09-13T11:26:53.541 INFO:teuthology.orchestra.run.smithi001.stdout: "last_force_op_resend": "0", 2024-09-13T11:26:53.541 INFO:teuthology.orchestra.run.smithi001.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-13T11:26:53.541 INFO:teuthology.orchestra.run.smithi001.stdout: "last_force_op_resend_preluminous": "0", 2024-09-13T11:26:53.541 INFO:teuthology.orchestra.run.smithi001.stdout: "auid": 0, 2024-09-13T11:26:53.541 INFO:teuthology.orchestra.run.smithi001.stdout: "snap_mode": "selfmanaged", 2024-09-13T11:26:53.542 INFO:teuthology.orchestra.run.smithi001.stdout: "snap_seq": 0, 2024-09-13T11:26:53.542 INFO:teuthology.orchestra.run.smithi001.stdout: "snap_epoch": 0, 2024-09-13T11:26:53.542 INFO:teuthology.orchestra.run.smithi001.stdout: "pool_snaps": [], 2024-09-13T11:26:53.542 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_snaps": "[]", 2024-09-13T11:26:53.542 INFO:teuthology.orchestra.run.smithi001.stdout: "quota_max_bytes": 0, 2024-09-13T11:26:53.542 INFO:teuthology.orchestra.run.smithi001.stdout: "quota_max_objects": 0, 2024-09-13T11:26:53.542 INFO:teuthology.orchestra.run.smithi001.stdout: "tiers": [], 2024-09-13T11:26:53.542 INFO:teuthology.orchestra.run.smithi001.stdout: "tier_of": -1, 2024-09-13T11:26:53.542 INFO:teuthology.orchestra.run.smithi001.stdout: "read_tier": -1, 2024-09-13T11:26:53.542 INFO:teuthology.orchestra.run.smithi001.stdout: "write_tier": -1, 2024-09-13T11:26:53.542 INFO:teuthology.orchestra.run.smithi001.stdout: "cache_mode": "none", 2024-09-13T11:26:53.543 INFO:teuthology.orchestra.run.smithi001.stdout: "target_max_bytes": 0, 2024-09-13T11:26:53.543 INFO:teuthology.orchestra.run.smithi001.stdout: "target_max_objects": 0, 2024-09-13T11:26:53.543 INFO:teuthology.orchestra.run.smithi001.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-13T11:26:53.543 INFO:teuthology.orchestra.run.smithi001.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-13T11:26:53.543 INFO:teuthology.orchestra.run.smithi001.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-13T11:26:53.543 INFO:teuthology.orchestra.run.smithi001.stdout: "cache_min_flush_age": 0, 2024-09-13T11:26:53.543 INFO:teuthology.orchestra.run.smithi001.stdout: "cache_min_evict_age": 0, 2024-09-13T11:26:53.543 INFO:teuthology.orchestra.run.smithi001.stdout: "erasure_code_profile": "", 2024-09-13T11:26:53.543 INFO:teuthology.orchestra.run.smithi001.stdout: "hit_set_params": { 2024-09-13T11:26:53.543 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "none" 2024-09-13T11:26:53.543 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:53.543 INFO:teuthology.orchestra.run.smithi001.stdout: "hit_set_period": 0, 2024-09-13T11:26:53.544 INFO:teuthology.orchestra.run.smithi001.stdout: "hit_set_count": 0, 2024-09-13T11:26:53.544 INFO:teuthology.orchestra.run.smithi001.stdout: "use_gmt_hitset": true, 2024-09-13T11:26:53.544 INFO:teuthology.orchestra.run.smithi001.stdout: "min_read_recency_for_promote": 0, 2024-09-13T11:26:53.544 INFO:teuthology.orchestra.run.smithi001.stdout: "min_write_recency_for_promote": 0, 2024-09-13T11:26:53.544 INFO:teuthology.orchestra.run.smithi001.stdout: "hit_set_grade_decay_rate": 0, 2024-09-13T11:26:53.544 INFO:teuthology.orchestra.run.smithi001.stdout: "hit_set_search_last_n": 0, 2024-09-13T11:26:53.544 INFO:teuthology.orchestra.run.smithi001.stdout: "grade_table": [], 2024-09-13T11:26:53.544 INFO:teuthology.orchestra.run.smithi001.stdout: "stripe_width": 0, 2024-09-13T11:26:53.544 INFO:teuthology.orchestra.run.smithi001.stdout: "expected_num_objects": 0, 2024-09-13T11:26:53.544 INFO:teuthology.orchestra.run.smithi001.stdout: "fast_read": false, 2024-09-13T11:26:53.544 INFO:teuthology.orchestra.run.smithi001.stdout: "options": { 2024-09-13T11:26:53.545 INFO:teuthology.orchestra.run.smithi001.stdout: "pg_num_max": 32, 2024-09-13T11:26:53.545 INFO:teuthology.orchestra.run.smithi001.stdout: "pg_num_min": 1 2024-09-13T11:26:53.545 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:53.545 INFO:teuthology.orchestra.run.smithi001.stdout: "application_metadata": { 2024-09-13T11:26:53.545 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": {} 2024-09-13T11:26:53.545 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:53.545 INFO:teuthology.orchestra.run.smithi001.stdout: "read_balance": { 2024-09-13T11:26:53.545 INFO:teuthology.orchestra.run.smithi001.stdout: "score_type": "Fair distribution", 2024-09-13T11:26:53.545 INFO:teuthology.orchestra.run.smithi001.stdout: "score_acting": 15.380000114440918, 2024-09-13T11:26:53.545 INFO:teuthology.orchestra.run.smithi001.stdout: "score_stable": 15.380000114440918, 2024-09-13T11:26:53.545 INFO:teuthology.orchestra.run.smithi001.stdout: "optimal_score": 0.12999999523162842, 2024-09-13T11:26:53.546 INFO:teuthology.orchestra.run.smithi001.stdout: "raw_score_acting": 2, 2024-09-13T11:26:53.546 INFO:teuthology.orchestra.run.smithi001.stdout: "raw_score_stable": 2, 2024-09-13T11:26:53.546 INFO:teuthology.orchestra.run.smithi001.stdout: "primary_affinity_weighted": 0.92666339874267578, 2024-09-13T11:26:53.546 INFO:teuthology.orchestra.run.smithi001.stdout: "average_primary_affinity": 0.99000000953674316, 2024-09-13T11:26:53.546 INFO:teuthology.orchestra.run.smithi001.stdout: "average_primary_affinity_weighted": 0.93000000715255737 2024-09-13T11:26:53.546 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:53.546 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:53.546 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:53.546 INFO:teuthology.orchestra.run.smithi001.stdout: "pool": 2, 2024-09-13T11:26:53.546 INFO:teuthology.orchestra.run.smithi001.stdout: "pool_name": "rbd", 2024-09-13T11:26:53.546 INFO:teuthology.orchestra.run.smithi001.stdout: "create_time": "2024-09-13T11:22:12.235549+0000", 2024-09-13T11:26:53.547 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 8193, 2024-09-13T11:26:53.547 INFO:teuthology.orchestra.run.smithi001.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-09-13T11:26:53.547 INFO:teuthology.orchestra.run.smithi001.stdout: "type": 1, 2024-09-13T11:26:53.547 INFO:teuthology.orchestra.run.smithi001.stdout: "size": 2, 2024-09-13T11:26:53.547 INFO:teuthology.orchestra.run.smithi001.stdout: "min_size": 2, 2024-09-13T11:26:53.547 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_rule": 0, 2024-09-13T11:26:53.547 INFO:teuthology.orchestra.run.smithi001.stdout: "peering_crush_bucket_count": 0, 2024-09-13T11:26:53.547 INFO:teuthology.orchestra.run.smithi001.stdout: "peering_crush_bucket_target": 0, 2024-09-13T11:26:53.547 INFO:teuthology.orchestra.run.smithi001.stdout: "peering_crush_bucket_barrier": 0, 2024-09-13T11:26:53.547 INFO:teuthology.orchestra.run.smithi001.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-13T11:26:53.547 INFO:teuthology.orchestra.run.smithi001.stdout: "is_stretch_pool": false, 2024-09-13T11:26:53.547 INFO:teuthology.orchestra.run.smithi001.stdout: "object_hash": 2, 2024-09-13T11:26:53.548 INFO:teuthology.orchestra.run.smithi001.stdout: "pg_autoscale_mode": "off", 2024-09-13T11:26:53.548 INFO:teuthology.orchestra.run.smithi001.stdout: "pg_num": 8, 2024-09-13T11:26:53.548 INFO:teuthology.orchestra.run.smithi001.stdout: "pg_placement_num": 8, 2024-09-13T11:26:53.548 INFO:teuthology.orchestra.run.smithi001.stdout: "pg_placement_num_target": 8, 2024-09-13T11:26:53.548 INFO:teuthology.orchestra.run.smithi001.stdout: "pg_num_target": 8, 2024-09-13T11:26:53.548 INFO:teuthology.orchestra.run.smithi001.stdout: "pg_num_pending": 8, 2024-09-13T11:26:53.548 INFO:teuthology.orchestra.run.smithi001.stdout: "last_pg_merge_meta": { 2024-09-13T11:26:53.548 INFO:teuthology.orchestra.run.smithi001.stdout: "source_pgid": "0.0", 2024-09-13T11:26:53.548 INFO:teuthology.orchestra.run.smithi001.stdout: "ready_epoch": 0, 2024-09-13T11:26:53.548 INFO:teuthology.orchestra.run.smithi001.stdout: "last_epoch_started": 0, 2024-09-13T11:26:53.548 INFO:teuthology.orchestra.run.smithi001.stdout: "last_epoch_clean": 0, 2024-09-13T11:26:53.548 INFO:teuthology.orchestra.run.smithi001.stdout: "source_version": "0'0", 2024-09-13T11:26:53.549 INFO:teuthology.orchestra.run.smithi001.stdout: "target_version": "0'0" 2024-09-13T11:26:53.549 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:53.549 INFO:teuthology.orchestra.run.smithi001.stdout: "last_change": "28", 2024-09-13T11:26:53.549 INFO:teuthology.orchestra.run.smithi001.stdout: "last_force_op_resend": "0", 2024-09-13T11:26:53.549 INFO:teuthology.orchestra.run.smithi001.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-13T11:26:53.549 INFO:teuthology.orchestra.run.smithi001.stdout: "last_force_op_resend_preluminous": "0", 2024-09-13T11:26:53.549 INFO:teuthology.orchestra.run.smithi001.stdout: "auid": 0, 2024-09-13T11:26:53.549 INFO:teuthology.orchestra.run.smithi001.stdout: "snap_mode": "selfmanaged", 2024-09-13T11:26:53.551 INFO:teuthology.orchestra.run.smithi001.stdout: "snap_seq": 2, 2024-09-13T11:26:53.551 INFO:teuthology.orchestra.run.smithi001.stdout: "snap_epoch": 28, 2024-09-13T11:26:53.551 INFO:teuthology.orchestra.run.smithi001.stdout: "pool_snaps": [], 2024-09-13T11:26:53.551 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_snaps": "[]", 2024-09-13T11:26:53.551 INFO:teuthology.orchestra.run.smithi001.stdout: "quota_max_bytes": 0, 2024-09-13T11:26:53.551 INFO:teuthology.orchestra.run.smithi001.stdout: "quota_max_objects": 0, 2024-09-13T11:26:53.551 INFO:teuthology.orchestra.run.smithi001.stdout: "tiers": [], 2024-09-13T11:26:53.551 INFO:teuthology.orchestra.run.smithi001.stdout: "tier_of": -1, 2024-09-13T11:26:53.551 INFO:teuthology.orchestra.run.smithi001.stdout: "read_tier": -1, 2024-09-13T11:26:53.551 INFO:teuthology.orchestra.run.smithi001.stdout: "write_tier": -1, 2024-09-13T11:26:53.552 INFO:teuthology.orchestra.run.smithi001.stdout: "cache_mode": "none", 2024-09-13T11:26:53.552 INFO:teuthology.orchestra.run.smithi001.stdout: "target_max_bytes": 0, 2024-09-13T11:26:53.552 INFO:teuthology.orchestra.run.smithi001.stdout: "target_max_objects": 0, 2024-09-13T11:26:53.552 INFO:teuthology.orchestra.run.smithi001.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-13T11:26:53.552 INFO:teuthology.orchestra.run.smithi001.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-13T11:26:53.552 INFO:teuthology.orchestra.run.smithi001.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-13T11:26:53.552 INFO:teuthology.orchestra.run.smithi001.stdout: "cache_min_flush_age": 0, 2024-09-13T11:26:53.552 INFO:teuthology.orchestra.run.smithi001.stdout: "cache_min_evict_age": 0, 2024-09-13T11:26:53.552 INFO:teuthology.orchestra.run.smithi001.stdout: "erasure_code_profile": "", 2024-09-13T11:26:53.552 INFO:teuthology.orchestra.run.smithi001.stdout: "hit_set_params": { 2024-09-13T11:26:53.552 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "none" 2024-09-13T11:26:53.552 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:53.553 INFO:teuthology.orchestra.run.smithi001.stdout: "hit_set_period": 0, 2024-09-13T11:26:53.553 INFO:teuthology.orchestra.run.smithi001.stdout: "hit_set_count": 0, 2024-09-13T11:26:53.553 INFO:teuthology.orchestra.run.smithi001.stdout: "use_gmt_hitset": true, 2024-09-13T11:26:53.553 INFO:teuthology.orchestra.run.smithi001.stdout: "min_read_recency_for_promote": 0, 2024-09-13T11:26:53.553 INFO:teuthology.orchestra.run.smithi001.stdout: "min_write_recency_for_promote": 0, 2024-09-13T11:26:53.553 INFO:teuthology.orchestra.run.smithi001.stdout: "hit_set_grade_decay_rate": 0, 2024-09-13T11:26:53.553 INFO:teuthology.orchestra.run.smithi001.stdout: "hit_set_search_last_n": 0, 2024-09-13T11:26:53.553 INFO:teuthology.orchestra.run.smithi001.stdout: "grade_table": [], 2024-09-13T11:26:53.553 INFO:teuthology.orchestra.run.smithi001.stdout: "stripe_width": 0, 2024-09-13T11:26:53.553 INFO:teuthology.orchestra.run.smithi001.stdout: "expected_num_objects": 0, 2024-09-13T11:26:53.553 INFO:teuthology.orchestra.run.smithi001.stdout: "fast_read": false, 2024-09-13T11:26:53.553 INFO:teuthology.orchestra.run.smithi001.stdout: "options": {}, 2024-09-13T11:26:53.553 INFO:teuthology.orchestra.run.smithi001.stdout: "application_metadata": { 2024-09-13T11:26:53.553 INFO:teuthology.orchestra.run.smithi001.stdout: "rbd": {} 2024-09-13T11:26:53.554 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:53.554 INFO:teuthology.orchestra.run.smithi001.stdout: "read_balance": { 2024-09-13T11:26:53.554 INFO:teuthology.orchestra.run.smithi001.stdout: "score_type": "Fair distribution", 2024-09-13T11:26:53.554 INFO:teuthology.orchestra.run.smithi001.stdout: "score_acting": 3.75, 2024-09-13T11:26:53.554 INFO:teuthology.orchestra.run.smithi001.stdout: "score_stable": 3.75, 2024-09-13T11:26:53.554 INFO:teuthology.orchestra.run.smithi001.stdout: "optimal_score": 0.60000002384185791, 2024-09-13T11:26:53.554 INFO:teuthology.orchestra.run.smithi001.stdout: "raw_score_acting": 2.25, 2024-09-13T11:26:53.554 INFO:teuthology.orchestra.run.smithi001.stdout: "raw_score_stable": 2.25, 2024-09-13T11:26:53.554 INFO:teuthology.orchestra.run.smithi001.stdout: "primary_affinity_weighted": 0.92666339874267578, 2024-09-13T11:26:53.554 INFO:teuthology.orchestra.run.smithi001.stdout: "average_primary_affinity": 0.99000000953674316, 2024-09-13T11:26:53.554 INFO:teuthology.orchestra.run.smithi001.stdout: "average_primary_affinity_weighted": 0.93000000715255737 2024-09-13T11:26:53.554 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:53.554 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:53.555 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:53.555 INFO:teuthology.orchestra.run.smithi001.stdout: "pool": 3, 2024-09-13T11:26:53.555 INFO:teuthology.orchestra.run.smithi001.stdout: "pool_name": "unique_pool_0", 2024-09-13T11:26:53.555 INFO:teuthology.orchestra.run.smithi001.stdout: "create_time": "2024-09-13T11:22:24.305521+0000", 2024-09-13T11:26:53.555 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-09-13T11:26:53.555 INFO:teuthology.orchestra.run.smithi001.stdout: "flags_names": "hashpspool", 2024-09-13T11:26:53.555 INFO:teuthology.orchestra.run.smithi001.stdout: "type": 1, 2024-09-13T11:26:53.555 INFO:teuthology.orchestra.run.smithi001.stdout: "size": 2, 2024-09-13T11:26:53.555 INFO:teuthology.orchestra.run.smithi001.stdout: "min_size": 2, 2024-09-13T11:26:53.555 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_rule": 0, 2024-09-13T11:26:53.555 INFO:teuthology.orchestra.run.smithi001.stdout: "peering_crush_bucket_count": 0, 2024-09-13T11:26:53.555 INFO:teuthology.orchestra.run.smithi001.stdout: "peering_crush_bucket_target": 0, 2024-09-13T11:26:53.555 INFO:teuthology.orchestra.run.smithi001.stdout: "peering_crush_bucket_barrier": 0, 2024-09-13T11:26:53.556 INFO:teuthology.orchestra.run.smithi001.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-13T11:26:53.556 INFO:teuthology.orchestra.run.smithi001.stdout: "is_stretch_pool": false, 2024-09-13T11:26:53.556 INFO:teuthology.orchestra.run.smithi001.stdout: "object_hash": 2, 2024-09-13T11:26:53.556 INFO:teuthology.orchestra.run.smithi001.stdout: "pg_autoscale_mode": "off", 2024-09-13T11:26:53.556 INFO:teuthology.orchestra.run.smithi001.stdout: "pg_num": 56, 2024-09-13T11:26:53.556 INFO:teuthology.orchestra.run.smithi001.stdout: "pg_placement_num": 56, 2024-09-13T11:26:53.556 INFO:teuthology.orchestra.run.smithi001.stdout: "pg_placement_num_target": 56, 2024-09-13T11:26:53.556 INFO:teuthology.orchestra.run.smithi001.stdout: "pg_num_target": 56, 2024-09-13T11:26:53.556 INFO:teuthology.orchestra.run.smithi001.stdout: "pg_num_pending": 56, 2024-09-13T11:26:53.556 INFO:teuthology.orchestra.run.smithi001.stdout: "last_pg_merge_meta": { 2024-09-13T11:26:53.556 INFO:teuthology.orchestra.run.smithi001.stdout: "source_pgid": "0.0", 2024-09-13T11:26:53.556 INFO:teuthology.orchestra.run.smithi001.stdout: "ready_epoch": 0, 2024-09-13T11:26:53.557 INFO:teuthology.orchestra.run.smithi001.stdout: "last_epoch_started": 0, 2024-09-13T11:26:53.557 INFO:teuthology.orchestra.run.smithi001.stdout: "last_epoch_clean": 0, 2024-09-13T11:26:53.557 INFO:teuthology.orchestra.run.smithi001.stdout: "source_version": "0'0", 2024-09-13T11:26:53.557 INFO:teuthology.orchestra.run.smithi001.stdout: "target_version": "0'0" 2024-09-13T11:26:53.557 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:53.557 INFO:teuthology.orchestra.run.smithi001.stdout: "last_change": "226", 2024-09-13T11:26:53.557 INFO:teuthology.orchestra.run.smithi001.stdout: "last_force_op_resend": "0", 2024-09-13T11:26:53.557 INFO:teuthology.orchestra.run.smithi001.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-13T11:26:53.557 INFO:teuthology.orchestra.run.smithi001.stdout: "last_force_op_resend_preluminous": "224", 2024-09-13T11:26:53.557 INFO:teuthology.orchestra.run.smithi001.stdout: "auid": 0, 2024-09-13T11:26:53.557 INFO:teuthology.orchestra.run.smithi001.stdout: "snap_mode": "selfmanaged", 2024-09-13T11:26:53.558 INFO:teuthology.orchestra.run.smithi001.stdout: "snap_seq": 0, 2024-09-13T11:26:53.558 INFO:teuthology.orchestra.run.smithi001.stdout: "snap_epoch": 0, 2024-09-13T11:26:53.558 INFO:teuthology.orchestra.run.smithi001.stdout: "pool_snaps": [], 2024-09-13T11:26:53.558 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_snaps": "[]", 2024-09-13T11:26:53.558 INFO:teuthology.orchestra.run.smithi001.stdout: "quota_max_bytes": 0, 2024-09-13T11:26:53.558 INFO:teuthology.orchestra.run.smithi001.stdout: "quota_max_objects": 0, 2024-09-13T11:26:53.558 INFO:teuthology.orchestra.run.smithi001.stdout: "tiers": [], 2024-09-13T11:26:53.558 INFO:teuthology.orchestra.run.smithi001.stdout: "tier_of": -1, 2024-09-13T11:26:53.558 INFO:teuthology.orchestra.run.smithi001.stdout: "read_tier": -1, 2024-09-13T11:26:53.558 INFO:teuthology.orchestra.run.smithi001.stdout: "write_tier": -1, 2024-09-13T11:26:53.558 INFO:teuthology.orchestra.run.smithi001.stdout: "cache_mode": "none", 2024-09-13T11:26:53.558 INFO:teuthology.orchestra.run.smithi001.stdout: "target_max_bytes": 0, 2024-09-13T11:26:53.558 INFO:teuthology.orchestra.run.smithi001.stdout: "target_max_objects": 0, 2024-09-13T11:26:53.559 INFO:teuthology.orchestra.run.smithi001.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-13T11:26:53.559 INFO:teuthology.orchestra.run.smithi001.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-13T11:26:53.559 INFO:teuthology.orchestra.run.smithi001.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-13T11:26:53.559 INFO:teuthology.orchestra.run.smithi001.stdout: "cache_min_flush_age": 0, 2024-09-13T11:26:53.559 INFO:teuthology.orchestra.run.smithi001.stdout: "cache_min_evict_age": 0, 2024-09-13T11:26:53.559 INFO:teuthology.orchestra.run.smithi001.stdout: "erasure_code_profile": "", 2024-09-13T11:26:53.559 INFO:teuthology.orchestra.run.smithi001.stdout: "hit_set_params": { 2024-09-13T11:26:53.559 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "none" 2024-09-13T11:26:53.559 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:53.559 INFO:teuthology.orchestra.run.smithi001.stdout: "hit_set_period": 0, 2024-09-13T11:26:53.559 INFO:teuthology.orchestra.run.smithi001.stdout: "hit_set_count": 0, 2024-09-13T11:26:53.559 INFO:teuthology.orchestra.run.smithi001.stdout: "use_gmt_hitset": true, 2024-09-13T11:26:53.559 INFO:teuthology.orchestra.run.smithi001.stdout: "min_read_recency_for_promote": 0, 2024-09-13T11:26:53.560 INFO:teuthology.orchestra.run.smithi001.stdout: "min_write_recency_for_promote": 0, 2024-09-13T11:26:53.560 INFO:teuthology.orchestra.run.smithi001.stdout: "hit_set_grade_decay_rate": 0, 2024-09-13T11:26:53.560 INFO:teuthology.orchestra.run.smithi001.stdout: "hit_set_search_last_n": 0, 2024-09-13T11:26:53.560 INFO:teuthology.orchestra.run.smithi001.stdout: "grade_table": [], 2024-09-13T11:26:53.560 INFO:teuthology.orchestra.run.smithi001.stdout: "stripe_width": 0, 2024-09-13T11:26:53.560 INFO:teuthology.orchestra.run.smithi001.stdout: "expected_num_objects": 0, 2024-09-13T11:26:53.560 INFO:teuthology.orchestra.run.smithi001.stdout: "fast_read": false, 2024-09-13T11:26:53.560 INFO:teuthology.orchestra.run.smithi001.stdout: "options": {}, 2024-09-13T11:26:53.560 INFO:teuthology.orchestra.run.smithi001.stdout: "application_metadata": { 2024-09-13T11:26:53.560 INFO:teuthology.orchestra.run.smithi001.stdout: "rados": {} 2024-09-13T11:26:53.560 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:53.560 INFO:teuthology.orchestra.run.smithi001.stdout: "read_balance": { 2024-09-13T11:26:53.560 INFO:teuthology.orchestra.run.smithi001.stdout: "score_type": "Fair distribution", 2024-09-13T11:26:53.561 INFO:teuthology.orchestra.run.smithi001.stdout: "score_acting": 1.8700000047683716, 2024-09-13T11:26:53.561 INFO:teuthology.orchestra.run.smithi001.stdout: "score_stable": 2.1400001049041748, 2024-09-13T11:26:53.561 INFO:teuthology.orchestra.run.smithi001.stdout: "optimal_score": 1.0700000524520874, 2024-09-13T11:26:53.561 INFO:teuthology.orchestra.run.smithi001.stdout: "raw_score_acting": 2, 2024-09-13T11:26:53.561 INFO:teuthology.orchestra.run.smithi001.stdout: "raw_score_stable": 2.2899999618530273, 2024-09-13T11:26:53.561 INFO:teuthology.orchestra.run.smithi001.stdout: "primary_affinity_weighted": 0.92666339874267578, 2024-09-13T11:26:53.561 INFO:teuthology.orchestra.run.smithi001.stdout: "average_primary_affinity": 0.99000000953674316, 2024-09-13T11:26:53.561 INFO:teuthology.orchestra.run.smithi001.stdout: "average_primary_affinity_weighted": 0.93000000715255737 2024-09-13T11:26:53.561 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:53.561 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:53.561 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-09-13T11:26:53.561 INFO:teuthology.orchestra.run.smithi001.stdout: "osds": [ 2024-09-13T11:26:53.561 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:53.562 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": 0, 2024-09-13T11:26:53.562 INFO:teuthology.orchestra.run.smithi001.stdout: "uuid": "0fc12b3c-e0de-4546-b935-db77efe7c567", 2024-09-13T11:26:53.562 INFO:teuthology.orchestra.run.smithi001.stdout: "up": 1, 2024-09-13T11:26:53.562 INFO:teuthology.orchestra.run.smithi001.stdout: "in": 1, 2024-09-13T11:26:53.562 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 1, 2024-09-13T11:26:53.562 INFO:teuthology.orchestra.run.smithi001.stdout: "primary_affinity": 1, 2024-09-13T11:26:53.562 INFO:teuthology.orchestra.run.smithi001.stdout: "last_clean_begin": 0, 2024-09-13T11:26:53.562 INFO:teuthology.orchestra.run.smithi001.stdout: "last_clean_end": 0, 2024-09-13T11:26:53.562 INFO:teuthology.orchestra.run.smithi001.stdout: "up_from": 22, 2024-09-13T11:26:53.562 INFO:teuthology.orchestra.run.smithi001.stdout: "up_thru": 227, 2024-09-13T11:26:53.562 INFO:teuthology.orchestra.run.smithi001.stdout: "down_at": 0, 2024-09-13T11:26:53.562 INFO:teuthology.orchestra.run.smithi001.stdout: "lost_at": 0, 2024-09-13T11:26:53.563 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-09-13T11:26:53.563 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:26:53.563 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:53.563 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:26:53.563 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6800", 2024-09-13T11:26:53.563 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 1544776477 2024-09-13T11:26:53.563 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:53.563 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:53.563 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:53.563 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_addrs": { 2024-09-13T11:26:53.563 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:26:53.563 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:53.563 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:26:53.563 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6801", 2024-09-13T11:26:53.563 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 1544776477 2024-09-13T11:26:53.564 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:53.564 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:53.564 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:53.564 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_back_addrs": { 2024-09-13T11:26:53.564 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:26:53.564 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:53.564 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:26:53.564 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6803", 2024-09-13T11:26:53.564 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 1544776477 2024-09-13T11:26:53.564 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:53.564 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:53.564 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:53.564 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_front_addrs": { 2024-09-13T11:26:53.564 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:26:53.565 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:53.565 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:26:53.565 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6802", 2024-09-13T11:26:53.565 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 1544776477 2024-09-13T11:26:53.565 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:53.565 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:53.565 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:53.565 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6800/1544776477", 2024-09-13T11:26:53.565 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_addr": "172.21.15.1:6801/1544776477", 2024-09-13T11:26:53.568 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_back_addr": "172.21.15.1:6803/1544776477", 2024-09-13T11:26:53.568 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_front_addr": "172.21.15.1:6802/1544776477", 2024-09-13T11:26:53.568 INFO:teuthology.orchestra.run.smithi001.stdout: "state": [ 2024-09-13T11:26:53.568 INFO:teuthology.orchestra.run.smithi001.stdout: "exists", 2024-09-13T11:26:53.568 INFO:teuthology.orchestra.run.smithi001.stdout: "up" 2024-09-13T11:26:53.569 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:53.569 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:53.569 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:53.569 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": 1, 2024-09-13T11:26:53.569 INFO:teuthology.orchestra.run.smithi001.stdout: "uuid": "b77fe331-cc92-4adb-b146-af5c380f6d4f", 2024-09-13T11:26:53.569 INFO:teuthology.orchestra.run.smithi001.stdout: "up": 1, 2024-09-13T11:26:53.569 INFO:teuthology.orchestra.run.smithi001.stdout: "in": 1, 2024-09-13T11:26:53.569 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 1, 2024-09-13T11:26:53.569 INFO:teuthology.orchestra.run.smithi001.stdout: "primary_affinity": 1, 2024-09-13T11:26:53.569 INFO:teuthology.orchestra.run.smithi001.stdout: "last_clean_begin": 23, 2024-09-13T11:26:53.569 INFO:teuthology.orchestra.run.smithi001.stdout: "last_clean_end": 29, 2024-09-13T11:26:53.569 INFO:teuthology.orchestra.run.smithi001.stdout: "up_from": 40, 2024-09-13T11:26:53.569 INFO:teuthology.orchestra.run.smithi001.stdout: "up_thru": 226, 2024-09-13T11:26:53.570 INFO:teuthology.orchestra.run.smithi001.stdout: "down_at": 30, 2024-09-13T11:26:53.570 INFO:teuthology.orchestra.run.smithi001.stdout: "lost_at": 0, 2024-09-13T11:26:53.570 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-09-13T11:26:53.570 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:26:53.570 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:53.570 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:26:53.570 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.92:6800", 2024-09-13T11:26:53.570 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 3591430847 2024-09-13T11:26:53.571 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:53.571 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:53.571 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:53.571 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_addrs": { 2024-09-13T11:26:53.571 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:26:53.571 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:53.571 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:26:53.571 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.92:6801", 2024-09-13T11:26:53.571 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 3591430847 2024-09-13T11:26:53.571 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:53.571 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:53.571 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:53.571 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_back_addrs": { 2024-09-13T11:26:53.572 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:26:53.572 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:53.572 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:26:53.572 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.92:6803", 2024-09-13T11:26:53.572 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 3591430847 2024-09-13T11:26:53.572 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:53.572 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:53.572 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:53.572 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_front_addrs": { 2024-09-13T11:26:53.572 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:26:53.572 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:53.572 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:26:53.572 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.92:6802", 2024-09-13T11:26:53.572 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 3591430847 2024-09-13T11:26:53.573 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:53.573 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:53.573 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:53.573 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.92:6800/3591430847", 2024-09-13T11:26:53.573 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_addr": "172.21.15.92:6801/3591430847", 2024-09-13T11:26:53.573 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_back_addr": "172.21.15.92:6803/3591430847", 2024-09-13T11:26:53.573 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_front_addr": "172.21.15.92:6802/3591430847", 2024-09-13T11:26:53.573 INFO:teuthology.orchestra.run.smithi001.stdout: "state": [ 2024-09-13T11:26:53.573 INFO:teuthology.orchestra.run.smithi001.stdout: "exists", 2024-09-13T11:26:53.573 INFO:teuthology.orchestra.run.smithi001.stdout: "up" 2024-09-13T11:26:53.573 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:53.573 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:53.574 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:53.574 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": 2, 2024-09-13T11:26:53.574 INFO:teuthology.orchestra.run.smithi001.stdout: "uuid": "c9d031c8-e885-4229-affa-08e1729d00e8", 2024-09-13T11:26:53.574 INFO:teuthology.orchestra.run.smithi001.stdout: "up": 1, 2024-09-13T11:26:53.574 INFO:teuthology.orchestra.run.smithi001.stdout: "in": 1, 2024-09-13T11:26:53.574 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 1, 2024-09-13T11:26:53.574 INFO:teuthology.orchestra.run.smithi001.stdout: "primary_affinity": 1, 2024-09-13T11:26:53.574 INFO:teuthology.orchestra.run.smithi001.stdout: "last_clean_begin": 0, 2024-09-13T11:26:53.574 INFO:teuthology.orchestra.run.smithi001.stdout: "last_clean_end": 0, 2024-09-13T11:26:53.574 INFO:teuthology.orchestra.run.smithi001.stdout: "up_from": 22, 2024-09-13T11:26:53.574 INFO:teuthology.orchestra.run.smithi001.stdout: "up_thru": 226, 2024-09-13T11:26:53.574 INFO:teuthology.orchestra.run.smithi001.stdout: "down_at": 0, 2024-09-13T11:26:53.575 INFO:teuthology.orchestra.run.smithi001.stdout: "lost_at": 0, 2024-09-13T11:26:53.575 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-09-13T11:26:53.575 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:26:53.575 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:53.575 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:26:53.575 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.162:6812", 2024-09-13T11:26:53.575 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 2712148101 2024-09-13T11:26:53.575 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:53.575 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:53.575 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:53.575 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_addrs": { 2024-09-13T11:26:53.575 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:26:53.576 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:53.576 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:26:53.576 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.162:6813", 2024-09-13T11:26:53.576 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 2712148101 2024-09-13T11:26:53.576 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:53.576 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:53.576 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:53.576 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_back_addrs": { 2024-09-13T11:26:53.576 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:26:53.576 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:53.576 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:26:53.576 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.162:6815", 2024-09-13T11:26:53.577 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 2712148101 2024-09-13T11:26:53.577 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:53.577 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:53.577 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:53.577 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_front_addrs": { 2024-09-13T11:26:53.577 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:26:53.577 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:53.577 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:26:53.577 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.162:6814", 2024-09-13T11:26:53.577 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 2712148101 2024-09-13T11:26:53.577 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:53.577 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:53.578 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:53.578 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.162:6812/2712148101", 2024-09-13T11:26:53.578 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_addr": "172.21.15.162:6813/2712148101", 2024-09-13T11:26:53.578 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_back_addr": "172.21.15.162:6815/2712148101", 2024-09-13T11:26:53.578 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_front_addr": "172.21.15.162:6814/2712148101", 2024-09-13T11:26:53.578 INFO:teuthology.orchestra.run.smithi001.stdout: "state": [ 2024-09-13T11:26:53.578 INFO:teuthology.orchestra.run.smithi001.stdout: "exists", 2024-09-13T11:26:53.578 INFO:teuthology.orchestra.run.smithi001.stdout: "up" 2024-09-13T11:26:53.578 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:53.578 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:53.578 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:53.578 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": 3, 2024-09-13T11:26:53.579 INFO:teuthology.orchestra.run.smithi001.stdout: "uuid": "bf50d71a-6c3a-4fbc-a407-cb81bad14a7b", 2024-09-13T11:26:53.579 INFO:teuthology.orchestra.run.smithi001.stdout: "up": 1, 2024-09-13T11:26:53.579 INFO:teuthology.orchestra.run.smithi001.stdout: "in": 1, 2024-09-13T11:26:53.579 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 1, 2024-09-13T11:26:53.579 INFO:teuthology.orchestra.run.smithi001.stdout: "primary_affinity": 1, 2024-09-13T11:26:53.579 INFO:teuthology.orchestra.run.smithi001.stdout: "last_clean_begin": 0, 2024-09-13T11:26:53.579 INFO:teuthology.orchestra.run.smithi001.stdout: "last_clean_end": 0, 2024-09-13T11:26:53.579 INFO:teuthology.orchestra.run.smithi001.stdout: "up_from": 21, 2024-09-13T11:26:53.579 INFO:teuthology.orchestra.run.smithi001.stdout: "up_thru": 227, 2024-09-13T11:26:53.579 INFO:teuthology.orchestra.run.smithi001.stdout: "down_at": 0, 2024-09-13T11:26:53.579 INFO:teuthology.orchestra.run.smithi001.stdout: "lost_at": 0, 2024-09-13T11:26:53.579 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-09-13T11:26:53.580 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:26:53.580 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:53.580 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:26:53.580 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.177:6801", 2024-09-13T11:26:53.580 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 3367751690 2024-09-13T11:26:53.580 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:53.580 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:53.580 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:53.580 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_addrs": { 2024-09-13T11:26:53.580 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:26:53.580 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:53.580 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:26:53.581 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.177:6802", 2024-09-13T11:26:53.581 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 3367751690 2024-09-13T11:26:53.581 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:53.581 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:53.581 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:53.581 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_back_addrs": { 2024-09-13T11:26:53.581 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:26:53.581 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:53.581 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:26:53.581 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.177:6804", 2024-09-13T11:26:53.581 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 3367751690 2024-09-13T11:26:53.581 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:53.582 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:53.582 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:53.582 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_front_addrs": { 2024-09-13T11:26:53.582 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:26:53.582 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:53.582 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:26:53.582 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.177:6803", 2024-09-13T11:26:53.582 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 3367751690 2024-09-13T11:26:53.582 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:53.582 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:53.582 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:53.582 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.177:6801/3367751690", 2024-09-13T11:26:53.582 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_addr": "172.21.15.177:6802/3367751690", 2024-09-13T11:26:53.582 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_back_addr": "172.21.15.177:6804/3367751690", 2024-09-13T11:26:53.583 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_front_addr": "172.21.15.177:6803/3367751690", 2024-09-13T11:26:53.583 INFO:teuthology.orchestra.run.smithi001.stdout: "state": [ 2024-09-13T11:26:53.583 INFO:teuthology.orchestra.run.smithi001.stdout: "exists", 2024-09-13T11:26:53.583 INFO:teuthology.orchestra.run.smithi001.stdout: "up" 2024-09-13T11:26:53.583 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:53.583 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:53.583 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:53.583 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": 4, 2024-09-13T11:26:53.583 INFO:teuthology.orchestra.run.smithi001.stdout: "uuid": "d1fe3f63-df3f-4a73-a1e2-8d7c8a3f1fa0", 2024-09-13T11:26:53.583 INFO:teuthology.orchestra.run.smithi001.stdout: "up": 1, 2024-09-13T11:26:53.583 INFO:teuthology.orchestra.run.smithi001.stdout: "in": 1, 2024-09-13T11:26:53.583 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 1, 2024-09-13T11:26:53.584 INFO:teuthology.orchestra.run.smithi001.stdout: "primary_affinity": 1, 2024-09-13T11:26:53.584 INFO:teuthology.orchestra.run.smithi001.stdout: "last_clean_begin": 0, 2024-09-13T11:26:53.584 INFO:teuthology.orchestra.run.smithi001.stdout: "last_clean_end": 0, 2024-09-13T11:26:53.584 INFO:teuthology.orchestra.run.smithi001.stdout: "up_from": 22, 2024-09-13T11:26:53.584 INFO:teuthology.orchestra.run.smithi001.stdout: "up_thru": 227, 2024-09-13T11:26:53.584 INFO:teuthology.orchestra.run.smithi001.stdout: "down_at": 0, 2024-09-13T11:26:53.584 INFO:teuthology.orchestra.run.smithi001.stdout: "lost_at": 0, 2024-09-13T11:26:53.584 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-09-13T11:26:53.584 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:26:53.584 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:53.584 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:26:53.584 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6808", 2024-09-13T11:26:53.584 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 3648540926 2024-09-13T11:26:53.585 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:53.585 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:53.585 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:53.585 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_addrs": { 2024-09-13T11:26:53.585 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:26:53.585 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:53.585 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:26:53.585 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6809", 2024-09-13T11:26:53.585 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 3648540926 2024-09-13T11:26:53.585 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:53.585 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:53.585 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:53.586 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_back_addrs": { 2024-09-13T11:26:53.586 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:26:53.586 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:53.586 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:26:53.586 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6811", 2024-09-13T11:26:53.586 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 3648540926 2024-09-13T11:26:53.586 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:53.586 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:53.586 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:53.586 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_front_addrs": { 2024-09-13T11:26:53.586 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:26:53.586 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:53.587 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:26:53.587 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6810", 2024-09-13T11:26:53.587 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 3648540926 2024-09-13T11:26:53.587 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:53.587 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:53.587 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:53.587 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6808/3648540926", 2024-09-13T11:26:53.587 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_addr": "172.21.15.1:6809/3648540926", 2024-09-13T11:26:53.587 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_back_addr": "172.21.15.1:6811/3648540926", 2024-09-13T11:26:53.587 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_front_addr": "172.21.15.1:6810/3648540926", 2024-09-13T11:26:53.587 INFO:teuthology.orchestra.run.smithi001.stdout: "state": [ 2024-09-13T11:26:53.587 INFO:teuthology.orchestra.run.smithi001.stdout: "exists", 2024-09-13T11:26:53.588 INFO:teuthology.orchestra.run.smithi001.stdout: "up" 2024-09-13T11:26:53.588 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:53.588 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:53.588 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:53.588 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": 5, 2024-09-13T11:26:53.588 INFO:teuthology.orchestra.run.smithi001.stdout: "uuid": "e2ead447-2f2c-40c9-82e7-53b8dc41e2af", 2024-09-13T11:26:53.588 INFO:teuthology.orchestra.run.smithi001.stdout: "up": 1, 2024-09-13T11:26:53.588 INFO:teuthology.orchestra.run.smithi001.stdout: "in": 1, 2024-09-13T11:26:53.588 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 1, 2024-09-13T11:26:53.588 INFO:teuthology.orchestra.run.smithi001.stdout: "primary_affinity": 1, 2024-09-13T11:26:53.588 INFO:teuthology.orchestra.run.smithi001.stdout: "last_clean_begin": 0, 2024-09-13T11:26:53.588 INFO:teuthology.orchestra.run.smithi001.stdout: "last_clean_end": 0, 2024-09-13T11:26:53.589 INFO:teuthology.orchestra.run.smithi001.stdout: "up_from": 21, 2024-09-13T11:26:53.589 INFO:teuthology.orchestra.run.smithi001.stdout: "up_thru": 227, 2024-09-13T11:26:53.589 INFO:teuthology.orchestra.run.smithi001.stdout: "down_at": 0, 2024-09-13T11:26:53.589 INFO:teuthology.orchestra.run.smithi001.stdout: "lost_at": 0, 2024-09-13T11:26:53.589 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-09-13T11:26:53.589 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:26:53.589 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:53.589 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:26:53.589 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.92:6804", 2024-09-13T11:26:53.589 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 3399529091 2024-09-13T11:26:53.589 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:53.589 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:53.589 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:53.590 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_addrs": { 2024-09-13T11:26:53.590 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:26:53.590 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:53.590 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:26:53.590 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.92:6805", 2024-09-13T11:26:53.590 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 3399529091 2024-09-13T11:26:53.590 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:53.590 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:53.590 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:53.590 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_back_addrs": { 2024-09-13T11:26:53.590 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:26:53.590 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:53.591 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:26:53.591 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.92:6807", 2024-09-13T11:26:53.591 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 3399529091 2024-09-13T11:26:53.591 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:53.591 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:53.591 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:53.591 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_front_addrs": { 2024-09-13T11:26:53.591 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:26:53.591 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:53.591 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:26:53.591 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.92:6806", 2024-09-13T11:26:53.592 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 3399529091 2024-09-13T11:26:53.592 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:53.592 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:53.592 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:53.592 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.92:6804/3399529091", 2024-09-13T11:26:53.592 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_addr": "172.21.15.92:6805/3399529091", 2024-09-13T11:26:53.592 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_back_addr": "172.21.15.92:6807/3399529091", 2024-09-13T11:26:53.592 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_front_addr": "172.21.15.92:6806/3399529091", 2024-09-13T11:26:53.592 INFO:teuthology.orchestra.run.smithi001.stdout: "state": [ 2024-09-13T11:26:53.592 INFO:teuthology.orchestra.run.smithi001.stdout: "exists", 2024-09-13T11:26:53.592 INFO:teuthology.orchestra.run.smithi001.stdout: "up" 2024-09-13T11:26:53.592 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:53.593 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:53.593 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:53.593 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": 6, 2024-09-13T11:26:53.593 INFO:teuthology.orchestra.run.smithi001.stdout: "uuid": "dbc20439-144e-4ec7-89de-674eb783f263", 2024-09-13T11:26:53.593 INFO:teuthology.orchestra.run.smithi001.stdout: "up": 1, 2024-09-13T11:26:53.593 INFO:teuthology.orchestra.run.smithi001.stdout: "in": 1, 2024-09-13T11:26:53.593 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 1, 2024-09-13T11:26:53.593 INFO:teuthology.orchestra.run.smithi001.stdout: "primary_affinity": 1, 2024-09-13T11:26:53.593 INFO:teuthology.orchestra.run.smithi001.stdout: "last_clean_begin": 0, 2024-09-13T11:26:53.593 INFO:teuthology.orchestra.run.smithi001.stdout: "last_clean_end": 0, 2024-09-13T11:26:53.593 INFO:teuthology.orchestra.run.smithi001.stdout: "up_from": 22, 2024-09-13T11:26:53.593 INFO:teuthology.orchestra.run.smithi001.stdout: "up_thru": 226, 2024-09-13T11:26:53.593 INFO:teuthology.orchestra.run.smithi001.stdout: "down_at": 0, 2024-09-13T11:26:53.594 INFO:teuthology.orchestra.run.smithi001.stdout: "lost_at": 0, 2024-09-13T11:26:53.594 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-09-13T11:26:53.594 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:26:53.594 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:53.594 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:26:53.594 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.162:6808", 2024-09-13T11:26:53.594 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 2287343806 2024-09-13T11:26:53.594 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:53.594 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:53.594 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:53.594 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_addrs": { 2024-09-13T11:26:53.594 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:26:53.594 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:53.595 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:26:53.595 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.162:6809", 2024-09-13T11:26:53.595 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 2287343806 2024-09-13T11:26:53.595 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:53.595 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:53.595 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:53.595 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_back_addrs": { 2024-09-13T11:26:53.595 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:26:53.595 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:53.595 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:26:53.595 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.162:6811", 2024-09-13T11:26:53.595 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 2287343806 2024-09-13T11:26:53.595 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:53.596 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:53.596 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:53.596 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_front_addrs": { 2024-09-13T11:26:53.596 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:26:53.596 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:53.596 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:26:53.596 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.162:6810", 2024-09-13T11:26:53.596 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 2287343806 2024-09-13T11:26:53.596 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:53.596 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:53.596 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:53.596 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.162:6808/2287343806", 2024-09-13T11:26:53.597 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_addr": "172.21.15.162:6809/2287343806", 2024-09-13T11:26:53.597 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_back_addr": "172.21.15.162:6811/2287343806", 2024-09-13T11:26:53.597 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_front_addr": "172.21.15.162:6810/2287343806", 2024-09-13T11:26:53.597 INFO:teuthology.orchestra.run.smithi001.stdout: "state": [ 2024-09-13T11:26:53.597 INFO:teuthology.orchestra.run.smithi001.stdout: "exists", 2024-09-13T11:26:53.597 INFO:teuthology.orchestra.run.smithi001.stdout: "up" 2024-09-13T11:26:53.597 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:53.597 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:53.597 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:53.597 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": 7, 2024-09-13T11:26:53.597 INFO:teuthology.orchestra.run.smithi001.stdout: "uuid": "60aa5206-cace-40f8-8d1a-10ec28d97096", 2024-09-13T11:26:53.597 INFO:teuthology.orchestra.run.smithi001.stdout: "up": 1, 2024-09-13T11:26:53.598 INFO:teuthology.orchestra.run.smithi001.stdout: "in": 1, 2024-09-13T11:26:53.598 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 1, 2024-09-13T11:26:53.598 INFO:teuthology.orchestra.run.smithi001.stdout: "primary_affinity": 1, 2024-09-13T11:26:53.598 INFO:teuthology.orchestra.run.smithi001.stdout: "last_clean_begin": 0, 2024-09-13T11:26:53.598 INFO:teuthology.orchestra.run.smithi001.stdout: "last_clean_end": 0, 2024-09-13T11:26:53.598 INFO:teuthology.orchestra.run.smithi001.stdout: "up_from": 22, 2024-09-13T11:26:53.598 INFO:teuthology.orchestra.run.smithi001.stdout: "up_thru": 227, 2024-09-13T11:26:53.598 INFO:teuthology.orchestra.run.smithi001.stdout: "down_at": 0, 2024-09-13T11:26:53.598 INFO:teuthology.orchestra.run.smithi001.stdout: "lost_at": 0, 2024-09-13T11:26:53.598 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-09-13T11:26:53.598 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:26:53.598 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:53.598 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:26:53.599 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.177:6813", 2024-09-13T11:26:53.599 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 2110809906 2024-09-13T11:26:53.599 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:53.599 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:53.599 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:53.599 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_addrs": { 2024-09-13T11:26:53.599 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:26:53.599 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:53.599 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:26:53.599 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.177:6814", 2024-09-13T11:26:53.599 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 2110809906 2024-09-13T11:26:53.600 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:53.600 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:53.600 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:53.600 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_back_addrs": { 2024-09-13T11:26:53.600 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:26:53.600 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:53.600 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:26:53.600 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.177:6816", 2024-09-13T11:26:53.600 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 2110809906 2024-09-13T11:26:53.600 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:53.600 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:53.600 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:53.600 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_front_addrs": { 2024-09-13T11:26:53.601 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:26:53.601 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:53.601 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:26:53.601 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.177:6815", 2024-09-13T11:26:53.601 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 2110809906 2024-09-13T11:26:53.601 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:53.601 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:53.601 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:53.601 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.177:6813/2110809906", 2024-09-13T11:26:53.601 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_addr": "172.21.15.177:6814/2110809906", 2024-09-13T11:26:53.601 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_back_addr": "172.21.15.177:6816/2110809906", 2024-09-13T11:26:53.601 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_front_addr": "172.21.15.177:6815/2110809906", 2024-09-13T11:26:53.601 INFO:teuthology.orchestra.run.smithi001.stdout: "state": [ 2024-09-13T11:26:53.602 INFO:teuthology.orchestra.run.smithi001.stdout: "exists", 2024-09-13T11:26:53.602 INFO:teuthology.orchestra.run.smithi001.stdout: "up" 2024-09-13T11:26:53.602 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:53.602 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:53.602 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:53.602 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": 8, 2024-09-13T11:26:53.602 INFO:teuthology.orchestra.run.smithi001.stdout: "uuid": "f3d99325-48ff-403c-8662-1d7745f7c97d", 2024-09-13T11:26:53.602 INFO:teuthology.orchestra.run.smithi001.stdout: "up": 1, 2024-09-13T11:26:53.602 INFO:teuthology.orchestra.run.smithi001.stdout: "in": 1, 2024-09-13T11:26:53.602 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 1, 2024-09-13T11:26:53.602 INFO:teuthology.orchestra.run.smithi001.stdout: "primary_affinity": 1, 2024-09-13T11:26:53.602 INFO:teuthology.orchestra.run.smithi001.stdout: "last_clean_begin": 0, 2024-09-13T11:26:53.602 INFO:teuthology.orchestra.run.smithi001.stdout: "last_clean_end": 0, 2024-09-13T11:26:53.603 INFO:teuthology.orchestra.run.smithi001.stdout: "up_from": 22, 2024-09-13T11:26:53.603 INFO:teuthology.orchestra.run.smithi001.stdout: "up_thru": 225, 2024-09-13T11:26:53.603 INFO:teuthology.orchestra.run.smithi001.stdout: "down_at": 0, 2024-09-13T11:26:53.603 INFO:teuthology.orchestra.run.smithi001.stdout: "lost_at": 0, 2024-09-13T11:26:53.603 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-09-13T11:26:53.603 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:26:53.603 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:53.603 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:26:53.603 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6804", 2024-09-13T11:26:53.603 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 2766567910 2024-09-13T11:26:53.603 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:53.603 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:53.603 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:53.604 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_addrs": { 2024-09-13T11:26:53.604 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:26:53.604 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:53.604 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:26:53.604 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6805", 2024-09-13T11:26:53.604 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 2766567910 2024-09-13T11:26:53.604 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:53.604 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:53.604 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:53.604 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_back_addrs": { 2024-09-13T11:26:53.604 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:26:53.604 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:53.604 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:26:53.604 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6807", 2024-09-13T11:26:53.605 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 2766567910 2024-09-13T11:26:53.605 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:53.605 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:53.605 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:53.605 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_front_addrs": { 2024-09-13T11:26:53.605 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:26:53.605 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:53.605 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:26:53.605 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6806", 2024-09-13T11:26:53.605 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 2766567910 2024-09-13T11:26:53.605 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:53.605 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:53.605 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:53.605 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6804/2766567910", 2024-09-13T11:26:53.606 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_addr": "172.21.15.1:6805/2766567910", 2024-09-13T11:26:53.606 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_back_addr": "172.21.15.1:6807/2766567910", 2024-09-13T11:26:53.606 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_front_addr": "172.21.15.1:6806/2766567910", 2024-09-13T11:26:53.606 INFO:teuthology.orchestra.run.smithi001.stdout: "state": [ 2024-09-13T11:26:53.606 INFO:teuthology.orchestra.run.smithi001.stdout: "exists", 2024-09-13T11:26:53.606 INFO:teuthology.orchestra.run.smithi001.stdout: "up" 2024-09-13T11:26:53.606 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:53.606 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:53.606 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:53.606 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": 9, 2024-09-13T11:26:53.606 INFO:teuthology.orchestra.run.smithi001.stdout: "uuid": "88511e24-3675-4551-968c-1a52e7e28f96", 2024-09-13T11:26:53.606 INFO:teuthology.orchestra.run.smithi001.stdout: "up": 1, 2024-09-13T11:26:53.606 INFO:teuthology.orchestra.run.smithi001.stdout: "in": 1, 2024-09-13T11:26:53.606 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 1, 2024-09-13T11:26:53.607 INFO:teuthology.orchestra.run.smithi001.stdout: "primary_affinity": 1, 2024-09-13T11:26:53.607 INFO:teuthology.orchestra.run.smithi001.stdout: "last_clean_begin": 0, 2024-09-13T11:26:53.607 INFO:teuthology.orchestra.run.smithi001.stdout: "last_clean_end": 0, 2024-09-13T11:26:53.607 INFO:teuthology.orchestra.run.smithi001.stdout: "up_from": 23, 2024-09-13T11:26:53.607 INFO:teuthology.orchestra.run.smithi001.stdout: "up_thru": 191, 2024-09-13T11:26:53.607 INFO:teuthology.orchestra.run.smithi001.stdout: "down_at": 0, 2024-09-13T11:26:53.607 INFO:teuthology.orchestra.run.smithi001.stdout: "lost_at": 0, 2024-09-13T11:26:53.607 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-09-13T11:26:53.607 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:26:53.607 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:53.607 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:26:53.607 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.92:6812", 2024-09-13T11:26:53.607 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 2488187511 2024-09-13T11:26:53.608 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:53.608 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:53.608 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:53.608 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_addrs": { 2024-09-13T11:26:53.608 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:26:53.608 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:53.608 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:26:53.608 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.92:6813", 2024-09-13T11:26:53.608 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 2488187511 2024-09-13T11:26:53.608 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:53.608 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:53.608 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:53.609 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_back_addrs": { 2024-09-13T11:26:53.609 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:26:53.609 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:53.609 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:26:53.609 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.92:6815", 2024-09-13T11:26:53.609 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 2488187511 2024-09-13T11:26:53.609 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:53.609 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:53.609 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:53.609 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_front_addrs": { 2024-09-13T11:26:53.609 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:26:53.609 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:53.609 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:26:53.609 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.92:6814", 2024-09-13T11:26:53.610 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 2488187511 2024-09-13T11:26:53.610 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:53.610 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:53.610 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:53.610 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.92:6812/2488187511", 2024-09-13T11:26:53.610 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_addr": "172.21.15.92:6813/2488187511", 2024-09-13T11:26:53.610 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_back_addr": "172.21.15.92:6815/2488187511", 2024-09-13T11:26:53.610 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_front_addr": "172.21.15.92:6814/2488187511", 2024-09-13T11:26:53.610 INFO:teuthology.orchestra.run.smithi001.stdout: "state": [ 2024-09-13T11:26:53.610 INFO:teuthology.orchestra.run.smithi001.stdout: "exists", 2024-09-13T11:26:53.610 INFO:teuthology.orchestra.run.smithi001.stdout: "up" 2024-09-13T11:26:53.610 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:53.610 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:53.610 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:53.611 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": 10, 2024-09-13T11:26:53.611 INFO:teuthology.orchestra.run.smithi001.stdout: "uuid": "acf4c64d-99af-42be-8406-413c86f54670", 2024-09-13T11:26:53.611 INFO:teuthology.orchestra.run.smithi001.stdout: "up": 1, 2024-09-13T11:26:53.611 INFO:teuthology.orchestra.run.smithi001.stdout: "in": 1, 2024-09-13T11:26:53.611 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 1, 2024-09-13T11:26:53.611 INFO:teuthology.orchestra.run.smithi001.stdout: "primary_affinity": 1, 2024-09-13T11:26:53.611 INFO:teuthology.orchestra.run.smithi001.stdout: "last_clean_begin": 0, 2024-09-13T11:26:53.611 INFO:teuthology.orchestra.run.smithi001.stdout: "last_clean_end": 0, 2024-09-13T11:26:53.611 INFO:teuthology.orchestra.run.smithi001.stdout: "up_from": 22, 2024-09-13T11:26:53.611 INFO:teuthology.orchestra.run.smithi001.stdout: "up_thru": 226, 2024-09-13T11:26:53.611 INFO:teuthology.orchestra.run.smithi001.stdout: "down_at": 0, 2024-09-13T11:26:53.611 INFO:teuthology.orchestra.run.smithi001.stdout: "lost_at": 0, 2024-09-13T11:26:53.611 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-09-13T11:26:53.612 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:26:53.612 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:53.612 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:26:53.612 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.162:6804", 2024-09-13T11:26:53.612 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 1450266459 2024-09-13T11:26:53.612 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:53.612 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:53.612 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:53.612 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_addrs": { 2024-09-13T11:26:53.612 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:26:53.612 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:53.612 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:26:53.612 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.162:6805", 2024-09-13T11:26:53.612 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 1450266459 2024-09-13T11:26:53.613 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:53.613 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:53.613 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:53.613 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_back_addrs": { 2024-09-13T11:26:53.613 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:26:53.613 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:53.613 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:26:53.613 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.162:6807", 2024-09-13T11:26:53.613 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 1450266459 2024-09-13T11:26:53.613 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:53.613 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:53.613 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:53.613 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_front_addrs": { 2024-09-13T11:26:53.613 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:26:53.614 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:53.614 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:26:53.614 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.162:6806", 2024-09-13T11:26:53.614 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 1450266459 2024-09-13T11:26:53.614 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:53.614 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:53.614 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:53.614 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.162:6804/1450266459", 2024-09-13T11:26:53.614 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_addr": "172.21.15.162:6805/1450266459", 2024-09-13T11:26:53.614 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_back_addr": "172.21.15.162:6807/1450266459", 2024-09-13T11:26:53.614 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_front_addr": "172.21.15.162:6806/1450266459", 2024-09-13T11:26:53.614 INFO:teuthology.orchestra.run.smithi001.stdout: "state": [ 2024-09-13T11:26:53.614 INFO:teuthology.orchestra.run.smithi001.stdout: "exists", 2024-09-13T11:26:53.614 INFO:teuthology.orchestra.run.smithi001.stdout: "up" 2024-09-13T11:26:53.615 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:53.615 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:53.615 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:53.615 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": 11, 2024-09-13T11:26:53.615 INFO:teuthology.orchestra.run.smithi001.stdout: "uuid": "a4bfdfbe-b586-4080-8e09-6f26428e59b5", 2024-09-13T11:26:53.615 INFO:teuthology.orchestra.run.smithi001.stdout: "up": 1, 2024-09-13T11:26:53.615 INFO:teuthology.orchestra.run.smithi001.stdout: "in": 1, 2024-09-13T11:26:53.615 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 1, 2024-09-13T11:26:53.615 INFO:teuthology.orchestra.run.smithi001.stdout: "primary_affinity": 1, 2024-09-13T11:26:53.615 INFO:teuthology.orchestra.run.smithi001.stdout: "last_clean_begin": 0, 2024-09-13T11:26:53.615 INFO:teuthology.orchestra.run.smithi001.stdout: "last_clean_end": 0, 2024-09-13T11:26:53.615 INFO:teuthology.orchestra.run.smithi001.stdout: "up_from": 22, 2024-09-13T11:26:53.615 INFO:teuthology.orchestra.run.smithi001.stdout: "up_thru": 227, 2024-09-13T11:26:53.616 INFO:teuthology.orchestra.run.smithi001.stdout: "down_at": 0, 2024-09-13T11:26:53.616 INFO:teuthology.orchestra.run.smithi001.stdout: "lost_at": 0, 2024-09-13T11:26:53.616 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-09-13T11:26:53.616 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:26:53.616 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:53.616 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:26:53.616 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.177:6809", 2024-09-13T11:26:53.616 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 3239521954 2024-09-13T11:26:53.616 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:53.616 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:53.616 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:53.616 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_addrs": { 2024-09-13T11:26:53.616 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:26:53.616 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:53.617 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:26:53.617 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.177:6810", 2024-09-13T11:26:53.617 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 3239521954 2024-09-13T11:26:53.617 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:53.617 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:53.617 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:53.617 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_back_addrs": { 2024-09-13T11:26:53.617 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:26:53.617 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:53.617 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:26:53.617 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.177:6812", 2024-09-13T11:26:53.617 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 3239521954 2024-09-13T11:26:53.617 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:53.618 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:53.618 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:53.618 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_front_addrs": { 2024-09-13T11:26:53.618 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:26:53.618 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:53.618 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:26:53.618 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.177:6811", 2024-09-13T11:26:53.618 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 3239521954 2024-09-13T11:26:53.618 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:53.618 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:53.618 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:53.618 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.177:6809/3239521954", 2024-09-13T11:26:53.618 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_addr": "172.21.15.177:6810/3239521954", 2024-09-13T11:26:53.619 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_back_addr": "172.21.15.177:6812/3239521954", 2024-09-13T11:26:53.619 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_front_addr": "172.21.15.177:6811/3239521954", 2024-09-13T11:26:53.619 INFO:teuthology.orchestra.run.smithi001.stdout: "state": [ 2024-09-13T11:26:53.619 INFO:teuthology.orchestra.run.smithi001.stdout: "exists", 2024-09-13T11:26:53.619 INFO:teuthology.orchestra.run.smithi001.stdout: "up" 2024-09-13T11:26:53.619 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:53.619 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:53.619 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:53.619 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": 12, 2024-09-13T11:26:53.619 INFO:teuthology.orchestra.run.smithi001.stdout: "uuid": "dc571fb6-d8b5-465b-bb25-afa7d0c47b5a", 2024-09-13T11:26:53.619 INFO:teuthology.orchestra.run.smithi001.stdout: "up": 1, 2024-09-13T11:26:53.619 INFO:teuthology.orchestra.run.smithi001.stdout: "in": 1, 2024-09-13T11:26:53.619 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 1, 2024-09-13T11:26:53.619 INFO:teuthology.orchestra.run.smithi001.stdout: "primary_affinity": 1, 2024-09-13T11:26:53.619 INFO:teuthology.orchestra.run.smithi001.stdout: "last_clean_begin": 0, 2024-09-13T11:26:53.620 INFO:teuthology.orchestra.run.smithi001.stdout: "last_clean_end": 0, 2024-09-13T11:26:53.620 INFO:teuthology.orchestra.run.smithi001.stdout: "up_from": 22, 2024-09-13T11:26:53.620 INFO:teuthology.orchestra.run.smithi001.stdout: "up_thru": 141, 2024-09-13T11:26:53.620 INFO:teuthology.orchestra.run.smithi001.stdout: "down_at": 0, 2024-09-13T11:26:53.620 INFO:teuthology.orchestra.run.smithi001.stdout: "lost_at": 0, 2024-09-13T11:26:53.620 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-09-13T11:26:53.620 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:26:53.620 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:53.620 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:26:53.620 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6812", 2024-09-13T11:26:53.620 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 4267062742 2024-09-13T11:26:53.620 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:53.620 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:53.621 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:53.621 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_addrs": { 2024-09-13T11:26:53.621 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:26:53.621 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:53.621 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:26:53.621 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6813", 2024-09-13T11:26:53.621 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 4267062742 2024-09-13T11:26:53.621 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:53.621 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:53.621 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:53.621 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_back_addrs": { 2024-09-13T11:26:53.621 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:26:53.621 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:53.621 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:26:53.622 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6815", 2024-09-13T11:26:53.622 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 4267062742 2024-09-13T11:26:53.622 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:53.622 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:53.622 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:53.622 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_front_addrs": { 2024-09-13T11:26:53.622 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:26:53.622 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:53.622 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:26:53.622 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6814", 2024-09-13T11:26:53.622 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 4267062742 2024-09-13T11:26:53.622 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:53.623 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:53.623 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:53.623 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6812/4267062742", 2024-09-13T11:26:53.623 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_addr": "172.21.15.1:6813/4267062742", 2024-09-13T11:26:53.623 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_back_addr": "172.21.15.1:6815/4267062742", 2024-09-13T11:26:53.623 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_front_addr": "172.21.15.1:6814/4267062742", 2024-09-13T11:26:53.623 INFO:teuthology.orchestra.run.smithi001.stdout: "state": [ 2024-09-13T11:26:53.623 INFO:teuthology.orchestra.run.smithi001.stdout: "exists", 2024-09-13T11:26:53.623 INFO:teuthology.orchestra.run.smithi001.stdout: "up" 2024-09-13T11:26:53.623 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:53.623 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:53.623 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:53.623 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": 13, 2024-09-13T11:26:53.623 INFO:teuthology.orchestra.run.smithi001.stdout: "uuid": "5d321d4b-1b26-47fe-bf84-25751b8cde7d", 2024-09-13T11:26:53.623 INFO:teuthology.orchestra.run.smithi001.stdout: "up": 1, 2024-09-13T11:26:53.624 INFO:teuthology.orchestra.run.smithi001.stdout: "in": 1, 2024-09-13T11:26:53.624 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 1, 2024-09-13T11:26:53.624 INFO:teuthology.orchestra.run.smithi001.stdout: "primary_affinity": 1, 2024-09-13T11:26:53.624 INFO:teuthology.orchestra.run.smithi001.stdout: "last_clean_begin": 0, 2024-09-13T11:26:53.624 INFO:teuthology.orchestra.run.smithi001.stdout: "last_clean_end": 0, 2024-09-13T11:26:53.624 INFO:teuthology.orchestra.run.smithi001.stdout: "up_from": 23, 2024-09-13T11:26:53.624 INFO:teuthology.orchestra.run.smithi001.stdout: "up_thru": 205, 2024-09-13T11:26:53.624 INFO:teuthology.orchestra.run.smithi001.stdout: "down_at": 0, 2024-09-13T11:26:53.624 INFO:teuthology.orchestra.run.smithi001.stdout: "lost_at": 0, 2024-09-13T11:26:53.624 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-09-13T11:26:53.624 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:26:53.624 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:53.624 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:26:53.625 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.92:6808", 2024-09-13T11:26:53.625 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 1759585046 2024-09-13T11:26:53.625 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:53.625 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:53.625 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:53.625 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_addrs": { 2024-09-13T11:26:53.625 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:26:53.625 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:53.625 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:26:53.625 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.92:6809", 2024-09-13T11:26:53.625 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 1759585046 2024-09-13T11:26:53.625 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:53.625 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:53.625 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:53.626 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_back_addrs": { 2024-09-13T11:26:53.626 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:26:53.626 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:53.626 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:26:53.626 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.92:6811", 2024-09-13T11:26:53.626 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 1759585046 2024-09-13T11:26:53.626 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:53.626 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:53.626 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:53.626 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_front_addrs": { 2024-09-13T11:26:53.626 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:26:53.626 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:53.626 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:26:53.626 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.92:6810", 2024-09-13T11:26:53.627 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 1759585046 2024-09-13T11:26:53.627 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:53.627 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:53.627 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:53.627 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.92:6808/1759585046", 2024-09-13T11:26:53.627 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_addr": "172.21.15.92:6809/1759585046", 2024-09-13T11:26:53.627 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_back_addr": "172.21.15.92:6811/1759585046", 2024-09-13T11:26:53.627 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_front_addr": "172.21.15.92:6810/1759585046", 2024-09-13T11:26:53.627 INFO:teuthology.orchestra.run.smithi001.stdout: "state": [ 2024-09-13T11:26:53.627 INFO:teuthology.orchestra.run.smithi001.stdout: "exists", 2024-09-13T11:26:53.627 INFO:teuthology.orchestra.run.smithi001.stdout: "up" 2024-09-13T11:26:53.627 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:53.627 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:53.627 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:53.628 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": 14, 2024-09-13T11:26:53.628 INFO:teuthology.orchestra.run.smithi001.stdout: "uuid": "fc3e2499-0c2a-414d-a25e-2628876ae630", 2024-09-13T11:26:53.628 INFO:teuthology.orchestra.run.smithi001.stdout: "up": 1, 2024-09-13T11:26:53.628 INFO:teuthology.orchestra.run.smithi001.stdout: "in": 1, 2024-09-13T11:26:53.628 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 1, 2024-09-13T11:26:53.628 INFO:teuthology.orchestra.run.smithi001.stdout: "primary_affinity": 0.8266143798828125, 2024-09-13T11:26:53.628 INFO:teuthology.orchestra.run.smithi001.stdout: "last_clean_begin": 22, 2024-09-13T11:26:53.628 INFO:teuthology.orchestra.run.smithi001.stdout: "last_clean_end": 120, 2024-09-13T11:26:53.628 INFO:teuthology.orchestra.run.smithi001.stdout: "up_from": 142, 2024-09-13T11:26:53.628 INFO:teuthology.orchestra.run.smithi001.stdout: "up_thru": 226, 2024-09-13T11:26:53.628 INFO:teuthology.orchestra.run.smithi001.stdout: "down_at": 121, 2024-09-13T11:26:53.628 INFO:teuthology.orchestra.run.smithi001.stdout: "lost_at": 0, 2024-09-13T11:26:53.628 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-09-13T11:26:53.628 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:26:53.629 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:53.629 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:26:53.629 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.162:6800", 2024-09-13T11:26:53.629 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 3999392664 2024-09-13T11:26:53.629 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:53.629 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:53.629 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:53.629 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_addrs": { 2024-09-13T11:26:53.629 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:26:53.629 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:53.629 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:26:53.629 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.162:6801", 2024-09-13T11:26:53.629 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 3999392664 2024-09-13T11:26:53.630 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:53.630 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:53.630 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:53.630 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_back_addrs": { 2024-09-13T11:26:53.630 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:26:53.630 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:53.630 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:26:53.630 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.162:6803", 2024-09-13T11:26:53.630 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 3999392664 2024-09-13T11:26:53.630 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:53.630 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:53.630 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:53.630 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_front_addrs": { 2024-09-13T11:26:53.630 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:26:53.631 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:53.631 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:26:53.631 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.162:6802", 2024-09-13T11:26:53.631 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 3999392664 2024-09-13T11:26:53.631 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:53.631 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:53.631 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:53.631 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.162:6800/3999392664", 2024-09-13T11:26:53.631 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_addr": "172.21.15.162:6801/3999392664", 2024-09-13T11:26:53.631 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_back_addr": "172.21.15.162:6803/3999392664", 2024-09-13T11:26:53.631 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_front_addr": "172.21.15.162:6802/3999392664", 2024-09-13T11:26:53.631 INFO:teuthology.orchestra.run.smithi001.stdout: "state": [ 2024-09-13T11:26:53.631 INFO:teuthology.orchestra.run.smithi001.stdout: "exists", 2024-09-13T11:26:53.631 INFO:teuthology.orchestra.run.smithi001.stdout: "up" 2024-09-13T11:26:53.632 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:53.632 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:53.632 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:53.632 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": 15, 2024-09-13T11:26:53.632 INFO:teuthology.orchestra.run.smithi001.stdout: "uuid": "d2f69775-edfe-4527-91ec-f7c7e558bec8", 2024-09-13T11:26:53.632 INFO:teuthology.orchestra.run.smithi001.stdout: "up": 1, 2024-09-13T11:26:53.632 INFO:teuthology.orchestra.run.smithi001.stdout: "in": 1, 2024-09-13T11:26:53.632 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 1, 2024-09-13T11:26:53.632 INFO:teuthology.orchestra.run.smithi001.stdout: "primary_affinity": 0, 2024-09-13T11:26:53.632 INFO:teuthology.orchestra.run.smithi001.stdout: "last_clean_begin": 0, 2024-09-13T11:26:53.632 INFO:teuthology.orchestra.run.smithi001.stdout: "last_clean_end": 0, 2024-09-13T11:26:53.632 INFO:teuthology.orchestra.run.smithi001.stdout: "up_from": 22, 2024-09-13T11:26:53.632 INFO:teuthology.orchestra.run.smithi001.stdout: "up_thru": 122, 2024-09-13T11:26:53.633 INFO:teuthology.orchestra.run.smithi001.stdout: "down_at": 0, 2024-09-13T11:26:53.633 INFO:teuthology.orchestra.run.smithi001.stdout: "lost_at": 0, 2024-09-13T11:26:53.633 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-09-13T11:26:53.633 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:26:53.633 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:53.633 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:26:53.633 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.177:6805", 2024-09-13T11:26:53.633 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 1294337172 2024-09-13T11:26:53.633 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:53.633 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:53.633 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:53.633 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_addrs": { 2024-09-13T11:26:53.633 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:26:53.633 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:53.634 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:26:53.634 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.177:6806", 2024-09-13T11:26:53.634 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 1294337172 2024-09-13T11:26:53.634 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:53.634 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:53.634 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:53.634 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_back_addrs": { 2024-09-13T11:26:53.634 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:26:53.634 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:53.634 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:26:53.634 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.177:6808", 2024-09-13T11:26:53.634 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 1294337172 2024-09-13T11:26:53.634 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:53.634 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:53.635 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:53.635 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_front_addrs": { 2024-09-13T11:26:53.635 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-13T11:26:53.635 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:53.635 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-13T11:26:53.635 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.177:6807", 2024-09-13T11:26:53.635 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 1294337172 2024-09-13T11:26:53.635 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:53.635 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:53.635 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:53.635 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.177:6805/1294337172", 2024-09-13T11:26:53.635 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_addr": "172.21.15.177:6806/1294337172", 2024-09-13T11:26:53.635 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_back_addr": "172.21.15.177:6808/1294337172", 2024-09-13T11:26:53.635 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_front_addr": "172.21.15.177:6807/1294337172", 2024-09-13T11:26:53.636 INFO:teuthology.orchestra.run.smithi001.stdout: "state": [ 2024-09-13T11:26:53.636 INFO:teuthology.orchestra.run.smithi001.stdout: "exists", 2024-09-13T11:26:53.636 INFO:teuthology.orchestra.run.smithi001.stdout: "up" 2024-09-13T11:26:53.636 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:53.636 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:53.636 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-09-13T11:26:53.636 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_xinfo": [ 2024-09-13T11:26:53.636 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:53.636 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": 0, 2024-09-13T11:26:53.636 INFO:teuthology.orchestra.run.smithi001.stdout: "down_stamp": "0.000000", 2024-09-13T11:26:53.636 INFO:teuthology.orchestra.run.smithi001.stdout: "laggy_probability": 0, 2024-09-13T11:26:53.636 INFO:teuthology.orchestra.run.smithi001.stdout: "laggy_interval": 0, 2024-09-13T11:26:53.636 INFO:teuthology.orchestra.run.smithi001.stdout: "features": 4540719139924082687, 2024-09-13T11:26:53.637 INFO:teuthology.orchestra.run.smithi001.stdout: "old_weight": 0, 2024-09-13T11:26:53.637 INFO:teuthology.orchestra.run.smithi001.stdout: "last_purged_snaps_scrub": "2024-09-13T11:26:19.725882+0000", 2024-09-13T11:26:53.637 INFO:teuthology.orchestra.run.smithi001.stdout: "dead_epoch": 0 2024-09-13T11:26:53.637 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:53.637 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:53.637 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": 1, 2024-09-13T11:26:53.637 INFO:teuthology.orchestra.run.smithi001.stdout: "down_stamp": "0.000000", 2024-09-13T11:26:53.637 INFO:teuthology.orchestra.run.smithi001.stdout: "laggy_probability": 0, 2024-09-13T11:26:53.637 INFO:teuthology.orchestra.run.smithi001.stdout: "laggy_interval": 0, 2024-09-13T11:26:53.637 INFO:teuthology.orchestra.run.smithi001.stdout: "features": 4540719139924082687, 2024-09-13T11:26:53.637 INFO:teuthology.orchestra.run.smithi001.stdout: "old_weight": 0, 2024-09-13T11:26:53.637 INFO:teuthology.orchestra.run.smithi001.stdout: "last_purged_snaps_scrub": "2024-09-13T11:25:47.780833+0000", 2024-09-13T11:26:53.638 INFO:teuthology.orchestra.run.smithi001.stdout: "dead_epoch": 30 2024-09-13T11:26:53.638 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:53.638 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:53.638 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": 2, 2024-09-13T11:26:53.638 INFO:teuthology.orchestra.run.smithi001.stdout: "down_stamp": "0.000000", 2024-09-13T11:26:53.638 INFO:teuthology.orchestra.run.smithi001.stdout: "laggy_probability": 0, 2024-09-13T11:26:53.638 INFO:teuthology.orchestra.run.smithi001.stdout: "laggy_interval": 0, 2024-09-13T11:26:53.638 INFO:teuthology.orchestra.run.smithi001.stdout: "features": 4540719139924082687, 2024-09-13T11:26:53.638 INFO:teuthology.orchestra.run.smithi001.stdout: "old_weight": 0, 2024-09-13T11:26:53.638 INFO:teuthology.orchestra.run.smithi001.stdout: "last_purged_snaps_scrub": "2024-09-13T11:26:18.925695+0000", 2024-09-13T11:26:53.638 INFO:teuthology.orchestra.run.smithi001.stdout: "dead_epoch": 0 2024-09-13T11:26:53.638 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:53.638 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:53.638 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": 3, 2024-09-13T11:26:53.639 INFO:teuthology.orchestra.run.smithi001.stdout: "down_stamp": "0.000000", 2024-09-13T11:26:53.639 INFO:teuthology.orchestra.run.smithi001.stdout: "laggy_probability": 0, 2024-09-13T11:26:53.639 INFO:teuthology.orchestra.run.smithi001.stdout: "laggy_interval": 0, 2024-09-13T11:26:53.639 INFO:teuthology.orchestra.run.smithi001.stdout: "features": 4540719139924082687, 2024-09-13T11:26:53.639 INFO:teuthology.orchestra.run.smithi001.stdout: "old_weight": 0, 2024-09-13T11:26:53.639 INFO:teuthology.orchestra.run.smithi001.stdout: "last_purged_snaps_scrub": "2024-09-13T11:26:17.200385+0000", 2024-09-13T11:26:53.639 INFO:teuthology.orchestra.run.smithi001.stdout: "dead_epoch": 0 2024-09-13T11:26:53.639 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:53.639 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:53.639 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": 4, 2024-09-13T11:26:53.639 INFO:teuthology.orchestra.run.smithi001.stdout: "down_stamp": "0.000000", 2024-09-13T11:26:53.640 INFO:teuthology.orchestra.run.smithi001.stdout: "laggy_probability": 0, 2024-09-13T11:26:53.640 INFO:teuthology.orchestra.run.smithi001.stdout: "laggy_interval": 0, 2024-09-13T11:26:53.640 INFO:teuthology.orchestra.run.smithi001.stdout: "features": 4540719139924082687, 2024-09-13T11:26:53.640 INFO:teuthology.orchestra.run.smithi001.stdout: "old_weight": 0, 2024-09-13T11:26:53.640 INFO:teuthology.orchestra.run.smithi001.stdout: "last_purged_snaps_scrub": "2024-09-13T11:25:59.330894+0000", 2024-09-13T11:26:53.640 INFO:teuthology.orchestra.run.smithi001.stdout: "dead_epoch": 0 2024-09-13T11:26:53.640 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:53.640 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:53.640 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": 5, 2024-09-13T11:26:53.640 INFO:teuthology.orchestra.run.smithi001.stdout: "down_stamp": "0.000000", 2024-09-13T11:26:53.640 INFO:teuthology.orchestra.run.smithi001.stdout: "laggy_probability": 0, 2024-09-13T11:26:53.640 INFO:teuthology.orchestra.run.smithi001.stdout: "laggy_interval": 0, 2024-09-13T11:26:53.640 INFO:teuthology.orchestra.run.smithi001.stdout: "features": 4540719139924082687, 2024-09-13T11:26:53.640 INFO:teuthology.orchestra.run.smithi001.stdout: "old_weight": 0, 2024-09-13T11:26:53.641 INFO:teuthology.orchestra.run.smithi001.stdout: "last_purged_snaps_scrub": "2024-09-13T11:26:21.996515+0000", 2024-09-13T11:26:53.641 INFO:teuthology.orchestra.run.smithi001.stdout: "dead_epoch": 0 2024-09-13T11:26:53.641 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:53.641 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:53.641 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": 6, 2024-09-13T11:26:53.641 INFO:teuthology.orchestra.run.smithi001.stdout: "down_stamp": "0.000000", 2024-09-13T11:26:53.641 INFO:teuthology.orchestra.run.smithi001.stdout: "laggy_probability": 0, 2024-09-13T11:26:53.641 INFO:teuthology.orchestra.run.smithi001.stdout: "laggy_interval": 0, 2024-09-13T11:26:53.641 INFO:teuthology.orchestra.run.smithi001.stdout: "features": 4540719139924082687, 2024-09-13T11:26:53.641 INFO:teuthology.orchestra.run.smithi001.stdout: "old_weight": 0, 2024-09-13T11:26:53.641 INFO:teuthology.orchestra.run.smithi001.stdout: "last_purged_snaps_scrub": "2024-09-13T11:26:16.679806+0000", 2024-09-13T11:26:53.641 INFO:teuthology.orchestra.run.smithi001.stdout: "dead_epoch": 0 2024-09-13T11:26:53.641 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:53.642 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:53.642 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": 7, 2024-09-13T11:26:53.642 INFO:teuthology.orchestra.run.smithi001.stdout: "down_stamp": "0.000000", 2024-09-13T11:26:53.642 INFO:teuthology.orchestra.run.smithi001.stdout: "laggy_probability": 0, 2024-09-13T11:26:53.642 INFO:teuthology.orchestra.run.smithi001.stdout: "laggy_interval": 0, 2024-09-13T11:26:53.642 INFO:teuthology.orchestra.run.smithi001.stdout: "features": 4540719139924082687, 2024-09-13T11:26:53.642 INFO:teuthology.orchestra.run.smithi001.stdout: "old_weight": 0, 2024-09-13T11:26:53.642 INFO:teuthology.orchestra.run.smithi001.stdout: "last_purged_snaps_scrub": "2024-09-13T11:26:01.857996+0000", 2024-09-13T11:26:53.642 INFO:teuthology.orchestra.run.smithi001.stdout: "dead_epoch": 0 2024-09-13T11:26:53.642 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:53.642 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:53.642 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": 8, 2024-09-13T11:26:53.642 INFO:teuthology.orchestra.run.smithi001.stdout: "down_stamp": "0.000000", 2024-09-13T11:26:53.642 INFO:teuthology.orchestra.run.smithi001.stdout: "laggy_probability": 0, 2024-09-13T11:26:53.643 INFO:teuthology.orchestra.run.smithi001.stdout: "laggy_interval": 0, 2024-09-13T11:26:53.643 INFO:teuthology.orchestra.run.smithi001.stdout: "features": 4540719139924082687, 2024-09-13T11:26:53.643 INFO:teuthology.orchestra.run.smithi001.stdout: "old_weight": 0, 2024-09-13T11:26:53.643 INFO:teuthology.orchestra.run.smithi001.stdout: "last_purged_snaps_scrub": "2024-09-13T11:26:14.379046+0000", 2024-09-13T11:26:53.643 INFO:teuthology.orchestra.run.smithi001.stdout: "dead_epoch": 0 2024-09-13T11:26:53.643 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:53.643 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:53.643 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": 9, 2024-09-13T11:26:53.643 INFO:teuthology.orchestra.run.smithi001.stdout: "down_stamp": "0.000000", 2024-09-13T11:26:53.643 INFO:teuthology.orchestra.run.smithi001.stdout: "laggy_probability": 0, 2024-09-13T11:26:53.643 INFO:teuthology.orchestra.run.smithi001.stdout: "laggy_interval": 0, 2024-09-13T11:26:53.643 INFO:teuthology.orchestra.run.smithi001.stdout: "features": 4540719139924082687, 2024-09-13T11:26:53.643 INFO:teuthology.orchestra.run.smithi001.stdout: "old_weight": 0, 2024-09-13T11:26:53.644 INFO:teuthology.orchestra.run.smithi001.stdout: "last_purged_snaps_scrub": "2024-09-13T11:26:09.278966+0000", 2024-09-13T11:26:53.644 INFO:teuthology.orchestra.run.smithi001.stdout: "dead_epoch": 0 2024-09-13T11:26:53.644 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:53.644 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:53.644 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": 10, 2024-09-13T11:26:53.644 INFO:teuthology.orchestra.run.smithi001.stdout: "down_stamp": "0.000000", 2024-09-13T11:26:53.644 INFO:teuthology.orchestra.run.smithi001.stdout: "laggy_probability": 0, 2024-09-13T11:26:53.644 INFO:teuthology.orchestra.run.smithi001.stdout: "laggy_interval": 0, 2024-09-13T11:26:53.644 INFO:teuthology.orchestra.run.smithi001.stdout: "features": 4540719139924082687, 2024-09-13T11:26:53.644 INFO:teuthology.orchestra.run.smithi001.stdout: "old_weight": 0, 2024-09-13T11:26:53.644 INFO:teuthology.orchestra.run.smithi001.stdout: "last_purged_snaps_scrub": "2024-09-13T11:26:02.999347+0000", 2024-09-13T11:26:53.644 INFO:teuthology.orchestra.run.smithi001.stdout: "dead_epoch": 0 2024-09-13T11:26:53.644 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:53.645 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:53.645 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": 11, 2024-09-13T11:26:53.645 INFO:teuthology.orchestra.run.smithi001.stdout: "down_stamp": "0.000000", 2024-09-13T11:26:53.645 INFO:teuthology.orchestra.run.smithi001.stdout: "laggy_probability": 0, 2024-09-13T11:26:53.645 INFO:teuthology.orchestra.run.smithi001.stdout: "laggy_interval": 0, 2024-09-13T11:26:53.645 INFO:teuthology.orchestra.run.smithi001.stdout: "features": 4540719139924082687, 2024-09-13T11:26:53.645 INFO:teuthology.orchestra.run.smithi001.stdout: "old_weight": 0, 2024-09-13T11:26:53.645 INFO:teuthology.orchestra.run.smithi001.stdout: "last_purged_snaps_scrub": "2024-09-13T11:26:27.208176+0000", 2024-09-13T11:26:53.645 INFO:teuthology.orchestra.run.smithi001.stdout: "dead_epoch": 0 2024-09-13T11:26:53.645 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:53.645 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:53.645 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": 12, 2024-09-13T11:26:53.645 INFO:teuthology.orchestra.run.smithi001.stdout: "down_stamp": "0.000000", 2024-09-13T11:26:53.645 INFO:teuthology.orchestra.run.smithi001.stdout: "laggy_probability": 0, 2024-09-13T11:26:53.646 INFO:teuthology.orchestra.run.smithi001.stdout: "laggy_interval": 0, 2024-09-13T11:26:53.646 INFO:teuthology.orchestra.run.smithi001.stdout: "features": 4540719139924082687, 2024-09-13T11:26:53.646 INFO:teuthology.orchestra.run.smithi001.stdout: "old_weight": 0, 2024-09-13T11:26:53.646 INFO:teuthology.orchestra.run.smithi001.stdout: "last_purged_snaps_scrub": "2024-09-13T11:25:47.548386+0000", 2024-09-13T11:26:53.646 INFO:teuthology.orchestra.run.smithi001.stdout: "dead_epoch": 0 2024-09-13T11:26:53.646 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:53.646 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:53.646 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": 13, 2024-09-13T11:26:53.646 INFO:teuthology.orchestra.run.smithi001.stdout: "down_stamp": "0.000000", 2024-09-13T11:26:53.646 INFO:teuthology.orchestra.run.smithi001.stdout: "laggy_probability": 0, 2024-09-13T11:26:53.646 INFO:teuthology.orchestra.run.smithi001.stdout: "laggy_interval": 0, 2024-09-13T11:26:53.646 INFO:teuthology.orchestra.run.smithi001.stdout: "features": 4540719139924082687, 2024-09-13T11:26:53.646 INFO:teuthology.orchestra.run.smithi001.stdout: "old_weight": 0, 2024-09-13T11:26:53.647 INFO:teuthology.orchestra.run.smithi001.stdout: "last_purged_snaps_scrub": "2024-09-13T11:25:45.821466+0000", 2024-09-13T11:26:53.647 INFO:teuthology.orchestra.run.smithi001.stdout: "dead_epoch": 0 2024-09-13T11:26:53.647 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:53.647 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:53.647 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": 14, 2024-09-13T11:26:53.647 INFO:teuthology.orchestra.run.smithi001.stdout: "down_stamp": "0.000000", 2024-09-13T11:26:53.647 INFO:teuthology.orchestra.run.smithi001.stdout: "laggy_probability": 0, 2024-09-13T11:26:53.647 INFO:teuthology.orchestra.run.smithi001.stdout: "laggy_interval": 0, 2024-09-13T11:26:53.647 INFO:teuthology.orchestra.run.smithi001.stdout: "features": 4540719139924082687, 2024-09-13T11:26:53.647 INFO:teuthology.orchestra.run.smithi001.stdout: "old_weight": 0, 2024-09-13T11:26:53.647 INFO:teuthology.orchestra.run.smithi001.stdout: "last_purged_snaps_scrub": "2024-09-13T11:26:03.927526+0000", 2024-09-13T11:26:53.647 INFO:teuthology.orchestra.run.smithi001.stdout: "dead_epoch": 121 2024-09-13T11:26:53.647 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:53.648 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:53.648 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": 15, 2024-09-13T11:26:53.648 INFO:teuthology.orchestra.run.smithi001.stdout: "down_stamp": "0.000000", 2024-09-13T11:26:53.648 INFO:teuthology.orchestra.run.smithi001.stdout: "laggy_probability": 0, 2024-09-13T11:26:53.648 INFO:teuthology.orchestra.run.smithi001.stdout: "laggy_interval": 0, 2024-09-13T11:26:53.648 INFO:teuthology.orchestra.run.smithi001.stdout: "features": 4540719139924082687, 2024-09-13T11:26:53.648 INFO:teuthology.orchestra.run.smithi001.stdout: "old_weight": 0, 2024-09-13T11:26:53.648 INFO:teuthology.orchestra.run.smithi001.stdout: "last_purged_snaps_scrub": "2024-09-13T11:25:57.987079+0000", 2024-09-13T11:26:53.648 INFO:teuthology.orchestra.run.smithi001.stdout: "dead_epoch": 0 2024-09-13T11:26:53.648 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:53.648 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-09-13T11:26:53.648 INFO:teuthology.orchestra.run.smithi001.stdout: "pg_upmap": [ 2024-09-13T11:26:53.648 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:53.648 INFO:teuthology.orchestra.run.smithi001.stdout: "pgid": "3.e", 2024-09-13T11:26:53.649 INFO:teuthology.orchestra.run.smithi001.stdout: "osds": [ 2024-09-13T11:26:53.649 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-09-13T11:26:53.649 INFO:teuthology.orchestra.run.smithi001.stdout: 1 2024-09-13T11:26:53.649 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:53.649 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:53.649 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:53.649 INFO:teuthology.orchestra.run.smithi001.stdout: "pgid": "3.22", 2024-09-13T11:26:53.649 INFO:teuthology.orchestra.run.smithi001.stdout: "osds": [ 2024-09-13T11:26:53.649 INFO:teuthology.orchestra.run.smithi001.stdout: 14, 2024-09-13T11:26:53.649 INFO:teuthology.orchestra.run.smithi001.stdout: 12 2024-09-13T11:26:53.649 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:53.649 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:53.649 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-09-13T11:26:53.649 INFO:teuthology.orchestra.run.smithi001.stdout: "pg_upmap_items": [], 2024-09-13T11:26:53.649 INFO:teuthology.orchestra.run.smithi001.stdout: "pg_upmap_primaries": [], 2024-09-13T11:26:53.650 INFO:teuthology.orchestra.run.smithi001.stdout: "pg_temp": [ 2024-09-13T11:26:53.650 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:53.650 INFO:teuthology.orchestra.run.smithi001.stdout: "pgid": "3.12", 2024-09-13T11:26:53.650 INFO:teuthology.orchestra.run.smithi001.stdout: "osds": [ 2024-09-13T11:26:53.650 INFO:teuthology.orchestra.run.smithi001.stdout: 5, 2024-09-13T11:26:53.650 INFO:teuthology.orchestra.run.smithi001.stdout: 7 2024-09-13T11:26:53.650 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:53.650 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:53.650 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:53.650 INFO:teuthology.orchestra.run.smithi001.stdout: "pgid": "3.22", 2024-09-13T11:26:53.650 INFO:teuthology.orchestra.run.smithi001.stdout: "osds": [ 2024-09-13T11:26:53.650 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-09-13T11:26:53.650 INFO:teuthology.orchestra.run.smithi001.stdout: 14 2024-09-13T11:26:53.650 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:53.651 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:53.651 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:53.651 INFO:teuthology.orchestra.run.smithi001.stdout: "pgid": "3.2e", 2024-09-13T11:26:53.651 INFO:teuthology.orchestra.run.smithi001.stdout: "osds": [ 2024-09-13T11:26:53.651 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-09-13T11:26:53.651 INFO:teuthology.orchestra.run.smithi001.stdout: 9 2024-09-13T11:26:53.651 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:53.651 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:53.651 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:53.651 INFO:teuthology.orchestra.run.smithi001.stdout: "pgid": "3.2f", 2024-09-13T11:26:53.651 INFO:teuthology.orchestra.run.smithi001.stdout: "osds": [ 2024-09-13T11:26:53.651 INFO:teuthology.orchestra.run.smithi001.stdout: 11, 2024-09-13T11:26:53.651 INFO:teuthology.orchestra.run.smithi001.stdout: 4 2024-09-13T11:26:53.651 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:53.651 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:53.652 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:53.652 INFO:teuthology.orchestra.run.smithi001.stdout: "pgid": "3.30", 2024-09-13T11:26:53.652 INFO:teuthology.orchestra.run.smithi001.stdout: "osds": [ 2024-09-13T11:26:53.652 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-09-13T11:26:53.652 INFO:teuthology.orchestra.run.smithi001.stdout: 3 2024-09-13T11:26:53.652 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:53.652 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:53.652 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:53.652 INFO:teuthology.orchestra.run.smithi001.stdout: "pgid": "3.31", 2024-09-13T11:26:53.652 INFO:teuthology.orchestra.run.smithi001.stdout: "osds": [ 2024-09-13T11:26:53.652 INFO:teuthology.orchestra.run.smithi001.stdout: 4, 2024-09-13T11:26:53.652 INFO:teuthology.orchestra.run.smithi001.stdout: 1 2024-09-13T11:26:53.652 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:53.653 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:53.653 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:53.653 INFO:teuthology.orchestra.run.smithi001.stdout: "pgid": "3.32", 2024-09-13T11:26:53.653 INFO:teuthology.orchestra.run.smithi001.stdout: "osds": [ 2024-09-13T11:26:53.653 INFO:teuthology.orchestra.run.smithi001.stdout: 5, 2024-09-13T11:26:53.653 INFO:teuthology.orchestra.run.smithi001.stdout: 0 2024-09-13T11:26:53.653 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:53.653 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:53.653 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:53.653 INFO:teuthology.orchestra.run.smithi001.stdout: "pgid": "3.33", 2024-09-13T11:26:53.653 INFO:teuthology.orchestra.run.smithi001.stdout: "osds": [ 2024-09-13T11:26:53.653 INFO:teuthology.orchestra.run.smithi001.stdout: 7, 2024-09-13T11:26:53.653 INFO:teuthology.orchestra.run.smithi001.stdout: 2 2024-09-13T11:26:53.653 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:53.654 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:53.654 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:53.654 INFO:teuthology.orchestra.run.smithi001.stdout: "pgid": "3.34", 2024-09-13T11:26:53.654 INFO:teuthology.orchestra.run.smithi001.stdout: "osds": [ 2024-09-13T11:26:53.654 INFO:teuthology.orchestra.run.smithi001.stdout: 3, 2024-09-13T11:26:53.654 INFO:teuthology.orchestra.run.smithi001.stdout: 10 2024-09-13T11:26:53.654 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:53.654 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:53.654 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:53.654 INFO:teuthology.orchestra.run.smithi001.stdout: "pgid": "3.35", 2024-09-13T11:26:53.654 INFO:teuthology.orchestra.run.smithi001.stdout: "osds": [ 2024-09-13T11:26:53.654 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-09-13T11:26:53.654 INFO:teuthology.orchestra.run.smithi001.stdout: 7 2024-09-13T11:26:53.654 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:53.654 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:53.655 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:53.655 INFO:teuthology.orchestra.run.smithi001.stdout: "pgid": "3.36", 2024-09-13T11:26:53.655 INFO:teuthology.orchestra.run.smithi001.stdout: "osds": [ 2024-09-13T11:26:53.655 INFO:teuthology.orchestra.run.smithi001.stdout: 11, 2024-09-13T11:26:53.655 INFO:teuthology.orchestra.run.smithi001.stdout: 0 2024-09-13T11:26:53.655 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:53.655 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:53.655 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:26:53.655 INFO:teuthology.orchestra.run.smithi001.stdout: "pgid": "3.37", 2024-09-13T11:26:53.655 INFO:teuthology.orchestra.run.smithi001.stdout: "osds": [ 2024-09-13T11:26:53.655 INFO:teuthology.orchestra.run.smithi001.stdout: 3, 2024-09-13T11:26:53.655 INFO:teuthology.orchestra.run.smithi001.stdout: 2 2024-09-13T11:26:53.655 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:26:53.655 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:53.655 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-09-13T11:26:53.656 INFO:teuthology.orchestra.run.smithi001.stdout: "primary_temp": [], 2024-09-13T11:26:53.656 INFO:teuthology.orchestra.run.smithi001.stdout: "blocklist": {}, 2024-09-13T11:26:53.656 INFO:teuthology.orchestra.run.smithi001.stdout: "range_blocklist": {}, 2024-09-13T11:26:53.656 INFO:teuthology.orchestra.run.smithi001.stdout: "erasure_code_profiles": { 2024-09-13T11:26:53.656 INFO:teuthology.orchestra.run.smithi001.stdout: "default": { 2024-09-13T11:26:53.656 INFO:teuthology.orchestra.run.smithi001.stdout: "crush-failure-domain": "osd", 2024-09-13T11:26:53.656 INFO:teuthology.orchestra.run.smithi001.stdout: "k": "2", 2024-09-13T11:26:53.656 INFO:teuthology.orchestra.run.smithi001.stdout: "m": "1", 2024-09-13T11:26:53.656 INFO:teuthology.orchestra.run.smithi001.stdout: "plugin": "jerasure", 2024-09-13T11:26:53.656 INFO:teuthology.orchestra.run.smithi001.stdout: "technique": "reed_sol_van" 2024-09-13T11:26:53.656 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:53.656 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:26:53.656 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_snaps_queue": [], 2024-09-13T11:26:53.656 INFO:teuthology.orchestra.run.smithi001.stdout: "new_removed_snaps": [], 2024-09-13T11:26:53.657 INFO:teuthology.orchestra.run.smithi001.stdout: "new_purged_snaps": [], 2024-09-13T11:26:53.657 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_node_flags": {}, 2024-09-13T11:26:53.657 INFO:teuthology.orchestra.run.smithi001.stdout: "device_class_flags": {}, 2024-09-13T11:26:53.657 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": { 2024-09-13T11:26:53.657 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode_enabled": false, 2024-09-13T11:26:53.657 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_bucket_count": 0, 2024-09-13T11:26:53.657 INFO:teuthology.orchestra.run.smithi001.stdout: "degraded_stretch_mode": 0, 2024-09-13T11:26:53.657 INFO:teuthology.orchestra.run.smithi001.stdout: "recovering_stretch_mode": 0, 2024-09-13T11:26:53.657 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode_bucket": 0 2024-09-13T11:26:53.657 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:26:53.657 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-13T11:26:53.659 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-09-13T11:26:53.659 INFO:teuthology.orchestra.run.smithi092.stdout: "ops": [], 2024-09-13T11:26:53.660 INFO:teuthology.orchestra.run.smithi092.stdout: "num_ops": 0 2024-09-13T11:26:53.660 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-09-13T11:26:53.661 INFO:tasks.thrashosds.thrasher:j is {'epoch': 228, 'fsid': '2fd68ac7-5581-4d90-80a4-30bcf5252a7a', 'created': '2024-09-13T11:21:57.223153+0000', 'modified': '2024-09-13T11:26:52.959392+0000', 'last_up_change': '2024-09-13T11:24:55.968636+0000', 'last_in_change': '2024-09-13T11:26:04.007363+0000', 'flags': 'noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5806080, 'flags_set': ['nodeep-scrub', 'noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 7, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 3, 'max_osd': 16, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-13T11:22:10.162266+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 15.380000114440918, 'score_stable': 15.380000114440918, 'optimal_score': 0.12999999523162842, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.9266633987426758, 'average_primary_affinity': 0.9900000095367432, 'average_primary_affinity_weighted': 0.9300000071525574}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2024-09-13T11:22:12.235549+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '28', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 2, 'snap_epoch': 28, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 3.75, 'score_stable': 3.75, 'optimal_score': 0.6000000238418579, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9266633987426758, 'average_primary_affinity': 0.9900000095367432, 'average_primary_affinity_weighted': 0.9300000071525574}}, {'pool': 3, 'pool_name': 'unique_pool_0', 'create_time': '2024-09-13T11:22:24.305521+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 56, 'pg_placement_num': 56, 'pg_placement_num_target': 56, 'pg_num_target': 56, 'pg_num_pending': 56, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '226', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '224', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8700000047683716, 'score_stable': 2.140000104904175, 'optimal_score': 1.0700000524520874, 'raw_score_acting': 2, 'raw_score_stable': 2.2899999618530273, 'primary_affinity_weighted': 0.9266633987426758, 'average_primary_affinity': 0.9900000095367432, 'average_primary_affinity_weighted': 0.9300000071525574}}], 'osds': [{'osd': 0, 'uuid': '0fc12b3c-e0de-4546-b935-db77efe7c567', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 22, 'up_thru': 227, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.1:6800', 'nonce': 1544776477}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.1:6801', 'nonce': 1544776477}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.1:6803', 'nonce': 1544776477}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.1:6802', 'nonce': 1544776477}]}, 'public_addr': '172.21.15.1:6800/1544776477', 'cluster_addr': '172.21.15.1:6801/1544776477', 'heartbeat_back_addr': '172.21.15.1:6803/1544776477', 'heartbeat_front_addr': '172.21.15.1:6802/1544776477', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': 'b77fe331-cc92-4adb-b146-af5c380f6d4f', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 23, 'last_clean_end': 29, 'up_from': 40, 'up_thru': 226, 'down_at': 30, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.92:6800', 'nonce': 3591430847}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.92:6801', 'nonce': 3591430847}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.92:6803', 'nonce': 3591430847}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.92:6802', 'nonce': 3591430847}]}, 'public_addr': '172.21.15.92:6800/3591430847', 'cluster_addr': '172.21.15.92:6801/3591430847', 'heartbeat_back_addr': '172.21.15.92:6803/3591430847', 'heartbeat_front_addr': '172.21.15.92:6802/3591430847', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': 'c9d031c8-e885-4229-affa-08e1729d00e8', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 22, 'up_thru': 226, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.162:6812', 'nonce': 2712148101}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.162:6813', 'nonce': 2712148101}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.162:6815', 'nonce': 2712148101}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.162:6814', 'nonce': 2712148101}]}, 'public_addr': '172.21.15.162:6812/2712148101', 'cluster_addr': '172.21.15.162:6813/2712148101', 'heartbeat_back_addr': '172.21.15.162:6815/2712148101', 'heartbeat_front_addr': '172.21.15.162:6814/2712148101', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': 'bf50d71a-6c3a-4fbc-a407-cb81bad14a7b', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 21, 'up_thru': 227, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.177:6801', 'nonce': 3367751690}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.177:6802', 'nonce': 3367751690}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.177:6804', 'nonce': 3367751690}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.177:6803', 'nonce': 3367751690}]}, 'public_addr': '172.21.15.177:6801/3367751690', 'cluster_addr': '172.21.15.177:6802/3367751690', 'heartbeat_back_addr': '172.21.15.177:6804/3367751690', 'heartbeat_front_addr': '172.21.15.177:6803/3367751690', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': 'd1fe3f63-df3f-4a73-a1e2-8d7c8a3f1fa0', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 22, 'up_thru': 227, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.1:6808', 'nonce': 3648540926}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.1:6809', 'nonce': 3648540926}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.1:6811', 'nonce': 3648540926}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.1:6810', 'nonce': 3648540926}]}, 'public_addr': '172.21.15.1:6808/3648540926', 'cluster_addr': '172.21.15.1:6809/3648540926', 'heartbeat_back_addr': '172.21.15.1:6811/3648540926', 'heartbeat_front_addr': '172.21.15.1:6810/3648540926', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': 'e2ead447-2f2c-40c9-82e7-53b8dc41e2af', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 21, 'up_thru': 227, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.92:6804', 'nonce': 3399529091}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.92:6805', 'nonce': 3399529091}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.92:6807', 'nonce': 3399529091}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.92:6806', 'nonce': 3399529091}]}, 'public_addr': '172.21.15.92:6804/3399529091', 'cluster_addr': '172.21.15.92:6805/3399529091', 'heartbeat_back_addr': '172.21.15.92:6807/3399529091', 'heartbeat_front_addr': '172.21.15.92:6806/3399529091', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': 'dbc20439-144e-4ec7-89de-674eb783f263', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 22, 'up_thru': 226, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.162:6808', 'nonce': 2287343806}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.162:6809', 'nonce': 2287343806}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.162:6811', 'nonce': 2287343806}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.162:6810', 'nonce': 2287343806}]}, 'public_addr': '172.21.15.162:6808/2287343806', 'cluster_addr': '172.21.15.162:6809/2287343806', 'heartbeat_back_addr': '172.21.15.162:6811/2287343806', 'heartbeat_front_addr': '172.21.15.162:6810/2287343806', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '60aa5206-cace-40f8-8d1a-10ec28d97096', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 22, 'up_thru': 227, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.177:6813', 'nonce': 2110809906}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.177:6814', 'nonce': 2110809906}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.177:6816', 'nonce': 2110809906}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.177:6815', 'nonce': 2110809906}]}, 'public_addr': '172.21.15.177:6813/2110809906', 'cluster_addr': '172.21.15.177:6814/2110809906', 'heartbeat_back_addr': '172.21.15.177:6816/2110809906', 'heartbeat_front_addr': '172.21.15.177:6815/2110809906', 'state': ['exists', 'up']}, {'osd': 8, 'uuid': 'f3d99325-48ff-403c-8662-1d7745f7c97d', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 22, 'up_thru': 225, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.1:6804', 'nonce': 2766567910}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.1:6805', 'nonce': 2766567910}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.1:6807', 'nonce': 2766567910}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.1:6806', 'nonce': 2766567910}]}, 'public_addr': '172.21.15.1:6804/2766567910', 'cluster_addr': '172.21.15.1:6805/2766567910', 'heartbeat_back_addr': '172.21.15.1:6807/2766567910', 'heartbeat_front_addr': '172.21.15.1:6806/2766567910', 'state': ['exists', 'up']}, {'osd': 9, 'uuid': '88511e24-3675-4551-968c-1a52e7e28f96', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 23, 'up_thru': 191, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.92:6812', 'nonce': 2488187511}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.92:6813', 'nonce': 2488187511}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.92:6815', 'nonce': 2488187511}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.92:6814', 'nonce': 2488187511}]}, 'public_addr': '172.21.15.92:6812/2488187511', 'cluster_addr': '172.21.15.92:6813/2488187511', 'heartbeat_back_addr': '172.21.15.92:6815/2488187511', 'heartbeat_front_addr': '172.21.15.92:6814/2488187511', 'state': ['exists', 'up']}, {'osd': 10, 'uuid': 'acf4c64d-99af-42be-8406-413c86f54670', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 22, 'up_thru': 226, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.162:6804', 'nonce': 1450266459}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.162:6805', 'nonce': 1450266459}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.162:6807', 'nonce': 1450266459}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.162:6806', 'nonce': 1450266459}]}, 'public_addr': '172.21.15.162:6804/1450266459', 'cluster_addr': '172.21.15.162:6805/1450266459', 'heartbeat_back_addr': '172.21.15.162:6807/1450266459', 'heartbeat_front_addr': '172.21.15.162:6806/1450266459', 'state': ['exists', 'up']}, {'osd': 11, 'uuid': 'a4bfdfbe-b586-4080-8e09-6f26428e59b5', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 22, 'up_thru': 227, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.177:6809', 'nonce': 3239521954}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.177:6810', 'nonce': 3239521954}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.177:6812', 'nonce': 3239521954}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.177:6811', 'nonce': 3239521954}]}, 'public_addr': '172.21.15.177:6809/3239521954', 'cluster_addr': '172.21.15.177:6810/3239521954', 'heartbeat_back_addr': '172.21.15.177:6812/3239521954', 'heartbeat_front_addr': '172.21.15.177:6811/3239521954', 'state': ['exists', 'up']}, {'osd': 12, 'uuid': 'dc571fb6-d8b5-465b-bb25-afa7d0c47b5a', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 22, 'up_thru': 141, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.1:6812', 'nonce': 4267062742}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.1:6813', 'nonce': 4267062742}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.1:6815', 'nonce': 4267062742}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.1:6814', 'nonce': 4267062742}]}, 'public_addr': '172.21.15.1:6812/4267062742', 'cluster_addr': '172.21.15.1:6813/4267062742', 'heartbeat_back_addr': '172.21.15.1:6815/4267062742', 'heartbeat_front_addr': '172.21.15.1:6814/4267062742', 'state': ['exists', 'up']}, {'osd': 13, 'uuid': '5d321d4b-1b26-47fe-bf84-25751b8cde7d', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 23, 'up_thru': 205, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.92:6808', 'nonce': 1759585046}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.92:6809', 'nonce': 1759585046}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.92:6811', 'nonce': 1759585046}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.92:6810', 'nonce': 1759585046}]}, 'public_addr': '172.21.15.92:6808/1759585046', 'cluster_addr': '172.21.15.92:6809/1759585046', 'heartbeat_back_addr': '172.21.15.92:6811/1759585046', 'heartbeat_front_addr': '172.21.15.92:6810/1759585046', 'state': ['exists', 'up']}, {'osd': 14, 'uuid': 'fc3e2499-0c2a-414d-a25e-2628876ae630', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.8266143798828125, 'last_clean_begin': 22, 'last_clean_end': 120, 'up_from': 142, 'up_thru': 226, 'down_at': 121, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.162:6800', 'nonce': 3999392664}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.162:6801', 'nonce': 3999392664}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.162:6803', 'nonce': 3999392664}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.162:6802', 'nonce': 3999392664}]}, 'public_addr': '172.21.15.162:6800/3999392664', 'cluster_addr': '172.21.15.162:6801/3999392664', 'heartbeat_back_addr': '172.21.15.162:6803/3999392664', 'heartbeat_front_addr': '172.21.15.162:6802/3999392664', 'state': ['exists', 'up']}, {'osd': 15, 'uuid': 'd2f69775-edfe-4527-91ec-f7c7e558bec8', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 22, 'up_thru': 122, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.177:6805', 'nonce': 1294337172}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.177:6806', 'nonce': 1294337172}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.177:6808', 'nonce': 1294337172}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.177:6807', 'nonce': 1294337172}]}, 'public_addr': '172.21.15.177:6805/1294337172', 'cluster_addr': '172.21.15.177:6806/1294337172', 'heartbeat_back_addr': '172.21.15.177:6808/1294337172', 'heartbeat_front_addr': '172.21.15.177:6807/1294337172', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-13T11:26:19.725882+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-13T11:25:47.780833+0000', 'dead_epoch': 30}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-13T11:26:18.925695+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-13T11:26:17.200385+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-13T11:25:59.330894+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-13T11:26:21.996515+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-13T11:26:16.679806+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-13T11:26:01.857996+0000', 'dead_epoch': 0}, {'osd': 8, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-13T11:26:14.379046+0000', 'dead_epoch': 0}, {'osd': 9, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-13T11:26:09.278966+0000', 'dead_epoch': 0}, {'osd': 10, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-13T11:26:02.999347+0000', 'dead_epoch': 0}, {'osd': 11, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-13T11:26:27.208176+0000', 'dead_epoch': 0}, {'osd': 12, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-13T11:25:47.548386+0000', 'dead_epoch': 0}, {'osd': 13, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-13T11:25:45.821466+0000', 'dead_epoch': 0}, {'osd': 14, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-13T11:26:03.927526+0000', 'dead_epoch': 121}, {'osd': 15, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-13T11:25:57.987079+0000', 'dead_epoch': 0}], 'pg_upmap': [{'pgid': '3.e', 'osds': [0, 1]}, {'pgid': '3.22', 'osds': [14, 12]}], 'pg_upmap_items': [], 'pg_upmap_primaries': [], 'pg_temp': [{'pgid': '3.12', 'osds': [5, 7]}, {'pgid': '3.22', 'osds': [0, 14]}, {'pgid': '3.2e', 'osds': [0, 9]}, {'pgid': '3.2f', 'osds': [11, 4]}, {'pgid': '3.30', 'osds': [0, 3]}, {'pgid': '3.31', 'osds': [4, 1]}, {'pgid': '3.32', 'osds': [5, 0]}, {'pgid': '3.33', 'osds': [7, 2]}, {'pgid': '3.34', 'osds': [3, 10]}, {'pgid': '3.35', 'osds': [0, 7]}, {'pgid': '3.36', 'osds': [11, 0]}, {'pgid': '3.37', 'osds': [3, 2]}], 'primary_temp': [], 'blocklist': {}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-09-13T11:26:53.661 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-13T11:26:53.662 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-09-13T11:26:53.662 INFO:tasks.ceph.osd.9.smithi092.stderr:2024-09-13T11:26:53.655+0000 7ff5ab3d0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26348) UID: 0 2024-09-13T11:26:53.760 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-09-13T11:26:53.755+0000 7f97befc9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26315) UID: 0 2024-09-13T11:26:53.841 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-09-13T11:26:53.842 INFO:teuthology.orchestra.run.smithi092.stdout: "ops": [], 2024-09-13T11:26:53.842 INFO:teuthology.orchestra.run.smithi092.stdout: "complaint_time": 30, 2024-09-13T11:26:53.842 INFO:teuthology.orchestra.run.smithi092.stdout: "num_blocked_ops": 0 2024-09-13T11:26:53.842 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-09-13T11:26:53.858 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-09-13T11:26:53.863 INFO:tasks.ceph.osd.4.smithi001.stderr:2024-09-13T11:26:53.858+0000 7ff8e0218640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26310) UID: 0 2024-09-13T11:26:53.960 INFO:tasks.ceph.osd.10.smithi162.stderr:2024-09-13T11:26:53.959+0000 7fb59c9f0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26317) UID: 0 2024-09-13T11:26:53.995 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-09-13T11:26:53.995 INFO:teuthology.orchestra.run.smithi092.stdout: "size": 20, 2024-09-13T11:26:53.995 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 600, 2024-09-13T11:26:53.995 INFO:teuthology.orchestra.run.smithi092.stdout: "ops": [ 2024-09-13T11:26:53.995 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:53.996 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "MOSDPGPush(3.19 76/70 [PushOp(3:9eacf3d0:::benchmark_data_smithi177_31116_object19747:head, version: 54'1236, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:9eacf3d0:::benchmark_data_smithi177_31116_object19747:head@54'1236, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:26:53.996 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:32.704676+0000", 2024-09-13T11:26:53.996 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 201.28940995799999, 2024-09-13T11:26:53.996 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.23525014899999999, 2024-09-13T11:26:53.996 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:26:53.996 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:26:53.996 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-09-13T11:26:53.996 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:26:53.996 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:53.996 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:26:53.996 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:32.704676+0000", 2024-09-13T11:26:53.996 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:53.997 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:53.997 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:53.997 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:26:53.997 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:32.704676+0000", 2024-09-13T11:26:53.997 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:53.997 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:53.997 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:53.997 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:26:53.997 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:32.704677+0000", 2024-09-13T11:26:53.997 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.531e-06 2024-09-13T11:26:53.997 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:53.997 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:53.998 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:26:53.998 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:32.704749+0000", 2024-09-13T11:26:53.998 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.1451000000000004e-05 2024-09-13T11:26:53.998 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:53.998 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:53.998 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:26:53.998 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:32.938053+0000", 2024-09-13T11:26:53.998 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.233304752 2024-09-13T11:26:53.998 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:53.998 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:53.998 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:26:53.998 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:32.938087+0000", 2024-09-13T11:26:53.998 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.4016000000000001e-05 2024-09-13T11:26:53.999 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:53.999 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:53.999 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:53.999 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:32.938351+0000", 2024-09-13T11:26:53.999 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00026397500000000001 2024-09-13T11:26:53.999 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:53.999 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:53.999 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:26:53.999 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:32.938381+0000", 2024-09-13T11:26:53.999 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.0001999999999998e-05 2024-09-13T11:26:53.999 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:53.999 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:53.999 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:26:53.999 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:32.939926+0000", 2024-09-13T11:26:54.000 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0015444219999999999 2024-09-13T11:26:54.000 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:54.000 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:54.000 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:54.000 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.000 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.000 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "MOSDPGPush(3.23 80/78 [PushOp(3:c59ecc6d:::benchmark_data_smithi177_31116_object17005:head, version: 52'1054, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:c59ecc6d:::benchmark_data_smithi177_31116_object17005:head@52'1054, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:26:54.000 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:37.433040+0000", 2024-09-13T11:26:54.000 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 196.561045982, 2024-09-13T11:26:54.000 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.26658219100000002, 2024-09-13T11:26:54.000 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:26:54.000 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:26:54.000 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-09-13T11:26:54.001 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:26:54.001 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.001 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:26:54.001 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:37.433040+0000", 2024-09-13T11:26:54.001 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:54.001 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.001 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.001 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:26:54.001 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:37.433040+0000", 2024-09-13T11:26:54.001 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:54.001 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.001 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.001 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:26:54.001 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:37.433042+0000", 2024-09-13T11:26:54.002 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.3999999999999999e-06 2024-09-13T11:26:54.002 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.002 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.002 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:26:54.002 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:37.433090+0000", 2024-09-13T11:26:54.002 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.8065999999999997e-05 2024-09-13T11:26:54.002 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.002 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.002 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:26:54.002 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:37.698797+0000", 2024-09-13T11:26:54.002 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.26570674500000002 2024-09-13T11:26:54.002 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.002 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.002 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:26:54.003 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:37.698827+0000", 2024-09-13T11:26:54.003 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.0505e-05 2024-09-13T11:26:54.003 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.003 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.003 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:54.003 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:37.698941+0000", 2024-09-13T11:26:54.003 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00011403799999999999 2024-09-13T11:26:54.003 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.003 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.003 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:26:54.003 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:37.698956+0000", 2024-09-13T11:26:54.003 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.4765e-05 2024-09-13T11:26:54.003 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.003 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.004 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:26:54.004 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:37.699622+0000", 2024-09-13T11:26:54.004 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00066567199999999997 2024-09-13T11:26:54.004 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:54.004 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:54.004 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:54.004 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.004 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.004 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "MOSDPGPush(3.1a 85/78 [PushOp(3:5dc85525:::benchmark_data_smithi177_31116_object11755:head, version: 49'723, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:5dc85525:::benchmark_data_smithi177_31116_object11755:head@49'723, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:26:54.004 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:42.422061+0000", 2024-09-13T11:26:54.004 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 191.572024529, 2024-09-13T11:26:54.004 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.52875617500000005, 2024-09-13T11:26:54.004 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:26:54.004 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:26:54.005 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-09-13T11:26:54.005 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:26:54.005 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.005 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:26:54.005 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.422061+0000", 2024-09-13T11:26:54.005 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:54.005 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.005 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.005 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:26:54.005 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.422061+0000", 2024-09-13T11:26:54.005 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:54.005 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.005 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.005 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:26:54.006 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.422064+0000", 2024-09-13T11:26:54.006 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.5409999999999999e-06 2024-09-13T11:26:54.006 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.006 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.006 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:26:54.006 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.422139+0000", 2024-09-13T11:26:54.006 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.5488999999999995e-05 2024-09-13T11:26:54.006 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.006 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.006 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:26:54.006 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.948781+0000", 2024-09-13T11:26:54.006 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.52664230999999995 2024-09-13T11:26:54.006 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.006 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.007 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:26:54.007 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.948841+0000", 2024-09-13T11:26:54.007 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.9450999999999997e-05 2024-09-13T11:26:54.007 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.007 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.007 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:54.007 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.949035+0000", 2024-09-13T11:26:54.007 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00019380600000000001 2024-09-13T11:26:54.007 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.007 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.007 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:26:54.007 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.949060+0000", 2024-09-13T11:26:54.008 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.5381999999999999e-05 2024-09-13T11:26:54.008 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.008 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.008 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:26:54.008 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.950817+0000", 2024-09-13T11:26:54.008 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0017571959999999999 2024-09-13T11:26:54.008 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:54.008 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:54.008 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:54.008 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.008 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.008 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "MOSDPGPush(3.1a 89/78 [PushOp(3:59270734:::benchmark_data_smithi177_31116_object20039:head, version: 54'1256, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:59270734:::benchmark_data_smithi177_31116_object20039:head@54'1256, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:26:54.008 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:46.607562+0000", 2024-09-13T11:26:54.008 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 187.38652375199999, 2024-09-13T11:26:54.009 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.32123945300000001, 2024-09-13T11:26:54.009 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:26:54.009 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:26:54.009 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-09-13T11:26:54.009 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:26:54.009 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.009 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:26:54.009 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.607562+0000", 2024-09-13T11:26:54.009 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:54.009 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.009 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.009 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:26:54.009 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.607562+0000", 2024-09-13T11:26:54.009 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:54.010 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.010 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.010 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:26:54.010 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.607564+0000", 2024-09-13T11:26:54.010 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.8610000000000001e-06 2024-09-13T11:26:54.010 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.010 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.010 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:26:54.010 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.607663+0000", 2024-09-13T11:26:54.010 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.9210999999999996e-05 2024-09-13T11:26:54.010 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.010 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.010 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:26:54.010 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.925193+0000", 2024-09-13T11:26:54.010 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.317530586 2024-09-13T11:26:54.011 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.011 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.011 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:26:54.011 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.925250+0000", 2024-09-13T11:26:54.011 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.6925000000000001e-05 2024-09-13T11:26:54.011 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.011 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.011 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:54.011 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.927013+0000", 2024-09-13T11:26:54.011 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.001762853 2024-09-13T11:26:54.011 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.011 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.011 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:26:54.012 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.927067+0000", 2024-09-13T11:26:54.012 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.3601999999999999e-05 2024-09-13T11:26:54.012 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.012 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.012 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:26:54.012 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.928801+0000", 2024-09-13T11:26:54.012 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.001734415 2024-09-13T11:26:54.012 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:54.012 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:54.012 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:54.012 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.012 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.012 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "MOSDPGPush(3.1a 91/78 [PushOp(3:5e5010d2:::benchmark_data_smithi177_31116_object25733:head, version: 58'1609, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:5e5010d2:::benchmark_data_smithi177_31116_object25733:head@58'1609, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:26:54.013 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:48.608485+0000", 2024-09-13T11:26:54.013 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 185.38560063400001, 2024-09-13T11:26:54.013 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.35035599699999997, 2024-09-13T11:26:54.013 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:26:54.013 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:26:54.013 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-09-13T11:26:54.013 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:26:54.013 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.013 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:26:54.013 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:48.608485+0000", 2024-09-13T11:26:54.013 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:54.013 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.013 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.014 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:26:54.014 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:48.608485+0000", 2024-09-13T11:26:54.014 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:54.014 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.014 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.014 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:26:54.014 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:48.608488+0000", 2024-09-13T11:26:54.014 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.9399999999999998e-06 2024-09-13T11:26:54.014 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.014 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.014 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:26:54.014 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:48.608552+0000", 2024-09-13T11:26:54.014 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.3731000000000004e-05 2024-09-13T11:26:54.015 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.015 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.015 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:26:54.015 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:48.957330+0000", 2024-09-13T11:26:54.015 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.34877806099999997 2024-09-13T11:26:54.015 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.015 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.015 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:26:54.015 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:48.957361+0000", 2024-09-13T11:26:54.015 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.0852000000000002e-05 2024-09-13T11:26:54.015 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.015 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.015 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:54.015 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:48.957544+0000", 2024-09-13T11:26:54.016 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000183537 2024-09-13T11:26:54.016 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.016 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.016 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:26:54.016 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:48.957582+0000", 2024-09-13T11:26:54.016 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.7472999999999998e-05 2024-09-13T11:26:54.016 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.016 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.016 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:26:54.016 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:48.958841+0000", 2024-09-13T11:26:54.016 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0012594030000000001 2024-09-13T11:26:54.016 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:54.016 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:54.016 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:54.017 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.017 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.017 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop(client.4665.0:72923 3.23 e97/90 3:c6230123:::benchmark_data_smithi177_31116_object72922:head v 97'3418, mlcod=97'3414)", 2024-09-13T11:26:54.017 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:55.047734+0000", 2024-09-13T11:26:54.017 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 178.94635111700001, 2024-09-13T11:26:54.017 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.20075528000000001, 2024-09-13T11:26:54.017 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:26:54.017 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:26:54.017 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:54.017 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:26:54.017 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.017 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:26:54.017 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:55.047734+0000", 2024-09-13T11:26:54.018 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:54.018 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.018 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.018 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:26:54.018 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:55.047734+0000", 2024-09-13T11:26:54.018 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:54.018 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.018 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.018 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:26:54.018 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:55.047736+0000", 2024-09-13T11:26:54.018 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.466e-06 2024-09-13T11:26:54.018 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.018 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.018 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:26:54.019 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:55.047772+0000", 2024-09-13T11:26:54.019 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.6576999999999999e-05 2024-09-13T11:26:54.019 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.019 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.019 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:26:54.019 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:55.246478+0000", 2024-09-13T11:26:54.019 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.19870581300000001 2024-09-13T11:26:54.019 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.019 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.019 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:26:54.019 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:55.246488+0000", 2024-09-13T11:26:54.019 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.5419999999999994e-06 2024-09-13T11:26:54.019 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.020 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.020 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:54.020 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:55.246631+0000", 2024-09-13T11:26:54.020 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00014287300000000001 2024-09-13T11:26:54.020 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.020 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.020 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:26:54.020 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:55.246665+0000", 2024-09-13T11:26:54.020 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.4684999999999997e-05 2024-09-13T11:26:54.020 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.020 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.020 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:26:54.020 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:55.248453+0000", 2024-09-13T11:26:54.020 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0017879459999999999 2024-09-13T11:26:54.021 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.021 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.021 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:26:54.021 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:55.248490+0000", 2024-09-13T11:26:54.021 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.6378000000000003e-05 2024-09-13T11:26:54.021 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:54.021 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:54.021 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:54.021 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.021 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.021 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop(client.4665.0:98582 3.1a e113/105 3:59f9cd75:::benchmark_data_smithi177_31116_object98581:head v 113'4743, mlcod=113'4741)", 2024-09-13T11:26:54.021 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:24:15.432027+0000", 2024-09-13T11:26:54.021 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 158.56205850200001, 2024-09-13T11:26:54.021 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.196879784, 2024-09-13T11:26:54.022 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:26:54.022 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:26:54.022 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:54.022 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:26:54.022 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.022 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:26:54.022 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:15.432027+0000", 2024-09-13T11:26:54.022 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:54.022 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.022 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.022 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:26:54.022 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:15.432027+0000", 2024-09-13T11:26:54.022 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:54.022 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.023 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.023 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:26:54.023 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:15.432029+0000", 2024-09-13T11:26:54.023 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.838e-06 2024-09-13T11:26:54.023 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.023 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.023 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:26:54.023 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:15.432088+0000", 2024-09-13T11:26:54.023 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.8767999999999997e-05 2024-09-13T11:26:54.023 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.023 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.023 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:26:54.023 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:15.624731+0000", 2024-09-13T11:26:54.023 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.192643221 2024-09-13T11:26:54.023 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.023 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.024 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:26:54.024 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:15.624746+0000", 2024-09-13T11:26:54.024 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.4705000000000001e-05 2024-09-13T11:26:54.024 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.024 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.024 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:54.024 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:15.624882+0000", 2024-09-13T11:26:54.024 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00013602300000000001 2024-09-13T11:26:54.024 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.024 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.024 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:26:54.024 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:15.624941+0000", 2024-09-13T11:26:54.024 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.9351000000000001e-05 2024-09-13T11:26:54.024 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.025 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.025 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:26:54.025 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:15.628836+0000", 2024-09-13T11:26:54.025 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0038955109999999999 2024-09-13T11:26:54.025 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.025 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.025 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:26:54.025 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:15.628907+0000", 2024-09-13T11:26:54.025 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.0366999999999999e-05 2024-09-13T11:26:54.025 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:54.025 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:54.025 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:54.026 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.026 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.026 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop(client.4665.0:99105 3.1a e113/105 3:5ac5090c:::benchmark_data_smithi177_31116_object99104:head v 113'4749, mlcod=113'4746)", 2024-09-13T11:26:54.026 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:24:15.963602+0000", 2024-09-13T11:26:54.026 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 158.03048332, 2024-09-13T11:26:54.026 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.23699906300000001, 2024-09-13T11:26:54.026 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:26:54.026 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:26:54.026 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:54.026 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:26:54.026 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.026 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:26:54.027 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:15.963602+0000", 2024-09-13T11:26:54.027 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:54.027 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.027 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.027 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:26:54.027 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:15.963602+0000", 2024-09-13T11:26:54.027 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:54.027 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.027 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.027 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:26:54.027 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:15.963605+0000", 2024-09-13T11:26:54.027 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.0400000000000001e-06 2024-09-13T11:26:54.027 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.028 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.028 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:26:54.028 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:15.963718+0000", 2024-09-13T11:26:54.028 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000112796 2024-09-13T11:26:54.028 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.028 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.028 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:26:54.028 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:16.197994+0000", 2024-09-13T11:26:54.028 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.234276285 2024-09-13T11:26:54.028 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.028 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.028 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:26:54.028 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:16.198004+0000", 2024-09-13T11:26:54.029 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.5559999999999996e-06 2024-09-13T11:26:54.029 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.029 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.029 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:54.029 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:16.198110+0000", 2024-09-13T11:26:54.029 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00010614 2024-09-13T11:26:54.029 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.029 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.029 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:26:54.029 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:16.198152+0000", 2024-09-13T11:26:54.029 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.1899e-05 2024-09-13T11:26:54.029 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.030 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.030 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:26:54.030 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:16.200528+0000", 2024-09-13T11:26:54.030 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00237587 2024-09-13T11:26:54.030 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.030 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.030 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:26:54.030 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:16.200601+0000", 2024-09-13T11:26:54.030 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.3477000000000001e-05 2024-09-13T11:26:54.030 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:54.030 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:54.030 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:54.030 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.030 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.031 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop(client.4665.0:129921 3.1a e134/105 3:5fb9088b:::benchmark_data_smithi177_31116_object129920:head v 134'5758, mlcod=134'5755)", 2024-09-13T11:26:54.031 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:24:47.178021+0000", 2024-09-13T11:26:54.031 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 126.816064314, 2024-09-13T11:26:54.031 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.33723278600000001, 2024-09-13T11:26:54.031 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:26:54.031 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:26:54.031 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:54.031 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:26:54.031 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.031 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:26:54.031 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:47.178021+0000", 2024-09-13T11:26:54.031 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:54.032 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.032 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.032 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:26:54.032 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:47.178021+0000", 2024-09-13T11:26:54.032 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:54.032 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.032 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.032 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:26:54.032 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:47.178024+0000", 2024-09-13T11:26:54.032 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.3609999999999999e-06 2024-09-13T11:26:54.032 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.032 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.032 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:26:54.032 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:47.178102+0000", 2024-09-13T11:26:54.033 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.8325000000000006e-05 2024-09-13T11:26:54.033 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.033 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.033 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:26:54.033 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:47.512707+0000", 2024-09-13T11:26:54.033 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.33460516000000001 2024-09-13T11:26:54.033 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.033 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.033 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:26:54.033 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:47.512716+0000", 2024-09-13T11:26:54.033 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.0389999999999993e-06 2024-09-13T11:26:54.033 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.034 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.034 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:54.034 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:47.512788+0000", 2024-09-13T11:26:54.034 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.1840000000000003e-05 2024-09-13T11:26:54.034 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.034 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.034 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:26:54.034 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:47.512809+0000", 2024-09-13T11:26:54.034 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.1005e-05 2024-09-13T11:26:54.034 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.034 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.034 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:26:54.034 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:47.515182+0000", 2024-09-13T11:26:54.034 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0023731939999999999 2024-09-13T11:26:54.034 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.035 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.035 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:26:54.035 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:47.515254+0000", 2024-09-13T11:26:54.035 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.1861999999999997e-05 2024-09-13T11:26:54.035 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:54.035 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:54.035 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:54.035 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.035 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.035 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop(client.4665.0:138370 3.23 e140/90 3:c5730cd2:::benchmark_data_smithi177_31116_object138369:head v 140'4458, mlcod=140'4455)", 2024-09-13T11:26:54.035 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:24:54.416118+0000", 2024-09-13T11:26:54.035 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 119.577967836, 2024-09-13T11:26:54.035 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.244024991, 2024-09-13T11:26:54.036 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:26:54.036 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:26:54.036 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:54.036 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:26:54.036 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.036 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:26:54.036 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:54.416118+0000", 2024-09-13T11:26:54.036 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:54.036 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.036 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.036 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:26:54.036 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:54.416118+0000", 2024-09-13T11:26:54.037 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:54.037 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.037 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.037 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:26:54.037 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:54.416120+0000", 2024-09-13T11:26:54.037 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.6280000000000001e-06 2024-09-13T11:26:54.037 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.037 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.037 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:26:54.037 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:54.416161+0000", 2024-09-13T11:26:54.037 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.0877000000000002e-05 2024-09-13T11:26:54.037 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.037 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.038 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:26:54.038 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:54.657128+0000", 2024-09-13T11:26:54.038 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.24096696300000001 2024-09-13T11:26:54.038 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.038 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.038 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:26:54.038 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:54.657142+0000", 2024-09-13T11:26:54.038 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.358e-05 2024-09-13T11:26:54.038 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.038 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.038 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:54.038 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:54.657290+0000", 2024-09-13T11:26:54.039 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00014794400000000001 2024-09-13T11:26:54.039 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.039 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.039 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:26:54.039 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:54.657325+0000", 2024-09-13T11:26:54.039 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.5623000000000001e-05 2024-09-13T11:26:54.039 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.039 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.039 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:26:54.039 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:54.660079+0000", 2024-09-13T11:26:54.039 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0027536489999999999 2024-09-13T11:26:54.039 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.039 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.039 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:26:54.040 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:54.660143+0000", 2024-09-13T11:26:54.040 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.3726999999999995e-05 2024-09-13T11:26:54.040 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:54.040 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:54.040 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:54.040 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.040 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.040 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:209225 3.24 3:2478ed54:::benchmark_data_smithi177_31116_object68922:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e185)", 2024-09-13T11:26:54.040 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:25:53.632417+0000", 2024-09-13T11:26:54.040 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 60.361668403000003, 2024-09-13T11:26:54.040 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.219758709, 2024-09-13T11:26:54.040 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:26:54.040 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:26:54.041 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:54.041 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:26:54.041 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:26:54.041 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:54.041 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 209225 2024-09-13T11:26:54.041 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.041 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:26:54.041 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.041 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:26:54.041 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.632417+0000", 2024-09-13T11:26:54.041 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:54.041 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.042 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.042 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:26:54.042 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.632417+0000", 2024-09-13T11:26:54.042 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:54.042 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.042 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.042 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:26:54.042 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.632420+0000", 2024-09-13T11:26:54.042 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.4980000000000001e-06 2024-09-13T11:26:54.042 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.042 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.042 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:26:54.042 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.632427+0000", 2024-09-13T11:26:54.043 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.5120000000000002e-06 2024-09-13T11:26:54.043 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.043 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.043 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:26:54.043 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.632441+0000", 2024-09-13T11:26:54.043 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.4072e-05 2024-09-13T11:26:54.043 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.043 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.043 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:26:54.043 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.632454+0000", 2024-09-13T11:26:54.043 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.2886e-05 2024-09-13T11:26:54.044 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.044 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.044 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:54.044 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.632529+0000", 2024-09-13T11:26:54.044 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.5326000000000004e-05 2024-09-13T11:26:54.044 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.044 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.044 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:26:54.044 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.632642+0000", 2024-09-13T11:26:54.044 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00011249200000000001 2024-09-13T11:26:54.044 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.044 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.044 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1", 2024-09-13T11:26:54.044 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.632712+0000", 2024-09-13T11:26:54.045 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.0376000000000005e-05 2024-09-13T11:26:54.045 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.045 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.045 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:26:54.045 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.635270+0000", 2024-09-13T11:26:54.045 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0025574339999999999 2024-09-13T11:26:54.045 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.045 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.045 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:54.045 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.851915+0000", 2024-09-13T11:26:54.045 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.21664515400000001 2024-09-13T11:26:54.045 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.045 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.045 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:26:54.046 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.852065+0000", 2024-09-13T11:26:54.046 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00014972799999999999 2024-09-13T11:26:54.046 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.046 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.046 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:26:54.046 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.852176+0000", 2024-09-13T11:26:54.046 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00011123100000000001 2024-09-13T11:26:54.046 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:54.046 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:54.046 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:54.046 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.046 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.047 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:209229 3.19 3:9a976dcb:::benchmark_data_smithi177_31116_object68926:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e185)", 2024-09-13T11:26:54.047 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:25:53.633903+0000", 2024-09-13T11:26:54.047 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 60.360182498, 2024-09-13T11:26:54.047 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.25298130200000002, 2024-09-13T11:26:54.047 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:26:54.047 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:26:54.047 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:54.047 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:26:54.047 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:26:54.047 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:54.047 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 209229 2024-09-13T11:26:54.047 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.047 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:26:54.047 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.048 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:26:54.048 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.633903+0000", 2024-09-13T11:26:54.048 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:54.048 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.048 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.048 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:26:54.048 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.633903+0000", 2024-09-13T11:26:54.048 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:54.048 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.048 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.048 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:26:54.048 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.633905+0000", 2024-09-13T11:26:54.048 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.1600000000000001e-06 2024-09-13T11:26:54.048 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.049 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.049 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:26:54.049 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.633917+0000", 2024-09-13T11:26:54.049 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.1881e-05 2024-09-13T11:26:54.049 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.049 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.049 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:26:54.049 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.633931+0000", 2024-09-13T11:26:54.049 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.432e-05 2024-09-13T11:26:54.049 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.049 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.049 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:26:54.049 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.633941+0000", 2024-09-13T11:26:54.050 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.8079999999999999e-06 2024-09-13T11:26:54.050 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.050 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.050 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:54.050 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.634009+0000", 2024-09-13T11:26:54.050 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.7895e-05 2024-09-13T11:26:54.050 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.050 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.050 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:26:54.050 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.634137+0000", 2024-09-13T11:26:54.050 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000127644 2024-09-13T11:26:54.050 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.050 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.050 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1", 2024-09-13T11:26:54.050 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.634245+0000", 2024-09-13T11:26:54.051 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00010776399999999999 2024-09-13T11:26:54.051 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.051 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.051 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:26:54.051 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.637244+0000", 2024-09-13T11:26:54.051 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0029993149999999998 2024-09-13T11:26:54.051 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.051 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.051 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:54.051 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.886675+0000", 2024-09-13T11:26:54.051 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.249431456 2024-09-13T11:26:54.051 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.051 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.051 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:26:54.052 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.886785+0000", 2024-09-13T11:26:54.052 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000109326 2024-09-13T11:26:54.052 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.052 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.052 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:26:54.052 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.886884+0000", 2024-09-13T11:26:54.052 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.9733000000000002e-05 2024-09-13T11:26:54.052 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:54.052 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:54.052 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:54.052 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.052 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.052 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:209252 3.24 3:25ab37ee:::benchmark_data_smithi177_31116_object68949:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e185)", 2024-09-13T11:26:54.052 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:25:53.647942+0000", 2024-09-13T11:26:54.053 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 60.346143425999998, 2024-09-13T11:26:54.053 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.21571831799999999, 2024-09-13T11:26:54.053 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:26:54.053 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:26:54.053 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:54.053 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:26:54.053 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:26:54.053 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:54.053 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 209252 2024-09-13T11:26:54.053 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.053 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:26:54.053 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.053 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:26:54.054 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.647942+0000", 2024-09-13T11:26:54.054 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:54.054 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.054 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.054 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:26:54.054 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.647942+0000", 2024-09-13T11:26:54.054 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:54.054 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.054 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.054 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:26:54.054 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.647945+0000", 2024-09-13T11:26:54.054 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.717e-06 2024-09-13T11:26:54.054 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.054 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.054 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:26:54.055 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.647951+0000", 2024-09-13T11:26:54.055 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.353e-06 2024-09-13T11:26:54.055 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.055 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.055 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:26:54.055 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.647966+0000", 2024-09-13T11:26:54.055 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.4877e-05 2024-09-13T11:26:54.055 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.055 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.055 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:26:54.055 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.647973+0000", 2024-09-13T11:26:54.055 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.2640000000000002e-06 2024-09-13T11:26:54.056 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.056 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.056 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:54.056 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.648038+0000", 2024-09-13T11:26:54.056 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.4366e-05 2024-09-13T11:26:54.056 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.056 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.056 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:26:54.056 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.648131+0000", 2024-09-13T11:26:54.056 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.3017999999999996e-05 2024-09-13T11:26:54.056 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.056 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.056 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1", 2024-09-13T11:26:54.057 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.648200+0000", 2024-09-13T11:26:54.057 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.9244000000000003e-05 2024-09-13T11:26:54.057 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.057 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.057 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:26:54.057 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.649817+0000", 2024-09-13T11:26:54.057 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0016166920000000001 2024-09-13T11:26:54.057 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.057 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.057 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:54.057 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.863329+0000", 2024-09-13T11:26:54.057 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.21351207699999999 2024-09-13T11:26:54.057 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.058 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.058 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:26:54.058 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.863524+0000", 2024-09-13T11:26:54.058 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00019551 2024-09-13T11:26:54.058 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.058 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.058 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:26:54.058 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.863660+0000", 2024-09-13T11:26:54.058 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0001362 2024-09-13T11:26:54.058 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:54.058 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:54.058 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:54.058 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.059 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.059 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:209264 3.19 3:9fb0ed8f:::benchmark_data_smithi177_31116_object68961:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e185)", 2024-09-13T11:26:54.059 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:25:53.655043+0000", 2024-09-13T11:26:54.059 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 60.339042513000003, 2024-09-13T11:26:54.059 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.232156005, 2024-09-13T11:26:54.059 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:26:54.059 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:26:54.059 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:54.059 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:26:54.059 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:26:54.059 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:54.059 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 209264 2024-09-13T11:26:54.059 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.059 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:26:54.060 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.060 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:26:54.060 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.655043+0000", 2024-09-13T11:26:54.060 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:54.060 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.060 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.060 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:26:54.060 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.655043+0000", 2024-09-13T11:26:54.060 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:54.060 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.060 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.060 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:26:54.060 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.655045+0000", 2024-09-13T11:26:54.060 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.7090000000000001e-06 2024-09-13T11:26:54.061 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.061 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.061 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:26:54.061 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.655051+0000", 2024-09-13T11:26:54.061 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.3239999999999999e-06 2024-09-13T11:26:54.061 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.061 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.061 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:26:54.061 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.655065+0000", 2024-09-13T11:26:54.061 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.4137e-05 2024-09-13T11:26:54.061 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.061 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.062 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:26:54.062 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.655072+0000", 2024-09-13T11:26:54.062 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.827e-06 2024-09-13T11:26:54.062 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.062 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.062 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:54.062 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.655133+0000", 2024-09-13T11:26:54.062 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.0912000000000002e-05 2024-09-13T11:26:54.062 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.062 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.062 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:26:54.062 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.655228+0000", 2024-09-13T11:26:54.062 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.5440999999999996e-05 2024-09-13T11:26:54.062 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.063 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.063 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1", 2024-09-13T11:26:54.063 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.655296+0000", 2024-09-13T11:26:54.063 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.7316000000000004e-05 2024-09-13T11:26:54.063 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.063 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.063 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:26:54.063 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.658618+0000", 2024-09-13T11:26:54.063 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0033226179999999998 2024-09-13T11:26:54.063 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.063 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.063 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:54.063 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.887043+0000", 2024-09-13T11:26:54.063 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.22842430699999999 2024-09-13T11:26:54.063 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.064 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.064 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:26:54.064 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.887132+0000", 2024-09-13T11:26:54.064 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.9635000000000005e-05 2024-09-13T11:26:54.064 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.064 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.064 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:26:54.064 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.887199+0000", 2024-09-13T11:26:54.064 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.6779000000000006e-05 2024-09-13T11:26:54.064 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:54.064 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:54.064 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:54.064 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.064 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.064 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:209273 3.24 3:24d19ad8:::benchmark_data_smithi177_31116_object68970:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e185)", 2024-09-13T11:26:54.065 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:25:53.660601+0000", 2024-09-13T11:26:54.065 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 60.333484290000001, 2024-09-13T11:26:54.065 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.203427316, 2024-09-13T11:26:54.065 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:26:54.065 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:26:54.065 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:54.065 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:26:54.065 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:26:54.065 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:54.065 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 209273 2024-09-13T11:26:54.065 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.065 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:26:54.065 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.065 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:26:54.066 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.660601+0000", 2024-09-13T11:26:54.066 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:54.066 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.066 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.066 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:26:54.066 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.660601+0000", 2024-09-13T11:26:54.066 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:54.066 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.066 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.066 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:26:54.066 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.660603+0000", 2024-09-13T11:26:54.066 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.7039999999999999e-06 2024-09-13T11:26:54.066 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.066 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.067 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:26:54.067 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.660609+0000", 2024-09-13T11:26:54.067 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.5910000000000003e-06 2024-09-13T11:26:54.067 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.067 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.067 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:26:54.067 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.660623+0000", 2024-09-13T11:26:54.067 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.4338000000000001e-05 2024-09-13T11:26:54.067 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.067 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.067 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:26:54.067 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.660631+0000", 2024-09-13T11:26:54.067 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.1349999999999992e-06 2024-09-13T11:26:54.067 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.067 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.068 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:54.068 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.660694+0000", 2024-09-13T11:26:54.068 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.2755000000000005e-05 2024-09-13T11:26:54.068 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.068 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.068 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:26:54.068 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.660816+0000", 2024-09-13T11:26:54.068 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000121716 2024-09-13T11:26:54.068 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.068 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.068 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1", 2024-09-13T11:26:54.068 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.660900+0000", 2024-09-13T11:26:54.068 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.4924000000000002e-05 2024-09-13T11:26:54.068 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.068 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.069 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:26:54.069 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.662468+0000", 2024-09-13T11:26:54.069 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0015679279999999999 2024-09-13T11:26:54.069 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.069 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.069 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:54.069 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.863779+0000", 2024-09-13T11:26:54.069 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.201310406 2024-09-13T11:26:54.069 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.069 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.069 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:26:54.069 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.863921+0000", 2024-09-13T11:26:54.070 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00014240100000000001 2024-09-13T11:26:54.070 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.070 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.070 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:26:54.070 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.864029+0000", 2024-09-13T11:26:54.070 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00010741799999999999 2024-09-13T11:26:54.070 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:54.070 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:54.070 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:54.070 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.070 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.070 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:209298 3.19 3:9e1ecea7:::benchmark_data_smithi177_31116_object68995:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e185)", 2024-09-13T11:26:54.071 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:25:53.682311+0000", 2024-09-13T11:26:54.071 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 60.311774067999998, 2024-09-13T11:26:54.071 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.20511669699999999, 2024-09-13T11:26:54.071 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:26:54.071 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:26:54.071 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:54.071 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:26:54.071 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:26:54.071 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:54.071 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 209298 2024-09-13T11:26:54.071 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.072 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:26:54.072 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.072 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:26:54.072 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.682311+0000", 2024-09-13T11:26:54.072 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:54.072 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.072 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.072 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:26:54.072 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.682311+0000", 2024-09-13T11:26:54.072 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:54.072 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.073 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.073 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:26:54.073 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.682314+0000", 2024-09-13T11:26:54.073 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.588e-06 2024-09-13T11:26:54.073 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.073 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.073 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:26:54.073 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.682320+0000", 2024-09-13T11:26:54.073 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.3949999999999998e-06 2024-09-13T11:26:54.073 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.073 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.073 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:26:54.073 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.682335+0000", 2024-09-13T11:26:54.073 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.4789e-05 2024-09-13T11:26:54.074 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.074 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.074 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:26:54.074 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.682343+0000", 2024-09-13T11:26:54.074 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.3919999999999997e-06 2024-09-13T11:26:54.074 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.074 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.074 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:54.074 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.682456+0000", 2024-09-13T11:26:54.074 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00011341999999999999 2024-09-13T11:26:54.074 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.074 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.074 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:26:54.074 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.682583+0000", 2024-09-13T11:26:54.075 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00012655500000000001 2024-09-13T11:26:54.075 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.075 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.075 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1", 2024-09-13T11:26:54.075 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.682681+0000", 2024-09-13T11:26:54.075 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.8158000000000004e-05 2024-09-13T11:26:54.075 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.075 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.075 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:26:54.075 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.685667+0000", 2024-09-13T11:26:54.075 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0029857540000000002 2024-09-13T11:26:54.075 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.075 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.076 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:54.076 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.887298+0000", 2024-09-13T11:26:54.076 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.201631591 2024-09-13T11:26:54.076 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.076 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.076 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:26:54.076 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.887369+0000", 2024-09-13T11:26:54.076 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.0743999999999999e-05 2024-09-13T11:26:54.076 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.076 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.076 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:26:54.076 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.887428+0000", 2024-09-13T11:26:54.076 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.9311000000000002e-05 2024-09-13T11:26:54.077 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:54.077 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:54.077 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:54.077 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.077 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.077 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:221512 3.24 3:25f85d9e:::benchmark_data_smithi177_31116_object81209:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e192)", 2024-09-13T11:26:54.077 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:26:05.060948+0000", 2024-09-13T11:26:54.077 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 48.933137946000002, 2024-09-13T11:26:54.077 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.21371807700000001, 2024-09-13T11:26:54.077 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:26:54.077 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:26:54.077 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:54.078 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:26:54.078 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:26:54.078 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:54.078 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 221512 2024-09-13T11:26:54.078 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.078 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:26:54.078 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.078 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:26:54.078 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:05.060948+0000", 2024-09-13T11:26:54.078 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:54.078 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.079 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.079 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:26:54.079 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:05.060948+0000", 2024-09-13T11:26:54.079 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:54.079 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.079 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.079 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:26:54.079 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:05.060950+0000", 2024-09-13T11:26:54.079 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.6469999999999998e-06 2024-09-13T11:26:54.079 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.079 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.079 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:26:54.080 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:05.060959+0000", 2024-09-13T11:26:54.080 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.2460000000000003e-06 2024-09-13T11:26:54.080 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.080 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.080 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:26:54.080 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:05.060977+0000", 2024-09-13T11:26:54.080 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.8779000000000001e-05 2024-09-13T11:26:54.080 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.080 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.080 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:26:54.080 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:05.060993+0000", 2024-09-13T11:26:54.081 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.5464999999999998e-05 2024-09-13T11:26:54.081 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.081 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.081 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:54.081 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:05.062155+0000", 2024-09-13T11:26:54.081 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.001162724 2024-09-13T11:26:54.081 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.081 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.081 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-09-13T11:26:54.081 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:05.062181+0000", 2024-09-13T11:26:54.081 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.5157e-05 2024-09-13T11:26:54.081 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.082 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.082 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:54.082 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:05.257730+0000", 2024-09-13T11:26:54.082 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.19554980299999999 2024-09-13T11:26:54.082 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.082 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.082 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for degraded object", 2024-09-13T11:26:54.082 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:05.258342+0000", 2024-09-13T11:26:54.082 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00061164900000000003 2024-09-13T11:26:54.082 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.082 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.083 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:54.083 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:05.266413+0000", 2024-09-13T11:26:54.083 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0080705780000000001 2024-09-13T11:26:54.083 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.083 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.083 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:26:54.083 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:05.266509+0000", 2024-09-13T11:26:54.083 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.6644999999999994e-05 2024-09-13T11:26:54.083 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.083 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.084 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 14", 2024-09-13T11:26:54.084 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:05.266646+0000", 2024-09-13T11:26:54.084 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00013685499999999999 2024-09-13T11:26:54.084 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.084 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.084 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:54.084 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:05.273976+0000", 2024-09-13T11:26:54.084 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0073293719999999998 2024-09-13T11:26:54.084 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.084 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.084 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:26:54.084 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:05.273999+0000", 2024-09-13T11:26:54.085 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.3252e-05 2024-09-13T11:26:54.085 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.085 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.085 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:26:54.085 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:05.274075+0000", 2024-09-13T11:26:54.085 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.6415999999999995e-05 2024-09-13T11:26:54.085 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.085 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.085 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:26:54.085 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:05.274666+0000", 2024-09-13T11:26:54.085 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00059048900000000003 2024-09-13T11:26:54.085 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:54.085 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:54.085 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:54.086 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.086 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.086 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "MOSDPGPushReply(3.24 192/191 [PushReplyOp(3:26a69d79:::benchmark_data_smithi177_31116_object86105:head)])", 2024-09-13T11:26:54.086 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:26:05.749387+0000", 2024-09-13T11:26:54.086 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 48.244698122999999, 2024-09-13T11:26:54.086 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.34518297100000001, 2024-09-13T11:26:54.086 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:26:54.086 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:26:54.086 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "reached pg", 2024-09-13T11:26:54.086 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:26:54.086 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.086 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:26:54.086 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:05.749387+0000", 2024-09-13T11:26:54.087 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:54.087 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.087 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.087 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:26:54.087 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:05.749387+0000", 2024-09-13T11:26:54.087 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:54.087 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.087 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.087 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:26:54.087 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:05.749391+0000", 2024-09-13T11:26:54.087 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.54e-06 2024-09-13T11:26:54.087 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.087 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.088 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:26:54.088 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:05.749403+0000", 2024-09-13T11:26:54.088 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.1573000000000001e-05 2024-09-13T11:26:54.088 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.088 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.088 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:26:54.088 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:06.092306+0000", 2024-09-13T11:26:54.088 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.34290318600000003 2024-09-13T11:26:54.088 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.088 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.088 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:26:54.088 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:06.092334+0000", 2024-09-13T11:26:54.088 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.7993000000000001e-05 2024-09-13T11:26:54.089 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.089 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.089 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:54.089 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:06.092512+0000", 2024-09-13T11:26:54.089 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000178658 2024-09-13T11:26:54.089 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.089 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.089 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:26:54.089 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:06.094570+0000", 2024-09-13T11:26:54.089 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0020580210000000002 2024-09-13T11:26:54.089 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:54.089 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:54.089 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:54.090 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.090 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.090 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:228454 3.24 3:25beae58:::benchmark_data_smithi177_31116_object88151:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e198)", 2024-09-13T11:26:54.090 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:26:12.032335+0000", 2024-09-13T11:26:54.090 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 41.961750584999997, 2024-09-13T11:26:54.090 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.206015589, 2024-09-13T11:26:54.090 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:26:54.090 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:26:54.090 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:54.090 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:26:54.090 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:26:54.090 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:54.090 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 228454 2024-09-13T11:26:54.090 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.091 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:26:54.091 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.091 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:26:54.091 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:12.032335+0000", 2024-09-13T11:26:54.091 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:54.091 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.091 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.091 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:26:54.091 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:12.032335+0000", 2024-09-13T11:26:54.091 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:54.091 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.091 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.091 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:26:54.092 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:12.032337+0000", 2024-09-13T11:26:54.092 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.2290000000000002e-06 2024-09-13T11:26:54.092 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.092 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.092 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:26:54.092 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:12.032342+0000", 2024-09-13T11:26:54.092 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.0200000000000002e-06 2024-09-13T11:26:54.092 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.092 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.092 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:26:54.092 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:12.032353+0000", 2024-09-13T11:26:54.092 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.1168e-05 2024-09-13T11:26:54.093 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.093 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.093 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:26:54.093 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:12.032360+0000", 2024-09-13T11:26:54.093 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.7209999999999997e-06 2024-09-13T11:26:54.093 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.093 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.093 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:54.093 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:12.032430+0000", 2024-09-13T11:26:54.093 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.9651000000000001e-05 2024-09-13T11:26:54.093 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.093 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.094 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:26:54.094 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:12.032600+0000", 2024-09-13T11:26:54.094 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00017053000000000001 2024-09-13T11:26:54.094 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.094 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.094 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 14", 2024-09-13T11:26:54.094 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:12.032732+0000", 2024-09-13T11:26:54.094 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00013139099999999999 2024-09-13T11:26:54.094 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.094 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.094 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:26:54.094 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:12.034663+0000", 2024-09-13T11:26:54.095 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.001931012 2024-09-13T11:26:54.095 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.095 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.095 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:54.095 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:12.237696+0000", 2024-09-13T11:26:54.095 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.203033767 2024-09-13T11:26:54.095 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.095 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.095 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:26:54.095 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:12.237753+0000", 2024-09-13T11:26:54.095 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.6146000000000001e-05 2024-09-13T11:26:54.095 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.095 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.096 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:26:54.096 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:12.238351+0000", 2024-09-13T11:26:54.096 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00059795400000000004 2024-09-13T11:26:54.096 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:54.096 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:54.096 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:54.096 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.096 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.096 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "MOSDPGPushReply(3.24 200/191 [PushReplyOp(3:243cc29e:::benchmark_data_smithi177_31116_object132309:head)])", 2024-09-13T11:26:54.096 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:26:16.643152+0000", 2024-09-13T11:26:54.096 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 37.350933159999997, 2024-09-13T11:26:54.096 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.222406362, 2024-09-13T11:26:54.096 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:26:54.097 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:26:54.097 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "reached pg", 2024-09-13T11:26:54.097 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:26:54.097 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.097 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:26:54.097 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:16.643152+0000", 2024-09-13T11:26:54.097 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:54.097 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.097 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.097 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:26:54.097 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:16.643152+0000", 2024-09-13T11:26:54.097 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:54.097 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.098 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.098 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:26:54.098 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:16.643154+0000", 2024-09-13T11:26:54.098 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.0080000000000001e-06 2024-09-13T11:26:54.098 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.098 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.098 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:26:54.098 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:16.643166+0000", 2024-09-13T11:26:54.098 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.1739999999999999e-05 2024-09-13T11:26:54.098 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.098 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.098 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:26:54.098 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:16.864486+0000", 2024-09-13T11:26:54.099 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.221319708 2024-09-13T11:26:54.099 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.099 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.099 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:26:54.099 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:16.864501+0000", 2024-09-13T11:26:54.099 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.5226e-05 2024-09-13T11:26:54.099 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.099 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.099 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:54.099 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:16.864629+0000", 2024-09-13T11:26:54.099 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00012821499999999999 2024-09-13T11:26:54.099 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.099 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.100 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:26:54.100 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:16.865559+0000", 2024-09-13T11:26:54.100 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00092946499999999998 2024-09-13T11:26:54.100 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:54.100 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:54.100 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:54.100 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:54.100 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:54.100 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-09-13T11:26:54.102 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_ops_in_flight 2024-09-13T11:26:54.102 INFO:tasks.ceph.osd.15.smithi177.stderr:2024-09-13T11:26:54.101+0000 7f9d6ce77640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26578) UID: 0 2024-09-13T11:26:54.157 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-13T11:26:54.158 INFO:teuthology.orchestra.run.smithi001.stderr:dumped all 2024-09-13T11:26:54.184 INFO:teuthology.orchestra.run.smithi001.stdout:{"pg_ready":true,"pg_map":{"version":355,"stamp":"2024-09-13T11:26:52.969514+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":787317323,"num_objects":12008,"num_object_clones":0,"num_object_copies":24016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":167,"num_objects_unfound":0,"num_objects_dirty":12008,"num_whiteouts":0,"num_read":87,"num_read_kb":74,"num_write":157854,"num_write_kb":2901107,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24444,"num_bytes_recovered":1249378342,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":416714,"ondisk_log_size":416714,"up":126,"acting":126,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":134,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":16,"kb":1509949440,"kb_used":3722112,"kb_used_data":2011072,"kb_used_omap":68,"kb_used_meta":1710715,"kb_avail":1506227328,"statfs":{"total":1546188226560,"available":1542376783872,"internally_reserved":0,"allocated":2059337728,"data_stored":2052745550,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":70045,"internal_metadata":1751772771},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1,0,0,0,1,0,1,4],"upper_bound":2048},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-254967808,"num_objects":-3892,"num_object_clones":0,"num_object_copies":-7784,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1,"num_objects_misplaced":-902,"num_objects_unfound":0,"num_objects_dirty":-3892,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-16345,"num_write_kb":-455616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-3152,"num_bytes_recovered":-220332032,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"3.997278"},"pg_stats":[{"pgid":"3.37","version":"0'0","reported_seq":1,"reported_epoch":226,"state":"unknown","last_fresh":"2024-09-13T11:26:50.996896+0000","last_change":"2024-09-13T11:26:51.097734+0000","last_active":"0.000000","last_peered":"0.000000","last_clean":"0.000000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T11:26:50.996896+0000","last_undegraded":"2024-09-13T11:26:50.996896+0000","last_fullsized":"2024-09-13T11:26:50.996896+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"206'8719","last_scrub_stamp":"2024-09-13T11:26:25.299674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:26:25.299674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,12],"acting":[2,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.36","version":"225'9946","reported_seq":14938,"reported_epoch":227,"state":"remapped+peering","last_fresh":"2024-09-13T11:26:51.968160+0000","last_change":"2024-09-13T11:26:51.968160+0000","last_active":"2024-09-13T11:26:51.968160+0000","last_peered":"2024-09-13T11:26:50.851925+0000","last_clean":"2024-09-13T11:26:50.851925+0000","last_became_active":"2024-09-13T11:26:50.031958+0000","last_became_peered":"2024-09-13T11:26:50.031958+0000","last_unstale":"2024-09-13T11:26:51.968160+0000","last_undegraded":"2024-09-13T11:26:51.968160+0000","last_fullsized":"2024-09-13T11:26:51.968160+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":6,"last_scrub":"204'9074","last_scrub_stamp":"2024-09-13T11:26:21.348185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:26:21.348185+0000","objects_scrubbed":1320,"log_size":9946,"log_dups_size":0,"ondisk_log_size":9946,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":479,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13959168,"num_objects":213,"num_object_clones":0,"num_object_copies":426,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":213,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3239,"num_write_kb":77600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":1,"reported_epoch":226,"state":"unknown","last_fresh":"2024-09-13T11:26:50.997519+0000","last_change":"2024-09-13T11:26:51.230596+0000","last_active":"0.000000","last_peered":"0.000000","last_clean":"0.000000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T11:26:50.997519+0000","last_undegraded":"2024-09-13T11:26:50.997519+0000","last_fullsized":"2024-09-13T11:26:50.997519+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"181'7535","last_scrub_stamp":"2024-09-13T11:25:48.075568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:48.075568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,12],"acting":[7,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":1,"reported_epoch":226,"state":"unknown","last_fresh":"2024-09-13T11:26:51.002125+0000","last_change":"2024-09-13T11:26:51.156362+0000","last_active":"0.000000","last_peered":"0.000000","last_clean":"0.000000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T11:26:51.002125+0000","last_undegraded":"2024-09-13T11:26:51.002125+0000","last_fullsized":"2024-09-13T11:26:51.002125+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"174'7321","last_scrub_stamp":"2024-09-13T11:25:38.085667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:38.085667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,11],"acting":[10,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":1,"reported_epoch":226,"state":"unknown","last_fresh":"2024-09-13T11:26:50.994257+0000","last_change":"2024-09-13T11:26:51.085950+0000","last_active":"0.000000","last_peered":"0.000000","last_clean":"0.000000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T11:26:50.994257+0000","last_undegraded":"2024-09-13T11:26:50.994257+0000","last_fullsized":"2024-09-13T11:26:50.994257+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'9393","last_scrub_stamp":"2024-09-13T11:26:36.941193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:26:36.941193+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"223'9738","reported_seq":20875,"reported_epoch":226,"state":"remapped","last_fresh":"2024-09-13T11:26:50.987627+0000","last_change":"2024-09-13T11:26:51.081649+0000","last_active":"2024-09-13T11:26:50.987627+0000","last_peered":"2024-09-13T11:26:48.964097+0000","last_clean":"2024-09-13T11:26:03.227632+0000","last_became_active":"2024-09-13T11:26:06.225212+0000","last_became_peered":"2024-09-13T11:26:06.225212+0000","last_unstale":"2024-09-13T11:26:50.987627+0000","last_undegraded":"2024-09-13T11:26:50.987627+0000","last_fullsized":"2024-09-13T11:26:50.987627+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":6,"last_scrub":"186'8089","last_scrub_stamp":"2024-09-13T11:25:57.085837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:57.085837+0000","objects_scrubbed":2130,"log_size":9738,"log_dups_size":0,"ondisk_log_size":9738,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1023,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15368192,"num_objects":235,"num_object_clones":0,"num_object_copies":470,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":235,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3315,"num_write_kb":85856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1943,"num_bytes_recovered":127336448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"226'9781","reported_seq":13165,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-13T11:26:51.969220+0000","last_change":"2024-09-13T11:26:51.969220+0000","last_active":"2024-09-13T11:26:51.969220+0000","last_peered":"2024-09-13T11:26:51.969220+0000","last_clean":"2024-09-13T11:26:51.969220+0000","last_became_active":"2024-09-13T11:26:49.986829+0000","last_became_peered":"2024-09-13T11:26:49.986829+0000","last_unstale":"2024-09-13T11:26:51.969220+0000","last_undegraded":"2024-09-13T11:26:51.969220+0000","last_fullsized":"2024-09-13T11:26:51.969220+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":5,"last_scrub":"213'9393","last_scrub_stamp":"2024-09-13T11:26:36.941193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:26:36.941193+0000","objects_scrubbed":0,"log_size":9781,"log_dups_size":0,"ondisk_log_size":9781,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"user requested scrub scheduled @ 2024-09-13T11:26:56.891612+0000","scrub_duration":119,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13139968,"num_objects":200,"num_object_clones":0,"num_object_copies":400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":200,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3197,"num_write_kb":77024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7],"acting":[11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.12","version":"225'9750","reported_seq":20899,"reported_epoch":226,"state":"active+remapped+backfill_toofull","last_fresh":"2024-09-13T11:26:50.987652+0000","last_change":"2024-09-13T11:26:50.021305+0000","last_active":"2024-09-13T11:26:50.987652+0000","last_peered":"2024-09-13T11:26:50.987652+0000","last_clean":"2024-09-13T11:26:03.227632+0000","last_became_active":"2024-09-13T11:26:49.990894+0000","last_became_peered":"2024-09-13T11:26:49.990894+0000","last_unstale":"2024-09-13T11:26:50.987652+0000","last_undegraded":"2024-09-13T11:26:50.987652+0000","last_fullsized":"2024-09-13T11:26:50.987652+0000","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":5,"last_scrub":"186'8089","last_scrub_stamp":"2024-09-13T11:25:57.085837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:57.085837+0000","objects_scrubbed":2130,"log_size":9750,"log_dups_size":0,"ondisk_log_size":9750,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1023,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14581760,"num_objects":222,"num_object_clones":0,"num_object_copies":444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":53,"num_objects_unfound":0,"num_objects_dirty":222,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3326,"num_write_kb":85856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1943,"num_bytes_recovered":127336448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,7],"acting":[5,7],"avail_no_missing":["5","7"],"object_location_counts":[{"shards":"5,7","objects":222}],"blocked_by":[],"up_primary":14,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.11","version":"226'9733","reported_seq":13162,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-13T11:26:50.987005+0000","last_change":"2024-09-13T11:26:50.986771+0000","last_active":"2024-09-13T11:26:50.987005+0000","last_peered":"2024-09-13T11:26:50.987005+0000","last_clean":"2024-09-13T11:26:50.987005+0000","last_became_active":"2024-09-13T11:26:50.004919+0000","last_became_peered":"2024-09-13T11:26:50.004919+0000","last_unstale":"2024-09-13T11:26:50.987005+0000","last_undegraded":"2024-09-13T11:26:50.987005+0000","last_fullsized":"2024-09-13T11:26:50.987005+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":5,"last_scrub":"165'6821","last_scrub_stamp":"2024-09-13T11:25:24.287082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:24.287082+0000","objects_scrubbed":0,"log_size":9733,"log_dups_size":0,"ondisk_log_size":9733,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":947,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15532032,"num_objects":237,"num_object_clones":0,"num_object_copies":474,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":237,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3199,"num_write_kb":77792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11],"acting":[4,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"225'9841","reported_seq":14518,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-13T11:26:50.994220+0000","last_change":"2024-09-13T11:26:50.993797+0000","last_active":"2024-09-13T11:26:50.994220+0000","last_peered":"2024-09-13T11:26:50.994220+0000","last_clean":"2024-09-13T11:26:50.994220+0000","last_became_active":"2024-09-13T11:26:49.984837+0000","last_became_peered":"2024-09-13T11:26:49.984837+0000","last_unstale":"2024-09-13T11:26:50.994220+0000","last_undegraded":"2024-09-13T11:26:50.994220+0000","last_fullsized":"2024-09-13T11:26:50.994220+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":5,"last_scrub":"204'8963","last_scrub_stamp":"2024-09-13T11:26:22.296795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:26:22.296795+0000","objects_scrubbed":0,"log_size":9841,"log_dups_size":0,"ondisk_log_size":9841,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":195,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12943360,"num_objects":197,"num_object_clones":0,"num_object_copies":394,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":197,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3193,"num_write_kb":76000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"225'10074","reported_seq":10605,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-13T11:26:51.966871+0000","last_change":"2024-09-13T11:26:50.022789+0000","last_active":"2024-09-13T11:26:51.966871+0000","last_peered":"2024-09-13T11:26:51.966871+0000","last_clean":"2024-09-13T11:26:51.966871+0000","last_became_active":"2024-09-13T11:26:49.990894+0000","last_became_peered":"2024-09-13T11:26:49.990894+0000","last_unstale":"2024-09-13T11:26:51.966871+0000","last_undegraded":"2024-09-13T11:26:51.966871+0000","last_fullsized":"2024-09-13T11:26:51.966871+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"211'9653","last_scrub_stamp":"2024-09-13T11:26:34.940878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:26:34.940878+0000","objects_scrubbed":877,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":119,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14516224,"num_objects":221,"num_object_clones":0,"num_object_copies":442,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":221,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3331,"num_write_kb":85280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,12],"acting":[11,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.e","version":"225'10099","reported_seq":15524,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-13T11:26:50.985757+0000","last_change":"2024-09-13T11:26:50.839154+0000","last_active":"2024-09-13T11:26:50.985757+0000","last_peered":"2024-09-13T11:26:50.985757+0000","last_clean":"2024-09-13T11:26:50.985757+0000","last_became_active":"2024-09-13T11:26:49.986357+0000","last_became_peered":"2024-09-13T11:26:49.986357+0000","last_unstale":"2024-09-13T11:26:50.985757+0000","last_undegraded":"2024-09-13T11:26:50.985757+0000","last_fullsized":"2024-09-13T11:26:50.985757+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"180'8032","last_scrub_stamp":"2024-09-13T11:25:46.438107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:46.438107+0000","objects_scrubbed":0,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":327,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12451840,"num_objects":190,"num_object_clones":0,"num_object_copies":380,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":190,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3289,"num_write_kb":82944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.d","version":"225'7572","reported_seq":8066,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-13T11:26:51.003971+0000","last_change":"2024-09-13T11:25:57.320482+0000","last_active":"2024-09-13T11:26:51.003971+0000","last_peered":"2024-09-13T11:26:51.003971+0000","last_clean":"2024-09-13T11:26:51.003971+0000","last_became_active":"2024-09-13T11:24:24.087224+0000","last_became_peered":"2024-09-13T11:24:24.087224+0000","last_unstale":"2024-09-13T11:26:51.003971+0000","last_undegraded":"2024-09-13T11:26:51.003971+0000","last_fullsized":"2024-09-13T11:26:51.003971+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"186'6735","last_scrub_stamp":"2024-09-13T11:25:57.320378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:57.320378+0000","objects_scrubbed":1064,"log_size":7572,"log_dups_size":0,"ondisk_log_size":7572,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:27:06.312652+0000","scrub_duration":211,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14811136,"num_objects":226,"num_object_clones":0,"num_object_copies":452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":226,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2465,"num_write_kb":30912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.c","version":"226'7585","reported_seq":8085,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-13T11:26:51.966412+0000","last_change":"2024-09-13T11:25:36.279464+0000","last_active":"2024-09-13T11:26:51.966412+0000","last_peered":"2024-09-13T11:26:51.966412+0000","last_clean":"2024-09-13T11:26:51.966412+0000","last_became_active":"2024-09-13T11:24:24.090847+0000","last_became_peered":"2024-09-13T11:24:24.090847+0000","last_unstale":"2024-09-13T11:26:51.966412+0000","last_undegraded":"2024-09-13T11:26:51.966412+0000","last_fullsized":"2024-09-13T11:26:51.966412+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"174'6334","last_scrub_stamp":"2024-09-13T11:25:36.279390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:36.279390+0000","objects_scrubbed":1456,"log_size":7585,"log_dups_size":0,"ondisk_log_size":7585,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":235,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13369344,"num_objects":204,"num_object_clones":0,"num_object_copies":408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":204,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2436,"num_write_kb":29440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,13],"acting":[11,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.b","version":"225'7581","reported_seq":8090,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-13T11:26:51.983115+0000","last_change":"2024-09-13T11:25:58.292249+0000","last_active":"2024-09-13T11:26:51.983115+0000","last_peered":"2024-09-13T11:26:51.983115+0000","last_clean":"2024-09-13T11:26:51.983115+0000","last_became_active":"2024-09-13T11:24:24.085684+0000","last_became_peered":"2024-09-13T11:24:24.085684+0000","last_unstale":"2024-09-13T11:26:51.983115+0000","last_undegraded":"2024-09-13T11:26:51.983115+0000","last_fullsized":"2024-09-13T11:26:51.983115+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"186'6759","last_scrub_stamp":"2024-09-13T11:25:58.292153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:58.292153+0000","objects_scrubbed":1030,"log_size":7581,"log_dups_size":0,"ondisk_log_size":7581,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:27:10.748450+0000","scrub_duration":119,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13565952,"num_objects":207,"num_object_clones":0,"num_object_copies":414,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":207,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2469,"num_write_kb":30656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,1],"acting":[13,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"3.a","version":"225'7622","reported_seq":8114,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-09-13T11:26:49.983890+0000","last_change":"2024-09-13T11:25:59.695285+0000","last_active":"2024-09-13T11:26:49.983890+0000","last_peered":"2024-09-13T11:26:49.983890+0000","last_clean":"2024-09-13T11:26:49.983890+0000","last_became_active":"2024-09-13T11:24:24.089253+0000","last_became_peered":"2024-09-13T11:24:24.089253+0000","last_unstale":"2024-09-13T11:26:49.983890+0000","last_undegraded":"2024-09-13T11:26:49.983890+0000","last_fullsized":"2024-09-13T11:26:49.983890+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"187'6833","last_scrub_stamp":"2024-09-13T11:25:59.695136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:59.695136+0000","objects_scrubbed":973,"log_size":7622,"log_dups_size":0,"ondisk_log_size":7622,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:27:13.269622+0000","scrub_duration":131,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11993088,"num_objects":183,"num_object_clones":0,"num_object_copies":366,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":183,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2468,"num_write_kb":27968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.9","version":"225'7280","reported_seq":7753,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-13T11:26:50.996165+0000","last_change":"2024-09-13T11:24:24.928964+0000","last_active":"2024-09-13T11:26:50.996165+0000","last_peered":"2024-09-13T11:26:50.996165+0000","last_clean":"2024-09-13T11:26:50.996165+0000","last_became_active":"2024-09-13T11:24:24.082504+0000","last_became_peered":"2024-09-13T11:24:24.082504+0000","last_unstale":"2024-09-13T11:26:50.996165+0000","last_undegraded":"2024-09-13T11:26:50.996165+0000","last_fullsized":"2024-09-13T11:26:50.996165+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'4799","last_scrub_stamp":"2024-09-13T11:24:24.928724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:24:24.928724+0000","objects_scrubbed":1705,"log_size":7280,"log_dups_size":0,"ondisk_log_size":7280,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":255,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12648448,"num_objects":193,"num_object_clones":0,"num_object_copies":386,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":193,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2484,"num_write_kb":31104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,2],"acting":[12,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[]},{"pgid":"3.8","version":"225'7199","reported_seq":7690,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-13T11:26:50.994873+0000","last_change":"2024-09-13T11:26:00.418168+0000","last_active":"2024-09-13T11:26:50.994873+0000","last_peered":"2024-09-13T11:26:50.994873+0000","last_clean":"2024-09-13T11:26:50.994873+0000","last_became_active":"2024-09-13T11:24:24.096220+0000","last_became_peered":"2024-09-13T11:24:24.096220+0000","last_unstale":"2024-09-13T11:26:50.994873+0000","last_undegraded":"2024-09-13T11:26:50.994873+0000","last_fullsized":"2024-09-13T11:26:50.994873+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"188'6425","last_scrub_stamp":"2024-09-13T11:26:00.417942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:26:00.417942+0000","objects_scrubbed":989,"log_size":7199,"log_dups_size":0,"ondisk_log_size":7199,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:27:09.204623+0000","scrub_duration":199,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14090240,"num_objects":215,"num_object_clones":0,"num_object_copies":430,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":215,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2451,"num_write_kb":31232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"26'112","reported_seq":575,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-13T11:26:50.995307+0000","last_change":"2024-09-13T11:26:46.996802+0000","last_active":"2024-09-13T11:26:50.995307+0000","last_peered":"2024-09-13T11:26:50.995307+0000","last_clean":"2024-09-13T11:26:50.995307+0000","last_became_active":"2024-09-13T11:22:12.129721+0000","last_became_peered":"2024-09-13T11:22:12.129721+0000","last_unstale":"2024-09-13T11:26:50.995307+0000","last_undegraded":"2024-09-13T11:26:50.995307+0000","last_fullsized":"2024-09-13T11:26:50.995307+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"26'112","last_scrub_stamp":"2024-09-13T11:26:46.996697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:11.098388+0000","last_clean_scrub_stamp":"2024-09-13T11:26:46.996697+0000","objects_scrubbed":2,"log_size":112,"log_dups_size":0,"ondisk_log_size":112,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:27:50.591478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.3","version":"26'1","reported_seq":81,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-09-13T11:26:49.982234+0000","last_change":"2024-09-13T11:26:05.046376+0000","last_active":"2024-09-13T11:26:49.982234+0000","last_peered":"2024-09-13T11:26:49.982234+0000","last_clean":"2024-09-13T11:26:49.982234+0000","last_became_active":"2024-09-13T11:26:05.041032+0000","last_became_peered":"2024-09-13T11:26:05.041032+0000","last_unstale":"2024-09-13T11:26:49.982234+0000","last_undegraded":"2024-09-13T11:26:49.982234+0000","last_fullsized":"2024-09-13T11:26:49.982234+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"26'1","last_scrub_stamp":"2024-09-13T11:25:23.034247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:25:23.034247+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,15],"acting":[14,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"3.2","version":"223'6475","reported_seq":14900,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-09-13T11:26:49.984889+0000","last_change":"2024-09-13T11:26:43.033025+0000","last_active":"2024-09-13T11:26:49.984889+0000","last_peered":"2024-09-13T11:26:49.984889+0000","last_clean":"2024-09-13T11:26:49.984889+0000","last_became_active":"2024-09-13T11:26:42.985163+0000","last_became_peered":"2024-09-13T11:26:42.985163+0000","last_unstale":"2024-09-13T11:26:49.984889+0000","last_undegraded":"2024-09-13T11:26:49.984889+0000","last_fullsized":"2024-09-13T11:26:49.984889+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"164'4994","last_scrub_stamp":"2024-09-13T11:25:23.765047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:23.765047+0000","objects_scrubbed":1689,"log_size":6475,"log_dups_size":0,"ondisk_log_size":6475,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":283,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13434880,"num_objects":205,"num_object_clones":0,"num_object_copies":410,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":205,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3418,"num_write_kb":89024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3963,"num_bytes_recovered":162332672,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,12],"acting":[14,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":431,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-09-13T11:26:49.981991+0000","last_change":"2024-09-13T11:25:56.557174+0000","last_active":"2024-09-13T11:26:49.981991+0000","last_peered":"2024-09-13T11:26:49.981991+0000","last_clean":"2024-09-13T11:26:49.981991+0000","last_became_active":"2024-09-13T11:22:14.211435+0000","last_became_peered":"2024-09-13T11:22:14.211435+0000","last_unstale":"2024-09-13T11:26:49.981991+0000","last_undegraded":"2024-09-13T11:26:49.981991+0000","last_fullsized":"2024-09-13T11:26:49.981991+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:25:56.557113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:25:56.557113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:27:18.356524+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0010010360000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"225'6352","reported_seq":6860,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-13T11:26:51.983071+0000","last_change":"2024-09-13T11:26:33.441490+0000","last_active":"2024-09-13T11:26:51.983071+0000","last_peered":"2024-09-13T11:26:51.983071+0000","last_clean":"2024-09-13T11:26:51.983071+0000","last_became_active":"2024-09-13T11:23:31.926528+0000","last_became_peered":"2024-09-13T11:23:31.926528+0000","last_unstale":"2024-09-13T11:26:51.983071+0000","last_undegraded":"2024-09-13T11:26:51.983071+0000","last_fullsized":"2024-09-13T11:26:51.983071+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"211'6118","last_scrub_stamp":"2024-09-13T11:26:33.441303+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:26:33.441303+0000","objects_scrubbed":453,"log_size":6352,"log_dups_size":0,"ondisk_log_size":6352,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:27:53.732036+0000","scrub_duration":71,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14286848,"num_objects":218,"num_object_clones":0,"num_object_copies":436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":218,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3282,"num_write_kb":85056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,11],"acting":[13,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":77,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-09-13T11:26:49.982091+0000","last_change":"2024-09-13T11:26:05.047291+0000","last_active":"2024-09-13T11:26:49.982091+0000","last_peered":"2024-09-13T11:26:49.982091+0000","last_clean":"2024-09-13T11:26:49.982091+0000","last_became_active":"2024-09-13T11:26:05.046717+0000","last_became_peered":"2024-09-13T11:26:05.046717+0000","last_unstale":"2024-09-13T11:26:49.982091+0000","last_undegraded":"2024-09-13T11:26:49.982091+0000","last_fullsized":"2024-09-13T11:26:49.982091+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:25:34.822831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:25:34.822831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00081225399999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,1],"acting":[14,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"3.0","version":"225'6375","reported_seq":6866,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-13T11:26:50.996345+0000","last_change":"2024-09-13T11:25:09.842875+0000","last_active":"2024-09-13T11:26:50.996345+0000","last_peered":"2024-09-13T11:26:50.996345+0000","last_clean":"2024-09-13T11:26:50.996345+0000","last_became_active":"2024-09-13T11:23:31.921388+0000","last_became_peered":"2024-09-13T11:23:31.921388+0000","last_unstale":"2024-09-13T11:26:50.996345+0000","last_undegraded":"2024-09-13T11:26:50.996345+0000","last_fullsized":"2024-09-13T11:26:50.996345+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"151'4644","last_scrub_stamp":"2024-09-13T11:25:09.842728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:09.842728+0000","objects_scrubbed":1944,"log_size":6375,"log_dups_size":0,"ondisk_log_size":6375,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":395,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13959168,"num_objects":213,"num_object_clones":0,"num_object_copies":426,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":213,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3269,"num_write_kb":84032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,15],"acting":[12,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[]},{"pgid":"2.2","version":"28'2","reported_seq":41,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-09-13T11:26:49.983740+0000","last_change":"2024-09-13T11:26:36.271659+0000","last_active":"2024-09-13T11:26:49.983740+0000","last_peered":"2024-09-13T11:26:49.983740+0000","last_clean":"2024-09-13T11:26:49.983740+0000","last_became_active":"2024-09-13T11:26:36.267081+0000","last_became_peered":"2024-09-13T11:26:36.267081+0000","last_unstale":"2024-09-13T11:26:49.983740+0000","last_undegraded":"2024-09-13T11:26:49.983740+0000","last_fullsized":"2024-09-13T11:26:49.983740+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2","last_scrub_stamp":"2024-09-13T11:25:13.869413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:25:13.869413+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0025826730000000002,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9],"acting":[3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.3","version":"225'6330","reported_seq":6823,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-13T11:26:51.003822+0000","last_change":"2024-09-13T11:25:08.394545+0000","last_active":"2024-09-13T11:26:51.003822+0000","last_peered":"2024-09-13T11:26:51.003822+0000","last_clean":"2024-09-13T11:26:51.003822+0000","last_became_active":"2024-09-13T11:23:31.930448+0000","last_became_peered":"2024-09-13T11:23:31.930448+0000","last_unstale":"2024-09-13T11:26:51.003822+0000","last_undegraded":"2024-09-13T11:26:51.003822+0000","last_fullsized":"2024-09-13T11:26:51.003822+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"151'4587","last_scrub_stamp":"2024-09-13T11:25:08.394441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:08.394441+0000","objects_scrubbed":1942,"log_size":6330,"log_dups_size":0,"ondisk_log_size":6330,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":219,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12845056,"num_objects":196,"num_object_clones":0,"num_object_copies":392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":196,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3208,"num_write_kb":80896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,7],"acting":[10,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.5","version":"225'7156","reported_seq":7656,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-13T11:26:51.983144+0000","last_change":"2024-09-13T11:26:24.022070+0000","last_active":"2024-09-13T11:26:51.983144+0000","last_peered":"2024-09-13T11:26:51.983144+0000","last_clean":"2024-09-13T11:26:51.983144+0000","last_became_active":"2024-09-13T11:26:23.988370+0000","last_became_peered":"2024-09-13T11:26:23.988370+0000","last_unstale":"2024-09-13T11:26:51.983144+0000","last_undegraded":"2024-09-13T11:26:51.983144+0000","last_fullsized":"2024-09-13T11:26:51.983144+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"182'6171","last_scrub_stamp":"2024-09-13T11:25:49.468616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:49.468616+0000","objects_scrubbed":1202,"log_size":7156,"log_dups_size":0,"ondisk_log_size":7156,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:27:13.323295+0000","scrub_duration":175,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14221312,"num_objects":217,"num_object_clones":0,"num_object_copies":434,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":217,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2419,"num_write_kb":32384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,15],"acting":[13,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":418,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-09-13T11:26:49.985578+0000","last_change":"2024-09-13T11:25:45.851854+0000","last_active":"2024-09-13T11:26:49.985578+0000","last_peered":"2024-09-13T11:26:49.985578+0000","last_clean":"2024-09-13T11:26:49.985578+0000","last_became_active":"2024-09-13T11:22:47.719636+0000","last_became_peered":"2024-09-13T11:22:47.719636+0000","last_unstale":"2024-09-13T11:26:49.985578+0000","last_undegraded":"2024-09-13T11:26:49.985578+0000","last_fullsized":"2024-09-13T11:26:49.985578+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:25:45.851784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:25:45.851784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:26:50.817831+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00087476000000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,9],"acting":[1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"225'7221","reported_seq":7691,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-09-13T11:26:50.193408+0000","last_change":"2024-09-13T11:25:48.001902+0000","last_active":"2024-09-13T11:26:50.193408+0000","last_peered":"2024-09-13T11:26:50.193408+0000","last_clean":"2024-09-13T11:26:50.193408+0000","last_became_active":"2024-09-13T11:24:24.083077+0000","last_became_peered":"2024-09-13T11:24:24.083077+0000","last_unstale":"2024-09-13T11:26:50.193408+0000","last_undegraded":"2024-09-13T11:26:50.193408+0000","last_fullsized":"2024-09-13T11:26:50.193408+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"181'6222","last_scrub_stamp":"2024-09-13T11:25:48.001778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:48.001778+0000","objects_scrubbed":1224,"log_size":7221,"log_dups_size":0,"ondisk_log_size":7221,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:27:07.237007+0000","scrub_duration":171,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14745600,"num_objects":225,"num_object_clones":0,"num_object_copies":450,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":225,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2487,"num_write_kb":32704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,12],"acting":[1,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":433,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-13T11:26:50.981369+0000","last_change":"2024-09-13T11:25:33.957283+0000","last_active":"2024-09-13T11:26:50.981369+0000","last_peered":"2024-09-13T11:26:50.981369+0000","last_clean":"2024-09-13T11:26:50.981369+0000","last_became_active":"2024-09-13T11:22:14.581458+0000","last_became_peered":"2024-09-13T11:22:14.581458+0000","last_unstale":"2024-09-13T11:26:50.981369+0000","last_undegraded":"2024-09-13T11:26:50.981369+0000","last_fullsized":"2024-09-13T11:26:50.981369+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:25:33.957179+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:25:33.957179+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:26:52.126360+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.001026047,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5],"acting":[8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.7","version":"225'7188","reported_seq":7690,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-13T11:26:51.983208+0000","last_change":"2024-09-13T11:25:59.279849+0000","last_active":"2024-09-13T11:26:51.983208+0000","last_peered":"2024-09-13T11:26:51.983208+0000","last_clean":"2024-09-13T11:26:51.983208+0000","last_became_active":"2024-09-13T11:24:24.092975+0000","last_became_peered":"2024-09-13T11:24:24.092975+0000","last_unstale":"2024-09-13T11:26:51.983208+0000","last_undegraded":"2024-09-13T11:26:51.983208+0000","last_fullsized":"2024-09-13T11:26:51.983208+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"187'6375","last_scrub_stamp":"2024-09-13T11:25:59.279677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:59.279677+0000","objects_scrubbed":1033,"log_size":7188,"log_dups_size":0,"ondisk_log_size":7188,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:27:06.179340+0000","scrub_duration":139,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14417920,"num_objects":220,"num_object_clones":0,"num_object_copies":440,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":220,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2370,"num_write_kb":26368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,8],"acting":[13,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":442,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-13T11:26:51.971454+0000","last_change":"2024-09-13T11:26:33.048163+0000","last_active":"2024-09-13T11:26:51.971454+0000","last_peered":"2024-09-13T11:26:51.971454+0000","last_clean":"2024-09-13T11:26:51.971454+0000","last_became_active":"2024-09-13T11:22:14.211923+0000","last_became_peered":"2024-09-13T11:22:14.211923+0000","last_unstale":"2024-09-13T11:26:51.971454+0000","last_undegraded":"2024-09-13T11:26:51.971454+0000","last_fullsized":"2024-09-13T11:26:51.971454+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:26:33.048033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:26:33.048033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:27:40.864163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011689529999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0],"acting":[9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.6","version":"225'7367","reported_seq":15836,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-13T11:26:51.983163+0000","last_change":"2024-09-13T11:26:32.392786+0000","last_active":"2024-09-13T11:26:51.983163+0000","last_peered":"2024-09-13T11:26:51.983163+0000","last_clean":"2024-09-13T11:26:51.983163+0000","last_became_active":"2024-09-13T11:26:05.246522+0000","last_became_peered":"2024-09-13T11:26:05.246522+0000","last_unstale":"2024-09-13T11:26:51.983163+0000","last_undegraded":"2024-09-13T11:26:51.983163+0000","last_fullsized":"2024-09-13T11:26:51.983163+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"211'7110","last_scrub_stamp":"2024-09-13T11:26:32.392666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:26:32.392666+0000","objects_scrubbed":482,"log_size":7367,"log_dups_size":0,"ondisk_log_size":7367,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:27:54.979779+0000","scrub_duration":59,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14745600,"num_objects":225,"num_object_clones":0,"num_object_copies":450,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":225,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2152,"num_write_kb":9408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2230,"num_bytes_recovered":61800448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,14],"acting":[13,14],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":52,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-13T11:26:51.004697+0000","last_change":"2024-09-13T11:26:23.991137+0000","last_active":"2024-09-13T11:26:51.004697+0000","last_peered":"2024-09-13T11:26:51.004697+0000","last_clean":"2024-09-13T11:26:51.004697+0000","last_became_active":"2024-09-13T11:26:23.990749+0000","last_became_peered":"2024-09-13T11:26:23.990749+0000","last_unstale":"2024-09-13T11:26:51.004697+0000","last_undegraded":"2024-09-13T11:26:51.004697+0000","last_fullsized":"2024-09-13T11:26:51.004697+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:25:46.222806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:25:46.222806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:27:14.224983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,15],"acting":[10,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.14","version":"225'9827","reported_seq":22351,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-09-13T11:26:50.107478+0000","last_change":"2024-09-13T11:26:50.014657+0000","last_active":"2024-09-13T11:26:50.107478+0000","last_peered":"2024-09-13T11:26:50.107478+0000","last_clean":"2024-09-13T11:26:50.107478+0000","last_became_active":"2024-09-13T11:26:49.995599+0000","last_became_peered":"2024-09-13T11:26:49.995599+0000","last_unstale":"2024-09-13T11:26:50.107478+0000","last_undegraded":"2024-09-13T11:26:50.107478+0000","last_fullsized":"2024-09-13T11:26:50.107478+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":5,"last_scrub":"174'7321","last_scrub_stamp":"2024-09-13T11:25:38.085667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:38.085667+0000","objects_scrubbed":2938,"log_size":9827,"log_dups_size":0,"ondisk_log_size":9827,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":431,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13664256,"num_objects":208,"num_object_clones":0,"num_object_copies":416,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":208,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3237,"num_write_kb":79296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1994,"num_bytes_recovered":130711552,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,14],"acting":[3,14],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"223'9499","reported_seq":22597,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-09-13T11:26:50.032470+0000","last_change":"2024-09-13T11:26:50.032470+0000","last_active":"2024-09-13T11:26:50.032470+0000","last_peered":"2024-09-13T11:26:50.032470+0000","last_clean":"2024-09-13T11:26:50.032470+0000","last_became_active":"2024-09-13T11:26:50.003619+0000","last_became_peered":"2024-09-13T11:26:50.003619+0000","last_unstale":"2024-09-13T11:26:50.032470+0000","last_undegraded":"2024-09-13T11:26:50.032470+0000","last_fullsized":"2024-09-13T11:26:50.032470+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":5,"last_scrub":"181'7535","last_scrub_stamp":"2024-09-13T11:25:48.075568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:48.075568+0000","objects_scrubbed":2413,"log_size":9499,"log_dups_size":0,"ondisk_log_size":9499,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":991,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13959168,"num_objects":213,"num_object_clones":0,"num_object_copies":426,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":213,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3058,"num_write_kb":72160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2093,"num_bytes_recovered":137199616,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,0],"acting":[14,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"3.16","version":"225'9950","reported_seq":14946,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-13T11:26:51.966521+0000","last_change":"2024-09-13T11:26:50.034002+0000","last_active":"2024-09-13T11:26:51.966521+0000","last_peered":"2024-09-13T11:26:51.966521+0000","last_clean":"2024-09-13T11:26:51.966521+0000","last_became_active":"2024-09-13T11:26:50.003909+0000","last_became_peered":"2024-09-13T11:26:50.003909+0000","last_unstale":"2024-09-13T11:26:51.966521+0000","last_undegraded":"2024-09-13T11:26:51.966521+0000","last_fullsized":"2024-09-13T11:26:51.966521+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":5,"last_scrub":"204'9074","last_scrub_stamp":"2024-09-13T11:26:21.348185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:26:21.348185+0000","objects_scrubbed":1320,"log_size":9950,"log_dups_size":0,"ondisk_log_size":9950,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":479,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13697024,"num_objects":209,"num_object_clones":0,"num_object_copies":418,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":209,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3243,"num_write_kb":77600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,15],"acting":[11,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.17","version":"225'9507","reported_seq":13869,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-13T11:26:51.004449+0000","last_change":"2024-09-13T11:26:50.240596+0000","last_active":"2024-09-13T11:26:51.004449+0000","last_peered":"2024-09-13T11:26:51.004449+0000","last_clean":"2024-09-13T11:26:51.004449+0000","last_became_active":"2024-09-13T11:26:49.983871+0000","last_became_peered":"2024-09-13T11:26:49.983871+0000","last_unstale":"2024-09-13T11:26:51.004449+0000","last_undegraded":"2024-09-13T11:26:51.004449+0000","last_fullsized":"2024-09-13T11:26:51.004449+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":5,"last_scrub":"206'8719","last_scrub_stamp":"2024-09-13T11:26:25.299674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:26:25.299674+0000","objects_scrubbed":0,"log_size":9507,"log_dups_size":0,"ondisk_log_size":9507,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":919,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13959168,"num_objects":213,"num_object_clones":0,"num_object_copies":426,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":213,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3081,"num_write_kb":73344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3],"acting":[10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.18","version":"225'9715","reported_seq":14611,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-13T11:26:51.003565+0000","last_change":"2024-09-13T11:25:37.928304+0000","last_active":"2024-09-13T11:26:51.003565+0000","last_peered":"2024-09-13T11:26:51.003565+0000","last_clean":"2024-09-13T11:26:51.003565+0000","last_became_active":"2024-09-13T11:23:48.926014+0000","last_became_peered":"2024-09-13T11:23:48.926014+0000","last_unstale":"2024-09-13T11:26:51.003565+0000","last_undegraded":"2024-09-13T11:26:51.003565+0000","last_fullsized":"2024-09-13T11:26:51.003565+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":5,"last_scrub":"174'7306","last_scrub_stamp":"2024-09-13T11:25:37.928052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:37.928052+0000","objects_scrubbed":2815,"log_size":9715,"log_dups_size":0,"ondisk_log_size":9715,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:26:55.771991+0000","scrub_duration":631,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25690112,"num_objects":392,"num_object_clones":0,"num_object_copies":784,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":392,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5348,"num_write_kb":86528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0],"acting":[10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.19","version":"225'9554","reported_seq":14170,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-13T11:26:51.970171+0000","last_change":"2024-09-13T11:26:34.674723+0000","last_active":"2024-09-13T11:26:51.970171+0000","last_peered":"2024-09-13T11:26:51.970171+0000","last_clean":"2024-09-13T11:26:51.970171+0000","last_became_active":"2024-09-13T11:23:41.924941+0000","last_became_peered":"2024-09-13T11:23:41.924941+0000","last_unstale":"2024-09-13T11:26:51.970171+0000","last_undegraded":"2024-09-13T11:26:51.970171+0000","last_fullsized":"2024-09-13T11:26:51.970171+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":5,"last_scrub":"211'9143","last_scrub_stamp":"2024-09-13T11:26:34.674563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:26:34.674563+0000","objects_scrubbed":778,"log_size":9554,"log_dups_size":0,"ondisk_log_size":9554,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:27:45.242098+0000","scrub_duration":647,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23920640,"num_objects":365,"num_object_clones":0,"num_object_copies":730,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":365,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6152,"num_write_kb":145216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1],"acting":[9,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1a","version":"225'9945","reported_seq":15390,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-13T11:26:50.986782+0000","last_change":"2024-09-13T11:25:46.321420+0000","last_active":"2024-09-13T11:26:50.986782+0000","last_peered":"2024-09-13T11:26:50.986782+0000","last_clean":"2024-09-13T11:26:50.986782+0000","last_became_active":"2024-09-13T11:24:03.935185+0000","last_became_peered":"2024-09-13T11:24:03.935185+0000","last_unstale":"2024-09-13T11:26:50.986782+0000","last_undegraded":"2024-09-13T11:26:50.986782+0000","last_fullsized":"2024-09-13T11:26:50.986782+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":5,"last_scrub":"179'7826","last_scrub_stamp":"2024-09-13T11:25:46.321284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:46.321284+0000","objects_scrubbed":2526,"log_size":9945,"log_dups_size":0,"ondisk_log_size":9945,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:26:55.735552+0000","scrub_duration":935,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25886720,"num_objects":395,"num_object_clones":0,"num_object_copies":790,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":395,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5523,"num_write_kb":104448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9],"acting":[4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"225'10119","reported_seq":15901,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-13T11:26:50.996314+0000","last_change":"2024-09-13T11:25:49.890080+0000","last_active":"2024-09-13T11:26:50.996314+0000","last_peered":"2024-09-13T11:26:50.996314+0000","last_clean":"2024-09-13T11:26:50.996314+0000","last_became_active":"2024-09-13T11:24:04.950821+0000","last_became_peered":"2024-09-13T11:24:04.950821+0000","last_unstale":"2024-09-13T11:26:50.996314+0000","last_undegraded":"2024-09-13T11:26:50.996314+0000","last_fullsized":"2024-09-13T11:26:50.996314+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":5,"last_scrub":"182'8139","last_scrub_stamp":"2024-09-13T11:25:49.889914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:49.889914+0000","objects_scrubbed":2406,"log_size":10119,"log_dups_size":0,"ondisk_log_size":10119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:27:16.988785+0000","scrub_duration":519,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27328512,"num_objects":417,"num_object_clones":0,"num_object_copies":834,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":417,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5475,"num_write_kb":94656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,15],"acting":[12,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[]},{"pgid":"3.1c","version":"225'10129","reported_seq":14092,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-13T11:26:51.966293+0000","last_change":"2024-09-13T11:25:46.456795+0000","last_active":"2024-09-13T11:26:51.966293+0000","last_peered":"2024-09-13T11:26:51.966293+0000","last_clean":"2024-09-13T11:26:51.966293+0000","last_became_active":"2024-09-13T11:23:33.989492+0000","last_became_peered":"2024-09-13T11:23:33.989492+0000","last_unstale":"2024-09-13T11:26:51.966293+0000","last_undegraded":"2024-09-13T11:26:51.966293+0000","last_fullsized":"2024-09-13T11:26:51.966293+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":5,"last_scrub":"179'8037","last_scrub_stamp":"2024-09-13T11:25:46.456644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:46.456644+0000","objects_scrubbed":2528,"log_size":10129,"log_dups_size":0,"ondisk_log_size":10129,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":451,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28180480,"num_objects":430,"num_object_clones":0,"num_object_copies":860,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":430,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5854,"num_write_kb":119680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7],"acting":[11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1d","version":"225'9974","reported_seq":15579,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-13T11:26:50.994746+0000","last_change":"2024-09-13T11:25:46.627072+0000","last_active":"2024-09-13T11:26:50.994746+0000","last_peered":"2024-09-13T11:26:50.994746+0000","last_clean":"2024-09-13T11:26:50.994746+0000","last_became_active":"2024-09-13T11:24:01.258225+0000","last_became_peered":"2024-09-13T11:24:01.258225+0000","last_unstale":"2024-09-13T11:26:50.994746+0000","last_undegraded":"2024-09-13T11:26:50.994746+0000","last_fullsized":"2024-09-13T11:26:50.994746+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":5,"last_scrub":"180'7891","last_scrub_stamp":"2024-09-13T11:25:46.626952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:46.626952+0000","objects_scrubbed":2517,"log_size":9974,"log_dups_size":0,"ondisk_log_size":9974,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:27:01.444687+0000","scrub_duration":875,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27787288,"num_objects":425,"num_object_clones":0,"num_object_copies":850,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":425,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5408,"num_write_kb":94593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,15],"acting":[2,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1e","version":"225'10198","reported_seq":20840,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-13T11:26:50.987561+0000","last_change":"2024-09-13T11:24:08.338485+0000","last_active":"2024-09-13T11:26:50.987561+0000","last_peered":"2024-09-13T11:26:50.987561+0000","last_clean":"2024-09-13T11:26:50.987561+0000","last_became_active":"2024-09-13T11:24:01.261448+0000","last_became_peered":"2024-09-13T11:24:01.261448+0000","last_unstale":"2024-09-13T11:26:50.987561+0000","last_undegraded":"2024-09-13T11:26:50.987561+0000","last_fullsized":"2024-09-13T11:26:50.987561+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":5,"last_scrub":"110'4537","last_scrub_stamp":"2024-09-13T11:24:08.338354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:24:08.338354+0000","objects_scrubbed":2760,"log_size":10198,"log_dups_size":0,"ondisk_log_size":10198,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":367,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26869760,"num_objects":410,"num_object_clones":0,"num_object_copies":820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":410,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5670,"num_write_kb":106240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1f","version":"225'9883","reported_seq":23034,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-13T11:26:51.003895+0000","last_change":"2024-09-13T11:26:49.174841+0000","last_active":"2024-09-13T11:26:51.003895+0000","last_peered":"2024-09-13T11:26:51.003895+0000","last_clean":"2024-09-13T11:26:51.003895+0000","last_became_active":"2024-09-13T11:26:49.140281+0000","last_became_peered":"2024-09-13T11:26:49.140281+0000","last_unstale":"2024-09-13T11:26:51.003895+0000","last_undegraded":"2024-09-13T11:26:51.003895+0000","last_fullsized":"2024-09-13T11:26:51.003895+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":5,"last_scrub":"161'6865","last_scrub_stamp":"2024-09-13T11:25:20.773641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:20.773641+0000","objects_scrubbed":3425,"log_size":9883,"log_dups_size":0,"ondisk_log_size":9883,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":535,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26345472,"num_objects":402,"num_object_clones":0,"num_object_copies":804,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":402,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5463,"num_write_kb":102848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3836,"num_bytes_recovered":251396096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,14],"acting":[10,14],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.20","version":"225'6478","reported_seq":8778,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-13T11:26:50.996807+0000","last_change":"2024-09-13T11:25:35.773483+0000","last_active":"2024-09-13T11:26:50.996807+0000","last_peered":"2024-09-13T11:26:50.996807+0000","last_clean":"2024-09-13T11:26:50.996807+0000","last_became_active":"2024-09-13T11:23:33.954946+0000","last_became_peered":"2024-09-13T11:23:33.954946+0000","last_unstale":"2024-09-13T11:26:50.996807+0000","last_undegraded":"2024-09-13T11:26:50.996807+0000","last_fullsized":"2024-09-13T11:26:50.996807+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":6,"last_scrub":"174'5240","last_scrub_stamp":"2024-09-13T11:25:35.773409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:35.773409+0000","objects_scrubbed":1424,"log_size":6478,"log_dups_size":0,"ondisk_log_size":6478,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":215,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12058624,"num_objects":184,"num_object_clones":0,"num_object_copies":368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":184,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3213,"num_write_kb":74752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,0],"acting":[12,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[]},{"pgid":"3.21","version":"225'6278","reported_seq":9288,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-13T11:26:50.985893+0000","last_change":"2024-09-13T11:25:49.226484+0000","last_active":"2024-09-13T11:26:50.985893+0000","last_peered":"2024-09-13T11:26:50.985893+0000","last_clean":"2024-09-13T11:26:50.985893+0000","last_became_active":"2024-09-13T11:24:01.250308+0000","last_became_peered":"2024-09-13T11:24:01.250308+0000","last_unstale":"2024-09-13T11:26:50.985893+0000","last_undegraded":"2024-09-13T11:26:50.985893+0000","last_fullsized":"2024-09-13T11:26:50.985893+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":6,"last_scrub":"182'5306","last_scrub_stamp":"2024-09-13T11:25:49.226272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:49.226272+0000","objects_scrubbed":1176,"log_size":6278,"log_dups_size":0,"ondisk_log_size":6278,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:27:08.537716+0000","scrub_duration":151,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13238272,"num_objects":202,"num_object_clones":0,"num_object_copies":404,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":202,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2611,"num_write_kb":44224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8],"acting":[0,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"225'6441","reported_seq":18113,"reported_epoch":226,"state":"active+recovering+remapped","last_fresh":"2024-09-13T11:26:51.148292+0000","last_change":"2024-09-13T11:26:49.521922+0000","last_active":"2024-09-13T11:26:51.148292+0000","last_peered":"2024-09-13T11:26:51.148292+0000","last_clean":"2024-09-13T11:26:03.171705+0000","last_became_active":"2024-09-13T11:26:42.027697+0000","last_became_peered":"2024-09-13T11:26:42.027697+0000","last_unstale":"2024-09-13T11:26:51.148292+0000","last_undegraded":"2024-09-13T11:26:51.148292+0000","last_fullsized":"2024-09-13T11:26:51.148292+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":6,"last_scrub":"186'5611","last_scrub_stamp":"2024-09-13T11:25:57.123569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:57.123569+0000","objects_scrubbed":1032,"log_size":6441,"log_dups_size":0,"ondisk_log_size":6441,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":147,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13238272,"num_objects":202,"num_object_clones":0,"num_object_copies":404,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":114,"num_objects_unfound":0,"num_objects_dirty":202,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3197,"num_write_kb":75712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4192,"num_bytes_recovered":190251008,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,14],"acting":[0,14],"avail_no_missing":["0","5","14"],"object_location_counts":[{"shards":"0,5,14","objects":4},{"shards":"0,14","objects":88}],"blocked_by":[],"up_primary":12,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.23","version":"225'6541","reported_seq":9549,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-13T11:26:50.986449+0000","last_change":"2024-09-13T11:25:56.586649+0000","last_active":"2024-09-13T11:26:50.986449+0000","last_peered":"2024-09-13T11:26:50.986449+0000","last_clean":"2024-09-13T11:26:50.986449+0000","last_became_active":"2024-09-13T11:23:47.926677+0000","last_became_peered":"2024-09-13T11:23:47.926677+0000","last_unstale":"2024-09-13T11:26:50.986449+0000","last_undegraded":"2024-09-13T11:26:50.986449+0000","last_fullsized":"2024-09-13T11:26:50.986449+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":6,"last_scrub":"186'5649","last_scrub_stamp":"2024-09-13T11:25:56.586425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:56.586425+0000","objects_scrubbed":1098,"log_size":6541,"log_dups_size":0,"ondisk_log_size":6541,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:27:01.886969+0000","scrub_duration":135,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13434880,"num_objects":205,"num_object_clones":0,"num_object_copies":410,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":205,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3264,"num_write_kb":78080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9],"acting":[5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.24","version":"225'7298","reported_seq":15858,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-13T11:26:51.971468+0000","last_change":"2024-09-13T11:26:30.786438+0000","last_active":"2024-09-13T11:26:51.971468+0000","last_peered":"2024-09-13T11:26:51.971468+0000","last_clean":"2024-09-13T11:26:51.971468+0000","last_became_active":"2024-09-13T11:26:05.240552+0000","last_became_peered":"2024-09-13T11:26:05.240552+0000","last_unstale":"2024-09-13T11:26:51.971468+0000","last_undegraded":"2024-09-13T11:26:51.971468+0000","last_fullsized":"2024-09-13T11:26:51.971468+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":6,"last_scrub":"188'6512","last_scrub_stamp":"2024-09-13T11:26:01.118438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:26:01.118438+0000","objects_scrubbed":972,"log_size":7298,"log_dups_size":0,"ondisk_log_size":7298,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:27:06.263451+0000","scrub_duration":183,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12189696,"num_objects":186,"num_object_clones":0,"num_object_copies":372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":186,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2206,"num_write_kb":9280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2247,"num_bytes_recovered":61014016,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,14],"acting":[9,14],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.25","version":"225'7254","reported_seq":13108,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-13T11:26:50.996240+0000","last_change":"2024-09-13T11:24:57.073162+0000","last_active":"2024-09-13T11:26:50.996240+0000","last_peered":"2024-09-13T11:26:50.996240+0000","last_clean":"2024-09-13T11:26:50.996240+0000","last_became_active":"2024-09-13T11:24:55.999019+0000","last_became_peered":"2024-09-13T11:24:55.999019+0000","last_unstale":"2024-09-13T11:26:50.996240+0000","last_undegraded":"2024-09-13T11:26:50.996240+0000","last_fullsized":"2024-09-13T11:26:50.996240+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":6,"last_scrub":"142'5214","last_scrub_stamp":"2024-09-13T11:24:57.073061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:24:57.073061+0000","objects_scrubbed":2249,"log_size":7254,"log_dups_size":0,"ondisk_log_size":7254,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":571,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13893632,"num_objects":212,"num_object_clones":0,"num_object_copies":424,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":212,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2043,"num_write_kb":192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,2],"acting":[12,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[]},{"pgid":"3.26","version":"225'7364","reported_seq":13075,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-13T11:26:50.981567+0000","last_change":"2024-09-13T11:26:11.113925+0000","last_active":"2024-09-13T11:26:50.981567+0000","last_peered":"2024-09-13T11:26:50.981567+0000","last_clean":"2024-09-13T11:26:50.981567+0000","last_became_active":"2024-09-13T11:24:54.976919+0000","last_became_peered":"2024-09-13T11:24:54.976919+0000","last_unstale":"2024-09-13T11:26:50.981567+0000","last_undegraded":"2024-09-13T11:26:50.981567+0000","last_fullsized":"2024-09-13T11:26:50.981567+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":6,"last_scrub":"196'6743","last_scrub_stamp":"2024-09-13T11:26:11.113811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:26:11.113811+0000","objects_scrubbed":840,"log_size":7364,"log_dups_size":0,"ondisk_log_size":7364,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:27:18.228492+0000","scrub_duration":103,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14352384,"num_objects":219,"num_object_clones":0,"num_object_copies":438,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":219,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1939,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,12],"acting":[8,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.27","version":"225'7267","reported_seq":12929,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-13T11:26:50.986905+0000","last_change":"2024-09-13T11:25:37.344464+0000","last_active":"2024-09-13T11:26:50.986905+0000","last_peered":"2024-09-13T11:26:50.986905+0000","last_clean":"2024-09-13T11:26:50.986905+0000","last_became_active":"2024-09-13T11:25:30.257911+0000","last_became_peered":"2024-09-13T11:25:30.257911+0000","last_unstale":"2024-09-13T11:26:50.986905+0000","last_undegraded":"2024-09-13T11:26:50.986905+0000","last_fullsized":"2024-09-13T11:26:50.986905+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":6,"last_scrub":"174'6026","last_scrub_stamp":"2024-09-13T11:25:37.344335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:37.344335+0000","objects_scrubbed":1463,"log_size":7267,"log_dups_size":0,"ondisk_log_size":7267,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":195,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14483456,"num_objects":221,"num_object_clones":0,"num_object_copies":442,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":221,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1242,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,11],"acting":[5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.28","version":"225'7175","reported_seq":12718,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-13T11:26:50.981428+0000","last_change":"2024-09-13T11:25:36.143261+0000","last_active":"2024-09-13T11:26:50.981428+0000","last_peered":"2024-09-13T11:26:50.981428+0000","last_clean":"2024-09-13T11:26:50.981428+0000","last_became_active":"2024-09-13T11:25:34.278640+0000","last_became_peered":"2024-09-13T11:25:34.278640+0000","last_unstale":"2024-09-13T11:26:50.981428+0000","last_undegraded":"2024-09-13T11:26:50.981428+0000","last_fullsized":"2024-09-13T11:26:50.981428+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":6,"last_scrub":"174'5936","last_scrub_stamp":"2024-09-13T11:25:36.143113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:36.143113+0000","objects_scrubbed":1447,"log_size":7175,"log_dups_size":0,"ondisk_log_size":7175,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:27:05.839849+0000","scrub_duration":1147,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12386304,"num_objects":189,"num_object_clones":0,"num_object_copies":378,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":189,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1258,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5],"acting":[8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.29","version":"225'7316","reported_seq":13132,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-13T11:26:50.986505+0000","last_change":"2024-09-13T11:25:13.333062+0000","last_active":"2024-09-13T11:26:50.986505+0000","last_peered":"2024-09-13T11:26:50.986505+0000","last_clean":"2024-09-13T11:26:50.986505+0000","last_became_active":"2024-09-13T11:25:12.944028+0000","last_became_peered":"2024-09-13T11:25:12.944028+0000","last_unstale":"2024-09-13T11:26:50.986505+0000","last_undegraded":"2024-09-13T11:26:50.986505+0000","last_fullsized":"2024-09-13T11:26:50.986505+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":6,"last_scrub":"154'5579","last_scrub_stamp":"2024-09-13T11:25:13.332959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:13.332959+0000","objects_scrubbed":1937,"log_size":7316,"log_dups_size":0,"ondisk_log_size":7316,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":243,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12976128,"num_objects":198,"num_object_clones":0,"num_object_copies":396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":198,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1739,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2a","version":"225'7717","reported_seq":11735,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-13T11:26:51.003709+0000","last_change":"2024-09-13T11:26:09.477886+0000","last_active":"2024-09-13T11:26:51.003709+0000","last_peered":"2024-09-13T11:26:51.003709+0000","last_clean":"2024-09-13T11:26:51.003709+0000","last_became_active":"2024-09-13T11:24:45.320540+0000","last_became_peered":"2024-09-13T11:24:45.320540+0000","last_unstale":"2024-09-13T11:26:51.003709+0000","last_undegraded":"2024-09-13T11:26:51.003709+0000","last_fullsized":"2024-09-13T11:26:51.003709+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":6,"last_scrub":"195'7004","last_scrub_stamp":"2024-09-13T11:26:09.477793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:26:09.477793+0000","objects_scrubbed":929,"log_size":7717,"log_dups_size":0,"ondisk_log_size":7717,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:27:14.239925+0000","scrub_duration":115,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14155776,"num_objects":216,"num_object_clones":0,"num_object_copies":432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":216,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2228,"num_write_kb":11200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,7],"acting":[10,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.2b","version":"225'7471","reported_seq":11299,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-09-13T11:26:50.111290+0000","last_change":"2024-09-13T11:26:01.126562+0000","last_active":"2024-09-13T11:26:50.111290+0000","last_peered":"2024-09-13T11:26:50.111290+0000","last_clean":"2024-09-13T11:26:50.111290+0000","last_became_active":"2024-09-13T11:24:26.180578+0000","last_became_peered":"2024-09-13T11:24:26.180578+0000","last_unstale":"2024-09-13T11:26:50.111290+0000","last_undegraded":"2024-09-13T11:26:50.111290+0000","last_fullsized":"2024-09-13T11:26:50.111290+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":6,"last_scrub":"188'6745","last_scrub_stamp":"2024-09-13T11:26:01.126423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:26:01.126423+0000","objects_scrubbed":916,"log_size":7471,"log_dups_size":0,"ondisk_log_size":7471,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:27:28.092997+0000","scrub_duration":391,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11993088,"num_objects":183,"num_object_clones":0,"num_object_copies":366,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":183,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2089,"num_write_kb":10624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2c","version":"225'7667","reported_seq":13136,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-13T11:26:51.003817+0000","last_change":"2024-09-13T11:26:34.433494+0000","last_active":"2024-09-13T11:26:51.003817+0000","last_peered":"2024-09-13T11:26:51.003817+0000","last_clean":"2024-09-13T11:26:51.003817+0000","last_became_active":"2024-09-13T11:24:59.021895+0000","last_became_peered":"2024-09-13T11:24:59.021895+0000","last_unstale":"2024-09-13T11:26:51.003817+0000","last_undegraded":"2024-09-13T11:26:51.003817+0000","last_fullsized":"2024-09-13T11:26:51.003817+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":6,"last_scrub":"211'7422","last_scrub_stamp":"2024-09-13T11:26:34.433371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:26:34.433371+0000","objects_scrubbed":442,"log_size":7667,"log_dups_size":0,"ondisk_log_size":7667,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:27:39.180701+0000","scrub_duration":63,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12910592,"num_objects":197,"num_object_clones":0,"num_object_copies":394,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":197,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1995,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,7],"acting":[10,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.2d","version":"225'7599","reported_seq":13238,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-13T11:26:50.985310+0000","last_change":"2024-09-13T11:25:45.355259+0000","last_active":"2024-09-13T11:26:50.985310+0000","last_peered":"2024-09-13T11:26:50.985310+0000","last_clean":"2024-09-13T11:26:50.985310+0000","last_became_active":"2024-09-13T11:25:39.334408+0000","last_became_peered":"2024-09-13T11:25:39.334408+0000","last_unstale":"2024-09-13T11:26:50.985310+0000","last_undegraded":"2024-09-13T11:26:50.985310+0000","last_fullsized":"2024-09-13T11:26:50.985310+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":6,"last_scrub":"179'6515","last_scrub_stamp":"2024-09-13T11:25:45.355191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:45.355191+0000","objects_scrubbed":1320,"log_size":7599,"log_dups_size":0,"ondisk_log_size":7599,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:26:52.101869+0000","scrub_duration":199,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15204352,"num_objects":232,"num_object_clones":0,"num_object_copies":464,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":232,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1088,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2e","version":"223'10094","reported_seq":15504,"reported_epoch":226,"state":"remapped","last_fresh":"2024-09-13T11:26:50.985046+0000","last_change":"2024-09-13T11:26:51.150228+0000","last_active":"2024-09-13T11:26:50.985046+0000","last_peered":"2024-09-13T11:26:48.728528+0000","last_clean":"2024-09-13T11:26:48.728528+0000","last_became_active":"2024-09-13T11:23:31.925869+0000","last_became_peered":"2024-09-13T11:23:31.925869+0000","last_unstale":"2024-09-13T11:26:50.985046+0000","last_undegraded":"2024-09-13T11:26:50.985046+0000","last_fullsized":"2024-09-13T11:26:50.985046+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":6,"last_scrub":"180'8032","last_scrub_stamp":"2024-09-13T11:25:46.438107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:46.438107+0000","objects_scrubbed":2456,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":327,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12779520,"num_objects":195,"num_object_clones":0,"num_object_copies":390,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":195,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3284,"num_write_kb":82944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11],"acting":[0,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2f","version":"225'10074","reported_seq":10601,"reported_epoch":227,"state":"remapped+peering","last_fresh":"2024-09-13T11:26:51.967535+0000","last_change":"2024-09-13T11:26:51.967535+0000","last_active":"2024-09-13T11:26:51.967535+0000","last_peered":"2024-09-13T11:26:50.828909+0000","last_clean":"2024-09-13T11:26:50.828909+0000","last_became_active":"2024-09-13T11:26:50.028809+0000","last_became_peered":"2024-09-13T11:26:50.028809+0000","last_unstale":"2024-09-13T11:26:51.967535+0000","last_undegraded":"2024-09-13T11:26:51.967535+0000","last_fullsized":"2024-09-13T11:26:51.967535+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":6,"last_scrub":"211'9653","last_scrub_stamp":"2024-09-13T11:26:34.940878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:26:34.940878+0000","objects_scrubbed":877,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":119,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14516224,"num_objects":222,"num_object_clones":0,"num_object_copies":444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":222,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3332,"num_write_kb":85280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,4],"acting":[11,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T11:26:48.959745+0000","last_change":"2024-09-13T11:26:48.959745+0000","last_active":"2024-09-13T11:26:48.959745+0000","last_peered":"2024-09-13T11:26:48.959745+0000","last_clean":"2024-09-13T11:26:48.959745+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T11:26:48.959745+0000","last_undegraded":"2024-09-13T11:26:48.959745+0000","last_fullsized":"2024-09-13T11:26:48.959745+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:26:48.959745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:26:48.959745+0000","last_clean_scrub_stamp":"2024-09-13T11:26:48.959745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T11:26:48.959745+0000","last_change":"2024-09-13T11:26:48.959745+0000","last_active":"2024-09-13T11:26:48.959745+0000","last_peered":"2024-09-13T11:26:48.959745+0000","last_clean":"2024-09-13T11:26:48.959745+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T11:26:48.959745+0000","last_undegraded":"2024-09-13T11:26:48.959745+0000","last_fullsized":"2024-09-13T11:26:48.959745+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:26:48.959745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:26:48.959745+0000","last_clean_scrub_stamp":"2024-09-13T11:26:48.959745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":56,"stat_sum":{"num_bytes":786726936,"num_objects":12004,"num_object_clones":0,"num_object_copies":24008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":167,"num_objects_unfound":0,"num_objects_dirty":12004,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":157707,"num_write_kb":2898433,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24441,"num_bytes_recovered":1249378304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2020745216,"data_stored":2020737072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":43282,"internal_metadata":0},"log_size":416599,"ondisk_log_size":416599,"up":108,"acting":108,"num_store_stats":16},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":11},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":112,"ondisk_log_size":112,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":13,"up_from":23,"seq":98784247868,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":195512,"kb_used_data":83816,"kb_used_omap":1,"kb_used_meta":111678,"kb_avail":94176328,"statfs":{"total":96636764160,"available":96436559872,"internally_reserved":0,"allocated":85827584,"data_stored":85388180,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":114358725},"hb_peers":[0,1,2,4,6,8,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":23,"seq":98784247867,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":142048,"kb_used_data":74768,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94229792,"statfs":{"total":96636764160,"available":96491307008,"internally_reserved":0,"allocated":76562432,"data_stored":76151537,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":68876736},"hb_peers":[0,1,3,8,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":40,"seq":171798691890,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":232044,"kb_used_data":127836,"kb_used_omap":4,"kb_used_meta":104187,"kb_avail":94139796,"statfs":{"total":96636764160,"available":96399151104,"internally_reserved":0,"allocated":130904064,"data_stored":130523532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4433,"internal_metadata":106688175},"hb_peers":[0,2,4,6,7,8,9,11,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":21,"seq":90194313275,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":257040,"kb_used_data":117568,"kb_used_omap":1,"kb_used_meta":139454,"kb_avail":94114800,"statfs":{"total":96636764160,"available":96373555200,"internally_reserved":0,"allocated":120389632,"data_stored":119974461,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":142801351},"hb_peers":[0,2,4,7,9,10,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,1,4],"upper_bound":2048},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":21,"seq":90194313278,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":332808,"kb_used_data":207224,"kb_used_omap":1,"kb_used_meta":125566,"kb_avail":94039032,"statfs":{"total":96636764160,"available":96295968768,"internally_reserved":0,"allocated":212197376,"data_stored":211796713,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":128580039},"hb_peers":[0,2,4,6,7,9,11,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":22,"seq":94489280570,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":304224,"kb_used_data":156432,"kb_used_omap":1,"kb_used_meta":147774,"kb_avail":94067616,"statfs":{"total":96636764160,"available":96325238784,"internally_reserved":0,"allocated":160186368,"data_stored":159762975,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":151321029},"hb_peers":[1,3,5,7,8,10,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":22,"seq":94489280571,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":121632,"kb_used_data":54352,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94250208,"statfs":{"total":96636764160,"available":96512212992,"internally_reserved":0,"allocated":55656448,"data_stored":55232560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,1,3,4,5,7,8,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":15,"up_from":22,"seq":94489280571,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":273924,"kb_used_data":134452,"kb_used_omap":1,"kb_used_meta":139454,"kb_avail":94097916,"statfs":{"total":96636764160,"available":96356265984,"internally_reserved":0,"allocated":137678848,"data_stored":137281260,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":142801350},"hb_peers":[0,4,5,6,7,8,10,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":22,"seq":94489280572,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":167756,"kb_used_data":96316,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":94204084,"statfs":{"total":96636764160,"available":96464982016,"internally_reserved":0,"allocated":98627584,"data_stored":98222599,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73136581},"hb_peers":[3,5,6,7,9,10,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":22,"seq":94489280571,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":132828,"kb_used_data":57228,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94239012,"statfs":{"total":96636764160,"available":96500748288,"internally_reserved":0,"allocated":58601472,"data_stored":58182049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":77396422},"hb_peers":[0,1,3,5,7,8,10,11,12,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":22,"seq":94489280572,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":301372,"kb_used_data":161900,"kb_used_omap":1,"kb_used_meta":139454,"kb_avail":94070468,"statfs":{"total":96636764160,"available":96328159232,"internally_reserved":0,"allocated":165785600,"data_stored":165394639,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":142801354},"hb_peers":[0,3,5,6,8,9,10,11,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":22,"seq":94489280571,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":117988,"kb_used_data":54868,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94253852,"statfs":{"total":96636764160,"available":96515944448,"internally_reserved":0,"allocated":56184832,"data_stored":55757508,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[1,5,6,7,9,10,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":22,"seq":94489280571,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":232160,"kb_used_data":120464,"kb_used_omap":1,"kb_used_meta":111678,"kb_avail":94139680,"statfs":{"total":96636764160,"available":96399032320,"internally_reserved":0,"allocated":123355136,"data_stored":122931520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":114358720},"hb_peers":[0,1,3,5,7,9,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":11,"up_from":22,"seq":94489280572,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":320760,"kb_used_data":181288,"kb_used_omap":1,"kb_used_meta":139454,"kb_avail":94051080,"statfs":{"total":96636764160,"available":96308305920,"internally_reserved":0,"allocated":185638912,"data_stored":185199220,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":142801348},"hb_peers":[0,1,2,4,7,8,10,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":12,"up_from":22,"seq":94489280571,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298140,"kb_used_data":158668,"kb_used_omap":1,"kb_used_meta":139454,"kb_avail":94073700,"statfs":{"total":96636764160,"available":96331468800,"internally_reserved":0,"allocated":162476032,"data_stored":162056088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":142801349},"hb_peers":[0,1,2,3,4,5,10,11,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":14,"up_from":142,"seq":609885356056,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":291876,"kb_used_data":223892,"kb_used_omap":42,"kb_used_meta":67925,"kb_avail":94079964,"statfs":{"total":96636764160,"available":96337883136,"internally_reserved":0,"allocated":229265408,"data_stored":228890709,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":43282,"internal_metadata":69555950},"hb_peers":[0,1,3,5,7,8,9,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":157941760,"data_stored":157941760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":126287872,"data_stored":126287872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54333440,"data_stored":54329368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":112066560,"data_stored":112066560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":96731136,"data_stored":96731136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":210173952,"data_stored":210173952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":56885248,"data_stored":56885248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":163053568,"data_stored":163053568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":54394880,"data_stored":54394880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":75431936,"data_stored":75431936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":119996416,"data_stored":119996416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":184483840,"data_stored":184483840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":163577856,"data_stored":163577856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":84606976,"data_stored":84606976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":227213312,"data_stored":227213312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":43282,"internal_metadata":0},{"poolid":3,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":133566464,"data_stored":133562392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T11:26:54.187 INFO:tasks.thrashosds.thrasher:Setting 3.17 to [9, 13, 8, 5] 2024-09-13T11:26:54.188 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap-items', '3.17', '9', '13', '8', '5'] 2024-09-13T11:26:54.188 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pg-upmap-items 3.17 9 13 8 5 2024-09-13T11:26:54.202 INFO:tasks.ceph.osd.2.smithi162.stderr:2024-09-13T11:26:54.199+0000 7fe92b35a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26313) UID: 0 2024-09-13T11:26:54.244 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-09-13T11:26:54.244 INFO:teuthology.orchestra.run.smithi092.stdout: "ops": [], 2024-09-13T11:26:54.244 INFO:teuthology.orchestra.run.smithi092.stdout: "num_ops": 0 2024-09-13T11:26:54.245 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-09-13T11:26:54.261 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_blocked_ops 2024-09-13T11:26:54.305 INFO:tasks.ceph.osd.10.smithi162.stderr:2024-09-13T11:26:54.303+0000 7fb59c9f0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26317) UID: 0 2024-09-13T11:26:54.396 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-09-13T11:26:54.396 INFO:teuthology.orchestra.run.smithi092.stdout: "ops": [], 2024-09-13T11:26:54.397 INFO:teuthology.orchestra.run.smithi092.stdout: "complaint_time": 30, 2024-09-13T11:26:54.397 INFO:teuthology.orchestra.run.smithi092.stdout: "num_blocked_ops": 0 2024-09-13T11:26:54.397 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-09-13T11:26:54.406 INFO:tasks.ceph.osd.7.smithi177.stderr:2024-09-13T11:26:54.405+0000 7f897b51d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26575) UID: 0 2024-09-13T11:26:54.414 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_historic_ops 2024-09-13T11:26:54.507 INFO:tasks.ceph.osd.12.smithi001.stderr:2024-09-13T11:26:54.502+0000 7f892c6e6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26314) UID: 0 2024-09-13T11:26:54.553 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-09-13T11:26:54.553 INFO:teuthology.orchestra.run.smithi092.stdout: "size": 20, 2024-09-13T11:26:54.553 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 600, 2024-09-13T11:26:54.553 INFO:teuthology.orchestra.run.smithi092.stdout: "ops": [ 2024-09-13T11:26:54.554 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.554 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:79 3.1 3:841df5b5:::benchmark_data_smithi177_31116_object78:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-09-13T11:26:54.554 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:27.056083+0000", 2024-09-13T11:26:54.554 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 267.49525378999999, 2024-09-13T11:26:54.554 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.89469730199999997, 2024-09-13T11:26:54.554 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:26:54.554 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:26:54.554 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:54.554 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:26:54.554 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:26:54.554 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:54.555 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 79 2024-09-13T11:26:54.555 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.555 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:26:54.555 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.555 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:26:54.555 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.056083+0000", 2024-09-13T11:26:54.555 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:54.555 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.555 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.555 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:26:54.555 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.056083+0000", 2024-09-13T11:26:54.556 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:54.556 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.556 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.556 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:26:54.556 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.056087+0000", 2024-09-13T11:26:54.556 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.0659999999999997e-06 2024-09-13T11:26:54.556 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.556 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.556 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:26:54.556 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.056148+0000", 2024-09-13T11:26:54.556 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.1376000000000003e-05 2024-09-13T11:26:54.556 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.557 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.557 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:26:54.557 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.056180+0000", 2024-09-13T11:26:54.557 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.2370999999999999e-05 2024-09-13T11:26:54.557 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.557 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.557 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:26:54.557 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.056195+0000", 2024-09-13T11:26:54.558 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.5119e-05 2024-09-13T11:26:54.558 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.558 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.558 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:54.558 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.056304+0000", 2024-09-13T11:26:54.558 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000108365 2024-09-13T11:26:54.558 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.558 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.558 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:26:54.558 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.056478+0000", 2024-09-13T11:26:54.558 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000174542 2024-09-13T11:26:54.558 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.559 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.559 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 11", 2024-09-13T11:26:54.559 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.056690+0000", 2024-09-13T11:26:54.559 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00021167500000000001 2024-09-13T11:26:54.559 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.559 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.559 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:26:54.559 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.058063+0000", 2024-09-13T11:26:54.559 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0013731819999999999 2024-09-13T11:26:54.559 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.559 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.559 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:54.560 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.950477+0000", 2024-09-13T11:26:54.560 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.89241369400000004 2024-09-13T11:26:54.560 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.560 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.560 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:26:54.560 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.950652+0000", 2024-09-13T11:26:54.560 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00017526400000000001 2024-09-13T11:26:54.560 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.560 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.560 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:26:54.560 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.950780+0000", 2024-09-13T11:26:54.560 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00012764800000000001 2024-09-13T11:26:54.561 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:54.561 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:54.561 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:54.561 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.561 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.561 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:7 3.b 3:de78393b:::benchmark_data_smithi177_31116_object6:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T11:26:54.561 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:48.313952+0000", 2024-09-13T11:26:54.561 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 246.23738404100001, 2024-09-13T11:26:54.561 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.86421288600000001, 2024-09-13T11:26:54.561 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:26:54.561 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:26:54.561 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:54.562 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:26:54.562 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:26:54.562 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:54.562 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 7 2024-09-13T11:26:54.562 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.562 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:26:54.562 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.562 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:26:54.562 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.313952+0000", 2024-09-13T11:26:54.562 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:54.562 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.563 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.563 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:26:54.563 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.313952+0000", 2024-09-13T11:26:54.563 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:54.563 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.563 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.563 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:26:54.563 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.313957+0000", 2024-09-13T11:26:54.563 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.3599999999999998e-06 2024-09-13T11:26:54.563 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.563 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.563 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:26:54.564 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314043+0000", 2024-09-13T11:26:54.564 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.6766000000000002e-05 2024-09-13T11:26:54.564 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.564 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.564 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:26:54.564 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314065+0000", 2024-09-13T11:26:54.564 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.1416e-05 2024-09-13T11:26:54.564 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.564 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.564 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:26:54.564 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314080+0000", 2024-09-13T11:26:54.564 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.5519000000000002e-05 2024-09-13T11:26:54.565 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.565 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.565 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:54.565 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315478+0000", 2024-09-13T11:26:54.565 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.001397765 2024-09-13T11:26:54.565 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.565 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.565 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-09-13T11:26:54.565 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315507+0000", 2024-09-13T11:26:54.565 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.9128999999999998e-05 2024-09-13T11:26:54.565 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.565 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.565 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:54.566 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.167978+0000", 2024-09-13T11:26:54.566 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.85247068400000003 2024-09-13T11:26:54.566 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.566 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.566 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:26:54.566 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.168307+0000", 2024-09-13T11:26:54.566 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00032886699999999999 2024-09-13T11:26:54.566 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.566 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.566 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1", 2024-09-13T11:26:54.566 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.168647+0000", 2024-09-13T11:26:54.566 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00033998399999999999 2024-09-13T11:26:54.567 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.567 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.567 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:54.567 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.178011+0000", 2024-09-13T11:26:54.567 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0093639169999999994 2024-09-13T11:26:54.567 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.567 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.567 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:26:54.567 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.178026+0000", 2024-09-13T11:26:54.567 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.5274999999999999e-05 2024-09-13T11:26:54.567 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.567 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.567 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:26:54.568 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.178080+0000", 2024-09-13T11:26:54.568 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.3601999999999999e-05 2024-09-13T11:26:54.568 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.568 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.568 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:26:54.568 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.178165+0000", 2024-09-13T11:26:54.568 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.5601999999999998e-05 2024-09-13T11:26:54.568 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:54.568 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:54.568 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:54.568 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.568 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.569 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:9 3.b 3:dafadff6:::benchmark_data_smithi177_31116_object8:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T11:26:54.569 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:48.314114+0000", 2024-09-13T11:26:54.569 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 246.23722222699999, 2024-09-13T11:26:54.569 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.86613615899999996, 2024-09-13T11:26:54.569 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:26:54.569 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:26:54.569 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:54.569 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:26:54.569 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:26:54.569 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:54.569 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 9 2024-09-13T11:26:54.569 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.570 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:26:54.570 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.570 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:26:54.570 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314114+0000", 2024-09-13T11:26:54.570 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:54.570 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.570 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.570 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:26:54.570 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314114+0000", 2024-09-13T11:26:54.570 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:54.570 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.570 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.571 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:26:54.571 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314117+0000", 2024-09-13T11:26:54.571 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.3860000000000001e-06 2024-09-13T11:26:54.571 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.571 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.571 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:26:54.571 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314215+0000", 2024-09-13T11:26:54.571 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.8900000000000005e-05 2024-09-13T11:26:54.571 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.571 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.571 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:26:54.572 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314229+0000", 2024-09-13T11:26:54.572 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.3444999999999999e-05 2024-09-13T11:26:54.572 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.572 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.572 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:26:54.572 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314239+0000", 2024-09-13T11:26:54.572 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.8470000000000008e-06 2024-09-13T11:26:54.572 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.572 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.573 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:54.573 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315648+0000", 2024-09-13T11:26:54.573 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.001409291 2024-09-13T11:26:54.573 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.573 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.573 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-09-13T11:26:54.573 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315671+0000", 2024-09-13T11:26:54.573 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.3169000000000001e-05 2024-09-13T11:26:54.573 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.573 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.573 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:54.573 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.176036+0000", 2024-09-13T11:26:54.574 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.86036497499999998 2024-09-13T11:26:54.574 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.574 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.574 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:26:54.574 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.176189+0000", 2024-09-13T11:26:54.574 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00015271100000000001 2024-09-13T11:26:54.574 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.574 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.575 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1", 2024-09-13T11:26:54.575 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.176339+0000", 2024-09-13T11:26:54.575 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00015057499999999999 2024-09-13T11:26:54.575 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.575 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.575 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:54.575 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.180148+0000", 2024-09-13T11:26:54.575 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.003808891 2024-09-13T11:26:54.575 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.575 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.575 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:26:54.575 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.180168+0000", 2024-09-13T11:26:54.576 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.9507000000000001e-05 2024-09-13T11:26:54.576 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.576 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.576 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:26:54.576 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.180213+0000", 2024-09-13T11:26:54.576 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.5504999999999999e-05 2024-09-13T11:26:54.576 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.576 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.576 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:26:54.576 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.180250+0000", 2024-09-13T11:26:54.576 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.6956999999999999e-05 2024-09-13T11:26:54.576 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:54.576 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:54.577 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:54.577 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.577 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.577 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:10 3.b 3:dbe028a8:::benchmark_data_smithi177_31116_object9:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T11:26:54.577 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:48.314259+0000", 2024-09-13T11:26:54.577 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 246.237077097, 2024-09-13T11:26:54.577 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.86637313599999999, 2024-09-13T11:26:54.577 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:26:54.577 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:26:54.577 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:54.577 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:26:54.578 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:26:54.578 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:54.578 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 10 2024-09-13T11:26:54.578 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.578 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:26:54.578 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.578 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:26:54.578 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314259+0000", 2024-09-13T11:26:54.578 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:54.578 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.578 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.578 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:26:54.578 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314259+0000", 2024-09-13T11:26:54.579 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:54.579 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.579 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.579 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:26:54.579 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314261+0000", 2024-09-13T11:26:54.579 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.1500000000000002e-06 2024-09-13T11:26:54.579 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.579 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.579 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:26:54.579 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314349+0000", 2024-09-13T11:26:54.579 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.8070000000000002e-05 2024-09-13T11:26:54.579 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.579 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.580 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:26:54.580 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314365+0000", 2024-09-13T11:26:54.580 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.5835999999999999e-05 2024-09-13T11:26:54.580 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.580 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.580 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:26:54.580 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314377+0000", 2024-09-13T11:26:54.580 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.1589e-05 2024-09-13T11:26:54.580 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.580 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.580 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:54.581 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315788+0000", 2024-09-13T11:26:54.581 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0014109579999999999 2024-09-13T11:26:54.581 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.581 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.581 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-09-13T11:26:54.581 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315809+0000", 2024-09-13T11:26:54.581 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.0985999999999999e-05 2024-09-13T11:26:54.581 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.581 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.581 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:54.581 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.176822+0000", 2024-09-13T11:26:54.581 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.86101310799999997 2024-09-13T11:26:54.582 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.582 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.582 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:26:54.582 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.177021+0000", 2024-09-13T11:26:54.582 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00019911199999999999 2024-09-13T11:26:54.582 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.582 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.582 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1", 2024-09-13T11:26:54.582 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.177226+0000", 2024-09-13T11:26:54.582 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00020516600000000001 2024-09-13T11:26:54.582 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.582 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.582 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:26:54.583 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.180264+0000", 2024-09-13T11:26:54.583 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0030381739999999998 2024-09-13T11:26:54.583 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.583 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.583 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:54.583 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.180387+0000", 2024-09-13T11:26:54.583 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00012257699999999999 2024-09-13T11:26:54.583 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.583 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.583 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:26:54.583 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.180518+0000", 2024-09-13T11:26:54.583 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00013135800000000001 2024-09-13T11:26:54.583 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.584 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.584 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:26:54.584 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.180632+0000", 2024-09-13T11:26:54.584 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000114052 2024-09-13T11:26:54.584 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:54.584 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:54.584 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:54.584 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.584 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.584 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:39 3.b 3:d91854b3:::benchmark_data_smithi177_31116_object38:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T11:26:54.584 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:48.314393+0000", 2024-09-13T11:26:54.584 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 246.23694355500001, 2024-09-13T11:26:54.584 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.86850121700000005, 2024-09-13T11:26:54.585 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:26:54.585 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:26:54.585 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:54.585 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:26:54.585 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:26:54.585 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:54.585 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 39 2024-09-13T11:26:54.585 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.585 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:26:54.585 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.585 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:26:54.585 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314393+0000", 2024-09-13T11:26:54.585 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:54.586 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.586 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.586 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:26:54.586 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314393+0000", 2024-09-13T11:26:54.586 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:54.586 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.586 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.586 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:26:54.586 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314394+0000", 2024-09-13T11:26:54.586 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.513e-06 2024-09-13T11:26:54.586 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.586 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.587 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:26:54.587 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314463+0000", 2024-09-13T11:26:54.587 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.8261000000000003e-05 2024-09-13T11:26:54.587 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.587 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.587 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:26:54.587 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314477+0000", 2024-09-13T11:26:54.587 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.4619e-05 2024-09-13T11:26:54.587 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.587 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.587 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:26:54.587 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314485+0000", 2024-09-13T11:26:54.588 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.1699999999999997e-06 2024-09-13T11:26:54.588 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.588 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.588 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:54.588 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315917+0000", 2024-09-13T11:26:54.588 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0014316089999999999 2024-09-13T11:26:54.588 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.588 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.588 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-09-13T11:26:54.588 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315933+0000", 2024-09-13T11:26:54.588 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.5999999999999999e-05 2024-09-13T11:26:54.588 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.589 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.589 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:54.589 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.178205+0000", 2024-09-13T11:26:54.589 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.86227247100000004 2024-09-13T11:26:54.589 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.589 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.589 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:26:54.589 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.178316+0000", 2024-09-13T11:26:54.589 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000110788 2024-09-13T11:26:54.589 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.589 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.590 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1", 2024-09-13T11:26:54.590 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.178448+0000", 2024-09-13T11:26:54.590 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000132056 2024-09-13T11:26:54.590 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.590 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.590 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:26:54.590 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.182721+0000", 2024-09-13T11:26:54.590 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0042729780000000002 2024-09-13T11:26:54.590 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.590 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.590 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:54.591 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.182801+0000", 2024-09-13T11:26:54.591 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.0209000000000004e-05 2024-09-13T11:26:54.591 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.591 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.591 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:26:54.591 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.182855+0000", 2024-09-13T11:26:54.591 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.3749000000000003e-05 2024-09-13T11:26:54.591 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.591 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.591 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:26:54.591 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.182894+0000", 2024-09-13T11:26:54.591 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.8794000000000002e-05 2024-09-13T11:26:54.591 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:54.592 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:54.592 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:54.592 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.592 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.592 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:50 3.b 3:d1c144b8:::benchmark_data_smithi177_31116_object49:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T11:26:54.592 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:48.314498+0000", 2024-09-13T11:26:54.592 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 246.23683814, 2024-09-13T11:26:54.592 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.86878673299999998, 2024-09-13T11:26:54.592 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:26:54.592 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:26:54.592 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:54.592 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:26:54.592 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:26:54.593 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:54.593 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 50 2024-09-13T11:26:54.593 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.593 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:26:54.593 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.593 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:26:54.593 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314498+0000", 2024-09-13T11:26:54.593 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:54.593 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.593 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.593 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:26:54.593 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314498+0000", 2024-09-13T11:26:54.594 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:54.594 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.594 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.594 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:26:54.594 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314499+0000", 2024-09-13T11:26:54.594 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.1969999999999999e-06 2024-09-13T11:26:54.594 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.594 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.594 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:26:54.594 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314565+0000", 2024-09-13T11:26:54.594 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.5628000000000003e-05 2024-09-13T11:26:54.595 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.595 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.595 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:26:54.595 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314579+0000", 2024-09-13T11:26:54.595 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.4222e-05 2024-09-13T11:26:54.595 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.595 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.595 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:26:54.595 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314590+0000", 2024-09-13T11:26:54.595 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.1240000000000001e-05 2024-09-13T11:26:54.595 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.596 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.596 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:54.596 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316060+0000", 2024-09-13T11:26:54.596 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.001469515 2024-09-13T11:26:54.596 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.596 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.596 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-09-13T11:26:54.596 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316076+0000", 2024-09-13T11:26:54.596 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.5608999999999999e-05 2024-09-13T11:26:54.596 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.596 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.597 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:54.597 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.178857+0000", 2024-09-13T11:26:54.597 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.86278099900000005 2024-09-13T11:26:54.597 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.597 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.597 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:26:54.597 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.178961+0000", 2024-09-13T11:26:54.597 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000104664 2024-09-13T11:26:54.597 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.597 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.597 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1", 2024-09-13T11:26:54.597 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.179123+0000", 2024-09-13T11:26:54.598 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000161738 2024-09-13T11:26:54.598 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.598 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.598 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:26:54.598 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.182755+0000", 2024-09-13T11:26:54.598 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0036323200000000001 2024-09-13T11:26:54.598 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.598 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.598 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:54.598 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.183023+0000", 2024-09-13T11:26:54.598 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00026749100000000002 2024-09-13T11:26:54.598 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.599 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.599 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:26:54.599 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.183168+0000", 2024-09-13T11:26:54.599 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00014499099999999999 2024-09-13T11:26:54.599 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.599 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.599 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:26:54.599 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.183285+0000", 2024-09-13T11:26:54.599 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000117119 2024-09-13T11:26:54.599 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:54.599 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:54.599 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:54.599 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.600 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.600 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:60 3.b 3:d67218b0:::benchmark_data_smithi177_31116_object59:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T11:26:54.600 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:48.314603+0000", 2024-09-13T11:26:54.600 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 246.236733577, 2024-09-13T11:26:54.600 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.87106201100000002, 2024-09-13T11:26:54.600 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:26:54.600 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:26:54.600 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:54.600 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:26:54.600 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:26:54.600 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:54.600 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 60 2024-09-13T11:26:54.600 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.601 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:26:54.601 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.601 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:26:54.601 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314603+0000", 2024-09-13T11:26:54.601 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:54.601 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.601 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.601 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:26:54.601 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314603+0000", 2024-09-13T11:26:54.601 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:54.601 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.602 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.602 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:26:54.602 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314604+0000", 2024-09-13T11:26:54.602 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.2359999999999999e-06 2024-09-13T11:26:54.602 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.602 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.602 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:26:54.602 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314667+0000", 2024-09-13T11:26:54.602 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.3077999999999995e-05 2024-09-13T11:26:54.602 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.602 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.602 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:26:54.602 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314699+0000", 2024-09-13T11:26:54.603 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.1965999999999999e-05 2024-09-13T11:26:54.603 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.603 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.603 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:26:54.603 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314710+0000", 2024-09-13T11:26:54.603 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.1148999999999999e-05 2024-09-13T11:26:54.603 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.603 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.603 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:54.603 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316291+0000", 2024-09-13T11:26:54.603 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0015804440000000001 2024-09-13T11:26:54.603 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.604 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.604 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-09-13T11:26:54.604 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316310+0000", 2024-09-13T11:26:54.604 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.9799000000000001e-05 2024-09-13T11:26:54.604 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.604 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.604 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:54.604 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.179555+0000", 2024-09-13T11:26:54.604 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.86324469500000001 2024-09-13T11:26:54.604 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.604 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.604 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:26:54.604 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.179666+0000", 2024-09-13T11:26:54.605 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000110443 2024-09-13T11:26:54.605 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.605 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.605 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1", 2024-09-13T11:26:54.605 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.179791+0000", 2024-09-13T11:26:54.605 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00012560499999999999 2024-09-13T11:26:54.605 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.605 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.605 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:26:54.605 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.182761+0000", 2024-09-13T11:26:54.605 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0029697019999999998 2024-09-13T11:26:54.605 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.606 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.606 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:54.606 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.185572+0000", 2024-09-13T11:26:54.606 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.002811512 2024-09-13T11:26:54.606 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.606 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.606 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:26:54.606 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.185625+0000", 2024-09-13T11:26:54.606 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.2716000000000001e-05 2024-09-13T11:26:54.606 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.606 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.606 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:26:54.607 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.185665+0000", 2024-09-13T11:26:54.607 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.9666000000000003e-05 2024-09-13T11:26:54.607 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:54.607 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:54.607 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:54.607 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.607 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.607 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:62 3.b 3:d20ae5a8:::benchmark_data_smithi177_31116_object61:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T11:26:54.607 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:48.314723+0000", 2024-09-13T11:26:54.607 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 246.236613035, 2024-09-13T11:26:54.607 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.87338985099999999, 2024-09-13T11:26:54.607 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:26:54.607 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:26:54.608 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:54.608 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:26:54.608 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:26:54.608 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:54.608 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 62 2024-09-13T11:26:54.608 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.608 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:26:54.608 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.608 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:26:54.608 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314723+0000", 2024-09-13T11:26:54.608 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:54.608 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.609 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.609 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:26:54.609 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314723+0000", 2024-09-13T11:26:54.609 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:54.609 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.609 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.609 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:26:54.609 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314725+0000", 2024-09-13T11:26:54.609 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.3069999999999999e-06 2024-09-13T11:26:54.609 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.609 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.609 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:26:54.609 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314797+0000", 2024-09-13T11:26:54.610 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.2652999999999997e-05 2024-09-13T11:26:54.610 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.610 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.610 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:26:54.610 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314809+0000", 2024-09-13T11:26:54.610 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.1496e-05 2024-09-13T11:26:54.610 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.610 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.610 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:26:54.610 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314816+0000", 2024-09-13T11:26:54.610 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.6639999999999998e-06 2024-09-13T11:26:54.610 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.610 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.610 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:54.610 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316424+0000", 2024-09-13T11:26:54.611 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0016077559999999999 2024-09-13T11:26:54.611 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.611 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.611 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-09-13T11:26:54.611 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316441+0000", 2024-09-13T11:26:54.611 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.7305000000000001e-05 2024-09-13T11:26:54.611 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.611 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.611 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:54.611 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.180688+0000", 2024-09-13T11:26:54.611 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.86424692800000003 2024-09-13T11:26:54.612 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.612 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.612 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:26:54.612 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.180807+0000", 2024-09-13T11:26:54.612 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000118977 2024-09-13T11:26:54.612 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.612 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.612 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1", 2024-09-13T11:26:54.612 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.180938+0000", 2024-09-13T11:26:54.612 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00013049700000000001 2024-09-13T11:26:54.612 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.613 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.613 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:26:54.613 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.185680+0000", 2024-09-13T11:26:54.613 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0047417029999999999 2024-09-13T11:26:54.613 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.613 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.613 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:54.613 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.187856+0000", 2024-09-13T11:26:54.613 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0021760550000000001 2024-09-13T11:26:54.613 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.613 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.613 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:26:54.614 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.188007+0000", 2024-09-13T11:26:54.614 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00015185099999999999 2024-09-13T11:26:54.614 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.614 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.614 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:26:54.614 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.188113+0000", 2024-09-13T11:26:54.614 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000105659 2024-09-13T11:26:54.614 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:54.614 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:54.614 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:54.614 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.615 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.615 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:73 3.b 3:d0060d5f:::benchmark_data_smithi177_31116_object72:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T11:26:54.615 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:48.314829+0000", 2024-09-13T11:26:54.615 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 246.23650768600001, 2024-09-13T11:26:54.615 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.87364757199999998, 2024-09-13T11:26:54.615 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:26:54.615 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:26:54.615 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:54.615 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:26:54.615 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:26:54.615 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:54.616 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 73 2024-09-13T11:26:54.616 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.616 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:26:54.616 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.616 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:26:54.616 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314829+0000", 2024-09-13T11:26:54.616 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:54.616 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.616 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.616 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:26:54.616 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314829+0000", 2024-09-13T11:26:54.617 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:54.617 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.617 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.617 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:26:54.617 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314830+0000", 2024-09-13T11:26:54.617 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.243e-06 2024-09-13T11:26:54.617 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.617 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.617 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:26:54.617 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314965+0000", 2024-09-13T11:26:54.618 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00013470800000000001 2024-09-13T11:26:54.618 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.618 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.618 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:26:54.618 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314979+0000", 2024-09-13T11:26:54.618 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.4375999999999999e-05 2024-09-13T11:26:54.618 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.618 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.618 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:26:54.618 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314988+0000", 2024-09-13T11:26:54.618 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.8570000000000001e-06 2024-09-13T11:26:54.619 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.619 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.619 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:54.619 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316545+0000", 2024-09-13T11:26:54.619 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.001556878 2024-09-13T11:26:54.619 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.619 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.619 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-09-13T11:26:54.619 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316556+0000", 2024-09-13T11:26:54.619 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.1453e-05 2024-09-13T11:26:54.619 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.619 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.620 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:54.620 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.182124+0000", 2024-09-13T11:26:54.620 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.86556744200000002 2024-09-13T11:26:54.620 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.620 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.620 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:26:54.620 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.182246+0000", 2024-09-13T11:26:54.620 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00012210500000000001 2024-09-13T11:26:54.620 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.620 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.620 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1", 2024-09-13T11:26:54.620 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.182375+0000", 2024-09-13T11:26:54.620 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00012920999999999999 2024-09-13T11:26:54.621 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.621 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.621 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:26:54.621 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.185684+0000", 2024-09-13T11:26:54.621 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0033093969999999999 2024-09-13T11:26:54.621 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.621 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.621 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:54.621 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.188244+0000", 2024-09-13T11:26:54.621 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0025601550000000002 2024-09-13T11:26:54.621 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.622 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.622 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:26:54.622 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.188379+0000", 2024-09-13T11:26:54.622 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00013463400000000001 2024-09-13T11:26:54.622 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.622 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.622 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:26:54.622 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.188476+0000", 2024-09-13T11:26:54.622 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.7114000000000006e-05 2024-09-13T11:26:54.622 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:54.622 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:54.622 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:54.623 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.623 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.623 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:81 3.b 3:d705fd70:::benchmark_data_smithi177_31116_object80:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T11:26:54.623 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:48.315000+0000", 2024-09-13T11:26:54.623 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 246.236335849, 2024-09-13T11:26:54.623 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.87561418300000005, 2024-09-13T11:26:54.623 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:26:54.623 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:26:54.623 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:54.623 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:26:54.623 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:26:54.624 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:54.624 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 81 2024-09-13T11:26:54.624 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.624 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:26:54.624 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.624 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:26:54.624 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315000+0000", 2024-09-13T11:26:54.624 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:54.624 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.624 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.624 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:26:54.624 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315000+0000", 2024-09-13T11:26:54.625 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:54.625 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.625 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.625 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:26:54.625 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315002+0000", 2024-09-13T11:26:54.625 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.3799999999999999e-06 2024-09-13T11:26:54.625 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.625 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.625 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:26:54.625 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315119+0000", 2024-09-13T11:26:54.625 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000116925 2024-09-13T11:26:54.625 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.626 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.626 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:26:54.626 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315133+0000", 2024-09-13T11:26:54.626 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.3971999999999999e-05 2024-09-13T11:26:54.626 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.626 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.626 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:26:54.626 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315142+0000", 2024-09-13T11:26:54.626 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.4399999999999994e-06 2024-09-13T11:26:54.626 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.626 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.626 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:54.626 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316652+0000", 2024-09-13T11:26:54.627 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.001509907 2024-09-13T11:26:54.627 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.627 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.627 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-09-13T11:26:54.627 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316668+0000", 2024-09-13T11:26:54.627 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.5945999999999999e-05 2024-09-13T11:26:54.627 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.627 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.627 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:54.627 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.183405+0000", 2024-09-13T11:26:54.627 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.866737384 2024-09-13T11:26:54.627 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.628 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.628 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:26:54.628 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.183701+0000", 2024-09-13T11:26:54.628 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00029511099999999997 2024-09-13T11:26:54.628 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.628 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.628 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1", 2024-09-13T11:26:54.628 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.184078+0000", 2024-09-13T11:26:54.628 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00037741000000000001 2024-09-13T11:26:54.628 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.628 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.628 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:26:54.629 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.188504+0000", 2024-09-13T11:26:54.629 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0044260280000000003 2024-09-13T11:26:54.629 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.629 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.629 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:54.629 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.190562+0000", 2024-09-13T11:26:54.629 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0020580099999999999 2024-09-13T11:26:54.629 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.629 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.629 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:26:54.629 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.190592+0000", 2024-09-13T11:26:54.630 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.0057999999999999e-05 2024-09-13T11:26:54.630 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.630 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.630 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:26:54.630 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.190615+0000", 2024-09-13T11:26:54.630 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.2612e-05 2024-09-13T11:26:54.630 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:54.630 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:54.630 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:54.630 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.630 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.630 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:87 3.b 3:dd7de0ad:::benchmark_data_smithi177_31116_object86:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T11:26:54.630 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:48.315155+0000", 2024-09-13T11:26:54.631 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 246.23618105200001, 2024-09-13T11:26:54.631 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.87555916099999997, 2024-09-13T11:26:54.631 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:26:54.631 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:26:54.631 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:54.631 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:26:54.631 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:26:54.631 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:54.631 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 87 2024-09-13T11:26:54.631 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.631 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:26:54.631 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.631 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:26:54.632 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315155+0000", 2024-09-13T11:26:54.632 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:54.632 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.632 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.632 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:26:54.632 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315155+0000", 2024-09-13T11:26:54.632 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:54.632 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.632 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.632 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:26:54.632 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315157+0000", 2024-09-13T11:26:54.633 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.3829999999999999e-06 2024-09-13T11:26:54.633 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.633 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.633 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:26:54.633 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315271+0000", 2024-09-13T11:26:54.633 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000114539 2024-09-13T11:26:54.633 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.633 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.633 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:26:54.633 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315285+0000", 2024-09-13T11:26:54.633 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.4042e-05 2024-09-13T11:26:54.633 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.634 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.634 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:26:54.634 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315295+0000", 2024-09-13T11:26:54.634 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.2739999999999996e-06 2024-09-13T11:26:54.634 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.634 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.634 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:54.634 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316739+0000", 2024-09-13T11:26:54.634 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0014446960000000001 2024-09-13T11:26:54.634 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.634 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.634 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-09-13T11:26:54.634 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316747+0000", 2024-09-13T11:26:54.635 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.0790000000000001e-06 2024-09-13T11:26:54.635 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.635 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.635 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:54.635 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.184922+0000", 2024-09-13T11:26:54.635 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.86817446399999998 2024-09-13T11:26:54.635 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.635 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.635 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:26:54.635 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.185037+0000", 2024-09-13T11:26:54.635 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00011565099999999999 2024-09-13T11:26:54.635 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.635 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.636 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1", 2024-09-13T11:26:54.636 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.185171+0000", 2024-09-13T11:26:54.636 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00013316900000000001 2024-09-13T11:26:54.636 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.636 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.636 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:26:54.636 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.188520+0000", 2024-09-13T11:26:54.636 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0033489150000000001 2024-09-13T11:26:54.636 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.636 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.636 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:54.636 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.190652+0000", 2024-09-13T11:26:54.637 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0021325649999999999 2024-09-13T11:26:54.637 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.637 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.637 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:26:54.637 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.190686+0000", 2024-09-13T11:26:54.637 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.3827e-05 2024-09-13T11:26:54.637 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.637 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.637 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:26:54.637 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.190714+0000", 2024-09-13T11:26:54.637 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.8557000000000001e-05 2024-09-13T11:26:54.638 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:54.638 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:54.638 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:54.638 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.638 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.638 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:99 3.b 3:d011b600:::benchmark_data_smithi177_31116_object98:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T11:26:54.638 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:48.315307+0000", 2024-09-13T11:26:54.638 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 246.23602922500001, 2024-09-13T11:26:54.638 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.87551770799999995, 2024-09-13T11:26:54.638 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:26:54.638 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:26:54.638 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:54.639 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:26:54.639 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:26:54.639 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:54.639 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 99 2024-09-13T11:26:54.639 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.639 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:26:54.639 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.639 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:26:54.639 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315307+0000", 2024-09-13T11:26:54.639 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:54.639 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.639 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.640 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:26:54.640 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315307+0000", 2024-09-13T11:26:54.640 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:54.640 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.640 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.640 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:26:54.640 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315308+0000", 2024-09-13T11:26:54.640 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.316e-06 2024-09-13T11:26:54.640 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.640 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.640 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:26:54.640 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315453+0000", 2024-09-13T11:26:54.641 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00014473200000000001 2024-09-13T11:26:54.641 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.641 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.641 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:26:54.641 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315471+0000", 2024-09-13T11:26:54.641 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.8019000000000001e-05 2024-09-13T11:26:54.641 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.641 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.641 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:26:54.641 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315489+0000", 2024-09-13T11:26:54.641 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.7861999999999999e-05 2024-09-13T11:26:54.642 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.642 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.642 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:54.642 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316837+0000", 2024-09-13T11:26:54.642 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.001347787 2024-09-13T11:26:54.642 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.642 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.642 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-09-13T11:26:54.642 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316858+0000", 2024-09-13T11:26:54.642 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.1103000000000001e-05 2024-09-13T11:26:54.642 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.643 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.643 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:54.643 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.185713+0000", 2024-09-13T11:26:54.643 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.86885540800000005 2024-09-13T11:26:54.643 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.643 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.643 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:26:54.643 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.185820+0000", 2024-09-13T11:26:54.643 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000107081 2024-09-13T11:26:54.643 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.643 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.643 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1", 2024-09-13T11:26:54.644 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.185948+0000", 2024-09-13T11:26:54.644 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00012775800000000001 2024-09-13T11:26:54.644 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.644 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.644 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:26:54.644 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.190527+0000", 2024-09-13T11:26:54.644 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0045783910000000002 2024-09-13T11:26:54.644 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.644 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.644 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:54.644 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.190759+0000", 2024-09-13T11:26:54.645 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00023266199999999999 2024-09-13T11:26:54.645 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.645 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.645 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:26:54.645 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.190789+0000", 2024-09-13T11:26:54.645 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.9666999999999999e-05 2024-09-13T11:26:54.645 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.645 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.645 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:26:54.645 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.190825+0000", 2024-09-13T11:26:54.645 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.5921999999999999e-05 2024-09-13T11:26:54.645 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:54.646 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:54.646 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:54.646 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.646 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.646 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:108 3.b 3:d5c90217:::benchmark_data_smithi177_31116_object107:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T11:26:54.646 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:48.315513+0000", 2024-09-13T11:26:54.646 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 246.23582352, 2024-09-13T11:26:54.646 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.87544772500000001, 2024-09-13T11:26:54.646 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:26:54.646 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:26:54.646 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:54.646 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:26:54.647 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:26:54.647 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:54.647 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 108 2024-09-13T11:26:54.647 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.647 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:26:54.647 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.647 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:26:54.647 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315513+0000", 2024-09-13T11:26:54.647 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:54.647 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.647 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.647 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:26:54.648 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315513+0000", 2024-09-13T11:26:54.648 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:54.648 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.648 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.648 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:26:54.648 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315515+0000", 2024-09-13T11:26:54.648 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.9369999999999998e-06 2024-09-13T11:26:54.648 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.648 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.648 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:26:54.648 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315644+0000", 2024-09-13T11:26:54.649 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000129011 2024-09-13T11:26:54.649 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.649 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.649 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:26:54.649 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315657+0000", 2024-09-13T11:26:54.649 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.3689999999999999e-05 2024-09-13T11:26:54.649 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.649 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.649 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:26:54.649 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315668+0000", 2024-09-13T11:26:54.649 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0589e-05 2024-09-13T11:26:54.650 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.650 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.650 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:54.650 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316925+0000", 2024-09-13T11:26:54.650 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0012568340000000001 2024-09-13T11:26:54.650 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.650 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.650 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-09-13T11:26:54.650 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316932+0000", 2024-09-13T11:26:54.650 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.2230000000000001e-06 2024-09-13T11:26:54.650 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.650 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.651 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:54.651 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.186619+0000", 2024-09-13T11:26:54.651 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.86968679599999998 2024-09-13T11:26:54.651 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.651 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.651 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:26:54.651 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.186914+0000", 2024-09-13T11:26:54.651 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00029528900000000001 2024-09-13T11:26:54.651 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.651 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.651 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1", 2024-09-13T11:26:54.651 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.187217+0000", 2024-09-13T11:26:54.652 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00030232099999999998 2024-09-13T11:26:54.652 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.652 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.652 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:26:54.652 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.190532+0000", 2024-09-13T11:26:54.652 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.003315391 2024-09-13T11:26:54.652 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.652 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.652 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:54.652 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.190875+0000", 2024-09-13T11:26:54.652 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00034326300000000001 2024-09-13T11:26:54.652 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.652 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.653 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:26:54.653 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.190925+0000", 2024-09-13T11:26:54.653 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.9429999999999999e-05 2024-09-13T11:26:54.653 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.653 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.653 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:26:54.653 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.190961+0000", 2024-09-13T11:26:54.653 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.5951000000000002e-05 2024-09-13T11:26:54.653 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:54.653 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:54.653 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:54.654 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.654 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.654 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:172 3.b 3:dbec1eeb:::benchmark_data_smithi177_31116_object171:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T11:26:54.654 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:48.315684+0000", 2024-09-13T11:26:54.654 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 246.23565236300001, 2024-09-13T11:26:54.654 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.87615312999999995, 2024-09-13T11:26:54.654 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:26:54.654 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:26:54.654 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:54.654 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:26:54.654 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:26:54.654 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:54.655 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 172 2024-09-13T11:26:54.655 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.655 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:26:54.655 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.655 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:26:54.655 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315684+0000", 2024-09-13T11:26:54.655 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:54.655 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.655 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.655 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:26:54.655 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315684+0000", 2024-09-13T11:26:54.655 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:54.656 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.656 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.656 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:26:54.656 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315686+0000", 2024-09-13T11:26:54.656 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.886e-06 2024-09-13T11:26:54.656 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.656 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.656 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:26:54.656 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315820+0000", 2024-09-13T11:26:54.656 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00013388500000000001 2024-09-13T11:26:54.656 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.657 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.657 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:26:54.657 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315837+0000", 2024-09-13T11:26:54.657 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.6756e-05 2024-09-13T11:26:54.657 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.657 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.657 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:26:54.657 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315851+0000", 2024-09-13T11:26:54.657 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.4566000000000001e-05 2024-09-13T11:26:54.657 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.657 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.658 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:54.658 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.317021+0000", 2024-09-13T11:26:54.658 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0011701769999999999 2024-09-13T11:26:54.658 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.658 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.658 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-09-13T11:26:54.658 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.317037+0000", 2024-09-13T11:26:54.658 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.609e-05 2024-09-13T11:26:54.658 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.658 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.658 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:54.658 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.188634+0000", 2024-09-13T11:26:54.659 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.87159711900000003 2024-09-13T11:26:54.659 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.659 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.659 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:26:54.659 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.188974+0000", 2024-09-13T11:26:54.659 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00033922299999999999 2024-09-13T11:26:54.659 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.659 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.659 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1", 2024-09-13T11:26:54.659 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.189510+0000", 2024-09-13T11:26:54.659 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00053618999999999997 2024-09-13T11:26:54.659 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.660 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.661 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:54.661 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.191027+0000", 2024-09-13T11:26:54.661 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0015168390000000001 2024-09-13T11:26:54.662 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.662 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.662 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:26:54.662 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.191766+0000", 2024-09-13T11:26:54.662 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000739064 2024-09-13T11:26:54.662 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.662 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.662 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:26:54.662 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.191810+0000", 2024-09-13T11:26:54.662 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.3755999999999999e-05 2024-09-13T11:26:54.662 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.662 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.663 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:26:54.663 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.191837+0000", 2024-09-13T11:26:54.663 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.7579000000000001e-05 2024-09-13T11:26:54.663 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:54.663 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:54.663 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:54.663 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.663 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.663 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:177 3.b 3:de89c779:::benchmark_data_smithi177_31116_object176:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T11:26:54.663 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:48.315898+0000", 2024-09-13T11:26:54.663 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 246.23543804799999, 2024-09-13T11:26:54.663 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.87600712599999997, 2024-09-13T11:26:54.664 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:26:54.664 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:26:54.664 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:54.664 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:26:54.664 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:26:54.664 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:54.664 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 177 2024-09-13T11:26:54.664 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.664 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:26:54.664 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.664 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:26:54.665 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315898+0000", 2024-09-13T11:26:54.665 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:54.665 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.665 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.665 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:26:54.665 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315898+0000", 2024-09-13T11:26:54.665 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:54.665 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.665 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.665 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:26:54.665 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315900+0000", 2024-09-13T11:26:54.665 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.4929999999999999e-06 2024-09-13T11:26:54.665 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.666 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.666 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:26:54.666 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316027+0000", 2024-09-13T11:26:54.666 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000127568 2024-09-13T11:26:54.666 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.666 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.666 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:26:54.666 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316048+0000", 2024-09-13T11:26:54.666 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.0795999999999999e-05 2024-09-13T11:26:54.666 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.666 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.667 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:26:54.667 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316061+0000", 2024-09-13T11:26:54.667 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.3283e-05 2024-09-13T11:26:54.667 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.667 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.667 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:54.667 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.317107+0000", 2024-09-13T11:26:54.667 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0010452949999999999 2024-09-13T11:26:54.667 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.667 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.667 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-09-13T11:26:54.668 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.317114+0000", 2024-09-13T11:26:54.668 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.4569999999999999e-06 2024-09-13T11:26:54.668 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.668 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.668 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:54.668 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.190017+0000", 2024-09-13T11:26:54.668 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.87290258499999995 2024-09-13T11:26:54.668 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.668 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.668 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:26:54.668 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.190138+0000", 2024-09-13T11:26:54.668 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000121418 2024-09-13T11:26:54.668 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.669 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.669 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1", 2024-09-13T11:26:54.669 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.190216+0000", 2024-09-13T11:26:54.669 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.7410999999999995e-05 2024-09-13T11:26:54.669 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.669 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.669 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:54.669 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.191755+0000", 2024-09-13T11:26:54.669 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0015395420000000001 2024-09-13T11:26:54.669 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.669 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.669 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:26:54.669 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.191843+0000", 2024-09-13T11:26:54.670 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.8022999999999994e-05 2024-09-13T11:26:54.670 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.670 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.670 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:26:54.670 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.191876+0000", 2024-09-13T11:26:54.670 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.2619000000000001e-05 2024-09-13T11:26:54.670 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.670 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.670 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:26:54.670 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.191905+0000", 2024-09-13T11:26:54.670 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.9635999999999999e-05 2024-09-13T11:26:54.670 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:54.671 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:54.671 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:54.671 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.671 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.671 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:187 3.b 3:da03c8dd:::benchmark_data_smithi177_31116_object186:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T11:26:54.671 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:48.316083+0000", 2024-09-13T11:26:54.671 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 246.23525309199999, 2024-09-13T11:26:54.671 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.87835499500000003, 2024-09-13T11:26:54.671 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:26:54.671 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:26:54.671 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:54.671 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:26:54.672 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:26:54.672 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:54.672 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 187 2024-09-13T11:26:54.672 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.672 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:26:54.672 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.672 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:26:54.672 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316083+0000", 2024-09-13T11:26:54.672 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:54.673 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.673 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.673 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:26:54.673 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316083+0000", 2024-09-13T11:26:54.673 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:54.673 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.673 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.673 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:26:54.673 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316086+0000", 2024-09-13T11:26:54.673 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.5560000000000002e-06 2024-09-13T11:26:54.673 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.674 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.674 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:26:54.674 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316199+0000", 2024-09-13T11:26:54.674 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000113522 2024-09-13T11:26:54.674 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.674 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.674 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:26:54.674 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316215+0000", 2024-09-13T11:26:54.674 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.5265999999999999e-05 2024-09-13T11:26:54.674 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.674 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.674 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:26:54.675 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316226+0000", 2024-09-13T11:26:54.675 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0967e-05 2024-09-13T11:26:54.675 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.675 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.675 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:54.675 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.317198+0000", 2024-09-13T11:26:54.675 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00097194100000000004 2024-09-13T11:26:54.675 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.675 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.675 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-09-13T11:26:54.675 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.317213+0000", 2024-09-13T11:26:54.675 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.5472999999999999e-05 2024-09-13T11:26:54.676 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.676 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.676 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:54.676 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.191077+0000", 2024-09-13T11:26:54.676 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.87386435200000001 2024-09-13T11:26:54.676 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.676 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.676 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:26:54.676 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.191181+0000", 2024-09-13T11:26:54.676 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000103618 2024-09-13T11:26:54.676 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.677 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.677 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1", 2024-09-13T11:26:54.677 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.191305+0000", 2024-09-13T11:26:54.677 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00012358599999999999 2024-09-13T11:26:54.677 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.677 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.677 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:54.677 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.194279+0000", 2024-09-13T11:26:54.677 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0029746970000000001 2024-09-13T11:26:54.677 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.677 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.678 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:26:54.678 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.194336+0000", 2024-09-13T11:26:54.678 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.7244000000000003e-05 2024-09-13T11:26:54.678 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.678 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.678 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:26:54.678 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.194397+0000", 2024-09-13T11:26:54.678 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.0278000000000001e-05 2024-09-13T11:26:54.678 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.678 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.678 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:26:54.678 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.194438+0000", 2024-09-13T11:26:54.679 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.1495000000000002e-05 2024-09-13T11:26:54.679 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:54.679 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:54.679 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:54.679 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.679 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.679 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:200 3.b 3:d90039f1:::benchmark_data_smithi177_31116_object199:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T11:26:54.679 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:48.316280+0000", 2024-09-13T11:26:54.679 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 246.23505662700001, 2024-09-13T11:26:54.679 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.87879273000000002, 2024-09-13T11:26:54.679 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:26:54.679 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:26:54.680 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:54.680 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:26:54.680 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:26:54.680 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:54.680 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 200 2024-09-13T11:26:54.680 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.680 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:26:54.680 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.680 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:26:54.680 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316280+0000", 2024-09-13T11:26:54.680 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:54.681 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.681 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.681 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:26:54.681 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316280+0000", 2024-09-13T11:26:54.681 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:54.681 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.681 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.681 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:26:54.681 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316282+0000", 2024-09-13T11:26:54.681 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.497e-06 2024-09-13T11:26:54.681 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.681 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.682 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:26:54.682 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316411+0000", 2024-09-13T11:26:54.682 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00012924799999999999 2024-09-13T11:26:54.682 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.682 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.682 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:26:54.682 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316426+0000", 2024-09-13T11:26:54.682 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.4183e-05 2024-09-13T11:26:54.682 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.682 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.682 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:26:54.682 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316435+0000", 2024-09-13T11:26:54.683 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.0990000000000005e-06 2024-09-13T11:26:54.683 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.683 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.683 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:54.683 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.317250+0000", 2024-09-13T11:26:54.683 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00081501100000000003 2024-09-13T11:26:54.683 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.683 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.683 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-09-13T11:26:54.683 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.317257+0000", 2024-09-13T11:26:54.683 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.2169999999999997e-06 2024-09-13T11:26:54.683 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.684 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.684 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:54.684 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.191936+0000", 2024-09-13T11:26:54.684 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.87467922099999995 2024-09-13T11:26:54.684 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.684 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.684 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:26:54.684 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.192056+0000", 2024-09-13T11:26:54.684 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000119638 2024-09-13T11:26:54.684 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.684 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.684 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1", 2024-09-13T11:26:54.684 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.192184+0000", 2024-09-13T11:26:54.685 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00012813399999999999 2024-09-13T11:26:54.685 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.685 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.685 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:54.685 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.194324+0000", 2024-09-13T11:26:54.685 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0021399420000000001 2024-09-13T11:26:54.685 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.685 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.685 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:26:54.685 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.194802+0000", 2024-09-13T11:26:54.685 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00047798799999999999 2024-09-13T11:26:54.685 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.686 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.686 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:26:54.686 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.194951+0000", 2024-09-13T11:26:54.686 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0001494 2024-09-13T11:26:54.686 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.686 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.686 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:26:54.686 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.195072+0000", 2024-09-13T11:26:54.686 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00012115199999999999 2024-09-13T11:26:54.686 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:54.686 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:54.686 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:54.686 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.686 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.687 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop(client.4665.0:79801 3.5 e102/67 3:a6efdf84:::benchmark_data_smithi177_31116_object79800:head v 102'3851, mlcod=102'3848)", 2024-09-13T11:26:54.687 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:24:00.403754+0000", 2024-09-13T11:26:54.687 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 174.14758227499999, 2024-09-13T11:26:54.687 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.31749079899999999, 2024-09-13T11:26:54.687 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:26:54.687 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:26:54.687 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:54.687 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:26:54.687 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.687 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:26:54.687 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:00.403754+0000", 2024-09-13T11:26:54.687 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:54.687 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.688 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.688 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:26:54.688 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:00.403754+0000", 2024-09-13T11:26:54.688 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:54.688 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.688 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.688 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:26:54.688 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:00.403758+0000", 2024-09-13T11:26:54.688 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.507e-06 2024-09-13T11:26:54.688 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.688 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.688 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:26:54.688 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:00.403819+0000", 2024-09-13T11:26:54.688 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.1042000000000003e-05 2024-09-13T11:26:54.688 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.689 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.689 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:26:54.689 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:00.717963+0000", 2024-09-13T11:26:54.689 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.314144594 2024-09-13T11:26:54.689 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.689 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.689 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:26:54.689 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:00.717976+0000", 2024-09-13T11:26:54.689 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.3029e-05 2024-09-13T11:26:54.689 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.689 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.689 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:54.689 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:00.718143+0000", 2024-09-13T11:26:54.689 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00016639000000000001 2024-09-13T11:26:54.690 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.690 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.690 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:26:54.690 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:00.718184+0000", 2024-09-13T11:26:54.690 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.1801999999999997e-05 2024-09-13T11:26:54.690 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.690 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.690 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:26:54.690 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:00.721162+0000", 2024-09-13T11:26:54.690 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0029772800000000001 2024-09-13T11:26:54.690 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.690 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.690 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:26:54.690 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:00.721245+0000", 2024-09-13T11:26:54.690 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.3154999999999999e-05 2024-09-13T11:26:54.691 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:54.691 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:54.691 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:54.691 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.691 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.691 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "MOSDPGPushReply(3.26 121/121 [PushReplyOp(3:670b9aab:::benchmark_data_smithi177_31116_object93:head)])", 2024-09-13T11:26:54.691 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:24:27.300318+0000", 2024-09-13T11:26:54.691 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 147.251018407, 2024-09-13T11:26:54.691 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.811436728, 2024-09-13T11:26:54.691 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:26:54.691 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:26:54.691 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "reached pg", 2024-09-13T11:26:54.691 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:26:54.691 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.692 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:26:54.692 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:27.300318+0000", 2024-09-13T11:26:54.692 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:54.692 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.692 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.692 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:26:54.692 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:27.300318+0000", 2024-09-13T11:26:54.692 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:54.692 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.692 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.692 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:26:54.692 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:27.300320+0000", 2024-09-13T11:26:54.692 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.778e-06 2024-09-13T11:26:54.692 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.693 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.693 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:26:54.693 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:27.300326+0000", 2024-09-13T11:26:54.693 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.7700000000000004e-06 2024-09-13T11:26:54.693 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.693 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.693 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:26:54.693 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:28.109665+0000", 2024-09-13T11:26:54.693 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.80933853499999997 2024-09-13T11:26:54.693 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.693 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.693 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:26:54.693 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:28.109678+0000", 2024-09-13T11:26:54.693 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.2870999999999999e-05 2024-09-13T11:26:54.693 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.694 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.694 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:54.694 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:28.109800+0000", 2024-09-13T11:26:54.694 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000122328 2024-09-13T11:26:54.694 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.694 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.694 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:26:54.694 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:28.111755+0000", 2024-09-13T11:26:54.694 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0019544459999999999 2024-09-13T11:26:54.694 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:54.694 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:54.694 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:54.694 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.695 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.695 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:184026 3.b 3:d3052347:::benchmark_data_smithi177_31116_object43723:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e171)", 2024-09-13T11:26:54.695 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:25:32.645242+0000", 2024-09-13T11:26:54.695 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 81.906094366999994, 2024-09-13T11:26:54.695 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.348925019, 2024-09-13T11:26:54.695 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:26:54.695 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:26:54.695 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:54.695 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:26:54.695 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:26:54.695 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:54.695 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 184026 2024-09-13T11:26:54.695 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.695 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:26:54.696 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.696 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:26:54.696 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:32.645242+0000", 2024-09-13T11:26:54.696 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:54.696 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.696 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.696 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:26:54.696 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:32.645242+0000", 2024-09-13T11:26:54.696 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:54.696 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.696 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.696 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:26:54.696 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:32.645244+0000", 2024-09-13T11:26:54.697 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.368e-06 2024-09-13T11:26:54.697 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.697 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.697 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:26:54.697 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:32.645251+0000", 2024-09-13T11:26:54.697 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.2519999999999996e-06 2024-09-13T11:26:54.697 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.697 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.697 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:26:54.697 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:32.645267+0000", 2024-09-13T11:26:54.697 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.6529e-05 2024-09-13T11:26:54.697 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.697 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.698 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:26:54.698 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:32.645278+0000", 2024-09-13T11:26:54.698 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.119e-05 2024-09-13T11:26:54.698 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.698 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.698 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:54.698 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:32.645348+0000", 2024-09-13T11:26:54.698 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.9333999999999997e-05 2024-09-13T11:26:54.698 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.698 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.698 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:26:54.698 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:32.645479+0000", 2024-09-13T11:26:54.698 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00013153199999999999 2024-09-13T11:26:54.698 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.699 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.699 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1", 2024-09-13T11:26:54.699 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:32.645599+0000", 2024-09-13T11:26:54.699 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000119481 2024-09-13T11:26:54.699 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.699 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.699 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:26:54.699 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:32.648042+0000", 2024-09-13T11:26:54.699 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0024437059999999999 2024-09-13T11:26:54.699 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.699 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.699 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:54.699 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:32.994082+0000", 2024-09-13T11:26:54.700 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.34604006399999998 2024-09-13T11:26:54.700 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.700 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.700 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:26:54.700 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:32.994124+0000", 2024-09-13T11:26:54.700 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.1573999999999999e-05 2024-09-13T11:26:54.700 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:54.700 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:54.700 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:26:54.700 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:32.994167+0000", 2024-09-13T11:26:54.700 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.2988999999999998e-05 2024-09-13T11:26:54.700 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:54.700 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:54.700 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:54.701 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:54.701 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:54.701 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-09-13T11:26:54.702 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-13T11:26:54.703 INFO:tasks.ceph.osd.9.smithi092.stderr:2024-09-13T11:26:54.699+0000 7ff5ab3d0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26348) UID: 0 2024-09-13T11:26:54.803 INFO:tasks.ceph.osd.4.smithi001.stderr:2024-09-13T11:26:54.798+0000 7ff8e0218640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26310) UID: 0 2024-09-13T11:26:54.844 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-13T11:26:54.844 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [], 2024-09-13T11:26:54.844 INFO:teuthology.orchestra.run.smithi162.stdout: "num_ops": 0 2024-09-13T11:26:54.844 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-13T11:26:54.864 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-13T11:26:54.904 INFO:tasks.ceph.osd.1.smithi092.stderr:2024-09-13T11:26:54.899+0000 7fe7789c5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33718) UID: 0 2024-09-13T11:26:55.002 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-13T11:26:55.002 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [], 2024-09-13T11:26:55.002 INFO:teuthology.orchestra.run.smithi162.stdout: "complaint_time": 30, 2024-09-13T11:26:55.002 INFO:teuthology.orchestra.run.smithi162.stdout: "num_blocked_ops": 0 2024-09-13T11:26:55.002 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-13T11:26:55.005 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-13T11:26:55.005 INFO:tasks.ceph.osd.3.smithi177.stderr:2024-09-13T11:26:55.001+0000 7f10adba4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26574) UID: 0 2024-09-13T11:26:55.022 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-13T11:26:55.106 INFO:tasks.ceph.osd.13.smithi092.stderr:2024-09-13T11:26:55.103+0000 7f3e5afea640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26350) UID: 0 2024-09-13T11:26:55.161 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-13T11:26:55.161 INFO:teuthology.orchestra.run.smithi162.stdout: "size": 20, 2024-09-13T11:26:55.161 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 600, 2024-09-13T11:26:55.161 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [ 2024-09-13T11:26:55.161 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.162 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.4665.0:171 3.9 e41/30 3:9e1c7eeb:::benchmark_data_smithi177_31116_object170:head v 41'13, mlcod=41'9)", 2024-09-13T11:26:55.162 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:22:47.418602+0000", 2024-09-13T11:26:55.162 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 247.74131601600001, 2024-09-13T11:26:55.162 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.67235367300000004, 2024-09-13T11:26:55.162 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:26:55.162 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:26:55.162 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:55.162 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:26:55.162 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.162 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:26:55.162 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:47.418602+0000", 2024-09-13T11:26:55.162 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:55.163 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.163 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.163 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:26:55.163 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:47.418602+0000", 2024-09-13T11:26:55.163 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:55.163 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.163 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.163 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:26:55.163 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:47.418605+0000", 2024-09-13T11:26:55.163 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.2200000000000001e-06 2024-09-13T11:26:55.163 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.164 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.164 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:26:55.164 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:47.418668+0000", 2024-09-13T11:26:55.164 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.2921000000000003e-05 2024-09-13T11:26:55.164 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.164 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.164 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:26:55.164 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:48.089380+0000", 2024-09-13T11:26:55.164 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.67071191100000005 2024-09-13T11:26:55.164 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.164 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.165 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:26:55.165 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:48.089423+0000", 2024-09-13T11:26:55.165 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.2916e-05 2024-09-13T11:26:55.165 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.165 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.165 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:26:55.165 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:48.089533+0000", 2024-09-13T11:26:55.165 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000109996 2024-09-13T11:26:55.165 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.165 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.165 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:26:55.165 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:48.089570+0000", 2024-09-13T11:26:55.166 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.7314999999999998e-05 2024-09-13T11:26:55.166 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.166 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.166 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:26:55.166 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:48.090852+0000", 2024-09-13T11:26:55.166 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001281641 2024-09-13T11:26:55.166 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.166 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.166 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:26:55.166 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:48.090956+0000", 2024-09-13T11:26:55.166 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00010375299999999999 2024-09-13T11:26:55.166 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:55.167 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:26:55.167 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:55.167 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.167 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.167 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPushReply(3.19 74/70 [PushReplyOp(3:9edf939c:::benchmark_data_smithi177_31116_object17207:head)])", 2024-09-13T11:26:55.167 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:30.897971+0000", 2024-09-13T11:26:55.167 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 204.26194716500001, 2024-09-13T11:26:55.167 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.40616071100000001, 2024-09-13T11:26:55.167 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:26:55.167 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:26:55.167 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-09-13T11:26:55.167 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:26:55.168 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.168 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:26:55.168 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.897971+0000", 2024-09-13T11:26:55.168 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:55.168 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.168 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.168 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:26:55.168 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.897971+0000", 2024-09-13T11:26:55.168 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:55.168 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.168 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.168 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:26:55.169 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.897972+0000", 2024-09-13T11:26:55.169 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1429999999999999e-06 2024-09-13T11:26:55.169 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.169 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.169 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:26:55.169 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.897977+0000", 2024-09-13T11:26:55.169 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.5199999999999999e-06 2024-09-13T11:26:55.169 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.169 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.169 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:26:55.169 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:31.302500+0000", 2024-09-13T11:26:55.170 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.40452312400000001 2024-09-13T11:26:55.170 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.170 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.170 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:26:55.170 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:31.302513+0000", 2024-09-13T11:26:55.170 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2753e-05 2024-09-13T11:26:55.170 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.170 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.170 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:26:55.170 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:31.302645+0000", 2024-09-13T11:26:55.170 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00013292100000000001 2024-09-13T11:26:55.170 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.171 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.171 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:26:55.171 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:31.304132+0000", 2024-09-13T11:26:55.171 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0014862499999999999 2024-09-13T11:26:55.171 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:55.171 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:26:55.171 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:55.171 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.171 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.171 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPushReply(3.19 75/70 [PushReplyOp(3:9d1489b1:::benchmark_data_smithi177_31116_object18698:head)])", 2024-09-13T11:26:55.171 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:31.697212+0000", 2024-09-13T11:26:55.171 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 203.462706164, 2024-09-13T11:26:55.172 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.22850633200000001, 2024-09-13T11:26:55.172 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:26:55.172 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:26:55.172 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-09-13T11:26:55.172 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:26:55.172 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.172 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:26:55.172 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:31.697212+0000", 2024-09-13T11:26:55.172 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:55.172 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.172 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.172 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:26:55.173 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:31.697212+0000", 2024-09-13T11:26:55.173 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:55.173 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.173 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.173 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:26:55.173 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:31.697214+0000", 2024-09-13T11:26:55.173 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.7439999999999999e-06 2024-09-13T11:26:55.173 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.173 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.173 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:26:55.173 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:31.697224+0000", 2024-09-13T11:26:55.174 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0173000000000001e-05 2024-09-13T11:26:55.174 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.174 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.174 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:26:55.174 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:31.923931+0000", 2024-09-13T11:26:55.174 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.226706926 2024-09-13T11:26:55.174 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.174 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.174 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:26:55.174 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:31.923950+0000", 2024-09-13T11:26:55.174 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.8899e-05 2024-09-13T11:26:55.175 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.175 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.175 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:26:55.175 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:31.924449+0000", 2024-09-13T11:26:55.175 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00049907800000000002 2024-09-13T11:26:55.175 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.175 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.175 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:26:55.175 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:31.925718+0000", 2024-09-13T11:26:55.175 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001269512 2024-09-13T11:26:55.175 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:55.175 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:26:55.175 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:55.176 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.176 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.176 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPushReply(3.19 76/70 [PushReplyOp(3:9faf6d5f:::benchmark_data_smithi177_31116_object18954:head)])", 2024-09-13T11:26:55.176 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:32.042874+0000", 2024-09-13T11:26:55.176 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 203.117044295, 2024-09-13T11:26:55.176 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.40212404299999999, 2024-09-13T11:26:55.176 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:26:55.176 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:26:55.176 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-09-13T11:26:55.176 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:26:55.176 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.177 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:26:55.177 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.042874+0000", 2024-09-13T11:26:55.177 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:55.177 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.177 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.177 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:26:55.177 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.042874+0000", 2024-09-13T11:26:55.177 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:55.177 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.177 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.177 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:26:55.177 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.042875+0000", 2024-09-13T11:26:55.177 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.432e-06 2024-09-13T11:26:55.178 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.178 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.178 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:26:55.178 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.042886+0000", 2024-09-13T11:26:55.178 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0265000000000001e-05 2024-09-13T11:26:55.178 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.178 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.178 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:26:55.178 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.443455+0000", 2024-09-13T11:26:55.178 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.40056951400000002 2024-09-13T11:26:55.178 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.178 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.179 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:26:55.179 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.443480+0000", 2024-09-13T11:26:55.179 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.4929000000000001e-05 2024-09-13T11:26:55.179 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.179 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.179 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:26:55.179 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.443666+0000", 2024-09-13T11:26:55.179 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000186446 2024-09-13T11:26:55.179 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.179 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.179 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:26:55.180 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.444998+0000", 2024-09-13T11:26:55.180 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0013314570000000001 2024-09-13T11:26:55.180 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:55.180 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:26:55.180 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:55.180 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.180 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.180 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPush(3.1d 85/78 [PushOp(3:bc4b9336:::benchmark_data_smithi177_31116_object11944:head, version: 49'734, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:bc4b9336:::benchmark_data_smithi177_31116_object11944:head@49'734, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:26:55.180 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:42.225966+0000", 2024-09-13T11:26:55.180 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 192.93395208199999, 2024-09-13T11:26:55.180 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.23879947800000001, 2024-09-13T11:26:55.181 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:26:55.181 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:26:55.181 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:26:55.181 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:26:55.181 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.181 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:26:55.181 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.225966+0000", 2024-09-13T11:26:55.181 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:55.181 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.181 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.181 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:26:55.181 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.225966+0000", 2024-09-13T11:26:55.181 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:55.182 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.182 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.182 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:26:55.182 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.225969+0000", 2024-09-13T11:26:55.182 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.0400000000000001e-06 2024-09-13T11:26:55.182 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.182 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.182 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:26:55.182 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.226036+0000", 2024-09-13T11:26:55.182 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.6978999999999997e-05 2024-09-13T11:26:55.182 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.183 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.183 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:26:55.183 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.463110+0000", 2024-09-13T11:26:55.183 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.23707400000000001 2024-09-13T11:26:55.183 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.183 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.183 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:26:55.183 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.463140+0000", 2024-09-13T11:26:55.183 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.9575999999999999e-05 2024-09-13T11:26:55.183 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.183 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.184 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:26:55.184 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.463326+0000", 2024-09-13T11:26:55.184 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000186298 2024-09-13T11:26:55.184 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.184 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.184 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:26:55.184 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.463351+0000", 2024-09-13T11:26:55.184 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.4862999999999999e-05 2024-09-13T11:26:55.184 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.184 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.184 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:26:55.185 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.464766+0000", 2024-09-13T11:26:55.185 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001414722 2024-09-13T11:26:55.185 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:55.185 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:26:55.185 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:55.185 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.185 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.185 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPush(3.1d 89/78 [PushOp(3:bc1b7cfb:::benchmark_data_smithi177_31116_object22629:head, version: 55'1387, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:bc1b7cfb:::benchmark_data_smithi177_31116_object22629:head@55'1387, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:26:55.185 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:46.660855+0000", 2024-09-13T11:26:55.185 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 188.49906358600001, 2024-09-13T11:26:55.185 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.26622785599999998, 2024-09-13T11:26:55.185 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:26:55.186 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:26:55.186 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:26:55.186 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:26:55.186 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.186 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:26:55.186 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:46.660855+0000", 2024-09-13T11:26:55.186 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:55.186 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.186 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.186 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:26:55.186 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:46.660855+0000", 2024-09-13T11:26:55.186 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:55.187 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.187 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.187 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:26:55.187 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:46.660857+0000", 2024-09-13T11:26:55.187 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.6560000000000001e-06 2024-09-13T11:26:55.187 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.187 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.187 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:26:55.187 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:46.660931+0000", 2024-09-13T11:26:55.187 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.4041999999999993e-05 2024-09-13T11:26:55.187 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.187 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.188 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:26:55.188 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:46.924654+0000", 2024-09-13T11:26:55.188 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.26372239800000002 2024-09-13T11:26:55.188 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.188 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.188 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:26:55.188 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:46.924729+0000", 2024-09-13T11:26:55.188 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.5061999999999994e-05 2024-09-13T11:26:55.188 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.188 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.188 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:26:55.189 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:46.924986+0000", 2024-09-13T11:26:55.189 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00025762000000000002 2024-09-13T11:26:55.189 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.189 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.189 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:26:55.189 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:46.925013+0000", 2024-09-13T11:26:55.189 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.6179000000000001e-05 2024-09-13T11:26:55.189 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.189 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.189 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:26:55.189 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:46.927082+0000", 2024-09-13T11:26:55.189 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.002069899 2024-09-13T11:26:55.190 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:55.190 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:26:55.190 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:55.190 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.190 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.190 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPush(3.1d 90/78 [PushOp(3:bc0e7228:::benchmark_data_smithi177_31116_object24733:head, version: 55'1512, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:bc0e7228:::benchmark_data_smithi177_31116_object24733:head@55'1512, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:26:55.190 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:47.529126+0000", 2024-09-13T11:26:55.190 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 187.63079261199999, 2024-09-13T11:26:55.190 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.21343989499999999, 2024-09-13T11:26:55.190 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:26:55.190 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:26:55.190 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:26:55.191 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:26:55.191 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.191 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:26:55.191 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:47.529126+0000", 2024-09-13T11:26:55.191 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:55.191 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.191 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.191 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:26:55.191 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:47.529126+0000", 2024-09-13T11:26:55.191 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:55.191 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.192 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.192 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:26:55.192 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:47.529127+0000", 2024-09-13T11:26:55.192 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.252e-06 2024-09-13T11:26:55.192 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.192 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.192 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:26:55.192 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:47.529185+0000", 2024-09-13T11:26:55.192 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.8224999999999999e-05 2024-09-13T11:26:55.192 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.193 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.193 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:26:55.193 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:47.740216+0000", 2024-09-13T11:26:55.193 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.21103060800000001 2024-09-13T11:26:55.193 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.193 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.193 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:26:55.193 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:47.740248+0000", 2024-09-13T11:26:55.193 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.2821999999999998e-05 2024-09-13T11:26:55.193 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.193 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.194 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:26:55.194 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:47.740504+0000", 2024-09-13T11:26:55.194 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00025518399999999999 2024-09-13T11:26:55.194 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.194 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.194 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:26:55.194 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:47.740544+0000", 2024-09-13T11:26:55.194 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.0438999999999997e-05 2024-09-13T11:26:55.194 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.194 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.194 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:26:55.194 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:47.742565+0000", 2024-09-13T11:26:55.195 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0020213649999999998 2024-09-13T11:26:55.195 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:55.195 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:26:55.195 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:55.195 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.195 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.195 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:83199 3.1d 3:be2c6fae:::benchmark_data_smithi177_31116_object83198:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106)", 2024-09-13T11:26:55.195 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:24:04.297306+0000", 2024-09-13T11:26:55.195 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 170.86261266, 2024-09-13T11:26:55.195 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.25114587999999999, 2024-09-13T11:26:55.195 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:26:55.196 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:26:55.196 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:55.196 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:26:55.196 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:26:55.196 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:55.196 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 83199 2024-09-13T11:26:55.196 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.196 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:26:55.196 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.196 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:26:55.196 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.297306+0000", 2024-09-13T11:26:55.196 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:55.197 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.197 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.197 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:26:55.197 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.297306+0000", 2024-09-13T11:26:55.197 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:55.200 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.200 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.201 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:26:55.201 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.297308+0000", 2024-09-13T11:26:55.201 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.4150000000000002e-06 2024-09-13T11:26:55.201 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.201 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.201 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:26:55.201 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.297352+0000", 2024-09-13T11:26:55.201 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.4431000000000002e-05 2024-09-13T11:26:55.201 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.201 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.201 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:26:55.201 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.297369+0000", 2024-09-13T11:26:55.201 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.6685e-05 2024-09-13T11:26:55.202 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.202 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.202 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:26:55.202 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.297376+0000", 2024-09-13T11:26:55.202 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.7299999999999999e-06 2024-09-13T11:26:55.202 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.202 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.202 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:26:55.202 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.297427+0000", 2024-09-13T11:26:55.202 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.1461000000000001e-05 2024-09-13T11:26:55.202 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.202 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.203 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:26:55.203 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.297528+0000", 2024-09-13T11:26:55.203 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00010028599999999999 2024-09-13T11:26:55.203 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.203 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.203 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 15", 2024-09-13T11:26:55.203 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.297685+0000", 2024-09-13T11:26:55.203 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00015760500000000001 2024-09-13T11:26:55.203 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.203 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.203 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:26:55.203 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.300779+0000", 2024-09-13T11:26:55.203 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0030936700000000002 2024-09-13T11:26:55.204 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.204 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.204 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:55.204 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.548279+0000", 2024-09-13T11:26:55.204 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.24750059999999999 2024-09-13T11:26:55.204 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.204 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.204 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:26:55.204 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.548373+0000", 2024-09-13T11:26:55.204 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.3451999999999999e-05 2024-09-13T11:26:55.204 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.205 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.205 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:26:55.205 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.548451+0000", 2024-09-13T11:26:55.205 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.8545000000000001e-05 2024-09-13T11:26:55.205 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:55.205 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:26:55.205 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:55.205 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.205 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.205 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:84621 3.1d 3:becd72aa:::benchmark_data_smithi177_31116_object84620:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-09-13T11:26:55.206 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:24:05.741008+0000", 2024-09-13T11:26:55.206 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 169.41891065999999, 2024-09-13T11:26:55.206 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.17994101100000001, 2024-09-13T11:26:55.206 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:26:55.206 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:26:55.206 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:55.206 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:26:55.206 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:26:55.206 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:55.206 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 84621 2024-09-13T11:26:55.206 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.206 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:26:55.206 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.207 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:26:55.207 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.741008+0000", 2024-09-13T11:26:55.207 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:55.207 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.207 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.207 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:26:55.207 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.741008+0000", 2024-09-13T11:26:55.207 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:55.207 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.207 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.207 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:26:55.207 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.741010+0000", 2024-09-13T11:26:55.207 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.898e-06 2024-09-13T11:26:55.208 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.208 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.208 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:26:55.208 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.741061+0000", 2024-09-13T11:26:55.208 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.0898e-05 2024-09-13T11:26:55.208 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.208 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.208 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:26:55.208 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.741085+0000", 2024-09-13T11:26:55.208 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.3558e-05 2024-09-13T11:26:55.208 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.208 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.208 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:26:55.209 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.741093+0000", 2024-09-13T11:26:55.209 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.3890000000000005e-06 2024-09-13T11:26:55.209 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.209 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.209 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:26:55.209 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.741178+0000", 2024-09-13T11:26:55.209 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.4792999999999999e-05 2024-09-13T11:26:55.209 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.209 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.209 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:26:55.209 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.741386+0000", 2024-09-13T11:26:55.210 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000207505 2024-09-13T11:26:55.210 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.210 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.210 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 15", 2024-09-13T11:26:55.210 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.741597+0000", 2024-09-13T11:26:55.210 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00021158200000000001 2024-09-13T11:26:55.210 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.210 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.210 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:26:55.210 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.743392+0000", 2024-09-13T11:26:55.210 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00179518 2024-09-13T11:26:55.211 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.211 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.211 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:55.211 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.920866+0000", 2024-09-13T11:26:55.211 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.17747373699999999 2024-09-13T11:26:55.211 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.211 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.211 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:26:55.211 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.920913+0000", 2024-09-13T11:26:55.211 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.6451e-05 2024-09-13T11:26:55.211 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.212 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.212 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:26:55.212 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.920949+0000", 2024-09-13T11:26:55.212 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.6019999999999997e-05 2024-09-13T11:26:55.212 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:55.212 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:26:55.212 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:55.212 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.212 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.212 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPush(3.25 124/120 [PushOp(3:a6d4ee3f:::benchmark_data_smithi177_31116_object14712:head, version: 51'872, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:a6d4ee3f:::benchmark_data_smithi177_31116_object14712:head@51'872, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:26:55.212 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:24:31.996253+0000", 2024-09-13T11:26:55.213 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 143.163664888, 2024-09-13T11:26:55.213 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.17694863499999999, 2024-09-13T11:26:55.213 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:26:55.213 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:26:55.213 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:26:55.213 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:26:55.213 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.213 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:26:55.213 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:31.996253+0000", 2024-09-13T11:26:55.213 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:55.213 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.214 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.214 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:26:55.214 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:31.996253+0000", 2024-09-13T11:26:55.214 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:55.214 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.214 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.214 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:26:55.214 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:31.996257+0000", 2024-09-13T11:26:55.214 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.2629999999999999e-06 2024-09-13T11:26:55.214 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.214 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.215 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:26:55.215 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:31.996359+0000", 2024-09-13T11:26:55.215 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000102456 2024-09-13T11:26:55.215 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.215 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.215 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:26:55.215 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:32.172313+0000", 2024-09-13T11:26:55.215 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.17595388000000001 2024-09-13T11:26:55.215 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.215 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.215 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:26:55.215 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:32.172326+0000", 2024-09-13T11:26:55.216 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2632999999999999e-05 2024-09-13T11:26:55.216 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.216 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.216 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:26:55.216 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:32.172423+0000", 2024-09-13T11:26:55.216 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.7084999999999996e-05 2024-09-13T11:26:55.216 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.216 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.216 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:26:55.216 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:32.172438+0000", 2024-09-13T11:26:55.216 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5775999999999999e-05 2024-09-13T11:26:55.217 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.217 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.217 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:26:55.217 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:32.173202+0000", 2024-09-13T11:26:55.217 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00076354199999999998 2024-09-13T11:26:55.217 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:55.217 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:26:55.217 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:55.217 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.217 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.217 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:126039 3.29 3:96100181:::benchmark_data_smithi177_31116_object126038:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e130)", 2024-09-13T11:26:55.218 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:24:43.012785+0000", 2024-09-13T11:26:55.218 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 132.14713353799999, 2024-09-13T11:26:55.218 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.26685311900000003, 2024-09-13T11:26:55.218 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:26:55.218 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:26:55.218 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:55.218 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:26:55.218 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:26:55.218 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:55.218 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 126039 2024-09-13T11:26:55.218 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.218 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:26:55.218 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.219 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:26:55.219 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:43.012785+0000", 2024-09-13T11:26:55.219 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:55.219 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.219 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.219 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:26:55.219 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:43.012785+0000", 2024-09-13T11:26:55.219 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:55.219 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.219 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.219 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:26:55.219 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:43.012787+0000", 2024-09-13T11:26:55.220 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.875e-06 2024-09-13T11:26:55.220 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.220 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.220 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:26:55.220 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:43.012826+0000", 2024-09-13T11:26:55.220 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.9483999999999997e-05 2024-09-13T11:26:55.220 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.220 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.220 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:26:55.220 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:43.012839+0000", 2024-09-13T11:26:55.220 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3405e-05 2024-09-13T11:26:55.220 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.221 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.221 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:26:55.221 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:43.012845+0000", 2024-09-13T11:26:55.221 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.9100000000000002e-06 2024-09-13T11:26:55.221 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.221 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.221 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:26:55.221 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:43.012903+0000", 2024-09-13T11:26:55.221 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.8131000000000003e-05 2024-09-13T11:26:55.221 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.221 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.222 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:26:55.222 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:43.013066+0000", 2024-09-13T11:26:55.222 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000162058 2024-09-13T11:26:55.222 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.222 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.222 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 4,5", 2024-09-13T11:26:55.222 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:43.013235+0000", 2024-09-13T11:26:55.222 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00016929600000000001 2024-09-13T11:26:55.222 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.222 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.222 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:55.222 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:43.017063+0000", 2024-09-13T11:26:55.223 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0038283890000000002 2024-09-13T11:26:55.223 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.223 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.223 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:26:55.223 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:43.017086+0000", 2024-09-13T11:26:55.223 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.2934000000000001e-05 2024-09-13T11:26:55.223 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.223 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.223 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:55.223 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:43.278804+0000", 2024-09-13T11:26:55.223 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.26171751799999998 2024-09-13T11:26:55.223 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.224 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.224 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:26:55.224 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:43.278862+0000", 2024-09-13T11:26:55.224 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.8431999999999999e-05 2024-09-13T11:26:55.224 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.224 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.224 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:26:55.224 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:43.279638+0000", 2024-09-13T11:26:55.224 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00077568699999999995 2024-09-13T11:26:55.224 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:55.224 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:26:55.224 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:55.225 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.225 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.225 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop_reply(client.4665.0:126039 3.29 e130/120 ondisk, result = 0)", 2024-09-13T11:26:55.225 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:24:43.021691+0000", 2024-09-13T11:26:55.225 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 132.13822761200001, 2024-09-13T11:26:55.225 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.25795579699999999, 2024-09-13T11:26:55.225 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:26:55.225 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:26:55.225 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:26:55.225 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:26:55.225 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.226 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:26:55.226 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:43.021691+0000", 2024-09-13T11:26:55.226 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:55.226 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.226 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.226 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:26:55.226 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:43.021691+0000", 2024-09-13T11:26:55.226 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:55.226 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.226 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.226 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:26:55.226 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:43.021692+0000", 2024-09-13T11:26:55.226 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3880000000000001e-06 2024-09-13T11:26:55.227 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.227 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.227 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:26:55.227 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:43.021700+0000", 2024-09-13T11:26:55.227 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.9310000000000008e-06 2024-09-13T11:26:55.227 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.227 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.227 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:26:55.227 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:43.278719+0000", 2024-09-13T11:26:55.227 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.25701909299999998 2024-09-13T11:26:55.227 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.227 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.227 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:26:55.228 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:43.278728+0000", 2024-09-13T11:26:55.228 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.5000000000000005e-06 2024-09-13T11:26:55.228 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.228 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.228 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:26:55.228 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:43.278786+0000", 2024-09-13T11:26:55.228 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.7640999999999999e-05 2024-09-13T11:26:55.228 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.228 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.228 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:26:55.228 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:43.278796+0000", 2024-09-13T11:26:55.228 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.9199999999999999e-06 2024-09-13T11:26:55.229 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.229 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.229 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:26:55.229 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:43.279646+0000", 2024-09-13T11:26:55.229 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00085032399999999996 2024-09-13T11:26:55.229 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:55.229 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:26:55.229 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:55.229 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.229 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.229 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.4665.0:138294 3.9 e140/117 3:90012dab:::benchmark_data_smithi177_31116_object138293:head v 140'5246, mlcod=140'5243)", 2024-09-13T11:26:55.229 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:24:54.252227+0000", 2024-09-13T11:26:55.230 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 120.907691124, 2024-09-13T11:26:55.230 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.32919836400000002, 2024-09-13T11:26:55.230 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:26:55.230 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:26:55.230 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:55.230 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:26:55.230 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.230 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:26:55.230 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:54.252227+0000", 2024-09-13T11:26:55.230 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:55.230 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.230 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.231 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:26:55.231 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:54.252227+0000", 2024-09-13T11:26:55.231 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:55.231 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.231 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.231 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:26:55.231 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:54.252228+0000", 2024-09-13T11:26:55.231 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.8899999999999998e-07 2024-09-13T11:26:55.231 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.231 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.231 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:26:55.231 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:54.252276+0000", 2024-09-13T11:26:55.232 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.8137999999999999e-05 2024-09-13T11:26:55.232 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.232 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.232 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:26:55.232 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:54.578216+0000", 2024-09-13T11:26:55.232 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.32594014399999999 2024-09-13T11:26:55.232 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.232 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.232 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:26:55.232 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:54.578235+0000", 2024-09-13T11:26:55.232 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.8943000000000001e-05 2024-09-13T11:26:55.232 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.232 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.233 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:26:55.233 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:54.578396+0000", 2024-09-13T11:26:55.233 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000160922 2024-09-13T11:26:55.233 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.233 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.233 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:26:55.233 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:54.578457+0000", 2024-09-13T11:26:55.233 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.0665999999999998e-05 2024-09-13T11:26:55.233 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.233 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.233 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:26:55.233 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:54.581339+0000", 2024-09-13T11:26:55.233 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0028826839999999999 2024-09-13T11:26:55.234 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.234 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.234 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:26:55.234 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:54.581425+0000", 2024-09-13T11:26:55.234 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.6077999999999998e-05 2024-09-13T11:26:55.234 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:55.234 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:26:55.234 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:55.234 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.234 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.234 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPushReply(3.29 143/120 [PushReplyOp(3:973d2047:::benchmark_data_smithi177_31116_object49749:head)])", 2024-09-13T11:26:55.234 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:24:57.387106+0000", 2024-09-13T11:26:55.235 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 117.772811797, 2024-09-13T11:26:55.235 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.64813637000000002, 2024-09-13T11:26:55.235 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:26:55.235 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:26:55.235 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-09-13T11:26:55.235 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:26:55.235 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.235 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:26:55.235 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:57.387106+0000", 2024-09-13T11:26:55.235 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:55.235 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.235 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.235 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:26:55.236 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:57.387106+0000", 2024-09-13T11:26:55.236 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:55.236 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.236 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.236 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:26:55.236 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:57.387116+0000", 2024-09-13T11:26:55.236 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.1530000000000003e-06 2024-09-13T11:26:55.236 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.236 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.236 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:26:55.236 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:57.387127+0000", 2024-09-13T11:26:55.236 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1197e-05 2024-09-13T11:26:55.237 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.237 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.237 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:26:55.237 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:58.034264+0000", 2024-09-13T11:26:55.237 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.64713691799999995 2024-09-13T11:26:55.237 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.237 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.237 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:26:55.237 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:58.034273+0000", 2024-09-13T11:26:55.237 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.0929999999999992e-06 2024-09-13T11:26:55.237 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.238 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.238 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:26:55.238 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:58.034321+0000", 2024-09-13T11:26:55.238 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.8396999999999998e-05 2024-09-13T11:26:55.238 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.238 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.238 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:26:55.238 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:58.035243+0000", 2024-09-13T11:26:55.238 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00092161200000000004 2024-09-13T11:26:55.238 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:55.238 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:26:55.238 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:55.238 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.239 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.239 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPushReply(3.29 144/120 [PushReplyOp(3:94a6fc3a:::benchmark_data_smithi177_31116_object50245:head)])", 2024-09-13T11:26:55.239 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:24:58.320977+0000", 2024-09-13T11:26:55.239 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 116.838941214, 2024-09-13T11:26:55.239 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.31110569599999999, 2024-09-13T11:26:55.239 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:26:55.239 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:26:55.239 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-09-13T11:26:55.239 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:26:55.239 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.239 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:26:55.239 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:58.320977+0000", 2024-09-13T11:26:55.239 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:55.240 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.240 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.240 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:26:55.240 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:58.320977+0000", 2024-09-13T11:26:55.240 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:55.240 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.240 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.240 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:26:55.240 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:58.320978+0000", 2024-09-13T11:26:55.240 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.8700000000000004e-07 2024-09-13T11:26:55.240 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.241 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.241 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:26:55.241 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:58.320985+0000", 2024-09-13T11:26:55.241 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.742e-06 2024-09-13T11:26:55.241 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.241 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.241 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:26:55.241 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:58.631147+0000", 2024-09-13T11:26:55.241 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.31016212900000001 2024-09-13T11:26:55.241 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.241 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.241 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:26:55.242 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:58.631159+0000", 2024-09-13T11:26:55.242 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1946999999999999e-05 2024-09-13T11:26:55.242 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.242 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.242 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:26:55.242 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:58.631257+0000", 2024-09-13T11:26:55.242 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.8096000000000004e-05 2024-09-13T11:26:55.242 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.242 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.242 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:26:55.242 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:58.632083+0000", 2024-09-13T11:26:55.242 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00082589499999999999 2024-09-13T11:26:55.242 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:55.242 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:26:55.243 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:55.243 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.243 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.243 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.4665.0:170721 3.9 e162/117 3:93244b1b:::benchmark_data_smithi177_31116_object30418:head v 162'5743, mlcod=162'5740)", 2024-09-13T11:26:55.243 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:25:21.721723+0000", 2024-09-13T11:26:55.243 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 93.438195499000003, 2024-09-13T11:26:55.243 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.19722754300000001, 2024-09-13T11:26:55.243 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:26:55.243 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:26:55.243 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:55.243 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:26:55.243 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.244 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:26:55.244 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:21.721723+0000", 2024-09-13T11:26:55.244 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:55.244 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.244 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.244 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:26:55.244 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:21.721723+0000", 2024-09-13T11:26:55.244 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:55.244 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.244 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.244 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:26:55.244 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:21.721726+0000", 2024-09-13T11:26:55.245 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.1930000000000002e-06 2024-09-13T11:26:55.245 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.245 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.245 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:26:55.245 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:21.721741+0000", 2024-09-13T11:26:55.245 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5299999999999999e-05 2024-09-13T11:26:55.245 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.245 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.245 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:26:55.245 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:21.915450+0000", 2024-09-13T11:26:55.245 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.193708519 2024-09-13T11:26:55.245 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.245 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.245 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:26:55.246 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:21.915462+0000", 2024-09-13T11:26:55.247 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2414e-05 2024-09-13T11:26:55.247 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.247 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.247 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:26:55.248 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:21.915603+0000", 2024-09-13T11:26:55.248 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00014116500000000001 2024-09-13T11:26:55.248 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.248 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.248 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:26:55.248 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:21.915668+0000", 2024-09-13T11:26:55.248 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.4488999999999999e-05 2024-09-13T11:26:55.248 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.248 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.248 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:26:55.249 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:21.918850+0000", 2024-09-13T11:26:55.249 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0031825310000000002 2024-09-13T11:26:55.249 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.249 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.249 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:26:55.249 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:21.918950+0000", 2024-09-13T11:26:55.249 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.9932000000000005e-05 2024-09-13T11:26:55.249 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:55.249 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:26:55.249 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:55.249 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.249 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.250 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:191058 3.1d 3:bd6e7029:::benchmark_data_smithi177_31116_object50755:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e175)", 2024-09-13T11:26:55.250 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:25:38.512403+0000", 2024-09-13T11:26:55.250 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 76.647514954000002, 2024-09-13T11:26:55.250 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.38726333200000002, 2024-09-13T11:26:55.250 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:26:55.250 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:26:55.250 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:55.250 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:26:55.250 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:26:55.250 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:55.250 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 191058 2024-09-13T11:26:55.250 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.251 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:26:55.251 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.251 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:26:55.251 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:38.512403+0000", 2024-09-13T11:26:55.251 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:55.251 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.251 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.251 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:26:55.251 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:38.512403+0000", 2024-09-13T11:26:55.251 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:55.251 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.251 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.252 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:26:55.252 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:38.512406+0000", 2024-09-13T11:26:55.252 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.4059999999999999e-06 2024-09-13T11:26:55.252 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.252 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.252 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:26:55.252 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:38.512412+0000", 2024-09-13T11:26:55.252 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.8170000000000001e-06 2024-09-13T11:26:55.252 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.252 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.252 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:26:55.252 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:38.512430+0000", 2024-09-13T11:26:55.253 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.7397999999999998e-05 2024-09-13T11:26:55.253 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.253 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.253 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:26:55.253 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:38.512438+0000", 2024-09-13T11:26:55.253 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.3459999999999994e-06 2024-09-13T11:26:55.253 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.253 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.253 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:26:55.253 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:38.512526+0000", 2024-09-13T11:26:55.253 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.8146999999999995e-05 2024-09-13T11:26:55.254 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.254 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.254 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:26:55.254 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:38.512722+0000", 2024-09-13T11:26:55.254 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00019587599999999999 2024-09-13T11:26:55.254 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.254 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.254 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 15", 2024-09-13T11:26:55.254 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:38.512882+0000", 2024-09-13T11:26:55.254 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00015957300000000001 2024-09-13T11:26:55.254 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.254 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.255 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:26:55.255 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:38.517854+0000", 2024-09-13T11:26:55.255 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0049725500000000001 2024-09-13T11:26:55.255 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.255 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.255 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:55.255 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:38.899562+0000", 2024-09-13T11:26:55.255 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.38170719800000003 2024-09-13T11:26:55.255 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.255 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.255 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:26:55.255 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:38.899622+0000", 2024-09-13T11:26:55.255 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.0396999999999998e-05 2024-09-13T11:26:55.255 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.256 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.256 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:26:55.256 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:38.899667+0000", 2024-09-13T11:26:55.256 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.4623999999999998e-05 2024-09-13T11:26:55.256 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:55.256 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:26:55.256 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:55.256 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.256 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.256 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.4665.0:191547 3.25 e176/141 3:a7ac3e5b:::benchmark_data_smithi177_31116_object51244:head v 176'6020, mlcod=175'6018)", 2024-09-13T11:26:55.256 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:25:39.496290+0000", 2024-09-13T11:26:55.256 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 75.663628392999996, 2024-09-13T11:26:55.256 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.179206057, 2024-09-13T11:26:55.257 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:26:55.257 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:26:55.257 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:55.257 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:26:55.257 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.257 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:26:55.257 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:39.496290+0000", 2024-09-13T11:26:55.257 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:55.257 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.257 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.257 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:26:55.257 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:39.496290+0000", 2024-09-13T11:26:55.258 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:55.258 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.258 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.258 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:26:55.258 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:39.496293+0000", 2024-09-13T11:26:55.258 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.8499999999999998e-06 2024-09-13T11:26:55.258 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.258 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.258 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:26:55.258 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:39.496305+0000", 2024-09-13T11:26:55.258 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2286999999999999e-05 2024-09-13T11:26:55.258 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.259 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.259 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:26:55.259 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:39.670614+0000", 2024-09-13T11:26:55.259 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.17430884699999999 2024-09-13T11:26:55.259 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.259 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.259 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:26:55.259 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:39.670625+0000", 2024-09-13T11:26:55.259 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1233000000000001e-05 2024-09-13T11:26:55.259 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.259 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.259 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:26:55.259 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:39.670722+0000", 2024-09-13T11:26:55.260 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.6885999999999994e-05 2024-09-13T11:26:55.260 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.260 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.260 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:26:55.260 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:39.670756+0000", 2024-09-13T11:26:55.260 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.4125999999999998e-05 2024-09-13T11:26:55.260 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.260 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.260 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:26:55.260 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:39.675411+0000", 2024-09-13T11:26:55.260 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0046546160000000003 2024-09-13T11:26:55.260 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.261 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.261 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:26:55.261 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:39.675496+0000", 2024-09-13T11:26:55.261 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.5211999999999997e-05 2024-09-13T11:26:55.261 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:55.261 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:26:55.261 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:55.261 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.261 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.261 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.4665.0:250598 3.9 e211/117 3:929b3711:::benchmark_data_smithi177_31116_object110295:head v 211'6996, mlcod=211'6992)", 2024-09-13T11:26:55.261 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:26:32.474084+0000", 2024-09-13T11:26:55.261 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 22.685834364000002, 2024-09-13T11:26:55.261 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.24130304, 2024-09-13T11:26:55.261 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:26:55.262 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:26:55.262 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:55.262 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:26:55.262 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.262 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:26:55.262 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:32.474084+0000", 2024-09-13T11:26:55.262 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:55.262 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.262 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.262 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:26:55.262 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:32.474084+0000", 2024-09-13T11:26:55.262 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:55.262 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.263 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.263 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:26:55.263 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:32.474085+0000", 2024-09-13T11:26:55.263 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.2699999999999998e-07 2024-09-13T11:26:55.263 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.263 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.263 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:26:55.263 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:32.474090+0000", 2024-09-13T11:26:55.263 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.8339999999999998e-06 2024-09-13T11:26:55.263 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.263 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.264 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:26:55.264 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:32.713078+0000", 2024-09-13T11:26:55.264 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.23898883900000001 2024-09-13T11:26:55.264 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.264 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.264 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:26:55.264 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:32.713088+0000", 2024-09-13T11:26:55.264 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.5170000000000005e-06 2024-09-13T11:26:55.264 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.264 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.264 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:26:55.264 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:32.713150+0000", 2024-09-13T11:26:55.264 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.1715000000000001e-05 2024-09-13T11:26:55.264 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.265 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.265 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:26:55.265 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:32.713165+0000", 2024-09-13T11:26:55.265 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5804999999999999e-05 2024-09-13T11:26:55.265 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.265 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.265 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:26:55.265 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:32.715350+0000", 2024-09-13T11:26:55.265 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0021848979999999998 2024-09-13T11:26:55.265 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.265 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.265 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:26:55.266 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:32.715387+0000", 2024-09-13T11:26:55.266 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.6504999999999997e-05 2024-09-13T11:26:55.266 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:55.266 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:26:55.266 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:55.266 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.266 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.266 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.4665.0:268374 3.25 e228/141 3:a5681ce4:::benchmark_data_smithi177_31116_object128071:head v 228'7272, mlcod=228'7269)", 2024-09-13T11:26:55.266 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:26:54.256363+0000", 2024-09-13T11:26:55.266 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 0.90355541299999997, 2024-09-13T11:26:55.266 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.17220392000000001, 2024-09-13T11:26:55.266 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:26:55.266 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:26:55.267 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:55.267 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:26:55.267 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.267 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:26:55.267 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:54.256363+0000", 2024-09-13T11:26:55.267 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:55.267 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.267 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.267 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:26:55.267 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:54.256363+0000", 2024-09-13T11:26:55.267 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:55.268 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.268 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.268 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:26:55.268 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:54.256365+0000", 2024-09-13T11:26:55.268 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.782e-06 2024-09-13T11:26:55.268 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.268 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.268 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:26:55.268 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:54.256377+0000", 2024-09-13T11:26:55.268 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2092999999999999e-05 2024-09-13T11:26:55.268 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.268 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.268 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:26:55.269 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:54.426547+0000", 2024-09-13T11:26:55.269 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.17017012000000001 2024-09-13T11:26:55.269 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.269 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.269 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:26:55.269 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:54.426551+0000", 2024-09-13T11:26:55.269 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.4279999999999998e-06 2024-09-13T11:26:55.269 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.269 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.269 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:26:55.269 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:54.426594+0000", 2024-09-13T11:26:55.269 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.2518000000000002e-05 2024-09-13T11:26:55.270 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.270 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.270 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:26:55.270 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:54.426615+0000", 2024-09-13T11:26:55.270 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.1633000000000001e-05 2024-09-13T11:26:55.270 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.270 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.270 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:26:55.270 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:54.428519+0000", 2024-09-13T11:26:55.270 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0019041430000000001 2024-09-13T11:26:55.270 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.271 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.271 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:26:55.271 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:54.428567+0000", 2024-09-13T11:26:55.271 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.7203000000000002e-05 2024-09-13T11:26:55.271 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:55.271 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:26:55.271 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:55.271 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:55.271 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:26:55.271 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-13T11:26:55.273 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-13T11:26:55.273 INFO:tasks.ceph.osd.3.smithi177.stderr:2024-09-13T11:26:55.269+0000 7f10adba4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26574) UID: 0 2024-09-13T11:26:55.374 INFO:tasks.ceph.osd.2.smithi162.stderr:2024-09-13T11:26:55.371+0000 7fe92b35a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26313) UID: 0 2024-09-13T11:26:55.378 INFO:teuthology.orchestra.run.smithi001.stderr:set 3.17 pg_upmap_items mapping to [9->13,8->5] 2024-09-13T11:26:55.383 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T11:26:55.383 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-13T11:26:55.378+0000 7f6a5d1a2640 -1 osd.0 228 handle_osd_map injecting map crc failure 2024-09-13T11:26:55.417 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-13T11:26:55.417 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [], 2024-09-13T11:26:55.418 INFO:teuthology.orchestra.run.smithi162.stdout: "num_ops": 0 2024-09-13T11:26:55.418 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-13T11:26:55.438 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-13T11:26:55.475 INFO:tasks.ceph.osd.1.smithi092.stderr:2024-09-13T11:26:55.471+0000 7fe7789c5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33718) UID: 0 2024-09-13T11:26:55.575 INFO:tasks.ceph.osd.11.smithi177.stderr:2024-09-13T11:26:55.573+0000 7f24792c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26580) UID: 0 2024-09-13T11:26:55.627 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-13T11:26:55.627 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [], 2024-09-13T11:26:55.627 INFO:teuthology.orchestra.run.smithi162.stdout: "complaint_time": 30, 2024-09-13T11:26:55.627 INFO:teuthology.orchestra.run.smithi162.stdout: "num_blocked_ops": 0 2024-09-13T11:26:55.627 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-13T11:26:55.646 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-13T11:26:55.673 INFO:teuthology.orchestra.run.smithi001.stdout:osd.0: {} 2024-09-13T11:26:55.673 INFO:teuthology.orchestra.run.smithi001.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T11:26:55.681 INFO:tasks.ceph.osd.11.smithi177.stderr:2024-09-13T11:26:55.673+0000 7f24792c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26580) UID: 0 2024-09-13T11:26:55.706 INFO:teuthology.orchestra.run.smithi001.stdout:osd.1: {} 2024-09-13T11:26:55.706 INFO:teuthology.orchestra.run.smithi001.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T11:26:55.743 INFO:teuthology.orchestra.run.smithi001.stdout:osd.2: {} 2024-09-13T11:26:55.744 INFO:teuthology.orchestra.run.smithi001.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T11:26:55.779 INFO:tasks.ceph.osd.3.smithi177.stderr:2024-09-13T11:26:55.773+0000 7f10adba4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26574) UID: 0 2024-09-13T11:26:55.782 INFO:teuthology.orchestra.run.smithi001.stdout:osd.3: {} 2024-09-13T11:26:55.782 INFO:teuthology.orchestra.run.smithi001.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T11:26:55.786 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-13T11:26:55.786 INFO:teuthology.orchestra.run.smithi162.stdout: "size": 20, 2024-09-13T11:26:55.786 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 600, 2024-09-13T11:26:55.786 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [ 2024-09-13T11:26:55.786 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.786 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4257.0:80 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 262144~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-13T11:26:55.786 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:22:12.516065+0000", 2024-09-13T11:26:55.786 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 283.26933886699999, 2024-09-13T11:26:55.786 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.46461997999999999, 2024-09-13T11:26:55.786 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:26:55.786 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:26:55.786 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:55.787 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:26:55.787 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4257", 2024-09-13T11:26:55.787 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/1488131527", 2024-09-13T11:26:55.787 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 80 2024-09-13T11:26:55.787 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.787 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:26:55.787 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.787 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:26:55.787 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.516065+0000", 2024-09-13T11:26:55.787 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:55.787 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.787 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.787 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:26:55.788 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.516065+0000", 2024-09-13T11:26:55.788 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:55.788 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.788 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.788 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:26:55.788 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.516066+0000", 2024-09-13T11:26:55.788 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0869999999999999e-06 2024-09-13T11:26:55.788 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.788 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.788 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:26:55.788 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.516112+0000", 2024-09-13T11:26:55.788 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.5503000000000001e-05 2024-09-13T11:26:55.788 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.789 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.789 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:26:55.789 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.516128+0000", 2024-09-13T11:26:55.789 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5891000000000001e-05 2024-09-13T11:26:55.789 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.789 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.789 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:26:55.789 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.516138+0000", 2024-09-13T11:26:55.789 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.8120000000000002e-06 2024-09-13T11:26:55.789 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.789 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.789 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:26:55.790 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.520398+0000", 2024-09-13T11:26:55.790 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0042605450000000001 2024-09-13T11:26:55.790 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.790 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.790 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:26:55.790 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.520526+0000", 2024-09-13T11:26:55.790 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000128355 2024-09-13T11:26:55.790 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.790 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.790 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 0", 2024-09-13T11:26:55.790 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.520730+0000", 2024-09-13T11:26:55.790 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00020337799999999999 2024-09-13T11:26:55.790 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.791 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.791 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:26:55.791 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.522417+0000", 2024-09-13T11:26:55.791 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001687184 2024-09-13T11:26:55.791 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.791 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.791 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:55.791 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.980408+0000", 2024-09-13T11:26:55.791 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.45799068399999998 2024-09-13T11:26:55.791 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.791 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.791 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:26:55.791 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.980565+0000", 2024-09-13T11:26:55.791 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00015760899999999999 2024-09-13T11:26:55.792 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.792 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.792 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:26:55.792 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.980685+0000", 2024-09-13T11:26:55.792 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000119932 2024-09-13T11:26:55.792 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:55.792 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:26:55.792 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:55.792 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.792 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.792 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop_reply(client.4257.0:80 1.0 e24/23 ondisk, result = 0)", 2024-09-13T11:26:55.792 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:22:12.522938+0000", 2024-09-13T11:26:55.792 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 283.26246583400001, 2024-09-13T11:26:55.793 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.45776420600000001, 2024-09-13T11:26:55.793 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:26:55.793 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:26:55.793 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:26:55.793 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:26:55.793 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.793 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:26:55.793 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.522938+0000", 2024-09-13T11:26:55.793 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:55.793 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.793 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.793 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:26:55.793 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.522938+0000", 2024-09-13T11:26:55.794 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:55.794 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.794 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.794 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:26:55.794 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.522941+0000", 2024-09-13T11:26:55.794 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.0290000000000001e-06 2024-09-13T11:26:55.794 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.794 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.794 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:26:55.794 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.522950+0000", 2024-09-13T11:26:55.794 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.7499999999999992e-06 2024-09-13T11:26:55.794 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.794 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.795 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:26:55.795 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.980215+0000", 2024-09-13T11:26:55.795 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.45726523800000002 2024-09-13T11:26:55.795 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.795 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.795 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:26:55.795 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.980229+0000", 2024-09-13T11:26:55.795 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3549e-05 2024-09-13T11:26:55.795 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.795 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.795 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:26:55.795 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.980365+0000", 2024-09-13T11:26:55.795 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00013608700000000001 2024-09-13T11:26:55.796 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.796 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.796 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:26:55.796 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.980388+0000", 2024-09-13T11:26:55.796 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.3112000000000002e-05 2024-09-13T11:26:55.796 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.796 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.796 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:26:55.796 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.980702+0000", 2024-09-13T11:26:55.796 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00031444099999999999 2024-09-13T11:26:55.796 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:55.796 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:26:55.796 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:55.796 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.797 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.797 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:25416 3.8 3:1a1fbdd1:::benchmark_data_smithi177_31116_object25415:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e56)", 2024-09-13T11:26:55.797 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:04.512188+0000", 2024-09-13T11:26:55.797 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 231.273215867, 2024-09-13T11:26:55.797 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.210208697, 2024-09-13T11:26:55.797 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:26:55.797 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:26:55.797 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:55.797 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:26:55.797 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:26:55.797 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:55.797 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 25416 2024-09-13T11:26:55.798 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.798 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:26:55.798 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.798 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:26:55.798 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.512188+0000", 2024-09-13T11:26:55.798 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:55.798 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.798 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.798 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:26:55.798 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.512188+0000", 2024-09-13T11:26:55.798 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:55.798 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.798 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.798 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:26:55.798 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.512192+0000", 2024-09-13T11:26:55.799 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.2660000000000003e-06 2024-09-13T11:26:55.799 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.799 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.799 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:26:55.799 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.512247+0000", 2024-09-13T11:26:55.799 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.4889000000000003e-05 2024-09-13T11:26:55.799 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.799 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.799 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:26:55.799 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.512279+0000", 2024-09-13T11:26:55.799 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.1146000000000003e-05 2024-09-13T11:26:55.799 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.799 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.799 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:26:55.800 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.512297+0000", 2024-09-13T11:26:55.800 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.825e-05 2024-09-13T11:26:55.800 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.800 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.800 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:26:55.800 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.512377+0000", 2024-09-13T11:26:55.800 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.9875000000000004e-05 2024-09-13T11:26:55.800 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.800 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.800 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:26:55.800 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.512570+0000", 2024-09-13T11:26:55.800 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00019312100000000001 2024-09-13T11:26:55.800 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.801 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.801 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 1", 2024-09-13T11:26:55.801 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.512798+0000", 2024-09-13T11:26:55.801 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00022847899999999999 2024-09-13T11:26:55.801 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.801 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.801 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:26:55.801 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.514975+0000", 2024-09-13T11:26:55.801 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0021769979999999999 2024-09-13T11:26:55.801 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.801 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.801 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:55.801 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.722207+0000", 2024-09-13T11:26:55.802 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.20723217099999999 2024-09-13T11:26:55.802 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.802 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.802 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:26:55.802 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.722314+0000", 2024-09-13T11:26:55.802 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00010648400000000001 2024-09-13T11:26:55.802 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.802 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.802 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:26:55.802 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.722397+0000", 2024-09-13T11:26:55.802 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.3017999999999997e-05 2024-09-13T11:26:55.802 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:55.802 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:26:55.802 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:55.803 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.803 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.803 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop_reply(client.4665.0:25416 3.8 e56/40 ondisk, result = 0)", 2024-09-13T11:26:55.803 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:04.515635+0000", 2024-09-13T11:26:55.803 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 231.269769215, 2024-09-13T11:26:55.803 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.20677600500000001, 2024-09-13T11:26:55.803 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:26:55.803 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:26:55.803 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:26:55.803 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:26:55.803 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.803 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:26:55.804 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.515635+0000", 2024-09-13T11:26:55.804 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:55.804 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.804 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.804 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:26:55.804 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.515635+0000", 2024-09-13T11:26:55.804 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:55.804 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.804 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.804 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:26:55.804 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.515639+0000", 2024-09-13T11:26:55.804 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.619e-06 2024-09-13T11:26:55.804 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.805 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.805 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:26:55.805 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.515650+0000", 2024-09-13T11:26:55.805 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.111e-05 2024-09-13T11:26:55.805 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.805 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.805 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:26:55.805 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.722004+0000", 2024-09-13T11:26:55.805 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.206354805 2024-09-13T11:26:55.805 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.805 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.805 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:26:55.805 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.722017+0000", 2024-09-13T11:26:55.805 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2840000000000001e-05 2024-09-13T11:26:55.806 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.806 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.806 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:26:55.806 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.722186+0000", 2024-09-13T11:26:55.806 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00016898299999999999 2024-09-13T11:26:55.806 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.806 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.806 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:26:55.806 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.722199+0000", 2024-09-13T11:26:55.806 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.289e-05 2024-09-13T11:26:55.806 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.806 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.806 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:26:55.807 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.722411+0000", 2024-09-13T11:26:55.807 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00021175799999999999 2024-09-13T11:26:55.807 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:55.807 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:26:55.807 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:55.807 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.807 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.807 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:30833 3.8 3:1c203f8d:::benchmark_data_smithi177_31116_object30832:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e59)", 2024-09-13T11:26:55.807 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:09.506656+0000", 2024-09-13T11:26:55.807 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 226.27874831700001, 2024-09-13T11:26:55.807 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.224862282, 2024-09-13T11:26:55.807 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:26:55.808 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:26:55.808 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:55.808 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:26:55.808 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:26:55.808 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:55.808 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 30833 2024-09-13T11:26:55.808 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.808 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:26:55.808 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.808 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:26:55.808 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.506656+0000", 2024-09-13T11:26:55.808 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:55.808 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.808 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.809 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:26:55.809 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.506656+0000", 2024-09-13T11:26:55.809 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:55.809 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.809 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.809 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:26:55.809 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.506660+0000", 2024-09-13T11:26:55.809 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.6e-06 2024-09-13T11:26:55.809 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.809 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.809 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:26:55.809 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.506717+0000", 2024-09-13T11:26:55.809 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.6131999999999997e-05 2024-09-13T11:26:55.810 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.810 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.810 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:26:55.810 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.506751+0000", 2024-09-13T11:26:55.810 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.4746000000000002e-05 2024-09-13T11:26:55.810 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.810 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.810 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:26:55.810 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.506774+0000", 2024-09-13T11:26:55.810 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.2636000000000001e-05 2024-09-13T11:26:55.810 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.811 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.811 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:26:55.811 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.506878+0000", 2024-09-13T11:26:55.811 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000104057 2024-09-13T11:26:55.811 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.811 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.811 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:26:55.811 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.507052+0000", 2024-09-13T11:26:55.811 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00017402800000000001 2024-09-13T11:26:55.811 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.811 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.811 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 1", 2024-09-13T11:26:55.811 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.507217+0000", 2024-09-13T11:26:55.812 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00016516 2024-09-13T11:26:55.812 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.812 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.812 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:26:55.812 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.508992+0000", 2024-09-13T11:26:55.812 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0017743990000000001 2024-09-13T11:26:55.812 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.812 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.812 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:55.812 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.731429+0000", 2024-09-13T11:26:55.812 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.22243718000000001 2024-09-13T11:26:55.812 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.812 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.812 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:26:55.813 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.731479+0000", 2024-09-13T11:26:55.813 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.0146999999999999e-05 2024-09-13T11:26:55.813 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.813 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.813 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:26:55.813 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.731518+0000", 2024-09-13T11:26:55.813 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.9196999999999998e-05 2024-09-13T11:26:55.813 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:55.813 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:26:55.813 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:55.816 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.817 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.817 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop_reply(client.4665.0:30833 3.8 e59/40 ondisk, result = 0)", 2024-09-13T11:26:55.817 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:09.508907+0000", 2024-09-13T11:26:55.817 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 226.27649722300001, 2024-09-13T11:26:55.817 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.222618862, 2024-09-13T11:26:55.817 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:26:55.817 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:26:55.817 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:26:55.817 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:26:55.817 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.817 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:26:55.817 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.508907+0000", 2024-09-13T11:26:55.817 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:55.817 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.818 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.818 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:26:55.818 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.508907+0000", 2024-09-13T11:26:55.818 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:55.818 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.818 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.818 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:26:55.818 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.508910+0000", 2024-09-13T11:26:55.818 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.993e-06 2024-09-13T11:26:55.818 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.818 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.818 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:26:55.818 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.508918+0000", 2024-09-13T11:26:55.819 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.7000000000000008e-06 2024-09-13T11:26:55.819 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.819 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.819 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:26:55.819 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.731341+0000", 2024-09-13T11:26:55.819 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.22242308699999999 2024-09-13T11:26:55.819 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.819 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.819 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:26:55.819 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.731350+0000", 2024-09-13T11:26:55.819 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.3179999999999996e-06 2024-09-13T11:26:55.819 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.819 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.819 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:26:55.819 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.731418+0000", 2024-09-13T11:26:55.820 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.7586e-05 2024-09-13T11:26:55.820 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.820 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.820 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:26:55.820 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.731424+0000", 2024-09-13T11:26:55.820 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.844e-06 2024-09-13T11:26:55.820 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.820 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.820 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:26:55.820 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.731526+0000", 2024-09-13T11:26:55.820 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000101334 2024-09-13T11:26:55.820 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:55.820 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:26:55.821 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:55.821 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.821 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.821 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPush(3.10 76/70 [PushOp(3:0e7d2700:::benchmark_data_smithi177_31116_object17406:head, version: 52'1075, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:0e7d2700:::benchmark_data_smithi177_31116_object17406:head@52'1075, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:26:55.821 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:32.206198+0000", 2024-09-13T11:26:55.821 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 203.57920651399999, 2024-09-13T11:26:55.821 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.25193395000000002, 2024-09-13T11:26:55.821 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:26:55.821 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:26:55.821 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:26:55.821 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:26:55.821 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.821 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:26:55.821 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.206198+0000", 2024-09-13T11:26:55.822 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:55.822 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.822 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.822 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:26:55.822 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.206198+0000", 2024-09-13T11:26:55.822 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:55.822 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.822 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.822 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:26:55.822 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.206199+0000", 2024-09-13T11:26:55.822 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.46e-06 2024-09-13T11:26:55.822 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.822 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.822 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:26:55.823 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.206250+0000", 2024-09-13T11:26:55.823 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.113e-05 2024-09-13T11:26:55.823 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.823 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.823 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:26:55.823 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.456042+0000", 2024-09-13T11:26:55.823 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.24979182999999999 2024-09-13T11:26:55.823 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.823 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.823 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:26:55.823 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.456084+0000", 2024-09-13T11:26:55.823 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.1814000000000003e-05 2024-09-13T11:26:55.823 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.823 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.824 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:26:55.824 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.456342+0000", 2024-09-13T11:26:55.824 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000258404 2024-09-13T11:26:55.824 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.824 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.824 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:26:55.824 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.456367+0000", 2024-09-13T11:26:55.824 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.5099000000000001e-05 2024-09-13T11:26:55.824 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.824 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.824 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:26:55.824 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.458132+0000", 2024-09-13T11:26:55.824 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0017642129999999999 2024-09-13T11:26:55.825 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:55.825 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:26:55.825 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:55.825 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.825 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.825 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:60417 3.10 3:092808b0:::benchmark_data_smithi177_31116_object60416:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e85)", 2024-09-13T11:26:55.825 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:42.173238+0000", 2024-09-13T11:26:55.825 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 193.61216560599999, 2024-09-13T11:26:55.825 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.38965994599999998, 2024-09-13T11:26:55.825 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:26:55.825 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:26:55.825 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:55.825 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:26:55.825 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:26:55.826 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:55.826 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 60417 2024-09-13T11:26:55.826 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.826 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:26:55.826 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.826 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:26:55.826 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.173238+0000", 2024-09-13T11:26:55.826 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:55.826 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.826 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.826 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:26:55.826 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.173238+0000", 2024-09-13T11:26:55.826 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:55.826 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.827 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.827 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:26:55.827 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.173241+0000", 2024-09-13T11:26:55.827 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.6819999999999999e-06 2024-09-13T11:26:55.827 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.827 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.827 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:26:55.827 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.173335+0000", 2024-09-13T11:26:55.827 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.3957999999999997e-05 2024-09-13T11:26:55.827 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.827 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.827 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:26:55.827 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.173358+0000", 2024-09-13T11:26:55.827 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.3023999999999998e-05 2024-09-13T11:26:55.828 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.828 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.828 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:26:55.828 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.173373+0000", 2024-09-13T11:26:55.828 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4867e-05 2024-09-13T11:26:55.828 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.828 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.828 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:26:55.828 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.173466+0000", 2024-09-13T11:26:55.828 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.2542999999999998e-05 2024-09-13T11:26:55.828 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.828 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.828 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:26:55.828 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.173754+0000", 2024-09-13T11:26:55.829 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000288657 2024-09-13T11:26:55.829 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.829 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.829 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 0", 2024-09-13T11:26:55.829 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.174066+0000", 2024-09-13T11:26:55.829 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00031170999999999998 2024-09-13T11:26:55.829 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.829 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.829 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:26:55.829 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.178245+0000", 2024-09-13T11:26:55.829 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00417864 2024-09-13T11:26:55.829 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.829 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.829 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:55.830 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.562617+0000", 2024-09-13T11:26:55.830 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.38437253199999999 2024-09-13T11:26:55.830 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.830 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.830 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:26:55.830 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.562782+0000", 2024-09-13T11:26:55.830 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00016499200000000001 2024-09-13T11:26:55.830 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.830 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.830 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:26:55.830 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.562898+0000", 2024-09-13T11:26:55.830 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000116341 2024-09-13T11:26:55.830 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:55.830 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:26:55.831 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:55.831 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.831 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.831 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:83193 3.8 3:10066d83:::benchmark_data_smithi177_31116_object83192:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106)", 2024-09-13T11:26:55.831 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:24:04.283923+0000", 2024-09-13T11:26:55.831 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 171.501480923, 2024-09-13T11:26:55.831 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.64775463499999997, 2024-09-13T11:26:55.831 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:26:55.831 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:26:55.831 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:55.831 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:26:55.831 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:26:55.831 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:55.831 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 83193 2024-09-13T11:26:55.832 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.832 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:26:55.832 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.832 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:26:55.832 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.283923+0000", 2024-09-13T11:26:55.832 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:55.832 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.832 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.832 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:26:55.832 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.283923+0000", 2024-09-13T11:26:55.832 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:55.832 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.832 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.833 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:26:55.833 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.283925+0000", 2024-09-13T11:26:55.833 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.9489999999999999e-06 2024-09-13T11:26:55.833 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.833 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.833 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:26:55.833 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.283960+0000", 2024-09-13T11:26:55.833 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.5063e-05 2024-09-13T11:26:55.833 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.833 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.833 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:26:55.833 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.283974+0000", 2024-09-13T11:26:55.833 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3546e-05 2024-09-13T11:26:55.833 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.834 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.834 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:26:55.834 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.283983+0000", 2024-09-13T11:26:55.834 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.2499999999999995e-06 2024-09-13T11:26:55.834 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.834 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.834 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:26:55.834 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.284061+0000", 2024-09-13T11:26:55.834 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.7946000000000002e-05 2024-09-13T11:26:55.834 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.834 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.834 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:26:55.834 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.284270+0000", 2024-09-13T11:26:55.834 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00020876799999999999 2024-09-13T11:26:55.834 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.835 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.835 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 1", 2024-09-13T11:26:55.835 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.284489+0000", 2024-09-13T11:26:55.835 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00021956800000000001 2024-09-13T11:26:55.835 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.835 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.835 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:26:55.835 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.286456+0000", 2024-09-13T11:26:55.835 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0019665809999999998 2024-09-13T11:26:55.835 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.835 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.835 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:55.835 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.931497+0000", 2024-09-13T11:26:55.835 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.64504147700000003 2024-09-13T11:26:55.836 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.836 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.836 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:26:55.836 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.931589+0000", 2024-09-13T11:26:55.836 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.2041999999999997e-05 2024-09-13T11:26:55.836 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.836 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.836 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:26:55.836 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.931678+0000", 2024-09-13T11:26:55.836 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.8444999999999998e-05 2024-09-13T11:26:55.836 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:55.836 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:26:55.836 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:55.836 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.837 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.837 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:115060 3.8 3:11200ba9:::benchmark_data_smithi177_31116_object115059:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e125)", 2024-09-13T11:26:55.837 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:24:33.602178+0000", 2024-09-13T11:26:55.837 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 142.183225929, 2024-09-13T11:26:55.837 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.23668186799999999, 2024-09-13T11:26:55.837 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:26:55.837 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:26:55.837 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:55.837 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:26:55.837 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:26:55.837 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:55.837 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 115060 2024-09-13T11:26:55.837 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.838 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:26:55.838 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.838 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:26:55.838 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:33.602178+0000", 2024-09-13T11:26:55.838 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:55.838 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.838 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.838 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:26:55.838 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:33.602178+0000", 2024-09-13T11:26:55.838 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:55.838 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.838 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.838 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:26:55.838 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:33.602181+0000", 2024-09-13T11:26:55.839 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.0369999999999999e-06 2024-09-13T11:26:55.839 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.839 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.839 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:26:55.839 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:33.602217+0000", 2024-09-13T11:26:55.839 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.5837999999999998e-05 2024-09-13T11:26:55.839 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.839 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.839 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:26:55.839 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:33.602235+0000", 2024-09-13T11:26:55.839 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.7571000000000002e-05 2024-09-13T11:26:55.839 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.839 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.839 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:26:55.840 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:33.602245+0000", 2024-09-13T11:26:55.840 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0360000000000001e-05 2024-09-13T11:26:55.840 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.840 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.840 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:26:55.840 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:33.602312+0000", 2024-09-13T11:26:55.840 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.6989000000000005e-05 2024-09-13T11:26:55.840 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.840 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.840 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:26:55.840 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:33.602490+0000", 2024-09-13T11:26:55.840 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000178464 2024-09-13T11:26:55.840 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.840 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.841 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 1", 2024-09-13T11:26:55.841 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:33.602670+0000", 2024-09-13T11:26:55.841 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00018002799999999999 2024-09-13T11:26:55.841 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.841 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.841 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:26:55.841 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:33.604382+0000", 2024-09-13T11:26:55.841 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001711314 2024-09-13T11:26:55.841 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.842 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.842 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:55.842 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:33.838738+0000", 2024-09-13T11:26:55.842 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.23435634699999999 2024-09-13T11:26:55.842 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.842 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.842 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:26:55.842 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:33.838815+0000", 2024-09-13T11:26:55.842 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.7381999999999999e-05 2024-09-13T11:26:55.842 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.842 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.842 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:26:55.842 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:33.838860+0000", 2024-09-13T11:26:55.842 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.4537999999999999e-05 2024-09-13T11:26:55.843 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:55.843 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:26:55.843 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:55.843 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.843 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.843 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop_reply(client.4665.0:115060 3.8 e125/117 ondisk, result = 0)", 2024-09-13T11:26:55.843 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:24:33.604404+0000", 2024-09-13T11:26:55.843 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 142.180999671, 2024-09-13T11:26:55.843 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.23446235400000001, 2024-09-13T11:26:55.843 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:26:55.843 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:26:55.843 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:26:55.843 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:26:55.843 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.844 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:26:55.844 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:33.604404+0000", 2024-09-13T11:26:55.844 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:55.844 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.844 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.844 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:26:55.844 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:33.604404+0000", 2024-09-13T11:26:55.844 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:55.844 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.844 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.844 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:26:55.844 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:33.604406+0000", 2024-09-13T11:26:55.844 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.418e-06 2024-09-13T11:26:55.844 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.845 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.845 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:26:55.845 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:33.604411+0000", 2024-09-13T11:26:55.845 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.3519999999999997e-06 2024-09-13T11:26:55.845 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.845 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.845 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:26:55.845 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:33.838578+0000", 2024-09-13T11:26:55.845 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.23416656199999999 2024-09-13T11:26:55.845 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.845 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.845 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:26:55.845 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:33.838586+0000", 2024-09-13T11:26:55.845 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.9920000000000007e-06 2024-09-13T11:26:55.846 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.846 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.846 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:26:55.846 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:33.838655+0000", 2024-09-13T11:26:55.846 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.9463999999999998e-05 2024-09-13T11:26:55.846 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.846 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.846 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:26:55.846 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:33.838731+0000", 2024-09-13T11:26:55.846 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.5487000000000004e-05 2024-09-13T11:26:55.847 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.847 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.847 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:26:55.847 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:33.838867+0000", 2024-09-13T11:26:55.847 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000136079 2024-09-13T11:26:55.847 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:55.847 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:26:55.847 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:55.847 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.847 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.847 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPush(3.6 129/122 [PushOp(3:6261f3b6:::benchmark_data_smithi177_31116_object64090:head, version: 90'3486, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:6261f3b6:::benchmark_data_smithi177_31116_object64090:head@90'3486, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:26:55.847 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:24:38.641487+0000", 2024-09-13T11:26:55.847 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 137.14391725999999, 2024-09-13T11:26:55.847 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.209889098, 2024-09-13T11:26:55.848 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:26:55.848 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:26:55.848 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:26:55.848 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:26:55.848 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.848 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:26:55.848 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:38.641487+0000", 2024-09-13T11:26:55.848 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:55.848 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.848 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.848 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:26:55.848 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:38.641487+0000", 2024-09-13T11:26:55.848 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:55.849 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.849 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.849 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:26:55.849 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:38.641489+0000", 2024-09-13T11:26:55.849 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.0980000000000001e-06 2024-09-13T11:26:55.849 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.849 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.849 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:26:55.849 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:38.641560+0000", 2024-09-13T11:26:55.849 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.1404999999999998e-05 2024-09-13T11:26:55.849 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.849 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.849 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:26:55.849 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:38.850498+0000", 2024-09-13T11:26:55.850 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.20893800500000001 2024-09-13T11:26:55.850 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.850 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.850 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:26:55.850 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:38.850510+0000", 2024-09-13T11:26:55.850 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1762e-05 2024-09-13T11:26:55.850 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.850 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.850 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:26:55.850 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:38.850610+0000", 2024-09-13T11:26:55.850 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00010011200000000001 2024-09-13T11:26:55.850 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.850 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.850 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:26:55.851 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:38.850628+0000", 2024-09-13T11:26:55.851 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.7291000000000001e-05 2024-09-13T11:26:55.851 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.851 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.851 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:26:55.851 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:38.851376+0000", 2024-09-13T11:26:55.851 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00074842500000000005 2024-09-13T11:26:55.851 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:55.851 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:26:55.851 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:55.851 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.851 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.851 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.4665.0:126919 3.6 e130/122 3:634ce08e:::benchmark_data_smithi177_31116_object126918:head v 130'5137, mlcod=130'5134)", 2024-09-13T11:26:55.852 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:24:43.780120+0000", 2024-09-13T11:26:55.852 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 132.00528436900001, 2024-09-13T11:26:55.852 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.298086295, 2024-09-13T11:26:55.852 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:26:55.852 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:26:55.852 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:55.852 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:26:55.852 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.852 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:26:55.852 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:43.780120+0000", 2024-09-13T11:26:55.852 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:55.852 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.852 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.853 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:26:55.853 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:43.780120+0000", 2024-09-13T11:26:55.853 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:55.853 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.853 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.853 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:26:55.853 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:43.780123+0000", 2024-09-13T11:26:55.853 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.79e-06 2024-09-13T11:26:55.853 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.853 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.853 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:26:55.853 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:43.780160+0000", 2024-09-13T11:26:55.853 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.7945999999999999e-05 2024-09-13T11:26:55.854 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.854 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.854 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:26:55.854 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:44.075853+0000", 2024-09-13T11:26:55.854 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.29569234999999999 2024-09-13T11:26:55.854 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.854 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.854 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:26:55.854 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:44.075861+0000", 2024-09-13T11:26:55.854 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.9510000000000006e-06 2024-09-13T11:26:55.854 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.854 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.854 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:26:55.854 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:44.075924+0000", 2024-09-13T11:26:55.854 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.2946000000000003e-05 2024-09-13T11:26:55.855 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.855 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.855 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:26:55.855 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:44.075942+0000", 2024-09-13T11:26:55.855 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.8108e-05 2024-09-13T11:26:55.855 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.855 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.855 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:26:55.855 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:44.078164+0000", 2024-09-13T11:26:55.855 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.002222375 2024-09-13T11:26:55.855 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.855 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.855 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:26:55.855 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:44.078206+0000", 2024-09-13T11:26:55.856 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.1829000000000002e-05 2024-09-13T11:26:55.856 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:55.856 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:26:55.856 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:55.856 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.856 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.856 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:137473 3.10 3:094c0efb:::benchmark_data_smithi177_31116_object137472:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e139)", 2024-09-13T11:26:55.856 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:24:53.523319+0000", 2024-09-13T11:26:55.856 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 122.26208533000001, 2024-09-13T11:26:55.856 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.18297369299999999, 2024-09-13T11:26:55.856 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:26:55.856 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:26:55.856 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:55.856 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:26:55.857 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:26:55.857 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:55.857 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 137473 2024-09-13T11:26:55.857 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.857 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:26:55.857 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.857 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:26:55.857 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:53.523319+0000", 2024-09-13T11:26:55.857 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:55.857 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.857 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.857 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:26:55.857 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:53.523319+0000", 2024-09-13T11:26:55.857 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:55.858 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.858 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.858 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:26:55.858 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:53.523322+0000", 2024-09-13T11:26:55.858 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.2119999999999999e-06 2024-09-13T11:26:55.858 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.858 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.858 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:26:55.858 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:53.523370+0000", 2024-09-13T11:26:55.858 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.7788999999999999e-05 2024-09-13T11:26:55.858 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.858 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.858 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:26:55.858 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:53.523395+0000", 2024-09-13T11:26:55.859 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.5202e-05 2024-09-13T11:26:55.859 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.859 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.859 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:26:55.859 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:53.523409+0000", 2024-09-13T11:26:55.859 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4263999999999999e-05 2024-09-13T11:26:55.859 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.859 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.859 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:26:55.859 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:53.523477+0000", 2024-09-13T11:26:55.859 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.7578999999999998e-05 2024-09-13T11:26:55.859 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.859 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.859 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:26:55.860 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:53.523638+0000", 2024-09-13T11:26:55.860 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00016078300000000001 2024-09-13T11:26:55.860 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.860 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.860 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 0", 2024-09-13T11:26:55.860 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:53.523800+0000", 2024-09-13T11:26:55.860 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000162564 2024-09-13T11:26:55.860 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.860 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.860 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:26:55.860 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:53.526960+0000", 2024-09-13T11:26:55.860 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0031593950000000002 2024-09-13T11:26:55.860 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.861 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.861 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:55.861 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:53.706180+0000", 2024-09-13T11:26:55.861 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.17922080600000001 2024-09-13T11:26:55.861 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.861 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.861 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:26:55.861 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:53.706237+0000", 2024-09-13T11:26:55.861 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.6845000000000002e-05 2024-09-13T11:26:55.861 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.861 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.861 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:26:55.861 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:53.706292+0000", 2024-09-13T11:26:55.861 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.5254000000000003e-05 2024-09-13T11:26:55.861 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:55.862 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:26:55.862 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:55.862 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.862 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.862 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:137780 3.8 3:1047f268:::benchmark_data_smithi177_31116_object137779:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e139)", 2024-09-13T11:26:55.862 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:24:53.795212+0000", 2024-09-13T11:26:55.862 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 121.990192585, 2024-09-13T11:26:55.862 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.21530981399999999, 2024-09-13T11:26:55.862 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:26:55.862 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:26:55.862 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:55.862 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:26:55.862 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:26:55.862 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:55.863 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 137780 2024-09-13T11:26:55.863 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.863 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:26:55.863 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.863 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:26:55.863 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:53.795212+0000", 2024-09-13T11:26:55.863 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:55.863 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.863 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.863 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:26:55.863 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:53.795212+0000", 2024-09-13T11:26:55.863 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:55.863 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.863 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.864 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:26:55.864 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:53.795214+0000", 2024-09-13T11:26:55.864 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.481e-06 2024-09-13T11:26:55.864 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.864 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.864 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:26:55.864 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:53.795247+0000", 2024-09-13T11:26:55.864 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.3105999999999998e-05 2024-09-13T11:26:55.864 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.864 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.864 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:26:55.864 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:53.795258+0000", 2024-09-13T11:26:55.864 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0872e-05 2024-09-13T11:26:55.864 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.865 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.865 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:26:55.865 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:53.795265+0000", 2024-09-13T11:26:55.865 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.5729999999999996e-06 2024-09-13T11:26:55.865 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.865 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.865 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:26:55.865 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:53.795309+0000", 2024-09-13T11:26:55.865 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.4617999999999999e-05 2024-09-13T11:26:55.865 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.865 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.865 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:26:55.865 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:53.795397+0000", 2024-09-13T11:26:55.865 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.7863999999999998e-05 2024-09-13T11:26:55.866 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.866 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.866 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 1", 2024-09-13T11:26:55.866 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:53.795493+0000", 2024-09-13T11:26:55.866 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.6062000000000003e-05 2024-09-13T11:26:55.866 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.866 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.866 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:26:55.866 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:53.797245+0000", 2024-09-13T11:26:55.866 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0017521559999999999 2024-09-13T11:26:55.866 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.866 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.866 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:55.866 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:54.010409+0000", 2024-09-13T11:26:55.867 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.21316385299999999 2024-09-13T11:26:55.867 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.867 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.867 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:26:55.867 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:54.010474+0000", 2024-09-13T11:26:55.867 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.4998000000000003e-05 2024-09-13T11:26:55.867 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.867 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.867 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:26:55.867 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:54.010521+0000", 2024-09-13T11:26:55.867 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.7231000000000002e-05 2024-09-13T11:26:55.867 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:55.867 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:26:55.868 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:55.868 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.868 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.868 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop_reply(client.4665.0:137780 3.8 e139/117 ondisk, result = 0)", 2024-09-13T11:26:55.868 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:24:53.797821+0000", 2024-09-13T11:26:55.868 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 121.987582687, 2024-09-13T11:26:55.868 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.21270765599999999, 2024-09-13T11:26:55.868 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:26:55.868 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:26:55.868 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:26:55.868 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:26:55.868 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.868 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:26:55.868 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:53.797821+0000", 2024-09-13T11:26:55.869 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:55.869 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.869 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.869 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:26:55.869 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:53.797821+0000", 2024-09-13T11:26:55.869 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:55.869 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.869 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.869 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:26:55.869 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:53.797822+0000", 2024-09-13T11:26:55.869 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.012e-06 2024-09-13T11:26:55.869 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.869 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.869 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:26:55.869 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:53.797829+0000", 2024-09-13T11:26:55.870 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.8009999999999998e-06 2024-09-13T11:26:55.870 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.870 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.870 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:26:55.870 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:54.010334+0000", 2024-09-13T11:26:55.870 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.212504477 2024-09-13T11:26:55.870 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.870 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.870 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:26:55.870 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:54.010342+0000", 2024-09-13T11:26:55.870 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.6549999999999993e-06 2024-09-13T11:26:55.870 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.870 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.870 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:26:55.870 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:54.010394+0000", 2024-09-13T11:26:55.871 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.1875000000000001e-05 2024-09-13T11:26:55.871 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.871 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.871 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:26:55.871 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:54.010402+0000", 2024-09-13T11:26:55.871 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.6499999999999996e-06 2024-09-13T11:26:55.871 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.871 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.871 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:26:55.871 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:54.010529+0000", 2024-09-13T11:26:55.871 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000127186 2024-09-13T11:26:55.871 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:55.871 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:26:55.871 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:55.872 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.872 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.872 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.4665.0:196314 3.6 e178/122 3:62e79b1c:::benchmark_data_smithi177_31116_object56011:head v 178'6256, mlcod=178'6253)", 2024-09-13T11:26:55.872 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:25:43.340693+0000", 2024-09-13T11:26:55.872 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 72.444711106, 2024-09-13T11:26:55.872 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.24109011699999999, 2024-09-13T11:26:55.872 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:26:55.872 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:26:55.872 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:55.872 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:26:55.872 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.872 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:26:55.872 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:43.340693+0000", 2024-09-13T11:26:55.872 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:55.873 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.873 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.873 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:26:55.873 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:43.340693+0000", 2024-09-13T11:26:55.873 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:55.873 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.873 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.873 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:26:55.873 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:43.340704+0000", 2024-09-13T11:26:55.873 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1049e-05 2024-09-13T11:26:55.873 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.873 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.873 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:26:55.873 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:43.340718+0000", 2024-09-13T11:26:55.874 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4214e-05 2024-09-13T11:26:55.874 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.874 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.874 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:26:55.874 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:43.579964+0000", 2024-09-13T11:26:55.874 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.23924590200000001 2024-09-13T11:26:55.874 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.874 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.874 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:26:55.874 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:43.579972+0000", 2024-09-13T11:26:55.874 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.2789999999999998e-06 2024-09-13T11:26:55.874 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.874 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.874 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:26:55.875 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:43.580043+0000", 2024-09-13T11:26:55.875 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.0856000000000001e-05 2024-09-13T11:26:55.875 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.875 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.875 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:26:55.875 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:43.580065+0000", 2024-09-13T11:26:55.875 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.1795999999999999e-05 2024-09-13T11:26:55.875 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.875 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.875 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:26:55.875 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:43.581741+0000", 2024-09-13T11:26:55.875 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0016759190000000001 2024-09-13T11:26:55.875 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.875 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.876 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:26:55.876 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:43.581783+0000", 2024-09-13T11:26:55.876 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.2101999999999998e-05 2024-09-13T11:26:55.876 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:55.876 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:26:55.876 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:55.876 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.876 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.876 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:227446 3.10 3:0a4bd1eb:::benchmark_data_smithi177_31116_object87143:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e196)", 2024-09-13T11:26:55.876 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:26:11.077279+0000", 2024-09-13T11:26:55.876 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 44.708125338000002, 2024-09-13T11:26:55.876 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.18995021400000001, 2024-09-13T11:26:55.876 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:26:55.876 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:26:55.877 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:55.877 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:26:55.877 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:26:55.877 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:55.877 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 227446 2024-09-13T11:26:55.877 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.877 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:26:55.877 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.877 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:26:55.877 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:11.077279+0000", 2024-09-13T11:26:55.877 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:55.877 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.877 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.877 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:26:55.878 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:11.077279+0000", 2024-09-13T11:26:55.878 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:55.878 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.878 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.878 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:26:55.878 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:11.077281+0000", 2024-09-13T11:26:55.878 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.3429999999999998e-06 2024-09-13T11:26:55.878 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.878 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.878 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:26:55.878 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:11.077287+0000", 2024-09-13T11:26:55.878 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.353e-06 2024-09-13T11:26:55.878 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.878 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.879 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:26:55.879 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:11.077303+0000", 2024-09-13T11:26:55.879 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5523e-05 2024-09-13T11:26:55.879 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.879 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.879 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:26:55.879 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:11.077312+0000", 2024-09-13T11:26:55.879 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.6049999999999998e-06 2024-09-13T11:26:55.879 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.879 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.879 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:26:55.879 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:11.077359+0000", 2024-09-13T11:26:55.879 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.7388000000000001e-05 2024-09-13T11:26:55.879 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.880 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.880 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:26:55.880 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:11.077448+0000", 2024-09-13T11:26:55.880 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.9431999999999993e-05 2024-09-13T11:26:55.880 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.880 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.880 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 0", 2024-09-13T11:26:55.880 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:11.077513+0000", 2024-09-13T11:26:55.880 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.4543999999999997e-05 2024-09-13T11:26:55.880 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.880 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.880 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:26:55.880 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:11.080314+0000", 2024-09-13T11:26:55.880 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.002801441 2024-09-13T11:26:55.881 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.881 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.881 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:55.881 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:11.267144+0000", 2024-09-13T11:26:55.881 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.18682944700000001 2024-09-13T11:26:55.881 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.881 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.881 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:26:55.881 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:11.267191+0000", 2024-09-13T11:26:55.881 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.7227e-05 2024-09-13T11:26:55.881 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.881 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.881 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:26:55.881 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:11.267229+0000", 2024-09-13T11:26:55.882 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.7910999999999997e-05 2024-09-13T11:26:55.882 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:55.882 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:26:55.882 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:55.882 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.882 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.882 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop_reply(client.4665.0:227446 3.10 e196/84 ondisk, result = 0)", 2024-09-13T11:26:55.882 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:26:11.079968+0000", 2024-09-13T11:26:55.882 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 44.705436544000001, 2024-09-13T11:26:55.882 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.18726515199999999, 2024-09-13T11:26:55.882 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:26:55.882 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:26:55.882 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:26:55.882 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:26:55.883 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.883 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:26:55.883 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:11.079968+0000", 2024-09-13T11:26:55.883 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:55.883 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.883 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.883 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:26:55.883 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:11.079968+0000", 2024-09-13T11:26:55.883 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:55.883 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.883 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.883 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:26:55.883 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:11.079969+0000", 2024-09-13T11:26:55.883 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.42e-06 2024-09-13T11:26:55.884 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.884 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.884 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:26:55.884 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:11.079974+0000", 2024-09-13T11:26:55.884 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.3739999999999997e-06 2024-09-13T11:26:55.884 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.884 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.884 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:26:55.884 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:11.267071+0000", 2024-09-13T11:26:55.884 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.18709714699999999 2024-09-13T11:26:55.884 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.884 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.884 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:26:55.885 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:11.267080+0000", 2024-09-13T11:26:55.885 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.8589999999999994e-06 2024-09-13T11:26:55.885 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.885 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.885 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:26:55.885 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:11.267134+0000", 2024-09-13T11:26:55.885 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.4131000000000001e-05 2024-09-13T11:26:55.885 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.885 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.885 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:26:55.885 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:11.267140+0000", 2024-09-13T11:26:55.885 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.9499999999999998e-06 2024-09-13T11:26:55.885 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.885 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.885 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:26:55.886 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:11.267233+0000", 2024-09-13T11:26:55.886 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.2270999999999998e-05 2024-09-13T11:26:55.886 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:55.886 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:26:55.886 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:55.886 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.886 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.886 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:266873 3.30 3:0f4a798b:::benchmark_data_smithi177_31116_object126570:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e224)", 2024-09-13T11:26:55.886 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:26:49.175028+0000", 2024-09-13T11:26:55.886 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 6.6103764189999996, 2024-09-13T11:26:55.886 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.89281703099999998, 2024-09-13T11:26:55.886 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:26:55.886 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:26:55.887 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:55.887 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:26:55.887 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:26:55.887 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:55.887 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 266873 2024-09-13T11:26:55.887 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.887 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:26:55.887 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.887 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:26:55.887 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:49.175028+0000", 2024-09-13T11:26:55.887 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:55.887 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.887 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.887 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:26:55.888 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:49.175028+0000", 2024-09-13T11:26:55.888 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:55.888 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.888 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.888 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:26:55.888 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:49.175031+0000", 2024-09-13T11:26:55.888 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.4300000000000002e-06 2024-09-13T11:26:55.888 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.888 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.888 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:26:55.888 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:49.175041+0000", 2024-09-13T11:26:55.888 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.6299999999999993e-06 2024-09-13T11:26:55.888 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.888 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.889 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:26:55.889 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:49.175066+0000", 2024-09-13T11:26:55.889 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.5193000000000001e-05 2024-09-13T11:26:55.889 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.889 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.889 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:26:55.889 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:49.175084+0000", 2024-09-13T11:26:55.889 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.853e-05 2024-09-13T11:26:55.889 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.889 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.889 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:26:55.889 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:49.253283+0000", 2024-09-13T11:26:55.889 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.078198765000000003 2024-09-13T11:26:55.889 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.890 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.890 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-09-13T11:26:55.890 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:49.253292+0000", 2024-09-13T11:26:55.890 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.0920000000000004e-06 2024-09-13T11:26:55.890 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.890 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.890 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:26:55.890 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:50.059052+0000", 2024-09-13T11:26:55.890 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.80575967599999998 2024-09-13T11:26:55.890 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.891 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.891 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:26:55.891 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:50.059226+0000", 2024-09-13T11:26:55.891 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00017433499999999999 2024-09-13T11:26:55.891 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.891 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.891 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 0", 2024-09-13T11:26:55.891 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:50.059374+0000", 2024-09-13T11:26:55.891 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00014768600000000001 2024-09-13T11:26:55.891 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.891 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.891 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:26:55.891 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:50.062440+0000", 2024-09-13T11:26:55.891 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.003066086 2024-09-13T11:26:55.892 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.892 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.892 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:55.892 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:50.067764+0000", 2024-09-13T11:26:55.892 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0053240370000000002 2024-09-13T11:26:55.892 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.892 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.892 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:26:55.892 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:50.067802+0000", 2024-09-13T11:26:55.892 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.7462000000000001e-05 2024-09-13T11:26:55.892 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:55.892 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:55.892 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:26:55.892 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:50.067845+0000", 2024-09-13T11:26:55.893 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.3108999999999997e-05 2024-09-13T11:26:55.893 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:55.893 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:26:55.893 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:55.893 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:55.893 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:26:55.893 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-13T11:26:55.894 INFO:teuthology.orchestra.run.smithi001.stdout:osd.4: {} 2024-09-13T11:26:55.894 INFO:teuthology.orchestra.run.smithi001.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T11:26:55.895 INFO:teuthology.orchestra.run.smithi001.stdout:osd.5: {} 2024-09-13T11:26:55.895 INFO:teuthology.orchestra.run.smithi001.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T11:26:55.896 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-09-13T11:26:55.896 INFO:tasks.ceph.osd.8.smithi001.stderr:2024-09-13T11:26:55.890+0000 7fb9b48ae640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26312) UID: 0 2024-09-13T11:26:55.997 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-13T11:26:55.994+0000 7f6a68775640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26307) UID: 0 2024-09-13T11:26:56.039 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-13T11:26:56.040 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [ 2024-09-13T11:26:56.040 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:56.040 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:270212 3.18 3:186a49d4:::benchmark_data_smithi177_31116_object129909:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e229)", 2024-09-13T11:26:56.040 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:26:55.972673+0000", 2024-09-13T11:26:56.040 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 0.068118209999999998, 2024-09-13T11:26:56.040 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.068143393999999996, 2024-09-13T11:26:56.040 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:26:56.040 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:26:56.040 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "waiting for sub ops", 2024-09-13T11:26:56.040 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:26:56.040 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:26:56.041 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:56.041 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 270212 2024-09-13T11:26:56.041 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:56.041 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:26:56.041 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:56.041 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:26:56.041 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:55.972673+0000", 2024-09-13T11:26:56.041 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:56.041 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:56.041 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:56.041 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:26:56.041 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:55.972673+0000", 2024-09-13T11:26:56.041 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:56.041 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:56.042 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:56.042 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:26:56.042 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:55.972673+0000", 2024-09-13T11:26:56.042 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.0500000000000004e-07 2024-09-13T11:26:56.042 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:56.042 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:56.042 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:26:56.042 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:55.972676+0000", 2024-09-13T11:26:56.042 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.6230000000000001e-06 2024-09-13T11:26:56.042 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:56.043 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:56.043 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:26:56.043 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:55.972682+0000", 2024-09-13T11:26:56.043 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.5049999999999998e-06 2024-09-13T11:26:56.043 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:56.043 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:56.043 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:26:56.043 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:55.972685+0000", 2024-09-13T11:26:56.043 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.444e-06 2024-09-13T11:26:56.043 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:56.043 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:56.043 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:26:56.043 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:55.976248+0000", 2024-09-13T11:26:56.044 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0035631389999999999 2024-09-13T11:26:56.044 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:56.044 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:56.044 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:26:56.044 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:55.976376+0000", 2024-09-13T11:26:56.044 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00012788799999999999 2024-09-13T11:26:56.044 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:56.044 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:56.044 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 0", 2024-09-13T11:26:56.044 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:55.976484+0000", 2024-09-13T11:26:56.044 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000108207 2024-09-13T11:26:56.044 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:56.044 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:56.044 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:26:56.045 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:55.983727+0000", 2024-09-13T11:26:56.045 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.007243284 2024-09-13T11:26:56.045 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:56.045 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:26:56.045 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:56.045 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:56.045 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:56.045 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:270294 3.18 3:1b6b65d9:::benchmark_data_smithi177_31116_object129991:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e229)", 2024-09-13T11:26:56.045 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:26:56.039570+0000", 2024-09-13T11:26:56.045 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 0.001221162, 2024-09-13T11:26:56.045 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001320559, 2024-09-13T11:26:56.045 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:26:56.045 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:26:56.045 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "waiting for sub ops", 2024-09-13T11:26:56.046 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:26:56.046 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:26:56.046 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:56.046 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 270294 2024-09-13T11:26:56.046 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:56.046 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:26:56.046 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:56.046 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:26:56.046 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:56.039570+0000", 2024-09-13T11:26:56.046 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:56.046 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:56.046 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:56.047 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:26:56.047 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:56.039570+0000", 2024-09-13T11:26:56.047 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:56.047 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:56.047 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:56.047 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:26:56.047 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:56.039571+0000", 2024-09-13T11:26:56.047 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4500000000000001e-06 2024-09-13T11:26:56.047 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:56.047 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:56.047 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:26:56.047 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:56.039575+0000", 2024-09-13T11:26:56.047 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.8689999999999997e-06 2024-09-13T11:26:56.047 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:56.048 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:56.048 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:26:56.048 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:56.039585+0000", 2024-09-13T11:26:56.048 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.5000000000000005e-06 2024-09-13T11:26:56.048 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:56.048 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:56.048 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:26:56.048 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:56.039590+0000", 2024-09-13T11:26:56.048 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.9840000000000001e-06 2024-09-13T11:26:56.048 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:56.048 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:56.048 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:26:56.048 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:56.039632+0000", 2024-09-13T11:26:56.048 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.2361999999999999e-05 2024-09-13T11:26:56.048 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:56.049 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:56.049 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:26:56.049 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:56.039735+0000", 2024-09-13T11:26:56.049 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00010340099999999999 2024-09-13T11:26:56.049 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:56.049 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:56.049 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 0", 2024-09-13T11:26:56.049 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:56.039800+0000", 2024-09-13T11:26:56.049 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.4134000000000006e-05 2024-09-13T11:26:56.049 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:56.049 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:26:56.049 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:56.049 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:56.049 INFO:teuthology.orchestra.run.smithi162.stdout: ], 2024-09-13T11:26:56.050 INFO:teuthology.orchestra.run.smithi162.stdout: "num_ops": 2 2024-09-13T11:26:56.050 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-13T11:26:56.062 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-09-13T11:26:56.096 INFO:tasks.ceph.osd.8.smithi001.stderr:2024-09-13T11:26:56.094+0000 7fb9b48ae640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26312) UID: 0 2024-09-13T11:26:56.197 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-13T11:26:56.194+0000 7f6a68775640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26307) UID: 0 2024-09-13T11:26:56.199 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-13T11:26:56.199 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [], 2024-09-13T11:26:56.199 INFO:teuthology.orchestra.run.smithi162.stdout: "complaint_time": 30, 2024-09-13T11:26:56.199 INFO:teuthology.orchestra.run.smithi162.stdout: "num_blocked_ops": 0 2024-09-13T11:26:56.199 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-13T11:26:56.217 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-09-13T11:26:56.300 INFO:tasks.ceph.osd.4.smithi001.stderr:2024-09-13T11:26:56.294+0000 7ff8e0218640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26310) UID: 0 2024-09-13T11:26:56.354 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-13T11:26:56.354 INFO:teuthology.orchestra.run.smithi162.stdout: "size": 20, 2024-09-13T11:26:56.354 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 600, 2024-09-13T11:26:56.355 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [ 2024-09-13T11:26:56.355 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:56.355 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:45810 3.17 3.9bd13e57 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e69)", 2024-09-13T11:26:56.355 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:21.347960+0000", 2024-09-13T11:26:56.355 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 215.00524391499999, 2024-09-13T11:26:56.355 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.56814464200000003, 2024-09-13T11:26:56.355 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:26:56.355 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:26:56.355 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-09-13T11:26:56.355 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:26:56.355 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:26:56.355 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:56.355 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 45810 2024-09-13T11:26:56.356 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:56.356 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:26:56.356 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:56.356 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:26:56.356 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:21.347960+0000", 2024-09-13T11:26:56.356 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:56.356 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:56.356 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:56.356 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:26:56.356 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:21.347960+0000", 2024-09-13T11:26:56.356 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:56.356 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:56.356 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:56.357 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:26:56.357 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:21.347965+0000", 2024-09-13T11:26:56.357 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.6720000000000003e-06 2024-09-13T11:26:56.357 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:56.357 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:56.357 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:26:56.357 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:21.348019+0000", 2024-09-13T11:26:56.357 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.4391000000000002e-05 2024-09-13T11:26:56.357 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:56.357 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:56.357 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:26:56.357 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:21.348052+0000", 2024-09-13T11:26:56.357 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.2246000000000003e-05 2024-09-13T11:26:56.357 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:56.358 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:56.358 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:26:56.358 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:21.348072+0000", 2024-09-13T11:26:56.358 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.0622e-05 2024-09-13T11:26:56.358 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:56.358 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:56.358 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:26:56.358 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:21.348278+0000", 2024-09-13T11:26:56.358 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000205391 2024-09-13T11:26:56.358 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:56.358 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:56.358 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-09-13T11:26:56.358 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:21.348292+0000", 2024-09-13T11:26:56.358 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4759e-05 2024-09-13T11:26:56.359 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:56.359 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:56.359 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:26:56.359 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:21.916064+0000", 2024-09-13T11:26:56.359 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.56777128600000004 2024-09-13T11:26:56.359 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:56.359 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:56.359 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:26:56.359 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:21.916105+0000", 2024-09-13T11:26:56.359 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.1275e-05 2024-09-13T11:26:56.359 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:56.359 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:26:56.359 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:56.359 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:56.360 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:56.360 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:54653 3.23 3:c7a54800:::benchmark_data_smithi177_31116_object54652:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e75)", 2024-09-13T11:26:56.360 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:30.979059+0000", 2024-09-13T11:26:56.360 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 205.374145416, 2024-09-13T11:26:56.360 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.97284747100000002, 2024-09-13T11:26:56.360 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:26:56.360 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:26:56.360 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-09-13T11:26:56.360 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:26:56.360 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:26:56.360 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:56.360 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 54653 2024-09-13T11:26:56.360 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:56.360 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:26:56.361 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:56.361 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:26:56.361 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.979059+0000", 2024-09-13T11:26:56.361 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:56.361 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:56.361 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:56.361 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:26:56.361 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.979059+0000", 2024-09-13T11:26:56.361 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:56.361 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:56.361 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:56.361 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:26:56.361 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.979061+0000", 2024-09-13T11:26:56.361 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.4279999999999999e-06 2024-09-13T11:26:56.362 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:56.362 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:56.362 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:26:56.362 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.979114+0000", 2024-09-13T11:26:56.362 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.3254000000000002e-05 2024-09-13T11:26:56.362 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:56.362 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:56.362 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:26:56.362 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.979157+0000", 2024-09-13T11:26:56.362 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.2018000000000003e-05 2024-09-13T11:26:56.362 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:56.362 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:56.362 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:26:56.362 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.979172+0000", 2024-09-13T11:26:56.363 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5636000000000001e-05 2024-09-13T11:26:56.363 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:56.363 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:56.363 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:26:56.363 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:31.045109+0000", 2024-09-13T11:26:56.363 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.065936982000000005 2024-09-13T11:26:56.363 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:56.363 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:56.363 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-09-13T11:26:56.363 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:31.045121+0000", 2024-09-13T11:26:56.363 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1608e-05 2024-09-13T11:26:56.363 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:56.363 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:56.363 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:26:56.364 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:31.951636+0000", 2024-09-13T11:26:56.364 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.90651503899999997 2024-09-13T11:26:56.364 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:56.364 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:56.364 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:26:56.364 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:31.951906+0000", 2024-09-13T11:26:56.364 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000270506 2024-09-13T11:26:56.364 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:56.364 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:26:56.364 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:56.364 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:56.365 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:56.365 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPush(3.18 76/70 [PushOp(3:1a32c9cd:::benchmark_data_smithi177_31116_object12833:head, version: 49'818, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:1a32c9cd:::benchmark_data_smithi177_31116_object12833:head@49'818, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:26:56.365 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:32.522305+0000", 2024-09-13T11:26:56.365 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 203.830899332, 2024-09-13T11:26:56.365 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.40201207100000003, 2024-09-13T11:26:56.365 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:26:56.365 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:26:56.365 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:26:56.365 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:26:56.365 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:56.365 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:26:56.365 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.522305+0000", 2024-09-13T11:26:56.365 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:56.366 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:56.366 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:56.366 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:26:56.366 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.522305+0000", 2024-09-13T11:26:56.366 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:56.366 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:56.366 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:56.366 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:26:56.366 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.522309+0000", 2024-09-13T11:26:56.366 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.9829999999999998e-06 2024-09-13T11:26:56.366 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:56.366 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:56.366 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:26:56.366 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.522377+0000", 2024-09-13T11:26:56.367 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.8069000000000002e-05 2024-09-13T11:26:56.367 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:56.367 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:56.367 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:26:56.367 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.921950+0000", 2024-09-13T11:26:56.367 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.399573438 2024-09-13T11:26:56.367 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:56.367 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:56.367 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:26:56.367 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.921983+0000", 2024-09-13T11:26:56.367 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.2858000000000003e-05 2024-09-13T11:26:56.367 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:56.368 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:56.368 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:26:56.368 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.922389+0000", 2024-09-13T11:26:56.368 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00040612999999999999 2024-09-13T11:26:56.368 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:56.368 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:56.368 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:26:56.368 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.922421+0000", 2024-09-13T11:26:56.368 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.1356999999999998e-05 2024-09-13T11:26:56.368 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:56.368 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:56.368 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:26:56.368 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.924317+0000", 2024-09-13T11:26:56.369 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0018962359999999999 2024-09-13T11:26:56.369 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:56.369 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:26:56.369 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:56.369 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:56.369 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:56.369 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:57435 3.3 3:c0db4777:::benchmark_data_smithi177_31116_object57434:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e80)", 2024-09-13T11:26:56.369 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:37.436034+0000", 2024-09-13T11:26:56.369 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 198.91717042400001, 2024-09-13T11:26:56.369 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.35949142000000001, 2024-09-13T11:26:56.369 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:26:56.369 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:26:56.369 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:56.370 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:26:56.370 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:26:56.370 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:56.370 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 57435 2024-09-13T11:26:56.370 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:56.370 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:26:56.370 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:56.370 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:26:56.370 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:37.436034+0000", 2024-09-13T11:26:56.370 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:56.370 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:56.370 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:56.370 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:26:56.370 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:37.436034+0000", 2024-09-13T11:26:56.371 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:56.371 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:56.371 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:56.371 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:26:56.371 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:37.436037+0000", 2024-09-13T11:26:56.371 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.7829999999999999e-06 2024-09-13T11:26:56.371 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:56.371 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:56.371 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:26:56.372 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:37.436090+0000", 2024-09-13T11:26:56.372 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.3665999999999997e-05 2024-09-13T11:26:56.372 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:56.372 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:56.372 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:26:56.372 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:37.436112+0000", 2024-09-13T11:26:56.372 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.1556000000000002e-05 2024-09-13T11:26:56.372 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:56.372 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:56.372 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:26:56.372 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:37.436122+0000", 2024-09-13T11:26:56.372 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0098000000000001e-05 2024-09-13T11:26:56.372 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:56.372 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:56.373 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:26:56.373 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:37.436187+0000", 2024-09-13T11:26:56.373 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.4857999999999995e-05 2024-09-13T11:26:56.373 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:56.373 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:56.373 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:26:56.373 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:37.436285+0000", 2024-09-13T11:26:56.373 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.8643999999999999e-05 2024-09-13T11:26:56.373 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:56.373 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:56.373 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 7", 2024-09-13T11:26:56.373 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:37.436423+0000", 2024-09-13T11:26:56.373 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00013809299999999999 2024-09-13T11:26:56.373 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:56.374 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:56.374 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:26:56.374 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:37.440125+0000", 2024-09-13T11:26:56.374 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0037015239999999999 2024-09-13T11:26:56.374 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:56.374 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:56.374 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:56.374 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:37.795397+0000", 2024-09-13T11:26:56.374 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.35527234400000002 2024-09-13T11:26:56.374 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:56.374 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:56.374 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:26:56.374 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:37.795485+0000", 2024-09-13T11:26:56.375 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.7538000000000001e-05 2024-09-13T11:26:56.375 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:56.375 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:56.375 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:26:56.375 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:37.795525+0000", 2024-09-13T11:26:56.375 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.0315999999999998e-05 2024-09-13T11:26:56.375 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:56.375 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:26:56.375 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:56.375 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:56.375 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:56.375 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:60433 3.3 3:c01f7ddc:::benchmark_data_smithi177_31116_object60432:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e85)", 2024-09-13T11:26:56.375 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:42.199382+0000", 2024-09-13T11:26:56.375 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 194.15382253300001, 2024-09-13T11:26:56.376 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.33918965200000001, 2024-09-13T11:26:56.376 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:26:56.376 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:26:56.376 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:56.376 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:26:56.376 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:26:56.376 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:56.376 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 60433 2024-09-13T11:26:56.376 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:56.376 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:26:56.376 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:56.376 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:26:56.376 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.199382+0000", 2024-09-13T11:26:56.376 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:56.377 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:56.377 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:56.377 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:26:56.377 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.199382+0000", 2024-09-13T11:26:56.377 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:56.377 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:56.377 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:56.377 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:26:56.377 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.199386+0000", 2024-09-13T11:26:56.377 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.8759999999999998e-06 2024-09-13T11:26:56.377 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:56.377 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:56.377 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:26:56.377 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.199455+0000", 2024-09-13T11:26:56.378 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.9305000000000001e-05 2024-09-13T11:26:56.378 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:56.378 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:56.378 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:26:56.378 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.199482+0000", 2024-09-13T11:26:56.378 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.6749000000000001e-05 2024-09-13T11:26:56.378 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:56.378 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:56.378 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:26:56.378 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.199503+0000", 2024-09-13T11:26:56.378 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.1137000000000001e-05 2024-09-13T11:26:56.378 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:56.378 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:56.378 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:26:56.379 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.199602+0000", 2024-09-13T11:26:56.379 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.8743e-05 2024-09-13T11:26:56.379 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:56.379 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:56.379 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:26:56.379 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.199784+0000", 2024-09-13T11:26:56.379 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00018265100000000001 2024-09-13T11:26:56.379 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:56.379 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:56.379 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 7", 2024-09-13T11:26:56.379 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.199955+0000", 2024-09-13T11:26:56.379 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00017067100000000001 2024-09-13T11:26:56.379 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:56.379 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:56.380 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:26:56.380 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.202075+0000", 2024-09-13T11:26:56.380 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0021206749999999998 2024-09-13T11:26:56.380 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:56.380 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:56.380 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:56.380 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.538395+0000", 2024-09-13T11:26:56.380 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.33631963599999998 2024-09-13T11:26:56.380 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:56.380 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:56.380 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:26:56.380 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.538498+0000", 2024-09-13T11:26:56.380 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00010267099999999999 2024-09-13T11:26:56.380 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:56.380 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:56.381 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:26:56.381 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.538571+0000", 2024-09-13T11:26:56.381 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.3538e-05 2024-09-13T11:26:56.381 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:56.381 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:26:56.381 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:56.381 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:56.381 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:56.381 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop_reply(client.4665.0:60433 3.3 e85/75 ondisk, result = 0)", 2024-09-13T11:26:56.381 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:42.203186+0000", 2024-09-13T11:26:56.381 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 194.15001835499999, 2024-09-13T11:26:56.381 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.33540096699999999, 2024-09-13T11:26:56.381 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:26:56.381 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:26:56.382 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:26:56.382 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:26:56.382 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:56.382 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:26:56.382 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.203186+0000", 2024-09-13T11:26:56.382 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:56.382 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:56.382 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:56.382 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:26:56.382 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.203186+0000", 2024-09-13T11:26:56.382 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:56.382 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:56.382 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:56.383 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:26:56.383 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.203194+0000", 2024-09-13T11:26:56.383 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.0020000000000006e-06 2024-09-13T11:26:56.383 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:56.383 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:56.383 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:26:56.383 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.203202+0000", 2024-09-13T11:26:56.383 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.9829999999999996e-06 2024-09-13T11:26:56.383 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:56.383 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:56.383 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:26:56.383 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.538182+0000", 2024-09-13T11:26:56.383 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.33498029899999998 2024-09-13T11:26:56.383 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:56.384 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:56.384 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:26:56.384 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.538196+0000", 2024-09-13T11:26:56.384 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3807e-05 2024-09-13T11:26:56.384 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:56.384 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:56.384 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:26:56.384 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.538371+0000", 2024-09-13T11:26:56.384 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00017541200000000001 2024-09-13T11:26:56.385 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:56.385 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:56.385 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:26:56.385 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.538386+0000", 2024-09-13T11:26:56.385 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4833e-05 2024-09-13T11:26:56.385 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:56.385 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:56.385 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:26:56.385 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.538587+0000", 2024-09-13T11:26:56.385 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000200631 2024-09-13T11:26:56.385 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:56.385 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:26:56.385 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:56.385 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:56.386 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:56.386 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:83144 3.18 3:1c2a5085:::benchmark_data_smithi177_31116_object83143:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106)", 2024-09-13T11:26:56.386 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:24:04.186395+0000", 2024-09-13T11:26:56.386 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 172.16680923600001, 2024-09-13T11:26:56.386 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.29655527599999998, 2024-09-13T11:26:56.386 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:26:56.386 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:26:56.386 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:56.386 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:26:56.386 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:26:56.386 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:56.386 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 83144 2024-09-13T11:26:56.386 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:56.386 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:26:56.387 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:56.387 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:26:56.387 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.186395+0000", 2024-09-13T11:26:56.387 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:56.387 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:56.387 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:56.387 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:26:56.387 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.186395+0000", 2024-09-13T11:26:56.387 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:56.387 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:56.387 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:56.387 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:26:56.387 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.186398+0000", 2024-09-13T11:26:56.387 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.5299999999999999e-06 2024-09-13T11:26:56.388 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:56.388 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:56.388 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:26:56.388 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.186438+0000", 2024-09-13T11:26:56.388 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.0293000000000002e-05 2024-09-13T11:26:56.388 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:56.388 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:56.388 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:26:56.388 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.186450+0000", 2024-09-13T11:26:56.388 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2687000000000001e-05 2024-09-13T11:26:56.388 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:56.388 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:56.388 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:26:56.389 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.186456+0000", 2024-09-13T11:26:56.389 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.446e-06 2024-09-13T11:26:56.389 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:56.389 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:56.389 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:26:56.389 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.186503+0000", 2024-09-13T11:26:56.389 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.7491000000000003e-05 2024-09-13T11:26:56.389 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:56.389 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:56.389 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:26:56.389 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.186618+0000", 2024-09-13T11:26:56.389 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000114999 2024-09-13T11:26:56.389 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:56.389 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:56.390 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 0", 2024-09-13T11:26:56.390 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.186712+0000", 2024-09-13T11:26:56.390 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.3805000000000001e-05 2024-09-13T11:26:56.390 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:56.390 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:56.390 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:26:56.390 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.189753+0000", 2024-09-13T11:26:56.390 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.003040713 2024-09-13T11:26:56.390 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:56.390 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:56.390 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:56.390 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.482662+0000", 2024-09-13T11:26:56.390 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.29290945699999998 2024-09-13T11:26:56.390 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:56.391 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:56.391 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:26:56.391 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.482831+0000", 2024-09-13T11:26:56.391 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00016900699999999999 2024-09-13T11:26:56.391 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:56.391 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:56.391 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:26:56.391 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.482950+0000", 2024-09-13T11:26:56.391 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000118848 2024-09-13T11:26:56.391 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:56.391 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:26:56.392 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:56.392 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:56.392 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:56.392 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop_reply(client.4665.0:83144 3.18 e106/91 ondisk, result = 0)", 2024-09-13T11:26:56.392 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:24:04.188413+0000", 2024-09-13T11:26:56.392 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 172.16479122600001, 2024-09-13T11:26:56.392 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.294557562, 2024-09-13T11:26:56.392 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:26:56.393 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:26:56.393 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:26:56.393 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:26:56.393 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:56.393 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:26:56.393 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.188413+0000", 2024-09-13T11:26:56.393 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:56.393 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:56.393 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:56.393 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:26:56.393 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.188413+0000", 2024-09-13T11:26:56.393 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:56.393 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:56.394 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:56.394 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:26:56.394 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.188414+0000", 2024-09-13T11:26:56.394 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4309999999999999e-06 2024-09-13T11:26:56.394 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:56.394 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:56.394 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:26:56.394 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.188419+0000", 2024-09-13T11:26:56.394 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.2889999999999999e-06 2024-09-13T11:26:56.394 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:56.394 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:56.394 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:26:56.394 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.482442+0000", 2024-09-13T11:26:56.395 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.294023168 2024-09-13T11:26:56.395 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:56.395 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:56.395 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:26:56.395 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.482459+0000", 2024-09-13T11:26:56.395 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.7013000000000001e-05 2024-09-13T11:26:56.395 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:56.395 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:56.395 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:26:56.395 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.482615+0000", 2024-09-13T11:26:56.395 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00015620400000000001 2024-09-13T11:26:56.395 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:56.395 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:56.395 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:26:56.396 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.482644+0000", 2024-09-13T11:26:56.396 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.9093000000000001e-05 2024-09-13T11:26:56.396 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:56.396 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:56.396 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:26:56.396 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.482971+0000", 2024-09-13T11:26:56.396 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00032636399999999999 2024-09-13T11:26:56.396 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:56.396 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:26:56.396 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:56.396 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:56.396 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:56.397 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPush(3.1f 106/78 [PushOp(3:fd4be819:::benchmark_data_smithi177_31116_object54560:head, version: 74'3410, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:fd4be819:::benchmark_data_smithi177_31116_object54560:head@74'3410, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:26:56.397 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:24:04.397161+0000", 2024-09-13T11:26:56.397 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 171.956042817, 2024-09-13T11:26:56.397 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.51611096199999995, 2024-09-13T11:26:56.397 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:26:56.397 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:26:56.397 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:26:56.397 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:26:56.397 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:56.397 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:26:56.397 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.397161+0000", 2024-09-13T11:26:56.397 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:56.397 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:56.398 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:56.398 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:26:56.398 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.397161+0000", 2024-09-13T11:26:56.398 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:56.398 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:56.398 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:56.398 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:26:56.398 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.397163+0000", 2024-09-13T11:26:56.398 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4720000000000001e-06 2024-09-13T11:26:56.398 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:56.398 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:56.398 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:26:56.398 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.397228+0000", 2024-09-13T11:26:56.398 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.5141999999999994e-05 2024-09-13T11:26:56.398 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:56.399 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:56.399 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:26:56.399 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.911583+0000", 2024-09-13T11:26:56.399 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.51435473700000001 2024-09-13T11:26:56.399 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:56.399 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:56.399 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:26:56.399 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.911615+0000", 2024-09-13T11:26:56.399 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.2620000000000003e-05 2024-09-13T11:26:56.399 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:56.399 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:56.399 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:26:56.399 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.911759+0000", 2024-09-13T11:26:56.399 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00014376300000000001 2024-09-13T11:26:56.400 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:56.400 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:56.400 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:26:56.400 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.911779+0000", 2024-09-13T11:26:56.400 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.0003000000000001e-05 2024-09-13T11:26:56.400 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:56.400 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:56.400 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:26:56.400 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.913272+0000", 2024-09-13T11:26:56.400 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0014932249999999999 2024-09-13T11:26:56.400 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:56.400 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:26:56.400 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:56.401 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:56.401 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:56.401 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPush(3.2a 121/120 [PushOp(3:5595108b:::benchmark_data_smithi177_31116_object899:head, version: 43'62, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:5595108b:::benchmark_data_smithi177_31116_object899:head@43'62, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:26:56.401 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:24:27.352355+0000", 2024-09-13T11:26:56.401 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 149.00084952200001, 2024-09-13T11:26:56.401 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.61628666099999996, 2024-09-13T11:26:56.401 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:26:56.401 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:26:56.401 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:26:56.401 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:26:56.401 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:56.401 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:26:56.401 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:27.352355+0000", 2024-09-13T11:26:56.402 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:56.402 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:56.402 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:56.402 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:26:56.402 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:27.352355+0000", 2024-09-13T11:26:56.402 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:56.402 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:56.402 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:56.402 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:26:56.402 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:27.352357+0000", 2024-09-13T11:26:56.402 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.328e-06 2024-09-13T11:26:56.402 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:56.402 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:56.402 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:26:56.403 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:27.352428+0000", 2024-09-13T11:26:56.403 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.1118999999999995e-05 2024-09-13T11:26:56.403 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:56.403 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:56.403 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:26:56.403 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:27.966950+0000", 2024-09-13T11:26:56.403 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.61452202499999997 2024-09-13T11:26:56.403 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:56.403 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:56.403 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:26:56.403 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:27.966980+0000", 2024-09-13T11:26:56.403 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.9433000000000001e-05 2024-09-13T11:26:56.403 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:56.403 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:56.404 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:26:56.404 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:27.967220+0000", 2024-09-13T11:26:56.404 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00024065499999999999 2024-09-13T11:26:56.404 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:56.404 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:56.404 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:26:56.404 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:27.967244+0000", 2024-09-13T11:26:56.404 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.4216e-05 2024-09-13T11:26:56.404 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:56.404 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:56.404 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:26:56.404 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:27.968641+0000", 2024-09-13T11:26:56.404 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0013968850000000001 2024-09-13T11:26:56.405 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:56.405 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:26:56.405 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:56.405 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:56.405 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:56.405 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:162073 3.1f 3:fd1a00ae:::benchmark_data_smithi177_31116_object21770:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e156)", 2024-09-13T11:26:56.405 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:25:15.454353+0000", 2024-09-13T11:26:56.405 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 100.898850924, 2024-09-13T11:26:56.405 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.27516647, 2024-09-13T11:26:56.405 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:26:56.405 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:26:56.405 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:56.405 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:26:56.406 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:26:56.406 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:56.406 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 162073 2024-09-13T11:26:56.406 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:56.406 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:26:56.406 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:56.406 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:26:56.406 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:15.454353+0000", 2024-09-13T11:26:56.406 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:56.406 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:56.406 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:56.407 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:26:56.407 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:15.454353+0000", 2024-09-13T11:26:56.407 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:56.407 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:56.407 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:56.407 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:26:56.407 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:15.454358+0000", 2024-09-13T11:26:56.407 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.3499999999999999e-06 2024-09-13T11:26:56.407 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:56.407 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:56.407 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:26:56.407 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:15.454371+0000", 2024-09-13T11:26:56.407 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3594000000000001e-05 2024-09-13T11:26:56.408 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:56.408 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:56.408 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:26:56.408 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:15.454408+0000", 2024-09-13T11:26:56.408 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.6619999999999998e-05 2024-09-13T11:26:56.408 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:56.408 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:56.408 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:26:56.408 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:15.454431+0000", 2024-09-13T11:26:56.408 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.3546000000000001e-05 2024-09-13T11:26:56.408 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:56.408 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:56.408 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:26:56.409 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:15.454587+0000", 2024-09-13T11:26:56.409 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000155248 2024-09-13T11:26:56.409 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:56.409 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:56.409 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:26:56.409 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:15.454897+0000", 2024-09-13T11:26:56.409 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000309955 2024-09-13T11:26:56.409 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:56.409 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:56.409 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 5", 2024-09-13T11:26:56.409 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:15.455203+0000", 2024-09-13T11:26:56.409 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00030604999999999999 2024-09-13T11:26:56.409 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:56.410 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:56.410 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:26:56.410 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:15.460534+0000", 2024-09-13T11:26:56.410 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0053310199999999997 2024-09-13T11:26:56.410 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:56.410 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:56.410 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:56.410 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:15.729356+0000", 2024-09-13T11:26:56.410 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.26882267199999998 2024-09-13T11:26:56.410 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:56.410 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:56.410 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:26:56.410 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:15.729442+0000", 2024-09-13T11:26:56.410 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.5209000000000004e-05 2024-09-13T11:26:56.411 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:56.411 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:56.411 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:26:56.411 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:15.729520+0000", 2024-09-13T11:26:56.411 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.8206000000000003e-05 2024-09-13T11:26:56.411 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:56.411 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:26:56.411 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:56.411 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:56.411 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:56.411 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:162102 3.1f 3:fc0cda6c:::benchmark_data_smithi177_31116_object21799:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e156)", 2024-09-13T11:26:56.411 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:25:15.477044+0000", 2024-09-13T11:26:56.411 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 100.87616007, 2024-09-13T11:26:56.411 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.28798895800000002, 2024-09-13T11:26:56.412 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:26:56.412 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:26:56.412 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:56.412 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:26:56.412 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:26:56.412 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:56.412 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 162102 2024-09-13T11:26:56.412 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:56.412 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:26:56.412 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:56.412 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:26:56.412 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:15.477044+0000", 2024-09-13T11:26:56.412 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:56.413 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:56.413 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:56.413 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:26:56.413 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:15.477044+0000", 2024-09-13T11:26:56.413 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:56.413 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:56.413 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:56.413 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:26:56.413 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:15.477048+0000", 2024-09-13T11:26:56.413 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.3569999999999998e-06 2024-09-13T11:26:56.413 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:56.413 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:56.413 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:26:56.413 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:15.477058+0000", 2024-09-13T11:26:56.414 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.9899999999999992e-06 2024-09-13T11:26:56.414 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:56.414 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:56.414 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:26:56.414 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:15.477087+0000", 2024-09-13T11:26:56.414 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.9026999999999998e-05 2024-09-13T11:26:56.414 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:56.414 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:56.414 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:26:56.414 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:15.477101+0000", 2024-09-13T11:26:56.414 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4583000000000001e-05 2024-09-13T11:26:56.414 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:56.414 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:56.414 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:26:56.415 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:15.477212+0000", 2024-09-13T11:26:56.415 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00011071200000000001 2024-09-13T11:26:56.415 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:56.415 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:56.415 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:26:56.415 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:15.477492+0000", 2024-09-13T11:26:56.415 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00027995400000000001 2024-09-13T11:26:56.415 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:56.415 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:56.415 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 5", 2024-09-13T11:26:56.415 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:15.477682+0000", 2024-09-13T11:26:56.415 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000189929 2024-09-13T11:26:56.415 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:56.416 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:56.416 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:26:56.416 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:15.481903+0000", 2024-09-13T11:26:56.416 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0042214649999999998 2024-09-13T11:26:56.416 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:56.416 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:56.416 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:56.416 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:15.764891+0000", 2024-09-13T11:26:56.416 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.28298753599999998 2024-09-13T11:26:56.416 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:56.416 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:56.416 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:26:56.416 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:15.764972+0000", 2024-09-13T11:26:56.416 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.1230000000000007e-05 2024-09-13T11:26:56.417 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:56.417 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:56.417 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:26:56.417 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:15.765033+0000", 2024-09-13T11:26:56.417 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.1174999999999996e-05 2024-09-13T11:26:56.417 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:56.417 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:26:56.417 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:56.417 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:56.417 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:56.417 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:221532 3.17 3:ec6fb681:::benchmark_data_smithi177_31116_object81229:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e191)", 2024-09-13T11:26:56.417 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:26:04.303286+0000", 2024-09-13T11:26:56.417 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 52.049918323, 2024-09-13T11:26:56.418 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.79092058600000004, 2024-09-13T11:26:56.418 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:26:56.418 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:26:56.418 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:56.418 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:26:56.418 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:26:56.418 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:56.418 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 221532 2024-09-13T11:26:56.418 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:56.418 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:26:56.418 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:56.418 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:26:56.418 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:04.303286+0000", 2024-09-13T11:26:56.418 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:56.418 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:56.419 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:56.419 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:26:56.419 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:04.303286+0000", 2024-09-13T11:26:56.419 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:56.419 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:56.419 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:56.419 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:26:56.419 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:04.303290+0000", 2024-09-13T11:26:56.419 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.9600000000000002e-06 2024-09-13T11:26:56.419 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:56.419 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:56.419 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:26:56.419 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:04.303302+0000", 2024-09-13T11:26:56.419 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2126e-05 2024-09-13T11:26:56.420 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:56.420 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:56.420 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:26:56.420 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:04.303332+0000", 2024-09-13T11:26:56.420 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.0015999999999999e-05 2024-09-13T11:26:56.420 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:56.420 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:56.420 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:26:56.420 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:04.303347+0000", 2024-09-13T11:26:56.420 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5324000000000001e-05 2024-09-13T11:26:56.420 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:56.420 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:56.420 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:26:56.420 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:04.303484+0000", 2024-09-13T11:26:56.421 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00013702199999999999 2024-09-13T11:26:56.421 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:56.421 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:56.421 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:26:56.421 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:04.303700+0000", 2024-09-13T11:26:56.421 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00021570500000000001 2024-09-13T11:26:56.421 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:56.421 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:56.421 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 3", 2024-09-13T11:26:56.421 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:04.303868+0000", 2024-09-13T11:26:56.421 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00016751799999999999 2024-09-13T11:26:56.421 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:56.421 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:56.421 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:26:56.422 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:04.309489+0000", 2024-09-13T11:26:56.422 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.005621455 2024-09-13T11:26:56.422 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:56.422 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:56.422 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:56.422 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:05.094014+0000", 2024-09-13T11:26:56.422 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.78452480499999999 2024-09-13T11:26:56.422 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:56.422 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:56.422 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:26:56.422 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:05.094132+0000", 2024-09-13T11:26:56.422 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000118093 2024-09-13T11:26:56.422 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:56.422 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:56.423 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:26:56.423 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:05.094206+0000", 2024-09-13T11:26:56.423 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.4561999999999995e-05 2024-09-13T11:26:56.423 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:56.423 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:26:56.423 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:56.423 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:56.423 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:56.423 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop_reply(client.4665.0:221532 3.17 e191/85 ondisk, result = 0)", 2024-09-13T11:26:56.423 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:26:04.309386+0000", 2024-09-13T11:26:56.423 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 52.043818020000003, 2024-09-13T11:26:56.423 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.78483394299999998, 2024-09-13T11:26:56.423 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:26:56.424 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:26:56.424 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:26:56.424 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:26:56.424 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:56.424 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:26:56.424 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:04.309386+0000", 2024-09-13T11:26:56.424 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:56.424 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:56.424 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:56.424 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:26:56.424 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:04.309386+0000", 2024-09-13T11:26:56.424 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:56.424 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:56.425 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:56.425 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:26:56.425 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:04.309390+0000", 2024-09-13T11:26:56.425 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.5700000000000001e-06 2024-09-13T11:26:56.425 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:56.425 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:56.425 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:26:56.425 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:04.309403+0000", 2024-09-13T11:26:56.425 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3567000000000001e-05 2024-09-13T11:26:56.425 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:56.425 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:56.425 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:26:56.425 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:05.093783+0000", 2024-09-13T11:26:56.425 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.78437944699999995 2024-09-13T11:26:56.426 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:56.426 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:56.426 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:26:56.426 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:05.093802+0000", 2024-09-13T11:26:56.426 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.8816000000000001e-05 2024-09-13T11:26:56.426 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:56.426 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:56.426 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:26:56.426 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:05.093968+0000", 2024-09-13T11:26:56.426 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000166572 2024-09-13T11:26:56.426 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:56.426 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:56.426 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:26:56.427 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:05.093999+0000", 2024-09-13T11:26:56.427 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.0352999999999999e-05 2024-09-13T11:26:56.427 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:56.427 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:56.427 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:26:56.427 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:05.094220+0000", 2024-09-13T11:26:56.427 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000221618 2024-09-13T11:26:56.427 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:56.427 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:26:56.427 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:56.427 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:56.427 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:56.427 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPushReply(3.1f 219/192 [PushReplyOp(3:f85f8546:::benchmark_data_smithi177_31116_object127445:head)])", 2024-09-13T11:26:56.428 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:26:42.440545+0000", 2024-09-13T11:26:56.428 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 13.912659469999999, 2024-09-13T11:26:56.428 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.49957432099999999, 2024-09-13T11:26:56.428 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:26:56.428 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:26:56.428 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-09-13T11:26:56.428 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:26:56.428 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:56.428 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:26:56.428 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:42.440545+0000", 2024-09-13T11:26:56.428 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:56.428 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:56.428 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:56.429 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:26:56.429 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:42.440545+0000", 2024-09-13T11:26:56.429 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:56.429 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:56.429 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:56.429 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:26:56.429 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:42.440545+0000", 2024-09-13T11:26:56.429 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.4300000000000003e-07 2024-09-13T11:26:56.429 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:56.429 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:56.429 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:26:56.429 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:42.440552+0000", 2024-09-13T11:26:56.429 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.4829999999999996e-06 2024-09-13T11:26:56.429 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:56.430 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:56.430 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:26:56.430 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:42.939600+0000", 2024-09-13T11:26:56.430 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.49904801300000001 2024-09-13T11:26:56.430 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:56.430 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:56.430 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:26:56.430 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:42.939620+0000", 2024-09-13T11:26:56.430 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.0305999999999998e-05 2024-09-13T11:26:56.430 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:56.430 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:56.430 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:26:56.430 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:42.939773+0000", 2024-09-13T11:26:56.430 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00015300599999999999 2024-09-13T11:26:56.431 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:56.431 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:56.431 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:26:56.431 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:42.940119+0000", 2024-09-13T11:26:56.431 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00034597000000000002 2024-09-13T11:26:56.431 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:56.431 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:26:56.431 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:56.431 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:56.431 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:56.431 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:266710 3.1f 3:fa3f4753:::benchmark_data_smithi177_31116_object126407:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e223)", 2024-09-13T11:26:56.431 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:26:48.978129+0000", 2024-09-13T11:26:56.431 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 7.3750757150000004, 2024-09-13T11:26:56.431 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.274037694, 2024-09-13T11:26:56.432 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:26:56.432 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:26:56.432 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:56.432 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:26:56.432 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:26:56.432 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:56.432 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 266710 2024-09-13T11:26:56.432 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:56.432 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:26:56.432 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:56.432 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:26:56.432 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:48.978129+0000", 2024-09-13T11:26:56.432 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:56.432 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:56.433 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:56.433 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:26:56.433 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:48.978129+0000", 2024-09-13T11:26:56.433 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:56.433 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:56.433 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:56.433 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:26:56.433 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:48.978130+0000", 2024-09-13T11:26:56.433 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.747e-06 2024-09-13T11:26:56.433 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:56.433 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:56.433 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:26:56.433 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:48.978137+0000", 2024-09-13T11:26:56.433 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.2200000000000003e-06 2024-09-13T11:26:56.434 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:56.434 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:56.434 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:26:56.434 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:48.978152+0000", 2024-09-13T11:26:56.434 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4922999999999999e-05 2024-09-13T11:26:56.434 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:56.434 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:56.434 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:26:56.434 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:48.978165+0000", 2024-09-13T11:26:56.434 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2567e-05 2024-09-13T11:26:56.434 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:56.434 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:56.434 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:26:56.435 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:48.978887+0000", 2024-09-13T11:26:56.435 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00072206699999999996 2024-09-13T11:26:56.435 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:56.435 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:56.435 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-09-13T11:26:56.435 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:48.978904+0000", 2024-09-13T11:26:56.435 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.6657e-05 2024-09-13T11:26:56.435 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:56.435 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:56.435 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:26:56.435 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:49.178833+0000", 2024-09-13T11:26:56.435 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.19992964199999999 2024-09-13T11:26:56.435 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:56.435 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:56.436 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:26:56.436 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:49.178988+0000", 2024-09-13T11:26:56.436 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000154869 2024-09-13T11:26:56.436 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:56.436 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:56.436 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 14", 2024-09-13T11:26:56.436 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:49.179124+0000", 2024-09-13T11:26:56.436 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000135532 2024-09-13T11:26:56.436 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:56.436 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:56.436 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:26:56.436 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:49.194638+0000", 2024-09-13T11:26:56.436 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.015514412 2024-09-13T11:26:56.437 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:56.437 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:56.437 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:56.437 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:49.252059+0000", 2024-09-13T11:26:56.437 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.057420818999999998 2024-09-13T11:26:56.437 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:56.437 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:56.437 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:26:56.437 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:49.252120+0000", 2024-09-13T11:26:56.437 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.1048000000000002e-05 2024-09-13T11:26:56.437 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:56.437 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:56.437 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:26:56.437 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:49.252166+0000", 2024-09-13T11:26:56.437 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.6190999999999999e-05 2024-09-13T11:26:56.438 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:56.438 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:26:56.438 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:56.438 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:56.438 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:56.438 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:266723 3.1f 3:f8daff18:::benchmark_data_smithi177_31116_object126420:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e223)", 2024-09-13T11:26:56.438 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:26:48.978181+0000", 2024-09-13T11:26:56.438 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 7.3750235379999998, 2024-09-13T11:26:56.438 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.27412631599999998, 2024-09-13T11:26:56.438 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:26:56.438 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:26:56.438 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:56.438 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:26:56.439 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:26:56.439 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:56.439 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 266723 2024-09-13T11:26:56.439 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:56.439 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:26:56.439 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:56.439 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:26:56.439 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:48.978181+0000", 2024-09-13T11:26:56.439 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:56.439 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:56.439 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:56.439 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:26:56.439 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:48.978181+0000", 2024-09-13T11:26:56.439 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:56.440 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:56.440 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:56.440 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:26:56.440 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:48.978182+0000", 2024-09-13T11:26:56.440 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.72e-06 2024-09-13T11:26:56.440 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:56.440 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:56.440 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:26:56.440 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:48.978189+0000", 2024-09-13T11:26:56.440 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.9700000000000002e-06 2024-09-13T11:26:56.440 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:56.440 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:56.440 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:26:56.440 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:48.978205+0000", 2024-09-13T11:26:56.441 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5746000000000001e-05 2024-09-13T11:26:56.441 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:56.441 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:56.441 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:26:56.441 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:48.978221+0000", 2024-09-13T11:26:56.441 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5679999999999999e-05 2024-09-13T11:26:56.441 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:56.441 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:56.441 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:26:56.441 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:48.978961+0000", 2024-09-13T11:26:56.441 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0007406 2024-09-13T11:26:56.441 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:56.441 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:56.441 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-09-13T11:26:56.442 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:48.978978+0000", 2024-09-13T11:26:56.442 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.6663e-05 2024-09-13T11:26:56.442 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:56.442 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:56.442 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:26:56.442 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:49.183721+0000", 2024-09-13T11:26:56.442 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.20474339799999999 2024-09-13T11:26:56.442 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:56.442 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:56.442 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:26:56.442 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:49.183856+0000", 2024-09-13T11:26:56.442 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00013502300000000001 2024-09-13T11:26:56.442 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:56.442 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:56.443 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 14", 2024-09-13T11:26:56.443 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:49.183980+0000", 2024-09-13T11:26:56.443 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000123473 2024-09-13T11:26:56.443 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:56.443 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:56.443 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:26:56.443 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:49.194643+0000", 2024-09-13T11:26:56.443 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.010662914000000001 2024-09-13T11:26:56.443 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:56.443 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:56.443 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:56.443 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:49.252225+0000", 2024-09-13T11:26:56.443 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.057581637999999997 2024-09-13T11:26:56.443 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:56.444 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:56.444 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:26:56.444 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:49.252273+0000", 2024-09-13T11:26:56.444 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.8158000000000002e-05 2024-09-13T11:26:56.444 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:56.444 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:56.444 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:26:56.444 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:49.252307+0000", 2024-09-13T11:26:56.444 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.4332999999999998e-05 2024-09-13T11:26:56.444 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:56.444 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:26:56.444 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:56.444 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:56.445 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:56.445 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:266816 3.1f 3:f851e4b5:::benchmark_data_smithi177_31116_object126513:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e223)", 2024-09-13T11:26:56.445 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:26:48.978237+0000", 2024-09-13T11:26:56.445 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 7.3749672029999997, 2024-09-13T11:26:56.445 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.27421219200000002, 2024-09-13T11:26:56.445 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:26:56.445 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:26:56.445 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:56.445 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:26:56.445 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:26:56.445 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:56.445 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 266816 2024-09-13T11:26:56.445 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:56.446 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:26:56.446 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:56.446 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:26:56.446 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:48.978237+0000", 2024-09-13T11:26:56.446 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:56.446 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:56.446 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:56.446 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:26:56.446 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:48.978237+0000", 2024-09-13T11:26:56.446 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:56.446 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:56.446 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:56.447 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:26:56.447 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:48.978239+0000", 2024-09-13T11:26:56.447 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5340000000000001e-06 2024-09-13T11:26:56.447 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:56.447 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:56.447 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:26:56.447 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:48.978245+0000", 2024-09-13T11:26:56.447 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.8360000000000003e-06 2024-09-13T11:26:56.447 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:56.447 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:56.447 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:26:56.447 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:48.978263+0000", 2024-09-13T11:26:56.447 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.7246999999999999e-05 2024-09-13T11:26:56.447 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:56.448 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:56.448 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:26:56.448 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:48.978276+0000", 2024-09-13T11:26:56.448 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.331e-05 2024-09-13T11:26:56.448 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:56.448 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:56.448 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:26:56.448 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:48.979037+0000", 2024-09-13T11:26:56.448 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00076063699999999999 2024-09-13T11:26:56.448 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:56.448 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:56.448 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-09-13T11:26:56.448 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:48.979052+0000", 2024-09-13T11:26:56.449 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5359999999999999e-05 2024-09-13T11:26:56.449 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:56.449 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:56.449 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:26:56.449 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:49.187185+0000", 2024-09-13T11:26:56.449 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.20813316500000001 2024-09-13T11:26:56.449 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:56.449 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:56.449 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:26:56.449 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:49.187317+0000", 2024-09-13T11:26:56.449 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000132364 2024-09-13T11:26:56.449 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:56.449 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:56.449 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 14", 2024-09-13T11:26:56.450 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:49.187439+0000", 2024-09-13T11:26:56.450 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000121999 2024-09-13T11:26:56.450 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:56.450 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:56.450 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:26:56.450 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:49.194646+0000", 2024-09-13T11:26:56.450 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.007206128 2024-09-13T11:26:56.450 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:56.450 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:56.450 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:56.450 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:49.252363+0000", 2024-09-13T11:26:56.450 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.057717888000000002 2024-09-13T11:26:56.450 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:56.450 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:56.450 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:26:56.451 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:49.252415+0000", 2024-09-13T11:26:56.451 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.1379999999999999e-05 2024-09-13T11:26:56.451 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:56.451 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:56.451 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:26:56.451 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:49.252449+0000", 2024-09-13T11:26:56.451 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.4344000000000002e-05 2024-09-13T11:26:56.451 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:56.451 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:26:56.451 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:56.451 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:56.451 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:56.451 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:266861 3.1f 3:feb21dac:::benchmark_data_smithi177_31116_object126558:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e223)", 2024-09-13T11:26:56.451 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:26:48.978290+0000", 2024-09-13T11:26:56.452 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 7.3749138490000004, 2024-09-13T11:26:56.452 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.27428401000000002, 2024-09-13T11:26:56.452 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:26:56.452 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:26:56.452 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:56.452 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:26:56.452 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:26:56.452 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:56.452 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 266861 2024-09-13T11:26:56.452 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:56.452 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:26:56.452 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:56.452 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:26:56.453 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:48.978290+0000", 2024-09-13T11:26:56.453 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:56.453 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:56.453 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:56.453 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:26:56.453 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:48.978290+0000", 2024-09-13T11:26:56.453 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:56.453 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:56.453 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:56.453 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:26:56.453 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:48.978292+0000", 2024-09-13T11:26:56.453 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.266e-06 2024-09-13T11:26:56.453 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:56.454 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:56.454 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:26:56.454 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:48.978301+0000", 2024-09-13T11:26:56.454 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.1200000000000008e-06 2024-09-13T11:26:56.454 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:56.454 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:56.454 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:26:56.454 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:48.978315+0000", 2024-09-13T11:26:56.454 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.452e-05 2024-09-13T11:26:56.454 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:56.454 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:56.454 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:26:56.454 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:48.978328+0000", 2024-09-13T11:26:56.454 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2743e-05 2024-09-13T11:26:56.455 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:56.455 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:56.455 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:26:56.455 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:48.979107+0000", 2024-09-13T11:26:56.455 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00077939999999999997 2024-09-13T11:26:56.455 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:56.455 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:56.455 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-09-13T11:26:56.455 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:48.979125+0000", 2024-09-13T11:26:56.455 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.7977000000000001e-05 2024-09-13T11:26:56.455 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:56.455 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:56.455 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:26:56.455 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:49.191293+0000", 2024-09-13T11:26:56.456 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.212168046 2024-09-13T11:26:56.456 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:56.456 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:56.456 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:26:56.456 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:49.191420+0000", 2024-09-13T11:26:56.456 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00012653100000000001 2024-09-13T11:26:56.456 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:56.456 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:56.456 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 14", 2024-09-13T11:26:56.456 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:49.191522+0000", 2024-09-13T11:26:56.456 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00010239900000000001 2024-09-13T11:26:56.456 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:56.456 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:56.456 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:26:56.457 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:49.194891+0000", 2024-09-13T11:26:56.457 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0033688020000000002 2024-09-13T11:26:56.457 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:56.457 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:56.457 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:56.457 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:49.252493+0000", 2024-09-13T11:26:56.457 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.057601806999999998 2024-09-13T11:26:56.457 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:56.457 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:56.457 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:26:56.457 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:49.252542+0000", 2024-09-13T11:26:56.457 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.8519000000000001e-05 2024-09-13T11:26:56.458 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:56.458 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:56.458 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:26:56.458 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:49.252574+0000", 2024-09-13T11:26:56.458 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.2879999999999997e-05 2024-09-13T11:26:56.458 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:56.458 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:26:56.458 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:56.458 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:56.458 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:56.458 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:266884 3.1f 3:fb121a9e:::benchmark_data_smithi177_31116_object126581:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e224)", 2024-09-13T11:26:56.458 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:26:49.275993+0000", 2024-09-13T11:26:56.458 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 7.0772108779999998, 2024-09-13T11:26:56.458 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.54084032800000004, 2024-09-13T11:26:56.459 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:26:56.459 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:26:56.459 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:56.459 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:26:56.459 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:26:56.459 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:56.459 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 266884 2024-09-13T11:26:56.459 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:56.459 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:26:56.459 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:56.459 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:26:56.459 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:49.275993+0000", 2024-09-13T11:26:56.459 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:56.459 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:56.460 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:56.460 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:26:56.460 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:49.275993+0000", 2024-09-13T11:26:56.460 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:26:56.460 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:56.460 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:56.460 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:26:56.460 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:49.275997+0000", 2024-09-13T11:26:56.460 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.2600000000000001e-06 2024-09-13T11:26:56.460 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:56.460 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:56.460 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:26:56.460 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:49.276005+0000", 2024-09-13T11:26:56.461 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.0779999999999996e-06 2024-09-13T11:26:56.461 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:56.461 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:56.461 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:26:56.461 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:49.276022+0000", 2024-09-13T11:26:56.461 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.6849000000000001e-05 2024-09-13T11:26:56.461 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:56.461 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:56.461 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:26:56.461 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:49.276029+0000", 2024-09-13T11:26:56.461 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.6969999999999993e-06 2024-09-13T11:26:56.461 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:56.461 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:56.461 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:26:56.462 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:49.276084+0000", 2024-09-13T11:26:56.462 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.4928e-05 2024-09-13T11:26:56.462 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:56.462 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:56.462 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:26:56.462 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:49.276181+0000", 2024-09-13T11:26:56.462 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.6834000000000002e-05 2024-09-13T11:26:56.462 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:56.462 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:56.462 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 14", 2024-09-13T11:26:56.462 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:49.276251+0000", 2024-09-13T11:26:56.462 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.0028000000000001e-05 2024-09-13T11:26:56.462 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:56.462 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:56.463 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:26:56.463 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:49.281808+0000", 2024-09-13T11:26:56.463 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0055566770000000003 2024-09-13T11:26:56.463 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:56.463 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:56.463 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:56.463 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:49.816659+0000", 2024-09-13T11:26:56.463 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.53485174899999999 2024-09-13T11:26:56.463 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:56.463 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:56.463 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:26:56.463 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:49.816755+0000", 2024-09-13T11:26:56.463 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.5073000000000003e-05 2024-09-13T11:26:56.464 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:26:56.464 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:26:56.464 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:26:56.464 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:49.816834+0000", 2024-09-13T11:26:56.464 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.9154999999999997e-05 2024-09-13T11:26:56.464 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:56.464 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:26:56.464 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:56.464 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:26:56.464 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:26:56.464 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-13T11:26:56.465 INFO:tasks.ceph.osd.7.smithi177.stderr:2024-09-13T11:26:56.393+0000 7f896ff4a640 -1 osd.7 229 handle_osd_map injecting map crc failure 2024-09-13T11:26:56.466 INFO:tasks.ceph.osd.15.smithi177.stderr:2024-09-13T11:26:56.393+0000 7f9d618a4640 -1 osd.15 229 handle_osd_map injecting map crc failure 2024-09-13T11:26:56.467 INFO:tasks.ceph.osd.15.smithi177.stderr:2024-09-13T11:26:56.393+0000 7f9d618a4640 -1 osd.15 229 handle_osd_map injecting map crc failure 2024-09-13T11:26:56.467 INFO:tasks.ceph.osd.3.smithi177.stderr:2024-09-13T11:26:56.461+0000 7f10adba4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26574) UID: 0 2024-09-13T11:26:56.467 INFO:teuthology.orchestra.run.smithi001.stderr:noscrub is unset 2024-09-13T11:26:56.468 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-13T11:26:56.566 INFO:tasks.ceph.osd.11.smithi177.stderr:2024-09-13T11:26:56.565+0000 7f24792c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26580) UID: 0 2024-09-13T11:26:56.667 INFO:tasks.ceph.osd.4.smithi001.stderr:2024-09-13T11:26:56.666+0000 7ff8e0218640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26310) UID: 0 2024-09-13T11:26:56.673 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-09-13T11:26:56.673 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:26:56.681 INFO:teuthology.orchestra.run.smithi001.stdout:osd.6: {} 2024-09-13T11:26:56.681 INFO:teuthology.orchestra.run.smithi001.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T11:26:56.709 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-13T11:26:56.712 INFO:teuthology.orchestra.run.smithi001.stdout:osd.7: {} 2024-09-13T11:26:56.712 INFO:teuthology.orchestra.run.smithi001.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T11:26:56.740 INFO:teuthology.orchestra.run.smithi001.stdout:osd.8: {} 2024-09-13T11:26:56.741 INFO:teuthology.orchestra.run.smithi001.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-13T11:26:56.768 INFO:tasks.ceph.osd.9.smithi092.stderr:2024-09-13T11:26:56.763+0000 7ff5ab3d0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26348) UID: 0 2024-09-13T11:26:56.777 INFO:teuthology.orchestra.run.smithi001.stdout:osd.9: {} 2024-09-13T11:26:56.777 INFO:teuthology.orchestra.run.smithi001.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-13T11:26:56.807 INFO:teuthology.orchestra.run.smithi001.stdout:osd.10: {} 2024-09-13T11:26:56.807 INFO:teuthology.orchestra.run.smithi001.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-13T11:26:56.840 INFO:teuthology.orchestra.run.smithi001.stdout:osd.11: {} 2024-09-13T11:26:56.841 INFO:teuthology.orchestra.run.smithi001.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-13T11:26:56.868 INFO:teuthology.orchestra.run.smithi001.stdout:osd.12: {} 2024-09-13T11:26:56.869 INFO:teuthology.orchestra.run.smithi001.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-13T11:26:56.870 INFO:tasks.ceph.osd.3.smithi177.stderr:2024-09-13T11:26:56.865+0000 7f10adba4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26574) UID: 0 2024-09-13T11:26:56.897 INFO:teuthology.orchestra.run.smithi001.stdout:osd.13: {} 2024-09-13T11:26:56.897 INFO:teuthology.orchestra.run.smithi001.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-13T11:26:56.912 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-09-13T11:26:56.912 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:26:56.920 INFO:teuthology.orchestra.run.smithi001.stdout:osd.14: {} 2024-09-13T11:26:56.921 INFO:teuthology.orchestra.run.smithi001.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-13T11:26:56.935 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-13T11:26:56.950 INFO:teuthology.orchestra.run.smithi001.stdout:osd.15: {} 2024-09-13T11:26:56.950 INFO:teuthology.orchestra.run.smithi001.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-13T11:26:56.970 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-09-13T11:26:56.967+0000 7f97befc9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26315) UID: 0 2024-09-13T11:26:57.070 INFO:tasks.ceph.osd.2.smithi162.stderr:2024-09-13T11:26:57.067+0000 7fe92b35a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26313) UID: 0 2024-09-13T11:26:57.148 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-09-13T11:26:57.149 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:26:57.168 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-13T11:26:57.171 INFO:tasks.ceph.osd.10.smithi162.stderr:2024-09-13T11:26:57.167+0000 7fb59c9f0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26317) UID: 0 2024-09-13T11:26:57.272 INFO:tasks.ceph.osd.2.smithi162.stderr:2024-09-13T11:26:57.267+0000 7fe92b35a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26313) UID: 0 2024-09-13T11:26:57.325 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-09-13T11:26:57.325 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:26:57.344 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-13T11:26:57.372 INFO:tasks.ceph.osd.12.smithi001.stderr:2024-09-13T11:26:57.370+0000 7f892c6e6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26314) UID: 0 2024-09-13T11:26:57.473 INFO:tasks.ceph.osd.11.smithi177.stderr:2024-09-13T11:26:57.469+0000 7f24792c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26580) UID: 0 2024-09-13T11:26:57.496 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-09-13T11:26:57.496 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:26:57.519 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-13T11:26:57.610 INFO:tasks.ceph.osd.15.smithi177.stderr:2024-09-13T11:26:57.609+0000 7f9d6ce77640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26578) UID: 0 2024-09-13T11:26:57.675 INFO:tasks.ceph.osd.1.smithi092.stderr:2024-09-13T11:26:57.671+0000 7fe7789c5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33718) UID: 0 2024-09-13T11:26:57.776 INFO:tasks.ceph.osd.8.smithi001.stderr:2024-09-13T11:26:57.774+0000 7fb9b48ae640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26312) UID: 0 2024-09-13T11:26:57.783 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-09-13T11:26:57.784 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:26:57.821 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-09-13T11:26:57.879 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-09-13T11:26:57.875+0000 7f97befc9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26315) UID: 0 2024-09-13T11:26:57.978 INFO:tasks.ceph.osd.5.smithi092.stderr:2024-09-13T11:26:57.975+0000 7f6f0b705640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26346) UID: 0 2024-09-13T11:26:58.055 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-09-13T11:26:58.055 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:26:58.080 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-09-13T11:26:58.080 INFO:tasks.ceph.osd.4.smithi001.stderr:2024-09-13T11:26:58.078+0000 7ff8e0218640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26310) UID: 0 2024-09-13T11:26:58.181 INFO:tasks.ceph.osd.9.smithi092.stderr:2024-09-13T11:26:58.179+0000 7ff5ab3d0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26348) UID: 0 2024-09-13T11:26:58.268 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-09-13T11:26:58.268 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:26:58.282 INFO:tasks.ceph.osd.11.smithi177.stderr:2024-09-13T11:26:58.281+0000 7f24792c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26580) UID: 0 2024-09-13T11:26:58.295 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-09-13T11:26:58.382 INFO:tasks.ceph.osd.5.smithi092.stderr:2024-09-13T11:26:58.379+0000 7f6f0b705640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26346) UID: 0 2024-09-13T11:26:58.399 INFO:tasks.ceph.osd.5.smithi092.stderr:2024-09-13T11:26:58.395+0000 7f6f02136640 -1 osd.5 230 handle_osd_map injecting map crc failure 2024-09-13T11:26:58.469 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-13T11:26:58.483 INFO:tasks.ceph.osd.14.smithi162.stderr:2024-09-13T11:26:58.479+0000 7f3df5903640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33154) UID: 0 2024-09-13T11:26:58.536 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-09-13T11:26:58.536 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:26:58.572 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_ops_in_flight 2024-09-13T11:26:58.584 INFO:tasks.ceph.osd.13.smithi092.stderr:2024-09-13T11:26:58.579+0000 7f3e5afea640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26350) UID: 0 2024-09-13T11:26:58.689 INFO:tasks.ceph.osd.3.smithi177.stderr:2024-09-13T11:26:58.681+0000 7f10adba4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26574) UID: 0 2024-09-13T11:26:58.786 INFO:tasks.ceph.osd.1.smithi092.stderr:2024-09-13T11:26:58.783+0000 7fe7789c5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33718) UID: 0 2024-09-13T11:26:58.820 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-09-13T11:26:58.821 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:26:58.843 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_blocked_ops 2024-09-13T11:26:58.886 INFO:tasks.ceph.osd.3.smithi177.stderr:2024-09-13T11:26:58.885+0000 7f10adba4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26574) UID: 0 2024-09-13T11:26:58.979 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T11:26:59.023 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-13T11:26:59.022+0000 7f6a68775640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26307) UID: 0 2024-09-13T11:26:59.039 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-09-13T11:26:59.040 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:26:59.057 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_historic_ops 2024-09-13T11:26:59.088 INFO:tasks.ceph.osd.13.smithi092.stderr:2024-09-13T11:26:59.087+0000 7f3e5afea640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26350) UID: 0 2024-09-13T11:26:59.142 INFO:teuthology.orchestra.run.smithi001.stdout:osd.0: {} 2024-09-13T11:26:59.143 INFO:teuthology.orchestra.run.smithi001.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T11:26:59.168 INFO:teuthology.orchestra.run.smithi001.stdout:osd.1: {} 2024-09-13T11:26:59.169 INFO:teuthology.orchestra.run.smithi001.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T11:26:59.174 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-09-13T11:26:59.174 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:26:59.188 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-13T11:26:59.190 INFO:tasks.ceph.osd.2.smithi162.stderr:2024-09-13T11:26:59.187+0000 7fe92b35a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26313) UID: 0 2024-09-13T11:26:59.199 INFO:teuthology.orchestra.run.smithi001.stdout:osd.2: {} 2024-09-13T11:26:59.199 INFO:teuthology.orchestra.run.smithi001.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T11:26:59.219 INFO:teuthology.orchestra.run.smithi001.stdout:osd.3: {} 2024-09-13T11:26:59.220 INFO:teuthology.orchestra.run.smithi001.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T11:26:59.242 INFO:teuthology.orchestra.run.smithi001.stdout:osd.4: {} 2024-09-13T11:26:59.242 INFO:teuthology.orchestra.run.smithi001.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T11:26:59.263 INFO:teuthology.orchestra.run.smithi001.stdout:osd.5: {} 2024-09-13T11:26:59.264 INFO:teuthology.orchestra.run.smithi001.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T11:26:59.289 INFO:teuthology.orchestra.run.smithi001.stdout:osd.6: {} 2024-09-13T11:26:59.289 INFO:teuthology.orchestra.run.smithi001.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T11:26:59.290 INFO:tasks.ceph.osd.9.smithi092.stderr:2024-09-13T11:26:59.287+0000 7ff5ab3d0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26348) UID: 0 2024-09-13T11:26:59.319 INFO:teuthology.orchestra.run.smithi001.stdout:osd.7: {} 2024-09-13T11:26:59.319 INFO:teuthology.orchestra.run.smithi001.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T11:26:59.338 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-09-13T11:26:59.338 INFO:teuthology.orchestra.run.smithi092.stdout: "ops": [], 2024-09-13T11:26:59.338 INFO:teuthology.orchestra.run.smithi092.stdout: "num_ops": 0 2024-09-13T11:26:59.338 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-09-13T11:26:59.341 INFO:teuthology.orchestra.run.smithi001.stdout:osd.8: {} 2024-09-13T11:26:59.341 INFO:teuthology.orchestra.run.smithi001.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-13T11:26:59.353 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-13T11:26:59.364 INFO:teuthology.orchestra.run.smithi001.stdout:osd.9: {} 2024-09-13T11:26:59.364 INFO:teuthology.orchestra.run.smithi001.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-13T11:26:59.390 INFO:teuthology.orchestra.run.smithi001.stdout:osd.10: {} 2024-09-13T11:26:59.391 INFO:teuthology.orchestra.run.smithi001.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-13T11:26:59.397 INFO:tasks.ceph.osd.13.smithi092.stderr:2024-09-13T11:26:59.395+0000 7f3e5afea640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26350) UID: 0 2024-09-13T11:26:59.406 INFO:teuthology.orchestra.run.smithi001.stderr:nodeep-scrub is unset 2024-09-13T11:26:59.407 INFO:tasks.ceph.osd.9.smithi092.stderr:2024-09-13T11:26:59.403+0000 7ff59ddf9640 -1 osd.9 231 handle_osd_map injecting map crc failure 2024-09-13T11:26:59.422 INFO:tasks.ceph.osd.2.smithi162.stderr:2024-09-13T11:26:59.419+0000 7fe91fd87640 -1 osd.2 231 handle_osd_map injecting map crc failure 2024-09-13T11:26:59.424 INFO:teuthology.orchestra.run.smithi001.stdout:osd.11: {} 2024-09-13T11:26:59.425 INFO:teuthology.orchestra.run.smithi001.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-13T11:26:59.457 INFO:teuthology.orchestra.run.smithi001.stdout:osd.12: {} 2024-09-13T11:26:59.457 INFO:teuthology.orchestra.run.smithi001.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-13T11:26:59.492 INFO:tasks.ceph.osd.2.smithi162.stderr:2024-09-13T11:26:59.487+0000 7fe92b35a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26313) UID: 0 2024-09-13T11:26:59.493 INFO:teuthology.orchestra.run.smithi001.stdout:osd.13: {} 2024-09-13T11:26:59.493 INFO:teuthology.orchestra.run.smithi001.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-13T11:26:59.528 INFO:teuthology.orchestra.run.smithi001.stdout:osd.14: {} 2024-09-13T11:26:59.529 INFO:teuthology.orchestra.run.smithi001.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-13T11:26:59.543 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-09-13T11:26:59.543 INFO:teuthology.orchestra.run.smithi092.stdout: "ops": [], 2024-09-13T11:26:59.543 INFO:teuthology.orchestra.run.smithi092.stdout: "complaint_time": 30, 2024-09-13T11:26:59.544 INFO:teuthology.orchestra.run.smithi092.stdout: "num_blocked_ops": 0 2024-09-13T11:26:59.544 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-09-13T11:26:59.565 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-13T11:26:59.566 INFO:teuthology.orchestra.run.smithi001.stdout:osd.15: {} 2024-09-13T11:26:59.566 INFO:teuthology.orchestra.run.smithi001.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-13T11:26:59.592 INFO:tasks.ceph.osd.3.smithi177.stderr:2024-09-13T11:26:59.589+0000 7f10adba4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26574) UID: 0 2024-09-13T11:26:59.693 INFO:tasks.ceph.osd.5.smithi092.stderr:2024-09-13T11:26:59.691+0000 7f6f0b705640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26346) UID: 0 2024-09-13T11:26:59.754 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-09-13T11:26:59.754 INFO:teuthology.orchestra.run.smithi092.stdout: "size": 20, 2024-09-13T11:26:59.754 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 600, 2024-09-13T11:26:59.754 INFO:teuthology.orchestra.run.smithi092.stdout: "ops": [ 2024-09-13T11:26:59.755 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:59.755 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:3 3.4 3.125e09e4 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e40)", 2024-09-13T11:26:59.755 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:46.304641+0000", 2024-09-13T11:26:59.755 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 253.44874093199999, 2024-09-13T11:26:59.755 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.374372119, 2024-09-13T11:26:59.755 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:26:59.755 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:26:59.755 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "reached pg", 2024-09-13T11:26:59.755 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:26:59.755 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:26:59.755 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:59.755 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 3 2024-09-13T11:26:59.755 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:59.756 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:26:59.756 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:59.756 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:26:59.756 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.304641+0000", 2024-09-13T11:26:59.756 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:59.756 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:59.756 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:59.756 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:26:59.756 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.304641+0000", 2024-09-13T11:26:59.756 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:59.756 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:59.756 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:59.756 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:26:59.757 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.304644+0000", 2024-09-13T11:26:59.757 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.5130000000000001e-06 2024-09-13T11:26:59.757 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:59.757 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:59.757 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:26:59.757 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305058+0000", 2024-09-13T11:26:59.757 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000414277 2024-09-13T11:26:59.757 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:59.757 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:59.757 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:26:59.757 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305124+0000", 2024-09-13T11:26:59.757 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.5813000000000002e-05 2024-09-13T11:26:59.757 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:59.758 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:59.758 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:26:59.758 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305156+0000", 2024-09-13T11:26:59.758 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.1489000000000003e-05 2024-09-13T11:26:59.758 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:59.758 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:59.758 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:59.758 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.678939+0000", 2024-09-13T11:26:59.758 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.37378339999999999 2024-09-13T11:26:59.758 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:59.758 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:59.758 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:26:59.758 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.679013+0000", 2024-09-13T11:26:59.759 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.3627000000000005e-05 2024-09-13T11:26:59.759 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:59.759 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:59.759 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:59.759 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:59.759 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:59.759 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:29 3.4 3.b7bbfbe4 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-09-13T11:26:59.759 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:46.305215+0000", 2024-09-13T11:26:59.759 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 253.44816662900001, 2024-09-13T11:26:59.759 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.37387964600000001, 2024-09-13T11:26:59.759 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:26:59.759 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:26:59.759 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "reached pg", 2024-09-13T11:26:59.759 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:26:59.760 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:26:59.760 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:59.760 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 29 2024-09-13T11:26:59.760 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:59.760 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:26:59.760 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:59.760 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:26:59.760 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305215+0000", 2024-09-13T11:26:59.760 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:59.760 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:59.760 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:59.760 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:26:59.760 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305215+0000", 2024-09-13T11:26:59.761 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:59.761 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:59.761 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:59.761 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:26:59.761 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305218+0000", 2024-09-13T11:26:59.761 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.7030000000000002e-06 2024-09-13T11:26:59.761 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:59.761 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:59.761 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:26:59.761 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305366+0000", 2024-09-13T11:26:59.761 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00014825400000000001 2024-09-13T11:26:59.761 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:59.761 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:59.761 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:26:59.762 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305386+0000", 2024-09-13T11:26:59.762 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.9689000000000001e-05 2024-09-13T11:26:59.762 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:59.762 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:59.762 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:26:59.762 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305403+0000", 2024-09-13T11:26:59.762 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.7600000000000001e-05 2024-09-13T11:26:59.762 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:59.762 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:59.762 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:59.762 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.679067+0000", 2024-09-13T11:26:59.762 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.37366339100000001 2024-09-13T11:26:59.762 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:59.762 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:59.763 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:26:59.763 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.679095+0000", 2024-09-13T11:26:59.763 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.8008999999999999e-05 2024-09-13T11:26:59.763 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:59.763 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:59.763 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:59.763 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:59.763 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:59.763 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:30 3.4 3.547ec0f4 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-09-13T11:26:59.763 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:46.305453+0000", 2024-09-13T11:26:59.763 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 253.44792850499999, 2024-09-13T11:26:59.763 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.373764914, 2024-09-13T11:26:59.763 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:26:59.763 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:26:59.764 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "reached pg", 2024-09-13T11:26:59.764 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:26:59.764 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:26:59.764 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:59.764 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 30 2024-09-13T11:26:59.764 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:59.764 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:26:59.764 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:59.764 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:26:59.764 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305453+0000", 2024-09-13T11:26:59.764 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:59.764 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:59.765 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:59.765 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:26:59.765 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305453+0000", 2024-09-13T11:26:59.765 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:59.765 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:59.765 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:59.765 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:26:59.765 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305456+0000", 2024-09-13T11:26:59.765 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.9560000000000002e-06 2024-09-13T11:26:59.765 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:59.765 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:59.765 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:26:59.765 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305597+0000", 2024-09-13T11:26:59.766 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00014122700000000001 2024-09-13T11:26:59.766 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:59.766 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:59.766 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:26:59.766 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305615+0000", 2024-09-13T11:26:59.766 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.8139999999999999e-05 2024-09-13T11:26:59.766 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:59.766 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:59.766 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:26:59.766 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305632+0000", 2024-09-13T11:26:59.766 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.6101999999999999e-05 2024-09-13T11:26:59.766 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:59.766 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:59.767 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:59.767 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.679195+0000", 2024-09-13T11:26:59.767 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.373563486 2024-09-13T11:26:59.767 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:59.767 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:59.767 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:26:59.767 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.679218+0000", 2024-09-13T11:26:59.767 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.3003e-05 2024-09-13T11:26:59.767 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:59.767 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:59.767 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:59.767 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:59.767 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:59.767 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:70 3.4 3.1404a604 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-09-13T11:26:59.768 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:46.305665+0000", 2024-09-13T11:26:59.768 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 253.44771635699999, 2024-09-13T11:26:59.768 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.37365550199999997, 2024-09-13T11:26:59.768 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:26:59.768 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:26:59.768 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "reached pg", 2024-09-13T11:26:59.768 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:26:59.768 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:26:59.768 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:59.768 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 70 2024-09-13T11:26:59.768 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:59.768 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:26:59.768 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:59.769 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:26:59.769 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305665+0000", 2024-09-13T11:26:59.769 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:59.769 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:59.769 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:59.769 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:26:59.769 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305665+0000", 2024-09-13T11:26:59.769 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:59.769 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:59.769 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:59.769 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:26:59.769 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305668+0000", 2024-09-13T11:26:59.769 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.5890000000000001e-06 2024-09-13T11:26:59.770 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:59.770 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:59.770 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:26:59.770 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305791+0000", 2024-09-13T11:26:59.770 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000123545 2024-09-13T11:26:59.770 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:59.770 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:59.770 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:26:59.770 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305808+0000", 2024-09-13T11:26:59.770 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.668e-05 2024-09-13T11:26:59.770 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:59.770 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:59.770 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:26:59.770 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305824+0000", 2024-09-13T11:26:59.771 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.6385e-05 2024-09-13T11:26:59.771 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:59.771 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:59.771 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:59.771 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.679305+0000", 2024-09-13T11:26:59.771 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.37348059900000002 2024-09-13T11:26:59.771 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:59.771 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:59.771 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:26:59.771 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.679321+0000", 2024-09-13T11:26:59.771 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.5704000000000001e-05 2024-09-13T11:26:59.771 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:59.771 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:59.772 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:59.772 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:59.772 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:59.772 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:25408 3.4 3:294e23bd:::benchmark_data_smithi177_31116_object25407:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e56)", 2024-09-13T11:26:59.772 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:04.495455+0000", 2024-09-13T11:26:59.772 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 235.25792670499999, 2024-09-13T11:26:59.772 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.29897367600000002, 2024-09-13T11:26:59.772 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:26:59.772 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:26:59.772 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:59.772 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:26:59.772 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:26:59.772 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:59.773 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 25408 2024-09-13T11:26:59.773 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:59.773 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:26:59.773 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:59.773 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:26:59.773 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.495455+0000", 2024-09-13T11:26:59.773 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:59.773 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:59.773 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:59.773 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:26:59.773 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.495455+0000", 2024-09-13T11:26:59.773 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:59.773 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:59.774 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:59.774 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:26:59.774 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.495459+0000", 2024-09-13T11:26:59.774 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.5630000000000002e-06 2024-09-13T11:26:59.774 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:59.774 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:59.774 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:26:59.774 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.495545+0000", 2024-09-13T11:26:59.774 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.5704000000000005e-05 2024-09-13T11:26:59.774 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:59.774 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:59.774 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:26:59.774 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.495581+0000", 2024-09-13T11:26:59.775 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.5689000000000003e-05 2024-09-13T11:26:59.775 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:59.775 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:59.775 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:26:59.775 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.495597+0000", 2024-09-13T11:26:59.775 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.6053e-05 2024-09-13T11:26:59.775 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:59.775 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:59.775 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:59.775 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.495721+0000", 2024-09-13T11:26:59.775 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00012379099999999999 2024-09-13T11:26:59.775 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:59.775 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:59.775 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:26:59.776 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.495940+0000", 2024-09-13T11:26:59.776 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00021967699999999999 2024-09-13T11:26:59.776 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:59.776 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:59.776 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 12", 2024-09-13T11:26:59.776 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.496162+0000", 2024-09-13T11:26:59.776 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000221917 2024-09-13T11:26:59.776 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:59.776 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:59.776 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:26:59.776 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.498699+0000", 2024-09-13T11:26:59.776 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0025366690000000001 2024-09-13T11:26:59.776 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:59.777 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:59.777 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:59.777 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.794139+0000", 2024-09-13T11:26:59.777 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.295440025 2024-09-13T11:26:59.777 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:59.777 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:59.777 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:26:59.777 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.794306+0000", 2024-09-13T11:26:59.777 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00016689799999999999 2024-09-13T11:26:59.777 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:59.777 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:59.777 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:26:59.777 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.794429+0000", 2024-09-13T11:26:59.778 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00012269000000000001 2024-09-13T11:26:59.778 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:59.778 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:59.778 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:59.778 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:59.778 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:59.778 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "MOSDPGPushReply(3.14 75/70 [PushReplyOp(3:2f14928a:::benchmark_data_smithi177_31116_object15123:head)])", 2024-09-13T11:26:59.778 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:31.569662+0000", 2024-09-13T11:26:59.778 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 208.18371993299999, 2024-09-13T11:26:59.778 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.35759148299999999, 2024-09-13T11:26:59.778 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:26:59.778 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:26:59.778 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "reached pg", 2024-09-13T11:26:59.778 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:26:59.779 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:59.779 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:26:59.779 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:31.569662+0000", 2024-09-13T11:26:59.779 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:59.779 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:59.779 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:59.779 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:26:59.779 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:31.569662+0000", 2024-09-13T11:26:59.779 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:59.779 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:59.779 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:59.779 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:26:59.779 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:31.569663+0000", 2024-09-13T11:26:59.780 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.333e-06 2024-09-13T11:26:59.780 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:59.780 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:59.780 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:26:59.780 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:31.569667+0000", 2024-09-13T11:26:59.780 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.8929999999999998e-06 2024-09-13T11:26:59.780 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:59.780 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:59.780 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:26:59.780 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:31.923221+0000", 2024-09-13T11:26:59.780 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.35355413699999999 2024-09-13T11:26:59.780 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:59.780 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:59.780 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:26:59.781 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:31.923230+0000", 2024-09-13T11:26:59.781 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.6829999999999997e-06 2024-09-13T11:26:59.781 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:59.781 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:59.781 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:59.781 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:31.923807+0000", 2024-09-13T11:26:59.781 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00057686800000000002 2024-09-13T11:26:59.781 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:59.781 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:59.781 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:26:59.781 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:31.927253+0000", 2024-09-13T11:26:59.781 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.003446569 2024-09-13T11:26:59.781 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:59.782 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:59.782 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:59.782 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:59.782 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:59.782 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "MOSDPGPushReply(3.18 85/70 [PushReplyOp(3:1e410119:::benchmark_data_smithi177_31116_object35700:head)])", 2024-09-13T11:26:59.782 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:42.216614+0000", 2024-09-13T11:26:59.782 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 197.53676758899999, 2024-09-13T11:26:59.782 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.54260090699999997, 2024-09-13T11:26:59.782 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:26:59.782 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:26:59.782 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "reached pg", 2024-09-13T11:26:59.782 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:26:59.782 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:59.783 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:26:59.783 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.216614+0000", 2024-09-13T11:26:59.783 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:59.783 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:59.783 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:59.783 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:26:59.783 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.216614+0000", 2024-09-13T11:26:59.783 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:59.783 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:59.783 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:59.783 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:26:59.783 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.216615+0000", 2024-09-13T11:26:59.783 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.0200000000000001e-07 2024-09-13T11:26:59.783 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:59.784 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:59.784 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:26:59.784 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.216619+0000", 2024-09-13T11:26:59.784 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.805e-06 2024-09-13T11:26:59.784 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:59.784 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:59.784 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:26:59.784 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.757476+0000", 2024-09-13T11:26:59.784 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.54085762000000004 2024-09-13T11:26:59.784 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:59.784 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:59.784 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:26:59.784 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.757498+0000", 2024-09-13T11:26:59.785 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.1936000000000001e-05 2024-09-13T11:26:59.785 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:59.785 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:59.785 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:59.785 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.757678+0000", 2024-09-13T11:26:59.785 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00018006599999999999 2024-09-13T11:26:59.785 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:59.785 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:59.785 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:26:59.785 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.759215+0000", 2024-09-13T11:26:59.785 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0015366780000000001 2024-09-13T11:26:59.785 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:59.786 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:59.786 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:59.786 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:59.786 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:59.786 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "MOSDPGPushReply(3.1b 85/78 [PushReplyOp(3:dabda1cc:::benchmark_data_smithi177_31116_object12988:head)])", 2024-09-13T11:26:59.786 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:42.485171+0000", 2024-09-13T11:26:59.786 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 197.268210681, 2024-09-13T11:26:59.786 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.308420427, 2024-09-13T11:26:59.786 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:26:59.786 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:26:59.786 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "reached pg", 2024-09-13T11:26:59.786 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:26:59.786 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:59.786 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:26:59.787 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.485171+0000", 2024-09-13T11:26:59.787 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:59.787 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:59.787 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:59.787 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:26:59.787 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.485171+0000", 2024-09-13T11:26:59.787 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:59.787 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:59.787 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:59.787 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:26:59.787 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.485173+0000", 2024-09-13T11:26:59.787 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.1569999999999998e-06 2024-09-13T11:26:59.787 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:59.787 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:59.787 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:26:59.788 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.485182+0000", 2024-09-13T11:26:59.788 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.6759999999999996e-06 2024-09-13T11:26:59.788 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:59.788 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:59.788 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:26:59.788 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.791336+0000", 2024-09-13T11:26:59.788 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.30615446499999999 2024-09-13T11:26:59.788 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:59.788 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:59.788 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:26:59.788 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.791369+0000", 2024-09-13T11:26:59.788 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.3155999999999999e-05 2024-09-13T11:26:59.788 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:59.789 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:59.789 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:59.789 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.791472+0000", 2024-09-13T11:26:59.789 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00010247900000000001 2024-09-13T11:26:59.789 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:59.789 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:59.789 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:26:59.789 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.793591+0000", 2024-09-13T11:26:59.789 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.002119494 2024-09-13T11:26:59.789 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:59.789 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:59.789 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:59.789 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:59.790 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:59.790 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:64066 3.1b 3:dd84e0ec:::benchmark_data_smithi177_31116_object64065:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89)", 2024-09-13T11:26:59.790 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:46.633608+0000", 2024-09-13T11:26:59.790 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 193.11977324200001, 2024-09-13T11:26:59.790 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.32564315399999999, 2024-09-13T11:26:59.790 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:26:59.790 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:26:59.790 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:59.790 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:26:59.790 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:26:59.790 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:59.790 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 64066 2024-09-13T11:26:59.791 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:59.791 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:26:59.791 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:59.791 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:26:59.791 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.633608+0000", 2024-09-13T11:26:59.791 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:59.791 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:59.791 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:59.791 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:26:59.791 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.633608+0000", 2024-09-13T11:26:59.791 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:59.791 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:59.791 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:59.792 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:26:59.792 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.633612+0000", 2024-09-13T11:26:59.792 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.856e-06 2024-09-13T11:26:59.792 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:59.792 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:59.792 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:26:59.792 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.633683+0000", 2024-09-13T11:26:59.792 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.1000999999999993e-05 2024-09-13T11:26:59.792 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:59.792 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:59.792 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:26:59.792 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.633711+0000", 2024-09-13T11:26:59.792 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.8204999999999998e-05 2024-09-13T11:26:59.793 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:59.793 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:59.793 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:26:59.793 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.633732+0000", 2024-09-13T11:26:59.793 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.0662999999999999e-05 2024-09-13T11:26:59.793 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:59.793 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:59.793 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:59.793 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.635573+0000", 2024-09-13T11:26:59.793 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.001841365 2024-09-13T11:26:59.793 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:59.793 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:59.793 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:26:59.793 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.635995+0000", 2024-09-13T11:26:59.794 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00042113699999999998 2024-09-13T11:26:59.794 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:59.794 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:59.794 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 12,15", 2024-09-13T11:26:59.794 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.636475+0000", 2024-09-13T11:26:59.794 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00048067699999999999 2024-09-13T11:26:59.794 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:59.794 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:59.794 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:59.794 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.640247+0000", 2024-09-13T11:26:59.794 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0037716450000000001 2024-09-13T11:26:59.794 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:59.794 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:59.795 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:26:59.795 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.649308+0000", 2024-09-13T11:26:59.795 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0090613240000000008 2024-09-13T11:26:59.795 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:59.795 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:59.795 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:59.795 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.956912+0000", 2024-09-13T11:26:59.795 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.30760338500000001 2024-09-13T11:26:59.795 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:59.795 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:59.795 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:26:59.795 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.957156+0000", 2024-09-13T11:26:59.795 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000244031 2024-09-13T11:26:59.796 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:59.796 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:59.796 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:26:59.796 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.959252+0000", 2024-09-13T11:26:59.796 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0020958650000000001 2024-09-13T11:26:59.796 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:59.796 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:59.796 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:59.796 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:59.796 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:59.796 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop_reply(client.4665.0:64066 3.1b e89/78 ondisk, result = 0)", 2024-09-13T11:26:59.796 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:46.651596+0000", 2024-09-13T11:26:59.796 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 193.10178541100001, 2024-09-13T11:26:59.796 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.30767612599999999, 2024-09-13T11:26:59.797 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:26:59.797 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:26:59.797 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-09-13T11:26:59.797 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:26:59.797 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:59.797 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:26:59.797 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.651596+0000", 2024-09-13T11:26:59.797 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:59.797 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:59.797 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:59.797 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:26:59.798 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.651596+0000", 2024-09-13T11:26:59.798 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:59.798 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:59.798 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:59.798 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:26:59.798 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.651600+0000", 2024-09-13T11:26:59.798 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.6969999999999999e-06 2024-09-13T11:26:59.798 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:59.798 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:59.798 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:26:59.798 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.651611+0000", 2024-09-13T11:26:59.798 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.1516e-05 2024-09-13T11:26:59.798 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:59.799 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:59.799 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:26:59.799 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.948118+0000", 2024-09-13T11:26:59.799 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.29650663500000002 2024-09-13T11:26:59.799 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:59.799 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:59.799 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:26:59.799 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.948137+0000", 2024-09-13T11:26:59.799 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.8529000000000002e-05 2024-09-13T11:26:59.799 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:59.799 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:59.799 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:59.799 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.956870+0000", 2024-09-13T11:26:59.800 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0087336380000000002 2024-09-13T11:26:59.800 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:59.800 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:59.800 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:26:59.800 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.956893+0000", 2024-09-13T11:26:59.800 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.317e-05 2024-09-13T11:26:59.800 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:59.800 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:59.800 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:26:59.800 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.959272+0000", 2024-09-13T11:26:59.800 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0023789409999999999 2024-09-13T11:26:59.800 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:59.800 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:59.801 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:59.801 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:59.801 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:59.801 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "MOSDPGPushReply(3.1b 91/78 [PushReplyOp(3:d9ac6889:::benchmark_data_smithi177_31116_object22254:head)])", 2024-09-13T11:26:59.801 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:48.392729+0000", 2024-09-13T11:26:59.801 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 191.360652496, 2024-09-13T11:26:59.801 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.30924993099999998, 2024-09-13T11:26:59.801 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:26:59.801 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:26:59.801 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "reached pg", 2024-09-13T11:26:59.801 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:26:59.801 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:59.801 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:26:59.802 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:48.392729+0000", 2024-09-13T11:26:59.802 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:59.802 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:59.802 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:59.802 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:26:59.802 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:48.392729+0000", 2024-09-13T11:26:59.802 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:59.802 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:59.802 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:59.802 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:26:59.802 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:48.392730+0000", 2024-09-13T11:26:59.802 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.336e-06 2024-09-13T11:26:59.803 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:59.803 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:59.803 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:26:59.803 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:48.392739+0000", 2024-09-13T11:26:59.803 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.8580000000000006e-06 2024-09-13T11:26:59.803 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:59.803 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:59.803 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:26:59.803 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:48.700180+0000", 2024-09-13T11:26:59.803 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.30744035600000003 2024-09-13T11:26:59.803 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:59.803 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:59.803 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:26:59.803 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:48.700197+0000", 2024-09-13T11:26:59.804 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.7649999999999999e-05 2024-09-13T11:26:59.804 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:59.804 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:59.804 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:59.804 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:48.700398+0000", 2024-09-13T11:26:59.804 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000201113 2024-09-13T11:26:59.804 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:59.804 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:59.804 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:26:59.804 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:48.701979+0000", 2024-09-13T11:26:59.804 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.001580618 2024-09-13T11:26:59.804 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:59.804 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:59.805 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:59.805 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:59.805 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:59.805 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop(client.4665.0:83193 3.8 e106/67 3:10066d83:::benchmark_data_smithi177_31116_object83192:head v 106'3925, mlcod=106'3922)", 2024-09-13T11:26:59.805 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:24:04.285068+0000", 2024-09-13T11:26:59.805 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 175.468313741, 2024-09-13T11:26:59.805 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.64647716600000005, 2024-09-13T11:26:59.805 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:26:59.805 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:26:59.805 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:59.805 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:26:59.805 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:59.805 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:26:59.805 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:04.285068+0000", 2024-09-13T11:26:59.806 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:59.806 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:59.806 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:59.806 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:26:59.806 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:04.285068+0000", 2024-09-13T11:26:59.806 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:59.806 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:59.806 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:59.806 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:26:59.806 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:04.285071+0000", 2024-09-13T11:26:59.806 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.729e-06 2024-09-13T11:26:59.807 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:59.807 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:59.807 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:26:59.807 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:04.285215+0000", 2024-09-13T11:26:59.807 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00014410000000000001 2024-09-13T11:26:59.807 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:59.807 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:59.807 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:26:59.807 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:04.928221+0000", 2024-09-13T11:26:59.807 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.64300653299999999 2024-09-13T11:26:59.807 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:59.807 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:59.807 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:26:59.807 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:04.928240+0000", 2024-09-13T11:26:59.808 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.8678999999999999e-05 2024-09-13T11:26:59.808 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:59.808 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:59.808 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:59.808 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:04.928388+0000", 2024-09-13T11:26:59.808 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00014809 2024-09-13T11:26:59.808 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:59.808 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:59.808 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:26:59.808 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:04.928428+0000", 2024-09-13T11:26:59.808 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.9776e-05 2024-09-13T11:26:59.808 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:59.808 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:59.809 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:26:59.809 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:04.931463+0000", 2024-09-13T11:26:59.809 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.003035168 2024-09-13T11:26:59.809 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:59.809 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:59.809 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:26:59.809 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:04.931545+0000", 2024-09-13T11:26:59.809 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.2090999999999997e-05 2024-09-13T11:26:59.809 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:59.809 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:59.809 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:59.809 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:59.809 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:59.809 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop(client.4665.0:83239 3.e e106/75 3:74547f17:::benchmark_data_smithi177_31116_object83238:head v 106'4391, mlcod=106'4388)", 2024-09-13T11:26:59.810 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:24:04.581196+0000", 2024-09-13T11:26:59.810 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 175.172185235, 2024-09-13T11:26:59.810 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.33617203600000001, 2024-09-13T11:26:59.810 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:26:59.810 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:26:59.810 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:59.810 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:26:59.810 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:59.810 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:26:59.810 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:04.581196+0000", 2024-09-13T11:26:59.810 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:59.810 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:59.810 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:59.810 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:26:59.811 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:04.581196+0000", 2024-09-13T11:26:59.811 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:59.811 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:59.811 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:59.811 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:26:59.811 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:04.581200+0000", 2024-09-13T11:26:59.811 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.19e-06 2024-09-13T11:26:59.811 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:59.811 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:59.811 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:26:59.811 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:04.581257+0000", 2024-09-13T11:26:59.811 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.7531999999999997e-05 2024-09-13T11:26:59.811 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:59.811 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:59.812 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:26:59.812 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:04.914011+0000", 2024-09-13T11:26:59.812 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.33275413700000001 2024-09-13T11:26:59.812 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:59.812 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:59.812 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:26:59.812 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:04.914028+0000", 2024-09-13T11:26:59.812 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.7082999999999999e-05 2024-09-13T11:26:59.812 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:59.812 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:59.812 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:59.812 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:04.914151+0000", 2024-09-13T11:26:59.812 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00012283300000000001 2024-09-13T11:26:59.812 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:59.813 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:59.813 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:26:59.813 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:04.914185+0000", 2024-09-13T11:26:59.813 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.4255999999999999e-05 2024-09-13T11:26:59.813 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:59.813 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:59.813 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:26:59.813 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:04.917300+0000", 2024-09-13T11:26:59.813 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0031141599999999999 2024-09-13T11:26:59.813 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:59.813 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:59.813 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:26:59.813 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:04.917368+0000", 2024-09-13T11:26:59.814 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.8844999999999996e-05 2024-09-13T11:26:59.814 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:59.814 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:59.814 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:59.814 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:59.814 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:59.814 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:131687 3.28 3:14f65078:::benchmark_data_smithi177_31116_object131686:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e135)", 2024-09-13T11:26:59.814 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:24:48.925565+0000", 2024-09-13T11:26:59.814 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 130.827816238, 2024-09-13T11:26:59.814 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.41421604099999998, 2024-09-13T11:26:59.814 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:26:59.814 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:26:59.814 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:59.815 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:26:59.815 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:26:59.815 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:59.815 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 131687 2024-09-13T11:26:59.815 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:59.815 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:26:59.815 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:59.815 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:26:59.815 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:48.925565+0000", 2024-09-13T11:26:59.815 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:59.815 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:59.815 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:59.815 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:26:59.815 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:48.925565+0000", 2024-09-13T11:26:59.816 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:59.816 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:59.816 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:59.816 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:26:59.816 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:48.925568+0000", 2024-09-13T11:26:59.816 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.441e-06 2024-09-13T11:26:59.816 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:59.816 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:59.816 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:26:59.816 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:48.925617+0000", 2024-09-13T11:26:59.816 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.8739000000000002e-05 2024-09-13T11:26:59.816 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:59.816 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:59.817 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:26:59.817 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:48.925631+0000", 2024-09-13T11:26:59.817 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.4463e-05 2024-09-13T11:26:59.817 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:59.817 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:59.817 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:26:59.817 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:48.925638+0000", 2024-09-13T11:26:59.817 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.2119999999999997e-06 2024-09-13T11:26:59.817 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:59.817 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:59.817 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:59.817 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:48.925703+0000", 2024-09-13T11:26:59.817 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.4794000000000003e-05 2024-09-13T11:26:59.818 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:59.818 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:59.818 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:26:59.818 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:48.925857+0000", 2024-09-13T11:26:59.818 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00015427 2024-09-13T11:26:59.818 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:59.818 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:59.818 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 5,8", 2024-09-13T11:26:59.818 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:48.925995+0000", 2024-09-13T11:26:59.818 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000137365 2024-09-13T11:26:59.818 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:59.818 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:59.818 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:26:59.818 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:48.927261+0000", 2024-09-13T11:26:59.819 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.001266021 2024-09-13T11:26:59.819 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:59.819 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:59.819 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:59.819 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:48.930015+0000", 2024-09-13T11:26:59.819 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0027545130000000001 2024-09-13T11:26:59.819 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:59.819 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:59.819 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:59.819 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:49.338898+0000", 2024-09-13T11:26:59.819 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.40888254299999999 2024-09-13T11:26:59.819 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:59.819 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:59.820 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:26:59.820 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:49.338967+0000", 2024-09-13T11:26:59.820 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.9402999999999999e-05 2024-09-13T11:26:59.820 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:59.820 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:59.820 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:26:59.820 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:49.339781+0000", 2024-09-13T11:26:59.820 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00081427699999999997 2024-09-13T11:26:59.820 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:59.820 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:59.820 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:59.820 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:59.820 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:59.821 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop_reply(client.4665.0:131687 3.28 e135/120 ondisk, result = 0)", 2024-09-13T11:26:59.821 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:24:48.928935+0000", 2024-09-13T11:26:59.821 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 130.824446937, 2024-09-13T11:26:59.821 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.41085638499999999, 2024-09-13T11:26:59.821 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:26:59.821 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:26:59.821 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-09-13T11:26:59.821 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:26:59.821 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:59.821 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:26:59.821 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:48.928935+0000", 2024-09-13T11:26:59.821 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:59.821 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:59.821 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:59.822 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:26:59.822 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:48.928935+0000", 2024-09-13T11:26:59.822 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:59.822 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:59.822 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:59.822 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:26:59.822 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:48.928936+0000", 2024-09-13T11:26:59.822 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.519e-06 2024-09-13T11:26:59.822 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:59.822 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:59.822 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:26:59.822 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:48.928941+0000", 2024-09-13T11:26:59.822 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.0710000000000002e-06 2024-09-13T11:26:59.822 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:59.823 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:59.823 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:26:59.823 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:49.338799+0000", 2024-09-13T11:26:59.823 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.40985808499999998 2024-09-13T11:26:59.823 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:59.823 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:59.823 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:26:59.823 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:49.338809+0000", 2024-09-13T11:26:59.823 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.4390000000000006e-06 2024-09-13T11:26:59.823 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:59.823 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:59.823 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:59.823 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:49.338878+0000", 2024-09-13T11:26:59.823 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.9429000000000002e-05 2024-09-13T11:26:59.824 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:59.824 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:59.824 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:26:59.824 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:49.338890+0000", 2024-09-13T11:26:59.824 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.2217e-05 2024-09-13T11:26:59.824 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:59.824 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:59.824 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:26:59.824 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:49.339791+0000", 2024-09-13T11:26:59.824 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00090062499999999995 2024-09-13T11:26:59.824 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:59.824 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:59.824 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:59.824 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:59.825 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:59.825 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "MOSDPGPushReply(3.28 163/120 [PushReplyOp(3:169fe9fd:::benchmark_data_smithi177_31116_object59744:head)])", 2024-09-13T11:26:59.825 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:25:22.263786+0000", 2024-09-13T11:26:59.825 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 97.489595668999996, 2024-09-13T11:26:59.825 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.28927712799999999, 2024-09-13T11:26:59.825 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:26:59.825 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:26:59.825 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "reached pg", 2024-09-13T11:26:59.825 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:26:59.825 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:59.825 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:26:59.825 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:22.263786+0000", 2024-09-13T11:26:59.825 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:59.826 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:59.826 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:59.826 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:26:59.826 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:22.263786+0000", 2024-09-13T11:26:59.826 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:59.826 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:59.826 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:59.826 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:26:59.826 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:22.263788+0000", 2024-09-13T11:26:59.826 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.542e-06 2024-09-13T11:26:59.826 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:59.826 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:59.826 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:26:59.826 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:22.263797+0000", 2024-09-13T11:26:59.827 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.234e-06 2024-09-13T11:26:59.827 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:59.827 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:59.827 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:26:59.827 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:22.552310+0000", 2024-09-13T11:26:59.827 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.28851281499999998 2024-09-13T11:26:59.827 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:59.827 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:59.827 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:26:59.827 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:22.552315+0000", 2024-09-13T11:26:59.827 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.5990000000000001e-06 2024-09-13T11:26:59.827 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:59.827 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:59.828 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:59.828 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:22.552362+0000", 2024-09-13T11:26:59.828 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.7144000000000001e-05 2024-09-13T11:26:59.828 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:59.828 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:59.828 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:26:59.828 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:22.553063+0000", 2024-09-13T11:26:59.828 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00070079400000000003 2024-09-13T11:26:59.828 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:59.828 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:59.828 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:59.828 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:59.828 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:59.829 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:181661 3.4 3:2019f47a:::benchmark_data_smithi177_31116_object41358:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e170)", 2024-09-13T11:26:59.829 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:25:31.131054+0000", 2024-09-13T11:26:59.829 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 88.622327170999995, 2024-09-13T11:26:59.829 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.44761152399999998, 2024-09-13T11:26:59.829 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:26:59.829 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:26:59.829 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:59.829 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:26:59.829 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:26:59.829 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:26:59.829 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 181661 2024-09-13T11:26:59.829 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:59.829 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:26:59.830 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:59.830 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:26:59.830 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:31.131054+0000", 2024-09-13T11:26:59.830 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:59.830 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:59.830 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:59.830 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:26:59.830 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:31.131054+0000", 2024-09-13T11:26:59.830 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:59.830 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:59.830 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:59.830 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:26:59.830 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:31.131056+0000", 2024-09-13T11:26:59.830 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.8640000000000001e-06 2024-09-13T11:26:59.831 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:59.831 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:59.831 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:26:59.831 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:31.131061+0000", 2024-09-13T11:26:59.831 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.6249999999999998e-06 2024-09-13T11:26:59.831 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:59.831 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:59.831 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:26:59.831 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:31.131069+0000", 2024-09-13T11:26:59.831 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.1049999999999995e-06 2024-09-13T11:26:59.831 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:59.831 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:59.831 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:26:59.832 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:31.131074+0000", 2024-09-13T11:26:59.832 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.344e-06 2024-09-13T11:26:59.832 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:59.832 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:59.832 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:59.832 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:31.131121+0000", 2024-09-13T11:26:59.832 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.6794e-05 2024-09-13T11:26:59.832 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:59.832 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:59.832 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:26:59.832 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:31.131225+0000", 2024-09-13T11:26:59.832 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000103502 2024-09-13T11:26:59.832 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:59.833 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:59.833 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 12", 2024-09-13T11:26:59.833 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:31.131289+0000", 2024-09-13T11:26:59.833 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.4790999999999997e-05 2024-09-13T11:26:59.833 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:59.833 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:59.833 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:26:59.833 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:31.133191+0000", 2024-09-13T11:26:59.833 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0019013859999999999 2024-09-13T11:26:59.833 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:59.833 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:59.833 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:26:59.833 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:31.578585+0000", 2024-09-13T11:26:59.834 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.44539403 2024-09-13T11:26:59.834 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:59.834 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:59.834 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:26:59.834 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:31.578620+0000", 2024-09-13T11:26:59.834 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.4657999999999999e-05 2024-09-13T11:26:59.834 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:59.834 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:59.834 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:26:59.834 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:31.578666+0000", 2024-09-13T11:26:59.834 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.6424999999999997e-05 2024-09-13T11:26:59.834 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:59.834 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:59.835 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:59.835 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:59.835 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:59.835 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop_reply(client.4665.0:181661 3.4 e170/117 ondisk, result = 0)", 2024-09-13T11:26:59.835 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:25:31.135385+0000", 2024-09-13T11:26:59.835 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 88.617996351000002, 2024-09-13T11:26:59.835 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.44328512599999997, 2024-09-13T11:26:59.835 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:26:59.835 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:26:59.835 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-09-13T11:26:59.835 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:26:59.835 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:59.835 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:26:59.836 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:31.135385+0000", 2024-09-13T11:26:59.836 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:59.836 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:59.836 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:59.836 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:26:59.836 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:31.135385+0000", 2024-09-13T11:26:59.836 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:59.836 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:59.836 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:59.836 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:26:59.836 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:31.135387+0000", 2024-09-13T11:26:59.836 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.217e-06 2024-09-13T11:26:59.836 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:59.837 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:59.837 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:26:59.837 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:31.135391+0000", 2024-09-13T11:26:59.837 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.0820000000000001e-06 2024-09-13T11:26:59.837 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:59.837 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:59.837 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:26:59.837 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:31.578504+0000", 2024-09-13T11:26:59.837 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.44311382100000002 2024-09-13T11:26:59.837 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:59.837 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:59.837 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:26:59.837 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:31.578530+0000", 2024-09-13T11:26:59.838 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.5367e-05 2024-09-13T11:26:59.838 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:59.838 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:59.838 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:59.838 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:31.578576+0000", 2024-09-13T11:26:59.838 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.5840999999999997e-05 2024-09-13T11:26:59.838 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:59.838 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:59.838 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:26:59.838 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:31.578581+0000", 2024-09-13T11:26:59.838 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.5620000000000003e-06 2024-09-13T11:26:59.838 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:59.838 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:59.839 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:26:59.839 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:31.578670+0000", 2024-09-13T11:26:59.839 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.9235999999999997e-05 2024-09-13T11:26:59.839 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:59.839 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:59.839 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:59.839 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:59.839 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:59.839 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop(client.4665.0:184026 3.b e171/117 3:d3052347:::benchmark_data_smithi177_31116_object43723:head v 171'6273, mlcod=171'6269)", 2024-09-13T11:26:59.839 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:25:32.645695+0000", 2024-09-13T11:26:59.839 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 87.107686616999999, 2024-09-13T11:26:59.839 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.34822244200000002, 2024-09-13T11:26:59.839 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:26:59.840 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:26:59.840 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:59.840 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:26:59.840 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:59.840 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:26:59.840 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:32.645695+0000", 2024-09-13T11:26:59.840 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:59.840 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:59.840 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:59.840 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:26:59.841 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:32.645695+0000", 2024-09-13T11:26:59.841 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:59.841 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:59.841 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:59.841 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:26:59.841 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:32.645696+0000", 2024-09-13T11:26:59.841 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.4780000000000001e-06 2024-09-13T11:26:59.841 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:59.841 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:59.841 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:26:59.841 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:32.645704+0000", 2024-09-13T11:26:59.841 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.1980000000000004e-06 2024-09-13T11:26:59.841 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:59.842 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:59.842 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:26:59.842 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:32.992228+0000", 2024-09-13T11:26:59.842 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.346523992 2024-09-13T11:26:59.842 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:59.842 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:59.842 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:26:59.842 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:32.992232+0000", 2024-09-13T11:26:59.842 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.3109999999999999e-06 2024-09-13T11:26:59.842 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:59.842 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:59.842 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:59.842 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:32.992275+0000", 2024-09-13T11:26:59.842 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.2589999999999997e-05 2024-09-13T11:26:59.843 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:59.843 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:59.843 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:26:59.843 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:32.992289+0000", 2024-09-13T11:26:59.843 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.4116e-05 2024-09-13T11:26:59.843 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:59.843 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:59.843 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:26:59.843 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:32.993877+0000", 2024-09-13T11:26:59.843 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0015879259999999999 2024-09-13T11:26:59.843 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:59.843 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:59.843 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:26:59.844 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:32.993917+0000", 2024-09-13T11:26:59.844 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.0831000000000003e-05 2024-09-13T11:26:59.844 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:59.844 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:59.844 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:59.844 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:59.844 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:59.844 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop(client.4665.0:184072 3.b e171/117 3:d319cf93:::benchmark_data_smithi177_31116_object43769:head v 171'6274, mlcod=171'6270)", 2024-09-13T11:26:59.844 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:25:32.701516+0000", 2024-09-13T11:26:59.844 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 87.051865929000002, 2024-09-13T11:26:59.844 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.30463955399999998, 2024-09-13T11:26:59.844 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:26:59.844 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:26:59.845 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:26:59.845 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:26:59.845 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:59.845 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:26:59.845 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:32.701516+0000", 2024-09-13T11:26:59.845 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:59.845 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:59.845 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:59.845 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:26:59.845 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:32.701516+0000", 2024-09-13T11:26:59.845 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:26:59.845 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:59.845 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:59.845 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:26:59.846 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:32.701519+0000", 2024-09-13T11:26:59.846 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.0029999999999999e-06 2024-09-13T11:26:59.846 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:59.846 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:59.846 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:26:59.846 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:32.701530+0000", 2024-09-13T11:26:59.846 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.1759999999999999e-05 2024-09-13T11:26:59.846 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:59.846 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:59.846 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:26:59.846 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:33.003551+0000", 2024-09-13T11:26:59.846 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.302020819 2024-09-13T11:26:59.846 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:59.847 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:59.847 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:26:59.847 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:33.003558+0000", 2024-09-13T11:26:59.847 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.2299999999999996e-06 2024-09-13T11:26:59.847 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:59.847 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:59.847 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:26:59.847 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:33.003630+0000", 2024-09-13T11:26:59.847 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.2125000000000005e-05 2024-09-13T11:26:59.847 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:59.847 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:59.847 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:26:59.847 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:33.003648+0000", 2024-09-13T11:26:59.847 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.8672e-05 2024-09-13T11:26:59.848 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:59.848 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:59.848 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:26:59.848 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:33.006114+0000", 2024-09-13T11:26:59.848 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0024656790000000001 2024-09-13T11:26:59.848 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:26:59.848 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:26:59.848 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:26:59.848 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:33.006155+0000", 2024-09-13T11:26:59.848 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.1266000000000001e-05 2024-09-13T11:26:59.848 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:59.848 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:59.848 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:59.848 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:26:59.849 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:26:59.849 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-09-13T11:26:59.850 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_ops_in_flight 2024-09-13T11:26:59.851 INFO:tasks.ceph.osd.9.smithi092.stderr:2024-09-13T11:26:59.847+0000 7ff5ab3d0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26348) UID: 0 2024-09-13T11:26:59.953 INFO:tasks.ceph.osd.9.smithi092.stderr:2024-09-13T11:26:59.951+0000 7ff5ab3d0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26348) UID: 0 2024-09-13T11:27:00.001 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-13T11:27:00.001 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [ 2024-09-13T11:27:00.001 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:00.001 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:273835 3.15 3:aa83e6b2:::benchmark_data_smithi177_31116_object133532:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e232)", 2024-09-13T11:27:00.001 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:26:59.996551+0000", 2024-09-13T11:27:00.001 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 0.0055243230000000003, 2024-09-13T11:27:00.002 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0055527600000000003, 2024-09-13T11:27:00.002 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:27:00.002 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:27:00.002 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "waiting for sub ops", 2024-09-13T11:27:00.002 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:27:00.002 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:27:00.002 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:27:00.002 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 273835 2024-09-13T11:27:00.002 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:00.002 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:27:00.002 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:00.002 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:27:00.002 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:59.996551+0000", 2024-09-13T11:27:00.003 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:00.003 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:00.003 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:00.003 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:27:00.003 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:59.996551+0000", 2024-09-13T11:27:00.003 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:00.003 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:00.003 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:00.003 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:27:00.003 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:59.996553+0000", 2024-09-13T11:27:00.003 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.6960000000000001e-06 2024-09-13T11:27:00.003 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:00.004 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:00.004 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:27:00.004 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:59.996560+0000", 2024-09-13T11:27:00.004 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.7839999999999998e-06 2024-09-13T11:27:00.004 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:00.004 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:00.004 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:27:00.004 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:59.996578+0000", 2024-09-13T11:27:00.005 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.7944999999999998e-05 2024-09-13T11:27:00.005 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:00.005 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:00.005 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:27:00.005 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:59.996588+0000", 2024-09-13T11:27:00.005 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.5400000000000001e-06 2024-09-13T11:27:00.005 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:00.006 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:00.006 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:27:00.006 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:59.996657+0000", 2024-09-13T11:27:00.006 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.9581999999999999e-05 2024-09-13T11:27:00.006 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:00.006 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:00.006 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:27:00.006 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:59.996803+0000", 2024-09-13T11:27:00.007 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00014564299999999999 2024-09-13T11:27:00.007 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:00.007 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:00.007 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 0", 2024-09-13T11:27:00.007 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:59.996918+0000", 2024-09-13T11:27:00.007 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00011534799999999999 2024-09-13T11:27:00.007 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:00.007 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:00.007 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:27:00.007 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:59.999153+0000", 2024-09-13T11:27:00.008 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0022351350000000001 2024-09-13T11:27:00.008 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:00.008 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:27:00.008 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:00.008 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:00.008 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:00.008 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:273802 3.2 3:433fd1a7:::benchmark_data_smithi177_31116_object133499:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e232)", 2024-09-13T11:27:00.008 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:26:59.966236+0000", 2024-09-13T11:27:00.008 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 0.035838652999999998, 2024-09-13T11:27:00.008 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.035942333, 2024-09-13T11:27:00.008 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:27:00.008 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:27:00.008 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "waiting for sub ops", 2024-09-13T11:27:00.008 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:27:00.009 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:27:00.009 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:27:00.009 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 273802 2024-09-13T11:27:00.009 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:00.009 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:27:00.009 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:00.009 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:27:00.009 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:59.966236+0000", 2024-09-13T11:27:00.009 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:00.009 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:00.009 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:00.009 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:27:00.009 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:59.966236+0000", 2024-09-13T11:27:00.009 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:00.010 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:00.010 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:00.010 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:27:00.010 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:59.966239+0000", 2024-09-13T11:27:00.010 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.9919999999999999e-06 2024-09-13T11:27:00.010 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:00.010 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:00.010 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:27:00.010 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:59.966246+0000", 2024-09-13T11:27:00.010 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.2219999999999996e-06 2024-09-13T11:27:00.010 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:00.010 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:00.010 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:27:00.010 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:59.966259+0000", 2024-09-13T11:27:00.011 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3023000000000001e-05 2024-09-13T11:27:00.011 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:00.011 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:00.011 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:27:00.011 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:59.966268+0000", 2024-09-13T11:27:00.011 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.9730000000000003e-06 2024-09-13T11:27:00.011 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:00.011 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:00.011 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:27:00.011 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:59.966331+0000", 2024-09-13T11:27:00.011 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.2561e-05 2024-09-13T11:27:00.011 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:00.011 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:00.011 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:27:00.012 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:59.966468+0000", 2024-09-13T11:27:00.012 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00013707199999999999 2024-09-13T11:27:00.012 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:00.012 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:00.012 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 12", 2024-09-13T11:27:00.012 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:59.966580+0000", 2024-09-13T11:27:00.012 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000112169 2024-09-13T11:27:00.012 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:00.012 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:00.012 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:27:00.012 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:59.968823+0000", 2024-09-13T11:27:00.012 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0022422760000000001 2024-09-13T11:27:00.012 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:00.012 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:27:00.013 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:00.013 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:00.013 INFO:teuthology.orchestra.run.smithi162.stdout: ], 2024-09-13T11:27:00.013 INFO:teuthology.orchestra.run.smithi162.stdout: "num_ops": 2 2024-09-13T11:27:00.013 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-13T11:27:00.021 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_blocked_ops 2024-09-13T11:27:00.051 INFO:tasks.ceph.osd.15.smithi177.stderr:2024-09-13T11:27:00.049+0000 7f9d6ce77640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26578) UID: 0 2024-09-13T11:27:00.152 INFO:tasks.ceph.osd.15.smithi177.stderr:2024-09-13T11:27:00.149+0000 7f9d6ce77640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26578) UID: 0 2024-09-13T11:27:00.162 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-13T11:27:00.162 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [], 2024-09-13T11:27:00.162 INFO:teuthology.orchestra.run.smithi162.stdout: "complaint_time": 30, 2024-09-13T11:27:00.163 INFO:teuthology.orchestra.run.smithi162.stdout: "num_blocked_ops": 0 2024-09-13T11:27:00.163 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-13T11:27:00.183 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_historic_ops 2024-09-13T11:27:00.261 INFO:tasks.ceph.osd.3.smithi177.stderr:2024-09-13T11:27:00.253+0000 7f10adba4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26574) UID: 0 2024-09-13T11:27:00.322 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-13T11:27:00.322 INFO:teuthology.orchestra.run.smithi162.stdout: "size": 20, 2024-09-13T11:27:00.322 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 600, 2024-09-13T11:27:00.322 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [ 2024-09-13T11:27:00.322 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:00.323 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:221526 3.2 3.ea98b382 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e191)", 2024-09-13T11:27:00.323 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:26:04.295546+0000", 2024-09-13T11:27:00.323 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 56.026231269, 2024-09-13T11:27:00.323 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.73989113200000001, 2024-09-13T11:27:00.323 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:27:00.323 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:27:00.323 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-09-13T11:27:00.323 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:27:00.323 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:27:00.323 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:27:00.323 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 221526 2024-09-13T11:27:00.323 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:00.323 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:27:00.323 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:00.323 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:27:00.324 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:04.295546+0000", 2024-09-13T11:27:00.324 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:00.324 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:00.324 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:00.324 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:27:00.324 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:04.295546+0000", 2024-09-13T11:27:00.324 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:00.324 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:00.324 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:00.324 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:27:00.324 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:04.295551+0000", 2024-09-13T11:27:00.324 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.6940000000000003e-06 2024-09-13T11:27:00.325 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:00.325 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:00.325 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:27:00.325 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:04.295567+0000", 2024-09-13T11:27:00.325 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5896000000000002e-05 2024-09-13T11:27:00.325 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:00.325 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:00.325 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:27:00.325 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:04.295599+0000", 2024-09-13T11:27:00.325 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.243e-05 2024-09-13T11:27:00.325 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:00.325 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:00.325 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:27:00.325 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:04.295622+0000", 2024-09-13T11:27:00.326 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.3149000000000001e-05 2024-09-13T11:27:00.326 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:00.326 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:00.326 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:27:00.326 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:04.295764+0000", 2024-09-13T11:27:00.326 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00014193300000000001 2024-09-13T11:27:00.326 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:00.326 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:00.326 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-09-13T11:27:00.326 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:04.295778+0000", 2024-09-13T11:27:00.326 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3763e-05 2024-09-13T11:27:00.326 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:00.326 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:00.326 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:27:00.327 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:05.035420+0000", 2024-09-13T11:27:00.327 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.73964228300000001 2024-09-13T11:27:00.327 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:00.327 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:00.327 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:27:00.327 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:05.035437+0000", 2024-09-13T11:27:00.327 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.6983999999999999e-05 2024-09-13T11:27:00.327 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:00.327 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:27:00.327 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:00.327 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:00.327 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:00.327 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:221528 3.15 3.b7a45bf5 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e191)", 2024-09-13T11:27:00.328 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:26:04.296174+0000", 2024-09-13T11:27:00.328 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 56.025603623999999, 2024-09-13T11:27:00.328 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.73909410399999997, 2024-09-13T11:27:00.328 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:27:00.328 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:27:00.328 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-09-13T11:27:00.328 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:27:00.328 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:27:00.328 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:27:00.328 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 221528 2024-09-13T11:27:00.328 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:00.328 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:27:00.328 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:00.329 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:27:00.329 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:04.296174+0000", 2024-09-13T11:27:00.329 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:00.329 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:00.329 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:00.329 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:27:00.329 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:04.296174+0000", 2024-09-13T11:27:00.329 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:00.329 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:00.329 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:00.329 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:27:00.329 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:04.296188+0000", 2024-09-13T11:27:00.329 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4037e-05 2024-09-13T11:27:00.330 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:00.330 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:00.330 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:27:00.330 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:04.296201+0000", 2024-09-13T11:27:00.330 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3096e-05 2024-09-13T11:27:00.330 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:00.330 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:00.330 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:27:00.330 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:04.296242+0000", 2024-09-13T11:27:00.330 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.1103000000000003e-05 2024-09-13T11:27:00.330 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:00.330 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:00.330 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:27:00.330 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:04.296264+0000", 2024-09-13T11:27:00.331 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.173e-05 2024-09-13T11:27:00.331 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:00.331 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:00.331 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:27:00.331 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:04.296410+0000", 2024-09-13T11:27:00.331 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00014574600000000001 2024-09-13T11:27:00.331 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:00.331 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:00.331 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-09-13T11:27:00.331 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:04.296424+0000", 2024-09-13T11:27:00.331 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3926e-05 2024-09-13T11:27:00.331 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:00.331 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:00.331 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:27:00.331 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:05.035221+0000", 2024-09-13T11:27:00.332 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.73879784999999998 2024-09-13T11:27:00.332 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:00.332 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:00.332 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:27:00.332 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:05.035268+0000", 2024-09-13T11:27:00.332 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.6616000000000002e-05 2024-09-13T11:27:00.332 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:00.332 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:27:00.332 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:00.332 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:00.332 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:00.332 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:221544 3.12 3.ec3fbdf2 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e191)", 2024-09-13T11:27:00.332 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:26:04.384452+0000", 2024-09-13T11:27:00.333 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 55.937325031999997, 2024-09-13T11:27:00.333 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.65065541999999998, 2024-09-13T11:27:00.333 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:27:00.333 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:27:00.333 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-09-13T11:27:00.333 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:27:00.333 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:27:00.333 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:27:00.333 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 221544 2024-09-13T11:27:00.333 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:00.333 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:27:00.333 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:00.333 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:27:00.333 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:04.384452+0000", 2024-09-13T11:27:00.334 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:00.334 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:00.334 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:00.334 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:27:00.334 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:04.384452+0000", 2024-09-13T11:27:00.334 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:00.334 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:00.334 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:00.334 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:27:00.334 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:04.384457+0000", 2024-09-13T11:27:00.334 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.6469999999999997e-06 2024-09-13T11:27:00.334 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:00.334 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:00.334 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:27:00.335 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:04.384470+0000", 2024-09-13T11:27:00.335 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3186999999999999e-05 2024-09-13T11:27:00.335 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:00.335 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:00.335 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:27:00.335 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:04.384504+0000", 2024-09-13T11:27:00.335 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.3413e-05 2024-09-13T11:27:00.335 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:00.335 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:00.335 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:27:00.335 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:04.384526+0000", 2024-09-13T11:27:00.335 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.2690000000000001e-05 2024-09-13T11:27:00.335 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:00.335 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:00.336 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:27:00.336 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:04.384653+0000", 2024-09-13T11:27:00.336 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00012697199999999999 2024-09-13T11:27:00.336 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:00.336 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:00.336 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-09-13T11:27:00.336 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:04.384668+0000", 2024-09-13T11:27:00.336 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4936000000000001e-05 2024-09-13T11:27:00.336 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:00.336 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:00.336 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:27:00.336 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:05.035087+0000", 2024-09-13T11:27:00.336 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.65041902900000004 2024-09-13T11:27:00.337 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:00.337 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:00.337 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:27:00.337 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:05.035108+0000", 2024-09-13T11:27:00.337 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.0545999999999999e-05 2024-09-13T11:27:00.337 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:00.337 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:27:00.337 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:00.337 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:00.337 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:00.337 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPush(3.24 198/191 [PushOp(3:259c1bd3:::benchmark_data_smithi177_31116_object114151:head, version: 125'4839, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:259c1bd3:::benchmark_data_smithi177_31116_object114151:head@125'4839, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:27:00.337 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:26:12.041809+0000", 2024-09-13T11:27:00.338 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 48.279968383000003, 2024-09-13T11:27:00.338 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.195123457, 2024-09-13T11:27:00.338 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:27:00.338 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:27:00.338 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:27:00.338 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:27:00.338 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:00.338 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:27:00.338 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:12.041809+0000", 2024-09-13T11:27:00.338 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:00.338 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:00.338 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:00.338 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:27:00.338 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:12.041809+0000", 2024-09-13T11:27:00.339 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:00.339 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:00.339 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:00.339 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:27:00.339 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:12.041811+0000", 2024-09-13T11:27:00.339 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.1469999999999999e-06 2024-09-13T11:27:00.339 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:00.339 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:00.339 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:27:00.339 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:12.041875+0000", 2024-09-13T11:27:00.339 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.3785e-05 2024-09-13T11:27:00.339 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:00.339 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:00.339 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:27:00.339 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:12.235597+0000", 2024-09-13T11:27:00.340 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.193721853 2024-09-13T11:27:00.340 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:00.340 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:00.340 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:27:00.340 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:12.235610+0000", 2024-09-13T11:27:00.340 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.31e-05 2024-09-13T11:27:00.340 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:00.340 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:00.340 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:27:00.340 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:12.235733+0000", 2024-09-13T11:27:00.340 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00012339600000000001 2024-09-13T11:27:00.340 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:00.340 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:00.341 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:27:00.341 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:12.235744+0000", 2024-09-13T11:27:00.341 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0665999999999999e-05 2024-09-13T11:27:00.341 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:00.341 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:00.341 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:27:00.341 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:12.236933+0000", 2024-09-13T11:27:00.341 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00118851 2024-09-13T11:27:00.341 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:00.341 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:27:00.341 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:00.341 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:00.341 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:00.342 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.4665.0:233786 3.1f e200/192 3:fbb85971:::benchmark_data_smithi177_31116_object93483:head v 200'8854, mlcod=200'8851)", 2024-09-13T11:27:00.342 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:26:16.784220+0000", 2024-09-13T11:27:00.342 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 43.537557855000003, 2024-09-13T11:27:00.342 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.138243172, 2024-09-13T11:27:00.342 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:27:00.342 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:27:00.342 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:27:00.342 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:27:00.342 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:00.342 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:27:00.342 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:16.784220+0000", 2024-09-13T11:27:00.342 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:00.342 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:00.343 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:00.343 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:27:00.343 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:16.784220+0000", 2024-09-13T11:27:00.343 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:00.343 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:00.343 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:00.343 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:27:00.343 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:16.784224+0000", 2024-09-13T11:27:00.343 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.1860000000000002e-06 2024-09-13T11:27:00.343 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:00.343 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:00.343 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:27:00.343 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:16.784240+0000", 2024-09-13T11:27:00.343 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5707000000000001e-05 2024-09-13T11:27:00.343 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:00.344 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:00.344 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:27:00.344 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:16.917820+0000", 2024-09-13T11:27:00.344 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.13358021 2024-09-13T11:27:00.344 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:00.344 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:00.344 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:27:00.344 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:16.917838+0000", 2024-09-13T11:27:00.344 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.7853999999999999e-05 2024-09-13T11:27:00.344 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:00.344 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:00.344 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:27:00.344 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:16.917982+0000", 2024-09-13T11:27:00.345 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00014408499999999999 2024-09-13T11:27:00.345 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:00.345 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:00.345 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:27:00.345 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:16.918024+0000", 2024-09-13T11:27:00.345 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.2345999999999997e-05 2024-09-13T11:27:00.345 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:00.345 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:00.345 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:27:00.345 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:16.922345+0000", 2024-09-13T11:27:00.345 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0043210480000000001 2024-09-13T11:27:00.345 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:00.345 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:00.346 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:27:00.346 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:16.922463+0000", 2024-09-13T11:27:00.346 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000117736 2024-09-13T11:27:00.346 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:00.346 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:27:00.346 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:00.346 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:00.346 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:00.346 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:234164 3.22 3.d472cf62 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e201)", 2024-09-13T11:27:00.346 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:26:17.191424+0000", 2024-09-13T11:27:00.346 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 43.130353913, 2024-09-13T11:27:00.346 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.95096563000000001, 2024-09-13T11:27:00.346 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:27:00.346 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:27:00.346 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-09-13T11:27:00.347 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:27:00.347 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:27:00.347 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:27:00.347 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 234164 2024-09-13T11:27:00.347 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:00.347 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:27:00.347 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:00.347 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:27:00.347 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:17.191424+0000", 2024-09-13T11:27:00.347 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:00.347 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:00.347 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:00.347 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:27:00.347 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:17.191424+0000", 2024-09-13T11:27:00.348 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:00.348 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:00.348 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:00.348 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:27:00.348 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:17.191426+0000", 2024-09-13T11:27:00.348 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.3269999999999999e-06 2024-09-13T11:27:00.348 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:00.348 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:00.348 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:27:00.348 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:17.191434+0000", 2024-09-13T11:27:00.348 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.5490000000000006e-06 2024-09-13T11:27:00.348 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:00.348 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:00.349 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:27:00.349 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:17.191450+0000", 2024-09-13T11:27:00.349 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.596e-05 2024-09-13T11:27:00.349 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:00.349 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:00.349 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:27:00.349 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:17.191458+0000", 2024-09-13T11:27:00.349 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.6070000000000002e-06 2024-09-13T11:27:00.349 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:00.349 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:00.349 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:27:00.349 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:17.191518+0000", 2024-09-13T11:27:00.349 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.9488999999999999e-05 2024-09-13T11:27:00.349 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:00.350 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:00.350 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-09-13T11:27:00.350 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:17.191526+0000", 2024-09-13T11:27:00.350 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.8049999999999996e-06 2024-09-13T11:27:00.350 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:00.350 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:00.350 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:27:00.350 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:18.142341+0000", 2024-09-13T11:27:00.350 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.95081438299999999 2024-09-13T11:27:00.350 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:00.350 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:00.350 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:27:00.350 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:18.142389+0000", 2024-09-13T11:27:00.351 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.8510000000000001e-05 2024-09-13T11:27:00.351 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:00.351 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:27:00.351 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:00.351 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:00.351 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:00.351 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:234214 3.22 3.3ebf73a2 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e201)", 2024-09-13T11:27:00.351 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:26:17.239598+0000", 2024-09-13T11:27:00.351 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 43.082179642, 2024-09-13T11:27:00.351 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.90310166300000005, 2024-09-13T11:27:00.351 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:27:00.351 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:27:00.351 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-09-13T11:27:00.351 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:27:00.352 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:27:00.352 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:27:00.352 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 234214 2024-09-13T11:27:00.352 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:00.352 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:27:00.352 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:00.352 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:27:00.352 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:17.239598+0000", 2024-09-13T11:27:00.352 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:00.352 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:00.352 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:00.352 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:27:00.352 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:17.239598+0000", 2024-09-13T11:27:00.353 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:00.353 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:00.353 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:00.353 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:27:00.353 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:17.239601+0000", 2024-09-13T11:27:00.353 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.021e-06 2024-09-13T11:27:00.353 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:00.353 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:00.353 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:27:00.353 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:17.239609+0000", 2024-09-13T11:27:00.353 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.9459999999999998e-06 2024-09-13T11:27:00.353 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:00.353 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:00.353 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:27:00.354 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:17.239628+0000", 2024-09-13T11:27:00.354 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.9029e-05 2024-09-13T11:27:00.354 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:00.354 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:00.354 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:27:00.354 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:17.239637+0000", 2024-09-13T11:27:00.354 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.7530000000000007e-06 2024-09-13T11:27:00.354 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:00.354 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:00.354 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:27:00.354 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:17.239716+0000", 2024-09-13T11:27:00.354 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.9789000000000005e-05 2024-09-13T11:27:00.354 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:00.354 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:00.355 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-09-13T11:27:00.355 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:17.239729+0000", 2024-09-13T11:27:00.355 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2215000000000001e-05 2024-09-13T11:27:00.355 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:00.355 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:00.355 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:27:00.355 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:18.142672+0000", 2024-09-13T11:27:00.355 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.90294309299999997 2024-09-13T11:27:00.355 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:00.355 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:00.355 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:27:00.355 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:18.142700+0000", 2024-09-13T11:27:00.355 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.7817000000000002e-05 2024-09-13T11:27:00.355 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:00.356 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:27:00.356 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:00.356 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:00.356 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:00.356 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:234350 3.22 3.6c9f6f62 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e201)", 2024-09-13T11:27:00.356 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:26:17.412568+0000", 2024-09-13T11:27:00.356 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 42.909209842999999, 2024-09-13T11:27:00.356 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.73023408300000003, 2024-09-13T11:27:00.356 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:27:00.356 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:27:00.356 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-09-13T11:27:00.356 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:27:00.356 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:27:00.356 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:27:00.357 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 234350 2024-09-13T11:27:00.357 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:00.357 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:27:00.357 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:00.357 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:27:00.357 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:17.412568+0000", 2024-09-13T11:27:00.357 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:00.357 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:00.357 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:00.357 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:27:00.357 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:17.412568+0000", 2024-09-13T11:27:00.357 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:00.357 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:00.357 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:00.358 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:27:00.358 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:17.412571+0000", 2024-09-13T11:27:00.358 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.044e-06 2024-09-13T11:27:00.358 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:00.358 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:00.358 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:27:00.358 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:17.412579+0000", 2024-09-13T11:27:00.358 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.7639999999999994e-06 2024-09-13T11:27:00.358 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:00.358 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:00.358 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:27:00.358 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:17.412595+0000", 2024-09-13T11:27:00.358 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5637999999999998e-05 2024-09-13T11:27:00.358 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:00.359 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:00.359 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:27:00.359 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:17.412606+0000", 2024-09-13T11:27:00.359 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1175999999999999e-05 2024-09-13T11:27:00.359 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:00.359 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:00.359 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:27:00.359 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:17.412690+0000", 2024-09-13T11:27:00.359 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.3486000000000007e-05 2024-09-13T11:27:00.359 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:00.359 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:00.359 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-09-13T11:27:00.359 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:17.412703+0000", 2024-09-13T11:27:00.360 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3196e-05 2024-09-13T11:27:00.360 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:00.360 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:00.360 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:27:00.360 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:18.142767+0000", 2024-09-13T11:27:00.360 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.73006412600000004 2024-09-13T11:27:00.360 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:00.360 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:00.360 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:27:00.360 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:18.142802+0000", 2024-09-13T11:27:00.360 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.4653000000000002e-05 2024-09-13T11:27:00.360 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:00.360 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:27:00.360 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:00.361 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:00.361 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:00.361 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGRecoveryDelete(3.6 e201,191 3:60767a7b:::benchmark_data_smithi177_31116_object14927:head,151'5598)", 2024-09-13T11:27:00.361 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:26:17.890539+0000", 2024-09-13T11:27:00.361 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 42.431238620999999, 2024-09-13T11:27:00.361 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.29203489500000002, 2024-09-13T11:27:00.361 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:27:00.361 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:27:00.361 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:27:00.361 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:27:00.361 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:00.361 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:27:00.361 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:17.890539+0000", 2024-09-13T11:27:00.361 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:00.362 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:00.362 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:00.362 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:27:00.362 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:17.890539+0000", 2024-09-13T11:27:00.362 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:00.362 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:00.362 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:00.362 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:27:00.362 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:17.890541+0000", 2024-09-13T11:27:00.362 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.795e-06 2024-09-13T11:27:00.362 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:00.362 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:00.362 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:27:00.362 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:17.890547+0000", 2024-09-13T11:27:00.363 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.3770000000000001e-06 2024-09-13T11:27:00.363 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:00.363 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:00.363 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:27:00.363 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:18.182175+0000", 2024-09-13T11:27:00.363 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.29162787000000001 2024-09-13T11:27:00.363 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:00.363 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:00.363 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:27:00.363 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:18.182190+0000", 2024-09-13T11:27:00.363 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4976e-05 2024-09-13T11:27:00.363 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:00.363 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:00.363 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:27:00.364 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:18.182397+0000", 2024-09-13T11:27:00.364 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00020751499999999999 2024-09-13T11:27:00.364 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:00.364 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:00.364 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:27:00.364 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:18.182425+0000", 2024-09-13T11:27:00.364 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.7076999999999998e-05 2024-09-13T11:27:00.364 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:00.364 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:00.364 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:27:00.364 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:18.182574+0000", 2024-09-13T11:27:00.364 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000149285 2024-09-13T11:27:00.364 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:00.365 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:27:00.365 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:00.365 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:00.365 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:00.365 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:234778 3.22 3.36a6aba2 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e201)", 2024-09-13T11:27:00.365 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:26:17.909436+0000", 2024-09-13T11:27:00.365 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 42.412341638999997, 2024-09-13T11:27:00.365 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.233461425, 2024-09-13T11:27:00.365 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:27:00.365 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:27:00.365 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-09-13T11:27:00.365 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:27:00.365 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:27:00.365 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:27:00.366 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 234778 2024-09-13T11:27:00.366 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:00.366 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:27:00.366 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:00.366 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:27:00.366 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:17.909436+0000", 2024-09-13T11:27:00.366 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:00.366 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:00.366 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:00.366 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:27:00.366 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:17.909436+0000", 2024-09-13T11:27:00.366 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:00.366 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:00.367 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:00.367 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:27:00.367 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:17.909439+0000", 2024-09-13T11:27:00.367 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.717e-06 2024-09-13T11:27:00.367 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:00.367 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:00.367 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:27:00.367 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:17.909446+0000", 2024-09-13T11:27:00.367 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.1980000000000004e-06 2024-09-13T11:27:00.367 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:00.367 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:00.367 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:27:00.367 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:17.909462+0000", 2024-09-13T11:27:00.368 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5809999999999999e-05 2024-09-13T11:27:00.368 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:00.368 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:00.368 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:27:00.368 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:17.909471+0000", 2024-09-13T11:27:00.368 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.3100000000000006e-06 2024-09-13T11:27:00.368 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:00.368 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:00.368 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:27:00.368 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:17.909554+0000", 2024-09-13T11:27:00.368 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.2917000000000006e-05 2024-09-13T11:27:00.368 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:00.368 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:00.368 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-09-13T11:27:00.369 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:17.909569+0000", 2024-09-13T11:27:00.369 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5201000000000001e-05 2024-09-13T11:27:00.369 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:00.369 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:00.369 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:27:00.369 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:18.142863+0000", 2024-09-13T11:27:00.369 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.23329387900000001 2024-09-13T11:27:00.369 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:00.369 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:00.369 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:27:00.369 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:18.142897+0000", 2024-09-13T11:27:00.369 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.4393000000000001e-05 2024-09-13T11:27:00.369 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:00.369 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:27:00.370 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:00.370 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:00.370 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:00.370 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:234793 3.22 3.215b2da2 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e201)", 2024-09-13T11:27:00.370 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:26:17.923001+0000", 2024-09-13T11:27:00.370 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 42.398776253999998, 2024-09-13T11:27:00.370 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.220029066, 2024-09-13T11:27:00.370 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:27:00.370 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:27:00.370 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-09-13T11:27:00.370 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:27:00.370 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:27:00.370 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:27:00.370 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 234793 2024-09-13T11:27:00.371 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:00.371 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:27:00.371 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:00.371 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:27:00.371 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:17.923001+0000", 2024-09-13T11:27:00.371 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:00.371 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:00.371 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:00.371 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:27:00.371 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:17.923001+0000", 2024-09-13T11:27:00.371 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:00.371 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:00.371 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:00.372 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:27:00.372 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:17.923004+0000", 2024-09-13T11:27:00.372 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.4619999999999999e-06 2024-09-13T11:27:00.372 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:00.372 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:00.372 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:27:00.372 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:17.923010+0000", 2024-09-13T11:27:00.372 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.759e-06 2024-09-13T11:27:00.372 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:00.372 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:00.372 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:27:00.372 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:17.923022+0000", 2024-09-13T11:27:00.372 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1361e-05 2024-09-13T11:27:00.373 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:00.373 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:00.373 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:27:00.373 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:17.923029+0000", 2024-09-13T11:27:00.373 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.7349999999999999e-06 2024-09-13T11:27:00.373 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:00.373 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:00.373 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:27:00.373 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:17.923080+0000", 2024-09-13T11:27:00.373 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.1535e-05 2024-09-13T11:27:00.373 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:00.373 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:00.373 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-09-13T11:27:00.373 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:17.923087+0000", 2024-09-13T11:27:00.374 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.1389999999999998e-06 2024-09-13T11:27:00.374 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:00.374 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:00.374 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:27:00.374 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:18.142999+0000", 2024-09-13T11:27:00.374 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.21991205799999999 2024-09-13T11:27:00.374 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:00.374 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:00.374 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:27:00.374 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:18.143030+0000", 2024-09-13T11:27:00.374 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.1016999999999998e-05 2024-09-13T11:27:00.374 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:00.374 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:27:00.374 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:00.375 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:00.375 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:00.375 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPush(3.2 201/192 [PushOp(3:409d6ad2:::benchmark_data_smithi177_31116_object97365:head, version: 113'3767, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:409d6ad2:::benchmark_data_smithi177_31116_object97365:head@113'3767, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:27:00.375 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:26:17.942627+0000", 2024-09-13T11:27:00.375 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 42.379150967999998, 2024-09-13T11:27:00.375 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.25342641300000002, 2024-09-13T11:27:00.375 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:27:00.375 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:27:00.375 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:27:00.375 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:27:00.375 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:00.375 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:27:00.375 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:17.942627+0000", 2024-09-13T11:27:00.376 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:00.376 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:00.376 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:00.376 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:27:00.376 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:17.942627+0000", 2024-09-13T11:27:00.376 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:00.376 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:00.376 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:00.376 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:27:00.376 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:17.942628+0000", 2024-09-13T11:27:00.376 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3179999999999999e-06 2024-09-13T11:27:00.376 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:00.376 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:00.376 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:27:00.377 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:17.942665+0000", 2024-09-13T11:27:00.377 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.6748999999999997e-05 2024-09-13T11:27:00.377 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:00.377 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:00.377 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:27:00.377 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:18.192821+0000", 2024-09-13T11:27:00.377 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.25015650900000003 2024-09-13T11:27:00.377 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:00.377 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:00.377 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:27:00.377 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:18.192853+0000", 2024-09-13T11:27:00.377 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.1579999999999999e-05 2024-09-13T11:27:00.377 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:00.377 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:00.378 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:27:00.378 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:18.193147+0000", 2024-09-13T11:27:00.378 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000294424 2024-09-13T11:27:00.378 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:00.378 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:00.378 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:27:00.378 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:18.193191+0000", 2024-09-13T11:27:00.378 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.3463e-05 2024-09-13T11:27:00.378 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:00.378 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:00.378 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:27:00.378 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:18.196053+0000", 2024-09-13T11:27:00.378 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0028623699999999999 2024-09-13T11:27:00.378 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:00.379 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:27:00.379 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:00.379 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:00.379 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:00.379 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGRecoveryDelete(3.6 e204,191 3:60665223:::benchmark_data_smithi177_31116_object38983:head,168'5973)", 2024-09-13T11:27:00.379 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:26:21.915203+0000", 2024-09-13T11:27:00.379 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 38.406574765000002, 2024-09-13T11:27:00.379 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.144080137, 2024-09-13T11:27:00.379 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:27:00.379 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:27:00.379 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:27:00.379 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:27:00.379 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:00.379 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:27:00.379 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:21.915203+0000", 2024-09-13T11:27:00.380 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:00.380 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:00.380 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:00.380 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:27:00.380 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:21.915203+0000", 2024-09-13T11:27:00.380 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:00.380 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:00.380 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:00.380 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:27:00.380 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:21.915205+0000", 2024-09-13T11:27:00.380 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.9360000000000002e-06 2024-09-13T11:27:00.380 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:00.380 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:00.380 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:27:00.381 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:21.915215+0000", 2024-09-13T11:27:00.381 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0641e-05 2024-09-13T11:27:00.381 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:00.381 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:00.381 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:27:00.381 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:22.059132+0000", 2024-09-13T11:27:00.381 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.14391625399999999 2024-09-13T11:27:00.381 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:00.381 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:00.381 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:27:00.381 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:22.059141+0000", 2024-09-13T11:27:00.381 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.3300000000000005e-06 2024-09-13T11:27:00.381 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:00.382 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:00.382 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:27:00.382 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:22.059207+0000", 2024-09-13T11:27:00.382 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.5778000000000006e-05 2024-09-13T11:27:00.382 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:00.382 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:00.382 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:27:00.382 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:22.059214+0000", 2024-09-13T11:27:00.382 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.2760000000000003e-06 2024-09-13T11:27:00.382 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:00.382 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:00.382 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:27:00.382 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:22.059283+0000", 2024-09-13T11:27:00.382 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.8922000000000002e-05 2024-09-13T11:27:00.382 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:00.383 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:27:00.383 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:00.383 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:00.383 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:00.383 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPush(3.2 205/192 [PushOp(3:42b70fdc:::benchmark_data_smithi177_31116_object116542:head, version: 126'4065, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:42b70fdc:::benchmark_data_smithi177_31116_object116542:head@126'4065, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:27:00.383 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:26:23.424423+0000", 2024-09-13T11:27:00.383 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 36.897354096999997, 2024-09-13T11:27:00.383 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.178307523, 2024-09-13T11:27:00.383 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:27:00.383 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:27:00.383 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:27:00.383 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:27:00.383 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:00.383 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:27:00.384 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:23.424423+0000", 2024-09-13T11:27:00.384 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:00.384 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:00.384 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:00.384 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:27:00.384 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:23.424423+0000", 2024-09-13T11:27:00.384 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:00.384 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:00.384 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:00.384 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:27:00.384 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:23.424432+0000", 2024-09-13T11:27:00.384 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.4260000000000003e-06 2024-09-13T11:27:00.384 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:00.384 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:00.385 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:27:00.385 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:23.424502+0000", 2024-09-13T11:27:00.385 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.0476999999999996e-05 2024-09-13T11:27:00.385 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:00.385 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:00.385 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:27:00.385 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:23.599928+0000", 2024-09-13T11:27:00.385 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.17542566600000001 2024-09-13T11:27:00.385 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:00.385 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:00.385 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:27:00.385 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:23.599953+0000", 2024-09-13T11:27:00.385 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.5389999999999999e-05 2024-09-13T11:27:00.385 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:00.386 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:00.386 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:27:00.386 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:23.600144+0000", 2024-09-13T11:27:00.386 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00019051800000000001 2024-09-13T11:27:00.386 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:00.386 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:00.386 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:27:00.386 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:23.600164+0000", 2024-09-13T11:27:00.386 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.0230000000000001e-05 2024-09-13T11:27:00.386 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:00.386 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:00.386 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:27:00.386 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:23.602731+0000", 2024-09-13T11:27:00.386 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.002566816 2024-09-13T11:27:00.387 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:00.387 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:27:00.387 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:00.387 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:00.387 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:00.387 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.4665.0:252195 3.12 e211/192 3:4fa4efb0:::benchmark_data_smithi177_31116_object111892:head v 211'9304, mlcod=211'9301)", 2024-09-13T11:27:00.387 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:26:33.883066+0000", 2024-09-13T11:27:00.387 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 26.438711830999999, 2024-09-13T11:27:00.387 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.15765170100000001, 2024-09-13T11:27:00.387 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:27:00.387 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:27:00.387 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:27:00.387 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:27:00.387 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:00.388 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:27:00.388 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:33.883066+0000", 2024-09-13T11:27:00.388 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:00.388 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:00.388 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:00.388 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:27:00.388 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:33.883066+0000", 2024-09-13T11:27:00.388 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:00.388 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:00.388 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:00.388 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:27:00.388 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:33.883071+0000", 2024-09-13T11:27:00.388 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.7300000000000002e-06 2024-09-13T11:27:00.388 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:00.389 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:00.389 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:27:00.389 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:33.883079+0000", 2024-09-13T11:27:00.389 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.3100000000000003e-06 2024-09-13T11:27:00.389 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:00.389 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:00.389 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:27:00.389 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:34.036891+0000", 2024-09-13T11:27:00.389 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.15381271599999999 2024-09-13T11:27:00.389 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:00.389 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:00.389 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:27:00.389 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:34.036898+0000", 2024-09-13T11:27:00.389 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.1770000000000003e-06 2024-09-13T11:27:00.390 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:00.390 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:00.390 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:27:00.390 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:34.036961+0000", 2024-09-13T11:27:00.390 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.3016999999999997e-05 2024-09-13T11:27:00.390 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:00.390 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:00.390 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:27:00.390 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:34.036980+0000", 2024-09-13T11:27:00.390 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.9153000000000001e-05 2024-09-13T11:27:00.390 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:00.390 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:00.390 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:27:00.391 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:34.040661+0000", 2024-09-13T11:27:00.391 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0036812030000000001 2024-09-13T11:27:00.391 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:00.391 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:00.391 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:27:00.391 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:34.040717+0000", 2024-09-13T11:27:00.391 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.6394999999999998e-05 2024-09-13T11:27:00.391 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:00.391 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:27:00.391 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:00.391 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:00.391 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:00.391 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGRecoveryDelete(3.22 e223,218 3:46bd2d6a:::benchmark_data_smithi177_31116_object75483:head,187'5642)", 2024-09-13T11:27:00.391 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:26:48.314416+0000", 2024-09-13T11:27:00.392 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 12.007361885, 2024-09-13T11:27:00.392 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.13940249199999999, 2024-09-13T11:27:00.392 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:27:00.392 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:27:00.392 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:27:00.392 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:27:00.392 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:00.392 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:27:00.392 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:48.314416+0000", 2024-09-13T11:27:00.392 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:00.392 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:00.392 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:00.392 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:27:00.393 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:48.314416+0000", 2024-09-13T11:27:00.393 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:00.393 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:00.393 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:00.393 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:27:00.393 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:48.314419+0000", 2024-09-13T11:27:00.393 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.3270000000000001e-06 2024-09-13T11:27:00.393 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:00.393 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:00.393 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:27:00.393 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:48.314430+0000", 2024-09-13T11:27:00.393 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1486e-05 2024-09-13T11:27:00.393 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:00.394 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:00.394 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:27:00.394 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:48.453443+0000", 2024-09-13T11:27:00.394 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.13901245800000001 2024-09-13T11:27:00.394 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:00.394 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:00.394 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:27:00.394 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:48.453459+0000", 2024-09-13T11:27:00.394 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.6002999999999999e-05 2024-09-13T11:27:00.394 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:00.394 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:00.394 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:27:00.394 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:48.453660+0000", 2024-09-13T11:27:00.394 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00020151199999999999 2024-09-13T11:27:00.395 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:00.395 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:00.395 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:27:00.395 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:48.453678+0000", 2024-09-13T11:27:00.395 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.7989999999999999e-05 2024-09-13T11:27:00.395 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:00.395 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:00.395 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:27:00.395 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:48.453818+0000", 2024-09-13T11:27:00.395 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00013971599999999999 2024-09-13T11:27:00.395 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:00.395 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:27:00.395 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:00.396 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:00.396 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:00.396 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGRecoveryDelete(3.22 e223,218 3:47eb4e8b:::benchmark_data_smithi177_31116_object75722:head,187'5647)", 2024-09-13T11:27:00.396 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:26:48.562782+0000", 2024-09-13T11:27:00.396 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 11.758995285999999, 2024-09-13T11:27:00.396 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.307749461, 2024-09-13T11:27:00.396 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:27:00.396 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:27:00.396 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:27:00.396 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:27:00.396 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:00.396 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:27:00.396 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:48.562782+0000", 2024-09-13T11:27:00.397 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:00.397 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:00.397 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:00.397 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:27:00.397 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:48.562782+0000", 2024-09-13T11:27:00.397 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:00.397 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:00.397 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:00.397 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:27:00.397 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:48.562785+0000", 2024-09-13T11:27:00.397 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.9900000000000002e-06 2024-09-13T11:27:00.397 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:00.397 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:00.397 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:27:00.398 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:48.562796+0000", 2024-09-13T11:27:00.398 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1046999999999999e-05 2024-09-13T11:27:00.398 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:00.398 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:00.398 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:27:00.398 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:48.862609+0000", 2024-09-13T11:27:00.398 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.29981280399999999 2024-09-13T11:27:00.398 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:00.398 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:00.398 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:27:00.398 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:48.862624+0000", 2024-09-13T11:27:00.398 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5214e-05 2024-09-13T11:27:00.398 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:00.399 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:00.399 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:27:00.399 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:48.870361+0000", 2024-09-13T11:27:00.399 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0077370670000000002 2024-09-13T11:27:00.399 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:00.399 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:00.399 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:27:00.399 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:48.870385+0000", 2024-09-13T11:27:00.399 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.3693000000000002e-05 2024-09-13T11:27:00.399 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:00.399 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:00.399 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:27:00.399 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:48.870532+0000", 2024-09-13T11:27:00.399 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00014664600000000001 2024-09-13T11:27:00.400 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:00.400 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:27:00.400 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:00.400 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:00.400 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:00.400 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:266847 3.15 3:aced3906:::benchmark_data_smithi177_31116_object126544:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e223)", 2024-09-13T11:27:00.400 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:26:48.588305+0000", 2024-09-13T11:27:00.400 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 11.733472098, 2024-09-13T11:27:00.400 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.29333894599999999, 2024-09-13T11:27:00.400 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:27:00.400 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:27:00.400 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:27:00.400 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:27:00.401 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:27:00.401 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:27:00.401 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 266847 2024-09-13T11:27:00.401 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:00.401 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:27:00.401 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:00.401 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:27:00.401 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:48.588305+0000", 2024-09-13T11:27:00.401 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:00.401 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:00.401 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:00.401 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:27:00.401 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:48.588305+0000", 2024-09-13T11:27:00.401 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:00.401 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:00.402 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:00.402 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:27:00.402 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:48.588309+0000", 2024-09-13T11:27:00.402 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.41e-06 2024-09-13T11:27:00.402 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:00.402 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:00.402 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:27:00.402 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:48.588319+0000", 2024-09-13T11:27:00.402 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.9359999999999994e-06 2024-09-13T11:27:00.402 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:00.402 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:00.402 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:27:00.402 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:48.588348+0000", 2024-09-13T11:27:00.403 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.8940000000000001e-05 2024-09-13T11:27:00.403 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:00.403 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:00.403 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:27:00.403 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:48.588362+0000", 2024-09-13T11:27:00.403 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4752999999999999e-05 2024-09-13T11:27:00.403 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:00.403 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:00.403 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:27:00.403 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:48.588510+0000", 2024-09-13T11:27:00.403 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00014728500000000001 2024-09-13T11:27:00.403 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:00.403 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:00.403 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:27:00.404 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:48.588731+0000", 2024-09-13T11:27:00.404 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000220762 2024-09-13T11:27:00.404 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:00.404 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:00.404 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 0", 2024-09-13T11:27:00.404 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:48.588906+0000", 2024-09-13T11:27:00.404 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00017593900000000001 2024-09-13T11:27:00.404 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:00.404 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:00.404 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:27:00.404 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:48.594579+0000", 2024-09-13T11:27:00.404 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.005672351 2024-09-13T11:27:00.404 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:00.404 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:00.405 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:27:00.405 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:48.881375+0000", 2024-09-13T11:27:00.405 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.28679606600000002 2024-09-13T11:27:00.405 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:00.405 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:00.405 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:27:00.405 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:48.881524+0000", 2024-09-13T11:27:00.405 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000148935 2024-09-13T11:27:00.405 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:00.405 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:00.405 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:27:00.405 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:48.881644+0000", 2024-09-13T11:27:00.405 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000120569 2024-09-13T11:27:00.405 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:00.406 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:27:00.406 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:00.406 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:00.406 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:00.406 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.4665.0:266884 3.1f e224/223 3:fb121a9e:::benchmark_data_smithi177_31116_object126581:head v 224'9867, mlcod=224'9865)", 2024-09-13T11:27:00.406 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:26:49.276350+0000", 2024-09-13T11:27:00.406 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 11.04542745, 2024-09-13T11:27:00.406 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.53992503999999997, 2024-09-13T11:27:00.406 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:27:00.406 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:27:00.406 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:27:00.406 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:27:00.407 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:00.407 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:27:00.407 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:49.276350+0000", 2024-09-13T11:27:00.407 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:00.407 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:00.407 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:00.407 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:27:00.407 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:49.276350+0000", 2024-09-13T11:27:00.407 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:00.407 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:00.407 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:00.407 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:27:00.407 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:49.276352+0000", 2024-09-13T11:27:00.408 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.6020000000000001e-06 2024-09-13T11:27:00.408 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:00.408 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:00.408 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:27:00.408 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:49.276360+0000", 2024-09-13T11:27:00.408 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.6319999999999997e-06 2024-09-13T11:27:00.408 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:00.408 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:00.408 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:27:00.408 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:49.810598+0000", 2024-09-13T11:27:00.408 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.53423792199999998 2024-09-13T11:27:00.408 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:00.408 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:00.409 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:27:00.409 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:49.810616+0000", 2024-09-13T11:27:00.409 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.7456999999999999e-05 2024-09-13T11:27:00.409 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:00.409 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:00.409 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:27:00.409 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:49.810770+0000", 2024-09-13T11:27:00.409 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00015476199999999999 2024-09-13T11:27:00.409 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:00.409 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:00.409 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:27:00.409 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:49.810838+0000", 2024-09-13T11:27:00.409 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.7788999999999998e-05 2024-09-13T11:27:00.410 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:00.410 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:00.410 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:27:00.410 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:49.816185+0000", 2024-09-13T11:27:00.410 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0053467369999999998 2024-09-13T11:27:00.410 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:00.410 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:00.410 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:27:00.410 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:49.816275+0000", 2024-09-13T11:27:00.410 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.0138999999999999e-05 2024-09-13T11:27:00.410 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:00.410 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:27:00.410 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:00.410 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:00.411 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:00.411 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.4665.0:273268 3.12 e232/224 3:48daa4a0:::benchmark_data_smithi177_31116_object132965:head v 232'9853, mlcod=231'9850)", 2024-09-13T11:27:00.411 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:26:59.476629+0000", 2024-09-13T11:27:00.411 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 0.84514897600000005, 2024-09-13T11:27:00.411 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.161299567, 2024-09-13T11:27:00.411 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:27:00.411 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:27:00.411 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:27:00.411 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:27:00.411 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:00.411 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:27:00.411 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:59.476629+0000", 2024-09-13T11:27:00.411 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:00.411 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:00.412 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:00.412 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:27:00.412 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:59.476629+0000", 2024-09-13T11:27:00.412 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:00.412 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:00.412 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:00.412 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:27:00.412 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:59.476630+0000", 2024-09-13T11:27:00.412 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.066e-06 2024-09-13T11:27:00.412 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:00.412 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:00.412 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:27:00.412 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:59.476642+0000", 2024-09-13T11:27:00.412 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2486999999999999e-05 2024-09-13T11:27:00.413 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:00.413 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:00.413 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:27:00.413 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:59.632687+0000", 2024-09-13T11:27:00.413 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.156045403 2024-09-13T11:27:00.413 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:00.413 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:00.413 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:27:00.413 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:59.632694+0000", 2024-09-13T11:27:00.413 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.4699999999999999e-06 2024-09-13T11:27:00.413 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:00.413 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:00.413 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:27:00.413 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:59.632752+0000", 2024-09-13T11:27:00.414 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.8054999999999999e-05 2024-09-13T11:27:00.414 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:00.414 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:00.414 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:27:00.414 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:59.632778+0000", 2024-09-13T11:27:00.414 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.6060000000000001e-05 2024-09-13T11:27:00.414 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:00.414 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:00.414 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:27:00.414 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:59.637869+0000", 2024-09-13T11:27:00.414 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0050908760000000003 2024-09-13T11:27:00.414 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:00.414 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:00.414 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:27:00.414 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:59.637928+0000", 2024-09-13T11:27:00.415 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.9150000000000001e-05 2024-09-13T11:27:00.415 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:00.415 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:27:00.415 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:00.415 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:00.415 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:27:00.415 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-13T11:27:00.416 INFO:tasks.ceph.osd.10.smithi162.stderr:2024-09-13T11:27:00.415+0000 7fb59c9f0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26317) UID: 0 2024-09-13T11:27:00.417 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-13T11:27:00.425 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 15, 14] out_osds: [] dead_osds: [] live_osds: [0, 4, 8, 12, 5, 9, 13, 2, 6, 10, 3, 7, 11, 15, 1, 14] 2024-09-13T11:27:00.425 INFO:tasks.thrashosds.thrasher:choose_action: min_in 2 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-13T11:27:00.425 INFO:tasks.thrashosds.thrasher:Growing pool unique_pool_0 2024-09-13T11:27:00.426 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-13T11:27:00.517 INFO:tasks.ceph.osd.13.smithi092.stderr:2024-09-13T11:27:00.515+0000 7f3e5afea640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26350) UID: 0 2024-09-13T11:27:00.581 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-13T11:27:00.581 INFO:teuthology.orchestra.run.smithi001.stdout: "ops": [ 2024-09-13T11:27:00.581 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:00.581 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:274387 3.35 3:ae5cbac3:::benchmark_data_smithi177_31116_object134084:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e233)", 2024-09-13T11:27:00.581 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:27:00.580595+0000", 2024-09-13T11:27:00.581 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 0.00037884099999999999, 2024-09-13T11:27:00.581 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00039966999999999997, 2024-09-13T11:27:00.582 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:27:00.582 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:27:00.582 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "waiting for sub ops", 2024-09-13T11:27:00.582 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:27:00.582 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:27:00.582 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:27:00.582 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 274387 2024-09-13T11:27:00.582 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:00.582 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:27:00.582 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:00.582 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:27:00.582 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:27:00.580595+0000", 2024-09-13T11:27:00.582 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:27:00.582 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:00.583 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:00.583 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:27:00.583 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:27:00.580595+0000", 2024-09-13T11:27:00.583 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:27:00.583 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:00.583 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:00.583 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:27:00.583 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:27:00.580597+0000", 2024-09-13T11:27:00.583 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.1320000000000001e-06 2024-09-13T11:27:00.583 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:00.583 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:00.583 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:27:00.583 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:27:00.580604+0000", 2024-09-13T11:27:00.583 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.7490000000000001e-06 2024-09-13T11:27:00.584 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:00.584 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:00.584 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:27:00.584 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:27:00.580619+0000", 2024-09-13T11:27:00.584 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.5268999999999999e-05 2024-09-13T11:27:00.584 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:00.584 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:00.584 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:27:00.584 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:27:00.580627+0000", 2024-09-13T11:27:00.584 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.9119999999999998e-06 2024-09-13T11:27:00.584 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:00.584 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:00.584 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:27:00.585 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:27:00.580689+0000", 2024-09-13T11:27:00.585 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.1889000000000003e-05 2024-09-13T11:27:00.585 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:00.585 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:00.585 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:27:00.585 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:27:00.580856+0000", 2024-09-13T11:27:00.585 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00016641099999999999 2024-09-13T11:27:00.585 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:00.585 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:00.585 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 7,12", 2024-09-13T11:27:00.585 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:27:00.580983+0000", 2024-09-13T11:27:00.585 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00012737800000000001 2024-09-13T11:27:00.585 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:27:00.585 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:27:00.586 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:27:00.586 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:27:00.586 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-09-13T11:27:00.586 INFO:teuthology.orchestra.run.smithi001.stdout: "num_ops": 1 2024-09-13T11:27:00.586 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-13T11:27:00.605 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-13T11:27:00.617 INFO:tasks.ceph.osd.14.smithi162.stderr:2024-09-13T11:27:00.615+0000 7f3df5903640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33154) UID: 0 2024-09-13T11:27:00.717 INFO:tasks.ceph.osd.13.smithi092.stderr:2024-09-13T11:27:00.715+0000 7f3e5afea640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26350) UID: 0 2024-09-13T11:27:00.774 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-13T11:27:00.774 INFO:teuthology.orchestra.run.smithi001.stdout: "ops": [], 2024-09-13T11:27:00.774 INFO:teuthology.orchestra.run.smithi001.stdout: "complaint_time": 30, 2024-09-13T11:27:00.774 INFO:teuthology.orchestra.run.smithi001.stdout: "num_blocked_ops": 0 2024-09-13T11:27:00.774 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-13T11:27:00.799 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-13T11:27:00.818 INFO:tasks.ceph.osd.10.smithi162.stderr:2024-09-13T11:27:00.815+0000 7fb59c9f0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26317) UID: 0 2024-09-13T11:27:00.910 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-13T11:27:00.911 INFO:teuthology.orchestra.run.smithi001.stderr:dumped all 2024-09-13T11:27:00.919 INFO:tasks.ceph.osd.7.smithi177.stderr:2024-09-13T11:27:00.917+0000 7f897b51d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26575) UID: 0 2024-09-13T11:27:00.934 INFO:teuthology.orchestra.run.smithi001.stdout:{"pg_ready":true,"pg_map":{"version":364,"stamp":"2024-09-13T11:27:00.420261+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":620790347,"num_objects":9469,"num_object_clones":0,"num_object_copies":18938,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2129,"num_objects_misplaced":2902,"num_objects_unfound":0,"num_objects_dirty":9469,"num_whiteouts":0,"num_read":87,"num_read_kb":74,"num_write":174185,"num_write_kb":3205267,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26908,"num_bytes_recovered":1357971494,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":478692,"ondisk_log_size":478692,"up":130,"acting":130,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":156,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":16,"kb":1509949440,"kb_used":3409588,"kb_used_data":1651252,"kb_used_omap":68,"kb_used_meta":1757883,"kb_avail":1506539852,"statfs":{"total":1546188226560,"available":1542696808448,"internally_reserved":0,"allocated":1690882048,"data_stored":1683996904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":70045,"internal_metadata":1800072803},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-123666432,"num_objects":-1886,"num_object_clones":0,"num_object_copies":-3772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-451,"num_objects_unfound":0,"num_objects_dirty":-1886,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-1161,"num_write_kb":-74432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-3081,"num_bytes_recovered":-212271104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.235928"},"pg_stats":[{"pgid":"3.37","version":"232'9618","reported_seq":15096,"reported_epoch":232,"state":"active+recovering+degraded+remapped","last_fresh":"2024-09-13T11:27:00.137369+0000","last_change":"2024-09-13T11:27:00.137568+0000","last_active":"2024-09-13T11:27:00.137369+0000","last_peered":"2024-09-13T11:27:00.137369+0000","last_clean":"2024-09-13T11:26:50.332529+0000","last_became_active":"2024-09-13T11:26:53.261387+0000","last_became_peered":"2024-09-13T11:26:53.261387+0000","last_unstale":"2024-09-13T11:27:00.137369+0000","last_undegraded":"2024-09-13T11:27:00.137369+0000","last_fullsized":"2024-09-13T11:27:00.137369+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":6,"last_scrub":"206'8719","last_scrub_stamp":"2024-09-13T11:26:25.299674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:26:25.299674+0000","objects_scrubbed":1237,"log_size":9618,"log_dups_size":0,"ondisk_log_size":9618,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":919,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6684672,"num_objects":102,"num_object_clones":0,"num_object_copies":204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3192,"num_write_kb":73344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":369,"num_bytes_recovered":15269888,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,12],"acting":[3,2],"avail_no_missing":["3","10"],"object_location_counts":[{"shards":"2,3","objects":102}],"blocked_by":[],"up_primary":2,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.36","version":"230'10016","reported_seq":15606,"reported_epoch":230,"state":"active+recovering+remapped","last_fresh":"2024-09-13T11:26:57.107287+0000","last_change":"2024-09-13T11:26:53.396883+0000","last_active":"2024-09-13T11:26:57.107287+0000","last_peered":"2024-09-13T11:26:57.107287+0000","last_clean":"2024-09-13T11:26:50.851925+0000","last_became_active":"2024-09-13T11:26:53.327720+0000","last_became_peered":"2024-09-13T11:26:53.327720+0000","last_unstale":"2024-09-13T11:26:57.107287+0000","last_undegraded":"2024-09-13T11:26:57.107287+0000","last_fullsized":"2024-09-13T11:26:57.107287+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":6,"last_scrub":"204'9074","last_scrub_stamp":"2024-09-13T11:26:21.348185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:26:21.348185+0000","objects_scrubbed":1320,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":479,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9371648,"num_objects":143,"num_object_clones":0,"num_object_copies":286,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":29,"num_objects_unfound":0,"num_objects_dirty":143,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3309,"num_write_kb":77600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":195,"num_bytes_recovered":12779520,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[11,0],"avail_no_missing":["11","15"],"object_location_counts":[{"shards":"0,11","objects":128},{"shards":"0,11,15","objects":1},{"shards":"11,15","objects":14}],"blocked_by":[],"up_primary":2,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.35","version":"229'9561","reported_seq":23178,"reported_epoch":229,"state":"active+recovering+remapped","last_fresh":"2024-09-13T11:26:56.145385+0000","last_change":"2024-09-13T11:26:53.263547+0000","last_active":"2024-09-13T11:26:56.145385+0000","last_peered":"2024-09-13T11:26:56.145385+0000","last_clean":"2024-09-13T11:26:50.245669+0000","last_became_active":"2024-09-13T11:26:53.227748+0000","last_became_peered":"2024-09-13T11:26:53.227748+0000","last_unstale":"2024-09-13T11:26:56.145385+0000","last_undegraded":"2024-09-13T11:26:56.145385+0000","last_fullsized":"2024-09-13T11:26:56.145385+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":6,"last_scrub":"181'7535","last_scrub_stamp":"2024-09-13T11:25:48.075568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:48.075568+0000","objects_scrubbed":2413,"log_size":9561,"log_dups_size":0,"ondisk_log_size":9561,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":991,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9895936,"num_objects":151,"num_object_clones":0,"num_object_copies":302,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":80,"num_objects_unfound":0,"num_objects_dirty":151,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3120,"num_write_kb":72160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2263,"num_bytes_recovered":148275200,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,12],"acting":[0,7],"avail_no_missing":["0","3","14"],"object_location_counts":[{"shards":"0,7","objects":111},{"shards":"0,14","objects":40}],"blocked_by":[],"up_primary":7,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.34","version":"232'9948","reported_seq":23216,"reported_epoch":232,"state":"active+recovering+remapped","last_fresh":"2024-09-13T11:27:00.131586+0000","last_change":"2024-09-13T11:27:00.145788+0000","last_active":"2024-09-13T11:27:00.131586+0000","last_peered":"2024-09-13T11:27:00.131586+0000","last_clean":"2024-09-13T11:26:50.965843+0000","last_became_active":"2024-09-13T11:26:53.259085+0000","last_became_peered":"2024-09-13T11:26:53.259085+0000","last_unstale":"2024-09-13T11:27:00.131586+0000","last_undegraded":"2024-09-13T11:27:00.131449+0000","last_fullsized":"2024-09-13T11:27:00.131586+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":6,"last_scrub":"174'7321","last_scrub_stamp":"2024-09-13T11:25:38.085667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:38.085667+0000","objects_scrubbed":2938,"log_size":9948,"log_dups_size":0,"ondisk_log_size":9948,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":431,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5734400,"num_objects":88,"num_object_clones":0,"num_object_copies":176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":88,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3358,"num_write_kb":79296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2237,"num_bytes_recovered":145195008,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,11],"acting":[3,10],"avail_no_missing":["3","14","15"],"object_location_counts":[{"shards":"3,10","objects":88}],"blocked_by":[],"up_primary":10,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.33","version":"229'9827","reported_seq":13741,"reported_epoch":229,"state":"active+recovering+remapped","last_fresh":"2024-09-13T11:26:56.229406+0000","last_change":"2024-09-13T11:26:53.317459+0000","last_active":"2024-09-13T11:26:56.229406+0000","last_peered":"2024-09-13T11:26:56.229406+0000","last_clean":"2024-09-13T11:26:50.882386+0000","last_became_active":"2024-09-13T11:26:53.282703+0000","last_became_peered":"2024-09-13T11:26:53.282703+0000","last_unstale":"2024-09-13T11:26:56.229406+0000","last_undegraded":"2024-09-13T11:26:56.229406+0000","last_fullsized":"2024-09-13T11:26:56.229406+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":6,"last_scrub":"213'9393","last_scrub_stamp":"2024-09-13T11:26:36.941193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:26:36.941193+0000","objects_scrubbed":798,"log_size":9827,"log_dups_size":0,"ondisk_log_size":9827,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":119,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10125312,"num_objects":155,"num_object_clones":0,"num_object_copies":310,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":50,"num_objects_unfound":0,"num_objects_dirty":155,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3243,"num_write_kb":77024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":177,"num_bytes_recovered":11599872,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[7,2],"avail_no_missing":["7","11"],"object_location_counts":[{"shards":"2,7","objects":130},{"shards":"7,11","objects":25}],"blocked_by":[],"up_primary":6,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.32","version":"229'9797","reported_seq":21518,"reported_epoch":229,"state":"active+recovering+remapped","last_fresh":"2024-09-13T11:26:56.072468+0000","last_change":"2024-09-13T11:26:53.324636+0000","last_active":"2024-09-13T11:26:56.072468+0000","last_peered":"2024-09-13T11:26:56.072468+0000","last_clean":"2024-09-13T11:26:03.227632+0000","last_became_active":"2024-09-13T11:26:53.252695+0000","last_became_peered":"2024-09-13T11:26:53.252695+0000","last_unstale":"2024-09-13T11:26:56.072468+0000","last_undegraded":"2024-09-13T11:26:56.072468+0000","last_fullsized":"2024-09-13T11:26:56.072468+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":6,"last_scrub":"186'8089","last_scrub_stamp":"2024-09-13T11:25:57.085837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:57.085837+0000","objects_scrubbed":2130,"log_size":9797,"log_dups_size":0,"ondisk_log_size":9797,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1023,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11501568,"num_objects":176,"num_object_clones":0,"num_object_copies":352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":72,"num_objects_unfound":0,"num_objects_dirty":176,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3374,"num_write_kb":85856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2134,"num_bytes_recovered":139853824,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0],"acting":[5,0],"avail_no_missing":["5","7","14"],"object_location_counts":[{"shards":"0,5","objects":140},{"shards":"5,7","objects":36}],"blocked_by":[],"up_primary":10,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"230'9848","reported_seq":13238,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-09-13T11:26:57.011788+0000","last_change":"2024-09-13T11:26:51.969220+0000","last_active":"2024-09-13T11:26:57.011788+0000","last_peered":"2024-09-13T11:26:57.011788+0000","last_clean":"2024-09-13T11:26:57.011788+0000","last_became_active":"2024-09-13T11:26:49.986829+0000","last_became_peered":"2024-09-13T11:26:49.986829+0000","last_unstale":"2024-09-13T11:26:57.011788+0000","last_undegraded":"2024-09-13T11:26:57.011788+0000","last_fullsized":"2024-09-13T11:26:57.011788+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":5,"last_scrub":"213'9393","last_scrub_stamp":"2024-09-13T11:26:36.941193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:26:36.941193+0000","objects_scrubbed":0,"log_size":9848,"log_dups_size":0,"ondisk_log_size":9848,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":119,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8749056,"num_objects":133,"num_object_clones":0,"num_object_copies":266,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":133,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3264,"num_write_kb":77024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7],"acting":[11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.12","version":"229'9797","reported_seq":20951,"reported_epoch":229,"state":"active+remapped+backfill_toofull","last_fresh":"2024-09-13T11:26:56.011601+0000","last_change":"2024-09-13T11:26:50.021305+0000","last_active":"2024-09-13T11:26:56.011601+0000","last_peered":"2024-09-13T11:26:56.011601+0000","last_clean":"2024-09-13T11:26:03.227632+0000","last_became_active":"2024-09-13T11:26:49.990894+0000","last_became_peered":"2024-09-13T11:26:49.990894+0000","last_unstale":"2024-09-13T11:26:56.011601+0000","last_undegraded":"2024-09-13T11:26:56.011601+0000","last_fullsized":"2024-09-13T11:26:56.011601+0000","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":5,"last_scrub":"186'8089","last_scrub_stamp":"2024-09-13T11:25:57.085837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:57.085837+0000","objects_scrubbed":2130,"log_size":9797,"log_dups_size":0,"ondisk_log_size":9797,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1023,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11501568,"num_objects":175,"num_object_clones":0,"num_object_copies":350,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6,"num_objects_unfound":0,"num_objects_dirty":175,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3373,"num_write_kb":85856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1943,"num_bytes_recovered":127336448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,7],"acting":[5,7],"avail_no_missing":["5","7"],"object_location_counts":[{"shards":"5,7","objects":175}],"blocked_by":[],"up_primary":14,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.11","version":"229'9789","reported_seq":13227,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-09-13T11:26:56.026639+0000","last_change":"2024-09-13T11:26:51.961292+0000","last_active":"2024-09-13T11:26:56.026639+0000","last_peered":"2024-09-13T11:26:56.026639+0000","last_clean":"2024-09-13T11:26:56.026639+0000","last_became_active":"2024-09-13T11:26:50.004919+0000","last_became_peered":"2024-09-13T11:26:50.004919+0000","last_unstale":"2024-09-13T11:26:56.026639+0000","last_undegraded":"2024-09-13T11:26:56.026639+0000","last_fullsized":"2024-09-13T11:26:56.026639+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":5,"last_scrub":"165'6821","last_scrub_stamp":"2024-09-13T11:25:24.287082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:24.287082+0000","objects_scrubbed":0,"log_size":9789,"log_dups_size":0,"ondisk_log_size":9789,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"user requested scrub scheduled @ 2024-09-13T11:26:56.608552+0000","scrub_duration":947,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11862016,"num_objects":181,"num_object_clones":0,"num_object_copies":362,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":181,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3255,"num_write_kb":77792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11],"acting":[4,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"229'9893","reported_seq":14579,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-09-13T11:26:55.971084+0000","last_change":"2024-09-13T11:26:51.973154+0000","last_active":"2024-09-13T11:26:55.971084+0000","last_peered":"2024-09-13T11:26:55.971084+0000","last_clean":"2024-09-13T11:26:55.971084+0000","last_became_active":"2024-09-13T11:26:49.984837+0000","last_became_peered":"2024-09-13T11:26:49.984837+0000","last_unstale":"2024-09-13T11:26:55.971084+0000","last_undegraded":"2024-09-13T11:26:55.971084+0000","last_fullsized":"2024-09-13T11:26:55.971084+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":5,"last_scrub":"204'8963","last_scrub_stamp":"2024-09-13T11:26:22.296795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:26:22.296795+0000","objects_scrubbed":0,"log_size":9893,"log_dups_size":0,"ondisk_log_size":9893,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"user requested scrub scheduled @ 2024-09-13T11:26:56.939649+0000","scrub_duration":195,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9535488,"num_objects":145,"num_object_clones":0,"num_object_copies":290,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":145,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3245,"num_write_kb":76000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"230'10152","reported_seq":10693,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-09-13T11:26:56.976062+0000","last_change":"2024-09-13T11:26:54.062342+0000","last_active":"2024-09-13T11:26:56.976062+0000","last_peered":"2024-09-13T11:26:56.976062+0000","last_clean":"2024-09-13T11:26:56.976062+0000","last_became_active":"2024-09-13T11:26:49.990894+0000","last_became_peered":"2024-09-13T11:26:49.990894+0000","last_unstale":"2024-09-13T11:26:56.976062+0000","last_undegraded":"2024-09-13T11:26:56.976062+0000","last_fullsized":"2024-09-13T11:26:56.976062+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"211'9653","last_scrub_stamp":"2024-09-13T11:26:34.940878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:26:34.940878+0000","objects_scrubbed":0,"log_size":10152,"log_dups_size":0,"ondisk_log_size":10152,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"user requested scrub scheduled @ 2024-09-13T11:26:58.906834+0000","scrub_duration":119,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9404416,"num_objects":143,"num_object_clones":0,"num_object_copies":286,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":143,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3409,"num_write_kb":85280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,12],"acting":[11,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.e","version":"229'10138","reported_seq":15572,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-09-13T11:26:55.853731+0000","last_change":"2024-09-13T11:26:51.968465+0000","last_active":"2024-09-13T11:26:55.853731+0000","last_peered":"2024-09-13T11:26:55.853731+0000","last_clean":"2024-09-13T11:26:55.853731+0000","last_became_active":"2024-09-13T11:26:49.986357+0000","last_became_peered":"2024-09-13T11:26:49.986357+0000","last_unstale":"2024-09-13T11:26:55.853731+0000","last_undegraded":"2024-09-13T11:26:55.853731+0000","last_fullsized":"2024-09-13T11:26:55.853731+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"180'8032","last_scrub_stamp":"2024-09-13T11:25:46.438107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:46.438107+0000","objects_scrubbed":0,"log_size":10138,"log_dups_size":0,"ondisk_log_size":10138,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"user requested scrub scheduled @ 2024-09-13T11:26:56.780178+0000","scrub_duration":327,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9895936,"num_objects":151,"num_object_clones":0,"num_object_copies":302,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":151,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3328,"num_write_kb":82944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.d","version":"229'7637","reported_seq":8138,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-09-13T11:26:56.238783+0000","last_change":"2024-09-13T11:25:57.320482+0000","last_active":"2024-09-13T11:26:56.238783+0000","last_peered":"2024-09-13T11:26:56.238783+0000","last_clean":"2024-09-13T11:26:56.238783+0000","last_became_active":"2024-09-13T11:24:24.087224+0000","last_became_peered":"2024-09-13T11:24:24.087224+0000","last_unstale":"2024-09-13T11:26:56.238783+0000","last_undegraded":"2024-09-13T11:26:56.238783+0000","last_fullsized":"2024-09-13T11:26:56.238783+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"186'6735","last_scrub_stamp":"2024-09-13T11:25:57.320378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:57.320378+0000","objects_scrubbed":1064,"log_size":7637,"log_dups_size":0,"ondisk_log_size":7637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:27:06.312652+0000","scrub_duration":211,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10551296,"num_objects":161,"num_object_clones":0,"num_object_copies":322,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":161,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2530,"num_write_kb":30912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.c","version":"230'7653","reported_seq":8160,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-09-13T11:26:57.039492+0000","last_change":"2024-09-13T11:25:36.279464+0000","last_active":"2024-09-13T11:26:57.039492+0000","last_peered":"2024-09-13T11:26:57.039492+0000","last_clean":"2024-09-13T11:26:57.039492+0000","last_became_active":"2024-09-13T11:24:24.090847+0000","last_became_peered":"2024-09-13T11:24:24.090847+0000","last_unstale":"2024-09-13T11:26:57.039492+0000","last_undegraded":"2024-09-13T11:26:57.039492+0000","last_fullsized":"2024-09-13T11:26:57.039492+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"174'6334","last_scrub_stamp":"2024-09-13T11:25:36.279390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:36.279390+0000","objects_scrubbed":1456,"log_size":7653,"log_dups_size":0,"ondisk_log_size":7653,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":235,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8912896,"num_objects":136,"num_object_clones":0,"num_object_copies":272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":136,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2504,"num_write_kb":29440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,13],"acting":[11,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.b","version":"230'7642","reported_seq":8158,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-09-13T11:26:56.868597+0000","last_change":"2024-09-13T11:25:58.292249+0000","last_active":"2024-09-13T11:26:56.868597+0000","last_peered":"2024-09-13T11:26:56.868597+0000","last_clean":"2024-09-13T11:26:56.868597+0000","last_became_active":"2024-09-13T11:24:24.085684+0000","last_became_peered":"2024-09-13T11:24:24.085684+0000","last_unstale":"2024-09-13T11:26:56.868597+0000","last_undegraded":"2024-09-13T11:26:56.868597+0000","last_fullsized":"2024-09-13T11:26:56.868597+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"186'6759","last_scrub_stamp":"2024-09-13T11:25:58.292153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:58.292153+0000","objects_scrubbed":1030,"log_size":7642,"log_dups_size":0,"ondisk_log_size":7642,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:27:10.748450+0000","scrub_duration":119,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9568256,"num_objects":146,"num_object_clones":0,"num_object_copies":292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":146,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2530,"num_write_kb":30656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,1],"acting":[13,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"3.a","version":"232'7718","reported_seq":8225,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-13T11:27:00.057515+0000","last_change":"2024-09-13T11:25:59.695285+0000","last_active":"2024-09-13T11:27:00.057515+0000","last_peered":"2024-09-13T11:27:00.057515+0000","last_clean":"2024-09-13T11:27:00.057515+0000","last_became_active":"2024-09-13T11:24:24.089253+0000","last_became_peered":"2024-09-13T11:24:24.089253+0000","last_unstale":"2024-09-13T11:27:00.057515+0000","last_undegraded":"2024-09-13T11:27:00.057515+0000","last_fullsized":"2024-09-13T11:27:00.057515+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"187'6833","last_scrub_stamp":"2024-09-13T11:25:59.695136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:59.695136+0000","objects_scrubbed":973,"log_size":7718,"log_dups_size":0,"ondisk_log_size":7718,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:27:12.200514+0000","scrub_duration":131,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5701632,"num_objects":87,"num_object_clones":0,"num_object_copies":174,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":87,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2564,"num_write_kb":27968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.9","version":"229'7317","reported_seq":7796,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-09-13T11:26:56.107568+0000","last_change":"2024-09-13T11:24:24.928964+0000","last_active":"2024-09-13T11:26:56.107568+0000","last_peered":"2024-09-13T11:26:56.107568+0000","last_clean":"2024-09-13T11:26:56.107568+0000","last_became_active":"2024-09-13T11:24:24.082504+0000","last_became_peered":"2024-09-13T11:24:24.082504+0000","last_unstale":"2024-09-13T11:26:56.107568+0000","last_undegraded":"2024-09-13T11:26:56.107568+0000","last_fullsized":"2024-09-13T11:26:56.107568+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'4799","last_scrub_stamp":"2024-09-13T11:24:24.928724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:24:24.928724+0000","objects_scrubbed":1705,"log_size":7317,"log_dups_size":0,"ondisk_log_size":7317,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":255,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10223616,"num_objects":156,"num_object_clones":0,"num_object_copies":312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":156,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2521,"num_write_kb":31104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,2],"acting":[12,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[]},{"pgid":"3.8","version":"229'7255","reported_seq":7752,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-09-13T11:26:56.062854+0000","last_change":"2024-09-13T11:26:00.418168+0000","last_active":"2024-09-13T11:26:56.062854+0000","last_peered":"2024-09-13T11:26:56.062854+0000","last_clean":"2024-09-13T11:26:56.062854+0000","last_became_active":"2024-09-13T11:24:24.096220+0000","last_became_peered":"2024-09-13T11:24:24.096220+0000","last_unstale":"2024-09-13T11:26:56.062854+0000","last_undegraded":"2024-09-13T11:26:56.062854+0000","last_fullsized":"2024-09-13T11:26:56.062854+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"188'6425","last_scrub_stamp":"2024-09-13T11:26:00.417942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:26:00.417942+0000","objects_scrubbed":989,"log_size":7255,"log_dups_size":0,"ondisk_log_size":7255,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:27:09.204623+0000","scrub_duration":199,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10420224,"num_objects":159,"num_object_clones":0,"num_object_copies":318,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":159,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2507,"num_write_kb":31232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"26'112","reported_seq":581,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-09-13T11:26:55.386538+0000","last_change":"2024-09-13T11:26:46.996802+0000","last_active":"2024-09-13T11:26:55.386538+0000","last_peered":"2024-09-13T11:26:55.386538+0000","last_clean":"2024-09-13T11:26:55.386538+0000","last_became_active":"2024-09-13T11:22:12.129721+0000","last_became_peered":"2024-09-13T11:22:12.129721+0000","last_unstale":"2024-09-13T11:26:55.386538+0000","last_undegraded":"2024-09-13T11:26:55.386538+0000","last_fullsized":"2024-09-13T11:26:55.386538+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"26'112","last_scrub_stamp":"2024-09-13T11:26:46.996697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:11.098388+0000","last_clean_scrub_stamp":"2024-09-13T11:26:46.996697+0000","objects_scrubbed":2,"log_size":112,"log_dups_size":0,"ondisk_log_size":112,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:27:50.591478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.3","version":"26'1","reported_seq":95,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-13T11:26:59.415604+0000","last_change":"2024-09-13T11:26:05.046376+0000","last_active":"2024-09-13T11:26:59.415604+0000","last_peered":"2024-09-13T11:26:59.415604+0000","last_clean":"2024-09-13T11:26:59.415604+0000","last_became_active":"2024-09-13T11:26:05.041032+0000","last_became_peered":"2024-09-13T11:26:05.041032+0000","last_unstale":"2024-09-13T11:26:59.415604+0000","last_undegraded":"2024-09-13T11:26:59.415604+0000","last_fullsized":"2024-09-13T11:26:59.415604+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"26'1","last_scrub_stamp":"2024-09-13T11:25:23.034247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:25:23.034247+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,15],"acting":[14,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"3.2","version":"232'6585","reported_seq":15025,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-13T11:26:59.857514+0000","last_change":"2024-09-13T11:26:43.033025+0000","last_active":"2024-09-13T11:26:59.857514+0000","last_peered":"2024-09-13T11:26:59.857514+0000","last_clean":"2024-09-13T11:26:59.857514+0000","last_became_active":"2024-09-13T11:26:42.985163+0000","last_became_peered":"2024-09-13T11:26:42.985163+0000","last_unstale":"2024-09-13T11:26:59.857514+0000","last_undegraded":"2024-09-13T11:26:59.857514+0000","last_fullsized":"2024-09-13T11:26:59.857514+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"164'4994","last_scrub_stamp":"2024-09-13T11:25:23.765047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:23.765047+0000","objects_scrubbed":1689,"log_size":6585,"log_dups_size":0,"ondisk_log_size":6585,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":283,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6225920,"num_objects":95,"num_object_clones":0,"num_object_copies":190,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":95,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3528,"num_write_kb":89024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3963,"num_bytes_recovered":162332672,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,12],"acting":[14,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":445,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-13T11:26:59.431966+0000","last_change":"2024-09-13T11:25:56.557174+0000","last_active":"2024-09-13T11:26:59.431966+0000","last_peered":"2024-09-13T11:26:59.431966+0000","last_clean":"2024-09-13T11:26:59.431966+0000","last_became_active":"2024-09-13T11:22:14.211435+0000","last_became_peered":"2024-09-13T11:22:14.211435+0000","last_unstale":"2024-09-13T11:26:59.431966+0000","last_undegraded":"2024-09-13T11:26:59.431966+0000","last_fullsized":"2024-09-13T11:26:59.431966+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:25:56.557113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:25:56.557113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:27:18.356524+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0010010360000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"230'6422","reported_seq":6936,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-09-13T11:26:56.936918+0000","last_change":"2024-09-13T11:26:33.441490+0000","last_active":"2024-09-13T11:26:56.936918+0000","last_peered":"2024-09-13T11:26:56.936918+0000","last_clean":"2024-09-13T11:26:56.936918+0000","last_became_active":"2024-09-13T11:23:31.926528+0000","last_became_peered":"2024-09-13T11:23:31.926528+0000","last_unstale":"2024-09-13T11:26:56.936918+0000","last_undegraded":"2024-09-13T11:26:56.936918+0000","last_fullsized":"2024-09-13T11:26:56.936918+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"211'6118","last_scrub_stamp":"2024-09-13T11:26:33.441303+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:26:33.441303+0000","objects_scrubbed":453,"log_size":6422,"log_dups_size":0,"ondisk_log_size":6422,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:27:53.732036+0000","scrub_duration":71,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9699328,"num_objects":148,"num_object_clones":0,"num_object_copies":296,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":148,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3352,"num_write_kb":85056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,11],"acting":[13,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":91,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-13T11:26:59.415474+0000","last_change":"2024-09-13T11:26:05.047291+0000","last_active":"2024-09-13T11:26:59.415474+0000","last_peered":"2024-09-13T11:26:59.415474+0000","last_clean":"2024-09-13T11:26:59.415474+0000","last_became_active":"2024-09-13T11:26:05.046717+0000","last_became_peered":"2024-09-13T11:26:05.046717+0000","last_unstale":"2024-09-13T11:26:59.415474+0000","last_undegraded":"2024-09-13T11:26:59.415474+0000","last_fullsized":"2024-09-13T11:26:59.415474+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:25:34.822831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:25:34.822831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00081225399999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,1],"acting":[14,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"3.0","version":"229'6417","reported_seq":6914,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-09-13T11:26:55.960249+0000","last_change":"2024-09-13T11:25:09.842875+0000","last_active":"2024-09-13T11:26:55.960249+0000","last_peered":"2024-09-13T11:26:55.960249+0000","last_clean":"2024-09-13T11:26:55.960249+0000","last_became_active":"2024-09-13T11:23:31.921388+0000","last_became_peered":"2024-09-13T11:23:31.921388+0000","last_unstale":"2024-09-13T11:26:55.960249+0000","last_undegraded":"2024-09-13T11:26:55.960249+0000","last_fullsized":"2024-09-13T11:26:55.960249+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"151'4644","last_scrub_stamp":"2024-09-13T11:25:09.842728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:09.842728+0000","objects_scrubbed":1944,"log_size":6417,"log_dups_size":0,"ondisk_log_size":6417,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":395,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11206656,"num_objects":171,"num_object_clones":0,"num_object_copies":342,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":171,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3311,"num_write_kb":84032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,15],"acting":[12,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[]},{"pgid":"2.2","version":"28'2","reported_seq":55,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-13T11:26:59.429215+0000","last_change":"2024-09-13T11:26:36.271659+0000","last_active":"2024-09-13T11:26:59.429215+0000","last_peered":"2024-09-13T11:26:59.429215+0000","last_clean":"2024-09-13T11:26:59.429215+0000","last_became_active":"2024-09-13T11:26:36.267081+0000","last_became_peered":"2024-09-13T11:26:36.267081+0000","last_unstale":"2024-09-13T11:26:59.429215+0000","last_undegraded":"2024-09-13T11:26:59.429215+0000","last_fullsized":"2024-09-13T11:26:59.429215+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2","last_scrub_stamp":"2024-09-13T11:25:13.869413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:25:13.869413+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0025826730000000002,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9],"acting":[3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.3","version":"229'6378","reported_seq":6877,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-09-13T11:26:56.035964+0000","last_change":"2024-09-13T11:25:08.394545+0000","last_active":"2024-09-13T11:26:56.035964+0000","last_peered":"2024-09-13T11:26:56.035964+0000","last_clean":"2024-09-13T11:26:56.035964+0000","last_became_active":"2024-09-13T11:23:31.930448+0000","last_became_peered":"2024-09-13T11:23:31.930448+0000","last_unstale":"2024-09-13T11:26:56.035964+0000","last_undegraded":"2024-09-13T11:26:56.035964+0000","last_fullsized":"2024-09-13T11:26:56.035964+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"151'4587","last_scrub_stamp":"2024-09-13T11:25:08.394441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:08.394441+0000","objects_scrubbed":1942,"log_size":6378,"log_dups_size":0,"ondisk_log_size":6378,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:26:57.146916+0000","scrub_duration":219,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9699328,"num_objects":148,"num_object_clones":0,"num_object_copies":296,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":148,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3256,"num_write_kb":80896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,7],"acting":[10,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.5","version":"230'7211","reported_seq":7717,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-09-13T11:26:57.026187+0000","last_change":"2024-09-13T11:26:24.022070+0000","last_active":"2024-09-13T11:26:57.026187+0000","last_peered":"2024-09-13T11:26:57.026187+0000","last_clean":"2024-09-13T11:26:57.026187+0000","last_became_active":"2024-09-13T11:26:23.988370+0000","last_became_peered":"2024-09-13T11:26:23.988370+0000","last_unstale":"2024-09-13T11:26:57.026187+0000","last_undegraded":"2024-09-13T11:26:57.026187+0000","last_fullsized":"2024-09-13T11:26:57.026187+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"182'6171","last_scrub_stamp":"2024-09-13T11:25:49.468616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:49.468616+0000","objects_scrubbed":1202,"log_size":7211,"log_dups_size":0,"ondisk_log_size":7211,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:27:13.323295+0000","scrub_duration":175,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10616832,"num_objects":162,"num_object_clones":0,"num_object_copies":324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":162,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2474,"num_write_kb":32384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,15],"acting":[13,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":433,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-13T11:26:59.441986+0000","last_change":"2024-09-13T11:25:45.851854+0000","last_active":"2024-09-13T11:26:59.441986+0000","last_peered":"2024-09-13T11:26:59.441986+0000","last_clean":"2024-09-13T11:26:59.441986+0000","last_became_active":"2024-09-13T11:22:47.719636+0000","last_became_peered":"2024-09-13T11:22:47.719636+0000","last_unstale":"2024-09-13T11:26:59.441986+0000","last_undegraded":"2024-09-13T11:26:59.441986+0000","last_fullsized":"2024-09-13T11:26:59.441986+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:25:45.851784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:25:45.851784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00087476000000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,9],"acting":[1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"232'7340","reported_seq":7826,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-13T11:27:00.383064+0000","last_change":"2024-09-13T11:25:48.001902+0000","last_active":"2024-09-13T11:27:00.383064+0000","last_peered":"2024-09-13T11:27:00.383064+0000","last_clean":"2024-09-13T11:27:00.383064+0000","last_became_active":"2024-09-13T11:24:24.083077+0000","last_became_peered":"2024-09-13T11:24:24.083077+0000","last_unstale":"2024-09-13T11:27:00.383064+0000","last_undegraded":"2024-09-13T11:27:00.383064+0000","last_fullsized":"2024-09-13T11:27:00.383064+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"181'6222","last_scrub_stamp":"2024-09-13T11:25:48.001778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:48.001778+0000","objects_scrubbed":1224,"log_size":7340,"log_dups_size":0,"ondisk_log_size":7340,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":171,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6946816,"num_objects":106,"num_object_clones":0,"num_object_copies":212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":106,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2606,"num_write_kb":32704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,12],"acting":[1,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":439,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-09-13T11:26:55.382965+0000","last_change":"2024-09-13T11:25:33.957283+0000","last_active":"2024-09-13T11:26:55.382965+0000","last_peered":"2024-09-13T11:26:55.382965+0000","last_clean":"2024-09-13T11:26:55.382965+0000","last_became_active":"2024-09-13T11:22:14.581458+0000","last_became_peered":"2024-09-13T11:22:14.581458+0000","last_unstale":"2024-09-13T11:26:55.382965+0000","last_undegraded":"2024-09-13T11:26:55.382965+0000","last_fullsized":"2024-09-13T11:26:55.382965+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:25:33.957179+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:25:33.957179+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:26:57.298596+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.001026047,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5],"acting":[8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.7","version":"230'7253","reported_seq":7761,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-09-13T11:26:57.077289+0000","last_change":"2024-09-13T11:25:59.279849+0000","last_active":"2024-09-13T11:26:57.077289+0000","last_peered":"2024-09-13T11:26:57.077289+0000","last_clean":"2024-09-13T11:26:57.077289+0000","last_became_active":"2024-09-13T11:24:24.092975+0000","last_became_peered":"2024-09-13T11:24:24.092975+0000","last_unstale":"2024-09-13T11:26:57.077289+0000","last_undegraded":"2024-09-13T11:26:57.077289+0000","last_fullsized":"2024-09-13T11:26:57.077289+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"187'6375","last_scrub_stamp":"2024-09-13T11:25:59.279677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:59.279677+0000","objects_scrubbed":1033,"log_size":7253,"log_dups_size":0,"ondisk_log_size":7253,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:27:06.179340+0000","scrub_duration":139,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10158080,"num_objects":155,"num_object_clones":0,"num_object_copies":310,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":155,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2435,"num_write_kb":26368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,8],"acting":[13,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":448,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-09-13T11:26:56.398413+0000","last_change":"2024-09-13T11:26:33.048163+0000","last_active":"2024-09-13T11:26:56.398413+0000","last_peered":"2024-09-13T11:26:56.398413+0000","last_clean":"2024-09-13T11:26:56.398413+0000","last_became_active":"2024-09-13T11:22:14.211923+0000","last_became_peered":"2024-09-13T11:22:14.211923+0000","last_unstale":"2024-09-13T11:26:56.398413+0000","last_undegraded":"2024-09-13T11:26:56.398413+0000","last_fullsized":"2024-09-13T11:26:56.398413+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:26:33.048033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:26:33.048033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:27:40.864163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011689529999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0],"acting":[9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.6","version":"230'7426","reported_seq":15901,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-09-13T11:26:56.885028+0000","last_change":"2024-09-13T11:26:32.392786+0000","last_active":"2024-09-13T11:26:56.885028+0000","last_peered":"2024-09-13T11:26:56.885028+0000","last_clean":"2024-09-13T11:26:56.885028+0000","last_became_active":"2024-09-13T11:26:05.246522+0000","last_became_peered":"2024-09-13T11:26:05.246522+0000","last_unstale":"2024-09-13T11:26:56.885028+0000","last_undegraded":"2024-09-13T11:26:56.885028+0000","last_fullsized":"2024-09-13T11:26:56.885028+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"211'7110","last_scrub_stamp":"2024-09-13T11:26:32.392666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:26:32.392666+0000","objects_scrubbed":482,"log_size":7426,"log_dups_size":0,"ondisk_log_size":7426,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:27:54.979779+0000","scrub_duration":59,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10878976,"num_objects":166,"num_object_clones":0,"num_object_copies":332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":166,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2211,"num_write_kb":9408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2230,"num_bytes_recovered":61800448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,14],"acting":[13,14],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":58,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-09-13T11:26:55.397737+0000","last_change":"2024-09-13T11:26:23.991137+0000","last_active":"2024-09-13T11:26:55.397737+0000","last_peered":"2024-09-13T11:26:55.397737+0000","last_clean":"2024-09-13T11:26:55.397737+0000","last_became_active":"2024-09-13T11:26:23.990749+0000","last_became_peered":"2024-09-13T11:26:23.990749+0000","last_unstale":"2024-09-13T11:26:55.397737+0000","last_undegraded":"2024-09-13T11:26:55.397737+0000","last_fullsized":"2024-09-13T11:26:55.397737+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:25:46.222806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:25:46.222806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:27:14.224983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,15],"acting":[10,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.14","version":"232'9919","reported_seq":22474,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-13T11:27:00.123505+0000","last_change":"2024-09-13T11:26:57.171391+0000","last_active":"2024-09-13T11:27:00.123505+0000","last_peered":"2024-09-13T11:27:00.123505+0000","last_clean":"2024-09-13T11:27:00.123505+0000","last_became_active":"2024-09-13T11:26:49.995599+0000","last_became_peered":"2024-09-13T11:26:49.995599+0000","last_unstale":"2024-09-13T11:27:00.123505+0000","last_undegraded":"2024-09-13T11:27:00.123505+0000","last_fullsized":"2024-09-13T11:27:00.123505+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":5,"last_scrub":"230'9887","last_scrub_stamp":"2024-09-13T11:26:57.171290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:26:57.171290+0000","objects_scrubbed":131,"log_size":9919,"log_dups_size":0,"ondisk_log_size":9919,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:28:13.022860+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6488064,"num_objects":99,"num_object_clones":0,"num_object_copies":198,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":99,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":32,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,14],"acting":[3,14],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"232'9611","reported_seq":22741,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-13T11:27:00.004636+0000","last_change":"2024-09-13T11:26:57.258064+0000","last_active":"2024-09-13T11:27:00.004636+0000","last_peered":"2024-09-13T11:27:00.004636+0000","last_clean":"2024-09-13T11:27:00.004636+0000","last_became_active":"2024-09-13T11:26:50.003619+0000","last_became_peered":"2024-09-13T11:26:50.003619+0000","last_unstale":"2024-09-13T11:27:00.004636+0000","last_undegraded":"2024-09-13T11:27:00.004636+0000","last_fullsized":"2024-09-13T11:27:00.004636+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":5,"last_scrub":"230'9569","last_scrub_stamp":"2024-09-13T11:26:57.257731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:26:57.257731+0000","objects_scrubbed":140,"log_size":9611,"log_dups_size":0,"ondisk_log_size":9611,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:28:11.995229+0000","scrub_duration":215,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6422528,"num_objects":98,"num_object_clones":0,"num_object_copies":196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":42,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,0],"acting":[14,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"3.16","version":"230'10010","reported_seq":15015,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-09-13T11:26:57.060279+0000","last_change":"2024-09-13T11:26:52.979237+0000","last_active":"2024-09-13T11:26:57.060279+0000","last_peered":"2024-09-13T11:26:57.060279+0000","last_clean":"2024-09-13T11:26:57.060279+0000","last_became_active":"2024-09-13T11:26:50.003909+0000","last_became_peered":"2024-09-13T11:26:50.003909+0000","last_unstale":"2024-09-13T11:26:57.060279+0000","last_undegraded":"2024-09-13T11:26:57.060279+0000","last_fullsized":"2024-09-13T11:26:57.060279+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":5,"last_scrub":"204'9074","last_scrub_stamp":"2024-09-13T11:26:21.348185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:26:21.348185+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"user requested scrub scheduled @ 2024-09-13T11:26:57.931856+0000","scrub_duration":479,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9764864,"num_objects":149,"num_object_clones":0,"num_object_copies":298,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":149,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3303,"num_write_kb":77600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,15],"acting":[11,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.17","version":"229'9548","reported_seq":13919,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-09-13T11:26:56.096820+0000","last_change":"2024-09-13T11:26:51.966436+0000","last_active":"2024-09-13T11:26:56.096820+0000","last_peered":"2024-09-13T11:26:56.096820+0000","last_clean":"2024-09-13T11:26:56.096820+0000","last_became_active":"2024-09-13T11:26:49.983871+0000","last_became_peered":"2024-09-13T11:26:49.983871+0000","last_unstale":"2024-09-13T11:26:56.096820+0000","last_undegraded":"2024-09-13T11:26:56.096820+0000","last_fullsized":"2024-09-13T11:26:56.096820+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":5,"last_scrub":"206'8719","last_scrub_stamp":"2024-09-13T11:26:25.299674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:26:25.299674+0000","objects_scrubbed":0,"log_size":9548,"log_dups_size":0,"ondisk_log_size":9548,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"user requested scrub scheduled @ 2024-09-13T11:26:56.172643+0000","scrub_duration":919,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11272192,"num_objects":172,"num_object_clones":0,"num_object_copies":344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":172,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3122,"num_write_kb":73344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3],"acting":[10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.18","version":"229'9800","reported_seq":14702,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-09-13T11:26:56.053818+0000","last_change":"2024-09-13T11:25:37.928304+0000","last_active":"2024-09-13T11:26:56.053818+0000","last_peered":"2024-09-13T11:26:56.053818+0000","last_clean":"2024-09-13T11:26:56.053818+0000","last_became_active":"2024-09-13T11:23:48.926014+0000","last_became_peered":"2024-09-13T11:23:48.926014+0000","last_unstale":"2024-09-13T11:26:56.053818+0000","last_undegraded":"2024-09-13T11:26:56.053818+0000","last_fullsized":"2024-09-13T11:26:56.053818+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":5,"last_scrub":"174'7306","last_scrub_stamp":"2024-09-13T11:25:37.928052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:37.928052+0000","objects_scrubbed":2815,"log_size":9800,"log_dups_size":0,"ondisk_log_size":9800,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":631,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20119552,"num_objects":307,"num_object_clones":0,"num_object_copies":614,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":307,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5433,"num_write_kb":86528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0],"acting":[10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.19","version":"230'9667","reported_seq":14289,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-09-13T11:26:57.047464+0000","last_change":"2024-09-13T11:26:34.674723+0000","last_active":"2024-09-13T11:26:57.047464+0000","last_peered":"2024-09-13T11:26:57.047464+0000","last_clean":"2024-09-13T11:26:57.047464+0000","last_became_active":"2024-09-13T11:23:41.924941+0000","last_became_peered":"2024-09-13T11:23:41.924941+0000","last_unstale":"2024-09-13T11:26:57.047464+0000","last_undegraded":"2024-09-13T11:26:57.047464+0000","last_fullsized":"2024-09-13T11:26:57.047464+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":5,"last_scrub":"211'9143","last_scrub_stamp":"2024-09-13T11:26:34.674563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:26:34.674563+0000","objects_scrubbed":778,"log_size":9667,"log_dups_size":0,"ondisk_log_size":9667,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:27:45.242098+0000","scrub_duration":647,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16515072,"num_objects":252,"num_object_clones":0,"num_object_copies":504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":252,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6265,"num_write_kb":145216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1],"acting":[9,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1a","version":"229'10044","reported_seq":15495,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-09-13T11:26:55.965019+0000","last_change":"2024-09-13T11:25:46.321420+0000","last_active":"2024-09-13T11:26:55.965019+0000","last_peered":"2024-09-13T11:26:55.965019+0000","last_clean":"2024-09-13T11:26:55.965019+0000","last_became_active":"2024-09-13T11:24:03.935185+0000","last_became_peered":"2024-09-13T11:24:03.935185+0000","last_unstale":"2024-09-13T11:26:55.965019+0000","last_undegraded":"2024-09-13T11:26:55.965019+0000","last_fullsized":"2024-09-13T11:26:55.965019+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":5,"last_scrub":"179'7826","last_scrub_stamp":"2024-09-13T11:25:46.321284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:46.321284+0000","objects_scrubbed":2526,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":935,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19398656,"num_objects":296,"num_object_clones":0,"num_object_copies":592,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":296,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5622,"num_write_kb":104448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9],"acting":[4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"229'10218","reported_seq":16006,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-09-13T11:26:56.067876+0000","last_change":"2024-09-13T11:25:49.890080+0000","last_active":"2024-09-13T11:26:56.067876+0000","last_peered":"2024-09-13T11:26:56.067876+0000","last_clean":"2024-09-13T11:26:56.067876+0000","last_became_active":"2024-09-13T11:24:04.950821+0000","last_became_peered":"2024-09-13T11:24:04.950821+0000","last_unstale":"2024-09-13T11:26:56.067876+0000","last_undegraded":"2024-09-13T11:26:56.067876+0000","last_fullsized":"2024-09-13T11:26:56.067876+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":5,"last_scrub":"182'8139","last_scrub_stamp":"2024-09-13T11:25:49.889914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:49.889914+0000","objects_scrubbed":2406,"log_size":10218,"log_dups_size":0,"ondisk_log_size":10218,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:27:16.988785+0000","scrub_duration":519,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20840448,"num_objects":318,"num_object_clones":0,"num_object_copies":636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":318,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5574,"num_write_kb":94656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,15],"acting":[12,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[]},{"pgid":"3.1c","version":"230'10253","reported_seq":14222,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-09-13T11:26:57.060874+0000","last_change":"2024-09-13T11:25:46.456795+0000","last_active":"2024-09-13T11:26:57.060874+0000","last_peered":"2024-09-13T11:26:57.060874+0000","last_clean":"2024-09-13T11:26:57.060874+0000","last_became_active":"2024-09-13T11:23:33.989492+0000","last_became_peered":"2024-09-13T11:23:33.989492+0000","last_unstale":"2024-09-13T11:26:57.060874+0000","last_undegraded":"2024-09-13T11:26:57.060874+0000","last_fullsized":"2024-09-13T11:26:57.060874+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":5,"last_scrub":"179'8037","last_scrub_stamp":"2024-09-13T11:25:46.456644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:46.456644+0000","objects_scrubbed":2528,"log_size":10253,"log_dups_size":0,"ondisk_log_size":10253,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":451,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20054016,"num_objects":306,"num_object_clones":0,"num_object_copies":612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":306,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5978,"num_write_kb":119680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7],"acting":[11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1d","version":"229'10071","reported_seq":15682,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-09-13T11:26:56.025415+0000","last_change":"2024-09-13T11:25:46.627072+0000","last_active":"2024-09-13T11:26:56.025415+0000","last_peered":"2024-09-13T11:26:56.025415+0000","last_clean":"2024-09-13T11:26:56.025415+0000","last_became_active":"2024-09-13T11:24:01.258225+0000","last_became_peered":"2024-09-13T11:24:01.258225+0000","last_unstale":"2024-09-13T11:26:56.025415+0000","last_undegraded":"2024-09-13T11:26:56.025415+0000","last_fullsized":"2024-09-13T11:26:56.025415+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":5,"last_scrub":"180'7891","last_scrub_stamp":"2024-09-13T11:25:46.626952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:46.626952+0000","objects_scrubbed":2517,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:27:01.444687+0000","scrub_duration":875,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21430296,"num_objects":328,"num_object_clones":0,"num_object_copies":656,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":328,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5505,"num_write_kb":94593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,15],"acting":[2,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1e","version":"229'10295","reported_seq":20943,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-09-13T11:26:56.046773+0000","last_change":"2024-09-13T11:24:08.338485+0000","last_active":"2024-09-13T11:26:56.046773+0000","last_peered":"2024-09-13T11:26:56.046773+0000","last_clean":"2024-09-13T11:26:56.046773+0000","last_became_active":"2024-09-13T11:24:01.261448+0000","last_became_peered":"2024-09-13T11:24:01.261448+0000","last_unstale":"2024-09-13T11:26:56.046773+0000","last_undegraded":"2024-09-13T11:26:56.046773+0000","last_fullsized":"2024-09-13T11:26:56.046773+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":5,"last_scrub":"110'4537","last_scrub_stamp":"2024-09-13T11:24:08.338354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:24:08.338354+0000","objects_scrubbed":2760,"log_size":10295,"log_dups_size":0,"ondisk_log_size":10295,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:26:57.646144+0000","scrub_duration":367,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20512768,"num_objects":313,"num_object_clones":0,"num_object_copies":626,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":313,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5767,"num_write_kb":106240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1f","version":"229'9976","reported_seq":23133,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-09-13T11:26:55.999751+0000","last_change":"2024-09-13T11:26:49.174841+0000","last_active":"2024-09-13T11:26:55.999751+0000","last_peered":"2024-09-13T11:26:55.999751+0000","last_clean":"2024-09-13T11:26:55.999751+0000","last_became_active":"2024-09-13T11:26:49.140281+0000","last_became_peered":"2024-09-13T11:26:49.140281+0000","last_unstale":"2024-09-13T11:26:55.999751+0000","last_undegraded":"2024-09-13T11:26:55.999751+0000","last_fullsized":"2024-09-13T11:26:55.999751+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":5,"last_scrub":"161'6865","last_scrub_stamp":"2024-09-13T11:25:20.773641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:20.773641+0000","objects_scrubbed":3425,"log_size":9976,"log_dups_size":0,"ondisk_log_size":9976,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":535,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20250624,"num_objects":309,"num_object_clones":0,"num_object_copies":618,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":309,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5556,"num_write_kb":102848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3836,"num_bytes_recovered":251396096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,14],"acting":[10,14],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.20","version":"229'6525","reported_seq":8831,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-09-13T11:26:56.091052+0000","last_change":"2024-09-13T11:25:35.773483+0000","last_active":"2024-09-13T11:26:56.091052+0000","last_peered":"2024-09-13T11:26:56.091052+0000","last_clean":"2024-09-13T11:26:56.091052+0000","last_became_active":"2024-09-13T11:23:33.954946+0000","last_became_peered":"2024-09-13T11:23:33.954946+0000","last_unstale":"2024-09-13T11:26:56.091052+0000","last_undegraded":"2024-09-13T11:26:56.091052+0000","last_fullsized":"2024-09-13T11:26:56.091052+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":6,"last_scrub":"174'5240","last_scrub_stamp":"2024-09-13T11:25:35.773409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:35.773409+0000","objects_scrubbed":1424,"log_size":6525,"log_dups_size":0,"ondisk_log_size":6525,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":215,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8978432,"num_objects":137,"num_object_clones":0,"num_object_copies":274,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":137,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3260,"num_write_kb":74752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,0],"acting":[12,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[]},{"pgid":"3.21","version":"229'6333","reported_seq":9349,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-09-13T11:26:55.987169+0000","last_change":"2024-09-13T11:25:49.226484+0000","last_active":"2024-09-13T11:26:55.987169+0000","last_peered":"2024-09-13T11:26:55.987169+0000","last_clean":"2024-09-13T11:26:55.987169+0000","last_became_active":"2024-09-13T11:24:01.250308+0000","last_became_peered":"2024-09-13T11:24:01.250308+0000","last_unstale":"2024-09-13T11:26:55.987169+0000","last_undegraded":"2024-09-13T11:26:55.987169+0000","last_fullsized":"2024-09-13T11:26:55.987169+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":6,"last_scrub":"182'5306","last_scrub_stamp":"2024-09-13T11:25:49.226272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:49.226272+0000","objects_scrubbed":1176,"log_size":6333,"log_dups_size":0,"ondisk_log_size":6333,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:27:08.537716+0000","scrub_duration":151,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9633792,"num_objects":147,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":147,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2666,"num_write_kb":44224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8],"acting":[0,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"229'6496","reported_seq":19478,"reported_epoch":229,"state":"active+recovering+remapped","last_fresh":"2024-09-13T11:26:56.142631+0000","last_change":"2024-09-13T11:26:49.521922+0000","last_active":"2024-09-13T11:26:56.142631+0000","last_peered":"2024-09-13T11:26:56.142631+0000","last_clean":"2024-09-13T11:26:03.171705+0000","last_became_active":"2024-09-13T11:26:42.027697+0000","last_became_peered":"2024-09-13T11:26:42.027697+0000","last_unstale":"2024-09-13T11:26:56.142631+0000","last_undegraded":"2024-09-13T11:26:56.142631+0000","last_fullsized":"2024-09-13T11:26:56.142631+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":6,"last_scrub":"186'5611","last_scrub_stamp":"2024-09-13T11:25:57.123569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:57.123569+0000","objects_scrubbed":1032,"log_size":6496,"log_dups_size":0,"ondisk_log_size":6496,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":147,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9633792,"num_objects":147,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":110,"num_objects_unfound":0,"num_objects_dirty":147,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3252,"num_write_kb":75712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4844,"num_bytes_recovered":190513152,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,14],"acting":[0,14],"avail_no_missing":["0","5","14"],"object_location_counts":[{"shards":"0,14","objects":37}],"blocked_by":[],"up_primary":12,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.23","version":"229'6580","reported_seq":9594,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-09-13T11:26:56.008832+0000","last_change":"2024-09-13T11:25:56.586649+0000","last_active":"2024-09-13T11:26:56.008832+0000","last_peered":"2024-09-13T11:26:56.008832+0000","last_clean":"2024-09-13T11:26:56.008832+0000","last_became_active":"2024-09-13T11:23:47.926677+0000","last_became_peered":"2024-09-13T11:23:47.926677+0000","last_unstale":"2024-09-13T11:26:56.008832+0000","last_undegraded":"2024-09-13T11:26:56.008832+0000","last_fullsized":"2024-09-13T11:26:56.008832+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":6,"last_scrub":"186'5649","last_scrub_stamp":"2024-09-13T11:25:56.586425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:56.586425+0000","objects_scrubbed":1098,"log_size":6580,"log_dups_size":0,"ondisk_log_size":6580,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:27:01.886969+0000","scrub_duration":135,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10878976,"num_objects":166,"num_object_clones":0,"num_object_copies":332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":166,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3303,"num_write_kb":78080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9],"acting":[5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.24","version":"230'7347","reported_seq":15914,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-09-13T11:26:57.016689+0000","last_change":"2024-09-13T11:26:30.786438+0000","last_active":"2024-09-13T11:26:57.016689+0000","last_peered":"2024-09-13T11:26:57.016689+0000","last_clean":"2024-09-13T11:26:57.016689+0000","last_became_active":"2024-09-13T11:26:05.240552+0000","last_became_peered":"2024-09-13T11:26:05.240552+0000","last_unstale":"2024-09-13T11:26:57.016689+0000","last_undegraded":"2024-09-13T11:26:57.016689+0000","last_fullsized":"2024-09-13T11:26:57.016689+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":6,"last_scrub":"188'6512","last_scrub_stamp":"2024-09-13T11:26:01.118438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:26:01.118438+0000","objects_scrubbed":972,"log_size":7347,"log_dups_size":0,"ondisk_log_size":7347,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:27:06.263451+0000","scrub_duration":183,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8978432,"num_objects":137,"num_object_clones":0,"num_object_copies":274,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":137,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2255,"num_write_kb":9280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2247,"num_bytes_recovered":61014016,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,14],"acting":[9,14],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.25","version":"229'7295","reported_seq":13155,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-09-13T11:26:56.093744+0000","last_change":"2024-09-13T11:24:57.073162+0000","last_active":"2024-09-13T11:26:56.093744+0000","last_peered":"2024-09-13T11:26:56.093744+0000","last_clean":"2024-09-13T11:26:56.093744+0000","last_became_active":"2024-09-13T11:24:55.999019+0000","last_became_peered":"2024-09-13T11:24:55.999019+0000","last_unstale":"2024-09-13T11:26:56.093744+0000","last_undegraded":"2024-09-13T11:26:56.093744+0000","last_fullsized":"2024-09-13T11:26:56.093744+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":6,"last_scrub":"142'5214","last_scrub_stamp":"2024-09-13T11:24:57.073061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:24:57.073061+0000","objects_scrubbed":2249,"log_size":7295,"log_dups_size":0,"ondisk_log_size":7295,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":571,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11206656,"num_objects":171,"num_object_clones":0,"num_object_copies":342,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":171,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2084,"num_write_kb":192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,2],"acting":[12,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[]},{"pgid":"3.26","version":"229'7417","reported_seq":13134,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-09-13T11:26:56.087057+0000","last_change":"2024-09-13T11:26:11.113925+0000","last_active":"2024-09-13T11:26:56.087057+0000","last_peered":"2024-09-13T11:26:56.087057+0000","last_clean":"2024-09-13T11:26:56.087057+0000","last_became_active":"2024-09-13T11:24:54.976919+0000","last_became_peered":"2024-09-13T11:24:54.976919+0000","last_unstale":"2024-09-13T11:26:56.087057+0000","last_undegraded":"2024-09-13T11:26:56.087057+0000","last_fullsized":"2024-09-13T11:26:56.087057+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":6,"last_scrub":"196'6743","last_scrub_stamp":"2024-09-13T11:26:11.113811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:26:11.113811+0000","objects_scrubbed":840,"log_size":7417,"log_dups_size":0,"ondisk_log_size":7417,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:27:18.228492+0000","scrub_duration":103,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10878976,"num_objects":166,"num_object_clones":0,"num_object_copies":332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":166,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1992,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,12],"acting":[8,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.27","version":"229'7315","reported_seq":12983,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-09-13T11:26:55.916348+0000","last_change":"2024-09-13T11:25:37.344464+0000","last_active":"2024-09-13T11:26:55.916348+0000","last_peered":"2024-09-13T11:26:55.916348+0000","last_clean":"2024-09-13T11:26:55.916348+0000","last_became_active":"2024-09-13T11:25:30.257911+0000","last_became_peered":"2024-09-13T11:25:30.257911+0000","last_unstale":"2024-09-13T11:26:55.916348+0000","last_undegraded":"2024-09-13T11:26:55.916348+0000","last_fullsized":"2024-09-13T11:26:55.916348+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":6,"last_scrub":"174'6026","last_scrub_stamp":"2024-09-13T11:25:37.344335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:37.344335+0000","objects_scrubbed":1463,"log_size":7315,"log_dups_size":0,"ondisk_log_size":7315,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":195,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11337728,"num_objects":173,"num_object_clones":0,"num_object_copies":346,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":173,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1290,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,11],"acting":[5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.28","version":"229'7221","reported_seq":12770,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-09-13T11:26:56.030206+0000","last_change":"2024-09-13T11:25:36.143261+0000","last_active":"2024-09-13T11:26:56.030206+0000","last_peered":"2024-09-13T11:26:56.030206+0000","last_clean":"2024-09-13T11:26:56.030206+0000","last_became_active":"2024-09-13T11:25:34.278640+0000","last_became_peered":"2024-09-13T11:25:34.278640+0000","last_unstale":"2024-09-13T11:26:56.030206+0000","last_undegraded":"2024-09-13T11:26:56.030206+0000","last_fullsized":"2024-09-13T11:26:56.030206+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":6,"last_scrub":"174'5936","last_scrub_stamp":"2024-09-13T11:25:36.143113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:36.143113+0000","objects_scrubbed":1447,"log_size":7221,"log_dups_size":0,"ondisk_log_size":7221,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:27:05.839849+0000","scrub_duration":1147,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9371648,"num_objects":143,"num_object_clones":0,"num_object_copies":286,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":143,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1304,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5],"acting":[8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.29","version":"229'7362","reported_seq":13184,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-09-13T11:26:55.894927+0000","last_change":"2024-09-13T11:25:13.333062+0000","last_active":"2024-09-13T11:26:55.894927+0000","last_peered":"2024-09-13T11:26:55.894927+0000","last_clean":"2024-09-13T11:26:55.894927+0000","last_became_active":"2024-09-13T11:25:12.944028+0000","last_became_peered":"2024-09-13T11:25:12.944028+0000","last_unstale":"2024-09-13T11:26:55.894927+0000","last_undegraded":"2024-09-13T11:26:55.894927+0000","last_fullsized":"2024-09-13T11:26:55.894927+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":6,"last_scrub":"154'5579","last_scrub_stamp":"2024-09-13T11:25:13.332959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:13.332959+0000","objects_scrubbed":1937,"log_size":7362,"log_dups_size":0,"ondisk_log_size":7362,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":243,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9961472,"num_objects":152,"num_object_clones":0,"num_object_copies":304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":152,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1785,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2a","version":"229'7767","reported_seq":11791,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-09-13T11:26:56.008839+0000","last_change":"2024-09-13T11:26:09.477886+0000","last_active":"2024-09-13T11:26:56.008839+0000","last_peered":"2024-09-13T11:26:56.008839+0000","last_clean":"2024-09-13T11:26:56.008839+0000","last_became_active":"2024-09-13T11:24:45.320540+0000","last_became_peered":"2024-09-13T11:24:45.320540+0000","last_unstale":"2024-09-13T11:26:56.008839+0000","last_undegraded":"2024-09-13T11:26:56.008839+0000","last_fullsized":"2024-09-13T11:26:56.008839+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":6,"last_scrub":"195'7004","last_scrub_stamp":"2024-09-13T11:26:09.477793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:26:09.477793+0000","objects_scrubbed":929,"log_size":7767,"log_dups_size":0,"ondisk_log_size":7767,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:27:14.239925+0000","scrub_duration":115,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10878976,"num_objects":166,"num_object_clones":0,"num_object_copies":332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":166,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2278,"num_write_kb":11200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,7],"acting":[10,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.2b","version":"232'7576","reported_seq":11419,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-13T11:27:00.196879+0000","last_change":"2024-09-13T11:26:01.126562+0000","last_active":"2024-09-13T11:27:00.196879+0000","last_peered":"2024-09-13T11:27:00.196879+0000","last_clean":"2024-09-13T11:27:00.196879+0000","last_became_active":"2024-09-13T11:24:26.180578+0000","last_became_peered":"2024-09-13T11:24:26.180578+0000","last_unstale":"2024-09-13T11:27:00.196879+0000","last_undegraded":"2024-09-13T11:27:00.196879+0000","last_fullsized":"2024-09-13T11:27:00.196879+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":6,"last_scrub":"188'6745","last_scrub_stamp":"2024-09-13T11:26:01.126423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:26:01.126423+0000","objects_scrubbed":916,"log_size":7576,"log_dups_size":0,"ondisk_log_size":7576,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:27:23.916998+0000","scrub_duration":391,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5111808,"num_objects":78,"num_object_clones":0,"num_object_copies":156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":78,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2194,"num_write_kb":10624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2c","version":"229'7710","reported_seq":13185,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-09-13T11:26:56.078893+0000","last_change":"2024-09-13T11:26:34.433494+0000","last_active":"2024-09-13T11:26:56.078893+0000","last_peered":"2024-09-13T11:26:56.078893+0000","last_clean":"2024-09-13T11:26:56.078893+0000","last_became_active":"2024-09-13T11:24:59.021895+0000","last_became_peered":"2024-09-13T11:24:59.021895+0000","last_unstale":"2024-09-13T11:26:56.078893+0000","last_undegraded":"2024-09-13T11:26:56.078893+0000","last_fullsized":"2024-09-13T11:26:56.078893+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":6,"last_scrub":"211'7422","last_scrub_stamp":"2024-09-13T11:26:34.433371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:26:34.433371+0000","objects_scrubbed":442,"log_size":7710,"log_dups_size":0,"ondisk_log_size":7710,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:27:39.180701+0000","scrub_duration":63,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10092544,"num_objects":154,"num_object_clones":0,"num_object_copies":308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":154,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2038,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,7],"acting":[10,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.2d","version":"229'7645","reported_seq":13290,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-09-13T11:26:56.076310+0000","last_change":"2024-09-13T11:25:45.355259+0000","last_active":"2024-09-13T11:26:56.076310+0000","last_peered":"2024-09-13T11:26:56.076310+0000","last_clean":"2024-09-13T11:26:56.076310+0000","last_became_active":"2024-09-13T11:25:39.334408+0000","last_became_peered":"2024-09-13T11:25:39.334408+0000","last_unstale":"2024-09-13T11:26:56.076310+0000","last_undegraded":"2024-09-13T11:26:56.076310+0000","last_fullsized":"2024-09-13T11:26:56.076310+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":6,"last_scrub":"179'6515","last_scrub_stamp":"2024-09-13T11:25:45.355191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:45.355191+0000","objects_scrubbed":1320,"log_size":7645,"log_dups_size":0,"ondisk_log_size":7645,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":199,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12189696,"num_objects":186,"num_object_clones":0,"num_object_copies":372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":186,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1134,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2e","version":"229'10139","reported_seq":15694,"reported_epoch":229,"state":"active+recovery_wait+degraded+remapped","last_fresh":"2024-09-13T11:26:56.123855+0000","last_change":"2024-09-13T11:26:53.322743+0000","last_active":"2024-09-13T11:26:56.123855+0000","last_peered":"2024-09-13T11:26:56.123855+0000","last_clean":"2024-09-13T11:26:48.728528+0000","last_became_active":"2024-09-13T11:26:53.301660+0000","last_became_peered":"2024-09-13T11:26:53.301660+0000","last_unstale":"2024-09-13T11:26:56.123855+0000","last_undegraded":"2024-09-13T11:26:53.143465+0000","last_fullsized":"2024-09-13T11:26:56.123855+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":6,"last_scrub":"180'8032","last_scrub_stamp":"2024-09-13T11:25:46.438107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:46.438107+0000","objects_scrubbed":2456,"log_size":10139,"log_dups_size":0,"ondisk_log_size":10139,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":327,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9830400,"num_objects":150,"num_object_clones":0,"num_object_copies":300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2129,"num_objects_misplaced":2129,"num_objects_unfound":0,"num_objects_dirty":150,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3329,"num_write_kb":82944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":2949120,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11],"acting":[0,9],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":162},{"shards":"0,9","objects":-12}],"blocked_by":[],"up_primary":9,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2f","version":"230'10133","reported_seq":11136,"reported_epoch":230,"state":"active+recovering+remapped","last_fresh":"2024-09-13T11:26:57.130213+0000","last_change":"2024-09-13T11:26:53.315481+0000","last_active":"2024-09-13T11:26:57.130213+0000","last_peered":"2024-09-13T11:26:57.130213+0000","last_clean":"2024-09-13T11:26:50.828909+0000","last_became_active":"2024-09-13T11:26:53.275643+0000","last_became_peered":"2024-09-13T11:26:53.275643+0000","last_unstale":"2024-09-13T11:26:57.130213+0000","last_undegraded":"2024-09-13T11:26:57.130213+0000","last_fullsized":"2024-09-13T11:26:57.130213+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":6,"last_scrub":"211'9653","last_scrub_stamp":"2024-09-13T11:26:34.940878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:26:34.940878+0000","objects_scrubbed":877,"log_size":10133,"log_dups_size":0,"ondisk_log_size":10133,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":119,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10649600,"num_objects":163,"num_object_clones":0,"num_object_copies":326,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":114,"num_objects_unfound":0,"num_objects_dirty":163,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3391,"num_write_kb":85280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":155,"num_bytes_recovered":10158080,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,4],"acting":[11,4],"avail_no_missing":["11","12"],"object_location_counts":[{"shards":"4,11","objects":106},{"shards":"11,12","objects":57}],"blocked_by":[],"up_primary":14,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.30","version":"229'9884","reported_seq":14979,"reported_epoch":229,"state":"active+recovering+remapped","last_fresh":"2024-09-13T11:26:56.146089+0000","last_change":"2024-09-13T11:26:53.427506+0000","last_active":"2024-09-13T11:26:56.146089+0000","last_peered":"2024-09-13T11:26:56.146089+0000","last_clean":"2024-09-13T11:26:50.184017+0000","last_became_active":"2024-09-13T11:26:53.295253+0000","last_became_peered":"2024-09-13T11:26:53.295253+0000","last_unstale":"2024-09-13T11:26:56.146089+0000","last_undegraded":"2024-09-13T11:26:56.146089+0000","last_fullsized":"2024-09-13T11:26:56.146089+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":6,"last_scrub":"204'8963","last_scrub_stamp":"2024-09-13T11:26:22.296795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:26:22.296795+0000","objects_scrubbed":1288,"log_size":9884,"log_dups_size":0,"ondisk_log_size":9884,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":195,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10125312,"num_objects":155,"num_object_clones":0,"num_object_copies":310,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":105,"num_objects_unfound":0,"num_objects_dirty":155,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3236,"num_write_kb":76000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":138,"num_bytes_recovered":9043968,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,15],"acting":[0,3],"avail_no_missing":["0","6"],"object_location_counts":[{"shards":"0,3","objects":102},{"shards":"0,3,6","objects":1},{"shards":"0,6","objects":52}],"blocked_by":[],"up_primary":3,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"229'9782","reported_seq":13604,"reported_epoch":229,"state":"active+recovering+remapped","last_fresh":"2024-09-13T11:26:56.095926+0000","last_change":"2024-09-13T11:26:53.310009+0000","last_active":"2024-09-13T11:26:56.095926+0000","last_peered":"2024-09-13T11:26:56.095926+0000","last_clean":"2024-09-13T11:26:50.942461+0000","last_became_active":"2024-09-13T11:26:53.272926+0000","last_became_peered":"2024-09-13T11:26:53.272926+0000","last_unstale":"2024-09-13T11:26:56.095926+0000","last_undegraded":"2024-09-13T11:26:56.095926+0000","last_fullsized":"2024-09-13T11:26:56.095926+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":6,"last_scrub":"165'6821","last_scrub_stamp":"2024-09-13T11:25:24.287082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:24.896055+0000","last_clean_scrub_stamp":"2024-09-13T11:25:24.287082+0000","objects_scrubbed":3406,"log_size":9782,"log_dups_size":0,"ondisk_log_size":9782,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":947,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12320768,"num_objects":188,"num_object_clones":0,"num_object_copies":376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":207,"num_objects_unfound":0,"num_objects_dirty":188,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3249,"num_write_kb":77792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":129,"num_bytes_recovered":8454144,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[4,1],"avail_no_missing":["4","11"],"object_location_counts":[{"shards":"1,4","objects":84},{"shards":"1,4,11","objects":1},{"shards":"4,11","objects":103}],"blocked_by":[],"up_primary":1,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":56,"stat_sum":{"num_bytes":620199960,"num_objects":9465,"num_object_clones":0,"num_object_copies":18930,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2129,"num_objects_misplaced":2902,"num_objects_unfound":0,"num_objects_dirty":9465,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":174038,"num_write_kb":3202593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26905,"num_bytes_recovered":1357971456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1604395008,"data_stored":1604386864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":43282,"internal_metadata":0},"log_size":478577,"ondisk_log_size":478577,"up":112,"acting":112,"num_store_stats":16},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":11},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":112,"ondisk_log_size":112,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":13,"up_from":23,"seq":98784247869,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":176292,"kb_used_data":64596,"kb_used_omap":1,"kb_used_meta":111678,"kb_avail":94195548,"statfs":{"total":96636764160,"available":96456241152,"internally_reserved":0,"allocated":66146304,"data_stored":65692541,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":114358725},"hb_peers":[0,1,2,4,6,8,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":23,"seq":98784247868,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":125688,"kb_used_data":58408,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94246152,"statfs":{"total":96636764160,"available":96508059648,"internally_reserved":0,"allocated":59809792,"data_stored":59389824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":68876736},"hb_peers":[0,1,3,8,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":40,"seq":171798691892,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":182860,"kb_used_data":74492,"kb_used_omap":4,"kb_used_meta":108347,"kb_avail":94188980,"statfs":{"total":96636764160,"available":96449515520,"internally_reserved":0,"allocated":76279808,"data_stored":75863086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4433,"internal_metadata":110948015},"hb_peers":[0,2,3,4,7,8,9,11,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":21,"seq":90194313277,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":224484,"kb_used_data":80852,"kb_used_omap":1,"kb_used_meta":143614,"kb_avail":94147356,"statfs":{"total":96636764160,"available":96406892544,"internally_reserved":0,"allocated":82792448,"data_stored":82345677,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":147061191},"hb_peers":[0,2,4,6,7,9,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":21,"seq":90194313279,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":300084,"kb_used_data":174436,"kb_used_omap":1,"kb_used_meta":125566,"kb_avail":94071756,"statfs":{"total":96636764160,"available":96329478144,"internally_reserved":0,"allocated":178622464,"data_stored":178207969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":128580039},"hb_peers":[0,2,4,6,7,9,10,11,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":22,"seq":94489280571,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":272332,"kb_used_data":137020,"kb_used_omap":1,"kb_used_meta":135294,"kb_avail":94099508,"statfs":{"total":96636764160,"available":96357896192,"internally_reserved":0,"allocated":140308480,"data_stored":139871255,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":138541509},"hb_peers":[1,3,5,6,7,8,9,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":22,"seq":94489280572,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":134800,"kb_used_data":63360,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":94237040,"statfs":{"total":96636764160,"available":96498728960,"internally_reserved":0,"allocated":64880640,"data_stored":64439085,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73136581},"hb_peers":[0,1,3,7,8,10,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":15,"up_from":22,"seq":94489280572,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":257768,"kb_used_data":114136,"kb_used_omap":1,"kb_used_meta":143614,"kb_avail":94114072,"statfs":{"total":96636764160,"available":96372809728,"internally_reserved":0,"allocated":116875264,"data_stored":116464561,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":147061190},"hb_peers":[0,4,5,6,7,8,10,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":22,"seq":94489280573,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":150908,"kb_used_data":79468,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":94220932,"statfs":{"total":96636764160,"available":96482234368,"internally_reserved":0,"allocated":81375232,"data_stored":80952580,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73136581},"hb_peers":[1,3,5,7,9,10,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":22,"seq":94489280572,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":123848,"kb_used_data":48248,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94247992,"statfs":{"total":96636764160,"available":96509943808,"internally_reserved":0,"allocated":49405952,"data_stored":48972666,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":77396422},"hb_peers":[0,1,3,5,7,8,10,11,12,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":22,"seq":94489280573,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":280236,"kb_used_data":140764,"kb_used_omap":1,"kb_used_meta":139454,"kb_avail":94091604,"statfs":{"total":96636764160,"available":96349802496,"internally_reserved":0,"allocated":144142336,"data_stored":143733708,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":142801354},"hb_peers":[0,2,3,5,6,8,11,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":22,"seq":94489280572,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":112840,"kb_used_data":45560,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94259000,"statfs":{"total":96636764160,"available":96521216000,"internally_reserved":0,"allocated":46653440,"data_stored":46213001,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,1,5,7,9,10,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":22,"seq":94489280572,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":220420,"kb_used_data":108724,"kb_used_omap":1,"kb_used_meta":111678,"kb_avail":94151420,"statfs":{"total":96636764160,"available":96411054080,"internally_reserved":0,"allocated":111333376,"data_stored":110896645,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":114358720},"hb_peers":[0,1,3,5,7,9,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":11,"up_from":22,"seq":94489280573,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":284264,"kb_used_data":140568,"kb_used_omap":1,"kb_used_meta":143614,"kb_avail":94087576,"statfs":{"total":96636764160,"available":96345677824,"internally_reserved":0,"allocated":143941632,"data_stored":143484091,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":147061188},"hb_peers":[0,2,4,7,8,10,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":12,"up_from":22,"seq":94489280572,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":291980,"kb_used_data":144188,"kb_used_omap":1,"kb_used_meta":147774,"kb_avail":94079860,"statfs":{"total":96636764160,"available":96337776640,"internally_reserved":0,"allocated":147648512,"data_stored":147210901,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":151321029},"hb_peers":[0,1,2,3,4,5,10,11,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":14,"up_from":142,"seq":609885356058,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":270784,"kb_used_data":176432,"kb_used_omap":42,"kb_used_meta":94293,"kb_avail":94101056,"statfs":{"total":96636764160,"available":96359481344,"internally_reserved":0,"allocated":180666368,"data_stored":180259314,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":43282,"internal_metadata":96556782},"hb_peers":[0,1,4,5,7,9,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":136183808,"data_stored":136183808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65470464,"data_stored":65470464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":64819200,"data_stored":64815128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":75038720,"data_stored":75038720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":78249984,"data_stored":78249984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":172359680,"data_stored":172359680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":47775744,"data_stored":47775744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":135004160,"data_stored":135004160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":40960000,"data_stored":40960000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":52232192,"data_stored":52232192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":107347968,"data_stored":107347968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":138805248,"data_stored":138805248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":142737408,"data_stored":142737408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":59834368,"data_stored":59834368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":178585600,"data_stored":178585600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":43282,"internal_metadata":0},{"poolid":3,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":108990464,"data_stored":108986392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T11:27:00.937 INFO:tasks.ceph.ceph_manager.ceph:increase pool size by 10 2024-09-13T11:27:00.938 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool set unique_pool_0 pg_num 66 2024-09-13T11:27:00.959 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-13T11:27:00.959 INFO:teuthology.orchestra.run.smithi001.stdout: "size": 20, 2024-09-13T11:27:00.959 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 600, 2024-09-13T11:27:00.960 INFO:teuthology.orchestra.run.smithi001.stdout: "ops": [ 2024-09-13T11:27:00.960 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:00.960 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDPGPushReply(3.1e 84/78 [PushReplyOp(3:79ee54b9:::benchmark_data_smithi177_31116_object11701:head)])", 2024-09-13T11:27:00.960 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:41.407335+0000", 2024-09-13T11:27:00.960 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 199.550445559, 2024-09-13T11:27:00.960 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.51622350400000006, 2024-09-13T11:27:00.960 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:27:00.960 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:27:00.960 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-13T11:27:00.960 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:27:00.960 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:00.960 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:27:00.961 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:41.407335+0000", 2024-09-13T11:27:00.961 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:27:00.961 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:00.961 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:00.961 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:27:00.961 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:41.407335+0000", 2024-09-13T11:27:00.961 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:27:00.961 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:00.961 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:00.961 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:27:00.961 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:41.407337+0000", 2024-09-13T11:27:00.961 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.505e-06 2024-09-13T11:27:00.961 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:00.961 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:00.961 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:27:00.962 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:41.407343+0000", 2024-09-13T11:27:00.962 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.9089999999999997e-06 2024-09-13T11:27:00.962 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:00.962 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:00.962 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:27:00.962 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:41.921977+0000", 2024-09-13T11:27:00.962 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.51463463499999995 2024-09-13T11:27:00.962 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:00.962 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:00.962 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:27:00.962 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:41.921988+0000", 2024-09-13T11:27:00.962 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.092e-05 2024-09-13T11:27:00.962 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:00.962 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:00.963 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:27:00.963 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:41.922088+0000", 2024-09-13T11:27:00.963 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.9528e-05 2024-09-13T11:27:00.963 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:00.963 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:00.963 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:27:00.963 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:41.923559+0000", 2024-09-13T11:27:00.963 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0014710070000000001 2024-09-13T11:27:00.963 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:27:00.963 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:27:00.963 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:27:00.963 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:00.963 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:00.963 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4665.0:60417 3.10 e85/84 3:092808b0:::benchmark_data_smithi177_31116_object60416:head v 85'3295, mlcod=85'3292)", 2024-09-13T11:27:00.964 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:42.174488+0000", 2024-09-13T11:27:00.964 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 198.78329300499999, 2024-09-13T11:27:00.964 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.387908896, 2024-09-13T11:27:00.964 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:27:00.964 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:27:00.964 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:27:00.964 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:27:00.964 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:00.964 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:27:00.964 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.174488+0000", 2024-09-13T11:27:00.964 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:27:00.964 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:00.964 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:00.965 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:27:00.965 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.174488+0000", 2024-09-13T11:27:00.965 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:27:00.965 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:00.965 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:00.965 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:27:00.965 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.174490+0000", 2024-09-13T11:27:00.965 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.9099999999999999e-06 2024-09-13T11:27:00.965 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:00.965 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:00.965 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:27:00.965 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.174527+0000", 2024-09-13T11:27:00.965 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.7543000000000003e-05 2024-09-13T11:27:00.966 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:00.966 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:00.966 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:27:00.966 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.561093+0000", 2024-09-13T11:27:00.966 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.38656578400000002 2024-09-13T11:27:00.966 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:00.966 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:00.966 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:27:00.966 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.561101+0000", 2024-09-13T11:27:00.966 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.6090000000000003e-06 2024-09-13T11:27:00.966 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:00.966 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:00.966 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:27:00.966 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.561143+0000", 2024-09-13T11:27:00.967 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.2435000000000003e-05 2024-09-13T11:27:00.967 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:00.967 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:00.967 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:27:00.967 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.561164+0000", 2024-09-13T11:27:00.967 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.1276000000000001e-05 2024-09-13T11:27:00.967 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:00.967 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:00.967 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:27:00.967 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.562360+0000", 2024-09-13T11:27:00.967 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00119596 2024-09-13T11:27:00.967 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:00.967 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:00.967 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:27:00.968 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.562397+0000", 2024-09-13T11:27:00.968 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.6378999999999998e-05 2024-09-13T11:27:00.968 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:27:00.968 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:27:00.968 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:27:00.968 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:00.968 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:00.968 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDPGPush(3.22 85/78 [PushOp(3:456e8765:::benchmark_data_smithi177_31116_object16212:head, version: 52'1006, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:456e8765:::benchmark_data_smithi177_31116_object16212:head@52'1006, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:27:00.968 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:42.673067+0000", 2024-09-13T11:27:00.968 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 198.28471386300001, 2024-09-13T11:27:00.968 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.27935889899999999, 2024-09-13T11:27:00.968 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:27:00.968 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:27:00.968 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:27:00.969 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:27:00.969 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:00.969 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:27:00.969 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.673067+0000", 2024-09-13T11:27:00.969 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:27:00.969 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:00.969 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:00.969 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:27:00.969 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.673067+0000", 2024-09-13T11:27:00.969 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:27:00.969 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:00.969 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:00.969 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:27:00.970 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.673068+0000", 2024-09-13T11:27:00.970 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.9699999999999994e-07 2024-09-13T11:27:00.970 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:00.970 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:00.970 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:27:00.970 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.673090+0000", 2024-09-13T11:27:00.970 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.2416999999999999e-05 2024-09-13T11:27:00.970 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:00.970 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:00.971 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:27:00.971 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.951363+0000", 2024-09-13T11:27:00.971 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.27827291199999998 2024-09-13T11:27:00.971 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:00.971 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:00.971 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:27:00.971 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.951386+0000", 2024-09-13T11:27:00.971 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.2226999999999999e-05 2024-09-13T11:27:00.971 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:00.971 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:00.972 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:27:00.972 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.951581+0000", 2024-09-13T11:27:00.972 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000195713 2024-09-13T11:27:00.972 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:00.972 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:00.973 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:27:00.973 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.951605+0000", 2024-09-13T11:27:00.973 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.3767000000000001e-05 2024-09-13T11:27:00.973 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:00.973 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:00.973 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:27:00.973 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.952426+0000", 2024-09-13T11:27:00.973 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00082086599999999996 2024-09-13T11:27:00.973 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:27:00.973 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:27:00.974 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:27:00.974 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:00.974 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:00.974 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "replica_scrub(pg: 3.20,from:0'0,to:89'3254,epoch:89/77,start:3:04000000::::0,end:3:045ba805:::benchmark_data_smithi177_31116_object43251:0,chunky:1,deep:0,version:9,allow_preemption:1,priority=5)", 2024-09-13T11:27:00.974 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:46.641364+0000", 2024-09-13T11:27:00.974 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 194.31641718200001, 2024-09-13T11:27:00.974 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.27141393000000003, 2024-09-13T11:27:00.974 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:27:00.974 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:27:00.974 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:27:00.975 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:27:00.975 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:00.975 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:27:00.975 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.641364+0000", 2024-09-13T11:27:00.975 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:27:00.975 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:00.975 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:00.975 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:27:00.975 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.641364+0000", 2024-09-13T11:27:00.976 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:27:00.976 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:00.976 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:00.976 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:27:00.976 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.641365+0000", 2024-09-13T11:27:00.976 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.266e-06 2024-09-13T11:27:00.976 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:00.976 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:00.976 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:27:00.977 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.641370+0000", 2024-09-13T11:27:00.977 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.8539999999999996e-06 2024-09-13T11:27:00.977 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:00.977 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:00.977 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:27:00.977 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.912491+0000", 2024-09-13T11:27:00.977 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.27112103100000001 2024-09-13T11:27:00.977 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:00.977 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:00.978 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:27:00.978 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.912520+0000", 2024-09-13T11:27:00.978 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.9379000000000001e-05 2024-09-13T11:27:00.978 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:00.978 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:00.978 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:27:00.978 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.912679+0000", 2024-09-13T11:27:00.978 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00015915800000000001 2024-09-13T11:27:00.978 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:00.978 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:00.979 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:27:00.979 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.912714+0000", 2024-09-13T11:27:00.979 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.4687999999999997e-05 2024-09-13T11:27:00.979 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:00.979 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:00.979 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:27:00.979 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.912778+0000", 2024-09-13T11:27:00.979 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.3553999999999995e-05 2024-09-13T11:27:00.979 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:27:00.980 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:27:00.980 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:27:00.980 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:00.980 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:00.980 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDPGPush(3.18 89/70 [PushOp(3:1ac6c361:::benchmark_data_smithi177_31116_object44972:head, version: 68'2767, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:1ac6c361:::benchmark_data_smithi177_31116_object44972:head@68'2767, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:27:00.980 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:46.671018+0000", 2024-09-13T11:27:00.980 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 194.28676304999999, 2024-09-13T11:27:00.981 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.271282931, 2024-09-13T11:27:00.981 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:27:00.981 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:27:00.981 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:27:00.981 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:27:00.981 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:00.981 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:27:00.981 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.671018+0000", 2024-09-13T11:27:00.982 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:27:00.982 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:00.982 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:00.982 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:27:00.982 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.671018+0000", 2024-09-13T11:27:00.982 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:27:00.982 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:00.982 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:00.982 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:27:00.983 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.671020+0000", 2024-09-13T11:27:00.983 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.018e-06 2024-09-13T11:27:00.983 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:00.983 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:00.983 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:27:00.983 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.671097+0000", 2024-09-13T11:27:00.983 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.7218999999999994e-05 2024-09-13T11:27:00.983 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:00.983 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:00.984 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:27:00.984 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.939773+0000", 2024-09-13T11:27:00.984 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.26867550200000001 2024-09-13T11:27:00.984 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:00.984 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:00.984 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:27:00.984 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.939791+0000", 2024-09-13T11:27:00.984 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.8185999999999998e-05 2024-09-13T11:27:00.985 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:00.985 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:00.985 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:27:00.985 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.939926+0000", 2024-09-13T11:27:00.985 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000135291 2024-09-13T11:27:00.985 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:00.985 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:00.985 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:27:00.985 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.939938+0000", 2024-09-13T11:27:00.985 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2393e-05 2024-09-13T11:27:00.985 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:00.986 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:00.986 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:27:00.986 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.942301+0000", 2024-09-13T11:27:00.986 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.002362322 2024-09-13T11:27:00.986 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:27:00.986 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:27:00.986 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:27:00.986 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:00.986 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:00.987 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "replica_scrub(pg: 3.20,from:0'0,to:90'3264,epoch:90/77,start:3:0650aac8:::benchmark_data_smithi177_31116_object1734:0,end:3:06af01e3:::benchmark_data_smithi177_31116_object31572:0,chunky:1,deep:0,version:9,allow_preemption:1,priority=5)", 2024-09-13T11:27:00.987 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:47.380615+0000", 2024-09-13T11:27:00.987 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 193.577165627, 2024-09-13T11:27:00.987 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.51951398400000004, 2024-09-13T11:27:00.987 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:27:00.987 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:27:00.987 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:27:00.987 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:27:00.987 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:00.987 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:27:00.988 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:47.380615+0000", 2024-09-13T11:27:00.988 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:27:00.988 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:00.988 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:00.988 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:27:00.988 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:47.380615+0000", 2024-09-13T11:27:00.988 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:27:00.988 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:00.988 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:00.988 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:27:00.989 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:47.380617+0000", 2024-09-13T11:27:00.989 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.671e-06 2024-09-13T11:27:00.989 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:00.989 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:00.989 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:27:00.989 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:47.380623+0000", 2024-09-13T11:27:00.989 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.1190000000000002e-06 2024-09-13T11:27:00.989 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:00.989 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:00.989 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:27:00.990 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:47.900050+0000", 2024-09-13T11:27:00.990 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.51942737400000005 2024-09-13T11:27:00.990 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:00.990 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:00.990 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:27:00.990 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:47.900057+0000", 2024-09-13T11:27:00.990 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.4359999999999999e-06 2024-09-13T11:27:00.990 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:00.991 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:00.991 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:27:00.991 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:47.900101+0000", 2024-09-13T11:27:00.991 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.4675000000000002e-05 2024-09-13T11:27:00.991 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:00.991 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:00.991 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:27:00.991 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:47.900110+0000", 2024-09-13T11:27:00.991 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.0569999999999993e-06 2024-09-13T11:27:00.991 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:00.991 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:00.992 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:27:00.992 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:47.900129+0000", 2024-09-13T11:27:00.992 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.9652000000000001e-05 2024-09-13T11:27:00.992 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:27:00.992 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:27:00.992 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:27:00.992 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:00.993 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:00.993 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDPGPush(3.21 91/78 [PushOp(3:8492321f:::benchmark_data_smithi177_31116_object13767:head, version: 50'840, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:8492321f:::benchmark_data_smithi177_31116_object13767:head@50'840, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:27:00.993 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:48.319251+0000", 2024-09-13T11:27:00.993 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 192.63853009799999, 2024-09-13T11:27:00.993 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.60903802200000001, 2024-09-13T11:27:00.993 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:27:00.993 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:27:00.993 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:27:00.993 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:27:00.993 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:00.994 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:27:00.994 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:48.319251+0000", 2024-09-13T11:27:00.994 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:27:00.994 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:00.994 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:00.994 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:27:00.994 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:48.319251+0000", 2024-09-13T11:27:00.994 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:27:00.994 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:00.994 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:00.994 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:27:00.995 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:48.319252+0000", 2024-09-13T11:27:00.995 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2640000000000001e-06 2024-09-13T11:27:00.995 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:00.995 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:00.995 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:27:00.995 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:48.319363+0000", 2024-09-13T11:27:00.995 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000110559 2024-09-13T11:27:00.995 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:00.995 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:00.995 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:27:00.996 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:48.927115+0000", 2024-09-13T11:27:00.996 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.60775277800000005 2024-09-13T11:27:00.996 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:00.996 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:00.996 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:27:00.996 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:48.927144+0000", 2024-09-13T11:27:00.996 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.8861e-05 2024-09-13T11:27:00.996 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:00.996 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:00.997 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:27:00.997 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:48.927318+0000", 2024-09-13T11:27:00.997 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00017399000000000001 2024-09-13T11:27:00.997 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:00.997 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:00.997 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:27:00.997 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:48.927340+0000", 2024-09-13T11:27:00.997 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.1433999999999998e-05 2024-09-13T11:27:00.997 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:00.997 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:00.998 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:27:00.998 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:48.928289+0000", 2024-09-13T11:27:00.998 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00094913599999999995 2024-09-13T11:27:00.998 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:27:00.998 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:27:00.998 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:27:00.999 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:00.999 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:00.999 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:83239 3.e 3:74547f17:::benchmark_data_smithi177_31116_object83238:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106)", 2024-09-13T11:27:00.999 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:24:04.580085+0000", 2024-09-13T11:27:00.999 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 176.37769601700001, 2024-09-13T11:27:00.999 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.33785762600000002, 2024-09-13T11:27:00.999 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:27:00.999 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:27:00.999 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:27:00.999 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:27:00.999 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:27:01.000 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:27:01.000 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 83239 2024-09-13T11:27:01.000 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.000 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:27:01.000 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.000 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:27:01.000 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:04.580085+0000", 2024-09-13T11:27:01.001 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:27:01.001 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.001 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.001 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:27:01.001 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:04.580085+0000", 2024-09-13T11:27:01.001 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:27:01.001 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.001 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.001 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:27:01.001 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:04.580089+0000", 2024-09-13T11:27:01.001 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.053e-06 2024-09-13T11:27:01.002 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.002 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.002 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:27:01.002 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:04.580166+0000", 2024-09-13T11:27:01.002 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.7102999999999997e-05 2024-09-13T11:27:01.002 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.002 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.002 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:27:01.002 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:04.580196+0000", 2024-09-13T11:27:01.002 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.0196e-05 2024-09-13T11:27:01.002 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.002 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.003 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:27:01.003 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:04.580212+0000", 2024-09-13T11:27:01.003 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.535e-05 2024-09-13T11:27:01.003 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.003 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.003 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:27:01.003 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:04.580336+0000", 2024-09-13T11:27:01.003 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00012431899999999999 2024-09-13T11:27:01.003 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.003 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.003 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:27:01.004 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:04.580575+0000", 2024-09-13T11:27:01.004 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00023924100000000001 2024-09-13T11:27:01.004 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.004 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.004 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 1", 2024-09-13T11:27:01.004 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:04.580798+0000", 2024-09-13T11:27:01.004 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00022301499999999999 2024-09-13T11:27:01.004 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.004 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.004 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:27:01.004 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:04.583404+0000", 2024-09-13T11:27:01.005 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0026056120000000002 2024-09-13T11:27:01.005 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.005 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.005 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:27:01.005 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:04.917768+0000", 2024-09-13T11:27:01.005 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.33436387499999998 2024-09-13T11:27:01.005 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.005 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.005 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:27:01.005 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:04.917859+0000", 2024-09-13T11:27:01.006 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.1732999999999997e-05 2024-09-13T11:27:01.006 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.006 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.006 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:27:01.006 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:04.917942+0000", 2024-09-13T11:27:01.006 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.3128999999999996e-05 2024-09-13T11:27:01.006 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:27:01.006 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:27:01.006 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:27:01.006 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.006 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.007 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:110271 3.2d 3.3c2681ad (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e119)", 2024-09-13T11:27:01.007 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:24:25.403006+0000", 2024-09-13T11:27:01.007 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 155.554774533, 2024-09-13T11:27:01.007 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.68768755999999998, 2024-09-13T11:27:01.007 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:27:01.007 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:27:01.007 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-13T11:27:01.007 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:27:01.007 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:27:01.007 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:27:01.008 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 110271 2024-09-13T11:27:01.008 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.008 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:27:01.008 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.008 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:27:01.008 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:25.403006+0000", 2024-09-13T11:27:01.008 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:27:01.008 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.009 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.009 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:27:01.009 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:25.403006+0000", 2024-09-13T11:27:01.009 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:27:01.009 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.009 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.009 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:27:01.009 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:25.403011+0000", 2024-09-13T11:27:01.009 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.5199999999999999e-06 2024-09-13T11:27:01.009 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.010 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.010 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:27:01.010 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:25.403065+0000", 2024-09-13T11:27:01.010 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.4478999999999998e-05 2024-09-13T11:27:01.010 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.010 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.010 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:27:01.010 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:25.403097+0000", 2024-09-13T11:27:01.011 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.1636e-05 2024-09-13T11:27:01.011 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.011 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.011 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:27:01.011 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:25.403121+0000", 2024-09-13T11:27:01.011 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.3924e-05 2024-09-13T11:27:01.011 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.011 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.011 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:27:01.011 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:25.403216+0000", 2024-09-13T11:27:01.011 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.5557999999999995e-05 2024-09-13T11:27:01.012 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.012 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.012 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for peered", 2024-09-13T11:27:01.012 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:25.403234+0000", 2024-09-13T11:27:01.012 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.7104000000000001e-05 2024-09-13T11:27:01.012 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.012 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.012 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:27:01.012 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:26.090684+0000", 2024-09-13T11:27:01.012 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.68745000700000003 2024-09-13T11:27:01.012 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.013 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.013 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:27:01.013 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:26.090694+0000", 2024-09-13T11:27:01.013 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0332e-05 2024-09-13T11:27:01.013 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:27:01.013 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:27:01.013 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:27:01.013 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.013 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.013 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4665.0:129901 3.20 e134/77 3:067f5dfc:::benchmark_data_smithi177_31116_object129900:head v 134'4294, mlcod=134'4291)", 2024-09-13T11:27:01.013 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:24:47.145530+0000", 2024-09-13T11:27:01.014 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 133.81225045799999, 2024-09-13T11:27:01.014 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.333688349, 2024-09-13T11:27:01.014 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:27:01.014 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:27:01.014 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:27:01.014 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:27:01.014 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.014 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:27:01.014 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:47.145530+0000", 2024-09-13T11:27:01.014 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:27:01.014 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.014 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.015 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:27:01.015 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:47.145530+0000", 2024-09-13T11:27:01.015 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:27:01.015 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.015 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.015 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:27:01.015 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:47.145532+0000", 2024-09-13T11:27:01.015 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.7999999999999999e-06 2024-09-13T11:27:01.015 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.015 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.015 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:27:01.016 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:47.145557+0000", 2024-09-13T11:27:01.016 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.4678999999999999e-05 2024-09-13T11:27:01.016 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.016 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.016 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:27:01.016 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:47.477539+0000", 2024-09-13T11:27:01.016 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.33198164099999999 2024-09-13T11:27:01.016 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.016 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.016 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:27:01.016 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:47.477544+0000", 2024-09-13T11:27:01.017 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.9059999999999999e-06 2024-09-13T11:27:01.017 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.017 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.017 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:27:01.017 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:47.477597+0000", 2024-09-13T11:27:01.017 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.295e-05 2024-09-13T11:27:01.017 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.017 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.017 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:27:01.017 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:47.477619+0000", 2024-09-13T11:27:01.017 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.1835999999999999e-05 2024-09-13T11:27:01.017 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.018 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.018 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:27:01.018 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:47.479175+0000", 2024-09-13T11:27:01.019 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0015560470000000001 2024-09-13T11:27:01.019 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.019 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.019 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:27:01.019 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:47.479219+0000", 2024-09-13T11:27:01.019 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.3489999999999999e-05 2024-09-13T11:27:01.019 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:27:01.019 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:27:01.019 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:27:01.019 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.019 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.019 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:140840 3.21 3:87ae3db6:::benchmark_data_smithi177_31116_object537:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e144)", 2024-09-13T11:27:01.019 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:24:58.810939+0000", 2024-09-13T11:27:01.020 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 122.146841345, 2024-09-13T11:27:01.020 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.25628309300000002, 2024-09-13T11:27:01.020 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:27:01.020 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:27:01.020 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:27:01.020 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:27:01.020 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:27:01.020 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:27:01.020 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 140840 2024-09-13T11:27:01.020 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.020 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:27:01.020 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.021 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:27:01.021 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.810939+0000", 2024-09-13T11:27:01.021 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:27:01.021 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.021 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.021 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:27:01.021 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.810939+0000", 2024-09-13T11:27:01.021 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:27:01.021 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.021 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.021 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:27:01.021 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.810942+0000", 2024-09-13T11:27:01.021 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.6259999999999999e-06 2024-09-13T11:27:01.022 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.022 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.022 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:27:01.022 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.810950+0000", 2024-09-13T11:27:01.022 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.9500000000000001e-06 2024-09-13T11:27:01.022 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.022 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.022 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:27:01.022 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.810970+0000", 2024-09-13T11:27:01.022 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.9539999999999999e-05 2024-09-13T11:27:01.022 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.022 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.022 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:27:01.022 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.810978+0000", 2024-09-13T11:27:01.023 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.8680000000000005e-06 2024-09-13T11:27:01.023 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.023 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.023 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:27:01.023 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.811060+0000", 2024-09-13T11:27:01.023 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.1068000000000004e-05 2024-09-13T11:27:01.023 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.023 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.023 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:27:01.023 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.811174+0000", 2024-09-13T11:27:01.023 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000114385 2024-09-13T11:27:01.023 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.023 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.023 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 8", 2024-09-13T11:27:01.024 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.811255+0000", 2024-09-13T11:27:01.024 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.1298000000000007e-05 2024-09-13T11:27:01.024 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.024 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.024 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:27:01.024 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.812869+0000", 2024-09-13T11:27:01.024 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.001614234 2024-09-13T11:27:01.024 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.024 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.024 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:27:01.025 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:59.067155+0000", 2024-09-13T11:27:01.025 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.25428578299999999 2024-09-13T11:27:01.025 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.025 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.025 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:27:01.025 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:59.067192+0000", 2024-09-13T11:27:01.025 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.659e-05 2024-09-13T11:27:01.025 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.025 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.025 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:27:01.025 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:59.067223+0000", 2024-09-13T11:27:01.025 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.0750999999999997e-05 2024-09-13T11:27:01.025 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:27:01.026 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:27:01.026 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:27:01.026 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.026 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.026 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:201234 3.21 3:85402570:::benchmark_data_smithi177_31116_object60931:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e181)", 2024-09-13T11:27:01.026 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:25:47.835618+0000", 2024-09-13T11:27:01.026 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 73.122163180000001, 2024-09-13T11:27:01.026 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.26153554899999998, 2024-09-13T11:27:01.026 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:27:01.027 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:27:01.027 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:27:01.027 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:27:01.027 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:27:01.027 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:27:01.027 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 201234 2024-09-13T11:27:01.027 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.027 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:27:01.027 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.027 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:27:01.027 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:47.835618+0000", 2024-09-13T11:27:01.028 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:27:01.028 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.028 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.028 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:27:01.028 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:47.835618+0000", 2024-09-13T11:27:01.028 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:27:01.028 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.028 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.028 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:27:01.028 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:47.835619+0000", 2024-09-13T11:27:01.028 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.638e-06 2024-09-13T11:27:01.028 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.028 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.029 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:27:01.029 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:47.835627+0000", 2024-09-13T11:27:01.029 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.0150000000000003e-06 2024-09-13T11:27:01.029 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.029 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.029 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:27:01.029 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:47.835639+0000", 2024-09-13T11:27:01.029 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.1939e-05 2024-09-13T11:27:01.029 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.029 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.029 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:27:01.029 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:47.835644+0000", 2024-09-13T11:27:01.030 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.006e-06 2024-09-13T11:27:01.030 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.030 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.030 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:27:01.030 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:47.835696+0000", 2024-09-13T11:27:01.030 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.1668000000000001e-05 2024-09-13T11:27:01.031 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.031 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.031 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:27:01.031 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:47.835775+0000", 2024-09-13T11:27:01.031 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.8737000000000002e-05 2024-09-13T11:27:01.031 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.031 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.032 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 8", 2024-09-13T11:27:01.032 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:47.835857+0000", 2024-09-13T11:27:01.032 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.2508000000000004e-05 2024-09-13T11:27:01.032 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.032 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.032 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:27:01.032 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:47.837366+0000", 2024-09-13T11:27:01.032 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.001509292 2024-09-13T11:27:01.032 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.033 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.033 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:27:01.033 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:48.097051+0000", 2024-09-13T11:27:01.033 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.25968466699999998 2024-09-13T11:27:01.033 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.033 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.033 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:27:01.033 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:48.097109+0000", 2024-09-13T11:27:01.033 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.7778000000000001e-05 2024-09-13T11:27:01.034 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.034 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.034 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:27:01.034 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:48.097153+0000", 2024-09-13T11:27:01.034 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.4301000000000002e-05 2024-09-13T11:27:01.034 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:27:01.034 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:27:01.034 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:27:01.035 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.035 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.035 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:217082 3.22 3:449e4b87:::benchmark_data_smithi177_31116_object76779:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e187)", 2024-09-13T11:27:01.035 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:25:59.883418+0000", 2024-09-13T11:27:01.035 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 61.074363183000003, 2024-09-13T11:27:01.035 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.25557793200000001, 2024-09-13T11:27:01.035 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:27:01.035 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:27:01.036 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:27:01.036 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:27:01.036 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:27:01.036 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:27:01.036 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 217082 2024-09-13T11:27:01.036 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.036 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:27:01.036 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.037 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:27:01.037 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:59.883418+0000", 2024-09-13T11:27:01.037 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:27:01.037 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.037 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.037 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:27:01.037 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:59.883418+0000", 2024-09-13T11:27:01.038 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:27:01.038 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.038 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.038 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:27:01.038 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:59.883419+0000", 2024-09-13T11:27:01.038 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.235e-06 2024-09-13T11:27:01.038 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.039 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.039 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:27:01.039 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:59.883423+0000", 2024-09-13T11:27:01.039 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.7280000000000001e-06 2024-09-13T11:27:01.039 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.039 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.039 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:27:01.039 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:59.883433+0000", 2024-09-13T11:27:01.040 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0498e-05 2024-09-13T11:27:01.040 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.040 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.040 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:27:01.040 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:59.883439+0000", 2024-09-13T11:27:01.040 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.5890000000000002e-06 2024-09-13T11:27:01.041 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.041 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.041 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:27:01.041 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:59.883483+0000", 2024-09-13T11:27:01.041 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.4710999999999999e-05 2024-09-13T11:27:01.041 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.042 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.042 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:27:01.042 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:59.883558+0000", 2024-09-13T11:27:01.042 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.4308999999999996e-05 2024-09-13T11:27:01.042 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.042 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.043 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 5", 2024-09-13T11:27:01.043 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:59.883615+0000", 2024-09-13T11:27:01.043 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.7745999999999999e-05 2024-09-13T11:27:01.043 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.043 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.043 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:27:01.044 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:59.885044+0000", 2024-09-13T11:27:01.044 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0014282940000000001 2024-09-13T11:27:01.044 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.044 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.044 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:27:01.044 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:00.138864+0000", 2024-09-13T11:27:01.045 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.253819923 2024-09-13T11:27:01.045 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.045 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.045 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:27:01.045 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:00.138936+0000", 2024-09-13T11:27:01.045 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.2268999999999995e-05 2024-09-13T11:27:01.046 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.046 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.046 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:27:01.046 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:00.138996+0000", 2024-09-13T11:27:01.046 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.9630000000000003e-05 2024-09-13T11:27:01.046 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:27:01.046 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:27:01.047 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:27:01.047 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.047 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.047 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:221503 3.15 3:ab531595:::benchmark_data_smithi177_31116_object81200:head [delete] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e193)", 2024-09-13T11:27:01.047 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:26:06.058069+0000", 2024-09-13T11:27:01.047 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 54.899712125999997, 2024-09-13T11:27:01.047 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.24966487400000001, 2024-09-13T11:27:01.047 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:27:01.047 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:27:01.048 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:27:01.048 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:27:01.048 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:27:01.048 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:27:01.048 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 221503 2024-09-13T11:27:01.048 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.048 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:27:01.049 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.049 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:27:01.049 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:06.058069+0000", 2024-09-13T11:27:01.049 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:27:01.049 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.049 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.049 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:27:01.049 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:06.058069+0000", 2024-09-13T11:27:01.049 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:27:01.050 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.050 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.050 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:27:01.050 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:06.058070+0000", 2024-09-13T11:27:01.050 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.7000000000000003e-07 2024-09-13T11:27:01.051 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.051 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.051 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:27:01.051 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:06.058074+0000", 2024-09-13T11:27:01.051 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.5070000000000002e-06 2024-09-13T11:27:01.051 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.052 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.052 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:27:01.052 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:06.058089+0000", 2024-09-13T11:27:01.052 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.4593000000000001e-05 2024-09-13T11:27:01.052 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.053 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.053 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:27:01.053 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:06.058102+0000", 2024-09-13T11:27:01.053 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.3349999999999999e-05 2024-09-13T11:27:01.053 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.053 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.054 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:27:01.054 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:06.058550+0000", 2024-09-13T11:27:01.054 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00044800299999999999 2024-09-13T11:27:01.054 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.054 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.054 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for peered", 2024-09-13T11:27:01.054 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:06.058560+0000", 2024-09-13T11:27:01.054 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.5200000000000003e-06 2024-09-13T11:27:01.055 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.055 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.055 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:27:01.055 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:06.286141+0000", 2024-09-13T11:27:01.055 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.22758104200000001 2024-09-13T11:27:01.055 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.055 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.055 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:27:01.056 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:06.286335+0000", 2024-09-13T11:27:01.056 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00019429700000000001 2024-09-13T11:27:01.056 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.056 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.056 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 3,14", 2024-09-13T11:27:01.057 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:06.286518+0000", 2024-09-13T11:27:01.057 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00018350500000000001 2024-09-13T11:27:01.057 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.057 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.057 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:27:01.057 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:06.296043+0000", 2024-09-13T11:27:01.057 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0095243600000000008 2024-09-13T11:27:01.057 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.058 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.058 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:27:01.058 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:06.296384+0000", 2024-09-13T11:27:01.058 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00034132100000000001 2024-09-13T11:27:01.058 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.058 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.058 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:27:01.058 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:06.307532+0000", 2024-09-13T11:27:01.058 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.011148025000000001 2024-09-13T11:27:01.059 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.059 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.059 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:27:01.059 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:06.307611+0000", 2024-09-13T11:27:01.059 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.8465000000000002e-05 2024-09-13T11:27:01.059 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.059 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.060 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:27:01.060 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:06.307734+0000", 2024-09-13T11:27:01.060 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00012301599999999999 2024-09-13T11:27:01.060 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:27:01.060 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:27:01.061 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:27:01.061 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.061 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.061 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:221528 3.15 3:afda25ed:::benchmark_data_smithi177_31116_object81225:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e193)", 2024-09-13T11:27:01.061 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:26:06.058152+0000", 2024-09-13T11:27:01.061 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 54.899629269000002, 2024-09-13T11:27:01.061 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.25716789899999998, 2024-09-13T11:27:01.062 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:27:01.062 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:27:01.062 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:27:01.062 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:27:01.062 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:27:01.062 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:27:01.062 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 221528 2024-09-13T11:27:01.062 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.062 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:27:01.062 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.063 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:27:01.063 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:06.058152+0000", 2024-09-13T11:27:01.063 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:27:01.063 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.063 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.063 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:27:01.063 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:06.058152+0000", 2024-09-13T11:27:01.063 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:27:01.063 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.063 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.063 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:27:01.063 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:06.058153+0000", 2024-09-13T11:27:01.063 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.773e-06 2024-09-13T11:27:01.064 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.064 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.064 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:27:01.064 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:06.058161+0000", 2024-09-13T11:27:01.064 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.9160000000000001e-06 2024-09-13T11:27:01.064 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.064 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.064 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:27:01.065 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:06.058186+0000", 2024-09-13T11:27:01.065 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.5037000000000001e-05 2024-09-13T11:27:01.065 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.065 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.065 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:27:01.065 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:06.058202+0000", 2024-09-13T11:27:01.066 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.5709000000000002e-05 2024-09-13T11:27:01.066 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.066 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.066 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:27:01.066 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:06.058640+0000", 2024-09-13T11:27:01.066 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00043771400000000002 2024-09-13T11:27:01.066 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.066 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.066 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for peered", 2024-09-13T11:27:01.066 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:06.058664+0000", 2024-09-13T11:27:01.066 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.385e-05 2024-09-13T11:27:01.067 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.067 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.067 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:27:01.067 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:06.291953+0000", 2024-09-13T11:27:01.067 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.23328986800000001 2024-09-13T11:27:01.067 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.067 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.067 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:27:01.067 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:06.292123+0000", 2024-09-13T11:27:01.067 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00016977600000000001 2024-09-13T11:27:01.067 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.067 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.067 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 3,14", 2024-09-13T11:27:01.067 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:06.292276+0000", 2024-09-13T11:27:01.068 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000153196 2024-09-13T11:27:01.068 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.068 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.068 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:27:01.068 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:06.296387+0000", 2024-09-13T11:27:01.068 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0041105459999999996 2024-09-13T11:27:01.068 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.068 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.068 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:27:01.068 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:06.307779+0000", 2024-09-13T11:27:01.068 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.011391712 2024-09-13T11:27:01.069 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.069 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.069 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:27:01.069 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:06.315151+0000", 2024-09-13T11:27:01.069 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0073723169999999998 2024-09-13T11:27:01.069 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.069 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.069 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:27:01.069 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:06.315219+0000", 2024-09-13T11:27:01.069 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.7984999999999994e-05 2024-09-13T11:27:01.069 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.069 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.069 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:27:01.069 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:06.315319+0000", 2024-09-13T11:27:01.070 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0001005 2024-09-13T11:27:01.070 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:27:01.070 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:27:01.070 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:27:01.070 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.070 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.070 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:243211 3.22 3:445e4152:::benchmark_data_smithi177_31116_object102908:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e207)", 2024-09-13T11:27:01.070 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:26:26.318899+0000", 2024-09-13T11:27:01.070 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 34.638882199000001, 2024-09-13T11:27:01.070 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2078296390000001, 2024-09-13T11:27:01.070 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:27:01.070 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:27:01.070 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-13T11:27:01.070 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:27:01.071 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:27:01.071 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:27:01.071 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 243211 2024-09-13T11:27:01.071 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.071 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:27:01.071 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.071 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:27:01.071 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:26.318899+0000", 2024-09-13T11:27:01.071 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:27:01.071 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.071 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.071 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:27:01.071 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:26.318899+0000", 2024-09-13T11:27:01.072 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:27:01.072 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.072 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.072 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:27:01.072 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:26.318899+0000", 2024-09-13T11:27:01.072 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.7199999999999999e-07 2024-09-13T11:27:01.072 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.072 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.072 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:27:01.072 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:26.318902+0000", 2024-09-13T11:27:01.072 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.616e-06 2024-09-13T11:27:01.072 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.072 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.072 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:27:01.073 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:26.318906+0000", 2024-09-13T11:27:01.073 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.2440000000000004e-06 2024-09-13T11:27:01.073 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.073 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.073 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:27:01.073 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:26.318909+0000", 2024-09-13T11:27:01.073 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.9830000000000001e-06 2024-09-13T11:27:01.073 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.073 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.073 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:27:01.073 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:26.319101+0000", 2024-09-13T11:27:01.073 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00019198800000000001 2024-09-13T11:27:01.073 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.073 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.074 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for missing object", 2024-09-13T11:27:01.074 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:26.319166+0000", 2024-09-13T11:27:01.074 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.4906999999999994e-05 2024-09-13T11:27:01.074 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.074 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.074 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:27:01.074 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:27.526716+0000", 2024-09-13T11:27:01.074 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.207550323 2024-09-13T11:27:01.074 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.074 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.074 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:27:01.074 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:27.526728+0000", 2024-09-13T11:27:01.074 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2006e-05 2024-09-13T11:27:01.075 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:27:01.075 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:27:01.075 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:27:01.075 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.075 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.075 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDPGRecoveryDeleteReply(3.22 e223,218 3:47eb4e8b:::benchmark_data_smithi177_31116_object75722:head,187'5647)", 2024-09-13T11:27:01.075 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:26:48.565998+0000", 2024-09-13T11:27:01.075 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 12.391783249, 2024-09-13T11:27:01.075 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.42944626299999999, 2024-09-13T11:27:01.075 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:27:01.075 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:27:01.076 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-13T11:27:01.076 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:27:01.076 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.076 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:27:01.076 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:48.565998+0000", 2024-09-13T11:27:01.076 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:27:01.076 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.076 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.076 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:27:01.076 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:48.565998+0000", 2024-09-13T11:27:01.076 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:27:01.076 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.077 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.077 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:27:01.077 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:48.566001+0000", 2024-09-13T11:27:01.077 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.0199999999999999e-06 2024-09-13T11:27:01.077 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.077 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.077 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:27:01.077 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:48.566011+0000", 2024-09-13T11:27:01.077 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0817e-05 2024-09-13T11:27:01.077 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.077 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.077 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:27:01.077 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:48.994167+0000", 2024-09-13T11:27:01.078 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.428155548 2024-09-13T11:27:01.078 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.078 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.078 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:27:01.078 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:48.994178+0000", 2024-09-13T11:27:01.078 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.1321999999999999e-05 2024-09-13T11:27:01.078 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.078 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.078 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:27:01.078 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:48.994497+0000", 2024-09-13T11:27:01.078 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00031834000000000002 2024-09-13T11:27:01.078 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.078 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.078 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:27:01.079 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:48.995444+0000", 2024-09-13T11:27:01.079 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00094721599999999997 2024-09-13T11:27:01.079 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:27:01.079 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:27:01.079 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:27:01.079 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.079 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.079 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4665.0:266847 3.15 e223/216 3:aced3906:::benchmark_data_smithi177_31116_object126544:head v 223'9499, mlcod=223'9497)", 2024-09-13T11:27:01.079 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:26:48.588378+0000", 2024-09-13T11:27:01.079 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 12.369403054999999, 2024-09-13T11:27:01.079 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.29172203699999999, 2024-09-13T11:27:01.079 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:27:01.079 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:27:01.079 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:27:01.080 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:27:01.080 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.080 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:27:01.080 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:48.588378+0000", 2024-09-13T11:27:01.080 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:27:01.080 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.080 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.080 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:27:01.080 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:48.588378+0000", 2024-09-13T11:27:01.080 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:27:01.080 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.080 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.080 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:27:01.081 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:48.588381+0000", 2024-09-13T11:27:01.081 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.2629999999999999e-06 2024-09-13T11:27:01.081 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.081 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.081 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:27:01.081 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:48.588395+0000", 2024-09-13T11:27:01.081 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.401e-05 2024-09-13T11:27:01.081 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.081 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.081 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:27:01.081 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:48.873812+0000", 2024-09-13T11:27:01.081 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.28541680699999999 2024-09-13T11:27:01.081 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.082 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.082 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:27:01.082 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:48.873824+0000", 2024-09-13T11:27:01.082 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2143e-05 2024-09-13T11:27:01.082 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.082 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.082 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:27:01.082 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:48.873981+0000", 2024-09-13T11:27:01.082 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000156768 2024-09-13T11:27:01.082 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.082 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.082 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:27:01.082 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:48.874019+0000", 2024-09-13T11:27:01.082 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.8050000000000003e-05 2024-09-13T11:27:01.083 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.083 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.083 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:27:01.083 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:48.879999+0000", 2024-09-13T11:27:01.083 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0059797139999999997 2024-09-13T11:27:01.083 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.083 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.083 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:27:01.083 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:48.880100+0000", 2024-09-13T11:27:01.083 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000101282 2024-09-13T11:27:01.083 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:27:01.083 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:27:01.083 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:27:01.083 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.084 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.084 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:268442 3.2e 3:76ee2334:::benchmark_data_smithi177_31116_object128139:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e228)", 2024-09-13T11:27:01.084 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:26:54.320795+0000", 2024-09-13T11:27:01.084 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 6.6369861849999996, 2024-09-13T11:27:01.084 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.25309999799999999, 2024-09-13T11:27:01.084 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:27:01.084 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:27:01.084 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:27:01.084 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:27:01.084 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:27:01.084 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:27:01.084 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 268442 2024-09-13T11:27:01.084 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.085 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:27:01.085 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.085 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:27:01.085 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:54.320795+0000", 2024-09-13T11:27:01.085 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:27:01.085 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.085 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.085 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:27:01.085 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:54.320795+0000", 2024-09-13T11:27:01.085 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:27:01.085 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.086 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.086 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:27:01.086 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:54.320797+0000", 2024-09-13T11:27:01.086 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.328e-06 2024-09-13T11:27:01.086 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.086 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.086 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:27:01.086 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:54.320802+0000", 2024-09-13T11:27:01.086 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.062e-06 2024-09-13T11:27:01.086 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.086 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.086 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:27:01.086 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:54.320814+0000", 2024-09-13T11:27:01.086 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.1980999999999999e-05 2024-09-13T11:27:01.086 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.087 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.087 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:27:01.087 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:54.320823+0000", 2024-09-13T11:27:01.087 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.5699999999999993e-06 2024-09-13T11:27:01.087 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.087 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.087 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:27:01.087 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:54.320869+0000", 2024-09-13T11:27:01.087 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.6675999999999998e-05 2024-09-13T11:27:01.087 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.087 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.087 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for degraded object", 2024-09-13T11:27:01.087 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:54.321417+0000", 2024-09-13T11:27:01.087 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00054810099999999995 2024-09-13T11:27:01.088 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.088 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.088 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:27:01.088 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:54.328578+0000", 2024-09-13T11:27:01.088 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0071610349999999996 2024-09-13T11:27:01.088 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.088 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.088 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:27:01.088 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:54.328637+0000", 2024-09-13T11:27:01.088 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.8196000000000003e-05 2024-09-13T11:27:01.088 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.088 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.088 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 9,11", 2024-09-13T11:27:01.089 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:54.328902+0000", 2024-09-13T11:27:01.089 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00026555699999999998 2024-09-13T11:27:01.089 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.089 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.089 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:27:01.089 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:54.330757+0000", 2024-09-13T11:27:01.089 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.001854444 2024-09-13T11:27:01.089 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.089 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.089 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:27:01.089 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:54.337932+0000", 2024-09-13T11:27:01.089 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0071758079999999997 2024-09-13T11:27:01.089 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.089 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.090 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:27:01.090 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:54.573543+0000", 2024-09-13T11:27:01.090 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.235610983 2024-09-13T11:27:01.090 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.090 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.090 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:27:01.090 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:54.573612+0000", 2024-09-13T11:27:01.090 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.8973999999999994e-05 2024-09-13T11:27:01.090 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.090 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.090 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:27:01.090 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:54.573895+0000", 2024-09-13T11:27:01.090 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00028228299999999998 2024-09-13T11:27:01.090 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:27:01.091 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:27:01.091 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:27:01.091 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.091 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.091 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDPGPushReply(3.2e 231/227 [PushReplyOp(3:76e0d842:::benchmark_data_smithi177_31116_object136425:head)])", 2024-09-13T11:27:01.091 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:26:59.123267+0000", 2024-09-13T11:27:01.091 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 1.8345138009999999, 2024-09-13T11:27:01.091 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.307351559, 2024-09-13T11:27:01.091 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:27:01.091 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:27:01.091 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-13T11:27:01.091 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:27:01.091 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.091 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:27:01.092 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:59.123267+0000", 2024-09-13T11:27:01.092 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:27:01.092 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.092 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.092 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:27:01.092 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:59.123267+0000", 2024-09-13T11:27:01.092 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:27:01.092 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.092 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.092 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:27:01.092 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:59.123268+0000", 2024-09-13T11:27:01.092 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.1200000000000001e-06 2024-09-13T11:27:01.092 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.092 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.093 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:27:01.093 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:59.123273+0000", 2024-09-13T11:27:01.093 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.5630000000000002e-06 2024-09-13T11:27:01.093 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.093 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.093 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:27:01.093 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:59.429858+0000", 2024-09-13T11:27:01.093 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.30658561099999998 2024-09-13T11:27:01.093 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.093 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.093 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:27:01.093 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:59.429867+0000", 2024-09-13T11:27:01.093 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.1939999999999998e-06 2024-09-13T11:27:01.094 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.094 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.094 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:27:01.094 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:59.429940+0000", 2024-09-13T11:27:01.094 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.3848000000000002e-05 2024-09-13T11:27:01.094 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.094 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.094 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:27:01.094 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:59.430619+0000", 2024-09-13T11:27:01.094 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00067822300000000004 2024-09-13T11:27:01.094 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:27:01.094 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:27:01.094 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:27:01.094 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:27:01.095 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:27:01.095 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-13T11:27:01.096 INFO:tasks.ceph.osd.11.smithi177.stderr:2024-09-13T11:27:01.093+0000 7f24792c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26580) UID: 0 2024-09-13T11:27:01.096 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-13T11:27:01.197 INFO:tasks.ceph.osd.15.smithi177.stderr:2024-09-13T11:27:01.193+0000 7f9d6ce77640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26578) UID: 0 2024-09-13T11:27:01.299 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-09-13T11:27:01.295+0000 7f97befc9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26315) UID: 0 2024-09-13T11:27:01.308 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-13T11:27:01.308 INFO:teuthology.orchestra.run.smithi001.stdout: "ops": [], 2024-09-13T11:27:01.308 INFO:teuthology.orchestra.run.smithi001.stdout: "num_ops": 0 2024-09-13T11:27:01.308 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-13T11:27:01.327 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-13T11:27:01.398 INFO:tasks.ceph.osd.9.smithi092.stderr:2024-09-13T11:27:01.395+0000 7ff5ab3d0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26348) UID: 0 2024-09-13T11:27:01.433 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-13T11:27:01.439 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-13T11:27:01.426+0000 7f6a5f1a6640 -1 osd.0 233 handle_osd_map injecting map crc failure 2024-09-13T11:27:01.439 INFO:teuthology.orchestra.run.smithi001.stderr:set pool 3 pg_num to 66 2024-09-13T11:27:01.489 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-13T11:27:01.489 INFO:teuthology.orchestra.run.smithi001.stdout: "ops": [], 2024-09-13T11:27:01.489 INFO:teuthology.orchestra.run.smithi001.stdout: "complaint_time": 30, 2024-09-13T11:27:01.489 INFO:teuthology.orchestra.run.smithi001.stdout: "num_blocked_ops": 0 2024-09-13T11:27:01.490 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-13T11:27:01.499 INFO:tasks.ceph.osd.12.smithi001.stderr:2024-09-13T11:27:01.494+0000 7f892c6e6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26314) UID: 0 2024-09-13T11:27:01.514 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-13T11:27:01.593 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T11:27:01.600 INFO:tasks.ceph.osd.13.smithi092.stderr:2024-09-13T11:27:01.595+0000 7f3e5afea640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26350) UID: 0 2024-09-13T11:27:01.701 INFO:tasks.ceph.osd.2.smithi162.stderr:2024-09-13T11:27:01.699+0000 7fe92b35a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26313) UID: 0 2024-09-13T11:27:01.769 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-13T11:27:01.770 INFO:teuthology.orchestra.run.smithi001.stdout: "size": 20, 2024-09-13T11:27:01.770 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 600, 2024-09-13T11:27:01.770 INFO:teuthology.orchestra.run.smithi001.stdout: "ops": [ 2024-09-13T11:27:01.770 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.770 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDPGPush(3.1a 102/78 [PushOp(3:5df50766:::benchmark_data_smithi177_31116_object50543:head, version: 72'3152, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:5df50766:::benchmark_data_smithi177_31116_object50543:head@72'3152, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:27:01.770 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:24:00.499311+0000", 2024-09-13T11:27:01.770 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 181.26367183299999, 2024-09-13T11:27:01.770 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.26146664800000002, 2024-09-13T11:27:01.770 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:27:01.770 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:27:01.770 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:27:01.770 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:27:01.771 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.771 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:27:01.771 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:00.499311+0000", 2024-09-13T11:27:01.771 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:27:01.771 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.771 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.771 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:27:01.771 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:00.499311+0000", 2024-09-13T11:27:01.771 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:27:01.771 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.771 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.771 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:27:01.771 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:00.499320+0000", 2024-09-13T11:27:01.772 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.6400000000000003e-06 2024-09-13T11:27:01.772 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.772 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.772 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:27:01.772 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:00.499384+0000", 2024-09-13T11:27:01.772 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.4554999999999994e-05 2024-09-13T11:27:01.772 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.772 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.772 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:27:01.772 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:00.759713+0000", 2024-09-13T11:27:01.772 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.260328742 2024-09-13T11:27:01.772 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.772 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.772 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:27:01.773 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:00.759742+0000", 2024-09-13T11:27:01.773 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.9298e-05 2024-09-13T11:27:01.773 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.773 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.773 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:27:01.773 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:00.759871+0000", 2024-09-13T11:27:01.773 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000128992 2024-09-13T11:27:01.773 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.773 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.773 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:27:01.773 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:00.759884+0000", 2024-09-13T11:27:01.773 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2687999999999999e-05 2024-09-13T11:27:01.773 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.773 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.774 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:27:01.774 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:00.760778+0000", 2024-09-13T11:27:01.774 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00089373299999999996 2024-09-13T11:27:01.774 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:27:01.774 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:27:01.774 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:27:01.774 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.774 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.774 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4665.0:82769 3.1e e105/102 3:7a4e88b5:::benchmark_data_smithi177_31116_object82768:head v 105'4381, mlcod=105'4379)", 2024-09-13T11:27:01.774 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:24:03.745047+0000", 2024-09-13T11:27:01.774 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 178.01793560799999, 2024-09-13T11:27:01.774 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.213806145, 2024-09-13T11:27:01.774 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:27:01.774 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:27:01.775 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:27:01.775 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:27:01.775 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.775 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:27:01.775 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:03.745047+0000", 2024-09-13T11:27:01.775 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:27:01.775 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.775 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.775 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:27:01.775 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:03.745047+0000", 2024-09-13T11:27:01.775 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:27:01.775 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.775 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.776 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:27:01.776 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:03.745049+0000", 2024-09-13T11:27:01.776 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.5710000000000001e-06 2024-09-13T11:27:01.776 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.776 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.776 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:27:01.776 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:03.745107+0000", 2024-09-13T11:27:01.776 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.8799999999999999e-05 2024-09-13T11:27:01.776 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.776 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.776 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:27:01.776 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:03.955145+0000", 2024-09-13T11:27:01.776 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.210037735 2024-09-13T11:27:01.777 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.777 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.777 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:27:01.777 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:03.955163+0000", 2024-09-13T11:27:01.777 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.7520000000000002e-05 2024-09-13T11:27:01.777 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.777 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.777 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:27:01.777 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:03.955473+0000", 2024-09-13T11:27:01.777 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00031014399999999999 2024-09-13T11:27:01.777 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.777 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.777 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:27:01.777 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:03.955538+0000", 2024-09-13T11:27:01.778 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.5059000000000002e-05 2024-09-13T11:27:01.778 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.778 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.778 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:27:01.778 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:03.958753+0000", 2024-09-13T11:27:01.778 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0032152769999999999 2024-09-13T11:27:01.778 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.778 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.778 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:27:01.778 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:03.958853+0000", 2024-09-13T11:27:01.778 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000100039 2024-09-13T11:27:01.778 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:27:01.778 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:27:01.779 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:27:01.779 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.779 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.779 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:98582 3.1a 3:59f9cd75:::benchmark_data_smithi177_31116_object98581:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113)", 2024-09-13T11:27:01.779 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:24:15.431226+0000", 2024-09-13T11:27:01.779 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 166.33175713899999, 2024-09-13T11:27:01.779 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.19820755800000001, 2024-09-13T11:27:01.779 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:27:01.779 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:27:01.779 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:27:01.779 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:27:01.780 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:27:01.780 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:27:01.780 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 98582 2024-09-13T11:27:01.780 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.780 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:27:01.780 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.780 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:27:01.780 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.431226+0000", 2024-09-13T11:27:01.780 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:27:01.780 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.780 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.780 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:27:01.780 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.431226+0000", 2024-09-13T11:27:01.781 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:27:01.781 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.781 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.781 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:27:01.781 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.431229+0000", 2024-09-13T11:27:01.781 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.7100000000000001e-06 2024-09-13T11:27:01.781 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.781 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.781 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:27:01.781 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.431334+0000", 2024-09-13T11:27:01.781 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00010506900000000001 2024-09-13T11:27:01.781 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.781 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.781 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:27:01.782 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.431364+0000", 2024-09-13T11:27:01.782 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.9607e-05 2024-09-13T11:27:01.782 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.782 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.782 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:27:01.782 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.431388+0000", 2024-09-13T11:27:01.782 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.3808999999999999e-05 2024-09-13T11:27:01.782 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.782 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.782 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:27:01.782 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.431483+0000", 2024-09-13T11:27:01.782 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.5402999999999994e-05 2024-09-13T11:27:01.782 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.783 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.783 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:27:01.783 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.431642+0000", 2024-09-13T11:27:01.783 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000158412 2024-09-13T11:27:01.783 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.783 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.783 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 9", 2024-09-13T11:27:01.783 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.431796+0000", 2024-09-13T11:27:01.783 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00015467899999999999 2024-09-13T11:27:01.783 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.783 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.783 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:27:01.783 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.433731+0000", 2024-09-13T11:27:01.783 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.001934945 2024-09-13T11:27:01.784 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.784 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.784 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:27:01.784 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.629245+0000", 2024-09-13T11:27:01.784 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.19551390199999999 2024-09-13T11:27:01.784 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.784 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.784 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:27:01.784 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.629346+0000", 2024-09-13T11:27:01.784 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000100869 2024-09-13T11:27:01.784 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.784 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.784 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:27:01.785 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.629433+0000", 2024-09-13T11:27:01.785 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.7152999999999997e-05 2024-09-13T11:27:01.785 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:27:01.785 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:27:01.785 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:27:01.785 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.785 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.785 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:99105 3.1a 3:5ac5090c:::benchmark_data_smithi177_31116_object99104:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113)", 2024-09-13T11:27:01.785 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:24:15.962466+0000", 2024-09-13T11:27:01.785 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 165.80051685000001, 2024-09-13T11:27:01.785 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.23832849, 2024-09-13T11:27:01.785 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:27:01.785 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:27:01.786 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:27:01.786 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:27:01.786 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:27:01.786 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:27:01.786 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 99105 2024-09-13T11:27:01.786 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.786 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:27:01.786 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.786 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:27:01.786 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.962466+0000", 2024-09-13T11:27:01.786 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:27:01.786 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.786 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.787 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:27:01.787 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.962466+0000", 2024-09-13T11:27:01.787 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:27:01.787 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.787 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.787 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:27:01.787 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.962471+0000", 2024-09-13T11:27:01.787 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.9570000000000001e-06 2024-09-13T11:27:01.787 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.787 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.787 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:27:01.787 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.962547+0000", 2024-09-13T11:27:01.787 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.5947999999999999e-05 2024-09-13T11:27:01.787 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.788 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.788 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:27:01.788 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.962576+0000", 2024-09-13T11:27:01.788 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.9547999999999999e-05 2024-09-13T11:27:01.788 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.788 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.788 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:27:01.788 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.962594+0000", 2024-09-13T11:27:01.788 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.8023e-05 2024-09-13T11:27:01.788 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.788 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.788 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:27:01.788 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.962770+0000", 2024-09-13T11:27:01.788 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00017551099999999999 2024-09-13T11:27:01.789 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.789 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.789 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:27:01.789 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.963114+0000", 2024-09-13T11:27:01.789 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00034440399999999998 2024-09-13T11:27:01.789 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.789 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.789 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 9", 2024-09-13T11:27:01.789 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.963351+0000", 2024-09-13T11:27:01.789 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000237209 2024-09-13T11:27:01.789 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.789 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.789 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:27:01.789 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.966345+0000", 2024-09-13T11:27:01.790 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0029933519999999999 2024-09-13T11:27:01.790 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.790 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.790 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:27:01.790 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:16.200724+0000", 2024-09-13T11:27:01.790 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.23437934599999999 2024-09-13T11:27:01.790 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.790 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.790 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:27:01.790 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:16.200761+0000", 2024-09-13T11:27:01.790 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.6406999999999999e-05 2024-09-13T11:27:01.790 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.790 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.791 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:27:01.791 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:16.200794+0000", 2024-09-13T11:27:01.791 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.3785000000000003e-05 2024-09-13T11:27:01.791 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:27:01.791 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:27:01.791 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:27:01.791 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.791 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.791 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:129921 3.1a 3:5fb9088b:::benchmark_data_smithi177_31116_object129920:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e134)", 2024-09-13T11:27:01.791 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:24:47.177556+0000", 2024-09-13T11:27:01.791 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 134.585426571, 2024-09-13T11:27:01.791 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.33804469999999998, 2024-09-13T11:27:01.791 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:27:01.791 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:27:01.792 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:27:01.792 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:27:01.792 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:27:01.792 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:27:01.792 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 129921 2024-09-13T11:27:01.792 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.792 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:27:01.792 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.792 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:27:01.792 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:47.177556+0000", 2024-09-13T11:27:01.792 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:27:01.792 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.792 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.793 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:27:01.793 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:47.177556+0000", 2024-09-13T11:27:01.793 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:27:01.793 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.793 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.793 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:27:01.793 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:47.177559+0000", 2024-09-13T11:27:01.793 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.0790000000000001e-06 2024-09-13T11:27:01.793 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.793 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.793 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:27:01.793 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:47.177589+0000", 2024-09-13T11:27:01.793 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.0173999999999999e-05 2024-09-13T11:27:01.793 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.794 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.794 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:27:01.794 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:47.177606+0000", 2024-09-13T11:27:01.794 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.6475999999999999e-05 2024-09-13T11:27:01.794 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.794 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.794 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:27:01.794 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:47.177615+0000", 2024-09-13T11:27:01.794 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.5729999999999997e-06 2024-09-13T11:27:01.794 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.794 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.794 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:27:01.794 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:47.177680+0000", 2024-09-13T11:27:01.795 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.4349000000000003e-05 2024-09-13T11:27:01.795 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.795 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.795 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:27:01.795 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:47.177794+0000", 2024-09-13T11:27:01.795 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000114119 2024-09-13T11:27:01.795 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.795 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.795 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 9", 2024-09-13T11:27:01.795 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:47.177895+0000", 2024-09-13T11:27:01.795 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000101078 2024-09-13T11:27:01.795 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.795 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.796 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:27:01.796 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:47.179634+0000", 2024-09-13T11:27:01.796 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.001738606 2024-09-13T11:27:01.796 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.796 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.796 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:27:01.796 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:47.515495+0000", 2024-09-13T11:27:01.796 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.33586143000000002 2024-09-13T11:27:01.796 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.796 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.796 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:27:01.796 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:47.515551+0000", 2024-09-13T11:27:01.796 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.6048999999999998e-05 2024-09-13T11:27:01.797 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.797 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.797 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:27:01.797 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:47.515601+0000", 2024-09-13T11:27:01.797 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.9767e-05 2024-09-13T11:27:01.797 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:27:01.797 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:27:01.797 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:27:01.797 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.797 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.797 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4665.0:140742 3.1e e144/102 3:79c694a0:::benchmark_data_smithi177_31116_object439:head v 144'6204, mlcod=144'6200)", 2024-09-13T11:27:01.797 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:24:58.321534+0000", 2024-09-13T11:27:01.797 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 123.44144904700001, 2024-09-13T11:27:01.798 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.23629921500000001, 2024-09-13T11:27:01.798 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:27:01.798 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:27:01.798 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:27:01.798 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:27:01.798 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.798 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:27:01.798 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.321534+0000", 2024-09-13T11:27:01.798 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:27:01.798 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.798 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.798 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:27:01.798 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.321534+0000", 2024-09-13T11:27:01.799 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:27:01.799 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.799 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.799 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:27:01.799 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.321535+0000", 2024-09-13T11:27:01.799 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.159e-06 2024-09-13T11:27:01.799 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.799 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.799 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:27:01.799 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.321545+0000", 2024-09-13T11:27:01.799 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.9219999999999992e-06 2024-09-13T11:27:01.799 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.799 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.800 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:27:01.800 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.553719+0000", 2024-09-13T11:27:01.800 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.23217401700000001 2024-09-13T11:27:01.800 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.800 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.800 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:27:01.800 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.553734+0000", 2024-09-13T11:27:01.800 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.5344000000000001e-05 2024-09-13T11:27:01.800 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.800 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.800 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:27:01.800 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.553858+0000", 2024-09-13T11:27:01.800 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00012357500000000001 2024-09-13T11:27:01.800 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.801 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.801 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:27:01.801 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.553903+0000", 2024-09-13T11:27:01.801 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.5374999999999998e-05 2024-09-13T11:27:01.801 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.801 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.801 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:27:01.801 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.557719+0000", 2024-09-13T11:27:01.801 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0038158670000000001 2024-09-13T11:27:01.801 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.801 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.801 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:27:01.801 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.557833+0000", 2024-09-13T11:27:01.802 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000113956 2024-09-13T11:27:01.802 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:27:01.802 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:27:01.802 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:27:01.802 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.802 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.802 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4665.0:140796 3.1e e144/102 3:7987a652:::benchmark_data_smithi177_31116_object493:head v 144'6205, mlcod=144'6202)", 2024-09-13T11:27:01.802 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:24:58.553657+0000", 2024-09-13T11:27:01.802 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 123.20932607100001, 2024-09-13T11:27:01.802 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.25308323100000002, 2024-09-13T11:27:01.802 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:27:01.802 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:27:01.802 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:27:01.802 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:27:01.803 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.803 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:27:01.803 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.553657+0000", 2024-09-13T11:27:01.803 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:27:01.803 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.803 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.803 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:27:01.803 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.553657+0000", 2024-09-13T11:27:01.803 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:27:01.803 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.803 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.803 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:27:01.803 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.553660+0000", 2024-09-13T11:27:01.804 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.7519999999999998e-06 2024-09-13T11:27:01.804 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.804 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.804 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:27:01.804 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.553678+0000", 2024-09-13T11:27:01.804 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.7156e-05 2024-09-13T11:27:01.804 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.804 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.804 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:27:01.804 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.804342+0000", 2024-09-13T11:27:01.804 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.25066436199999997 2024-09-13T11:27:01.804 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.804 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.805 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:27:01.805 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.804353+0000", 2024-09-13T11:27:01.805 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.1147e-05 2024-09-13T11:27:01.805 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.805 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.805 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:27:01.805 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.804433+0000", 2024-09-13T11:27:01.805 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.9586999999999995e-05 2024-09-13T11:27:01.805 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.805 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.805 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:27:01.805 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.804457+0000", 2024-09-13T11:27:01.805 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.4108e-05 2024-09-13T11:27:01.806 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.806 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.806 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:27:01.806 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.806684+0000", 2024-09-13T11:27:01.806 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0022271309999999998 2024-09-13T11:27:01.806 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.806 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.806 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:27:01.806 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.806740+0000", 2024-09-13T11:27:01.806 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.5988e-05 2024-09-13T11:27:01.806 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:27:01.806 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:27:01.806 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:27:01.807 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.807 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.807 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:149955 3.11 3:8fa29838:::benchmark_data_smithi177_31116_object9652:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e149)", 2024-09-13T11:27:01.807 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:25:05.905492+0000", 2024-09-13T11:27:01.807 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 115.857490697, 2024-09-13T11:27:01.807 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.204181955, 2024-09-13T11:27:01.807 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:27:01.807 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:27:01.807 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:27:01.807 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:27:01.807 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:27:01.807 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:27:01.807 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 149955 2024-09-13T11:27:01.807 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.808 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:27:01.808 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.808 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:27:01.808 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:05.905492+0000", 2024-09-13T11:27:01.808 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:27:01.808 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.808 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.808 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:27:01.808 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:05.905492+0000", 2024-09-13T11:27:01.808 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:27:01.808 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.808 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.808 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:27:01.808 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:05.905494+0000", 2024-09-13T11:27:01.809 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.9369999999999998e-06 2024-09-13T11:27:01.809 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.809 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.809 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:27:01.809 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:05.905500+0000", 2024-09-13T11:27:01.809 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.7039999999999999e-06 2024-09-13T11:27:01.809 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.809 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.809 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:27:01.809 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:05.905512+0000", 2024-09-13T11:27:01.809 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.257e-05 2024-09-13T11:27:01.809 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.809 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.809 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:27:01.810 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:05.905519+0000", 2024-09-13T11:27:01.810 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.5420000000000002e-06 2024-09-13T11:27:01.810 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.810 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.810 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:27:01.810 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:05.905584+0000", 2024-09-13T11:27:01.810 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.4819000000000004e-05 2024-09-13T11:27:01.810 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.810 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.810 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:27:01.810 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:05.905696+0000", 2024-09-13T11:27:01.810 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00011248599999999999 2024-09-13T11:27:01.811 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.811 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.811 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 11", 2024-09-13T11:27:01.811 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:05.905759+0000", 2024-09-13T11:27:01.811 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.2451000000000002e-05 2024-09-13T11:27:01.811 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.811 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.811 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:27:01.811 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:05.907584+0000", 2024-09-13T11:27:01.811 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0018250110000000001 2024-09-13T11:27:01.811 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.811 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.811 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:27:01.812 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:06.109588+0000", 2024-09-13T11:27:01.812 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.20200470300000001 2024-09-13T11:27:01.812 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.812 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.812 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:27:01.812 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:06.109637+0000", 2024-09-13T11:27:01.812 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.8439000000000002e-05 2024-09-13T11:27:01.812 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.812 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.812 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:27:01.812 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:06.109674+0000", 2024-09-13T11:27:01.812 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.7292999999999997e-05 2024-09-13T11:27:01.813 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:27:01.813 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:27:01.813 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:27:01.813 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.813 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.813 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4665.0:191081 3.1e e175/102 3:7e784823:::benchmark_data_smithi177_31116_object50778:head v 175'7812, mlcod=175'7809)", 2024-09-13T11:27:01.813 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:25:38.547539+0000", 2024-09-13T11:27:01.813 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 83.215443446999998, 2024-09-13T11:27:01.813 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.213922061, 2024-09-13T11:27:01.813 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:27:01.813 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:27:01.813 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:27:01.813 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:27:01.814 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.814 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:27:01.814 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:38.547539+0000", 2024-09-13T11:27:01.814 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:27:01.814 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.814 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.814 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:27:01.814 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:38.547539+0000", 2024-09-13T11:27:01.814 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:27:01.814 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.814 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.814 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:27:01.814 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:38.547543+0000", 2024-09-13T11:27:01.815 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.3629999999999998e-06 2024-09-13T11:27:01.815 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.815 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.815 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:27:01.815 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:38.547558+0000", 2024-09-13T11:27:01.815 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.5043e-05 2024-09-13T11:27:01.815 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.815 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.815 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:27:01.815 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:38.756778+0000", 2024-09-13T11:27:01.815 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.209219936 2024-09-13T11:27:01.815 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.815 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.815 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:27:01.816 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:38.756790+0000", 2024-09-13T11:27:01.816 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2775999999999999e-05 2024-09-13T11:27:01.816 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.816 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.816 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:27:01.816 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:38.756934+0000", 2024-09-13T11:27:01.816 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00014401600000000001 2024-09-13T11:27:01.816 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.816 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.816 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:27:01.816 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:38.757001+0000", 2024-09-13T11:27:01.816 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.6716000000000003e-05 2024-09-13T11:27:01.816 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.816 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.817 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:27:01.817 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:38.761376+0000", 2024-09-13T11:27:01.817 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0043747320000000001 2024-09-13T11:27:01.817 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.817 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.817 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:27:01.817 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:38.761461+0000", 2024-09-13T11:27:01.817 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.5478999999999999e-05 2024-09-13T11:27:01.817 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:27:01.817 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:27:01.817 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:27:01.817 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.817 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.818 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4665.0:191087 3.1e e175/102 3:7a697202:::benchmark_data_smithi177_31116_object50784:head v 175'7813, mlcod=175'7809)", 2024-09-13T11:27:01.818 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:25:38.561827+0000", 2024-09-13T11:27:01.818 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 83.201155236999995, 2024-09-13T11:27:01.818 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.209395781, 2024-09-13T11:27:01.818 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:27:01.818 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:27:01.818 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:27:01.818 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:27:01.818 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.818 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:27:01.818 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:38.561827+0000", 2024-09-13T11:27:01.818 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:27:01.818 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.819 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.819 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:27:01.819 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:38.561827+0000", 2024-09-13T11:27:01.819 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:27:01.819 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.819 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.819 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:27:01.819 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:38.561837+0000", 2024-09-13T11:27:01.819 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.2329999999999995e-06 2024-09-13T11:27:01.819 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.819 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.819 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:27:01.819 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:38.561849+0000", 2024-09-13T11:27:01.820 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2593e-05 2024-09-13T11:27:01.820 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.820 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.820 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:27:01.820 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:38.766804+0000", 2024-09-13T11:27:01.820 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.20495488000000001 2024-09-13T11:27:01.820 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.820 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.820 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:27:01.820 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:38.766823+0000", 2024-09-13T11:27:01.820 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.9143e-05 2024-09-13T11:27:01.820 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.820 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.821 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:27:01.821 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:38.766973+0000", 2024-09-13T11:27:01.821 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000149226 2024-09-13T11:27:01.821 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.821 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.821 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:27:01.821 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:38.767014+0000", 2024-09-13T11:27:01.821 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.1183000000000002e-05 2024-09-13T11:27:01.821 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.821 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.821 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:27:01.821 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:38.771145+0000", 2024-09-13T11:27:01.821 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0041315270000000003 2024-09-13T11:27:01.821 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.822 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.822 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:27:01.822 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:38.771223+0000", 2024-09-13T11:27:01.822 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.7996000000000003e-05 2024-09-13T11:27:01.822 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:27:01.822 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:27:01.822 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:27:01.822 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.822 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.822 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:233802 3.11 3:8e27a54a:::benchmark_data_smithi177_31116_object93499:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e200)", 2024-09-13T11:27:01.822 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:26:16.812014+0000", 2024-09-13T11:27:01.822 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 44.950968969000002, 2024-09-13T11:27:01.822 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.34299280599999998, 2024-09-13T11:27:01.823 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:27:01.823 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:27:01.823 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:27:01.823 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:27:01.823 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:27:01.823 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:27:01.823 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 233802 2024-09-13T11:27:01.823 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.823 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:27:01.823 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.823 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:27:01.823 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:16.812014+0000", 2024-09-13T11:27:01.823 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:27:01.823 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.824 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.824 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:27:01.824 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:16.812014+0000", 2024-09-13T11:27:01.824 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:27:01.824 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.824 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.824 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:27:01.824 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:16.812015+0000", 2024-09-13T11:27:01.824 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.19e-06 2024-09-13T11:27:01.824 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.824 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.824 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:27:01.824 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:16.812019+0000", 2024-09-13T11:27:01.824 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.8720000000000004e-06 2024-09-13T11:27:01.825 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.825 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.825 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:27:01.825 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:16.812025+0000", 2024-09-13T11:27:01.825 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.3339999999999998e-06 2024-09-13T11:27:01.825 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.825 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.825 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:27:01.825 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:16.812029+0000", 2024-09-13T11:27:01.825 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.2989999999999998e-06 2024-09-13T11:27:01.825 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.825 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.825 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:27:01.826 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:16.812160+0000", 2024-09-13T11:27:01.826 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000130288 2024-09-13T11:27:01.826 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.826 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.826 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:27:01.826 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:16.812298+0000", 2024-09-13T11:27:01.826 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00013822500000000001 2024-09-13T11:27:01.826 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.826 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.826 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 11", 2024-09-13T11:27:01.826 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:16.812412+0000", 2024-09-13T11:27:01.826 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000113947 2024-09-13T11:27:01.826 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.827 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.827 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:27:01.827 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:16.815915+0000", 2024-09-13T11:27:01.827 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0035030259999999998 2024-09-13T11:27:01.827 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.827 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.827 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:27:01.827 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:17.154942+0000", 2024-09-13T11:27:01.827 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.33902705700000002 2024-09-13T11:27:01.827 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.827 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.827 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:27:01.827 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:17.154978+0000", 2024-09-13T11:27:01.828 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.5549999999999997e-05 2024-09-13T11:27:01.828 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.828 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.828 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:27:01.828 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:17.155007+0000", 2024-09-13T11:27:01.828 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.9017999999999999e-05 2024-09-13T11:27:01.828 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:27:01.828 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:27:01.828 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:27:01.828 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.828 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.828 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop_reply(client.4665.0:233802 3.11 e200/81 ondisk, result = 0)", 2024-09-13T11:27:01.828 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:26:16.815681+0000", 2024-09-13T11:27:01.829 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 44.947301469999999, 2024-09-13T11:27:01.829 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.33932907800000001, 2024-09-13T11:27:01.829 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:27:01.829 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:27:01.829 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:27:01.829 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:27:01.829 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.829 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:27:01.829 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:16.815681+0000", 2024-09-13T11:27:01.829 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:27:01.829 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.829 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.829 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:27:01.830 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:16.815681+0000", 2024-09-13T11:27:01.830 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:27:01.830 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.830 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.830 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:27:01.830 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:16.815682+0000", 2024-09-13T11:27:01.830 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.1230000000000001e-06 2024-09-13T11:27:01.830 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.830 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.830 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:27:01.830 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:16.815689+0000", 2024-09-13T11:27:01.830 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.5599999999999999e-06 2024-09-13T11:27:01.830 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.830 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.831 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:27:01.831 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:17.154882+0000", 2024-09-13T11:27:01.831 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.33919308599999998 2024-09-13T11:27:01.831 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.831 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.831 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:27:01.831 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:17.154890+0000", 2024-09-13T11:27:01.831 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.6149999999999999e-06 2024-09-13T11:27:01.831 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.831 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.831 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:27:01.831 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:17.154931+0000", 2024-09-13T11:27:01.831 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.1057999999999999e-05 2024-09-13T11:27:01.832 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.832 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.832 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:27:01.832 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:17.154937+0000", 2024-09-13T11:27:01.832 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.6719999999999998e-06 2024-09-13T11:27:01.832 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.832 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.832 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:27:01.832 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:17.155010+0000", 2024-09-13T11:27:01.832 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.2964000000000001e-05 2024-09-13T11:27:01.832 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:27:01.832 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:27:01.832 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:27:01.833 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.833 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.833 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:234291 3.11 3:8c4e5be8:::benchmark_data_smithi177_31116_object93988:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e201)", 2024-09-13T11:27:01.833 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:26:17.320509+0000", 2024-09-13T11:27:01.833 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 44.442474144000002, 2024-09-13T11:27:01.833 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.231498433, 2024-09-13T11:27:01.833 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:27:01.833 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:27:01.833 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:27:01.833 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:27:01.833 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:27:01.833 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:27:01.833 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 234291 2024-09-13T11:27:01.834 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.834 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:27:01.834 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.834 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:27:01.834 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:17.320509+0000", 2024-09-13T11:27:01.834 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:27:01.834 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.834 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.834 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:27:01.834 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:17.320509+0000", 2024-09-13T11:27:01.834 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:27:01.834 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.834 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.835 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:27:01.835 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:17.320510+0000", 2024-09-13T11:27:01.835 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.1999999999999999e-06 2024-09-13T11:27:01.835 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.835 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.835 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:27:01.835 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:17.320514+0000", 2024-09-13T11:27:01.835 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.8139999999999998e-06 2024-09-13T11:27:01.835 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.835 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.835 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:27:01.835 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:17.320523+0000", 2024-09-13T11:27:01.835 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.7589999999999992e-06 2024-09-13T11:27:01.835 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.836 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.836 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:27:01.836 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:17.320529+0000", 2024-09-13T11:27:01.836 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.2499999999999997e-06 2024-09-13T11:27:01.836 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.836 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.836 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:27:01.836 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:17.320580+0000", 2024-09-13T11:27:01.836 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.0980999999999999e-05 2024-09-13T11:27:01.836 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.836 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.836 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:27:01.836 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:17.320704+0000", 2024-09-13T11:27:01.837 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00012416599999999999 2024-09-13T11:27:01.837 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.837 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.837 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 11", 2024-09-13T11:27:01.837 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:17.320812+0000", 2024-09-13T11:27:01.837 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000108383 2024-09-13T11:27:01.837 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.837 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.837 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:27:01.837 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:17.323672+0000", 2024-09-13T11:27:01.837 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0028601329999999999 2024-09-13T11:27:01.837 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.837 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.838 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:27:01.838 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:17.551809+0000", 2024-09-13T11:27:01.838 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.22813635400000001 2024-09-13T11:27:01.838 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.838 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.838 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:27:01.838 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:17.551920+0000", 2024-09-13T11:27:01.838 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000111805 2024-09-13T11:27:01.838 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.838 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.838 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:27:01.838 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:17.552007+0000", 2024-09-13T11:27:01.838 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.6588000000000005e-05 2024-09-13T11:27:01.838 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:27:01.839 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:27:01.839 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:27:01.839 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.839 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.839 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop_reply(client.4665.0:234291 3.11 e201/81 ondisk, result = 0)", 2024-09-13T11:27:01.839 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:26:17.324111+0000", 2024-09-13T11:27:01.839 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 44.438871689000003, 2024-09-13T11:27:01.839 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.22790548199999999, 2024-09-13T11:27:01.839 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:27:01.839 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:27:01.839 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:27:01.839 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:27:01.839 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.840 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:27:01.840 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:17.324111+0000", 2024-09-13T11:27:01.840 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:27:01.840 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.840 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.840 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:27:01.840 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:17.324111+0000", 2024-09-13T11:27:01.840 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:27:01.840 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.840 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.840 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:27:01.840 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:17.324112+0000", 2024-09-13T11:27:01.840 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.313e-06 2024-09-13T11:27:01.841 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.841 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.841 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:27:01.841 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:17.324117+0000", 2024-09-13T11:27:01.841 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.5870000000000003e-06 2024-09-13T11:27:01.841 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.841 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.841 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:27:01.841 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:17.551656+0000", 2024-09-13T11:27:01.841 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.22753944200000001 2024-09-13T11:27:01.841 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.841 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.841 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:27:01.841 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:17.551669+0000", 2024-09-13T11:27:01.842 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2233e-05 2024-09-13T11:27:01.842 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.842 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.842 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:27:01.842 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:17.551780+0000", 2024-09-13T11:27:01.842 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000111471 2024-09-13T11:27:01.842 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.842 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.842 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:27:01.842 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:17.551798+0000", 2024-09-13T11:27:01.842 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.7887e-05 2024-09-13T11:27:01.842 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.842 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.842 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:27:01.843 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:17.552017+0000", 2024-09-13T11:27:01.843 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00021854900000000001 2024-09-13T11:27:01.843 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:27:01.843 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:27:01.843 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:27:01.843 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.843 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.843 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:243814 3.1a 3:5aaae69c:::benchmark_data_smithi177_31116_object103511:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e207)", 2024-09-13T11:27:01.843 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:26:26.761070+0000", 2024-09-13T11:27:01.843 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 35.001912638, 2024-09-13T11:27:01.843 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.222488768, 2024-09-13T11:27:01.843 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:27:01.843 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:27:01.844 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:27:01.844 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:27:01.844 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:27:01.844 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:27:01.844 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 243814 2024-09-13T11:27:01.844 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.844 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:27:01.844 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.844 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:27:01.844 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:26.761070+0000", 2024-09-13T11:27:01.844 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:27:01.844 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.844 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.845 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:27:01.845 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:26.761070+0000", 2024-09-13T11:27:01.845 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:27:01.845 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.845 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.845 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:27:01.845 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:26.761073+0000", 2024-09-13T11:27:01.845 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.9119999999999998e-06 2024-09-13T11:27:01.845 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.845 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.845 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:27:01.845 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:26.761082+0000", 2024-09-13T11:27:01.845 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.7960000000000001e-06 2024-09-13T11:27:01.846 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.846 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.846 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:27:01.846 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:26.761105+0000", 2024-09-13T11:27:01.846 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.2976000000000002e-05 2024-09-13T11:27:01.846 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.846 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.846 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:27:01.846 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:26.761117+0000", 2024-09-13T11:27:01.846 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2323999999999999e-05 2024-09-13T11:27:01.846 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.846 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.846 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:27:01.846 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:26.761207+0000", 2024-09-13T11:27:01.847 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.9534e-05 2024-09-13T11:27:01.847 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.847 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.847 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:27:01.847 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:26.761326+0000", 2024-09-13T11:27:01.847 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000119478 2024-09-13T11:27:01.847 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.847 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.847 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 9", 2024-09-13T11:27:01.847 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:26.761422+0000", 2024-09-13T11:27:01.847 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.577e-05 2024-09-13T11:27:01.847 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.847 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.848 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:27:01.848 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:26.765182+0000", 2024-09-13T11:27:01.848 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0037604280000000001 2024-09-13T11:27:01.848 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.848 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.848 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:27:01.848 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:26.983388+0000", 2024-09-13T11:27:01.848 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.21820598199999999 2024-09-13T11:27:01.848 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.848 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.848 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:27:01.848 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:26.983485+0000", 2024-09-13T11:27:01.848 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.7072999999999997e-05 2024-09-13T11:27:01.849 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.849 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.849 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:27:01.849 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:26.983559+0000", 2024-09-13T11:27:01.849 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.3495e-05 2024-09-13T11:27:01.849 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:27:01.849 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:27:01.849 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:27:01.849 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.849 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.849 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:256318 3.1a 3:58998337:::benchmark_data_smithi177_31116_object116015:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e214)", 2024-09-13T11:27:01.849 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:26:37.418760+0000", 2024-09-13T11:27:01.849 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 24.344223100000001, 2024-09-13T11:27:01.849 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.19965370800000001, 2024-09-13T11:27:01.850 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:27:01.850 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:27:01.850 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:27:01.850 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:27:01.850 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:27:01.850 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:27:01.850 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 256318 2024-09-13T11:27:01.850 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.850 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:27:01.850 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.850 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:27:01.850 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:37.418760+0000", 2024-09-13T11:27:01.850 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:27:01.851 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.851 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.851 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:27:01.851 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:37.418760+0000", 2024-09-13T11:27:01.851 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:27:01.851 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.851 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.851 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:27:01.851 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:37.418763+0000", 2024-09-13T11:27:01.851 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.2770000000000001e-06 2024-09-13T11:27:01.851 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.851 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.851 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:27:01.851 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:37.418771+0000", 2024-09-13T11:27:01.852 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.4209999999999995e-06 2024-09-13T11:27:01.852 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.852 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.852 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:27:01.852 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:37.418790+0000", 2024-09-13T11:27:01.852 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.8898000000000001e-05 2024-09-13T11:27:01.852 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.852 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.852 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:27:01.852 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:37.418800+0000", 2024-09-13T11:27:01.852 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.3819999999999993e-06 2024-09-13T11:27:01.852 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.852 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.852 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:27:01.853 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:37.418868+0000", 2024-09-13T11:27:01.853 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.7972000000000006e-05 2024-09-13T11:27:01.853 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.853 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.853 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:27:01.853 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:37.418974+0000", 2024-09-13T11:27:01.853 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000106036 2024-09-13T11:27:01.853 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.853 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.853 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 9", 2024-09-13T11:27:01.853 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:37.419044+0000", 2024-09-13T11:27:01.853 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.0516e-05 2024-09-13T11:27:01.854 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.854 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.854 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:27:01.854 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:37.421773+0000", 2024-09-13T11:27:01.854 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.002729319 2024-09-13T11:27:01.854 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.854 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.854 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:27:01.854 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:37.618309+0000", 2024-09-13T11:27:01.854 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.196535667 2024-09-13T11:27:01.854 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.854 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.854 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:27:01.855 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:37.618369+0000", 2024-09-13T11:27:01.855 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.0340999999999997e-05 2024-09-13T11:27:01.855 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.855 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.855 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:27:01.855 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:37.618413+0000", 2024-09-13T11:27:01.855 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.3878999999999998e-05 2024-09-13T11:27:01.855 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:27:01.855 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:27:01.855 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:27:01.855 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.855 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.855 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop_reply(client.4665.0:260146 3.1a e219/105 ondisk, result = 0)", 2024-09-13T11:27:01.856 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:26:42.138512+0000", 2024-09-13T11:27:01.856 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 19.624470789, 2024-09-13T11:27:01.856 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.240341576, 2024-09-13T11:27:01.856 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:27:01.856 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:27:01.856 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:27:01.856 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:27:01.856 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.856 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:27:01.856 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:42.138512+0000", 2024-09-13T11:27:01.856 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:27:01.856 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.857 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.857 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:27:01.857 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:42.138512+0000", 2024-09-13T11:27:01.857 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:27:01.857 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.857 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.857 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:27:01.857 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:42.138515+0000", 2024-09-13T11:27:01.857 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.9560000000000002e-06 2024-09-13T11:27:01.857 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.857 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.857 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:27:01.857 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:42.138524+0000", 2024-09-13T11:27:01.857 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.0350000000000007e-06 2024-09-13T11:27:01.858 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.858 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.858 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:27:01.858 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:42.378633+0000", 2024-09-13T11:27:01.858 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.24010957299999999 2024-09-13T11:27:01.858 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.858 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.858 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:27:01.858 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:42.378642+0000", 2024-09-13T11:27:01.858 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.7390000000000005e-06 2024-09-13T11:27:01.858 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.858 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.858 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:27:01.859 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:42.378709+0000", 2024-09-13T11:27:01.859 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.6586000000000002e-05 2024-09-13T11:27:01.859 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.859 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.859 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:27:01.859 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:42.378718+0000", 2024-09-13T11:27:01.859 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.5899999999999997e-06 2024-09-13T11:27:01.859 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.859 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.859 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:27:01.859 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:42.378854+0000", 2024-09-13T11:27:01.859 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00013509700000000001 2024-09-13T11:27:01.859 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:27:01.860 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:27:01.860 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:27:01.860 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.860 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.860 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:260356 3.1a 3:5c5cb7ed:::benchmark_data_smithi177_31116_object120053:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e219)", 2024-09-13T11:27:01.860 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:26:42.487175+0000", 2024-09-13T11:27:01.860 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 19.275807349000001, 2024-09-13T11:27:01.860 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.20567476500000001, 2024-09-13T11:27:01.860 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:27:01.860 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:27:01.860 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:27:01.860 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:27:01.860 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:27:01.861 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:27:01.861 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 260356 2024-09-13T11:27:01.861 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.861 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:27:01.861 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.861 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:27:01.861 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:42.487175+0000", 2024-09-13T11:27:01.861 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:27:01.861 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.861 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.861 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:27:01.861 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:42.487175+0000", 2024-09-13T11:27:01.861 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:27:01.861 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.862 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.862 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:27:01.862 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:42.487180+0000", 2024-09-13T11:27:01.862 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.7570000000000004e-06 2024-09-13T11:27:01.862 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.862 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.862 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:27:01.862 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:42.487195+0000", 2024-09-13T11:27:01.862 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.5016e-05 2024-09-13T11:27:01.862 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.862 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.862 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:27:01.863 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:42.487227+0000", 2024-09-13T11:27:01.863 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.1909999999999998e-05 2024-09-13T11:27:01.863 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.863 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.863 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:27:01.863 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:42.487252+0000", 2024-09-13T11:27:01.863 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.4902999999999999e-05 2024-09-13T11:27:01.863 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.864 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.864 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:27:01.864 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:42.487451+0000", 2024-09-13T11:27:01.864 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000198839 2024-09-13T11:27:01.864 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.864 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.864 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:27:01.864 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:42.487696+0000", 2024-09-13T11:27:01.864 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00024567399999999998 2024-09-13T11:27:01.864 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.864 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.864 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 9", 2024-09-13T11:27:01.865 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:42.487919+0000", 2024-09-13T11:27:01.865 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00022273099999999999 2024-09-13T11:27:01.865 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.865 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.865 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:27:01.865 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:42.491000+0000", 2024-09-13T11:27:01.865 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0030812769999999999 2024-09-13T11:27:01.865 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.865 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.865 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:27:01.865 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:42.692668+0000", 2024-09-13T11:27:01.865 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.20166735199999999 2024-09-13T11:27:01.866 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.866 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.866 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:27:01.866 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:42.692765+0000", 2024-09-13T11:27:01.866 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.7607000000000002e-05 2024-09-13T11:27:01.866 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.866 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.866 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:27:01.866 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:42.692850+0000", 2024-09-13T11:27:01.866 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.4698999999999997e-05 2024-09-13T11:27:01.866 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:27:01.867 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:27:01.867 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:27:01.867 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.867 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.867 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop_reply(client.4665.0:260356 3.1a e219/105 ondisk, result = 0)", 2024-09-13T11:27:01.867 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:26:42.494201+0000", 2024-09-13T11:27:01.867 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 19.268782183999999, 2024-09-13T11:27:01.867 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.198661267, 2024-09-13T11:27:01.867 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:27:01.867 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:27:01.867 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:27:01.867 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:27:01.867 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.867 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:27:01.868 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:42.494201+0000", 2024-09-13T11:27:01.868 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:27:01.868 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.868 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.868 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:27:01.868 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:42.494201+0000", 2024-09-13T11:27:01.868 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:27:01.868 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.868 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.868 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:27:01.868 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:42.494202+0000", 2024-09-13T11:27:01.868 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.852e-06 2024-09-13T11:27:01.868 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.869 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.869 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:27:01.869 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:42.494209+0000", 2024-09-13T11:27:01.869 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.7440000000000001e-06 2024-09-13T11:27:01.869 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.869 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.869 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:27:01.869 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:42.692508+0000", 2024-09-13T11:27:01.869 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.198299117 2024-09-13T11:27:01.869 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.869 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.869 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:27:01.869 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:42.692522+0000", 2024-09-13T11:27:01.869 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.3672999999999999e-05 2024-09-13T11:27:01.870 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.870 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.870 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:27:01.870 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:42.692641+0000", 2024-09-13T11:27:01.870 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000118682 2024-09-13T11:27:01.870 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.870 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.870 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:27:01.870 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:42.692657+0000", 2024-09-13T11:27:01.870 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.6103000000000002e-05 2024-09-13T11:27:01.870 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.870 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.870 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:27:01.871 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:42.692862+0000", 2024-09-13T11:27:01.871 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00020509599999999999 2024-09-13T11:27:01.871 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:27:01.871 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:27:01.871 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:27:01.871 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.871 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.871 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDPGRecoveryDeleteReply(3.31 e232,227 3:8f08ebd6:::benchmark_data_smithi177_31116_object4865:head,148'5911)", 2024-09-13T11:27:01.871 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:26:59.958563+0000", 2024-09-13T11:27:01.871 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 1.804419333, 2024-09-13T11:27:01.871 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.22245790700000001, 2024-09-13T11:27:01.871 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:27:01.871 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:27:01.871 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-13T11:27:01.872 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:27:01.872 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.872 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:27:01.872 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:59.958563+0000", 2024-09-13T11:27:01.872 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:27:01.872 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.872 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.872 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:27:01.872 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:59.958563+0000", 2024-09-13T11:27:01.872 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:27:01.872 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.872 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.872 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:27:01.873 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:59.958565+0000", 2024-09-13T11:27:01.873 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.759e-06 2024-09-13T11:27:01.873 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.873 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.873 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:27:01.873 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:59.958572+0000", 2024-09-13T11:27:01.873 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.1500000000000002e-06 2024-09-13T11:27:01.873 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.873 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.873 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:27:01.873 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:27:00.180446+0000", 2024-09-13T11:27:01.873 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.22187378999999999 2024-09-13T11:27:01.873 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.874 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.874 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:27:01.874 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:27:00.180456+0000", 2024-09-13T11:27:01.874 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.9480000000000003e-06 2024-09-13T11:27:01.874 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.874 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.874 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:27:01.874 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:27:00.180531+0000", 2024-09-13T11:27:01.874 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.5436000000000001e-05 2024-09-13T11:27:01.874 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:01.874 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:01.874 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:27:01.874 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:27:00.181021+0000", 2024-09-13T11:27:01.874 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00048982400000000003 2024-09-13T11:27:01.874 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:27:01.875 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:27:01.875 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:27:01.875 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:27:01.875 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:27:01.875 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-13T11:27:01.876 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-09-13T11:27:01.880 INFO:tasks.ceph.osd.8.smithi001.stderr:2024-09-13T11:27:01.878+0000 7fb9b48ae640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26312) UID: 0 2024-09-13T11:27:01.927 INFO:teuthology.orchestra.run.smithi001.stdout:osd.0: {} 2024-09-13T11:27:01.928 INFO:teuthology.orchestra.run.smithi001.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T11:27:01.978 INFO:tasks.ceph.osd.12.smithi001.stderr:2024-09-13T11:27:01.974+0000 7f892c6e6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26314) UID: 0 2024-09-13T11:27:01.983 INFO:teuthology.orchestra.run.smithi001.stdout:osd.1: {} 2024-09-13T11:27:01.983 INFO:teuthology.orchestra.run.smithi001.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T11:27:02.029 INFO:teuthology.orchestra.run.smithi001.stdout:osd.2: {} 2024-09-13T11:27:02.029 INFO:teuthology.orchestra.run.smithi001.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T11:27:02.071 INFO:teuthology.orchestra.run.smithi001.stdout:osd.3: {} 2024-09-13T11:27:02.072 INFO:teuthology.orchestra.run.smithi001.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T11:27:02.078 INFO:tasks.ceph.osd.4.smithi001.stderr:2024-09-13T11:27:02.074+0000 7ff8e0218640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26310) UID: 0 2024-09-13T11:27:02.084 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-13T11:27:02.084 INFO:teuthology.orchestra.run.smithi001.stdout: "ops": [ 2024-09-13T11:27:02.084 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:02.084 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:275648 3.28 3:15cbbb14:::benchmark_data_smithi177_31116_object135345:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e234)", 2024-09-13T11:27:02.084 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:27:01.939173+0000", 2024-09-13T11:27:02.084 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 0.14502805799999999, 2024-09-13T11:27:02.085 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.145057408, 2024-09-13T11:27:02.085 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:27:02.085 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:27:02.085 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "waiting for sub ops", 2024-09-13T11:27:02.085 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:27:02.085 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:27:02.085 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:27:02.085 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 275648 2024-09-13T11:27:02.085 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:02.085 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:27:02.085 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:02.085 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:27:02.085 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:27:01.939173+0000", 2024-09-13T11:27:02.085 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:27:02.086 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:02.086 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:02.086 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:27:02.086 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:27:01.939173+0000", 2024-09-13T11:27:02.086 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:27:02.086 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:02.086 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:02.086 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:27:02.086 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:27:01.939176+0000", 2024-09-13T11:27:02.086 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.5019999999999999e-06 2024-09-13T11:27:02.086 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:02.086 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:02.086 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:27:02.086 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:27:01.939184+0000", 2024-09-13T11:27:02.087 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.0239999999999997e-06 2024-09-13T11:27:02.087 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:02.087 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:02.087 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:27:02.087 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:27:01.939205+0000", 2024-09-13T11:27:02.087 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.1157000000000001e-05 2024-09-13T11:27:02.087 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:02.087 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:02.087 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:27:02.087 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:27:01.939215+0000", 2024-09-13T11:27:02.087 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.6879999999999994e-06 2024-09-13T11:27:02.087 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:02.087 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:02.087 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:27:02.088 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:27:01.939291+0000", 2024-09-13T11:27:02.088 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.6128e-05 2024-09-13T11:27:02.088 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:02.088 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:02.088 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:27:02.088 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:27:01.939444+0000", 2024-09-13T11:27:02.088 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000152941 2024-09-13T11:27:02.088 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:02.088 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:02.088 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 5", 2024-09-13T11:27:02.088 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:27:01.939556+0000", 2024-09-13T11:27:02.088 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000111939 2024-09-13T11:27:02.088 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:02.088 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:02.089 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:27:02.089 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:27:01.941794+0000", 2024-09-13T11:27:02.089 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0022379380000000001 2024-09-13T11:27:02.089 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:27:02.089 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:27:02.089 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:27:02.089 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:27:02.089 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-09-13T11:27:02.089 INFO:teuthology.orchestra.run.smithi001.stdout: "num_ops": 1 2024-09-13T11:27:02.089 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-13T11:27:02.107 INFO:teuthology.orchestra.run.smithi001.stdout:osd.4: {} 2024-09-13T11:27:02.107 INFO:teuthology.orchestra.run.smithi001.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T11:27:02.110 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-09-13T11:27:02.140 INFO:teuthology.orchestra.run.smithi001.stdout:osd.5: {} 2024-09-13T11:27:02.140 INFO:teuthology.orchestra.run.smithi001.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T11:27:02.167 INFO:teuthology.orchestra.run.smithi001.stdout:osd.6: {} 2024-09-13T11:27:02.167 INFO:teuthology.orchestra.run.smithi001.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T11:27:02.181 INFO:tasks.ceph.osd.4.smithi001.stderr:2024-09-13T11:27:02.174+0000 7ff8e0218640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26310) UID: 0 2024-09-13T11:27:02.196 INFO:teuthology.orchestra.run.smithi001.stdout:osd.7: {} 2024-09-13T11:27:02.197 INFO:teuthology.orchestra.run.smithi001.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T11:27:02.279 INFO:tasks.ceph.osd.7.smithi177.stderr:2024-09-13T11:27:02.277+0000 7f897b51d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26575) UID: 0 2024-09-13T11:27:02.293 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-13T11:27:02.293 INFO:teuthology.orchestra.run.smithi001.stdout: "ops": [], 2024-09-13T11:27:02.293 INFO:teuthology.orchestra.run.smithi001.stdout: "complaint_time": 30, 2024-09-13T11:27:02.293 INFO:teuthology.orchestra.run.smithi001.stdout: "num_blocked_ops": 0 2024-09-13T11:27:02.293 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-13T11:27:02.311 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-09-13T11:27:02.331 INFO:teuthology.orchestra.run.smithi001.stdout:osd.8: {} 2024-09-13T11:27:02.332 INFO:teuthology.orchestra.run.smithi001.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-13T11:27:02.373 INFO:teuthology.orchestra.run.smithi001.stdout:osd.9: {} 2024-09-13T11:27:02.373 INFO:teuthology.orchestra.run.smithi001.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-13T11:27:02.380 INFO:tasks.ceph.osd.5.smithi092.stderr:2024-09-13T11:27:02.375+0000 7f6f0b705640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26346) UID: 0 2024-09-13T11:27:02.405 INFO:teuthology.orchestra.run.smithi001.stdout:osd.10: {} 2024-09-13T11:27:02.405 INFO:teuthology.orchestra.run.smithi001.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-13T11:27:02.440 INFO:teuthology.orchestra.run.smithi001.stdout:osd.11: {} 2024-09-13T11:27:02.441 INFO:teuthology.orchestra.run.smithi001.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-13T11:27:02.473 INFO:teuthology.orchestra.run.smithi001.stdout:osd.12: {} 2024-09-13T11:27:02.474 INFO:teuthology.orchestra.run.smithi001.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-13T11:27:02.481 INFO:tasks.ceph.osd.13.smithi092.stderr:2024-09-13T11:27:02.479+0000 7f3e5afea640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26350) UID: 0 2024-09-13T11:27:02.495 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-13T11:27:02.495 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:27:02.503 INFO:teuthology.orchestra.run.smithi001.stdout:osd.13: {} 2024-09-13T11:27:02.504 INFO:teuthology.orchestra.run.smithi001.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-13T11:27:02.514 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_ops_in_flight 2024-09-13T11:27:02.534 INFO:teuthology.orchestra.run.smithi001.stdout:osd.14: {} 2024-09-13T11:27:02.534 INFO:teuthology.orchestra.run.smithi001.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-13T11:27:02.562 INFO:teuthology.orchestra.run.smithi001.stdout:osd.15: {} 2024-09-13T11:27:02.562 INFO:teuthology.orchestra.run.smithi001.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-13T11:27:02.581 INFO:tasks.ceph.osd.14.smithi162.stderr:2024-09-13T11:27:02.579+0000 7f3df5903640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33154) UID: 0 2024-09-13T11:27:02.682 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-13T11:27:02.678+0000 7f6a68775640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26307) UID: 0 2024-09-13T11:27:02.687 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-13T11:27:02.687 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:27:02.710 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_blocked_ops 2024-09-13T11:27:02.783 INFO:tasks.ceph.osd.9.smithi092.stderr:2024-09-13T11:27:02.779+0000 7ff5ab3d0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26348) UID: 0 2024-09-13T11:27:02.884 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-09-13T11:27:02.883+0000 7f97befc9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26315) UID: 0 2024-09-13T11:27:02.914 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-13T11:27:02.914 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:27:02.933 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_historic_ops 2024-09-13T11:27:02.967 INFO:tasks.ceph.osd.2.smithi162.stderr:2024-09-13T11:27:02.963+0000 7fe921d8b640 -1 osd.2 234 handle_osd_map injecting map crc failure 2024-09-13T11:27:02.970 INFO:teuthology.orchestra.run.smithi001.stderr:noscrub is set 2024-09-13T11:27:02.994 INFO:tasks.ceph.osd.3.smithi177.stderr:2024-09-13T11:27:02.981+0000 7f10adba4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26574) UID: 0 2024-09-13T11:27:03.091 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-13T11:27:03.082+0000 7f6a68775640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26307) UID: 0 2024-09-13T11:27:03.108 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-13T11:27:03.108 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:27:03.131 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-13T11:27:03.186 INFO:tasks.ceph.osd.10.smithi162.stderr:2024-09-13T11:27:03.183+0000 7fb59c9f0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26317) UID: 0 2024-09-13T11:27:03.287 INFO:tasks.ceph.osd.4.smithi001.stderr:2024-09-13T11:27:03.286+0000 7ff8e0218640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26310) UID: 0 2024-09-13T11:27:03.288 INFO:teuthology.orchestra.run.smithi092.stdout:ERROR: (22) Invalid argument 2024-09-13T11:27:03.288 INFO:teuthology.orchestra.run.smithi092.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:27:03.303 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-13T11:27:03.388 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-09-13T11:27:03.387+0000 7f97befc9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26315) UID: 0 2024-09-13T11:27:03.422 INFO:teuthology.orchestra.run.smithi092.stdout:ERROR: (22) Invalid argument 2024-09-13T11:27:03.422 INFO:teuthology.orchestra.run.smithi092.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:27:03.437 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-13T11:27:03.493 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-13T11:27:03.486+0000 7f6a68775640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26307) UID: 0 2024-09-13T11:27:03.568 INFO:teuthology.orchestra.run.smithi092.stdout:ERROR: (22) Invalid argument 2024-09-13T11:27:03.568 INFO:teuthology.orchestra.run.smithi092.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:27:03.584 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-09-13T11:27:03.629 INFO:tasks.ceph.osd.5.smithi092.stderr:2024-09-13T11:27:03.627+0000 7f6f0b705640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26346) UID: 0 2024-09-13T11:27:03.692 INFO:tasks.ceph.osd.3.smithi177.stderr:2024-09-13T11:27:03.689+0000 7f10adba4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26574) UID: 0 2024-09-13T11:27:03.708 INFO:teuthology.orchestra.run.smithi092.stdout:ERROR: (22) Invalid argument 2024-09-13T11:27:03.708 INFO:teuthology.orchestra.run.smithi092.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:27:03.723 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-09-13T11:27:03.809 INFO:tasks.ceph.osd.5.smithi092.stderr:2024-09-13T11:27:03.807+0000 7f6f0b705640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26346) UID: 0 2024-09-13T11:27:03.891 INFO:teuthology.orchestra.run.smithi092.stdout:ERROR: (22) Invalid argument 2024-09-13T11:27:03.891 INFO:teuthology.orchestra.run.smithi092.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:27:03.894 INFO:tasks.ceph.osd.1.smithi092.stderr:2024-09-13T11:27:03.891+0000 7fe7789c5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33718) UID: 0 2024-09-13T11:27:03.906 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-09-13T11:27:03.971 INFO:tasks.ceph.osd.9.smithi092.stderr:2024-09-13T11:27:03.967+0000 7ff59ddf9640 -1 osd.9 235 handle_osd_map injecting map crc failure 2024-09-13T11:27:03.971 INFO:tasks.ceph.osd.11.smithi177.stderr:2024-09-13T11:27:03.969+0000 7f246bcf0640 -1 osd.11 235 handle_osd_map injecting map crc failure 2024-09-13T11:27:03.973 INFO:tasks.ceph.osd.9.smithi092.stderr:2024-09-13T11:27:03.971+0000 7ff59bdf5640 -1 osd.9 235 handle_osd_map injecting map crc failure 2024-09-13T11:27:03.994 INFO:tasks.ceph.osd.12.smithi001.stderr:2024-09-13T11:27:03.990+0000 7f892c6e6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26314) UID: 0 2024-09-13T11:27:04.095 INFO:tasks.ceph.osd.14.smithi162.stderr:2024-09-13T11:27:04.091+0000 7f3df5903640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33154) UID: 0 2024-09-13T11:27:04.117 INFO:teuthology.orchestra.run.smithi092.stdout:ERROR: (22) Invalid argument 2024-09-13T11:27:04.117 INFO:teuthology.orchestra.run.smithi092.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:27:04.136 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_ops_in_flight 2024-09-13T11:27:04.196 INFO:tasks.ceph.osd.14.smithi162.stderr:2024-09-13T11:27:04.191+0000 7f3df5903640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33154) UID: 0 2024-09-13T11:27:04.297 INFO:tasks.ceph.osd.10.smithi162.stderr:2024-09-13T11:27:04.295+0000 7fb59c9f0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26317) UID: 0 2024-09-13T11:27:04.326 INFO:teuthology.orchestra.run.smithi092.stdout:ERROR: (22) Invalid argument 2024-09-13T11:27:04.326 INFO:teuthology.orchestra.run.smithi092.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:27:04.345 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_blocked_ops 2024-09-13T11:27:04.398 INFO:tasks.ceph.osd.1.smithi092.stderr:2024-09-13T11:27:04.395+0000 7fe7789c5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33718) UID: 0 2024-09-13T11:27:04.488 INFO:teuthology.orchestra.run.smithi092.stdout:ERROR: (22) Invalid argument 2024-09-13T11:27:04.488 INFO:teuthology.orchestra.run.smithi092.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:27:04.499 INFO:tasks.ceph.osd.1.smithi092.stderr:2024-09-13T11:27:04.495+0000 7fe7789c5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33718) UID: 0 2024-09-13T11:27:04.505 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_historic_ops 2024-09-13T11:27:04.587 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T11:27:04.636 INFO:tasks.ceph.osd.4.smithi001.stderr:2024-09-13T11:27:04.634+0000 7ff8e0218640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26310) UID: 0 2024-09-13T11:27:04.692 INFO:teuthology.orchestra.run.smithi092.stdout:ERROR: (22) Invalid argument 2024-09-13T11:27:04.692 INFO:teuthology.orchestra.run.smithi092.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:27:04.701 INFO:tasks.ceph.osd.14.smithi162.stderr:2024-09-13T11:27:04.699+0000 7f3df5903640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33154) UID: 0 2024-09-13T11:27:04.712 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-13T11:27:04.802 INFO:tasks.ceph.osd.3.smithi177.stderr:2024-09-13T11:27:04.801+0000 7f10adba4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26574) UID: 0 2024-09-13T11:27:04.821 INFO:teuthology.orchestra.run.smithi001.stdout:osd.0: {} 2024-09-13T11:27:04.821 INFO:teuthology.orchestra.run.smithi001.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T11:27:04.862 INFO:teuthology.orchestra.run.smithi001.stdout:osd.1: {} 2024-09-13T11:27:04.863 INFO:teuthology.orchestra.run.smithi001.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T11:27:04.870 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-13T11:27:04.870 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:27:04.894 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-13T11:27:04.897 INFO:teuthology.orchestra.run.smithi001.stdout:osd.2: {} 2024-09-13T11:27:04.897 INFO:teuthology.orchestra.run.smithi001.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T11:27:04.904 INFO:tasks.ceph.osd.5.smithi092.stderr:2024-09-13T11:27:04.899+0000 7f6f0b705640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26346) UID: 0 2024-09-13T11:27:04.931 INFO:teuthology.orchestra.run.smithi001.stdout:osd.3: {} 2024-09-13T11:27:04.932 INFO:teuthology.orchestra.run.smithi001.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T11:27:04.964 INFO:teuthology.orchestra.run.smithi001.stdout:osd.4: {} 2024-09-13T11:27:04.964 INFO:teuthology.orchestra.run.smithi001.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T11:27:04.969 INFO:tasks.ceph.osd.8.smithi001.stderr:2024-09-13T11:27:04.966+0000 7fb9ab2df640 -1 osd.8 236 handle_osd_map injecting map crc failure 2024-09-13T11:27:04.987 INFO:tasks.ceph.osd.2.smithi162.stderr:2024-09-13T11:27:04.983+0000 7fe91fd87640 -1 osd.2 236 handle_osd_map injecting map crc failure 2024-09-13T11:27:05.003 INFO:tasks.ceph.osd.10.smithi162.stderr:2024-09-13T11:27:04.999+0000 7fb59c9f0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26317) UID: 0 2024-09-13T11:27:05.007 INFO:teuthology.orchestra.run.smithi001.stdout:osd.5: {} 2024-09-13T11:27:05.007 INFO:teuthology.orchestra.run.smithi001.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T11:27:05.022 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-13T11:27:05.038 INFO:teuthology.orchestra.run.smithi001.stdout:osd.6: {} 2024-09-13T11:27:05.039 INFO:teuthology.orchestra.run.smithi001.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T11:27:05.042 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-13T11:27:05.042 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [], 2024-09-13T11:27:05.043 INFO:teuthology.orchestra.run.smithi162.stdout: "complaint_time": 30, 2024-09-13T11:27:05.043 INFO:teuthology.orchestra.run.smithi162.stdout: "num_blocked_ops": 0 2024-09-13T11:27:05.043 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-13T11:27:05.063 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-13T11:27:05.078 INFO:teuthology.orchestra.run.smithi001.stdout:osd.7: {} 2024-09-13T11:27:05.078 INFO:teuthology.orchestra.run.smithi001.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T11:27:05.104 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-09-13T11:27:05.103+0000 7f97befc9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26315) UID: 0 2024-09-13T11:27:05.107 INFO:teuthology.orchestra.run.smithi001.stdout:osd.8: {} 2024-09-13T11:27:05.107 INFO:teuthology.orchestra.run.smithi001.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-13T11:27:05.141 INFO:teuthology.orchestra.run.smithi001.stdout:osd.9: {} 2024-09-13T11:27:05.141 INFO:teuthology.orchestra.run.smithi001.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-13T11:27:05.171 INFO:teuthology.orchestra.run.smithi001.stdout:osd.10: {} 2024-09-13T11:27:05.171 INFO:teuthology.orchestra.run.smithi001.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-13T11:27:05.203 INFO:teuthology.orchestra.run.smithi001.stdout:osd.11: {} 2024-09-13T11:27:05.204 INFO:teuthology.orchestra.run.smithi001.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-13T11:27:05.205 INFO:tasks.ceph.osd.10.smithi162.stderr:2024-09-13T11:27:05.203+0000 7fb59c9f0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26317) UID: 0 2024-09-13T11:27:05.246 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-13T11:27:05.246 INFO:teuthology.orchestra.run.smithi162.stdout: "size": 20, 2024-09-13T11:27:05.247 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 600, 2024-09-13T11:27:05.247 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [ 2024-09-13T11:27:05.247 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.247 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.4665.0:171 3.9 e41/30 3:9e1c7eeb:::benchmark_data_smithi177_31116_object170:head v 41'13, mlcod=41'9)", 2024-09-13T11:27:05.247 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:22:47.418602+0000", 2024-09-13T11:27:05.247 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 257.82696233299998, 2024-09-13T11:27:05.247 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.67235367300000004, 2024-09-13T11:27:05.247 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:27:05.247 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:27:05.247 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:27:05.247 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:27:05.247 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.247 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:27:05.247 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:47.418602+0000", 2024-09-13T11:27:05.248 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:05.248 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.248 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.248 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:27:05.248 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:47.418602+0000", 2024-09-13T11:27:05.248 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:05.248 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.248 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.248 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:27:05.248 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:47.418605+0000", 2024-09-13T11:27:05.248 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.2200000000000001e-06 2024-09-13T11:27:05.248 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.248 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.249 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:27:05.249 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:47.418668+0000", 2024-09-13T11:27:05.249 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.2921000000000003e-05 2024-09-13T11:27:05.249 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.249 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.249 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:27:05.249 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:48.089380+0000", 2024-09-13T11:27:05.249 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.67071191100000005 2024-09-13T11:27:05.249 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.249 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.249 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:27:05.249 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:48.089423+0000", 2024-09-13T11:27:05.250 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.2916e-05 2024-09-13T11:27:05.250 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.250 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.250 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:27:05.250 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:48.089533+0000", 2024-09-13T11:27:05.250 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000109996 2024-09-13T11:27:05.250 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.250 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.250 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:27:05.250 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:48.089570+0000", 2024-09-13T11:27:05.250 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.7314999999999998e-05 2024-09-13T11:27:05.250 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.250 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.250 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:27:05.251 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:48.090852+0000", 2024-09-13T11:27:05.251 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001281641 2024-09-13T11:27:05.251 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.251 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.251 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:27:05.251 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:48.090956+0000", 2024-09-13T11:27:05.251 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00010375299999999999 2024-09-13T11:27:05.251 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:05.251 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:27:05.251 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:05.251 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.251 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.251 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPushReply(3.19 74/70 [PushReplyOp(3:9edf939c:::benchmark_data_smithi177_31116_object17207:head)])", 2024-09-13T11:27:05.251 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:30.897971+0000", 2024-09-13T11:27:05.252 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 214.34759348200001, 2024-09-13T11:27:05.252 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.40616071100000001, 2024-09-13T11:27:05.252 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:27:05.252 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:27:05.252 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-09-13T11:27:05.252 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:27:05.252 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.252 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:27:05.252 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.897971+0000", 2024-09-13T11:27:05.252 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:05.252 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.252 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.253 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:27:05.253 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.897971+0000", 2024-09-13T11:27:05.253 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:05.253 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.253 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.253 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:27:05.253 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.897972+0000", 2024-09-13T11:27:05.253 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1429999999999999e-06 2024-09-13T11:27:05.253 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.253 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.253 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:27:05.253 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.897977+0000", 2024-09-13T11:27:05.253 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.5199999999999999e-06 2024-09-13T11:27:05.253 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.254 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.254 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:27:05.254 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:31.302500+0000", 2024-09-13T11:27:05.254 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.40452312400000001 2024-09-13T11:27:05.254 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.254 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.254 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:27:05.254 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:31.302513+0000", 2024-09-13T11:27:05.254 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2753e-05 2024-09-13T11:27:05.254 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.254 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.254 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:27:05.254 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:31.302645+0000", 2024-09-13T11:27:05.254 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00013292100000000001 2024-09-13T11:27:05.255 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.255 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.255 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:27:05.255 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:31.304132+0000", 2024-09-13T11:27:05.255 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0014862499999999999 2024-09-13T11:27:05.255 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:05.255 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:27:05.255 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:05.255 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.255 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.255 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPushReply(3.19 75/70 [PushReplyOp(3:9d1489b1:::benchmark_data_smithi177_31116_object18698:head)])", 2024-09-13T11:27:05.255 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:31.697212+0000", 2024-09-13T11:27:05.255 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 213.54835248099999, 2024-09-13T11:27:05.255 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.22850633200000001, 2024-09-13T11:27:05.256 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:27:05.256 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:27:05.256 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-09-13T11:27:05.256 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:27:05.256 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.256 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:27:05.256 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:31.697212+0000", 2024-09-13T11:27:05.256 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:05.256 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.256 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.256 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:27:05.256 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:31.697212+0000", 2024-09-13T11:27:05.256 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:05.256 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.257 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.257 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:27:05.257 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:31.697214+0000", 2024-09-13T11:27:05.257 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.7439999999999999e-06 2024-09-13T11:27:05.257 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.257 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.257 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:27:05.257 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:31.697224+0000", 2024-09-13T11:27:05.257 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0173000000000001e-05 2024-09-13T11:27:05.257 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.257 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.257 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:27:05.257 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:31.923931+0000", 2024-09-13T11:27:05.258 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.226706926 2024-09-13T11:27:05.258 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.258 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.258 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:27:05.258 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:31.923950+0000", 2024-09-13T11:27:05.258 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.8899e-05 2024-09-13T11:27:05.258 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.258 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.258 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:27:05.258 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:31.924449+0000", 2024-09-13T11:27:05.259 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00049907800000000002 2024-09-13T11:27:05.259 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.259 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.259 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:27:05.259 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:31.925718+0000", 2024-09-13T11:27:05.259 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001269512 2024-09-13T11:27:05.259 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:05.259 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:27:05.259 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:05.259 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.259 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.259 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPushReply(3.19 76/70 [PushReplyOp(3:9faf6d5f:::benchmark_data_smithi177_31116_object18954:head)])", 2024-09-13T11:27:05.259 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:32.042874+0000", 2024-09-13T11:27:05.259 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 213.202690612, 2024-09-13T11:27:05.260 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.40212404299999999, 2024-09-13T11:27:05.260 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:27:05.260 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:27:05.260 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-09-13T11:27:05.260 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:27:05.260 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.260 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:27:05.260 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.042874+0000", 2024-09-13T11:27:05.260 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:05.260 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.260 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.260 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:27:05.261 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.042874+0000", 2024-09-13T11:27:05.261 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:05.261 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.261 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.261 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:27:05.261 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.042875+0000", 2024-09-13T11:27:05.261 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.432e-06 2024-09-13T11:27:05.261 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.261 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.261 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:27:05.261 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.042886+0000", 2024-09-13T11:27:05.261 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0265000000000001e-05 2024-09-13T11:27:05.261 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.261 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.262 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:27:05.262 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.443455+0000", 2024-09-13T11:27:05.262 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.40056951400000002 2024-09-13T11:27:05.262 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.262 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.262 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:27:05.262 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.443480+0000", 2024-09-13T11:27:05.262 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.4929000000000001e-05 2024-09-13T11:27:05.262 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.262 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.262 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:27:05.262 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.443666+0000", 2024-09-13T11:27:05.262 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000186446 2024-09-13T11:27:05.262 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.263 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.263 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:27:05.263 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.444998+0000", 2024-09-13T11:27:05.263 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0013314570000000001 2024-09-13T11:27:05.263 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:05.263 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:27:05.263 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:05.263 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.263 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.263 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPush(3.1d 85/78 [PushOp(3:bc4b9336:::benchmark_data_smithi177_31116_object11944:head, version: 49'734, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:bc4b9336:::benchmark_data_smithi177_31116_object11944:head@49'734, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:27:05.263 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:42.225966+0000", 2024-09-13T11:27:05.263 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 203.01959839899999, 2024-09-13T11:27:05.263 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.23879947800000001, 2024-09-13T11:27:05.264 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:27:05.264 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:27:05.264 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:27:05.264 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:27:05.264 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.264 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:27:05.264 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.225966+0000", 2024-09-13T11:27:05.264 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:05.264 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.264 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.264 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:27:05.264 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.225966+0000", 2024-09-13T11:27:05.264 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:05.264 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.265 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.265 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:27:05.265 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.225969+0000", 2024-09-13T11:27:05.265 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.0400000000000001e-06 2024-09-13T11:27:05.265 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.265 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.265 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:27:05.265 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.226036+0000", 2024-09-13T11:27:05.265 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.6978999999999997e-05 2024-09-13T11:27:05.265 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.265 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.265 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:27:05.265 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.463110+0000", 2024-09-13T11:27:05.265 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.23707400000000001 2024-09-13T11:27:05.266 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.266 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.266 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:27:05.266 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.463140+0000", 2024-09-13T11:27:05.266 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.9575999999999999e-05 2024-09-13T11:27:05.266 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.266 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.266 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:27:05.266 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.463326+0000", 2024-09-13T11:27:05.266 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000186298 2024-09-13T11:27:05.266 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.266 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.266 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:27:05.266 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.463351+0000", 2024-09-13T11:27:05.267 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.4862999999999999e-05 2024-09-13T11:27:05.267 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.267 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.267 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:27:05.267 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.464766+0000", 2024-09-13T11:27:05.267 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001414722 2024-09-13T11:27:05.267 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:05.267 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:27:05.267 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:05.267 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.267 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.267 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPush(3.1d 89/78 [PushOp(3:bc1b7cfb:::benchmark_data_smithi177_31116_object22629:head, version: 55'1387, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:bc1b7cfb:::benchmark_data_smithi177_31116_object22629:head@55'1387, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:27:05.267 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:46.660855+0000", 2024-09-13T11:27:05.267 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 198.584709903, 2024-09-13T11:27:05.268 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.26622785599999998, 2024-09-13T11:27:05.268 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:27:05.268 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:27:05.268 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:27:05.268 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:27:05.268 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.268 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:27:05.268 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:46.660855+0000", 2024-09-13T11:27:05.268 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:05.268 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.268 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.268 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:27:05.269 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:46.660855+0000", 2024-09-13T11:27:05.269 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:05.269 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.269 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.269 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:27:05.269 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:46.660857+0000", 2024-09-13T11:27:05.269 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.6560000000000001e-06 2024-09-13T11:27:05.269 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.269 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.269 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:27:05.269 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:46.660931+0000", 2024-09-13T11:27:05.269 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.4041999999999993e-05 2024-09-13T11:27:05.270 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.270 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.270 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:27:05.270 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:46.924654+0000", 2024-09-13T11:27:05.270 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.26372239800000002 2024-09-13T11:27:05.270 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.270 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.270 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:27:05.270 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:46.924729+0000", 2024-09-13T11:27:05.270 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.5061999999999994e-05 2024-09-13T11:27:05.270 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.270 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.270 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:27:05.271 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:46.924986+0000", 2024-09-13T11:27:05.271 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00025762000000000002 2024-09-13T11:27:05.271 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.271 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.271 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:27:05.271 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:46.925013+0000", 2024-09-13T11:27:05.271 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.6179000000000001e-05 2024-09-13T11:27:05.271 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.271 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.271 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:27:05.271 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:46.927082+0000", 2024-09-13T11:27:05.271 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.002069899 2024-09-13T11:27:05.271 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:05.271 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:27:05.272 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:05.272 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.272 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.272 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPush(3.1d 90/78 [PushOp(3:bc0e7228:::benchmark_data_smithi177_31116_object24733:head, version: 55'1512, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:bc0e7228:::benchmark_data_smithi177_31116_object24733:head@55'1512, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:27:05.272 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:47.529126+0000", 2024-09-13T11:27:05.272 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 197.71643892899999, 2024-09-13T11:27:05.272 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.21343989499999999, 2024-09-13T11:27:05.272 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:27:05.272 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:27:05.272 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:27:05.272 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:27:05.272 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.272 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:27:05.273 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:47.529126+0000", 2024-09-13T11:27:05.273 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:05.273 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.273 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.273 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:27:05.273 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:47.529126+0000", 2024-09-13T11:27:05.273 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:05.273 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.273 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.273 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:27:05.273 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:47.529127+0000", 2024-09-13T11:27:05.273 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.252e-06 2024-09-13T11:27:05.273 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.274 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.274 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:27:05.274 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:47.529185+0000", 2024-09-13T11:27:05.274 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.8224999999999999e-05 2024-09-13T11:27:05.274 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.274 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.274 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:27:05.274 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:47.740216+0000", 2024-09-13T11:27:05.274 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.21103060800000001 2024-09-13T11:27:05.274 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.274 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.274 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:27:05.274 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:47.740248+0000", 2024-09-13T11:27:05.274 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.2821999999999998e-05 2024-09-13T11:27:05.275 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.275 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.275 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:27:05.275 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:47.740504+0000", 2024-09-13T11:27:05.275 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00025518399999999999 2024-09-13T11:27:05.275 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.275 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.275 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:27:05.275 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:47.740544+0000", 2024-09-13T11:27:05.275 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.0438999999999997e-05 2024-09-13T11:27:05.275 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.275 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.275 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:27:05.275 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:47.742565+0000", 2024-09-13T11:27:05.276 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0020213649999999998 2024-09-13T11:27:05.276 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:05.276 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:27:05.276 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:05.276 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.276 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.276 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:83199 3.1d 3:be2c6fae:::benchmark_data_smithi177_31116_object83198:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106)", 2024-09-13T11:27:05.276 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:24:04.297306+0000", 2024-09-13T11:27:05.276 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 180.94825897699999, 2024-09-13T11:27:05.276 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.25114587999999999, 2024-09-13T11:27:05.276 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:27:05.276 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:27:05.277 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:27:05.277 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:27:05.277 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:27:05.277 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:27:05.277 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 83199 2024-09-13T11:27:05.277 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.277 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:27:05.277 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.277 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:27:05.277 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.297306+0000", 2024-09-13T11:27:05.277 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:05.277 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.277 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.277 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:27:05.278 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.297306+0000", 2024-09-13T11:27:05.278 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:05.278 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.278 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.278 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:27:05.278 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.297308+0000", 2024-09-13T11:27:05.278 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.4150000000000002e-06 2024-09-13T11:27:05.278 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.278 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.278 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:27:05.278 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.297352+0000", 2024-09-13T11:27:05.278 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.4431000000000002e-05 2024-09-13T11:27:05.278 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.278 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.279 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:27:05.279 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.297369+0000", 2024-09-13T11:27:05.279 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.6685e-05 2024-09-13T11:27:05.279 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.279 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.279 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:27:05.279 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.297376+0000", 2024-09-13T11:27:05.279 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.7299999999999999e-06 2024-09-13T11:27:05.279 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.279 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.279 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:27:05.279 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.297427+0000", 2024-09-13T11:27:05.280 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.1461000000000001e-05 2024-09-13T11:27:05.280 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.280 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.280 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:27:05.280 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.297528+0000", 2024-09-13T11:27:05.280 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00010028599999999999 2024-09-13T11:27:05.280 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.280 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.280 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 15", 2024-09-13T11:27:05.280 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.297685+0000", 2024-09-13T11:27:05.280 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00015760500000000001 2024-09-13T11:27:05.280 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.280 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.281 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:27:05.281 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.300779+0000", 2024-09-13T11:27:05.281 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0030936700000000002 2024-09-13T11:27:05.281 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.281 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.281 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:27:05.281 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.548279+0000", 2024-09-13T11:27:05.281 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.24750059999999999 2024-09-13T11:27:05.281 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.281 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.281 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:27:05.281 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.548373+0000", 2024-09-13T11:27:05.281 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.3451999999999999e-05 2024-09-13T11:27:05.281 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.282 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.282 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:27:05.282 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.548451+0000", 2024-09-13T11:27:05.282 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.8545000000000001e-05 2024-09-13T11:27:05.282 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:05.282 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:27:05.282 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:05.282 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.282 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.282 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:84621 3.1d 3:becd72aa:::benchmark_data_smithi177_31116_object84620:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-09-13T11:27:05.282 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:24:05.741008+0000", 2024-09-13T11:27:05.282 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 179.50455697699999, 2024-09-13T11:27:05.282 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.17994101100000001, 2024-09-13T11:27:05.282 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:27:05.283 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:27:05.283 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:27:05.283 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:27:05.283 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:27:05.283 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:27:05.283 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 84621 2024-09-13T11:27:05.283 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.283 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:27:05.283 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.283 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:27:05.283 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.741008+0000", 2024-09-13T11:27:05.283 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:05.283 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.283 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.284 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:27:05.284 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.741008+0000", 2024-09-13T11:27:05.284 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:05.284 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.284 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.284 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:27:05.284 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.741010+0000", 2024-09-13T11:27:05.284 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.898e-06 2024-09-13T11:27:05.284 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.284 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.284 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:27:05.284 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.741061+0000", 2024-09-13T11:27:05.284 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.0898e-05 2024-09-13T11:27:05.285 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.285 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.285 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:27:05.285 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.741085+0000", 2024-09-13T11:27:05.285 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.3558e-05 2024-09-13T11:27:05.285 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.285 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.285 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:27:05.285 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.741093+0000", 2024-09-13T11:27:05.285 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.3890000000000005e-06 2024-09-13T11:27:05.285 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.285 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.285 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:27:05.285 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.741178+0000", 2024-09-13T11:27:05.286 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.4792999999999999e-05 2024-09-13T11:27:05.286 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.286 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.286 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:27:05.286 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.741386+0000", 2024-09-13T11:27:05.286 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000207505 2024-09-13T11:27:05.286 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.286 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.286 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 15", 2024-09-13T11:27:05.286 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.741597+0000", 2024-09-13T11:27:05.286 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00021158200000000001 2024-09-13T11:27:05.286 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.286 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.287 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:27:05.287 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.743392+0000", 2024-09-13T11:27:05.287 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00179518 2024-09-13T11:27:05.287 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.287 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.287 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:27:05.287 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.920866+0000", 2024-09-13T11:27:05.287 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.17747373699999999 2024-09-13T11:27:05.287 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.287 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.287 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:27:05.287 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.920913+0000", 2024-09-13T11:27:05.287 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.6451e-05 2024-09-13T11:27:05.288 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.288 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.288 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:27:05.288 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:05.920949+0000", 2024-09-13T11:27:05.288 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.6019999999999997e-05 2024-09-13T11:27:05.288 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:05.288 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:27:05.288 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:05.288 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.288 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.288 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPush(3.25 124/120 [PushOp(3:a6d4ee3f:::benchmark_data_smithi177_31116_object14712:head, version: 51'872, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:a6d4ee3f:::benchmark_data_smithi177_31116_object14712:head@51'872, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:27:05.288 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:24:31.996253+0000", 2024-09-13T11:27:05.288 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 153.249311205, 2024-09-13T11:27:05.288 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.17694863499999999, 2024-09-13T11:27:05.289 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:27:05.289 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:27:05.289 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:27:05.289 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:27:05.289 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.289 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:27:05.289 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:31.996253+0000", 2024-09-13T11:27:05.289 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:05.289 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.290 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.290 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:27:05.290 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:31.996253+0000", 2024-09-13T11:27:05.290 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:05.290 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.290 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.290 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:27:05.290 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:31.996257+0000", 2024-09-13T11:27:05.290 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.2629999999999999e-06 2024-09-13T11:27:05.290 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.290 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.290 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:27:05.290 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:31.996359+0000", 2024-09-13T11:27:05.290 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000102456 2024-09-13T11:27:05.291 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.291 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.291 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:27:05.291 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:32.172313+0000", 2024-09-13T11:27:05.291 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.17595388000000001 2024-09-13T11:27:05.291 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.291 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.291 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:27:05.291 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:32.172326+0000", 2024-09-13T11:27:05.291 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2632999999999999e-05 2024-09-13T11:27:05.291 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.291 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.291 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:27:05.291 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:32.172423+0000", 2024-09-13T11:27:05.292 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.7084999999999996e-05 2024-09-13T11:27:05.292 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.292 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.292 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:27:05.292 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:32.172438+0000", 2024-09-13T11:27:05.292 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5775999999999999e-05 2024-09-13T11:27:05.292 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.292 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.292 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:27:05.292 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:32.173202+0000", 2024-09-13T11:27:05.292 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00076354199999999998 2024-09-13T11:27:05.292 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:05.292 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:27:05.292 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:05.293 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.293 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.293 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:126039 3.29 3:96100181:::benchmark_data_smithi177_31116_object126038:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e130)", 2024-09-13T11:27:05.293 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:24:43.012785+0000", 2024-09-13T11:27:05.293 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 142.23277985499999, 2024-09-13T11:27:05.293 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.26685311900000003, 2024-09-13T11:27:05.293 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:27:05.293 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:27:05.294 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:27:05.294 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:27:05.294 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:27:05.294 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:27:05.294 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 126039 2024-09-13T11:27:05.294 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.294 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:27:05.294 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.294 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:27:05.294 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:43.012785+0000", 2024-09-13T11:27:05.294 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:05.294 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.294 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.295 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:27:05.295 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:43.012785+0000", 2024-09-13T11:27:05.295 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:05.295 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.295 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.295 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:27:05.295 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:43.012787+0000", 2024-09-13T11:27:05.295 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.875e-06 2024-09-13T11:27:05.295 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.295 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.295 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:27:05.295 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:43.012826+0000", 2024-09-13T11:27:05.295 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.9483999999999997e-05 2024-09-13T11:27:05.295 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.296 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.296 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:27:05.296 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:43.012839+0000", 2024-09-13T11:27:05.296 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3405e-05 2024-09-13T11:27:05.296 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.296 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.296 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:27:05.296 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:43.012845+0000", 2024-09-13T11:27:05.296 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.9100000000000002e-06 2024-09-13T11:27:05.296 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.296 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.296 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:27:05.296 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:43.012903+0000", 2024-09-13T11:27:05.296 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.8131000000000003e-05 2024-09-13T11:27:05.297 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.297 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.297 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:27:05.297 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:43.013066+0000", 2024-09-13T11:27:05.297 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000162058 2024-09-13T11:27:05.297 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.297 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.297 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 4,5", 2024-09-13T11:27:05.297 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:43.013235+0000", 2024-09-13T11:27:05.297 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00016929600000000001 2024-09-13T11:27:05.297 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.297 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.297 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:27:05.298 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:43.017063+0000", 2024-09-13T11:27:05.298 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0038283890000000002 2024-09-13T11:27:05.298 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.298 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.298 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:27:05.298 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:43.017086+0000", 2024-09-13T11:27:05.298 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.2934000000000001e-05 2024-09-13T11:27:05.298 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.298 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.298 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:27:05.298 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:43.278804+0000", 2024-09-13T11:27:05.298 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.26171751799999998 2024-09-13T11:27:05.298 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.298 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.298 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:27:05.299 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:43.278862+0000", 2024-09-13T11:27:05.299 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.8431999999999999e-05 2024-09-13T11:27:05.299 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.299 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.299 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:27:05.299 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:43.279638+0000", 2024-09-13T11:27:05.299 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00077568699999999995 2024-09-13T11:27:05.299 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:05.299 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:27:05.299 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:05.299 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.299 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.299 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop_reply(client.4665.0:126039 3.29 e130/120 ondisk, result = 0)", 2024-09-13T11:27:05.300 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:24:43.021691+0000", 2024-09-13T11:27:05.300 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 142.22387392900001, 2024-09-13T11:27:05.300 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.25795579699999999, 2024-09-13T11:27:05.300 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:27:05.300 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:27:05.300 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:27:05.300 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:27:05.300 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.300 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:27:05.300 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:43.021691+0000", 2024-09-13T11:27:05.300 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:05.300 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.300 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.300 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:27:05.301 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:43.021691+0000", 2024-09-13T11:27:05.301 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:05.301 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.301 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.301 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:27:05.301 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:43.021692+0000", 2024-09-13T11:27:05.301 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3880000000000001e-06 2024-09-13T11:27:05.301 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.301 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.301 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:27:05.301 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:43.021700+0000", 2024-09-13T11:27:05.301 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.9310000000000008e-06 2024-09-13T11:27:05.301 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.302 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.302 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:27:05.302 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:43.278719+0000", 2024-09-13T11:27:05.302 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.25701909299999998 2024-09-13T11:27:05.302 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.302 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.302 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:27:05.302 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:43.278728+0000", 2024-09-13T11:27:05.302 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.5000000000000005e-06 2024-09-13T11:27:05.302 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.302 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.302 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:27:05.302 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:43.278786+0000", 2024-09-13T11:27:05.302 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.7640999999999999e-05 2024-09-13T11:27:05.303 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.303 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.303 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:27:05.303 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:43.278796+0000", 2024-09-13T11:27:05.303 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.9199999999999999e-06 2024-09-13T11:27:05.303 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.303 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.303 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:27:05.303 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:43.279646+0000", 2024-09-13T11:27:05.303 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00085032399999999996 2024-09-13T11:27:05.303 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:05.303 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:27:05.303 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:05.304 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.304 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.304 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.4665.0:138294 3.9 e140/117 3:90012dab:::benchmark_data_smithi177_31116_object138293:head v 140'5246, mlcod=140'5243)", 2024-09-13T11:27:05.304 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:24:54.252227+0000", 2024-09-13T11:27:05.304 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 130.99333744099999, 2024-09-13T11:27:05.304 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.32919836400000002, 2024-09-13T11:27:05.304 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:27:05.304 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:27:05.304 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:27:05.304 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:27:05.304 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.304 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:27:05.304 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:54.252227+0000", 2024-09-13T11:27:05.304 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:05.305 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.305 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.305 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:27:05.305 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:54.252227+0000", 2024-09-13T11:27:05.305 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:05.305 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.305 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.305 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:27:05.305 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:54.252228+0000", 2024-09-13T11:27:05.305 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.8899999999999998e-07 2024-09-13T11:27:05.305 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.305 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.305 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:27:05.305 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:54.252276+0000", 2024-09-13T11:27:05.306 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.8137999999999999e-05 2024-09-13T11:27:05.306 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.306 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.306 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:27:05.306 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:54.578216+0000", 2024-09-13T11:27:05.306 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.32594014399999999 2024-09-13T11:27:05.306 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.306 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.306 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:27:05.306 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:54.578235+0000", 2024-09-13T11:27:05.306 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.8943000000000001e-05 2024-09-13T11:27:05.306 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.306 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.307 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:27:05.307 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:54.578396+0000", 2024-09-13T11:27:05.307 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000160922 2024-09-13T11:27:05.307 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.307 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.307 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:27:05.307 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:54.578457+0000", 2024-09-13T11:27:05.307 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.0665999999999998e-05 2024-09-13T11:27:05.307 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.307 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.307 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:27:05.307 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:54.581339+0000", 2024-09-13T11:27:05.307 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0028826839999999999 2024-09-13T11:27:05.308 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.308 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.308 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:27:05.308 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:54.581425+0000", 2024-09-13T11:27:05.308 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.6077999999999998e-05 2024-09-13T11:27:05.308 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:05.308 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:27:05.308 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:05.308 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.308 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.308 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPushReply(3.29 143/120 [PushReplyOp(3:973d2047:::benchmark_data_smithi177_31116_object49749:head)])", 2024-09-13T11:27:05.308 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:24:57.387106+0000", 2024-09-13T11:27:05.308 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 127.858458114, 2024-09-13T11:27:05.308 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.64813637000000002, 2024-09-13T11:27:05.309 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:27:05.309 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:27:05.309 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-09-13T11:27:05.309 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:27:05.309 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.309 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:27:05.309 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:57.387106+0000", 2024-09-13T11:27:05.309 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:05.309 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.309 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.309 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:27:05.309 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:57.387106+0000", 2024-09-13T11:27:05.309 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:05.310 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.310 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.310 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:27:05.310 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:57.387116+0000", 2024-09-13T11:27:05.310 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.1530000000000003e-06 2024-09-13T11:27:05.310 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.310 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.310 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:27:05.310 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:57.387127+0000", 2024-09-13T11:27:05.310 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1197e-05 2024-09-13T11:27:05.310 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.310 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.310 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:27:05.310 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:58.034264+0000", 2024-09-13T11:27:05.311 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.64713691799999995 2024-09-13T11:27:05.311 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.311 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.311 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:27:05.311 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:58.034273+0000", 2024-09-13T11:27:05.311 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.0929999999999992e-06 2024-09-13T11:27:05.311 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.311 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.311 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:27:05.311 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:58.034321+0000", 2024-09-13T11:27:05.311 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.8396999999999998e-05 2024-09-13T11:27:05.311 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.311 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.311 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:27:05.312 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:58.035243+0000", 2024-09-13T11:27:05.312 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00092161200000000004 2024-09-13T11:27:05.312 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:05.312 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:27:05.312 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:05.312 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.312 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.312 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPushReply(3.29 144/120 [PushReplyOp(3:94a6fc3a:::benchmark_data_smithi177_31116_object50245:head)])", 2024-09-13T11:27:05.312 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:24:58.320977+0000", 2024-09-13T11:27:05.312 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 126.924587531, 2024-09-13T11:27:05.312 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.31110569599999999, 2024-09-13T11:27:05.312 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:27:05.312 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:27:05.312 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-09-13T11:27:05.313 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:27:05.313 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.313 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:27:05.313 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:58.320977+0000", 2024-09-13T11:27:05.313 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:05.313 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.313 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.313 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:27:05.313 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:58.320977+0000", 2024-09-13T11:27:05.313 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:05.313 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.313 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.313 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:27:05.314 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:58.320978+0000", 2024-09-13T11:27:05.314 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.8700000000000004e-07 2024-09-13T11:27:05.314 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.314 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.314 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:27:05.314 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:58.320985+0000", 2024-09-13T11:27:05.314 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.742e-06 2024-09-13T11:27:05.314 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.314 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.314 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:27:05.314 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:58.631147+0000", 2024-09-13T11:27:05.314 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.31016212900000001 2024-09-13T11:27:05.314 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.314 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.315 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:27:05.315 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:58.631159+0000", 2024-09-13T11:27:05.315 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1946999999999999e-05 2024-09-13T11:27:05.315 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.315 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.315 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:27:05.315 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:58.631257+0000", 2024-09-13T11:27:05.315 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.8096000000000004e-05 2024-09-13T11:27:05.315 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.315 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.315 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:27:05.315 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:58.632083+0000", 2024-09-13T11:27:05.316 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00082589499999999999 2024-09-13T11:27:05.316 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:05.316 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:27:05.316 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:05.316 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.316 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.316 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.4665.0:170721 3.9 e162/117 3:93244b1b:::benchmark_data_smithi177_31116_object30418:head v 162'5743, mlcod=162'5740)", 2024-09-13T11:27:05.316 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:25:21.721723+0000", 2024-09-13T11:27:05.316 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 103.523841816, 2024-09-13T11:27:05.316 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.19722754300000001, 2024-09-13T11:27:05.317 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:27:05.317 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:27:05.317 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:27:05.317 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:27:05.317 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.317 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:27:05.317 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:21.721723+0000", 2024-09-13T11:27:05.317 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:05.317 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.317 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.317 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:27:05.317 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:21.721723+0000", 2024-09-13T11:27:05.317 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:05.318 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.318 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.318 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:27:05.318 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:21.721726+0000", 2024-09-13T11:27:05.318 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.1930000000000002e-06 2024-09-13T11:27:05.318 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.318 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.318 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:27:05.318 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:21.721741+0000", 2024-09-13T11:27:05.318 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5299999999999999e-05 2024-09-13T11:27:05.318 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.318 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.318 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:27:05.319 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:21.915450+0000", 2024-09-13T11:27:05.319 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.193708519 2024-09-13T11:27:05.319 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.319 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.319 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:27:05.319 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:21.915462+0000", 2024-09-13T11:27:05.319 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2414e-05 2024-09-13T11:27:05.319 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.319 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.319 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:27:05.319 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:21.915603+0000", 2024-09-13T11:27:05.319 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00014116500000000001 2024-09-13T11:27:05.319 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.319 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.320 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:27:05.320 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:21.915668+0000", 2024-09-13T11:27:05.320 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.4488999999999999e-05 2024-09-13T11:27:05.320 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.320 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.320 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:27:05.320 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:21.918850+0000", 2024-09-13T11:27:05.320 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0031825310000000002 2024-09-13T11:27:05.320 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.320 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.320 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:27:05.320 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:21.918950+0000", 2024-09-13T11:27:05.320 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.9932000000000005e-05 2024-09-13T11:27:05.320 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:05.321 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:27:05.321 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:05.321 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.321 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.321 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:191058 3.1d 3:bd6e7029:::benchmark_data_smithi177_31116_object50755:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e175)", 2024-09-13T11:27:05.321 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:25:38.512403+0000", 2024-09-13T11:27:05.321 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 86.733161271, 2024-09-13T11:27:05.321 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.38726333200000002, 2024-09-13T11:27:05.321 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:27:05.321 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:27:05.321 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:27:05.321 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:27:05.321 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:27:05.322 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:27:05.322 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 191058 2024-09-13T11:27:05.322 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.322 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:27:05.322 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.322 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:27:05.322 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:38.512403+0000", 2024-09-13T11:27:05.322 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:05.322 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.322 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.322 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:27:05.322 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:38.512403+0000", 2024-09-13T11:27:05.322 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:05.323 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.323 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.323 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:27:05.323 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:38.512406+0000", 2024-09-13T11:27:05.323 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.4059999999999999e-06 2024-09-13T11:27:05.323 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.323 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.323 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:27:05.323 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:38.512412+0000", 2024-09-13T11:27:05.323 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.8170000000000001e-06 2024-09-13T11:27:05.323 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.323 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.323 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:27:05.324 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:38.512430+0000", 2024-09-13T11:27:05.324 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.7397999999999998e-05 2024-09-13T11:27:05.324 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.324 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.324 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:27:05.324 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:38.512438+0000", 2024-09-13T11:27:05.324 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.3459999999999994e-06 2024-09-13T11:27:05.324 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.324 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.324 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:27:05.324 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:38.512526+0000", 2024-09-13T11:27:05.324 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.8146999999999995e-05 2024-09-13T11:27:05.325 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.325 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.325 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:27:05.325 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:38.512722+0000", 2024-09-13T11:27:05.325 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00019587599999999999 2024-09-13T11:27:05.325 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.325 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.325 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 15", 2024-09-13T11:27:05.325 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:38.512882+0000", 2024-09-13T11:27:05.325 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00015957300000000001 2024-09-13T11:27:05.325 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.325 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.325 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:27:05.326 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:38.517854+0000", 2024-09-13T11:27:05.326 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0049725500000000001 2024-09-13T11:27:05.326 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.326 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.326 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:27:05.326 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:38.899562+0000", 2024-09-13T11:27:05.326 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.38170719800000003 2024-09-13T11:27:05.326 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.326 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.327 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:27:05.327 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:38.899622+0000", 2024-09-13T11:27:05.327 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.0396999999999998e-05 2024-09-13T11:27:05.327 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.327 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.327 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:27:05.327 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:38.899667+0000", 2024-09-13T11:27:05.327 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.4623999999999998e-05 2024-09-13T11:27:05.327 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:05.327 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:27:05.327 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:05.327 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.327 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.327 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.4665.0:191547 3.25 e176/141 3:a7ac3e5b:::benchmark_data_smithi177_31116_object51244:head v 176'6020, mlcod=175'6018)", 2024-09-13T11:27:05.328 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:25:39.496290+0000", 2024-09-13T11:27:05.328 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 85.749274709999995, 2024-09-13T11:27:05.328 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.179206057, 2024-09-13T11:27:05.328 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:27:05.328 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:27:05.328 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:27:05.328 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:27:05.328 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.328 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:27:05.328 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:39.496290+0000", 2024-09-13T11:27:05.328 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:05.328 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.329 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.329 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:27:05.329 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:39.496290+0000", 2024-09-13T11:27:05.329 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:05.329 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.329 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.329 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:27:05.329 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:39.496293+0000", 2024-09-13T11:27:05.329 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.8499999999999998e-06 2024-09-13T11:27:05.329 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.329 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.329 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:27:05.329 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:39.496305+0000", 2024-09-13T11:27:05.330 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2286999999999999e-05 2024-09-13T11:27:05.330 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.330 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.330 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:27:05.330 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:39.670614+0000", 2024-09-13T11:27:05.330 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.17430884699999999 2024-09-13T11:27:05.330 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.330 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.330 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:27:05.330 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:39.670625+0000", 2024-09-13T11:27:05.330 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1233000000000001e-05 2024-09-13T11:27:05.330 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.330 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.330 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:27:05.331 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:39.670722+0000", 2024-09-13T11:27:05.331 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.6885999999999994e-05 2024-09-13T11:27:05.331 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.331 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.331 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:27:05.331 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:39.670756+0000", 2024-09-13T11:27:05.331 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.4125999999999998e-05 2024-09-13T11:27:05.331 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.331 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.331 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:27:05.331 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:39.675411+0000", 2024-09-13T11:27:05.331 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0046546160000000003 2024-09-13T11:27:05.332 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.332 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.332 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:27:05.332 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:39.675496+0000", 2024-09-13T11:27:05.332 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.5211999999999997e-05 2024-09-13T11:27:05.332 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:05.332 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:27:05.332 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:05.332 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.332 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.332 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.4665.0:250598 3.9 e211/117 3:929b3711:::benchmark_data_smithi177_31116_object110295:head v 211'6996, mlcod=211'6992)", 2024-09-13T11:27:05.332 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:26:32.474084+0000", 2024-09-13T11:27:05.332 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 32.771480681, 2024-09-13T11:27:05.333 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.24130304, 2024-09-13T11:27:05.333 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:27:05.333 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:27:05.333 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:27:05.333 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:27:05.333 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.333 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:27:05.333 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:32.474084+0000", 2024-09-13T11:27:05.333 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:05.333 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.333 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.333 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:27:05.333 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:32.474084+0000", 2024-09-13T11:27:05.334 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:05.334 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.334 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.334 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:27:05.334 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:32.474085+0000", 2024-09-13T11:27:05.334 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.2699999999999998e-07 2024-09-13T11:27:05.334 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.334 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.334 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:27:05.334 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:32.474090+0000", 2024-09-13T11:27:05.334 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.8339999999999998e-06 2024-09-13T11:27:05.334 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.334 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.334 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:27:05.335 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:32.713078+0000", 2024-09-13T11:27:05.335 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.23898883900000001 2024-09-13T11:27:05.335 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.335 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.335 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:27:05.335 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:32.713088+0000", 2024-09-13T11:27:05.335 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.5170000000000005e-06 2024-09-13T11:27:05.335 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.335 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.335 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:27:05.335 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:32.713150+0000", 2024-09-13T11:27:05.335 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.1715000000000001e-05 2024-09-13T11:27:05.335 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.336 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.336 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:27:05.336 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:32.713165+0000", 2024-09-13T11:27:05.336 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5804999999999999e-05 2024-09-13T11:27:05.336 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.336 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.336 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:27:05.336 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:32.715350+0000", 2024-09-13T11:27:05.336 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0021848979999999998 2024-09-13T11:27:05.336 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.336 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.336 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:27:05.336 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:32.715387+0000", 2024-09-13T11:27:05.336 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.6504999999999997e-05 2024-09-13T11:27:05.337 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:05.337 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:27:05.337 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:05.337 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.337 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.337 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGRecoveryDelete(3.33 e231,227 3:cf137e7b:::benchmark_data_smithi177_31116_object9019:head,149'6092)", 2024-09-13T11:27:05.337 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:26:58.970211+0000", 2024-09-13T11:27:05.337 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 6.2753539280000004, 2024-09-13T11:27:05.337 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.41248640800000003, 2024-09-13T11:27:05.337 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:27:05.337 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:27:05.337 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:27:05.337 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:27:05.338 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.338 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:27:05.338 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:58.970211+0000", 2024-09-13T11:27:05.338 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:05.338 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.338 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.338 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:27:05.338 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:58.970211+0000", 2024-09-13T11:27:05.338 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:05.338 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.338 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.338 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:27:05.338 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:58.970218+0000", 2024-09-13T11:27:05.339 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.9019999999999999e-06 2024-09-13T11:27:05.339 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.339 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.339 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:27:05.339 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:58.970230+0000", 2024-09-13T11:27:05.339 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1341e-05 2024-09-13T11:27:05.339 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.339 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.339 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:27:05.339 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:59.382297+0000", 2024-09-13T11:27:05.339 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.41206738199999998 2024-09-13T11:27:05.339 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.339 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.339 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:27:05.340 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:59.382313+0000", 2024-09-13T11:27:05.340 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5519000000000002e-05 2024-09-13T11:27:05.340 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.340 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.340 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:27:05.340 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:59.382471+0000", 2024-09-13T11:27:05.340 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00015788900000000001 2024-09-13T11:27:05.340 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.340 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.340 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:27:05.340 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:59.382499+0000", 2024-09-13T11:27:05.340 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.8073e-05 2024-09-13T11:27:05.340 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.341 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.341 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:27:05.341 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:59.382697+0000", 2024-09-13T11:27:05.341 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000198302 2024-09-13T11:27:05.341 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:05.341 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:27:05.341 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:05.341 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:05.341 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:27:05.341 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-13T11:27:05.343 INFO:teuthology.orchestra.run.smithi001.stdout:osd.12: {} 2024-09-13T11:27:05.343 INFO:teuthology.orchestra.run.smithi001.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-13T11:27:05.344 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-13T11:27:05.345 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-09-13T11:27:05.339+0000 7f97befc9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26315) UID: 0 2024-09-13T11:27:05.345 INFO:teuthology.orchestra.run.smithi001.stdout:osd.13: {} 2024-09-13T11:27:05.345 INFO:teuthology.orchestra.run.smithi001.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-13T11:27:05.440 INFO:teuthology.orchestra.run.smithi001.stdout:osd.14: {} 2024-09-13T11:27:05.440 INFO:teuthology.orchestra.run.smithi001.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-13T11:27:05.443 INFO:tasks.ceph.osd.9.smithi092.stderr:2024-09-13T11:27:05.439+0000 7ff5ab3d0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26348) UID: 0 2024-09-13T11:27:05.473 INFO:teuthology.orchestra.run.smithi001.stdout:osd.15: {} 2024-09-13T11:27:05.473 INFO:teuthology.orchestra.run.smithi001.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-13T11:27:05.527 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-13T11:27:05.527 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [], 2024-09-13T11:27:05.527 INFO:teuthology.orchestra.run.smithi162.stdout: "num_ops": 0 2024-09-13T11:27:05.527 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-13T11:27:05.542 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-13T11:27:05.544 INFO:tasks.ceph.osd.4.smithi001.stderr:2024-09-13T11:27:05.542+0000 7ff8e0218640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26310) UID: 0 2024-09-13T11:27:05.645 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-13T11:27:05.642+0000 7f6a68775640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26307) UID: 0 2024-09-13T11:27:05.678 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-13T11:27:05.678 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [], 2024-09-13T11:27:05.678 INFO:teuthology.orchestra.run.smithi162.stdout: "complaint_time": 30, 2024-09-13T11:27:05.678 INFO:teuthology.orchestra.run.smithi162.stdout: "num_blocked_ops": 0 2024-09-13T11:27:05.678 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-13T11:27:05.693 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-13T11:27:05.746 INFO:tasks.ceph.osd.8.smithi001.stderr:2024-09-13T11:27:05.742+0000 7fb9b48ae640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26312) UID: 0 2024-09-13T11:27:05.827 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-13T11:27:05.827 INFO:teuthology.orchestra.run.smithi162.stdout: "size": 20, 2024-09-13T11:27:05.828 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 600, 2024-09-13T11:27:05.828 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [ 2024-09-13T11:27:05.828 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.828 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4257.0:80 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 262144~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-13T11:27:05.828 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:22:12.516065+0000", 2024-09-13T11:27:05.828 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 293.310196211, 2024-09-13T11:27:05.828 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.46461997999999999, 2024-09-13T11:27:05.828 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:27:05.828 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:27:05.828 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:27:05.828 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:27:05.829 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4257", 2024-09-13T11:27:05.829 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/1488131527", 2024-09-13T11:27:05.829 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 80 2024-09-13T11:27:05.829 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.829 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:27:05.829 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.829 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:27:05.829 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.516065+0000", 2024-09-13T11:27:05.829 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:05.829 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.829 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.829 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:27:05.829 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.516065+0000", 2024-09-13T11:27:05.830 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:05.830 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.830 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.830 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:27:05.830 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.516066+0000", 2024-09-13T11:27:05.830 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0869999999999999e-06 2024-09-13T11:27:05.830 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.830 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.830 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:27:05.830 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.516112+0000", 2024-09-13T11:27:05.830 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.5503000000000001e-05 2024-09-13T11:27:05.830 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.830 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.830 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:27:05.831 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.516128+0000", 2024-09-13T11:27:05.831 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5891000000000001e-05 2024-09-13T11:27:05.831 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.831 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.831 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:27:05.831 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.516138+0000", 2024-09-13T11:27:05.831 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.8120000000000002e-06 2024-09-13T11:27:05.831 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.831 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.831 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:27:05.831 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.520398+0000", 2024-09-13T11:27:05.831 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0042605450000000001 2024-09-13T11:27:05.831 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.832 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.832 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:27:05.832 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.520526+0000", 2024-09-13T11:27:05.832 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000128355 2024-09-13T11:27:05.832 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.832 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.832 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 0", 2024-09-13T11:27:05.832 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.520730+0000", 2024-09-13T11:27:05.832 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00020337799999999999 2024-09-13T11:27:05.832 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.832 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.832 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:27:05.832 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.522417+0000", 2024-09-13T11:27:05.833 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001687184 2024-09-13T11:27:05.833 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.833 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.833 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:27:05.833 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.980408+0000", 2024-09-13T11:27:05.833 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.45799068399999998 2024-09-13T11:27:05.833 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.833 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.833 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:27:05.833 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.980565+0000", 2024-09-13T11:27:05.833 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00015760899999999999 2024-09-13T11:27:05.833 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.833 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.834 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:27:05.834 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.980685+0000", 2024-09-13T11:27:05.834 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000119932 2024-09-13T11:27:05.834 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:05.834 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:27:05.834 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:05.834 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.834 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.834 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop_reply(client.4257.0:80 1.0 e24/23 ondisk, result = 0)", 2024-09-13T11:27:05.834 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:22:12.522938+0000", 2024-09-13T11:27:05.834 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 293.30332317800003, 2024-09-13T11:27:05.834 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.45776420600000001, 2024-09-13T11:27:05.834 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:27:05.835 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:27:05.835 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:27:05.835 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:27:05.835 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.835 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:27:05.835 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.522938+0000", 2024-09-13T11:27:05.835 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:05.835 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.835 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.835 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:27:05.835 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.522938+0000", 2024-09-13T11:27:05.835 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:05.835 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.836 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.836 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:27:05.836 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.522941+0000", 2024-09-13T11:27:05.836 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.0290000000000001e-06 2024-09-13T11:27:05.836 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.836 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.836 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:27:05.836 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.522950+0000", 2024-09-13T11:27:05.836 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.7499999999999992e-06 2024-09-13T11:27:05.836 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.836 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.836 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:27:05.836 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.980215+0000", 2024-09-13T11:27:05.837 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.45726523800000002 2024-09-13T11:27:05.837 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.837 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.837 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:27:05.837 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.980229+0000", 2024-09-13T11:27:05.837 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3549e-05 2024-09-13T11:27:05.837 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.837 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.837 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:27:05.837 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.980365+0000", 2024-09-13T11:27:05.837 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00013608700000000001 2024-09-13T11:27:05.837 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.837 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.838 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:27:05.838 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.980388+0000", 2024-09-13T11:27:05.838 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.3112000000000002e-05 2024-09-13T11:27:05.838 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.838 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.838 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:27:05.838 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.980702+0000", 2024-09-13T11:27:05.838 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00031444099999999999 2024-09-13T11:27:05.838 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:05.838 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:27:05.838 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:05.838 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.839 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.839 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:25416 3.8 3:1a1fbdd1:::benchmark_data_smithi177_31116_object25415:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e56)", 2024-09-13T11:27:05.839 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:04.512188+0000", 2024-09-13T11:27:05.839 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 241.31407321099999, 2024-09-13T11:27:05.839 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.210208697, 2024-09-13T11:27:05.839 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:27:05.839 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:27:05.839 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:27:05.839 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:27:05.839 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:27:05.839 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:27:05.839 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 25416 2024-09-13T11:27:05.839 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.840 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:27:05.840 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.840 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:27:05.840 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.512188+0000", 2024-09-13T11:27:05.840 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:05.840 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.840 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.840 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:27:05.840 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.512188+0000", 2024-09-13T11:27:05.840 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:05.840 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.840 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.840 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:27:05.840 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.512192+0000", 2024-09-13T11:27:05.841 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.2660000000000003e-06 2024-09-13T11:27:05.841 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.841 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.841 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:27:05.841 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.512247+0000", 2024-09-13T11:27:05.841 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.4889000000000003e-05 2024-09-13T11:27:05.841 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.841 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.841 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:27:05.841 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.512279+0000", 2024-09-13T11:27:05.841 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.1146000000000003e-05 2024-09-13T11:27:05.841 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.841 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.841 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:27:05.842 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.512297+0000", 2024-09-13T11:27:05.842 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.825e-05 2024-09-13T11:27:05.842 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.842 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.842 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:27:05.842 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.512377+0000", 2024-09-13T11:27:05.842 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.9875000000000004e-05 2024-09-13T11:27:05.842 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.842 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.842 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:27:05.842 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.512570+0000", 2024-09-13T11:27:05.842 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00019312100000000001 2024-09-13T11:27:05.842 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.843 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.844 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 1", 2024-09-13T11:27:05.844 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.512798+0000", 2024-09-13T11:27:05.844 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00022847899999999999 2024-09-13T11:27:05.844 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.844 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.844 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:27:05.844 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.514975+0000", 2024-09-13T11:27:05.844 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0021769979999999999 2024-09-13T11:27:05.844 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.845 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.845 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:27:05.845 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.722207+0000", 2024-09-13T11:27:05.845 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.20723217099999999 2024-09-13T11:27:05.845 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.845 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.845 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:27:05.845 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.722314+0000", 2024-09-13T11:27:05.845 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00010648400000000001 2024-09-13T11:27:05.845 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.846 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.846 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:27:05.846 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.722397+0000", 2024-09-13T11:27:05.846 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.3017999999999997e-05 2024-09-13T11:27:05.846 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:05.846 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:27:05.846 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:05.846 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.846 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.846 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop_reply(client.4665.0:25416 3.8 e56/40 ondisk, result = 0)", 2024-09-13T11:27:05.846 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:04.515635+0000", 2024-09-13T11:27:05.847 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 241.31062655900001, 2024-09-13T11:27:05.847 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.20677600500000001, 2024-09-13T11:27:05.847 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:27:05.847 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:27:05.847 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:27:05.847 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:27:05.847 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.847 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:27:05.847 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.515635+0000", 2024-09-13T11:27:05.847 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:05.848 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.848 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.848 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:27:05.848 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.515635+0000", 2024-09-13T11:27:05.848 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:05.848 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.848 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.848 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:27:05.848 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.515639+0000", 2024-09-13T11:27:05.848 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.619e-06 2024-09-13T11:27:05.849 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.849 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.849 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:27:05.849 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.515650+0000", 2024-09-13T11:27:05.849 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.111e-05 2024-09-13T11:27:05.849 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.849 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.849 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:27:05.849 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.722004+0000", 2024-09-13T11:27:05.849 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.206354805 2024-09-13T11:27:05.849 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.850 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.850 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:27:05.850 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.722017+0000", 2024-09-13T11:27:05.850 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2840000000000001e-05 2024-09-13T11:27:05.850 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.850 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.850 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:27:05.850 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.722186+0000", 2024-09-13T11:27:05.850 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00016898299999999999 2024-09-13T11:27:05.850 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.850 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.851 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:27:05.851 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.722199+0000", 2024-09-13T11:27:05.851 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.289e-05 2024-09-13T11:27:05.851 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.851 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.851 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:27:05.851 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:04.722411+0000", 2024-09-13T11:27:05.851 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00021175799999999999 2024-09-13T11:27:05.851 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:05.852 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:27:05.852 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:05.852 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.852 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.852 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:30833 3.8 3:1c203f8d:::benchmark_data_smithi177_31116_object30832:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e59)", 2024-09-13T11:27:05.852 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:09.506656+0000", 2024-09-13T11:27:05.852 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 236.319605661, 2024-09-13T11:27:05.852 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.224862282, 2024-09-13T11:27:05.852 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:27:05.852 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:27:05.853 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:27:05.853 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:27:05.853 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:27:05.853 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:27:05.853 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 30833 2024-09-13T11:27:05.853 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.853 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:27:05.853 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.853 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:27:05.853 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.506656+0000", 2024-09-13T11:27:05.854 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:05.854 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.854 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.854 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:27:05.854 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.506656+0000", 2024-09-13T11:27:05.854 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:05.854 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.854 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.854 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:27:05.854 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.506660+0000", 2024-09-13T11:27:05.854 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.6e-06 2024-09-13T11:27:05.855 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.855 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.855 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:27:05.855 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.506717+0000", 2024-09-13T11:27:05.855 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.6131999999999997e-05 2024-09-13T11:27:05.855 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.855 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.855 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:27:05.855 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.506751+0000", 2024-09-13T11:27:05.855 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.4746000000000002e-05 2024-09-13T11:27:05.856 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.856 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.856 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:27:05.856 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.506774+0000", 2024-09-13T11:27:05.856 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.2636000000000001e-05 2024-09-13T11:27:05.856 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.856 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.856 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:27:05.856 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.506878+0000", 2024-09-13T11:27:05.857 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000104057 2024-09-13T11:27:05.857 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.857 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.857 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:27:05.857 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.507052+0000", 2024-09-13T11:27:05.857 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00017402800000000001 2024-09-13T11:27:05.857 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.857 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.857 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 1", 2024-09-13T11:27:05.858 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.507217+0000", 2024-09-13T11:27:05.858 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00016516 2024-09-13T11:27:05.858 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.858 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.858 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:27:05.858 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.508992+0000", 2024-09-13T11:27:05.858 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0017743990000000001 2024-09-13T11:27:05.858 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.858 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.858 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:27:05.858 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.731429+0000", 2024-09-13T11:27:05.859 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.22243718000000001 2024-09-13T11:27:05.859 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.859 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.859 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:27:05.859 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.731479+0000", 2024-09-13T11:27:05.859 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.0146999999999999e-05 2024-09-13T11:27:05.859 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.859 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.859 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:27:05.859 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.731518+0000", 2024-09-13T11:27:05.859 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.9196999999999998e-05 2024-09-13T11:27:05.860 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:05.860 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:27:05.860 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:05.860 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.860 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.860 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop_reply(client.4665.0:30833 3.8 e59/40 ondisk, result = 0)", 2024-09-13T11:27:05.860 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:09.508907+0000", 2024-09-13T11:27:05.860 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 236.317354567, 2024-09-13T11:27:05.860 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.222618862, 2024-09-13T11:27:05.861 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:27:05.861 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:27:05.861 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:27:05.861 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:27:05.861 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.861 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:27:05.861 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.508907+0000", 2024-09-13T11:27:05.861 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:05.861 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.861 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.862 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:27:05.862 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.508907+0000", 2024-09-13T11:27:05.862 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:05.862 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.862 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.862 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:27:05.862 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.508910+0000", 2024-09-13T11:27:05.862 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.993e-06 2024-09-13T11:27:05.862 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.862 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.862 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:27:05.863 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.508918+0000", 2024-09-13T11:27:05.863 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.7000000000000008e-06 2024-09-13T11:27:05.863 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.863 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.863 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:27:05.863 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.731341+0000", 2024-09-13T11:27:05.863 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.22242308699999999 2024-09-13T11:27:05.863 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.863 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.863 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:27:05.863 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.731350+0000", 2024-09-13T11:27:05.864 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.3179999999999996e-06 2024-09-13T11:27:05.864 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.864 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.864 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:27:05.864 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.731418+0000", 2024-09-13T11:27:05.864 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.7586e-05 2024-09-13T11:27:05.864 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.864 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.864 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:27:05.864 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.731424+0000", 2024-09-13T11:27:05.864 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.844e-06 2024-09-13T11:27:05.864 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.865 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.865 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:27:05.865 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.731526+0000", 2024-09-13T11:27:05.865 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000101334 2024-09-13T11:27:05.865 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:05.865 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:27:05.865 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:05.865 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.865 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.865 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPush(3.10 76/70 [PushOp(3:0e7d2700:::benchmark_data_smithi177_31116_object17406:head, version: 52'1075, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:0e7d2700:::benchmark_data_smithi177_31116_object17406:head@52'1075, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:27:05.866 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:32.206198+0000", 2024-09-13T11:27:05.866 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 213.62006385800001, 2024-09-13T11:27:05.866 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.25193395000000002, 2024-09-13T11:27:05.866 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:27:05.866 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:27:05.866 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:27:05.866 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:27:05.866 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.866 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:27:05.866 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.206198+0000", 2024-09-13T11:27:05.866 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:05.867 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.867 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.867 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:27:05.867 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.206198+0000", 2024-09-13T11:27:05.867 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:05.867 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.867 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.867 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:27:05.867 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.206199+0000", 2024-09-13T11:27:05.867 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.46e-06 2024-09-13T11:27:05.867 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.868 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.868 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:27:05.868 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.206250+0000", 2024-09-13T11:27:05.868 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.113e-05 2024-09-13T11:27:05.868 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.868 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.868 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:27:05.868 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.456042+0000", 2024-09-13T11:27:05.868 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.24979182999999999 2024-09-13T11:27:05.868 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.869 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.869 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:27:05.869 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.456084+0000", 2024-09-13T11:27:05.869 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.1814000000000003e-05 2024-09-13T11:27:05.869 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.869 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.869 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:27:05.869 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.456342+0000", 2024-09-13T11:27:05.869 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000258404 2024-09-13T11:27:05.869 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.869 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.870 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:27:05.870 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.456367+0000", 2024-09-13T11:27:05.870 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.5099000000000001e-05 2024-09-13T11:27:05.870 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.870 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.870 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:27:05.870 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.458132+0000", 2024-09-13T11:27:05.870 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0017642129999999999 2024-09-13T11:27:05.870 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:05.870 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:27:05.870 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:05.870 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.871 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.871 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:60417 3.10 3:092808b0:::benchmark_data_smithi177_31116_object60416:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e85)", 2024-09-13T11:27:05.871 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:42.173238+0000", 2024-09-13T11:27:05.871 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 203.65302295000001, 2024-09-13T11:27:05.871 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.38965994599999998, 2024-09-13T11:27:05.871 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:27:05.871 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:27:05.871 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:27:05.871 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:27:05.871 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:27:05.871 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:27:05.871 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 60417 2024-09-13T11:27:05.872 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.872 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:27:05.872 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.872 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:27:05.872 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.173238+0000", 2024-09-13T11:27:05.872 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:05.872 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.872 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.872 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:27:05.872 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.173238+0000", 2024-09-13T11:27:05.872 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:05.872 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.873 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.873 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:27:05.873 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.173241+0000", 2024-09-13T11:27:05.873 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.6819999999999999e-06 2024-09-13T11:27:05.873 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.873 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.873 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:27:05.873 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.173335+0000", 2024-09-13T11:27:05.873 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.3957999999999997e-05 2024-09-13T11:27:05.873 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.874 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.874 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:27:05.874 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.173358+0000", 2024-09-13T11:27:05.874 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.3023999999999998e-05 2024-09-13T11:27:05.874 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.874 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.874 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:27:05.874 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.173373+0000", 2024-09-13T11:27:05.874 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4867e-05 2024-09-13T11:27:05.874 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.874 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.875 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:27:05.875 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.173466+0000", 2024-09-13T11:27:05.875 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.2542999999999998e-05 2024-09-13T11:27:05.875 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.875 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.875 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:27:05.875 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.173754+0000", 2024-09-13T11:27:05.875 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000288657 2024-09-13T11:27:05.875 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.875 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.875 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 0", 2024-09-13T11:27:05.876 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.174066+0000", 2024-09-13T11:27:05.876 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00031170999999999998 2024-09-13T11:27:05.876 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.876 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.876 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:27:05.876 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.178245+0000", 2024-09-13T11:27:05.876 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00417864 2024-09-13T11:27:05.876 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.876 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.876 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:27:05.876 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.562617+0000", 2024-09-13T11:27:05.877 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.38437253199999999 2024-09-13T11:27:05.877 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.877 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.877 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:27:05.877 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.562782+0000", 2024-09-13T11:27:05.877 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00016499200000000001 2024-09-13T11:27:05.877 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.877 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.877 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:27:05.877 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.562898+0000", 2024-09-13T11:27:05.877 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000116341 2024-09-13T11:27:05.878 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:05.878 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:27:05.878 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:05.878 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.878 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.878 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:83193 3.8 3:10066d83:::benchmark_data_smithi177_31116_object83192:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106)", 2024-09-13T11:27:05.878 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:24:04.283923+0000", 2024-09-13T11:27:05.878 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 181.54233826699999, 2024-09-13T11:27:05.878 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.64775463499999997, 2024-09-13T11:27:05.878 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:27:05.878 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:27:05.879 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:27:05.879 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:27:05.879 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:27:05.879 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:27:05.879 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 83193 2024-09-13T11:27:05.879 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.879 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:27:05.879 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.879 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:27:05.879 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.283923+0000", 2024-09-13T11:27:05.879 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:05.879 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.880 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.880 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:27:05.880 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.283923+0000", 2024-09-13T11:27:05.880 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:05.880 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.880 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.880 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:27:05.880 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.283925+0000", 2024-09-13T11:27:05.880 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.9489999999999999e-06 2024-09-13T11:27:05.880 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.880 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.880 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:27:05.881 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.283960+0000", 2024-09-13T11:27:05.881 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.5063e-05 2024-09-13T11:27:05.881 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.881 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.881 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:27:05.881 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.283974+0000", 2024-09-13T11:27:05.881 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3546e-05 2024-09-13T11:27:05.881 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.881 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.881 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:27:05.881 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.283983+0000", 2024-09-13T11:27:05.882 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.2499999999999995e-06 2024-09-13T11:27:05.882 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.882 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.882 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:27:05.882 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.284061+0000", 2024-09-13T11:27:05.882 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.7946000000000002e-05 2024-09-13T11:27:05.882 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.882 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.882 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:27:05.882 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.284270+0000", 2024-09-13T11:27:05.882 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00020876799999999999 2024-09-13T11:27:05.882 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.883 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.883 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 1", 2024-09-13T11:27:05.883 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.284489+0000", 2024-09-13T11:27:05.883 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00021956800000000001 2024-09-13T11:27:05.883 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.883 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.883 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:27:05.883 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.286456+0000", 2024-09-13T11:27:05.883 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0019665809999999998 2024-09-13T11:27:05.883 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.883 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.884 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:27:05.884 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.931497+0000", 2024-09-13T11:27:05.884 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.64504147700000003 2024-09-13T11:27:05.884 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.884 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.884 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:27:05.884 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.931589+0000", 2024-09-13T11:27:05.884 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.2041999999999997e-05 2024-09-13T11:27:05.884 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.884 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.885 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:27:05.885 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.931678+0000", 2024-09-13T11:27:05.885 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.8444999999999998e-05 2024-09-13T11:27:05.885 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:05.885 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:27:05.885 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:05.885 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.885 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.885 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:115060 3.8 3:11200ba9:::benchmark_data_smithi177_31116_object115059:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e125)", 2024-09-13T11:27:05.885 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:24:33.602178+0000", 2024-09-13T11:27:05.885 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 152.22408327299999, 2024-09-13T11:27:05.886 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.23668186799999999, 2024-09-13T11:27:05.886 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:27:05.886 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:27:05.886 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:27:05.886 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:27:05.886 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:27:05.886 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:27:05.886 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 115060 2024-09-13T11:27:05.886 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.886 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:27:05.887 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.887 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:27:05.887 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:33.602178+0000", 2024-09-13T11:27:05.887 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:05.887 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.887 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.887 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:27:05.887 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:33.602178+0000", 2024-09-13T11:27:05.888 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:05.888 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.888 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.888 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:27:05.888 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:33.602181+0000", 2024-09-13T11:27:05.888 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.0369999999999999e-06 2024-09-13T11:27:05.888 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.888 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.888 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:27:05.889 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:33.602217+0000", 2024-09-13T11:27:05.889 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.5837999999999998e-05 2024-09-13T11:27:05.889 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.889 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.889 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:27:05.889 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:33.602235+0000", 2024-09-13T11:27:05.889 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.7571000000000002e-05 2024-09-13T11:27:05.889 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.889 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.889 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:27:05.889 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:33.602245+0000", 2024-09-13T11:27:05.889 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0360000000000001e-05 2024-09-13T11:27:05.890 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.890 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.890 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:27:05.890 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:33.602312+0000", 2024-09-13T11:27:05.890 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.6989000000000005e-05 2024-09-13T11:27:05.890 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.890 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.890 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:27:05.890 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:33.602490+0000", 2024-09-13T11:27:05.890 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000178464 2024-09-13T11:27:05.891 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.891 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.891 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 1", 2024-09-13T11:27:05.891 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:33.602670+0000", 2024-09-13T11:27:05.891 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00018002799999999999 2024-09-13T11:27:05.891 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.891 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.891 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:27:05.891 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:33.604382+0000", 2024-09-13T11:27:05.891 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001711314 2024-09-13T11:27:05.892 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.892 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.892 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:27:05.892 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:33.838738+0000", 2024-09-13T11:27:05.892 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.23435634699999999 2024-09-13T11:27:05.892 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.892 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.892 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:27:05.892 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:33.838815+0000", 2024-09-13T11:27:05.892 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.7381999999999999e-05 2024-09-13T11:27:05.893 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.893 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.893 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:27:05.893 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:33.838860+0000", 2024-09-13T11:27:05.893 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.4537999999999999e-05 2024-09-13T11:27:05.893 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:05.893 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:27:05.893 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:05.893 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.893 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.893 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop_reply(client.4665.0:115060 3.8 e125/117 ondisk, result = 0)", 2024-09-13T11:27:05.894 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:24:33.604404+0000", 2024-09-13T11:27:05.894 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 152.22185701500001, 2024-09-13T11:27:05.894 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.23446235400000001, 2024-09-13T11:27:05.894 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:27:05.894 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:27:05.894 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:27:05.894 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:27:05.894 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.894 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:27:05.894 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:33.604404+0000", 2024-09-13T11:27:05.894 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:05.894 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.895 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.895 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:27:05.895 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:33.604404+0000", 2024-09-13T11:27:05.895 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:05.895 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.895 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.895 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:27:05.895 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:33.604406+0000", 2024-09-13T11:27:05.895 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.418e-06 2024-09-13T11:27:05.895 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.895 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.896 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:27:05.896 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:33.604411+0000", 2024-09-13T11:27:05.896 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.3519999999999997e-06 2024-09-13T11:27:05.896 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.896 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.896 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:27:05.896 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:33.838578+0000", 2024-09-13T11:27:05.896 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.23416656199999999 2024-09-13T11:27:05.896 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.896 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.896 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:27:05.897 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:33.838586+0000", 2024-09-13T11:27:05.897 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.9920000000000007e-06 2024-09-13T11:27:05.897 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.897 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.897 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:27:05.898 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:33.838655+0000", 2024-09-13T11:27:05.898 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.9463999999999998e-05 2024-09-13T11:27:05.898 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.898 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.898 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:27:05.898 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:33.838731+0000", 2024-09-13T11:27:05.898 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.5487000000000004e-05 2024-09-13T11:27:05.898 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.898 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.898 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:27:05.899 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:33.838867+0000", 2024-09-13T11:27:05.899 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000136079 2024-09-13T11:27:05.899 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:05.899 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:27:05.899 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:05.899 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.899 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.899 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPush(3.6 129/122 [PushOp(3:6261f3b6:::benchmark_data_smithi177_31116_object64090:head, version: 90'3486, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:6261f3b6:::benchmark_data_smithi177_31116_object64090:head@90'3486, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:27:05.899 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:24:38.641487+0000", 2024-09-13T11:27:05.899 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 147.18477460400001, 2024-09-13T11:27:05.900 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.209889098, 2024-09-13T11:27:05.900 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:27:05.900 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:27:05.900 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:27:05.900 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:27:05.900 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.900 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:27:05.900 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:38.641487+0000", 2024-09-13T11:27:05.900 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:05.900 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.900 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.900 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:27:05.901 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:38.641487+0000", 2024-09-13T11:27:05.901 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:05.901 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.901 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.901 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:27:05.901 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:38.641489+0000", 2024-09-13T11:27:05.901 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.0980000000000001e-06 2024-09-13T11:27:05.901 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.901 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.901 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:27:05.901 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:38.641560+0000", 2024-09-13T11:27:05.902 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.1404999999999998e-05 2024-09-13T11:27:05.902 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.902 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.902 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:27:05.902 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:38.850498+0000", 2024-09-13T11:27:05.902 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.20893800500000001 2024-09-13T11:27:05.902 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.902 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.902 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:27:05.902 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:38.850510+0000", 2024-09-13T11:27:05.902 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1762e-05 2024-09-13T11:27:05.902 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.902 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.903 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:27:05.903 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:38.850610+0000", 2024-09-13T11:27:05.903 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00010011200000000001 2024-09-13T11:27:05.903 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.903 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.903 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:27:05.903 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:38.850628+0000", 2024-09-13T11:27:05.903 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.7291000000000001e-05 2024-09-13T11:27:05.903 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.903 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.903 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:27:05.903 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:38.851376+0000", 2024-09-13T11:27:05.903 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00074842500000000005 2024-09-13T11:27:05.904 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:05.904 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:27:05.904 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:05.904 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.904 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.904 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.4665.0:126919 3.6 e130/122 3:634ce08e:::benchmark_data_smithi177_31116_object126918:head v 130'5137, mlcod=130'5134)", 2024-09-13T11:27:05.904 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:24:43.780120+0000", 2024-09-13T11:27:05.904 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 142.046141713, 2024-09-13T11:27:05.904 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.298086295, 2024-09-13T11:27:05.904 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:27:05.904 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:27:05.905 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:27:05.905 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:27:05.905 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.905 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:27:05.905 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:43.780120+0000", 2024-09-13T11:27:05.905 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:05.905 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.905 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.905 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:27:05.905 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:43.780120+0000", 2024-09-13T11:27:05.905 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:05.906 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.906 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.906 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:27:05.906 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:43.780123+0000", 2024-09-13T11:27:05.906 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.79e-06 2024-09-13T11:27:05.906 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.906 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.906 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:27:05.906 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:43.780160+0000", 2024-09-13T11:27:05.906 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.7945999999999999e-05 2024-09-13T11:27:05.906 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.907 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.907 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:27:05.907 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:44.075853+0000", 2024-09-13T11:27:05.907 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.29569234999999999 2024-09-13T11:27:05.907 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.907 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.907 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:27:05.907 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:44.075861+0000", 2024-09-13T11:27:05.907 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.9510000000000006e-06 2024-09-13T11:27:05.907 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.908 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.908 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:27:05.908 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:44.075924+0000", 2024-09-13T11:27:05.908 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.2946000000000003e-05 2024-09-13T11:27:05.908 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.908 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.908 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:27:05.908 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:44.075942+0000", 2024-09-13T11:27:05.908 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.8108e-05 2024-09-13T11:27:05.908 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.908 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.909 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:27:05.909 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:44.078164+0000", 2024-09-13T11:27:05.909 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.002222375 2024-09-13T11:27:05.909 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.909 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.909 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:27:05.909 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:44.078206+0000", 2024-09-13T11:27:05.909 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.1829000000000002e-05 2024-09-13T11:27:05.909 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:05.909 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:27:05.910 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:05.910 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.910 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.910 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:137780 3.8 3:1047f268:::benchmark_data_smithi177_31116_object137779:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e139)", 2024-09-13T11:27:05.910 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:24:53.795212+0000", 2024-09-13T11:27:05.910 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 132.03104992900001, 2024-09-13T11:27:05.910 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.21530981399999999, 2024-09-13T11:27:05.910 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:27:05.910 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:27:05.910 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:27:05.910 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:27:05.910 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:27:05.911 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:27:05.911 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 137780 2024-09-13T11:27:05.911 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.911 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:27:05.911 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.911 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:27:05.911 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:53.795212+0000", 2024-09-13T11:27:05.911 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:05.911 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.911 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.911 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:27:05.911 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:53.795212+0000", 2024-09-13T11:27:05.911 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:05.912 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.912 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.912 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:27:05.912 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:53.795214+0000", 2024-09-13T11:27:05.912 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.481e-06 2024-09-13T11:27:05.912 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.912 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.912 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:27:05.912 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:53.795247+0000", 2024-09-13T11:27:05.912 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.3105999999999998e-05 2024-09-13T11:27:05.912 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.912 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.913 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:27:05.913 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:53.795258+0000", 2024-09-13T11:27:05.913 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0872e-05 2024-09-13T11:27:05.913 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.913 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.913 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:27:05.913 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:53.795265+0000", 2024-09-13T11:27:05.913 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.5729999999999996e-06 2024-09-13T11:27:05.913 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.913 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.913 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:27:05.913 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:53.795309+0000", 2024-09-13T11:27:05.914 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.4617999999999999e-05 2024-09-13T11:27:05.914 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.914 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.914 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:27:05.914 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:53.795397+0000", 2024-09-13T11:27:05.914 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.7863999999999998e-05 2024-09-13T11:27:05.914 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.914 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.914 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 1", 2024-09-13T11:27:05.914 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:53.795493+0000", 2024-09-13T11:27:05.914 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.6062000000000003e-05 2024-09-13T11:27:05.914 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.915 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.915 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:27:05.915 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:53.797245+0000", 2024-09-13T11:27:05.915 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0017521559999999999 2024-09-13T11:27:05.915 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.915 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.915 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:27:05.915 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:54.010409+0000", 2024-09-13T11:27:05.915 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.21316385299999999 2024-09-13T11:27:05.915 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.916 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.916 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:27:05.916 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:54.010474+0000", 2024-09-13T11:27:05.916 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.4998000000000003e-05 2024-09-13T11:27:05.916 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.916 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.916 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:27:05.916 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:54.010521+0000", 2024-09-13T11:27:05.916 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.7231000000000002e-05 2024-09-13T11:27:05.916 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:05.916 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:27:05.917 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:05.917 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.917 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.917 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop_reply(client.4665.0:137780 3.8 e139/117 ondisk, result = 0)", 2024-09-13T11:27:05.917 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:24:53.797821+0000", 2024-09-13T11:27:05.917 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 132.028440031, 2024-09-13T11:27:05.917 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.21270765599999999, 2024-09-13T11:27:05.917 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:27:05.917 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:27:05.917 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:27:05.917 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:27:05.918 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.918 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:27:05.918 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:53.797821+0000", 2024-09-13T11:27:05.918 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:05.918 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.918 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.918 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:27:05.918 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:53.797821+0000", 2024-09-13T11:27:05.918 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:05.918 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.918 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.918 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:27:05.919 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:53.797822+0000", 2024-09-13T11:27:05.919 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.012e-06 2024-09-13T11:27:05.919 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.919 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.919 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:27:05.919 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:53.797829+0000", 2024-09-13T11:27:05.919 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.8009999999999998e-06 2024-09-13T11:27:05.919 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.919 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.919 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:27:05.919 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:54.010334+0000", 2024-09-13T11:27:05.920 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.212504477 2024-09-13T11:27:05.920 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.920 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.920 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:27:05.920 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:54.010342+0000", 2024-09-13T11:27:05.920 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.6549999999999993e-06 2024-09-13T11:27:05.920 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.920 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.920 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:27:05.920 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:54.010394+0000", 2024-09-13T11:27:05.920 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.1875000000000001e-05 2024-09-13T11:27:05.921 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.921 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.921 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:27:05.921 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:54.010402+0000", 2024-09-13T11:27:05.921 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.6499999999999996e-06 2024-09-13T11:27:05.921 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.921 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.921 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:27:05.921 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:54.010529+0000", 2024-09-13T11:27:05.921 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000127186 2024-09-13T11:27:05.921 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:05.922 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:27:05.922 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:05.922 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.922 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.922 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.4665.0:196314 3.6 e178/122 3:62e79b1c:::benchmark_data_smithi177_31116_object56011:head v 178'6256, mlcod=178'6253)", 2024-09-13T11:27:05.922 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:25:43.340693+0000", 2024-09-13T11:27:05.922 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 82.485568450000002, 2024-09-13T11:27:05.922 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.24109011699999999, 2024-09-13T11:27:05.922 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:27:05.922 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:27:05.922 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:27:05.923 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:27:05.923 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.923 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:27:05.923 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:43.340693+0000", 2024-09-13T11:27:05.923 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:05.923 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.923 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.923 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:27:05.923 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:43.340693+0000", 2024-09-13T11:27:05.923 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:05.924 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.924 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.924 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:27:05.924 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:43.340704+0000", 2024-09-13T11:27:05.924 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1049e-05 2024-09-13T11:27:05.924 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.924 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.924 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:27:05.924 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:43.340718+0000", 2024-09-13T11:27:05.924 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4214e-05 2024-09-13T11:27:05.924 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.924 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.924 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:27:05.925 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:43.579964+0000", 2024-09-13T11:27:05.925 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.23924590200000001 2024-09-13T11:27:05.925 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.925 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.925 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:27:05.925 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:43.579972+0000", 2024-09-13T11:27:05.925 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.2789999999999998e-06 2024-09-13T11:27:05.925 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.925 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.925 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:27:05.925 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:43.580043+0000", 2024-09-13T11:27:05.925 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.0856000000000001e-05 2024-09-13T11:27:05.926 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.926 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.926 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:27:05.926 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:43.580065+0000", 2024-09-13T11:27:05.926 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.1795999999999999e-05 2024-09-13T11:27:05.926 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.926 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.926 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:27:05.926 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:43.581741+0000", 2024-09-13T11:27:05.926 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0016759190000000001 2024-09-13T11:27:05.926 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.926 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.927 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:27:05.927 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:43.581783+0000", 2024-09-13T11:27:05.927 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.2101999999999998e-05 2024-09-13T11:27:05.927 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:05.927 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:27:05.927 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:05.927 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.927 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.927 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:227446 3.10 3:0a4bd1eb:::benchmark_data_smithi177_31116_object87143:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e196)", 2024-09-13T11:27:05.927 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:26:11.077279+0000", 2024-09-13T11:27:05.928 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 54.748982681999998, 2024-09-13T11:27:05.928 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.18995021400000001, 2024-09-13T11:27:05.928 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:27:05.928 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:27:05.928 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:27:05.928 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:27:05.928 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:27:05.928 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:27:05.928 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 227446 2024-09-13T11:27:05.928 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.928 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:27:05.929 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.929 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:27:05.929 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:11.077279+0000", 2024-09-13T11:27:05.929 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:05.929 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.929 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.929 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:27:05.929 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:11.077279+0000", 2024-09-13T11:27:05.929 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:05.929 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.929 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.930 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:27:05.930 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:11.077281+0000", 2024-09-13T11:27:05.930 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.3429999999999998e-06 2024-09-13T11:27:05.930 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.930 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.930 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:27:05.930 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:11.077287+0000", 2024-09-13T11:27:05.930 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.353e-06 2024-09-13T11:27:05.930 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.930 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.930 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:27:05.931 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:11.077303+0000", 2024-09-13T11:27:05.931 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5523e-05 2024-09-13T11:27:05.931 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.931 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.931 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:27:05.931 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:11.077312+0000", 2024-09-13T11:27:05.931 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.6049999999999998e-06 2024-09-13T11:27:05.931 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.931 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.931 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:27:05.931 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:11.077359+0000", 2024-09-13T11:27:05.931 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.7388000000000001e-05 2024-09-13T11:27:05.931 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.932 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.932 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:27:05.932 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:11.077448+0000", 2024-09-13T11:27:05.932 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.9431999999999993e-05 2024-09-13T11:27:05.932 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.932 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.932 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 0", 2024-09-13T11:27:05.932 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:11.077513+0000", 2024-09-13T11:27:05.932 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.4543999999999997e-05 2024-09-13T11:27:05.932 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.932 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.933 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:27:05.933 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:11.080314+0000", 2024-09-13T11:27:05.933 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.002801441 2024-09-13T11:27:05.933 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.933 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.933 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:27:05.933 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:11.267144+0000", 2024-09-13T11:27:05.933 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.18682944700000001 2024-09-13T11:27:05.933 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.933 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.933 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:27:05.934 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:11.267191+0000", 2024-09-13T11:27:05.934 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.7227e-05 2024-09-13T11:27:05.934 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.934 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.934 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:27:05.934 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:11.267229+0000", 2024-09-13T11:27:05.934 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.7910999999999997e-05 2024-09-13T11:27:05.934 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:05.934 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:27:05.934 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:05.934 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.935 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.935 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop_reply(client.4665.0:227446 3.10 e196/84 ondisk, result = 0)", 2024-09-13T11:27:05.935 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:26:11.079968+0000", 2024-09-13T11:27:05.935 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 54.746293887999997, 2024-09-13T11:27:05.935 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.18726515199999999, 2024-09-13T11:27:05.935 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:27:05.935 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:27:05.935 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:27:05.935 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:27:05.935 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.935 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:27:05.935 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:11.079968+0000", 2024-09-13T11:27:05.935 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:05.936 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.936 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.936 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:27:05.936 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:11.079968+0000", 2024-09-13T11:27:05.936 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:05.936 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.936 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.936 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:27:05.936 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:11.079969+0000", 2024-09-13T11:27:05.936 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.42e-06 2024-09-13T11:27:05.936 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.937 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.937 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:27:05.937 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:11.079974+0000", 2024-09-13T11:27:05.937 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.3739999999999997e-06 2024-09-13T11:27:05.937 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.937 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.937 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:27:05.937 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:11.267071+0000", 2024-09-13T11:27:05.937 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.18709714699999999 2024-09-13T11:27:05.937 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.937 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.938 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:27:05.938 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:11.267080+0000", 2024-09-13T11:27:05.938 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.8589999999999994e-06 2024-09-13T11:27:05.938 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.938 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.938 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:27:05.938 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:11.267134+0000", 2024-09-13T11:27:05.938 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.4131000000000001e-05 2024-09-13T11:27:05.938 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.938 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.939 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:27:05.939 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:11.267140+0000", 2024-09-13T11:27:05.939 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.9499999999999998e-06 2024-09-13T11:27:05.939 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.939 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.939 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:27:05.939 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:11.267233+0000", 2024-09-13T11:27:05.939 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.2270999999999998e-05 2024-09-13T11:27:05.939 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:05.939 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:27:05.939 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:05.939 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.939 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.940 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:266873 3.30 3:0f4a798b:::benchmark_data_smithi177_31116_object126570:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e224)", 2024-09-13T11:27:05.940 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:26:49.175028+0000", 2024-09-13T11:27:05.940 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 16.651233763, 2024-09-13T11:27:05.940 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.89281703099999998, 2024-09-13T11:27:05.940 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:27:05.940 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:27:05.940 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:27:05.940 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:27:05.940 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:27:05.940 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:27:05.940 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 266873 2024-09-13T11:27:05.941 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.941 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:27:05.941 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.941 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:27:05.941 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:49.175028+0000", 2024-09-13T11:27:05.941 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:05.941 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.941 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.941 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:27:05.941 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:49.175028+0000", 2024-09-13T11:27:05.942 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:05.942 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.942 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.942 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:27:05.942 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:49.175031+0000", 2024-09-13T11:27:05.942 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.4300000000000002e-06 2024-09-13T11:27:05.942 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.942 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.942 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:27:05.942 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:49.175041+0000", 2024-09-13T11:27:05.943 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.6299999999999993e-06 2024-09-13T11:27:05.943 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.943 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.943 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:27:05.943 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:49.175066+0000", 2024-09-13T11:27:05.943 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.5193000000000001e-05 2024-09-13T11:27:05.943 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.943 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.943 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:27:05.943 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:49.175084+0000", 2024-09-13T11:27:05.943 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.853e-05 2024-09-13T11:27:05.944 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.944 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.944 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:27:05.944 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:49.253283+0000", 2024-09-13T11:27:05.944 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.078198765000000003 2024-09-13T11:27:05.944 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.944 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.944 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-09-13T11:27:05.944 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:49.253292+0000", 2024-09-13T11:27:05.944 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.0920000000000004e-06 2024-09-13T11:27:05.944 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.945 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.945 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:27:05.945 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:50.059052+0000", 2024-09-13T11:27:05.945 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.80575967599999998 2024-09-13T11:27:05.945 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.945 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.945 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:27:05.945 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:50.059226+0000", 2024-09-13T11:27:05.945 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00017433499999999999 2024-09-13T11:27:05.945 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.945 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.946 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 0", 2024-09-13T11:27:05.946 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:50.059374+0000", 2024-09-13T11:27:05.946 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00014768600000000001 2024-09-13T11:27:05.946 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.946 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.946 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:27:05.946 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:50.062440+0000", 2024-09-13T11:27:05.946 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.003066086 2024-09-13T11:27:05.946 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.946 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.946 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:27:05.947 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:50.067764+0000", 2024-09-13T11:27:05.947 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0053240370000000002 2024-09-13T11:27:05.947 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.947 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.947 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:27:05.947 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:50.067802+0000", 2024-09-13T11:27:05.947 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.7462000000000001e-05 2024-09-13T11:27:05.947 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.947 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.947 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:27:05.947 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:50.067845+0000", 2024-09-13T11:27:05.948 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.3108999999999997e-05 2024-09-13T11:27:05.948 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:05.948 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:27:05.948 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:05.948 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.948 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.948 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGRecoveryDelete(3.33 e231,227 3:cf137e7b:::benchmark_data_smithi177_31116_object9019:head,149'6092)", 2024-09-13T11:27:05.948 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:26:58.969284+0000", 2024-09-13T11:27:05.948 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 6.856977401, 2024-09-13T11:27:05.948 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.41337520300000002, 2024-09-13T11:27:05.949 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:27:05.949 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:27:05.949 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:27:05.949 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:27:05.949 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.949 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:27:05.949 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:58.969284+0000", 2024-09-13T11:27:05.949 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:05.949 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.949 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.949 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:27:05.950 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:58.969284+0000", 2024-09-13T11:27:05.950 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:05.950 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.950 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.950 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:27:05.950 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:58.969286+0000", 2024-09-13T11:27:05.950 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.1330000000000002e-06 2024-09-13T11:27:05.950 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.950 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.950 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:27:05.951 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:58.969295+0000", 2024-09-13T11:27:05.951 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.6740000000000003e-06 2024-09-13T11:27:05.951 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.951 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.951 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:27:05.951 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:59.382366+0000", 2024-09-13T11:27:05.951 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.41307074799999999 2024-09-13T11:27:05.951 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.951 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.951 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:27:05.951 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:59.382384+0000", 2024-09-13T11:27:05.951 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.8202999999999998e-05 2024-09-13T11:27:05.952 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.952 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.952 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:27:05.952 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:59.382515+0000", 2024-09-13T11:27:05.952 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00013096599999999999 2024-09-13T11:27:05.952 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.952 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.952 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:27:05.952 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:59.382530+0000", 2024-09-13T11:27:05.952 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4797000000000001e-05 2024-09-13T11:27:05.952 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:05.953 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:05.953 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:27:05.953 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:59.382659+0000", 2024-09-13T11:27:05.953 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000129682 2024-09-13T11:27:05.953 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:05.953 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:27:05.953 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:05.953 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:05.953 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:27:05.953 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-13T11:27:05.955 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-09-13T11:27:05.956 INFO:tasks.ceph.osd.1.smithi092.stderr:2024-09-13T11:27:05.951+0000 7fe7789c5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33718) UID: 0 2024-09-13T11:27:05.980 INFO:teuthology.orchestra.run.smithi001.stderr:nodeep-scrub is set 2024-09-13T11:27:05.994 INFO:tasks.ceph.osd.1.smithi092.stderr:2024-09-13T11:27:05.991+0000 7fe76d3f2640 -1 osd.1 237 handle_osd_map injecting map crc failure 2024-09-13T11:27:06.067 INFO:tasks.ceph.osd.4.smithi001.stderr:2024-09-13T11:27:06.066+0000 7ff8e0218640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26310) UID: 0 2024-09-13T11:27:06.110 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-13T11:27:06.110 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [], 2024-09-13T11:27:06.110 INFO:teuthology.orchestra.run.smithi162.stdout: "num_ops": 0 2024-09-13T11:27:06.110 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-13T11:27:06.129 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-09-13T11:27:06.157 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-13T11:27:06.154+0000 7f6a68775640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26307) UID: 0 2024-09-13T11:27:06.258 INFO:tasks.ceph.osd.14.smithi162.stderr:2024-09-13T11:27:06.255+0000 7f3df5903640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33154) UID: 0 2024-09-13T11:27:06.265 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-13T11:27:06.265 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [], 2024-09-13T11:27:06.265 INFO:teuthology.orchestra.run.smithi162.stdout: "complaint_time": 30, 2024-09-13T11:27:06.265 INFO:teuthology.orchestra.run.smithi162.stdout: "num_blocked_ops": 0 2024-09-13T11:27:06.265 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-13T11:27:06.285 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-09-13T11:27:06.359 INFO:tasks.ceph.osd.7.smithi177.stderr:2024-09-13T11:27:06.357+0000 7f897b51d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26575) UID: 0 2024-09-13T11:27:06.460 INFO:tasks.ceph.osd.11.smithi177.stderr:2024-09-13T11:27:06.457+0000 7f24792c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26580) UID: 0 2024-09-13T11:27:06.468 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-13T11:27:06.468 INFO:teuthology.orchestra.run.smithi162.stdout: "size": 20, 2024-09-13T11:27:06.468 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 600, 2024-09-13T11:27:06.468 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [ 2024-09-13T11:27:06.468 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:06.468 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:45810 3.17 3.9bd13e57 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e69)", 2024-09-13T11:27:06.468 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:21.347960+0000", 2024-09-13T11:27:06.468 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 225.119068269, 2024-09-13T11:27:06.468 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.56814464200000003, 2024-09-13T11:27:06.468 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:27:06.469 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:27:06.469 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-09-13T11:27:06.469 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:27:06.469 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:27:06.469 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:27:06.469 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 45810 2024-09-13T11:27:06.469 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:06.469 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:27:06.469 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:06.469 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:27:06.469 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:21.347960+0000", 2024-09-13T11:27:06.469 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:06.469 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:06.470 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:06.470 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:27:06.470 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:21.347960+0000", 2024-09-13T11:27:06.470 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:06.470 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:06.470 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:06.470 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:27:06.470 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:21.347965+0000", 2024-09-13T11:27:06.470 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.6720000000000003e-06 2024-09-13T11:27:06.470 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:06.470 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:06.470 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:27:06.470 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:21.348019+0000", 2024-09-13T11:27:06.471 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.4391000000000002e-05 2024-09-13T11:27:06.471 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:06.471 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:06.471 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:27:06.471 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:21.348052+0000", 2024-09-13T11:27:06.471 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.2246000000000003e-05 2024-09-13T11:27:06.471 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:06.471 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:06.471 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:27:06.471 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:21.348072+0000", 2024-09-13T11:27:06.471 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.0622e-05 2024-09-13T11:27:06.471 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:06.471 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:06.471 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:27:06.472 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:21.348278+0000", 2024-09-13T11:27:06.472 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000205391 2024-09-13T11:27:06.472 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:06.472 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:06.472 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-09-13T11:27:06.472 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:21.348292+0000", 2024-09-13T11:27:06.472 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4759e-05 2024-09-13T11:27:06.472 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:06.472 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:06.472 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:27:06.472 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:21.916064+0000", 2024-09-13T11:27:06.472 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.56777128600000004 2024-09-13T11:27:06.472 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:06.473 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:06.473 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:27:06.473 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:21.916105+0000", 2024-09-13T11:27:06.473 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.1275e-05 2024-09-13T11:27:06.473 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:06.473 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:27:06.473 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:06.473 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:06.473 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:06.473 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:54653 3.23 3:c7a54800:::benchmark_data_smithi177_31116_object54652:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e75)", 2024-09-13T11:27:06.473 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:30.979059+0000", 2024-09-13T11:27:06.473 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 215.48796977000001, 2024-09-13T11:27:06.474 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.97284747100000002, 2024-09-13T11:27:06.474 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:27:06.474 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:27:06.474 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-09-13T11:27:06.474 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:27:06.474 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:27:06.474 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:27:06.474 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 54653 2024-09-13T11:27:06.474 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:06.474 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:27:06.474 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:06.474 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:27:06.474 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.979059+0000", 2024-09-13T11:27:06.475 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:06.475 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:06.475 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:06.475 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:27:06.475 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.979059+0000", 2024-09-13T11:27:06.475 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:06.475 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:06.475 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:06.475 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:27:06.475 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.979061+0000", 2024-09-13T11:27:06.475 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.4279999999999999e-06 2024-09-13T11:27:06.475 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:06.475 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:06.475 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:27:06.476 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.979114+0000", 2024-09-13T11:27:06.476 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.3254000000000002e-05 2024-09-13T11:27:06.476 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:06.476 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:06.476 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:27:06.476 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.979157+0000", 2024-09-13T11:27:06.476 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.2018000000000003e-05 2024-09-13T11:27:06.476 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:06.476 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:06.476 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:27:06.476 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.979172+0000", 2024-09-13T11:27:06.476 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5636000000000001e-05 2024-09-13T11:27:06.476 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:06.476 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:06.477 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:27:06.477 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:31.045109+0000", 2024-09-13T11:27:06.477 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.065936982000000005 2024-09-13T11:27:06.477 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:06.477 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:06.477 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-09-13T11:27:06.477 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:31.045121+0000", 2024-09-13T11:27:06.477 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1608e-05 2024-09-13T11:27:06.477 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:06.477 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:06.477 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:27:06.477 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:31.951636+0000", 2024-09-13T11:27:06.478 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.90651503899999997 2024-09-13T11:27:06.478 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:06.478 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:06.478 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:27:06.478 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:31.951906+0000", 2024-09-13T11:27:06.478 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000270506 2024-09-13T11:27:06.478 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:06.478 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:27:06.478 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:06.478 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:06.478 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:06.478 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPush(3.18 76/70 [PushOp(3:1a32c9cd:::benchmark_data_smithi177_31116_object12833:head, version: 49'818, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:1a32c9cd:::benchmark_data_smithi177_31116_object12833:head@49'818, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:27:06.478 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:32.522305+0000", 2024-09-13T11:27:06.479 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 213.944723686, 2024-09-13T11:27:06.479 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.40201207100000003, 2024-09-13T11:27:06.479 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:27:06.479 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:27:06.479 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:27:06.479 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:27:06.479 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:06.479 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:27:06.479 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.522305+0000", 2024-09-13T11:27:06.479 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:06.479 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:06.479 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:06.479 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:27:06.480 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.522305+0000", 2024-09-13T11:27:06.480 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:06.480 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:06.480 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:06.480 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:27:06.480 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.522309+0000", 2024-09-13T11:27:06.480 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.9829999999999998e-06 2024-09-13T11:27:06.480 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:06.480 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:06.480 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:27:06.480 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.522377+0000", 2024-09-13T11:27:06.480 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.8069000000000002e-05 2024-09-13T11:27:06.480 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:06.480 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:06.481 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:27:06.481 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.921950+0000", 2024-09-13T11:27:06.481 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.399573438 2024-09-13T11:27:06.481 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:06.481 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:06.481 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:27:06.481 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.921983+0000", 2024-09-13T11:27:06.481 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.2858000000000003e-05 2024-09-13T11:27:06.481 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:06.481 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:06.481 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:27:06.481 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.922389+0000", 2024-09-13T11:27:06.481 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00040612999999999999 2024-09-13T11:27:06.482 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:06.482 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:06.482 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:27:06.482 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.922421+0000", 2024-09-13T11:27:06.482 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.1356999999999998e-05 2024-09-13T11:27:06.482 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:06.482 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:06.482 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:27:06.482 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.924317+0000", 2024-09-13T11:27:06.482 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0018962359999999999 2024-09-13T11:27:06.482 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:06.482 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:27:06.482 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:06.482 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:06.483 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:06.483 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:57435 3.3 3:c0db4777:::benchmark_data_smithi177_31116_object57434:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e80)", 2024-09-13T11:27:06.483 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:37.436034+0000", 2024-09-13T11:27:06.483 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 209.03099477800001, 2024-09-13T11:27:06.483 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.35949142000000001, 2024-09-13T11:27:06.483 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:27:06.483 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:27:06.483 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:27:06.483 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:27:06.483 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:27:06.483 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:27:06.483 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 57435 2024-09-13T11:27:06.483 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:06.484 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:27:06.484 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:06.484 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:27:06.484 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:37.436034+0000", 2024-09-13T11:27:06.484 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:06.484 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:06.484 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:06.484 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:27:06.484 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:37.436034+0000", 2024-09-13T11:27:06.484 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:06.484 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:06.484 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:06.484 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:27:06.484 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:37.436037+0000", 2024-09-13T11:27:06.485 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.7829999999999999e-06 2024-09-13T11:27:06.485 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:06.485 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:06.485 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:27:06.485 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:37.436090+0000", 2024-09-13T11:27:06.485 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.3665999999999997e-05 2024-09-13T11:27:06.485 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:06.485 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:06.485 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:27:06.485 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:37.436112+0000", 2024-09-13T11:27:06.485 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.1556000000000002e-05 2024-09-13T11:27:06.485 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:06.485 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:06.486 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:27:06.486 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:37.436122+0000", 2024-09-13T11:27:06.486 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0098000000000001e-05 2024-09-13T11:27:06.486 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:06.486 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:06.486 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:27:06.486 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:37.436187+0000", 2024-09-13T11:27:06.486 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.4857999999999995e-05 2024-09-13T11:27:06.486 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:06.486 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:06.486 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:27:06.486 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:37.436285+0000", 2024-09-13T11:27:06.486 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.8643999999999999e-05 2024-09-13T11:27:06.487 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:06.487 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:06.487 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 7", 2024-09-13T11:27:06.487 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:37.436423+0000", 2024-09-13T11:27:06.487 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00013809299999999999 2024-09-13T11:27:06.487 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:06.487 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:06.487 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:27:06.487 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:37.440125+0000", 2024-09-13T11:27:06.487 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0037015239999999999 2024-09-13T11:27:06.487 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:06.487 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:06.487 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:27:06.487 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:37.795397+0000", 2024-09-13T11:27:06.488 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.35527234400000002 2024-09-13T11:27:06.488 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:06.488 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:06.488 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:27:06.488 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:37.795485+0000", 2024-09-13T11:27:06.488 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.7538000000000001e-05 2024-09-13T11:27:06.488 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:06.488 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:06.488 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:27:06.488 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:37.795525+0000", 2024-09-13T11:27:06.488 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.0315999999999998e-05 2024-09-13T11:27:06.488 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:06.488 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:27:06.489 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:06.489 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:06.489 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:06.489 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:60433 3.3 3:c01f7ddc:::benchmark_data_smithi177_31116_object60432:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e85)", 2024-09-13T11:27:06.489 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:42.199382+0000", 2024-09-13T11:27:06.489 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 204.26764688700001, 2024-09-13T11:27:06.489 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.33918965200000001, 2024-09-13T11:27:06.489 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:27:06.489 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:27:06.489 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:27:06.489 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:27:06.489 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:27:06.489 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:27:06.490 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 60433 2024-09-13T11:27:06.490 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:06.490 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:27:06.490 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:06.490 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:27:06.490 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.199382+0000", 2024-09-13T11:27:06.490 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:06.490 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:06.490 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:06.490 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:27:06.490 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.199382+0000", 2024-09-13T11:27:06.490 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:06.490 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:06.490 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:06.491 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:27:06.491 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.199386+0000", 2024-09-13T11:27:06.491 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.8759999999999998e-06 2024-09-13T11:27:06.491 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:06.491 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:06.491 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:27:06.491 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.199455+0000", 2024-09-13T11:27:06.491 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.9305000000000001e-05 2024-09-13T11:27:06.491 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:06.491 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:06.491 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:27:06.491 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.199482+0000", 2024-09-13T11:27:06.491 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.6749000000000001e-05 2024-09-13T11:27:06.491 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:06.492 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:06.492 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:27:06.492 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.199503+0000", 2024-09-13T11:27:06.492 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.1137000000000001e-05 2024-09-13T11:27:06.492 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:06.492 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:06.492 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:27:06.492 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.199602+0000", 2024-09-13T11:27:06.492 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.8743e-05 2024-09-13T11:27:06.492 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:06.492 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:06.492 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:27:06.492 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.199784+0000", 2024-09-13T11:27:06.493 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00018265100000000001 2024-09-13T11:27:06.493 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:06.493 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:06.493 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 7", 2024-09-13T11:27:06.493 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.199955+0000", 2024-09-13T11:27:06.493 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00017067100000000001 2024-09-13T11:27:06.493 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:06.493 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:06.493 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:27:06.493 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.202075+0000", 2024-09-13T11:27:06.493 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0021206749999999998 2024-09-13T11:27:06.493 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:06.493 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:06.493 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:27:06.494 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.538395+0000", 2024-09-13T11:27:06.494 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.33631963599999998 2024-09-13T11:27:06.494 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:06.494 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:06.494 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:27:06.494 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.538498+0000", 2024-09-13T11:27:06.494 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00010267099999999999 2024-09-13T11:27:06.494 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:06.494 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:06.494 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:27:06.494 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.538571+0000", 2024-09-13T11:27:06.494 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.3538e-05 2024-09-13T11:27:06.494 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:06.494 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:27:06.495 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:06.495 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:06.495 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:06.495 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop_reply(client.4665.0:60433 3.3 e85/75 ondisk, result = 0)", 2024-09-13T11:27:06.495 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:42.203186+0000", 2024-09-13T11:27:06.495 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 204.26384270899999, 2024-09-13T11:27:06.495 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.33540096699999999, 2024-09-13T11:27:06.495 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:27:06.495 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:27:06.495 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:27:06.495 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:27:06.495 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:06.495 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:27:06.495 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.203186+0000", 2024-09-13T11:27:06.496 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:06.496 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:06.496 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:06.496 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:27:06.496 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.203186+0000", 2024-09-13T11:27:06.496 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:06.496 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:06.496 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:06.496 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:27:06.496 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.203194+0000", 2024-09-13T11:27:06.496 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.0020000000000006e-06 2024-09-13T11:27:06.496 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:06.497 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:06.497 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:27:06.497 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.203202+0000", 2024-09-13T11:27:06.497 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.9829999999999996e-06 2024-09-13T11:27:06.497 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:06.497 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:06.497 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:27:06.497 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.538182+0000", 2024-09-13T11:27:06.497 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.33498029899999998 2024-09-13T11:27:06.497 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:06.497 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:06.497 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:27:06.498 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.538196+0000", 2024-09-13T11:27:06.498 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3807e-05 2024-09-13T11:27:06.498 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:06.498 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:06.498 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:27:06.498 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.538371+0000", 2024-09-13T11:27:06.498 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00017541200000000001 2024-09-13T11:27:06.498 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:06.498 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:06.498 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:27:06.498 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.538386+0000", 2024-09-13T11:27:06.498 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4833e-05 2024-09-13T11:27:06.498 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:06.498 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:06.499 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:27:06.499 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.538587+0000", 2024-09-13T11:27:06.499 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000200631 2024-09-13T11:27:06.499 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:06.499 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:27:06.499 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:06.499 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:06.499 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:06.499 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:83144 3.18 3:1c2a5085:::benchmark_data_smithi177_31116_object83143:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106)", 2024-09-13T11:27:06.499 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:24:04.186395+0000", 2024-09-13T11:27:06.499 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 182.28063359000001, 2024-09-13T11:27:06.499 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.29655527599999998, 2024-09-13T11:27:06.499 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:27:06.500 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:27:06.500 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:27:06.500 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:27:06.500 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:27:06.500 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:27:06.500 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 83144 2024-09-13T11:27:06.500 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:06.500 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:27:06.500 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:06.500 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:27:06.500 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.186395+0000", 2024-09-13T11:27:06.500 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:06.501 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:06.501 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:06.501 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:27:06.501 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.186395+0000", 2024-09-13T11:27:06.501 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:06.501 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:06.501 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:06.501 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:27:06.501 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.186398+0000", 2024-09-13T11:27:06.501 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.5299999999999999e-06 2024-09-13T11:27:06.501 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:06.501 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:06.501 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:27:06.502 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.186438+0000", 2024-09-13T11:27:06.502 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.0293000000000002e-05 2024-09-13T11:27:06.502 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:06.502 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:06.502 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:27:06.502 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.186450+0000", 2024-09-13T11:27:06.502 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2687000000000001e-05 2024-09-13T11:27:06.502 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:06.502 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:06.502 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:27:06.502 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.186456+0000", 2024-09-13T11:27:06.502 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.446e-06 2024-09-13T11:27:06.502 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:06.502 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:06.503 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:27:06.503 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.186503+0000", 2024-09-13T11:27:06.503 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.7491000000000003e-05 2024-09-13T11:27:06.503 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:06.503 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:06.503 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:27:06.503 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.186618+0000", 2024-09-13T11:27:06.503 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000114999 2024-09-13T11:27:06.503 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:06.503 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:06.503 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 0", 2024-09-13T11:27:06.503 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.186712+0000", 2024-09-13T11:27:06.504 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.3805000000000001e-05 2024-09-13T11:27:06.504 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:06.504 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:06.504 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:27:06.504 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.189753+0000", 2024-09-13T11:27:06.504 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.003040713 2024-09-13T11:27:06.504 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:06.504 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:06.504 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:27:06.504 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.482662+0000", 2024-09-13T11:27:06.504 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.29290945699999998 2024-09-13T11:27:06.504 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:06.504 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:06.505 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:27:06.505 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.482831+0000", 2024-09-13T11:27:06.505 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00016900699999999999 2024-09-13T11:27:06.505 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:06.505 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:06.505 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:27:06.505 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.482950+0000", 2024-09-13T11:27:06.505 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000118848 2024-09-13T11:27:06.505 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:06.505 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:27:06.505 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:06.505 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:06.505 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:06.505 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop_reply(client.4665.0:83144 3.18 e106/91 ondisk, result = 0)", 2024-09-13T11:27:06.506 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:24:04.188413+0000", 2024-09-13T11:27:06.506 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 182.27861558000001, 2024-09-13T11:27:06.506 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.294557562, 2024-09-13T11:27:06.506 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:27:06.506 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:27:06.506 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:27:06.506 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:27:06.506 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:06.506 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:27:06.506 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.188413+0000", 2024-09-13T11:27:06.506 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:06.506 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:06.506 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:06.507 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:27:06.507 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.188413+0000", 2024-09-13T11:27:06.507 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:06.507 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:06.507 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:06.507 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:27:06.507 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.188414+0000", 2024-09-13T11:27:06.507 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4309999999999999e-06 2024-09-13T11:27:06.507 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:06.507 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:06.507 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:27:06.507 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.188419+0000", 2024-09-13T11:27:06.507 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.2889999999999999e-06 2024-09-13T11:27:06.507 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:06.508 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:06.508 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:27:06.508 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.482442+0000", 2024-09-13T11:27:06.508 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.294023168 2024-09-13T11:27:06.508 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:06.508 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:06.508 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:27:06.508 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.482459+0000", 2024-09-13T11:27:06.508 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.7013000000000001e-05 2024-09-13T11:27:06.508 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:06.508 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:06.508 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:27:06.509 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.482615+0000", 2024-09-13T11:27:06.509 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00015620400000000001 2024-09-13T11:27:06.509 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:06.509 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:06.509 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:27:06.509 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.482644+0000", 2024-09-13T11:27:06.509 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.9093000000000001e-05 2024-09-13T11:27:06.509 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:06.509 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:06.509 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:27:06.509 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.482971+0000", 2024-09-13T11:27:06.509 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00032636399999999999 2024-09-13T11:27:06.509 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:06.510 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:27:06.510 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:06.510 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:06.510 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:06.510 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPush(3.1f 106/78 [PushOp(3:fd4be819:::benchmark_data_smithi177_31116_object54560:head, version: 74'3410, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:fd4be819:::benchmark_data_smithi177_31116_object54560:head@74'3410, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:27:06.510 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:24:04.397161+0000", 2024-09-13T11:27:06.510 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 182.069867171, 2024-09-13T11:27:06.510 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.51611096199999995, 2024-09-13T11:27:06.510 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:27:06.510 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:27:06.510 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:27:06.510 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:27:06.510 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:06.510 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:27:06.511 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.397161+0000", 2024-09-13T11:27:06.511 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:06.511 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:06.511 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:06.511 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:27:06.511 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.397161+0000", 2024-09-13T11:27:06.511 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:06.511 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:06.511 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:06.511 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:27:06.511 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.397163+0000", 2024-09-13T11:27:06.511 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4720000000000001e-06 2024-09-13T11:27:06.511 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:06.512 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:06.512 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:27:06.512 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.397228+0000", 2024-09-13T11:27:06.512 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.5141999999999994e-05 2024-09-13T11:27:06.512 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:06.512 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:06.512 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:27:06.512 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.911583+0000", 2024-09-13T11:27:06.512 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.51435473700000001 2024-09-13T11:27:06.512 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:06.512 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:06.512 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:27:06.512 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.911615+0000", 2024-09-13T11:27:06.513 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.2620000000000003e-05 2024-09-13T11:27:06.513 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:06.513 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:06.513 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:27:06.513 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.911759+0000", 2024-09-13T11:27:06.513 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00014376300000000001 2024-09-13T11:27:06.513 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:06.513 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:06.513 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:27:06.513 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.911779+0000", 2024-09-13T11:27:06.513 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.0003000000000001e-05 2024-09-13T11:27:06.513 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:06.513 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:06.513 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:27:06.514 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.913272+0000", 2024-09-13T11:27:06.514 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0014932249999999999 2024-09-13T11:27:06.514 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:06.514 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:27:06.514 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:06.514 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:06.514 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:06.514 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPush(3.2a 121/120 [PushOp(3:5595108b:::benchmark_data_smithi177_31116_object899:head, version: 43'62, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:5595108b:::benchmark_data_smithi177_31116_object899:head@43'62, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:27:06.514 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:24:27.352355+0000", 2024-09-13T11:27:06.514 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 159.11467387600001, 2024-09-13T11:27:06.514 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.61628666099999996, 2024-09-13T11:27:06.514 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:27:06.514 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:27:06.515 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:27:06.515 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:27:06.515 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:06.515 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:27:06.515 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:27.352355+0000", 2024-09-13T11:27:06.515 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:06.515 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:06.515 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:06.515 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:27:06.515 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:27.352355+0000", 2024-09-13T11:27:06.515 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:06.515 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:06.515 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:06.516 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:27:06.516 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:27.352357+0000", 2024-09-13T11:27:06.516 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.328e-06 2024-09-13T11:27:06.516 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:06.516 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:06.516 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:27:06.516 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:27.352428+0000", 2024-09-13T11:27:06.516 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.1118999999999995e-05 2024-09-13T11:27:06.516 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:06.516 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:06.516 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:27:06.516 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:27.966950+0000", 2024-09-13T11:27:06.516 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.61452202499999997 2024-09-13T11:27:06.517 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:06.517 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:06.517 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:27:06.517 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:27.966980+0000", 2024-09-13T11:27:06.517 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.9433000000000001e-05 2024-09-13T11:27:06.517 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:06.517 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:06.517 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:27:06.517 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:27.967220+0000", 2024-09-13T11:27:06.517 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00024065499999999999 2024-09-13T11:27:06.517 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:06.517 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:06.517 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:27:06.517 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:27.967244+0000", 2024-09-13T11:27:06.518 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.4216e-05 2024-09-13T11:27:06.518 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:06.518 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:06.518 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:27:06.518 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:27.968641+0000", 2024-09-13T11:27:06.518 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0013968850000000001 2024-09-13T11:27:06.518 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:06.518 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:27:06.518 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:06.518 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:06.518 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:06.518 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:162073 3.1f 3:fd1a00ae:::benchmark_data_smithi177_31116_object21770:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e156)", 2024-09-13T11:27:06.518 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:25:15.454353+0000", 2024-09-13T11:27:06.519 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 111.012675278, 2024-09-13T11:27:06.519 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.27516647, 2024-09-13T11:27:06.519 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:27:06.519 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:27:06.519 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:27:06.519 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:27:06.519 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:27:06.519 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:27:06.519 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 162073 2024-09-13T11:27:06.519 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:06.519 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:27:06.519 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:06.519 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:27:06.519 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:15.454353+0000", 2024-09-13T11:27:06.519 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:06.520 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:06.520 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:06.520 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:27:06.520 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:15.454353+0000", 2024-09-13T11:27:06.520 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:06.520 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:06.520 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:06.520 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:27:06.520 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:15.454358+0000", 2024-09-13T11:27:06.520 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.3499999999999999e-06 2024-09-13T11:27:06.520 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:06.520 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:06.520 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:27:06.521 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:15.454371+0000", 2024-09-13T11:27:06.521 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3594000000000001e-05 2024-09-13T11:27:06.521 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:06.521 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:06.521 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:27:06.521 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:15.454408+0000", 2024-09-13T11:27:06.521 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.6619999999999998e-05 2024-09-13T11:27:06.521 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:06.521 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:06.521 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:27:06.521 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:15.454431+0000", 2024-09-13T11:27:06.521 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.3546000000000001e-05 2024-09-13T11:27:06.521 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:06.521 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:06.522 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:27:06.522 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:15.454587+0000", 2024-09-13T11:27:06.522 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000155248 2024-09-13T11:27:06.522 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:06.522 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:06.522 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:27:06.522 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:15.454897+0000", 2024-09-13T11:27:06.522 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000309955 2024-09-13T11:27:06.522 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:06.522 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:06.522 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 5", 2024-09-13T11:27:06.522 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:15.455203+0000", 2024-09-13T11:27:06.522 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00030604999999999999 2024-09-13T11:27:06.522 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:06.523 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:06.523 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:27:06.523 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:15.460534+0000", 2024-09-13T11:27:06.523 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0053310199999999997 2024-09-13T11:27:06.523 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:06.523 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:06.523 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:27:06.523 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:15.729356+0000", 2024-09-13T11:27:06.523 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.26882267199999998 2024-09-13T11:27:06.523 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:06.523 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:06.523 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:27:06.523 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:15.729442+0000", 2024-09-13T11:27:06.524 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.5209000000000004e-05 2024-09-13T11:27:06.524 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:06.524 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:06.524 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:27:06.524 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:15.729520+0000", 2024-09-13T11:27:06.524 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.8206000000000003e-05 2024-09-13T11:27:06.524 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:06.524 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:27:06.524 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:06.524 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:06.524 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:06.524 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:162102 3.1f 3:fc0cda6c:::benchmark_data_smithi177_31116_object21799:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e156)", 2024-09-13T11:27:06.524 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:25:15.477044+0000", 2024-09-13T11:27:06.525 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 110.989984424, 2024-09-13T11:27:06.525 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.28798895800000002, 2024-09-13T11:27:06.525 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:27:06.525 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:27:06.525 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:27:06.525 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:27:06.525 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:27:06.525 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:27:06.525 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 162102 2024-09-13T11:27:06.525 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:06.525 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:27:06.525 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:06.525 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:27:06.525 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:15.477044+0000", 2024-09-13T11:27:06.526 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:06.526 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:06.526 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:06.526 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:27:06.526 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:15.477044+0000", 2024-09-13T11:27:06.526 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:06.526 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:06.526 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:06.526 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:27:06.526 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:15.477048+0000", 2024-09-13T11:27:06.526 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.3569999999999998e-06 2024-09-13T11:27:06.526 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:06.526 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:06.527 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:27:06.527 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:15.477058+0000", 2024-09-13T11:27:06.527 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.9899999999999992e-06 2024-09-13T11:27:06.527 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:06.527 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:06.527 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:27:06.527 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:15.477087+0000", 2024-09-13T11:27:06.527 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.9026999999999998e-05 2024-09-13T11:27:06.527 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:06.527 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:06.527 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:27:06.527 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:15.477101+0000", 2024-09-13T11:27:06.527 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4583000000000001e-05 2024-09-13T11:27:06.527 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:06.528 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:06.528 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:27:06.528 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:15.477212+0000", 2024-09-13T11:27:06.528 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00011071200000000001 2024-09-13T11:27:06.528 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:06.528 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:06.528 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:27:06.528 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:15.477492+0000", 2024-09-13T11:27:06.528 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00027995400000000001 2024-09-13T11:27:06.528 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:06.528 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:06.528 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 5", 2024-09-13T11:27:06.528 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:15.477682+0000", 2024-09-13T11:27:06.529 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000189929 2024-09-13T11:27:06.529 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:06.529 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:06.529 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:27:06.529 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:15.481903+0000", 2024-09-13T11:27:06.529 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0042214649999999998 2024-09-13T11:27:06.529 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:06.529 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:06.529 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:27:06.529 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:15.764891+0000", 2024-09-13T11:27:06.529 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.28298753599999998 2024-09-13T11:27:06.529 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:06.529 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:06.530 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:27:06.530 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:15.764972+0000", 2024-09-13T11:27:06.530 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.1230000000000007e-05 2024-09-13T11:27:06.530 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:06.530 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:06.530 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:27:06.530 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:15.765033+0000", 2024-09-13T11:27:06.530 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.1174999999999996e-05 2024-09-13T11:27:06.530 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:06.530 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:27:06.531 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:06.531 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:06.531 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:06.531 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:221532 3.17 3:ec6fb681:::benchmark_data_smithi177_31116_object81229:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e191)", 2024-09-13T11:27:06.531 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:26:04.303286+0000", 2024-09-13T11:27:06.531 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 62.163742677000002, 2024-09-13T11:27:06.531 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.79092058600000004, 2024-09-13T11:27:06.531 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:27:06.531 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:27:06.531 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:27:06.531 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:27:06.531 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:27:06.531 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:27:06.532 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 221532 2024-09-13T11:27:06.532 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:06.532 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:27:06.532 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:06.532 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:27:06.532 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:04.303286+0000", 2024-09-13T11:27:06.532 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:06.532 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:06.532 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:06.532 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:27:06.532 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:04.303286+0000", 2024-09-13T11:27:06.532 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:06.532 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:06.532 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:06.533 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:27:06.533 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:04.303290+0000", 2024-09-13T11:27:06.533 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.9600000000000002e-06 2024-09-13T11:27:06.533 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:06.533 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:06.533 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:27:06.533 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:04.303302+0000", 2024-09-13T11:27:06.533 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2126e-05 2024-09-13T11:27:06.533 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:06.533 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:06.533 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:27:06.533 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:04.303332+0000", 2024-09-13T11:27:06.533 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.0015999999999999e-05 2024-09-13T11:27:06.534 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:06.534 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:06.534 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:27:06.534 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:04.303347+0000", 2024-09-13T11:27:06.534 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5324000000000001e-05 2024-09-13T11:27:06.534 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:06.534 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:06.534 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:27:06.534 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:04.303484+0000", 2024-09-13T11:27:06.534 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00013702199999999999 2024-09-13T11:27:06.534 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:06.534 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:06.534 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:27:06.534 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:04.303700+0000", 2024-09-13T11:27:06.535 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00021570500000000001 2024-09-13T11:27:06.535 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:06.535 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:06.535 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 3", 2024-09-13T11:27:06.535 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:04.303868+0000", 2024-09-13T11:27:06.535 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00016751799999999999 2024-09-13T11:27:06.535 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:06.535 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:06.535 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:27:06.535 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:04.309489+0000", 2024-09-13T11:27:06.535 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.005621455 2024-09-13T11:27:06.535 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:06.535 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:06.535 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:27:06.536 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:05.094014+0000", 2024-09-13T11:27:06.536 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.78452480499999999 2024-09-13T11:27:06.536 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:06.536 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:06.536 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:27:06.536 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:05.094132+0000", 2024-09-13T11:27:06.536 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000118093 2024-09-13T11:27:06.536 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:06.536 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:06.536 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:27:06.536 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:05.094206+0000", 2024-09-13T11:27:06.536 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.4561999999999995e-05 2024-09-13T11:27:06.536 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:06.537 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:27:06.537 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:06.537 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:06.537 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:06.537 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop_reply(client.4665.0:221532 3.17 e191/85 ondisk, result = 0)", 2024-09-13T11:27:06.537 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:26:04.309386+0000", 2024-09-13T11:27:06.537 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 62.157642373999998, 2024-09-13T11:27:06.537 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.78483394299999998, 2024-09-13T11:27:06.537 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:27:06.537 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:27:06.537 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:27:06.537 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:27:06.537 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:06.538 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:27:06.538 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:04.309386+0000", 2024-09-13T11:27:06.538 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:06.538 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:06.538 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:06.538 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:27:06.538 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:04.309386+0000", 2024-09-13T11:27:06.538 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:06.538 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:06.538 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:06.538 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:27:06.538 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:04.309390+0000", 2024-09-13T11:27:06.538 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.5700000000000001e-06 2024-09-13T11:27:06.538 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:06.538 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:06.539 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:27:06.539 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:04.309403+0000", 2024-09-13T11:27:06.539 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3567000000000001e-05 2024-09-13T11:27:06.539 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:06.539 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:06.539 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:27:06.539 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:05.093783+0000", 2024-09-13T11:27:06.539 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.78437944699999995 2024-09-13T11:27:06.539 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:06.539 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:06.539 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:27:06.539 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:05.093802+0000", 2024-09-13T11:27:06.539 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.8816000000000001e-05 2024-09-13T11:27:06.539 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:06.540 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:06.540 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:27:06.540 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:05.093968+0000", 2024-09-13T11:27:06.540 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000166572 2024-09-13T11:27:06.540 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:06.540 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:06.540 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:27:06.540 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:05.093999+0000", 2024-09-13T11:27:06.540 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.0352999999999999e-05 2024-09-13T11:27:06.540 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:06.540 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:06.540 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:27:06.540 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:05.094220+0000", 2024-09-13T11:27:06.540 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000221618 2024-09-13T11:27:06.541 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:06.541 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:27:06.541 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:06.541 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:06.541 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:06.541 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPushReply(3.1f 219/192 [PushReplyOp(3:f85f8546:::benchmark_data_smithi177_31116_object127445:head)])", 2024-09-13T11:27:06.541 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:26:42.440545+0000", 2024-09-13T11:27:06.541 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 24.026483824, 2024-09-13T11:27:06.541 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.49957432099999999, 2024-09-13T11:27:06.541 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:27:06.541 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:27:06.541 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-09-13T11:27:06.541 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:27:06.541 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:06.542 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:27:06.542 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:42.440545+0000", 2024-09-13T11:27:06.542 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:06.542 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:06.542 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:06.542 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:27:06.542 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:42.440545+0000", 2024-09-13T11:27:06.542 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:06.542 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:06.542 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:06.542 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:27:06.542 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:42.440545+0000", 2024-09-13T11:27:06.542 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.4300000000000003e-07 2024-09-13T11:27:06.542 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:06.543 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:06.543 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:27:06.543 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:42.440552+0000", 2024-09-13T11:27:06.543 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.4829999999999996e-06 2024-09-13T11:27:06.543 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:06.543 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:06.543 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:27:06.543 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:42.939600+0000", 2024-09-13T11:27:06.543 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.49904801300000001 2024-09-13T11:27:06.543 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:06.543 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:06.543 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:27:06.543 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:42.939620+0000", 2024-09-13T11:27:06.544 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.0305999999999998e-05 2024-09-13T11:27:06.544 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:06.544 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:06.544 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:27:06.544 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:42.939773+0000", 2024-09-13T11:27:06.544 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00015300599999999999 2024-09-13T11:27:06.544 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:06.544 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:06.544 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:27:06.544 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:42.940119+0000", 2024-09-13T11:27:06.544 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00034597000000000002 2024-09-13T11:27:06.544 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:06.544 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:27:06.545 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:06.545 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:06.545 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:06.545 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:266723 3.1f 3:f8daff18:::benchmark_data_smithi177_31116_object126420:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e223)", 2024-09-13T11:27:06.545 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:26:48.978181+0000", 2024-09-13T11:27:06.545 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 17.488847891999999, 2024-09-13T11:27:06.545 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.27412631599999998, 2024-09-13T11:27:06.545 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:27:06.545 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:27:06.545 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:27:06.545 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:27:06.545 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:27:06.545 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:27:06.546 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 266723 2024-09-13T11:27:06.546 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:06.546 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:27:06.546 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:06.546 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:27:06.546 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:48.978181+0000", 2024-09-13T11:27:06.546 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:06.546 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:06.546 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:06.546 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:27:06.546 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:48.978181+0000", 2024-09-13T11:27:06.546 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:06.546 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:06.546 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:06.547 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:27:06.547 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:48.978182+0000", 2024-09-13T11:27:06.547 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.72e-06 2024-09-13T11:27:06.547 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:06.547 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:06.547 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:27:06.547 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:48.978189+0000", 2024-09-13T11:27:06.547 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.9700000000000002e-06 2024-09-13T11:27:06.547 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:06.547 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:06.547 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:27:06.547 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:48.978205+0000", 2024-09-13T11:27:06.547 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5746000000000001e-05 2024-09-13T11:27:06.548 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:06.548 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:06.548 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:27:06.548 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:48.978221+0000", 2024-09-13T11:27:06.548 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5679999999999999e-05 2024-09-13T11:27:06.548 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:06.548 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:06.548 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:27:06.548 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:48.978961+0000", 2024-09-13T11:27:06.548 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0007406 2024-09-13T11:27:06.548 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:06.548 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:06.548 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-09-13T11:27:06.549 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:48.978978+0000", 2024-09-13T11:27:06.549 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.6663e-05 2024-09-13T11:27:06.549 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:06.549 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:06.549 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:27:06.549 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:49.183721+0000", 2024-09-13T11:27:06.549 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.20474339799999999 2024-09-13T11:27:06.549 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:06.549 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:06.549 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:27:06.549 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:49.183856+0000", 2024-09-13T11:27:06.549 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00013502300000000001 2024-09-13T11:27:06.549 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:06.549 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:06.550 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 14", 2024-09-13T11:27:06.550 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:49.183980+0000", 2024-09-13T11:27:06.550 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000123473 2024-09-13T11:27:06.550 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:06.550 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:06.550 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:27:06.550 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:49.194643+0000", 2024-09-13T11:27:06.550 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.010662914000000001 2024-09-13T11:27:06.550 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:06.550 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:06.550 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:27:06.550 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:49.252225+0000", 2024-09-13T11:27:06.550 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.057581637999999997 2024-09-13T11:27:06.551 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:06.551 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:06.551 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:27:06.551 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:49.252273+0000", 2024-09-13T11:27:06.551 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.8158000000000002e-05 2024-09-13T11:27:06.551 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:06.551 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:06.551 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:27:06.551 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:49.252307+0000", 2024-09-13T11:27:06.551 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.4332999999999998e-05 2024-09-13T11:27:06.551 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:06.551 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:27:06.551 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:06.551 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:06.552 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:06.552 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:266816 3.1f 3:f851e4b5:::benchmark_data_smithi177_31116_object126513:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e223)", 2024-09-13T11:27:06.552 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:26:48.978237+0000", 2024-09-13T11:27:06.552 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 17.488791556999999, 2024-09-13T11:27:06.552 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.27421219200000002, 2024-09-13T11:27:06.552 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:27:06.552 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:27:06.552 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:27:06.552 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:27:06.552 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:27:06.552 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:27:06.552 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 266816 2024-09-13T11:27:06.552 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:06.553 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:27:06.553 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:06.553 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:27:06.553 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:48.978237+0000", 2024-09-13T11:27:06.553 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:06.553 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:06.553 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:06.553 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:27:06.553 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:48.978237+0000", 2024-09-13T11:27:06.553 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:06.553 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:06.553 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:06.553 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:27:06.554 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:48.978239+0000", 2024-09-13T11:27:06.554 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5340000000000001e-06 2024-09-13T11:27:06.554 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:06.554 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:06.554 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:27:06.554 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:48.978245+0000", 2024-09-13T11:27:06.554 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.8360000000000003e-06 2024-09-13T11:27:06.554 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:06.554 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:06.554 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:27:06.554 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:48.978263+0000", 2024-09-13T11:27:06.554 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.7246999999999999e-05 2024-09-13T11:27:06.554 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:06.554 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:06.555 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:27:06.555 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:48.978276+0000", 2024-09-13T11:27:06.555 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.331e-05 2024-09-13T11:27:06.555 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:06.555 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:06.555 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:27:06.555 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:48.979037+0000", 2024-09-13T11:27:06.555 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00076063699999999999 2024-09-13T11:27:06.555 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:06.555 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:06.555 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-09-13T11:27:06.555 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:48.979052+0000", 2024-09-13T11:27:06.555 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5359999999999999e-05 2024-09-13T11:27:06.556 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:06.556 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:06.556 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:27:06.556 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:49.187185+0000", 2024-09-13T11:27:06.556 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.20813316500000001 2024-09-13T11:27:06.556 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:06.556 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:06.556 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:27:06.556 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:49.187317+0000", 2024-09-13T11:27:06.556 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000132364 2024-09-13T11:27:06.556 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:06.556 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:06.556 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 14", 2024-09-13T11:27:06.557 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:49.187439+0000", 2024-09-13T11:27:06.557 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000121999 2024-09-13T11:27:06.557 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:06.557 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:06.557 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:27:06.557 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:49.194646+0000", 2024-09-13T11:27:06.557 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.007206128 2024-09-13T11:27:06.557 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:06.557 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:06.557 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:27:06.557 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:49.252363+0000", 2024-09-13T11:27:06.557 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.057717888000000002 2024-09-13T11:27:06.557 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:06.558 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:06.558 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:27:06.558 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:49.252415+0000", 2024-09-13T11:27:06.558 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.1379999999999999e-05 2024-09-13T11:27:06.558 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:06.558 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:06.558 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:27:06.558 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:49.252449+0000", 2024-09-13T11:27:06.558 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.4344000000000002e-05 2024-09-13T11:27:06.558 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:06.558 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:27:06.558 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:06.558 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:06.558 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:06.559 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:266861 3.1f 3:feb21dac:::benchmark_data_smithi177_31116_object126558:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e223)", 2024-09-13T11:27:06.559 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:26:48.978290+0000", 2024-09-13T11:27:06.559 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 17.488738203, 2024-09-13T11:27:06.559 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.27428401000000002, 2024-09-13T11:27:06.559 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:27:06.559 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:27:06.559 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:27:06.559 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:27:06.559 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:27:06.559 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:27:06.559 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 266861 2024-09-13T11:27:06.559 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:06.559 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:27:06.559 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:06.560 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:27:06.560 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:48.978290+0000", 2024-09-13T11:27:06.560 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:06.560 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:06.560 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:06.560 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:27:06.560 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:48.978290+0000", 2024-09-13T11:27:06.560 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:06.560 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:06.560 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:06.560 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:27:06.560 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:48.978292+0000", 2024-09-13T11:27:06.560 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.266e-06 2024-09-13T11:27:06.561 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:06.561 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:06.561 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:27:06.561 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:48.978301+0000", 2024-09-13T11:27:06.561 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.1200000000000008e-06 2024-09-13T11:27:06.561 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:06.561 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:06.561 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:27:06.561 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:48.978315+0000", 2024-09-13T11:27:06.561 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.452e-05 2024-09-13T11:27:06.561 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:06.561 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:06.561 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:27:06.562 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:48.978328+0000", 2024-09-13T11:27:06.562 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2743e-05 2024-09-13T11:27:06.562 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:06.562 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:06.562 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:27:06.562 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:48.979107+0000", 2024-09-13T11:27:06.562 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00077939999999999997 2024-09-13T11:27:06.562 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:06.562 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:06.562 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-09-13T11:27:06.562 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:48.979125+0000", 2024-09-13T11:27:06.562 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.7977000000000001e-05 2024-09-13T11:27:06.562 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:06.562 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:06.563 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:27:06.563 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:49.191293+0000", 2024-09-13T11:27:06.563 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.212168046 2024-09-13T11:27:06.563 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:06.563 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:06.563 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:27:06.563 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:49.191420+0000", 2024-09-13T11:27:06.563 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00012653100000000001 2024-09-13T11:27:06.563 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:06.563 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:06.563 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 14", 2024-09-13T11:27:06.563 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:49.191522+0000", 2024-09-13T11:27:06.563 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00010239900000000001 2024-09-13T11:27:06.563 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:06.564 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:06.564 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:27:06.564 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:49.194891+0000", 2024-09-13T11:27:06.564 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0033688020000000002 2024-09-13T11:27:06.564 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:06.564 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:06.564 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:27:06.564 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:49.252493+0000", 2024-09-13T11:27:06.564 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.057601806999999998 2024-09-13T11:27:06.564 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:06.564 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:06.564 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:27:06.564 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:49.252542+0000", 2024-09-13T11:27:06.565 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.8519000000000001e-05 2024-09-13T11:27:06.565 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:06.565 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:06.565 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:27:06.565 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:49.252574+0000", 2024-09-13T11:27:06.565 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.2879999999999997e-05 2024-09-13T11:27:06.565 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:06.565 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:27:06.565 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:06.565 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:06.565 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:06.565 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:266884 3.1f 3:fb121a9e:::benchmark_data_smithi177_31116_object126581:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e224)", 2024-09-13T11:27:06.565 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:26:49.275993+0000", 2024-09-13T11:27:06.565 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 17.191035232000001, 2024-09-13T11:27:06.566 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.54084032800000004, 2024-09-13T11:27:06.566 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:27:06.566 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:27:06.566 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:27:06.566 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:27:06.566 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:27:06.566 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:27:06.566 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 266884 2024-09-13T11:27:06.566 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:06.566 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:27:06.566 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:06.566 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:27:06.566 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:49.275993+0000", 2024-09-13T11:27:06.567 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:06.567 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:06.567 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:06.567 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:27:06.567 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:49.275993+0000", 2024-09-13T11:27:06.567 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:06.567 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:06.567 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:06.567 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:27:06.567 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:49.275997+0000", 2024-09-13T11:27:06.567 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.2600000000000001e-06 2024-09-13T11:27:06.567 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:06.567 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:06.567 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:27:06.568 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:49.276005+0000", 2024-09-13T11:27:06.568 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.0779999999999996e-06 2024-09-13T11:27:06.568 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:06.568 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:06.568 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:27:06.568 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:49.276022+0000", 2024-09-13T11:27:06.568 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.6849000000000001e-05 2024-09-13T11:27:06.568 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:06.568 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:06.568 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:27:06.568 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:49.276029+0000", 2024-09-13T11:27:06.568 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.6969999999999993e-06 2024-09-13T11:27:06.568 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:06.569 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:06.569 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:27:06.569 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:49.276084+0000", 2024-09-13T11:27:06.569 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.4928e-05 2024-09-13T11:27:06.569 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:06.569 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:06.569 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:27:06.569 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:49.276181+0000", 2024-09-13T11:27:06.569 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.6834000000000002e-05 2024-09-13T11:27:06.569 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:06.569 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:06.569 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 14", 2024-09-13T11:27:06.569 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:49.276251+0000", 2024-09-13T11:27:06.569 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.0028000000000001e-05 2024-09-13T11:27:06.570 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:06.570 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:06.570 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:27:06.570 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:49.281808+0000", 2024-09-13T11:27:06.570 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0055566770000000003 2024-09-13T11:27:06.570 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:06.570 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:06.570 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:27:06.570 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:49.816659+0000", 2024-09-13T11:27:06.570 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.53485174899999999 2024-09-13T11:27:06.570 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:06.570 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:06.570 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:27:06.570 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:49.816755+0000", 2024-09-13T11:27:06.571 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.5073000000000003e-05 2024-09-13T11:27:06.571 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:06.571 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:06.571 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:27:06.571 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:49.816834+0000", 2024-09-13T11:27:06.571 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.9154999999999997e-05 2024-09-13T11:27:06.571 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:06.571 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:27:06.571 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:06.571 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:06.571 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:06.571 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPush(3.34 231/227 [PushOp(3:2c60772f:::benchmark_data_smithi177_31116_object139269:head, version: 142'5805, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:2c60772f:::benchmark_data_smithi177_31116_object139269:head@142'5805, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:27:06.571 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:26:59.024261+0000", 2024-09-13T11:27:06.571 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 7.4427680509999998, 2024-09-13T11:27:06.572 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.42593106200000003, 2024-09-13T11:27:06.572 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:27:06.572 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:27:06.572 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:27:06.572 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:27:06.572 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:06.572 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:27:06.572 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:59.024261+0000", 2024-09-13T11:27:06.572 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:06.572 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:06.572 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:06.572 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:27:06.572 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:59.024261+0000", 2024-09-13T11:27:06.573 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:06.573 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:06.573 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:06.573 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:27:06.573 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:59.024263+0000", 2024-09-13T11:27:06.573 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.2000000000000001e-06 2024-09-13T11:27:06.573 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:06.573 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:06.573 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:27:06.573 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:59.024363+0000", 2024-09-13T11:27:06.573 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.9920999999999994e-05 2024-09-13T11:27:06.573 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:06.573 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:06.574 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:27:06.574 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:59.446512+0000", 2024-09-13T11:27:06.574 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.42214886800000001 2024-09-13T11:27:06.574 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:06.574 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:06.574 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:27:06.574 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:59.446541+0000", 2024-09-13T11:27:06.574 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.9672e-05 2024-09-13T11:27:06.574 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:06.574 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:06.574 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:27:06.574 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:59.446748+0000", 2024-09-13T11:27:06.574 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00020683600000000001 2024-09-13T11:27:06.574 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:06.574 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:06.575 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:27:06.575 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:59.446776+0000", 2024-09-13T11:27:06.575 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.7832000000000001e-05 2024-09-13T11:27:06.575 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:06.575 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:06.575 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:27:06.575 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:59.450192+0000", 2024-09-13T11:27:06.575 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0034157329999999998 2024-09-13T11:27:06.575 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:06.575 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:27:06.575 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:06.575 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:06.575 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:27:06.575 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-13T11:27:06.576 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 15, 14] out_osds: [] dead_osds: [] live_osds: [0, 4, 8, 12, 5, 9, 13, 2, 6, 10, 3, 7, 11, 15, 1, 14] 2024-09-13T11:27:06.576 INFO:tasks.thrashosds.thrasher:choose_action: min_in 2 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-13T11:27:06.576 INFO:tasks.thrashosds.thrasher:test_bluestore_reshard started 2024-09-13T11:27:06.576 INFO:tasks.thrashosds.thrasher:Killing osd 10, live_osds are [0, 4, 8, 12, 5, 9, 13, 2, 6, 10, 3, 7, 11, 15, 1, 14] 2024-09-13T11:27:06.576 DEBUG:tasks.ceph.osd.10:waiting for process to exit 2024-09-13T11:27:06.576 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T11:27:06.578 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-13T11:27:06.579 INFO:tasks.ceph.osd.5.smithi092.stderr:2024-09-13T11:27:06.575+0000 7f6f0b705640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26346) UID: 0 2024-09-13T11:27:06.679 INFO:tasks.ceph.osd.15.smithi177.stderr:2024-09-13T11:27:06.677+0000 7f9d6ce77640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26578) UID: 0 2024-09-13T11:27:06.723 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-09-13T11:27:06.723 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [], 2024-09-13T11:27:06.723 INFO:teuthology.orchestra.run.smithi177.stdout: "num_ops": 0 2024-09-13T11:27:06.723 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-09-13T11:27:06.743 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-13T11:27:06.780 INFO:tasks.ceph.osd.5.smithi092.stderr:2024-09-13T11:27:06.775+0000 7f6f0b705640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26346) UID: 0 2024-09-13T11:27:06.881 INFO:tasks.ceph.osd.14.smithi162.stderr:2024-09-13T11:27:06.879+0000 7f3df5903640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33154) UID: 0 2024-09-13T11:27:06.931 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-09-13T11:27:06.931 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [], 2024-09-13T11:27:06.931 INFO:teuthology.orchestra.run.smithi177.stdout: "complaint_time": 30, 2024-09-13T11:27:06.931 INFO:teuthology.orchestra.run.smithi177.stdout: "num_blocked_ops": 0 2024-09-13T11:27:06.931 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-09-13T11:27:06.951 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-13T11:27:06.981 INFO:tasks.ceph.osd.14.smithi162.stderr:2024-09-13T11:27:06.979+0000 7f3df5903640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33154) UID: 0 2024-09-13T11:27:06.997 INFO:tasks.ceph.osd.3.smithi177.stderr:2024-09-13T11:27:06.993+0000 7f10a25d1640 -1 osd.3 238 handle_osd_map injecting map crc failure 2024-09-13T11:27:07.082 INFO:tasks.ceph.osd.7.smithi177.stderr:2024-09-13T11:27:07.081+0000 7f897b51d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26575) UID: 0 2024-09-13T11:27:07.120 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-09-13T11:27:07.120 INFO:teuthology.orchestra.run.smithi177.stdout: "size": 20, 2024-09-13T11:27:07.120 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 600, 2024-09-13T11:27:07.121 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [ 2024-09-13T11:27:07.121 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.121 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPushReply(3.13 75/70 [PushReplyOp(3:cfab03d6:::benchmark_data_smithi177_31116_object16599:head)])", 2024-09-13T11:27:07.121 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:31.163480+0000", 2024-09-13T11:27:07.121 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 215.95657342000001, 2024-09-13T11:27:07.121 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.63009029299999997, 2024-09-13T11:27:07.121 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:27:07.121 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:27:07.121 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-13T11:27:07.121 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:27:07.121 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.121 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:27:07.121 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.163480+0000", 2024-09-13T11:27:07.122 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:07.122 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.122 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.122 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:27:07.122 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.163480+0000", 2024-09-13T11:27:07.122 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:07.122 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.122 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.122 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:27:07.122 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.163481+0000", 2024-09-13T11:27:07.122 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0410000000000001e-06 2024-09-13T11:27:07.122 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.122 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.123 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:27:07.123 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.163485+0000", 2024-09-13T11:27:07.123 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.1629999999999998e-06 2024-09-13T11:27:07.123 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.123 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.123 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:27:07.123 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.792166+0000", 2024-09-13T11:27:07.123 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.62868067400000005 2024-09-13T11:27:07.123 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.123 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.123 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:27:07.123 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.792184+0000", 2024-09-13T11:27:07.124 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.7898999999999999e-05 2024-09-13T11:27:07.124 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.124 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.124 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:27:07.124 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.792312+0000", 2024-09-13T11:27:07.124 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00012871000000000001 2024-09-13T11:27:07.124 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.124 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.124 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:27:07.124 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.793570+0000", 2024-09-13T11:27:07.125 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001257806 2024-09-13T11:27:07.125 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:07.125 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:27:07.125 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:07.125 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.125 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.125 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPushReply(3.13 76/70 [PushReplyOp(3:c9d43a82:::benchmark_data_smithi177_31116_object20026:head)])", 2024-09-13T11:27:07.125 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:32.427709+0000", 2024-09-13T11:27:07.125 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 214.692343905, 2024-09-13T11:27:07.125 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.46547316300000002, 2024-09-13T11:27:07.125 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:27:07.125 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:27:07.126 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-13T11:27:07.126 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:27:07.126 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.126 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:27:07.126 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:32.427709+0000", 2024-09-13T11:27:07.126 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:07.126 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.126 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.126 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:27:07.126 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:32.427709+0000", 2024-09-13T11:27:07.126 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:07.126 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.126 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.127 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:27:07.127 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:32.427711+0000", 2024-09-13T11:27:07.127 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5099999999999999e-06 2024-09-13T11:27:07.127 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.127 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.127 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:27:07.127 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:32.427717+0000", 2024-09-13T11:27:07.127 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.1569999999999997e-06 2024-09-13T11:27:07.127 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.127 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.127 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:27:07.127 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:32.892390+0000", 2024-09-13T11:27:07.127 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.46467251100000001 2024-09-13T11:27:07.128 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.128 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.128 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:27:07.128 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:32.892405+0000", 2024-09-13T11:27:07.128 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5296999999999999e-05 2024-09-13T11:27:07.128 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.128 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.128 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:27:07.128 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:32.892503+0000", 2024-09-13T11:27:07.128 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.7820999999999997e-05 2024-09-13T11:27:07.128 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.128 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.129 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:27:07.129 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:32.893183+0000", 2024-09-13T11:27:07.129 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00067986700000000001 2024-09-13T11:27:07.129 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:07.129 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:27:07.129 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:07.129 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.129 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.129 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:57435 3.3 e80/75 3:c0db4777:::benchmark_data_smithi177_31116_object57434:head v 80'3155, mlcod=80'3153)", 2024-09-13T11:27:07.130 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:37.437070+0000", 2024-09-13T11:27:07.130 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 209.68298308799999, 2024-09-13T11:27:07.130 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.35861845999999997, 2024-09-13T11:27:07.130 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:27:07.130 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:27:07.130 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:27:07.130 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:27:07.130 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.130 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:27:07.130 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:37.437070+0000", 2024-09-13T11:27:07.130 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:07.130 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.130 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.131 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:27:07.131 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:37.437070+0000", 2024-09-13T11:27:07.131 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:07.131 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.131 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.131 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:27:07.131 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:37.437072+0000", 2024-09-13T11:27:07.131 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5549999999999999e-06 2024-09-13T11:27:07.131 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.131 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.131 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:27:07.131 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:37.437141+0000", 2024-09-13T11:27:07.132 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.9494999999999998e-05 2024-09-13T11:27:07.132 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.132 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.132 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:27:07.132 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:37.793400+0000", 2024-09-13T11:27:07.132 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.35625817999999998 2024-09-13T11:27:07.132 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.132 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.132 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:27:07.133 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:37.793406+0000", 2024-09-13T11:27:07.133 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.6830000000000002e-06 2024-09-13T11:27:07.133 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.133 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.133 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:27:07.133 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:37.793472+0000", 2024-09-13T11:27:07.133 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.5671000000000002e-05 2024-09-13T11:27:07.133 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.133 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.133 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:27:07.133 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:37.793492+0000", 2024-09-13T11:27:07.133 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.0051000000000002e-05 2024-09-13T11:27:07.134 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.134 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.134 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:27:07.134 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:37.795626+0000", 2024-09-13T11:27:07.134 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0021344990000000002 2024-09-13T11:27:07.134 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.134 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.134 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:27:07.134 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:37.795689+0000", 2024-09-13T11:27:07.134 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.2325999999999999e-05 2024-09-13T11:27:07.134 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:07.134 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:27:07.135 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:07.135 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.135 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.135 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPushReply(3.23 84/78 [PushReplyOp(3:c7e44616:::benchmark_data_smithi177_31116_object32371:head)])", 2024-09-13T11:27:07.135 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:41.338210+0000", 2024-09-13T11:27:07.135 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 205.781843017, 2024-09-13T11:27:07.135 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.41118492299999998, 2024-09-13T11:27:07.135 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:27:07.135 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:27:07.135 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-13T11:27:07.135 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:27:07.135 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.136 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:27:07.136 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.338210+0000", 2024-09-13T11:27:07.136 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:07.136 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.136 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.136 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:27:07.136 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.338210+0000", 2024-09-13T11:27:07.136 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:07.136 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.136 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.136 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:27:07.136 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.338212+0000", 2024-09-13T11:27:07.137 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.314e-06 2024-09-13T11:27:07.137 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.137 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.137 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:27:07.137 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.338219+0000", 2024-09-13T11:27:07.137 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.6469999999999998e-06 2024-09-13T11:27:07.137 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.137 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.137 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:27:07.137 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.748675+0000", 2024-09-13T11:27:07.137 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.41045567500000002 2024-09-13T11:27:07.137 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.137 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.138 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:27:07.138 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.748684+0000", 2024-09-13T11:27:07.138 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.2150000000000007e-06 2024-09-13T11:27:07.138 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.138 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.138 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:27:07.138 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.748739+0000", 2024-09-13T11:27:07.138 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.4543000000000003e-05 2024-09-13T11:27:07.138 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.138 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.138 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:27:07.138 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.749395+0000", 2024-09-13T11:27:07.138 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00065652899999999999 2024-09-13T11:27:07.139 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:07.139 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:27:07.139 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:07.139 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.139 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.139 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPushReply(3.23 85/78 [PushReplyOp(3:c5c6dce1:::benchmark_data_smithi177_31116_object37293:head)])", 2024-09-13T11:27:07.139 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:42.377862+0000", 2024-09-13T11:27:07.139 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 204.742191558, 2024-09-13T11:27:07.139 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.62538808599999995, 2024-09-13T11:27:07.139 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:27:07.139 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:27:07.139 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-13T11:27:07.139 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:27:07.140 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.140 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:27:07.140 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.377862+0000", 2024-09-13T11:27:07.140 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:07.140 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.140 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.140 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:27:07.140 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.377862+0000", 2024-09-13T11:27:07.140 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:07.140 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.140 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.140 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:27:07.141 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.377863+0000", 2024-09-13T11:27:07.141 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.6899999999999996e-07 2024-09-13T11:27:07.141 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.141 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.141 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:27:07.141 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.377866+0000", 2024-09-13T11:27:07.141 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.219e-06 2024-09-13T11:27:07.141 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.141 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.141 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:27:07.141 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:43.001857+0000", 2024-09-13T11:27:07.141 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.62399084299999996 2024-09-13T11:27:07.141 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.141 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.142 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:27:07.142 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:43.001869+0000", 2024-09-13T11:27:07.142 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.2415999999999999e-05 2024-09-13T11:27:07.142 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.142 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.142 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:27:07.142 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:43.002194+0000", 2024-09-13T11:27:07.142 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000324774 2024-09-13T11:27:07.142 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.142 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.142 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:27:07.142 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:43.003250+0000", 2024-09-13T11:27:07.142 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001055865 2024-09-13T11:27:07.143 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:07.143 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:27:07.143 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:07.143 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.143 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.143 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPush(3.1c 89/77 [PushOp(3:38bcbc71:::benchmark_data_smithi177_31116_object31448:head, version: 59'1946, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:38bcbc71:::benchmark_data_smithi177_31116_object31448:head@59'1946, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:27:07.143 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:46.670723+0000", 2024-09-13T11:27:07.143 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 200.449330334, 2024-09-13T11:27:07.143 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.27576065, 2024-09-13T11:27:07.143 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:27:07.143 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:27:07.143 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:27:07.144 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:27:07.144 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.144 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:27:07.144 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.670723+0000", 2024-09-13T11:27:07.144 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:07.144 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.144 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.144 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:27:07.144 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.670723+0000", 2024-09-13T11:27:07.144 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:07.144 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.144 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.144 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:27:07.144 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.670724+0000", 2024-09-13T11:27:07.145 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.4389999999999999e-06 2024-09-13T11:27:07.145 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.145 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.145 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:27:07.145 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.670754+0000", 2024-09-13T11:27:07.145 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.9224999999999999e-05 2024-09-13T11:27:07.145 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.145 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.145 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:27:07.145 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.944369+0000", 2024-09-13T11:27:07.145 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.27361529800000001 2024-09-13T11:27:07.145 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.145 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.145 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:27:07.146 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.944422+0000", 2024-09-13T11:27:07.146 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.3318e-05 2024-09-13T11:27:07.146 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.146 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.146 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:27:07.146 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.944711+0000", 2024-09-13T11:27:07.146 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00028912399999999998 2024-09-13T11:27:07.146 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.146 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.147 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:27:07.147 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.944735+0000", 2024-09-13T11:27:07.147 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.3366999999999998e-05 2024-09-13T11:27:07.147 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.147 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.147 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:27:07.147 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.946484+0000", 2024-09-13T11:27:07.147 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0017488790000000001 2024-09-13T11:27:07.147 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:07.147 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:27:07.147 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:07.147 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.147 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.148 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPush(3.1c 91/77 [PushOp(3:3e18abc2:::benchmark_data_smithi177_31116_object35874:head, version: 62'2235, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:3e18abc2:::benchmark_data_smithi177_31116_object35874:head@62'2235, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:27:07.148 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:48.350968+0000", 2024-09-13T11:27:07.148 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 198.769085021, 2024-09-13T11:27:07.148 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.19257698000000001, 2024-09-13T11:27:07.148 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:27:07.148 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:27:07.148 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:27:07.148 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:27:07.148 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.148 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:27:07.148 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.350968+0000", 2024-09-13T11:27:07.148 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:07.148 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.149 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.149 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:27:07.149 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.350968+0000", 2024-09-13T11:27:07.149 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:07.149 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.149 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.149 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:27:07.149 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.350970+0000", 2024-09-13T11:27:07.149 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.796e-06 2024-09-13T11:27:07.150 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.150 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.150 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:27:07.150 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.351003+0000", 2024-09-13T11:27:07.150 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.2631e-05 2024-09-13T11:27:07.150 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.150 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.150 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:27:07.150 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.542140+0000", 2024-09-13T11:27:07.150 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.191136737 2024-09-13T11:27:07.150 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.150 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.151 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:27:07.151 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.542158+0000", 2024-09-13T11:27:07.151 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.8896999999999999e-05 2024-09-13T11:27:07.151 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.151 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.151 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:27:07.151 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.542263+0000", 2024-09-13T11:27:07.151 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000104158 2024-09-13T11:27:07.151 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.151 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.151 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:27:07.151 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.542279+0000", 2024-09-13T11:27:07.151 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5926999999999998e-05 2024-09-13T11:27:07.152 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.152 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.152 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:27:07.152 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.543545+0000", 2024-09-13T11:27:07.152 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0012668340000000001 2024-09-13T11:27:07.152 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:07.152 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:27:07.152 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:07.152 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.152 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.152 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:83223 3.1c e106/77 3:3bc6018d:::benchmark_data_smithi177_31116_object83222:head v 106'4363, mlcod=106'4360)", 2024-09-13T11:27:07.152 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:04.512139+0000", 2024-09-13T11:27:07.152 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 182.60791422299999, 2024-09-13T11:27:07.153 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.471923235, 2024-09-13T11:27:07.153 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:27:07.153 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:27:07.153 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:27:07.153 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:27:07.153 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.153 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:27:07.153 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.512139+0000", 2024-09-13T11:27:07.153 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:07.153 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.153 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.153 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:27:07.154 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.512139+0000", 2024-09-13T11:27:07.154 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:07.154 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.154 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.154 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:27:07.154 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.512143+0000", 2024-09-13T11:27:07.154 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.427e-06 2024-09-13T11:27:07.154 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.154 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.154 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:27:07.155 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.512227+0000", 2024-09-13T11:27:07.155 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.3965e-05 2024-09-13T11:27:07.155 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.155 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.155 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:27:07.155 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.976330+0000", 2024-09-13T11:27:07.155 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.46410342599999999 2024-09-13T11:27:07.155 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.155 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.155 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:27:07.155 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.976350+0000", 2024-09-13T11:27:07.155 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.9533000000000001e-05 2024-09-13T11:27:07.155 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.155 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.156 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:27:07.156 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.976488+0000", 2024-09-13T11:27:07.156 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00013799400000000001 2024-09-13T11:27:07.156 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.156 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.156 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:27:07.156 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.976543+0000", 2024-09-13T11:27:07.156 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.5442000000000002e-05 2024-09-13T11:27:07.156 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.156 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.156 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:27:07.156 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.983227+0000", 2024-09-13T11:27:07.156 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0066842899999999998 2024-09-13T11:27:07.157 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.157 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.157 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:27:07.157 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.984062+0000", 2024-09-13T11:27:07.157 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00083515800000000002 2024-09-13T11:27:07.157 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:07.157 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:27:07.157 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:07.157 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.157 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.157 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:110256 3.13 e119/83 3:cb00d078:::benchmark_data_smithi177_31116_object110255:head v 119'4823, mlcod=119'4821)", 2024-09-13T11:27:07.158 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:25.284530+0000", 2024-09-13T11:27:07.158 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 161.835523763, 2024-09-13T11:27:07.158 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.343491771, 2024-09-13T11:27:07.158 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:27:07.158 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:27:07.158 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:27:07.158 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:27:07.158 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.158 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:27:07.158 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.284530+0000", 2024-09-13T11:27:07.158 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:07.159 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.159 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.159 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:27:07.159 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.284530+0000", 2024-09-13T11:27:07.159 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:07.159 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.159 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.159 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:27:07.159 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.284532+0000", 2024-09-13T11:27:07.159 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.8870000000000001e-06 2024-09-13T11:27:07.159 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.159 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.159 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:27:07.159 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.284558+0000", 2024-09-13T11:27:07.160 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.6899000000000001e-05 2024-09-13T11:27:07.160 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.160 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.160 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:27:07.160 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.626172+0000", 2024-09-13T11:27:07.160 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.34161370800000002 2024-09-13T11:27:07.160 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.160 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.160 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:27:07.160 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.626187+0000", 2024-09-13T11:27:07.160 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.485e-05 2024-09-13T11:27:07.160 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.160 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.160 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:27:07.161 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.626245+0000", 2024-09-13T11:27:07.161 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.7998000000000003e-05 2024-09-13T11:27:07.161 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.161 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.161 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:27:07.161 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.626260+0000", 2024-09-13T11:27:07.161 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5429000000000001e-05 2024-09-13T11:27:07.161 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.161 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.161 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:27:07.161 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.627924+0000", 2024-09-13T11:27:07.161 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0016634060000000001 2024-09-13T11:27:07.161 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.161 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.162 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:27:07.162 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.628021+0000", 2024-09-13T11:27:07.162 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.7594000000000001e-05 2024-09-13T11:27:07.162 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:07.162 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:27:07.162 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:07.162 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.162 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.162 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPush(3.2b 121/119 [PushOp(3:d53a23e8:::benchmark_data_smithi177_31116_object5366:head, version: 45'342, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:d53a23e8:::benchmark_data_smithi177_31116_object5366:head@45'342, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:27:07.162 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:27.268731+0000", 2024-09-13T11:27:07.162 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 159.85132230100001, 2024-09-13T11:27:07.162 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.72228345000000005, 2024-09-13T11:27:07.162 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:27:07.162 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:27:07.163 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:27:07.163 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:27:07.163 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.163 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:27:07.163 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:27.268731+0000", 2024-09-13T11:27:07.163 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:07.163 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.163 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.163 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:27:07.163 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:27.268731+0000", 2024-09-13T11:27:07.163 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:07.163 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.163 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.163 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:27:07.164 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:27.268733+0000", 2024-09-13T11:27:07.164 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.9740000000000001e-06 2024-09-13T11:27:07.164 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.164 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.164 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:27:07.164 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:27.268778+0000", 2024-09-13T11:27:07.164 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.4679999999999999e-05 2024-09-13T11:27:07.164 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.164 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.164 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:27:07.164 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:27.988594+0000", 2024-09-13T11:27:07.164 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.71981667299999996 2024-09-13T11:27:07.164 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.164 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.165 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:27:07.165 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:27.988652+0000", 2024-09-13T11:27:07.165 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.7640999999999999e-05 2024-09-13T11:27:07.165 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.165 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.165 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:27:07.165 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:27.988889+0000", 2024-09-13T11:27:07.165 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00023706500000000001 2024-09-13T11:27:07.165 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.165 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.166 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:27:07.166 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:27.988933+0000", 2024-09-13T11:27:07.166 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.3865000000000001e-05 2024-09-13T11:27:07.166 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.166 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.166 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:27:07.166 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:27.991015+0000", 2024-09-13T11:27:07.166 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.002081552 2024-09-13T11:27:07.166 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:07.166 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:27:07.166 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:07.166 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.166 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.166 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPush(3.2b 124/119 [PushOp(3:d70bd127:::benchmark_data_smithi177_31116_object32365:head, version: 60'2034, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:d70bd127:::benchmark_data_smithi177_31116_object32365:head@60'2034, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:27:07.167 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:32.037338+0000", 2024-09-13T11:27:07.167 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 155.082715028, 2024-09-13T11:27:07.167 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.24358922699999999, 2024-09-13T11:27:07.167 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:27:07.167 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:27:07.167 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:27:07.167 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:27:07.167 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.167 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:27:07.167 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:32.037338+0000", 2024-09-13T11:27:07.167 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:07.167 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.167 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.168 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:27:07.168 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:32.037338+0000", 2024-09-13T11:27:07.168 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:07.168 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.168 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.168 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:27:07.168 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:32.037340+0000", 2024-09-13T11:27:07.168 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3790000000000001e-06 2024-09-13T11:27:07.168 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.168 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.168 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:27:07.168 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:32.037411+0000", 2024-09-13T11:27:07.168 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.1254000000000006e-05 2024-09-13T11:27:07.169 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.169 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.169 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:27:07.169 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:32.277467+0000", 2024-09-13T11:27:07.169 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.24005563199999999 2024-09-13T11:27:07.169 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.169 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.169 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:27:07.169 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:32.277494+0000", 2024-09-13T11:27:07.169 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.7120000000000001e-05 2024-09-13T11:27:07.169 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.169 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.170 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:27:07.170 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:32.279860+0000", 2024-09-13T11:27:07.170 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0023667559999999998 2024-09-13T11:27:07.170 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.170 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.170 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:27:07.170 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:32.279883+0000", 2024-09-13T11:27:07.170 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.2138999999999999e-05 2024-09-13T11:27:07.170 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.170 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.170 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:27:07.170 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:32.280928+0000", 2024-09-13T11:27:07.170 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0010449470000000001 2024-09-13T11:27:07.170 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:07.171 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:27:07.171 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:07.171 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.171 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.171 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPush(3.2b 125/119 [PushOp(3:d4c5ede9:::benchmark_data_smithi177_31116_object33277:head, version: 60'2092, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:d4c5ede9:::benchmark_data_smithi177_31116_object33277:head@60'2092, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:27:07.171 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:32.403929+0000", 2024-09-13T11:27:07.171 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 154.71612457800001, 2024-09-13T11:27:07.171 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.194592864, 2024-09-13T11:27:07.171 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:27:07.171 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:27:07.171 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:27:07.171 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:27:07.171 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.171 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:27:07.172 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:32.403929+0000", 2024-09-13T11:27:07.172 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:07.172 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.172 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.172 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:27:07.172 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:32.403929+0000", 2024-09-13T11:27:07.172 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:07.172 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.172 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.172 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:27:07.172 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:32.403930+0000", 2024-09-13T11:27:07.172 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.6729999999999999e-06 2024-09-13T11:27:07.172 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.173 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.173 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:27:07.173 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:32.403979+0000", 2024-09-13T11:27:07.173 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.8242000000000003e-05 2024-09-13T11:27:07.173 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.173 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.173 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:27:07.173 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:32.597445+0000", 2024-09-13T11:27:07.173 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.19346676500000001 2024-09-13T11:27:07.173 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.173 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.173 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:27:07.173 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:32.597455+0000", 2024-09-13T11:27:07.173 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.2939999999999994e-06 2024-09-13T11:27:07.174 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.174 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.174 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:27:07.174 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:32.597514+0000", 2024-09-13T11:27:07.174 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.9509000000000002e-05 2024-09-13T11:27:07.174 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.174 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.174 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:27:07.174 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:32.597522+0000", 2024-09-13T11:27:07.174 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.0350000000000001e-06 2024-09-13T11:27:07.174 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.174 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.174 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:27:07.175 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:32.598522+0000", 2024-09-13T11:27:07.175 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00099934599999999991 2024-09-13T11:27:07.175 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:07.175 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:27:07.175 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:07.175 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.175 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.175 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPush(3.2b 128/119 [PushOp(3:d6ff590f:::benchmark_data_smithi177_31116_object64581:head, version: 90'3750, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:d6ff590f:::benchmark_data_smithi177_31116_object64581:head@90'3750, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:27:07.175 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:37.982062+0000", 2024-09-13T11:27:07.175 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 149.13799167900001, 2024-09-13T11:27:07.175 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.28899675200000002, 2024-09-13T11:27:07.175 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:27:07.175 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:27:07.175 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:27:07.176 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:27:07.176 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.176 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:27:07.176 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:37.982062+0000", 2024-09-13T11:27:07.176 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:07.176 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.176 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.176 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:27:07.176 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:37.982062+0000", 2024-09-13T11:27:07.176 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:07.176 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.176 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.176 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:27:07.177 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:37.982064+0000", 2024-09-13T11:27:07.177 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.2259999999999999e-06 2024-09-13T11:27:07.177 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.177 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.177 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:27:07.177 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:37.982114+0000", 2024-09-13T11:27:07.177 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.0330000000000001e-05 2024-09-13T11:27:07.177 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.177 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.177 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:27:07.177 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:38.269983+0000", 2024-09-13T11:27:07.177 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.28786869999999998 2024-09-13T11:27:07.177 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.178 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.178 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:27:07.178 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:38.270000+0000", 2024-09-13T11:27:07.178 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.6977999999999999e-05 2024-09-13T11:27:07.178 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.178 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.178 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:27:07.178 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:38.270092+0000", 2024-09-13T11:27:07.178 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.221e-05 2024-09-13T11:27:07.178 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.178 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.178 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:27:07.178 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:38.270103+0000", 2024-09-13T11:27:07.179 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.1253999999999999e-05 2024-09-13T11:27:07.179 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.179 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.179 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:27:07.179 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:38.271058+0000", 2024-09-13T11:27:07.179 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00095505399999999999 2024-09-13T11:27:07.179 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:07.179 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:27:07.179 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:07.179 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.179 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.179 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:183827 3.2b e171/119 3:d63d3986:::benchmark_data_smithi177_31116_object43524:head v 171'6235, mlcod=171'6232)", 2024-09-13T11:27:07.179 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:25:32.487782+0000", 2024-09-13T11:27:07.179 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 94.632271466000006, 2024-09-13T11:27:07.180 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.21250508400000001, 2024-09-13T11:27:07.180 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:27:07.180 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:27:07.180 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:27:07.180 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:27:07.180 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.180 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:27:07.180 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:32.487782+0000", 2024-09-13T11:27:07.180 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:07.180 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.180 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.180 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:27:07.180 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:32.487782+0000", 2024-09-13T11:27:07.180 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:07.181 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.181 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.181 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:27:07.181 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:32.487783+0000", 2024-09-13T11:27:07.181 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.9699999999999994e-07 2024-09-13T11:27:07.181 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.181 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.181 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:27:07.181 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:32.487787+0000", 2024-09-13T11:27:07.181 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.5689999999999998e-06 2024-09-13T11:27:07.181 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.181 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.181 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:27:07.182 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:32.699019+0000", 2024-09-13T11:27:07.182 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.21123154499999999 2024-09-13T11:27:07.182 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.182 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.182 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:27:07.182 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:32.699024+0000", 2024-09-13T11:27:07.182 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.6229999999999996e-06 2024-09-13T11:27:07.182 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.182 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.182 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:27:07.182 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:32.699068+0000", 2024-09-13T11:27:07.182 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.4722000000000003e-05 2024-09-13T11:27:07.183 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.183 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.183 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:27:07.183 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:32.699084+0000", 2024-09-13T11:27:07.183 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5282999999999999e-05 2024-09-13T11:27:07.183 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.183 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.183 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:27:07.183 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:32.700246+0000", 2024-09-13T11:27:07.183 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001162735 2024-09-13T11:27:07.183 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.183 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.184 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:27:07.184 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:32.700287+0000", 2024-09-13T11:27:07.184 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.0609999999999999e-05 2024-09-13T11:27:07.184 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:07.184 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:27:07.184 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:07.184 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.184 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.184 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:191665 3.a e176/117 3:53060b28:::benchmark_data_smithi177_31116_object51362:head v 176'6424, mlcod=176'6422)", 2024-09-13T11:27:07.184 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:25:39.657701+0000", 2024-09-13T11:27:07.184 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 87.462351952000006, 2024-09-13T11:27:07.184 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.25090320900000002, 2024-09-13T11:27:07.184 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:27:07.184 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:27:07.185 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:27:07.185 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:27:07.185 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.185 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:27:07.185 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:39.657701+0000", 2024-09-13T11:27:07.185 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:07.185 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.185 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.185 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:27:07.185 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:39.657701+0000", 2024-09-13T11:27:07.185 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:07.185 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.185 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.186 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:27:07.186 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:39.657704+0000", 2024-09-13T11:27:07.186 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.7650000000000002e-06 2024-09-13T11:27:07.186 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.186 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.186 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:27:07.186 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:39.657714+0000", 2024-09-13T11:27:07.186 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.5719999999999992e-06 2024-09-13T11:27:07.186 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.186 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.186 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:27:07.186 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:39.904490+0000", 2024-09-13T11:27:07.186 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.246775831 2024-09-13T11:27:07.186 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.186 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.187 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:27:07.187 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:39.904503+0000", 2024-09-13T11:27:07.187 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3723000000000001e-05 2024-09-13T11:27:07.187 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.187 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.187 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:27:07.187 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:39.904660+0000", 2024-09-13T11:27:07.187 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00015705499999999999 2024-09-13T11:27:07.187 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.187 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.187 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:27:07.187 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:39.904699+0000", 2024-09-13T11:27:07.187 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.8877000000000001e-05 2024-09-13T11:27:07.187 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.188 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.188 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:27:07.188 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:39.908544+0000", 2024-09-13T11:27:07.188 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0038448739999999999 2024-09-13T11:27:07.188 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.188 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.188 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:27:07.188 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:39.908605+0000", 2024-09-13T11:27:07.188 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.0511999999999999e-05 2024-09-13T11:27:07.188 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:07.188 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:27:07.188 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:07.188 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.188 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.189 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:227427 3.a e196/117 3:52db129e:::benchmark_data_smithi177_31116_object87124:head v 196'6993, mlcod=196'6991)", 2024-09-13T11:27:07.189 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:26:11.046359+0000", 2024-09-13T11:27:07.189 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 56.073694387000003, 2024-09-13T11:27:07.189 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.19900411000000001, 2024-09-13T11:27:07.189 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:27:07.189 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:27:07.189 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:27:07.189 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:27:07.189 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.189 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:27:07.189 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:11.046359+0000", 2024-09-13T11:27:07.189 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:07.189 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.189 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.190 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:27:07.190 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:11.046359+0000", 2024-09-13T11:27:07.190 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:07.190 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.190 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.190 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:27:07.190 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:11.046360+0000", 2024-09-13T11:27:07.190 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.215e-06 2024-09-13T11:27:07.190 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.190 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.190 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:27:07.190 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:11.046365+0000", 2024-09-13T11:27:07.190 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.7060000000000003e-06 2024-09-13T11:27:07.190 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.191 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.191 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:27:07.191 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:11.239636+0000", 2024-09-13T11:27:07.191 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.193270687 2024-09-13T11:27:07.191 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.191 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.191 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:27:07.191 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:11.239651+0000", 2024-09-13T11:27:07.191 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5067e-05 2024-09-13T11:27:07.191 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.191 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.191 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:27:07.191 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:11.239970+0000", 2024-09-13T11:27:07.191 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00031894300000000003 2024-09-13T11:27:07.192 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.192 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.192 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:27:07.192 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:11.240027+0000", 2024-09-13T11:27:07.192 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.7419e-05 2024-09-13T11:27:07.192 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.192 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.192 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:27:07.192 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:11.243833+0000", 2024-09-13T11:27:07.192 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0038059629999999999 2024-09-13T11:27:07.192 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.192 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.192 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:27:07.193 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:11.245363+0000", 2024-09-13T11:27:07.193 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0015301100000000001 2024-09-13T11:27:07.193 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:07.193 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:27:07.193 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:07.193 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.193 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.193 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGRecoveryDelete(3.35 e231,227 3:af1a9c89:::benchmark_data_smithi177_31116_object6576:head,148'5831)", 2024-09-13T11:27:07.193 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:26:59.140312+0000", 2024-09-13T11:27:07.193 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 7.9797416139999999, 2024-09-13T11:27:07.193 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.27474032799999998, 2024-09-13T11:27:07.193 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:27:07.194 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:27:07.194 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:27:07.194 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:27:07.194 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.194 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:27:07.194 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:59.140312+0000", 2024-09-13T11:27:07.194 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:07.194 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.194 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.194 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:27:07.194 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:59.140312+0000", 2024-09-13T11:27:07.194 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:07.194 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.194 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.195 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:27:07.195 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:59.140313+0000", 2024-09-13T11:27:07.195 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.418e-06 2024-09-13T11:27:07.195 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.195 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.195 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:27:07.195 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:59.140319+0000", 2024-09-13T11:27:07.195 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.49e-06 2024-09-13T11:27:07.195 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.195 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.195 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:27:07.195 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:59.414669+0000", 2024-09-13T11:27:07.195 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.27435077400000002 2024-09-13T11:27:07.196 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.196 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.196 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:27:07.196 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:59.414685+0000", 2024-09-13T11:27:07.196 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5276000000000001e-05 2024-09-13T11:27:07.196 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.196 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.196 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:27:07.196 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:59.414839+0000", 2024-09-13T11:27:07.196 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00015430500000000001 2024-09-13T11:27:07.196 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.196 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.196 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:27:07.197 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:59.414868+0000", 2024-09-13T11:27:07.197 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.9393000000000001e-05 2024-09-13T11:27:07.197 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.197 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.197 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:27:07.197 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:59.415052+0000", 2024-09-13T11:27:07.197 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00018367200000000001 2024-09-13T11:27:07.197 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:07.197 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:27:07.197 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:07.197 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.197 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.197 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:274423 3.1c e233/77 3:391423c9:::benchmark_data_smithi177_31116_object134120:head v 233'10364, mlcod=233'10362)", 2024-09-13T11:27:07.198 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:27:00.621698+0000", 2024-09-13T11:27:07.198 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 6.4983552620000005, 2024-09-13T11:27:07.198 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.34328379999999997, 2024-09-13T11:27:07.198 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:27:07.198 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:27:07.198 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:27:07.198 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:27:07.198 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.198 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:27:07.198 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:27:00.621698+0000", 2024-09-13T11:27:07.198 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:07.198 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.198 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.199 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:27:07.199 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:27:00.621698+0000", 2024-09-13T11:27:07.199 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:07.199 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.199 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.199 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:27:07.199 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:27:00.621701+0000", 2024-09-13T11:27:07.199 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.396e-06 2024-09-13T11:27:07.199 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.199 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.199 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:27:07.199 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:27:00.621708+0000", 2024-09-13T11:27:07.199 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.8650000000000001e-06 2024-09-13T11:27:07.200 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.200 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.200 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:27:07.200 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:27:00.959179+0000", 2024-09-13T11:27:07.200 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.33747026200000002 2024-09-13T11:27:07.200 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.200 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.200 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:27:07.200 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:27:00.959188+0000", 2024-09-13T11:27:07.200 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.0310000000000004e-06 2024-09-13T11:27:07.200 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.200 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.200 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:27:07.201 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:27:00.959244+0000", 2024-09-13T11:27:07.201 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.6536999999999998e-05 2024-09-13T11:27:07.201 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.201 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.201 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:27:07.201 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:27:00.959263+0000", 2024-09-13T11:27:07.201 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.8802999999999999e-05 2024-09-13T11:27:07.201 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.201 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.201 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:27:07.201 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:27:00.964940+0000", 2024-09-13T11:27:07.201 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0056765670000000004 2024-09-13T11:27:07.201 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.202 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.202 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:27:07.202 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:27:00.964982+0000", 2024-09-13T11:27:07.202 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.2339000000000002e-05 2024-09-13T11:27:07.202 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:07.202 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:27:07.202 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:07.202 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.202 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.202 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:278044 3.d 3:b34c5ad6:::benchmark_data_smithi177_31116_object137741:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e237)", 2024-09-13T11:27:07.202 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:27:05.131032+0000", 2024-09-13T11:27:07.203 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 1.989021777, 2024-09-13T11:27:07.203 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.72037806999999998, 2024-09-13T11:27:07.203 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:27:07.203 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:27:07.203 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:27:07.203 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:27:07.203 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:27:07.203 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:27:07.203 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 278044 2024-09-13T11:27:07.203 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.203 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:27:07.203 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.204 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:27:07.204 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:27:05.131032+0000", 2024-09-13T11:27:07.204 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:07.204 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.204 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.204 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:27:07.204 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:27:05.131032+0000", 2024-09-13T11:27:07.204 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:07.204 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.204 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.204 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:27:07.204 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:27:05.131034+0000", 2024-09-13T11:27:07.204 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.7470000000000001e-06 2024-09-13T11:27:07.205 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.205 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.205 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:27:07.205 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:27:05.131041+0000", 2024-09-13T11:27:07.205 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.2419999999999997e-06 2024-09-13T11:27:07.205 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.205 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.205 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:27:07.205 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:27:05.131056+0000", 2024-09-13T11:27:07.205 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5031999999999999e-05 2024-09-13T11:27:07.205 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.205 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.205 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:27:07.205 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:27:05.131063+0000", 2024-09-13T11:27:07.206 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.3810000000000001e-06 2024-09-13T11:27:07.206 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.206 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.206 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:27:07.206 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:27:05.131118+0000", 2024-09-13T11:27:07.206 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.4688999999999998e-05 2024-09-13T11:27:07.206 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.206 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.206 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:27:07.206 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:27:05.131274+0000", 2024-09-13T11:27:07.206 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00015649500000000001 2024-09-13T11:27:07.206 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.206 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.207 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 3", 2024-09-13T11:27:07.207 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:27:05.131399+0000", 2024-09-13T11:27:07.207 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000124941 2024-09-13T11:27:07.207 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.207 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.207 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:27:07.207 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:27:05.133762+0000", 2024-09-13T11:27:07.207 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0023625819999999998 2024-09-13T11:27:07.207 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.207 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.207 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:27:07.207 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:27:05.851342+0000", 2024-09-13T11:27:07.207 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.71758002200000004 2024-09-13T11:27:07.208 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.208 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.208 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:27:07.208 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:27:05.851380+0000", 2024-09-13T11:27:07.208 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.8031000000000002e-05 2024-09-13T11:27:07.208 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.208 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.208 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:27:07.208 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:27:05.851410+0000", 2024-09-13T11:27:07.208 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.9907999999999999e-05 2024-09-13T11:27:07.208 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:07.208 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:27:07.208 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:07.208 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.209 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.209 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGRecoveryDeleteReply(3.33 e237,227 3:cdfb90c8:::benchmark_data_smithi177_31116_object37981:head,168'7001)", 2024-09-13T11:27:07.209 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:27:05.805541+0000", 2024-09-13T11:27:07.209 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 1.31451253, 2024-09-13T11:27:07.209 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.20135983700000001, 2024-09-13T11:27:07.209 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:27:07.209 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:27:07.209 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-13T11:27:07.209 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:27:07.209 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.209 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:27:07.209 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:27:05.805541+0000", 2024-09-13T11:27:07.209 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:07.210 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.210 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.210 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:27:07.210 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:27:05.805541+0000", 2024-09-13T11:27:07.210 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:07.210 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.210 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.210 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:27:07.210 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:27:05.805543+0000", 2024-09-13T11:27:07.210 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.0379999999999998e-06 2024-09-13T11:27:07.210 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.210 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.210 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:27:07.210 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:27:05.805551+0000", 2024-09-13T11:27:07.211 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.4600000000000003e-06 2024-09-13T11:27:07.211 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.211 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.211 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:27:07.211 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:27:06.006164+0000", 2024-09-13T11:27:07.211 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.20061280300000001 2024-09-13T11:27:07.211 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.211 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.211 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:27:07.211 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:27:06.006174+0000", 2024-09-13T11:27:07.211 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.8600000000000005e-06 2024-09-13T11:27:07.211 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.211 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.211 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:27:07.212 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:27:06.006298+0000", 2024-09-13T11:27:07.212 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00012429800000000001 2024-09-13T11:27:07.212 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.212 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.212 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:27:07.212 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:27:06.006901+0000", 2024-09-13T11:27:07.212 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00060237799999999998 2024-09-13T11:27:07.212 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:07.212 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:27:07.212 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:07.212 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:07.212 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:27:07.212 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-09-13T11:27:07.214 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-09-13T11:27:07.214 INFO:tasks.ceph.osd.15.smithi177.stderr:2024-09-13T11:27:07.213+0000 7f9d6ce77640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26578) UID: 0 2024-09-13T11:27:07.314 INFO:tasks.ceph.osd.12.smithi001.stderr:2024-09-13T11:27:07.310+0000 7f892c6e6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26314) UID: 0 2024-09-13T11:27:07.415 INFO:tasks.ceph.osd.12.smithi001.stderr:2024-09-13T11:27:07.410+0000 7f892c6e6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26314) UID: 0 2024-09-13T11:27:07.422 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-09-13T11:27:07.422 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [ 2024-09-13T11:27:07.422 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.422 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGRecoveryDeleteReply(3.2f e239,227 3:f499524e:::benchmark_data_smithi177_31116_object23819:head,157'6880)", 2024-09-13T11:27:07.422 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:27:07.423578+0000", 2024-09-13T11:27:07.422 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 0.00018344800000000001, 2024-09-13T11:27:07.422 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000202229, 2024-09-13T11:27:07.422 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:27:07.422 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:27:07.422 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-13T11:27:07.423 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:27:07.423 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.423 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:27:07.423 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:27:07.423578+0000", 2024-09-13T11:27:07.423 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:07.423 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.423 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.423 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:27:07.423 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:27:07.423578+0000", 2024-09-13T11:27:07.423 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:07.423 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.423 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.423 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:27:07.423 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:27:07.423579+0000", 2024-09-13T11:27:07.424 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.24e-06 2024-09-13T11:27:07.424 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.424 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.424 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:27:07.424 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:27:07.423584+0000", 2024-09-13T11:27:07.424 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.5009999999999998e-06 2024-09-13T11:27:07.424 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.424 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.424 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:27:07.424 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:27:07.423599+0000", 2024-09-13T11:27:07.424 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5358000000000001e-05 2024-09-13T11:27:07.424 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.424 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.424 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:27:07.425 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:27:07.423607+0000", 2024-09-13T11:27:07.425 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.1249999999999993e-06 2024-09-13T11:27:07.425 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.425 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.425 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:27:07.425 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:27:07.423662+0000", 2024-09-13T11:27:07.425 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.4645999999999998e-05 2024-09-13T11:27:07.425 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:07.425 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:27:07.425 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:07.425 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:07.425 INFO:teuthology.orchestra.run.smithi177.stdout: ], 2024-09-13T11:27:07.425 INFO:teuthology.orchestra.run.smithi177.stdout: "num_ops": 1 2024-09-13T11:27:07.426 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-09-13T11:27:07.439 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-09-13T11:27:07.497 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T11:27:07.515 INFO:tasks.ceph.osd.3.smithi177.stderr:2024-09-13T11:27:07.513+0000 7f10adba4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26574) UID: 0 2024-09-13T11:27:07.578 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-09-13T11:27:07.578 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [], 2024-09-13T11:27:07.578 INFO:teuthology.orchestra.run.smithi177.stdout: "complaint_time": 30, 2024-09-13T11:27:07.578 INFO:teuthology.orchestra.run.smithi177.stdout: "num_blocked_ops": 0 2024-09-13T11:27:07.579 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-09-13T11:27:07.597 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-09-13T11:27:07.616 INFO:tasks.ceph.osd.2.smithi162.stderr:2024-09-13T11:27:07.615+0000 7fe92b35a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26313) UID: 0 2024-09-13T11:27:07.694 INFO:teuthology.orchestra.run.smithi001.stdout:osd.0: {} 2024-09-13T11:27:07.694 INFO:teuthology.orchestra.run.smithi001.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T11:27:07.717 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-09-13T11:27:07.715+0000 7f97befc9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26315) UID: 0 2024-09-13T11:27:07.732 INFO:teuthology.orchestra.run.smithi001.stdout:osd.1: {} 2024-09-13T11:27:07.732 INFO:teuthology.orchestra.run.smithi001.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T11:27:07.769 INFO:teuthology.orchestra.run.smithi001.stdout:osd.2: {} 2024-09-13T11:27:07.769 INFO:teuthology.orchestra.run.smithi001.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T11:27:07.783 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-09-13T11:27:07.784 INFO:teuthology.orchestra.run.smithi177.stdout: "size": 20, 2024-09-13T11:27:07.784 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 600, 2024-09-13T11:27:07.784 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [ 2024-09-13T11:27:07.784 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.784 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:79 3.1 e32/30 3:841df5b5:::benchmark_data_smithi177_31116_object78:head v 32'6, mlcod=32'4)", 2024-09-13T11:27:07.784 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:22:27.057292+0000", 2024-09-13T11:27:07.784 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 280.72632139500001, 2024-09-13T11:27:07.784 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.89296786800000005, 2024-09-13T11:27:07.784 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:27:07.784 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:27:07.784 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:27:07.784 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:27:07.784 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.785 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:27:07.785 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.057292+0000", 2024-09-13T11:27:07.785 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:07.785 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.785 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.785 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:27:07.785 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.057292+0000", 2024-09-13T11:27:07.785 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:07.785 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.785 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.785 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:27:07.785 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.057293+0000", 2024-09-13T11:27:07.786 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.6109999999999999e-06 2024-09-13T11:27:07.786 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.786 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.786 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:27:07.786 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.057330+0000", 2024-09-13T11:27:07.786 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.6721999999999998e-05 2024-09-13T11:27:07.786 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.786 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.786 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:27:07.786 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.948197+0000", 2024-09-13T11:27:07.786 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.890867042 2024-09-13T11:27:07.786 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.786 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.786 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:27:07.787 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.948213+0000", 2024-09-13T11:27:07.787 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5826000000000001e-05 2024-09-13T11:27:07.787 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.787 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.787 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:27:07.787 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.948377+0000", 2024-09-13T11:27:07.787 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00016348600000000001 2024-09-13T11:27:07.787 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.787 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.787 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:27:07.787 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.948447+0000", 2024-09-13T11:27:07.787 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.0891999999999998e-05 2024-09-13T11:27:07.787 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.787 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.788 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:27:07.788 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.950150+0000", 2024-09-13T11:27:07.788 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001702329 2024-09-13T11:27:07.788 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.788 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.788 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:27:07.788 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.950260+0000", 2024-09-13T11:27:07.788 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00010996 2024-09-13T11:27:07.788 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:07.788 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:27:07.788 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:07.788 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.788 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.789 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:45783 3.16 3.17c10a76 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e69)", 2024-09-13T11:27:07.789 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:20.947774+0000", 2024-09-13T11:27:07.789 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 226.835839144, 2024-09-13T11:27:07.789 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.96682922400000004, 2024-09-13T11:27:07.789 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:27:07.789 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:27:07.789 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-13T11:27:07.789 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:27:07.789 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:27:07.789 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:27:07.789 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 45783 2024-09-13T11:27:07.789 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.789 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:27:07.790 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.790 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:27:07.790 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:20.947774+0000", 2024-09-13T11:27:07.790 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:07.790 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.790 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.790 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:27:07.790 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:20.947774+0000", 2024-09-13T11:27:07.790 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:07.790 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.790 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.790 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:27:07.790 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:20.947777+0000", 2024-09-13T11:27:07.791 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.57e-06 2024-09-13T11:27:07.791 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.791 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.791 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:27:07.791 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:20.947802+0000", 2024-09-13T11:27:07.791 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.5528999999999999e-05 2024-09-13T11:27:07.791 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.791 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.791 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:27:07.791 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:20.947813+0000", 2024-09-13T11:27:07.791 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0473e-05 2024-09-13T11:27:07.791 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.791 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.791 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:27:07.792 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:20.947821+0000", 2024-09-13T11:27:07.792 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.3440000000000001e-06 2024-09-13T11:27:07.792 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.792 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.792 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:27:07.792 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:20.947933+0000", 2024-09-13T11:27:07.792 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00011210200000000001 2024-09-13T11:27:07.792 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.792 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.792 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for peered", 2024-09-13T11:27:07.792 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:20.947942+0000", 2024-09-13T11:27:07.792 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.6610000000000006e-06 2024-09-13T11:27:07.792 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.793 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.793 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:27:07.793 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:21.914579+0000", 2024-09-13T11:27:07.793 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.96663672300000003 2024-09-13T11:27:07.793 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.793 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.793 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:27:07.793 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:21.914603+0000", 2024-09-13T11:27:07.793 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.4822000000000001e-05 2024-09-13T11:27:07.793 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:07.793 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:27:07.793 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:07.793 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.793 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.794 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:54625 3.1f 3:f812cdfb:::benchmark_data_smithi177_31116_object54624:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e75)", 2024-09-13T11:27:07.794 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:30.979327+0000", 2024-09-13T11:27:07.794 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 216.80428635499999, 2024-09-13T11:27:07.794 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.96163670899999998, 2024-09-13T11:27:07.794 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:27:07.794 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:27:07.794 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-13T11:27:07.794 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:27:07.794 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:27:07.794 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:27:07.794 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 54625 2024-09-13T11:27:07.794 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.794 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:27:07.794 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.795 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:27:07.795 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.979327+0000", 2024-09-13T11:27:07.795 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:07.795 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.795 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.795 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:27:07.795 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.979327+0000", 2024-09-13T11:27:07.795 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:07.795 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.795 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.795 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:27:07.795 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.979329+0000", 2024-09-13T11:27:07.795 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.686e-06 2024-09-13T11:27:07.795 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.796 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.796 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:27:07.796 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.979364+0000", 2024-09-13T11:27:07.796 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.5750000000000002e-05 2024-09-13T11:27:07.796 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.796 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.796 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:27:07.796 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.979408+0000", 2024-09-13T11:27:07.796 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.3795999999999999e-05 2024-09-13T11:27:07.796 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.796 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.796 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:27:07.796 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.979427+0000", 2024-09-13T11:27:07.797 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.9040999999999999e-05 2024-09-13T11:27:07.797 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.797 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.797 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:27:07.797 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.122291+0000", 2024-09-13T11:27:07.797 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.14286373599999999 2024-09-13T11:27:07.797 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.797 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.797 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for peered", 2024-09-13T11:27:07.797 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.122298+0000", 2024-09-13T11:27:07.797 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.3719999999999999e-06 2024-09-13T11:27:07.797 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.797 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.797 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:27:07.798 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.940341+0000", 2024-09-13T11:27:07.798 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.81804249699999998 2024-09-13T11:27:07.798 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.798 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.798 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:27:07.798 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.940964+0000", 2024-09-13T11:27:07.798 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00062283100000000001 2024-09-13T11:27:07.798 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:07.798 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:27:07.798 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:07.798 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.798 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.798 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:54649 3.1c 3:3a457a93:::benchmark_data_smithi177_31116_object54648:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e75)", 2024-09-13T11:27:07.799 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:30.979460+0000", 2024-09-13T11:27:07.799 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 216.804153165, 2024-09-13T11:27:07.799 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.95579424000000002, 2024-09-13T11:27:07.799 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:27:07.799 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:27:07.799 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-13T11:27:07.799 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:27:07.799 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:27:07.799 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:27:07.799 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 54649 2024-09-13T11:27:07.799 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.799 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:27:07.799 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.799 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:27:07.800 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.979460+0000", 2024-09-13T11:27:07.800 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:07.800 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.800 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.800 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:27:07.800 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.979460+0000", 2024-09-13T11:27:07.800 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:07.800 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.800 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.800 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:27:07.800 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.979462+0000", 2024-09-13T11:27:07.800 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5799999999999999e-06 2024-09-13T11:27:07.800 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.801 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.801 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:27:07.801 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.979524+0000", 2024-09-13T11:27:07.801 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.2521999999999995e-05 2024-09-13T11:27:07.801 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.801 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.801 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:27:07.801 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.979561+0000", 2024-09-13T11:27:07.801 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.6349e-05 2024-09-13T11:27:07.801 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.801 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.801 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:27:07.801 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.979571+0000", 2024-09-13T11:27:07.802 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0457e-05 2024-09-13T11:27:07.802 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.802 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.803 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:27:07.803 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.121029+0000", 2024-09-13T11:27:07.803 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.14145766200000001 2024-09-13T11:27:07.803 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.803 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.803 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for peered", 2024-09-13T11:27:07.803 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.121039+0000", 2024-09-13T11:27:07.803 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0084e-05 2024-09-13T11:27:07.803 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.803 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.803 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:27:07.803 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.935157+0000", 2024-09-13T11:27:07.804 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.81411807899999999 2024-09-13T11:27:07.804 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.804 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.804 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:27:07.804 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.935254+0000", 2024-09-13T11:27:07.804 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.7507e-05 2024-09-13T11:27:07.804 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:07.804 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:27:07.804 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:07.804 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.804 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.804 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:54654 3.1c 3:3f0bb31a:::benchmark_data_smithi177_31116_object54653:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e75)", 2024-09-13T11:27:07.804 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:30.979665+0000", 2024-09-13T11:27:07.804 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 216.80394799999999, 2024-09-13T11:27:07.805 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.95623908099999999, 2024-09-13T11:27:07.805 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:27:07.805 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:27:07.805 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-13T11:27:07.805 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:27:07.806 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:27:07.806 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:27:07.806 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 54654 2024-09-13T11:27:07.806 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.806 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:27:07.806 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.806 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:27:07.806 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.979665+0000", 2024-09-13T11:27:07.806 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:07.806 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.807 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.807 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:27:07.807 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.979665+0000", 2024-09-13T11:27:07.807 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:07.807 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.807 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.807 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:27:07.807 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.979667+0000", 2024-09-13T11:27:07.807 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3540000000000001e-06 2024-09-13T11:27:07.807 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.807 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.807 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:27:07.807 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.979703+0000", 2024-09-13T11:27:07.807 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.6758000000000003e-05 2024-09-13T11:27:07.808 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.808 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.808 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:27:07.808 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.979714+0000", 2024-09-13T11:27:07.808 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0804e-05 2024-09-13T11:27:07.808 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.808 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.808 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:27:07.808 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.979724+0000", 2024-09-13T11:27:07.808 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0040999999999999e-05 2024-09-13T11:27:07.808 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.808 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.808 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:27:07.809 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.121061+0000", 2024-09-13T11:27:07.809 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.141336713 2024-09-13T11:27:07.809 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.809 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.809 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for peered", 2024-09-13T11:27:07.809 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.121067+0000", 2024-09-13T11:27:07.809 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.4150000000000004e-06 2024-09-13T11:27:07.809 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.809 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.809 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:27:07.809 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.935776+0000", 2024-09-13T11:27:07.809 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.81470863100000002 2024-09-13T11:27:07.809 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.810 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.810 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:27:07.810 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.935904+0000", 2024-09-13T11:27:07.810 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00012836499999999999 2024-09-13T11:27:07.810 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:07.810 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:27:07.810 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:07.810 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.810 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.810 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPushReply(3.11 75/70 [PushReplyOp(3:8df10771:::benchmark_data_smithi177_31116_object24734:head)])", 2024-09-13T11:27:07.810 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:31.149798+0000", 2024-09-13T11:27:07.810 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 216.63381498999999, 2024-09-13T11:27:07.810 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.32402702500000002, 2024-09-13T11:27:07.810 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:27:07.811 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:27:07.811 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-13T11:27:07.811 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:27:07.811 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.811 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:27:07.811 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.149798+0000", 2024-09-13T11:27:07.811 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:07.811 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.811 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.811 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:27:07.811 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.149798+0000", 2024-09-13T11:27:07.811 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:07.811 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.812 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.812 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:27:07.812 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.149799+0000", 2024-09-13T11:27:07.812 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.5700000000000001e-07 2024-09-13T11:27:07.812 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.812 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.812 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:27:07.812 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.149803+0000", 2024-09-13T11:27:07.812 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.0040000000000001e-06 2024-09-13T11:27:07.812 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.813 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.813 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:27:07.813 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.472172+0000", 2024-09-13T11:27:07.813 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.32236856699999999 2024-09-13T11:27:07.813 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.813 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.813 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:27:07.813 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.472187+0000", 2024-09-13T11:27:07.813 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5016e-05 2024-09-13T11:27:07.813 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.813 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.813 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:27:07.813 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.472302+0000", 2024-09-13T11:27:07.813 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000115121 2024-09-13T11:27:07.814 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.814 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.814 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:27:07.814 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.473825+0000", 2024-09-13T11:27:07.814 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0015234599999999999 2024-09-13T11:27:07.814 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:07.814 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:27:07.814 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:07.814 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.814 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.814 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:54668 3.1c 3:3d2c5e40:::benchmark_data_smithi177_31116_object54667:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e75)", 2024-09-13T11:27:07.814 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:31.245525+0000", 2024-09-13T11:27:07.814 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 216.538088733, 2024-09-13T11:27:07.815 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.69828973000000005, 2024-09-13T11:27:07.815 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:27:07.815 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:27:07.815 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:27:07.815 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:27:07.815 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:27:07.815 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:27:07.815 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 54668 2024-09-13T11:27:07.815 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.815 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:27:07.815 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.816 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:27:07.816 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.245525+0000", 2024-09-13T11:27:07.816 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:07.816 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.816 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.816 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:27:07.816 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.245525+0000", 2024-09-13T11:27:07.816 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:07.816 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.816 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.816 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:27:07.816 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.245526+0000", 2024-09-13T11:27:07.817 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.719e-06 2024-09-13T11:27:07.817 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.817 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.817 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:27:07.817 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.245547+0000", 2024-09-13T11:27:07.817 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.1084999999999999e-05 2024-09-13T11:27:07.817 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.817 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.817 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:27:07.817 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.245558+0000", 2024-09-13T11:27:07.817 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0934000000000001e-05 2024-09-13T11:27:07.817 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.817 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.817 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:27:07.818 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.245564+0000", 2024-09-13T11:27:07.818 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.6269999999999997e-06 2024-09-13T11:27:07.818 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.818 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.818 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:27:07.818 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.245620+0000", 2024-09-13T11:27:07.818 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.6289999999999998e-05 2024-09-13T11:27:07.818 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.818 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.818 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for peered", 2024-09-13T11:27:07.818 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.245632+0000", 2024-09-13T11:27:07.818 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.1748e-05 2024-09-13T11:27:07.818 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.818 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.819 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:27:07.819 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.935976+0000", 2024-09-13T11:27:07.819 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.69034455299999997 2024-09-13T11:27:07.819 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.819 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.819 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:27:07.819 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.936302+0000", 2024-09-13T11:27:07.819 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000325397 2024-09-13T11:27:07.819 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.819 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.819 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 13", 2024-09-13T11:27:07.820 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.936459+0000", 2024-09-13T11:27:07.820 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00015754300000000001 2024-09-13T11:27:07.820 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.820 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.820 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:27:07.820 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.938813+0000", 2024-09-13T11:27:07.820 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0023530999999999999 2024-09-13T11:27:07.820 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.820 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.820 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:27:07.820 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.943695+0000", 2024-09-13T11:27:07.820 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.004882506 2024-09-13T11:27:07.820 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.821 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.821 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:27:07.821 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.943767+0000", 2024-09-13T11:27:07.821 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.2454999999999997e-05 2024-09-13T11:27:07.821 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.821 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.821 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:27:07.821 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.943814+0000", 2024-09-13T11:27:07.821 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.6773000000000001e-05 2024-09-13T11:27:07.821 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:07.821 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:27:07.822 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:07.822 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.822 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.822 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPush(3.16 84/70 [PushOp(3:6f7be541:::benchmark_data_smithi177_31116_object43332:head, version: 66'2788, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:6f7be541:::benchmark_data_smithi177_31116_object43332:head@66'2788, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:27:07.822 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:41.207515+0000", 2024-09-13T11:27:07.822 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 206.57609796200001, 2024-09-13T11:27:07.823 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.52643771500000003, 2024-09-13T11:27:07.823 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:27:07.823 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:27:07.823 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:27:07.824 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:27:07.824 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.824 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:27:07.824 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.207515+0000", 2024-09-13T11:27:07.824 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:07.824 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.824 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.824 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:27:07.824 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.207515+0000", 2024-09-13T11:27:07.824 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:07.824 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.824 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.824 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:27:07.825 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.207516+0000", 2024-09-13T11:27:07.825 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.096e-06 2024-09-13T11:27:07.825 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.825 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.825 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:27:07.825 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.207717+0000", 2024-09-13T11:27:07.825 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000200199 2024-09-13T11:27:07.825 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.825 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.825 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:27:07.825 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.732785+0000", 2024-09-13T11:27:07.825 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.52506869599999995 2024-09-13T11:27:07.826 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.826 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.826 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:27:07.826 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.732801+0000", 2024-09-13T11:27:07.826 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5305999999999999e-05 2024-09-13T11:27:07.826 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.826 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.826 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:27:07.826 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.732889+0000", 2024-09-13T11:27:07.826 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.8157000000000003e-05 2024-09-13T11:27:07.826 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.826 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.826 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:27:07.827 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.732901+0000", 2024-09-13T11:27:07.827 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.1866e-05 2024-09-13T11:27:07.827 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.827 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.827 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:27:07.827 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.733953+0000", 2024-09-13T11:27:07.827 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001052395 2024-09-13T11:27:07.827 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:07.827 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:27:07.827 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:07.827 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.827 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.827 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPushReply(3.1f 85/78 [PushReplyOp(3:fd221d21:::benchmark_data_smithi177_31116_object7639:head)])", 2024-09-13T11:27:07.827 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:42.198715+0000", 2024-09-13T11:27:07.828 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 205.584898393, 2024-09-13T11:27:07.828 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.48930391099999998, 2024-09-13T11:27:07.828 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:27:07.828 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:27:07.828 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-13T11:27:07.828 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:27:07.828 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.828 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:27:07.828 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.198715+0000", 2024-09-13T11:27:07.828 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:07.828 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.828 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.829 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:27:07.829 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.198715+0000", 2024-09-13T11:27:07.829 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:07.829 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.829 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.829 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:27:07.829 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.198716+0000", 2024-09-13T11:27:07.829 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.3500000000000005e-07 2024-09-13T11:27:07.829 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.829 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.829 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:27:07.829 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.198719+0000", 2024-09-13T11:27:07.830 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.4539999999999999e-06 2024-09-13T11:27:07.830 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.830 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.830 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:27:07.830 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.684732+0000", 2024-09-13T11:27:07.830 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.48601262099999998 2024-09-13T11:27:07.830 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.830 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.830 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:27:07.830 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.684746+0000", 2024-09-13T11:27:07.830 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.4518999999999999e-05 2024-09-13T11:27:07.830 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.830 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.830 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:27:07.831 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.684911+0000", 2024-09-13T11:27:07.831 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000164728 2024-09-13T11:27:07.831 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.831 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.831 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:27:07.831 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.688019+0000", 2024-09-13T11:27:07.831 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0031077539999999999 2024-09-13T11:27:07.831 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:07.831 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:27:07.831 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:07.831 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.831 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.832 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPushReply(3.1c 85/77 [PushReplyOp(3:3a669437:::benchmark_data_smithi177_31116_object25280:head)])", 2024-09-13T11:27:07.832 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:42.352466+0000", 2024-09-13T11:27:07.832 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 205.43114749899999, 2024-09-13T11:27:07.832 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.36670725300000001, 2024-09-13T11:27:07.832 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:27:07.832 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:27:07.832 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-13T11:27:07.832 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:27:07.832 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.832 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:27:07.832 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.352466+0000", 2024-09-13T11:27:07.832 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:07.832 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.833 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.833 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:27:07.833 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.352466+0000", 2024-09-13T11:27:07.833 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:07.833 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.833 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.833 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:27:07.833 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.352467+0000", 2024-09-13T11:27:07.833 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.1429999999999999e-06 2024-09-13T11:27:07.833 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.833 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.834 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:27:07.834 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.352474+0000", 2024-09-13T11:27:07.834 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.6100000000000002e-06 2024-09-13T11:27:07.834 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.834 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.834 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:27:07.834 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.717011+0000", 2024-09-13T11:27:07.834 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.36453790200000002 2024-09-13T11:27:07.834 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.834 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.834 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:27:07.834 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.717032+0000", 2024-09-13T11:27:07.834 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.0999e-05 2024-09-13T11:27:07.834 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.835 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.835 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:27:07.835 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.717162+0000", 2024-09-13T11:27:07.835 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000129159 2024-09-13T11:27:07.835 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.835 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.835 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:27:07.835 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.719173+0000", 2024-09-13T11:27:07.835 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0020114400000000002 2024-09-13T11:27:07.835 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:07.835 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:27:07.835 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:07.835 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.835 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.836 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPushReply(3.21 89/78 [PushReplyOp(3:8534d087:::benchmark_data_smithi177_31116_object5474:head)])", 2024-09-13T11:27:07.836 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:46.632834+0000", 2024-09-13T11:27:07.836 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 201.15077921700001, 2024-09-13T11:27:07.836 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.324805556, 2024-09-13T11:27:07.836 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:27:07.836 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:27:07.836 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-13T11:27:07.836 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:27:07.836 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.836 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:27:07.836 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.632834+0000", 2024-09-13T11:27:07.836 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:07.836 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.836 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.837 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:27:07.837 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.632834+0000", 2024-09-13T11:27:07.837 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:07.837 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.837 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.837 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:27:07.837 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.632835+0000", 2024-09-13T11:27:07.837 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.4270000000000001e-06 2024-09-13T11:27:07.837 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.837 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.837 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:27:07.837 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.632840+0000", 2024-09-13T11:27:07.837 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.7809999999999996e-06 2024-09-13T11:27:07.837 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.838 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.838 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:27:07.838 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.956282+0000", 2024-09-13T11:27:07.838 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.323441639 2024-09-13T11:27:07.838 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.838 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.838 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:27:07.838 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.956293+0000", 2024-09-13T11:27:07.838 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.083e-05 2024-09-13T11:27:07.838 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.838 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.838 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:27:07.838 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.956395+0000", 2024-09-13T11:27:07.838 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00010184200000000001 2024-09-13T11:27:07.839 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.839 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.839 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:27:07.839 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.957640+0000", 2024-09-13T11:27:07.839 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0012450370000000001 2024-09-13T11:27:07.839 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:07.839 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:27:07.839 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:07.839 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.839 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.839 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:83223 3.1c 3:3bc6018d:::benchmark_data_smithi177_31116_object83222:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106)", 2024-09-13T11:27:07.839 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:04.511163+0000", 2024-09-13T11:27:07.839 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 183.27244984000001, 2024-09-13T11:27:07.839 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.47558093200000001, 2024-09-13T11:27:07.840 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:27:07.840 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:27:07.840 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:27:07.840 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:27:07.840 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:27:07.840 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:27:07.840 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 83223 2024-09-13T11:27:07.840 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.840 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:27:07.840 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.840 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:27:07.840 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.511163+0000", 2024-09-13T11:27:07.840 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:07.840 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.841 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.841 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:27:07.841 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.511163+0000", 2024-09-13T11:27:07.841 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:07.841 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.841 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.841 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:27:07.841 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.511167+0000", 2024-09-13T11:27:07.841 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.8330000000000004e-06 2024-09-13T11:27:07.841 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.841 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.841 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:27:07.841 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.511219+0000", 2024-09-13T11:27:07.841 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.1771999999999998e-05 2024-09-13T11:27:07.842 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.842 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.842 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:27:07.842 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.511252+0000", 2024-09-13T11:27:07.842 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.3315999999999997e-05 2024-09-13T11:27:07.842 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.842 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.842 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:27:07.842 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.511268+0000", 2024-09-13T11:27:07.842 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5812999999999999e-05 2024-09-13T11:27:07.842 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.842 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.842 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:27:07.842 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.511418+0000", 2024-09-13T11:27:07.843 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000150023 2024-09-13T11:27:07.843 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.843 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.843 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:27:07.843 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.511670+0000", 2024-09-13T11:27:07.843 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00025185499999999999 2024-09-13T11:27:07.843 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.843 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.843 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 7", 2024-09-13T11:27:07.843 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.511905+0000", 2024-09-13T11:27:07.843 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00023493799999999999 2024-09-13T11:27:07.843 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.843 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.843 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:27:07.843 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.514960+0000", 2024-09-13T11:27:07.844 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.003054977 2024-09-13T11:27:07.844 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.844 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.844 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:27:07.844 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.986432+0000", 2024-09-13T11:27:07.844 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.47147209200000001 2024-09-13T11:27:07.844 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.844 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.844 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:27:07.844 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.986635+0000", 2024-09-13T11:27:07.844 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00020309899999999999 2024-09-13T11:27:07.844 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.845 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.845 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:27:07.845 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.986744+0000", 2024-09-13T11:27:07.845 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000109214 2024-09-13T11:27:07.845 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:07.845 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:27:07.845 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:07.845 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.845 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.845 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop_reply(client.4665.0:84803 3.1f e107/78)", 2024-09-13T11:27:07.845 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:05.912627+0000", 2024-09-13T11:27:07.845 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 181.87098593299999, 2024-09-13T11:27:07.845 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.33726566600000002, 2024-09-13T11:27:07.845 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:27:07.845 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:27:07.846 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-13T11:27:07.846 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:27:07.846 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.846 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:27:07.846 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:05.912627+0000", 2024-09-13T11:27:07.846 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:07.846 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.846 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.846 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:27:07.846 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:05.912627+0000", 2024-09-13T11:27:07.846 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:07.846 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.846 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.846 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:27:07.847 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:05.912629+0000", 2024-09-13T11:27:07.847 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.4440000000000001e-06 2024-09-13T11:27:07.847 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.847 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.847 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:27:07.847 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:05.912634+0000", 2024-09-13T11:27:07.847 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.6869999999999999e-06 2024-09-13T11:27:07.847 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.847 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.847 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:27:07.847 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:06.249722+0000", 2024-09-13T11:27:07.847 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.33708766099999998 2024-09-13T11:27:07.847 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.847 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.848 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:27:07.848 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:06.249736+0000", 2024-09-13T11:27:07.848 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3623e-05 2024-09-13T11:27:07.848 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.848 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.848 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:27:07.848 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:06.249878+0000", 2024-09-13T11:27:07.848 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00014264599999999999 2024-09-13T11:27:07.848 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.848 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.848 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:27:07.848 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:06.249893+0000", 2024-09-13T11:27:07.848 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.4605e-05 2024-09-13T11:27:07.848 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:07.849 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:27:07.849 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:07.849 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.849 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.849 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:110265 3.1 e119/75 3:8236484d:::benchmark_data_smithi177_31116_object110264:head v 119'3935, mlcod=118'3933)", 2024-09-13T11:27:07.849 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:25.343348+0000", 2024-09-13T11:27:07.849 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 162.44026569100001, 2024-09-13T11:27:07.849 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.60013013500000001, 2024-09-13T11:27:07.849 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:27:07.849 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:27:07.849 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:27:07.849 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:27:07.849 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.849 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:27:07.850 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.343348+0000", 2024-09-13T11:27:07.850 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:07.850 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.850 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.850 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:27:07.850 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.343348+0000", 2024-09-13T11:27:07.850 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:07.850 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.850 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.850 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:27:07.850 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.343350+0000", 2024-09-13T11:27:07.850 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.576e-06 2024-09-13T11:27:07.850 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.850 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.851 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:27:07.851 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.343443+0000", 2024-09-13T11:27:07.851 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.2989e-05 2024-09-13T11:27:07.851 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.851 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.851 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:27:07.851 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.940863+0000", 2024-09-13T11:27:07.851 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.59742003300000002 2024-09-13T11:27:07.851 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.851 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.851 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:27:07.851 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.940875+0000", 2024-09-13T11:27:07.851 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.2147000000000001e-05 2024-09-13T11:27:07.851 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.851 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.852 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:27:07.852 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.940993+0000", 2024-09-13T11:27:07.852 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000117903 2024-09-13T11:27:07.852 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.852 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.852 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:27:07.852 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.941017+0000", 2024-09-13T11:27:07.852 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.3442999999999999e-05 2024-09-13T11:27:07.852 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.852 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.852 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:27:07.852 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.943347+0000", 2024-09-13T11:27:07.852 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0023299869999999999 2024-09-13T11:27:07.853 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.853 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.853 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:27:07.853 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.943478+0000", 2024-09-13T11:27:07.853 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00013105699999999999 2024-09-13T11:27:07.853 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:07.853 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:27:07.853 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:07.853 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.853 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.853 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:260347 3.16 3:6d3745c1:::benchmark_data_smithi177_31116_object120044:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e219)", 2024-09-13T11:27:07.853 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:26:42.470302+0000", 2024-09-13T11:27:07.853 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 25.3133111, 2024-09-13T11:27:07.853 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.43228830200000001, 2024-09-13T11:27:07.854 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:27:07.854 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:27:07.854 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:27:07.854 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:27:07.854 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:27:07.854 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:27:07.854 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 260347 2024-09-13T11:27:07.854 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.854 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:27:07.854 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.854 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:27:07.854 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:42.470302+0000", 2024-09-13T11:27:07.854 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:07.854 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.854 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.855 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:27:07.855 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:42.470302+0000", 2024-09-13T11:27:07.855 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:07.855 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.855 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.855 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:27:07.855 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:42.470306+0000", 2024-09-13T11:27:07.855 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.2899999999999996e-06 2024-09-13T11:27:07.855 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.855 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.855 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:27:07.855 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:42.470318+0000", 2024-09-13T11:27:07.855 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.1426e-05 2024-09-13T11:27:07.855 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.856 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.856 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:27:07.856 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:42.470361+0000", 2024-09-13T11:27:07.856 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.35e-05 2024-09-13T11:27:07.856 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.856 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.856 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:27:07.856 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:42.470371+0000", 2024-09-13T11:27:07.856 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.1740000000000006e-06 2024-09-13T11:27:07.856 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.856 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.856 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:27:07.856 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:42.470474+0000", 2024-09-13T11:27:07.856 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000103225 2024-09-13T11:27:07.857 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.857 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.857 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:27:07.857 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:42.470735+0000", 2024-09-13T11:27:07.857 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00026167399999999999 2024-09-13T11:27:07.857 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.857 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.857 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 15", 2024-09-13T11:27:07.857 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:42.470939+0000", 2024-09-13T11:27:07.857 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00020340000000000001 2024-09-13T11:27:07.857 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.857 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.857 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:27:07.857 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:42.477467+0000", 2024-09-13T11:27:07.858 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0065277319999999996 2024-09-13T11:27:07.858 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.858 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.858 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:27:07.858 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:42.902321+0000", 2024-09-13T11:27:07.858 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.42485422099999998 2024-09-13T11:27:07.858 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.858 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.858 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:27:07.858 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:42.902468+0000", 2024-09-13T11:27:07.858 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000147591 2024-09-13T11:27:07.858 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.858 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.858 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:27:07.859 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:42.902590+0000", 2024-09-13T11:27:07.859 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000122069 2024-09-13T11:27:07.859 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:07.859 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:27:07.859 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:07.859 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.859 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.859 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:266874 3.2f 3:f5648933:::benchmark_data_smithi177_31116_object126571:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e224)", 2024-09-13T11:27:07.859 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:26:49.252945+0000", 2024-09-13T11:27:07.859 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 18.530668269, 2024-09-13T11:27:07.859 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.79831508200000001, 2024-09-13T11:27:07.859 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:27:07.859 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:27:07.859 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:27:07.860 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:27:07.860 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:27:07.860 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:27:07.860 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 266874 2024-09-13T11:27:07.860 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.860 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:27:07.860 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.860 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:27:07.860 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:49.252945+0000", 2024-09-13T11:27:07.860 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:07.860 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.860 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.860 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:27:07.860 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:49.252945+0000", 2024-09-13T11:27:07.861 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:07.861 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.861 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.861 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:27:07.861 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:49.252949+0000", 2024-09-13T11:27:07.861 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.4390000000000002e-06 2024-09-13T11:27:07.861 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.861 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.861 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:27:07.861 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:49.252961+0000", 2024-09-13T11:27:07.861 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.1891e-05 2024-09-13T11:27:07.861 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.861 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.862 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:27:07.862 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:49.252989+0000", 2024-09-13T11:27:07.862 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.7384000000000001e-05 2024-09-13T11:27:07.862 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.862 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.862 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:27:07.862 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:49.253004+0000", 2024-09-13T11:27:07.862 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5013e-05 2024-09-13T11:27:07.862 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.862 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.862 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:27:07.862 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:49.253095+0000", 2024-09-13T11:27:07.862 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.1150000000000007e-05 2024-09-13T11:27:07.862 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.862 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.863 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for peered", 2024-09-13T11:27:07.863 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:49.253106+0000", 2024-09-13T11:27:07.863 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0749e-05 2024-09-13T11:27:07.863 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.863 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.863 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:27:07.863 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:50.047276+0000", 2024-09-13T11:27:07.863 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.79417086299999995 2024-09-13T11:27:07.863 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.863 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.863 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:27:07.863 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:50.047392+0000", 2024-09-13T11:27:07.863 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000115777 2024-09-13T11:27:07.864 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.864 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.864 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 12", 2024-09-13T11:27:07.864 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:50.047515+0000", 2024-09-13T11:27:07.864 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00012257699999999999 2024-09-13T11:27:07.864 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.864 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.864 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:27:07.864 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:50.050394+0000", 2024-09-13T11:27:07.864 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.002879038 2024-09-13T11:27:07.864 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.864 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.864 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:27:07.864 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:50.051179+0000", 2024-09-13T11:27:07.865 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000785 2024-09-13T11:27:07.865 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.865 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.865 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:27:07.865 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:50.051223+0000", 2024-09-13T11:27:07.865 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.4202000000000001e-05 2024-09-13T11:27:07.865 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.865 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.865 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:27:07.865 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:50.051260+0000", 2024-09-13T11:27:07.865 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.6999000000000003e-05 2024-09-13T11:27:07.865 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:07.865 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:27:07.865 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:07.866 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.866 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.866 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGRecoveryDeleteReply(3.2f e231,227 3:f4d0ce4f:::benchmark_data_smithi177_31116_object254:head,144'6093)", 2024-09-13T11:27:07.866 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:26:58.970945+0000", 2024-09-13T11:27:07.866 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 8.8126679199999991, 2024-09-13T11:27:07.866 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.45467746999999997, 2024-09-13T11:27:07.866 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:27:07.866 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:27:07.866 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-13T11:27:07.866 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:27:07.866 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.866 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:27:07.866 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:58.970945+0000", 2024-09-13T11:27:07.866 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:07.866 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.867 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.867 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:27:07.867 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:58.970945+0000", 2024-09-13T11:27:07.867 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:07.867 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.867 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.867 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:27:07.867 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:58.970947+0000", 2024-09-13T11:27:07.867 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.668e-06 2024-09-13T11:27:07.867 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.867 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.867 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:27:07.867 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:58.970961+0000", 2024-09-13T11:27:07.867 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3655999999999999e-05 2024-09-13T11:27:07.868 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.868 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.868 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:27:07.868 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:59.424664+0000", 2024-09-13T11:27:07.868 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.45370305599999999 2024-09-13T11:27:07.868 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.868 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.868 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:27:07.868 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:59.424682+0000", 2024-09-13T11:27:07.868 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.7949999999999999e-05 2024-09-13T11:27:07.868 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.868 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.868 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:27:07.868 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:59.425051+0000", 2024-09-13T11:27:07.868 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00036941300000000002 2024-09-13T11:27:07.869 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.869 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.869 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:27:07.869 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:59.425623+0000", 2024-09-13T11:27:07.869 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00057172699999999996 2024-09-13T11:27:07.869 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:07.869 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:27:07.869 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:07.869 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.869 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.869 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:273225 3.1 e231/75 3:83f859ad:::benchmark_data_smithi177_31116_object132922:head v 231'6454, mlcod=231'6451)", 2024-09-13T11:27:07.869 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:26:59.235477+0000", 2024-09-13T11:27:07.869 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 8.5481357960000004, 2024-09-13T11:27:07.869 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.30826847099999999, 2024-09-13T11:27:07.870 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:27:07.870 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:27:07.870 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:27:07.870 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:27:07.870 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.870 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:27:07.870 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:59.235477+0000", 2024-09-13T11:27:07.870 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:07.870 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.870 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.870 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:27:07.870 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:59.235477+0000", 2024-09-13T11:27:07.870 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:07.870 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.871 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.871 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:27:07.871 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:59.235479+0000", 2024-09-13T11:27:07.871 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0640000000000001e-06 2024-09-13T11:27:07.871 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.871 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.871 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:27:07.871 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:59.235484+0000", 2024-09-13T11:27:07.871 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.2800000000000003e-06 2024-09-13T11:27:07.871 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.871 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.871 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:27:07.871 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:59.540261+0000", 2024-09-13T11:27:07.871 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.30477764299999999 2024-09-13T11:27:07.872 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.872 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.872 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:27:07.872 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:59.540269+0000", 2024-09-13T11:27:07.872 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.1119999999999998e-06 2024-09-13T11:27:07.872 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.872 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.872 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:27:07.872 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:59.540357+0000", 2024-09-13T11:27:07.872 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.8158999999999994e-05 2024-09-13T11:27:07.872 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.872 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.872 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:27:07.872 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:59.540379+0000", 2024-09-13T11:27:07.873 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.2158e-05 2024-09-13T11:27:07.873 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.873 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.873 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:27:07.873 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:59.543702+0000", 2024-09-13T11:27:07.873 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0033230009999999999 2024-09-13T11:27:07.873 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.873 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.873 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:27:07.873 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:59.543746+0000", 2024-09-13T11:27:07.873 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.4054000000000002e-05 2024-09-13T11:27:07.873 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:07.873 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:27:07.873 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:07.873 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.874 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.874 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:274423 3.1c 3:391423c9:::benchmark_data_smithi177_31116_object134120:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e233)", 2024-09-13T11:27:07.874 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:27:00.620397+0000", 2024-09-13T11:27:07.874 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 7.1632163389999999, 2024-09-13T11:27:07.874 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.34487627199999998, 2024-09-13T11:27:07.874 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:27:07.874 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:27:07.874 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:27:07.874 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:27:07.874 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:27:07.874 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:27:07.874 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 274423 2024-09-13T11:27:07.874 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.874 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:27:07.875 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.875 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:27:07.875 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:27:00.620397+0000", 2024-09-13T11:27:07.875 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:07.875 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.875 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.875 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:27:07.875 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:27:00.620397+0000", 2024-09-13T11:27:07.875 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:07.875 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.875 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.875 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:27:07.875 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:27:00.620399+0000", 2024-09-13T11:27:07.875 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.657e-06 2024-09-13T11:27:07.876 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.876 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.876 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:27:07.876 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:27:00.620403+0000", 2024-09-13T11:27:07.876 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.3069999999999996e-06 2024-09-13T11:27:07.876 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.876 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.876 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:27:07.876 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:27:00.620413+0000", 2024-09-13T11:27:07.876 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0355e-05 2024-09-13T11:27:07.876 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.876 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.876 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:27:07.876 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:27:00.620421+0000", 2024-09-13T11:27:07.877 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.0250000000000002e-06 2024-09-13T11:27:07.877 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.877 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.877 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:27:07.877 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:27:00.620851+0000", 2024-09-13T11:27:07.877 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00042989900000000003 2024-09-13T11:27:07.877 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.877 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.877 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:27:07.877 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:27:00.620967+0000", 2024-09-13T11:27:07.877 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00011627 2024-09-13T11:27:07.877 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.877 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.877 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 7", 2024-09-13T11:27:07.878 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:27:00.621613+0000", 2024-09-13T11:27:07.878 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00064577099999999995 2024-09-13T11:27:07.878 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.878 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.878 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:27:07.878 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:27:00.627319+0000", 2024-09-13T11:27:07.878 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0057055200000000004 2024-09-13T11:27:07.878 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.878 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.878 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:27:07.878 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:27:00.965178+0000", 2024-09-13T11:27:07.878 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.33785940599999997 2024-09-13T11:27:07.878 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.878 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.878 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:27:07.879 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:27:00.965231+0000", 2024-09-13T11:27:07.879 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.2944e-05 2024-09-13T11:27:07.879 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.879 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.879 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:27:07.879 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:27:00.965273+0000", 2024-09-13T11:27:07.879 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.2117999999999999e-05 2024-09-13T11:27:07.879 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:07.879 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:27:07.879 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:07.880 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.880 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.880 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGRecoveryDeleteReply(3.2f e238,227 3:f487a72d:::benchmark_data_smithi177_31116_object20739:head,155'6770)", 2024-09-13T11:27:07.880 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:27:06.435293+0000", 2024-09-13T11:27:07.880 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 1.348320513, 2024-09-13T11:27:07.880 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.398313372, 2024-09-13T11:27:07.880 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:27:07.880 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:27:07.880 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-13T11:27:07.880 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:27:07.880 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.880 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:27:07.880 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:27:06.435293+0000", 2024-09-13T11:27:07.880 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:07.881 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.881 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.881 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:27:07.881 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:27:06.435293+0000", 2024-09-13T11:27:07.881 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:07.881 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.881 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.881 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:27:07.881 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:27:06.435294+0000", 2024-09-13T11:27:07.881 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.234e-06 2024-09-13T11:27:07.881 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.881 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.881 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:27:07.881 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:27:06.435302+0000", 2024-09-13T11:27:07.882 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.7959999999999996e-06 2024-09-13T11:27:07.882 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.882 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.882 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:27:07.882 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:27:06.833212+0000", 2024-09-13T11:27:07.882 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.39791068000000002 2024-09-13T11:27:07.882 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.883 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.885 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:27:07.885 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:27:06.833219+0000", 2024-09-13T11:27:07.885 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.7449999999999998e-06 2024-09-13T11:27:07.885 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.885 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.885 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:27:07.885 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:27:06.833270+0000", 2024-09-13T11:27:07.885 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.0717000000000002e-05 2024-09-13T11:27:07.885 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:07.885 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:07.885 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:27:07.886 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:27:06.833606+0000", 2024-09-13T11:27:07.886 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00033619999999999999 2024-09-13T11:27:07.886 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:07.886 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:27:07.886 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:07.886 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:07.886 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:27:07.886 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-09-13T11:27:07.888 INFO:teuthology.orchestra.run.smithi001.stdout:osd.3: {} 2024-09-13T11:27:07.889 INFO:teuthology.orchestra.run.smithi001.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T11:27:07.889 INFO:teuthology.orchestra.run.smithi001.stdout:osd.4: {} 2024-09-13T11:27:07.889 INFO:teuthology.orchestra.run.smithi001.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T11:27:07.890 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_ops_in_flight 2024-09-13T11:27:07.890 INFO:tasks.ceph.osd.7.smithi177.stderr:2024-09-13T11:27:07.885+0000 7f897b51d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26575) UID: 0 2024-09-13T11:27:07.890 INFO:teuthology.orchestra.run.smithi001.stdout:osd.5: {} 2024-09-13T11:27:07.891 INFO:teuthology.orchestra.run.smithi001.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T11:27:07.896 INFO:teuthology.orchestra.run.smithi001.stdout:osd.6: {} 2024-09-13T11:27:07.896 INFO:teuthology.orchestra.run.smithi001.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T11:27:07.926 INFO:teuthology.orchestra.run.smithi001.stdout:osd.7: {} 2024-09-13T11:27:07.927 INFO:teuthology.orchestra.run.smithi001.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T11:27:07.955 INFO:teuthology.orchestra.run.smithi001.stdout:osd.8: {} 2024-09-13T11:27:07.955 INFO:teuthology.orchestra.run.smithi001.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-13T11:27:07.988 INFO:teuthology.orchestra.run.smithi001.stdout:osd.9: {} 2024-09-13T11:27:07.989 INFO:teuthology.orchestra.run.smithi001.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-13T11:27:07.989 INFO:teuthology.orchestra.run.smithi001.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-09-13T11:27:07.989 INFO:tasks.ceph.osd.1.smithi092.stderr:2024-09-13T11:27:07.987+0000 7fe7789c5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33718) UID: 0 2024-09-13T11:27:08.028 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-13T11:27:08.029 INFO:teuthology.orchestra.run.smithi001.stdout:osd.11: {} 2024-09-13T11:27:08.029 INFO:teuthology.orchestra.run.smithi001.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-13T11:27:08.073 INFO:teuthology.orchestra.run.smithi001.stdout:osd.12: {} 2024-09-13T11:27:08.074 INFO:teuthology.orchestra.run.smithi001.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-13T11:27:08.089 INFO:tasks.ceph.osd.11.smithi177.stderr:2024-09-13T11:27:08.085+0000 7f24792c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26580) UID: 0 2024-09-13T11:27:08.107 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-09-13T11:27:08.107 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [], 2024-09-13T11:27:08.107 INFO:teuthology.orchestra.run.smithi177.stdout: "num_ops": 0 2024-09-13T11:27:08.107 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-09-13T11:27:08.130 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_blocked_ops 2024-09-13T11:27:08.132 INFO:teuthology.orchestra.run.smithi001.stdout:osd.13: {} 2024-09-13T11:27:08.132 INFO:teuthology.orchestra.run.smithi001.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-13T11:27:08.147 INFO:tasks.ceph.osd.13.smithi092.stderr:2024-09-13T11:27:08.143+0000 7f3e4ba0f640 -1 osd.13 239 handle_osd_map injecting map crc failure 2024-09-13T11:27:08.190 INFO:tasks.ceph.osd.9.smithi092.stderr:2024-09-13T11:27:08.187+0000 7ff5ab3d0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26348) UID: 0 2024-09-13T11:27:08.205 INFO:teuthology.orchestra.run.smithi001.stdout:osd.14: {} 2024-09-13T11:27:08.206 INFO:teuthology.orchestra.run.smithi001.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-13T11:27:08.237 INFO:teuthology.orchestra.run.smithi001.stdout:osd.15: {} 2024-09-13T11:27:08.237 INFO:teuthology.orchestra.run.smithi001.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-13T11:27:08.241 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-13T11:27:08.242 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-13T11:27:08.291 INFO:tasks.ceph.osd.7.smithi177.stderr:2024-09-13T11:27:08.289+0000 7f897b51d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26575) UID: 0 2024-09-13T11:27:08.317 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-09-13T11:27:08.317 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:27:08.338 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_historic_ops 2024-09-13T11:27:08.392 INFO:tasks.ceph.osd.14.smithi162.stderr:2024-09-13T11:27:08.391+0000 7f3df5903640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33154) UID: 0 2024-09-13T11:27:08.494 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-13T11:27:08.490+0000 7f6a68775640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26307) UID: 0 2024-09-13T11:27:08.522 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-09-13T11:27:08.523 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:27:08.542 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-13T11:27:08.594 INFO:tasks.ceph.osd.11.smithi177.stderr:2024-09-13T11:27:08.593+0000 7f24792c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26580) UID: 0 2024-09-13T11:27:08.689 INFO:teuthology.orchestra.run.smithi092.stdout:ERROR: (22) Invalid argument 2024-09-13T11:27:08.689 INFO:teuthology.orchestra.run.smithi092.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:27:08.696 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-13T11:27:08.694+0000 7f6a68775640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26307) UID: 0 2024-09-13T11:27:08.708 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-13T11:27:08.796 INFO:tasks.ceph.osd.12.smithi001.stderr:2024-09-13T11:27:08.794+0000 7f892c6e6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26314) UID: 0 2024-09-13T11:27:08.837 INFO:teuthology.orchestra.run.smithi092.stdout:ERROR: (22) Invalid argument 2024-09-13T11:27:08.837 INFO:teuthology.orchestra.run.smithi092.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:27:08.854 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-13T11:27:08.897 INFO:tasks.ceph.osd.14.smithi162.stderr:2024-09-13T11:27:08.895+0000 7f3df5903640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33154) UID: 0 2024-09-13T11:27:08.991 INFO:teuthology.orchestra.run.smithi092.stdout:ERROR: (22) Invalid argument 2024-09-13T11:27:08.991 INFO:teuthology.orchestra.run.smithi092.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:27:08.998 INFO:tasks.ceph.osd.14.smithi162.stderr:2024-09-13T11:27:08.995+0000 7f3df5903640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33154) UID: 0 2024-09-13T11:27:08.999 INFO:teuthology.orchestra.run.smithi001.stderr:noscrub is unset 2024-09-13T11:27:09.011 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_ops_in_flight 2024-09-13T11:27:09.037 INFO:tasks.ceph.osd.8.smithi001.stderr:2024-09-13T11:27:09.034+0000 7fb9ab2df640 -1 osd.8 240 handle_osd_map injecting map crc failure 2024-09-13T11:27:09.098 INFO:tasks.ceph.osd.14.smithi162.stderr:2024-09-13T11:27:09.095+0000 7f3df5903640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33154) UID: 0 2024-09-13T11:27:09.170 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-13T11:27:09.170 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:27:09.191 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_blocked_ops 2024-09-13T11:27:09.203 INFO:tasks.ceph.osd.4.smithi001.stderr:2024-09-13T11:27:09.194+0000 7ff8e0218640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26310) UID: 0 2024-09-13T11:27:09.300 INFO:tasks.ceph.osd.1.smithi092.stderr:2024-09-13T11:27:09.295+0000 7fe7789c5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33718) UID: 0 2024-09-13T11:27:09.369 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-13T11:27:09.369 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:27:09.386 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_historic_ops 2024-09-13T11:27:09.432 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-09-13T11:27:09.431+0000 7f97befc9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26315) UID: 0 2024-09-13T11:27:09.501 INFO:tasks.ceph.osd.11.smithi177.stderr:2024-09-13T11:27:09.501+0000 7f24792c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26580) UID: 0 2024-09-13T11:27:09.521 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-13T11:27:09.521 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:27:09.536 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-13T11:27:09.603 INFO:tasks.ceph.osd.1.smithi092.stderr:2024-09-13T11:27:09.599+0000 7fe7789c5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33718) UID: 0 2024-09-13T11:27:09.679 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-13T11:27:09.680 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:27:09.696 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-13T11:27:09.704 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-09-13T11:27:09.699+0000 7f97befc9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26315) UID: 0 2024-09-13T11:27:09.804 INFO:tasks.ceph.osd.8.smithi001.stderr:2024-09-13T11:27:09.802+0000 7fb9b48ae640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26312) UID: 0 2024-09-13T11:27:09.835 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-13T11:27:09.836 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:27:09.854 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-13T11:27:09.906 INFO:tasks.ceph.osd.14.smithi162.stderr:2024-09-13T11:27:09.903+0000 7f3df5903640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33154) UID: 0 2024-09-13T11:27:10.007 INFO:tasks.ceph.osd.11.smithi177.stderr:2024-09-13T11:27:10.005+0000 7f24792c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26580) UID: 0 2024-09-13T11:27:10.039 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-13T11:27:10.039 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:27:10.060 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-13T11:27:10.108 INFO:tasks.ceph.osd.11.smithi177.stderr:2024-09-13T11:27:10.105+0000 7f24792c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26580) UID: 0 2024-09-13T11:27:10.208 INFO:tasks.ceph.osd.8.smithi001.stderr:2024-09-13T11:27:10.206+0000 7fb9b48ae640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26312) UID: 0 2024-09-13T11:27:10.213 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-13T11:27:10.213 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:27:10.231 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-13T11:27:10.242 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T11:27:10.309 INFO:tasks.ceph.osd.1.smithi092.stderr:2024-09-13T11:27:10.307+0000 7fe7789c5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33718) UID: 0 2024-09-13T11:27:10.410 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-09-13T11:27:10.407+0000 7f97befc9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26315) UID: 0 2024-09-13T11:27:10.432 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-13T11:27:10.432 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:27:10.455 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-13T11:27:10.473 INFO:teuthology.orchestra.run.smithi001.stdout:osd.0: {} 2024-09-13T11:27:10.473 INFO:teuthology.orchestra.run.smithi001.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T11:27:10.505 INFO:teuthology.orchestra.run.smithi001.stdout:osd.1: {} 2024-09-13T11:27:10.505 INFO:teuthology.orchestra.run.smithi001.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T11:27:10.511 INFO:tasks.ceph.osd.12.smithi001.stderr:2024-09-13T11:27:10.506+0000 7f892c6e6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26314) UID: 0 2024-09-13T11:27:10.539 INFO:teuthology.orchestra.run.smithi001.stdout:osd.2: {} 2024-09-13T11:27:10.539 INFO:teuthology.orchestra.run.smithi001.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T11:27:10.573 INFO:teuthology.orchestra.run.smithi001.stdout:osd.3: {} 2024-09-13T11:27:10.574 INFO:teuthology.orchestra.run.smithi001.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T11:27:10.600 INFO:teuthology.orchestra.run.smithi001.stdout:osd.4: {} 2024-09-13T11:27:10.600 INFO:teuthology.orchestra.run.smithi001.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T11:27:10.611 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-09-13T11:27:10.607+0000 7f97befc9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26315) UID: 0 2024-09-13T11:27:10.612 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-13T11:27:10.613 INFO:teuthology.orchestra.run.smithi001.stdout: "size": 20, 2024-09-13T11:27:10.613 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 600, 2024-09-13T11:27:10.613 INFO:teuthology.orchestra.run.smithi001.stdout: "ops": [ 2024-09-13T11:27:10.613 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:10.613 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDPGPush(3.1a 102/78 [PushOp(3:5df50766:::benchmark_data_smithi177_31116_object50543:head, version: 72'3152, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:5df50766:::benchmark_data_smithi177_31116_object50543:head@72'3152, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:27:10.613 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:24:00.499311+0000", 2024-09-13T11:27:10.613 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 190.11145584600001, 2024-09-13T11:27:10.613 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.26146664800000002, 2024-09-13T11:27:10.613 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:27:10.613 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:27:10.613 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:27:10.613 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:27:10.613 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:10.614 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:27:10.614 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:00.499311+0000", 2024-09-13T11:27:10.614 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:27:10.614 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:10.614 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:10.614 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:27:10.614 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:00.499311+0000", 2024-09-13T11:27:10.614 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:27:10.614 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:10.614 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:10.614 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:27:10.614 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:00.499320+0000", 2024-09-13T11:27:10.614 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.6400000000000003e-06 2024-09-13T11:27:10.614 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:10.615 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:10.615 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:27:10.615 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:00.499384+0000", 2024-09-13T11:27:10.615 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.4554999999999994e-05 2024-09-13T11:27:10.615 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:10.615 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:10.615 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:27:10.615 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:00.759713+0000", 2024-09-13T11:27:10.615 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.260328742 2024-09-13T11:27:10.615 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:10.615 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:10.615 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:27:10.615 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:00.759742+0000", 2024-09-13T11:27:10.615 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.9298e-05 2024-09-13T11:27:10.616 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:10.616 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:10.616 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:27:10.616 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:00.759871+0000", 2024-09-13T11:27:10.616 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000128992 2024-09-13T11:27:10.616 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:10.616 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:10.616 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:27:10.616 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:00.759884+0000", 2024-09-13T11:27:10.616 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2687999999999999e-05 2024-09-13T11:27:10.616 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:10.616 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:10.616 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:27:10.616 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:00.760778+0000", 2024-09-13T11:27:10.617 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00089373299999999996 2024-09-13T11:27:10.617 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:27:10.617 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:27:10.617 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:27:10.617 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:10.617 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:10.617 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4665.0:82769 3.1e e105/102 3:7a4e88b5:::benchmark_data_smithi177_31116_object82768:head v 105'4381, mlcod=105'4379)", 2024-09-13T11:27:10.617 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:24:03.745047+0000", 2024-09-13T11:27:10.617 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 186.86571962100001, 2024-09-13T11:27:10.617 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.213806145, 2024-09-13T11:27:10.617 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:27:10.617 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:27:10.617 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:27:10.618 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:27:10.618 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:10.618 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:27:10.618 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:03.745047+0000", 2024-09-13T11:27:10.618 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:27:10.618 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:10.618 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:10.618 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:27:10.618 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:03.745047+0000", 2024-09-13T11:27:10.618 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:27:10.618 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:10.618 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:10.618 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:27:10.619 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:03.745049+0000", 2024-09-13T11:27:10.619 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.5710000000000001e-06 2024-09-13T11:27:10.619 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:10.619 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:10.619 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:27:10.619 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:03.745107+0000", 2024-09-13T11:27:10.619 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.8799999999999999e-05 2024-09-13T11:27:10.619 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:10.619 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:10.619 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:27:10.619 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:03.955145+0000", 2024-09-13T11:27:10.619 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.210037735 2024-09-13T11:27:10.619 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:10.619 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:10.619 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:27:10.620 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:03.955163+0000", 2024-09-13T11:27:10.620 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.7520000000000002e-05 2024-09-13T11:27:10.620 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:10.620 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:10.620 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:27:10.620 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:03.955473+0000", 2024-09-13T11:27:10.620 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00031014399999999999 2024-09-13T11:27:10.621 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:10.621 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:10.621 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:27:10.621 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:03.955538+0000", 2024-09-13T11:27:10.621 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.5059000000000002e-05 2024-09-13T11:27:10.622 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:10.622 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:10.622 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:27:10.622 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:03.958753+0000", 2024-09-13T11:27:10.622 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0032152769999999999 2024-09-13T11:27:10.622 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:10.622 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:10.622 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:27:10.622 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:03.958853+0000", 2024-09-13T11:27:10.622 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000100039 2024-09-13T11:27:10.622 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:27:10.622 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:27:10.622 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:27:10.622 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:10.623 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:10.623 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:99105 3.1a 3:5ac5090c:::benchmark_data_smithi177_31116_object99104:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113)", 2024-09-13T11:27:10.623 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:24:15.962466+0000", 2024-09-13T11:27:10.623 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 174.648300863, 2024-09-13T11:27:10.623 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.23832849, 2024-09-13T11:27:10.623 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:27:10.623 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:27:10.623 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:27:10.623 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:27:10.623 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:27:10.623 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:27:10.623 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 99105 2024-09-13T11:27:10.623 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:10.623 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:27:10.623 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:10.623 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:27:10.624 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.962466+0000", 2024-09-13T11:27:10.624 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:27:10.624 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:10.624 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:10.624 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:27:10.624 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.962466+0000", 2024-09-13T11:27:10.624 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:27:10.624 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:10.624 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:10.624 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:27:10.624 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.962471+0000", 2024-09-13T11:27:10.624 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.9570000000000001e-06 2024-09-13T11:27:10.625 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:10.625 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:10.625 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:27:10.625 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.962547+0000", 2024-09-13T11:27:10.625 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.5947999999999999e-05 2024-09-13T11:27:10.625 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:10.625 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:10.625 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:27:10.625 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.962576+0000", 2024-09-13T11:27:10.625 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.9547999999999999e-05 2024-09-13T11:27:10.625 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:10.625 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:10.625 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:27:10.625 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.962594+0000", 2024-09-13T11:27:10.625 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.8023e-05 2024-09-13T11:27:10.626 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:10.626 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:10.626 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:27:10.626 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.962770+0000", 2024-09-13T11:27:10.626 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00017551099999999999 2024-09-13T11:27:10.626 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:10.626 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:10.626 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:27:10.626 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.963114+0000", 2024-09-13T11:27:10.626 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00034440399999999998 2024-09-13T11:27:10.626 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:10.626 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:10.626 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 9", 2024-09-13T11:27:10.627 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.963351+0000", 2024-09-13T11:27:10.627 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000237209 2024-09-13T11:27:10.627 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:10.627 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:10.627 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:27:10.627 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:15.966345+0000", 2024-09-13T11:27:10.627 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0029933519999999999 2024-09-13T11:27:10.627 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:10.627 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:10.627 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:27:10.627 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:16.200724+0000", 2024-09-13T11:27:10.627 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.23437934599999999 2024-09-13T11:27:10.627 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:10.627 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:10.628 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:27:10.628 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:16.200761+0000", 2024-09-13T11:27:10.628 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.6406999999999999e-05 2024-09-13T11:27:10.628 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:10.628 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:10.628 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:27:10.628 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:16.200794+0000", 2024-09-13T11:27:10.628 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.3785000000000003e-05 2024-09-13T11:27:10.628 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:27:10.628 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:27:10.628 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:27:10.628 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:10.629 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:10.629 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:129921 3.1a 3:5fb9088b:::benchmark_data_smithi177_31116_object129920:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e134)", 2024-09-13T11:27:10.629 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:24:47.177556+0000", 2024-09-13T11:27:10.629 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 143.43321058399999, 2024-09-13T11:27:10.629 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.33804469999999998, 2024-09-13T11:27:10.629 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:27:10.629 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:27:10.629 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:27:10.629 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:27:10.629 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:27:10.629 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:27:10.629 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 129921 2024-09-13T11:27:10.630 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:10.630 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:27:10.630 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:10.630 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:27:10.630 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:47.177556+0000", 2024-09-13T11:27:10.630 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:27:10.630 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:10.630 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:10.630 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:27:10.630 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:47.177556+0000", 2024-09-13T11:27:10.630 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:27:10.630 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:10.630 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:10.630 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:27:10.631 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:47.177559+0000", 2024-09-13T11:27:10.631 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.0790000000000001e-06 2024-09-13T11:27:10.631 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:10.631 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:10.631 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:27:10.631 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:47.177589+0000", 2024-09-13T11:27:10.631 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.0173999999999999e-05 2024-09-13T11:27:10.631 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:10.631 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:10.631 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:27:10.631 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:47.177606+0000", 2024-09-13T11:27:10.631 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.6475999999999999e-05 2024-09-13T11:27:10.631 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:10.631 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:10.631 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:27:10.631 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:47.177615+0000", 2024-09-13T11:27:10.632 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.5729999999999997e-06 2024-09-13T11:27:10.632 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:10.632 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:10.632 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:27:10.632 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:47.177680+0000", 2024-09-13T11:27:10.632 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.4349000000000003e-05 2024-09-13T11:27:10.632 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:10.632 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:10.632 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:27:10.633 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:47.177794+0000", 2024-09-13T11:27:10.633 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000114119 2024-09-13T11:27:10.633 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:10.633 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:10.633 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 9", 2024-09-13T11:27:10.633 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:47.177895+0000", 2024-09-13T11:27:10.633 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000101078 2024-09-13T11:27:10.633 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:10.633 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:10.634 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:27:10.634 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:47.179634+0000", 2024-09-13T11:27:10.634 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.001738606 2024-09-13T11:27:10.634 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:10.634 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:10.634 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:27:10.634 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:47.515495+0000", 2024-09-13T11:27:10.634 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.33586143000000002 2024-09-13T11:27:10.634 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:10.634 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:10.634 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:27:10.634 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:47.515551+0000", 2024-09-13T11:27:10.634 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.6048999999999998e-05 2024-09-13T11:27:10.635 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:10.635 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:10.635 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:27:10.635 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:47.515601+0000", 2024-09-13T11:27:10.635 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.9767e-05 2024-09-13T11:27:10.635 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:27:10.635 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:27:10.635 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:27:10.635 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:10.635 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:10.635 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4665.0:140742 3.1e e144/102 3:79c694a0:::benchmark_data_smithi177_31116_object439:head v 144'6204, mlcod=144'6200)", 2024-09-13T11:27:10.635 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:24:58.321534+0000", 2024-09-13T11:27:10.635 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 132.28923305999999, 2024-09-13T11:27:10.635 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.23629921500000001, 2024-09-13T11:27:10.635 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:27:10.636 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:27:10.636 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:27:10.636 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:27:10.636 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:10.636 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:27:10.636 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.321534+0000", 2024-09-13T11:27:10.636 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:27:10.636 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:10.636 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:10.637 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:27:10.637 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.321534+0000", 2024-09-13T11:27:10.637 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:27:10.637 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:10.637 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:10.637 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:27:10.637 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.321535+0000", 2024-09-13T11:27:10.637 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.159e-06 2024-09-13T11:27:10.637 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:10.637 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:10.637 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:27:10.637 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.321545+0000", 2024-09-13T11:27:10.637 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.9219999999999992e-06 2024-09-13T11:27:10.637 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:10.638 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:10.638 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:27:10.638 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.553719+0000", 2024-09-13T11:27:10.638 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.23217401700000001 2024-09-13T11:27:10.638 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:10.638 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:10.638 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:27:10.638 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.553734+0000", 2024-09-13T11:27:10.638 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.5344000000000001e-05 2024-09-13T11:27:10.638 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:10.638 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:10.638 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:27:10.638 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.553858+0000", 2024-09-13T11:27:10.638 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00012357500000000001 2024-09-13T11:27:10.639 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:10.639 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:10.639 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:27:10.639 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.553903+0000", 2024-09-13T11:27:10.639 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.5374999999999998e-05 2024-09-13T11:27:10.639 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:10.639 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:10.639 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:27:10.639 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.557719+0000", 2024-09-13T11:27:10.639 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0038158670000000001 2024-09-13T11:27:10.639 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:10.639 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:10.639 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:27:10.639 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.557833+0000", 2024-09-13T11:27:10.639 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000113956 2024-09-13T11:27:10.640 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:27:10.640 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:27:10.640 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:27:10.640 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:10.640 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:10.640 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4665.0:140796 3.1e e144/102 3:7987a652:::benchmark_data_smithi177_31116_object493:head v 144'6205, mlcod=144'6202)", 2024-09-13T11:27:10.640 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:24:58.553657+0000", 2024-09-13T11:27:10.640 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 132.05711008399999, 2024-09-13T11:27:10.640 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.25308323100000002, 2024-09-13T11:27:10.640 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:27:10.640 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:27:10.640 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:27:10.640 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:27:10.640 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:10.641 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:27:10.641 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.553657+0000", 2024-09-13T11:27:10.641 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:27:10.641 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:10.641 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:10.641 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:27:10.641 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.553657+0000", 2024-09-13T11:27:10.641 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:27:10.641 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:10.641 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:10.641 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:27:10.641 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.553660+0000", 2024-09-13T11:27:10.641 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.7519999999999998e-06 2024-09-13T11:27:10.641 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:10.642 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:10.642 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:27:10.642 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.553678+0000", 2024-09-13T11:27:10.642 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.7156e-05 2024-09-13T11:27:10.642 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:10.642 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:10.642 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:27:10.642 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.804342+0000", 2024-09-13T11:27:10.642 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.25066436199999997 2024-09-13T11:27:10.642 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:10.642 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:10.642 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:27:10.642 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.804353+0000", 2024-09-13T11:27:10.642 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.1147e-05 2024-09-13T11:27:10.643 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:10.643 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:10.643 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:27:10.643 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.804433+0000", 2024-09-13T11:27:10.643 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.9586999999999995e-05 2024-09-13T11:27:10.643 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:10.643 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:10.643 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:27:10.643 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.804457+0000", 2024-09-13T11:27:10.643 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.4108e-05 2024-09-13T11:27:10.643 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:10.643 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:10.643 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:27:10.643 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.806684+0000", 2024-09-13T11:27:10.644 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0022271309999999998 2024-09-13T11:27:10.644 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:10.644 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:10.644 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:27:10.644 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.806740+0000", 2024-09-13T11:27:10.644 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.5988e-05 2024-09-13T11:27:10.644 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:27:10.644 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:27:10.644 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:27:10.644 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:10.644 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:10.644 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4665.0:191081 3.1e e175/102 3:7e784823:::benchmark_data_smithi177_31116_object50778:head v 175'7812, mlcod=175'7809)", 2024-09-13T11:27:10.644 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:25:38.547539+0000", 2024-09-13T11:27:10.644 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 92.063227459999993, 2024-09-13T11:27:10.644 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.213922061, 2024-09-13T11:27:10.645 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:27:10.645 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:27:10.645 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:27:10.645 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:27:10.645 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:10.645 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:27:10.645 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:38.547539+0000", 2024-09-13T11:27:10.645 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:27:10.645 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:10.645 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:10.645 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:27:10.645 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:38.547539+0000", 2024-09-13T11:27:10.645 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:27:10.645 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:10.646 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:10.646 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:27:10.646 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:38.547543+0000", 2024-09-13T11:27:10.646 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.3629999999999998e-06 2024-09-13T11:27:10.646 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:10.646 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:10.646 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:27:10.646 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:38.547558+0000", 2024-09-13T11:27:10.646 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.5043e-05 2024-09-13T11:27:10.646 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:10.646 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:10.646 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:27:10.646 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:38.756778+0000", 2024-09-13T11:27:10.646 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.209219936 2024-09-13T11:27:10.646 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:10.647 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:10.647 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:27:10.647 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:38.756790+0000", 2024-09-13T11:27:10.647 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2775999999999999e-05 2024-09-13T11:27:10.647 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:10.647 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:10.647 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:27:10.647 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:38.756934+0000", 2024-09-13T11:27:10.647 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00014401600000000001 2024-09-13T11:27:10.647 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:10.647 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:10.647 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:27:10.647 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:38.757001+0000", 2024-09-13T11:27:10.647 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.6716000000000003e-05 2024-09-13T11:27:10.647 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:10.648 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:10.648 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:27:10.648 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:38.761376+0000", 2024-09-13T11:27:10.648 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0043747320000000001 2024-09-13T11:27:10.648 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:10.648 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:10.648 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:27:10.648 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:38.761461+0000", 2024-09-13T11:27:10.648 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.5478999999999999e-05 2024-09-13T11:27:10.648 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:27:10.648 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:27:10.648 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:27:10.648 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:10.648 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:10.649 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4665.0:191087 3.1e e175/102 3:7a697202:::benchmark_data_smithi177_31116_object50784:head v 175'7813, mlcod=175'7809)", 2024-09-13T11:27:10.649 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:25:38.561827+0000", 2024-09-13T11:27:10.649 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 92.048939250000004, 2024-09-13T11:27:10.649 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.209395781, 2024-09-13T11:27:10.649 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:27:10.649 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:27:10.649 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:27:10.649 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:27:10.649 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:10.649 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:27:10.649 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:38.561827+0000", 2024-09-13T11:27:10.649 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:27:10.649 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:10.650 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:10.650 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:27:10.650 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:38.561827+0000", 2024-09-13T11:27:10.650 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:27:10.650 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:10.650 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:10.650 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:27:10.650 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:38.561837+0000", 2024-09-13T11:27:10.650 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.2329999999999995e-06 2024-09-13T11:27:10.650 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:10.650 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:10.650 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:27:10.650 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:38.561849+0000", 2024-09-13T11:27:10.650 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2593e-05 2024-09-13T11:27:10.650 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:10.651 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:10.651 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:27:10.651 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:38.766804+0000", 2024-09-13T11:27:10.651 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.20495488000000001 2024-09-13T11:27:10.651 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:10.651 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:10.651 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:27:10.651 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:38.766823+0000", 2024-09-13T11:27:10.651 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.9143e-05 2024-09-13T11:27:10.651 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:10.651 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:10.651 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:27:10.651 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:38.766973+0000", 2024-09-13T11:27:10.651 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000149226 2024-09-13T11:27:10.652 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:10.652 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:10.652 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:27:10.652 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:38.767014+0000", 2024-09-13T11:27:10.652 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.1183000000000002e-05 2024-09-13T11:27:10.652 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:10.652 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:10.652 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:27:10.652 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:38.771145+0000", 2024-09-13T11:27:10.652 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0041315270000000003 2024-09-13T11:27:10.652 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:10.652 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:10.652 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:27:10.652 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:38.771223+0000", 2024-09-13T11:27:10.653 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.7996000000000003e-05 2024-09-13T11:27:10.653 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:27:10.653 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:27:10.653 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:27:10.653 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:10.653 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:10.653 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:233802 3.11 3:8e27a54a:::benchmark_data_smithi177_31116_object93499:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e200)", 2024-09-13T11:27:10.653 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:26:16.812014+0000", 2024-09-13T11:27:10.653 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 53.798752982000003, 2024-09-13T11:27:10.653 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.34299280599999998, 2024-09-13T11:27:10.653 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:27:10.653 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:27:10.653 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:27:10.653 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:27:10.654 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:27:10.654 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:27:10.654 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 233802 2024-09-13T11:27:10.654 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:10.654 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:27:10.654 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:10.654 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:27:10.654 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:16.812014+0000", 2024-09-13T11:27:10.654 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:27:10.654 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:10.654 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:10.654 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:27:10.654 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:16.812014+0000", 2024-09-13T11:27:10.654 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:27:10.655 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:10.655 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:10.655 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:27:10.655 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:16.812015+0000", 2024-09-13T11:27:10.655 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.19e-06 2024-09-13T11:27:10.655 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:10.655 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:10.655 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:27:10.655 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:16.812019+0000", 2024-09-13T11:27:10.655 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.8720000000000004e-06 2024-09-13T11:27:10.655 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:10.655 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:10.655 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:27:10.655 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:16.812025+0000", 2024-09-13T11:27:10.656 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.3339999999999998e-06 2024-09-13T11:27:10.656 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:10.656 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:10.656 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:27:10.656 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:16.812029+0000", 2024-09-13T11:27:10.656 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.2989999999999998e-06 2024-09-13T11:27:10.656 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:10.656 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:10.656 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:27:10.656 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:16.812160+0000", 2024-09-13T11:27:10.656 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000130288 2024-09-13T11:27:10.656 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:10.656 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:10.657 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:27:10.657 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:16.812298+0000", 2024-09-13T11:27:10.657 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00013822500000000001 2024-09-13T11:27:10.657 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:10.657 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:10.657 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 11", 2024-09-13T11:27:10.657 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:16.812412+0000", 2024-09-13T11:27:10.657 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000113947 2024-09-13T11:27:10.657 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:10.657 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:10.657 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:27:10.657 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:16.815915+0000", 2024-09-13T11:27:10.657 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0035030259999999998 2024-09-13T11:27:10.658 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:10.658 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:10.658 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:27:10.658 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:17.154942+0000", 2024-09-13T11:27:10.658 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.33902705700000002 2024-09-13T11:27:10.658 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:10.658 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:10.658 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:27:10.658 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:17.154978+0000", 2024-09-13T11:27:10.658 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.5549999999999997e-05 2024-09-13T11:27:10.658 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:10.658 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:10.658 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:27:10.658 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:17.155007+0000", 2024-09-13T11:27:10.659 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.9017999999999999e-05 2024-09-13T11:27:10.659 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:27:10.659 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:27:10.659 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:27:10.659 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:10.659 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:10.659 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop_reply(client.4665.0:233802 3.11 e200/81 ondisk, result = 0)", 2024-09-13T11:27:10.659 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:26:16.815681+0000", 2024-09-13T11:27:10.659 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 53.795085483000001, 2024-09-13T11:27:10.659 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.33932907800000001, 2024-09-13T11:27:10.659 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:27:10.659 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:27:10.659 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:27:10.659 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:27:10.660 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:10.660 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:27:10.660 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:16.815681+0000", 2024-09-13T11:27:10.660 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:27:10.660 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:10.660 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:10.660 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:27:10.660 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:16.815681+0000", 2024-09-13T11:27:10.660 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:27:10.660 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:10.660 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:10.660 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:27:10.660 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:16.815682+0000", 2024-09-13T11:27:10.661 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.1230000000000001e-06 2024-09-13T11:27:10.661 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:10.661 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:10.661 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:27:10.661 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:16.815689+0000", 2024-09-13T11:27:10.661 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.5599999999999999e-06 2024-09-13T11:27:10.661 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:10.661 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:10.661 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:27:10.661 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:17.154882+0000", 2024-09-13T11:27:10.661 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.33919308599999998 2024-09-13T11:27:10.661 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:10.661 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:10.661 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:27:10.662 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:17.154890+0000", 2024-09-13T11:27:10.662 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.6149999999999999e-06 2024-09-13T11:27:10.662 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:10.662 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:10.662 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:27:10.662 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:17.154931+0000", 2024-09-13T11:27:10.662 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.1057999999999999e-05 2024-09-13T11:27:10.662 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:10.662 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:10.662 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:27:10.662 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:17.154937+0000", 2024-09-13T11:27:10.662 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.6719999999999998e-06 2024-09-13T11:27:10.662 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:10.662 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:10.663 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:27:10.663 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:17.155010+0000", 2024-09-13T11:27:10.663 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.2964000000000001e-05 2024-09-13T11:27:10.663 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:27:10.663 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:27:10.663 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:27:10.663 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:10.663 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:10.663 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:234291 3.11 3:8c4e5be8:::benchmark_data_smithi177_31116_object93988:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e201)", 2024-09-13T11:27:10.663 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:26:17.320509+0000", 2024-09-13T11:27:10.663 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 53.290258156999997, 2024-09-13T11:27:10.663 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.231498433, 2024-09-13T11:27:10.663 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:27:10.663 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:27:10.664 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:27:10.664 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:27:10.664 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:27:10.664 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:27:10.664 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 234291 2024-09-13T11:27:10.664 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:10.664 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:27:10.664 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:10.664 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:27:10.664 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:17.320509+0000", 2024-09-13T11:27:10.664 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:27:10.664 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:10.664 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:10.665 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:27:10.665 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:17.320509+0000", 2024-09-13T11:27:10.665 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:27:10.665 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:10.665 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:10.665 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:27:10.665 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:17.320510+0000", 2024-09-13T11:27:10.665 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.1999999999999999e-06 2024-09-13T11:27:10.665 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:10.665 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:10.665 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:27:10.665 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:17.320514+0000", 2024-09-13T11:27:10.665 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.8139999999999998e-06 2024-09-13T11:27:10.666 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:10.666 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:10.666 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:27:10.666 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:17.320523+0000", 2024-09-13T11:27:10.666 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.7589999999999992e-06 2024-09-13T11:27:10.666 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:10.666 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:10.666 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:27:10.666 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:17.320529+0000", 2024-09-13T11:27:10.666 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.2499999999999997e-06 2024-09-13T11:27:10.666 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:10.666 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:10.666 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:27:10.666 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:17.320580+0000", 2024-09-13T11:27:10.667 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.0980999999999999e-05 2024-09-13T11:27:10.667 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:10.667 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:10.667 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:27:10.667 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:17.320704+0000", 2024-09-13T11:27:10.667 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00012416599999999999 2024-09-13T11:27:10.667 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:10.667 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:10.667 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 11", 2024-09-13T11:27:10.667 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:17.320812+0000", 2024-09-13T11:27:10.667 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000108383 2024-09-13T11:27:10.667 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:10.667 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:10.667 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:27:10.668 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:17.323672+0000", 2024-09-13T11:27:10.668 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0028601329999999999 2024-09-13T11:27:10.668 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:10.668 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:10.668 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:27:10.668 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:17.551809+0000", 2024-09-13T11:27:10.668 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.22813635400000001 2024-09-13T11:27:10.668 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:10.668 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:10.668 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:27:10.668 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:17.551920+0000", 2024-09-13T11:27:10.668 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000111805 2024-09-13T11:27:10.668 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:10.669 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:10.669 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:27:10.669 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:17.552007+0000", 2024-09-13T11:27:10.669 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.6588000000000005e-05 2024-09-13T11:27:10.669 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:27:10.669 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:27:10.669 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:27:10.669 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:10.669 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:10.669 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop_reply(client.4665.0:234291 3.11 e201/81 ondisk, result = 0)", 2024-09-13T11:27:10.669 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:26:17.324111+0000", 2024-09-13T11:27:10.669 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 53.286655701999997, 2024-09-13T11:27:10.669 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.22790548199999999, 2024-09-13T11:27:10.670 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:27:10.670 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:27:10.670 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:27:10.670 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:27:10.670 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:10.670 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:27:10.670 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:17.324111+0000", 2024-09-13T11:27:10.670 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:27:10.670 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:10.670 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:10.670 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:27:10.670 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:17.324111+0000", 2024-09-13T11:27:10.670 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:27:10.670 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:10.671 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:10.671 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:27:10.671 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:17.324112+0000", 2024-09-13T11:27:10.671 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.313e-06 2024-09-13T11:27:10.671 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:10.671 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:10.671 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:27:10.671 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:17.324117+0000", 2024-09-13T11:27:10.671 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.5870000000000003e-06 2024-09-13T11:27:10.671 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:10.671 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:10.671 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:27:10.671 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:17.551656+0000", 2024-09-13T11:27:10.671 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.22753944200000001 2024-09-13T11:27:10.671 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:10.672 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:10.672 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:27:10.672 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:17.551669+0000", 2024-09-13T11:27:10.672 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2233e-05 2024-09-13T11:27:10.672 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:10.672 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:10.672 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:27:10.672 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:17.551780+0000", 2024-09-13T11:27:10.672 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000111471 2024-09-13T11:27:10.672 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:10.672 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:10.672 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:27:10.672 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:17.551798+0000", 2024-09-13T11:27:10.672 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.7887e-05 2024-09-13T11:27:10.673 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:10.673 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:10.673 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:27:10.673 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:17.552017+0000", 2024-09-13T11:27:10.673 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00021854900000000001 2024-09-13T11:27:10.673 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:27:10.673 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:27:10.673 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:27:10.673 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:10.673 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:10.673 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:243814 3.1a 3:5aaae69c:::benchmark_data_smithi177_31116_object103511:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e207)", 2024-09-13T11:27:10.673 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:26:26.761070+0000", 2024-09-13T11:27:10.673 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 43.849696651000002, 2024-09-13T11:27:10.673 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.222488768, 2024-09-13T11:27:10.674 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:27:10.674 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:27:10.674 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:27:10.674 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:27:10.674 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:27:10.674 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:27:10.674 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 243814 2024-09-13T11:27:10.674 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:10.674 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:27:10.674 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:10.674 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:27:10.674 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:26.761070+0000", 2024-09-13T11:27:10.674 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:27:10.674 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:10.675 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:10.675 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:27:10.675 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:26.761070+0000", 2024-09-13T11:27:10.675 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:27:10.675 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:10.675 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:10.675 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:27:10.675 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:26.761073+0000", 2024-09-13T11:27:10.675 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.9119999999999998e-06 2024-09-13T11:27:10.675 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:10.675 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:10.675 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:27:10.675 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:26.761082+0000", 2024-09-13T11:27:10.675 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.7960000000000001e-06 2024-09-13T11:27:10.675 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:10.676 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:10.676 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:27:10.676 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:26.761105+0000", 2024-09-13T11:27:10.676 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.2976000000000002e-05 2024-09-13T11:27:10.676 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:10.676 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:10.676 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:27:10.676 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:26.761117+0000", 2024-09-13T11:27:10.676 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2323999999999999e-05 2024-09-13T11:27:10.676 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:10.676 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:10.676 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:27:10.676 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:26.761207+0000", 2024-09-13T11:27:10.676 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.9534e-05 2024-09-13T11:27:10.677 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:10.677 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:10.677 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:27:10.677 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:26.761326+0000", 2024-09-13T11:27:10.677 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000119478 2024-09-13T11:27:10.677 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:10.677 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:10.677 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 9", 2024-09-13T11:27:10.677 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:26.761422+0000", 2024-09-13T11:27:10.677 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.577e-05 2024-09-13T11:27:10.678 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:10.678 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:10.678 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:27:10.678 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:26.765182+0000", 2024-09-13T11:27:10.678 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0037604280000000001 2024-09-13T11:27:10.678 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:10.679 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:10.679 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:27:10.679 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:26.983388+0000", 2024-09-13T11:27:10.679 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.21820598199999999 2024-09-13T11:27:10.679 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:10.679 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:10.679 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:27:10.679 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:26.983485+0000", 2024-09-13T11:27:10.679 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.7072999999999997e-05 2024-09-13T11:27:10.679 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:10.679 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:10.679 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:27:10.679 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:26.983559+0000", 2024-09-13T11:27:10.679 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.3495e-05 2024-09-13T11:27:10.679 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:27:10.680 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:27:10.680 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:27:10.680 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:10.680 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:10.680 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop_reply(client.4665.0:260146 3.1a e219/105 ondisk, result = 0)", 2024-09-13T11:27:10.680 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:26:42.138512+0000", 2024-09-13T11:27:10.680 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 28.472254801999998, 2024-09-13T11:27:10.680 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.240341576, 2024-09-13T11:27:10.680 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:27:10.680 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:27:10.680 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:27:10.680 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:27:10.680 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:10.681 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:27:10.681 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:42.138512+0000", 2024-09-13T11:27:10.681 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:27:10.681 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:10.681 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:10.681 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:27:10.681 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:42.138512+0000", 2024-09-13T11:27:10.681 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:27:10.681 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:10.681 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:10.681 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:27:10.681 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:42.138515+0000", 2024-09-13T11:27:10.681 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.9560000000000002e-06 2024-09-13T11:27:10.681 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:10.682 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:10.682 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:27:10.682 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:42.138524+0000", 2024-09-13T11:27:10.682 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.0350000000000007e-06 2024-09-13T11:27:10.682 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:10.682 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:10.682 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:27:10.682 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:42.378633+0000", 2024-09-13T11:27:10.682 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.24010957299999999 2024-09-13T11:27:10.682 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:10.682 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:10.682 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:27:10.682 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:42.378642+0000", 2024-09-13T11:27:10.682 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.7390000000000005e-06 2024-09-13T11:27:10.682 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:10.683 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:10.683 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:27:10.683 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:42.378709+0000", 2024-09-13T11:27:10.683 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.6586000000000002e-05 2024-09-13T11:27:10.683 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:10.683 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:10.683 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:27:10.683 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:42.378718+0000", 2024-09-13T11:27:10.683 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.5899999999999997e-06 2024-09-13T11:27:10.683 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:10.683 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:10.683 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:27:10.683 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:42.378854+0000", 2024-09-13T11:27:10.683 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00013509700000000001 2024-09-13T11:27:10.684 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:27:10.684 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:27:10.684 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:27:10.684 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:10.684 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:10.684 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDPGRecoveryDeleteReply(3.31 e232,227 3:8f08ebd6:::benchmark_data_smithi177_31116_object4865:head,148'5911)", 2024-09-13T11:27:10.684 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:26:59.958563+0000", 2024-09-13T11:27:10.684 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 10.652203346, 2024-09-13T11:27:10.684 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.22245790700000001, 2024-09-13T11:27:10.684 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:27:10.684 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:27:10.684 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-13T11:27:10.684 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:27:10.684 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:10.685 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:27:10.685 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:59.958563+0000", 2024-09-13T11:27:10.685 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:27:10.685 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:10.685 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:10.685 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:27:10.685 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:59.958563+0000", 2024-09-13T11:27:10.685 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:27:10.685 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:10.685 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:10.685 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:27:10.685 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:59.958565+0000", 2024-09-13T11:27:10.685 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.759e-06 2024-09-13T11:27:10.685 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:10.686 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:10.686 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:27:10.686 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:59.958572+0000", 2024-09-13T11:27:10.686 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.1500000000000002e-06 2024-09-13T11:27:10.686 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:10.686 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:10.686 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:27:10.686 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:27:00.180446+0000", 2024-09-13T11:27:10.686 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.22187378999999999 2024-09-13T11:27:10.686 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:10.686 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:10.686 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:27:10.686 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:27:00.180456+0000", 2024-09-13T11:27:10.686 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.9480000000000003e-06 2024-09-13T11:27:10.687 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:10.687 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:10.687 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:27:10.687 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:27:00.180531+0000", 2024-09-13T11:27:10.687 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.5436000000000001e-05 2024-09-13T11:27:10.687 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:10.687 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:10.687 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:27:10.687 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:27:00.181021+0000", 2024-09-13T11:27:10.687 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00048982400000000003 2024-09-13T11:27:10.688 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:27:10.688 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:27:10.688 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:27:10.688 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:10.688 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:10.688 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDPGRecoveryDelete(3.2f e237,227 3:f7b2a17e:::benchmark_data_smithi177_31116_object18197:head,153'6692)", 2024-09-13T11:27:10.688 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:27:05.269087+0000", 2024-09-13T11:27:10.688 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 5.3416801129999998, 2024-09-13T11:27:10.688 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.21128809000000001, 2024-09-13T11:27:10.688 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:27:10.688 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:27:10.688 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:27:10.688 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:27:10.689 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:10.689 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:27:10.689 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:27:05.269087+0000", 2024-09-13T11:27:10.689 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:27:10.689 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:10.689 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:10.689 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:27:10.689 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:27:05.269087+0000", 2024-09-13T11:27:10.689 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:27:10.689 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:10.689 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:10.689 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:27:10.689 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:27:05.269089+0000", 2024-09-13T11:27:10.689 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.1579999999999999e-06 2024-09-13T11:27:10.690 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:10.690 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:10.690 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:27:10.690 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:27:05.269095+0000", 2024-09-13T11:27:10.690 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.6769999999999998e-06 2024-09-13T11:27:10.690 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:10.690 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:10.690 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:27:10.690 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:27:05.480209+0000", 2024-09-13T11:27:10.690 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.21111343599999999 2024-09-13T11:27:10.690 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:10.690 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:10.690 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:27:10.690 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:27:05.480219+0000", 2024-09-13T11:27:10.691 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.6080000000000002e-06 2024-09-13T11:27:10.691 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:10.691 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:10.691 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:27:10.691 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:27:05.480285+0000", 2024-09-13T11:27:10.691 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.6260000000000006e-05 2024-09-13T11:27:10.691 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:10.691 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:10.691 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:27:10.691 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:27:05.480295+0000", 2024-09-13T11:27:10.691 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0265000000000001e-05 2024-09-13T11:27:10.691 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:10.691 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:10.691 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:27:10.691 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:27:05.480375+0000", 2024-09-13T11:27:10.692 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.9685999999999996e-05 2024-09-13T11:27:10.692 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:27:10.692 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:27:10.692 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:27:10.692 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:10.692 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:10.692 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDPGRecoveryDelete(3.2f e237,227 3:f51cf26c:::benchmark_data_smithi177_31116_object18255:head,153'6693)", 2024-09-13T11:27:10.692 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:27:05.278170+0000", 2024-09-13T11:27:10.692 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 5.3325963669999998, 2024-09-13T11:27:10.692 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.20993941099999999, 2024-09-13T11:27:10.692 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:27:10.692 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:27:10.692 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:27:10.693 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:27:10.693 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:10.693 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:27:10.693 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:27:05.278170+0000", 2024-09-13T11:27:10.693 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:27:10.693 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:10.693 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:10.693 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:27:10.693 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:27:05.278170+0000", 2024-09-13T11:27:10.693 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:27:10.693 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:10.693 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:10.693 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:27:10.693 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:27:05.278172+0000", 2024-09-13T11:27:10.693 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.0540000000000002e-06 2024-09-13T11:27:10.694 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:10.694 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:10.694 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:27:10.694 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:27:05.278179+0000", 2024-09-13T11:27:10.694 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.793e-06 2024-09-13T11:27:10.694 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:10.694 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:10.694 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:27:10.694 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:27:05.487933+0000", 2024-09-13T11:27:10.694 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.20975418900000001 2024-09-13T11:27:10.694 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:10.694 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:10.694 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:27:10.694 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:27:05.487941+0000", 2024-09-13T11:27:10.694 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.277e-06 2024-09-13T11:27:10.695 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:10.695 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:10.695 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:27:10.695 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:27:05.488006+0000", 2024-09-13T11:27:10.695 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.5524999999999994e-05 2024-09-13T11:27:10.695 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:10.695 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:10.695 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:27:10.695 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:27:05.488018+0000", 2024-09-13T11:27:10.695 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.1749999999999999e-05 2024-09-13T11:27:10.695 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:10.695 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:10.695 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:27:10.696 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:27:05.488110+0000", 2024-09-13T11:27:10.696 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.1823000000000005e-05 2024-09-13T11:27:10.696 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:27:10.696 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:27:10.696 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:27:10.696 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:10.696 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:10.696 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDPGRecoveryDelete(3.2f e237,227 3:f4fdd62b:::benchmark_data_smithi177_31116_object18349:head,153'6695)", 2024-09-13T11:27:10.696 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:27:05.487904+0000", 2024-09-13T11:27:10.696 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 5.1228626049999999, 2024-09-13T11:27:10.696 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.50371594200000003, 2024-09-13T11:27:10.696 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:27:10.696 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:27:10.697 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:27:10.697 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:27:10.697 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:10.697 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:27:10.697 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:27:05.487904+0000", 2024-09-13T11:27:10.697 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:27:10.697 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:10.697 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:10.697 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:27:10.697 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:27:05.487904+0000", 2024-09-13T11:27:10.697 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:27:10.697 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:10.697 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:10.698 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:27:10.698 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:27:05.487906+0000", 2024-09-13T11:27:10.698 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.6619999999999999e-06 2024-09-13T11:27:10.698 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:10.698 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:10.698 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:27:10.698 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:27:05.487912+0000", 2024-09-13T11:27:10.698 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.28e-06 2024-09-13T11:27:10.698 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:10.698 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:10.698 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:27:10.698 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:27:05.991429+0000", 2024-09-13T11:27:10.698 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.50351741800000005 2024-09-13T11:27:10.698 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:10.699 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:10.699 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:27:10.699 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:27:05.991438+0000", 2024-09-13T11:27:10.699 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.3359999999999995e-06 2024-09-13T11:27:10.699 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:10.699 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:10.699 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:27:10.699 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:27:05.991518+0000", 2024-09-13T11:27:10.699 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.9915999999999999e-05 2024-09-13T11:27:10.699 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:10.699 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:10.699 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:27:10.699 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:27:05.991528+0000", 2024-09-13T11:27:10.700 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.8199999999999992e-06 2024-09-13T11:27:10.700 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:10.700 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:10.700 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:27:10.700 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:27:05.991620+0000", 2024-09-13T11:27:10.700 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.2509999999999993e-05 2024-09-13T11:27:10.700 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:27:10.700 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:27:10.700 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:27:10.700 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:10.700 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:10.700 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDPGRecoveryDelete(3.2f e237,227 3:f5082d95:::benchmark_data_smithi177_31116_object18391:head,154'6696)", 2024-09-13T11:27:10.700 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:27:05.495151+0000", 2024-09-13T11:27:10.701 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 5.1156157699999998, 2024-09-13T11:27:10.701 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.496581518, 2024-09-13T11:27:10.701 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:27:10.701 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:27:10.701 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:27:10.701 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:27:10.701 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:10.701 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:27:10.701 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:27:05.495151+0000", 2024-09-13T11:27:10.701 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:27:10.701 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:10.701 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:10.701 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:27:10.701 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:27:05.495151+0000", 2024-09-13T11:27:10.702 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:27:10.702 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:10.702 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:10.702 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:27:10.702 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:27:05.495153+0000", 2024-09-13T11:27:10.702 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.368e-06 2024-09-13T11:27:10.702 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:10.702 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:10.702 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:27:10.702 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:27:05.495165+0000", 2024-09-13T11:27:10.702 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.1246e-05 2024-09-13T11:27:10.702 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:10.702 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:10.703 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:27:10.703 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:27:05.991466+0000", 2024-09-13T11:27:10.703 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.49630176799999998 2024-09-13T11:27:10.703 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:10.703 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:10.703 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:27:10.703 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:27:05.991473+0000", 2024-09-13T11:27:10.703 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.1999999999999999e-06 2024-09-13T11:27:10.703 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:10.703 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:10.703 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:27:10.703 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:27:05.991656+0000", 2024-09-13T11:27:10.703 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00018344800000000001 2024-09-13T11:27:10.703 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:10.704 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:10.704 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:27:10.704 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:27:05.991665+0000", 2024-09-13T11:27:10.704 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.1600000000000004e-06 2024-09-13T11:27:10.704 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:10.704 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:10.704 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:27:10.704 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:27:05.991732+0000", 2024-09-13T11:27:10.704 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.7328000000000003e-05 2024-09-13T11:27:10.704 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:27:10.704 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:27:10.704 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:27:10.704 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:10.704 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:10.705 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDPGRecoveryDeleteReply(3.31 e238,227 3:8f7e8810:::benchmark_data_smithi177_31116_object26861:head,160'6607)", 2024-09-13T11:27:10.705 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:27:06.081624+0000", 2024-09-13T11:27:10.705 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 4.5291423020000003, 2024-09-13T11:27:10.705 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.34901254300000001, 2024-09-13T11:27:10.705 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:27:10.705 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:27:10.705 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-13T11:27:10.705 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:27:10.705 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:10.705 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:27:10.705 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:27:06.081624+0000", 2024-09-13T11:27:10.705 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:27:10.705 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:10.705 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:10.706 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:27:10.706 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:27:06.081624+0000", 2024-09-13T11:27:10.706 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:27:10.706 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:10.706 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:10.706 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:27:10.706 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:27:06.081626+0000", 2024-09-13T11:27:10.706 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.432e-06 2024-09-13T11:27:10.706 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:10.706 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:10.706 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:27:10.706 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:27:06.081631+0000", 2024-09-13T11:27:10.706 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.5790000000000003e-06 2024-09-13T11:27:10.706 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:10.707 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:10.707 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:27:10.707 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:27:06.430157+0000", 2024-09-13T11:27:10.707 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.348525109 2024-09-13T11:27:10.707 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:10.707 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:10.707 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:27:10.707 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:27:06.430166+0000", 2024-09-13T11:27:10.707 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.7070000000000004e-06 2024-09-13T11:27:10.707 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:10.707 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:10.707 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:27:10.707 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:27:06.430232+0000", 2024-09-13T11:27:10.707 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.6223999999999995e-05 2024-09-13T11:27:10.708 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:10.708 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:10.708 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:27:10.708 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:27:06.430637+0000", 2024-09-13T11:27:10.708 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000404492 2024-09-13T11:27:10.708 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:27:10.708 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:27:10.708 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:27:10.708 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:27:10.708 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:27:10.708 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-13T11:27:10.710 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-09-13T11:27:10.710 INFO:teuthology.orchestra.run.smithi001.stdout:osd.5: {} 2024-09-13T11:27:10.710 INFO:teuthology.orchestra.run.smithi001.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T11:27:10.711 INFO:teuthology.orchestra.run.smithi001.stdout:osd.6: {} 2024-09-13T11:27:10.711 INFO:teuthology.orchestra.run.smithi001.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T11:27:10.711 INFO:teuthology.orchestra.run.smithi001.stdout:osd.7: {} 2024-09-13T11:27:10.711 INFO:teuthology.orchestra.run.smithi001.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T11:27:10.715 INFO:teuthology.orchestra.run.smithi001.stdout:osd.8: {} 2024-09-13T11:27:10.715 INFO:teuthology.orchestra.run.smithi001.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-13T11:27:10.715 INFO:tasks.ceph.osd.8.smithi001.stderr:2024-09-13T11:27:10.714+0000 7fb9b48ae640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26312) UID: 0 2024-09-13T11:27:10.749 INFO:teuthology.orchestra.run.smithi001.stdout:osd.9: {} 2024-09-13T11:27:10.749 INFO:teuthology.orchestra.run.smithi001.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-13T11:27:10.750 INFO:teuthology.orchestra.run.smithi001.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-09-13T11:27:10.781 INFO:teuthology.orchestra.run.smithi001.stdout:osd.11: {} 2024-09-13T11:27:10.781 INFO:teuthology.orchestra.run.smithi001.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-13T11:27:10.806 INFO:teuthology.orchestra.run.smithi001.stdout:osd.12: {} 2024-09-13T11:27:10.806 INFO:teuthology.orchestra.run.smithi001.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-13T11:27:10.813 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-09-13T11:27:10.811+0000 7f97befc9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26315) UID: 0 2024-09-13T11:27:10.837 INFO:teuthology.orchestra.run.smithi001.stdout:osd.13: {} 2024-09-13T11:27:10.838 INFO:teuthology.orchestra.run.smithi001.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-13T11:27:10.855 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-13T11:27:10.856 INFO:teuthology.orchestra.run.smithi001.stdout: "ops": [], 2024-09-13T11:27:10.856 INFO:teuthology.orchestra.run.smithi001.stdout: "num_ops": 0 2024-09-13T11:27:10.856 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-13T11:27:10.868 INFO:teuthology.orchestra.run.smithi001.stdout:osd.14: {} 2024-09-13T11:27:10.868 INFO:teuthology.orchestra.run.smithi001.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-13T11:27:10.872 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-09-13T11:27:10.894 INFO:teuthology.orchestra.run.smithi001.stdout:osd.15: {} 2024-09-13T11:27:10.895 INFO:teuthology.orchestra.run.smithi001.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-13T11:27:10.897 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-13T11:27:10.897 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-13T11:27:10.914 INFO:tasks.ceph.osd.2.smithi162.stderr:2024-09-13T11:27:10.911+0000 7fe92b35a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26313) UID: 0 2024-09-13T11:27:11.014 INFO:tasks.ceph.osd.4.smithi001.stderr:2024-09-13T11:27:11.010+0000 7ff8e0218640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26310) UID: 0 2024-09-13T11:27:11.033 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-13T11:27:11.037 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-13T11:27:11.037 INFO:teuthology.orchestra.run.smithi001.stdout: "ops": [], 2024-09-13T11:27:11.037 INFO:teuthology.orchestra.run.smithi001.stdout: "complaint_time": 30, 2024-09-13T11:27:11.037 INFO:teuthology.orchestra.run.smithi001.stdout: "num_blocked_ops": 0 2024-09-13T11:27:11.037 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-13T11:27:11.055 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-09-13T11:27:11.115 INFO:tasks.ceph.osd.11.smithi177.stderr:2024-09-13T11:27:11.113+0000 7f24792c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26580) UID: 0 2024-09-13T11:27:11.215 INFO:tasks.ceph.osd.11.smithi177.stderr:2024-09-13T11:27:11.213+0000 7f24792c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26580) UID: 0 2024-09-13T11:27:11.244 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-13T11:27:11.244 INFO:teuthology.orchestra.run.smithi001.stdout: "size": 20, 2024-09-13T11:27:11.244 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 600, 2024-09-13T11:27:11.244 INFO:teuthology.orchestra.run.smithi001.stdout: "ops": [ 2024-09-13T11:27:11.245 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.245 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:54656 3.1e 3.6a49163e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e75)", 2024-09-13T11:27:11.245 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:30.979993+0000", 2024-09-13T11:27:11.245 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 220.26251436199999, 2024-09-13T11:27:11.245 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.9269557150000001, 2024-09-13T11:27:11.245 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:27:11.245 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:27:11.245 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-13T11:27:11.245 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:27:11.246 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:27:11.246 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:27:11.246 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 54656 2024-09-13T11:27:11.246 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.246 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:27:11.246 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.246 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:27:11.246 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:30.979993+0000", 2024-09-13T11:27:11.246 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:27:11.246 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.247 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.247 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:27:11.247 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:30.979993+0000", 2024-09-13T11:27:11.247 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:27:11.247 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.247 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.247 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:27:11.247 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:30.979997+0000", 2024-09-13T11:27:11.247 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.0319999999999997e-06 2024-09-13T11:27:11.247 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.247 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.248 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:27:11.248 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:30.980057+0000", 2024-09-13T11:27:11.248 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.0341999999999999e-05 2024-09-13T11:27:11.248 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.248 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.248 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:27:11.248 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:30.980094+0000", 2024-09-13T11:27:11.248 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.6770000000000002e-05 2024-09-13T11:27:11.248 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.248 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.248 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:27:11.248 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:30.980113+0000", 2024-09-13T11:27:11.248 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.8791999999999999e-05 2024-09-13T11:27:11.249 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.249 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.249 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:27:11.249 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.097773+0000", 2024-09-13T11:27:11.249 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.117660379 2024-09-13T11:27:11.249 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.249 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.249 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for peered", 2024-09-13T11:27:11.249 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.097784+0000", 2024-09-13T11:27:11.249 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.1025e-05 2024-09-13T11:27:11.249 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.249 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.249 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:27:11.249 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.906937+0000", 2024-09-13T11:27:11.250 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.8091523789999999 2024-09-13T11:27:11.250 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.250 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.250 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:27:11.250 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.906949+0000", 2024-09-13T11:27:11.250 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.1996e-05 2024-09-13T11:27:11.250 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:27:11.250 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:27:11.250 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:27:11.250 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.250 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.250 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:54720 3.1e 3.a944303e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-13T11:27:11.250 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:31.973315+0000", 2024-09-13T11:27:11.251 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 219.269192127, 2024-09-13T11:27:11.251 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.93366527200000005, 2024-09-13T11:27:11.251 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:27:11.251 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:27:11.251 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-13T11:27:11.251 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:27:11.251 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:27:11.251 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:27:11.251 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 54720 2024-09-13T11:27:11.251 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.251 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:27:11.251 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.251 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:27:11.251 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.973315+0000", 2024-09-13T11:27:11.252 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:27:11.252 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.252 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.252 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:27:11.252 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.973315+0000", 2024-09-13T11:27:11.252 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:27:11.252 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.252 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.252 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:27:11.252 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.973318+0000", 2024-09-13T11:27:11.252 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.5830000000000001e-06 2024-09-13T11:27:11.252 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.252 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.253 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:27:11.253 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.973465+0000", 2024-09-13T11:27:11.253 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000146816 2024-09-13T11:27:11.253 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.253 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.253 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:27:11.253 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.973478+0000", 2024-09-13T11:27:11.253 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2931e-05 2024-09-13T11:27:11.253 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.253 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.253 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:27:11.253 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.973483+0000", 2024-09-13T11:27:11.253 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.8969999999999996e-06 2024-09-13T11:27:11.254 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.254 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.254 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:27:11.254 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.973548+0000", 2024-09-13T11:27:11.254 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.4771999999999996e-05 2024-09-13T11:27:11.254 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.254 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.254 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for peered", 2024-09-13T11:27:11.254 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.973557+0000", 2024-09-13T11:27:11.254 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.0359999999999995e-06 2024-09-13T11:27:11.254 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.254 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.254 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:27:11.254 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.906974+0000", 2024-09-13T11:27:11.255 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.93341706000000002 2024-09-13T11:27:11.255 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.255 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.255 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:27:11.255 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.906980+0000", 2024-09-13T11:27:11.255 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.1770000000000003e-06 2024-09-13T11:27:11.255 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:27:11.255 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:27:11.255 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:27:11.255 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.255 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.255 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:54734 3.1e 3.d7f6f7e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-13T11:27:11.255 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:31.983342+0000", 2024-09-13T11:27:11.255 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 219.25916532400001, 2024-09-13T11:27:11.256 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.92367058599999996, 2024-09-13T11:27:11.256 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:27:11.256 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:27:11.256 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-13T11:27:11.256 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:27:11.256 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:27:11.256 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:27:11.256 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 54734 2024-09-13T11:27:11.256 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.256 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:27:11.256 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.256 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:27:11.256 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.983342+0000", 2024-09-13T11:27:11.257 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:27:11.257 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.257 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.257 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:27:11.257 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.983342+0000", 2024-09-13T11:27:11.257 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:27:11.257 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.257 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.257 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:27:11.257 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.983345+0000", 2024-09-13T11:27:11.257 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.6220000000000001e-06 2024-09-13T11:27:11.257 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.257 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.257 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:27:11.258 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.983438+0000", 2024-09-13T11:27:11.258 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.3517000000000006e-05 2024-09-13T11:27:11.258 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.258 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.258 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:27:11.258 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.983451+0000", 2024-09-13T11:27:11.258 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2527999999999999e-05 2024-09-13T11:27:11.258 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.258 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.258 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:27:11.258 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.983458+0000", 2024-09-13T11:27:11.258 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.6380000000000004e-06 2024-09-13T11:27:11.258 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.258 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.259 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:27:11.259 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.983521+0000", 2024-09-13T11:27:11.259 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.2729000000000002e-05 2024-09-13T11:27:11.259 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.259 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.259 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for peered", 2024-09-13T11:27:11.259 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.983548+0000", 2024-09-13T11:27:11.259 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.6537999999999999e-05 2024-09-13T11:27:11.259 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.259 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.259 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:27:11.259 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.907005+0000", 2024-09-13T11:27:11.259 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.923456991 2024-09-13T11:27:11.260 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.260 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.260 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:27:11.260 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.907013+0000", 2024-09-13T11:27:11.260 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.0229999999999992e-06 2024-09-13T11:27:11.260 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:27:11.260 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:27:11.260 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:27:11.260 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.260 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.260 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:54760 3.1e 3.8c32c13e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-13T11:27:11.260 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:31.999685+0000", 2024-09-13T11:27:11.260 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 219.24282209200001, 2024-09-13T11:27:11.261 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.90735639599999995, 2024-09-13T11:27:11.261 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:27:11.261 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:27:11.261 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-13T11:27:11.261 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:27:11.261 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:27:11.261 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:27:11.261 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 54760 2024-09-13T11:27:11.261 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.261 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:27:11.261 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.261 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:27:11.261 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.999685+0000", 2024-09-13T11:27:11.261 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:27:11.262 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.262 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.262 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:27:11.262 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.999685+0000", 2024-09-13T11:27:11.262 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:27:11.262 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.262 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.262 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:27:11.262 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.999690+0000", 2024-09-13T11:27:11.262 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.7879999999999997e-06 2024-09-13T11:27:11.262 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.262 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.262 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:27:11.262 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.999729+0000", 2024-09-13T11:27:11.263 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.9032000000000002e-05 2024-09-13T11:27:11.263 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.263 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.263 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:27:11.263 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.999741+0000", 2024-09-13T11:27:11.263 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2136999999999999e-05 2024-09-13T11:27:11.263 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.263 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.263 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:27:11.263 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.999748+0000", 2024-09-13T11:27:11.263 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.9120000000000001e-06 2024-09-13T11:27:11.263 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.263 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.264 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:27:11.264 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.999803+0000", 2024-09-13T11:27:11.264 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.5064e-05 2024-09-13T11:27:11.264 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.264 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.264 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for peered", 2024-09-13T11:27:11.264 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:31.999815+0000", 2024-09-13T11:27:11.264 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2041e-05 2024-09-13T11:27:11.264 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.265 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.265 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:27:11.265 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.907034+0000", 2024-09-13T11:27:11.265 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.90721858200000005 2024-09-13T11:27:11.265 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.265 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.265 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:27:11.265 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.907042+0000", 2024-09-13T11:27:11.265 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.8399999999999995e-06 2024-09-13T11:27:11.265 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:27:11.265 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:27:11.265 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:27:11.265 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.265 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.266 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:54768 3.1e 3.6dad499e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-13T11:27:11.266 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:32.003482+0000", 2024-09-13T11:27:11.266 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 219.23902544200001, 2024-09-13T11:27:11.266 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.90358804699999995, 2024-09-13T11:27:11.266 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:27:11.266 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:27:11.266 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-13T11:27:11.266 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:27:11.266 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:27:11.266 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:27:11.266 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 54768 2024-09-13T11:27:11.266 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.266 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:27:11.267 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.267 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:27:11.267 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.003482+0000", 2024-09-13T11:27:11.267 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:27:11.267 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.267 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.267 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:27:11.267 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.003482+0000", 2024-09-13T11:27:11.267 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:27:11.267 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.267 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.267 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:27:11.267 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.003486+0000", 2024-09-13T11:27:11.267 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.1030000000000004e-06 2024-09-13T11:27:11.268 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.268 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.268 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:27:11.268 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.003523+0000", 2024-09-13T11:27:11.268 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.6732e-05 2024-09-13T11:27:11.268 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.268 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.268 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:27:11.268 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.003535+0000", 2024-09-13T11:27:11.268 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2520999999999999e-05 2024-09-13T11:27:11.268 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.268 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.268 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:27:11.269 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.003541+0000", 2024-09-13T11:27:11.269 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.8690000000000001e-06 2024-09-13T11:27:11.269 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.269 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.269 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:27:11.269 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.003593+0000", 2024-09-13T11:27:11.269 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.1786000000000002e-05 2024-09-13T11:27:11.269 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.269 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.269 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for peered", 2024-09-13T11:27:11.269 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.003602+0000", 2024-09-13T11:27:11.269 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.9679999999999995e-06 2024-09-13T11:27:11.269 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.269 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.270 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:27:11.270 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.907063+0000", 2024-09-13T11:27:11.270 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.90346116600000004 2024-09-13T11:27:11.270 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.270 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.270 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:27:11.270 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.907070+0000", 2024-09-13T11:27:11.270 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.9020000000000002e-06 2024-09-13T11:27:11.270 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:27:11.270 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:27:11.270 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:27:11.270 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.270 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.270 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:54785 3.1e 3.d32f1ede (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-13T11:27:11.271 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:32.016238+0000", 2024-09-13T11:27:11.271 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 219.22626923199999, 2024-09-13T11:27:11.271 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.89086243399999998, 2024-09-13T11:27:11.271 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:27:11.271 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:27:11.271 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-13T11:27:11.271 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:27:11.271 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:27:11.271 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:27:11.271 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 54785 2024-09-13T11:27:11.271 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.271 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:27:11.271 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.271 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:27:11.272 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.016238+0000", 2024-09-13T11:27:11.272 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:27:11.272 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.272 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.272 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:27:11.272 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.016238+0000", 2024-09-13T11:27:11.272 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:27:11.272 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.272 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.272 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:27:11.272 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.016241+0000", 2024-09-13T11:27:11.272 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.6469999999999998e-06 2024-09-13T11:27:11.272 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.272 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.273 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:27:11.273 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.016271+0000", 2024-09-13T11:27:11.273 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.0505999999999999e-05 2024-09-13T11:27:11.273 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.273 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.273 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:27:11.273 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.016287+0000", 2024-09-13T11:27:11.273 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.6073999999999999e-05 2024-09-13T11:27:11.273 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.273 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.273 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:27:11.273 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.016295+0000", 2024-09-13T11:27:11.273 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.0590000000000003e-06 2024-09-13T11:27:11.273 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.274 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.274 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:27:11.274 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.016373+0000", 2024-09-13T11:27:11.274 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.7634999999999998e-05 2024-09-13T11:27:11.274 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.274 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.274 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for peered", 2024-09-13T11:27:11.274 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.016383+0000", 2024-09-13T11:27:11.274 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0246e-05 2024-09-13T11:27:11.274 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.274 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.274 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:27:11.274 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.907092+0000", 2024-09-13T11:27:11.275 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.89070849200000002 2024-09-13T11:27:11.275 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.275 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.275 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:27:11.275 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.907101+0000", 2024-09-13T11:27:11.275 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.7749999999999998e-06 2024-09-13T11:27:11.275 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:27:11.275 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:27:11.275 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:27:11.275 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.275 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.275 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:54811 3.1e 3.3754367e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-13T11:27:11.275 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:32.045424+0000", 2024-09-13T11:27:11.275 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 219.19708358099999, 2024-09-13T11:27:11.276 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.86170719399999995, 2024-09-13T11:27:11.276 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:27:11.276 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:27:11.276 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-13T11:27:11.276 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:27:11.276 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:27:11.276 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:27:11.276 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 54811 2024-09-13T11:27:11.276 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.276 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:27:11.276 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.276 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:27:11.276 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.045424+0000", 2024-09-13T11:27:11.276 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:27:11.277 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.277 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.277 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:27:11.277 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.045424+0000", 2024-09-13T11:27:11.277 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:27:11.277 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.277 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.277 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:27:11.277 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.045426+0000", 2024-09-13T11:27:11.277 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.0260000000000002e-06 2024-09-13T11:27:11.277 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.277 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.277 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:27:11.278 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.045556+0000", 2024-09-13T11:27:11.278 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00013003099999999999 2024-09-13T11:27:11.278 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.278 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.278 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:27:11.278 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.045568+0000", 2024-09-13T11:27:11.278 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.1727e-05 2024-09-13T11:27:11.278 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.278 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.278 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:27:11.278 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.045576+0000", 2024-09-13T11:27:11.278 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.6459999999999999e-06 2024-09-13T11:27:11.278 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.278 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.278 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:27:11.279 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.045650+0000", 2024-09-13T11:27:11.279 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.4000999999999998e-05 2024-09-13T11:27:11.279 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.279 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.279 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for peered", 2024-09-13T11:27:11.279 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.045658+0000", 2024-09-13T11:27:11.279 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.3869999999999997e-06 2024-09-13T11:27:11.279 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.279 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.279 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:27:11.279 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.907123+0000", 2024-09-13T11:27:11.279 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.86146508399999999 2024-09-13T11:27:11.279 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.279 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.280 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:27:11.280 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.907131+0000", 2024-09-13T11:27:11.280 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.2919999999999996e-06 2024-09-13T11:27:11.280 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:27:11.280 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:27:11.280 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:27:11.280 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.280 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.280 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:54817 3.1e 3.36dd2fde (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-13T11:27:11.280 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:32.050108+0000", 2024-09-13T11:27:11.280 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 219.19239968900001, 2024-09-13T11:27:11.280 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.85705549199999997, 2024-09-13T11:27:11.280 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:27:11.281 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:27:11.281 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-13T11:27:11.281 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:27:11.281 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:27:11.281 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:27:11.281 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 54817 2024-09-13T11:27:11.281 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.281 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:27:11.281 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.281 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:27:11.281 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.050108+0000", 2024-09-13T11:27:11.281 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:27:11.281 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.281 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.282 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:27:11.282 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.050108+0000", 2024-09-13T11:27:11.282 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:27:11.282 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.282 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.282 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:27:11.282 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.050110+0000", 2024-09-13T11:27:11.282 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.0209999999999998e-06 2024-09-13T11:27:11.282 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.282 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.282 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:27:11.282 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.050166+0000", 2024-09-13T11:27:11.282 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.6026000000000002e-05 2024-09-13T11:27:11.282 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.283 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.283 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:27:11.283 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.050180+0000", 2024-09-13T11:27:11.283 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.4307e-05 2024-09-13T11:27:11.283 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.283 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.283 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:27:11.283 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.050188+0000", 2024-09-13T11:27:11.283 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.5410000000000003e-06 2024-09-13T11:27:11.283 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.283 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.283 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:27:11.283 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.050260+0000", 2024-09-13T11:27:11.283 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.2157999999999996e-05 2024-09-13T11:27:11.284 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.284 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.284 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for peered", 2024-09-13T11:27:11.284 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.050267+0000", 2024-09-13T11:27:11.284 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.0190000000000001e-06 2024-09-13T11:27:11.284 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.284 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.284 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:27:11.284 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.907155+0000", 2024-09-13T11:27:11.284 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.85688824399999997 2024-09-13T11:27:11.284 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.284 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.284 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:27:11.284 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.907163+0000", 2024-09-13T11:27:11.285 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.1759999999999993e-06 2024-09-13T11:27:11.285 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:27:11.285 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:27:11.285 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:27:11.285 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.285 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.285 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:54818 3.1e 3.5cfcc69e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-13T11:27:11.285 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:32.051374+0000", 2024-09-13T11:27:11.285 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 219.19113327299999, 2024-09-13T11:27:11.285 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.85581781000000001, 2024-09-13T11:27:11.285 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:27:11.285 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:27:11.285 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-13T11:27:11.286 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:27:11.286 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:27:11.286 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:27:11.286 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 54818 2024-09-13T11:27:11.286 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.286 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:27:11.286 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.286 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:27:11.286 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.051374+0000", 2024-09-13T11:27:11.286 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:27:11.286 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.286 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.286 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:27:11.286 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.051374+0000", 2024-09-13T11:27:11.287 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:27:11.287 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.287 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.287 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:27:11.287 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.051375+0000", 2024-09-13T11:27:11.287 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.4440000000000001e-06 2024-09-13T11:27:11.287 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.287 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.287 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:27:11.287 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.051426+0000", 2024-09-13T11:27:11.287 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.0809000000000001e-05 2024-09-13T11:27:11.287 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.287 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.287 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:27:11.288 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.051441+0000", 2024-09-13T11:27:11.288 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.4460999999999999e-05 2024-09-13T11:27:11.288 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.288 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.288 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:27:11.288 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.051449+0000", 2024-09-13T11:27:11.288 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.4880000000000007e-06 2024-09-13T11:27:11.288 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.288 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.288 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:27:11.288 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.051499+0000", 2024-09-13T11:27:11.288 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.9691999999999998e-05 2024-09-13T11:27:11.288 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.289 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.289 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for peered", 2024-09-13T11:27:11.289 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.051505+0000", 2024-09-13T11:27:11.289 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.2029999999999997e-06 2024-09-13T11:27:11.289 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.289 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.289 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:27:11.289 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.907184+0000", 2024-09-13T11:27:11.289 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.85567906599999999 2024-09-13T11:27:11.289 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.289 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.289 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:27:11.289 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.907192+0000", 2024-09-13T11:27:11.290 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.6469999999999998e-06 2024-09-13T11:27:11.290 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:27:11.290 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:27:11.290 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:27:11.290 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.290 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.290 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:54828 3.1e 3.e920c11e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-13T11:27:11.290 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:32.063344+0000", 2024-09-13T11:27:11.290 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 219.179162846, 2024-09-13T11:27:11.290 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.84387899099999997, 2024-09-13T11:27:11.290 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:27:11.290 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:27:11.290 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-13T11:27:11.290 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:27:11.291 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:27:11.291 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:27:11.291 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 54828 2024-09-13T11:27:11.291 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.291 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:27:11.291 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.291 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:27:11.291 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.063344+0000", 2024-09-13T11:27:11.291 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:27:11.291 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.291 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.292 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:27:11.292 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.063344+0000", 2024-09-13T11:27:11.292 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:27:11.292 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.292 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.292 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:27:11.292 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.063347+0000", 2024-09-13T11:27:11.292 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.2299999999999998e-06 2024-09-13T11:27:11.292 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.292 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.292 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:27:11.292 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.063384+0000", 2024-09-13T11:27:11.292 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.7290999999999999e-05 2024-09-13T11:27:11.292 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.293 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.293 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:27:11.293 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.063397+0000", 2024-09-13T11:27:11.293 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2955999999999999e-05 2024-09-13T11:27:11.293 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.293 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.293 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:27:11.293 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.063404+0000", 2024-09-13T11:27:11.293 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.4889999999999998e-06 2024-09-13T11:27:11.293 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.293 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.293 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:27:11.293 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.063470+0000", 2024-09-13T11:27:11.294 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.5516000000000001e-05 2024-09-13T11:27:11.294 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.294 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.294 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for peered", 2024-09-13T11:27:11.294 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.063482+0000", 2024-09-13T11:27:11.294 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2065000000000001e-05 2024-09-13T11:27:11.294 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.294 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.294 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:27:11.294 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.907216+0000", 2024-09-13T11:27:11.294 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.84373388900000001 2024-09-13T11:27:11.294 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.294 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.294 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:27:11.295 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.907223+0000", 2024-09-13T11:27:11.295 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.5549999999999997e-06 2024-09-13T11:27:11.295 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:27:11.295 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:27:11.295 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:27:11.295 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.295 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.295 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:54829 3.1e 3.192f62be (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-13T11:27:11.295 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:32.065642+0000", 2024-09-13T11:27:11.295 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 219.17686577800001, 2024-09-13T11:27:11.296 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.84161483100000001, 2024-09-13T11:27:11.296 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:27:11.296 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:27:11.296 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-13T11:27:11.296 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:27:11.296 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:27:11.296 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:27:11.296 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 54829 2024-09-13T11:27:11.296 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.297 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:27:11.297 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.297 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:27:11.297 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.065642+0000", 2024-09-13T11:27:11.297 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:27:11.297 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.298 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.298 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:27:11.298 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.065642+0000", 2024-09-13T11:27:11.298 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:27:11.298 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.298 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.298 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:27:11.298 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.065644+0000", 2024-09-13T11:27:11.298 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.1780000000000002e-06 2024-09-13T11:27:11.298 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.298 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.299 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:27:11.299 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.065695+0000", 2024-09-13T11:27:11.299 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.1632000000000003e-05 2024-09-13T11:27:11.299 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.299 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.299 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:27:11.299 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.065711+0000", 2024-09-13T11:27:11.299 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.5522000000000001e-05 2024-09-13T11:27:11.299 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.299 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.299 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:27:11.299 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.065719+0000", 2024-09-13T11:27:11.299 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.3480000000000004e-06 2024-09-13T11:27:11.299 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.300 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.300 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:27:11.300 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.065785+0000", 2024-09-13T11:27:11.300 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.5627e-05 2024-09-13T11:27:11.300 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.300 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.300 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for peered", 2024-09-13T11:27:11.300 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.065797+0000", 2024-09-13T11:27:11.300 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.1766000000000001e-05 2024-09-13T11:27:11.301 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.301 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.301 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:27:11.301 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.907248+0000", 2024-09-13T11:27:11.301 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.84145150300000005 2024-09-13T11:27:11.301 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.301 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.301 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:27:11.301 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.907256+0000", 2024-09-13T11:27:11.301 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.2549999999999997e-06 2024-09-13T11:27:11.301 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:27:11.301 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:27:11.301 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:27:11.301 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.302 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.302 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:54830 3.1e 3.b31554de (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-13T11:27:11.302 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:32.068259+0000", 2024-09-13T11:27:11.302 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 219.17424790999999, 2024-09-13T11:27:11.302 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.83902489599999996, 2024-09-13T11:27:11.302 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:27:11.302 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:27:11.302 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-13T11:27:11.302 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:27:11.302 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:27:11.302 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:27:11.302 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 54830 2024-09-13T11:27:11.302 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.302 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:27:11.303 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.303 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:27:11.303 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.068259+0000", 2024-09-13T11:27:11.303 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:27:11.303 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.303 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.303 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:27:11.303 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.068259+0000", 2024-09-13T11:27:11.303 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:27:11.303 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.303 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.303 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:27:11.303 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.068262+0000", 2024-09-13T11:27:11.303 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.1799999999999999e-06 2024-09-13T11:27:11.304 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.304 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.304 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:27:11.304 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.068301+0000", 2024-09-13T11:27:11.304 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.9289999999999998e-05 2024-09-13T11:27:11.304 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.304 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.304 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:27:11.304 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.068314+0000", 2024-09-13T11:27:11.304 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.3502999999999999e-05 2024-09-13T11:27:11.304 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.304 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.304 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:27:11.304 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.068324+0000", 2024-09-13T11:27:11.305 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.1630000000000002e-06 2024-09-13T11:27:11.305 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.305 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.305 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:27:11.305 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.068375+0000", 2024-09-13T11:27:11.305 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.1805000000000003e-05 2024-09-13T11:27:11.305 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.305 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.305 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for peered", 2024-09-13T11:27:11.305 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.068382+0000", 2024-09-13T11:27:11.305 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.224e-06 2024-09-13T11:27:11.305 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.305 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.306 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:27:11.306 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.907277+0000", 2024-09-13T11:27:11.306 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.83889509500000004 2024-09-13T11:27:11.306 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.306 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.306 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:27:11.306 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.907284+0000", 2024-09-13T11:27:11.306 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.6359999999999994e-06 2024-09-13T11:27:11.306 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:27:11.306 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:27:11.307 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:27:11.307 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.307 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.307 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:54846 3.1e 3.742e049e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-13T11:27:11.307 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:32.153567+0000", 2024-09-13T11:27:11.307 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 219.088940686, 2024-09-13T11:27:11.307 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.75378642399999995, 2024-09-13T11:27:11.307 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:27:11.307 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:27:11.307 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-13T11:27:11.307 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:27:11.307 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:27:11.307 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:27:11.308 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 54846 2024-09-13T11:27:11.308 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.308 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:27:11.308 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.308 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:27:11.308 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.153567+0000", 2024-09-13T11:27:11.308 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:27:11.308 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.308 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.308 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:27:11.309 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.153567+0000", 2024-09-13T11:27:11.309 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:27:11.309 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.309 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.309 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:27:11.309 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.153569+0000", 2024-09-13T11:27:11.309 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.4660000000000002e-06 2024-09-13T11:27:11.309 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.309 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.309 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:27:11.309 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.153619+0000", 2024-09-13T11:27:11.309 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.9617000000000003e-05 2024-09-13T11:27:11.309 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.310 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.310 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:27:11.310 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.153634+0000", 2024-09-13T11:27:11.310 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.4904e-05 2024-09-13T11:27:11.310 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.310 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.310 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:27:11.310 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.153642+0000", 2024-09-13T11:27:11.310 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.5599999999999994e-06 2024-09-13T11:27:11.310 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.310 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.310 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:27:11.310 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.153721+0000", 2024-09-13T11:27:11.310 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.8603999999999995e-05 2024-09-13T11:27:11.311 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.311 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.311 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for peered", 2024-09-13T11:27:11.311 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.153730+0000", 2024-09-13T11:27:11.311 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.3840000000000003e-06 2024-09-13T11:27:11.311 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.311 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.311 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:27:11.311 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.907346+0000", 2024-09-13T11:27:11.311 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.75361563099999995 2024-09-13T11:27:11.311 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.311 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.311 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:27:11.312 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.907353+0000", 2024-09-13T11:27:11.312 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.2579999999999998e-06 2024-09-13T11:27:11.312 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:27:11.312 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:27:11.312 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:27:11.312 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.312 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.312 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:54888 3.1e 3.43e14ede (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-13T11:27:11.312 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:32.423064+0000", 2024-09-13T11:27:11.312 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 218.81944301499999, 2024-09-13T11:27:11.312 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.48433585299999998, 2024-09-13T11:27:11.312 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:27:11.312 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:27:11.313 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-13T11:27:11.313 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:27:11.313 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:27:11.313 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:27:11.313 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 54888 2024-09-13T11:27:11.313 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.313 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:27:11.313 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.313 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:27:11.313 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.423064+0000", 2024-09-13T11:27:11.313 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:27:11.313 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.313 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.313 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:27:11.314 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.423064+0000", 2024-09-13T11:27:11.314 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:27:11.314 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.314 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.314 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:27:11.314 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.423069+0000", 2024-09-13T11:27:11.314 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.7389999999999999e-06 2024-09-13T11:27:11.314 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.314 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.314 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:27:11.314 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.423115+0000", 2024-09-13T11:27:11.315 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.5847000000000003e-05 2024-09-13T11:27:11.315 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.315 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.315 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:27:11.315 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.423126+0000", 2024-09-13T11:27:11.315 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.1418e-05 2024-09-13T11:27:11.315 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.315 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.315 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:27:11.315 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.423132+0000", 2024-09-13T11:27:11.315 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.57e-06 2024-09-13T11:27:11.315 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.315 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.315 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:27:11.316 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.423191+0000", 2024-09-13T11:27:11.316 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.9250999999999999e-05 2024-09-13T11:27:11.316 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.316 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.316 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for peered", 2024-09-13T11:27:11.316 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.423198+0000", 2024-09-13T11:27:11.316 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.9179999999999997e-06 2024-09-13T11:27:11.316 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.316 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.316 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:27:11.316 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.907392+0000", 2024-09-13T11:27:11.316 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.48419437500000001 2024-09-13T11:27:11.316 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.316 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.317 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:27:11.317 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:32.907400+0000", 2024-09-13T11:27:11.317 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.7349999999999996e-06 2024-09-13T11:27:11.317 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:27:11.317 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:27:11.317 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:27:11.317 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.317 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.317 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4665.0:59977 3.21 e83/78 3:86b4d8f6:::benchmark_data_smithi177_31116_object59976:head v 83'3136, mlcod=0'0)", 2024-09-13T11:27:11.317 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:40.571147+0000", 2024-09-13T11:27:11.317 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 210.671360566, 2024-09-13T11:27:11.317 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.51907542799999995, 2024-09-13T11:27:11.317 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:27:11.317 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:27:11.318 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:27:11.318 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:27:11.318 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.318 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:27:11.318 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:40.571147+0000", 2024-09-13T11:27:11.318 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:27:11.318 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.318 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.318 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:27:11.318 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:40.571147+0000", 2024-09-13T11:27:11.318 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:27:11.318 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.318 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.318 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:27:11.319 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:40.571151+0000", 2024-09-13T11:27:11.319 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.5650000000000003e-06 2024-09-13T11:27:11.319 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.319 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.319 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:27:11.319 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:40.571204+0000", 2024-09-13T11:27:11.319 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.2720999999999999e-05 2024-09-13T11:27:11.319 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.319 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.319 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:27:11.319 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:41.088332+0000", 2024-09-13T11:27:11.319 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.51712822000000003 2024-09-13T11:27:11.319 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.319 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.320 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:27:11.320 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:41.088341+0000", 2024-09-13T11:27:11.320 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.3100000000000001e-06 2024-09-13T11:27:11.320 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.320 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.320 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:27:11.320 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:41.088444+0000", 2024-09-13T11:27:11.320 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000103708 2024-09-13T11:27:11.320 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.320 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.320 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:27:11.320 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:41.088475+0000", 2024-09-13T11:27:11.320 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.0497e-05 2024-09-13T11:27:11.321 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.321 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.321 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:27:11.321 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:41.090192+0000", 2024-09-13T11:27:11.321 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.001717288 2024-09-13T11:27:11.321 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.321 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.321 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:27:11.321 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:41.090222+0000", 2024-09-13T11:27:11.321 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.0119000000000001e-05 2024-09-13T11:27:11.321 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:27:11.321 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:27:11.321 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:27:11.322 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.322 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.322 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:110286 3.26 3.c41c0fa6 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e119)", 2024-09-13T11:27:11.322 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:24:25.654272+0000", 2024-09-13T11:27:11.322 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 165.58823487399999, 2024-09-13T11:27:11.322 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.43337235400000002, 2024-09-13T11:27:11.322 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:27:11.322 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:27:11.322 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-13T11:27:11.322 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:27:11.322 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:27:11.322 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:27:11.322 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 110286 2024-09-13T11:27:11.322 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.323 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:27:11.323 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.323 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:27:11.323 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:25.654272+0000", 2024-09-13T11:27:11.323 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:27:11.323 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.323 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.323 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:27:11.323 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:25.654272+0000", 2024-09-13T11:27:11.323 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:27:11.323 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.323 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.323 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:27:11.323 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:25.654275+0000", 2024-09-13T11:27:11.324 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.3199999999999998e-06 2024-09-13T11:27:11.324 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.324 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.324 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:27:11.324 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:25.654517+0000", 2024-09-13T11:27:11.324 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00024270199999999999 2024-09-13T11:27:11.324 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.324 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.324 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:27:11.324 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:25.654550+0000", 2024-09-13T11:27:11.324 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.2549000000000003e-05 2024-09-13T11:27:11.324 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.324 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.325 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:27:11.325 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:25.654566+0000", 2024-09-13T11:27:11.325 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.5517000000000001e-05 2024-09-13T11:27:11.325 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.325 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.325 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:27:11.325 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:25.654655+0000", 2024-09-13T11:27:11.325 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.9961000000000001e-05 2024-09-13T11:27:11.325 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.325 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.325 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for peered", 2024-09-13T11:27:11.325 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:25.654665+0000", 2024-09-13T11:27:11.325 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.0520000000000007e-06 2024-09-13T11:27:11.325 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.326 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.326 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:27:11.326 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:26.087629+0000", 2024-09-13T11:27:11.326 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.43296476900000003 2024-09-13T11:27:11.326 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.326 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.326 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:27:11.326 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:26.087645+0000", 2024-09-13T11:27:11.326 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.5483999999999999e-05 2024-09-13T11:27:11.326 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:27:11.326 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:27:11.326 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:27:11.326 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.327 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.327 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:177913 3.27 3:e5268e7a:::benchmark_data_smithi177_31116_object37610:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e168)", 2024-09-13T11:27:11.327 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:25:27.607015+0000", 2024-09-13T11:27:11.327 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 103.635492524, 2024-09-13T11:27:11.327 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.301660555, 2024-09-13T11:27:11.327 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:27:11.327 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:27:11.327 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:27:11.327 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:27:11.327 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:27:11.327 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:27:11.327 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 177913 2024-09-13T11:27:11.327 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.327 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:27:11.328 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.328 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:27:11.328 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:27.607015+0000", 2024-09-13T11:27:11.328 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:27:11.328 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.328 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.328 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:27:11.328 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:27.607015+0000", 2024-09-13T11:27:11.328 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:27:11.328 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.328 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.328 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:27:11.328 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:27.607018+0000", 2024-09-13T11:27:11.329 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.7000000000000002e-06 2024-09-13T11:27:11.329 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.329 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.329 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:27:11.329 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:27.607029+0000", 2024-09-13T11:27:11.329 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0933e-05 2024-09-13T11:27:11.329 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.329 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.329 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:27:11.329 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:27.607062+0000", 2024-09-13T11:27:11.329 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.2786000000000001e-05 2024-09-13T11:27:11.329 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.329 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.329 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:27:11.330 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:27.607078+0000", 2024-09-13T11:27:11.330 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.6232999999999999e-05 2024-09-13T11:27:11.330 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.330 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.330 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:27:11.330 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:27.607204+0000", 2024-09-13T11:27:11.330 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000125458 2024-09-13T11:27:11.330 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.330 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.330 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:27:11.330 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:27.607544+0000", 2024-09-13T11:27:11.330 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00033990600000000001 2024-09-13T11:27:11.330 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.330 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.331 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 5,11", 2024-09-13T11:27:11.331 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:27.607882+0000", 2024-09-13T11:27:11.331 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00033788399999999999 2024-09-13T11:27:11.331 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.331 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.331 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:27:11.331 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:27.610120+0000", 2024-09-13T11:27:11.331 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0022379930000000002 2024-09-13T11:27:11.331 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.331 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.331 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:27:11.331 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:27.613170+0000", 2024-09-13T11:27:11.331 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.003050545 2024-09-13T11:27:11.331 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.332 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.332 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:27:11.332 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:27.908398+0000", 2024-09-13T11:27:11.332 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.29522760599999998 2024-09-13T11:27:11.332 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.332 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.332 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:27:11.332 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:27.908458+0000", 2024-09-13T11:27:11.332 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.9718e-05 2024-09-13T11:27:11.332 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.332 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.332 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:27:11.332 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:27.908675+0000", 2024-09-13T11:27:11.332 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00021779300000000001 2024-09-13T11:27:11.333 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:27:11.333 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:27:11.333 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:27:11.333 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.333 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.333 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop_reply(client.4665.0:177913 3.27 e168/120 ondisk, result = 0)", 2024-09-13T11:27:11.333 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:25:27.610053+0000", 2024-09-13T11:27:11.333 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 103.632454365, 2024-09-13T11:27:11.333 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.29862817200000003, 2024-09-13T11:27:11.333 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:27:11.333 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:27:11.333 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:27:11.333 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:27:11.333 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.334 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:27:11.334 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:27.610053+0000", 2024-09-13T11:27:11.334 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:27:11.334 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.334 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.334 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:27:11.334 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:27.610053+0000", 2024-09-13T11:27:11.334 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:27:11.334 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.334 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.334 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:27:11.334 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:27.610056+0000", 2024-09-13T11:27:11.334 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.49e-06 2024-09-13T11:27:11.334 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.335 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.335 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:27:11.335 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:27.610067+0000", 2024-09-13T11:27:11.335 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0599e-05 2024-09-13T11:27:11.335 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.335 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.335 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:27:11.335 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:27.908314+0000", 2024-09-13T11:27:11.335 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.29824728700000003 2024-09-13T11:27:11.335 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.335 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.335 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:27:11.335 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:27.908320+0000", 2024-09-13T11:27:11.336 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.9100000000000002e-06 2024-09-13T11:27:11.336 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.336 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.336 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:27:11.336 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:27.908378+0000", 2024-09-13T11:27:11.336 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.7575999999999999e-05 2024-09-13T11:27:11.336 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.336 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.336 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:27:11.336 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:27.908390+0000", 2024-09-13T11:27:11.336 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.187e-05 2024-09-13T11:27:11.336 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.336 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.336 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:27:11.337 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:27.908681+0000", 2024-09-13T11:27:11.337 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00029144000000000002 2024-09-13T11:27:11.337 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:27:11.337 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:27:11.337 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:27:11.337 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.337 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.337 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDPGPushReply(3.27 168/120 [PushReplyOp(3:e535cc18:::benchmark_data_smithi177_31116_object103295:head)])", 2024-09-13T11:27:11.337 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:25:27.615014+0000", 2024-09-13T11:27:11.337 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 103.627493031, 2024-09-13T11:27:11.337 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.31810488799999997, 2024-09-13T11:27:11.337 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:27:11.337 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:27:11.337 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-13T11:27:11.338 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:27:11.338 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.338 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:27:11.338 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:27.615014+0000", 2024-09-13T11:27:11.338 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:27:11.338 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.338 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.338 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:27:11.338 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:27.615014+0000", 2024-09-13T11:27:11.338 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:27:11.338 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.338 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.338 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:27:11.338 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:27.615017+0000", 2024-09-13T11:27:11.339 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.6199999999999999e-06 2024-09-13T11:27:11.339 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.339 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.339 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:27:11.339 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:27.615030+0000", 2024-09-13T11:27:11.339 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2833000000000001e-05 2024-09-13T11:27:11.339 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.339 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.339 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:27:11.339 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:27.932588+0000", 2024-09-13T11:27:11.339 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.31755857700000001 2024-09-13T11:27:11.339 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.339 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.339 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:27:11.340 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:27.932595+0000", 2024-09-13T11:27:11.340 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.2160000000000003e-06 2024-09-13T11:27:11.340 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.340 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.340 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:27:11.340 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:27.932658+0000", 2024-09-13T11:27:11.340 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.3618e-05 2024-09-13T11:27:11.340 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.340 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.340 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:27:11.340 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:27.933119+0000", 2024-09-13T11:27:11.340 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00046102399999999998 2024-09-13T11:27:11.340 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:27:11.341 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:27:11.341 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:27:11.341 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.341 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.341 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:233663 3.26 3:67af7a71:::benchmark_data_smithi177_31116_object93360:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e200)", 2024-09-13T11:27:11.341 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:26:16.613407+0000", 2024-09-13T11:27:11.341 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 54.629100051000002, 2024-09-13T11:27:11.341 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.32259047600000001, 2024-09-13T11:27:11.341 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:27:11.341 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:27:11.341 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:27:11.341 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:27:11.341 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:27:11.341 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:27:11.342 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 233663 2024-09-13T11:27:11.342 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.342 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:27:11.342 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.342 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:27:11.342 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:16.613407+0000", 2024-09-13T11:27:11.342 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:27:11.342 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.342 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.342 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:27:11.342 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:16.613407+0000", 2024-09-13T11:27:11.342 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:27:11.343 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.343 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.343 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:27:11.343 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:16.613409+0000", 2024-09-13T11:27:11.343 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.8220000000000001e-06 2024-09-13T11:27:11.343 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.343 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.343 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:27:11.343 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:16.613414+0000", 2024-09-13T11:27:11.343 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.8049999999999997e-06 2024-09-13T11:27:11.344 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.344 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.344 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:27:11.344 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:16.613426+0000", 2024-09-13T11:27:11.344 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2319e-05 2024-09-13T11:27:11.344 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.344 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.344 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:27:11.344 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:16.613432+0000", 2024-09-13T11:27:11.344 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.604e-06 2024-09-13T11:27:11.344 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.344 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.344 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:27:11.345 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:16.613490+0000", 2024-09-13T11:27:11.345 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.7742999999999999e-05 2024-09-13T11:27:11.345 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.345 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.345 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:27:11.345 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:16.613614+0000", 2024-09-13T11:27:11.345 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00012410899999999999 2024-09-13T11:27:11.345 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.345 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.345 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 12", 2024-09-13T11:27:11.345 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:16.613672+0000", 2024-09-13T11:27:11.345 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.8053999999999997e-05 2024-09-13T11:27:11.345 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.345 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.346 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:27:11.346 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:16.615571+0000", 2024-09-13T11:27:11.346 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0018996830000000001 2024-09-13T11:27:11.346 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.346 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.346 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:27:11.346 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:16.935742+0000", 2024-09-13T11:27:11.346 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.32017025599999999 2024-09-13T11:27:11.346 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.346 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.346 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:27:11.346 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:16.935888+0000", 2024-09-13T11:27:11.346 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000146716 2024-09-13T11:27:11.346 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.347 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.347 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:27:11.347 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:16.935998+0000", 2024-09-13T11:27:11.347 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00010936499999999999 2024-09-13T11:27:11.347 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:27:11.347 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:27:11.347 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:27:11.347 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:27:11.347 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:27:11.347 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-13T11:27:11.348 INFO:tasks.ceph.osd.3.smithi177.stderr:2024-09-13T11:27:11.345+0000 7f10adba4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26574) UID: 0 2024-09-13T11:27:11.349 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_ops_in_flight 2024-09-13T11:27:11.449 INFO:tasks.ceph.osd.7.smithi177.stderr:2024-09-13T11:27:11.445+0000 7f897b51d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26575) UID: 0 2024-09-13T11:27:11.457 INFO:teuthology.orchestra.run.smithi001.stderr:nodeep-scrub is unset 2024-09-13T11:27:11.490 INFO:tasks.ceph.osd.1.smithi092.stderr:2024-09-13T11:27:11.487+0000 7fe76d3f2640 -1 osd.1 241 handle_osd_map injecting map crc failure 2024-09-13T11:27:11.495 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-13T11:27:11.495 INFO:teuthology.orchestra.run.smithi001.stdout: "ops": [], 2024-09-13T11:27:11.495 INFO:teuthology.orchestra.run.smithi001.stdout: "num_ops": 0 2024-09-13T11:27:11.495 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-13T11:27:11.521 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_blocked_ops 2024-09-13T11:27:11.564 INFO:tasks.ceph.osd.8.smithi001.stderr:2024-09-13T11:27:11.562+0000 7fb9b48ae640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26312) UID: 0 2024-09-13T11:27:11.653 INFO:tasks.ceph.osd.14.smithi162.stderr:2024-09-13T11:27:11.651+0000 7f3df5903640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33154) UID: 0 2024-09-13T11:27:11.719 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-13T11:27:11.719 INFO:teuthology.orchestra.run.smithi001.stdout: "ops": [], 2024-09-13T11:27:11.719 INFO:teuthology.orchestra.run.smithi001.stdout: "complaint_time": 30, 2024-09-13T11:27:11.719 INFO:teuthology.orchestra.run.smithi001.stdout: "num_blocked_ops": 0 2024-09-13T11:27:11.719 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-13T11:27:11.739 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_historic_ops 2024-09-13T11:27:11.753 INFO:tasks.ceph.osd.11.smithi177.stderr:2024-09-13T11:27:11.749+0000 7f24792c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26580) UID: 0 2024-09-13T11:27:11.853 INFO:tasks.ceph.osd.5.smithi092.stderr:2024-09-13T11:27:11.851+0000 7f6f0b705640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26346) UID: 0 2024-09-13T11:27:11.903 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-13T11:27:11.903 INFO:teuthology.orchestra.run.smithi001.stdout: "size": 20, 2024-09-13T11:27:11.903 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 600, 2024-09-13T11:27:11.903 INFO:teuthology.orchestra.run.smithi001.stdout: "ops": [ 2024-09-13T11:27:11.903 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.903 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop_reply(client.4665.0:167 3.9 e41/30 ondisk, result = 0)", 2024-09-13T11:27:11.903 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:22:47.419696+0000", 2024-09-13T11:27:11.903 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 264.48157551700001, 2024-09-13T11:27:11.903 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.64615778199999996, 2024-09-13T11:27:11.903 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:27:11.903 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:27:11.903 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:27:11.904 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:27:11.904 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.904 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:27:11.904 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:47.419696+0000", 2024-09-13T11:27:11.904 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:27:11.904 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.904 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.904 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:27:11.904 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:47.419696+0000", 2024-09-13T11:27:11.904 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:27:11.904 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.904 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.904 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:27:11.905 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:47.419697+0000", 2024-09-13T11:27:11.905 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.15e-06 2024-09-13T11:27:11.905 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.905 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.905 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:27:11.905 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:47.419700+0000", 2024-09-13T11:27:11.905 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.2449999999999998e-06 2024-09-13T11:27:11.905 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.905 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.905 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:27:11.905 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:48.065637+0000", 2024-09-13T11:27:11.905 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.64593674300000004 2024-09-13T11:27:11.905 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.906 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.906 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:27:11.906 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:48.065648+0000", 2024-09-13T11:27:11.906 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0865e-05 2024-09-13T11:27:11.906 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.906 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.906 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:27:11.906 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:48.065720+0000", 2024-09-13T11:27:11.906 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.2020999999999994e-05 2024-09-13T11:27:11.906 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.906 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.906 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:27:11.906 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:48.065728+0000", 2024-09-13T11:27:11.906 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.7660000000000004e-06 2024-09-13T11:27:11.907 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.907 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.907 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:27:11.907 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:22:48.065853+0000", 2024-09-13T11:27:11.907 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00012499200000000001 2024-09-13T11:27:11.907 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:27:11.907 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:27:11.907 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:27:11.907 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.907 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.907 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4665.0:25408 3.4 e56/40 3:294e23bd:::benchmark_data_smithi177_31116_object25407:head v 56'1645, mlcod=55'1642)", 2024-09-13T11:27:11.907 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:04.494686+0000", 2024-09-13T11:27:11.907 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 247.406585189, 2024-09-13T11:27:11.907 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.29725566399999998, 2024-09-13T11:27:11.908 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:27:11.908 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:27:11.908 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:27:11.908 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:27:11.908 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.908 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:27:11.908 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.494686+0000", 2024-09-13T11:27:11.908 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:27:11.908 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.909 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.909 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:27:11.909 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.494686+0000", 2024-09-13T11:27:11.909 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:27:11.909 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.909 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.909 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:27:11.909 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.494686+0000", 2024-09-13T11:27:11.909 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.2399999999999999e-07 2024-09-13T11:27:11.909 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.909 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.909 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:27:11.909 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.494809+0000", 2024-09-13T11:27:11.909 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00012308600000000001 2024-09-13T11:27:11.910 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.910 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.910 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:27:11.910 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.789410+0000", 2024-09-13T11:27:11.910 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.29460066899999998 2024-09-13T11:27:11.910 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.910 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.910 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:27:11.910 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.789425+0000", 2024-09-13T11:27:11.910 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.4600000000000001e-05 2024-09-13T11:27:11.910 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.910 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.910 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:27:11.910 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.789507+0000", 2024-09-13T11:27:11.911 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.2785000000000001e-05 2024-09-13T11:27:11.911 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.911 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.911 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:27:11.911 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.789525+0000", 2024-09-13T11:27:11.911 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.8117999999999998e-05 2024-09-13T11:27:11.911 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.911 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.911 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:27:11.911 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.791852+0000", 2024-09-13T11:27:11.911 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0023264459999999998 2024-09-13T11:27:11.911 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.911 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.911 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:27:11.912 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:04.791942+0000", 2024-09-13T11:27:11.912 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.9635999999999994e-05 2024-09-13T11:27:11.912 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:27:11.912 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:27:11.912 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:27:11.912 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.912 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.912 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4665.0:30828 3.f e59/30 3:f3a9c6a9:::benchmark_data_smithi177_31116_object30827:head v 59'1920, mlcod=59'1916)", 2024-09-13T11:27:11.912 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:09.502607+0000", 2024-09-13T11:27:11.912 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 242.39866414599999, 2024-09-13T11:27:11.912 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.26374408300000002, 2024-09-13T11:27:11.912 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:27:11.912 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:27:11.912 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:27:11.913 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:27:11.913 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.913 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:27:11.913 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.502607+0000", 2024-09-13T11:27:11.913 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:27:11.913 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.913 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.913 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:27:11.913 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.502607+0000", 2024-09-13T11:27:11.913 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:27:11.913 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.913 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.913 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:27:11.913 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.502609+0000", 2024-09-13T11:27:11.914 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.84e-06 2024-09-13T11:27:11.914 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.914 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.914 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:27:11.914 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.502663+0000", 2024-09-13T11:27:11.914 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.3971000000000002e-05 2024-09-13T11:27:11.914 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.914 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.914 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:27:11.914 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.763912+0000", 2024-09-13T11:27:11.914 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.26124904300000001 2024-09-13T11:27:11.914 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.914 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.914 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:27:11.915 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.763920+0000", 2024-09-13T11:27:11.915 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.1149999999999994e-06 2024-09-13T11:27:11.915 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.915 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.915 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:27:11.915 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.764024+0000", 2024-09-13T11:27:11.915 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00010380899999999999 2024-09-13T11:27:11.915 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.915 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.915 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:27:11.915 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.764068+0000", 2024-09-13T11:27:11.915 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.4638999999999997e-05 2024-09-13T11:27:11.915 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.915 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.915 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:27:11.916 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.766308+0000", 2024-09-13T11:27:11.916 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0022392039999999998 2024-09-13T11:27:11.916 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.916 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.916 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:27:11.916 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:09.766351+0000", 2024-09-13T11:27:11.916 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.3461999999999998e-05 2024-09-13T11:27:11.916 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:27:11.916 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:27:11.916 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:27:11.916 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.916 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.916 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDPGPush(3.1b 84/78 [PushOp(3:dcddd139:::benchmark_data_smithi177_31116_object10384:head, version: 48'673, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:dcddd139:::benchmark_data_smithi177_31116_object10384:head@48'673, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:27:11.917 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:41.400602+0000", 2024-09-13T11:27:11.917 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 210.50066866099999, 2024-09-13T11:27:11.917 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.43365821999999998, 2024-09-13T11:27:11.917 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:27:11.917 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:27:11.917 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:27:11.917 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:27:11.917 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.917 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:27:11.917 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:41.400602+0000", 2024-09-13T11:27:11.917 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:27:11.917 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.917 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.917 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:27:11.918 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:41.400602+0000", 2024-09-13T11:27:11.918 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:27:11.918 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.918 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.918 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:27:11.918 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:41.400604+0000", 2024-09-13T11:27:11.918 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.99e-06 2024-09-13T11:27:11.918 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.918 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.918 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:27:11.918 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:41.400653+0000", 2024-09-13T11:27:11.918 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.8195000000000002e-05 2024-09-13T11:27:11.918 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.918 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.919 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:27:11.919 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:41.833180+0000", 2024-09-13T11:27:11.919 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.43252769200000002 2024-09-13T11:27:11.919 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.919 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.919 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:27:11.919 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:41.833213+0000", 2024-09-13T11:27:11.919 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.2737999999999997e-05 2024-09-13T11:27:11.919 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.919 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.919 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:27:11.919 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:41.833408+0000", 2024-09-13T11:27:11.919 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00019473799999999999 2024-09-13T11:27:11.919 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.920 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.920 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:27:11.920 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:41.833417+0000", 2024-09-13T11:27:11.920 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.5349999999999993e-06 2024-09-13T11:27:11.920 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.920 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.920 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:27:11.920 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:41.834261+0000", 2024-09-13T11:27:11.920 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00084333200000000002 2024-09-13T11:27:11.920 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:27:11.920 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:27:11.920 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:27:11.920 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.921 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.921 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDPGPushReply(3.20 85/77 [PushReplyOp(3:066982a6:::benchmark_data_smithi177_31116_object16400:head)])", 2024-09-13T11:27:11.921 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:42.671551+0000", 2024-09-13T11:27:11.921 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 209.22971983799999, 2024-09-13T11:27:11.921 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.28249085499999999, 2024-09-13T11:27:11.921 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:27:11.921 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:27:11.921 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-13T11:27:11.921 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:27:11.921 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.921 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:27:11.921 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.671551+0000", 2024-09-13T11:27:11.921 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:27:11.921 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.922 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.922 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:27:11.922 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.671551+0000", 2024-09-13T11:27:11.922 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:27:11.922 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.922 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.922 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:27:11.922 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.671552+0000", 2024-09-13T11:27:11.922 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.097e-06 2024-09-13T11:27:11.922 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.922 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.922 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:27:11.922 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.671556+0000", 2024-09-13T11:27:11.923 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.8519999999999997e-06 2024-09-13T11:27:11.923 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.923 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.923 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:27:11.923 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.951769+0000", 2024-09-13T11:27:11.923 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.28021293000000003 2024-09-13T11:27:11.923 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.923 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.923 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:27:11.923 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.951780+0000", 2024-09-13T11:27:11.923 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.1059e-05 2024-09-13T11:27:11.923 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.923 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.923 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:27:11.924 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.951992+0000", 2024-09-13T11:27:11.924 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000211478 2024-09-13T11:27:11.924 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.924 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.924 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:27:11.924 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:42.954042+0000", 2024-09-13T11:27:11.924 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0020504389999999998 2024-09-13T11:27:11.924 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:27:11.924 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:27:11.924 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:27:11.924 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.924 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.924 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDPGPush(3.1b 89/78 [PushOp(3:dc4c79e7:::benchmark_data_smithi177_31116_object19857:head, version: 54'1266, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:dc4c79e7:::benchmark_data_smithi177_31116_object19857:head@54'1266, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:27:11.924 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:46.649128+0000", 2024-09-13T11:27:11.925 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 205.25214273, 2024-09-13T11:27:11.925 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.29963603, 2024-09-13T11:27:11.925 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:27:11.925 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:27:11.925 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:27:11.925 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:27:11.925 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.925 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:27:11.925 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.649128+0000", 2024-09-13T11:27:11.925 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:27:11.925 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.925 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.925 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:27:11.925 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.649128+0000", 2024-09-13T11:27:11.926 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:27:11.926 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.926 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.926 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:27:11.926 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.649130+0000", 2024-09-13T11:27:11.926 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2950000000000001e-06 2024-09-13T11:27:11.926 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.926 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.926 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:27:11.926 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.649197+0000", 2024-09-13T11:27:11.926 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.7033999999999995e-05 2024-09-13T11:27:11.926 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.926 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.926 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:27:11.927 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.947691+0000", 2024-09-13T11:27:11.927 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.29849387300000002 2024-09-13T11:27:11.927 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.927 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.927 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:27:11.927 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.947702+0000", 2024-09-13T11:27:11.927 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.1239e-05 2024-09-13T11:27:11.927 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.927 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.927 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:27:11.927 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.947799+0000", 2024-09-13T11:27:11.927 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.7283999999999999e-05 2024-09-13T11:27:11.927 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.927 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.927 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:27:11.928 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.947809+0000", 2024-09-13T11:27:11.928 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.008e-05 2024-09-13T11:27:11.928 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.928 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.928 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:27:11.928 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:46.948764+0000", 2024-09-13T11:27:11.928 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00095522499999999998 2024-09-13T11:27:11.928 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:27:11.928 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:27:11.928 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:27:11.928 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.928 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.928 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:71927 3.0 3:0197244a:::benchmark_data_smithi177_31116_object71926:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2024-09-13T11:27:11.929 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:23:54.199022+0000", 2024-09-13T11:27:11.929 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 197.70224937399999, 2024-09-13T11:27:11.929 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.26819617800000001, 2024-09-13T11:27:11.929 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:27:11.929 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:27:11.929 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:27:11.929 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:27:11.929 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:27:11.929 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:27:11.929 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 71927 2024-09-13T11:27:11.929 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.930 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:27:11.930 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.930 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:27:11.930 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:54.199022+0000", 2024-09-13T11:27:11.930 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:27:11.930 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.930 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.930 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:27:11.930 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:54.199022+0000", 2024-09-13T11:27:11.930 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:27:11.930 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.930 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.930 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:27:11.931 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:54.199025+0000", 2024-09-13T11:27:11.931 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.2200000000000001e-06 2024-09-13T11:27:11.931 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.931 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.931 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:27:11.931 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:54.199076+0000", 2024-09-13T11:27:11.931 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.1184000000000003e-05 2024-09-13T11:27:11.931 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.931 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.931 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:27:11.931 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:54.199095+0000", 2024-09-13T11:27:11.931 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.9253999999999999e-05 2024-09-13T11:27:11.931 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.932 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.932 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:27:11.932 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:54.199107+0000", 2024-09-13T11:27:11.932 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.1916000000000001e-05 2024-09-13T11:27:11.932 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.932 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.932 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:27:11.932 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:54.199196+0000", 2024-09-13T11:27:11.932 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.8645000000000003e-05 2024-09-13T11:27:11.932 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.932 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.932 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:27:11.932 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:54.199421+0000", 2024-09-13T11:27:11.932 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000224868 2024-09-13T11:27:11.933 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.933 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.933 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 15", 2024-09-13T11:27:11.933 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:54.199647+0000", 2024-09-13T11:27:11.933 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00022666099999999999 2024-09-13T11:27:11.933 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.933 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.933 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:27:11.933 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:54.201333+0000", 2024-09-13T11:27:11.933 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.001685761 2024-09-13T11:27:11.933 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.933 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.933 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:27:11.933 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:54.467097+0000", 2024-09-13T11:27:11.934 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.26576393399999998 2024-09-13T11:27:11.934 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.934 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.934 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:27:11.934 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:54.467163+0000", 2024-09-13T11:27:11.934 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.6088999999999997e-05 2024-09-13T11:27:11.934 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.934 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.934 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:27:11.934 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:23:54.467218+0000", 2024-09-13T11:27:11.935 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.4645999999999998e-05 2024-09-13T11:27:11.935 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:27:11.935 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:27:11.935 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:27:11.935 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.935 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.935 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDPGPush(3.25 121/120 [PushOp(3:a5de2229:::benchmark_data_smithi177_31116_object1005:head, version: 43'62, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:a5de2229:::benchmark_data_smithi177_31116_object1005:head@43'62, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:27:11.935 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:24:27.379880+0000", 2024-09-13T11:27:11.935 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 164.52139153600001, 2024-09-13T11:27:11.935 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.72647719499999996, 2024-09-13T11:27:11.935 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:27:11.935 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:27:11.935 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:27:11.935 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:27:11.936 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.936 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:27:11.936 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:27.379880+0000", 2024-09-13T11:27:11.936 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:27:11.936 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.936 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.936 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:27:11.936 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:27.379880+0000", 2024-09-13T11:27:11.936 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:27:11.936 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.936 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.936 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:27:11.936 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:27.379882+0000", 2024-09-13T11:27:11.936 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.1050000000000002e-06 2024-09-13T11:27:11.937 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.937 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.937 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:27:11.937 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:27.379976+0000", 2024-09-13T11:27:11.937 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.4468000000000004e-05 2024-09-13T11:27:11.937 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.937 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.937 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:27:11.937 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:28.105075+0000", 2024-09-13T11:27:11.937 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.72509911299999996 2024-09-13T11:27:11.937 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.937 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.937 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:27:11.937 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:28.105111+0000", 2024-09-13T11:27:11.938 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.5403e-05 2024-09-13T11:27:11.938 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.938 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.938 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:27:11.938 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:28.105267+0000", 2024-09-13T11:27:11.938 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00015663600000000001 2024-09-13T11:27:11.938 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.938 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.938 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:27:11.938 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:28.105291+0000", 2024-09-13T11:27:11.938 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.4164000000000001e-05 2024-09-13T11:27:11.938 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.938 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.938 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:27:11.939 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:28.106357+0000", 2024-09-13T11:27:11.939 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.001065306 2024-09-13T11:27:11.939 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:27:11.939 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:27:11.939 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:27:11.939 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.939 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.939 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:138294 3.9 3:90012dab:::benchmark_data_smithi177_31116_object138293:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e140)", 2024-09-13T11:27:11.939 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:24:54.250266+0000", 2024-09-13T11:27:11.939 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 137.651004863, 2024-09-13T11:27:11.939 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.33165556200000001, 2024-09-13T11:27:11.939 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:27:11.939 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:27:11.939 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:27:11.940 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:27:11.940 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:27:11.940 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:27:11.940 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 138294 2024-09-13T11:27:11.940 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.940 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:27:11.940 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.940 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:27:11.940 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:54.250266+0000", 2024-09-13T11:27:11.940 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:27:11.940 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.940 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.940 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:27:11.940 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:54.250266+0000", 2024-09-13T11:27:11.941 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:27:11.941 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.941 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.941 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:27:11.941 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:54.250269+0000", 2024-09-13T11:27:11.941 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.3159999999999999e-06 2024-09-13T11:27:11.941 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.941 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.941 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:27:11.941 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:54.250362+0000", 2024-09-13T11:27:11.941 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.3051000000000001e-05 2024-09-13T11:27:11.941 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.941 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.942 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:27:11.942 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:54.250374+0000", 2024-09-13T11:27:11.942 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2327000000000001e-05 2024-09-13T11:27:11.942 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.942 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.942 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:27:11.942 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:54.250382+0000", 2024-09-13T11:27:11.942 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.7249999999999997e-06 2024-09-13T11:27:11.942 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.942 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.942 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:27:11.942 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:54.250459+0000", 2024-09-13T11:27:11.942 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.7218999999999994e-05 2024-09-13T11:27:11.942 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.942 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.943 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:27:11.943 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:54.250611+0000", 2024-09-13T11:27:11.943 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00015241000000000001 2024-09-13T11:27:11.943 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.943 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.943 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 2", 2024-09-13T11:27:11.943 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:54.250777+0000", 2024-09-13T11:27:11.943 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00016575199999999999 2024-09-13T11:27:11.943 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.943 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.944 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:27:11.944 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:54.252776+0000", 2024-09-13T11:27:11.944 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0019987260000000001 2024-09-13T11:27:11.944 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.944 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.944 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:27:11.944 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:54.581758+0000", 2024-09-13T11:27:11.944 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.32898232799999999 2024-09-13T11:27:11.944 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.944 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.944 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:27:11.944 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:54.581854+0000", 2024-09-13T11:27:11.944 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.5468000000000001e-05 2024-09-13T11:27:11.945 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.945 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.945 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:27:11.945 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:54.581922+0000", 2024-09-13T11:27:11.945 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.8239999999999997e-05 2024-09-13T11:27:11.945 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:27:11.945 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:27:11.945 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:27:11.945 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.945 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.945 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop_reply(client.4665.0:140790 3.20 e144/77 ondisk, result = 0)", 2024-09-13T11:27:11.946 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:24:58.385972+0000", 2024-09-13T11:27:11.946 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 133.51529929500001, 2024-09-13T11:27:11.946 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.51829568500000001, 2024-09-13T11:27:11.946 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:27:11.946 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:27:11.946 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:27:11.946 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:27:11.946 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.946 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:27:11.946 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.385972+0000", 2024-09-13T11:27:11.946 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:27:11.946 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.946 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.946 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:27:11.947 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.385972+0000", 2024-09-13T11:27:11.947 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:27:11.947 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.947 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.947 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:27:11.947 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.385974+0000", 2024-09-13T11:27:11.947 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.7400000000000001e-06 2024-09-13T11:27:11.947 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.947 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.947 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:27:11.947 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.385979+0000", 2024-09-13T11:27:11.947 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.091e-06 2024-09-13T11:27:11.947 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.948 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.948 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:27:11.948 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.904091+0000", 2024-09-13T11:27:11.948 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.51811194900000002 2024-09-13T11:27:11.948 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.948 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.948 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:27:11.948 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.904098+0000", 2024-09-13T11:27:11.948 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.0609999999999998e-06 2024-09-13T11:27:11.949 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.949 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.949 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:27:11.949 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.904156+0000", 2024-09-13T11:27:11.949 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.7902000000000002e-05 2024-09-13T11:27:11.949 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.949 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.949 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:27:11.949 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.904163+0000", 2024-09-13T11:27:11.949 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.1430000000000001e-06 2024-09-13T11:27:11.949 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.949 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.949 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:27:11.949 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:24:58.904267+0000", 2024-09-13T11:27:11.950 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000104799 2024-09-13T11:27:11.950 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:27:11.950 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:27:11.950 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:27:11.950 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.950 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.950 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4665.0:191082 3.26 e175/140 3:67bd986b:::benchmark_data_smithi177_31116_object50779:head v 175'6183, mlcod=175'6180)", 2024-09-13T11:27:11.950 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:25:38.551017+0000", 2024-09-13T11:27:11.950 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 93.350253875000007, 2024-09-13T11:27:11.950 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.29063407899999999, 2024-09-13T11:27:11.950 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:27:11.950 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:27:11.950 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:27:11.951 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:27:11.951 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.951 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:27:11.951 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:38.551017+0000", 2024-09-13T11:27:11.951 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:27:11.951 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.951 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.951 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:27:11.951 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:38.551017+0000", 2024-09-13T11:27:11.951 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:27:11.951 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.951 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.951 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:27:11.951 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:38.551020+0000", 2024-09-13T11:27:11.952 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.9170000000000002e-06 2024-09-13T11:27:11.952 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.952 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.952 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:27:11.952 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:38.551033+0000", 2024-09-13T11:27:11.952 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2843000000000001e-05 2024-09-13T11:27:11.952 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.952 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.952 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:27:11.952 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:38.838514+0000", 2024-09-13T11:27:11.952 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.28748061600000002 2024-09-13T11:27:11.952 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.952 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.952 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:27:11.953 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:38.838532+0000", 2024-09-13T11:27:11.953 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.8012999999999998e-05 2024-09-13T11:27:11.953 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.953 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.953 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:27:11.953 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:38.838646+0000", 2024-09-13T11:27:11.953 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000114596 2024-09-13T11:27:11.953 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.953 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.953 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:27:11.953 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:38.838681+0000", 2024-09-13T11:27:11.953 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.5099999999999999e-05 2024-09-13T11:27:11.953 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.953 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.954 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:27:11.954 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:38.841593+0000", 2024-09-13T11:27:11.954 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0029117050000000001 2024-09-13T11:27:11.954 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.954 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.954 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:27:11.954 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:38.841651+0000", 2024-09-13T11:27:11.954 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.8288999999999997e-05 2024-09-13T11:27:11.954 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:27:11.954 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:27:11.954 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:27:11.954 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.954 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.954 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4665.0:191198 3.26 e175/140 3:64ff4057:::benchmark_data_smithi177_31116_object50895:head v 175'6185, mlcod=175'6182)", 2024-09-13T11:27:11.955 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:25:38.948739+0000", 2024-09-13T11:27:11.955 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 92.952532414000004, 2024-09-13T11:27:11.955 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.25772718100000003, 2024-09-13T11:27:11.955 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:27:11.955 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:27:11.955 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:27:11.955 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:27:11.955 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.955 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:27:11.955 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:38.948739+0000", 2024-09-13T11:27:11.955 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:27:11.955 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.955 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.955 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:27:11.956 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:38.948739+0000", 2024-09-13T11:27:11.956 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:27:11.956 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.956 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.956 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:27:11.956 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:38.948743+0000", 2024-09-13T11:27:11.956 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.0600000000000001e-06 2024-09-13T11:27:11.956 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.956 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.956 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:27:11.956 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:38.948760+0000", 2024-09-13T11:27:11.956 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.7540000000000001e-05 2024-09-13T11:27:11.956 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.956 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.957 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:27:11.957 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:39.204732+0000", 2024-09-13T11:27:11.957 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.25597220199999998 2024-09-13T11:27:11.957 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.957 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.957 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:27:11.957 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:39.204742+0000", 2024-09-13T11:27:11.957 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.6579999999999997e-06 2024-09-13T11:27:11.957 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.957 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.957 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:27:11.957 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:39.204834+0000", 2024-09-13T11:27:11.957 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.1565999999999997e-05 2024-09-13T11:27:11.957 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.957 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.958 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:27:11.958 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:39.204854+0000", 2024-09-13T11:27:11.958 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.0809e-05 2024-09-13T11:27:11.958 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.958 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.958 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:27:11.958 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:39.206423+0000", 2024-09-13T11:27:11.958 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0015685219999999999 2024-09-13T11:27:11.958 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.958 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.958 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:27:11.958 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:39.206466+0000", 2024-09-13T11:27:11.958 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.2824000000000002e-05 2024-09-13T11:27:11.959 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:27:11.959 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:27:11.959 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:27:11.959 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.959 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.959 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:211295 3.2 3:433aa750:::benchmark_data_smithi177_31116_object70992:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e185)", 2024-09-13T11:27:11.959 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:25:55.459639+0000", 2024-09-13T11:27:11.959 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 76.441632064999993, 2024-09-13T11:27:11.959 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.28751246200000002, 2024-09-13T11:27:11.959 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:27:11.959 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:27:11.959 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:27:11.959 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:27:11.959 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:27:11.960 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:27:11.960 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 211295 2024-09-13T11:27:11.960 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.960 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:27:11.960 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.960 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:27:11.960 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:55.459639+0000", 2024-09-13T11:27:11.960 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:27:11.960 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.960 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.960 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:27:11.960 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:55.459639+0000", 2024-09-13T11:27:11.961 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:27:11.961 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.961 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.961 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:27:11.961 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:55.459642+0000", 2024-09-13T11:27:11.961 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.5940000000000001e-06 2024-09-13T11:27:11.961 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.961 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.961 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:27:11.961 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:55.459649+0000", 2024-09-13T11:27:11.961 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.0899999999999999e-06 2024-09-13T11:27:11.961 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.961 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.962 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:27:11.962 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:55.459669+0000", 2024-09-13T11:27:11.962 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.0305999999999998e-05 2024-09-13T11:27:11.962 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.962 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.962 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:27:11.962 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:55.459683+0000", 2024-09-13T11:27:11.962 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.4219999999999999e-05 2024-09-13T11:27:11.962 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.962 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.962 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:27:11.962 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:55.459761+0000", 2024-09-13T11:27:11.962 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.8072999999999996e-05 2024-09-13T11:27:11.962 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.963 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.963 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:27:11.963 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:55.459912+0000", 2024-09-13T11:27:11.963 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000150304 2024-09-13T11:27:11.963 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.963 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.963 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 3", 2024-09-13T11:27:11.963 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:55.460013+0000", 2024-09-13T11:27:11.963 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000101828 2024-09-13T11:27:11.963 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.963 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.963 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:27:11.963 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:55.462204+0000", 2024-09-13T11:27:11.963 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.002191082 2024-09-13T11:27:11.964 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.964 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.964 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:27:11.964 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:55.747086+0000", 2024-09-13T11:27:11.964 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.28488175100000002 2024-09-13T11:27:11.964 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.964 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.964 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:27:11.964 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:55.747120+0000", 2024-09-13T11:27:11.964 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.3448000000000002e-05 2024-09-13T11:27:11.964 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.964 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.964 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:27:11.964 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:25:55.747151+0000", 2024-09-13T11:27:11.965 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.1766e-05 2024-09-13T11:27:11.965 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:27:11.965 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:27:11.965 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:27:11.965 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.965 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.965 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop_reply(client.4665.0:221502 3.20 e190/77 ondisk, result = 0)", 2024-09-13T11:27:11.965 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:26:03.873964+0000", 2024-09-13T11:27:11.965 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 68.02730751, 2024-09-13T11:27:11.965 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.48664245099999998, 2024-09-13T11:27:11.965 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:27:11.965 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:27:11.965 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:27:11.965 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:27:11.966 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.966 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:27:11.966 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:03.873964+0000", 2024-09-13T11:27:11.966 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:27:11.966 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.966 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.966 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:27:11.966 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:03.873964+0000", 2024-09-13T11:27:11.966 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:27:11.966 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.966 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.966 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:27:11.967 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:03.873965+0000", 2024-09-13T11:27:11.967 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.801e-06 2024-09-13T11:27:11.967 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.967 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.967 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:27:11.967 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:03.873972+0000", 2024-09-13T11:27:11.967 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.3029999999999996e-06 2024-09-13T11:27:11.967 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.967 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.968 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:27:11.968 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:04.360178+0000", 2024-09-13T11:27:11.968 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.48620634400000001 2024-09-13T11:27:11.968 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.968 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.968 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:27:11.968 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:04.360183+0000", 2024-09-13T11:27:11.968 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.2669999999999997e-06 2024-09-13T11:27:11.968 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.968 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.968 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:27:11.968 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:04.360368+0000", 2024-09-13T11:27:11.968 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00018470100000000001 2024-09-13T11:27:11.969 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.969 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.969 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:27:11.969 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:04.360387+0000", 2024-09-13T11:27:11.969 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.874e-05 2024-09-13T11:27:11.969 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.969 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.969 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:27:11.969 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:04.360606+0000", 2024-09-13T11:27:11.969 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00021929499999999999 2024-09-13T11:27:11.969 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:27:11.969 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:27:11.969 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:27:11.969 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.970 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.970 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop(client.4665.0:226607 3.4 e196/117 3:218872c8:::benchmark_data_smithi177_31116_object86304:head v 196'6594, mlcod=195'6591)", 2024-09-13T11:27:11.970 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:26:10.278888+0000", 2024-09-13T11:27:11.970 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 61.622382758999997, 2024-09-13T11:27:11.970 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.25848660899999998, 2024-09-13T11:27:11.970 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:27:11.970 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:27:11.970 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:27:11.970 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:27:11.970 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.970 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:27:11.970 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:10.278888+0000", 2024-09-13T11:27:11.970 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:27:11.970 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.971 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.971 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:27:11.971 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:10.278888+0000", 2024-09-13T11:27:11.971 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:27:11.971 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.971 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.971 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:27:11.971 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:10.278890+0000", 2024-09-13T11:27:11.971 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.392e-06 2024-09-13T11:27:11.971 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.971 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.971 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:27:11.971 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:10.278896+0000", 2024-09-13T11:27:11.971 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.562e-06 2024-09-13T11:27:11.972 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.972 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.972 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:27:11.972 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:10.535554+0000", 2024-09-13T11:27:11.972 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.256657723 2024-09-13T11:27:11.972 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.972 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.972 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:27:11.972 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:10.535559+0000", 2024-09-13T11:27:11.972 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.4720000000000003e-06 2024-09-13T11:27:11.972 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.972 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.972 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:27:11.973 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:10.535621+0000", 2024-09-13T11:27:11.973 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.2012000000000002e-05 2024-09-13T11:27:11.973 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.973 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.973 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:27:11.973 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:10.535644+0000", 2024-09-13T11:27:11.973 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.2911000000000001e-05 2024-09-13T11:27:11.973 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.973 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.973 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:27:11.973 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:10.537316+0000", 2024-09-13T11:27:11.973 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.001671692 2024-09-13T11:27:11.973 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.974 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.974 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:27:11.974 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:10.537375+0000", 2024-09-13T11:27:11.974 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.8845000000000003e-05 2024-09-13T11:27:11.974 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:27:11.974 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:27:11.974 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:27:11.974 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.974 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.974 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4665.0:260386 3.9 3:93b945ad:::benchmark_data_smithi177_31116_object120083:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e219)", 2024-09-13T11:27:11.974 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:26:42.629421+0000", 2024-09-13T11:27:11.974 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 29.271849996, 2024-09-13T11:27:11.974 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.27837949899999997, 2024-09-13T11:27:11.974 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:27:11.975 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:27:11.975 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:27:11.975 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-13T11:27:11.975 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4665", 2024-09-13T11:27:11.975 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:27:11.975 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 260386 2024-09-13T11:27:11.975 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.975 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:27:11.975 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.975 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:27:11.975 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:42.629421+0000", 2024-09-13T11:27:11.975 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:27:11.975 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.976 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.976 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:27:11.976 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:42.629421+0000", 2024-09-13T11:27:11.976 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:27:11.976 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.976 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.976 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:27:11.976 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:42.629426+0000", 2024-09-13T11:27:11.976 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.6630000000000001e-06 2024-09-13T11:27:11.976 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.976 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.976 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:27:11.976 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:42.629438+0000", 2024-09-13T11:27:11.976 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2500000000000001e-05 2024-09-13T11:27:11.977 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.977 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.977 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:27:11.977 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:42.629471+0000", 2024-09-13T11:27:11.977 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.3179999999999997e-05 2024-09-13T11:27:11.977 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.977 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.977 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:27:11.977 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:42.629488+0000", 2024-09-13T11:27:11.977 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.6946e-05 2024-09-13T11:27:11.977 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.977 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.977 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:27:11.978 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:42.629628+0000", 2024-09-13T11:27:11.978 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000139369 2024-09-13T11:27:11.978 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.978 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.978 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:27:11.978 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:42.629892+0000", 2024-09-13T11:27:11.978 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00026411499999999999 2024-09-13T11:27:11.978 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.978 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.978 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for subops from 2", 2024-09-13T11:27:11.978 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:42.630094+0000", 2024-09-13T11:27:11.978 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00020248799999999999 2024-09-13T11:27:11.978 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.978 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.979 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op_commit", 2024-09-13T11:27:11.979 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:42.633648+0000", 2024-09-13T11:27:11.979 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.003553956 2024-09-13T11:27:11.979 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.979 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.979 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:27:11.979 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:42.907584+0000", 2024-09-13T11:27:11.979 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.27393531500000001 2024-09-13T11:27:11.979 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.979 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.979 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-13T11:27:11.979 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:42.907708+0000", 2024-09-13T11:27:11.979 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00012468200000000001 2024-09-13T11:27:11.979 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.980 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.980 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:27:11.980 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:42.907801+0000", 2024-09-13T11:27:11.980 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.2285000000000002e-05 2024-09-13T11:27:11.980 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:27:11.980 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:27:11.980 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:27:11.980 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.980 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.980 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_repop_reply(client.4665.0:260386 3.9 e219/117 ondisk, result = 0)", 2024-09-13T11:27:11.980 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:26:42.634218+0000", 2024-09-13T11:27:11.980 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 29.267052777, 2024-09-13T11:27:11.980 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.27359229099999999, 2024-09-13T11:27:11.981 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:27:11.981 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:27:11.981 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:27:11.981 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:27:11.981 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.981 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:27:11.981 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:42.634218+0000", 2024-09-13T11:27:11.981 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:27:11.981 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.981 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.981 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:27:11.981 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:42.634218+0000", 2024-09-13T11:27:11.981 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:27:11.981 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.982 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.982 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:27:11.982 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:42.634221+0000", 2024-09-13T11:27:11.982 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.7800000000000001e-06 2024-09-13T11:27:11.982 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.982 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.982 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:27:11.982 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:42.634230+0000", 2024-09-13T11:27:11.982 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.0699999999999996e-06 2024-09-13T11:27:11.982 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.982 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.982 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:27:11.982 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:42.907405+0000", 2024-09-13T11:27:11.982 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.27317502199999999 2024-09-13T11:27:11.983 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.983 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.983 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:27:11.983 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:42.907423+0000", 2024-09-13T11:27:11.983 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.7932999999999999e-05 2024-09-13T11:27:11.983 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.983 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.983 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:27:11.983 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:42.907554+0000", 2024-09-13T11:27:11.983 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000130708 2024-09-13T11:27:11.983 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.983 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.983 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:27:11.984 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:42.907571+0000", 2024-09-13T11:27:11.984 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.6722999999999999e-05 2024-09-13T11:27:11.984 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.984 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.984 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:27:11.984 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:26:42.907811+0000", 2024-09-13T11:27:11.984 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000240055 2024-09-13T11:27:11.984 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:27:11.984 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:27:11.984 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:27:11.984 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.984 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.984 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDPGRecoveryDelete(3.3c e238,237 3:3c620a77:::benchmark_data_smithi177_31116_object853:head,145'6165)", 2024-09-13T11:27:11.984 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:27:06.527752+0000", 2024-09-13T11:27:11.985 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 5.3735189390000002, 2024-09-13T11:27:11.985 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.46456126599999997, 2024-09-13T11:27:11.985 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:27:11.985 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:27:11.985 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:27:11.985 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:27:11.985 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.985 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:27:11.985 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:27:06.527752+0000", 2024-09-13T11:27:11.985 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:27:11.985 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.985 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.985 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:27:11.985 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:27:06.527752+0000", 2024-09-13T11:27:11.986 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:27:11.986 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.986 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.986 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:27:11.986 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:27:06.527754+0000", 2024-09-13T11:27:11.986 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.6169999999999999e-06 2024-09-13T11:27:11.986 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.986 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.986 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:27:11.986 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:27:06.527760+0000", 2024-09-13T11:27:11.986 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.2099999999999998e-06 2024-09-13T11:27:11.986 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.986 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.986 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:27:11.987 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:27:06.992148+0000", 2024-09-13T11:27:11.987 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.46438807999999998 2024-09-13T11:27:11.987 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.987 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.987 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:27:11.987 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:27:06.992158+0000", 2024-09-13T11:27:11.987 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.859e-06 2024-09-13T11:27:11.987 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.987 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.987 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:27:11.987 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:27:06.992226+0000", 2024-09-13T11:27:11.987 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.8182999999999994e-05 2024-09-13T11:27:11.987 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.988 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.988 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:27:11.988 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:27:06.992239+0000", 2024-09-13T11:27:11.988 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.3006e-05 2024-09-13T11:27:11.988 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.988 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.988 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:27:11.988 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:27:06.992313+0000", 2024-09-13T11:27:11.988 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.4311000000000001e-05 2024-09-13T11:27:11.988 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:27:11.988 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:27:11.988 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:27:11.988 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.988 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.989 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDPGRecoveryDelete(3.3c e241,237 3:3dfb804f:::benchmark_data_smithi177_31116_object26311:head,159'6940)", 2024-09-13T11:27:11.989 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:27:10.866536+0000", 2024-09-13T11:27:11.989 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 1.034735124, 2024-09-13T11:27:11.989 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.51903578699999997, 2024-09-13T11:27:11.989 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:27:11.989 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:27:11.989 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:27:11.989 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:27:11.989 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.989 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:27:11.989 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:27:10.866536+0000", 2024-09-13T11:27:11.989 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:27:11.989 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.990 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.990 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:27:11.990 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:27:10.866536+0000", 2024-09-13T11:27:11.990 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:27:11.990 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.990 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.990 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:27:11.990 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:27:10.866537+0000", 2024-09-13T11:27:11.990 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.305e-06 2024-09-13T11:27:11.990 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.990 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.990 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:27:11.990 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:27:10.866543+0000", 2024-09-13T11:27:11.990 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.4820000000000002e-06 2024-09-13T11:27:11.991 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.991 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.991 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:27:11.991 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:27:11.385445+0000", 2024-09-13T11:27:11.991 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.51890206900000002 2024-09-13T11:27:11.991 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.991 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.991 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:27:11.991 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:27:11.385450+0000", 2024-09-13T11:27:11.991 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.1869999999999996e-06 2024-09-13T11:27:11.991 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.991 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.991 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:27:11.991 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:27:11.385498+0000", 2024-09-13T11:27:11.992 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.7812999999999998e-05 2024-09-13T11:27:11.992 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.992 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.992 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:27:11.992 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:27:11.385502+0000", 2024-09-13T11:27:11.992 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.5689999999999998e-06 2024-09-13T11:27:11.992 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.992 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.992 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:27:11.992 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:27:11.385572+0000", 2024-09-13T11:27:11.992 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.9362000000000004e-05 2024-09-13T11:27:11.992 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:27:11.992 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:27:11.992 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:27:11.993 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.993 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.993 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDPGRecoveryDelete(3.37 e241,227 3:eca5861f:::benchmark_data_smithi177_31116_object52715:head,176'7256)", 2024-09-13T11:27:11.993 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-13T11:27:10.999035+0000", 2024-09-13T11:27:11.993 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 0.902236068, 2024-09-13T11:27:11.993 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.47244847800000001, 2024-09-13T11:27:11.993 INFO:teuthology.orchestra.run.smithi001.stdout: "continuous": false, 2024-09-13T11:27:11.993 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-13T11:27:11.993 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-13T11:27:11.993 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-13T11:27:11.993 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.993 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-13T11:27:11.993 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:27:10.999035+0000", 2024-09-13T11:27:11.994 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:27:11.994 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.994 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.994 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-13T11:27:11.994 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:27:10.999035+0000", 2024-09-13T11:27:11.994 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-13T11:27:11.994 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.996 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.996 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-13T11:27:11.996 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:27:10.999037+0000", 2024-09-13T11:27:11.996 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.624e-06 2024-09-13T11:27:11.996 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.996 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.996 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-13T11:27:11.996 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:27:10.999042+0000", 2024-09-13T11:27:11.996 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.7790000000000001e-06 2024-09-13T11:27:11.996 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.996 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.996 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-13T11:27:11.996 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:27:11.471285+0000", 2024-09-13T11:27:11.997 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.47224272299999998 2024-09-13T11:27:11.997 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.997 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.997 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-13T11:27:11.997 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:27:11.471293+0000", 2024-09-13T11:27:11.997 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.3119999999999994e-06 2024-09-13T11:27:11.997 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.997 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.997 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-13T11:27:11.997 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:27:11.471381+0000", 2024-09-13T11:27:11.997 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.8017999999999997e-05 2024-09-13T11:27:11.997 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.997 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.997 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-13T11:27:11.998 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:27:11.471391+0000", 2024-09-13T11:27:11.998 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.611e-06 2024-09-13T11:27:11.998 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-13T11:27:11.998 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-13T11:27:11.998 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-13T11:27:11.998 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-13T11:27:11.471483+0000", 2024-09-13T11:27:11.998 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.2410999999999993e-05 2024-09-13T11:27:11.998 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:27:11.998 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:27:11.998 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:27:11.998 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-13T11:27:11.998 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-13T11:27:11.998 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-13T11:27:12.000 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-13T11:27:12.001 INFO:tasks.ceph.osd.5.smithi092.stderr:2024-09-13T11:27:11.995+0000 7f6f0b705640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26346) UID: 0 2024-09-13T11:27:12.100 INFO:tasks.ceph.osd.15.smithi177.stderr:2024-09-13T11:27:12.097+0000 7f9d6ce77640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26578) UID: 0 2024-09-13T11:27:12.151 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-09-13T11:27:12.151 INFO:teuthology.orchestra.run.smithi092.stdout: "ops": [], 2024-09-13T11:27:12.152 INFO:teuthology.orchestra.run.smithi092.stdout: "num_ops": 0 2024-09-13T11:27:12.152 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-09-13T11:27:12.170 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-13T11:27:12.201 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-09-13T11:27:12.199+0000 7f97befc9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26315) UID: 0 2024-09-13T11:27:12.302 INFO:tasks.ceph.osd.14.smithi162.stderr:2024-09-13T11:27:12.299+0000 7f3df5903640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33154) UID: 0 2024-09-13T11:27:12.312 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-09-13T11:27:12.312 INFO:teuthology.orchestra.run.smithi092.stdout: "ops": [], 2024-09-13T11:27:12.312 INFO:teuthology.orchestra.run.smithi092.stdout: "complaint_time": 30, 2024-09-13T11:27:12.312 INFO:teuthology.orchestra.run.smithi092.stdout: "num_blocked_ops": 0 2024-09-13T11:27:12.312 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-09-13T11:27:12.328 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-13T11:27:12.404 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-09-13T11:27:12.399+0000 7f97befc9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26315) UID: 0 2024-09-13T11:27:12.465 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-09-13T11:27:12.465 INFO:teuthology.orchestra.run.smithi092.stdout: "size": 20, 2024-09-13T11:27:12.466 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 600, 2024-09-13T11:27:12.466 INFO:teuthology.orchestra.run.smithi092.stdout: "ops": [ 2024-09-13T11:27:12.466 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:12.466 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "MOSDPGPush(3.23 83/78 [PushOp(3:c61f6bab:::benchmark_data_smithi177_31116_object29479:head, version: 59'1860, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:c61f6bab:::benchmark_data_smithi177_31116_object29479:head@59'1860, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:27:12.466 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:40.770864+0000", 2024-09-13T11:27:12.466 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 211.69147144999999, 2024-09-13T11:27:12.466 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.31229662800000002, 2024-09-13T11:27:12.466 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:27:12.466 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:27:12.467 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-09-13T11:27:12.467 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:27:12.467 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:12.467 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:27:12.467 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:40.770864+0000", 2024-09-13T11:27:12.467 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:27:12.467 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:12.467 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:12.467 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:27:12.467 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:40.770864+0000", 2024-09-13T11:27:12.467 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:27:12.468 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:12.468 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:12.468 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:27:12.468 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:40.770865+0000", 2024-09-13T11:27:12.468 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.288e-06 2024-09-13T11:27:12.468 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:12.468 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:12.468 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:27:12.468 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:40.770955+0000", 2024-09-13T11:27:12.468 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.9871999999999996e-05 2024-09-13T11:27:12.469 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:12.469 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:12.469 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:27:12.469 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:41.082105+0000", 2024-09-13T11:27:12.469 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.31114963400000001 2024-09-13T11:27:12.469 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:12.469 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:12.469 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:27:12.469 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:41.082136+0000", 2024-09-13T11:27:12.469 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.0972000000000001e-05 2024-09-13T11:27:12.470 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:12.470 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:12.470 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:27:12.470 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:41.082404+0000", 2024-09-13T11:27:12.470 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00026844800000000002 2024-09-13T11:27:12.470 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:12.470 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:12.470 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:27:12.470 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:41.082429+0000", 2024-09-13T11:27:12.471 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.5021e-05 2024-09-13T11:27:12.471 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:12.471 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:12.471 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:27:12.471 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:41.083161+0000", 2024-09-13T11:27:12.471 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000731393 2024-09-13T11:27:12.471 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:27:12.471 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:27:12.471 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:27:12.471 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:12.471 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:12.472 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:110289 3.29 3.27287969 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e119)", 2024-09-13T11:27:12.472 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:24:25.774634+0000", 2024-09-13T11:27:12.472 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 166.68770137499999, 2024-09-13T11:27:12.472 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.31721633999999999, 2024-09-13T11:27:12.472 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:27:12.472 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:27:12.472 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "reached pg", 2024-09-13T11:27:12.472 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:27:12.472 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:27:12.473 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:27:12.473 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 110289 2024-09-13T11:27:12.473 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:12.473 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:27:12.473 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:12.473 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:27:12.473 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:25.774634+0000", 2024-09-13T11:27:12.473 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:27:12.473 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:12.473 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:12.473 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:27:12.473 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:25.774634+0000", 2024-09-13T11:27:12.474 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:27:12.474 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:12.474 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:12.474 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:27:12.474 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:25.774639+0000", 2024-09-13T11:27:12.474 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.6029999999999998e-06 2024-09-13T11:27:12.474 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:12.474 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:12.474 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:27:12.474 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:25.774695+0000", 2024-09-13T11:27:12.474 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.6694999999999998e-05 2024-09-13T11:27:12.475 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:12.475 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:12.475 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:27:12.475 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:25.774728+0000", 2024-09-13T11:27:12.475 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.2502000000000002e-05 2024-09-13T11:27:12.475 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:12.475 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:12.475 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:27:12.475 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:25.774750+0000", 2024-09-13T11:27:12.476 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.2002999999999999e-05 2024-09-13T11:27:12.476 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:12.476 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:12.476 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:27:12.476 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:25.774942+0000", 2024-09-13T11:27:12.476 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000192265 2024-09-13T11:27:12.476 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:12.476 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:12.476 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-09-13T11:27:12.477 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:25.774960+0000", 2024-09-13T11:27:12.477 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.8275999999999999e-05 2024-09-13T11:27:12.477 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:12.477 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:12.477 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:27:12.477 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:26.091821+0000", 2024-09-13T11:27:12.477 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.31686065000000002 2024-09-13T11:27:12.477 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:12.477 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:12.477 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:27:12.477 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:26.091850+0000", 2024-09-13T11:27:12.478 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.9346e-05 2024-09-13T11:27:12.478 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:27:12.478 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:27:12.478 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:27:12.478 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:12.478 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:12.478 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "MOSDPGPush(3.28 168/120 [PushOp(3:14eb8ed9:::benchmark_data_smithi177_31116_object87538:head, version: 110'4071, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:14eb8ed9:::benchmark_data_smithi177_31116_object87538:head@110'4071, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:27:12.478 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:25:27.274802+0000", 2024-09-13T11:27:12.478 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 105.18753361, 2024-09-13T11:27:12.478 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.369268239, 2024-09-13T11:27:12.479 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:27:12.479 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:27:12.479 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-09-13T11:27:12.479 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:27:12.479 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:12.479 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:27:12.479 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:27.274802+0000", 2024-09-13T11:27:12.479 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:27:12.479 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:12.479 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:12.480 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:27:12.480 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:27.274802+0000", 2024-09-13T11:27:12.480 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:27:12.480 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:12.480 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:12.480 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:27:12.480 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:27.274803+0000", 2024-09-13T11:27:12.480 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.7579999999999999e-06 2024-09-13T11:27:12.480 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:12.480 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:12.480 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:27:12.480 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:27.274831+0000", 2024-09-13T11:27:12.480 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.7464e-05 2024-09-13T11:27:12.481 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:12.481 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:12.481 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:27:12.481 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:27.641355+0000", 2024-09-13T11:27:12.481 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.36652425199999999 2024-09-13T11:27:12.481 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:12.481 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:12.481 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:27:12.481 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:27.641385+0000", 2024-09-13T11:27:12.482 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.9433000000000001e-05 2024-09-13T11:27:12.482 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:12.482 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:12.482 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:27:12.482 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:27.641611+0000", 2024-09-13T11:27:12.482 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00022646299999999999 2024-09-13T11:27:12.482 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:12.482 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:12.482 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:27:12.483 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:27.641638+0000", 2024-09-13T11:27:12.483 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.6482000000000002e-05 2024-09-13T11:27:12.483 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:12.483 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:12.483 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:27:12.483 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:27.644070+0000", 2024-09-13T11:27:12.483 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0024323869999999998 2024-09-13T11:27:12.483 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:27:12.483 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:27:12.483 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:27:12.483 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:12.483 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:12.484 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:191021 3.23 3:c4df18dd:::benchmark_data_smithi177_31116_object50718:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e175)", 2024-09-13T11:27:12.484 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:25:38.460411+0000", 2024-09-13T11:27:12.484 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 94.001924184999993, 2024-09-13T11:27:12.484 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.40138001400000001, 2024-09-13T11:27:12.484 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:27:12.484 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:27:12.484 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:27:12.484 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:27:12.484 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:27:12.484 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:27:12.485 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 191021 2024-09-13T11:27:12.485 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:12.485 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:27:12.485 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:12.485 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:27:12.485 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:38.460411+0000", 2024-09-13T11:27:12.485 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:27:12.485 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:12.485 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:12.485 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:27:12.485 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:38.460411+0000", 2024-09-13T11:27:12.486 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:27:12.486 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:12.486 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:12.486 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:27:12.486 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:38.460413+0000", 2024-09-13T11:27:12.486 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.075e-06 2024-09-13T11:27:12.486 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:12.486 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:12.486 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:27:12.486 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:38.460419+0000", 2024-09-13T11:27:12.486 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.6869999999999999e-06 2024-09-13T11:27:12.487 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:12.487 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:12.487 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:27:12.487 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:38.460431+0000", 2024-09-13T11:27:12.487 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.2451e-05 2024-09-13T11:27:12.487 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:12.487 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:12.487 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:27:12.487 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:38.460440+0000", 2024-09-13T11:27:12.487 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.5140000000000001e-06 2024-09-13T11:27:12.487 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:12.488 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:12.488 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:27:12.488 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:38.460501+0000", 2024-09-13T11:27:12.488 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.0772e-05 2024-09-13T11:27:12.488 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:12.488 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:12.488 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:27:12.488 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:38.460632+0000", 2024-09-13T11:27:12.488 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00013175200000000001 2024-09-13T11:27:12.489 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:12.489 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:12.489 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 9", 2024-09-13T11:27:12.489 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:38.460731+0000", 2024-09-13T11:27:12.489 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.8647000000000006e-05 2024-09-13T11:27:12.489 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:12.489 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:12.489 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:27:12.489 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:38.462322+0000", 2024-09-13T11:27:12.490 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.001590658 2024-09-13T11:27:12.490 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:12.490 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:12.490 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:27:12.490 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:38.861634+0000", 2024-09-13T11:27:12.490 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.39931206499999999 2024-09-13T11:27:12.490 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:12.490 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:12.491 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:27:12.491 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:38.861719+0000", 2024-09-13T11:27:12.491 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.4968000000000004e-05 2024-09-13T11:27:12.491 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:12.491 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:12.491 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:27:12.491 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:38.861791+0000", 2024-09-13T11:27:12.491 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.2424999999999999e-05 2024-09-13T11:27:12.491 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:27:12.491 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:27:12.492 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:27:12.492 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:12.492 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:12.493 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop_reply(client.4665.0:191021 3.23 e175/90 ondisk, result = 0)", 2024-09-13T11:27:12.493 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:25:38.462483+0000", 2024-09-13T11:27:12.493 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 93.999852763000007, 2024-09-13T11:27:12.493 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.39932300900000001, 2024-09-13T11:27:12.493 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:27:12.493 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:27:12.493 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-09-13T11:27:12.493 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:27:12.493 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:12.493 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:27:12.493 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:38.462483+0000", 2024-09-13T11:27:12.493 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:27:12.494 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:12.494 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:12.494 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:27:12.494 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:38.462483+0000", 2024-09-13T11:27:12.494 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:27:12.494 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:12.494 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:12.494 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:27:12.494 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:38.462484+0000", 2024-09-13T11:27:12.494 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.1990000000000001e-06 2024-09-13T11:27:12.494 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:12.494 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:12.494 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:27:12.494 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:38.462488+0000", 2024-09-13T11:27:12.494 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.985e-06 2024-09-13T11:27:12.495 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:12.495 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:12.495 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:27:12.495 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:38.861475+0000", 2024-09-13T11:27:12.495 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.39898754800000003 2024-09-13T11:27:12.495 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:12.495 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:12.495 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:27:12.495 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:38.861488+0000", 2024-09-13T11:27:12.495 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.3037000000000001e-05 2024-09-13T11:27:12.495 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:12.495 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:12.495 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:27:12.495 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:38.861607+0000", 2024-09-13T11:27:12.496 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00011835 2024-09-13T11:27:12.496 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:12.496 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:12.496 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:27:12.496 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:38.861624+0000", 2024-09-13T11:27:12.496 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.7730000000000001e-05 2024-09-13T11:27:12.496 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:12.496 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:12.496 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:27:12.496 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:38.861806+0000", 2024-09-13T11:27:12.496 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00018116000000000001 2024-09-13T11:27:12.496 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:27:12.496 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:27:12.496 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:27:12.497 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:12.497 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:12.497 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:209869 3.27 3:e6b1fcc5:::benchmark_data_smithi177_31116_object69566:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e185)", 2024-09-13T11:27:12.497 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:25:54.300263+0000", 2024-09-13T11:27:12.497 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 78.162071866000005, 2024-09-13T11:27:12.497 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.30266981599999998, 2024-09-13T11:27:12.497 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:27:12.497 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:27:12.497 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:27:12.497 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:27:12.497 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:27:12.497 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:27:12.497 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 209869 2024-09-13T11:27:12.498 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:12.499 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:27:12.499 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:12.499 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:27:12.499 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:54.300263+0000", 2024-09-13T11:27:12.499 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:27:12.499 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:12.499 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:12.499 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:27:12.499 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:54.300263+0000", 2024-09-13T11:27:12.499 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:27:12.499 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:12.500 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:12.500 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:27:12.500 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:54.300267+0000", 2024-09-13T11:27:12.500 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.2200000000000001e-06 2024-09-13T11:27:12.500 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:12.500 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:12.500 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:27:12.500 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:54.300276+0000", 2024-09-13T11:27:12.500 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.3570000000000004e-06 2024-09-13T11:27:12.500 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:12.500 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:12.500 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:27:12.501 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:54.300301+0000", 2024-09-13T11:27:12.501 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.5191e-05 2024-09-13T11:27:12.501 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:12.501 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:12.501 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:27:12.501 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:54.300321+0000", 2024-09-13T11:27:12.501 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.9967999999999999e-05 2024-09-13T11:27:12.501 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:12.501 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:12.501 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:27:12.501 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:54.300427+0000", 2024-09-13T11:27:12.501 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000105781 2024-09-13T11:27:12.501 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:12.502 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:12.502 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:27:12.502 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:54.300590+0000", 2024-09-13T11:27:12.502 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00016341000000000001 2024-09-13T11:27:12.502 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:12.502 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:12.502 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 11", 2024-09-13T11:27:12.502 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:54.300724+0000", 2024-09-13T11:27:12.502 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000133323 2024-09-13T11:27:12.502 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:12.502 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:12.502 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:27:12.502 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:54.303373+0000", 2024-09-13T11:27:12.503 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0026488050000000002 2024-09-13T11:27:12.503 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:12.503 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:12.503 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:27:12.503 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:54.602650+0000", 2024-09-13T11:27:12.503 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.299277976 2024-09-13T11:27:12.503 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:12.503 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:12.503 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:27:12.503 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:54.602795+0000", 2024-09-13T11:27:12.503 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000144127 2024-09-13T11:27:12.503 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:12.504 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:12.504 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:27:12.504 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:54.602933+0000", 2024-09-13T11:27:12.504 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000138658 2024-09-13T11:27:12.504 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:27:12.504 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:27:12.504 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:27:12.504 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:12.504 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:12.504 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:221567 3.12 3:4823d14b:::benchmark_data_smithi177_31116_object81264:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e193)", 2024-09-13T11:27:12.504 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:26:06.064686+0000", 2024-09-13T11:27:12.504 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 66.397649627999996, 2024-09-13T11:27:12.504 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.28539763299999998, 2024-09-13T11:27:12.505 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:27:12.505 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:27:12.505 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:27:12.505 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:27:12.505 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:27:12.505 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:27:12.505 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 221567 2024-09-13T11:27:12.505 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:12.505 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:27:12.505 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:12.505 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:27:12.505 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:06.064686+0000", 2024-09-13T11:27:12.506 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:27:12.506 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:12.506 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:12.506 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:27:12.506 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:06.064686+0000", 2024-09-13T11:27:12.506 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:27:12.506 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:12.506 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:12.506 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:27:12.506 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:06.064687+0000", 2024-09-13T11:27:12.506 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.6300000000000001e-06 2024-09-13T11:27:12.506 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:12.506 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:12.507 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:27:12.507 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:06.064693+0000", 2024-09-13T11:27:12.507 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.7339999999999996e-06 2024-09-13T11:27:12.507 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:12.507 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:12.507 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:27:12.507 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:06.064704+0000", 2024-09-13T11:27:12.507 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.1253000000000001e-05 2024-09-13T11:27:12.507 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:12.507 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:12.507 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:27:12.507 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:06.064717+0000", 2024-09-13T11:27:12.508 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.2293000000000001e-05 2024-09-13T11:27:12.508 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:12.508 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:12.508 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:27:12.508 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:06.064866+0000", 2024-09-13T11:27:12.508 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00014957499999999999 2024-09-13T11:27:12.508 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:12.508 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:12.508 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-09-13T11:27:12.508 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:06.064893+0000", 2024-09-13T11:27:12.508 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.6614e-05 2024-09-13T11:27:12.508 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:12.509 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:12.509 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:27:12.509 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:06.282646+0000", 2024-09-13T11:27:12.509 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.21775271800000001 2024-09-13T11:27:12.509 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:12.509 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:12.509 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:27:12.509 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:06.283003+0000", 2024-09-13T11:27:12.509 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00035780799999999999 2024-09-13T11:27:12.509 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:12.509 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:12.510 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 7,14", 2024-09-13T11:27:12.510 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:06.283345+0000", 2024-09-13T11:27:12.510 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00034164200000000002 2024-09-13T11:27:12.510 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:12.510 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:12.510 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:27:12.510 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:06.288935+0000", 2024-09-13T11:27:12.510 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0055902429999999999 2024-09-13T11:27:12.510 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:12.510 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:12.510 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:27:12.511 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:06.289795+0000", 2024-09-13T11:27:12.511 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00085952200000000004 2024-09-13T11:27:12.511 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:12.511 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:12.511 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:27:12.511 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:06.349852+0000", 2024-09-13T11:27:12.511 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.060057352000000001 2024-09-13T11:27:12.511 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:12.511 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:12.512 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:27:12.512 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:06.349922+0000", 2024-09-13T11:27:12.512 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.0272999999999996e-05 2024-09-13T11:27:12.512 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:12.512 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:12.512 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:27:12.512 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:06.350083+0000", 2024-09-13T11:27:12.512 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00016097600000000001 2024-09-13T11:27:12.512 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:27:12.512 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:27:12.512 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:27:12.512 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:12.513 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:12.513 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:227066 3.23 3:c67f2f28:::benchmark_data_smithi177_31116_object86763:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e196)", 2024-09-13T11:27:12.513 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:26:10.753624+0000", 2024-09-13T11:27:12.513 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 61.708711555000001, 2024-09-13T11:27:12.513 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.28670347299999999, 2024-09-13T11:27:12.513 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:27:12.513 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:27:12.513 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:27:12.513 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:27:12.513 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:27:12.513 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:27:12.514 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 227066 2024-09-13T11:27:12.514 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:12.514 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:27:12.514 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:12.514 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:27:12.514 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:10.753624+0000", 2024-09-13T11:27:12.514 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:27:12.514 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:12.514 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:12.514 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:27:12.514 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:10.753624+0000", 2024-09-13T11:27:12.514 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:27:12.514 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:12.515 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:12.515 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:27:12.515 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:10.753627+0000", 2024-09-13T11:27:12.515 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.8030000000000001e-06 2024-09-13T11:27:12.515 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:12.515 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:12.515 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:27:12.515 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:10.753634+0000", 2024-09-13T11:27:12.515 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.9659999999999999e-06 2024-09-13T11:27:12.515 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:12.515 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:12.515 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:27:12.516 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:10.753650+0000", 2024-09-13T11:27:12.516 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.6167999999999998e-05 2024-09-13T11:27:12.516 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:12.516 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:12.516 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:27:12.516 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:10.753657+0000", 2024-09-13T11:27:12.516 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.7600000000000002e-06 2024-09-13T11:27:12.516 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:12.516 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:12.516 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:27:12.516 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:10.753741+0000", 2024-09-13T11:27:12.516 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.3523999999999995e-05 2024-09-13T11:27:12.517 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:12.517 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:12.517 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:27:12.517 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:10.753878+0000", 2024-09-13T11:27:12.517 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000137207 2024-09-13T11:27:12.517 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:12.517 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:12.517 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 9", 2024-09-13T11:27:12.517 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:10.754027+0000", 2024-09-13T11:27:12.517 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00014856599999999999 2024-09-13T11:27:12.517 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:12.517 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:12.518 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:27:12.518 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:10.756313+0000", 2024-09-13T11:27:12.518 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.002286045 2024-09-13T11:27:12.518 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:12.518 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:12.518 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:27:12.518 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:11.040266+0000", 2024-09-13T11:27:12.518 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.283953434 2024-09-13T11:27:12.518 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:12.518 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:12.518 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:27:12.518 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:11.040300+0000", 2024-09-13T11:27:12.518 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.4227000000000003e-05 2024-09-13T11:27:12.519 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:12.519 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:12.519 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:27:12.519 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:11.040327+0000", 2024-09-13T11:27:12.519 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.6772999999999999e-05 2024-09-13T11:27:12.519 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:27:12.519 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:27:12.519 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:27:12.519 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:12.519 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:12.519 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:227420 3.23 3:c4add80a:::benchmark_data_smithi177_31116_object87117:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e196)", 2024-09-13T11:27:12.519 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:26:11.037912+0000", 2024-09-13T11:27:12.520 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 61.424423535000003, 2024-09-13T11:27:12.520 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.33121650499999999, 2024-09-13T11:27:12.520 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:27:12.520 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:27:12.520 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:27:12.520 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:27:12.520 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:27:12.520 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:27:12.520 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 227420 2024-09-13T11:27:12.520 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:12.520 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:27:12.520 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:12.521 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:27:12.521 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:11.037912+0000", 2024-09-13T11:27:12.521 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:27:12.521 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:12.521 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:12.521 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:27:12.521 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:11.037912+0000", 2024-09-13T11:27:12.521 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:27:12.521 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:12.521 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:12.521 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:27:12.521 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:11.037913+0000", 2024-09-13T11:27:12.522 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.606e-06 2024-09-13T11:27:12.522 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:12.522 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:12.522 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:27:12.522 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:11.037918+0000", 2024-09-13T11:27:12.522 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.0509999999999996e-06 2024-09-13T11:27:12.522 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:12.522 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:12.522 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:27:12.522 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:11.037930+0000", 2024-09-13T11:27:12.522 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.1466e-05 2024-09-13T11:27:12.522 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:12.522 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:12.523 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:27:12.523 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:11.037936+0000", 2024-09-13T11:27:12.523 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.5259999999999999e-06 2024-09-13T11:27:12.523 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:12.523 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:12.523 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:27:12.523 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:11.037994+0000", 2024-09-13T11:27:12.523 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.7776999999999999e-05 2024-09-13T11:27:12.523 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:12.523 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:12.523 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:27:12.523 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:11.038129+0000", 2024-09-13T11:27:12.524 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00013500299999999999 2024-09-13T11:27:12.524 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:12.524 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:12.524 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 9", 2024-09-13T11:27:12.524 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:11.038208+0000", 2024-09-13T11:27:12.524 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.9016000000000004e-05 2024-09-13T11:27:12.524 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:12.524 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:12.524 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:27:12.524 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:11.041461+0000", 2024-09-13T11:27:12.524 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00325246 2024-09-13T11:27:12.525 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:12.525 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:12.525 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:27:12.525 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:11.369025+0000", 2024-09-13T11:27:12.525 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.32756474800000002 2024-09-13T11:27:12.525 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:12.525 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:12.525 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:27:12.525 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:11.369086+0000", 2024-09-13T11:27:12.525 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.0140999999999999e-05 2024-09-13T11:27:12.525 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:12.526 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:12.526 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:27:12.526 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:11.369128+0000", 2024-09-13T11:27:12.526 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.2710999999999998e-05 2024-09-13T11:27:12.526 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:27:12.526 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:27:12.526 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:27:12.526 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:12.526 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:12.526 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop_reply(client.4665.0:227420 3.23 e196/90 ondisk, result = 0)", 2024-09-13T11:27:12.526 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:26:11.040152+0000", 2024-09-13T11:27:12.527 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 61.422183697000001, 2024-09-13T11:27:12.527 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.328983201, 2024-09-13T11:27:12.527 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:27:12.527 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:27:12.527 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-09-13T11:27:12.527 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:27:12.527 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:12.527 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:27:12.527 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:11.040152+0000", 2024-09-13T11:27:12.527 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:27:12.527 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:12.527 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:12.527 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:27:12.528 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:11.040152+0000", 2024-09-13T11:27:12.528 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:27:12.528 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:12.528 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:12.528 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:27:12.528 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:11.040157+0000", 2024-09-13T11:27:12.528 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.1309999999999997e-06 2024-09-13T11:27:12.528 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:12.528 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:12.528 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:27:12.529 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:11.040162+0000", 2024-09-13T11:27:12.529 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.8720000000000001e-06 2024-09-13T11:27:12.529 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:12.529 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:12.529 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:27:12.529 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:11.368952+0000", 2024-09-13T11:27:12.529 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.32879022400000002 2024-09-13T11:27:12.529 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:12.529 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:12.529 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:27:12.529 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:11.368960+0000", 2024-09-13T11:27:12.529 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.8460000000000008e-06 2024-09-13T11:27:12.529 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:12.529 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:12.530 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:27:12.530 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:11.369013+0000", 2024-09-13T11:27:12.530 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.3201000000000001e-05 2024-09-13T11:27:12.530 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:12.530 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:12.530 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:27:12.530 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:11.369020+0000", 2024-09-13T11:27:12.530 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.3050000000000004e-06 2024-09-13T11:27:12.530 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:12.530 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:12.530 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:27:12.530 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:11.369135+0000", 2024-09-13T11:27:12.530 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000114622 2024-09-13T11:27:12.530 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:27:12.530 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:27:12.531 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:27:12.531 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:12.531 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:12.531 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:244694 3.27 3:e45f60ea:::benchmark_data_smithi177_31116_object104391:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e208)", 2024-09-13T11:27:12.531 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:26:27.471915+0000", 2024-09-13T11:27:12.531 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 44.990420043999997, 2024-09-13T11:27:12.531 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.28616412299999999, 2024-09-13T11:27:12.531 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:27:12.531 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:27:12.531 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:27:12.531 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:27:12.531 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:27:12.531 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:27:12.531 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 244694 2024-09-13T11:27:12.532 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:12.532 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:27:12.532 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:12.532 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:27:12.532 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:27.471915+0000", 2024-09-13T11:27:12.532 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:27:12.532 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:12.532 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:12.532 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:27:12.532 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:27.471915+0000", 2024-09-13T11:27:12.532 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:27:12.532 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:12.532 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:12.533 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:27:12.533 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:27.471916+0000", 2024-09-13T11:27:12.533 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.8100000000000001e-07 2024-09-13T11:27:12.533 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:12.533 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:12.533 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:27:12.533 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:27.471919+0000", 2024-09-13T11:27:12.533 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.2179999999999999e-06 2024-09-13T11:27:12.533 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:12.533 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:12.533 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:27:12.533 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:27.471928+0000", 2024-09-13T11:27:12.533 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.4279999999999996e-06 2024-09-13T11:27:12.533 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:12.534 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:12.534 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:27:12.534 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:27.471934+0000", 2024-09-13T11:27:12.534 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.3180000000000003e-06 2024-09-13T11:27:12.534 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:12.534 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:12.534 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:27:12.534 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:27.471983+0000", 2024-09-13T11:27:12.534 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.8393000000000002e-05 2024-09-13T11:27:12.534 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:12.534 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:12.534 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:27:12.534 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:27.472085+0000", 2024-09-13T11:27:12.534 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000102723 2024-09-13T11:27:12.535 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:12.535 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:12.535 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 11", 2024-09-13T11:27:12.535 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:27.472218+0000", 2024-09-13T11:27:12.535 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00013276600000000001 2024-09-13T11:27:12.535 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:12.535 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:12.535 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:27:12.535 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:27.474061+0000", 2024-09-13T11:27:12.535 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.001842797 2024-09-13T11:27:12.535 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:12.535 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:12.535 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:27:12.535 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:27.757978+0000", 2024-09-13T11:27:12.535 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.28391740999999998 2024-09-13T11:27:12.536 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:12.536 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:12.536 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:27:12.536 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:27.758033+0000", 2024-09-13T11:27:12.536 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.4997000000000002e-05 2024-09-13T11:27:12.536 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:12.536 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:12.536 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:27:12.536 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:27.758079+0000", 2024-09-13T11:27:12.536 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.6091999999999998e-05 2024-09-13T11:27:12.536 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:27:12.536 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:27:12.536 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:27:12.537 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:12.537 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:12.537 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:266944 3.32 3:4c199f39:::benchmark_data_smithi177_31116_object126641:head [delete] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e227)", 2024-09-13T11:27:12.537 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:26:53.035771+0000", 2024-09-13T11:27:12.537 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 19.426564063000001, 2024-09-13T11:27:12.537 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.30792321099999997, 2024-09-13T11:27:12.537 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:27:12.537 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:27:12.537 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:27:12.537 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:27:12.537 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:27:12.537 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:27:12.537 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 266944 2024-09-13T11:27:12.537 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:12.538 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:27:12.538 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:12.538 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:27:12.538 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:53.035771+0000", 2024-09-13T11:27:12.538 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:27:12.538 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:12.538 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:12.538 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:27:12.538 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:53.035771+0000", 2024-09-13T11:27:12.538 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:27:12.538 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:12.538 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:12.538 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:27:12.538 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:53.035774+0000", 2024-09-13T11:27:12.539 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.4669999999999998e-06 2024-09-13T11:27:12.539 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:12.539 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:12.539 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:27:12.539 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:53.035781+0000", 2024-09-13T11:27:12.539 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.661e-06 2024-09-13T11:27:12.539 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:12.539 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:12.539 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:27:12.539 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:53.035794+0000", 2024-09-13T11:27:12.539 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.2774e-05 2024-09-13T11:27:12.539 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:12.539 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:12.540 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:27:12.540 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:53.035806+0000", 2024-09-13T11:27:12.540 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.1426e-05 2024-09-13T11:27:12.540 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:12.540 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:12.540 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:27:12.540 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:53.035930+0000", 2024-09-13T11:27:12.540 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00012434600000000001 2024-09-13T11:27:12.540 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:12.540 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:12.540 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-09-13T11:27:12.541 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:53.035951+0000", 2024-09-13T11:27:12.541 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.0837000000000001e-05 2024-09-13T11:27:12.541 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:12.541 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:12.541 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:27:12.541 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:53.308946+0000", 2024-09-13T11:27:12.541 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.27299521900000001 2024-09-13T11:27:12.541 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:12.541 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:12.541 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for degraded object", 2024-09-13T11:27:12.541 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:53.309423+0000", 2024-09-13T11:27:12.541 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00047727799999999998 2024-09-13T11:27:12.541 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:12.542 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:12.542 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:27:12.542 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:53.320723+0000", 2024-09-13T11:27:12.542 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.011299774 2024-09-13T11:27:12.542 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:12.542 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:12.542 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:27:12.542 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:53.320867+0000", 2024-09-13T11:27:12.542 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000143511 2024-09-13T11:27:12.542 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:12.542 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:12.542 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 0,10", 2024-09-13T11:27:12.542 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:53.321063+0000", 2024-09-13T11:27:12.543 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000196763 2024-09-13T11:27:12.543 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:12.543 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:12.543 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:27:12.543 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:53.331226+0000", 2024-09-13T11:27:12.543 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.010162492 2024-09-13T11:27:12.543 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:12.543 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:12.543 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:27:12.543 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:53.343345+0000", 2024-09-13T11:27:12.543 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.012119512000000001 2024-09-13T11:27:12.544 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:12.544 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:12.544 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:27:12.544 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:53.343381+0000", 2024-09-13T11:27:12.544 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.5392000000000003e-05 2024-09-13T11:27:12.544 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:12.544 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:12.544 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:27:12.544 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:53.343433+0000", 2024-09-13T11:27:12.544 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.1990000000000002e-05 2024-09-13T11:27:12.544 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:12.545 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:12.545 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:27:12.545 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:53.343694+0000", 2024-09-13T11:27:12.545 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00026176900000000002 2024-09-13T11:27:12.545 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:27:12.545 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:27:12.545 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:27:12.545 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:12.545 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:12.545 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:266995 3.32 3:4edb55ad:::benchmark_data_smithi177_31116_object126692:head [delete] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e227)", 2024-09-13T11:27:12.546 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:26:53.035828+0000", 2024-09-13T11:27:12.546 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 19.426507156, 2024-09-13T11:27:12.546 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.30850358999999999, 2024-09-13T11:27:12.546 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:27:12.546 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:27:12.546 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:27:12.546 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:27:12.546 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:27:12.546 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:27:12.546 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 266995 2024-09-13T11:27:12.547 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:12.547 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:27:12.547 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:12.547 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:27:12.547 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:53.035828+0000", 2024-09-13T11:27:12.547 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:27:12.547 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:12.547 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:12.547 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:27:12.547 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:53.035828+0000", 2024-09-13T11:27:12.547 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:27:12.547 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:12.547 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:12.547 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:27:12.548 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:53.035829+0000", 2024-09-13T11:27:12.548 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.0100000000000003e-07 2024-09-13T11:27:12.548 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:12.548 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:12.548 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:27:12.548 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:53.035833+0000", 2024-09-13T11:27:12.548 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.1350000000000002e-06 2024-09-13T11:27:12.548 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:12.548 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:12.548 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:27:12.548 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:53.035841+0000", 2024-09-13T11:27:12.548 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.627e-06 2024-09-13T11:27:12.548 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:12.549 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:12.549 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:27:12.549 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:53.035847+0000", 2024-09-13T11:27:12.549 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.8390000000000003e-06 2024-09-13T11:27:12.549 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:12.549 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:12.549 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:27:12.549 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:53.036011+0000", 2024-09-13T11:27:12.549 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00016409399999999999 2024-09-13T11:27:12.549 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:12.549 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:12.549 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-09-13T11:27:12.549 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:53.036020+0000", 2024-09-13T11:27:12.549 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.533e-06 2024-09-13T11:27:12.550 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:12.550 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:12.550 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:27:12.550 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:53.309896+0000", 2024-09-13T11:27:12.550 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.27387577699999999 2024-09-13T11:27:12.550 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:12.550 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:12.550 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for degraded object", 2024-09-13T11:27:12.550 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:53.310678+0000", 2024-09-13T11:27:12.550 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00078151900000000005 2024-09-13T11:27:12.550 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:12.550 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:12.550 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:27:12.551 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:53.325984+0000", 2024-09-13T11:27:12.551 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.01530629 2024-09-13T11:27:12.551 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:12.551 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:12.551 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:27:12.551 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:53.326144+0000", 2024-09-13T11:27:12.551 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00016016500000000001 2024-09-13T11:27:12.551 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:12.551 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:12.551 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 0,10", 2024-09-13T11:27:12.551 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:53.326254+0000", 2024-09-13T11:27:12.551 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00010958799999999999 2024-09-13T11:27:12.551 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:12.552 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:12.552 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:27:12.552 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:53.334318+0000", 2024-09-13T11:27:12.552 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0080645769999999999 2024-09-13T11:27:12.552 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:12.552 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:12.552 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:27:12.552 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:53.343837+0000", 2024-09-13T11:27:12.552 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0095191330000000008 2024-09-13T11:27:12.552 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:12.552 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:12.552 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:27:12.552 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:53.343952+0000", 2024-09-13T11:27:12.552 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0001149 2024-09-13T11:27:12.553 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:12.553 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:12.553 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:27:12.553 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:53.344009+0000", 2024-09-13T11:27:12.553 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.6665e-05 2024-09-13T11:27:12.553 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:12.553 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:12.553 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:27:12.553 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:53.344332+0000", 2024-09-13T11:27:12.553 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00032284700000000002 2024-09-13T11:27:12.553 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:27:12.553 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:27:12.553 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:27:12.554 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:12.554 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:12.554 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:267198 3.32 3:4f6daeb8:::benchmark_data_smithi177_31116_object126895:head [delete] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e227)", 2024-09-13T11:27:12.554 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:26:53.035978+0000", 2024-09-13T11:27:12.554 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 19.426357417999998, 2024-09-13T11:27:12.554 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.30891107800000001, 2024-09-13T11:27:12.554 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:27:12.554 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:27:12.554 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:27:12.554 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:27:12.554 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:27:12.554 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:27:12.554 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 267198 2024-09-13T11:27:12.555 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:12.555 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:27:12.555 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:12.555 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:27:12.555 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:53.035978+0000", 2024-09-13T11:27:12.555 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:27:12.555 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:12.555 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:12.555 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:27:12.555 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:53.035978+0000", 2024-09-13T11:27:12.555 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:27:12.555 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:12.555 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:12.556 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:27:12.556 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:53.035980+0000", 2024-09-13T11:27:12.556 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.3719999999999999e-06 2024-09-13T11:27:12.556 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:12.556 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:12.556 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:27:12.556 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:53.035990+0000", 2024-09-13T11:27:12.556 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.4769999999999992e-06 2024-09-13T11:27:12.556 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:12.556 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:12.556 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:27:12.556 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:53.036003+0000", 2024-09-13T11:27:12.556 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.3223e-05 2024-09-13T11:27:12.557 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:12.557 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:12.557 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:27:12.557 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:53.036016+0000", 2024-09-13T11:27:12.557 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.2943e-05 2024-09-13T11:27:12.557 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:12.557 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:12.557 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:27:12.557 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:53.036091+0000", 2024-09-13T11:27:12.557 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.5346999999999995e-05 2024-09-13T11:27:12.557 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:12.557 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:12.557 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-09-13T11:27:12.557 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:53.036100+0000", 2024-09-13T11:27:12.558 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.9469999999999992e-06 2024-09-13T11:27:12.558 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:12.558 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:12.558 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:27:12.558 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:53.310844+0000", 2024-09-13T11:27:12.558 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.27474415899999999 2024-09-13T11:27:12.558 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:12.558 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:12.558 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for degraded object", 2024-09-13T11:27:12.558 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:53.312126+0000", 2024-09-13T11:27:12.558 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.001282107 2024-09-13T11:27:12.558 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:12.558 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:12.559 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:27:12.559 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:53.329190+0000", 2024-09-13T11:27:12.559 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.017063389000000002 2024-09-13T11:27:12.559 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:12.559 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:12.559 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:27:12.559 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:53.329276+0000", 2024-09-13T11:27:12.559 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.5692999999999994e-05 2024-09-13T11:27:12.559 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:12.559 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:12.559 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 0,10", 2024-09-13T11:27:12.559 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:53.329383+0000", 2024-09-13T11:27:12.559 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000107717 2024-09-13T11:27:12.559 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:12.560 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:12.560 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:27:12.560 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:53.335683+0000", 2024-09-13T11:27:12.560 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0062998009999999998 2024-09-13T11:27:12.560 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:12.560 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:12.560 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:27:12.560 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:53.344465+0000", 2024-09-13T11:27:12.560 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.008782069 2024-09-13T11:27:12.560 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:12.560 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:12.560 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:27:12.560 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:53.344565+0000", 2024-09-13T11:27:12.560 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.9406000000000003e-05 2024-09-13T11:27:12.561 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:12.561 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:12.561 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:27:12.561 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:53.344623+0000", 2024-09-13T11:27:12.561 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.8476000000000001e-05 2024-09-13T11:27:12.561 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:12.561 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:12.561 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:27:12.561 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:53.344889+0000", 2024-09-13T11:27:12.561 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00026595200000000002 2024-09-13T11:27:12.561 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:27:12.562 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:27:12.562 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:27:12.562 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:12.562 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:12.562 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:267211 3.32 3:4ccc7b14:::benchmark_data_smithi177_31116_object126908:head [delete] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e227)", 2024-09-13T11:27:12.562 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:26:53.036055+0000", 2024-09-13T11:27:12.562 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 19.426280647999999, 2024-09-13T11:27:12.562 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.30934412999999999, 2024-09-13T11:27:12.562 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:27:12.562 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:27:12.562 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:27:12.563 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:27:12.563 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:27:12.563 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:27:12.563 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 267211 2024-09-13T11:27:12.563 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:12.563 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:27:12.563 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:12.563 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:27:12.563 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:53.036055+0000", 2024-09-13T11:27:12.564 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:27:12.564 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:12.564 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:12.564 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:27:12.564 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:53.036055+0000", 2024-09-13T11:27:12.564 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:27:12.564 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:12.564 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:12.564 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:27:12.564 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:53.036056+0000", 2024-09-13T11:27:12.564 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.46e-06 2024-09-13T11:27:12.564 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:12.565 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:12.565 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:27:12.565 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:53.036061+0000", 2024-09-13T11:27:12.565 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.3079999999999998e-06 2024-09-13T11:27:12.565 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:12.565 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:12.565 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:27:12.565 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:53.036070+0000", 2024-09-13T11:27:12.565 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.8939999999999999e-06 2024-09-13T11:27:12.565 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:12.565 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:12.565 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:27:12.565 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:53.036077+0000", 2024-09-13T11:27:12.565 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.0199999999999997e-06 2024-09-13T11:27:12.566 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:12.566 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:12.566 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:27:12.566 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:53.036163+0000", 2024-09-13T11:27:12.566 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.5993000000000002e-05 2024-09-13T11:27:12.566 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:12.566 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:12.566 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-09-13T11:27:12.566 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:53.036171+0000", 2024-09-13T11:27:12.566 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.199e-06 2024-09-13T11:27:12.566 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:12.566 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:12.566 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:27:12.566 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:53.312204+0000", 2024-09-13T11:27:12.567 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.276033419 2024-09-13T11:27:12.567 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:12.567 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:12.567 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for degraded object", 2024-09-13T11:27:12.567 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:53.312573+0000", 2024-09-13T11:27:12.567 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00036915800000000002 2024-09-13T11:27:12.567 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:12.567 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:12.567 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:27:12.567 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:53.332647+0000", 2024-09-13T11:27:12.567 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.020074195999999999 2024-09-13T11:27:12.567 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:12.567 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:12.567 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:27:12.568 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:53.332792+0000", 2024-09-13T11:27:12.568 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00014504000000000001 2024-09-13T11:27:12.568 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:12.568 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:12.568 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 0,10", 2024-09-13T11:27:12.568 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:53.332995+0000", 2024-09-13T11:27:12.568 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00020285600000000001 2024-09-13T11:27:12.568 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:12.568 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:12.568 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:27:12.568 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:53.339667+0000", 2024-09-13T11:27:12.568 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0066719170000000003 2024-09-13T11:27:12.568 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:12.568 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:12.569 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:27:12.569 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:53.344997+0000", 2024-09-13T11:27:12.569 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0053296000000000003 2024-09-13T11:27:12.569 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:12.569 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:12.569 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:27:12.569 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:53.345086+0000", 2024-09-13T11:27:12.569 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.8980000000000005e-05 2024-09-13T11:27:12.569 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:12.569 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:12.569 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:27:12.569 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:53.345142+0000", 2024-09-13T11:27:12.569 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.6428000000000003e-05 2024-09-13T11:27:12.569 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:12.570 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:12.570 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:27:12.570 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:53.345399+0000", 2024-09-13T11:27:12.570 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000256662 2024-09-13T11:27:12.570 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:27:12.570 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:27:12.570 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:27:12.570 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:12.570 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:12.570 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:267319 3.32 3:4e44814a:::benchmark_data_smithi177_31116_object127016:head [delete] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e227)", 2024-09-13T11:27:12.570 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:26:53.036090+0000", 2024-09-13T11:27:12.570 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 19.426245653999999, 2024-09-13T11:27:12.570 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.30974317299999998, 2024-09-13T11:27:12.571 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:27:12.571 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:27:12.571 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:27:12.571 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:27:12.571 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:27:12.571 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:27:12.571 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 267319 2024-09-13T11:27:12.571 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:12.571 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:27:12.571 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:12.571 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:27:12.571 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:53.036090+0000", 2024-09-13T11:27:12.571 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:27:12.571 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:12.571 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:12.572 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:27:12.572 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:53.036090+0000", 2024-09-13T11:27:12.572 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:27:12.572 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:12.572 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:12.572 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:27:12.572 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:53.036090+0000", 2024-09-13T11:27:12.572 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.5499999999999997e-07 2024-09-13T11:27:12.572 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:12.572 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:12.572 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:27:12.572 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:53.036099+0000", 2024-09-13T11:27:12.572 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.2290000000000003e-06 2024-09-13T11:27:12.573 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:12.573 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:12.573 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:27:12.573 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:53.036105+0000", 2024-09-13T11:27:12.573 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.781e-06 2024-09-13T11:27:12.573 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:12.573 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:12.573 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:27:12.573 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:53.036112+0000", 2024-09-13T11:27:12.573 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.1090000000000003e-06 2024-09-13T11:27:12.573 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:12.573 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:12.573 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:27:12.573 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:53.036192+0000", 2024-09-13T11:27:12.573 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.0712999999999998e-05 2024-09-13T11:27:12.574 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:12.574 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:12.574 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-09-13T11:27:12.574 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:53.036199+0000", 2024-09-13T11:27:12.574 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.7039999999999997e-06 2024-09-13T11:27:12.574 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:12.574 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:12.574 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:27:12.574 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:53.312645+0000", 2024-09-13T11:27:12.574 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.27644615700000003 2024-09-13T11:27:12.574 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:12.574 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:12.574 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for degraded object", 2024-09-13T11:27:12.574 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:53.313622+0000", 2024-09-13T11:27:12.575 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00097722499999999997 2024-09-13T11:27:12.575 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:12.575 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:12.575 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:27:12.575 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:53.335768+0000", 2024-09-13T11:27:12.575 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.022145821999999999 2024-09-13T11:27:12.575 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:12.575 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:12.575 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:27:12.575 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:53.336028+0000", 2024-09-13T11:27:12.575 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00026027599999999999 2024-09-13T11:27:12.575 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:12.575 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:12.575 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 0,10", 2024-09-13T11:27:12.576 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:53.336417+0000", 2024-09-13T11:27:12.576 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00038850100000000003 2024-09-13T11:27:12.576 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:12.576 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:12.576 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:27:12.576 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:53.340517+0000", 2024-09-13T11:27:12.576 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.004100495 2024-09-13T11:27:12.576 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:12.576 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:12.576 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:27:12.576 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:53.345465+0000", 2024-09-13T11:27:12.576 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.004947094 2024-09-13T11:27:12.576 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:12.576 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:12.577 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:27:12.577 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:53.345512+0000", 2024-09-13T11:27:12.577 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.7120000000000003e-05 2024-09-13T11:27:12.577 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:12.577 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:12.577 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:27:12.577 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:53.345571+0000", 2024-09-13T11:27:12.577 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.9648999999999998e-05 2024-09-13T11:27:12.577 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:12.577 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:12.577 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:27:12.577 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:53.345833+0000", 2024-09-13T11:27:12.578 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00026154299999999999 2024-09-13T11:27:12.578 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:27:12.578 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:27:12.578 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:27:12.578 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:12.578 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:12.578 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:267359 3.32 3:4fbe1345:::benchmark_data_smithi177_31116_object127056:head [delete] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e227)", 2024-09-13T11:27:12.578 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:26:53.036130+0000", 2024-09-13T11:27:12.578 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 19.426204873, 2024-09-13T11:27:12.578 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.31064794699999998, 2024-09-13T11:27:12.578 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:27:12.578 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:27:12.579 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:27:12.579 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:27:12.579 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:27:12.579 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:27:12.579 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 267359 2024-09-13T11:27:12.579 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:12.579 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:27:12.579 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:12.579 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:27:12.579 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:53.036130+0000", 2024-09-13T11:27:12.579 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:27:12.579 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:12.580 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:12.580 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:27:12.580 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:53.036130+0000", 2024-09-13T11:27:12.580 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:27:12.580 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:12.580 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:12.580 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:27:12.580 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:53.036131+0000", 2024-09-13T11:27:12.580 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.7199999999999998e-07 2024-09-13T11:27:12.580 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:12.580 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:12.581 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:27:12.581 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:53.036135+0000", 2024-09-13T11:27:12.581 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.6150000000000001e-06 2024-09-13T11:27:12.581 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:12.581 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:12.581 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:27:12.581 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:53.036142+0000", 2024-09-13T11:27:12.581 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.1840000000000002e-06 2024-09-13T11:27:12.581 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:12.582 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:12.582 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:27:12.582 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:53.036148+0000", 2024-09-13T11:27:12.582 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.0839999999999997e-06 2024-09-13T11:27:12.582 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:12.582 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:12.582 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:27:12.582 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:53.036221+0000", 2024-09-13T11:27:12.582 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.2540999999999995e-05 2024-09-13T11:27:12.582 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:12.582 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:12.582 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-09-13T11:27:12.583 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:53.036227+0000", 2024-09-13T11:27:12.583 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.8240000000000002e-06 2024-09-13T11:27:12.583 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:12.583 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:12.583 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:27:12.583 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:53.313744+0000", 2024-09-13T11:27:12.583 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.27751620599999999 2024-09-13T11:27:12.583 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:12.583 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:12.583 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for degraded object", 2024-09-13T11:27:12.583 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:53.314981+0000", 2024-09-13T11:27:12.583 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.001237082 2024-09-13T11:27:12.584 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:12.584 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:12.584 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:27:12.584 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:53.340571+0000", 2024-09-13T11:27:12.584 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.025590695 2024-09-13T11:27:12.584 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:12.584 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:12.584 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:27:12.584 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:53.340735+0000", 2024-09-13T11:27:12.584 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00016394099999999999 2024-09-13T11:27:12.584 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:12.585 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:12.585 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 0,10", 2024-09-13T11:27:12.585 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:53.340953+0000", 2024-09-13T11:27:12.585 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00021797799999999999 2024-09-13T11:27:12.585 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:12.585 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:12.585 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:27:12.585 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:53.344905+0000", 2024-09-13T11:27:12.585 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0039517880000000003 2024-09-13T11:27:12.585 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:12.585 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:12.586 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:27:12.586 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:53.345964+0000", 2024-09-13T11:27:12.586 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.001058701 2024-09-13T11:27:12.586 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:12.586 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:12.586 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:27:12.586 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:53.346095+0000", 2024-09-13T11:27:12.586 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00013125699999999999 2024-09-13T11:27:12.586 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:12.586 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:12.586 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:27:12.587 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:53.346248+0000", 2024-09-13T11:27:12.587 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000152836 2024-09-13T11:27:12.587 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:12.587 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:12.587 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:27:12.587 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:53.346778+0000", 2024-09-13T11:27:12.587 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00053044300000000002 2024-09-13T11:27:12.587 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:27:12.587 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:27:12.587 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:27:12.587 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:12.588 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:12.588 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:273235 3.12 3:4b3e9dde:::benchmark_data_smithi177_31116_object132932:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e231)", 2024-09-13T11:27:12.588 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:26:59.292209+0000", 2024-09-13T11:27:12.588 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 13.170126355000001, 2024-09-13T11:27:12.588 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.34537550099999997, 2024-09-13T11:27:12.588 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:27:12.588 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:27:12.588 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:27:12.588 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:27:12.588 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:27:12.588 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:27:12.589 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 273235 2024-09-13T11:27:12.589 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:12.589 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:27:12.589 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:12.589 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:27:12.589 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:59.292209+0000", 2024-09-13T11:27:12.589 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:27:12.589 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:12.589 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:12.589 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:27:12.589 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:59.292209+0000", 2024-09-13T11:27:12.589 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:27:12.590 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:12.590 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:12.590 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:27:12.590 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:59.292212+0000", 2024-09-13T11:27:12.590 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.5490000000000001e-06 2024-09-13T11:27:12.590 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:12.590 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:12.590 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:27:12.590 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:59.292218+0000", 2024-09-13T11:27:12.590 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.1430000000000003e-06 2024-09-13T11:27:12.590 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:12.591 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:12.591 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:27:12.591 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:59.292232+0000", 2024-09-13T11:27:12.591 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.4078e-05 2024-09-13T11:27:12.591 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:12.591 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:12.591 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:27:12.591 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:59.292241+0000", 2024-09-13T11:27:12.591 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.7800000000000006e-06 2024-09-13T11:27:12.591 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:12.591 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:12.592 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:27:12.592 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:59.292296+0000", 2024-09-13T11:27:12.592 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.5414000000000002e-05 2024-09-13T11:27:12.592 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:12.592 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:12.592 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:27:12.592 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:59.292464+0000", 2024-09-13T11:27:12.592 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000168548 2024-09-13T11:27:12.592 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:12.592 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:12.592 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 7,14", 2024-09-13T11:27:12.592 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:59.292611+0000", 2024-09-13T11:27:12.593 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00014622499999999999 2024-09-13T11:27:12.593 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:12.593 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:12.593 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:27:12.593 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:59.295250+0000", 2024-09-13T11:27:12.593 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0026396290000000001 2024-09-13T11:27:12.593 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:12.593 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:12.593 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:27:12.593 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:59.295347+0000", 2024-09-13T11:27:12.593 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.6712000000000005e-05 2024-09-13T11:27:12.593 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:12.593 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:12.593 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:27:12.594 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:59.637449+0000", 2024-09-13T11:27:12.594 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.34210160899999997 2024-09-13T11:27:12.594 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:12.594 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:12.594 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:27:12.594 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:59.637494+0000", 2024-09-13T11:27:12.594 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.5775999999999997e-05 2024-09-13T11:27:12.594 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:12.594 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:12.594 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:27:12.594 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:59.637584+0000", 2024-09-13T11:27:12.594 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.0037999999999994e-05 2024-09-13T11:27:12.594 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:27:12.594 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:27:12.595 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:27:12.595 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:12.595 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:12.595 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:273235 3.12 3:4b3e9dde:::benchmark_data_smithi177_31116_object132932:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e231)", 2024-09-13T11:27:12.595 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:26:59.297179+0000", 2024-09-13T11:27:12.595 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 13.165156392, 2024-09-13T11:27:12.595 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.34032614700000002, 2024-09-13T11:27:12.595 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:27:12.595 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:27:12.595 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "waiting for ondisk", 2024-09-13T11:27:12.595 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:27:12.595 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:27:12.595 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:27:12.595 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 273235 2024-09-13T11:27:12.596 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:12.596 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:27:12.596 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:12.596 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:27:12.596 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:59.297179+0000", 2024-09-13T11:27:12.596 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:27:12.596 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:12.596 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:12.596 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:27:12.596 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:59.297179+0000", 2024-09-13T11:27:12.596 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:27:12.596 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:12.596 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:12.596 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:27:12.597 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:59.297179+0000", 2024-09-13T11:27:12.597 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.89e-07 2024-09-13T11:27:12.597 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:12.597 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:12.597 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:27:12.597 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:59.297181+0000", 2024-09-13T11:27:12.597 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.0590000000000001e-06 2024-09-13T11:27:12.597 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:12.597 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:12.597 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:27:12.597 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:59.297186+0000", 2024-09-13T11:27:12.597 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.5630000000000002e-06 2024-09-13T11:27:12.597 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:12.598 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:12.598 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:27:12.598 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:59.297190+0000", 2024-09-13T11:27:12.598 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.8160000000000004e-06 2024-09-13T11:27:12.598 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:12.598 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:12.598 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:27:12.598 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:59.297583+0000", 2024-09-13T11:27:12.598 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00039354100000000002 2024-09-13T11:27:12.598 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:12.598 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:12.599 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for ondisk", 2024-09-13T11:27:12.599 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:59.297634+0000", 2024-09-13T11:27:12.599 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.0722e-05 2024-09-13T11:27:12.599 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:12.599 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:12.599 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:27:12.599 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:59.637505+0000", 2024-09-13T11:27:12.599 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.33987095699999997 2024-09-13T11:27:12.599 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:27:12.599 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:27:12.599 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:27:12.599 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:12.599 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:12.599 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "MOSDPGRecoveryDelete(3.3b e241,237 3:df60fe88:::benchmark_data_smithi177_31116_object26630:head,159'6976)", 2024-09-13T11:27:12.600 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:27:10.482925+0000", 2024-09-13T11:27:12.600 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 1.9794099200000002, 2024-09-13T11:27:12.600 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.716230056, 2024-09-13T11:27:12.600 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:27:12.600 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:27:12.600 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-09-13T11:27:12.600 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:27:12.600 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:12.600 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:27:12.600 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:27:10.482925+0000", 2024-09-13T11:27:12.600 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:27:12.600 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:12.600 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:12.601 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:27:12.601 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:27:10.482925+0000", 2024-09-13T11:27:12.601 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:27:12.601 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:12.601 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:12.601 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:27:12.601 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:27:10.482928+0000", 2024-09-13T11:27:12.601 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.1239999999999999e-06 2024-09-13T11:27:12.601 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:12.601 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:12.601 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:27:12.601 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:27:10.482938+0000", 2024-09-13T11:27:12.601 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0093e-05 2024-09-13T11:27:12.601 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:12.602 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:12.602 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:27:12.602 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:27:11.198174+0000", 2024-09-13T11:27:12.602 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.71523653899999995 2024-09-13T11:27:12.602 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:12.602 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:12.602 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:27:12.602 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:27:11.198186+0000", 2024-09-13T11:27:12.602 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.2071e-05 2024-09-13T11:27:12.602 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:12.602 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:12.602 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:27:12.602 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:27:11.198922+0000", 2024-09-13T11:27:12.602 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00073529300000000004 2024-09-13T11:27:12.603 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:12.603 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:12.603 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:27:12.603 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:27:11.198950+0000", 2024-09-13T11:27:12.603 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.8293000000000002e-05 2024-09-13T11:27:12.603 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:12.603 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:12.603 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:27:12.603 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:27:11.199155+0000", 2024-09-13T11:27:12.603 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00020564300000000001 2024-09-13T11:27:12.603 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:27:12.603 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:27:12.603 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:27:12.603 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:27:12.604 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:27:12.604 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-09-13T11:27:12.607 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-09-13T11:27:12.608 INFO:tasks.ceph.osd.9.smithi092.stderr:2024-09-13T11:27:12.603+0000 7ff5ab3d0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26348) UID: 0 2024-09-13T11:27:12.678 INFO:tasks.ceph.osd.10:Stopped 2024-09-13T11:27:12.678 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd down 10 2024-09-13T11:27:12.706 INFO:tasks.ceph.osd.13.smithi092.stderr:2024-09-13T11:27:12.703+0000 7f3e5afea640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26350) UID: 0 2024-09-13T11:27:12.790 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-09-13T11:27:12.790 INFO:teuthology.orchestra.run.smithi092.stdout: "ops": [], 2024-09-13T11:27:12.790 INFO:teuthology.orchestra.run.smithi092.stdout: "num_ops": 0 2024-09-13T11:27:12.791 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-09-13T11:27:12.806 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-09-13T11:27:12.807 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-13T11:27:12.806+0000 7f6a68775640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26307) UID: 0 2024-09-13T11:27:12.898 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T11:27:12.908 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-09-13T11:27:12.907+0000 7f97befc9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26315) UID: 0 2024-09-13T11:27:12.984 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-09-13T11:27:12.984 INFO:teuthology.orchestra.run.smithi092.stdout: "ops": [], 2024-09-13T11:27:12.984 INFO:teuthology.orchestra.run.smithi092.stdout: "complaint_time": 30, 2024-09-13T11:27:12.984 INFO:teuthology.orchestra.run.smithi092.stdout: "num_blocked_ops": 0 2024-09-13T11:27:12.984 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-09-13T11:27:12.999 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-09-13T11:27:13.009 INFO:tasks.ceph.osd.2.smithi162.stderr:2024-09-13T11:27:13.007+0000 7fe92b35a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26313) UID: 0 2024-09-13T11:27:13.068 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-09-13T11:27:13.066+0000 7fb2a5596640 -1 mon.a@0(leader).osd e242 definitely_dead 0 2024-09-13T11:27:13.068 INFO:teuthology.orchestra.run.smithi001.stderr:osd.10 is already down. 2024-09-13T11:27:13.075 INFO:teuthology.orchestra.run.smithi001.stdout:osd.0: {} 2024-09-13T11:27:13.075 INFO:teuthology.orchestra.run.smithi001.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T11:27:13.093 INFO:tasks.thrashosds.thrasher:Removing osd 10, in_osds are: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 15, 14] 2024-09-13T11:27:13.093 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd out 10 2024-09-13T11:27:13.105 INFO:teuthology.orchestra.run.smithi001.stdout:osd.1: {} 2024-09-13T11:27:13.105 INFO:teuthology.orchestra.run.smithi001.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T11:27:13.110 INFO:tasks.ceph.osd.15.smithi177.stderr:2024-09-13T11:27:13.109+0000 7f9d6ce77640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26578) UID: 0 2024-09-13T11:27:13.132 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-09-13T11:27:13.133 INFO:teuthology.orchestra.run.smithi092.stdout: "size": 20, 2024-09-13T11:27:13.133 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 600, 2024-09-13T11:27:13.133 INFO:teuthology.orchestra.run.smithi092.stdout: "ops": [ 2024-09-13T11:27:13.133 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:13.133 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "MOSDPGPush(3.19 76/70 [PushOp(3:9eacf3d0:::benchmark_data_smithi177_31116_object19747:head, version: 54'1236, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:9eacf3d0:::benchmark_data_smithi177_31116_object19747:head@54'1236, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:27:13.133 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:32.704676+0000", 2024-09-13T11:27:13.133 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 220.427186917, 2024-09-13T11:27:13.133 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.23525014899999999, 2024-09-13T11:27:13.133 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:27:13.134 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:27:13.134 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-09-13T11:27:13.134 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:27:13.134 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:13.134 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:27:13.134 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:32.704676+0000", 2024-09-13T11:27:13.135 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:27:13.135 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:13.135 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:13.136 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:27:13.136 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:32.704676+0000", 2024-09-13T11:27:13.136 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:27:13.136 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:13.136 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:13.136 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:27:13.136 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:32.704677+0000", 2024-09-13T11:27:13.136 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.531e-06 2024-09-13T11:27:13.136 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:13.137 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:13.137 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:27:13.137 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:32.704749+0000", 2024-09-13T11:27:13.137 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.1451000000000004e-05 2024-09-13T11:27:13.137 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:13.137 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:13.137 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:27:13.137 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:32.938053+0000", 2024-09-13T11:27:13.137 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.233304752 2024-09-13T11:27:13.137 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:13.137 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:13.137 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:27:13.137 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:32.938087+0000", 2024-09-13T11:27:13.137 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.4016000000000001e-05 2024-09-13T11:27:13.138 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:13.138 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:13.138 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:27:13.138 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:32.938351+0000", 2024-09-13T11:27:13.138 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00026397500000000001 2024-09-13T11:27:13.138 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:13.138 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:13.138 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:27:13.138 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:32.938381+0000", 2024-09-13T11:27:13.138 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.0001999999999998e-05 2024-09-13T11:27:13.138 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:13.138 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:13.138 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:27:13.138 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:32.939926+0000", 2024-09-13T11:27:13.139 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0015444219999999999 2024-09-13T11:27:13.139 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:27:13.139 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:27:13.139 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:27:13.139 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:13.139 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:13.139 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "MOSDPGPush(3.23 80/78 [PushOp(3:c59ecc6d:::benchmark_data_smithi177_31116_object17005:head, version: 52'1054, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:c59ecc6d:::benchmark_data_smithi177_31116_object17005:head@52'1054, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:27:13.139 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:37.433040+0000", 2024-09-13T11:27:13.139 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 215.698822941, 2024-09-13T11:27:13.139 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.26658219100000002, 2024-09-13T11:27:13.139 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:27:13.139 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:27:13.139 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-09-13T11:27:13.139 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:27:13.140 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:13.140 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:27:13.140 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:37.433040+0000", 2024-09-13T11:27:13.140 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:27:13.140 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:13.140 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:13.140 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:27:13.140 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:37.433040+0000", 2024-09-13T11:27:13.140 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:27:13.140 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:13.140 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:13.140 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:27:13.140 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:37.433042+0000", 2024-09-13T11:27:13.140 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.3999999999999999e-06 2024-09-13T11:27:13.141 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:13.141 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:13.141 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:27:13.141 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:37.433090+0000", 2024-09-13T11:27:13.141 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.8065999999999997e-05 2024-09-13T11:27:13.141 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:13.142 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:13.142 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:27:13.142 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:37.698797+0000", 2024-09-13T11:27:13.142 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.26570674500000002 2024-09-13T11:27:13.142 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:13.142 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:13.142 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:27:13.143 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:37.698827+0000", 2024-09-13T11:27:13.143 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.0505e-05 2024-09-13T11:27:13.143 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:13.143 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:13.143 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:27:13.143 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:37.698941+0000", 2024-09-13T11:27:13.143 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00011403799999999999 2024-09-13T11:27:13.143 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:13.143 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:13.143 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:27:13.143 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:37.698956+0000", 2024-09-13T11:27:13.143 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.4765e-05 2024-09-13T11:27:13.143 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:13.143 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:13.144 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:27:13.144 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:37.699622+0000", 2024-09-13T11:27:13.144 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00066567199999999997 2024-09-13T11:27:13.144 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:27:13.144 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:27:13.144 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:27:13.144 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:13.144 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:13.144 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "MOSDPGPush(3.1a 85/78 [PushOp(3:5dc85525:::benchmark_data_smithi177_31116_object11755:head, version: 49'723, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:5dc85525:::benchmark_data_smithi177_31116_object11755:head@49'723, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:27:13.144 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:42.422061+0000", 2024-09-13T11:27:13.144 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 210.70980148800001, 2024-09-13T11:27:13.144 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.52875617500000005, 2024-09-13T11:27:13.144 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:27:13.145 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:27:13.145 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-09-13T11:27:13.145 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:27:13.145 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:13.145 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:27:13.145 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.422061+0000", 2024-09-13T11:27:13.145 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:27:13.145 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:13.145 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:13.145 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:27:13.145 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.422061+0000", 2024-09-13T11:27:13.145 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:27:13.145 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:13.145 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:13.146 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:27:13.146 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.422064+0000", 2024-09-13T11:27:13.146 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.5409999999999999e-06 2024-09-13T11:27:13.146 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:13.146 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:13.146 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:27:13.146 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.422139+0000", 2024-09-13T11:27:13.146 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.5488999999999995e-05 2024-09-13T11:27:13.146 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:13.146 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:13.146 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:27:13.146 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.948781+0000", 2024-09-13T11:27:13.146 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.52664230999999995 2024-09-13T11:27:13.146 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:13.147 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:13.147 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:27:13.147 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.948841+0000", 2024-09-13T11:27:13.147 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.9450999999999997e-05 2024-09-13T11:27:13.147 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:13.147 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:13.147 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:27:13.147 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.949035+0000", 2024-09-13T11:27:13.147 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00019380600000000001 2024-09-13T11:27:13.147 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:13.147 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:13.147 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:27:13.147 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.949060+0000", 2024-09-13T11:27:13.147 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.5381999999999999e-05 2024-09-13T11:27:13.148 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:13.148 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:13.148 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:27:13.148 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.950817+0000", 2024-09-13T11:27:13.148 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0017571959999999999 2024-09-13T11:27:13.148 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:27:13.148 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:27:13.148 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:27:13.148 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:13.148 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:13.148 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "MOSDPGPush(3.1a 89/78 [PushOp(3:59270734:::benchmark_data_smithi177_31116_object20039:head, version: 54'1256, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:59270734:::benchmark_data_smithi177_31116_object20039:head@54'1256, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:27:13.148 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:46.607562+0000", 2024-09-13T11:27:13.148 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 206.524300711, 2024-09-13T11:27:13.148 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.32123945300000001, 2024-09-13T11:27:13.149 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:27:13.149 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:27:13.149 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-09-13T11:27:13.149 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:27:13.149 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:13.149 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:27:13.149 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.607562+0000", 2024-09-13T11:27:13.149 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:27:13.149 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:13.149 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:13.149 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:27:13.149 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.607562+0000", 2024-09-13T11:27:13.149 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:27:13.149 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:13.150 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:13.150 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:27:13.150 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.607564+0000", 2024-09-13T11:27:13.150 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.8610000000000001e-06 2024-09-13T11:27:13.150 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:13.150 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:13.150 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:27:13.150 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.607663+0000", 2024-09-13T11:27:13.150 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.9210999999999996e-05 2024-09-13T11:27:13.150 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:13.150 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:13.150 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:27:13.150 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.925193+0000", 2024-09-13T11:27:13.150 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.317530586 2024-09-13T11:27:13.151 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:13.151 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:13.151 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:27:13.151 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.925250+0000", 2024-09-13T11:27:13.151 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.6925000000000001e-05 2024-09-13T11:27:13.151 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:13.151 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:13.151 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:27:13.151 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.927013+0000", 2024-09-13T11:27:13.151 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.001762853 2024-09-13T11:27:13.151 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:13.152 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:13.152 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:27:13.152 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.927067+0000", 2024-09-13T11:27:13.152 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.3601999999999999e-05 2024-09-13T11:27:13.152 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:13.152 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:13.152 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:27:13.152 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.928801+0000", 2024-09-13T11:27:13.152 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.001734415 2024-09-13T11:27:13.152 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:27:13.152 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:27:13.152 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:27:13.152 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:13.152 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:13.153 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "MOSDPGPush(3.1a 91/78 [PushOp(3:5e5010d2:::benchmark_data_smithi177_31116_object25733:head, version: 58'1609, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:5e5010d2:::benchmark_data_smithi177_31116_object25733:head@58'1609, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:27:13.153 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:48.608485+0000", 2024-09-13T11:27:13.153 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 204.52337759299999, 2024-09-13T11:27:13.153 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.35035599699999997, 2024-09-13T11:27:13.153 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:27:13.153 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:27:13.153 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-09-13T11:27:13.153 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:27:13.153 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:13.153 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:27:13.153 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:48.608485+0000", 2024-09-13T11:27:13.153 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:27:13.153 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:13.154 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:13.154 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:27:13.154 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:48.608485+0000", 2024-09-13T11:27:13.154 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:27:13.154 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:13.154 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:13.154 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:27:13.154 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:48.608488+0000", 2024-09-13T11:27:13.154 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.9399999999999998e-06 2024-09-13T11:27:13.154 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:13.154 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:13.154 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:27:13.154 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:48.608552+0000", 2024-09-13T11:27:13.154 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.3731000000000004e-05 2024-09-13T11:27:13.154 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:13.155 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:13.155 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:27:13.155 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:48.957330+0000", 2024-09-13T11:27:13.155 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.34877806099999997 2024-09-13T11:27:13.155 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:13.155 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:13.155 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:27:13.155 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:48.957361+0000", 2024-09-13T11:27:13.155 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.0852000000000002e-05 2024-09-13T11:27:13.155 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:13.156 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:13.156 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:27:13.156 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:48.957544+0000", 2024-09-13T11:27:13.156 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000183537 2024-09-13T11:27:13.156 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:13.156 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:13.156 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:27:13.156 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:48.957582+0000", 2024-09-13T11:27:13.156 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.7472999999999998e-05 2024-09-13T11:27:13.156 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:13.156 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:13.156 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:27:13.156 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:48.958841+0000", 2024-09-13T11:27:13.156 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0012594030000000001 2024-09-13T11:27:13.157 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:27:13.157 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:27:13.157 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:27:13.157 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:13.157 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:13.157 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop(client.4665.0:99105 3.1a e113/105 3:5ac5090c:::benchmark_data_smithi177_31116_object99104:head v 113'4749, mlcod=113'4746)", 2024-09-13T11:27:13.157 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:24:15.963602+0000", 2024-09-13T11:27:13.157 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 177.16826027900001, 2024-09-13T11:27:13.157 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.23699906300000001, 2024-09-13T11:27:13.157 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:27:13.157 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:27:13.157 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:27:13.157 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:27:13.157 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:13.158 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:27:13.158 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:15.963602+0000", 2024-09-13T11:27:13.158 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:27:13.158 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:13.158 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:13.158 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:27:13.158 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:15.963602+0000", 2024-09-13T11:27:13.158 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:27:13.158 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:13.158 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:13.158 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:27:13.158 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:15.963605+0000", 2024-09-13T11:27:13.158 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.0400000000000001e-06 2024-09-13T11:27:13.158 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:13.159 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:13.159 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:27:13.159 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:15.963718+0000", 2024-09-13T11:27:13.159 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000112796 2024-09-13T11:27:13.159 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:13.159 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:13.159 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:27:13.159 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:16.197994+0000", 2024-09-13T11:27:13.159 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.234276285 2024-09-13T11:27:13.160 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:13.160 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:13.160 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:27:13.160 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:16.198004+0000", 2024-09-13T11:27:13.160 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.5559999999999996e-06 2024-09-13T11:27:13.160 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:13.160 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:13.160 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:27:13.160 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:16.198110+0000", 2024-09-13T11:27:13.160 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00010614 2024-09-13T11:27:13.160 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:13.160 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:13.160 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:27:13.161 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:16.198152+0000", 2024-09-13T11:27:13.161 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.1899e-05 2024-09-13T11:27:13.161 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:13.161 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:13.161 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:27:13.161 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:16.200528+0000", 2024-09-13T11:27:13.161 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00237587 2024-09-13T11:27:13.161 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:13.161 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:13.161 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:27:13.161 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:16.200601+0000", 2024-09-13T11:27:13.161 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.3477000000000001e-05 2024-09-13T11:27:13.161 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:27:13.161 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:27:13.162 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:27:13.162 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:13.162 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:13.162 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop(client.4665.0:129921 3.1a e134/105 3:5fb9088b:::benchmark_data_smithi177_31116_object129920:head v 134'5758, mlcod=134'5755)", 2024-09-13T11:27:13.162 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:24:47.178021+0000", 2024-09-13T11:27:13.162 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 145.95384127299999, 2024-09-13T11:27:13.162 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.33723278600000001, 2024-09-13T11:27:13.162 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:27:13.162 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:27:13.162 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:27:13.162 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:27:13.162 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:13.162 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:27:13.162 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:47.178021+0000", 2024-09-13T11:27:13.163 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:27:13.163 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:13.163 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:13.163 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:27:13.163 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:47.178021+0000", 2024-09-13T11:27:13.163 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:27:13.163 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:13.163 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:13.163 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:27:13.163 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:47.178024+0000", 2024-09-13T11:27:13.163 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.3609999999999999e-06 2024-09-13T11:27:13.163 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:13.163 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:13.163 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:27:13.164 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:47.178102+0000", 2024-09-13T11:27:13.164 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.8325000000000006e-05 2024-09-13T11:27:13.164 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:13.164 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:13.164 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:27:13.164 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:47.512707+0000", 2024-09-13T11:27:13.164 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.33460516000000001 2024-09-13T11:27:13.164 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:13.164 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:13.164 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:27:13.164 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:47.512716+0000", 2024-09-13T11:27:13.164 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.0389999999999993e-06 2024-09-13T11:27:13.164 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:13.164 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:13.165 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:27:13.165 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:47.512788+0000", 2024-09-13T11:27:13.165 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.1840000000000003e-05 2024-09-13T11:27:13.165 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:13.165 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:13.165 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:27:13.165 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:47.512809+0000", 2024-09-13T11:27:13.165 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.1005e-05 2024-09-13T11:27:13.165 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:13.165 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:13.165 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:27:13.165 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:47.515182+0000", 2024-09-13T11:27:13.165 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0023731939999999999 2024-09-13T11:27:13.165 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:13.166 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:13.166 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:27:13.166 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:47.515254+0000", 2024-09-13T11:27:13.166 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.1861999999999997e-05 2024-09-13T11:27:13.166 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:27:13.166 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:27:13.166 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:27:13.166 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:13.166 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:13.166 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop(client.4665.0:138370 3.23 e140/90 3:c5730cd2:::benchmark_data_smithi177_31116_object138369:head v 140'4458, mlcod=140'4455)", 2024-09-13T11:27:13.166 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:24:54.416118+0000", 2024-09-13T11:27:13.166 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 138.71574479500001, 2024-09-13T11:27:13.166 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.244024991, 2024-09-13T11:27:13.166 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:27:13.167 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:27:13.167 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:27:13.167 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:27:13.167 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:13.167 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:27:13.167 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:54.416118+0000", 2024-09-13T11:27:13.167 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:27:13.167 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:13.167 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:13.167 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:27:13.167 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:54.416118+0000", 2024-09-13T11:27:13.167 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:27:13.167 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:13.167 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:13.168 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:27:13.168 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:54.416120+0000", 2024-09-13T11:27:13.168 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.6280000000000001e-06 2024-09-13T11:27:13.168 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:13.168 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:13.168 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:27:13.168 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:54.416161+0000", 2024-09-13T11:27:13.168 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.0877000000000002e-05 2024-09-13T11:27:13.168 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:13.168 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:13.168 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:27:13.168 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:54.657128+0000", 2024-09-13T11:27:13.168 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.24096696300000001 2024-09-13T11:27:13.169 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:13.169 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:13.169 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:27:13.169 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:54.657142+0000", 2024-09-13T11:27:13.169 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.358e-05 2024-09-13T11:27:13.169 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:13.169 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:13.169 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:27:13.169 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:54.657290+0000", 2024-09-13T11:27:13.169 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00014794400000000001 2024-09-13T11:27:13.169 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:13.169 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:13.169 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:27:13.170 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:54.657325+0000", 2024-09-13T11:27:13.170 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.5623000000000001e-05 2024-09-13T11:27:13.170 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:13.170 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:13.170 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:27:13.170 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:54.660079+0000", 2024-09-13T11:27:13.170 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0027536489999999999 2024-09-13T11:27:13.170 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:13.170 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:13.170 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:27:13.170 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:54.660143+0000", 2024-09-13T11:27:13.171 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.3726999999999995e-05 2024-09-13T11:27:13.171 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:27:13.171 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:27:13.171 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:27:13.171 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:13.171 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:13.171 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:209225 3.24 3:2478ed54:::benchmark_data_smithi177_31116_object68922:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e185)", 2024-09-13T11:27:13.171 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:25:53.632417+0000", 2024-09-13T11:27:13.171 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 79.499445362000003, 2024-09-13T11:27:13.171 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.219758709, 2024-09-13T11:27:13.171 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:27:13.171 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:27:13.171 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:27:13.171 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:27:13.171 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:27:13.172 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:27:13.172 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 209225 2024-09-13T11:27:13.172 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:13.172 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:27:13.172 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:13.172 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:27:13.172 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.632417+0000", 2024-09-13T11:27:13.172 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:27:13.172 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:13.172 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:13.172 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:27:13.172 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.632417+0000", 2024-09-13T11:27:13.172 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:27:13.172 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:13.173 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:13.173 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:27:13.173 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.632420+0000", 2024-09-13T11:27:13.173 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.4980000000000001e-06 2024-09-13T11:27:13.173 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:13.173 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:13.173 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:27:13.173 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.632427+0000", 2024-09-13T11:27:13.173 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.5120000000000002e-06 2024-09-13T11:27:13.173 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:13.173 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:13.173 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:27:13.173 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.632441+0000", 2024-09-13T11:27:13.174 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.4072e-05 2024-09-13T11:27:13.174 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:13.174 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:13.174 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:27:13.174 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.632454+0000", 2024-09-13T11:27:13.174 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.2886e-05 2024-09-13T11:27:13.174 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:13.174 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:13.174 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:27:13.174 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.632529+0000", 2024-09-13T11:27:13.174 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.5326000000000004e-05 2024-09-13T11:27:13.174 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:13.174 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:13.174 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:27:13.175 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.632642+0000", 2024-09-13T11:27:13.175 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00011249200000000001 2024-09-13T11:27:13.175 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:13.175 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:13.175 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1", 2024-09-13T11:27:13.175 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.632712+0000", 2024-09-13T11:27:13.175 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.0376000000000005e-05 2024-09-13T11:27:13.175 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:13.175 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:13.175 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:27:13.175 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.635270+0000", 2024-09-13T11:27:13.175 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0025574339999999999 2024-09-13T11:27:13.175 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:13.175 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:13.176 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:27:13.176 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.851915+0000", 2024-09-13T11:27:13.176 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.21664515400000001 2024-09-13T11:27:13.176 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:13.176 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:13.176 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:27:13.176 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.852065+0000", 2024-09-13T11:27:13.176 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00014972799999999999 2024-09-13T11:27:13.176 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:13.176 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:13.176 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:27:13.176 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.852176+0000", 2024-09-13T11:27:13.176 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00011123100000000001 2024-09-13T11:27:13.177 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:27:13.177 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:27:13.177 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:27:13.177 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:13.177 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:13.177 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:209229 3.19 3:9a976dcb:::benchmark_data_smithi177_31116_object68926:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e185)", 2024-09-13T11:27:13.177 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:25:53.633903+0000", 2024-09-13T11:27:13.177 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 79.497959456999993, 2024-09-13T11:27:13.177 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.25298130200000002, 2024-09-13T11:27:13.177 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:27:13.177 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:27:13.177 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:27:13.177 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:27:13.178 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:27:13.178 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:27:13.178 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 209229 2024-09-13T11:27:13.178 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:13.178 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:27:13.178 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:13.178 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:27:13.178 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.633903+0000", 2024-09-13T11:27:13.178 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:27:13.178 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:13.178 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:13.178 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:27:13.178 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.633903+0000", 2024-09-13T11:27:13.178 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:27:13.179 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:13.179 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:13.179 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:27:13.179 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.633905+0000", 2024-09-13T11:27:13.179 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.1600000000000001e-06 2024-09-13T11:27:13.179 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:13.179 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:13.179 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:27:13.179 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.633917+0000", 2024-09-13T11:27:13.179 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.1881e-05 2024-09-13T11:27:13.179 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:13.179 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:13.179 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:27:13.179 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.633931+0000", 2024-09-13T11:27:13.180 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.432e-05 2024-09-13T11:27:13.180 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:13.180 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:13.180 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:27:13.180 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.633941+0000", 2024-09-13T11:27:13.180 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.8079999999999999e-06 2024-09-13T11:27:13.180 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:13.180 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:13.180 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:27:13.180 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.634009+0000", 2024-09-13T11:27:13.180 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.7895e-05 2024-09-13T11:27:13.180 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:13.180 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:13.180 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:27:13.181 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.634137+0000", 2024-09-13T11:27:13.181 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000127644 2024-09-13T11:27:13.181 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:13.181 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:13.181 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1", 2024-09-13T11:27:13.181 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.634245+0000", 2024-09-13T11:27:13.181 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00010776399999999999 2024-09-13T11:27:13.181 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:13.181 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:13.181 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:27:13.181 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.637244+0000", 2024-09-13T11:27:13.181 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0029993149999999998 2024-09-13T11:27:13.181 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:13.181 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:13.182 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:27:13.182 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.886675+0000", 2024-09-13T11:27:13.182 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.249431456 2024-09-13T11:27:13.182 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:13.182 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:13.182 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:27:13.182 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.886785+0000", 2024-09-13T11:27:13.182 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000109326 2024-09-13T11:27:13.182 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:13.182 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:13.182 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:27:13.182 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.886884+0000", 2024-09-13T11:27:13.182 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.9733000000000002e-05 2024-09-13T11:27:13.182 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:27:13.182 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:27:13.183 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:27:13.183 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:13.183 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:13.183 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:209252 3.24 3:25ab37ee:::benchmark_data_smithi177_31116_object68949:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e185)", 2024-09-13T11:27:13.183 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:25:53.647942+0000", 2024-09-13T11:27:13.183 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 79.483920385000005, 2024-09-13T11:27:13.183 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.21571831799999999, 2024-09-13T11:27:13.183 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:27:13.183 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:27:13.183 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:27:13.183 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:27:13.183 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:27:13.183 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:27:13.183 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 209252 2024-09-13T11:27:13.184 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:13.184 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:27:13.184 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:13.184 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:27:13.184 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.647942+0000", 2024-09-13T11:27:13.184 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:27:13.184 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:13.184 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:13.184 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:27:13.184 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.647942+0000", 2024-09-13T11:27:13.184 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:27:13.184 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:13.184 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:13.184 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:27:13.185 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.647945+0000", 2024-09-13T11:27:13.185 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.717e-06 2024-09-13T11:27:13.185 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:13.185 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:13.185 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:27:13.185 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.647951+0000", 2024-09-13T11:27:13.185 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.353e-06 2024-09-13T11:27:13.185 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:13.185 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:13.185 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:27:13.185 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.647966+0000", 2024-09-13T11:27:13.185 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.4877e-05 2024-09-13T11:27:13.185 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:13.185 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:13.186 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:27:13.186 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.647973+0000", 2024-09-13T11:27:13.186 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.2640000000000002e-06 2024-09-13T11:27:13.186 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:13.186 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:13.186 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:27:13.186 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.648038+0000", 2024-09-13T11:27:13.186 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.4366e-05 2024-09-13T11:27:13.186 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:13.186 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:13.186 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:27:13.186 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.648131+0000", 2024-09-13T11:27:13.186 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.3017999999999996e-05 2024-09-13T11:27:13.186 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:13.187 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:13.187 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1", 2024-09-13T11:27:13.187 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.648200+0000", 2024-09-13T11:27:13.187 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.9244000000000003e-05 2024-09-13T11:27:13.187 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:13.187 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:13.187 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:27:13.187 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.649817+0000", 2024-09-13T11:27:13.187 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0016166920000000001 2024-09-13T11:27:13.187 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:13.187 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:13.187 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:27:13.187 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.863329+0000", 2024-09-13T11:27:13.187 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.21351207699999999 2024-09-13T11:27:13.188 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:13.188 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:13.188 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:27:13.188 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.863524+0000", 2024-09-13T11:27:13.188 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00019551 2024-09-13T11:27:13.188 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:13.188 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:13.188 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:27:13.188 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.863660+0000", 2024-09-13T11:27:13.188 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0001362 2024-09-13T11:27:13.189 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:27:13.189 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:27:13.189 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:27:13.189 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:13.189 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:13.189 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:209264 3.19 3:9fb0ed8f:::benchmark_data_smithi177_31116_object68961:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e185)", 2024-09-13T11:27:13.189 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:25:53.655043+0000", 2024-09-13T11:27:13.190 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 79.476819472000003, 2024-09-13T11:27:13.190 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.232156005, 2024-09-13T11:27:13.190 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:27:13.190 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:27:13.190 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:27:13.190 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:27:13.190 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:27:13.190 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:27:13.190 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 209264 2024-09-13T11:27:13.190 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:13.191 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:27:13.191 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:13.191 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:27:13.191 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.655043+0000", 2024-09-13T11:27:13.191 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:27:13.191 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:13.191 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:13.191 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:27:13.191 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.655043+0000", 2024-09-13T11:27:13.191 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:27:13.191 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:13.192 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:13.192 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:27:13.192 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.655045+0000", 2024-09-13T11:27:13.192 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.7090000000000001e-06 2024-09-13T11:27:13.192 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:13.192 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:13.193 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:27:13.193 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.655051+0000", 2024-09-13T11:27:13.193 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.3239999999999999e-06 2024-09-13T11:27:13.193 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:13.193 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:13.193 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:27:13.193 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.655065+0000", 2024-09-13T11:27:13.193 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.4137e-05 2024-09-13T11:27:13.193 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:13.193 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:13.193 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:27:13.194 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.655072+0000", 2024-09-13T11:27:13.194 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.827e-06 2024-09-13T11:27:13.194 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:13.194 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:13.194 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:27:13.194 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.655133+0000", 2024-09-13T11:27:13.194 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.0912000000000002e-05 2024-09-13T11:27:13.194 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:13.194 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:13.194 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:27:13.194 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.655228+0000", 2024-09-13T11:27:13.195 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.5440999999999996e-05 2024-09-13T11:27:13.195 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:13.195 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:13.195 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1", 2024-09-13T11:27:13.195 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.655296+0000", 2024-09-13T11:27:13.195 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.7316000000000004e-05 2024-09-13T11:27:13.195 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:13.195 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:13.195 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:27:13.195 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.658618+0000", 2024-09-13T11:27:13.195 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0033226179999999998 2024-09-13T11:27:13.196 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:13.196 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:13.196 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:27:13.196 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.887043+0000", 2024-09-13T11:27:13.196 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.22842430699999999 2024-09-13T11:27:13.196 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:13.196 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:13.196 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:27:13.196 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.887132+0000", 2024-09-13T11:27:13.197 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.9635000000000005e-05 2024-09-13T11:27:13.197 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:13.197 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:13.197 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:27:13.197 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:53.887199+0000", 2024-09-13T11:27:13.197 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.6779000000000006e-05 2024-09-13T11:27:13.197 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:27:13.197 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:27:13.197 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:27:13.197 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:13.198 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:13.198 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:221512 3.24 3:25f85d9e:::benchmark_data_smithi177_31116_object81209:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e192)", 2024-09-13T11:27:13.198 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:26:05.060948+0000", 2024-09-13T11:27:13.198 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 68.070914904999995, 2024-09-13T11:27:13.198 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.21371807700000001, 2024-09-13T11:27:13.198 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:27:13.198 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:27:13.199 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:27:13.199 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:27:13.199 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:27:13.199 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:27:13.199 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 221512 2024-09-13T11:27:13.200 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:13.200 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:27:13.200 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:13.200 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:27:13.201 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:05.060948+0000", 2024-09-13T11:27:13.201 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:27:13.201 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:13.201 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:13.201 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:27:13.201 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:05.060948+0000", 2024-09-13T11:27:13.201 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:27:13.201 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:13.201 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:13.202 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:27:13.202 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:05.060950+0000", 2024-09-13T11:27:13.202 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.6469999999999998e-06 2024-09-13T11:27:13.202 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:13.203 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:13.203 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:27:13.203 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:05.060959+0000", 2024-09-13T11:27:13.203 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.2460000000000003e-06 2024-09-13T11:27:13.203 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:13.203 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:13.204 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:27:13.204 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:05.060977+0000", 2024-09-13T11:27:13.204 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.8779000000000001e-05 2024-09-13T11:27:13.204 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:13.204 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:13.205 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:27:13.205 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:05.060993+0000", 2024-09-13T11:27:13.205 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.5464999999999998e-05 2024-09-13T11:27:13.205 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:13.205 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:13.205 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:27:13.205 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:05.062155+0000", 2024-09-13T11:27:13.205 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.001162724 2024-09-13T11:27:13.205 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:13.205 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:13.205 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-09-13T11:27:13.206 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:05.062181+0000", 2024-09-13T11:27:13.206 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.5157e-05 2024-09-13T11:27:13.206 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:13.206 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:13.206 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:27:13.206 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:05.257730+0000", 2024-09-13T11:27:13.206 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.19554980299999999 2024-09-13T11:27:13.207 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:13.207 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:13.207 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for degraded object", 2024-09-13T11:27:13.207 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:05.258342+0000", 2024-09-13T11:27:13.207 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00061164900000000003 2024-09-13T11:27:13.207 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:13.207 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:13.208 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:27:13.208 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:05.266413+0000", 2024-09-13T11:27:13.208 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0080705780000000001 2024-09-13T11:27:13.208 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:13.208 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:13.208 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:27:13.208 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:05.266509+0000", 2024-09-13T11:27:13.208 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.6644999999999994e-05 2024-09-13T11:27:13.208 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:13.208 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:13.208 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 14", 2024-09-13T11:27:13.208 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:05.266646+0000", 2024-09-13T11:27:13.208 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00013685499999999999 2024-09-13T11:27:13.209 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:13.209 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:13.209 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:27:13.209 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:05.273976+0000", 2024-09-13T11:27:13.209 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0073293719999999998 2024-09-13T11:27:13.209 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:13.209 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:13.209 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:27:13.209 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:05.273999+0000", 2024-09-13T11:27:13.209 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.3252e-05 2024-09-13T11:27:13.209 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:13.209 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:13.209 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:27:13.209 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:05.274075+0000", 2024-09-13T11:27:13.210 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.6415999999999995e-05 2024-09-13T11:27:13.210 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:13.210 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:13.210 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:27:13.210 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:05.274666+0000", 2024-09-13T11:27:13.210 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00059048900000000003 2024-09-13T11:27:13.210 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:27:13.210 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:27:13.210 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:27:13.210 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:13.210 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:13.210 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "MOSDPGPushReply(3.24 192/191 [PushReplyOp(3:26a69d79:::benchmark_data_smithi177_31116_object86105:head)])", 2024-09-13T11:27:13.210 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:26:05.749387+0000", 2024-09-13T11:27:13.210 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 67.382475081999999, 2024-09-13T11:27:13.211 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.34518297100000001, 2024-09-13T11:27:13.211 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:27:13.211 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:27:13.211 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "reached pg", 2024-09-13T11:27:13.211 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:27:13.211 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:13.211 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:27:13.211 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:05.749387+0000", 2024-09-13T11:27:13.211 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:27:13.211 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:13.211 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:13.211 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:27:13.211 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:05.749387+0000", 2024-09-13T11:27:13.212 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:27:13.212 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:13.212 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:13.212 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:27:13.212 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:05.749391+0000", 2024-09-13T11:27:13.212 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.54e-06 2024-09-13T11:27:13.212 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:13.212 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:13.212 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:27:13.212 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:05.749403+0000", 2024-09-13T11:27:13.212 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.1573000000000001e-05 2024-09-13T11:27:13.212 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:13.212 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:13.213 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:27:13.213 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:06.092306+0000", 2024-09-13T11:27:13.213 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.34290318600000003 2024-09-13T11:27:13.213 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:13.213 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:13.213 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:27:13.213 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:06.092334+0000", 2024-09-13T11:27:13.213 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.7993000000000001e-05 2024-09-13T11:27:13.213 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:13.213 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:13.213 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:27:13.213 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:06.092512+0000", 2024-09-13T11:27:13.213 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000178658 2024-09-13T11:27:13.213 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:13.213 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:13.214 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:27:13.214 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:06.094570+0000", 2024-09-13T11:27:13.214 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0020580210000000002 2024-09-13T11:27:13.214 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:27:13.214 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:27:13.214 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:27:13.214 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:13.214 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:13.214 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:228454 3.24 3:25beae58:::benchmark_data_smithi177_31116_object88151:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e198)", 2024-09-13T11:27:13.214 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:26:12.032335+0000", 2024-09-13T11:27:13.214 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 61.099527543999997, 2024-09-13T11:27:13.214 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.206015589, 2024-09-13T11:27:13.214 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:27:13.214 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:27:13.215 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:27:13.215 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:27:13.215 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:27:13.215 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:27:13.215 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 228454 2024-09-13T11:27:13.215 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:13.215 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:27:13.215 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:13.215 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:27:13.215 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:12.032335+0000", 2024-09-13T11:27:13.215 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:27:13.215 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:13.215 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:13.215 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:27:13.216 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:12.032335+0000", 2024-09-13T11:27:13.216 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:27:13.216 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:13.216 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:13.216 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:27:13.216 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:12.032337+0000", 2024-09-13T11:27:13.216 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.2290000000000002e-06 2024-09-13T11:27:13.216 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:13.216 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:13.216 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:27:13.216 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:12.032342+0000", 2024-09-13T11:27:13.216 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.0200000000000002e-06 2024-09-13T11:27:13.216 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:13.216 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:13.217 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:27:13.217 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:12.032353+0000", 2024-09-13T11:27:13.217 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.1168e-05 2024-09-13T11:27:13.217 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:13.217 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:13.217 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:27:13.217 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:12.032360+0000", 2024-09-13T11:27:13.217 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.7209999999999997e-06 2024-09-13T11:27:13.217 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:13.217 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:13.217 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:27:13.217 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:12.032430+0000", 2024-09-13T11:27:13.217 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.9651000000000001e-05 2024-09-13T11:27:13.217 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:13.218 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:13.218 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:27:13.218 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:12.032600+0000", 2024-09-13T11:27:13.218 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00017053000000000001 2024-09-13T11:27:13.218 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:13.218 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:13.218 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 14", 2024-09-13T11:27:13.218 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:12.032732+0000", 2024-09-13T11:27:13.218 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00013139099999999999 2024-09-13T11:27:13.218 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:13.218 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:13.218 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:27:13.218 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:12.034663+0000", 2024-09-13T11:27:13.218 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.001931012 2024-09-13T11:27:13.218 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:13.219 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:13.219 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:27:13.219 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:12.237696+0000", 2024-09-13T11:27:13.219 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.203033767 2024-09-13T11:27:13.219 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:13.219 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:13.219 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:27:13.219 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:12.237753+0000", 2024-09-13T11:27:13.219 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.6146000000000001e-05 2024-09-13T11:27:13.219 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:13.219 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:13.219 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:27:13.219 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:12.238351+0000", 2024-09-13T11:27:13.219 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00059795400000000004 2024-09-13T11:27:13.220 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:27:13.220 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:27:13.220 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:27:13.220 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:13.220 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:13.220 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "MOSDPGPushReply(3.24 200/191 [PushReplyOp(3:243cc29e:::benchmark_data_smithi177_31116_object132309:head)])", 2024-09-13T11:27:13.220 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:26:16.643152+0000", 2024-09-13T11:27:13.220 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 56.488710118999997, 2024-09-13T11:27:13.220 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.222406362, 2024-09-13T11:27:13.220 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:27:13.220 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:27:13.220 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "reached pg", 2024-09-13T11:27:13.220 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:27:13.220 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:13.221 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:27:13.221 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:16.643152+0000", 2024-09-13T11:27:13.221 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:27:13.221 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:13.221 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:13.221 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:27:13.221 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:16.643152+0000", 2024-09-13T11:27:13.221 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:27:13.221 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:13.221 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:13.221 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:27:13.221 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:16.643154+0000", 2024-09-13T11:27:13.221 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.0080000000000001e-06 2024-09-13T11:27:13.223 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:13.223 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:13.224 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:27:13.224 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:16.643166+0000", 2024-09-13T11:27:13.224 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.1739999999999999e-05 2024-09-13T11:27:13.224 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:13.224 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:13.224 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:27:13.224 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:16.864486+0000", 2024-09-13T11:27:13.224 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.221319708 2024-09-13T11:27:13.224 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:13.224 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:13.224 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:27:13.224 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:16.864501+0000", 2024-09-13T11:27:13.224 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.5226e-05 2024-09-13T11:27:13.224 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:13.225 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:13.225 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:27:13.225 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:16.864629+0000", 2024-09-13T11:27:13.225 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00012821499999999999 2024-09-13T11:27:13.225 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:13.225 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:13.225 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:27:13.225 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:16.865559+0000", 2024-09-13T11:27:13.225 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00092946499999999998 2024-09-13T11:27:13.225 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:27:13.225 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:27:13.225 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:27:13.225 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:13.225 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:13.226 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop(client.4665.0:268442 3.2e e228/227 3:76ee2334:::benchmark_data_smithi177_31116_object128139:head v 228'10116, mlcod=132'5760)", 2024-09-13T11:27:13.226 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:26:54.329270+0000", 2024-09-13T11:27:13.226 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 18.80259251, 2024-09-13T11:27:13.226 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.24437125900000001, 2024-09-13T11:27:13.226 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:27:13.226 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:27:13.226 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:27:13.226 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:27:13.226 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:13.226 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:27:13.226 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:54.329270+0000", 2024-09-13T11:27:13.226 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:27:13.226 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:13.227 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:13.227 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:27:13.227 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:54.329270+0000", 2024-09-13T11:27:13.227 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:27:13.227 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:13.227 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:13.227 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:27:13.227 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:54.329271+0000", 2024-09-13T11:27:13.227 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.463e-06 2024-09-13T11:27:13.227 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:13.227 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:13.227 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:27:13.227 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:54.329278+0000", 2024-09-13T11:27:13.227 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.4699999999999999e-06 2024-09-13T11:27:13.227 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:13.228 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:13.228 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:27:13.228 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:54.570320+0000", 2024-09-13T11:27:13.228 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.241041911 2024-09-13T11:27:13.228 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:13.228 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:13.228 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:27:13.228 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:54.570328+0000", 2024-09-13T11:27:13.228 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.2870000000000004e-06 2024-09-13T11:27:13.228 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:13.228 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:13.228 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:27:13.228 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:54.570394+0000", 2024-09-13T11:27:13.228 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.5675999999999999e-05 2024-09-13T11:27:13.229 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:13.229 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:13.229 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:27:13.229 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:54.570423+0000", 2024-09-13T11:27:13.229 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.9005000000000001e-05 2024-09-13T11:27:13.229 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:13.229 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:13.229 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:27:13.229 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:54.573581+0000", 2024-09-13T11:27:13.229 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.003158054 2024-09-13T11:27:13.229 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:13.229 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:13.229 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:27:13.229 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:54.573641+0000", 2024-09-13T11:27:13.230 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.0393000000000003e-05 2024-09-13T11:27:13.230 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:27:13.230 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:27:13.230 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:27:13.230 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:13.230 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:13.230 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "MOSDPGRecoveryDelete(3.2e e233,227 3:77cadc7d:::benchmark_data_smithi177_31116_object457:head,144'6132)", 2024-09-13T11:27:13.230 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:27:00.978843+0000", 2024-09-13T11:27:13.230 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 12.153019367000001, 2024-09-13T11:27:13.230 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.29752768400000001, 2024-09-13T11:27:13.230 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:27:13.230 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:27:13.230 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-09-13T11:27:13.230 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:27:13.231 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:13.231 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:27:13.231 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:27:00.978843+0000", 2024-09-13T11:27:13.231 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:27:13.231 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:13.231 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:13.231 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:27:13.231 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:27:00.978843+0000", 2024-09-13T11:27:13.231 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:27:13.231 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:13.232 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:13.232 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:27:13.232 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:27:00.978845+0000", 2024-09-13T11:27:13.232 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.04e-06 2024-09-13T11:27:13.232 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:13.232 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:13.233 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:27:13.233 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:27:00.978853+0000", 2024-09-13T11:27:13.233 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.1640000000000001e-06 2024-09-13T11:27:13.233 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:13.233 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:13.233 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:27:13.233 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:27:01.276188+0000", 2024-09-13T11:27:13.233 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.29733416600000001 2024-09-13T11:27:13.233 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:13.233 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:13.233 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:27:13.233 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:27:01.276197+0000", 2024-09-13T11:27:13.233 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.2080000000000006e-06 2024-09-13T11:27:13.234 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:13.234 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:13.234 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:27:13.234 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:27:01.276281+0000", 2024-09-13T11:27:13.234 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.4159999999999994e-05 2024-09-13T11:27:13.234 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:13.234 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:13.234 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:27:13.234 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:27:01.276291+0000", 2024-09-13T11:27:13.234 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.9019999999999994e-06 2024-09-13T11:27:13.234 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:13.234 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:13.234 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:27:13.234 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:27:01.276371+0000", 2024-09-13T11:27:13.234 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.0043999999999995e-05 2024-09-13T11:27:13.235 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:27:13.235 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:27:13.235 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:27:13.235 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:13.235 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:13.235 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop_reply(client.4665.0:278037 3.24 e237/191 ondisk, result = 0)", 2024-09-13T11:27:13.235 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:27:05.098993+0000", 2024-09-13T11:27:13.235 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 8.0328691980000002, 2024-09-13T11:27:13.235 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.35564582700000003, 2024-09-13T11:27:13.235 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:27:13.235 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:27:13.235 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-09-13T11:27:13.235 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:27:13.235 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:13.236 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:27:13.236 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:27:05.098993+0000", 2024-09-13T11:27:13.236 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:27:13.236 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:13.236 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:13.236 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:27:13.236 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:27:05.098993+0000", 2024-09-13T11:27:13.236 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:27:13.236 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:13.236 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:13.236 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:27:13.236 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:27:05.098994+0000", 2024-09-13T11:27:13.236 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.128e-06 2024-09-13T11:27:13.237 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:13.237 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:13.237 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:27:13.237 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:27:05.099001+0000", 2024-09-13T11:27:13.237 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.9460000000000001e-06 2024-09-13T11:27:13.237 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:13.237 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:13.237 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:27:13.237 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:27:05.454282+0000", 2024-09-13T11:27:13.237 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.35528074599999998 2024-09-13T11:27:13.237 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:13.237 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:13.237 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:27:13.237 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:27:05.454303+0000", 2024-09-13T11:27:13.238 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.0735999999999999e-05 2024-09-13T11:27:13.238 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:13.238 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:13.238 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:27:13.238 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:27:05.454436+0000", 2024-09-13T11:27:13.238 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00013330300000000001 2024-09-13T11:27:13.238 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:13.238 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:13.238 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:27:13.238 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:27:05.454451+0000", 2024-09-13T11:27:13.238 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.5176e-05 2024-09-13T11:27:13.238 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:13.238 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:13.238 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:27:13.239 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:27:05.454639+0000", 2024-09-13T11:27:13.239 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00018779199999999999 2024-09-13T11:27:13.239 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:27:13.239 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:27:13.239 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:27:13.239 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:13.239 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:13.239 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "MOSDPGRecoveryDelete(3.2e e237,227 3:743b8fc3:::benchmark_data_smithi177_31116_object18332:head,153'6743)", 2024-09-13T11:27:13.239 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:27:05.313002+0000", 2024-09-13T11:27:13.239 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 7.818860248, 2024-09-13T11:27:13.239 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.68388622799999998, 2024-09-13T11:27:13.239 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:27:13.239 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:27:13.239 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-09-13T11:27:13.239 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:27:13.240 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:13.240 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:27:13.240 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:27:05.313002+0000", 2024-09-13T11:27:13.240 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:27:13.240 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:13.240 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:13.240 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:27:13.240 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:27:05.313002+0000", 2024-09-13T11:27:13.240 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:27:13.240 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:13.240 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:13.240 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:27:13.240 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:27:05.313005+0000", 2024-09-13T11:27:13.241 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.9859999999999999e-06 2024-09-13T11:27:13.241 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:13.241 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:13.241 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:27:13.241 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:27:05.313016+0000", 2024-09-13T11:27:13.241 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0934000000000001e-05 2024-09-13T11:27:13.241 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:13.241 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:13.241 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:27:13.241 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:27:05.996141+0000", 2024-09-13T11:27:13.241 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.683125027 2024-09-13T11:27:13.242 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:13.242 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:13.242 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:27:13.242 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:27:05.996170+0000", 2024-09-13T11:27:13.242 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.8710000000000001e-05 2024-09-13T11:27:13.242 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:13.242 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:13.242 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:27:13.242 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:27:05.996696+0000", 2024-09-13T11:27:13.242 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00052609599999999999 2024-09-13T11:27:13.242 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:13.242 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:13.243 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:27:13.243 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:27:05.996723+0000", 2024-09-13T11:27:13.243 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.6933000000000001e-05 2024-09-13T11:27:13.243 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:13.243 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:13.243 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:27:13.243 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:27:05.996889+0000", 2024-09-13T11:27:13.243 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00016554199999999999 2024-09-13T11:27:13.243 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:27:13.243 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:27:13.244 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:27:13.244 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:27:13.244 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:27:13.244 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-09-13T11:27:13.245 INFO:teuthology.orchestra.run.smithi001.stdout:osd.2: {} 2024-09-13T11:27:13.246 INFO:teuthology.orchestra.run.smithi001.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T11:27:13.247 INFO:teuthology.orchestra.run.smithi001.stdout:osd.3: {} 2024-09-13T11:27:13.248 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_ops_in_flight 2024-09-13T11:27:13.248 INFO:teuthology.orchestra.run.smithi001.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T11:27:13.248 INFO:teuthology.orchestra.run.smithi001.stdout:osd.4: {} 2024-09-13T11:27:13.249 INFO:teuthology.orchestra.run.smithi001.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T11:27:13.249 INFO:teuthology.orchestra.run.smithi001.stdout:osd.5: {} 2024-09-13T11:27:13.249 INFO:teuthology.orchestra.run.smithi001.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T11:27:13.250 INFO:tasks.ceph.osd.8.smithi001.stderr:2024-09-13T11:27:13.242+0000 7fb9b48ae640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26312) UID: 0 2024-09-13T11:27:13.281 INFO:teuthology.orchestra.run.smithi001.stdout:osd.6: {} 2024-09-13T11:27:13.281 INFO:teuthology.orchestra.run.smithi001.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T11:27:13.322 INFO:teuthology.orchestra.run.smithi001.stdout:osd.7: {} 2024-09-13T11:27:13.323 INFO:teuthology.orchestra.run.smithi001.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T11:27:13.346 INFO:tasks.ceph.osd.4.smithi001.stderr:2024-09-13T11:27:13.342+0000 7ff8e0218640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26310) UID: 0 2024-09-13T11:27:13.359 INFO:teuthology.orchestra.run.smithi001.stdout:osd.8: {} 2024-09-13T11:27:13.359 INFO:teuthology.orchestra.run.smithi001.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-13T11:27:13.392 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-09-13T11:27:13.392 INFO:teuthology.orchestra.run.smithi092.stdout: "ops": [], 2024-09-13T11:27:13.392 INFO:teuthology.orchestra.run.smithi092.stdout: "num_ops": 0 2024-09-13T11:27:13.392 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-09-13T11:27:13.394 INFO:teuthology.orchestra.run.smithi001.stdout:osd.9: {} 2024-09-13T11:27:13.394 INFO:teuthology.orchestra.run.smithi001.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-13T11:27:13.394 INFO:teuthology.orchestra.run.smithi001.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-09-13T11:27:13.411 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_blocked_ops 2024-09-13T11:27:13.421 INFO:teuthology.orchestra.run.smithi001.stdout:osd.11: {} 2024-09-13T11:27:13.422 INFO:teuthology.orchestra.run.smithi001.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-13T11:27:13.447 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-13T11:27:13.446+0000 7f6a68775640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26307) UID: 0 2024-09-13T11:27:13.449 INFO:teuthology.orchestra.run.smithi001.stdout:osd.12: {} 2024-09-13T11:27:13.449 INFO:teuthology.orchestra.run.smithi001.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-13T11:27:13.486 INFO:teuthology.orchestra.run.smithi001.stdout:osd.13: {} 2024-09-13T11:27:13.486 INFO:teuthology.orchestra.run.smithi001.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-13T11:27:13.487 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-13T11:27:13.530 INFO:teuthology.orchestra.run.smithi001.stdout:osd.14: {} 2024-09-13T11:27:13.531 INFO:teuthology.orchestra.run.smithi001.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-13T11:27:13.548 INFO:teuthology.orchestra.run.smithi092.stdout:ERROR: (22) Invalid argument 2024-09-13T11:27:13.548 INFO:teuthology.orchestra.run.smithi092.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:27:13.548 INFO:tasks.ceph.osd.3.smithi177.stderr:2024-09-13T11:27:13.545+0000 7f10adba4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26574) UID: 0 2024-09-13T11:27:13.556 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-09-13T11:27:13.554+0000 7fb2a5596640 -1 mon.a@0(leader).osd e242 definitely_dead 0 2024-09-13T11:27:13.565 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_historic_ops 2024-09-13T11:27:13.566 INFO:teuthology.orchestra.run.smithi001.stdout:osd.15: {} 2024-09-13T11:27:13.567 INFO:teuthology.orchestra.run.smithi001.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-13T11:27:13.570 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-13T11:27:13.571 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-13T11:27:13.649 INFO:tasks.ceph.osd.7.smithi177.stderr:2024-09-13T11:27:13.645+0000 7f897b51d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26575) UID: 0 2024-09-13T11:27:13.705 INFO:teuthology.orchestra.run.smithi092.stdout:ERROR: (22) Invalid argument 2024-09-13T11:27:13.706 INFO:teuthology.orchestra.run.smithi092.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:27:13.723 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-13T11:27:13.768 INFO:tasks.ceph.osd.14.smithi162.stderr:2024-09-13T11:27:13.767+0000 7f3df5903640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33154) UID: 0 2024-09-13T11:27:13.850 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-13T11:27:13.846+0000 7f6a68775640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26307) UID: 0 2024-09-13T11:27:13.876 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-13T11:27:13.877 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:27:13.897 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-13T11:27:13.952 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-13T11:27:13.950+0000 7f6a68775640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26307) UID: 0 2024-09-13T11:27:14.053 INFO:tasks.ceph.osd.15.smithi177.stderr:2024-09-13T11:27:14.049+0000 7f9d6ce77640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26578) UID: 0 2024-09-13T11:27:14.080 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-13T11:27:14.081 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:27:14.097 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-13T11:27:14.154 INFO:tasks.ceph.osd.1.smithi092.stderr:2024-09-13T11:27:14.151+0000 7fe7789c5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33718) UID: 0 2024-09-13T11:27:14.230 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-13T11:27:14.230 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:27:14.246 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-13T11:27:14.255 INFO:tasks.ceph.osd.13.smithi092.stderr:2024-09-13T11:27:14.251+0000 7f3e5afea640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26350) UID: 0 2024-09-13T11:27:14.356 INFO:tasks.ceph.osd.13.smithi092.stderr:2024-09-13T11:27:14.351+0000 7f3e5afea640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26350) UID: 0 2024-09-13T11:27:14.379 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-13T11:27:14.379 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:27:14.395 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-13T11:27:14.457 INFO:tasks.ceph.osd.1.smithi092.stderr:2024-09-13T11:27:14.455+0000 7fe7789c5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33718) UID: 0 2024-09-13T11:27:14.490 INFO:teuthology.orchestra.run.smithi001.stderr:marked out osd.10. 2024-09-13T11:27:14.491 INFO:teuthology.orchestra.run.smithi001.stderr:noscrub is set 2024-09-13T11:27:14.505 INFO:tasks.ceph.osd.4.smithi001.stderr:2024-09-13T11:27:14.502+0000 7ff8d0c3d640 -1 osd.4 242 handle_osd_map injecting map crc failure 2024-09-13T11:27:14.511 INFO:tasks.ceph.osd.11.smithi177.stderr:2024-09-13T11:27:14.505+0000 7f2469cec640 -1 osd.11 242 handle_osd_map injecting map crc failure 2024-09-13T11:27:14.527 INFO:tasks.thrashosds.thrasher:Testing ceph-objectstore-tool on down osd.10 2024-09-13T11:27:14.528 INFO:tasks.thrashosds.thrasher:remote for osd 10 is ubuntu@smithi162.front.sepia.ceph.com 2024-09-13T11:27:14.528 DEBUG:teuthology.orchestra.run.smithi162:> type ceph-objectstore-tool 2024-09-13T11:27:14.532 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-13T11:27:14.532 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:27:14.533 INFO:teuthology.orchestra.run.smithi162.stdout:ceph-objectstore-tool is /usr/bin/ceph-objectstore-tool 2024-09-13T11:27:14.533 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-10 --journal-path /var/lib/ceph/osd/ceph-10/journal --op list-pgs 2024-09-13T11:27:14.553 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-13T11:27:14.558 INFO:tasks.ceph.osd.5.smithi092.stderr:2024-09-13T11:27:14.555+0000 7f6f0b705640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26346) UID: 0 2024-09-13T11:27:14.659 INFO:tasks.ceph.osd.15.smithi177.stderr:2024-09-13T11:27:14.657+0000 7f9d6ce77640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26578) UID: 0 2024-09-13T11:27:14.730 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-13T11:27:14.731 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:27:14.749 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-13T11:27:14.760 INFO:tasks.ceph.osd.13.smithi092.stderr:2024-09-13T11:27:14.755+0000 7f3e5afea640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26350) UID: 0 2024-09-13T11:27:14.861 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-13T11:27:14.858+0000 7f6a68775640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26307) UID: 0 2024-09-13T11:27:14.914 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-09-13T11:27:14.914 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:27:14.937 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-13T11:27:14.961 INFO:tasks.ceph.osd.4.smithi001.stderr:2024-09-13T11:27:14.958+0000 7ff8e0218640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26310) UID: 0 2024-09-13T11:27:15.065 INFO:tasks.ceph.osd.14.smithi162.stderr:2024-09-13T11:27:15.063+0000 7f3df5903640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33154) UID: 0 2024-09-13T11:27:15.097 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-09-13T11:27:15.097 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:27:15.121 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-13T11:27:15.166 INFO:tasks.ceph.osd.4.smithi001.stderr:2024-09-13T11:27:15.162+0000 7ff8e0218640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26310) UID: 0 2024-09-13T11:27:15.268 INFO:tasks.ceph.osd.2.smithi162.stderr:2024-09-13T11:27:15.267+0000 7fe92b35a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26313) UID: 0 2024-09-13T11:27:15.283 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-09-13T11:27:15.284 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:27:15.306 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-13T11:27:15.369 INFO:tasks.ceph.osd.9.smithi092.stderr:2024-09-13T11:27:15.367+0000 7ff5ab3d0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26348) UID: 0 2024-09-13T11:27:15.458 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-09-13T11:27:15.458 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:27:15.471 INFO:tasks.ceph.osd.5.smithi092.stderr:2024-09-13T11:27:15.467+0000 7f6f0b705640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26346) UID: 0 2024-09-13T11:27:15.478 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-13T11:27:15.495 INFO:tasks.ceph.osd.5.smithi092.stderr:2024-09-13T11:27:15.491+0000 7f6f02136640 -1 osd.5 243 handle_osd_map injecting map crc failure 2024-09-13T11:27:15.497 INFO:tasks.ceph.osd.2.smithi162.stderr:2024-09-13T11:27:15.495+0000 7fe921d8b640 -1 osd.2 243 handle_osd_map injecting map crc failure 2024-09-13T11:27:15.498 INFO:tasks.ceph.osd.3.smithi177.stderr:2024-09-13T11:27:15.493+0000 7f10a25d1640 -1 osd.3 243 handle_osd_map injecting map crc failure 2024-09-13T11:27:15.498 INFO:tasks.ceph.osd.14.smithi162.stderr:2024-09-13T11:27:15.495+0000 7f3dea330640 -1 osd.14 243 handle_osd_map injecting map crc failure 2024-09-13T11:27:15.499 INFO:tasks.ceph.osd.8.smithi001.stderr:2024-09-13T11:27:15.494+0000 7fb9a92db640 -1 osd.8 243 handle_osd_map injecting map crc failure 2024-09-13T11:27:15.504 INFO:tasks.ceph.osd.12.smithi001.stderr:2024-09-13T11:27:15.502+0000 7f891d10b640 -1 osd.12 243 handle_osd_map injecting map crc failure 2024-09-13T11:27:15.571 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T11:27:15.572 INFO:tasks.ceph.osd.14.smithi162.stderr:2024-09-13T11:27:15.567+0000 7f3df5903640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33154) UID: 0 2024-09-13T11:27:15.647 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-09-13T11:27:15.648 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:27:15.667 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-13T11:27:15.672 INFO:tasks.ceph.osd.4.smithi001.stderr:2024-09-13T11:27:15.670+0000 7ff8e0218640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26310) UID: 0 2024-09-13T11:27:15.774 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-09-13T11:27:15.771+0000 7f97befc9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26315) UID: 0 2024-09-13T11:27:15.822 INFO:teuthology.orchestra.run.smithi162.stdout:3.3f 2024-09-13T11:27:15.822 INFO:teuthology.orchestra.run.smithi162.stdout:2.7 2024-09-13T11:27:15.822 INFO:teuthology.orchestra.run.smithi162.stdout:3.2a 2024-09-13T11:27:15.822 INFO:teuthology.orchestra.run.smithi162.stdout:3.17 2024-09-13T11:27:15.822 INFO:teuthology.orchestra.run.smithi162.stdout:3.3 2024-09-13T11:27:15.823 INFO:teuthology.orchestra.run.smithi162.stdout:3.32 2024-09-13T11:27:15.823 INFO:teuthology.orchestra.run.smithi162.stdout:3.1f 2024-09-13T11:27:15.823 INFO:teuthology.orchestra.run.smithi162.stdout:3.3d 2024-09-13T11:27:15.823 INFO:teuthology.orchestra.run.smithi162.stdout:3.2c 2024-09-13T11:27:15.823 INFO:teuthology.orchestra.run.smithi162.stdout:3.34 2024-09-13T11:27:15.823 INFO:teuthology.orchestra.run.smithi162.stdout:3.37 2024-09-13T11:27:15.823 INFO:teuthology.orchestra.run.smithi162.stdout:3.3a 2024-09-13T11:27:15.823 INFO:teuthology.orchestra.run.smithi162.stdout:3.18 2024-09-13T11:27:15.823 INFO:teuthology.orchestra.run.smithi162.stdout:3.38 2024-09-13T11:27:15.824 INFO:teuthology.orchestra.run.smithi001.stdout:osd.0: {} 2024-09-13T11:27:15.824 INFO:teuthology.orchestra.run.smithi001.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T11:27:15.851 INFO:teuthology.orchestra.run.smithi001.stdout:osd.1: {} 2024-09-13T11:27:15.851 INFO:teuthology.orchestra.run.smithi001.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T11:27:15.870 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-09-13T11:27:15.870 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:27:15.875 INFO:tasks.ceph.osd.15.smithi177.stderr:2024-09-13T11:27:15.873+0000 7f9d6ce77640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26578) UID: 0 2024-09-13T11:27:15.895 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-09-13T11:27:15.900 INFO:teuthology.orchestra.run.smithi001.stdout:osd.2: {} 2024-09-13T11:27:15.901 INFO:teuthology.orchestra.run.smithi001.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T11:27:15.930 INFO:teuthology.orchestra.run.smithi001.stdout:osd.3: {} 2024-09-13T11:27:15.930 INFO:teuthology.orchestra.run.smithi001.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T11:27:15.960 INFO:teuthology.orchestra.run.smithi001.stdout:osd.4: {} 2024-09-13T11:27:15.960 INFO:teuthology.orchestra.run.smithi001.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T11:27:15.976 INFO:tasks.ceph.osd.8.smithi001.stderr:2024-09-13T11:27:15.974+0000 7fb9b48ae640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26312) UID: 0 2024-09-13T11:27:15.988 INFO:teuthology.orchestra.run.smithi001.stdout:osd.5: {} 2024-09-13T11:27:15.988 INFO:teuthology.orchestra.run.smithi001.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T11:27:16.019 INFO:teuthology.orchestra.run.smithi001.stdout:osd.6: {} 2024-09-13T11:27:16.019 INFO:teuthology.orchestra.run.smithi001.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T11:27:16.052 INFO:teuthology.orchestra.run.smithi001.stdout:osd.7: {} 2024-09-13T11:27:16.052 INFO:teuthology.orchestra.run.smithi001.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T11:27:16.078 INFO:tasks.ceph.osd.15.smithi177.stderr:2024-09-13T11:27:16.077+0000 7f9d6ce77640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26578) UID: 0 2024-09-13T11:27:16.083 INFO:teuthology.orchestra.run.smithi001.stdout:osd.8: {} 2024-09-13T11:27:16.084 INFO:teuthology.orchestra.run.smithi001.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-13T11:27:16.089 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-09-13T11:27:16.089 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:27:16.108 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-09-13T11:27:16.114 INFO:teuthology.orchestra.run.smithi001.stdout:osd.9: {} 2024-09-13T11:27:16.114 INFO:teuthology.orchestra.run.smithi001.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-13T11:27:16.115 INFO:teuthology.orchestra.run.smithi001.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-09-13T11:27:16.125 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-10 --journal-path /var/lib/ceph/osd/ceph-10/journal --op export --pgid 3.32 --file /var/log/ceph/exp.3.32.10 2024-09-13T11:27:16.143 INFO:teuthology.orchestra.run.smithi001.stdout:osd.11: {} 2024-09-13T11:27:16.144 INFO:teuthology.orchestra.run.smithi001.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-13T11:27:16.172 INFO:teuthology.orchestra.run.smithi001.stdout:osd.12: {} 2024-09-13T11:27:16.172 INFO:teuthology.orchestra.run.smithi001.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-13T11:27:16.179 INFO:tasks.ceph.osd.1.smithi092.stderr:2024-09-13T11:27:16.175+0000 7fe7789c5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33718) UID: 0 2024-09-13T11:27:16.205 INFO:teuthology.orchestra.run.smithi001.stdout:osd.13: {} 2024-09-13T11:27:16.205 INFO:teuthology.orchestra.run.smithi001.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-13T11:27:16.236 INFO:teuthology.orchestra.run.smithi001.stdout:osd.14: {} 2024-09-13T11:27:16.236 INFO:teuthology.orchestra.run.smithi001.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-13T11:27:16.261 INFO:teuthology.orchestra.run.smithi001.stdout:osd.15: {} 2024-09-13T11:27:16.261 INFO:teuthology.orchestra.run.smithi001.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-13T11:27:16.264 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-13T11:27:16.264 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-13T11:27:16.280 INFO:tasks.ceph.osd.14.smithi162.stderr:2024-09-13T11:27:16.275+0000 7f3df5903640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33154) UID: 0 2024-09-13T11:27:16.286 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-09-13T11:27:16.286 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [], 2024-09-13T11:27:16.286 INFO:teuthology.orchestra.run.smithi177.stdout: "complaint_time": 30, 2024-09-13T11:27:16.286 INFO:teuthology.orchestra.run.smithi177.stdout: "num_blocked_ops": 0 2024-09-13T11:27:16.286 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-09-13T11:27:16.304 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-09-13T11:27:16.381 INFO:tasks.ceph.osd.5.smithi092.stderr:2024-09-13T11:27:16.379+0000 7f6f0b705640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26346) UID: 0 2024-09-13T11:27:16.439 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-09-13T11:27:16.439 INFO:teuthology.orchestra.run.smithi177.stdout: "size": 20, 2024-09-13T11:27:16.439 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 600, 2024-09-13T11:27:16.439 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [ 2024-09-13T11:27:16.439 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:16.439 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:79 3.1 e32/30 3:841df5b5:::benchmark_data_smithi177_31116_object78:head v 32'6, mlcod=32'4)", 2024-09-13T11:27:16.439 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:22:27.057292+0000", 2024-09-13T11:27:16.439 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 289.38232603799997, 2024-09-13T11:27:16.440 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.89296786800000005, 2024-09-13T11:27:16.440 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:27:16.440 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:27:16.440 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:27:16.440 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:27:16.440 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:16.440 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:27:16.440 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.057292+0000", 2024-09-13T11:27:16.440 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:16.440 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:16.440 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:16.440 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:27:16.440 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.057292+0000", 2024-09-13T11:27:16.440 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:16.441 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:16.441 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:16.441 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:27:16.441 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.057293+0000", 2024-09-13T11:27:16.441 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.6109999999999999e-06 2024-09-13T11:27:16.441 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:16.441 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:16.441 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:27:16.441 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.057330+0000", 2024-09-13T11:27:16.441 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.6721999999999998e-05 2024-09-13T11:27:16.441 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:16.441 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:16.441 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:27:16.441 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.948197+0000", 2024-09-13T11:27:16.442 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.890867042 2024-09-13T11:27:16.442 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:16.442 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:16.442 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:27:16.442 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.948213+0000", 2024-09-13T11:27:16.442 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5826000000000001e-05 2024-09-13T11:27:16.442 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:16.442 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:16.442 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:27:16.442 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.948377+0000", 2024-09-13T11:27:16.442 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00016348600000000001 2024-09-13T11:27:16.442 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:16.442 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:16.442 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:27:16.442 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.948447+0000", 2024-09-13T11:27:16.443 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.0891999999999998e-05 2024-09-13T11:27:16.443 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:16.443 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:16.443 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:27:16.443 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.950150+0000", 2024-09-13T11:27:16.443 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001702329 2024-09-13T11:27:16.443 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:16.443 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:16.443 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:27:16.443 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.950260+0000", 2024-09-13T11:27:16.443 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00010996 2024-09-13T11:27:16.443 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:16.443 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:27:16.443 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:16.444 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:16.444 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:16.444 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:45783 3.16 3.17c10a76 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e69)", 2024-09-13T11:27:16.444 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:20.947774+0000", 2024-09-13T11:27:16.444 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 235.49184378699999, 2024-09-13T11:27:16.444 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.96682922400000004, 2024-09-13T11:27:16.444 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:27:16.444 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:27:16.444 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-13T11:27:16.444 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:27:16.444 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:27:16.444 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:27:16.444 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 45783 2024-09-13T11:27:16.444 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:16.445 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:27:16.445 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:16.445 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:27:16.445 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:20.947774+0000", 2024-09-13T11:27:16.445 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:16.445 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:16.445 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:16.445 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:27:16.445 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:20.947774+0000", 2024-09-13T11:27:16.445 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:16.445 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:16.445 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:16.445 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:27:16.446 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:20.947777+0000", 2024-09-13T11:27:16.446 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.57e-06 2024-09-13T11:27:16.446 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:16.446 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:16.446 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:27:16.446 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:20.947802+0000", 2024-09-13T11:27:16.446 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.5528999999999999e-05 2024-09-13T11:27:16.446 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:16.446 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:16.446 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:27:16.446 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:20.947813+0000", 2024-09-13T11:27:16.446 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0473e-05 2024-09-13T11:27:16.446 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:16.446 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:16.447 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:27:16.447 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:20.947821+0000", 2024-09-13T11:27:16.447 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.3440000000000001e-06 2024-09-13T11:27:16.447 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:16.447 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:16.447 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:27:16.447 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:20.947933+0000", 2024-09-13T11:27:16.447 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00011210200000000001 2024-09-13T11:27:16.447 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:16.447 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:16.447 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for peered", 2024-09-13T11:27:16.447 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:20.947942+0000", 2024-09-13T11:27:16.447 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.6610000000000006e-06 2024-09-13T11:27:16.447 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:16.448 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:16.448 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:27:16.448 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:21.914579+0000", 2024-09-13T11:27:16.448 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.96663672300000003 2024-09-13T11:27:16.448 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:16.448 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:16.448 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:27:16.448 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:21.914603+0000", 2024-09-13T11:27:16.448 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.4822000000000001e-05 2024-09-13T11:27:16.448 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:16.448 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:27:16.448 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:16.448 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:16.449 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:16.449 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:54625 3.1f 3:f812cdfb:::benchmark_data_smithi177_31116_object54624:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e75)", 2024-09-13T11:27:16.449 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:30.979327+0000", 2024-09-13T11:27:16.449 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 225.460290998, 2024-09-13T11:27:16.449 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.96163670899999998, 2024-09-13T11:27:16.449 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:27:16.449 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:27:16.449 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-13T11:27:16.449 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:27:16.449 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:27:16.449 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:27:16.449 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 54625 2024-09-13T11:27:16.449 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:16.450 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:27:16.450 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:16.450 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:27:16.450 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.979327+0000", 2024-09-13T11:27:16.450 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:16.450 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:16.450 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:16.450 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:27:16.450 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.979327+0000", 2024-09-13T11:27:16.450 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:16.450 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:16.450 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:16.450 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:27:16.450 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.979329+0000", 2024-09-13T11:27:16.450 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.686e-06 2024-09-13T11:27:16.451 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:16.451 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:16.451 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:27:16.451 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.979364+0000", 2024-09-13T11:27:16.451 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.5750000000000002e-05 2024-09-13T11:27:16.451 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:16.451 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:16.451 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:27:16.451 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.979408+0000", 2024-09-13T11:27:16.451 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.3795999999999999e-05 2024-09-13T11:27:16.451 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:16.451 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:16.451 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:27:16.451 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.979427+0000", 2024-09-13T11:27:16.451 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.9040999999999999e-05 2024-09-13T11:27:16.452 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:16.452 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:16.452 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:27:16.452 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.122291+0000", 2024-09-13T11:27:16.452 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.14286373599999999 2024-09-13T11:27:16.452 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:16.452 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:16.452 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for peered", 2024-09-13T11:27:16.452 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.122298+0000", 2024-09-13T11:27:16.452 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.3719999999999999e-06 2024-09-13T11:27:16.452 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:16.452 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:16.452 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:27:16.452 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.940341+0000", 2024-09-13T11:27:16.453 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.81804249699999998 2024-09-13T11:27:16.453 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:16.453 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:16.453 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:27:16.453 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.940964+0000", 2024-09-13T11:27:16.453 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00062283100000000001 2024-09-13T11:27:16.453 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:16.453 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:27:16.453 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:16.453 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:16.453 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:16.453 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:54649 3.1c 3:3a457a93:::benchmark_data_smithi177_31116_object54648:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e75)", 2024-09-13T11:27:16.453 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:30.979460+0000", 2024-09-13T11:27:16.453 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 225.46015780799999, 2024-09-13T11:27:16.454 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.95579424000000002, 2024-09-13T11:27:16.454 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:27:16.454 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:27:16.454 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-13T11:27:16.454 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:27:16.454 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:27:16.454 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:27:16.454 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 54649 2024-09-13T11:27:16.454 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:16.454 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:27:16.454 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:16.454 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:27:16.454 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.979460+0000", 2024-09-13T11:27:16.454 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:16.455 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:16.455 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:16.455 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:27:16.455 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.979460+0000", 2024-09-13T11:27:16.455 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:16.455 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:16.455 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:16.455 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:27:16.455 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.979462+0000", 2024-09-13T11:27:16.455 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5799999999999999e-06 2024-09-13T11:27:16.455 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:16.455 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:16.455 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:27:16.455 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.979524+0000", 2024-09-13T11:27:16.455 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.2521999999999995e-05 2024-09-13T11:27:16.456 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:16.456 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:16.456 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:27:16.456 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.979561+0000", 2024-09-13T11:27:16.456 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.6349e-05 2024-09-13T11:27:16.456 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:16.456 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:16.456 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:27:16.456 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.979571+0000", 2024-09-13T11:27:16.456 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0457e-05 2024-09-13T11:27:16.456 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:16.456 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:16.457 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:27:16.457 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.121029+0000", 2024-09-13T11:27:16.457 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.14145766200000001 2024-09-13T11:27:16.457 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:16.457 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:16.457 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for peered", 2024-09-13T11:27:16.457 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.121039+0000", 2024-09-13T11:27:16.457 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0084e-05 2024-09-13T11:27:16.457 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:16.457 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:16.457 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:27:16.457 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.935157+0000", 2024-09-13T11:27:16.457 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.81411807899999999 2024-09-13T11:27:16.457 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:16.457 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:16.458 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:27:16.458 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.935254+0000", 2024-09-13T11:27:16.458 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.7507e-05 2024-09-13T11:27:16.458 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:16.458 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:27:16.458 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:16.458 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:16.458 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:16.458 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:54654 3.1c 3:3f0bb31a:::benchmark_data_smithi177_31116_object54653:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e75)", 2024-09-13T11:27:16.458 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:30.979665+0000", 2024-09-13T11:27:16.458 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 225.45995264300001, 2024-09-13T11:27:16.458 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.95623908099999999, 2024-09-13T11:27:16.458 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:27:16.458 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:27:16.459 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-13T11:27:16.459 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:27:16.459 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:27:16.459 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:27:16.459 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 54654 2024-09-13T11:27:16.459 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:16.459 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:27:16.459 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:16.459 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:27:16.459 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.979665+0000", 2024-09-13T11:27:16.459 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:16.459 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:16.459 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:16.460 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:27:16.460 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.979665+0000", 2024-09-13T11:27:16.460 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:16.460 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:16.460 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:16.460 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:27:16.460 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.979667+0000", 2024-09-13T11:27:16.460 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3540000000000001e-06 2024-09-13T11:27:16.460 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:16.460 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:16.460 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:27:16.460 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.979703+0000", 2024-09-13T11:27:16.460 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.6758000000000003e-05 2024-09-13T11:27:16.460 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:16.461 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:16.461 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:27:16.461 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.979714+0000", 2024-09-13T11:27:16.461 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0804e-05 2024-09-13T11:27:16.461 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:16.461 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:16.461 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:27:16.461 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.979724+0000", 2024-09-13T11:27:16.461 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0040999999999999e-05 2024-09-13T11:27:16.461 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:16.461 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:16.461 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:27:16.461 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.121061+0000", 2024-09-13T11:27:16.461 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.141336713 2024-09-13T11:27:16.462 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:16.462 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:16.462 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for peered", 2024-09-13T11:27:16.462 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.121067+0000", 2024-09-13T11:27:16.462 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.4150000000000004e-06 2024-09-13T11:27:16.462 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:16.462 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:16.462 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:27:16.462 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.935776+0000", 2024-09-13T11:27:16.462 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.81470863100000002 2024-09-13T11:27:16.462 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:16.462 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:16.462 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:27:16.462 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.935904+0000", 2024-09-13T11:27:16.462 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00012836499999999999 2024-09-13T11:27:16.463 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:16.463 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:27:16.463 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:16.463 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:16.463 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:16.463 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPushReply(3.11 75/70 [PushReplyOp(3:8df10771:::benchmark_data_smithi177_31116_object24734:head)])", 2024-09-13T11:27:16.463 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:31.149798+0000", 2024-09-13T11:27:16.463 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 225.28981963300001, 2024-09-13T11:27:16.463 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.32402702500000002, 2024-09-13T11:27:16.463 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:27:16.463 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:27:16.463 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-13T11:27:16.463 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:27:16.463 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:16.464 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:27:16.464 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.149798+0000", 2024-09-13T11:27:16.464 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:16.464 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:16.464 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:16.464 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:27:16.464 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.149798+0000", 2024-09-13T11:27:16.464 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:16.464 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:16.464 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:16.464 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:27:16.464 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.149799+0000", 2024-09-13T11:27:16.464 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.5700000000000001e-07 2024-09-13T11:27:16.464 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:16.464 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:16.465 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:27:16.465 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.149803+0000", 2024-09-13T11:27:16.465 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.0040000000000001e-06 2024-09-13T11:27:16.465 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:16.465 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:16.465 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:27:16.465 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.472172+0000", 2024-09-13T11:27:16.465 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.32236856699999999 2024-09-13T11:27:16.465 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:16.465 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:16.465 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:27:16.466 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.472187+0000", 2024-09-13T11:27:16.466 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5016e-05 2024-09-13T11:27:16.466 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:16.466 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:16.466 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:27:16.466 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.472302+0000", 2024-09-13T11:27:16.466 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000115121 2024-09-13T11:27:16.466 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:16.466 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:16.466 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:27:16.466 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.473825+0000", 2024-09-13T11:27:16.466 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0015234599999999999 2024-09-13T11:27:16.467 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:16.467 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:27:16.467 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:16.467 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:16.467 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:16.467 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:54668 3.1c 3:3d2c5e40:::benchmark_data_smithi177_31116_object54667:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e75)", 2024-09-13T11:27:16.467 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:31.245525+0000", 2024-09-13T11:27:16.467 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 225.19409337600001, 2024-09-13T11:27:16.467 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.69828973000000005, 2024-09-13T11:27:16.467 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:27:16.467 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:27:16.467 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:27:16.467 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:27:16.468 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:27:16.468 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:27:16.468 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 54668 2024-09-13T11:27:16.468 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:16.468 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:27:16.468 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:16.468 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:27:16.468 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.245525+0000", 2024-09-13T11:27:16.468 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:16.468 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:16.468 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:16.468 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:27:16.468 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.245525+0000", 2024-09-13T11:27:16.469 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:16.469 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:16.469 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:16.469 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:27:16.469 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.245526+0000", 2024-09-13T11:27:16.469 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.719e-06 2024-09-13T11:27:16.469 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:16.469 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:16.469 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:27:16.469 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.245547+0000", 2024-09-13T11:27:16.469 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.1084999999999999e-05 2024-09-13T11:27:16.469 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:16.469 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:16.469 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:27:16.469 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.245558+0000", 2024-09-13T11:27:16.470 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0934000000000001e-05 2024-09-13T11:27:16.470 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:16.470 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:16.470 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:27:16.470 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.245564+0000", 2024-09-13T11:27:16.470 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.6269999999999997e-06 2024-09-13T11:27:16.470 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:16.470 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:16.470 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:27:16.470 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.245620+0000", 2024-09-13T11:27:16.470 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.6289999999999998e-05 2024-09-13T11:27:16.470 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:16.470 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:16.470 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for peered", 2024-09-13T11:27:16.471 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.245632+0000", 2024-09-13T11:27:16.471 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.1748e-05 2024-09-13T11:27:16.471 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:16.471 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:16.471 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:27:16.471 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.935976+0000", 2024-09-13T11:27:16.471 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.69034455299999997 2024-09-13T11:27:16.471 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:16.471 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:16.471 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:27:16.471 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.936302+0000", 2024-09-13T11:27:16.471 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000325397 2024-09-13T11:27:16.471 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:16.471 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:16.471 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 13", 2024-09-13T11:27:16.472 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.936459+0000", 2024-09-13T11:27:16.472 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00015754300000000001 2024-09-13T11:27:16.472 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:16.472 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:16.472 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:27:16.472 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.938813+0000", 2024-09-13T11:27:16.472 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0023530999999999999 2024-09-13T11:27:16.472 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:16.472 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:16.472 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:27:16.472 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.943695+0000", 2024-09-13T11:27:16.472 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.004882506 2024-09-13T11:27:16.472 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:16.472 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:16.473 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:27:16.473 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.943767+0000", 2024-09-13T11:27:16.473 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.2454999999999997e-05 2024-09-13T11:27:16.473 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:16.473 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:16.473 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:27:16.473 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.943814+0000", 2024-09-13T11:27:16.473 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.6773000000000001e-05 2024-09-13T11:27:16.473 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:16.473 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:27:16.473 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:16.473 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:16.473 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:16.473 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPush(3.16 84/70 [PushOp(3:6f7be541:::benchmark_data_smithi177_31116_object43332:head, version: 66'2788, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:6f7be541:::benchmark_data_smithi177_31116_object43332:head@66'2788, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:27:16.474 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:41.207515+0000", 2024-09-13T11:27:16.474 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 215.23210260499999, 2024-09-13T11:27:16.474 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.52643771500000003, 2024-09-13T11:27:16.474 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:27:16.474 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:27:16.474 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:27:16.474 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:27:16.474 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:16.474 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:27:16.474 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.207515+0000", 2024-09-13T11:27:16.474 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:16.474 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:16.474 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:16.474 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:27:16.475 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.207515+0000", 2024-09-13T11:27:16.475 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:16.475 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:16.475 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:16.475 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:27:16.475 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.207516+0000", 2024-09-13T11:27:16.475 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.096e-06 2024-09-13T11:27:16.475 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:16.475 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:16.475 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:27:16.475 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.207717+0000", 2024-09-13T11:27:16.475 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000200199 2024-09-13T11:27:16.475 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:16.475 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:16.476 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:27:16.476 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.732785+0000", 2024-09-13T11:27:16.476 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.52506869599999995 2024-09-13T11:27:16.476 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:16.476 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:16.476 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:27:16.476 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.732801+0000", 2024-09-13T11:27:16.476 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5305999999999999e-05 2024-09-13T11:27:16.476 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:16.476 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:16.476 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:27:16.476 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.732889+0000", 2024-09-13T11:27:16.476 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.8157000000000003e-05 2024-09-13T11:27:16.476 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:16.477 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:16.477 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:27:16.477 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.732901+0000", 2024-09-13T11:27:16.477 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.1866e-05 2024-09-13T11:27:16.480 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:16.481 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:16.481 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:27:16.481 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.733953+0000", 2024-09-13T11:27:16.481 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001052395 2024-09-13T11:27:16.481 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:16.481 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:27:16.481 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:16.481 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:16.481 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:16.481 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPushReply(3.1f 85/78 [PushReplyOp(3:fd221d21:::benchmark_data_smithi177_31116_object7639:head)])", 2024-09-13T11:27:16.481 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:42.198715+0000", 2024-09-13T11:27:16.481 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 214.24090303599999, 2024-09-13T11:27:16.481 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.48930391099999998, 2024-09-13T11:27:16.482 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:27:16.482 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:27:16.482 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-13T11:27:16.482 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:27:16.482 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:16.482 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:27:16.482 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.198715+0000", 2024-09-13T11:27:16.482 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:16.482 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:16.482 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:16.482 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:27:16.482 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.198715+0000", 2024-09-13T11:27:16.482 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:16.482 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:16.482 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:16.483 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:27:16.483 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.198716+0000", 2024-09-13T11:27:16.483 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.3500000000000005e-07 2024-09-13T11:27:16.483 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:16.483 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:16.483 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:27:16.483 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.198719+0000", 2024-09-13T11:27:16.483 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.4539999999999999e-06 2024-09-13T11:27:16.483 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:16.483 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:16.483 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:27:16.483 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.684732+0000", 2024-09-13T11:27:16.483 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.48601262099999998 2024-09-13T11:27:16.484 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:16.484 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:16.484 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:27:16.484 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.684746+0000", 2024-09-13T11:27:16.484 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.4518999999999999e-05 2024-09-13T11:27:16.484 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:16.484 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:16.484 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:27:16.484 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.684911+0000", 2024-09-13T11:27:16.484 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000164728 2024-09-13T11:27:16.484 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:16.484 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:16.484 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:27:16.485 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.688019+0000", 2024-09-13T11:27:16.485 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0031077539999999999 2024-09-13T11:27:16.485 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:16.485 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:27:16.485 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:16.485 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:16.485 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:16.485 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPushReply(3.1c 85/77 [PushReplyOp(3:3a669437:::benchmark_data_smithi177_31116_object25280:head)])", 2024-09-13T11:27:16.485 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:42.352466+0000", 2024-09-13T11:27:16.485 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 214.08715214200001, 2024-09-13T11:27:16.485 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.36670725300000001, 2024-09-13T11:27:16.485 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:27:16.485 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:27:16.485 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-13T11:27:16.486 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:27:16.486 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:16.486 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:27:16.486 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.352466+0000", 2024-09-13T11:27:16.486 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:16.486 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:16.486 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:16.486 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:27:16.486 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.352466+0000", 2024-09-13T11:27:16.486 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:16.486 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:16.486 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:16.486 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:27:16.486 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.352467+0000", 2024-09-13T11:27:16.486 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.1429999999999999e-06 2024-09-13T11:27:16.487 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:16.487 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:16.487 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:27:16.487 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.352474+0000", 2024-09-13T11:27:16.487 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.6100000000000002e-06 2024-09-13T11:27:16.487 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:16.487 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:16.487 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:27:16.487 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.717011+0000", 2024-09-13T11:27:16.487 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.36453790200000002 2024-09-13T11:27:16.487 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:16.487 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:16.487 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:27:16.487 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.717032+0000", 2024-09-13T11:27:16.488 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.0999e-05 2024-09-13T11:27:16.488 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:16.488 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:16.488 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:27:16.488 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.717162+0000", 2024-09-13T11:27:16.488 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000129159 2024-09-13T11:27:16.488 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:16.488 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:16.488 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:27:16.488 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.719173+0000", 2024-09-13T11:27:16.488 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0020114400000000002 2024-09-13T11:27:16.488 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:16.488 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:27:16.488 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:16.489 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:16.489 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:16.489 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPushReply(3.21 89/78 [PushReplyOp(3:8534d087:::benchmark_data_smithi177_31116_object5474:head)])", 2024-09-13T11:27:16.489 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:46.632834+0000", 2024-09-13T11:27:16.489 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 209.80678386, 2024-09-13T11:27:16.489 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.324805556, 2024-09-13T11:27:16.489 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:27:16.489 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:27:16.489 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-13T11:27:16.489 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:27:16.489 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:16.489 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:27:16.489 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.632834+0000", 2024-09-13T11:27:16.489 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:16.489 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:16.490 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:16.490 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:27:16.490 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.632834+0000", 2024-09-13T11:27:16.490 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:16.490 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:16.490 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:16.490 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:27:16.490 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.632835+0000", 2024-09-13T11:27:16.490 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.4270000000000001e-06 2024-09-13T11:27:16.490 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:16.490 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:16.490 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:27:16.490 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.632840+0000", 2024-09-13T11:27:16.490 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.7809999999999996e-06 2024-09-13T11:27:16.491 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:16.491 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:16.491 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:27:16.491 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.956282+0000", 2024-09-13T11:27:16.491 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.323441639 2024-09-13T11:27:16.491 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:16.491 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:16.491 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:27:16.491 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.956293+0000", 2024-09-13T11:27:16.491 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.083e-05 2024-09-13T11:27:16.491 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:16.491 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:16.491 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:27:16.491 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.956395+0000", 2024-09-13T11:27:16.492 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00010184200000000001 2024-09-13T11:27:16.492 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:16.492 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:16.492 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:27:16.492 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.957640+0000", 2024-09-13T11:27:16.492 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0012450370000000001 2024-09-13T11:27:16.492 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:16.492 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:27:16.492 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:16.492 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:16.492 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:16.492 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:83223 3.1c 3:3bc6018d:::benchmark_data_smithi177_31116_object83222:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106)", 2024-09-13T11:27:16.492 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:04.511163+0000", 2024-09-13T11:27:16.492 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 191.928454483, 2024-09-13T11:27:16.493 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.47558093200000001, 2024-09-13T11:27:16.493 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:27:16.493 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:27:16.493 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:27:16.493 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:27:16.493 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:27:16.493 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:27:16.493 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 83223 2024-09-13T11:27:16.493 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:16.493 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:27:16.493 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:16.493 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:27:16.493 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.511163+0000", 2024-09-13T11:27:16.493 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:16.494 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:16.494 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:16.494 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:27:16.494 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.511163+0000", 2024-09-13T11:27:16.494 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:16.494 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:16.494 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:16.494 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:27:16.494 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.511167+0000", 2024-09-13T11:27:16.494 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.8330000000000004e-06 2024-09-13T11:27:16.494 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:16.494 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:16.494 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:27:16.494 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.511219+0000", 2024-09-13T11:27:16.495 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.1771999999999998e-05 2024-09-13T11:27:16.495 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:16.495 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:16.495 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:27:16.495 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.511252+0000", 2024-09-13T11:27:16.495 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.3315999999999997e-05 2024-09-13T11:27:16.495 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:16.495 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:16.495 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:27:16.495 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.511268+0000", 2024-09-13T11:27:16.495 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5812999999999999e-05 2024-09-13T11:27:16.495 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:16.495 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:16.495 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:27:16.495 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.511418+0000", 2024-09-13T11:27:16.496 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000150023 2024-09-13T11:27:16.496 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:16.496 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:16.496 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:27:16.496 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.511670+0000", 2024-09-13T11:27:16.496 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00025185499999999999 2024-09-13T11:27:16.496 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:16.496 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:16.496 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 7", 2024-09-13T11:27:16.496 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.511905+0000", 2024-09-13T11:27:16.496 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00023493799999999999 2024-09-13T11:27:16.496 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:16.496 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:16.497 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:27:16.497 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.514960+0000", 2024-09-13T11:27:16.497 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.003054977 2024-09-13T11:27:16.497 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:16.497 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:16.497 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:27:16.497 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.986432+0000", 2024-09-13T11:27:16.497 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.47147209200000001 2024-09-13T11:27:16.497 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:16.497 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:16.498 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:27:16.498 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.986635+0000", 2024-09-13T11:27:16.498 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00020309899999999999 2024-09-13T11:27:16.498 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:16.498 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:16.498 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:27:16.498 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.986744+0000", 2024-09-13T11:27:16.498 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000109214 2024-09-13T11:27:16.498 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:16.498 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:27:16.498 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:16.498 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:16.498 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:16.499 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop_reply(client.4665.0:84803 3.1f e107/78)", 2024-09-13T11:27:16.499 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:05.912627+0000", 2024-09-13T11:27:16.499 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 190.526990576, 2024-09-13T11:27:16.499 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.33726566600000002, 2024-09-13T11:27:16.499 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:27:16.499 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:27:16.499 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-13T11:27:16.499 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:27:16.499 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:16.499 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:27:16.499 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:05.912627+0000", 2024-09-13T11:27:16.499 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:16.499 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:16.499 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:16.500 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:27:16.500 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:05.912627+0000", 2024-09-13T11:27:16.500 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:16.500 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:16.500 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:16.500 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:27:16.500 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:05.912629+0000", 2024-09-13T11:27:16.500 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.4440000000000001e-06 2024-09-13T11:27:16.500 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:16.500 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:16.500 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:27:16.500 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:05.912634+0000", 2024-09-13T11:27:16.500 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.6869999999999999e-06 2024-09-13T11:27:16.500 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:16.501 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:16.501 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:27:16.501 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:06.249722+0000", 2024-09-13T11:27:16.501 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.33708766099999998 2024-09-13T11:27:16.501 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:16.501 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:16.501 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:27:16.501 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:06.249736+0000", 2024-09-13T11:27:16.501 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3623e-05 2024-09-13T11:27:16.501 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:16.501 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:16.501 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:27:16.501 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:06.249878+0000", 2024-09-13T11:27:16.501 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00014264599999999999 2024-09-13T11:27:16.502 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:16.502 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:16.502 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:27:16.502 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:06.249893+0000", 2024-09-13T11:27:16.502 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.4605e-05 2024-09-13T11:27:16.502 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:16.502 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:27:16.502 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:16.502 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:16.502 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:16.502 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:110265 3.1 e119/75 3:8236484d:::benchmark_data_smithi177_31116_object110264:head v 119'3935, mlcod=118'3933)", 2024-09-13T11:27:16.502 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:25.343348+0000", 2024-09-13T11:27:16.502 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 171.096270334, 2024-09-13T11:27:16.502 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.60013013500000001, 2024-09-13T11:27:16.503 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:27:16.503 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:27:16.503 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:27:16.503 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:27:16.503 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:16.503 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:27:16.503 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.343348+0000", 2024-09-13T11:27:16.503 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:16.503 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:16.503 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:16.503 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:27:16.503 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.343348+0000", 2024-09-13T11:27:16.503 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:16.503 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:16.503 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:16.504 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:27:16.504 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.343350+0000", 2024-09-13T11:27:16.504 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.576e-06 2024-09-13T11:27:16.504 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:16.504 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:16.504 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:27:16.504 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.343443+0000", 2024-09-13T11:27:16.504 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.2989e-05 2024-09-13T11:27:16.504 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:16.504 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:16.504 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:27:16.504 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.940863+0000", 2024-09-13T11:27:16.504 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.59742003300000002 2024-09-13T11:27:16.504 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:16.505 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:16.505 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:27:16.505 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.940875+0000", 2024-09-13T11:27:16.505 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.2147000000000001e-05 2024-09-13T11:27:16.505 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:16.505 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:16.505 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:27:16.505 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.940993+0000", 2024-09-13T11:27:16.505 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000117903 2024-09-13T11:27:16.505 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:16.505 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:16.505 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:27:16.505 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.941017+0000", 2024-09-13T11:27:16.505 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.3442999999999999e-05 2024-09-13T11:27:16.506 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:16.506 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:16.506 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:27:16.506 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.943347+0000", 2024-09-13T11:27:16.506 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0023299869999999999 2024-09-13T11:27:16.506 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:16.506 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:16.506 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:27:16.506 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.943478+0000", 2024-09-13T11:27:16.506 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00013105699999999999 2024-09-13T11:27:16.506 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:16.506 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:27:16.506 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:16.506 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:16.506 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:16.507 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:260347 3.16 3:6d3745c1:::benchmark_data_smithi177_31116_object120044:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e219)", 2024-09-13T11:27:16.507 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:26:42.470302+0000", 2024-09-13T11:27:16.507 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 33.969315743000003, 2024-09-13T11:27:16.507 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.43228830200000001, 2024-09-13T11:27:16.507 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:27:16.507 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:27:16.507 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:27:16.507 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:27:16.507 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:27:16.507 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:27:16.507 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 260347 2024-09-13T11:27:16.507 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:16.507 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:27:16.507 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:16.508 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:27:16.508 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:42.470302+0000", 2024-09-13T11:27:16.508 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:16.508 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:16.508 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:16.508 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:27:16.508 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:42.470302+0000", 2024-09-13T11:27:16.508 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:16.508 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:16.508 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:16.508 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:27:16.508 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:42.470306+0000", 2024-09-13T11:27:16.508 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.2899999999999996e-06 2024-09-13T11:27:16.508 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:16.509 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:16.509 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:27:16.509 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:42.470318+0000", 2024-09-13T11:27:16.509 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.1426e-05 2024-09-13T11:27:16.509 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:16.509 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:16.509 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:27:16.509 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:42.470361+0000", 2024-09-13T11:27:16.509 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.35e-05 2024-09-13T11:27:16.509 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:16.509 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:16.509 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:27:16.509 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:42.470371+0000", 2024-09-13T11:27:16.509 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.1740000000000006e-06 2024-09-13T11:27:16.510 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:16.510 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:16.510 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:27:16.510 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:42.470474+0000", 2024-09-13T11:27:16.510 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000103225 2024-09-13T11:27:16.510 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:16.510 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:16.510 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:27:16.510 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:42.470735+0000", 2024-09-13T11:27:16.510 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00026167399999999999 2024-09-13T11:27:16.510 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:16.510 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:16.510 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 15", 2024-09-13T11:27:16.510 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:42.470939+0000", 2024-09-13T11:27:16.510 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00020340000000000001 2024-09-13T11:27:16.511 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:16.511 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:16.511 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:27:16.511 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:42.477467+0000", 2024-09-13T11:27:16.511 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0065277319999999996 2024-09-13T11:27:16.511 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:16.511 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:16.511 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:27:16.511 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:42.902321+0000", 2024-09-13T11:27:16.511 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.42485422099999998 2024-09-13T11:27:16.511 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:16.511 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:16.511 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:27:16.511 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:42.902468+0000", 2024-09-13T11:27:16.512 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000147591 2024-09-13T11:27:16.512 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:16.512 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:16.512 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:27:16.512 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:42.902590+0000", 2024-09-13T11:27:16.512 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000122069 2024-09-13T11:27:16.512 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:16.512 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:27:16.512 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:16.512 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:16.512 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:16.512 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:266874 3.2f 3:f5648933:::benchmark_data_smithi177_31116_object126571:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e224)", 2024-09-13T11:27:16.512 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:26:49.252945+0000", 2024-09-13T11:27:16.513 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 27.186672911999999, 2024-09-13T11:27:16.513 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.79831508200000001, 2024-09-13T11:27:16.513 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:27:16.513 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:27:16.513 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:27:16.513 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:27:16.513 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:27:16.513 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:27:16.513 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 266874 2024-09-13T11:27:16.513 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:16.513 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:27:16.513 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:16.513 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:27:16.514 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:49.252945+0000", 2024-09-13T11:27:16.514 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:16.514 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:16.514 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:16.514 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:27:16.514 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:49.252945+0000", 2024-09-13T11:27:16.514 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:16.514 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:16.514 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:16.514 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:27:16.514 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:49.252949+0000", 2024-09-13T11:27:16.514 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.4390000000000002e-06 2024-09-13T11:27:16.514 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:16.514 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:16.515 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:27:16.515 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:49.252961+0000", 2024-09-13T11:27:16.515 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.1891e-05 2024-09-13T11:27:16.515 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:16.515 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:16.515 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:27:16.515 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:49.252989+0000", 2024-09-13T11:27:16.515 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.7384000000000001e-05 2024-09-13T11:27:16.515 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:16.515 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:16.515 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:27:16.515 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:49.253004+0000", 2024-09-13T11:27:16.515 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5013e-05 2024-09-13T11:27:16.515 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:16.515 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:16.516 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:27:16.516 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:49.253095+0000", 2024-09-13T11:27:16.516 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.1150000000000007e-05 2024-09-13T11:27:16.516 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:16.516 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:16.516 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for peered", 2024-09-13T11:27:16.516 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:49.253106+0000", 2024-09-13T11:27:16.516 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0749e-05 2024-09-13T11:27:16.516 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:16.516 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:16.516 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:27:16.516 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:50.047276+0000", 2024-09-13T11:27:16.516 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.79417086299999995 2024-09-13T11:27:16.517 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:16.517 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:16.517 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:27:16.517 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:50.047392+0000", 2024-09-13T11:27:16.517 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000115777 2024-09-13T11:27:16.517 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:16.517 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:16.517 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 12", 2024-09-13T11:27:16.517 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:50.047515+0000", 2024-09-13T11:27:16.517 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00012257699999999999 2024-09-13T11:27:16.517 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:16.517 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:16.517 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:27:16.517 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:50.050394+0000", 2024-09-13T11:27:16.518 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.002879038 2024-09-13T11:27:16.518 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:16.518 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:16.518 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:27:16.518 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:50.051179+0000", 2024-09-13T11:27:16.518 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000785 2024-09-13T11:27:16.518 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:16.518 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:16.518 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:27:16.518 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:50.051223+0000", 2024-09-13T11:27:16.519 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.4202000000000001e-05 2024-09-13T11:27:16.519 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:16.519 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:16.519 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:27:16.519 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:50.051260+0000", 2024-09-13T11:27:16.519 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.6999000000000003e-05 2024-09-13T11:27:16.519 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:16.519 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:27:16.519 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:16.519 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:16.519 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:16.519 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGRecoveryDeleteReply(3.2f e231,227 3:f4d0ce4f:::benchmark_data_smithi177_31116_object254:head,144'6093)", 2024-09-13T11:27:16.519 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:26:58.970945+0000", 2024-09-13T11:27:16.520 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 17.468672562999998, 2024-09-13T11:27:16.520 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.45467746999999997, 2024-09-13T11:27:16.520 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:27:16.520 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:27:16.520 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-13T11:27:16.520 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:27:16.520 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:16.520 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:27:16.520 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:58.970945+0000", 2024-09-13T11:27:16.520 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:16.520 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:16.520 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:16.520 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:27:16.520 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:58.970945+0000", 2024-09-13T11:27:16.521 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:16.521 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:16.521 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:16.521 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:27:16.521 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:58.970947+0000", 2024-09-13T11:27:16.521 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.668e-06 2024-09-13T11:27:16.521 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:16.521 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:16.521 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:27:16.521 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:58.970961+0000", 2024-09-13T11:27:16.521 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3655999999999999e-05 2024-09-13T11:27:16.521 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:16.521 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:16.522 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:27:16.522 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:59.424664+0000", 2024-09-13T11:27:16.522 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.45370305599999999 2024-09-13T11:27:16.522 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:16.522 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:16.522 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:27:16.522 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:59.424682+0000", 2024-09-13T11:27:16.522 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.7949999999999999e-05 2024-09-13T11:27:16.522 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:16.522 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:16.522 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:27:16.522 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:59.425051+0000", 2024-09-13T11:27:16.522 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00036941300000000002 2024-09-13T11:27:16.522 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:16.523 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:16.523 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:27:16.523 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:59.425623+0000", 2024-09-13T11:27:16.523 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00057172699999999996 2024-09-13T11:27:16.523 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:16.523 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:27:16.523 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:16.523 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:16.523 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:16.523 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:274423 3.1c 3:391423c9:::benchmark_data_smithi177_31116_object134120:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e233)", 2024-09-13T11:27:16.523 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:27:00.620397+0000", 2024-09-13T11:27:16.523 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 15.819220982000001, 2024-09-13T11:27:16.523 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.34487627199999998, 2024-09-13T11:27:16.523 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:27:16.523 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:27:16.524 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:27:16.524 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:27:16.524 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:27:16.524 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:27:16.524 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 274423 2024-09-13T11:27:16.524 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:16.524 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:27:16.524 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:16.524 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:27:16.524 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:27:00.620397+0000", 2024-09-13T11:27:16.524 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:16.524 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:16.524 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:16.524 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:27:16.525 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:27:00.620397+0000", 2024-09-13T11:27:16.525 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:16.525 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:16.525 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:16.525 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:27:16.525 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:27:00.620399+0000", 2024-09-13T11:27:16.525 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.657e-06 2024-09-13T11:27:16.525 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:16.525 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:16.525 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:27:16.525 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:27:00.620403+0000", 2024-09-13T11:27:16.525 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.3069999999999996e-06 2024-09-13T11:27:16.525 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:16.525 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:16.526 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:27:16.526 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:27:00.620413+0000", 2024-09-13T11:27:16.526 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0355e-05 2024-09-13T11:27:16.526 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:16.526 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:16.526 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:27:16.526 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:27:00.620421+0000", 2024-09-13T11:27:16.526 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.0250000000000002e-06 2024-09-13T11:27:16.526 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:16.526 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:16.526 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:27:16.526 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:27:00.620851+0000", 2024-09-13T11:27:16.526 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00042989900000000003 2024-09-13T11:27:16.526 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:16.526 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:16.527 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:27:16.527 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:27:00.620967+0000", 2024-09-13T11:27:16.527 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00011627 2024-09-13T11:27:16.527 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:16.527 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:16.527 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 7", 2024-09-13T11:27:16.527 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:27:00.621613+0000", 2024-09-13T11:27:16.527 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00064577099999999995 2024-09-13T11:27:16.527 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:16.527 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:16.527 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:27:16.527 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:27:00.627319+0000", 2024-09-13T11:27:16.527 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0057055200000000004 2024-09-13T11:27:16.527 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:16.528 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:16.528 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:27:16.528 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:27:00.965178+0000", 2024-09-13T11:27:16.528 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.33785940599999997 2024-09-13T11:27:16.528 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:16.528 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:16.528 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:27:16.528 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:27:00.965231+0000", 2024-09-13T11:27:16.528 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.2944e-05 2024-09-13T11:27:16.528 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:16.528 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:16.528 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:27:16.528 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:27:00.965273+0000", 2024-09-13T11:27:16.528 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.2117999999999999e-05 2024-09-13T11:27:16.529 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:16.529 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:27:16.529 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:16.529 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:16.529 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:16.529 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGRecoveryDeleteReply(3.2f e238,227 3:f487a72d:::benchmark_data_smithi177_31116_object20739:head,155'6770)", 2024-09-13T11:27:16.529 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:27:06.435293+0000", 2024-09-13T11:27:16.529 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 10.004325156, 2024-09-13T11:27:16.529 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.398313372, 2024-09-13T11:27:16.529 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:27:16.529 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:27:16.529 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-13T11:27:16.529 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:27:16.529 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:16.530 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:27:16.530 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:27:06.435293+0000", 2024-09-13T11:27:16.530 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:16.530 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:16.530 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:16.530 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:27:16.530 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:27:06.435293+0000", 2024-09-13T11:27:16.530 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:16.530 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:16.530 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:16.530 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:27:16.530 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:27:06.435294+0000", 2024-09-13T11:27:16.530 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.234e-06 2024-09-13T11:27:16.530 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:16.530 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:16.531 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:27:16.531 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:27:06.435302+0000", 2024-09-13T11:27:16.531 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.7959999999999996e-06 2024-09-13T11:27:16.531 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:16.531 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:16.531 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:27:16.531 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:27:06.833212+0000", 2024-09-13T11:27:16.531 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.39791068000000002 2024-09-13T11:27:16.531 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:16.531 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:16.531 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:27:16.531 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:27:06.833219+0000", 2024-09-13T11:27:16.531 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.7449999999999998e-06 2024-09-13T11:27:16.531 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:16.531 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:16.532 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:27:16.532 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:27:06.833270+0000", 2024-09-13T11:27:16.532 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.0717000000000002e-05 2024-09-13T11:27:16.532 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:16.532 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:16.532 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:27:16.532 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:27:06.833606+0000", 2024-09-13T11:27:16.532 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00033619999999999999 2024-09-13T11:27:16.532 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:16.532 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:27:16.532 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:16.532 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:16.532 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:16.532 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGRecoveryDeleteReply(3.36 e239,227 3:6c15f3c5:::benchmark_data_smithi177_31116_object28159:head,161'6777)", 2024-09-13T11:27:16.533 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:27:07.338993+0000", 2024-09-13T11:27:16.533 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 9.1006250150000003, 2024-09-13T11:27:16.533 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.68653944099999997, 2024-09-13T11:27:16.533 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:27:16.533 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:27:16.533 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-13T11:27:16.533 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:27:16.533 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:16.533 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:27:16.533 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:27:07.338993+0000", 2024-09-13T11:27:16.533 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:16.533 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:16.533 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:16.533 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:27:16.533 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:27:07.338993+0000", 2024-09-13T11:27:16.534 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:16.534 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:16.534 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:16.534 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:27:16.534 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:27:07.338994+0000", 2024-09-13T11:27:16.534 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.2979999999999999e-06 2024-09-13T11:27:16.534 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:16.534 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:16.534 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:27:16.534 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:27:07.339000+0000", 2024-09-13T11:27:16.534 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.8239999999999996e-06 2024-09-13T11:27:16.534 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:16.534 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:16.534 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:27:16.535 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:27:08.025032+0000", 2024-09-13T11:27:16.535 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.68603244399999996 2024-09-13T11:27:16.535 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:16.535 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:16.535 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:27:16.535 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:27:08.025042+0000", 2024-09-13T11:27:16.535 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.2269999999999999e-06 2024-09-13T11:27:16.535 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:16.535 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:16.535 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:27:16.535 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:27:08.025135+0000", 2024-09-13T11:27:16.535 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.3321999999999999e-05 2024-09-13T11:27:16.535 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:16.535 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:16.536 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:27:16.536 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:27:08.025532+0000", 2024-09-13T11:27:16.536 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00039732599999999998 2024-09-13T11:27:16.536 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:16.536 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:27:16.536 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:16.536 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:16.536 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:27:16.536 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-09-13T11:27:16.537 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-13T11:27:16.538 INFO:tasks.ceph.osd.14.smithi162.stderr:2024-09-13T11:27:16.507+0000 7f3dea330640 -1 osd.14 244 handle_osd_map injecting map crc failure 2024-09-13T11:27:16.538 INFO:tasks.ceph.osd.2.smithi162.stderr:2024-09-13T11:27:16.523+0000 7fe91fd87640 -1 osd.2 244 handle_osd_map injecting map crc failure 2024-09-13T11:27:16.539 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_ops_in_flight 2024-09-13T11:27:16.540 INFO:tasks.ceph.osd.7.smithi177.stderr:2024-09-13T11:27:16.533+0000 7f897b51d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26575) UID: 0 2024-09-13T11:27:16.638 INFO:tasks.ceph.osd.9.smithi092.stderr:2024-09-13T11:27:16.635+0000 7ff5ab3d0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26348) UID: 0 2024-09-13T11:27:16.728 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-09-13T11:27:16.729 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [], 2024-09-13T11:27:16.729 INFO:teuthology.orchestra.run.smithi177.stdout: "num_ops": 0 2024-09-13T11:27:16.729 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-09-13T11:27:16.739 INFO:tasks.ceph.osd.1.smithi092.stderr:2024-09-13T11:27:16.735+0000 7fe7789c5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33718) UID: 0 2024-09-13T11:27:16.750 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_blocked_ops 2024-09-13T11:27:16.840 INFO:tasks.ceph.osd.13.smithi092.stderr:2024-09-13T11:27:16.839+0000 7f3e5afea640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26350) UID: 0 2024-09-13T11:27:16.889 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-09-13T11:27:16.889 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [], 2024-09-13T11:27:16.889 INFO:teuthology.orchestra.run.smithi177.stdout: "complaint_time": 30, 2024-09-13T11:27:16.889 INFO:teuthology.orchestra.run.smithi177.stdout: "num_blocked_ops": 0 2024-09-13T11:27:16.889 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-09-13T11:27:16.908 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_historic_ops 2024-09-13T11:27:16.941 INFO:tasks.ceph.osd.12.smithi001.stderr:2024-09-13T11:27:16.938+0000 7f892c6e6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26314) UID: 0 2024-09-13T11:27:17.008 INFO:tasks.ceph.osd.1.smithi092.stderr:2024-09-13T11:27:17.003+0000 7fe76d3f2640 -1 osd.1 245 handle_osd_map injecting map crc failure 2024-09-13T11:27:17.042 INFO:tasks.ceph.osd.9.smithi092.stderr:2024-09-13T11:27:17.039+0000 7ff5ab3d0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26348) UID: 0 2024-09-13T11:27:17.062 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-09-13T11:27:17.063 INFO:teuthology.orchestra.run.smithi177.stdout: "size": 20, 2024-09-13T11:27:17.063 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 600, 2024-09-13T11:27:17.063 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [ 2024-09-13T11:27:17.063 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:17.063 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:25383 3.0 e55/30 3:0a6bb869:::benchmark_data_smithi177_31116_object25382:head v 55'1620, mlcod=55'1617)", 2024-09-13T11:27:17.063 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:04.449956+0000", 2024-09-13T11:27:17.063 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 252.61303059900001, 2024-09-13T11:27:17.063 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.21143479600000001, 2024-09-13T11:27:17.063 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:27:17.063 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:27:17.063 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:27:17.063 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:27:17.064 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:17.064 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:27:17.064 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.449956+0000", 2024-09-13T11:27:17.064 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:17.064 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:17.064 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:17.064 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:27:17.064 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.449956+0000", 2024-09-13T11:27:17.064 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:17.064 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:17.064 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:17.064 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:27:17.064 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.449959+0000", 2024-09-13T11:27:17.064 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.393e-06 2024-09-13T11:27:17.065 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:17.065 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:17.065 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:27:17.065 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.450014+0000", 2024-09-13T11:27:17.065 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.4200000000000003e-05 2024-09-13T11:27:17.065 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:17.065 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:17.065 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:27:17.065 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.658637+0000", 2024-09-13T11:27:17.065 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.20862334799999999 2024-09-13T11:27:17.065 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:17.066 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:17.066 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:27:17.066 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.658650+0000", 2024-09-13T11:27:17.066 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3148e-05 2024-09-13T11:27:17.066 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:17.066 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:17.066 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:27:17.066 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.658867+0000", 2024-09-13T11:27:17.066 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00021673499999999999 2024-09-13T11:27:17.066 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:17.066 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:17.066 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:27:17.066 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.658931+0000", 2024-09-13T11:27:17.067 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.4045000000000001e-05 2024-09-13T11:27:17.067 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:17.068 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:17.068 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:27:17.068 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.661317+0000", 2024-09-13T11:27:17.068 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.002385823 2024-09-13T11:27:17.068 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:17.068 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:17.069 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:27:17.069 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.661391+0000", 2024-09-13T11:27:17.069 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.4103999999999994e-05 2024-09-13T11:27:17.069 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:17.069 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:27:17.069 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:17.069 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:17.069 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:17.069 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPush(3.16 85/70 [PushOp(3:69aac9fd:::benchmark_data_smithi177_31116_object43914:head, version: 66'2837, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:69aac9fd:::benchmark_data_smithi177_31116_object43914:head@66'2837, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:27:17.069 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:42.295995+0000", 2024-09-13T11:27:17.069 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 214.766991279, 2024-09-13T11:27:17.070 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.215089431, 2024-09-13T11:27:17.070 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:27:17.070 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:27:17.070 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:27:17.070 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:27:17.070 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:17.070 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:27:17.071 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.295995+0000", 2024-09-13T11:27:17.071 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:17.071 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:17.071 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:17.071 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:27:17.071 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.295995+0000", 2024-09-13T11:27:17.071 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:17.071 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:17.072 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:17.072 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:27:17.072 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.295997+0000", 2024-09-13T11:27:17.072 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3170000000000001e-06 2024-09-13T11:27:17.072 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:17.072 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:17.072 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:27:17.072 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.296058+0000", 2024-09-13T11:27:17.073 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.1690000000000001e-05 2024-09-13T11:27:17.073 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:17.073 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:17.073 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:27:17.073 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.509606+0000", 2024-09-13T11:27:17.073 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.21354748300000001 2024-09-13T11:27:17.073 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:17.073 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:17.073 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:27:17.073 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.509638+0000", 2024-09-13T11:27:17.074 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.1644999999999999e-05 2024-09-13T11:27:17.074 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:17.074 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:17.074 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:27:17.074 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.509814+0000", 2024-09-13T11:27:17.074 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00017679499999999999 2024-09-13T11:27:17.074 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:17.074 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:17.074 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:27:17.074 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.509840+0000", 2024-09-13T11:27:17.074 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.5871e-05 2024-09-13T11:27:17.075 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:17.075 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:17.075 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:27:17.075 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.511085+0000", 2024-09-13T11:27:17.075 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00124463 2024-09-13T11:27:17.075 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:17.075 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:27:17.075 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:17.075 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:17.075 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:17.075 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPush(3.1d 91/78 [PushOp(3:be6a0ae4:::benchmark_data_smithi177_31116_object25671:head, version: 58'1569, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:be6a0ae4:::benchmark_data_smithi177_31116_object25671:head@58'1569, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:27:17.076 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:48.481971+0000", 2024-09-13T11:27:17.076 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 208.58101603399999, 2024-09-13T11:27:17.076 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.47945454199999998, 2024-09-13T11:27:17.076 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:27:17.076 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:27:17.076 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:27:17.076 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:27:17.076 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:17.076 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:27:17.076 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.481971+0000", 2024-09-13T11:27:17.076 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:17.077 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:17.077 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:17.077 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:27:17.077 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.481971+0000", 2024-09-13T11:27:17.077 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:17.077 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:17.077 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:17.077 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:27:17.077 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.481972+0000", 2024-09-13T11:27:17.077 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5179999999999999e-06 2024-09-13T11:27:17.077 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:17.077 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:17.077 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:27:17.077 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.482000+0000", 2024-09-13T11:27:17.078 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.8249e-05 2024-09-13T11:27:17.078 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:17.078 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:17.078 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:27:17.078 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.959926+0000", 2024-09-13T11:27:17.078 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.47792585100000001 2024-09-13T11:27:17.078 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:17.078 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:17.078 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:27:17.078 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.959944+0000", 2024-09-13T11:27:17.078 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.7620000000000001e-05 2024-09-13T11:27:17.078 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:17.078 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:17.078 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:27:17.079 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.960059+0000", 2024-09-13T11:27:17.079 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000115188 2024-09-13T11:27:17.079 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:17.079 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:17.079 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:27:17.079 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.960074+0000", 2024-09-13T11:27:17.079 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.4738e-05 2024-09-13T11:27:17.079 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:17.079 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:17.079 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:27:17.079 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.961425+0000", 2024-09-13T11:27:17.079 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0013513780000000001 2024-09-13T11:27:17.079 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:17.079 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:27:17.080 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:17.080 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:17.080 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:17.080 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:71884 3.16 e96/88 3:6ca846b2:::benchmark_data_smithi177_31116_object71883:head v 96'3755, mlcod=96'3753)", 2024-09-13T11:27:17.080 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:54.108299+0000", 2024-09-13T11:27:17.080 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 202.954687861, 2024-09-13T11:27:17.080 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.20685140099999999, 2024-09-13T11:27:17.080 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:27:17.080 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:27:17.080 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:27:17.080 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:27:17.080 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:17.080 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:27:17.081 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.108299+0000", 2024-09-13T11:27:17.081 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:17.081 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:17.081 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:17.081 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:27:17.081 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.108299+0000", 2024-09-13T11:27:17.081 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:17.081 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:17.081 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:17.081 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:27:17.081 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.108300+0000", 2024-09-13T11:27:17.081 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.173e-06 2024-09-13T11:27:17.081 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:17.081 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:17.082 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:27:17.082 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.108323+0000", 2024-09-13T11:27:17.082 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.2591000000000001e-05 2024-09-13T11:27:17.082 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:17.082 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:17.082 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:27:17.082 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.312377+0000", 2024-09-13T11:27:17.082 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.20405456599999999 2024-09-13T11:27:17.082 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:17.082 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:17.082 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:27:17.082 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.312382+0000", 2024-09-13T11:27:17.082 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.1429999999999997e-06 2024-09-13T11:27:17.082 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:17.083 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:17.083 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:27:17.083 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.312480+0000", 2024-09-13T11:27:17.083 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.7742e-05 2024-09-13T11:27:17.083 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:17.083 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:17.083 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:27:17.083 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.312521+0000", 2024-09-13T11:27:17.083 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.1369000000000003e-05 2024-09-13T11:27:17.083 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:17.083 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:17.083 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:27:17.083 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.315105+0000", 2024-09-13T11:27:17.083 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0025835060000000002 2024-09-13T11:27:17.084 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:17.084 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:17.084 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:27:17.084 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.315150+0000", 2024-09-13T11:27:17.084 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.5311e-05 2024-09-13T11:27:17.084 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:17.084 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:27:17.084 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:17.084 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:17.084 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:17.084 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:71927 3.0 e97/75 3:0197244a:::benchmark_data_smithi177_31116_object71926:head v 97'3351, mlcod=96'3348)", 2024-09-13T11:27:17.084 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:54.200161+0000", 2024-09-13T11:27:17.084 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 202.862825633, 2024-09-13T11:27:17.085 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.26691738999999998, 2024-09-13T11:27:17.085 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:27:17.085 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:27:17.085 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:27:17.085 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:27:17.085 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:17.085 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:27:17.085 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.200161+0000", 2024-09-13T11:27:17.085 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:17.085 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:17.085 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:17.085 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:27:17.085 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.200161+0000", 2024-09-13T11:27:17.085 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:17.086 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:17.086 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:17.086 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:27:17.086 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.200162+0000", 2024-09-13T11:27:17.086 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.78e-07 2024-09-13T11:27:17.086 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:17.086 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:17.086 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:27:17.086 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.200200+0000", 2024-09-13T11:27:17.086 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.8121000000000003e-05 2024-09-13T11:27:17.086 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:17.086 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:17.086 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:27:17.086 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.465397+0000", 2024-09-13T11:27:17.087 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.26519681099999998 2024-09-13T11:27:17.087 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:17.087 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:17.087 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:27:17.087 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.465404+0000", 2024-09-13T11:27:17.087 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.7399999999999998e-06 2024-09-13T11:27:17.087 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:17.087 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:17.087 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:27:17.087 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.465456+0000", 2024-09-13T11:27:17.087 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.2626000000000001e-05 2024-09-13T11:27:17.087 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:17.087 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:17.087 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:27:17.088 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.465482+0000", 2024-09-13T11:27:17.088 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.6086000000000001e-05 2024-09-13T11:27:17.088 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:17.088 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:17.088 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:27:17.088 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.467035+0000", 2024-09-13T11:27:17.088 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0015523729999999999 2024-09-13T11:27:17.088 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:17.088 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:17.088 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:27:17.088 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.467078+0000", 2024-09-13T11:27:17.088 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.3754999999999997e-05 2024-09-13T11:27:17.088 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:17.088 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:27:17.089 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:17.089 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:17.089 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:17.089 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPush(3.1b 102/78 [PushOp(3:df786fc7:::benchmark_data_smithi177_31116_object48840:head, version: 71'3079, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:df786fc7:::benchmark_data_smithi177_31116_object48840:head@71'3079, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:27:17.089 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:00.290536+0000", 2024-09-13T11:27:17.089 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 196.77245087099999, 2024-09-13T11:27:17.089 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.20017585399999999, 2024-09-13T11:27:17.089 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:27:17.089 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:27:17.090 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:27:17.090 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:27:17.090 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:17.090 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:27:17.090 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.290536+0000", 2024-09-13T11:27:17.090 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:17.090 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:17.090 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:17.090 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:27:17.090 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.290536+0000", 2024-09-13T11:27:17.090 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:17.090 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:17.090 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:17.091 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:27:17.091 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.290537+0000", 2024-09-13T11:27:17.091 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.327e-06 2024-09-13T11:27:17.091 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:17.091 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:17.091 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:27:17.091 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.290581+0000", 2024-09-13T11:27:17.091 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.3893999999999997e-05 2024-09-13T11:27:17.091 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:17.091 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:17.091 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:27:17.091 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.488588+0000", 2024-09-13T11:27:17.091 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.198006708 2024-09-13T11:27:17.091 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:17.092 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:17.092 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:27:17.092 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.488607+0000", 2024-09-13T11:27:17.092 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.9162999999999999e-05 2024-09-13T11:27:17.092 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:17.092 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:17.092 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:27:17.092 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.488747+0000", 2024-09-13T11:27:17.092 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00014040799999999999 2024-09-13T11:27:17.092 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:17.092 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:17.092 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:27:17.092 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.488770+0000", 2024-09-13T11:27:17.093 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.279e-05 2024-09-13T11:27:17.093 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:17.093 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:17.093 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:27:17.093 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.490712+0000", 2024-09-13T11:27:17.093 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001941564 2024-09-13T11:27:17.093 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:17.094 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:27:17.094 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:17.094 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:17.094 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:17.094 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPush(3.1b 102/78 [PushOp(3:d82bf6ec:::benchmark_data_smithi177_31116_object48847:head, version: 71'3080, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:d82bf6ec:::benchmark_data_smithi177_31116_object48847:head@71'3080, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:27:17.094 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:00.293979+0000", 2024-09-13T11:27:17.094 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 196.76900734399999, 2024-09-13T11:27:17.094 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.20314832099999999, 2024-09-13T11:27:17.095 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:27:17.095 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:27:17.095 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:27:17.095 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:27:17.095 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:17.095 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:27:17.095 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.293979+0000", 2024-09-13T11:27:17.095 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:17.095 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:17.095 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:17.095 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:27:17.095 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.293979+0000", 2024-09-13T11:27:17.095 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:17.095 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:17.095 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:17.096 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:27:17.096 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.293981+0000", 2024-09-13T11:27:17.096 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.4449999999999999e-06 2024-09-13T11:27:17.096 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:17.096 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:17.096 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:27:17.096 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.294085+0000", 2024-09-13T11:27:17.096 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000103848 2024-09-13T11:27:17.096 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:17.096 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:17.096 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:27:17.096 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.495571+0000", 2024-09-13T11:27:17.096 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.20148640800000001 2024-09-13T11:27:17.096 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:17.097 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:17.097 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:27:17.097 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.495609+0000", 2024-09-13T11:27:17.097 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.7904000000000002e-05 2024-09-13T11:27:17.097 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:17.097 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:17.097 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:27:17.097 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.495729+0000", 2024-09-13T11:27:17.097 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000120556 2024-09-13T11:27:17.097 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:17.097 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:17.097 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:27:17.097 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.495751+0000", 2024-09-13T11:27:17.098 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.1511000000000001e-05 2024-09-13T11:27:17.098 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:17.098 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:17.098 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:27:17.098 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.497128+0000", 2024-09-13T11:27:17.098 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001376649 2024-09-13T11:27:17.098 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:17.098 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:27:17.098 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:17.098 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:17.098 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:17.098 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:79801 3.5 3:a6efdf84:::benchmark_data_smithi177_31116_object79800:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102)", 2024-09-13T11:27:17.098 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:00.403201+0000", 2024-09-13T11:27:17.099 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 196.65978575700001, 2024-09-13T11:27:17.099 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.31868626300000003, 2024-09-13T11:27:17.099 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:27:17.099 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:27:17.099 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:27:17.099 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:27:17.099 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:27:17.099 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:27:17.099 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 79801 2024-09-13T11:27:17.099 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:17.099 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:27:17.099 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:17.099 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:27:17.099 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.403201+0000", 2024-09-13T11:27:17.100 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:17.100 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:17.100 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:17.100 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:27:17.100 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.403201+0000", 2024-09-13T11:27:17.100 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:17.100 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:17.100 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:17.100 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:27:17.100 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.403204+0000", 2024-09-13T11:27:17.100 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.0819999999999999e-06 2024-09-13T11:27:17.100 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:17.100 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:17.100 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:27:17.101 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.403260+0000", 2024-09-13T11:27:17.101 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.5624999999999997e-05 2024-09-13T11:27:17.101 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:17.101 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:17.101 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:27:17.101 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.403280+0000", 2024-09-13T11:27:17.101 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.0060000000000001e-05 2024-09-13T11:27:17.101 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:17.101 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:17.101 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:27:17.101 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.403291+0000", 2024-09-13T11:27:17.101 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.1335000000000001e-05 2024-09-13T11:27:17.101 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:17.101 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:17.102 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:27:17.102 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.403344+0000", 2024-09-13T11:27:17.102 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.3217000000000002e-05 2024-09-13T11:27:17.102 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:17.102 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:17.102 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:27:17.102 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.403456+0000", 2024-09-13T11:27:17.102 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000111706 2024-09-13T11:27:17.102 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:17.102 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:17.102 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 13", 2024-09-13T11:27:17.102 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.403547+0000", 2024-09-13T11:27:17.102 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.1556000000000002e-05 2024-09-13T11:27:17.102 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:17.103 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:17.103 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:27:17.103 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.407436+0000", 2024-09-13T11:27:17.103 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.003888856 2024-09-13T11:27:17.103 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:17.103 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:17.103 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:27:17.103 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.721722+0000", 2024-09-13T11:27:17.103 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.31428562700000001 2024-09-13T11:27:17.103 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:17.103 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:17.103 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:27:17.103 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.721814+0000", 2024-09-13T11:27:17.103 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.1564000000000006e-05 2024-09-13T11:27:17.104 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:17.104 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:17.104 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:27:17.104 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.721887+0000", 2024-09-13T11:27:17.104 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.3634999999999995e-05 2024-09-13T11:27:17.104 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:17.104 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:27:17.104 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:17.104 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:17.104 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:17.104 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:83199 3.1d e106/102 3:be2c6fae:::benchmark_data_smithi177_31116_object83198:head v 106'4244, mlcod=106'4242)", 2024-09-13T11:27:17.104 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:04.298299+0000", 2024-09-13T11:27:17.104 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 192.76468763, 2024-09-13T11:27:17.104 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.25001107099999997, 2024-09-13T11:27:17.105 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:27:17.105 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:27:17.105 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:27:17.105 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:27:17.105 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:17.105 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:27:17.105 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.298299+0000", 2024-09-13T11:27:17.105 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:17.105 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:17.105 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:17.105 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:27:17.105 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.298299+0000", 2024-09-13T11:27:17.105 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:17.105 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:17.105 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:17.106 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:27:17.106 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.298301+0000", 2024-09-13T11:27:17.106 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5859999999999999e-06 2024-09-13T11:27:17.106 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:17.106 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:17.106 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:27:17.106 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.298368+0000", 2024-09-13T11:27:17.106 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.6928999999999996e-05 2024-09-13T11:27:17.106 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:17.106 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:17.106 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:27:17.106 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.545180+0000", 2024-09-13T11:27:17.106 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.24681209700000001 2024-09-13T11:27:17.107 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:17.107 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:17.107 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:27:17.107 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.545193+0000", 2024-09-13T11:27:17.107 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3427e-05 2024-09-13T11:27:17.107 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:17.107 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:17.107 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:27:17.107 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.545327+0000", 2024-09-13T11:27:17.107 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00013359400000000001 2024-09-13T11:27:17.107 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:17.107 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:17.107 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:27:17.107 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.545385+0000", 2024-09-13T11:27:17.108 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.8492000000000002e-05 2024-09-13T11:27:17.108 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:17.108 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:17.108 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:27:17.108 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.548225+0000", 2024-09-13T11:27:17.108 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.002839472 2024-09-13T11:27:17.108 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:17.108 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:17.108 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:27:17.108 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.548310+0000", 2024-09-13T11:27:17.108 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.5474000000000002e-05 2024-09-13T11:27:17.108 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:17.108 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:27:17.108 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:17.109 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:17.109 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:17.109 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:137062 3.5 3:a16e178f:::benchmark_data_smithi177_31116_object137061:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e139)", 2024-09-13T11:27:17.109 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:53.140525+0000", 2024-09-13T11:27:17.109 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 143.922462037, 2024-09-13T11:27:17.109 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.20697954599999999, 2024-09-13T11:27:17.109 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:27:17.109 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:27:17.109 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:27:17.109 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:27:17.109 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:27:17.109 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:27:17.109 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 137062 2024-09-13T11:27:17.109 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:17.110 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:27:17.110 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:17.110 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:27:17.110 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:53.140525+0000", 2024-09-13T11:27:17.110 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:17.110 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:17.110 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:17.110 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:27:17.110 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:53.140525+0000", 2024-09-13T11:27:17.110 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:17.110 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:17.110 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:17.110 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:27:17.110 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:53.140527+0000", 2024-09-13T11:27:17.111 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.1739999999999999e-06 2024-09-13T11:27:17.111 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:17.111 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:17.111 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:27:17.111 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:53.140553+0000", 2024-09-13T11:27:17.111 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.6631e-05 2024-09-13T11:27:17.111 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:17.111 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:17.111 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:27:17.111 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:53.140566+0000", 2024-09-13T11:27:17.111 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.2268e-05 2024-09-13T11:27:17.111 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:17.111 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:17.111 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:27:17.112 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:53.140574+0000", 2024-09-13T11:27:17.112 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.5669999999999995e-06 2024-09-13T11:27:17.112 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:17.112 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:17.112 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:27:17.112 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:53.140622+0000", 2024-09-13T11:27:17.112 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.8143999999999998e-05 2024-09-13T11:27:17.112 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:17.112 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:17.112 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:27:17.112 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:53.140759+0000", 2024-09-13T11:27:17.112 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000136499 2024-09-13T11:27:17.112 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:17.112 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:17.113 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 13", 2024-09-13T11:27:17.113 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:53.140887+0000", 2024-09-13T11:27:17.113 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00012796300000000001 2024-09-13T11:27:17.113 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:17.113 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:17.113 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:27:17.113 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:53.142675+0000", 2024-09-13T11:27:17.113 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001788073 2024-09-13T11:27:17.113 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:17.113 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:17.113 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:27:17.113 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:53.347407+0000", 2024-09-13T11:27:17.113 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.20473206899999999 2024-09-13T11:27:17.113 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:17.114 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:17.114 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:27:17.114 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:53.347464+0000", 2024-09-13T11:27:17.114 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.6536000000000002e-05 2024-09-13T11:27:17.114 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:17.114 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:17.114 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:27:17.114 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:53.347504+0000", 2024-09-13T11:27:17.114 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.0621999999999998e-05 2024-09-13T11:27:17.114 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:17.114 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:27:17.114 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:17.114 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:17.114 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:17.116 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:191058 3.1d e175/102 3:bd6e7029:::benchmark_data_smithi177_31116_object50755:head v 175'7612, mlcod=175'7609)", 2024-09-13T11:27:17.116 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:25:38.513527+0000", 2024-09-13T11:27:17.116 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 98.549459380000002, 2024-09-13T11:27:17.116 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.386104477, 2024-09-13T11:27:17.116 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:27:17.116 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:27:17.116 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:27:17.116 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:27:17.116 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:17.117 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:27:17.117 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:38.513527+0000", 2024-09-13T11:27:17.117 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:17.117 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:17.117 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:17.117 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:27:17.117 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:38.513527+0000", 2024-09-13T11:27:17.117 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:17.117 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:17.117 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:17.117 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:27:17.117 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:38.513530+0000", 2024-09-13T11:27:17.117 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.0029999999999999e-06 2024-09-13T11:27:17.117 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:17.118 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:17.118 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:27:17.118 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:38.513542+0000", 2024-09-13T11:27:17.118 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.1477e-05 2024-09-13T11:27:17.118 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:17.118 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:17.118 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:27:17.118 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:38.891617+0000", 2024-09-13T11:27:17.118 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.378075045 2024-09-13T11:27:17.118 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:17.118 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:17.118 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:27:17.118 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:38.891635+0000", 2024-09-13T11:27:17.118 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.7697e-05 2024-09-13T11:27:17.118 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:17.119 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:17.119 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:27:17.119 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:38.891775+0000", 2024-09-13T11:27:17.119 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000140665 2024-09-13T11:27:17.119 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:17.119 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:17.119 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:27:17.119 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:38.891833+0000", 2024-09-13T11:27:17.119 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.7769999999999997e-05 2024-09-13T11:27:17.119 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:17.119 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:17.119 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:27:17.119 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:38.899567+0000", 2024-09-13T11:27:17.119 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0077339970000000003 2024-09-13T11:27:17.120 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:17.120 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:17.120 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:27:17.120 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:38.899632+0000", 2024-09-13T11:27:17.120 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.4822999999999999e-05 2024-09-13T11:27:17.120 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:17.120 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:27:17.120 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:17.120 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:17.120 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:17.120 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:197236 3.5 3:a3127298:::benchmark_data_smithi177_31116_object56933:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e178)", 2024-09-13T11:27:17.120 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:25:44.300667+0000", 2024-09-13T11:27:17.120 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 92.762319289000004, 2024-09-13T11:27:17.120 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.37481205200000001, 2024-09-13T11:27:17.121 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:27:17.121 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:27:17.121 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:27:17.121 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:27:17.121 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:27:17.121 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:27:17.121 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 197236 2024-09-13T11:27:17.121 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:17.121 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:27:17.121 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:17.121 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:27:17.121 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:44.300667+0000", 2024-09-13T11:27:17.121 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:17.122 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:17.122 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:17.122 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:27:17.122 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:44.300667+0000", 2024-09-13T11:27:17.122 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:17.122 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:17.122 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:17.122 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:27:17.122 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:44.300670+0000", 2024-09-13T11:27:17.122 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.7259999999999998e-06 2024-09-13T11:27:17.122 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:17.122 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:17.122 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:27:17.122 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:44.300677+0000", 2024-09-13T11:27:17.123 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.1550000000000001e-06 2024-09-13T11:27:17.123 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:17.123 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:17.123 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:27:17.123 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:44.300695+0000", 2024-09-13T11:27:17.123 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.7552999999999999e-05 2024-09-13T11:27:17.123 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:17.123 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:17.123 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:27:17.123 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:44.300704+0000", 2024-09-13T11:27:17.123 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.5710000000000004e-06 2024-09-13T11:27:17.123 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:17.123 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:17.123 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:27:17.123 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:44.300762+0000", 2024-09-13T11:27:17.124 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.7351e-05 2024-09-13T11:27:17.124 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:17.124 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:17.124 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:27:17.124 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:44.300888+0000", 2024-09-13T11:27:17.124 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00012617999999999999 2024-09-13T11:27:17.124 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:17.124 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:17.124 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 13", 2024-09-13T11:27:17.124 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:44.300993+0000", 2024-09-13T11:27:17.124 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00010505600000000001 2024-09-13T11:27:17.124 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:17.124 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:17.125 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:27:17.125 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:44.302936+0000", 2024-09-13T11:27:17.125 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0019428749999999999 2024-09-13T11:27:17.125 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:17.125 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:17.125 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:27:17.125 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:44.675352+0000", 2024-09-13T11:27:17.125 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.37241603400000001 2024-09-13T11:27:17.125 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:17.125 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:17.125 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:27:17.125 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:44.675412+0000", 2024-09-13T11:27:17.125 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.9679999999999998e-05 2024-09-13T11:27:17.125 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:17.125 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:17.126 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:27:17.126 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:44.675479+0000", 2024-09-13T11:27:17.126 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.7871000000000001e-05 2024-09-13T11:27:17.126 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:17.126 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:27:17.126 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:17.126 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:17.126 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:17.126 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop_reply(client.4665.0:197236 3.5 e178/117 ondisk, result = 0)", 2024-09-13T11:27:17.126 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:25:44.304480+0000", 2024-09-13T11:27:17.126 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 92.758507022000003, 2024-09-13T11:27:17.126 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.37100864099999997, 2024-09-13T11:27:17.126 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:27:17.127 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:27:17.127 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:27:17.127 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:27:17.127 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:17.127 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:27:17.127 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:44.304480+0000", 2024-09-13T11:27:17.127 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:17.127 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:17.127 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:17.127 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:27:17.127 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:44.304480+0000", 2024-09-13T11:27:17.127 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:17.128 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:17.128 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:17.128 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:27:17.128 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:44.304481+0000", 2024-09-13T11:27:17.128 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.7349999999999999e-06 2024-09-13T11:27:17.128 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:17.128 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:17.128 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:27:17.128 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:44.304492+0000", 2024-09-13T11:27:17.129 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0394000000000001e-05 2024-09-13T11:27:17.129 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:17.129 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:17.129 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:27:17.129 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:44.675037+0000", 2024-09-13T11:27:17.129 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.37054519400000002 2024-09-13T11:27:17.129 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:17.129 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:17.129 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:27:17.129 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:44.675045+0000", 2024-09-13T11:27:17.129 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.1370000000000002e-06 2024-09-13T11:27:17.129 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:17.129 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:17.129 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:27:17.130 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:44.675334+0000", 2024-09-13T11:27:17.130 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00028848699999999998 2024-09-13T11:27:17.130 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:17.130 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:17.130 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:27:17.130 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:44.675345+0000", 2024-09-13T11:27:17.130 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.1198e-05 2024-09-13T11:27:17.130 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:17.130 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:17.130 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:27:17.131 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:44.675488+0000", 2024-09-13T11:27:17.131 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00014349600000000001 2024-09-13T11:27:17.131 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:17.131 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:27:17.131 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:17.131 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:17.131 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:17.131 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:217032 3.16 e187/88 3:6d28a27c:::benchmark_data_smithi177_31116_object76729:head v 187'8398, mlcod=187'8395)", 2024-09-13T11:27:17.131 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:25:59.836895+0000", 2024-09-13T11:27:17.131 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 77.226092136999995, 2024-09-13T11:27:17.131 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.22574986, 2024-09-13T11:27:17.131 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:27:17.131 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:27:17.131 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:27:17.132 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:27:17.132 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:17.132 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:27:17.132 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:59.836895+0000", 2024-09-13T11:27:17.132 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:17.132 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:17.132 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:17.132 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:27:17.132 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:59.836895+0000", 2024-09-13T11:27:17.132 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:17.132 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:17.132 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:17.132 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:27:17.133 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:59.836896+0000", 2024-09-13T11:27:17.133 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.432e-06 2024-09-13T11:27:17.133 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:17.133 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:17.133 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:27:17.133 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:59.836904+0000", 2024-09-13T11:27:17.133 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.9149999999999996e-06 2024-09-13T11:27:17.133 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:17.133 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:17.133 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:27:17.133 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:00.053571+0000", 2024-09-13T11:27:17.134 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.21666679799999999 2024-09-13T11:27:17.134 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:17.134 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:17.134 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:27:17.134 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:00.053582+0000", 2024-09-13T11:27:17.134 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0927999999999999e-05 2024-09-13T11:27:17.134 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:17.134 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:17.134 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:27:17.134 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:00.053811+0000", 2024-09-13T11:27:17.134 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00022964700000000001 2024-09-13T11:27:17.134 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:17.134 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:17.135 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:27:17.135 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:00.053864+0000", 2024-09-13T11:27:17.135 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.2710000000000002e-05 2024-09-13T11:27:17.135 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:17.135 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:17.135 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:27:17.135 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:00.062547+0000", 2024-09-13T11:27:17.135 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0086832650000000008 2024-09-13T11:27:17.135 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:17.135 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:17.135 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:27:17.135 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:00.062644+0000", 2024-09-13T11:27:17.135 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.7164999999999996e-05 2024-09-13T11:27:17.136 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:17.136 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:27:17.136 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:17.136 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:17.136 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:17.136 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:234789 3.5 3:a0df9e27:::benchmark_data_smithi177_31116_object94486:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e201)", 2024-09-13T11:27:17.136 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:26:17.919815+0000", 2024-09-13T11:27:17.136 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 59.143171991999999, 2024-09-13T11:27:17.136 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.34227359400000001, 2024-09-13T11:27:17.136 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:27:17.136 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:27:17.136 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:27:17.136 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:27:17.137 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:27:17.137 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:27:17.137 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 234789 2024-09-13T11:27:17.137 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:17.137 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:27:17.137 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:17.137 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:27:17.137 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:17.919815+0000", 2024-09-13T11:27:17.137 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:17.137 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:17.137 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:17.137 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:27:17.137 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:17.919815+0000", 2024-09-13T11:27:17.138 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:17.138 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:17.138 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:17.138 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:27:17.138 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:17.919817+0000", 2024-09-13T11:27:17.138 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.6369999999999999e-06 2024-09-13T11:27:17.138 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:17.138 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:17.138 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:27:17.138 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:17.919824+0000", 2024-09-13T11:27:17.138 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.9290000000000001e-06 2024-09-13T11:27:17.138 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:17.138 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:17.139 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:27:17.139 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:17.919839+0000", 2024-09-13T11:27:17.139 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.4763000000000001e-05 2024-09-13T11:27:17.139 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:17.139 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:17.139 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:27:17.139 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:17.919846+0000", 2024-09-13T11:27:17.139 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.1829999999999997e-06 2024-09-13T11:27:17.139 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:17.139 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:17.139 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:27:17.139 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:17.919894+0000", 2024-09-13T11:27:17.139 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.7475999999999997e-05 2024-09-13T11:27:17.140 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:17.140 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:17.140 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:27:17.140 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:17.919996+0000", 2024-09-13T11:27:17.140 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00010262199999999999 2024-09-13T11:27:17.140 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:17.140 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:17.140 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 13", 2024-09-13T11:27:17.140 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:17.920092+0000", 2024-09-13T11:27:17.140 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.5546999999999998e-05 2024-09-13T11:27:17.140 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:17.140 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:17.140 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:27:17.140 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:17.921865+0000", 2024-09-13T11:27:17.141 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00177271 2024-09-13T11:27:17.141 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:17.141 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:17.141 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:27:17.141 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:18.261907+0000", 2024-09-13T11:27:17.141 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.34004223300000003 2024-09-13T11:27:17.141 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:17.141 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:17.141 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:27:17.141 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:18.262002+0000", 2024-09-13T11:27:17.141 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.4745000000000002e-05 2024-09-13T11:27:17.141 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:17.141 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:17.142 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:27:17.142 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:18.262088+0000", 2024-09-13T11:27:17.142 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.6749000000000006e-05 2024-09-13T11:27:17.142 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:17.142 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:27:17.142 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:17.142 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:17.142 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:17.142 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop_reply(client.4665.0:234789 3.5 e201/117 ondisk, result = 0)", 2024-09-13T11:27:17.142 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:26:17.922332+0000", 2024-09-13T11:27:17.142 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 59.140654654000002, 2024-09-13T11:27:17.142 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.33978349299999999, 2024-09-13T11:27:17.142 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:27:17.142 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:27:17.143 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:27:17.143 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:27:17.143 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:17.143 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:27:17.143 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:17.922332+0000", 2024-09-13T11:27:17.143 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:17.143 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:17.143 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:17.143 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:27:17.143 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:17.922332+0000", 2024-09-13T11:27:17.143 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:17.143 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:17.143 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:17.144 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:27:17.144 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:17.922333+0000", 2024-09-13T11:27:17.144 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.2929999999999999e-06 2024-09-13T11:27:17.144 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:17.144 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:17.144 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:27:17.144 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:17.922337+0000", 2024-09-13T11:27:17.144 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.7469999999999999e-06 2024-09-13T11:27:17.144 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:17.144 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:17.144 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:27:17.144 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:18.261064+0000", 2024-09-13T11:27:17.144 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.33872685499999999 2024-09-13T11:27:17.144 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:17.145 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:17.145 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:27:17.145 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:18.261084+0000", 2024-09-13T11:27:17.145 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.9893000000000001e-05 2024-09-13T11:27:17.145 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:17.145 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:17.145 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:27:17.145 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:18.261880+0000", 2024-09-13T11:27:17.145 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00079579399999999995 2024-09-13T11:27:17.145 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:17.145 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:17.145 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:27:17.145 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:18.261897+0000", 2024-09-13T11:27:17.146 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.7730000000000001e-05 2024-09-13T11:27:17.146 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:17.146 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:17.146 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:27:17.146 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:18.262116+0000", 2024-09-13T11:27:17.146 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00021818099999999999 2024-09-13T11:27:17.146 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:17.146 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:27:17.146 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:17.146 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:17.146 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:17.146 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:260347 3.16 e219/88 3:6d3745c1:::benchmark_data_smithi177_31116_object120044:head v 219'9755, mlcod=219'9752)", 2024-09-13T11:27:17.146 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:26:42.471118+0000", 2024-09-13T11:27:17.147 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 34.591869060000001, 2024-09-13T11:27:17.147 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.43076131200000001, 2024-09-13T11:27:17.147 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:27:17.147 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:27:17.147 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:27:17.147 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:27:17.147 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:17.147 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:27:17.147 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:42.471118+0000", 2024-09-13T11:27:17.147 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:17.147 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:17.147 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:17.147 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:27:17.147 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:42.471118+0000", 2024-09-13T11:27:17.148 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:17.148 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:17.148 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:17.148 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:27:17.148 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:42.471122+0000", 2024-09-13T11:27:17.148 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.233e-06 2024-09-13T11:27:17.148 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:17.148 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:17.148 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:27:17.148 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:42.471139+0000", 2024-09-13T11:27:17.148 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.7442999999999999e-05 2024-09-13T11:27:17.148 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:17.148 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:17.148 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:27:17.149 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:42.893124+0000", 2024-09-13T11:27:17.149 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.42198470300000002 2024-09-13T11:27:17.149 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:17.149 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:17.149 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:27:17.149 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:42.893137+0000", 2024-09-13T11:27:17.149 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3233e-05 2024-09-13T11:27:17.149 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:17.149 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:17.149 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:27:17.149 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:42.893272+0000", 2024-09-13T11:27:17.149 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00013433499999999999 2024-09-13T11:27:17.149 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:17.150 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:17.150 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:27:17.150 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:42.893335+0000", 2024-09-13T11:27:17.150 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.3373000000000005e-05 2024-09-13T11:27:17.150 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:17.150 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:17.150 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:27:17.150 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:42.901776+0000", 2024-09-13T11:27:17.150 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0084411899999999995 2024-09-13T11:27:17.150 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:17.150 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:17.150 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:27:17.150 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:42.901879+0000", 2024-09-13T11:27:17.150 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000102802 2024-09-13T11:27:17.150 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:17.151 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:27:17.151 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:17.151 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:17.151 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:17.151 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:266456 3.5 e222/205 3:a058f943:::benchmark_data_smithi177_31116_object126153:head v 222'7143, mlcod=222'7139)", 2024-09-13T11:27:17.151 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:26:47.702034+0000", 2024-09-13T11:27:17.151 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 29.360952205, 2024-09-13T11:27:17.151 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.20573660499999999, 2024-09-13T11:27:17.151 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:27:17.151 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:27:17.151 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:27:17.151 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:27:17.152 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:17.152 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:27:17.152 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:47.702034+0000", 2024-09-13T11:27:17.152 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:17.152 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:17.152 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:17.152 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:27:17.152 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:47.702034+0000", 2024-09-13T11:27:17.152 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:17.152 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:17.152 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:17.152 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:27:17.152 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:47.702036+0000", 2024-09-13T11:27:17.152 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.265e-06 2024-09-13T11:27:17.153 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:17.153 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:17.153 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:27:17.153 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:47.702043+0000", 2024-09-13T11:27:17.153 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.9820000000000002e-06 2024-09-13T11:27:17.153 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:17.153 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:17.153 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:27:17.153 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:47.905566+0000", 2024-09-13T11:27:17.153 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.20352285000000001 2024-09-13T11:27:17.153 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:17.153 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:17.153 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:27:17.153 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:47.905575+0000", 2024-09-13T11:27:17.154 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.5570000000000001e-06 2024-09-13T11:27:17.154 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:17.154 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:17.154 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:27:17.154 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:47.905657+0000", 2024-09-13T11:27:17.154 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.2088000000000004e-05 2024-09-13T11:27:17.154 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:17.154 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:17.154 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:27:17.154 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:47.905676+0000", 2024-09-13T11:27:17.154 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.8561999999999999e-05 2024-09-13T11:27:17.154 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:17.154 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:17.154 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:27:17.155 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:47.907724+0000", 2024-09-13T11:27:17.155 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0020485239999999999 2024-09-13T11:27:17.155 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:17.155 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:17.155 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:27:17.155 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:47.907771+0000", 2024-09-13T11:27:17.155 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.6777000000000003e-05 2024-09-13T11:27:17.155 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:17.155 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:27:17.155 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:17.155 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:17.155 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:17.155 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:266464 3.5 e222/205 3:a2b966a1:::benchmark_data_smithi177_31116_object126161:head v 222'7144, mlcod=222'7140)", 2024-09-13T11:27:17.156 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:26:47.709996+0000", 2024-09-13T11:27:17.156 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 29.352991115999998, 2024-09-13T11:27:17.156 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.209382814, 2024-09-13T11:27:17.156 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:27:17.156 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:27:17.156 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:27:17.156 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:27:17.156 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:17.156 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:27:17.156 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:47.709996+0000", 2024-09-13T11:27:17.156 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:17.156 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:17.156 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:17.157 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:27:17.157 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:47.709996+0000", 2024-09-13T11:27:17.157 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:17.157 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:17.157 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:17.157 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:27:17.157 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:47.709997+0000", 2024-09-13T11:27:17.157 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.652e-06 2024-09-13T11:27:17.157 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:17.157 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:17.157 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:27:17.157 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:47.710006+0000", 2024-09-13T11:27:17.157 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.3480000000000004e-06 2024-09-13T11:27:17.158 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:17.158 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:17.158 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:27:17.158 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:47.917127+0000", 2024-09-13T11:27:17.158 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.20712151400000001 2024-09-13T11:27:17.158 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:17.158 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:17.158 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:27:17.158 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:47.917136+0000", 2024-09-13T11:27:17.158 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.3500000000000003e-06 2024-09-13T11:27:17.158 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:17.158 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:17.158 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:27:17.158 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:47.917204+0000", 2024-09-13T11:27:17.158 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.7793000000000006e-05 2024-09-13T11:27:17.159 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:17.159 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:17.159 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:27:17.159 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:47.917230+0000", 2024-09-13T11:27:17.159 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.5721e-05 2024-09-13T11:27:17.159 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:17.159 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:17.159 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:27:17.159 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:47.919340+0000", 2024-09-13T11:27:17.159 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0021102550000000001 2024-09-13T11:27:17.159 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:17.159 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:17.159 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:27:17.159 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:47.919378+0000", 2024-09-13T11:27:17.160 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.8180999999999999e-05 2024-09-13T11:27:17.160 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:17.160 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:27:17.160 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:17.160 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:17.160 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:17.160 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:266535 3.5 e222/205 3:a07a94a7:::benchmark_data_smithi177_31116_object126232:head v 222'7145, mlcod=222'7141)", 2024-09-13T11:27:17.160 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:26:47.759387+0000", 2024-09-13T11:27:17.160 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 29.303599899000002, 2024-09-13T11:27:17.160 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.23350773499999999, 2024-09-13T11:27:17.160 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:27:17.160 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:27:17.161 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:27:17.161 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:27:17.161 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:17.161 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:27:17.161 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:47.759387+0000", 2024-09-13T11:27:17.161 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:17.161 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:17.161 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:17.161 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:27:17.161 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:47.759387+0000", 2024-09-13T11:27:17.161 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:17.161 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:17.161 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:17.161 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:27:17.162 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:47.759389+0000", 2024-09-13T11:27:17.162 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.069e-06 2024-09-13T11:27:17.162 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:17.162 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:17.162 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:27:17.162 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:47.759397+0000", 2024-09-13T11:27:17.162 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.7780000000000007e-06 2024-09-13T11:27:17.162 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:17.162 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:17.162 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:27:17.162 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:47.990513+0000", 2024-09-13T11:27:17.162 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.23111675800000001 2024-09-13T11:27:17.162 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:17.163 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:17.163 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:27:17.163 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:47.990523+0000", 2024-09-13T11:27:17.163 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.3600000000000002e-06 2024-09-13T11:27:17.163 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:17.163 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:17.163 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:27:17.163 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:47.990649+0000", 2024-09-13T11:27:17.163 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00012615700000000001 2024-09-13T11:27:17.163 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:17.163 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:17.163 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:27:17.163 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:47.990671+0000", 2024-09-13T11:27:17.163 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.1934999999999999e-05 2024-09-13T11:27:17.164 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:17.164 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:17.164 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:27:17.164 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:47.992852+0000", 2024-09-13T11:27:17.164 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0021815060000000002 2024-09-13T11:27:17.164 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:17.164 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:17.164 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:27:17.164 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:47.992895+0000", 2024-09-13T11:27:17.164 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.2172000000000002e-05 2024-09-13T11:27:17.164 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:17.164 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:27:17.164 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:17.164 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:17.165 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:27:17.165 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-09-13T11:27:17.166 INFO:tasks.ceph.osd.13.smithi092.stderr:2024-09-13T11:27:17.095+0000 7f3e4da13640 -1 osd.13 245 handle_osd_map injecting map crc failure 2024-09-13T11:27:17.166 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-13T11:27:17.167 INFO:tasks.ceph.osd.7.smithi177.stderr:2024-09-13T11:27:17.165+0000 7f897b51d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26575) UID: 0 2024-09-13T11:27:17.269 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-13T11:27:17.262+0000 7f6a68775640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26307) UID: 0 2024-09-13T11:27:17.323 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-09-13T11:27:17.324 INFO:teuthology.orchestra.run.smithi092.stdout: "ops": [ 2024-09-13T11:27:17.324 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:17.324 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:278607 3.4 3:23d6f674:::benchmark_data_smithi177_31116_object138304:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e246)", 2024-09-13T11:27:17.324 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:27:17.322244+0000", 2024-09-13T11:27:17.324 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 0.0021590559999999999, 2024-09-13T11:27:17.324 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00217789, 2024-09-13T11:27:17.324 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:27:17.324 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:27:17.324 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "waiting for sub ops", 2024-09-13T11:27:17.324 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:27:17.324 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:27:17.324 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:27:17.325 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 278607 2024-09-13T11:27:17.325 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:17.325 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:27:17.325 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:17.325 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:27:17.325 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:27:17.322244+0000", 2024-09-13T11:27:17.325 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:27:17.325 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:17.325 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:17.325 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:27:17.325 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:27:17.322244+0000", 2024-09-13T11:27:17.325 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:27:17.325 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:17.325 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:17.326 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:27:17.326 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:27:17.322246+0000", 2024-09-13T11:27:17.326 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.9759999999999998e-06 2024-09-13T11:27:17.326 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:17.326 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:17.326 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:27:17.326 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:27:17.322251+0000", 2024-09-13T11:27:17.326 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.4360000000000001e-06 2024-09-13T11:27:17.326 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:17.326 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:17.326 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:27:17.326 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:27:17.322262+0000", 2024-09-13T11:27:17.326 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0417e-05 2024-09-13T11:27:17.326 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:17.327 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:17.327 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:27:17.327 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:27:17.322269+0000", 2024-09-13T11:27:17.327 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.3649999999999998e-06 2024-09-13T11:27:17.327 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:17.327 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:17.327 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:27:17.327 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:27:17.322319+0000", 2024-09-13T11:27:17.327 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.9743000000000001e-05 2024-09-13T11:27:17.327 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:17.327 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:17.327 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:27:17.327 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:27:17.322433+0000", 2024-09-13T11:27:17.327 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000113791 2024-09-13T11:27:17.328 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:17.328 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:17.328 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 12", 2024-09-13T11:27:17.328 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:27:17.322528+0000", 2024-09-13T11:27:17.328 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.5402000000000006e-05 2024-09-13T11:27:17.328 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:27:17.328 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:27:17.328 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:27:17.328 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:17.328 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:17.328 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "MOSDPGRecoveryDelete(3.31 e246,227 3:8fca52ef:::benchmark_data_smithi177_31116_object85855:head,195'8472)", 2024-09-13T11:27:17.328 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:27:17.323759+0000", 2024-09-13T11:27:17.328 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 0.00064382399999999998, 2024-09-13T11:27:17.328 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00070429900000000003, 2024-09-13T11:27:17.329 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:27:17.329 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:27:17.329 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "queued for pg", 2024-09-13T11:27:17.329 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:27:17.329 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:17.329 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:27:17.329 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:27:17.323759+0000", 2024-09-13T11:27:17.329 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:27:17.329 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:17.329 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:17.329 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:27:17.329 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:27:17.323759+0000", 2024-09-13T11:27:17.329 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:27:17.329 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:17.329 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:17.330 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:27:17.330 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:27:17.323761+0000", 2024-09-13T11:27:17.330 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.4160000000000001e-06 2024-09-13T11:27:17.330 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:17.330 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:17.330 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:27:17.330 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:27:17.323767+0000", 2024-09-13T11:27:17.330 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.285e-06 2024-09-13T11:27:17.330 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:17.330 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:17.330 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:27:17.330 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:27:17.323798+0000", 2024-09-13T11:27:17.330 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.1062000000000002e-05 2024-09-13T11:27:17.330 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:17.331 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:17.331 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:27:17.331 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:27:17.323806+0000", 2024-09-13T11:27:17.331 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.9650000000000008e-06 2024-09-13T11:27:17.331 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:27:17.331 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:27:17.331 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:27:17.331 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:27:17.331 INFO:teuthology.orchestra.run.smithi092.stdout: ], 2024-09-13T11:27:17.331 INFO:teuthology.orchestra.run.smithi092.stdout: "num_ops": 2 2024-09-13T11:27:17.331 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-09-13T11:27:17.350 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-13T11:27:17.369 INFO:tasks.ceph.osd.12.smithi001.stderr:2024-09-13T11:27:17.366+0000 7f892c6e6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26314) UID: 0 2024-09-13T11:27:17.372 INFO:teuthology.orchestra.run.smithi162.stderr:Exporting 3.32 info 3.32( v 237'9929 lc 171'7167 (0'0,237'9929] local-lis/les=0/0 n=44 ec=224/30 lis/c=0/122 les/c/f=0/160/0 sis=227) 2024-09-13T11:27:17.411 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4c36832c:::benchmark_data_smithi177_31116_object139842:head# 2024-09-13T11:27:17.416 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4c8b6c2c:::benchmark_data_smithi177_31116_object138330:head# 2024-09-13T11:27:17.417 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4ca618c4:::benchmark_data_smithi177_31116_object138601:head# 2024-09-13T11:27:17.417 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4cce3ca4:::benchmark_data_smithi177_31116_object138517:head# 2024-09-13T11:27:17.418 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4cdefb85:::benchmark_data_smithi177_31116_object138131:head# 2024-09-13T11:27:17.419 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4cdfe228:::benchmark_data_smithi177_31116_object138507:head# 2024-09-13T11:27:17.419 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4d09234a:::benchmark_data_smithi177_31116_object138495:head# 2024-09-13T11:27:17.422 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4d45afc8:::benchmark_data_smithi177_31116_object139901:head# 2024-09-13T11:27:17.423 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4d70efd9:::benchmark_data_smithi177_31116_object139003:head# 2024-09-13T11:27:17.424 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4d79a7be:::benchmark_data_smithi177_31116_object138000:head# 2024-09-13T11:27:17.427 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4da9e252:::benchmark_data_smithi177_31116_object139857:head# 2024-09-13T11:27:17.430 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4dd0f18d:::benchmark_data_smithi177_31116_object137806:head# 2024-09-13T11:27:17.433 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4dd1c71b:::benchmark_data_smithi177_31116_object138813:head# 2024-09-13T11:27:17.435 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4dd68623:::benchmark_data_smithi177_31116_object140295:head# 2024-09-13T11:27:17.436 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4de1fb58:::benchmark_data_smithi177_31116_object140223:head# 2024-09-13T11:27:17.438 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4df30a72:::benchmark_data_smithi177_31116_object139649:head# 2024-09-13T11:27:17.439 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4dfff984:::benchmark_data_smithi177_31116_object138372:head# 2024-09-13T11:27:17.439 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4e0caca5:::benchmark_data_smithi177_31116_object138189:head# 2024-09-13T11:27:17.442 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4e106809:::benchmark_data_smithi177_31116_object139020:head# 2024-09-13T11:27:17.442 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4e2dac0e:::benchmark_data_smithi177_31116_object139986:head# 2024-09-13T11:27:17.443 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4e2e55c4:::benchmark_data_smithi177_31116_object139822:head# 2024-09-13T11:27:17.443 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4e487caf:::benchmark_data_smithi177_31116_object137938:head# 2024-09-13T11:27:17.444 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4e50d499:::benchmark_data_smithi177_31116_object138038:head# 2024-09-13T11:27:17.444 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4e5c15f0:::benchmark_data_smithi177_31116_object139138:head# 2024-09-13T11:27:17.445 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4e8f93b7:::benchmark_data_smithi177_31116_object138120:head# 2024-09-13T11:27:17.445 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4ea0254f:::benchmark_data_smithi177_31116_object139194:head# 2024-09-13T11:27:17.447 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4ea73683:::benchmark_data_smithi177_31116_object139004:head# 2024-09-13T11:27:17.447 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4ebae361:::benchmark_data_smithi177_31116_object138102:head# 2024-09-13T11:27:17.448 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4f0452aa:::benchmark_data_smithi177_31116_object139766:head# 2024-09-13T11:27:17.449 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4f08b27e:::benchmark_data_smithi177_31116_object138488:head# 2024-09-13T11:27:17.449 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4f326442:::benchmark_data_smithi177_31116_object137928:head# 2024-09-13T11:27:17.450 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4f637648:::benchmark_data_smithi177_31116_object139045:head# 2024-09-13T11:27:17.450 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4f915478:::benchmark_data_smithi177_31116_object138703:head# 2024-09-13T11:27:17.451 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4f9cd3bd:::benchmark_data_smithi177_31116_object139079:head# 2024-09-13T11:27:17.451 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4fd2ea6a:::benchmark_data_smithi177_31116_object138815:head# 2024-09-13T11:27:17.473 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-13T11:27:17.470+0000 7f6a68775640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26307) UID: 0 2024-09-13T11:27:17.488 INFO:teuthology.orchestra.run.smithi162.stderr:Read #3:4fd3b5f4:::benchmark_data_smithi177_31116_object139197:head# 2024-09-13T11:27:17.490 INFO:teuthology.orchestra.run.smithi162.stderr:Export successful 2024-09-13T11:27:17.501 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-09-13T11:27:17.502 INFO:teuthology.orchestra.run.smithi092.stdout: "ops": [], 2024-09-13T11:27:17.502 INFO:teuthology.orchestra.run.smithi092.stdout: "complaint_time": 30, 2024-09-13T11:27:17.502 INFO:teuthology.orchestra.run.smithi092.stdout: "num_blocked_ops": 0 2024-09-13T11:27:17.502 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-09-13T11:27:17.529 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-13T11:27:17.570 INFO:tasks.ceph.osd.2.smithi162.stderr:2024-09-13T11:27:17.567+0000 7fe92b35a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26313) UID: 0 2024-09-13T11:27:17.671 INFO:tasks.ceph.osd.1.smithi092.stderr:2024-09-13T11:27:17.667+0000 7fe7789c5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33718) UID: 0 2024-09-13T11:27:17.706 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-09-13T11:27:17.706 INFO:teuthology.orchestra.run.smithi092.stdout: "size": 20, 2024-09-13T11:27:17.706 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 600, 2024-09-13T11:27:17.706 INFO:teuthology.orchestra.run.smithi092.stdout: "ops": [ 2024-09-13T11:27:17.706 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:17.706 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:3 3.4 3.125e09e4 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e40)", 2024-09-13T11:27:17.707 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:46.304641+0000", 2024-09-13T11:27:17.707 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 271.40023101200001, 2024-09-13T11:27:17.707 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.374372119, 2024-09-13T11:27:17.707 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:27:17.707 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:27:17.707 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "reached pg", 2024-09-13T11:27:17.707 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:27:17.707 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:27:17.707 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:27:17.707 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 3 2024-09-13T11:27:17.707 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:17.708 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:27:17.708 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:17.708 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:27:17.708 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.304641+0000", 2024-09-13T11:27:17.708 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:27:17.708 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:17.708 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:17.708 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:27:17.708 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.304641+0000", 2024-09-13T11:27:17.708 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:27:17.708 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:17.709 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:17.709 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:27:17.709 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.304644+0000", 2024-09-13T11:27:17.709 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.5130000000000001e-06 2024-09-13T11:27:17.709 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:17.709 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:17.709 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:27:17.709 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305058+0000", 2024-09-13T11:27:17.709 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000414277 2024-09-13T11:27:17.709 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:17.709 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:17.710 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:27:17.710 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305124+0000", 2024-09-13T11:27:17.710 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.5813000000000002e-05 2024-09-13T11:27:17.710 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:17.710 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:17.710 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:27:17.710 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305156+0000", 2024-09-13T11:27:17.710 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.1489000000000003e-05 2024-09-13T11:27:17.710 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:17.710 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:17.710 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:27:17.711 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.678939+0000", 2024-09-13T11:27:17.711 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.37378339999999999 2024-09-13T11:27:17.711 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:17.711 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:17.711 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:27:17.711 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.679013+0000", 2024-09-13T11:27:17.711 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.3627000000000005e-05 2024-09-13T11:27:17.711 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:27:17.711 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:27:17.711 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:27:17.711 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:17.711 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:17.712 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:29 3.4 3.b7bbfbe4 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-09-13T11:27:17.712 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:46.305215+0000", 2024-09-13T11:27:17.712 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 271.399656709, 2024-09-13T11:27:17.712 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.37387964600000001, 2024-09-13T11:27:17.712 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:27:17.712 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:27:17.712 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "reached pg", 2024-09-13T11:27:17.712 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:27:17.712 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:27:17.712 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:27:17.712 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 29 2024-09-13T11:27:17.712 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:17.713 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:27:17.713 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:17.713 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:27:17.713 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305215+0000", 2024-09-13T11:27:17.713 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:27:17.713 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:17.713 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:17.713 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:27:17.713 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305215+0000", 2024-09-13T11:27:17.713 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:27:17.713 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:17.713 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:17.714 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:27:17.714 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305218+0000", 2024-09-13T11:27:17.714 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.7030000000000002e-06 2024-09-13T11:27:17.714 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:17.714 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:17.714 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:27:17.714 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305366+0000", 2024-09-13T11:27:17.714 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00014825400000000001 2024-09-13T11:27:17.714 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:17.714 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:17.714 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:27:17.714 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305386+0000", 2024-09-13T11:27:17.715 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.9689000000000001e-05 2024-09-13T11:27:17.715 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:17.715 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:17.715 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:27:17.715 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305403+0000", 2024-09-13T11:27:17.715 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.7600000000000001e-05 2024-09-13T11:27:17.715 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:17.715 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:17.715 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:27:17.715 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.679067+0000", 2024-09-13T11:27:17.715 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.37366339100000001 2024-09-13T11:27:17.715 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:17.716 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:17.716 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:27:17.716 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.679095+0000", 2024-09-13T11:27:17.716 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.8008999999999999e-05 2024-09-13T11:27:17.716 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:27:17.716 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:27:17.716 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:27:17.716 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:17.716 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:17.716 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:30 3.4 3.547ec0f4 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-09-13T11:27:17.716 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:46.305453+0000", 2024-09-13T11:27:17.717 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 271.39941858499998, 2024-09-13T11:27:17.717 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.373764914, 2024-09-13T11:27:17.717 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:27:17.717 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:27:17.717 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "reached pg", 2024-09-13T11:27:17.717 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:27:17.717 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:27:17.717 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:27:17.717 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 30 2024-09-13T11:27:17.717 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:17.717 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:27:17.718 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:17.718 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:27:17.718 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305453+0000", 2024-09-13T11:27:17.718 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:27:17.718 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:17.718 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:17.718 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:27:17.718 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305453+0000", 2024-09-13T11:27:17.719 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:27:17.719 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:17.719 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:17.719 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:27:17.719 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305456+0000", 2024-09-13T11:27:17.719 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.9560000000000002e-06 2024-09-13T11:27:17.719 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:17.719 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:17.719 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:27:17.719 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305597+0000", 2024-09-13T11:27:17.719 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00014122700000000001 2024-09-13T11:27:17.719 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:17.720 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:17.720 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:27:17.720 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305615+0000", 2024-09-13T11:27:17.720 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.8139999999999999e-05 2024-09-13T11:27:17.720 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:17.720 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:17.720 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:27:17.720 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305632+0000", 2024-09-13T11:27:17.720 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.6101999999999999e-05 2024-09-13T11:27:17.720 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:17.720 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:17.720 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:27:17.721 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.679195+0000", 2024-09-13T11:27:17.721 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.373563486 2024-09-13T11:27:17.721 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:17.721 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:17.721 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:27:17.722 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.679218+0000", 2024-09-13T11:27:17.722 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.3003e-05 2024-09-13T11:27:17.722 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:27:17.722 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:27:17.722 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:27:17.722 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:17.722 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:17.722 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:70 3.4 3.1404a604 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-09-13T11:27:17.722 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:46.305665+0000", 2024-09-13T11:27:17.723 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 271.39920643699998, 2024-09-13T11:27:17.723 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.37365550199999997, 2024-09-13T11:27:17.723 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:27:17.723 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:27:17.723 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "reached pg", 2024-09-13T11:27:17.723 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:27:17.723 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:27:17.723 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:27:17.724 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 70 2024-09-13T11:27:17.724 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:17.724 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:27:17.724 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:17.724 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:27:17.724 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305665+0000", 2024-09-13T11:27:17.724 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:27:17.725 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:17.725 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:17.725 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:27:17.725 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305665+0000", 2024-09-13T11:27:17.725 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:27:17.725 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:17.725 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:17.725 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:27:17.726 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305668+0000", 2024-09-13T11:27:17.726 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.5890000000000001e-06 2024-09-13T11:27:17.726 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:17.726 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:17.726 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:27:17.726 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305791+0000", 2024-09-13T11:27:17.726 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000123545 2024-09-13T11:27:17.726 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:17.726 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:17.726 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:27:17.726 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305808+0000", 2024-09-13T11:27:17.727 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.668e-05 2024-09-13T11:27:17.727 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:17.727 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:17.727 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:27:17.727 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305824+0000", 2024-09-13T11:27:17.727 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.6385e-05 2024-09-13T11:27:17.727 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:17.727 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:17.727 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:27:17.728 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.679305+0000", 2024-09-13T11:27:17.728 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.37348059900000002 2024-09-13T11:27:17.728 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:17.728 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:17.728 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:27:17.728 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.679321+0000", 2024-09-13T11:27:17.728 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.5704000000000001e-05 2024-09-13T11:27:17.728 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:27:17.728 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:27:17.728 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:27:17.729 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:17.729 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:17.729 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:25408 3.4 3:294e23bd:::benchmark_data_smithi177_31116_object25407:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e56)", 2024-09-13T11:27:17.729 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:04.495455+0000", 2024-09-13T11:27:17.729 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 253.209416785, 2024-09-13T11:27:17.729 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.29897367600000002, 2024-09-13T11:27:17.729 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:27:17.729 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:27:17.729 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:27:17.729 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:27:17.730 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:27:17.730 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:27:17.730 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 25408 2024-09-13T11:27:17.730 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:17.730 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:27:17.730 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:17.730 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:27:17.730 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.495455+0000", 2024-09-13T11:27:17.730 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:27:17.730 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:17.730 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:17.730 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:27:17.730 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.495455+0000", 2024-09-13T11:27:17.731 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:27:17.731 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:17.731 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:17.731 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:27:17.731 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.495459+0000", 2024-09-13T11:27:17.731 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.5630000000000002e-06 2024-09-13T11:27:17.731 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:17.731 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:17.731 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:27:17.731 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.495545+0000", 2024-09-13T11:27:17.731 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.5704000000000005e-05 2024-09-13T11:27:17.731 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:17.731 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:17.731 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:27:17.732 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.495581+0000", 2024-09-13T11:27:17.732 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.5689000000000003e-05 2024-09-13T11:27:17.732 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:17.732 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:17.732 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:27:17.732 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.495597+0000", 2024-09-13T11:27:17.732 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.6053e-05 2024-09-13T11:27:17.732 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:17.732 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:17.732 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:27:17.732 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.495721+0000", 2024-09-13T11:27:17.732 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00012379099999999999 2024-09-13T11:27:17.732 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:17.733 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:17.733 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:27:17.733 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.495940+0000", 2024-09-13T11:27:17.733 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00021967699999999999 2024-09-13T11:27:17.733 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:17.733 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:17.733 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 12", 2024-09-13T11:27:17.733 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.496162+0000", 2024-09-13T11:27:17.733 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000221917 2024-09-13T11:27:17.733 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:17.733 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:17.733 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:27:17.734 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.498699+0000", 2024-09-13T11:27:17.734 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0025366690000000001 2024-09-13T11:27:17.734 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:17.734 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:17.734 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:27:17.734 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.794139+0000", 2024-09-13T11:27:17.734 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.295440025 2024-09-13T11:27:17.734 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:17.734 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:17.734 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:27:17.734 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.794306+0000", 2024-09-13T11:27:17.734 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00016689799999999999 2024-09-13T11:27:17.734 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:17.734 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:17.735 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:27:17.735 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:04.794429+0000", 2024-09-13T11:27:17.735 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00012269000000000001 2024-09-13T11:27:17.735 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:27:17.735 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:27:17.735 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:27:17.735 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:17.735 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:17.735 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "MOSDPGPushReply(3.14 75/70 [PushReplyOp(3:2f14928a:::benchmark_data_smithi177_31116_object15123:head)])", 2024-09-13T11:27:17.735 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:31.569662+0000", 2024-09-13T11:27:17.735 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 226.13521001300001, 2024-09-13T11:27:17.735 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.35759148299999999, 2024-09-13T11:27:17.735 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:27:17.736 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:27:17.736 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "reached pg", 2024-09-13T11:27:17.736 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:27:17.736 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:17.736 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:27:17.736 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:31.569662+0000", 2024-09-13T11:27:17.736 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:27:17.736 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:17.736 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:17.736 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:27:17.736 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:31.569662+0000", 2024-09-13T11:27:17.736 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:27:17.736 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:17.736 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:17.737 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:27:17.737 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:31.569663+0000", 2024-09-13T11:27:17.737 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.333e-06 2024-09-13T11:27:17.737 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:17.737 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:17.737 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:27:17.737 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:31.569667+0000", 2024-09-13T11:27:17.737 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.8929999999999998e-06 2024-09-13T11:27:17.737 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:17.737 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:17.737 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:27:17.737 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:31.923221+0000", 2024-09-13T11:27:17.737 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.35355413699999999 2024-09-13T11:27:17.737 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:17.738 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:17.738 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:27:17.738 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:31.923230+0000", 2024-09-13T11:27:17.738 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.6829999999999997e-06 2024-09-13T11:27:17.738 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:17.738 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:17.738 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:27:17.738 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:31.923807+0000", 2024-09-13T11:27:17.738 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00057686800000000002 2024-09-13T11:27:17.738 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:17.738 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:17.738 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:27:17.738 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:31.927253+0000", 2024-09-13T11:27:17.738 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.003446569 2024-09-13T11:27:17.739 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:27:17.739 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:27:17.739 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:27:17.739 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:17.739 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:17.739 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "MOSDPGPushReply(3.18 85/70 [PushReplyOp(3:1e410119:::benchmark_data_smithi177_31116_object35700:head)])", 2024-09-13T11:27:17.739 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:42.216614+0000", 2024-09-13T11:27:17.739 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 215.48825766900001, 2024-09-13T11:27:17.739 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.54260090699999997, 2024-09-13T11:27:17.739 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:27:17.739 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:27:17.739 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "reached pg", 2024-09-13T11:27:17.739 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:27:17.739 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:17.740 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:27:17.740 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.216614+0000", 2024-09-13T11:27:17.740 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:27:17.740 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:17.740 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:17.740 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:27:17.740 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.216614+0000", 2024-09-13T11:27:17.740 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:27:17.740 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:17.740 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:17.740 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:27:17.740 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.216615+0000", 2024-09-13T11:27:17.740 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.0200000000000001e-07 2024-09-13T11:27:17.740 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:17.741 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:17.741 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:27:17.741 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.216619+0000", 2024-09-13T11:27:17.741 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.805e-06 2024-09-13T11:27:17.741 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:17.741 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:17.741 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:27:17.741 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.757476+0000", 2024-09-13T11:27:17.741 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.54085762000000004 2024-09-13T11:27:17.741 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:17.741 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:17.741 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:27:17.741 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.757498+0000", 2024-09-13T11:27:17.742 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.1936000000000001e-05 2024-09-13T11:27:17.742 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:17.742 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:17.742 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:27:17.742 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.757678+0000", 2024-09-13T11:27:17.742 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00018006599999999999 2024-09-13T11:27:17.742 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:17.742 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:17.742 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:27:17.742 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.759215+0000", 2024-09-13T11:27:17.742 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0015366780000000001 2024-09-13T11:27:17.742 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:27:17.742 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:27:17.743 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:27:17.743 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:17.743 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:17.743 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "MOSDPGPushReply(3.1b 85/78 [PushReplyOp(3:dabda1cc:::benchmark_data_smithi177_31116_object12988:head)])", 2024-09-13T11:27:17.743 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:42.485171+0000", 2024-09-13T11:27:17.743 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 215.21970076100001, 2024-09-13T11:27:17.743 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.308420427, 2024-09-13T11:27:17.743 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:27:17.743 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:27:17.743 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "reached pg", 2024-09-13T11:27:17.743 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:27:17.743 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:17.743 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:27:17.743 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.485171+0000", 2024-09-13T11:27:17.744 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:27:17.744 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:17.744 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:17.744 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:27:17.744 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.485171+0000", 2024-09-13T11:27:17.744 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:27:17.744 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:17.744 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:17.744 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:27:17.744 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.485173+0000", 2024-09-13T11:27:17.744 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.1569999999999998e-06 2024-09-13T11:27:17.744 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:17.744 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:17.745 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:27:17.745 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.485182+0000", 2024-09-13T11:27:17.745 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.6759999999999996e-06 2024-09-13T11:27:17.745 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:17.745 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:17.745 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:27:17.745 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.791336+0000", 2024-09-13T11:27:17.745 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.30615446499999999 2024-09-13T11:27:17.745 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:17.745 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:17.745 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:27:17.745 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.791369+0000", 2024-09-13T11:27:17.745 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.3155999999999999e-05 2024-09-13T11:27:17.746 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:17.746 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:17.746 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:27:17.746 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.791472+0000", 2024-09-13T11:27:17.746 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00010247900000000001 2024-09-13T11:27:17.746 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:17.746 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:17.746 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:27:17.746 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.793591+0000", 2024-09-13T11:27:17.746 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.002119494 2024-09-13T11:27:17.746 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:27:17.746 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:27:17.746 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:27:17.746 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:17.747 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:17.747 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:64066 3.1b 3:dd84e0ec:::benchmark_data_smithi177_31116_object64065:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89)", 2024-09-13T11:27:17.747 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:46.633608+0000", 2024-09-13T11:27:17.747 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 211.07126332199999, 2024-09-13T11:27:17.747 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.32564315399999999, 2024-09-13T11:27:17.747 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:27:17.747 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:27:17.747 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:27:17.747 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:27:17.747 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:27:17.747 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:27:17.747 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 64066 2024-09-13T11:27:17.747 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:17.747 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:27:17.748 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:17.748 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:27:17.748 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.633608+0000", 2024-09-13T11:27:17.748 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:27:17.748 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:17.748 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:17.748 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:27:17.748 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.633608+0000", 2024-09-13T11:27:17.748 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:27:17.748 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:17.748 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:17.748 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:27:17.748 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.633612+0000", 2024-09-13T11:27:17.748 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.856e-06 2024-09-13T11:27:17.749 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:17.749 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:17.749 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:27:17.749 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.633683+0000", 2024-09-13T11:27:17.749 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.1000999999999993e-05 2024-09-13T11:27:17.749 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:17.749 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:17.749 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:27:17.749 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.633711+0000", 2024-09-13T11:27:17.749 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.8204999999999998e-05 2024-09-13T11:27:17.749 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:17.749 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:17.749 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:27:17.749 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.633732+0000", 2024-09-13T11:27:17.750 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.0662999999999999e-05 2024-09-13T11:27:17.750 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:17.750 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:17.750 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:27:17.750 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.635573+0000", 2024-09-13T11:27:17.750 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.001841365 2024-09-13T11:27:17.750 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:17.750 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:17.750 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:27:17.750 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.635995+0000", 2024-09-13T11:27:17.750 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00042113699999999998 2024-09-13T11:27:17.750 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:17.750 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:17.751 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 12,15", 2024-09-13T11:27:17.751 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.636475+0000", 2024-09-13T11:27:17.751 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00048067699999999999 2024-09-13T11:27:17.751 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:17.751 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:17.751 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:27:17.751 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.640247+0000", 2024-09-13T11:27:17.751 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0037716450000000001 2024-09-13T11:27:17.751 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:17.751 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:17.751 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:27:17.751 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.649308+0000", 2024-09-13T11:27:17.751 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0090613240000000008 2024-09-13T11:27:17.751 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:17.752 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:17.752 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:27:17.752 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.956912+0000", 2024-09-13T11:27:17.752 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.30760338500000001 2024-09-13T11:27:17.752 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:17.752 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:17.752 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:27:17.752 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.957156+0000", 2024-09-13T11:27:17.752 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000244031 2024-09-13T11:27:17.752 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:17.752 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:17.752 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:27:17.752 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.959252+0000", 2024-09-13T11:27:17.752 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0020958650000000001 2024-09-13T11:27:17.753 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:27:17.753 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:27:17.753 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:27:17.753 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:17.753 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:17.753 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop_reply(client.4665.0:64066 3.1b e89/78 ondisk, result = 0)", 2024-09-13T11:27:17.753 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:46.651596+0000", 2024-09-13T11:27:17.753 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 211.05327549099999, 2024-09-13T11:27:17.753 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.30767612599999999, 2024-09-13T11:27:17.753 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:27:17.753 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:27:17.753 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-09-13T11:27:17.753 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:27:17.753 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:17.754 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:27:17.754 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.651596+0000", 2024-09-13T11:27:17.754 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:27:17.754 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:17.754 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:17.754 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:27:17.754 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.651596+0000", 2024-09-13T11:27:17.754 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:27:17.754 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:17.754 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:17.754 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:27:17.754 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.651600+0000", 2024-09-13T11:27:17.754 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.6969999999999999e-06 2024-09-13T11:27:17.754 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:17.755 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:17.755 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:27:17.755 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.651611+0000", 2024-09-13T11:27:17.755 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.1516e-05 2024-09-13T11:27:17.755 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:17.755 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:17.755 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:27:17.755 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.948118+0000", 2024-09-13T11:27:17.755 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.29650663500000002 2024-09-13T11:27:17.755 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:17.755 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:17.755 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:27:17.755 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.948137+0000", 2024-09-13T11:27:17.755 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.8529000000000002e-05 2024-09-13T11:27:17.756 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:17.756 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:17.756 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:27:17.756 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.956870+0000", 2024-09-13T11:27:17.756 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0087336380000000002 2024-09-13T11:27:17.756 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:17.756 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:17.756 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:27:17.756 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.956893+0000", 2024-09-13T11:27:17.756 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.317e-05 2024-09-13T11:27:17.756 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:17.756 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:17.756 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:27:17.757 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.959272+0000", 2024-09-13T11:27:17.757 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0023789409999999999 2024-09-13T11:27:17.757 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:27:17.757 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:27:17.757 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:27:17.757 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:17.757 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:17.757 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "MOSDPGPushReply(3.1b 91/78 [PushReplyOp(3:d9ac6889:::benchmark_data_smithi177_31116_object22254:head)])", 2024-09-13T11:27:17.757 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:48.392729+0000", 2024-09-13T11:27:17.757 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 209.31214257600001, 2024-09-13T11:27:17.757 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.30924993099999998, 2024-09-13T11:27:17.757 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:27:17.757 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:27:17.757 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "reached pg", 2024-09-13T11:27:17.758 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:27:17.758 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:17.758 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:27:17.758 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:48.392729+0000", 2024-09-13T11:27:17.758 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:27:17.758 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:17.758 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:17.758 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:27:17.758 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:48.392729+0000", 2024-09-13T11:27:17.758 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:27:17.758 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:17.758 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:17.758 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:27:17.758 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:48.392730+0000", 2024-09-13T11:27:17.759 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.336e-06 2024-09-13T11:27:17.759 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:17.759 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:17.759 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:27:17.759 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:48.392739+0000", 2024-09-13T11:27:17.759 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.8580000000000006e-06 2024-09-13T11:27:17.759 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:17.759 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:17.759 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:27:17.759 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:48.700180+0000", 2024-09-13T11:27:17.759 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.30744035600000003 2024-09-13T11:27:17.759 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:17.759 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:17.759 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:27:17.760 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:48.700197+0000", 2024-09-13T11:27:17.760 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.7649999999999999e-05 2024-09-13T11:27:17.760 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:17.760 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:17.760 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:27:17.760 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:48.700398+0000", 2024-09-13T11:27:17.760 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000201113 2024-09-13T11:27:17.760 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:17.760 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:17.760 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:27:17.760 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:48.701979+0000", 2024-09-13T11:27:17.760 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.001580618 2024-09-13T11:27:17.760 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:27:17.761 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:27:17.761 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:27:17.761 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:17.761 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:17.761 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop(client.4665.0:83193 3.8 e106/67 3:10066d83:::benchmark_data_smithi177_31116_object83192:head v 106'3925, mlcod=106'3922)", 2024-09-13T11:27:17.761 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:24:04.285068+0000", 2024-09-13T11:27:17.761 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 193.41980382099999, 2024-09-13T11:27:17.761 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.64647716600000005, 2024-09-13T11:27:17.761 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:27:17.761 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:27:17.761 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:27:17.761 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:27:17.761 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:17.762 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:27:17.762 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:04.285068+0000", 2024-09-13T11:27:17.762 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:27:17.762 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:17.762 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:17.762 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:27:17.762 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:04.285068+0000", 2024-09-13T11:27:17.762 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:27:17.762 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:17.762 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:17.762 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:27:17.762 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:04.285071+0000", 2024-09-13T11:27:17.762 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.729e-06 2024-09-13T11:27:17.762 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:17.763 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:17.763 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:27:17.763 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:04.285215+0000", 2024-09-13T11:27:17.763 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00014410000000000001 2024-09-13T11:27:17.763 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:17.763 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:17.763 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:27:17.763 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:04.928221+0000", 2024-09-13T11:27:17.763 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.64300653299999999 2024-09-13T11:27:17.763 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:17.763 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:17.763 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:27:17.763 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:04.928240+0000", 2024-09-13T11:27:17.763 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.8678999999999999e-05 2024-09-13T11:27:17.764 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:17.764 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:17.764 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:27:17.764 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:04.928388+0000", 2024-09-13T11:27:17.764 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00014809 2024-09-13T11:27:17.764 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:17.764 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:17.764 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:27:17.764 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:04.928428+0000", 2024-09-13T11:27:17.764 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.9776e-05 2024-09-13T11:27:17.764 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:17.764 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:17.764 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:27:17.765 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:04.931463+0000", 2024-09-13T11:27:17.765 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.003035168 2024-09-13T11:27:17.765 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:17.765 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:17.765 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:27:17.765 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:04.931545+0000", 2024-09-13T11:27:17.765 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.2090999999999997e-05 2024-09-13T11:27:17.765 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:27:17.765 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:27:17.765 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:27:17.765 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:17.765 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:17.765 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop(client.4665.0:83239 3.e e106/75 3:74547f17:::benchmark_data_smithi177_31116_object83238:head v 106'4391, mlcod=106'4388)", 2024-09-13T11:27:17.765 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:24:04.581196+0000", 2024-09-13T11:27:17.766 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 193.12367531500001, 2024-09-13T11:27:17.766 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.33617203600000001, 2024-09-13T11:27:17.766 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:27:17.766 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:27:17.766 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:27:17.766 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:27:17.766 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:17.766 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:27:17.766 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:04.581196+0000", 2024-09-13T11:27:17.766 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:27:17.766 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:17.766 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:17.766 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:27:17.766 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:04.581196+0000", 2024-09-13T11:27:17.767 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:27:17.767 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:17.767 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:17.767 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:27:17.767 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:04.581200+0000", 2024-09-13T11:27:17.767 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.19e-06 2024-09-13T11:27:17.767 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:17.767 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:17.767 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:27:17.767 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:04.581257+0000", 2024-09-13T11:27:17.767 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.7531999999999997e-05 2024-09-13T11:27:17.767 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:17.767 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:17.767 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:27:17.768 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:04.914011+0000", 2024-09-13T11:27:17.768 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.33275413700000001 2024-09-13T11:27:17.768 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:17.768 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:17.768 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:27:17.768 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:04.914028+0000", 2024-09-13T11:27:17.768 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.7082999999999999e-05 2024-09-13T11:27:17.768 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:17.768 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:17.768 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:27:17.768 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:04.914151+0000", 2024-09-13T11:27:17.768 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00012283300000000001 2024-09-13T11:27:17.768 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:17.768 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:17.769 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:27:17.769 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:04.914185+0000", 2024-09-13T11:27:17.769 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.4255999999999999e-05 2024-09-13T11:27:17.769 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:17.769 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:17.769 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:27:17.769 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:04.917300+0000", 2024-09-13T11:27:17.769 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0031141599999999999 2024-09-13T11:27:17.769 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:17.769 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:17.769 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:27:17.770 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:04.917368+0000", 2024-09-13T11:27:17.770 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.8844999999999996e-05 2024-09-13T11:27:17.770 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:27:17.770 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:27:17.770 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:27:17.770 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:17.770 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:17.770 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:131687 3.28 3:14f65078:::benchmark_data_smithi177_31116_object131686:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e135)", 2024-09-13T11:27:17.770 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:24:48.925565+0000", 2024-09-13T11:27:17.770 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 148.77930631800001, 2024-09-13T11:27:17.770 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.41421604099999998, 2024-09-13T11:27:17.770 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:27:17.771 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:27:17.771 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:27:17.771 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:27:17.771 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:27:17.771 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:27:17.771 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 131687 2024-09-13T11:27:17.771 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:17.771 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:27:17.771 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:17.771 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:27:17.771 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:48.925565+0000", 2024-09-13T11:27:17.771 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:27:17.771 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:17.771 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:17.772 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:27:17.772 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:48.925565+0000", 2024-09-13T11:27:17.772 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:27:17.772 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:17.772 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:17.772 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:27:17.772 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:48.925568+0000", 2024-09-13T11:27:17.772 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.441e-06 2024-09-13T11:27:17.772 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:17.772 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:17.772 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:27:17.772 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:48.925617+0000", 2024-09-13T11:27:17.772 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.8739000000000002e-05 2024-09-13T11:27:17.772 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:17.773 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:17.773 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:27:17.773 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:48.925631+0000", 2024-09-13T11:27:17.773 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.4463e-05 2024-09-13T11:27:17.773 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:17.773 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:17.773 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:27:17.773 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:48.925638+0000", 2024-09-13T11:27:17.773 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.2119999999999997e-06 2024-09-13T11:27:17.773 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:17.773 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:17.773 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:27:17.773 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:48.925703+0000", 2024-09-13T11:27:17.773 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.4794000000000003e-05 2024-09-13T11:27:17.774 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:17.774 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:17.774 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:27:17.774 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:48.925857+0000", 2024-09-13T11:27:17.774 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00015427 2024-09-13T11:27:17.774 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:17.774 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:17.774 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 5,8", 2024-09-13T11:27:17.774 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:48.925995+0000", 2024-09-13T11:27:17.774 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000137365 2024-09-13T11:27:17.774 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:17.774 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:17.774 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:27:17.775 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:48.927261+0000", 2024-09-13T11:27:17.775 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.001266021 2024-09-13T11:27:17.775 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:17.775 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:17.775 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:27:17.775 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:48.930015+0000", 2024-09-13T11:27:17.775 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0027545130000000001 2024-09-13T11:27:17.775 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:17.775 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:17.775 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:27:17.775 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:49.338898+0000", 2024-09-13T11:27:17.775 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.40888254299999999 2024-09-13T11:27:17.775 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:17.775 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:17.776 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:27:17.776 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:49.338967+0000", 2024-09-13T11:27:17.776 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.9402999999999999e-05 2024-09-13T11:27:17.776 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:17.776 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:17.776 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:27:17.776 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:49.339781+0000", 2024-09-13T11:27:17.776 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00081427699999999997 2024-09-13T11:27:17.776 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:27:17.776 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:27:17.776 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:27:17.776 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:17.776 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:17.776 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop_reply(client.4665.0:131687 3.28 e135/120 ondisk, result = 0)", 2024-09-13T11:27:17.777 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:24:48.928935+0000", 2024-09-13T11:27:17.777 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 148.77593701699999, 2024-09-13T11:27:17.777 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.41085638499999999, 2024-09-13T11:27:17.777 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:27:17.777 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:27:17.777 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-09-13T11:27:17.777 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:27:17.777 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:17.777 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:27:17.777 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:48.928935+0000", 2024-09-13T11:27:17.777 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:27:17.777 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:17.777 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:17.777 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:27:17.778 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:48.928935+0000", 2024-09-13T11:27:17.778 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:27:17.778 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:17.778 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:17.778 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:27:17.778 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:48.928936+0000", 2024-09-13T11:27:17.778 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.519e-06 2024-09-13T11:27:17.778 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:17.778 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:17.778 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:27:17.778 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:48.928941+0000", 2024-09-13T11:27:17.778 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.0710000000000002e-06 2024-09-13T11:27:17.778 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:17.778 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:17.779 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:27:17.779 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:49.338799+0000", 2024-09-13T11:27:17.779 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.40985808499999998 2024-09-13T11:27:17.779 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:17.779 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:17.779 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:27:17.779 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:49.338809+0000", 2024-09-13T11:27:17.779 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.4390000000000006e-06 2024-09-13T11:27:17.779 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:17.779 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:17.779 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:27:17.779 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:49.338878+0000", 2024-09-13T11:27:17.779 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.9429000000000002e-05 2024-09-13T11:27:17.779 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:17.780 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:17.780 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:27:17.780 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:49.338890+0000", 2024-09-13T11:27:17.780 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.2217e-05 2024-09-13T11:27:17.780 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:17.780 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:17.780 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:27:17.780 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:49.339791+0000", 2024-09-13T11:27:17.780 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00090062499999999995 2024-09-13T11:27:17.780 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:27:17.780 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:27:17.780 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:27:17.780 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:17.780 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:17.781 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:181661 3.4 3:2019f47a:::benchmark_data_smithi177_31116_object41358:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e170)", 2024-09-13T11:27:17.781 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:25:31.131054+0000", 2024-09-13T11:27:17.781 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 106.57381725099999, 2024-09-13T11:27:17.781 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.44761152399999998, 2024-09-13T11:27:17.781 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:27:17.781 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:27:17.781 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:27:17.781 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:27:17.781 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:27:17.781 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:27:17.781 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 181661 2024-09-13T11:27:17.781 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:17.781 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:27:17.781 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:17.782 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:27:17.782 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:31.131054+0000", 2024-09-13T11:27:17.782 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:27:17.782 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:17.782 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:17.782 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:27:17.782 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:31.131054+0000", 2024-09-13T11:27:17.782 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:27:17.782 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:17.782 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:17.782 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:27:17.782 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:31.131056+0000", 2024-09-13T11:27:17.782 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.8640000000000001e-06 2024-09-13T11:27:17.782 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:17.783 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:17.783 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:27:17.783 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:31.131061+0000", 2024-09-13T11:27:17.783 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.6249999999999998e-06 2024-09-13T11:27:17.783 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:17.783 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:17.783 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:27:17.783 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:31.131069+0000", 2024-09-13T11:27:17.783 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.1049999999999995e-06 2024-09-13T11:27:17.783 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:17.783 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:17.783 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:27:17.783 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:31.131074+0000", 2024-09-13T11:27:17.783 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.344e-06 2024-09-13T11:27:17.784 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:17.784 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:17.784 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:27:17.784 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:31.131121+0000", 2024-09-13T11:27:17.784 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.6794e-05 2024-09-13T11:27:17.784 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:17.784 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:17.784 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:27:17.784 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:31.131225+0000", 2024-09-13T11:27:17.784 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000103502 2024-09-13T11:27:17.784 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:17.784 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:17.784 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 12", 2024-09-13T11:27:17.784 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:31.131289+0000", 2024-09-13T11:27:17.785 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.4790999999999997e-05 2024-09-13T11:27:17.785 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:17.785 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:17.785 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:27:17.785 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:31.133191+0000", 2024-09-13T11:27:17.785 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0019013859999999999 2024-09-13T11:27:17.785 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:17.785 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:17.785 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:27:17.785 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:31.578585+0000", 2024-09-13T11:27:17.785 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.44539403 2024-09-13T11:27:17.785 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:17.785 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:17.786 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:27:17.786 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:31.578620+0000", 2024-09-13T11:27:17.786 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.4657999999999999e-05 2024-09-13T11:27:17.786 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:17.786 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:17.786 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:27:17.786 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:31.578666+0000", 2024-09-13T11:27:17.786 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.6424999999999997e-05 2024-09-13T11:27:17.786 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:27:17.786 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:27:17.786 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:27:17.786 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:17.786 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:17.786 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop_reply(client.4665.0:181661 3.4 e170/117 ondisk, result = 0)", 2024-09-13T11:27:17.786 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:25:31.135385+0000", 2024-09-13T11:27:17.787 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 106.569486431, 2024-09-13T11:27:17.787 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.44328512599999997, 2024-09-13T11:27:17.787 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:27:17.787 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:27:17.787 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-09-13T11:27:17.787 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:27:17.787 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:17.787 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:27:17.787 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:31.135385+0000", 2024-09-13T11:27:17.787 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:27:17.787 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:17.787 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:17.787 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:27:17.788 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:31.135385+0000", 2024-09-13T11:27:17.788 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:27:17.788 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:17.788 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:17.788 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:27:17.788 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:31.135387+0000", 2024-09-13T11:27:17.788 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.217e-06 2024-09-13T11:27:17.788 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:17.788 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:17.788 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:27:17.788 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:31.135391+0000", 2024-09-13T11:27:17.788 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.0820000000000001e-06 2024-09-13T11:27:17.788 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:17.788 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:17.789 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:27:17.789 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:31.578504+0000", 2024-09-13T11:27:17.789 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.44311382100000002 2024-09-13T11:27:17.789 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:17.789 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:17.789 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:27:17.789 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:31.578530+0000", 2024-09-13T11:27:17.789 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.5367e-05 2024-09-13T11:27:17.789 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:17.789 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:17.789 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:27:17.789 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:31.578576+0000", 2024-09-13T11:27:17.789 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.5840999999999997e-05 2024-09-13T11:27:17.789 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:17.790 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:17.790 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:27:17.790 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:31.578581+0000", 2024-09-13T11:27:17.790 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.5620000000000003e-06 2024-09-13T11:27:17.790 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:17.790 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:17.790 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:27:17.790 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:31.578670+0000", 2024-09-13T11:27:17.790 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.9235999999999997e-05 2024-09-13T11:27:17.790 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:27:17.791 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:27:17.791 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:27:17.791 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:17.791 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:17.791 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop(client.4665.0:184026 3.b e171/117 3:d3052347:::benchmark_data_smithi177_31116_object43723:head v 171'6273, mlcod=171'6269)", 2024-09-13T11:27:17.791 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:25:32.645695+0000", 2024-09-13T11:27:17.791 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 105.059176697, 2024-09-13T11:27:17.791 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.34822244200000002, 2024-09-13T11:27:17.791 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:27:17.791 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:27:17.791 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:27:17.791 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:27:17.791 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:17.792 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:27:17.792 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:32.645695+0000", 2024-09-13T11:27:17.792 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:27:17.792 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:17.792 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:17.792 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:27:17.792 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:32.645695+0000", 2024-09-13T11:27:17.792 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:27:17.792 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:17.792 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:17.792 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:27:17.792 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:32.645696+0000", 2024-09-13T11:27:17.792 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.4780000000000001e-06 2024-09-13T11:27:17.792 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:17.793 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:17.793 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:27:17.793 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:32.645704+0000", 2024-09-13T11:27:17.793 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.1980000000000004e-06 2024-09-13T11:27:17.793 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:17.793 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:17.793 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:27:17.793 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:32.992228+0000", 2024-09-13T11:27:17.793 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.346523992 2024-09-13T11:27:17.793 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:17.793 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:17.793 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:27:17.793 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:32.992232+0000", 2024-09-13T11:27:17.793 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.3109999999999999e-06 2024-09-13T11:27:17.794 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:17.794 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:17.794 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:27:17.794 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:32.992275+0000", 2024-09-13T11:27:17.794 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.2589999999999997e-05 2024-09-13T11:27:17.794 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:17.794 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:17.794 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:27:17.794 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:32.992289+0000", 2024-09-13T11:27:17.794 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.4116e-05 2024-09-13T11:27:17.794 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:17.794 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:17.794 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:27:17.794 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:32.993877+0000", 2024-09-13T11:27:17.795 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0015879259999999999 2024-09-13T11:27:17.795 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:17.795 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:17.795 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:27:17.795 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:32.993917+0000", 2024-09-13T11:27:17.795 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.0831000000000003e-05 2024-09-13T11:27:17.795 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:27:17.795 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:27:17.795 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:27:17.795 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:17.795 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:17.795 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop(client.4665.0:184072 3.b e171/117 3:d319cf93:::benchmark_data_smithi177_31116_object43769:head v 171'6274, mlcod=171'6270)", 2024-09-13T11:27:17.795 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:25:32.701516+0000", 2024-09-13T11:27:17.795 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 105.003356009, 2024-09-13T11:27:17.796 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.30463955399999998, 2024-09-13T11:27:17.796 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:27:17.796 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:27:17.796 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:27:17.796 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:27:17.796 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:17.796 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:27:17.796 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:32.701516+0000", 2024-09-13T11:27:17.796 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:27:17.796 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:17.796 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:17.796 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:27:17.796 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:32.701516+0000", 2024-09-13T11:27:17.797 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:27:17.797 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:17.797 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:17.797 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:27:17.797 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:32.701519+0000", 2024-09-13T11:27:17.797 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.0029999999999999e-06 2024-09-13T11:27:17.797 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:17.797 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:17.797 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:27:17.797 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:32.701530+0000", 2024-09-13T11:27:17.797 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.1759999999999999e-05 2024-09-13T11:27:17.797 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:17.797 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:17.797 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:27:17.798 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:33.003551+0000", 2024-09-13T11:27:17.798 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.302020819 2024-09-13T11:27:17.798 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:17.798 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:17.798 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:27:17.798 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:33.003558+0000", 2024-09-13T11:27:17.798 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.2299999999999996e-06 2024-09-13T11:27:17.798 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:17.798 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:17.798 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:27:17.798 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:33.003630+0000", 2024-09-13T11:27:17.798 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.2125000000000005e-05 2024-09-13T11:27:17.798 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:17.798 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:17.799 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:27:17.799 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:33.003648+0000", 2024-09-13T11:27:17.799 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.8672e-05 2024-09-13T11:27:17.799 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:17.799 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:17.799 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:27:17.799 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:33.006114+0000", 2024-09-13T11:27:17.799 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0024656790000000001 2024-09-13T11:27:17.799 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:17.799 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:17.799 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:27:17.799 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:33.006155+0000", 2024-09-13T11:27:17.799 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.1266000000000001e-05 2024-09-13T11:27:17.799 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:27:17.800 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:27:17.800 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:27:17.800 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:17.800 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:17.800 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "MOSDPGRecoveryDeleteReply(3.39 e242,238 3:9f6b69ce:::benchmark_data_smithi177_31116_object27023:head,160'6505)", 2024-09-13T11:27:17.800 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:27:11.545923+0000", 2024-09-13T11:27:17.800 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 6.1589489009999996, 2024-09-13T11:27:17.800 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.50434392299999997, 2024-09-13T11:27:17.800 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:27:17.800 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:27:17.800 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "reached pg", 2024-09-13T11:27:17.800 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:27:17.800 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:17.800 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:27:17.801 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:27:11.545923+0000", 2024-09-13T11:27:17.801 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:27:17.801 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:17.801 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:17.801 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:27:17.801 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:27:11.545923+0000", 2024-09-13T11:27:17.801 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:27:17.801 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:17.801 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:17.801 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:27:17.801 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:27:11.545924+0000", 2024-09-13T11:27:17.801 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.4020000000000001e-06 2024-09-13T11:27:17.801 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:17.801 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:17.802 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:27:17.802 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:27:11.545930+0000", 2024-09-13T11:27:17.802 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.4580000000000001e-06 2024-09-13T11:27:17.802 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:17.802 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:17.802 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:27:17.802 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:27:12.049823+0000", 2024-09-13T11:27:17.802 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.50389338100000003 2024-09-13T11:27:17.802 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:17.802 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:17.802 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:27:17.802 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:27:12.049834+0000", 2024-09-13T11:27:17.802 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.1199999999999999e-05 2024-09-13T11:27:17.802 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:17.803 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:17.803 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:27:17.803 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:27:12.049897+0000", 2024-09-13T11:27:17.803 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.3225000000000005e-05 2024-09-13T11:27:17.803 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:17.803 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:17.803 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:27:17.803 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:27:12.050267+0000", 2024-09-13T11:27:17.803 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00036925700000000001 2024-09-13T11:27:17.803 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:27:17.803 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:27:17.803 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:27:17.803 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:27:17.803 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:27:17.804 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-09-13T11:27:17.806 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-10 --journal-path /var/lib/ceph/osd/ceph-10/journal --force --op remove --pgid 3.32 2024-09-13T11:27:17.806 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_ops_in_flight 2024-09-13T11:27:17.806 INFO:tasks.ceph.osd.13.smithi092.stderr:2024-09-13T11:27:17.803+0000 7f3e5afea640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26350) UID: 0 2024-09-13T11:27:17.906 INFO:tasks.ceph.osd.8.smithi001.stderr:2024-09-13T11:27:17.902+0000 7fb9b48ae640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26312) UID: 0 2024-09-13T11:27:17.966 INFO:tasks.ceph.osd.14.smithi162.stderr:2024-09-13T11:27:17.963+0000 7f3dec334640 -1 osd.14 246 handle_osd_map injecting map crc failure 2024-09-13T11:27:17.969 INFO:teuthology.orchestra.run.smithi001.stderr:nodeep-scrub is set 2024-09-13T11:27:17.983 INFO:tasks.ceph.osd.8.smithi001.stderr:2024-09-13T11:27:17.978+0000 7fb9a92db640 -1 osd.8 246 handle_osd_map injecting map crc failure 2024-09-13T11:27:17.984 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-13T11:27:17.984 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [ 2024-09-13T11:27:17.984 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:17.984 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:279308 3.3f 3:fffa926b:::benchmark_data_smithi177_31116_object139005:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e247)", 2024-09-13T11:27:17.984 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:27:17.977946+0000", 2024-09-13T11:27:17.984 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 0.0072940970000000003, 2024-09-13T11:27:17.984 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0073229549999999999, 2024-09-13T11:27:17.984 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:27:17.984 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:27:17.984 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "waiting for degraded object", 2024-09-13T11:27:17.985 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:27:17.985 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:27:17.985 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:27:17.985 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 279308 2024-09-13T11:27:17.985 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:17.985 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:27:17.985 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:17.985 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:27:17.985 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:27:17.977946+0000", 2024-09-13T11:27:17.985 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:17.985 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:17.985 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:17.985 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:27:17.985 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:27:17.977946+0000", 2024-09-13T11:27:17.986 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:17.986 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:17.986 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:17.986 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:27:17.986 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:27:17.977948+0000", 2024-09-13T11:27:17.986 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.711e-06 2024-09-13T11:27:17.986 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:17.986 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:17.986 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:27:17.986 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:27:17.977953+0000", 2024-09-13T11:27:17.986 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.6949999999999999e-06 2024-09-13T11:27:17.986 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:17.986 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:17.986 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:27:17.987 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:27:17.977963+0000", 2024-09-13T11:27:17.987 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0552e-05 2024-09-13T11:27:17.987 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:17.987 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:17.987 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:27:17.987 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:27:17.977971+0000", 2024-09-13T11:27:17.987 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.7719999999999994e-06 2024-09-13T11:27:17.987 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:17.987 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:17.987 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:27:17.987 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:27:17.978026+0000", 2024-09-13T11:27:17.987 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.4654999999999998e-05 2024-09-13T11:27:17.987 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:17.987 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:17.988 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for degraded object", 2024-09-13T11:27:17.988 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:27:17.981343+0000", 2024-09-13T11:27:17.988 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.003317544 2024-09-13T11:27:17.988 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:17.988 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:27:17.988 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:17.988 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:17.988 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:17.988 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGRecoveryDeleteReply(3.1f e247,243 3:fb7ee849:::benchmark_data_smithi177_31116_object14050:head,151'6472)", 2024-09-13T11:27:17.988 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:27:17.984432+0000", 2024-09-13T11:27:17.988 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 0.00080868699999999999, 2024-09-13T11:27:17.988 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00090405100000000001, 2024-09-13T11:27:17.988 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:27:17.988 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:27:17.989 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "queued for pg", 2024-09-13T11:27:17.989 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:27:17.989 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:17.989 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:27:17.989 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:27:17.984432+0000", 2024-09-13T11:27:17.989 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:17.989 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:17.989 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:17.989 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:27:17.989 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:27:17.984432+0000", 2024-09-13T11:27:17.989 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:17.989 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:17.989 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:17.989 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:27:17.990 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:27:17.984433+0000", 2024-09-13T11:27:17.990 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2929999999999999e-06 2024-09-13T11:27:17.990 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:17.990 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:17.990 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:27:17.990 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:27:17.984438+0000", 2024-09-13T11:27:17.990 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.0989999999999998e-06 2024-09-13T11:27:17.990 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:17.990 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:17.990 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:27:17.990 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:27:17.984456+0000", 2024-09-13T11:27:17.990 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.7638999999999998e-05 2024-09-13T11:27:17.990 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:17.990 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:17.991 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:27:17.991 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:27:17.984464+0000", 2024-09-13T11:27:17.991 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.4970000000000001e-06 2024-09-13T11:27:17.991 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:17.991 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:27:17.991 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:17.991 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:17.991 INFO:teuthology.orchestra.run.smithi162.stdout: ], 2024-09-13T11:27:17.991 INFO:teuthology.orchestra.run.smithi162.stdout: "num_ops": 2 2024-09-13T11:27:17.991 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-13T11:27:18.004 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_blocked_ops 2024-09-13T11:27:18.008 INFO:tasks.ceph.osd.12.smithi001.stderr:2024-09-13T11:27:18.006+0000 7f892c6e6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26314) UID: 0 2024-09-13T11:27:18.109 INFO:tasks.ceph.osd.1.smithi092.stderr:2024-09-13T11:27:18.107+0000 7fe7789c5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33718) UID: 0 2024-09-13T11:27:18.152 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-13T11:27:18.152 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [], 2024-09-13T11:27:18.152 INFO:teuthology.orchestra.run.smithi162.stdout: "complaint_time": 30, 2024-09-13T11:27:18.152 INFO:teuthology.orchestra.run.smithi162.stdout: "num_blocked_ops": 0 2024-09-13T11:27:18.152 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-13T11:27:18.175 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_historic_ops 2024-09-13T11:27:18.209 INFO:tasks.ceph.osd.9.smithi092.stderr:2024-09-13T11:27:18.207+0000 7ff5ab3d0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26348) UID: 0 2024-09-13T11:27:18.265 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T11:27:18.311 INFO:tasks.ceph.osd.4.smithi001.stderr:2024-09-13T11:27:18.310+0000 7ff8e0218640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26310) UID: 0 2024-09-13T11:27:18.332 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-13T11:27:18.332 INFO:teuthology.orchestra.run.smithi162.stdout: "size": 20, 2024-09-13T11:27:18.332 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 600, 2024-09-13T11:27:18.332 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [ 2024-09-13T11:27:18.332 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:18.332 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:221526 3.2 3.ea98b382 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e191)", 2024-09-13T11:27:18.332 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:26:04.295546+0000", 2024-09-13T11:27:18.332 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 74.035933685000003, 2024-09-13T11:27:18.332 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.73989113200000001, 2024-09-13T11:27:18.332 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:27:18.332 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:27:18.333 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-09-13T11:27:18.333 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:27:18.333 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:27:18.333 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:27:18.333 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 221526 2024-09-13T11:27:18.333 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:18.333 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:27:18.333 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:18.333 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:27:18.333 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:04.295546+0000", 2024-09-13T11:27:18.333 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:18.333 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:18.333 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:18.334 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:27:18.334 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:04.295546+0000", 2024-09-13T11:27:18.334 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:18.334 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:18.334 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:18.334 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:27:18.334 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:04.295551+0000", 2024-09-13T11:27:18.334 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.6940000000000003e-06 2024-09-13T11:27:18.334 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:18.334 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:18.335 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:27:18.335 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:04.295567+0000", 2024-09-13T11:27:18.335 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5896000000000002e-05 2024-09-13T11:27:18.335 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:18.335 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:18.335 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:27:18.335 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:04.295599+0000", 2024-09-13T11:27:18.335 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.243e-05 2024-09-13T11:27:18.335 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:18.335 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:18.335 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:27:18.335 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:04.295622+0000", 2024-09-13T11:27:18.336 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.3149000000000001e-05 2024-09-13T11:27:18.336 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:18.336 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:18.336 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:27:18.336 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:04.295764+0000", 2024-09-13T11:27:18.336 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00014193300000000001 2024-09-13T11:27:18.336 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:18.336 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:18.336 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-09-13T11:27:18.336 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:04.295778+0000", 2024-09-13T11:27:18.336 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3763e-05 2024-09-13T11:27:18.336 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:18.336 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:18.337 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:27:18.337 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:05.035420+0000", 2024-09-13T11:27:18.337 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.73964228300000001 2024-09-13T11:27:18.337 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:18.337 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:18.337 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:27:18.337 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:05.035437+0000", 2024-09-13T11:27:18.337 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.6983999999999999e-05 2024-09-13T11:27:18.337 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:18.337 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:27:18.337 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:18.337 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:18.337 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:18.337 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:221528 3.15 3.b7a45bf5 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e191)", 2024-09-13T11:27:18.338 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:26:04.296174+0000", 2024-09-13T11:27:18.338 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 74.035306039999995, 2024-09-13T11:27:18.338 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.73909410399999997, 2024-09-13T11:27:18.338 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:27:18.338 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:27:18.338 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-09-13T11:27:18.338 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:27:18.338 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:27:18.338 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:27:18.338 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 221528 2024-09-13T11:27:18.338 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:18.338 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:27:18.338 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:18.339 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:27:18.339 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:04.296174+0000", 2024-09-13T11:27:18.339 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:18.339 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:18.339 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:18.339 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:27:18.339 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:04.296174+0000", 2024-09-13T11:27:18.339 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:18.339 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:18.339 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:18.339 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:27:18.339 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:04.296188+0000", 2024-09-13T11:27:18.339 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4037e-05 2024-09-13T11:27:18.340 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:18.340 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:18.340 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:27:18.340 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:04.296201+0000", 2024-09-13T11:27:18.340 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3096e-05 2024-09-13T11:27:18.340 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:18.340 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:18.340 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:27:18.340 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:04.296242+0000", 2024-09-13T11:27:18.340 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.1103000000000003e-05 2024-09-13T11:27:18.340 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:18.340 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:18.340 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:27:18.341 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:04.296264+0000", 2024-09-13T11:27:18.341 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.173e-05 2024-09-13T11:27:18.341 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:18.341 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:18.341 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:27:18.341 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:04.296410+0000", 2024-09-13T11:27:18.341 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00014574600000000001 2024-09-13T11:27:18.341 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:18.341 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:18.341 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-09-13T11:27:18.341 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:04.296424+0000", 2024-09-13T11:27:18.341 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3926e-05 2024-09-13T11:27:18.341 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:18.342 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:18.342 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:27:18.342 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:05.035221+0000", 2024-09-13T11:27:18.342 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.73879784999999998 2024-09-13T11:27:18.342 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:18.342 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:18.342 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:27:18.342 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:05.035268+0000", 2024-09-13T11:27:18.342 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.6616000000000002e-05 2024-09-13T11:27:18.342 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:18.342 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:27:18.342 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:18.342 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:18.343 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:18.343 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:221544 3.12 3.ec3fbdf2 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e191)", 2024-09-13T11:27:18.343 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:26:04.384452+0000", 2024-09-13T11:27:18.343 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 73.947027448, 2024-09-13T11:27:18.343 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.65065541999999998, 2024-09-13T11:27:18.343 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:27:18.343 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:27:18.343 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-09-13T11:27:18.343 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:27:18.343 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:27:18.343 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:27:18.343 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 221544 2024-09-13T11:27:18.343 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:18.343 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:27:18.344 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:18.344 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:27:18.344 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:04.384452+0000", 2024-09-13T11:27:18.344 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:18.344 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:18.344 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:18.344 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:27:18.344 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:04.384452+0000", 2024-09-13T11:27:18.344 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:18.344 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:18.344 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:18.344 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:27:18.344 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:04.384457+0000", 2024-09-13T11:27:18.345 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.6469999999999997e-06 2024-09-13T11:27:18.345 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:18.345 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:18.345 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:27:18.345 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:04.384470+0000", 2024-09-13T11:27:18.345 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3186999999999999e-05 2024-09-13T11:27:18.345 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:18.345 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:18.345 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:27:18.345 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:04.384504+0000", 2024-09-13T11:27:18.345 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.3413e-05 2024-09-13T11:27:18.345 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:18.346 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:18.346 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:27:18.346 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:04.384526+0000", 2024-09-13T11:27:18.346 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.2690000000000001e-05 2024-09-13T11:27:18.346 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:18.346 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:18.346 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:27:18.346 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:04.384653+0000", 2024-09-13T11:27:18.346 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00012697199999999999 2024-09-13T11:27:18.346 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:18.346 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:18.346 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-09-13T11:27:18.346 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:04.384668+0000", 2024-09-13T11:27:18.346 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4936000000000001e-05 2024-09-13T11:27:18.347 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:18.347 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:18.347 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:27:18.347 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:05.035087+0000", 2024-09-13T11:27:18.347 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.65041902900000004 2024-09-13T11:27:18.347 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:18.347 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:18.347 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:27:18.347 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:05.035108+0000", 2024-09-13T11:27:18.347 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.0545999999999999e-05 2024-09-13T11:27:18.347 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:18.347 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:27:18.347 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:18.348 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:18.348 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:18.348 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:234164 3.22 3.d472cf62 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e201)", 2024-09-13T11:27:18.348 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:26:17.191424+0000", 2024-09-13T11:27:18.348 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 61.140056328999997, 2024-09-13T11:27:18.348 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.95096563000000001, 2024-09-13T11:27:18.348 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:27:18.348 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:27:18.348 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-09-13T11:27:18.348 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:27:18.348 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:27:18.348 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:27:18.348 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 234164 2024-09-13T11:27:18.349 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:18.349 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:27:18.349 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:18.349 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:27:18.349 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:17.191424+0000", 2024-09-13T11:27:18.349 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:18.349 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:18.349 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:18.349 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:27:18.349 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:17.191424+0000", 2024-09-13T11:27:18.349 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:18.349 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:18.349 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:18.350 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:27:18.350 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:17.191426+0000", 2024-09-13T11:27:18.350 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.3269999999999999e-06 2024-09-13T11:27:18.350 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:18.350 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:18.350 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:27:18.350 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:17.191434+0000", 2024-09-13T11:27:18.350 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.5490000000000006e-06 2024-09-13T11:27:18.350 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:18.350 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:18.350 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:27:18.350 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:17.191450+0000", 2024-09-13T11:27:18.350 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.596e-05 2024-09-13T11:27:18.350 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:18.351 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:18.351 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:27:18.351 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:17.191458+0000", 2024-09-13T11:27:18.351 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.6070000000000002e-06 2024-09-13T11:27:18.351 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:18.351 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:18.351 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:27:18.351 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:17.191518+0000", 2024-09-13T11:27:18.351 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.9488999999999999e-05 2024-09-13T11:27:18.351 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:18.351 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:18.351 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-09-13T11:27:18.351 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:17.191526+0000", 2024-09-13T11:27:18.352 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.8049999999999996e-06 2024-09-13T11:27:18.352 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:18.352 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:18.352 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:27:18.352 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:18.142341+0000", 2024-09-13T11:27:18.352 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.95081438299999999 2024-09-13T11:27:18.352 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:18.352 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:18.352 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:27:18.352 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:18.142389+0000", 2024-09-13T11:27:18.352 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.8510000000000001e-05 2024-09-13T11:27:18.352 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:18.352 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:27:18.352 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:18.353 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:18.353 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:18.353 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:234214 3.22 3.3ebf73a2 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e201)", 2024-09-13T11:27:18.353 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:26:17.239598+0000", 2024-09-13T11:27:18.353 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 61.091882058000003, 2024-09-13T11:27:18.353 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.90310166300000005, 2024-09-13T11:27:18.353 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:27:18.353 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:27:18.353 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-09-13T11:27:18.353 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:27:18.356 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:27:18.356 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:27:18.356 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 234214 2024-09-13T11:27:18.356 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:18.356 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:27:18.356 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:18.356 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:27:18.356 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:17.239598+0000", 2024-09-13T11:27:18.356 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:18.356 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:18.356 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:18.357 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:27:18.357 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:17.239598+0000", 2024-09-13T11:27:18.357 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:18.357 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:18.357 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:18.357 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:27:18.357 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:17.239601+0000", 2024-09-13T11:27:18.357 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.021e-06 2024-09-13T11:27:18.357 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:18.357 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:18.357 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:27:18.357 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:17.239609+0000", 2024-09-13T11:27:18.357 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.9459999999999998e-06 2024-09-13T11:27:18.358 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:18.358 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:18.358 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:27:18.358 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:17.239628+0000", 2024-09-13T11:27:18.358 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.9029e-05 2024-09-13T11:27:18.358 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:18.358 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:18.358 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:27:18.358 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:17.239637+0000", 2024-09-13T11:27:18.358 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.7530000000000007e-06 2024-09-13T11:27:18.358 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:18.358 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:18.358 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:27:18.358 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:17.239716+0000", 2024-09-13T11:27:18.359 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.9789000000000005e-05 2024-09-13T11:27:18.359 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:18.359 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:18.359 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-09-13T11:27:18.359 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:17.239729+0000", 2024-09-13T11:27:18.359 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2215000000000001e-05 2024-09-13T11:27:18.359 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:18.359 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:18.359 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:27:18.359 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:18.142672+0000", 2024-09-13T11:27:18.359 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.90294309299999997 2024-09-13T11:27:18.359 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:18.359 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:18.360 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:27:18.360 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:18.142700+0000", 2024-09-13T11:27:18.360 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.7817000000000002e-05 2024-09-13T11:27:18.360 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:18.360 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:27:18.360 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:18.360 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:18.360 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:18.360 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:234350 3.22 3.6c9f6f62 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e201)", 2024-09-13T11:27:18.360 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:26:17.412568+0000", 2024-09-13T11:27:18.360 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 60.918912259000003, 2024-09-13T11:27:18.360 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.73023408300000003, 2024-09-13T11:27:18.360 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:27:18.361 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:27:18.361 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-09-13T11:27:18.361 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:27:18.361 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:27:18.361 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:27:18.361 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 234350 2024-09-13T11:27:18.361 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:18.361 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:27:18.361 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:18.361 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:27:18.361 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:17.412568+0000", 2024-09-13T11:27:18.361 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:18.361 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:18.361 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:18.362 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:27:18.362 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:17.412568+0000", 2024-09-13T11:27:18.362 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:18.362 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:18.362 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:18.362 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:27:18.362 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:17.412571+0000", 2024-09-13T11:27:18.362 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.044e-06 2024-09-13T11:27:18.362 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:18.362 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:18.362 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:27:18.362 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:17.412579+0000", 2024-09-13T11:27:18.362 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.7639999999999994e-06 2024-09-13T11:27:18.362 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:18.363 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:18.363 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:27:18.363 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:17.412595+0000", 2024-09-13T11:27:18.363 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5637999999999998e-05 2024-09-13T11:27:18.363 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:18.363 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:18.363 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:27:18.363 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:17.412606+0000", 2024-09-13T11:27:18.363 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1175999999999999e-05 2024-09-13T11:27:18.363 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:18.363 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:18.363 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:27:18.363 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:17.412690+0000", 2024-09-13T11:27:18.363 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.3486000000000007e-05 2024-09-13T11:27:18.364 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:18.364 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:18.364 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-09-13T11:27:18.364 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:17.412703+0000", 2024-09-13T11:27:18.364 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3196e-05 2024-09-13T11:27:18.364 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:18.364 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:18.364 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:27:18.364 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:18.142767+0000", 2024-09-13T11:27:18.364 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.73006412600000004 2024-09-13T11:27:18.364 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:18.364 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:18.364 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:27:18.365 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:18.142802+0000", 2024-09-13T11:27:18.365 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.4653000000000002e-05 2024-09-13T11:27:18.365 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:18.365 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:27:18.365 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:18.365 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:18.365 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:18.365 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGRecoveryDelete(3.6 e201,191 3:60767a7b:::benchmark_data_smithi177_31116_object14927:head,151'5598)", 2024-09-13T11:27:18.365 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:26:17.890539+0000", 2024-09-13T11:27:18.365 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 60.440941037000002, 2024-09-13T11:27:18.365 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.29203489500000002, 2024-09-13T11:27:18.365 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:27:18.365 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:27:18.365 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:27:18.366 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:27:18.366 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:18.366 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:27:18.366 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:17.890539+0000", 2024-09-13T11:27:18.366 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:18.366 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:18.366 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:18.366 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:27:18.366 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:17.890539+0000", 2024-09-13T11:27:18.366 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:18.367 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:18.367 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:18.367 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:27:18.367 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:17.890541+0000", 2024-09-13T11:27:18.367 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.795e-06 2024-09-13T11:27:18.367 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:18.367 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:18.367 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:27:18.367 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:17.890547+0000", 2024-09-13T11:27:18.367 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.3770000000000001e-06 2024-09-13T11:27:18.367 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:18.367 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:18.367 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:27:18.368 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:18.182175+0000", 2024-09-13T11:27:18.368 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.29162787000000001 2024-09-13T11:27:18.368 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:18.368 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:18.368 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:27:18.368 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:18.182190+0000", 2024-09-13T11:27:18.368 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4976e-05 2024-09-13T11:27:18.368 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:18.368 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:18.368 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:27:18.368 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:18.182397+0000", 2024-09-13T11:27:18.368 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00020751499999999999 2024-09-13T11:27:18.368 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:18.368 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:18.369 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:27:18.369 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:18.182425+0000", 2024-09-13T11:27:18.369 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.7076999999999998e-05 2024-09-13T11:27:18.369 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:18.369 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:18.369 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:27:18.369 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:18.182574+0000", 2024-09-13T11:27:18.369 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000149285 2024-09-13T11:27:18.369 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:18.369 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:27:18.369 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:18.369 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:18.369 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:18.370 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:234778 3.22 3.36a6aba2 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e201)", 2024-09-13T11:27:18.370 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:26:17.909436+0000", 2024-09-13T11:27:18.370 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 60.422044055000001, 2024-09-13T11:27:18.370 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.233461425, 2024-09-13T11:27:18.370 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:27:18.370 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:27:18.370 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-09-13T11:27:18.370 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:27:18.370 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:27:18.370 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:27:18.370 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 234778 2024-09-13T11:27:18.370 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:18.370 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:27:18.371 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:18.371 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:27:18.371 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:17.909436+0000", 2024-09-13T11:27:18.371 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:18.371 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:18.371 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:18.371 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:27:18.371 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:17.909436+0000", 2024-09-13T11:27:18.371 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:18.371 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:18.371 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:18.371 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:27:18.371 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:17.909439+0000", 2024-09-13T11:27:18.371 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.717e-06 2024-09-13T11:27:18.372 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:18.372 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:18.372 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:27:18.372 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:17.909446+0000", 2024-09-13T11:27:18.372 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.1980000000000004e-06 2024-09-13T11:27:18.372 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:18.372 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:18.372 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:27:18.372 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:17.909462+0000", 2024-09-13T11:27:18.372 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5809999999999999e-05 2024-09-13T11:27:18.372 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:18.372 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:18.372 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:27:18.372 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:17.909471+0000", 2024-09-13T11:27:18.373 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.3100000000000006e-06 2024-09-13T11:27:18.373 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:18.373 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:18.373 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:27:18.373 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:17.909554+0000", 2024-09-13T11:27:18.373 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.2917000000000006e-05 2024-09-13T11:27:18.373 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:18.373 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:18.373 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-09-13T11:27:18.373 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:17.909569+0000", 2024-09-13T11:27:18.373 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5201000000000001e-05 2024-09-13T11:27:18.374 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:18.374 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:18.374 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:27:18.374 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:18.142863+0000", 2024-09-13T11:27:18.374 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.23329387900000001 2024-09-13T11:27:18.374 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:18.374 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:18.374 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:27:18.374 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:18.142897+0000", 2024-09-13T11:27:18.374 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.4393000000000001e-05 2024-09-13T11:27:18.374 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:18.374 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:27:18.374 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:18.375 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:18.375 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:18.375 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPush(3.2 201/192 [PushOp(3:409d6ad2:::benchmark_data_smithi177_31116_object97365:head, version: 113'3767, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:409d6ad2:::benchmark_data_smithi177_31116_object97365:head@113'3767, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:27:18.375 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:26:17.942627+0000", 2024-09-13T11:27:18.375 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 60.388853384000001, 2024-09-13T11:27:18.375 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.25342641300000002, 2024-09-13T11:27:18.375 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:27:18.375 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:27:18.375 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:27:18.375 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:27:18.375 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:18.375 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:27:18.375 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:17.942627+0000", 2024-09-13T11:27:18.376 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:18.376 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:18.376 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:18.376 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:27:18.376 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:17.942627+0000", 2024-09-13T11:27:18.376 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:18.376 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:18.376 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:18.376 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:27:18.376 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:17.942628+0000", 2024-09-13T11:27:18.376 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3179999999999999e-06 2024-09-13T11:27:18.376 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:18.376 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:18.376 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:27:18.377 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:17.942665+0000", 2024-09-13T11:27:18.377 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.6748999999999997e-05 2024-09-13T11:27:18.377 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:18.377 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:18.377 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:27:18.377 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:18.192821+0000", 2024-09-13T11:27:18.377 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.25015650900000003 2024-09-13T11:27:18.377 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:18.377 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:18.377 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:27:18.377 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:18.192853+0000", 2024-09-13T11:27:18.377 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.1579999999999999e-05 2024-09-13T11:27:18.377 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:18.377 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:18.378 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:27:18.378 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:18.193147+0000", 2024-09-13T11:27:18.378 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000294424 2024-09-13T11:27:18.378 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:18.378 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:18.378 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:27:18.378 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:18.193191+0000", 2024-09-13T11:27:18.378 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.3463e-05 2024-09-13T11:27:18.378 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:18.378 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:18.378 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:27:18.378 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:18.196053+0000", 2024-09-13T11:27:18.379 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0028623699999999999 2024-09-13T11:27:18.379 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:18.379 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:27:18.379 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:18.379 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:18.379 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:18.379 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGRecoveryDelete(3.22 e223,218 3:47eb4e8b:::benchmark_data_smithi177_31116_object75722:head,187'5647)", 2024-09-13T11:27:18.379 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:26:48.562782+0000", 2024-09-13T11:27:18.379 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 29.768697702000001, 2024-09-13T11:27:18.379 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.307749461, 2024-09-13T11:27:18.379 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:27:18.379 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:27:18.379 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:27:18.379 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:27:18.380 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:18.380 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:27:18.380 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:48.562782+0000", 2024-09-13T11:27:18.380 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:18.380 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:18.380 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:18.380 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:27:18.380 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:48.562782+0000", 2024-09-13T11:27:18.380 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:18.380 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:18.380 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:18.380 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:27:18.380 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:48.562785+0000", 2024-09-13T11:27:18.381 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.9900000000000002e-06 2024-09-13T11:27:18.381 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:18.381 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:18.381 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:27:18.381 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:48.562796+0000", 2024-09-13T11:27:18.381 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1046999999999999e-05 2024-09-13T11:27:18.381 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:18.381 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:18.381 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:27:18.381 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:48.862609+0000", 2024-09-13T11:27:18.381 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.29981280399999999 2024-09-13T11:27:18.381 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:18.381 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:18.382 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:27:18.382 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:48.862624+0000", 2024-09-13T11:27:18.382 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5214e-05 2024-09-13T11:27:18.382 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:18.382 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:18.382 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:27:18.382 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:48.870361+0000", 2024-09-13T11:27:18.382 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0077370670000000002 2024-09-13T11:27:18.382 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:18.382 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:18.382 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:27:18.382 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:48.870385+0000", 2024-09-13T11:27:18.382 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.3693000000000002e-05 2024-09-13T11:27:18.383 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:18.383 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:18.383 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:27:18.383 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:48.870532+0000", 2024-09-13T11:27:18.383 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00014664600000000001 2024-09-13T11:27:18.383 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:18.383 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:27:18.383 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:18.383 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:18.383 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:18.383 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:266847 3.15 3:aced3906:::benchmark_data_smithi177_31116_object126544:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e223)", 2024-09-13T11:27:18.383 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:26:48.588305+0000", 2024-09-13T11:27:18.383 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 29.743174514, 2024-09-13T11:27:18.384 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.29333894599999999, 2024-09-13T11:27:18.384 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:27:18.384 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:27:18.384 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:27:18.384 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:27:18.384 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:27:18.384 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:27:18.384 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 266847 2024-09-13T11:27:18.384 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:18.384 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:27:18.384 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:18.384 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:27:18.384 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:48.588305+0000", 2024-09-13T11:27:18.385 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:18.385 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:18.385 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:18.385 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:27:18.385 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:48.588305+0000", 2024-09-13T11:27:18.385 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:18.385 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:18.385 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:18.385 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:27:18.385 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:48.588309+0000", 2024-09-13T11:27:18.385 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.41e-06 2024-09-13T11:27:18.386 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:18.386 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:18.386 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:27:18.386 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:48.588319+0000", 2024-09-13T11:27:18.386 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.9359999999999994e-06 2024-09-13T11:27:18.386 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:18.386 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:18.386 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:27:18.386 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:48.588348+0000", 2024-09-13T11:27:18.386 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.8940000000000001e-05 2024-09-13T11:27:18.386 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:18.386 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:18.386 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:27:18.387 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:48.588362+0000", 2024-09-13T11:27:18.387 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4752999999999999e-05 2024-09-13T11:27:18.387 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:18.387 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:18.387 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:27:18.387 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:48.588510+0000", 2024-09-13T11:27:18.387 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00014728500000000001 2024-09-13T11:27:18.387 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:18.387 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:18.387 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:27:18.387 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:48.588731+0000", 2024-09-13T11:27:18.387 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000220762 2024-09-13T11:27:18.387 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:18.387 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:18.388 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 0", 2024-09-13T11:27:18.388 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:48.588906+0000", 2024-09-13T11:27:18.388 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00017593900000000001 2024-09-13T11:27:18.388 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:18.388 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:18.388 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:27:18.388 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:48.594579+0000", 2024-09-13T11:27:18.388 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.005672351 2024-09-13T11:27:18.388 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:18.388 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:18.388 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:27:18.388 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:48.881375+0000", 2024-09-13T11:27:18.388 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.28679606600000002 2024-09-13T11:27:18.389 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:18.389 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:18.389 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:27:18.389 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:48.881524+0000", 2024-09-13T11:27:18.389 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000148935 2024-09-13T11:27:18.389 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:18.389 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:18.389 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:27:18.389 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:48.881644+0000", 2024-09-13T11:27:18.389 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000120569 2024-09-13T11:27:18.389 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:18.389 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:27:18.389 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:18.389 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:18.390 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:18.390 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.4665.0:266884 3.1f e224/223 3:fb121a9e:::benchmark_data_smithi177_31116_object126581:head v 224'9867, mlcod=224'9865)", 2024-09-13T11:27:18.390 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:26:49.276350+0000", 2024-09-13T11:27:18.390 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 29.055129866000001, 2024-09-13T11:27:18.390 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.53992503999999997, 2024-09-13T11:27:18.390 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:27:18.390 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:27:18.390 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:27:18.390 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:27:18.390 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:18.390 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:27:18.390 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:49.276350+0000", 2024-09-13T11:27:18.390 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:18.391 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:18.391 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:18.391 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:27:18.391 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:49.276350+0000", 2024-09-13T11:27:18.391 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:18.391 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:18.391 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:18.391 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:27:18.391 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:49.276352+0000", 2024-09-13T11:27:18.391 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.6020000000000001e-06 2024-09-13T11:27:18.391 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:18.391 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:18.391 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:27:18.391 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:49.276360+0000", 2024-09-13T11:27:18.392 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.6319999999999997e-06 2024-09-13T11:27:18.392 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:18.392 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:18.392 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:27:18.392 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:49.810598+0000", 2024-09-13T11:27:18.392 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.53423792199999998 2024-09-13T11:27:18.392 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:18.392 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:18.392 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:27:18.392 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:49.810616+0000", 2024-09-13T11:27:18.392 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.7456999999999999e-05 2024-09-13T11:27:18.392 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:18.392 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:18.393 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:27:18.393 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:49.810770+0000", 2024-09-13T11:27:18.393 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00015476199999999999 2024-09-13T11:27:18.393 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:18.393 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:18.393 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:27:18.393 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:49.810838+0000", 2024-09-13T11:27:18.393 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.7788999999999998e-05 2024-09-13T11:27:18.393 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:18.393 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:18.393 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:27:18.393 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:49.816185+0000", 2024-09-13T11:27:18.393 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0053467369999999998 2024-09-13T11:27:18.394 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:18.394 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:18.394 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:27:18.394 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:49.816275+0000", 2024-09-13T11:27:18.394 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.0138999999999999e-05 2024-09-13T11:27:18.394 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:18.394 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:27:18.394 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:18.394 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:18.394 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:18.394 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:277975 3.3f 3.e0a6dd7f (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e239)", 2024-09-13T11:27:18.394 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:27:06.991878+0000", 2024-09-13T11:27:18.394 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 11.339601763999999, 2024-09-13T11:27:18.395 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.011507309, 2024-09-13T11:27:18.395 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:27:18.395 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:27:18.395 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-09-13T11:27:18.395 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:27:18.395 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:27:18.395 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:27:18.395 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 277975 2024-09-13T11:27:18.395 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:18.395 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:27:18.395 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:18.395 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:27:18.395 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:27:06.991878+0000", 2024-09-13T11:27:18.395 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:18.396 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:18.396 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:18.396 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:27:18.396 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:27:06.991878+0000", 2024-09-13T11:27:18.396 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:18.396 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:18.396 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:18.396 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:27:18.396 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:27:06.991884+0000", 2024-09-13T11:27:18.396 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.677e-06 2024-09-13T11:27:18.396 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:18.396 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:18.396 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:27:18.397 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:27:06.991899+0000", 2024-09-13T11:27:18.397 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5271999999999999e-05 2024-09-13T11:27:18.397 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:18.397 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:18.397 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:27:18.397 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:27:06.991934+0000", 2024-09-13T11:27:18.397 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.4754e-05 2024-09-13T11:27:18.397 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:18.397 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:18.397 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:27:18.397 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:27:06.991951+0000", 2024-09-13T11:27:18.397 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.7076999999999999e-05 2024-09-13T11:27:18.397 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:18.397 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:18.398 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:27:18.398 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:27:06.992111+0000", 2024-09-13T11:27:18.398 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000159875 2024-09-13T11:27:18.398 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:18.398 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:18.398 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op must wait for map", 2024-09-13T11:27:18.398 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:27:06.992127+0000", 2024-09-13T11:27:18.398 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.6334e-05 2024-09-13T11:27:18.398 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:18.398 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:18.398 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:27:18.398 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:27:07.010082+0000", 2024-09-13T11:27:18.398 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.017955314999999999 2024-09-13T11:27:18.399 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:18.399 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:18.399 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-09-13T11:27:18.399 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:27:07.010102+0000", 2024-09-13T11:27:18.399 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.9425000000000001e-05 2024-09-13T11:27:18.399 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:18.399 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:18.399 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:27:18.399 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:27:08.003314+0000", 2024-09-13T11:27:18.399 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.99321246799999996 2024-09-13T11:27:18.399 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:18.399 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:18.399 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:27:18.399 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:27:08.003385+0000", 2024-09-13T11:27:18.400 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.1112000000000006e-05 2024-09-13T11:27:18.400 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:18.400 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:27:18.400 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:18.400 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:18.400 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:18.400 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:278046 3.3f 3.10b6dbff (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e239)", 2024-09-13T11:27:18.400 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:27:06.991985+0000", 2024-09-13T11:27:18.400 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 11.339494619, 2024-09-13T11:27:18.400 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0116638360000001, 2024-09-13T11:27:18.400 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:27:18.400 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:27:18.400 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-09-13T11:27:18.401 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:27:18.401 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:27:18.401 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:27:18.401 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 278046 2024-09-13T11:27:18.401 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:18.401 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:27:18.401 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:18.401 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:27:18.401 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:27:06.991985+0000", 2024-09-13T11:27:18.401 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:18.401 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:18.401 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:18.401 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:27:18.402 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:27:06.991985+0000", 2024-09-13T11:27:18.402 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:18.402 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:18.402 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:18.402 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:27:18.402 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:27:06.991987+0000", 2024-09-13T11:27:18.402 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5939999999999999e-06 2024-09-13T11:27:18.402 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:18.402 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:18.402 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:27:18.402 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:27:06.991998+0000", 2024-09-13T11:27:18.402 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1195e-05 2024-09-13T11:27:18.402 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:18.403 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:18.403 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:27:18.403 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:27:06.992043+0000", 2024-09-13T11:27:18.403 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.4827999999999998e-05 2024-09-13T11:27:18.403 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:18.403 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:18.403 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:27:18.403 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:27:06.992065+0000", 2024-09-13T11:27:18.403 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.2544999999999998e-05 2024-09-13T11:27:18.403 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:18.403 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:18.403 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:27:18.403 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:27:06.992225+0000", 2024-09-13T11:27:18.403 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00015911300000000001 2024-09-13T11:27:18.404 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:18.404 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:18.404 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting_for_map not empty", 2024-09-13T11:27:18.404 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:27:06.992242+0000", 2024-09-13T11:27:18.404 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.7025999999999999e-05 2024-09-13T11:27:18.404 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:18.404 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:18.404 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:27:18.404 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:27:07.010420+0000", 2024-09-13T11:27:18.404 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.018178577000000001 2024-09-13T11:27:18.404 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:18.404 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:18.405 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-09-13T11:27:18.405 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:27:07.010436+0000", 2024-09-13T11:27:18.405 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.6101999999999999e-05 2024-09-13T11:27:18.405 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:18.405 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:18.405 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:27:18.405 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:27:08.003629+0000", 2024-09-13T11:27:18.405 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.99319262399999997 2024-09-13T11:27:18.405 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:18.405 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:18.405 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:27:18.405 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:27:08.003649+0000", 2024-09-13T11:27:18.405 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.0231999999999999e-05 2024-09-13T11:27:18.405 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:18.406 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:27:18.406 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:18.406 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:18.406 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:18.406 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:278067 3.3f 3.9fc82f3f (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e239)", 2024-09-13T11:27:18.406 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:27:06.992123+0000", 2024-09-13T11:27:18.406 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 11.339357375000001, 2024-09-13T11:27:18.406 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.011590293, 2024-09-13T11:27:18.406 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:27:18.406 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:27:18.406 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-09-13T11:27:18.406 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:27:18.406 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:27:18.407 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:27:18.407 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 278067 2024-09-13T11:27:18.407 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:18.407 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:27:18.407 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:18.407 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:27:18.407 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:27:06.992123+0000", 2024-09-13T11:27:18.407 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:18.407 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:18.407 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:18.407 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:27:18.407 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:27:06.992123+0000", 2024-09-13T11:27:18.407 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:18.407 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:18.407 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:18.408 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:27:18.408 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:27:06.992125+0000", 2024-09-13T11:27:18.408 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.644e-06 2024-09-13T11:27:18.408 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:18.408 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:18.408 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:27:18.408 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:27:06.992136+0000", 2024-09-13T11:27:18.408 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0961999999999999e-05 2024-09-13T11:27:18.408 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:18.408 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:18.408 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:27:18.408 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:27:06.992159+0000", 2024-09-13T11:27:18.409 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.2934000000000001e-05 2024-09-13T11:27:18.409 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:18.409 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:18.409 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:27:18.409 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:27:06.992176+0000", 2024-09-13T11:27:18.409 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.6518e-05 2024-09-13T11:27:18.409 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:18.409 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:18.409 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:27:18.409 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:27:06.992297+0000", 2024-09-13T11:27:18.409 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00012105299999999999 2024-09-13T11:27:18.409 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:18.409 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:18.409 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting_for_map not empty", 2024-09-13T11:27:18.410 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:27:06.992304+0000", 2024-09-13T11:27:18.410 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.9079999999999998e-06 2024-09-13T11:27:18.410 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:18.410 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:18.410 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:27:18.410 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:27:07.010485+0000", 2024-09-13T11:27:18.410 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.018181342999999999 2024-09-13T11:27:18.410 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:18.410 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:18.410 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-09-13T11:27:18.410 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:27:07.010498+0000", 2024-09-13T11:27:18.410 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2914999999999999e-05 2024-09-13T11:27:18.410 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:18.411 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:18.411 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:27:18.411 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:27:08.003699+0000", 2024-09-13T11:27:18.411 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.99320087000000001 2024-09-13T11:27:18.411 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:18.411 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:18.411 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:27:18.411 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:27:08.003713+0000", 2024-09-13T11:27:18.411 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4146e-05 2024-09-13T11:27:18.411 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:18.411 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:27:18.411 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:18.411 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:18.411 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:18.412 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:278101 3.3f 3.cbc5fa3f (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e239)", 2024-09-13T11:27:18.412 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:27:06.992204+0000", 2024-09-13T11:27:18.412 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 11.339275972999999, 2024-09-13T11:27:18.412 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0115713200000001, 2024-09-13T11:27:18.412 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:27:18.412 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:27:18.412 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-09-13T11:27:18.412 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:27:18.412 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:27:18.412 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:27:18.412 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 278101 2024-09-13T11:27:18.412 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:18.412 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:27:18.413 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:18.413 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:27:18.413 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:27:06.992204+0000", 2024-09-13T11:27:18.413 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:18.413 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:18.413 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:18.413 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:27:18.413 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:27:06.992204+0000", 2024-09-13T11:27:18.413 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:18.413 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:18.413 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:18.413 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:27:18.413 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:27:06.992206+0000", 2024-09-13T11:27:18.413 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5999999999999999e-06 2024-09-13T11:27:18.414 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:18.414 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:18.414 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:27:18.414 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:27:06.992212+0000", 2024-09-13T11:27:18.414 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.2759999999999997e-06 2024-09-13T11:27:18.414 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:18.414 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:18.414 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:27:18.414 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:27:06.992224+0000", 2024-09-13T11:27:18.414 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1963e-05 2024-09-13T11:27:18.414 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:18.414 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:18.414 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:27:18.414 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:27:06.992235+0000", 2024-09-13T11:27:18.415 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1262e-05 2024-09-13T11:27:18.415 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:18.415 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:18.415 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:27:18.415 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:27:06.992354+0000", 2024-09-13T11:27:18.415 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000118713 2024-09-13T11:27:18.415 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:18.415 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:18.415 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting_for_map not empty", 2024-09-13T11:27:18.415 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:27:06.992361+0000", 2024-09-13T11:27:18.415 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.7719999999999997e-06 2024-09-13T11:27:18.415 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:18.415 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:18.416 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:27:18.416 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:27:07.010539+0000", 2024-09-13T11:27:18.416 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.018178659999999999 2024-09-13T11:27:18.416 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:18.416 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:18.416 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-09-13T11:27:18.416 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:27:07.010559+0000", 2024-09-13T11:27:18.416 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.9638000000000001e-05 2024-09-13T11:27:18.416 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:18.416 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:18.416 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:27:18.416 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:27:08.003761+0000", 2024-09-13T11:27:18.416 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.99320253000000003 2024-09-13T11:27:18.416 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:18.417 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:18.417 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:27:18.417 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:27:08.003775+0000", 2024-09-13T11:27:18.417 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3906e-05 2024-09-13T11:27:18.417 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:18.417 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:27:18.417 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:18.417 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:18.417 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:18.417 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:278107 3.3f 3.24b842bf (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e239)", 2024-09-13T11:27:18.417 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:27:06.992250+0000", 2024-09-13T11:27:18.417 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 11.33922993, 2024-09-13T11:27:18.417 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.011585081, 2024-09-13T11:27:18.417 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:27:18.418 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:27:18.418 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-09-13T11:27:18.418 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:27:18.418 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:27:18.418 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:27:18.418 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 278107 2024-09-13T11:27:18.418 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:18.418 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:27:18.418 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:18.418 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:27:18.418 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:27:06.992250+0000", 2024-09-13T11:27:18.418 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:18.418 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:18.418 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:18.419 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:27:18.419 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:27:06.992250+0000", 2024-09-13T11:27:18.419 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:18.419 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:18.419 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:18.419 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:27:18.419 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:27:06.992251+0000", 2024-09-13T11:27:18.419 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4139999999999999e-06 2024-09-13T11:27:18.419 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:18.419 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:18.419 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:27:18.419 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:27:06.992257+0000", 2024-09-13T11:27:18.419 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.4639999999999997e-06 2024-09-13T11:27:18.419 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:18.420 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:18.420 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:27:18.420 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:27:06.992267+0000", 2024-09-13T11:27:18.420 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0518e-05 2024-09-13T11:27:18.420 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:18.420 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:18.420 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:27:18.420 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:27:06.992278+0000", 2024-09-13T11:27:18.420 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0180000000000001e-05 2024-09-13T11:27:18.420 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:18.420 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:18.420 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:27:18.420 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:27:06.992397+0000", 2024-09-13T11:27:18.421 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00011923500000000001 2024-09-13T11:27:18.421 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:18.421 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:18.421 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting_for_map not empty", 2024-09-13T11:27:18.421 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:27:06.992403+0000", 2024-09-13T11:27:18.421 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.4740000000000002e-06 2024-09-13T11:27:18.421 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:18.421 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:18.421 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:27:18.421 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:27:07.010605+0000", 2024-09-13T11:27:18.421 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.018201697999999999 2024-09-13T11:27:18.421 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:18.421 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:18.422 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-09-13T11:27:18.422 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:27:07.010617+0000", 2024-09-13T11:27:18.422 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.154e-05 2024-09-13T11:27:18.422 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:18.422 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:18.422 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:27:18.422 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:27:08.003823+0000", 2024-09-13T11:27:18.422 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.99320655300000005 2024-09-13T11:27:18.422 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:18.422 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:18.422 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:27:18.422 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:27:08.003835+0000", 2024-09-13T11:27:18.422 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2004999999999999e-05 2024-09-13T11:27:18.422 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:18.423 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:27:18.423 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:18.423 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:18.423 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:18.423 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGRecoveryDelete(3.2f e241,227 3:f68554f1:::benchmark_data_smithi177_31116_object47862:head,174'7646)", 2024-09-13T11:27:18.423 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:27:11.254029+0000", 2024-09-13T11:27:18.423 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 7.0774506610000003, 2024-09-13T11:27:18.423 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.225005441, 2024-09-13T11:27:18.423 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:27:18.423 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:27:18.423 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:27:18.423 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:27:18.423 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:18.424 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:27:18.424 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:27:11.254029+0000", 2024-09-13T11:27:18.424 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:18.424 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:18.424 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:18.424 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:27:18.424 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:27:11.254029+0000", 2024-09-13T11:27:18.424 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:18.424 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:18.424 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:18.424 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:27:18.424 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:27:11.254032+0000", 2024-09-13T11:27:18.424 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.9569999999999998e-06 2024-09-13T11:27:18.425 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:18.425 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:18.425 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:27:18.425 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:27:11.254043+0000", 2024-09-13T11:27:18.425 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0703000000000001e-05 2024-09-13T11:27:18.425 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:18.425 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:18.425 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:27:18.425 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:27:11.478618+0000", 2024-09-13T11:27:18.425 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.22457474099999999 2024-09-13T11:27:18.425 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:18.425 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:18.425 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:27:18.426 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:27:11.478633+0000", 2024-09-13T11:27:18.426 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5786000000000001e-05 2024-09-13T11:27:18.426 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:18.426 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:18.426 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:27:18.426 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:27:11.478825+0000", 2024-09-13T11:27:18.427 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000191719 2024-09-13T11:27:18.427 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:18.427 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:18.427 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:27:18.427 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:27:11.478847+0000", 2024-09-13T11:27:18.427 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.1662999999999999e-05 2024-09-13T11:27:18.427 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:18.427 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:18.427 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:27:18.427 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:27:11.479035+0000", 2024-09-13T11:27:18.427 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00018787200000000001 2024-09-13T11:27:18.427 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:18.427 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:27:18.428 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:18.428 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:18.428 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:18.428 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGRecoveryDelete(3.2f e244,227 3:f784bcf5:::benchmark_data_smithi177_31116_object67166:head,184'8217)", 2024-09-13T11:27:18.428 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:27:16.064546+0000", 2024-09-13T11:27:18.428 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 2.2669336449999999, 2024-09-13T11:27:18.428 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.48785419200000002, 2024-09-13T11:27:18.428 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:27:18.428 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:27:18.428 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:27:18.429 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:27:18.429 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:18.429 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:27:18.429 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:27:16.064546+0000", 2024-09-13T11:27:18.429 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:18.429 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:18.429 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:18.429 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:27:18.429 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:27:16.064546+0000", 2024-09-13T11:27:18.429 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:18.429 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:18.429 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:18.429 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:27:18.430 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:27:16.064548+0000", 2024-09-13T11:27:18.430 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.435e-06 2024-09-13T11:27:18.430 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:18.430 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:18.430 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:27:18.430 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:27:16.064553+0000", 2024-09-13T11:27:18.430 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.1499999999999998e-06 2024-09-13T11:27:18.430 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:18.430 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:18.430 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:27:18.431 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:27:16.552082+0000", 2024-09-13T11:27:18.431 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.48752930999999999 2024-09-13T11:27:18.431 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:18.431 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:18.431 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:27:18.431 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:27:16.552091+0000", 2024-09-13T11:27:18.431 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.2350000000000005e-06 2024-09-13T11:27:18.431 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:18.431 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:18.431 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:27:18.431 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:27:16.552281+0000", 2024-09-13T11:27:18.431 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000189798 2024-09-13T11:27:18.431 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:18.431 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:18.432 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:27:18.432 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:27:16.552297+0000", 2024-09-13T11:27:18.432 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5736e-05 2024-09-13T11:27:18.432 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:18.432 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:18.432 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:27:18.432 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:27:16.552400+0000", 2024-09-13T11:27:18.432 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000103528 2024-09-13T11:27:18.432 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:18.432 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:27:18.432 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:18.432 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:18.432 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:18.433 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:278438 3.15 3:ab6b0d5d:::benchmark_data_smithi177_31116_object138135:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e245)", 2024-09-13T11:27:18.433 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:27:16.736707+0000", 2024-09-13T11:27:18.433 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 1.5947724490000001, 2024-09-13T11:27:18.433 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.35539664300000001, 2024-09-13T11:27:18.433 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:27:18.433 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:27:18.433 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:27:18.433 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:27:18.433 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:27:18.433 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:27:18.433 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 278438 2024-09-13T11:27:18.433 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:18.433 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:27:18.433 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:18.434 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:27:18.434 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:27:16.736707+0000", 2024-09-13T11:27:18.434 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:18.434 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:18.434 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:18.434 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:27:18.434 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:27:16.736707+0000", 2024-09-13T11:27:18.434 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:18.434 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:18.434 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:18.434 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:27:18.434 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:27:16.736709+0000", 2024-09-13T11:27:18.434 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.863e-06 2024-09-13T11:27:18.435 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:18.435 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:18.435 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:27:18.435 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:27:16.736715+0000", 2024-09-13T11:27:18.435 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.3380000000000004e-06 2024-09-13T11:27:18.435 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:18.435 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:18.435 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:27:18.435 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:27:16.736726+0000", 2024-09-13T11:27:18.435 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1365e-05 2024-09-13T11:27:18.436 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:18.436 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:18.436 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:27:18.436 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:27:16.736732+0000", 2024-09-13T11:27:18.436 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.9479999999999996e-06 2024-09-13T11:27:18.436 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:18.436 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:18.436 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:27:18.436 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:27:16.736790+0000", 2024-09-13T11:27:18.436 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.8477999999999998e-05 2024-09-13T11:27:18.436 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:18.436 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:18.436 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:27:18.437 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:27:16.736912+0000", 2024-09-13T11:27:18.437 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000121279 2024-09-13T11:27:18.437 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:18.437 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:18.437 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 0", 2024-09-13T11:27:18.437 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:27:16.736989+0000", 2024-09-13T11:27:18.437 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.6922999999999995e-05 2024-09-13T11:27:18.437 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:18.437 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:18.437 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:27:18.437 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:27:16.738822+0000", 2024-09-13T11:27:18.437 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0018333959999999999 2024-09-13T11:27:18.437 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:18.438 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:18.438 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:27:18.438 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:27:17.092030+0000", 2024-09-13T11:27:18.438 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.35320780899999998 2024-09-13T11:27:18.438 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:18.438 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:18.438 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:27:18.438 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:27:17.092068+0000", 2024-09-13T11:27:18.438 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.7710000000000003e-05 2024-09-13T11:27:18.438 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:18.438 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:18.438 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:27:18.438 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:27:17.092104+0000", 2024-09-13T11:27:18.438 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.6534e-05 2024-09-13T11:27:18.439 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:18.439 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:27:18.439 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:18.439 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:18.439 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:27:18.439 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-13T11:27:18.440 INFO:tasks.ceph.osd.14.smithi162.stderr:2024-09-13T11:27:18.439+0000 7f3df5903640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33154) UID: 0 2024-09-13T11:27:18.441 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-13T11:27:18.484 INFO:teuthology.orchestra.run.smithi001.stdout:osd.0: {} 2024-09-13T11:27:18.485 INFO:teuthology.orchestra.run.smithi001.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T11:27:18.516 INFO:teuthology.orchestra.run.smithi001.stdout:osd.1: {} 2024-09-13T11:27:18.517 INFO:teuthology.orchestra.run.smithi001.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T11:27:18.546 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-13T11:27:18.538+0000 7f6a68775640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26307) UID: 0 2024-09-13T11:27:18.552 INFO:teuthology.orchestra.run.smithi001.stdout:osd.2: {} 2024-09-13T11:27:18.552 INFO:teuthology.orchestra.run.smithi001.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T11:27:18.588 INFO:teuthology.orchestra.run.smithi001.stdout:osd.3: {} 2024-09-13T11:27:18.588 INFO:teuthology.orchestra.run.smithi001.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T11:27:18.614 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-13T11:27:18.614 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:27:18.628 INFO:teuthology.orchestra.run.smithi001.stdout:osd.4: {} 2024-09-13T11:27:18.628 INFO:teuthology.orchestra.run.smithi001.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T11:27:18.639 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-13T11:27:18.666 INFO:teuthology.orchestra.run.smithi001.stdout:osd.5: {} 2024-09-13T11:27:18.667 INFO:teuthology.orchestra.run.smithi001.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T11:27:18.667 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-13T11:27:18.662+0000 7f6a68775640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26307) UID: 0 2024-09-13T11:27:18.708 INFO:teuthology.orchestra.run.smithi001.stdout:osd.6: {} 2024-09-13T11:27:18.708 INFO:teuthology.orchestra.run.smithi001.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T11:27:18.743 INFO:tasks.ceph.osd.15.smithi177.stderr:2024-09-13T11:27:18.741+0000 7f9d6ce77640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26578) UID: 0 2024-09-13T11:27:18.778 INFO:teuthology.orchestra.run.smithi001.stdout:osd.7: {} 2024-09-13T11:27:18.778 INFO:teuthology.orchestra.run.smithi001.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T11:27:18.842 INFO:teuthology.orchestra.run.smithi001.stdout:osd.8: {} 2024-09-13T11:27:18.842 INFO:teuthology.orchestra.run.smithi001.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-13T11:27:18.844 INFO:tasks.ceph.osd.3.smithi177.stderr:2024-09-13T11:27:18.841+0000 7f10adba4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26574) UID: 0 2024-09-13T11:27:18.912 INFO:teuthology.orchestra.run.smithi001.stdout:osd.9: {} 2024-09-13T11:27:18.913 INFO:teuthology.orchestra.run.smithi001.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-13T11:27:18.913 INFO:teuthology.orchestra.run.smithi001.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-09-13T11:27:18.922 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-13T11:27:18.922 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:27:18.945 INFO:tasks.ceph.osd.3.smithi177.stderr:2024-09-13T11:27:18.941+0000 7f10adba4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26574) UID: 0 2024-09-13T11:27:18.950 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-13T11:27:18.965 INFO:teuthology.orchestra.run.smithi001.stdout:osd.11: {} 2024-09-13T11:27:18.966 INFO:teuthology.orchestra.run.smithi001.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-13T11:27:19.016 INFO:teuthology.orchestra.run.smithi001.stdout:osd.12: {} 2024-09-13T11:27:19.016 INFO:teuthology.orchestra.run.smithi001.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-13T11:27:19.045 INFO:tasks.ceph.osd.2.smithi162.stderr:2024-09-13T11:27:19.043+0000 7fe92b35a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26313) UID: 0 2024-09-13T11:27:19.062 INFO:teuthology.orchestra.run.smithi001.stdout:osd.13: {} 2024-09-13T11:27:19.063 INFO:teuthology.orchestra.run.smithi001.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-13T11:27:19.084 INFO:teuthology.orchestra.run.smithi001.stdout:osd.14: {} 2024-09-13T11:27:19.085 INFO:teuthology.orchestra.run.smithi001.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-13T11:27:19.105 INFO:teuthology.orchestra.run.smithi001.stdout:osd.15: {} 2024-09-13T11:27:19.105 INFO:teuthology.orchestra.run.smithi001.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-13T11:27:19.107 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-13T11:27:19.108 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-13T11:27:19.126 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-13T11:27:19.126 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:27:19.139 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-13T11:27:19.146 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-09-13T11:27:19.143+0000 7f97befc9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26315) UID: 0 2024-09-13T11:27:19.248 INFO:tasks.ceph.osd.1.smithi092.stderr:2024-09-13T11:27:19.243+0000 7fe7789c5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33718) UID: 0 2024-09-13T11:27:19.348 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-13T11:27:19.348 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:27:19.350 INFO:tasks.ceph.osd.4.smithi001.stderr:2024-09-13T11:27:19.346+0000 7ff8e0218640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26310) UID: 0 2024-09-13T11:27:19.370 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-13T11:27:19.450 INFO:tasks.ceph.osd.2.smithi162.stderr:2024-09-13T11:27:19.447+0000 7fe92b35a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26313) UID: 0 2024-09-13T11:27:19.550 INFO:tasks.ceph.osd.13.smithi092.stderr:2024-09-13T11:27:19.547+0000 7f3e5afea640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26350) UID: 0 2024-09-13T11:27:19.571 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-13T11:27:19.572 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:27:19.594 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-13T11:27:19.651 INFO:tasks.ceph.osd.2.smithi162.stderr:2024-09-13T11:27:19.647+0000 7fe92b35a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26313) UID: 0 2024-09-13T11:27:19.724 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-13T11:27:19.724 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:27:19.741 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-09-13T11:27:19.752 INFO:tasks.ceph.osd.13.smithi092.stderr:2024-09-13T11:27:19.747+0000 7f3e5afea640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26350) UID: 0 2024-09-13T11:27:19.852 INFO:tasks.ceph.osd.1.smithi092.stderr:2024-09-13T11:27:19.851+0000 7fe7789c5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33718) UID: 0 2024-09-13T11:27:19.871 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-13T11:27:19.871 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:27:19.887 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-09-13T11:27:19.953 INFO:tasks.ceph.osd.12.smithi001.stderr:2024-09-13T11:27:19.950+0000 7f892c6e6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26314) UID: 0 2024-09-13T11:27:19.998 INFO:tasks.ceph.osd.14.smithi162.stderr:2024-09-13T11:27:19.995+0000 7f3dea330640 -1 osd.14 248 handle_osd_map injecting map crc failure 2024-09-13T11:27:20.016 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-13T11:27:20.049 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-13T11:27:20.049 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:27:20.054 INFO:tasks.ceph.osd.3.smithi177.stderr:2024-09-13T11:27:20.052+0000 7f10adba4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26574) UID: 0 2024-09-13T11:27:20.067 INFO:tasks.radosbench.radosbench.0.smithi177.stdout:Removed 140300 objects 2024-09-13T11:27:20.067 INFO:tasks.radosbench.radosbench.0.smithi177.stdout:Clean up completed and total clean up time :142.044 2024-09-13T11:27:20.071 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-09-13T11:27:20.155 INFO:tasks.ceph.osd.2.smithi162.stderr:2024-09-13T11:27:20.151+0000 7fe92b35a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26313) UID: 0 2024-09-13T11:27:20.256 INFO:tasks.ceph.osd.9.smithi092.stderr:2024-09-13T11:27:20.251+0000 7ff5ab3d0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26348) UID: 0 2024-09-13T11:27:20.291 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-13T11:27:20.291 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:27:20.315 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_ops_in_flight 2024-09-13T11:27:20.361 INFO:tasks.ceph.osd.12.smithi001.stderr:2024-09-13T11:27:20.358+0000 7f892c6e6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26314) UID: 0 2024-09-13T11:27:20.458 INFO:tasks.ceph.osd.2.smithi162.stderr:2024-09-13T11:27:20.455+0000 7fe92b35a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26313) UID: 0 2024-09-13T11:27:20.506 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-13T11:27:20.507 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:27:20.529 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_blocked_ops 2024-09-13T11:27:20.571 INFO:tasks.ceph.osd.4.smithi001.stderr:2024-09-13T11:27:20.570+0000 7ff8e0218640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26310) UID: 0 2024-09-13T11:27:20.660 INFO:tasks.ceph.osd.1.smithi092.stderr:2024-09-13T11:27:20.655+0000 7fe7789c5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33718) UID: 0 2024-09-13T11:27:20.711 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-13T11:27:20.711 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:27:20.731 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_historic_ops 2024-09-13T11:27:20.775 INFO:tasks.ceph.osd.8.smithi001.stderr:2024-09-13T11:27:20.774+0000 7fb9b48ae640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26312) UID: 0 2024-09-13T11:27:20.862 INFO:tasks.ceph.osd.8.smithi001.stderr:2024-09-13T11:27:20.858+0000 7fb9b48ae640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26312) UID: 0 2024-09-13T11:27:20.870 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-13T11:27:20.871 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:27:20.889 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-13T11:27:20.962 INFO:tasks.ceph.osd.14.smithi162.stderr:2024-09-13T11:27:20.959+0000 7f3df5903640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33154) UID: 0 2024-09-13T11:27:21.007 INFO:teuthology.orchestra.run.smithi001.stderr:noscrub is unset 2024-09-13T11:27:21.043 INFO:teuthology.orchestra.run.smithi092.stdout:ERROR: (22) Invalid argument 2024-09-13T11:27:21.043 INFO:teuthology.orchestra.run.smithi092.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:27:21.064 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-09-13T11:27:21.063+0000 7f97befc9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26315) UID: 0 2024-09-13T11:27:21.067 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-13T11:27:21.109 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T11:27:21.164 INFO:tasks.ceph.osd.1.smithi092.stderr:2024-09-13T11:27:21.163+0000 7fe7789c5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33718) UID: 0 2024-09-13T11:27:21.214 INFO:teuthology.orchestra.run.smithi092.stdout:ERROR: (22) Invalid argument 2024-09-13T11:27:21.214 INFO:teuthology.orchestra.run.smithi092.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:27:21.233 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-13T11:27:21.276 INFO:tasks.ceph.osd.9.smithi092.stderr:2024-09-13T11:27:21.275+0000 7ff5ab3d0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26348) UID: 0 2024-09-13T11:27:21.286 INFO:teuthology.orchestra.run.smithi001.stdout:osd.0: {} 2024-09-13T11:27:21.286 INFO:teuthology.orchestra.run.smithi001.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T11:27:21.314 INFO:teuthology.orchestra.run.smithi001.stdout:osd.1: {} 2024-09-13T11:27:21.314 INFO:teuthology.orchestra.run.smithi001.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T11:27:21.341 INFO:teuthology.orchestra.run.smithi001.stdout:osd.2: {} 2024-09-13T11:27:21.341 INFO:teuthology.orchestra.run.smithi001.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T11:27:21.365 INFO:tasks.ceph.osd.4.smithi001.stderr:2024-09-13T11:27:21.362+0000 7ff8e0218640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26310) UID: 0 2024-09-13T11:27:21.369 INFO:teuthology.orchestra.run.smithi001.stdout:osd.3: {} 2024-09-13T11:27:21.369 INFO:teuthology.orchestra.run.smithi001.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T11:27:21.395 INFO:teuthology.orchestra.run.smithi001.stdout:osd.4: {} 2024-09-13T11:27:21.395 INFO:teuthology.orchestra.run.smithi001.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T11:27:21.424 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-09-13T11:27:21.424 INFO:teuthology.orchestra.run.smithi092.stdout: "size": 20, 2024-09-13T11:27:21.424 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 600, 2024-09-13T11:27:21.424 INFO:teuthology.orchestra.run.smithi092.stdout: "ops": [ 2024-09-13T11:27:21.424 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:21.424 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "MOSDPGPush(3.23 83/78 [PushOp(3:c61f6bab:::benchmark_data_smithi177_31116_object29479:head, version: 59'1860, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:c61f6bab:::benchmark_data_smithi177_31116_object29479:head@59'1860, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:27:21.424 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:40.770864+0000", 2024-09-13T11:27:21.424 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 220.65168558400001, 2024-09-13T11:27:21.425 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.31229662800000002, 2024-09-13T11:27:21.425 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:27:21.425 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:27:21.425 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-09-13T11:27:21.425 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:27:21.425 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:21.425 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:27:21.425 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:40.770864+0000", 2024-09-13T11:27:21.425 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:27:21.425 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:21.425 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:21.425 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:27:21.425 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:40.770864+0000", 2024-09-13T11:27:21.426 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:27:21.426 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:21.426 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:21.426 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:27:21.426 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:40.770865+0000", 2024-09-13T11:27:21.426 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.288e-06 2024-09-13T11:27:21.426 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:21.426 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:21.426 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:27:21.426 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:40.770955+0000", 2024-09-13T11:27:21.426 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.9871999999999996e-05 2024-09-13T11:27:21.426 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:21.426 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:21.427 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:27:21.427 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:41.082105+0000", 2024-09-13T11:27:21.427 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.31114963400000001 2024-09-13T11:27:21.427 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:21.427 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:21.427 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:27:21.427 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:41.082136+0000", 2024-09-13T11:27:21.427 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.0972000000000001e-05 2024-09-13T11:27:21.427 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:21.427 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:21.427 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:27:21.427 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:41.082404+0000", 2024-09-13T11:27:21.427 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00026844800000000002 2024-09-13T11:27:21.428 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:21.428 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:21.428 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:27:21.428 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:41.082429+0000", 2024-09-13T11:27:21.428 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.5021e-05 2024-09-13T11:27:21.428 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:21.428 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:21.428 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:27:21.428 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:41.083161+0000", 2024-09-13T11:27:21.428 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000731393 2024-09-13T11:27:21.428 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:27:21.428 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:27:21.428 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:27:21.428 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:21.429 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:21.429 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:110289 3.29 3.27287969 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e119)", 2024-09-13T11:27:21.429 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:24:25.774634+0000", 2024-09-13T11:27:21.429 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 175.647915509, 2024-09-13T11:27:21.429 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.31721633999999999, 2024-09-13T11:27:21.429 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:27:21.429 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:27:21.429 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "reached pg", 2024-09-13T11:27:21.429 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:27:21.429 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:27:21.429 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:27:21.429 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 110289 2024-09-13T11:27:21.429 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:21.430 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:27:21.430 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:21.430 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:27:21.430 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:25.774634+0000", 2024-09-13T11:27:21.430 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:27:21.430 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:21.430 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:21.430 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:27:21.430 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:25.774634+0000", 2024-09-13T11:27:21.430 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:27:21.430 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:21.430 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:21.430 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:27:21.430 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:25.774639+0000", 2024-09-13T11:27:21.431 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.6029999999999998e-06 2024-09-13T11:27:21.431 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:21.431 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:21.431 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:27:21.431 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:25.774695+0000", 2024-09-13T11:27:21.431 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.6694999999999998e-05 2024-09-13T11:27:21.431 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:21.431 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:21.431 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:27:21.431 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:25.774728+0000", 2024-09-13T11:27:21.431 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.2502000000000002e-05 2024-09-13T11:27:21.431 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:21.431 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:21.431 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:27:21.432 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:25.774750+0000", 2024-09-13T11:27:21.432 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.2002999999999999e-05 2024-09-13T11:27:21.432 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:21.432 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:21.432 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:27:21.432 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:25.774942+0000", 2024-09-13T11:27:21.432 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000192265 2024-09-13T11:27:21.432 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:21.432 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:21.432 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-09-13T11:27:21.432 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:25.774960+0000", 2024-09-13T11:27:21.433 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.8275999999999999e-05 2024-09-13T11:27:21.433 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:21.433 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:21.433 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:27:21.433 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:26.091821+0000", 2024-09-13T11:27:21.433 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.31686065000000002 2024-09-13T11:27:21.433 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:21.433 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:21.433 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:27:21.433 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:26.091850+0000", 2024-09-13T11:27:21.433 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.9346e-05 2024-09-13T11:27:21.433 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:27:21.433 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:27:21.433 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:27:21.434 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:21.434 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:21.434 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "MOSDPGPush(3.28 168/120 [PushOp(3:14eb8ed9:::benchmark_data_smithi177_31116_object87538:head, version: 110'4071, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:14eb8ed9:::benchmark_data_smithi177_31116_object87538:head@110'4071, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:27:21.434 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:25:27.274802+0000", 2024-09-13T11:27:21.434 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 114.147747744, 2024-09-13T11:27:21.434 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.369268239, 2024-09-13T11:27:21.434 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:27:21.434 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:27:21.434 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-09-13T11:27:21.434 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:27:21.436 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:21.436 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:27:21.436 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:27.274802+0000", 2024-09-13T11:27:21.437 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:27:21.437 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:21.437 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:21.437 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:27:21.437 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:27.274802+0000", 2024-09-13T11:27:21.437 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:27:21.437 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:21.437 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:21.437 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:27:21.437 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:27.274803+0000", 2024-09-13T11:27:21.437 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.7579999999999999e-06 2024-09-13T11:27:21.437 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:21.437 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:21.437 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:27:21.438 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:27.274831+0000", 2024-09-13T11:27:21.438 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.7464e-05 2024-09-13T11:27:21.438 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:21.438 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:21.438 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:27:21.438 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:27.641355+0000", 2024-09-13T11:27:21.438 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.36652425199999999 2024-09-13T11:27:21.438 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:21.438 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:21.438 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:27:21.438 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:27.641385+0000", 2024-09-13T11:27:21.438 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.9433000000000001e-05 2024-09-13T11:27:21.438 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:21.438 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:21.439 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:27:21.439 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:27.641611+0000", 2024-09-13T11:27:21.439 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00022646299999999999 2024-09-13T11:27:21.439 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:21.439 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:21.439 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:27:21.439 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:27.641638+0000", 2024-09-13T11:27:21.439 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.6482000000000002e-05 2024-09-13T11:27:21.439 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:21.439 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:21.439 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:27:21.439 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:27.644070+0000", 2024-09-13T11:27:21.439 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0024323869999999998 2024-09-13T11:27:21.440 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:27:21.440 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:27:21.440 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:27:21.440 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:21.440 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:21.440 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:191021 3.23 3:c4df18dd:::benchmark_data_smithi177_31116_object50718:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e175)", 2024-09-13T11:27:21.440 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:25:38.460411+0000", 2024-09-13T11:27:21.440 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 102.962138319, 2024-09-13T11:27:21.440 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.40138001400000001, 2024-09-13T11:27:21.440 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:27:21.440 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:27:21.440 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:27:21.440 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:27:21.440 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:27:21.441 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:27:21.441 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 191021 2024-09-13T11:27:21.441 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:21.441 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:27:21.441 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:21.441 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:27:21.441 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:38.460411+0000", 2024-09-13T11:27:21.441 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:27:21.441 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:21.441 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:21.442 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:27:21.442 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:38.460411+0000", 2024-09-13T11:27:21.442 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:27:21.442 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:21.442 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:21.442 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:27:21.442 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:38.460413+0000", 2024-09-13T11:27:21.442 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.075e-06 2024-09-13T11:27:21.442 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:21.442 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:21.442 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:27:21.442 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:38.460419+0000", 2024-09-13T11:27:21.442 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.6869999999999999e-06 2024-09-13T11:27:21.442 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:21.443 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:21.443 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:27:21.443 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:38.460431+0000", 2024-09-13T11:27:21.443 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.2451e-05 2024-09-13T11:27:21.443 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:21.443 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:21.443 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:27:21.443 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:38.460440+0000", 2024-09-13T11:27:21.443 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.5140000000000001e-06 2024-09-13T11:27:21.443 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:21.444 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:21.444 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:27:21.444 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:38.460501+0000", 2024-09-13T11:27:21.444 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.0772e-05 2024-09-13T11:27:21.444 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:21.444 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:21.444 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:27:21.444 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:38.460632+0000", 2024-09-13T11:27:21.444 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00013175200000000001 2024-09-13T11:27:21.444 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:21.444 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:21.444 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 9", 2024-09-13T11:27:21.444 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:38.460731+0000", 2024-09-13T11:27:21.444 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.8647000000000006e-05 2024-09-13T11:27:21.445 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:21.445 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:21.445 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:27:21.445 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:38.462322+0000", 2024-09-13T11:27:21.445 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.001590658 2024-09-13T11:27:21.445 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:21.445 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:21.445 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:27:21.445 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:38.861634+0000", 2024-09-13T11:27:21.445 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.39931206499999999 2024-09-13T11:27:21.445 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:21.445 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:21.445 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:27:21.446 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:38.861719+0000", 2024-09-13T11:27:21.446 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.4968000000000004e-05 2024-09-13T11:27:21.446 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:21.446 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:21.446 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:27:21.446 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:38.861791+0000", 2024-09-13T11:27:21.446 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.2424999999999999e-05 2024-09-13T11:27:21.446 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:27:21.446 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:27:21.446 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:27:21.446 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:21.446 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:21.446 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop_reply(client.4665.0:191021 3.23 e175/90 ondisk, result = 0)", 2024-09-13T11:27:21.446 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:25:38.462483+0000", 2024-09-13T11:27:21.447 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 102.960066897, 2024-09-13T11:27:21.447 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.39932300900000001, 2024-09-13T11:27:21.447 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:27:21.447 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:27:21.447 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-09-13T11:27:21.447 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:27:21.447 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:21.447 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:27:21.447 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:38.462483+0000", 2024-09-13T11:27:21.447 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:27:21.447 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:21.447 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:21.447 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:27:21.447 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:38.462483+0000", 2024-09-13T11:27:21.448 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:27:21.448 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:21.448 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:21.448 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:27:21.448 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:38.462484+0000", 2024-09-13T11:27:21.448 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.1990000000000001e-06 2024-09-13T11:27:21.448 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:21.448 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:21.448 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:27:21.448 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:38.462488+0000", 2024-09-13T11:27:21.448 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.985e-06 2024-09-13T11:27:21.448 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:21.448 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:21.448 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:27:21.449 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:38.861475+0000", 2024-09-13T11:27:21.449 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.39898754800000003 2024-09-13T11:27:21.449 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:21.449 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:21.449 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:27:21.449 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:38.861488+0000", 2024-09-13T11:27:21.449 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.3037000000000001e-05 2024-09-13T11:27:21.449 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:21.449 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:21.449 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:27:21.449 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:38.861607+0000", 2024-09-13T11:27:21.449 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00011835 2024-09-13T11:27:21.449 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:21.449 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:21.450 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:27:21.450 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:38.861624+0000", 2024-09-13T11:27:21.450 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.7730000000000001e-05 2024-09-13T11:27:21.450 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:21.450 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:21.450 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:27:21.450 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:38.861806+0000", 2024-09-13T11:27:21.450 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00018116000000000001 2024-09-13T11:27:21.450 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:27:21.450 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:27:21.450 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:27:21.450 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:21.450 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:21.450 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:209869 3.27 3:e6b1fcc5:::benchmark_data_smithi177_31116_object69566:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e185)", 2024-09-13T11:27:21.451 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:25:54.300263+0000", 2024-09-13T11:27:21.451 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 87.122286000000003, 2024-09-13T11:27:21.451 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.30266981599999998, 2024-09-13T11:27:21.451 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:27:21.451 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:27:21.451 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:27:21.451 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:27:21.451 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:27:21.451 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:27:21.451 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 209869 2024-09-13T11:27:21.451 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:21.451 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:27:21.451 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:21.451 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:27:21.452 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:54.300263+0000", 2024-09-13T11:27:21.452 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:27:21.452 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:21.452 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:21.452 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:27:21.452 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:54.300263+0000", 2024-09-13T11:27:21.452 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:27:21.452 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:21.452 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:21.452 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:27:21.452 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:54.300267+0000", 2024-09-13T11:27:21.452 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.2200000000000001e-06 2024-09-13T11:27:21.452 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:21.453 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:21.453 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:27:21.453 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:54.300276+0000", 2024-09-13T11:27:21.453 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.3570000000000004e-06 2024-09-13T11:27:21.453 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:21.453 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:21.453 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:27:21.453 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:54.300301+0000", 2024-09-13T11:27:21.453 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.5191e-05 2024-09-13T11:27:21.453 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:21.453 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:21.453 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:27:21.453 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:54.300321+0000", 2024-09-13T11:27:21.454 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.9967999999999999e-05 2024-09-13T11:27:21.454 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:21.454 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:21.454 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:27:21.454 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:54.300427+0000", 2024-09-13T11:27:21.454 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000105781 2024-09-13T11:27:21.454 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:21.454 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:21.454 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:27:21.454 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:54.300590+0000", 2024-09-13T11:27:21.454 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00016341000000000001 2024-09-13T11:27:21.454 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:21.454 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:21.454 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 11", 2024-09-13T11:27:21.455 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:54.300724+0000", 2024-09-13T11:27:21.455 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000133323 2024-09-13T11:27:21.455 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:21.455 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:21.455 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:27:21.455 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:54.303373+0000", 2024-09-13T11:27:21.455 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0026488050000000002 2024-09-13T11:27:21.455 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:21.455 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:21.455 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:27:21.455 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:54.602650+0000", 2024-09-13T11:27:21.455 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.299277976 2024-09-13T11:27:21.455 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:21.455 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:21.456 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:27:21.456 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:54.602795+0000", 2024-09-13T11:27:21.456 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000144127 2024-09-13T11:27:21.456 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:21.456 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:21.456 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:27:21.456 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:54.602933+0000", 2024-09-13T11:27:21.456 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000138658 2024-09-13T11:27:21.456 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:27:21.456 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:27:21.456 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:27:21.456 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:21.456 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:21.456 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:227066 3.23 3:c67f2f28:::benchmark_data_smithi177_31116_object86763:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e196)", 2024-09-13T11:27:21.457 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:26:10.753624+0000", 2024-09-13T11:27:21.457 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 70.668925689000005, 2024-09-13T11:27:21.457 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.28670347299999999, 2024-09-13T11:27:21.457 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:27:21.457 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:27:21.457 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:27:21.457 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:27:21.457 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:27:21.457 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:27:21.457 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 227066 2024-09-13T11:27:21.457 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:21.457 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:27:21.457 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:21.457 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:27:21.458 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:10.753624+0000", 2024-09-13T11:27:21.458 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:27:21.458 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:21.458 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:21.458 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:27:21.458 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:10.753624+0000", 2024-09-13T11:27:21.458 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:27:21.458 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:21.458 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:21.458 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:27:21.458 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:10.753627+0000", 2024-09-13T11:27:21.458 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.8030000000000001e-06 2024-09-13T11:27:21.458 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:21.458 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:21.459 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:27:21.459 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:10.753634+0000", 2024-09-13T11:27:21.459 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.9659999999999999e-06 2024-09-13T11:27:21.459 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:21.459 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:21.459 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:27:21.459 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:10.753650+0000", 2024-09-13T11:27:21.459 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.6167999999999998e-05 2024-09-13T11:27:21.459 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:21.459 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:21.459 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:27:21.459 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:10.753657+0000", 2024-09-13T11:27:21.459 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.7600000000000002e-06 2024-09-13T11:27:21.459 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:21.460 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:21.460 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:27:21.460 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:10.753741+0000", 2024-09-13T11:27:21.460 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.3523999999999995e-05 2024-09-13T11:27:21.460 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:21.460 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:21.460 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:27:21.460 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:10.753878+0000", 2024-09-13T11:27:21.460 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000137207 2024-09-13T11:27:21.460 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:21.460 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:21.460 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 9", 2024-09-13T11:27:21.460 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:10.754027+0000", 2024-09-13T11:27:21.461 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00014856599999999999 2024-09-13T11:27:21.461 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:21.461 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:21.461 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:27:21.461 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:10.756313+0000", 2024-09-13T11:27:21.461 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.002286045 2024-09-13T11:27:21.461 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:21.461 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:21.461 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:27:21.461 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:11.040266+0000", 2024-09-13T11:27:21.461 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.283953434 2024-09-13T11:27:21.461 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:21.461 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:21.461 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:27:21.461 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:11.040300+0000", 2024-09-13T11:27:21.462 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.4227000000000003e-05 2024-09-13T11:27:21.462 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:21.462 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:21.462 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:27:21.462 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:11.040327+0000", 2024-09-13T11:27:21.462 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.6772999999999999e-05 2024-09-13T11:27:21.462 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:27:21.462 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:27:21.462 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:27:21.462 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:21.462 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:21.462 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:227420 3.23 3:c4add80a:::benchmark_data_smithi177_31116_object87117:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e196)", 2024-09-13T11:27:21.462 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:26:11.037912+0000", 2024-09-13T11:27:21.462 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 70.384637669, 2024-09-13T11:27:21.463 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.33121650499999999, 2024-09-13T11:27:21.463 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:27:21.463 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:27:21.463 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:27:21.463 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:27:21.463 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:27:21.463 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:27:21.463 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 227420 2024-09-13T11:27:21.463 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:21.463 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:27:21.463 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:21.463 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:27:21.463 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:11.037912+0000", 2024-09-13T11:27:21.463 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:27:21.464 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:21.464 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:21.464 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:27:21.464 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:11.037912+0000", 2024-09-13T11:27:21.464 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:27:21.464 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:21.464 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:21.464 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:27:21.464 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:11.037913+0000", 2024-09-13T11:27:21.464 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.606e-06 2024-09-13T11:27:21.464 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:21.464 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:21.464 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:27:21.464 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:11.037918+0000", 2024-09-13T11:27:21.465 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.0509999999999996e-06 2024-09-13T11:27:21.465 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:21.465 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:21.465 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:27:21.465 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:11.037930+0000", 2024-09-13T11:27:21.465 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.1466e-05 2024-09-13T11:27:21.465 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:21.465 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:21.465 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:27:21.465 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:11.037936+0000", 2024-09-13T11:27:21.465 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.5259999999999999e-06 2024-09-13T11:27:21.465 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:21.465 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:21.465 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:27:21.466 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:11.037994+0000", 2024-09-13T11:27:21.466 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.7776999999999999e-05 2024-09-13T11:27:21.466 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:21.466 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:21.466 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:27:21.466 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:11.038129+0000", 2024-09-13T11:27:21.466 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00013500299999999999 2024-09-13T11:27:21.466 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:21.466 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:21.466 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 9", 2024-09-13T11:27:21.466 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:11.038208+0000", 2024-09-13T11:27:21.466 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.9016000000000004e-05 2024-09-13T11:27:21.466 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:21.466 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:21.467 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:27:21.467 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:11.041461+0000", 2024-09-13T11:27:21.467 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00325246 2024-09-13T11:27:21.467 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:21.467 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:21.467 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:27:21.467 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:11.369025+0000", 2024-09-13T11:27:21.467 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.32756474800000002 2024-09-13T11:27:21.467 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:21.467 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:21.467 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:27:21.467 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:11.369086+0000", 2024-09-13T11:27:21.467 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.0140999999999999e-05 2024-09-13T11:27:21.468 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:21.468 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:21.468 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:27:21.468 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:11.369128+0000", 2024-09-13T11:27:21.468 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.2710999999999998e-05 2024-09-13T11:27:21.468 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:27:21.468 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:27:21.468 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:27:21.468 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:21.468 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:21.468 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop_reply(client.4665.0:227420 3.23 e196/90 ondisk, result = 0)", 2024-09-13T11:27:21.468 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:26:11.040152+0000", 2024-09-13T11:27:21.468 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 70.382397831000006, 2024-09-13T11:27:21.468 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.328983201, 2024-09-13T11:27:21.469 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:27:21.469 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:27:21.469 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-09-13T11:27:21.469 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:27:21.469 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:21.469 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:27:21.469 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:11.040152+0000", 2024-09-13T11:27:21.469 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:27:21.469 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:21.469 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:21.469 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:27:21.469 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:11.040152+0000", 2024-09-13T11:27:21.469 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:27:21.469 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:21.470 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:21.470 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:27:21.470 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:11.040157+0000", 2024-09-13T11:27:21.470 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.1309999999999997e-06 2024-09-13T11:27:21.470 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:21.470 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:21.470 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:27:21.470 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:11.040162+0000", 2024-09-13T11:27:21.470 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.8720000000000001e-06 2024-09-13T11:27:21.470 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:21.470 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:21.470 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:27:21.470 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:11.368952+0000", 2024-09-13T11:27:21.470 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.32879022400000002 2024-09-13T11:27:21.471 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:21.471 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:21.471 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:27:21.471 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:11.368960+0000", 2024-09-13T11:27:21.471 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.8460000000000008e-06 2024-09-13T11:27:21.471 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:21.471 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:21.471 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:27:21.471 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:11.369013+0000", 2024-09-13T11:27:21.471 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.3201000000000001e-05 2024-09-13T11:27:21.471 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:21.471 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:21.471 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:27:21.471 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:11.369020+0000", 2024-09-13T11:27:21.472 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.3050000000000004e-06 2024-09-13T11:27:21.472 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:21.472 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:21.472 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:27:21.472 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:11.369135+0000", 2024-09-13T11:27:21.472 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000114622 2024-09-13T11:27:21.472 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:27:21.472 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:27:21.472 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:27:21.472 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:21.472 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:21.472 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:244694 3.27 3:e45f60ea:::benchmark_data_smithi177_31116_object104391:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e208)", 2024-09-13T11:27:21.472 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:26:27.471915+0000", 2024-09-13T11:27:21.472 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 53.950634178000001, 2024-09-13T11:27:21.473 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.28616412299999999, 2024-09-13T11:27:21.473 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:27:21.473 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:27:21.473 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:27:21.473 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:27:21.473 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:27:21.473 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:27:21.473 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 244694 2024-09-13T11:27:21.473 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:21.473 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:27:21.473 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:21.473 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:27:21.473 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:27.471915+0000", 2024-09-13T11:27:21.474 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:27:21.474 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:21.474 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:21.474 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:27:21.474 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:27.471915+0000", 2024-09-13T11:27:21.474 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:27:21.474 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:21.474 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:21.474 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:27:21.474 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:27.471916+0000", 2024-09-13T11:27:21.474 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.8100000000000001e-07 2024-09-13T11:27:21.474 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:21.474 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:21.474 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:27:21.475 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:27.471919+0000", 2024-09-13T11:27:21.475 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.2179999999999999e-06 2024-09-13T11:27:21.475 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:21.475 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:21.475 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:27:21.475 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:27.471928+0000", 2024-09-13T11:27:21.475 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.4279999999999996e-06 2024-09-13T11:27:21.475 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:21.475 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:21.475 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:27:21.475 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:27.471934+0000", 2024-09-13T11:27:21.475 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.3180000000000003e-06 2024-09-13T11:27:21.475 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:21.475 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:21.476 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:27:21.476 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:27.471983+0000", 2024-09-13T11:27:21.476 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.8393000000000002e-05 2024-09-13T11:27:21.476 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:21.476 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:21.476 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:27:21.476 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:27.472085+0000", 2024-09-13T11:27:21.476 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000102723 2024-09-13T11:27:21.476 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:21.476 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:21.476 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 11", 2024-09-13T11:27:21.476 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:27.472218+0000", 2024-09-13T11:27:21.476 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00013276600000000001 2024-09-13T11:27:21.476 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:21.477 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:21.477 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:27:21.477 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:27.474061+0000", 2024-09-13T11:27:21.477 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.001842797 2024-09-13T11:27:21.477 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:21.477 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:21.477 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:27:21.477 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:27.757978+0000", 2024-09-13T11:27:21.477 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.28391740999999998 2024-09-13T11:27:21.477 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:21.477 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:21.477 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:27:21.477 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:27.758033+0000", 2024-09-13T11:27:21.477 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.4997000000000002e-05 2024-09-13T11:27:21.478 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:21.478 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:21.478 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:27:21.478 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:27.758079+0000", 2024-09-13T11:27:21.478 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.6091999999999998e-05 2024-09-13T11:27:21.478 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:27:21.478 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:27:21.478 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:27:21.478 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:21.478 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:21.478 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:266944 3.32 3:4c199f39:::benchmark_data_smithi177_31116_object126641:head [delete] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e227)", 2024-09-13T11:27:21.478 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:26:53.035771+0000", 2024-09-13T11:27:21.478 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 28.386778197000002, 2024-09-13T11:27:21.478 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.30792321099999997, 2024-09-13T11:27:21.478 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:27:21.479 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:27:21.479 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:27:21.479 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:27:21.479 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:27:21.479 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:27:21.479 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 266944 2024-09-13T11:27:21.479 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:21.479 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:27:21.479 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:21.479 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:27:21.479 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:53.035771+0000", 2024-09-13T11:27:21.479 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:27:21.479 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:21.480 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:21.480 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:27:21.480 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:53.035771+0000", 2024-09-13T11:27:21.480 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:27:21.480 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:21.480 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:21.480 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:27:21.480 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:53.035774+0000", 2024-09-13T11:27:21.480 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.4669999999999998e-06 2024-09-13T11:27:21.480 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:21.480 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:21.480 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:27:21.480 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:53.035781+0000", 2024-09-13T11:27:21.480 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.661e-06 2024-09-13T11:27:21.481 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:21.481 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:21.481 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:27:21.481 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:53.035794+0000", 2024-09-13T11:27:21.481 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.2774e-05 2024-09-13T11:27:21.481 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:21.481 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:21.481 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:27:21.481 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:53.035806+0000", 2024-09-13T11:27:21.481 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.1426e-05 2024-09-13T11:27:21.481 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:21.481 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:21.481 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:27:21.482 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:53.035930+0000", 2024-09-13T11:27:21.482 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00012434600000000001 2024-09-13T11:27:21.482 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:21.482 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:21.482 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-09-13T11:27:21.482 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:53.035951+0000", 2024-09-13T11:27:21.482 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.0837000000000001e-05 2024-09-13T11:27:21.482 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:21.482 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:21.482 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:27:21.482 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:53.308946+0000", 2024-09-13T11:27:21.482 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.27299521900000001 2024-09-13T11:27:21.482 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:21.482 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:21.483 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for degraded object", 2024-09-13T11:27:21.483 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:53.309423+0000", 2024-09-13T11:27:21.483 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00047727799999999998 2024-09-13T11:27:21.483 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:21.483 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:21.483 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:27:21.483 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:53.320723+0000", 2024-09-13T11:27:21.483 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.011299774 2024-09-13T11:27:21.483 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:21.483 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:21.483 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:27:21.483 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:53.320867+0000", 2024-09-13T11:27:21.483 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000143511 2024-09-13T11:27:21.483 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:21.484 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:21.484 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 0,10", 2024-09-13T11:27:21.484 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:53.321063+0000", 2024-09-13T11:27:21.484 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000196763 2024-09-13T11:27:21.484 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:21.484 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:21.484 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:27:21.484 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:53.331226+0000", 2024-09-13T11:27:21.484 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.010162492 2024-09-13T11:27:21.484 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:21.484 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:21.484 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:27:21.484 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:53.343345+0000", 2024-09-13T11:27:21.485 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.012119512000000001 2024-09-13T11:27:21.485 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:21.485 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:21.485 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:27:21.485 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:53.343381+0000", 2024-09-13T11:27:21.485 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.5392000000000003e-05 2024-09-13T11:27:21.485 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:21.485 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:21.485 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:27:21.485 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:53.343433+0000", 2024-09-13T11:27:21.485 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.1990000000000002e-05 2024-09-13T11:27:21.485 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:21.485 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:21.485 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:27:21.486 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:53.343694+0000", 2024-09-13T11:27:21.486 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00026176900000000002 2024-09-13T11:27:21.486 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:27:21.486 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:27:21.486 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:27:21.486 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:21.486 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:21.486 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:266995 3.32 3:4edb55ad:::benchmark_data_smithi177_31116_object126692:head [delete] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e227)", 2024-09-13T11:27:21.486 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:26:53.035828+0000", 2024-09-13T11:27:21.486 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 28.386721290000001, 2024-09-13T11:27:21.486 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.30850358999999999, 2024-09-13T11:27:21.486 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:27:21.486 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:27:21.486 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:27:21.487 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:27:21.487 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:27:21.487 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:27:21.487 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 266995 2024-09-13T11:27:21.487 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:21.487 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:27:21.487 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:21.487 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:27:21.487 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:53.035828+0000", 2024-09-13T11:27:21.487 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:27:21.487 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:21.487 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:21.487 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:27:21.487 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:53.035828+0000", 2024-09-13T11:27:21.488 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:27:21.488 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:21.488 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:21.488 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:27:21.488 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:53.035829+0000", 2024-09-13T11:27:21.488 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.0100000000000003e-07 2024-09-13T11:27:21.488 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:21.488 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:21.488 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:27:21.488 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:53.035833+0000", 2024-09-13T11:27:21.488 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.1350000000000002e-06 2024-09-13T11:27:21.488 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:21.488 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:21.488 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:27:21.489 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:53.035841+0000", 2024-09-13T11:27:21.489 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.627e-06 2024-09-13T11:27:21.489 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:21.489 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:21.489 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:27:21.489 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:53.035847+0000", 2024-09-13T11:27:21.489 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.8390000000000003e-06 2024-09-13T11:27:21.489 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:21.489 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:21.489 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:27:21.489 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:53.036011+0000", 2024-09-13T11:27:21.489 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00016409399999999999 2024-09-13T11:27:21.489 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:21.489 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:21.490 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-09-13T11:27:21.490 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:53.036020+0000", 2024-09-13T11:27:21.490 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.533e-06 2024-09-13T11:27:21.490 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:21.490 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:21.490 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:27:21.490 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:53.309896+0000", 2024-09-13T11:27:21.490 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.27387577699999999 2024-09-13T11:27:21.490 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:21.490 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:21.490 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for degraded object", 2024-09-13T11:27:21.490 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:53.310678+0000", 2024-09-13T11:27:21.490 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00078151900000000005 2024-09-13T11:27:21.490 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:21.491 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:21.491 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:27:21.491 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:53.325984+0000", 2024-09-13T11:27:21.491 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.01530629 2024-09-13T11:27:21.491 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:21.491 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:21.491 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:27:21.491 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:53.326144+0000", 2024-09-13T11:27:21.491 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00016016500000000001 2024-09-13T11:27:21.491 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:21.491 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:21.491 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 0,10", 2024-09-13T11:27:21.491 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:53.326254+0000", 2024-09-13T11:27:21.491 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00010958799999999999 2024-09-13T11:27:21.492 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:21.492 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:21.492 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:27:21.492 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:53.334318+0000", 2024-09-13T11:27:21.492 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0080645769999999999 2024-09-13T11:27:21.492 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:21.492 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:21.492 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:27:21.492 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:53.343837+0000", 2024-09-13T11:27:21.492 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0095191330000000008 2024-09-13T11:27:21.492 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:21.492 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:21.492 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:27:21.492 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:53.343952+0000", 2024-09-13T11:27:21.493 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0001149 2024-09-13T11:27:21.493 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:21.493 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:21.493 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:27:21.493 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:53.344009+0000", 2024-09-13T11:27:21.493 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.6665e-05 2024-09-13T11:27:21.493 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:21.493 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:21.493 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:27:21.493 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:53.344332+0000", 2024-09-13T11:27:21.493 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00032284700000000002 2024-09-13T11:27:21.493 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:27:21.493 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:27:21.493 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:27:21.493 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:21.494 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:21.494 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:267198 3.32 3:4f6daeb8:::benchmark_data_smithi177_31116_object126895:head [delete] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e227)", 2024-09-13T11:27:21.494 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:26:53.035978+0000", 2024-09-13T11:27:21.494 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 28.386571551999999, 2024-09-13T11:27:21.494 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.30891107800000001, 2024-09-13T11:27:21.494 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:27:21.494 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:27:21.494 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:27:21.494 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:27:21.494 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:27:21.494 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:27:21.494 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 267198 2024-09-13T11:27:21.494 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:21.494 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:27:21.495 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:21.495 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:27:21.495 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:53.035978+0000", 2024-09-13T11:27:21.495 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:27:21.495 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:21.495 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:21.495 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:27:21.495 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:53.035978+0000", 2024-09-13T11:27:21.495 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:27:21.495 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:21.495 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:21.495 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:27:21.495 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:53.035980+0000", 2024-09-13T11:27:21.496 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.3719999999999999e-06 2024-09-13T11:27:21.496 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:21.496 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:21.496 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:27:21.496 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:53.035990+0000", 2024-09-13T11:27:21.496 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.4769999999999992e-06 2024-09-13T11:27:21.496 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:21.496 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:21.496 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:27:21.496 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:53.036003+0000", 2024-09-13T11:27:21.496 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.3223e-05 2024-09-13T11:27:21.496 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:21.496 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:21.496 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:27:21.497 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:53.036016+0000", 2024-09-13T11:27:21.497 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.2943e-05 2024-09-13T11:27:21.497 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:21.497 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:21.497 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:27:21.497 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:53.036091+0000", 2024-09-13T11:27:21.497 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.5346999999999995e-05 2024-09-13T11:27:21.497 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:21.497 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:21.497 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-09-13T11:27:21.497 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:53.036100+0000", 2024-09-13T11:27:21.497 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.9469999999999992e-06 2024-09-13T11:27:21.497 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:21.497 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:21.498 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:27:21.498 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:53.310844+0000", 2024-09-13T11:27:21.498 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.27474415899999999 2024-09-13T11:27:21.498 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:21.498 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:21.498 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for degraded object", 2024-09-13T11:27:21.498 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:53.312126+0000", 2024-09-13T11:27:21.498 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.001282107 2024-09-13T11:27:21.498 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:21.498 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:21.498 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:27:21.498 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:53.329190+0000", 2024-09-13T11:27:21.498 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.017063389000000002 2024-09-13T11:27:21.499 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:21.499 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:21.499 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:27:21.499 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:53.329276+0000", 2024-09-13T11:27:21.499 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.5692999999999994e-05 2024-09-13T11:27:21.499 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:21.499 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:21.499 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 0,10", 2024-09-13T11:27:21.499 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:53.329383+0000", 2024-09-13T11:27:21.499 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000107717 2024-09-13T11:27:21.499 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:21.499 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:21.499 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:27:21.499 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:53.335683+0000", 2024-09-13T11:27:21.500 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0062998009999999998 2024-09-13T11:27:21.500 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:21.500 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:21.500 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:27:21.500 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:53.344465+0000", 2024-09-13T11:27:21.500 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.008782069 2024-09-13T11:27:21.500 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:21.500 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:21.500 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:27:21.500 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:53.344565+0000", 2024-09-13T11:27:21.500 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.9406000000000003e-05 2024-09-13T11:27:21.500 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:21.500 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:21.500 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:27:21.501 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:53.344623+0000", 2024-09-13T11:27:21.501 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.8476000000000001e-05 2024-09-13T11:27:21.501 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:21.501 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:21.501 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:27:21.501 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:53.344889+0000", 2024-09-13T11:27:21.501 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00026595200000000002 2024-09-13T11:27:21.501 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:27:21.501 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:27:21.501 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:27:21.501 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:21.501 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:21.501 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:267211 3.32 3:4ccc7b14:::benchmark_data_smithi177_31116_object126908:head [delete] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e227)", 2024-09-13T11:27:21.501 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:26:53.036055+0000", 2024-09-13T11:27:21.502 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 28.386494782, 2024-09-13T11:27:21.502 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.30934412999999999, 2024-09-13T11:27:21.502 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:27:21.502 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:27:21.502 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:27:21.502 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:27:21.502 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:27:21.502 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:27:21.502 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 267211 2024-09-13T11:27:21.502 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:21.502 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:27:21.502 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:21.502 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:27:21.502 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:53.036055+0000", 2024-09-13T11:27:21.503 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:27:21.503 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:21.503 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:21.503 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:27:21.503 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:53.036055+0000", 2024-09-13T11:27:21.503 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:27:21.503 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:21.503 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:21.503 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:27:21.503 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:53.036056+0000", 2024-09-13T11:27:21.503 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.46e-06 2024-09-13T11:27:21.503 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:21.503 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:21.503 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:27:21.504 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:53.036061+0000", 2024-09-13T11:27:21.504 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.3079999999999998e-06 2024-09-13T11:27:21.504 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:21.504 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:21.504 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:27:21.504 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:53.036070+0000", 2024-09-13T11:27:21.504 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.8939999999999999e-06 2024-09-13T11:27:21.504 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:21.504 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:21.504 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:27:21.504 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:53.036077+0000", 2024-09-13T11:27:21.504 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.0199999999999997e-06 2024-09-13T11:27:21.504 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:21.504 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:21.505 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:27:21.505 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:53.036163+0000", 2024-09-13T11:27:21.505 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.5993000000000002e-05 2024-09-13T11:27:21.505 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:21.505 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:21.505 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-09-13T11:27:21.505 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:53.036171+0000", 2024-09-13T11:27:21.505 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.199e-06 2024-09-13T11:27:21.505 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:21.505 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:21.505 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:27:21.505 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:53.312204+0000", 2024-09-13T11:27:21.505 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.276033419 2024-09-13T11:27:21.505 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:21.506 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:21.506 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for degraded object", 2024-09-13T11:27:21.506 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:53.312573+0000", 2024-09-13T11:27:21.506 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00036915800000000002 2024-09-13T11:27:21.506 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:21.506 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:21.506 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:27:21.506 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:53.332647+0000", 2024-09-13T11:27:21.506 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.020074195999999999 2024-09-13T11:27:21.506 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:21.506 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:21.506 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:27:21.506 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:53.332792+0000", 2024-09-13T11:27:21.506 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00014504000000000001 2024-09-13T11:27:21.507 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:21.507 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:21.507 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 0,10", 2024-09-13T11:27:21.507 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:53.332995+0000", 2024-09-13T11:27:21.507 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00020285600000000001 2024-09-13T11:27:21.507 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:21.507 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:21.507 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:27:21.507 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:53.339667+0000", 2024-09-13T11:27:21.507 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0066719170000000003 2024-09-13T11:27:21.507 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:21.507 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:21.507 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:27:21.507 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:53.344997+0000", 2024-09-13T11:27:21.508 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0053296000000000003 2024-09-13T11:27:21.508 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:21.508 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:21.508 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:27:21.508 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:53.345086+0000", 2024-09-13T11:27:21.508 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.8980000000000005e-05 2024-09-13T11:27:21.508 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:21.508 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:21.508 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:27:21.508 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:53.345142+0000", 2024-09-13T11:27:21.508 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.6428000000000003e-05 2024-09-13T11:27:21.508 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:21.508 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:21.509 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:27:21.509 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:53.345399+0000", 2024-09-13T11:27:21.509 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000256662 2024-09-13T11:27:21.509 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:27:21.509 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:27:21.509 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:27:21.509 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:21.509 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:21.509 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:267319 3.32 3:4e44814a:::benchmark_data_smithi177_31116_object127016:head [delete] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e227)", 2024-09-13T11:27:21.509 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:26:53.036090+0000", 2024-09-13T11:27:21.509 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 28.386459788, 2024-09-13T11:27:21.509 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.30974317299999998, 2024-09-13T11:27:21.509 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:27:21.509 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:27:21.509 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:27:21.510 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:27:21.510 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:27:21.510 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:27:21.510 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 267319 2024-09-13T11:27:21.510 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:21.510 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:27:21.510 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:21.510 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:27:21.510 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:53.036090+0000", 2024-09-13T11:27:21.510 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:27:21.510 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:21.510 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:21.510 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:27:21.511 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:53.036090+0000", 2024-09-13T11:27:21.511 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:27:21.511 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:21.511 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:21.511 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:27:21.511 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:53.036090+0000", 2024-09-13T11:27:21.511 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.5499999999999997e-07 2024-09-13T11:27:21.511 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:21.511 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:21.511 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:27:21.511 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:53.036099+0000", 2024-09-13T11:27:21.511 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.2290000000000003e-06 2024-09-13T11:27:21.511 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:21.511 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:21.512 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:27:21.512 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:53.036105+0000", 2024-09-13T11:27:21.512 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.781e-06 2024-09-13T11:27:21.512 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:21.512 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:21.512 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:27:21.512 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:53.036112+0000", 2024-09-13T11:27:21.512 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.1090000000000003e-06 2024-09-13T11:27:21.512 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:21.512 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:21.512 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:27:21.512 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:53.036192+0000", 2024-09-13T11:27:21.512 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.0712999999999998e-05 2024-09-13T11:27:21.512 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:21.513 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:21.513 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-09-13T11:27:21.513 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:53.036199+0000", 2024-09-13T11:27:21.513 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.7039999999999997e-06 2024-09-13T11:27:21.513 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:21.513 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:21.513 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:27:21.513 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:53.312645+0000", 2024-09-13T11:27:21.513 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.27644615700000003 2024-09-13T11:27:21.513 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:21.513 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:21.513 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for degraded object", 2024-09-13T11:27:21.513 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:53.313622+0000", 2024-09-13T11:27:21.513 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00097722499999999997 2024-09-13T11:27:21.514 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:21.514 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:21.514 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:27:21.514 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:53.335768+0000", 2024-09-13T11:27:21.514 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.022145821999999999 2024-09-13T11:27:21.514 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:21.514 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:21.514 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:27:21.514 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:53.336028+0000", 2024-09-13T11:27:21.514 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00026027599999999999 2024-09-13T11:27:21.514 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:21.514 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:21.514 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 0,10", 2024-09-13T11:27:21.514 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:53.336417+0000", 2024-09-13T11:27:21.515 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00038850100000000003 2024-09-13T11:27:21.515 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:21.515 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:21.515 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:27:21.515 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:53.340517+0000", 2024-09-13T11:27:21.515 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.004100495 2024-09-13T11:27:21.515 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:21.515 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:21.515 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:27:21.515 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:53.345465+0000", 2024-09-13T11:27:21.515 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.004947094 2024-09-13T11:27:21.515 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:21.515 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:21.515 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:27:21.516 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:53.345512+0000", 2024-09-13T11:27:21.516 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.7120000000000003e-05 2024-09-13T11:27:21.516 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:21.516 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:21.516 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:27:21.516 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:53.345571+0000", 2024-09-13T11:27:21.516 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.9648999999999998e-05 2024-09-13T11:27:21.516 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:21.516 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:21.516 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:27:21.516 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:53.345833+0000", 2024-09-13T11:27:21.516 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00026154299999999999 2024-09-13T11:27:21.516 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:27:21.517 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:27:21.517 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:27:21.517 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:21.517 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:21.517 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:267359 3.32 3:4fbe1345:::benchmark_data_smithi177_31116_object127056:head [delete] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e227)", 2024-09-13T11:27:21.517 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:26:53.036130+0000", 2024-09-13T11:27:21.517 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 28.386419007000001, 2024-09-13T11:27:21.517 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.31064794699999998, 2024-09-13T11:27:21.517 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:27:21.517 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:27:21.517 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:27:21.517 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:27:21.517 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:27:21.517 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:27:21.518 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 267359 2024-09-13T11:27:21.518 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:21.518 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:27:21.518 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:21.518 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:27:21.518 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:53.036130+0000", 2024-09-13T11:27:21.518 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:27:21.518 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:21.518 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:21.518 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:27:21.518 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:53.036130+0000", 2024-09-13T11:27:21.518 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:27:21.518 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:21.518 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:21.519 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:27:21.519 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:53.036131+0000", 2024-09-13T11:27:21.519 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.7199999999999998e-07 2024-09-13T11:27:21.519 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:21.519 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:21.519 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:27:21.519 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:53.036135+0000", 2024-09-13T11:27:21.519 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.6150000000000001e-06 2024-09-13T11:27:21.519 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:21.519 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:21.519 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:27:21.519 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:53.036142+0000", 2024-09-13T11:27:21.519 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.1840000000000002e-06 2024-09-13T11:27:21.519 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:21.520 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:21.520 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:27:21.520 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:53.036148+0000", 2024-09-13T11:27:21.520 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.0839999999999997e-06 2024-09-13T11:27:21.520 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:21.520 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:21.520 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:27:21.520 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:53.036221+0000", 2024-09-13T11:27:21.520 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.2540999999999995e-05 2024-09-13T11:27:21.520 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:21.520 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:21.520 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-09-13T11:27:21.520 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:53.036227+0000", 2024-09-13T11:27:21.521 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.8240000000000002e-06 2024-09-13T11:27:21.521 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:21.521 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:21.521 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:27:21.521 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:53.313744+0000", 2024-09-13T11:27:21.521 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.27751620599999999 2024-09-13T11:27:21.521 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:21.521 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:21.521 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for degraded object", 2024-09-13T11:27:21.521 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:53.314981+0000", 2024-09-13T11:27:21.521 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.001237082 2024-09-13T11:27:21.521 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:21.521 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:21.521 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:27:21.522 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:53.340571+0000", 2024-09-13T11:27:21.522 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.025590695 2024-09-13T11:27:21.522 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:21.522 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:21.522 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:27:21.522 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:53.340735+0000", 2024-09-13T11:27:21.522 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00016394099999999999 2024-09-13T11:27:21.522 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:21.522 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:21.522 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 0,10", 2024-09-13T11:27:21.522 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:53.340953+0000", 2024-09-13T11:27:21.522 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00021797799999999999 2024-09-13T11:27:21.522 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:21.523 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:21.523 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:27:21.523 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:53.344905+0000", 2024-09-13T11:27:21.523 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0039517880000000003 2024-09-13T11:27:21.523 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:21.523 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:21.523 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:27:21.523 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:53.345964+0000", 2024-09-13T11:27:21.523 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.001058701 2024-09-13T11:27:21.523 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:21.523 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:21.523 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:27:21.523 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:53.346095+0000", 2024-09-13T11:27:21.523 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00013125699999999999 2024-09-13T11:27:21.524 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:21.524 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:21.524 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:27:21.524 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:53.346248+0000", 2024-09-13T11:27:21.524 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000152836 2024-09-13T11:27:21.524 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:21.524 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:21.524 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:27:21.524 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:53.346778+0000", 2024-09-13T11:27:21.524 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00053044300000000002 2024-09-13T11:27:21.524 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:27:21.524 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:27:21.524 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:27:21.524 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:21.525 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:21.525 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:273235 3.12 3:4b3e9dde:::benchmark_data_smithi177_31116_object132932:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e231)", 2024-09-13T11:27:21.525 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:26:59.292209+0000", 2024-09-13T11:27:21.525 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 22.130340489000002, 2024-09-13T11:27:21.525 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.34537550099999997, 2024-09-13T11:27:21.525 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:27:21.525 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:27:21.525 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:27:21.525 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:27:21.525 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:27:21.525 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:27:21.525 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 273235 2024-09-13T11:27:21.525 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:21.525 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:27:21.526 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:21.526 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:27:21.526 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:59.292209+0000", 2024-09-13T11:27:21.526 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:27:21.526 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:21.526 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:21.526 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:27:21.526 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:59.292209+0000", 2024-09-13T11:27:21.526 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:27:21.526 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:21.526 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:21.526 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:27:21.526 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:59.292212+0000", 2024-09-13T11:27:21.526 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.5490000000000001e-06 2024-09-13T11:27:21.527 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:21.527 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:21.527 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:27:21.527 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:59.292218+0000", 2024-09-13T11:27:21.527 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.1430000000000003e-06 2024-09-13T11:27:21.527 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:21.527 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:21.527 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:27:21.527 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:59.292232+0000", 2024-09-13T11:27:21.527 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.4078e-05 2024-09-13T11:27:21.527 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:21.527 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:21.527 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:27:21.527 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:59.292241+0000", 2024-09-13T11:27:21.528 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.7800000000000006e-06 2024-09-13T11:27:21.528 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:21.528 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:21.528 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:27:21.528 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:59.292296+0000", 2024-09-13T11:27:21.528 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.5414000000000002e-05 2024-09-13T11:27:21.528 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:21.528 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:21.528 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:27:21.528 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:59.292464+0000", 2024-09-13T11:27:21.528 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000168548 2024-09-13T11:27:21.528 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:21.528 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:21.528 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 7,14", 2024-09-13T11:27:21.529 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:59.292611+0000", 2024-09-13T11:27:21.529 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00014622499999999999 2024-09-13T11:27:21.529 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:21.529 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:21.529 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:27:21.529 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:59.295250+0000", 2024-09-13T11:27:21.529 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0026396290000000001 2024-09-13T11:27:21.529 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:21.529 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:21.529 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:27:21.529 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:59.295347+0000", 2024-09-13T11:27:21.529 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.6712000000000005e-05 2024-09-13T11:27:21.529 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:21.529 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:21.530 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:27:21.530 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:59.637449+0000", 2024-09-13T11:27:21.530 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.34210160899999997 2024-09-13T11:27:21.530 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:21.530 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:21.530 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:27:21.530 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:59.637494+0000", 2024-09-13T11:27:21.530 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.5775999999999997e-05 2024-09-13T11:27:21.530 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:21.530 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:21.530 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:27:21.530 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:59.637584+0000", 2024-09-13T11:27:21.530 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.0037999999999994e-05 2024-09-13T11:27:21.530 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:27:21.531 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:27:21.531 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:27:21.531 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:21.531 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:21.531 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:273235 3.12 3:4b3e9dde:::benchmark_data_smithi177_31116_object132932:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e231)", 2024-09-13T11:27:21.531 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:26:59.297179+0000", 2024-09-13T11:27:21.531 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 22.125370526000001, 2024-09-13T11:27:21.531 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.34032614700000002, 2024-09-13T11:27:21.531 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:27:21.531 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:27:21.531 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "waiting for ondisk", 2024-09-13T11:27:21.531 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:27:21.531 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:27:21.531 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:27:21.532 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 273235 2024-09-13T11:27:21.532 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:21.532 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:27:21.532 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:21.532 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:27:21.532 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:59.297179+0000", 2024-09-13T11:27:21.532 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:27:21.532 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:21.532 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:21.532 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:27:21.532 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:59.297179+0000", 2024-09-13T11:27:21.532 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:27:21.532 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:21.532 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:21.533 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:27:21.533 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:59.297179+0000", 2024-09-13T11:27:21.533 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.89e-07 2024-09-13T11:27:21.533 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:21.533 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:21.533 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:27:21.533 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:59.297181+0000", 2024-09-13T11:27:21.533 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.0590000000000001e-06 2024-09-13T11:27:21.533 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:21.533 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:21.533 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:27:21.533 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:59.297186+0000", 2024-09-13T11:27:21.533 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.5630000000000002e-06 2024-09-13T11:27:21.533 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:21.534 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:21.534 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:27:21.534 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:59.297190+0000", 2024-09-13T11:27:21.534 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.8160000000000004e-06 2024-09-13T11:27:21.534 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:21.534 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:21.534 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:27:21.534 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:59.297583+0000", 2024-09-13T11:27:21.534 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00039354100000000002 2024-09-13T11:27:21.534 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:21.534 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:21.534 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for ondisk", 2024-09-13T11:27:21.534 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:59.297634+0000", 2024-09-13T11:27:21.534 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.0722e-05 2024-09-13T11:27:21.535 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:21.535 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:21.535 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:27:21.535 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:26:59.637505+0000", 2024-09-13T11:27:21.535 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.33987095699999997 2024-09-13T11:27:21.535 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:27:21.535 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:27:21.535 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:27:21.535 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:21.535 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:21.536 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "MOSDPGRecoveryDelete(3.3b e241,237 3:df60fe88:::benchmark_data_smithi177_31116_object26630:head,159'6976)", 2024-09-13T11:27:21.536 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:27:10.482925+0000", 2024-09-13T11:27:21.536 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 10.939624053999999, 2024-09-13T11:27:21.536 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.716230056, 2024-09-13T11:27:21.536 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:27:21.536 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:27:21.536 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-09-13T11:27:21.536 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:27:21.536 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:21.536 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:27:21.536 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:27:10.482925+0000", 2024-09-13T11:27:21.536 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:27:21.536 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:21.537 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:21.537 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:27:21.537 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:27:10.482925+0000", 2024-09-13T11:27:21.537 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:27:21.537 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:21.537 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:21.537 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:27:21.537 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:27:10.482928+0000", 2024-09-13T11:27:21.537 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.1239999999999999e-06 2024-09-13T11:27:21.537 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:21.537 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:21.537 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:27:21.537 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:27:10.482938+0000", 2024-09-13T11:27:21.537 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0093e-05 2024-09-13T11:27:21.538 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:21.538 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:21.538 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:27:21.538 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:27:11.198174+0000", 2024-09-13T11:27:21.538 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.71523653899999995 2024-09-13T11:27:21.538 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:21.538 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:21.538 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:27:21.538 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:27:11.198186+0000", 2024-09-13T11:27:21.538 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.2071e-05 2024-09-13T11:27:21.538 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:21.538 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:21.538 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:27:21.538 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:27:11.198922+0000", 2024-09-13T11:27:21.539 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00073529300000000004 2024-09-13T11:27:21.539 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:21.539 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:21.539 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:27:21.539 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:27:11.198950+0000", 2024-09-13T11:27:21.539 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.8293000000000002e-05 2024-09-13T11:27:21.539 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:21.539 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:21.539 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:27:21.539 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:27:11.199155+0000", 2024-09-13T11:27:21.539 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00020564300000000001 2024-09-13T11:27:21.539 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:27:21.539 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:27:21.539 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:27:21.540 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:21.540 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:21.540 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "MOSDPGRecoveryDelete(3.38 e242,239 3:1cece8e7:::benchmark_data_smithi177_31116_object34010:head,165'6790)", 2024-09-13T11:27:21.540 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:27:12.396619+0000", 2024-09-13T11:27:21.540 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 9.0259305110000003, 2024-09-13T11:27:21.540 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.69161329400000005, 2024-09-13T11:27:21.540 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:27:21.540 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:27:21.540 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-09-13T11:27:21.540 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:27:21.540 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:21.540 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:27:21.540 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:27:12.396619+0000", 2024-09-13T11:27:21.541 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:27:21.541 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:21.541 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:21.541 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:27:21.541 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:27:12.396619+0000", 2024-09-13T11:27:21.541 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:27:21.541 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:21.541 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:21.541 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:27:21.541 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:27:12.396621+0000", 2024-09-13T11:27:21.541 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.688e-06 2024-09-13T11:27:21.541 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:21.541 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:21.541 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:27:21.542 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:27:12.396626+0000", 2024-09-13T11:27:21.542 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.0950000000000003e-06 2024-09-13T11:27:21.542 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:21.542 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:21.542 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:27:21.542 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:27:13.088038+0000", 2024-09-13T11:27:21.542 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.69141244700000004 2024-09-13T11:27:21.542 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:21.542 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:21.542 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:27:21.542 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:27:13.088046+0000", 2024-09-13T11:27:21.542 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.6510000000000001e-06 2024-09-13T11:27:21.542 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:21.542 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:21.543 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:27:21.543 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:27:13.088141+0000", 2024-09-13T11:27:21.543 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.5157999999999999e-05 2024-09-13T11:27:21.543 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:21.543 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:21.543 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:27:21.543 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:27:13.088154+0000", 2024-09-13T11:27:21.543 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.253e-05 2024-09-13T11:27:21.543 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:21.543 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:21.543 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:27:21.543 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:27:13.088232+0000", 2024-09-13T11:27:21.543 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.8725000000000003e-05 2024-09-13T11:27:21.544 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:27:21.544 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:27:21.544 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:27:21.544 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:27:21.544 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:27:21.544 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-09-13T11:27:21.545 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-09-13T11:27:21.546 INFO:tasks.ceph.osd.14.smithi162.stderr:2024-09-13T11:27:21.543+0000 7f3df5903640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33154) UID: 0 2024-09-13T11:27:21.646 INFO:tasks.ceph.osd.12.smithi001.stderr:2024-09-13T11:27:21.642+0000 7f892c6e6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26314) UID: 0 2024-09-13T11:27:21.692 INFO:teuthology.orchestra.run.smithi092.stdout:ERROR: (22) Invalid argument 2024-09-13T11:27:21.693 INFO:teuthology.orchestra.run.smithi092.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:27:21.709 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-09-13T11:27:21.746 INFO:tasks.ceph.osd.11.smithi177.stderr:2024-09-13T11:27:21.744+0000 7f24792c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26580) UID: 0 2024-09-13T11:27:21.847 INFO:tasks.ceph.osd.3.smithi177.stderr:2024-09-13T11:27:21.844+0000 7f10adba4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26574) UID: 0 2024-09-13T11:27:21.849 INFO:teuthology.orchestra.run.smithi092.stdout:ERROR: (22) Invalid argument 2024-09-13T11:27:21.849 INFO:teuthology.orchestra.run.smithi092.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:27:21.867 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-09-13T11:27:21.888 INFO:tasks.ceph.ceph_manager.ceph:removing pool_name unique_pool_0 2024-09-13T11:27:21.888 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm unique_pool_0 unique_pool_0 --yes-i-really-really-mean-it 2024-09-13T11:27:21.948 INFO:tasks.ceph.osd.11.smithi177.stderr:2024-09-13T11:27:21.944+0000 7f24792c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26580) UID: 0 2024-09-13T11:27:21.965 INFO:tasks.ceph.osd.14.smithi162.stderr:2024-09-13T11:27:21.963+0000 7f3dec334640 -1 osd.14 250 handle_osd_map injecting map crc failure 2024-09-13T11:27:22.009 INFO:teuthology.orchestra.run.smithi092.stdout:ERROR: (22) Invalid argument 2024-09-13T11:27:22.009 INFO:teuthology.orchestra.run.smithi092.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:27:22.032 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_ops_in_flight 2024-09-13T11:27:22.049 INFO:tasks.ceph.osd.4.smithi001.stderr:2024-09-13T11:27:22.046+0000 7ff8e0218640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26310) UID: 0 2024-09-13T11:27:22.050 INFO:tasks.ceph.osd.15.smithi177.stderr:2024-09-13T11:27:22.044+0000 7f9d618a4640 -1 osd.15 250 handle_osd_map injecting map crc failure 2024-09-13T11:27:22.150 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-09-13T11:27:22.147+0000 7f97befc9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26315) UID: 0 2024-09-13T11:27:22.184 INFO:teuthology.orchestra.run.smithi092.stdout:ERROR: (22) Invalid argument 2024-09-13T11:27:22.184 INFO:teuthology.orchestra.run.smithi092.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:27:22.205 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_blocked_ops 2024-09-13T11:27:22.224 INFO:teuthology.orchestra.run.smithi001.stdout:osd.5: {} 2024-09-13T11:27:22.224 INFO:teuthology.orchestra.run.smithi001.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T11:27:22.250 INFO:tasks.ceph.osd.8.smithi001.stderr:2024-09-13T11:27:22.246+0000 7fb9b48ae640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26312) UID: 0 2024-09-13T11:27:22.256 INFO:teuthology.orchestra.run.smithi001.stdout:osd.6: {} 2024-09-13T11:27:22.256 INFO:teuthology.orchestra.run.smithi001.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T11:27:22.291 INFO:teuthology.orchestra.run.smithi001.stdout:osd.7: {} 2024-09-13T11:27:22.291 INFO:teuthology.orchestra.run.smithi001.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T11:27:22.322 INFO:teuthology.orchestra.run.smithi001.stdout:osd.8: {} 2024-09-13T11:27:22.323 INFO:teuthology.orchestra.run.smithi001.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-13T11:27:22.347 INFO:teuthology.orchestra.run.smithi092.stdout:ERROR: (22) Invalid argument 2024-09-13T11:27:22.347 INFO:teuthology.orchestra.run.smithi092.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:27:22.351 INFO:tasks.ceph.osd.13.smithi092.stderr:2024-09-13T11:27:22.347+0000 7f3e5afea640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26350) UID: 0 2024-09-13T11:27:22.354 INFO:teuthology.orchestra.run.smithi001.stdout:osd.9: {} 2024-09-13T11:27:22.354 INFO:teuthology.orchestra.run.smithi001.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-13T11:27:22.354 INFO:teuthology.orchestra.run.smithi001.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-09-13T11:27:22.367 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_historic_ops 2024-09-13T11:27:22.383 INFO:teuthology.orchestra.run.smithi001.stdout:osd.11: {} 2024-09-13T11:27:22.384 INFO:teuthology.orchestra.run.smithi001.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-13T11:27:22.412 INFO:teuthology.orchestra.run.smithi001.stdout:osd.12: {} 2024-09-13T11:27:22.412 INFO:teuthology.orchestra.run.smithi001.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-13T11:27:22.441 INFO:teuthology.orchestra.run.smithi001.stdout:osd.13: {} 2024-09-13T11:27:22.441 INFO:teuthology.orchestra.run.smithi001.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-13T11:27:22.451 INFO:tasks.ceph.osd.7.smithi177.stderr:2024-09-13T11:27:22.448+0000 7f897b51d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26575) UID: 0 2024-09-13T11:27:22.469 INFO:teuthology.orchestra.run.smithi001.stdout:osd.14: {} 2024-09-13T11:27:22.469 INFO:teuthology.orchestra.run.smithi001.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-13T11:27:22.496 INFO:teuthology.orchestra.run.smithi001.stdout:osd.15: {} 2024-09-13T11:27:22.497 INFO:teuthology.orchestra.run.smithi001.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-13T11:27:22.500 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-13T11:27:22.500 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-13T11:27:22.553 INFO:tasks.ceph.osd.7.smithi177.stderr:2024-09-13T11:27:22.548+0000 7f897b51d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26575) UID: 0 2024-09-13T11:27:22.555 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-09-13T11:27:22.555 INFO:teuthology.orchestra.run.smithi092.stdout: "size": 20, 2024-09-13T11:27:22.555 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 600, 2024-09-13T11:27:22.555 INFO:teuthology.orchestra.run.smithi092.stdout: "ops": [ 2024-09-13T11:27:22.555 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.555 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:79 3.1 3:841df5b5:::benchmark_data_smithi177_31116_object78:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-09-13T11:27:22.555 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:27.056083+0000", 2024-09-13T11:27:22.556 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 295.497561577, 2024-09-13T11:27:22.556 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.89469730199999997, 2024-09-13T11:27:22.556 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:27:22.556 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:27:22.556 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:27:22.556 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:27:22.556 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:27:22.556 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:27:22.556 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 79 2024-09-13T11:27:22.556 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:22.556 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:27:22.556 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.556 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:27:22.557 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.056083+0000", 2024-09-13T11:27:22.557 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:27:22.557 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:22.557 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.557 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:27:22.557 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.056083+0000", 2024-09-13T11:27:22.557 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:27:22.557 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:22.557 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.557 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:27:22.557 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.056087+0000", 2024-09-13T11:27:22.557 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.0659999999999997e-06 2024-09-13T11:27:22.557 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:22.557 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.557 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:27:22.558 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.056148+0000", 2024-09-13T11:27:22.558 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.1376000000000003e-05 2024-09-13T11:27:22.558 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:22.558 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.558 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:27:22.558 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.056180+0000", 2024-09-13T11:27:22.558 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.2370999999999999e-05 2024-09-13T11:27:22.558 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:22.558 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.558 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:27:22.558 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.056195+0000", 2024-09-13T11:27:22.558 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.5119e-05 2024-09-13T11:27:22.558 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:22.558 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.559 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:27:22.559 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.056304+0000", 2024-09-13T11:27:22.559 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000108365 2024-09-13T11:27:22.559 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:22.559 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.559 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:27:22.559 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.056478+0000", 2024-09-13T11:27:22.559 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000174542 2024-09-13T11:27:22.559 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:22.559 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.559 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 11", 2024-09-13T11:27:22.559 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.056690+0000", 2024-09-13T11:27:22.559 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00021167500000000001 2024-09-13T11:27:22.559 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:22.560 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.560 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:27:22.560 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.058063+0000", 2024-09-13T11:27:22.560 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0013731819999999999 2024-09-13T11:27:22.560 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:22.560 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.560 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:27:22.560 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.950477+0000", 2024-09-13T11:27:22.560 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.89241369400000004 2024-09-13T11:27:22.560 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:22.560 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.560 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:27:22.560 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.950652+0000", 2024-09-13T11:27:22.561 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00017526400000000001 2024-09-13T11:27:22.561 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:22.561 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.561 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:27:22.561 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:27.950780+0000", 2024-09-13T11:27:22.561 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00012764800000000001 2024-09-13T11:27:22.561 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:27:22.561 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:27:22.561 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:27:22.561 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:22.561 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.561 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:7 3.b 3:de78393b:::benchmark_data_smithi177_31116_object6:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T11:27:22.561 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:48.313952+0000", 2024-09-13T11:27:22.561 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 274.23969182799999, 2024-09-13T11:27:22.562 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.86421288600000001, 2024-09-13T11:27:22.562 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:27:22.562 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:27:22.562 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:27:22.562 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:27:22.562 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:27:22.562 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:27:22.562 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 7 2024-09-13T11:27:22.562 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:22.562 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:27:22.562 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.562 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:27:22.562 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.313952+0000", 2024-09-13T11:27:22.563 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:27:22.563 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:22.563 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.563 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:27:22.563 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.313952+0000", 2024-09-13T11:27:22.563 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:27:22.563 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:22.563 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.563 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:27:22.563 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.313957+0000", 2024-09-13T11:27:22.563 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.3599999999999998e-06 2024-09-13T11:27:22.563 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:22.563 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.563 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:27:22.564 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314043+0000", 2024-09-13T11:27:22.564 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.6766000000000002e-05 2024-09-13T11:27:22.564 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:22.564 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.564 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:27:22.564 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314065+0000", 2024-09-13T11:27:22.564 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.1416e-05 2024-09-13T11:27:22.564 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:22.564 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.564 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:27:22.564 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314080+0000", 2024-09-13T11:27:22.564 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.5519000000000002e-05 2024-09-13T11:27:22.564 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:22.564 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.565 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:27:22.565 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315478+0000", 2024-09-13T11:27:22.565 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.001397765 2024-09-13T11:27:22.565 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:22.565 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.565 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-09-13T11:27:22.565 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315507+0000", 2024-09-13T11:27:22.565 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.9128999999999998e-05 2024-09-13T11:27:22.565 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:22.565 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.565 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:27:22.565 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.167978+0000", 2024-09-13T11:27:22.565 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.85247068400000003 2024-09-13T11:27:22.565 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:22.566 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.566 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:27:22.566 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.168307+0000", 2024-09-13T11:27:22.566 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00032886699999999999 2024-09-13T11:27:22.566 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:22.566 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.566 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1", 2024-09-13T11:27:22.566 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.168647+0000", 2024-09-13T11:27:22.566 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00033998399999999999 2024-09-13T11:27:22.566 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:22.566 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.566 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:27:22.566 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.178011+0000", 2024-09-13T11:27:22.566 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0093639169999999994 2024-09-13T11:27:22.566 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:22.567 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.567 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:27:22.567 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.178026+0000", 2024-09-13T11:27:22.567 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.5274999999999999e-05 2024-09-13T11:27:22.567 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:22.567 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.567 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:27:22.567 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.178080+0000", 2024-09-13T11:27:22.567 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.3601999999999999e-05 2024-09-13T11:27:22.567 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:22.567 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.567 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:27:22.567 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.178165+0000", 2024-09-13T11:27:22.567 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.5601999999999998e-05 2024-09-13T11:27:22.568 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:27:22.568 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:27:22.568 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:27:22.568 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:22.568 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.568 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:9 3.b 3:dafadff6:::benchmark_data_smithi177_31116_object8:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T11:27:22.568 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:48.314114+0000", 2024-09-13T11:27:22.568 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 274.23953001400002, 2024-09-13T11:27:22.568 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.86613615899999996, 2024-09-13T11:27:22.568 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:27:22.568 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:27:22.568 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:27:22.568 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:27:22.568 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:27:22.569 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:27:22.569 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 9 2024-09-13T11:27:22.569 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:22.569 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:27:22.569 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.569 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:27:22.569 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314114+0000", 2024-09-13T11:27:22.569 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:27:22.569 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:22.569 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.569 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:27:22.569 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314114+0000", 2024-09-13T11:27:22.569 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:27:22.569 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:22.570 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.570 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:27:22.570 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314117+0000", 2024-09-13T11:27:22.570 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.3860000000000001e-06 2024-09-13T11:27:22.570 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:22.570 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.570 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:27:22.570 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314215+0000", 2024-09-13T11:27:22.570 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.8900000000000005e-05 2024-09-13T11:27:22.570 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:22.570 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.570 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:27:22.570 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314229+0000", 2024-09-13T11:27:22.570 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.3444999999999999e-05 2024-09-13T11:27:22.571 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:22.571 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.571 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:27:22.571 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314239+0000", 2024-09-13T11:27:22.571 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.8470000000000008e-06 2024-09-13T11:27:22.571 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:22.571 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.571 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:27:22.571 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315648+0000", 2024-09-13T11:27:22.571 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.001409291 2024-09-13T11:27:22.571 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:22.571 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.571 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-09-13T11:27:22.571 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315671+0000", 2024-09-13T11:27:22.572 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.3169000000000001e-05 2024-09-13T11:27:22.572 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:22.572 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.572 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:27:22.572 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.176036+0000", 2024-09-13T11:27:22.572 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.86036497499999998 2024-09-13T11:27:22.572 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:22.572 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.572 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:27:22.572 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.176189+0000", 2024-09-13T11:27:22.572 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00015271100000000001 2024-09-13T11:27:22.572 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:22.572 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.572 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1", 2024-09-13T11:27:22.573 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.176339+0000", 2024-09-13T11:27:22.573 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00015057499999999999 2024-09-13T11:27:22.573 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:22.573 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.573 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:27:22.573 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.180148+0000", 2024-09-13T11:27:22.573 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.003808891 2024-09-13T11:27:22.573 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:22.573 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.573 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:27:22.573 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.180168+0000", 2024-09-13T11:27:22.573 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.9507000000000001e-05 2024-09-13T11:27:22.573 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:22.573 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.574 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:27:22.574 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.180213+0000", 2024-09-13T11:27:22.574 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.5504999999999999e-05 2024-09-13T11:27:22.574 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:22.574 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.574 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:27:22.574 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.180250+0000", 2024-09-13T11:27:22.574 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.6956999999999999e-05 2024-09-13T11:27:22.574 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:27:22.574 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:27:22.574 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:27:22.574 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:22.574 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.574 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:10 3.b 3:dbe028a8:::benchmark_data_smithi177_31116_object9:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T11:27:22.575 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:48.314259+0000", 2024-09-13T11:27:22.575 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 274.239384884, 2024-09-13T11:27:22.575 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.86637313599999999, 2024-09-13T11:27:22.575 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:27:22.575 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:27:22.575 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:27:22.575 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:27:22.575 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:27:22.575 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:27:22.575 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 10 2024-09-13T11:27:22.575 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:22.575 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:27:22.575 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.575 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:27:22.576 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314259+0000", 2024-09-13T11:27:22.576 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:27:22.576 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:22.576 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.576 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:27:22.576 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314259+0000", 2024-09-13T11:27:22.576 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:27:22.576 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:22.576 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.576 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:27:22.576 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314261+0000", 2024-09-13T11:27:22.576 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.1500000000000002e-06 2024-09-13T11:27:22.576 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:22.577 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.577 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:27:22.577 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314349+0000", 2024-09-13T11:27:22.577 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.8070000000000002e-05 2024-09-13T11:27:22.577 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:22.577 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.577 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:27:22.577 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314365+0000", 2024-09-13T11:27:22.577 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.5835999999999999e-05 2024-09-13T11:27:22.577 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:22.577 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.577 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:27:22.577 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314377+0000", 2024-09-13T11:27:22.577 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.1589e-05 2024-09-13T11:27:22.578 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:22.578 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.578 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:27:22.578 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315788+0000", 2024-09-13T11:27:22.578 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0014109579999999999 2024-09-13T11:27:22.578 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:22.578 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.578 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-09-13T11:27:22.578 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315809+0000", 2024-09-13T11:27:22.578 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.0985999999999999e-05 2024-09-13T11:27:22.578 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:22.578 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.578 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:27:22.578 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.176822+0000", 2024-09-13T11:27:22.578 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.86101310799999997 2024-09-13T11:27:22.579 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:22.579 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.579 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:27:22.579 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.177021+0000", 2024-09-13T11:27:22.579 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00019911199999999999 2024-09-13T11:27:22.579 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:22.579 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.579 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1", 2024-09-13T11:27:22.579 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.177226+0000", 2024-09-13T11:27:22.579 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00020516600000000001 2024-09-13T11:27:22.579 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:22.579 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.579 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:27:22.579 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.180264+0000", 2024-09-13T11:27:22.580 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0030381739999999998 2024-09-13T11:27:22.580 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:22.580 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.580 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:27:22.580 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.180387+0000", 2024-09-13T11:27:22.580 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00012257699999999999 2024-09-13T11:27:22.580 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:22.580 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.580 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:27:22.580 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.180518+0000", 2024-09-13T11:27:22.580 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00013135800000000001 2024-09-13T11:27:22.580 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:22.580 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.580 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:27:22.581 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.180632+0000", 2024-09-13T11:27:22.581 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000114052 2024-09-13T11:27:22.581 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:27:22.581 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:27:22.581 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:27:22.581 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:22.581 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.581 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:39 3.b 3:d91854b3:::benchmark_data_smithi177_31116_object38:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T11:27:22.581 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:48.314393+0000", 2024-09-13T11:27:22.581 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 274.23925134199999, 2024-09-13T11:27:22.581 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.86850121700000005, 2024-09-13T11:27:22.581 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:27:22.581 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:27:22.581 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:27:22.581 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:27:22.582 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:27:22.582 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:27:22.582 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 39 2024-09-13T11:27:22.582 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:22.582 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:27:22.582 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.582 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:27:22.582 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314393+0000", 2024-09-13T11:27:22.582 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:27:22.582 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:22.582 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.582 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:27:22.582 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314393+0000", 2024-09-13T11:27:22.582 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:27:22.583 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:22.583 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.583 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:27:22.583 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314394+0000", 2024-09-13T11:27:22.583 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.513e-06 2024-09-13T11:27:22.583 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:22.583 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.583 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:27:22.583 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314463+0000", 2024-09-13T11:27:22.583 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.8261000000000003e-05 2024-09-13T11:27:22.583 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:22.583 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.583 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:27:22.583 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314477+0000", 2024-09-13T11:27:22.584 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.4619e-05 2024-09-13T11:27:22.584 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:22.584 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.584 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:27:22.584 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314485+0000", 2024-09-13T11:27:22.584 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.1699999999999997e-06 2024-09-13T11:27:22.584 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:22.584 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.584 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:27:22.584 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315917+0000", 2024-09-13T11:27:22.584 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0014316089999999999 2024-09-13T11:27:22.584 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:22.584 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.584 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-09-13T11:27:22.585 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315933+0000", 2024-09-13T11:27:22.585 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.5999999999999999e-05 2024-09-13T11:27:22.585 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:22.585 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.585 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:27:22.585 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.178205+0000", 2024-09-13T11:27:22.585 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.86227247100000004 2024-09-13T11:27:22.585 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:22.585 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.585 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:27:22.585 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.178316+0000", 2024-09-13T11:27:22.585 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000110788 2024-09-13T11:27:22.585 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:22.585 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.586 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1", 2024-09-13T11:27:22.586 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.178448+0000", 2024-09-13T11:27:22.586 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000132056 2024-09-13T11:27:22.586 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:22.586 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.586 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:27:22.586 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.182721+0000", 2024-09-13T11:27:22.586 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0042729780000000002 2024-09-13T11:27:22.586 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:22.586 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.586 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:27:22.586 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.182801+0000", 2024-09-13T11:27:22.586 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.0209000000000004e-05 2024-09-13T11:27:22.586 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:22.587 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.587 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:27:22.587 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.182855+0000", 2024-09-13T11:27:22.587 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.3749000000000003e-05 2024-09-13T11:27:22.587 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:22.587 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.587 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:27:22.587 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.182894+0000", 2024-09-13T11:27:22.587 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.8794000000000002e-05 2024-09-13T11:27:22.587 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:27:22.587 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:27:22.587 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:27:22.587 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:22.588 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.588 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:50 3.b 3:d1c144b8:::benchmark_data_smithi177_31116_object49:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T11:27:22.588 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:48.314498+0000", 2024-09-13T11:27:22.588 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 274.23914592699998, 2024-09-13T11:27:22.588 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.86878673299999998, 2024-09-13T11:27:22.588 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:27:22.588 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:27:22.588 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:27:22.588 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:27:22.588 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:27:22.588 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:27:22.588 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 50 2024-09-13T11:27:22.588 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:22.588 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:27:22.589 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.589 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:27:22.589 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314498+0000", 2024-09-13T11:27:22.589 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:27:22.589 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:22.589 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.589 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:27:22.589 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314498+0000", 2024-09-13T11:27:22.589 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:27:22.589 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:22.589 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.589 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:27:22.589 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314499+0000", 2024-09-13T11:27:22.589 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.1969999999999999e-06 2024-09-13T11:27:22.590 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:22.590 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.590 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:27:22.590 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314565+0000", 2024-09-13T11:27:22.590 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.5628000000000003e-05 2024-09-13T11:27:22.590 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:22.590 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.590 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:27:22.590 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314579+0000", 2024-09-13T11:27:22.590 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.4222e-05 2024-09-13T11:27:22.590 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:22.590 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.590 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:27:22.590 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314590+0000", 2024-09-13T11:27:22.591 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.1240000000000001e-05 2024-09-13T11:27:22.591 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:22.591 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.591 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:27:22.591 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316060+0000", 2024-09-13T11:27:22.591 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.001469515 2024-09-13T11:27:22.591 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:22.591 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.591 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-09-13T11:27:22.591 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316076+0000", 2024-09-13T11:27:22.591 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.5608999999999999e-05 2024-09-13T11:27:22.591 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:22.591 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.591 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:27:22.592 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.178857+0000", 2024-09-13T11:27:22.592 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.86278099900000005 2024-09-13T11:27:22.592 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:22.592 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.592 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:27:22.592 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.178961+0000", 2024-09-13T11:27:22.592 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000104664 2024-09-13T11:27:22.592 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:22.592 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.592 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1", 2024-09-13T11:27:22.592 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.179123+0000", 2024-09-13T11:27:22.592 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000161738 2024-09-13T11:27:22.592 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:22.592 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.593 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:27:22.593 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.182755+0000", 2024-09-13T11:27:22.593 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0036323200000000001 2024-09-13T11:27:22.593 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:22.593 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.593 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:27:22.593 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.183023+0000", 2024-09-13T11:27:22.593 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00026749100000000002 2024-09-13T11:27:22.593 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:22.593 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.593 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:27:22.593 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.183168+0000", 2024-09-13T11:27:22.593 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00014499099999999999 2024-09-13T11:27:22.593 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:22.594 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.594 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:27:22.594 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.183285+0000", 2024-09-13T11:27:22.594 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000117119 2024-09-13T11:27:22.594 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:27:22.594 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:27:22.594 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:27:22.594 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:22.594 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.594 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:60 3.b 3:d67218b0:::benchmark_data_smithi177_31116_object59:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T11:27:22.594 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:48.314603+0000", 2024-09-13T11:27:22.594 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 274.239041364, 2024-09-13T11:27:22.594 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.87106201100000002, 2024-09-13T11:27:22.594 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:27:22.594 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:27:22.595 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:27:22.595 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:27:22.595 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:27:22.595 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:27:22.595 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 60 2024-09-13T11:27:22.595 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:22.595 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:27:22.595 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.595 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:27:22.595 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314603+0000", 2024-09-13T11:27:22.595 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:27:22.595 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:22.595 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.595 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:27:22.596 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314603+0000", 2024-09-13T11:27:22.596 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:27:22.596 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:22.596 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.596 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:27:22.596 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314604+0000", 2024-09-13T11:27:22.596 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.2359999999999999e-06 2024-09-13T11:27:22.596 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:22.596 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.596 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:27:22.596 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314667+0000", 2024-09-13T11:27:22.596 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.3077999999999995e-05 2024-09-13T11:27:22.596 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:22.597 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.597 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:27:22.597 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314699+0000", 2024-09-13T11:27:22.597 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.1965999999999999e-05 2024-09-13T11:27:22.597 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:22.597 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.597 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:27:22.597 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314710+0000", 2024-09-13T11:27:22.597 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.1148999999999999e-05 2024-09-13T11:27:22.597 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:22.597 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.597 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:27:22.597 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316291+0000", 2024-09-13T11:27:22.597 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0015804440000000001 2024-09-13T11:27:22.598 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:22.598 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.598 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-09-13T11:27:22.598 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316310+0000", 2024-09-13T11:27:22.598 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.9799000000000001e-05 2024-09-13T11:27:22.598 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:22.598 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.598 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:27:22.598 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.179555+0000", 2024-09-13T11:27:22.598 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.86324469500000001 2024-09-13T11:27:22.598 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:22.598 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.598 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:27:22.598 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.179666+0000", 2024-09-13T11:27:22.599 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000110443 2024-09-13T11:27:22.599 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:22.599 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.599 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1", 2024-09-13T11:27:22.599 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.179791+0000", 2024-09-13T11:27:22.599 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00012560499999999999 2024-09-13T11:27:22.599 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:22.599 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.599 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:27:22.599 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.182761+0000", 2024-09-13T11:27:22.599 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0029697019999999998 2024-09-13T11:27:22.599 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:22.599 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.599 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:27:22.600 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.185572+0000", 2024-09-13T11:27:22.600 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.002811512 2024-09-13T11:27:22.600 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:22.600 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.600 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:27:22.600 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.185625+0000", 2024-09-13T11:27:22.600 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.2716000000000001e-05 2024-09-13T11:27:22.600 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:22.600 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.600 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:27:22.600 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.185665+0000", 2024-09-13T11:27:22.600 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.9666000000000003e-05 2024-09-13T11:27:22.600 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:27:22.600 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:27:22.601 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:27:22.601 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:22.601 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.601 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:62 3.b 3:d20ae5a8:::benchmark_data_smithi177_31116_object61:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T11:27:22.601 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:48.314723+0000", 2024-09-13T11:27:22.601 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 274.23892082200001, 2024-09-13T11:27:22.601 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.87338985099999999, 2024-09-13T11:27:22.601 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:27:22.601 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:27:22.601 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:27:22.601 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:27:22.601 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:27:22.601 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:27:22.601 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 62 2024-09-13T11:27:22.602 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:22.602 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:27:22.602 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.602 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:27:22.602 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314723+0000", 2024-09-13T11:27:22.602 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:27:22.602 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:22.602 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.602 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:27:22.602 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314723+0000", 2024-09-13T11:27:22.602 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:27:22.602 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:22.602 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.602 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:27:22.603 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314725+0000", 2024-09-13T11:27:22.603 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.3069999999999999e-06 2024-09-13T11:27:22.603 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:22.603 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.603 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:27:22.603 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314797+0000", 2024-09-13T11:27:22.603 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.2652999999999997e-05 2024-09-13T11:27:22.603 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:22.603 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.603 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:27:22.603 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314809+0000", 2024-09-13T11:27:22.603 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.1496e-05 2024-09-13T11:27:22.603 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:22.603 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.604 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:27:22.604 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314816+0000", 2024-09-13T11:27:22.604 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.6639999999999998e-06 2024-09-13T11:27:22.604 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:22.604 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.604 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:27:22.604 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316424+0000", 2024-09-13T11:27:22.604 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0016077559999999999 2024-09-13T11:27:22.604 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:22.604 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.604 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-09-13T11:27:22.604 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316441+0000", 2024-09-13T11:27:22.604 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.7305000000000001e-05 2024-09-13T11:27:22.604 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:22.605 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.605 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:27:22.605 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.180688+0000", 2024-09-13T11:27:22.605 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.86424692800000003 2024-09-13T11:27:22.605 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:22.605 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.605 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:27:22.605 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.180807+0000", 2024-09-13T11:27:22.605 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000118977 2024-09-13T11:27:22.605 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:22.605 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.605 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1", 2024-09-13T11:27:22.605 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.180938+0000", 2024-09-13T11:27:22.605 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00013049700000000001 2024-09-13T11:27:22.605 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:22.606 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.606 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:27:22.606 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.185680+0000", 2024-09-13T11:27:22.606 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0047417029999999999 2024-09-13T11:27:22.606 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:22.606 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.606 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:27:22.606 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.187856+0000", 2024-09-13T11:27:22.606 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0021760550000000001 2024-09-13T11:27:22.606 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:22.606 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.606 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:27:22.606 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.188007+0000", 2024-09-13T11:27:22.607 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00015185099999999999 2024-09-13T11:27:22.607 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:22.607 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.607 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:27:22.607 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.188113+0000", 2024-09-13T11:27:22.607 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000105659 2024-09-13T11:27:22.607 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:27:22.607 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:27:22.607 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:27:22.607 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:22.607 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.607 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:73 3.b 3:d0060d5f:::benchmark_data_smithi177_31116_object72:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T11:27:22.607 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:48.314829+0000", 2024-09-13T11:27:22.607 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 274.23881547299999, 2024-09-13T11:27:22.608 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.87364757199999998, 2024-09-13T11:27:22.608 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:27:22.608 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:27:22.608 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:27:22.608 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:27:22.608 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:27:22.608 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:27:22.608 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 73 2024-09-13T11:27:22.608 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:22.608 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:27:22.608 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.608 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:27:22.608 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314829+0000", 2024-09-13T11:27:22.608 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:27:22.609 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:22.609 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.609 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:27:22.609 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314829+0000", 2024-09-13T11:27:22.609 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:27:22.609 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:22.609 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.609 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:27:22.609 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314830+0000", 2024-09-13T11:27:22.609 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.243e-06 2024-09-13T11:27:22.609 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:22.609 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.609 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:27:22.609 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314965+0000", 2024-09-13T11:27:22.610 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00013470800000000001 2024-09-13T11:27:22.610 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:22.610 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.610 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:27:22.610 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314979+0000", 2024-09-13T11:27:22.610 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.4375999999999999e-05 2024-09-13T11:27:22.610 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:22.610 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.610 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:27:22.610 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.314988+0000", 2024-09-13T11:27:22.610 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.8570000000000001e-06 2024-09-13T11:27:22.610 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:22.610 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.610 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:27:22.611 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316545+0000", 2024-09-13T11:27:22.611 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.001556878 2024-09-13T11:27:22.611 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:22.611 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.611 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-09-13T11:27:22.611 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316556+0000", 2024-09-13T11:27:22.611 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.1453e-05 2024-09-13T11:27:22.611 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:22.611 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.611 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:27:22.611 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.182124+0000", 2024-09-13T11:27:22.611 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.86556744200000002 2024-09-13T11:27:22.611 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:22.611 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.611 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:27:22.612 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.182246+0000", 2024-09-13T11:27:22.612 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00012210500000000001 2024-09-13T11:27:22.612 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:22.612 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.612 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1", 2024-09-13T11:27:22.612 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.182375+0000", 2024-09-13T11:27:22.612 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00012920999999999999 2024-09-13T11:27:22.612 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:22.612 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.612 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:27:22.612 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.185684+0000", 2024-09-13T11:27:22.612 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0033093969999999999 2024-09-13T11:27:22.612 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:22.613 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.613 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:27:22.613 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.188244+0000", 2024-09-13T11:27:22.613 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0025601550000000002 2024-09-13T11:27:22.613 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:22.613 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.613 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:27:22.613 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.188379+0000", 2024-09-13T11:27:22.613 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00013463400000000001 2024-09-13T11:27:22.613 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:22.613 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.613 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:27:22.613 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.188476+0000", 2024-09-13T11:27:22.613 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.7114000000000006e-05 2024-09-13T11:27:22.614 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:27:22.614 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:27:22.614 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:27:22.614 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:22.614 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.614 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:81 3.b 3:d705fd70:::benchmark_data_smithi177_31116_object80:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T11:27:22.614 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:48.315000+0000", 2024-09-13T11:27:22.614 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 274.23864363600001, 2024-09-13T11:27:22.614 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.87561418300000005, 2024-09-13T11:27:22.614 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:27:22.614 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:27:22.614 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:27:22.614 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:27:22.615 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:27:22.615 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:27:22.615 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 81 2024-09-13T11:27:22.615 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:22.615 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:27:22.615 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.615 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:27:22.615 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315000+0000", 2024-09-13T11:27:22.615 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:27:22.615 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:22.615 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.615 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:27:22.615 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315000+0000", 2024-09-13T11:27:22.616 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:27:22.616 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:22.616 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.616 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:27:22.616 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315002+0000", 2024-09-13T11:27:22.616 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.3799999999999999e-06 2024-09-13T11:27:22.616 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:22.616 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.616 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:27:22.616 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315119+0000", 2024-09-13T11:27:22.616 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000116925 2024-09-13T11:27:22.616 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:22.616 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.616 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:27:22.617 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315133+0000", 2024-09-13T11:27:22.617 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.3971999999999999e-05 2024-09-13T11:27:22.617 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:22.617 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.617 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:27:22.617 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315142+0000", 2024-09-13T11:27:22.617 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.4399999999999994e-06 2024-09-13T11:27:22.617 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:22.617 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.617 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:27:22.617 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316652+0000", 2024-09-13T11:27:22.617 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.001509907 2024-09-13T11:27:22.617 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:22.617 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.618 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-09-13T11:27:22.618 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316668+0000", 2024-09-13T11:27:22.618 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.5945999999999999e-05 2024-09-13T11:27:22.618 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:22.618 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.618 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:27:22.618 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.183405+0000", 2024-09-13T11:27:22.618 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.866737384 2024-09-13T11:27:22.618 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:22.618 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.618 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:27:22.618 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.183701+0000", 2024-09-13T11:27:22.618 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00029511099999999997 2024-09-13T11:27:22.618 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:22.619 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.619 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1", 2024-09-13T11:27:22.619 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.184078+0000", 2024-09-13T11:27:22.619 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00037741000000000001 2024-09-13T11:27:22.619 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:22.619 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.619 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:27:22.619 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.188504+0000", 2024-09-13T11:27:22.619 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0044260280000000003 2024-09-13T11:27:22.619 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:22.619 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.619 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:27:22.619 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.190562+0000", 2024-09-13T11:27:22.619 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0020580099999999999 2024-09-13T11:27:22.620 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:22.620 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.620 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:27:22.620 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.190592+0000", 2024-09-13T11:27:22.620 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.0057999999999999e-05 2024-09-13T11:27:22.620 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:22.620 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.620 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:27:22.620 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.190615+0000", 2024-09-13T11:27:22.620 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.2612e-05 2024-09-13T11:27:22.620 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:27:22.620 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:27:22.620 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:27:22.620 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:22.621 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.621 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:87 3.b 3:dd7de0ad:::benchmark_data_smithi177_31116_object86:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T11:27:22.621 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:48.315155+0000", 2024-09-13T11:27:22.621 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 274.23848883900001, 2024-09-13T11:27:22.621 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.87555916099999997, 2024-09-13T11:27:22.621 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:27:22.621 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:27:22.621 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:27:22.621 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:27:22.621 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:27:22.621 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:27:22.621 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 87 2024-09-13T11:27:22.621 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:22.621 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:27:22.622 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.622 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:27:22.622 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315155+0000", 2024-09-13T11:27:22.622 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:27:22.622 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:22.622 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.622 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:27:22.622 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315155+0000", 2024-09-13T11:27:22.622 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:27:22.622 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:22.622 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.622 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:27:22.622 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315157+0000", 2024-09-13T11:27:22.623 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.3829999999999999e-06 2024-09-13T11:27:22.623 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:22.623 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.623 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:27:22.623 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315271+0000", 2024-09-13T11:27:22.623 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000114539 2024-09-13T11:27:22.623 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:22.623 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.623 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:27:22.623 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315285+0000", 2024-09-13T11:27:22.623 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.4042e-05 2024-09-13T11:27:22.623 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:22.624 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.624 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:27:22.624 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315295+0000", 2024-09-13T11:27:22.624 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.2739999999999996e-06 2024-09-13T11:27:22.624 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:22.624 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.624 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:27:22.624 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316739+0000", 2024-09-13T11:27:22.624 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0014446960000000001 2024-09-13T11:27:22.624 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:22.624 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.624 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-09-13T11:27:22.624 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316747+0000", 2024-09-13T11:27:22.625 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.0790000000000001e-06 2024-09-13T11:27:22.625 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:22.625 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.625 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:27:22.625 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.184922+0000", 2024-09-13T11:27:22.625 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.86817446399999998 2024-09-13T11:27:22.625 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:22.625 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.625 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:27:22.625 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.185037+0000", 2024-09-13T11:27:22.625 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00011565099999999999 2024-09-13T11:27:22.625 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:22.625 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.625 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1", 2024-09-13T11:27:22.626 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.185171+0000", 2024-09-13T11:27:22.626 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00013316900000000001 2024-09-13T11:27:22.626 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:22.626 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.626 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:27:22.626 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.188520+0000", 2024-09-13T11:27:22.626 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0033489150000000001 2024-09-13T11:27:22.626 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:22.626 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.626 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:27:22.626 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.190652+0000", 2024-09-13T11:27:22.626 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0021325649999999999 2024-09-13T11:27:22.626 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:22.626 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.627 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:27:22.627 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.190686+0000", 2024-09-13T11:27:22.627 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.3827e-05 2024-09-13T11:27:22.627 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:22.627 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.627 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:27:22.627 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.190714+0000", 2024-09-13T11:27:22.627 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.8557000000000001e-05 2024-09-13T11:27:22.627 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:27:22.627 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:27:22.627 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:27:22.627 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:22.627 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.627 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:99 3.b 3:d011b600:::benchmark_data_smithi177_31116_object98:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T11:27:22.628 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:48.315307+0000", 2024-09-13T11:27:22.628 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 274.23833701199999, 2024-09-13T11:27:22.628 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.87551770799999995, 2024-09-13T11:27:22.628 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:27:22.628 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:27:22.628 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:27:22.628 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:27:22.628 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:27:22.628 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:27:22.628 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 99 2024-09-13T11:27:22.628 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:22.628 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:27:22.628 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.629 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:27:22.629 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315307+0000", 2024-09-13T11:27:22.629 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:27:22.629 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:22.629 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.629 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:27:22.629 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315307+0000", 2024-09-13T11:27:22.629 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:27:22.629 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:22.629 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.629 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:27:22.629 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315308+0000", 2024-09-13T11:27:22.629 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.316e-06 2024-09-13T11:27:22.629 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:22.630 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.630 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:27:22.630 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315453+0000", 2024-09-13T11:27:22.630 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00014473200000000001 2024-09-13T11:27:22.630 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:22.630 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.630 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:27:22.630 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315471+0000", 2024-09-13T11:27:22.630 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.8019000000000001e-05 2024-09-13T11:27:22.630 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:22.630 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.630 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:27:22.630 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315489+0000", 2024-09-13T11:27:22.630 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.7861999999999999e-05 2024-09-13T11:27:22.631 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:22.631 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.631 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:27:22.631 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316837+0000", 2024-09-13T11:27:22.631 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.001347787 2024-09-13T11:27:22.631 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:22.631 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.631 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-09-13T11:27:22.631 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316858+0000", 2024-09-13T11:27:22.631 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.1103000000000001e-05 2024-09-13T11:27:22.631 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:22.631 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.631 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:27:22.631 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.185713+0000", 2024-09-13T11:27:22.632 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.86885540800000005 2024-09-13T11:27:22.632 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:22.632 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.632 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:27:22.632 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.185820+0000", 2024-09-13T11:27:22.632 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000107081 2024-09-13T11:27:22.632 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:22.632 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.632 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1", 2024-09-13T11:27:22.632 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.185948+0000", 2024-09-13T11:27:22.632 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00012775800000000001 2024-09-13T11:27:22.632 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:22.632 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.633 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:27:22.633 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.190527+0000", 2024-09-13T11:27:22.633 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0045783910000000002 2024-09-13T11:27:22.633 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:22.633 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.633 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:27:22.633 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.190759+0000", 2024-09-13T11:27:22.633 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00023266199999999999 2024-09-13T11:27:22.633 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:22.633 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.633 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:27:22.633 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.190789+0000", 2024-09-13T11:27:22.633 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.9666999999999999e-05 2024-09-13T11:27:22.633 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:22.634 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.634 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:27:22.634 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.190825+0000", 2024-09-13T11:27:22.634 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.5921999999999999e-05 2024-09-13T11:27:22.634 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:27:22.634 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:27:22.634 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:27:22.634 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:22.634 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.634 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:108 3.b 3:d5c90217:::benchmark_data_smithi177_31116_object107:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T11:27:22.634 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:48.315513+0000", 2024-09-13T11:27:22.634 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 274.238131307, 2024-09-13T11:27:22.634 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.87544772500000001, 2024-09-13T11:27:22.634 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:27:22.635 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:27:22.635 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:27:22.635 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:27:22.635 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:27:22.635 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:27:22.635 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 108 2024-09-13T11:27:22.635 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:22.635 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:27:22.635 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.635 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:27:22.635 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315513+0000", 2024-09-13T11:27:22.635 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:27:22.635 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:22.635 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.636 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:27:22.636 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315513+0000", 2024-09-13T11:27:22.636 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:27:22.636 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:22.636 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.636 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:27:22.636 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315515+0000", 2024-09-13T11:27:22.636 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.9369999999999998e-06 2024-09-13T11:27:22.636 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:22.636 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.636 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:27:22.636 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315644+0000", 2024-09-13T11:27:22.636 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000129011 2024-09-13T11:27:22.637 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:22.637 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.637 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:27:22.637 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315657+0000", 2024-09-13T11:27:22.637 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.3689999999999999e-05 2024-09-13T11:27:22.637 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:22.637 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.637 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:27:22.637 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315668+0000", 2024-09-13T11:27:22.637 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0589e-05 2024-09-13T11:27:22.637 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:22.637 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.637 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:27:22.637 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316925+0000", 2024-09-13T11:27:22.638 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0012568340000000001 2024-09-13T11:27:22.638 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:22.638 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.638 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-09-13T11:27:22.638 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316932+0000", 2024-09-13T11:27:22.638 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.2230000000000001e-06 2024-09-13T11:27:22.638 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:22.638 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.638 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:27:22.638 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.186619+0000", 2024-09-13T11:27:22.638 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.86968679599999998 2024-09-13T11:27:22.638 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:22.639 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.639 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:27:22.639 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.186914+0000", 2024-09-13T11:27:22.639 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00029528900000000001 2024-09-13T11:27:22.639 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:22.639 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.639 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1", 2024-09-13T11:27:22.639 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.187217+0000", 2024-09-13T11:27:22.639 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00030232099999999998 2024-09-13T11:27:22.639 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:22.639 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.639 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:27:22.639 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.190532+0000", 2024-09-13T11:27:22.639 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.003315391 2024-09-13T11:27:22.639 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:22.640 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.640 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:27:22.640 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.190875+0000", 2024-09-13T11:27:22.640 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00034326300000000001 2024-09-13T11:27:22.640 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:22.640 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.640 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:27:22.640 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.190925+0000", 2024-09-13T11:27:22.640 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.9429999999999999e-05 2024-09-13T11:27:22.640 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:22.640 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.640 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:27:22.640 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.190961+0000", 2024-09-13T11:27:22.641 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.5951000000000002e-05 2024-09-13T11:27:22.641 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:27:22.641 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:27:22.641 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:27:22.641 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:22.641 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.641 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:172 3.b 3:dbec1eeb:::benchmark_data_smithi177_31116_object171:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T11:27:22.641 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:48.315684+0000", 2024-09-13T11:27:22.641 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 274.23796014999999, 2024-09-13T11:27:22.641 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.87615312999999995, 2024-09-13T11:27:22.641 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:27:22.641 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:27:22.641 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:27:22.641 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:27:22.642 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:27:22.642 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:27:22.642 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 172 2024-09-13T11:27:22.642 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:22.642 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:27:22.642 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.642 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:27:22.642 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315684+0000", 2024-09-13T11:27:22.642 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:27:22.642 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:22.642 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.642 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:27:22.642 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315684+0000", 2024-09-13T11:27:22.643 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:27:22.643 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:22.643 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.643 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:27:22.643 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315686+0000", 2024-09-13T11:27:22.643 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.886e-06 2024-09-13T11:27:22.643 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:22.643 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.643 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:27:22.643 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315820+0000", 2024-09-13T11:27:22.643 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00013388500000000001 2024-09-13T11:27:22.643 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:22.643 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.643 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:27:22.644 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315837+0000", 2024-09-13T11:27:22.644 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.6756e-05 2024-09-13T11:27:22.644 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:22.644 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.644 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:27:22.644 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315851+0000", 2024-09-13T11:27:22.644 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.4566000000000001e-05 2024-09-13T11:27:22.644 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:22.644 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.644 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:27:22.644 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.317021+0000", 2024-09-13T11:27:22.644 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0011701769999999999 2024-09-13T11:27:22.644 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:22.644 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.645 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-09-13T11:27:22.645 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.317037+0000", 2024-09-13T11:27:22.645 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.609e-05 2024-09-13T11:27:22.645 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:22.645 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.645 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:27:22.645 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.188634+0000", 2024-09-13T11:27:22.645 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.87159711900000003 2024-09-13T11:27:22.645 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:22.645 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.645 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:27:22.645 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.188974+0000", 2024-09-13T11:27:22.645 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00033922299999999999 2024-09-13T11:27:22.645 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:22.646 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.646 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1", 2024-09-13T11:27:22.646 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.189510+0000", 2024-09-13T11:27:22.646 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00053618999999999997 2024-09-13T11:27:22.646 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:22.646 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.646 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:27:22.646 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.191027+0000", 2024-09-13T11:27:22.646 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0015168390000000001 2024-09-13T11:27:22.646 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:22.646 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.646 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:27:22.646 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.191766+0000", 2024-09-13T11:27:22.646 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000739064 2024-09-13T11:27:22.647 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:22.647 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.647 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:27:22.647 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.191810+0000", 2024-09-13T11:27:22.647 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.3755999999999999e-05 2024-09-13T11:27:22.647 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:22.647 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.647 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:27:22.647 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.191837+0000", 2024-09-13T11:27:22.647 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.7579000000000001e-05 2024-09-13T11:27:22.647 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:27:22.647 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:27:22.647 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:27:22.647 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:22.648 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.648 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:177 3.b 3:de89c779:::benchmark_data_smithi177_31116_object176:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T11:27:22.648 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:48.315898+0000", 2024-09-13T11:27:22.648 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 274.237745835, 2024-09-13T11:27:22.648 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.87600712599999997, 2024-09-13T11:27:22.648 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:27:22.648 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:27:22.648 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:27:22.648 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:27:22.648 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:27:22.648 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:27:22.648 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 177 2024-09-13T11:27:22.648 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:22.649 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:27:22.649 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.649 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:27:22.649 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315898+0000", 2024-09-13T11:27:22.649 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:27:22.649 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:22.649 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.649 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:27:22.649 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315898+0000", 2024-09-13T11:27:22.649 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:27:22.649 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:22.649 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.649 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:27:22.649 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.315900+0000", 2024-09-13T11:27:22.650 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.4929999999999999e-06 2024-09-13T11:27:22.650 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:22.650 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.650 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:27:22.650 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316027+0000", 2024-09-13T11:27:22.650 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000127568 2024-09-13T11:27:22.650 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:22.650 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.650 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:27:22.650 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316048+0000", 2024-09-13T11:27:22.650 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.0795999999999999e-05 2024-09-13T11:27:22.650 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:22.650 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.650 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:27:22.651 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316061+0000", 2024-09-13T11:27:22.651 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.3283e-05 2024-09-13T11:27:22.651 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:22.651 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.651 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:27:22.651 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.317107+0000", 2024-09-13T11:27:22.651 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0010452949999999999 2024-09-13T11:27:22.651 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:22.651 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.651 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-09-13T11:27:22.651 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.317114+0000", 2024-09-13T11:27:22.651 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.4569999999999999e-06 2024-09-13T11:27:22.651 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:22.651 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.652 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:27:22.652 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.190017+0000", 2024-09-13T11:27:22.652 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.87290258499999995 2024-09-13T11:27:22.652 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:22.652 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.652 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:27:22.652 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.190138+0000", 2024-09-13T11:27:22.652 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000121418 2024-09-13T11:27:22.652 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:22.652 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.652 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1", 2024-09-13T11:27:22.652 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.190216+0000", 2024-09-13T11:27:22.652 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.7410999999999995e-05 2024-09-13T11:27:22.652 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:22.653 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.653 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:27:22.653 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.191755+0000", 2024-09-13T11:27:22.653 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0015395420000000001 2024-09-13T11:27:22.653 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:22.653 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.653 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:27:22.653 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.191843+0000", 2024-09-13T11:27:22.653 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.8022999999999994e-05 2024-09-13T11:27:22.653 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:22.653 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.653 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:27:22.653 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.191876+0000", 2024-09-13T11:27:22.653 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.2619000000000001e-05 2024-09-13T11:27:22.654 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:22.654 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.654 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:27:22.654 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.191905+0000", 2024-09-13T11:27:22.654 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.9635999999999999e-05 2024-09-13T11:27:22.654 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:27:22.654 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:27:22.654 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:27:22.654 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:22.654 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.654 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:187 3.b 3:da03c8dd:::benchmark_data_smithi177_31116_object186:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T11:27:22.654 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:48.316083+0000", 2024-09-13T11:27:22.654 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 274.237560879, 2024-09-13T11:27:22.654 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.87835499500000003, 2024-09-13T11:27:22.655 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:27:22.655 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:27:22.655 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:27:22.655 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:27:22.655 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:27:22.655 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:27:22.655 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 187 2024-09-13T11:27:22.655 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:22.655 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:27:22.655 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.655 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:27:22.655 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316083+0000", 2024-09-13T11:27:22.655 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:27:22.655 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:22.656 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.656 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:27:22.656 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316083+0000", 2024-09-13T11:27:22.656 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:27:22.656 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:22.656 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.656 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:27:22.656 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316086+0000", 2024-09-13T11:27:22.656 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.5560000000000002e-06 2024-09-13T11:27:22.656 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:22.656 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.656 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:27:22.656 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316199+0000", 2024-09-13T11:27:22.657 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000113522 2024-09-13T11:27:22.657 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:22.657 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.657 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:27:22.657 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316215+0000", 2024-09-13T11:27:22.657 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.5265999999999999e-05 2024-09-13T11:27:22.657 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:22.657 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.657 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:27:22.657 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316226+0000", 2024-09-13T11:27:22.657 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0967e-05 2024-09-13T11:27:22.660 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:22.660 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.660 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:27:22.660 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.317198+0000", 2024-09-13T11:27:22.660 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00097194100000000004 2024-09-13T11:27:22.660 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:22.660 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.660 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-09-13T11:27:22.660 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.317213+0000", 2024-09-13T11:27:22.660 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.5472999999999999e-05 2024-09-13T11:27:22.661 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:22.661 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.661 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:27:22.661 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.191077+0000", 2024-09-13T11:27:22.661 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.87386435200000001 2024-09-13T11:27:22.661 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:22.661 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.661 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:27:22.661 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.191181+0000", 2024-09-13T11:27:22.661 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000103618 2024-09-13T11:27:22.661 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:22.661 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.661 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1", 2024-09-13T11:27:22.661 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.191305+0000", 2024-09-13T11:27:22.662 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00012358599999999999 2024-09-13T11:27:22.662 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:22.662 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.662 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:27:22.662 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.194279+0000", 2024-09-13T11:27:22.662 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0029746970000000001 2024-09-13T11:27:22.662 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:22.662 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.662 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:27:22.662 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.194336+0000", 2024-09-13T11:27:22.662 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.7244000000000003e-05 2024-09-13T11:27:22.662 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:22.662 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.662 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:27:22.663 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.194397+0000", 2024-09-13T11:27:22.663 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.0278000000000001e-05 2024-09-13T11:27:22.663 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:22.663 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.663 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:27:22.663 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.194438+0000", 2024-09-13T11:27:22.663 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.1495000000000002e-05 2024-09-13T11:27:22.663 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:27:22.663 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:27:22.663 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:27:22.663 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:22.663 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.663 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:200 3.b 3:d90039f1:::benchmark_data_smithi177_31116_object199:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T11:27:22.663 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:48.316280+0000", 2024-09-13T11:27:22.664 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 274.23736441400001, 2024-09-13T11:27:22.664 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.87879273000000002, 2024-09-13T11:27:22.664 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:27:22.664 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:27:22.664 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:27:22.664 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:27:22.664 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:27:22.664 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:27:22.664 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 200 2024-09-13T11:27:22.664 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:22.664 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:27:22.664 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.664 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:27:22.664 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316280+0000", 2024-09-13T11:27:22.665 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:27:22.665 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:22.665 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.665 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:27:22.665 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316280+0000", 2024-09-13T11:27:22.665 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:27:22.665 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:22.665 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.665 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:27:22.665 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316282+0000", 2024-09-13T11:27:22.665 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.497e-06 2024-09-13T11:27:22.665 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:22.665 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.665 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:27:22.666 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316411+0000", 2024-09-13T11:27:22.666 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00012924799999999999 2024-09-13T11:27:22.666 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:22.666 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.666 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:27:22.666 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316426+0000", 2024-09-13T11:27:22.666 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.4183e-05 2024-09-13T11:27:22.666 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:22.666 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.666 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:27:22.666 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.316435+0000", 2024-09-13T11:27:22.666 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.0990000000000005e-06 2024-09-13T11:27:22.666 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:22.667 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.667 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:27:22.667 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.317250+0000", 2024-09-13T11:27:22.667 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00081501100000000003 2024-09-13T11:27:22.667 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:22.667 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.667 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-09-13T11:27:22.667 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:48.317257+0000", 2024-09-13T11:27:22.667 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.2169999999999997e-06 2024-09-13T11:27:22.667 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:22.667 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.667 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:27:22.667 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.191936+0000", 2024-09-13T11:27:22.667 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.87467922099999995 2024-09-13T11:27:22.668 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:22.668 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.668 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:27:22.668 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.192056+0000", 2024-09-13T11:27:22.668 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000119638 2024-09-13T11:27:22.668 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:22.668 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.668 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1", 2024-09-13T11:27:22.668 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.192184+0000", 2024-09-13T11:27:22.668 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00012813399999999999 2024-09-13T11:27:22.668 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:22.668 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.668 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:27:22.668 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.194324+0000", 2024-09-13T11:27:22.669 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0021399420000000001 2024-09-13T11:27:22.669 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:22.669 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.669 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:27:22.669 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.194802+0000", 2024-09-13T11:27:22.669 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00047798799999999999 2024-09-13T11:27:22.669 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:22.669 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.669 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:27:22.669 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.194951+0000", 2024-09-13T11:27:22.669 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0001494 2024-09-13T11:27:22.669 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:22.669 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.669 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:27:22.670 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:49.195072+0000", 2024-09-13T11:27:22.670 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00012115199999999999 2024-09-13T11:27:22.670 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:27:22.670 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:27:22.670 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:27:22.670 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:22.670 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.670 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop(client.4665.0:79801 3.5 e102/67 3:a6efdf84:::benchmark_data_smithi177_31116_object79800:head v 102'3851, mlcod=102'3848)", 2024-09-13T11:27:22.670 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:24:00.403754+0000", 2024-09-13T11:27:22.670 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 202.149890062, 2024-09-13T11:27:22.670 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.31749079899999999, 2024-09-13T11:27:22.670 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:27:22.670 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:27:22.671 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:27:22.671 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:27:22.671 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.671 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:27:22.671 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:00.403754+0000", 2024-09-13T11:27:22.671 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:27:22.671 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:22.671 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.671 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:27:22.671 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:00.403754+0000", 2024-09-13T11:27:22.671 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:27:22.671 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:22.671 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.671 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:27:22.672 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:00.403758+0000", 2024-09-13T11:27:22.672 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.507e-06 2024-09-13T11:27:22.672 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:22.672 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.672 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:27:22.672 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:00.403819+0000", 2024-09-13T11:27:22.672 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.1042000000000003e-05 2024-09-13T11:27:22.672 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:22.672 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.672 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:27:22.672 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:00.717963+0000", 2024-09-13T11:27:22.672 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.314144594 2024-09-13T11:27:22.672 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:22.672 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.673 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:27:22.673 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:00.717976+0000", 2024-09-13T11:27:22.673 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.3029e-05 2024-09-13T11:27:22.673 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:22.673 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.673 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:27:22.673 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:00.718143+0000", 2024-09-13T11:27:22.673 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00016639000000000001 2024-09-13T11:27:22.673 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:22.673 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.673 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:27:22.673 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:00.718184+0000", 2024-09-13T11:27:22.673 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.1801999999999997e-05 2024-09-13T11:27:22.673 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:22.674 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.674 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:27:22.674 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:00.721162+0000", 2024-09-13T11:27:22.674 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0029772800000000001 2024-09-13T11:27:22.674 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:22.674 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.674 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:27:22.674 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:00.721245+0000", 2024-09-13T11:27:22.674 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.3154999999999999e-05 2024-09-13T11:27:22.674 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:27:22.674 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:27:22.674 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:27:22.674 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:22.675 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.675 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "MOSDPGPushReply(3.26 121/121 [PushReplyOp(3:670b9aab:::benchmark_data_smithi177_31116_object93:head)])", 2024-09-13T11:27:22.675 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:24:27.300318+0000", 2024-09-13T11:27:22.675 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 175.25332619400001, 2024-09-13T11:27:22.675 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.811436728, 2024-09-13T11:27:22.675 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:27:22.675 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:27:22.675 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "reached pg", 2024-09-13T11:27:22.675 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:27:22.675 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.675 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:27:22.675 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:27.300318+0000", 2024-09-13T11:27:22.675 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:27:22.675 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:22.676 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.676 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:27:22.676 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:27.300318+0000", 2024-09-13T11:27:22.676 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:27:22.676 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:22.676 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.676 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:27:22.676 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:27.300320+0000", 2024-09-13T11:27:22.676 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.778e-06 2024-09-13T11:27:22.676 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:22.676 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.676 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:27:22.676 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:27.300326+0000", 2024-09-13T11:27:22.676 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.7700000000000004e-06 2024-09-13T11:27:22.676 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:22.677 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.677 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:27:22.677 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:28.109665+0000", 2024-09-13T11:27:22.677 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.80933853499999997 2024-09-13T11:27:22.677 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:22.677 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.677 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:27:22.677 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:28.109678+0000", 2024-09-13T11:27:22.677 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.2870999999999999e-05 2024-09-13T11:27:22.677 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:22.677 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.677 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:27:22.677 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:28.109800+0000", 2024-09-13T11:27:22.677 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000122328 2024-09-13T11:27:22.678 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:22.678 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.678 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:27:22.678 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:28.111755+0000", 2024-09-13T11:27:22.678 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0019544459999999999 2024-09-13T11:27:22.678 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:27:22.678 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:27:22.678 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:27:22.678 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:22.678 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.678 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:184026 3.b 3:d3052347:::benchmark_data_smithi177_31116_object43723:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e171)", 2024-09-13T11:27:22.678 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:25:32.645242+0000", 2024-09-13T11:27:22.678 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 109.908402154, 2024-09-13T11:27:22.678 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.348925019, 2024-09-13T11:27:22.678 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:27:22.679 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:27:22.679 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:27:22.679 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:27:22.679 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:27:22.679 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:27:22.679 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 184026 2024-09-13T11:27:22.679 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:22.679 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:27:22.679 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.679 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:27:22.679 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:32.645242+0000", 2024-09-13T11:27:22.679 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:27:22.679 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:22.679 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.680 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:27:22.680 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:32.645242+0000", 2024-09-13T11:27:22.680 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:27:22.680 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:22.680 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.680 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:27:22.680 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:32.645244+0000", 2024-09-13T11:27:22.680 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.368e-06 2024-09-13T11:27:22.680 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:22.680 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.680 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:27:22.680 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:32.645251+0000", 2024-09-13T11:27:22.680 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.2519999999999996e-06 2024-09-13T11:27:22.680 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:22.681 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.681 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:27:22.681 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:32.645267+0000", 2024-09-13T11:27:22.681 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.6529e-05 2024-09-13T11:27:22.681 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:22.681 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.681 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:27:22.681 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:32.645278+0000", 2024-09-13T11:27:22.681 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.119e-05 2024-09-13T11:27:22.681 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:22.681 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.681 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:27:22.681 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:32.645348+0000", 2024-09-13T11:27:22.681 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.9333999999999997e-05 2024-09-13T11:27:22.682 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:22.682 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.682 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:27:22.682 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:32.645479+0000", 2024-09-13T11:27:22.682 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00013153199999999999 2024-09-13T11:27:22.682 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:22.682 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.682 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1", 2024-09-13T11:27:22.682 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:32.645599+0000", 2024-09-13T11:27:22.682 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000119481 2024-09-13T11:27:22.682 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:22.682 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.682 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:27:22.682 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:32.648042+0000", 2024-09-13T11:27:22.682 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0024437059999999999 2024-09-13T11:27:22.683 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:22.683 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.683 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:27:22.683 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:32.994082+0000", 2024-09-13T11:27:22.683 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.34604006399999998 2024-09-13T11:27:22.683 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:22.683 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.683 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:27:22.683 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:32.994124+0000", 2024-09-13T11:27:22.683 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.1573999999999999e-05 2024-09-13T11:27:22.683 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:22.683 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:22.683 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:27:22.683 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:32.994167+0000", 2024-09-13T11:27:22.684 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.2988999999999998e-05 2024-09-13T11:27:22.684 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:27:22.684 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:27:22.684 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:27:22.684 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:27:22.684 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:27:22.684 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-09-13T11:27:22.685 INFO:tasks.ceph.osd.15.smithi177.stderr:2024-09-13T11:27:22.680+0000 7f9d6ce77640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26578) UID: 0 2024-09-13T11:27:22.686 INFO:teuthology.orchestra.run.smithi162.stdout: marking collection for removal 2024-09-13T11:27:22.686 INFO:teuthology.orchestra.run.smithi162.stdout:setting '_remove' omap key 2024-09-13T11:27:22.686 INFO:teuthology.orchestra.run.smithi162.stdout:finish_remove_pgs 3.32_head removing 3.32 2024-09-13T11:27:22.686 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-13T11:27:22.692 INFO:teuthology.orchestra.run.smithi162.stdout:Remove successful 2024-09-13T11:27:22.787 INFO:tasks.ceph.osd.8.smithi001.stderr:2024-09-13T11:27:22.782+0000 7fb9b48ae640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26312) UID: 0 2024-09-13T11:27:22.805 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-13T11:27:22.805 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [], 2024-09-13T11:27:22.805 INFO:teuthology.orchestra.run.smithi162.stdout: "num_ops": 0 2024-09-13T11:27:22.806 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-13T11:27:22.821 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-13T11:27:22.888 INFO:tasks.ceph.osd.12.smithi001.stderr:2024-09-13T11:27:22.886+0000 7f892c6e6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26314) UID: 0 2024-09-13T11:27:22.977 INFO:teuthology.orchestra.run.smithi001.stderr:pool 'unique_pool_0' removed 2024-09-13T11:27:22.993 INFO:tasks.ceph.osd.2.smithi162.stderr:2024-09-13T11:27:22.987+0000 7fe92b35a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26313) UID: 0 2024-09-13T11:27:23.000 DEBUG:teuthology.run_tasks:Unwinding manager thrashosds 2024-09-13T11:27:23.009 INFO:tasks.thrashosds:joining thrashosds 2024-09-13T11:27:23.016 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-13T11:27:23.016 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [], 2024-09-13T11:27:23.016 INFO:teuthology.orchestra.run.smithi162.stdout: "complaint_time": 30, 2024-09-13T11:27:23.016 INFO:teuthology.orchestra.run.smithi162.stdout: "num_blocked_ops": 0 2024-09-13T11:27:23.016 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-13T11:27:23.036 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-13T11:27:23.037 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-13T11:27:23.211 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-13T11:27:23.211 INFO:teuthology.orchestra.run.smithi162.stdout: "size": 20, 2024-09-13T11:27:23.211 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 600, 2024-09-13T11:27:23.211 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [ 2024-09-13T11:27:23.211 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.211 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.4665.0:171 3.9 e41/30 3:9e1c7eeb:::benchmark_data_smithi177_31116_object170:head v 41'13, mlcod=41'9)", 2024-09-13T11:27:23.211 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:22:47.418602+0000", 2024-09-13T11:27:23.211 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 275.791973789, 2024-09-13T11:27:23.211 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.67235367300000004, 2024-09-13T11:27:23.211 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:27:23.211 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:27:23.212 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:27:23.212 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:27:23.212 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.212 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:27:23.212 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:47.418602+0000", 2024-09-13T11:27:23.212 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:23.212 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.212 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.212 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:27:23.212 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:47.418602+0000", 2024-09-13T11:27:23.212 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:23.212 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.213 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.213 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:27:23.213 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:47.418605+0000", 2024-09-13T11:27:23.213 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.2200000000000001e-06 2024-09-13T11:27:23.213 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.213 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.213 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:27:23.213 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:47.418668+0000", 2024-09-13T11:27:23.213 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.2921000000000003e-05 2024-09-13T11:27:23.213 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.213 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.213 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:27:23.213 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:48.089380+0000", 2024-09-13T11:27:23.213 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.67071191100000005 2024-09-13T11:27:23.214 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.214 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.214 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:27:23.214 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:48.089423+0000", 2024-09-13T11:27:23.214 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.2916e-05 2024-09-13T11:27:23.214 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.214 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.214 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:27:23.214 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:48.089533+0000", 2024-09-13T11:27:23.214 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000109996 2024-09-13T11:27:23.214 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.214 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.214 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:27:23.214 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:48.089570+0000", 2024-09-13T11:27:23.215 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.7314999999999998e-05 2024-09-13T11:27:23.215 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.215 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.215 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:27:23.215 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:48.090852+0000", 2024-09-13T11:27:23.215 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001281641 2024-09-13T11:27:23.216 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.216 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.216 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:27:23.216 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:48.090956+0000", 2024-09-13T11:27:23.216 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00010375299999999999 2024-09-13T11:27:23.216 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:23.216 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:27:23.216 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:23.216 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.216 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.216 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPushReply(3.19 74/70 [PushReplyOp(3:9edf939c:::benchmark_data_smithi177_31116_object17207:head)])", 2024-09-13T11:27:23.216 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:30.897971+0000", 2024-09-13T11:27:23.217 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 232.31260493799999, 2024-09-13T11:27:23.217 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.40616071100000001, 2024-09-13T11:27:23.217 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:27:23.217 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:27:23.217 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-09-13T11:27:23.217 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:27:23.217 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.217 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:27:23.217 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.897971+0000", 2024-09-13T11:27:23.218 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:23.218 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.218 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.218 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:27:23.218 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.897971+0000", 2024-09-13T11:27:23.218 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:23.218 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.218 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.218 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:27:23.218 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.897972+0000", 2024-09-13T11:27:23.218 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1429999999999999e-06 2024-09-13T11:27:23.218 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.219 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.219 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:27:23.219 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:30.897977+0000", 2024-09-13T11:27:23.219 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.5199999999999999e-06 2024-09-13T11:27:23.219 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.219 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.219 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:27:23.219 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:31.302500+0000", 2024-09-13T11:27:23.219 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.40452312400000001 2024-09-13T11:27:23.219 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.220 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.220 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:27:23.220 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:31.302513+0000", 2024-09-13T11:27:23.220 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2753e-05 2024-09-13T11:27:23.220 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.220 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.220 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:27:23.220 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:31.302645+0000", 2024-09-13T11:27:23.220 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00013292100000000001 2024-09-13T11:27:23.220 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.220 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.220 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:27:23.221 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:31.304132+0000", 2024-09-13T11:27:23.221 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0014862499999999999 2024-09-13T11:27:23.221 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:23.221 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:27:23.221 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:23.221 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.221 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.221 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPushReply(3.19 76/70 [PushReplyOp(3:9faf6d5f:::benchmark_data_smithi177_31116_object18954:head)])", 2024-09-13T11:27:23.221 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:32.042874+0000", 2024-09-13T11:27:23.221 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 231.16770206800001, 2024-09-13T11:27:23.221 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.40212404299999999, 2024-09-13T11:27:23.221 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:27:23.221 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:27:23.222 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-09-13T11:27:23.222 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:27:23.222 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.222 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:27:23.222 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.042874+0000", 2024-09-13T11:27:23.222 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:23.222 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.222 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.222 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:27:23.222 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.042874+0000", 2024-09-13T11:27:23.222 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:23.222 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.222 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.223 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:27:23.223 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.042875+0000", 2024-09-13T11:27:23.223 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.432e-06 2024-09-13T11:27:23.223 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.223 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.223 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:27:23.223 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.042886+0000", 2024-09-13T11:27:23.223 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0265000000000001e-05 2024-09-13T11:27:23.223 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.223 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.223 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:27:23.223 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.443455+0000", 2024-09-13T11:27:23.223 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.40056951400000002 2024-09-13T11:27:23.224 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.224 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.224 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:27:23.224 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.443480+0000", 2024-09-13T11:27:23.224 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.4929000000000001e-05 2024-09-13T11:27:23.224 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.224 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.224 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:27:23.224 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.443666+0000", 2024-09-13T11:27:23.224 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000186446 2024-09-13T11:27:23.225 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.225 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.225 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:27:23.225 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.444998+0000", 2024-09-13T11:27:23.225 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0013314570000000001 2024-09-13T11:27:23.225 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:23.225 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:27:23.225 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:23.225 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.226 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.226 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPush(3.1d 85/78 [PushOp(3:bc4b9336:::benchmark_data_smithi177_31116_object11944:head, version: 49'734, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:bc4b9336:::benchmark_data_smithi177_31116_object11944:head@49'734, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:27:23.226 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:42.225966+0000", 2024-09-13T11:27:23.226 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 220.984609855, 2024-09-13T11:27:23.226 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.23879947800000001, 2024-09-13T11:27:23.226 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:27:23.226 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:27:23.226 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:27:23.226 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:27:23.226 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.226 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:27:23.226 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.225966+0000", 2024-09-13T11:27:23.226 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:23.227 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.227 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.227 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:27:23.227 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.225966+0000", 2024-09-13T11:27:23.227 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:23.227 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.227 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.227 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:27:23.227 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.225969+0000", 2024-09-13T11:27:23.227 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.0400000000000001e-06 2024-09-13T11:27:23.227 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.227 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.228 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:27:23.228 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.226036+0000", 2024-09-13T11:27:23.228 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.6978999999999997e-05 2024-09-13T11:27:23.228 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.228 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.229 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:27:23.229 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.463110+0000", 2024-09-13T11:27:23.229 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.23707400000000001 2024-09-13T11:27:23.229 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.229 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.229 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:27:23.229 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.463140+0000", 2024-09-13T11:27:23.229 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.9575999999999999e-05 2024-09-13T11:27:23.229 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.229 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.229 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:27:23.229 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.463326+0000", 2024-09-13T11:27:23.229 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000186298 2024-09-13T11:27:23.230 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.230 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.230 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:27:23.230 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.463351+0000", 2024-09-13T11:27:23.230 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.4862999999999999e-05 2024-09-13T11:27:23.230 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.230 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.230 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:27:23.230 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.464766+0000", 2024-09-13T11:27:23.230 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001414722 2024-09-13T11:27:23.230 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:23.230 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:27:23.230 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:23.230 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.231 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.231 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPush(3.1d 89/78 [PushOp(3:bc1b7cfb:::benchmark_data_smithi177_31116_object22629:head, version: 55'1387, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:bc1b7cfb:::benchmark_data_smithi177_31116_object22629:head@55'1387, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:27:23.231 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:46.660855+0000", 2024-09-13T11:27:23.231 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 216.54972135899999, 2024-09-13T11:27:23.231 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.26622785599999998, 2024-09-13T11:27:23.231 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:27:23.231 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:27:23.231 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:27:23.231 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:27:23.231 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.231 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:27:23.232 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:46.660855+0000", 2024-09-13T11:27:23.232 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:23.232 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.232 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.232 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:27:23.232 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:46.660855+0000", 2024-09-13T11:27:23.232 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:23.232 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.232 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.232 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:27:23.232 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:46.660857+0000", 2024-09-13T11:27:23.232 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.6560000000000001e-06 2024-09-13T11:27:23.232 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.233 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.233 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:27:23.233 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:46.660931+0000", 2024-09-13T11:27:23.233 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.4041999999999993e-05 2024-09-13T11:27:23.233 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.233 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.233 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:27:23.233 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:46.924654+0000", 2024-09-13T11:27:23.233 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.26372239800000002 2024-09-13T11:27:23.233 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.233 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.233 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:27:23.233 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:46.924729+0000", 2024-09-13T11:27:23.233 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.5061999999999994e-05 2024-09-13T11:27:23.234 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.234 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.234 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:27:23.234 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:46.924986+0000", 2024-09-13T11:27:23.234 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00025762000000000002 2024-09-13T11:27:23.234 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.234 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.234 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:27:23.234 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:46.925013+0000", 2024-09-13T11:27:23.234 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.6179000000000001e-05 2024-09-13T11:27:23.234 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.234 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.234 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:27:23.235 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:46.927082+0000", 2024-09-13T11:27:23.235 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.002069899 2024-09-13T11:27:23.235 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:23.235 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:27:23.235 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:23.235 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.235 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.235 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:83199 3.1d 3:be2c6fae:::benchmark_data_smithi177_31116_object83198:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106)", 2024-09-13T11:27:23.235 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:24:04.297306+0000", 2024-09-13T11:27:23.235 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 198.91327043300001, 2024-09-13T11:27:23.235 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.25114587999999999, 2024-09-13T11:27:23.235 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:27:23.236 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:27:23.236 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:27:23.236 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:27:23.236 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:27:23.236 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:27:23.236 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 83199 2024-09-13T11:27:23.236 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.236 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:27:23.236 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.236 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:27:23.236 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.297306+0000", 2024-09-13T11:27:23.236 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:23.237 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.237 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.237 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:27:23.237 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.297306+0000", 2024-09-13T11:27:23.237 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:23.237 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.237 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.237 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:27:23.237 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.297308+0000", 2024-09-13T11:27:23.237 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.4150000000000002e-06 2024-09-13T11:27:23.237 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.237 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.237 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:27:23.238 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.297352+0000", 2024-09-13T11:27:23.238 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.4431000000000002e-05 2024-09-13T11:27:23.238 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.238 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.238 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:27:23.238 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.297369+0000", 2024-09-13T11:27:23.238 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.6685e-05 2024-09-13T11:27:23.238 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.238 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.239 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:27:23.239 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.297376+0000", 2024-09-13T11:27:23.239 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.7299999999999999e-06 2024-09-13T11:27:23.239 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.239 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.239 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:27:23.239 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.297427+0000", 2024-09-13T11:27:23.239 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.1461000000000001e-05 2024-09-13T11:27:23.239 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.239 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.239 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:27:23.239 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.297528+0000", 2024-09-13T11:27:23.239 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00010028599999999999 2024-09-13T11:27:23.240 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.240 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.240 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 15", 2024-09-13T11:27:23.240 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.297685+0000", 2024-09-13T11:27:23.240 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00015760500000000001 2024-09-13T11:27:23.240 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.240 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.240 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:27:23.240 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.300779+0000", 2024-09-13T11:27:23.240 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0030936700000000002 2024-09-13T11:27:23.240 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.241 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.241 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:27:23.241 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.548279+0000", 2024-09-13T11:27:23.241 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.24750059999999999 2024-09-13T11:27:23.241 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.241 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.241 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:27:23.241 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.548373+0000", 2024-09-13T11:27:23.241 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.3451999999999999e-05 2024-09-13T11:27:23.241 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.241 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.241 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:27:23.242 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.548451+0000", 2024-09-13T11:27:23.242 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.8545000000000001e-05 2024-09-13T11:27:23.242 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:23.242 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:27:23.242 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:23.242 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.242 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.242 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:126039 3.29 3:96100181:::benchmark_data_smithi177_31116_object126038:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e130)", 2024-09-13T11:27:23.243 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:24:43.012785+0000", 2024-09-13T11:27:23.243 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 160.197791311, 2024-09-13T11:27:23.243 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.26685311900000003, 2024-09-13T11:27:23.243 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:27:23.243 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:27:23.243 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:27:23.243 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:27:23.243 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:27:23.243 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:27:23.243 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 126039 2024-09-13T11:27:23.243 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.243 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:27:23.243 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.244 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:27:23.244 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:43.012785+0000", 2024-09-13T11:27:23.244 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:23.244 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.244 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.244 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:27:23.244 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:43.012785+0000", 2024-09-13T11:27:23.244 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:23.244 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.244 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.244 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:27:23.244 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:43.012787+0000", 2024-09-13T11:27:23.244 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.875e-06 2024-09-13T11:27:23.245 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.245 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.245 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:27:23.245 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:43.012826+0000", 2024-09-13T11:27:23.245 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.9483999999999997e-05 2024-09-13T11:27:23.245 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.245 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.245 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:27:23.245 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:43.012839+0000", 2024-09-13T11:27:23.245 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3405e-05 2024-09-13T11:27:23.245 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.245 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.245 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:27:23.245 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:43.012845+0000", 2024-09-13T11:27:23.246 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.9100000000000002e-06 2024-09-13T11:27:23.246 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.246 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.246 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:27:23.246 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:43.012903+0000", 2024-09-13T11:27:23.246 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.8131000000000003e-05 2024-09-13T11:27:23.246 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.246 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.246 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:27:23.246 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:43.013066+0000", 2024-09-13T11:27:23.246 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000162058 2024-09-13T11:27:23.246 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.246 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.246 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 4,5", 2024-09-13T11:27:23.247 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:43.013235+0000", 2024-09-13T11:27:23.247 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00016929600000000001 2024-09-13T11:27:23.247 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.247 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.247 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:27:23.247 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:43.017063+0000", 2024-09-13T11:27:23.247 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0038283890000000002 2024-09-13T11:27:23.247 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.247 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.247 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:27:23.247 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:43.017086+0000", 2024-09-13T11:27:23.247 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.2934000000000001e-05 2024-09-13T11:27:23.247 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.247 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.248 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:27:23.248 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:43.278804+0000", 2024-09-13T11:27:23.248 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.26171751799999998 2024-09-13T11:27:23.248 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.248 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.248 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:27:23.248 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:43.278862+0000", 2024-09-13T11:27:23.248 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.8431999999999999e-05 2024-09-13T11:27:23.248 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.249 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.249 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:27:23.249 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:43.279638+0000", 2024-09-13T11:27:23.249 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00077568699999999995 2024-09-13T11:27:23.249 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:23.249 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:27:23.249 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:23.249 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.249 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.249 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop_reply(client.4665.0:126039 3.29 e130/120 ondisk, result = 0)", 2024-09-13T11:27:23.249 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:24:43.021691+0000", 2024-09-13T11:27:23.250 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 160.18888538499999, 2024-09-13T11:27:23.250 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.25795579699999999, 2024-09-13T11:27:23.250 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:27:23.250 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:27:23.250 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:27:23.250 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:27:23.250 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.250 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:27:23.250 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:43.021691+0000", 2024-09-13T11:27:23.250 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:23.250 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.250 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.250 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:27:23.251 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:43.021691+0000", 2024-09-13T11:27:23.251 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:23.251 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.251 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.251 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:27:23.251 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:43.021692+0000", 2024-09-13T11:27:23.251 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3880000000000001e-06 2024-09-13T11:27:23.251 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.251 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.251 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:27:23.251 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:43.021700+0000", 2024-09-13T11:27:23.251 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.9310000000000008e-06 2024-09-13T11:27:23.251 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.251 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.252 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:27:23.252 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:43.278719+0000", 2024-09-13T11:27:23.252 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.25701909299999998 2024-09-13T11:27:23.252 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.252 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.252 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:27:23.252 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:43.278728+0000", 2024-09-13T11:27:23.252 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.5000000000000005e-06 2024-09-13T11:27:23.252 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.252 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.252 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:27:23.252 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:43.278786+0000", 2024-09-13T11:27:23.253 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.7640999999999999e-05 2024-09-13T11:27:23.253 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.253 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.253 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:27:23.253 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:43.278796+0000", 2024-09-13T11:27:23.253 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.9199999999999999e-06 2024-09-13T11:27:23.253 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.253 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.253 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:27:23.253 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:43.279646+0000", 2024-09-13T11:27:23.253 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00085032399999999996 2024-09-13T11:27:23.253 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:23.254 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:27:23.254 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:23.254 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.254 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.254 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.4665.0:138294 3.9 e140/117 3:90012dab:::benchmark_data_smithi177_31116_object138293:head v 140'5246, mlcod=140'5243)", 2024-09-13T11:27:23.254 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:24:54.252227+0000", 2024-09-13T11:27:23.254 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 148.95834889700001, 2024-09-13T11:27:23.254 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.32919836400000002, 2024-09-13T11:27:23.254 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:27:23.254 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:27:23.254 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:27:23.254 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:27:23.255 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.255 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:27:23.255 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:54.252227+0000", 2024-09-13T11:27:23.255 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:23.255 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.255 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.255 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:27:23.255 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:54.252227+0000", 2024-09-13T11:27:23.255 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:23.255 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.255 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.255 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:27:23.255 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:54.252228+0000", 2024-09-13T11:27:23.256 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.8899999999999998e-07 2024-09-13T11:27:23.256 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.256 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.256 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:27:23.256 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:54.252276+0000", 2024-09-13T11:27:23.256 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.8137999999999999e-05 2024-09-13T11:27:23.256 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.256 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.256 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:27:23.256 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:54.578216+0000", 2024-09-13T11:27:23.256 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.32594014399999999 2024-09-13T11:27:23.256 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.256 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.257 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:27:23.257 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:54.578235+0000", 2024-09-13T11:27:23.257 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.8943000000000001e-05 2024-09-13T11:27:23.257 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.257 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.257 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:27:23.257 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:54.578396+0000", 2024-09-13T11:27:23.257 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000160922 2024-09-13T11:27:23.257 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.257 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.257 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:27:23.257 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:54.578457+0000", 2024-09-13T11:27:23.257 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.0665999999999998e-05 2024-09-13T11:27:23.258 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.258 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.258 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:27:23.258 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:54.581339+0000", 2024-09-13T11:27:23.258 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0028826839999999999 2024-09-13T11:27:23.258 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.258 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.258 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:27:23.258 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:54.581425+0000", 2024-09-13T11:27:23.258 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.6077999999999998e-05 2024-09-13T11:27:23.258 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:23.258 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:27:23.258 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:23.258 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.259 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.259 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPushReply(3.29 143/120 [PushReplyOp(3:973d2047:::benchmark_data_smithi177_31116_object49749:head)])", 2024-09-13T11:27:23.259 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:24:57.387106+0000", 2024-09-13T11:27:23.259 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 145.82346956999999, 2024-09-13T11:27:23.259 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.64813637000000002, 2024-09-13T11:27:23.259 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:27:23.259 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:27:23.259 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-09-13T11:27:23.259 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:27:23.259 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.259 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:27:23.259 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:57.387106+0000", 2024-09-13T11:27:23.259 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:23.260 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.260 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.260 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:27:23.260 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:57.387106+0000", 2024-09-13T11:27:23.260 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:23.260 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.260 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.260 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:27:23.260 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:57.387116+0000", 2024-09-13T11:27:23.260 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.1530000000000003e-06 2024-09-13T11:27:23.260 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.260 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.260 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:27:23.261 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:57.387127+0000", 2024-09-13T11:27:23.261 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1197e-05 2024-09-13T11:27:23.261 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.261 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.261 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:27:23.261 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:58.034264+0000", 2024-09-13T11:27:23.261 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.64713691799999995 2024-09-13T11:27:23.261 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.261 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.261 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:27:23.261 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:58.034273+0000", 2024-09-13T11:27:23.261 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.0929999999999992e-06 2024-09-13T11:27:23.261 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.261 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.262 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:27:23.262 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:58.034321+0000", 2024-09-13T11:27:23.262 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.8396999999999998e-05 2024-09-13T11:27:23.262 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.262 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.262 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:27:23.262 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:58.035243+0000", 2024-09-13T11:27:23.262 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00092161200000000004 2024-09-13T11:27:23.262 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:23.262 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:27:23.262 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:23.262 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.262 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.263 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPushReply(3.29 144/120 [PushReplyOp(3:94a6fc3a:::benchmark_data_smithi177_31116_object50245:head)])", 2024-09-13T11:27:23.263 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:24:58.320977+0000", 2024-09-13T11:27:23.263 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 144.889598987, 2024-09-13T11:27:23.263 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.31110569599999999, 2024-09-13T11:27:23.263 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:27:23.263 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:27:23.263 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-09-13T11:27:23.263 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:27:23.263 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.264 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:27:23.264 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:58.320977+0000", 2024-09-13T11:27:23.264 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:23.264 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.264 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.264 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:27:23.264 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:58.320977+0000", 2024-09-13T11:27:23.264 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:23.264 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.264 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.264 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:27:23.264 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:58.320978+0000", 2024-09-13T11:27:23.264 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.8700000000000004e-07 2024-09-13T11:27:23.264 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.265 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.265 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:27:23.265 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:58.320985+0000", 2024-09-13T11:27:23.265 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.742e-06 2024-09-13T11:27:23.265 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.265 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.265 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:27:23.265 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:58.631147+0000", 2024-09-13T11:27:23.265 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.31016212900000001 2024-09-13T11:27:23.265 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.265 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.265 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:27:23.266 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:58.631159+0000", 2024-09-13T11:27:23.266 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1946999999999999e-05 2024-09-13T11:27:23.266 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.266 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.266 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:27:23.266 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:58.631257+0000", 2024-09-13T11:27:23.266 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.8096000000000004e-05 2024-09-13T11:27:23.266 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.266 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.266 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:27:23.266 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:58.632083+0000", 2024-09-13T11:27:23.266 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00082589499999999999 2024-09-13T11:27:23.266 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:23.266 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:27:23.267 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:23.267 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.267 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.267 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:191058 3.1d 3:bd6e7029:::benchmark_data_smithi177_31116_object50755:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e175)", 2024-09-13T11:27:23.267 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:25:38.512403+0000", 2024-09-13T11:27:23.267 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 104.698172727, 2024-09-13T11:27:23.267 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.38726333200000002, 2024-09-13T11:27:23.267 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:27:23.267 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:27:23.267 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:27:23.267 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:27:23.267 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:27:23.268 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:27:23.268 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 191058 2024-09-13T11:27:23.268 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.268 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:27:23.268 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.268 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:27:23.268 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:38.512403+0000", 2024-09-13T11:27:23.268 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:23.268 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.268 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.268 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:27:23.268 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:38.512403+0000", 2024-09-13T11:27:23.268 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:23.268 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.269 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.269 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:27:23.269 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:38.512406+0000", 2024-09-13T11:27:23.269 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.4059999999999999e-06 2024-09-13T11:27:23.269 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.269 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.269 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:27:23.269 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:38.512412+0000", 2024-09-13T11:27:23.269 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.8170000000000001e-06 2024-09-13T11:27:23.269 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.269 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.269 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:27:23.269 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:38.512430+0000", 2024-09-13T11:27:23.270 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.7397999999999998e-05 2024-09-13T11:27:23.270 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.270 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.270 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:27:23.270 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:38.512438+0000", 2024-09-13T11:27:23.270 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.3459999999999994e-06 2024-09-13T11:27:23.270 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.270 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.270 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:27:23.270 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:38.512526+0000", 2024-09-13T11:27:23.270 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.8146999999999995e-05 2024-09-13T11:27:23.270 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.270 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.271 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:27:23.271 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:38.512722+0000", 2024-09-13T11:27:23.271 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00019587599999999999 2024-09-13T11:27:23.271 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.271 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.271 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 15", 2024-09-13T11:27:23.271 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:38.512882+0000", 2024-09-13T11:27:23.271 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00015957300000000001 2024-09-13T11:27:23.271 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.271 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.271 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:27:23.271 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:38.517854+0000", 2024-09-13T11:27:23.272 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0049725500000000001 2024-09-13T11:27:23.272 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.272 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.272 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:27:23.272 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:38.899562+0000", 2024-09-13T11:27:23.272 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.38170719800000003 2024-09-13T11:27:23.272 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.272 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.272 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:27:23.272 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:38.899622+0000", 2024-09-13T11:27:23.272 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.0396999999999998e-05 2024-09-13T11:27:23.272 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.272 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.273 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:27:23.273 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:38.899667+0000", 2024-09-13T11:27:23.273 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.4623999999999998e-05 2024-09-13T11:27:23.273 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:23.273 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:27:23.273 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:23.273 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.273 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.273 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.4665.0:250598 3.9 e211/117 3:929b3711:::benchmark_data_smithi177_31116_object110295:head v 211'6996, mlcod=211'6992)", 2024-09-13T11:27:23.273 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:26:32.474084+0000", 2024-09-13T11:27:23.273 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 50.736492136999999, 2024-09-13T11:27:23.273 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.24130304, 2024-09-13T11:27:23.273 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:27:23.274 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:27:23.274 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:27:23.274 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:27:23.274 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.274 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:27:23.274 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:32.474084+0000", 2024-09-13T11:27:23.274 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:23.274 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.274 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.274 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:27:23.274 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:32.474084+0000", 2024-09-13T11:27:23.274 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:23.274 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.275 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.275 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:27:23.275 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:32.474085+0000", 2024-09-13T11:27:23.275 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.2699999999999998e-07 2024-09-13T11:27:23.275 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.275 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.275 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:27:23.275 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:32.474090+0000", 2024-09-13T11:27:23.275 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.8339999999999998e-06 2024-09-13T11:27:23.275 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.275 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.275 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:27:23.275 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:32.713078+0000", 2024-09-13T11:27:23.275 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.23898883900000001 2024-09-13T11:27:23.276 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.276 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.276 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:27:23.276 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:32.713088+0000", 2024-09-13T11:27:23.276 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.5170000000000005e-06 2024-09-13T11:27:23.276 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.276 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.276 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:27:23.276 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:32.713150+0000", 2024-09-13T11:27:23.276 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.1715000000000001e-05 2024-09-13T11:27:23.276 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.276 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.276 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:27:23.277 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:32.713165+0000", 2024-09-13T11:27:23.277 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5804999999999999e-05 2024-09-13T11:27:23.277 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.277 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.277 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:27:23.277 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:32.715350+0000", 2024-09-13T11:27:23.277 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0021848979999999998 2024-09-13T11:27:23.277 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.277 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.277 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:27:23.277 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:32.715387+0000", 2024-09-13T11:27:23.277 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.6504999999999997e-05 2024-09-13T11:27:23.277 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:23.277 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:27:23.278 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:23.280 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.280 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.280 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGRecoveryDelete(3.33 e231,227 3:cf137e7b:::benchmark_data_smithi177_31116_object9019:head,149'6092)", 2024-09-13T11:27:23.280 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:26:58.970211+0000", 2024-09-13T11:27:23.280 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 24.240365384, 2024-09-13T11:27:23.280 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.41248640800000003, 2024-09-13T11:27:23.280 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:27:23.280 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:27:23.281 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:27:23.281 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:27:23.281 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.281 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:27:23.281 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:58.970211+0000", 2024-09-13T11:27:23.281 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:23.281 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.281 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.281 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:27:23.281 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:58.970211+0000", 2024-09-13T11:27:23.282 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:23.282 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.282 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.282 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:27:23.282 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:58.970218+0000", 2024-09-13T11:27:23.282 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.9019999999999999e-06 2024-09-13T11:27:23.282 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.282 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.282 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:27:23.282 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:58.970230+0000", 2024-09-13T11:27:23.282 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1341e-05 2024-09-13T11:27:23.283 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.283 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.283 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:27:23.283 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:59.382297+0000", 2024-09-13T11:27:23.283 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.41206738199999998 2024-09-13T11:27:23.283 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.283 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.283 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:27:23.283 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:59.382313+0000", 2024-09-13T11:27:23.283 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5519000000000002e-05 2024-09-13T11:27:23.283 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.283 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.284 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:27:23.284 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:59.382471+0000", 2024-09-13T11:27:23.284 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00015788900000000001 2024-09-13T11:27:23.284 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.284 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.284 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:27:23.284 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:59.382499+0000", 2024-09-13T11:27:23.284 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.8073e-05 2024-09-13T11:27:23.284 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.284 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.284 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:27:23.284 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:59.382697+0000", 2024-09-13T11:27:23.285 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000198302 2024-09-13T11:27:23.285 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:23.285 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:27:23.285 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:23.285 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.285 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.285 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGRecoveryDelete(3.37 e237,227 3:ed5582aa:::benchmark_data_smithi177_31116_object27975:head,161'6495)", 2024-09-13T11:27:23.285 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:27:05.058731+0000", 2024-09-13T11:27:23.285 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 18.151845078000001, 2024-09-13T11:27:23.285 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.55157706299999998, 2024-09-13T11:27:23.285 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:27:23.285 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:27:23.286 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:27:23.286 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:27:23.286 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.286 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:27:23.286 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:27:05.058731+0000", 2024-09-13T11:27:23.286 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:23.286 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.286 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.286 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:27:23.286 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:27:05.058731+0000", 2024-09-13T11:27:23.286 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:23.286 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.286 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.287 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:27:23.287 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:27:05.058733+0000", 2024-09-13T11:27:23.287 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.1260000000000001e-06 2024-09-13T11:27:23.287 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.287 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.287 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:27:23.287 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:27:05.058739+0000", 2024-09-13T11:27:23.287 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.3779999999999998e-06 2024-09-13T11:27:23.287 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.287 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.287 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:27:23.287 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:27:05.610144+0000", 2024-09-13T11:27:23.288 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.55140456900000001 2024-09-13T11:27:23.288 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.288 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.288 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:27:23.288 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:27:05.610156+0000", 2024-09-13T11:27:23.288 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1803e-05 2024-09-13T11:27:23.288 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.288 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.288 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:27:23.288 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:27:05.610223+0000", 2024-09-13T11:27:23.288 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.7254000000000004e-05 2024-09-13T11:27:23.288 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.288 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.289 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:27:23.289 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:27:05.610238+0000", 2024-09-13T11:27:23.289 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4841000000000001e-05 2024-09-13T11:27:23.289 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.289 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.289 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:27:23.289 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:27:05.610308+0000", 2024-09-13T11:27:23.289 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.0092000000000006e-05 2024-09-13T11:27:23.289 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:23.289 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:27:23.289 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:23.290 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.290 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.290 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:278048 3.1d 3:b90a5f9d:::benchmark_data_smithi177_31116_object137745:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e237)", 2024-09-13T11:27:23.290 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:27:05.170542+0000", 2024-09-13T11:27:23.290 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 18.040033545, 2024-09-13T11:27:23.290 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.287824792, 2024-09-13T11:27:23.290 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:27:23.290 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:27:23.290 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:27:23.290 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:27:23.290 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:27:23.290 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:27:23.291 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 278048 2024-09-13T11:27:23.291 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.291 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:27:23.291 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.291 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:27:23.291 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:27:05.170542+0000", 2024-09-13T11:27:23.291 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:23.291 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.292 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.292 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:27:23.292 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:27:05.170542+0000", 2024-09-13T11:27:23.292 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:23.292 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.292 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.292 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:27:23.292 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:27:05.170545+0000", 2024-09-13T11:27:23.292 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.3980000000000002e-06 2024-09-13T11:27:23.292 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.292 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.292 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:27:23.293 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:27:05.170551+0000", 2024-09-13T11:27:23.293 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.494e-06 2024-09-13T11:27:23.293 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.293 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.293 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:27:23.293 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:27:05.170564+0000", 2024-09-13T11:27:23.293 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2529e-05 2024-09-13T11:27:23.293 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.293 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.293 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:27:23.293 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:27:05.170571+0000", 2024-09-13T11:27:23.294 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.9160000000000004e-06 2024-09-13T11:27:23.294 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.294 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.294 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:27:23.294 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:27:05.170640+0000", 2024-09-13T11:27:23.294 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.9587999999999998e-05 2024-09-13T11:27:23.294 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.294 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.294 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:27:23.294 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:27:05.170797+0000", 2024-09-13T11:27:23.294 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00015668999999999999 2024-09-13T11:27:23.295 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.295 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.295 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 15", 2024-09-13T11:27:23.295 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:27:05.171209+0000", 2024-09-13T11:27:23.295 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00041150799999999999 2024-09-13T11:27:23.295 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.295 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.295 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:27:23.295 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:27:05.173854+0000", 2024-09-13T11:27:23.295 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0026449970000000001 2024-09-13T11:27:23.295 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.295 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.295 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:27:23.296 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:27:05.458264+0000", 2024-09-13T11:27:23.296 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.28441028699999998 2024-09-13T11:27:23.296 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.296 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.296 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:27:23.296 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:27:05.458320+0000", 2024-09-13T11:27:23.296 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.6100000000000002e-05 2024-09-13T11:27:23.296 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.296 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.296 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:27:23.296 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:27:05.458367+0000", 2024-09-13T11:27:23.296 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.7284999999999999e-05 2024-09-13T11:27:23.297 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:23.297 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:27:23.297 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:23.297 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.297 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.297 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGRecoveryDelete(3.33 e237,227 3:cfbf8319:::benchmark_data_smithi177_31116_object37164:head,168'6983)", 2024-09-13T11:27:23.297 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:27:05.427390+0000", 2024-09-13T11:27:23.297 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 17.783186006000001, 2024-09-13T11:27:23.297 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.27642200500000003, 2024-09-13T11:27:23.297 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:27:23.297 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:27:23.297 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:27:23.298 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:27:23.298 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.298 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:27:23.298 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:27:05.427390+0000", 2024-09-13T11:27:23.298 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:23.298 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.298 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.298 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:27:23.298 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:27:05.427390+0000", 2024-09-13T11:27:23.298 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:23.299 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.299 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.299 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:27:23.299 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:27:05.427391+0000", 2024-09-13T11:27:23.299 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.443e-06 2024-09-13T11:27:23.299 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.299 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.299 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:27:23.299 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:27:05.427397+0000", 2024-09-13T11:27:23.299 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.5559999999999998e-06 2024-09-13T11:27:23.300 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.300 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.300 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:27:23.300 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:27:05.703569+0000", 2024-09-13T11:27:23.300 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.27617228900000002 2024-09-13T11:27:23.300 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.300 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.300 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:27:23.300 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:27:05.703579+0000", 2024-09-13T11:27:23.300 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.7950000000000002e-06 2024-09-13T11:27:23.300 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.300 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.300 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:27:23.301 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:27:05.703700+0000", 2024-09-13T11:27:23.301 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000120906 2024-09-13T11:27:23.301 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.301 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.301 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:27:23.301 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:27:05.703708+0000", 2024-09-13T11:27:23.301 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.5150000000000006e-06 2024-09-13T11:27:23.301 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.301 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.301 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:27:23.301 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:27:05.703812+0000", 2024-09-13T11:27:23.301 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000103501 2024-09-13T11:27:23.302 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:23.302 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:27:23.302 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:23.302 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.302 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.302 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGRecoveryDelete(3.33 e238,227 3:cc092e70:::benchmark_data_smithi177_31116_object40980:head,170'7093)", 2024-09-13T11:27:23.302 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:27:06.544415+0000", 2024-09-13T11:27:23.302 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 16.666160585, 2024-09-13T11:27:23.302 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.291114812, 2024-09-13T11:27:23.302 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:27:23.302 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:27:23.303 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:27:23.303 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:27:23.303 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.303 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:27:23.303 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:27:06.544415+0000", 2024-09-13T11:27:23.303 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:23.303 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.303 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.303 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:27:23.303 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:27:06.544415+0000", 2024-09-13T11:27:23.303 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:23.303 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.304 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.304 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:27:23.304 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:27:06.544417+0000", 2024-09-13T11:27:23.304 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.61e-06 2024-09-13T11:27:23.304 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.304 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.304 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:27:23.304 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:27:06.544427+0000", 2024-09-13T11:27:23.304 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0104e-05 2024-09-13T11:27:23.304 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.304 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.304 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:27:23.304 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:27:06.835180+0000", 2024-09-13T11:27:23.304 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.290753398 2024-09-13T11:27:23.305 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.305 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.305 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:27:23.305 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:27:06.835198+0000", 2024-09-13T11:27:23.305 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.7819999999999999e-05 2024-09-13T11:27:23.305 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.305 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.305 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:27:23.305 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:27:06.835342+0000", 2024-09-13T11:27:23.305 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000143418 2024-09-13T11:27:23.305 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.305 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.305 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:27:23.306 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:27:06.835366+0000", 2024-09-13T11:27:23.306 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.4286000000000001e-05 2024-09-13T11:27:23.306 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.306 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.306 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:27:23.306 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:27:06.835530+0000", 2024-09-13T11:27:23.306 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00016417600000000001 2024-09-13T11:27:23.306 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:23.306 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:27:23.306 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:23.307 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.307 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.307 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGRecoveryDelete(3.33 e238,227 3:cebf5a5b:::benchmark_data_smithi177_31116_object41040:head,170'7096)", 2024-09-13T11:27:23.307 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:27:06.550510+0000", 2024-09-13T11:27:23.307 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 16.660065695, 2024-09-13T11:27:23.307 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.29742270900000001, 2024-09-13T11:27:23.307 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:27:23.307 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:27:23.307 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:27:23.307 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:27:23.307 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.307 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:27:23.308 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:27:06.550510+0000", 2024-09-13T11:27:23.308 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:23.308 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.308 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.308 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:27:23.308 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:27:06.550510+0000", 2024-09-13T11:27:23.308 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:23.308 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.308 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.308 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:27:23.308 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:27:06.550519+0000", 2024-09-13T11:27:23.309 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.4200000000000007e-06 2024-09-13T11:27:23.309 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.309 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.309 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:27:23.309 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:27:06.550529+0000", 2024-09-13T11:27:23.309 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0017000000000001e-05 2024-09-13T11:27:23.310 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.310 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.310 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:27:23.310 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:27:06.847633+0000", 2024-09-13T11:27:23.310 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.29710405099999998 2024-09-13T11:27:23.310 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.310 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.310 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:27:23.310 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:27:06.847654+0000", 2024-09-13T11:27:23.310 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.1603e-05 2024-09-13T11:27:23.310 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.310 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.310 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:27:23.311 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:27:06.847798+0000", 2024-09-13T11:27:23.311 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00014326900000000001 2024-09-13T11:27:23.311 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.311 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.311 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:27:23.311 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:27:06.847814+0000", 2024-09-13T11:27:23.311 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.6826000000000001e-05 2024-09-13T11:27:23.311 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.311 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.311 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:27:23.311 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:27:06.847933+0000", 2024-09-13T11:27:23.312 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000118523 2024-09-13T11:27:23.312 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:23.312 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:27:23.312 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:23.312 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.312 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.312 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGRecoveryDelete(3.37 e246,227 3:ee1375fa:::benchmark_data_smithi177_31116_object103292:head,207'8773)", 2024-09-13T11:27:23.312 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:27:17.585798+0000", 2024-09-13T11:27:23.312 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 5.6247784010000004, 2024-09-13T11:27:23.312 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.246478583, 2024-09-13T11:27:23.312 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:27:23.312 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:27:23.313 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:27:23.313 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:27:23.313 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.313 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:27:23.313 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:27:17.585798+0000", 2024-09-13T11:27:23.313 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:23.313 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.313 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.313 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:27:23.313 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:27:17.585798+0000", 2024-09-13T11:27:23.313 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:23.313 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.313 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.314 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:27:23.314 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:27:17.585799+0000", 2024-09-13T11:27:23.314 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3459999999999999e-06 2024-09-13T11:27:23.314 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.314 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.314 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:27:23.314 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:27:17.585805+0000", 2024-09-13T11:27:23.314 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.0240000000000003e-06 2024-09-13T11:27:23.314 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.314 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.314 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:27:23.314 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:27:17.832155+0000", 2024-09-13T11:27:23.314 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.24635038500000001 2024-09-13T11:27:23.315 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.315 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.315 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:27:23.315 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:27:17.832163+0000", 2024-09-13T11:27:23.315 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.1920000000000005e-06 2024-09-13T11:27:23.315 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.315 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.315 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:27:23.315 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:27:17.832214+0000", 2024-09-13T11:27:23.315 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.0322999999999999e-05 2024-09-13T11:27:23.315 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.315 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.315 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:27:23.316 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:27:17.832220+0000", 2024-09-13T11:27:23.316 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.4790000000000001e-06 2024-09-13T11:27:23.316 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.316 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.316 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:27:23.316 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:27:17.832276+0000", 2024-09-13T11:27:23.316 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.5834000000000001e-05 2024-09-13T11:27:23.316 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:23.316 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:27:23.316 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:23.316 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:23.316 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:27:23.317 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-13T11:27:23.317 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-09-13T11:27:23.219+0000 7f97b39f6640 -1 osd.6 251 handle_osd_map injecting map crc failure 2024-09-13T11:27:23.318 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-13T11:27:23.320 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-objectstore-tool --err-to-stderr --data-path /var/lib/ceph/osd/ceph-10 --journal-path /var/lib/ceph/osd/ceph-10/journal '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --op import --file /var/log/ceph/exp.3.32.10 2024-09-13T11:27:23.487 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-13T11:27:23.487 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [], 2024-09-13T11:27:23.487 INFO:teuthology.orchestra.run.smithi162.stdout: "num_ops": 0 2024-09-13T11:27:23.488 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-13T11:27:23.502 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-13T11:27:23.624 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-13T11:27:23.624 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [], 2024-09-13T11:27:23.624 INFO:teuthology.orchestra.run.smithi162.stdout: "complaint_time": 30, 2024-09-13T11:27:23.624 INFO:teuthology.orchestra.run.smithi162.stdout: "num_blocked_ops": 0 2024-09-13T11:27:23.624 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-13T11:27:23.638 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-13T11:27:23.763 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-13T11:27:23.763 INFO:teuthology.orchestra.run.smithi162.stdout: "size": 20, 2024-09-13T11:27:23.763 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 600, 2024-09-13T11:27:23.763 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [ 2024-09-13T11:27:23.763 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.764 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4257.0:80 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 262144~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-13T11:27:23.764 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:22:12.516065+0000", 2024-09-13T11:27:23.764 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 311.246634384, 2024-09-13T11:27:23.764 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.46461997999999999, 2024-09-13T11:27:23.764 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:27:23.764 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:27:23.764 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:27:23.764 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:27:23.764 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4257", 2024-09-13T11:27:23.764 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/1488131527", 2024-09-13T11:27:23.764 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 80 2024-09-13T11:27:23.764 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.764 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:27:23.765 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.765 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:27:23.765 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.516065+0000", 2024-09-13T11:27:23.765 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:23.765 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.765 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.765 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:27:23.765 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.516065+0000", 2024-09-13T11:27:23.765 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:23.765 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.765 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.765 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:27:23.765 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.516066+0000", 2024-09-13T11:27:23.765 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0869999999999999e-06 2024-09-13T11:27:23.766 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.766 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.766 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:27:23.766 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.516112+0000", 2024-09-13T11:27:23.766 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.5503000000000001e-05 2024-09-13T11:27:23.766 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.766 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.766 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:27:23.766 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.516128+0000", 2024-09-13T11:27:23.766 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5891000000000001e-05 2024-09-13T11:27:23.766 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.766 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.766 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:27:23.766 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.516138+0000", 2024-09-13T11:27:23.767 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.8120000000000002e-06 2024-09-13T11:27:23.767 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.767 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.767 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:27:23.767 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.520398+0000", 2024-09-13T11:27:23.767 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0042605450000000001 2024-09-13T11:27:23.767 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.767 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.767 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:27:23.767 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.520526+0000", 2024-09-13T11:27:23.767 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000128355 2024-09-13T11:27:23.767 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.767 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.768 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 0", 2024-09-13T11:27:23.768 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.520730+0000", 2024-09-13T11:27:23.768 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00020337799999999999 2024-09-13T11:27:23.768 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.768 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.768 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:27:23.768 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.522417+0000", 2024-09-13T11:27:23.768 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001687184 2024-09-13T11:27:23.768 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.768 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.768 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:27:23.768 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.980408+0000", 2024-09-13T11:27:23.768 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.45799068399999998 2024-09-13T11:27:23.769 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.769 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.769 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:27:23.769 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.980565+0000", 2024-09-13T11:27:23.769 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00015760899999999999 2024-09-13T11:27:23.769 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.769 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.769 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:27:23.769 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.980685+0000", 2024-09-13T11:27:23.769 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000119932 2024-09-13T11:27:23.769 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:23.769 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:27:23.769 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:23.769 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.770 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.770 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop_reply(client.4257.0:80 1.0 e24/23 ondisk, result = 0)", 2024-09-13T11:27:23.770 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:22:12.522938+0000", 2024-09-13T11:27:23.770 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 311.23976135100003, 2024-09-13T11:27:23.770 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.45776420600000001, 2024-09-13T11:27:23.770 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:27:23.770 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:27:23.770 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:27:23.770 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:27:23.770 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.770 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:27:23.770 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.522938+0000", 2024-09-13T11:27:23.770 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:23.771 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.771 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.771 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:27:23.771 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.522938+0000", 2024-09-13T11:27:23.771 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:23.771 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.771 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.771 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:27:23.771 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.522941+0000", 2024-09-13T11:27:23.771 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.0290000000000001e-06 2024-09-13T11:27:23.771 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.771 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.771 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:27:23.771 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.522950+0000", 2024-09-13T11:27:23.772 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.7499999999999992e-06 2024-09-13T11:27:23.772 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.772 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.772 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:27:23.772 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.980215+0000", 2024-09-13T11:27:23.772 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.45726523800000002 2024-09-13T11:27:23.772 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.772 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.772 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:27:23.772 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.980229+0000", 2024-09-13T11:27:23.772 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3549e-05 2024-09-13T11:27:23.772 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.772 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.772 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:27:23.773 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.980365+0000", 2024-09-13T11:27:23.773 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00013608700000000001 2024-09-13T11:27:23.773 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.773 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.773 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:27:23.773 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.980388+0000", 2024-09-13T11:27:23.773 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.3112000000000002e-05 2024-09-13T11:27:23.773 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.773 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.773 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:27:23.773 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:22:12.980702+0000", 2024-09-13T11:27:23.773 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00031444099999999999 2024-09-13T11:27:23.773 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:23.774 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:27:23.774 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:23.774 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.774 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.774 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:30833 3.8 3:1c203f8d:::benchmark_data_smithi177_31116_object30832:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e59)", 2024-09-13T11:27:23.774 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:09.506656+0000", 2024-09-13T11:27:23.774 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 254.256043834, 2024-09-13T11:27:23.774 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.224862282, 2024-09-13T11:27:23.774 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:27:23.774 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:27:23.774 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:27:23.774 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:27:23.774 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:27:23.775 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:27:23.775 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 30833 2024-09-13T11:27:23.775 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.775 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:27:23.775 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.775 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:27:23.775 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.506656+0000", 2024-09-13T11:27:23.775 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:23.775 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.776 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.776 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:27:23.776 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.506656+0000", 2024-09-13T11:27:23.776 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:23.776 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.776 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.776 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:27:23.776 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.506660+0000", 2024-09-13T11:27:23.776 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.6e-06 2024-09-13T11:27:23.776 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.776 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.776 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:27:23.776 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.506717+0000", 2024-09-13T11:27:23.776 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.6131999999999997e-05 2024-09-13T11:27:23.777 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.777 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.777 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:27:23.777 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.506751+0000", 2024-09-13T11:27:23.777 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.4746000000000002e-05 2024-09-13T11:27:23.777 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.777 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.777 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:27:23.777 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.506774+0000", 2024-09-13T11:27:23.777 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.2636000000000001e-05 2024-09-13T11:27:23.777 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.777 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.777 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:27:23.777 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.506878+0000", 2024-09-13T11:27:23.778 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000104057 2024-09-13T11:27:23.778 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.778 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.778 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:27:23.778 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.507052+0000", 2024-09-13T11:27:23.778 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00017402800000000001 2024-09-13T11:27:23.778 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.778 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.778 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 1", 2024-09-13T11:27:23.778 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.507217+0000", 2024-09-13T11:27:23.778 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00016516 2024-09-13T11:27:23.778 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.779 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.779 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:27:23.779 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.508992+0000", 2024-09-13T11:27:23.779 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0017743990000000001 2024-09-13T11:27:23.779 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.779 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.779 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:27:23.779 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.731429+0000", 2024-09-13T11:27:23.779 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.22243718000000001 2024-09-13T11:27:23.779 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.779 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.779 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:27:23.779 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.731479+0000", 2024-09-13T11:27:23.779 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.0146999999999999e-05 2024-09-13T11:27:23.779 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.780 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.780 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:27:23.780 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.731518+0000", 2024-09-13T11:27:23.780 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.9196999999999998e-05 2024-09-13T11:27:23.780 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:23.780 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:27:23.780 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:23.780 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.780 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.780 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop_reply(client.4665.0:30833 3.8 e59/40 ondisk, result = 0)", 2024-09-13T11:27:23.780 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:09.508907+0000", 2024-09-13T11:27:23.780 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 254.25379273999999, 2024-09-13T11:27:23.780 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.222618862, 2024-09-13T11:27:23.781 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:27:23.781 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:27:23.781 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:27:23.781 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:27:23.781 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.781 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:27:23.781 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.508907+0000", 2024-09-13T11:27:23.781 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:23.781 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.781 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.781 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:27:23.781 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.508907+0000", 2024-09-13T11:27:23.781 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:23.781 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.782 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.782 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:27:23.782 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.508910+0000", 2024-09-13T11:27:23.782 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.993e-06 2024-09-13T11:27:23.782 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.782 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.782 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:27:23.782 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.508918+0000", 2024-09-13T11:27:23.782 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.7000000000000008e-06 2024-09-13T11:27:23.782 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.782 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.782 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:27:23.782 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.731341+0000", 2024-09-13T11:27:23.783 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.22242308699999999 2024-09-13T11:27:23.783 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.783 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.783 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:27:23.783 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.731350+0000", 2024-09-13T11:27:23.783 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.3179999999999996e-06 2024-09-13T11:27:23.783 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.783 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.783 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:27:23.783 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.731418+0000", 2024-09-13T11:27:23.783 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.7586e-05 2024-09-13T11:27:23.783 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.783 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.783 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:27:23.784 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.731424+0000", 2024-09-13T11:27:23.784 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.844e-06 2024-09-13T11:27:23.784 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.784 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.784 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:27:23.784 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:09.731526+0000", 2024-09-13T11:27:23.784 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000101334 2024-09-13T11:27:23.784 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:23.784 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:27:23.784 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:23.784 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.784 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.784 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPush(3.10 76/70 [PushOp(3:0e7d2700:::benchmark_data_smithi177_31116_object17406:head, version: 52'1075, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:0e7d2700:::benchmark_data_smithi177_31116_object17406:head@52'1075, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:27:23.784 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:32.206198+0000", 2024-09-13T11:27:23.785 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 231.55650203100001, 2024-09-13T11:27:23.785 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.25193395000000002, 2024-09-13T11:27:23.785 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:27:23.785 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:27:23.785 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:27:23.785 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:27:23.785 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.785 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:27:23.785 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.206198+0000", 2024-09-13T11:27:23.785 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:23.785 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.785 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.785 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:27:23.786 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.206198+0000", 2024-09-13T11:27:23.786 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:23.786 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.786 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.786 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:27:23.786 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.206199+0000", 2024-09-13T11:27:23.786 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.46e-06 2024-09-13T11:27:23.786 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.786 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.786 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:27:23.786 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.206250+0000", 2024-09-13T11:27:23.786 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.113e-05 2024-09-13T11:27:23.786 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.786 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.786 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:27:23.787 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.456042+0000", 2024-09-13T11:27:23.787 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.24979182999999999 2024-09-13T11:27:23.787 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.787 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.787 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:27:23.787 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.456084+0000", 2024-09-13T11:27:23.787 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.1814000000000003e-05 2024-09-13T11:27:23.787 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.787 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.787 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:27:23.787 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.456342+0000", 2024-09-13T11:27:23.787 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000258404 2024-09-13T11:27:23.787 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.787 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.788 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:27:23.788 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.456367+0000", 2024-09-13T11:27:23.788 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.5099000000000001e-05 2024-09-13T11:27:23.788 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.788 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.788 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:27:23.788 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:32.458132+0000", 2024-09-13T11:27:23.788 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0017642129999999999 2024-09-13T11:27:23.788 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:23.788 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:27:23.788 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:23.788 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.788 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.789 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:60417 3.10 3:092808b0:::benchmark_data_smithi177_31116_object60416:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e85)", 2024-09-13T11:27:23.789 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:23:42.173238+0000", 2024-09-13T11:27:23.789 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 221.58946112300001, 2024-09-13T11:27:23.789 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.38965994599999998, 2024-09-13T11:27:23.789 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:27:23.789 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:27:23.789 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:27:23.789 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:27:23.789 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:27:23.789 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:27:23.789 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 60417 2024-09-13T11:27:23.789 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.789 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:27:23.789 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.790 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:27:23.790 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.173238+0000", 2024-09-13T11:27:23.790 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:23.790 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.790 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.790 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:27:23.790 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.173238+0000", 2024-09-13T11:27:23.790 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:23.790 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.790 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.790 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:27:23.790 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.173241+0000", 2024-09-13T11:27:23.790 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.6819999999999999e-06 2024-09-13T11:27:23.790 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.791 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.791 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:27:23.791 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.173335+0000", 2024-09-13T11:27:23.791 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.3957999999999997e-05 2024-09-13T11:27:23.791 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.791 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.791 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:27:23.791 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.173358+0000", 2024-09-13T11:27:23.791 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.3023999999999998e-05 2024-09-13T11:27:23.791 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.791 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.791 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:27:23.792 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.173373+0000", 2024-09-13T11:27:23.792 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4867e-05 2024-09-13T11:27:23.792 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.792 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.792 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:27:23.792 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.173466+0000", 2024-09-13T11:27:23.792 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.2542999999999998e-05 2024-09-13T11:27:23.792 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.792 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.792 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:27:23.792 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.173754+0000", 2024-09-13T11:27:23.792 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000288657 2024-09-13T11:27:23.793 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.793 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.793 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 0", 2024-09-13T11:27:23.793 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.174066+0000", 2024-09-13T11:27:23.793 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00031170999999999998 2024-09-13T11:27:23.793 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.793 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.793 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:27:23.793 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.178245+0000", 2024-09-13T11:27:23.793 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00417864 2024-09-13T11:27:23.793 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.793 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.793 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:27:23.794 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.562617+0000", 2024-09-13T11:27:23.794 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.38437253199999999 2024-09-13T11:27:23.794 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.794 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.794 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:27:23.794 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.562782+0000", 2024-09-13T11:27:23.794 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00016499200000000001 2024-09-13T11:27:23.794 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.794 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.794 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:27:23.794 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:23:42.562898+0000", 2024-09-13T11:27:23.794 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000116341 2024-09-13T11:27:23.794 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:23.795 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:27:23.795 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:23.795 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.795 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.795 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:83193 3.8 3:10066d83:::benchmark_data_smithi177_31116_object83192:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106)", 2024-09-13T11:27:23.795 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:24:04.283923+0000", 2024-09-13T11:27:23.795 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 199.47877643999999, 2024-09-13T11:27:23.795 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.64775463499999997, 2024-09-13T11:27:23.795 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:27:23.795 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:27:23.795 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:27:23.795 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:27:23.795 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:27:23.796 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:27:23.796 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 83193 2024-09-13T11:27:23.796 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.796 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:27:23.796 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.796 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:27:23.796 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.283923+0000", 2024-09-13T11:27:23.796 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:23.796 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.796 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.796 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:27:23.796 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.283923+0000", 2024-09-13T11:27:23.796 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:23.797 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.797 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.797 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:27:23.797 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.283925+0000", 2024-09-13T11:27:23.797 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.9489999999999999e-06 2024-09-13T11:27:23.797 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.797 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.797 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:27:23.797 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.283960+0000", 2024-09-13T11:27:23.797 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.5063e-05 2024-09-13T11:27:23.797 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.797 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.797 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:27:23.797 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.283974+0000", 2024-09-13T11:27:23.798 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3546e-05 2024-09-13T11:27:23.798 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.798 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.798 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:27:23.798 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.283983+0000", 2024-09-13T11:27:23.798 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.2499999999999995e-06 2024-09-13T11:27:23.798 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.798 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.798 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:27:23.798 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.284061+0000", 2024-09-13T11:27:23.798 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.7946000000000002e-05 2024-09-13T11:27:23.798 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.798 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.798 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:27:23.799 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.284270+0000", 2024-09-13T11:27:23.799 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00020876799999999999 2024-09-13T11:27:23.799 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.799 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.799 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 1", 2024-09-13T11:27:23.799 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.284489+0000", 2024-09-13T11:27:23.799 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00021956800000000001 2024-09-13T11:27:23.799 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.799 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.799 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:27:23.799 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.286456+0000", 2024-09-13T11:27:23.799 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0019665809999999998 2024-09-13T11:27:23.799 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.799 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.800 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:27:23.800 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.931497+0000", 2024-09-13T11:27:23.800 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.64504147700000003 2024-09-13T11:27:23.800 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.800 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.800 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:27:23.800 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.931589+0000", 2024-09-13T11:27:23.800 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.2041999999999997e-05 2024-09-13T11:27:23.800 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.800 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.800 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:27:23.800 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:04.931678+0000", 2024-09-13T11:27:23.800 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.8444999999999998e-05 2024-09-13T11:27:23.801 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:23.801 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:27:23.801 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:23.801 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.801 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.801 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:115060 3.8 3:11200ba9:::benchmark_data_smithi177_31116_object115059:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e125)", 2024-09-13T11:27:23.801 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:24:33.602178+0000", 2024-09-13T11:27:23.801 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 170.16052144599999, 2024-09-13T11:27:23.801 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.23668186799999999, 2024-09-13T11:27:23.801 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:27:23.801 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:27:23.801 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:27:23.801 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:27:23.802 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:27:23.802 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:27:23.802 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 115060 2024-09-13T11:27:23.802 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.802 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:27:23.802 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.802 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:27:23.802 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:33.602178+0000", 2024-09-13T11:27:23.802 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:23.802 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.802 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.802 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:27:23.802 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:33.602178+0000", 2024-09-13T11:27:23.802 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:23.803 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.803 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.803 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:27:23.803 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:33.602181+0000", 2024-09-13T11:27:23.803 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.0369999999999999e-06 2024-09-13T11:27:23.803 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.803 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.803 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:27:23.803 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:33.602217+0000", 2024-09-13T11:27:23.803 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.5837999999999998e-05 2024-09-13T11:27:23.803 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.803 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.803 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:27:23.803 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:33.602235+0000", 2024-09-13T11:27:23.803 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.7571000000000002e-05 2024-09-13T11:27:23.804 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.804 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.804 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:27:23.804 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:33.602245+0000", 2024-09-13T11:27:23.804 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0360000000000001e-05 2024-09-13T11:27:23.804 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.804 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.804 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:27:23.804 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:33.602312+0000", 2024-09-13T11:27:23.804 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.6989000000000005e-05 2024-09-13T11:27:23.804 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.804 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.804 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:27:23.804 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:33.602490+0000", 2024-09-13T11:27:23.805 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000178464 2024-09-13T11:27:23.805 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.805 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.805 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 1", 2024-09-13T11:27:23.805 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:33.602670+0000", 2024-09-13T11:27:23.805 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00018002799999999999 2024-09-13T11:27:23.805 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.805 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.805 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:27:23.805 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:33.604382+0000", 2024-09-13T11:27:23.805 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001711314 2024-09-13T11:27:23.805 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.805 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.805 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:27:23.805 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:33.838738+0000", 2024-09-13T11:27:23.806 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.23435634699999999 2024-09-13T11:27:23.806 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.806 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.806 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:27:23.806 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:33.838815+0000", 2024-09-13T11:27:23.806 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.7381999999999999e-05 2024-09-13T11:27:23.806 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.806 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.806 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:27:23.806 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:33.838860+0000", 2024-09-13T11:27:23.806 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.4537999999999999e-05 2024-09-13T11:27:23.806 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:23.806 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:27:23.807 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:23.807 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.807 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.807 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop_reply(client.4665.0:115060 3.8 e125/117 ondisk, result = 0)", 2024-09-13T11:27:23.807 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:24:33.604404+0000", 2024-09-13T11:27:23.807 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 170.15829518800001, 2024-09-13T11:27:23.807 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.23446235400000001, 2024-09-13T11:27:23.807 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:27:23.807 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:27:23.807 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:27:23.807 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:27:23.807 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.807 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:27:23.807 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:33.604404+0000", 2024-09-13T11:27:23.808 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:23.808 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.808 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.808 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:27:23.808 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:33.604404+0000", 2024-09-13T11:27:23.808 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:23.808 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.808 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.808 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:27:23.808 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:33.604406+0000", 2024-09-13T11:27:23.808 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.418e-06 2024-09-13T11:27:23.808 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.808 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.808 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:27:23.809 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:33.604411+0000", 2024-09-13T11:27:23.809 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.3519999999999997e-06 2024-09-13T11:27:23.809 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.809 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.809 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:27:23.809 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:33.838578+0000", 2024-09-13T11:27:23.809 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.23416656199999999 2024-09-13T11:27:23.809 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.809 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.809 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:27:23.809 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:33.838586+0000", 2024-09-13T11:27:23.809 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.9920000000000007e-06 2024-09-13T11:27:23.809 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.810 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.810 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:27:23.810 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:33.838655+0000", 2024-09-13T11:27:23.810 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.9463999999999998e-05 2024-09-13T11:27:23.810 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.810 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.810 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:27:23.810 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:33.838731+0000", 2024-09-13T11:27:23.810 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.5487000000000004e-05 2024-09-13T11:27:23.810 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.810 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.810 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:27:23.810 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:33.838867+0000", 2024-09-13T11:27:23.810 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000136079 2024-09-13T11:27:23.811 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:23.811 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:27:23.811 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:23.811 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.811 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.811 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.4665.0:126919 3.6 e130/122 3:634ce08e:::benchmark_data_smithi177_31116_object126918:head v 130'5137, mlcod=130'5134)", 2024-09-13T11:27:23.811 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:24:43.780120+0000", 2024-09-13T11:27:23.811 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 159.982579886, 2024-09-13T11:27:23.811 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.298086295, 2024-09-13T11:27:23.811 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:27:23.811 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:27:23.811 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:27:23.811 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:27:23.812 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.812 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:27:23.812 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:43.780120+0000", 2024-09-13T11:27:23.812 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:23.812 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.812 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.812 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:27:23.812 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:43.780120+0000", 2024-09-13T11:27:23.812 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:23.812 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.812 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.812 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:27:23.812 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:43.780123+0000", 2024-09-13T11:27:23.812 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.79e-06 2024-09-13T11:27:23.813 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.813 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.813 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:27:23.813 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:43.780160+0000", 2024-09-13T11:27:23.813 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.7945999999999999e-05 2024-09-13T11:27:23.813 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.813 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.813 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:27:23.813 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:44.075853+0000", 2024-09-13T11:27:23.813 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.29569234999999999 2024-09-13T11:27:23.813 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.813 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.813 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:27:23.813 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:44.075861+0000", 2024-09-13T11:27:23.814 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.9510000000000006e-06 2024-09-13T11:27:23.814 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.814 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.814 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:27:23.814 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:44.075924+0000", 2024-09-13T11:27:23.814 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.2946000000000003e-05 2024-09-13T11:27:23.814 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.814 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.814 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:27:23.814 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:44.075942+0000", 2024-09-13T11:27:23.814 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.8108e-05 2024-09-13T11:27:23.814 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.814 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.815 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:27:23.815 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:44.078164+0000", 2024-09-13T11:27:23.815 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.002222375 2024-09-13T11:27:23.815 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.815 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.815 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:27:23.815 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:44.078206+0000", 2024-09-13T11:27:23.815 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.1829000000000002e-05 2024-09-13T11:27:23.815 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:23.815 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:27:23.815 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:23.815 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.815 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.816 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:137780 3.8 3:1047f268:::benchmark_data_smithi177_31116_object137779:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e139)", 2024-09-13T11:27:23.816 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:24:53.795212+0000", 2024-09-13T11:27:23.816 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 149.967488102, 2024-09-13T11:27:23.816 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.21530981399999999, 2024-09-13T11:27:23.816 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:27:23.816 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:27:23.816 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:27:23.816 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:27:23.816 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:27:23.816 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:27:23.816 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 137780 2024-09-13T11:27:23.816 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.816 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:27:23.817 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.817 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:27:23.817 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:53.795212+0000", 2024-09-13T11:27:23.817 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:23.817 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.817 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.817 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:27:23.817 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:53.795212+0000", 2024-09-13T11:27:23.817 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:23.817 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.817 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.817 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:27:23.817 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:53.795214+0000", 2024-09-13T11:27:23.817 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.481e-06 2024-09-13T11:27:23.818 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.818 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.818 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:27:23.818 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:53.795247+0000", 2024-09-13T11:27:23.818 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.3105999999999998e-05 2024-09-13T11:27:23.818 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.818 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.818 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:27:23.818 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:53.795258+0000", 2024-09-13T11:27:23.818 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0872e-05 2024-09-13T11:27:23.818 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.818 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.818 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:27:23.818 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:53.795265+0000", 2024-09-13T11:27:23.818 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.5729999999999996e-06 2024-09-13T11:27:23.819 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.819 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.819 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:27:23.819 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:53.795309+0000", 2024-09-13T11:27:23.819 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.4617999999999999e-05 2024-09-13T11:27:23.819 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.819 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.819 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:27:23.819 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:53.795397+0000", 2024-09-13T11:27:23.819 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.7863999999999998e-05 2024-09-13T11:27:23.819 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.819 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.819 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 1", 2024-09-13T11:27:23.819 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:53.795493+0000", 2024-09-13T11:27:23.820 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.6062000000000003e-05 2024-09-13T11:27:23.820 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.820 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.820 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:27:23.820 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:53.797245+0000", 2024-09-13T11:27:23.820 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0017521559999999999 2024-09-13T11:27:23.820 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.820 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.820 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:27:23.820 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:54.010409+0000", 2024-09-13T11:27:23.820 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.21316385299999999 2024-09-13T11:27:23.820 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.820 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.820 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:27:23.821 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:54.010474+0000", 2024-09-13T11:27:23.821 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.4998000000000003e-05 2024-09-13T11:27:23.821 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.821 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.821 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:27:23.821 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:54.010521+0000", 2024-09-13T11:27:23.821 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.7231000000000002e-05 2024-09-13T11:27:23.821 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:23.821 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:27:23.821 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:23.821 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.821 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.821 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop_reply(client.4665.0:137780 3.8 e139/117 ondisk, result = 0)", 2024-09-13T11:27:23.821 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:24:53.797821+0000", 2024-09-13T11:27:23.822 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 149.964878204, 2024-09-13T11:27:23.822 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.21270765599999999, 2024-09-13T11:27:23.822 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:27:23.822 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:27:23.822 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:27:23.822 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:27:23.822 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.822 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:27:23.822 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:53.797821+0000", 2024-09-13T11:27:23.822 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:23.822 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.822 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.822 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:27:23.822 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:53.797821+0000", 2024-09-13T11:27:23.823 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:23.823 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.823 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.823 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:27:23.823 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:53.797822+0000", 2024-09-13T11:27:23.823 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.012e-06 2024-09-13T11:27:23.823 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.823 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.823 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:27:23.823 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:53.797829+0000", 2024-09-13T11:27:23.823 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.8009999999999998e-06 2024-09-13T11:27:23.823 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.823 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.824 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:27:23.824 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:54.010334+0000", 2024-09-13T11:27:23.824 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.212504477 2024-09-13T11:27:23.824 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.824 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.824 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:27:23.824 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:54.010342+0000", 2024-09-13T11:27:23.824 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.6549999999999993e-06 2024-09-13T11:27:23.824 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.824 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.824 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:27:23.824 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:54.010394+0000", 2024-09-13T11:27:23.824 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.1875000000000001e-05 2024-09-13T11:27:23.824 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.825 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.825 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:27:23.825 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:54.010402+0000", 2024-09-13T11:27:23.825 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.6499999999999996e-06 2024-09-13T11:27:23.825 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.825 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.825 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:27:23.825 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:24:54.010529+0000", 2024-09-13T11:27:23.825 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000127186 2024-09-13T11:27:23.825 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:23.825 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:27:23.825 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:23.825 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.826 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.826 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.4665.0:196314 3.6 e178/122 3:62e79b1c:::benchmark_data_smithi177_31116_object56011:head v 178'6256, mlcod=178'6253)", 2024-09-13T11:27:23.826 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:25:43.340693+0000", 2024-09-13T11:27:23.826 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 100.422006623, 2024-09-13T11:27:23.826 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.24109011699999999, 2024-09-13T11:27:23.826 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:27:23.826 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:27:23.826 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:27:23.826 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:27:23.826 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.826 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:27:23.826 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:43.340693+0000", 2024-09-13T11:27:23.826 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:23.827 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.827 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.827 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:27:23.827 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:43.340693+0000", 2024-09-13T11:27:23.827 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:23.827 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.827 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.827 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:27:23.827 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:43.340704+0000", 2024-09-13T11:27:23.827 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1049e-05 2024-09-13T11:27:23.827 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.827 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.827 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:27:23.827 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:43.340718+0000", 2024-09-13T11:27:23.828 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4214e-05 2024-09-13T11:27:23.828 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.828 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.828 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:27:23.828 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:43.579964+0000", 2024-09-13T11:27:23.828 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.23924590200000001 2024-09-13T11:27:23.828 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.828 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.828 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:27:23.828 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:43.579972+0000", 2024-09-13T11:27:23.828 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.2789999999999998e-06 2024-09-13T11:27:23.828 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.828 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.829 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:27:23.829 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:43.580043+0000", 2024-09-13T11:27:23.829 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.0856000000000001e-05 2024-09-13T11:27:23.829 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.829 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.829 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:27:23.829 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:43.580065+0000", 2024-09-13T11:27:23.829 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.1795999999999999e-05 2024-09-13T11:27:23.829 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.829 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.829 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:27:23.829 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:43.581741+0000", 2024-09-13T11:27:23.829 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0016759190000000001 2024-09-13T11:27:23.829 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.830 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.830 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:27:23.830 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:25:43.581783+0000", 2024-09-13T11:27:23.830 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.2101999999999998e-05 2024-09-13T11:27:23.830 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:23.830 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:27:23.830 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:23.830 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.830 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.830 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:266873 3.30 3:0f4a798b:::benchmark_data_smithi177_31116_object126570:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e224)", 2024-09-13T11:27:23.830 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:26:49.175028+0000", 2024-09-13T11:27:23.830 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 34.587671936, 2024-09-13T11:27:23.830 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.89281703099999998, 2024-09-13T11:27:23.830 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:27:23.831 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:27:23.831 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:27:23.831 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:27:23.831 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:27:23.831 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:27:23.831 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 266873 2024-09-13T11:27:23.831 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.831 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:27:23.831 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.831 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:27:23.831 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:49.175028+0000", 2024-09-13T11:27:23.831 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:23.831 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.831 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.832 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:27:23.832 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:49.175028+0000", 2024-09-13T11:27:23.832 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:23.832 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.832 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.832 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:27:23.832 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:49.175031+0000", 2024-09-13T11:27:23.832 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.4300000000000002e-06 2024-09-13T11:27:23.832 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.832 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.832 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:27:23.832 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:49.175041+0000", 2024-09-13T11:27:23.832 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.6299999999999993e-06 2024-09-13T11:27:23.832 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.832 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.833 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:27:23.833 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:49.175066+0000", 2024-09-13T11:27:23.833 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.5193000000000001e-05 2024-09-13T11:27:23.833 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.833 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.833 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:27:23.833 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:49.175084+0000", 2024-09-13T11:27:23.833 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.853e-05 2024-09-13T11:27:23.833 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.833 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.833 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:27:23.833 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:49.253283+0000", 2024-09-13T11:27:23.833 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.078198765000000003 2024-09-13T11:27:23.833 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.834 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.834 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-09-13T11:27:23.834 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:49.253292+0000", 2024-09-13T11:27:23.834 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.0920000000000004e-06 2024-09-13T11:27:23.834 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.834 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.834 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:27:23.834 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:50.059052+0000", 2024-09-13T11:27:23.834 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.80575967599999998 2024-09-13T11:27:23.834 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.834 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.834 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:27:23.834 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:50.059226+0000", 2024-09-13T11:27:23.834 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00017433499999999999 2024-09-13T11:27:23.835 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.835 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.835 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 0", 2024-09-13T11:27:23.835 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:50.059374+0000", 2024-09-13T11:27:23.835 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00014768600000000001 2024-09-13T11:27:23.835 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.835 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.835 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:27:23.835 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:50.062440+0000", 2024-09-13T11:27:23.835 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.003066086 2024-09-13T11:27:23.835 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.835 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.835 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:27:23.835 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:50.067764+0000", 2024-09-13T11:27:23.835 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0053240370000000002 2024-09-13T11:27:23.836 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.836 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.836 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:27:23.836 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:50.067802+0000", 2024-09-13T11:27:23.836 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.7462000000000001e-05 2024-09-13T11:27:23.836 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.836 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.836 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:27:23.836 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:50.067845+0000", 2024-09-13T11:27:23.836 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.3108999999999997e-05 2024-09-13T11:27:23.836 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:23.836 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:27:23.836 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:23.837 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.837 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.837 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGRecoveryDelete(3.33 e231,227 3:cf137e7b:::benchmark_data_smithi177_31116_object9019:head,149'6092)", 2024-09-13T11:27:23.837 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:26:58.969284+0000", 2024-09-13T11:27:23.837 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 24.793415574000001, 2024-09-13T11:27:23.837 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.41337520300000002, 2024-09-13T11:27:23.837 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:27:23.837 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:27:23.837 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:27:23.837 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:27:23.837 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.837 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:27:23.837 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:58.969284+0000", 2024-09-13T11:27:23.838 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:23.838 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.838 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.838 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:27:23.838 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:58.969284+0000", 2024-09-13T11:27:23.838 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:23.838 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.838 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.838 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:27:23.838 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:58.969286+0000", 2024-09-13T11:27:23.838 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.1330000000000002e-06 2024-09-13T11:27:23.838 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.838 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.839 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:27:23.839 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:58.969295+0000", 2024-09-13T11:27:23.839 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.6740000000000003e-06 2024-09-13T11:27:23.839 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.839 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.839 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:27:23.839 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:59.382366+0000", 2024-09-13T11:27:23.839 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.41307074799999999 2024-09-13T11:27:23.839 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.839 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.839 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:27:23.839 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:59.382384+0000", 2024-09-13T11:27:23.839 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.8202999999999998e-05 2024-09-13T11:27:23.840 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.840 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.840 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:27:23.840 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:59.382515+0000", 2024-09-13T11:27:23.840 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00013096599999999999 2024-09-13T11:27:23.840 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.840 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.840 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:27:23.840 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:59.382530+0000", 2024-09-13T11:27:23.840 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4797000000000001e-05 2024-09-13T11:27:23.840 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.840 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.840 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:27:23.840 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:59.382659+0000", 2024-09-13T11:27:23.841 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000129682 2024-09-13T11:27:23.841 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:23.841 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:27:23.841 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:23.841 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.841 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.841 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGRecoveryDelete(3.33 e242,227 3:cf7fdc1d:::benchmark_data_smithi177_31116_object66087:head,183'7846)", 2024-09-13T11:27:23.841 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:27:12.101840+0000", 2024-09-13T11:27:23.841 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 11.660859267999999, 2024-09-13T11:27:23.841 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.47295727399999998, 2024-09-13T11:27:23.841 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:27:23.841 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:27:23.841 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:27:23.842 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:27:23.842 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.842 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:27:23.842 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:27:12.101840+0000", 2024-09-13T11:27:23.842 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:23.842 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.842 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.842 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:27:23.842 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:27:12.101840+0000", 2024-09-13T11:27:23.842 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:23.842 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.842 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.842 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:27:23.843 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:27:12.101843+0000", 2024-09-13T11:27:23.843 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.3039999999999999e-06 2024-09-13T11:27:23.843 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.843 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.843 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:27:23.843 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:27:12.101855+0000", 2024-09-13T11:27:23.843 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2330000000000001e-05 2024-09-13T11:27:23.843 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.843 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.843 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:27:23.844 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:27:12.574539+0000", 2024-09-13T11:27:23.844 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.47268410199999999 2024-09-13T11:27:23.844 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.844 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.844 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:27:23.844 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:27:12.574557+0000", 2024-09-13T11:27:23.844 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.7682000000000001e-05 2024-09-13T11:27:23.844 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.844 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.844 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:27:23.844 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:27:12.574654+0000", 2024-09-13T11:27:23.844 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.7104999999999999e-05 2024-09-13T11:27:23.845 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.845 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.845 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:27:23.845 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:27:12.574672+0000", 2024-09-13T11:27:23.845 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.7677e-05 2024-09-13T11:27:23.845 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.845 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.845 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:27:23.845 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:27:12.574798+0000", 2024-09-13T11:27:23.845 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000126074 2024-09-13T11:27:23.845 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:23.845 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:27:23.845 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:23.845 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.846 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.846 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGRecoveryDelete(3.40 e245,238 3:024098ef:::benchmark_data_smithi177_31116_object117632:head,215'6224)", 2024-09-13T11:27:23.846 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:27:16.683144+0000", 2024-09-13T11:27:23.846 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 7.0795557770000004, 2024-09-13T11:27:23.846 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.30693769799999998, 2024-09-13T11:27:23.846 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:27:23.846 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:27:23.846 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:27:23.846 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:27:23.846 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.846 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:27:23.846 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:27:16.683144+0000", 2024-09-13T11:27:23.846 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:23.846 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.847 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.847 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:27:23.847 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:27:16.683144+0000", 2024-09-13T11:27:23.847 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:23.847 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.847 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.847 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:27:23.847 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:27:16.683146+0000", 2024-09-13T11:27:23.847 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.903e-06 2024-09-13T11:27:23.848 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.848 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.848 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:27:23.848 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:27:16.683152+0000", 2024-09-13T11:27:23.848 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.1290000000000001e-06 2024-09-13T11:27:23.848 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.848 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.848 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:27:23.848 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:27:16.989921+0000", 2024-09-13T11:27:23.848 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.30676933000000001 2024-09-13T11:27:23.848 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.848 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.848 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:27:23.848 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:27:16.989931+0000", 2024-09-13T11:27:23.849 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0138e-05 2024-09-13T11:27:23.849 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.849 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.849 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:27:23.849 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:27:16.990000+0000", 2024-09-13T11:27:23.849 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.8347000000000001e-05 2024-09-13T11:27:23.849 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.849 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.849 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:27:23.849 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:27:16.990010+0000", 2024-09-13T11:27:23.849 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.9890000000000004e-06 2024-09-13T11:27:23.849 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.849 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.850 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:27:23.850 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:27:16.990082+0000", 2024-09-13T11:27:23.850 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.1861999999999997e-05 2024-09-13T11:27:23.850 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:23.850 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:27:23.850 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:23.850 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.850 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.850 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGRecoveryDelete(3.40 e246,238 3:03994b16:::benchmark_data_smithi177_31116_object122064:head,221'6295)", 2024-09-13T11:27:23.850 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:27:17.355362+0000", 2024-09-13T11:27:23.850 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 6.407337804, 2024-09-13T11:27:23.850 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.32633605799999998, 2024-09-13T11:27:23.850 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:27:23.851 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:27:23.851 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:27:23.851 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:27:23.851 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.851 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:27:23.851 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:27:17.355362+0000", 2024-09-13T11:27:23.851 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:23.851 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.851 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.851 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:27:23.851 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:27:17.355362+0000", 2024-09-13T11:27:23.851 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:23.851 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.851 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.852 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:27:23.852 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:27:17.355363+0000", 2024-09-13T11:27:23.852 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.209e-06 2024-09-13T11:27:23.852 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.852 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.852 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:27:23.852 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:27:17.355370+0000", 2024-09-13T11:27:23.852 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.6490000000000002e-06 2024-09-13T11:27:23.852 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.852 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.852 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:27:23.852 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:27:17.681544+0000", 2024-09-13T11:27:23.852 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.32617453499999999 2024-09-13T11:27:23.853 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.853 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.853 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:27:23.853 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:27:17.681554+0000", 2024-09-13T11:27:23.853 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.6709999999999994e-06 2024-09-13T11:27:23.853 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.853 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.853 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:27:23.853 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:27:17.681635+0000", 2024-09-13T11:27:23.853 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.1105000000000004e-05 2024-09-13T11:27:23.853 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.853 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.853 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:27:23.854 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:27:17.681645+0000", 2024-09-13T11:27:23.854 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.9520000000000006e-06 2024-09-13T11:27:23.854 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.854 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.854 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:27:23.854 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:27:17.681698+0000", 2024-09-13T11:27:23.854 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.2936999999999998e-05 2024-09-13T11:27:23.854 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:23.854 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:27:23.854 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:23.854 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.854 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.854 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGRecoveryDelete(3.40 e246,238 3:02631d46:::benchmark_data_smithi177_31116_object122078:head,221'6296)", 2024-09-13T11:27:23.855 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:27:17.686169+0000", 2024-09-13T11:27:23.855 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 6.0765308420000004, 2024-09-13T11:27:23.855 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.28719174600000003, 2024-09-13T11:27:23.855 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:27:23.855 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:27:23.855 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:27:23.855 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:27:23.855 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.855 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:27:23.855 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:27:17.686169+0000", 2024-09-13T11:27:23.855 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:23.855 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.855 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.855 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:27:23.856 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:27:17.686169+0000", 2024-09-13T11:27:23.856 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:23.856 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.856 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.856 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:27:23.856 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:27:17.686170+0000", 2024-09-13T11:27:23.856 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.8899999999999998e-07 2024-09-13T11:27:23.856 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.856 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.856 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:27:23.856 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:27:17.686174+0000", 2024-09-13T11:27:23.857 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.8550000000000004e-06 2024-09-13T11:27:23.857 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.857 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.857 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:27:23.857 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:27:17.973191+0000", 2024-09-13T11:27:23.857 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.287017305 2024-09-13T11:27:23.857 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.857 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.857 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:27:23.857 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:27:17.973202+0000", 2024-09-13T11:27:23.857 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1464000000000001e-05 2024-09-13T11:27:23.857 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.857 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.857 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:27:23.858 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:27:17.973285+0000", 2024-09-13T11:27:23.858 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.2435000000000006e-05 2024-09-13T11:27:23.858 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.858 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.858 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:27:23.858 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:27:17.973293+0000", 2024-09-13T11:27:23.858 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.8809999999999996e-06 2024-09-13T11:27:23.858 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.858 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.858 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:27:23.858 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:27:17.973361+0000", 2024-09-13T11:27:23.858 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.7916999999999994e-05 2024-09-13T11:27:23.858 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:23.858 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:27:23.858 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:23.859 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.859 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.859 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGRecoveryDelete(3.2a e251,243 3:55e7aaec:::benchmark_data_smithi177_31116_object44237:head,171'6366)", 2024-09-13T11:27:23.859 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:27:22.497370+0000", 2024-09-13T11:27:23.859 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 1.2653297809999999, 2024-09-13T11:27:23.859 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.89375647599999997, 2024-09-13T11:27:23.859 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:27:23.859 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:27:23.859 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "queued for pg", 2024-09-13T11:27:23.859 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:27:23.859 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.859 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:27:23.860 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:27:22.497370+0000", 2024-09-13T11:27:23.860 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:23.860 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.860 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.860 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:27:23.860 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:27:22.497370+0000", 2024-09-13T11:27:23.860 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:23.860 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.860 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.860 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:27:23.860 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:27:22.497372+0000", 2024-09-13T11:27:23.861 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.0660000000000002e-06 2024-09-13T11:27:23.861 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.861 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.861 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:27:23.861 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:27:22.497379+0000", 2024-09-13T11:27:23.861 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.2969999999999998e-06 2024-09-13T11:27:23.861 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.861 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.861 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:27:23.861 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:27:23.390236+0000", 2024-09-13T11:27:23.861 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.89285665700000005 2024-09-13T11:27:23.861 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.862 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.862 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:27:23.862 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:27:23.390251+0000", 2024-09-13T11:27:23.862 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4868000000000001e-05 2024-09-13T11:27:23.862 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:23.862 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:23.862 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:27:23.862 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:27:23.391126+0000", 2024-09-13T11:27:23.862 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00087558799999999995 2024-09-13T11:27:23.862 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:23.862 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:27:23.863 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:23.863 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:23.863 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:27:23.863 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-13T11:27:23.864 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-13T11:27:23.995 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-09-13T11:27:24.044 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [], 2024-09-13T11:27:24.045 INFO:teuthology.orchestra.run.smithi177.stdout: "num_ops": 0 2024-09-13T11:27:24.045 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-09-13T11:27:24.045 INFO:teuthology.orchestra.run.smithi001.stderr:noscrub is unset 2024-09-13T11:27:24.046 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-13T11:27:24.050 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-13T11:27:24.177 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-09-13T11:27:24.177 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [], 2024-09-13T11:27:24.177 INFO:teuthology.orchestra.run.smithi177.stdout: "complaint_time": 30, 2024-09-13T11:27:24.177 INFO:teuthology.orchestra.run.smithi177.stdout: "num_blocked_ops": 0 2024-09-13T11:27:24.178 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-09-13T11:27:24.195 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-13T11:27:24.321 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-09-13T11:27:24.321 INFO:teuthology.orchestra.run.smithi177.stdout: "size": 20, 2024-09-13T11:27:24.321 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 600, 2024-09-13T11:27:24.321 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [ 2024-09-13T11:27:24.321 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.321 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPush(3.17 75/70 [PushOp(3:eda94455:::benchmark_data_smithi177_31116_object13017:head, version: 50'788, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:eda94455:::benchmark_data_smithi177_31116_object13017:head@50'788, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:27:24.321 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:31.038029+0000", 2024-09-13T11:27:24.321 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 233.28185387299999, 2024-09-13T11:27:24.321 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.42064623499999998, 2024-09-13T11:27:24.322 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:27:24.322 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:27:24.322 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:27:24.322 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:27:24.322 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.322 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:27:24.322 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.038029+0000", 2024-09-13T11:27:24.322 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:24.322 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.322 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.322 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:27:24.322 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.038029+0000", 2024-09-13T11:27:24.322 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:24.322 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.323 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.323 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:27:24.323 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.038030+0000", 2024-09-13T11:27:24.323 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.499e-06 2024-09-13T11:27:24.323 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.323 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.323 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:27:24.323 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.038088+0000", 2024-09-13T11:27:24.323 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.7771e-05 2024-09-13T11:27:24.323 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.323 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.323 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:27:24.323 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.457025+0000", 2024-09-13T11:27:24.323 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.41893754900000002 2024-09-13T11:27:24.324 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.324 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.324 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:27:24.324 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.457073+0000", 2024-09-13T11:27:24.324 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.7450999999999997e-05 2024-09-13T11:27:24.324 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.324 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.324 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:27:24.324 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.457304+0000", 2024-09-13T11:27:24.324 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000231351 2024-09-13T11:27:24.324 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.324 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.324 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:27:24.324 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.457331+0000", 2024-09-13T11:27:24.325 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.6372000000000001e-05 2024-09-13T11:27:24.325 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.325 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.325 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:27:24.325 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.458675+0000", 2024-09-13T11:27:24.325 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001344242 2024-09-13T11:27:24.325 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:24.325 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:27:24.325 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:24.325 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.325 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.325 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPushReply(3.1a 84/78 [PushReplyOp(3:581ea97e:::benchmark_data_smithi177_31116_object10255:head)])", 2024-09-13T11:27:24.325 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:41.514228+0000", 2024-09-13T11:27:24.325 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 222.80565398100001, 2024-09-13T11:27:24.326 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.40695564200000001, 2024-09-13T11:27:24.326 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:27:24.326 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:27:24.326 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-13T11:27:24.326 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:27:24.326 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.326 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:27:24.326 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.514228+0000", 2024-09-13T11:27:24.326 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:24.326 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.326 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.326 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:27:24.326 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.514228+0000", 2024-09-13T11:27:24.326 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:24.327 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.327 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.327 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:27:24.327 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.514230+0000", 2024-09-13T11:27:24.327 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.499e-06 2024-09-13T11:27:24.327 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.327 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.327 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:27:24.327 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.514236+0000", 2024-09-13T11:27:24.327 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.5389999999999998e-06 2024-09-13T11:27:24.327 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.327 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.327 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:27:24.327 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.919787+0000", 2024-09-13T11:27:24.328 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.40555152999999999 2024-09-13T11:27:24.328 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.328 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.328 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:27:24.328 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.919796+0000", 2024-09-13T11:27:24.328 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.4829999999999999e-06 2024-09-13T11:27:24.328 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.328 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.328 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:27:24.328 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.920366+0000", 2024-09-13T11:27:24.328 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00057061700000000004 2024-09-13T11:27:24.328 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.328 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.328 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:27:24.329 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.921184+0000", 2024-09-13T11:27:24.329 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00081797399999999996 2024-09-13T11:27:24.329 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:24.329 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:27:24.329 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:24.329 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.329 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.329 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:60427 3.14 3:2afea229:::benchmark_data_smithi177_31116_object60426:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e85)", 2024-09-13T11:27:24.329 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:42.187231+0000", 2024-09-13T11:27:24.329 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 222.13265111199999, 2024-09-13T11:27:24.329 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.83900596999999999, 2024-09-13T11:27:24.329 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:27:24.329 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:27:24.329 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:27:24.330 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:27:24.330 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:27:24.330 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:27:24.330 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 60427 2024-09-13T11:27:24.330 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.330 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:27:24.330 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.330 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:27:24.330 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.187231+0000", 2024-09-13T11:27:24.330 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:24.330 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.330 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.330 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:27:24.330 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.187231+0000", 2024-09-13T11:27:24.330 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:24.331 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.331 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.331 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:27:24.331 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.187235+0000", 2024-09-13T11:27:24.331 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.3579999999999999e-06 2024-09-13T11:27:24.331 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.331 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.331 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:27:24.331 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.187261+0000", 2024-09-13T11:27:24.331 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.6089e-05 2024-09-13T11:27:24.331 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.331 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.331 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:27:24.331 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.187275+0000", 2024-09-13T11:27:24.332 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.4653e-05 2024-09-13T11:27:24.332 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.332 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.332 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:27:24.332 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.187282+0000", 2024-09-13T11:27:24.332 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.8870000000000003e-06 2024-09-13T11:27:24.332 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.332 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.332 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:27:24.332 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.187336+0000", 2024-09-13T11:27:24.332 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.3875999999999997e-05 2024-09-13T11:27:24.332 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.332 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.332 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:27:24.333 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.187480+0000", 2024-09-13T11:27:24.333 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00014330899999999999 2024-09-13T11:27:24.333 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.333 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.333 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 14", 2024-09-13T11:27:24.333 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.187658+0000", 2024-09-13T11:27:24.333 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00017861900000000001 2024-09-13T11:27:24.333 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.333 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.333 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:27:24.333 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.189353+0000", 2024-09-13T11:27:24.333 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001695144 2024-09-13T11:27:24.333 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.333 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.333 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:27:24.334 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:43.026081+0000", 2024-09-13T11:27:24.334 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.83672736199999997 2024-09-13T11:27:24.334 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.334 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.334 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:27:24.334 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:43.026169+0000", 2024-09-13T11:27:24.334 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.8411000000000004e-05 2024-09-13T11:27:24.334 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.334 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.334 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:27:24.334 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:43.026237+0000", 2024-09-13T11:27:24.334 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.8262000000000005e-05 2024-09-13T11:27:24.334 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:24.335 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:27:24.335 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:24.335 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.335 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.335 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop_reply(client.4665.0:60427 3.14 e85/84 ondisk, result = 0)", 2024-09-13T11:27:24.335 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:42.190914+0000", 2024-09-13T11:27:24.335 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 222.12896844799999, 2024-09-13T11:27:24.335 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.83534071499999996, 2024-09-13T11:27:24.335 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:27:24.335 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:27:24.335 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:27:24.335 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:27:24.335 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.335 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:27:24.335 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.190914+0000", 2024-09-13T11:27:24.336 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:24.336 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.336 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.336 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:27:24.336 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.190914+0000", 2024-09-13T11:27:24.336 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:24.336 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.336 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.336 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:27:24.336 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.190916+0000", 2024-09-13T11:27:24.336 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.84e-06 2024-09-13T11:27:24.336 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.336 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.336 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:27:24.337 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.190921+0000", 2024-09-13T11:27:24.337 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.2039999999999996e-06 2024-09-13T11:27:24.337 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.337 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.337 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:27:24.337 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:43.025926+0000", 2024-09-13T11:27:24.337 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.83500542 2024-09-13T11:27:24.337 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.337 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.337 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:27:24.337 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:43.025937+0000", 2024-09-13T11:27:24.337 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0597e-05 2024-09-13T11:27:24.337 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.337 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.338 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:27:24.338 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:43.026052+0000", 2024-09-13T11:27:24.338 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000114838 2024-09-13T11:27:24.338 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.338 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.338 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:27:24.338 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:43.026070+0000", 2024-09-13T11:27:24.338 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.8131e-05 2024-09-13T11:27:24.338 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.338 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.338 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:27:24.338 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:43.026255+0000", 2024-09-13T11:27:24.338 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000184685 2024-09-13T11:27:24.338 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:24.339 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:27:24.339 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:24.339 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.339 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.339 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "rep_scrubmap(3.14 e84 from shard 14)", 2024-09-13T11:27:24.339 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:48.313615+0000", 2024-09-13T11:27:24.339 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 216.00626747199999, 2024-09-13T11:27:24.339 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.35633836099999999, 2024-09-13T11:27:24.339 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:27:24.339 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:27:24.339 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:27:24.339 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:27:24.339 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.339 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:27:24.339 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.313615+0000", 2024-09-13T11:27:24.340 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:24.340 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.340 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.340 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:27:24.340 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.313615+0000", 2024-09-13T11:27:24.340 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:24.340 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.340 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.340 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:27:24.340 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.313617+0000", 2024-09-13T11:27:24.340 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.7990000000000001e-06 2024-09-13T11:27:24.340 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.340 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.341 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:27:24.341 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.313688+0000", 2024-09-13T11:27:24.341 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.1481000000000002e-05 2024-09-13T11:27:24.341 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.341 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.341 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:27:24.341 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.669785+0000", 2024-09-13T11:27:24.341 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.356096466 2024-09-13T11:27:24.341 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.341 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.341 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:27:24.341 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.669792+0000", 2024-09-13T11:27:24.341 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.1740000000000003e-06 2024-09-13T11:27:24.341 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.342 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.342 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:27:24.342 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.669839+0000", 2024-09-13T11:27:24.342 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.7323000000000001e-05 2024-09-13T11:27:24.342 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.342 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.342 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:27:24.342 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.669847+0000", 2024-09-13T11:27:24.342 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.0469999999999994e-06 2024-09-13T11:27:24.342 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.342 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.342 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:27:24.342 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.669953+0000", 2024-09-13T11:27:24.342 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00010607099999999999 2024-09-13T11:27:24.342 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:24.343 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:27:24.343 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:24.343 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.343 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.343 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPush(3.15 143/122 [PushOp(3:ae52bb45:::benchmark_data_smithi177_31116_object61893:head, version: 87'3272, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:ae52bb45:::benchmark_data_smithi177_31116_object61893:head@87'3272, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:27:24.343 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:57.633939+0000", 2024-09-13T11:27:24.343 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 146.68594333499999, 2024-09-13T11:27:24.343 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.339405067, 2024-09-13T11:27:24.343 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:27:24.343 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:27:24.343 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:27:24.343 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:27:24.343 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.343 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:27:24.344 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:57.633939+0000", 2024-09-13T11:27:24.344 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:24.344 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.344 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.344 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:27:24.344 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:57.633939+0000", 2024-09-13T11:27:24.344 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:24.344 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.344 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.344 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:27:24.344 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:57.633941+0000", 2024-09-13T11:27:24.345 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.911e-06 2024-09-13T11:27:24.345 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.345 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.345 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:27:24.345 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:57.633994+0000", 2024-09-13T11:27:24.345 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.2753999999999997e-05 2024-09-13T11:27:24.345 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.345 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.345 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:27:24.345 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:57.968403+0000", 2024-09-13T11:27:24.345 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.334409336 2024-09-13T11:27:24.345 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.346 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.346 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:27:24.346 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:57.968451+0000", 2024-09-13T11:27:24.346 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.7525000000000003e-05 2024-09-13T11:27:24.346 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.346 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.346 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:27:24.346 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:57.968660+0000", 2024-09-13T11:27:24.346 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000209204 2024-09-13T11:27:24.346 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.346 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.346 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:27:24.346 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:57.968680+0000", 2024-09-13T11:27:24.346 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.9639999999999998e-05 2024-09-13T11:27:24.347 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.347 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.347 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:27:24.347 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:57.973344+0000", 2024-09-13T11:27:24.347 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0046646969999999998 2024-09-13T11:27:24.347 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:24.347 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:27:24.347 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:24.347 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.347 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.347 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:221537 3.14 3.d5a7b194 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e191)", 2024-09-13T11:27:24.347 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:26:04.320591+0000", 2024-09-13T11:27:24.347 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 79.999291313000001, 2024-09-13T11:27:24.347 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.72342120499999996, 2024-09-13T11:27:24.348 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:27:24.348 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:27:24.348 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-13T11:27:24.348 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:27:24.348 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:27:24.348 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:27:24.348 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 221537 2024-09-13T11:27:24.348 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.348 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:27:24.348 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.348 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:27:24.348 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:04.320591+0000", 2024-09-13T11:27:24.348 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:24.348 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.349 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.349 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:27:24.349 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:04.320591+0000", 2024-09-13T11:27:24.349 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:24.349 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.349 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.349 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:27:24.349 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:04.320595+0000", 2024-09-13T11:27:24.349 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.6500000000000002e-06 2024-09-13T11:27:24.349 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.349 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.349 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:27:24.349 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:04.320604+0000", 2024-09-13T11:27:24.350 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.5400000000000001e-06 2024-09-13T11:27:24.350 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.350 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.350 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:27:24.350 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:04.320630+0000", 2024-09-13T11:27:24.350 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.5340000000000001e-05 2024-09-13T11:27:24.350 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.350 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.350 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:27:24.350 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:04.320644+0000", 2024-09-13T11:27:24.350 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.4008e-05 2024-09-13T11:27:24.350 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.350 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.350 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:27:24.351 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:04.320708+0000", 2024-09-13T11:27:24.351 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.4268000000000002e-05 2024-09-13T11:27:24.351 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.351 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.351 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for peered", 2024-09-13T11:27:24.351 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:04.320716+0000", 2024-09-13T11:27:24.351 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.2139999999999996e-06 2024-09-13T11:27:24.351 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.351 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.351 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:27:24.351 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:05.043955+0000", 2024-09-13T11:27:24.351 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.72323873599999999 2024-09-13T11:27:24.351 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.351 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.351 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:27:24.352 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:05.044012+0000", 2024-09-13T11:27:24.352 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.7448999999999998e-05 2024-09-13T11:27:24.352 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:24.352 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:27:24.352 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:24.352 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.352 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.352 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:266700 3.34 3:2d55c559:::benchmark_data_smithi177_31116_object126397:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e224)", 2024-09-13T11:27:24.352 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:26:48.996871+0000", 2024-09-13T11:27:24.352 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 35.323011100000002, 2024-09-13T11:27:24.352 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0522031489999999, 2024-09-13T11:27:24.352 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:27:24.352 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:27:24.353 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:27:24.353 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:27:24.353 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:27:24.353 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:27:24.353 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 266700 2024-09-13T11:27:24.353 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.353 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:27:24.353 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.353 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:27:24.353 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:48.996871+0000", 2024-09-13T11:27:24.353 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:24.353 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.353 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.353 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:27:24.353 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:48.996871+0000", 2024-09-13T11:27:24.354 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:24.354 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.354 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.354 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:27:24.354 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:48.996872+0000", 2024-09-13T11:27:24.354 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.9100000000000003e-07 2024-09-13T11:27:24.354 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.354 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.354 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:27:24.354 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:48.996875+0000", 2024-09-13T11:27:24.354 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.1470000000000001e-06 2024-09-13T11:27:24.354 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.354 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.354 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:27:24.355 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:48.996882+0000", 2024-09-13T11:27:24.355 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.2019999999999998e-06 2024-09-13T11:27:24.355 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.355 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.355 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:27:24.355 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:48.996888+0000", 2024-09-13T11:27:24.355 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.0549999999999996e-06 2024-09-13T11:27:24.355 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.355 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.355 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:27:24.355 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:49.228485+0000", 2024-09-13T11:27:24.355 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.231596424 2024-09-13T11:27:24.355 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.355 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.356 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for peered", 2024-09-13T11:27:24.356 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:49.228494+0000", 2024-09-13T11:27:24.356 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.2450000000000004e-06 2024-09-13T11:27:24.356 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.356 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.356 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:27:24.356 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:50.042660+0000", 2024-09-13T11:27:24.356 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.81416628099999999 2024-09-13T11:27:24.356 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.356 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.356 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:27:24.356 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:50.042810+0000", 2024-09-13T11:27:24.356 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00014997900000000001 2024-09-13T11:27:24.356 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.357 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.357 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 14", 2024-09-13T11:27:24.357 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:50.042937+0000", 2024-09-13T11:27:24.357 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000127019 2024-09-13T11:27:24.357 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.357 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.357 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:27:24.357 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:50.045264+0000", 2024-09-13T11:27:24.357 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0023268609999999999 2024-09-13T11:27:24.357 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.357 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.357 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:27:24.357 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:50.048977+0000", 2024-09-13T11:27:24.357 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0037126960000000001 2024-09-13T11:27:24.357 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.358 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.358 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:27:24.358 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:50.049033+0000", 2024-09-13T11:27:24.358 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.6458000000000001e-05 2024-09-13T11:27:24.358 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.358 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.358 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:27:24.358 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:50.049075+0000", 2024-09-13T11:27:24.358 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.1090999999999997e-05 2024-09-13T11:27:24.358 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:24.358 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:27:24.358 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:24.358 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.358 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.359 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:266735 3.34 3:2f49a883:::benchmark_data_smithi177_31116_object126432:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e224)", 2024-09-13T11:27:24.359 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:26:48.996930+0000", 2024-09-13T11:27:24.359 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 35.322952495999999, 2024-09-13T11:27:24.359 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0551774199999999, 2024-09-13T11:27:24.359 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:27:24.359 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:27:24.359 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:27:24.359 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:27:24.359 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:27:24.359 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:27:24.359 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 266735 2024-09-13T11:27:24.359 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.359 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:27:24.359 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.360 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:27:24.360 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:48.996930+0000", 2024-09-13T11:27:24.360 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:24.360 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.360 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.360 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:27:24.360 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:48.996930+0000", 2024-09-13T11:27:24.360 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:24.360 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.360 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.360 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:27:24.360 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:48.996931+0000", 2024-09-13T11:27:24.360 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.569999999999999e-07 2024-09-13T11:27:24.360 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.361 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.361 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:27:24.361 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:48.996934+0000", 2024-09-13T11:27:24.361 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.8399999999999999e-06 2024-09-13T11:27:24.361 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.361 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.361 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:27:24.361 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:48.996944+0000", 2024-09-13T11:27:24.361 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.026e-05 2024-09-13T11:27:24.361 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.361 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.361 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:27:24.361 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:48.996951+0000", 2024-09-13T11:27:24.362 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.5470000000000002e-06 2024-09-13T11:27:24.362 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.362 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.362 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:27:24.362 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:49.228514+0000", 2024-09-13T11:27:24.362 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.231563614 2024-09-13T11:27:24.362 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.362 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.362 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for peered", 2024-09-13T11:27:24.362 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:49.228520+0000", 2024-09-13T11:27:24.362 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.7309999999999998e-06 2024-09-13T11:27:24.362 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.362 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.363 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:27:24.363 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:50.045685+0000", 2024-09-13T11:27:24.363 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.817165366 2024-09-13T11:27:24.363 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.363 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.363 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:27:24.363 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:50.045830+0000", 2024-09-13T11:27:24.363 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00014434499999999999 2024-09-13T11:27:24.363 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.363 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.363 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 14", 2024-09-13T11:27:24.363 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:50.045892+0000", 2024-09-13T11:27:24.363 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.2169999999999996e-05 2024-09-13T11:27:24.363 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.364 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.364 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:27:24.364 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:50.049086+0000", 2024-09-13T11:27:24.364 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0031937929999999999 2024-09-13T11:27:24.364 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.364 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.364 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:27:24.364 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:50.052007+0000", 2024-09-13T11:27:24.364 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0029213160000000002 2024-09-13T11:27:24.364 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.364 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.364 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:27:24.364 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:50.052057+0000", 2024-09-13T11:27:24.364 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.9778999999999999e-05 2024-09-13T11:27:24.365 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.365 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.365 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:27:24.365 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:50.052107+0000", 2024-09-13T11:27:24.365 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.0701999999999997e-05 2024-09-13T11:27:24.365 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:24.365 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:27:24.365 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:24.365 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.365 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.365 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:266744 3.34 3:2d363f39:::benchmark_data_smithi177_31116_object126441:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e224)", 2024-09-13T11:27:24.365 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:26:48.996957+0000", 2024-09-13T11:27:24.365 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 35.322925062000003, 2024-09-13T11:27:24.366 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0552655930000001, 2024-09-13T11:27:24.366 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:27:24.366 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:27:24.366 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:27:24.366 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:27:24.366 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:27:24.366 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:27:24.366 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 266744 2024-09-13T11:27:24.366 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.366 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:27:24.366 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.366 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:27:24.366 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:48.996957+0000", 2024-09-13T11:27:24.366 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:24.367 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.367 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.367 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:27:24.367 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:48.996957+0000", 2024-09-13T11:27:24.367 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:24.367 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.367 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.367 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:27:24.367 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:48.996958+0000", 2024-09-13T11:27:24.367 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.5199999999999996e-07 2024-09-13T11:27:24.367 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.367 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.367 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:27:24.367 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:48.996961+0000", 2024-09-13T11:27:24.368 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.886e-06 2024-09-13T11:27:24.368 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.368 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.368 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:27:24.368 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:48.996972+0000", 2024-09-13T11:27:24.368 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.1201e-05 2024-09-13T11:27:24.368 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.368 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.368 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:27:24.368 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:48.996978+0000", 2024-09-13T11:27:24.368 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.9920000000000004e-06 2024-09-13T11:27:24.368 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.368 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.368 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:27:24.369 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:49.228539+0000", 2024-09-13T11:27:24.369 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.23156088599999999 2024-09-13T11:27:24.369 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.369 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.369 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for peered", 2024-09-13T11:27:24.369 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:49.228544+0000", 2024-09-13T11:27:24.369 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.1290000000000004e-06 2024-09-13T11:27:24.369 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.369 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.369 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:27:24.369 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:50.047158+0000", 2024-09-13T11:27:24.369 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.818614273 2024-09-13T11:27:24.369 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.369 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.370 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:27:24.370 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:50.047281+0000", 2024-09-13T11:27:24.370 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00012297799999999999 2024-09-13T11:27:24.370 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.370 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.370 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 14", 2024-09-13T11:27:24.370 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:50.047384+0000", 2024-09-13T11:27:24.370 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000103089 2024-09-13T11:27:24.370 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.370 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.370 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:27:24.370 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:50.050763+0000", 2024-09-13T11:27:24.370 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.003378388 2024-09-13T11:27:24.371 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.371 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.371 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:27:24.371 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:50.052143+0000", 2024-09-13T11:27:24.371 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001380575 2024-09-13T11:27:24.371 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.371 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.371 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:27:24.371 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:50.052190+0000", 2024-09-13T11:27:24.371 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.6771000000000003e-05 2024-09-13T11:27:24.371 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.371 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.371 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:27:24.371 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:50.052223+0000", 2024-09-13T11:27:24.372 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.2772999999999999e-05 2024-09-13T11:27:24.372 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:24.372 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:27:24.372 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:24.372 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.372 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.372 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:266872 3.34 3:2d48bf1b:::benchmark_data_smithi177_31116_object126569:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e224)", 2024-09-13T11:27:24.372 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:26:48.997053+0000", 2024-09-13T11:27:24.372 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 35.322829345000002, 2024-09-13T11:27:24.372 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.058285779, 2024-09-13T11:27:24.372 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:27:24.372 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:27:24.372 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:27:24.372 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:27:24.373 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:27:24.373 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:27:24.373 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 266872 2024-09-13T11:27:24.373 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.373 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:27:24.373 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.373 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:27:24.373 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:48.997053+0000", 2024-09-13T11:27:24.373 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:24.373 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.373 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.373 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:27:24.373 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:48.997053+0000", 2024-09-13T11:27:24.373 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:24.373 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.374 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.374 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:27:24.374 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:48.997054+0000", 2024-09-13T11:27:24.374 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.8700000000000005e-07 2024-09-13T11:27:24.374 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.374 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.374 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:27:24.374 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:48.997057+0000", 2024-09-13T11:27:24.374 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.9160000000000001e-06 2024-09-13T11:27:24.374 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.374 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.374 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:27:24.374 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:48.997068+0000", 2024-09-13T11:27:24.374 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.1243999999999999e-05 2024-09-13T11:27:24.375 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.375 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.375 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:27:24.375 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:48.997075+0000", 2024-09-13T11:27:24.375 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.511e-06 2024-09-13T11:27:24.375 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.375 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.375 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:27:24.375 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:49.228564+0000", 2024-09-13T11:27:24.375 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.23148927799999999 2024-09-13T11:27:24.375 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.375 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.375 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for peered", 2024-09-13T11:27:24.376 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:49.228569+0000", 2024-09-13T11:27:24.376 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.0919999999999997e-06 2024-09-13T11:27:24.376 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.376 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.376 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:27:24.376 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:50.049116+0000", 2024-09-13T11:27:24.376 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.82054686200000004 2024-09-13T11:27:24.376 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.376 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.376 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:27:24.376 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:50.049254+0000", 2024-09-13T11:27:24.377 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000138085 2024-09-13T11:27:24.377 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.377 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.377 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 14", 2024-09-13T11:27:24.377 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:50.049356+0000", 2024-09-13T11:27:24.377 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000102444 2024-09-13T11:27:24.377 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.377 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.377 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:27:24.377 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:50.052916+0000", 2024-09-13T11:27:24.377 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0035596849999999999 2024-09-13T11:27:24.377 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.377 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.377 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:27:24.378 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:50.055190+0000", 2024-09-13T11:27:24.378 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0022735289999999998 2024-09-13T11:27:24.378 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.378 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.378 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:27:24.378 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:50.055290+0000", 2024-09-13T11:27:24.378 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00010009900000000001 2024-09-13T11:27:24.378 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.378 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.378 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:27:24.378 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:50.055339+0000", 2024-09-13T11:27:24.378 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.9246999999999998e-05 2024-09-13T11:27:24.378 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:24.378 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:27:24.379 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:24.379 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.379 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.379 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:266877 3.34 3:2c38142a:::benchmark_data_smithi177_31116_object126574:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e224)", 2024-09-13T11:27:24.379 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:26:49.253226+0000", 2024-09-13T11:27:24.379 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 35.066656535, 2024-09-13T11:27:24.379 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.80224803300000003, 2024-09-13T11:27:24.379 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:27:24.379 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:27:24.379 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:27:24.379 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:27:24.379 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:27:24.379 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:27:24.379 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 266877 2024-09-13T11:27:24.380 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.380 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:27:24.380 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.380 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:27:24.380 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:49.253226+0000", 2024-09-13T11:27:24.380 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:24.380 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.380 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.380 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:27:24.380 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:49.253226+0000", 2024-09-13T11:27:24.380 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:24.380 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.380 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.381 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:27:24.381 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:49.253230+0000", 2024-09-13T11:27:24.381 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.9960000000000004e-06 2024-09-13T11:27:24.381 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.381 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.381 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:27:24.381 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:49.253239+0000", 2024-09-13T11:27:24.381 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.8969999999999997e-06 2024-09-13T11:27:24.381 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.381 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.381 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:27:24.381 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:49.253264+0000", 2024-09-13T11:27:24.381 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.5012999999999999e-05 2024-09-13T11:27:24.381 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.381 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.382 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:27:24.382 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:49.253276+0000", 2024-09-13T11:27:24.382 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.1932e-05 2024-09-13T11:27:24.382 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.382 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.382 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:27:24.382 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:49.253374+0000", 2024-09-13T11:27:24.382 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.8189000000000005e-05 2024-09-13T11:27:24.382 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.382 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.382 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for peered", 2024-09-13T11:27:24.382 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:49.253384+0000", 2024-09-13T11:27:24.382 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.9499999999999996e-06 2024-09-13T11:27:24.382 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.383 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.383 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:27:24.383 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:50.050795+0000", 2024-09-13T11:27:24.383 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.79741076099999997 2024-09-13T11:27:24.383 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.383 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.383 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:27:24.383 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:50.050907+0000", 2024-09-13T11:27:24.383 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000111832 2024-09-13T11:27:24.383 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.383 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.383 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 14", 2024-09-13T11:27:24.383 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:50.051021+0000", 2024-09-13T11:27:24.383 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000114435 2024-09-13T11:27:24.384 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.384 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.384 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:27:24.384 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:50.053293+0000", 2024-09-13T11:27:24.384 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0022717409999999999 2024-09-13T11:27:24.384 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.384 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.384 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:27:24.384 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:50.055353+0000", 2024-09-13T11:27:24.384 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.002060227 2024-09-13T11:27:24.384 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.384 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.384 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:27:24.384 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:50.055436+0000", 2024-09-13T11:27:24.385 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.3436000000000005e-05 2024-09-13T11:27:24.385 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.385 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.385 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:27:24.385 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:50.055474+0000", 2024-09-13T11:27:24.385 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.7623999999999998e-05 2024-09-13T11:27:24.385 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:24.385 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:27:24.385 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:24.385 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.385 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.385 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGRecoveryDeleteReply(3.37 e231,227 3:ec9a2027:::benchmark_data_smithi177_31116_object6072:head,148'5854)", 2024-09-13T11:27:24.385 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:26:59.120542+0000", 2024-09-13T11:27:24.385 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 25.199340501000002, 2024-09-13T11:27:24.386 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.33395871199999999, 2024-09-13T11:27:24.386 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:27:24.386 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:27:24.386 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-13T11:27:24.386 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:27:24.386 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.386 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:27:24.386 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:59.120542+0000", 2024-09-13T11:27:24.386 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:24.386 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.386 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.386 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:27:24.386 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:59.120542+0000", 2024-09-13T11:27:24.386 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:24.386 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.387 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.387 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:27:24.387 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:59.120543+0000", 2024-09-13T11:27:24.387 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.494e-06 2024-09-13T11:27:24.387 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.387 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.387 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:27:24.387 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:59.120549+0000", 2024-09-13T11:27:24.387 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.0249999999999999e-06 2024-09-13T11:27:24.387 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.387 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.387 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:27:24.387 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:59.453758+0000", 2024-09-13T11:27:24.387 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.33320845500000001 2024-09-13T11:27:24.388 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.388 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.388 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:27:24.388 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:59.453770+0000", 2024-09-13T11:27:24.388 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.1912e-05 2024-09-13T11:27:24.388 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.388 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.388 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:27:24.388 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:59.453905+0000", 2024-09-13T11:27:24.388 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00013504399999999999 2024-09-13T11:27:24.388 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.388 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.388 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:27:24.388 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:59.454501+0000", 2024-09-13T11:27:24.389 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000595782 2024-09-13T11:27:24.389 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:24.389 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:27:24.389 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:24.389 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.389 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.389 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:278044 3.d e237/117 3:b34c5ad6:::benchmark_data_smithi177_31116_object137741:head v 237'7756, mlcod=236'7753)", 2024-09-13T11:27:24.389 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:27:05.131513+0000", 2024-09-13T11:27:24.389 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 19.18836941, 2024-09-13T11:27:24.389 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.71971197600000003, 2024-09-13T11:27:24.389 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:27:24.389 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:27:24.389 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:27:24.389 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:27:24.390 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.390 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:27:24.390 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:27:05.131513+0000", 2024-09-13T11:27:24.390 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:24.390 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.390 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.390 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:27:24.390 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:27:05.131513+0000", 2024-09-13T11:27:24.390 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:24.390 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.390 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.390 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:27:24.390 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:27:05.131514+0000", 2024-09-13T11:27:24.390 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.0100000000000005e-07 2024-09-13T11:27:24.390 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.391 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.391 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:27:24.391 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:27:05.131520+0000", 2024-09-13T11:27:24.391 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.5200000000000003e-06 2024-09-13T11:27:24.391 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.391 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.391 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:27:24.391 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:27:05.848709+0000", 2024-09-13T11:27:24.391 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.717188783 2024-09-13T11:27:24.391 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.391 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.391 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:27:24.391 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:27:05.848715+0000", 2024-09-13T11:27:24.391 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.9819999999999996e-06 2024-09-13T11:27:24.392 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.392 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.392 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:27:24.392 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:27:05.848860+0000", 2024-09-13T11:27:24.392 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000144955 2024-09-13T11:27:24.392 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.392 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.392 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:27:24.392 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:27:05.848876+0000", 2024-09-13T11:27:24.392 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.6594000000000001e-05 2024-09-13T11:27:24.392 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.392 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.392 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:27:24.392 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:27:05.851187+0000", 2024-09-13T11:27:24.393 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.002310791 2024-09-13T11:27:24.393 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.393 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.393 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:27:24.393 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:27:05.851225+0000", 2024-09-13T11:27:24.393 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.7849999999999998e-05 2024-09-13T11:27:24.393 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:24.393 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:27:24.393 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:24.393 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.394 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.394 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGRecoveryDelete(3.31 e238,227 3:8fb2e1d6:::benchmark_data_smithi177_31116_object26864:head,160'6608)", 2024-09-13T11:27:24.394 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:27:06.086941+0000", 2024-09-13T11:27:24.394 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 18.232941628999999, 2024-09-13T11:27:24.394 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.35038828, 2024-09-13T11:27:24.394 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:27:24.394 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:27:24.394 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:27:24.394 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:27:24.394 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.394 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:27:24.394 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:27:06.086941+0000", 2024-09-13T11:27:24.394 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:24.394 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.395 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.395 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:27:24.395 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:27:06.086941+0000", 2024-09-13T11:27:24.395 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:24.395 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.395 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.395 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:27:24.395 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:27:06.086942+0000", 2024-09-13T11:27:24.395 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3319999999999999e-06 2024-09-13T11:27:24.395 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.395 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.395 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:27:24.395 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:27:06.086947+0000", 2024-09-13T11:27:24.395 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.237e-06 2024-09-13T11:27:24.396 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.396 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.396 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:27:24.396 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:27:06.437152+0000", 2024-09-13T11:27:24.396 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.35020466500000003 2024-09-13T11:27:24.396 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.396 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.396 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:27:24.396 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:27:06.437160+0000", 2024-09-13T11:27:24.396 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.678e-06 2024-09-13T11:27:24.396 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.396 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.396 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:27:24.397 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:27:06.437238+0000", 2024-09-13T11:27:24.397 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.8392000000000004e-05 2024-09-13T11:27:24.397 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.397 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.397 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:27:24.397 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:27:06.437246+0000", 2024-09-13T11:27:24.397 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.0800000000000006e-06 2024-09-13T11:27:24.397 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.397 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.397 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:27:24.397 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:27:06.437329+0000", 2024-09-13T11:27:24.397 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.2896000000000001e-05 2024-09-13T11:27:24.397 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:24.397 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:27:24.398 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:24.398 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.398 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.398 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGRecoveryDeleteReply(3.34 e241,240 3:2d319442:::benchmark_data_smithi177_31116_object33792:head,165'6841)", 2024-09-13T11:27:24.398 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:27:10.240612+0000", 2024-09-13T11:27:24.398 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 14.079270609, 2024-09-13T11:27:24.398 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.83602851499999997, 2024-09-13T11:27:24.398 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:27:24.398 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:27:24.398 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-13T11:27:24.398 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:27:24.398 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.398 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:27:24.398 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:27:10.240612+0000", 2024-09-13T11:27:24.399 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:24.399 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.399 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.399 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:27:24.399 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:27:10.240612+0000", 2024-09-13T11:27:24.399 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:24.399 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.399 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.399 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:27:24.399 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:27:10.240613+0000", 2024-09-13T11:27:24.399 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0759999999999999e-06 2024-09-13T11:27:24.399 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.399 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.399 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:27:24.400 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:27:10.240617+0000", 2024-09-13T11:27:24.400 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.0439999999999997e-06 2024-09-13T11:27:24.400 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.400 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.400 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:27:24.400 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:27:11.076132+0000", 2024-09-13T11:27:24.400 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.83551476499999999 2024-09-13T11:27:24.400 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.400 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.400 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:27:24.400 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:27:11.076142+0000", 2024-09-13T11:27:24.400 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.7739999999999999e-06 2024-09-13T11:27:24.400 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.400 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.401 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:27:24.401 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:27:11.076208+0000", 2024-09-13T11:27:24.401 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.6752000000000001e-05 2024-09-13T11:27:24.401 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.401 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.401 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:27:24.401 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:27:11.076640+0000", 2024-09-13T11:27:24.401 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000432104 2024-09-13T11:27:24.401 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:24.401 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:27:24.401 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:24.401 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.401 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.401 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGRecoveryDeleteReply(3.37 e241,227 3:edc9a6c9:::benchmark_data_smithi177_31116_object50649:head,175'7196)", 2024-09-13T11:27:24.402 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:27:10.278139+0000", 2024-09-13T11:27:24.402 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 14.041743831, 2024-09-13T11:27:24.402 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.40257930800000002, 2024-09-13T11:27:24.402 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:27:24.402 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:27:24.402 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-13T11:27:24.402 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:27:24.402 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.402 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:27:24.402 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:27:10.278139+0000", 2024-09-13T11:27:24.402 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:24.402 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.402 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.402 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:27:24.403 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:27:10.278139+0000", 2024-09-13T11:27:24.403 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:24.403 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.403 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.403 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:27:24.403 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:27:10.278140+0000", 2024-09-13T11:27:24.403 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.068e-06 2024-09-13T11:27:24.403 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.403 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.403 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:27:24.403 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:27:10.278144+0000", 2024-09-13T11:27:24.403 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.1139999999999999e-06 2024-09-13T11:27:24.403 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.403 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.404 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:27:24.404 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:27:10.680267+0000", 2024-09-13T11:27:24.404 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.40212351600000001 2024-09-13T11:27:24.404 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.404 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.404 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:27:24.404 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:27:10.680274+0000", 2024-09-13T11:27:24.404 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.7320000000000001e-06 2024-09-13T11:27:24.404 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.404 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.404 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:27:24.404 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:27:10.680349+0000", 2024-09-13T11:27:24.404 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.5056999999999996e-05 2024-09-13T11:27:24.404 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.405 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.405 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:27:24.405 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:27:10.680718+0000", 2024-09-13T11:27:24.405 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00036882100000000003 2024-09-13T11:27:24.405 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:24.405 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:27:24.405 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:24.405 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.405 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.405 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGRecoveryDelete(3.31 e242,227 3:8e255f15:::benchmark_data_smithi177_31116_object63203:head,182'7773)", 2024-09-13T11:27:24.405 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:27:12.439895+0000", 2024-09-13T11:27:24.405 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 11.879987716, 2024-09-13T11:27:24.405 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.68240246199999999, 2024-09-13T11:27:24.405 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:27:24.406 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:27:24.406 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:27:24.406 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:27:24.406 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.406 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:27:24.406 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:27:12.439895+0000", 2024-09-13T11:27:24.406 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:24.406 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.406 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.406 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:27:24.406 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:27:12.439895+0000", 2024-09-13T11:27:24.406 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:24.406 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.406 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.406 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:27:24.407 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:27:12.439896+0000", 2024-09-13T11:27:24.407 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.2589999999999999e-06 2024-09-13T11:27:24.407 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.407 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.407 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:27:24.407 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:27:12.439902+0000", 2024-09-13T11:27:24.407 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.6879999999999996e-06 2024-09-13T11:27:24.407 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.407 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.407 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:27:24.407 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:27:13.122063+0000", 2024-09-13T11:27:24.407 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.68216121100000005 2024-09-13T11:27:24.407 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.407 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.408 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:27:24.408 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:27:13.122079+0000", 2024-09-13T11:27:24.408 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.6155999999999999e-05 2024-09-13T11:27:24.408 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.408 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.408 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:27:24.408 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:27:13.122170+0000", 2024-09-13T11:27:24.408 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.1304999999999994e-05 2024-09-13T11:27:24.408 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.408 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.408 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:27:24.408 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:27:13.122181+0000", 2024-09-13T11:27:24.408 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0987e-05 2024-09-13T11:27:24.408 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.409 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.409 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:27:24.409 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:27:13.122297+0000", 2024-09-13T11:27:24.409 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000115856 2024-09-13T11:27:24.409 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:24.409 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:27:24.409 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:24.409 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.409 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.409 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGRecoveryDelete(3.31 e242,227 3:8d6a2c26:::benchmark_data_smithi177_31116_object63237:head,182'7774)", 2024-09-13T11:27:24.409 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:27:12.447056+0000", 2024-09-13T11:27:24.409 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 11.872826745999999, 2024-09-13T11:27:24.409 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.67796466300000002, 2024-09-13T11:27:24.409 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:27:24.410 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:27:24.410 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:27:24.410 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:27:24.410 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.410 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:27:24.410 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:27:12.447056+0000", 2024-09-13T11:27:24.410 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:24.410 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.410 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.410 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:27:24.410 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:27:12.447056+0000", 2024-09-13T11:27:24.410 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:24.410 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.410 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.411 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:27:24.411 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:27:12.447057+0000", 2024-09-13T11:27:24.411 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.232e-06 2024-09-13T11:27:24.411 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.411 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.411 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:27:24.411 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:27:12.447062+0000", 2024-09-13T11:27:24.411 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.5210000000000002e-06 2024-09-13T11:27:24.411 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.411 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.411 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:27:24.411 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:27:13.124249+0000", 2024-09-13T11:27:24.411 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.677186442 2024-09-13T11:27:24.411 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.412 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.412 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:27:24.412 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:27:13.124262+0000", 2024-09-13T11:27:24.412 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3372e-05 2024-09-13T11:27:24.412 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.412 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.412 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:27:24.412 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:27:13.124863+0000", 2024-09-13T11:27:24.412 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00060077500000000001 2024-09-13T11:27:24.412 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.412 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.412 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:27:24.412 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:27:13.124880+0000", 2024-09-13T11:27:24.412 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.7167999999999999e-05 2024-09-13T11:27:24.413 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.413 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.413 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:27:24.413 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:27:13.125020+0000", 2024-09-13T11:27:24.413 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00014015300000000001 2024-09-13T11:27:24.413 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:24.413 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:27:24.413 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:24.413 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.413 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.413 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGRecoveryDelete(3.31 e250,227 3:8dba42e2:::benchmark_data_smithi177_31116_object112298:head,211'9288)", 2024-09-13T11:27:24.413 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:27:21.602205+0000", 2024-09-13T11:27:24.413 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 2.7176775439999998, 2024-09-13T11:27:24.413 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.37971433799999998, 2024-09-13T11:27:24.414 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:27:24.414 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:27:24.414 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:27:24.414 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:27:24.414 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.414 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:27:24.414 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:27:21.602205+0000", 2024-09-13T11:27:24.414 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:24.414 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.414 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.414 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:27:24.414 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:27:21.602205+0000", 2024-09-13T11:27:24.414 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:24.414 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.415 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.415 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:27:24.415 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:27:21.602206+0000", 2024-09-13T11:27:24.415 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.325e-06 2024-09-13T11:27:24.415 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.415 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.415 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:27:24.415 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:27:21.602211+0000", 2024-09-13T11:27:24.415 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.2880000000000003e-06 2024-09-13T11:27:24.415 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.415 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.415 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:27:24.415 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:27:21.981726+0000", 2024-09-13T11:27:24.415 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.37951518400000001 2024-09-13T11:27:24.415 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.416 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.416 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:27:24.416 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:27:21.981734+0000", 2024-09-13T11:27:24.416 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.5769999999999994e-06 2024-09-13T11:27:24.416 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.416 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.416 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:27:24.416 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:27:21.981831+0000", 2024-09-13T11:27:24.416 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.6232999999999998e-05 2024-09-13T11:27:24.416 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.416 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.416 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:27:24.416 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:27:21.981841+0000", 2024-09-13T11:27:24.416 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0105999999999999e-05 2024-09-13T11:27:24.417 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.417 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.417 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:27:24.417 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:27:21.981919+0000", 2024-09-13T11:27:24.417 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.8625e-05 2024-09-13T11:27:24.417 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:24.417 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:27:24.417 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:24.417 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:24.417 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:27:24.417 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-09-13T11:27:24.419 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-13T11:27:24.545 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-09-13T11:27:24.546 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [], 2024-09-13T11:27:24.546 INFO:teuthology.orchestra.run.smithi177.stdout: "num_ops": 0 2024-09-13T11:27:24.546 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-09-13T11:27:24.560 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-13T11:27:24.676 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-09-13T11:27:24.676 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [], 2024-09-13T11:27:24.676 INFO:teuthology.orchestra.run.smithi177.stdout: "complaint_time": 30, 2024-09-13T11:27:24.676 INFO:teuthology.orchestra.run.smithi177.stdout: "num_blocked_ops": 0 2024-09-13T11:27:24.676 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-09-13T11:27:24.690 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-13T11:27:24.810 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-09-13T11:27:24.811 INFO:teuthology.orchestra.run.smithi177.stdout: "size": 20, 2024-09-13T11:27:24.811 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 600, 2024-09-13T11:27:24.811 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [ 2024-09-13T11:27:24.811 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.811 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPushReply(3.13 75/70 [PushReplyOp(3:cfab03d6:::benchmark_data_smithi177_31116_object16599:head)])", 2024-09-13T11:27:24.811 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:31.163480+0000", 2024-09-13T11:27:24.811 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 233.646559, 2024-09-13T11:27:24.811 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.63009029299999997, 2024-09-13T11:27:24.811 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:27:24.811 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:27:24.811 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-13T11:27:24.811 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:27:24.811 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.812 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:27:24.812 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.163480+0000", 2024-09-13T11:27:24.812 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:24.812 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.812 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.812 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:27:24.812 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.163480+0000", 2024-09-13T11:27:24.812 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:24.812 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.812 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.812 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:27:24.812 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.163481+0000", 2024-09-13T11:27:24.812 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0410000000000001e-06 2024-09-13T11:27:24.813 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.813 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.813 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:27:24.813 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.163485+0000", 2024-09-13T11:27:24.813 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.1629999999999998e-06 2024-09-13T11:27:24.813 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.813 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.813 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:27:24.813 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.792166+0000", 2024-09-13T11:27:24.813 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.62868067400000005 2024-09-13T11:27:24.813 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.813 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.813 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:27:24.813 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.792184+0000", 2024-09-13T11:27:24.814 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.7898999999999999e-05 2024-09-13T11:27:24.814 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.814 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.814 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:27:24.814 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.792312+0000", 2024-09-13T11:27:24.814 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00012871000000000001 2024-09-13T11:27:24.814 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.814 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.814 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:27:24.814 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.793570+0000", 2024-09-13T11:27:24.814 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001257806 2024-09-13T11:27:24.814 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:24.814 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:27:24.814 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:24.814 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.815 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.815 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPushReply(3.13 76/70 [PushReplyOp(3:c9d43a82:::benchmark_data_smithi177_31116_object20026:head)])", 2024-09-13T11:27:24.815 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:32.427709+0000", 2024-09-13T11:27:24.815 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 232.38232948500001, 2024-09-13T11:27:24.815 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.46547316300000002, 2024-09-13T11:27:24.815 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:27:24.815 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:27:24.815 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-13T11:27:24.815 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:27:24.815 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.815 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:27:24.815 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:32.427709+0000", 2024-09-13T11:27:24.815 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:24.815 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.816 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.816 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:27:24.816 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:32.427709+0000", 2024-09-13T11:27:24.816 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:24.816 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.816 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.816 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:27:24.816 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:32.427711+0000", 2024-09-13T11:27:24.816 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5099999999999999e-06 2024-09-13T11:27:24.816 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.816 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.816 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:27:24.816 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:32.427717+0000", 2024-09-13T11:27:24.817 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.1569999999999997e-06 2024-09-13T11:27:24.817 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.817 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.817 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:27:24.817 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:32.892390+0000", 2024-09-13T11:27:24.817 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.46467251100000001 2024-09-13T11:27:24.817 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.817 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.817 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:27:24.817 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:32.892405+0000", 2024-09-13T11:27:24.817 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5296999999999999e-05 2024-09-13T11:27:24.817 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.817 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.817 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:27:24.818 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:32.892503+0000", 2024-09-13T11:27:24.818 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.7820999999999997e-05 2024-09-13T11:27:24.818 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.818 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.818 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:27:24.818 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:32.893183+0000", 2024-09-13T11:27:24.818 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00067986700000000001 2024-09-13T11:27:24.818 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:24.818 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:27:24.818 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:24.818 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.818 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.818 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:57435 3.3 e80/75 3:c0db4777:::benchmark_data_smithi177_31116_object57434:head v 80'3155, mlcod=80'3153)", 2024-09-13T11:27:24.818 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:37.437070+0000", 2024-09-13T11:27:24.819 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 227.372968668, 2024-09-13T11:27:24.819 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.35861845999999997, 2024-09-13T11:27:24.819 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:27:24.819 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:27:24.819 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:27:24.819 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:27:24.819 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.819 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:27:24.819 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:37.437070+0000", 2024-09-13T11:27:24.819 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:24.819 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.819 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.819 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:27:24.819 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:37.437070+0000", 2024-09-13T11:27:24.820 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:24.820 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.820 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.820 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:27:24.820 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:37.437072+0000", 2024-09-13T11:27:24.820 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5549999999999999e-06 2024-09-13T11:27:24.820 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.820 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.820 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:27:24.820 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:37.437141+0000", 2024-09-13T11:27:24.820 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.9494999999999998e-05 2024-09-13T11:27:24.820 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.820 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.820 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:27:24.821 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:37.793400+0000", 2024-09-13T11:27:24.821 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.35625817999999998 2024-09-13T11:27:24.821 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.821 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.821 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:27:24.821 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:37.793406+0000", 2024-09-13T11:27:24.821 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.6830000000000002e-06 2024-09-13T11:27:24.821 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.821 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.821 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:27:24.821 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:37.793472+0000", 2024-09-13T11:27:24.821 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.5671000000000002e-05 2024-09-13T11:27:24.821 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.821 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.822 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:27:24.822 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:37.793492+0000", 2024-09-13T11:27:24.822 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.0051000000000002e-05 2024-09-13T11:27:24.822 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.822 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.822 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:27:24.822 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:37.795626+0000", 2024-09-13T11:27:24.822 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0021344990000000002 2024-09-13T11:27:24.822 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.822 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.822 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:27:24.822 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:37.795689+0000", 2024-09-13T11:27:24.822 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.2325999999999999e-05 2024-09-13T11:27:24.822 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:24.822 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:27:24.823 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:24.823 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.823 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.823 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPushReply(3.23 84/78 [PushReplyOp(3:c7e44616:::benchmark_data_smithi177_31116_object32371:head)])", 2024-09-13T11:27:24.823 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:41.338210+0000", 2024-09-13T11:27:24.823 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 223.47182859700001, 2024-09-13T11:27:24.823 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.41118492299999998, 2024-09-13T11:27:24.823 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:27:24.823 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:27:24.823 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-13T11:27:24.823 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:27:24.823 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.823 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:27:24.823 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.338210+0000", 2024-09-13T11:27:24.824 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:24.824 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.824 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.824 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:27:24.824 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.338210+0000", 2024-09-13T11:27:24.824 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:24.824 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.824 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.824 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:27:24.824 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.338212+0000", 2024-09-13T11:27:24.824 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.314e-06 2024-09-13T11:27:24.824 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.824 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.825 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:27:24.825 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.338219+0000", 2024-09-13T11:27:24.825 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.6469999999999998e-06 2024-09-13T11:27:24.825 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.825 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.825 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:27:24.825 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.748675+0000", 2024-09-13T11:27:24.825 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.41045567500000002 2024-09-13T11:27:24.825 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.825 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.825 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:27:24.825 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.748684+0000", 2024-09-13T11:27:24.825 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.2150000000000007e-06 2024-09-13T11:27:24.825 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.826 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.826 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:27:24.826 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.748739+0000", 2024-09-13T11:27:24.826 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.4543000000000003e-05 2024-09-13T11:27:24.826 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.826 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.826 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:27:24.826 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.749395+0000", 2024-09-13T11:27:24.826 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00065652899999999999 2024-09-13T11:27:24.826 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:24.826 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:27:24.826 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:24.826 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.827 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.827 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPushReply(3.23 85/78 [PushReplyOp(3:c5c6dce1:::benchmark_data_smithi177_31116_object37293:head)])", 2024-09-13T11:27:24.827 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:42.377862+0000", 2024-09-13T11:27:24.827 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 222.43217713799999, 2024-09-13T11:27:24.827 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.62538808599999995, 2024-09-13T11:27:24.827 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:27:24.827 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:27:24.827 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-13T11:27:24.827 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:27:24.827 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.827 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:27:24.827 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.377862+0000", 2024-09-13T11:27:24.827 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:24.827 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.828 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.828 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:27:24.828 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.377862+0000", 2024-09-13T11:27:24.828 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:24.828 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.828 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.828 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:27:24.828 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.377863+0000", 2024-09-13T11:27:24.828 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.6899999999999996e-07 2024-09-13T11:27:24.828 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.828 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.828 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:27:24.828 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.377866+0000", 2024-09-13T11:27:24.829 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.219e-06 2024-09-13T11:27:24.829 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.829 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.829 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:27:24.829 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:43.001857+0000", 2024-09-13T11:27:24.829 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.62399084299999996 2024-09-13T11:27:24.829 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.829 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.829 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:27:24.829 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:43.001869+0000", 2024-09-13T11:27:24.829 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.2415999999999999e-05 2024-09-13T11:27:24.829 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.829 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.830 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:27:24.830 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:43.002194+0000", 2024-09-13T11:27:24.830 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000324774 2024-09-13T11:27:24.830 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.830 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.830 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:27:24.830 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:43.003250+0000", 2024-09-13T11:27:24.830 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001055865 2024-09-13T11:27:24.830 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:24.830 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:27:24.830 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:24.830 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.830 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.830 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPush(3.1c 89/77 [PushOp(3:38bcbc71:::benchmark_data_smithi177_31116_object31448:head, version: 59'1946, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:38bcbc71:::benchmark_data_smithi177_31116_object31448:head@59'1946, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:27:24.831 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:46.670723+0000", 2024-09-13T11:27:24.831 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 218.13931591400001, 2024-09-13T11:27:24.831 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.27576065, 2024-09-13T11:27:24.831 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:27:24.831 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:27:24.831 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:27:24.831 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:27:24.831 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.831 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:27:24.831 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.670723+0000", 2024-09-13T11:27:24.831 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:24.831 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.831 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.831 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:27:24.832 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.670723+0000", 2024-09-13T11:27:24.832 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:24.832 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.832 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.832 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:27:24.832 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.670724+0000", 2024-09-13T11:27:24.832 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.4389999999999999e-06 2024-09-13T11:27:24.832 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.832 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.832 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:27:24.832 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.670754+0000", 2024-09-13T11:27:24.832 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.9224999999999999e-05 2024-09-13T11:27:24.832 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.832 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.833 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:27:24.833 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.944369+0000", 2024-09-13T11:27:24.833 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.27361529800000001 2024-09-13T11:27:24.833 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.833 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.833 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:27:24.833 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.944422+0000", 2024-09-13T11:27:24.833 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.3318e-05 2024-09-13T11:27:24.833 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.833 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.833 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:27:24.833 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.944711+0000", 2024-09-13T11:27:24.833 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00028912399999999998 2024-09-13T11:27:24.833 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.834 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.834 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:27:24.834 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.944735+0000", 2024-09-13T11:27:24.834 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.3366999999999998e-05 2024-09-13T11:27:24.834 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.834 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.834 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:27:24.834 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.946484+0000", 2024-09-13T11:27:24.834 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0017488790000000001 2024-09-13T11:27:24.834 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:24.834 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:27:24.834 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:24.834 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.834 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.835 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:83223 3.1c e106/77 3:3bc6018d:::benchmark_data_smithi177_31116_object83222:head v 106'4363, mlcod=106'4360)", 2024-09-13T11:27:24.835 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:04.512139+0000", 2024-09-13T11:27:24.835 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 200.29789980300001, 2024-09-13T11:27:24.835 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.471923235, 2024-09-13T11:27:24.835 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:27:24.835 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:27:24.835 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:27:24.835 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:27:24.835 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.835 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:27:24.835 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.512139+0000", 2024-09-13T11:27:24.835 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:24.835 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.835 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.836 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:27:24.836 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.512139+0000", 2024-09-13T11:27:24.836 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:24.836 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.836 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.836 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:27:24.836 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.512143+0000", 2024-09-13T11:27:24.836 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.427e-06 2024-09-13T11:27:24.836 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.836 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.836 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:27:24.836 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.512227+0000", 2024-09-13T11:27:24.836 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.3965e-05 2024-09-13T11:27:24.836 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.837 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.837 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:27:24.837 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.976330+0000", 2024-09-13T11:27:24.837 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.46410342599999999 2024-09-13T11:27:24.837 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.837 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.837 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:27:24.837 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.976350+0000", 2024-09-13T11:27:24.837 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.9533000000000001e-05 2024-09-13T11:27:24.837 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.837 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.837 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:27:24.837 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.976488+0000", 2024-09-13T11:27:24.837 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00013799400000000001 2024-09-13T11:27:24.838 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.838 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.838 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:27:24.838 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.976543+0000", 2024-09-13T11:27:24.838 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.5442000000000002e-05 2024-09-13T11:27:24.838 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.838 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.838 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:27:24.838 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.983227+0000", 2024-09-13T11:27:24.838 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0066842899999999998 2024-09-13T11:27:24.838 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.838 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.838 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:27:24.838 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.984062+0000", 2024-09-13T11:27:24.839 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00083515800000000002 2024-09-13T11:27:24.839 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:24.839 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:27:24.839 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:24.839 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.839 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.839 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:110256 3.13 e119/83 3:cb00d078:::benchmark_data_smithi177_31116_object110255:head v 119'4823, mlcod=119'4821)", 2024-09-13T11:27:24.839 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:25.284530+0000", 2024-09-13T11:27:24.839 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 179.52550934300001, 2024-09-13T11:27:24.839 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.343491771, 2024-09-13T11:27:24.839 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:27:24.839 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:27:24.839 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:27:24.839 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:27:24.840 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.840 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:27:24.840 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.284530+0000", 2024-09-13T11:27:24.840 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:24.840 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.840 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.840 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:27:24.840 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.284530+0000", 2024-09-13T11:27:24.840 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:24.840 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.840 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.840 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:27:24.840 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.284532+0000", 2024-09-13T11:27:24.840 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.8870000000000001e-06 2024-09-13T11:27:24.841 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.841 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.841 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:27:24.841 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.284558+0000", 2024-09-13T11:27:24.841 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.6899000000000001e-05 2024-09-13T11:27:24.841 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.841 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.841 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:27:24.841 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.626172+0000", 2024-09-13T11:27:24.841 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.34161370800000002 2024-09-13T11:27:24.841 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.841 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.841 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:27:24.841 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.626187+0000", 2024-09-13T11:27:24.842 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.485e-05 2024-09-13T11:27:24.842 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.842 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.842 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:27:24.842 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.626245+0000", 2024-09-13T11:27:24.842 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.7998000000000003e-05 2024-09-13T11:27:24.842 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.842 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.842 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:27:24.842 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.626260+0000", 2024-09-13T11:27:24.842 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5429000000000001e-05 2024-09-13T11:27:24.842 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.842 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.842 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:27:24.843 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.627924+0000", 2024-09-13T11:27:24.843 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0016634060000000001 2024-09-13T11:27:24.843 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.843 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.843 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:27:24.843 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.628021+0000", 2024-09-13T11:27:24.843 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.7594000000000001e-05 2024-09-13T11:27:24.843 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:24.843 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:27:24.843 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:24.843 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.843 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.843 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPush(3.2b 121/119 [PushOp(3:d53a23e8:::benchmark_data_smithi177_31116_object5366:head, version: 45'342, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:d53a23e8:::benchmark_data_smithi177_31116_object5366:head@45'342, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:27:24.843 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:27.268731+0000", 2024-09-13T11:27:24.844 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 177.54130788099999, 2024-09-13T11:27:24.844 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.72228345000000005, 2024-09-13T11:27:24.844 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:27:24.844 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:27:24.844 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:27:24.844 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:27:24.844 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.844 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:27:24.844 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:27.268731+0000", 2024-09-13T11:27:24.844 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:24.844 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.844 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.844 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:27:24.844 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:27.268731+0000", 2024-09-13T11:27:24.845 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:24.845 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.845 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.845 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:27:24.845 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:27.268733+0000", 2024-09-13T11:27:24.845 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.9740000000000001e-06 2024-09-13T11:27:24.845 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.845 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.845 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:27:24.845 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:27.268778+0000", 2024-09-13T11:27:24.845 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.4679999999999999e-05 2024-09-13T11:27:24.845 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.845 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.845 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:27:24.846 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:27.988594+0000", 2024-09-13T11:27:24.846 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.71981667299999996 2024-09-13T11:27:24.846 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.846 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.846 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:27:24.846 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:27.988652+0000", 2024-09-13T11:27:24.846 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.7640999999999999e-05 2024-09-13T11:27:24.846 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.846 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.846 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:27:24.846 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:27.988889+0000", 2024-09-13T11:27:24.846 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00023706500000000001 2024-09-13T11:27:24.846 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.847 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.847 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:27:24.847 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:27.988933+0000", 2024-09-13T11:27:24.847 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.3865000000000001e-05 2024-09-13T11:27:24.847 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.847 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.847 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:27:24.847 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:27.991015+0000", 2024-09-13T11:27:24.847 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.002081552 2024-09-13T11:27:24.847 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:24.847 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:27:24.847 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:24.847 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.847 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.848 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPush(3.2b 128/119 [PushOp(3:d6ff590f:::benchmark_data_smithi177_31116_object64581:head, version: 90'3750, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:d6ff590f:::benchmark_data_smithi177_31116_object64581:head@90'3750, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:27:24.848 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:37.982062+0000", 2024-09-13T11:27:24.848 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 166.82797725899999, 2024-09-13T11:27:24.848 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.28899675200000002, 2024-09-13T11:27:24.848 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:27:24.848 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:27:24.848 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:27:24.848 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:27:24.848 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.848 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:27:24.848 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:37.982062+0000", 2024-09-13T11:27:24.848 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:24.848 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.848 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.849 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:27:24.849 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:37.982062+0000", 2024-09-13T11:27:24.849 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:24.849 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.849 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.849 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:27:24.849 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:37.982064+0000", 2024-09-13T11:27:24.849 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.2259999999999999e-06 2024-09-13T11:27:24.849 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.849 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.849 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:27:24.849 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:37.982114+0000", 2024-09-13T11:27:24.849 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.0330000000000001e-05 2024-09-13T11:27:24.850 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.850 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.850 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:27:24.850 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:38.269983+0000", 2024-09-13T11:27:24.850 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.28786869999999998 2024-09-13T11:27:24.850 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.850 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.850 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:27:24.850 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:38.270000+0000", 2024-09-13T11:27:24.850 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.6977999999999999e-05 2024-09-13T11:27:24.850 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.850 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.850 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:27:24.850 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:38.270092+0000", 2024-09-13T11:27:24.851 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.221e-05 2024-09-13T11:27:24.851 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.851 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.851 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:27:24.851 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:38.270103+0000", 2024-09-13T11:27:24.851 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.1253999999999999e-05 2024-09-13T11:27:24.851 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.851 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.851 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:27:24.851 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:38.271058+0000", 2024-09-13T11:27:24.851 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00095505399999999999 2024-09-13T11:27:24.851 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:24.851 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:27:24.851 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:24.852 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.852 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.852 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:191665 3.a e176/117 3:53060b28:::benchmark_data_smithi177_31116_object51362:head v 176'6424, mlcod=176'6422)", 2024-09-13T11:27:24.852 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:25:39.657701+0000", 2024-09-13T11:27:24.852 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 105.152337532, 2024-09-13T11:27:24.852 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.25090320900000002, 2024-09-13T11:27:24.852 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:27:24.852 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:27:24.852 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:27:24.852 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:27:24.852 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.852 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:27:24.852 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:39.657701+0000", 2024-09-13T11:27:24.852 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:24.853 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.853 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.853 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:27:24.853 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:39.657701+0000", 2024-09-13T11:27:24.853 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:24.853 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.853 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.853 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:27:24.853 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:39.657704+0000", 2024-09-13T11:27:24.853 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.7650000000000002e-06 2024-09-13T11:27:24.853 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.853 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.853 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:27:24.853 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:39.657714+0000", 2024-09-13T11:27:24.854 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.5719999999999992e-06 2024-09-13T11:27:24.854 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.854 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.854 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:27:24.854 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:39.904490+0000", 2024-09-13T11:27:24.854 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.246775831 2024-09-13T11:27:24.854 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.854 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.854 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:27:24.854 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:39.904503+0000", 2024-09-13T11:27:24.854 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3723000000000001e-05 2024-09-13T11:27:24.854 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.854 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.854 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:27:24.855 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:39.904660+0000", 2024-09-13T11:27:24.855 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00015705499999999999 2024-09-13T11:27:24.855 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.855 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.855 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:27:24.855 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:39.904699+0000", 2024-09-13T11:27:24.855 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.8877000000000001e-05 2024-09-13T11:27:24.855 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.855 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.855 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:27:24.855 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:39.908544+0000", 2024-09-13T11:27:24.855 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0038448739999999999 2024-09-13T11:27:24.855 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.855 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.856 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:27:24.856 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:39.908605+0000", 2024-09-13T11:27:24.856 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.0511999999999999e-05 2024-09-13T11:27:24.856 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:24.856 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:27:24.856 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:24.856 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.856 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.856 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGRecoveryDelete(3.35 e231,227 3:af1a9c89:::benchmark_data_smithi177_31116_object6576:head,148'5831)", 2024-09-13T11:27:24.856 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:26:59.140312+0000", 2024-09-13T11:27:24.856 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 25.669727194, 2024-09-13T11:27:24.856 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.27474032799999998, 2024-09-13T11:27:24.856 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:27:24.857 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:27:24.857 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:27:24.857 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:27:24.857 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.857 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:27:24.857 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:59.140312+0000", 2024-09-13T11:27:24.857 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:24.857 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.857 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.857 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:27:24.857 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:59.140312+0000", 2024-09-13T11:27:24.857 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:24.857 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.857 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.857 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:27:24.858 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:59.140313+0000", 2024-09-13T11:27:24.858 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.418e-06 2024-09-13T11:27:24.858 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.858 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.858 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:27:24.858 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:59.140319+0000", 2024-09-13T11:27:24.858 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.49e-06 2024-09-13T11:27:24.858 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.858 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.858 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:27:24.858 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:59.414669+0000", 2024-09-13T11:27:24.858 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.27435077400000002 2024-09-13T11:27:24.858 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.859 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.859 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:27:24.859 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:59.414685+0000", 2024-09-13T11:27:24.859 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5276000000000001e-05 2024-09-13T11:27:24.859 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.859 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.859 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:27:24.859 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:59.414839+0000", 2024-09-13T11:27:24.859 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00015430500000000001 2024-09-13T11:27:24.859 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.859 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.859 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:27:24.859 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:59.414868+0000", 2024-09-13T11:27:24.859 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.9393000000000001e-05 2024-09-13T11:27:24.859 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.860 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.860 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:27:24.860 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:59.415052+0000", 2024-09-13T11:27:24.860 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00018367200000000001 2024-09-13T11:27:24.860 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:24.860 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:27:24.860 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:24.860 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.860 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.860 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:274423 3.1c e233/77 3:391423c9:::benchmark_data_smithi177_31116_object134120:head v 233'10364, mlcod=233'10362)", 2024-09-13T11:27:24.860 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:27:00.621698+0000", 2024-09-13T11:27:24.860 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 24.188340841999999, 2024-09-13T11:27:24.860 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.34328379999999997, 2024-09-13T11:27:24.860 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:27:24.861 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:27:24.861 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:27:24.861 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:27:24.861 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.861 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:27:24.861 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:27:00.621698+0000", 2024-09-13T11:27:24.861 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:24.861 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.861 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.861 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:27:24.861 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:27:00.621698+0000", 2024-09-13T11:27:24.861 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:24.861 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.861 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.862 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:27:24.862 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:27:00.621701+0000", 2024-09-13T11:27:24.862 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.396e-06 2024-09-13T11:27:24.862 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.862 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.862 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:27:24.862 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:27:00.621708+0000", 2024-09-13T11:27:24.862 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.8650000000000001e-06 2024-09-13T11:27:24.862 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.862 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.862 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:27:24.862 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:27:00.959179+0000", 2024-09-13T11:27:24.862 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.33747026200000002 2024-09-13T11:27:24.862 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.863 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.863 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:27:24.863 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:27:00.959188+0000", 2024-09-13T11:27:24.863 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.0310000000000004e-06 2024-09-13T11:27:24.863 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.863 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.863 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:27:24.863 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:27:00.959244+0000", 2024-09-13T11:27:24.863 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.6536999999999998e-05 2024-09-13T11:27:24.863 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.863 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.863 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:27:24.863 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:27:00.959263+0000", 2024-09-13T11:27:24.863 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.8802999999999999e-05 2024-09-13T11:27:24.864 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.864 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.864 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:27:24.864 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:27:00.964940+0000", 2024-09-13T11:27:24.864 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0056765670000000004 2024-09-13T11:27:24.864 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.864 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.864 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:27:24.864 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:27:00.964982+0000", 2024-09-13T11:27:24.864 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.2339000000000002e-05 2024-09-13T11:27:24.864 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:24.864 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:27:24.864 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:24.864 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.865 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.865 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:278044 3.d 3:b34c5ad6:::benchmark_data_smithi177_31116_object137741:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e237)", 2024-09-13T11:27:24.865 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:27:05.131032+0000", 2024-09-13T11:27:24.865 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 19.679007357, 2024-09-13T11:27:24.865 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.72037806999999998, 2024-09-13T11:27:24.865 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:27:24.865 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:27:24.865 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:27:24.865 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:27:24.865 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:27:24.865 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:27:24.865 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 278044 2024-09-13T11:27:24.865 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.865 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:27:24.866 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.866 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:27:24.866 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:27:05.131032+0000", 2024-09-13T11:27:24.866 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:24.866 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.866 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.866 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:27:24.866 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:27:05.131032+0000", 2024-09-13T11:27:24.866 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:24.866 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.866 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.866 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:27:24.866 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:27:05.131034+0000", 2024-09-13T11:27:24.866 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.7470000000000001e-06 2024-09-13T11:27:24.866 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.867 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.867 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:27:24.867 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:27:05.131041+0000", 2024-09-13T11:27:24.867 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.2419999999999997e-06 2024-09-13T11:27:24.867 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.867 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.867 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:27:24.867 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:27:05.131056+0000", 2024-09-13T11:27:24.867 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5031999999999999e-05 2024-09-13T11:27:24.867 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.867 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.867 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:27:24.867 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:27:05.131063+0000", 2024-09-13T11:27:24.867 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.3810000000000001e-06 2024-09-13T11:27:24.868 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.868 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.868 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:27:24.868 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:27:05.131118+0000", 2024-09-13T11:27:24.868 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.4688999999999998e-05 2024-09-13T11:27:24.868 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.868 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.868 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:27:24.868 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:27:05.131274+0000", 2024-09-13T11:27:24.868 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00015649500000000001 2024-09-13T11:27:24.868 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.868 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.868 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 3", 2024-09-13T11:27:24.868 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:27:05.131399+0000", 2024-09-13T11:27:24.869 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000124941 2024-09-13T11:27:24.869 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.869 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.869 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:27:24.869 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:27:05.133762+0000", 2024-09-13T11:27:24.869 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0023625819999999998 2024-09-13T11:27:24.869 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.869 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.869 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:27:24.869 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:27:05.851342+0000", 2024-09-13T11:27:24.869 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.71758002200000004 2024-09-13T11:27:24.869 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.869 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.870 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:27:24.870 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:27:05.851380+0000", 2024-09-13T11:27:24.870 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.8031000000000002e-05 2024-09-13T11:27:24.870 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.870 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.870 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:27:24.870 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:27:05.851410+0000", 2024-09-13T11:27:24.870 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.9907999999999999e-05 2024-09-13T11:27:24.870 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:24.870 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:27:24.870 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:24.870 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.870 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.870 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGRecoveryDeleteReply(3.33 e241,227 3:cce407df:::benchmark_data_smithi177_31116_object60783:head,181'7682)", 2024-09-13T11:27:24.870 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:27:09.920975+0000", 2024-09-13T11:27:24.871 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 14.889063461999999, 2024-09-13T11:27:24.871 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.83160401699999997, 2024-09-13T11:27:24.871 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:27:24.871 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:27:24.871 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-13T11:27:24.871 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:27:24.871 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.871 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:27:24.871 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:27:09.920975+0000", 2024-09-13T11:27:24.871 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:24.871 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.871 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.871 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:27:24.871 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:27:09.920975+0000", 2024-09-13T11:27:24.872 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:24.872 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.872 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.872 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:27:24.872 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:27:09.920976+0000", 2024-09-13T11:27:24.872 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.8999999999999995e-07 2024-09-13T11:27:24.872 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.872 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.872 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:27:24.872 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:27:09.920983+0000", 2024-09-13T11:27:24.872 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.6880000000000002e-06 2024-09-13T11:27:24.872 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.872 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.873 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:27:24.873 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:27:10.752182+0000", 2024-09-13T11:27:24.873 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.83119937899999996 2024-09-13T11:27:24.873 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.873 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.873 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:27:24.873 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:27:10.752192+0000", 2024-09-13T11:27:24.873 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0047000000000001e-05 2024-09-13T11:27:24.873 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.873 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.873 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:27:24.873 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:27:10.752241+0000", 2024-09-13T11:27:24.873 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.8940000000000002e-05 2024-09-13T11:27:24.873 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.874 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.874 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:27:24.874 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:27:10.752580+0000", 2024-09-13T11:27:24.874 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00033817299999999999 2024-09-13T11:27:24.874 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:24.874 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:27:24.874 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:24.874 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.874 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.874 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGRecoveryDeleteReply(3.33 e241,227 3:cd2a70c2:::benchmark_data_smithi177_31116_object63502:head,182'7754)", 2024-09-13T11:27:24.874 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:27:11.037803+0000", 2024-09-13T11:27:24.874 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 13.772235658, 2024-09-13T11:27:24.874 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.45914262500000003, 2024-09-13T11:27:24.874 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:27:24.874 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:27:24.875 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-13T11:27:24.875 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:27:24.875 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.875 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:27:24.875 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:27:11.037803+0000", 2024-09-13T11:27:24.875 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:24.875 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.875 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.875 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:27:24.875 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:27:11.037803+0000", 2024-09-13T11:27:24.875 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:24.875 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.875 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.876 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:27:24.876 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:27:11.037805+0000", 2024-09-13T11:27:24.876 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.376e-06 2024-09-13T11:27:24.876 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.876 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.876 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:27:24.876 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:27:11.037809+0000", 2024-09-13T11:27:24.876 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.4959999999999998e-06 2024-09-13T11:27:24.876 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.876 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.876 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:27:24.876 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:27:11.496436+0000", 2024-09-13T11:27:24.876 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.45862708699999999 2024-09-13T11:27:24.876 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.877 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.877 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:27:24.877 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:27:11.496447+0000", 2024-09-13T11:27:24.877 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0594999999999999e-05 2024-09-13T11:27:24.877 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.877 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.877 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:27:24.877 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:27:11.496546+0000", 2024-09-13T11:27:24.877 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.8690999999999994e-05 2024-09-13T11:27:24.877 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.877 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.877 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:27:24.877 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:27:11.496946+0000", 2024-09-13T11:27:24.878 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00040037999999999999 2024-09-13T11:27:24.878 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:24.878 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:27:24.878 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:24.878 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.878 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.878 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGRecoveryDeleteReply(3.33 e242,227 3:cf8ec38a:::benchmark_data_smithi177_31116_object65988:head,183'7844)", 2024-09-13T11:27:24.878 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:27:12.111155+0000", 2024-09-13T11:27:24.878 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 12.698883728, 2024-09-13T11:27:24.878 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.47315008200000003, 2024-09-13T11:27:24.878 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:27:24.878 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:27:24.878 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-13T11:27:24.879 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:27:24.879 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.879 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:27:24.879 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:27:12.111155+0000", 2024-09-13T11:27:24.879 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:24.879 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.879 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.879 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:27:24.879 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:27:12.111155+0000", 2024-09-13T11:27:24.879 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:24.879 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.879 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.879 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:27:24.879 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:27:12.111160+0000", 2024-09-13T11:27:24.880 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.9300000000000002e-06 2024-09-13T11:27:24.880 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.880 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.880 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:27:24.880 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:27:12.111166+0000", 2024-09-13T11:27:24.880 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.7049999999999996e-06 2024-09-13T11:27:24.880 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.880 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.880 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:27:24.880 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:27:12.583950+0000", 2024-09-13T11:27:24.880 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.47278400300000001 2024-09-13T11:27:24.880 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.880 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.881 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:27:24.881 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:27:12.583955+0000", 2024-09-13T11:27:24.881 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.4750000000000001e-06 2024-09-13T11:27:24.881 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.881 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.881 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:27:24.881 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:27:12.584016+0000", 2024-09-13T11:27:24.881 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.0503e-05 2024-09-13T11:27:24.881 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.881 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.881 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:27:24.881 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:27:12.584305+0000", 2024-09-13T11:27:24.881 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000289466 2024-09-13T11:27:24.881 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:24.882 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:27:24.882 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:24.882 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.882 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.882 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:278385 3.13 e244/224 3:c8289c54:::benchmark_data_smithi177_31116_object138082:head v 244'9946, mlcod=241'9943)", 2024-09-13T11:27:24.882 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:27:16.135172+0000", 2024-09-13T11:27:24.882 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 8.6748669550000006, 2024-09-13T11:27:24.882 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.30351028899999999, 2024-09-13T11:27:24.882 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:27:24.882 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:27:24.882 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:27:24.882 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:27:24.882 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.882 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:27:24.883 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:27:16.135172+0000", 2024-09-13T11:27:24.883 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:24.883 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.883 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.883 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:27:24.883 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:27:16.135172+0000", 2024-09-13T11:27:24.883 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:24.883 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.883 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.883 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:27:24.883 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:27:16.135174+0000", 2024-09-13T11:27:24.883 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5850000000000001e-06 2024-09-13T11:27:24.883 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.883 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.883 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:27:24.884 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:27:16.135182+0000", 2024-09-13T11:27:24.884 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.7749999999999998e-06 2024-09-13T11:27:24.884 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.884 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.884 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:27:24.884 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:27:16.436542+0000", 2024-09-13T11:27:24.884 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.301359661 2024-09-13T11:27:24.884 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.884 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.884 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:27:24.884 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:27:16.436547+0000", 2024-09-13T11:27:24.884 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.5770000000000004e-06 2024-09-13T11:27:24.884 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.885 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.885 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:27:24.885 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:27:16.436584+0000", 2024-09-13T11:27:24.885 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.6965000000000003e-05 2024-09-13T11:27:24.885 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.885 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.885 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:27:24.885 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:27:16.436600+0000", 2024-09-13T11:27:24.885 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.6025999999999999e-05 2024-09-13T11:27:24.885 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.885 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.885 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:27:24.885 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:27:16.438648+0000", 2024-09-13T11:27:24.885 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0020481470000000002 2024-09-13T11:27:24.885 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.886 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.886 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:27:24.886 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:27:16.438682+0000", 2024-09-13T11:27:24.886 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.4552999999999999e-05 2024-09-13T11:27:24.886 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:24.886 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:27:24.886 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:24.886 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.886 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.886 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGRecoveryDeleteReply(3.2c e251,243 3:371b039e:::benchmark_data_smithi177_31116_object42421:head,170'6333)", 2024-09-13T11:27:24.886 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:27:22.199860+0000", 2024-09-13T11:27:24.886 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 2.6101793669999998, 2024-09-13T11:27:24.886 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.47269027499999999, 2024-09-13T11:27:24.886 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:27:24.887 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:27:24.887 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-13T11:27:24.887 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:27:24.887 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.887 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:27:24.887 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:27:22.199860+0000", 2024-09-13T11:27:24.887 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:24.887 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.887 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.887 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:27:24.887 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:27:22.199860+0000", 2024-09-13T11:27:24.887 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:24.887 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.887 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.888 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:27:24.888 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:27:22.199861+0000", 2024-09-13T11:27:24.888 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.6309999999999999e-06 2024-09-13T11:27:24.888 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.888 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.888 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:27:24.888 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:27:22.199871+0000", 2024-09-13T11:27:24.888 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.5100000000000004e-06 2024-09-13T11:27:24.888 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.888 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.888 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:27:24.888 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:27:22.672227+0000", 2024-09-13T11:27:24.888 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.47235632 2024-09-13T11:27:24.888 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.889 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.889 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:27:24.889 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:27:22.672238+0000", 2024-09-13T11:27:24.889 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0866999999999999e-05 2024-09-13T11:27:24.889 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.889 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.889 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:27:24.889 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:27:22.672295+0000", 2024-09-13T11:27:24.889 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.7522999999999998e-05 2024-09-13T11:27:24.889 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.889 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.889 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:27:24.889 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:27:22.672550+0000", 2024-09-13T11:27:24.889 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000254424 2024-09-13T11:27:24.890 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:24.890 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:27:24.890 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:24.890 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.890 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.890 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGRecoveryDeleteReply(3.3 e251,243 3:c3cc63e4:::benchmark_data_smithi177_31116_object33777:head,165'4932)", 2024-09-13T11:27:24.890 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:27:22.995869+0000", 2024-09-13T11:27:24.890 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 1.814170302, 2024-09-13T11:27:24.890 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.85287645199999995, 2024-09-13T11:27:24.890 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:27:24.890 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:27:24.890 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "queued for pg", 2024-09-13T11:27:24.890 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:27:24.890 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.890 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:27:24.891 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:27:22.995869+0000", 2024-09-13T11:27:24.891 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:24.891 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.891 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.891 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:27:24.891 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:27:22.995869+0000", 2024-09-13T11:27:24.891 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:24.891 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.891 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.891 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:27:24.891 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:27:22.995870+0000", 2024-09-13T11:27:24.891 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.624e-06 2024-09-13T11:27:24.891 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.891 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.891 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:27:24.892 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:27:22.995877+0000", 2024-09-13T11:27:24.892 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.268e-06 2024-09-13T11:27:24.892 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.892 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.892 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:27:24.892 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:27:23.848583+0000", 2024-09-13T11:27:24.892 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.85270599499999999 2024-09-13T11:27:24.892 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.892 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.892 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:27:24.892 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:27:23.848600+0000", 2024-09-13T11:27:24.892 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.7742999999999999e-05 2024-09-13T11:27:24.892 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:24.893 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:24.893 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:27:24.893 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:27:23.848745+0000", 2024-09-13T11:27:24.893 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00014482199999999999 2024-09-13T11:27:24.893 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:24.893 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:27:24.893 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:24.893 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:24.893 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:27:24.893 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-09-13T11:27:24.895 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-09-13T11:27:24.923 INFO:tasks.ceph.osd.12.smithi001.stderr:2024-09-13T11:27:24.918+0000 7f891d10b640 -1 osd.12 252 handle_osd_map injecting map crc failure 2024-09-13T11:27:24.955 INFO:teuthology.orchestra.run.smithi162.stdout:get_pg_num_history pg_num_history pg_num_history(e238 pg_nums {1={23=1},2={25=8},3={117=46,224=56,235=66}} deleted_pools ) 2024-09-13T11:27:24.955 INFO:teuthology.orchestra.run.smithi162.stdout:Importing pgid 3.32 2024-09-13T11:27:24.970 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4c36832c:::benchmark_data_smithi177_31116_object139842:head# 2024-09-13T11:27:24.970 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:27:24.972 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4c8b6c2c:::benchmark_data_smithi177_31116_object138330:head# 2024-09-13T11:27:24.972 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:27:24.972 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4ca618c4:::benchmark_data_smithi177_31116_object138601:head# 2024-09-13T11:27:24.972 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:27:24.973 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4cce3ca4:::benchmark_data_smithi177_31116_object138517:head# 2024-09-13T11:27:24.973 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:27:24.973 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4cdefb85:::benchmark_data_smithi177_31116_object138131:head# 2024-09-13T11:27:24.973 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:27:24.974 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4cdfe228:::benchmark_data_smithi177_31116_object138507:head# 2024-09-13T11:27:24.974 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:27:24.975 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4d09234a:::benchmark_data_smithi177_31116_object138495:head# 2024-09-13T11:27:24.975 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:27:24.977 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4d45afc8:::benchmark_data_smithi177_31116_object139901:head# 2024-09-13T11:27:24.977 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:27:24.980 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4d70efd9:::benchmark_data_smithi177_31116_object139003:head# 2024-09-13T11:27:24.980 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:27:24.983 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4d79a7be:::benchmark_data_smithi177_31116_object138000:head# 2024-09-13T11:27:24.983 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:27:24.986 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4da9e252:::benchmark_data_smithi177_31116_object139857:head# 2024-09-13T11:27:24.986 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:27:24.988 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4dd0f18d:::benchmark_data_smithi177_31116_object137806:head# 2024-09-13T11:27:24.989 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:27:24.991 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4dd1c71b:::benchmark_data_smithi177_31116_object138813:head# 2024-09-13T11:27:24.991 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:27:24.994 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4dd68623:::benchmark_data_smithi177_31116_object140295:head# 2024-09-13T11:27:24.994 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:27:24.995 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4de1fb58:::benchmark_data_smithi177_31116_object140223:head# 2024-09-13T11:27:24.995 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:27:24.998 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4df30a72:::benchmark_data_smithi177_31116_object139649:head# 2024-09-13T11:27:24.998 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:27:25.001 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4dfff984:::benchmark_data_smithi177_31116_object138372:head# 2024-09-13T11:27:25.001 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:27:25.003 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4e0caca5:::benchmark_data_smithi177_31116_object138189:head# 2024-09-13T11:27:25.004 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:27:25.006 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4e106809:::benchmark_data_smithi177_31116_object139020:head# 2024-09-13T11:27:25.006 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:27:25.009 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4e2dac0e:::benchmark_data_smithi177_31116_object139986:head# 2024-09-13T11:27:25.009 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:27:25.012 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4e2e55c4:::benchmark_data_smithi177_31116_object139822:head# 2024-09-13T11:27:25.013 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:27:25.015 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4e487caf:::benchmark_data_smithi177_31116_object137938:head# 2024-09-13T11:27:25.015 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:27:25.016 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4e50d499:::benchmark_data_smithi177_31116_object138038:head# 2024-09-13T11:27:25.016 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:27:25.018 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-09-13T11:27:25.018 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [], 2024-09-13T11:27:25.018 INFO:teuthology.orchestra.run.smithi177.stdout: "num_ops": 0 2024-09-13T11:27:25.018 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-09-13T11:27:25.019 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4e5c15f0:::benchmark_data_smithi177_31116_object139138:head# 2024-09-13T11:27:25.019 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:27:25.021 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4e8f93b7:::benchmark_data_smithi177_31116_object138120:head# 2024-09-13T11:27:25.021 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:27:25.024 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4ea0254f:::benchmark_data_smithi177_31116_object139194:head# 2024-09-13T11:27:25.024 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:27:25.026 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4ea73683:::benchmark_data_smithi177_31116_object139004:head# 2024-09-13T11:27:25.026 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:27:25.029 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4ebae361:::benchmark_data_smithi177_31116_object138102:head# 2024-09-13T11:27:25.029 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:27:25.032 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4f0452aa:::benchmark_data_smithi177_31116_object139766:head# 2024-09-13T11:27:25.032 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:27:25.033 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-09-13T11:27:25.034 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4f08b27e:::benchmark_data_smithi177_31116_object138488:head# 2024-09-13T11:27:25.034 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:27:25.035 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4f326442:::benchmark_data_smithi177_31116_object137928:head# 2024-09-13T11:27:25.035 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:27:25.036 INFO:teuthology.orchestra.run.smithi001.stderr:nodeep-scrub is unset 2024-09-13T11:27:25.038 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4f637648:::benchmark_data_smithi177_31116_object139045:head# 2024-09-13T11:27:25.038 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:27:25.041 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4f915478:::benchmark_data_smithi177_31116_object138703:head# 2024-09-13T11:27:25.041 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:27:25.042 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4f9cd3bd:::benchmark_data_smithi177_31116_object139079:head# 2024-09-13T11:27:25.042 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:27:25.043 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4fd2ea6a:::benchmark_data_smithi177_31116_object138815:head# 2024-09-13T11:27:25.043 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:27:25.044 INFO:teuthology.orchestra.run.smithi162.stdout:Write #3:4fd3b5f4:::benchmark_data_smithi177_31116_object139197:head# 2024-09-13T11:27:25.044 INFO:teuthology.orchestra.run.smithi162.stdout:snapset 0=[]:{} 2024-09-13T11:27:25.052 INFO:teuthology.orchestra.run.smithi162.stdout:write_pg epoch 238 info 3.32( v 237'9929 lc 171'7167 (0'0,237'9929] local-lis/les=0/0 n=44 ec=224/30 lis/c=0/122 les/c/f=0/160/0 sis=227) 2024-09-13T11:27:25.122 INFO:teuthology.orchestra.run.smithi162.stdout:Import successful 2024-09-13T11:27:25.160 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-09-13T11:27:25.160 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [], 2024-09-13T11:27:25.160 INFO:teuthology.orchestra.run.smithi177.stdout: "complaint_time": 30, 2024-09-13T11:27:25.160 INFO:teuthology.orchestra.run.smithi177.stdout: "num_blocked_ops": 0 2024-09-13T11:27:25.160 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-09-13T11:27:25.175 INFO:tasks.ceph.osd.14.smithi162.stderr:2024-09-13T11:27:25.171+0000 7f3dea330640 -1 osd.14 252 handle_osd_map injecting map crc failure 2024-09-13T11:27:25.177 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-09-13T11:27:25.302 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-09-13T11:27:25.303 INFO:teuthology.orchestra.run.smithi177.stdout: "size": 20, 2024-09-13T11:27:25.303 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 600, 2024-09-13T11:27:25.303 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [ 2024-09-13T11:27:25.303 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.303 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:79 3.1 e32/30 3:841df5b5:::benchmark_data_smithi177_31116_object78:head v 32'6, mlcod=32'4)", 2024-09-13T11:27:25.303 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:22:27.057292+0000", 2024-09-13T11:27:25.303 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 298.24440157700002, 2024-09-13T11:27:25.303 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.89296786800000005, 2024-09-13T11:27:25.303 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:27:25.303 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:27:25.303 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:27:25.303 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:27:25.303 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.304 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:27:25.304 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.057292+0000", 2024-09-13T11:27:25.304 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:25.304 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.304 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.304 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:27:25.304 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.057292+0000", 2024-09-13T11:27:25.304 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:25.304 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.304 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.304 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:27:25.304 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.057293+0000", 2024-09-13T11:27:25.304 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.6109999999999999e-06 2024-09-13T11:27:25.305 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.305 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.305 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:27:25.305 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.057330+0000", 2024-09-13T11:27:25.305 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.6721999999999998e-05 2024-09-13T11:27:25.305 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.305 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.305 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:27:25.305 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.948197+0000", 2024-09-13T11:27:25.305 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.890867042 2024-09-13T11:27:25.306 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.306 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.306 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:27:25.306 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.948213+0000", 2024-09-13T11:27:25.306 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5826000000000001e-05 2024-09-13T11:27:25.306 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.306 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.306 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:27:25.307 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.948377+0000", 2024-09-13T11:27:25.307 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00016348600000000001 2024-09-13T11:27:25.307 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.307 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.307 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:27:25.307 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.948447+0000", 2024-09-13T11:27:25.307 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.0891999999999998e-05 2024-09-13T11:27:25.307 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.307 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.307 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:27:25.307 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.950150+0000", 2024-09-13T11:27:25.307 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001702329 2024-09-13T11:27:25.308 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.308 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.308 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:27:25.308 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:22:27.950260+0000", 2024-09-13T11:27:25.308 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00010996 2024-09-13T11:27:25.308 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:25.308 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:27:25.308 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:25.308 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.309 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.309 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:45783 3.16 3.17c10a76 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e69)", 2024-09-13T11:27:25.309 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:20.947774+0000", 2024-09-13T11:27:25.309 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 244.35391932600001, 2024-09-13T11:27:25.309 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.96682922400000004, 2024-09-13T11:27:25.309 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:27:25.309 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:27:25.309 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-13T11:27:25.310 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:27:25.310 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:27:25.310 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:27:25.310 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 45783 2024-09-13T11:27:25.310 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.310 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:27:25.310 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.310 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:27:25.310 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:20.947774+0000", 2024-09-13T11:27:25.311 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:25.311 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.311 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.311 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:27:25.311 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:20.947774+0000", 2024-09-13T11:27:25.311 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:25.311 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.311 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.311 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:27:25.311 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:20.947777+0000", 2024-09-13T11:27:25.311 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.57e-06 2024-09-13T11:27:25.312 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.312 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.312 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:27:25.312 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:20.947802+0000", 2024-09-13T11:27:25.312 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.5528999999999999e-05 2024-09-13T11:27:25.312 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.312 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.312 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:27:25.312 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:20.947813+0000", 2024-09-13T11:27:25.312 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0473e-05 2024-09-13T11:27:25.312 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.313 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.313 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:27:25.313 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:20.947821+0000", 2024-09-13T11:27:25.313 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.3440000000000001e-06 2024-09-13T11:27:25.313 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.313 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.313 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:27:25.313 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:20.947933+0000", 2024-09-13T11:27:25.313 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00011210200000000001 2024-09-13T11:27:25.313 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.313 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.314 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for peered", 2024-09-13T11:27:25.314 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:20.947942+0000", 2024-09-13T11:27:25.314 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.6610000000000006e-06 2024-09-13T11:27:25.314 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.314 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.314 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:27:25.314 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:21.914579+0000", 2024-09-13T11:27:25.314 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.96663672300000003 2024-09-13T11:27:25.314 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.314 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.314 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:27:25.314 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:21.914603+0000", 2024-09-13T11:27:25.314 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.4822000000000001e-05 2024-09-13T11:27:25.314 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:25.315 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:27:25.315 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:25.315 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.315 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.315 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:54625 3.1f 3:f812cdfb:::benchmark_data_smithi177_31116_object54624:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e75)", 2024-09-13T11:27:25.315 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:30.979327+0000", 2024-09-13T11:27:25.315 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 234.32236653699999, 2024-09-13T11:27:25.315 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.96163670899999998, 2024-09-13T11:27:25.315 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:27:25.315 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:27:25.315 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-13T11:27:25.315 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:27:25.315 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:27:25.316 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:27:25.316 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 54625 2024-09-13T11:27:25.316 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.316 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:27:25.316 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.316 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:27:25.316 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.979327+0000", 2024-09-13T11:27:25.316 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:25.316 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.316 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.316 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:27:25.316 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.979327+0000", 2024-09-13T11:27:25.316 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:25.317 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.317 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.317 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:27:25.317 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.979329+0000", 2024-09-13T11:27:25.317 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.686e-06 2024-09-13T11:27:25.317 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.317 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.317 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:27:25.317 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.979364+0000", 2024-09-13T11:27:25.317 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.5750000000000002e-05 2024-09-13T11:27:25.317 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.317 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.317 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:27:25.317 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.979408+0000", 2024-09-13T11:27:25.318 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.3795999999999999e-05 2024-09-13T11:27:25.318 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.318 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.318 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:27:25.318 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.979427+0000", 2024-09-13T11:27:25.318 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.9040999999999999e-05 2024-09-13T11:27:25.318 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.318 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.318 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:27:25.318 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.122291+0000", 2024-09-13T11:27:25.318 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.14286373599999999 2024-09-13T11:27:25.318 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.318 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.319 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for peered", 2024-09-13T11:27:25.319 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.122298+0000", 2024-09-13T11:27:25.319 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.3719999999999999e-06 2024-09-13T11:27:25.319 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.319 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.319 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:27:25.319 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.940341+0000", 2024-09-13T11:27:25.319 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.81804249699999998 2024-09-13T11:27:25.319 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.319 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.319 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:27:25.319 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.940964+0000", 2024-09-13T11:27:25.319 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00062283100000000001 2024-09-13T11:27:25.320 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:25.320 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:27:25.320 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:25.320 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.320 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.320 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:54649 3.1c 3:3a457a93:::benchmark_data_smithi177_31116_object54648:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e75)", 2024-09-13T11:27:25.320 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:30.979460+0000", 2024-09-13T11:27:25.320 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 234.32223334700001, 2024-09-13T11:27:25.320 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.95579424000000002, 2024-09-13T11:27:25.320 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:27:25.320 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:27:25.320 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-13T11:27:25.321 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:27:25.321 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:27:25.321 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:27:25.321 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 54649 2024-09-13T11:27:25.321 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.321 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:27:25.321 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.321 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:27:25.321 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.979460+0000", 2024-09-13T11:27:25.321 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:25.321 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.321 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.321 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:27:25.322 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.979460+0000", 2024-09-13T11:27:25.322 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:25.322 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.322 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.322 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:27:25.322 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.979462+0000", 2024-09-13T11:27:25.322 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5799999999999999e-06 2024-09-13T11:27:25.322 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.322 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.322 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:27:25.322 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.979524+0000", 2024-09-13T11:27:25.322 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.2521999999999995e-05 2024-09-13T11:27:25.322 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.322 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.323 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:27:25.323 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.979561+0000", 2024-09-13T11:27:25.323 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.6349e-05 2024-09-13T11:27:25.323 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.323 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.323 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:27:25.323 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.979571+0000", 2024-09-13T11:27:25.323 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0457e-05 2024-09-13T11:27:25.323 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.323 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.323 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:27:25.323 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.121029+0000", 2024-09-13T11:27:25.323 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.14145766200000001 2024-09-13T11:27:25.323 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.324 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.324 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for peered", 2024-09-13T11:27:25.324 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.121039+0000", 2024-09-13T11:27:25.324 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0084e-05 2024-09-13T11:27:25.324 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.324 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.324 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:27:25.324 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.935157+0000", 2024-09-13T11:27:25.324 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.81411807899999999 2024-09-13T11:27:25.324 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.324 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.324 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:27:25.325 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.935254+0000", 2024-09-13T11:27:25.325 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.7507e-05 2024-09-13T11:27:25.325 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:25.325 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:27:25.325 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:25.325 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.325 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.325 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:54654 3.1c 3:3f0bb31a:::benchmark_data_smithi177_31116_object54653:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e75)", 2024-09-13T11:27:25.325 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:30.979665+0000", 2024-09-13T11:27:25.325 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 234.322028182, 2024-09-13T11:27:25.325 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.95623908099999999, 2024-09-13T11:27:25.325 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:27:25.325 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:27:25.326 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-13T11:27:25.326 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:27:25.326 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:27:25.326 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:27:25.326 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 54654 2024-09-13T11:27:25.326 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.326 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:27:25.326 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.326 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:27:25.326 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.979665+0000", 2024-09-13T11:27:25.326 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:25.326 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.326 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.327 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:27:25.327 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.979665+0000", 2024-09-13T11:27:25.327 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:25.327 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.327 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.327 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:27:25.327 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.979667+0000", 2024-09-13T11:27:25.327 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3540000000000001e-06 2024-09-13T11:27:25.327 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.327 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.327 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:27:25.327 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.979703+0000", 2024-09-13T11:27:25.327 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.6758000000000003e-05 2024-09-13T11:27:25.328 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.328 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.328 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:27:25.328 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.979714+0000", 2024-09-13T11:27:25.328 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0804e-05 2024-09-13T11:27:25.328 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.328 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.328 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:27:25.328 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:30.979724+0000", 2024-09-13T11:27:25.328 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0040999999999999e-05 2024-09-13T11:27:25.328 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.328 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.328 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:27:25.329 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.121061+0000", 2024-09-13T11:27:25.329 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.141336713 2024-09-13T11:27:25.329 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.329 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.329 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for peered", 2024-09-13T11:27:25.329 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.121067+0000", 2024-09-13T11:27:25.329 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.4150000000000004e-06 2024-09-13T11:27:25.329 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.329 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.329 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:27:25.329 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.935776+0000", 2024-09-13T11:27:25.329 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.81470863100000002 2024-09-13T11:27:25.329 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.329 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.330 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:27:25.330 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.935904+0000", 2024-09-13T11:27:25.330 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00012836499999999999 2024-09-13T11:27:25.330 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:25.330 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:27:25.330 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:25.330 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.330 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.330 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:54668 3.1c 3:3d2c5e40:::benchmark_data_smithi177_31116_object54667:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e75)", 2024-09-13T11:27:25.330 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:31.245525+0000", 2024-09-13T11:27:25.332 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 234.056168915, 2024-09-13T11:27:25.332 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.69828973000000005, 2024-09-13T11:27:25.332 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:27:25.332 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:27:25.332 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:27:25.332 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:27:25.332 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:27:25.332 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:27:25.333 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 54668 2024-09-13T11:27:25.333 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.333 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:27:25.333 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.333 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:27:25.333 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.245525+0000", 2024-09-13T11:27:25.333 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:25.333 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.333 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.334 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:27:25.334 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.245525+0000", 2024-09-13T11:27:25.334 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:25.334 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.334 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.334 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:27:25.334 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.245526+0000", 2024-09-13T11:27:25.334 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.719e-06 2024-09-13T11:27:25.334 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.334 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.334 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:27:25.335 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.245547+0000", 2024-09-13T11:27:25.335 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.1084999999999999e-05 2024-09-13T11:27:25.335 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.335 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.335 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:27:25.335 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.245558+0000", 2024-09-13T11:27:25.335 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0934000000000001e-05 2024-09-13T11:27:25.335 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.335 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.335 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:27:25.335 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.245564+0000", 2024-09-13T11:27:25.335 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.6269999999999997e-06 2024-09-13T11:27:25.336 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.336 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.336 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:27:25.336 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.245620+0000", 2024-09-13T11:27:25.336 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.6289999999999998e-05 2024-09-13T11:27:25.336 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.336 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.336 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for peered", 2024-09-13T11:27:25.336 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.245632+0000", 2024-09-13T11:27:25.336 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.1748e-05 2024-09-13T11:27:25.336 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.336 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.337 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:27:25.337 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.935976+0000", 2024-09-13T11:27:25.337 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.69034455299999997 2024-09-13T11:27:25.337 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.337 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.337 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:27:25.337 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.936302+0000", 2024-09-13T11:27:25.337 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000325397 2024-09-13T11:27:25.337 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.337 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.337 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 13", 2024-09-13T11:27:25.337 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.936459+0000", 2024-09-13T11:27:25.337 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00015754300000000001 2024-09-13T11:27:25.338 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.338 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.338 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:27:25.338 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.938813+0000", 2024-09-13T11:27:25.338 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0023530999999999999 2024-09-13T11:27:25.338 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.338 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.338 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:27:25.338 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.943695+0000", 2024-09-13T11:27:25.338 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.004882506 2024-09-13T11:27:25.338 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.338 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.338 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:27:25.339 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.943767+0000", 2024-09-13T11:27:25.339 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.2454999999999997e-05 2024-09-13T11:27:25.339 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.339 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.339 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:27:25.339 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:31.943814+0000", 2024-09-13T11:27:25.339 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.6773000000000001e-05 2024-09-13T11:27:25.339 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:25.339 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:27:25.339 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:25.339 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.339 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.339 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPush(3.16 84/70 [PushOp(3:6f7be541:::benchmark_data_smithi177_31116_object43332:head, version: 66'2788, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:6f7be541:::benchmark_data_smithi177_31116_object43332:head@66'2788, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:27:25.339 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:41.207515+0000", 2024-09-13T11:27:25.340 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 224.09417814400001, 2024-09-13T11:27:25.340 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.52643771500000003, 2024-09-13T11:27:25.340 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:27:25.340 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:27:25.340 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:27:25.340 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:27:25.340 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.340 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:27:25.340 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.207515+0000", 2024-09-13T11:27:25.341 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:25.341 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.341 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.341 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:27:25.341 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.207515+0000", 2024-09-13T11:27:25.341 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:25.341 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.341 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.341 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:27:25.341 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.207516+0000", 2024-09-13T11:27:25.341 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.096e-06 2024-09-13T11:27:25.341 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.342 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.342 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:27:25.342 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.207717+0000", 2024-09-13T11:27:25.342 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000200199 2024-09-13T11:27:25.342 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.342 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.342 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:27:25.342 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.732785+0000", 2024-09-13T11:27:25.342 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.52506869599999995 2024-09-13T11:27:25.342 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.342 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.342 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:27:25.342 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.732801+0000", 2024-09-13T11:27:25.343 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5305999999999999e-05 2024-09-13T11:27:25.343 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.343 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.343 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:27:25.343 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.732889+0000", 2024-09-13T11:27:25.343 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.8157000000000003e-05 2024-09-13T11:27:25.343 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.343 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.343 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:27:25.343 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.732901+0000", 2024-09-13T11:27:25.343 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.1866e-05 2024-09-13T11:27:25.343 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.343 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.343 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:27:25.344 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:41.733953+0000", 2024-09-13T11:27:25.344 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001052395 2024-09-13T11:27:25.344 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:25.344 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:27:25.344 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:25.344 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.344 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.344 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPushReply(3.1f 85/78 [PushReplyOp(3:fd221d21:::benchmark_data_smithi177_31116_object7639:head)])", 2024-09-13T11:27:25.344 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:42.198715+0000", 2024-09-13T11:27:25.344 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 223.10297857500001, 2024-09-13T11:27:25.344 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.48930391099999998, 2024-09-13T11:27:25.344 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:27:25.345 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:27:25.345 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-13T11:27:25.345 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:27:25.345 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.345 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:27:25.345 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.198715+0000", 2024-09-13T11:27:25.345 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:25.345 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.345 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.345 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:27:25.345 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.198715+0000", 2024-09-13T11:27:25.345 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:25.345 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.346 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.346 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:27:25.346 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.198716+0000", 2024-09-13T11:27:25.346 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.3500000000000005e-07 2024-09-13T11:27:25.346 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.346 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.346 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:27:25.346 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.198719+0000", 2024-09-13T11:27:25.346 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.4539999999999999e-06 2024-09-13T11:27:25.346 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.346 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.346 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:27:25.347 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.684732+0000", 2024-09-13T11:27:25.347 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.48601262099999998 2024-09-13T11:27:25.347 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.347 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.347 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:27:25.347 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.684746+0000", 2024-09-13T11:27:25.347 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.4518999999999999e-05 2024-09-13T11:27:25.347 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.347 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.347 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:27:25.347 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.684911+0000", 2024-09-13T11:27:25.347 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000164728 2024-09-13T11:27:25.348 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.348 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.348 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:27:25.348 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.688019+0000", 2024-09-13T11:27:25.348 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0031077539999999999 2024-09-13T11:27:25.348 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:25.348 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:27:25.348 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:25.348 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.348 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.348 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPushReply(3.1c 85/77 [PushReplyOp(3:3a669437:::benchmark_data_smithi177_31116_object25280:head)])", 2024-09-13T11:27:25.348 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:42.352466+0000", 2024-09-13T11:27:25.349 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 222.949227681, 2024-09-13T11:27:25.349 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.36670725300000001, 2024-09-13T11:27:25.349 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:27:25.349 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:27:25.349 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-13T11:27:25.349 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:27:25.349 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.349 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:27:25.349 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.352466+0000", 2024-09-13T11:27:25.349 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:25.349 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.349 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.350 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:27:25.350 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.352466+0000", 2024-09-13T11:27:25.350 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:25.350 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.350 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.350 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:27:25.350 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.352467+0000", 2024-09-13T11:27:25.350 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.1429999999999999e-06 2024-09-13T11:27:25.350 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.350 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.350 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:27:25.350 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.352474+0000", 2024-09-13T11:27:25.350 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.6100000000000002e-06 2024-09-13T11:27:25.351 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.351 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.351 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:27:25.351 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.717011+0000", 2024-09-13T11:27:25.351 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.36453790200000002 2024-09-13T11:27:25.351 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.351 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.351 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:27:25.351 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.717032+0000", 2024-09-13T11:27:25.351 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.0999e-05 2024-09-13T11:27:25.351 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.352 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.352 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:27:25.352 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.717162+0000", 2024-09-13T11:27:25.352 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000129159 2024-09-13T11:27:25.352 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.352 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.352 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:27:25.352 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.719173+0000", 2024-09-13T11:27:25.352 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0020114400000000002 2024-09-13T11:27:25.352 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:25.352 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:27:25.352 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:25.352 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.353 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.353 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPushReply(3.21 89/78 [PushReplyOp(3:8534d087:::benchmark_data_smithi177_31116_object5474:head)])", 2024-09-13T11:27:25.353 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:46.632834+0000", 2024-09-13T11:27:25.353 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 218.66885939900001, 2024-09-13T11:27:25.353 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.324805556, 2024-09-13T11:27:25.353 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:27:25.353 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:27:25.353 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-13T11:27:25.354 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:27:25.354 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.354 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:27:25.354 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.632834+0000", 2024-09-13T11:27:25.354 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:25.354 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.354 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.354 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:27:25.354 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.632834+0000", 2024-09-13T11:27:25.354 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:25.354 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.354 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.354 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:27:25.355 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.632835+0000", 2024-09-13T11:27:25.355 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.4270000000000001e-06 2024-09-13T11:27:25.355 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.355 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.355 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:27:25.355 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.632840+0000", 2024-09-13T11:27:25.355 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.7809999999999996e-06 2024-09-13T11:27:25.355 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.355 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.355 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:27:25.356 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.956282+0000", 2024-09-13T11:27:25.356 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.323441639 2024-09-13T11:27:25.356 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.356 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.356 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:27:25.356 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.956293+0000", 2024-09-13T11:27:25.356 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.083e-05 2024-09-13T11:27:25.356 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.357 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.357 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:27:25.357 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.956395+0000", 2024-09-13T11:27:25.357 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00010184200000000001 2024-09-13T11:27:25.357 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.357 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.357 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:27:25.357 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:46.957640+0000", 2024-09-13T11:27:25.357 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0012450370000000001 2024-09-13T11:27:25.357 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:25.357 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:27:25.357 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:25.357 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.358 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.358 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:83223 3.1c 3:3bc6018d:::benchmark_data_smithi177_31116_object83222:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106)", 2024-09-13T11:27:25.358 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:04.511163+0000", 2024-09-13T11:27:25.358 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 200.79053002200001, 2024-09-13T11:27:25.358 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.47558093200000001, 2024-09-13T11:27:25.358 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:27:25.358 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:27:25.358 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:27:25.358 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:27:25.358 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:27:25.358 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:27:25.358 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 83223 2024-09-13T11:27:25.359 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.359 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:27:25.359 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.359 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:27:25.359 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.511163+0000", 2024-09-13T11:27:25.359 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:25.359 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.359 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.359 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:27:25.359 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.511163+0000", 2024-09-13T11:27:25.359 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:25.359 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.359 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.360 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:27:25.360 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.511167+0000", 2024-09-13T11:27:25.360 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.8330000000000004e-06 2024-09-13T11:27:25.360 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.360 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.360 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:27:25.360 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.511219+0000", 2024-09-13T11:27:25.360 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.1771999999999998e-05 2024-09-13T11:27:25.360 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.360 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.360 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:27:25.361 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.511252+0000", 2024-09-13T11:27:25.361 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.3315999999999997e-05 2024-09-13T11:27:25.361 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.361 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.361 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:27:25.361 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.511268+0000", 2024-09-13T11:27:25.361 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5812999999999999e-05 2024-09-13T11:27:25.361 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.361 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.361 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:27:25.361 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.511418+0000", 2024-09-13T11:27:25.361 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000150023 2024-09-13T11:27:25.362 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.362 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.362 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:27:25.362 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.511670+0000", 2024-09-13T11:27:25.362 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00025185499999999999 2024-09-13T11:27:25.362 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.362 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.362 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 7", 2024-09-13T11:27:25.362 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.511905+0000", 2024-09-13T11:27:25.362 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00023493799999999999 2024-09-13T11:27:25.362 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.362 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.362 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:27:25.363 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.514960+0000", 2024-09-13T11:27:25.363 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.003054977 2024-09-13T11:27:25.363 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.363 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.363 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:27:25.363 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.986432+0000", 2024-09-13T11:27:25.363 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.47147209200000001 2024-09-13T11:27:25.363 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.363 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.363 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:27:25.363 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.986635+0000", 2024-09-13T11:27:25.363 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00020309899999999999 2024-09-13T11:27:25.363 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.364 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.364 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:27:25.364 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.986744+0000", 2024-09-13T11:27:25.364 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000109214 2024-09-13T11:27:25.364 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:25.364 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:27:25.364 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:25.364 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.364 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.364 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop_reply(client.4665.0:84803 3.1f e107/78)", 2024-09-13T11:27:25.364 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:05.912627+0000", 2024-09-13T11:27:25.364 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 199.38906611499999, 2024-09-13T11:27:25.364 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.33726566600000002, 2024-09-13T11:27:25.365 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:27:25.365 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:27:25.365 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-13T11:27:25.365 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:27:25.365 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.365 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:27:25.365 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:05.912627+0000", 2024-09-13T11:27:25.365 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:25.365 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.365 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.365 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:27:25.365 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:05.912627+0000", 2024-09-13T11:27:25.365 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:25.366 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.366 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.366 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:27:25.366 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:05.912629+0000", 2024-09-13T11:27:25.366 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.4440000000000001e-06 2024-09-13T11:27:25.366 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.366 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.366 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:27:25.366 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:05.912634+0000", 2024-09-13T11:27:25.366 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.6869999999999999e-06 2024-09-13T11:27:25.366 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.366 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.367 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:27:25.367 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:06.249722+0000", 2024-09-13T11:27:25.367 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.33708766099999998 2024-09-13T11:27:25.367 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.367 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.367 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:27:25.367 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:06.249736+0000", 2024-09-13T11:27:25.367 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3623e-05 2024-09-13T11:27:25.367 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.367 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.367 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:27:25.367 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:06.249878+0000", 2024-09-13T11:27:25.368 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00014264599999999999 2024-09-13T11:27:25.368 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.368 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.368 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:27:25.368 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:06.249893+0000", 2024-09-13T11:27:25.368 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.4605e-05 2024-09-13T11:27:25.368 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:25.368 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:27:25.368 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:25.368 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.368 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.369 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:110265 3.1 e119/75 3:8236484d:::benchmark_data_smithi177_31116_object110264:head v 119'3935, mlcod=118'3933)", 2024-09-13T11:27:25.369 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:25.343348+0000", 2024-09-13T11:27:25.369 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 179.95834587300001, 2024-09-13T11:27:25.369 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.60013013500000001, 2024-09-13T11:27:25.369 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:27:25.369 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:27:25.369 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:27:25.369 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:27:25.369 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.369 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:27:25.369 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.343348+0000", 2024-09-13T11:27:25.369 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:25.369 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.370 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.370 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:27:25.370 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.343348+0000", 2024-09-13T11:27:25.370 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:25.370 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.370 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.370 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:27:25.370 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.343350+0000", 2024-09-13T11:27:25.370 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.576e-06 2024-09-13T11:27:25.370 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.370 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.370 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:27:25.370 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.343443+0000", 2024-09-13T11:27:25.371 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.2989e-05 2024-09-13T11:27:25.371 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.371 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.371 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:27:25.371 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.940863+0000", 2024-09-13T11:27:25.371 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.59742003300000002 2024-09-13T11:27:25.371 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.371 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.371 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:27:25.371 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.940875+0000", 2024-09-13T11:27:25.371 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.2147000000000001e-05 2024-09-13T11:27:25.371 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.372 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.372 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:27:25.372 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.940993+0000", 2024-09-13T11:27:25.372 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000117903 2024-09-13T11:27:25.372 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.372 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.372 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:27:25.372 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.941017+0000", 2024-09-13T11:27:25.372 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.3442999999999999e-05 2024-09-13T11:27:25.372 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.372 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.372 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:27:25.372 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.943347+0000", 2024-09-13T11:27:25.373 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0023299869999999999 2024-09-13T11:27:25.373 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.373 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.373 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:27:25.373 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:25.943478+0000", 2024-09-13T11:27:25.373 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00013105699999999999 2024-09-13T11:27:25.373 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:25.373 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:27:25.373 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:25.373 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.373 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.373 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:260347 3.16 3:6d3745c1:::benchmark_data_smithi177_31116_object120044:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e219)", 2024-09-13T11:27:25.374 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:26:42.470302+0000", 2024-09-13T11:27:25.374 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 42.831391281999998, 2024-09-13T11:27:25.374 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.43228830200000001, 2024-09-13T11:27:25.374 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:27:25.374 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:27:25.374 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:27:25.374 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:27:25.374 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:27:25.374 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:27:25.374 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 260347 2024-09-13T11:27:25.374 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.374 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:27:25.374 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.375 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:27:25.375 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:42.470302+0000", 2024-09-13T11:27:25.375 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:25.375 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.375 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.375 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:27:25.375 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:42.470302+0000", 2024-09-13T11:27:25.375 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:25.375 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.375 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.375 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:27:25.375 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:42.470306+0000", 2024-09-13T11:27:25.375 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.2899999999999996e-06 2024-09-13T11:27:25.376 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.376 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.376 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:27:25.376 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:42.470318+0000", 2024-09-13T11:27:25.376 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.1426e-05 2024-09-13T11:27:25.376 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.376 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.376 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:27:25.376 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:42.470361+0000", 2024-09-13T11:27:25.376 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.35e-05 2024-09-13T11:27:25.377 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.377 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.377 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:27:25.377 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:42.470371+0000", 2024-09-13T11:27:25.377 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.1740000000000006e-06 2024-09-13T11:27:25.377 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.377 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.377 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:27:25.377 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:42.470474+0000", 2024-09-13T11:27:25.377 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000103225 2024-09-13T11:27:25.377 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.377 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.378 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:27:25.378 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:42.470735+0000", 2024-09-13T11:27:25.378 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00026167399999999999 2024-09-13T11:27:25.378 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.378 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.378 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 15", 2024-09-13T11:27:25.378 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:42.470939+0000", 2024-09-13T11:27:25.378 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00020340000000000001 2024-09-13T11:27:25.378 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.378 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.378 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:27:25.378 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:42.477467+0000", 2024-09-13T11:27:25.378 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0065277319999999996 2024-09-13T11:27:25.379 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.379 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.379 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:27:25.379 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:42.902321+0000", 2024-09-13T11:27:25.379 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.42485422099999998 2024-09-13T11:27:25.379 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.379 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.379 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:27:25.379 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:42.902468+0000", 2024-09-13T11:27:25.379 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000147591 2024-09-13T11:27:25.379 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.379 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.379 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:27:25.379 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:42.902590+0000", 2024-09-13T11:27:25.380 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000122069 2024-09-13T11:27:25.380 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:25.380 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:27:25.380 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:25.380 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.380 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.380 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:266874 3.2f 3:f5648933:::benchmark_data_smithi177_31116_object126571:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e224)", 2024-09-13T11:27:25.380 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:26:49.252945+0000", 2024-09-13T11:27:25.380 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 36.048748451000002, 2024-09-13T11:27:25.380 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.79831508200000001, 2024-09-13T11:27:25.380 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:27:25.380 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:27:25.380 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:27:25.381 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:27:25.381 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:27:25.381 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:27:25.381 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 266874 2024-09-13T11:27:25.381 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.381 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:27:25.381 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.381 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:27:25.381 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:49.252945+0000", 2024-09-13T11:27:25.381 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:25.381 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.381 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.382 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:27:25.382 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:49.252945+0000", 2024-09-13T11:27:25.382 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:25.382 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.382 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.382 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:27:25.382 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:49.252949+0000", 2024-09-13T11:27:25.382 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.4390000000000002e-06 2024-09-13T11:27:25.382 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.382 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.382 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:27:25.382 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:49.252961+0000", 2024-09-13T11:27:25.382 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.1891e-05 2024-09-13T11:27:25.383 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.383 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.383 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:27:25.383 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:49.252989+0000", 2024-09-13T11:27:25.383 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.7384000000000001e-05 2024-09-13T11:27:25.383 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.383 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.383 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:27:25.383 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:49.253004+0000", 2024-09-13T11:27:25.383 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5013e-05 2024-09-13T11:27:25.383 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.383 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.383 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:27:25.383 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:49.253095+0000", 2024-09-13T11:27:25.384 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.1150000000000007e-05 2024-09-13T11:27:25.384 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.384 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.384 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for peered", 2024-09-13T11:27:25.384 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:49.253106+0000", 2024-09-13T11:27:25.384 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0749e-05 2024-09-13T11:27:25.384 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.384 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.384 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:27:25.384 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:50.047276+0000", 2024-09-13T11:27:25.384 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.79417086299999995 2024-09-13T11:27:25.385 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.385 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.385 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:27:25.385 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:50.047392+0000", 2024-09-13T11:27:25.385 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000115777 2024-09-13T11:27:25.385 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.385 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.385 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 12", 2024-09-13T11:27:25.385 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:50.047515+0000", 2024-09-13T11:27:25.385 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00012257699999999999 2024-09-13T11:27:25.385 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.385 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.385 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:27:25.386 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:50.050394+0000", 2024-09-13T11:27:25.386 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.002879038 2024-09-13T11:27:25.386 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.386 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.386 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:27:25.386 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:50.051179+0000", 2024-09-13T11:27:25.386 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000785 2024-09-13T11:27:25.386 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.386 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.386 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:27:25.386 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:50.051223+0000", 2024-09-13T11:27:25.386 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.4202000000000001e-05 2024-09-13T11:27:25.386 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.387 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.387 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:27:25.387 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:50.051260+0000", 2024-09-13T11:27:25.387 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.6999000000000003e-05 2024-09-13T11:27:25.387 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:25.387 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:27:25.387 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:25.387 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.387 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.387 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGRecoveryDeleteReply(3.2f e231,227 3:f4d0ce4f:::benchmark_data_smithi177_31116_object254:head,144'6093)", 2024-09-13T11:27:25.387 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:26:58.970945+0000", 2024-09-13T11:27:25.387 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 26.330748102000001, 2024-09-13T11:27:25.387 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.45467746999999997, 2024-09-13T11:27:25.388 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:27:25.388 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:27:25.388 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-13T11:27:25.388 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:27:25.388 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.388 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:27:25.388 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:58.970945+0000", 2024-09-13T11:27:25.388 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:25.388 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.388 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.388 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:27:25.388 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:58.970945+0000", 2024-09-13T11:27:25.388 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:25.388 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.389 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.389 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:27:25.389 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:58.970947+0000", 2024-09-13T11:27:25.389 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.668e-06 2024-09-13T11:27:25.389 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.389 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.389 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:27:25.389 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:58.970961+0000", 2024-09-13T11:27:25.389 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3655999999999999e-05 2024-09-13T11:27:25.389 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.389 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.389 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:27:25.390 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:59.424664+0000", 2024-09-13T11:27:25.390 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.45370305599999999 2024-09-13T11:27:25.390 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.390 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.390 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:27:25.390 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:59.424682+0000", 2024-09-13T11:27:25.390 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.7949999999999999e-05 2024-09-13T11:27:25.390 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.390 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.390 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:27:25.390 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:59.425051+0000", 2024-09-13T11:27:25.390 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00036941300000000002 2024-09-13T11:27:25.390 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.391 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.391 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:27:25.391 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:59.425623+0000", 2024-09-13T11:27:25.391 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00057172699999999996 2024-09-13T11:27:25.391 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:25.391 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:27:25.391 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:25.391 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.391 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.391 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:274423 3.1c 3:391423c9:::benchmark_data_smithi177_31116_object134120:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e233)", 2024-09-13T11:27:25.391 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:27:00.620397+0000", 2024-09-13T11:27:25.391 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 24.681296521, 2024-09-13T11:27:25.391 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.34487627199999998, 2024-09-13T11:27:25.391 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:27:25.392 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:27:25.392 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:27:25.392 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:27:25.392 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:27:25.392 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:27:25.392 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 274423 2024-09-13T11:27:25.392 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.392 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:27:25.392 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.392 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:27:25.392 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:27:00.620397+0000", 2024-09-13T11:27:25.392 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:25.392 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.392 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.393 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:27:25.393 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:27:00.620397+0000", 2024-09-13T11:27:25.393 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:25.393 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.393 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.393 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:27:25.393 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:27:00.620399+0000", 2024-09-13T11:27:25.393 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.657e-06 2024-09-13T11:27:25.393 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.393 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.393 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:27:25.393 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:27:00.620403+0000", 2024-09-13T11:27:25.393 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.3069999999999996e-06 2024-09-13T11:27:25.393 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.394 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.394 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:27:25.394 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:27:00.620413+0000", 2024-09-13T11:27:25.394 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0355e-05 2024-09-13T11:27:25.394 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.394 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.394 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:27:25.394 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:27:00.620421+0000", 2024-09-13T11:27:25.394 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.0250000000000002e-06 2024-09-13T11:27:25.394 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.394 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.394 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:27:25.394 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:27:00.620851+0000", 2024-09-13T11:27:25.394 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00042989900000000003 2024-09-13T11:27:25.395 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.395 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.395 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:27:25.395 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:27:00.620967+0000", 2024-09-13T11:27:25.395 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00011627 2024-09-13T11:27:25.395 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.395 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.395 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 7", 2024-09-13T11:27:25.395 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:27:00.621613+0000", 2024-09-13T11:27:25.395 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00064577099999999995 2024-09-13T11:27:25.395 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.395 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.395 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:27:25.395 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:27:00.627319+0000", 2024-09-13T11:27:25.396 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0057055200000000004 2024-09-13T11:27:25.396 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.396 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.396 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:27:25.396 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:27:00.965178+0000", 2024-09-13T11:27:25.396 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.33785940599999997 2024-09-13T11:27:25.396 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.396 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.396 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:27:25.396 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:27:00.965231+0000", 2024-09-13T11:27:25.396 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.2944e-05 2024-09-13T11:27:25.396 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.396 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.397 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:27:25.397 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:27:00.965273+0000", 2024-09-13T11:27:25.397 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.2117999999999999e-05 2024-09-13T11:27:25.397 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:25.397 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:27:25.397 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:25.397 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.397 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.397 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGRecoveryDeleteReply(3.2f e238,227 3:f487a72d:::benchmark_data_smithi177_31116_object20739:head,155'6770)", 2024-09-13T11:27:25.397 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:27:06.435293+0000", 2024-09-13T11:27:25.397 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 18.866400694999999, 2024-09-13T11:27:25.397 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.398313372, 2024-09-13T11:27:25.397 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:27:25.398 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:27:25.398 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-13T11:27:25.398 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:27:25.398 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.398 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:27:25.398 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:27:06.435293+0000", 2024-09-13T11:27:25.398 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:25.398 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.398 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.398 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:27:25.398 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:27:06.435293+0000", 2024-09-13T11:27:25.398 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:25.398 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.399 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.399 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:27:25.399 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:27:06.435294+0000", 2024-09-13T11:27:25.399 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.234e-06 2024-09-13T11:27:25.399 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.399 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.399 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:27:25.399 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:27:06.435302+0000", 2024-09-13T11:27:25.399 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.7959999999999996e-06 2024-09-13T11:27:25.399 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.399 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.399 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:27:25.399 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:27:06.833212+0000", 2024-09-13T11:27:25.400 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.39791068000000002 2024-09-13T11:27:25.400 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.400 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.400 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:27:25.400 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:27:06.833219+0000", 2024-09-13T11:27:25.400 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.7449999999999998e-06 2024-09-13T11:27:25.400 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.400 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.400 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:27:25.400 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:27:06.833270+0000", 2024-09-13T11:27:25.400 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.0717000000000002e-05 2024-09-13T11:27:25.400 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.400 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.400 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:27:25.401 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:27:06.833606+0000", 2024-09-13T11:27:25.401 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00033619999999999999 2024-09-13T11:27:25.401 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:25.401 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:27:25.401 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:25.401 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.401 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.401 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGRecoveryDeleteReply(3.36 e239,227 3:6c15f3c5:::benchmark_data_smithi177_31116_object28159:head,161'6777)", 2024-09-13T11:27:25.401 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:27:07.338993+0000", 2024-09-13T11:27:25.401 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 17.962700554000001, 2024-09-13T11:27:25.401 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.68653944099999997, 2024-09-13T11:27:25.401 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:27:25.401 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:27:25.401 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-13T11:27:25.402 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:27:25.402 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.402 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:27:25.402 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:27:07.338993+0000", 2024-09-13T11:27:25.402 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:25.402 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.402 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.402 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:27:25.402 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:27:07.338993+0000", 2024-09-13T11:27:25.402 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:25.402 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.402 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.402 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:27:25.402 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:27:07.338994+0000", 2024-09-13T11:27:25.403 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.2979999999999999e-06 2024-09-13T11:27:25.403 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.403 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.403 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:27:25.403 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:27:07.339000+0000", 2024-09-13T11:27:25.403 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.8239999999999996e-06 2024-09-13T11:27:25.403 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.403 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.403 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:27:25.403 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:27:08.025032+0000", 2024-09-13T11:27:25.403 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.68603244399999996 2024-09-13T11:27:25.403 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.404 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.404 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:27:25.404 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:27:08.025042+0000", 2024-09-13T11:27:25.404 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.2269999999999999e-06 2024-09-13T11:27:25.404 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.404 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.404 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:27:25.404 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:27:08.025135+0000", 2024-09-13T11:27:25.404 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.3321999999999999e-05 2024-09-13T11:27:25.404 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.404 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.404 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:27:25.404 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:27:08.025532+0000", 2024-09-13T11:27:25.404 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00039732599999999998 2024-09-13T11:27:25.405 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:25.405 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:27:25.405 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:25.405 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.405 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.405 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGRecoveryDeleteReply(3.2f e251,227 3:f7605a46:::benchmark_data_smithi177_31116_object96330:head,204'9121)", 2024-09-13T11:27:25.405 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:27:22.373764+0000", 2024-09-13T11:27:25.405 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 2.9279293429999997, 2024-09-13T11:27:25.405 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.85736502000000003, 2024-09-13T11:27:25.405 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:27:25.405 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:27:25.405 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "queued for pg", 2024-09-13T11:27:25.405 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:27:25.405 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.406 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:27:25.406 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:27:22.373764+0000", 2024-09-13T11:27:25.406 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:25.406 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.406 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.406 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:27:25.406 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:27:22.373764+0000", 2024-09-13T11:27:25.406 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:25.406 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.406 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.406 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:27:25.406 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:27:22.373766+0000", 2024-09-13T11:27:25.406 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.412e-06 2024-09-13T11:27:25.406 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.407 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.407 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:27:25.407 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:27:22.373776+0000", 2024-09-13T11:27:25.407 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0081000000000001e-05 2024-09-13T11:27:25.407 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.407 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.407 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:27:25.407 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:27:23.231051+0000", 2024-09-13T11:27:25.407 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.85727519500000005 2024-09-13T11:27:25.407 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.407 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.407 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:27:25.407 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:27:23.231061+0000", 2024-09-13T11:27:25.407 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.7170000000000003e-06 2024-09-13T11:27:25.408 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.408 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.408 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:27:25.408 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:27:23.231129+0000", 2024-09-13T11:27:25.408 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.8615000000000006e-05 2024-09-13T11:27:25.408 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:25.408 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:27:25.408 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:25.408 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:25.408 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:27:25.408 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-09-13T11:27:25.410 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_ops_in_flight 2024-09-13T11:27:25.479 INFO:tasks.ceph.osd.1.smithi092.stderr:2024-09-13T11:27:25.475+0000 7fe76d3f2640 -1 osd.1 252 handle_osd_map injecting map crc failure 2024-09-13T11:27:25.535 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-09-13T11:27:25.535 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [], 2024-09-13T11:27:25.535 INFO:teuthology.orchestra.run.smithi177.stdout: "num_ops": 0 2024-09-13T11:27:25.535 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-09-13T11:27:25.549 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_blocked_ops 2024-09-13T11:27:25.600 DEBUG:teuthology.orchestra.run.smithi162:> sudo rm -f /var/log/ceph/exp.3.32.10 2024-09-13T11:27:25.614 INFO:tasks.thrashosds.thrasher:checking if target objectstore is bluestore on osd.10 2024-09-13T11:27:25.615 DEBUG:teuthology.orchestra.run.smithi162:> sudo ceph-bluestore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/bluestore_tool.$pid.log' --log-level=10 --path /var/lib/ceph/osd/ceph-10 show-label 2024-09-13T11:27:25.673 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-09-13T11:27:25.673 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [], 2024-09-13T11:27:25.673 INFO:teuthology.orchestra.run.smithi177.stdout: "complaint_time": 30, 2024-09-13T11:27:25.673 INFO:teuthology.orchestra.run.smithi177.stdout: "num_blocked_ops": 0 2024-09-13T11:27:25.673 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-09-13T11:27:25.688 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_historic_ops 2024-09-13T11:27:25.695 INFO:teuthology.orchestra.run.smithi162.stdout:inferring bluefs devices from bluestore path 2024-09-13T11:27:25.815 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-09-13T11:27:25.815 INFO:teuthology.orchestra.run.smithi177.stdout: "size": 20, 2024-09-13T11:27:25.815 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 600, 2024-09-13T11:27:25.815 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [ 2024-09-13T11:27:25.815 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.816 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:25383 3.0 e55/30 3:0a6bb869:::benchmark_data_smithi177_31116_object25382:head v 55'1620, mlcod=55'1617)", 2024-09-13T11:27:25.816 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:04.449956+0000", 2024-09-13T11:27:25.816 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 261.36481441699999, 2024-09-13T11:27:25.816 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.21143479600000001, 2024-09-13T11:27:25.816 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:27:25.816 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:27:25.816 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:27:25.816 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:27:25.816 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.816 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:27:25.816 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.449956+0000", 2024-09-13T11:27:25.816 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:25.817 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.817 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.817 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:27:25.817 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.449956+0000", 2024-09-13T11:27:25.817 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:25.817 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.817 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.817 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:27:25.817 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.449959+0000", 2024-09-13T11:27:25.817 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.393e-06 2024-09-13T11:27:25.817 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.817 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.818 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:27:25.818 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.450014+0000", 2024-09-13T11:27:25.818 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.4200000000000003e-05 2024-09-13T11:27:25.818 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.818 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.818 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:27:25.818 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.658637+0000", 2024-09-13T11:27:25.818 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.20862334799999999 2024-09-13T11:27:25.818 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.818 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.818 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:27:25.818 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.658650+0000", 2024-09-13T11:27:25.818 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3148e-05 2024-09-13T11:27:25.819 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.819 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.819 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:27:25.819 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.658867+0000", 2024-09-13T11:27:25.819 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00021673499999999999 2024-09-13T11:27:25.819 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.819 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.819 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:27:25.819 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.658931+0000", 2024-09-13T11:27:25.819 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.4045000000000001e-05 2024-09-13T11:27:25.819 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.819 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.819 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:27:25.819 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.661317+0000", 2024-09-13T11:27:25.820 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.002385823 2024-09-13T11:27:25.820 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.820 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.820 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:27:25.820 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:04.661391+0000", 2024-09-13T11:27:25.820 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.4103999999999994e-05 2024-09-13T11:27:25.820 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:25.820 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:27:25.820 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:25.820 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.820 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.820 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPush(3.16 85/70 [PushOp(3:69aac9fd:::benchmark_data_smithi177_31116_object43914:head, version: 66'2837, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:69aac9fd:::benchmark_data_smithi177_31116_object43914:head@66'2837, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:27:25.820 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:42.295995+0000", 2024-09-13T11:27:25.821 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 223.518775097, 2024-09-13T11:27:25.821 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.215089431, 2024-09-13T11:27:25.821 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:27:25.821 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:27:25.821 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:27:25.821 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:27:25.821 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.821 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:27:25.821 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.295995+0000", 2024-09-13T11:27:25.821 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:25.821 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.821 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.822 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:27:25.822 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.295995+0000", 2024-09-13T11:27:25.822 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:25.822 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.822 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.822 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:27:25.822 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.295997+0000", 2024-09-13T11:27:25.822 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3170000000000001e-06 2024-09-13T11:27:25.822 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.822 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.822 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:27:25.822 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.296058+0000", 2024-09-13T11:27:25.822 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.1690000000000001e-05 2024-09-13T11:27:25.822 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.823 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.823 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:27:25.823 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.509606+0000", 2024-09-13T11:27:25.823 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.21354748300000001 2024-09-13T11:27:25.823 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.823 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.823 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:27:25.823 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.509638+0000", 2024-09-13T11:27:25.823 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.1644999999999999e-05 2024-09-13T11:27:25.823 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.823 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.823 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:27:25.823 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.509814+0000", 2024-09-13T11:27:25.823 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00017679499999999999 2024-09-13T11:27:25.824 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.824 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.824 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:27:25.824 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.509840+0000", 2024-09-13T11:27:25.824 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.5871e-05 2024-09-13T11:27:25.824 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.824 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.824 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:27:25.824 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:42.511085+0000", 2024-09-13T11:27:25.824 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00124463 2024-09-13T11:27:25.824 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:25.824 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:27:25.824 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:25.825 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.825 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.825 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPush(3.1d 91/78 [PushOp(3:be6a0ae4:::benchmark_data_smithi177_31116_object25671:head, version: 58'1569, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:be6a0ae4:::benchmark_data_smithi177_31116_object25671:head@58'1569, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:27:25.825 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:48.481971+0000", 2024-09-13T11:27:25.825 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 217.33279985199999, 2024-09-13T11:27:25.825 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.47945454199999998, 2024-09-13T11:27:25.825 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:27:25.825 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:27:25.825 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:27:25.825 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:27:25.825 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.825 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:27:25.825 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.481971+0000", 2024-09-13T11:27:25.825 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:25.826 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.826 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.826 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:27:25.826 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.481971+0000", 2024-09-13T11:27:25.826 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:25.826 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.826 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.826 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:27:25.826 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.481972+0000", 2024-09-13T11:27:25.826 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5179999999999999e-06 2024-09-13T11:27:25.826 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.826 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.826 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:27:25.826 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.482000+0000", 2024-09-13T11:27:25.827 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.8249e-05 2024-09-13T11:27:25.827 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.827 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.827 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:27:25.827 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.959926+0000", 2024-09-13T11:27:25.827 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.47792585100000001 2024-09-13T11:27:25.827 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.827 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.827 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:27:25.827 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.959944+0000", 2024-09-13T11:27:25.827 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.7620000000000001e-05 2024-09-13T11:27:25.827 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.827 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.827 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:27:25.828 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.960059+0000", 2024-09-13T11:27:25.828 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000115188 2024-09-13T11:27:25.828 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.828 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.828 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:27:25.828 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.960074+0000", 2024-09-13T11:27:25.828 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.4738e-05 2024-09-13T11:27:25.828 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.828 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.828 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:27:25.828 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:48.961425+0000", 2024-09-13T11:27:25.828 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0013513780000000001 2024-09-13T11:27:25.828 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:25.829 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:27:25.829 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:25.829 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.829 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.829 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:71884 3.16 e96/88 3:6ca846b2:::benchmark_data_smithi177_31116_object71883:head v 96'3755, mlcod=96'3753)", 2024-09-13T11:27:25.829 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:54.108299+0000", 2024-09-13T11:27:25.829 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 211.706471679, 2024-09-13T11:27:25.829 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.20685140099999999, 2024-09-13T11:27:25.829 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:27:25.829 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:27:25.829 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:27:25.829 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:27:25.829 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.829 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:27:25.830 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.108299+0000", 2024-09-13T11:27:25.830 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:25.830 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.830 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.830 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:27:25.830 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.108299+0000", 2024-09-13T11:27:25.830 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:25.830 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.830 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.830 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:27:25.830 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.108300+0000", 2024-09-13T11:27:25.830 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.173e-06 2024-09-13T11:27:25.830 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.831 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.831 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:27:25.831 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.108323+0000", 2024-09-13T11:27:25.831 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.2591000000000001e-05 2024-09-13T11:27:25.831 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.831 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.831 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:27:25.831 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.312377+0000", 2024-09-13T11:27:25.831 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.20405456599999999 2024-09-13T11:27:25.831 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.831 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.831 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:27:25.831 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.312382+0000", 2024-09-13T11:27:25.831 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.1429999999999997e-06 2024-09-13T11:27:25.832 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.832 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.832 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:27:25.832 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.312480+0000", 2024-09-13T11:27:25.832 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.7742e-05 2024-09-13T11:27:25.832 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.832 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.832 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:27:25.832 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.312521+0000", 2024-09-13T11:27:25.832 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.1369000000000003e-05 2024-09-13T11:27:25.832 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.832 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.832 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:27:25.833 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.315105+0000", 2024-09-13T11:27:25.833 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0025835060000000002 2024-09-13T11:27:25.833 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.833 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.833 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:27:25.833 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.315150+0000", 2024-09-13T11:27:25.833 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.5311e-05 2024-09-13T11:27:25.833 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:25.834 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:27:25.834 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:25.834 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.834 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.834 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:71927 3.0 e97/75 3:0197244a:::benchmark_data_smithi177_31116_object71926:head v 97'3351, mlcod=96'3348)", 2024-09-13T11:27:25.834 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:23:54.200161+0000", 2024-09-13T11:27:25.834 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 211.61460945100001, 2024-09-13T11:27:25.834 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.26691738999999998, 2024-09-13T11:27:25.834 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:27:25.834 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:27:25.834 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:27:25.834 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:27:25.835 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.835 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:27:25.835 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.200161+0000", 2024-09-13T11:27:25.835 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:25.835 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.835 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.835 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:27:25.835 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.200161+0000", 2024-09-13T11:27:25.835 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:25.835 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.836 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.836 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:27:25.836 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.200162+0000", 2024-09-13T11:27:25.836 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.78e-07 2024-09-13T11:27:25.836 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.836 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.836 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:27:25.836 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.200200+0000", 2024-09-13T11:27:25.836 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.8121000000000003e-05 2024-09-13T11:27:25.836 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.836 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.837 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:27:25.837 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.465397+0000", 2024-09-13T11:27:25.837 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.26519681099999998 2024-09-13T11:27:25.837 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.837 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.837 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:27:25.837 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.465404+0000", 2024-09-13T11:27:25.837 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.7399999999999998e-06 2024-09-13T11:27:25.837 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.837 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.837 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:27:25.838 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.465456+0000", 2024-09-13T11:27:25.838 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.2626000000000001e-05 2024-09-13T11:27:25.838 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.838 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.838 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:27:25.838 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.465482+0000", 2024-09-13T11:27:25.838 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.6086000000000001e-05 2024-09-13T11:27:25.838 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.838 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.838 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:27:25.838 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.467035+0000", 2024-09-13T11:27:25.838 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0015523729999999999 2024-09-13T11:27:25.839 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.839 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.839 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:27:25.839 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:23:54.467078+0000", 2024-09-13T11:27:25.839 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.3754999999999997e-05 2024-09-13T11:27:25.839 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:25.839 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:27:25.839 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:25.839 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.839 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.840 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPush(3.1b 102/78 [PushOp(3:df786fc7:::benchmark_data_smithi177_31116_object48840:head, version: 71'3079, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:df786fc7:::benchmark_data_smithi177_31116_object48840:head@71'3079, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:27:25.840 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:00.290536+0000", 2024-09-13T11:27:25.840 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 205.524234689, 2024-09-13T11:27:25.840 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.20017585399999999, 2024-09-13T11:27:25.840 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:27:25.840 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:27:25.840 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:27:25.840 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:27:25.840 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.840 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:27:25.840 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.290536+0000", 2024-09-13T11:27:25.840 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:25.841 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.841 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.841 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:27:25.841 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.290536+0000", 2024-09-13T11:27:25.841 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:25.841 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.841 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.841 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:27:25.841 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.290537+0000", 2024-09-13T11:27:25.842 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.327e-06 2024-09-13T11:27:25.842 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.842 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.842 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:27:25.842 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.290581+0000", 2024-09-13T11:27:25.842 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.3893999999999997e-05 2024-09-13T11:27:25.842 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.842 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.842 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:27:25.842 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.488588+0000", 2024-09-13T11:27:25.843 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.198006708 2024-09-13T11:27:25.843 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.843 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.843 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:27:25.843 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.488607+0000", 2024-09-13T11:27:25.843 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.9162999999999999e-05 2024-09-13T11:27:25.843 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.843 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.843 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:27:25.843 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.488747+0000", 2024-09-13T11:27:25.843 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00014040799999999999 2024-09-13T11:27:25.844 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.844 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.844 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:27:25.844 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.488770+0000", 2024-09-13T11:27:25.844 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.279e-05 2024-09-13T11:27:25.844 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.844 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.844 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:27:25.845 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.490712+0000", 2024-09-13T11:27:25.845 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001941564 2024-09-13T11:27:25.845 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:25.845 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:27:25.845 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:25.845 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.845 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.845 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPush(3.1b 102/78 [PushOp(3:d82bf6ec:::benchmark_data_smithi177_31116_object48847:head, version: 71'3080, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:d82bf6ec:::benchmark_data_smithi177_31116_object48847:head@71'3080, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:27:25.845 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:00.293979+0000", 2024-09-13T11:27:25.846 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 205.52079116199999, 2024-09-13T11:27:25.846 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.20314832099999999, 2024-09-13T11:27:25.846 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:27:25.846 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:27:25.846 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:27:25.846 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:27:25.846 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.846 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:27:25.846 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.293979+0000", 2024-09-13T11:27:25.846 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:25.846 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.846 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.846 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:27:25.846 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.293979+0000", 2024-09-13T11:27:25.847 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:25.847 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.847 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.847 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:27:25.847 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.293981+0000", 2024-09-13T11:27:25.847 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.4449999999999999e-06 2024-09-13T11:27:25.847 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.847 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.847 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:27:25.847 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.294085+0000", 2024-09-13T11:27:25.847 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000103848 2024-09-13T11:27:25.847 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.847 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.847 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:27:25.848 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.495571+0000", 2024-09-13T11:27:25.848 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.20148640800000001 2024-09-13T11:27:25.848 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.848 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.848 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:27:25.848 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.495609+0000", 2024-09-13T11:27:25.848 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.7904000000000002e-05 2024-09-13T11:27:25.848 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.848 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.848 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:27:25.848 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.495729+0000", 2024-09-13T11:27:25.848 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000120556 2024-09-13T11:27:25.848 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.849 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.849 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:27:25.849 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.495751+0000", 2024-09-13T11:27:25.849 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.1511000000000001e-05 2024-09-13T11:27:25.849 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.849 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.849 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:27:25.849 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.497128+0000", 2024-09-13T11:27:25.849 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001376649 2024-09-13T11:27:25.849 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:25.849 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:27:25.849 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:25.849 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.850 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.850 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:79801 3.5 3:a6efdf84:::benchmark_data_smithi177_31116_object79800:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102)", 2024-09-13T11:27:25.850 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:00.403201+0000", 2024-09-13T11:27:25.850 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 205.41156957499999, 2024-09-13T11:27:25.850 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.31868626300000003, 2024-09-13T11:27:25.850 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:27:25.850 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:27:25.850 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:27:25.850 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:27:25.850 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:27:25.850 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:27:25.850 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 79801 2024-09-13T11:27:25.850 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.850 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:27:25.851 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.851 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:27:25.851 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.403201+0000", 2024-09-13T11:27:25.851 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:25.851 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.851 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.851 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:27:25.851 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.403201+0000", 2024-09-13T11:27:25.851 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:25.851 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.851 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.851 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:27:25.851 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.403204+0000", 2024-09-13T11:27:25.852 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.0819999999999999e-06 2024-09-13T11:27:25.852 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.852 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.852 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:27:25.852 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.403260+0000", 2024-09-13T11:27:25.852 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.5624999999999997e-05 2024-09-13T11:27:25.852 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.852 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.852 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:27:25.852 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.403280+0000", 2024-09-13T11:27:25.852 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.0060000000000001e-05 2024-09-13T11:27:25.852 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.852 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.853 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:27:25.853 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.403291+0000", 2024-09-13T11:27:25.853 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.1335000000000001e-05 2024-09-13T11:27:25.853 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.853 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.853 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:27:25.853 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.403344+0000", 2024-09-13T11:27:25.853 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.3217000000000002e-05 2024-09-13T11:27:25.853 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.853 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.853 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:27:25.853 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.403456+0000", 2024-09-13T11:27:25.853 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000111706 2024-09-13T11:27:25.853 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.854 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.854 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 13", 2024-09-13T11:27:25.854 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.403547+0000", 2024-09-13T11:27:25.854 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.1556000000000002e-05 2024-09-13T11:27:25.854 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.854 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.854 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:27:25.854 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.407436+0000", 2024-09-13T11:27:25.854 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.003888856 2024-09-13T11:27:25.854 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.854 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.854 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:27:25.854 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.721722+0000", 2024-09-13T11:27:25.854 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.31428562700000001 2024-09-13T11:27:25.854 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.855 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.855 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:27:25.855 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.721814+0000", 2024-09-13T11:27:25.855 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.1564000000000006e-05 2024-09-13T11:27:25.855 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.855 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.855 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:27:25.855 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:00.721887+0000", 2024-09-13T11:27:25.855 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.3634999999999995e-05 2024-09-13T11:27:25.855 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:25.855 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:27:25.855 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:25.855 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.856 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.856 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:83199 3.1d e106/102 3:be2c6fae:::benchmark_data_smithi177_31116_object83198:head v 106'4244, mlcod=106'4242)", 2024-09-13T11:27:25.856 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:04.298299+0000", 2024-09-13T11:27:25.856 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 201.516471448, 2024-09-13T11:27:25.856 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.25001107099999997, 2024-09-13T11:27:25.856 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:27:25.856 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:27:25.856 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:27:25.856 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:27:25.856 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.856 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:27:25.856 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.298299+0000", 2024-09-13T11:27:25.856 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:25.857 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.857 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.857 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:27:25.857 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.298299+0000", 2024-09-13T11:27:25.857 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:25.857 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.857 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.857 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:27:25.857 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.298301+0000", 2024-09-13T11:27:25.857 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5859999999999999e-06 2024-09-13T11:27:25.857 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.857 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.857 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:27:25.858 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.298368+0000", 2024-09-13T11:27:25.858 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.6928999999999996e-05 2024-09-13T11:27:25.858 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.858 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.858 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:27:25.858 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.545180+0000", 2024-09-13T11:27:25.858 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.24681209700000001 2024-09-13T11:27:25.858 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.858 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.858 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:27:25.858 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.545193+0000", 2024-09-13T11:27:25.858 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3427e-05 2024-09-13T11:27:25.858 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.859 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.859 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:27:25.859 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.545327+0000", 2024-09-13T11:27:25.859 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00013359400000000001 2024-09-13T11:27:25.859 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.859 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.859 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:27:25.859 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.545385+0000", 2024-09-13T11:27:25.859 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.8492000000000002e-05 2024-09-13T11:27:25.859 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.859 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.859 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:27:25.859 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.548225+0000", 2024-09-13T11:27:25.860 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.002839472 2024-09-13T11:27:25.860 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.860 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.860 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:27:25.860 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:04.548310+0000", 2024-09-13T11:27:25.860 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.5474000000000002e-05 2024-09-13T11:27:25.860 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:25.860 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:27:25.860 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:25.860 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.860 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.860 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:137062 3.5 3:a16e178f:::benchmark_data_smithi177_31116_object137061:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e139)", 2024-09-13T11:27:25.860 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:24:53.140525+0000", 2024-09-13T11:27:25.860 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 152.67424585500001, 2024-09-13T11:27:25.861 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.20697954599999999, 2024-09-13T11:27:25.861 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:27:25.861 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:27:25.861 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:27:25.861 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:27:25.861 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:27:25.861 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:27:25.861 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 137062 2024-09-13T11:27:25.861 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.861 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:27:25.861 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.861 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:27:25.861 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:53.140525+0000", 2024-09-13T11:27:25.862 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:25.862 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.862 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.862 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:27:25.862 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:53.140525+0000", 2024-09-13T11:27:25.862 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:25.862 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.862 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.862 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:27:25.862 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:53.140527+0000", 2024-09-13T11:27:25.862 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.1739999999999999e-06 2024-09-13T11:27:25.862 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.862 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.862 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:27:25.863 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:53.140553+0000", 2024-09-13T11:27:25.863 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.6631e-05 2024-09-13T11:27:25.863 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.863 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.863 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:27:25.863 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:53.140566+0000", 2024-09-13T11:27:25.863 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.2268e-05 2024-09-13T11:27:25.863 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.863 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.863 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:27:25.863 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:53.140574+0000", 2024-09-13T11:27:25.863 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.5669999999999995e-06 2024-09-13T11:27:25.863 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.863 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.864 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:27:25.864 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:53.140622+0000", 2024-09-13T11:27:25.864 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.8143999999999998e-05 2024-09-13T11:27:25.864 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.864 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.864 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:27:25.864 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:53.140759+0000", 2024-09-13T11:27:25.864 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000136499 2024-09-13T11:27:25.864 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.864 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.864 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 13", 2024-09-13T11:27:25.864 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:53.140887+0000", 2024-09-13T11:27:25.864 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00012796300000000001 2024-09-13T11:27:25.865 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.865 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.865 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:27:25.865 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:53.142675+0000", 2024-09-13T11:27:25.865 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001788073 2024-09-13T11:27:25.865 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.865 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.865 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:27:25.865 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:53.347407+0000", 2024-09-13T11:27:25.865 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.20473206899999999 2024-09-13T11:27:25.865 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.865 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.865 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:27:25.865 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:53.347464+0000", 2024-09-13T11:27:25.866 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.6536000000000002e-05 2024-09-13T11:27:25.866 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.866 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.866 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:27:25.866 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:24:53.347504+0000", 2024-09-13T11:27:25.866 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.0621999999999998e-05 2024-09-13T11:27:25.866 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:25.866 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:27:25.867 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:25.867 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.867 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.867 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:191058 3.1d e175/102 3:bd6e7029:::benchmark_data_smithi177_31116_object50755:head v 175'7612, mlcod=175'7609)", 2024-09-13T11:27:25.867 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:25:38.513527+0000", 2024-09-13T11:27:25.867 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 107.30124319799999, 2024-09-13T11:27:25.867 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.386104477, 2024-09-13T11:27:25.867 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:27:25.867 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:27:25.867 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:27:25.867 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:27:25.867 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.867 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:27:25.867 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:38.513527+0000", 2024-09-13T11:27:25.868 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:25.868 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.868 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.868 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:27:25.868 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:38.513527+0000", 2024-09-13T11:27:25.868 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:25.868 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.868 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.868 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:27:25.868 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:38.513530+0000", 2024-09-13T11:27:25.868 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.0029999999999999e-06 2024-09-13T11:27:25.869 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.869 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.869 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:27:25.869 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:38.513542+0000", 2024-09-13T11:27:25.869 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.1477e-05 2024-09-13T11:27:25.869 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.869 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.869 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:27:25.869 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:38.891617+0000", 2024-09-13T11:27:25.869 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.378075045 2024-09-13T11:27:25.869 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.869 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.870 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:27:25.870 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:38.891635+0000", 2024-09-13T11:27:25.870 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.7697e-05 2024-09-13T11:27:25.870 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.870 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.870 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:27:25.870 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:38.891775+0000", 2024-09-13T11:27:25.870 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000140665 2024-09-13T11:27:25.870 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.870 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.870 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:27:25.870 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:38.891833+0000", 2024-09-13T11:27:25.870 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.7769999999999997e-05 2024-09-13T11:27:25.871 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.871 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.871 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:27:25.871 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:38.899567+0000", 2024-09-13T11:27:25.871 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0077339970000000003 2024-09-13T11:27:25.871 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.871 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.871 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:27:25.871 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:38.899632+0000", 2024-09-13T11:27:25.871 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.4822999999999999e-05 2024-09-13T11:27:25.871 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:25.871 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:27:25.871 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:25.872 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.872 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.872 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:197236 3.5 3:a3127298:::benchmark_data_smithi177_31116_object56933:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e178)", 2024-09-13T11:27:25.872 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:25:44.300667+0000", 2024-09-13T11:27:25.872 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 101.514103107, 2024-09-13T11:27:25.872 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.37481205200000001, 2024-09-13T11:27:25.872 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:27:25.872 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:27:25.872 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:27:25.872 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:27:25.872 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:27:25.872 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:27:25.872 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 197236 2024-09-13T11:27:25.873 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.873 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:27:25.873 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.873 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:27:25.873 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:44.300667+0000", 2024-09-13T11:27:25.873 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:25.873 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.873 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.873 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:27:25.873 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:44.300667+0000", 2024-09-13T11:27:25.873 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:25.873 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.873 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.874 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:27:25.874 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:44.300670+0000", 2024-09-13T11:27:25.874 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.7259999999999998e-06 2024-09-13T11:27:25.874 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.874 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.874 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:27:25.874 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:44.300677+0000", 2024-09-13T11:27:25.874 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.1550000000000001e-06 2024-09-13T11:27:25.874 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.874 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.874 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:27:25.874 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:44.300695+0000", 2024-09-13T11:27:25.874 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.7552999999999999e-05 2024-09-13T11:27:25.875 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.875 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.875 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:27:25.875 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:44.300704+0000", 2024-09-13T11:27:25.875 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.5710000000000004e-06 2024-09-13T11:27:25.875 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.875 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.875 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:27:25.875 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:44.300762+0000", 2024-09-13T11:27:25.875 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.7351e-05 2024-09-13T11:27:25.875 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.875 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.875 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:27:25.876 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:44.300888+0000", 2024-09-13T11:27:25.876 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00012617999999999999 2024-09-13T11:27:25.876 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.876 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.876 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 13", 2024-09-13T11:27:25.876 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:44.300993+0000", 2024-09-13T11:27:25.876 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00010505600000000001 2024-09-13T11:27:25.876 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.876 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.876 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:27:25.877 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:44.302936+0000", 2024-09-13T11:27:25.877 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0019428749999999999 2024-09-13T11:27:25.877 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.877 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.877 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:27:25.877 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:44.675352+0000", 2024-09-13T11:27:25.877 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.37241603400000001 2024-09-13T11:27:25.877 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.877 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.877 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:27:25.877 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:44.675412+0000", 2024-09-13T11:27:25.877 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.9679999999999998e-05 2024-09-13T11:27:25.877 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.878 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.878 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:27:25.878 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:44.675479+0000", 2024-09-13T11:27:25.878 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.7871000000000001e-05 2024-09-13T11:27:25.878 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:25.878 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:27:25.878 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:25.878 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.878 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.878 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop_reply(client.4665.0:197236 3.5 e178/117 ondisk, result = 0)", 2024-09-13T11:27:25.879 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:25:44.304480+0000", 2024-09-13T11:27:25.879 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 101.51029084, 2024-09-13T11:27:25.879 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.37100864099999997, 2024-09-13T11:27:25.879 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:27:25.879 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:27:25.879 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:27:25.879 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:27:25.879 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.879 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:27:25.879 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:44.304480+0000", 2024-09-13T11:27:25.879 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:25.880 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.880 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.880 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:27:25.880 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:44.304480+0000", 2024-09-13T11:27:25.880 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:25.880 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.880 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.880 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:27:25.881 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:44.304481+0000", 2024-09-13T11:27:25.881 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.7349999999999999e-06 2024-09-13T11:27:25.881 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.881 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.881 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:27:25.881 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:44.304492+0000", 2024-09-13T11:27:25.881 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0394000000000001e-05 2024-09-13T11:27:25.881 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.881 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.882 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:27:25.882 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:44.675037+0000", 2024-09-13T11:27:25.882 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.37054519400000002 2024-09-13T11:27:25.882 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.882 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.882 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:27:25.882 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:44.675045+0000", 2024-09-13T11:27:25.882 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.1370000000000002e-06 2024-09-13T11:27:25.883 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.883 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.883 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:27:25.883 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:44.675334+0000", 2024-09-13T11:27:25.883 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00028848699999999998 2024-09-13T11:27:25.883 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.883 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.883 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:27:25.883 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:44.675345+0000", 2024-09-13T11:27:25.883 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.1198e-05 2024-09-13T11:27:25.883 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.883 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.883 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:27:25.884 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:44.675488+0000", 2024-09-13T11:27:25.884 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00014349600000000001 2024-09-13T11:27:25.884 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:25.884 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:27:25.884 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:25.884 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.884 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.884 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:217032 3.16 e187/88 3:6d28a27c:::benchmark_data_smithi177_31116_object76729:head v 187'8398, mlcod=187'8395)", 2024-09-13T11:27:25.884 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:25:59.836895+0000", 2024-09-13T11:27:25.884 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 85.977875955000002, 2024-09-13T11:27:25.884 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.22574986, 2024-09-13T11:27:25.884 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:27:25.884 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:27:25.884 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:27:25.885 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:27:25.885 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.885 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:27:25.885 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:59.836895+0000", 2024-09-13T11:27:25.885 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:25.885 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.885 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.885 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:27:25.885 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:59.836895+0000", 2024-09-13T11:27:25.885 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:25.885 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.885 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.885 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:27:25.886 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:59.836896+0000", 2024-09-13T11:27:25.886 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.432e-06 2024-09-13T11:27:25.886 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.886 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.886 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:27:25.886 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:25:59.836904+0000", 2024-09-13T11:27:25.886 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.9149999999999996e-06 2024-09-13T11:27:25.886 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.886 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.886 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:27:25.886 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:00.053571+0000", 2024-09-13T11:27:25.886 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.21666679799999999 2024-09-13T11:27:25.886 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.886 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.887 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:27:25.887 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:00.053582+0000", 2024-09-13T11:27:25.887 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0927999999999999e-05 2024-09-13T11:27:25.887 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.887 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.887 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:27:25.887 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:00.053811+0000", 2024-09-13T11:27:25.887 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00022964700000000001 2024-09-13T11:27:25.887 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.887 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.887 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:27:25.887 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:00.053864+0000", 2024-09-13T11:27:25.887 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.2710000000000002e-05 2024-09-13T11:27:25.887 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.888 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.888 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:27:25.888 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:00.062547+0000", 2024-09-13T11:27:25.888 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0086832650000000008 2024-09-13T11:27:25.888 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.888 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.888 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:27:25.888 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:00.062644+0000", 2024-09-13T11:27:25.888 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.7164999999999996e-05 2024-09-13T11:27:25.888 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:25.888 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:27:25.888 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:25.888 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.889 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.889 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4665.0:234789 3.5 3:a0df9e27:::benchmark_data_smithi177_31116_object94486:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e201)", 2024-09-13T11:27:25.889 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:26:17.919815+0000", 2024-09-13T11:27:25.889 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 67.894955809999999, 2024-09-13T11:27:25.889 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.34227359400000001, 2024-09-13T11:27:25.889 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:27:25.889 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:27:25.889 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:27:25.889 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-13T11:27:25.889 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4665", 2024-09-13T11:27:25.889 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:27:25.889 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 234789 2024-09-13T11:27:25.889 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.890 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:27:25.890 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.890 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:27:25.890 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:17.919815+0000", 2024-09-13T11:27:25.890 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:25.890 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.890 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.890 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:27:25.890 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:17.919815+0000", 2024-09-13T11:27:25.890 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:25.890 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.890 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.890 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:27:25.891 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:17.919817+0000", 2024-09-13T11:27:25.891 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.6369999999999999e-06 2024-09-13T11:27:25.891 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.891 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.891 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:27:25.891 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:17.919824+0000", 2024-09-13T11:27:25.891 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.9290000000000001e-06 2024-09-13T11:27:25.891 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.891 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.891 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:27:25.891 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:17.919839+0000", 2024-09-13T11:27:25.891 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.4763000000000001e-05 2024-09-13T11:27:25.891 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.891 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.892 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:27:25.892 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:17.919846+0000", 2024-09-13T11:27:25.892 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.1829999999999997e-06 2024-09-13T11:27:25.892 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.892 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.892 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:27:25.892 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:17.919894+0000", 2024-09-13T11:27:25.892 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.7475999999999997e-05 2024-09-13T11:27:25.892 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.892 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.892 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:27:25.892 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:17.919996+0000", 2024-09-13T11:27:25.892 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00010262199999999999 2024-09-13T11:27:25.893 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.893 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.893 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 13", 2024-09-13T11:27:25.893 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:17.920092+0000", 2024-09-13T11:27:25.893 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.5546999999999998e-05 2024-09-13T11:27:25.893 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.893 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.893 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-13T11:27:25.893 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:17.921865+0000", 2024-09-13T11:27:25.893 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00177271 2024-09-13T11:27:25.893 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.893 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.893 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:27:25.893 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:18.261907+0000", 2024-09-13T11:27:25.894 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.34004223300000003 2024-09-13T11:27:25.894 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.894 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.894 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:27:25.894 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:18.262002+0000", 2024-09-13T11:27:25.894 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.4745000000000002e-05 2024-09-13T11:27:25.894 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.894 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.894 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:27:25.894 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:18.262088+0000", 2024-09-13T11:27:25.894 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.6749000000000006e-05 2024-09-13T11:27:25.894 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:25.894 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:27:25.895 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:25.895 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.895 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.895 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop_reply(client.4665.0:234789 3.5 e201/117 ondisk, result = 0)", 2024-09-13T11:27:25.895 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:26:17.922332+0000", 2024-09-13T11:27:25.895 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 67.892438471999995, 2024-09-13T11:27:25.895 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.33978349299999999, 2024-09-13T11:27:25.895 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:27:25.895 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:27:25.895 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-13T11:27:25.895 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:27:25.895 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.895 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:27:25.896 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:17.922332+0000", 2024-09-13T11:27:25.896 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:25.896 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.896 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.896 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:27:25.896 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:17.922332+0000", 2024-09-13T11:27:25.896 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:25.896 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.896 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.896 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:27:25.896 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:17.922333+0000", 2024-09-13T11:27:25.896 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.2929999999999999e-06 2024-09-13T11:27:25.896 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.896 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.897 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:27:25.897 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:17.922337+0000", 2024-09-13T11:27:25.897 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.7469999999999999e-06 2024-09-13T11:27:25.897 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.897 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.897 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:27:25.897 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:18.261064+0000", 2024-09-13T11:27:25.897 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.33872685499999999 2024-09-13T11:27:25.897 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.897 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.897 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:27:25.897 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:18.261084+0000", 2024-09-13T11:27:25.897 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.9893000000000001e-05 2024-09-13T11:27:25.897 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.898 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.898 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:27:25.898 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:18.261880+0000", 2024-09-13T11:27:25.898 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00079579399999999995 2024-09-13T11:27:25.898 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.898 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.898 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:27:25.898 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:18.261897+0000", 2024-09-13T11:27:25.898 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.7730000000000001e-05 2024-09-13T11:27:25.898 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.898 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.898 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:27:25.898 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:18.262116+0000", 2024-09-13T11:27:25.899 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00021818099999999999 2024-09-13T11:27:25.899 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:25.899 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:27:25.899 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:25.899 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.899 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.899 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:260347 3.16 e219/88 3:6d3745c1:::benchmark_data_smithi177_31116_object120044:head v 219'9755, mlcod=219'9752)", 2024-09-13T11:27:25.899 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:26:42.471118+0000", 2024-09-13T11:27:25.899 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 43.343652878, 2024-09-13T11:27:25.899 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.43076131200000001, 2024-09-13T11:27:25.899 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:27:25.899 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:27:25.899 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:27:25.900 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:27:25.900 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.900 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:27:25.900 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:42.471118+0000", 2024-09-13T11:27:25.900 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:25.900 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.900 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.900 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:27:25.900 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:42.471118+0000", 2024-09-13T11:27:25.900 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:25.900 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.900 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.900 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:27:25.900 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:42.471122+0000", 2024-09-13T11:27:25.901 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.233e-06 2024-09-13T11:27:25.901 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.901 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.901 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:27:25.901 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:42.471139+0000", 2024-09-13T11:27:25.901 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.7442999999999999e-05 2024-09-13T11:27:25.901 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.901 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.901 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:27:25.901 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:42.893124+0000", 2024-09-13T11:27:25.901 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.42198470300000002 2024-09-13T11:27:25.901 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.901 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.902 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:27:25.902 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:42.893137+0000", 2024-09-13T11:27:25.902 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3233e-05 2024-09-13T11:27:25.902 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.902 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.902 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:27:25.902 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:42.893272+0000", 2024-09-13T11:27:25.902 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00013433499999999999 2024-09-13T11:27:25.902 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.902 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.902 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:27:25.902 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:42.893335+0000", 2024-09-13T11:27:25.902 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.3373000000000005e-05 2024-09-13T11:27:25.903 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.903 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.903 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:27:25.903 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:42.901776+0000", 2024-09-13T11:27:25.903 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0084411899999999995 2024-09-13T11:27:25.903 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.903 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.903 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:27:25.903 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:42.901879+0000", 2024-09-13T11:27:25.904 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000102802 2024-09-13T11:27:25.904 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:25.904 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:27:25.904 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:25.904 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.904 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.904 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:266456 3.5 e222/205 3:a058f943:::benchmark_data_smithi177_31116_object126153:head v 222'7143, mlcod=222'7139)", 2024-09-13T11:27:25.904 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:26:47.702034+0000", 2024-09-13T11:27:25.904 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 38.112736022999997, 2024-09-13T11:27:25.904 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.20573660499999999, 2024-09-13T11:27:25.904 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:27:25.904 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:27:25.905 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:27:25.905 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:27:25.905 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.905 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:27:25.905 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:47.702034+0000", 2024-09-13T11:27:25.905 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:25.905 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.905 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.905 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:27:25.905 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:47.702034+0000", 2024-09-13T11:27:25.905 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:25.906 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.906 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.906 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:27:25.906 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:47.702036+0000", 2024-09-13T11:27:25.906 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.265e-06 2024-09-13T11:27:25.906 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.906 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.906 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:27:25.906 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:47.702043+0000", 2024-09-13T11:27:25.906 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.9820000000000002e-06 2024-09-13T11:27:25.906 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.906 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.907 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:27:25.907 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:47.905566+0000", 2024-09-13T11:27:25.907 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.20352285000000001 2024-09-13T11:27:25.907 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.907 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.907 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:27:25.907 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:47.905575+0000", 2024-09-13T11:27:25.907 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.5570000000000001e-06 2024-09-13T11:27:25.907 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.907 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.907 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:27:25.908 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:47.905657+0000", 2024-09-13T11:27:25.908 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.2088000000000004e-05 2024-09-13T11:27:25.908 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.908 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.908 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:27:25.908 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:47.905676+0000", 2024-09-13T11:27:25.908 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.8561999999999999e-05 2024-09-13T11:27:25.908 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.908 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.908 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:27:25.909 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:47.907724+0000", 2024-09-13T11:27:25.909 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0020485239999999999 2024-09-13T11:27:25.909 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.909 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.909 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:27:25.909 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:47.907771+0000", 2024-09-13T11:27:25.909 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.6777000000000003e-05 2024-09-13T11:27:25.909 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:25.909 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:27:25.909 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:25.909 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.910 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.910 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:266464 3.5 e222/205 3:a2b966a1:::benchmark_data_smithi177_31116_object126161:head v 222'7144, mlcod=222'7140)", 2024-09-13T11:27:25.910 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:26:47.709996+0000", 2024-09-13T11:27:25.910 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 38.104774933999998, 2024-09-13T11:27:25.910 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.209382814, 2024-09-13T11:27:25.910 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:27:25.910 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:27:25.910 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:27:25.910 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:27:25.910 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.910 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:27:25.910 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:47.709996+0000", 2024-09-13T11:27:25.911 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:25.911 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.911 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.911 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:27:25.911 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:47.709996+0000", 2024-09-13T11:27:25.911 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:25.911 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.911 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.911 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:27:25.912 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:47.709997+0000", 2024-09-13T11:27:25.912 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.652e-06 2024-09-13T11:27:25.912 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.912 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.912 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:27:25.912 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:47.710006+0000", 2024-09-13T11:27:25.912 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.3480000000000004e-06 2024-09-13T11:27:25.912 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.912 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.912 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:27:25.912 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:47.917127+0000", 2024-09-13T11:27:25.913 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.20712151400000001 2024-09-13T11:27:25.913 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.913 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.913 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:27:25.913 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:47.917136+0000", 2024-09-13T11:27:25.913 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.3500000000000003e-06 2024-09-13T11:27:25.913 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.913 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.914 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:27:25.914 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:47.917204+0000", 2024-09-13T11:27:25.914 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.7793000000000006e-05 2024-09-13T11:27:25.914 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.914 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.914 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:27:25.914 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:47.917230+0000", 2024-09-13T11:27:25.914 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.5721e-05 2024-09-13T11:27:25.914 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.914 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.915 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:27:25.915 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:47.919340+0000", 2024-09-13T11:27:25.915 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0021102550000000001 2024-09-13T11:27:25.915 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.915 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.915 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:27:25.915 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:47.919378+0000", 2024-09-13T11:27:25.915 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.8180999999999999e-05 2024-09-13T11:27:25.915 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:25.916 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:27:25.916 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:25.916 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.916 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.916 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4665.0:266535 3.5 e222/205 3:a07a94a7:::benchmark_data_smithi177_31116_object126232:head v 222'7145, mlcod=222'7141)", 2024-09-13T11:27:25.916 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-13T11:26:47.759387+0000", 2024-09-13T11:27:25.916 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 38.055383716999998, 2024-09-13T11:27:25.916 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.23350773499999999, 2024-09-13T11:27:25.916 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-13T11:27:25.916 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-13T11:27:25.916 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:27:25.917 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-13T11:27:25.917 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.917 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-13T11:27:25.917 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:47.759387+0000", 2024-09-13T11:27:25.917 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:25.917 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.917 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.917 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-13T11:27:25.917 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:47.759387+0000", 2024-09-13T11:27:25.917 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-13T11:27:25.917 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.917 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.917 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-13T11:27:25.917 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:47.759389+0000", 2024-09-13T11:27:25.918 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.069e-06 2024-09-13T11:27:25.918 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.918 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.918 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-13T11:27:25.918 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:47.759397+0000", 2024-09-13T11:27:25.918 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.7780000000000007e-06 2024-09-13T11:27:25.918 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.918 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.918 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-13T11:27:25.918 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:47.990513+0000", 2024-09-13T11:27:25.918 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.23111675800000001 2024-09-13T11:27:25.918 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.918 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.919 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-13T11:27:25.919 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:47.990523+0000", 2024-09-13T11:27:25.919 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.3600000000000002e-06 2024-09-13T11:27:25.919 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.919 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.919 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-13T11:27:25.919 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:47.990649+0000", 2024-09-13T11:27:25.919 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00012615700000000001 2024-09-13T11:27:25.919 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.919 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.919 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-13T11:27:25.919 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:47.990671+0000", 2024-09-13T11:27:25.919 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.1934999999999999e-05 2024-09-13T11:27:25.919 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.919 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.920 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-13T11:27:25.920 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:47.992852+0000", 2024-09-13T11:27:25.920 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0021815060000000002 2024-09-13T11:27:25.920 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-13T11:27:25.920 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-13T11:27:25.920 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-13T11:27:25.920 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-13T11:26:47.992895+0000", 2024-09-13T11:27:25.920 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.2172000000000002e-05 2024-09-13T11:27:25.920 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:25.920 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:27:25.920 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:25.920 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-13T11:27:25.920 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-13T11:27:25.921 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-09-13T11:27:25.922 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-13T11:27:25.997 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-13T11:27:25.997 INFO:teuthology.orchestra.run.smithi162.stdout: "/var/lib/ceph/osd/ceph-10/block": { 2024-09-13T11:27:25.997 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_uuid": "acf4c64d-99af-42be-8406-413c86f54670", 2024-09-13T11:27:25.998 INFO:teuthology.orchestra.run.smithi162.stdout: "size": 96636764160, 2024-09-13T11:27:25.998 INFO:teuthology.orchestra.run.smithi162.stdout: "btime": "2024-09-13T11:21:26.338387+0000", 2024-09-13T11:27:25.998 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "main", 2024-09-13T11:27:25.998 INFO:teuthology.orchestra.run.smithi162.stdout: "bfm_blocks": "23592960", 2024-09-13T11:27:25.998 INFO:teuthology.orchestra.run.smithi162.stdout: "bfm_blocks_per_key": "128", 2024-09-13T11:27:25.998 INFO:teuthology.orchestra.run.smithi162.stdout: "bfm_bytes_per_block": "4096", 2024-09-13T11:27:25.998 INFO:teuthology.orchestra.run.smithi162.stdout: "bfm_size": "96636764160", 2024-09-13T11:27:25.998 INFO:teuthology.orchestra.run.smithi162.stdout: "bluefs": "1", 2024-09-13T11:27:25.998 INFO:teuthology.orchestra.run.smithi162.stdout: "ceph_fsid": "2fd68ac7-5581-4d90-80a4-30bcf5252a7a", 2024-09-13T11:27:25.998 INFO:teuthology.orchestra.run.smithi162.stdout: "ceph_version_when_created": "ceph version 19.3.0-4697-gd58f7306 (d58f73069a8157a9ace12a0247ee3e8497707ea9) squid (dev)", 2024-09-13T11:27:25.998 INFO:teuthology.orchestra.run.smithi162.stdout: "created_at": "2024-09-13T11:21:28.937108Z", 2024-09-13T11:27:25.998 INFO:teuthology.orchestra.run.smithi162.stdout: "elastic_shared_blobs": "1", 2024-09-13T11:27:25.998 INFO:teuthology.orchestra.run.smithi162.stdout: "epoch": "16", 2024-09-13T11:27:25.999 INFO:teuthology.orchestra.run.smithi162.stdout: "kv_backend": "rocksdb", 2024-09-13T11:27:25.999 INFO:teuthology.orchestra.run.smithi162.stdout: "magic": "ceph osd volume v026", 2024-09-13T11:27:25.999 INFO:teuthology.orchestra.run.smithi162.stdout: "multi": "yes", 2024-09-13T11:27:25.999 INFO:teuthology.orchestra.run.smithi162.stdout: "ready": "ready", 2024-09-13T11:27:25.999 INFO:teuthology.orchestra.run.smithi162.stdout: "require_osd_release": "19", 2024-09-13T11:27:25.999 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "bluestore", 2024-09-13T11:27:25.999 INFO:teuthology.orchestra.run.smithi162.stdout: "whoami": "10" 2024-09-13T11:27:25.999 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:25.999 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-13T11:27:26.000 INFO:tasks.thrashosds.thrasher:checking if target bluestore supports sharding on osd.10 2024-09-13T11:27:26.000 DEBUG:teuthology.orchestra.run.smithi162:> sudo ceph-bluestore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/bluestore_tool.$pid.log' --log-level=10 --path /var/lib/ceph/osd/ceph-10 show-sharding 2024-09-13T11:27:26.055 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-09-13T11:27:26.055 INFO:teuthology.orchestra.run.smithi092.stdout: "ops": [], 2024-09-13T11:27:26.055 INFO:teuthology.orchestra.run.smithi092.stdout: "num_ops": 0 2024-09-13T11:27:26.055 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-09-13T11:27:26.070 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-13T11:27:26.192 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-09-13T11:27:26.193 INFO:teuthology.orchestra.run.smithi092.stdout: "ops": [], 2024-09-13T11:27:26.193 INFO:teuthology.orchestra.run.smithi092.stdout: "complaint_time": 30, 2024-09-13T11:27:26.193 INFO:teuthology.orchestra.run.smithi092.stdout: "num_blocked_ops": 0 2024-09-13T11:27:26.193 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-09-13T11:27:26.207 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-13T11:27:26.332 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-09-13T11:27:26.333 INFO:teuthology.orchestra.run.smithi092.stdout: "size": 20, 2024-09-13T11:27:26.333 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 600, 2024-09-13T11:27:26.333 INFO:teuthology.orchestra.run.smithi092.stdout: "ops": [ 2024-09-13T11:27:26.333 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:26.333 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:3 3.4 3.125e09e4 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e40)", 2024-09-13T11:27:26.333 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:46.304641+0000", 2024-09-13T11:27:26.333 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 280.02608411099999, 2024-09-13T11:27:26.333 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.374372119, 2024-09-13T11:27:26.333 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:27:26.333 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:27:26.333 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "reached pg", 2024-09-13T11:27:26.333 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:27:26.334 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:27:26.334 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:27:26.334 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 3 2024-09-13T11:27:26.334 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:26.334 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:27:26.334 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:26.334 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:27:26.334 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.304641+0000", 2024-09-13T11:27:26.334 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:27:26.334 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:26.334 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:26.334 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:27:26.335 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.304641+0000", 2024-09-13T11:27:26.335 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:27:26.335 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:26.335 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:26.335 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:27:26.335 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.304644+0000", 2024-09-13T11:27:26.335 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.5130000000000001e-06 2024-09-13T11:27:26.335 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:26.335 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:26.335 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:27:26.335 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305058+0000", 2024-09-13T11:27:26.335 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000414277 2024-09-13T11:27:26.336 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:26.336 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:26.336 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:27:26.336 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305124+0000", 2024-09-13T11:27:26.336 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.5813000000000002e-05 2024-09-13T11:27:26.336 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:26.336 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:26.336 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:27:26.336 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305156+0000", 2024-09-13T11:27:26.336 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.1489000000000003e-05 2024-09-13T11:27:26.336 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:26.336 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:26.337 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:27:26.337 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.678939+0000", 2024-09-13T11:27:26.337 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.37378339999999999 2024-09-13T11:27:26.337 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:26.337 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:26.337 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:27:26.337 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.679013+0000", 2024-09-13T11:27:26.337 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.3627000000000005e-05 2024-09-13T11:27:26.337 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:27:26.337 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:27:26.337 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:27:26.337 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:26.338 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:26.338 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:29 3.4 3.b7bbfbe4 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-09-13T11:27:26.338 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:46.305215+0000", 2024-09-13T11:27:26.338 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 280.02550980799998, 2024-09-13T11:27:26.338 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.37387964600000001, 2024-09-13T11:27:26.338 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:27:26.338 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:27:26.338 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "reached pg", 2024-09-13T11:27:26.338 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:27:26.338 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:27:26.338 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:27:26.338 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 29 2024-09-13T11:27:26.338 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:26.339 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:27:26.339 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:26.339 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:27:26.339 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305215+0000", 2024-09-13T11:27:26.339 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:27:26.339 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:26.339 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:26.339 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:27:26.339 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305215+0000", 2024-09-13T11:27:26.339 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:27:26.339 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:26.339 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:26.340 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:27:26.340 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305218+0000", 2024-09-13T11:27:26.340 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.7030000000000002e-06 2024-09-13T11:27:26.340 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:26.340 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:26.340 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:27:26.340 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305366+0000", 2024-09-13T11:27:26.340 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00014825400000000001 2024-09-13T11:27:26.340 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:26.340 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:26.340 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:27:26.340 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305386+0000", 2024-09-13T11:27:26.340 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.9689000000000001e-05 2024-09-13T11:27:26.341 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:26.341 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:26.341 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:27:26.341 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305403+0000", 2024-09-13T11:27:26.341 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.7600000000000001e-05 2024-09-13T11:27:26.341 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:26.341 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:26.341 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:27:26.341 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.679067+0000", 2024-09-13T11:27:26.341 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.37366339100000001 2024-09-13T11:27:26.341 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:26.341 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:26.341 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:27:26.342 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.679095+0000", 2024-09-13T11:27:26.342 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.8008999999999999e-05 2024-09-13T11:27:26.342 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:27:26.342 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:27:26.342 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:27:26.342 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:26.342 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:26.342 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:30 3.4 3.547ec0f4 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-09-13T11:27:26.342 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:46.305453+0000", 2024-09-13T11:27:26.342 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 280.02527168400002, 2024-09-13T11:27:26.342 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.373764914, 2024-09-13T11:27:26.342 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:27:26.342 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:27:26.343 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "reached pg", 2024-09-13T11:27:26.343 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:27:26.343 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:27:26.343 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:27:26.343 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 30 2024-09-13T11:27:26.343 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:26.343 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:27:26.343 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:26.343 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:27:26.343 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305453+0000", 2024-09-13T11:27:26.343 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:27:26.343 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:26.343 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:26.344 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:27:26.344 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305453+0000", 2024-09-13T11:27:26.344 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:27:26.344 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:26.344 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:26.344 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:27:26.344 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305456+0000", 2024-09-13T11:27:26.344 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.9560000000000002e-06 2024-09-13T11:27:26.344 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:26.344 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:26.344 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:27:26.344 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305597+0000", 2024-09-13T11:27:26.345 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00014122700000000001 2024-09-13T11:27:26.345 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:26.345 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:26.345 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:27:26.345 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305615+0000", 2024-09-13T11:27:26.345 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.8139999999999999e-05 2024-09-13T11:27:26.345 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:26.345 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:26.345 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:27:26.345 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305632+0000", 2024-09-13T11:27:26.345 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.6101999999999999e-05 2024-09-13T11:27:26.346 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:26.346 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:26.346 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:27:26.346 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.679195+0000", 2024-09-13T11:27:26.346 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.373563486 2024-09-13T11:27:26.346 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:26.346 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:26.346 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:27:26.346 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.679218+0000", 2024-09-13T11:27:26.346 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.3003e-05 2024-09-13T11:27:26.346 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:27:26.346 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:27:26.347 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:27:26.347 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:26.347 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:26.347 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:70 3.4 3.1404a604 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-09-13T11:27:26.347 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:22:46.305665+0000", 2024-09-13T11:27:26.347 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 280.02505953600001, 2024-09-13T11:27:26.347 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.37365550199999997, 2024-09-13T11:27:26.347 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:27:26.347 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:27:26.347 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "reached pg", 2024-09-13T11:27:26.347 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:27:26.347 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:27:26.348 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:27:26.348 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 70 2024-09-13T11:27:26.348 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:26.348 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:27:26.348 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:26.348 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:27:26.348 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305665+0000", 2024-09-13T11:27:26.348 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:27:26.348 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:26.348 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:26.348 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:27:26.349 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305665+0000", 2024-09-13T11:27:26.349 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:27:26.349 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:26.349 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:26.349 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:27:26.349 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305668+0000", 2024-09-13T11:27:26.349 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.5890000000000001e-06 2024-09-13T11:27:26.349 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:26.349 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:26.349 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:27:26.349 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305791+0000", 2024-09-13T11:27:26.349 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000123545 2024-09-13T11:27:26.350 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:26.350 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:26.350 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:27:26.350 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305808+0000", 2024-09-13T11:27:26.350 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.668e-05 2024-09-13T11:27:26.350 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:26.350 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:26.350 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:27:26.350 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.305824+0000", 2024-09-13T11:27:26.350 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.6385e-05 2024-09-13T11:27:26.350 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:26.351 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:26.351 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:27:26.351 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.679305+0000", 2024-09-13T11:27:26.351 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.37348059900000002 2024-09-13T11:27:26.351 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:26.351 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:26.351 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:27:26.351 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:22:46.679321+0000", 2024-09-13T11:27:26.351 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.5704000000000001e-05 2024-09-13T11:27:26.351 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:27:26.351 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:27:26.351 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:27:26.351 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:26.352 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:26.352 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "MOSDPGPushReply(3.14 75/70 [PushReplyOp(3:2f14928a:::benchmark_data_smithi177_31116_object15123:head)])", 2024-09-13T11:27:26.352 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:31.569662+0000", 2024-09-13T11:27:26.352 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 234.76106311199999, 2024-09-13T11:27:26.352 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.35759148299999999, 2024-09-13T11:27:26.352 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:27:26.352 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:27:26.352 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "reached pg", 2024-09-13T11:27:26.352 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:27:26.352 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:26.352 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:27:26.353 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:31.569662+0000", 2024-09-13T11:27:26.353 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:27:26.353 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:26.353 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:26.353 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:27:26.353 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:31.569662+0000", 2024-09-13T11:27:26.353 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:27:26.353 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:26.353 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:26.353 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:27:26.353 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:31.569663+0000", 2024-09-13T11:27:26.354 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.333e-06 2024-09-13T11:27:26.354 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:26.354 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:26.354 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:27:26.354 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:31.569667+0000", 2024-09-13T11:27:26.354 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.8929999999999998e-06 2024-09-13T11:27:26.354 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:26.354 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:26.354 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:27:26.354 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:31.923221+0000", 2024-09-13T11:27:26.354 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.35355413699999999 2024-09-13T11:27:26.354 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:26.354 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:26.355 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:27:26.355 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:31.923230+0000", 2024-09-13T11:27:26.355 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.6829999999999997e-06 2024-09-13T11:27:26.355 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:26.355 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:26.355 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:27:26.355 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:31.923807+0000", 2024-09-13T11:27:26.355 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00057686800000000002 2024-09-13T11:27:26.355 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:26.355 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:26.355 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:27:26.355 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:31.927253+0000", 2024-09-13T11:27:26.356 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.003446569 2024-09-13T11:27:26.356 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:27:26.356 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:27:26.356 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:27:26.356 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:26.356 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:26.356 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "MOSDPGPushReply(3.18 85/70 [PushReplyOp(3:1e410119:::benchmark_data_smithi177_31116_object35700:head)])", 2024-09-13T11:27:26.356 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:42.216614+0000", 2024-09-13T11:27:26.356 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 224.11411076799999, 2024-09-13T11:27:26.356 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.54260090699999997, 2024-09-13T11:27:26.356 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:27:26.356 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:27:26.357 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "reached pg", 2024-09-13T11:27:26.357 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:27:26.357 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:26.357 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:27:26.357 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.216614+0000", 2024-09-13T11:27:26.357 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:27:26.357 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:26.357 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:26.357 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:27:26.357 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.216614+0000", 2024-09-13T11:27:26.357 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:27:26.357 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:26.357 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:26.357 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:27:26.358 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.216615+0000", 2024-09-13T11:27:26.358 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.0200000000000001e-07 2024-09-13T11:27:26.360 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:26.360 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:26.360 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:27:26.360 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.216619+0000", 2024-09-13T11:27:26.360 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.805e-06 2024-09-13T11:27:26.360 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:26.360 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:26.360 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:27:26.361 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.757476+0000", 2024-09-13T11:27:26.361 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.54085762000000004 2024-09-13T11:27:26.361 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:26.361 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:26.361 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:27:26.361 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.757498+0000", 2024-09-13T11:27:26.361 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.1936000000000001e-05 2024-09-13T11:27:26.361 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:26.361 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:26.361 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:27:26.361 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.757678+0000", 2024-09-13T11:27:26.361 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00018006599999999999 2024-09-13T11:27:26.362 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:26.362 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:26.362 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:27:26.362 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.759215+0000", 2024-09-13T11:27:26.362 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0015366780000000001 2024-09-13T11:27:26.362 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:27:26.362 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:27:26.362 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:27:26.362 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:26.362 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:26.362 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "MOSDPGPushReply(3.1b 85/78 [PushReplyOp(3:dabda1cc:::benchmark_data_smithi177_31116_object12988:head)])", 2024-09-13T11:27:26.362 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:42.485171+0000", 2024-09-13T11:27:26.363 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 223.84555386, 2024-09-13T11:27:26.363 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.308420427, 2024-09-13T11:27:26.363 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:27:26.363 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:27:26.363 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "reached pg", 2024-09-13T11:27:26.363 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:27:26.363 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:26.363 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:27:26.363 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.485171+0000", 2024-09-13T11:27:26.363 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:27:26.363 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:26.363 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:26.364 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:27:26.364 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.485171+0000", 2024-09-13T11:27:26.364 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:27:26.364 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:26.364 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:26.364 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:27:26.364 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.485173+0000", 2024-09-13T11:27:26.364 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.1569999999999998e-06 2024-09-13T11:27:26.364 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:26.364 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:26.364 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:27:26.364 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.485182+0000", 2024-09-13T11:27:26.365 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.6759999999999996e-06 2024-09-13T11:27:26.365 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:26.365 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:26.365 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:27:26.365 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.791336+0000", 2024-09-13T11:27:26.365 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.30615446499999999 2024-09-13T11:27:26.365 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:26.365 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:26.365 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:27:26.365 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.791369+0000", 2024-09-13T11:27:26.365 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.3155999999999999e-05 2024-09-13T11:27:26.365 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:26.366 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:26.366 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:27:26.366 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.791472+0000", 2024-09-13T11:27:26.366 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00010247900000000001 2024-09-13T11:27:26.366 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:26.366 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:26.366 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:27:26.366 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:42.793591+0000", 2024-09-13T11:27:26.366 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.002119494 2024-09-13T11:27:26.366 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:27:26.366 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:27:26.366 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:27:26.367 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:26.367 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:26.367 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:64066 3.1b 3:dd84e0ec:::benchmark_data_smithi177_31116_object64065:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89)", 2024-09-13T11:27:26.367 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:46.633608+0000", 2024-09-13T11:27:26.367 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 219.697116421, 2024-09-13T11:27:26.367 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.32564315399999999, 2024-09-13T11:27:26.367 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:27:26.367 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:27:26.367 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:27:26.367 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:27:26.367 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:27:26.367 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:27:26.367 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 64066 2024-09-13T11:27:26.368 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:26.368 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:27:26.368 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:26.368 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:27:26.368 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.633608+0000", 2024-09-13T11:27:26.368 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:27:26.368 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:26.368 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:26.368 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:27:26.368 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.633608+0000", 2024-09-13T11:27:26.369 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:27:26.369 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:26.369 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:26.369 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:27:26.369 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.633612+0000", 2024-09-13T11:27:26.369 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.856e-06 2024-09-13T11:27:26.369 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:26.369 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:26.369 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:27:26.369 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.633683+0000", 2024-09-13T11:27:26.369 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.1000999999999993e-05 2024-09-13T11:27:26.369 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:26.369 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:26.370 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:27:26.370 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.633711+0000", 2024-09-13T11:27:26.370 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.8204999999999998e-05 2024-09-13T11:27:26.370 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:26.370 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:26.370 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:27:26.370 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.633732+0000", 2024-09-13T11:27:26.370 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.0662999999999999e-05 2024-09-13T11:27:26.370 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:26.370 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:26.370 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:27:26.370 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.635573+0000", 2024-09-13T11:27:26.371 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.001841365 2024-09-13T11:27:26.371 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:26.371 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:26.371 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:27:26.371 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.635995+0000", 2024-09-13T11:27:26.371 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00042113699999999998 2024-09-13T11:27:26.371 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:26.371 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:26.371 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 12,15", 2024-09-13T11:27:26.371 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.636475+0000", 2024-09-13T11:27:26.371 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00048067699999999999 2024-09-13T11:27:26.371 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:26.371 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:26.372 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:27:26.372 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.640247+0000", 2024-09-13T11:27:26.372 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0037716450000000001 2024-09-13T11:27:26.372 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:26.372 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:26.372 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:27:26.372 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.649308+0000", 2024-09-13T11:27:26.372 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0090613240000000008 2024-09-13T11:27:26.372 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:26.372 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:26.372 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:27:26.372 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.956912+0000", 2024-09-13T11:27:26.372 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.30760338500000001 2024-09-13T11:27:26.373 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:26.373 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:26.373 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:27:26.373 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.957156+0000", 2024-09-13T11:27:26.373 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000244031 2024-09-13T11:27:26.373 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:26.373 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:26.373 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:27:26.373 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.959252+0000", 2024-09-13T11:27:26.373 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0020958650000000001 2024-09-13T11:27:26.373 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:27:26.373 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:27:26.373 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:27:26.374 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:26.374 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:26.374 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop_reply(client.4665.0:64066 3.1b e89/78 ondisk, result = 0)", 2024-09-13T11:27:26.374 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:46.651596+0000", 2024-09-13T11:27:26.374 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 219.67912859, 2024-09-13T11:27:26.374 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.30767612599999999, 2024-09-13T11:27:26.374 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:27:26.374 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:27:26.374 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-09-13T11:27:26.374 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:27:26.374 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:26.374 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:27:26.374 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.651596+0000", 2024-09-13T11:27:26.375 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:27:26.375 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:26.375 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:26.375 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:27:26.375 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.651596+0000", 2024-09-13T11:27:26.375 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:27:26.375 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:26.375 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:26.375 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:27:26.375 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.651600+0000", 2024-09-13T11:27:26.375 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.6969999999999999e-06 2024-09-13T11:27:26.375 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:26.375 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:26.375 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:27:26.376 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.651611+0000", 2024-09-13T11:27:26.376 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.1516e-05 2024-09-13T11:27:26.376 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:26.376 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:26.376 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:27:26.376 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.948118+0000", 2024-09-13T11:27:26.376 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.29650663500000002 2024-09-13T11:27:26.376 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:26.376 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:26.376 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:27:26.376 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.948137+0000", 2024-09-13T11:27:26.376 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.8529000000000002e-05 2024-09-13T11:27:26.376 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:26.377 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:26.377 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:27:26.377 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.956870+0000", 2024-09-13T11:27:26.377 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0087336380000000002 2024-09-13T11:27:26.377 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:26.377 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:26.377 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:27:26.377 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.956893+0000", 2024-09-13T11:27:26.377 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.317e-05 2024-09-13T11:27:26.377 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:26.377 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:26.377 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:27:26.378 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:46.959272+0000", 2024-09-13T11:27:26.378 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0023789409999999999 2024-09-13T11:27:26.378 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:27:26.378 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:27:26.378 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:27:26.378 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:26.378 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:26.378 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "MOSDPGPushReply(3.1b 91/78 [PushReplyOp(3:d9ac6889:::benchmark_data_smithi177_31116_object22254:head)])", 2024-09-13T11:27:26.378 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:23:48.392729+0000", 2024-09-13T11:27:26.378 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 217.937995675, 2024-09-13T11:27:26.378 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.30924993099999998, 2024-09-13T11:27:26.378 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:27:26.378 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:27:26.378 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "reached pg", 2024-09-13T11:27:26.379 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:27:26.379 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:26.379 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:27:26.379 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:48.392729+0000", 2024-09-13T11:27:26.379 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:27:26.379 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:26.379 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:26.379 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:27:26.379 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:48.392729+0000", 2024-09-13T11:27:26.379 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:27:26.379 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:26.379 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:26.379 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:27:26.380 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:48.392730+0000", 2024-09-13T11:27:26.380 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.336e-06 2024-09-13T11:27:26.380 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:26.380 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:26.380 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:27:26.380 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:48.392739+0000", 2024-09-13T11:27:26.380 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.8580000000000006e-06 2024-09-13T11:27:26.380 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:26.380 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:26.380 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:27:26.380 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:48.700180+0000", 2024-09-13T11:27:26.380 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.30744035600000003 2024-09-13T11:27:26.380 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:26.381 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:26.381 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:27:26.381 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:48.700197+0000", 2024-09-13T11:27:26.381 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.7649999999999999e-05 2024-09-13T11:27:26.381 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:26.381 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:26.381 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:27:26.381 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:48.700398+0000", 2024-09-13T11:27:26.381 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000201113 2024-09-13T11:27:26.381 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:26.381 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:26.381 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:27:26.381 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:23:48.701979+0000", 2024-09-13T11:27:26.382 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.001580618 2024-09-13T11:27:26.382 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:27:26.382 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:27:26.382 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:27:26.382 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:26.382 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:26.382 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop(client.4665.0:83193 3.8 e106/67 3:10066d83:::benchmark_data_smithi177_31116_object83192:head v 106'3925, mlcod=106'3922)", 2024-09-13T11:27:26.382 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:24:04.285068+0000", 2024-09-13T11:27:26.382 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 202.04565692, 2024-09-13T11:27:26.382 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.64647716600000005, 2024-09-13T11:27:26.382 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:27:26.382 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:27:26.382 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:27:26.382 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:27:26.383 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:26.383 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:27:26.383 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:04.285068+0000", 2024-09-13T11:27:26.383 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:27:26.383 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:26.383 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:26.383 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:27:26.383 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:04.285068+0000", 2024-09-13T11:27:26.383 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:27:26.383 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:26.383 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:26.383 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:27:26.383 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:04.285071+0000", 2024-09-13T11:27:26.384 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.729e-06 2024-09-13T11:27:26.384 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:26.384 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:26.384 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:27:26.384 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:04.285215+0000", 2024-09-13T11:27:26.384 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00014410000000000001 2024-09-13T11:27:26.384 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:26.384 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:26.384 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:27:26.384 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:04.928221+0000", 2024-09-13T11:27:26.384 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.64300653299999999 2024-09-13T11:27:26.384 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:26.384 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:26.385 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:27:26.385 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:04.928240+0000", 2024-09-13T11:27:26.385 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.8678999999999999e-05 2024-09-13T11:27:26.385 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:26.385 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:26.385 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:27:26.385 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:04.928388+0000", 2024-09-13T11:27:26.385 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00014809 2024-09-13T11:27:26.385 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:26.385 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:26.385 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:27:26.386 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:04.928428+0000", 2024-09-13T11:27:26.386 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.9776e-05 2024-09-13T11:27:26.386 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:26.386 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:26.386 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:27:26.386 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:04.931463+0000", 2024-09-13T11:27:26.386 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.003035168 2024-09-13T11:27:26.386 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:26.386 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:26.386 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:27:26.386 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:04.931545+0000", 2024-09-13T11:27:26.386 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.2090999999999997e-05 2024-09-13T11:27:26.387 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:27:26.387 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:27:26.387 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:27:26.387 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:26.387 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:26.389 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop(client.4665.0:83239 3.e e106/75 3:74547f17:::benchmark_data_smithi177_31116_object83238:head v 106'4391, mlcod=106'4388)", 2024-09-13T11:27:26.389 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:24:04.581196+0000", 2024-09-13T11:27:26.389 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 201.749528414, 2024-09-13T11:27:26.389 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.33617203600000001, 2024-09-13T11:27:26.389 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:27:26.389 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:27:26.389 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:27:26.389 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:27:26.389 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:26.389 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:27:26.389 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:04.581196+0000", 2024-09-13T11:27:26.389 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:27:26.390 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:26.390 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:26.390 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:27:26.390 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:04.581196+0000", 2024-09-13T11:27:26.390 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:27:26.390 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:26.390 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:26.390 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:27:26.390 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:04.581200+0000", 2024-09-13T11:27:26.390 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.19e-06 2024-09-13T11:27:26.390 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:26.390 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:26.390 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:27:26.391 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:04.581257+0000", 2024-09-13T11:27:26.391 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.7531999999999997e-05 2024-09-13T11:27:26.391 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:26.391 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:26.391 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:27:26.391 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:04.914011+0000", 2024-09-13T11:27:26.391 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.33275413700000001 2024-09-13T11:27:26.391 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:26.391 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:26.391 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:27:26.391 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:04.914028+0000", 2024-09-13T11:27:26.391 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.7082999999999999e-05 2024-09-13T11:27:26.392 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:26.392 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:26.392 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:27:26.392 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:04.914151+0000", 2024-09-13T11:27:26.392 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00012283300000000001 2024-09-13T11:27:26.392 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:26.392 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:26.392 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:27:26.392 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:04.914185+0000", 2024-09-13T11:27:26.392 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.4255999999999999e-05 2024-09-13T11:27:26.392 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:26.392 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:26.392 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:27:26.393 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:04.917300+0000", 2024-09-13T11:27:26.393 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0031141599999999999 2024-09-13T11:27:26.393 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:26.393 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:26.393 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:27:26.393 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:04.917368+0000", 2024-09-13T11:27:26.393 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.8844999999999996e-05 2024-09-13T11:27:26.393 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:27:26.393 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:27:26.393 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:27:26.393 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:26.393 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:26.393 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:131687 3.28 3:14f65078:::benchmark_data_smithi177_31116_object131686:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e135)", 2024-09-13T11:27:26.394 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:24:48.925565+0000", 2024-09-13T11:27:26.394 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 157.40515941699999, 2024-09-13T11:27:26.394 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.41421604099999998, 2024-09-13T11:27:26.394 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:27:26.394 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:27:26.394 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:27:26.394 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:27:26.394 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:27:26.394 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:27:26.394 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 131687 2024-09-13T11:27:26.394 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:26.394 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:27:26.394 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:26.394 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:27:26.395 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:48.925565+0000", 2024-09-13T11:27:26.395 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:27:26.395 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:26.395 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:26.395 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:27:26.395 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:48.925565+0000", 2024-09-13T11:27:26.395 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:27:26.395 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:26.395 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:26.395 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:27:26.395 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:48.925568+0000", 2024-09-13T11:27:26.395 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.441e-06 2024-09-13T11:27:26.395 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:26.395 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:26.396 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:27:26.396 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:48.925617+0000", 2024-09-13T11:27:26.396 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.8739000000000002e-05 2024-09-13T11:27:26.396 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:26.396 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:26.396 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:27:26.396 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:48.925631+0000", 2024-09-13T11:27:26.396 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.4463e-05 2024-09-13T11:27:26.396 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:26.396 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:26.396 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:27:26.396 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:48.925638+0000", 2024-09-13T11:27:26.396 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.2119999999999997e-06 2024-09-13T11:27:26.397 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:26.397 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:26.397 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:27:26.397 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:48.925703+0000", 2024-09-13T11:27:26.397 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.4794000000000003e-05 2024-09-13T11:27:26.397 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:26.397 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:26.397 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:27:26.397 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:48.925857+0000", 2024-09-13T11:27:26.397 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00015427 2024-09-13T11:27:26.397 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:26.397 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:26.398 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 5,8", 2024-09-13T11:27:26.398 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:48.925995+0000", 2024-09-13T11:27:26.398 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000137365 2024-09-13T11:27:26.398 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:26.398 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:26.398 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:27:26.398 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:48.927261+0000", 2024-09-13T11:27:26.398 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.001266021 2024-09-13T11:27:26.398 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:26.398 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:26.398 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:27:26.398 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:48.930015+0000", 2024-09-13T11:27:26.398 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0027545130000000001 2024-09-13T11:27:26.399 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:26.399 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:26.399 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:27:26.399 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:49.338898+0000", 2024-09-13T11:27:26.399 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.40888254299999999 2024-09-13T11:27:26.399 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:26.399 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:26.399 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:27:26.399 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:49.338967+0000", 2024-09-13T11:27:26.399 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.9402999999999999e-05 2024-09-13T11:27:26.399 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:26.399 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:26.399 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:27:26.400 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:49.339781+0000", 2024-09-13T11:27:26.400 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00081427699999999997 2024-09-13T11:27:26.400 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:27:26.400 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:27:26.400 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:27:26.400 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:26.400 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:26.400 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop_reply(client.4665.0:131687 3.28 e135/120 ondisk, result = 0)", 2024-09-13T11:27:26.400 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:24:48.928935+0000", 2024-09-13T11:27:26.400 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 157.401790116, 2024-09-13T11:27:26.400 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.41085638499999999, 2024-09-13T11:27:26.400 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:27:26.400 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:27:26.401 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-09-13T11:27:26.401 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:27:26.401 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:26.401 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:27:26.401 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:48.928935+0000", 2024-09-13T11:27:26.401 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:27:26.401 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:26.401 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:26.401 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:27:26.401 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:48.928935+0000", 2024-09-13T11:27:26.401 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:27:26.401 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:26.401 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:26.402 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:27:26.402 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:48.928936+0000", 2024-09-13T11:27:26.402 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.519e-06 2024-09-13T11:27:26.402 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:26.402 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:26.402 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:27:26.402 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:48.928941+0000", 2024-09-13T11:27:26.402 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.0710000000000002e-06 2024-09-13T11:27:26.402 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:26.402 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:26.402 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:27:26.402 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:49.338799+0000", 2024-09-13T11:27:26.403 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.40985808499999998 2024-09-13T11:27:26.403 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:26.403 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:26.403 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:27:26.403 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:49.338809+0000", 2024-09-13T11:27:26.403 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.4390000000000006e-06 2024-09-13T11:27:26.403 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:26.403 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:26.403 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:27:26.403 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:49.338878+0000", 2024-09-13T11:27:26.403 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.9429000000000002e-05 2024-09-13T11:27:26.403 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:26.403 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:26.404 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:27:26.404 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:49.338890+0000", 2024-09-13T11:27:26.404 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.2217e-05 2024-09-13T11:27:26.404 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:26.404 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:26.404 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:27:26.404 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:24:49.339791+0000", 2024-09-13T11:27:26.404 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00090062499999999995 2024-09-13T11:27:26.404 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:27:26.404 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:27:26.404 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:27:26.404 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:26.404 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:26.405 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.4665.0:181661 3.4 3:2019f47a:::benchmark_data_smithi177_31116_object41358:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e170)", 2024-09-13T11:27:26.405 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:25:31.131054+0000", 2024-09-13T11:27:26.405 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 115.19967035000001, 2024-09-13T11:27:26.405 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.44761152399999998, 2024-09-13T11:27:26.405 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:27:26.405 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:27:26.405 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:27:26.405 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-09-13T11:27:26.405 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.4665", 2024-09-13T11:27:26.405 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:27:26.405 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 181661 2024-09-13T11:27:26.405 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:26.405 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:27:26.406 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:26.406 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:27:26.406 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:31.131054+0000", 2024-09-13T11:27:26.406 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:27:26.406 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:26.406 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:26.406 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:27:26.406 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:31.131054+0000", 2024-09-13T11:27:26.406 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:27:26.406 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:26.406 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:26.406 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:27:26.406 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:31.131056+0000", 2024-09-13T11:27:26.406 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.8640000000000001e-06 2024-09-13T11:27:26.407 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:26.407 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:26.407 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:27:26.407 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:31.131061+0000", 2024-09-13T11:27:26.407 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.6249999999999998e-06 2024-09-13T11:27:26.407 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:26.407 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:26.407 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:27:26.407 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:31.131069+0000", 2024-09-13T11:27:26.407 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.1049999999999995e-06 2024-09-13T11:27:26.407 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:26.407 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:26.407 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:27:26.408 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:31.131074+0000", 2024-09-13T11:27:26.408 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.344e-06 2024-09-13T11:27:26.408 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:26.408 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:26.408 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:27:26.408 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:31.131121+0000", 2024-09-13T11:27:26.408 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.6794e-05 2024-09-13T11:27:26.408 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:26.408 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:26.408 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:27:26.408 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:31.131225+0000", 2024-09-13T11:27:26.408 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000103502 2024-09-13T11:27:26.408 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:26.409 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:26.409 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 12", 2024-09-13T11:27:26.409 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:31.131289+0000", 2024-09-13T11:27:26.409 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.4790999999999997e-05 2024-09-13T11:27:26.409 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:26.409 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:26.409 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-09-13T11:27:26.409 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:31.133191+0000", 2024-09-13T11:27:26.409 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0019013859999999999 2024-09-13T11:27:26.409 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:26.409 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:26.409 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:27:26.409 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:31.578585+0000", 2024-09-13T11:27:26.410 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.44539403 2024-09-13T11:27:26.410 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:26.410 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:26.410 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:27:26.410 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:31.578620+0000", 2024-09-13T11:27:26.410 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.4657999999999999e-05 2024-09-13T11:27:26.410 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:26.410 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:26.410 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:27:26.410 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:31.578666+0000", 2024-09-13T11:27:26.410 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.6424999999999997e-05 2024-09-13T11:27:26.410 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:27:26.410 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:27:26.411 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:27:26.411 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:26.411 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:26.411 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop_reply(client.4665.0:181661 3.4 e170/117 ondisk, result = 0)", 2024-09-13T11:27:26.411 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:25:31.135385+0000", 2024-09-13T11:27:26.411 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 115.19533953, 2024-09-13T11:27:26.411 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.44328512599999997, 2024-09-13T11:27:26.411 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:27:26.411 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:27:26.411 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-09-13T11:27:26.411 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:27:26.411 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:26.411 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:27:26.412 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:31.135385+0000", 2024-09-13T11:27:26.412 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:27:26.412 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:26.412 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:26.412 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:27:26.412 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:31.135385+0000", 2024-09-13T11:27:26.412 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:27:26.412 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:26.412 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:26.412 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:27:26.412 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:31.135387+0000", 2024-09-13T11:27:26.412 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.217e-06 2024-09-13T11:27:26.412 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:26.413 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:26.413 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:27:26.413 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:31.135391+0000", 2024-09-13T11:27:26.413 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.0820000000000001e-06 2024-09-13T11:27:26.413 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:26.413 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:26.413 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:27:26.413 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:31.578504+0000", 2024-09-13T11:27:26.413 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.44311382100000002 2024-09-13T11:27:26.413 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:26.413 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:26.413 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:27:26.413 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:31.578530+0000", 2024-09-13T11:27:26.413 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.5367e-05 2024-09-13T11:27:26.414 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:26.414 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:26.414 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:27:26.414 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:31.578576+0000", 2024-09-13T11:27:26.414 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.5840999999999997e-05 2024-09-13T11:27:26.414 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:26.414 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:26.414 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:27:26.414 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:31.578581+0000", 2024-09-13T11:27:26.414 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.5620000000000003e-06 2024-09-13T11:27:26.414 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:26.414 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:26.414 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:27:26.414 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:31.578670+0000", 2024-09-13T11:27:26.415 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.9235999999999997e-05 2024-09-13T11:27:26.415 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:27:26.415 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:27:26.415 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:27:26.415 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:26.415 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:26.415 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop(client.4665.0:184026 3.b e171/117 3:d3052347:::benchmark_data_smithi177_31116_object43723:head v 171'6273, mlcod=171'6269)", 2024-09-13T11:27:26.415 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:25:32.645695+0000", 2024-09-13T11:27:26.415 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 113.68502979599999, 2024-09-13T11:27:26.415 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.34822244200000002, 2024-09-13T11:27:26.415 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:27:26.415 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:27:26.415 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:27:26.416 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:27:26.416 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:26.416 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:27:26.416 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:32.645695+0000", 2024-09-13T11:27:26.416 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:27:26.416 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:26.416 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:26.416 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:27:26.416 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:32.645695+0000", 2024-09-13T11:27:26.416 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:27:26.416 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:26.416 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:26.416 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:27:26.417 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:32.645696+0000", 2024-09-13T11:27:26.417 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.4780000000000001e-06 2024-09-13T11:27:26.417 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:26.417 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:26.417 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:27:26.417 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:32.645704+0000", 2024-09-13T11:27:26.417 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.1980000000000004e-06 2024-09-13T11:27:26.417 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:26.417 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:26.417 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:27:26.417 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:32.992228+0000", 2024-09-13T11:27:26.417 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.346523992 2024-09-13T11:27:26.418 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:26.418 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:26.418 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:27:26.418 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:32.992232+0000", 2024-09-13T11:27:26.418 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.3109999999999999e-06 2024-09-13T11:27:26.418 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:26.418 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:26.418 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:27:26.418 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:32.992275+0000", 2024-09-13T11:27:26.418 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.2589999999999997e-05 2024-09-13T11:27:26.418 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:26.418 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:26.418 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:27:26.419 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:32.992289+0000", 2024-09-13T11:27:26.419 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.4116e-05 2024-09-13T11:27:26.419 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:26.419 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:26.419 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:27:26.419 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:32.993877+0000", 2024-09-13T11:27:26.419 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0015879259999999999 2024-09-13T11:27:26.419 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:26.419 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:26.419 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:27:26.419 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:32.993917+0000", 2024-09-13T11:27:26.419 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.0831000000000003e-05 2024-09-13T11:27:26.420 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:27:26.420 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:27:26.420 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:27:26.420 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:26.420 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:26.420 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop(client.4665.0:184072 3.b e171/117 3:d319cf93:::benchmark_data_smithi177_31116_object43769:head v 171'6274, mlcod=171'6270)", 2024-09-13T11:27:26.420 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:25:32.701516+0000", 2024-09-13T11:27:26.420 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 113.629209108, 2024-09-13T11:27:26.420 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.30463955399999998, 2024-09-13T11:27:26.420 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:27:26.420 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:27:26.420 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:27:26.420 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:27:26.421 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:26.421 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:27:26.421 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:32.701516+0000", 2024-09-13T11:27:26.421 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:27:26.421 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:26.421 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:26.421 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:27:26.421 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:32.701516+0000", 2024-09-13T11:27:26.421 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:27:26.421 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:26.421 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:26.421 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:27:26.421 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:32.701519+0000", 2024-09-13T11:27:26.422 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.0029999999999999e-06 2024-09-13T11:27:26.422 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:26.422 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:26.422 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:27:26.422 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:32.701530+0000", 2024-09-13T11:27:26.422 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.1759999999999999e-05 2024-09-13T11:27:26.422 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:26.422 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:26.422 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:27:26.422 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:33.003551+0000", 2024-09-13T11:27:26.422 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.302020819 2024-09-13T11:27:26.422 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:26.422 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:26.423 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:27:26.423 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:33.003558+0000", 2024-09-13T11:27:26.423 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.2299999999999996e-06 2024-09-13T11:27:26.423 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:26.423 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:26.423 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:27:26.423 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:33.003630+0000", 2024-09-13T11:27:26.423 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.2125000000000005e-05 2024-09-13T11:27:26.423 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:26.423 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:26.423 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:27:26.423 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:33.003648+0000", 2024-09-13T11:27:26.423 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.8672e-05 2024-09-13T11:27:26.424 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:26.424 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:26.424 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-09-13T11:27:26.424 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:33.006114+0000", 2024-09-13T11:27:26.424 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0024656790000000001 2024-09-13T11:27:26.424 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:26.424 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:26.424 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:27:26.424 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:25:33.006155+0000", 2024-09-13T11:27:26.424 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.1266000000000001e-05 2024-09-13T11:27:26.424 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:27:26.424 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:27:26.424 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:27:26.424 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:26.425 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:26.425 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "MOSDPGRecoveryDeleteReply(3.39 e242,238 3:9f6b69ce:::benchmark_data_smithi177_31116_object27023:head,160'6505)", 2024-09-13T11:27:26.425 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:27:11.545923+0000", 2024-09-13T11:27:26.425 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 14.784801999999999, 2024-09-13T11:27:26.425 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.50434392299999997, 2024-09-13T11:27:26.425 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:27:26.425 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:27:26.425 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "reached pg", 2024-09-13T11:27:26.425 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:27:26.425 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:26.425 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:27:26.425 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:27:11.545923+0000", 2024-09-13T11:27:26.425 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:27:26.425 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:26.426 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:26.426 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:27:26.426 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:27:11.545923+0000", 2024-09-13T11:27:26.426 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:27:26.426 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:26.426 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:26.426 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:27:26.426 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:27:11.545924+0000", 2024-09-13T11:27:26.426 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.4020000000000001e-06 2024-09-13T11:27:26.426 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:26.426 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:26.426 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:27:26.427 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:27:11.545930+0000", 2024-09-13T11:27:26.427 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.4580000000000001e-06 2024-09-13T11:27:26.427 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:26.427 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:26.427 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:27:26.427 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:27:12.049823+0000", 2024-09-13T11:27:26.427 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.50389338100000003 2024-09-13T11:27:26.427 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:26.427 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:26.427 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:27:26.427 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:27:12.049834+0000", 2024-09-13T11:27:26.427 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.1199999999999999e-05 2024-09-13T11:27:26.427 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:26.428 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:26.428 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:27:26.428 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:27:12.049897+0000", 2024-09-13T11:27:26.428 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.3225000000000005e-05 2024-09-13T11:27:26.428 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:26.428 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:26.428 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:27:26.428 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:27:12.050267+0000", 2024-09-13T11:27:26.428 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00036925700000000001 2024-09-13T11:27:26.428 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:27:26.428 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:27:26.428 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:27:26.429 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:26.429 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:26.429 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "MOSDPGRecoveryDelete(3.18 e250,243 3:1a1fbdd1:::benchmark_data_smithi177_31116_object25415:head,158'6524)", 2024-09-13T11:27:26.429 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-09-13T11:27:21.214305+0000", 2024-09-13T11:27:26.429 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 5.1164197580000002, 2024-09-13T11:27:26.429 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.74344127699999996, 2024-09-13T11:27:26.429 INFO:teuthology.orchestra.run.smithi092.stdout: "continuous": false, 2024-09-13T11:27:26.429 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-09-13T11:27:26.429 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-09-13T11:27:26.429 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-09-13T11:27:26.429 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:26.429 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-09-13T11:27:26.429 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:27:21.214305+0000", 2024-09-13T11:27:26.429 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:27:26.430 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:26.430 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:26.430 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-09-13T11:27:26.430 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:27:21.214305+0000", 2024-09-13T11:27:26.430 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-09-13T11:27:26.430 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:26.430 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:26.430 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-09-13T11:27:26.430 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:27:21.214307+0000", 2024-09-13T11:27:26.430 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.7120000000000001e-06 2024-09-13T11:27:26.430 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:26.430 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:26.430 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-09-13T11:27:26.431 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:27:21.214313+0000", 2024-09-13T11:27:26.431 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.0850000000000002e-06 2024-09-13T11:27:26.431 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:26.431 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:26.431 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-09-13T11:27:26.431 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:27:21.957528+0000", 2024-09-13T11:27:26.431 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.74321472799999999 2024-09-13T11:27:26.431 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:26.431 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:26.431 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-09-13T11:27:26.431 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:27:21.957534+0000", 2024-09-13T11:27:26.431 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.4989999999999999e-06 2024-09-13T11:27:26.432 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:26.432 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:26.432 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-09-13T11:27:26.432 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:27:21.957616+0000", 2024-09-13T11:27:26.432 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.2083999999999995e-05 2024-09-13T11:27:26.432 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:26.432 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:26.432 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-09-13T11:27:26.432 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:27:21.957628+0000", 2024-09-13T11:27:26.432 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.1814000000000001e-05 2024-09-13T11:27:26.432 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-09-13T11:27:26.432 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-09-13T11:27:26.433 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-09-13T11:27:26.433 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-09-13T11:27:21.957746+0000", 2024-09-13T11:27:26.433 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000118355 2024-09-13T11:27:26.433 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:27:26.433 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:27:26.433 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:27:26.433 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-09-13T11:27:26.433 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-09-13T11:27:26.433 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-09-13T11:27:26.435 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_ops_in_flight 2024-09-13T11:27:26.559 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-13T11:27:26.561 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [], 2024-09-13T11:27:26.561 INFO:teuthology.orchestra.run.smithi162.stdout: "num_ops": 0 2024-09-13T11:27:26.561 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-13T11:27:26.574 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_blocked_ops 2024-09-13T11:27:26.704 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-13T11:27:26.704 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [], 2024-09-13T11:27:26.704 INFO:teuthology.orchestra.run.smithi162.stdout: "complaint_time": 30, 2024-09-13T11:27:26.705 INFO:teuthology.orchestra.run.smithi162.stdout: "num_blocked_ops": 0 2024-09-13T11:27:26.705 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-13T11:27:26.720 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_historic_ops 2024-09-13T11:27:26.840 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-13T11:27:26.840 INFO:teuthology.orchestra.run.smithi162.stdout: "size": 20, 2024-09-13T11:27:26.840 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 600, 2024-09-13T11:27:26.840 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [ 2024-09-13T11:27:26.841 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:26.841 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:221526 3.2 3.ea98b382 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e191)", 2024-09-13T11:27:26.841 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:26:04.295546+0000", 2024-09-13T11:27:26.841 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 82.544227946000007, 2024-09-13T11:27:26.841 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.73989113200000001, 2024-09-13T11:27:26.841 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:27:26.841 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:27:26.841 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-09-13T11:27:26.841 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:27:26.841 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:27:26.841 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:27:26.841 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 221526 2024-09-13T11:27:26.841 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:26.842 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:27:26.842 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:26.842 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:27:26.842 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:04.295546+0000", 2024-09-13T11:27:26.842 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:26.842 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:26.842 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:26.842 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:27:26.842 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:04.295546+0000", 2024-09-13T11:27:26.842 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:26.842 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:26.842 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:26.842 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:27:26.843 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:04.295551+0000", 2024-09-13T11:27:26.843 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.6940000000000003e-06 2024-09-13T11:27:26.843 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:26.843 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:26.843 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:27:26.843 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:04.295567+0000", 2024-09-13T11:27:26.843 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5896000000000002e-05 2024-09-13T11:27:26.843 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:26.843 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:26.843 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:27:26.843 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:04.295599+0000", 2024-09-13T11:27:26.843 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.243e-05 2024-09-13T11:27:26.843 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:26.844 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:26.844 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:27:26.844 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:04.295622+0000", 2024-09-13T11:27:26.844 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.3149000000000001e-05 2024-09-13T11:27:26.844 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:26.844 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:26.844 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:27:26.844 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:04.295764+0000", 2024-09-13T11:27:26.844 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00014193300000000001 2024-09-13T11:27:26.844 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:26.844 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:26.844 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-09-13T11:27:26.844 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:04.295778+0000", 2024-09-13T11:27:26.845 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3763e-05 2024-09-13T11:27:26.845 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:26.845 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:26.845 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:27:26.845 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:05.035420+0000", 2024-09-13T11:27:26.845 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.73964228300000001 2024-09-13T11:27:26.845 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:26.845 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:26.845 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:27:26.845 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:05.035437+0000", 2024-09-13T11:27:26.845 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.6983999999999999e-05 2024-09-13T11:27:26.845 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:26.845 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:27:26.846 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:26.846 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:26.846 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:26.846 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:221528 3.15 3.b7a45bf5 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e191)", 2024-09-13T11:27:26.846 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:26:04.296174+0000", 2024-09-13T11:27:26.846 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 82.543600300999998, 2024-09-13T11:27:26.846 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.73909410399999997, 2024-09-13T11:27:26.846 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:27:26.846 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:27:26.846 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-09-13T11:27:26.846 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:27:26.846 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:27:26.846 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:27:26.847 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 221528 2024-09-13T11:27:26.847 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:26.847 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:27:26.847 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:26.847 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:27:26.847 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:04.296174+0000", 2024-09-13T11:27:26.847 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:26.847 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:26.847 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:26.847 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:27:26.847 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:04.296174+0000", 2024-09-13T11:27:26.847 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:26.847 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:26.848 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:26.848 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:27:26.848 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:04.296188+0000", 2024-09-13T11:27:26.848 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4037e-05 2024-09-13T11:27:26.848 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:26.848 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:26.848 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:27:26.848 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:04.296201+0000", 2024-09-13T11:27:26.848 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3096e-05 2024-09-13T11:27:26.848 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:26.848 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:26.848 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:27:26.848 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:04.296242+0000", 2024-09-13T11:27:26.849 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.1103000000000003e-05 2024-09-13T11:27:26.849 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:26.849 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:26.849 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:27:26.849 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:04.296264+0000", 2024-09-13T11:27:26.849 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.173e-05 2024-09-13T11:27:26.849 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:26.849 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:26.849 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:27:26.849 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:04.296410+0000", 2024-09-13T11:27:26.850 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00014574600000000001 2024-09-13T11:27:26.850 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:26.850 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:26.850 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-09-13T11:27:26.850 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:04.296424+0000", 2024-09-13T11:27:26.850 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3926e-05 2024-09-13T11:27:26.850 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:26.850 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:26.850 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:27:26.850 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:05.035221+0000", 2024-09-13T11:27:26.850 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.73879784999999998 2024-09-13T11:27:26.850 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:26.850 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:26.851 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:27:26.851 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:05.035268+0000", 2024-09-13T11:27:26.851 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.6616000000000002e-05 2024-09-13T11:27:26.851 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:26.851 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:27:26.851 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:26.851 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:26.851 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:26.851 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:221544 3.12 3.ec3fbdf2 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e191)", 2024-09-13T11:27:26.851 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:26:04.384452+0000", 2024-09-13T11:27:26.851 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 82.455321709000003, 2024-09-13T11:27:26.852 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.65065541999999998, 2024-09-13T11:27:26.852 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:27:26.852 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:27:26.852 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-09-13T11:27:26.852 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:27:26.852 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:27:26.852 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:27:26.852 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 221544 2024-09-13T11:27:26.853 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:26.853 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:27:26.853 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:26.853 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:27:26.853 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:04.384452+0000", 2024-09-13T11:27:26.853 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:26.853 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:26.853 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:26.853 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:27:26.853 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:04.384452+0000", 2024-09-13T11:27:26.853 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:26.853 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:26.854 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:26.854 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:27:26.854 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:04.384457+0000", 2024-09-13T11:27:26.854 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.6469999999999997e-06 2024-09-13T11:27:26.854 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:26.854 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:26.854 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:27:26.854 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:04.384470+0000", 2024-09-13T11:27:26.854 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3186999999999999e-05 2024-09-13T11:27:26.855 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:26.855 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:26.855 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:27:26.855 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:04.384504+0000", 2024-09-13T11:27:26.855 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.3413e-05 2024-09-13T11:27:26.855 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:26.855 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:26.856 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:27:26.856 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:04.384526+0000", 2024-09-13T11:27:26.856 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.2690000000000001e-05 2024-09-13T11:27:26.856 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:26.856 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:26.856 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:27:26.856 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:04.384653+0000", 2024-09-13T11:27:26.856 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00012697199999999999 2024-09-13T11:27:26.856 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:26.856 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:26.856 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-09-13T11:27:26.856 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:04.384668+0000", 2024-09-13T11:27:26.857 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4936000000000001e-05 2024-09-13T11:27:26.857 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:26.857 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:26.857 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:27:26.857 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:05.035087+0000", 2024-09-13T11:27:26.857 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.65041902900000004 2024-09-13T11:27:26.857 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:26.857 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:26.857 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:27:26.858 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:05.035108+0000", 2024-09-13T11:27:26.858 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.0545999999999999e-05 2024-09-13T11:27:26.858 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:26.858 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:27:26.858 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:26.858 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:26.858 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:26.858 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:234164 3.22 3.d472cf62 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e201)", 2024-09-13T11:27:26.858 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:26:17.191424+0000", 2024-09-13T11:27:26.858 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 69.648350590000007, 2024-09-13T11:27:26.858 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.95096563000000001, 2024-09-13T11:27:26.859 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:27:26.859 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:27:26.859 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-09-13T11:27:26.859 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:27:26.859 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:27:26.859 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:27:26.859 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 234164 2024-09-13T11:27:26.859 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:26.859 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:27:26.860 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:26.860 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:27:26.860 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:17.191424+0000", 2024-09-13T11:27:26.860 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:26.860 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:26.860 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:26.860 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:27:26.860 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:17.191424+0000", 2024-09-13T11:27:26.861 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:26.861 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:26.861 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:26.861 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:27:26.861 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:17.191426+0000", 2024-09-13T11:27:26.861 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.3269999999999999e-06 2024-09-13T11:27:26.861 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:26.861 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:26.862 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:27:26.862 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:17.191434+0000", 2024-09-13T11:27:26.862 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.5490000000000006e-06 2024-09-13T11:27:26.862 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:26.862 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:26.862 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:27:26.862 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:17.191450+0000", 2024-09-13T11:27:26.862 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.596e-05 2024-09-13T11:27:26.862 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:26.862 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:26.862 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:27:26.862 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:17.191458+0000", 2024-09-13T11:27:26.863 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.6070000000000002e-06 2024-09-13T11:27:26.863 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:26.863 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:26.863 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:27:26.863 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:17.191518+0000", 2024-09-13T11:27:26.863 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.9488999999999999e-05 2024-09-13T11:27:26.863 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:26.863 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:26.863 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-09-13T11:27:26.863 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:17.191526+0000", 2024-09-13T11:27:26.863 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.8049999999999996e-06 2024-09-13T11:27:26.863 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:26.863 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:26.863 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:27:26.864 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:18.142341+0000", 2024-09-13T11:27:26.864 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.95081438299999999 2024-09-13T11:27:26.864 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:26.864 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:26.864 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:27:26.864 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:18.142389+0000", 2024-09-13T11:27:26.864 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.8510000000000001e-05 2024-09-13T11:27:26.864 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:26.864 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:27:26.864 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:26.864 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:26.864 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:26.864 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:234214 3.22 3.3ebf73a2 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e201)", 2024-09-13T11:27:26.865 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:26:17.239598+0000", 2024-09-13T11:27:26.865 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 69.600176318999999, 2024-09-13T11:27:26.865 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.90310166300000005, 2024-09-13T11:27:26.865 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:27:26.865 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:27:26.865 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-09-13T11:27:26.865 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:27:26.865 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:27:26.865 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:27:26.865 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 234214 2024-09-13T11:27:26.865 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:26.865 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:27:26.865 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:26.866 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:27:26.866 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:17.239598+0000", 2024-09-13T11:27:26.866 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:26.866 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:26.866 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:26.866 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:27:26.866 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:17.239598+0000", 2024-09-13T11:27:26.866 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:26.866 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:26.866 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:26.866 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:27:26.866 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:17.239601+0000", 2024-09-13T11:27:26.866 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.021e-06 2024-09-13T11:27:26.867 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:26.867 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:26.867 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:27:26.867 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:17.239609+0000", 2024-09-13T11:27:26.867 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.9459999999999998e-06 2024-09-13T11:27:26.867 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:26.867 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:26.867 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:27:26.867 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:17.239628+0000", 2024-09-13T11:27:26.867 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.9029e-05 2024-09-13T11:27:26.867 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:26.867 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:26.867 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:27:26.868 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:17.239637+0000", 2024-09-13T11:27:26.868 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.7530000000000007e-06 2024-09-13T11:27:26.868 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:26.868 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:26.868 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:27:26.868 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:17.239716+0000", 2024-09-13T11:27:26.868 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.9789000000000005e-05 2024-09-13T11:27:26.868 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:26.868 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:26.868 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-09-13T11:27:26.868 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:17.239729+0000", 2024-09-13T11:27:26.868 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2215000000000001e-05 2024-09-13T11:27:26.868 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:26.869 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:26.869 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:27:26.869 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:18.142672+0000", 2024-09-13T11:27:26.869 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.90294309299999997 2024-09-13T11:27:26.869 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:26.869 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:26.869 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:27:26.869 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:18.142700+0000", 2024-09-13T11:27:26.869 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.7817000000000002e-05 2024-09-13T11:27:26.869 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:26.869 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:27:26.869 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:26.869 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:26.869 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:26.870 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:234350 3.22 3.6c9f6f62 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e201)", 2024-09-13T11:27:26.870 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:26:17.412568+0000", 2024-09-13T11:27:26.870 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 69.427206519999999, 2024-09-13T11:27:26.870 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.73023408300000003, 2024-09-13T11:27:26.870 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:27:26.870 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:27:26.870 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-09-13T11:27:26.870 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:27:26.870 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:27:26.870 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:27:26.870 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 234350 2024-09-13T11:27:26.870 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:26.870 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:27:26.871 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:26.871 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:27:26.871 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:17.412568+0000", 2024-09-13T11:27:26.871 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:26.871 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:26.871 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:26.871 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:27:26.871 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:17.412568+0000", 2024-09-13T11:27:26.871 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:26.871 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:26.871 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:26.871 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:27:26.871 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:17.412571+0000", 2024-09-13T11:27:26.871 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.044e-06 2024-09-13T11:27:26.872 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:26.872 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:26.872 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:27:26.872 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:17.412579+0000", 2024-09-13T11:27:26.872 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.7639999999999994e-06 2024-09-13T11:27:26.872 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:26.872 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:26.872 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:27:26.872 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:17.412595+0000", 2024-09-13T11:27:26.872 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5637999999999998e-05 2024-09-13T11:27:26.872 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:26.872 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:26.872 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:27:26.873 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:17.412606+0000", 2024-09-13T11:27:26.873 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1175999999999999e-05 2024-09-13T11:27:26.873 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:26.873 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:26.873 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:27:26.873 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:17.412690+0000", 2024-09-13T11:27:26.873 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.3486000000000007e-05 2024-09-13T11:27:26.873 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:26.873 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:26.873 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-09-13T11:27:26.873 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:17.412703+0000", 2024-09-13T11:27:26.873 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3196e-05 2024-09-13T11:27:26.873 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:26.873 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:26.874 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:27:26.874 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:18.142767+0000", 2024-09-13T11:27:26.874 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.73006412600000004 2024-09-13T11:27:26.874 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:26.874 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:26.874 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:27:26.874 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:18.142802+0000", 2024-09-13T11:27:26.874 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.4653000000000002e-05 2024-09-13T11:27:26.874 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:26.874 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:27:26.874 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:26.874 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:26.874 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:26.875 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGRecoveryDelete(3.6 e201,191 3:60767a7b:::benchmark_data_smithi177_31116_object14927:head,151'5598)", 2024-09-13T11:27:26.875 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:26:17.890539+0000", 2024-09-13T11:27:26.875 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 68.949235298000005, 2024-09-13T11:27:26.875 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.29203489500000002, 2024-09-13T11:27:26.875 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:27:26.875 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:27:26.875 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:27:26.875 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:27:26.875 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:26.875 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:27:26.875 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:17.890539+0000", 2024-09-13T11:27:26.875 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:26.875 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:26.875 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:26.876 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:27:26.876 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:17.890539+0000", 2024-09-13T11:27:26.876 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:26.876 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:26.876 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:26.876 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:27:26.876 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:17.890541+0000", 2024-09-13T11:27:26.876 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.795e-06 2024-09-13T11:27:26.876 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:26.876 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:26.876 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:27:26.876 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:17.890547+0000", 2024-09-13T11:27:26.876 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.3770000000000001e-06 2024-09-13T11:27:26.877 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:26.877 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:26.877 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:27:26.877 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:18.182175+0000", 2024-09-13T11:27:26.877 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.29162787000000001 2024-09-13T11:27:26.877 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:26.877 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:26.877 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:27:26.877 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:18.182190+0000", 2024-09-13T11:27:26.877 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4976e-05 2024-09-13T11:27:26.877 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:26.877 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:26.877 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:27:26.878 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:18.182397+0000", 2024-09-13T11:27:26.878 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00020751499999999999 2024-09-13T11:27:26.878 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:26.878 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:26.878 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:27:26.878 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:18.182425+0000", 2024-09-13T11:27:26.878 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.7076999999999998e-05 2024-09-13T11:27:26.878 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:26.878 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:26.878 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:27:26.878 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:18.182574+0000", 2024-09-13T11:27:26.878 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000149285 2024-09-13T11:27:26.878 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:26.878 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:27:26.879 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:26.879 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:26.879 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:26.879 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:234778 3.22 3.36a6aba2 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e201)", 2024-09-13T11:27:26.879 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:26:17.909436+0000", 2024-09-13T11:27:26.879 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 68.930338316000004, 2024-09-13T11:27:26.879 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.233461425, 2024-09-13T11:27:26.879 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:27:26.879 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:27:26.879 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-09-13T11:27:26.879 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:27:26.879 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:27:26.879 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:27:26.879 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 234778 2024-09-13T11:27:26.880 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:26.880 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:27:26.880 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:26.880 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:27:26.880 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:17.909436+0000", 2024-09-13T11:27:26.880 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:26.880 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:26.880 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:26.880 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:27:26.880 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:17.909436+0000", 2024-09-13T11:27:26.880 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:26.880 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:26.880 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:26.881 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:27:26.881 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:17.909439+0000", 2024-09-13T11:27:26.881 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.717e-06 2024-09-13T11:27:26.881 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:26.881 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:26.881 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:27:26.881 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:17.909446+0000", 2024-09-13T11:27:26.881 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.1980000000000004e-06 2024-09-13T11:27:26.881 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:26.881 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:26.881 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:27:26.881 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:17.909462+0000", 2024-09-13T11:27:26.881 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5809999999999999e-05 2024-09-13T11:27:26.881 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:26.882 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:26.882 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:27:26.882 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:17.909471+0000", 2024-09-13T11:27:26.882 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.3100000000000006e-06 2024-09-13T11:27:26.882 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:26.882 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:26.882 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:27:26.882 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:17.909554+0000", 2024-09-13T11:27:26.882 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.2917000000000006e-05 2024-09-13T11:27:26.882 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:26.883 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:26.883 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-09-13T11:27:26.883 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:17.909569+0000", 2024-09-13T11:27:26.883 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5201000000000001e-05 2024-09-13T11:27:26.883 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:26.883 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:26.883 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:27:26.883 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:18.142863+0000", 2024-09-13T11:27:26.883 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.23329387900000001 2024-09-13T11:27:26.883 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:26.883 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:26.883 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:27:26.884 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:18.142897+0000", 2024-09-13T11:27:26.884 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.4393000000000001e-05 2024-09-13T11:27:26.884 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:26.884 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:27:26.884 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:26.884 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:26.884 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:26.884 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPush(3.2 201/192 [PushOp(3:409d6ad2:::benchmark_data_smithi177_31116_object97365:head, version: 113'3767, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:409d6ad2:::benchmark_data_smithi177_31116_object97365:head@113'3767, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:27:26.884 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:26:17.942627+0000", 2024-09-13T11:27:26.884 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 68.897147645000004, 2024-09-13T11:27:26.885 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.25342641300000002, 2024-09-13T11:27:26.885 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:27:26.885 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:27:26.885 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:27:26.885 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:27:26.885 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:26.885 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:27:26.885 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:17.942627+0000", 2024-09-13T11:27:26.885 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:26.886 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:26.886 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:26.886 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:27:26.886 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:17.942627+0000", 2024-09-13T11:27:26.886 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:26.886 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:26.886 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:26.886 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:27:26.886 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:17.942628+0000", 2024-09-13T11:27:26.886 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3179999999999999e-06 2024-09-13T11:27:26.886 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:26.886 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:26.886 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:27:26.887 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:17.942665+0000", 2024-09-13T11:27:26.887 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.6748999999999997e-05 2024-09-13T11:27:26.887 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:26.887 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:26.887 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:27:26.887 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:18.192821+0000", 2024-09-13T11:27:26.887 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.25015650900000003 2024-09-13T11:27:26.887 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:26.887 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:26.888 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:27:26.888 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:18.192853+0000", 2024-09-13T11:27:26.888 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.1579999999999999e-05 2024-09-13T11:27:26.888 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:26.888 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:26.888 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:27:26.888 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:18.193147+0000", 2024-09-13T11:27:26.888 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000294424 2024-09-13T11:27:26.888 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:26.889 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:26.889 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:27:26.889 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:18.193191+0000", 2024-09-13T11:27:26.889 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.3463e-05 2024-09-13T11:27:26.889 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:26.889 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:26.889 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:27:26.889 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:18.196053+0000", 2024-09-13T11:27:26.889 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0028623699999999999 2024-09-13T11:27:26.889 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:26.889 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:27:26.890 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:26.890 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:26.890 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:26.890 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGRecoveryDelete(3.22 e223,218 3:47eb4e8b:::benchmark_data_smithi177_31116_object75722:head,187'5647)", 2024-09-13T11:27:26.890 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:26:48.562782+0000", 2024-09-13T11:27:26.890 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 38.276991963, 2024-09-13T11:27:26.890 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.307749461, 2024-09-13T11:27:26.890 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:27:26.890 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:27:26.890 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:27:26.891 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:27:26.891 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:26.891 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:27:26.891 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:48.562782+0000", 2024-09-13T11:27:26.891 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:26.891 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:26.891 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:26.891 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:27:26.891 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:48.562782+0000", 2024-09-13T11:27:26.891 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:26.891 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:26.892 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:26.892 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:27:26.892 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:48.562785+0000", 2024-09-13T11:27:26.892 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.9900000000000002e-06 2024-09-13T11:27:26.892 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:26.892 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:26.892 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:27:26.892 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:48.562796+0000", 2024-09-13T11:27:26.892 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1046999999999999e-05 2024-09-13T11:27:26.893 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:26.893 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:26.893 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:27:26.893 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:48.862609+0000", 2024-09-13T11:27:26.893 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.29981280399999999 2024-09-13T11:27:26.893 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:26.893 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:26.893 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:27:26.894 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:48.862624+0000", 2024-09-13T11:27:26.894 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5214e-05 2024-09-13T11:27:26.894 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:26.894 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:26.894 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:27:26.894 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:48.870361+0000", 2024-09-13T11:27:26.894 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0077370670000000002 2024-09-13T11:27:26.894 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:26.894 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:26.895 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:27:26.895 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:48.870385+0000", 2024-09-13T11:27:26.895 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.3693000000000002e-05 2024-09-13T11:27:26.895 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:26.895 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:26.895 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:27:26.895 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:48.870532+0000", 2024-09-13T11:27:26.895 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00014664600000000001 2024-09-13T11:27:26.895 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:26.895 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:27:26.895 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:26.895 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:26.895 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:26.896 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:266847 3.15 3:aced3906:::benchmark_data_smithi177_31116_object126544:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e223)", 2024-09-13T11:27:26.896 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:26:48.588305+0000", 2024-09-13T11:27:26.896 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 38.251468774999999, 2024-09-13T11:27:26.896 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.29333894599999999, 2024-09-13T11:27:26.896 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:27:26.896 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:27:26.896 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:27:26.896 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:27:26.896 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:27:26.896 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:27:26.896 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 266847 2024-09-13T11:27:26.896 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:26.897 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:27:26.897 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:26.897 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:27:26.897 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:48.588305+0000", 2024-09-13T11:27:26.897 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:26.897 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:26.897 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:26.897 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:27:26.897 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:48.588305+0000", 2024-09-13T11:27:26.897 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:26.897 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:26.897 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:26.897 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:27:26.897 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:48.588309+0000", 2024-09-13T11:27:26.898 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.41e-06 2024-09-13T11:27:26.898 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:26.898 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:26.898 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:27:26.898 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:48.588319+0000", 2024-09-13T11:27:26.898 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.9359999999999994e-06 2024-09-13T11:27:26.898 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:26.898 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:26.898 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:27:26.898 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:48.588348+0000", 2024-09-13T11:27:26.898 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.8940000000000001e-05 2024-09-13T11:27:26.898 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:26.898 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:26.898 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:27:26.899 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:48.588362+0000", 2024-09-13T11:27:26.899 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4752999999999999e-05 2024-09-13T11:27:26.899 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:26.899 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:26.899 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:27:26.899 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:48.588510+0000", 2024-09-13T11:27:26.899 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00014728500000000001 2024-09-13T11:27:26.899 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:26.899 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:26.899 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:27:26.899 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:48.588731+0000", 2024-09-13T11:27:26.899 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000220762 2024-09-13T11:27:26.899 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:26.900 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:26.900 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 0", 2024-09-13T11:27:26.900 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:48.588906+0000", 2024-09-13T11:27:26.900 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00017593900000000001 2024-09-13T11:27:26.900 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:26.900 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:26.900 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:27:26.900 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:48.594579+0000", 2024-09-13T11:27:26.900 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.005672351 2024-09-13T11:27:26.900 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:26.900 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:26.900 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:27:26.900 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:48.881375+0000", 2024-09-13T11:27:26.901 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.28679606600000002 2024-09-13T11:27:26.901 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:26.901 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:26.901 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:27:26.901 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:48.881524+0000", 2024-09-13T11:27:26.901 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000148935 2024-09-13T11:27:26.901 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:26.901 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:26.901 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:27:26.901 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:48.881644+0000", 2024-09-13T11:27:26.901 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000120569 2024-09-13T11:27:26.901 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:26.901 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:27:26.902 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:26.902 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:26.902 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:26.902 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.4665.0:266884 3.1f e224/223 3:fb121a9e:::benchmark_data_smithi177_31116_object126581:head v 224'9867, mlcod=224'9865)", 2024-09-13T11:27:26.902 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:26:49.276350+0000", 2024-09-13T11:27:26.902 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 37.563424126999998, 2024-09-13T11:27:26.902 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.53992503999999997, 2024-09-13T11:27:26.902 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:27:26.902 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:27:26.902 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:27:26.902 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:27:26.902 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:26.902 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:27:26.902 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:49.276350+0000", 2024-09-13T11:27:26.902 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:26.903 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:26.903 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:26.903 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:27:26.903 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:49.276350+0000", 2024-09-13T11:27:26.903 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:26.903 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:26.903 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:26.903 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:27:26.903 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:49.276352+0000", 2024-09-13T11:27:26.903 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.6020000000000001e-06 2024-09-13T11:27:26.903 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:26.903 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:26.903 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:27:26.903 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:49.276360+0000", 2024-09-13T11:27:26.904 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.6319999999999997e-06 2024-09-13T11:27:26.904 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:26.904 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:26.904 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:27:26.904 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:49.810598+0000", 2024-09-13T11:27:26.904 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.53423792199999998 2024-09-13T11:27:26.904 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:26.904 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:26.904 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:27:26.904 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:49.810616+0000", 2024-09-13T11:27:26.904 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.7456999999999999e-05 2024-09-13T11:27:26.904 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:26.904 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:26.905 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:27:26.905 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:49.810770+0000", 2024-09-13T11:27:26.905 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00015476199999999999 2024-09-13T11:27:26.905 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:26.905 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:26.905 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:27:26.905 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:49.810838+0000", 2024-09-13T11:27:26.905 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.7788999999999998e-05 2024-09-13T11:27:26.905 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:26.905 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:26.905 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:27:26.905 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:49.816185+0000", 2024-09-13T11:27:26.905 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0053467369999999998 2024-09-13T11:27:26.905 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:26.906 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:26.906 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:27:26.906 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:26:49.816275+0000", 2024-09-13T11:27:26.906 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.0138999999999999e-05 2024-09-13T11:27:26.906 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:26.906 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:27:26.906 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:26.906 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:26.906 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:26.906 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:277975 3.3f 3.e0a6dd7f (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e239)", 2024-09-13T11:27:26.906 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:27:06.991878+0000", 2024-09-13T11:27:26.906 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 19.847896025000001, 2024-09-13T11:27:26.906 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.011507309, 2024-09-13T11:27:26.907 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:27:26.907 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:27:26.907 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-09-13T11:27:26.907 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:27:26.907 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:27:26.907 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:27:26.907 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 277975 2024-09-13T11:27:26.907 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:26.907 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:27:26.907 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:26.907 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:27:26.907 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:27:06.991878+0000", 2024-09-13T11:27:26.907 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:26.908 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:26.908 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:26.908 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:27:26.908 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:27:06.991878+0000", 2024-09-13T11:27:26.908 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:26.908 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:26.908 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:26.908 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:27:26.908 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:27:06.991884+0000", 2024-09-13T11:27:26.908 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.677e-06 2024-09-13T11:27:26.908 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:26.908 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:26.908 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:27:26.908 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:27:06.991899+0000", 2024-09-13T11:27:26.909 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5271999999999999e-05 2024-09-13T11:27:26.909 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:26.909 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:26.909 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:27:26.909 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:27:06.991934+0000", 2024-09-13T11:27:26.909 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.4754e-05 2024-09-13T11:27:26.909 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:26.909 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:26.909 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:27:26.909 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:27:06.991951+0000", 2024-09-13T11:27:26.909 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.7076999999999999e-05 2024-09-13T11:27:26.909 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:26.909 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:26.909 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:27:26.910 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:27:06.992111+0000", 2024-09-13T11:27:26.910 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000159875 2024-09-13T11:27:26.910 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:26.910 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:26.910 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op must wait for map", 2024-09-13T11:27:26.910 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:27:06.992127+0000", 2024-09-13T11:27:26.910 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.6334e-05 2024-09-13T11:27:26.910 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:26.910 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:26.910 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:27:26.910 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:27:07.010082+0000", 2024-09-13T11:27:26.910 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.017955314999999999 2024-09-13T11:27:26.910 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:26.910 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:26.911 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-09-13T11:27:26.911 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:27:07.010102+0000", 2024-09-13T11:27:26.911 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.9425000000000001e-05 2024-09-13T11:27:26.911 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:26.911 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:26.911 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:27:26.911 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:27:08.003314+0000", 2024-09-13T11:27:26.911 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.99321246799999996 2024-09-13T11:27:26.911 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:26.911 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:26.911 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:27:26.911 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:27:08.003385+0000", 2024-09-13T11:27:26.911 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.1112000000000006e-05 2024-09-13T11:27:26.912 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:26.912 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:27:26.912 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:26.912 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:26.912 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:26.912 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:278046 3.3f 3.10b6dbff (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e239)", 2024-09-13T11:27:26.912 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:27:06.991985+0000", 2024-09-13T11:27:26.912 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 19.84778888, 2024-09-13T11:27:26.912 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0116638360000001, 2024-09-13T11:27:26.912 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:27:26.912 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:27:26.912 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-09-13T11:27:26.912 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:27:26.913 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:27:26.913 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:27:26.913 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 278046 2024-09-13T11:27:26.913 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:26.913 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:27:26.913 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:26.913 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:27:26.913 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:27:06.991985+0000", 2024-09-13T11:27:26.913 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:26.913 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:26.913 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:26.913 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:27:26.913 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:27:06.991985+0000", 2024-09-13T11:27:26.914 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:26.914 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:26.914 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:26.914 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:27:26.914 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:27:06.991987+0000", 2024-09-13T11:27:26.914 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5939999999999999e-06 2024-09-13T11:27:26.914 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:26.914 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:26.914 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:27:26.914 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:27:06.991998+0000", 2024-09-13T11:27:26.914 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1195e-05 2024-09-13T11:27:26.914 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:26.914 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:26.914 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:27:26.915 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:27:06.992043+0000", 2024-09-13T11:27:26.915 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.4827999999999998e-05 2024-09-13T11:27:26.915 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:26.915 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:26.915 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:27:26.915 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:27:06.992065+0000", 2024-09-13T11:27:26.915 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.2544999999999998e-05 2024-09-13T11:27:26.915 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:26.915 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:26.915 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:27:26.916 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:27:06.992225+0000", 2024-09-13T11:27:26.916 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00015911300000000001 2024-09-13T11:27:26.916 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:26.916 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:26.916 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting_for_map not empty", 2024-09-13T11:27:26.916 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:27:06.992242+0000", 2024-09-13T11:27:26.916 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.7025999999999999e-05 2024-09-13T11:27:26.916 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:26.916 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:26.916 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:27:26.916 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:27:07.010420+0000", 2024-09-13T11:27:26.916 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.018178577000000001 2024-09-13T11:27:26.917 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:26.917 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:26.917 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-09-13T11:27:26.917 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:27:07.010436+0000", 2024-09-13T11:27:26.917 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.6101999999999999e-05 2024-09-13T11:27:26.917 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:26.917 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:26.917 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:27:26.917 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:27:08.003629+0000", 2024-09-13T11:27:26.917 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.99319262399999997 2024-09-13T11:27:26.918 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:26.918 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:26.918 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:27:26.918 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:27:08.003649+0000", 2024-09-13T11:27:26.918 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.0231999999999999e-05 2024-09-13T11:27:26.918 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:26.918 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:27:26.918 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:26.918 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:26.918 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:26.918 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:278067 3.3f 3.9fc82f3f (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e239)", 2024-09-13T11:27:26.919 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:27:06.992123+0000", 2024-09-13T11:27:26.919 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 19.847651635999998, 2024-09-13T11:27:26.919 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.011590293, 2024-09-13T11:27:26.919 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:27:26.919 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:27:26.919 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-09-13T11:27:26.919 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:27:26.919 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:27:26.919 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:27:26.919 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 278067 2024-09-13T11:27:26.919 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:26.919 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:27:26.919 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:26.919 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:27:26.920 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:27:06.992123+0000", 2024-09-13T11:27:26.920 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:26.920 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:26.920 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:26.920 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:27:26.920 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:27:06.992123+0000", 2024-09-13T11:27:26.920 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:26.920 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:26.920 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:26.920 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:27:26.920 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:27:06.992125+0000", 2024-09-13T11:27:26.920 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.644e-06 2024-09-13T11:27:26.920 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:26.921 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:26.921 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:27:26.921 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:27:06.992136+0000", 2024-09-13T11:27:26.921 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0961999999999999e-05 2024-09-13T11:27:26.921 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:26.921 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:26.921 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:27:26.921 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:27:06.992159+0000", 2024-09-13T11:27:26.921 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.2934000000000001e-05 2024-09-13T11:27:26.921 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:26.921 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:26.921 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:27:26.921 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:27:06.992176+0000", 2024-09-13T11:27:26.922 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.6518e-05 2024-09-13T11:27:26.922 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:26.922 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:26.922 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:27:26.922 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:27:06.992297+0000", 2024-09-13T11:27:26.922 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00012105299999999999 2024-09-13T11:27:26.922 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:26.922 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:26.922 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting_for_map not empty", 2024-09-13T11:27:26.922 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:27:06.992304+0000", 2024-09-13T11:27:26.922 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.9079999999999998e-06 2024-09-13T11:27:26.922 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:26.922 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:26.922 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:27:26.923 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:27:07.010485+0000", 2024-09-13T11:27:26.923 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.018181342999999999 2024-09-13T11:27:26.923 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:26.923 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:26.923 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-09-13T11:27:26.923 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:27:07.010498+0000", 2024-09-13T11:27:26.923 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2914999999999999e-05 2024-09-13T11:27:26.923 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:26.923 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:26.923 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:27:26.923 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:27:08.003699+0000", 2024-09-13T11:27:26.923 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.99320087000000001 2024-09-13T11:27:26.924 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:26.924 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:26.924 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:27:26.924 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:27:08.003713+0000", 2024-09-13T11:27:26.924 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4146e-05 2024-09-13T11:27:26.924 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:26.924 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:27:26.924 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:26.924 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:26.924 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:26.924 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:278101 3.3f 3.cbc5fa3f (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e239)", 2024-09-13T11:27:26.924 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:27:06.992204+0000", 2024-09-13T11:27:26.924 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 19.847570233999999, 2024-09-13T11:27:26.925 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0115713200000001, 2024-09-13T11:27:26.925 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:27:26.925 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:27:26.925 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-09-13T11:27:26.925 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:27:26.925 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:27:26.925 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:27:26.925 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 278101 2024-09-13T11:27:26.925 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:26.925 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:27:26.925 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:26.925 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:27:26.926 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:27:06.992204+0000", 2024-09-13T11:27:26.926 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:26.926 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:26.926 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:26.926 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:27:26.926 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:27:06.992204+0000", 2024-09-13T11:27:26.926 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:26.926 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:26.926 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:26.927 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:27:26.927 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:27:06.992206+0000", 2024-09-13T11:27:26.927 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5999999999999999e-06 2024-09-13T11:27:26.927 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:26.927 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:26.927 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:27:26.927 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:27:06.992212+0000", 2024-09-13T11:27:26.927 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.2759999999999997e-06 2024-09-13T11:27:26.927 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:26.928 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:26.928 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:27:26.928 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:27:06.992224+0000", 2024-09-13T11:27:26.928 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1963e-05 2024-09-13T11:27:26.928 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:26.928 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:26.928 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:27:26.928 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:27:06.992235+0000", 2024-09-13T11:27:26.928 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1262e-05 2024-09-13T11:27:26.928 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:26.928 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:26.929 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:27:26.929 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:27:06.992354+0000", 2024-09-13T11:27:26.929 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000118713 2024-09-13T11:27:26.929 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:26.929 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:26.929 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting_for_map not empty", 2024-09-13T11:27:26.929 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:27:06.992361+0000", 2024-09-13T11:27:26.929 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.7719999999999997e-06 2024-09-13T11:27:26.930 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:26.930 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:26.930 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:27:26.930 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:27:07.010539+0000", 2024-09-13T11:27:26.930 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.018178659999999999 2024-09-13T11:27:26.930 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:26.930 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:26.930 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-09-13T11:27:26.930 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:27:07.010559+0000", 2024-09-13T11:27:26.930 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.9638000000000001e-05 2024-09-13T11:27:26.931 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:26.931 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:26.931 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:27:26.931 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:27:08.003761+0000", 2024-09-13T11:27:26.931 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.99320253000000003 2024-09-13T11:27:26.931 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:26.931 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:26.931 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:27:26.931 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:27:08.003775+0000", 2024-09-13T11:27:26.932 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3906e-05 2024-09-13T11:27:26.932 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:26.932 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:27:26.932 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:26.932 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:26.932 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:26.932 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:278107 3.3f 3.24b842bf (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e239)", 2024-09-13T11:27:26.932 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:27:06.992250+0000", 2024-09-13T11:27:26.932 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 19.847524191000002, 2024-09-13T11:27:26.932 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.011585081, 2024-09-13T11:27:26.932 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:27:26.932 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:27:26.932 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-09-13T11:27:26.933 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:27:26.933 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:27:26.933 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:27:26.933 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 278107 2024-09-13T11:27:26.933 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:26.933 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:27:26.933 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:26.933 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:27:26.933 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:27:06.992250+0000", 2024-09-13T11:27:26.933 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:26.933 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:26.933 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:26.933 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:27:26.933 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:27:06.992250+0000", 2024-09-13T11:27:26.934 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:26.934 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:26.934 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:26.934 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:27:26.934 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:27:06.992251+0000", 2024-09-13T11:27:26.934 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4139999999999999e-06 2024-09-13T11:27:26.934 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:26.934 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:26.934 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:27:26.934 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:27:06.992257+0000", 2024-09-13T11:27:26.934 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.4639999999999997e-06 2024-09-13T11:27:26.934 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:26.934 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:26.934 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:27:26.935 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:27:06.992267+0000", 2024-09-13T11:27:26.935 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0518e-05 2024-09-13T11:27:26.935 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:26.935 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:26.935 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:27:26.935 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:27:06.992278+0000", 2024-09-13T11:27:26.935 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0180000000000001e-05 2024-09-13T11:27:26.935 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:26.935 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:26.935 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:27:26.935 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:27:06.992397+0000", 2024-09-13T11:27:26.935 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00011923500000000001 2024-09-13T11:27:26.935 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:26.936 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:26.936 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting_for_map not empty", 2024-09-13T11:27:26.936 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:27:06.992403+0000", 2024-09-13T11:27:26.936 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.4740000000000002e-06 2024-09-13T11:27:26.936 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:26.936 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:26.936 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:27:26.936 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:27:07.010605+0000", 2024-09-13T11:27:26.936 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.018201697999999999 2024-09-13T11:27:26.936 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:26.936 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:26.936 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-09-13T11:27:26.936 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:27:07.010617+0000", 2024-09-13T11:27:26.936 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.154e-05 2024-09-13T11:27:26.937 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:26.937 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:26.937 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:27:26.937 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:27:08.003823+0000", 2024-09-13T11:27:26.937 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.99320655300000005 2024-09-13T11:27:26.937 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:26.937 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:26.937 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:27:26.937 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:27:08.003835+0000", 2024-09-13T11:27:26.937 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2004999999999999e-05 2024-09-13T11:27:26.937 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:26.937 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:27:26.938 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:26.938 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:26.938 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:26.938 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGRecoveryDelete(3.2f e241,227 3:f68554f1:::benchmark_data_smithi177_31116_object47862:head,174'7646)", 2024-09-13T11:27:26.938 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:27:11.254029+0000", 2024-09-13T11:27:26.938 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 15.585744922, 2024-09-13T11:27:26.938 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.225005441, 2024-09-13T11:27:26.938 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:27:26.938 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:27:26.938 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:27:26.938 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:27:26.938 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:26.938 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:27:26.938 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:27:11.254029+0000", 2024-09-13T11:27:26.939 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:26.939 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:26.939 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:26.939 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:27:26.939 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:27:11.254029+0000", 2024-09-13T11:27:26.939 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:26.939 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:26.939 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:26.939 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:27:26.939 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:27:11.254032+0000", 2024-09-13T11:27:26.939 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.9569999999999998e-06 2024-09-13T11:27:26.939 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:26.939 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:26.940 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:27:26.940 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:27:11.254043+0000", 2024-09-13T11:27:26.940 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0703000000000001e-05 2024-09-13T11:27:26.940 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:26.940 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:26.940 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:27:26.940 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:27:11.478618+0000", 2024-09-13T11:27:26.940 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.22457474099999999 2024-09-13T11:27:26.940 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:26.940 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:26.940 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:27:26.940 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:27:11.478633+0000", 2024-09-13T11:27:26.940 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5786000000000001e-05 2024-09-13T11:27:26.941 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:26.941 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:26.941 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:27:26.941 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:27:11.478825+0000", 2024-09-13T11:27:26.941 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000191719 2024-09-13T11:27:26.941 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:26.941 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:26.941 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:27:26.941 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:27:11.478847+0000", 2024-09-13T11:27:26.941 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.1662999999999999e-05 2024-09-13T11:27:26.941 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:26.941 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:26.941 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:27:26.942 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:27:11.479035+0000", 2024-09-13T11:27:26.942 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00018787200000000001 2024-09-13T11:27:26.942 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:26.942 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:27:26.942 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:26.942 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:26.942 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:26.942 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGRecoveryDelete(3.2f e244,227 3:f784bcf5:::benchmark_data_smithi177_31116_object67166:head,184'8217)", 2024-09-13T11:27:26.942 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:27:16.064546+0000", 2024-09-13T11:27:26.942 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 10.775227906, 2024-09-13T11:27:26.942 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.48785419200000002, 2024-09-13T11:27:26.942 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:27:26.942 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:27:26.943 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-13T11:27:26.943 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:27:26.943 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:26.943 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:27:26.943 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:27:16.064546+0000", 2024-09-13T11:27:26.943 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:26.943 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:26.943 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:26.943 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:27:26.943 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:27:16.064546+0000", 2024-09-13T11:27:26.943 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:26.943 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:26.943 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:26.943 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:27:26.944 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:27:16.064548+0000", 2024-09-13T11:27:26.944 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.435e-06 2024-09-13T11:27:26.944 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:26.944 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:26.944 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:27:26.944 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:27:16.064553+0000", 2024-09-13T11:27:26.944 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.1499999999999998e-06 2024-09-13T11:27:26.944 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:26.944 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:26.944 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:27:26.944 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:27:16.552082+0000", 2024-09-13T11:27:26.944 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.48752930999999999 2024-09-13T11:27:26.944 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:26.945 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:26.945 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:27:26.945 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:27:16.552091+0000", 2024-09-13T11:27:26.945 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.2350000000000005e-06 2024-09-13T11:27:26.945 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:26.945 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:26.945 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:27:26.945 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:27:16.552281+0000", 2024-09-13T11:27:26.945 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000189798 2024-09-13T11:27:26.945 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:26.945 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:26.945 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:27:26.945 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:27:16.552297+0000", 2024-09-13T11:27:26.946 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5736e-05 2024-09-13T11:27:26.946 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:26.946 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:26.946 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:27:26.946 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:27:16.552400+0000", 2024-09-13T11:27:26.946 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000103528 2024-09-13T11:27:26.946 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:26.946 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:27:26.946 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:26.946 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:26.946 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:26.946 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.4665.0:278438 3.15 3:ab6b0d5d:::benchmark_data_smithi177_31116_object138135:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e245)", 2024-09-13T11:27:26.946 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-13T11:27:16.736707+0000", 2024-09-13T11:27:26.946 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 10.10306671, 2024-09-13T11:27:26.947 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.35539664300000001, 2024-09-13T11:27:26.947 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-13T11:27:26.947 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-13T11:27:26.947 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:27:26.947 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-13T11:27:26.947 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.4665", 2024-09-13T11:27:26.947 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "v1:172.21.15.177:0/789083725", 2024-09-13T11:27:26.947 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 278438 2024-09-13T11:27:26.947 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:26.947 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-13T11:27:26.947 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:26.947 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-13T11:27:26.947 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:27:16.736707+0000", 2024-09-13T11:27:26.948 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:26.948 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:26.948 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:26.948 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-13T11:27:26.948 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:27:16.736707+0000", 2024-09-13T11:27:26.948 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-13T11:27:26.948 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:26.948 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:26.948 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-13T11:27:26.948 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:27:16.736709+0000", 2024-09-13T11:27:26.948 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.863e-06 2024-09-13T11:27:26.948 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:26.948 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:26.948 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-13T11:27:26.949 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:27:16.736715+0000", 2024-09-13T11:27:26.949 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.3380000000000004e-06 2024-09-13T11:27:26.949 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:26.949 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:26.949 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-13T11:27:26.949 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:27:16.736726+0000", 2024-09-13T11:27:26.949 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1365e-05 2024-09-13T11:27:26.949 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:26.949 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:26.949 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-13T11:27:26.949 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:27:16.736732+0000", 2024-09-13T11:27:26.949 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.9479999999999996e-06 2024-09-13T11:27:26.949 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:26.949 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:26.950 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-13T11:27:26.950 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:27:16.736790+0000", 2024-09-13T11:27:26.950 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.8477999999999998e-05 2024-09-13T11:27:26.950 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:26.950 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:26.950 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-13T11:27:26.950 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:27:16.736912+0000", 2024-09-13T11:27:26.950 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000121279 2024-09-13T11:27:26.950 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:26.950 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:26.950 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 0", 2024-09-13T11:27:26.950 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:27:16.736989+0000", 2024-09-13T11:27:26.950 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.6922999999999995e-05 2024-09-13T11:27:26.950 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:26.951 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:26.951 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-13T11:27:26.951 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:27:16.738822+0000", 2024-09-13T11:27:26.951 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0018333959999999999 2024-09-13T11:27:26.951 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:26.951 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:26.951 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:27:26.951 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:27:17.092030+0000", 2024-09-13T11:27:26.951 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.35320780899999998 2024-09-13T11:27:26.951 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:26.951 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:26.951 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-13T11:27:26.951 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:27:17.092068+0000", 2024-09-13T11:27:26.951 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.7710000000000003e-05 2024-09-13T11:27:26.952 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-13T11:27:26.952 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-13T11:27:26.952 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-13T11:27:26.952 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-13T11:27:17.092104+0000", 2024-09-13T11:27:26.952 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.6534e-05 2024-09-13T11:27:26.952 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:26.952 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:27:26.952 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:26.952 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-13T11:27:26.952 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-13T11:27:26.952 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-13T11:27:26.968 INFO:tasks.ceph.osd.8.smithi001.stderr:2024-09-13T11:27:26.966+0000 7fb9ab2df640 -1 osd.8 254 handle_osd_map injecting map crc failure 2024-09-13T11:27:27.176 INFO:teuthology.orchestra.run.smithi162.stdout:m(3) p(3,0-12) O(3,0-13)=block_cache={type=binned_lru} L=min_write_buffer_number_to_merge=32 P=min_write_buffer_number_to_merge=32 2024-09-13T11:27:27.183 INFO:tasks.thrashosds.thrasher:applying new sharding to bluestore on osd.10 2024-09-13T11:27:27.184 INFO:tasks.thrashosds.thrasher:generate random sharding 2024-09-13T11:27:27.184 INFO:tasks.thrashosds.thrasher:applying new sharding: O(4) P(1,0-) 2024-09-13T11:27:27.184 DEBUG:teuthology.orchestra.run.smithi162:> sudo ceph-bluestore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/bluestore_tool.$pid.log' --log-level=10 --path /var/lib/ceph/osd/ceph-10 --sharding 'O(4) P(1,0-)' reshard 2024-09-13T11:27:27.936 INFO:teuthology.orchestra.run.smithi162.stdout:reshard success 2024-09-13T11:27:28.253 INFO:tasks.ceph.osd.3.smithi177.stderr:2024-09-13T11:27:28.252+0000 7f10a25d1640 -1 osd.3 254 handle_osd_map injecting map crc failure 2024-09-13T11:27:28.315 INFO:tasks.thrashosds.thrasher:running fsck to verify new sharding on osd.10 2024-09-13T11:27:28.315 DEBUG:teuthology.orchestra.run.smithi162:> sudo ceph-bluestore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/bluestore_tool.$pid.log' --log-level=10 --path /var/lib/ceph/osd/ceph-10 fsck 2024-09-13T11:27:28.352 INFO:tasks.ceph.osd.5.smithi092.stderr:2024-09-13T11:27:28.350+0000 7f6f00132640 -1 osd.5 254 handle_osd_map injecting map crc failure 2024-09-13T11:27:29.704 INFO:teuthology.orchestra.run.smithi162.stdout:fsck success 2024-09-13T11:27:29.720 INFO:tasks.thrashosds.thrasher:resharding successfully completed 2024-09-13T11:27:29.720 INFO:tasks.thrashosds.thrasher:Reviving osd 10 2024-09-13T11:27:29.720 INFO:tasks.ceph.osd.10:Restarting daemon 2024-09-13T11:27:29.720 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 10 2024-09-13T11:27:29.722 INFO:tasks.ceph.osd.10:Started 2024-09-13T11:27:29.723 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-09-13T11:27:29.839 INFO:teuthology.orchestra.run.smithi162.stderr:no valid command found; 10 closest matches: 2024-09-13T11:27:29.840 INFO:teuthology.orchestra.run.smithi162.stderr:0 2024-09-13T11:27:29.840 INFO:teuthology.orchestra.run.smithi162.stderr:1 2024-09-13T11:27:29.840 INFO:teuthology.orchestra.run.smithi162.stderr:2 2024-09-13T11:27:29.840 INFO:teuthology.orchestra.run.smithi162.stderr:abort 2024-09-13T11:27:29.840 INFO:teuthology.orchestra.run.smithi162.stderr:assert 2024-09-13T11:27:29.840 INFO:teuthology.orchestra.run.smithi162.stderr:config diff 2024-09-13T11:27:29.840 INFO:teuthology.orchestra.run.smithi162.stderr:config diff get 2024-09-13T11:27:29.840 INFO:teuthology.orchestra.run.smithi162.stderr:config get 2024-09-13T11:27:29.840 INFO:teuthology.orchestra.run.smithi162.stderr:config help [] 2024-09-13T11:27:29.840 INFO:teuthology.orchestra.run.smithi162.stderr:config set ... 2024-09-13T11:27:29.840 INFO:teuthology.orchestra.run.smithi162.stderr:admin_socket: invalid command 2024-09-13T11:27:29.842 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-13T11:27:29.842 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-10, ['dump_ops_in_flight'] 2024-09-13T11:27:31.054 INFO:tasks.ceph.osd.10.smithi162.stderr:2024-09-13T11:27:31.051+0000 7ff898f387c0 -1 Falling back to public interface 2024-09-13T11:27:32.480 INFO:tasks.ceph.osd.8.smithi001.stderr:2024-09-13T11:27:32.478+0000 7fb9a92db640 -1 osd.8 257 handle_osd_map injecting map crc failure 2024-09-13T11:27:34.843 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-09-13T11:27:34.984 INFO:teuthology.orchestra.run.smithi162.stderr:no valid command found; 10 closest matches: 2024-09-13T11:27:34.984 INFO:teuthology.orchestra.run.smithi162.stderr:0 2024-09-13T11:27:34.984 INFO:teuthology.orchestra.run.smithi162.stderr:1 2024-09-13T11:27:34.984 INFO:teuthology.orchestra.run.smithi162.stderr:2 2024-09-13T11:27:34.984 INFO:teuthology.orchestra.run.smithi162.stderr:abort 2024-09-13T11:27:34.984 INFO:teuthology.orchestra.run.smithi162.stderr:assert 2024-09-13T11:27:34.984 INFO:teuthology.orchestra.run.smithi162.stderr:bluefs debug_inject_read_zeros 2024-09-13T11:27:34.984 INFO:teuthology.orchestra.run.smithi162.stderr:bluefs files list 2024-09-13T11:27:34.984 INFO:teuthology.orchestra.run.smithi162.stderr:bluefs stats 2024-09-13T11:27:34.984 INFO:teuthology.orchestra.run.smithi162.stderr:bluestore allocator dump block 2024-09-13T11:27:34.984 INFO:teuthology.orchestra.run.smithi162.stderr:bluestore allocator fragmentation block 2024-09-13T11:27:34.985 INFO:teuthology.orchestra.run.smithi162.stderr:admin_socket: invalid command 2024-09-13T11:27:34.987 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-13T11:27:34.988 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-10, ['dump_ops_in_flight'] 2024-09-13T11:27:35.736 INFO:tasks.ceph.osd.10.smithi162.stderr:2024-09-13T11:27:35.735+0000 7ff898f387c0 -1 osd.10 238 log_to_monitors true 2024-09-13T11:27:36.581 INFO:tasks.ceph.osd.15.smithi177.stderr:2024-09-13T11:27:36.576+0000 7f9d618a4640 -1 osd.15 258 handle_osd_map injecting map crc failure 2024-09-13T11:27:36.627 INFO:tasks.ceph.osd.14.smithi162.stderr:2024-09-13T11:27:36.623+0000 7f3dea330640 -1 osd.14 258 handle_osd_map injecting map crc failure 2024-09-13T11:27:37.780 INFO:tasks.ceph.osd.10.smithi162.stderr:2024-09-13T11:27:37.779+0000 7ff88cd13640 -1 osd.10 238 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-13T11:27:39.329 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-09-13T11:27:39.327+0000 7f97b39f6640 -1 osd.6 260 handle_osd_map injecting map crc failure 2024-09-13T11:27:39.331 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-09-13T11:27:39.327+0000 7f97b39f6640 -1 osd.6 260 handle_osd_map injecting map crc failure 2024-09-13T11:27:39.356 INFO:tasks.ceph.osd.12.smithi001.stderr:2024-09-13T11:27:39.354+0000 7f891d10b640 -1 osd.12 260 handle_osd_map injecting map crc failure 2024-09-13T11:27:39.357 INFO:tasks.ceph.osd.2.smithi162.stderr:2024-09-13T11:27:39.355+0000 7fe91fd87640 -1 osd.2 260 handle_osd_map injecting map crc failure 2024-09-13T11:27:39.988 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-09-13T11:27:40.119 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-13T11:27:40.119 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [], 2024-09-13T11:27:40.119 INFO:teuthology.orchestra.run.smithi162.stdout: "num_ops": 0 2024-09-13T11:27:40.119 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-13T11:27:40.133 INFO:tasks.thrashosds.thrasher:test_bluestore_reshard completed 2024-09-13T11:27:40.304 INFO:tasks.ceph.osd.13.smithi092.stderr:2024-09-13T11:27:40.302+0000 7f3e4ba0f640 -1 osd.13 261 handle_osd_map injecting map crc failure 2024-09-13T11:27:41.746 INFO:tasks.ceph.osd.12.smithi001.stderr:2024-09-13T11:27:41.742+0000 7f891d10b640 -1 osd.12 262 handle_osd_map injecting map crc failure 2024-09-13T11:27:42.141 INFO:tasks.ceph.osd.12.smithi001.stderr:2024-09-13T11:27:42.137+0000 7f891d10b640 -1 osd.12 263 handle_osd_map injecting map crc failure 2024-09-13T11:27:45.134 INFO:tasks.thrashosds.thrasher:inning osd 2024-09-13T11:27:45.134 INFO:tasks.thrashosds.thrasher:Adding osd 10 2024-09-13T11:27:45.134 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd in 10 2024-09-13T11:27:45.471 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-09-13T11:27:45.469+0000 7fb2a5596640 -1 mon.a@0(leader).osd e265 definitely_dead 0 2024-09-13T11:27:46.325 INFO:teuthology.orchestra.run.smithi001.stderr:marked in osd.10. 2024-09-13T11:27:46.326 INFO:tasks.ceph.osd.2.smithi162.stderr:2024-09-13T11:27:46.322+0000 7fe921d8b640 -1 osd.2 265 handle_osd_map injecting map crc failure 2024-09-13T11:27:46.352 INFO:tasks.thrashosds.thrasher:Added osd 10 2024-09-13T11:27:46.353 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'mon.*' injectargs --mon_osd_down_out_interval=600 2024-09-13T11:27:46.504 INFO:teuthology.orchestra.run.smithi001.stdout:mon.a: {} 2024-09-13T11:27:46.504 INFO:teuthology.orchestra.run.smithi001.stderr:mon.a: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-13T11:27:46.524 INFO:teuthology.orchestra.run.smithi001.stdout:mon.b: {} 2024-09-13T11:27:46.524 INFO:teuthology.orchestra.run.smithi001.stderr:mon.b: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-13T11:27:46.544 INFO:teuthology.orchestra.run.smithi001.stdout:mon.c: {} 2024-09-13T11:27:46.545 INFO:teuthology.orchestra.run.smithi001.stderr:mon.c: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-13T11:27:46.563 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 0 1 2024-09-13T11:27:46.577 INFO:tasks.ceph.osd.15.smithi177.stderr:2024-09-13T11:27:46.576+0000 7f9d618a4640 -1 osd.15 264 handle_osd_map injecting map crc failure 2024-09-13T11:27:46.826 INFO:tasks.ceph.osd.9.smithi092.stderr:2024-09-13T11:27:46.822+0000 7ff59bdf5640 -1 osd.9 265 handle_osd_map injecting map crc failure 2024-09-13T11:27:46.832 INFO:tasks.ceph.osd.5.smithi092.stderr:2024-09-13T11:27:46.830+0000 7f6f00132640 -1 osd.5 264 handle_osd_map injecting map crc failure 2024-09-13T11:27:47.345 INFO:teuthology.orchestra.run.smithi001.stderr:reweighted osd.0 to 1 (10000) 2024-09-13T11:27:47.367 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 0 1 2024-09-13T11:27:48.347 INFO:teuthology.orchestra.run.smithi001.stderr:set osd.0 primary-affinity to 1 (8655362) 2024-09-13T11:27:48.372 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 4 1 2024-09-13T11:27:49.363 INFO:teuthology.orchestra.run.smithi001.stderr:reweighted osd.4 to 1 (10000) 2024-09-13T11:27:49.394 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 4 1 2024-09-13T11:27:49.509 INFO:tasks.ceph.osd.13.smithi092.stderr:2024-09-13T11:27:49.506+0000 7f3e4ba0f640 -1 osd.13 267 handle_osd_map injecting map crc failure 2024-09-13T11:27:49.684 INFO:tasks.ceph.osd.1.smithi092.stderr:2024-09-13T11:27:49.682+0000 7fe76f3f6640 -1 osd.1 267 handle_osd_map injecting map crc failure 2024-09-13T11:27:50.373 INFO:teuthology.orchestra.run.smithi001.stderr:set osd.4 primary-affinity to 1 (8655362) 2024-09-13T11:27:50.396 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 8 1 2024-09-13T11:27:51.388 INFO:teuthology.orchestra.run.smithi001.stderr:reweighted osd.8 to 1 (10000) 2024-09-13T11:27:51.411 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 8 1 2024-09-13T11:27:51.559 INFO:tasks.ceph.osd.14.smithi162.stderr:2024-09-13T11:27:51.554+0000 7f3dea330640 -1 osd.14 269 handle_osd_map injecting map crc failure 2024-09-13T11:27:51.559 INFO:tasks.ceph.osd.7.smithi177.stderr:2024-09-13T11:27:51.556+0000 7f896ff4a640 -1 osd.7 269 handle_osd_map injecting map crc failure 2024-09-13T11:27:51.684 INFO:tasks.ceph.osd.3.smithi177.stderr:2024-09-13T11:27:51.680+0000 7f10a25d1640 -1 osd.3 269 handle_osd_map injecting map crc failure 2024-09-13T11:27:51.684 INFO:tasks.ceph.osd.4.smithi001.stderr:2024-09-13T11:27:51.681+0000 7ff8d0c3d640 -1 osd.4 269 handle_osd_map injecting map crc failure 2024-09-13T11:27:51.831 INFO:tasks.ceph.osd.1.smithi092.stderr:2024-09-13T11:27:51.830+0000 7fe76d3f2640 -1 osd.1 270 handle_osd_map injecting map crc failure 2024-09-13T11:27:51.981 INFO:teuthology.orchestra.run.smithi001.stderr:set osd.8 primary-affinity to 1 (8655362) 2024-09-13T11:27:52.007 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 12 1 2024-09-13T11:27:52.983 INFO:teuthology.orchestra.run.smithi001.stderr:reweighted osd.12 to 1 (10000) 2024-09-13T11:27:52.984 INFO:tasks.ceph.osd.13.smithi092.stderr:2024-09-13T11:27:52.982+0000 7f3e4da13640 -1 osd.13 271 handle_osd_map injecting map crc failure 2024-09-13T11:27:53.012 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 12 1 2024-09-13T11:27:53.284 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-09-13T11:27:53.282+0000 7f97b39f6640 -1 osd.6 272 handle_osd_map injecting map crc failure 2024-09-13T11:27:53.385 INFO:tasks.ceph.osd.15.smithi177.stderr:2024-09-13T11:27:53.384+0000 7f9d618a4640 -1 osd.15 271 handle_osd_map injecting map crc failure 2024-09-13T11:27:53.993 INFO:teuthology.orchestra.run.smithi001.stderr:set osd.12 primary-affinity to 1 (8655362) 2024-09-13T11:27:54.016 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 5 1 2024-09-13T11:27:55.001 INFO:teuthology.orchestra.run.smithi001.stderr:reweighted osd.5 to 1 (10000) 2024-09-13T11:27:55.215 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 5 1 2024-09-13T11:27:55.248 INFO:tasks.ceph.osd.5.smithi092.stderr:2024-09-13T11:27:55.246+0000 7f6f00132640 -1 osd.5 273 handle_osd_map injecting map crc failure 2024-09-13T11:27:56.003 INFO:teuthology.orchestra.run.smithi001.stderr:set osd.5 primary-affinity to 1 (8655362) 2024-09-13T11:27:56.004 INFO:tasks.ceph.osd.12.smithi001.stderr:2024-09-13T11:27:56.001+0000 7f891f10f640 -1 osd.12 274 handle_osd_map injecting map crc failure 2024-09-13T11:27:56.005 INFO:tasks.ceph.osd.10.smithi162.stderr:2024-09-13T11:27:56.002+0000 7ff88891d640 -1 osd.10 274 handle_osd_map injecting map crc failure 2024-09-13T11:27:56.028 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 9 1 2024-09-13T11:27:56.164 INFO:tasks.ceph.osd.15.smithi177.stderr:2024-09-13T11:27:56.164+0000 7f9d638a8640 -1 osd.15 274 handle_osd_map injecting map crc failure 2024-09-13T11:27:56.243 INFO:tasks.ceph.osd.9.smithi092.stderr:2024-09-13T11:27:56.238+0000 7ff59bdf5640 -1 osd.9 275 handle_osd_map injecting map crc failure 2024-09-13T11:27:57.025 INFO:teuthology.orchestra.run.smithi001.stderr:reweighted osd.9 to 1 (10000) 2024-09-13T11:27:57.050 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 9 1 2024-09-13T11:27:57.131 INFO:tasks.ceph.osd.1.smithi092.stderr:2024-09-13T11:27:57.126+0000 7fe76d3f2640 -1 osd.1 276 handle_osd_map injecting map crc failure 2024-09-13T11:27:58.028 INFO:teuthology.orchestra.run.smithi001.stderr:set osd.9 primary-affinity to 1 (8655362) 2024-09-13T11:27:58.029 INFO:tasks.ceph.osd.11.smithi177.stderr:2024-09-13T11:27:58.028+0000 7f246bcf0640 -1 osd.11 277 handle_osd_map injecting map crc failure 2024-09-13T11:27:58.029 INFO:tasks.ceph.osd.10.smithi162.stderr:2024-09-13T11:27:58.026+0000 7ff88891d640 -1 osd.10 277 handle_osd_map injecting map crc failure 2024-09-13T11:27:58.055 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 13 1 2024-09-13T11:27:59.044 INFO:teuthology.orchestra.run.smithi001.stderr:reweighted osd.13 to 1 (10000) 2024-09-13T11:27:59.070 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 13 1 2024-09-13T11:27:59.946 INFO:tasks.ceph.osd.8.smithi001.stderr:2024-09-13T11:27:59.941+0000 7fb9a92db640 -1 osd.8 277 handle_osd_map injecting map crc failure 2024-09-13T11:27:59.952 INFO:tasks.ceph.osd.8.smithi001.stderr:2024-09-13T11:27:59.949+0000 7fb9ab2df640 -1 osd.8 278 handle_osd_map injecting map crc failure 2024-09-13T11:28:00.046 INFO:teuthology.orchestra.run.smithi001.stderr:set osd.13 primary-affinity to 1 (8655362) 2024-09-13T11:28:00.047 INFO:tasks.ceph.osd.10.smithi162.stderr:2024-09-13T11:28:00.046+0000 7ff88891d640 -1 osd.10 279 handle_osd_map injecting map crc failure 2024-09-13T11:28:00.070 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 2 1 2024-09-13T11:28:01.061 INFO:teuthology.orchestra.run.smithi001.stderr:reweighted osd.2 to 1 (10000) 2024-09-13T11:28:01.062 INFO:tasks.ceph.osd.8.smithi001.stderr:2024-09-13T11:28:01.057+0000 7fb9ab2df640 -1 osd.8 280 handle_osd_map injecting map crc failure 2024-09-13T11:28:01.085 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 2 1 2024-09-13T11:28:01.984 INFO:teuthology.orchestra.run.smithi001.stderr:set osd.2 primary-affinity to 1 (8655362) 2024-09-13T11:28:02.011 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 6 1 2024-09-13T11:28:02.661 INFO:tasks.ceph.osd.13.smithi092.stderr:2024-09-13T11:28:02.658+0000 7f3e4ba0f640 -1 osd.13 281 handle_osd_map injecting map crc failure 2024-09-13T11:28:03.072 INFO:teuthology.orchestra.run.smithi001.stderr:reweighted osd.6 to 1 (10000) 2024-09-13T11:28:03.073 INFO:tasks.ceph.osd.14.smithi162.stderr:2024-09-13T11:28:03.070+0000 7f3dec334640 -1 osd.14 282 handle_osd_map injecting map crc failure 2024-09-13T11:28:03.087 INFO:tasks.ceph.osd.7.smithi177.stderr:2024-09-13T11:28:03.084+0000 7f896ff4a640 -1 osd.7 281 handle_osd_map injecting map crc failure 2024-09-13T11:28:03.098 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 6 1 2024-09-13T11:28:04.075 INFO:teuthology.orchestra.run.smithi001.stderr:set osd.6 primary-affinity to 1 (8655362) 2024-09-13T11:28:04.098 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 3 1 2024-09-13T11:28:05.090 INFO:teuthology.orchestra.run.smithi001.stderr:reweighted osd.3 to 1 (10000) 2024-09-13T11:28:05.116 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 3 1 2024-09-13T11:28:06.112 INFO:teuthology.orchestra.run.smithi001.stderr:set osd.3 primary-affinity to 1 (8655362) 2024-09-13T11:28:06.113 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-09-13T11:28:06.110+0000 7f97b59fa640 -1 osd.6 285 handle_osd_map injecting map crc failure 2024-09-13T11:28:06.135 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 7 1 2024-09-13T11:28:06.586 INFO:tasks.ceph.osd.7.smithi177.stderr:2024-09-13T11:28:06.584+0000 7f896ff4a640 -1 osd.7 285 handle_osd_map injecting map crc failure 2024-09-13T11:28:07.114 INFO:teuthology.orchestra.run.smithi001.stderr:reweighted osd.7 to 1 (10000) 2024-09-13T11:28:07.115 INFO:tasks.ceph.osd.9.smithi092.stderr:2024-09-13T11:28:07.114+0000 7ff59ddf9640 -1 osd.9 285 handle_osd_map injecting map crc failure 2024-09-13T11:28:07.143 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 7 1 2024-09-13T11:28:08.124 INFO:teuthology.orchestra.run.smithi001.stderr:set osd.7 primary-affinity to 1 (8655362) 2024-09-13T11:28:08.125 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-09-13T11:28:08.122+0000 7f97b59fa640 -1 osd.6 286 handle_osd_map injecting map crc failure 2024-09-13T11:28:08.153 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 11 1 2024-09-13T11:28:08.450 INFO:tasks.ceph.osd.14.smithi162.stderr:2024-09-13T11:28:08.446+0000 7f3dea330640 -1 osd.14 286 handle_osd_map injecting map crc failure 2024-09-13T11:28:08.451 INFO:tasks.ceph.osd.3.smithi177.stderr:2024-09-13T11:28:08.448+0000 7f10a25d1640 -1 osd.3 285 handle_osd_map injecting map crc failure 2024-09-13T11:28:09.139 INFO:teuthology.orchestra.run.smithi001.stderr:reweighted osd.11 to 1 (10000) 2024-09-13T11:28:09.165 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 11 1 2024-09-13T11:28:10.142 INFO:teuthology.orchestra.run.smithi001.stderr:set osd.11 primary-affinity to 1 (8655362) 2024-09-13T11:28:10.172 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 15 1 2024-09-13T11:28:10.183 INFO:tasks.ceph.osd.5.smithi092.stderr:2024-09-13T11:28:10.178+0000 7f6f00132640 -1 osd.5 288 handle_osd_map injecting map crc failure 2024-09-13T11:28:10.247 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-13T11:28:10.245+0000 7f6a5d1a2640 -1 osd.0 288 handle_osd_map injecting map crc failure 2024-09-13T11:28:10.248 INFO:tasks.ceph.osd.7.smithi177.stderr:2024-09-13T11:28:10.244+0000 7f896ff4a640 -1 osd.7 288 handle_osd_map injecting map crc failure 2024-09-13T11:28:11.159 INFO:teuthology.orchestra.run.smithi001.stderr:reweighted osd.15 to 1 (10000) 2024-09-13T11:28:11.192 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 15 1 2024-09-13T11:28:11.348 INFO:tasks.ceph.osd.7.smithi177.stderr:2024-09-13T11:28:11.344+0000 7f896ff4a640 -1 osd.7 290 handle_osd_map injecting map crc failure 2024-09-13T11:28:11.349 INFO:tasks.ceph.osd.9.smithi092.stderr:2024-09-13T11:28:11.346+0000 7ff59bdf5640 -1 osd.9 290 handle_osd_map injecting map crc failure 2024-09-13T11:28:11.733 INFO:tasks.ceph.osd.2.smithi162.stderr:2024-09-13T11:28:11.730+0000 7fe91fd87640 -1 osd.2 290 handle_osd_map injecting map crc failure 2024-09-13T11:28:11.811 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-09-13T11:28:11.806+0000 7f97b39f6640 -1 osd.6 288 handle_osd_map injecting map crc failure 2024-09-13T11:28:11.988 INFO:teuthology.orchestra.run.smithi001.stderr:set osd.15 primary-affinity to 1 (8655362) 2024-09-13T11:28:12.014 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 1 1 2024-09-13T11:28:12.303 INFO:tasks.ceph.osd.4.smithi001.stderr:2024-09-13T11:28:12.297+0000 7ff8d0c3d640 -1 osd.4 291 handle_osd_map injecting map crc failure 2024-09-13T11:28:12.990 INFO:teuthology.orchestra.run.smithi001.stderr:reweighted osd.1 to 1 (10000) 2024-09-13T11:28:13.014 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 1 1 2024-09-13T11:28:13.087 INFO:tasks.ceph.osd.2.smithi162.stderr:2024-09-13T11:28:13.082+0000 7fe91fd87640 -1 osd.2 291 handle_osd_map injecting map crc failure 2024-09-13T11:28:14.178 INFO:teuthology.orchestra.run.smithi001.stderr:set osd.1 primary-affinity to 1 (8655362) 2024-09-13T11:28:14.203 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 14 1 2024-09-13T11:28:15.186 INFO:teuthology.orchestra.run.smithi001.stderr:reweighted osd.14 to 1 (10000) 2024-09-13T11:28:15.187 INFO:tasks.ceph.osd.2.smithi162.stderr:2024-09-13T11:28:15.182+0000 7fe921d8b640 -1 osd.2 294 handle_osd_map injecting map crc failure 2024-09-13T11:28:15.215 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 14 1 2024-09-13T11:28:15.564 INFO:tasks.ceph.osd.5.smithi092.stderr:2024-09-13T11:28:15.562+0000 7f6f00132640 -1 osd.5 294 handle_osd_map injecting map crc failure 2024-09-13T11:28:15.652 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-13T11:28:15.649+0000 7f6a5d1a2640 -1 osd.0 294 handle_osd_map injecting map crc failure 2024-09-13T11:28:16.189 INFO:teuthology.orchestra.run.smithi001.stderr:set osd.14 primary-affinity to 1 (8655362) 2024-09-13T11:28:16.213 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 10 1 2024-09-13T11:28:16.510 INFO:tasks.ceph.osd.13.smithi092.stderr:2024-09-13T11:28:16.510+0000 7f3e4ba0f640 -1 osd.13 295 handle_osd_map injecting map crc failure 2024-09-13T11:28:16.649 INFO:tasks.ceph.osd.15.smithi177.stderr:2024-09-13T11:28:16.648+0000 7f9d618a4640 -1 osd.15 295 handle_osd_map injecting map crc failure 2024-09-13T11:28:17.204 INFO:teuthology.orchestra.run.smithi001.stderr:reweighted osd.10 to 1 (10000) 2024-09-13T11:28:17.205 INFO:tasks.ceph.osd.7.smithi177.stderr:2024-09-13T11:28:17.204+0000 7f8971f4e640 -1 osd.7 296 handle_osd_map injecting map crc failure 2024-09-13T11:28:17.229 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 10 1 2024-09-13T11:28:17.604 INFO:tasks.ceph.osd.12.smithi001.stderr:2024-09-13T11:28:17.601+0000 7f891d10b640 -1 osd.12 296 handle_osd_map injecting map crc failure 2024-09-13T11:28:18.214 INFO:teuthology.orchestra.run.smithi001.stderr:set osd.10 primary-affinity to 1 (8655362) 2024-09-13T11:28:18.237 INFO:tasks.thrashosds.thrasher:joining the do_sighup greenlet 2024-09-13T11:28:18.237 INFO:tasks.thrashosds.thrasher:joining the do_optrack_toggle greenlet 2024-09-13T11:28:18.238 INFO:tasks.thrashosds.thrasher:joining the do_dump_ops greenlet 2024-09-13T11:28:18.238 INFO:tasks.thrashosds.thrasher:joining the do_noscrub_toggle greenlet 2024-09-13T11:28:18.238 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-09-13T11:28:18.238 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T11:28:18.552 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-13T11:28:18.552 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":298,"fsid":"2fd68ac7-5581-4d90-80a4-30bcf5252a7a","created":"2024-09-13T11:21:57.223153+0000","modified":"2024-09-13T11:28:18.205074+0000","last_up_change":"2024-09-13T11:27:38.782633+0000","last_in_change":"2024-09-13T11:27:46.310884+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":8,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T11:22:10.162266+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":15.380000114440918,"score_stable":15.380000114440918,"optimal_score":0.12999999523162842,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-13T11:22:12.235549+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"28","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":28,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4.0199999809265137,"score_stable":4.0199999809265137,"optimal_score":0.56000000238418579,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"0fc12b3c-e0de-4546-b935-db77efe7c567","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":250,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6800","nonce":1544776477}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6801","nonce":1544776477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6803","nonce":1544776477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6802","nonce":1544776477}]},"public_addr":"172.21.15.1:6800/1544776477","cluster_addr":"172.21.15.1:6801/1544776477","heartbeat_back_addr":"172.21.15.1:6803/1544776477","heartbeat_front_addr":"172.21.15.1:6802/1544776477","state":["exists","up"]},{"osd":1,"uuid":"b77fe331-cc92-4adb-b146-af5c380f6d4f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":23,"last_clean_end":29,"up_from":40,"up_thru":243,"down_at":30,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.92:6800","nonce":3591430847}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.92:6801","nonce":3591430847}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.92:6803","nonce":3591430847}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.92:6802","nonce":3591430847}]},"public_addr":"172.21.15.92:6800/3591430847","cluster_addr":"172.21.15.92:6801/3591430847","heartbeat_back_addr":"172.21.15.92:6803/3591430847","heartbeat_front_addr":"172.21.15.92:6802/3591430847","state":["exists","up"]},{"osd":2,"uuid":"c9d031c8-e885-4229-affa-08e1729d00e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":251,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.162:6812","nonce":2712148101}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.162:6813","nonce":2712148101}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.162:6815","nonce":2712148101}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.162:6814","nonce":2712148101}]},"public_addr":"172.21.15.162:6812/2712148101","cluster_addr":"172.21.15.162:6813/2712148101","heartbeat_back_addr":"172.21.15.162:6815/2712148101","heartbeat_front_addr":"172.21.15.162:6814/2712148101","state":["exists","up"]},{"osd":3,"uuid":"bf50d71a-6c3a-4fbc-a407-cb81bad14a7b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":250,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.177:6801","nonce":3367751690}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.177:6802","nonce":3367751690}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.177:6804","nonce":3367751690}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.177:6803","nonce":3367751690}]},"public_addr":"172.21.15.177:6801/3367751690","cluster_addr":"172.21.15.177:6802/3367751690","heartbeat_back_addr":"172.21.15.177:6804/3367751690","heartbeat_front_addr":"172.21.15.177:6803/3367751690","state":["exists","up"]},{"osd":4,"uuid":"d1fe3f63-df3f-4a73-a1e2-8d7c8a3f1fa0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":239,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6808","nonce":3648540926}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6809","nonce":3648540926}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6811","nonce":3648540926}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6810","nonce":3648540926}]},"public_addr":"172.21.15.1:6808/3648540926","cluster_addr":"172.21.15.1:6809/3648540926","heartbeat_back_addr":"172.21.15.1:6811/3648540926","heartbeat_front_addr":"172.21.15.1:6810/3648540926","state":["exists","up"]},{"osd":5,"uuid":"e2ead447-2f2c-40c9-82e7-53b8dc41e2af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":249,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.92:6804","nonce":3399529091}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.92:6805","nonce":3399529091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.92:6807","nonce":3399529091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.92:6806","nonce":3399529091}]},"public_addr":"172.21.15.92:6804/3399529091","cluster_addr":"172.21.15.92:6805/3399529091","heartbeat_back_addr":"172.21.15.92:6807/3399529091","heartbeat_front_addr":"172.21.15.92:6806/3399529091","state":["exists","up"]},{"osd":6,"uuid":"dbc20439-144e-4ec7-89de-674eb783f263","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":250,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.162:6808","nonce":2287343806}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.162:6809","nonce":2287343806}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.162:6811","nonce":2287343806}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.162:6810","nonce":2287343806}]},"public_addr":"172.21.15.162:6808/2287343806","cluster_addr":"172.21.15.162:6809/2287343806","heartbeat_back_addr":"172.21.15.162:6811/2287343806","heartbeat_front_addr":"172.21.15.162:6810/2287343806","state":["exists","up"]},{"osd":7,"uuid":"60aa5206-cace-40f8-8d1a-10ec28d97096","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":251,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.177:6813","nonce":2110809906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.177:6814","nonce":2110809906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.177:6816","nonce":2110809906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.177:6815","nonce":2110809906}]},"public_addr":"172.21.15.177:6813/2110809906","cluster_addr":"172.21.15.177:6814/2110809906","heartbeat_back_addr":"172.21.15.177:6816/2110809906","heartbeat_front_addr":"172.21.15.177:6815/2110809906","state":["exists","up"]},{"osd":8,"uuid":"f3d99325-48ff-403c-8662-1d7745f7c97d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6804","nonce":2766567910}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6805","nonce":2766567910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6807","nonce":2766567910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6806","nonce":2766567910}]},"public_addr":"172.21.15.1:6804/2766567910","cluster_addr":"172.21.15.1:6805/2766567910","heartbeat_back_addr":"172.21.15.1:6807/2766567910","heartbeat_front_addr":"172.21.15.1:6806/2766567910","state":["exists","up"]},{"osd":9,"uuid":"88511e24-3675-4551-968c-1a52e7e28f96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":244,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.92:6812","nonce":2488187511}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.92:6813","nonce":2488187511}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.92:6815","nonce":2488187511}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.92:6814","nonce":2488187511}]},"public_addr":"172.21.15.92:6812/2488187511","cluster_addr":"172.21.15.92:6813/2488187511","heartbeat_back_addr":"172.21.15.92:6815/2488187511","heartbeat_front_addr":"172.21.15.92:6814/2488187511","state":["exists","up"]},{"osd":10,"uuid":"acf4c64d-99af-42be-8406-413c86f54670","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":22,"last_clean_end":238,"up_from":261,"up_thru":266,"down_at":239,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.162:6804","nonce":4258911154}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.162:6805","nonce":4258911154}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.162:6807","nonce":4258911154}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.162:6806","nonce":4258911154}]},"public_addr":"172.21.15.162:6804/4258911154","cluster_addr":"172.21.15.162:6805/4258911154","heartbeat_back_addr":"172.21.15.162:6807/4258911154","heartbeat_front_addr":"172.21.15.162:6806/4258911154","state":["exists","up"]},{"osd":11,"uuid":"a4bfdfbe-b586-4080-8e09-6f26428e59b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":247,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.177:6809","nonce":3239521954}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.177:6810","nonce":3239521954}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.177:6812","nonce":3239521954}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.177:6811","nonce":3239521954}]},"public_addr":"172.21.15.177:6809/3239521954","cluster_addr":"172.21.15.177:6810/3239521954","heartbeat_back_addr":"172.21.15.177:6812/3239521954","heartbeat_front_addr":"172.21.15.177:6811/3239521954","state":["exists","up"]},{"osd":12,"uuid":"dc571fb6-d8b5-465b-bb25-afa7d0c47b5a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":249,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6812","nonce":4267062742}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6813","nonce":4267062742}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6815","nonce":4267062742}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6814","nonce":4267062742}]},"public_addr":"172.21.15.1:6812/4267062742","cluster_addr":"172.21.15.1:6813/4267062742","heartbeat_back_addr":"172.21.15.1:6815/4267062742","heartbeat_front_addr":"172.21.15.1:6814/4267062742","state":["exists","up"]},{"osd":13,"uuid":"5d321d4b-1b26-47fe-bf84-25751b8cde7d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":235,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.92:6808","nonce":1759585046}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.92:6809","nonce":1759585046}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.92:6811","nonce":1759585046}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.92:6810","nonce":1759585046}]},"public_addr":"172.21.15.92:6808/1759585046","cluster_addr":"172.21.15.92:6809/1759585046","heartbeat_back_addr":"172.21.15.92:6811/1759585046","heartbeat_front_addr":"172.21.15.92:6810/1759585046","state":["exists","up"]},{"osd":14,"uuid":"fc3e2499-0c2a-414d-a25e-2628876ae630","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":22,"last_clean_end":120,"up_from":142,"up_thru":296,"down_at":121,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.162:6800","nonce":3999392664}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.162:6801","nonce":3999392664}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.162:6803","nonce":3999392664}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.162:6802","nonce":3999392664}]},"public_addr":"172.21.15.162:6800/3999392664","cluster_addr":"172.21.15.162:6801/3999392664","heartbeat_back_addr":"172.21.15.162:6803/3999392664","heartbeat_front_addr":"172.21.15.162:6802/3999392664","state":["exists","up"]},{"osd":15,"uuid":"d2f69775-edfe-4527-91ec-f7c7e558bec8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":292,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.177:6805","nonce":1294337172}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.177:6806","nonce":1294337172}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.177:6808","nonce":1294337172}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.177:6807","nonce":1294337172}]},"public_addr":"172.21.15.177:6805/1294337172","cluster_addr":"172.21.15.177:6806/1294337172","heartbeat_back_addr":"172.21.15.177:6808/1294337172","heartbeat_front_addr":"172.21.15.177:6807/1294337172","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T11:27:40.342776+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T11:27:07.613933+0000","dead_epoch":30},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T11:27:22.828451+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T11:27:21.614437+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T11:27:25.568873+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T11:27:28.185934+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T11:27:46.757486+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T11:27:21.994608+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T11:27:44.589810+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T11:27:20.603346+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T11:27:37.802899+0000","dead_epoch":239},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T11:27:44.791122+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T11:26:59.682735+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T11:28:15.562220+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T11:27:06.849942+0000","dead_epoch":121},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T11:27:15.020976+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T11:28:18.570 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-09-13T11:28:18.570 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T11:28:18.681 INFO:tasks.ceph.osd.12.smithi001.stderr:2024-09-13T11:28:18.677+0000 7f891d10b640 -1 osd.12 297 handle_osd_map injecting map crc failure 2024-09-13T11:28:18.882 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-13T11:28:18.882 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":298,"fsid":"2fd68ac7-5581-4d90-80a4-30bcf5252a7a","created":"2024-09-13T11:21:57.223153+0000","modified":"2024-09-13T11:28:18.205074+0000","last_up_change":"2024-09-13T11:27:38.782633+0000","last_in_change":"2024-09-13T11:27:46.310884+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":8,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T11:22:10.162266+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":15.380000114440918,"score_stable":15.380000114440918,"optimal_score":0.12999999523162842,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-13T11:22:12.235549+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"28","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":28,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4.0199999809265137,"score_stable":4.0199999809265137,"optimal_score":0.56000000238418579,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"0fc12b3c-e0de-4546-b935-db77efe7c567","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":250,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6800","nonce":1544776477}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6801","nonce":1544776477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6803","nonce":1544776477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6802","nonce":1544776477}]},"public_addr":"172.21.15.1:6800/1544776477","cluster_addr":"172.21.15.1:6801/1544776477","heartbeat_back_addr":"172.21.15.1:6803/1544776477","heartbeat_front_addr":"172.21.15.1:6802/1544776477","state":["exists","up"]},{"osd":1,"uuid":"b77fe331-cc92-4adb-b146-af5c380f6d4f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":23,"last_clean_end":29,"up_from":40,"up_thru":243,"down_at":30,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.92:6800","nonce":3591430847}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.92:6801","nonce":3591430847}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.92:6803","nonce":3591430847}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.92:6802","nonce":3591430847}]},"public_addr":"172.21.15.92:6800/3591430847","cluster_addr":"172.21.15.92:6801/3591430847","heartbeat_back_addr":"172.21.15.92:6803/3591430847","heartbeat_front_addr":"172.21.15.92:6802/3591430847","state":["exists","up"]},{"osd":2,"uuid":"c9d031c8-e885-4229-affa-08e1729d00e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":251,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.162:6812","nonce":2712148101}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.162:6813","nonce":2712148101}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.162:6815","nonce":2712148101}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.162:6814","nonce":2712148101}]},"public_addr":"172.21.15.162:6812/2712148101","cluster_addr":"172.21.15.162:6813/2712148101","heartbeat_back_addr":"172.21.15.162:6815/2712148101","heartbeat_front_addr":"172.21.15.162:6814/2712148101","state":["exists","up"]},{"osd":3,"uuid":"bf50d71a-6c3a-4fbc-a407-cb81bad14a7b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":250,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.177:6801","nonce":3367751690}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.177:6802","nonce":3367751690}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.177:6804","nonce":3367751690}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.177:6803","nonce":3367751690}]},"public_addr":"172.21.15.177:6801/3367751690","cluster_addr":"172.21.15.177:6802/3367751690","heartbeat_back_addr":"172.21.15.177:6804/3367751690","heartbeat_front_addr":"172.21.15.177:6803/3367751690","state":["exists","up"]},{"osd":4,"uuid":"d1fe3f63-df3f-4a73-a1e2-8d7c8a3f1fa0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":239,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6808","nonce":3648540926}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6809","nonce":3648540926}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6811","nonce":3648540926}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6810","nonce":3648540926}]},"public_addr":"172.21.15.1:6808/3648540926","cluster_addr":"172.21.15.1:6809/3648540926","heartbeat_back_addr":"172.21.15.1:6811/3648540926","heartbeat_front_addr":"172.21.15.1:6810/3648540926","state":["exists","up"]},{"osd":5,"uuid":"e2ead447-2f2c-40c9-82e7-53b8dc41e2af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":249,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.92:6804","nonce":3399529091}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.92:6805","nonce":3399529091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.92:6807","nonce":3399529091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.92:6806","nonce":3399529091}]},"public_addr":"172.21.15.92:6804/3399529091","cluster_addr":"172.21.15.92:6805/3399529091","heartbeat_back_addr":"172.21.15.92:6807/3399529091","heartbeat_front_addr":"172.21.15.92:6806/3399529091","state":["exists","up"]},{"osd":6,"uuid":"dbc20439-144e-4ec7-89de-674eb783f263","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":250,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.162:6808","nonce":2287343806}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.162:6809","nonce":2287343806}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.162:6811","nonce":2287343806}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.162:6810","nonce":2287343806}]},"public_addr":"172.21.15.162:6808/2287343806","cluster_addr":"172.21.15.162:6809/2287343806","heartbeat_back_addr":"172.21.15.162:6811/2287343806","heartbeat_front_addr":"172.21.15.162:6810/2287343806","state":["exists","up"]},{"osd":7,"uuid":"60aa5206-cace-40f8-8d1a-10ec28d97096","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":251,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.177:6813","nonce":2110809906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.177:6814","nonce":2110809906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.177:6816","nonce":2110809906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.177:6815","nonce":2110809906}]},"public_addr":"172.21.15.177:6813/2110809906","cluster_addr":"172.21.15.177:6814/2110809906","heartbeat_back_addr":"172.21.15.177:6816/2110809906","heartbeat_front_addr":"172.21.15.177:6815/2110809906","state":["exists","up"]},{"osd":8,"uuid":"f3d99325-48ff-403c-8662-1d7745f7c97d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6804","nonce":2766567910}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6805","nonce":2766567910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6807","nonce":2766567910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6806","nonce":2766567910}]},"public_addr":"172.21.15.1:6804/2766567910","cluster_addr":"172.21.15.1:6805/2766567910","heartbeat_back_addr":"172.21.15.1:6807/2766567910","heartbeat_front_addr":"172.21.15.1:6806/2766567910","state":["exists","up"]},{"osd":9,"uuid":"88511e24-3675-4551-968c-1a52e7e28f96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":244,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.92:6812","nonce":2488187511}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.92:6813","nonce":2488187511}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.92:6815","nonce":2488187511}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.92:6814","nonce":2488187511}]},"public_addr":"172.21.15.92:6812/2488187511","cluster_addr":"172.21.15.92:6813/2488187511","heartbeat_back_addr":"172.21.15.92:6815/2488187511","heartbeat_front_addr":"172.21.15.92:6814/2488187511","state":["exists","up"]},{"osd":10,"uuid":"acf4c64d-99af-42be-8406-413c86f54670","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":22,"last_clean_end":238,"up_from":261,"up_thru":266,"down_at":239,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.162:6804","nonce":4258911154}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.162:6805","nonce":4258911154}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.162:6807","nonce":4258911154}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.162:6806","nonce":4258911154}]},"public_addr":"172.21.15.162:6804/4258911154","cluster_addr":"172.21.15.162:6805/4258911154","heartbeat_back_addr":"172.21.15.162:6807/4258911154","heartbeat_front_addr":"172.21.15.162:6806/4258911154","state":["exists","up"]},{"osd":11,"uuid":"a4bfdfbe-b586-4080-8e09-6f26428e59b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":247,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.177:6809","nonce":3239521954}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.177:6810","nonce":3239521954}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.177:6812","nonce":3239521954}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.177:6811","nonce":3239521954}]},"public_addr":"172.21.15.177:6809/3239521954","cluster_addr":"172.21.15.177:6810/3239521954","heartbeat_back_addr":"172.21.15.177:6812/3239521954","heartbeat_front_addr":"172.21.15.177:6811/3239521954","state":["exists","up"]},{"osd":12,"uuid":"dc571fb6-d8b5-465b-bb25-afa7d0c47b5a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":249,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6812","nonce":4267062742}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6813","nonce":4267062742}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6815","nonce":4267062742}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6814","nonce":4267062742}]},"public_addr":"172.21.15.1:6812/4267062742","cluster_addr":"172.21.15.1:6813/4267062742","heartbeat_back_addr":"172.21.15.1:6815/4267062742","heartbeat_front_addr":"172.21.15.1:6814/4267062742","state":["exists","up"]},{"osd":13,"uuid":"5d321d4b-1b26-47fe-bf84-25751b8cde7d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":235,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.92:6808","nonce":1759585046}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.92:6809","nonce":1759585046}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.92:6811","nonce":1759585046}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.92:6810","nonce":1759585046}]},"public_addr":"172.21.15.92:6808/1759585046","cluster_addr":"172.21.15.92:6809/1759585046","heartbeat_back_addr":"172.21.15.92:6811/1759585046","heartbeat_front_addr":"172.21.15.92:6810/1759585046","state":["exists","up"]},{"osd":14,"uuid":"fc3e2499-0c2a-414d-a25e-2628876ae630","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":22,"last_clean_end":120,"up_from":142,"up_thru":296,"down_at":121,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.162:6800","nonce":3999392664}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.162:6801","nonce":3999392664}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.162:6803","nonce":3999392664}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.162:6802","nonce":3999392664}]},"public_addr":"172.21.15.162:6800/3999392664","cluster_addr":"172.21.15.162:6801/3999392664","heartbeat_back_addr":"172.21.15.162:6803/3999392664","heartbeat_front_addr":"172.21.15.162:6802/3999392664","state":["exists","up"]},{"osd":15,"uuid":"d2f69775-edfe-4527-91ec-f7c7e558bec8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":292,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.177:6805","nonce":1294337172}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.177:6806","nonce":1294337172}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.177:6808","nonce":1294337172}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.177:6807","nonce":1294337172}]},"public_addr":"172.21.15.177:6805/1294337172","cluster_addr":"172.21.15.177:6806/1294337172","heartbeat_back_addr":"172.21.15.177:6808/1294337172","heartbeat_front_addr":"172.21.15.177:6807/1294337172","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T11:27:40.342776+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T11:27:07.613933+0000","dead_epoch":30},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T11:27:22.828451+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T11:27:21.614437+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T11:27:25.568873+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T11:27:28.185934+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T11:27:46.757486+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T11:27:21.994608+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T11:27:44.589810+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T11:27:20.603346+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T11:27:37.802899+0000","dead_epoch":239},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T11:27:44.791122+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T11:26:59.682735+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T11:28:15.562220+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T11:27:06.849942+0000","dead_epoch":121},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T11:27:15.020976+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T11:28:18.902 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-09-13T11:28:18.903 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-09-13T11:28:18.903 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-09-13T11:28:18.903 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-09-13T11:28:18.904 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2024-09-13T11:28:18.904 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2024-09-13T11:28:18.905 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2024-09-13T11:28:18.906 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2024-09-13T11:28:18.906 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.8 flush_pg_stats 2024-09-13T11:28:18.907 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.9 flush_pg_stats 2024-09-13T11:28:18.907 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.10 flush_pg_stats 2024-09-13T11:28:18.908 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.11 flush_pg_stats 2024-09-13T11:28:18.908 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.12 flush_pg_stats 2024-09-13T11:28:18.909 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.13 flush_pg_stats 2024-09-13T11:28:18.910 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.14 flush_pg_stats 2024-09-13T11:28:18.910 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.15 flush_pg_stats 2024-09-13T11:28:19.335 INFO:teuthology.orchestra.run.smithi001.stdout:94489280589 2024-09-13T11:28:19.336 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-13T11:28:19.341 INFO:teuthology.orchestra.run.smithi001.stdout:171798691908 2024-09-13T11:28:19.341 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-13T11:28:19.372 INFO:teuthology.orchestra.run.smithi001.stdout:609885356074 2024-09-13T11:28:19.373 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.14 2024-09-13T11:28:19.380 INFO:teuthology.orchestra.run.smithi001.stdout:94489280589 2024-09-13T11:28:19.380 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.12 2024-09-13T11:28:19.390 INFO:teuthology.orchestra.run.smithi001.stdout:94489280590 2024-09-13T11:28:19.390 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-09-13T11:28:19.393 INFO:teuthology.orchestra.run.smithi001.stdout:98784247885 2024-09-13T11:28:19.394 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.9 2024-09-13T11:28:19.398 INFO:teuthology.orchestra.run.smithi001.stdout:1120986464266 2024-09-13T11:28:19.399 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.10 2024-09-13T11:28:19.430 INFO:teuthology.orchestra.run.smithi001.stdout:90194313296 2024-09-13T11:28:19.431 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-09-13T11:28:19.476 INFO:teuthology.orchestra.run.smithi001.stdout:90194313293 2024-09-13T11:28:19.477 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-13T11:28:19.477 INFO:teuthology.orchestra.run.smithi001.stdout:94489280589 2024-09-13T11:28:19.478 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.15 2024-09-13T11:28:19.496 INFO:teuthology.orchestra.run.smithi001.stdout:94489280590 2024-09-13T11:28:19.497 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.11 2024-09-13T11:28:19.500 INFO:teuthology.orchestra.run.smithi001.stdout:98784247886 2024-09-13T11:28:19.500 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.13 2024-09-13T11:28:19.506 INFO:teuthology.orchestra.run.smithi001.stdout:94489280589 2024-09-13T11:28:19.507 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-13T11:28:19.509 INFO:teuthology.orchestra.run.smithi001.stdout:94489280589 2024-09-13T11:28:19.509 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.8 2024-09-13T11:28:19.513 INFO:teuthology.orchestra.run.smithi001.stdout:94489280589 2024-09-13T11:28:19.513 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-09-13T11:28:19.526 INFO:teuthology.orchestra.run.smithi001.stdout:94489280590 2024-09-13T11:28:19.526 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-09-13T11:28:20.168 INFO:tasks.ceph.osd.14.smithi162.stderr:2024-09-13T11:28:20.162+0000 7f3dea330640 -1 osd.14 298 handle_osd_map injecting map crc failure 2024-09-13T11:28:20.169 INFO:teuthology.orchestra.run.smithi001.stdout:98784247884 2024-09-13T11:28:20.191 INFO:teuthology.orchestra.run.smithi001.stdout:609885356073 2024-09-13T11:28:20.214 INFO:tasks.ceph.ceph_manager.ceph:need seq 98784247885 got 98784247884 for osd.9 2024-09-13T11:28:20.254 INFO:tasks.ceph.ceph_manager.ceph:need seq 609885356074 got 609885356073 for osd.14 2024-09-13T11:28:20.335 INFO:tasks.ceph.osd.11.smithi177.stderr:2024-09-13T11:28:20.336+0000 7f2469cec640 -1 osd.11 298 handle_osd_map injecting map crc failure 2024-09-13T11:28:20.356 INFO:teuthology.orchestra.run.smithi001.stdout:94489280590 2024-09-13T11:28:20.379 INFO:teuthology.orchestra.run.smithi001.stdout:90194313296 2024-09-13T11:28:20.394 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280590 got 94489280590 for osd.7 2024-09-13T11:28:20.394 DEBUG:teuthology.parallel:result is None 2024-09-13T11:28:20.405 INFO:teuthology.orchestra.run.smithi001.stdout:94489280589 2024-09-13T11:28:20.433 INFO:tasks.ceph.ceph_manager.ceph:need seq 90194313296 got 90194313296 for osd.5 2024-09-13T11:28:20.433 DEBUG:teuthology.parallel:result is None 2024-09-13T11:28:20.438 INFO:teuthology.orchestra.run.smithi001.stdout:94489280589 2024-09-13T11:28:20.450 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280589 got 94489280589 for osd.2 2024-09-13T11:28:20.450 DEBUG:teuthology.parallel:result is None 2024-09-13T11:28:20.452 INFO:teuthology.orchestra.run.smithi001.stdout:171798691908 2024-09-13T11:28:20.467 INFO:teuthology.orchestra.run.smithi001.stdout:90194313294 2024-09-13T11:28:20.475 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280589 got 94489280589 for osd.12 2024-09-13T11:28:20.475 DEBUG:teuthology.parallel:result is None 2024-09-13T11:28:20.505 INFO:tasks.ceph.ceph_manager.ceph:need seq 171798691908 got 171798691908 for osd.1 2024-09-13T11:28:20.505 DEBUG:teuthology.parallel:result is None 2024-09-13T11:28:20.507 INFO:tasks.ceph.ceph_manager.ceph:need seq 90194313293 got 90194313294 for osd.3 2024-09-13T11:28:20.508 DEBUG:teuthology.parallel:result is None 2024-09-13T11:28:20.515 INFO:teuthology.orchestra.run.smithi001.stdout:94489280590 2024-09-13T11:28:20.539 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280590 got 94489280590 for osd.11 2024-09-13T11:28:20.539 DEBUG:teuthology.parallel:result is None 2024-09-13T11:28:20.557 INFO:teuthology.orchestra.run.smithi001.stdout:1120986464266 2024-09-13T11:28:20.568 INFO:teuthology.orchestra.run.smithi001.stdout:98784247886 2024-09-13T11:28:20.569 INFO:teuthology.orchestra.run.smithi001.stdout:94489280590 2024-09-13T11:28:20.577 INFO:tasks.ceph.ceph_manager.ceph:need seq 1120986464266 got 1120986464266 for osd.10 2024-09-13T11:28:20.577 DEBUG:teuthology.parallel:result is None 2024-09-13T11:28:20.579 INFO:teuthology.orchestra.run.smithi001.stdout:94489280589 2024-09-13T11:28:20.601 INFO:tasks.ceph.ceph_manager.ceph:need seq 98784247886 got 98784247886 for osd.13 2024-09-13T11:28:20.602 DEBUG:teuthology.parallel:result is None 2024-09-13T11:28:20.602 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280590 got 94489280590 for osd.4 2024-09-13T11:28:20.602 DEBUG:teuthology.parallel:result is None 2024-09-13T11:28:20.603 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280589 got 94489280589 for osd.15 2024-09-13T11:28:20.603 DEBUG:teuthology.parallel:result is None 2024-09-13T11:28:20.610 INFO:teuthology.orchestra.run.smithi001.stdout:94489280589 2024-09-13T11:28:20.613 INFO:teuthology.orchestra.run.smithi001.stdout:94489280589 2024-09-13T11:28:20.630 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280589 got 94489280589 for osd.0 2024-09-13T11:28:20.630 DEBUG:teuthology.parallel:result is None 2024-09-13T11:28:20.632 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280589 got 94489280589 for osd.8 2024-09-13T11:28:20.632 DEBUG:teuthology.parallel:result is None 2024-09-13T11:28:20.638 INFO:teuthology.orchestra.run.smithi001.stdout:94489280589 2024-09-13T11:28:20.655 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280589 got 94489280589 for osd.6 2024-09-13T11:28:20.655 DEBUG:teuthology.parallel:result is None 2024-09-13T11:28:21.214 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.9 2024-09-13T11:28:21.255 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.14 2024-09-13T11:28:21.541 INFO:teuthology.orchestra.run.smithi001.stdout:98784247885 2024-09-13T11:28:21.560 INFO:tasks.ceph.ceph_manager.ceph:need seq 98784247885 got 98784247885 for osd.9 2024-09-13T11:28:21.560 DEBUG:teuthology.parallel:result is None 2024-09-13T11:28:21.570 INFO:teuthology.orchestra.run.smithi001.stdout:609885356075 2024-09-13T11:28:21.587 INFO:tasks.ceph.ceph_manager.ceph:need seq 609885356074 got 609885356075 for osd.14 2024-09-13T11:28:21.587 DEBUG:teuthology.parallel:result is None 2024-09-13T11:28:21.587 INFO:tasks.ceph.ceph_manager.ceph:waiting for recovery to complete 2024-09-13T11:28:21.588 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-13T11:28:21.931 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-13T11:28:21.931 INFO:teuthology.orchestra.run.smithi001.stderr:dumped all 2024-09-13T11:28:21.948 INFO:teuthology.orchestra.run.smithi001.stdout:{"pg_ready":true,"pg_map":{"version":470,"stamp":"2024-09-13T11:28:20.209692+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":147,"num_write_kb":2674,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":115,"ondisk_log_size":115,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":10,"kb":1509949440,"kb_used":1964212,"kb_used_data":15092,"kb_used_omap":69,"kb_used_meta":1948794,"kb_avail":1507985228,"statfs":{"total":1546188226560,"available":1544176873472,"internally_reserved":0,"allocated":15454208,"data_stored":9894824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":71147,"internal_metadata":1995565589},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":20,"apply_latency_ms":20,"commit_latency_ns":20000000,"apply_latency_ns":20000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.016458"},"pg_stats":[{"pgid":"1.0","version":"26'112","reported_seq":723,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-09-13T11:28:18.740419+0000","last_change":"2024-09-13T11:27:51.016799+0000","last_active":"2024-09-13T11:28:18.740419+0000","last_peered":"2024-09-13T11:28:18.740419+0000","last_clean":"2024-09-13T11:28:18.740419+0000","last_became_active":"2024-09-13T11:22:12.129721+0000","last_became_peered":"2024-09-13T11:22:12.129721+0000","last_unstale":"2024-09-13T11:28:18.740419+0000","last_undegraded":"2024-09-13T11:28:18.740419+0000","last_fullsized":"2024-09-13T11:28:18.740419+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"26'112","last_scrub_stamp":"2024-09-13T11:27:51.016625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:11.098388+0000","last_clean_scrub_stamp":"2024-09-13T11:27:51.016625+0000","objects_scrubbed":2,"log_size":112,"log_dups_size":0,"ondisk_log_size":112,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:29:12.312000+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.3","version":"26'1","reported_seq":232,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-09-13T11:28:18.220535+0000","last_change":"2024-09-13T11:28:17.223911+0000","last_active":"2024-09-13T11:28:18.220535+0000","last_peered":"2024-09-13T11:28:18.220535+0000","last_clean":"2024-09-13T11:28:18.220535+0000","last_became_active":"2024-09-13T11:28:17.223376+0000","last_became_peered":"2024-09-13T11:28:17.223376+0000","last_unstale":"2024-09-13T11:28:18.220535+0000","last_undegraded":"2024-09-13T11:28:18.220535+0000","last_fullsized":"2024-09-13T11:28:18.220535+0000","mapping_epoch":296,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":297,"parent":"0.0","parent_split_bits":0,"last_scrub":"26'1","last_scrub_stamp":"2024-09-13T11:27:25.752752+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:27:25.752752+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:28:40.883396+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,15],"acting":[14,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":578,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-09-13T11:28:18.687300+0000","last_change":"2024-09-13T11:27:27.219367+0000","last_active":"2024-09-13T11:28:18.687300+0000","last_peered":"2024-09-13T11:28:18.687300+0000","last_clean":"2024-09-13T11:28:18.687300+0000","last_became_active":"2024-09-13T11:22:14.211435+0000","last_became_peered":"2024-09-13T11:22:14.211435+0000","last_unstale":"2024-09-13T11:28:18.687300+0000","last_undegraded":"2024-09-13T11:28:18.687300+0000","last_fullsized":"2024-09-13T11:28:18.687300+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:27:27.219203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:27:27.219203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:28:30.848078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010010360000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":230,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-09-13T11:28:18.220511+0000","last_change":"2024-09-13T11:27:26.709818+0000","last_active":"2024-09-13T11:28:18.220511+0000","last_peered":"2024-09-13T11:28:18.220511+0000","last_clean":"2024-09-13T11:28:18.220511+0000","last_became_active":"2024-09-13T11:26:05.046717+0000","last_became_peered":"2024-09-13T11:26:05.046717+0000","last_unstale":"2024-09-13T11:28:18.220511+0000","last_undegraded":"2024-09-13T11:28:18.220511+0000","last_fullsized":"2024-09-13T11:28:18.220511+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:27:26.709721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:27:26.709721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:28:41.582217+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00081225399999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,1],"acting":[14,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"2.2","version":"28'2","reported_seq":187,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-09-13T11:28:18.687303+0000","last_change":"2024-09-13T11:27:25.229244+0000","last_active":"2024-09-13T11:28:18.687303+0000","last_peered":"2024-09-13T11:28:18.687303+0000","last_clean":"2024-09-13T11:28:18.687303+0000","last_became_active":"2024-09-13T11:26:36.267081+0000","last_became_peered":"2024-09-13T11:26:36.267081+0000","last_unstale":"2024-09-13T11:28:18.687303+0000","last_undegraded":"2024-09-13T11:28:18.687303+0000","last_fullsized":"2024-09-13T11:28:18.687303+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2","last_scrub_stamp":"2024-09-13T11:27:25.229120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:27:25.229120+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:28:40.171329+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0025826730000000002,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9],"acting":[3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":570,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-13T11:28:19.235519+0000","last_change":"2024-09-13T11:27:26.521004+0000","last_active":"2024-09-13T11:28:19.235519+0000","last_peered":"2024-09-13T11:28:19.235519+0000","last_clean":"2024-09-13T11:28:19.235519+0000","last_became_active":"2024-09-13T11:22:47.719636+0000","last_became_peered":"2024-09-13T11:22:47.719636+0000","last_unstale":"2024-09-13T11:28:19.235519+0000","last_undegraded":"2024-09-13T11:28:19.235519+0000","last_fullsized":"2024-09-13T11:28:19.235519+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:27:26.520834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:27:26.520834+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:28:32.826137+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00087476000000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,9],"acting":[1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":585,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-13T11:28:19.232406+0000","last_change":"2024-09-13T11:26:58.241079+0000","last_active":"2024-09-13T11:28:19.232406+0000","last_peered":"2024-09-13T11:28:19.232406+0000","last_clean":"2024-09-13T11:28:19.232406+0000","last_became_active":"2024-09-13T11:22:14.581458+0000","last_became_peered":"2024-09-13T11:22:14.581458+0000","last_unstale":"2024-09-13T11:28:19.232406+0000","last_undegraded":"2024-09-13T11:28:19.232406+0000","last_fullsized":"2024-09-13T11:28:19.232406+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:26:58.240985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:26:58.240985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:28:23.342920+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.001026047,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5],"acting":[8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":589,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-09-13T11:28:18.220145+0000","last_change":"2024-09-13T11:27:58.082895+0000","last_active":"2024-09-13T11:28:18.220145+0000","last_peered":"2024-09-13T11:28:18.220145+0000","last_clean":"2024-09-13T11:28:18.220145+0000","last_became_active":"2024-09-13T11:22:14.211923+0000","last_became_peered":"2024-09-13T11:22:14.211923+0000","last_unstale":"2024-09-13T11:28:18.220145+0000","last_undegraded":"2024-09-13T11:28:18.220145+0000","last_fullsized":"2024-09-13T11:28:18.220145+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:27:58.082738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:27:58.082738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:29:11.583617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011689529999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0],"acting":[9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":420,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-09-13T11:28:18.688180+0000","last_change":"2024-09-13T11:28:13.159189+0000","last_active":"2024-09-13T11:28:18.688180+0000","last_peered":"2024-09-13T11:28:18.688180+0000","last_clean":"2024-09-13T11:28:18.688180+0000","last_became_active":"2024-09-13T11:28:13.158553+0000","last_became_peered":"2024-09-13T11:28:13.158553+0000","last_unstale":"2024-09-13T11:28:18.688180+0000","last_undegraded":"2024-09-13T11:28:18.688180+0000","last_fullsized":"2024-09-13T11:28:18.688180+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:27:27.561975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:27:27.561975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:28:56.783029+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,10],"acting":[15,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":11},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":112,"ondisk_log_size":112,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":13,"up_from":23,"seq":98784247886,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":100120,"kb_used_data":904,"kb_used_omap":1,"kb_used_meta":99198,"kb_avail":94271720,"statfs":{"total":96636764160,"available":96534241280,"internally_reserved":0,"allocated":925696,"data_stored":563618,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":101579205},"hb_peers":[0,1,2,4,6,8,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":9,"up_from":23,"seq":98784247885,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":94520,"kb_used_data":872,"kb_used_omap":1,"kb_used_meta":93630,"kb_avail":94277320,"statfs":{"total":96636764160,"available":96539975680,"internally_reserved":0,"allocated":892928,"data_stored":540962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":95877568},"hb_peers":[0,3,4,8,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":40,"seq":171798691908,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":141164,"kb_used_data":860,"kb_used_omap":4,"kb_used_meta":140283,"kb_avail":94230676,"statfs":{"total":96636764160,"available":96492212224,"internally_reserved":0,"allocated":880640,"data_stored":540555,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4433,"internal_metadata":143650479},"hb_peers":[0,2,4,7,8,9,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":21,"seq":90194313294,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":158396,"kb_used_data":876,"kb_used_omap":1,"kb_used_meta":157502,"kb_avail":94213444,"statfs":{"total":96636764160,"available":96474566656,"internally_reserved":0,"allocated":897024,"data_stored":540669,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":161282503},"hb_peers":[0,2,4,9,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":21,"seq":90194313296,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":166684,"kb_used_data":844,"kb_used_omap":1,"kb_used_meta":165822,"kb_avail":94205156,"statfs":{"total":96636764160,"available":96466079744,"internally_reserved":0,"allocated":864256,"data_stored":539135,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":169802183},"hb_peers":[0,2,4,6,7,8,9,11,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":22,"seq":94489280589,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":177016,"kb_used_data":1448,"kb_used_omap":1,"kb_used_meta":175550,"kb_avail":94194824,"statfs":{"total":96636764160,"available":96455499776,"internally_reserved":0,"allocated":1482752,"data_stored":1130786,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":179763653},"hb_peers":[1,3,5,8,9,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":22,"seq":94489280589,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":94496,"kb_used_data":848,"kb_used_omap":1,"kb_used_meta":93630,"kb_avail":94277344,"statfs":{"total":96636764160,"available":96540000256,"internally_reserved":0,"allocated":868352,"data_stored":538858,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":95877573},"hb_peers":[0,1,3,4,7,8,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":15,"up_from":22,"seq":94489280589,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":144492,"kb_used_data":860,"kb_used_omap":1,"kb_used_meta":143614,"kb_avail":94227348,"statfs":{"total":96636764160,"available":96488804352,"internally_reserved":0,"allocated":880640,"data_stored":538837,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":147061190},"hb_peers":[0,4,5,6,7,8,10,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":22,"seq":94489280590,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":86200,"kb_used_data":872,"kb_used_omap":1,"kb_used_meta":85310,"kb_avail":94285640,"statfs":{"total":96636764160,"available":96548495360,"internally_reserved":0,"allocated":892928,"data_stored":541293,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":87357893},"hb_peers":[3,5,6,7,9,10,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":22,"seq":94489280589,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72868,"kb_used_data":1428,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":94298972,"statfs":{"total":96636764160,"available":96562147328,"internally_reserved":0,"allocated":1462272,"data_stored":1129416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":73136582},"hb_peers":[0,1,2,3,5,7,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":7,"up_from":22,"seq":94489280590,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":162560,"kb_used_data":880,"kb_used_omap":1,"kb_used_meta":161662,"kb_avail":94209280,"statfs":{"total":96636764160,"available":96470302720,"internally_reserved":0,"allocated":901120,"data_stored":562723,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":165542346},"hb_peers":[0,2,5,6,8,9,11,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":8,"up_from":22,"seq":94489280589,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":100076,"kb_used_data":860,"kb_used_omap":1,"kb_used_meta":99198,"kb_avail":94271764,"statfs":{"total":96636764160,"available":96534286336,"internally_reserved":0,"allocated":880640,"data_stored":540086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":101579205},"hb_peers":[1,2,5,7,9,10,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":10,"up_from":261,"seq":1120986464266,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35564,"kb_used_data":924,"kb_used_omap":2,"kb_used_meta":34621,"kb_avail":94336276,"statfs":{"total":96636764160,"available":96600346624,"internally_reserved":0,"allocated":946176,"data_stored":565968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2702,"internal_metadata":35452274},"hb_peers":[0,1,3,7,9,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":11,"up_from":22,"seq":94489280590,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":154240,"kb_used_data":880,"kb_used_omap":1,"kb_used_meta":153342,"kb_avail":94217600,"statfs":{"total":96636764160,"available":96478822400,"internally_reserved":0,"allocated":901120,"data_stored":541670,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":157022660},"hb_peers":[2,4,7,8,9,10,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":12,"up_from":22,"seq":94489280589,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":166704,"kb_used_data":864,"kb_used_omap":1,"kb_used_meta":165822,"kb_avail":94205136,"statfs":{"total":96636764160,"available":96466059264,"internally_reserved":0,"allocated":884736,"data_stored":540398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":169802181},"hb_peers":[1,2,3,5,6,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":14,"up_from":142,"seq":609885356075,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":109112,"kb_used_data":872,"kb_used_omap":42,"kb_used_meta":108181,"kb_avail":94262728,"statfs":{"total":96636764160,"available":96525033472,"internally_reserved":0,"allocated":892928,"data_stored":539850,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":43282,"internal_metadata":110778094},"hb_peers":[0,1,4,5,7,9,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T11:28:21.949 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-13T11:28:22.239 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-13T11:28:22.239 INFO:teuthology.orchestra.run.smithi001.stderr:dumped all 2024-09-13T11:28:22.256 INFO:teuthology.orchestra.run.smithi001.stdout:{"pg_ready":true,"pg_map":{"version":472,"stamp":"2024-09-13T11:28:22.210466+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":147,"num_write_kb":2674,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":115,"ondisk_log_size":115,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":10,"kb":1509949440,"kb_used":1964276,"kb_used_data":15156,"kb_used_omap":69,"kb_used_meta":1948794,"kb_avail":1507985164,"statfs":{"total":1546188226560,"available":1544176807936,"internally_reserved":0,"allocated":15519744,"data_stored":9922474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":71147,"internal_metadata":1995565589},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.001914"},"pg_stats":[{"pgid":"1.0","version":"26'112","reported_seq":725,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-13T11:28:20.440039+0000","last_change":"2024-09-13T11:27:51.016799+0000","last_active":"2024-09-13T11:28:20.440039+0000","last_peered":"2024-09-13T11:28:20.440039+0000","last_clean":"2024-09-13T11:28:20.440039+0000","last_became_active":"2024-09-13T11:22:12.129721+0000","last_became_peered":"2024-09-13T11:22:12.129721+0000","last_unstale":"2024-09-13T11:28:20.440039+0000","last_undegraded":"2024-09-13T11:28:20.440039+0000","last_fullsized":"2024-09-13T11:28:20.440039+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"26'112","last_scrub_stamp":"2024-09-13T11:27:51.016625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:11.098388+0000","last_clean_scrub_stamp":"2024-09-13T11:27:51.016625+0000","objects_scrubbed":2,"log_size":112,"log_dups_size":0,"ondisk_log_size":112,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:29:12.312000+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.3","version":"26'1","reported_seq":232,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-09-13T11:28:18.220535+0000","last_change":"2024-09-13T11:28:17.223911+0000","last_active":"2024-09-13T11:28:18.220535+0000","last_peered":"2024-09-13T11:28:18.220535+0000","last_clean":"2024-09-13T11:28:18.220535+0000","last_became_active":"2024-09-13T11:28:17.223376+0000","last_became_peered":"2024-09-13T11:28:17.223376+0000","last_unstale":"2024-09-13T11:28:18.220535+0000","last_undegraded":"2024-09-13T11:28:18.220535+0000","last_fullsized":"2024-09-13T11:28:18.220535+0000","mapping_epoch":296,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":297,"parent":"0.0","parent_split_bits":0,"last_scrub":"26'1","last_scrub_stamp":"2024-09-13T11:27:25.752752+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:27:25.752752+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:28:40.883396+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,15],"acting":[14,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":578,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-09-13T11:28:18.687300+0000","last_change":"2024-09-13T11:27:27.219367+0000","last_active":"2024-09-13T11:28:18.687300+0000","last_peered":"2024-09-13T11:28:18.687300+0000","last_clean":"2024-09-13T11:28:18.687300+0000","last_became_active":"2024-09-13T11:22:14.211435+0000","last_became_peered":"2024-09-13T11:22:14.211435+0000","last_unstale":"2024-09-13T11:28:18.687300+0000","last_undegraded":"2024-09-13T11:28:18.687300+0000","last_fullsized":"2024-09-13T11:28:18.687300+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:27:27.219203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:27:27.219203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:28:30.848078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010010360000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":230,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-09-13T11:28:18.220511+0000","last_change":"2024-09-13T11:27:26.709818+0000","last_active":"2024-09-13T11:28:18.220511+0000","last_peered":"2024-09-13T11:28:18.220511+0000","last_clean":"2024-09-13T11:28:18.220511+0000","last_became_active":"2024-09-13T11:26:05.046717+0000","last_became_peered":"2024-09-13T11:26:05.046717+0000","last_unstale":"2024-09-13T11:28:18.220511+0000","last_undegraded":"2024-09-13T11:28:18.220511+0000","last_fullsized":"2024-09-13T11:28:18.220511+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:27:26.709721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:27:26.709721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:28:41.582217+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00081225399999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,1],"acting":[14,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"2.2","version":"28'2","reported_seq":187,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-09-13T11:28:18.687303+0000","last_change":"2024-09-13T11:27:25.229244+0000","last_active":"2024-09-13T11:28:18.687303+0000","last_peered":"2024-09-13T11:28:18.687303+0000","last_clean":"2024-09-13T11:28:18.687303+0000","last_became_active":"2024-09-13T11:26:36.267081+0000","last_became_peered":"2024-09-13T11:26:36.267081+0000","last_unstale":"2024-09-13T11:28:18.687303+0000","last_undegraded":"2024-09-13T11:28:18.687303+0000","last_fullsized":"2024-09-13T11:28:18.687303+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2","last_scrub_stamp":"2024-09-13T11:27:25.229120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:27:25.229120+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:28:40.171329+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0025826730000000002,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9],"acting":[3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":570,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-13T11:28:19.235519+0000","last_change":"2024-09-13T11:27:26.521004+0000","last_active":"2024-09-13T11:28:19.235519+0000","last_peered":"2024-09-13T11:28:19.235519+0000","last_clean":"2024-09-13T11:28:19.235519+0000","last_became_active":"2024-09-13T11:22:47.719636+0000","last_became_peered":"2024-09-13T11:22:47.719636+0000","last_unstale":"2024-09-13T11:28:19.235519+0000","last_undegraded":"2024-09-13T11:28:19.235519+0000","last_fullsized":"2024-09-13T11:28:19.235519+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:27:26.520834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:27:26.520834+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:28:32.826137+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00087476000000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,9],"acting":[1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":585,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-13T11:28:19.232406+0000","last_change":"2024-09-13T11:26:58.241079+0000","last_active":"2024-09-13T11:28:19.232406+0000","last_peered":"2024-09-13T11:28:19.232406+0000","last_clean":"2024-09-13T11:28:19.232406+0000","last_became_active":"2024-09-13T11:22:14.581458+0000","last_became_peered":"2024-09-13T11:22:14.581458+0000","last_unstale":"2024-09-13T11:28:19.232406+0000","last_undegraded":"2024-09-13T11:28:19.232406+0000","last_fullsized":"2024-09-13T11:28:19.232406+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:26:58.240985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:26:58.240985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:28:23.342920+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.001026047,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5],"acting":[8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":591,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-13T11:28:19.786575+0000","last_change":"2024-09-13T11:27:58.082895+0000","last_active":"2024-09-13T11:28:19.786575+0000","last_peered":"2024-09-13T11:28:19.786575+0000","last_clean":"2024-09-13T11:28:19.786575+0000","last_became_active":"2024-09-13T11:22:14.211923+0000","last_became_peered":"2024-09-13T11:22:14.211923+0000","last_unstale":"2024-09-13T11:28:19.786575+0000","last_undegraded":"2024-09-13T11:28:19.786575+0000","last_fullsized":"2024-09-13T11:28:19.786575+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:27:58.082738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:27:58.082738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:29:11.583617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011689529999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0],"acting":[9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":422,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-13T11:28:20.170034+0000","last_change":"2024-09-13T11:28:13.159189+0000","last_active":"2024-09-13T11:28:20.170034+0000","last_peered":"2024-09-13T11:28:20.170034+0000","last_clean":"2024-09-13T11:28:20.170034+0000","last_became_active":"2024-09-13T11:28:13.158553+0000","last_became_peered":"2024-09-13T11:28:13.158553+0000","last_unstale":"2024-09-13T11:28:20.170034+0000","last_undegraded":"2024-09-13T11:28:20.170034+0000","last_fullsized":"2024-09-13T11:28:20.170034+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:27:27.561975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:27:27.561975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:28:56.783029+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,10],"acting":[15,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":11},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":112,"ondisk_log_size":112,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":13,"up_from":23,"seq":98784247887,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":100096,"kb_used_data":880,"kb_used_omap":1,"kb_used_meta":99198,"kb_avail":94271744,"statfs":{"total":96636764160,"available":96534265856,"internally_reserved":0,"allocated":901120,"data_stored":546978,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":101579205},"hb_peers":[0,1,2,4,6,8,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":9,"up_from":23,"seq":98784247886,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":94532,"kb_used_data":884,"kb_used_omap":1,"kb_used_meta":93630,"kb_avail":94277308,"statfs":{"total":96636764160,"available":96539963392,"internally_reserved":0,"allocated":905216,"data_stored":547960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":95877568},"hb_peers":[0,3,4,8,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":40,"seq":171798691909,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":141176,"kb_used_data":872,"kb_used_omap":4,"kb_used_meta":140283,"kb_avail":94230664,"statfs":{"total":96636764160,"available":96492199936,"internally_reserved":0,"allocated":892928,"data_stored":547553,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4433,"internal_metadata":143650479},"hb_peers":[0,2,4,7,8,9,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":21,"seq":90194313294,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":158396,"kb_used_data":876,"kb_used_omap":1,"kb_used_meta":157502,"kb_avail":94213444,"statfs":{"total":96636764160,"available":96474566656,"internally_reserved":0,"allocated":897024,"data_stored":540669,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":161282503},"hb_peers":[0,2,4,9,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":21,"seq":90194313297,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":166696,"kb_used_data":856,"kb_used_omap":1,"kb_used_meta":165822,"kb_avail":94205144,"statfs":{"total":96636764160,"available":96466067456,"internally_reserved":0,"allocated":876544,"data_stored":546133,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":169802183},"hb_peers":[0,2,4,6,7,8,9,11,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":22,"seq":94489280590,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":177028,"kb_used_data":1460,"kb_used_omap":1,"kb_used_meta":175550,"kb_avail":94194812,"statfs":{"total":96636764160,"available":96455487488,"internally_reserved":0,"allocated":1495040,"data_stored":1137784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":179763653},"hb_peers":[1,3,5,8,9,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":22,"seq":94489280590,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":94508,"kb_used_data":860,"kb_used_omap":1,"kb_used_meta":93630,"kb_avail":94277332,"statfs":{"total":96636764160,"available":96539987968,"internally_reserved":0,"allocated":880640,"data_stored":545856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":95877573},"hb_peers":[0,1,3,4,7,8,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":15,"up_from":22,"seq":94489280590,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":144504,"kb_used_data":872,"kb_used_omap":1,"kb_used_meta":143614,"kb_avail":94227336,"statfs":{"total":96636764160,"available":96488792064,"internally_reserved":0,"allocated":892928,"data_stored":545835,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":147061190},"hb_peers":[0,4,5,6,7,8,10,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":22,"seq":94489280591,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":86212,"kb_used_data":884,"kb_used_omap":1,"kb_used_meta":85310,"kb_avail":94285628,"statfs":{"total":96636764160,"available":96548483072,"internally_reserved":0,"allocated":905216,"data_stored":548291,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":87357893},"hb_peers":[3,5,6,7,9,10,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":22,"seq":94489280590,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72880,"kb_used_data":1440,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":94298960,"statfs":{"total":96636764160,"available":96562135040,"internally_reserved":0,"allocated":1474560,"data_stored":1136414,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":73136582},"hb_peers":[0,1,2,3,5,7,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":22,"seq":94489280591,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":162544,"kb_used_data":864,"kb_used_omap":1,"kb_used_meta":161662,"kb_avail":94209296,"statfs":{"total":96636764160,"available":96470319104,"internally_reserved":0,"allocated":884736,"data_stored":546969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":165542346},"hb_peers":[0,2,5,6,8,9,11,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":22,"seq":94489280590,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":100088,"kb_used_data":872,"kb_used_omap":1,"kb_used_meta":99198,"kb_avail":94271752,"statfs":{"total":96636764160,"available":96534274048,"internally_reserved":0,"allocated":892928,"data_stored":547084,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":101579205},"hb_peers":[1,2,5,7,9,10,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":261,"seq":1120986464267,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35540,"kb_used_data":900,"kb_used_omap":2,"kb_used_meta":34621,"kb_avail":94336300,"statfs":{"total":96636764160,"available":96600371200,"internally_reserved":0,"allocated":921600,"data_stored":549328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2702,"internal_metadata":35452274},"hb_peers":[0,1,3,7,9,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":11,"up_from":22,"seq":94489280591,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":154248,"kb_used_data":888,"kb_used_omap":1,"kb_used_meta":153342,"kb_avail":94217592,"statfs":{"total":96636764160,"available":96478814208,"internally_reserved":0,"allocated":909312,"data_stored":548374,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":157022660},"hb_peers":[2,4,7,8,9,10,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":12,"up_from":22,"seq":94489280590,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":166716,"kb_used_data":876,"kb_used_omap":1,"kb_used_meta":165822,"kb_avail":94205124,"statfs":{"total":96636764160,"available":96466046976,"internally_reserved":0,"allocated":897024,"data_stored":547396,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":169802181},"hb_peers":[1,2,3,5,6,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":14,"up_from":142,"seq":609885356075,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":109112,"kb_used_data":872,"kb_used_omap":42,"kb_used_meta":108181,"kb_avail":94262728,"statfs":{"total":96636764160,"available":96525033472,"internally_reserved":0,"allocated":892928,"data_stored":539850,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":43282,"internal_metadata":110778094},"hb_peers":[0,1,4,5,7,9,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T11:28:22.257 INFO:tasks.ceph.ceph_manager.ceph:recovered! 2024-09-13T11:28:22.258 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr_debug_aggressive_pg_num_changes 2024-09-13T11:28:22.579 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-09-13T11:28:22.588 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-13T11:28:22.589 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-13T11:28:22.891 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-13T11:28:22.892 INFO:teuthology.orchestra.run.smithi001.stderr:dumped all 2024-09-13T11:28:22.908 INFO:teuthology.orchestra.run.smithi001.stdout:{"pg_ready":true,"pg_map":{"version":472,"stamp":"2024-09-13T11:28:22.210466+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":147,"num_write_kb":2674,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":115,"ondisk_log_size":115,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":10,"kb":1509949440,"kb_used":1964276,"kb_used_data":15156,"kb_used_omap":69,"kb_used_meta":1948794,"kb_avail":1507985164,"statfs":{"total":1546188226560,"available":1544176807936,"internally_reserved":0,"allocated":15519744,"data_stored":9922474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":71147,"internal_metadata":1995565589},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.001914"},"pg_stats":[{"pgid":"1.0","version":"26'112","reported_seq":725,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-13T11:28:20.440039+0000","last_change":"2024-09-13T11:27:51.016799+0000","last_active":"2024-09-13T11:28:20.440039+0000","last_peered":"2024-09-13T11:28:20.440039+0000","last_clean":"2024-09-13T11:28:20.440039+0000","last_became_active":"2024-09-13T11:22:12.129721+0000","last_became_peered":"2024-09-13T11:22:12.129721+0000","last_unstale":"2024-09-13T11:28:20.440039+0000","last_undegraded":"2024-09-13T11:28:20.440039+0000","last_fullsized":"2024-09-13T11:28:20.440039+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"26'112","last_scrub_stamp":"2024-09-13T11:27:51.016625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:11.098388+0000","last_clean_scrub_stamp":"2024-09-13T11:27:51.016625+0000","objects_scrubbed":2,"log_size":112,"log_dups_size":0,"ondisk_log_size":112,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:29:12.312000+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.3","version":"26'1","reported_seq":232,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-09-13T11:28:18.220535+0000","last_change":"2024-09-13T11:28:17.223911+0000","last_active":"2024-09-13T11:28:18.220535+0000","last_peered":"2024-09-13T11:28:18.220535+0000","last_clean":"2024-09-13T11:28:18.220535+0000","last_became_active":"2024-09-13T11:28:17.223376+0000","last_became_peered":"2024-09-13T11:28:17.223376+0000","last_unstale":"2024-09-13T11:28:18.220535+0000","last_undegraded":"2024-09-13T11:28:18.220535+0000","last_fullsized":"2024-09-13T11:28:18.220535+0000","mapping_epoch":296,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":297,"parent":"0.0","parent_split_bits":0,"last_scrub":"26'1","last_scrub_stamp":"2024-09-13T11:27:25.752752+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:27:25.752752+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:28:40.883396+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,15],"acting":[14,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":578,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-09-13T11:28:18.687300+0000","last_change":"2024-09-13T11:27:27.219367+0000","last_active":"2024-09-13T11:28:18.687300+0000","last_peered":"2024-09-13T11:28:18.687300+0000","last_clean":"2024-09-13T11:28:18.687300+0000","last_became_active":"2024-09-13T11:22:14.211435+0000","last_became_peered":"2024-09-13T11:22:14.211435+0000","last_unstale":"2024-09-13T11:28:18.687300+0000","last_undegraded":"2024-09-13T11:28:18.687300+0000","last_fullsized":"2024-09-13T11:28:18.687300+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:27:27.219203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:27:27.219203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:28:30.848078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010010360000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":230,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-09-13T11:28:18.220511+0000","last_change":"2024-09-13T11:27:26.709818+0000","last_active":"2024-09-13T11:28:18.220511+0000","last_peered":"2024-09-13T11:28:18.220511+0000","last_clean":"2024-09-13T11:28:18.220511+0000","last_became_active":"2024-09-13T11:26:05.046717+0000","last_became_peered":"2024-09-13T11:26:05.046717+0000","last_unstale":"2024-09-13T11:28:18.220511+0000","last_undegraded":"2024-09-13T11:28:18.220511+0000","last_fullsized":"2024-09-13T11:28:18.220511+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:27:26.709721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:27:26.709721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:28:41.582217+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00081225399999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,1],"acting":[14,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"2.2","version":"28'2","reported_seq":187,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-09-13T11:28:18.687303+0000","last_change":"2024-09-13T11:27:25.229244+0000","last_active":"2024-09-13T11:28:18.687303+0000","last_peered":"2024-09-13T11:28:18.687303+0000","last_clean":"2024-09-13T11:28:18.687303+0000","last_became_active":"2024-09-13T11:26:36.267081+0000","last_became_peered":"2024-09-13T11:26:36.267081+0000","last_unstale":"2024-09-13T11:28:18.687303+0000","last_undegraded":"2024-09-13T11:28:18.687303+0000","last_fullsized":"2024-09-13T11:28:18.687303+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2","last_scrub_stamp":"2024-09-13T11:27:25.229120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:27:25.229120+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:28:40.171329+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0025826730000000002,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9],"acting":[3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":570,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-13T11:28:19.235519+0000","last_change":"2024-09-13T11:27:26.521004+0000","last_active":"2024-09-13T11:28:19.235519+0000","last_peered":"2024-09-13T11:28:19.235519+0000","last_clean":"2024-09-13T11:28:19.235519+0000","last_became_active":"2024-09-13T11:22:47.719636+0000","last_became_peered":"2024-09-13T11:22:47.719636+0000","last_unstale":"2024-09-13T11:28:19.235519+0000","last_undegraded":"2024-09-13T11:28:19.235519+0000","last_fullsized":"2024-09-13T11:28:19.235519+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:27:26.520834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:27:26.520834+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:28:32.826137+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00087476000000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,9],"acting":[1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":585,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-13T11:28:19.232406+0000","last_change":"2024-09-13T11:26:58.241079+0000","last_active":"2024-09-13T11:28:19.232406+0000","last_peered":"2024-09-13T11:28:19.232406+0000","last_clean":"2024-09-13T11:28:19.232406+0000","last_became_active":"2024-09-13T11:22:14.581458+0000","last_became_peered":"2024-09-13T11:22:14.581458+0000","last_unstale":"2024-09-13T11:28:19.232406+0000","last_undegraded":"2024-09-13T11:28:19.232406+0000","last_fullsized":"2024-09-13T11:28:19.232406+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:26:58.240985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:26:58.240985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:28:23.342920+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.001026047,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5],"acting":[8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":591,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-13T11:28:19.786575+0000","last_change":"2024-09-13T11:27:58.082895+0000","last_active":"2024-09-13T11:28:19.786575+0000","last_peered":"2024-09-13T11:28:19.786575+0000","last_clean":"2024-09-13T11:28:19.786575+0000","last_became_active":"2024-09-13T11:22:14.211923+0000","last_became_peered":"2024-09-13T11:22:14.211923+0000","last_unstale":"2024-09-13T11:28:19.786575+0000","last_undegraded":"2024-09-13T11:28:19.786575+0000","last_fullsized":"2024-09-13T11:28:19.786575+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:27:58.082738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:27:58.082738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:29:11.583617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011689529999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0],"acting":[9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":422,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-13T11:28:20.170034+0000","last_change":"2024-09-13T11:28:13.159189+0000","last_active":"2024-09-13T11:28:20.170034+0000","last_peered":"2024-09-13T11:28:20.170034+0000","last_clean":"2024-09-13T11:28:20.170034+0000","last_became_active":"2024-09-13T11:28:13.158553+0000","last_became_peered":"2024-09-13T11:28:13.158553+0000","last_unstale":"2024-09-13T11:28:20.170034+0000","last_undegraded":"2024-09-13T11:28:20.170034+0000","last_fullsized":"2024-09-13T11:28:20.170034+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:27:27.561975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:27:27.561975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:28:56.783029+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,10],"acting":[15,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":11},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":112,"ondisk_log_size":112,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":13,"up_from":23,"seq":98784247887,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":100096,"kb_used_data":880,"kb_used_omap":1,"kb_used_meta":99198,"kb_avail":94271744,"statfs":{"total":96636764160,"available":96534265856,"internally_reserved":0,"allocated":901120,"data_stored":546978,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":101579205},"hb_peers":[0,1,2,4,6,8,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":9,"up_from":23,"seq":98784247886,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":94532,"kb_used_data":884,"kb_used_omap":1,"kb_used_meta":93630,"kb_avail":94277308,"statfs":{"total":96636764160,"available":96539963392,"internally_reserved":0,"allocated":905216,"data_stored":547960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":95877568},"hb_peers":[0,3,4,8,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":40,"seq":171798691909,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":141176,"kb_used_data":872,"kb_used_omap":4,"kb_used_meta":140283,"kb_avail":94230664,"statfs":{"total":96636764160,"available":96492199936,"internally_reserved":0,"allocated":892928,"data_stored":547553,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4433,"internal_metadata":143650479},"hb_peers":[0,2,4,7,8,9,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":21,"seq":90194313294,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":158396,"kb_used_data":876,"kb_used_omap":1,"kb_used_meta":157502,"kb_avail":94213444,"statfs":{"total":96636764160,"available":96474566656,"internally_reserved":0,"allocated":897024,"data_stored":540669,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":161282503},"hb_peers":[0,2,4,9,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":21,"seq":90194313297,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":166696,"kb_used_data":856,"kb_used_omap":1,"kb_used_meta":165822,"kb_avail":94205144,"statfs":{"total":96636764160,"available":96466067456,"internally_reserved":0,"allocated":876544,"data_stored":546133,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":169802183},"hb_peers":[0,2,4,6,7,8,9,11,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":22,"seq":94489280590,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":177028,"kb_used_data":1460,"kb_used_omap":1,"kb_used_meta":175550,"kb_avail":94194812,"statfs":{"total":96636764160,"available":96455487488,"internally_reserved":0,"allocated":1495040,"data_stored":1137784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":179763653},"hb_peers":[1,3,5,8,9,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":22,"seq":94489280590,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":94508,"kb_used_data":860,"kb_used_omap":1,"kb_used_meta":93630,"kb_avail":94277332,"statfs":{"total":96636764160,"available":96539987968,"internally_reserved":0,"allocated":880640,"data_stored":545856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":95877573},"hb_peers":[0,1,3,4,7,8,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":15,"up_from":22,"seq":94489280590,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":144504,"kb_used_data":872,"kb_used_omap":1,"kb_used_meta":143614,"kb_avail":94227336,"statfs":{"total":96636764160,"available":96488792064,"internally_reserved":0,"allocated":892928,"data_stored":545835,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":147061190},"hb_peers":[0,4,5,6,7,8,10,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":22,"seq":94489280591,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":86212,"kb_used_data":884,"kb_used_omap":1,"kb_used_meta":85310,"kb_avail":94285628,"statfs":{"total":96636764160,"available":96548483072,"internally_reserved":0,"allocated":905216,"data_stored":548291,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":87357893},"hb_peers":[3,5,6,7,9,10,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":22,"seq":94489280590,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72880,"kb_used_data":1440,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":94298960,"statfs":{"total":96636764160,"available":96562135040,"internally_reserved":0,"allocated":1474560,"data_stored":1136414,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":73136582},"hb_peers":[0,1,2,3,5,7,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":22,"seq":94489280591,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":162544,"kb_used_data":864,"kb_used_omap":1,"kb_used_meta":161662,"kb_avail":94209296,"statfs":{"total":96636764160,"available":96470319104,"internally_reserved":0,"allocated":884736,"data_stored":546969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":165542346},"hb_peers":[0,2,5,6,8,9,11,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":22,"seq":94489280590,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":100088,"kb_used_data":872,"kb_used_omap":1,"kb_used_meta":99198,"kb_avail":94271752,"statfs":{"total":96636764160,"available":96534274048,"internally_reserved":0,"allocated":892928,"data_stored":547084,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":101579205},"hb_peers":[1,2,5,7,9,10,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":261,"seq":1120986464267,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35540,"kb_used_data":900,"kb_used_omap":2,"kb_used_meta":34621,"kb_avail":94336300,"statfs":{"total":96636764160,"available":96600371200,"internally_reserved":0,"allocated":921600,"data_stored":549328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2702,"internal_metadata":35452274},"hb_peers":[0,1,3,7,9,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":11,"up_from":22,"seq":94489280591,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":154248,"kb_used_data":888,"kb_used_omap":1,"kb_used_meta":153342,"kb_avail":94217592,"statfs":{"total":96636764160,"available":96478814208,"internally_reserved":0,"allocated":909312,"data_stored":548374,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":157022660},"hb_peers":[2,4,7,8,9,10,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":12,"up_from":22,"seq":94489280590,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":166716,"kb_used_data":876,"kb_used_omap":1,"kb_used_meta":165822,"kb_avail":94205124,"statfs":{"total":96636764160,"available":96466046976,"internally_reserved":0,"allocated":897024,"data_stored":547396,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":169802181},"hb_peers":[1,2,3,5,6,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":14,"up_from":142,"seq":609885356075,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":109112,"kb_used_data":872,"kb_used_omap":42,"kb_used_meta":108181,"kb_avail":94262728,"statfs":{"total":96636764160,"available":96525033472,"internally_reserved":0,"allocated":892928,"data_stored":539850,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":43282,"internal_metadata":110778094},"hb_peers":[0,1,4,5,7,9,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T11:28:22.909 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-13T11:28:23.200 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-13T11:28:23.200 INFO:teuthology.orchestra.run.smithi001.stderr:dumped all 2024-09-13T11:28:23.217 INFO:teuthology.orchestra.run.smithi001.stdout:{"pg_ready":true,"pg_map":{"version":472,"stamp":"2024-09-13T11:28:22.210466+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":147,"num_write_kb":2674,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":115,"ondisk_log_size":115,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":10,"kb":1509949440,"kb_used":1964276,"kb_used_data":15156,"kb_used_omap":69,"kb_used_meta":1948794,"kb_avail":1507985164,"statfs":{"total":1546188226560,"available":1544176807936,"internally_reserved":0,"allocated":15519744,"data_stored":9922474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":71147,"internal_metadata":1995565589},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.001914"},"pg_stats":[{"pgid":"1.0","version":"26'112","reported_seq":725,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-13T11:28:20.440039+0000","last_change":"2024-09-13T11:27:51.016799+0000","last_active":"2024-09-13T11:28:20.440039+0000","last_peered":"2024-09-13T11:28:20.440039+0000","last_clean":"2024-09-13T11:28:20.440039+0000","last_became_active":"2024-09-13T11:22:12.129721+0000","last_became_peered":"2024-09-13T11:22:12.129721+0000","last_unstale":"2024-09-13T11:28:20.440039+0000","last_undegraded":"2024-09-13T11:28:20.440039+0000","last_fullsized":"2024-09-13T11:28:20.440039+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"26'112","last_scrub_stamp":"2024-09-13T11:27:51.016625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:11.098388+0000","last_clean_scrub_stamp":"2024-09-13T11:27:51.016625+0000","objects_scrubbed":2,"log_size":112,"log_dups_size":0,"ondisk_log_size":112,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:29:12.312000+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.3","version":"26'1","reported_seq":232,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-09-13T11:28:18.220535+0000","last_change":"2024-09-13T11:28:17.223911+0000","last_active":"2024-09-13T11:28:18.220535+0000","last_peered":"2024-09-13T11:28:18.220535+0000","last_clean":"2024-09-13T11:28:18.220535+0000","last_became_active":"2024-09-13T11:28:17.223376+0000","last_became_peered":"2024-09-13T11:28:17.223376+0000","last_unstale":"2024-09-13T11:28:18.220535+0000","last_undegraded":"2024-09-13T11:28:18.220535+0000","last_fullsized":"2024-09-13T11:28:18.220535+0000","mapping_epoch":296,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":297,"parent":"0.0","parent_split_bits":0,"last_scrub":"26'1","last_scrub_stamp":"2024-09-13T11:27:25.752752+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:27:25.752752+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:28:40.883396+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,15],"acting":[14,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":578,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-09-13T11:28:18.687300+0000","last_change":"2024-09-13T11:27:27.219367+0000","last_active":"2024-09-13T11:28:18.687300+0000","last_peered":"2024-09-13T11:28:18.687300+0000","last_clean":"2024-09-13T11:28:18.687300+0000","last_became_active":"2024-09-13T11:22:14.211435+0000","last_became_peered":"2024-09-13T11:22:14.211435+0000","last_unstale":"2024-09-13T11:28:18.687300+0000","last_undegraded":"2024-09-13T11:28:18.687300+0000","last_fullsized":"2024-09-13T11:28:18.687300+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:27:27.219203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:27:27.219203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:28:30.848078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010010360000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":230,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-09-13T11:28:18.220511+0000","last_change":"2024-09-13T11:27:26.709818+0000","last_active":"2024-09-13T11:28:18.220511+0000","last_peered":"2024-09-13T11:28:18.220511+0000","last_clean":"2024-09-13T11:28:18.220511+0000","last_became_active":"2024-09-13T11:26:05.046717+0000","last_became_peered":"2024-09-13T11:26:05.046717+0000","last_unstale":"2024-09-13T11:28:18.220511+0000","last_undegraded":"2024-09-13T11:28:18.220511+0000","last_fullsized":"2024-09-13T11:28:18.220511+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:27:26.709721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:27:26.709721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:28:41.582217+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00081225399999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,1],"acting":[14,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"2.2","version":"28'2","reported_seq":187,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-09-13T11:28:18.687303+0000","last_change":"2024-09-13T11:27:25.229244+0000","last_active":"2024-09-13T11:28:18.687303+0000","last_peered":"2024-09-13T11:28:18.687303+0000","last_clean":"2024-09-13T11:28:18.687303+0000","last_became_active":"2024-09-13T11:26:36.267081+0000","last_became_peered":"2024-09-13T11:26:36.267081+0000","last_unstale":"2024-09-13T11:28:18.687303+0000","last_undegraded":"2024-09-13T11:28:18.687303+0000","last_fullsized":"2024-09-13T11:28:18.687303+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2","last_scrub_stamp":"2024-09-13T11:27:25.229120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:27:25.229120+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:28:40.171329+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0025826730000000002,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9],"acting":[3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":570,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-13T11:28:19.235519+0000","last_change":"2024-09-13T11:27:26.521004+0000","last_active":"2024-09-13T11:28:19.235519+0000","last_peered":"2024-09-13T11:28:19.235519+0000","last_clean":"2024-09-13T11:28:19.235519+0000","last_became_active":"2024-09-13T11:22:47.719636+0000","last_became_peered":"2024-09-13T11:22:47.719636+0000","last_unstale":"2024-09-13T11:28:19.235519+0000","last_undegraded":"2024-09-13T11:28:19.235519+0000","last_fullsized":"2024-09-13T11:28:19.235519+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:27:26.520834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:27:26.520834+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:28:32.826137+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00087476000000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,9],"acting":[1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":585,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-13T11:28:19.232406+0000","last_change":"2024-09-13T11:26:58.241079+0000","last_active":"2024-09-13T11:28:19.232406+0000","last_peered":"2024-09-13T11:28:19.232406+0000","last_clean":"2024-09-13T11:28:19.232406+0000","last_became_active":"2024-09-13T11:22:14.581458+0000","last_became_peered":"2024-09-13T11:22:14.581458+0000","last_unstale":"2024-09-13T11:28:19.232406+0000","last_undegraded":"2024-09-13T11:28:19.232406+0000","last_fullsized":"2024-09-13T11:28:19.232406+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:26:58.240985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:26:58.240985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:28:23.342920+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.001026047,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5],"acting":[8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":591,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-13T11:28:19.786575+0000","last_change":"2024-09-13T11:27:58.082895+0000","last_active":"2024-09-13T11:28:19.786575+0000","last_peered":"2024-09-13T11:28:19.786575+0000","last_clean":"2024-09-13T11:28:19.786575+0000","last_became_active":"2024-09-13T11:22:14.211923+0000","last_became_peered":"2024-09-13T11:22:14.211923+0000","last_unstale":"2024-09-13T11:28:19.786575+0000","last_undegraded":"2024-09-13T11:28:19.786575+0000","last_fullsized":"2024-09-13T11:28:19.786575+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:27:58.082738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:27:58.082738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:29:11.583617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011689529999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0],"acting":[9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":422,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-13T11:28:20.170034+0000","last_change":"2024-09-13T11:28:13.159189+0000","last_active":"2024-09-13T11:28:20.170034+0000","last_peered":"2024-09-13T11:28:20.170034+0000","last_clean":"2024-09-13T11:28:20.170034+0000","last_became_active":"2024-09-13T11:28:13.158553+0000","last_became_peered":"2024-09-13T11:28:13.158553+0000","last_unstale":"2024-09-13T11:28:20.170034+0000","last_undegraded":"2024-09-13T11:28:20.170034+0000","last_fullsized":"2024-09-13T11:28:20.170034+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:27:27.561975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:27:27.561975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:28:56.783029+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,10],"acting":[15,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":11},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":112,"ondisk_log_size":112,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":13,"up_from":23,"seq":98784247887,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":100096,"kb_used_data":880,"kb_used_omap":1,"kb_used_meta":99198,"kb_avail":94271744,"statfs":{"total":96636764160,"available":96534265856,"internally_reserved":0,"allocated":901120,"data_stored":546978,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":101579205},"hb_peers":[0,1,2,4,6,8,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":9,"up_from":23,"seq":98784247886,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":94532,"kb_used_data":884,"kb_used_omap":1,"kb_used_meta":93630,"kb_avail":94277308,"statfs":{"total":96636764160,"available":96539963392,"internally_reserved":0,"allocated":905216,"data_stored":547960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":95877568},"hb_peers":[0,3,4,8,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":40,"seq":171798691909,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":141176,"kb_used_data":872,"kb_used_omap":4,"kb_used_meta":140283,"kb_avail":94230664,"statfs":{"total":96636764160,"available":96492199936,"internally_reserved":0,"allocated":892928,"data_stored":547553,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4433,"internal_metadata":143650479},"hb_peers":[0,2,4,7,8,9,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":21,"seq":90194313294,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":158396,"kb_used_data":876,"kb_used_omap":1,"kb_used_meta":157502,"kb_avail":94213444,"statfs":{"total":96636764160,"available":96474566656,"internally_reserved":0,"allocated":897024,"data_stored":540669,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":161282503},"hb_peers":[0,2,4,9,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":21,"seq":90194313297,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":166696,"kb_used_data":856,"kb_used_omap":1,"kb_used_meta":165822,"kb_avail":94205144,"statfs":{"total":96636764160,"available":96466067456,"internally_reserved":0,"allocated":876544,"data_stored":546133,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":169802183},"hb_peers":[0,2,4,6,7,8,9,11,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":22,"seq":94489280590,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":177028,"kb_used_data":1460,"kb_used_omap":1,"kb_used_meta":175550,"kb_avail":94194812,"statfs":{"total":96636764160,"available":96455487488,"internally_reserved":0,"allocated":1495040,"data_stored":1137784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":179763653},"hb_peers":[1,3,5,8,9,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":22,"seq":94489280590,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":94508,"kb_used_data":860,"kb_used_omap":1,"kb_used_meta":93630,"kb_avail":94277332,"statfs":{"total":96636764160,"available":96539987968,"internally_reserved":0,"allocated":880640,"data_stored":545856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":95877573},"hb_peers":[0,1,3,4,7,8,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":15,"up_from":22,"seq":94489280590,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":144504,"kb_used_data":872,"kb_used_omap":1,"kb_used_meta":143614,"kb_avail":94227336,"statfs":{"total":96636764160,"available":96488792064,"internally_reserved":0,"allocated":892928,"data_stored":545835,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":147061190},"hb_peers":[0,4,5,6,7,8,10,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":22,"seq":94489280591,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":86212,"kb_used_data":884,"kb_used_omap":1,"kb_used_meta":85310,"kb_avail":94285628,"statfs":{"total":96636764160,"available":96548483072,"internally_reserved":0,"allocated":905216,"data_stored":548291,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":87357893},"hb_peers":[3,5,6,7,9,10,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":22,"seq":94489280590,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72880,"kb_used_data":1440,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":94298960,"statfs":{"total":96636764160,"available":96562135040,"internally_reserved":0,"allocated":1474560,"data_stored":1136414,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":73136582},"hb_peers":[0,1,2,3,5,7,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":22,"seq":94489280591,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":162544,"kb_used_data":864,"kb_used_omap":1,"kb_used_meta":161662,"kb_avail":94209296,"statfs":{"total":96636764160,"available":96470319104,"internally_reserved":0,"allocated":884736,"data_stored":546969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":165542346},"hb_peers":[0,2,5,6,8,9,11,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":22,"seq":94489280590,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":100088,"kb_used_data":872,"kb_used_omap":1,"kb_used_meta":99198,"kb_avail":94271752,"statfs":{"total":96636764160,"available":96534274048,"internally_reserved":0,"allocated":892928,"data_stored":547084,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":101579205},"hb_peers":[1,2,5,7,9,10,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":261,"seq":1120986464267,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35540,"kb_used_data":900,"kb_used_omap":2,"kb_used_meta":34621,"kb_avail":94336300,"statfs":{"total":96636764160,"available":96600371200,"internally_reserved":0,"allocated":921600,"data_stored":549328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2702,"internal_metadata":35452274},"hb_peers":[0,1,3,7,9,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":11,"up_from":22,"seq":94489280591,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":154248,"kb_used_data":888,"kb_used_omap":1,"kb_used_meta":153342,"kb_avail":94217592,"statfs":{"total":96636764160,"available":96478814208,"internally_reserved":0,"allocated":909312,"data_stored":548374,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":157022660},"hb_peers":[2,4,7,8,9,10,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":12,"up_from":22,"seq":94489280590,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":166716,"kb_used_data":876,"kb_used_omap":1,"kb_used_meta":165822,"kb_avail":94205124,"statfs":{"total":96636764160,"available":96466046976,"internally_reserved":0,"allocated":897024,"data_stored":547396,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":169802181},"hb_peers":[1,2,3,5,6,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":14,"up_from":142,"seq":609885356075,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":109112,"kb_used_data":872,"kb_used_omap":42,"kb_used_meta":108181,"kb_avail":94262728,"statfs":{"total":96636764160,"available":96525033472,"internally_reserved":0,"allocated":892928,"data_stored":539850,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":43282,"internal_metadata":110778094},"hb_peers":[0,1,4,5,7,9,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T11:28:23.218 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-13T11:28:23.218 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-13T11:28:23.508 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-13T11:28:23.508 INFO:teuthology.orchestra.run.smithi001.stderr:dumped all 2024-09-13T11:28:23.525 INFO:teuthology.orchestra.run.smithi001.stdout:{"pg_ready":true,"pg_map":{"version":472,"stamp":"2024-09-13T11:28:22.210466+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":147,"num_write_kb":2674,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":115,"ondisk_log_size":115,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":10,"kb":1509949440,"kb_used":1964276,"kb_used_data":15156,"kb_used_omap":69,"kb_used_meta":1948794,"kb_avail":1507985164,"statfs":{"total":1546188226560,"available":1544176807936,"internally_reserved":0,"allocated":15519744,"data_stored":9922474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":71147,"internal_metadata":1995565589},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.001914"},"pg_stats":[{"pgid":"1.0","version":"26'112","reported_seq":725,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-13T11:28:20.440039+0000","last_change":"2024-09-13T11:27:51.016799+0000","last_active":"2024-09-13T11:28:20.440039+0000","last_peered":"2024-09-13T11:28:20.440039+0000","last_clean":"2024-09-13T11:28:20.440039+0000","last_became_active":"2024-09-13T11:22:12.129721+0000","last_became_peered":"2024-09-13T11:22:12.129721+0000","last_unstale":"2024-09-13T11:28:20.440039+0000","last_undegraded":"2024-09-13T11:28:20.440039+0000","last_fullsized":"2024-09-13T11:28:20.440039+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"26'112","last_scrub_stamp":"2024-09-13T11:27:51.016625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:11.098388+0000","last_clean_scrub_stamp":"2024-09-13T11:27:51.016625+0000","objects_scrubbed":2,"log_size":112,"log_dups_size":0,"ondisk_log_size":112,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:29:12.312000+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.3","version":"26'1","reported_seq":232,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-09-13T11:28:18.220535+0000","last_change":"2024-09-13T11:28:17.223911+0000","last_active":"2024-09-13T11:28:18.220535+0000","last_peered":"2024-09-13T11:28:18.220535+0000","last_clean":"2024-09-13T11:28:18.220535+0000","last_became_active":"2024-09-13T11:28:17.223376+0000","last_became_peered":"2024-09-13T11:28:17.223376+0000","last_unstale":"2024-09-13T11:28:18.220535+0000","last_undegraded":"2024-09-13T11:28:18.220535+0000","last_fullsized":"2024-09-13T11:28:18.220535+0000","mapping_epoch":296,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":297,"parent":"0.0","parent_split_bits":0,"last_scrub":"26'1","last_scrub_stamp":"2024-09-13T11:27:25.752752+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:27:25.752752+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:28:40.883396+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,15],"acting":[14,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":578,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-09-13T11:28:18.687300+0000","last_change":"2024-09-13T11:27:27.219367+0000","last_active":"2024-09-13T11:28:18.687300+0000","last_peered":"2024-09-13T11:28:18.687300+0000","last_clean":"2024-09-13T11:28:18.687300+0000","last_became_active":"2024-09-13T11:22:14.211435+0000","last_became_peered":"2024-09-13T11:22:14.211435+0000","last_unstale":"2024-09-13T11:28:18.687300+0000","last_undegraded":"2024-09-13T11:28:18.687300+0000","last_fullsized":"2024-09-13T11:28:18.687300+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:27:27.219203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:27:27.219203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:28:30.848078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010010360000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":230,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-09-13T11:28:18.220511+0000","last_change":"2024-09-13T11:27:26.709818+0000","last_active":"2024-09-13T11:28:18.220511+0000","last_peered":"2024-09-13T11:28:18.220511+0000","last_clean":"2024-09-13T11:28:18.220511+0000","last_became_active":"2024-09-13T11:26:05.046717+0000","last_became_peered":"2024-09-13T11:26:05.046717+0000","last_unstale":"2024-09-13T11:28:18.220511+0000","last_undegraded":"2024-09-13T11:28:18.220511+0000","last_fullsized":"2024-09-13T11:28:18.220511+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:27:26.709721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:27:26.709721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:28:41.582217+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00081225399999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,1],"acting":[14,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"2.2","version":"28'2","reported_seq":187,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-09-13T11:28:18.687303+0000","last_change":"2024-09-13T11:27:25.229244+0000","last_active":"2024-09-13T11:28:18.687303+0000","last_peered":"2024-09-13T11:28:18.687303+0000","last_clean":"2024-09-13T11:28:18.687303+0000","last_became_active":"2024-09-13T11:26:36.267081+0000","last_became_peered":"2024-09-13T11:26:36.267081+0000","last_unstale":"2024-09-13T11:28:18.687303+0000","last_undegraded":"2024-09-13T11:28:18.687303+0000","last_fullsized":"2024-09-13T11:28:18.687303+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2","last_scrub_stamp":"2024-09-13T11:27:25.229120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:27:25.229120+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:28:40.171329+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0025826730000000002,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9],"acting":[3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":570,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-13T11:28:19.235519+0000","last_change":"2024-09-13T11:27:26.521004+0000","last_active":"2024-09-13T11:28:19.235519+0000","last_peered":"2024-09-13T11:28:19.235519+0000","last_clean":"2024-09-13T11:28:19.235519+0000","last_became_active":"2024-09-13T11:22:47.719636+0000","last_became_peered":"2024-09-13T11:22:47.719636+0000","last_unstale":"2024-09-13T11:28:19.235519+0000","last_undegraded":"2024-09-13T11:28:19.235519+0000","last_fullsized":"2024-09-13T11:28:19.235519+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:27:26.520834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:27:26.520834+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:28:32.826137+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00087476000000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,9],"acting":[1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":585,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-13T11:28:19.232406+0000","last_change":"2024-09-13T11:26:58.241079+0000","last_active":"2024-09-13T11:28:19.232406+0000","last_peered":"2024-09-13T11:28:19.232406+0000","last_clean":"2024-09-13T11:28:19.232406+0000","last_became_active":"2024-09-13T11:22:14.581458+0000","last_became_peered":"2024-09-13T11:22:14.581458+0000","last_unstale":"2024-09-13T11:28:19.232406+0000","last_undegraded":"2024-09-13T11:28:19.232406+0000","last_fullsized":"2024-09-13T11:28:19.232406+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:26:58.240985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:26:58.240985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:28:23.342920+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.001026047,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5],"acting":[8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":591,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-13T11:28:19.786575+0000","last_change":"2024-09-13T11:27:58.082895+0000","last_active":"2024-09-13T11:28:19.786575+0000","last_peered":"2024-09-13T11:28:19.786575+0000","last_clean":"2024-09-13T11:28:19.786575+0000","last_became_active":"2024-09-13T11:22:14.211923+0000","last_became_peered":"2024-09-13T11:22:14.211923+0000","last_unstale":"2024-09-13T11:28:19.786575+0000","last_undegraded":"2024-09-13T11:28:19.786575+0000","last_fullsized":"2024-09-13T11:28:19.786575+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:27:58.082738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:27:58.082738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:29:11.583617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011689529999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0],"acting":[9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":422,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-13T11:28:20.170034+0000","last_change":"2024-09-13T11:28:13.159189+0000","last_active":"2024-09-13T11:28:20.170034+0000","last_peered":"2024-09-13T11:28:20.170034+0000","last_clean":"2024-09-13T11:28:20.170034+0000","last_became_active":"2024-09-13T11:28:13.158553+0000","last_became_peered":"2024-09-13T11:28:13.158553+0000","last_unstale":"2024-09-13T11:28:20.170034+0000","last_undegraded":"2024-09-13T11:28:20.170034+0000","last_fullsized":"2024-09-13T11:28:20.170034+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:27:27.561975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:27:27.561975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:28:56.783029+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,10],"acting":[15,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":11},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":112,"ondisk_log_size":112,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":13,"up_from":23,"seq":98784247887,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":100096,"kb_used_data":880,"kb_used_omap":1,"kb_used_meta":99198,"kb_avail":94271744,"statfs":{"total":96636764160,"available":96534265856,"internally_reserved":0,"allocated":901120,"data_stored":546978,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":101579205},"hb_peers":[0,1,2,4,6,8,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":9,"up_from":23,"seq":98784247886,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":94532,"kb_used_data":884,"kb_used_omap":1,"kb_used_meta":93630,"kb_avail":94277308,"statfs":{"total":96636764160,"available":96539963392,"internally_reserved":0,"allocated":905216,"data_stored":547960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":95877568},"hb_peers":[0,3,4,8,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":40,"seq":171798691909,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":141176,"kb_used_data":872,"kb_used_omap":4,"kb_used_meta":140283,"kb_avail":94230664,"statfs":{"total":96636764160,"available":96492199936,"internally_reserved":0,"allocated":892928,"data_stored":547553,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4433,"internal_metadata":143650479},"hb_peers":[0,2,4,7,8,9,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":21,"seq":90194313294,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":158396,"kb_used_data":876,"kb_used_omap":1,"kb_used_meta":157502,"kb_avail":94213444,"statfs":{"total":96636764160,"available":96474566656,"internally_reserved":0,"allocated":897024,"data_stored":540669,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":161282503},"hb_peers":[0,2,4,9,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":21,"seq":90194313297,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":166696,"kb_used_data":856,"kb_used_omap":1,"kb_used_meta":165822,"kb_avail":94205144,"statfs":{"total":96636764160,"available":96466067456,"internally_reserved":0,"allocated":876544,"data_stored":546133,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":169802183},"hb_peers":[0,2,4,6,7,8,9,11,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":22,"seq":94489280590,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":177028,"kb_used_data":1460,"kb_used_omap":1,"kb_used_meta":175550,"kb_avail":94194812,"statfs":{"total":96636764160,"available":96455487488,"internally_reserved":0,"allocated":1495040,"data_stored":1137784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":179763653},"hb_peers":[1,3,5,8,9,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":22,"seq":94489280590,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":94508,"kb_used_data":860,"kb_used_omap":1,"kb_used_meta":93630,"kb_avail":94277332,"statfs":{"total":96636764160,"available":96539987968,"internally_reserved":0,"allocated":880640,"data_stored":545856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":95877573},"hb_peers":[0,1,3,4,7,8,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":15,"up_from":22,"seq":94489280590,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":144504,"kb_used_data":872,"kb_used_omap":1,"kb_used_meta":143614,"kb_avail":94227336,"statfs":{"total":96636764160,"available":96488792064,"internally_reserved":0,"allocated":892928,"data_stored":545835,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":147061190},"hb_peers":[0,4,5,6,7,8,10,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":22,"seq":94489280591,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":86212,"kb_used_data":884,"kb_used_omap":1,"kb_used_meta":85310,"kb_avail":94285628,"statfs":{"total":96636764160,"available":96548483072,"internally_reserved":0,"allocated":905216,"data_stored":548291,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":87357893},"hb_peers":[3,5,6,7,9,10,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":22,"seq":94489280590,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72880,"kb_used_data":1440,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":94298960,"statfs":{"total":96636764160,"available":96562135040,"internally_reserved":0,"allocated":1474560,"data_stored":1136414,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":73136582},"hb_peers":[0,1,2,3,5,7,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":22,"seq":94489280591,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":162544,"kb_used_data":864,"kb_used_omap":1,"kb_used_meta":161662,"kb_avail":94209296,"statfs":{"total":96636764160,"available":96470319104,"internally_reserved":0,"allocated":884736,"data_stored":546969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":165542346},"hb_peers":[0,2,5,6,8,9,11,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":22,"seq":94489280590,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":100088,"kb_used_data":872,"kb_used_omap":1,"kb_used_meta":99198,"kb_avail":94271752,"statfs":{"total":96636764160,"available":96534274048,"internally_reserved":0,"allocated":892928,"data_stored":547084,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":101579205},"hb_peers":[1,2,5,7,9,10,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":261,"seq":1120986464267,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35540,"kb_used_data":900,"kb_used_omap":2,"kb_used_meta":34621,"kb_avail":94336300,"statfs":{"total":96636764160,"available":96600371200,"internally_reserved":0,"allocated":921600,"data_stored":549328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2702,"internal_metadata":35452274},"hb_peers":[0,1,3,7,9,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":11,"up_from":22,"seq":94489280591,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":154248,"kb_used_data":888,"kb_used_omap":1,"kb_used_meta":153342,"kb_avail":94217592,"statfs":{"total":96636764160,"available":96478814208,"internally_reserved":0,"allocated":909312,"data_stored":548374,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":157022660},"hb_peers":[2,4,7,8,9,10,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":12,"up_from":22,"seq":94489280590,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":166716,"kb_used_data":876,"kb_used_omap":1,"kb_used_meta":165822,"kb_avail":94205124,"statfs":{"total":96636764160,"available":96466046976,"internally_reserved":0,"allocated":897024,"data_stored":547396,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":169802181},"hb_peers":[1,2,3,5,6,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":14,"up_from":142,"seq":609885356075,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":109112,"kb_used_data":872,"kb_used_omap":42,"kb_used_meta":108181,"kb_avail":94262728,"statfs":{"total":96636764160,"available":96525033472,"internally_reserved":0,"allocated":892928,"data_stored":539850,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":43282,"internal_metadata":110778094},"hb_peers":[0,1,4,5,7,9,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T11:28:23.526 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T11:28:23.821 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-13T11:28:23.821 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":300,"fsid":"2fd68ac7-5581-4d90-80a4-30bcf5252a7a","created":"2024-09-13T11:21:57.223153+0000","modified":"2024-09-13T11:28:21.989952+0000","last_up_change":"2024-09-13T11:27:38.782633+0000","last_in_change":"2024-09-13T11:27:46.310884+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":8,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T11:22:10.162266+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":15.380000114440918,"score_stable":15.380000114440918,"optimal_score":0.12999999523162842,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-13T11:22:12.235549+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"28","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":28,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4.0199999809265137,"score_stable":4.0199999809265137,"optimal_score":0.56000000238418579,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"0fc12b3c-e0de-4546-b935-db77efe7c567","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":250,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6800","nonce":1544776477}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6801","nonce":1544776477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6803","nonce":1544776477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6802","nonce":1544776477}]},"public_addr":"172.21.15.1:6800/1544776477","cluster_addr":"172.21.15.1:6801/1544776477","heartbeat_back_addr":"172.21.15.1:6803/1544776477","heartbeat_front_addr":"172.21.15.1:6802/1544776477","state":["exists","up"]},{"osd":1,"uuid":"b77fe331-cc92-4adb-b146-af5c380f6d4f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":23,"last_clean_end":29,"up_from":40,"up_thru":243,"down_at":30,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.92:6800","nonce":3591430847}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.92:6801","nonce":3591430847}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.92:6803","nonce":3591430847}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.92:6802","nonce":3591430847}]},"public_addr":"172.21.15.92:6800/3591430847","cluster_addr":"172.21.15.92:6801/3591430847","heartbeat_back_addr":"172.21.15.92:6803/3591430847","heartbeat_front_addr":"172.21.15.92:6802/3591430847","state":["exists","up"]},{"osd":2,"uuid":"c9d031c8-e885-4229-affa-08e1729d00e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":251,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.162:6812","nonce":2712148101}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.162:6813","nonce":2712148101}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.162:6815","nonce":2712148101}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.162:6814","nonce":2712148101}]},"public_addr":"172.21.15.162:6812/2712148101","cluster_addr":"172.21.15.162:6813/2712148101","heartbeat_back_addr":"172.21.15.162:6815/2712148101","heartbeat_front_addr":"172.21.15.162:6814/2712148101","state":["exists","up"]},{"osd":3,"uuid":"bf50d71a-6c3a-4fbc-a407-cb81bad14a7b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":250,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.177:6801","nonce":3367751690}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.177:6802","nonce":3367751690}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.177:6804","nonce":3367751690}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.177:6803","nonce":3367751690}]},"public_addr":"172.21.15.177:6801/3367751690","cluster_addr":"172.21.15.177:6802/3367751690","heartbeat_back_addr":"172.21.15.177:6804/3367751690","heartbeat_front_addr":"172.21.15.177:6803/3367751690","state":["exists","up"]},{"osd":4,"uuid":"d1fe3f63-df3f-4a73-a1e2-8d7c8a3f1fa0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":239,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6808","nonce":3648540926}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6809","nonce":3648540926}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6811","nonce":3648540926}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6810","nonce":3648540926}]},"public_addr":"172.21.15.1:6808/3648540926","cluster_addr":"172.21.15.1:6809/3648540926","heartbeat_back_addr":"172.21.15.1:6811/3648540926","heartbeat_front_addr":"172.21.15.1:6810/3648540926","state":["exists","up"]},{"osd":5,"uuid":"e2ead447-2f2c-40c9-82e7-53b8dc41e2af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":249,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.92:6804","nonce":3399529091}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.92:6805","nonce":3399529091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.92:6807","nonce":3399529091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.92:6806","nonce":3399529091}]},"public_addr":"172.21.15.92:6804/3399529091","cluster_addr":"172.21.15.92:6805/3399529091","heartbeat_back_addr":"172.21.15.92:6807/3399529091","heartbeat_front_addr":"172.21.15.92:6806/3399529091","state":["exists","up"]},{"osd":6,"uuid":"dbc20439-144e-4ec7-89de-674eb783f263","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":250,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.162:6808","nonce":2287343806}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.162:6809","nonce":2287343806}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.162:6811","nonce":2287343806}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.162:6810","nonce":2287343806}]},"public_addr":"172.21.15.162:6808/2287343806","cluster_addr":"172.21.15.162:6809/2287343806","heartbeat_back_addr":"172.21.15.162:6811/2287343806","heartbeat_front_addr":"172.21.15.162:6810/2287343806","state":["exists","up"]},{"osd":7,"uuid":"60aa5206-cace-40f8-8d1a-10ec28d97096","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":251,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.177:6813","nonce":2110809906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.177:6814","nonce":2110809906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.177:6816","nonce":2110809906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.177:6815","nonce":2110809906}]},"public_addr":"172.21.15.177:6813/2110809906","cluster_addr":"172.21.15.177:6814/2110809906","heartbeat_back_addr":"172.21.15.177:6816/2110809906","heartbeat_front_addr":"172.21.15.177:6815/2110809906","state":["exists","up"]},{"osd":8,"uuid":"f3d99325-48ff-403c-8662-1d7745f7c97d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6804","nonce":2766567910}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6805","nonce":2766567910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6807","nonce":2766567910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6806","nonce":2766567910}]},"public_addr":"172.21.15.1:6804/2766567910","cluster_addr":"172.21.15.1:6805/2766567910","heartbeat_back_addr":"172.21.15.1:6807/2766567910","heartbeat_front_addr":"172.21.15.1:6806/2766567910","state":["exists","up"]},{"osd":9,"uuid":"88511e24-3675-4551-968c-1a52e7e28f96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":244,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.92:6812","nonce":2488187511}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.92:6813","nonce":2488187511}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.92:6815","nonce":2488187511}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.92:6814","nonce":2488187511}]},"public_addr":"172.21.15.92:6812/2488187511","cluster_addr":"172.21.15.92:6813/2488187511","heartbeat_back_addr":"172.21.15.92:6815/2488187511","heartbeat_front_addr":"172.21.15.92:6814/2488187511","state":["exists","up"]},{"osd":10,"uuid":"acf4c64d-99af-42be-8406-413c86f54670","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":22,"last_clean_end":238,"up_from":261,"up_thru":266,"down_at":239,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.162:6804","nonce":4258911154}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.162:6805","nonce":4258911154}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.162:6807","nonce":4258911154}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.162:6806","nonce":4258911154}]},"public_addr":"172.21.15.162:6804/4258911154","cluster_addr":"172.21.15.162:6805/4258911154","heartbeat_back_addr":"172.21.15.162:6807/4258911154","heartbeat_front_addr":"172.21.15.162:6806/4258911154","state":["exists","up"]},{"osd":11,"uuid":"a4bfdfbe-b586-4080-8e09-6f26428e59b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":247,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.177:6809","nonce":3239521954}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.177:6810","nonce":3239521954}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.177:6812","nonce":3239521954}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.177:6811","nonce":3239521954}]},"public_addr":"172.21.15.177:6809/3239521954","cluster_addr":"172.21.15.177:6810/3239521954","heartbeat_back_addr":"172.21.15.177:6812/3239521954","heartbeat_front_addr":"172.21.15.177:6811/3239521954","state":["exists","up"]},{"osd":12,"uuid":"dc571fb6-d8b5-465b-bb25-afa7d0c47b5a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":249,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6812","nonce":4267062742}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6813","nonce":4267062742}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6815","nonce":4267062742}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.1:6814","nonce":4267062742}]},"public_addr":"172.21.15.1:6812/4267062742","cluster_addr":"172.21.15.1:6813/4267062742","heartbeat_back_addr":"172.21.15.1:6815/4267062742","heartbeat_front_addr":"172.21.15.1:6814/4267062742","state":["exists","up"]},{"osd":13,"uuid":"5d321d4b-1b26-47fe-bf84-25751b8cde7d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":235,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.92:6808","nonce":1759585046}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.92:6809","nonce":1759585046}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.92:6811","nonce":1759585046}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.92:6810","nonce":1759585046}]},"public_addr":"172.21.15.92:6808/1759585046","cluster_addr":"172.21.15.92:6809/1759585046","heartbeat_back_addr":"172.21.15.92:6811/1759585046","heartbeat_front_addr":"172.21.15.92:6810/1759585046","state":["exists","up"]},{"osd":14,"uuid":"fc3e2499-0c2a-414d-a25e-2628876ae630","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":22,"last_clean_end":120,"up_from":142,"up_thru":296,"down_at":121,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.162:6800","nonce":3999392664}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.162:6801","nonce":3999392664}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.162:6803","nonce":3999392664}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.162:6802","nonce":3999392664}]},"public_addr":"172.21.15.162:6800/3999392664","cluster_addr":"172.21.15.162:6801/3999392664","heartbeat_back_addr":"172.21.15.162:6803/3999392664","heartbeat_front_addr":"172.21.15.162:6802/3999392664","state":["exists","up"]},{"osd":15,"uuid":"d2f69775-edfe-4527-91ec-f7c7e558bec8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":292,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.177:6805","nonce":1294337172}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.177:6806","nonce":1294337172}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.177:6808","nonce":1294337172}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.177:6807","nonce":1294337172}]},"public_addr":"172.21.15.177:6805/1294337172","cluster_addr":"172.21.15.177:6806/1294337172","heartbeat_back_addr":"172.21.15.177:6808/1294337172","heartbeat_front_addr":"172.21.15.177:6807/1294337172","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T11:27:40.342776+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T11:27:07.613933+0000","dead_epoch":30},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T11:27:22.828451+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T11:27:21.614437+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T11:27:25.568873+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T11:27:28.185934+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T11:27:46.757486+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T11:27:21.994608+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T11:27:44.589810+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T11:27:20.603346+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T11:27:37.802899+0000","dead_epoch":239},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T11:27:44.791122+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T11:28:19.094640+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T11:28:15.562220+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T11:27:06.849942+0000","dead_epoch":121},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T11:27:15.020976+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T11:28:24.249 INFO:tasks.ceph.osd.14.smithi162.stderr:2024-09-13T11:28:24.246+0000 7f3dec334640 -1 osd.14 300 handle_osd_map injecting map crc failure 2024-09-13T11:28:24.612 INFO:tasks.ceph.osd.13.smithi092.stderr:2024-09-13T11:28:24.610+0000 7f3e4ba0f640 -1 osd.13 300 handle_osd_map injecting map crc failure 2024-09-13T11:28:24.842 INFO:tasks.ceph:Scrubbing osd.0 2024-09-13T11:28:24.842 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-09-13T11:28:25.013 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-13T11:28:25.013 INFO:teuthology.orchestra.run.smithi001.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-13T11:28:25.013 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-13T11:28:25.029 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2024-09-13T11:28:25.338 INFO:teuthology.orchestra.run.smithi001.stderr:instructed osd(s) 0 to deep-scrub 2024-09-13T11:28:25.359 INFO:tasks.ceph:Scrubbing osd.4 2024-09-13T11:28:25.359 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 config set osd_debug_deep_scrub_sleep 0 2024-09-13T11:28:25.513 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-13T11:28:25.513 INFO:teuthology.orchestra.run.smithi001.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-13T11:28:25.514 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-13T11:28:25.530 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 4 2024-09-13T11:28:25.836 INFO:teuthology.orchestra.run.smithi001.stderr:instructed osd(s) 4 to deep-scrub 2024-09-13T11:28:25.851 INFO:tasks.ceph:Scrubbing osd.8 2024-09-13T11:28:25.852 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.8 config set osd_debug_deep_scrub_sleep 0 2024-09-13T11:28:26.013 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-13T11:28:26.013 INFO:teuthology.orchestra.run.smithi001.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-13T11:28:26.013 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-13T11:28:26.030 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 8 2024-09-13T11:28:26.336 INFO:teuthology.orchestra.run.smithi001.stderr:instructed osd(s) 8 to deep-scrub 2024-09-13T11:28:26.354 INFO:tasks.ceph:Scrubbing osd.12 2024-09-13T11:28:26.355 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.12 config set osd_debug_deep_scrub_sleep 0 2024-09-13T11:28:26.512 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-13T11:28:26.512 INFO:teuthology.orchestra.run.smithi001.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-13T11:28:26.512 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-13T11:28:26.527 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 12 2024-09-13T11:28:26.816 INFO:teuthology.orchestra.run.smithi001.stderr:instructed osd(s) 12 to deep-scrub 2024-09-13T11:28:26.834 INFO:tasks.ceph:Scrubbing osd.1 2024-09-13T11:28:26.834 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-09-13T11:28:26.985 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-13T11:28:26.985 INFO:teuthology.orchestra.run.smithi001.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-13T11:28:26.986 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-13T11:28:27.001 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2024-09-13T11:28:27.306 INFO:teuthology.orchestra.run.smithi001.stderr:instructed osd(s) 1 to deep-scrub 2024-09-13T11:28:27.325 INFO:tasks.ceph:Scrubbing osd.5 2024-09-13T11:28:27.326 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 config set osd_debug_deep_scrub_sleep 0 2024-09-13T11:28:27.491 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-13T11:28:27.491 INFO:teuthology.orchestra.run.smithi001.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-13T11:28:27.491 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-13T11:28:27.508 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 5 2024-09-13T11:28:27.798 INFO:teuthology.orchestra.run.smithi001.stderr:instructed osd(s) 5 to deep-scrub 2024-09-13T11:28:27.815 INFO:tasks.ceph:Scrubbing osd.9 2024-09-13T11:28:27.816 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.9 config set osd_debug_deep_scrub_sleep 0 2024-09-13T11:28:27.976 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-13T11:28:27.976 INFO:teuthology.orchestra.run.smithi001.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-13T11:28:27.976 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-13T11:28:27.990 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 9 2024-09-13T11:28:28.293 INFO:teuthology.orchestra.run.smithi001.stderr:instructed osd(s) 9 to deep-scrub 2024-09-13T11:28:28.311 INFO:tasks.ceph:Scrubbing osd.13 2024-09-13T11:28:28.311 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.13 config set osd_debug_deep_scrub_sleep 0 2024-09-13T11:28:28.475 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-13T11:28:28.476 INFO:teuthology.orchestra.run.smithi001.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-13T11:28:28.476 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-13T11:28:28.492 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 13 2024-09-13T11:28:28.798 INFO:teuthology.orchestra.run.smithi001.stderr:instructed osd(s) 13 to deep-scrub 2024-09-13T11:28:28.817 INFO:tasks.ceph:Scrubbing osd.2 2024-09-13T11:28:28.817 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-09-13T11:28:28.972 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-13T11:28:28.972 INFO:teuthology.orchestra.run.smithi001.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-13T11:28:28.972 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-13T11:28:28.986 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2024-09-13T11:28:29.285 INFO:teuthology.orchestra.run.smithi001.stderr:instructed osd(s) 2 to deep-scrub 2024-09-13T11:28:29.305 INFO:tasks.ceph:Scrubbing osd.6 2024-09-13T11:28:29.305 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 config set osd_debug_deep_scrub_sleep 0 2024-09-13T11:28:29.467 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-13T11:28:29.467 INFO:teuthology.orchestra.run.smithi001.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-13T11:28:29.467 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-13T11:28:29.483 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 6 2024-09-13T11:28:29.793 INFO:teuthology.orchestra.run.smithi001.stderr:instructed osd(s) 6 to deep-scrub 2024-09-13T11:28:29.812 INFO:tasks.ceph:Scrubbing osd.10 2024-09-13T11:28:29.813 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.10 config set osd_debug_deep_scrub_sleep 0 2024-09-13T11:28:29.982 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-13T11:28:29.983 INFO:teuthology.orchestra.run.smithi001.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-13T11:28:29.983 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-13T11:28:30.001 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 10 2024-09-13T11:28:30.307 INFO:teuthology.orchestra.run.smithi001.stderr:instructed osd(s) 10 to deep-scrub 2024-09-13T11:28:30.326 INFO:tasks.ceph:Scrubbing osd.14 2024-09-13T11:28:30.326 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.14 config set osd_debug_deep_scrub_sleep 0 2024-09-13T11:28:30.491 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-13T11:28:30.491 INFO:teuthology.orchestra.run.smithi001.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-13T11:28:30.491 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-13T11:28:30.507 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 14 2024-09-13T11:28:30.740 INFO:tasks.ceph.osd.2.smithi162.stderr:2024-09-13T11:28:30.738+0000 7fe91fd87640 -1 osd.2 302 handle_osd_map injecting map crc failure 2024-09-13T11:28:30.741 INFO:tasks.ceph.osd.7.smithi177.stderr:2024-09-13T11:28:30.739+0000 7f896ff4a640 -1 osd.7 302 handle_osd_map injecting map crc failure 2024-09-13T11:28:30.820 INFO:teuthology.orchestra.run.smithi001.stderr:instructed osd(s) 14 to deep-scrub 2024-09-13T11:28:30.841 INFO:tasks.ceph:Scrubbing osd.3 2024-09-13T11:28:30.841 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2024-09-13T11:28:31.009 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-13T11:28:31.010 INFO:teuthology.orchestra.run.smithi001.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-13T11:28:31.010 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-13T11:28:31.027 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2024-09-13T11:28:31.343 INFO:teuthology.orchestra.run.smithi001.stderr:instructed osd(s) 3 to deep-scrub 2024-09-13T11:28:31.363 INFO:tasks.ceph:Scrubbing osd.7 2024-09-13T11:28:31.363 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 config set osd_debug_deep_scrub_sleep 0 2024-09-13T11:28:31.531 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-13T11:28:31.531 INFO:teuthology.orchestra.run.smithi001.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-13T11:28:31.531 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-13T11:28:31.548 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 7 2024-09-13T11:28:31.850 INFO:teuthology.orchestra.run.smithi001.stderr:instructed osd(s) 7 to deep-scrub 2024-09-13T11:28:31.868 INFO:tasks.ceph:Scrubbing osd.11 2024-09-13T11:28:31.869 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.11 config set osd_debug_deep_scrub_sleep 0 2024-09-13T11:28:31.996 INFO:tasks.ceph.osd.9.smithi092.stderr:2024-09-13T11:28:31.994+0000 7ff59ddf9640 -1 osd.9 303 handle_osd_map injecting map crc failure 2024-09-13T11:28:32.023 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-13T11:28:32.023 INFO:teuthology.orchestra.run.smithi001.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-13T11:28:32.023 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-13T11:28:32.039 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 11 2024-09-13T11:28:32.056 INFO:tasks.ceph.osd.10.smithi162.stderr:2024-09-13T11:28:32.054+0000 7ff886919640 -1 osd.10 303 handle_osd_map injecting map crc failure 2024-09-13T11:28:32.186 INFO:tasks.ceph.osd.14.smithi162.stderr:2024-09-13T11:28:32.182+0000 7f3dea330640 -1 osd.14 303 handle_osd_map injecting map crc failure 2024-09-13T11:28:32.347 INFO:teuthology.orchestra.run.smithi001.stderr:instructed osd(s) 11 to deep-scrub 2024-09-13T11:28:32.366 INFO:tasks.ceph:Scrubbing osd.15 2024-09-13T11:28:32.367 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.15 config set osd_debug_deep_scrub_sleep 0 2024-09-13T11:28:32.525 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-13T11:28:32.526 INFO:teuthology.orchestra.run.smithi001.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-13T11:28:32.526 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-13T11:28:32.541 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 15 2024-09-13T11:28:32.832 INFO:teuthology.orchestra.run.smithi001.stderr:instructed osd(s) 15 to deep-scrub 2024-09-13T11:28:32.851 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-13T11:28:33.142 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-13T11:28:33.142 INFO:teuthology.orchestra.run.smithi001.stderr:dumped all 2024-09-13T11:28:33.159 INFO:teuthology.orchestra.run.smithi001.stdout:{"pg_ready":true,"pg_map":{"version":481,"stamp":"2024-09-13T11:28:32.213401+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":147,"num_write_kb":2674,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":10,"kb":1509949440,"kb_used":1968220,"kb_used_data":14940,"kb_used_omap":69,"kb_used_meta":1952954,"kb_avail":1507981220,"statfs":{"total":1546188226560,"available":1544172769280,"internally_reserved":0,"allocated":15298560,"data_stored":9756064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":71147,"internal_metadata":1999825429},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.945896"},"pg_stats":[{"pgid":"1.0","version":"302'114","reported_seq":741,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-13T11:28:29.934970+0000","last_change":"2024-09-13T11:28:29.934970+0000","last_active":"2024-09-13T11:28:29.934970+0000","last_peered":"2024-09-13T11:28:29.934970+0000","last_clean":"2024-09-13T11:28:29.934970+0000","last_became_active":"2024-09-13T11:22:12.129721+0000","last_became_peered":"2024-09-13T11:22:12.129721+0000","last_unstale":"2024-09-13T11:28:29.934970+0000","last_undegraded":"2024-09-13T11:28:29.934970+0000","last_fullsized":"2024-09-13T11:28:29.934970+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"302'114","last_scrub_stamp":"2024-09-13T11:28:29.934796+0000","last_deep_scrub":"302'114","last_deep_scrub_stamp":"2024-09-13T11:28:29.934796+0000","last_clean_scrub_stamp":"2024-09-13T11:28:29.934796+0000","objects_scrubbed":2,"log_size":114,"log_dups_size":0,"ondisk_log_size":114,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:29:56.601457+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.3","version":"26'1","reported_seq":240,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-13T11:28:25.268407+0000","last_change":"2024-09-13T11:28:17.223911+0000","last_active":"2024-09-13T11:28:25.268407+0000","last_peered":"2024-09-13T11:28:25.268407+0000","last_clean":"2024-09-13T11:28:25.268407+0000","last_became_active":"2024-09-13T11:28:17.223376+0000","last_became_peered":"2024-09-13T11:28:17.223376+0000","last_unstale":"2024-09-13T11:28:25.268407+0000","last_undegraded":"2024-09-13T11:28:25.268407+0000","last_fullsized":"2024-09-13T11:28:25.268407+0000","mapping_epoch":296,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":297,"parent":"0.0","parent_split_bits":0,"last_scrub":"26'1","last_scrub_stamp":"2024-09-13T11:27:25.752752+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:27:25.752752+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:28:40.883396+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,15],"acting":[14,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":586,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-13T11:28:25.996704+0000","last_change":"2024-09-13T11:27:27.219367+0000","last_active":"2024-09-13T11:28:25.996704+0000","last_peered":"2024-09-13T11:28:25.996704+0000","last_clean":"2024-09-13T11:28:25.996704+0000","last_became_active":"2024-09-13T11:22:14.211435+0000","last_became_peered":"2024-09-13T11:22:14.211435+0000","last_unstale":"2024-09-13T11:28:25.996704+0000","last_undegraded":"2024-09-13T11:28:25.996704+0000","last_fullsized":"2024-09-13T11:28:25.996704+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:27:27.219203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:27:27.219203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:28:30.848078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010010360000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":238,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-13T11:28:25.268407+0000","last_change":"2024-09-13T11:27:26.709818+0000","last_active":"2024-09-13T11:28:25.268407+0000","last_peered":"2024-09-13T11:28:25.268407+0000","last_clean":"2024-09-13T11:28:25.268407+0000","last_became_active":"2024-09-13T11:26:05.046717+0000","last_became_peered":"2024-09-13T11:26:05.046717+0000","last_unstale":"2024-09-13T11:28:25.268407+0000","last_undegraded":"2024-09-13T11:28:25.268407+0000","last_fullsized":"2024-09-13T11:28:25.268407+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:27:26.709721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:27:26.709721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:28:41.582217+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00081225399999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,1],"acting":[14,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"2.2","version":"28'2","reported_seq":195,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-13T11:28:25.996689+0000","last_change":"2024-09-13T11:27:25.229244+0000","last_active":"2024-09-13T11:28:25.996689+0000","last_peered":"2024-09-13T11:28:25.996689+0000","last_clean":"2024-09-13T11:28:25.996689+0000","last_became_active":"2024-09-13T11:26:36.267081+0000","last_became_peered":"2024-09-13T11:26:36.267081+0000","last_unstale":"2024-09-13T11:28:25.996689+0000","last_undegraded":"2024-09-13T11:28:25.996689+0000","last_fullsized":"2024-09-13T11:28:25.996689+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2","last_scrub_stamp":"2024-09-13T11:27:25.229120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:27:25.229120+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:28:40.171329+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0025826730000000002,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9],"acting":[3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":586,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-09-13T11:28:30.301707+0000","last_change":"2024-09-13T11:28:27.979991+0000","last_active":"2024-09-13T11:28:30.301707+0000","last_peered":"2024-09-13T11:28:30.301707+0000","last_clean":"2024-09-13T11:28:30.301707+0000","last_became_active":"2024-09-13T11:22:47.719636+0000","last_became_peered":"2024-09-13T11:22:47.719636+0000","last_unstale":"2024-09-13T11:28:30.301707+0000","last_undegraded":"2024-09-13T11:28:30.301707+0000","last_fullsized":"2024-09-13T11:28:30.301707+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:28:27.979830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:28:27.979830+0000","last_clean_scrub_stamp":"2024-09-13T11:28:27.979830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:29:36.815622+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00087476000000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,9],"acting":[1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":610,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-09-13T11:28:30.744850+0000","last_change":"2024-09-13T11:28:26.626031+0000","last_active":"2024-09-13T11:28:30.744850+0000","last_peered":"2024-09-13T11:28:30.744850+0000","last_clean":"2024-09-13T11:28:30.744850+0000","last_became_active":"2024-09-13T11:22:14.581458+0000","last_became_peered":"2024-09-13T11:22:14.581458+0000","last_unstale":"2024-09-13T11:28:30.744850+0000","last_undegraded":"2024-09-13T11:28:30.744850+0000","last_fullsized":"2024-09-13T11:28:30.744850+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:28:26.625959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:28:26.625959+0000","last_clean_scrub_stamp":"2024-09-13T11:28:26.625959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:29:56.128806+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.001026047,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5],"acting":[8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":609,"reported_epoch":304,"state":"active+clean","last_fresh":"2024-09-13T11:28:32.004825+0000","last_change":"2024-09-13T11:28:29.142706+0000","last_active":"2024-09-13T11:28:32.004825+0000","last_peered":"2024-09-13T11:28:32.004825+0000","last_clean":"2024-09-13T11:28:32.004825+0000","last_became_active":"2024-09-13T11:22:14.211923+0000","last_became_peered":"2024-09-13T11:22:14.211923+0000","last_unstale":"2024-09-13T11:28:32.004825+0000","last_undegraded":"2024-09-13T11:28:32.004825+0000","last_fullsized":"2024-09-13T11:28:32.004825+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:28:29.142543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:28:29.142543+0000","last_clean_scrub_stamp":"2024-09-13T11:28:29.142543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:29:35.806459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011689529999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0],"acting":[9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":430,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-09-13T11:28:30.748041+0000","last_change":"2024-09-13T11:28:13.159189+0000","last_active":"2024-09-13T11:28:30.748041+0000","last_peered":"2024-09-13T11:28:30.748041+0000","last_clean":"2024-09-13T11:28:30.748041+0000","last_became_active":"2024-09-13T11:28:13.158553+0000","last_became_peered":"2024-09-13T11:28:13.158553+0000","last_unstale":"2024-09-13T11:28:30.748041+0000","last_undegraded":"2024-09-13T11:28:30.748041+0000","last_fullsized":"2024-09-13T11:28:30.748041+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:27:27.561975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:22:13.113563+0000","last_clean_scrub_stamp":"2024-09-13T11:27:27.561975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:28:56.783029+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,10],"acting":[15,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":11},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":114,"ondisk_log_size":114,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":13,"up_from":23,"seq":98784247889,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":100000,"kb_used_data":784,"kb_used_omap":1,"kb_used_meta":99198,"kb_avail":94271840,"statfs":{"total":96636764160,"available":96534364160,"internally_reserved":0,"allocated":802816,"data_stored":481556,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":101579205},"hb_peers":[0,1,2,4,6,8,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":9,"up_from":23,"seq":98784247888,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":94444,"kb_used_data":796,"kb_used_omap":1,"kb_used_meta":93630,"kb_avail":94277396,"statfs":{"total":96636764160,"available":96540053504,"internally_reserved":0,"allocated":815104,"data_stored":482765,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":95877568},"hb_peers":[0,3,4,8,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":40,"seq":171798691911,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":141208,"kb_used_data":904,"kb_used_omap":4,"kb_used_meta":140283,"kb_avail":94230632,"statfs":{"total":96636764160,"available":96492167168,"internally_reserved":0,"allocated":925696,"data_stored":568203,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4433,"internal_metadata":143650479},"hb_peers":[0,2,4,7,8,9,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":21,"seq":90194313296,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":158440,"kb_used_data":920,"kb_used_omap":1,"kb_used_meta":157502,"kb_avail":94213400,"statfs":{"total":96636764160,"available":96474521600,"internally_reserved":0,"allocated":942080,"data_stored":568317,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":161282503},"hb_peers":[0,2,4,9,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":21,"seq":90194313299,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":166732,"kb_used_data":892,"kb_used_omap":1,"kb_used_meta":165822,"kb_avail":94205108,"statfs":{"total":96636764160,"available":96466030592,"internally_reserved":0,"allocated":913408,"data_stored":567077,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":169802183},"hb_peers":[0,2,4,6,7,8,9,11,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":22,"seq":94489280592,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":177060,"kb_used_data":1492,"kb_used_omap":1,"kb_used_meta":175550,"kb_avail":94194780,"statfs":{"total":96636764160,"available":96455454720,"internally_reserved":0,"allocated":1527808,"data_stored":1158434,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":179763653},"hb_peers":[1,3,5,8,9,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":22,"seq":94489280592,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":94540,"kb_used_data":892,"kb_used_omap":1,"kb_used_meta":93630,"kb_avail":94277300,"statfs":{"total":96636764160,"available":96539955200,"internally_reserved":0,"allocated":913408,"data_stored":566506,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":95877573},"hb_peers":[0,1,3,4,7,8,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":15,"up_from":22,"seq":94489280592,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":144536,"kb_used_data":904,"kb_used_omap":1,"kb_used_meta":143614,"kb_avail":94227304,"statfs":{"total":96636764160,"available":96488759296,"internally_reserved":0,"allocated":925696,"data_stored":566485,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":147061190},"hb_peers":[0,4,5,6,7,8,10,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":22,"seq":94489280593,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":86116,"kb_used_data":788,"kb_used_omap":1,"kb_used_meta":85310,"kb_avail":94285724,"statfs":{"total":96636764160,"available":96548581376,"internally_reserved":0,"allocated":806912,"data_stored":482557,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":87357893},"hb_peers":[3,5,6,7,9,10,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":22,"seq":94489280592,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72912,"kb_used_data":1472,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":94298928,"statfs":{"total":96636764160,"available":96562102272,"internally_reserved":0,"allocated":1507328,"data_stored":1157064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":73136582},"hb_peers":[0,1,2,3,5,7,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":22,"seq":94489280593,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":162576,"kb_used_data":896,"kb_used_omap":1,"kb_used_meta":161662,"kb_avail":94209264,"statfs":{"total":96636764160,"available":96470286336,"internally_reserved":0,"allocated":917504,"data_stored":567619,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":165542346},"hb_peers":[0,2,5,6,8,9,11,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":8,"up_from":22,"seq":94489280592,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":100120,"kb_used_data":904,"kb_used_omap":1,"kb_used_meta":99198,"kb_avail":94271720,"statfs":{"total":96636764160,"available":96534241280,"internally_reserved":0,"allocated":925696,"data_stored":567734,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":101579205},"hb_peers":[1,2,5,7,9,10,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":10,"up_from":261,"seq":1120986464269,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":39604,"kb_used_data":804,"kb_used_omap":2,"kb_used_meta":38781,"kb_avail":94332236,"statfs":{"total":96636764160,"available":96596209664,"internally_reserved":0,"allocated":823296,"data_stored":483573,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2702,"internal_metadata":39712114},"hb_peers":[0,1,3,7,9,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":11,"up_from":22,"seq":94489280593,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":154164,"kb_used_data":804,"kb_used_omap":1,"kb_used_meta":153342,"kb_avail":94217676,"statfs":{"total":96636764160,"available":96478900224,"internally_reserved":0,"allocated":823296,"data_stored":489602,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":157022660},"hb_peers":[2,4,7,8,9,10,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":12,"up_from":22,"seq":94489280592,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":166616,"kb_used_data":776,"kb_used_omap":1,"kb_used_meta":165822,"kb_avail":94205224,"statfs":{"total":96636764160,"available":96466149376,"internally_reserved":0,"allocated":794624,"data_stored":481368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":169802181},"hb_peers":[1,2,3,5,6,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":14,"up_from":142,"seq":609885356077,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":109152,"kb_used_data":912,"kb_used_omap":42,"kb_used_meta":108181,"kb_avail":94262688,"statfs":{"total":96636764160,"available":96524992512,"internally_reserved":0,"allocated":933888,"data_stored":567204,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":43282,"internal_metadata":110778094},"hb_peers":[0,1,4,5,7,9,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T11:28:33.163 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2024-09-13T11:27:25.752752+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=11, tm_min=27, tm_sec=25, tm_wday=4, tm_yday=257, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=11, tm_min=28, tm_sec=23, tm_wday=4, tm_yday=257, tm_isdst=0) 2024-09-13T11:28:33.164 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2024-09-13T11:27:27.219203+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=11, tm_min=27, tm_sec=27, tm_wday=4, tm_yday=257, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=11, tm_min=28, tm_sec=23, tm_wday=4, tm_yday=257, tm_isdst=0) 2024-09-13T11:28:33.164 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2024-09-13T11:27:26.709721+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=11, tm_min=27, tm_sec=26, tm_wday=4, tm_yday=257, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=11, tm_min=28, tm_sec=23, tm_wday=4, tm_yday=257, tm_isdst=0) 2024-09-13T11:28:33.164 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2024-09-13T11:27:25.229120+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=11, tm_min=27, tm_sec=25, tm_wday=4, tm_yday=257, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=11, tm_min=28, tm_sec=23, tm_wday=4, tm_yday=257, tm_isdst=0) 2024-09-13T11:28:33.164 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2024-09-13T11:27:27.561975+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=11, tm_min=27, tm_sec=27, tm_wday=4, tm_yday=257, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=11, tm_min=28, tm_sec=23, tm_wday=4, tm_yday=257, tm_isdst=0) 2024-09-13T11:28:33.164 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-09-13T11:28:36.317 INFO:tasks.ceph.osd.11.smithi177.stderr:2024-09-13T11:28:36.319+0000 7f246bcf0640 -1 osd.11 304 handle_osd_map injecting map crc failure 2024-09-13T11:28:43.294 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-09-13T11:28:43.290+0000 7f97b39f6640 -1 osd.6 306 handle_osd_map injecting map crc failure 2024-09-13T11:28:46.356 INFO:tasks.ceph.osd.3.smithi177.stderr:2024-09-13T11:28:46.355+0000 7f10a25d1640 -1 osd.3 307 handle_osd_map injecting map crc failure 2024-09-13T11:28:46.356 INFO:tasks.ceph.osd.5.smithi092.stderr:2024-09-13T11:28:46.354+0000 7f6f00132640 -1 osd.5 307 handle_osd_map injecting map crc failure 2024-09-13T11:28:46.357 INFO:tasks.ceph.osd.14.smithi162.stderr:2024-09-13T11:28:46.354+0000 7f3dea330640 -1 osd.14 307 handle_osd_map injecting map crc failure 2024-09-13T11:28:51.284 INFO:tasks.ceph.osd.10.smithi162.stderr:2024-09-13T11:28:51.282+0000 7ff88891d640 -1 osd.10 309 handle_osd_map injecting map crc failure 2024-09-13T11:28:52.245 INFO:tasks.ceph.osd.13.smithi092.stderr:2024-09-13T11:28:52.242+0000 7f3e4ba0f640 -1 osd.13 309 handle_osd_map injecting map crc failure 2024-09-13T11:28:52.270 INFO:tasks.ceph.osd.8.smithi001.stderr:2024-09-13T11:28:52.269+0000 7fb9a92db640 -1 osd.8 309 handle_osd_map injecting map crc failure 2024-09-13T11:28:53.166 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-13T11:28:53.491 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-13T11:28:53.491 INFO:teuthology.orchestra.run.smithi001.stderr:dumped all 2024-09-13T11:28:53.510 INFO:teuthology.orchestra.run.smithi001.stdout:{"pg_ready":true,"pg_map":{"version":497,"stamp":"2024-09-13T11:28:52.219380+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":147,"num_write_kb":2674,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":10,"kb":1509949440,"kb_used":1965404,"kb_used_data":12124,"kb_used_omap":69,"kb_used_meta":1952954,"kb_avail":1507984036,"statfs":{"total":1546188226560,"available":1544175652864,"internally_reserved":0,"allocated":12414976,"data_stored":7780315,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":71147,"internal_metadata":1999825429},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":21,"apply_latency_ms":21,"commit_latency_ns":21000000,"apply_latency_ns":21000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.106308"},"pg_stats":[{"pgid":"1.0","version":"302'114","reported_seq":755,"reported_epoch":309,"state":"active+clean","last_fresh":"2024-09-13T11:28:49.055012+0000","last_change":"2024-09-13T11:28:29.934970+0000","last_active":"2024-09-13T11:28:49.055012+0000","last_peered":"2024-09-13T11:28:49.055012+0000","last_clean":"2024-09-13T11:28:49.055012+0000","last_became_active":"2024-09-13T11:22:12.129721+0000","last_became_peered":"2024-09-13T11:22:12.129721+0000","last_unstale":"2024-09-13T11:28:49.055012+0000","last_undegraded":"2024-09-13T11:28:49.055012+0000","last_fullsized":"2024-09-13T11:28:49.055012+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"302'114","last_scrub_stamp":"2024-09-13T11:28:29.934796+0000","last_deep_scrub":"302'114","last_deep_scrub_stamp":"2024-09-13T11:28:29.934796+0000","last_clean_scrub_stamp":"2024-09-13T11:28:29.934796+0000","objects_scrubbed":2,"log_size":114,"log_dups_size":0,"ondisk_log_size":114,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:29:56.601457+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.3","version":"304'2","reported_seq":263,"reported_epoch":309,"state":"active+clean","last_fresh":"2024-09-13T11:28:48.129341+0000","last_change":"2024-09-13T11:28:32.788278+0000","last_active":"2024-09-13T11:28:48.129341+0000","last_peered":"2024-09-13T11:28:48.129341+0000","last_clean":"2024-09-13T11:28:48.129341+0000","last_became_active":"2024-09-13T11:28:17.223376+0000","last_became_peered":"2024-09-13T11:28:17.223376+0000","last_unstale":"2024-09-13T11:28:48.129341+0000","last_undegraded":"2024-09-13T11:28:48.129341+0000","last_fullsized":"2024-09-13T11:28:48.129341+0000","mapping_epoch":296,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":297,"parent":"0.0","parent_split_bits":0,"last_scrub":"304'2","last_scrub_stamp":"2024-09-13T11:28:32.788072+0000","last_deep_scrub":"304'2","last_deep_scrub_stamp":"2024-09-13T11:28:32.788072+0000","last_clean_scrub_stamp":"2024-09-13T11:28:32.788072+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:29:57.490180+0000","scrub_duration":75,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,15],"acting":[14,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":608,"reported_epoch":309,"state":"active+clean","last_fresh":"2024-09-13T11:28:49.056163+0000","last_change":"2024-09-13T11:28:33.217738+0000","last_active":"2024-09-13T11:28:49.056163+0000","last_peered":"2024-09-13T11:28:49.056163+0000","last_clean":"2024-09-13T11:28:49.056163+0000","last_became_active":"2024-09-13T11:22:14.211435+0000","last_became_peered":"2024-09-13T11:22:14.211435+0000","last_unstale":"2024-09-13T11:28:49.056163+0000","last_undegraded":"2024-09-13T11:28:49.056163+0000","last_fullsized":"2024-09-13T11:28:49.056163+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:28:33.217520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:28:33.217520+0000","last_clean_scrub_stamp":"2024-09-13T11:28:33.217520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:29:56.601175+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0010010360000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":260,"reported_epoch":309,"state":"active+clean","last_fresh":"2024-09-13T11:28:48.129303+0000","last_change":"2024-09-13T11:28:31.668034+0000","last_active":"2024-09-13T11:28:48.129303+0000","last_peered":"2024-09-13T11:28:48.129303+0000","last_clean":"2024-09-13T11:28:48.129303+0000","last_became_active":"2024-09-13T11:26:05.046717+0000","last_became_peered":"2024-09-13T11:26:05.046717+0000","last_unstale":"2024-09-13T11:28:48.129303+0000","last_undegraded":"2024-09-13T11:28:48.129303+0000","last_fullsized":"2024-09-13T11:28:48.129303+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:28:31.667864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:28:31.667864+0000","last_clean_scrub_stamp":"2024-09-13T11:28:31.667864+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:29:39.916285+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00081225399999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,1],"acting":[14,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"2.2","version":"303'3","reported_seq":219,"reported_epoch":309,"state":"active+clean","last_fresh":"2024-09-13T11:28:49.056223+0000","last_change":"2024-09-13T11:28:32.192011+0000","last_active":"2024-09-13T11:28:49.056223+0000","last_peered":"2024-09-13T11:28:49.056223+0000","last_clean":"2024-09-13T11:28:49.056223+0000","last_became_active":"2024-09-13T11:26:36.267081+0000","last_became_peered":"2024-09-13T11:26:36.267081+0000","last_unstale":"2024-09-13T11:28:49.056223+0000","last_undegraded":"2024-09-13T11:28:49.056223+0000","last_fullsized":"2024-09-13T11:28:49.056223+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"303'3","last_scrub_stamp":"2024-09-13T11:28:32.191526+0000","last_deep_scrub":"303'3","last_deep_scrub_stamp":"2024-09-13T11:28:32.191526+0000","last_clean_scrub_stamp":"2024-09-13T11:28:32.191526+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:29:58.157308+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0025826730000000002,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9],"acting":[3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":597,"reported_epoch":309,"state":"active+clean","last_fresh":"2024-09-13T11:28:48.965456+0000","last_change":"2024-09-13T11:28:27.979991+0000","last_active":"2024-09-13T11:28:48.965456+0000","last_peered":"2024-09-13T11:28:48.965456+0000","last_clean":"2024-09-13T11:28:48.965456+0000","last_became_active":"2024-09-13T11:22:47.719636+0000","last_became_peered":"2024-09-13T11:22:47.719636+0000","last_unstale":"2024-09-13T11:28:48.965456+0000","last_undegraded":"2024-09-13T11:28:48.965456+0000","last_fullsized":"2024-09-13T11:28:48.965456+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:28:27.979830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:28:27.979830+0000","last_clean_scrub_stamp":"2024-09-13T11:28:27.979830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:29:36.815622+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00087476000000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,9],"acting":[1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":622,"reported_epoch":309,"state":"active+clean","last_fresh":"2024-09-13T11:28:49.147314+0000","last_change":"2024-09-13T11:28:26.626031+0000","last_active":"2024-09-13T11:28:49.147314+0000","last_peered":"2024-09-13T11:28:49.147314+0000","last_clean":"2024-09-13T11:28:49.147314+0000","last_became_active":"2024-09-13T11:22:14.581458+0000","last_became_peered":"2024-09-13T11:22:14.581458+0000","last_unstale":"2024-09-13T11:28:49.147314+0000","last_undegraded":"2024-09-13T11:28:49.147314+0000","last_fullsized":"2024-09-13T11:28:49.147314+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:28:26.625959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:28:26.625959+0000","last_clean_scrub_stamp":"2024-09-13T11:28:26.625959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:29:56.128806+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.001026047,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5],"acting":[8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":622,"reported_epoch":310,"state":"active+clean","last_fresh":"2024-09-13T11:28:51.694814+0000","last_change":"2024-09-13T11:28:29.142706+0000","last_active":"2024-09-13T11:28:51.694814+0000","last_peered":"2024-09-13T11:28:51.694814+0000","last_clean":"2024-09-13T11:28:51.694814+0000","last_became_active":"2024-09-13T11:22:14.211923+0000","last_became_peered":"2024-09-13T11:22:14.211923+0000","last_unstale":"2024-09-13T11:28:51.694814+0000","last_undegraded":"2024-09-13T11:28:51.694814+0000","last_fullsized":"2024-09-13T11:28:51.694814+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:28:29.142543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:28:29.142543+0000","last_clean_scrub_stamp":"2024-09-13T11:28:29.142543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:29:35.806459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011689529999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0],"acting":[9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":450,"reported_epoch":309,"state":"active+clean","last_fresh":"2024-09-13T11:28:48.799668+0000","last_change":"2024-09-13T11:28:33.225132+0000","last_active":"2024-09-13T11:28:48.799668+0000","last_peered":"2024-09-13T11:28:48.799668+0000","last_clean":"2024-09-13T11:28:48.799668+0000","last_became_active":"2024-09-13T11:28:13.158553+0000","last_became_peered":"2024-09-13T11:28:13.158553+0000","last_unstale":"2024-09-13T11:28:48.799668+0000","last_undegraded":"2024-09-13T11:28:48.799668+0000","last_fullsized":"2024-09-13T11:28:48.799668+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:28:33.224960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:28:33.224960+0000","last_clean_scrub_stamp":"2024-09-13T11:28:33.224960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:29:46.516784+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,10],"acting":[15,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":16,"acting":16,"num_store_stats":11},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":114,"ondisk_log_size":114,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":13,"up_from":23,"seq":98784247893,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":99920,"kb_used_data":704,"kb_used_omap":1,"kb_used_meta":99198,"kb_avail":94271920,"statfs":{"total":96636764160,"available":96534446080,"internally_reserved":0,"allocated":720896,"data_stored":413962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":101579205},"hb_peers":[0,1,2,4,6,8,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":9,"up_from":23,"seq":98784247892,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":94340,"kb_used_data":692,"kb_used_omap":1,"kb_used_meta":93630,"kb_avail":94277500,"statfs":{"total":96636764160,"available":96540160000,"internally_reserved":0,"allocated":708608,"data_stored":412939,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":95877568},"hb_peers":[0,3,4,8,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":40,"seq":171798691915,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":140988,"kb_used_data":684,"kb_used_omap":4,"kb_used_meta":140283,"kb_avail":94230852,"statfs":{"total":96636764160,"available":96492392448,"internally_reserved":0,"allocated":700416,"data_stored":412739,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4433,"internal_metadata":143650479},"hb_peers":[0,2,4,7,8,9,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":21,"seq":90194313300,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":158204,"kb_used_data":684,"kb_used_omap":1,"kb_used_meta":157502,"kb_avail":94213636,"statfs":{"total":96636764160,"available":96474763264,"internally_reserved":0,"allocated":700416,"data_stored":412422,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":161282503},"hb_peers":[0,2,4,9,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":21,"seq":90194313303,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":166512,"kb_used_data":672,"kb_used_omap":1,"kb_used_meta":165822,"kb_avail":94205328,"statfs":{"total":96636764160,"available":96466255872,"internally_reserved":0,"allocated":688128,"data_stored":411786,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":169802183},"hb_peers":[0,2,4,6,7,8,9,11,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":22,"seq":94489280596,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":176836,"kb_used_data":1268,"kb_used_omap":1,"kb_used_meta":175550,"kb_avail":94195004,"statfs":{"total":96636764160,"available":96455684096,"internally_reserved":0,"allocated":1298432,"data_stored":1003359,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":179763653},"hb_peers":[1,3,5,8,9,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":22,"seq":94489280596,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":94320,"kb_used_data":672,"kb_used_omap":1,"kb_used_meta":93630,"kb_avail":94277520,"statfs":{"total":96636764160,"available":96540180480,"internally_reserved":0,"allocated":688128,"data_stored":411632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":95877573},"hb_peers":[0,1,3,4,7,8,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":15,"up_from":22,"seq":94489280596,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":144312,"kb_used_data":680,"kb_used_omap":1,"kb_used_meta":143614,"kb_avail":94227528,"statfs":{"total":96636764160,"available":96488988672,"internally_reserved":0,"allocated":696320,"data_stored":410729,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":147061190},"hb_peers":[0,4,5,6,7,8,10,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":22,"seq":94489280597,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":86016,"kb_used_data":688,"kb_used_omap":1,"kb_used_meta":85310,"kb_avail":94285824,"statfs":{"total":96636764160,"available":96548683776,"internally_reserved":0,"allocated":704512,"data_stored":412983,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":87357893},"hb_peers":[3,5,6,7,9,10,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":22,"seq":94489280596,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72688,"kb_used_data":1248,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":94299152,"statfs":{"total":96636764160,"available":96562331648,"internally_reserved":0,"allocated":1277952,"data_stored":1001467,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":73136582},"hb_peers":[0,1,2,3,5,7,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":22,"seq":94489280597,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":162356,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":161662,"kb_avail":94209484,"statfs":{"total":96636764160,"available":96470511616,"internally_reserved":0,"allocated":692224,"data_stored":412219,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":165542346},"hb_peers":[0,2,5,6,8,9,11,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":22,"seq":94489280596,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":99904,"kb_used_data":688,"kb_used_omap":1,"kb_used_meta":99198,"kb_avail":94271936,"statfs":{"total":96636764160,"available":96534462464,"internally_reserved":0,"allocated":704512,"data_stored":412991,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":101579205},"hb_peers":[1,2,5,7,9,10,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":10,"up_from":261,"seq":1120986464273,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":39504,"kb_used_data":704,"kb_used_omap":2,"kb_used_meta":38781,"kb_avail":94332336,"statfs":{"total":96636764160,"available":96596312064,"internally_reserved":0,"allocated":720896,"data_stored":414041,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2702,"internal_metadata":39712114},"hb_peers":[0,1,3,7,9,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":11,"up_from":22,"seq":94489280597,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":154056,"kb_used_data":696,"kb_used_omap":1,"kb_used_meta":153342,"kb_avail":94217784,"statfs":{"total":96636764160,"available":96479010816,"internally_reserved":0,"allocated":712704,"data_stored":413374,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":157022660},"hb_peers":[2,4,7,8,9,10,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":12,"up_from":22,"seq":94489280596,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":166520,"kb_used_data":680,"kb_used_omap":1,"kb_used_meta":165822,"kb_avail":94205320,"statfs":{"total":96636764160,"available":96466247680,"internally_reserved":0,"allocated":696320,"data_stored":412207,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":169802181},"hb_peers":[1,2,3,5,6,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":14,"up_from":142,"seq":609885356081,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":108928,"kb_used_data":688,"kb_used_omap":42,"kb_used_meta":108181,"kb_avail":94262912,"statfs":{"total":96636764160,"available":96525221888,"internally_reserved":0,"allocated":704512,"data_stored":411465,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":43282,"internal_metadata":110778094},"hb_peers":[0,1,4,5,7,9,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T11:28:53.511 DEBUG:teuthology.orchestra.run.smithi001:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2024-09-13T11:28:53.854 INFO:teuthology.misc:Shutting down mds daemons... 2024-09-13T11:28:53.854 INFO:teuthology.misc:Shutting down osd daemons... 2024-09-13T11:28:53.854 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-09-13T11:28:53.854 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T11:28:54.839 INFO:tasks.ceph.osd.11.smithi177.stderr:2024-09-13T11:28:54.839+0000 7f246bcf0640 -1 osd.11 310 handle_osd_map injecting map crc failure 2024-09-13T11:28:56.619 INFO:tasks.ceph.osd.1.smithi092.stderr:2024-09-13T11:28:56.618+0000 7fe76d3f2640 -1 osd.1 311 handle_osd_map injecting map crc failure 2024-09-13T11:28:56.619 INFO:tasks.ceph.osd.8.smithi001.stderr:2024-09-13T11:28:56.617+0000 7fb9a92db640 -1 osd.8 311 handle_osd_map injecting map crc failure 2024-09-13T11:28:57.894 INFO:tasks.ceph.osd.5.smithi092.stderr:2024-09-13T11:28:57.890+0000 7f6f00132640 -1 osd.5 312 handle_osd_map injecting map crc failure 2024-09-13T11:28:57.899 INFO:tasks.ceph.osd.5.smithi092.stderr:2024-09-13T11:28:57.898+0000 7f6f00132640 -1 osd.5 312 handle_osd_map injecting map crc failure 2024-09-13T11:28:59.956 INFO:tasks.ceph.osd.0:Stopped 2024-09-13T11:28:59.956 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-09-13T11:28:59.956 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T11:29:00.392 INFO:tasks.ceph.osd.9.smithi092.stderr:2024-09-13T11:29:00.390+0000 7ff59bdf5640 -1 osd.9 314 handle_osd_map injecting map crc failure 2024-09-13T11:29:00.433 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~0s 2024-09-13T11:29:01.162 INFO:tasks.ceph.osd.13.smithi092.stderr:2024-09-13T11:29:01.162+0000 7f3e4ba0f640 -1 osd.13 314 handle_osd_map injecting map crc failure 2024-09-13T11:29:01.345 INFO:tasks.ceph.osd.1.smithi092.stderr:2024-09-13T11:29:01.342+0000 7fe76d3f2640 -1 osd.1 315 handle_osd_map injecting map crc failure 2024-09-13T11:29:01.452 INFO:tasks.ceph.osd.3.smithi177.stderr:2024-09-13T11:29:01.451+0000 7f10a25d1640 -1 osd.3 314 handle_osd_map injecting map crc failure 2024-09-13T11:29:06.058 INFO:tasks.ceph.osd.4:Stopped 2024-09-13T11:29:06.058 DEBUG:tasks.ceph.osd.8:waiting for process to exit 2024-09-13T11:29:06.058 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T11:29:06.370 INFO:tasks.ceph.osd.7.smithi177.stderr:2024-09-13T11:29:06.371+0000 7f8971f4e640 -1 osd.7 316 handle_osd_map injecting map crc failure 2024-09-13T11:29:06.371 INFO:tasks.ceph.osd.1.smithi092.stderr:2024-09-13T11:29:06.370+0000 7fe76f3f6640 -1 osd.1 316 handle_osd_map injecting map crc failure 2024-09-13T11:29:06.593 INFO:tasks.ceph.osd.14.smithi162.stderr:2024-09-13T11:29:06.590+0000 7f3dea330640 -1 osd.14 316 handle_osd_map injecting map crc failure 2024-09-13T11:29:07.291 INFO:tasks.ceph.osd.9.smithi092.stderr:2024-09-13T11:29:07.290+0000 7ff59ddf9640 -1 osd.9 317 handle_osd_map injecting map crc failure 2024-09-13T11:29:07.344 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~0s 2024-09-13T11:29:07.344 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 has been restored 2024-09-13T11:29:08.352 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-09-13T11:29:08.350+0000 7f97b39f6640 -1 osd.6 317 handle_osd_map injecting map crc failure 2024-09-13T11:29:08.354 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-09-13T11:29:08.350+0000 7f97b39f6640 -1 osd.6 317 handle_osd_map injecting map crc failure 2024-09-13T11:29:08.777 INFO:tasks.ceph.osd.13.smithi092.stderr:2024-09-13T11:29:08.778+0000 7f3e4ba0f640 -1 osd.13 318 handle_osd_map injecting map crc failure 2024-09-13T11:29:12.160 INFO:tasks.ceph.osd.8:Stopped 2024-09-13T11:29:12.161 DEBUG:tasks.ceph.osd.12:waiting for process to exit 2024-09-13T11:29:12.161 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T11:29:12.325 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-09-13T11:29:12.322+0000 7f97b59fa640 -1 osd.6 319 handle_osd_map injecting map crc failure 2024-09-13T11:29:14.058 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.12 is failed for ~0s 2024-09-13T11:29:14.058 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 has been restored 2024-09-13T11:29:18.263 INFO:tasks.ceph.osd.12:Stopped 2024-09-13T11:29:18.264 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-09-13T11:29:18.264 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T11:29:19.398 INFO:tasks.ceph.osd.11.smithi177.stderr:2024-09-13T11:29:19.399+0000 7f2469cec640 -1 osd.11 320 handle_osd_map injecting map crc failure 2024-09-13T11:29:20.672 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~0s 2024-09-13T11:29:20.673 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.12 has been restored 2024-09-13T11:29:21.066 INFO:tasks.ceph.osd.2.smithi162.stderr:2024-09-13T11:29:21.066+0000 7fe91fd87640 -1 osd.2 321 handle_osd_map injecting map crc failure 2024-09-13T11:29:21.066 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-09-13T11:29:21.066+0000 7f97b39f6640 -1 osd.6 321 handle_osd_map injecting map crc failure 2024-09-13T11:29:21.524 INFO:tasks.ceph.osd.3.smithi177.stderr:2024-09-13T11:29:21.527+0000 7f10a25d1640 -1 osd.3 321 handle_osd_map injecting map crc failure 2024-09-13T11:29:24.366 INFO:tasks.ceph.osd.1:Stopped 2024-09-13T11:29:24.367 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-09-13T11:29:24.367 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T11:29:27.181 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~0s 2024-09-13T11:29:27.196 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 has been restored 2024-09-13T11:29:30.469 INFO:tasks.ceph.osd.5:Stopped 2024-09-13T11:29:30.470 DEBUG:tasks.ceph.osd.9:waiting for process to exit 2024-09-13T11:29:30.470 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T11:29:33.607 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.9 is failed for ~0s 2024-09-13T11:29:33.607 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 has been restored 2024-09-13T11:29:33.670 INFO:tasks.ceph.osd.14.smithi162.stderr:2024-09-13T11:29:33.669+0000 7f3dea330640 -1 osd.14 325 handle_osd_map injecting map crc failure 2024-09-13T11:29:36.160 INFO:tasks.ceph.osd.3.smithi177.stderr:2024-09-13T11:29:36.159+0000 7f10a25d1640 -1 osd.3 325 handle_osd_map injecting map crc failure 2024-09-13T11:29:36.572 INFO:tasks.ceph.osd.9:Stopped 2024-09-13T11:29:36.572 DEBUG:tasks.ceph.osd.13:waiting for process to exit 2024-09-13T11:29:36.572 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T11:29:37.497 INFO:tasks.ceph.osd.7.smithi177.stderr:2024-09-13T11:29:37.499+0000 7f896ff4a640 -1 osd.7 326 handle_osd_map injecting map crc failure 2024-09-13T11:29:39.920 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.13 is failed for ~0s 2024-09-13T11:29:39.920 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.9 has been restored 2024-09-13T11:29:42.674 INFO:tasks.ceph.osd.13:Stopped 2024-09-13T11:29:42.675 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-09-13T11:29:42.675 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T11:29:43.369 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-09-13T11:29:43.365+0000 7f97b39f6640 -1 osd.6 327 handle_osd_map injecting map crc failure 2024-09-13T11:29:46.126 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~0s 2024-09-13T11:29:46.126 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.13 has been restored 2024-09-13T11:29:48.776 INFO:tasks.ceph.osd.2:Stopped 2024-09-13T11:29:48.776 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2024-09-13T11:29:48.776 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T11:29:48.849 INFO:tasks.ceph.osd.6:Stopped 2024-09-13T11:29:48.849 DEBUG:tasks.ceph.osd.10:waiting for process to exit 2024-09-13T11:29:48.849 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T11:29:49.361 INFO:tasks.ceph.osd.15.smithi177.stderr:2024-09-13T11:29:49.362+0000 7f9d638a8640 -1 osd.15 328 handle_osd_map injecting map crc failure 2024-09-13T11:29:52.133 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.10 is failed for ~0s 2024-09-13T11:29:52.133 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 has been restored 2024-09-13T11:29:54.950 INFO:tasks.ceph.osd.10:Stopped 2024-09-13T11:29:54.951 DEBUG:tasks.ceph.osd.14:waiting for process to exit 2024-09-13T11:29:54.951 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T11:29:55.337 INFO:tasks.ceph.osd.7.smithi177.stderr:2024-09-13T11:29:55.338+0000 7f8971f4e640 -1 osd.7 331 handle_osd_map injecting map crc failure 2024-09-13T11:29:58.039 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.14 is failed for ~0s 2024-09-13T11:29:58.039 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.10 has been restored 2024-09-13T11:30:01.053 INFO:tasks.ceph.osd.14:Stopped 2024-09-13T11:30:01.054 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-09-13T11:30:01.054 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T11:30:03.847 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~0s 2024-09-13T11:30:03.847 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.14 has been restored 2024-09-13T11:30:07.155 INFO:tasks.ceph.osd.3:Stopped 2024-09-13T11:30:07.156 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2024-09-13T11:30:07.156 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T11:30:09.551 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~0s 2024-09-13T11:30:09.551 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 has been restored 2024-09-13T11:30:13.258 INFO:tasks.ceph.osd.7:Stopped 2024-09-13T11:30:13.259 DEBUG:tasks.ceph.osd.11:waiting for process to exit 2024-09-13T11:30:13.259 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T11:30:15.156 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.11 is failed for ~0s 2024-09-13T11:30:15.156 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 has been restored 2024-09-13T11:30:19.361 INFO:tasks.ceph.osd.11:Stopped 2024-09-13T11:30:19.361 DEBUG:tasks.ceph.osd.15:waiting for process to exit 2024-09-13T11:30:19.361 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T11:30:19.453 INFO:tasks.ceph.osd.15:Stopped 2024-09-13T11:30:19.454 INFO:teuthology.misc:Shutting down mgr daemons... 2024-09-13T11:30:19.454 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2024-09-13T11:30:19.454 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T11:30:19.486 INFO:tasks.ceph.mgr.y:Stopped 2024-09-13T11:30:19.487 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-09-13T11:30:19.487 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T11:30:19.520 INFO:tasks.ceph.mgr.x:Stopped 2024-09-13T11:30:19.521 INFO:teuthology.misc:Shutting down mon daemons... 2024-09-13T11:30:19.521 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-09-13T11:30:19.521 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T11:30:19.549 INFO:tasks.ceph.mon.a:Stopped 2024-09-13T11:30:19.550 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-09-13T11:30:19.550 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T11:30:19.576 INFO:tasks.ceph.mon.b:Stopped 2024-09-13T11:30:19.576 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-09-13T11:30:19.576 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T11:30:19.600 INFO:tasks.ceph.mon.c:Stopped 2024-09-13T11:30:19.601 INFO:tasks.ceph:Checking cluster log for badness... 2024-09-13T11:30:19.601 DEBUG:teuthology.orchestra.run.smithi001:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(REQUEST_STUCK\)' | egrep -v 'failed to encode map' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v 'overall HEALTH_' | egrep -v '\(OSDMAP_FLAGS\)' | egrep -v '\(OSD_' | egrep -v '\(PG_' | egrep -v '\(POOL_' | egrep -v '\(CACHE_POOL_' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(OBJECT_' | egrep -v SLOW_OPS | egrep -v '\(REQUEST_SLOW\)' | egrep -v '\(TOO_FEW_PGS\)' | egrep -v 'slow request' | egrep -v 'timeout on replica' | egrep -v 'late reservation from' | egrep -v MON_DOWN | egrep -v OSDMAP_FLAGS | egrep -v OSD_DOWN | egrep -v PG_DEGRADED | egrep -v PG_AVAILABILITY | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'mons down' | egrep -v 'mon down' | egrep -v 'out of quorum' | egrep -v noscrub | egrep -v nodeep-scrub | egrep -v '\(POOL_APP_NOT_ENABLED\)' | head -n 1 2024-09-13T11:30:19.621 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi001.front.sepia.ceph.com 2024-09-13T11:30:19.621 DEBUG:teuthology.orchestra.run.smithi001:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-09-13T11:30:19.842 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi001.front.sepia.ceph.com 2024-09-13T11:30:19.842 DEBUG:teuthology.orchestra.run.smithi001:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-09-13T11:30:20.004 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-8 on ubuntu@smithi001.front.sepia.ceph.com 2024-09-13T11:30:20.004 DEBUG:teuthology.orchestra.run.smithi001:> sync && sudo umount -f /var/lib/ceph/osd/ceph-8 2024-09-13T11:30:20.157 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.11 has been restored 2024-09-13T11:30:20.163 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-12 on ubuntu@smithi001.front.sepia.ceph.com 2024-09-13T11:30:20.163 DEBUG:teuthology.orchestra.run.smithi001:> sync && sudo umount -f /var/lib/ceph/osd/ceph-12 2024-09-13T11:30:20.320 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi092.front.sepia.ceph.com 2024-09-13T11:30:20.320 DEBUG:teuthology.orchestra.run.smithi092:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-09-13T11:30:20.468 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi092.front.sepia.ceph.com 2024-09-13T11:30:20.468 DEBUG:teuthology.orchestra.run.smithi092:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-09-13T11:30:20.642 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-9 on ubuntu@smithi092.front.sepia.ceph.com 2024-09-13T11:30:20.642 DEBUG:teuthology.orchestra.run.smithi092:> sync && sudo umount -f /var/lib/ceph/osd/ceph-9 2024-09-13T11:30:20.825 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-13 on ubuntu@smithi092.front.sepia.ceph.com 2024-09-13T11:30:20.826 DEBUG:teuthology.orchestra.run.smithi092:> sync && sudo umount -f /var/lib/ceph/osd/ceph-13 2024-09-13T11:30:21.022 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi162.front.sepia.ceph.com 2024-09-13T11:30:21.023 DEBUG:teuthology.orchestra.run.smithi162:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-09-13T11:30:21.139 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi162.front.sepia.ceph.com 2024-09-13T11:30:21.140 DEBUG:teuthology.orchestra.run.smithi162:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2024-09-13T11:30:21.294 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-10 on ubuntu@smithi162.front.sepia.ceph.com 2024-09-13T11:30:21.294 DEBUG:teuthology.orchestra.run.smithi162:> sync && sudo umount -f /var/lib/ceph/osd/ceph-10 2024-09-13T11:30:21.455 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-14 on ubuntu@smithi162.front.sepia.ceph.com 2024-09-13T11:30:21.455 DEBUG:teuthology.orchestra.run.smithi162:> sync && sudo umount -f /var/lib/ceph/osd/ceph-14 2024-09-13T11:30:21.602 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi177.front.sepia.ceph.com 2024-09-13T11:30:21.603 DEBUG:teuthology.orchestra.run.smithi177:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-09-13T11:30:21.889 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi177.front.sepia.ceph.com 2024-09-13T11:30:21.890 DEBUG:teuthology.orchestra.run.smithi177:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2024-09-13T11:30:22.102 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-11 on ubuntu@smithi177.front.sepia.ceph.com 2024-09-13T11:30:22.102 DEBUG:teuthology.orchestra.run.smithi177:> sync && sudo umount -f /var/lib/ceph/osd/ceph-11 2024-09-13T11:30:22.304 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-15 on ubuntu@smithi177.front.sepia.ceph.com 2024-09-13T11:30:22.304 DEBUG:teuthology.orchestra.run.smithi177:> sync && sudo umount -f /var/lib/ceph/osd/ceph-15 2024-09-13T11:30:22.479 INFO:tasks.ceph:Archiving mon data... 2024-09-13T11:30:22.480 DEBUG:teuthology.misc:Transferring archived files from smithi001:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/akupczyk-2024-09-13_09:37:56-rados-wip-aclamk-cbt-combined-distro-default-smithi/7903409/data/mon.a.tgz 2024-09-13T11:30:22.480 DEBUG:teuthology.orchestra.run.smithi001:> mktemp 2024-09-13T11:30:22.486 INFO:teuthology.orchestra.run.smithi001.stdout:/tmp/tmp.YVcet9dMBY 2024-09-13T11:30:22.486 DEBUG:teuthology.orchestra.run.smithi001:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.YVcet9dMBY 2024-09-13T11:30:22.660 DEBUG:teuthology.orchestra.run.smithi001:> sudo chmod 0666 /tmp/tmp.YVcet9dMBY 2024-09-13T11:30:22.724 DEBUG:teuthology.orchestra.remote:smithi001:/tmp/tmp.YVcet9dMBY is 559KB 2024-09-13T11:30:22.789 DEBUG:teuthology.orchestra.run.smithi001:> rm -fr /tmp/tmp.YVcet9dMBY 2024-09-13T11:30:22.795 DEBUG:teuthology.misc:Transferring archived files from smithi092:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/akupczyk-2024-09-13_09:37:56-rados-wip-aclamk-cbt-combined-distro-default-smithi/7903409/data/mon.b.tgz 2024-09-13T11:30:22.795 DEBUG:teuthology.orchestra.run.smithi092:> mktemp 2024-09-13T11:30:22.801 INFO:teuthology.orchestra.run.smithi092.stdout:/tmp/tmp.U9RwKd3IEi 2024-09-13T11:30:22.801 DEBUG:teuthology.orchestra.run.smithi092:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.U9RwKd3IEi 2024-09-13T11:30:22.963 DEBUG:teuthology.orchestra.run.smithi092:> sudo chmod 0666 /tmp/tmp.U9RwKd3IEi 2024-09-13T11:30:23.026 DEBUG:teuthology.orchestra.remote:smithi092:/tmp/tmp.U9RwKd3IEi is 560KB 2024-09-13T11:30:23.099 DEBUG:teuthology.orchestra.run.smithi092:> rm -fr /tmp/tmp.U9RwKd3IEi 2024-09-13T11:30:23.104 DEBUG:teuthology.misc:Transferring archived files from smithi162:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/akupczyk-2024-09-13_09:37:56-rados-wip-aclamk-cbt-combined-distro-default-smithi/7903409/data/mon.c.tgz 2024-09-13T11:30:23.104 DEBUG:teuthology.orchestra.run.smithi162:> mktemp 2024-09-13T11:30:23.110 INFO:teuthology.orchestra.run.smithi162.stdout:/tmp/tmp.zozKIRLFQE 2024-09-13T11:30:23.110 DEBUG:teuthology.orchestra.run.smithi162:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.zozKIRLFQE 2024-09-13T11:30:23.274 DEBUG:teuthology.orchestra.run.smithi162:> sudo chmod 0666 /tmp/tmp.zozKIRLFQE 2024-09-13T11:30:23.338 DEBUG:teuthology.orchestra.remote:smithi162:/tmp/tmp.zozKIRLFQE is 560KB 2024-09-13T11:30:23.400 DEBUG:teuthology.orchestra.run.smithi162:> rm -fr /tmp/tmp.zozKIRLFQE 2024-09-13T11:30:23.405 INFO:tasks.ceph:Cleaning ceph cluster... 2024-09-13T11:30:23.406 DEBUG:teuthology.orchestra.run.smithi001:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-09-13T11:30:23.408 DEBUG:teuthology.orchestra.run.smithi092:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-09-13T11:30:23.410 DEBUG:teuthology.orchestra.run.smithi162:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-09-13T11:30:23.448 DEBUG:teuthology.orchestra.run.smithi177:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-09-13T11:30:23.470 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-09-13T11:30:23.476 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-09-13T11:30:23.506 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-09-13T11:30:23.512 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-09-13T11:30:23.512 INFO:tasks.ceph:Archiving crash dumps... 2024-09-13T11:30:23.513 DEBUG:teuthology.misc:Transferring archived files from smithi001:/var/lib/ceph/crash to /home/teuthworker/archive/akupczyk-2024-09-13_09:37:56-rados-wip-aclamk-cbt-combined-distro-default-smithi/7903409/remote/smithi001/crash 2024-09-13T11:30:23.514 DEBUG:teuthology.orchestra.run.smithi001:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-09-13T11:30:23.528 DEBUG:teuthology.misc:Transferring archived files from smithi092:/var/lib/ceph/crash to /home/teuthworker/archive/akupczyk-2024-09-13_09:37:56-rados-wip-aclamk-cbt-combined-distro-default-smithi/7903409/remote/smithi092/crash 2024-09-13T11:30:23.529 DEBUG:teuthology.orchestra.run.smithi092:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-09-13T11:30:23.542 DEBUG:teuthology.misc:Transferring archived files from smithi162:/var/lib/ceph/crash to /home/teuthworker/archive/akupczyk-2024-09-13_09:37:56-rados-wip-aclamk-cbt-combined-distro-default-smithi/7903409/remote/smithi162/crash 2024-09-13T11:30:23.543 DEBUG:teuthology.orchestra.run.smithi162:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-09-13T11:30:23.563 DEBUG:teuthology.misc:Transferring archived files from smithi177:/var/lib/ceph/crash to /home/teuthworker/archive/akupczyk-2024-09-13_09:37:56-rados-wip-aclamk-cbt-combined-distro-default-smithi/7903409/remote/smithi177/crash 2024-09-13T11:30:23.563 DEBUG:teuthology.orchestra.run.smithi177:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-09-13T11:30:23.576 INFO:tasks.ceph:Compressing logs... 2024-09-13T11:30:23.576 DEBUG:teuthology.orchestra.run.smithi001:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-13T11:30:23.579 DEBUG:teuthology.orchestra.run.smithi092:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-13T11:30:23.590 DEBUG:teuthology.orchestra.run.smithi162:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-13T11:30:23.597 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47905.log 2024-09-13T11:30:23.597 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43887.log 2024-09-13T11:30:23.598 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.47905.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47905.log.gz 2024-09-13T11:30:23.598 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44537.log 2024-09-13T11:30:23.600 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40924.log 2024-09-13T11:30:23.600 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.43887.log: /var/log/ceph/ceph-client.admin.44537.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45602.log 2024-09-13T11:30:23.600 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44537.log.gz 2024-09-13T11:30:23.600 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43887.log.gz 2024-09-13T11:30:23.600 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.40924.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36674.log 2024-09-13T11:30:23.600 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40924.log.gz 2024-09-13T11:30:23.600 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.45602.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46239.log 2024-09-13T11:30:23.600 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45602.log.gz 2024-09-13T11:30:23.601 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.36674.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43502.log 2024-09-13T11:30:23.601 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36674.log.gz 2024-09-13T11:30:23.605 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37268.log 2024-09-13T11:30:23.606 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.46239.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46239.log.gz 2024-09-13T11:30:23.606 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.43502.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40499.log 2024-09-13T11:30:23.606 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43502.log.gz 2024-09-13T11:30:23.606 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.37268.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37324.log 2024-09-13T11:30:23.606 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37268.log.gz 2024-09-13T11:30:23.606 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.40499.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31489.log 2024-09-13T11:30:23.606 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40499.log.gz 2024-09-13T11:30:23.606 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38097.log 2024-09-13T11:30:23.606 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.37324.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37324.log.gz 2024-09-13T11:30:23.606 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.31489.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34056.log 2024-09-13T11:30:23.606 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31489.log.gz 2024-09-13T11:30:23.606 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.38097.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38269.log 2024-09-13T11:30:23.606 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38097.log.gz 2024-09-13T11:30:23.607 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41737.log 2024-09-13T11:30:23.607 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.34056.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34056.log.gz 2024-09-13T11:30:23.607 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.38269.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38269.log.gz 2024-09-13T11:30:23.607 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46527.log 2024-09-13T11:30:23.607 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.41737.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31056.log 2024-09-13T11:30:23.607 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41737.log.gz 2024-09-13T11:30:23.607 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.46527.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.y.log 2024-09-13T11:30:23.607 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46527.log.gz 2024-09-13T11:30:23.607 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.31056.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31056.log.gz 2024-09-13T11:30:23.607 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40681.log 2024-09-13T11:30:23.607 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45164.log 2024-09-13T11:30:23.607 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46473.log 2024-09-13T11:30:23.607 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.40681.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40681.log.gz 2024-09-13T11:30:23.607 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.45164.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48373.log 2024-09-13T11:30:23.607 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45164.log.gz 2024-09-13T11:30:23.614 DEBUG:teuthology.orchestra.run.smithi177:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-13T11:30:23.626 INFO:teuthology.orchestra.run.smithi092.stderr:gzip -5 --verbose -- /var/log/ceph/objectstore_tool.31755.log 2024-09-13T11:30:23.626 INFO:teuthology.orchestra.run.smithi092.stderr:gzip -5 --verbose -- /var/log/ceph/objectstore_tool.32927.log 2024-09-13T11:30:23.626 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/objectstore_tool.31755.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2024-09-13T11:30:23.626 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/objectstore_tool.32927.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-09-13T11:30:23.626 INFO:teuthology.orchestra.run.smithi092.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-09-13T11:30:23.627 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/ceph-osd.5.log: /var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-09-13T11:30:23.627 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.9.log 2024-09-13T11:30:23.627 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/objectstore_tool.31951.log 2024-09-13T11:30:23.627 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/ceph-osd.9.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.13.log 2024-09-13T11:30:23.627 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/objectstore_tool.31951.log: 89.2% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-09-13T11:30:23.627 INFO:teuthology.orchestra.run.smithi092.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2024-09-13T11:30:23.627 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/ceph-osd.13.log: 94.2% -- replaced with /var/log/ceph/objectstore_tool.31755.log.gz 2024-09-13T11:30:23.627 INFO:teuthology.orchestra.run.smithi092.stderr:gzip -5 --verbose -- /var/log/ceph/objectstore_tool.32147.log 2024-09-13T11:30:23.627 INFO:teuthology.orchestra.run.smithi092.stderr: 86.8% -- replaced with /var/log/ceph/ceph.log.gz 2024-09-13T11:30:23.631 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/ceph-mon.b.log: /var/log/ceph/objectstore_tool.32147.log: 94.2% -- replaced with /var/log/ceph/objectstore_tool.31951.log.gz 2024-09-13T11:30:23.631 INFO:teuthology.orchestra.run.smithi092.stderr: 93.2% -- replaced with /var/log/ceph/objectstore_tool.32927.log.gz 2024-09-13T11:30:23.637 INFO:teuthology.orchestra.run.smithi162.stderr:gzip -5 --verbose -- /var/log/ceph/objectstore_tool.30307.log 2024-09-13T11:30:23.638 INFO:teuthology.orchestra.run.smithi162.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-09-13T11:30:23.638 INFO:teuthology.orchestra.run.smithi162.stderr:/var/log/ceph/objectstore_tool.30307.log: gzip -5 --verbose -- /var/log/ceph/bluestore_tool.36614.log 2024-09-13T11:30:23.638 INFO:teuthology.orchestra.run.smithi162.stderr:/var/log/ceph/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.6.log 2024-09-13T11:30:23.638 INFO:teuthology.orchestra.run.smithi162.stderr:/var/log/ceph/bluestore_tool.36614.log: gzip -5 --verbose -- /var/log/ceph/objectstore_tool.30725.log 2024-09-13T11:30:23.638 INFO:teuthology.orchestra.run.smithi162.stderr:gzip -5 --verbose -- /var/log/ceph/objectstore_tool.34691.log 2024-09-13T11:30:23.638 INFO:teuthology.orchestra.run.smithi162.stderr:/var/log/ceph/ceph-osd.6.log: /var/log/ceph/objectstore_tool.30725.log: gzip -5 --verbose -- /var/log/ceph/objectstore_tool.35461.log 2024-09-13T11:30:23.638 INFO:teuthology.orchestra.run.smithi162.stderr:/var/log/ceph/objectstore_tool.34691.log: gzip -5 --verbose -- /var/log/ceph/bluestore_tool.32646.log 2024-09-13T11:30:23.638 INFO:teuthology.orchestra.run.smithi162.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.14.log 2024-09-13T11:30:23.647 INFO:teuthology.orchestra.run.smithi162.stderr:/var/log/ceph/objectstore_tool.35461.log: /var/log/ceph/bluestore_tool.32646.log: gzip -5 --verbose -- /var/log/ceph/objectstore_tool.30503.log 2024-09-13T11:30:23.660 INFO:teuthology.orchestra.run.smithi162.stderr:/var/log/ceph/ceph-osd.14.log: 95.6% -- replaced with /var/log/ceph/bluestore_tool.36614.log.gz 2024-09-13T11:30:23.662 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26499.log 2024-09-13T11:30:23.662 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26299.log 2024-09-13T11:30:23.662 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.26499.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26499.log.gz 2024-09-13T11:30:23.662 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26547.log 2024-09-13T11:30:23.662 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.26299.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26299.log.gz 2024-09-13T11:30:23.662 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.7.log 2024-09-13T11:30:23.662 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26323.log 2024-09-13T11:30:23.662 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.26547.log: /var/log/ceph/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26450.log 2024-09-13T11:30:23.662 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26547.log.gz 2024-09-13T11:30:23.662 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.26323.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26275.log 2024-09-13T11:30:23.662 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26323.log.gz 2024-09-13T11:30:23.662 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.26450.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26450.log.gz 2024-09-13T11:30:23.663 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-09-13T11:30:23.663 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.11.log/var/log/ceph/ceph-client.admin.26275.log: 2024-09-13T11:30:23.663 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26275.log.gz 2024-09-13T11:30:23.663 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.15.log 2024-09-13T11:30:23.663 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-osd.11.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26251.log 2024-09-13T11:30:23.663 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-osd.15.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-09-13T11:30:23.663 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.26251.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26203.log 2024-09-13T11:30:23.663 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26251.log.gz 2024-09-13T11:30:23.663 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26523.log 2024-09-13T11:30:23.663 INFO:teuthology.orchestra.run.smithi177.stderr:gzip/var/log/ceph/ceph-client.admin.26203.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.26378.log 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26203.log.gz 2024-09-13T11:30:23.663 INFO:teuthology.orchestra.run.smithi177.stderr: 2024-09-13T11:30:23.663 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.26523.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26523.log.gz 2024-09-13T11:30:23.663 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26347.log 2024-09-13T11:30:23.663 INFO:teuthology.orchestra.run.smithi177.stderr:gzip/var/log/ceph/ceph-client.admin.26378.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.26227.log 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26378.log.gz 2024-09-13T11:30:23.664 INFO:teuthology.orchestra.run.smithi177.stderr: 2024-09-13T11:30:23.664 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.26347.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26347.log.gz 2024-09-13T11:30:23.664 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.31116.log 2024-09-13T11:30:23.664 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.26227.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26426.log 2024-09-13T11:30:23.664 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0%/var/log/ceph/ceph-client.0.31116.log: -- replaced with /var/log/ceph/ceph-client.admin.26227.log.gz 2024-09-13T11:30:23.664 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26474.log 2024-09-13T11:30:23.664 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.26426.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26426.log.gz 2024-09-13T11:30:23.664 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26402.log 2024-09-13T11:30:23.664 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26179.log 2024-09-13T11:30:23.664 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.26474.log: 95.0% -- replaced with /var/log/ceph/ceph-client.0.31116.log.gz 2024-09-13T11:30:23.664 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26474.log.gz 2024-09-13T11:30:23.664 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.26402.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26402.log.gz 2024-09-13T11:30:23.665 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.26179.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26179.log.gz 2024-09-13T11:30:23.667 INFO:teuthology.orchestra.run.smithi092.stderr: 94.1% -- replaced with /var/log/ceph/objectstore_tool.32147.log.gz 2024-09-13T11:30:23.667 INFO:teuthology.orchestra.run.smithi162.stderr:gzip -5 --verbose -- /var/log/ceph/bluestore_tool.36229.log 2024-09-13T11:30:23.681 INFO:teuthology.orchestra.run.smithi162.stderr:/var/log/ceph/objectstore_tool.30503.log: 95.0% -- replaced with /var/log/ceph/objectstore_tool.30307.log.gz 2024-09-13T11:30:23.688 INFO:teuthology.orchestra.run.smithi162.stderr:gzip -5 --verbose -- /var/log/ceph/bluestore_tool.32256.log 2024-09-13T11:30:23.695 INFO:teuthology.orchestra.run.smithi162.stderr:/var/log/ceph/bluestore_tool.36229.log: 94.1% -- replaced with /var/log/ceph/bluestore_tool.36229.log.gz 2024-09-13T11:30:23.695 INFO:teuthology.orchestra.run.smithi162.stderr: 95.3% -- replaced with /var/log/ceph/bluestore_tool.32646.log.gz 2024-09-13T11:30:23.696 INFO:teuthology.orchestra.run.smithi162.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-09-13T11:30:23.702 INFO:teuthology.orchestra.run.smithi162.stderr:/var/log/ceph/bluestore_tool.32256.log: 85.8% -- replaced with /var/log/ceph/bluestore_tool.32256.log.gz 2024-09-13T11:30:23.702 INFO:teuthology.orchestra.run.smithi162.stderr:gzip -5 --verbose -- /var/log/ceph/bluestore_tool.36224.log 2024-09-13T11:30:23.710 INFO:teuthology.orchestra.run.smithi162.stderr:/var/log/ceph/ceph.log: 86.8% -- replaced with /var/log/ceph/ceph.log.gz 2024-09-13T11:30:23.710 INFO:teuthology.orchestra.run.smithi162.stderr:gzip -5 --verbose -- /var/log/ceph/bluestore_tool.36987.log 2024-09-13T11:30:23.710 INFO:teuthology.orchestra.run.smithi162.stderr:/var/log/ceph/bluestore_tool.36224.log: 85.9% -- replaced with /var/log/ceph/bluestore_tool.36224.log.gz 2024-09-13T11:30:23.710 INFO:teuthology.orchestra.run.smithi162.stderr:gzip -5 --verbose -- /var/log/ceph/objectstore_tool.34294.log 2024-09-13T11:30:23.714 INFO:teuthology.orchestra.run.smithi162.stderr:/var/log/ceph/bluestore_tool.36987.log: 95.0% -- replaced with /var/log/ceph/objectstore_tool.35461.log.gz 2024-09-13T11:30:23.721 INFO:teuthology.orchestra.run.smithi162.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2024-09-13T11:30:23.733 INFO:teuthology.orchestra.run.smithi162.stderr:/var/log/ceph/objectstore_tool.34294.log: gzip -5 --verbose -- /var/log/ceph/objectstore_tool.31479.log 2024-09-13T11:30:23.733 INFO:teuthology.orchestra.run.smithi162.stderr: 95.6% -- replaced with /var/log/ceph/bluestore_tool.36987.log.gz 2024-09-13T11:30:23.746 INFO:teuthology.orchestra.run.smithi162.stderr:/var/log/ceph/ceph-mon.c.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-09-13T11:30:23.768 INFO:teuthology.orchestra.run.smithi162.stderr:/var/log/ceph/objectstore_tool.31479.log: gzip -5 --verbose -- /var/log/ceph/bluestore_tool.33019.log 2024-09-13T11:30:23.768 INFO:teuthology.orchestra.run.smithi162.stderr: 95.2% -- replaced with /var/log/ceph/objectstore_tool.34294.log.gz 2024-09-13T11:30:23.768 INFO:teuthology.orchestra.run.smithi162.stderr:gzip -5 --verbose -- /var/log/ceph/bluestore_tool.32273.log 2024-09-13T11:30:23.768 INFO:teuthology.orchestra.run.smithi162.stderr:/var/log/ceph/ceph.audit.log: /var/log/ceph/bluestore_tool.33019.log: 89.2% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-09-13T11:30:23.779 INFO:teuthology.orchestra.run.smithi162.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.10.log 2024-09-13T11:30:23.798 INFO:teuthology.orchestra.run.smithi162.stderr:/var/log/ceph/bluestore_tool.32273.log: gzip -5 --verbose -- /var/log/ceph/objectstore_tool.34493.log 2024-09-13T11:30:23.802 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.46473.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40171.log 2024-09-13T11:30:23.802 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46473.log.gz 2024-09-13T11:30:23.802 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47076.log 2024-09-13T11:30:23.802 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.48373.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48373.log.gz 2024-09-13T11:30:23.802 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.40171.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40171.log.gz 2024-09-13T11:30:23.802 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34945.log 2024-09-13T11:30:23.802 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.47076.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48269.log 2024-09-13T11:30:23.802 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47076.log.gz 2024-09-13T11:30:23.802 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.34945.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35992.log 2024-09-13T11:30:23.803 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34945.log.gz 2024-09-13T11:30:23.803 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.48269.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35503.log 2024-09-13T11:30:23.803 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48269.log.gz 2024-09-13T11:30:23.803 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.35992.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35992.log.gz 2024-09-13T11:30:23.803 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33577.log 2024-09-13T11:30:23.803 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43069.log 2024-09-13T11:30:23.803 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.35503.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35503.log.gz 2024-09-13T11:30:23.803 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.33577.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35080.log 2024-09-13T11:30:23.803 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33577.log.gz 2024-09-13T11:30:23.803 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.43069.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42221.log 2024-09-13T11:30:23.803 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43069.log.gz 2024-09-13T11:30:23.803 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.35080.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40119.log 2024-09-13T11:30:23.803 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35080.log.gz 2024-09-13T11:30:23.804 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.42221.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48685.log 2024-09-13T11:30:23.804 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42221.log.gz 2024-09-13T11:30:23.804 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.40119.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46813.log 2024-09-13T11:30:23.804 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40119.log.gz 2024-09-13T11:30:23.804 INFO:teuthology.orchestra.run.smithi001.stderr:gzip/var/log/ceph/ceph-client.admin.48685.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.39209.log 2024-09-13T11:30:23.804 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48685.log.gz 2024-09-13T11:30:23.804 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41536.log 2024-09-13T11:30:23.804 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.46813.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46813.log.gz 2024-09-13T11:30:23.804 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.39209.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30646.log 2024-09-13T11:30:23.804 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39209.log.gz 2024-09-13T11:30:23.804 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.41536.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48139.log 2024-09-13T11:30:23.804 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41536.log.gz 2024-09-13T11:30:23.805 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39821.log 2024-09-13T11:30:23.805 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.30646.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30646.log.gz 2024-09-13T11:30:23.805 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.48139.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-09-13T11:30:23.805 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48139.log.gz 2024-09-13T11:30:23.805 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.39821.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32145.log 2024-09-13T11:30:23.805 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39821.log.gz 2024-09-13T11:30:23.805 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31037.log 2024-09-13T11:30:23.805 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-osd.0.log: /var/log/ceph/ceph-client.admin.32145.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32988.log 2024-09-13T11:30:23.805 INFO:teuthology.orchestra.run.smithi001.stderr: 93.6% -- replaced with /var/log/ceph/ceph-mgr.y.log.gz 2024-09-13T11:30:23.805 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32145.log.gz 2024-09-13T11:30:23.805 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.31037.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48659.log 2024-09-13T11:30:23.805 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31037.log.gz 2024-09-13T11:30:23.805 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42818.log 2024-09-13T11:30:23.806 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.32988.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32988.log.gz 2024-09-13T11:30:23.806 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.48659.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32907.log 2024-09-13T11:30:23.806 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48659.log.gz 2024-09-13T11:30:23.806 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.42818.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47931.log 2024-09-13T11:30:23.806 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42818.log.gz 2024-09-13T11:30:23.806 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36841.log 2024-09-13T11:30:23.806 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.32907.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32907.log.gz 2024-09-13T11:30:23.806 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.47931.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30624.log 2024-09-13T11:30:23.806 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47931.log.gz 2024-09-13T11:30:23.806 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.36841.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46709.log 2024-09-13T11:30:23.806 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36841.log.gz 2024-09-13T11:30:23.806 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46553.log 2024-09-13T11:30:23.806 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.30624.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30624.log.gz 2024-09-13T11:30:23.806 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.46709.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46631.log 2024-09-13T11:30:23.807 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46709.log.gz 2024-09-13T11:30:23.807 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.46553.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30659.log 2024-09-13T11:30:23.807 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46553.log.gz 2024-09-13T11:30:23.807 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34972.log 2024-09-13T11:30:23.807 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.46631.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46631.log.gz 2024-09-13T11:30:23.807 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.30659.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47492.log 2024-09-13T11:30:23.807 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30659.log.gz 2024-09-13T11:30:23.807 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.34972.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40878.log 2024-09-13T11:30:23.807 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34972.log.gz 2024-09-13T11:30:23.807 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.47492.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37168.log 2024-09-13T11:30:23.807 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47492.log.gz 2024-09-13T11:30:23.807 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.40878.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34565.log 2024-09-13T11:30:23.807 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40878.log.gz 2024-09-13T11:30:23.807 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.37168.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40781.log 2024-09-13T11:30:23.807 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37168.log.gz 2024-09-13T11:30:23.808 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.34565.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47464.log 2024-09-13T11:30:23.808 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34565.log.gz 2024-09-13T11:30:23.808 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.40781.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37669.log 2024-09-13T11:30:23.808 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40781.log.gz 2024-09-13T11:30:23.808 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.47464.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47464.log.gz 2024-09-13T11:30:23.808 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46683.log 2024-09-13T11:30:23.808 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.37669.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36815.log 2024-09-13T11:30:23.808 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37669.log.gz 2024-09-13T11:30:23.808 INFO:teuthology.orchestra.run.smithi001.stderr:gzip/var/log/ceph/ceph-client.admin.46683.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.34540.log 2024-09-13T11:30:23.808 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46683.log.gz 2024-09-13T11:30:23.808 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.36815.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36815.log.gz 2024-09-13T11:30:23.808 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39994.log 2024-09-13T11:30:23.809 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34316.log 2024-09-13T11:30:23.809 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.34540.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34540.log.gz 2024-09-13T11:30:23.809 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.39994.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39994.log.gz 2024-09-13T11:30:23.809 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33492.log 2024-09-13T11:30:23.809 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.34316.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45981.log 2024-09-13T11:30:23.809 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34316.log.gz 2024-09-13T11:30:23.809 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45492.log 2024-09-13T11:30:23.809 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.33492.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33492.log.gz 2024-09-13T11:30:23.809 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.45981.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45981.log.gz 2024-09-13T11:30:23.809 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39329.log 2024-09-13T11:30:23.809 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39769.log 2024-09-13T11:30:23.809 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.45492.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45492.log.gz 2024-09-13T11:30:23.809 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.39329.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39329.log.gz 2024-09-13T11:30:23.810 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31495.log 2024-09-13T11:30:23.810 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.39769.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31903.log 2024-09-13T11:30:23.810 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39769.log.gz 2024-09-13T11:30:23.810 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41043.log 2024-09-13T11:30:23.810 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.31495.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31495.log.gz 2024-09-13T11:30:23.810 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.31903.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32041.log 2024-09-13T11:30:23.810 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31903.log.gz 2024-09-13T11:30:23.810 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39687.log 2024-09-13T11:30:23.810 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.41043.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41043.log.gz 2024-09-13T11:30:23.810 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.32041.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32041.log.gz 2024-09-13T11:30:23.810 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40022.log 2024-09-13T11:30:23.810 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47983.log 2024-09-13T11:30:23.810 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.39687.log: /var/log/ceph/ceph-client.admin.40022.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39687.log.gz 2024-09-13T11:30:23.810 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40022.log.gz 2024-09-13T11:30:23.811 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33237.log 2024-09-13T11:30:23.811 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30650.log 2024-09-13T11:30:23.811 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.47983.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47983.log.gz 2024-09-13T11:30:23.811 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.33237.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33237.log.gz 2024-09-13T11:30:23.811 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48451.log 2024-09-13T11:30:23.811 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.30650.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30650.log.gz 2024-09-13T11:30:23.811 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48790.log 2024-09-13T11:30:23.811 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.48451.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45130.log 2024-09-13T11:30:23.811 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48451.log.gz 2024-09-13T11:30:23.811 INFO:teuthology.orchestra.run.smithi001.stderr:gzip/var/log/ceph/ceph-client.admin.48790.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.46265.log 2024-09-13T11:30:23.811 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48790.log.gz 2024-09-13T11:30:23.811 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.45130.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48087.log 2024-09-13T11:30:23.811 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45130.log.gz 2024-09-13T11:30:23.811 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.46265.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34789.log 2024-09-13T11:30:23.812 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46265.log.gz 2024-09-13T11:30:23.812 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.48087.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43367.log 2024-09-13T11:30:23.812 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48087.log.gz 2024-09-13T11:30:23.812 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.34789.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41207.log 2024-09-13T11:30:23.812 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34789.log.gz 2024-09-13T11:30:23.812 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.43367.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46104.log 2024-09-13T11:30:23.812 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43367.log.gz 2024-09-13T11:30:23.812 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.41207.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33611.log 2024-09-13T11:30:23.812 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41207.log.gz 2024-09-13T11:30:23.812 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.46104.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46104.log.gz 2024-09-13T11:30:23.812 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35605.log 2024-09-13T11:30:23.812 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.33611.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45903.log 2024-09-13T11:30:23.812 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33611.log.gz 2024-09-13T11:30:23.812 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.35605.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39185.log 2024-09-13T11:30:23.812 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35605.log.gz 2024-09-13T11:30:23.813 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.45903.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48009.log 2024-09-13T11:30:23.813 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45903.log.gz 2024-09-13T11:30:23.813 INFO:teuthology.orchestra.run.smithi001.stderr:gzip/var/log/ceph/ceph-client.admin.39185.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.40678.log 2024-09-13T11:30:23.813 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39185.log.gz 2024-09-13T11:30:23.813 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.48009.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48009.log.gz 2024-09-13T11:30:23.813 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42083.log 2024-09-13T11:30:23.813 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36063.log 2024-09-13T11:30:23.813 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.40678.log: /var/log/ceph/ceph-client.admin.42083.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40678.log.gz 2024-09-13T11:30:23.813 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42083.log.gz 2024-09-13T11:30:23.813 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39281.log 2024-09-13T11:30:23.813 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48113.log 2024-09-13T11:30:23.813 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.36063.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36063.log.gz 2024-09-13T11:30:23.813 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.39281.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39281.log.gz 2024-09-13T11:30:23.813 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32566.log 2024-09-13T11:30:23.814 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33610.log 2024-09-13T11:30:23.814 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.48113.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48113.log.gz 2024-09-13T11:30:23.814 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.32566.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32566.log.gz 2024-09-13T11:30:23.814 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46421.log 2024-09-13T11:30:23.814 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42536.log 2024-09-13T11:30:23.814 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.33610.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33610.log.gz 2024-09-13T11:30:23.814 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.46421.log: gzip 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46421.log.gz 2024-09-13T11:30:23.814 INFO:teuthology.orchestra.run.smithi001.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.36947.log 2024-09-13T11:30:23.814 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31061.log 2024-09-13T11:30:23.814 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.42536.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42536.log.gz 2024-09-13T11:30:23.814 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.36947.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36947.log.gz 2024-09-13T11:30:23.814 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39922.log 2024-09-13T11:30:23.814 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44984.log 2024-09-13T11:30:23.814 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.31061.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31061.log.gz 2024-09-13T11:30:23.815 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.39922.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39922.log.gz 2024-09-13T11:30:23.815 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46343.log 2024-09-13T11:30:23.815 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48295.log 2024-09-13T11:30:23.815 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.44984.log: 0.0%/var/log/ceph/ceph-client.admin.46343.log: -- replaced with /var/log/ceph/ceph-client.admin.44984.log.gz 2024-09-13T11:30:23.815 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46343.log.gz 2024-09-13T11:30:23.815 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46447.log 2024-09-13T11:30:23.815 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30652.log 2024-09-13T11:30:23.815 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.48295.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48295.log.gz 2024-09-13T11:30:23.815 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.46447.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46447.log.gz 2024-09-13T11:30:23.815 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45735.log 2024-09-13T11:30:23.815 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.8.log 2024-09-13T11:30:23.815 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.30652.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30652.log.gz 2024-09-13T11:30:23.815 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.45735.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45735.log.gz 2024-09-13T11:30:23.815 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48165.log 2024-09-13T11:30:23.815 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31024.log 2024-09-13T11:30:23.816 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-osd.8.log: /var/log/ceph/ceph-client.admin.48165.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48165.log.gzgzip 2024-09-13T11:30:23.816 INFO:teuthology.orchestra.run.smithi001.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.47445.log 2024-09-13T11:30:23.816 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34431.log 2024-09-13T11:30:23.816 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.31024.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31024.log.gz 2024-09-13T11:30:23.816 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.47445.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47445.log.gz 2024-09-13T11:30:23.816 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46761.log 2024-09-13T11:30:23.816 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.34431.log: gzip 0.0% -5 -- replaced with /var/log/ceph/ceph-client.admin.34431.log.gz 2024-09-13T11:30:23.816 INFO:teuthology.orchestra.run.smithi001.stderr: --verbose -- /var/log/ceph/ceph-client.admin.34113.log 2024-09-13T11:30:23.816 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33034.log 2024-09-13T11:30:23.816 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.46761.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46761.log.gz 2024-09-13T11:30:23.816 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.34113.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34113.log.gz 2024-09-13T11:30:23.816 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47045.log 2024-09-13T11:30:23.816 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39479.log 2024-09-13T11:30:23.817 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.33034.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33034.log.gz 2024-09-13T11:30:23.817 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.47045.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47045.log.gz 2024-09-13T11:30:23.817 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30521.log 2024-09-13T11:30:23.817 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.39479.log: /var/log/ceph/ceph-client.admin.34899.log 2024-09-13T11:30:23.817 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39479.log.gz 2024-09-13T11:30:23.817 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.30521.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30521.log.gz 2024-09-13T11:30:23.817 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35164.log 2024-09-13T11:30:23.817 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37694.log 2024-09-13T11:30:23.817 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.34899.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34899.log.gz 2024-09-13T11:30:23.817 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.35164.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35164.log.gz 2024-09-13T11:30:23.817 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46943.log 2024-09-13T11:30:23.817 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47484.log 2024-09-13T11:30:23.817 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.37694.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37694.log.gz 2024-09-13T11:30:23.817 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.46943.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46943.log.gz 2024-09-13T11:30:23.818 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39357.log 2024-09-13T11:30:23.818 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41962.log 2024-09-13T11:30:23.818 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.47484.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47484.log.gz 2024-09-13T11:30:23.818 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.39357.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39357.log.gz 2024-09-13T11:30:23.818 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47062.log 2024-09-13T11:30:23.818 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39859.log 2024-09-13T11:30:23.818 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.41962.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41962.log.gz 2024-09-13T11:30:23.818 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.47062.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47062.log.gz 2024-09-13T11:30:23.818 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42320.log 2024-09-13T11:30:23.818 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38016.log 2024-09-13T11:30:23.818 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.39859.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39859.log.gz 2024-09-13T11:30:23.818 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.42320.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42320.log.gz 2024-09-13T11:30:23.818 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34837.log 2024-09-13T11:30:23.819 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.38016.log: /var/log/ceph/ceph-client.admin.35726.log 2024-09-13T11:30:23.819 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38016.log.gz 2024-09-13T11:30:23.819 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.34837.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34837.log.gz 2024-09-13T11:30:23.819 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44293.log 2024-09-13T11:30:23.819 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31717.log 2024-09-13T11:30:23.819 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.35726.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35726.log.gz 2024-09-13T11:30:23.819 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.44293.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44293.log.gz 2024-09-13T11:30:23.819 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46052.log 2024-09-13T11:30:23.819 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40754.log 2024-09-13T11:30:23.819 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.31717.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44092.log 2024-09-13T11:30:23.819 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.46052.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31717.log.gz 2024-09-13T11:30:23.819 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46052.log.gz 2024-09-13T11:30:23.819 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.40754.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40754.log.gz 2024-09-13T11:30:23.819 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45945.log 2024-09-13T11:30:23.820 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-09-13T11:30:23.820 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.44092.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44092.log.gz 2024-09-13T11:30:23.820 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.45945.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45945.log.gz 2024-09-13T11:30:23.820 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26323.log 2024-09-13T11:30:23.820 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46135.log 2024-09-13T11:30:23.820 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-mon.a.log: /var/log/ceph/ceph-client.admin.26323.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26323.log.gz 2024-09-13T11:30:23.820 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46026.log 2024-09-13T11:30:23.820 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43611.log 2024-09-13T11:30:23.820 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.46135.log: /var/log/ceph/ceph-client.admin.46026.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46026.log.gz 2024-09-13T11:30:23.820 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46135.log.gz 2024-09-13T11:30:23.820 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35907.log 2024-09-13T11:30:23.820 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39604.log 2024-09-13T11:30:23.820 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.43611.log: /var/log/ceph/ceph-client.admin.35907.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43611.log.gz 2024-09-13T11:30:23.821 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35907.log.gz 2024-09-13T11:30:23.821 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39918.log 2024-09-13T11:30:23.821 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43419.log 2024-09-13T11:30:23.821 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.39604.log: /var/log/ceph/ceph-client.admin.39918.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39918.log.gz 2024-09-13T11:30:23.821 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39604.log.gz 2024-09-13T11:30:23.821 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41976.log 2024-09-13T11:30:23.821 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39160.log 2024-09-13T11:30:23.821 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.43419.log: /var/log/ceph/ceph-client.admin.41976.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43419.log.gz 2024-09-13T11:30:23.821 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41976.log.gz 2024-09-13T11:30:23.821 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40326.log 2024-09-13T11:30:23.821 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31063.log 2024-09-13T11:30:23.821 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.39160.log: /var/log/ceph/ceph-client.admin.40326.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40326.log.gz 2024-09-13T11:30:23.821 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39160.log.gz 2024-09-13T11:30:23.821 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30630.log 2024-09-13T11:30:23.822 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34455.log 2024-09-13T11:30:23.822 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.31063.log: /var/log/ceph/ceph-client.admin.30630.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31063.log.gz 2024-09-13T11:30:23.822 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30630.log.gz 2024-09-13T11:30:23.822 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36656.log 2024-09-13T11:30:23.822 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37403.log 2024-09-13T11:30:23.822 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.34455.log: /var/log/ceph/ceph-client.admin.36656.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36656.log.gz 2024-09-13T11:30:23.822 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34455.log.gz 2024-09-13T11:30:23.822 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41125.log 2024-09-13T11:30:23.822 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48920.log 2024-09-13T11:30:23.822 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.37403.log: /var/log/ceph/ceph-client.admin.41125.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37403.log.gz 2024-09-13T11:30:23.822 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41125.log.gz 2024-09-13T11:30:23.822 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37429.log 2024-09-13T11:30:23.822 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36758.log 2024-09-13T11:30:23.823 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.48920.log: /var/log/ceph/ceph-client.admin.37429.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48920.log.gz 2024-09-13T11:30:23.823 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37429.log.gz 2024-09-13T11:30:23.823 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46995.log 2024-09-13T11:30:23.823 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48035.log 2024-09-13T11:30:23.823 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.36758.log: /var/log/ceph/ceph-client.admin.46995.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36758.log.gz 2024-09-13T11:30:23.823 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46995.log.gz 2024-09-13T11:30:23.823 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35402.log 2024-09-13T11:30:23.823 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33790.log 2024-09-13T11:30:23.823 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.48035.log: /var/log/ceph/ceph-client.admin.35402.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35402.log.gz 2024-09-13T11:30:23.823 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.48035.log.gz 2024-09-13T11:30:23.823 INFO:teuthology.orchestra.run.smithi001.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.31431.log 2024-09-13T11:30:23.823 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34191.log 2024-09-13T11:30:23.823 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.33790.log: /var/log/ceph/ceph-client.admin.31431.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33790.log.gz 2024-09-13T11:30:23.824 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31431.log.gz 2024-09-13T11:30:23.824 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33712.log 2024-09-13T11:30:23.824 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30658.log 2024-09-13T11:30:23.824 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.34191.log: /var/log/ceph/ceph-client.admin.33712.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34191.log.gz 2024-09-13T11:30:23.824 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33712.log.gz 2024-09-13T11:30:23.824 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42454.log 2024-09-13T11:30:23.824 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39486.log 2024-09-13T11:30:23.824 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.30658.log: /var/log/ceph/ceph-client.admin.42454.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30658.log.gz 2024-09-13T11:30:23.824 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42454.log.gz 2024-09-13T11:30:23.824 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46291.log 2024-09-13T11:30:23.824 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31463.log 2024-09-13T11:30:23.824 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.39486.log: /var/log/ceph/ceph-client.admin.46291.log: 0.0% 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46291.log.gz 2024-09-13T11:30:23.824 INFO:teuthology.orchestra.run.smithi001.stderr: -- replaced with /var/log/ceph/ceph-client.admin.39486.log.gz 2024-09-13T11:30:23.825 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42752.log 2024-09-13T11:30:23.825 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33148.log 2024-09-13T11:30:23.825 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.31463.log: /var/log/ceph/ceph-client.admin.42752.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31463.log.gz 2024-09-13T11:30:23.825 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42752.log.gz 2024-09-13T11:30:23.825 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40089.log 2024-09-13T11:30:23.825 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32299.log 2024-09-13T11:30:23.825 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.33148.log: /var/log/ceph/ceph-client.admin.40089.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40089.log.gz 2024-09-13T11:30:23.825 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33148.log.gz 2024-09-13T11:30:23.825 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38194.log 2024-09-13T11:30:23.825 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42865.log 2024-09-13T11:30:23.825 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.32299.log: /var/log/ceph/ceph-client.admin.38194.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38194.log.gz 2024-09-13T11:30:23.825 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32299.log.gz 2024-09-13T11:30:23.825 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39058.log 2024-09-13T11:30:23.826 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47067.log 2024-09-13T11:30:23.826 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.42865.log: /var/log/ceph/ceph-client.admin.39058.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42865.log.gz 0.0% 2024-09-13T11:30:23.826 INFO:teuthology.orchestra.run.smithi001.stderr: -- replaced with /var/log/ceph/ceph-client.admin.39058.log.gz 2024-09-13T11:30:23.826 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47077.log 2024-09-13T11:30:23.826 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36464.log 2024-09-13T11:30:23.826 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.47067.log: /var/log/ceph/ceph-client.admin.47077.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47067.log.gz 2024-09-13T11:30:23.826 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47077.log.gz 2024-09-13T11:30:23.826 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33764.log 2024-09-13T11:30:23.826 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41890.log 2024-09-13T11:30:23.826 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.36464.log: /var/log/ceph/ceph-client.admin.33764.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33764.log.gz 2024-09-13T11:30:23.826 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36464.log.gz 2024-09-13T11:30:23.826 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31522.log 2024-09-13T11:30:23.827 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37133.log 2024-09-13T11:30:23.827 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.41890.log: /var/log/ceph/ceph-client.admin.31522.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41890.log.gz 2024-09-13T11:30:23.827 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31522.log.gz 2024-09-13T11:30:23.827 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31947.log 2024-09-13T11:30:23.827 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.37133.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37133.log.gz 2024-09-13T11:30:23.827 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41914.log 2024-09-13T11:30:23.827 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.31947.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31947.log.gz 2024-09-13T11:30:23.827 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33124.log 2024-09-13T11:30:23.827 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38653.log 2024-09-13T11:30:23.827 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.41914.log: /var/log/ceph/ceph-client.admin.33124.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41914.log.gz 2024-09-13T11:30:23.827 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33124.log.gz 2024-09-13T11:30:23.827 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35352.log 2024-09-13T11:30:23.827 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46213.log 2024-09-13T11:30:23.828 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.38653.log: /var/log/ceph/ceph-client.admin.35352.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38653.log.gz 2024-09-13T11:30:23.828 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35352.log.gz 2024-09-13T11:30:23.828 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36227.log 2024-09-13T11:30:23.828 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41824.log 2024-09-13T11:30:23.828 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.46213.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46213.log.gz 2024-09-13T11:30:23.828 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.36227.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36227.log.gz 2024-09-13T11:30:23.828 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32647.log 2024-09-13T11:30:23.828 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35857.log 2024-09-13T11:30:23.828 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.41824.log: /var/log/ceph/ceph-client.admin.32647.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41824.log.gz 2024-09-13T11:30:23.828 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32647.log.gz 2024-09-13T11:30:23.828 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45626.log 2024-09-13T11:30:23.828 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36724.log 2024-09-13T11:30:23.828 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.35857.log: /var/log/ceph/ceph-client.admin.45626.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35857.log.gz 2024-09-13T11:30:23.828 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45626.log.gz 2024-09-13T11:30:23.829 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43476.log 2024-09-13T11:30:23.829 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31490.log 2024-09-13T11:30:23.829 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.36724.log: /var/log/ceph/ceph-client.admin.43476.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36724.log.gz 2024-09-13T11:30:23.829 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43476.log.gz 2024-09-13T11:30:23.829 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34894.log 2024-09-13T11:30:23.829 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39548.log 2024-09-13T11:30:23.829 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.31490.log: /var/log/ceph/ceph-client.admin.34894.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31490.log.gz 2024-09-13T11:30:23.829 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34894.log.gz 2024-09-13T11:30:23.829 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46161.log 2024-09-13T11:30:23.829 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32706.log 2024-09-13T11:30:23.829 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.39548.log: /var/log/ceph/ceph-client.admin.46161.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39548.log.gz 2024-09-13T11:30:23.829 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46161.log.gz 2024-09-13T11:30:23.830 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35557.log 2024-09-13T11:30:23.830 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31048.log 2024-09-13T11:30:23.830 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.32706.log: /var/log/ceph/ceph-client.admin.35557.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32706.log.gz 2024-09-13T11:30:23.830 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35557.log.gz 2024-09-13T11:30:23.830 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41741.log 2024-09-13T11:30:23.830 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45657.log 2024-09-13T11:30:23.830 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.31048.log: /var/log/ceph/ceph-client.admin.41741.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31048.log.gz 2024-09-13T11:30:23.830 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41741.log.gz 2024-09-13T11:30:23.830 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42584.log 2024-09-13T11:30:23.830 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46787.log 2024-09-13T11:30:23.830 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.45657.log: /var/log/ceph/ceph-client.admin.42584.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45657.log.gz 2024-09-13T11:30:23.830 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42584.log.gz 2024-09-13T11:30:23.830 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30547.log 2024-09-13T11:30:23.830 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43924.log 2024-09-13T11:30:23.831 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.46787.log: /var/log/ceph/ceph-client.admin.30547.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46787.log.gz 2024-09-13T11:30:23.831 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30547.log.gz 2024-09-13T11:30:23.831 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37075.log 2024-09-13T11:30:23.831 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45560.log 2024-09-13T11:30:23.831 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.43924.log: /var/log/ceph/ceph-client.admin.37075.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43924.log.gz 2024-09-13T11:30:23.831 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37075.log.gz 2024-09-13T11:30:23.831 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38556.log 2024-09-13T11:30:23.831 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46969.log 2024-09-13T11:30:23.831 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.45560.log: /var/log/ceph/ceph-client.admin.38556.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45560.log.gz 2024-09-13T11:30:23.831 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38556.log.gz 2024-09-13T11:30:23.831 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39661.log 2024-09-13T11:30:23.831 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.46969.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42261.log 2024-09-13T11:30:23.831 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46969.log.gz 2024-09-13T11:30:23.831 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.39661.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39661.log.gz 2024-09-13T11:30:23.832 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48868.log 2024-09-13T11:30:23.832 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35269.log 2024-09-13T11:30:23.832 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.42261.log: /var/log/ceph/ceph-client.admin.48868.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42261.log.gz 2024-09-13T11:30:23.832 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48868.log.gz 2024-09-13T11:30:23.832 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41263.log 2024-09-13T11:30:23.832 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32445.log 2024-09-13T11:30:23.832 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.35269.log: 0.0%/var/log/ceph/ceph-client.admin.41263.log: -- replaced with /var/log/ceph/ceph-client.admin.35269.log.gz 2024-09-13T11:30:23.832 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41263.log.gz 2024-09-13T11:30:23.832 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36030.log 2024-09-13T11:30:23.832 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36362.log 2024-09-13T11:30:23.832 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.32445.log: /var/log/ceph/ceph-client.admin.36030.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32445.log.gz 2024-09-13T11:30:23.833 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36030.log.gz 2024-09-13T11:30:23.833 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43337.log 2024-09-13T11:30:23.833 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41329.log 2024-09-13T11:30:23.833 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.36362.log: /var/log/ceph/ceph-client.admin.43337.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36362.log.gz 2024-09-13T11:30:23.833 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43337.log.gz 2024-09-13T11:30:23.833 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32791.log 2024-09-13T11:30:23.833 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.41329.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44820.log 2024-09-13T11:30:23.833 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41329.log.gz 2024-09-13T11:30:23.833 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.32791.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32791.log.gz 2024-09-13T11:30:23.833 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47957.log 2024-09-13T11:30:23.833 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46579.log 2024-09-13T11:30:23.833 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.44820.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44820.log.gz 2024-09-13T11:30:23.834 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.47957.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47957.log.gz 2024-09-13T11:30:23.834 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35883.log 2024-09-13T11:30:23.834 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43733.log 2024-09-13T11:30:23.834 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.46579.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46579.log.gz 2024-09-13T11:30:23.834 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.35883.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47853.log 2024-09-13T11:30:23.834 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35883.log.gz 2024-09-13T11:30:23.834 INFO:teuthology.orchestra.run.smithi001.stderr:gzip/var/log/ceph/ceph-client.admin.43733.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.48347.log 2024-09-13T11:30:23.834 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43733.log.gz 2024-09-13T11:30:23.834 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.47853.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36201.log 2024-09-13T11:30:23.834 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47853.log.gz 2024-09-13T11:30:23.834 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.48347.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48347.log.gz 2024-09-13T11:30:23.834 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40586.log 2024-09-13T11:30:23.834 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34754.log 2024-09-13T11:30:23.834 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.36201.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36201.log.gz 2024-09-13T11:30:23.835 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.40586.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36620.log 2024-09-13T11:30:23.835 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40586.log.gz 2024-09-13T11:30:23.835 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.34754.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34754.log.gz 2024-09-13T11:30:23.835 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33541.log 2024-09-13T11:30:23.835 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41580.log 2024-09-13T11:30:23.835 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.36620.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36620.log.gz 2024-09-13T11:30:23.835 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.33541.log: gzip -5 --verbose -- 0.0% /var/log/ceph/ceph-client.admin.35029.log 2024-09-13T11:30:23.835 INFO:teuthology.orchestra.run.smithi001.stderr: -- replaced with /var/log/ceph/ceph-client.admin.33541.log.gz 2024-09-13T11:30:23.835 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.41580.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41580.log.gz 2024-09-13T11:30:23.835 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40300.log 2024-09-13T11:30:23.835 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37506.log 2024-09-13T11:30:23.835 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.35029.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35029.log.gz 2024-09-13T11:30:23.835 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.40300.log: gzip 0.0% -5 -- replaced with /var/log/ceph/ceph-client.admin.40300.log.gz --verbose 2024-09-13T11:30:23.835 INFO:teuthology.orchestra.run.smithi001.stderr: -- /var/log/ceph/ceph-client.admin.41506.log 2024-09-13T11:30:23.835 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.37506.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37506.log.gz 2024-09-13T11:30:23.836 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33442.log 2024-09-13T11:30:23.836 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48503.log 2024-09-13T11:30:23.836 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.41506.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41506.log.gz 2024-09-13T11:30:23.836 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.33442.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43184.log 2024-09-13T11:30:23.836 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33442.log.gz 2024-09-13T11:30:23.836 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.48503.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48503.log.gz 2024-09-13T11:30:23.836 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44430.log 2024-09-13T11:30:23.836 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39233.log 2024-09-13T11:30:23.836 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.43184.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43184.log.gz 2024-09-13T11:30:23.836 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.44430.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33312.log 2024-09-13T11:30:23.836 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44430.log.gz 2024-09-13T11:30:23.836 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.39233.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39233.log.gz 2024-09-13T11:30:23.836 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32091.log 2024-09-13T11:30:23.836 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32905.log 2024-09-13T11:30:23.837 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.33312.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33312.log.gz 2024-09-13T11:30:23.837 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.32091.log: gzip -5 --verbose 0.0% -- -- replaced with /var/log/ceph/ceph-client.admin.32091.log.gz /var/log/ceph/ceph-client.admin.36870.log 2024-09-13T11:30:23.837 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-09-13T11:30:23.837 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.32905.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32905.log.gz 2024-09-13T11:30:23.837 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31795.log 2024-09-13T11:30:23.837 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46891.log 2024-09-13T11:30:23.837 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.36870.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36870.log.gz 2024-09-13T11:30:23.837 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.31795.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44020.log 2024-09-13T11:30:23.837 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31795.log.gz 2024-09-13T11:30:23.837 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.46891.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46891.log.gz 2024-09-13T11:30:23.837 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44153.log 2024-09-13T11:30:23.837 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47454.log 2024-09-13T11:30:23.837 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.44020.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44020.log.gz 2024-09-13T11:30:23.838 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.44153.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40457.log 2024-09-13T11:30:23.838 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44153.log.gz 2024-09-13T11:30:23.838 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.47454.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47454.log.gz 2024-09-13T11:30:23.838 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44908.log 2024-09-13T11:30:23.838 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38307.log 2024-09-13T11:30:23.838 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.40457.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40457.log.gz 2024-09-13T11:30:23.838 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.44908.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39385.log 2024-09-13T11:30:23.838 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44908.log.gz 2024-09-13T11:30:23.838 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.38307.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38307.log.gz 2024-09-13T11:30:23.838 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36231.log 2024-09-13T11:30:23.838 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33865.log 2024-09-13T11:30:23.838 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.39385.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39385.log.gz 2024-09-13T11:30:23.838 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.36231.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45224.log 2024-09-13T11:30:23.838 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36231.log.gz 2024-09-13T11:30:23.839 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.33865.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33865.log.gz 2024-09-13T11:30:23.839 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38443.log 2024-09-13T11:30:23.839 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46839.log 2024-09-13T11:30:23.839 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.45224.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45224.log.gz 2024-09-13T11:30:23.839 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.38443.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31527.log 2024-09-13T11:30:23.839 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38443.log.gz 2024-09-13T11:30:23.839 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.46839.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46839.log.gz 2024-09-13T11:30:23.839 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31044.log 2024-09-13T11:30:23.839 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33233.log 2024-09-13T11:30:23.839 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.31527.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31527.log.gz 2024-09-13T11:30:23.839 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.31044.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40645.log 2024-09-13T11:30:23.839 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31044.log.gz 2024-09-13T11:30:23.840 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.33233.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33233.log.gz 2024-09-13T11:30:23.840 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35912.log 2024-09-13T11:30:23.840 INFO:teuthology.orchestra.run.smithi001.stderr:gzip/var/log/ceph/ceph-client.admin.40645.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.44934.log 2024-09-13T11:30:23.840 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40645.log.gz 2024-09-13T11:30:23.840 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.35912.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30654.log 2024-09-13T11:30:23.840 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35912.log.gz 2024-09-13T11:30:23.840 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.44934.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44934.log.gz 2024-09-13T11:30:23.840 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47506.log 2024-09-13T11:30:23.840 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36976.log 2024-09-13T11:30:23.840 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.30654.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30654.log.gz 2024-09-13T11:30:23.840 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.47506.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30648.log 2024-09-13T11:30:23.840 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47506.log.gz 2024-09-13T11:30:23.840 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.36976.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36976.log.gz 2024-09-13T11:30:23.841 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39135.log 2024-09-13T11:30:23.841 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33961.log 2024-09-13T11:30:23.841 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.30648.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30648.log.gz 2024-09-13T11:30:23.841 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.39135.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31440.log 2024-09-13T11:30:23.841 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39135.log.gz 2024-09-13T11:30:23.841 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.33961.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33961.log.gz 2024-09-13T11:30:23.841 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47488.log 2024-09-13T11:30:23.841 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47081.log 2024-09-13T11:30:23.841 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.31440.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31440.log.gz 2024-09-13T11:30:23.841 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.47488.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36388.log 2024-09-13T11:30:23.841 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47488.log.gz 2024-09-13T11:30:23.841 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.47081.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47081.log.gz 2024-09-13T11:30:23.841 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46735.log 2024-09-13T11:30:23.841 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44018.log 2024-09-13T11:30:23.842 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.36388.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36388.log.gz 2024-09-13T11:30:23.842 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.46735.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36507.log 2024-09-13T11:30:23.842 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46735.log.gz 2024-09-13T11:30:23.842 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.44018.log: gzip 0.0% -5 -- replaced with /var/log/ceph/ceph-client.admin.44018.log.gz --verbose 2024-09-13T11:30:23.842 INFO:teuthology.orchestra.run.smithi001.stderr: -- /var/log/ceph/ceph-client.admin.42938.log 2024-09-13T11:30:23.842 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36133.log 2024-09-13T11:30:23.842 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.36507.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36507.log.gz 2024-09-13T11:30:23.842 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.42938.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43014.log 2024-09-13T11:30:23.842 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42938.log.gz 2024-09-13T11:30:23.842 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.36133.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36133.log.gz 2024-09-13T11:30:23.842 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39438.log 2024-09-13T11:30:23.842 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42034.log 2024-09-13T11:30:23.842 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.43014.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43014.log.gz 2024-09-13T11:30:23.843 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.39438.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31533.log 2024-09-13T11:30:23.843 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39438.log.gz 2024-09-13T11:30:23.843 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.42034.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42034.log.gz 2024-09-13T11:30:23.843 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39257.log 2024-09-13T11:30:23.843 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47490.log 2024-09-13T11:30:23.843 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.31533.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31533.log.gz 2024-09-13T11:30:23.843 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.39257.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31520.log 2024-09-13T11:30:23.843 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39257.log.gz 2024-09-13T11:30:23.843 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.47490.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38982.log 2024-09-13T11:30:23.843 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47490.log.gz 2024-09-13T11:30:23.843 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35356.log 2024-09-13T11:30:23.843 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.31520.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31520.log.gz 2024-09-13T11:30:23.843 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.38982.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47080.log 2024-09-13T11:30:23.843 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38982.log.gz 2024-09-13T11:30:23.844 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.35356.log: gzip 0.0% -5 --verbose -- replaced with /var/log/ceph/ceph-client.admin.35356.log.gz 2024-09-13T11:30:23.844 INFO:teuthology.orchestra.run.smithi001.stderr: -- /var/log/ceph/ceph-client.admin.30655.log 2024-09-13T11:30:23.844 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39796.log 2024-09-13T11:30:23.844 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.47080.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47080.log.gz 2024-09-13T11:30:23.844 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.30655.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46605.log 2024-09-13T11:30:23.844 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30655.log.gz 2024-09-13T11:30:23.844 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.39796.log: gzip -5 0.0% --verbose -- replaced with /var/log/ceph/ceph-client.admin.39796.log.gz -- 2024-09-13T11:30:23.844 INFO:teuthology.orchestra.run.smithi001.stderr: /var/log/ceph/ceph-client.admin.48633.log 2024-09-13T11:30:23.844 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34482.log 2024-09-13T11:30:23.844 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.46605.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46605.log.gz 2024-09-13T11:30:23.844 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.48633.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44533.log 2024-09-13T11:30:23.844 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48633.log.gz 2024-09-13T11:30:23.844 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.34482.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39305.log 2024-09-13T11:30:23.844 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34482.log.gz 2024-09-13T11:30:23.845 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43559.log 2024-09-13T11:30:23.845 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.44533.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44533.log.gz 2024-09-13T11:30:23.845 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.39305.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41765.log 2024-09-13T11:30:23.845 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39305.log.gz 2024-09-13T11:30:23.845 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.43559.log: gzip -5 --verbose -- 0.0% /var/log/ceph/ceph-client.admin.34629.log 2024-09-13T11:30:23.845 INFO:teuthology.orchestra.run.smithi001.stderr: -- replaced with /var/log/ceph/ceph-client.admin.43559.log.gz 2024-09-13T11:30:23.845 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-09-13T11:30:23.845 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.41765.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41765.log.gz 2024-09-13T11:30:23.845 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.34629.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48946.log 2024-09-13T11:30:23.845 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34629.log.gz 2024-09-13T11:30:23.845 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30395.log 2024-09-13T11:30:23.845 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.48946.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35212.log 2024-09-13T11:30:23.845 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48946.log.gz 2024-09-13T11:30:23.846 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.30395.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37752.log 2024-09-13T11:30:23.846 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30395.log.gz 2024-09-13T11:30:23.846 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.35212.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40620.log 2024-09-13T11:30:23.846 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35212.log.gz 2024-09-13T11:30:23.846 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.37752.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37002.log 2024-09-13T11:30:23.846 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37752.log.gz 2024-09-13T11:30:23.846 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47493.log 2024-09-13T11:30:23.846 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.40620.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40620.log.gz 2024-09-13T11:30:23.846 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.37002.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36291.log 2024-09-13T11:30:23.846 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37002.log.gz 2024-09-13T11:30:23.846 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.47493.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41990.log 2024-09-13T11:30:23.846 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47493.log.gz 2024-09-13T11:30:23.846 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.36291.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31769.log 2024-09-13T11:30:23.846 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36291.log.gz 2024-09-13T11:30:23.847 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.41990.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34274.log 2024-09-13T11:30:23.847 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41990.log.gz 2024-09-13T11:30:23.847 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37970.log 2024-09-13T11:30:23.847 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.31769.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31769.log.gz 2024-09-13T11:30:23.847 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.34274.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44870.log 2024-09-13T11:30:23.847 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34274.log.gz 2024-09-13T11:30:23.847 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.37970.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45043.log 2024-09-13T11:30:23.847 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37970.log.gz 2024-09-13T11:30:23.847 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.44870.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31821.log 2024-09-13T11:30:23.847 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44870.log.gz 2024-09-13T11:30:23.847 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.45043.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30420.log 2024-09-13T11:30:23.847 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45043.log.gz 2024-09-13T11:30:23.847 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.31821.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33072.log 2024-09-13T11:30:23.847 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31821.log.gz 2024-09-13T11:30:23.848 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33283.log 2024-09-13T11:30:23.848 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.30420.log: /var/log/ceph/ceph-client.admin.33072.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46499.log 2024-09-13T11:30:23.848 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30420.log.gz 2024-09-13T11:30:23.848 INFO:teuthology.orchestra.run.smithi001.stderr: 86.8% -- replaced with /var/log/ceph/ceph.log.gz 2024-09-13T11:30:23.848 INFO:teuthology.orchestra.run.smithi001.stderr: -- replaced with /var/log/ceph/ceph-client.admin.33072.log.gz 2024-09-13T11:30:23.848 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.33283.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42756.log 2024-09-13T11:30:23.848 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33283.log.gz 2024-09-13T11:30:23.848 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.46499.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31069.log 2024-09-13T11:30:23.848 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46499.log.gz 2024-09-13T11:30:23.848 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.42756.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48842.log 2024-09-13T11:30:23.848 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42756.log.gz 2024-09-13T11:30:23.848 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.31069.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31847.log 2024-09-13T11:30:23.848 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31069.log.gz 2024-09-13T11:30:23.848 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.48842.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47078.log 2024-09-13T11:30:23.849 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48842.log.gz 2024-09-13T11:30:23.849 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.31847.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41387.log 2024-09-13T11:30:23.849 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31847.log.gz 2024-09-13T11:30:23.849 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.47078.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.47078.log.gz -5 2024-09-13T11:30:23.849 INFO:teuthology.orchestra.run.smithi001.stderr: --verbose -- /var/log/ceph/ceph-client.admin.39742.log 2024-09-13T11:30:23.849 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.41387.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30647.log 2024-09-13T11:30:23.849 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41387.log.gz 2024-09-13T11:30:23.849 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.39742.log: gzip 0.0% -5 --verbose -- replaced with /var/log/ceph/ceph-client.admin.39742.log.gz -- 2024-09-13T11:30:23.849 INFO:teuthology.orchestra.run.smithi001.stderr: /var/log/ceph/ceph-client.admin.33609.log 2024-09-13T11:30:23.849 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.30647.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37320.log 2024-09-13T11:30:23.849 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30647.log.gz 2024-09-13T11:30:23.849 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.33609.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48477.log 0.0% 2024-09-13T11:30:23.849 INFO:teuthology.orchestra.run.smithi001.stderr: -- replaced with /var/log/ceph/ceph-client.admin.33609.log.gz 2024-09-13T11:30:23.850 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.37320.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47071.log 2024-09-13T11:30:23.850 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37320.log.gz 2024-09-13T11:30:23.850 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.48477.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48477.log.gz 2024-09-13T11:30:23.850 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47857.log 2024-09-13T11:30:23.850 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.47071.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31942.log 2024-09-13T11:30:23.850 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47071.log.gz 2024-09-13T11:30:23.850 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.47857.log: gzip 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47857.log.gz -5 2024-09-13T11:30:23.850 INFO:teuthology.orchestra.run.smithi001.stderr: --verbose -- /var/log/ceph/ceph-client.admin.32604.log 2024-09-13T11:30:23.851 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.31942.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33446.log 2024-09-13T11:30:23.852 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31942.log.gz 2024-09-13T11:30:23.852 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.32604.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31074.log 2024-09-13T11:30:23.852 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32604.log.gz 2024-09-13T11:30:23.852 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.33446.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35455.log 2024-09-13T11:30:23.852 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33446.log.gz 2024-09-13T11:30:23.852 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.31074.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36556.log 2024-09-13T11:30:23.852 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31074.log.gz 2024-09-13T11:30:23.852 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.35455.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47055.log 2024-09-13T11:30:23.852 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35455.log.gz 2024-09-13T11:30:23.852 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.36556.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.36556.log.gz 2024-09-13T11:30:23.852 INFO:teuthology.orchestra.run.smithi001.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.38020.log 2024-09-13T11:30:23.852 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.47055.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31435.log 2024-09-13T11:30:23.852 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47055.log.gz 2024-09-13T11:30:23.852 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.38020.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38944.log 0.0% 2024-09-13T11:30:23.853 INFO:teuthology.orchestra.run.smithi001.stderr: -- replaced with /var/log/ceph/ceph-client.admin.38020.log.gz 2024-09-13T11:30:23.853 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.31435.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46395.log 2024-09-13T11:30:23.853 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31435.log.gz 2024-09-13T11:30:23.853 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.38944.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45440.log 2024-09-13T11:30:23.853 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38944.log.gz 2024-09-13T11:30:23.853 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.46395.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41003.log 2024-09-13T11:30:23.853 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46395.log.gz 2024-09-13T11:30:23.853 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.45440.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32473.log 2024-09-13T11:30:23.853 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45440.log.gz 2024-09-13T11:30:23.853 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.41003.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48607.log 2024-09-13T11:30:23.853 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41003.log.gz 2024-09-13T11:30:23.853 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.32473.log: gzip 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32473.log.gz -5 2024-09-13T11:30:23.853 INFO:teuthology.orchestra.run.smithi001.stderr: --verbose -- /var/log/ceph/ceph-client.admin.31055.log 2024-09-13T11:30:23.853 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.48607.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30495.log 2024-09-13T11:30:23.854 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48607.log.gz 2024-09-13T11:30:23.854 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.31055.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33816.log 0.0% 2024-09-13T11:30:23.854 INFO:teuthology.orchestra.run.smithi001.stderr: -- replaced with /var/log/ceph/ceph-client.admin.31055.log.gz 2024-09-13T11:30:23.854 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.30495.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35722.log 2024-09-13T11:30:23.854 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30495.log.gz 2024-09-13T11:30:23.854 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.33816.log: gzip -5 --verbose -- 0.0% /var/log/ceph/ceph-client.admin.43803.log 2024-09-13T11:30:23.854 INFO:teuthology.orchestra.run.smithi001.stderr: -- replaced with /var/log/ceph/ceph-client.admin.33816.log.gz 2024-09-13T11:30:23.854 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.35722.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36440.log 2024-09-13T11:30:23.854 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35722.log.gz 2024-09-13T11:30:23.854 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.43803.log: gzip -5 --verbose -- 0.0% /var/log/ceph/ceph-client.admin.40375.log -- replaced with /var/log/ceph/ceph-client.admin.43803.log.gz 2024-09-13T11:30:23.854 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-09-13T11:30:23.854 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.36440.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38151.log 2024-09-13T11:30:23.854 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36440.log.gz 2024-09-13T11:30:23.855 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.40375.log: gzip 0.0% -5 --verbose -- replaced with /var/log/ceph/ceph-client.admin.40375.log.gz -- 2024-09-13T11:30:23.855 INFO:teuthology.orchestra.run.smithi001.stderr: /var/log/ceph/ceph-client.admin.41466.log 2024-09-13T11:30:23.855 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.38151.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44210.log 2024-09-13T11:30:23.855 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38151.log.gz 2024-09-13T11:30:23.855 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.41466.log: gzip 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41466.log.gz -5 2024-09-13T11:30:23.855 INFO:teuthology.orchestra.run.smithi001.stderr: --verbose -- /var/log/ceph/ceph-client.admin.36128.log 2024-09-13T11:30:23.855 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.44210.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38831.log 2024-09-13T11:30:23.855 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44210.log.gz 2024-09-13T11:30:23.855 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.36128.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31940.log 0.0% 2024-09-13T11:30:23.855 INFO:teuthology.orchestra.run.smithi001.stderr: -- replaced with /var/log/ceph/ceph-client.admin.36128.log.gz 2024-09-13T11:30:23.855 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.38831.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48555.log 2024-09-13T11:30:23.855 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38831.log.gz 2024-09-13T11:30:23.855 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.31940.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31940.log.gz 2024-09-13T11:30:23.855 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37756.log 2024-09-13T11:30:23.855 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.48555.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42059.log 2024-09-13T11:30:23.856 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48555.log.gz 2024-09-13T11:30:23.856 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.37756.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41864.log 2024-09-13T11:30:23.856 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37756.log.gz 2024-09-13T11:30:23.856 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.42059.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42132.log 2024-09-13T11:30:23.856 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42059.log.gz 2024-09-13T11:30:23.856 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.41864.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48581.log 2024-09-13T11:30:23.856 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41864.log.gz 2024-09-13T11:30:23.856 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.42132.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37559.log 2024-09-13T11:30:23.856 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42132.log.gz 2024-09-13T11:30:23.856 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48738.log 2024-09-13T11:30:23.856 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.48581.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48581.log.gz 2024-09-13T11:30:23.856 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.37559.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.21415.log 2024-09-13T11:30:23.856 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37559.log.gz 2024-09-13T11:30:23.856 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.48738.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47465.log 2024-09-13T11:30:23.857 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48738.log.gz 2024-09-13T11:30:23.857 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38468.log 2024-09-13T11:30:23.857 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph.tmp-client.admin.21415.log: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.21415.log.gz 2024-09-13T11:30:23.857 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.47465.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42107.log 2024-09-13T11:30:23.857 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47465.log.gz 2024-09-13T11:30:23.857 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.38468.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43308.log 2024-09-13T11:30:23.857 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38468.log.gz 2024-09-13T11:30:23.857 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41689.log 2024-09-13T11:30:23.857 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.42107.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42107.log.gz 2024-09-13T11:30:23.857 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.43308.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26436.log 2024-09-13T11:30:23.857 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43308.log.gz 2024-09-13T11:30:23.857 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.41689.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31006.log 2024-09-13T11:30:23.857 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41689.log.gz 2024-09-13T11:30:23.857 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32355.log 2024-09-13T11:30:23.858 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.26436.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26436.log.gz 2024-09-13T11:30:23.858 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.31006.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41234.log 2024-09-13T11:30:23.858 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31006.log.gz 2024-09-13T11:30:23.858 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.32355.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35160.log 2024-09-13T11:30:23.858 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32355.log.gz 2024-09-13T11:30:23.858 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38589.log 2024-09-13T11:30:23.858 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.41234.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41234.log.gz 2024-09-13T11:30:23.858 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.35160.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30634.log 2024-09-13T11:30:23.858 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35160.log.gz 2024-09-13T11:30:23.858 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.38589.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48970.log 2024-09-13T11:30:23.858 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38589.log.gz 2024-09-13T11:30:23.858 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44783.log 2024-09-13T11:30:23.858 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.30634.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30634.log.gz 2024-09-13T11:30:23.858 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.48970.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47482.log 2024-09-13T11:30:23.858 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48970.log.gz 2024-09-13T11:30:23.859 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.44783.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-09-13T11:30:23.859 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44783.log.gz 2024-09-13T11:30:23.859 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47056.log 2024-09-13T11:30:23.859 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.47482.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47482.log.gz 2024-09-13T11:30:23.859 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46317.log 2024-09-13T11:30:23.859 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.47056.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46657.log 2024-09-13T11:30:23.859 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47056.log.gz 2024-09-13T11:30:23.859 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.46317.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48399.log 2024-09-13T11:30:23.859 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46317.log.gz 2024-09-13T11:30:23.859 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.46657.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37207.log 2024-09-13T11:30:23.859 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46657.log.gz 2024-09-13T11:30:23.859 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.48399.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43101.log 2024-09-13T11:30:23.859 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48399.log.gz 2024-09-13T11:30:23.859 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.37207.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35295.log 2024-09-13T11:30:23.860 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37207.log.gz 2024-09-13T11:30:23.860 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.43101.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41386.log 2024-09-13T11:30:23.860 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43101.log.gz 2024-09-13T11:30:23.860 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.35295.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35295.log.gz 2024-09-13T11:30:23.860 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41637.log 2024-09-13T11:30:23.860 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.41386.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36871.log 2024-09-13T11:30:23.860 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41386.log.gz 2024-09-13T11:30:23.860 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.41637.log: 89.2% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-09-13T11:30:23.860 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35668.log 2024-09-13T11:30:23.860 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41637.log.gz 2024-09-13T11:30:23.860 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.36871.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30999.log 2024-09-13T11:30:23.860 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36871.log.gz 2024-09-13T11:30:23.860 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.35668.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47463.log 2024-09-13T11:30:23.860 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35668.log.gz 2024-09-13T11:30:23.861 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.30999.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39027.log 2024-09-13T11:30:23.861 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30999.log.gz 2024-09-13T11:30:23.861 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37839.log 2024-09-13T11:30:23.861 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.47463.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47463.log.gz 2024-09-13T11:30:23.861 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.39027.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34019.log 2024-09-13T11:30:23.861 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39027.log.gz 2024-09-13T11:30:23.861 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.37839.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37839.log.gz 2024-09-13T11:30:23.861 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30626.log 2024-09-13T11:30:23.861 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45816.log 2024-09-13T11:30:23.861 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.34019.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34019.log.gz 2024-09-13T11:30:23.861 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.30626.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38362.log 2024-09-13T11:30:23.861 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30626.log.gz 2024-09-13T11:30:23.861 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.45816.log: gzip 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45816.log.gz 2024-09-13T11:30:23.861 INFO:teuthology.orchestra.run.smithi001.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.46369.log 2024-09-13T11:30:23.862 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48217.log 2024-09-13T11:30:23.862 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.38362.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38362.log.gz 2024-09-13T11:30:23.862 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.46369.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30451.log 2024-09-13T11:30:23.862 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46369.log.gz 2024-09-13T11:30:23.862 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.48217.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48321.log 2024-09-13T11:30:23.862 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48217.log.gz 2024-09-13T11:30:23.862 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26202.log 2024-09-13T11:30:23.862 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.30451.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30451.log.gz 2024-09-13T11:30:23.862 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.48321.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47066.log 2024-09-13T11:30:23.862 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48321.log.gz 2024-09-13T11:30:23.862 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.26202.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48712.log 2024-09-13T11:30:23.862 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26202.log.gz 2024-09-13T11:30:23.862 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36295.log 2024-09-13T11:30:23.862 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.47066.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47066.log.gz 2024-09-13T11:30:23.863 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.48712.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40932.log 2024-09-13T11:30:23.863 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48712.log.gz 2024-09-13T11:30:23.863 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.36295.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.36295.log.gz 2024-09-13T11:30:23.863 INFO:teuthology.orchestra.run.smithi001.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.48191.log 2024-09-13T11:30:23.863 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34164.log 2024-09-13T11:30:23.863 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.40932.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40932.log.gz 2024-09-13T11:30:23.863 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.48191.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44693.log 2024-09-13T11:30:23.863 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48191.log.gz 2024-09-13T11:30:23.863 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.34164.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44610.log 2024-09-13T11:30:23.863 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34164.log.gz 2024-09-13T11:30:23.863 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.12.log 2024-09-13T11:30:23.863 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.44693.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44693.log.gz 2024-09-13T11:30:23.863 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.44610.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47533.log 2024-09-13T11:30:23.863 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44610.log.gz 2024-09-13T11:30:23.864 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-osd.12.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48816.log 2024-09-13T11:30:23.864 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.47533.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48243.log 2024-09-13T11:30:23.864 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47533.log.gz 2024-09-13T11:30:23.864 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.48816.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32823.log 2024-09-13T11:30:23.864 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48816.log.gz 2024-09-13T11:30:23.864 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.48243.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45704.log 2024-09-13T11:30:23.864 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48243.log.gz 2024-09-13T11:30:23.864 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.32823.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37922.log 2024-09-13T11:30:23.864 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32823.log.gz 2024-09-13T11:30:23.864 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.45704.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45068.log 2024-09-13T11:30:23.864 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45704.log.gz 2024-09-13T11:30:23.864 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.37922.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30618.log 2024-09-13T11:30:23.865 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37922.log.gz 2024-09-13T11:30:23.865 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.45068.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44476.log 2024-09-13T11:30:23.865 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45068.log.gz 2024-09-13T11:30:23.865 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.30618.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43218.log 2024-09-13T11:30:23.865 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30618.log.gz 2024-09-13T11:30:23.865 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.44476.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44476.log.gzgzip 2024-09-13T11:30:23.865 INFO:teuthology.orchestra.run.smithi001.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.48894.log 2024-09-13T11:30:23.865 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.43218.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34623.log 2024-09-13T11:30:23.865 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43218.log.gz 2024-09-13T11:30:23.865 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.48894.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34117.log 2024-09-13T11:30:23.865 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48894.log.gz 2024-09-13T11:30:23.865 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.34623.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43807.log 2024-09-13T11:30:23.865 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34623.log.gz 2024-09-13T11:30:23.865 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.34117.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32183.log 2024-09-13T11:30:23.865 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34117.log.gz 2024-09-13T11:30:23.866 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.43807.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44236.log 2024-09-13T11:30:23.866 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43807.log.gz 2024-09-13T11:30:23.866 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.32183.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43982.log 2024-09-13T11:30:23.866 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32183.log.gz 2024-09-13T11:30:23.866 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.44236.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39094.log 2024-09-13T11:30:23.866 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44236.log.gz 2024-09-13T11:30:23.866 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.43982.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45790.log 2024-09-13T11:30:23.866 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43982.log.gz 2024-09-13T11:30:23.866 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.39094.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41077.log 2024-09-13T11:30:23.866 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39094.log.gz 2024-09-13T11:30:23.866 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.45790.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37635.log 2024-09-13T11:30:23.866 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45790.log.gz 2024-09-13T11:30:23.866 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.41077.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35552.log 2024-09-13T11:30:23.867 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41077.log.gz 2024-09-13T11:30:23.867 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.37635.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48061.log 2024-09-13T11:30:23.867 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37635.log.gz 2024-09-13T11:30:23.867 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.35552.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31107.log 2024-09-13T11:30:23.867 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35552.log.gz 2024-09-13T11:30:23.867 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.48061.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47047.log 2024-09-13T11:30:23.867 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48061.log.gz 2024-09-13T11:30:23.867 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.31107.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42992.log 2024-09-13T11:30:23.867 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31107.log.gz 2024-09-13T11:30:23.867 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.47047.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43127.log 2024-09-13T11:30:23.867 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47047.log.gz 2024-09-13T11:30:23.867 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.42992.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41938.log 2024-09-13T11:30:23.867 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42992.log.gz 2024-09-13T11:30:23.867 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.43127.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37103.log 2024-09-13T11:30:23.868 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43127.log.gz 2024-09-13T11:30:23.868 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.41938.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31873.log 2024-09-13T11:30:23.868 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41938.log.gz 2024-09-13T11:30:23.868 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.37103.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43696.log 2024-09-13T11:30:23.868 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37103.log.gz 2024-09-13T11:30:23.868 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38388.log 2024-09-13T11:30:23.868 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.31873.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31873.log.gz 2024-09-13T11:30:23.868 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.43696.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47441.log 2024-09-13T11:30:23.868 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43696.log.gz 2024-09-13T11:30:23.868 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.38388.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30627.log 2024-09-13T11:30:23.868 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38388.log.gz 2024-09-13T11:30:23.868 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.47441.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47451.log 2024-09-13T11:30:23.868 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47441.log.gz 2024-09-13T11:30:23.869 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.30627.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37294.log 2024-09-13T11:30:23.869 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30627.log.gz 2024-09-13T11:30:23.869 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.47451.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45309.log 2024-09-13T11:30:23.869 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47451.log.gz 2024-09-13T11:30:23.869 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.37294.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43563.log 2024-09-13T11:30:23.869 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37294.log.gz 2024-09-13T11:30:23.869 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.45309.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41264.log 2024-09-13T11:30:23.869 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45309.log.gz 2024-09-13T11:30:23.869 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.43563.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48529.log 2024-09-13T11:30:23.869 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43563.log.gz 2024-09-13T11:30:23.869 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.41264.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33738.log 2024-09-13T11:30:23.869 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41264.log.gz 2024-09-13T11:30:23.869 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.48529.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34405.log 2024-09-13T11:30:23.869 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48529.log.gz 2024-09-13T11:30:23.870 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.33738.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47079.log 2024-09-13T11:30:23.870 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33738.log.gz 2024-09-13T11:30:23.870 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.34405.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44959.log 2024-09-13T11:30:23.870 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34405.log.gz 2024-09-13T11:30:23.870 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.47079.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46078.log 2024-09-13T11:30:23.870 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47079.log.gz 2024-09-13T11:30:23.870 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.44959.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45348.log 2024-09-13T11:30:23.870 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44959.log.gz 2024-09-13T11:30:23.870 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.46078.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42368.log 2024-09-13T11:30:23.870 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46078.log.gz 2024-09-13T11:30:23.870 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.45348.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33207.log 2024-09-13T11:30:23.870 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45348.log.gz 2024-09-13T11:30:23.870 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.42368.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30573.log 2024-09-13T11:30:23.871 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42368.log.gz 2024-09-13T11:30:23.871 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.33207.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2024-09-13T11:30:23.871 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33207.log.gz 2024-09-13T11:30:23.871 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.30573.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41182.log 2024-09-13T11:30:23.871 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30573.log.gz 2024-09-13T11:30:23.871 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48425.log 2024-09-13T11:30:23.871 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.41182.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32380.log 2024-09-13T11:30:23.871 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41182.log.gz 2024-09-13T11:30:23.871 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.48425.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48425.log.gz 2024-09-13T11:30:23.871 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31047.log 2024-09-13T11:30:23.871 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44668.log 2024-09-13T11:30:23.871 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.32380.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32380.log.gz/var/log/ceph/ceph-client.admin.31047.log: 2024-09-13T11:30:23.871 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40807.log 2024-09-13T11:30:23.872 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31047.log.gz 2024-09-13T11:30:23.872 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.44668.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44668.log.gz 2024-09-13T11:30:23.872 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38738.log 2024-09-13T11:30:23.872 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.40807.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48764.log 2024-09-13T11:30:23.872 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40807.log.gz 2024-09-13T11:30:23.872 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.38738.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38678.log 2024-09-13T11:30:23.872 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38738.log.gz 2024-09-13T11:30:23.872 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.48764.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48764.log.gz 2024-09-13T11:30:23.872 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37551.log 2024-09-13T11:30:23.872 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.38678.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41662.log 2024-09-13T11:30:23.872 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38678.log.gz 2024-09-13T11:30:23.872 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.37551.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37551.log.gz 2024-09-13T11:30:23.872 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44584.log 2024-09-13T11:30:23.872 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43214.log 2024-09-13T11:30:23.873 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.41662.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41662.log.gz 2024-09-13T11:30:23.873 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.44584.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40250.log 2024-09-13T11:30:23.873 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44584.log.gz 2024-09-13T11:30:23.873 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.43214.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43214.log.gz 2024-09-13T11:30:23.873 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32754.log 2024-09-13T11:30:23.873 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.40250.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38872.log 2024-09-13T11:30:23.873 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40250.log.gz 2024-09-13T11:30:23.873 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.32754.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33840.log 2024-09-13T11:30:23.873 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32754.log.gz 2024-09-13T11:30:23.873 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.38872.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38872.log.gz 2024-09-13T11:30:23.873 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40115.log 2024-09-13T11:30:23.873 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.33840.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36414.log 2024-09-13T11:30:23.873 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33840.log.gz 2024-09-13T11:30:23.874 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.40115.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40115.log.gz 2024-09-13T11:30:23.874 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32702.log 2024-09-13T11:30:23.874 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35730.log 2024-09-13T11:30:23.874 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.36414.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36414.log.gz 2024-09-13T11:30:23.874 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.32702.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40167.log 2024-09-13T11:30:23.874 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32702.log.gz 2024-09-13T11:30:23.874 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41713.log 2024-09-13T11:30:23.874 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.35730.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35730.log.gz 2024-09-13T11:30:23.874 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.40167.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47053.log 2024-09-13T11:30:23.874 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40167.log.gz 2024-09-13T11:30:23.874 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.41713.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41713.log.gz 2024-09-13T11:30:23.874 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45198.log 2024-09-13T11:30:23.874 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.47053.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31070.log 2024-09-13T11:30:23.874 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47053.log.gz 2024-09-13T11:30:23.874 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.45198.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33076.log 2024-09-13T11:30:23.875 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45198.log.gz 2024-09-13T11:30:23.875 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30657.log 2024-09-13T11:30:23.875 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.31070.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31070.log.gz 2024-09-13T11:30:23.875 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.33076.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45392.log 2024-09-13T11:30:23.875 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33076.log.gz 2024-09-13T11:30:23.875 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.30657.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30657.log.gz 2024-09-13T11:30:23.875 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31743.log 2024-09-13T11:30:23.875 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.45392.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35696.log 2024-09-13T11:30:23.875 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45392.log.gz 2024-09-13T11:30:23.875 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.31743.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42534.log 2024-09-13T11:30:23.875 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31743.log.gz 2024-09-13T11:30:23.875 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.35696.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35696.log.gz 2024-09-13T11:30:23.875 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31022.log 2024-09-13T11:30:23.876 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.42534.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32274.log 2024-09-13T11:30:23.876 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42534.log.gz 2024-09-13T11:30:23.876 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.31022.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32963.log 2024-09-13T11:30:23.876 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31022.log.gz 2024-09-13T11:30:23.876 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47050.log 2024-09-13T11:30:23.876 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.32274.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32274.log.gz 2024-09-13T11:30:23.876 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.32963.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44345.log 2024-09-13T11:30:23.876 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32963.log.gz 2024-09-13T11:30:23.876 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.47050.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47050.log.gz 2024-09-13T11:30:23.876 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46865.log 2024-09-13T11:30:23.876 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.44345.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40461.log 2024-09-13T11:30:23.876 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44345.log.gz 2024-09-13T11:30:23.876 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.46865.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40833.log 2024-09-13T11:30:23.876 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46865.log.gz 2024-09-13T11:30:23.877 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37885.log 2024-09-13T11:30:23.877 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.40461.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40461.log.gz 2024-09-13T11:30:23.877 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.40833.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47386.log 2024-09-13T11:30:23.877 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40833.log.gz 2024-09-13T11:30:23.877 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.37885.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37885.log.gz 2024-09-13T11:30:23.877 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40275.log 2024-09-13T11:30:23.877 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.47386.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35831.log 2024-09-13T11:30:23.877 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47386.log.gz 2024-09-13T11:30:23.877 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.40275.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42429.log 2024-09-13T11:30:23.877 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40275.log.gz 2024-09-13T11:30:23.877 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.35831.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35831.log.gz 2024-09-13T11:30:23.877 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42672.log 2024-09-13T11:30:23.877 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.42429.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42324.log 2024-09-13T11:30:23.877 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42429.log.gz 2024-09-13T11:30:23.877 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.42672.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34216.log 2024-09-13T11:30:23.878 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42672.log.gz 2024-09-13T11:30:23.878 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.42324.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33388.log 2024-09-13T11:30:23.878 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42324.log.gz 2024-09-13T11:30:23.878 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.34216.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34216.log.gz 2024-09-13T11:30:23.878 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44297.log 2024-09-13T11:30:23.878 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.33388.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46917.log 2024-09-13T11:30:23.878 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33388.log.gz 2024-09-13T11:30:23.878 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.44297.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40418.log 2024-09-13T11:30:23.878 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44297.log.gz 2024-09-13T11:30:23.878 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46187.log 2024-09-13T11:30:23.878 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.46917.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46917.log.gz 2024-09-13T11:30:23.878 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.40418.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42673.log 2024-09-13T11:30:23.878 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40418.log.gz 2024-09-13T11:30:23.878 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.46187.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46187.log.gz 2024-09-13T11:30:23.878 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38734.log 2024-09-13T11:30:23.879 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.42673.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39389.log 2024-09-13T11:30:23.879 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42673.log.gz 2024-09-13T11:30:23.879 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.38734.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38734.log.gz 2024-09-13T11:30:23.879 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.39389.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39389.log.gz 2024-09-13T11:30:23.880 INFO:teuthology.orchestra.run.smithi177.stderr: 91.5% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-09-13T11:30:23.881 INFO:teuthology.orchestra.run.smithi162.stderr:/var/log/ceph/ceph-osd.10.log: 95.5%/var/log/ceph/objectstore_tool.34493.log: -- replaced with /var/log/ceph/bluestore_tool.32273.log.gz 2024-09-13T11:30:23.881 INFO:teuthology.orchestra.run.smithi162.stderr: 94.4% -- replaced with /var/log/ceph/bluestore_tool.33019.log.gz 2024-09-13T11:30:23.881 INFO:teuthology.orchestra.run.smithi162.stderr: 95.3% -- replaced with /var/log/ceph/objectstore_tool.34691.log.gz 2024-09-13T11:30:23.881 INFO:teuthology.orchestra.run.smithi162.stderr: 95.4% -- replaced with /var/log/ceph/objectstore_tool.30503.log.gz 2024-09-13T11:30:23.881 INFO:teuthology.orchestra.run.smithi162.stderr: 95.4% -- replaced with /var/log/ceph/objectstore_tool.34493.log.gz 2024-09-13T11:30:24.135 INFO:teuthology.orchestra.run.smithi092.stderr: 90.1% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2024-09-13T11:30:24.221 INFO:teuthology.orchestra.run.smithi162.stderr: 95.8% -- replaced with /var/log/ceph/objectstore_tool.31479.log.gz 2024-09-13T11:30:24.223 INFO:teuthology.orchestra.run.smithi162.stderr: 89.9% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2024-09-13T11:30:24.251 INFO:teuthology.orchestra.run.smithi162.stderr: 96.0% -- replaced with /var/log/ceph/objectstore_tool.30725.log.gz 2024-09-13T11:30:24.381 INFO:teuthology.orchestra.run.smithi001.stderr: 90.0% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-09-13T11:30:32.889 INFO:teuthology.orchestra.run.smithi162.stderr: 91.7% -- replaced with /var/log/ceph/ceph-osd.6.log.gz 2024-09-13T11:30:36.286 INFO:teuthology.orchestra.run.smithi177.stderr: 92.1% -- replaced with /var/log/ceph/ceph-osd.15.log.gz 2024-09-13T11:30:38.562 INFO:teuthology.orchestra.run.smithi092.stderr: 91.1% -- replaced with /var/log/ceph/ceph-osd.9.log.gz 2024-09-13T11:30:40.209 INFO:teuthology.orchestra.run.smithi162.stderr: 91.0% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-09-13T11:30:42.994 INFO:teuthology.orchestra.run.smithi001.stderr: 91.0% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2024-09-13T11:30:44.375 INFO:teuthology.orchestra.run.smithi162.stderr: 91.3% -- replaced with /var/log/ceph/ceph-osd.14.log.gz 2024-09-13T11:30:45.782 INFO:teuthology.orchestra.run.smithi001.stderr: 90.7% -- replaced with /var/log/ceph/ceph-osd.8.log.gz 2024-09-13T11:30:50.089 INFO:teuthology.orchestra.run.smithi092.stderr: 90.8% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2024-09-13T11:30:51.402 INFO:teuthology.orchestra.run.smithi162.stderr: 91.0% -- replaced with /var/log/ceph/ceph-osd.10.log.gz 2024-09-13T11:30:51.404 INFO:teuthology.orchestra.run.smithi162.stderr: 2024-09-13T11:30:51.404 INFO:teuthology.orchestra.run.smithi162.stderr:real 0m27.793s 2024-09-13T11:30:51.405 INFO:teuthology.orchestra.run.smithi162.stderr:user 1m14.002s 2024-09-13T11:30:51.405 INFO:teuthology.orchestra.run.smithi162.stderr:sys 0m2.401s 2024-09-13T11:31:00.594 INFO:teuthology.orchestra.run.smithi001.stderr: 90.8% -- replaced with /var/log/ceph/ceph-osd.12.log.gz 2024-09-13T11:31:03.560 INFO:teuthology.orchestra.run.smithi092.stderr: 90.4% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-09-13T11:31:06.129 INFO:teuthology.orchestra.run.smithi092.stderr: 90.4% -- replaced with /var/log/ceph/ceph-osd.13.log.gz 2024-09-13T11:31:06.131 INFO:teuthology.orchestra.run.smithi092.stderr: 2024-09-13T11:31:06.131 INFO:teuthology.orchestra.run.smithi092.stderr:real 0m42.540s 2024-09-13T11:31:06.131 INFO:teuthology.orchestra.run.smithi092.stderr:user 2m0.711s 2024-09-13T11:31:06.131 INFO:teuthology.orchestra.run.smithi092.stderr:sys 0m3.764s 2024-09-13T11:31:09.362 INFO:teuthology.orchestra.run.smithi177.stderr: 90.3% -- replaced with /var/log/ceph/ceph-osd.7.log.gz 2024-09-13T11:31:19.320 INFO:teuthology.orchestra.run.smithi177.stderr: 90.4% -- replaced with /var/log/ceph/ceph-osd.11.log.gz 2024-09-13T11:31:26.958 INFO:teuthology.orchestra.run.smithi177.stderr: 90.0% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-09-13T11:31:26.959 INFO:teuthology.orchestra.run.smithi177.stderr: 2024-09-13T11:31:26.959 INFO:teuthology.orchestra.run.smithi177.stderr:real 1m3.336s 2024-09-13T11:31:26.959 INFO:teuthology.orchestra.run.smithi177.stderr:user 2m52.155s 2024-09-13T11:31:26.959 INFO:teuthology.orchestra.run.smithi177.stderr:sys 0m5.104s 2024-09-13T11:31:35.442 INFO:teuthology.orchestra.run.smithi001.stderr: 90.0% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-09-13T11:31:35.445 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-09-13T11:31:35.445 INFO:teuthology.orchestra.run.smithi001.stderr:real 1m11.862s 2024-09-13T11:31:35.445 INFO:teuthology.orchestra.run.smithi001.stderr:user 2m27.121s 2024-09-13T11:31:35.445 INFO:teuthology.orchestra.run.smithi001.stderr:sys 0m4.139s 2024-09-13T11:31:35.446 INFO:tasks.ceph:Archiving logs... 2024-09-13T11:31:35.447 DEBUG:teuthology.misc:Transferring archived files from smithi001:/var/log/ceph to /home/teuthworker/archive/akupczyk-2024-09-13_09:37:56-rados-wip-aclamk-cbt-combined-distro-default-smithi/7903409/remote/smithi001/log 2024-09-13T11:31:35.448 DEBUG:teuthology.orchestra.run.smithi001:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-13T11:31:59.097 DEBUG:teuthology.misc:Transferring archived files from smithi092:/var/log/ceph to /home/teuthworker/archive/akupczyk-2024-09-13_09:37:56-rados-wip-aclamk-cbt-combined-distro-default-smithi/7903409/remote/smithi092/log 2024-09-13T11:31:59.098 DEBUG:teuthology.orchestra.run.smithi092:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-13T11:32:18.416 DEBUG:teuthology.misc:Transferring archived files from smithi162:/var/log/ceph to /home/teuthworker/archive/akupczyk-2024-09-13_09:37:56-rados-wip-aclamk-cbt-combined-distro-default-smithi/7903409/remote/smithi162/log 2024-09-13T11:32:18.417 DEBUG:teuthology.orchestra.run.smithi162:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-13T11:32:31.985 DEBUG:teuthology.misc:Transferring archived files from smithi177:/var/log/ceph to /home/teuthworker/archive/akupczyk-2024-09-13_09:37:56-rados-wip-aclamk-cbt-combined-distro-default-smithi/7903409/remote/smithi177/log 2024-09-13T11:32:31.986 DEBUG:teuthology.orchestra.run.smithi177:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-13T11:33:00.111 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-09-13T11:33:00.122 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-09-13T11:33:00.123 DEBUG:teuthology.orchestra.run.smithi001:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-13T11:33:00.126 DEBUG:teuthology.orchestra.run.smithi092:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-13T11:33:00.128 DEBUG:teuthology.orchestra.run.smithi162:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-13T11:33:00.131 DEBUG:teuthology.orchestra.run.smithi177:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-13T11:33:00.244 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-09-13T11:33:00.254 INFO:teuthology.task.clock:Checking final clock skew... 2024-09-13T11:33:00.255 DEBUG:teuthology.orchestra.run.smithi001:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-13T11:33:00.257 DEBUG:teuthology.orchestra.run.smithi092:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-13T11:33:00.260 DEBUG:teuthology.orchestra.run.smithi162:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-13T11:33:00.262 DEBUG:teuthology.orchestra.run.smithi177:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-13T11:33:00.269 INFO:teuthology.orchestra.run.smithi001.stdout: remote refid st t when poll reach delay offset jitter 2024-09-13T11:33:00.269 INFO:teuthology.orchestra.run.smithi001.stdout:============================================================================== 2024-09-13T11:33:00.269 INFO:teuthology.orchestra.run.smithi001.stdout:*hv01.front.sepi 67.205.162.81 3 u 5 64 377 0.081 -0.673 0.802 2024-09-13T11:33:00.269 INFO:teuthology.orchestra.run.smithi001.stdout:+hv02.front.sepi 63.231.80.2 3 u 6 64 377 0.083 -0.998 1.013 2024-09-13T11:33:00.269 INFO:teuthology.orchestra.run.smithi001.stdout:+hv03.front.sepi 74.6.168.72 3 u 63 64 377 0.070 +0.002 0.770 2024-09-13T11:33:00.269 INFO:teuthology.orchestra.run.smithi001.stdout: hv04.front.sepi .INIT. 16 u - 256 0 0.000 +0.000 0.000 2024-09-13T11:33:00.271 INFO:teuthology.orchestra.run.smithi092.stdout: remote refid st t when poll reach delay offset jitter 2024-09-13T11:33:00.271 INFO:teuthology.orchestra.run.smithi092.stdout:============================================================================== 2024-09-13T11:33:00.271 INFO:teuthology.orchestra.run.smithi092.stdout:+hv01.front.sepi 67.205.162.81 3 u 2 64 377 0.083 -5.538 3.147 2024-09-13T11:33:00.271 INFO:teuthology.orchestra.run.smithi092.stdout:+hv02.front.sepi 63.231.80.2 3 u 3 64 377 0.078 -1.304 4.179 2024-09-13T11:33:00.271 INFO:teuthology.orchestra.run.smithi092.stdout:*hv03.front.sepi 74.6.168.72 3 u 4 64 377 0.065 -0.796 3.697 2024-09-13T11:33:00.271 INFO:teuthology.orchestra.run.smithi092.stdout: hv04.front.sepi .INIT. 16 u - 256 0 0.000 +0.000 0.000 2024-09-13T11:33:00.273 INFO:teuthology.orchestra.run.smithi162.stdout: remote refid st t when poll reach delay offset jitter 2024-09-13T11:33:00.273 INFO:teuthology.orchestra.run.smithi162.stdout:============================================================================== 2024-09-13T11:33:00.273 INFO:teuthology.orchestra.run.smithi162.stdout:*hv01.front.sepi 67.205.162.81 3 u 6 64 377 0.083 -5.028 2.667 2024-09-13T11:33:00.273 INFO:teuthology.orchestra.run.smithi162.stdout:+hv02.front.sepi 63.231.80.2 3 u 64 64 377 0.080 -5.120 2.863 2024-09-13T11:33:00.273 INFO:teuthology.orchestra.run.smithi162.stdout:+hv03.front.sepi 74.6.168.72 3 u 65 64 377 0.096 +0.568 3.050 2024-09-13T11:33:00.274 INFO:teuthology.orchestra.run.smithi162.stdout: hv04.front.sepi .INIT. 16 u - 256 0 0.000 +0.000 0.000 2024-09-13T11:33:00.275 INFO:teuthology.orchestra.run.smithi177.stdout: remote refid st t when poll reach delay offset jitter 2024-09-13T11:33:00.275 INFO:teuthology.orchestra.run.smithi177.stdout:============================================================================== 2024-09-13T11:33:00.275 INFO:teuthology.orchestra.run.smithi177.stdout:+hv01.front.sepi 67.205.162.81 3 u 3 64 377 0.078 -0.944 4.367 2024-09-13T11:33:00.275 INFO:teuthology.orchestra.run.smithi177.stdout:+hv02.front.sepi 63.231.80.2 3 u 61 64 377 0.087 -1.499 3.588 2024-09-13T11:33:00.275 INFO:teuthology.orchestra.run.smithi177.stdout:*hv03.front.sepi 74.6.168.72 3 u 3 64 377 0.061 -1.408 3.553 2024-09-13T11:33:00.275 INFO:teuthology.orchestra.run.smithi177.stdout: hv04.front.sepi .INIT. 16 u - 256 0 0.000 +0.000 0.000 2024-09-13T11:33:00.276 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-09-13T11:33:00.285 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-09-13T11:33:00.285 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-09-13T11:33:00.294 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-09-13T11:33:00.303 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-09-13T11:33:00.312 INFO:teuthology.task.internal:Duration was 1330.212622 seconds 2024-09-13T11:33:00.313 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-09-13T11:33:00.322 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-09-13T11:33:00.322 DEBUG:teuthology.orchestra.run.smithi001:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-13T11:33:00.325 DEBUG:teuthology.orchestra.run.smithi092:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-13T11:33:00.327 DEBUG:teuthology.orchestra.run.smithi162:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-13T11:33:00.329 DEBUG:teuthology.orchestra.run.smithi177:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-13T11:33:00.366 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-09-13T11:33:00.367 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi001.front.sepia.ceph.com 2024-09-13T11:33:00.367 DEBUG:teuthology.orchestra.run.smithi001:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-13T11:33:00.411 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi092.front.sepia.ceph.com 2024-09-13T11:33:00.412 DEBUG:teuthology.orchestra.run.smithi092:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-13T11:33:00.421 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi162.front.sepia.ceph.com 2024-09-13T11:33:00.422 DEBUG:teuthology.orchestra.run.smithi162:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-13T11:33:00.432 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi177.front.sepia.ceph.com 2024-09-13T11:33:00.432 DEBUG:teuthology.orchestra.run.smithi177:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-13T11:33:00.443 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-09-13T11:33:00.443 DEBUG:teuthology.orchestra.run.smithi001:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-13T11:33:00.456 DEBUG:teuthology.orchestra.run.smithi092:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-13T11:33:00.467 DEBUG:teuthology.orchestra.run.smithi162:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-13T11:33:00.478 DEBUG:teuthology.orchestra.run.smithi177:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-13T11:33:00.513 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-09-13T11:33:00.513 DEBUG:teuthology.orchestra.run.smithi001:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-13T11:33:00.531 DEBUG:teuthology.orchestra.run.smithi092:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-13T11:33:00.539 DEBUG:teuthology.orchestra.run.smithi162:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-13T11:33:00.550 DEBUG:teuthology.orchestra.run.smithi177:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-13T11:33:00.625 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-09-13T11:33:00.635 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-09-13T11:33:00.635 DEBUG:teuthology.orchestra.run.smithi001:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-13T11:33:00.646 DEBUG:teuthology.orchestra.run.smithi092:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-13T11:33:00.657 DEBUG:teuthology.orchestra.run.smithi162:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-13T11:33:00.670 DEBUG:teuthology.orchestra.run.smithi177:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-13T11:33:00.683 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-09-13T11:33:00.693 DEBUG:teuthology.orchestra.run.smithi001:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-13T11:33:00.695 DEBUG:teuthology.orchestra.run.smithi092:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-13T11:33:00.703 DEBUG:teuthology.orchestra.run.smithi162:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-13T11:33:00.704 INFO:teuthology.orchestra.run.smithi001.stdout:kernel.core_pattern = core 2024-09-13T11:33:00.713 INFO:teuthology.orchestra.run.smithi092.stdout:kernel.core_pattern = core 2024-09-13T11:33:00.714 DEBUG:teuthology.orchestra.run.smithi177:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-13T11:33:00.721 INFO:teuthology.orchestra.run.smithi162.stdout:kernel.core_pattern = core 2024-09-13T11:33:00.735 INFO:teuthology.orchestra.run.smithi177.stdout:kernel.core_pattern = core 2024-09-13T11:33:00.753 DEBUG:teuthology.orchestra.run.smithi001:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-13T11:33:00.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T11:33:00.769 DEBUG:teuthology.orchestra.run.smithi092:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-13T11:33:00.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T11:33:00.781 DEBUG:teuthology.orchestra.run.smithi162:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-13T11:33:00.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T11:33:00.786 DEBUG:teuthology.orchestra.run.smithi177:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-13T11:33:00.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T11:33:00.800 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-09-13T11:33:00.810 INFO:teuthology.task.internal:Transferring archived files... 2024-09-13T11:33:00.811 DEBUG:teuthology.misc:Transferring archived files from smithi001:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/akupczyk-2024-09-13_09:37:56-rados-wip-aclamk-cbt-combined-distro-default-smithi/7903409/remote/smithi001 2024-09-13T11:33:00.812 DEBUG:teuthology.orchestra.run.smithi001:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-13T11:33:00.834 DEBUG:teuthology.misc:Transferring archived files from smithi092:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/akupczyk-2024-09-13_09:37:56-rados-wip-aclamk-cbt-combined-distro-default-smithi/7903409/remote/smithi092 2024-09-13T11:33:00.834 DEBUG:teuthology.orchestra.run.smithi092:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-13T11:33:00.855 DEBUG:teuthology.misc:Transferring archived files from smithi162:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/akupczyk-2024-09-13_09:37:56-rados-wip-aclamk-cbt-combined-distro-default-smithi/7903409/remote/smithi162 2024-09-13T11:33:00.856 DEBUG:teuthology.orchestra.run.smithi162:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-13T11:33:00.877 DEBUG:teuthology.misc:Transferring archived files from smithi177:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/akupczyk-2024-09-13_09:37:56-rados-wip-aclamk-cbt-combined-distro-default-smithi/7903409/remote/smithi177 2024-09-13T11:33:00.878 DEBUG:teuthology.orchestra.run.smithi177:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-13T11:33:00.900 INFO:teuthology.task.internal:Removing archive directory... 2024-09-13T11:33:00.900 DEBUG:teuthology.orchestra.run.smithi001:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-13T11:33:00.902 DEBUG:teuthology.orchestra.run.smithi092:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-13T11:33:00.904 DEBUG:teuthology.orchestra.run.smithi162:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-13T11:33:00.918 DEBUG:teuthology.orchestra.run.smithi177:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-13T11:33:00.945 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-09-13T11:33:00.957 INFO:teuthology.task.internal:Not uploading archives. 2024-09-13T11:33:00.957 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-09-13T11:33:00.967 INFO:teuthology.task.internal:Tidying up after the test... 2024-09-13T11:33:00.967 DEBUG:teuthology.orchestra.run.smithi001:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-13T11:33:00.969 DEBUG:teuthology.orchestra.run.smithi092:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-13T11:33:00.972 DEBUG:teuthology.orchestra.run.smithi162:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-13T11:33:00.973 INFO:teuthology.orchestra.run.smithi001.stdout: 658642 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 13 11:33 /home/ubuntu/cephtest 2024-09-13T11:33:00.973 DEBUG:teuthology.orchestra.run.smithi177:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-13T11:33:00.975 INFO:teuthology.orchestra.run.smithi092.stdout: 658623 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 13 11:33 /home/ubuntu/cephtest 2024-09-13T11:33:00.976 INFO:teuthology.orchestra.run.smithi162.stdout: 658622 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 13 11:33 /home/ubuntu/cephtest 2024-09-13T11:33:00.993 INFO:teuthology.orchestra.run.smithi177.stdout: 658640 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 13 11:33 /home/ubuntu/cephtest 2024-09-13T11:33:00.995 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-09-13T11:33:01.005 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-09-13T11:33:01.035 INFO:teuthology.run:Summary data: description: rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{max-simultaneous-scrubs-5} backoff/peering ceph clusters/{fixed-4 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/classic msgr-failures/osd-delay msgr/async-v1only objectstore/{bluestore-options/write$/{write_v2} bluestore/bluestore-stupid} rados supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/admin_socket_objecter_requests} duration: 1330.2126216888428 flavor: default owner: scheduled_akupczyk@teuthology success: true 2024-09-13T11:33:01.035 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-13T11:33:01.111 INFO:teuthology.run:pass